Max level shown:
1 starting January 28, 2025 at 09:54:11 PM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.26870692
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.022513928
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.021545315
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start ds: 8820
14 start ds: 8830
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8810
17 Starting downstairs at port 8820
18 Make output file at "/tmp/dsc/downstairs-8820.txt"
19 Make output file at "/tmp/dsc/downstairs-8810.txt"
20 Starting downstairs at port 8830
21 Make output file at "/tmp/dsc/downstairs-8830.txt"
22 [8810][0] initial start wait reports Starting
23 [8820][1] initial start wait reports Starting
24 [8830][2] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1192)
26 [8820][1] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1193)
28 [8830][2] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1194)
30 [8810][0] initial start wait reports Running
31 All downstairs are running
32 Jan 28 21:54:12.085 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Jan 28 21:54:17.151 INFO accepted connection, remote_addr: 127.0.0.1:59731, local_addr: 127.0.0.1:9998
36 Jan 28 21:54:17.153 INFO request completed, latency_us: 718, response_code: 200, uri: /regioninfo, method: GET, req_id: fd61d42b-da66-4607-8e09-6525b2ebe21c, remote_addr: 127.0.0.1:59731, local_addr: 127.0.0.1:9998
37 Jan 28 21:54:17.155 INFO request completed, latency_us: 92, response_code: 200, uri: /regioncount, method: GET, req_id: 7d6abd65-7aa0-4417-b67f-330dd169a153, remote_addr: 127.0.0.1:59731, local_addr: 127.0.0.1:9998
382025-01-28T21:54:17.155ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-01-28T21:54:17.155ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Jan 28 21:54:17.156 INFO request completed, latency_us: 123, response_code: 200, uri: /port/cid/0, method: GET, req_id: 8a7a5a47-6bed-41b1-89d1-2f8ac7396fd8, remote_addr: 127.0.0.1:59731, local_addr: 127.0.0.1:9998
41 Jan 28 21:54:17.156 INFO request completed, latency_us: 96, response_code: 200, uri: /port/cid/1, method: GET, req_id: 51d2c1a5-26a6-4fb6-b7da-02d90607630f, remote_addr: 127.0.0.1:59731, local_addr: 127.0.0.1:9998
42 Jan 28 21:54:17.157 INFO request completed, latency_us: 90, response_code: 200, uri: /port/cid/2, method: GET, req_id: f279838a-d71a-4e0a-85d5-459672894f10, remote_addr: 127.0.0.1:59731, local_addr: 127.0.0.1:9998
432025-01-28T21:54:17.157ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-01-28T21:54:17.158ZINFOcrucible: Upstairs starts
452025-01-28T21:54:17.158ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "7817632486f84ab6865c5ce71e16196ec0c710fc", git_commit_timestamp: "2025-01-28T21:38:04.000000000Z", git_branch: "mkeeter/early-io-rejection", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-01-28T21:54:17.158ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-01-28T21:54:17.158ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-01-28T21:54:17.159ZINFOcrucible: Crucible a4545362-d178-4b95-896b-66dcd9ef73ea has session id: abae0d2d-3a82-4705-a9b8-0e138c018a34 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
492025-01-28T21:54:17.159ZINFOcrucible: Upstairs opts: Upstairs UUID: a4545362-d178-4b95-896b-66dcd9ef73ea, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
502025-01-28T21:54:17.159ZINFOcrucible: Crucible stats registered with UUID: a4545362-d178-4b95-896b-66dcd9ef73ea session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
512025-01-28T21:54:17.159ZWARNcrucible: could not find Downstairs address for Nexus session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
522025-01-28T21:54:17.160ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-01-28T21:54:17.160ZINFOcrucible: The guest has requested activation with gen:1
542025-01-28T21:54:17.160ZINFOcrucible: a4545362-d178-4b95-896b-66dcd9ef73ea active request set session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
552025-01-28T21:54:17.160ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
562025-01-28T21:54:17.160ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
572025-01-28T21:54:17.160ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
582025-01-28T21:54:17.160ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
592025-01-28T21:54:17.161ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
602025-01-28T21:54:17.161ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
612025-01-28T21:54:17.161ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
622025-01-28T21:54:17.161ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
632025-01-28T21:54:17.161ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
642025-01-28T21:54:17.161ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61083) = io task client = 0 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
652025-01-28T21:54:17.161ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:57376) = io task client = 1 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
662025-01-28T21:54:17.161ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58299) = io task client = 2 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
672025-01-28T21:54:17.163ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
682025-01-28T21:54:17.163ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
692025-01-28T21:54:17.163ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
702025-01-28T21:54:17.164ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
712025-01-28T21:54:17.164ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
722025-01-28T21:54:17.164ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
732025-01-28T21:54:17.165ZINFOcrucible: Waiting for more clients to be ready session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
742025-01-28T21:54:17.165ZINFOcrucible: Waiting for more clients to be ready session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
752025-01-28T21:54:17.165ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
762025-01-28T21:54:17.165ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
772025-01-28T21:54:17.165ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
782025-01-28T21:54:17.165ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
792025-01-28T21:54:17.165ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
802025-01-28T21:54:17.165ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
812025-01-28T21:54:17.165ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
822025-01-28T21:54:17.165ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
832025-01-28T21:54:17.165ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
842025-01-28T21:54:17.165ZINFOcrucible: Max found gen is 1 = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
852025-01-28T21:54:17.165ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
862025-01-28T21:54:17.165ZINFOcrucible: Next flush: 1 = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
872025-01-28T21:54:17.165ZINFOcrucible: All extents match = downstairs session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
882025-01-28T21:54:17.165ZINFOcrucible: No downstairs reconciliation required session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
892025-01-28T21:54:17.165ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
902025-01-28T21:54:17.165ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
912025-01-28T21:54:17.165ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
922025-01-28T21:54:17.165ZINFOcrucible: All required reconciliation work is completed session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
932025-01-28T21:54:17.165ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
94 {"msg":"Activate successful
95 a4545362-d178-4b95-896b-66dcd9ef73ea is now active with session: abae0d2d-3a82-4705-a9b8-0e138c018a34"Wait for a query_work_queue command to finish before sending IO
96 ,"v":0,"name":"crucible","level":30,"time":"2025-01-28T21:54:17.165647648Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1198,"session_id":"abae0d2d-3a82-4705-a9b8-0e138c018a34"}
972025-01-28T21:54:17.165ZINFOcrucible: Set Active after no reconciliation session_id = abae0d2d-3a82-4705-a9b8-0e138c018a34
982025-01-28T21:54:17.165ZINFOcrucible: The guest has finished waiting for activation with:1
99 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
100 Fill test
101 Read and Verify all blocks (0..5000 range:false)
102 Wrote out file "/tmp/test_replay_verify.log"
103 CLIENT: Tests done. All submitted work has been ACK'd
104 ----------------------------------------------------------------
105 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
106 States: Active Active Active
107 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
108 1021 Acked Read 256 Done Done Done false
109 1022 Acked Read 256 Done Done Done false
110 1023 Acked Read 256 Done Done Done false
111 1024 Acked Read 256 Done Done Done false
112 1025 Acked Read 256 Done Done Done false
113 1026 Acked Read 256 Done Done Done false
114 1027 Acked Read 256 Done Done Done false
115 1028 Acked Read 256 Done Done Done false
116 1029 Acked Read 256 Done Done Done false
117 1030 Acked Read 256 Done Done Done false
118 1031 Acked Read 256 Done Done Done false
119 1032 Acked Read 256 Done Done Done false
120 1033 Acked Read 256 Done Done Done false
121 1034 Acked Read 256 Done Done Done false
122 1035 Acked Read 256 Done Done Done false
123 1036 Acked Read 256 Done Done Done false
124 1037 Acked Read 136 Done Done Done false
125 1038 Acked Read 256 Done Done Done false
126 1039 Acked Read 256 Done Done Done false
127 1040 Acked Read 256 Done Done Done false
128 STATES DS:0 DS:1 DS:2 TOTAL
129 Sent 0 0 0 0
130 Done 20 20 20 60
131 Skipped 0 0 0 0
132 Error 0 0 0 0
133 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
134 Downstairs last five completed: 1020 1019 1018 1017 1016
135 Upstairs last five completed: 1040 1039 1038 1037 1036
136 CLIENT: Up:0 ds:20 act:3
137 ----------------------------------------------------------------
138 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
139 Downstairs last five completed: 1041 1040 1039 1038 1037
140 Upstairs last five completed: 1041 1040 1039 1038 1037
141 CLIENT: Up:0 ds:0 act:3
142 CLIENT: All crucible jobs finished, exiting program
143 Replay loop starts now January 28, 2025 at 09:54:21 PM UTC
144 Jan 28 21:54:21.829 INFO accepted connection, remote_addr: 127.0.0.1:43826, local_addr: 127.0.0.1:9998
145 Jan 28 21:54:21.830 INFO request completed, latency_us: 163, response_code: 200, uri: /regioninfo, method: GET, req_id: 1d3455c3-7e47-4c89-855f-631dc3e48cba, remote_addr: 127.0.0.1:43826, local_addr: 127.0.0.1:9998
146 Jan 28 21:54:21.831 INFO request completed, latency_us: 105, response_code: 200, uri: /regioncount, method: GET, req_id:{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:54:21.830757714Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"task":"crutest"}
147 3c715670-d20b-412d-a99c-aa99e4610615, remote_addr: 127.0.0.1:43826, local_addr: 127.0.0.1:9998
1482025-01-28T21:54:21.831ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
149 Jan 28 21:54:21.831 INFO request completed, latency_us: 114, response_code: 200, uri: /port/cid/0, method: GET, req_id: e99ce8a8-ca28-4368-a2a4-29e0c2540f8e, remote_addr: 127.0.0.1:43826, local_addr: 127.0.0.1:9998
150 Jan 28 21:54:21.832 INFO request completed, latency_us: 92, response_code: 200, uri: /port/cid/1, method: GET, req_id: cdde3b32-16db-4f9e-ad4f-df4ebe3ecd6b, remote_addr: 127.0.0.1:43826, local_addr: 127.0.0.1:9998
151 Jan 28 21:54:21.832 INFO request completed, latency_us: 90, response_code: 200, uri: /port/cid/2, method: GET, req_id: 927925de-2280-46ad-9bc0-7ada04a130a8, remote_addr: 127.0.0.1:43826, local_addr: 127.0.0.1:9998
1522025-01-28T21:54:21.833ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1532025-01-28T21:54:21.833ZINFOcrucible: Upstairs starts
1542025-01-28T21:54:21.833ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "7817632486f84ab6865c5ce71e16196ec0c710fc", git_commit_timestamp: "2025-01-28T21:38:04.000000000Z", git_branch: "mkeeter/early-io-rejection", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1552025-01-28T21:54:21.833ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1562025-01-28T21:54:21.833ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1572025-01-28T21:54:21.833ZINFOcrucible: Crucible 3777bfc0-2609-43b8-98c1-2062d11ba149 has session id: 97f0d889-45eb-4682-bc34-b4ef4ba454a8 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1582025-01-28T21:54:21.833ZINFOcrucible: Upstairs opts: Upstairs UUID: 3777bfc0-2609-43b8-98c1-2062d11ba149, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1592025-01-28T21:54:21.834ZINFOcrucible: Crucible stats registered with UUID: 3777bfc0-2609-43b8-98c1-2062d11ba149 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1602025-01-28T21:54:21.834ZWARNcrucible: could not find Downstairs address for Nexus session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1612025-01-28T21:54:21.835ZINFOcrucible: Downstairs in volume = 3 task = crutest
1622025-01-28T21:54:21.835ZINFOcrucible: The guest has requested activation with gen:2
1632025-01-28T21:54:21.835ZINFOcrucible: 3777bfc0-2609-43b8-98c1-2062d11ba149 active request set session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1642025-01-28T21:54:21.835ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1652025-01-28T21:54:21.835ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1662025-01-28T21:54:21.835ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1672025-01-28T21:54:21.835ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1682025-01-28T21:54:21.835ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1692025-01-28T21:54:21.835ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1702025-01-28T21:54:21.835ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1712025-01-28T21:54:21.835ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1722025-01-28T21:54:21.835ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1732025-01-28T21:54:21.835ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49366) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1742025-01-28T21:54:21.835ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44711) = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1752025-01-28T21:54:21.835ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62549) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1762025-01-28T21:54:21.836ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1772025-01-28T21:54:21.836ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1782025-01-28T21:54:21.836ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1792025-01-28T21:54:21.837ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1802025-01-28T21:54:21.837ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1812025-01-28T21:54:21.837ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1822025-01-28T21:54:21.837ZINFOcrucible: Waiting for more clients to be ready session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1832025-01-28T21:54:21.838ZINFOcrucible: Waiting for more clients to be ready session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1842025-01-28T21:54:21.838ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1852025-01-28T21:54:21.838ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1862025-01-28T21:54:21.838ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1872025-01-28T21:54:21.838ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1882025-01-28T21:54:21.838ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1892025-01-28T21:54:21.838ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1902025-01-28T21:54:21.838ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1912025-01-28T21:54:21.838ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1922025-01-28T21:54:21.838ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
193 {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:54:21.838324416Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"downstairs","":"downstairs"Activate successful
194 ,"Wait for a query_work_queue command to finish before sending IO
195 session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
1962025-01-28T21:54:21.838ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1972025-01-28T21:54:21.838ZINFOcrucible: Next flush: 2 = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1982025-01-28T21:54:21.838ZINFOcrucible: All extents match = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1992025-01-28T21:54:21.838ZINFOcrucible: No downstairs reconciliation required session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2002025-01-28T21:54:21.838ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2012025-01-28T21:54:21.838ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2022025-01-28T21:54:21.838ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2032025-01-28T21:54:21.838ZINFOcrucible: All required reconciliation work is completed session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2042025-01-28T21:54:21.838ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2052025-01-28T21:54:21.838ZINFOcrucible: 3777bfc0-2609-43b8-98c1-2062d11ba149 is now active with session: 97f0d889-45eb-4682-bc34-b4ef4ba454a8 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2062025-01-28T21:54:21.838ZINFOcrucible: Set Active after no reconciliation session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2072025-01-28T21:54:21.838ZINFOcrucible: The guest has finished waiting for activation with:2
208 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
209 Loading write count information from file "/tmp/test_replay_verify.log"
210 Jan 28 21:54:21.872 INFO accepted connection, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
211 Main task has work to do, go find it
212 got dsc Stop(1)
213 Jan 28 21:54:21.873stop 1
214 INFO request completed, latency_us: 237, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 34135861-c5e7-4c69-acc0-f846d039257e, remote_addr: 127.0.0.[8820] Got stop action so:false kr:false
215 1:36665, local_addr: 127.0.0.1:9998
2162025-01-28T21:54:21.874ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
217 Jan 28 21:54:21.874 INFO request completed, latency_us: 191, response_code: 200, uri: /state/cid/1, method{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:54:21.874572455Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
218 : GET, req_id: 97e9839c-b33f-4b88-888e-df112fc6bcf2, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
2192025-01-28T21:54:21.874ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2202025-01-28T21:54:21.874ZINFOcrucible: client task is exiting = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
221 [8820] Exited with: ExitStatus(unix_wait_status(9))
222 [8820][1] reports Exit
223 Jan 28 21:54:25.876 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/1, method: GET, req_id: 0d4e8879-2399-493b-87cc-f439e003774f, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
224 001/300 Write block 4877 len 32768 data: 2 2 2 2 2 2 2 2
225 002/300 Write block 3760 len 36864 data: 2 2 2 2 2 2 2 2 2
226 003/300 Write block 4477 len 32768 data: 2 2 2 2 2 2 2 2
227 004/300 Write block 1904 len 24576 data: 2 2 2 2 2 2
228 005/300 Write block 4532 len 20480 data: 2 2 2 2 2
229 006/300 Read block 4045 len 8192
230 007/300 Write block 1052 len 36864 data: 2 2 2 2 2 2 2 2 2
231 008/300 Read block 1786 len 28672
232 009/300 Read block 4947 len 16384
233 010/300 Flush
234 011/300 Read block 4365 len 4096
235 012/300 Read block 681 len 16384
236 013/300 Write block 2231 len 40960 data: 2 2 2 2 2 2 2 2 2 2
237 014/300 Write block 2597 len 20480 data: 2 2 2 2 2
238 015/300 Read block 2571 len 8192
239 016/300 Write block 1353 len 16384 data: 2 2 2 2
240 017/300 Write block 64 len 8192 data: 2 2
241 018/300 Write block 666 len 20480 data: 2 2 2 2 2
242 019/300 Write block 4164 len 4096 data: 2
243 020/300 Write block 3528 len 28672 data: 2 2 2 2 2 2 2
244 021/300 Write block 2151 len 40960 data: 2 2 2 2 2 2 2 2 2 2
245 022/300 Write block 538 len 28672 data: 2 2 2 2 2 2 2
246 023/300 Write block 243 len 32768 data: 2 2 2 2 2 2 2 2
247 024/300 Read block 4241 len 32768
248 025/300 Read block 4430 len 8192
249 026/300 Write block 4274 len 16384 data: 2 2 2 2
250 027/300 Read block 1170 len 28672
251 028/300 Write block 1717 len 28672 data: 2 2 2 2 2 2 2
252 029/300 Write block 3996 len 40960 data: 2 2 2 2 2 2 2 2 2 2
253 030/300 Write block 2857 len 4096 data: 2
254 031/300 Read block 2702 len 24576
255 032/300 Write block 2419 len 4096 data: 2
256 033/300 Write block 3880 len 20480 data: 2 2 2 2 2
257 034/300 Write block 3393 len 28672 data: 2 2 2 2 2 2 2
258 035/300 Flush
259 036/300 Write block 2069 len 8192 data: 2 2
260 037/300 Flush
261 038/300 Read block 4169 len 4096
262 039/300 Write block 2591 len 12288 data: 2 2 2
263 040/300 Read block 4049 len 40960
264 041/300 Read block 146 len 28672
265 042/300 Write block 1149 len 40960 data: 2 2 2 2 2 2 2 2 2 2
266 043/300 Read block 34 len 4096
267 044/300 Read block 2479 len 32768
268 045/300 Write block 1332 len 32768 data: 2 2 2 2 2 2 2 2
269 046/300 Read block 1419 len 36864
270 047/300 Read block 3846 len 12288
271 048/300 Write block 265 len 16384 data: 2 2 2 2
272 049/300 Read block 4211 len 16384
273 050/300 Read block 4075 len 20480
274 051/300 Read block 157 len 40960
275 052/300 Read block 2658 len 8192
276 053/300 Read block 500 len 4096
277 054/300 Write block 4334 len 36864 data: 2 2 2 2 2 2 2 2 2
278 055/300 Read block 618 len 16384
279 056/300 Write block 3355 len 32768 data: 2 2 2 2 2 2 2 2
280 057/300 Write block 3957 len 40960 data: 2 2 2 2 2 2 2 2 2 2
281 058/300 Read block 4796 len 4096
282 059/300 Write block 1358 len 8192 data: 2 2
283 060/300 Write block 628 len 40960 data: 2 2 2 2 2 2 2 2 2 2
284 061/300 Write block 3941 len 40960 data: 2 2 2 2 2 2 2 2 2 2
285 062/300 Read block 2470 len 16384
286 063/300 Read block 3934 len 24576
287 064/300 Read block 3499 len 36864
288 065/300 Flush
289 066/300 Write block 4203 len 32768 data: 2 2 2 2 2 2 2 2
290 067/300 Read block 2509 len 12288
291 068/300 Read block 2972 len 12288
292 069/300 Write block 2325 len 12288 data: 2 2 2
293 070/300 Write block 3767 len 20480 data: 3 3 2 2 2
294 071/300 Write block 2135 len 40960 data: 2 2 2 2 2 2 2 2 2 2
295 072/300 Read block 3337 len 20480
296 073/300 Write block 1762 len 32768 data: 2 2 2 2 2 2 2 2
297 074/300 Read block 2100 len 40960
298 075/300 Write block 2202 len 12288 data: 2 2 2
299 076/300 Write block 2124 len 4096 data: 2
300 077/300 Read block 721 len 28672
301 078/300 Flush
302 079/300 Read block 4762 len 24576
303 080/300 Write block 3668 len 32768 data: 2 2 2 2 2 2 2 2
304 081/300 Flush
305 082/300 Read block 3134 len 36864
306 083/300 Read block 3413 len 24576
307 084/300 Read block 3007 len 4096
308 085/300 Read block 3669 len 40960
309 086/300 Write block 2159 len 40960 data: 3 3 2 2 2 2 2 2 2 2
310 087/300 Read block 3422 len 36864
311 088/300 Read block 3324 len 16384
312 089/300 Flush
313 090/300 Read block 1825 len 20480
314 091/300 Read block 3922 len 36864
315 092/300 Flush
316 093/300 Flush
317 094/300 Read block 4637 len 4096
318 095/300 Write block 62 len 36864 data: 2 2 3 3 2 2 2 2 2
319 096/300 Read block 3580 len 24576
320 097/300 Write block 3059 len 40960 data: 2 2 2 2 2 2 2 2 2 2
321 098/300 Read block 2639 len 36864
322 099/300 Write block 4981 len 20480 data: 2 2 2 2 2
323 100/300 Read block 4490 len 20480
324 101/300 Write block 1364 len 32768 data: 2 2 2 2 2 2 2 2
325 102/300 Read block 687 len 12288
326 103/300 Read block 4144 len 40960
327 104/300 Write block 604 len 20480 data: 2 2 2 2 2
328 105/300 Flush
329 106/300 Read block 552 len 8192
330 107/300 Write block 473 len 32768 data: 2 2 2 2 2 2 2 2
331 108/300 Read block 1171 len 24576
332 109/300 Write block 4619 len 12288 data: 2 2 2
333 110/300 Read block 4970 len 4096
334 111/300 Write block 1517 len 36864 data: 2 2 2 2 2 2 2 2 2
335 112/300 Read block 4189 len 8192
336 113/300 Read block 1352 len 16384
337 114/300 Read block 277 len 20480
338 115/300 Read block 964 len 40960
339 116/300 Read block 1991 len 24576
340 117/300 Flush
341 118/300 Write block 316 len 20480 data: 2 2 2 2 2
342 119/300 Write block 4166 len 8192 data: 2 2
343 120/300 Read block 3606 len 28672
344 121/300 Read block 2387 len 8192
345 122/300 Read block 919 len 16384
346 123/300 Write block 4073 len 16384 data: 2 2 2 2
347 124/300 Read block 2982 len 40960
348 125/300 Read block 507 len 20480
349 126/300 Flush
350 127/300 Read block 303 len 40960
351 128/300 Write block 3979 len 24576 data: 2 2 2 2 2 2
352 129/300 Read block 270 len 8192
353 130/300 Read block 3179 len 28672
354 131/300 Read block 1024 len 8192
355 132/300 Write block 4732 len 24576 data: 2 2 2 2 2 2
356 133/300 Read block 3492 len 24576
357 134/300 Read block 3086 len 4096
358 135/300 Read block 96 len 16384
359 136/300 Write block 2846 len 20480 data: 2 2 2 2 2
360 137/300 Write block 626 len 4096 data: 2
361 138/300 Write block 3349 len 4096 data: 2
362 139/300 Read block 4494 len 16384
363 140/300 Write block 3320 len 28672 data: 2 2 2 2 2 2 2
364 141/300 Flush
365 142/300 Read block 4954 len 12288
366 143/300 Write block 2846 len 4096 data: 3
367 144/300 Write block 3537 len 16384 data: 2 2 2 2
368 145/300 Write block 1919 len 12288 data: 2 2 2
369 146/300 Read block 4294 len 32768
370 147/300 Read block 2947 len 32768
371 148/300 Read block 4284 len 28672
372 149/300 Read block 2090 len 32768
373 150/300 Write block 1325 len 4096 data: 2
374 151/300 Write block 3895 len 32768 data: 2 2 2 2 2 2 2 2
375 152/300 Read block 1668 len 8192
376 153/300 Read block 30 len 40960
377 154/300 Write block 3322 len 40960 data: 3 3 3 3 3 2 2 2 2 2
378 155/300 Write block 598 len 28672 data: 2 2 2 2 2 2 3
379 156/300 Read block 3361 len 20480
380 157/300 Write block 1711 len 4096 data: 2
381 158/300 Read block 821 len 36864
382 159/300 Write block 3514 len 40960 data: 2 2 2 2 2 2 2 2 2 2
383 160/300 Read block 4650 len 20480
384 161/300 Write block 689 len 28672 data: 2 2 2 2 2 2 2
385 162/300 Read block 232 len 40960
386 163/300 Flush
387 164/300 Read block 7 len 32768
388 165/300 Write block 4840 len 32768 data: 2 2 2 2 2 2 2 2
389 166/300 Write block 2119 len 16384 data: 2 2 2 2
390 167/300 Write block 3021 len 24576 data: 2 2 2 2 2 2
391 168/300 Read block 3403 len 8192
392 169/300 Write block 1407 len 36864 data: 2 2 2 2 2 2 2 2 2
393 170/300 Write block 3085 len 8192 data: 2 2
394 171/300 Read block 620 len 16384
395 172/300 Read block 255 len 20480
396 173/300 Write block 3668 len 4096 data: 3
397 174/300 Write block 4102 len 28672 data: 2 2 2 2 2 2 2
398 175/300 Write block 24 len 4096 data: 2
399 176/300 Read block 1005 len 20480
400 177/300 Flush
401 178/300 Write block 4292 len 36864 data: 2 2 2 2 2 2 2 2 2
402 179/300 Read block 115 len 8192
403 180/300 Read block 196 len 16384
404 181/300 Write block 868 len 16384 data: 2 2 2 2
405 182/300 Read block 4359 len 32768
406 183/300 Read block 2634 len 12288
407 184/300 Write block 4865 len 28672 data: 2 2 2 2 2 2 2
408 185/300 Read block 430 len 28672
409 186/300 Write block 1942 len 12288 data: 2 2 2
410 187/300 Read block 451 len 32768
411 188/300 Write block 3356 len 12288 data: 3 3 3
412 189/300 Read block 2451 len 12288
413 190/300 Flush
414 191/300 Read block 130 len 8192
415 192/300 Read block 4621 len 8192
416 193/300 Read block 3773 len 16384
417 194/300 Read block 1092 len 8192
418 195/300 Write block 2988 len 32768 data: 2 2 2 2 2 2 2 2
419 196/300 Read block 4878 len 24576
420 197/300 Read block 3094 len 28672
421 198/300 Write block 963 len 32768 data: 2 2 2 2 2 2 2 2
422 199/300 Read block 700 len 4096
423 200/300 Read block 3179 len 12288
424 201/300 Read block 4767 len 24576
425 202/300 Read block 2794 len 28672
426 203/300 Read block 2964 len 8192
427 204/300 Write block 1036 len 36864 data: 2 2 2 2 2 2 2 2 2
428 205/300 Write block 2674 len 32768 data: 2 2 2 2 2 2 2 2
429 206/300 Read block 3059 len 8192
430 207/300 Read block 912 len 28672
431 208/300 Read block 1269 len 16384
432 209/300 Read block 1377 len 4096
433 210/300 Write block 3713 len 40960 data: 2 2 2 2 2 2 2 2 2 2
434 211/300 Write block 4479 len 32768 data: 3 3 3 3 3 3 2 2
435 212/300 Read block 3957 len 24576
436 213/300 Read block 1854 len 12288
437 214/300 Read block 4317 len 4096
438 215/300 Read block 3088 len 24576
439 216/300 Write block 4183 len 20480 data: 2 2 2 2 2
440 217/300 Read block 418 len 20480
441 218/300 Write block 3524 len 16384 data: 2 2 2 2
442 219/300 Read block 2255 len 28672
443 220/300 Read block 2531 len 8192
444 221/300 Write block 4372 len 20480 data: 2 2 2 2 2
445 222/300 Read block 3749 len 12288
446 223/300 Write block 3430 len 16384 data: 2 2 2 2
447 224/300 Flush
448 225/300 Read block 4800 len 8192
449 226/300 Write block 4074 len 32768 data: 3 3 3 2 2 2 2 2
450 227/300 Write block 4197 len 4096 data: 2
451 228/300 Write block 298 len 36864 data: 2 2 2 2 2 2 2 2 2
452 229/300 Write block 1344 len 12288 data: 2 2 2
453 230/300 Flush
454 231/300 Write block 4690 len 12288 data: 2 2 2
455 232/300 Read block 2997 len 40960
456 233/300 Write block 3624 len 28672 data: 2 2 2 2 2 2 2
457 234/300 Read block 2610 len 8192
458 235/300 Read block 420 len 32768
459 236/300 Write block 4794 len 8192 data: 2 2
460 237/300 Read block 534 len 28672
461 238/300 Write block 2343 len 20480 data: 2 2 2 2 2
462 239/300 Write block 2329 len 24576 data: 2 2 2 2 2 2
463 240/300 Read block 3442 len 8192
464 241/300 Write block 1527 len 8192 data: 2 2
465 242/300 Flush
466 243/300 Read block 2883 len 12288
467 244/300 Read block 3036 len 20480
468 245/300 Write block 1108 len 24576 data: 2 2 2 2 2 2
469 246/300 Read block 2999 len 8192
470 247/300 Read block 3504 len 4096
471 248/300 Read block 2611 len 12288
472 249/300 Write block 1051 len 28672 data: 2 3 3 3 3 3 3
473 250/300 Read block 892 len 24576
474 251/300 Flush
475 252/300 Write block 2407 len 32768 data: 2 2 2 2 2 2 2 2
476 253/300 Write block 852 len 24576 data: 2 2 2 2 2 2
477 254/300 Read block 2953 len 16384
478 255/300 Flush
479 256/300 Write block 3192 len 16384 data: 2 2 2 2
480 257/300 Write block 3243 len 16384 data: 2 2 2 2
481 258/300 Write block 1251 len 24576 data: 2 2 2 2 2 2
482 259/300 Read block 2602 len 40960
483 260/300 Write block 737 len 20480 data: 2 2 2 2 2
484 261/300 Write block 4947 len 4096 data: 2
485 262/300 Read block 1620 len 8192
486 263/300 Write block 885 len 20480 data: 2 2 2 2 2
487 264/300 Read block 3497 len 36864
488 265/300 Write block 2478 len 8192 data: 2 2
489 266/300 Write block 1629 len 20480 data: 2 2 2 2 2
490 267/300 Flush
491 268/300 Write block 4228 len 20480 data: 2 2 2 2 2
492 269/300 Write block 2275 len 28672 data: 2 2 2 2 2 2 2
493 270/300 Write block 3187 len 36864 data: 2 2 2 2 2 3 3 3 3
494 271/300 Flush
495 272/300 Read block 1785 len 28672
496 273/300 Read block 4784 len 8192
497 274/300 Write block 839 len 28672 data: 2 2 2 2 2 2 2
498 275/300 Flush
499 276/300 Read block 4532 len 28672
500 277/300 Write block 523 len 20480 data: 2 2 2 2 2
501 278/300 Read block 1166 len 36864
502 279/300 Write block 1057 len 40960 data: 4 3 3 3 2 2 2 2 2 2
503 280/300 Read block 2639 len 40960
504 281/300 Flush
505 282/300 Write block 979 len 32768 data: 2 2 2 2 2 2 2 2
506 283/300 Read block 3970 len 12288
507 284/300 Flush
508 285/300 Flush
509 286/300 Read block 684 len 24576
510 287/300 Read block 4938 len 4096
511 288/300 Read block 4666 len 28672
512 289/300 Write block 4887 len 8192 data: 2 2
513 290/300 Flush
514 291/300 Write block 1502 len 28672 data: 2 2 2 2 2 2 2
515 292/300 Read block 2925 len 28672
516 293/300 Write block 533 len 36864 data: 2 2 2 2 2 3 3 3 3
517 294/300 Read block 41 len 24576
518 295/300 Write block 2168 len 32768 data: 3 2 2 2 2 2 2 2
519 296/300 Write block 4480 len 16384 data: 4 4 4 4
520 297/300 Read block 1404 len 16384
521 298/300 Write block 1204 len 36864 data: 2 2 2 2 2 2 2 2 2
522 299/300 Read block 2258 len 36864
523 300/300 Flush
524 301/300 Write block 2884 len 16384 data: 2 2 2 2
525 Main task has work to do, go find it
526 got dsc Start(1)
527 start 1
528 Jan 28 21:54:26.112 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/1, method: POST, req_id: f43a467e-88f7-4082-86fb-2ca03557d8a3, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
529 Replay: started 1, returned:Ok(())
530 ----------------------------------------------------------------
531 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
532 States: Active Offline Active
533 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
534 1000 Acked Write 8 Done Sent Done false
535 1001 Acked Write 9 Done Sent Done false
536 1002 Acked Write 8 Done Sent Done false
537 1003 Acked Write 6 Done Sent Done false
538 1004 Acked Write 5 Done Sent Done false
539 1005 Acked Read 2 Done Sent Done false
540 1006 Acked Write 9 Done Sent Done false
541 1007 Acked Read 7 Done Sent Done false
542 1008 Acked Read 4 Done Sent Done false
543 1009 Acked Flush 0 Done Sent Done false
544 1010 Acked Read 1 Done Sent Done false
545 1011 Acked Read 4 Done Sent Done false
546 1012 Acked Write 10 Done Sent Done false
547 1013 Acked Write 5 Done Sent Done false
548 1014 Acked Read 2 Done Sent Done false
549 1015 Acked Write 4 Done Sent Done false
550 1016 Acked Write 2 Done Sent Done false
551 1017 Acked Write 5 Done Sent Done false
552 1018 Acked Write 1 Done Sent Done false
553 1019 Acked Write 7 Done Sent Done false
554 1020 Acked Write 10 Done Sent Done false
555 1021 Acked Write 7 Done Sent Done false
556 1022 Acked Write 8 Done Sent Done false
557 1023 Acked Read 8 Done Sent Done false
558 1024 Acked Read 2 Done Sent Done false
559 1025 Acked Write 4 Done Sent Done false
560 1026 Acked Read 7 Done Sent Done false
561 1027 Acked Write 7 Done Sent Done false
562 1028 Acked Write 10 Done Sent Done false
563 1029 Acked Write 1 Done Sent Done false
564 1030 Acked Read 6 Done Sent Done false
565 1031 Acked Write 1 Done Sent Done false
566 1032 Acked Write 5 Done Sent Done false
567 1033 Acked Write 7 Done Sent Done false
568 1034 Acked Flush 0 Done Sent Done false
569 1035 Acked Write 2 Done Sent Done false
570 1036 Acked Flush 0 Done Sent Done false
571 1037 Acked Read 1 Done Sent Done false
572 1038 Acked Write 3 Done Sent Done false
573 1039 Acked Read 10 Done Sent Done false
574 1040 Acked Read 7 Done Sent Done false
575 1041 Acked Write 10 Done Sent Done false
576 1042 Acked Read 1 Done Sent Done false
577 1043 Acked Read 8 Done Sent Done false
578 1044 Acked Write 8 Done Sent Done false
579 1045 Acked Read 9 Done Sent Done false
580 1046 Acked Read 3 Done Sent Done false
581 1047 Acked Write 4 Done Sent Done false
582 1048 Acked Read 4 Done Sent Done false
583 1049 Acked Read 5 Done Sent Done false
584 1050 Acked Read 10 Done Sent Done false
585 1051 Acked Read 2 Done Sent Done false
586 1052 Acked Read 1 Done Sent Done false
587 1053 Acked Write 9 Done Sent Done false
588 1054 Acked Read 4 Done Sent Done false
589 1055 Acked Write 8 Done Sent Done false
590 1056 Acked Write 10 Done Sent Done false
591 1057 Acked Read 1 Done Sent Done false
592 1058 Acked Write 2 Done Sent Done false
593 1059 Acked Write 10 Done Sent Done false
594 1060 Acked Write 10 Done Sent Done false
595 1061 Acked Read 4 Done Sent Done false
596 1062 Acked Read 6 Done Sent Done false
597 1063 Acked Read 9 Done Sent Done false
598 1064 Acked Flush 0 Done Sent Done false
599 1065 Acked Write 8 Done Sent Done false
600 1066 Acked Read 3 Done Sent Done false
601 1067 Acked Read 3 Done Sent Done false
602 1068 Acked Write 3 Done Sent Done false
603 1069 Acked Write 5 Done Sent Done false
604 1070 Acked Write 10 Done Sent Done false
605 1071 Acked Read 5 Done Sent Done false
606 1072 Acked Write 8 Done Sent Done false
607 1073 Acked Read 10 Done Sent Done false
608 1074 Acked Write 3 Done Sent Done false
609 1075 Acked Write 1 Done Sent Done false
610 1076 Acked Read 7 Done Sent Done false
611 1077 Acked Flush 0 Done Sent Done false
612 1078 Acked Read 6 Done Sent Done false
613 1079 Acked Write 8 Done Sent Done false
614 1080 Acked Flush 0 Done Sent Done false
615 1081 Acked Read 9 Done Sent Done false
616 1082 Acked Read 6 Done Sent Done false
617 1083 Acked Read 1 Done Sent Done false
618 1084 Acked Read 10 Done Sent Done false
619 1085 Acked Write 10 Done Sent Done false
620 1086 Acked Read 9 Done Sent Done false
621 1087 Acked Read 4 Done Sent Done false
622 1088 Acked Flush 0 Done Sent Done false
623 1089 Acked Read 5 Done Sent Done false
624 1090 Acked Read 9 Done Sent Done false
625 1091 Acked Flush 0 Done Sent Done false
626 1092 Acked Flush 0 Done Sent Done false
627 1093 Acked Read 1 Done Sent Done false
628 1094 Acked Write 9 Done Sent Done false
629 1095 Acked Read 6 Done Sent Done false
630 1096 Acked Write 10 Done Sent Done false
631 1097 Acked Read 9 Done Sent Done false
632 1098 Acked Write 5 Done Sent Done false
633 1099 Acked Read 5 Done Sent Done false
634 1100 Acked Write 8 Done Sent Done false
635 1101 Acked Read 3 Done Sent Done false
636 1102 Acked Read 10 Done Sent Done false
637 1103 Acked Write 5 Done Sent Done false
638 1104 Acked Flush 0 Done Sent Done false
639 1105 Acked Read 2 Done Sent Done false
640 1106 Acked Write 8 Done Sent Done false
641 1107 Acked Read 6 Done Sent Done false
642 1108 Acked Write 3 Done Sent Done false
643 1109 Acked Read 1 Done Sent Done false
644 1110 Acked Write 9 Done Sent Done false
645 1111 Acked Read 2 Done Sent Done false
646 1112 Acked Read 4 Done Sent Done false
647 1113 Acked Read 5 Done Sent Done false
648 1114 Acked Read 10 Done Sent Done false
649 1115 Acked Read 6 Done Sent Done false
650 1116 Acked Flush 0 Done Sent Done false
651 1117 Acked Write 5 Done Sent Done false
652 1118 Acked Write 2 Done Sent Done false
653 1119 Acked Read 7 Done Sent Done false
654 1120 Acked Read 2 Done Sent Done false
655 1121 Acked Read 4 Done Sent Done false
656 1122 Acked Write 4 Done Sent Done false
657 1123 Acked Read 10 Done Sent Done false
658 1124 Acked Read 5 Done Sent Done false
659 1125 Acked Flush 0 Done Sent Done false
660 1126 Acked Read 10 Done Sent Done false
661 1127 Acked Write 6 Done Sent Done false
662 1128 Acked Read 2 Done Sent Done false
663 1129 Acked Read 7 Done Sent Done false
664 1130 Acked Read 2 Done Sent Done false
665 1131 Acked Write 6 Done Sent Done false
666 1132 Acked Read 6 Done Sent Done false
667 1133 Acked Read 1 Done Sent Done false
668 1134 Acked Read 4 Done Sent Done false
669 1135 Acked Write 5 Done Sent Done false
670 1136 Acked Write 1 Done Sent Done false
671 1137 Acked Write 1 Done Sent Done false
672 1138 Acked Read 4 Done Sent Done false
673 1139 Acked Write 7 Done Sent Done false
674 1140 Acked Flush 0 Done Sent Done false
675 1141 Acked Read 3 Done Sent Done false
676 1142 Acked Write 1 Done Sent Done false
677 1143 Acked Write 4 Done Sent Done false
678 1144 Acked Write 3 Done Sent Done false
679 1145 Acked Read 8 Done Sent Done false
680 1146 Acked Read 8 Done Sent Done false
681 1147 Acked Read 7 Done Sent Done false
682 1148 Acked Read 8 Done Sent Done false
683 1149 Acked Write 1 Done Sent Done false
684 1150 Acked Write 8 Done Sent Done false
685 1151 Acked Read 2 Done Sent Done false
686 1152 Acked Read 10 Done Sent Done false
687 1153 Acked Write 10 Done Sent Done false
688 1154 Acked Write 7 Done Sent Done false
689 1155 Acked Read 5 Done Sent Done false
690 1156 Acked Write 1 Done Sent Done false
691 1157 Acked Read 9 Done Sent Done false
692 1158 Acked Write 10 Done Sent Done false
693 1159 Acked Read 5 Done Sent Done false
694 1160 Acked Write 7 Done Sent Done false
695 1161 Acked Read 10 Done Sent Done false
696 1162 Acked Flush 0 Done Sent Done false
697 1163 Acked Read 8 Done Sent Done false
698 1164 Acked Write 8 Done Sent Done false
699 1165 Acked Write 4 Done Sent Done false
700 1166 Acked Write 6 Done Sent Done false
701 1167 Acked Read 2 Done Sent Done false
702 1168 Acked Write 9 Done Sent Done false
703 1169 Acked Write 2 Done Sent Done false
704 1170 Acked Read 4 Done Sent Done false
705 1171 Acked Read 5 Done Sent Done false
706 1172 Acked Write 1 Done Sent Done false
707 1173 Acked Write 7 Done Sent Done false
708 1174 Acked Write 1 Done Sent Done false
709 1175 Acked Read 5 Done Sent Done false
710 1176 Acked Flush 0 Done Sent Done false
711 1177 Acked Write 9 Done Sent Done false
712 1178 Acked Read 2 Done Sent Done false
713 1179 Acked Read 4 Done Sent Done false
714 1180 Acked Write 4 Done Sent Done false
715 1181 Acked Read 8 Done Sent Done false
716 1182 Acked Read 3 Done Sent Done false
717 1183 Acked Write 7 Done Sent Done false
718 1184 Acked Read 7 Done Sent Done false
719 1185 Acked Write 3 Done Sent Done false
720 1186 Acked Read 8 Done Sent Done false
721 1187 Acked Write 3 Done Sent Done false
722 1188 Acked Read 3 Done Sent Done false
723 1189 Acked Flush 0 Done Sent Done false
724 1190 Acked Read 2 Done Sent Done false
725 1191 Acked Read 2 Done Sent Done false
726 1192 Acked Read 4 Done Sent Done false
727 1193 Acked Read 2 Done Sent Done false
728 1194 Acked Write 8 Done Sent Done false
729 1195 Acked Read 6 Done Sent Done false
730 1196 Acked Read 7 Done Sent Done false
731 1197 Acked Write 8 Done Sent Done false
732 1198 Acked Read 1 Done Sent Done false
733 1199 Acked Read 3 Done Sent Done false
734 1200 Acked Read 6 Done Sent Done false
735 1201 Acked Read 7 Done Sent Done false
736 1202 Acked Read 2 Done Sent Done false
737 1203 Acked Write 9 Done Sent Done false
738 1204 Acked Write 8 Done Sent Done false
739 1205 Acked Read 2 Done Sent Done false
740 1206 Acked Read 7 Done Sent Done false
741 1207 Acked Read 4 Done Sent Done false
742 1208 Acked Read 1 Done Sent Done false
743 1209 Acked Write 10 Done Sent Done false
744 1210 Acked Write 8 Done Sent Done false
745 1211 Acked Read 6 Done Sent Done false
746 1212 Acked Read 3 Done Sent Done false
747 1213 Acked Read 1 Done Sent Done false
748 1214 Acked Read 6 Done Sent Done false
749 1215 Acked Write 5 Done Sent Done false
750 1216 Acked Read 5 Done Sent Done false
751 1217 Acked Write 4 Done Sent Done false
752 1218 Acked Read 7 Done Sent Done false
753 1219 Acked Read 2 Done Sent Done false
754 1220 Acked Write 5 Done Sent Done false
755 1221 Acked Read 3 Done Sent Done false
756 1222 Acked Write 4 Done Sent Done false
757 1223 Acked Flush 0 Done Sent Done false
758 1224 Acked Read 2 Done Sent Done false
759 1225 Acked Write 8 Done Sent Done false
760 1226 Acked Write 1 Done Sent Done false
761 1227 Acked Write 9 Done Sent Done false
762 1228 Acked Write 3 Done Sent Done false
763 1229 Acked Flush 0 Done Sent Done false
764 1230 Acked Write 3 Done Sent Done false
765 1231 Acked Read 10 Done Sent Done false
766 1232 Acked Write 7 Done Sent Done false
767 1233 Acked Read 2 Done Sent Done false
768 1234 Acked Read 8 Done Sent Done false
769 1235 Acked Write 2 Done Sent Done false
770 1236 Acked Read 7 Done Sent Done false
771 1237 Acked Write 5 Done Sent Done false
772 1238 Acked Write 6 Done Sent Done false
773 1239 Acked Read 2 Done Sent Done false
774 1240 Acked Write 2 Done Sent Done false
775 1241 Acked Flush 0 Done Sent Done false
776 1242 Acked Read 3 Done Sent Done false
777 1243 Acked Read 5 Done Sent Done false
778 1244 Acked Write 6 Done Sent Done false
779 1245 Acked Read 2 Done Sent Done false
780 1246 Acked Read 1 Done Sent Done false
781 1247 Acked Read 3 Done Sent Done false
782 1248 Acked Write 7 Done Sent Done false
783 1249 Acked Read 6 Done Sent Done false
784 1250 Acked Flush 0 Done Sent Done false
785 1251 Acked Write 8 Done Sent Done false
786 1252 Acked Write 6 Done Sent Done false
787 1253 Acked Read 4 Done Sent Done false
788 1254 Acked Flush 0 Done Sent Done false
789 1255 Acked Write 4 Done Sent Done false
790 1256 Acked Write 4 Done Sent Done false
791 1257 Acked Write 6 Done Sent Done false
792 1258 Acked Read 10 Done Sent Done false
793 1259 Acked Write 5 Done Sent Done false
794 1260 Acked Write 1 Done Sent Done false
795 1261 Acked Read 2 Done Sent Done false
796 1262 Acked Write 5 Done Sent Done false
797 1263 Acked Read 9 Done Sent Done false
798 1264 Acked Write 2 Done Sent Done false
799 1265 Acked Write 5 Done Sent Done false
800 1266 Acked Flush 0 Done Sent Done false
801 1267 Acked Write 5 Done Sent Done false
802 1268 Acked Write 7 Done Sent Done false
803 1269 Acked Write 9 Done Sent Done false
804 1270 Acked Flush 0 Done Sent Done false
805 1271 Acked Read 7 Done Sent Done false
806 1272 Acked Read 2 Done Sent Done false
807 1273 Acked Write 7 Done Sent Done false
808 1274 Acked Flush 0 Done Sent Done false
809 1275 Acked Read 7 Done Sent Done false
810 1276 Acked Write 5 Done Sent Done false
811 1277 Acked Read 9 Done Sent Done false
812 1278 Acked Write 10 Done Sent Done false
813 1279 Acked Read 10 Done Sent Done false
814 1280 Acked Flush 0 Done Sent Done false
815 1281 Acked Write 8 Done Sent Done false
816 1282 Acked Read 3 Done Sent Done false
817 1283 Acked Flush 0 Done Sent Done false
818 1284 Acked Flush 0 Done Sent Done false
819 1285 Acked Read 6 Done Sent Done false
820 1286 Acked Read 1 Done Sent Done false
821 1287 Acked Read 7 Done Sent Done false
822 1288 Acked Write 2 Done Sent Done false
823 1289 Acked Flush 0 Done Sent Done false
824 1290 Acked Write 7 Done Sent Done false
825 1291 Acked Read 7 Done Sent Done false
826 1292 Acked Write 9 Done Sent Done false
827 1293 Acked Read 6 Done Sent Done false
828 1294 Acked Write 8 Done Sent Done false
829 1295 Acked Write 4 Done Sent Done false
830 1296 Acked Read 4 Done Sent Done false
831 1297 Acked Write 9 Done Sent Done false
832 1298 Acked Read 9 Done Sent Done false
833 1299 Acked Flush 0 Done Sent Done false
834 1300 Acked Write 4 Done Sent Done false
835 STATES DS:0 DS:1 DS:2 TOTAL
836 Sent 0 301 0 301
837 Done 301 0 301 602
838 Skipped 0 0 0 0
839 Error 0 0 0 0
840 Last Flush: Some(JobId(1299)) None Some(JobId(1299))
841 Downstairs last five completed:
842 Upstairs last five completed: 1300 1299 1298 1297 1296
843 CLIENT: Up:0 ds:301 act:2
8442025-01-28T21:54:26.614ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
8452025-01-28T21:54:27.342ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
8462025-01-28T21:54:28.343ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
8472025-01-28T21:54:29.344ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
848 [8820] got start action
849 [8820] I am going to restart
850 Starting downstairs at port 8820
851 Make output file at "/tmp/dsc/downstairs-8820.txt"
852 [8820][1] reports Starting
853 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1203)
854 [8820][1] reports Running
855 ----------------------------------------------------------------
856 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
857 States: Active Offline Active
858 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
859 1000 Acked Write 8 Done Sent Done false
860 1001 Acked Write 9 Done Sent Done false
861 1002 Acked Write 8 Done Sent Done false
862 1003 Acked Write 6 Done Sent Done false
863 1004 Acked Write 5 Done Sent Done false
864 1005 Acked Read 2 Done Sent Done false
865 1006 Acked Write 9 Done Sent Done false
866 1007 Acked Read 7 Done Sent Done false
867 1008 Acked Read 4 Done Sent Done false
868 1009 Acked Flush 0 Done Sent Done false
869 1010 Acked Read 1 Done Sent Done false
870 1011 Acked Read 4 Done Sent Done false
871 1012 Acked Write 10 Done Sent Done false
872 1013 Acked Write 5 Done Sent Done false
873 1014 Acked Read 2 Done Sent Done false
874 1015 Acked Write 4 Done Sent Done false
875 1016 Acked Write 2 Done Sent Done false
876 1017 Acked Write 5 Done Sent Done false
877 1018 Acked Write 1 Done Sent Done false
878 1019 Acked Write 7 Done Sent Done false
879 1020 Acked Write 10 Done Sent Done false
880 1021 Acked Write 7 Done Sent Done false
881 1022 Acked Write 8 Done Sent Done false
882 1023 Acked Read 8 Done Sent Done false
883 1024 Acked Read 2 Done Sent Done false
884 1025 Acked Write 4 Done Sent Done false
885 1026 Acked Read 7 Done Sent Done false
886 1027 Acked Write 7 Done Sent Done false
887 1028 Acked Write 10 Done Sent Done false
888 1029 Acked Write 1 Done Sent Done false
889 1030 Acked Read 6 Done Sent Done false
890 1031 Acked Write 1 Done Sent Done false
891 1032 Acked Write 5 Done Sent Done false
892 1033 Acked Write 7 Done Sent Done false
893 1034 Acked Flush 0 Done Sent Done false
894 1035 Acked Write 2 Done Sent Done false
895 1036 Acked Flush 0 Done Sent Done false
896 1037 Acked Read 1 Done Sent Done false
897 1038 Acked Write 3 Done Sent Done false
898 1039 Acked Read 10 Done Sent Done false
899 1040 Acked Read 7 Done Sent Done false
900 1041 Acked Write 10 Done Sent Done false
901 1042 Acked Read 1 Done Sent Done false
902 1043 Acked Read 8 Done Sent Done false
903 1044 Acked Write 8 Done Sent Done false
904 1045 Acked Read 9 Done Sent Done false
905 1046 Acked Read 3 Done Sent Done false
906 1047 Acked Write 4 Done Sent Done false
907 1048 Acked Read 4 Done Sent Done false
908 1049 Acked Read 5 Done Sent Done false
909 1050 Acked Read 10 Done Sent Done false
910 1051 Acked Read 2 Done Sent Done false
911 1052 Acked Read 1 Done Sent Done false
912 1053 Acked Write 9 Done Sent Done false
913 1054 Acked Read 4 Done Sent Done false
914 1055 Acked Write 8 Done Sent Done false
915 1056 Acked Write 10 Done Sent Done false
916 1057 Acked Read 1 Done Sent Done false
917 1058 Acked Write 2 Done Sent Done false
918 1059 Acked Write 10 Done Sent Done false
919 1060 Acked Write 10 Done Sent Done false
920 1061 Acked Read 4 Done Sent Done false
921 1062 Acked Read 6 Done Sent Done false
922 1063 Acked Read 9 Done Sent Done false
923 1064 Acked Flush 0 Done Sent Done false
924 1065 Acked Write 8 Done Sent Done false
925 1066 Acked Read 3 Done Sent Done false
926 1067 Acked Read 3 Done Sent Done false
927 1068 Acked Write 3 Done Sent Done false
928 1069 Acked Write 5 Done Sent Done false
929 1070 Acked Write 10 Done Sent Done false
930 1071 Acked Read 5 Done Sent Done false
931 1072 Acked Write 8 Done Sent Done false
932 1073 Acked Read 10 Done Sent Done false
933 1074 Acked Write 3 Done Sent Done false
934 1075 Acked Write 1 Done Sent Done false
935 1076 Acked Read 7 Done Sent Done false
936 1077 Acked Flush 0 Done Sent Done false
937 1078 Acked Read 6 Done Sent Done false
938 1079 Acked Write 8 Done Sent Done false
939 1080 Acked Flush 0 Done Sent Done false
940 1081 Acked Read 9 Done Sent Done false
941 1082 Acked Read 6 Done Sent Done false
942 1083 Acked Read 1 Done Sent Done false
943 1084 Acked Read 10 Done Sent Done false
944 1085 Acked Write 10 Done Sent Done false
945 1086 Acked Read 9 Done Sent Done false
946 1087 Acked Read 4 Done Sent Done false
947 1088 Acked Flush 0 Done Sent Done false
948 1089 Acked Read 5 Done Sent Done false
949 1090 Acked Read 9 Done Sent Done false
950 1091 Acked Flush 0 Done Sent Done false
951 1092 Acked Flush 0 Done Sent Done false
952 1093 Acked Read 1 Done Sent Done false
953 1094 Acked Write 9 Done Sent Done false
954 1095 Acked Read 6 Done Sent Done false
955 1096 Acked Write 10 Done Sent Done false
956 1097 Acked Read 9 Done Sent Done false
957 1098 Acked Write 5 Done Sent Done false
958 1099 Acked Read 5 Done Sent Done false
959 1100 Acked Write 8 Done Sent Done false
960 1101 Acked Read 3 Done Sent Done false
961 1102 Acked Read 10 Done Sent Done false
962 1103 Acked Write 5 Done Sent Done false
963 1104 Acked Flush 0 Done Sent Done false
964 1105 Acked Read 2 Done Sent Done false
965 1106 Acked Write 8 Done Sent Done false
966 1107 Acked Read 6 Done Sent Done false
967 1108 Acked Write 3 Done Sent Done false
968 1109 Acked Read 1 Done Sent Done false
969 1110 Acked Write 9 Done Sent Done false
970 1111 Acked Read 2 Done Sent Done false
971 1112 Acked Read 4 Done Sent Done false
972 1113 Acked Read 5 Done Sent Done false
973 1114 Acked Read 10 Done Sent Done false
974 1115 Acked Read 6 Done Sent Done false
975 1116 Acked Flush 0 Done Sent Done false
976 1117 Acked Write 5 Done Sent Done false
977 1118 Acked Write 2 Done Sent Done false
978 1119 Acked Read 7 Done Sent Done false
979 1120 Acked Read 2 Done Sent Done false
980 1121 Acked Read 4 Done Sent Done false
981 1122 Acked Write 4 Done Sent Done false
982 1123 Acked Read 10 Done Sent Done false
983 1124 Acked Read 5 Done Sent Done false
984 1125 Acked Flush 0 Done Sent Done false
985 1126 Acked Read 10 Done Sent Done false
986 1127 Acked Write 6 Done Sent Done false
987 1128 Acked Read 2 Done Sent Done false
988 1129 Acked Read 7 Done Sent Done false
989 1130 Acked Read 2 Done Sent Done false
990 1131 Acked Write 6 Done Sent Done false
991 1132 Acked Read 6 Done Sent Done false
992 1133 Acked Read 1 Done Sent Done false
993 1134 Acked Read 4 Done Sent Done false
994 1135 Acked Write 5 Done Sent Done false
995 1136 Acked Write 1 Done Sent Done false
996 1137 Acked Write 1 Done Sent Done false
997 1138 Acked Read 4 Done Sent Done false
998 1139 Acked Write 7 Done Sent Done false
999 1140 Acked Flush 0 Done Sent Done false
1000 1141 Acked Read 3 Done Sent Done false
1001 1142 Acked Write 1 Done Sent Done false
1002 1143 Acked Write 4 Done Sent Done false
1003 1144 Acked Write 3 Done Sent Done false
1004 1145 Acked Read 8 Done Sent Done false
1005 1146 Acked Read 8 Done Sent Done false
1006 1147 Acked Read 7 Done Sent Done false
1007 1148 Acked Read 8 Done Sent Done false
1008 1149 Acked Write 1 Done Sent Done false
1009 1150 Acked Write 8 Done Sent Done false
1010 1151 Acked Read 2 Done Sent Done false
1011 1152 Acked Read 10 Done Sent Done false
1012 1153 Acked Write 10 Done Sent Done false
1013 1154 Acked Write 7 Done Sent Done false
1014 1155 Acked Read 5 Done Sent Done false
1015 1156 Acked Write 1 Done Sent Done false
1016 1157 Acked Read 9 Done Sent Done false
1017 1158 Acked Write 10 Done Sent Done false
1018 1159 Acked Read 5 Done Sent Done false
1019 1160 Acked Write 7 Done Sent Done false
1020 1161 Acked Read 10 Done Sent Done false
1021 1162 Acked Flush 0 Done Sent Done false
1022 1163 Acked Read 8 Done Sent Done false
1023 1164 Acked Write 8 Done Sent Done false
1024 1165 Acked Write 4 Done Sent Done false
1025 1166 Acked Write 6 Done Sent Done false
1026 1167 Acked Read 2 Done Sent Done false
1027 1168 Acked Write 9 Done Sent Done false
1028 1169 Acked Write 2 Done Sent Done false
1029 1170 Acked Read 4 Done Sent Done false
1030 1171 Acked Read 5 Done Sent Done false
1031 1172 Acked Write 1 Done Sent Done false
1032 1173 Acked Write 7 Done Sent Done false
1033 1174 Acked Write 1 Done Sent Done false
1034 1175 Acked Read 5 Done Sent Done false
1035 1176 Acked Flush 0 Done Sent Done false
1036 1177 Acked Write 9 Done Sent Done false
1037 1178 Acked Read 2 Done Sent Done false
1038 1179 Acked Read 4 Done Sent Done false
1039 1180 Acked Write 4 Done Sent Done false
1040 1181 Acked Read 8 Done Sent Done false
1041 1182 Acked Read 3 Done Sent Done false
1042 1183 Acked Write 7 Done Sent Done false
1043 1184 Acked Read 7 Done Sent Done false
1044 1185 Acked Write 3 Done Sent Done false
1045 1186 Acked Read 8 Done Sent Done false
1046 1187 Acked Write 3 Done Sent Done false
1047 1188 Acked Read 3 Done Sent Done false
1048 1189 Acked Flush 0 Done Sent Done false
1049 1190 Acked Read 2 Done Sent Done false
1050 1191 Acked Read 2 Done Sent Done false
1051 1192 Acked Read 4 Done Sent Done false
1052 1193 Acked Read 2 Done Sent Done false
1053 1194 Acked Write 8 Done Sent Done false
1054 1195 Acked Read 6 Done Sent Done false
1055 1196 Acked Read 7 Done Sent Done false
1056 1197 Acked Write 8 Done Sent Done false
1057 1198 Acked Read 1 Done Sent Done false
1058 1199 Acked Read 3 Done Sent Done false
1059 1200 Acked Read 6 Done Sent Done false
1060 1201 Acked Read 7 Done Sent Done false
1061 1202 Acked Read 2 Done Sent Done false
1062 1203 Acked Write 9 Done Sent Done false
1063 1204 Acked Write 8 Done Sent Done false
1064 1205 Acked Read 2 Done Sent Done false
1065 1206 Acked Read 7 Done Sent Done false
1066 1207 Acked Read 4 Done Sent Done false
1067 1208 Acked Read 1 Done Sent Done false
1068 1209 Acked Write 10 Done Sent Done false
1069 1210 Acked Write 8 Done Sent Done false
1070 1211 Acked Read 6 Done Sent Done false
1071 1212 Acked Read 3 Done Sent Done false
1072 1213 Acked Read 1 Done Sent Done false
1073 1214 Acked Read 6 Done Sent Done false
1074 1215 Acked Write 5 Done Sent Done false
1075 1216 Acked Read 5 Done Sent Done false
1076 1217 Acked Write 4 Done Sent Done false
1077 1218 Acked Read 7 Done Sent Done false
1078 1219 Acked Read 2 Done Sent Done false
1079 1220 Acked Write 5 Done Sent Done false
1080 1221 Acked Read 3 Done Sent Done false
1081 1222 Acked Write 4 Done Sent Done false
1082 1223 Acked Flush 0 Done Sent Done false
1083 1224 Acked Read 2 Done Sent Done false
1084 1225 Acked Write 8 Done Sent Done false
1085 1226 Acked Write 1 Done Sent Done false
1086 1227 Acked Write 9 Done Sent Done false
1087 1228 Acked Write 3 Done Sent Done false
1088 1229 Acked Flush 0 Done Sent Done false
1089 1230 Acked Write 3 Done Sent Done false
1090 1231 Acked Read 10 Done Sent Done false
1091 1232 Acked Write 7 Done Sent Done false
1092 1233 Acked Read 2 Done Sent Done false
1093 1234 Acked Read 8 Done Sent Done false
1094 1235 Acked Write 2 Done Sent Done false
1095 1236 Acked Read 7 Done Sent Done false
1096 1237 Acked Write 5 Done Sent Done false
1097 1238 Acked Write 6 Done Sent Done false
1098 1239 Acked Read 2 Done Sent Done false
1099 1240 Acked Write 2 Done Sent Done false
1100 1241 Acked Flush 0 Done Sent Done false
1101 1242 Acked Read 3 Done Sent Done false
1102 1243 Acked Read 5 Done Sent Done false
1103 1244 Acked Write 6 Done Sent Done false
1104 1245 Acked Read 2 Done Sent Done false
1105 1246 Acked Read 1 Done Sent Done false
1106 1247 Acked Read 3 Done Sent Done false
1107 1248 Acked Write 7 Done Sent Done false
1108 1249 Acked Read 6 Done Sent Done false
1109 1250 Acked Flush 0 Done Sent Done false
1110 1251 Acked Write 8 Done Sent Done false
1111 1252 Acked Write 6 Done Sent Done false
1112 1253 Acked Read 4 Done Sent Done false
1113 1254 Acked Flush 0 Done Sent Done false
1114 1255 Acked Write 4 Done Sent Done false
1115 1256 Acked Write 4 Done Sent Done false
1116 1257 Acked Write 6 Done Sent Done false
1117 1258 Acked Read 10 Done Sent Done false
1118 1259 Acked Write 5 Done Sent Done false
1119 1260 Acked Write 1 Done Sent Done false
1120 1261 Acked Read 2 Done Sent Done false
1121 1262 Acked Write 5 Done Sent Done false
1122 1263 Acked Read 9 Done Sent Done false
1123 1264 Acked Write 2 Done Sent Done false
1124 1265 Acked Write 5 Done Sent Done false
1125 1266 Acked Flush 0 Done Sent Done false
1126 1267 Acked Write 5 Done Sent Done false
1127 1268 Acked Write 7 Done Sent Done false
1128 1269 Acked Write 9 Done Sent Done false
1129 1270 Acked Flush 0 Done Sent Done false
1130 1271 Acked Read 7 Done Sent Done false
1131 1272 Acked Read 2 Done Sent Done false
1132 1273 Acked Write 7 Done Sent Done false
1133 1274 Acked Flush 0 Done Sent Done false
1134 1275 Acked Read 7 Done Sent Done false
1135 1276 Acked Write 5 Done Sent Done false
1136 1277 Acked Read 9 Done Sent Done false
1137 1278 Acked Write 10 Done Sent Done false
1138 1279 Acked Read 10 Done Sent Done false
1139 1280 Acked Flush 0 Done Sent Done false
1140 1281 Acked Write 8 Done Sent Done false
1141 1282 Acked Read 3 Done Sent Done false
1142 1283 Acked Flush 0 Done Sent Done false
1143 1284 Acked Flush 0 Done Sent Done false
1144 1285 Acked Read 6 Done Sent Done false
1145 1286 Acked Read 1 Done Sent Done false
1146 1287 Acked Read 7 Done Sent Done false
1147 1288 Acked Write 2 Done Sent Done false
1148 1289 Acked Flush 0 Done Sent Done false
1149 1290 Acked Write 7 Done Sent Done false
1150 1291 Acked Read 7 Done Sent Done false
1151 1292 Acked Write 9 Done Sent Done false
1152 1293 Acked Read 6 Done Sent Done false
1153 1294 Acked Write 8 Done Sent Done false
1154 1295 Acked Write 4 Done Sent Done false
1155 1296 Acked Read 4 Done Sent Done false
1156 1297 Acked Write 9 Done Sent Done false
1157 1298 Acked Read 9 Done Sent Done false
1158 1299 Acked Flush 0 Done Sent Done false
1159 1300 Acked Write 4 Done Sent Done false
1160 STATES DS:0 DS:1 DS:2 TOTAL
1161 Sent 0 301 0 301
1162 Done 301 0 301 602
1163 Skipped 0 0 0 0
1164 Error 0 0 0 0
1165 Last Flush: Some(JobId(1299)) None Some(JobId(1299))
1166 Downstairs last five completed:
1167 Upstairs last five completed: 1300 1299 1298 1297 1296
1168 CLIENT: Up:0 ds:301 act:2
11692025-01-28T21:54:30.617ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11702025-01-28T21:54:31.347ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11712025-01-28T21:54:31.875ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11722025-01-28T21:54:31.875ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42724) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11732025-01-28T21:54:31.876ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11742025-01-28T21:54:31.876ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11752025-01-28T21:54:31.876ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11762025-01-28T21:54:31.876ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11772025-01-28T21:54:31.876ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11782025-01-28T21:54:31.876ZINFOcrucible: [1] client re-new 301 jobs since flush None = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11792025-01-28T21:54:31.877ZINFOcrucible: [1] Replayed 301 jobs since flush: None = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1180 ----------------------------------------------------------------
1181 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1182 Downstairs last five completed: 1301 1300 1299 1298 1297
1183 Upstairs last five completed: 1301 1300 1299 1298 1297
1184 CLIENT: Up:0 ds:0 act:3
1185 Replay: All jobs finished
1186 Main task has work to do, go find it
1187 got dsc Stop(2)
1188 stop 2
1189 Jan 28 21:54:34.117 INFO request completed, latency_us: 365, response_code: 204, uri: /stop/cid/2, method: POST, req_id: fc3a0ff9-b47a-4504-bae5-c23cae9e87a9, remote_addr: 127.0.0.1:36665[8830] Got stop action so:false kr:false
1190 , local_addr: 127.0.0.1:9998
11912025-01-28T21:54:34.118ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11922025-01-28T21:54:34.118ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1193 Jan 28 21:54:34.118 INFO request completed, latency_us: 123, response_code: 200, uri: /state/cid/2{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:54:34.11870833Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
1194 , method: GET, req_id: cc1c4d37-930d-4395-a678-9356134bef50, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
11952025-01-28T21:54:34.118ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1196 [8830] Exited with: ExitStatus(unix_wait_status(9))
1197 [8830][2] reports Exit
1198 Jan 28 21:54:38.121 INFO request completed, latency_us: 234, response_code: 200, uri: /state/cid/2, method: GET, req_id: 99f083a0-6c31-4d6e-aebc-275974b9c738, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
1199 001/300 Read block 3746 len 16384
1200 002/300 Read block 2947 len 36864
1201 003/300 Flush
1202 004/300 Read block 1094 len 8192
1203 005/300 Read block 114 len 36864
1204 006/300 Read block 3978 len 40960
1205 007/300 Write block 2733 len 4096 data: 2
1206 008/300 Write block 3181 len 40960 data: 2 2 2 2 2 2 3 3 3 3
1207 009/300 Write block 1031 len 40960 data: 2 2 2 2 2 3 3 3 3 3
1208 010/300 Write block 4676 len 28672 data: 2 2 2 2 2 2 2
1209 011/300 Write block 3708 len 20480 data: 2 2 2 2 2
1210 012/300 Read block 4190 len 20480
1211 013/300 Write block 588 len 4096 data: 2
1212 014/300 Flush
1213 015/300 Read block 1690 len 24576
1214 016/300 Write block 1749 len 24576 data: 2 2 2 2 2 2
1215 017/300 Read block 2282 len 36864
1216 018/300 Write block 3816 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1217 019/300 Read block 773 len 16384
1218 020/300 Flush
1219 021/300 Write block 812 len 20480 data: 2 2 2 2 2
1220 022/300 Read block 2267 len 8192
1221 023/300 Write block 1284 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1222 024/300 Write block 4884 len 16384 data: 3 2 2 3
1223 025/300 Read block 4785 len 16384
1224 026/300 Read block 4069 len 24576
1225 027/300 Write block 737 len 36864 data: 3 3 3 3 3 2 2 2 2
1226 028/300 Read block 3665 len 32768
1227 029/300 Write block 4658 len 36864 data: 2 2 2 2 2 2 2 2 2
1228 030/300 Flush
1229 031/300 Write block 4525 len 12288 data: 2 2 2
1230 032/300 Read block 746 len 8192
1231 033/300 Read block 4835 len 40960
1232 034/300 Write block 2314 len 32768 data: 2 2 2 2 2 2 2 2
1233 035/300 Read block 1593 len 32768
1234 036/300 Flush
1235 037/300 Write block 1202 len 12288 data: 2 2 3
1236 038/300 Read block 1911 len 40960
1237 039/300 Read block 4779 len 24576
1238 040/300 Write block 2616 len 24576 data: 2 2 2 2 2 2
1239 041/300 Read block 1725 len 36864
1240 042/300 Read block 700 len 40960
1241 043/300 Write block 1669 len 12288 data: 2 2 2
1242 044/300 Read block 3282 len 24576
1243 045/300 Read block 302 len 8192
1244 046/300 Read block 831 len 36864
1245 047/300 Write block 4615 len 12288 data: 2 2 2
1246 048/300 Read block 3447 len 40960
1247 049/300 Read block 2703 len 28672
1248 050/300 Read block 1854 len 12288
1249 051/300 Read block 721 len 36864
1250 052/300 Read block 879 len 8192
1251 053/300 Read block 2568 len 28672
1252 054/300 Flush
1253 055/300 Write block 4961 len 4096 data: 2
1254 056/300 Write block 1077 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1255 057/300 Write block 4083 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1256 058/300 Read block 239 len 32768
1257 059/300 Read block 733 len 16384
1258 060/300 Read block 1745 len 28672
1259 061/300 Write block 1150 len 40960 data: 3 3 3 3 3 3 3 3 3 2
1260 062/300 Flush
1261 063/300 Read block 2493 len 40960
1262 064/300 Write block 2337 len 16384 data: 2 2 2 2
1263 065/300 Write block 2894 len 12288 data: 2 2 2
1264 066/300 Write block 136 len 16384 data: 2 2 2 2
1265 067/300 Read block 1163 len 8192
1266 068/300 Read block 4259 len 28672
1267 069/300 Read block 3661 len 8192
1268 070/300 Write block 2475 len 28672 data: 2 2 2 3 3 2 2
1269 071/300 Write block 2392 len 36864 data: 2 2 2 2 2 2 2 2 2
1270 072/300 Read block 1080 len 4096
1271 073/300 Read block 3204 len 40960
1272 074/300 Write block 2808 len 8192 data: 2 2
1273 075/300 Write block 1514 len 40960 data: 2 2 2 3 3 3 3 3 3 3
1274 076/300 Flush
1275 077/300 Read block 2326 len 28672
1276 078/300 Flush
1277 079/300 Write block 864 len 40960 data: 2 2 2 2 3 3 3 3 2 2
1278 080/300 Write block 4863 len 8192 data: 2 2
1279 081/300 Read block 4229 len 20480
1280 082/300 Read block 4515 len 8192
1281 083/300 Flush
1282 084/300 Write block 3000 len 36864 data: 2 2 2 2 2 2 2 2 2
1283 085/300 Read block 4546 len 12288
1284 086/300 Write block 4587 len 8192 data: 2 2
1285 087/300 Write block 1982 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1286 088/300 Read block 2653 len 40960
1287 089/300 Read block 1855 len 36864
1288 090/300 Flush
1289 091/300 Read block 3663 len 12288
1290 092/300 Write block 4001 len 24576 data: 3 3 3 3 3 2
1291 093/300 Read block 2040 len 32768
1292 094/300 Flush
1293 095/300 Write block 2290 len 16384 data: 2 2 2 2
1294 096/300 Write block 1116 len 28672 data: 2 2 2 2 2 2 2
1295 097/300 Read block 482 len 12288
1296 098/300 Write block 3826 len 24576 data: 2 2 2 2 2 2
1297 099/300 Write block 4694 len 20480 data: 2 2 2 2 2
1298 100/300 Read block 1491 len 32768
1299 101/300 Read block 955 len 20480
1300 102/300 Write block 1179 len 8192 data: 2 2
1301 103/300 Write block 3102 len 28672 data: 2 2 2 2 2 2 2
1302 104/300 Flush
1303 105/300 Read block 536 len 8192
1304 106/300 Write block 4377 len 4096 data: 2
1305 107/300 Read block 4626 len 8192
1306 108/300 Write block 4462 len 24576 data: 2 2 2 2 2 2
1307 109/300 Write block 540 len 40960 data: 4 4 3 3 3 2 2 2 2 2
1308 110/300 Read block 3469 len 12288
1309 111/300 Read block 2979 len 28672
1310 112/300 Flush
1311 113/300 Read block 679 len 24576
1312 114/300 Read block 3938 len 40960
1313 115/300 Flush
1314 116/300 Read block 3889 len 32768
1315 117/300 Read block 3275 len 12288
1316 118/300 Write block 360 len 28672 data: 2 2 2 2 2 2 2
1317 119/300 Write block 1347 len 4096 data: 2
1318 120/300 Write block 4876 len 40960 data: 2 3 3 3 3 3 3 3 4 3
1319 121/300 Write block 926 len 24576 data: 2 2 2 2 2 2
1320 122/300 Write block 2697 len 8192 data: 2 2
1321 123/300 Read block 1024 len 4096
1322 124/300 Read block 641 len 16384
1323 125/300 Read block 3565 len 8192
1324 126/300 Write block 3079 len 20480 data: 2 2 2 2 2
1325 127/300 Read block 2233 len 28672
1326 128/300 Flush
1327 129/300 Read block 3374 len 8192
1328 130/300 Write block 3042 len 28672 data: 2 2 2 2 2 2 2
1329 131/300 Write block 2586 len 12288 data: 2 2 2
1330 132/300 Read block 3444 len 16384
1331 133/300 Write block 3352 len 16384 data: 2 2 2 3
1332 134/300 Read block 2901 len 40960
1333 135/300 Write block 1179 len 28672 data: 3 3 2 2 2 2 2
1334 136/300 Write block 1779 len 28672 data: 2 2 2 2 2 2 2
1335 137/300 Write block 2615 len 28672 data: 2 3 3 3 3 3 3
1336 138/300 Read block 1218 len 36864
1337 139/300 Write block 4842 len 4096 data: 3
1338 140/300 Flush
1339 141/300 Read block 194 len 4096
1340 142/300 Write block 4184 len 12288 data: 3 3 3
1341 143/300 Read block 3945 len 24576
1342 144/300 Read block 3699 len 32768
1343 145/300 Read block 966 len 8192
1344 146/300 Read block 2586 len 40960
1345 147/300 Read block 1884 len 20480
1346 148/300 Read block 914 len 40960
1347 149/300 Write block 3597 len 36864 data: 2 2 2 2 2 2 2 2 2
1348 150/300 Flush
1349 151/300 Write block 4850 len 20480 data: 2 2 2 2 2
1350 152/300 Write block 300 len 8192 data: 3 3
1351 153/300 Write block 1875 len 32768 data: 2 2 2 2 2 2 2 2
1352 154/300 Read block 4248 len 20480
1353 155/300 Write block 1935 len 8192 data: 2 2
1354 156/300 Read block 452 len 20480
1355 157/300 Write block 2555 len 32768 data: 2 2 2 2 2 2 2 2
1356 158/300 Read block 4917 len 16384
1357 159/300 Read block 3854 len 24576
1358 160/300 Read block 3207 len 32768
1359 161/300 Write block 4351 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1360 162/300 Write block 2283 len 12288 data: 2 2 2
1361 163/300 Read block 843 len 20480
1362 164/300 Write block 4627 len 12288 data: 2 2 2
1363 165/300 Read block 3948 len 16384
1364 166/300 Read block 3222 len 40960
1365 167/300 Read block 1367 len 20480
1366 168/300 Read block 882 len 36864
1367 169/300 Read block 2953 len 36864
1368 170/300 Write block 4457 len 40960 data: 2 2 2 2 2 3 3 3 3 3
1369 171/300 Read block 2997 len 16384
1370 172/300 Write block 4414 len 4096 data: 2
1371 173/300 Read block 3616 len 8192
1372 174/300 Write block 892 len 8192 data: 2 2
1373 175/300 Write block 4949 len 36864 data: 2 2 2 2 2 2 2 2 2
1374 176/300 Flush
1375 177/300 Write block 2914 len 8192 data: 2 2
1376 178/300 Write block 3714 len 4096 data: 3
1377 179/300 Read block 2596 len 40960
1378 180/300 Read block 3466 len 20480
1379 181/300 Flush
1380 182/300 Write block 2603 len 12288 data: 2 2 2
1381 183/300 Read block 4153 len 8192
1382 184/300 Write block 2290 len 32768 data: 3 3 3 3 2 2 2 2
1383 185/300 Write block 3870 len 24576 data: 2 2 2 2 2 2
1384 186/300 Flush
1385 187/300 Write block 2757 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1386 188/300 Write block 2885 len 8192 data: 3 3
1387 189/300 Read block 1962 len 4096
1388 190/300 Write block 4507 len 20480 data: 2 2 2 2 2
1389 191/300 Read block 2605 len 12288
1390 192/300 Write block 1467 len 12288 data: 2 2 2
1391 193/300 Read block 3249 len 4096
1392 194/300 Read block 1437 len 12288
1393 195/300 Read block 4782 len 36864
1394 196/300 Read block 4268 len 4096
1395 197/300 Write block 1442 len 24576 data: 2 2 2 2 2 2
1396 198/300 Read block 919 len 12288
1397 199/300 Read block 795 len 8192
1398 200/300 Write block 2286 len 16384 data: 2 2 2 2
1399 201/300 Write block 2971 len 36864 data: 2 2 2 2 2 2 2 2 2
1400 202/300 Read block 4744 len 28672
1401 203/300 Write block 2157 len 28672 data: 3 3 4 4 3 3 3
1402 204/300 Write block 2739 len 20480 data: 2 2 2 2 2
1403 205/300 Read block 3861 len 4096
1404 206/300 Write block 2675 len 32768 data: 3 3 3 3 3 3 3 2
1405 207/300 Read block 4683 len 20480
1406 208/300 Write block 3954 len 40960 data: 2 2 2 3 3 3 3 3 3 3
1407 209/300 Write block 4950 len 32768 data: 3 3 3 3 3 3 3 3
1408 210/300 Read block 2699 len 20480
1409 211/300 Read block 946 len 4096
1410 212/300 Read block 3969 len 12288
1411 213/300 Read block 1441 len 16384
1412 214/300 Write block 456 len 28672 data: 2 2 2 2 2 2 2
1413 215/300 Read block 1867 len 4096
1414 216/300 Read block 3041 len 4096
1415 217/300 Read block 3851 len 28672
1416 218/300 Write block 3878 len 32768 data: 2 2 3 3 3 3 3 2
1417 219/300 Write block 2441 len 16384 data: 2 2 2 2
1418 220/300 Write block 3086 len 12288 data: 3 2 2
1419 221/300 Read block 996 len 12288
1420 222/300 Flush
1421 223/300 Write block 1921 len 12288 data: 3 2 2
1422 224/300 Write block 936 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1423 225/300 Write block 1046 len 40960 data: 2 2 2 2 2 3 4 4 4 4
1424 226/300 Flush
1425 227/300 Read block 3315 len 24576
1426 228/300 Read block 4678 len 12288
1427 229/300 Read block 2748 len 16384
1428 230/300 Write block 1125 len 24576 data: 2 2 2 2 2 2
1429 231/300 Read block 2679 len 16384
1430 232/300 Write block 2789 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1431 233/300 Write block 825 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1432 234/300 Read block 992 len 28672
1433 235/300 Write block 3777 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1434 236/300 Read block 3319 len 12288
1435 237/300 Write block 1053 len 8192 data: 5 5
1436 238/300 Read block 1962 len 20480
1437 239/300 Read block 3449 len 32768
1438 240/300 Read block 3868 len 12288
1439 241/300 Read block 2870 len 40960
1440 242/300 Read block 2453 len 20480
1441 243/300 Write block 222 len 20480 data: 2 2 2 2 2
1442 244/300 Read block 4965 len 16384
1443 245/300 Write block 1308 len 36864 data: 2 2 2 2 2 2 2 2 2
1444 246/300 Flush
1445 247/300 Write block 4639 len 8192 data: 2 2
1446 248/300 Write block 4764 len 36864 data: 2 2 2 2 2 2 2 2 2
1447 249/300 Write block 367 len 8192 data: 2 2
1448 250/300 Write block 233 len 16384 data: 2 2 2 2
1449 251/300 Read block 2150 len 32768
1450 252/300 Read block 4922 len 24576
1451 253/300 Write block 2633 len 20480 data: 2 2 2 2 2
1452 254/300 Read block 2985 len 36864
1453 255/300 Read block 1119 len 4096
1454 256/300 Write block 1798 len 8192 data: 2 2
1455 257/300 Flush
1456 258/300 Read block 2283 len 40960
1457 259/300 Write block 2471 len 32768 data: 2 2 2 2 3 3 3 4
1458 260/300 Flush
1459 261/300 Write block 3249 len 4096 data: 2
1460 262/300 Read block 341 len 24576
1461 263/300 Read block 3460 len 24576
1462 264/300 Read block 4961 len 32768
1463 265/300 Flush
1464 266/300 Write block 2658 len 16384 data: 2 2 2 2
1465 267/300 Write block 4402 len 32768 data: 2 2 2 2 2 2 2 2
1466 268/300 Read block 254 len 40960
1467 269/300 Read block 572 len 32768
1468 270/300 Flush
1469 271/300 Write block 2132 len 16384 data: 2 2 2 3
1470 272/300 Flush
1471 273/300 Write block 1604 len 16384 data: 2 2 2 2
1472 274/300 Read block 996 len 40960
1473 275/300 Read block 2973 len 8192
1474 276/300 Read block 725 len 12288
1475 277/300 Read block 2979 len 36864
1476 278/300 Read block 4878 len 8192
1477 279/300 Write block 2826 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1478 280/300 Read block 4621 len 4096
1479 281/300 Write block 4016 len 28672 data: 2 2 2 2 2 2 2
1480 282/300 Read block 1871 len 36864
1481 283/300 Write block 1127 len 16384 data: 3 3 3 3
1482 284/300 Read block 1255 len 32768
1483 285/300 Read block 4283 len 28672
1484 286/300 Read block 136 len 4096
1485 287/300 Read block 4903 len 40960
1486 288/300 Write block 3282 len 32768 data: 2 2 2 2 2 2 2 2
1487 289/300 Read block 2643 len 36864
1488 290/300 Read block 1856 len 40960
1489 291/300 Write block 2574 len 24576 data: 2 2 2 2 2 2
1490 292/300 Write block 4565 len 28672 data: 2 2 2 2 2 2 2
1491 293/300 Read block 2994 len 40960
1492 294/300 Flush
1493 295/300 Write block 25 len 8192 data: 2 2
1494 296/300 Write block 905 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1495 297/300 Read block 1469 len 16384
1496 298/300 Write block 2454 len 32768 data: 2 2 2 2 2 2 2 2
1497 299/300 Write block 2680 len 4096 data: 4
1498 300/300 Read block 1167 len 16384
1499 301/300 Read block 2368 len 12288
1500 Main task has work to do, go find it
1501 got dsc Start(2)
1502 start 2
1503 Jan 28 21:54:38.362 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: 7146b2a3-c82e-49d2-888f-015e438962c0, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
1504 Replay: started 2, returned:Ok(())
1505 ----------------------------------------------------------------
1506 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1507 States: Active Active Offline
1508 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1509 1302 Acked Read 4 Done Done Sent false
1510 1303 Acked Read 9 Done Done Sent false
1511 1304 Acked Flush 0 Done Done Sent false
1512 1305 Acked Read 2 Done Done Sent false
1513 1306 Acked Read 9 Done Done Sent false
1514 1307 Acked Read 10 Done Done Sent false
1515 1308 Acked Write 1 Done Done Sent false
1516 1309 Acked Write 10 Done Done Sent false
1517 1310 Acked Write 10 Done Done Sent false
1518 1311 Acked Write 7 Done Done Sent false
1519 1312 Acked Write 5 Done Done Sent false
1520 1313 Acked Read 5 Done Done Sent false
1521 1314 Acked Write 1 Done Done Sent false
1522 1315 Acked Flush 0 Done Done Sent false
1523 1316 Acked Read 6 Done Done Sent false
1524 1317 Acked Write 6 Done Done Sent false
1525 1318 Acked Read 9 Done Done Sent false
1526 1319 Acked Write 10 Done Done Sent false
1527 1320 Acked Read 4 Done Done Sent false
1528 1321 Acked Flush 0 Done Done Sent false
1529 1322 Acked Write 5 Done Done Sent false
1530 1323 Acked Read 2 Done Done Sent false
1531 1324 Acked Write 10 Done Done Sent false
1532 1325 Acked Write 4 Done Done Sent false
1533 1326 Acked Read 4 Done Done Sent false
1534 1327 Acked Read 6 Done Done Sent false
1535 1328 Acked Write 9 Done Done Sent false
1536 1329 Acked Read 8 Done Done Sent false
1537 1330 Acked Write 9 Done Done Sent false
1538 1331 Acked Flush 0 Done Done Sent false
1539 1332 Acked Write 3 Done Done Sent false
1540 1333 Acked Read 2 Done Done Sent false
1541 1334 Acked Read 10 Done Done Sent false
1542 1335 Acked Write 8 Done Done Sent false
1543 1336 Acked Read 8 Done Done Sent false
1544 1337 Acked Flush 0 Done Done Sent false
1545 1338 Acked Write 3 Done Done Sent false
1546 1339 Acked Read 10 Done Done Sent false
1547 1340 Acked Read 6 Done Done Sent false
1548 1341 Acked Write 6 Done Done Sent false
1549 1342 Acked Read 9 Done Done Sent false
1550 1343 Acked Read 10 Done Done Sent false
1551 1344 Acked Write 3 Done Done Sent false
1552 1345 Acked Read 6 Done Done Sent false
1553 1346 Acked Read 2 Done Done Sent false
1554 1347 Acked Read 9 Done Done Sent false
1555 1348 Acked Write 3 Done Done Sent false
1556 1349 Acked Read 10 Done Done Sent false
1557 1350 Acked Read 7 Done Done Sent false
1558 1351 Acked Read 3 Done Done Sent false
1559 1352 Acked Read 9 Done Done Sent false
1560 1353 Acked Read 2 Done Done Sent false
1561 1354 Acked Read 7 Done Done Sent false
1562 1355 Acked Flush 0 Done Done Sent false
1563 1356 Acked Write 1 Done Done Sent false
1564 1357 Acked Write 10 Done Done Sent false
1565 1358 Acked Write 10 Done Done Sent false
1566 1359 Acked Read 8 Done Done Sent false
1567 1360 Acked Read 4 Done Done Sent false
1568 1361 Acked Read 7 Done Done Sent false
1569 1362 Acked Write 10 Done Done Sent false
1570 1363 Acked Flush 0 Done Done Sent false
1571 1364 Acked Read 10 Done Done Sent false
1572 1365 Acked Write 4 Done Done Sent false
1573 1366 Acked Write 3 Done Done Sent false
1574 1367 Acked Write 4 Done Done Sent false
1575 1368 Acked Read 2 Done Done Sent false
1576 1369 Acked Read 7 Done Done Sent false
1577 1370 Acked Read 2 Done Done Sent false
1578 1371 Acked Write 7 Done Done Sent false
1579 1372 Acked Write 9 Done Done Sent false
1580 1373 Acked Read 1 Done Done Sent false
1581 1374 Acked Read 10 Done Done Sent false
1582 1375 Acked Write 2 Done Done Sent false
1583 1376 Acked Write 10 Done Done Sent false
1584 1377 Acked Flush 0 Done Done Sent false
1585 1378 Acked Read 7 Done Done Sent false
1586 1379 Acked Flush 0 Done Done Sent false
1587 1380 Acked Write 10 Done Done Sent false
1588 1381 Acked Write 2 Done Done Sent false
1589 1382 Acked Read 5 Done Done Sent false
1590 1383 Acked Read 2 Done Done Sent false
1591 1384 Acked Flush 0 Done Done Sent false
1592 1385 Acked Write 9 Done Done Sent false
1593 1386 Acked Read 3 Done Done Sent false
1594 1387 Acked Write 2 Done Done Sent false
1595 1388 Acked Write 10 Done Done Sent false
1596 1389 Acked Read 10 Done Done Sent false
1597 1390 Acked Read 9 Done Done Sent false
1598 1391 Acked Flush 0 Done Done Sent false
1599 1392 Acked Read 3 Done Done Sent false
1600 1393 Acked Write 6 Done Done Sent false
1601 1394 Acked Read 8 Done Done Sent false
1602 1395 Acked Flush 0 Done Done Sent false
1603 1396 Acked Write 4 Done Done Sent false
1604 1397 Acked Write 7 Done Done Sent false
1605 1398 Acked Read 3 Done Done Sent false
1606 1399 Acked Write 6 Done Done Sent false
1607 1400 Acked Write 5 Done Done Sent false
1608 1401 Acked Read 8 Done Done Sent false
1609 1402 Acked Read 5 Done Done Sent false
1610 1403 Acked Write 2 Done Done Sent false
1611 1404 Acked Write 7 Done Done Sent false
1612 1405 Acked Flush 0 Done Done Sent false
1613 1406 Acked Read 2 Done Done Sent false
1614 1407 Acked Write 1 Done Done Sent false
1615 1408 Acked Read 2 Done Done Sent false
1616 1409 Acked Write 6 Done Done Sent false
1617 1410 Acked Write 10 Done Done Sent false
1618 1411 Acked Read 3 Done Done Sent false
1619 1412 Acked Read 7 Done Done Sent false
1620 1413 Acked Flush 0 Done Done Sent false
1621 1414 Acked Read 6 Done Done Sent false
1622 1415 Acked Read 10 Done Done Sent false
1623 1416 Acked Flush 0 Done Done Sent false
1624 1417 Acked Read 8 Done Done Sent false
1625 1418 Acked Read 3 Done Done Sent false
1626 1419 Acked Write 7 Done Done Sent false
1627 1420 Acked Write 1 Done Done Sent false
1628 1421 Acked Write 10 Done Done Sent false
1629 1422 Acked Write 6 Done Done Sent false
1630 1423 Acked Write 2 Done Done Sent false
1631 1424 Acked Read 1 Done Done Sent false
1632 1425 Acked Read 4 Done Done Sent false
1633 1426 Acked Read 2 Done Done Sent false
1634 1427 Acked Write 5 Done Done Sent false
1635 1428 Acked Read 7 Done Done Sent false
1636 1429 Acked Flush 0 Done Done Sent false
1637 1430 Acked Read 2 Done Done Sent false
1638 1431 Acked Write 7 Done Done Sent false
1639 1432 Acked Write 3 Done Done Sent false
1640 1433 Acked Read 4 Done Done Sent false
1641 1434 Acked Write 4 Done Done Sent false
1642 1435 Acked Read 10 Done Done Sent false
1643 1436 Acked Write 7 Done Done Sent false
1644 1437 Acked Write 7 Done Done Sent false
1645 1438 Acked Write 7 Done Done Sent false
1646 1439 Acked Read 9 Done Done Sent false
1647 1440 Acked Write 1 Done Done Sent false
1648 1441 Acked Flush 0 Done Done Sent false
1649 1442 Acked Read 1 Done Done Sent false
1650 1443 Acked Write 3 Done Done Sent false
1651 1444 Acked Read 6 Done Done Sent false
1652 1445 Acked Read 8 Done Done Sent false
1653 1446 Acked Read 2 Done Done Sent false
1654 1447 Acked Read 10 Done Done Sent false
1655 1448 Acked Read 5 Done Done Sent false
1656 1449 Acked Read 10 Done Done Sent false
1657 1450 Acked Write 9 Done Done Sent false
1658 1451 Acked Flush 0 Done Done Sent false
1659 1452 Acked Write 5 Done Done Sent false
1660 1453 Acked Write 2 Done Done Sent false
1661 1454 Acked Write 8 Done Done Sent false
1662 1455 Acked Read 5 Done Done Sent false
1663 1456 Acked Write 2 Done Done Sent false
1664 1457 Acked Read 5 Done Done Sent false
1665 1458 Acked Write 8 Done Done Sent false
1666 1459 Acked Read 4 Done Done Sent false
1667 1460 Acked Read 6 Done Done Sent false
1668 1461 Acked Read 8 Done Done Sent false
1669 1462 Acked Write 10 Done Done Sent false
1670 1463 Acked Write 3 Done Done Sent false
1671 1464 Acked Read 5 Done Done Sent false
1672 1465 Acked Write 3 Done Done Sent false
1673 1466 Acked Read 4 Done Done Sent false
1674 1467 Acked Read 10 Done Done Sent false
1675 1468 Acked Read 5 Done Done Sent false
1676 1469 Acked Read 9 Done Done Sent false
1677 1470 Acked Read 9 Done Done Sent false
1678 1471 Acked Write 10 Done Done Sent false
1679 1472 Acked Read 4 Done Done Sent false
1680 1473 Acked Write 1 Done Done Sent false
1681 1474 Acked Read 2 Done Done Sent false
1682 1475 Acked Write 2 Done Done Sent false
1683 1476 Acked Write 9 Done Done Sent false
1684 1477 Acked Flush 0 Done Done Sent false
1685 1478 Acked Write 2 Done Done Sent false
1686 1479 Acked Write 1 Done Done Sent false
1687 1480 Acked Read 10 Done Done Sent false
1688 1481 Acked Read 5 Done Done Sent false
1689 1482 Acked Flush 0 Done Done Sent false
1690 1483 Acked Write 3 Done Done Sent false
1691 1484 Acked Read 2 Done Done Sent false
1692 1485 Acked Write 8 Done Done Sent false
1693 1486 Acked Write 6 Done Done Sent false
1694 1487 Acked Flush 0 Done Done Sent false
1695 1488 Acked Write 10 Done Done Sent false
1696 1489 Acked Write 2 Done Done Sent false
1697 1490 Acked Read 1 Done Done Sent false
1698 1491 Acked Write 5 Done Done Sent false
1699 1492 Acked Read 3 Done Done Sent false
1700 1493 Acked Write 3 Done Done Sent false
1701 1494 Acked Read 1 Done Done Sent false
1702 1495 Acked Read 3 Done Done Sent false
1703 1496 Acked Read 9 Done Done Sent false
1704 1497 Acked Read 1 Done Done Sent false
1705 1498 Acked Write 6 Done Done Sent false
1706 1499 Acked Read 3 Done Done Sent false
1707 1500 Acked Read 2 Done Done Sent false
1708 1501 Acked Write 4 Done Done Sent false
1709 1502 Acked Write 9 Done Done Sent false
1710 1503 Acked Read 7 Done Done Sent false
1711 1504 Acked Write 7 Done Done Sent false
1712 1505 Acked Write 5 Done Done Sent false
1713 1506 Acked Read 1 Done Done Sent false
1714 1507 Acked Write 8 Done Done Sent false
1715 1508 Acked Read 5 Done Done Sent false
1716 1509 Acked Write 10 Done Done Sent false
1717 1510 Acked Write 8 Done Done Sent false
1718 1511 Acked Read 5 Done Done Sent false
1719 1512 Acked Read 1 Done Done Sent false
1720 1513 Acked Read 3 Done Done Sent false
1721 1514 Acked Read 4 Done Done Sent false
1722 1515 Acked Write 7 Done Done Sent false
1723 1516 Acked Read 1 Done Done Sent false
1724 1517 Acked Read 1 Done Done Sent false
1725 1518 Acked Read 7 Done Done Sent false
1726 1519 Acked Write 8 Done Done Sent false
1727 1520 Acked Write 4 Done Done Sent false
1728 1521 Acked Write 3 Done Done Sent false
1729 1522 Acked Read 3 Done Done Sent false
1730 1523 Acked Flush 0 Done Done Sent false
1731 1524 Acked Write 3 Done Done Sent false
1732 1525 Acked Write 10 Done Done Sent false
1733 1526 Acked Write 10 Done Done Sent false
1734 1527 Acked Flush 0 Done Done Sent false
1735 1528 Acked Read 6 Done Done Sent false
1736 1529 Acked Read 3 Done Done Sent false
1737 1530 Acked Read 4 Done Done Sent false
1738 1531 Acked Write 6 Done Done Sent false
1739 1532 Acked Read 4 Done Done Sent false
1740 1533 Acked Write 10 Done Done Sent false
1741 1534 Acked Write 10 Done Done Sent false
1742 1535 Acked Read 7 Done Done Sent false
1743 1536 Acked Write 10 Done Done Sent false
1744 1537 Acked Read 3 Done Done Sent false
1745 1538 Acked Write 2 Done Done Sent false
1746 1539 Acked Read 5 Done Done Sent false
1747 1540 Acked Read 8 Done Done Sent false
1748 1541 Acked Read 3 Done Done Sent false
1749 1542 Acked Read 10 Done Done Sent false
1750 1543 Acked Read 5 Done Done Sent false
1751 1544 Acked Write 5 Done Done Sent false
1752 1545 Acked Read 4 Done Done Sent false
1753 1546 Acked Write 9 Done Done Sent false
1754 1547 Acked Flush 0 Done Done Sent false
1755 1548 Acked Write 2 Done Done Sent false
1756 1549 Acked Write 9 Done Done Sent false
1757 1550 Acked Write 2 Done Done Sent false
1758 1551 Acked Write 4 Done Done Sent false
1759 1552 Acked Read 8 Done Done Sent false
1760 1553 Acked Read 6 Done Done Sent false
1761 1554 Acked Write 5 Done Done Sent false
1762 1555 Acked Read 9 Done Done Sent false
1763 1556 Acked Read 1 Done Done Sent false
1764 1557 Acked Write 2 Done Done Sent false
1765 1558 Acked Flush 0 Done Done Sent false
1766 1559 Acked Read 10 Done Done Sent false
1767 1560 Acked Write 8 Done Done Sent false
1768 1561 Acked Flush 0 Done Done Sent false
1769 1562 Acked Write 1 Done Done Sent false
1770 1563 Acked Read 6 Done Done Sent false
1771 1564 Acked Read 6 Done Done Sent false
1772 1565 Acked Read 8 Done Done Sent false
1773 1566 Acked Flush 0 Done Done Sent false
1774 1567 Acked Write 4 Done Done Sent false
1775 1568 Acked Write 8 Done Done Sent false
1776 1569 Acked Read 10 Done Done Sent false
1777 1570 Acked Read 8 Done Done Sent false
1778 1571 Acked Flush 0 Done Done Sent false
1779 1572 Acked Write 4 Done Done Sent false
1780 1573 Acked Flush 0 Done Done Sent false
1781 1574 Acked Write 4 Done Done Sent false
1782 1575 Acked Read 10 Done Done Sent false
1783 1576 Acked Read 2 Done Done Sent false
1784 1577 Acked Read 3 Done Done Sent false
1785 1578 Acked Read 9 Done Done Sent false
1786 1579 Acked Read 2 Done Done Sent false
1787 1580 Acked Write 10 Done Done Sent false
1788 1581 Acked Read 1 Done Done Sent false
1789 1582 Acked Write 7 Done Done Sent false
1790 1583 Acked Read 9 Done Done Sent false
1791 1584 Acked Write 4 Done Done Sent false
1792 1585 Acked Read 8 Done Done Sent false
1793 1586 Acked Read 7 Done Done Sent false
1794 1587 Acked Read 1 Done Done Sent false
1795 1588 Acked Read 10 Done Done Sent false
1796 1589 Acked Write 8 Done Done Sent false
1797 1590 Acked Read 9 Done Done Sent false
1798 1591 Acked Read 10 Done Done Sent false
1799 1592 Acked Write 6 Done Done Sent false
1800 1593 Acked Write 7 Done Done Sent false
1801 1594 Acked Read 10 Done Done Sent false
1802 1595 Acked Flush 0 Done Done Sent false
1803 1596 Acked Write 2 Done Done Sent false
1804 1597 Acked Write 10 Done Done Sent false
1805 1598 Acked Read 4 Done Done Sent false
1806 1599 Acked Write 8 Done Done Sent false
1807 1600 Acked Write 1 Done Done Sent false
1808 1601 Acked Read 4 Done Done Sent false
1809 1602 Acked Read 3 Done Done Sent false
1810 STATES DS:0 DS:1 DS:2 TOTAL
1811 Sent 0 0 301 301
1812 Done 301 301 0 602
1813 Skipped 0 0 0 0
1814 Error 0 0 0 0
1815 Last Flush: Some(JobId(1595)) Some(JobId(1595)) Some(JobId(1301))
1816 Downstairs last five completed: 1301 1300 1299 1298 1297
1817 Upstairs last five completed: 1602 1601 1600 1599 1598
1818 CLIENT: Up:0 ds:301 act:2
18192025-01-28T21:54:39.360ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
18202025-01-28T21:54:40.361ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
18212025-01-28T21:54:41.363ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
1822 ----------------------------------------------------------------
1823 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1824 States: Active Active Offline
1825 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1826 1302 Acked Read 4 Done Done Sent false
1827 1303 Acked Read 9 Done Done Sent false
1828 1304 Acked Flush 0 Done Done Sent false
1829 1305 Acked Read 2 Done Done Sent false
1830 1306 Acked Read 9 Done Done Sent false
1831 1307 Acked Read 10 Done Done Sent false
1832 1308 Acked Write 1 Done Done Sent false
1833 1309 Acked Write 10 Done Done Sent false
1834 1310 Acked Write 10 Done Done Sent false
1835 1311 Acked Write 7 Done Done Sent false
1836 1312 Acked Write 5 Done Done Sent false
1837 1313 Acked Read 5 Done Done Sent false
1838 1314 Acked Write 1 Done Done Sent false
1839 1315 Acked Flush 0 Done Done Sent false
1840 1316 Acked Read 6 Done Done Sent false
1841 1317 Acked Write 6 Done Done Sent false
1842 1318 Acked Read 9 Done Done Sent false
1843 1319 Acked Write 10 Done Done Sent false
1844 1320 Acked Read 4 Done Done Sent false
1845 1321 Acked Flush 0 Done Done Sent false
1846 1322 Acked Write 5 Done Done Sent false
1847 1323 Acked Read 2 Done Done Sent false
1848 1324 Acked Write 10 Done Done Sent false
1849 1325 Acked Write 4 Done Done Sent false
1850 1326 Acked Read 4 Done Done Sent false
1851 1327 Acked Read 6 Done Done Sent false
1852 1328 Acked Write 9 Done Done Sent false
1853 1329 Acked Read 8 Done Done Sent false
1854 1330 Acked Write 9 Done Done Sent false
1855 1331 Acked Flush 0 Done Done Sent false
1856 1332 Acked Write 3 Done Done Sent false
1857 1333 Acked Read 2 Done Done Sent false
1858 1334 Acked Read 10 Done Done Sent false
1859 1335 Acked Write 8 Done Done Sent false
1860 1336 Acked Read 8 Done Done Sent false
1861 1337 Acked Flush 0 Done Done Sent false
1862 1338 Acked Write 3 Done Done Sent false
1863 1339 Acked Read 10 Done Done Sent false
1864 1340 Acked Read 6 Done Done Sent false
1865 1341 Acked Write 6 Done Done Sent false
1866 1342 Acked Read 9 Done Done Sent false
1867 1343 Acked Read 10 Done Done Sent false
1868 1344 Acked Write 3 Done Done Sent false
1869 1345 Acked Read 6 Done Done Sent false
1870 1346 Acked Read 2 Done Done Sent false
1871 1347 Acked Read 9 Done Done Sent false
1872 1348 Acked Write 3 Done Done Sent false
1873 1349 Acked Read 10 Done Done Sent false
1874 1350 Acked Read 7 Done Done Sent false
1875 1351 Acked Read 3 Done Done Sent false
1876 1352 Acked Read 9 Done Done Sent false
1877 1353 Acked Read 2 Done Done Sent false
1878 1354 Acked Read 7 Done Done Sent false
1879 1355 Acked Flush 0 Done Done Sent false
1880 1356 Acked Write 1 Done Done Sent false
1881 1357 Acked Write 10 Done Done Sent false
1882 1358 Acked Write 10 Done Done Sent false
1883 1359 Acked Read 8 Done Done Sent false
1884 1360 Acked Read 4 Done Done Sent false
1885 1361 Acked Read 7 Done Done Sent false
1886 1362 Acked Write 10 Done Done Sent false
1887 1363 Acked Flush 0 Done Done Sent false
1888 1364 Acked Read 10 Done Done Sent false
1889 1365 Acked Write 4 Done Done Sent false
1890 1366 Acked Write 3 Done Done Sent false
1891 1367 Acked Write 4 Done Done Sent false
1892 1368 Acked Read 2 Done Done Sent false
1893 1369 Acked Read 7 Done Done Sent false
1894 1370 Acked Read 2 Done Done Sent false
1895 1371 Acked Write 7 Done Done Sent false
1896 1372 Acked Write 9 Done Done Sent false
1897 1373 Acked Read 1 Done Done Sent false
1898 1374 Acked Read 10 Done Done Sent false
1899 1375 Acked Write 2 Done Done Sent false
1900 1376 Acked Write 10 Done Done Sent false
1901 1377 Acked Flush 0 Done Done Sent false
1902 1378 Acked Read 7 Done Done Sent false
1903 1379 Acked Flush 0 Done Done Sent false
1904 1380 Acked Write 10 Done Done Sent false
1905 1381 Acked Write 2 Done Done Sent false
1906 1382 Acked Read 5 Done Done Sent false
1907 1383 Acked Read 2 Done Done Sent false
1908 1384 Acked Flush 0 Done Done Sent false
1909 1385 Acked Write 9 Done Done Sent false
1910 1386 Acked Read 3 Done Done Sent false
1911 1387 Acked Write 2 Done Done Sent false
1912 1388 Acked Write 10 Done Done Sent false
1913 1389 Acked Read 10 Done Done Sent false
1914 1390 Acked Read 9 Done Done Sent false
1915 1391 Acked Flush 0 Done Done Sent false
1916 1392 Acked Read 3 Done Done Sent false
1917 1393 Acked Write 6 Done Done Sent false
1918 1394 Acked Read 8 Done Done Sent false
1919 1395 Acked Flush 0 Done Done Sent false
1920 1396 Acked Write 4 Done Done Sent false
1921 1397 Acked Write 7 Done Done Sent false
1922 1398 Acked Read 3 Done Done Sent false
1923 1399 Acked Write 6 Done Done Sent false
1924 1400 Acked Write 5 Done Done Sent false
1925 1401 Acked Read 8 Done Done Sent false
1926 1402 Acked Read 5 Done Done Sent false
1927 1403 Acked Write 2 Done Done Sent false
1928 1404 Acked Write 7 Done Done Sent false
1929 1405 Acked Flush 0 Done Done Sent false
1930 1406 Acked Read 2 Done Done Sent false
1931 1407 Acked Write 1 Done Done Sent false
1932 1408 Acked Read 2 Done Done Sent false
1933 1409 Acked Write 6 Done Done Sent false
1934 1410 Acked Write 10 Done Done Sent false
1935 1411 Acked Read 3 Done Done Sent false
1936 1412 Acked Read 7 Done Done Sent false
1937 1413 Acked Flush 0 Done Done Sent false
1938 1414 Acked Read 6 Done Done Sent false
1939 1415 Acked Read 10 Done Done Sent false
1940 1416 Acked Flush 0 Done Done Sent false
1941 1417 Acked Read 8 Done Done Sent false
1942 1418 Acked Read 3 Done Done Sent false
1943 1419 Acked Write 7 Done Done Sent false
1944 1420 Acked Write 1 Done Done Sent false
1945 1421 Acked Write 10 Done Done Sent false
1946 1422 Acked Write 6 Done Done Sent false
1947 1423 Acked Write 2 Done Done Sent false
1948 1424 Acked Read 1 Done Done Sent false
1949 1425 Acked Read 4 Done Done Sent false
1950 1426 Acked Read 2 Done Done Sent false
1951 1427 Acked Write 5 Done Done Sent false
1952 1428 Acked Read 7 Done Done Sent false
1953 1429 Acked Flush 0 Done Done Sent false
1954 1430 Acked Read 2 Done Done Sent false
1955 1431 Acked Write 7 Done Done Sent false
1956 1432 Acked Write 3 Done Done Sent false
1957 1433 Acked Read 4 Done Done Sent false
1958 1434 Acked Write 4 Done Done Sent false
1959 1435 Acked Read 10 Done Done Sent false
1960 1436 Acked Write 7 Done Done Sent false
1961 1437 Acked Write 7 Done Done Sent false
1962 1438 Acked Write 7 Done Done Sent false
1963 1439 Acked Read 9 Done Done Sent false
1964 1440 Acked Write 1 Done Done Sent false
1965 1441 Acked Flush 0 Done Done Sent false
1966 1442 Acked Read 1 Done Done Sent false
1967 1443 Acked Write 3 Done Done Sent false
1968 1444 Acked Read 6 Done Done Sent false
1969 1445 Acked Read 8 Done Done Sent false
1970 1446 Acked Read 2 Done Done Sent false
1971 1447 Acked Read 10 Done Done Sent false
1972 1448 Acked Read 5 Done Done Sent false
1973 1449 Acked Read 10 Done Done Sent false
1974 1450 Acked Write 9 Done Done Sent false
1975 1451 Acked Flush 0 Done Done Sent false
1976 1452 Acked Write 5 Done Done Sent false
1977 1453 Acked Write 2 Done Done Sent false
1978 1454 Acked Write 8 Done Done Sent false
1979 1455 Acked Read 5 Done Done Sent false
1980 1456 Acked Write 2 Done Done Sent false
1981 1457 Acked Read 5 Done Done Sent false
1982 1458 Acked Write 8 Done Done Sent false
1983 1459 Acked Read 4 Done Done Sent false
1984 1460 Acked Read 6 Done Done Sent false
1985 1461 Acked Read 8 Done Done Sent false
1986 1462 Acked Write 10 Done Done Sent false
1987 1463 Acked Write 3 Done Done Sent false
1988 1464 Acked Read 5 Done Done Sent false
1989 1465 Acked Write 3 Done Done Sent false
1990 1466 Acked Read 4 Done Done Sent false
1991 1467 Acked Read 10 Done Done Sent false
1992 1468 Acked Read 5 Done Done Sent false
1993 1469 Acked Read 9 Done Done Sent false
1994 1470 Acked Read 9 Done Done Sent false
1995 1471 Acked Write 10 Done Done Sent false
1996 1472 Acked Read 4 Done Done Sent false
1997 1473 Acked Write 1 Done Done Sent false
1998 1474 Acked Read 2 Done Done Sent false
1999 1475 Acked Write 2 Done Done Sent false
2000 1476 Acked Write 9 Done Done Sent false
2001 1477 Acked Flush 0 Done Done Sent false
2002 1478 Acked Write 2 Done Done Sent false
2003 1479 Acked Write 1 Done Done Sent false
2004 1480 Acked Read 10 Done Done Sent false
2005 1481 Acked Read 5 Done Done Sent false
2006 1482 Acked Flush 0 Done Done Sent false
2007 1483 Acked Write 3 Done Done Sent false
2008 1484 Acked Read 2 Done Done Sent false
2009 1485 Acked Write 8 Done Done Sent false
2010 1486 Acked Write 6 Done Done Sent false
2011 1487 Acked Flush 0 Done Done Sent false
2012 1488 Acked Write 10 Done Done Sent false
2013 1489 Acked Write 2 Done Done Sent false
2014 1490 Acked Read 1 Done Done Sent false
2015 1491 Acked Write 5 Done Done Sent false
2016 1492 Acked Read 3 Done Done Sent false
2017 1493 Acked Write 3 Done Done Sent false
2018 1494 Acked Read 1 Done Done Sent false
2019 1495 Acked Read 3 Done Done Sent false
2020 1496 Acked Read 9 Done Done Sent false
2021 1497 Acked Read 1 Done Done Sent false
2022 1498 Acked Write 6 Done Done Sent false
2023 1499 Acked Read 3 Done Done Sent false
2024 1500 Acked Read 2 Done Done Sent false
2025 1501 Acked Write 4 Done Done Sent false
2026 1502 Acked Write 9 Done Done Sent false
2027 1503 Acked Read 7 Done Done Sent false
2028 1504 Acked Write 7 Done Done Sent false
2029 1505 Acked Write 5 Done Done Sent false
2030 1506 Acked Read 1 Done Done Sent false
2031 1507 Acked Write 8 Done Done Sent false
2032 1508 Acked Read 5 Done Done Sent false
2033 1509 Acked Write 10 Done Done Sent false
2034 1510 Acked Write 8 Done Done Sent false
2035 1511 Acked Read 5 Done Done Sent false
2036 1512 Acked Read 1 Done Done Sent false
2037 1513 Acked Read 3 Done Done Sent false
2038 1514 Acked Read 4 Done Done Sent false
2039 1515 Acked Write 7 Done Done Sent false
2040 1516 Acked Read 1 Done Done Sent false
2041 1517 Acked Read 1 Done Done Sent false
2042 1518 Acked Read 7 Done Done Sent false
2043 1519 Acked Write 8 Done Done Sent false
2044 1520 Acked Write 4 Done Done Sent false
2045 1521 Acked Write 3 Done Done Sent false
2046 1522 Acked Read 3 Done Done Sent false
2047 1523 Acked Flush 0 Done Done Sent false
2048 1524 Acked Write 3 Done Done Sent false
2049 1525 Acked Write 10 Done Done Sent false
2050 1526 Acked Write 10 Done Done Sent false
2051 1527 Acked Flush 0 Done Done Sent false
2052 1528 Acked Read 6 Done Done Sent false
2053 1529 Acked Read 3 Done Done Sent false
2054 1530 Acked Read 4 Done Done Sent false
2055 1531 Acked Write 6 Done Done Sent false
2056 1532 Acked Read 4 Done Done Sent false
2057 1533 Acked Write 10 Done Done Sent false
2058 1534 Acked Write 10 Done Done Sent false
2059 1535 Acked Read 7 Done Done Sent false
2060 1536 Acked Write 10 Done Done Sent false
2061 1537 Acked Read 3 Done Done Sent false
2062 1538 Acked Write 2 Done Done Sent false
2063 1539 Acked Read 5 Done Done Sent false
2064 1540 Acked Read 8 Done Done Sent false
2065 1541 Acked Read 3 Done Done Sent false
2066 1542 Acked Read 10 Done Done Sent false
2067 1543 Acked Read 5 Done Done Sent false
2068 1544 Acked Write 5 Done Done Sent false
2069 1545 Acked Read 4 Done Done Sent false
2070 1546 Acked Write 9 Done Done Sent false
2071 1547 Acked Flush 0 Done Done Sent false
2072 1548 Acked Write 2 Done Done Sent false
2073 1549 Acked Write 9 Done Done Sent false
2074 1550 Acked Write 2 Done Done Sent false
2075 1551 Acked Write 4 Done Done Sent false
2076 1552 Acked Read 8 Done Done Sent false
2077 1553 Acked Read 6 Done Done Sent false
2078 1554 Acked Write 5 Done Done Sent false
2079 1555 Acked Read 9 Done Done Sent false
2080 1556 Acked Read 1 Done Done Sent false
2081 1557 Acked Write 2 Done Done Sent false
2082 1558 Acked Flush 0 Done Done Sent false
2083 1559 Acked Read 10 Done Done Sent false
2084 1560 Acked Write 8 Done Done Sent false
2085 1561 Acked Flush 0 Done Done Sent false
2086 1562 Acked Write 1 Done Done Sent false
2087 1563 Acked Read 6 Done Done Sent false
2088 1564 Acked Read 6 Done Done Sent false
2089 1565 Acked Read 8 Done Done Sent false
2090 1566 Acked Flush 0 Done Done Sent false
2091 1567 Acked Write 4 Done Done Sent false
2092 1568 Acked Write 8 Done Done Sent false
2093 1569 Acked Read 10 Done Done Sent false
2094 1570 Acked Read 8 Done Done Sent false
2095 1571 Acked Flush 0 Done Done Sent false
2096 1572 Acked Write 4 Done Done Sent false
2097 1573 Acked Flush 0 Done Done Sent false
2098 1574 Acked Write 4 Done Done Sent false
2099 1575 Acked Read 10 Done Done Sent false
2100 1576 Acked Read 2 Done Done Sent false
2101 1577 Acked Read 3 Done Done Sent false
2102 1578 Acked Read 9 Done Done Sent false
2103 1579 Acked Read 2 Done Done Sent false
2104 1580 Acked Write 10 Done Done Sent false
2105 1581 Acked Read 1 Done Done Sent false
2106 1582 Acked Write 7 Done Done Sent false
2107 1583 Acked Read 9 Done Done Sent false
2108 1584 Acked Write 4 Done Done Sent false
2109 1585 Acked Read 8 Done Done Sent false
2110 1586 Acked Read 7 Done Done Sent false
2111 1587 Acked Read 1 Done Done Sent false
2112 1588 Acked Read 10 Done Done Sent false
2113 1589 Acked Write 8 Done Done Sent false
2114 1590 Acked Read 9 Done Done Sent false
2115 1591 Acked Read 10 Done Done Sent false
2116 1592 Acked Write 6 Done Done Sent false
2117 1593 Acked Write 7 Done Done Sent false
2118 1594 Acked Read 10 Done Done Sent false
2119 1595 Acked Flush 0 Done Done Sent false
2120 1596 Acked Write 2 Done Done Sent false
2121 1597 Acked Write 10 Done Done Sent false
2122 1598 Acked Read 4 Done Done Sent false
2123 1599 Acked Write 8 Done Done Sent false
2124 1600 Acked Write 1 Done Done Sent false
2125 1601 Acked Read 4 Done Done Sent false
2126 1602 Acked Read 3 Done Done Sent false
2127 STATES DS:0 DS:1 DS:2 TOTAL
2128 Sent 0 0 301 301
2129 Done 301 301 0 602
2130 Skipped 0 0 0 0
2131 Error 0 0 0 0
2132 Last Flush: Some(JobId(1595)) Some(JobId(1595)) Some(JobId(1301))
2133 Downstairs last five completed: 1301 1300 1299 1298 1297
2134 Upstairs last five completed: 1602 1601 1600 1599 1598
2135 CLIENT: Up:0 ds:301 act:2
21362025-01-28T21:54:43.365ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
21372025-01-28T21:54:44.119ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
21382025-01-28T21:54:44.120ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
21392025-01-28T21:54:44.120ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
21402025-01-28T21:54:44.120ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
21412025-01-28T21:54:44.120ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2142 [8830] got start action
2143 [8830] I am going to restart
2144 Starting downstairs at port 8830
2145 Make output file at "/tmp/dsc/downstairs-8830.txt"
2146 [8830][2] reports Starting
2147 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1204)
2148 [8830][2] reports Running
21492025-01-28T21:54:44.621ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
21502025-01-28T21:54:45.368ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2151 ----------------------------------------------------------------
2152 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2153 States: Active Active Offline
2154 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2155 1302 Acked Read 4 Done Done Sent false
2156 1303 Acked Read 9 Done Done Sent false
2157 1304 Acked Flush 0 Done Done Sent false
2158 1305 Acked Read 2 Done Done Sent false
2159 1306 Acked Read 9 Done Done Sent false
2160 1307 Acked Read 10 Done Done Sent false
2161 1308 Acked Write 1 Done Done Sent false
2162 1309 Acked Write 10 Done Done Sent false
2163 1310 Acked Write 10 Done Done Sent false
2164 1311 Acked Write 7 Done Done Sent false
2165 1312 Acked Write 5 Done Done Sent false
2166 1313 Acked Read 5 Done Done Sent false
2167 1314 Acked Write 1 Done Done Sent false
2168 1315 Acked Flush 0 Done Done Sent false
2169 1316 Acked Read 6 Done Done Sent false
2170 1317 Acked Write 6 Done Done Sent false
2171 1318 Acked Read 9 Done Done Sent false
2172 1319 Acked Write 10 Done Done Sent false
2173 1320 Acked Read 4 Done Done Sent false
2174 1321 Acked Flush 0 Done Done Sent false
2175 1322 Acked Write 5 Done Done Sent false
2176 1323 Acked Read 2 Done Done Sent false
2177 1324 Acked Write 10 Done Done Sent false
2178 1325 Acked Write 4 Done Done Sent false
2179 1326 Acked Read 4 Done Done Sent false
2180 1327 Acked Read 6 Done Done Sent false
2181 1328 Acked Write 9 Done Done Sent false
2182 1329 Acked Read 8 Done Done Sent false
2183 1330 Acked Write 9 Done Done Sent false
2184 1331 Acked Flush 0 Done Done Sent false
2185 1332 Acked Write 3 Done Done Sent false
2186 1333 Acked Read 2 Done Done Sent false
2187 1334 Acked Read 10 Done Done Sent false
2188 1335 Acked Write 8 Done Done Sent false
2189 1336 Acked Read 8 Done Done Sent false
2190 1337 Acked Flush 0 Done Done Sent false
2191 1338 Acked Write 3 Done Done Sent false
2192 1339 Acked Read 10 Done Done Sent false
2193 1340 Acked Read 6 Done Done Sent false
2194 1341 Acked Write 6 Done Done Sent false
2195 1342 Acked Read 9 Done Done Sent false
2196 1343 Acked Read 10 Done Done Sent false
2197 1344 Acked Write 3 Done Done Sent false
2198 1345 Acked Read 6 Done Done Sent false
2199 1346 Acked Read 2 Done Done Sent false
2200 1347 Acked Read 9 Done Done Sent false
2201 1348 Acked Write 3 Done Done Sent false
2202 1349 Acked Read 10 Done Done Sent false
2203 1350 Acked Read 7 Done Done Sent false
2204 1351 Acked Read 3 Done Done Sent false
2205 1352 Acked Read 9 Done Done Sent false
2206 1353 Acked Read 2 Done Done Sent false
2207 1354 Acked Read 7 Done Done Sent false
2208 1355 Acked Flush 0 Done Done Sent false
2209 1356 Acked Write 1 Done Done Sent false
2210 1357 Acked Write 10 Done Done Sent false
2211 1358 Acked Write 10 Done Done Sent false
2212 1359 Acked Read 8 Done Done Sent false
2213 1360 Acked Read 4 Done Done Sent false
2214 1361 Acked Read 7 Done Done Sent false
2215 1362 Acked Write 10 Done Done Sent false
2216 1363 Acked Flush 0 Done Done Sent false
2217 1364 Acked Read 10 Done Done Sent false
2218 1365 Acked Write 4 Done Done Sent false
2219 1366 Acked Write 3 Done Done Sent false
2220 1367 Acked Write 4 Done Done Sent false
2221 1368 Acked Read 2 Done Done Sent false
2222 1369 Acked Read 7 Done Done Sent false
2223 1370 Acked Read 2 Done Done Sent false
2224 1371 Acked Write 7 Done Done Sent false
2225 1372 Acked Write 9 Done Done Sent false
2226 1373 Acked Read 1 Done Done Sent false
2227 1374 Acked Read 10 Done Done Sent false
2228 1375 Acked Write 2 Done Done Sent false
2229 1376 Acked Write 10 Done Done Sent false
2230 1377 Acked Flush 0 Done Done Sent false
2231 1378 Acked Read 7 Done Done Sent false
2232 1379 Acked Flush 0 Done Done Sent false
2233 1380 Acked Write 10 Done Done Sent false
2234 1381 Acked Write 2 Done Done Sent false
2235 1382 Acked Read 5 Done Done Sent false
2236 1383 Acked Read 2 Done Done Sent false
2237 1384 Acked Flush 0 Done Done Sent false
2238 1385 Acked Write 9 Done Done Sent false
2239 1386 Acked Read 3 Done Done Sent false
2240 1387 Acked Write 2 Done Done Sent false
2241 1388 Acked Write 10 Done Done Sent false
2242 1389 Acked Read 10 Done Done Sent false
2243 1390 Acked Read 9 Done Done Sent false
2244 1391 Acked Flush 0 Done Done Sent false
2245 1392 Acked Read 3 Done Done Sent false
2246 1393 Acked Write 6 Done Done Sent false
2247 1394 Acked Read 8 Done Done Sent false
2248 1395 Acked Flush 0 Done Done Sent false
2249 1396 Acked Write 4 Done Done Sent false
2250 1397 Acked Write 7 Done Done Sent false
2251 1398 Acked Read 3 Done Done Sent false
2252 1399 Acked Write 6 Done Done Sent false
2253 1400 Acked Write 5 Done Done Sent false
2254 1401 Acked Read 8 Done Done Sent false
2255 1402 Acked Read 5 Done Done Sent false
2256 1403 Acked Write 2 Done Done Sent false
2257 1404 Acked Write 7 Done Done Sent false
2258 1405 Acked Flush 0 Done Done Sent false
2259 1406 Acked Read 2 Done Done Sent false
2260 1407 Acked Write 1 Done Done Sent false
2261 1408 Acked Read 2 Done Done Sent false
2262 1409 Acked Write 6 Done Done Sent false
2263 1410 Acked Write 10 Done Done Sent false
2264 1411 Acked Read 3 Done Done Sent false
2265 1412 Acked Read 7 Done Done Sent false
2266 1413 Acked Flush 0 Done Done Sent false
2267 1414 Acked Read 6 Done Done Sent false
2268 1415 Acked Read 10 Done Done Sent false
2269 1416 Acked Flush 0 Done Done Sent false
2270 1417 Acked Read 8 Done Done Sent false
2271 1418 Acked Read 3 Done Done Sent false
2272 1419 Acked Write 7 Done Done Sent false
2273 1420 Acked Write 1 Done Done Sent false
2274 1421 Acked Write 10 Done Done Sent false
2275 1422 Acked Write 6 Done Done Sent false
2276 1423 Acked Write 2 Done Done Sent false
2277 1424 Acked Read 1 Done Done Sent false
2278 1425 Acked Read 4 Done Done Sent false
2279 1426 Acked Read 2 Done Done Sent false
2280 1427 Acked Write 5 Done Done Sent false
2281 1428 Acked Read 7 Done Done Sent false
2282 1429 Acked Flush 0 Done Done Sent false
2283 1430 Acked Read 2 Done Done Sent false
2284 1431 Acked Write 7 Done Done Sent false
2285 1432 Acked Write 3 Done Done Sent false
2286 1433 Acked Read 4 Done Done Sent false
2287 1434 Acked Write 4 Done Done Sent false
2288 1435 Acked Read 10 Done Done Sent false
2289 1436 Acked Write 7 Done Done Sent false
2290 1437 Acked Write 7 Done Done Sent false
2291 1438 Acked Write 7 Done Done Sent false
2292 1439 Acked Read 9 Done Done Sent false
2293 1440 Acked Write 1 Done Done Sent false
2294 1441 Acked Flush 0 Done Done Sent false
2295 1442 Acked Read 1 Done Done Sent false
2296 1443 Acked Write 3 Done Done Sent false
2297 1444 Acked Read 6 Done Done Sent false
2298 1445 Acked Read 8 Done Done Sent false
2299 1446 Acked Read 2 Done Done Sent false
2300 1447 Acked Read 10 Done Done Sent false
2301 1448 Acked Read 5 Done Done Sent false
2302 1449 Acked Read 10 Done Done Sent false
2303 1450 Acked Write 9 Done Done Sent false
2304 1451 Acked Flush 0 Done Done Sent false
2305 1452 Acked Write 5 Done Done Sent false
2306 1453 Acked Write 2 Done Done Sent false
2307 1454 Acked Write 8 Done Done Sent false
2308 1455 Acked Read 5 Done Done Sent false
2309 1456 Acked Write 2 Done Done Sent false
2310 1457 Acked Read 5 Done Done Sent false
2311 1458 Acked Write 8 Done Done Sent false
2312 1459 Acked Read 4 Done Done Sent false
2313 1460 Acked Read 6 Done Done Sent false
2314 1461 Acked Read 8 Done Done Sent false
2315 1462 Acked Write 10 Done Done Sent false
2316 1463 Acked Write 3 Done Done Sent false
2317 1464 Acked Read 5 Done Done Sent false
2318 1465 Acked Write 3 Done Done Sent false
2319 1466 Acked Read 4 Done Done Sent false
2320 1467 Acked Read 10 Done Done Sent false
2321 1468 Acked Read 5 Done Done Sent false
2322 1469 Acked Read 9 Done Done Sent false
2323 1470 Acked Read 9 Done Done Sent false
2324 1471 Acked Write 10 Done Done Sent false
2325 1472 Acked Read 4 Done Done Sent false
2326 1473 Acked Write 1 Done Done Sent false
2327 1474 Acked Read 2 Done Done Sent false
2328 1475 Acked Write 2 Done Done Sent false
2329 1476 Acked Write 9 Done Done Sent false
2330 1477 Acked Flush 0 Done Done Sent false
2331 1478 Acked Write 2 Done Done Sent false
2332 1479 Acked Write 1 Done Done Sent false
2333 1480 Acked Read 10 Done Done Sent false
2334 1481 Acked Read 5 Done Done Sent false
2335 1482 Acked Flush 0 Done Done Sent false
2336 1483 Acked Write 3 Done Done Sent false
2337 1484 Acked Read 2 Done Done Sent false
2338 1485 Acked Write 8 Done Done Sent false
2339 1486 Acked Write 6 Done Done Sent false
2340 1487 Acked Flush 0 Done Done Sent false
2341 1488 Acked Write 10 Done Done Sent false
2342 1489 Acked Write 2 Done Done Sent false
2343 1490 Acked Read 1 Done Done Sent false
2344 1491 Acked Write 5 Done Done Sent false
2345 1492 Acked Read 3 Done Done Sent false
2346 1493 Acked Write 3 Done Done Sent false
2347 1494 Acked Read 1 Done Done Sent false
2348 1495 Acked Read 3 Done Done Sent false
2349 1496 Acked Read 9 Done Done Sent false
2350 1497 Acked Read 1 Done Done Sent false
2351 1498 Acked Write 6 Done Done Sent false
2352 1499 Acked Read 3 Done Done Sent false
2353 1500 Acked Read 2 Done Done Sent false
2354 1501 Acked Write 4 Done Done Sent false
2355 1502 Acked Write 9 Done Done Sent false
2356 1503 Acked Read 7 Done Done Sent false
2357 1504 Acked Write 7 Done Done Sent false
2358 1505 Acked Write 5 Done Done Sent false
2359 1506 Acked Read 1 Done Done Sent false
2360 1507 Acked Write 8 Done Done Sent false
2361 1508 Acked Read 5 Done Done Sent false
2362 1509 Acked Write 10 Done Done Sent false
2363 1510 Acked Write 8 Done Done Sent false
2364 1511 Acked Read 5 Done Done Sent false
2365 1512 Acked Read 1 Done Done Sent false
2366 1513 Acked Read 3 Done Done Sent false
2367 1514 Acked Read 4 Done Done Sent false
2368 1515 Acked Write 7 Done Done Sent false
2369 1516 Acked Read 1 Done Done Sent false
2370 1517 Acked Read 1 Done Done Sent false
2371 1518 Acked Read 7 Done Done Sent false
2372 1519 Acked Write 8 Done Done Sent false
2373 1520 Acked Write 4 Done Done Sent false
2374 1521 Acked Write 3 Done Done Sent false
2375 1522 Acked Read 3 Done Done Sent false
2376 1523 Acked Flush 0 Done Done Sent false
2377 1524 Acked Write 3 Done Done Sent false
2378 1525 Acked Write 10 Done Done Sent false
2379 1526 Acked Write 10 Done Done Sent false
2380 1527 Acked Flush 0 Done Done Sent false
2381 1528 Acked Read 6 Done Done Sent false
2382 1529 Acked Read 3 Done Done Sent false
2383 1530 Acked Read 4 Done Done Sent false
2384 1531 Acked Write 6 Done Done Sent false
2385 1532 Acked Read 4 Done Done Sent false
2386 1533 Acked Write 10 Done Done Sent false
2387 1534 Acked Write 10 Done Done Sent false
2388 1535 Acked Read 7 Done Done Sent false
2389 1536 Acked Write 10 Done Done Sent false
2390 1537 Acked Read 3 Done Done Sent false
2391 1538 Acked Write 2 Done Done Sent false
2392 1539 Acked Read 5 Done Done Sent false
2393 1540 Acked Read 8 Done Done Sent false
2394 1541 Acked Read 3 Done Done Sent false
2395 1542 Acked Read 10 Done Done Sent false
2396 1543 Acked Read 5 Done Done Sent false
2397 1544 Acked Write 5 Done Done Sent false
2398 1545 Acked Read 4 Done Done Sent false
2399 1546 Acked Write 9 Done Done Sent false
2400 1547 Acked Flush 0 Done Done Sent false
2401 1548 Acked Write 2 Done Done Sent false
2402 1549 Acked Write 9 Done Done Sent false
2403 1550 Acked Write 2 Done Done Sent false
2404 1551 Acked Write 4 Done Done Sent false
2405 1552 Acked Read 8 Done Done Sent false
2406 1553 Acked Read 6 Done Done Sent false
2407 1554 Acked Write 5 Done Done Sent false
2408 1555 Acked Read 9 Done Done Sent false
2409 1556 Acked Read 1 Done Done Sent false
2410 1557 Acked Write 2 Done Done Sent false
2411 1558 Acked Flush 0 Done Done Sent false
2412 1559 Acked Read 10 Done Done Sent false
2413 1560 Acked Write 8 Done Done Sent false
2414 1561 Acked Flush 0 Done Done Sent false
2415 1562 Acked Write 1 Done Done Sent false
2416 1563 Acked Read 6 Done Done Sent false
2417 1564 Acked Read 6 Done Done Sent false
2418 1565 Acked Read 8 Done Done Sent false
2419 1566 Acked Flush 0 Done Done Sent false
2420 1567 Acked Write 4 Done Done Sent false
2421 1568 Acked Write 8 Done Done Sent false
2422 1569 Acked Read 10 Done Done Sent false
2423 1570 Acked Read 8 Done Done Sent false
2424 1571 Acked Flush 0 Done Done Sent false
2425 1572 Acked Write 4 Done Done Sent false
2426 1573 Acked Flush 0 Done Done Sent false
2427 1574 Acked Write 4 Done Done Sent false
2428 1575 Acked Read 10 Done Done Sent false
2429 1576 Acked Read 2 Done Done Sent false
2430 1577 Acked Read 3 Done Done Sent false
2431 1578 Acked Read 9 Done Done Sent false
2432 1579 Acked Read 2 Done Done Sent false
2433 1580 Acked Write 10 Done Done Sent false
2434 1581 Acked Read 1 Done Done Sent false
2435 1582 Acked Write 7 Done Done Sent false
2436 1583 Acked Read 9 Done Done Sent false
2437 1584 Acked Write 4 Done Done Sent false
2438 1585 Acked Read 8 Done Done Sent false
2439 1586 Acked Read 7 Done Done Sent false
2440 1587 Acked Read 1 Done Done Sent false
2441 1588 Acked Read 10 Done Done Sent false
2442 1589 Acked Write 8 Done Done Sent false
2443 1590 Acked Read 9 Done Done Sent false
2444 1591 Acked Read 10 Done Done Sent false
2445 1592 Acked Write 6 Done Done Sent false
2446 1593 Acked Write 7 Done Done Sent false
2447 1594 Acked Read 10 Done Done Sent false
2448 1595 Acked Flush 0 Done Done Sent false
2449 1596 Acked Write 2 Done Done Sent false
2450 1597 Acked Write 10 Done Done Sent false
2451 1598 Acked Read 4 Done Done Sent false
2452 1599 Acked Write 8 Done Done Sent false
2453 1600 Acked Write 1 Done Done Sent false
2454 1601 Acked Read 4 Done Done Sent false
2455 1602 Acked Read 3 Done Done Sent false
2456 STATES DS:0 DS:1 DS:2 TOTAL
2457 Sent 0 0 301 301
2458 Done 301 301 0 602
2459 Skipped 0 0 0 0
2460 Error 0 0 0 0
2461 Last Flush: Some(JobId(1595)) Some(JobId(1595)) Some(JobId(1301))
2462 Downstairs last five completed: 1301 1300 1299 1298 1297
2463 Upstairs last five completed: 1602 1601 1600 1599 1598
2464 CLIENT: Up:0 ds:301 act:2
24652025-01-28T21:54:47.379ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
24662025-01-28T21:54:48.373ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
24672025-01-28T21:54:49.374ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2468 ----------------------------------------------------------------
2469 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2470 States: Active Active Offline
2471 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2472 1302 Acked Read 4 Done Done Sent false
2473 1303 Acked Read 9 Done Done Sent false
2474 1304 Acked Flush 0 Done Done Sent false
2475 1305 Acked Read 2 Done Done Sent false
2476 1306 Acked Read 9 Done Done Sent false
2477 1307 Acked Read 10 Done Done Sent false
2478 1308 Acked Write 1 Done Done Sent false
2479 1309 Acked Write 10 Done Done Sent false
2480 1310 Acked Write 10 Done Done Sent false
2481 1311 Acked Write 7 Done Done Sent false
2482 1312 Acked Write 5 Done Done Sent false
2483 1313 Acked Read 5 Done Done Sent false
2484 1314 Acked Write 1 Done Done Sent false
2485 1315 Acked Flush 0 Done Done Sent false
2486 1316 Acked Read 6 Done Done Sent false
2487 1317 Acked Write 6 Done Done Sent false
2488 1318 Acked Read 9 Done Done Sent false
2489 1319 Acked Write 10 Done Done Sent false
2490 1320 Acked Read 4 Done Done Sent false
2491 1321 Acked Flush 0 Done Done Sent false
2492 1322 Acked Write 5 Done Done Sent false
2493 1323 Acked Read 2 Done Done Sent false
2494 1324 Acked Write 10 Done Done Sent false
2495 1325 Acked Write 4 Done Done Sent false
2496 1326 Acked Read 4 Done Done Sent false
2497 1327 Acked Read 6 Done Done Sent false
2498 1328 Acked Write 9 Done Done Sent false
2499 1329 Acked Read 8 Done Done Sent false
2500 1330 Acked Write 9 Done Done Sent false
2501 1331 Acked Flush 0 Done Done Sent false
2502 1332 Acked Write 3 Done Done Sent false
2503 1333 Acked Read 2 Done Done Sent false
2504 1334 Acked Read 10 Done Done Sent false
2505 1335 Acked Write 8 Done Done Sent false
2506 1336 Acked Read 8 Done Done Sent false
2507 1337 Acked Flush 0 Done Done Sent false
2508 1338 Acked Write 3 Done Done Sent false
2509 1339 Acked Read 10 Done Done Sent false
2510 1340 Acked Read 6 Done Done Sent false
2511 1341 Acked Write 6 Done Done Sent false
2512 1342 Acked Read 9 Done Done Sent false
2513 1343 Acked Read 10 Done Done Sent false
2514 1344 Acked Write 3 Done Done Sent false
2515 1345 Acked Read 6 Done Done Sent false
2516 1346 Acked Read 2 Done Done Sent false
2517 1347 Acked Read 9 Done Done Sent false
2518 1348 Acked Write 3 Done Done Sent false
2519 1349 Acked Read 10 Done Done Sent false
2520 1350 Acked Read 7 Done Done Sent false
2521 1351 Acked Read 3 Done Done Sent false
2522 1352 Acked Read 9 Done Done Sent false
2523 1353 Acked Read 2 Done Done Sent false
2524 1354 Acked Read 7 Done Done Sent false
2525 1355 Acked Flush 0 Done Done Sent false
2526 1356 Acked Write 1 Done Done Sent false
2527 1357 Acked Write 10 Done Done Sent false
2528 1358 Acked Write 10 Done Done Sent false
2529 1359 Acked Read 8 Done Done Sent false
2530 1360 Acked Read 4 Done Done Sent false
2531 1361 Acked Read 7 Done Done Sent false
2532 1362 Acked Write 10 Done Done Sent false
2533 1363 Acked Flush 0 Done Done Sent false
2534 1364 Acked Read 10 Done Done Sent false
2535 1365 Acked Write 4 Done Done Sent false
2536 1366 Acked Write 3 Done Done Sent false
2537 1367 Acked Write 4 Done Done Sent false
2538 1368 Acked Read 2 Done Done Sent false
2539 1369 Acked Read 7 Done Done Sent false
2540 1370 Acked Read 2 Done Done Sent false
2541 1371 Acked Write 7 Done Done Sent false
2542 1372 Acked Write 9 Done Done Sent false
2543 1373 Acked Read 1 Done Done Sent false
2544 1374 Acked Read 10 Done Done Sent false
2545 1375 Acked Write 2 Done Done Sent false
2546 1376 Acked Write 10 Done Done Sent false
2547 1377 Acked Flush 0 Done Done Sent false
2548 1378 Acked Read 7 Done Done Sent false
2549 1379 Acked Flush 0 Done Done Sent false
2550 1380 Acked Write 10 Done Done Sent false
2551 1381 Acked Write 2 Done Done Sent false
2552 1382 Acked Read 5 Done Done Sent false
2553 1383 Acked Read 2 Done Done Sent false
2554 1384 Acked Flush 0 Done Done Sent false
2555 1385 Acked Write 9 Done Done Sent false
2556 1386 Acked Read 3 Done Done Sent false
2557 1387 Acked Write 2 Done Done Sent false
2558 1388 Acked Write 10 Done Done Sent false
2559 1389 Acked Read 10 Done Done Sent false
2560 1390 Acked Read 9 Done Done Sent false
2561 1391 Acked Flush 0 Done Done Sent false
2562 1392 Acked Read 3 Done Done Sent false
2563 1393 Acked Write 6 Done Done Sent false
2564 1394 Acked Read 8 Done Done Sent false
2565 1395 Acked Flush 0 Done Done Sent false
2566 1396 Acked Write 4 Done Done Sent false
2567 1397 Acked Write 7 Done Done Sent false
2568 1398 Acked Read 3 Done Done Sent false
2569 1399 Acked Write 6 Done Done Sent false
2570 1400 Acked Write 5 Done Done Sent false
2571 1401 Acked Read 8 Done Done Sent false
2572 1402 Acked Read 5 Done Done Sent false
2573 1403 Acked Write 2 Done Done Sent false
2574 1404 Acked Write 7 Done Done Sent false
2575 1405 Acked Flush 0 Done Done Sent false
2576 1406 Acked Read 2 Done Done Sent false
2577 1407 Acked Write 1 Done Done Sent false
2578 1408 Acked Read 2 Done Done Sent false
2579 1409 Acked Write 6 Done Done Sent false
2580 1410 Acked Write 10 Done Done Sent false
2581 1411 Acked Read 3 Done Done Sent false
2582 1412 Acked Read 7 Done Done Sent false
2583 1413 Acked Flush 0 Done Done Sent false
2584 1414 Acked Read 6 Done Done Sent false
2585 1415 Acked Read 10 Done Done Sent false
2586 1416 Acked Flush 0 Done Done Sent false
2587 1417 Acked Read 8 Done Done Sent false
2588 1418 Acked Read 3 Done Done Sent false
2589 1419 Acked Write 7 Done Done Sent false
2590 1420 Acked Write 1 Done Done Sent false
2591 1421 Acked Write 10 Done Done Sent false
2592 1422 Acked Write 6 Done Done Sent false
2593 1423 Acked Write 2 Done Done Sent false
2594 1424 Acked Read 1 Done Done Sent false
2595 1425 Acked Read 4 Done Done Sent false
2596 1426 Acked Read 2 Done Done Sent false
2597 1427 Acked Write 5 Done Done Sent false
2598 1428 Acked Read 7 Done Done Sent false
2599 1429 Acked Flush 0 Done Done Sent false
2600 1430 Acked Read 2 Done Done Sent false
2601 1431 Acked Write 7 Done Done Sent false
2602 1432 Acked Write 3 Done Done Sent false
2603 1433 Acked Read 4 Done Done Sent false
2604 1434 Acked Write 4 Done Done Sent false
2605 1435 Acked Read 10 Done Done Sent false
2606 1436 Acked Write 7 Done Done Sent false
2607 1437 Acked Write 7 Done Done Sent false
2608 1438 Acked Write 7 Done Done Sent false
2609 1439 Acked Read 9 Done Done Sent false
2610 1440 Acked Write 1 Done Done Sent false
2611 1441 Acked Flush 0 Done Done Sent false
2612 1442 Acked Read 1 Done Done Sent false
2613 1443 Acked Write 3 Done Done Sent false
2614 1444 Acked Read 6 Done Done Sent false
2615 1445 Acked Read 8 Done Done Sent false
2616 1446 Acked Read 2 Done Done Sent false
2617 1447 Acked Read 10 Done Done Sent false
2618 1448 Acked Read 5 Done Done Sent false
2619 1449 Acked Read 10 Done Done Sent false
2620 1450 Acked Write 9 Done Done Sent false
2621 1451 Acked Flush 0 Done Done Sent false
2622 1452 Acked Write 5 Done Done Sent false
2623 1453 Acked Write 2 Done Done Sent false
2624 1454 Acked Write 8 Done Done Sent false
2625 1455 Acked Read 5 Done Done Sent false
2626 1456 Acked Write 2 Done Done Sent false
2627 1457 Acked Read 5 Done Done Sent false
2628 1458 Acked Write 8 Done Done Sent false
2629 1459 Acked Read 4 Done Done Sent false
2630 1460 Acked Read 6 Done Done Sent false
2631 1461 Acked Read 8 Done Done Sent false
2632 1462 Acked Write 10 Done Done Sent false
2633 1463 Acked Write 3 Done Done Sent false
2634 1464 Acked Read 5 Done Done Sent false
2635 1465 Acked Write 3 Done Done Sent false
2636 1466 Acked Read 4 Done Done Sent false
2637 1467 Acked Read 10 Done Done Sent false
2638 1468 Acked Read 5 Done Done Sent false
2639 1469 Acked Read 9 Done Done Sent false
2640 1470 Acked Read 9 Done Done Sent false
2641 1471 Acked Write 10 Done Done Sent false
2642 1472 Acked Read 4 Done Done Sent false
2643 1473 Acked Write 1 Done Done Sent false
2644 1474 Acked Read 2 Done Done Sent false
2645 1475 Acked Write 2 Done Done Sent false
2646 1476 Acked Write 9 Done Done Sent false
2647 1477 Acked Flush 0 Done Done Sent false
2648 1478 Acked Write 2 Done Done Sent false
2649 1479 Acked Write 1 Done Done Sent false
2650 1480 Acked Read 10 Done Done Sent false
2651 1481 Acked Read 5 Done Done Sent false
2652 1482 Acked Flush 0 Done Done Sent false
2653 1483 Acked Write 3 Done Done Sent false
2654 1484 Acked Read 2 Done Done Sent false
2655 1485 Acked Write 8 Done Done Sent false
2656 1486 Acked Write 6 Done Done Sent false
2657 1487 Acked Flush 0 Done Done Sent false
2658 1488 Acked Write 10 Done Done Sent false
2659 1489 Acked Write 2 Done Done Sent false
2660 1490 Acked Read 1 Done Done Sent false
2661 1491 Acked Write 5 Done Done Sent false
2662 1492 Acked Read 3 Done Done Sent false
2663 1493 Acked Write 3 Done Done Sent false
2664 1494 Acked Read 1 Done Done Sent false
2665 1495 Acked Read 3 Done Done Sent false
2666 1496 Acked Read 9 Done Done Sent false
2667 1497 Acked Read 1 Done Done Sent false
2668 1498 Acked Write 6 Done Done Sent false
2669 1499 Acked Read 3 Done Done Sent false
2670 1500 Acked Read 2 Done Done Sent false
2671 1501 Acked Write 4 Done Done Sent false
2672 1502 Acked Write 9 Done Done Sent false
2673 1503 Acked Read 7 Done Done Sent false
2674 1504 Acked Write 7 Done Done Sent false
2675 1505 Acked Write 5 Done Done Sent false
2676 1506 Acked Read 1 Done Done Sent false
2677 1507 Acked Write 8 Done Done Sent false
2678 1508 Acked Read 5 Done Done Sent false
2679 1509 Acked Write 10 Done Done Sent false
2680 1510 Acked Write 8 Done Done Sent false
2681 1511 Acked Read 5 Done Done Sent false
2682 1512 Acked Read 1 Done Done Sent false
2683 1513 Acked Read 3 Done Done Sent false
2684 1514 Acked Read 4 Done Done Sent false
2685 1515 Acked Write 7 Done Done Sent false
2686 1516 Acked Read 1 Done Done Sent false
2687 1517 Acked Read 1 Done Done Sent false
2688 1518 Acked Read 7 Done Done Sent false
2689 1519 Acked Write 8 Done Done Sent false
2690 1520 Acked Write 4 Done Done Sent false
2691 1521 Acked Write 3 Done Done Sent false
2692 1522 Acked Read 3 Done Done Sent false
2693 1523 Acked Flush 0 Done Done Sent false
2694 1524 Acked Write 3 Done Done Sent false
2695 1525 Acked Write 10 Done Done Sent false
2696 1526 Acked Write 10 Done Done Sent false
2697 1527 Acked Flush 0 Done Done Sent false
2698 1528 Acked Read 6 Done Done Sent false
2699 1529 Acked Read 3 Done Done Sent false
2700 1530 Acked Read 4 Done Done Sent false
2701 1531 Acked Write 6 Done Done Sent false
2702 1532 Acked Read 4 Done Done Sent false
2703 1533 Acked Write 10 Done Done Sent false
2704 1534 Acked Write 10 Done Done Sent false
2705 1535 Acked Read 7 Done Done Sent false
2706 1536 Acked Write 10 Done Done Sent false
2707 1537 Acked Read 3 Done Done Sent false
2708 1538 Acked Write 2 Done Done Sent false
2709 1539 Acked Read 5 Done Done Sent false
2710 1540 Acked Read 8 Done Done Sent false
2711 1541 Acked Read 3 Done Done Sent false
2712 1542 Acked Read 10 Done Done Sent false
2713 1543 Acked Read 5 Done Done Sent false
2714 1544 Acked Write 5 Done Done Sent false
2715 1545 Acked Read 4 Done Done Sent false
2716 1546 Acked Write 9 Done Done Sent false
2717 1547 Acked Flush 0 Done Done Sent false
2718 1548 Acked Write 2 Done Done Sent false
2719 1549 Acked Write 9 Done Done Sent false
2720 1550 Acked Write 2 Done Done Sent false
2721 1551 Acked Write 4 Done Done Sent false
2722 1552 Acked Read 8 Done Done Sent false
2723 1553 Acked Read 6 Done Done Sent false
2724 1554 Acked Write 5 Done Done Sent false
2725 1555 Acked Read 9 Done Done Sent false
2726 1556 Acked Read 1 Done Done Sent false
2727 1557 Acked Write 2 Done Done Sent false
2728 1558 Acked Flush 0 Done Done Sent false
2729 1559 Acked Read 10 Done Done Sent false
2730 1560 Acked Write 8 Done Done Sent false
2731 1561 Acked Flush 0 Done Done Sent false
2732 1562 Acked Write 1 Done Done Sent false
2733 1563 Acked Read 6 Done Done Sent false
2734 1564 Acked Read 6 Done Done Sent false
2735 1565 Acked Read 8 Done Done Sent false
2736 1566 Acked Flush 0 Done Done Sent false
2737 1567 Acked Write 4 Done Done Sent false
2738 1568 Acked Write 8 Done Done Sent false
2739 1569 Acked Read 10 Done Done Sent false
2740 1570 Acked Read 8 Done Done Sent false
2741 1571 Acked Flush 0 Done Done Sent false
2742 1572 Acked Write 4 Done Done Sent false
2743 1573 Acked Flush 0 Done Done Sent false
2744 1574 Acked Write 4 Done Done Sent false
2745 1575 Acked Read 10 Done Done Sent false
2746 1576 Acked Read 2 Done Done Sent false
2747 1577 Acked Read 3 Done Done Sent false
2748 1578 Acked Read 9 Done Done Sent false
2749 1579 Acked Read 2 Done Done Sent false
2750 1580 Acked Write 10 Done Done Sent false
2751 1581 Acked Read 1 Done Done Sent false
2752 1582 Acked Write 7 Done Done Sent false
2753 1583 Acked Read 9 Done Done Sent false
2754 1584 Acked Write 4 Done Done Sent false
2755 1585 Acked Read 8 Done Done Sent false
2756 1586 Acked Read 7 Done Done Sent false
2757 1587 Acked Read 1 Done Done Sent false
2758 1588 Acked Read 10 Done Done Sent false
2759 1589 Acked Write 8 Done Done Sent false
2760 1590 Acked Read 9 Done Done Sent false
2761 1591 Acked Read 10 Done Done Sent false
2762 1592 Acked Write 6 Done Done Sent false
2763 1593 Acked Write 7 Done Done Sent false
2764 1594 Acked Read 10 Done Done Sent false
2765 1595 Acked Flush 0 Done Done Sent false
2766 1596 Acked Write 2 Done Done Sent false
2767 1597 Acked Write 10 Done Done Sent false
2768 1598 Acked Read 4 Done Done Sent false
2769 1599 Acked Write 8 Done Done Sent false
2770 1600 Acked Write 1 Done Done Sent false
2771 1601 Acked Read 4 Done Done Sent false
2772 1602 Acked Read 3 Done Done Sent false
2773 STATES DS:0 DS:1 DS:2 TOTAL
2774 Sent 0 0 301 301
2775 Done 301 301 0 602
2776 Skipped 0 0 0 0
2777 Error 0 0 0 0
2778 Last Flush: Some(JobId(1595)) Some(JobId(1595)) Some(JobId(1301))
2779 Downstairs last five completed: 1301 1300 1299 1298 1297
2780 Upstairs last five completed: 1602 1601 1600 1599 1598
2781 CLIENT: Up:0 ds:301 act:2
27822025-01-28T21:54:50.870ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27832025-01-28T21:54:51.375ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27842025-01-28T21:54:52.380ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27852025-01-28T21:54:53.378ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27862025-01-28T21:54:54.121ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27872025-01-28T21:54:54.121ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:35013) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27882025-01-28T21:54:54.122ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27892025-01-28T21:54:54.123ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27902025-01-28T21:54:54.123ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27912025-01-28T21:54:54.123ZINFOcrucible: send last flush ID to this DS: Some(JobId(1301)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27922025-01-28T21:54:54.123ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27932025-01-28T21:54:54.123ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1301)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27942025-01-28T21:54:54.123ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1301)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2795 ----------------------------------------------------------------
2796 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:7
2797 States: Active Active Active
2798 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2799 1596 Acked Write 2 Done Done Done true
2800 1597 Acked Write 10 Done Done Done true
2801 1598 Acked Read 4 Done Done Done true
2802 1599 Acked Write 8 Done Done Done true
2803 1600 Acked Write 1 Done Done Done true
2804 1601 Acked Read 4 Done Done Done true
2805 1602 Acked Read 3 Done Done Done true
2806 STATES DS:0 DS:1 DS:2 TOTAL
2807 Sent 0 0 0 0
2808 Done 7 7 7 21
2809 Skipped 0 0 0 0
2810 Error 0 0 0 0
2811 Last Flush: Some(JobId(1595)) Some(JobId(1595)) Some(JobId(1595))
2812 Downstairs last five completed: 1595 1594 1593 1592 1591
2813 Upstairs last five completed: 1602 1601 1600 1599 1598
2814 CLIENT: Up:0 ds:7 act:3
2815 ----------------------------------------------------------------
2816 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2817 Downstairs last five completed: 1603 1602 1601 1600 1599
2818 Upstairs last five completed: 1603 1602 1601 1600 1599
2819 CLIENT: Up:0 ds:0 act:3
2820 Replay: All jobs finished
2821 Main task has work to do, go find it
2822 got dsc Stop(2)
2823 stop 2
2824 Jan 28 21:54:58.372 INFO request completed, latency_us: 371, [8830] Got stop action so:false kr:false
2825 response_code: 204, uri: /stop/cid/2, method: POST, req_id: b7b73a87-a4de-417d-a0bb-be33ad1a5f1b, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
28262025-01-28T21:54:58.372ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
28272025-01-28T21:54:58.373ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
28282025-01-28T21:54:58.373ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
2829 Jan 28 21:54:58.373 INFO request completed, latency_us: 172, response_code{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:54:58.37320141Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
2830 : 200, uri: /state/cid/2, method: GET, req_id: f662c66a-13c9-4cbb-8144-7fa37aa24fc6, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
2831 [8830] Exited with: ExitStatus(unix_wait_status(9))
2832 [8830][2] reports Exit
2833 Jan 28 21:55:02.375 INFO request completed, latency_us: 375, response_code: 200, uri: /state/cid/2, method: GET, req_id: b9815c2f-dafd-42dc-8711-5ae858184a1a, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
2834 001/300 Write block 1425 len 8192 data: 2 2
2835 002/300 Write block 2860 len 8192 data: 2 2
2836 003/300 Write block 3315 len 40960 data: 2 2 2 2 2 3 3 4 4 4
2837 004/300 Write block 1524 len 36864 data: 3 3 2 3 3 2 2 2 2
2838 005/300 Read block 1263 len 20480
2839 006/300 Read block 3028 len 20480
2840 007/300 Read block 2894 len 8192
2841 008/300 Read block 1568 len 16384
2842 009/300 Read block 1093 len 20480
2843 010/300 Write block 3013 len 20480 data: 2 2 2 2 2
2844 011/300 Flush
2845 012/300 Flush
2846 013/300 Read block 1860 len 24576
2847 014/300 Read block 3261 len 20480
2848 015/300 Write block 4986 len 20480 data: 2 2 2 2 2
2849 016/300 Read block 4216 len 24576
2850 017/300 Write block 3044 len 28672 data: 3 3 3 3 3 2 2
2851 018/300 Read block 947 len 36864
2852 019/300 Read block 3247 len 32768
2853 020/300 Read block 1719 len 12288
2854 021/300 Read block 1594 len 32768
2855 022/300 Write block 624 len 8192 data: 2 2
2856 023/300 Read block 1324 len 8192
2857 024/300 Write block 498 len 12288 data: 2 2 2
2858 025/300 Read block 2048 len 32768
2859 026/300 Read block 656 len 16384
2860 027/300 Write block 1001 len 28672 data: 2 2 2 2 2 2 2
2861 028/300 Read block 4809 len 20480
2862 029/300 Write block 3842 len 24576 data: 2 2 2 2 2 2
2863 030/300 Write block 2876 len 8192 data: 2 2
2864 031/300 Read block 3313 len 12288
2865 032/300 Read block 2376 len 12288
2866 033/300 Read block 3034 len 8192
2867 034/300 Flush
2868 035/300 Write block 340 len 32768 data: 2 2 2 2 2 2 2 2
2869 036/300 Write block 2578 len 32768 data: 3 3 2 2 2 2 2 2
2870 037/300 Write block 3956 len 28672 data: 3 4 4 4 4 4 4
2871 038/300 Write block 4729 len 20480 data: 2 2 2 3 3
2872 039/300 Read block 4355 len 24576
2873 040/300 Flush
2874 041/300 Flush
2875 042/300 Write block 4707 len 24576 data: 2 2 2 2 2 2
2876 043/300 Read block 1197 len 24576
2877 044/300 Read block 3784 len 16384
2878 045/300 Read block 3359 len 20480
2879 046/300 Read block 3353 len 16384
2880 047/300 Read block 2487 len 12288
2881 048/300 Flush
2882 049/300 Read block 1661 len 16384
2883 050/300 Read block 4226 len 16384
2884 051/300 Write block 2368 len 28672 data: 2 2 2 2 2 2 2
2885 052/300 Read block 3314 len 40960
2886 053/300 Read block 343 len 28672
2887 054/300 Write block 353 len 24576 data: 2 2 2 2 2 2
2888 055/300 Write block 1818 len 16384 data: 2 2 2 2
2889 056/300 Read block 3291 len 12288
2890 057/300 Read block 1252 len 40960
2891 058/300 Write block 4767 len 32768 data: 3 3 3 3 3 3 2 2
2892 059/300 Read block 1414 len 32768
2893 060/300 Read block 4963 len 24576
2894 061/300 Read block 1753 len 24576
2895 062/300 Write block 4826 len 20480 data: 2 2 2 2 2
2896 063/300 Read block 4926 len 40960
2897 064/300 Write block 1904 len 4096 data: 3
2898 065/300 Write block 3789 len 24576 data: 2 2 2 2 2 2
2899 066/300 Read block 3515 len 16384
2900 067/300 Read block 3959 len 28672
2901 068/300 Read block 937 len 24576
2902 069/300 Read block 3487 len 8192
2903 070/300 Flush
2904 071/300 Read block 2947 len 24576
2905 072/300 Read block 1188 len 12288
2906 073/300 Read block 1984 len 12288
2907 074/300 Read block 691 len 4096
2908 075/300 Flush
2909 076/300 Read block 4121 len 12288
2910 077/300 Write block 1298 len 12288 data: 2 2 2
2911 078/300 Read block 4317 len 20480
2912 079/300 Flush
2913 080/300 Write block 3421 len 20480 data: 2 2 2 2 2
2914 081/300 Write block 3254 len 24576 data: 2 2 2 2 2 2
2915 082/300 Read block 3587 len 8192
2916 083/300 Read block 2562 len 24576
2917 084/300 Read block 3560 len 28672
2918 085/300 Write block 2376 len 36864 data: 2 2 2 2 2 2 2 2 2
2919 086/300 Read block 978 len 20480
2920 087/300 Flush
2921 088/300 Read block 263 len 32768
2922 089/300 Read block 1937 len 16384
2923 090/300 Read block 3678 len 24576
2924 091/300 Write block 1803 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2925 092/300 Write block 2157 len 28672 data: 4 4 5 5 4 4 4
2926 093/300 Write block 536 len 4096 data: 3
2927 094/300 Read block 4182 len 28672
2928 095/300 Read block 3945 len 32768
2929 096/300 Write block 4115 len 24576 data: 2 2 2 2 2 2
2930 097/300 Write block 3524 len 20480 data: 3 3 3 3 3
2931 098/300 Write block 933 len 36864 data: 2 2 2 3 3 3 3 3 3
2932 099/300 Read block 4298 len 40960
2933 100/300 Read block 2380 len 8192
2934 101/300 Write block 323 len 32768 data: 2 2 2 2 2 2 2 2
2935 102/300 Write block 3388 len 8192 data: 2 2
2936 103/300 Read block 2629 len 28672
2937 104/300 Read block 3256 len 12288
2938 105/300 Read block 1589 len 4096
2939 106/300 Read block 193 len 36864
2940 107/300 Write block 4579 len 20480 data: 2 2 2 2 2
2941 108/300 Write block 4478 len 36864 data: 3 4 5 5 5 5 4 3 3
2942 109/300 Flush
2943 110/300 Write block 3599 len 4096 data: 3
2944 111/300 Read block 328 len 8192
2945 112/300 Write block 694 len 4096 data: 3
2946 113/300 Write block 3679 len 12288 data: 2 2 2
2947 114/300 Read block 2229 len 16384
2948 115/300 Write block 2679 len 40960 data: 4 5 4 3 2 2 2 2 2 2
2949 116/300 Read block 2849 len 4096
2950 117/300 Write block 3895 len 16384 data: 3 3 3 3
2951 118/300 Read block 4360 len 8192
2952 119/300 Read block 1045 len 28672
2953 120/300 Flush
2954 121/300 Read block 4375 len 8192
2955 122/300 Read block 3697 len 24576
2956 123/300 Flush
2957 124/300 Write block 2213 len 12288 data: 2 2 2
2958 125/300 Read block 526 len 32768
2959 126/300 Write block 2764 len 8192 data: 3 3
2960 127/300 Flush
2961 128/300 Write block 998 len 40960 data: 2 2 2 3 3 3 3 3 3 3
2962 129/300 Write block 3075 len 40960 data: 2 2 2 2 3 3 3 3 3 2
2963 130/300 Read block 3470 len 12288
2964 131/300 Read block 4954 len 28672
2965 132/300 Write block 941 len 8192 data: 4 3
2966 133/300 Read block 2026 len 32768
2967 134/300 Read block 281 len 16384
2968 135/300 Read block 1824 len 36864
2969 136/300 Read block 2177 len 8192
2970 137/300 Write block 914 len 24576 data: 3 2 2 2 2 2
2971 138/300 Flush
2972 139/300 Read block 3044 len 36864
2973 140/300 Read block 1953 len 36864
2974 141/300 Read block 4796 len 16384
2975 142/300 Flush
2976 143/300 Write block 1065 len 20480 data: 3 3 2 2 2
2977 144/300 Write block 2165 len 20480 data: 3 3 3 4 3
2978 145/300 Read block 4737 len 16384
2979 146/300 Read block 3311 len 24576
2980 147/300 Read block 1718 len 12288
2981 148/300 Read block 2335 len 16384
2982 149/300 Write block 2425 len 36864 data: 2 2 2 2 2 2 2 2 2
2983 150/300 Write block 3541 len 20480 data: 2 2 2 2 2
2984 151/300 Read block 4729 len 4096
2985 152/300 Write block 3663 len 28672 data: 2 2 2 2 2 4 3
2986 153/300 Read block 4784 len 32768
2987 154/300 Read block 2455 len 24576
2988 155/300 Write block 3489 len 8192 data: 2 2
2989 156/300 Write block 4653 len 16384 data: 2 2 2 2
2990 157/300 Write block 3486 len 40960 data: 2 2 2 3 3 2 2 2 2 2
2991 158/300 Flush
2992 159/300 Read block 2278 len 16384
2993 160/300 Read block 1112 len 32768
2994 161/300 Read block 2558 len 24576
2995 162/300 Write block 3916 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2996 163/300 Flush
2997 164/300 Read block 3517 len 12288
2998 165/300 Write block 1700 len 28672 data: 2 2 2 2 2 2 2
2999 166/300 Write block 1306 len 4096 data: 2
3000 167/300 Write block 976 len 28672 data: 2 2 2 3 3 3 3
3001 168/300 Write block 2869 len 32768 data: 2 2 2 2 2 2 2 3
3002 169/300 Read block 4502 len 20480
3003 170/300 Write block 4696 len 8192 data: 3 3
3004 171/300 Write block 810 len 28672 data: 2 2 3 3 3 3 3
3005 172/300 Flush
3006 173/300 Read block 3204 len 24576
3007 174/300 Read block 4854 len 32768
3008 175/300 Write block 536 len 28672 data: 4 3 4 4 5 5 4
3009 176/300 Write block 2847 len 28672 data: 3 3 3 3 2 2 2
3010 177/300 Read block 217 len 28672
3011 178/300 Flush
3012 179/300 Read block 4234 len 32768
3013 180/300 Flush
3014 181/300 Read block 3310 len 8192
3015 182/300 Write block 1506 len 4096 data: 3
3016 183/300 Write block 1230 len 12288 data: 2 2 2
3017 184/300 Read block 2750 len 16384
3018 185/300 Read block 1064 len 8192
3019 186/300 Read block 1508 len 40960
3020 187/300 Read block 4114 len 20480
3021 188/300 Flush
3022 189/300 Write block 4250 len 16384 data: 2 2 2 2
3023 190/300 Flush
3024 191/300 Read block 2518 len 24576
3025 192/300 Flush
3026 193/300 Read block 1311 len 12288
3027 194/300 Write block 1368 len 24576 data: 3 3 3 3 2 2
3028 195/300 Read block 2777 len 32768
3029 196/300 Read block 4096 len 28672
3030 197/300 Read block 567 len 24576
3031 198/300 Write block 4118 len 12288 data: 3 3 3
3032 199/300 Read block 3649 len 40960
3033 200/300 Read block 4731 len 20480
3034 201/300 Read block 3231 len 16384
3035 202/300 Read block 3347 len 8192
3036 203/300 Write block 547 len 28672 data: 3 3 3 2 2 2 2
3037 204/300 Read block 704 len 4096
3038 205/300 Read block 4941 len 40960
3039 206/300 Write block 2138 len 4096 data: 3
3040 207/300 Read block 439 len 32768
3041 208/300 Read block 4288 len 28672
3042 209/300 Write block 2548 len 32768 data: 2 2 2 2 2 2 2 3
3043 210/300 Write block 4228 len 24576 data: 3 3 3 3 3 2
3044 211/300 Write block 462 len 4096 data: 3
3045 212/300 Write block 1979 len 12288 data: 2 2 2
3046 213/300 Read block 1338 len 32768
3047 214/300 Read block 4708 len 12288
3048 215/300 Write block 4081 len 20480 data: 3 2 3 3 3
3049 216/300 Write block 1636 len 36864 data: 2 2 2 2 2 2 2 2 2
3050 217/300 Read block 4607 len 32768
3051 218/300 Read block 391 len 20480
3052 219/300 Read block 2128 len 4096
3053 220/300 Read block 3284 len 20480
3054 221/300 Read block 3969 len 16384
3055 222/300 Read block 2250 len 36864
3056 223/300 Read block 1331 len 28672
3057 224/300 Read block 4179 len 40960
3058 225/300 Read block 723 len 24576
3059 226/300 Write block 4524 len 4096 data: 2
3060 227/300 Read block 3674 len 8192
3061 228/300 Write block 3769 len 36864 data: 3 3 3 2 2 2 2 2 3
3062 229/300 Write block 4306 len 32768 data: 2 2 2 2 2 2 2 2
3063 230/300 Write block 4371 len 28672 data: 2 3 3 3 3 3 3
3064 231/300 Write block 3396 len 8192 data: 3 3
3065 232/300 Read block 1500 len 32768
3066 233/300 Write block 3800 len 8192 data: 2 2
3067 234/300 Read block 580 len 12288
3068 235/300 Read block 1197 len 8192
3069 236/300 Read block 3031 len 16384
3070 237/300 Write block 491 len 24576 data: 2 2 2 2 2 2
3071 238/300 Read block 4308 len 28672
3072 239/300 Write block 1659 len 24576 data: 2 2 2 2 2 2
3073 240/300 Write block 2474 len 16384 data: 3 4 4 4
3074 241/300 Write block 443 len 4096 data: 2
3075 242/300 Write block 2909 len 28672 data: 2 2 2 2 2 3 3
3076 243/300 Read block 4727 len 16384
3077 244/300 Write block 91 len 28672 data: 2 2 2 2 2 2 2
3078 245/300 Flush
3079 246/300 Read block 4001 len 8192
3080 247/300 Read block 2167 len 20480
3081 248/300 Write block 1059 len 36864 data: 4 4 3 3 3 3 4 4 3
3082 249/300 Write block 4 len 16384 data: 2 2 2 2
3083 250/300 Read block 3783 len 16384
3084 251/300 Read block 3949 len 40960
3085 252/300 Write block 3730 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3086 253/300 Read block 4724 len 24576
3087 254/300 Flush
3088 255/300 Write block 2459 len 40960 data: 3 3 3 2 2 2 2 2 2 2
3089 256/300 Write block 4522 len 12288 data: 2 2 3
3090 257/300 Write block 2985 len 8192 data: 2 2
3091 258/300 Read block 2282 len 28672
3092 259/300 Read block 3759 len 40960
3093 260/300 Write block 2366 len 24576 data: 2 2 3 3 3 3
3094 261/300 Flush
3095 262/300 Write block 4401 len 4096 data: 2
3096 263/300 Read block 2372 len 40960
3097 264/300 Write block 4417 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3098 265/300 Write block 4545 len 8192 data: 2 2
3099 266/300 Read block 3821 len 20480
3100 267/300 Read block 342 len 20480
3101 268/300 Write block 4659 len 32768 data: 3 3 3 3 3 3 3 3
3102 269/300 Write block 2126 len 40960 data: 2 2 2 2 2 2 3 3 3 4
3103 270/300 Write block 2394 len 20480 data: 3 3 3 3 3
3104 271/300 Write block 1655 len 16384 data: 2 2 2 2
3105 272/300 Read block 3640 len 20480
3106 273/300 Read block 2941 len 28672
3107 274/300 Flush
3108 275/300 Read block 4185 len 36864
3109 276/300 Write block 1932 len 24576 data: 2 2 2 3 3 2
3110 277/300 Read block 3837 len 32768
3111 278/300 Read block 169 len 4096
3112 279/300 Write block 4244 len 12288 data: 2 2 2
3113 280/300 Read block 2677 len 16384
3114 281/300 Write block 2798 len 40960 data: 3 2 2 2 2 2 2 2 2 2
3115 282/300 Read block 3269 len 20480
3116 283/300 Write block 2991 len 32768 data: 3 3 3 3 3 2 2 2
3117 284/300 Write block 2009 len 20480 data: 2 2 2 2 2
3118 285/300 Read block 709 len 16384
3119 286/300 Flush
3120 287/300 Write block 4381 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3121 288/300 Write block 2427 len 40960 data: 3 3 3 3 3 3 3 2 2 2
3122 289/300 Write block 2593 len 12288 data: 3 2 2
3123 290/300 Read block 1905 len 36864
3124 291/300 Read block 2980 len 40960
3125 292/300 Write block 954 len 8192 data: 2 2
3126 293/300 Flush
3127 294/300 Read block 1329 len 4096
3128 295/300 Read block 2278 len 40960
3129 296/300 Write block 2133 len 32768 data: 4 4 5 3 3 4 3 3
3130 297/300 Write block 3756 len 24576 data: 2 2 2 2 3 3
3131 298/300 Read block 3324 len 20480
3132 299/300 Write block 3249 len 12288 data: 3 2 2
3133 300/300 Read block 411 len 36864
3134 301/300 Write block 4841 len 8192 data: 3 4
3135 Main task has work to do, go find it
3136 got dsc Start(2)
3137 start 2
3138 Jan 28 21:55:02.615 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/2, method: POST, req_id: e88ccb25-1fdc-4db1-8110-3712ff5cd9fc, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
3139 Replay: started 2, returned:Ok(())
3140 ----------------------------------------------------------------
3141 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3142 States: Active Active Offline
3143 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3144 1604 Acked Write 2 Done Done Sent false
3145 1605 Acked Write 2 Done Done Sent false
3146 1606 Acked Write 10 Done Done Sent false
3147 1607 Acked Write 9 Done Done Sent false
3148 1608 Acked Read 5 Done Done Sent false
3149 1609 Acked Read 5 Done Done Sent false
3150 1610 Acked Read 2 Done Done Sent false
3151 1611 Acked Read 4 Done Done Sent false
3152 1612 Acked Read 5 Done Done Sent false
3153 1613 Acked Write 5 Done Done Sent false
3154 1614 Acked Flush 0 Done Done Sent false
3155 1615 Acked Flush 0 Done Done Sent false
3156 1616 Acked Read 6 Done Done Sent false
3157 1617 Acked Read 5 Done Done Sent false
3158 1618 Acked Write 5 Done Done Sent false
3159 1619 Acked Read 6 Done Done Sent false
3160 1620 Acked Write 7 Done Done Sent false
3161 1621 Acked Read 9 Done Done Sent false
3162 1622 Acked Read 8 Done Done Sent false
3163 1623 Acked Read 3 Done Done Sent false
3164 1624 Acked Read 8 Done Done Sent false
3165 1625 Acked Write 2 Done Done Sent false
3166 1626 Acked Read 2 Done Done Sent false
3167 1627 Acked Write 3 Done Done Sent false
3168 1628 Acked Read 8 Done Done Sent false
3169 1629 Acked Read 4 Done Done Sent false
3170 1630 Acked Write 7 Done Done Sent false
3171 1631 Acked Read 5 Done Done Sent false
3172 1632 Acked Write 6 Done Done Sent false
3173 1633 Acked Write 2 Done Done Sent false
3174 1634 Acked Read 3 Done Done Sent false
3175 1635 Acked Read 3 Done Done Sent false
3176 1636 Acked Read 2 Done Done Sent false
3177 1637 Acked Flush 0 Done Done Sent false
3178 1638 Acked Write 8 Done Done Sent false
3179 1639 Acked Write 8 Done Done Sent false
3180 1640 Acked Write 7 Done Done Sent false
3181 1641 Acked Write 5 Done Done Sent false
3182 1642 Acked Read 6 Done Done Sent false
3183 1643 Acked Flush 0 Done Done Sent false
3184 1644 Acked Flush 0 Done Done Sent false
3185 1645 Acked Write 6 Done Done Sent false
3186 1646 Acked Read 6 Done Done Sent false
3187 1647 Acked Read 4 Done Done Sent false
3188 1648 Acked Read 5 Done Done Sent false
3189 1649 Acked Read 4 Done Done Sent false
3190 1650 Acked Read 3 Done Done Sent false
3191 1651 Acked Flush 0 Done Done Sent false
3192 1652 Acked Read 4 Done Done Sent false
3193 1653 Acked Read 4 Done Done Sent false
3194 1654 Acked Write 7 Done Done Sent false
3195 1655 Acked Read 10 Done Done Sent false
3196 1656 Acked Read 7 Done Done Sent false
3197 1657 Acked Write 6 Done Done Sent false
3198 1658 Acked Write 4 Done Done Sent false
3199 1659 Acked Read 3 Done Done Sent false
3200 1660 Acked Read 10 Done Done Sent false
3201 1661 Acked Write 8 Done Done Sent false
3202 1662 Acked Read 8 Done Done Sent false
3203 1663 Acked Read 6 Done Done Sent false
3204 1664 Acked Read 6 Done Done Sent false
3205 1665 Acked Write 5 Done Done Sent false
3206 1666 Acked Read 10 Done Done Sent false
3207 1667 Acked Write 1 Done Done Sent false
3208 1668 Acked Write 6 Done Done Sent false
3209 1669 Acked Read 4 Done Done Sent false
3210 1670 Acked Read 7 Done Done Sent false
3211 1671 Acked Read 6 Done Done Sent false
3212 1672 Acked Read 2 Done Done Sent false
3213 1673 Acked Flush 0 Done Done Sent false
3214 1674 Acked Read 6 Done Done Sent false
3215 1675 Acked Read 3 Done Done Sent false
3216 1676 Acked Read 3 Done Done Sent false
3217 1677 Acked Read 1 Done Done Sent false
3218 1678 Acked Flush 0 Done Done Sent false
3219 1679 Acked Read 3 Done Done Sent false
3220 1680 Acked Write 3 Done Done Sent false
3221 1681 Acked Read 5 Done Done Sent false
3222 1682 Acked Flush 0 Done Done Sent false
3223 1683 Acked Write 5 Done Done Sent false
3224 1684 Acked Write 6 Done Done Sent false
3225 1685 Acked Read 2 Done Done Sent false
3226 1686 Acked Read 6 Done Done Sent false
3227 1687 Acked Read 7 Done Done Sent false
3228 1688 Acked Write 9 Done Done Sent false
3229 1689 Acked Read 5 Done Done Sent false
3230 1690 Acked Flush 0 Done Done Sent false
3231 1691 Acked Read 8 Done Done Sent false
3232 1692 Acked Read 4 Done Done Sent false
3233 1693 Acked Read 6 Done Done Sent false
3234 1694 Acked Write 10 Done Done Sent false
3235 1695 Acked Write 7 Done Done Sent false
3236 1696 Acked Write 1 Done Done Sent false
3237 1697 Acked Read 7 Done Done Sent false
3238 1698 Acked Read 8 Done Done Sent false
3239 1699 Acked Write 6 Done Done Sent false
3240 1700 Acked Write 5 Done Done Sent false
3241 1701 Acked Write 9 Done Done Sent false
3242 1702 Acked Read 10 Done Done Sent false
3243 1703 Acked Read 2 Done Done Sent false
3244 1704 Acked Write 8 Done Done Sent false
3245 1705 Acked Write 2 Done Done Sent false
3246 1706 Acked Read 7 Done Done Sent false
3247 1707 Acked Read 3 Done Done Sent false
3248 1708 Acked Read 1 Done Done Sent false
3249 1709 Acked Read 9 Done Done Sent false
3250 1710 Acked Write 5 Done Done Sent false
3251 1711 Acked Write 9 Done Done Sent false
3252 1712 Acked Flush 0 Done Done Sent false
3253 1713 Acked Write 1 Done Done Sent false
3254 1714 Acked Read 2 Done Done Sent false
3255 1715 Acked Write 1 Done Done Sent false
3256 1716 Acked Write 3 Done Done Sent false
3257 1717 Acked Read 4 Done Done Sent false
3258 1718 Acked Write 10 Done Done Sent false
3259 1719 Acked Read 1 Done Done Sent false
3260 1720 Acked Write 4 Done Done Sent false
3261 1721 Acked Read 2 Done Done Sent false
3262 1722 Acked Read 7 Done Done Sent false
3263 1723 Acked Flush 0 Done Done Sent false
3264 1724 Acked Read 2 Done Done Sent false
3265 1725 Acked Read 6 Done Done Sent false
3266 1726 Acked Flush 0 Done Done Sent false
3267 1727 Acked Write 3 Done Done Sent false
3268 1728 Acked Read 8 Done Done Sent false
3269 1729 Acked Write 2 Done Done Sent false
3270 1730 Acked Flush 0 Done Done Sent false
3271 1731 Acked Write 10 Done Done Sent false
3272 1732 Acked Write 10 Done Done Sent false
3273 1733 Acked Read 3 Done Done Sent false
3274 1734 Acked Read 7 Done Done Sent false
3275 1735 Acked Write 2 Done Done Sent false
3276 1736 Acked Read 8 Done Done Sent false
3277 1737 Acked Read 4 Done Done Sent false
3278 1738 Acked Read 9 Done Done Sent false
3279 1739 Acked Read 2 Done Done Sent false
3280 1740 Acked Write 6 Done Done Sent false
3281 1741 Acked Flush 0 Done Done Sent false
3282 1742 Acked Read 9 Done Done Sent false
3283 1743 Acked Read 9 Done Done Sent false
3284 1744 Acked Read 4 Done Done Sent false
3285 1745 Acked Flush 0 Done Done Sent false
3286 1746 Acked Write 5 Done Done Sent false
3287 1747 Acked Write 5 Done Done Sent false
3288 1748 Acked Read 4 Done Done Sent false
3289 1749 Acked Read 6 Done Done Sent false
3290 1750 Acked Read 3 Done Done Sent false
3291 1751 Acked Read 4 Done Done Sent false
3292 1752 Acked Write 9 Done Done Sent false
3293 1753 Acked Write 5 Done Done Sent false
3294 1754 Acked Read 1 Done Done Sent false
3295 1755 Acked Write 7 Done Done Sent false
3296 1756 Acked Read 8 Done Done Sent false
3297 1757 Acked Read 6 Done Done Sent false
3298 1758 Acked Write 2 Done Done Sent false
3299 1759 Acked Write 4 Done Done Sent false
3300 1760 Acked Write 10 Done Done Sent false
3301 1761 Acked Flush 0 Done Done Sent false
3302 1762 Acked Read 4 Done Done Sent false
3303 1763 Acked Read 8 Done Done Sent false
3304 1764 Acked Read 6 Done Done Sent false
3305 1765 Acked Write 10 Done Done Sent false
3306 1766 Acked Flush 0 Done Done Sent false
3307 1767 Acked Read 3 Done Done Sent false
3308 1768 Acked Write 7 Done Done Sent false
3309 1769 Acked Write 1 Done Done Sent false
3310 1770 Acked Write 7 Done Done Sent false
3311 1771 Acked Write 8 Done Done Sent false
3312 1772 Acked Read 5 Done Done Sent false
3313 1773 Acked Write 2 Done Done Sent false
3314 1774 Acked Write 7 Done Done Sent false
3315 1775 Acked Flush 0 Done Done Sent false
3316 1776 Acked Read 6 Done Done Sent false
3317 1777 Acked Read 8 Done Done Sent false
3318 1778 Acked Write 7 Done Done Sent false
3319 1779 Acked Write 7 Done Done Sent false
3320 1780 Acked Read 7 Done Done Sent false
3321 1781 Acked Flush 0 Done Done Sent false
3322 1782 Acked Read 8 Done Done Sent false
3323 1783 Acked Flush 0 Done Done Sent false
3324 1784 Acked Read 2 Done Done Sent false
3325 1785 Acked Write 1 Done Done Sent false
3326 1786 Acked Write 3 Done Done Sent false
3327 1787 Acked Read 4 Done Done Sent false
3328 1788 Acked Read 2 Done Done Sent false
3329 1789 Acked Read 10 Done Done Sent false
3330 1790 Acked Read 5 Done Done Sent false
3331 1791 Acked Flush 0 Done Done Sent false
3332 1792 Acked Write 4 Done Done Sent false
3333 1793 Acked Flush 0 Done Done Sent false
3334 1794 Acked Read 6 Done Done Sent false
3335 1795 Acked Flush 0 Done Done Sent false
3336 1796 Acked Read 3 Done Done Sent false
3337 1797 Acked Write 6 Done Done Sent false
3338 1798 Acked Read 8 Done Done Sent false
3339 1799 Acked Read 7 Done Done Sent false
3340 1800 Acked Read 6 Done Done Sent false
3341 1801 Acked Write 3 Done Done Sent false
3342 1802 Acked Read 10 Done Done Sent false
3343 1803 Acked Read 5 Done Done Sent false
3344 1804 Acked Read 4 Done Done Sent false
3345 1805 Acked Read 2 Done Done Sent false
3346 1806 Acked Write 7 Done Done Sent false
3347 1807 Acked Read 1 Done Done Sent false
3348 1808 Acked Read 10 Done Done Sent false
3349 1809 Acked Write 1 Done Done Sent false
3350 1810 Acked Read 8 Done Done Sent false
3351 1811 Acked Read 7 Done Done Sent false
3352 1812 Acked Write 8 Done Done Sent false
3353 1813 Acked Write 6 Done Done Sent false
3354 1814 Acked Write 1 Done Done Sent false
3355 1815 Acked Write 3 Done Done Sent false
3356 1816 Acked Read 8 Done Done Sent false
3357 1817 Acked Read 3 Done Done Sent false
3358 1818 Acked Write 5 Done Done Sent false
3359 1819 Acked Write 9 Done Done Sent false
3360 1820 Acked Read 8 Done Done Sent false
3361 1821 Acked Read 5 Done Done Sent false
3362 1822 Acked Read 1 Done Done Sent false
3363 1823 Acked Read 5 Done Done Sent false
3364 1824 Acked Read 4 Done Done Sent false
3365 1825 Acked Read 9 Done Done Sent false
3366 1826 Acked Read 7 Done Done Sent false
3367 1827 Acked Read 10 Done Done Sent false
3368 1828 Acked Read 6 Done Done Sent false
3369 1829 Acked Write 1 Done Done Sent false
3370 1830 Acked Read 2 Done Done Sent false
3371 1831 Acked Write 9 Done Done Sent false
3372 1832 Acked Write 8 Done Done Sent false
3373 1833 Acked Write 7 Done Done Sent false
3374 1834 Acked Write 2 Done Done Sent false
3375 1835 Acked Read 8 Done Done Sent false
3376 1836 Acked Write 2 Done Done Sent false
3377 1837 Acked Read 3 Done Done Sent false
3378 1838 Acked Read 2 Done Done Sent false
3379 1839 Acked Read 4 Done Done Sent false
3380 1840 Acked Write 6 Done Done Sent false
3381 1841 Acked Read 7 Done Done Sent false
3382 1842 Acked Write 6 Done Done Sent false
3383 1843 Acked Write 4 Done Done Sent false
3384 1844 Acked Write 1 Done Done Sent false
3385 1845 Acked Write 7 Done Done Sent false
3386 1846 Acked Read 4 Done Done Sent false
3387 1847 Acked Write 7 Done Done Sent false
3388 1848 Acked Flush 0 Done Done Sent false
3389 1849 Acked Read 2 Done Done Sent false
3390 1850 Acked Read 5 Done Done Sent false
3391 1851 Acked Write 9 Done Done Sent false
3392 1852 Acked Write 4 Done Done Sent false
3393 1853 Acked Read 4 Done Done Sent false
3394 1854 Acked Read 10 Done Done Sent false
3395 1855 Acked Write 10 Done Done Sent false
3396 1856 Acked Read 6 Done Done Sent false
3397 1857 Acked Flush 0 Done Done Sent false
3398 1858 Acked Write 10 Done Done Sent false
3399 1859 Acked Write 3 Done Done Sent false
3400 1860 Acked Write 2 Done Done Sent false
3401 1861 Acked Read 7 Done Done Sent false
3402 1862 Acked Read 10 Done Done Sent false
3403 1863 Acked Write 6 Done Done Sent false
3404 1864 Acked Flush 0 Done Done Sent false
3405 1865 Acked Write 1 Done Done Sent false
3406 1866 Acked Read 10 Done Done Sent false
3407 1867 Acked Write 10 Done Done Sent false
3408 1868 Acked Write 2 Done Done Sent false
3409 1869 Acked Read 5 Done Done Sent false
3410 1870 Acked Read 5 Done Done Sent false
3411 1871 Acked Write 8 Done Done Sent false
3412 1872 Acked Write 10 Done Done Sent false
3413 1873 Acked Write 5 Done Done Sent false
3414 1874 Acked Write 4 Done Done Sent false
3415 1875 Acked Read 5 Done Done Sent false
3416 1876 Acked Read 7 Done Done Sent false
3417 1877 Acked Flush 0 Done Done Sent false
3418 1878 Acked Read 9 Done Done Sent false
3419 1879 Acked Write 6 Done Done Sent false
3420 1880 Acked Read 8 Done Done Sent false
3421 1881 Acked Read 1 Done Done Sent false
3422 1882 Acked Write 3 Done Done Sent false
3423 1883 Acked Read 4 Done Done Sent false
3424 1884 Acked Write 10 Done Done Sent false
3425 1885 Acked Read 5 Done Done Sent false
3426 1886 Acked Write 8 Done Done Sent false
3427 1887 Acked Write 5 Done Done Sent false
3428 1888 Acked Read 4 Done Done Sent false
3429 1889 Acked Flush 0 Done Done Sent false
3430 1890 Acked Write 10 Done Done Sent false
3431 1891 Acked Write 10 Done Done Sent false
3432 1892 Acked Write 3 Done Done Sent false
3433 1893 Acked Read 9 Done Done Sent false
3434 1894 Acked Read 10 Done Done Sent false
3435 1895 Acked Write 2 Done Done Sent false
3436 1896 Acked Flush 0 Done Done Sent false
3437 1897 Acked Read 1 Done Done Sent false
3438 1898 Acked Read 10 Done Done Sent false
3439 1899 Acked Write 8 Done Done Sent false
3440 1900 Acked Write 6 Done Done Sent false
3441 1901 Acked Read 5 Done Done Sent false
3442 1902 Acked Write 3 Done Done Sent false
3443 1903 Acked Read 9 Done Done Sent false
3444 1904 Acked Write 2 Done Done Sent false
3445 STATES DS:0 DS:1 DS:2 TOTAL
3446 Sent 0 0 301 301
3447 Done 301 301 0 602
3448 Skipped 0 0 0 0
3449 Error 0 0 0 0
3450 Last Flush: Some(JobId(1896)) Some(JobId(1896)) Some(JobId(1603))
3451 Downstairs last five completed: 1603 1602 1601 1600 1599
3452 Upstairs last five completed: 1904 1903 1902 1901 1900
3453 CLIENT: Up:0 ds:301 act:2
34542025-01-28T21:55:03.393ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
3455 [8830] got start action
3456 [8830] I am going to restart
3457 Starting downstairs at port 8830
3458 Make output file at "/tmp/dsc/downstairs-8830.txt"
3459 [8830][2] reports Starting
3460 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1205)
3461 [8830][2] reports Running
34622025-01-28T21:55:04.395ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
34632025-01-28T21:55:05.395ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
34642025-01-28T21:55:06.396ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
3465 ----------------------------------------------------------------
3466 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3467 States: Active Active Offline
3468 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3469 1604 Acked Write 2 Done Done Sent false
3470 1605 Acked Write 2 Done Done Sent false
3471 1606 Acked Write 10 Done Done Sent false
3472 1607 Acked Write 9 Done Done Sent false
3473 1608 Acked Read 5 Done Done Sent false
3474 1609 Acked Read 5 Done Done Sent false
3475 1610 Acked Read 2 Done Done Sent false
3476 1611 Acked Read 4 Done Done Sent false
3477 1612 Acked Read 5 Done Done Sent false
3478 1613 Acked Write 5 Done Done Sent false
3479 1614 Acked Flush 0 Done Done Sent false
3480 1615 Acked Flush 0 Done Done Sent false
3481 1616 Acked Read 6 Done Done Sent false
3482 1617 Acked Read 5 Done Done Sent false
3483 1618 Acked Write 5 Done Done Sent false
3484 1619 Acked Read 6 Done Done Sent false
3485 1620 Acked Write 7 Done Done Sent false
3486 1621 Acked Read 9 Done Done Sent false
3487 1622 Acked Read 8 Done Done Sent false
3488 1623 Acked Read 3 Done Done Sent false
3489 1624 Acked Read 8 Done Done Sent false
3490 1625 Acked Write 2 Done Done Sent false
3491 1626 Acked Read 2 Done Done Sent false
3492 1627 Acked Write 3 Done Done Sent false
3493 1628 Acked Read 8 Done Done Sent false
3494 1629 Acked Read 4 Done Done Sent false
3495 1630 Acked Write 7 Done Done Sent false
3496 1631 Acked Read 5 Done Done Sent false
3497 1632 Acked Write 6 Done Done Sent false
3498 1633 Acked Write 2 Done Done Sent false
3499 1634 Acked Read 3 Done Done Sent false
3500 1635 Acked Read 3 Done Done Sent false
3501 1636 Acked Read 2 Done Done Sent false
3502 1637 Acked Flush 0 Done Done Sent false
3503 1638 Acked Write 8 Done Done Sent false
3504 1639 Acked Write 8 Done Done Sent false
3505 1640 Acked Write 7 Done Done Sent false
3506 1641 Acked Write 5 Done Done Sent false
3507 1642 Acked Read 6 Done Done Sent false
3508 1643 Acked Flush 0 Done Done Sent false
3509 1644 Acked Flush 0 Done Done Sent false
3510 1645 Acked Write 6 Done Done Sent false
3511 1646 Acked Read 6 Done Done Sent false
3512 1647 Acked Read 4 Done Done Sent false
3513 1648 Acked Read 5 Done Done Sent false
3514 1649 Acked Read 4 Done Done Sent false
3515 1650 Acked Read 3 Done Done Sent false
3516 1651 Acked Flush 0 Done Done Sent false
3517 1652 Acked Read 4 Done Done Sent false
3518 1653 Acked Read 4 Done Done Sent false
3519 1654 Acked Write 7 Done Done Sent false
3520 1655 Acked Read 10 Done Done Sent false
3521 1656 Acked Read 7 Done Done Sent false
3522 1657 Acked Write 6 Done Done Sent false
3523 1658 Acked Write 4 Done Done Sent false
3524 1659 Acked Read 3 Done Done Sent false
3525 1660 Acked Read 10 Done Done Sent false
3526 1661 Acked Write 8 Done Done Sent false
3527 1662 Acked Read 8 Done Done Sent false
3528 1663 Acked Read 6 Done Done Sent false
3529 1664 Acked Read 6 Done Done Sent false
3530 1665 Acked Write 5 Done Done Sent false
3531 1666 Acked Read 10 Done Done Sent false
3532 1667 Acked Write 1 Done Done Sent false
3533 1668 Acked Write 6 Done Done Sent false
3534 1669 Acked Read 4 Done Done Sent false
3535 1670 Acked Read 7 Done Done Sent false
3536 1671 Acked Read 6 Done Done Sent false
3537 1672 Acked Read 2 Done Done Sent false
3538 1673 Acked Flush 0 Done Done Sent false
3539 1674 Acked Read 6 Done Done Sent false
3540 1675 Acked Read 3 Done Done Sent false
3541 1676 Acked Read 3 Done Done Sent false
3542 1677 Acked Read 1 Done Done Sent false
3543 1678 Acked Flush 0 Done Done Sent false
3544 1679 Acked Read 3 Done Done Sent false
3545 1680 Acked Write 3 Done Done Sent false
3546 1681 Acked Read 5 Done Done Sent false
3547 1682 Acked Flush 0 Done Done Sent false
3548 1683 Acked Write 5 Done Done Sent false
3549 1684 Acked Write 6 Done Done Sent false
3550 1685 Acked Read 2 Done Done Sent false
3551 1686 Acked Read 6 Done Done Sent false
3552 1687 Acked Read 7 Done Done Sent false
3553 1688 Acked Write 9 Done Done Sent false
3554 1689 Acked Read 5 Done Done Sent false
3555 1690 Acked Flush 0 Done Done Sent false
3556 1691 Acked Read 8 Done Done Sent false
3557 1692 Acked Read 4 Done Done Sent false
3558 1693 Acked Read 6 Done Done Sent false
3559 1694 Acked Write 10 Done Done Sent false
3560 1695 Acked Write 7 Done Done Sent false
3561 1696 Acked Write 1 Done Done Sent false
3562 1697 Acked Read 7 Done Done Sent false
3563 1698 Acked Read 8 Done Done Sent false
3564 1699 Acked Write 6 Done Done Sent false
3565 1700 Acked Write 5 Done Done Sent false
3566 1701 Acked Write 9 Done Done Sent false
3567 1702 Acked Read 10 Done Done Sent false
3568 1703 Acked Read 2 Done Done Sent false
3569 1704 Acked Write 8 Done Done Sent false
3570 1705 Acked Write 2 Done Done Sent false
3571 1706 Acked Read 7 Done Done Sent false
3572 1707 Acked Read 3 Done Done Sent false
3573 1708 Acked Read 1 Done Done Sent false
3574 1709 Acked Read 9 Done Done Sent false
3575 1710 Acked Write 5 Done Done Sent false
3576 1711 Acked Write 9 Done Done Sent false
3577 1712 Acked Flush 0 Done Done Sent false
3578 1713 Acked Write 1 Done Done Sent false
3579 1714 Acked Read 2 Done Done Sent false
3580 1715 Acked Write 1 Done Done Sent false
3581 1716 Acked Write 3 Done Done Sent false
3582 1717 Acked Read 4 Done Done Sent false
3583 1718 Acked Write 10 Done Done Sent false
3584 1719 Acked Read 1 Done Done Sent false
3585 1720 Acked Write 4 Done Done Sent false
3586 1721 Acked Read 2 Done Done Sent false
3587 1722 Acked Read 7 Done Done Sent false
3588 1723 Acked Flush 0 Done Done Sent false
3589 1724 Acked Read 2 Done Done Sent false
3590 1725 Acked Read 6 Done Done Sent false
3591 1726 Acked Flush 0 Done Done Sent false
3592 1727 Acked Write 3 Done Done Sent false
3593 1728 Acked Read 8 Done Done Sent false
3594 1729 Acked Write 2 Done Done Sent false
3595 1730 Acked Flush 0 Done Done Sent false
3596 1731 Acked Write 10 Done Done Sent false
3597 1732 Acked Write 10 Done Done Sent false
3598 1733 Acked Read 3 Done Done Sent false
3599 1734 Acked Read 7 Done Done Sent false
3600 1735 Acked Write 2 Done Done Sent false
3601 1736 Acked Read 8 Done Done Sent false
3602 1737 Acked Read 4 Done Done Sent false
3603 1738 Acked Read 9 Done Done Sent false
3604 1739 Acked Read 2 Done Done Sent false
3605 1740 Acked Write 6 Done Done Sent false
3606 1741 Acked Flush 0 Done Done Sent false
3607 1742 Acked Read 9 Done Done Sent false
3608 1743 Acked Read 9 Done Done Sent false
3609 1744 Acked Read 4 Done Done Sent false
3610 1745 Acked Flush 0 Done Done Sent false
3611 1746 Acked Write 5 Done Done Sent false
3612 1747 Acked Write 5 Done Done Sent false
3613 1748 Acked Read 4 Done Done Sent false
3614 1749 Acked Read 6 Done Done Sent false
3615 1750 Acked Read 3 Done Done Sent false
3616 1751 Acked Read 4 Done Done Sent false
3617 1752 Acked Write 9 Done Done Sent false
3618 1753 Acked Write 5 Done Done Sent false
3619 1754 Acked Read 1 Done Done Sent false
3620 1755 Acked Write 7 Done Done Sent false
3621 1756 Acked Read 8 Done Done Sent false
3622 1757 Acked Read 6 Done Done Sent false
3623 1758 Acked Write 2 Done Done Sent false
3624 1759 Acked Write 4 Done Done Sent false
3625 1760 Acked Write 10 Done Done Sent false
3626 1761 Acked Flush 0 Done Done Sent false
3627 1762 Acked Read 4 Done Done Sent false
3628 1763 Acked Read 8 Done Done Sent false
3629 1764 Acked Read 6 Done Done Sent false
3630 1765 Acked Write 10 Done Done Sent false
3631 1766 Acked Flush 0 Done Done Sent false
3632 1767 Acked Read 3 Done Done Sent false
3633 1768 Acked Write 7 Done Done Sent false
3634 1769 Acked Write 1 Done Done Sent false
3635 1770 Acked Write 7 Done Done Sent false
3636 1771 Acked Write 8 Done Done Sent false
3637 1772 Acked Read 5 Done Done Sent false
3638 1773 Acked Write 2 Done Done Sent false
3639 1774 Acked Write 7 Done Done Sent false
3640 1775 Acked Flush 0 Done Done Sent false
3641 1776 Acked Read 6 Done Done Sent false
3642 1777 Acked Read 8 Done Done Sent false
3643 1778 Acked Write 7 Done Done Sent false
3644 1779 Acked Write 7 Done Done Sent false
3645 1780 Acked Read 7 Done Done Sent false
3646 1781 Acked Flush 0 Done Done Sent false
3647 1782 Acked Read 8 Done Done Sent false
3648 1783 Acked Flush 0 Done Done Sent false
3649 1784 Acked Read 2 Done Done Sent false
3650 1785 Acked Write 1 Done Done Sent false
3651 1786 Acked Write 3 Done Done Sent false
3652 1787 Acked Read 4 Done Done Sent false
3653 1788 Acked Read 2 Done Done Sent false
3654 1789 Acked Read 10 Done Done Sent false
3655 1790 Acked Read 5 Done Done Sent false
3656 1791 Acked Flush 0 Done Done Sent false
3657 1792 Acked Write 4 Done Done Sent false
3658 1793 Acked Flush 0 Done Done Sent false
3659 1794 Acked Read 6 Done Done Sent false
3660 1795 Acked Flush 0 Done Done Sent false
3661 1796 Acked Read 3 Done Done Sent false
3662 1797 Acked Write 6 Done Done Sent false
3663 1798 Acked Read 8 Done Done Sent false
3664 1799 Acked Read 7 Done Done Sent false
3665 1800 Acked Read 6 Done Done Sent false
3666 1801 Acked Write 3 Done Done Sent false
3667 1802 Acked Read 10 Done Done Sent false
3668 1803 Acked Read 5 Done Done Sent false
3669 1804 Acked Read 4 Done Done Sent false
3670 1805 Acked Read 2 Done Done Sent false
3671 1806 Acked Write 7 Done Done Sent false
3672 1807 Acked Read 1 Done Done Sent false
3673 1808 Acked Read 10 Done Done Sent false
3674 1809 Acked Write 1 Done Done Sent false
3675 1810 Acked Read 8 Done Done Sent false
3676 1811 Acked Read 7 Done Done Sent false
3677 1812 Acked Write 8 Done Done Sent false
3678 1813 Acked Write 6 Done Done Sent false
3679 1814 Acked Write 1 Done Done Sent false
3680 1815 Acked Write 3 Done Done Sent false
3681 1816 Acked Read 8 Done Done Sent false
3682 1817 Acked Read 3 Done Done Sent false
3683 1818 Acked Write 5 Done Done Sent false
3684 1819 Acked Write 9 Done Done Sent false
3685 1820 Acked Read 8 Done Done Sent false
3686 1821 Acked Read 5 Done Done Sent false
3687 1822 Acked Read 1 Done Done Sent false
3688 1823 Acked Read 5 Done Done Sent false
3689 1824 Acked Read 4 Done Done Sent false
3690 1825 Acked Read 9 Done Done Sent false
3691 1826 Acked Read 7 Done Done Sent false
3692 1827 Acked Read 10 Done Done Sent false
3693 1828 Acked Read 6 Done Done Sent false
3694 1829 Acked Write 1 Done Done Sent false
3695 1830 Acked Read 2 Done Done Sent false
3696 1831 Acked Write 9 Done Done Sent false
3697 1832 Acked Write 8 Done Done Sent false
3698 1833 Acked Write 7 Done Done Sent false
3699 1834 Acked Write 2 Done Done Sent false
3700 1835 Acked Read 8 Done Done Sent false
3701 1836 Acked Write 2 Done Done Sent false
3702 1837 Acked Read 3 Done Done Sent false
3703 1838 Acked Read 2 Done Done Sent false
3704 1839 Acked Read 4 Done Done Sent false
3705 1840 Acked Write 6 Done Done Sent false
3706 1841 Acked Read 7 Done Done Sent false
3707 1842 Acked Write 6 Done Done Sent false
3708 1843 Acked Write 4 Done Done Sent false
3709 1844 Acked Write 1 Done Done Sent false
3710 1845 Acked Write 7 Done Done Sent false
3711 1846 Acked Read 4 Done Done Sent false
3712 1847 Acked Write 7 Done Done Sent false
3713 1848 Acked Flush 0 Done Done Sent false
3714 1849 Acked Read 2 Done Done Sent false
3715 1850 Acked Read 5 Done Done Sent false
3716 1851 Acked Write 9 Done Done Sent false
3717 1852 Acked Write 4 Done Done Sent false
3718 1853 Acked Read 4 Done Done Sent false
3719 1854 Acked Read 10 Done Done Sent false
3720 1855 Acked Write 10 Done Done Sent false
3721 1856 Acked Read 6 Done Done Sent false
3722 1857 Acked Flush 0 Done Done Sent false
3723 1858 Acked Write 10 Done Done Sent false
3724 1859 Acked Write 3 Done Done Sent false
3725 1860 Acked Write 2 Done Done Sent false
3726 1861 Acked Read 7 Done Done Sent false
3727 1862 Acked Read 10 Done Done Sent false
3728 1863 Acked Write 6 Done Done Sent false
3729 1864 Acked Flush 0 Done Done Sent false
3730 1865 Acked Write 1 Done Done Sent false
3731 1866 Acked Read 10 Done Done Sent false
3732 1867 Acked Write 10 Done Done Sent false
3733 1868 Acked Write 2 Done Done Sent false
3734 1869 Acked Read 5 Done Done Sent false
3735 1870 Acked Read 5 Done Done Sent false
3736 1871 Acked Write 8 Done Done Sent false
3737 1872 Acked Write 10 Done Done Sent false
3738 1873 Acked Write 5 Done Done Sent false
3739 1874 Acked Write 4 Done Done Sent false
3740 1875 Acked Read 5 Done Done Sent false
3741 1876 Acked Read 7 Done Done Sent false
3742 1877 Acked Flush 0 Done Done Sent false
3743 1878 Acked Read 9 Done Done Sent false
3744 1879 Acked Write 6 Done Done Sent false
3745 1880 Acked Read 8 Done Done Sent false
3746 1881 Acked Read 1 Done Done Sent false
3747 1882 Acked Write 3 Done Done Sent false
3748 1883 Acked Read 4 Done Done Sent false
3749 1884 Acked Write 10 Done Done Sent false
3750 1885 Acked Read 5 Done Done Sent false
3751 1886 Acked Write 8 Done Done Sent false
3752 1887 Acked Write 5 Done Done Sent false
3753 1888 Acked Read 4 Done Done Sent false
3754 1889 Acked Flush 0 Done Done Sent false
3755 1890 Acked Write 10 Done Done Sent false
3756 1891 Acked Write 10 Done Done Sent false
3757 1892 Acked Write 3 Done Done Sent false
3758 1893 Acked Read 9 Done Done Sent false
3759 1894 Acked Read 10 Done Done Sent false
3760 1895 Acked Write 2 Done Done Sent false
3761 1896 Acked Flush 0 Done Done Sent false
3762 1897 Acked Read 1 Done Done Sent false
3763 1898 Acked Read 10 Done Done Sent false
3764 1899 Acked Write 8 Done Done Sent false
3765 1900 Acked Write 6 Done Done Sent false
3766 1901 Acked Read 5 Done Done Sent false
3767 1902 Acked Write 3 Done Done Sent false
3768 1903 Acked Read 9 Done Done Sent false
3769 1904 Acked Write 2 Done Done Sent false
3770 STATES DS:0 DS:1 DS:2 TOTAL
3771 Sent 0 0 301 301
3772 Done 301 301 0 602
3773 Skipped 0 0 0 0
3774 Error 0 0 0 0
3775 Last Flush: Some(JobId(1896)) Some(JobId(1896)) Some(JobId(1603))
3776 Downstairs last five completed: 1603 1602 1601 1600 1599
3777 Upstairs last five completed: 1904 1903 1902 1901 1900
3778 CLIENT: Up:0 ds:301 act:2
37792025-01-28T21:55:07.397ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
37802025-01-28T21:55:08.374ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
37812025-01-28T21:55:08.374ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36201) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
37822025-01-28T21:55:08.375ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
37832025-01-28T21:55:08.376ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
37842025-01-28T21:55:08.376ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
37852025-01-28T21:55:08.376ZINFOcrucible: send last flush ID to this DS: Some(JobId(1603)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
37862025-01-28T21:55:08.376ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
37872025-01-28T21:55:08.376ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1603)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
37882025-01-28T21:55:08.376ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1603)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
3789 ----------------------------------------------------------------
3790 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3791 Downstairs last five completed: 1905 1904 1903 1902 1901
3792 Upstairs last five completed: 1905 1904 1903 1902 1901
3793 CLIENT: Up:0 ds:0 act:3
3794 Replay: All jobs finished
3795 Main task has work to do, go find it
3796 got dsc Stop(2)
3797 stop 2
3798 Jan 28 21:55:10.620 INFO request completed, latency_us[8830] Got stop action so:false kr:false
3799 : 359, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 0c85e2f3-5246-442a-8506-8d7179500d6c, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
3800 Jan 28 21:55:10.621 INFO request completed, latency_us: 110, response_code: 200, uri: /state/cid/2, method: GET, req_id: 31ba1d68-eaab-4240-9181-9faa152dfaf4, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
38012025-01-28T21:55:10.621ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
38022025-01-28T21:55:10.621ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
38032025-01-28T21:55:10.621ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
38042025-01-28T21:55:10.621ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
3805 [8830] Exited with: ExitStatus(unix_wait_status(9))
3806 [8830][2] reports Exit
3807 Jan 28 21:55:14.622 INFO request completed, latency_us: 252, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2980961a-21fa-4f31-9dd5-748fd9276d25, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
3808 001/300 Read block 1632 len 12288
3809 002/300 Write block 1479 len 12288 data: 2 2 2
3810 003/300 Read block 2632 len 36864
3811 004/300 Read block 4296 len 4096
3812 005/300 Flush
3813 006/300 Write block 2144 len 40960 data: 3 2 2 2 2 2 2 3 3 3
3814 007/300 Read block 3901 len 28672
3815 008/300 Read block 196 len 12288
3816 009/300 Read block 2081 len 8192
3817 010/300 Write block 3733 len 28672 data: 3 3 3 3 3 3 3
3818 011/300 Write block 2382 len 12288 data: 3 3 3
3819 012/300 Write block 1105 len 12288 data: 2 2 2
3820 013/300 Write block 3552 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3821 014/300 Write block 1221 len 12288 data: 2 2 2
3822 015/300 Read block 4520 len 4096
3823 016/300 Read block 1650 len 12288
3824 017/300 Read block 2927 len 16384
3825 018/300 Write block 1151 len 28672 data: 4 4 4 4 4 4 4
3826 019/300 Read block 3100 len 8192
3827 020/300 Read block 43 len 8192
3828 021/300 Write block 853 len 4096 data: 3
3829 022/300 Read block 2373 len 32768
3830 023/300 Write block 2658 len 20480 data: 3 3 3 3 2
3831 024/300 Read block 502 len 40960
3832 025/300 Write block 3906 len 32768 data: 2 2 2 2 2 2 2 2
3833 026/300 Read block 2183 len 20480
3834 027/300 Write block 521 len 16384 data: 2 2 3 3
3835 028/300 Write block 3397 len 16384 data: 4 3 3 2
3836 029/300 Write block 4443 len 28672 data: 2 2 2 2 2 2 2
3837 030/300 Read block 2357 len 12288
3838 031/300 Write block 3388 len 32768 data: 3 3 2 2 2 3 3 3
3839 032/300 Read block 1036 len 20480
3840 033/300 Flush
3841 034/300 Read block 3845 len 28672
3842 035/300 Write block 3576 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3843 036/300 Read block 2100 len 36864
3844 037/300 Write block 3665 len 4096 data: 3
3845 038/300 Write block 869 len 40960 data: 4 4 4 3 3 2 2 2 2 2
3846 039/300 Write block 152 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3847 040/300 Read block 2535 len 32768
3848 041/300 Write block 2728 len 12288 data: 2 2 2
3849 042/300 Read block 2986 len 40960
3850 043/300 Flush
3851 044/300 Write block 729 len 24576 data: 2 2 2 2 2 2
3852 045/300 Write block 2614 len 40960 data: 2 3 4 4 4 4 4 4 2 2
3853 046/300 Write block 2750 len 12288 data: 2 2 2
3854 047/300 Flush
3855 048/300 Flush
3856 049/300 Read block 3357 len 24576
3857 050/300 Read block 703 len 12288
3858 051/300 Read block 3849 len 36864
3859 052/300 Write block 1240 len 4096 data: 2
3860 053/300 Write block 4873 len 40960 data: 2 2 2 3 4 4 4 4 4 4
3861 054/300 Write block 2337 len 16384 data: 3 3 3 3
3862 055/300 Read block 1554 len 12288
3863 056/300 Write block 1042 len 12288 data: 3 3 3
3864 057/300 Flush
3865 058/300 Read block 2139 len 24576
3866 059/300 Flush
3867 060/300 Read block 148 len 24576
3868 061/300 Read block 4013 len 40960
3869 062/300 Write block 764 len 12288 data: 2 2 2
3870 063/300 Read block 2419 len 16384
3871 064/300 Write block 3227 len 8192 data: 2 2
3872 065/300 Read block 3909 len 16384
3873 066/300 Write block 1631 len 4096 data: 3
3874 067/300 Read block 1908 len 20480
3875 068/300 Write block 4355 len 40960 data: 3 3 3 3 3 3 2 2 2 2
3876 069/300 Read block 3044 len 12288
3877 070/300 Write block 1154 len 4096 data: 5
3878 071/300 Write block 40 len 28672 data: 2 2 2 2 2 2 2
3879 072/300 Read block 4986 len 12288
3880 073/300 Read block 3579 len 4096
3881 074/300 Flush
3882 075/300 Read block 4174 len 4096
3883 076/300 Read block 2096 len 16384
3884 077/300 Flush
3885 078/300 Write block 436 len 4096 data: 2
3886 079/300 Write block 376 len 8192 data: 2 2
3887 080/300 Write block 1458 len 12288 data: 2 2 2
3888 081/300 Write block 2048 len 8192 data: 2 2
3889 082/300 Read block 1590 len 8192
3890 083/300 Read block 1211 len 4096
3891 084/300 Read block 3575 len 24576
3892 085/300 Write block 114 len 4096 data: 2
3893 086/300 Read block 4501 len 16384
3894 087/300 Read block 1177 len 40960
3895 088/300 Write block 1913 len 32768 data: 2 2 2 2 2 2 3 3
3896 089/300 Read block 2988 len 12288
3897 090/300 Write block 4345 len 12288 data: 2 2 2
3898 091/300 Write block 3213 len 32768 data: 2 2 2 2 2 2 2 2
3899 092/300 Read block 939 len 28672
3900 093/300 Read block 533 len 32768
3901 094/300 Read block 3768 len 36864
3902 095/300 Write block 3816 len 36864 data: 3 3 3 3 3 3 3 3 3
3903 096/300 Flush
3904 097/300 Flush
3905 098/300 Flush
3906 099/300 Read block 4580 len 4096
3907 100/300 Write block 4089 len 24576 data: 3 3 3 3 2 2
3908 101/300 Flush
3909 102/300 Write block 3106 len 4096 data: 3
3910 103/300 Read block 2820 len 40960
3911 104/300 Read block 1403 len 28672
3912 105/300 Read block 2441 len 8192
3913 106/300 Read block 4579 len 24576
3914 107/300 Read block 2434 len 16384
3915 108/300 Write block 3526 len 40960 data: 4 4 4 3 3 3 3 3 3 2
3916 109/300 Flush
3917 110/300 Flush
3918 111/300 Write block 3931 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3919 112/300 Read block 2428 len 12288
3920 113/300 Write block 1697 len 16384 data: 2 2 2 3
3921 114/300 Read block 3959 len 12288
3922 115/300 Write block 345 len 28672 data: 3 3 3 2 2 2 2
3923 116/300 Write block 3133 len 24576 data: 2 2 2 2 2 2
3924 117/300 Read block 73 len 36864
3925 118/300 Flush
3926 119/300 Write block 3055 len 28672 data: 2 2 2 2 3 3 3
3927 120/300 Write block 4336 len 12288 data: 3 3 3
3928 121/300 Read block 626 len 36864
3929 122/300 Write block 3301 len 20480 data: 2 2 2 2 2
3930 123/300 Read block 4493 len 8192
3931 124/300 Read block 4400 len 20480
3932 125/300 Flush
3933 126/300 Read block 527 len 20480
3934 127/300 Flush
3935 128/300 Read block 2931 len 24576
3936 129/300 Read block 3424 len 28672
3937 130/300 Write block 511 len 8192 data: 2 2
3938 131/300 Write block 3293 len 28672 data: 2 2 2 2 2 2 2
3939 132/300 Read block 1373 len 36864
3940 133/300 Read block 2148 len 36864
3941 134/300 Write block 2857 len 12288 data: 3 2 2
3942 135/300 Read block 4083 len 20480
3943 136/300 Write block 3988 len 40960 data: 2 2 2 2 2 2 2 2 3 3
3944 137/300 Write block 358 len 20480 data: 3 2 3 3 3
3945 138/300 Write block 4402 len 12288 data: 3 3 3
3946 139/300 Write block 1344 len 40960 data: 3 3 3 3 2 2 2 2 2 3
3947 140/300 Read block 4616 len 16384
3948 141/300 Read block 2882 len 4096
3949 142/300 Flush
3950 143/300 Read block 2425 len 20480
3951 144/300 Read block 1695 len 20480
3952 145/300 Write block 2731 len 20480 data: 2 2 3 2 2
3953 146/300 Write block 89 len 12288 data: 2 2 3
3954 147/300 Write block 2201 len 32768 data: 2 3 3 3 2 2 2 2
3955 148/300 Write block 1953 len 20480 data: 2 2 2 2 2
3956 149/300 Write block 2008 len 28672 data: 2 3 3 3 3 3 2
3957 150/300 Read block 3681 len 20480
3958 151/300 Read block 1359 len 36864
3959 152/300 Flush
3960 153/300 Read block 671 len 24576
3961 154/300 Read block 3040 len 4096
3962 155/300 Write block 4770 len 8192 data: 4 4
3963 156/300 Read block 2462 len 28672
3964 157/300 Read block 3275 len 16384
3965 158/300 Read block 906 len 24576
3966 159/300 Read block 2222 len 28672
3967 160/300 Read block 6 len 16384
3968 161/300 Write block 1535 len 4096 data: 2
3969 162/300 Read block 257 len 4096
3970 163/300 Write block 123 len 16384 data: 2 2 2 2
3971 164/300 Write block 2856 len 12288 data: 2 4 3
3972 165/300 Read block 873 len 28672
3973 166/300 Write block 821 len 8192 data: 2 2
3974 167/300 Read block 761 len 32768
3975 168/300 Read block 781 len 16384
3976 169/300 Read block 1683 len 8192
3977 170/300 Write block 2507 len 32768 data: 2 2 2 2 2 2 2 2
3978 171/300 Write block 755 len 4096 data: 2
3979 172/300 Read block 2522 len 28672
3980 173/300 Read block 3718 len 36864
3981 174/300 Flush
3982 175/300 Read block 3422 len 40960
3983 176/300 Write block 2789 len 8192 data: 3 3
3984 177/300 Write block 3304 len 36864 data: 3 3 2 2 2 2 2 2 2
3985 178/300 Read block 3989 len 32768
3986 179/300 Write block 4058 len 24576 data: 2 2 2 2 2 2
3987 180/300 Write block 545 len 16384 data: 3 3 4 4
3988 181/300 Write block 3559 len 4096 data: 3
3989 182/300 Read block 1909 len 28672
3990 183/300 Read block 107 len 24576
3991 184/300 Read block 372 len 20480
3992 185/300 Write block 4101 len 36864 data: 2 3 3 3 3 3 3 3 2
3993 186/300 Read block 2991 len 16384
3994 187/300 Read block 3786 len 12288
3995 188/300 Write block 4804 len 16384 data: 2 2 2 2
3996 189/300 Write block 4288 len 12288 data: 2 2 2
3997 190/300 Read block 2441 len 8192
3998 191/300 Read block 3745 len 16384
3999 192/300 Write block 2615 len 4096 data: 4
4000 193/300 Write block 3328 len 28672 data: 3 3 3 3 2 2 2
4001 194/300 Write block 788 len 36864 data: 2 2 2 2 2 2 2 2 2
4002 195/300 Write block 74 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4003 196/300 Flush
4004 197/300 Write block 4771 len 20480 data: 5 4 3 3 2
4005 198/300 Read block 2477 len 16384
4006 199/300 Flush
4007 200/300 Write block 2820 len 8192 data: 2 2
4008 201/300 Read block 4502 len 12288
4009 202/300 Write block 2122 len 24576 data: 3 2 3 2 3 3
4010 203/300 Read block 931 len 32768
4011 204/300 Flush
4012 205/300 Read block 532 len 16384
4013 206/300 Flush
4014 207/300 Read block 4088 len 20480
4015 208/300 Write block 2927 len 32768 data: 2 2 2 2 2 2 2 2
4016 209/300 Flush
4017 210/300 Read block 1102 len 16384
4018 211/300 Write block 3271 len 24576 data: 2 2 2 2 2 2
4019 212/300 Read block 2362 len 40960
4020 213/300 Write block 1265 len 20480 data: 2 2 2 2 2
4021 214/300 Flush
4022 215/300 Write block 1530 len 36864 data: 3 3 3 2 2 3 2 2 2
4023 216/300 Read block 813 len 36864
4024 217/300 Read block 2286 len 32768
4025 218/300 Read block 1236 len 20480
4026 219/300 Read block 17 len 20480
4027 220/300 Write block 1731 len 36864 data: 2 2 2 2 2 2 2 2 2
4028 221/300 Read block 2084 len 32768
4029 222/300 Read block 3654 len 28672
4030 223/300 Read block 1252 len 24576
4031 224/300 Read block 1017 len 16384
4032 225/300 Read block 159 len 36864
4033 226/300 Read block 2161 len 24576
4034 227/300 Write block 3375 len 28672 data: 2 2 2 2 2 2 2
4035 228/300 Read block 2355 len 8192
4036 229/300 Write block 4820 len 16384 data: 2 2 2 2
4037 230/300 Read block 2027 len 20480
4038 231/300 Read block 281 len 20480
4039 232/300 Read block 4438 len 40960
4040 233/300 Read block 1636 len 24576
4041 234/300 Write block 2250 len 12288 data: 2 2 2
4042 235/300 Read block 1082 len 28672
4043 236/300 Read block 187 len 8192
4044 237/300 Write block 4555 len 32768 data: 2 2 2 2 2 2 2 2
4045 238/300 Read block 1345 len 32768
4046 239/300 Read block 3168 len 40960
4047 240/300 Flush
4048 241/300 Read block 968 len 8192
4049 242/300 Read block 1438 len 40960
4050 243/300 Write block 1533 len 20480 data: 3 3 4 3 3
4051 244/300 Write block 4486 len 20480 data: 4 2 2 2 2
4052 245/300 Write block 2271 len 36864 data: 2 2 2 2 3 3 3 3 3
4053 246/300 Write block 1289 len 36864 data: 3 3 3 3 3 2 2 2 2
4054 247/300 Write block 232 len 8192 data: 2 3
4055 248/300 Write block 751 len 28672 data: 2 2 2 2 3 2 2
4056 249/300 Read block 2375 len 40960
4057 250/300 Read block 236 len 12288
4058 251/300 Flush
4059 252/300 Read block 424 len 32768
4060 253/300 Write block 4137 len 4096 data: 2
4061 254/300 Read block 3897 len 12288
4062 255/300 Read block 4446 len 12288
4063 256/300 Read block 365 len 16384
4064 257/300 Read block 2656 len 12288
4065 258/300 Read block 1925 len 20480
4066 259/300 Write block 4663 len 20480 data: 4 4 4 4 2
4067 260/300 Read block 2291 len 16384
4068 261/300 Write block 4244 len 16384 data: 3 3 3 2
4069 262/300 Read block 1551 len 12288
4070 263/300 Write block 3158 len 32768 data: 2 2 2 2 2 2 2 2
4071 264/300 Write block 766 len 24576 data: 3 2 2 2 2 2
4072 265/300 Write block 782 len 32768 data: 2 2 2 2 2 2 3 3
4073 266/300 Read block 4422 len 16384
4074 267/300 Read block 3232 len 12288
4075 268/300 Read block 3254 len 36864
4076 269/300 Read block 2582 len 28672
4077 270/300 Read block 617 len 24576
4078 271/300 Write block 2016 len 16384 data: 2 2 2 2
4079 272/300 Read block 4639 len 32768
4080 273/300 Read block 108 len 8192
4081 274/300 Write block 2586 len 4096 data: 3
4082 275/300 Write block 4344 len 36864 data: 2 3 3 3 2 2 2 3 3
4083 276/300 Write block 1560 len 24576 data: 2 2 2 2 2 2
4084 277/300 Read block 2448 len 20480
4085 278/300 Read block 3367 len 4096
4086 279/300 Read block 4330 len 28672
4087 280/300 Flush
4088 281/300 Write block 4564 len 12288 data: 2 3 3
4089 282/300 Read block 2872 len 32768
4090 283/300 Read block 2962 len 4096
4091 284/300 Write block 2673 len 36864 data: 2 3 4 4 4 4 5 6 5
4092 285/300 Read block 56 len 4096
4093 286/300 Read block 1261 len 28672
4094 287/300 Read block 4971 len 40960
4095 288/300 Read block 2916 len 32768
4096 289/300 Flush
4097 290/300 Read block 2979 len 40960
4098 291/300 Flush
4099 292/300 Write block 1410 len 4096 data: 3
4100 293/300 Flush
4101 294/300 Write block 262 len 4096 data: 2
4102 295/300 Write block 4165 len 12288 data: 2 3 3
4103 296/300 Read block 1074 len 12288
4104 297/300 Flush
4105 298/300 Write block 4410 len 24576 data: 2 2 2 2 3 2
4106 299/300 Write block 2419 len 32768 data: 3 2 2 2 2 2 3 3
4107 300/300 Read block 4343 len 12288
4108 301/300 Write block 2730 len 36864 data: 3 3 3 4 3 3 2 2 2
4109 Main task has work to do, go find it
4110 got dsc Start(2)
4111 start 2
4112 Jan 28 21:55:14.860 INFO request completed, latency_us: 261, response_code: 204, uri: /start/cid/2, method: POST, req_id: e923cab6-6df1-437d-b8a5-3f55aba439c2, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
4113 Replay: started 2, returned:Ok(())
4114 ----------------------------------------------------------------
4115 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4116 States: Active Active Offline
4117 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4118 1906 Acked Read 3 Done Done Sent false
4119 1907 Acked Write 3 Done Done Sent false
4120 1908 Acked Read 9 Done Done Sent false
4121 1909 Acked Read 1 Done Done Sent false
4122 1910 Acked Flush 0 Done Done Sent false
4123 1911 Acked Write 10 Done Done Sent false
4124 1912 Acked Read 7 Done Done Sent false
4125 1913 Acked Read 3 Done Done Sent false
4126 1914 Acked Read 2 Done Done Sent false
4127 1915 Acked Write 7 Done Done Sent false
4128 1916 Acked Write 3 Done Done Sent false
4129 1917 Acked Write 3 Done Done Sent false
4130 1918 Acked Write 10 Done Done Sent false
4131 1919 Acked Write 3 Done Done Sent false
4132 1920 Acked Read 1 Done Done Sent false
4133 1921 Acked Read 3 Done Done Sent false
4134 1922 Acked Read 4 Done Done Sent false
4135 1923 Acked Write 7 Done Done Sent false
4136 1924 Acked Read 2 Done Done Sent false
4137 1925 Acked Read 2 Done Done Sent false
4138 1926 Acked Write 1 Done Done Sent false
4139 1927 Acked Read 8 Done Done Sent false
4140 1928 Acked Write 5 Done Done Sent false
4141 1929 Acked Read 10 Done Done Sent false
4142 1930 Acked Write 8 Done Done Sent false
4143 1931 Acked Read 5 Done Done Sent false
4144 1932 Acked Write 4 Done Done Sent false
4145 1933 Acked Write 4 Done Done Sent false
4146 1934 Acked Write 7 Done Done Sent false
4147 1935 Acked Read 3 Done Done Sent false
4148 1936 Acked Write 8 Done Done Sent false
4149 1937 Acked Read 5 Done Done Sent false
4150 1938 Acked Flush 0 Done Done Sent false
4151 1939 Acked Read 7 Done Done Sent false
4152 1940 Acked Write 10 Done Done Sent false
4153 1941 Acked Read 9 Done Done Sent false
4154 1942 Acked Write 1 Done Done Sent false
4155 1943 Acked Write 10 Done Done Sent false
4156 1944 Acked Write 10 Done Done Sent false
4157 1945 Acked Read 8 Done Done Sent false
4158 1946 Acked Write 3 Done Done Sent false
4159 1947 Acked Read 10 Done Done Sent false
4160 1948 Acked Flush 0 Done Done Sent false
4161 1949 Acked Write 6 Done Done Sent false
4162 1950 Acked Write 10 Done Done Sent false
4163 1951 Acked Write 3 Done Done Sent false
4164 1952 Acked Flush 0 Done Done Sent false
4165 1953 Acked Flush 0 Done Done Sent false
4166 1954 Acked Read 6 Done Done Sent false
4167 1955 Acked Read 3 Done Done Sent false
4168 1956 Acked Read 9 Done Done Sent false
4169 1957 Acked Write 1 Done Done Sent false
4170 1958 Acked Write 10 Done Done Sent false
4171 1959 Acked Write 4 Done Done Sent false
4172 1960 Acked Read 3 Done Done Sent false
4173 1961 Acked Write 3 Done Done Sent false
4174 1962 Acked Flush 0 Done Done Sent false
4175 1963 Acked Read 6 Done Done Sent false
4176 1964 Acked Flush 0 Done Done Sent false
4177 1965 Acked Read 6 Done Done Sent false
4178 1966 Acked Read 10 Done Done Sent false
4179 1967 Acked Write 3 Done Done Sent false
4180 1968 Acked Read 4 Done Done Sent false
4181 1969 Acked Write 2 Done Done Sent false
4182 1970 Acked Read 4 Done Done Sent false
4183 1971 Acked Write 1 Done Done Sent false
4184 1972 Acked Read 5 Done Done Sent false
4185 1973 Acked Write 10 Done Done Sent false
4186 1974 Acked Read 3 Done Done Sent false
4187 1975 Acked Write 1 Done Done Sent false
4188 1976 Acked Write 7 Done Done Sent false
4189 1977 Acked Read 3 Done Done Sent false
4190 1978 Acked Read 1 Done Done Sent false
4191 1979 Acked Flush 0 Done Done Sent false
4192 1980 Acked Read 1 Done Done Sent false
4193 1981 Acked Read 4 Done Done Sent false
4194 1982 Acked Flush 0 Done Done Sent false
4195 1983 Acked Write 1 Done Done Sent false
4196 1984 Acked Write 2 Done Done Sent false
4197 1985 Acked Write 3 Done Done Sent false
4198 1986 Acked Write 2 Done Done Sent false
4199 1987 Acked Read 2 Done Done Sent false
4200 1988 Acked Read 1 Done Done Sent false
4201 1989 Acked Read 6 Done Done Sent false
4202 1990 Acked Write 1 Done Done Sent false
4203 1991 Acked Read 4 Done Done Sent false
4204 1992 Acked Read 10 Done Done Sent false
4205 1993 Acked Write 8 Done Done Sent false
4206 1994 Acked Read 3 Done Done Sent false
4207 1995 Acked Write 3 Done Done Sent false
4208 1996 Acked Write 8 Done Done Sent false
4209 1997 Acked Read 7 Done Done Sent false
4210 1998 Acked Read 8 Done Done Sent false
4211 1999 Acked Read 9 Done Done Sent false
4212 2000 Acked Write 9 Done Done Sent false
4213 2001 Acked Flush 0 Done Done Sent false
4214 2002 Acked Flush 0 Done Done Sent false
4215 2003 Acked Flush 0 Done Done Sent false
4216 2004 Acked Read 1 Done Done Sent false
4217 2005 Acked Write 6 Done Done Sent false
4218 2006 Acked Flush 0 Done Done Sent false
4219 2007 Acked Write 1 Done Done Sent false
4220 2008 Acked Read 10 Done Done Sent false
4221 2009 Acked Read 7 Done Done Sent false
4222 2010 Acked Read 2 Done Done Sent false
4223 2011 Acked Read 6 Done Done Sent false
4224 2012 Acked Read 4 Done Done Sent false
4225 2013 Acked Write 10 Done Done Sent false
4226 2014 Acked Flush 0 Done Done Sent false
4227 2015 Acked Flush 0 Done Done Sent false
4228 2016 Acked Write 10 Done Done Sent false
4229 2017 Acked Read 3 Done Done Sent false
4230 2018 Acked Write 4 Done Done Sent false
4231 2019 Acked Read 3 Done Done Sent false
4232 2020 Acked Write 7 Done Done Sent false
4233 2021 Acked Write 6 Done Done Sent false
4234 2022 Acked Read 9 Done Done Sent false
4235 2023 Acked Flush 0 Done Done Sent false
4236 2024 Acked Write 7 Done Done Sent false
4237 2025 Acked Write 3 Done Done Sent false
4238 2026 Acked Read 9 Done Done Sent false
4239 2027 Acked Write 5 Done Done Sent false
4240 2028 Acked Read 2 Done Done Sent false
4241 2029 Acked Read 5 Done Done Sent false
4242 2030 Acked Flush 0 Done Done Sent false
4243 2031 Acked Read 5 Done Done Sent false
4244 2032 Acked Flush 0 Done Done Sent false
4245 2033 Acked Read 6 Done Done Sent false
4246 2034 Acked Read 7 Done Done Sent false
4247 2035 Acked Write 2 Done Done Sent false
4248 2036 Acked Write 7 Done Done Sent false
4249 2037 Acked Read 9 Done Done Sent false
4250 2038 Acked Read 9 Done Done Sent false
4251 2039 Acked Write 3 Done Done Sent false
4252 2040 Acked Read 5 Done Done Sent false
4253 2041 Acked Write 10 Done Done Sent false
4254 2042 Acked Write 5 Done Done Sent false
4255 2043 Acked Write 3 Done Done Sent false
4256 2044 Acked Write 10 Done Done Sent false
4257 2045 Acked Read 4 Done Done Sent false
4258 2046 Acked Read 1 Done Done Sent false
4259 2047 Acked Flush 0 Done Done Sent false
4260 2048 Acked Read 5 Done Done Sent false
4261 2049 Acked Read 5 Done Done Sent false
4262 2050 Acked Write 5 Done Done Sent false
4263 2051 Acked Write 3 Done Done Sent false
4264 2052 Acked Write 8 Done Done Sent false
4265 2053 Acked Write 5 Done Done Sent false
4266 2054 Acked Write 7 Done Done Sent false
4267 2055 Acked Read 5 Done Done Sent false
4268 2056 Acked Read 9 Done Done Sent false
4269 2057 Acked Flush 0 Done Done Sent false
4270 2058 Acked Read 6 Done Done Sent false
4271 2059 Acked Read 1 Done Done Sent false
4272 2060 Acked Write 2 Done Done Sent false
4273 2061 Acked Read 7 Done Done Sent false
4274 2062 Acked Read 4 Done Done Sent false
4275 2063 Acked Read 6 Done Done Sent false
4276 2064 Acked Read 7 Done Done Sent false
4277 2065 Acked Read 4 Done Done Sent false
4278 2066 Acked Write 1 Done Done Sent false
4279 2067 Acked Read 1 Done Done Sent false
4280 2068 Acked Write 4 Done Done Sent false
4281 2069 Acked Write 3 Done Done Sent false
4282 2070 Acked Read 7 Done Done Sent false
4283 2071 Acked Write 2 Done Done Sent false
4284 2072 Acked Read 8 Done Done Sent false
4285 2073 Acked Read 4 Done Done Sent false
4286 2074 Acked Read 2 Done Done Sent false
4287 2075 Acked Write 8 Done Done Sent false
4288 2076 Acked Write 1 Done Done Sent false
4289 2077 Acked Read 7 Done Done Sent false
4290 2078 Acked Read 9 Done Done Sent false
4291 2079 Acked Flush 0 Done Done Sent false
4292 2080 Acked Read 10 Done Done Sent false
4293 2081 Acked Write 2 Done Done Sent false
4294 2082 Acked Write 9 Done Done Sent false
4295 2083 Acked Read 8 Done Done Sent false
4296 2084 Acked Write 6 Done Done Sent false
4297 2085 Acked Write 4 Done Done Sent false
4298 2086 Acked Write 1 Done Done Sent false
4299 2087 Acked Read 7 Done Done Sent false
4300 2088 Acked Read 6 Done Done Sent false
4301 2089 Acked Read 5 Done Done Sent false
4302 2090 Acked Write 9 Done Done Sent false
4303 2091 Acked Read 4 Done Done Sent false
4304 2092 Acked Read 3 Done Done Sent false
4305 2093 Acked Write 4 Done Done Sent false
4306 2094 Acked Write 3 Done Done Sent false
4307 2095 Acked Read 2 Done Done Sent false
4308 2096 Acked Read 4 Done Done Sent false
4309 2097 Acked Write 1 Done Done Sent false
4310 2098 Acked Write 7 Done Done Sent false
4311 2099 Acked Write 9 Done Done Sent false
4312 2100 Acked Write 10 Done Done Sent false
4313 2101 Acked Flush 0 Done Done Sent false
4314 2102 Acked Write 5 Done Done Sent false
4315 2103 Acked Read 4 Done Done Sent false
4316 2104 Acked Flush 0 Done Done Sent false
4317 2105 Acked Write 2 Done Done Sent false
4318 2106 Acked Read 3 Done Done Sent false
4319 2107 Acked Write 6 Done Done Sent false
4320 2108 Acked Read 8 Done Done Sent false
4321 2109 Acked Flush 0 Done Done Sent false
4322 2110 Acked Read 4 Done Done Sent false
4323 2111 Acked Flush 0 Done Done Sent false
4324 2112 Acked Read 5 Done Done Sent false
4325 2113 Acked Write 8 Done Done Sent false
4326 2114 Acked Flush 0 Done Done Sent false
4327 2115 Acked Read 4 Done Done Sent false
4328 2116 Acked Write 6 Done Done Sent false
4329 2117 Acked Read 10 Done Done Sent false
4330 2118 Acked Write 5 Done Done Sent false
4331 2119 Acked Flush 0 Done Done Sent false
4332 2120 Acked Write 9 Done Done Sent false
4333 2121 Acked Read 9 Done Done Sent false
4334 2122 Acked Read 8 Done Done Sent false
4335 2123 Acked Read 5 Done Done Sent false
4336 2124 Acked Read 5 Done Done Sent false
4337 2125 Acked Write 9 Done Done Sent false
4338 2126 Acked Read 8 Done Done Sent false
4339 2127 Acked Read 7 Done Done Sent false
4340 2128 Acked Read 6 Done Done Sent false
4341 2129 Acked Read 4 Done Done Sent false
4342 2130 Acked Read 9 Done Done Sent false
4343 2131 Acked Read 6 Done Done Sent false
4344 2132 Acked Write 7 Done Done Sent false
4345 2133 Acked Read 2 Done Done Sent false
4346 2134 Acked Write 4 Done Done Sent false
4347 2135 Acked Read 5 Done Done Sent false
4348 2136 Acked Read 5 Done Done Sent false
4349 2137 Acked Read 10 Done Done Sent false
4350 2138 Acked Read 6 Done Done Sent false
4351 2139 Acked Write 3 Done Done Sent false
4352 2140 Acked Read 7 Done Done Sent false
4353 2141 Acked Read 2 Done Done Sent false
4354 2142 Acked Write 8 Done Done Sent false
4355 2143 Acked Read 8 Done Done Sent false
4356 2144 Acked Read 10 Done Done Sent false
4357 2145 Acked Flush 0 Done Done Sent false
4358 2146 Acked Read 2 Done Done Sent false
4359 2147 Acked Read 10 Done Done Sent false
4360 2148 Acked Write 5 Done Done Sent false
4361 2149 Acked Write 5 Done Done Sent false
4362 2150 Acked Write 9 Done Done Sent false
4363 2151 Acked Write 9 Done Done Sent false
4364 2152 Acked Write 2 Done Done Sent false
4365 2153 Acked Write 7 Done Done Sent false
4366 2154 Acked Read 10 Done Done Sent false
4367 2155 Acked Read 3 Done Done Sent false
4368 2156 Acked Flush 0 Done Done Sent false
4369 2157 Acked Read 8 Done Done Sent false
4370 2158 Acked Write 1 Done Done Sent false
4371 2159 Acked Read 3 Done Done Sent false
4372 2160 Acked Read 3 Done Done Sent false
4373 2161 Acked Read 4 Done Done Sent false
4374 2162 Acked Read 3 Done Done Sent false
4375 2163 Acked Read 5 Done Done Sent false
4376 2164 Acked Write 5 Done Done Sent false
4377 2165 Acked Read 4 Done Done Sent false
4378 2166 Acked Write 4 Done Done Sent false
4379 2167 Acked Read 3 Done Done Sent false
4380 2168 Acked Write 8 Done Done Sent false
4381 2169 Acked Write 6 Done Done Sent false
4382 2170 Acked Write 8 Done Done Sent false
4383 2171 Acked Read 4 Done Done Sent false
4384 2172 Acked Read 3 Done Done Sent false
4385 2173 Acked Read 9 Done Done Sent false
4386 2174 Acked Read 7 Done Done Sent false
4387 2175 Acked Read 6 Done Done Sent false
4388 2176 Acked Write 4 Done Done Sent false
4389 2177 Acked Read 8 Done Done Sent false
4390 2178 Acked Read 2 Done Done Sent false
4391 2179 Acked Write 1 Done Done Sent false
4392 2180 Acked Write 9 Done Done Sent false
4393 2181 Acked Write 6 Done Done Sent false
4394 2182 Acked Read 5 Done Done Sent false
4395 2183 Acked Read 1 Done Done Sent false
4396 2184 Acked Read 7 Done Done Sent false
4397 2185 Acked Flush 0 Done Done Sent false
4398 2186 Acked Write 3 Done Done Sent false
4399 2187 Acked Read 8 Done Done Sent false
4400 2188 Acked Read 1 Done Done Sent false
4401 2189 Acked Write 9 Done Done Sent false
4402 2190 Acked Read 1 Done Done Sent false
4403 2191 Acked Read 7 Done Done Sent false
4404 2192 Acked Read 10 Done Done Sent false
4405 2193 Acked Read 8 Done Done Sent false
4406 2194 Acked Flush 0 Done Done Sent false
4407 2195 Acked Read 10 Done Done Sent false
4408 2196 Acked Flush 0 Done Done Sent false
4409 2197 Acked Write 1 Done Done Sent false
4410 2198 Acked Flush 0 Done Done Sent false
4411 2199 Acked Write 1 Done Done Sent false
4412 2200 Acked Write 3 Done Done Sent false
4413 2201 Acked Read 3 Done Done Sent false
4414 2202 Acked Flush 0 Done Done Sent false
4415 2203 Acked Write 6 Done Done Sent false
4416 2204 Acked Write 8 Done Done Sent false
4417 2205 Acked Read 3 Done Done Sent false
4418 2206 Acked Write 9 Done Done Sent false
4419 STATES DS:0 DS:1 DS:2 TOTAL
4420 Sent 0 0 301 301
4421 Done 301 301 0 602
4422 Skipped 0 0 0 0
4423 Error 0 0 0 0
4424 Last Flush: Some(JobId(2202)) Some(JobId(2202)) Some(JobId(1905))
4425 Downstairs last five completed: 1905 1904 1903 1902 1901
4426 Upstairs last five completed: 2206 2205 2204 2203 2202
4427 CLIENT: Up:0 ds:301 act:2
44282025-01-28T21:55:15.411ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
44292025-01-28T21:55:16.413ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
4430 [8830] got start action
4431 [8830] I am going to restart
4432 Starting downstairs at port 8830
4433 Make output file at "/tmp/dsc/downstairs-8830.txt"
4434 [8830][2] reports Starting
4435 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1206)
4436 [8830][2] reports Running
44372025-01-28T21:55:17.415ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
44382025-01-28T21:55:18.417ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
4439 ----------------------------------------------------------------
4440 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4441 States: Active Active Offline
4442 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4443 1906 Acked Read 3 Done Done Sent false
4444 1907 Acked Write 3 Done Done Sent false
4445 1908 Acked Read 9 Done Done Sent false
4446 1909 Acked Read 1 Done Done Sent false
4447 1910 Acked Flush 0 Done Done Sent false
4448 1911 Acked Write 10 Done Done Sent false
4449 1912 Acked Read 7 Done Done Sent false
4450 1913 Acked Read 3 Done Done Sent false
4451 1914 Acked Read 2 Done Done Sent false
4452 1915 Acked Write 7 Done Done Sent false
4453 1916 Acked Write 3 Done Done Sent false
4454 1917 Acked Write 3 Done Done Sent false
4455 1918 Acked Write 10 Done Done Sent false
4456 1919 Acked Write 3 Done Done Sent false
4457 1920 Acked Read 1 Done Done Sent false
4458 1921 Acked Read 3 Done Done Sent false
4459 1922 Acked Read 4 Done Done Sent false
4460 1923 Acked Write 7 Done Done Sent false
4461 1924 Acked Read 2 Done Done Sent false
4462 1925 Acked Read 2 Done Done Sent false
4463 1926 Acked Write 1 Done Done Sent false
4464 1927 Acked Read 8 Done Done Sent false
4465 1928 Acked Write 5 Done Done Sent false
4466 1929 Acked Read 10 Done Done Sent false
4467 1930 Acked Write 8 Done Done Sent false
4468 1931 Acked Read 5 Done Done Sent false
4469 1932 Acked Write 4 Done Done Sent false
4470 1933 Acked Write 4 Done Done Sent false
4471 1934 Acked Write 7 Done Done Sent false
4472 1935 Acked Read 3 Done Done Sent false
4473 1936 Acked Write 8 Done Done Sent false
4474 1937 Acked Read 5 Done Done Sent false
4475 1938 Acked Flush 0 Done Done Sent false
4476 1939 Acked Read 7 Done Done Sent false
4477 1940 Acked Write 10 Done Done Sent false
4478 1941 Acked Read 9 Done Done Sent false
4479 1942 Acked Write 1 Done Done Sent false
4480 1943 Acked Write 10 Done Done Sent false
4481 1944 Acked Write 10 Done Done Sent false
4482 1945 Acked Read 8 Done Done Sent false
4483 1946 Acked Write 3 Done Done Sent false
4484 1947 Acked Read 10 Done Done Sent false
4485 1948 Acked Flush 0 Done Done Sent false
4486 1949 Acked Write 6 Done Done Sent false
4487 1950 Acked Write 10 Done Done Sent false
4488 1951 Acked Write 3 Done Done Sent false
4489 1952 Acked Flush 0 Done Done Sent false
4490 1953 Acked Flush 0 Done Done Sent false
4491 1954 Acked Read 6 Done Done Sent false
4492 1955 Acked Read 3 Done Done Sent false
4493 1956 Acked Read 9 Done Done Sent false
4494 1957 Acked Write 1 Done Done Sent false
4495 1958 Acked Write 10 Done Done Sent false
4496 1959 Acked Write 4 Done Done Sent false
4497 1960 Acked Read 3 Done Done Sent false
4498 1961 Acked Write 3 Done Done Sent false
4499 1962 Acked Flush 0 Done Done Sent false
4500 1963 Acked Read 6 Done Done Sent false
4501 1964 Acked Flush 0 Done Done Sent false
4502 1965 Acked Read 6 Done Done Sent false
4503 1966 Acked Read 10 Done Done Sent false
4504 1967 Acked Write 3 Done Done Sent false
4505 1968 Acked Read 4 Done Done Sent false
4506 1969 Acked Write 2 Done Done Sent false
4507 1970 Acked Read 4 Done Done Sent false
4508 1971 Acked Write 1 Done Done Sent false
4509 1972 Acked Read 5 Done Done Sent false
4510 1973 Acked Write 10 Done Done Sent false
4511 1974 Acked Read 3 Done Done Sent false
4512 1975 Acked Write 1 Done Done Sent false
4513 1976 Acked Write 7 Done Done Sent false
4514 1977 Acked Read 3 Done Done Sent false
4515 1978 Acked Read 1 Done Done Sent false
4516 1979 Acked Flush 0 Done Done Sent false
4517 1980 Acked Read 1 Done Done Sent false
4518 1981 Acked Read 4 Done Done Sent false
4519 1982 Acked Flush 0 Done Done Sent false
4520 1983 Acked Write 1 Done Done Sent false
4521 1984 Acked Write 2 Done Done Sent false
4522 1985 Acked Write 3 Done Done Sent false
4523 1986 Acked Write 2 Done Done Sent false
4524 1987 Acked Read 2 Done Done Sent false
4525 1988 Acked Read 1 Done Done Sent false
4526 1989 Acked Read 6 Done Done Sent false
4527 1990 Acked Write 1 Done Done Sent false
4528 1991 Acked Read 4 Done Done Sent false
4529 1992 Acked Read 10 Done Done Sent false
4530 1993 Acked Write 8 Done Done Sent false
4531 1994 Acked Read 3 Done Done Sent false
4532 1995 Acked Write 3 Done Done Sent false
4533 1996 Acked Write 8 Done Done Sent false
4534 1997 Acked Read 7 Done Done Sent false
4535 1998 Acked Read 8 Done Done Sent false
4536 1999 Acked Read 9 Done Done Sent false
4537 2000 Acked Write 9 Done Done Sent false
4538 2001 Acked Flush 0 Done Done Sent false
4539 2002 Acked Flush 0 Done Done Sent false
4540 2003 Acked Flush 0 Done Done Sent false
4541 2004 Acked Read 1 Done Done Sent false
4542 2005 Acked Write 6 Done Done Sent false
4543 2006 Acked Flush 0 Done Done Sent false
4544 2007 Acked Write 1 Done Done Sent false
4545 2008 Acked Read 10 Done Done Sent false
4546 2009 Acked Read 7 Done Done Sent false
4547 2010 Acked Read 2 Done Done Sent false
4548 2011 Acked Read 6 Done Done Sent false
4549 2012 Acked Read 4 Done Done Sent false
4550 2013 Acked Write 10 Done Done Sent false
4551 2014 Acked Flush 0 Done Done Sent false
4552 2015 Acked Flush 0 Done Done Sent false
4553 2016 Acked Write 10 Done Done Sent false
4554 2017 Acked Read 3 Done Done Sent false
4555 2018 Acked Write 4 Done Done Sent false
4556 2019 Acked Read 3 Done Done Sent false
4557 2020 Acked Write 7 Done Done Sent false
4558 2021 Acked Write 6 Done Done Sent false
4559 2022 Acked Read 9 Done Done Sent false
4560 2023 Acked Flush 0 Done Done Sent false
4561 2024 Acked Write 7 Done Done Sent false
4562 2025 Acked Write 3 Done Done Sent false
4563 2026 Acked Read 9 Done Done Sent false
4564 2027 Acked Write 5 Done Done Sent false
4565 2028 Acked Read 2 Done Done Sent false
4566 2029 Acked Read 5 Done Done Sent false
4567 2030 Acked Flush 0 Done Done Sent false
4568 2031 Acked Read 5 Done Done Sent false
4569 2032 Acked Flush 0 Done Done Sent false
4570 2033 Acked Read 6 Done Done Sent false
4571 2034 Acked Read 7 Done Done Sent false
4572 2035 Acked Write 2 Done Done Sent false
4573 2036 Acked Write 7 Done Done Sent false
4574 2037 Acked Read 9 Done Done Sent false
4575 2038 Acked Read 9 Done Done Sent false
4576 2039 Acked Write 3 Done Done Sent false
4577 2040 Acked Read 5 Done Done Sent false
4578 2041 Acked Write 10 Done Done Sent false
4579 2042 Acked Write 5 Done Done Sent false
4580 2043 Acked Write 3 Done Done Sent false
4581 2044 Acked Write 10 Done Done Sent false
4582 2045 Acked Read 4 Done Done Sent false
4583 2046 Acked Read 1 Done Done Sent false
4584 2047 Acked Flush 0 Done Done Sent false
4585 2048 Acked Read 5 Done Done Sent false
4586 2049 Acked Read 5 Done Done Sent false
4587 2050 Acked Write 5 Done Done Sent false
4588 2051 Acked Write 3 Done Done Sent false
4589 2052 Acked Write 8 Done Done Sent false
4590 2053 Acked Write 5 Done Done Sent false
4591 2054 Acked Write 7 Done Done Sent false
4592 2055 Acked Read 5 Done Done Sent false
4593 2056 Acked Read 9 Done Done Sent false
4594 2057 Acked Flush 0 Done Done Sent false
4595 2058 Acked Read 6 Done Done Sent false
4596 2059 Acked Read 1 Done Done Sent false
4597 2060 Acked Write 2 Done Done Sent false
4598 2061 Acked Read 7 Done Done Sent false
4599 2062 Acked Read 4 Done Done Sent false
4600 2063 Acked Read 6 Done Done Sent false
4601 2064 Acked Read 7 Done Done Sent false
4602 2065 Acked Read 4 Done Done Sent false
4603 2066 Acked Write 1 Done Done Sent false
4604 2067 Acked Read 1 Done Done Sent false
4605 2068 Acked Write 4 Done Done Sent false
4606 2069 Acked Write 3 Done Done Sent false
4607 2070 Acked Read 7 Done Done Sent false
4608 2071 Acked Write 2 Done Done Sent false
4609 2072 Acked Read 8 Done Done Sent false
4610 2073 Acked Read 4 Done Done Sent false
4611 2074 Acked Read 2 Done Done Sent false
4612 2075 Acked Write 8 Done Done Sent false
4613 2076 Acked Write 1 Done Done Sent false
4614 2077 Acked Read 7 Done Done Sent false
4615 2078 Acked Read 9 Done Done Sent false
4616 2079 Acked Flush 0 Done Done Sent false
4617 2080 Acked Read 10 Done Done Sent false
4618 2081 Acked Write 2 Done Done Sent false
4619 2082 Acked Write 9 Done Done Sent false
4620 2083 Acked Read 8 Done Done Sent false
4621 2084 Acked Write 6 Done Done Sent false
4622 2085 Acked Write 4 Done Done Sent false
4623 2086 Acked Write 1 Done Done Sent false
4624 2087 Acked Read 7 Done Done Sent false
4625 2088 Acked Read 6 Done Done Sent false
4626 2089 Acked Read 5 Done Done Sent false
4627 2090 Acked Write 9 Done Done Sent false
4628 2091 Acked Read 4 Done Done Sent false
4629 2092 Acked Read 3 Done Done Sent false
4630 2093 Acked Write 4 Done Done Sent false
4631 2094 Acked Write 3 Done Done Sent false
4632 2095 Acked Read 2 Done Done Sent false
4633 2096 Acked Read 4 Done Done Sent false
4634 2097 Acked Write 1 Done Done Sent false
4635 2098 Acked Write 7 Done Done Sent false
4636 2099 Acked Write 9 Done Done Sent false
4637 2100 Acked Write 10 Done Done Sent false
4638 2101 Acked Flush 0 Done Done Sent false
4639 2102 Acked Write 5 Done Done Sent false
4640 2103 Acked Read 4 Done Done Sent false
4641 2104 Acked Flush 0 Done Done Sent false
4642 2105 Acked Write 2 Done Done Sent false
4643 2106 Acked Read 3 Done Done Sent false
4644 2107 Acked Write 6 Done Done Sent false
4645 2108 Acked Read 8 Done Done Sent false
4646 2109 Acked Flush 0 Done Done Sent false
4647 2110 Acked Read 4 Done Done Sent false
4648 2111 Acked Flush 0 Done Done Sent false
4649 2112 Acked Read 5 Done Done Sent false
4650 2113 Acked Write 8 Done Done Sent false
4651 2114 Acked Flush 0 Done Done Sent false
4652 2115 Acked Read 4 Done Done Sent false
4653 2116 Acked Write 6 Done Done Sent false
4654 2117 Acked Read 10 Done Done Sent false
4655 2118 Acked Write 5 Done Done Sent false
4656 2119 Acked Flush 0 Done Done Sent false
4657 2120 Acked Write 9 Done Done Sent false
4658 2121 Acked Read 9 Done Done Sent false
4659 2122 Acked Read 8 Done Done Sent false
4660 2123 Acked Read 5 Done Done Sent false
4661 2124 Acked Read 5 Done Done Sent false
4662 2125 Acked Write 9 Done Done Sent false
4663 2126 Acked Read 8 Done Done Sent false
4664 2127 Acked Read 7 Done Done Sent false
4665 2128 Acked Read 6 Done Done Sent false
4666 2129 Acked Read 4 Done Done Sent false
4667 2130 Acked Read 9 Done Done Sent false
4668 2131 Acked Read 6 Done Done Sent false
4669 2132 Acked Write 7 Done Done Sent false
4670 2133 Acked Read 2 Done Done Sent false
4671 2134 Acked Write 4 Done Done Sent false
4672 2135 Acked Read 5 Done Done Sent false
4673 2136 Acked Read 5 Done Done Sent false
4674 2137 Acked Read 10 Done Done Sent false
4675 2138 Acked Read 6 Done Done Sent false
4676 2139 Acked Write 3 Done Done Sent false
4677 2140 Acked Read 7 Done Done Sent false
4678 2141 Acked Read 2 Done Done Sent false
4679 2142 Acked Write 8 Done Done Sent false
4680 2143 Acked Read 8 Done Done Sent false
4681 2144 Acked Read 10 Done Done Sent false
4682 2145 Acked Flush 0 Done Done Sent false
4683 2146 Acked Read 2 Done Done Sent false
4684 2147 Acked Read 10 Done Done Sent false
4685 2148 Acked Write 5 Done Done Sent false
4686 2149 Acked Write 5 Done Done Sent false
4687 2150 Acked Write 9 Done Done Sent false
4688 2151 Acked Write 9 Done Done Sent false
4689 2152 Acked Write 2 Done Done Sent false
4690 2153 Acked Write 7 Done Done Sent false
4691 2154 Acked Read 10 Done Done Sent false
4692 2155 Acked Read 3 Done Done Sent false
4693 2156 Acked Flush 0 Done Done Sent false
4694 2157 Acked Read 8 Done Done Sent false
4695 2158 Acked Write 1 Done Done Sent false
4696 2159 Acked Read 3 Done Done Sent false
4697 2160 Acked Read 3 Done Done Sent false
4698 2161 Acked Read 4 Done Done Sent false
4699 2162 Acked Read 3 Done Done Sent false
4700 2163 Acked Read 5 Done Done Sent false
4701 2164 Acked Write 5 Done Done Sent false
4702 2165 Acked Read 4 Done Done Sent false
4703 2166 Acked Write 4 Done Done Sent false
4704 2167 Acked Read 3 Done Done Sent false
4705 2168 Acked Write 8 Done Done Sent false
4706 2169 Acked Write 6 Done Done Sent false
4707 2170 Acked Write 8 Done Done Sent false
4708 2171 Acked Read 4 Done Done Sent false
4709 2172 Acked Read 3 Done Done Sent false
4710 2173 Acked Read 9 Done Done Sent false
4711 2174 Acked Read 7 Done Done Sent false
4712 2175 Acked Read 6 Done Done Sent false
4713 2176 Acked Write 4 Done Done Sent false
4714 2177 Acked Read 8 Done Done Sent false
4715 2178 Acked Read 2 Done Done Sent false
4716 2179 Acked Write 1 Done Done Sent false
4717 2180 Acked Write 9 Done Done Sent false
4718 2181 Acked Write 6 Done Done Sent false
4719 2182 Acked Read 5 Done Done Sent false
4720 2183 Acked Read 1 Done Done Sent false
4721 2184 Acked Read 7 Done Done Sent false
4722 2185 Acked Flush 0 Done Done Sent false
4723 2186 Acked Write 3 Done Done Sent false
4724 2187 Acked Read 8 Done Done Sent false
4725 2188 Acked Read 1 Done Done Sent false
4726 2189 Acked Write 9 Done Done Sent false
4727 2190 Acked Read 1 Done Done Sent false
4728 2191 Acked Read 7 Done Done Sent false
4729 2192 Acked Read 10 Done Done Sent false
4730 2193 Acked Read 8 Done Done Sent false
4731 2194 Acked Flush 0 Done Done Sent false
4732 2195 Acked Read 10 Done Done Sent false
4733 2196 Acked Flush 0 Done Done Sent false
4734 2197 Acked Write 1 Done Done Sent false
4735 2198 Acked Flush 0 Done Done Sent false
4736 2199 Acked Write 1 Done Done Sent false
4737 2200 Acked Write 3 Done Done Sent false
4738 2201 Acked Read 3 Done Done Sent false
4739 2202 Acked Flush 0 Done Done Sent false
4740 2203 Acked Write 6 Done Done Sent false
4741 2204 Acked Write 8 Done Done Sent false
4742 2205 Acked Read 3 Done Done Sent false
4743 2206 Acked Write 9 Done Done Sent false
4744 STATES DS:0 DS:1 DS:2 TOTAL
4745 Sent 0 0 301 301
4746 Done 301 301 0 602
4747 Skipped 0 0 0 0
4748 Error 0 0 0 0
4749 Last Flush: Some(JobId(2202)) Some(JobId(2202)) Some(JobId(1905))
4750 Downstairs last five completed: 1905 1904 1903 1902 1901
4751 Upstairs last five completed: 2206 2205 2204 2203 2202
4752 CLIENT: Up:0 ds:301 act:2
47532025-01-28T21:55:19.419ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
47542025-01-28T21:55:20.421ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
47552025-01-28T21:55:20.622ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
47562025-01-28T21:55:20.622ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59981) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
47572025-01-28T21:55:20.623ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
47582025-01-28T21:55:20.624ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
47592025-01-28T21:55:20.624ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
47602025-01-28T21:55:20.624ZINFOcrucible: send last flush ID to this DS: Some(JobId(1905)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
47612025-01-28T21:55:20.624ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
47622025-01-28T21:55:20.624ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1905)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
47632025-01-28T21:55:20.624ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1905)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
4764 ----------------------------------------------------------------
4765 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4766 Downstairs last five completed: 2207 2206 2205 2204 2203
4767 Upstairs last five completed: 2207 2206 2205 2204 2203
4768 CLIENT: Up:0 ds:0 act:3
4769 Replay: All jobs finished
4770 Main task has work to do, go find it
4771 got dsc Stop(1)
4772 stop 1
4773 Jan 28 21:55:22.865 INFO request completed, latency_us: 389, response_code: 204, uri: /stop/cid/1, [8820] Got stop action so:false kr:false
4774 method: POST, req_id: 2dcab679-9475-483d-99ba-a5c72f6382fd, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
47752025-01-28T21:55:22.866ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
4776 Jan 28 21:55:22.866 INFO request completed, latency_us: 138, response_code: 200, uri:{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:55:22.866524184Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
4777 /state/cid/1, method: GET, req_id: 5ee1aba5-7cf4-4ca8-b838-0dada088aad5, remote_addr{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:55:22.866542832Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
4778 : 127.0.0.1:36665, local_addr: 127.0.0.1:9998
47792025-01-28T21:55:22.866ZINFOcrucible: client task is exiting = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
4780 [8820] Exited with: ExitStatus(unix_wait_status(9))
4781 [8820][1] reports Exit
4782 Jan 28 21:55:26.868 INFO request completed, latency_us: 248, response_code: 200, uri: /state/cid/1, method: GET, req_id: c6c77388-e1fc-4c72-95d7-c6ca0beb8c74, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
4783 001/300 Read block 3618 len 16384
4784 002/300 Read block 3708 len 24576
4785 003/300 Read block 4547 len 32768
4786 004/300 Read block 451 len 12288
4787 005/300 Write block 460 len 8192 data: 3 3
4788 006/300 Write block 2723 len 32768 data: 2 2 2 2 2 3 3 4
4789 007/300 Read block 2660 len 36864
4790 008/300 Read block 2571 len 24576
4791 009/300 Write block 713 len 4096 data: 2
4792 010/300 Read block 3021 len 40960
4793 011/300 Flush
4794 012/300 Read block 419 len 16384
4795 013/300 Read block 3791 len 40960
4796 014/300 Read block 2644 len 24576
4797 015/300 Read block 3128 len 12288
4798 016/300 Write block 1739 len 8192 data: 3 2
4799 017/300 Write block 1692 len 36864 data: 2 2 2 2 2 3 3 3 4
4800 018/300 Read block 3634 len 20480
4801 019/300 Read block 4951 len 36864
4802 020/300 Read block 2777 len 12288
4803 021/300 Read block 765 len 24576
4804 022/300 Write block 2197 len 4096 data: 2
4805 023/300 Write block 687 len 20480 data: 2 2 3 3 3
4806 024/300 Read block 1519 len 32768
4807 025/300 Read block 1621 len 16384
4808 026/300 Read block 3157 len 20480
4809 027/300 Write block 881 len 36864 data: 2 2 2 2 3 3 3 3 3
4810 028/300 Flush
4811 029/300 Flush
4812 030/300 Write block 2195 len 28672 data: 2 2 3 2 2 2 3
4813 031/300 Write block 912 len 28672 data: 3 3 4 3 3 3 3
4814 032/300 Write block 3586 len 4096 data: 2
4815 033/300 Write block 2808 len 24576 data: 3 3 2 2 2 2
4816 034/300 Write block 1039 len 40960 data: 4 4 3 4 4 4 2 3 3 3
4817 035/300 Write block 4408 len 4096 data: 3
4818 036/300 Write block 1304 len 8192 data: 2 2
4819 037/300 Flush
4820 038/300 Write block 2408 len 40960 data: 3 3 3 3 3 3 3 2 2 2
4821 039/300 Read block 442 len 36864
4822 040/300 Flush
4823 041/300 Read block 2784 len 40960
4824 042/300 Read block 2002 len 32768
4825 043/300 Write block 3485 len 24576 data: 2 3 3 3 4 4
4826 044/300 Flush
4827 045/300 Write block 4740 len 16384 data: 2 2 2 2
4828 046/300 Flush
4829 047/300 Write block 3182 len 16384 data: 3 3 3 3
4830 048/300 Write block 1557 len 8192 data: 2 2
4831 049/300 Read block 2866 len 28672
4832 050/300 Write block 1664 len 28672 data: 3 2 2 2 2 3 3
4833 051/300 Write block 4537 len 4096 data: 2
4834 052/300 Flush
4835 053/300 Write block 1774 len 20480 data: 2 2 2 2 2
4836 054/300 Read block 637 len 28672
4837 055/300 Write block 2938 len 12288 data: 2 2 2
4838 056/300 Write block 170 len 28672 data: 2 2 2 2 2 2 2
4839 057/300 Read block 448 len 20480
4840 058/300 Write block 3477 len 12288 data: 2 2 2
4841 059/300 Read block 4391 len 24576
4842 060/300 Read block 4788 len 16384
4843 061/300 Read block 970 len 32768
4844 062/300 Read block 3777 len 36864
4845 063/300 Read block 1717 len 28672
4846 064/300 Flush
4847 065/300 Read block 3676 len 40960
4848 066/300 Write block 2751 len 24576 data: 3 3 2 2 2 2
4849 067/300 Read block 3568 len 24576
4850 068/300 Write block 1105 len 20480 data: 3 3 3 3 3
4851 069/300 Write block 3488 len 16384 data: 4 5 5 3
4852 070/300 Write block 916 len 16384 data: 4 4 4 3
4853 071/300 Write block 4563 len 40960 data: 2 3 4 4 3 3 3 3 3 2
4854 072/300 Read block 330 len 32768
4855 073/300 Write block 1647 len 4096 data: 2
4856 074/300 Read block 816 len 28672
4857 075/300 Flush
4858 076/300 Write block 3800 len 12288 data: 3 3 2
4859 077/300 Read block 2257 len 4096
4860 078/300 Write block 899 len 36864 data: 2 2 2 2 2 2 3 3 3
4861 079/300 Write block 4393 len 24576 data: 2 2 2 2 2 2
4862 080/300 Flush
4863 081/300 Read block 373 len 4096
4864 082/300 Write block 3905 len 20480 data: 2 3 3 3 3
4865 083/300 Read block 2525 len 8192
4866 084/300 Read block 1699 len 4096
4867 085/300 Flush
4868 086/300 Write block 4876 len 12288 data: 4 5 5
4869 087/300 Write block 4412 len 12288 data: 3 3 4
4870 088/300 Flush
4871 089/300 Write block 4115 len 20480 data: 3 3 3 4 4
4872 090/300 Flush
4873 091/300 Read block 4750 len 28672
4874 092/300 Flush
4875 093/300 Write block 3586 len 36864 data: 3 2 2 2 2 2 2 2 2
4876 094/300 Write block 3086 len 32768 data: 4 3 3 2 2 2 2 2
4877 095/300 Read block 1494 len 36864
4878 096/300 Flush
4879 097/300 Read block 3080 len 36864
4880 098/300 Flush
4881 099/300 Write block 2828 len 4096 data: 3
4882 100/300 Flush
4883 101/300 Read block 3377 len 24576
4884 102/300 Read block 1452 len 28672
4885 103/300 Read block 4924 len 36864
4886 104/300 Read block 2835 len 4096
4887 105/300 Write block 3569 len 20480 data: 2 2 2 2 2
4888 106/300 Read block 2244 len 12288
4889 107/300 Read block 1495 len 32768
4890 108/300 Read block 4651 len 20480
4891 109/300 Read block 2523 len 12288
4892 110/300 Read block 2104 len 16384
4893 111/300 Read block 3837 len 28672
4894 112/300 Read block 3503 len 24576
4895 113/300 Write block 3059 len 12288 data: 4 4 4
4896 114/300 Flush
4897 115/300 Read block 4260 len 12288
4898 116/300 Write block 2737 len 8192 data: 3 3
4899 117/300 Read block 624 len 24576
4900 118/300 Write block 4263 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4901 119/300 Read block 2391 len 8192
4902 120/300 Read block 1226 len 20480
4903 121/300 Write block 599 len 24576 data: 3 3 3 3 3 4
4904 122/300 Write block 4519 len 36864 data: 2 2 2 3 3 4 3 3 3
4905 123/300 Read block 995 len 32768
4906 124/300 Write block 2818 len 8192 data: 2 2
4907 125/300 Read block 2559 len 36864
4908 126/300 Read block 4411 len 12288
4909 127/300 Write block 1324 len 16384 data: 2 3 2 2
4910 128/300 Write block 3077 len 32768 data: 3 3 4 4 4 4 4 3
4911 129/300 Write block 1456 len 36864 data: 2 2 3 3 3 2 2 2 2
4912 130/300 Read block 2857 len 16384
4913 131/300 Read block 799 len 28672
4914 132/300 Write block 3547 len 28672 data: 2 2 2 2 2 3 3
4915 133/300 Read block 1205 len 4096
4916 134/300 Read block 4851 len 28672
4917 135/300 Read block 1167 len 8192
4918 136/300 Flush
4919 137/300 Read block 1558 len 16384
4920 138/300 Read block 4049 len 16384
4921 139/300 Write block 3025 len 8192 data: 3 3
4922 140/300 Write block 4048 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4923 141/300 Write block 1044 len 12288 data: 5 3 4
4924 142/300 Read block 2177 len 32768
4925 143/300 Read block 2428 len 20480
4926 144/300 Write block 1178 len 20480 data: 2 4 4 3 3
4927 145/300 Read block 70 len 8192
4928 146/300 Write block 4667 len 32768 data: 3 2 2 2 2 2 2 2
4929 147/300 Read block 2578 len 36864
4930 148/300 Read block 2570 len 4096
4931 149/300 Read block 2837 len 32768
4932 150/300 Write block 1444 len 4096 data: 3
4933 151/300 Read block 4534 len 32768
4934 152/300 Read block 3325 len 24576
4935 153/300 Read block 4280 len 24576
4936 154/300 Read block 1210 len 32768
4937 155/300 Read block 1171 len 24576
4938 156/300 Write block 2076 len 32768 data: 2 2 2 2 2 2 2 2
4939 157/300 Write block 2618 len 12288 data: 5 5 5
4940 158/300 Read block 4874 len 32768
4941 159/300 Read block 4173 len 40960
4942 160/300 Write block 3028 len 4096 data: 2
4943 161/300 Read block 3410 len 28672
4944 162/300 Write block 2302 len 36864 data: 2 2 2 2 2 2 2 2 2
4945 163/300 Write block 2159 len 4096 data: 6
4946 164/300 Read block 959 len 36864
4947 165/300 Read block 1214 len 4096
4948 166/300 Read block 4498 len 24576
4949 167/300 Read block 3938 len 24576
4950 168/300 Flush
4951 169/300 Write block 1819 len 12288 data: 3 3 3
4952 170/300 Read block 3964 len 16384
4953 171/300 Write block 1197 len 36864 data: 2 2 2 2 2 3 3 4 3
4954 172/300 Write block 2481 len 16384 data: 3 2 2 2
4955 173/300 Write block 373 len 32768 data: 2 2 2 3 3 2 2 2
4956 174/300 Write block 1190 len 4096 data: 2
4957 175/300 Write block 4529 len 36864 data: 2 2 2 3 3 3 3 3 3
4958 176/300 Write block 4380 len 24576 data: 2 3 3 3 3 3
4959 177/300 Read block 4713 len 4096
4960 178/300 Write block 4476 len 28672 data: 2 3 4 5 6 6 6
4961 179/300 Read block 4990 len 20480
4962 180/300 Read block 4480 len 8192
4963 181/300 Read block 3001 len 8192
4964 182/300 Read block 4100 len 12288
4965 183/300 Write block 3265 len 4096 data: 2
4966 184/300 Read block 4065 len 4096
4967 185/300 Write block 4658 len 40960 data: 3 4 4 4 4 5 5 5 5 4
4968 186/300 Read block 860 len 28672
4969 187/300 Read block 469 len 20480
4970 188/300 Read block 1034 len 32768
4971 189/300 Read block 4222 len 8192
4972 190/300 Read block 3021 len 8192
4973 191/300 Write block 1628 len 36864 data: 2 3 3 4 3 3 2 2 3
4974 192/300 Read block 3930 len 12288
4975 193/300 Read block 2686 len 20480
4976 194/300 Write block 4251 len 36864 data: 3 3 3 2 2 2 2 2 2
4977 195/300 Write block 1579 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4978 196/300 Write block 4793 len 32768 data: 2 3 3 2 2 2 2 2
4979 197/300 Read block 2618 len 28672
4980 198/300 Flush
4981 199/300 Flush
4982 200/300 Write block 4163 len 40960 data: 2 3 3 4 4 2 2 2 2 2
4983 201/300 Write block 1420 len 28672 data: 2 2 2 2 2 3 3
4984 202/300 Flush
4985 203/300 Write block 616 len 20480 data: 2 2 2 2 2
4986 204/300 Read block 424 len 12288
4987 205/300 Read block 4147 len 4096
4988 206/300 Read block 4828 len 4096
4989 207/300 Read block 2959 len 20480
4990 208/300 Read block 2382 len 4096
4991 209/300 Read block 1081 len 32768
4992 210/300 Read block 1432 len 24576
4993 211/300 Flush
4994 212/300 Read block 4967 len 40960
4995 213/300 Write block 4905 len 28672 data: 2 2 2 2 2 2 2
4996 214/300 Read block 3484 len 12288
4997 215/300 Read block 979 len 16384
4998 216/300 Read block 161 len 12288
4999 217/300 Write block 2945 len 24576 data: 2 2 2 2 2 2
5000 218/300 Write block 2827 len 4096 data: 3
5001 219/300 Read block 3460 len 20480
5002 220/300 Write block 1671 len 24576 data: 3 2 2 2 2 2
5003 221/300 Write block 1719 len 16384 data: 3 3 3 3
5004 222/300 Read block 2949 len 40960
5005 223/300 Read block 1619 len 28672
5006 224/300 Read block 1551 len 40960
5007 225/300 Write block 366 len 20480 data: 3 3 3 2 2
5008 226/300 Write block 4856 len 32768 data: 2 2 2 2 2 2 2 3
5009 227/300 Read block 1412 len 16384
5010 228/300 Write block 301 len 16384 data: 4 3 3 3
5011 229/300 Write block 688 len 24576 data: 3 4 4 4 3 3
5012 230/300 Read block 1203 len 28672
5013 231/300 Read block 3210 len 20480
5014 232/300 Read block 1873 len 32768
5015 233/300 Read block 4301 len 16384
5016 234/300 Write block 2347 len 40960 data: 3 2 2 2 2 2 2 2 2 2
5017 235/300 Read block 2734 len 36864
5018 236/300 Write block 4450 len 20480 data: 2 2 2 2 2
5019 237/300 Write block 3209 len 32768 data: 2 2 2 2 3 3 3 3
5020 238/300 Read block 4866 len 16384
5021 239/300 Flush
5022 240/300 Read block 4308 len 16384
5023 241/300 Read block 355 len 8192
5024 242/300 Write block 3470 len 8192 data: 2 2
5025 243/300 Read block 2310 len 12288
5026 244/300 Read block 164 len 20480
5027 245/300 Read block 4051 len 36864
5028 246/300 Read block 1843 len 12288
5029 247/300 Read block 3129 len 4096
5030 248/300 Write block 2070 len 16384 data: 3 2 2 2
5031 249/300 Read block 155 len 36864
5032 250/300 Read block 2233 len 16384
5033 251/300 Write block 1969 len 24576 data: 2 2 2 2 2 2
5034 252/300 Read block 3851 len 36864
5035 253/300 Write block 636 len 8192 data: 3 3
5036 254/300 Read block 571 len 32768
5037 255/300 Write block 2927 len 36864 data: 3 3 3 3 3 3 3 3 2
5038 256/300 Read block 1308 len 32768
5039 257/300 Read block 1346 len 32768
5040 258/300 Write block 1917 len 36864 data: 3 3 4 4 4 3 3 2 2
5041 259/300 Read block 876 len 36864
5042 260/300 Write block 2635 len 36864 data: 3 3 3 2 2 2 2 2 2
5043 261/300 Read block 1989 len 16384
5044 262/300 Write block 4984 len 20480 data: 3 3 3 3 3
5045 263/300 Read block 2601 len 28672
5046 264/300 Write block 545 len 16384 data: 4 4 5 5
5047 265/300 Write block 2302 len 20480 data: 3 3 3 3 3
5048 266/300 Flush
5049 267/300 Read block 1954 len 8192
5050 268/300 Flush
5051 269/300 Write block 721 len 8192 data: 2 2
5052 270/300 Flush
5053 271/300 Write block 644 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5054 272/300 Flush
5055 273/300 Write block 4317 len 32768 data: 2 2 2 2 2 2 2 2
5056 274/300 Write block 1118 len 28672 data: 3 3 3 3 3 2 2
5057 275/300 Read block 2712 len 16384
5058 276/300 Write block 398 len 28672 data: 2 2 2 2 2 2 2
5059 277/300 Read block 503 len 32768
5060 278/300 Write block 349 len 12288 data: 3 3 3
5061 279/300 Write block 2390 len 8192 data: 2 2
5062 280/300 Write block 4041 len 36864 data: 2 2 2 2 2 2 2 3 3
5063 281/300 Write block 1078 len 40960 data: 3 3 3 3 3 3 3 3 3 2
5064 282/300 Write block 4486 len 28672 data: 5 3 3 3 3 2 2
5065 283/300 Flush
5066 284/300 Flush
5067 285/300 Write block 1457 len 12288 data: 3 4 4
5068 286/300 Read block 1971 len 12288
5069 287/300 Read block 2968 len 8192
5070 288/300 Flush
5071 289/300 Write block 1403 len 12288 data: 2 2 2
5072 290/300 Read block 3574 len 40960
5073 291/300 Read block 4941 len 8192
5074 292/300 Write block 944 len 40960 data: 3 3 2 2 2 2 2 2 2 2
5075 293/300 Read block 4896 len 8192
5076 294/300 Flush
5077 295/300 Read block 4976 len 12288
5078 296/300 Read block 1084 len 4096
5079 297/300 Read block 3612 len 36864
5080 298/300 Write block 3925 len 28672 data: 3 2 2 2 2 2 3
5081 299/300 Read block 3844 len 20480
5082 300/300 Read block 2652 len 32768
5083 301/300 Write block 3555 len 28672 data: 3 3 3 3 4 3 3
5084 Main task has work to do, go find it
5085 got dsc Start(1)
5086 start 1
5087 Jan 28 21:55:27.108 INFO request completed, latency_us: 267, response_code: 204, uri: /start/cid/1, method: POST, req_id: 25b8c3c2-834a-4174-bf5c-c189db1788a5, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
5088 Replay: started 1, returned:Ok(())
5089 ----------------------------------------------------------------
5090 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5091 States: Active Offline Active
5092 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5093 2208 Acked Read 4 Done Sent Done false
5094 2209 Acked Read 6 Done Sent Done false
5095 2210 Acked Read 8 Done Sent Done false
5096 2211 Acked Read 3 Done Sent Done false
5097 2212 Acked Write 2 Done Sent Done false
5098 2213 Acked Write 8 Done Sent Done false
5099 2214 Acked Read 9 Done Sent Done false
5100 2215 Acked Read 6 Done Sent Done false
5101 2216 Acked Write 1 Done Sent Done false
5102 2217 Acked Read 10 Done Sent Done false
5103 2218 Acked Flush 0 Done Sent Done false
5104 2219 Acked Read 4 Done Sent Done false
5105 2220 Acked Read 10 Done Sent Done false
5106 2221 Acked Read 6 Done Sent Done false
5107 2222 Acked Read 3 Done Sent Done false
5108 2223 Acked Write 2 Done Sent Done false
5109 2224 Acked Write 9 Done Sent Done false
5110 2225 Acked Read 5 Done Sent Done false
5111 2226 Acked Read 9 Done Sent Done false
5112 2227 Acked Read 3 Done Sent Done false
5113 2228 Acked Read 6 Done Sent Done false
5114 2229 Acked Write 1 Done Sent Done false
5115 2230 Acked Write 5 Done Sent Done false
5116 2231 Acked Read 8 Done Sent Done false
5117 2232 Acked Read 4 Done Sent Done false
5118 2233 Acked Read 5 Done Sent Done false
5119 2234 Acked Write 9 Done Sent Done false
5120 2235 Acked Flush 0 Done Sent Done false
5121 2236 Acked Flush 0 Done Sent Done false
5122 2237 Acked Write 7 Done Sent Done false
5123 2238 Acked Write 7 Done Sent Done false
5124 2239 Acked Write 1 Done Sent Done false
5125 2240 Acked Write 6 Done Sent Done false
5126 2241 Acked Write 10 Done Sent Done false
5127 2242 Acked Write 1 Done Sent Done false
5128 2243 Acked Write 2 Done Sent Done false
5129 2244 Acked Flush 0 Done Sent Done false
5130 2245 Acked Write 10 Done Sent Done false
5131 2246 Acked Read 9 Done Sent Done false
5132 2247 Acked Flush 0 Done Sent Done false
5133 2248 Acked Read 10 Done Sent Done false
5134 2249 Acked Read 8 Done Sent Done false
5135 2250 Acked Write 6 Done Sent Done false
5136 2251 Acked Flush 0 Done Sent Done false
5137 2252 Acked Write 4 Done Sent Done false
5138 2253 Acked Flush 0 Done Sent Done false
5139 2254 Acked Write 4 Done Sent Done false
5140 2255 Acked Write 2 Done Sent Done false
5141 2256 Acked Read 7 Done Sent Done false
5142 2257 Acked Write 7 Done Sent Done false
5143 2258 Acked Write 1 Done Sent Done false
5144 2259 Acked Flush 0 Done Sent Done false
5145 2260 Acked Write 5 Done Sent Done false
5146 2261 Acked Read 7 Done Sent Done false
5147 2262 Acked Write 3 Done Sent Done false
5148 2263 Acked Write 7 Done Sent Done false
5149 2264 Acked Read 5 Done Sent Done false
5150 2265 Acked Write 3 Done Sent Done false
5151 2266 Acked Read 6 Done Sent Done false
5152 2267 Acked Read 4 Done Sent Done false
5153 2268 Acked Read 8 Done Sent Done false
5154 2269 Acked Read 9 Done Sent Done false
5155 2270 Acked Read 7 Done Sent Done false
5156 2271 Acked Flush 0 Done Sent Done false
5157 2272 Acked Read 10 Done Sent Done false
5158 2273 Acked Write 6 Done Sent Done false
5159 2274 Acked Read 6 Done Sent Done false
5160 2275 Acked Write 5 Done Sent Done false
5161 2276 Acked Write 4 Done Sent Done false
5162 2277 Acked Write 4 Done Sent Done false
5163 2278 Acked Write 10 Done Sent Done false
5164 2279 Acked Read 8 Done Sent Done false
5165 2280 Acked Write 1 Done Sent Done false
5166 2281 Acked Read 7 Done Sent Done false
5167 2282 Acked Flush 0 Done Sent Done false
5168 2283 Acked Write 3 Done Sent Done false
5169 2284 Acked Read 1 Done Sent Done false
5170 2285 Acked Write 9 Done Sent Done false
5171 2286 Acked Write 6 Done Sent Done false
5172 2287 Acked Flush 0 Done Sent Done false
5173 2288 Acked Read 1 Done Sent Done false
5174 2289 Acked Write 5 Done Sent Done false
5175 2290 Acked Read 2 Done Sent Done false
5176 2291 Acked Read 1 Done Sent Done false
5177 2292 Acked Flush 0 Done Sent Done false
5178 2293 Acked Write 3 Done Sent Done false
5179 2294 Acked Write 3 Done Sent Done false
5180 2295 Acked Flush 0 Done Sent Done false
5181 2296 Acked Write 5 Done Sent Done false
5182 2297 Acked Flush 0 Done Sent Done false
5183 2298 Acked Read 7 Done Sent Done false
5184 2299 Acked Flush 0 Done Sent Done false
5185 2300 Acked Write 9 Done Sent Done false
5186 2301 Acked Write 8 Done Sent Done false
5187 2302 Acked Read 9 Done Sent Done false
5188 2303 Acked Flush 0 Done Sent Done false
5189 2304 Acked Read 9 Done Sent Done false
5190 2305 Acked Flush 0 Done Sent Done false
5191 2306 Acked Write 1 Done Sent Done false
5192 2307 Acked Flush 0 Done Sent Done false
5193 2308 Acked Read 6 Done Sent Done false
5194 2309 Acked Read 7 Done Sent Done false
5195 2310 Acked Read 9 Done Sent Done false
5196 2311 Acked Read 1 Done Sent Done false
5197 2312 Acked Write 5 Done Sent Done false
5198 2313 Acked Read 3 Done Sent Done false
5199 2314 Acked Read 8 Done Sent Done false
5200 2315 Acked Read 5 Done Sent Done false
5201 2316 Acked Read 3 Done Sent Done false
5202 2317 Acked Read 4 Done Sent Done false
5203 2318 Acked Read 7 Done Sent Done false
5204 2319 Acked Read 6 Done Sent Done false
5205 2320 Acked Write 3 Done Sent Done false
5206 2321 Acked Flush 0 Done Sent Done false
5207 2322 Acked Read 3 Done Sent Done false
5208 2323 Acked Write 2 Done Sent Done false
5209 2324 Acked Read 6 Done Sent Done false
5210 2325 Acked Write 10 Done Sent Done false
5211 2326 Acked Read 2 Done Sent Done false
5212 2327 Acked Read 5 Done Sent Done false
5213 2328 Acked Write 6 Done Sent Done false
5214 2329 Acked Write 9 Done Sent Done false
5215 2330 Acked Read 8 Done Sent Done false
5216 2331 Acked Write 2 Done Sent Done false
5217 2332 Acked Read 9 Done Sent Done false
5218 2333 Acked Read 3 Done Sent Done false
5219 2334 Acked Write 4 Done Sent Done false
5220 2335 Acked Write 8 Done Sent Done false
5221 2336 Acked Write 9 Done Sent Done false
5222 2337 Acked Read 4 Done Sent Done false
5223 2338 Acked Read 7 Done Sent Done false
5224 2339 Acked Write 7 Done Sent Done false
5225 2340 Acked Read 1 Done Sent Done false
5226 2341 Acked Read 7 Done Sent Done false
5227 2342 Acked Read 2 Done Sent Done false
5228 2343 Acked Flush 0 Done Sent Done false
5229 2344 Acked Read 4 Done Sent Done false
5230 2345 Acked Read 4 Done Sent Done false
5231 2346 Acked Write 2 Done Sent Done false
5232 2347 Acked Write 10 Done Sent Done false
5233 2348 Acked Write 3 Done Sent Done false
5234 2349 Acked Read 8 Done Sent Done false
5235 2350 Acked Read 5 Done Sent Done false
5236 2351 Acked Write 5 Done Sent Done false
5237 2352 Acked Read 2 Done Sent Done false
5238 2353 Acked Write 8 Done Sent Done false
5239 2354 Acked Read 9 Done Sent Done false
5240 2355 Acked Read 1 Done Sent Done false
5241 2356 Acked Read 8 Done Sent Done false
5242 2357 Acked Write 1 Done Sent Done false
5243 2358 Acked Read 8 Done Sent Done false
5244 2359 Acked Read 6 Done Sent Done false
5245 2360 Acked Read 6 Done Sent Done false
5246 2361 Acked Read 8 Done Sent Done false
5247 2362 Acked Read 6 Done Sent Done false
5248 2363 Acked Write 8 Done Sent Done false
5249 2364 Acked Write 3 Done Sent Done false
5250 2365 Acked Read 8 Done Sent Done false
5251 2366 Acked Read 10 Done Sent Done false
5252 2367 Acked Write 1 Done Sent Done false
5253 2368 Acked Read 7 Done Sent Done false
5254 2369 Acked Write 9 Done Sent Done false
5255 2370 Acked Write 1 Done Sent Done false
5256 2371 Acked Read 9 Done Sent Done false
5257 2372 Acked Read 1 Done Sent Done false
5258 2373 Acked Read 6 Done Sent Done false
5259 2374 Acked Read 6 Done Sent Done false
5260 2375 Acked Flush 0 Done Sent Done false
5261 2376 Acked Write 3 Done Sent Done false
5262 2377 Acked Read 4 Done Sent Done false
5263 2378 Acked Write 9 Done Sent Done false
5264 2379 Acked Write 4 Done Sent Done false
5265 2380 Acked Write 8 Done Sent Done false
5266 2381 Acked Write 1 Done Sent Done false
5267 2382 Acked Write 9 Done Sent Done false
5268 2383 Acked Write 6 Done Sent Done false
5269 2384 Acked Read 1 Done Sent Done false
5270 2385 Acked Write 7 Done Sent Done false
5271 2386 Acked Read 5 Done Sent Done false
5272 2387 Acked Read 2 Done Sent Done false
5273 2388 Acked Read 2 Done Sent Done false
5274 2389 Acked Read 3 Done Sent Done false
5275 2390 Acked Write 1 Done Sent Done false
5276 2391 Acked Read 1 Done Sent Done false
5277 2392 Acked Write 10 Done Sent Done false
5278 2393 Acked Read 7 Done Sent Done false
5279 2394 Acked Read 5 Done Sent Done false
5280 2395 Acked Read 8 Done Sent Done false
5281 2396 Acked Read 2 Done Sent Done false
5282 2397 Acked Read 2 Done Sent Done false
5283 2398 Acked Write 9 Done Sent Done false
5284 2399 Acked Read 3 Done Sent Done false
5285 2400 Acked Read 5 Done Sent Done false
5286 2401 Acked Write 9 Done Sent Done false
5287 2402 Acked Write 10 Done Sent Done false
5288 2403 Acked Write 8 Done Sent Done false
5289 2404 Acked Read 7 Done Sent Done false
5290 2405 Acked Flush 0 Done Sent Done false
5291 2406 Acked Flush 0 Done Sent Done false
5292 2407 Acked Write 10 Done Sent Done false
5293 2408 Acked Write 7 Done Sent Done false
5294 2409 Acked Flush 0 Done Sent Done false
5295 2410 Acked Write 5 Done Sent Done false
5296 2411 Acked Read 3 Done Sent Done false
5297 2412 Acked Read 1 Done Sent Done false
5298 2413 Acked Read 1 Done Sent Done false
5299 2414 Acked Read 5 Done Sent Done false
5300 2415 Acked Read 1 Done Sent Done false
5301 2416 Acked Read 8 Done Sent Done false
5302 2417 Acked Read 6 Done Sent Done false
5303 2418 Acked Flush 0 Done Sent Done false
5304 2419 Acked Read 10 Done Sent Done false
5305 2420 Acked Write 7 Done Sent Done false
5306 2421 Acked Read 3 Done Sent Done false
5307 2422 Acked Read 4 Done Sent Done false
5308 2423 Acked Read 3 Done Sent Done false
5309 2424 Acked Write 6 Done Sent Done false
5310 2425 Acked Write 1 Done Sent Done false
5311 2426 Acked Read 5 Done Sent Done false
5312 2427 Acked Write 6 Done Sent Done false
5313 2428 Acked Write 4 Done Sent Done false
5314 2429 Acked Read 10 Done Sent Done false
5315 2430 Acked Read 7 Done Sent Done false
5316 2431 Acked Read 10 Done Sent Done false
5317 2432 Acked Write 5 Done Sent Done false
5318 2433 Acked Write 8 Done Sent Done false
5319 2434 Acked Read 4 Done Sent Done false
5320 2435 Acked Write 4 Done Sent Done false
5321 2436 Acked Write 6 Done Sent Done false
5322 2437 Acked Read 7 Done Sent Done false
5323 2438 Acked Read 5 Done Sent Done false
5324 2439 Acked Read 8 Done Sent Done false
5325 2440 Acked Read 4 Done Sent Done false
5326 2441 Acked Write 10 Done Sent Done false
5327 2442 Acked Read 9 Done Sent Done false
5328 2443 Acked Write 5 Done Sent Done false
5329 2444 Acked Write 8 Done Sent Done false
5330 2445 Acked Read 4 Done Sent Done false
5331 2446 Acked Flush 0 Done Sent Done false
5332 2447 Acked Read 4 Done Sent Done false
5333 2448 Acked Read 2 Done Sent Done false
5334 2449 Acked Write 2 Done Sent Done false
5335 2450 Acked Read 3 Done Sent Done false
5336 2451 Acked Read 5 Done Sent Done false
5337 2452 Acked Read 9 Done Sent Done false
5338 2453 Acked Read 3 Done Sent Done false
5339 2454 Acked Read 1 Done Sent Done false
5340 2455 Acked Write 4 Done Sent Done false
5341 2456 Acked Read 9 Done Sent Done false
5342 2457 Acked Read 4 Done Sent Done false
5343 2458 Acked Write 6 Done Sent Done false
5344 2459 Acked Read 9 Done Sent Done false
5345 2460 Acked Write 2 Done Sent Done false
5346 2461 Acked Read 8 Done Sent Done false
5347 2462 Acked Write 9 Done Sent Done false
5348 2463 Acked Read 8 Done Sent Done false
5349 2464 Acked Read 8 Done Sent Done false
5350 2465 Acked Write 9 Done Sent Done false
5351 2466 Acked Read 9 Done Sent Done false
5352 2467 Acked Write 9 Done Sent Done false
5353 2468 Acked Read 4 Done Sent Done false
5354 2469 Acked Write 5 Done Sent Done false
5355 2470 Acked Read 7 Done Sent Done false
5356 2471 Acked Write 4 Done Sent Done false
5357 2472 Acked Write 5 Done Sent Done false
5358 2473 Acked Flush 0 Done Sent Done false
5359 2474 Acked Read 2 Done Sent Done false
5360 2475 Acked Flush 0 Done Sent Done false
5361 2476 Acked Write 2 Done Sent Done false
5362 2477 Acked Flush 0 Done Sent Done false
5363 2478 Acked Write 10 Done Sent Done false
5364 2479 Acked Flush 0 Done Sent Done false
5365 2480 Acked Write 8 Done Sent Done false
5366 2481 Acked Write 7 Done Sent Done false
5367 2482 Acked Read 4 Done Sent Done false
5368 2483 Acked Write 7 Done Sent Done false
5369 2484 Acked Read 8 Done Sent Done false
5370 2485 Acked Write 3 Done Sent Done false
5371 2486 Acked Write 2 Done Sent Done false
5372 2487 Acked Write 9 Done Sent Done false
5373 2488 Acked Write 10 Done Sent Done false
5374 2489 Acked Write 7 Done Sent Done false
5375 2490 Acked Flush 0 Done Sent Done false
5376 2491 Acked Flush 0 Done Sent Done false
5377 2492 Acked Write 3 Done Sent Done false
5378 2493 Acked Read 3 Done Sent Done false
5379 2494 Acked Read 2 Done Sent Done false
5380 2495 Acked Flush 0 Done Sent Done false
5381 2496 Acked Write 3 Done Sent Done false
5382 2497 Acked Read 10 Done Sent Done false
5383 2498 Acked Read 2 Done Sent Done false
5384 2499 Acked Write 10 Done Sent Done false
5385 2500 Acked Read 2 Done Sent Done false
5386 2501 Acked Flush 0 Done Sent Done false
5387 2502 Acked Read 3 Done Sent Done false
5388 2503 Acked Read 1 Done Sent Done false
5389 2504 Acked Read 9 Done Sent Done false
5390 2505 Acked Write 7 Done Sent Done false
5391 2506 Acked Read 5 Done Sent Done false
5392 2507 Acked Read 8 Done Sent Done false
5393 2508 Acked Write 7 Done Sent Done false
5394 STATES DS:0 DS:1 DS:2 TOTAL
5395 Sent 0 301 0 301
5396 Done 301 0 301 602
5397 Skipped 0 0 0 0
5398 Error 0 0 0 0
5399 Last Flush: Some(JobId(2501)) Some(JobId(2207)) Some(JobId(2501))
5400 Downstairs last five completed: 2207 2206 2205 2204 2203
5401 Upstairs last five completed: 2508 2507 2506 2505 2504
5402 CLIENT: Up:0 ds:301 act:2
54032025-01-28T21:55:27.610ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
54042025-01-28T21:55:28.432ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
54052025-01-28T21:55:29.434ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
54062025-01-28T21:55:30.434ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
5407 [8820] got start action
5408 [8820] I am going to restart
5409 Starting downstairs at port 8820
5410 Make output file at "/tmp/dsc/downstairs-8820.txt"
5411 [8820][1] reports Starting
5412 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1207)
5413 [8820][1] reports Running
5414 ----------------------------------------------------------------
5415 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5416 States: Active Offline Active
5417 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5418 2208 Acked Read 4 Done Sent Done false
5419 2209 Acked Read 6 Done Sent Done false
5420 2210 Acked Read 8 Done Sent Done false
5421 2211 Acked Read 3 Done Sent Done false
5422 2212 Acked Write 2 Done Sent Done false
5423 2213 Acked Write 8 Done Sent Done false
5424 2214 Acked Read 9 Done Sent Done false
5425 2215 Acked Read 6 Done Sent Done false
5426 2216 Acked Write 1 Done Sent Done false
5427 2217 Acked Read 10 Done Sent Done false
5428 2218 Acked Flush 0 Done Sent Done false
5429 2219 Acked Read 4 Done Sent Done false
5430 2220 Acked Read 10 Done Sent Done false
5431 2221 Acked Read 6 Done Sent Done false
5432 2222 Acked Read 3 Done Sent Done false
5433 2223 Acked Write 2 Done Sent Done false
5434 2224 Acked Write 9 Done Sent Done false
5435 2225 Acked Read 5 Done Sent Done false
5436 2226 Acked Read 9 Done Sent Done false
5437 2227 Acked Read 3 Done Sent Done false
5438 2228 Acked Read 6 Done Sent Done false
5439 2229 Acked Write 1 Done Sent Done false
5440 2230 Acked Write 5 Done Sent Done false
5441 2231 Acked Read 8 Done Sent Done false
5442 2232 Acked Read 4 Done Sent Done false
5443 2233 Acked Read 5 Done Sent Done false
5444 2234 Acked Write 9 Done Sent Done false
5445 2235 Acked Flush 0 Done Sent Done false
5446 2236 Acked Flush 0 Done Sent Done false
5447 2237 Acked Write 7 Done Sent Done false
5448 2238 Acked Write 7 Done Sent Done false
5449 2239 Acked Write 1 Done Sent Done false
5450 2240 Acked Write 6 Done Sent Done false
5451 2241 Acked Write 10 Done Sent Done false
5452 2242 Acked Write 1 Done Sent Done false
5453 2243 Acked Write 2 Done Sent Done false
5454 2244 Acked Flush 0 Done Sent Done false
5455 2245 Acked Write 10 Done Sent Done false
5456 2246 Acked Read 9 Done Sent Done false
5457 2247 Acked Flush 0 Done Sent Done false
5458 2248 Acked Read 10 Done Sent Done false
5459 2249 Acked Read 8 Done Sent Done false
5460 2250 Acked Write 6 Done Sent Done false
5461 2251 Acked Flush 0 Done Sent Done false
5462 2252 Acked Write 4 Done Sent Done false
5463 2253 Acked Flush 0 Done Sent Done false
5464 2254 Acked Write 4 Done Sent Done false
5465 2255 Acked Write 2 Done Sent Done false
5466 2256 Acked Read 7 Done Sent Done false
5467 2257 Acked Write 7 Done Sent Done false
5468 2258 Acked Write 1 Done Sent Done false
5469 2259 Acked Flush 0 Done Sent Done false
5470 2260 Acked Write 5 Done Sent Done false
5471 2261 Acked Read 7 Done Sent Done false
5472 2262 Acked Write 3 Done Sent Done false
5473 2263 Acked Write 7 Done Sent Done false
5474 2264 Acked Read 5 Done Sent Done false
5475 2265 Acked Write 3 Done Sent Done false
5476 2266 Acked Read 6 Done Sent Done false
5477 2267 Acked Read 4 Done Sent Done false
5478 2268 Acked Read 8 Done Sent Done false
5479 2269 Acked Read 9 Done Sent Done false
5480 2270 Acked Read 7 Done Sent Done false
5481 2271 Acked Flush 0 Done Sent Done false
5482 2272 Acked Read 10 Done Sent Done false
5483 2273 Acked Write 6 Done Sent Done false
5484 2274 Acked Read 6 Done Sent Done false
5485 2275 Acked Write 5 Done Sent Done false
5486 2276 Acked Write 4 Done Sent Done false
5487 2277 Acked Write 4 Done Sent Done false
5488 2278 Acked Write 10 Done Sent Done false
5489 2279 Acked Read 8 Done Sent Done false
5490 2280 Acked Write 1 Done Sent Done false
5491 2281 Acked Read 7 Done Sent Done false
5492 2282 Acked Flush 0 Done Sent Done false
5493 2283 Acked Write 3 Done Sent Done false
5494 2284 Acked Read 1 Done Sent Done false
5495 2285 Acked Write 9 Done Sent Done false
5496 2286 Acked Write 6 Done Sent Done false
5497 2287 Acked Flush 0 Done Sent Done false
5498 2288 Acked Read 1 Done Sent Done false
5499 2289 Acked Write 5 Done Sent Done false
5500 2290 Acked Read 2 Done Sent Done false
5501 2291 Acked Read 1 Done Sent Done false
5502 2292 Acked Flush 0 Done Sent Done false
5503 2293 Acked Write 3 Done Sent Done false
5504 2294 Acked Write 3 Done Sent Done false
5505 2295 Acked Flush 0 Done Sent Done false
5506 2296 Acked Write 5 Done Sent Done false
5507 2297 Acked Flush 0 Done Sent Done false
5508 2298 Acked Read 7 Done Sent Done false
5509 2299 Acked Flush 0 Done Sent Done false
5510 2300 Acked Write 9 Done Sent Done false
5511 2301 Acked Write 8 Done Sent Done false
5512 2302 Acked Read 9 Done Sent Done false
5513 2303 Acked Flush 0 Done Sent Done false
5514 2304 Acked Read 9 Done Sent Done false
5515 2305 Acked Flush 0 Done Sent Done false
5516 2306 Acked Write 1 Done Sent Done false
5517 2307 Acked Flush 0 Done Sent Done false
5518 2308 Acked Read 6 Done Sent Done false
5519 2309 Acked Read 7 Done Sent Done false
5520 2310 Acked Read 9 Done Sent Done false
5521 2311 Acked Read 1 Done Sent Done false
5522 2312 Acked Write 5 Done Sent Done false
5523 2313 Acked Read 3 Done Sent Done false
5524 2314 Acked Read 8 Done Sent Done false
5525 2315 Acked Read 5 Done Sent Done false
5526 2316 Acked Read 3 Done Sent Done false
5527 2317 Acked Read 4 Done Sent Done false
5528 2318 Acked Read 7 Done Sent Done false
5529 2319 Acked Read 6 Done Sent Done false
5530 2320 Acked Write 3 Done Sent Done false
5531 2321 Acked Flush 0 Done Sent Done false
5532 2322 Acked Read 3 Done Sent Done false
5533 2323 Acked Write 2 Done Sent Done false
5534 2324 Acked Read 6 Done Sent Done false
5535 2325 Acked Write 10 Done Sent Done false
5536 2326 Acked Read 2 Done Sent Done false
5537 2327 Acked Read 5 Done Sent Done false
5538 2328 Acked Write 6 Done Sent Done false
5539 2329 Acked Write 9 Done Sent Done false
5540 2330 Acked Read 8 Done Sent Done false
5541 2331 Acked Write 2 Done Sent Done false
5542 2332 Acked Read 9 Done Sent Done false
5543 2333 Acked Read 3 Done Sent Done false
5544 2334 Acked Write 4 Done Sent Done false
5545 2335 Acked Write 8 Done Sent Done false
5546 2336 Acked Write 9 Done Sent Done false
5547 2337 Acked Read 4 Done Sent Done false
5548 2338 Acked Read 7 Done Sent Done false
5549 2339 Acked Write 7 Done Sent Done false
5550 2340 Acked Read 1 Done Sent Done false
5551 2341 Acked Read 7 Done Sent Done false
5552 2342 Acked Read 2 Done Sent Done false
5553 2343 Acked Flush 0 Done Sent Done false
5554 2344 Acked Read 4 Done Sent Done false
5555 2345 Acked Read 4 Done Sent Done false
5556 2346 Acked Write 2 Done Sent Done false
5557 2347 Acked Write 10 Done Sent Done false
5558 2348 Acked Write 3 Done Sent Done false
5559 2349 Acked Read 8 Done Sent Done false
5560 2350 Acked Read 5 Done Sent Done false
5561 2351 Acked Write 5 Done Sent Done false
5562 2352 Acked Read 2 Done Sent Done false
5563 2353 Acked Write 8 Done Sent Done false
5564 2354 Acked Read 9 Done Sent Done false
5565 2355 Acked Read 1 Done Sent Done false
5566 2356 Acked Read 8 Done Sent Done false
5567 2357 Acked Write 1 Done Sent Done false
5568 2358 Acked Read 8 Done Sent Done false
5569 2359 Acked Read 6 Done Sent Done false
5570 2360 Acked Read 6 Done Sent Done false
5571 2361 Acked Read 8 Done Sent Done false
5572 2362 Acked Read 6 Done Sent Done false
5573 2363 Acked Write 8 Done Sent Done false
5574 2364 Acked Write 3 Done Sent Done false
5575 2365 Acked Read 8 Done Sent Done false
5576 2366 Acked Read 10 Done Sent Done false
5577 2367 Acked Write 1 Done Sent Done false
5578 2368 Acked Read 7 Done Sent Done false
5579 2369 Acked Write 9 Done Sent Done false
5580 2370 Acked Write 1 Done Sent Done false
5581 2371 Acked Read 9 Done Sent Done false
5582 2372 Acked Read 1 Done Sent Done false
5583 2373 Acked Read 6 Done Sent Done false
5584 2374 Acked Read 6 Done Sent Done false
5585 2375 Acked Flush 0 Done Sent Done false
5586 2376 Acked Write 3 Done Sent Done false
5587 2377 Acked Read 4 Done Sent Done false
5588 2378 Acked Write 9 Done Sent Done false
5589 2379 Acked Write 4 Done Sent Done false
5590 2380 Acked Write 8 Done Sent Done false
5591 2381 Acked Write 1 Done Sent Done false
5592 2382 Acked Write 9 Done Sent Done false
5593 2383 Acked Write 6 Done Sent Done false
5594 2384 Acked Read 1 Done Sent Done false
5595 2385 Acked Write 7 Done Sent Done false
5596 2386 Acked Read 5 Done Sent Done false
5597 2387 Acked Read 2 Done Sent Done false
5598 2388 Acked Read 2 Done Sent Done false
5599 2389 Acked Read 3 Done Sent Done false
5600 2390 Acked Write 1 Done Sent Done false
5601 2391 Acked Read 1 Done Sent Done false
5602 2392 Acked Write 10 Done Sent Done false
5603 2393 Acked Read 7 Done Sent Done false
5604 2394 Acked Read 5 Done Sent Done false
5605 2395 Acked Read 8 Done Sent Done false
5606 2396 Acked Read 2 Done Sent Done false
5607 2397 Acked Read 2 Done Sent Done false
5608 2398 Acked Write 9 Done Sent Done false
5609 2399 Acked Read 3 Done Sent Done false
5610 2400 Acked Read 5 Done Sent Done false
5611 2401 Acked Write 9 Done Sent Done false
5612 2402 Acked Write 10 Done Sent Done false
5613 2403 Acked Write 8 Done Sent Done false
5614 2404 Acked Read 7 Done Sent Done false
5615 2405 Acked Flush 0 Done Sent Done false
5616 2406 Acked Flush 0 Done Sent Done false
5617 2407 Acked Write 10 Done Sent Done false
5618 2408 Acked Write 7 Done Sent Done false
5619 2409 Acked Flush 0 Done Sent Done false
5620 2410 Acked Write 5 Done Sent Done false
5621 2411 Acked Read 3 Done Sent Done false
5622 2412 Acked Read 1 Done Sent Done false
5623 2413 Acked Read 1 Done Sent Done false
5624 2414 Acked Read 5 Done Sent Done false
5625 2415 Acked Read 1 Done Sent Done false
5626 2416 Acked Read 8 Done Sent Done false
5627 2417 Acked Read 6 Done Sent Done false
5628 2418 Acked Flush 0 Done Sent Done false
5629 2419 Acked Read 10 Done Sent Done false
5630 2420 Acked Write 7 Done Sent Done false
5631 2421 Acked Read 3 Done Sent Done false
5632 2422 Acked Read 4 Done Sent Done false
5633 2423 Acked Read 3 Done Sent Done false
5634 2424 Acked Write 6 Done Sent Done false
5635 2425 Acked Write 1 Done Sent Done false
5636 2426 Acked Read 5 Done Sent Done false
5637 2427 Acked Write 6 Done Sent Done false
5638 2428 Acked Write 4 Done Sent Done false
5639 2429 Acked Read 10 Done Sent Done false
5640 2430 Acked Read 7 Done Sent Done false
5641 2431 Acked Read 10 Done Sent Done false
5642 2432 Acked Write 5 Done Sent Done false
5643 2433 Acked Write 8 Done Sent Done false
5644 2434 Acked Read 4 Done Sent Done false
5645 2435 Acked Write 4 Done Sent Done false
5646 2436 Acked Write 6 Done Sent Done false
5647 2437 Acked Read 7 Done Sent Done false
5648 2438 Acked Read 5 Done Sent Done false
5649 2439 Acked Read 8 Done Sent Done false
5650 2440 Acked Read 4 Done Sent Done false
5651 2441 Acked Write 10 Done Sent Done false
5652 2442 Acked Read 9 Done Sent Done false
5653 2443 Acked Write 5 Done Sent Done false
5654 2444 Acked Write 8 Done Sent Done false
5655 2445 Acked Read 4 Done Sent Done false
5656 2446 Acked Flush 0 Done Sent Done false
5657 2447 Acked Read 4 Done Sent Done false
5658 2448 Acked Read 2 Done Sent Done false
5659 2449 Acked Write 2 Done Sent Done false
5660 2450 Acked Read 3 Done Sent Done false
5661 2451 Acked Read 5 Done Sent Done false
5662 2452 Acked Read 9 Done Sent Done false
5663 2453 Acked Read 3 Done Sent Done false
5664 2454 Acked Read 1 Done Sent Done false
5665 2455 Acked Write 4 Done Sent Done false
5666 2456 Acked Read 9 Done Sent Done false
5667 2457 Acked Read 4 Done Sent Done false
5668 2458 Acked Write 6 Done Sent Done false
5669 2459 Acked Read 9 Done Sent Done false
5670 2460 Acked Write 2 Done Sent Done false
5671 2461 Acked Read 8 Done Sent Done false
5672 2462 Acked Write 9 Done Sent Done false
5673 2463 Acked Read 8 Done Sent Done false
5674 2464 Acked Read 8 Done Sent Done false
5675 2465 Acked Write 9 Done Sent Done false
5676 2466 Acked Read 9 Done Sent Done false
5677 2467 Acked Write 9 Done Sent Done false
5678 2468 Acked Read 4 Done Sent Done false
5679 2469 Acked Write 5 Done Sent Done false
5680 2470 Acked Read 7 Done Sent Done false
5681 2471 Acked Write 4 Done Sent Done false
5682 2472 Acked Write 5 Done Sent Done false
5683 2473 Acked Flush 0 Done Sent Done false
5684 2474 Acked Read 2 Done Sent Done false
5685 2475 Acked Flush 0 Done Sent Done false
5686 2476 Acked Write 2 Done Sent Done false
5687 2477 Acked Flush 0 Done Sent Done false
5688 2478 Acked Write 10 Done Sent Done false
5689 2479 Acked Flush 0 Done Sent Done false
5690 2480 Acked Write 8 Done Sent Done false
5691 2481 Acked Write 7 Done Sent Done false
5692 2482 Acked Read 4 Done Sent Done false
5693 2483 Acked Write 7 Done Sent Done false
5694 2484 Acked Read 8 Done Sent Done false
5695 2485 Acked Write 3 Done Sent Done false
5696 2486 Acked Write 2 Done Sent Done false
5697 2487 Acked Write 9 Done Sent Done false
5698 2488 Acked Write 10 Done Sent Done false
5699 2489 Acked Write 7 Done Sent Done false
5700 2490 Acked Flush 0 Done Sent Done false
5701 2491 Acked Flush 0 Done Sent Done false
5702 2492 Acked Write 3 Done Sent Done false
5703 2493 Acked Read 3 Done Sent Done false
5704 2494 Acked Read 2 Done Sent Done false
5705 2495 Acked Flush 0 Done Sent Done false
5706 2496 Acked Write 3 Done Sent Done false
5707 2497 Acked Read 10 Done Sent Done false
5708 2498 Acked Read 2 Done Sent Done false
5709 2499 Acked Write 10 Done Sent Done false
5710 2500 Acked Read 2 Done Sent Done false
5711 2501 Acked Flush 0 Done Sent Done false
5712 2502 Acked Read 3 Done Sent Done false
5713 2503 Acked Read 1 Done Sent Done false
5714 2504 Acked Read 9 Done Sent Done false
5715 2505 Acked Write 7 Done Sent Done false
5716 2506 Acked Read 5 Done Sent Done false
5717 2507 Acked Read 8 Done Sent Done false
5718 2508 Acked Write 7 Done Sent Done false
5719 STATES DS:0 DS:1 DS:2 TOTAL
5720 Sent 0 301 0 301
5721 Done 301 0 301 602
5722 Skipped 0 0 0 0
5723 Error 0 0 0 0
5724 Last Flush: Some(JobId(2501)) Some(JobId(2207)) Some(JobId(2501))
5725 Downstairs last five completed: 2207 2206 2205 2204 2203
5726 Upstairs last five completed: 2508 2507 2506 2505 2504
5727 CLIENT: Up:0 ds:301 act:2
57282025-01-28T21:55:31.612ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57292025-01-28T21:55:32.437ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57302025-01-28T21:55:32.867ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57312025-01-28T21:55:32.867ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49456) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57322025-01-28T21:55:32.868ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57332025-01-28T21:55:32.869ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57342025-01-28T21:55:32.869ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57352025-01-28T21:55:32.869ZINFOcrucible: send last flush ID to this DS: Some(JobId(2207)) = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57362025-01-28T21:55:32.869ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57372025-01-28T21:55:32.869ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2207)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57382025-01-28T21:55:32.869ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2207)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
5739 ----------------------------------------------------------------
5740 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5741 Downstairs last five completed: 2509 2508 2507 2506 2505
5742 Upstairs last five completed: 2509 2508 2507 2506 2505
5743 CLIENT: Up:0 ds:0 act:3
5744 Replay: All jobs finished
5745 Main task has work to do, go find it
5746 got dsc Stop(0)
5747 stop 0
5748 Jan 28 21:55:35.113 INFO request completed, latency_us: 345, response_code: 204, uri: /stop/cid/0[8810] Got stop action so:false kr:false
5749 , method: POST, req_id: 5060f653-e882-45d5-925c-724d7e21e3ef, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
5750 Jan 28 21:55:35.114 INFO request completed, latency_us: 122, response_code: 200, uri: /state/cid/0, method: GET, req_id: 277b7c35-ae33-45a9-a0ab-892966322403, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
57512025-01-28T21:55:35.114ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57522025-01-28T21:55:35.114ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57532025-01-28T21:55:35.114ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
57542025-01-28T21:55:35.114ZINFOcrucible: client task is exiting = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
5755 [8810] Exited with: ExitStatus(unix_wait_status(9))
5756 [8810][0] reports Exit
5757 Jan 28 21:55:39.115 INFO request completed, latency_us: 216, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9f1c5816-2500-4fcb-b996-82e14b4ac553, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
5758 001/300 Read block 3135 len 16384
5759 002/300 Read block 2627 len 20480
5760 003/300 Read block 2919 len 36864
5761 004/300 Read block 1822 len 36864
5762 005/300 Read block 3623 len 28672
5763 006/300 Read block 3051 len 36864
5764 007/300 Write block 4603 len 28672 data: 2 2 2 2 2 2 2
5765 008/300 Read block 849 len 20480
5766 009/300 Write block 238 len 28672 data: 2 2 2 2 2 3 3
5767 010/300 Write block 3178 len 24576 data: 2 2 2 3 4 4
5768 011/300 Write block 1974 len 16384 data: 3 2 2 2
5769 012/300 Write block 1204 len 36864 data: 5 4 3 3 3 3 3 3 3
5770 013/300 Read block 1002 len 24576
5771 014/300 Read block 1046 len 20480
5772 015/300 Flush
5773 016/300 Write block 4187 len 28672 data: 3 2 2 2 2 2 2
5774 017/300 Read block 4944 len 36864
5775 018/300 Write block 4727 len 24576 data: 2 2 3 3 3 4
5776 019/300 Write block 475 len 12288 data: 3 3 3
5777 020/300 Write block 4607 len 20480 data: 3 3 3 2 2
5778 021/300 Write block 892 len 8192 data: 3 3
5779 022/300 Flush
5780 023/300 Write block 4140 len 4096 data: 2
5781 024/300 Write block 635 len 12288 data: 3 4 4
5782 025/300 Write block 2216 len 12288 data: 2 2 2
5783 026/300 Flush
5784 027/300 Write block 3469 len 20480 data: 2 3 3 2 2
5785 028/300 Write block 1486 len 24576 data: 2 2 2 2 2 2
5786 029/300 Write block 930 len 20480 data: 3 3 2 3 3
5787 030/300 Flush
5788 031/300 Write block 3820 len 16384 data: 4 4 4 4
5789 032/300 Flush
5790 033/300 Write block 3611 len 28672 data: 2 2 2 2 2 2 2
5791 034/300 Read block 1185 len 16384
5792 035/300 Flush
5793 036/300 Read block 3747 len 8192
5794 037/300 Read block 1158 len 16384
5795 038/300 Write block 2333 len 8192 data: 3 3
5796 039/300 Write block 2228 len 12288 data: 2 2 2
5797 040/300 Write block 2328 len 20480 data: 2 3 3 3 3
5798 041/300 Flush
5799 042/300 Read block 3350 len 24576
5800 043/300 Read block 4660 len 40960
5801 044/300 Read block 1624 len 8192
5802 045/300 Write block 2940 len 36864 data: 3 2 2 2 2 3 3 3 3
5803 046/300 Write block 2244 len 40960 data: 2 2 2 2 2 2 3 3 3 2
5804 047/300 Write block 4469 len 40960 data: 2 2 2 2 2 2 2 3 4 5
5805 048/300 Write block 2078 len 4096 data: 3
5806 049/300 Write block 139 len 28672 data: 3 2 2 2 2 2 2
5807 050/300 Read block 2854 len 16384
5808 051/300 Write block 4912 len 20480 data: 2 2 2 2 2
5809 052/300 Read block 908 len 12288
5810 053/300 Read block 3964 len 4096
5811 054/300 Write block 3834 len 24576 data: 2 2 2 2 2 2
5812 055/300 Flush
5813 056/300 Read block 483 len 28672
5814 057/300 Read block 593 len 32768
5815 058/300 Read block 83 len 28672
5816 059/300 Write block 2149 len 28672 data: 3 3 4 4 4 3 3
5817 060/300 Read block 63 len 12288
5818 061/300 Write block 369 len 16384 data: 3 3 2 2
5819 062/300 Read block 4306 len 36864
5820 063/300 Write block 4171 len 36864 data: 3 3 2 2 2 2 2 2 2
5821 064/300 Write block 4967 len 16384 data: 2 2 2 2
5822 065/300 Read block 4905 len 28672
5823 066/300 Write block 2844 len 32768 data: 2 2 4 4 4 4 4 3
5824 067/300 Write block 6 len 12288 data: 3 3 2
5825 068/300 Read block 3585 len 36864
5826 069/300 Read block 354 len 16384
5827 070/300 Read block 3451 len 32768
5828 071/300 Write block 3050 len 40960 data: 3 2 2 2 2 3 3 3 3 5
5829 072/300 Read block 590 len 8192
5830 073/300 Write block 1191 len 32768 data: 2 2 2 2 2 2 3 3
5831 074/300 Read block 3806 len 20480
5832 075/300 Write block 4771 len 20480 data: 6 5 4 4 3
5833 076/300 Read block 964 len 32768
5834 077/300 Read block 3084 len 28672
5835 078/300 Read block 3302 len 28672
5836 079/300 Flush
5837 080/300 Read block 4113 len 40960
5838 081/300 Read block 2345 len 40960
5839 082/300 Read block 3109 len 28672
5840 083/300 Write block 4943 len 20480 data: 2 2 2 2 3
5841 084/300 Write block 441 len 36864 data: 2 2 3 2 2 2 2 2 2
5842 085/300 Read block 3573 len 32768
5843 086/300 Write block 4115 len 4096 data: 4
5844 087/300 Read block 554 len 32768
5845 088/300 Read block 3024 len 20480
5846 089/300 Read block 4934 len 16384
5847 090/300 Write block 774 len 12288 data: 2 2 2
5848 091/300 Write block 3574 len 20480 data: 2 2 3 3 3
5849 092/300 Read block 2306 len 40960
5850 093/300 Read block 4188 len 16384
5851 094/300 Read block 4922 len 16384
5852 095/300 Read block 4700 len 24576
5853 096/300 Flush
5854 097/300 Write block 3372 len 36864 data: 2 2 2 3 3 3 3 3 3
5855 098/300 Write block 2203 len 36864 data: 4 4 3 3 3 3 2 2 2
5856 099/300 Write block 4542 len 36864 data: 2 2 2 3 3 2 2 2 2
5857 100/300 Write block 3197 len 32768 data: 2 2 2 2 2 2 2 2
5858 101/300 Flush
5859 102/300 Read block 2121 len 4096
5860 103/300 Flush
5861 104/300 Read block 1750 len 20480
5862 105/300 Read block 1124 len 32768
5863 106/300 Read block 2218 len 8192
5864 107/300 Read block 4041 len 16384
5865 108/300 Write block 1938 len 16384 data: 2 2 2 2
5866 109/300 Flush
5867 110/300 Flush
5868 111/300 Read block 4712 len 32768
5869 112/300 Read block 1570 len 28672
5870 113/300 Read block 1078 len 32768
5871 114/300 Read block 2103 len 40960
5872 115/300 Write block 416 len 8192 data: 2 2
5873 116/300 Write block 2641 len 36864 data: 3 3 3 2 2 2 2 2 2
5874 117/300 Write block 664 len 40960 data: 2 2 3 3 3 3 3 2 2 2
5875 118/300 Flush
5876 119/300 Flush
5877 120/300 Write block 3129 len 16384 data: 2 2 2 2
5878 121/300 Write block 745 len 8192 data: 3 2
5879 122/300 Write block 835 len 32768 data: 2 2 2 2 3 3 3 3
5880 123/300 Read block 2306 len 36864
5881 124/300 Read block 3880 len 28672
5882 125/300 Read block 4907 len 20480
5883 126/300 Flush
5884 127/300 Flush
5885 128/300 Write block 3121 len 8192 data: 2 2
5886 129/300 Write block 3787 len 16384 data: 2 2 3 3
5887 130/300 Write block 584 len 32768 data: 2 2 2 2 3 2 2 2
5888 131/300 Read block 4139 len 8192
5889 132/300 Read block 1055 len 16384
5890 133/300 Flush
5891 134/300 Write block 567 len 8192 data: 2 2
5892 135/300 Write block 3218 len 8192 data: 3 3
5893 136/300 Read block 4511 len 16384
5894 137/300 Write block 2107 len 16384 data: 2 2 2 2
5895 138/300 Read block 647 len 20480
5896 139/300 Write block 3545 len 28672 data: 3 2 3 3 3 3 3
5897 140/300 Write block 3840 len 24576 data: 2 2 3 3 3 3
5898 141/300 Flush
5899 142/300 Write block 1167 len 32768 data: 2 2 2 2 2 2 2 2
5900 143/300 Write block 550 len 28672 data: 3 3 3 3 2 2 2
5901 144/300 Read block 1949 len 20480
5902 145/300 Read block 3412 len 16384
5903 146/300 Write block 4028 len 36864 data: 2 2 2 2 2 2 2 2 2
5904 147/300 Write block 2776 len 20480 data: 2 2 2 2 2
5905 148/300 Read block 4140 len 4096
5906 149/300 Read block 1075 len 4096
5907 150/300 Flush
5908 151/300 Read block 3521 len 24576
5909 152/300 Read block 4979 len 20480
5910 153/300 Write block 900 len 36864 data: 3 3 3 3 3 4 4 4 3
5911 154/300 Write block 1918 len 4096 data: 4
5912 155/300 Write block 4409 len 12288 data: 3 3 3
5913 156/300 Flush
5914 157/300 Read block 1657 len 8192
5915 158/300 Read block 1762 len 40960
5916 159/300 Read block 711 len 8192
5917 160/300 Write block 3489 len 8192 data: 6 6
5918 161/300 Write block 3176 len 4096 data: 2
5919 162/300 Write block 2282 len 8192 data: 2 3
5920 163/300 Write block 2388 len 12288 data: 2 2 3
5921 164/300 Flush
5922 165/300 Write block 4610 len 24576 data: 3 3 2 2 2 3
5923 166/300 Read block 3563 len 4096
5924 167/300 Write block 2934 len 40960 data: 4 3 2 2 3 3 4 3 3 3
5925 168/300 Write block 3048 len 24576 data: 4 3 4 3 3 3
5926 169/300 Read block 700 len 24576
5927 170/300 Write block 3050 len 24576 data: 5 4 4 4 3 4
5928 171/300 Write block 2687 len 40960 data: 3 3 2 2 2 2 2 2 2 2
5929 172/300 Write block 3117 len 24576 data: 2 2 2 2 3 3
5930 173/300 Read block 1590 len 32768
5931 174/300 Read block 2117 len 24576
5932 175/300 Read block 1613 len 20480
5933 176/300 Read block 1324 len 40960
5934 177/300 Flush
5935 178/300 Write block 4875 len 4096 data: 3
5936 179/300 Write block 758 len 36864 data: 2 2 2 2 2 2 3 3 4
5937 180/300 Write block 2201 len 40960 data: 4 4 5 5 4 4 4 4 3 3
5938 181/300 Write block 3190 len 20480 data: 4 3 4 4 4
5939 182/300 Write block 2678 len 4096 data: 5
5940 183/300 Write block 731 len 40960 data: 3 3 3 3 2 2 4 4 4 4
5941 184/300 Write block 4905 len 4096 data: 3
5942 185/300 Read block 638 len 36864
5943 186/300 Write block 1414 len 4096 data: 3
5944 187/300 Write block 4693 len 12288 data: 2 3 3
5945 188/300 Write block 4933 len 16384 data: 2 2 2 2
5946 189/300 Read block 3789 len 20480
5947 190/300 Read block 3156 len 4096
5948 191/300 Read block 1844 len 32768
5949 192/300 Read block 3182 len 32768
5950 193/300 Write block 41 len 36864 data: 3 3 3 3 3 3 2 2 2
5951 194/300 Write block 4189 len 16384 data: 3 3 3 3
5952 195/300 Write block 3625 len 4096 data: 3
5953 196/300 Write block 3567 len 8192 data: 2 2
5954 197/300 Read block 3647 len 36864
5955 198/300 Read block 3491 len 16384
5956 199/300 Write block 3899 len 4096 data: 3
5957 200/300 Read block 618 len 36864
5958 201/300 Write block 767 len 36864 data: 3 3 3 3 3 2 2 3 3
5959 202/300 Write block 4517 len 8192 data: 2 2
5960 203/300 Write block 2495 len 36864 data: 2 2 2 2 2 2 2 2 2
5961 204/300 Write block 371 len 28672 data: 3 3 3 3 3 4 4
5962 205/300 Write block 4433 len 24576 data: 2 2 2 2 2 2
5963 206/300 Read block 1939 len 24576
5964 207/300 Read block 1562 len 28672
5965 208/300 Write block 3640 len 36864 data: 2 2 2 2 2 2 2 2 2
5966 209/300 Flush
5967 210/300 Read block 4264 len 32768
5968 211/300 Read block 164 len 36864
5969 212/300 Write block 3726 len 20480 data: 2 2 2 2 3
5970 213/300 Read block 2682 len 16384
5971 214/300 Flush
5972 215/300 Write block 4588 len 8192 data: 3 2
5973 216/300 Write block 2368 len 12288 data: 4 4 4
5974 217/300 Write block 3909 len 28672 data: 4 3 3 3 3 2 2
5975 218/300 Read block 4476 len 40960
5976 219/300 Write block 2374 len 4096 data: 3
5977 220/300 Read block 3441 len 28672
5978 221/300 Write block 4684 len 8192 data: 2 2
5979 222/300 Read block 4478 len 12288
5980 223/300 Flush
5981 224/300 Read block 554 len 16384
5982 225/300 Read block 33 len 32768
5983 226/300 Read block 2750 len 28672
5984 227/300 Read block 793 len 20480
5985 228/300 Write block 4619 len 16384 data: 3 3 3 2
5986 229/300 Flush
5987 230/300 Write block 1850 len 28672 data: 2 2 2 2 2 2 2
5988 231/300 Write block 3723 len 40960 data: 2 2 2 3 3 3 3 4 3 3
5989 232/300 Read block 671 len 36864
5990 233/300 Write block 934 len 40960 data: 4 3 4 4 4 4 4 5 4 3
5991 234/300 Write block 575 len 20480 data: 2 2 2 2 2
5992 235/300 Write block 678 len 32768 data: 2 2 2 2 2 2 2 2
5993 236/300 Read block 684 len 4096
5994 237/300 Write block 2103 len 4096 data: 2
5995 238/300 Write block 2542 len 28672 data: 2 2 2 2 2 2 3
5996 239/300 Read block 3885 len 28672
5997 240/300 Read block 2775 len 32768
5998 241/300 Write block 1824 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5999 242/300 Read block 3513 len 36864
6000 243/300 Write block 527 len 32768 data: 3 2 2 2 2 2 3 3
6001 244/300 Read block 2111 len 28672
6002 245/300 Flush
6003 246/300 Read block 1683 len 40960
6004 247/300 Read block 3568 len 12288
6005 248/300 Read block 1363 len 40960
6006 249/300 Read block 3204 len 36864
6007 250/300 Write block 2646 len 24576 data: 3 3 3 3 2 2
6008 251/300 Write block 72 len 12288 data: 2 2 3
6009 252/300 Write block 1517 len 24576 data: 4 4 4 4 4 4
6010 253/300 Write block 2526 len 32768 data: 2 2 2 2 2 2 2 2
6011 254/300 Read block 3030 len 4096
6012 255/300 Write block 4737 len 12288 data: 3 2 2
6013 256/300 Read block 362 len 28672
6014 257/300 Write block 2610 len 36864 data: 2 2 2 2 3 5 5 5 6
6015 258/300 Flush
6016 259/300 Write block 1296 len 24576 data: 3 3 3 3 3 2
6017 260/300 Read block 4572 len 16384
6018 261/300 Write block 4402 len 4096 data: 4
6019 262/300 Write block 936 len 40960 data: 5 5 5 5 5 6 5 4 4 4
6020 263/300 Read block 2509 len 36864
6021 264/300 Flush
6022 265/300 Read block 889 len 8192
6023 266/300 Write block 4271 len 4096 data: 3
6024 267/300 Write block 606 len 24576 data: 3 3 3 2 2 2
6025 268/300 Read block 3961 len 16384
6026 269/300 Read block 1304 len 16384
6027 270/300 Read block 4683 len 36864
6028 271/300 Flush
6029 272/300 Read block 3595 len 16384
6030 273/300 Read block 3516 len 20480
6031 274/300 Write block 1506 len 4096 data: 4
6032 275/300 Write block 3436 len 36864 data: 2 2 2 2 2 2 2 2 2
6033 276/300 Read block 4165 len 4096
6034 277/300 Write block 3876 len 16384 data: 2 2 3 3
6035 278/300 Read block 2362 len 32768
6036 279/300 Flush
6037 280/300 Write block 2442 len 28672 data: 3 3 3 2 2 2 2
6038 281/300 Flush
6039 282/300 Write block 1068 len 40960 data: 3 3 2 2 2 2 2 2 2 3
6040 283/300 Write block 3688 len 16384 data: 2 2 2 2
6041 284/300 Read block 3572 len 28672
6042 285/300 Write block 2881 len 4096 data: 2
6043 286/300 Flush
6044 287/300 Read block 4240 len 24576
6045 288/300 Read block 303 len 28672
6046 289/300 Read block 450 len 40960
6047 290/300 Read block 1310 len 24576
6048 291/300 Write block 4095 len 28672 data: 2 2 2 2 2 2 3
6049 292/300 Read block 1144 len 36864
6050 293/300 Flush
6051 294/300 Read block 725 len 24576
6052 295/300 Write block 59 len 16384 data: 2 2 2 3
6053 296/300 Write block 304 len 40960 data: 4 3 3 2 2 2 2 2 2 2
6054 297/300 Read block 764 len 28672
6055 298/300 Write block 1486 len 20480 data: 3 3 3 3 3
6056 299/300 Read block 1152 len 4096
6057 300/300 Read block 3580 len 20480
6058 301/300 Read block 1715 len 24576
6059 Main task has work to do, go find it
6060 got dsc Start(0)
6061 start 0
6062 Jan 28 21:55:39.348 INFO request completed, latency_us: 252, response_code: 204, uri: /start/cid/0, method: POST, req_id: 93e33597-b0f3-4647-b121-0e7efa7a3187, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
6063 Replay: started 0, returned:Ok(())
6064 ----------------------------------------------------------------
6065 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6066 States: Offline Active Active
6067 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6068 2510 Acked Read 4 Sent Done Done false
6069 2511 Acked Read 5 Sent Done Done false
6070 2512 Acked Read 9 Sent Done Done false
6071 2513 Acked Read 9 Sent Done Done false
6072 2514 Acked Read 7 Sent Done Done false
6073 2515 Acked Read 9 Sent Done Done false
6074 2516 Acked Write 7 Sent Done Done false
6075 2517 Acked Read 5 Sent Done Done false
6076 2518 Acked Write 7 Sent Done Done false
6077 2519 Acked Write 6 Sent Done Done false
6078 2520 Acked Write 4 Sent Done Done false
6079 2521 Acked Write 9 Sent Done Done false
6080 2522 Acked Read 6 Sent Done Done false
6081 2523 Acked Read 5 Sent Done Done false
6082 2524 Acked Flush 0 Sent Done Done false
6083 2525 Acked Write 7 Sent Done Done false
6084 2526 Acked Read 9 Sent Done Done false
6085 2527 Acked Write 6 Sent Done Done false
6086 2528 Acked Write 3 Sent Done Done false
6087 2529 Acked Write 5 Sent Done Done false
6088 2530 Acked Write 2 Sent Done Done false
6089 2531 Acked Flush 0 Sent Done Done false
6090 2532 Acked Write 1 Sent Done Done false
6091 2533 Acked Write 3 Sent Done Done false
6092 2534 Acked Write 3 Sent Done Done false
6093 2535 Acked Flush 0 Sent Done Done false
6094 2536 Acked Write 5 Sent Done Done false
6095 2537 Acked Write 6 Sent Done Done false
6096 2538 Acked Write 5 Sent Done Done false
6097 2539 Acked Flush 0 Sent Done Done false
6098 2540 Acked Write 4 Sent Done Done false
6099 2541 Acked Flush 0 Sent Done Done false
6100 2542 Acked Write 7 Sent Done Done false
6101 2543 Acked Read 4 Sent Done Done false
6102 2544 Acked Flush 0 Sent Done Done false
6103 2545 Acked Read 2 Sent Done Done false
6104 2546 Acked Read 4 Sent Done Done false
6105 2547 Acked Write 2 Sent Done Done false
6106 2548 Acked Write 3 Sent Done Done false
6107 2549 Acked Write 5 Sent Done Done false
6108 2550 Acked Flush 0 Sent Done Done false
6109 2551 Acked Read 6 Sent Done Done false
6110 2552 Acked Read 10 Sent Done Done false
6111 2553 Acked Read 2 Sent Done Done false
6112 2554 Acked Write 9 Sent Done Done false
6113 2555 Acked Write 10 Sent Done Done false
6114 2556 Acked Write 10 Sent Done Done false
6115 2557 Acked Write 1 Sent Done Done false
6116 2558 Acked Write 7 Sent Done Done false
6117 2559 Acked Read 4 Sent Done Done false
6118 2560 Acked Write 5 Sent Done Done false
6119 2561 Acked Read 3 Sent Done Done false
6120 2562 Acked Read 1 Sent Done Done false
6121 2563 Acked Write 6 Sent Done Done false
6122 2564 Acked Flush 0 Sent Done Done false
6123 2565 Acked Read 7 Sent Done Done false
6124 2566 Acked Read 8 Sent Done Done false
6125 2567 Acked Read 7 Sent Done Done false
6126 2568 Acked Write 7 Sent Done Done false
6127 2569 Acked Read 3 Sent Done Done false
6128 2570 Acked Write 4 Sent Done Done false
6129 2571 Acked Read 9 Sent Done Done false
6130 2572 Acked Write 9 Sent Done Done false
6131 2573 Acked Write 4 Sent Done Done false
6132 2574 Acked Read 7 Sent Done Done false
6133 2575 Acked Write 8 Sent Done Done false
6134 2576 Acked Write 3 Sent Done Done false
6135 2577 Acked Read 9 Sent Done Done false
6136 2578 Acked Read 4 Sent Done Done false
6137 2579 Acked Read 8 Sent Done Done false
6138 2580 Acked Write 10 Sent Done Done false
6139 2581 Acked Read 2 Sent Done Done false
6140 2582 Acked Write 8 Sent Done Done false
6141 2583 Acked Read 5 Sent Done Done false
6142 2584 Acked Write 5 Sent Done Done false
6143 2585 Acked Read 8 Sent Done Done false
6144 2586 Acked Read 7 Sent Done Done false
6145 2587 Acked Read 7 Sent Done Done false
6146 2588 Acked Flush 0 Sent Done Done false
6147 2589 Acked Read 10 Sent Done Done false
6148 2590 Acked Read 10 Sent Done Done false
6149 2591 Acked Read 7 Sent Done Done false
6150 2592 Acked Write 5 Sent Done Done false
6151 2593 Acked Write 9 Sent Done Done false
6152 2594 Acked Read 8 Sent Done Done false
6153 2595 Acked Write 1 Sent Done Done false
6154 2596 Acked Read 8 Sent Done Done false
6155 2597 Acked Read 5 Sent Done Done false
6156 2598 Acked Read 4 Sent Done Done false
6157 2599 Acked Write 3 Sent Done Done false
6158 2600 Acked Write 5 Sent Done Done false
6159 2601 Acked Read 10 Sent Done Done false
6160 2602 Acked Read 4 Sent Done Done false
6161 2603 Acked Read 4 Sent Done Done false
6162 2604 Acked Read 6 Sent Done Done false
6163 2605 Acked Flush 0 Sent Done Done false
6164 2606 Acked Write 9 Sent Done Done false
6165 2607 Acked Write 9 Sent Done Done false
6166 2608 Acked Write 9 Sent Done Done false
6167 2609 Acked Write 8 Sent Done Done false
6168 2610 Acked Flush 0 Sent Done Done false
6169 2611 Acked Read 1 Sent Done Done false
6170 2612 Acked Flush 0 Sent Done Done false
6171 2613 Acked Read 5 Sent Done Done false
6172 2614 Acked Read 8 Sent Done Done false
6173 2615 Acked Read 2 Sent Done Done false
6174 2616 Acked Read 4 Sent Done Done false
6175 2617 Acked Write 4 Sent Done Done false
6176 2618 Acked Flush 0 Sent Done Done false
6177 2619 Acked Flush 0 Sent Done Done false
6178 2620 Acked Read 8 Sent Done Done false
6179 2621 Acked Read 7 Sent Done Done false
6180 2622 Acked Read 8 Sent Done Done false
6181 2623 Acked Read 10 Sent Done Done false
6182 2624 Acked Write 2 Sent Done Done false
6183 2625 Acked Write 9 Sent Done Done false
6184 2626 Acked Write 10 Sent Done Done false
6185 2627 Acked Flush 0 Sent Done Done false
6186 2628 Acked Flush 0 Sent Done Done false
6187 2629 Acked Write 4 Sent Done Done false
6188 2630 Acked Write 2 Sent Done Done false
6189 2631 Acked Write 8 Sent Done Done false
6190 2632 Acked Read 9 Sent Done Done false
6191 2633 Acked Read 7 Sent Done Done false
6192 2634 Acked Read 5 Sent Done Done false
6193 2635 Acked Flush 0 Sent Done Done false
6194 2636 Acked Flush 0 Sent Done Done false
6195 2637 Acked Write 2 Sent Done Done false
6196 2638 Acked Write 4 Sent Done Done false
6197 2639 Acked Write 8 Sent Done Done false
6198 2640 Acked Read 2 Sent Done Done false
6199 2641 Acked Read 4 Sent Done Done false
6200 2642 Acked Flush 0 Sent Done Done false
6201 2643 Acked Write 2 Sent Done Done false
6202 2644 Acked Write 2 Sent Done Done false
6203 2645 Acked Read 4 Sent Done Done false
6204 2646 Acked Write 4 Sent Done Done false
6205 2647 Acked Read 5 Sent Done Done false
6206 2648 Acked Write 7 Sent Done Done false
6207 2649 Acked Write 6 Sent Done Done false
6208 2650 Acked Flush 0 Sent Done Done false
6209 2651 Acked Write 8 Sent Done Done false
6210 2652 Acked Write 7 Sent Done Done false
6211 2653 Acked Read 5 Sent Done Done false
6212 2654 Acked Read 4 Sent Done Done false
6213 2655 Acked Write 9 Sent Done Done false
6214 2656 Acked Write 5 Sent Done Done false
6215 2657 Acked Read 1 Sent Done Done false
6216 2658 Acked Read 1 Sent Done Done false
6217 2659 Acked Flush 0 Sent Done Done false
6218 2660 Acked Read 6 Sent Done Done false
6219 2661 Acked Read 5 Sent Done Done false
6220 2662 Acked Write 9 Sent Done Done false
6221 2663 Acked Write 1 Sent Done Done false
6222 2664 Acked Write 3 Sent Done Done false
6223 2665 Acked Flush 0 Sent Done Done false
6224 2666 Acked Read 2 Sent Done Done false
6225 2667 Acked Read 10 Sent Done Done false
6226 2668 Acked Read 2 Sent Done Done false
6227 2669 Acked Write 2 Sent Done Done false
6228 2670 Acked Write 1 Sent Done Done false
6229 2671 Acked Write 2 Sent Done Done false
6230 2672 Acked Write 3 Sent Done Done false
6231 2673 Acked Flush 0 Sent Done Done false
6232 2674 Acked Write 6 Sent Done Done false
6233 2675 Acked Read 1 Sent Done Done false
6234 2676 Acked Write 10 Sent Done Done false
6235 2677 Acked Write 6 Sent Done Done false
6236 2678 Acked Read 6 Sent Done Done false
6237 2679 Acked Write 6 Sent Done Done false
6238 2680 Acked Write 10 Sent Done Done false
6239 2681 Acked Write 6 Sent Done Done false
6240 2682 Acked Read 8 Sent Done Done false
6241 2683 Acked Read 6 Sent Done Done false
6242 2684 Acked Read 5 Sent Done Done false
6243 2685 Acked Read 10 Sent Done Done false
6244 2686 Acked Flush 0 Sent Done Done false
6245 2687 Acked Write 1 Sent Done Done false
6246 2688 Acked Write 9 Sent Done Done false
6247 2689 Acked Write 10 Sent Done Done false
6248 2690 Acked Write 5 Sent Done Done false
6249 2691 Acked Write 1 Sent Done Done false
6250 2692 Acked Write 10 Sent Done Done false
6251 2693 Acked Write 1 Sent Done Done false
6252 2694 Acked Read 9 Sent Done Done false
6253 2695 Acked Write 1 Sent Done Done false
6254 2696 Acked Write 3 Sent Done Done false
6255 2697 Acked Write 4 Sent Done Done false
6256 2698 Acked Read 5 Sent Done Done false
6257 2699 Acked Read 1 Sent Done Done false
6258 2700 Acked Read 8 Sent Done Done false
6259 2701 Acked Read 8 Sent Done Done false
6260 2702 Acked Write 9 Sent Done Done false
6261 2703 Acked Write 4 Sent Done Done false
6262 2704 Acked Write 1 Sent Done Done false
6263 2705 Acked Write 2 Sent Done Done false
6264 2706 Acked Read 9 Sent Done Done false
6265 2707 Acked Read 4 Sent Done Done false
6266 2708 Acked Write 1 Sent Done Done false
6267 2709 Acked Read 9 Sent Done Done false
6268 2710 Acked Write 9 Sent Done Done false
6269 2711 Acked Write 2 Sent Done Done false
6270 2712 Acked Write 9 Sent Done Done false
6271 2713 Acked Write 7 Sent Done Done false
6272 2714 Acked Write 6 Sent Done Done false
6273 2715 Acked Read 6 Sent Done Done false
6274 2716 Acked Read 7 Sent Done Done false
6275 2717 Acked Write 9 Sent Done Done false
6276 2718 Acked Flush 0 Sent Done Done false
6277 2719 Acked Read 8 Sent Done Done false
6278 2720 Acked Read 9 Sent Done Done false
6279 2721 Acked Write 5 Sent Done Done false
6280 2722 Acked Read 4 Sent Done Done false
6281 2723 Acked Flush 0 Sent Done Done false
6282 2724 Acked Write 2 Sent Done Done false
6283 2725 Acked Write 3 Sent Done Done false
6284 2726 Acked Write 7 Sent Done Done false
6285 2727 Acked Read 10 Sent Done Done false
6286 2728 Acked Write 1 Sent Done Done false
6287 2729 Acked Read 7 Sent Done Done false
6288 2730 Acked Write 2 Sent Done Done false
6289 2731 Acked Read 3 Sent Done Done false
6290 2732 Acked Flush 0 Sent Done Done false
6291 2733 Acked Read 4 Sent Done Done false
6292 2734 Acked Read 8 Sent Done Done false
6293 2735 Acked Read 7 Sent Done Done false
6294 2736 Acked Read 5 Sent Done Done false
6295 2737 Acked Write 4 Sent Done Done false
6296 2738 Acked Flush 0 Sent Done Done false
6297 2739 Acked Write 7 Sent Done Done false
6298 2740 Acked Write 10 Sent Done Done false
6299 2741 Acked Read 9 Sent Done Done false
6300 2742 Acked Write 10 Sent Done Done false
6301 2743 Acked Write 5 Sent Done Done false
6302 2744 Acked Write 8 Sent Done Done false
6303 2745 Acked Read 1 Sent Done Done false
6304 2746 Acked Write 1 Sent Done Done false
6305 2747 Acked Write 7 Sent Done Done false
6306 2748 Acked Read 7 Sent Done Done false
6307 2749 Acked Read 8 Sent Done Done false
6308 2750 Acked Write 10 Sent Done Done false
6309 2751 Acked Read 9 Sent Done Done false
6310 2752 Acked Write 8 Sent Done Done false
6311 2753 Acked Read 7 Sent Done Done false
6312 2754 Acked Flush 0 Sent Done Done false
6313 2755 Acked Read 10 Sent Done Done false
6314 2756 Acked Read 3 Sent Done Done false
6315 2757 Acked Read 10 Sent Done Done false
6316 2758 Acked Read 9 Sent Done Done false
6317 2759 Acked Write 6 Sent Done Done false
6318 2760 Acked Write 3 Sent Done Done false
6319 2761 Acked Write 6 Sent Done Done false
6320 2762 Acked Write 8 Sent Done Done false
6321 2763 Acked Read 1 Sent Done Done false
6322 2764 Acked Write 3 Sent Done Done false
6323 2765 Acked Read 7 Sent Done Done false
6324 2766 Acked Write 9 Sent Done Done false
6325 2767 Acked Flush 0 Sent Done Done false
6326 2768 Acked Write 6 Sent Done Done false
6327 2769 Acked Read 4 Sent Done Done false
6328 2770 Acked Write 1 Sent Done Done false
6329 2771 Acked Write 10 Sent Done Done false
6330 2772 Acked Read 9 Sent Done Done false
6331 2773 Acked Flush 0 Sent Done Done false
6332 2774 Acked Read 2 Sent Done Done false
6333 2775 Acked Write 1 Sent Done Done false
6334 2776 Acked Write 6 Sent Done Done false
6335 2777 Acked Read 4 Sent Done Done false
6336 2778 Acked Read 4 Sent Done Done false
6337 2779 Acked Read 9 Sent Done Done false
6338 2780 Acked Flush 0 Sent Done Done false
6339 2781 Acked Read 4 Sent Done Done false
6340 2782 Acked Read 5 Sent Done Done false
6341 2783 Acked Write 1 Sent Done Done false
6342 2784 Acked Write 9 Sent Done Done false
6343 2785 Acked Read 1 Sent Done Done false
6344 2786 Acked Write 4 Sent Done Done false
6345 2787 Acked Read 8 Sent Done Done false
6346 2788 Acked Flush 0 Sent Done Done false
6347 2789 Acked Write 7 Sent Done Done false
6348 2790 Acked Flush 0 Sent Done Done false
6349 2791 Acked Write 10 Sent Done Done false
6350 2792 Acked Write 4 Sent Done Done false
6351 2793 Acked Read 7 Sent Done Done false
6352 2794 Acked Write 1 Sent Done Done false
6353 2795 Acked Flush 0 Sent Done Done false
6354 2796 Acked Read 6 Sent Done Done false
6355 2797 Acked Read 7 Sent Done Done false
6356 2798 Acked Read 10 Sent Done Done false
6357 2799 Acked Read 6 Sent Done Done false
6358 2800 Acked Write 7 Sent Done Done false
6359 2801 Acked Read 9 Sent Done Done false
6360 2802 Acked Flush 0 Sent Done Done false
6361 2803 Acked Read 6 Sent Done Done false
6362 2804 Acked Write 4 Sent Done Done false
6363 2805 Acked Write 10 Sent Done Done false
6364 2806 Acked Read 7 Sent Done Done false
6365 2807 Acked Write 5 Sent Done Done false
6366 2808 Acked Read 1 Sent Done Done false
6367 2809 Acked Read 5 Sent Done Done false
6368 2810 Acked Read 6 Sent Done Done false
6369 STATES DS:0 DS:1 DS:2 TOTAL
6370 Sent 301 0 0 301
6371 Done 0 301 301 602
6372 Skipped 0 0 0 0
6373 Error 0 0 0 0
6374 Last Flush: Some(JobId(2509)) Some(JobId(2802)) Some(JobId(2802))
6375 Downstairs last five completed: 2509 2508 2507 2506 2505
6376 Upstairs last five completed: 2810 2809 2808 2807 2806
6377 CLIENT: Up:0 ds:301 act:2
63782025-01-28T21:55:39.850ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
63792025-01-28T21:55:40.449ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
6380 [8810] got start action
6381 [8810] I am going to restart
6382 Starting downstairs at port 8810
6383 Make output file at "/tmp/dsc/downstairs-8810.txt"
6384 [8810][0] reports Starting
6385 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1208)
6386 [8810][0] reports Running
63872025-01-28T21:55:41.450ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
63882025-01-28T21:55:41.951ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
63892025-01-28T21:55:42.452ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
6390 ----------------------------------------------------------------
6391 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6392 States: Offline Active Active
6393 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6394 2510 Acked Read 4 Sent Done Done false
6395 2511 Acked Read 5 Sent Done Done false
6396 2512 Acked Read 9 Sent Done Done false
6397 2513 Acked Read 9 Sent Done Done false
6398 2514 Acked Read 7 Sent Done Done false
6399 2515 Acked Read 9 Sent Done Done false
6400 2516 Acked Write 7 Sent Done Done false
6401 2517 Acked Read 5 Sent Done Done false
6402 2518 Acked Write 7 Sent Done Done false
6403 2519 Acked Write 6 Sent Done Done false
6404 2520 Acked Write 4 Sent Done Done false
6405 2521 Acked Write 9 Sent Done Done false
6406 2522 Acked Read 6 Sent Done Done false
6407 2523 Acked Read 5 Sent Done Done false
6408 2524 Acked Flush 0 Sent Done Done false
6409 2525 Acked Write 7 Sent Done Done false
6410 2526 Acked Read 9 Sent Done Done false
6411 2527 Acked Write 6 Sent Done Done false
6412 2528 Acked Write 3 Sent Done Done false
6413 2529 Acked Write 5 Sent Done Done false
6414 2530 Acked Write 2 Sent Done Done false
6415 2531 Acked Flush 0 Sent Done Done false
6416 2532 Acked Write 1 Sent Done Done false
6417 2533 Acked Write 3 Sent Done Done false
6418 2534 Acked Write 3 Sent Done Done false
6419 2535 Acked Flush 0 Sent Done Done false
6420 2536 Acked Write 5 Sent Done Done false
6421 2537 Acked Write 6 Sent Done Done false
6422 2538 Acked Write 5 Sent Done Done false
6423 2539 Acked Flush 0 Sent Done Done false
6424 2540 Acked Write 4 Sent Done Done false
6425 2541 Acked Flush 0 Sent Done Done false
6426 2542 Acked Write 7 Sent Done Done false
6427 2543 Acked Read 4 Sent Done Done false
6428 2544 Acked Flush 0 Sent Done Done false
6429 2545 Acked Read 2 Sent Done Done false
6430 2546 Acked Read 4 Sent Done Done false
6431 2547 Acked Write 2 Sent Done Done false
6432 2548 Acked Write 3 Sent Done Done false
6433 2549 Acked Write 5 Sent Done Done false
6434 2550 Acked Flush 0 Sent Done Done false
6435 2551 Acked Read 6 Sent Done Done false
6436 2552 Acked Read 10 Sent Done Done false
6437 2553 Acked Read 2 Sent Done Done false
6438 2554 Acked Write 9 Sent Done Done false
6439 2555 Acked Write 10 Sent Done Done false
6440 2556 Acked Write 10 Sent Done Done false
6441 2557 Acked Write 1 Sent Done Done false
6442 2558 Acked Write 7 Sent Done Done false
6443 2559 Acked Read 4 Sent Done Done false
6444 2560 Acked Write 5 Sent Done Done false
6445 2561 Acked Read 3 Sent Done Done false
6446 2562 Acked Read 1 Sent Done Done false
6447 2563 Acked Write 6 Sent Done Done false
6448 2564 Acked Flush 0 Sent Done Done false
6449 2565 Acked Read 7 Sent Done Done false
6450 2566 Acked Read 8 Sent Done Done false
6451 2567 Acked Read 7 Sent Done Done false
6452 2568 Acked Write 7 Sent Done Done false
6453 2569 Acked Read 3 Sent Done Done false
6454 2570 Acked Write 4 Sent Done Done false
6455 2571 Acked Read 9 Sent Done Done false
6456 2572 Acked Write 9 Sent Done Done false
6457 2573 Acked Write 4 Sent Done Done false
6458 2574 Acked Read 7 Sent Done Done false
6459 2575 Acked Write 8 Sent Done Done false
6460 2576 Acked Write 3 Sent Done Done false
6461 2577 Acked Read 9 Sent Done Done false
6462 2578 Acked Read 4 Sent Done Done false
6463 2579 Acked Read 8 Sent Done Done false
6464 2580 Acked Write 10 Sent Done Done false
6465 2581 Acked Read 2 Sent Done Done false
6466 2582 Acked Write 8 Sent Done Done false
6467 2583 Acked Read 5 Sent Done Done false
6468 2584 Acked Write 5 Sent Done Done false
6469 2585 Acked Read 8 Sent Done Done false
6470 2586 Acked Read 7 Sent Done Done false
6471 2587 Acked Read 7 Sent Done Done false
6472 2588 Acked Flush 0 Sent Done Done false
6473 2589 Acked Read 10 Sent Done Done false
6474 2590 Acked Read 10 Sent Done Done false
6475 2591 Acked Read 7 Sent Done Done false
6476 2592 Acked Write 5 Sent Done Done false
6477 2593 Acked Write 9 Sent Done Done false
6478 2594 Acked Read 8 Sent Done Done false
6479 2595 Acked Write 1 Sent Done Done false
6480 2596 Acked Read 8 Sent Done Done false
6481 2597 Acked Read 5 Sent Done Done false
6482 2598 Acked Read 4 Sent Done Done false
6483 2599 Acked Write 3 Sent Done Done false
6484 2600 Acked Write 5 Sent Done Done false
6485 2601 Acked Read 10 Sent Done Done false
6486 2602 Acked Read 4 Sent Done Done false
6487 2603 Acked Read 4 Sent Done Done false
6488 2604 Acked Read 6 Sent Done Done false
6489 2605 Acked Flush 0 Sent Done Done false
6490 2606 Acked Write 9 Sent Done Done false
6491 2607 Acked Write 9 Sent Done Done false
6492 2608 Acked Write 9 Sent Done Done false
6493 2609 Acked Write 8 Sent Done Done false
6494 2610 Acked Flush 0 Sent Done Done false
6495 2611 Acked Read 1 Sent Done Done false
6496 2612 Acked Flush 0 Sent Done Done false
6497 2613 Acked Read 5 Sent Done Done false
6498 2614 Acked Read 8 Sent Done Done false
6499 2615 Acked Read 2 Sent Done Done false
6500 2616 Acked Read 4 Sent Done Done false
6501 2617 Acked Write 4 Sent Done Done false
6502 2618 Acked Flush 0 Sent Done Done false
6503 2619 Acked Flush 0 Sent Done Done false
6504 2620 Acked Read 8 Sent Done Done false
6505 2621 Acked Read 7 Sent Done Done false
6506 2622 Acked Read 8 Sent Done Done false
6507 2623 Acked Read 10 Sent Done Done false
6508 2624 Acked Write 2 Sent Done Done false
6509 2625 Acked Write 9 Sent Done Done false
6510 2626 Acked Write 10 Sent Done Done false
6511 2627 Acked Flush 0 Sent Done Done false
6512 2628 Acked Flush 0 Sent Done Done false
6513 2629 Acked Write 4 Sent Done Done false
6514 2630 Acked Write 2 Sent Done Done false
6515 2631 Acked Write 8 Sent Done Done false
6516 2632 Acked Read 9 Sent Done Done false
6517 2633 Acked Read 7 Sent Done Done false
6518 2634 Acked Read 5 Sent Done Done false
6519 2635 Acked Flush 0 Sent Done Done false
6520 2636 Acked Flush 0 Sent Done Done false
6521 2637 Acked Write 2 Sent Done Done false
6522 2638 Acked Write 4 Sent Done Done false
6523 2639 Acked Write 8 Sent Done Done false
6524 2640 Acked Read 2 Sent Done Done false
6525 2641 Acked Read 4 Sent Done Done false
6526 2642 Acked Flush 0 Sent Done Done false
6527 2643 Acked Write 2 Sent Done Done false
6528 2644 Acked Write 2 Sent Done Done false
6529 2645 Acked Read 4 Sent Done Done false
6530 2646 Acked Write 4 Sent Done Done false
6531 2647 Acked Read 5 Sent Done Done false
6532 2648 Acked Write 7 Sent Done Done false
6533 2649 Acked Write 6 Sent Done Done false
6534 2650 Acked Flush 0 Sent Done Done false
6535 2651 Acked Write 8 Sent Done Done false
6536 2652 Acked Write 7 Sent Done Done false
6537 2653 Acked Read 5 Sent Done Done false
6538 2654 Acked Read 4 Sent Done Done false
6539 2655 Acked Write 9 Sent Done Done false
6540 2656 Acked Write 5 Sent Done Done false
6541 2657 Acked Read 1 Sent Done Done false
6542 2658 Acked Read 1 Sent Done Done false
6543 2659 Acked Flush 0 Sent Done Done false
6544 2660 Acked Read 6 Sent Done Done false
6545 2661 Acked Read 5 Sent Done Done false
6546 2662 Acked Write 9 Sent Done Done false
6547 2663 Acked Write 1 Sent Done Done false
6548 2664 Acked Write 3 Sent Done Done false
6549 2665 Acked Flush 0 Sent Done Done false
6550 2666 Acked Read 2 Sent Done Done false
6551 2667 Acked Read 10 Sent Done Done false
6552 2668 Acked Read 2 Sent Done Done false
6553 2669 Acked Write 2 Sent Done Done false
6554 2670 Acked Write 1 Sent Done Done false
6555 2671 Acked Write 2 Sent Done Done false
6556 2672 Acked Write 3 Sent Done Done false
6557 2673 Acked Flush 0 Sent Done Done false
6558 2674 Acked Write 6 Sent Done Done false
6559 2675 Acked Read 1 Sent Done Done false
6560 2676 Acked Write 10 Sent Done Done false
6561 2677 Acked Write 6 Sent Done Done false
6562 2678 Acked Read 6 Sent Done Done false
6563 2679 Acked Write 6 Sent Done Done false
6564 2680 Acked Write 10 Sent Done Done false
6565 2681 Acked Write 6 Sent Done Done false
6566 2682 Acked Read 8 Sent Done Done false
6567 2683 Acked Read 6 Sent Done Done false
6568 2684 Acked Read 5 Sent Done Done false
6569 2685 Acked Read 10 Sent Done Done false
6570 2686 Acked Flush 0 Sent Done Done false
6571 2687 Acked Write 1 Sent Done Done false
6572 2688 Acked Write 9 Sent Done Done false
6573 2689 Acked Write 10 Sent Done Done false
6574 2690 Acked Write 5 Sent Done Done false
6575 2691 Acked Write 1 Sent Done Done false
6576 2692 Acked Write 10 Sent Done Done false
6577 2693 Acked Write 1 Sent Done Done false
6578 2694 Acked Read 9 Sent Done Done false
6579 2695 Acked Write 1 Sent Done Done false
6580 2696 Acked Write 3 Sent Done Done false
6581 2697 Acked Write 4 Sent Done Done false
6582 2698 Acked Read 5 Sent Done Done false
6583 2699 Acked Read 1 Sent Done Done false
6584 2700 Acked Read 8 Sent Done Done false
6585 2701 Acked Read 8 Sent Done Done false
6586 2702 Acked Write 9 Sent Done Done false
6587 2703 Acked Write 4 Sent Done Done false
6588 2704 Acked Write 1 Sent Done Done false
6589 2705 Acked Write 2 Sent Done Done false
6590 2706 Acked Read 9 Sent Done Done false
6591 2707 Acked Read 4 Sent Done Done false
6592 2708 Acked Write 1 Sent Done Done false
6593 2709 Acked Read 9 Sent Done Done false
6594 2710 Acked Write 9 Sent Done Done false
6595 2711 Acked Write 2 Sent Done Done false
6596 2712 Acked Write 9 Sent Done Done false
6597 2713 Acked Write 7 Sent Done Done false
6598 2714 Acked Write 6 Sent Done Done false
6599 2715 Acked Read 6 Sent Done Done false
6600 2716 Acked Read 7 Sent Done Done false
6601 2717 Acked Write 9 Sent Done Done false
6602 2718 Acked Flush 0 Sent Done Done false
6603 2719 Acked Read 8 Sent Done Done false
6604 2720 Acked Read 9 Sent Done Done false
6605 2721 Acked Write 5 Sent Done Done false
6606 2722 Acked Read 4 Sent Done Done false
6607 2723 Acked Flush 0 Sent Done Done false
6608 2724 Acked Write 2 Sent Done Done false
6609 2725 Acked Write 3 Sent Done Done false
6610 2726 Acked Write 7 Sent Done Done false
6611 2727 Acked Read 10 Sent Done Done false
6612 2728 Acked Write 1 Sent Done Done false
6613 2729 Acked Read 7 Sent Done Done false
6614 2730 Acked Write 2 Sent Done Done false
6615 2731 Acked Read 3 Sent Done Done false
6616 2732 Acked Flush 0 Sent Done Done false
6617 2733 Acked Read 4 Sent Done Done false
6618 2734 Acked Read 8 Sent Done Done false
6619 2735 Acked Read 7 Sent Done Done false
6620 2736 Acked Read 5 Sent Done Done false
6621 2737 Acked Write 4 Sent Done Done false
6622 2738 Acked Flush 0 Sent Done Done false
6623 2739 Acked Write 7 Sent Done Done false
6624 2740 Acked Write 10 Sent Done Done false
6625 2741 Acked Read 9 Sent Done Done false
6626 2742 Acked Write 10 Sent Done Done false
6627 2743 Acked Write 5 Sent Done Done false
6628 2744 Acked Write 8 Sent Done Done false
6629 2745 Acked Read 1 Sent Done Done false
6630 2746 Acked Write 1 Sent Done Done false
6631 2747 Acked Write 7 Sent Done Done false
6632 2748 Acked Read 7 Sent Done Done false
6633 2749 Acked Read 8 Sent Done Done false
6634 2750 Acked Write 10 Sent Done Done false
6635 2751 Acked Read 9 Sent Done Done false
6636 2752 Acked Write 8 Sent Done Done false
6637 2753 Acked Read 7 Sent Done Done false
6638 2754 Acked Flush 0 Sent Done Done false
6639 2755 Acked Read 10 Sent Done Done false
6640 2756 Acked Read 3 Sent Done Done false
6641 2757 Acked Read 10 Sent Done Done false
6642 2758 Acked Read 9 Sent Done Done false
6643 2759 Acked Write 6 Sent Done Done false
6644 2760 Acked Write 3 Sent Done Done false
6645 2761 Acked Write 6 Sent Done Done false
6646 2762 Acked Write 8 Sent Done Done false
6647 2763 Acked Read 1 Sent Done Done false
6648 2764 Acked Write 3 Sent Done Done false
6649 2765 Acked Read 7 Sent Done Done false
6650 2766 Acked Write 9 Sent Done Done false
6651 2767 Acked Flush 0 Sent Done Done false
6652 2768 Acked Write 6 Sent Done Done false
6653 2769 Acked Read 4 Sent Done Done false
6654 2770 Acked Write 1 Sent Done Done false
6655 2771 Acked Write 10 Sent Done Done false
6656 2772 Acked Read 9 Sent Done Done false
6657 2773 Acked Flush 0 Sent Done Done false
6658 2774 Acked Read 2 Sent Done Done false
6659 2775 Acked Write 1 Sent Done Done false
6660 2776 Acked Write 6 Sent Done Done false
6661 2777 Acked Read 4 Sent Done Done false
6662 2778 Acked Read 4 Sent Done Done false
6663 2779 Acked Read 9 Sent Done Done false
6664 2780 Acked Flush 0 Sent Done Done false
6665 2781 Acked Read 4 Sent Done Done false
6666 2782 Acked Read 5 Sent Done Done false
6667 2783 Acked Write 1 Sent Done Done false
6668 2784 Acked Write 9 Sent Done Done false
6669 2785 Acked Read 1 Sent Done Done false
6670 2786 Acked Write 4 Sent Done Done false
6671 2787 Acked Read 8 Sent Done Done false
6672 2788 Acked Flush 0 Sent Done Done false
6673 2789 Acked Write 7 Sent Done Done false
6674 2790 Acked Flush 0 Sent Done Done false
6675 2791 Acked Write 10 Sent Done Done false
6676 2792 Acked Write 4 Sent Done Done false
6677 2793 Acked Read 7 Sent Done Done false
6678 2794 Acked Write 1 Sent Done Done false
6679 2795 Acked Flush 0 Sent Done Done false
6680 2796 Acked Read 6 Sent Done Done false
6681 2797 Acked Read 7 Sent Done Done false
6682 2798 Acked Read 10 Sent Done Done false
6683 2799 Acked Read 6 Sent Done Done false
6684 2800 Acked Write 7 Sent Done Done false
6685 2801 Acked Read 9 Sent Done Done false
6686 2802 Acked Flush 0 Sent Done Done false
6687 2803 Acked Read 6 Sent Done Done false
6688 2804 Acked Write 4 Sent Done Done false
6689 2805 Acked Write 10 Sent Done Done false
6690 2806 Acked Read 7 Sent Done Done false
6691 2807 Acked Write 5 Sent Done Done false
6692 2808 Acked Read 1 Sent Done Done false
6693 2809 Acked Read 5 Sent Done Done false
6694 2810 Acked Read 6 Sent Done Done false
6695 STATES DS:0 DS:1 DS:2 TOTAL
6696 Sent 301 0 0 301
6697 Done 0 301 301 602
6698 Skipped 0 0 0 0
6699 Error 0 0 0 0
6700 Last Flush: Some(JobId(2509)) Some(JobId(2802)) Some(JobId(2802))
6701 Downstairs last five completed: 2509 2508 2507 2506 2505
6702 Upstairs last five completed: 2810 2809 2808 2807 2806
6703 CLIENT: Up:0 ds:301 act:2
67042025-01-28T21:55:43.852ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
67052025-01-28T21:55:44.456ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
67062025-01-28T21:55:45.115ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
67072025-01-28T21:55:45.115ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50045) = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
67082025-01-28T21:55:45.116ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
67092025-01-28T21:55:45.117ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
67102025-01-28T21:55:45.117ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
67112025-01-28T21:55:45.117ZINFOcrucible: send last flush ID to this DS: Some(JobId(2509)) = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
67122025-01-28T21:55:45.117ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
67132025-01-28T21:55:45.117ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2509)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
67142025-01-28T21:55:45.117ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2509)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
6715 ----------------------------------------------------------------
6716 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6717 Downstairs last five completed: 2811 2810 2809 2808 2807
6718 Upstairs last five completed: 2811 2810 2809 2808 2807
6719 CLIENT: Up:0 ds:0 act:3
6720 Replay: All jobs finished
6721 Main task has work to do, go find it
6722 got dsc Stop(0)
6723 stop 0
6724 Jan 28 21:55:47.354 INFO request completed, [8810] Got stop action so:false kr:false
6725 latency_us: 344, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 8ad9c8aa-f98e-4ac2-bfcc-1cb5e32314ad, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
67262025-01-28T21:55:47.354ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
67272025-01-28T21:55:47.354ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
67282025-01-28T21:55:47.354ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
6729 Jan 28 21:55:47.354 INFO request completed, latency_us: 174, response_code: 200, uri: /state/cid/0, method: GET, req_id: 59a83aee-5e68-44fb-82f6-3dab071ec315, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
67302025-01-28T21:55:47.355ZINFOcrucible: client task is exiting = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
6731 [8810] Exited with: ExitStatus(unix_wait_status(9))
6732 [8810][0] reports Exit
6733 Jan 28 21:55:51.357 INFO request completed, latency_us: 202, response_code: 200, uri: /state/cid/0, method: GET, req_id: 264328f5-9f1f-4a71-bc5d-5a296f03c71f, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
6734 001/300 Write block 468 len 40960 data: 2 2 2 2 2 3 3 4 4 4
6735 002/300 Read block 510 len 36864
6736 003/300 Write block 1100 len 8192 data: 2 2
6737 004/300 Write block 783 len 16384 data: 3 3 3 3
6738 005/300 Flush
6739 006/300 Write block 707 len 12288 data: 2 2 2
6740 007/300 Read block 1818 len 16384
6741 008/300 Flush
6742 009/300 Read block 773 len 4096
6743 010/300 Write block 3532 len 36864 data: 4 4 4 3 2 3 3 3 3
6744 011/300 Read block 3015 len 8192
6745 012/300 Read block 3663 len 12288
6746 013/300 Read block 4175 len 8192
6747 014/300 Flush
6748 015/300 Read block 3358 len 40960
6749 016/300 Read block 89 len 16384
6750 017/300 Read block 440 len 28672
6751 018/300 Read block 1615 len 40960
6752 019/300 Write block 1361 len 40960 data: 2 2 2 3 3 3 3 4 4 4
6753 020/300 Write block 637 len 16384 data: 5 2 2 2
6754 021/300 Read block 1896 len 28672
6755 022/300 Read block 2647 len 8192
6756 023/300 Write block 4562 len 40960 data: 3 3 4 5 5 4 4 4 4 4
6757 024/300 Write block 1331 len 24576 data: 2 3 3 3 3 3
6758 025/300 Flush
6759 026/300 Read block 1573 len 16384
6760 027/300 Read block 4629 len 16384
6761 028/300 Read block 2117 len 12288
6762 029/300 Flush
6763 030/300 Read block 4144 len 16384
6764 031/300 Write block 4769 len 24576 data: 4 5 7 6 5 5
6765 032/300 Write block 2928 len 16384 data: 4 4 4 4
6766 033/300 Write block 408 len 8192 data: 2 2
6767 034/300 Read block 4588 len 20480
6768 035/300 Write block 1003 len 20480 data: 4 4 4 4 4
6769 036/300 Read block 4768 len 8192
6770 037/300 Flush
6771 038/300 Write block 3095 len 8192 data: 2 2
6772 039/300 Read block 1664 len 16384
6773 040/300 Read block 2539 len 16384
6774 041/300 Write block 699 len 28672 data: 2 2 2 2 2 2 2
6775 042/300 Read block 4648 len 20480
6776 043/300 Write block 2010 len 8192 data: 4 4
6777 044/300 Read block 3351 len 8192
6778 045/300 Write block 3110 len 32768 data: 2 2 2 2 2 2 2 3
6779 046/300 Write block 3969 len 24576 data: 2 2 2 2 2 2
6780 047/300 Write block 1087 len 20480 data: 3 2 2 2 2
6781 048/300 Read block 2705 len 4096
6782 049/300 Read block 3692 len 4096
6783 050/300 Read block 3779 len 24576
6784 051/300 Write block 2823 len 40960 data: 2 2 2 3 4 4 3 3 3 3
6785 052/300 Read block 256 len 36864
6786 053/300 Write block 4490 len 24576 data: 4 3 3 2 2 2
6787 054/300 Write block 1703 len 32768 data: 3 3 3 3 2 2 2 2
6788 055/300 Write block 4426 len 12288 data: 3 2 2
6789 056/300 Flush
6790 057/300 Flush
6791 058/300 Read block 1175 len 12288
6792 059/300 Read block 4545 len 16384
6793 060/300 Write block 126 len 8192 data: 3 2
6794 061/300 Read block 1394 len 8192
6795 062/300 Read block 3396 len 8192
6796 063/300 Read block 884 len 16384
6797 064/300 Flush
6798 065/300 Read block 2960 len 28672
6799 066/300 Read block 1378 len 12288
6800 067/300 Flush
6801 068/300 Read block 2548 len 4096
6802 069/300 Read block 3638 len 40960
6803 070/300 Read block 2764 len 32768
6804 071/300 Read block 364 len 20480
6805 072/300 Write block 2563 len 20480 data: 2 2 2 2 2
6806 073/300 Write block 3225 len 8192 data: 2 2
6807 074/300 Flush
6808 075/300 Read block 1624 len 8192
6809 076/300 Read block 3090 len 12288
6810 077/300 Flush
6811 078/300 Write block 4683 len 36864 data: 2 3 3 2 2 2 2 3 3
6812 079/300 Write block 266 len 20480 data: 3 3 3 2 2
6813 080/300 Write block 1384 len 4096 data: 2
6814 081/300 Read block 760 len 36864
6815 082/300 Read block 3886 len 20480
6816 083/300 Read block 2774 len 16384
6817 084/300 Read block 267 len 20480
6818 085/300 Read block 1912 len 28672
6819 086/300 Write block 4326 len 16384 data: 2 2 2 2
6820 087/300 Read block 3782 len 12288
6821 088/300 Write block 2978 len 4096 data: 3
6822 089/300 Write block 3466 len 36864 data: 2 2 2 3 4 4 3 3 2
6823 090/300 Read block 2249 len 8192
6824 091/300 Read block 1157 len 36864
6825 092/300 Read block 4546 len 4096
6826 093/300 Read block 915 len 24576
6827 094/300 Write block 4920 len 32768 data: 2 2 2 2 2 2 2 2
6828 095/300 Write block 2423 len 36864 data: 3 3 4 4 4 4 4 4 4
6829 096/300 Write block 1145 len 20480 data: 2 2 2 2 3
6830 097/300 Write block 2067 len 40960 data: 2 2 3 4 3 3 3 2 2 3
6831 098/300 Write block 3447 len 24576 data: 2 2 2 2 2 2
6832 099/300 Write block 2994 len 32768 data: 4 4 3 3 3 2 3 3
6833 100/300 Write block 2257 len 32768 data: 2 2 2 2 2 2 2 2
6834 101/300 Read block 3015 len 24576
6835 102/300 Write block 1488 len 16384 data: 4 4 4 3
6836 103/300 Flush
6837 104/300 Write block 1602 len 8192 data: 2 2
6838 105/300 Write block 382 len 28672 data: 2 2 2 2 2 2 2
6839 106/300 Flush
6840 107/300 Read block 3505 len 20480
6841 108/300 Write block 966 len 20480 data: 3 3 3 3 3
6842 109/300 Write block 1194 len 4096 data: 3
6843 110/300 Read block 3143 len 20480
6844 111/300 Write block 3059 len 40960 data: 6 5 5 3 3 3 3 3 3 3
6845 112/300 Read block 4313 len 28672
6846 113/300 Read block 1556 len 32768
6847 114/300 Read block 1467 len 16384
6848 115/300 Read block 2788 len 4096
6849 116/300 Write block 2123 len 12288 data: 3 4 3
6850 117/300 Read block 2987 len 16384
6851 118/300 Read block 1135 len 16384
6852 119/300 Write block 2777 len 4096 data: 3
6853 120/300 Read block 4786 len 20480
6854 121/300 Read block 67 len 36864
6855 122/300 Read block 1619 len 32768
6856 123/300 Read block 1194 len 36864
6857 124/300 Read block 3963 len 20480
6858 125/300 Read block 4989 len 28672
6859 126/300 Write block 169 len 36864 data: 2 3 3 3 3 3 3 3 2
6860 127/300 Read block 2598 len 36864
6861 128/300 Read block 3462 len 36864
6862 129/300 Write block 4664 len 28672 data: 6 6 6 5 3 3 3
6863 130/300 Read block 1225 len 12288
6864 131/300 Read block 4242 len 8192
6865 132/300 Read block 527 len 24576
6866 133/300 Write block 1967 len 20480 data: 2 2 3 3 3
6867 134/300 Read block 1235 len 40960
6868 135/300 Read block 4111 len 4096
6869 136/300 Read block 744 len 12288
6870 137/300 Write block 1898 len 36864 data: 2 2 2 2 2 2 4 3 3
6871 138/300 Read block 935 len 32768
6872 139/300 Write block 4481 len 24576 data: 7 7 6 5 4 6
6873 140/300 Write block 3408 len 20480 data: 2 2 2 2 2
6874 141/300 Read block 3156 len 8192
6875 142/300 Flush
6876 143/300 Read block 2232 len 8192
6877 144/300 Read block 4948 len 40960
6878 145/300 Write block 3175 len 28672 data: 2 3 2 3 3 3 4
6879 146/300 Read block 3327 len 28672
6880 147/300 Write block 3781 len 32768 data: 3 3 3 3 3 3 3 3
6881 148/300 Read block 316 len 16384
6882 149/300 Read block 2596 len 32768
6883 150/300 Write block 2311 len 16384 data: 2 2 2 3
6884 151/300 Write block 1542 len 4096 data: 2
6885 152/300 Write block 4072 len 12288 data: 2 3 4
6886 153/300 Read block 859 len 32768
6887 154/300 Read block 2440 len 4096
6888 155/300 Write block 3020 len 8192 data: 2 3
6889 156/300 Write block 959 len 20480 data: 2 2 2 2 3
6890 157/300 Read block 3327 len 16384
6891 158/300 Read block 884 len 4096
6892 159/300 Read block 1944 len 20480
6893 160/300 Read block 802 len 4096
6894 161/300 Read block 3946 len 32768
6895 162/300 Read block 2544 len 8192
6896 163/300 Flush
6897 164/300 Read block 3739 len 16384
6898 165/300 Write block 1032 len 32768 data: 3 3 3 3 4 4 4 5
6899 166/300 Write block 1876 len 36864 data: 3 3 3 3 3 3 3 2 2
6900 167/300 Write block 651 len 28672 data: 3 3 3 2 2 2 2
6901 168/300 Read block 4554 len 20480
6902 169/300 Write block 1712 len 32768 data: 2 2 2 2 2 3 3 4
6903 170/300 Read block 3965 len 20480
6904 171/300 Read block 4001 len 32768
6905 172/300 Write block 1409 len 16384 data: 3 4 3 3
6906 173/300 Write block 4298 len 24576 data: 3 3 3 2 2 2
6907 174/300 Write block 4799 len 8192 data: 3 3
6908 175/300 Read block 2183 len 32768
6909 176/300 Read block 1105 len 8192
6910 177/300 Write block 4082 len 24576 data: 3 4 4 4 3 3
6911 178/300 Read block 4912 len 20480
6912 179/300 Write block 847 len 8192 data: 2 2
6913 180/300 Flush
6914 181/300 Read block 400 len 40960
6915 182/300 Read block 2562 len 24576
6916 183/300 Read block 882 len 28672
6917 184/300 Flush
6918 185/300 Read block 141 len 24576
6919 186/300 Read block 2204 len 36864
6920 187/300 Flush
6921 188/300 Read block 3994 len 20480
6922 189/300 Write block 3269 len 36864 data: 2 2 3 3 3 3 3 3 2
6923 190/300 Flush
6924 191/300 Write block 453 len 32768 data: 2 2 2 3 3 3 3 4
6925 192/300 Write block 1021 len 36864 data: 2 2 2 2 2 2 2 2 2
6926 193/300 Write block 1198 len 36864 data: 4 3 3 3 4 4 6 5 4
6927 194/300 Read block 3196 len 20480
6928 195/300 Write block 1378 len 28672 data: 2 2 2 2 2 2 3
6929 196/300 Read block 4504 len 12288
6930 197/300 Write block 2068 len 16384 data: 3 4 5 4
6931 198/300 Write block 2169 len 8192 data: 4 3
6932 199/300 Write block 889 len 20480 data: 4 2 2 4 4
6933 200/300 Read block 4749 len 28672
6934 201/300 Write block 1178 len 36864 data: 3 5 5 4 4 3 3 3 2
6935 202/300 Read block 2865 len 28672
6936 203/300 Write block 1008 len 28672 data: 2 2 2 2 2 2 2
6937 204/300 Read block 3249 len 36864
6938 205/300 Read block 3257 len 28672
6939 206/300 Flush
6940 207/300 Read block 299 len 36864
6941 208/300 Flush
6942 209/300 Write block 3368 len 12288 data: 2 2 2
6943 210/300 Read block 40 len 16384
6944 211/300 Read block 1339 len 16384
6945 212/300 Read block 2297 len 40960
6946 213/300 Write block 2000 len 8192 data: 2 2
6947 214/300 Read block 2960 len 28672
6948 215/300 Read block 2071 len 28672
6949 216/300 Read block 914 len 16384
6950 217/300 Read block 3576 len 40960
6951 218/300 Write block 4830 len 32768 data: 3 2 2 2 2 2 2 2
6952 219/300 Write block 3818 len 4096 data: 4
6953 220/300 Read block 4331 len 16384
6954 221/300 Flush
6955 222/300 Write block 3305 len 12288 data: 4 3 3
6956 223/300 Read block 1820 len 28672
6957 224/300 Write block 4718 len 32768 data: 2 2 2 2 2 2 2 2
6958 225/300 Read block 4378 len 40960
6959 226/300 Write block 2688 len 40960 data: 4 3 3 3 3 3 3 3 3 3
6960 227/300 Read block 1511 len 32768
6961 228/300 Read block 235 len 24576
6962 229/300 Write block 4860 len 32768 data: 3 3 3 4 3 3 3 3
6963 230/300 Read block 4778 len 4096
6964 231/300 Write block 3958 len 16384 data: 5 5 5 5
6965 232/300 Write block 297 len 40960 data: 2 3 3 4 5 4 4 5 4 4
6966 233/300 Read block 2899 len 4096
6967 234/300 Write block 2979 len 40960 data: 3 2 2 2 2 2 3 3 2 3
6968 235/300 Read block 4899 len 32768
6969 236/300 Read block 3827 len 4096
6970 237/300 Read block 1530 len 32768
6971 238/300 Read block 600 len 36864
6972 239/300 Read block 2019 len 40960
6973 240/300 Flush
6974 241/300 Read block 18 len 20480
6975 242/300 Flush
6976 243/300 Write block 1626 len 8192 data: 2 2
6977 244/300 Read block 3489 len 32768
6978 245/300 Write block 4611 len 12288 data: 4 3 3
6979 246/300 Read block 1811 len 16384
6980 247/300 Write block 4313 len 28672 data: 3 2 2 2 3 3 3
6981 248/300 Read block 3007 len 4096
6982 249/300 Read block 2181 len 16384
6983 250/300 Write block 57 len 8192 data: 2 2
6984 251/300 Write block 86 len 12288 data: 2 2 2
6985 252/300 Write block 388 len 40960 data: 3 2 2 2 2 2 2 2 2 2
6986 253/300 Flush
6987 254/300 Read block 1398 len 32768
6988 255/300 Write block 2963 len 12288 data: 2 2 2
6989 256/300 Write block 2773 len 16384 data: 2 2 2 3
6990 257/300 Read block 4481 len 4096
6991 258/300 Read block 1415 len 20480
6992 259/300 Write block 964 len 28672 data: 3 3 4 4 4 4 4
6993 260/300 Read block 3689 len 28672
6994 261/300 Read block 4836 len 16384
6995 262/300 Read block 205 len 24576
6996 263/300 Read block 3267 len 32768
6997 264/300 Write block 2473 len 20480 data: 3 4 5 5 5
6998 265/300 Read block 1598 len 28672
6999 266/300 Write block 1562 len 4096 data: 3
7000 267/300 Write block 3513 len 28672 data: 2 3 3 3 3 3 3
7001 268/300 Write block 4007 len 24576 data: 2 2 2 2 2 2
7002 269/300 Flush
7003 270/300 Write block 3635 len 16384 data: 2 2 2 2
7004 271/300 Flush
7005 272/300 Read block 1675 len 40960
7006 273/300 Read block 2465 len 20480
7007 274/300 Read block 1785 len 4096
7008 275/300 Read block 373 len 20480
7009 276/300 Read block 2944 len 28672
7010 277/300 Read block 1311 len 28672
7011 278/300 Read block 2805 len 24576
7012 279/300 Flush
7013 280/300 Read block 3665 len 28672
7014 281/300 Write block 3864 len 12288 data: 2 2 2
7015 282/300 Write block 4658 len 40960 data: 4 5 5 5 5 6 7 7 7 6
7016 283/300 Flush
7017 284/300 Write block 1653 len 32768 data: 2 2 3 3 3 3 3 3
7018 285/300 Read block 4683 len 12288
7019 286/300 Read block 53 len 28672
7020 287/300 Flush
7021 288/300 Write block 3391 len 8192 data: 3 3
7022 289/300 Flush
7023 290/300 Read block 1175 len 32768
7024 291/300 Flush
7025 292/300 Read block 4486 len 24576
7026 293/300 Read block 324 len 12288
7027 294/300 Read block 2601 len 12288
7028 295/300 Flush
7029 296/300 Flush
7030 297/300 Write block 2142 len 24576 data: 3 3 4 3 3 3
7031 298/300 Read block 4428 len 36864
7032 299/300 Write block 1619 len 8192 data: 2 2
7033 300/300 Write block 1861 len 20480 data: 2 2 2 2 2
7034 301/300 Read block 831 len 28672
7035 Main task has work to do, go find it
7036 got dsc Start(0)
7037 start 0
7038 Jan 28 21:55:51.597 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/0, method: POST, req_id: e72d1651-4e55-478b-a9e1-dd9b7075a83e, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
7039 Replay: started 0, returned:Ok(())
7040 ----------------------------------------------------------------
7041 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7042 States: Offline Active Active
7043 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7044 2812 Acked Write 10 Sent Done Done false
7045 2813 Acked Read 9 Sent Done Done false
7046 2814 Acked Write 2 Sent Done Done false
7047 2815 Acked Write 4 Sent Done Done false
7048 2816 Acked Flush 0 Sent Done Done false
7049 2817 Acked Write 3 Sent Done Done false
7050 2818 Acked Read 4 Sent Done Done false
7051 2819 Acked Flush 0 Sent Done Done false
7052 2820 Acked Read 1 Sent Done Done false
7053 2821 Acked Write 9 Sent Done Done false
7054 2822 Acked Read 2 Sent Done Done false
7055 2823 Acked Read 3 Sent Done Done false
7056 2824 Acked Read 2 Sent Done Done false
7057 2825 Acked Flush 0 Sent Done Done false
7058 2826 Acked Read 10 Sent Done Done false
7059 2827 Acked Read 4 Sent Done Done false
7060 2828 Acked Read 7 Sent Done Done false
7061 2829 Acked Read 10 Sent Done Done false
7062 2830 Acked Write 10 Sent Done Done false
7063 2831 Acked Write 4 Sent Done Done false
7064 2832 Acked Read 7 Sent Done Done false
7065 2833 Acked Read 2 Sent Done Done false
7066 2834 Acked Write 10 Sent Done Done false
7067 2835 Acked Write 6 Sent Done Done false
7068 2836 Acked Flush 0 Sent Done Done false
7069 2837 Acked Read 4 Sent Done Done false
7070 2838 Acked Read 4 Sent Done Done false
7071 2839 Acked Read 3 Sent Done Done false
7072 2840 Acked Flush 0 Sent Done Done false
7073 2841 Acked Read 4 Sent Done Done false
7074 2842 Acked Write 6 Sent Done Done false
7075 2843 Acked Write 4 Sent Done Done false
7076 2844 Acked Write 2 Sent Done Done false
7077 2845 Acked Read 5 Sent Done Done false
7078 2846 Acked Write 5 Sent Done Done false
7079 2847 Acked Read 2 Sent Done Done false
7080 2848 Acked Flush 0 Sent Done Done false
7081 2849 Acked Write 2 Sent Done Done false
7082 2850 Acked Read 4 Sent Done Done false
7083 2851 Acked Read 4 Sent Done Done false
7084 2852 Acked Write 7 Sent Done Done false
7085 2853 Acked Read 5 Sent Done Done false
7086 2854 Acked Write 2 Sent Done Done false
7087 2855 Acked Read 2 Sent Done Done false
7088 2856 Acked Write 8 Sent Done Done false
7089 2857 Acked Write 6 Sent Done Done false
7090 2858 Acked Write 5 Sent Done Done false
7091 2859 Acked Read 1 Sent Done Done false
7092 2860 Acked Read 1 Sent Done Done false
7093 2861 Acked Read 6 Sent Done Done false
7094 2862 Acked Write 10 Sent Done Done false
7095 2863 Acked Read 9 Sent Done Done false
7096 2864 Acked Write 6 Sent Done Done false
7097 2865 Acked Write 8 Sent Done Done false
7098 2866 Acked Write 3 Sent Done Done false
7099 2867 Acked Flush 0 Sent Done Done false
7100 2868 Acked Flush 0 Sent Done Done false
7101 2869 Acked Read 3 Sent Done Done false
7102 2870 Acked Read 4 Sent Done Done false
7103 2871 Acked Write 2 Sent Done Done false
7104 2872 Acked Read 2 Sent Done Done false
7105 2873 Acked Read 2 Sent Done Done false
7106 2874 Acked Read 4 Sent Done Done false
7107 2875 Acked Flush 0 Sent Done Done false
7108 2876 Acked Read 7 Sent Done Done false
7109 2877 Acked Read 3 Sent Done Done false
7110 2878 Acked Flush 0 Sent Done Done false
7111 2879 Acked Read 1 Sent Done Done false
7112 2880 Acked Read 10 Sent Done Done false
7113 2881 Acked Read 8 Sent Done Done false
7114 2882 Acked Read 5 Sent Done Done false
7115 2883 Acked Write 5 Sent Done Done false
7116 2884 Acked Write 2 Sent Done Done false
7117 2885 Acked Flush 0 Sent Done Done false
7118 2886 Acked Read 2 Sent Done Done false
7119 2887 Acked Read 3 Sent Done Done false
7120 2888 Acked Flush 0 Sent Done Done false
7121 2889 Acked Write 9 Sent Done Done false
7122 2890 Acked Write 5 Sent Done Done false
7123 2891 Acked Write 1 Sent Done Done false
7124 2892 Acked Read 9 Sent Done Done false
7125 2893 Acked Read 5 Sent Done Done false
7126 2894 Acked Read 4 Sent Done Done false
7127 2895 Acked Read 5 Sent Done Done false
7128 2896 Acked Read 7 Sent Done Done false
7129 2897 Acked Write 4 Sent Done Done false
7130 2898 Acked Read 3 Sent Done Done false
7131 2899 Acked Write 1 Sent Done Done false
7132 2900 Acked Write 9 Sent Done Done false
7133 2901 Acked Read 2 Sent Done Done false
7134 2902 Acked Read 9 Sent Done Done false
7135 2903 Acked Read 1 Sent Done Done false
7136 2904 Acked Read 6 Sent Done Done false
7137 2905 Acked Write 8 Sent Done Done false
7138 2906 Acked Write 9 Sent Done Done false
7139 2907 Acked Write 5 Sent Done Done false
7140 2908 Acked Write 10 Sent Done Done false
7141 2909 Acked Write 6 Sent Done Done false
7142 2910 Acked Write 8 Sent Done Done false
7143 2911 Acked Write 8 Sent Done Done false
7144 2912 Acked Read 6 Sent Done Done false
7145 2913 Acked Write 4 Sent Done Done false
7146 2914 Acked Flush 0 Sent Done Done false
7147 2915 Acked Write 2 Sent Done Done false
7148 2916 Acked Write 7 Sent Done Done false
7149 2917 Acked Flush 0 Sent Done Done false
7150 2918 Acked Read 5 Sent Done Done false
7151 2919 Acked Write 5 Sent Done Done false
7152 2920 Acked Write 1 Sent Done Done false
7153 2921 Acked Read 5 Sent Done Done false
7154 2922 Acked Write 10 Sent Done Done false
7155 2923 Acked Read 7 Sent Done Done false
7156 2924 Acked Read 8 Sent Done Done false
7157 2925 Acked Read 4 Sent Done Done false
7158 2926 Acked Read 1 Sent Done Done false
7159 2927 Acked Write 3 Sent Done Done false
7160 2928 Acked Read 4 Sent Done Done false
7161 2929 Acked Read 4 Sent Done Done false
7162 2930 Acked Write 1 Sent Done Done false
7163 2931 Acked Read 5 Sent Done Done false
7164 2932 Acked Read 9 Sent Done Done false
7165 2933 Acked Read 8 Sent Done Done false
7166 2934 Acked Read 9 Sent Done Done false
7167 2935 Acked Read 5 Sent Done Done false
7168 2936 Acked Read 7 Sent Done Done false
7169 2937 Acked Write 9 Sent Done Done false
7170 2938 Acked Read 9 Sent Done Done false
7171 2939 Acked Read 9 Sent Done Done false
7172 2940 Acked Write 7 Sent Done Done false
7173 2941 Acked Read 3 Sent Done Done false
7174 2942 Acked Read 2 Sent Done Done false
7175 2943 Acked Read 6 Sent Done Done false
7176 2944 Acked Write 5 Sent Done Done false
7177 2945 Acked Read 10 Sent Done Done false
7178 2946 Acked Read 1 Sent Done Done false
7179 2947 Acked Read 3 Sent Done Done false
7180 2948 Acked Write 9 Sent Done Done false
7181 2949 Acked Read 8 Sent Done Done false
7182 2950 Acked Write 6 Sent Done Done false
7183 2951 Acked Write 5 Sent Done Done false
7184 2952 Acked Read 2 Sent Done Done false
7185 2953 Acked Flush 0 Sent Done Done false
7186 2954 Acked Read 2 Sent Done Done false
7187 2955 Acked Read 10 Sent Done Done false
7188 2956 Acked Write 7 Sent Done Done false
7189 2957 Acked Read 7 Sent Done Done false
7190 2958 Acked Write 8 Sent Done Done false
7191 2959 Acked Read 4 Sent Done Done false
7192 2960 Acked Read 8 Sent Done Done false
7193 2961 Acked Write 4 Sent Done Done false
7194 2962 Acked Write 1 Sent Done Done false
7195 2963 Acked Write 3 Sent Done Done false
7196 2964 Acked Read 8 Sent Done Done false
7197 2965 Acked Read 1 Sent Done Done false
7198 2966 Acked Write 2 Sent Done Done false
7199 2967 Acked Write 5 Sent Done Done false
7200 2968 Acked Read 4 Sent Done Done false
7201 2969 Acked Read 1 Sent Done Done false
7202 2970 Acked Read 5 Sent Done Done false
7203 2971 Acked Read 1 Sent Done Done false
7204 2972 Acked Read 8 Sent Done Done false
7205 2973 Acked Read 2 Sent Done Done false
7206 2974 Acked Flush 0 Sent Done Done false
7207 2975 Acked Read 4 Sent Done Done false
7208 2976 Acked Write 8 Sent Done Done false
7209 2977 Acked Write 9 Sent Done Done false
7210 2978 Acked Write 7 Sent Done Done false
7211 2979 Acked Read 5 Sent Done Done false
7212 2980 Acked Write 8 Sent Done Done false
7213 2981 Acked Read 5 Sent Done Done false
7214 2982 Acked Read 8 Sent Done Done false
7215 2983 Acked Write 4 Sent Done Done false
7216 2984 Acked Write 6 Sent Done Done false
7217 2985 Acked Write 2 Sent Done Done false
7218 2986 Acked Read 8 Sent Done Done false
7219 2987 Acked Read 2 Sent Done Done false
7220 2988 Acked Write 6 Sent Done Done false
7221 2989 Acked Read 5 Sent Done Done false
7222 2990 Acked Write 2 Sent Done Done false
7223 2991 Acked Flush 0 Sent Done Done false
7224 2992 Acked Read 10 Sent Done Done false
7225 2993 Acked Read 6 Sent Done Done false
7226 2994 Acked Read 7 Sent Done Done false
7227 2995 Acked Flush 0 Sent Done Done false
7228 2996 Acked Read 6 Sent Done Done false
7229 2997 Acked Read 9 Sent Done Done false
7230 2998 Acked Flush 0 Sent Done Done false
7231 2999 Acked Read 5 Sent Done Done false
7232 3000 Acked Write 9 Sent Done Done false
7233 3001 Acked Flush 0 Sent Done Done false
7234 3002 Acked Write 8 Sent Done Done false
7235 3003 Acked Write 9 Sent Done Done false
7236 3004 Acked Write 9 Sent Done Done false
7237 3005 Acked Read 5 Sent Done Done false
7238 3006 Acked Write 7 Sent Done Done false
7239 3007 Acked Read 3 Sent Done Done false
7240 3008 Acked Write 4 Sent Done Done false
7241 3009 Acked Write 2 Sent Done Done false
7242 3010 Acked Write 5 Sent Done Done false
7243 3011 Acked Read 7 Sent Done Done false
7244 3012 Acked Write 9 Sent Done Done false
7245 3013 Acked Read 7 Sent Done Done false
7246 3014 Acked Write 7 Sent Done Done false
7247 3015 Acked Read 9 Sent Done Done false
7248 3016 Acked Read 7 Sent Done Done false
7249 3017 Acked Flush 0 Sent Done Done false
7250 3018 Acked Read 9 Sent Done Done false
7251 3019 Acked Flush 0 Sent Done Done false
7252 3020 Acked Write 3 Sent Done Done false
7253 3021 Acked Read 4 Sent Done Done false
7254 3022 Acked Read 4 Sent Done Done false
7255 3023 Acked Read 10 Sent Done Done false
7256 3024 Acked Write 2 Sent Done Done false
7257 3025 Acked Read 7 Sent Done Done false
7258 3026 Acked Read 7 Sent Done Done false
7259 3027 Acked Read 4 Sent Done Done false
7260 3028 Acked Read 10 Sent Done Done false
7261 3029 Acked Write 8 Sent Done Done false
7262 3030 Acked Write 1 Sent Done Done false
7263 3031 Acked Read 4 Sent Done Done false
7264 3032 Acked Flush 0 Sent Done Done false
7265 3033 Acked Write 3 Sent Done Done false
7266 3034 Acked Read 7 Sent Done Done false
7267 3035 Acked Write 8 Sent Done Done false
7268 3036 Acked Read 10 Sent Done Done false
7269 3037 Acked Write 10 Sent Done Done false
7270 3038 Acked Read 8 Sent Done Done false
7271 3039 Acked Read 6 Sent Done Done false
7272 3040 Acked Write 8 Sent Done Done false
7273 3041 Acked Read 1 Sent Done Done false
7274 3042 Acked Write 4 Sent Done Done false
7275 3043 Acked Write 10 Sent Done Done false
7276 3044 Acked Read 1 Sent Done Done false
7277 3045 Acked Write 10 Sent Done Done false
7278 3046 Acked Read 8 Sent Done Done false
7279 3047 Acked Read 1 Sent Done Done false
7280 3048 Acked Read 8 Sent Done Done false
7281 3049 Acked Read 9 Sent Done Done false
7282 3050 Acked Read 10 Sent Done Done false
7283 3051 Acked Flush 0 Sent Done Done false
7284 3052 Acked Read 5 Sent Done Done false
7285 3053 Acked Flush 0 Sent Done Done false
7286 3054 Acked Write 2 Sent Done Done false
7287 3055 Acked Read 8 Sent Done Done false
7288 3056 Acked Write 3 Sent Done Done false
7289 3057 Acked Read 4 Sent Done Done false
7290 3058 Acked Write 7 Sent Done Done false
7291 3059 Acked Read 1 Sent Done Done false
7292 3060 Acked Read 4 Sent Done Done false
7293 3061 Acked Write 2 Sent Done Done false
7294 3062 Acked Write 3 Sent Done Done false
7295 3063 Acked Write 10 Sent Done Done false
7296 3064 Acked Flush 0 Sent Done Done false
7297 3065 Acked Read 8 Sent Done Done false
7298 3066 Acked Write 3 Sent Done Done false
7299 3067 Acked Write 4 Sent Done Done false
7300 3068 Acked Read 1 Sent Done Done false
7301 3069 Acked Read 5 Sent Done Done false
7302 3070 Acked Write 7 Sent Done Done false
7303 3071 Acked Read 7 Sent Done Done false
7304 3072 Acked Read 4 Sent Done Done false
7305 3073 Acked Read 6 Sent Done Done false
7306 3074 Acked Read 8 Sent Done Done false
7307 3075 Acked Write 5 Sent Done Done false
7308 3076 Acked Read 7 Sent Done Done false
7309 3077 Acked Write 1 Sent Done Done false
7310 3078 Acked Write 7 Sent Done Done false
7311 3079 Acked Write 6 Sent Done Done false
7312 3080 Acked Flush 0 Sent Done Done false
7313 3081 Acked Write 4 Sent Done Done false
7314 3082 Acked Flush 0 Sent Done Done false
7315 3083 Acked Read 10 Sent Done Done false
7316 3084 Acked Read 5 Sent Done Done false
7317 3085 Acked Read 1 Sent Done Done false
7318 3086 Acked Read 5 Sent Done Done false
7319 3087 Acked Read 7 Sent Done Done false
7320 3088 Acked Read 7 Sent Done Done false
7321 3089 Acked Read 6 Sent Done Done false
7322 3090 Acked Flush 0 Sent Done Done false
7323 3091 Acked Read 7 Sent Done Done false
7324 3092 Acked Write 3 Sent Done Done false
7325 3093 Acked Write 10 Sent Done Done false
7326 3094 Acked Flush 0 Sent Done Done false
7327 3095 Acked Write 8 Sent Done Done false
7328 3096 Acked Read 3 Sent Done Done false
7329 3097 Acked Read 7 Sent Done Done false
7330 3098 Acked Flush 0 Sent Done Done false
7331 3099 Acked Write 2 Sent Done Done false
7332 3100 Acked Flush 0 Sent Done Done false
7333 3101 Acked Read 8 Sent Done Done false
7334 3102 Acked Flush 0 Sent Done Done false
7335 3103 Acked Read 6 Sent Done Done false
7336 3104 Acked Read 3 Sent Done Done false
7337 3105 Acked Read 3 Sent Done Done false
7338 3106 Acked Flush 0 Sent Done Done false
7339 3107 Acked Flush 0 Sent Done Done false
7340 3108 Acked Write 6 Sent Done Done false
7341 3109 Acked Read 9 Sent Done Done false
7342 3110 Acked Write 2 Sent Done Done false
7343 3111 Acked Write 5 Sent Done Done false
7344 3112 Acked Read 7 Sent Done Done false
7345 STATES DS:0 DS:1 DS:2 TOTAL
7346 Sent 301 0 0 301
7347 Done 0 301 301 602
7348 Skipped 0 0 0 0
7349 Error 0 0 0 0
7350 Last Flush: Some(JobId(2811)) Some(JobId(3107)) Some(JobId(3107))
7351 Downstairs last five completed: 2811 2810 2809 2808 2807
7352 Upstairs last five completed: 3112 3111 3110 3109 3108
7353 CLIENT: Up:0 ds:301 act:2
73542025-01-28T21:55:52.467ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
7355 [8810] got start action
7356 [8810] I am going to restart
7357 Starting downstairs at port 8810
7358 Make output file at "/tmp/dsc/downstairs-8810.txt"
7359 [8810][0] reports Starting
7360 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1209)
7361 [8810][0] reports Running
73622025-01-28T21:55:53.468ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
73632025-01-28T21:55:54.470ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
73642025-01-28T21:55:55.471ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
7365 ----------------------------------------------------------------
7366 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7367 States: Offline Active Active
7368 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7369 2812 Acked Write 10 Sent Done Done false
7370 2813 Acked Read 9 Sent Done Done false
7371 2814 Acked Write 2 Sent Done Done false
7372 2815 Acked Write 4 Sent Done Done false
7373 2816 Acked Flush 0 Sent Done Done false
7374 2817 Acked Write 3 Sent Done Done false
7375 2818 Acked Read 4 Sent Done Done false
7376 2819 Acked Flush 0 Sent Done Done false
7377 2820 Acked Read 1 Sent Done Done false
7378 2821 Acked Write 9 Sent Done Done false
7379 2822 Acked Read 2 Sent Done Done false
7380 2823 Acked Read 3 Sent Done Done false
7381 2824 Acked Read 2 Sent Done Done false
7382 2825 Acked Flush 0 Sent Done Done false
7383 2826 Acked Read 10 Sent Done Done false
7384 2827 Acked Read 4 Sent Done Done false
7385 2828 Acked Read 7 Sent Done Done false
7386 2829 Acked Read 10 Sent Done Done false
7387 2830 Acked Write 10 Sent Done Done false
7388 2831 Acked Write 4 Sent Done Done false
7389 2832 Acked Read 7 Sent Done Done false
7390 2833 Acked Read 2 Sent Done Done false
7391 2834 Acked Write 10 Sent Done Done false
7392 2835 Acked Write 6 Sent Done Done false
7393 2836 Acked Flush 0 Sent Done Done false
7394 2837 Acked Read 4 Sent Done Done false
7395 2838 Acked Read 4 Sent Done Done false
7396 2839 Acked Read 3 Sent Done Done false
7397 2840 Acked Flush 0 Sent Done Done false
7398 2841 Acked Read 4 Sent Done Done false
7399 2842 Acked Write 6 Sent Done Done false
7400 2843 Acked Write 4 Sent Done Done false
7401 2844 Acked Write 2 Sent Done Done false
7402 2845 Acked Read 5 Sent Done Done false
7403 2846 Acked Write 5 Sent Done Done false
7404 2847 Acked Read 2 Sent Done Done false
7405 2848 Acked Flush 0 Sent Done Done false
7406 2849 Acked Write 2 Sent Done Done false
7407 2850 Acked Read 4 Sent Done Done false
7408 2851 Acked Read 4 Sent Done Done false
7409 2852 Acked Write 7 Sent Done Done false
7410 2853 Acked Read 5 Sent Done Done false
7411 2854 Acked Write 2 Sent Done Done false
7412 2855 Acked Read 2 Sent Done Done false
7413 2856 Acked Write 8 Sent Done Done false
7414 2857 Acked Write 6 Sent Done Done false
7415 2858 Acked Write 5 Sent Done Done false
7416 2859 Acked Read 1 Sent Done Done false
7417 2860 Acked Read 1 Sent Done Done false
7418 2861 Acked Read 6 Sent Done Done false
7419 2862 Acked Write 10 Sent Done Done false
7420 2863 Acked Read 9 Sent Done Done false
7421 2864 Acked Write 6 Sent Done Done false
7422 2865 Acked Write 8 Sent Done Done false
7423 2866 Acked Write 3 Sent Done Done false
7424 2867 Acked Flush 0 Sent Done Done false
7425 2868 Acked Flush 0 Sent Done Done false
7426 2869 Acked Read 3 Sent Done Done false
7427 2870 Acked Read 4 Sent Done Done false
7428 2871 Acked Write 2 Sent Done Done false
7429 2872 Acked Read 2 Sent Done Done false
7430 2873 Acked Read 2 Sent Done Done false
7431 2874 Acked Read 4 Sent Done Done false
7432 2875 Acked Flush 0 Sent Done Done false
7433 2876 Acked Read 7 Sent Done Done false
7434 2877 Acked Read 3 Sent Done Done false
7435 2878 Acked Flush 0 Sent Done Done false
7436 2879 Acked Read 1 Sent Done Done false
7437 2880 Acked Read 10 Sent Done Done false
7438 2881 Acked Read 8 Sent Done Done false
7439 2882 Acked Read 5 Sent Done Done false
7440 2883 Acked Write 5 Sent Done Done false
7441 2884 Acked Write 2 Sent Done Done false
7442 2885 Acked Flush 0 Sent Done Done false
7443 2886 Acked Read 2 Sent Done Done false
7444 2887 Acked Read 3 Sent Done Done false
7445 2888 Acked Flush 0 Sent Done Done false
7446 2889 Acked Write 9 Sent Done Done false
7447 2890 Acked Write 5 Sent Done Done false
7448 2891 Acked Write 1 Sent Done Done false
7449 2892 Acked Read 9 Sent Done Done false
7450 2893 Acked Read 5 Sent Done Done false
7451 2894 Acked Read 4 Sent Done Done false
7452 2895 Acked Read 5 Sent Done Done false
7453 2896 Acked Read 7 Sent Done Done false
7454 2897 Acked Write 4 Sent Done Done false
7455 2898 Acked Read 3 Sent Done Done false
7456 2899 Acked Write 1 Sent Done Done false
7457 2900 Acked Write 9 Sent Done Done false
7458 2901 Acked Read 2 Sent Done Done false
7459 2902 Acked Read 9 Sent Done Done false
7460 2903 Acked Read 1 Sent Done Done false
7461 2904 Acked Read 6 Sent Done Done false
7462 2905 Acked Write 8 Sent Done Done false
7463 2906 Acked Write 9 Sent Done Done false
7464 2907 Acked Write 5 Sent Done Done false
7465 2908 Acked Write 10 Sent Done Done false
7466 2909 Acked Write 6 Sent Done Done false
7467 2910 Acked Write 8 Sent Done Done false
7468 2911 Acked Write 8 Sent Done Done false
7469 2912 Acked Read 6 Sent Done Done false
7470 2913 Acked Write 4 Sent Done Done false
7471 2914 Acked Flush 0 Sent Done Done false
7472 2915 Acked Write 2 Sent Done Done false
7473 2916 Acked Write 7 Sent Done Done false
7474 2917 Acked Flush 0 Sent Done Done false
7475 2918 Acked Read 5 Sent Done Done false
7476 2919 Acked Write 5 Sent Done Done false
7477 2920 Acked Write 1 Sent Done Done false
7478 2921 Acked Read 5 Sent Done Done false
7479 2922 Acked Write 10 Sent Done Done false
7480 2923 Acked Read 7 Sent Done Done false
7481 2924 Acked Read 8 Sent Done Done false
7482 2925 Acked Read 4 Sent Done Done false
7483 2926 Acked Read 1 Sent Done Done false
7484 2927 Acked Write 3 Sent Done Done false
7485 2928 Acked Read 4 Sent Done Done false
7486 2929 Acked Read 4 Sent Done Done false
7487 2930 Acked Write 1 Sent Done Done false
7488 2931 Acked Read 5 Sent Done Done false
7489 2932 Acked Read 9 Sent Done Done false
7490 2933 Acked Read 8 Sent Done Done false
7491 2934 Acked Read 9 Sent Done Done false
7492 2935 Acked Read 5 Sent Done Done false
7493 2936 Acked Read 7 Sent Done Done false
7494 2937 Acked Write 9 Sent Done Done false
7495 2938 Acked Read 9 Sent Done Done false
7496 2939 Acked Read 9 Sent Done Done false
7497 2940 Acked Write 7 Sent Done Done false
7498 2941 Acked Read 3 Sent Done Done false
7499 2942 Acked Read 2 Sent Done Done false
7500 2943 Acked Read 6 Sent Done Done false
7501 2944 Acked Write 5 Sent Done Done false
7502 2945 Acked Read 10 Sent Done Done false
7503 2946 Acked Read 1 Sent Done Done false
7504 2947 Acked Read 3 Sent Done Done false
7505 2948 Acked Write 9 Sent Done Done false
7506 2949 Acked Read 8 Sent Done Done false
7507 2950 Acked Write 6 Sent Done Done false
7508 2951 Acked Write 5 Sent Done Done false
7509 2952 Acked Read 2 Sent Done Done false
7510 2953 Acked Flush 0 Sent Done Done false
7511 2954 Acked Read 2 Sent Done Done false
7512 2955 Acked Read 10 Sent Done Done false
7513 2956 Acked Write 7 Sent Done Done false
7514 2957 Acked Read 7 Sent Done Done false
7515 2958 Acked Write 8 Sent Done Done false
7516 2959 Acked Read 4 Sent Done Done false
7517 2960 Acked Read 8 Sent Done Done false
7518 2961 Acked Write 4 Sent Done Done false
7519 2962 Acked Write 1 Sent Done Done false
7520 2963 Acked Write 3 Sent Done Done false
7521 2964 Acked Read 8 Sent Done Done false
7522 2965 Acked Read 1 Sent Done Done false
7523 2966 Acked Write 2 Sent Done Done false
7524 2967 Acked Write 5 Sent Done Done false
7525 2968 Acked Read 4 Sent Done Done false
7526 2969 Acked Read 1 Sent Done Done false
7527 2970 Acked Read 5 Sent Done Done false
7528 2971 Acked Read 1 Sent Done Done false
7529 2972 Acked Read 8 Sent Done Done false
7530 2973 Acked Read 2 Sent Done Done false
7531 2974 Acked Flush 0 Sent Done Done false
7532 2975 Acked Read 4 Sent Done Done false
7533 2976 Acked Write 8 Sent Done Done false
7534 2977 Acked Write 9 Sent Done Done false
7535 2978 Acked Write 7 Sent Done Done false
7536 2979 Acked Read 5 Sent Done Done false
7537 2980 Acked Write 8 Sent Done Done false
7538 2981 Acked Read 5 Sent Done Done false
7539 2982 Acked Read 8 Sent Done Done false
7540 2983 Acked Write 4 Sent Done Done false
7541 2984 Acked Write 6 Sent Done Done false
7542 2985 Acked Write 2 Sent Done Done false
7543 2986 Acked Read 8 Sent Done Done false
7544 2987 Acked Read 2 Sent Done Done false
7545 2988 Acked Write 6 Sent Done Done false
7546 2989 Acked Read 5 Sent Done Done false
7547 2990 Acked Write 2 Sent Done Done false
7548 2991 Acked Flush 0 Sent Done Done false
7549 2992 Acked Read 10 Sent Done Done false
7550 2993 Acked Read 6 Sent Done Done false
7551 2994 Acked Read 7 Sent Done Done false
7552 2995 Acked Flush 0 Sent Done Done false
7553 2996 Acked Read 6 Sent Done Done false
7554 2997 Acked Read 9 Sent Done Done false
7555 2998 Acked Flush 0 Sent Done Done false
7556 2999 Acked Read 5 Sent Done Done false
7557 3000 Acked Write 9 Sent Done Done false
7558 3001 Acked Flush 0 Sent Done Done false
7559 3002 Acked Write 8 Sent Done Done false
7560 3003 Acked Write 9 Sent Done Done false
7561 3004 Acked Write 9 Sent Done Done false
7562 3005 Acked Read 5 Sent Done Done false
7563 3006 Acked Write 7 Sent Done Done false
7564 3007 Acked Read 3 Sent Done Done false
7565 3008 Acked Write 4 Sent Done Done false
7566 3009 Acked Write 2 Sent Done Done false
7567 3010 Acked Write 5 Sent Done Done false
7568 3011 Acked Read 7 Sent Done Done false
7569 3012 Acked Write 9 Sent Done Done false
7570 3013 Acked Read 7 Sent Done Done false
7571 3014 Acked Write 7 Sent Done Done false
7572 3015 Acked Read 9 Sent Done Done false
7573 3016 Acked Read 7 Sent Done Done false
7574 3017 Acked Flush 0 Sent Done Done false
7575 3018 Acked Read 9 Sent Done Done false
7576 3019 Acked Flush 0 Sent Done Done false
7577 3020 Acked Write 3 Sent Done Done false
7578 3021 Acked Read 4 Sent Done Done false
7579 3022 Acked Read 4 Sent Done Done false
7580 3023 Acked Read 10 Sent Done Done false
7581 3024 Acked Write 2 Sent Done Done false
7582 3025 Acked Read 7 Sent Done Done false
7583 3026 Acked Read 7 Sent Done Done false
7584 3027 Acked Read 4 Sent Done Done false
7585 3028 Acked Read 10 Sent Done Done false
7586 3029 Acked Write 8 Sent Done Done false
7587 3030 Acked Write 1 Sent Done Done false
7588 3031 Acked Read 4 Sent Done Done false
7589 3032 Acked Flush 0 Sent Done Done false
7590 3033 Acked Write 3 Sent Done Done false
7591 3034 Acked Read 7 Sent Done Done false
7592 3035 Acked Write 8 Sent Done Done false
7593 3036 Acked Read 10 Sent Done Done false
7594 3037 Acked Write 10 Sent Done Done false
7595 3038 Acked Read 8 Sent Done Done false
7596 3039 Acked Read 6 Sent Done Done false
7597 3040 Acked Write 8 Sent Done Done false
7598 3041 Acked Read 1 Sent Done Done false
7599 3042 Acked Write 4 Sent Done Done false
7600 3043 Acked Write 10 Sent Done Done false
7601 3044 Acked Read 1 Sent Done Done false
7602 3045 Acked Write 10 Sent Done Done false
7603 3046 Acked Read 8 Sent Done Done false
7604 3047 Acked Read 1 Sent Done Done false
7605 3048 Acked Read 8 Sent Done Done false
7606 3049 Acked Read 9 Sent Done Done false
7607 3050 Acked Read 10 Sent Done Done false
7608 3051 Acked Flush 0 Sent Done Done false
7609 3052 Acked Read 5 Sent Done Done false
7610 3053 Acked Flush 0 Sent Done Done false
7611 3054 Acked Write 2 Sent Done Done false
7612 3055 Acked Read 8 Sent Done Done false
7613 3056 Acked Write 3 Sent Done Done false
7614 3057 Acked Read 4 Sent Done Done false
7615 3058 Acked Write 7 Sent Done Done false
7616 3059 Acked Read 1 Sent Done Done false
7617 3060 Acked Read 4 Sent Done Done false
7618 3061 Acked Write 2 Sent Done Done false
7619 3062 Acked Write 3 Sent Done Done false
7620 3063 Acked Write 10 Sent Done Done false
7621 3064 Acked Flush 0 Sent Done Done false
7622 3065 Acked Read 8 Sent Done Done false
7623 3066 Acked Write 3 Sent Done Done false
7624 3067 Acked Write 4 Sent Done Done false
7625 3068 Acked Read 1 Sent Done Done false
7626 3069 Acked Read 5 Sent Done Done false
7627 3070 Acked Write 7 Sent Done Done false
7628 3071 Acked Read 7 Sent Done Done false
7629 3072 Acked Read 4 Sent Done Done false
7630 3073 Acked Read 6 Sent Done Done false
7631 3074 Acked Read 8 Sent Done Done false
7632 3075 Acked Write 5 Sent Done Done false
7633 3076 Acked Read 7 Sent Done Done false
7634 3077 Acked Write 1 Sent Done Done false
7635 3078 Acked Write 7 Sent Done Done false
7636 3079 Acked Write 6 Sent Done Done false
7637 3080 Acked Flush 0 Sent Done Done false
7638 3081 Acked Write 4 Sent Done Done false
7639 3082 Acked Flush 0 Sent Done Done false
7640 3083 Acked Read 10 Sent Done Done false
7641 3084 Acked Read 5 Sent Done Done false
7642 3085 Acked Read 1 Sent Done Done false
7643 3086 Acked Read 5 Sent Done Done false
7644 3087 Acked Read 7 Sent Done Done false
7645 3088 Acked Read 7 Sent Done Done false
7646 3089 Acked Read 6 Sent Done Done false
7647 3090 Acked Flush 0 Sent Done Done false
7648 3091 Acked Read 7 Sent Done Done false
7649 3092 Acked Write 3 Sent Done Done false
7650 3093 Acked Write 10 Sent Done Done false
7651 3094 Acked Flush 0 Sent Done Done false
7652 3095 Acked Write 8 Sent Done Done false
7653 3096 Acked Read 3 Sent Done Done false
7654 3097 Acked Read 7 Sent Done Done false
7655 3098 Acked Flush 0 Sent Done Done false
7656 3099 Acked Write 2 Sent Done Done false
7657 3100 Acked Flush 0 Sent Done Done false
7658 3101 Acked Read 8 Sent Done Done false
7659 3102 Acked Flush 0 Sent Done Done false
7660 3103 Acked Read 6 Sent Done Done false
7661 3104 Acked Read 3 Sent Done Done false
7662 3105 Acked Read 3 Sent Done Done false
7663 3106 Acked Flush 0 Sent Done Done false
7664 3107 Acked Flush 0 Sent Done Done false
7665 3108 Acked Write 6 Sent Done Done false
7666 3109 Acked Read 9 Sent Done Done false
7667 3110 Acked Write 2 Sent Done Done false
7668 3111 Acked Write 5 Sent Done Done false
7669 3112 Acked Read 7 Sent Done Done false
7670 STATES DS:0 DS:1 DS:2 TOTAL
7671 Sent 301 0 0 301
7672 Done 0 301 301 602
7673 Skipped 0 0 0 0
7674 Error 0 0 0 0
7675 Last Flush: Some(JobId(2811)) Some(JobId(3107)) Some(JobId(3107))
7676 Downstairs last five completed: 2811 2810 2809 2808 2807
7677 Upstairs last five completed: 3112 3111 3110 3109 3108
7678 CLIENT: Up:0 ds:301 act:2
76792025-01-28T21:55:56.471ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
76802025-01-28T21:55:57.356ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
76812025-01-28T21:55:57.356ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44705) = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
76822025-01-28T21:55:57.357ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
76832025-01-28T21:55:57.358ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
76842025-01-28T21:55:57.358ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
76852025-01-28T21:55:57.358ZINFOcrucible: send last flush ID to this DS: Some(JobId(2811)) = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
76862025-01-28T21:55:57.358ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
76872025-01-28T21:55:57.358ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2811)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
76882025-01-28T21:55:57.358ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2811)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
7689 ----------------------------------------------------------------
7690 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7691 Downstairs last five completed: 3113 3112 3111 3110 3109
7692 Upstairs last five completed: 3113 3112 3111 3110 3109
7693 CLIENT: Up:0 ds:0 act:3
7694 Replay: All jobs finished
7695 Main task has work to do, go find it
7696 got dsc Stop(0)
7697 stop 0
7698 Jan 28 21:55:59.603[8810] Got stop action so:false kr:false
7699 INFO request completed, latency_us: 364, response_code: 204, uri: /stop/cid/0, method: POST, req_id: f0af1f17-070c-4adc-bfb2-74b021d1575e, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
7700 Jan 28 21:55:59.604 INFO request completed, latency_us: 129, response_code: 200, uri: /state/cid/0, method{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:55:59.603971913Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
7701 : GET, req_id: d4246d36-82a5-47ec-9d5e-d05074f641b0, remote_addr: 127.0.0.1:36665, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:55:59.604061307Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
7702 local_addr: 127.0.0.1:9998
77032025-01-28T21:55:59.604ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
77042025-01-28T21:55:59.604ZINFOcrucible: client task is exiting = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
7705 [8810] Exited with: ExitStatus(unix_wait_status(9))
7706 [8810][0] reports Exit
7707 Jan 28 21:56:03.606 INFO request completed, latency_us: 228, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5add234a-2bc8-453e-adc1-e3115ef6249f, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
7708 001/300 Read block 1848 len 32768
7709 002/300 Read block 985 len 24576
7710 003/300 Read block 1280 len 40960
7711 004/300 Read block 4159 len 4096
7712 005/300 Write block 4066 len 20480 data: 2 2 2 2 2
7713 006/300 Read block 1311 len 8192
7714 007/300 Flush
7715 008/300 Read block 1569 len 40960
7716 009/300 Read block 1108 len 20480
7717 010/300 Write block 119 len 32768 data: 2 2 2 2 3 3 3 4
7718 011/300 Write block 2003 len 24576 data: 2 2 2 2 2 3
7719 012/300 Read block 2225 len 20480
7720 013/300 Read block 3647 len 4096
7721 014/300 Read block 1271 len 12288
7722 015/300 Read block 2746 len 28672
7723 016/300 Read block 536 len 36864
7724 017/300 Write block 3718 len 32768 data: 3 3 3 3 3 3 3 3
7725 018/300 Read block 3169 len 36864
7726 019/300 Write block 950 len 12288 data: 3 3 3
7727 020/300 Write block 2629 len 20480 data: 2 2 2 2 3
7728 021/300 Write block 1681 len 12288 data: 2 2 2
7729 022/300 Flush
7730 023/300 Read block 2589 len 32768
7731 024/300 Write block 4171 len 8192 data: 4 4
7732 025/300 Flush
7733 026/300 Write block 2937 len 40960 data: 3 4 4 5 4 4 4 3 4 4
7734 027/300 Flush
7735 028/300 Read block 2186 len 28672
7736 029/300 Flush
7737 030/300 Read block 407 len 28672
7738 031/300 Write block 4883 len 12288 data: 4 5 4
7739 032/300 Write block 2660 len 40960 data: 4 4 3 2 2 2 2 2 2 2
7740 033/300 Read block 4496 len 32768
7741 034/300 Write block 208 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7742 035/300 Write block 1825 len 12288 data: 3 3 3
7743 036/300 Read block 2300 len 8192
7744 037/300 Write block 4144 len 36864 data: 2 2 2 2 2 2 2 2 2
7745 038/300 Write block 3870 len 16384 data: 3 3 3 3
7746 039/300 Read block 2386 len 28672
7747 040/300 Read block 137 len 8192
7748 041/300 Read block 4492 len 28672
7749 042/300 Flush
7750 043/300 Read block 540 len 16384
7751 044/300 Write block 2784 len 4096 data: 2
7752 045/300 Write block 3524 len 32768 data: 4 4 5 5 5 4 4 4
7753 046/300 Write block 2036 len 20480 data: 2 2 2 2 2
7754 047/300 Write block 3779 len 20480 data: 3 3 4 4 4
7755 048/300 Read block 3245 len 28672
7756 049/300 Flush
7757 050/300 Write block 219 len 16384 data: 2 2 2 3
7758 051/300 Write block 4694 len 8192 data: 4 4
7759 052/300 Read block 3815 len 32768
7760 053/300 Read block 3471 len 40960
7761 054/300 Read block 1567 len 40960
7762 055/300 Write block 2123 len 20480 data: 4 5 4 4 4
7763 056/300 Write block 2858 len 8192 data: 4 3
7764 057/300 Read block 3697 len 36864
7765 058/300 Write block 2107 len 4096 data: 3
7766 059/300 Read block 1036 len 20480
7767 060/300 Read block 1967 len 28672
7768 061/300 Read block 2367 len 36864
7769 062/300 Read block 1344 len 20480
7770 063/300 Write block 4235 len 8192 data: 2 2
7771 064/300 Write block 1924 len 16384 data: 3 3 2 2
7772 065/300 Read block 3691 len 12288
7773 066/300 Write block 1951 len 40960 data: 2 2 3 3 3 3 3 2 2 2
7774 067/300 Flush
7775 068/300 Read block 726 len 8192
7776 069/300 Flush
7777 070/300 Read block 4385 len 8192
7778 071/300 Read block 2704 len 24576
7779 072/300 Write block 2967 len 12288 data: 2 2 2
7780 073/300 Write block 381 len 16384 data: 2 3 3 3
7781 074/300 Write block 778 len 12288 data: 2 2 2
7782 075/300 Write block 4156 len 8192 data: 2 2
7783 076/300 Write block 3220 len 24576 data: 3 2 2 2 2 3
7784 077/300 Read block 4191 len 24576
7785 078/300 Read block 1674 len 4096
7786 079/300 Write block 4141 len 20480 data: 2 2 2 3 3
7787 080/300 Read block 4067 len 16384
7788 081/300 Write block 847 len 20480 data: 3 3 2 2 2
7789 082/300 Write block 2970 len 12288 data: 2 3 3
7790 083/300 Read block 1723 len 8192
7791 084/300 Read block 1910 len 20480
7792 085/300 Read block 4100 len 32768
7793 086/300 Read block 4778 len 16384
7794 087/300 Read block 2831 len 8192
7795 088/300 Read block 3368 len 36864
7796 089/300 Read block 1976 len 28672
7797 090/300 Flush
7798 091/300 Read block 4466 len 20480
7799 092/300 Read block 2582 len 24576
7800 093/300 Read block 2497 len 4096
7801 094/300 Read block 625 len 8192
7802 095/300 Read block 659 len 20480
7803 096/300 Read block 1270 len 8192
7804 097/300 Flush
7805 098/300 Read block 4761 len 12288
7806 099/300 Read block 1945 len 4096
7807 100/300 Read block 2118 len 28672
7808 101/300 Flush
7809 102/300 Write block 4565 len 16384 data: 6 6 5 5
7810 103/300 Read block 409 len 40960
7811 104/300 Read block 3688 len 40960
7812 105/300 Read block 3354 len 40960
7813 106/300 Write block 3728 len 24576 data: 4 4 5 4 4 4
7814 107/300 Write block 1206 len 28672 data: 5 4 4 4 4 4 4
7815 108/300 Read block 2496 len 36864
7816 109/300 Read block 2911 len 8192
7817 110/300 Write block 1452 len 20480 data: 2 2 2 2 3
7818 111/300 Read block 1111 len 20480
7819 112/300 Write block 651 len 12288 data: 4 4 4
7820 113/300 Read block 3493 len 36864
7821 114/300 Read block 4630 len 16384
7822 115/300 Write block 1944 len 16384 data: 3 2 2 2
7823 116/300 Write block 741 len 8192 data: 4 3
7824 117/300 Write block 397 len 40960 data: 3 3 3 3 3 3 3 3 2 2
7825 118/300 Read block 2650 len 8192
7826 119/300 Write block 3588 len 20480 data: 3 3 3 3 3
7827 120/300 Read block 4990 len 20480
7828 121/300 Read block 1130 len 20480
7829 122/300 Read block 1601 len 40960
7830 123/300 Read block 2099 len 28672
7831 124/300 Write block 3517 len 28672 data: 4 4 4 3 3 3 3
7832 125/300 Read block 3452 len 12288
7833 126/300 Read block 2149 len 4096
7834 127/300 Flush
7835 128/300 Write block 1816 len 16384 data: 2 2 3 4
7836 129/300 Read block 4678 len 28672
7837 130/300 Write block 1574 len 12288 data: 2 2 2
7838 131/300 Read block 166 len 12288
7839 132/300 Flush
7840 133/300 Write block 1033 len 8192 data: 4 4
7841 134/300 Read block 3107 len 20480
7842 135/300 Write block 1217 len 40960 data: 2 2 2 2 3 3 3 2 2 2
7843 136/300 Read block 439 len 32768
7844 137/300 Write block 3648 len 20480 data: 3 2 2 2 2
7845 138/300 Write block 2333 len 20480 data: 4 4 2 2 4
7846 139/300 Read block 3419 len 32768
7847 140/300 Flush
7848 141/300 Read block 2951 len 32768
7849 142/300 Write block 2241 len 8192 data: 2 2
7850 143/300 Write block 516 len 16384 data: 2 2 2 2
7851 144/300 Read block 1852 len 20480
7852 145/300 Write block 322 len 12288 data: 2 3 3
7853 146/300 Write block 2802 len 8192 data: 3 3
7854 147/300 Write block 4403 len 16384 data: 4 4 3 3
7855 148/300 Write block 1061 len 20480 data: 4 4 4 4 5
7856 149/300 Flush
7857 150/300 Flush
7858 151/300 Read block 724 len 24576
7859 152/300 Read block 3918 len 12288
7860 153/300 Write block 187 len 28672 data: 2 2 2 2 2 2 2
7861 154/300 Write block 1862 len 16384 data: 3 3 3 3
7862 155/300 Flush
7863 156/300 Read block 2911 len 20480
7864 157/300 Read block 2787 len 32768
7865 158/300 Write block 963 len 16384 data: 4 4 4 5
7866 159/300 Read block 4578 len 24576
7867 160/300 Read block 87 len 16384
7868 161/300 Read block 639 len 12288
7869 162/300 Write block 1794 len 24576 data: 2 2 2 2 3 3
7870 163/300 Read block 3444 len 36864
7871 164/300 Write block 1944 len 36864 data: 4 3 3 3 2 2 2 3 3
7872 165/300 Write block 3418 len 36864 data: 2 2 2 3 3 3 3 3 2
7873 166/300 Read block 1881 len 28672
7874 167/300 Flush
7875 168/300 Write block 3423 len 4096 data: 4
7876 169/300 Read block 2921 len 24576
7877 170/300 Read block 2959 len 32768
7878 171/300 Read block 1136 len 4096
7879 172/300 Write block 614 len 32768 data: 2 2 3 3 3 3 3 2
7880 173/300 Flush
7881 174/300 Read block 2065 len 12288
7882 175/300 Write block 1204 len 40960 data: 7 6 6 5 5 5 5 5 5 2
7883 176/300 Flush
7884 177/300 Read block 4283 len 4096
7885 178/300 Write block 4969 len 8192 data: 3 3
7886 179/300 Write block 181 len 20480 data: 2 2 2 2 2
7887 180/300 Write block 3633 len 4096 data: 2
7888 181/300 Read block 1818 len 20480
7889 182/300 Flush
7890 183/300 Write block 4946 len 28672 data: 3 4 2 3 4 4 4
7891 184/300 Flush
7892 185/300 Write block 4776 len 24576 data: 2 2 2 2 2 2
7893 186/300 Read block 2938 len 4096
7894 187/300 Read block 908 len 24576
7895 188/300 Write block 4157 len 36864 data: 3 2 2 2 2 2 3 4 4
7896 189/300 Read block 2560 len 20480
7897 190/300 Read block 2582 len 12288
7898 191/300 Write block 319 len 8192 data: 3 3
7899 192/300 Write block 395 len 24576 data: 3 3 4 4 4 4
7900 193/300 Write block 1193 len 4096 data: 3
7901 194/300 Read block 2432 len 40960
7902 195/300 Write block 3385 len 24576 data: 2 2 2 4 4 3
7903 196/300 Read block 3302 len 16384
7904 197/300 Write block 3385 len 36864 data: 3 3 3 5 5 4 4 4 4
7905 198/300 Write block 152 len 28672 data: 3 3 3 3 3 3 3
7906 199/300 Read block 2993 len 12288
7907 200/300 Read block 4354 len 12288
7908 201/300 Read block 2932 len 4096
7909 202/300 Write block 660 len 20480 data: 2 2 2 2 3
7910 203/300 Flush
7911 204/300 Read block 1486 len 20480
7912 205/300 Read block 2436 len 4096
7913 206/300 Write block 574 len 4096 data: 2
7914 207/300 Read block 3402 len 32768
7915 208/300 Write block 1468 len 24576 data: 3 3 2 2 2 2
7916 209/300 Write block 722 len 36864 data: 3 2 2 2 2 2 2 3 3
7917 210/300 Read block 2138 len 4096
7918 211/300 Write block 69 len 28672 data: 3 3 2 3 3 4 3
7919 212/300 Write block 4845 len 28672 data: 3 3 3 2 2 3 3
7920 213/300 Write block 3754 len 28672 data: 2 2 3 3 3 3 4
7921 214/300 Write block 4627 len 40960 data: 3 3 3 2 2 2 2 2 2 2
7922 215/300 Read block 4143 len 36864
7923 216/300 Read block 3831 len 36864
7924 217/300 Read block 2760 len 36864
7925 218/300 Read block 305 len 16384
7926 219/300 Write block 2099 len 28672 data: 2 2 2 2 3 2 2
7927 220/300 Read block 123 len 40960
7928 221/300 Read block 3670 len 40960
7929 222/300 Write block 2613 len 4096 data: 3
7930 223/300 Write block 3699 len 20480 data: 2 2 2 2 2
7931 224/300 Flush
7932 225/300 Read block 2778 len 8192
7933 226/300 Read block 692 len 28672
7934 227/300 Read block 3850 len 32768
7935 228/300 Write block 4792 len 4096 data: 2
7936 229/300 Read block 330 len 16384
7937 230/300 Read block 2866 len 24576
7938 231/300 Write block 2428 len 28672 data: 5 5 5 5 4 4 3
7939 232/300 Flush
7940 233/300 Read block 649 len 4096
7941 234/300 Write block 3239 len 8192 data: 2 2
7942 235/300 Flush
7943 236/300 Read block 1231 len 16384
7944 237/300 Read block 48 len 32768
7945 238/300 Read block 2401 len 40960
7946 239/300 Write block 1117 len 28672 data: 3 4 4 4 4 4 3
7947 240/300 Read block 2875 len 40960
7948 241/300 Read block 4502 len 12288
7949 242/300 Write block 1430 len 16384 data: 2 2 2 2
7950 243/300 Write block 3298 len 24576 data: 3 3 2 3 3 3
7951 244/300 Read block 2447 len 20480
7952 245/300 Read block 1424 len 20480
7953 246/300 Read block 1507 len 8192
7954 247/300 Flush
7955 248/300 Write block 2696 len 8192 data: 4 4
7956 249/300 Write block 2130 len 36864 data: 3 3 4 5 5 6 4 4 5
7957 250/300 Read block 3735 len 28672
7958 251/300 Write block 713 len 16384 data: 3 2 2 2
7959 252/300 Read block 3279 len 24576
7960 253/300 Write block 318 len 8192 data: 3 4
7961 254/300 Write block 172 len 20480 data: 4 4 4 4 4
7962 255/300 Write block 2052 len 28672 data: 2 2 2 2 2 2 2
7963 256/300 Write block 691 len 4096 data: 5
7964 257/300 Read block 4866 len 8192
7965 258/300 Read block 2578 len 4096
7966 259/300 Write block 3839 len 8192 data: 3 3
7967 260/300 Write block 4059 len 20480 data: 3 3 3 3 3
7968 261/300 Read block 3000 len 40960
7969 262/300 Read block 1171 len 12288
7970 263/300 Write block 1100 len 28672 data: 3 3 2 2 2 4 4
7971 264/300 Write block 2762 len 4096 data: 3
7972 265/300 Write block 3642 len 40960 data: 3 3 3 3 3 3 4 3 3 3
7973 266/300 Read block 3764 len 36864
7974 267/300 Flush
7975 268/300 Flush
7976 269/300 Write block 646 len 40960 data: 3 3 3 3 3 5 5 5 3 3
7977 270/300 Read block 3039 len 32768
7978 271/300 Write block 3819 len 8192 data: 4 5
7979 272/300 Write block 3597 len 36864 data: 3 3 4 3 3 3 3 3 3
7980 273/300 Read block 1568 len 20480
7981 274/300 Read block 2768 len 12288
7982 275/300 Read block 3929 len 36864
7983 276/300 Write block 2130 len 20480 data: 4 4 5 6 6
7984 277/300 Write block 4916 len 16384 data: 3 2 2 2
7985 278/300 Read block 3890 len 8192
7986 279/300 Read block 1361 len 16384
7987 280/300 Read block 3215 len 24576
7988 281/300 Write block 4364 len 4096 data: 3
7989 282/300 Read block 3250 len 36864
7990 283/300 Flush
7991 284/300 Write block 1280 len 4096 data: 2
7992 285/300 Read block 3379 len 24576
7993 286/300 Write block 2992 len 28672 data: 4 4 5 5 4 4 4
7994 287/300 Read block 4821 len 24576
7995 288/300 Write block 1980 len 36864 data: 3 3 3 3 3 3 3 3 3
7996 289/300 Write block 790 len 20480 data: 3 3 3 3 3
7997 290/300 Read block 4585 len 20480
7998 291/300 Flush
7999 292/300 Read block 438 len 32768
8000 293/300 Read block 1626 len 20480
8001 294/300 Read block 4844 len 32768
8002 295/300 Read block 1969 len 28672
8003 296/300 Read block 2451 len 20480
8004 297/300 Write block 4194 len 28672 data: 2 2 2 3 2 2 2
8005 298/300 Read block 3010 len 20480
8006 299/300 Write block 1033 len 4096 data: 5
8007 300/300 Write block 2705 len 32768 data: 2 2 2 2 2 2 2 2
8008 301/300 Read block 2154 len 36864
8009 Main task has work to do, go find it
8010 got dsc Start(0)
8011 start 0
8012 Jan 28 21:56:03.843 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/0, method: POST, req_id: 0bff2ba2-6486-4de5-8dd1-64efc89bf5ba, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
8013 Replay: started 0, returned:Ok(())
8014 ----------------------------------------------------------------
8015 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8016 States: Offline Active Active
8017 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8018 3114 Acked Read 8 Sent Done Done false
8019 3115 Acked Read 6 Sent Done Done false
8020 3116 Acked Read 10 Sent Done Done false
8021 3117 Acked Read 1 Sent Done Done false
8022 3118 Acked Write 5 Sent Done Done false
8023 3119 Acked Read 2 Sent Done Done false
8024 3120 Acked Flush 0 Sent Done Done false
8025 3121 Acked Read 10 Sent Done Done false
8026 3122 Acked Read 5 Sent Done Done false
8027 3123 Acked Write 8 Sent Done Done false
8028 3124 Acked Write 6 Sent Done Done false
8029 3125 Acked Read 5 Sent Done Done false
8030 3126 Acked Read 1 Sent Done Done false
8031 3127 Acked Read 3 Sent Done Done false
8032 3128 Acked Read 7 Sent Done Done false
8033 3129 Acked Read 9 Sent Done Done false
8034 3130 Acked Write 8 Sent Done Done false
8035 3131 Acked Read 9 Sent Done Done false
8036 3132 Acked Write 3 Sent Done Done false
8037 3133 Acked Write 5 Sent Done Done false
8038 3134 Acked Write 3 Sent Done Done false
8039 3135 Acked Flush 0 Sent Done Done false
8040 3136 Acked Read 8 Sent Done Done false
8041 3137 Acked Write 2 Sent Done Done false
8042 3138 Acked Flush 0 Sent Done Done false
8043 3139 Acked Write 10 Sent Done Done false
8044 3140 Acked Flush 0 Sent Done Done false
8045 3141 Acked Read 7 Sent Done Done false
8046 3142 Acked Flush 0 Sent Done Done false
8047 3143 Acked Read 7 Sent Done Done false
8048 3144 Acked Write 3 Sent Done Done false
8049 3145 Acked Write 10 Sent Done Done false
8050 3146 Acked Read 8 Sent Done Done false
8051 3147 Acked Write 10 Sent Done Done false
8052 3148 Acked Write 3 Sent Done Done false
8053 3149 Acked Read 2 Sent Done Done false
8054 3150 Acked Write 9 Sent Done Done false
8055 3151 Acked Write 4 Sent Done Done false
8056 3152 Acked Read 7 Sent Done Done false
8057 3153 Acked Read 2 Sent Done Done false
8058 3154 Acked Read 7 Sent Done Done false
8059 3155 Acked Flush 0 Sent Done Done false
8060 3156 Acked Read 4 Sent Done Done false
8061 3157 Acked Write 1 Sent Done Done false
8062 3158 Acked Write 8 Sent Done Done false
8063 3159 Acked Write 5 Sent Done Done false
8064 3160 Acked Write 5 Sent Done Done false
8065 3161 Acked Read 7 Sent Done Done false
8066 3162 Acked Flush 0 Sent Done Done false
8067 3163 Acked Write 4 Sent Done Done false
8068 3164 Acked Write 2 Sent Done Done false
8069 3165 Acked Read 8 Sent Done Done false
8070 3166 Acked Read 10 Sent Done Done false
8071 3167 Acked Read 10 Sent Done Done false
8072 3168 Acked Write 5 Sent Done Done false
8073 3169 Acked Write 2 Sent Done Done false
8074 3170 Acked Read 9 Sent Done Done false
8075 3171 Acked Write 1 Sent Done Done false
8076 3172 Acked Read 5 Sent Done Done false
8077 3173 Acked Read 7 Sent Done Done false
8078 3174 Acked Read 9 Sent Done Done false
8079 3175 Acked Read 5 Sent Done Done false
8080 3176 Acked Write 2 Sent Done Done false
8081 3177 Acked Write 4 Sent Done Done false
8082 3178 Acked Read 3 Sent Done Done false
8083 3179 Acked Write 10 Sent Done Done false
8084 3180 Acked Flush 0 Sent Done Done false
8085 3181 Acked Read 2 Sent Done Done false
8086 3182 Acked Flush 0 Sent Done Done false
8087 3183 Acked Read 2 Sent Done Done false
8088 3184 Acked Read 6 Sent Done Done false
8089 3185 Acked Write 3 Sent Done Done false
8090 3186 Acked Write 4 Sent Done Done false
8091 3187 Acked Write 3 Sent Done Done false
8092 3188 Acked Write 2 Sent Done Done false
8093 3189 Acked Write 6 Sent Done Done false
8094 3190 Acked Read 6 Sent Done Done false
8095 3191 Acked Read 1 Sent Done Done false
8096 3192 Acked Write 5 Sent Done Done false
8097 3193 Acked Read 4 Sent Done Done false
8098 3194 Acked Write 5 Sent Done Done false
8099 3195 Acked Write 3 Sent Done Done false
8100 3196 Acked Read 2 Sent Done Done false
8101 3197 Acked Read 5 Sent Done Done false
8102 3198 Acked Read 8 Sent Done Done false
8103 3199 Acked Read 4 Sent Done Done false
8104 3200 Acked Read 2 Sent Done Done false
8105 3201 Acked Read 9 Sent Done Done false
8106 3202 Acked Read 7 Sent Done Done false
8107 3203 Acked Flush 0 Sent Done Done false
8108 3204 Acked Read 5 Sent Done Done false
8109 3205 Acked Read 6 Sent Done Done false
8110 3206 Acked Read 1 Sent Done Done false
8111 3207 Acked Read 2 Sent Done Done false
8112 3208 Acked Read 5 Sent Done Done false
8113 3209 Acked Read 2 Sent Done Done false
8114 3210 Acked Flush 0 Sent Done Done false
8115 3211 Acked Read 3 Sent Done Done false
8116 3212 Acked Read 1 Sent Done Done false
8117 3213 Acked Read 7 Sent Done Done false
8118 3214 Acked Flush 0 Sent Done Done false
8119 3215 Acked Write 4 Sent Done Done false
8120 3216 Acked Read 10 Sent Done Done false
8121 3217 Acked Read 10 Sent Done Done false
8122 3218 Acked Read 10 Sent Done Done false
8123 3219 Acked Write 6 Sent Done Done false
8124 3220 Acked Write 7 Sent Done Done false
8125 3221 Acked Read 9 Sent Done Done false
8126 3222 Acked Read 2 Sent Done Done false
8127 3223 Acked Write 5 Sent Done Done false
8128 3224 Acked Read 5 Sent Done Done false
8129 3225 Acked Write 3 Sent Done Done false
8130 3226 Acked Read 9 Sent Done Done false
8131 3227 Acked Read 4 Sent Done Done false
8132 3228 Acked Write 4 Sent Done Done false
8133 3229 Acked Write 2 Sent Done Done false
8134 3230 Acked Write 10 Sent Done Done false
8135 3231 Acked Read 2 Sent Done Done false
8136 3232 Acked Write 5 Sent Done Done false
8137 3233 Acked Read 5 Sent Done Done false
8138 3234 Acked Read 5 Sent Done Done false
8139 3235 Acked Read 10 Sent Done Done false
8140 3236 Acked Read 7 Sent Done Done false
8141 3237 Acked Write 7 Sent Done Done false
8142 3238 Acked Read 3 Sent Done Done false
8143 3239 Acked Read 1 Sent Done Done false
8144 3240 Acked Flush 0 Sent Done Done false
8145 3241 Acked Write 4 Sent Done Done false
8146 3242 Acked Read 7 Sent Done Done false
8147 3243 Acked Write 3 Sent Done Done false
8148 3244 Acked Read 3 Sent Done Done false
8149 3245 Acked Flush 0 Sent Done Done false
8150 3246 Acked Write 2 Sent Done Done false
8151 3247 Acked Read 5 Sent Done Done false
8152 3248 Acked Write 10 Sent Done Done false
8153 3249 Acked Read 8 Sent Done Done false
8154 3250 Acked Write 5 Sent Done Done false
8155 3251 Acked Write 5 Sent Done Done false
8156 3252 Acked Read 8 Sent Done Done false
8157 3253 Acked Flush 0 Sent Done Done false
8158 3254 Acked Read 8 Sent Done Done false
8159 3255 Acked Write 2 Sent Done Done false
8160 3256 Acked Write 4 Sent Done Done false
8161 3257 Acked Read 5 Sent Done Done false
8162 3258 Acked Write 3 Sent Done Done false
8163 3259 Acked Write 2 Sent Done Done false
8164 3260 Acked Write 4 Sent Done Done false
8165 3261 Acked Write 5 Sent Done Done false
8166 3262 Acked Flush 0 Sent Done Done false
8167 3263 Acked Flush 0 Sent Done Done false
8168 3264 Acked Read 6 Sent Done Done false
8169 3265 Acked Read 3 Sent Done Done false
8170 3266 Acked Write 7 Sent Done Done false
8171 3267 Acked Write 4 Sent Done Done false
8172 3268 Acked Flush 0 Sent Done Done false
8173 3269 Acked Read 5 Sent Done Done false
8174 3270 Acked Read 8 Sent Done Done false
8175 3271 Acked Write 4 Sent Done Done false
8176 3272 Acked Read 6 Sent Done Done false
8177 3273 Acked Read 4 Sent Done Done false
8178 3274 Acked Read 3 Sent Done Done false
8179 3275 Acked Write 6 Sent Done Done false
8180 3276 Acked Read 9 Sent Done Done false
8181 3277 Acked Write 9 Sent Done Done false
8182 3278 Acked Write 9 Sent Done Done false
8183 3279 Acked Read 7 Sent Done Done false
8184 3280 Acked Flush 0 Sent Done Done false
8185 3281 Acked Write 1 Sent Done Done false
8186 3282 Acked Read 6 Sent Done Done false
8187 3283 Acked Read 8 Sent Done Done false
8188 3284 Acked Read 1 Sent Done Done false
8189 3285 Acked Write 8 Sent Done Done false
8190 3286 Acked Flush 0 Sent Done Done false
8191 3287 Acked Read 3 Sent Done Done false
8192 3288 Acked Write 10 Sent Done Done false
8193 3289 Acked Flush 0 Sent Done Done false
8194 3290 Acked Read 1 Sent Done Done false
8195 3291 Acked Write 2 Sent Done Done false
8196 3292 Acked Write 5 Sent Done Done false
8197 3293 Acked Write 1 Sent Done Done false
8198 3294 Acked Read 5 Sent Done Done false
8199 3295 Acked Flush 0 Sent Done Done false
8200 3296 Acked Write 7 Sent Done Done false
8201 3297 Acked Flush 0 Sent Done Done false
8202 3298 Acked Write 6 Sent Done Done false
8203 3299 Acked Read 1 Sent Done Done false
8204 3300 Acked Read 6 Sent Done Done false
8205 3301 Acked Write 9 Sent Done Done false
8206 3302 Acked Read 5 Sent Done Done false
8207 3303 Acked Read 3 Sent Done Done false
8208 3304 Acked Write 2 Sent Done Done false
8209 3305 Acked Write 6 Sent Done Done false
8210 3306 Acked Write 1 Sent Done Done false
8211 3307 Acked Read 10 Sent Done Done false
8212 3308 Acked Write 6 Sent Done Done false
8213 3309 Acked Read 4 Sent Done Done false
8214 3310 Acked Write 9 Sent Done Done false
8215 3311 Acked Write 7 Sent Done Done false
8216 3312 Acked Read 3 Sent Done Done false
8217 3313 Acked Read 3 Sent Done Done false
8218 3314 Acked Read 1 Sent Done Done false
8219 3315 Acked Write 5 Sent Done Done false
8220 3316 Acked Flush 0 Sent Done Done false
8221 3317 Acked Read 5 Sent Done Done false
8222 3318 Acked Read 1 Sent Done Done false
8223 3319 Acked Write 1 Sent Done Done false
8224 3320 Acked Read 8 Sent Done Done false
8225 3321 Acked Write 6 Sent Done Done false
8226 3322 Acked Write 9 Sent Done Done false
8227 3323 Acked Read 1 Sent Done Done false
8228 3324 Acked Write 7 Sent Done Done false
8229 3325 Acked Write 7 Sent Done Done false
8230 3326 Acked Write 7 Sent Done Done false
8231 3327 Acked Write 10 Sent Done Done false
8232 3328 Acked Read 9 Sent Done Done false
8233 3329 Acked Read 9 Sent Done Done false
8234 3330 Acked Read 9 Sent Done Done false
8235 3331 Acked Read 4 Sent Done Done false
8236 3332 Acked Write 7 Sent Done Done false
8237 3333 Acked Read 10 Sent Done Done false
8238 3334 Acked Read 10 Sent Done Done false
8239 3335 Acked Write 1 Sent Done Done false
8240 3336 Acked Write 5 Sent Done Done false
8241 3337 Acked Flush 0 Sent Done Done false
8242 3338 Acked Read 2 Sent Done Done false
8243 3339 Acked Read 7 Sent Done Done false
8244 3340 Acked Read 8 Sent Done Done false
8245 3341 Acked Write 1 Sent Done Done false
8246 3342 Acked Read 4 Sent Done Done false
8247 3343 Acked Read 6 Sent Done Done false
8248 3344 Acked Write 7 Sent Done Done false
8249 3345 Acked Flush 0 Sent Done Done false
8250 3346 Acked Read 1 Sent Done Done false
8251 3347 Acked Write 2 Sent Done Done false
8252 3348 Acked Flush 0 Sent Done Done false
8253 3349 Acked Read 4 Sent Done Done false
8254 3350 Acked Read 8 Sent Done Done false
8255 3351 Acked Read 10 Sent Done Done false
8256 3352 Acked Write 7 Sent Done Done false
8257 3353 Acked Read 10 Sent Done Done false
8258 3354 Acked Read 3 Sent Done Done false
8259 3355 Acked Write 4 Sent Done Done false
8260 3356 Acked Write 6 Sent Done Done false
8261 3357 Acked Read 5 Sent Done Done false
8262 3358 Acked Read 5 Sent Done Done false
8263 3359 Acked Read 2 Sent Done Done false
8264 3360 Acked Flush 0 Sent Done Done false
8265 3361 Acked Write 2 Sent Done Done false
8266 3362 Acked Write 9 Sent Done Done false
8267 3363 Acked Read 7 Sent Done Done false
8268 3364 Acked Write 4 Sent Done Done false
8269 3365 Acked Read 6 Sent Done Done false
8270 3366 Acked Write 2 Sent Done Done false
8271 3367 Acked Write 5 Sent Done Done false
8272 3368 Acked Write 7 Sent Done Done false
8273 3369 Acked Write 1 Sent Done Done false
8274 3370 Acked Read 2 Sent Done Done false
8275 3371 Acked Read 1 Sent Done Done false
8276 3372 Acked Write 2 Sent Done Done false
8277 3373 Acked Write 5 Sent Done Done false
8278 3374 Acked Read 10 Sent Done Done false
8279 3375 Acked Read 3 Sent Done Done false
8280 3376 Acked Write 7 Sent Done Done false
8281 3377 Acked Write 1 Sent Done Done false
8282 3378 Acked Write 10 Sent Done Done false
8283 3379 Acked Read 9 Sent Done Done false
8284 3380 Acked Flush 0 Sent Done Done false
8285 3381 Acked Flush 0 Sent Done Done false
8286 3382 Acked Write 10 Sent Done Done false
8287 3383 Acked Read 8 Sent Done Done false
8288 3384 Acked Write 2 Sent Done Done false
8289 3385 Acked Write 9 Sent Done Done false
8290 3386 Acked Read 5 Sent Done Done false
8291 3387 Acked Read 3 Sent Done Done false
8292 3388 Acked Read 9 Sent Done Done false
8293 3389 Acked Write 5 Sent Done Done false
8294 3390 Acked Write 4 Sent Done Done false
8295 3391 Acked Read 2 Sent Done Done false
8296 3392 Acked Read 4 Sent Done Done false
8297 3393 Acked Read 6 Sent Done Done false
8298 3394 Acked Write 1 Sent Done Done false
8299 3395 Acked Read 9 Sent Done Done false
8300 3396 Acked Flush 0 Sent Done Done false
8301 3397 Acked Write 1 Sent Done Done false
8302 3398 Acked Read 6 Sent Done Done false
8303 3399 Acked Write 7 Sent Done Done false
8304 3400 Acked Read 6 Sent Done Done false
8305 3401 Acked Write 9 Sent Done Done false
8306 3402 Acked Write 5 Sent Done Done false
8307 3403 Acked Read 5 Sent Done Done false
8308 3404 Acked Flush 0 Sent Done Done false
8309 3405 Acked Read 8 Sent Done Done false
8310 3406 Acked Read 5 Sent Done Done false
8311 3407 Acked Read 8 Sent Done Done false
8312 3408 Acked Read 7 Sent Done Done false
8313 3409 Acked Read 5 Sent Done Done false
8314 3410 Acked Write 7 Sent Done Done false
8315 3411 Acked Read 5 Sent Done Done false
8316 3412 Acked Write 1 Sent Done Done false
8317 3413 Acked Write 8 Sent Done Done false
8318 3414 Acked Read 9 Sent Done Done false
8319 STATES DS:0 DS:1 DS:2 TOTAL
8320 Sent 301 0 0 301
8321 Done 0 301 301 602
8322 Skipped 0 0 0 0
8323 Error 0 0 0 0
8324 Last Flush: Some(JobId(3113)) Some(JobId(3404)) Some(JobId(3404))
8325 Downstairs last five completed: 3113 3112 3111 3110 3109
8326 Upstairs last five completed: 3414 3413 3412 3411 3410
8327 CLIENT: Up:0 ds:301 act:2
83282025-01-28T21:56:04.482ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
83292025-01-28T21:56:05.483ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
83302025-01-28T21:56:06.484ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
83312025-01-28T21:56:07.486ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
8332 [8810] got start action
8333 [8810] I am going to restart
8334 Starting downstairs at port 8810
8335 Make output file at "/tmp/dsc/downstairs-8810.txt"
8336 [8810][0] reports Starting
8337 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1210)
8338 [8810][0] reports Running
8339 ----------------------------------------------------------------
8340 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8341 States: Offline Active Active
8342 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8343 3114 Acked Read 8 Sent Done Done false
8344 3115 Acked Read 6 Sent Done Done false
8345 3116 Acked Read 10 Sent Done Done false
8346 3117 Acked Read 1 Sent Done Done false
8347 3118 Acked Write 5 Sent Done Done false
8348 3119 Acked Read 2 Sent Done Done false
8349 3120 Acked Flush 0 Sent Done Done false
8350 3121 Acked Read 10 Sent Done Done false
8351 3122 Acked Read 5 Sent Done Done false
8352 3123 Acked Write 8 Sent Done Done false
8353 3124 Acked Write 6 Sent Done Done false
8354 3125 Acked Read 5 Sent Done Done false
8355 3126 Acked Read 1 Sent Done Done false
8356 3127 Acked Read 3 Sent Done Done false
8357 3128 Acked Read 7 Sent Done Done false
8358 3129 Acked Read 9 Sent Done Done false
8359 3130 Acked Write 8 Sent Done Done false
8360 3131 Acked Read 9 Sent Done Done false
8361 3132 Acked Write 3 Sent Done Done false
8362 3133 Acked Write 5 Sent Done Done false
8363 3134 Acked Write 3 Sent Done Done false
8364 3135 Acked Flush 0 Sent Done Done false
8365 3136 Acked Read 8 Sent Done Done false
8366 3137 Acked Write 2 Sent Done Done false
8367 3138 Acked Flush 0 Sent Done Done false
8368 3139 Acked Write 10 Sent Done Done false
8369 3140 Acked Flush 0 Sent Done Done false
8370 3141 Acked Read 7 Sent Done Done false
8371 3142 Acked Flush 0 Sent Done Done false
8372 3143 Acked Read 7 Sent Done Done false
8373 3144 Acked Write 3 Sent Done Done false
8374 3145 Acked Write 10 Sent Done Done false
8375 3146 Acked Read 8 Sent Done Done false
8376 3147 Acked Write 10 Sent Done Done false
8377 3148 Acked Write 3 Sent Done Done false
8378 3149 Acked Read 2 Sent Done Done false
8379 3150 Acked Write 9 Sent Done Done false
8380 3151 Acked Write 4 Sent Done Done false
8381 3152 Acked Read 7 Sent Done Done false
8382 3153 Acked Read 2 Sent Done Done false
8383 3154 Acked Read 7 Sent Done Done false
8384 3155 Acked Flush 0 Sent Done Done false
8385 3156 Acked Read 4 Sent Done Done false
8386 3157 Acked Write 1 Sent Done Done false
8387 3158 Acked Write 8 Sent Done Done false
8388 3159 Acked Write 5 Sent Done Done false
8389 3160 Acked Write 5 Sent Done Done false
8390 3161 Acked Read 7 Sent Done Done false
8391 3162 Acked Flush 0 Sent Done Done false
8392 3163 Acked Write 4 Sent Done Done false
8393 3164 Acked Write 2 Sent Done Done false
8394 3165 Acked Read 8 Sent Done Done false
8395 3166 Acked Read 10 Sent Done Done false
8396 3167 Acked Read 10 Sent Done Done false
8397 3168 Acked Write 5 Sent Done Done false
8398 3169 Acked Write 2 Sent Done Done false
8399 3170 Acked Read 9 Sent Done Done false
8400 3171 Acked Write 1 Sent Done Done false
8401 3172 Acked Read 5 Sent Done Done false
8402 3173 Acked Read 7 Sent Done Done false
8403 3174 Acked Read 9 Sent Done Done false
8404 3175 Acked Read 5 Sent Done Done false
8405 3176 Acked Write 2 Sent Done Done false
8406 3177 Acked Write 4 Sent Done Done false
8407 3178 Acked Read 3 Sent Done Done false
8408 3179 Acked Write 10 Sent Done Done false
8409 3180 Acked Flush 0 Sent Done Done false
8410 3181 Acked Read 2 Sent Done Done false
8411 3182 Acked Flush 0 Sent Done Done false
8412 3183 Acked Read 2 Sent Done Done false
8413 3184 Acked Read 6 Sent Done Done false
8414 3185 Acked Write 3 Sent Done Done false
8415 3186 Acked Write 4 Sent Done Done false
8416 3187 Acked Write 3 Sent Done Done false
8417 3188 Acked Write 2 Sent Done Done false
8418 3189 Acked Write 6 Sent Done Done false
8419 3190 Acked Read 6 Sent Done Done false
8420 3191 Acked Read 1 Sent Done Done false
8421 3192 Acked Write 5 Sent Done Done false
8422 3193 Acked Read 4 Sent Done Done false
8423 3194 Acked Write 5 Sent Done Done false
8424 3195 Acked Write 3 Sent Done Done false
8425 3196 Acked Read 2 Sent Done Done false
8426 3197 Acked Read 5 Sent Done Done false
8427 3198 Acked Read 8 Sent Done Done false
8428 3199 Acked Read 4 Sent Done Done false
8429 3200 Acked Read 2 Sent Done Done false
8430 3201 Acked Read 9 Sent Done Done false
8431 3202 Acked Read 7 Sent Done Done false
8432 3203 Acked Flush 0 Sent Done Done false
8433 3204 Acked Read 5 Sent Done Done false
8434 3205 Acked Read 6 Sent Done Done false
8435 3206 Acked Read 1 Sent Done Done false
8436 3207 Acked Read 2 Sent Done Done false
8437 3208 Acked Read 5 Sent Done Done false
8438 3209 Acked Read 2 Sent Done Done false
8439 3210 Acked Flush 0 Sent Done Done false
8440 3211 Acked Read 3 Sent Done Done false
8441 3212 Acked Read 1 Sent Done Done false
8442 3213 Acked Read 7 Sent Done Done false
8443 3214 Acked Flush 0 Sent Done Done false
8444 3215 Acked Write 4 Sent Done Done false
8445 3216 Acked Read 10 Sent Done Done false
8446 3217 Acked Read 10 Sent Done Done false
8447 3218 Acked Read 10 Sent Done Done false
8448 3219 Acked Write 6 Sent Done Done false
8449 3220 Acked Write 7 Sent Done Done false
8450 3221 Acked Read 9 Sent Done Done false
8451 3222 Acked Read 2 Sent Done Done false
8452 3223 Acked Write 5 Sent Done Done false
8453 3224 Acked Read 5 Sent Done Done false
8454 3225 Acked Write 3 Sent Done Done false
8455 3226 Acked Read 9 Sent Done Done false
8456 3227 Acked Read 4 Sent Done Done false
8457 3228 Acked Write 4 Sent Done Done false
8458 3229 Acked Write 2 Sent Done Done false
8459 3230 Acked Write 10 Sent Done Done false
8460 3231 Acked Read 2 Sent Done Done false
8461 3232 Acked Write 5 Sent Done Done false
8462 3233 Acked Read 5 Sent Done Done false
8463 3234 Acked Read 5 Sent Done Done false
8464 3235 Acked Read 10 Sent Done Done false
8465 3236 Acked Read 7 Sent Done Done false
8466 3237 Acked Write 7 Sent Done Done false
8467 3238 Acked Read 3 Sent Done Done false
8468 3239 Acked Read 1 Sent Done Done false
8469 3240 Acked Flush 0 Sent Done Done false
8470 3241 Acked Write 4 Sent Done Done false
8471 3242 Acked Read 7 Sent Done Done false
8472 3243 Acked Write 3 Sent Done Done false
8473 3244 Acked Read 3 Sent Done Done false
8474 3245 Acked Flush 0 Sent Done Done false
8475 3246 Acked Write 2 Sent Done Done false
8476 3247 Acked Read 5 Sent Done Done false
8477 3248 Acked Write 10 Sent Done Done false
8478 3249 Acked Read 8 Sent Done Done false
8479 3250 Acked Write 5 Sent Done Done false
8480 3251 Acked Write 5 Sent Done Done false
8481 3252 Acked Read 8 Sent Done Done false
8482 3253 Acked Flush 0 Sent Done Done false
8483 3254 Acked Read 8 Sent Done Done false
8484 3255 Acked Write 2 Sent Done Done false
8485 3256 Acked Write 4 Sent Done Done false
8486 3257 Acked Read 5 Sent Done Done false
8487 3258 Acked Write 3 Sent Done Done false
8488 3259 Acked Write 2 Sent Done Done false
8489 3260 Acked Write 4 Sent Done Done false
8490 3261 Acked Write 5 Sent Done Done false
8491 3262 Acked Flush 0 Sent Done Done false
8492 3263 Acked Flush 0 Sent Done Done false
8493 3264 Acked Read 6 Sent Done Done false
8494 3265 Acked Read 3 Sent Done Done false
8495 3266 Acked Write 7 Sent Done Done false
8496 3267 Acked Write 4 Sent Done Done false
8497 3268 Acked Flush 0 Sent Done Done false
8498 3269 Acked Read 5 Sent Done Done false
8499 3270 Acked Read 8 Sent Done Done false
8500 3271 Acked Write 4 Sent Done Done false
8501 3272 Acked Read 6 Sent Done Done false
8502 3273 Acked Read 4 Sent Done Done false
8503 3274 Acked Read 3 Sent Done Done false
8504 3275 Acked Write 6 Sent Done Done false
8505 3276 Acked Read 9 Sent Done Done false
8506 3277 Acked Write 9 Sent Done Done false
8507 3278 Acked Write 9 Sent Done Done false
8508 3279 Acked Read 7 Sent Done Done false
8509 3280 Acked Flush 0 Sent Done Done false
8510 3281 Acked Write 1 Sent Done Done false
8511 3282 Acked Read 6 Sent Done Done false
8512 3283 Acked Read 8 Sent Done Done false
8513 3284 Acked Read 1 Sent Done Done false
8514 3285 Acked Write 8 Sent Done Done false
8515 3286 Acked Flush 0 Sent Done Done false
8516 3287 Acked Read 3 Sent Done Done false
8517 3288 Acked Write 10 Sent Done Done false
8518 3289 Acked Flush 0 Sent Done Done false
8519 3290 Acked Read 1 Sent Done Done false
8520 3291 Acked Write 2 Sent Done Done false
8521 3292 Acked Write 5 Sent Done Done false
8522 3293 Acked Write 1 Sent Done Done false
8523 3294 Acked Read 5 Sent Done Done false
8524 3295 Acked Flush 0 Sent Done Done false
8525 3296 Acked Write 7 Sent Done Done false
8526 3297 Acked Flush 0 Sent Done Done false
8527 3298 Acked Write 6 Sent Done Done false
8528 3299 Acked Read 1 Sent Done Done false
8529 3300 Acked Read 6 Sent Done Done false
8530 3301 Acked Write 9 Sent Done Done false
8531 3302 Acked Read 5 Sent Done Done false
8532 3303 Acked Read 3 Sent Done Done false
8533 3304 Acked Write 2 Sent Done Done false
8534 3305 Acked Write 6 Sent Done Done false
8535 3306 Acked Write 1 Sent Done Done false
8536 3307 Acked Read 10 Sent Done Done false
8537 3308 Acked Write 6 Sent Done Done false
8538 3309 Acked Read 4 Sent Done Done false
8539 3310 Acked Write 9 Sent Done Done false
8540 3311 Acked Write 7 Sent Done Done false
8541 3312 Acked Read 3 Sent Done Done false
8542 3313 Acked Read 3 Sent Done Done false
8543 3314 Acked Read 1 Sent Done Done false
8544 3315 Acked Write 5 Sent Done Done false
8545 3316 Acked Flush 0 Sent Done Done false
8546 3317 Acked Read 5 Sent Done Done false
8547 3318 Acked Read 1 Sent Done Done false
8548 3319 Acked Write 1 Sent Done Done false
8549 3320 Acked Read 8 Sent Done Done false
8550 3321 Acked Write 6 Sent Done Done false
8551 3322 Acked Write 9 Sent Done Done false
8552 3323 Acked Read 1 Sent Done Done false
8553 3324 Acked Write 7 Sent Done Done false
8554 3325 Acked Write 7 Sent Done Done false
8555 3326 Acked Write 7 Sent Done Done false
8556 3327 Acked Write 10 Sent Done Done false
8557 3328 Acked Read 9 Sent Done Done false
8558 3329 Acked Read 9 Sent Done Done false
8559 3330 Acked Read 9 Sent Done Done false
8560 3331 Acked Read 4 Sent Done Done false
8561 3332 Acked Write 7 Sent Done Done false
8562 3333 Acked Read 10 Sent Done Done false
8563 3334 Acked Read 10 Sent Done Done false
8564 3335 Acked Write 1 Sent Done Done false
8565 3336 Acked Write 5 Sent Done Done false
8566 3337 Acked Flush 0 Sent Done Done false
8567 3338 Acked Read 2 Sent Done Done false
8568 3339 Acked Read 7 Sent Done Done false
8569 3340 Acked Read 8 Sent Done Done false
8570 3341 Acked Write 1 Sent Done Done false
8571 3342 Acked Read 4 Sent Done Done false
8572 3343 Acked Read 6 Sent Done Done false
8573 3344 Acked Write 7 Sent Done Done false
8574 3345 Acked Flush 0 Sent Done Done false
8575 3346 Acked Read 1 Sent Done Done false
8576 3347 Acked Write 2 Sent Done Done false
8577 3348 Acked Flush 0 Sent Done Done false
8578 3349 Acked Read 4 Sent Done Done false
8579 3350 Acked Read 8 Sent Done Done false
8580 3351 Acked Read 10 Sent Done Done false
8581 3352 Acked Write 7 Sent Done Done false
8582 3353 Acked Read 10 Sent Done Done false
8583 3354 Acked Read 3 Sent Done Done false
8584 3355 Acked Write 4 Sent Done Done false
8585 3356 Acked Write 6 Sent Done Done false
8586 3357 Acked Read 5 Sent Done Done false
8587 3358 Acked Read 5 Sent Done Done false
8588 3359 Acked Read 2 Sent Done Done false
8589 3360 Acked Flush 0 Sent Done Done false
8590 3361 Acked Write 2 Sent Done Done false
8591 3362 Acked Write 9 Sent Done Done false
8592 3363 Acked Read 7 Sent Done Done false
8593 3364 Acked Write 4 Sent Done Done false
8594 3365 Acked Read 6 Sent Done Done false
8595 3366 Acked Write 2 Sent Done Done false
8596 3367 Acked Write 5 Sent Done Done false
8597 3368 Acked Write 7 Sent Done Done false
8598 3369 Acked Write 1 Sent Done Done false
8599 3370 Acked Read 2 Sent Done Done false
8600 3371 Acked Read 1 Sent Done Done false
8601 3372 Acked Write 2 Sent Done Done false
8602 3373 Acked Write 5 Sent Done Done false
8603 3374 Acked Read 10 Sent Done Done false
8604 3375 Acked Read 3 Sent Done Done false
8605 3376 Acked Write 7 Sent Done Done false
8606 3377 Acked Write 1 Sent Done Done false
8607 3378 Acked Write 10 Sent Done Done false
8608 3379 Acked Read 9 Sent Done Done false
8609 3380 Acked Flush 0 Sent Done Done false
8610 3381 Acked Flush 0 Sent Done Done false
8611 3382 Acked Write 10 Sent Done Done false
8612 3383 Acked Read 8 Sent Done Done false
8613 3384 Acked Write 2 Sent Done Done false
8614 3385 Acked Write 9 Sent Done Done false
8615 3386 Acked Read 5 Sent Done Done false
8616 3387 Acked Read 3 Sent Done Done false
8617 3388 Acked Read 9 Sent Done Done false
8618 3389 Acked Write 5 Sent Done Done false
8619 3390 Acked Write 4 Sent Done Done false
8620 3391 Acked Read 2 Sent Done Done false
8621 3392 Acked Read 4 Sent Done Done false
8622 3393 Acked Read 6 Sent Done Done false
8623 3394 Acked Write 1 Sent Done Done false
8624 3395 Acked Read 9 Sent Done Done false
8625 3396 Acked Flush 0 Sent Done Done false
8626 3397 Acked Write 1 Sent Done Done false
8627 3398 Acked Read 6 Sent Done Done false
8628 3399 Acked Write 7 Sent Done Done false
8629 3400 Acked Read 6 Sent Done Done false
8630 3401 Acked Write 9 Sent Done Done false
8631 3402 Acked Write 5 Sent Done Done false
8632 3403 Acked Read 5 Sent Done Done false
8633 3404 Acked Flush 0 Sent Done Done false
8634 3405 Acked Read 8 Sent Done Done false
8635 3406 Acked Read 5 Sent Done Done false
8636 3407 Acked Read 8 Sent Done Done false
8637 3408 Acked Read 7 Sent Done Done false
8638 3409 Acked Read 5 Sent Done Done false
8639 3410 Acked Write 7 Sent Done Done false
8640 3411 Acked Read 5 Sent Done Done false
8641 3412 Acked Write 1 Sent Done Done false
8642 3413 Acked Write 8 Sent Done Done false
8643 3414 Acked Read 9 Sent Done Done false
8644 STATES DS:0 DS:1 DS:2 TOTAL
8645 Sent 301 0 0 301
8646 Done 0 301 301 602
8647 Skipped 0 0 0 0
8648 Error 0 0 0 0
8649 Last Flush: Some(JobId(3113)) Some(JobId(3404)) Some(JobId(3404))
8650 Downstairs last five completed: 3113 3112 3111 3110 3109
8651 Upstairs last five completed: 3414 3413 3412 3411 3410
8652 CLIENT: Up:0 ds:301 act:2
86532025-01-28T21:56:08.486ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
86542025-01-28T21:56:09.487ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
86552025-01-28T21:56:09.606ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
86562025-01-28T21:56:09.606ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42650) = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
86572025-01-28T21:56:09.607ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
86582025-01-28T21:56:09.607ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
86592025-01-28T21:56:09.607ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
86602025-01-28T21:56:09.607ZINFOcrucible: send last flush ID to this DS: Some(JobId(3113)) = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
86612025-01-28T21:56:09.607ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
86622025-01-28T21:56:09.607ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(3113)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
86632025-01-28T21:56:09.607ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(3113)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
8664 ----------------------------------------------------------------
8665 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8666 Downstairs last five completed: 3415 3414 3413 3412 3411
8667 Upstairs last five completed: 3415 3414 3413 3412 3411
8668 CLIENT: Up:0 ds:0 act:3
8669 Replay: All jobs finished
8670 Main task has work to do, go find it
8671 got dsc Stop(1)
8672 stop 1
8673 Jan 28 21:56:11.848 INFO[8820] Got stop action so:false kr:false
8674 request completed, latency_us: 371, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 738962a7-f7c9-4292-8c9f-47f4d1c2dafa, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
86752025-01-28T21:56:11.849ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
86762025-01-28T21:56:11.849ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
86772025-01-28T21:56:11.849ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
8678 Jan 28 21:56:11.849 INFO request completed, latency_us: 170, response_code: 200, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:56:11.849276267Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
8679 uri: /state/cid/1, method: GET, req_id: be2974dc-3127-4da8-af14-2a3969b57d08, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
8680 [8820] Exited with: ExitStatus(unix_wait_status(9))
8681 [8820][1] reports Exit
8682 Jan 28 21:56:15.851 INFO request completed, latency_us: 212, response_code: 200, uri: /state/cid/1, method: GET, req_id: 3c12a515-60fd-49b8-ae00-ef52685d251d, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
8683 001/300 Write block 385 len 8192 data: 3 3
8684 002/300 Write block 3409 len 16384 data: 3 3 3 3
8685 003/300 Write block 635 len 12288 data: 4 5 6
8686 004/300 Read block 4352 len 8192
8687 005/300 Write block 4214 len 24576 data: 2 2 2 2 2 2
8688 006/300 Write block 3882 len 28672 data: 4 4 4 3 2 2 2
8689 007/300 Write block 723 len 4096 data: 3
8690 008/300 Read block 1934 len 28672
8691 009/300 Write block 2688 len 32768 data: 5 4 4 4 4 4 4 4
8692 010/300 Read block 522 len 8192
8693 011/300 Write block 4702 len 36864 data: 2 2 2 2 2 3 3 3 3
8694 012/300 Flush
8695 013/300 Read block 1420 len 4096
8696 014/300 Write block 3788 len 8192 data: 4 4
8697 015/300 Read block 3066 len 16384
8698 016/300 Write block 3676 len 12288 data: 2 2 2
8699 017/300 Read block 4532 len 12288
8700 018/300 Write block 902 len 12288 data: 4 4 4
8701 019/300 Read block 4558 len 40960
8702 020/300 Write block 2748 len 32768 data: 2 2 3 4 4 3 3 3
8703 021/300 Write block 3740 len 16384 data: 2 2 2 2
8704 022/300 Flush
8705 023/300 Write block 4260 len 28672 data: 2 2 2 3 3 3 3
8706 024/300 Read block 2404 len 16384
8707 025/300 Read block 1422 len 16384
8708 026/300 Write block 2839 len 40960 data: 2 2 2 2 2 3 3 5 5 5
8709 027/300 Write block 3574 len 16384 data: 3 3 4 4
8710 028/300 Write block 2546 len 36864 data: 3 3 4 3 3 3 3 3 3
8711 029/300 Write block 1196 len 20480 data: 3 4 5 4 4
8712 030/300 Write block 379 len 4096 data: 3
8713 031/300 Write block 1399 len 28672 data: 2 2 2 2 3 3 3
8714 032/300 Write block 1197 len 36864 data: 5 6 5 5 4 5 5 8 7
8715 033/300 Read block 119 len 8192
8716 034/300 Read block 2382 len 36864
8717 035/300 Write block 2328 len 4096 data: 3
8718 036/300 Write block 1047 len 24576 data: 4 4 3 3 4 5
8719 037/300 Read block 3072 len 32768
8720 038/300 Flush
8721 039/300 Read block 2089 len 36864
8722 040/300 Write block 1029 len 8192 data: 3 2
8723 041/300 Write block 718 len 32768 data: 2 2 2 3 4 4 3 3
8724 042/300 Write block 2154 len 12288 data: 4 4 3
8725 043/300 Read block 3988 len 24576
8726 044/300 Write block 178 len 20480 data: 2 2 2 3 3
8727 045/300 Read block 4844 len 36864
8728 046/300 Read block 3872 len 20480
8729 047/300 Read block 3085 len 20480
8730 048/300 Read block 3458 len 28672
8731 049/300 Read block 353 len 32768
8732 050/300 Read block 3450 len 12288
8733 051/300 Read block 775 len 12288
8734 052/300 Write block 1307 len 40960 data: 2 3 3 3 3 3 3 3 3 3
8735 053/300 Read block 536 len 16384
8736 054/300 Write block 418 len 36864 data: 2 2 2 2 2 2 2 2 2
8737 055/300 Read block 3240 len 36864
8738 056/300 Flush
8739 057/300 Write block 1513 len 32768 data: 2 3 3 3 5 5 5 5
8740 058/300 Write block 1580 len 40960 data: 3 3 3 3 3 3 3 3 3 2
8741 059/300 Write block 1374 len 32768 data: 2 2 2 2 3 3 3 3
8742 060/300 Read block 4667 len 24576
8743 061/300 Write block 43 len 4096 data: 4
8744 062/300 Flush
8745 063/300 Write block 321 len 32768 data: 2 3 4 4 3 3 3 3
8746 064/300 Write block 1956 len 40960 data: 4 4 3 3 3 2 2 2 2 2
8747 065/300 Flush
8748 066/300 Write block 1254 len 40960 data: 3 3 3 2 2 2 2 2 2 2
8749 067/300 Write block 4926 len 20480 data: 3 3 2 2 2
8750 068/300 Write block 1603 len 12288 data: 3 3 3
8751 069/300 Write block 3884 len 24576 data: 5 4 3 3 3 2
8752 070/300 Write block 4481 len 36864 data: 8 8 7 6 5 7 4 4 4
8753 071/300 Write block 880 len 4096 data: 2
8754 072/300 Flush
8755 073/300 Read block 1576 len 24576
8756 074/300 Write block 1824 len 12288 data: 3 4 4
8757 075/300 Read block 1377 len 32768
8758 076/300 Read block 4400 len 16384
8759 077/300 Read block 2879 len 24576
8760 078/300 Write block 4426 len 20480 data: 4 3 3 2 2
8761 079/300 Read block 856 len 4096
8762 080/300 Flush
8763 081/300 Read block 1011 len 8192
8764 082/300 Write block 1125 len 4096 data: 3
8765 083/300 Read block 3757 len 16384
8766 084/300 Read block 2308 len 24576
8767 085/300 Write block 470 len 16384 data: 3 3 3 4
8768 086/300 Flush
8769 087/300 Read block 3134 len 20480
8770 088/300 Read block 2386 len 32768
8771 089/300 Flush
8772 090/300 Write block 76 len 28672 data: 3 3 3 3 3 3 3
8773 091/300 Write block 2245 len 16384 data: 3 3 3 3
8774 092/300 Write block 3293 len 12288 data: 3 3 3
8775 093/300 Read block 1044 len 8192
8776 094/300 Read block 2651 len 24576
8777 095/300 Write block 3199 len 20480 data: 3 3 3 3 3
8778 096/300 Write block 1968 len 20480 data: 3 4 4 4 3
8779 097/300 Flush
8780 098/300 Write block 2184 len 12288 data: 2 2 2
8781 099/300 Write block 2530 len 16384 data: 3 3 3 3
8782 100/300 Read block 3185 len 24576
8783 101/300 Read block 1381 len 32768
8784 102/300 Read block 3654 len 16384
8785 103/300 Read block 288 len 36864
8786 104/300 Write block 96 len 4096 data: 3
8787 105/300 Write block 3275 len 12288 data: 4 4 3
8788 106/300 Write block 501 len 12288 data: 2 2 2
8789 107/300 Write block 3213 len 12288 data: 4 4 4
8790 108/300 Flush
8791 109/300 Write block 1482 len 20480 data: 2 2 2 2 4
8792 110/300 Flush
8793 111/300 Read block 2810 len 16384
8794 112/300 Read block 2694 len 8192
8795 113/300 Read block 617 len 40960
8796 114/300 Write block 2381 len 40960 data: 3 4 4 4 2 2 2 3 3 4
8797 115/300 Write block 227 len 40960 data: 2 2 2 2 2 3 4 3 3 3
8798 116/300 Read block 1007 len 36864
8799 117/300 Write block 1298 len 36864 data: 4 4 4 3 2 2 3 3 3
8800 118/300 Read block 3477 len 28672
8801 119/300 Write block 4889 len 12288 data: 2 2 2
8802 120/300 Write block 1105 len 24576 data: 5 5 4 4 4 3
8803 121/300 Write block 3713 len 12288 data: 3 4 3
8804 122/300 Read block 1690 len 12288
8805 123/300 Read block 4849 len 40960
8806 124/300 Read block 3414 len 12288
8807 125/300 Write block 4017 len 8192 data: 3 3
8808 126/300 Read block 521 len 28672
8809 127/300 Read block 639 len 16384
8810 128/300 Write block 139 len 4096 data: 4
8811 129/300 Read block 4037 len 24576
8812 130/300 Flush
8813 131/300 Write block 3928 len 4096 data: 3
8814 132/300 Write block 1738 len 24576 data: 3 4 3 2 2 2
8815 133/300 Read block 1892 len 28672
8816 134/300 Write block 3093 len 36864 data: 3 2 3 3 2 2 2 2 2
8817 135/300 Read block 912 len 12288
8818 136/300 Read block 2572 len 40960
8819 137/300 Write block 1128 len 12288 data: 4 4 4
8820 138/300 Flush
8821 139/300 Write block 1079 len 16384 data: 4 4 4 4
8822 140/300 Read block 319 len 8192
8823 141/300 Read block 2233 len 8192
8824 142/300 Flush
8825 143/300 Write block 206 len 28672 data: 2 2 3 3 3 3 3
8826 144/300 Read block 4482 len 32768
8827 145/300 Write block 3719 len 40960 data: 4 4 4 4 4 4 4 4 4 5
8828 146/300 Write block 2122 len 40960 data: 4 5 6 5 5 5 3 3 5 5
8829 147/300 Write block 4307 len 36864 data: 3 3 3 3 3 3 4 3 3
8830 148/300 Flush
8831 149/300 Write block 2920 len 24576 data: 2 2 2 2 2 2
8832 150/300 Flush
8833 151/300 Read block 4447 len 12288
8834 152/300 Read block 2939 len 36864
8835 153/300 Read block 1520 len 12288
8836 154/300 Write block 2396 len 12288 data: 4 4 4
8837 155/300 Read block 2094 len 16384
8838 156/300 Flush
8839 157/300 Write block 4697 len 4096 data: 4
8840 158/300 Read block 694 len 24576
8841 159/300 Read block 3844 len 28672
8842 160/300 Read block 4833 len 4096
8843 161/300 Read block 4279 len 32768
8844 162/300 Read block 4193 len 8192
8845 163/300 Write block 1676 len 36864 data: 3 2 2 2 2 3 3 3 2
8846 164/300 Write block 1182 len 4096 data: 5
8847 165/300 Flush
8848 166/300 Write block 4247 len 40960 data: 3 2 2 3 4 4 4 3 3 3
8849 167/300 Read block 991 len 20480
8850 168/300 Read block 2175 len 40960
8851 169/300 Read block 4408 len 8192
8852 170/300 Read block 811 len 20480
8853 171/300 Read block 4292 len 12288
8854 172/300 Write block 3618 len 36864 data: 2 2 2 2 2 2 3 4 3
8855 173/300 Write block 1793 len 24576 data: 2 3 3 3 3 4
8856 174/300 Read block 24 len 28672
8857 175/300 Write block 1059 len 8192 data: 5 5
8858 176/300 Write block 2513 len 16384 data: 3 3 2 2
8859 177/300 Read block 1065 len 24576
8860 178/300 Read block 1684 len 40960
8861 179/300 Read block 4426 len 4096
8862 180/300 Flush
8863 181/300 Read block 4626 len 24576
8864 182/300 Read block 3623 len 16384
8865 183/300 Write block 3325 len 16384 data: 4 4 3 4
8866 184/300 Read block 1034 len 12288
8867 185/300 Write block 1046 len 12288 data: 5 5 5
8868 186/300 Read block 4796 len 28672
8869 187/300 Flush
8870 188/300 Write block 716 len 4096 data: 3
8871 189/300 Write block 3443 len 40960 data: 3 3 2 2 3 3 3 3 3 3
8872 190/300 Read block 4173 len 32768
8873 191/300 Write block 4337 len 28672 data: 4 4 3 3 3 3 2
8874 192/300 Write block 1003 len 4096 data: 5
8875 193/300 Write block 2155 len 28672 data: 5 4 5 5 7 6 5
8876 194/300 Read block 2554 len 40960
8877 195/300 Read block 2326 len 28672
8878 196/300 Read block 2686 len 8192
8879 197/300 Write block 1361 len 16384 data: 3 3 3 4
8880 198/300 Flush
8881 199/300 Read block 914 len 8192
8882 200/300 Write block 2471 len 20480 data: 3 3 4 5 6
8883 201/300 Write block 124 len 12288 data: 4 4 5
8884 202/300 Read block 2800 len 16384
8885 203/300 Write block 459 len 32768 data: 4 5 4 4 2 2 2 2
8886 204/300 Read block 1211 len 12288
8887 205/300 Read block 2261 len 8192
8888 206/300 Read block 4351 len 40960
8889 207/300 Read block 2308 len 40960
8890 208/300 Write block 2988 len 32768 data: 4 3 3 4 5 5 6 6
8891 209/300 Write block 1355 len 40960 data: 3 3 2 3 3 2 4 4 4 5
8892 210/300 Flush
8893 211/300 Read block 4620 len 16384
8894 212/300 Read block 3813 len 4096
8895 213/300 Flush
8896 214/300 Read block 3384 len 12288
8897 215/300 Read block 2564 len 4096
8898 216/300 Read block 2160 len 4096
8899 217/300 Write block 3975 len 36864 data: 2 2 2 2 3 3 3 3 3
8900 218/300 Read block 495 len 32768
8901 219/300 Flush
8902 220/300 Read block 4008 len 8192
8903 221/300 Read block 113 len 28672
8904 222/300 Read block 138 len 4096
8905 223/300 Read block 403 len 32768
8906 224/300 Read block 789 len 20480
8907 225/300 Write block 3110 len 28672 data: 3 3 3 3 3 3 3
8908 226/300 Read block 402 len 8192
8909 227/300 Write block 1659 len 28672 data: 4 4 3 3 3 4 3
8910 228/300 Write block 3635 len 20480 data: 3 3 3 3 2
8911 229/300 Write block 1242 len 16384 data: 2 2 2 2
8912 230/300 Read block 1340 len 24576
8913 231/300 Read block 1513 len 32768
8914 232/300 Read block 3631 len 4096
8915 233/300 Write block 1708 len 28672 data: 3 3 3 3 3 3 3
8916 234/300 Read block 129 len 28672
8917 235/300 Read block 4605 len 8192
8918 236/300 Read block 4773 len 40960
8919 237/300 Read block 2992 len 12288
8920 238/300 Read block 3314 len 24576
8921 239/300 Read block 562 len 24576
8922 240/300 Read block 4141 len 32768
8923 241/300 Read block 2661 len 16384
8924 242/300 Write block 231 len 32768 data: 3 4 5 4 4 4 2 3
8925 243/300 Read block 3633 len 40960
8926 244/300 Read block 396 len 28672
8927 245/300 Read block 1851 len 8192
8928 246/300 Write block 4436 len 24576 data: 3 3 3 2 2 2
8929 247/300 Read block 2926 len 12288
8930 248/300 Write block 4746 len 24576 data: 2 2 2 2 2 2
8931 249/300 Read block 3070 len 8192
8932 250/300 Read block 1556 len 36864
8933 251/300 Read block 290 len 40960
8934 252/300 Read block 1435 len 20480
8935 253/300 Read block 3220 len 4096
8936 254/300 Write block 2114 len 12288 data: 2 2 2
8937 255/300 Write block 77 len 36864 data: 4 4 4 4 4 4 3 2 2
8938 256/300 Read block 4879 len 32768
8939 257/300 Read block 2860 len 24576
8940 258/300 Write block 3605 len 24576 data: 4 2 2 2 2 2
8941 259/300 Read block 745 len 32768
8942 260/300 Read block 3588 len 16384
8943 261/300 Read block 1380 len 8192
8944 262/300 Read block 2707 len 36864
8945 263/300 Write block 1747 len 8192 data: 2 2
8946 264/300 Read block 4141 len 32768
8947 265/300 Write block 3484 len 8192 data: 2 3
8948 266/300 Write block 885 len 12288 data: 4 4 4
8949 267/300 Read block 1499 len 4096
8950 268/300 Read block 4700 len 8192
8951 269/300 Write block 3431 len 28672 data: 3 3 3 2 2 3 3
8952 270/300 Write block 4986 len 12288 data: 4 4 4
8953 271/300 Read block 4120 len 8192
8954 272/300 Write block 547 len 16384 data: 6 6 4 4
8955 273/300 Write block 4378 len 24576 data: 2 2 3 4 4 4
8956 274/300 Read block 4374 len 40960
8957 275/300 Read block 712 len 36864
8958 276/300 Read block 4531 len 16384
8959 277/300 Read block 2433 len 36864
8960 278/300 Read block 2290 len 40960
8961 279/300 Flush
8962 280/300 Write block 132 len 8192 data: 2 2
8963 281/300 Write block 4356 len 40960 data: 4 4 4 4 4 3 3 3 4 2
8964 282/300 Read block 412 len 28672
8965 283/300 Read block 15 len 12288
8966 284/300 Write block 137 len 40960 data: 3 3 5 3 3 3 3 3 3 2
8967 285/300 Flush
8968 286/300 Write block 3642 len 8192 data: 4 4
8969 287/300 Write block 2876 len 32768 data: 4 3 2 2 2 3 2 2
8970 288/300 Read block 2560 len 20480
8971 289/300 Flush
8972 290/300 Write block 2411 len 28672 data: 4 4 4 4 3 3 3
8973 291/300 Write block 4227 len 8192 data: 2 4
8974 292/300 Read block 2389 len 36864
8975 293/300 Write block 1195 len 36864 data: 3 4 6 7 6 6 5 6 6
8976 294/300 Read block 861 len 16384
8977 295/300 Read block 688 len 32768
8978 296/300 Read block 3340 len 4096
8979 297/300 Read block 2446 len 36864
8980 298/300 Read block 1344 len 8192
8981 299/300 Write block 4881 len 8192 data: 5 5
8982 300/300 Read block 1941 len 28672
8983 301/300 Read block 4506 len 8192
8984 Main task has work to do, go find it
8985 got dsc Start(1)
8986 start 1
8987 Jan 28 21:56:16.089 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/1, method: POST, req_id: 3d99b4c3-28aa-454d-9930-3c39cc046665, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
8988 Replay: started 1, returned:Ok(())
8989 ----------------------------------------------------------------
8990 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8991 States: Active Offline Active
8992 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8993 3416 Acked Write 2 Done Sent Done false
8994 3417 Acked Write 4 Done Sent Done false
8995 3418 Acked Write 3 Done Sent Done false
8996 3419 Acked Read 2 Done Sent Done false
8997 3420 Acked Write 6 Done Sent Done false
8998 3421 Acked Write 7 Done Sent Done false
8999 3422 Acked Write 1 Done Sent Done false
9000 3423 Acked Read 7 Done Sent Done false
9001 3424 Acked Write 8 Done Sent Done false
9002 3425 Acked Read 2 Done Sent Done false
9003 3426 Acked Write 9 Done Sent Done false
9004 3427 Acked Flush 0 Done Sent Done false
9005 3428 Acked Read 1 Done Sent Done false
9006 3429 Acked Write 2 Done Sent Done false
9007 3430 Acked Read 4 Done Sent Done false
9008 3431 Acked Write 3 Done Sent Done false
9009 3432 Acked Read 3 Done Sent Done false
9010 3433 Acked Write 3 Done Sent Done false
9011 3434 Acked Read 10 Done Sent Done false
9012 3435 Acked Write 8 Done Sent Done false
9013 3436 Acked Write 4 Done Sent Done false
9014 3437 Acked Flush 0 Done Sent Done false
9015 3438 Acked Write 7 Done Sent Done false
9016 3439 Acked Read 4 Done Sent Done false
9017 3440 Acked Read 4 Done Sent Done false
9018 3441 Acked Write 10 Done Sent Done false
9019 3442 Acked Write 4 Done Sent Done false
9020 3443 Acked Write 9 Done Sent Done false
9021 3444 Acked Write 5 Done Sent Done false
9022 3445 Acked Write 1 Done Sent Done false
9023 3446 Acked Write 7 Done Sent Done false
9024 3447 Acked Write 9 Done Sent Done false
9025 3448 Acked Read 2 Done Sent Done false
9026 3449 Acked Read 9 Done Sent Done false
9027 3450 Acked Write 1 Done Sent Done false
9028 3451 Acked Write 6 Done Sent Done false
9029 3452 Acked Read 8 Done Sent Done false
9030 3453 Acked Flush 0 Done Sent Done false
9031 3454 Acked Read 9 Done Sent Done false
9032 3455 Acked Write 2 Done Sent Done false
9033 3456 Acked Write 8 Done Sent Done false
9034 3457 Acked Write 3 Done Sent Done false
9035 3458 Acked Read 6 Done Sent Done false
9036 3459 Acked Write 5 Done Sent Done false
9037 3460 Acked Read 9 Done Sent Done false
9038 3461 Acked Read 5 Done Sent Done false
9039 3462 Acked Read 5 Done Sent Done false
9040 3463 Acked Read 7 Done Sent Done false
9041 3464 Acked Read 8 Done Sent Done false
9042 3465 Acked Read 3 Done Sent Done false
9043 3466 Acked Read 3 Done Sent Done false
9044 3467 Acked Write 10 Done Sent Done false
9045 3468 Acked Read 4 Done Sent Done false
9046 3469 Acked Write 9 Done Sent Done false
9047 3470 Acked Read 9 Done Sent Done false
9048 3471 Acked Flush 0 Done Sent Done false
9049 3472 Acked Write 8 Done Sent Done false
9050 3473 Acked Write 10 Done Sent Done false
9051 3474 Acked Write 8 Done Sent Done false
9052 3475 Acked Read 6 Done Sent Done false
9053 3476 Acked Write 1 Done Sent Done false
9054 3477 Acked Flush 0 Done Sent Done false
9055 3478 Acked Write 8 Done Sent Done false
9056 3479 Acked Write 10 Done Sent Done false
9057 3480 Acked Flush 0 Done Sent Done false
9058 3481 Acked Write 10 Done Sent Done false
9059 3482 Acked Write 5 Done Sent Done false
9060 3483 Acked Write 3 Done Sent Done false
9061 3484 Acked Write 6 Done Sent Done false
9062 3485 Acked Write 9 Done Sent Done false
9063 3486 Acked Write 1 Done Sent Done false
9064 3487 Acked Flush 0 Done Sent Done false
9065 3488 Acked Read 6 Done Sent Done false
9066 3489 Acked Write 3 Done Sent Done false
9067 3490 Acked Read 8 Done Sent Done false
9068 3491 Acked Read 4 Done Sent Done false
9069 3492 Acked Read 6 Done Sent Done false
9070 3493 Acked Write 5 Done Sent Done false
9071 3494 Acked Read 1 Done Sent Done false
9072 3495 Acked Flush 0 Done Sent Done false
9073 3496 Acked Read 2 Done Sent Done false
9074 3497 Acked Write 1 Done Sent Done false
9075 3498 Acked Read 4 Done Sent Done false
9076 3499 Acked Read 6 Done Sent Done false
9077 3500 Acked Write 4 Done Sent Done false
9078 3501 Acked Flush 0 Done Sent Done false
9079 3502 Acked Read 5 Done Sent Done false
9080 3503 Acked Read 8 Done Sent Done false
9081 3504 Acked Flush 0 Done Sent Done false
9082 3505 Acked Write 7 Done Sent Done false
9083 3506 Acked Write 4 Done Sent Done false
9084 3507 Acked Write 3 Done Sent Done false
9085 3508 Acked Read 2 Done Sent Done false
9086 3509 Acked Read 6 Done Sent Done false
9087 3510 Acked Write 5 Done Sent Done false
9088 3511 Acked Write 5 Done Sent Done false
9089 3512 Acked Flush 0 Done Sent Done false
9090 3513 Acked Write 3 Done Sent Done false
9091 3514 Acked Write 4 Done Sent Done false
9092 3515 Acked Read 6 Done Sent Done false
9093 3516 Acked Read 8 Done Sent Done false
9094 3517 Acked Read 4 Done Sent Done false
9095 3518 Acked Read 9 Done Sent Done false
9096 3519 Acked Write 1 Done Sent Done false
9097 3520 Acked Write 3 Done Sent Done false
9098 3521 Acked Write 3 Done Sent Done false
9099 3522 Acked Write 3 Done Sent Done false
9100 3523 Acked Flush 0 Done Sent Done false
9101 3524 Acked Write 5 Done Sent Done false
9102 3525 Acked Flush 0 Done Sent Done false
9103 3526 Acked Read 4 Done Sent Done false
9104 3527 Acked Read 2 Done Sent Done false
9105 3528 Acked Read 10 Done Sent Done false
9106 3529 Acked Write 10 Done Sent Done false
9107 3530 Acked Write 10 Done Sent Done false
9108 3531 Acked Read 9 Done Sent Done false
9109 3532 Acked Write 9 Done Sent Done false
9110 3533 Acked Read 7 Done Sent Done false
9111 3534 Acked Write 3 Done Sent Done false
9112 3535 Acked Write 6 Done Sent Done false
9113 3536 Acked Write 3 Done Sent Done false
9114 3537 Acked Read 3 Done Sent Done false
9115 3538 Acked Read 10 Done Sent Done false
9116 3539 Acked Read 3 Done Sent Done false
9117 3540 Acked Write 2 Done Sent Done false
9118 3541 Acked Read 7 Done Sent Done false
9119 3542 Acked Read 4 Done Sent Done false
9120 3543 Acked Write 1 Done Sent Done false
9121 3544 Acked Read 6 Done Sent Done false
9122 3545 Acked Flush 0 Done Sent Done false
9123 3546 Acked Write 1 Done Sent Done false
9124 3547 Acked Write 6 Done Sent Done false
9125 3548 Acked Read 7 Done Sent Done false
9126 3549 Acked Write 9 Done Sent Done false
9127 3550 Acked Read 3 Done Sent Done false
9128 3551 Acked Read 10 Done Sent Done false
9129 3552 Acked Write 3 Done Sent Done false
9130 3553 Acked Flush 0 Done Sent Done false
9131 3554 Acked Write 4 Done Sent Done false
9132 3555 Acked Read 2 Done Sent Done false
9133 3556 Acked Read 2 Done Sent Done false
9134 3557 Acked Flush 0 Done Sent Done false
9135 3558 Acked Write 7 Done Sent Done false
9136 3559 Acked Read 8 Done Sent Done false
9137 3560 Acked Write 10 Done Sent Done false
9138 3561 Acked Write 10 Done Sent Done false
9139 3562 Acked Write 9 Done Sent Done false
9140 3563 Acked Flush 0 Done Sent Done false
9141 3564 Acked Write 6 Done Sent Done false
9142 3565 Acked Flush 0 Done Sent Done false
9143 3566 Acked Read 3 Done Sent Done false
9144 3567 Acked Read 9 Done Sent Done false
9145 3568 Acked Read 3 Done Sent Done false
9146 3569 Acked Write 3 Done Sent Done false
9147 3570 Acked Read 4 Done Sent Done false
9148 3571 Acked Flush 0 Done Sent Done false
9149 3572 Acked Write 1 Done Sent Done false
9150 3573 Acked Read 6 Done Sent Done false
9151 3574 Acked Read 7 Done Sent Done false
9152 3575 Acked Read 1 Done Sent Done false
9153 3576 Acked Read 8 Done Sent Done false
9154 3577 Acked Read 2 Done Sent Done false
9155 3578 Acked Write 9 Done Sent Done false
9156 3579 Acked Write 1 Done Sent Done false
9157 3580 Acked Flush 0 Done Sent Done false
9158 3581 Acked Write 10 Done Sent Done false
9159 3582 Acked Read 5 Done Sent Done false
9160 3583 Acked Read 10 Done Sent Done false
9161 3584 Acked Read 2 Done Sent Done false
9162 3585 Acked Read 5 Done Sent Done false
9163 3586 Acked Read 3 Done Sent Done false
9164 3587 Acked Write 9 Done Sent Done false
9165 3588 Acked Write 6 Done Sent Done false
9166 3589 Acked Read 7 Done Sent Done false
9167 3590 Acked Write 2 Done Sent Done false
9168 3591 Acked Write 4 Done Sent Done false
9169 3592 Acked Read 6 Done Sent Done false
9170 3593 Acked Read 10 Done Sent Done false
9171 3594 Acked Read 1 Done Sent Done false
9172 3595 Acked Flush 0 Done Sent Done false
9173 3596 Acked Read 6 Done Sent Done false
9174 3597 Acked Read 4 Done Sent Done false
9175 3598 Acked Write 4 Done Sent Done false
9176 3599 Acked Read 3 Done Sent Done false
9177 3600 Acked Write 3 Done Sent Done false
9178 3601 Acked Read 7 Done Sent Done false
9179 3602 Acked Flush 0 Done Sent Done false
9180 3603 Acked Write 1 Done Sent Done false
9181 3604 Acked Write 10 Done Sent Done false
9182 3605 Acked Read 8 Done Sent Done false
9183 3606 Acked Write 7 Done Sent Done false
9184 3607 Acked Write 1 Done Sent Done false
9185 3608 Acked Write 7 Done Sent Done false
9186 3609 Acked Read 10 Done Sent Done false
9187 3610 Acked Read 7 Done Sent Done false
9188 3611 Acked Read 2 Done Sent Done false
9189 3612 Acked Write 4 Done Sent Done false
9190 3613 Acked Flush 0 Done Sent Done false
9191 3614 Acked Read 2 Done Sent Done false
9192 3615 Acked Write 5 Done Sent Done false
9193 3616 Acked Write 3 Done Sent Done false
9194 3617 Acked Read 4 Done Sent Done false
9195 3618 Acked Write 8 Done Sent Done false
9196 3619 Acked Read 3 Done Sent Done false
9197 3620 Acked Read 2 Done Sent Done false
9198 3621 Acked Read 10 Done Sent Done false
9199 3622 Acked Read 10 Done Sent Done false
9200 3623 Acked Write 8 Done Sent Done false
9201 3624 Acked Write 10 Done Sent Done false
9202 3625 Acked Flush 0 Done Sent Done false
9203 3626 Acked Read 4 Done Sent Done false
9204 3627 Acked Read 1 Done Sent Done false
9205 3628 Acked Flush 0 Done Sent Done false
9206 3629 Acked Read 3 Done Sent Done false
9207 3630 Acked Read 1 Done Sent Done false
9208 3631 Acked Read 1 Done Sent Done false
9209 3632 Acked Write 9 Done Sent Done false
9210 3633 Acked Read 8 Done Sent Done false
9211 3634 Acked Flush 0 Done Sent Done false
9212 3635 Acked Read 2 Done Sent Done false
9213 3636 Acked Read 7 Done Sent Done false
9214 3637 Acked Read 1 Done Sent Done false
9215 3638 Acked Read 8 Done Sent Done false
9216 3639 Acked Read 5 Done Sent Done false
9217 3640 Acked Write 7 Done Sent Done false
9218 3641 Acked Read 2 Done Sent Done false
9219 3642 Acked Write 7 Done Sent Done false
9220 3643 Acked Write 5 Done Sent Done false
9221 3644 Acked Write 4 Done Sent Done false
9222 3645 Acked Read 6 Done Sent Done false
9223 3646 Acked Read 8 Done Sent Done false
9224 3647 Acked Read 1 Done Sent Done false
9225 3648 Acked Write 7 Done Sent Done false
9226 3649 Acked Read 7 Done Sent Done false
9227 3650 Acked Read 2 Done Sent Done false
9228 3651 Acked Read 10 Done Sent Done false
9229 3652 Acked Read 3 Done Sent Done false
9230 3653 Acked Read 6 Done Sent Done false
9231 3654 Acked Read 6 Done Sent Done false
9232 3655 Acked Read 8 Done Sent Done false
9233 3656 Acked Read 4 Done Sent Done false
9234 3657 Acked Write 8 Done Sent Done false
9235 3658 Acked Read 10 Done Sent Done false
9236 3659 Acked Read 7 Done Sent Done false
9237 3660 Acked Read 2 Done Sent Done false
9238 3661 Acked Write 6 Done Sent Done false
9239 3662 Acked Read 3 Done Sent Done false
9240 3663 Acked Write 6 Done Sent Done false
9241 3664 Acked Read 2 Done Sent Done false
9242 3665 Acked Read 9 Done Sent Done false
9243 3666 Acked Read 10 Done Sent Done false
9244 3667 Acked Read 5 Done Sent Done false
9245 3668 Acked Read 1 Done Sent Done false
9246 3669 Acked Write 3 Done Sent Done false
9247 3670 Acked Write 9 Done Sent Done false
9248 3671 Acked Read 8 Done Sent Done false
9249 3672 Acked Read 6 Done Sent Done false
9250 3673 Acked Write 6 Done Sent Done false
9251 3674 Acked Read 8 Done Sent Done false
9252 3675 Acked Read 4 Done Sent Done false
9253 3676 Acked Read 2 Done Sent Done false
9254 3677 Acked Read 9 Done Sent Done false
9255 3678 Acked Write 2 Done Sent Done false
9256 3679 Acked Read 8 Done Sent Done false
9257 3680 Acked Write 2 Done Sent Done false
9258 3681 Acked Write 3 Done Sent Done false
9259 3682 Acked Read 1 Done Sent Done false
9260 3683 Acked Read 2 Done Sent Done false
9261 3684 Acked Write 7 Done Sent Done false
9262 3685 Acked Write 3 Done Sent Done false
9263 3686 Acked Read 2 Done Sent Done false
9264 3687 Acked Write 4 Done Sent Done false
9265 3688 Acked Write 6 Done Sent Done false
9266 3689 Acked Read 10 Done Sent Done false
9267 3690 Acked Read 9 Done Sent Done false
9268 3691 Acked Read 4 Done Sent Done false
9269 3692 Acked Read 9 Done Sent Done false
9270 3693 Acked Read 10 Done Sent Done false
9271 3694 Acked Flush 0 Done Sent Done false
9272 3695 Acked Write 2 Done Sent Done false
9273 3696 Acked Write 10 Done Sent Done false
9274 3697 Acked Read 7 Done Sent Done false
9275 3698 Acked Read 3 Done Sent Done false
9276 3699 Acked Write 10 Done Sent Done false
9277 3700 Acked Flush 0 Done Sent Done false
9278 3701 Acked Write 2 Done Sent Done false
9279 3702 Acked Write 8 Done Sent Done false
9280 3703 Acked Read 5 Done Sent Done false
9281 3704 Acked Flush 0 Done Sent Done false
9282 3705 Acked Write 7 Done Sent Done false
9283 3706 Acked Write 2 Done Sent Done false
9284 3707 Acked Read 9 Done Sent Done false
9285 3708 Acked Write 9 Done Sent Done false
9286 3709 Acked Read 4 Done Sent Done false
9287 3710 Acked Read 8 Done Sent Done false
9288 3711 Acked Read 1 Done Sent Done false
9289 3712 Acked Read 9 Done Sent Done false
9290 3713 Acked Read 2 Done Sent Done false
9291 3714 Acked Write 2 Done Sent Done false
9292 3715 Acked Read 7 Done Sent Done false
9293 3716 Acked Read 2 Done Sent Done false
9294 STATES DS:0 DS:1 DS:2 TOTAL
9295 Sent 0 301 0 301
9296 Done 301 0 301 602
9297 Skipped 0 0 0 0
9298 Error 0 0 0 0
9299 Last Flush: Some(JobId(3704)) Some(JobId(3415)) Some(JobId(3704))
9300 Downstairs last five completed: 3415 3414 3413 3412 3411
9301 Upstairs last five completed: 3716 3715 3714 3713 3712
9302 CLIENT: Up:0 ds:301 act:2
93032025-01-28T21:56:16.591ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
93042025-01-28T21:56:17.499ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
9305 [8820] got start action
9306 [8820] I am going to restart
9307 Starting downstairs at port 8820
9308 Make output file at "/tmp/dsc/downstairs-8820.txt"
9309 [8820][1] reports Starting
9310 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1211)
9311 [8820][1] reports Running
93122025-01-28T21:56:18.500ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
93132025-01-28T21:56:19.502ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
9314 ----------------------------------------------------------------
9315 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9316 States: Active Offline Active
9317 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9318 3416 Acked Write 2 Done Sent Done false
9319 3417 Acked Write 4 Done Sent Done false
9320 3418 Acked Write 3 Done Sent Done false
9321 3419 Acked Read 2 Done Sent Done false
9322 3420 Acked Write 6 Done Sent Done false
9323 3421 Acked Write 7 Done Sent Done false
9324 3422 Acked Write 1 Done Sent Done false
9325 3423 Acked Read 7 Done Sent Done false
9326 3424 Acked Write 8 Done Sent Done false
9327 3425 Acked Read 2 Done Sent Done false
9328 3426 Acked Write 9 Done Sent Done false
9329 3427 Acked Flush 0 Done Sent Done false
9330 3428 Acked Read 1 Done Sent Done false
9331 3429 Acked Write 2 Done Sent Done false
9332 3430 Acked Read 4 Done Sent Done false
9333 3431 Acked Write 3 Done Sent Done false
9334 3432 Acked Read 3 Done Sent Done false
9335 3433 Acked Write 3 Done Sent Done false
9336 3434 Acked Read 10 Done Sent Done false
9337 3435 Acked Write 8 Done Sent Done false
9338 3436 Acked Write 4 Done Sent Done false
9339 3437 Acked Flush 0 Done Sent Done false
9340 3438 Acked Write 7 Done Sent Done false
9341 3439 Acked Read 4 Done Sent Done false
9342 3440 Acked Read 4 Done Sent Done false
9343 3441 Acked Write 10 Done Sent Done false
9344 3442 Acked Write 4 Done Sent Done false
9345 3443 Acked Write 9 Done Sent Done false
9346 3444 Acked Write 5 Done Sent Done false
9347 3445 Acked Write 1 Done Sent Done false
9348 3446 Acked Write 7 Done Sent Done false
9349 3447 Acked Write 9 Done Sent Done false
9350 3448 Acked Read 2 Done Sent Done false
9351 3449 Acked Read 9 Done Sent Done false
9352 3450 Acked Write 1 Done Sent Done false
9353 3451 Acked Write 6 Done Sent Done false
9354 3452 Acked Read 8 Done Sent Done false
9355 3453 Acked Flush 0 Done Sent Done false
9356 3454 Acked Read 9 Done Sent Done false
9357 3455 Acked Write 2 Done Sent Done false
9358 3456 Acked Write 8 Done Sent Done false
9359 3457 Acked Write 3 Done Sent Done false
9360 3458 Acked Read 6 Done Sent Done false
9361 3459 Acked Write 5 Done Sent Done false
9362 3460 Acked Read 9 Done Sent Done false
9363 3461 Acked Read 5 Done Sent Done false
9364 3462 Acked Read 5 Done Sent Done false
9365 3463 Acked Read 7 Done Sent Done false
9366 3464 Acked Read 8 Done Sent Done false
9367 3465 Acked Read 3 Done Sent Done false
9368 3466 Acked Read 3 Done Sent Done false
9369 3467 Acked Write 10 Done Sent Done false
9370 3468 Acked Read 4 Done Sent Done false
9371 3469 Acked Write 9 Done Sent Done false
9372 3470 Acked Read 9 Done Sent Done false
9373 3471 Acked Flush 0 Done Sent Done false
9374 3472 Acked Write 8 Done Sent Done false
9375 3473 Acked Write 10 Done Sent Done false
9376 3474 Acked Write 8 Done Sent Done false
9377 3475 Acked Read 6 Done Sent Done false
9378 3476 Acked Write 1 Done Sent Done false
9379 3477 Acked Flush 0 Done Sent Done false
9380 3478 Acked Write 8 Done Sent Done false
9381 3479 Acked Write 10 Done Sent Done false
9382 3480 Acked Flush 0 Done Sent Done false
9383 3481 Acked Write 10 Done Sent Done false
9384 3482 Acked Write 5 Done Sent Done false
9385 3483 Acked Write 3 Done Sent Done false
9386 3484 Acked Write 6 Done Sent Done false
9387 3485 Acked Write 9 Done Sent Done false
9388 3486 Acked Write 1 Done Sent Done false
9389 3487 Acked Flush 0 Done Sent Done false
9390 3488 Acked Read 6 Done Sent Done false
9391 3489 Acked Write 3 Done Sent Done false
9392 3490 Acked Read 8 Done Sent Done false
9393 3491 Acked Read 4 Done Sent Done false
9394 3492 Acked Read 6 Done Sent Done false
9395 3493 Acked Write 5 Done Sent Done false
9396 3494 Acked Read 1 Done Sent Done false
9397 3495 Acked Flush 0 Done Sent Done false
9398 3496 Acked Read 2 Done Sent Done false
9399 3497 Acked Write 1 Done Sent Done false
9400 3498 Acked Read 4 Done Sent Done false
9401 3499 Acked Read 6 Done Sent Done false
9402 3500 Acked Write 4 Done Sent Done false
9403 3501 Acked Flush 0 Done Sent Done false
9404 3502 Acked Read 5 Done Sent Done false
9405 3503 Acked Read 8 Done Sent Done false
9406 3504 Acked Flush 0 Done Sent Done false
9407 3505 Acked Write 7 Done Sent Done false
9408 3506 Acked Write 4 Done Sent Done false
9409 3507 Acked Write 3 Done Sent Done false
9410 3508 Acked Read 2 Done Sent Done false
9411 3509 Acked Read 6 Done Sent Done false
9412 3510 Acked Write 5 Done Sent Done false
9413 3511 Acked Write 5 Done Sent Done false
9414 3512 Acked Flush 0 Done Sent Done false
9415 3513 Acked Write 3 Done Sent Done false
9416 3514 Acked Write 4 Done Sent Done false
9417 3515 Acked Read 6 Done Sent Done false
9418 3516 Acked Read 8 Done Sent Done false
9419 3517 Acked Read 4 Done Sent Done false
9420 3518 Acked Read 9 Done Sent Done false
9421 3519 Acked Write 1 Done Sent Done false
9422 3520 Acked Write 3 Done Sent Done false
9423 3521 Acked Write 3 Done Sent Done false
9424 3522 Acked Write 3 Done Sent Done false
9425 3523 Acked Flush 0 Done Sent Done false
9426 3524 Acked Write 5 Done Sent Done false
9427 3525 Acked Flush 0 Done Sent Done false
9428 3526 Acked Read 4 Done Sent Done false
9429 3527 Acked Read 2 Done Sent Done false
9430 3528 Acked Read 10 Done Sent Done false
9431 3529 Acked Write 10 Done Sent Done false
9432 3530 Acked Write 10 Done Sent Done false
9433 3531 Acked Read 9 Done Sent Done false
9434 3532 Acked Write 9 Done Sent Done false
9435 3533 Acked Read 7 Done Sent Done false
9436 3534 Acked Write 3 Done Sent Done false
9437 3535 Acked Write 6 Done Sent Done false
9438 3536 Acked Write 3 Done Sent Done false
9439 3537 Acked Read 3 Done Sent Done false
9440 3538 Acked Read 10 Done Sent Done false
9441 3539 Acked Read 3 Done Sent Done false
9442 3540 Acked Write 2 Done Sent Done false
9443 3541 Acked Read 7 Done Sent Done false
9444 3542 Acked Read 4 Done Sent Done false
9445 3543 Acked Write 1 Done Sent Done false
9446 3544 Acked Read 6 Done Sent Done false
9447 3545 Acked Flush 0 Done Sent Done false
9448 3546 Acked Write 1 Done Sent Done false
9449 3547 Acked Write 6 Done Sent Done false
9450 3548 Acked Read 7 Done Sent Done false
9451 3549 Acked Write 9 Done Sent Done false
9452 3550 Acked Read 3 Done Sent Done false
9453 3551 Acked Read 10 Done Sent Done false
9454 3552 Acked Write 3 Done Sent Done false
9455 3553 Acked Flush 0 Done Sent Done false
9456 3554 Acked Write 4 Done Sent Done false
9457 3555 Acked Read 2 Done Sent Done false
9458 3556 Acked Read 2 Done Sent Done false
9459 3557 Acked Flush 0 Done Sent Done false
9460 3558 Acked Write 7 Done Sent Done false
9461 3559 Acked Read 8 Done Sent Done false
9462 3560 Acked Write 10 Done Sent Done false
9463 3561 Acked Write 10 Done Sent Done false
9464 3562 Acked Write 9 Done Sent Done false
9465 3563 Acked Flush 0 Done Sent Done false
9466 3564 Acked Write 6 Done Sent Done false
9467 3565 Acked Flush 0 Done Sent Done false
9468 3566 Acked Read 3 Done Sent Done false
9469 3567 Acked Read 9 Done Sent Done false
9470 3568 Acked Read 3 Done Sent Done false
9471 3569 Acked Write 3 Done Sent Done false
9472 3570 Acked Read 4 Done Sent Done false
9473 3571 Acked Flush 0 Done Sent Done false
9474 3572 Acked Write 1 Done Sent Done false
9475 3573 Acked Read 6 Done Sent Done false
9476 3574 Acked Read 7 Done Sent Done false
9477 3575 Acked Read 1 Done Sent Done false
9478 3576 Acked Read 8 Done Sent Done false
9479 3577 Acked Read 2 Done Sent Done false
9480 3578 Acked Write 9 Done Sent Done false
9481 3579 Acked Write 1 Done Sent Done false
9482 3580 Acked Flush 0 Done Sent Done false
9483 3581 Acked Write 10 Done Sent Done false
9484 3582 Acked Read 5 Done Sent Done false
9485 3583 Acked Read 10 Done Sent Done false
9486 3584 Acked Read 2 Done Sent Done false
9487 3585 Acked Read 5 Done Sent Done false
9488 3586 Acked Read 3 Done Sent Done false
9489 3587 Acked Write 9 Done Sent Done false
9490 3588 Acked Write 6 Done Sent Done false
9491 3589 Acked Read 7 Done Sent Done false
9492 3590 Acked Write 2 Done Sent Done false
9493 3591 Acked Write 4 Done Sent Done false
9494 3592 Acked Read 6 Done Sent Done false
9495 3593 Acked Read 10 Done Sent Done false
9496 3594 Acked Read 1 Done Sent Done false
9497 3595 Acked Flush 0 Done Sent Done false
9498 3596 Acked Read 6 Done Sent Done false
9499 3597 Acked Read 4 Done Sent Done false
9500 3598 Acked Write 4 Done Sent Done false
9501 3599 Acked Read 3 Done Sent Done false
9502 3600 Acked Write 3 Done Sent Done false
9503 3601 Acked Read 7 Done Sent Done false
9504 3602 Acked Flush 0 Done Sent Done false
9505 3603 Acked Write 1 Done Sent Done false
9506 3604 Acked Write 10 Done Sent Done false
9507 3605 Acked Read 8 Done Sent Done false
9508 3606 Acked Write 7 Done Sent Done false
9509 3607 Acked Write 1 Done Sent Done false
9510 3608 Acked Write 7 Done Sent Done false
9511 3609 Acked Read 10 Done Sent Done false
9512 3610 Acked Read 7 Done Sent Done false
9513 3611 Acked Read 2 Done Sent Done false
9514 3612 Acked Write 4 Done Sent Done false
9515 3613 Acked Flush 0 Done Sent Done false
9516 3614 Acked Read 2 Done Sent Done false
9517 3615 Acked Write 5 Done Sent Done false
9518 3616 Acked Write 3 Done Sent Done false
9519 3617 Acked Read 4 Done Sent Done false
9520 3618 Acked Write 8 Done Sent Done false
9521 3619 Acked Read 3 Done Sent Done false
9522 3620 Acked Read 2 Done Sent Done false
9523 3621 Acked Read 10 Done Sent Done false
9524 3622 Acked Read 10 Done Sent Done false
9525 3623 Acked Write 8 Done Sent Done false
9526 3624 Acked Write 10 Done Sent Done false
9527 3625 Acked Flush 0 Done Sent Done false
9528 3626 Acked Read 4 Done Sent Done false
9529 3627 Acked Read 1 Done Sent Done false
9530 3628 Acked Flush 0 Done Sent Done false
9531 3629 Acked Read 3 Done Sent Done false
9532 3630 Acked Read 1 Done Sent Done false
9533 3631 Acked Read 1 Done Sent Done false
9534 3632 Acked Write 9 Done Sent Done false
9535 3633 Acked Read 8 Done Sent Done false
9536 3634 Acked Flush 0 Done Sent Done false
9537 3635 Acked Read 2 Done Sent Done false
9538 3636 Acked Read 7 Done Sent Done false
9539 3637 Acked Read 1 Done Sent Done false
9540 3638 Acked Read 8 Done Sent Done false
9541 3639 Acked Read 5 Done Sent Done false
9542 3640 Acked Write 7 Done Sent Done false
9543 3641 Acked Read 2 Done Sent Done false
9544 3642 Acked Write 7 Done Sent Done false
9545 3643 Acked Write 5 Done Sent Done false
9546 3644 Acked Write 4 Done Sent Done false
9547 3645 Acked Read 6 Done Sent Done false
9548 3646 Acked Read 8 Done Sent Done false
9549 3647 Acked Read 1 Done Sent Done false
9550 3648 Acked Write 7 Done Sent Done false
9551 3649 Acked Read 7 Done Sent Done false
9552 3650 Acked Read 2 Done Sent Done false
9553 3651 Acked Read 10 Done Sent Done false
9554 3652 Acked Read 3 Done Sent Done false
9555 3653 Acked Read 6 Done Sent Done false
9556 3654 Acked Read 6 Done Sent Done false
9557 3655 Acked Read 8 Done Sent Done false
9558 3656 Acked Read 4 Done Sent Done false
9559 3657 Acked Write 8 Done Sent Done false
9560 3658 Acked Read 10 Done Sent Done false
9561 3659 Acked Read 7 Done Sent Done false
9562 3660 Acked Read 2 Done Sent Done false
9563 3661 Acked Write 6 Done Sent Done false
9564 3662 Acked Read 3 Done Sent Done false
9565 3663 Acked Write 6 Done Sent Done false
9566 3664 Acked Read 2 Done Sent Done false
9567 3665 Acked Read 9 Done Sent Done false
9568 3666 Acked Read 10 Done Sent Done false
9569 3667 Acked Read 5 Done Sent Done false
9570 3668 Acked Read 1 Done Sent Done false
9571 3669 Acked Write 3 Done Sent Done false
9572 3670 Acked Write 9 Done Sent Done false
9573 3671 Acked Read 8 Done Sent Done false
9574 3672 Acked Read 6 Done Sent Done false
9575 3673 Acked Write 6 Done Sent Done false
9576 3674 Acked Read 8 Done Sent Done false
9577 3675 Acked Read 4 Done Sent Done false
9578 3676 Acked Read 2 Done Sent Done false
9579 3677 Acked Read 9 Done Sent Done false
9580 3678 Acked Write 2 Done Sent Done false
9581 3679 Acked Read 8 Done Sent Done false
9582 3680 Acked Write 2 Done Sent Done false
9583 3681 Acked Write 3 Done Sent Done false
9584 3682 Acked Read 1 Done Sent Done false
9585 3683 Acked Read 2 Done Sent Done false
9586 3684 Acked Write 7 Done Sent Done false
9587 3685 Acked Write 3 Done Sent Done false
9588 3686 Acked Read 2 Done Sent Done false
9589 3687 Acked Write 4 Done Sent Done false
9590 3688 Acked Write 6 Done Sent Done false
9591 3689 Acked Read 10 Done Sent Done false
9592 3690 Acked Read 9 Done Sent Done false
9593 3691 Acked Read 4 Done Sent Done false
9594 3692 Acked Read 9 Done Sent Done false
9595 3693 Acked Read 10 Done Sent Done false
9596 3694 Acked Flush 0 Done Sent Done false
9597 3695 Acked Write 2 Done Sent Done false
9598 3696 Acked Write 10 Done Sent Done false
9599 3697 Acked Read 7 Done Sent Done false
9600 3698 Acked Read 3 Done Sent Done false
9601 3699 Acked Write 10 Done Sent Done false
9602 3700 Acked Flush 0 Done Sent Done false
9603 3701 Acked Write 2 Done Sent Done false
9604 3702 Acked Write 8 Done Sent Done false
9605 3703 Acked Read 5 Done Sent Done false
9606 3704 Acked Flush 0 Done Sent Done false
9607 3705 Acked Write 7 Done Sent Done false
9608 3706 Acked Write 2 Done Sent Done false
9609 3707 Acked Read 9 Done Sent Done false
9610 3708 Acked Write 9 Done Sent Done false
9611 3709 Acked Read 4 Done Sent Done false
9612 3710 Acked Read 8 Done Sent Done false
9613 3711 Acked Read 1 Done Sent Done false
9614 3712 Acked Read 9 Done Sent Done false
9615 3713 Acked Read 2 Done Sent Done false
9616 3714 Acked Write 2 Done Sent Done false
9617 3715 Acked Read 7 Done Sent Done false
9618 3716 Acked Read 2 Done Sent Done false
9619 STATES DS:0 DS:1 DS:2 TOTAL
9620 Sent 0 301 0 301
9621 Done 301 0 301 602
9622 Skipped 0 0 0 0
9623 Error 0 0 0 0
9624 Last Flush: Some(JobId(3704)) Some(JobId(3415)) Some(JobId(3704))
9625 Downstairs last five completed: 3415 3414 3413 3412 3411
9626 Upstairs last five completed: 3716 3715 3714 3713 3712
9627 CLIENT: Up:0 ds:301 act:2
96282025-01-28T21:56:20.593ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96292025-01-28T21:56:21.505ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96302025-01-28T21:56:21.850ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96312025-01-28T21:56:21.850ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34429) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96322025-01-28T21:56:21.851ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96332025-01-28T21:56:21.852ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96342025-01-28T21:56:21.852ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96352025-01-28T21:56:21.852ZINFOcrucible: send last flush ID to this DS: Some(JobId(3415)) = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96362025-01-28T21:56:21.852ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96372025-01-28T21:56:21.852ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(3415)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96382025-01-28T21:56:21.852ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(3415)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
9639 ----------------------------------------------------------------
9640 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9641 Downstairs last five completed: 3717 3716 3715 3714 3713
9642 Upstairs last five completed: 3717 3716 3715 3714 3713
9643 CLIENT: Up:0 ds:0 act:3
9644 Replay: All jobs finished
9645 Main task has work to do, go find it
9646 got dsc Stop(2)
9647 stop 2
9648 Jan 28 21:56:24.094 INFO request completed, latency_us: 359, response_code:[8830] Got stop action so:false kr:false
9649 204, uri: /stop/cid/2, method: POST, req_id: 2c1eb69d-f6ba-4beb-afb5-a3b2cae00f8d, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
9650 Jan 28 21:56:24.095 INFO request completed, latency_us: 152, response_code: 200, uri: /state/cid/2, method: GET, req_id: d08a3857-c4e4-4bc9-a0cb-6e3e1073a478, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
96512025-01-28T21:56:24.095ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96522025-01-28T21:56:24.095ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96532025-01-28T21:56:24.095ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
96542025-01-28T21:56:24.095ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
9655 [8830] Exited with: ExitStatus(unix_wait_status(9))
9656 [8830][2] reports Exit
9657 Jan 28 21:56:28.097 INFO request completed, latency_us: 229, response_code: 200, uri: /state/cid/2, method: GET, req_id: 12f80b60-d7d8-4097-95eb-fbe53683e875, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
9658 001/300 Write block 4089 len 36864 data: 4 4 4 4 3 3 3 3 3
9659 002/300 Write block 4886 len 40960 data: 3 4 3 3 3 3 2 2 2 2
9660 003/300 Flush
9661 004/300 Read block 2425 len 20480
9662 005/300 Write block 623 len 12288 data: 2 3 3
9663 006/300 Read block 1970 len 4096
9664 007/300 Read block 1335 len 4096
9665 008/300 Flush
9666 009/300 Read block 2082 len 24576
9667 010/300 Write block 3433 len 32768 data: 4 3 3 4 4 3 3 3
9668 011/300 Read block 4546 len 4096
9669 012/300 Read block 1625 len 8192
9670 013/300 Read block 3557 len 24576
9671 014/300 Flush
9672 015/300 Read block 779 len 16384
9673 016/300 Write block 204 len 32768 data: 2 2 3 3 4 4 4 4
9674 017/300 Flush
9675 018/300 Read block 1258 len 12288
9676 019/300 Read block 1314 len 28672
9677 020/300 Write block 997 len 24576 data: 2 3 3 3 4 4
9678 021/300 Flush
9679 022/300 Read block 3587 len 4096
9680 023/300 Write block 2164 len 32768 data: 3 4 4 4 5 5 4 3
9681 024/300 Write block 4714 len 16384 data: 2 2 2 2
9682 025/300 Write block 3211 len 16384 data: 3 3 5 5
9683 026/300 Read block 77 len 24576
9684 027/300 Write block 2385 len 24576 data: 3 3 3 4 4 5
9685 028/300 Read block 1509 len 12288
9686 029/300 Write block 3112 len 40960 data: 4 4 4 4 4 4 3 3 3 4
9687 030/300 Write block 4853 len 4096 data: 3
9688 031/300 Read block 3412 len 24576
9689 032/300 Read block 4598 len 36864
9690 033/300 Write block 4922 len 40960 data: 3 3 3 3 4 4 3 3 3 2
9691 034/300 Read block 2116 len 4096
9692 035/300 Read block 2336 len 4096
9693 036/300 Write block 2261 len 8192 data: 3 3
9694 037/300 Write block 4752 len 12288 data: 2 2 2
9695 038/300 Write block 4736 len 40960 data: 3 4 3 3 3 3 3 3 2 2
9696 039/300 Read block 3883 len 24576
9697 040/300 Write block 2480 len 16384 data: 3 4 3 3
9698 041/300 Read block 2387 len 12288
9699 042/300 Read block 1996 len 16384
9700 043/300 Write block 2324 len 36864 data: 2 3 3 3 4 4 4 4 4
9701 044/300 Read block 1055 len 8192
9702 045/300 Write block 1321 len 12288 data: 2 2 2
9703 046/300 Flush
9704 047/300 Read block 93 len 20480
9705 048/300 Read block 2446 len 12288
9706 049/300 Flush
9707 050/300 Read block 587 len 40960
9708 051/300 Write block 4133 len 36864 data: 2 2 2 2 3 2 2 3 3
9709 052/300 Write block 2080 len 24576 data: 3 3 3 3 2 2
9710 053/300 Read block 3677 len 28672
9711 054/300 Flush
9712 055/300 Read block 2175 len 16384
9713 056/300 Write block 1140 len 20480 data: 2 2 2 2 2
9714 057/300 Flush
9715 058/300 Read block 4052 len 16384
9716 059/300 Write block 2485 len 4096 data: 2
9717 060/300 Write block 2784 len 8192 data: 3 2
9718 061/300 Read block 1659 len 4096
9719 062/300 Read block 912 len 40960
9720 063/300 Write block 3311 len 32768 data: 3 3 2 2 3 3 3 3
9721 064/300 Read block 1828 len 24576
9722 065/300 Read block 2104 len 8192
9723 066/300 Flush
9724 067/300 Read block 3525 len 12288
9725 068/300 Read block 869 len 40960
9726 069/300 Write block 3307 len 36864 data: 4 3 3 3 4 4 3 3 4
9727 070/300 Read block 2388 len 40960
9728 071/300 Write block 4174 len 40960 data: 3 3 3 3 3 3 2 2 2 3
9729 072/300 Read block 1409 len 20480
9730 073/300 Read block 2371 len 24576
9731 074/300 Write block 2966 len 24576 data: 2 3 3 3 3 4
9732 075/300 Read block 604 len 28672
9733 076/300 Write block 4875 len 4096 data: 4
9734 077/300 Read block 2604 len 20480
9735 078/300 Read block 1591 len 32768
9736 079/300 Read block 538 len 32768
9737 080/300 Write block 4678 len 40960 data: 3 3 3 3 3 3 4 4 3 3
9738 081/300 Read block 4360 len 24576
9739 082/300 Read block 3517 len 4096
9740 083/300 Read block 2654 len 8192
9741 084/300 Read block 2648 len 16384
9742 085/300 Read block 3971 len 20480
9743 086/300 Flush
9744 087/300 Write block 4892 len 8192 data: 3 3
9745 088/300 Read block 3971 len 16384
9746 089/300 Write block 1843 len 4096 data: 2
9747 090/300 Read block 2854 len 8192
9748 091/300 Write block 3528 len 20480 data: 6 5 5 5 5
9749 092/300 Write block 3925 len 16384 data: 4 3 3 4
9750 093/300 Write block 4134 len 36864 data: 3 3 3 4 3 3 4 4 3
9751 094/300 Read block 466 len 40960
9752 095/300 Write block 4728 len 24576 data: 3 4 4 4 5 4
9753 096/300 Write block 3554 len 32768 data: 3 4 4 4 4 5 4 4
9754 097/300 Read block 4135 len 12288
9755 098/300 Write block 4004 len 40960 data: 4 4 3 3 3 3 3 3 3 2
9756 099/300 Write block 2953 len 40960 data: 2 2 2 2 2 2 2 2 2 2
9757 100/300 Read block 227 len 36864
9758 101/300 Read block 3137 len 16384
9759 102/300 Read block 296 len 12288
9760 103/300 Read block 3356 len 28672
9761 104/300 Write block 439 len 4096 data: 2
9762 105/300 Read block 1048 len 32768
9763 106/300 Write block 3455 len 40960 data: 2 2 2 2 2 2 2 2 2 2
9764 107/300 Write block 3943 len 28672 data: 3 3 3 3 3 3 3
9765 108/300 Write block 1974 len 20480 data: 4 3 3 3 2
9766 109/300 Write block 421 len 36864 data: 3 3 3 3 3 3 2 2 2
9767 110/300 Write block 262 len 16384 data: 3 2 2 3
9768 111/300 Read block 4619 len 20480
9769 112/300 Flush
9770 113/300 Write block 1052 len 28672 data: 6 6 6 5 4 5 4
9771 114/300 Read block 888 len 36864
9772 115/300 Read block 3229 len 16384
9773 116/300 Write block 3224 len 28672 data: 3 4 3 3 3 2 2
9774 117/300 Read block 2609 len 28672
9775 118/300 Read block 1290 len 12288
9776 119/300 Write block 4591 len 16384 data: 2 2 2 2
9777 120/300 Read block 3551 len 24576
9778 121/300 Write block 3140 len 36864 data: 2 2 2 2 2 2 2 2 2
9779 122/300 Read block 1461 len 16384
9780 123/300 Flush
9781 124/300 Read block 3274 len 16384
9782 125/300 Write block 571 len 16384 data: 2 2 2 3
9783 126/300 Read block 2920 len 20480
9784 127/300 Read block 2309 len 4096
9785 128/300 Read block 2743 len 24576
9786 129/300 Read block 1063 len 36864
9787 130/300 Read block 4540 len 8192
9788 131/300 Read block 3078 len 32768
9789 132/300 Flush
9790 133/300 Flush
9791 134/300 Read block 3484 len 28672
9792 135/300 Read block 4227 len 8192
9793 136/300 Write block 911 len 32768 data: 3 4 4 5 4 5 5 5
9794 137/300 Read block 2870 len 4096
9795 138/300 Write block 3544 len 36864 data: 3 4 3 4 4 4 4 4 4
9796 139/300 Write block 1912 len 24576 data: 2 3 3 3 3 4
9797 140/300 Flush
9798 141/300 Write block 360 len 40960 data: 4 4 4 3 3 3 4 4 4 4
9799 142/300 Write block 1159 len 20480 data: 3 2 2 2 2
9800 143/300 Write block 4339 len 4096 data: 4
9801 144/300 Write block 2199 len 32768 data: 3 3 5 5 6 6 5 5
9802 145/300 Read block 4811 len 40960
9803 146/300 Read block 1710 len 8192
9804 147/300 Write block 4738 len 4096 data: 4
9805 148/300 Read block 654 len 40960
9806 149/300 Write block 3349 len 16384 data: 3 2 2 3
9807 150/300 Read block 948 len 12288
9808 151/300 Flush
9809 152/300 Write block 4078 len 40960 data: 3 3 3 4 4 5 5 5 4 4
9810 153/300 Read block 1445 len 36864
9811 154/300 Write block 4915 len 28672 data: 3 4 3 3 3 3 3
9812 155/300 Read block 3931 len 12288
9813 156/300 Flush
9814 157/300 Read block 1893 len 16384
9815 158/300 Read block 3317 len 40960
9816 159/300 Write block 1484 len 8192 data: 3 3
9817 160/300 Flush
9818 161/300 Write block 97 len 40960 data: 3 2 2 2 2 2 2 2 2 2
9819 162/300 Read block 2180 len 40960
9820 163/300 Write block 4797 len 32768 data: 3 3 4 4 2 2 2 3
9821 164/300 Read block 2013 len 4096
9822 165/300 Write block 1014 len 12288 data: 3 2 2
9823 166/300 Write block 3200 len 12288 data: 4 4 4
9824 167/300 Read block 3664 len 24576
9825 168/300 Read block 1917 len 24576
9826 169/300 Read block 4397 len 16384
9827 170/300 Read block 823 len 40960
9828 171/300 Read block 2468 len 32768
9829 172/300 Read block 4788 len 16384
9830 173/300 Read block 4420 len 12288
9831 174/300 Read block 4620 len 24576
9832 175/300 Read block 3639 len 32768
9833 176/300 Write block 2860 len 36864 data: 3 3 2 2 2 2 2 2 2
9834 177/300 Write block 3040 len 12288 data: 2 2 3
9835 178/300 Write block 2090 len 40960 data: 2 2 2 2 2 2 2 2 2 3
9836 179/300 Flush
9837 180/300 Write block 4061 len 24576 data: 4 4 4 2 2 3
9838 181/300 Flush
9839 182/300 Read block 93 len 24576
9840 183/300 Read block 4241 len 32768
9841 184/300 Read block 3294 len 32768
9842 185/300 Read block 1812 len 32768
9843 186/300 Flush
9844 187/300 Read block 1658 len 12288
9845 188/300 Write block 4111 len 32768 data: 2 2 2 2 5 4 4 5
9846 189/300 Write block 23 len 8192 data: 2 3
9847 190/300 Write block 586 len 8192 data: 3 3
9848 191/300 Flush
9849 192/300 Write block 24 len 32768 data: 4 3 3 2 2 2 2 2
9850 193/300 Read block 850 len 16384
9851 194/300 Write block 1110 len 12288 data: 4 3 3
9852 195/300 Write block 3306 len 16384 data: 4 5 4 4
9853 196/300 Read block 3679 len 40960
9854 197/300 Read block 921 len 16384
9855 198/300 Read block 4887 len 36864
9856 199/300 Read block 2717 len 28672
9857 200/300 Write block 1797 len 4096 data: 4
9858 201/300 Read block 3712 len 40960
9859 202/300 Write block 4370 len 28672 data: 2 3 4 4 4 4 4
9860 203/300 Read block 2960 len 4096
9861 204/300 Write block 4981 len 40960 data: 3 3 3 4 4 5 5 5 3 3
9862 205/300 Read block 2803 len 12288
9863 206/300 Read block 4788 len 28672
9864 207/300 Read block 3449 len 40960
9865 208/300 Read block 2337 len 36864
9866 209/300 Read block 4769 len 12288
9867 210/300 Read block 2904 len 32768
9868 211/300 Read block 2331 len 40960
9869 212/300 Read block 4610 len 16384
9870 213/300 Read block 4780 len 16384
9871 214/300 Read block 1603 len 40960
9872 215/300 Read block 4236 len 36864
9873 216/300 Read block 1710 len 12288
9874 217/300 Flush
9875 218/300 Write block 1454 len 8192 data: 3 3
9876 219/300 Read block 1486 len 36864
9877 220/300 Read block 996 len 32768
9878 221/300 Read block 1144 len 24576
9879 222/300 Write block 4074 len 40960 data: 5 4 4 3 4 4 4 5 5 6
9880 223/300 Write block 1979 len 28672 data: 3 4 4 4 4 4 4
9881 224/300 Read block 3854 len 32768
9882 225/300 Write block 2756 len 28672 data: 3 3 3 3 3 3 4
9883 226/300 Write block 3549 len 24576 data: 5 5 5 5 4 4
9884 227/300 Read block 883 len 8192
9885 228/300 Write block 1295 len 16384 data: 3 4 4 5
9886 229/300 Write block 2734 len 4096 data: 4
9887 230/300 Read block 1674 len 36864
9888 231/300 Write block 341 len 8192 data: 3 3
9889 232/300 Write block 1883 len 12288 data: 3 3 2
9890 233/300 Write block 4552 len 16384 data: 2 2 2 3
9891 234/300 Write block 270 len 12288 data: 3 2 2
9892 235/300 Write block 3679 len 4096 data: 3
9893 236/300 Write block 2897 len 36864 data: 2 2 2 2 2 2 2 2 2
9894 237/300 Read block 2949 len 36864
9895 238/300 Write block 4746 len 12288 data: 3 3 3
9896 239/300 Write block 355 len 28672 data: 3 3 3 4 3 5 5
9897 240/300 Read block 1980 len 36864
9898 241/300 Read block 4484 len 24576
9899 242/300 Read block 160 len 28672
9900 243/300 Flush
9901 244/300 Write block 3069 len 4096 data: 2
9902 245/300 Read block 2993 len 12288
9903 246/300 Write block 1841 len 40960 data: 2 2 3 2 2 2 2 2 2 3
9904 247/300 Write block 3480 len 20480 data: 2 2 2 2 3
9905 248/300 Read block 3333 len 40960
9906 249/300 Read block 4678 len 36864
9907 250/300 Write block 2518 len 28672 data: 2 2 2 2 2 2 2
9908 251/300 Write block 2266 len 8192 data: 2 2
9909 252/300 Read block 4679 len 40960
9910 253/300 Write block 2505 len 24576 data: 2 2 3 3 3 3
9911 254/300 Read block 3870 len 36864
9912 255/300 Write block 3670 len 28672 data: 3 3 3 3 3 3 3
9913 256/300 Read block 4823 len 24576
9914 257/300 Read block 2564 len 8192
9915 258/300 Write block 126 len 36864 data: 6 3 2 2 2 2 3 3 2
9916 259/300 Read block 4092 len 28672
9917 260/300 Read block 4639 len 20480
9918 261/300 Read block 2645 len 8192
9919 262/300 Read block 1823 len 36864
9920 263/300 Read block 1678 len 12288
9921 264/300 Write block 4894 len 32768 data: 3 3 2 2 2 2 2 2
9922 265/300 Read block 1062 len 36864
9923 266/300 Read block 1595 len 32768
9924 267/300 Write block 70 len 12288 data: 4 3 4
9925 268/300 Read block 3799 len 24576
9926 269/300 Read block 3275 len 8192
9927 270/300 Read block 4992 len 12288
9928 271/300 Read block 3362 len 20480
9929 272/300 Read block 4461 len 40960
9930 273/300 Read block 2327 len 4096
9931 274/300 Flush
9932 275/300 Flush
9933 276/300 Write block 2285 len 20480 data: 3 3 3 3 3
9934 277/300 Read block 3527 len 40960
9935 278/300 Write block 649 len 24576 data: 4 4 6 6 6 4
9936 279/300 Write block 2659 len 36864 data: 4 5 5 4 3 3 3 3 3
9937 280/300 Read block 1895 len 8192
9938 281/300 Write block 1342 len 28672 data: 2 2 4 4 4 4 3
9939 282/300 Flush
9940 283/300 Write block 3762 len 24576 data: 3 3 3 3 3 4
9941 284/300 Read block 1695 len 28672
9942 285/300 Write block 1505 len 16384 data: 3 5 3 3
9943 286/300 Flush
9944 287/300 Write block 347 len 32768 data: 4 3 4 4 4 2 3 3
9945 288/300 Read block 4254 len 4096
9946 289/300 Write block 2659 len 20480 data: 5 6 6 5 4
9947 290/300 Read block 2630 len 16384
9948 291/300 Flush
9949 292/300 Write block 2839 len 24576 data: 3 3 3 3 3 4
9950 293/300 Write block 1775 len 4096 data: 3
9951 294/300 Read block 2045 len 32768
9952 295/300 Read block 4951 len 20480
9953 296/300 Read block 2110 len 36864
9954 297/300 Read block 977 len 16384
9955 298/300 Read block 495 len 36864
9956 299/300 Write block 3262 len 36864 data: 2 2 2 3 2 2 2 3 3
9957 300/300 Read block 406 len 16384
9958 301/300 Write block 4016 len 16384 data: 3 4 4 3
9959 Main task has work to do, go find it
9960 got dsc Start(2)
9961 start 2
9962 Jan 28 21:56:28.346 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/2, method: POST, req_id: dfd6f089-5d6d-4ed0-9154-030244b3b80c, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
9963 Replay: started 2, returned:Ok(())
9964 ----------------------------------------------------------------
9965 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9966 States: Active Active Offline
9967 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9968 3718 Acked Write 9 Done Done Sent false
9969 3719 Acked Write 10 Done Done Sent false
9970 3720 Acked Flush 0 Done Done Sent false
9971 3721 Acked Read 5 Done Done Sent false
9972 3722 Acked Write 3 Done Done Sent false
9973 3723 Acked Read 1 Done Done Sent false
9974 3724 Acked Read 1 Done Done Sent false
9975 3725 Acked Flush 0 Done Done Sent false
9976 3726 Acked Read 6 Done Done Sent false
9977 3727 Acked Write 8 Done Done Sent false
9978 3728 Acked Read 1 Done Done Sent false
9979 3729 Acked Read 2 Done Done Sent false
9980 3730 Acked Read 6 Done Done Sent false
9981 3731 Acked Flush 0 Done Done Sent false
9982 3732 Acked Read 4 Done Done Sent false
9983 3733 Acked Write 8 Done Done Sent false
9984 3734 Acked Flush 0 Done Done Sent false
9985 3735 Acked Read 3 Done Done Sent false
9986 3736 Acked Read 7 Done Done Sent false
9987 3737 Acked Write 6 Done Done Sent false
9988 3738 Acked Flush 0 Done Done Sent false
9989 3739 Acked Read 1 Done Done Sent false
9990 3740 Acked Write 8 Done Done Sent false
9991 3741 Acked Write 4 Done Done Sent false
9992 3742 Acked Write 4 Done Done Sent false
9993 3743 Acked Read 6 Done Done Sent false
9994 3744 Acked Write 6 Done Done Sent false
9995 3745 Acked Read 3 Done Done Sent false
9996 3746 Acked Write 10 Done Done Sent false
9997 3747 Acked Write 1 Done Done Sent false
9998 3748 Acked Read 6 Done Done Sent false
9999 3749 Acked Read 9 Done Done Sent false
10000 3750 Acked Write 10 Done Done Sent false
10001 3751 Acked Read 1 Done Done Sent false
10002 3752 Acked Read 1 Done Done Sent false
10003 3753 Acked Write 2 Done Done Sent false
10004 3754 Acked Write 3 Done Done Sent false
10005 3755 Acked Write 10 Done Done Sent false
10006 3756 Acked Read 6 Done Done Sent false
10007 3757 Acked Write 4 Done Done Sent false
10008 3758 Acked Read 3 Done Done Sent false
10009 3759 Acked Read 4 Done Done Sent false
10010 3760 Acked Write 9 Done Done Sent false
10011 3761 Acked Read 2 Done Done Sent false
10012 3762 Acked Write 3 Done Done Sent false
10013 3763 Acked Flush 0 Done Done Sent false
10014 3764 Acked Read 5 Done Done Sent false
10015 3765 Acked Read 3 Done Done Sent false
10016 3766 Acked Flush 0 Done Done Sent false
10017 3767 Acked Read 10 Done Done Sent false
10018 3768 Acked Write 9 Done Done Sent false
10019 3769 Acked Write 6 Done Done Sent false
10020 3770 Acked Read 7 Done Done Sent false
10021 3771 Acked Flush 0 Done Done Sent false
10022 3772 Acked Read 4 Done Done Sent false
10023 3773 Acked Write 5 Done Done Sent false
10024 3774 Acked Flush 0 Done Done Sent false
10025 3775 Acked Read 4 Done Done Sent false
10026 3776 Acked Write 1 Done Done Sent false
10027 3777 Acked Write 2 Done Done Sent false
10028 3778 Acked Read 1 Done Done Sent false
10029 3779 Acked Read 10 Done Done Sent false
10030 3780 Acked Write 8 Done Done Sent false
10031 3781 Acked Read 6 Done Done Sent false
10032 3782 Acked Read 2 Done Done Sent false
10033 3783 Acked Flush 0 Done Done Sent false
10034 3784 Acked Read 3 Done Done Sent false
10035 3785 Acked Read 10 Done Done Sent false
10036 3786 Acked Write 9 Done Done Sent false
10037 3787 Acked Read 10 Done Done Sent false
10038 3788 Acked Write 10 Done Done Sent false
10039 3789 Acked Read 5 Done Done Sent false
10040 3790 Acked Read 6 Done Done Sent false
10041 3791 Acked Write 6 Done Done Sent false
10042 3792 Acked Read 7 Done Done Sent false
10043 3793 Acked Write 1 Done Done Sent false
10044 3794 Acked Read 5 Done Done Sent false
10045 3795 Acked Read 8 Done Done Sent false
10046 3796 Acked Read 8 Done Done Sent false
10047 3797 Acked Write 10 Done Done Sent false
10048 3798 Acked Read 6 Done Done Sent false
10049 3799 Acked Read 1 Done Done Sent false
10050 3800 Acked Read 2 Done Done Sent false
10051 3801 Acked Read 4 Done Done Sent false
10052 3802 Acked Read 5 Done Done Sent false
10053 3803 Acked Flush 0 Done Done Sent false
10054 3804 Acked Write 2 Done Done Sent false
10055 3805 Acked Read 4 Done Done Sent false
10056 3806 Acked Write 1 Done Done Sent false
10057 3807 Acked Read 2 Done Done Sent false
10058 3808 Acked Write 5 Done Done Sent false
10059 3809 Acked Write 4 Done Done Sent false
10060 3810 Acked Write 9 Done Done Sent false
10061 3811 Acked Read 10 Done Done Sent false
10062 3812 Acked Write 6 Done Done Sent false
10063 3813 Acked Write 8 Done Done Sent false
10064 3814 Acked Read 3 Done Done Sent false
10065 3815 Acked Write 10 Done Done Sent false
10066 3816 Acked Write 10 Done Done Sent false
10067 3817 Acked Read 9 Done Done Sent false
10068 3818 Acked Read 4 Done Done Sent false
10069 3819 Acked Read 3 Done Done Sent false
10070 3820 Acked Read 7 Done Done Sent false
10071 3821 Acked Write 1 Done Done Sent false
10072 3822 Acked Read 8 Done Done Sent false
10073 3823 Acked Write 10 Done Done Sent false
10074 3824 Acked Write 7 Done Done Sent false
10075 3825 Acked Write 5 Done Done Sent false
10076 3826 Acked Write 9 Done Done Sent false
10077 3827 Acked Write 4 Done Done Sent false
10078 3828 Acked Read 5 Done Done Sent false
10079 3829 Acked Flush 0 Done Done Sent false
10080 3830 Acked Write 7 Done Done Sent false
10081 3831 Acked Read 9 Done Done Sent false
10082 3832 Acked Read 4 Done Done Sent false
10083 3833 Acked Write 7 Done Done Sent false
10084 3834 Acked Read 7 Done Done Sent false
10085 3835 Acked Read 3 Done Done Sent false
10086 3836 Acked Write 4 Done Done Sent false
10087 3837 Acked Read 6 Done Done Sent false
10088 3838 Acked Write 9 Done Done Sent false
10089 3839 Acked Read 4 Done Done Sent false
10090 3840 Acked Flush 0 Done Done Sent false
10091 3841 Acked Read 4 Done Done Sent false
10092 3842 Acked Write 4 Done Done Sent false
10093 3843 Acked Read 5 Done Done Sent false
10094 3844 Acked Read 1 Done Done Sent false
10095 3845 Acked Read 6 Done Done Sent false
10096 3846 Acked Read 9 Done Done Sent false
10097 3847 Acked Read 2 Done Done Sent false
10098 3848 Acked Read 8 Done Done Sent false
10099 3849 Acked Flush 0 Done Done Sent false
10100 3850 Acked Flush 0 Done Done Sent false
10101 3851 Acked Read 7 Done Done Sent false
10102 3852 Acked Read 2 Done Done Sent false
10103 3853 Acked Write 8 Done Done Sent false
10104 3854 Acked Read 1 Done Done Sent false
10105 3855 Acked Write 9 Done Done Sent false
10106 3856 Acked Write 6 Done Done Sent false
10107 3857 Acked Flush 0 Done Done Sent false
10108 3858 Acked Write 10 Done Done Sent false
10109 3859 Acked Write 5 Done Done Sent false
10110 3860 Acked Write 1 Done Done Sent false
10111 3861 Acked Write 8 Done Done Sent false
10112 3862 Acked Read 10 Done Done Sent false
10113 3863 Acked Read 2 Done Done Sent false
10114 3864 Acked Write 1 Done Done Sent false
10115 3865 Acked Read 10 Done Done Sent false
10116 3866 Acked Write 4 Done Done Sent false
10117 3867 Acked Read 3 Done Done Sent false
10118 3868 Acked Flush 0 Done Done Sent false
10119 3869 Acked Write 10 Done Done Sent false
10120 3870 Acked Read 9 Done Done Sent false
10121 3871 Acked Write 7 Done Done Sent false
10122 3872 Acked Read 3 Done Done Sent false
10123 3873 Acked Flush 0 Done Done Sent false
10124 3874 Acked Read 4 Done Done Sent false
10125 3875 Acked Read 10 Done Done Sent false
10126 3876 Acked Write 2 Done Done Sent false
10127 3877 Acked Flush 0 Done Done Sent false
10128 3878 Acked Write 10 Done Done Sent false
10129 3879 Acked Read 10 Done Done Sent false
10130 3880 Acked Write 8 Done Done Sent false
10131 3881 Acked Read 1 Done Done Sent false
10132 3882 Acked Write 3 Done Done Sent false
10133 3883 Acked Write 3 Done Done Sent false
10134 3884 Acked Read 6 Done Done Sent false
10135 3885 Acked Read 6 Done Done Sent false
10136 3886 Acked Read 4 Done Done Sent false
10137 3887 Acked Read 10 Done Done Sent false
10138 3888 Acked Read 8 Done Done Sent false
10139 3889 Acked Read 4 Done Done Sent false
10140 3890 Acked Read 3 Done Done Sent false
10141 3891 Acked Read 6 Done Done Sent false
10142 3892 Acked Read 8 Done Done Sent false
10143 3893 Acked Write 9 Done Done Sent false
10144 3894 Acked Write 3 Done Done Sent false
10145 3895 Acked Write 10 Done Done Sent false
10146 3896 Acked Flush 0 Done Done Sent false
10147 3897 Acked Write 6 Done Done Sent false
10148 3898 Acked Flush 0 Done Done Sent false
10149 3899 Acked Read 6 Done Done Sent false
10150 3900 Acked Read 8 Done Done Sent false
10151 3901 Acked Read 8 Done Done Sent false
10152 3902 Acked Read 8 Done Done Sent false
10153 3903 Acked Flush 0 Done Done Sent false
10154 3904 Acked Read 3 Done Done Sent false
10155 3905 Acked Write 8 Done Done Sent false
10156 3906 Acked Write 2 Done Done Sent false
10157 3907 Acked Write 2 Done Done Sent false
10158 3908 Acked Flush 0 Done Done Sent false
10159 3909 Acked Write 8 Done Done Sent false
10160 3910 Acked Read 4 Done Done Sent false
10161 3911 Acked Write 3 Done Done Sent false
10162 3912 Acked Write 4 Done Done Sent false
10163 3913 Acked Read 10 Done Done Sent false
10164 3914 Acked Read 4 Done Done Sent false
10165 3915 Acked Read 9 Done Done Sent false
10166 3916 Acked Read 7 Done Done Sent false
10167 3917 Acked Write 1 Done Done Sent false
10168 3918 Acked Read 10 Done Done Sent false
10169 3919 Acked Write 7 Done Done Sent false
10170 3920 Acked Read 1 Done Done Sent false
10171 3921 Acked Write 10 Done Done Sent false
10172 3922 Acked Read 3 Done Done Sent false
10173 3923 Acked Read 7 Done Done Sent false
10174 3924 Acked Read 10 Done Done Sent false
10175 3925 Acked Read 9 Done Done Sent false
10176 3926 Acked Read 3 Done Done Sent false
10177 3927 Acked Read 8 Done Done Sent false
10178 3928 Acked Read 10 Done Done Sent false
10179 3929 Acked Read 4 Done Done Sent false
10180 3930 Acked Read 4 Done Done Sent false
10181 3931 Acked Read 10 Done Done Sent false
10182 3932 Acked Read 9 Done Done Sent false
10183 3933 Acked Read 3 Done Done Sent false
10184 3934 Acked Flush 0 Done Done Sent false
10185 3935 Acked Write 2 Done Done Sent false
10186 3936 Acked Read 9 Done Done Sent false
10187 3937 Acked Read 8 Done Done Sent false
10188 3938 Acked Read 6 Done Done Sent false
10189 3939 Acked Write 10 Done Done Sent false
10190 3940 Acked Write 7 Done Done Sent false
10191 3941 Acked Read 8 Done Done Sent false
10192 3942 Acked Write 7 Done Done Sent false
10193 3943 Acked Write 6 Done Done Sent false
10194 3944 Acked Read 2 Done Done Sent false
10195 3945 Acked Write 4 Done Done Sent false
10196 3946 Acked Write 1 Done Done Sent false
10197 3947 Acked Read 9 Done Done Sent false
10198 3948 Acked Write 2 Done Done Sent false
10199 3949 Acked Write 3 Done Done Sent false
10200 3950 Acked Write 4 Done Done Sent false
10201 3951 Acked Write 3 Done Done Sent false
10202 3952 Acked Write 1 Done Done Sent false
10203 3953 Acked Write 9 Done Done Sent false
10204 3954 Acked Read 9 Done Done Sent false
10205 3955 Acked Write 3 Done Done Sent false
10206 3956 Acked Write 7 Done Done Sent false
10207 3957 Acked Read 9 Done Done Sent false
10208 3958 Acked Read 6 Done Done Sent false
10209 3959 Acked Read 7 Done Done Sent false
10210 3960 Acked Flush 0 Done Done Sent false
10211 3961 Acked Write 1 Done Done Sent false
10212 3962 Acked Read 3 Done Done Sent false
10213 3963 Acked Write 10 Done Done Sent false
10214 3964 Acked Write 5 Done Done Sent false
10215 3965 Acked Read 10 Done Done Sent false
10216 3966 Acked Read 9 Done Done Sent false
10217 3967 Acked Write 7 Done Done Sent false
10218 3968 Acked Write 2 Done Done Sent false
10219 3969 Acked Read 10 Done Done Sent false
10220 3970 Acked Write 6 Done Done Sent false
10221 3971 Acked Read 9 Done Done Sent false
10222 3972 Acked Write 7 Done Done Sent false
10223 3973 Acked Read 6 Done Done Sent false
10224 3974 Acked Read 2 Done Done Sent false
10225 3975 Acked Write 9 Done Done Sent false
10226 3976 Acked Read 7 Done Done Sent false
10227 3977 Acked Read 5 Done Done Sent false
10228 3978 Acked Read 2 Done Done Sent false
10229 3979 Acked Read 9 Done Done Sent false
10230 3980 Acked Read 3 Done Done Sent false
10231 3981 Acked Write 8 Done Done Sent false
10232 3982 Acked Read 9 Done Done Sent false
10233 3983 Acked Read 8 Done Done Sent false
10234 3984 Acked Write 3 Done Done Sent false
10235 3985 Acked Read 6 Done Done Sent false
10236 3986 Acked Read 2 Done Done Sent false
10237 3987 Acked Read 3 Done Done Sent false
10238 3988 Acked Read 5 Done Done Sent false
10239 3989 Acked Read 10 Done Done Sent false
10240 3990 Acked Read 1 Done Done Sent false
10241 3991 Acked Flush 0 Done Done Sent false
10242 3992 Acked Flush 0 Done Done Sent false
10243 3993 Acked Write 5 Done Done Sent false
10244 3994 Acked Read 10 Done Done Sent false
10245 3995 Acked Write 6 Done Done Sent false
10246 3996 Acked Write 9 Done Done Sent false
10247 3997 Acked Read 2 Done Done Sent false
10248 3998 Acked Write 7 Done Done Sent false
10249 3999 Acked Flush 0 Done Done Sent false
10250 4000 Acked Write 6 Done Done Sent false
10251 4001 Acked Read 7 Done Done Sent false
10252 4002 Acked Write 4 Done Done Sent false
10253 4003 Acked Flush 0 Done Done Sent false
10254 4004 Acked Write 8 Done Done Sent false
10255 4005 Acked Read 1 Done Done Sent false
10256 4006 Acked Write 5 Done Done Sent false
10257 4007 Acked Read 4 Done Done Sent false
10258 4008 Acked Flush 0 Done Done Sent false
10259 4009 Acked Write 6 Done Done Sent false
10260 4010 Acked Write 1 Done Done Sent false
10261 4011 Acked Read 8 Done Done Sent false
10262 4012 Acked Read 5 Done Done Sent false
10263 4013 Acked Read 9 Done Done Sent false
10264 4014 Acked Read 4 Done Done Sent false
10265 4015 Acked Read 9 Done Done Sent false
10266 4016 Acked Write 9 Done Done Sent false
10267 4017 Acked Read 4 Done Done Sent false
10268 4018 Acked Write 4 Done Done Sent false
10269 STATES DS:0 DS:1 DS:2 TOTAL
10270 Sent 0 0 301 301
10271 Done 301 301 0 602
10272 Skipped 0 0 0 0
10273 Error 0 0 0 0
10274 Last Flush: Some(JobId(4008)) Some(JobId(4008)) Some(JobId(3717))
10275 Downstairs last five completed: 3717 3716 3715 3714 3713
10276 Upstairs last five completed: 4018 4017 4016 4015 4014
10277 CLIENT: Up:0 ds:301 act:2
102782025-01-28T21:56:28.848ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
102792025-01-28T21:56:29.518ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
10280 [8830] got start action
10281 [8830] I am going to restart
10282 Starting downstairs at port 8830
10283 Make output file at "/tmp/dsc/downstairs-8830.txt"
10284 [8830][2] reports Starting
10285 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1212)
10286 [8830][2] reports Running
102872025-01-28T21:56:30.519ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
102882025-01-28T21:56:31.520ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
10289 ----------------------------------------------------------------
10290 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10291 States: Active Active Offline
10292 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10293 3718 Acked Write 9 Done Done Sent false
10294 3719 Acked Write 10 Done Done Sent false
10295 3720 Acked Flush 0 Done Done Sent false
10296 3721 Acked Read 5 Done Done Sent false
10297 3722 Acked Write 3 Done Done Sent false
10298 3723 Acked Read 1 Done Done Sent false
10299 3724 Acked Read 1 Done Done Sent false
10300 3725 Acked Flush 0 Done Done Sent false
10301 3726 Acked Read 6 Done Done Sent false
10302 3727 Acked Write 8 Done Done Sent false
10303 3728 Acked Read 1 Done Done Sent false
10304 3729 Acked Read 2 Done Done Sent false
10305 3730 Acked Read 6 Done Done Sent false
10306 3731 Acked Flush 0 Done Done Sent false
10307 3732 Acked Read 4 Done Done Sent false
10308 3733 Acked Write 8 Done Done Sent false
10309 3734 Acked Flush 0 Done Done Sent false
10310 3735 Acked Read 3 Done Done Sent false
10311 3736 Acked Read 7 Done Done Sent false
10312 3737 Acked Write 6 Done Done Sent false
10313 3738 Acked Flush 0 Done Done Sent false
10314 3739 Acked Read 1 Done Done Sent false
10315 3740 Acked Write 8 Done Done Sent false
10316 3741 Acked Write 4 Done Done Sent false
10317 3742 Acked Write 4 Done Done Sent false
10318 3743 Acked Read 6 Done Done Sent false
10319 3744 Acked Write 6 Done Done Sent false
10320 3745 Acked Read 3 Done Done Sent false
10321 3746 Acked Write 10 Done Done Sent false
10322 3747 Acked Write 1 Done Done Sent false
10323 3748 Acked Read 6 Done Done Sent false
10324 3749 Acked Read 9 Done Done Sent false
10325 3750 Acked Write 10 Done Done Sent false
10326 3751 Acked Read 1 Done Done Sent false
10327 3752 Acked Read 1 Done Done Sent false
10328 3753 Acked Write 2 Done Done Sent false
10329 3754 Acked Write 3 Done Done Sent false
10330 3755 Acked Write 10 Done Done Sent false
10331 3756 Acked Read 6 Done Done Sent false
10332 3757 Acked Write 4 Done Done Sent false
10333 3758 Acked Read 3 Done Done Sent false
10334 3759 Acked Read 4 Done Done Sent false
10335 3760 Acked Write 9 Done Done Sent false
10336 3761 Acked Read 2 Done Done Sent false
10337 3762 Acked Write 3 Done Done Sent false
10338 3763 Acked Flush 0 Done Done Sent false
10339 3764 Acked Read 5 Done Done Sent false
10340 3765 Acked Read 3 Done Done Sent false
10341 3766 Acked Flush 0 Done Done Sent false
10342 3767 Acked Read 10 Done Done Sent false
10343 3768 Acked Write 9 Done Done Sent false
10344 3769 Acked Write 6 Done Done Sent false
10345 3770 Acked Read 7 Done Done Sent false
10346 3771 Acked Flush 0 Done Done Sent false
10347 3772 Acked Read 4 Done Done Sent false
10348 3773 Acked Write 5 Done Done Sent false
10349 3774 Acked Flush 0 Done Done Sent false
10350 3775 Acked Read 4 Done Done Sent false
10351 3776 Acked Write 1 Done Done Sent false
10352 3777 Acked Write 2 Done Done Sent false
10353 3778 Acked Read 1 Done Done Sent false
10354 3779 Acked Read 10 Done Done Sent false
10355 3780 Acked Write 8 Done Done Sent false
10356 3781 Acked Read 6 Done Done Sent false
10357 3782 Acked Read 2 Done Done Sent false
10358 3783 Acked Flush 0 Done Done Sent false
10359 3784 Acked Read 3 Done Done Sent false
10360 3785 Acked Read 10 Done Done Sent false
10361 3786 Acked Write 9 Done Done Sent false
10362 3787 Acked Read 10 Done Done Sent false
10363 3788 Acked Write 10 Done Done Sent false
10364 3789 Acked Read 5 Done Done Sent false
10365 3790 Acked Read 6 Done Done Sent false
10366 3791 Acked Write 6 Done Done Sent false
10367 3792 Acked Read 7 Done Done Sent false
10368 3793 Acked Write 1 Done Done Sent false
10369 3794 Acked Read 5 Done Done Sent false
10370 3795 Acked Read 8 Done Done Sent false
10371 3796 Acked Read 8 Done Done Sent false
10372 3797 Acked Write 10 Done Done Sent false
10373 3798 Acked Read 6 Done Done Sent false
10374 3799 Acked Read 1 Done Done Sent false
10375 3800 Acked Read 2 Done Done Sent false
10376 3801 Acked Read 4 Done Done Sent false
10377 3802 Acked Read 5 Done Done Sent false
10378 3803 Acked Flush 0 Done Done Sent false
10379 3804 Acked Write 2 Done Done Sent false
10380 3805 Acked Read 4 Done Done Sent false
10381 3806 Acked Write 1 Done Done Sent false
10382 3807 Acked Read 2 Done Done Sent false
10383 3808 Acked Write 5 Done Done Sent false
10384 3809 Acked Write 4 Done Done Sent false
10385 3810 Acked Write 9 Done Done Sent false
10386 3811 Acked Read 10 Done Done Sent false
10387 3812 Acked Write 6 Done Done Sent false
10388 3813 Acked Write 8 Done Done Sent false
10389 3814 Acked Read 3 Done Done Sent false
10390 3815 Acked Write 10 Done Done Sent false
10391 3816 Acked Write 10 Done Done Sent false
10392 3817 Acked Read 9 Done Done Sent false
10393 3818 Acked Read 4 Done Done Sent false
10394 3819 Acked Read 3 Done Done Sent false
10395 3820 Acked Read 7 Done Done Sent false
10396 3821 Acked Write 1 Done Done Sent false
10397 3822 Acked Read 8 Done Done Sent false
10398 3823 Acked Write 10 Done Done Sent false
10399 3824 Acked Write 7 Done Done Sent false
10400 3825 Acked Write 5 Done Done Sent false
10401 3826 Acked Write 9 Done Done Sent false
10402 3827 Acked Write 4 Done Done Sent false
10403 3828 Acked Read 5 Done Done Sent false
10404 3829 Acked Flush 0 Done Done Sent false
10405 3830 Acked Write 7 Done Done Sent false
10406 3831 Acked Read 9 Done Done Sent false
10407 3832 Acked Read 4 Done Done Sent false
10408 3833 Acked Write 7 Done Done Sent false
10409 3834 Acked Read 7 Done Done Sent false
10410 3835 Acked Read 3 Done Done Sent false
10411 3836 Acked Write 4 Done Done Sent false
10412 3837 Acked Read 6 Done Done Sent false
10413 3838 Acked Write 9 Done Done Sent false
10414 3839 Acked Read 4 Done Done Sent false
10415 3840 Acked Flush 0 Done Done Sent false
10416 3841 Acked Read 4 Done Done Sent false
10417 3842 Acked Write 4 Done Done Sent false
10418 3843 Acked Read 5 Done Done Sent false
10419 3844 Acked Read 1 Done Done Sent false
10420 3845 Acked Read 6 Done Done Sent false
10421 3846 Acked Read 9 Done Done Sent false
10422 3847 Acked Read 2 Done Done Sent false
10423 3848 Acked Read 8 Done Done Sent false
10424 3849 Acked Flush 0 Done Done Sent false
10425 3850 Acked Flush 0 Done Done Sent false
10426 3851 Acked Read 7 Done Done Sent false
10427 3852 Acked Read 2 Done Done Sent false
10428 3853 Acked Write 8 Done Done Sent false
10429 3854 Acked Read 1 Done Done Sent false
10430 3855 Acked Write 9 Done Done Sent false
10431 3856 Acked Write 6 Done Done Sent false
10432 3857 Acked Flush 0 Done Done Sent false
10433 3858 Acked Write 10 Done Done Sent false
10434 3859 Acked Write 5 Done Done Sent false
10435 3860 Acked Write 1 Done Done Sent false
10436 3861 Acked Write 8 Done Done Sent false
10437 3862 Acked Read 10 Done Done Sent false
10438 3863 Acked Read 2 Done Done Sent false
10439 3864 Acked Write 1 Done Done Sent false
10440 3865 Acked Read 10 Done Done Sent false
10441 3866 Acked Write 4 Done Done Sent false
10442 3867 Acked Read 3 Done Done Sent false
10443 3868 Acked Flush 0 Done Done Sent false
10444 3869 Acked Write 10 Done Done Sent false
10445 3870 Acked Read 9 Done Done Sent false
10446 3871 Acked Write 7 Done Done Sent false
10447 3872 Acked Read 3 Done Done Sent false
10448 3873 Acked Flush 0 Done Done Sent false
10449 3874 Acked Read 4 Done Done Sent false
10450 3875 Acked Read 10 Done Done Sent false
10451 3876 Acked Write 2 Done Done Sent false
10452 3877 Acked Flush 0 Done Done Sent false
10453 3878 Acked Write 10 Done Done Sent false
10454 3879 Acked Read 10 Done Done Sent false
10455 3880 Acked Write 8 Done Done Sent false
10456 3881 Acked Read 1 Done Done Sent false
10457 3882 Acked Write 3 Done Done Sent false
10458 3883 Acked Write 3 Done Done Sent false
10459 3884 Acked Read 6 Done Done Sent false
10460 3885 Acked Read 6 Done Done Sent false
10461 3886 Acked Read 4 Done Done Sent false
10462 3887 Acked Read 10 Done Done Sent false
10463 3888 Acked Read 8 Done Done Sent false
10464 3889 Acked Read 4 Done Done Sent false
10465 3890 Acked Read 3 Done Done Sent false
10466 3891 Acked Read 6 Done Done Sent false
10467 3892 Acked Read 8 Done Done Sent false
10468 3893 Acked Write 9 Done Done Sent false
10469 3894 Acked Write 3 Done Done Sent false
10470 3895 Acked Write 10 Done Done Sent false
10471 3896 Acked Flush 0 Done Done Sent false
10472 3897 Acked Write 6 Done Done Sent false
10473 3898 Acked Flush 0 Done Done Sent false
10474 3899 Acked Read 6 Done Done Sent false
10475 3900 Acked Read 8 Done Done Sent false
10476 3901 Acked Read 8 Done Done Sent false
10477 3902 Acked Read 8 Done Done Sent false
10478 3903 Acked Flush 0 Done Done Sent false
10479 3904 Acked Read 3 Done Done Sent false
10480 3905 Acked Write 8 Done Done Sent false
10481 3906 Acked Write 2 Done Done Sent false
10482 3907 Acked Write 2 Done Done Sent false
10483 3908 Acked Flush 0 Done Done Sent false
10484 3909 Acked Write 8 Done Done Sent false
10485 3910 Acked Read 4 Done Done Sent false
10486 3911 Acked Write 3 Done Done Sent false
10487 3912 Acked Write 4 Done Done Sent false
10488 3913 Acked Read 10 Done Done Sent false
10489 3914 Acked Read 4 Done Done Sent false
10490 3915 Acked Read 9 Done Done Sent false
10491 3916 Acked Read 7 Done Done Sent false
10492 3917 Acked Write 1 Done Done Sent false
10493 3918 Acked Read 10 Done Done Sent false
10494 3919 Acked Write 7 Done Done Sent false
10495 3920 Acked Read 1 Done Done Sent false
10496 3921 Acked Write 10 Done Done Sent false
10497 3922 Acked Read 3 Done Done Sent false
10498 3923 Acked Read 7 Done Done Sent false
10499 3924 Acked Read 10 Done Done Sent false
10500 3925 Acked Read 9 Done Done Sent false
10501 3926 Acked Read 3 Done Done Sent false
10502 3927 Acked Read 8 Done Done Sent false
10503 3928 Acked Read 10 Done Done Sent false
10504 3929 Acked Read 4 Done Done Sent false
10505 3930 Acked Read 4 Done Done Sent false
10506 3931 Acked Read 10 Done Done Sent false
10507 3932 Acked Read 9 Done Done Sent false
10508 3933 Acked Read 3 Done Done Sent false
10509 3934 Acked Flush 0 Done Done Sent false
10510 3935 Acked Write 2 Done Done Sent false
10511 3936 Acked Read 9 Done Done Sent false
10512 3937 Acked Read 8 Done Done Sent false
10513 3938 Acked Read 6 Done Done Sent false
10514 3939 Acked Write 10 Done Done Sent false
10515 3940 Acked Write 7 Done Done Sent false
10516 3941 Acked Read 8 Done Done Sent false
10517 3942 Acked Write 7 Done Done Sent false
10518 3943 Acked Write 6 Done Done Sent false
10519 3944 Acked Read 2 Done Done Sent false
10520 3945 Acked Write 4 Done Done Sent false
10521 3946 Acked Write 1 Done Done Sent false
10522 3947 Acked Read 9 Done Done Sent false
10523 3948 Acked Write 2 Done Done Sent false
10524 3949 Acked Write 3 Done Done Sent false
10525 3950 Acked Write 4 Done Done Sent false
10526 3951 Acked Write 3 Done Done Sent false
10527 3952 Acked Write 1 Done Done Sent false
10528 3953 Acked Write 9 Done Done Sent false
10529 3954 Acked Read 9 Done Done Sent false
10530 3955 Acked Write 3 Done Done Sent false
10531 3956 Acked Write 7 Done Done Sent false
10532 3957 Acked Read 9 Done Done Sent false
10533 3958 Acked Read 6 Done Done Sent false
10534 3959 Acked Read 7 Done Done Sent false
10535 3960 Acked Flush 0 Done Done Sent false
10536 3961 Acked Write 1 Done Done Sent false
10537 3962 Acked Read 3 Done Done Sent false
10538 3963 Acked Write 10 Done Done Sent false
10539 3964 Acked Write 5 Done Done Sent false
10540 3965 Acked Read 10 Done Done Sent false
10541 3966 Acked Read 9 Done Done Sent false
10542 3967 Acked Write 7 Done Done Sent false
10543 3968 Acked Write 2 Done Done Sent false
10544 3969 Acked Read 10 Done Done Sent false
10545 3970 Acked Write 6 Done Done Sent false
10546 3971 Acked Read 9 Done Done Sent false
10547 3972 Acked Write 7 Done Done Sent false
10548 3973 Acked Read 6 Done Done Sent false
10549 3974 Acked Read 2 Done Done Sent false
10550 3975 Acked Write 9 Done Done Sent false
10551 3976 Acked Read 7 Done Done Sent false
10552 3977 Acked Read 5 Done Done Sent false
10553 3978 Acked Read 2 Done Done Sent false
10554 3979 Acked Read 9 Done Done Sent false
10555 3980 Acked Read 3 Done Done Sent false
10556 3981 Acked Write 8 Done Done Sent false
10557 3982 Acked Read 9 Done Done Sent false
10558 3983 Acked Read 8 Done Done Sent false
10559 3984 Acked Write 3 Done Done Sent false
10560 3985 Acked Read 6 Done Done Sent false
10561 3986 Acked Read 2 Done Done Sent false
10562 3987 Acked Read 3 Done Done Sent false
10563 3988 Acked Read 5 Done Done Sent false
10564 3989 Acked Read 10 Done Done Sent false
10565 3990 Acked Read 1 Done Done Sent false
10566 3991 Acked Flush 0 Done Done Sent false
10567 3992 Acked Flush 0 Done Done Sent false
10568 3993 Acked Write 5 Done Done Sent false
10569 3994 Acked Read 10 Done Done Sent false
10570 3995 Acked Write 6 Done Done Sent false
10571 3996 Acked Write 9 Done Done Sent false
10572 3997 Acked Read 2 Done Done Sent false
10573 3998 Acked Write 7 Done Done Sent false
10574 3999 Acked Flush 0 Done Done Sent false
10575 4000 Acked Write 6 Done Done Sent false
10576 4001 Acked Read 7 Done Done Sent false
10577 4002 Acked Write 4 Done Done Sent false
10578 4003 Acked Flush 0 Done Done Sent false
10579 4004 Acked Write 8 Done Done Sent false
10580 4005 Acked Read 1 Done Done Sent false
10581 4006 Acked Write 5 Done Done Sent false
10582 4007 Acked Read 4 Done Done Sent false
10583 4008 Acked Flush 0 Done Done Sent false
10584 4009 Acked Write 6 Done Done Sent false
10585 4010 Acked Write 1 Done Done Sent false
10586 4011 Acked Read 8 Done Done Sent false
10587 4012 Acked Read 5 Done Done Sent false
10588 4013 Acked Read 9 Done Done Sent false
10589 4014 Acked Read 4 Done Done Sent false
10590 4015 Acked Read 9 Done Done Sent false
10591 4016 Acked Write 9 Done Done Sent false
10592 4017 Acked Read 4 Done Done Sent false
10593 4018 Acked Write 4 Done Done Sent false
10594 STATES DS:0 DS:1 DS:2 TOTAL
10595 Sent 0 0 301 301
10596 Done 301 301 0 602
10597 Skipped 0 0 0 0
10598 Error 0 0 0 0
10599 Last Flush: Some(JobId(4008)) Some(JobId(4008)) Some(JobId(3717))
10600 Downstairs last five completed: 3717 3716 3715 3714 3713
10601 Upstairs last five completed: 4018 4017 4016 4015 4014
10602 CLIENT: Up:0 ds:301 act:2
106032025-01-28T21:56:32.850ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106042025-01-28T21:56:33.524ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106052025-01-28T21:56:34.096ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106062025-01-28T21:56:34.096ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59020) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106072025-01-28T21:56:34.097ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106082025-01-28T21:56:34.098ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106092025-01-28T21:56:34.098ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106102025-01-28T21:56:34.098ZINFOcrucible: send last flush ID to this DS: Some(JobId(3717)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106112025-01-28T21:56:34.098ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106122025-01-28T21:56:34.098ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3717)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106132025-01-28T21:56:34.098ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3717)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
10614 ----------------------------------------------------------------
10615 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10616 Downstairs last five completed: 4019 4018 4017 4016 4015
10617 Upstairs last five completed: 4019 4018 4017 4016 4015
10618 CLIENT: Up:0 ds:0 act:3
10619 Replay: All jobs finished
10620 Main task has work to do, go find it
10621 got dsc Stop(2)
10622 stop 2
10623 Jan 28 21:56:36.352 INFO request completed, latency_us: 376, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 351a9835-3ad7-458c-aa97-ef874f03f69d, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0[8830] Got stop action so:false kr:false
10624 .1:9998
10625 Jan 28 21:56:36.352 INFO request completed, latency_us: 148, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8adae8a1-0456-4b37-9310-db69498e52e0, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
106262025-01-28T21:56:36.352ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106272025-01-28T21:56:36.352ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106282025-01-28T21:56:36.352ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
106292025-01-28T21:56:36.353ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
10630 [8830] Exited with: ExitStatus(unix_wait_status(9))
10631 [8830][2] reports Exit
10632 Jan 28 21:56:40.355 INFO request completed, latency_us: 221, response_code: 200, uri: /state/cid/2, method: GET, req_id: 02f2c640-36fd-4f39-9cf1-c5a4f1bf6f05, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
10633 001/300 Flush
10634 002/300 Read block 2902 len 40960
10635 003/300 Read block 2164 len 24576
10636 004/300 Write block 1281 len 20480 data: 2 2 2 3 3
10637 005/300 Flush
10638 006/300 Read block 2602 len 28672
10639 007/300 Read block 3994 len 36864
10640 008/300 Flush
10641 009/300 Read block 630 len 8192
10642 010/300 Write block 4812 len 24576 data: 2 2 2 2 2 2
10643 011/300 Read block 1121 len 12288
10644 012/300 Write block 2276 len 4096 data: 4
10645 013/300 Read block 1703 len 4096
10646 014/300 Read block 4725 len 32768
10647 015/300 Read block 2898 len 20480
10648 016/300 Read block 2264 len 28672
10649 017/300 Write block 1482 len 36864 data: 3 3 4 4 5 4 5 5 5
10650 018/300 Write block 1183 len 24576 data: 4 4 4 3 2 2
10651 019/300 Write block 278 len 32768 data: 2 2 2 2 2 2 2 2
10652 020/300 Read block 2259 len 32768
10653 021/300 Write block 349 len 12288 data: 5 5 5
10654 022/300 Write block 4679 len 32768 data: 4 4 4 4 4 5 5 4
10655 023/300 Flush
10656 024/300 Flush
10657 025/300 Read block 798 len 40960
10658 026/300 Write block 3337 len 8192 data: 2 2
10659 027/300 Read block 3465 len 16384
10660 028/300 Read block 548 len 8192
10661 029/300 Write block 2851 len 8192 data: 4 3
10662 030/300 Write block 78 len 16384 data: 5 5 5 5
10663 031/300 Read block 4799 len 4096
10664 032/300 Write block 3251 len 36864 data: 3 2 2 3 3 3 3 3 3
10665 033/300 Write block 607 len 16384 data: 4 4 3 3
10666 034/300 Write block 3101 len 24576 data: 3 3 3 3 3 4
10667 035/300 Write block 435 len 12288 data: 2 3 2
10668 036/300 Write block 3896 len 8192 data: 4 4
10669 037/300 Write block 1615 len 16384 data: 2 2 2 2
10670 038/300 Flush
10671 039/300 Write block 4998 len 4096 data: 2
10672 040/300 Read block 3344 len 40960
10673 041/300 Read block 899 len 8192
10674 042/300 Read block 1010 len 8192
10675 043/300 Write block 1052 len 32768 data: 7 7 7 6 5 6 5 6
10676 044/300 Write block 3163 len 8192 data: 3 3
10677 045/300 Write block 4753 len 12288 data: 3 3 2
10678 046/300 Read block 3204 len 40960
10679 047/300 Flush
10680 048/300 Read block 4742 len 28672
10681 049/300 Write block 918 len 28672 data: 6 4 2 2 2 2 2
10682 050/300 Read block 2568 len 36864
10683 051/300 Read block 971 len 16384
10684 052/300 Read block 4389 len 28672
10685 053/300 Read block 3554 len 36864
10686 054/300 Write block 2617 len 40960 data: 6 7 6 6 5 3 3 2 2 2
10687 055/300 Read block 4080 len 36864
10688 056/300 Write block 3499 len 12288 data: 2 2 2
10689 057/300 Write block 3948 len 20480 data: 4 4 3 2 2
10690 058/300 Write block 3335 len 36864 data: 2 2 3 3 2 2 2 2 2
10691 059/300 Write block 475 len 12288 data: 5 5 5
10692 060/300 Read block 2414 len 28672
10693 061/300 Read block 4796 len 28672
10694 062/300 Flush
10695 063/300 Read block 3065 len 16384
10696 064/300 Read block 757 len 28672
10697 065/300 Write block 4158 len 40960 data: 3 3 3 3 3 4 5 5 5 5
10698 066/300 Read block 3944 len 8192
10699 067/300 Write block 3244 len 16384 data: 3 3 3 2
10700 068/300 Write block 1661 len 20480 data: 4 4 4 5 4
10701 069/300 Read block 1023 len 28672
10702 070/300 Write block 972 len 8192 data: 2 2
10703 071/300 Read block 2070 len 36864
10704 072/300 Write block 758 len 28672 data: 3 3 3 3 3 3 4
10705 073/300 Read block 1447 len 8192
10706 074/300 Write block 2752 len 36864 data: 5 4 4 4 4 4 4 4 4
10707 075/300 Write block 4554 len 40960 data: 3 4 3 3 3 3 3 3 4 4
10708 076/300 Read block 460 len 20480
10709 077/300 Flush
10710 078/300 Write block 2859 len 16384 data: 4 4 4 3
10711 079/300 Read block 3954 len 40960
10712 080/300 Write block 2926 len 16384 data: 2 4 5 5
10713 081/300 Read block 297 len 24576
10714 082/300 Read block 220 len 24576
10715 083/300 Read block 4855 len 12288
10716 084/300 Read block 401 len 40960
10717 085/300 Read block 2737 len 8192
10718 086/300 Read block 4612 len 36864
10719 087/300 Read block 3056 len 12288
10720 088/300 Read block 4521 len 16384
10721 089/300 Write block 491 len 4096 data: 3
10722 090/300 Read block 4980 len 24576
10723 091/300 Write block 4671 len 8192 data: 3 3
10724 092/300 Flush
10725 093/300 Write block 4421 len 36864 data: 3 3 3 3 3 5 4 4 3
10726 094/300 Read block 990 len 24576
10727 095/300 Flush
10728 096/300 Read block 1713 len 4096
10729 097/300 Read block 2385 len 28672
10730 098/300 Read block 4590 len 16384
10731 099/300 Read block 3108 len 32768
10732 100/300 Read block 3431 len 40960
10733 101/300 Write block 886 len 16384 data: 5 5 4 5
10734 102/300 Read block 2307 len 4096
10735 103/300 Write block 2769 len 32768 data: 2 2 2 2 3 3 3 4
10736 104/300 Read block 87 len 8192
10737 105/300 Read block 3661 len 12288
10738 106/300 Read block 2276 len 28672
10739 107/300 Read block 3045 len 12288
10740 108/300 Flush
10741 109/300 Read block 3264 len 32768
10742 110/300 Read block 4210 len 28672
10743 111/300 Write block 2154 len 4096 data: 5
10744 112/300 Read block 3894 len 24576
10745 113/300 Flush
10746 114/300 Read block 3361 len 32768
10747 115/300 Flush
10748 116/300 Write block 1507 len 12288 data: 4 4 2
10749 117/300 Write block 2024 len 20480 data: 2 2 2 2 2
10750 118/300 Read block 3354 len 4096
10751 119/300 Read block 2374 len 12288
10752 120/300 Read block 3422 len 28672
10753 121/300 Write block 4932 len 20480 data: 2 3 3 3 3
10754 122/300 Write block 4637 len 16384 data: 2 2 3 3
10755 123/300 Read block 3576 len 16384
10756 124/300 Write block 2710 len 4096 data: 3
10757 125/300 Write block 2393 len 32768 data: 3 4 4 5 5 5 3 3
10758 126/300 Read block 4028 len 40960
10759 127/300 Flush
10760 128/300 Read block 4528 len 36864
10761 129/300 Flush
10762 130/300 Read block 2803 len 8192
10763 131/300 Write block 3862 len 28672 data: 2 2 3 3 3 2 2
10764 132/300 Read block 1505 len 8192
10765 133/300 Read block 940 len 4096
10766 134/300 Read block 3799 len 20480
10767 135/300 Read block 228 len 12288
10768 136/300 Read block 2579 len 40960
10769 137/300 Read block 4467 len 40960
10770 138/300 Read block 1149 len 4096
10771 139/300 Write block 928 len 4096 data: 3
10772 140/300 Read block 4743 len 4096
10773 141/300 Read block 1832 len 40960
10774 142/300 Flush
10775 143/300 Write block 1487 len 28672 data: 5 6 6 6 4 2 2
10776 144/300 Read block 1906 len 16384
10777 145/300 Write block 562 len 36864 data: 2 2 2 2 2 3 3 2 2
10778 146/300 Write block 4852 len 40960 data: 3 4 3 2 3 3 3 3 4 4
10779 147/300 Read block 2585 len 4096
10780 148/300 Write block 3039 len 12288 data: 2 3 3
10781 149/300 Read block 1328 len 40960
10782 150/300 Read block 2696 len 28672
10783 151/300 Write block 397 len 20480 data: 5 5 5 5 4
10784 152/300 Write block 2081 len 8192 data: 4 4
10785 153/300 Read block 1845 len 8192
10786 154/300 Read block 443 len 40960
10787 155/300 Read block 4667 len 4096
10788 156/300 Read block 1513 len 24576
10789 157/300 Flush
10790 158/300 Write block 1404 len 40960 data: 4 4 2 3 3 4 5 4 4 3
10791 159/300 Write block 1844 len 28672 data: 3 3 3 3 3 3 4
10792 160/300 Read block 1630 len 8192
10793 161/300 Write block 3793 len 20480 data: 3 3 2 2 2
10794 162/300 Flush
10795 163/300 Write block 4805 len 12288 data: 3 3 3
10796 164/300 Read block 2228 len 12288
10797 165/300 Write block 4523 len 24576 data: 4 5 4 4 4 2
10798 166/300 Read block 4940 len 8192
10799 167/300 Read block 1647 len 24576
10800 168/300 Write block 4778 len 20480 data: 3 3 3 3 2
10801 169/300 Read block 1715 len 24576
10802 170/300 Write block 3133 len 32768 data: 3 3 3 3 3 3 2 3
10803 171/300 Write block 1016 len 36864 data: 3 2 2 2 2 3 3 3 3
10804 172/300 Write block 1859 len 12288 data: 2 2 3
10805 173/300 Write block 4944 len 4096 data: 3
10806 174/300 Read block 1992 len 36864
10807 175/300 Read block 4307 len 40960
10808 176/300 Read block 3950 len 12288
10809 177/300 Read block 2226 len 20480
10810 178/300 Write block 4111 len 24576 data: 3 3 3 3 6 5
10811 179/300 Write block 3356 len 4096 data: 4
10812 180/300 Write block 3263 len 8192 data: 3 3
10813 181/300 Write block 66 len 16384 data: 3 3 3 4
10814 182/300 Read block 743 len 28672
10815 183/300 Write block 509 len 4096 data: 2
10816 184/300 Read block 3841 len 40960
10817 185/300 Read block 4114 len 40960
10818 186/300 Write block 4550 len 16384 data: 3 2 3 3
10819 187/300 Write block 3042 len 36864 data: 4 3 4 4 4 4 5 4 6
10820 188/300 Write block 3250 len 40960 data: 3 4 3 3 4 4 4 4 4 4
10821 189/300 Read block 1939 len 28672
10822 190/300 Read block 315 len 20480
10823 191/300 Flush
10824 192/300 Write block 1628 len 36864 data: 3 4 4 5 4 4 3 3 4
10825 193/300 Read block 3118 len 16384
10826 194/300 Read block 4524 len 20480
10827 195/300 Read block 4298 len 40960
10828 196/300 Write block 251 len 8192 data: 2 2
10829 197/300 Read block 2975 len 36864
10830 198/300 Write block 1867 len 24576 data: 2 2 2 2 2 2
10831 199/300 Write block 2339 len 12288 data: 4 4 2
10832 200/300 Read block 3258 len 40960
10833 201/300 Write block 2494 len 36864 data: 2 3 3 3 3 3 3 3 3
10834 202/300 Flush
10835 203/300 Read block 2160 len 20480
10836 204/300 Read block 2815 len 4096
10837 205/300 Read block 4380 len 32768
10838 206/300 Write block 1716 len 12288 data: 3 4 4
10839 207/300 Read block 2393 len 28672
10840 208/300 Read block 3812 len 8192
10841 209/300 Read block 206 len 28672
10842 210/300 Read block 866 len 12288
10843 211/300 Flush
10844 212/300 Flush
10845 213/300 Read block 733 len 28672
10846 214/300 Read block 2839 len 28672
10847 215/300 Write block 4599 len 36864 data: 2 2 2 2 3 3 3 3 4
10848 216/300 Read block 347 len 16384
10849 217/300 Flush
10850 218/300 Flush
10851 219/300 Write block 240 len 12288 data: 3 3 3
10852 220/300 Read block 4304 len 12288
10853 221/300 Write block 4980 len 28672 data: 2 4 4 4 5 5 6
10854 222/300 Read block 3142 len 20480
10855 223/300 Write block 4449 len 36864 data: 3 3 3 3 3 3 2 2 3
10856 224/300 Write block 900 len 36864 data: 4 4 5 5 5 5 5 5 4
10857 225/300 Write block 2218 len 16384 data: 3 2 2 2
10858 226/300 Write block 3330 len 4096 data: 4
10859 227/300 Read block 2284 len 24576
10860 228/300 Read block 454 len 4096
10861 229/300 Flush
10862 230/300 Read block 4190 len 28672
10863 231/300 Read block 4711 len 16384
10864 232/300 Write block 4849 len 36864 data: 3 4 4 4 5 4 3 4 4
10865 233/300 Write block 1183 len 28672 data: 5 5 5 4 3 3 2
10866 234/300 Write block 1712 len 40960 data: 4 4 4 3 4 5 5 5 4 4
10867 235/300 Read block 956 len 32768
10868 236/300 Read block 4536 len 12288
10869 237/300 Read block 1986 len 28672
10870 238/300 Read block 1486 len 8192
10871 239/300 Write block 2429 len 40960 data: 6 6 6 5 5 4 3 3 2 2
10872 240/300 Read block 2846 len 32768
10873 241/300 Read block 3205 len 8192
10874 242/300 Write block 223 len 12288 data: 3 3 3
10875 243/300 Write block 1770 len 36864 data: 2 2 2 2 3 4 3 3 3
10876 244/300 Read block 3241 len 32768
10877 245/300 Write block 2633 len 40960 data: 4 3 4 4 4 3 3 3 4 4
10878 246/300 Read block 1253 len 16384
10879 247/300 Read block 4928 len 16384
10880 248/300 Flush
10881 249/300 Read block 36 len 8192
10882 250/300 Read block 2607 len 8192
10883 251/300 Write block 1268 len 32768 data: 3 3 2 2 2 2 2 2
10884 252/300 Write block 62 len 12288 data: 4 3 4
10885 253/300 Write block 3220 len 12288 data: 4 3 3
10886 254/300 Read block 2439 len 20480
10887 255/300 Write block 4490 len 4096 data: 5
10888 256/300 Flush
10889 257/300 Write block 4270 len 24576 data: 3 4 3 2 3 3
10890 258/300 Flush
10891 259/300 Read block 4081 len 32768
10892 260/300 Read block 3918 len 36864
10893 261/300 Read block 142 len 8192
10894 262/300 Read block 2621 len 20480
10895 263/300 Read block 1090 len 12288
10896 264/300 Write block 4275 len 40960 data: 4 3 3 2 2 2 2 2 2 2
10897 265/300 Read block 3480 len 4096
10898 266/300 Read block 196 len 16384
10899 267/300 Read block 1156 len 28672
10900 268/300 Write block 3055 len 36864 data: 5 4 4 4 7 6 6 4 4
10901 269/300 Write block 3044 len 24576 data: 5 5 5 5 6 5
10902 270/300 Write block 340 len 28672 data: 3 4 4 3 3 4 4
10903 271/300 Flush
10904 272/300 Write block 4500 len 28672 data: 2 2 2 2 2 2 2
10905 273/300 Read block 3777 len 28672
10906 274/300 Write block 4127 len 16384 data: 2 2 2 2
10907 275/300 Read block 3863 len 8192
10908 276/300 Read block 1330 len 16384
10909 277/300 Write block 2571 len 4096 data: 2
10910 278/300 Read block 592 len 28672
10911 279/300 Read block 3168 len 28672
10912 280/300 Flush
10913 281/300 Read block 3739 len 4096
10914 282/300 Write block 235 len 24576 data: 5 5 3 4 3 4
10915 283/300 Read block 1145 len 20480
10916 284/300 Read block 4696 len 20480
10917 285/300 Read block 1064 len 28672
10918 286/300 Read block 1641 len 36864
10919 287/300 Read block 1947 len 20480
10920 288/300 Read block 3744 len 36864
10921 289/300 Write block 3294 len 36864 data: 4 4 3 3 4 4 3 4 4
10922 290/300 Write block 241 len 12288 data: 4 4 4
10923 291/300 Flush
10924 292/300 Write block 4837 len 20480 data: 3 2 2 3 4
10925 293/300 Read block 1530 len 8192
10926 294/300 Read block 1495 len 4096
10927 295/300 Write block 3987 len 16384 data: 2 3 3 3
10928 296/300 Read block 4151 len 36864
10929 297/300 Read block 2776 len 24576
10930 298/300 Read block 3206 len 12288
10931 299/300 Read block 3469 len 24576
10932 300/300 Write block 2861 len 4096 data: 5
10933 301/300 Write block 4792 len 32768 data: 3 3 4 4 3 4 4 5
10934 Main task has work to do, go find it
10935 got dsc Start(2)
10936 start 2
10937 Jan 28 21:56:40.600 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/2, method: POST, req_id: 363b4656-7f22-4b38-aee2-6abfb9a06894, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
10938 Replay: started 2, returned:Ok(())
10939 ----------------------------------------------------------------
10940 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10941 States: Active Active Offline
10942 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10943 4020 Acked Flush 0 Done Done Sent false
10944 4021 Acked Read 10 Done Done Sent false
10945 4022 Acked Read 6 Done Done Sent false
10946 4023 Acked Write 5 Done Done Sent false
10947 4024 Acked Flush 0 Done Done Sent false
10948 4025 Acked Read 7 Done Done Sent false
10949 4026 Acked Read 9 Done Done Sent false
10950 4027 Acked Flush 0 Done Done Sent false
10951 4028 Acked Read 2 Done Done Sent false
10952 4029 Acked Write 6 Done Done Sent false
10953 4030 Acked Read 3 Done Done Sent false
10954 4031 Acked Write 1 Done Done Sent false
10955 4032 Acked Read 1 Done Done Sent false
10956 4033 Acked Read 8 Done Done Sent false
10957 4034 Acked Read 5 Done Done Sent false
10958 4035 Acked Read 7 Done Done Sent false
10959 4036 Acked Write 9 Done Done Sent false
10960 4037 Acked Write 6 Done Done Sent false
10961 4038 Acked Write 8 Done Done Sent false
10962 4039 Acked Read 8 Done Done Sent false
10963 4040 Acked Write 3 Done Done Sent false
10964 4041 Acked Write 8 Done Done Sent false
10965 4042 Acked Flush 0 Done Done Sent false
10966 4043 Acked Flush 0 Done Done Sent false
10967 4044 Acked Read 10 Done Done Sent false
10968 4045 Acked Write 2 Done Done Sent false
10969 4046 Acked Read 4 Done Done Sent false
10970 4047 Acked Read 2 Done Done Sent false
10971 4048 Acked Write 2 Done Done Sent false
10972 4049 Acked Write 4 Done Done Sent false
10973 4050 Acked Read 1 Done Done Sent false
10974 4051 Acked Write 9 Done Done Sent false
10975 4052 Acked Write 4 Done Done Sent false
10976 4053 Acked Write 6 Done Done Sent false
10977 4054 Acked Write 3 Done Done Sent false
10978 4055 Acked Write 2 Done Done Sent false
10979 4056 Acked Write 4 Done Done Sent false
10980 4057 Acked Flush 0 Done Done Sent false
10981 4058 Acked Write 1 Done Done Sent false
10982 4059 Acked Read 10 Done Done Sent false
10983 4060 Acked Read 2 Done Done Sent false
10984 4061 Acked Read 2 Done Done Sent false
10985 4062 Acked Write 8 Done Done Sent false
10986 4063 Acked Write 2 Done Done Sent false
10987 4064 Acked Write 3 Done Done Sent false
10988 4065 Acked Read 10 Done Done Sent false
10989 4066 Acked Flush 0 Done Done Sent false
10990 4067 Acked Read 7 Done Done Sent false
10991 4068 Acked Write 7 Done Done Sent false
10992 4069 Acked Read 9 Done Done Sent false
10993 4070 Acked Read 4 Done Done Sent false
10994 4071 Acked Read 7 Done Done Sent false
10995 4072 Acked Read 9 Done Done Sent false
10996 4073 Acked Write 10 Done Done Sent false
10997 4074 Acked Read 9 Done Done Sent false
10998 4075 Acked Write 3 Done Done Sent false
10999 4076 Acked Write 5 Done Done Sent false
11000 4077 Acked Write 9 Done Done Sent false
11001 4078 Acked Write 3 Done Done Sent false
11002 4079 Acked Read 7 Done Done Sent false
11003 4080 Acked Read 7 Done Done Sent false
11004 4081 Acked Flush 0 Done Done Sent false
11005 4082 Acked Read 4 Done Done Sent false
11006 4083 Acked Read 7 Done Done Sent false
11007 4084 Acked Write 10 Done Done Sent false
11008 4085 Acked Read 2 Done Done Sent false
11009 4086 Acked Write 4 Done Done Sent false
11010 4087 Acked Write 5 Done Done Sent false
11011 4088 Acked Read 7 Done Done Sent false
11012 4089 Acked Write 2 Done Done Sent false
11013 4090 Acked Read 9 Done Done Sent false
11014 4091 Acked Write 7 Done Done Sent false
11015 4092 Acked Read 2 Done Done Sent false
11016 4093 Acked Write 9 Done Done Sent false
11017 4094 Acked Write 10 Done Done Sent false
11018 4095 Acked Read 5 Done Done Sent false
11019 4096 Acked Flush 0 Done Done Sent false
11020 4097 Acked Write 4 Done Done Sent false
11021 4098 Acked Read 10 Done Done Sent false
11022 4099 Acked Write 4 Done Done Sent false
11023 4100 Acked Read 6 Done Done Sent false
11024 4101 Acked Read 6 Done Done Sent false
11025 4102 Acked Read 3 Done Done Sent false
11026 4103 Acked Read 10 Done Done Sent false
11027 4104 Acked Read 2 Done Done Sent false
11028 4105 Acked Read 9 Done Done Sent false
11029 4106 Acked Read 3 Done Done Sent false
11030 4107 Acked Read 4 Done Done Sent false
11031 4108 Acked Write 1 Done Done Sent false
11032 4109 Acked Read 6 Done Done Sent false
11033 4110 Acked Write 2 Done Done Sent false
11034 4111 Acked Flush 0 Done Done Sent false
11035 4112 Acked Write 9 Done Done Sent false
11036 4113 Acked Read 6 Done Done Sent false
11037 4114 Acked Flush 0 Done Done Sent false
11038 4115 Acked Read 1 Done Done Sent false
11039 4116 Acked Read 7 Done Done Sent false
11040 4117 Acked Read 4 Done Done Sent false
11041 4118 Acked Read 8 Done Done Sent false
11042 4119 Acked Read 10 Done Done Sent false
11043 4120 Acked Write 4 Done Done Sent false
11044 4121 Acked Read 1 Done Done Sent false
11045 4122 Acked Write 8 Done Done Sent false
11046 4123 Acked Read 2 Done Done Sent false
11047 4124 Acked Read 3 Done Done Sent false
11048 4125 Acked Read 7 Done Done Sent false
11049 4126 Acked Read 3 Done Done Sent false
11050 4127 Acked Flush 0 Done Done Sent false
11051 4128 Acked Read 8 Done Done Sent false
11052 4129 Acked Read 7 Done Done Sent false
11053 4130 Acked Write 1 Done Done Sent false
11054 4131 Acked Read 6 Done Done Sent false
11055 4132 Acked Flush 0 Done Done Sent false
11056 4133 Acked Read 8 Done Done Sent false
11057 4134 Acked Flush 0 Done Done Sent false
11058 4135 Acked Write 3 Done Done Sent false
11059 4136 Acked Write 5 Done Done Sent false
11060 4137 Acked Read 1 Done Done Sent false
11061 4138 Acked Read 3 Done Done Sent false
11062 4139 Acked Read 7 Done Done Sent false
11063 4140 Acked Write 5 Done Done Sent false
11064 4141 Acked Write 4 Done Done Sent false
11065 4142 Acked Read 4 Done Done Sent false
11066 4143 Acked Write 1 Done Done Sent false
11067 4144 Acked Write 8 Done Done Sent false
11068 4145 Acked Read 10 Done Done Sent false
11069 4146 Acked Flush 0 Done Done Sent false
11070 4147 Acked Read 9 Done Done Sent false
11071 4148 Acked Flush 0 Done Done Sent false
11072 4149 Acked Read 2 Done Done Sent false
11073 4150 Acked Write 7 Done Done Sent false
11074 4151 Acked Read 2 Done Done Sent false
11075 4152 Acked Read 1 Done Done Sent false
11076 4153 Acked Read 5 Done Done Sent false
11077 4154 Acked Read 3 Done Done Sent false
11078 4155 Acked Read 10 Done Done Sent false
11079 4156 Acked Read 10 Done Done Sent false
11080 4157 Acked Read 1 Done Done Sent false
11081 4158 Acked Write 1 Done Done Sent false
11082 4159 Acked Read 1 Done Done Sent false
11083 4160 Acked Read 10 Done Done Sent false
11084 4161 Acked Flush 0 Done Done Sent false
11085 4162 Acked Write 7 Done Done Sent false
11086 4163 Acked Read 4 Done Done Sent false
11087 4164 Acked Write 9 Done Done Sent false
11088 4165 Acked Write 10 Done Done Sent false
11089 4166 Acked Read 1 Done Done Sent false
11090 4167 Acked Write 3 Done Done Sent false
11091 4168 Acked Read 10 Done Done Sent false
11092 4169 Acked Read 7 Done Done Sent false
11093 4170 Acked Write 5 Done Done Sent false
11094 4171 Acked Write 2 Done Done Sent false
11095 4172 Acked Read 2 Done Done Sent false
11096 4173 Acked Read 10 Done Done Sent false
11097 4174 Acked Read 1 Done Done Sent false
11098 4175 Acked Read 6 Done Done Sent false
11099 4176 Acked Flush 0 Done Done Sent false
11100 4177 Acked Write 10 Done Done Sent false
11101 4178 Acked Write 7 Done Done Sent false
11102 4179 Acked Read 2 Done Done Sent false
11103 4180 Acked Write 5 Done Done Sent false
11104 4181 Acked Flush 0 Done Done Sent false
11105 4182 Acked Write 3 Done Done Sent false
11106 4183 Acked Read 3 Done Done Sent false
11107 4184 Acked Write 6 Done Done Sent false
11108 4185 Acked Read 2 Done Done Sent false
11109 4186 Acked Read 6 Done Done Sent false
11110 4187 Acked Write 5 Done Done Sent false
11111 4188 Acked Read 6 Done Done Sent false
11112 4189 Acked Write 8 Done Done Sent false
11113 4190 Acked Write 9 Done Done Sent false
11114 4191 Acked Write 3 Done Done Sent false
11115 4192 Acked Write 1 Done Done Sent false
11116 4193 Acked Read 9 Done Done Sent false
11117 4194 Acked Read 10 Done Done Sent false
11118 4195 Acked Read 3 Done Done Sent false
11119 4196 Acked Read 5 Done Done Sent false
11120 4197 Acked Write 6 Done Done Sent false
11121 4198 Acked Write 1 Done Done Sent false
11122 4199 Acked Write 2 Done Done Sent false
11123 4200 Acked Write 4 Done Done Sent false
11124 4201 Acked Read 7 Done Done Sent false
11125 4202 Acked Write 1 Done Done Sent false
11126 4203 Acked Read 10 Done Done Sent false
11127 4204 Acked Read 10 Done Done Sent false
11128 4205 Acked Write 4 Done Done Sent false
11129 4206 Acked Write 9 Done Done Sent false
11130 4207 Acked Write 10 Done Done Sent false
11131 4208 Acked Read 7 Done Done Sent false
11132 4209 Acked Read 5 Done Done Sent false
11133 4210 Acked Flush 0 Done Done Sent false
11134 4211 Acked Write 9 Done Done Sent false
11135 4212 Acked Read 4 Done Done Sent false
11136 4213 Acked Read 5 Done Done Sent false
11137 4214 Acked Read 10 Done Done Sent false
11138 4215 Acked Write 2 Done Done Sent false
11139 4216 Acked Read 9 Done Done Sent false
11140 4217 Acked Write 6 Done Done Sent false
11141 4218 Acked Write 3 Done Done Sent false
11142 4219 Acked Read 10 Done Done Sent false
11143 4220 Acked Write 9 Done Done Sent false
11144 4221 Acked Flush 0 Done Done Sent false
11145 4222 Acked Read 5 Done Done Sent false
11146 4223 Acked Read 1 Done Done Sent false
11147 4224 Acked Read 8 Done Done Sent false
11148 4225 Acked Write 3 Done Done Sent false
11149 4226 Acked Read 7 Done Done Sent false
11150 4227 Acked Read 2 Done Done Sent false
11151 4228 Acked Read 7 Done Done Sent false
11152 4229 Acked Read 3 Done Done Sent false
11153 4230 Acked Flush 0 Done Done Sent false
11154 4231 Acked Flush 0 Done Done Sent false
11155 4232 Acked Read 7 Done Done Sent false
11156 4233 Acked Read 7 Done Done Sent false
11157 4234 Acked Write 9 Done Done Sent false
11158 4235 Acked Read 4 Done Done Sent false
11159 4236 Acked Flush 0 Done Done Sent false
11160 4237 Acked Flush 0 Done Done Sent false
11161 4238 Acked Write 3 Done Done Sent false
11162 4239 Acked Read 3 Done Done Sent false
11163 4240 Acked Write 7 Done Done Sent false
11164 4241 Acked Read 5 Done Done Sent false
11165 4242 Acked Write 9 Done Done Sent false
11166 4243 Acked Write 9 Done Done Sent false
11167 4244 Acked Write 4 Done Done Sent false
11168 4245 Acked Write 1 Done Done Sent false
11169 4246 Acked Read 6 Done Done Sent false
11170 4247 Acked Read 1 Done Done Sent false
11171 4248 Acked Flush 0 Done Done Sent false
11172 4249 Acked Read 7 Done Done Sent false
11173 4250 Acked Read 4 Done Done Sent false
11174 4251 Acked Write 9 Done Done Sent false
11175 4252 Acked Write 7 Done Done Sent false
11176 4253 Acked Write 10 Done Done Sent false
11177 4254 Acked Read 8 Done Done Sent false
11178 4255 Acked Read 3 Done Done Sent false
11179 4256 Acked Read 7 Done Done Sent false
11180 4257 Acked Read 2 Done Done Sent false
11181 4258 Acked Write 10 Done Done Sent false
11182 4259 Acked Read 8 Done Done Sent false
11183 4260 Acked Read 2 Done Done Sent false
11184 4261 Acked Write 3 Done Done Sent false
11185 4262 Acked Write 9 Done Done Sent false
11186 4263 Acked Read 8 Done Done Sent false
11187 4264 Acked Write 10 Done Done Sent false
11188 4265 Acked Read 4 Done Done Sent false
11189 4266 Acked Read 4 Done Done Sent false
11190 4267 Acked Flush 0 Done Done Sent false
11191 4268 Acked Read 2 Done Done Sent false
11192 4269 Acked Read 2 Done Done Sent false
11193 4270 Acked Write 8 Done Done Sent false
11194 4271 Acked Write 3 Done Done Sent false
11195 4272 Acked Write 3 Done Done Sent false
11196 4273 Acked Read 5 Done Done Sent false
11197 4274 Acked Write 1 Done Done Sent false
11198 4275 Acked Flush 0 Done Done Sent false
11199 4276 Acked Write 6 Done Done Sent false
11200 4277 Acked Flush 0 Done Done Sent false
11201 4278 Acked Read 8 Done Done Sent false
11202 4279 Acked Read 9 Done Done Sent false
11203 4280 Acked Read 2 Done Done Sent false
11204 4281 Acked Read 5 Done Done Sent false
11205 4282 Acked Read 3 Done Done Sent false
11206 4283 Acked Write 10 Done Done Sent false
11207 4284 Acked Read 1 Done Done Sent false
11208 4285 Acked Read 4 Done Done Sent false
11209 4286 Acked Read 7 Done Done Sent false
11210 4287 Acked Write 9 Done Done Sent false
11211 4288 Acked Write 6 Done Done Sent false
11212 4289 Acked Write 7 Done Done Sent false
11213 4290 Acked Flush 0 Done Done Sent false
11214 4291 Acked Write 7 Done Done Sent false
11215 4292 Acked Read 7 Done Done Sent false
11216 4293 Acked Write 4 Done Done Sent false
11217 4294 Acked Read 2 Done Done Sent false
11218 4295 Acked Read 4 Done Done Sent false
11219 4296 Acked Write 1 Done Done Sent false
11220 4297 Acked Read 7 Done Done Sent false
11221 4298 Acked Read 7 Done Done Sent false
11222 4299 Acked Flush 0 Done Done Sent false
11223 4300 Acked Read 1 Done Done Sent false
11224 4301 Acked Write 6 Done Done Sent false
11225 4302 Acked Read 5 Done Done Sent false
11226 4303 Acked Read 5 Done Done Sent false
11227 4304 Acked Read 7 Done Done Sent false
11228 4305 Acked Read 9 Done Done Sent false
11229 4306 Acked Read 5 Done Done Sent false
11230 4307 Acked Read 9 Done Done Sent false
11231 4308 Acked Write 9 Done Done Sent false
11232 4309 Acked Write 3 Done Done Sent false
11233 4310 Acked Flush 0 Done Done Sent false
11234 4311 Acked Write 5 Done Done Sent false
11235 4312 Acked Read 2 Done Done Sent false
11236 4313 Acked Read 1 Done Done Sent false
11237 4314 Acked Write 4 Done Done Sent false
11238 4315 Acked Read 9 Done Done Sent false
11239 4316 Acked Read 6 Done Done Sent false
11240 4317 Acked Read 3 Done Done Sent false
11241 4318 Acked Read 6 Done Done Sent false
11242 4319 Acked Write 1 Done Done Sent false
11243 4320 Acked Write 8 Sent Sent Sent false
11244 STATES DS:0 DS:1 DS:2 TOTAL
11245 Sent 1 1 301 303
11246 Done 300 300 0 600
11247 Skipped 0 0 0 0
11248 Error 0 0 0 0
11249 Last Flush: Some(JobId(4310)) Some(JobId(4310)) Some(JobId(4019))
11250 Downstairs last five completed: 4019 4018 4017 4016 4015
11251 Upstairs last five completed: 4320 4319 4318 4317 4316
11252 CLIENT: Up:0 ds:301 act:2
112532025-01-28T21:56:41.537ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
112542025-01-28T21:56:42.538ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
112552025-01-28T21:56:43.540ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
112562025-01-28T21:56:44.542ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11257 ----------------------------------------------------------------
11258 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11259 States: Active Active Offline
11260 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11261 4020 Acked Flush 0 Done Done Sent false
11262 4021 Acked Read 10 Done Done Sent false
11263 4022 Acked Read 6 Done Done Sent false
11264 4023 Acked Write 5 Done Done Sent false
11265 4024 Acked Flush 0 Done Done Sent false
11266 4025 Acked Read 7 Done Done Sent false
11267 4026 Acked Read 9 Done Done Sent false
11268 4027 Acked Flush 0 Done Done Sent false
11269 4028 Acked Read 2 Done Done Sent false
11270 4029 Acked Write 6 Done Done Sent false
11271 4030 Acked Read 3 Done Done Sent false
11272 4031 Acked Write 1 Done Done Sent false
11273 4032 Acked Read 1 Done Done Sent false
11274 4033 Acked Read 8 Done Done Sent false
11275 4034 Acked Read 5 Done Done Sent false
11276 4035 Acked Read 7 Done Done Sent false
11277 4036 Acked Write 9 Done Done Sent false
11278 4037 Acked Write 6 Done Done Sent false
11279 4038 Acked Write 8 Done Done Sent false
11280 4039 Acked Read 8 Done Done Sent false
11281 4040 Acked Write 3 Done Done Sent false
11282 4041 Acked Write 8 Done Done Sent false
11283 4042 Acked Flush 0 Done Done Sent false
11284 4043 Acked Flush 0 Done Done Sent false
11285 4044 Acked Read 10 Done Done Sent false
11286 4045 Acked Write 2 Done Done Sent false
11287 4046 Acked Read 4 Done Done Sent false
11288 4047 Acked Read 2 Done Done Sent false
11289 4048 Acked Write 2 Done Done Sent false
11290 4049 Acked Write 4 Done Done Sent false
11291 4050 Acked Read 1 Done Done Sent false
11292 4051 Acked Write 9 Done Done Sent false
11293 4052 Acked Write 4 Done Done Sent false
11294 4053 Acked Write 6 Done Done Sent false
11295 4054 Acked Write 3 Done Done Sent false
11296 4055 Acked Write 2 Done Done Sent false
11297 4056 Acked Write 4 Done Done Sent false
11298 4057 Acked Flush 0 Done Done Sent false
11299 4058 Acked Write 1 Done Done Sent false
11300 4059 Acked Read 10 Done Done Sent false
11301 4060 Acked Read 2 Done Done Sent false
11302 4061 Acked Read 2 Done Done Sent false
11303 4062 Acked Write 8 Done Done Sent false
11304 4063 Acked Write 2 Done Done Sent false
11305 4064 Acked Write 3 Done Done Sent false
11306 4065 Acked Read 10 Done Done Sent false
11307 4066 Acked Flush 0 Done Done Sent false
11308 4067 Acked Read 7 Done Done Sent false
11309 4068 Acked Write 7 Done Done Sent false
11310 4069 Acked Read 9 Done Done Sent false
11311 4070 Acked Read 4 Done Done Sent false
11312 4071 Acked Read 7 Done Done Sent false
11313 4072 Acked Read 9 Done Done Sent false
11314 4073 Acked Write 10 Done Done Sent false
11315 4074 Acked Read 9 Done Done Sent false
11316 4075 Acked Write 3 Done Done Sent false
11317 4076 Acked Write 5 Done Done Sent false
11318 4077 Acked Write 9 Done Done Sent false
11319 4078 Acked Write 3 Done Done Sent false
11320 4079 Acked Read 7 Done Done Sent false
11321 4080 Acked Read 7 Done Done Sent false
11322 4081 Acked Flush 0 Done Done Sent false
11323 4082 Acked Read 4 Done Done Sent false
11324 4083 Acked Read 7 Done Done Sent false
11325 4084 Acked Write 10 Done Done Sent false
11326 4085 Acked Read 2 Done Done Sent false
11327 4086 Acked Write 4 Done Done Sent false
11328 4087 Acked Write 5 Done Done Sent false
11329 4088 Acked Read 7 Done Done Sent false
11330 4089 Acked Write 2 Done Done Sent false
11331 4090 Acked Read 9 Done Done Sent false
11332 4091 Acked Write 7 Done Done Sent false
11333 4092 Acked Read 2 Done Done Sent false
11334 4093 Acked Write 9 Done Done Sent false
11335 4094 Acked Write 10 Done Done Sent false
11336 4095 Acked Read 5 Done Done Sent false
11337 4096 Acked Flush 0 Done Done Sent false
11338 4097 Acked Write 4 Done Done Sent false
11339 4098 Acked Read 10 Done Done Sent false
11340 4099 Acked Write 4 Done Done Sent false
11341 4100 Acked Read 6 Done Done Sent false
11342 4101 Acked Read 6 Done Done Sent false
11343 4102 Acked Read 3 Done Done Sent false
11344 4103 Acked Read 10 Done Done Sent false
11345 4104 Acked Read 2 Done Done Sent false
11346 4105 Acked Read 9 Done Done Sent false
11347 4106 Acked Read 3 Done Done Sent false
11348 4107 Acked Read 4 Done Done Sent false
11349 4108 Acked Write 1 Done Done Sent false
11350 4109 Acked Read 6 Done Done Sent false
11351 4110 Acked Write 2 Done Done Sent false
11352 4111 Acked Flush 0 Done Done Sent false
11353 4112 Acked Write 9 Done Done Sent false
11354 4113 Acked Read 6 Done Done Sent false
11355 4114 Acked Flush 0 Done Done Sent false
11356 4115 Acked Read 1 Done Done Sent false
11357 4116 Acked Read 7 Done Done Sent false
11358 4117 Acked Read 4 Done Done Sent false
11359 4118 Acked Read 8 Done Done Sent false
11360 4119 Acked Read 10 Done Done Sent false
11361 4120 Acked Write 4 Done Done Sent false
11362 4121 Acked Read 1 Done Done Sent false
11363 4122 Acked Write 8 Done Done Sent false
11364 4123 Acked Read 2 Done Done Sent false
11365 4124 Acked Read 3 Done Done Sent false
11366 4125 Acked Read 7 Done Done Sent false
11367 4126 Acked Read 3 Done Done Sent false
11368 4127 Acked Flush 0 Done Done Sent false
11369 4128 Acked Read 8 Done Done Sent false
11370 4129 Acked Read 7 Done Done Sent false
11371 4130 Acked Write 1 Done Done Sent false
11372 4131 Acked Read 6 Done Done Sent false
11373 4132 Acked Flush 0 Done Done Sent false
11374 4133 Acked Read 8 Done Done Sent false
11375 4134 Acked Flush 0 Done Done Sent false
11376 4135 Acked Write 3 Done Done Sent false
11377 4136 Acked Write 5 Done Done Sent false
11378 4137 Acked Read 1 Done Done Sent false
11379 4138 Acked Read 3 Done Done Sent false
11380 4139 Acked Read 7 Done Done Sent false
11381 4140 Acked Write 5 Done Done Sent false
11382 4141 Acked Write 4 Done Done Sent false
11383 4142 Acked Read 4 Done Done Sent false
11384 4143 Acked Write 1 Done Done Sent false
11385 4144 Acked Write 8 Done Done Sent false
11386 4145 Acked Read 10 Done Done Sent false
11387 4146 Acked Flush 0 Done Done Sent false
11388 4147 Acked Read 9 Done Done Sent false
11389 4148 Acked Flush 0 Done Done Sent false
11390 4149 Acked Read 2 Done Done Sent false
11391 4150 Acked Write 7 Done Done Sent false
11392 4151 Acked Read 2 Done Done Sent false
11393 4152 Acked Read 1 Done Done Sent false
11394 4153 Acked Read 5 Done Done Sent false
11395 4154 Acked Read 3 Done Done Sent false
11396 4155 Acked Read 10 Done Done Sent false
11397 4156 Acked Read 10 Done Done Sent false
11398 4157 Acked Read 1 Done Done Sent false
11399 4158 Acked Write 1 Done Done Sent false
11400 4159 Acked Read 1 Done Done Sent false
11401 4160 Acked Read 10 Done Done Sent false
11402 4161 Acked Flush 0 Done Done Sent false
11403 4162 Acked Write 7 Done Done Sent false
11404 4163 Acked Read 4 Done Done Sent false
11405 4164 Acked Write 9 Done Done Sent false
11406 4165 Acked Write 10 Done Done Sent false
11407 4166 Acked Read 1 Done Done Sent false
11408 4167 Acked Write 3 Done Done Sent false
11409 4168 Acked Read 10 Done Done Sent false
11410 4169 Acked Read 7 Done Done Sent false
11411 4170 Acked Write 5 Done Done Sent false
11412 4171 Acked Write 2 Done Done Sent false
11413 4172 Acked Read 2 Done Done Sent false
11414 4173 Acked Read 10 Done Done Sent false
11415 4174 Acked Read 1 Done Done Sent false
11416 4175 Acked Read 6 Done Done Sent false
11417 4176 Acked Flush 0 Done Done Sent false
11418 4177 Acked Write 10 Done Done Sent false
11419 4178 Acked Write 7 Done Done Sent false
11420 4179 Acked Read 2 Done Done Sent false
11421 4180 Acked Write 5 Done Done Sent false
11422 4181 Acked Flush 0 Done Done Sent false
11423 4182 Acked Write 3 Done Done Sent false
11424 4183 Acked Read 3 Done Done Sent false
11425 4184 Acked Write 6 Done Done Sent false
11426 4185 Acked Read 2 Done Done Sent false
11427 4186 Acked Read 6 Done Done Sent false
11428 4187 Acked Write 5 Done Done Sent false
11429 4188 Acked Read 6 Done Done Sent false
11430 4189 Acked Write 8 Done Done Sent false
11431 4190 Acked Write 9 Done Done Sent false
11432 4191 Acked Write 3 Done Done Sent false
11433 4192 Acked Write 1 Done Done Sent false
11434 4193 Acked Read 9 Done Done Sent false
11435 4194 Acked Read 10 Done Done Sent false
11436 4195 Acked Read 3 Done Done Sent false
11437 4196 Acked Read 5 Done Done Sent false
11438 4197 Acked Write 6 Done Done Sent false
11439 4198 Acked Write 1 Done Done Sent false
11440 4199 Acked Write 2 Done Done Sent false
11441 4200 Acked Write 4 Done Done Sent false
11442 4201 Acked Read 7 Done Done Sent false
11443 4202 Acked Write 1 Done Done Sent false
11444 4203 Acked Read 10 Done Done Sent false
11445 4204 Acked Read 10 Done Done Sent false
11446 4205 Acked Write 4 Done Done Sent false
11447 4206 Acked Write 9 Done Done Sent false
11448 4207 Acked Write 10 Done Done Sent false
11449 4208 Acked Read 7 Done Done Sent false
11450 4209 Acked Read 5 Done Done Sent false
11451 4210 Acked Flush 0 Done Done Sent false
11452 4211 Acked Write 9 Done Done Sent false
11453 4212 Acked Read 4 Done Done Sent false
11454 4213 Acked Read 5 Done Done Sent false
11455 4214 Acked Read 10 Done Done Sent false
11456 4215 Acked Write 2 Done Done Sent false
11457 4216 Acked Read 9 Done Done Sent false
11458 4217 Acked Write 6 Done Done Sent false
11459 4218 Acked Write 3 Done Done Sent false
11460 4219 Acked Read 10 Done Done Sent false
11461 4220 Acked Write 9 Done Done Sent false
11462 4221 Acked Flush 0 Done Done Sent false
11463 4222 Acked Read 5 Done Done Sent false
11464 4223 Acked Read 1 Done Done Sent false
11465 4224 Acked Read 8 Done Done Sent false
11466 4225 Acked Write 3 Done Done Sent false
11467 4226 Acked Read 7 Done Done Sent false
11468 4227 Acked Read 2 Done Done Sent false
11469 4228 Acked Read 7 Done Done Sent false
11470 4229 Acked Read 3 Done Done Sent false
11471 4230 Acked Flush 0 Done Done Sent false
11472 4231 Acked Flush 0 Done Done Sent false
11473 4232 Acked Read 7 Done Done Sent false
11474 4233 Acked Read 7 Done Done Sent false
11475 4234 Acked Write 9 Done Done Sent false
11476 4235 Acked Read 4 Done Done Sent false
11477 4236 Acked Flush 0 Done Done Sent false
11478 4237 Acked Flush 0 Done Done Sent false
11479 4238 Acked Write 3 Done Done Sent false
11480 4239 Acked Read 3 Done Done Sent false
11481 4240 Acked Write 7 Done Done Sent false
11482 4241 Acked Read 5 Done Done Sent false
11483 4242 Acked Write 9 Done Done Sent false
11484 4243 Acked Write 9 Done Done Sent false
11485 4244 Acked Write 4 Done Done Sent false
11486 4245 Acked Write 1 Done Done Sent false
11487 4246 Acked Read 6 Done Done Sent false
11488 4247 Acked Read 1 Done Done Sent false
11489 4248 Acked Flush 0 Done Done Sent false
11490 4249 Acked Read 7 Done Done Sent false
11491 4250 Acked Read 4 Done Done Sent false
11492 4251 Acked Write 9 Done Done Sent false
11493 4252 Acked Write 7 Done Done Sent false
11494 4253 Acked Write 10 Done Done Sent false
11495 4254 Acked Read 8 Done Done Sent false
11496 4255 Acked Read 3 Done Done Sent false
11497 4256 Acked Read 7 Done Done Sent false
11498 4257 Acked Read 2 Done Done Sent false
11499 4258 Acked Write 10 Done Done Sent false
11500 4259 Acked Read 8 Done Done Sent false
11501 4260 Acked Read 2 Done Done Sent false
11502 4261 Acked Write 3 Done Done Sent false
11503 4262 Acked Write 9 Done Done Sent false
11504 4263 Acked Read 8 Done Done Sent false
11505 4264 Acked Write 10 Done Done Sent false
11506 4265 Acked Read 4 Done Done Sent false
11507 4266 Acked Read 4 Done Done Sent false
11508 4267 Acked Flush 0 Done Done Sent false
11509 4268 Acked Read 2 Done Done Sent false
11510 4269 Acked Read 2 Done Done Sent false
11511 4270 Acked Write 8 Done Done Sent false
11512 4271 Acked Write 3 Done Done Sent false
11513 4272 Acked Write 3 Done Done Sent false
11514 4273 Acked Read 5 Done Done Sent false
11515 4274 Acked Write 1 Done Done Sent false
11516 4275 Acked Flush 0 Done Done Sent false
11517 4276 Acked Write 6 Done Done Sent false
11518 4277 Acked Flush 0 Done Done Sent false
11519 4278 Acked Read 8 Done Done Sent false
11520 4279 Acked Read 9 Done Done Sent false
11521 4280 Acked Read 2 Done Done Sent false
11522 4281 Acked Read 5 Done Done Sent false
11523 4282 Acked Read 3 Done Done Sent false
11524 4283 Acked Write 10 Done Done Sent false
11525 4284 Acked Read 1 Done Done Sent false
11526 4285 Acked Read 4 Done Done Sent false
11527 4286 Acked Read 7 Done Done Sent false
11528 4287 Acked Write 9 Done Done Sent false
11529 4288 Acked Write 6 Done Done Sent false
11530 4289 Acked Write 7 Done Done Sent false
11531 4290 Acked Flush 0 Done Done Sent false
11532 4291 Acked Write 7 Done Done Sent false
11533 4292 Acked Read 7 Done Done Sent false
11534 4293 Acked Write 4 Done Done Sent false
11535 4294 Acked Read 2 Done Done Sent false
11536 4295 Acked Read 4 Done Done Sent false
11537 4296 Acked Write 1 Done Done Sent false
11538 4297 Acked Read 7 Done Done Sent false
11539 4298 Acked Read 7 Done Done Sent false
11540 4299 Acked Flush 0 Done Done Sent false
11541 4300 Acked Read 1 Done Done Sent false
11542 4301 Acked Write 6 Done Done Sent false
11543 4302 Acked Read 5 Done Done Sent false
11544 4303 Acked Read 5 Done Done Sent false
11545 4304 Acked Read 7 Done Done Sent false
11546 4305 Acked Read 9 Done Done Sent false
11547 4306 Acked Read 5 Done Done Sent false
11548 4307 Acked Read 9 Done Done Sent false
11549 4308 Acked Write 9 Done Done Sent false
11550 4309 Acked Write 3 Done Done Sent false
11551 4310 Acked Flush 0 Done Done Sent false
11552 4311 Acked Write 5 Done Done Sent false
11553 4312 Acked Read 2 Done Done Sent false
11554 4313 Acked Read 1 Done Done Sent false
11555 4314 Acked Write 4 Done Done Sent false
11556 4315 Acked Read 9 Done Done Sent false
11557 4316 Acked Read 6 Done Done Sent false
11558 4317 Acked Read 3 Done Done Sent false
11559 4318 Acked Read 6 Done Done Sent false
11560 4319 Acked Write 1 Done Done Sent false
11561 4320 Acked Write 8 Done Done Sent false
11562 STATES DS:0 DS:1 DS:2 TOTAL
11563 Sent 0 0 301 301
11564 Done 301 301 0 602
11565 Skipped 0 0 0 0
11566 Error 0 0 0 0
11567 Last Flush: Some(JobId(4310)) Some(JobId(4310)) Some(JobId(4019))
11568 Downstairs last five completed: 4019 4018 4017 4016 4015
11569 Upstairs last five completed: 4320 4319 4318 4317 4316
11570 CLIENT: Up:0 ds:301 act:2
115712025-01-28T21:56:45.544ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
115722025-01-28T21:56:46.354ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
115732025-01-28T21:56:46.354ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
115742025-01-28T21:56:46.354ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
115752025-01-28T21:56:46.354ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
115762025-01-28T21:56:46.354ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
115772025-01-28T21:56:46.855ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
115782025-01-28T21:56:47.547ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11579 [8830] got start action
11580 [8830] I am going to restart
11581 Starting downstairs at port 8830
11582 Make output file at "/tmp/dsc/downstairs-8830.txt"
11583 [8830][2] reports Starting
11584 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1213)
11585 [8830][2] reports Running
115862025-01-28T21:56:48.548ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11587 ----------------------------------------------------------------
11588 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11589 States: Active Active Offline
11590 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11591 4020 Acked Flush 0 Done Done Sent false
11592 4021 Acked Read 10 Done Done Sent false
11593 4022 Acked Read 6 Done Done Sent false
11594 4023 Acked Write 5 Done Done Sent false
11595 4024 Acked Flush 0 Done Done Sent false
11596 4025 Acked Read 7 Done Done Sent false
11597 4026 Acked Read 9 Done Done Sent false
11598 4027 Acked Flush 0 Done Done Sent false
11599 4028 Acked Read 2 Done Done Sent false
11600 4029 Acked Write 6 Done Done Sent false
11601 4030 Acked Read 3 Done Done Sent false
11602 4031 Acked Write 1 Done Done Sent false
11603 4032 Acked Read 1 Done Done Sent false
11604 4033 Acked Read 8 Done Done Sent false
11605 4034 Acked Read 5 Done Done Sent false
11606 4035 Acked Read 7 Done Done Sent false
11607 4036 Acked Write 9 Done Done Sent false
11608 4037 Acked Write 6 Done Done Sent false
11609 4038 Acked Write 8 Done Done Sent false
11610 4039 Acked Read 8 Done Done Sent false
11611 4040 Acked Write 3 Done Done Sent false
11612 4041 Acked Write 8 Done Done Sent false
11613 4042 Acked Flush 0 Done Done Sent false
11614 4043 Acked Flush 0 Done Done Sent false
11615 4044 Acked Read 10 Done Done Sent false
11616 4045 Acked Write 2 Done Done Sent false
11617 4046 Acked Read 4 Done Done Sent false
11618 4047 Acked Read 2 Done Done Sent false
11619 4048 Acked Write 2 Done Done Sent false
11620 4049 Acked Write 4 Done Done Sent false
11621 4050 Acked Read 1 Done Done Sent false
11622 4051 Acked Write 9 Done Done Sent false
11623 4052 Acked Write 4 Done Done Sent false
11624 4053 Acked Write 6 Done Done Sent false
11625 4054 Acked Write 3 Done Done Sent false
11626 4055 Acked Write 2 Done Done Sent false
11627 4056 Acked Write 4 Done Done Sent false
11628 4057 Acked Flush 0 Done Done Sent false
11629 4058 Acked Write 1 Done Done Sent false
11630 4059 Acked Read 10 Done Done Sent false
11631 4060 Acked Read 2 Done Done Sent false
11632 4061 Acked Read 2 Done Done Sent false
11633 4062 Acked Write 8 Done Done Sent false
11634 4063 Acked Write 2 Done Done Sent false
11635 4064 Acked Write 3 Done Done Sent false
11636 4065 Acked Read 10 Done Done Sent false
11637 4066 Acked Flush 0 Done Done Sent false
11638 4067 Acked Read 7 Done Done Sent false
11639 4068 Acked Write 7 Done Done Sent false
11640 4069 Acked Read 9 Done Done Sent false
11641 4070 Acked Read 4 Done Done Sent false
11642 4071 Acked Read 7 Done Done Sent false
11643 4072 Acked Read 9 Done Done Sent false
11644 4073 Acked Write 10 Done Done Sent false
11645 4074 Acked Read 9 Done Done Sent false
11646 4075 Acked Write 3 Done Done Sent false
11647 4076 Acked Write 5 Done Done Sent false
11648 4077 Acked Write 9 Done Done Sent false
11649 4078 Acked Write 3 Done Done Sent false
11650 4079 Acked Read 7 Done Done Sent false
11651 4080 Acked Read 7 Done Done Sent false
11652 4081 Acked Flush 0 Done Done Sent false
11653 4082 Acked Read 4 Done Done Sent false
11654 4083 Acked Read 7 Done Done Sent false
11655 4084 Acked Write 10 Done Done Sent false
11656 4085 Acked Read 2 Done Done Sent false
11657 4086 Acked Write 4 Done Done Sent false
11658 4087 Acked Write 5 Done Done Sent false
11659 4088 Acked Read 7 Done Done Sent false
11660 4089 Acked Write 2 Done Done Sent false
11661 4090 Acked Read 9 Done Done Sent false
11662 4091 Acked Write 7 Done Done Sent false
11663 4092 Acked Read 2 Done Done Sent false
11664 4093 Acked Write 9 Done Done Sent false
11665 4094 Acked Write 10 Done Done Sent false
11666 4095 Acked Read 5 Done Done Sent false
11667 4096 Acked Flush 0 Done Done Sent false
11668 4097 Acked Write 4 Done Done Sent false
11669 4098 Acked Read 10 Done Done Sent false
11670 4099 Acked Write 4 Done Done Sent false
11671 4100 Acked Read 6 Done Done Sent false
11672 4101 Acked Read 6 Done Done Sent false
11673 4102 Acked Read 3 Done Done Sent false
11674 4103 Acked Read 10 Done Done Sent false
11675 4104 Acked Read 2 Done Done Sent false
11676 4105 Acked Read 9 Done Done Sent false
11677 4106 Acked Read 3 Done Done Sent false
11678 4107 Acked Read 4 Done Done Sent false
11679 4108 Acked Write 1 Done Done Sent false
11680 4109 Acked Read 6 Done Done Sent false
11681 4110 Acked Write 2 Done Done Sent false
11682 4111 Acked Flush 0 Done Done Sent false
11683 4112 Acked Write 9 Done Done Sent false
11684 4113 Acked Read 6 Done Done Sent false
11685 4114 Acked Flush 0 Done Done Sent false
11686 4115 Acked Read 1 Done Done Sent false
11687 4116 Acked Read 7 Done Done Sent false
11688 4117 Acked Read 4 Done Done Sent false
11689 4118 Acked Read 8 Done Done Sent false
11690 4119 Acked Read 10 Done Done Sent false
11691 4120 Acked Write 4 Done Done Sent false
11692 4121 Acked Read 1 Done Done Sent false
11693 4122 Acked Write 8 Done Done Sent false
11694 4123 Acked Read 2 Done Done Sent false
11695 4124 Acked Read 3 Done Done Sent false
11696 4125 Acked Read 7 Done Done Sent false
11697 4126 Acked Read 3 Done Done Sent false
11698 4127 Acked Flush 0 Done Done Sent false
11699 4128 Acked Read 8 Done Done Sent false
11700 4129 Acked Read 7 Done Done Sent false
11701 4130 Acked Write 1 Done Done Sent false
11702 4131 Acked Read 6 Done Done Sent false
11703 4132 Acked Flush 0 Done Done Sent false
11704 4133 Acked Read 8 Done Done Sent false
11705 4134 Acked Flush 0 Done Done Sent false
11706 4135 Acked Write 3 Done Done Sent false
11707 4136 Acked Write 5 Done Done Sent false
11708 4137 Acked Read 1 Done Done Sent false
11709 4138 Acked Read 3 Done Done Sent false
11710 4139 Acked Read 7 Done Done Sent false
11711 4140 Acked Write 5 Done Done Sent false
11712 4141 Acked Write 4 Done Done Sent false
11713 4142 Acked Read 4 Done Done Sent false
11714 4143 Acked Write 1 Done Done Sent false
11715 4144 Acked Write 8 Done Done Sent false
11716 4145 Acked Read 10 Done Done Sent false
11717 4146 Acked Flush 0 Done Done Sent false
11718 4147 Acked Read 9 Done Done Sent false
11719 4148 Acked Flush 0 Done Done Sent false
11720 4149 Acked Read 2 Done Done Sent false
11721 4150 Acked Write 7 Done Done Sent false
11722 4151 Acked Read 2 Done Done Sent false
11723 4152 Acked Read 1 Done Done Sent false
11724 4153 Acked Read 5 Done Done Sent false
11725 4154 Acked Read 3 Done Done Sent false
11726 4155 Acked Read 10 Done Done Sent false
11727 4156 Acked Read 10 Done Done Sent false
11728 4157 Acked Read 1 Done Done Sent false
11729 4158 Acked Write 1 Done Done Sent false
11730 4159 Acked Read 1 Done Done Sent false
11731 4160 Acked Read 10 Done Done Sent false
11732 4161 Acked Flush 0 Done Done Sent false
11733 4162 Acked Write 7 Done Done Sent false
11734 4163 Acked Read 4 Done Done Sent false
11735 4164 Acked Write 9 Done Done Sent false
11736 4165 Acked Write 10 Done Done Sent false
11737 4166 Acked Read 1 Done Done Sent false
11738 4167 Acked Write 3 Done Done Sent false
11739 4168 Acked Read 10 Done Done Sent false
11740 4169 Acked Read 7 Done Done Sent false
11741 4170 Acked Write 5 Done Done Sent false
11742 4171 Acked Write 2 Done Done Sent false
11743 4172 Acked Read 2 Done Done Sent false
11744 4173 Acked Read 10 Done Done Sent false
11745 4174 Acked Read 1 Done Done Sent false
11746 4175 Acked Read 6 Done Done Sent false
11747 4176 Acked Flush 0 Done Done Sent false
11748 4177 Acked Write 10 Done Done Sent false
11749 4178 Acked Write 7 Done Done Sent false
11750 4179 Acked Read 2 Done Done Sent false
11751 4180 Acked Write 5 Done Done Sent false
11752 4181 Acked Flush 0 Done Done Sent false
11753 4182 Acked Write 3 Done Done Sent false
11754 4183 Acked Read 3 Done Done Sent false
11755 4184 Acked Write 6 Done Done Sent false
11756 4185 Acked Read 2 Done Done Sent false
11757 4186 Acked Read 6 Done Done Sent false
11758 4187 Acked Write 5 Done Done Sent false
11759 4188 Acked Read 6 Done Done Sent false
11760 4189 Acked Write 8 Done Done Sent false
11761 4190 Acked Write 9 Done Done Sent false
11762 4191 Acked Write 3 Done Done Sent false
11763 4192 Acked Write 1 Done Done Sent false
11764 4193 Acked Read 9 Done Done Sent false
11765 4194 Acked Read 10 Done Done Sent false
11766 4195 Acked Read 3 Done Done Sent false
11767 4196 Acked Read 5 Done Done Sent false
11768 4197 Acked Write 6 Done Done Sent false
11769 4198 Acked Write 1 Done Done Sent false
11770 4199 Acked Write 2 Done Done Sent false
11771 4200 Acked Write 4 Done Done Sent false
11772 4201 Acked Read 7 Done Done Sent false
11773 4202 Acked Write 1 Done Done Sent false
11774 4203 Acked Read 10 Done Done Sent false
11775 4204 Acked Read 10 Done Done Sent false
11776 4205 Acked Write 4 Done Done Sent false
11777 4206 Acked Write 9 Done Done Sent false
11778 4207 Acked Write 10 Done Done Sent false
11779 4208 Acked Read 7 Done Done Sent false
11780 4209 Acked Read 5 Done Done Sent false
11781 4210 Acked Flush 0 Done Done Sent false
11782 4211 Acked Write 9 Done Done Sent false
11783 4212 Acked Read 4 Done Done Sent false
11784 4213 Acked Read 5 Done Done Sent false
11785 4214 Acked Read 10 Done Done Sent false
11786 4215 Acked Write 2 Done Done Sent false
11787 4216 Acked Read 9 Done Done Sent false
11788 4217 Acked Write 6 Done Done Sent false
11789 4218 Acked Write 3 Done Done Sent false
11790 4219 Acked Read 10 Done Done Sent false
11791 4220 Acked Write 9 Done Done Sent false
11792 4221 Acked Flush 0 Done Done Sent false
11793 4222 Acked Read 5 Done Done Sent false
11794 4223 Acked Read 1 Done Done Sent false
11795 4224 Acked Read 8 Done Done Sent false
11796 4225 Acked Write 3 Done Done Sent false
11797 4226 Acked Read 7 Done Done Sent false
11798 4227 Acked Read 2 Done Done Sent false
11799 4228 Acked Read 7 Done Done Sent false
11800 4229 Acked Read 3 Done Done Sent false
11801 4230 Acked Flush 0 Done Done Sent false
11802 4231 Acked Flush 0 Done Done Sent false
11803 4232 Acked Read 7 Done Done Sent false
11804 4233 Acked Read 7 Done Done Sent false
11805 4234 Acked Write 9 Done Done Sent false
11806 4235 Acked Read 4 Done Done Sent false
11807 4236 Acked Flush 0 Done Done Sent false
11808 4237 Acked Flush 0 Done Done Sent false
11809 4238 Acked Write 3 Done Done Sent false
11810 4239 Acked Read 3 Done Done Sent false
11811 4240 Acked Write 7 Done Done Sent false
11812 4241 Acked Read 5 Done Done Sent false
11813 4242 Acked Write 9 Done Done Sent false
11814 4243 Acked Write 9 Done Done Sent false
11815 4244 Acked Write 4 Done Done Sent false
11816 4245 Acked Write 1 Done Done Sent false
11817 4246 Acked Read 6 Done Done Sent false
11818 4247 Acked Read 1 Done Done Sent false
11819 4248 Acked Flush 0 Done Done Sent false
11820 4249 Acked Read 7 Done Done Sent false
11821 4250 Acked Read 4 Done Done Sent false
11822 4251 Acked Write 9 Done Done Sent false
11823 4252 Acked Write 7 Done Done Sent false
11824 4253 Acked Write 10 Done Done Sent false
11825 4254 Acked Read 8 Done Done Sent false
11826 4255 Acked Read 3 Done Done Sent false
11827 4256 Acked Read 7 Done Done Sent false
11828 4257 Acked Read 2 Done Done Sent false
11829 4258 Acked Write 10 Done Done Sent false
11830 4259 Acked Read 8 Done Done Sent false
11831 4260 Acked Read 2 Done Done Sent false
11832 4261 Acked Write 3 Done Done Sent false
11833 4262 Acked Write 9 Done Done Sent false
11834 4263 Acked Read 8 Done Done Sent false
11835 4264 Acked Write 10 Done Done Sent false
11836 4265 Acked Read 4 Done Done Sent false
11837 4266 Acked Read 4 Done Done Sent false
11838 4267 Acked Flush 0 Done Done Sent false
11839 4268 Acked Read 2 Done Done Sent false
11840 4269 Acked Read 2 Done Done Sent false
11841 4270 Acked Write 8 Done Done Sent false
11842 4271 Acked Write 3 Done Done Sent false
11843 4272 Acked Write 3 Done Done Sent false
11844 4273 Acked Read 5 Done Done Sent false
11845 4274 Acked Write 1 Done Done Sent false
11846 4275 Acked Flush 0 Done Done Sent false
11847 4276 Acked Write 6 Done Done Sent false
11848 4277 Acked Flush 0 Done Done Sent false
11849 4278 Acked Read 8 Done Done Sent false
11850 4279 Acked Read 9 Done Done Sent false
11851 4280 Acked Read 2 Done Done Sent false
11852 4281 Acked Read 5 Done Done Sent false
11853 4282 Acked Read 3 Done Done Sent false
11854 4283 Acked Write 10 Done Done Sent false
11855 4284 Acked Read 1 Done Done Sent false
11856 4285 Acked Read 4 Done Done Sent false
11857 4286 Acked Read 7 Done Done Sent false
11858 4287 Acked Write 9 Done Done Sent false
11859 4288 Acked Write 6 Done Done Sent false
11860 4289 Acked Write 7 Done Done Sent false
11861 4290 Acked Flush 0 Done Done Sent false
11862 4291 Acked Write 7 Done Done Sent false
11863 4292 Acked Read 7 Done Done Sent false
11864 4293 Acked Write 4 Done Done Sent false
11865 4294 Acked Read 2 Done Done Sent false
11866 4295 Acked Read 4 Done Done Sent false
11867 4296 Acked Write 1 Done Done Sent false
11868 4297 Acked Read 7 Done Done Sent false
11869 4298 Acked Read 7 Done Done Sent false
11870 4299 Acked Flush 0 Done Done Sent false
11871 4300 Acked Read 1 Done Done Sent false
11872 4301 Acked Write 6 Done Done Sent false
11873 4302 Acked Read 5 Done Done Sent false
11874 4303 Acked Read 5 Done Done Sent false
11875 4304 Acked Read 7 Done Done Sent false
11876 4305 Acked Read 9 Done Done Sent false
11877 4306 Acked Read 5 Done Done Sent false
11878 4307 Acked Read 9 Done Done Sent false
11879 4308 Acked Write 9 Done Done Sent false
11880 4309 Acked Write 3 Done Done Sent false
11881 4310 Acked Flush 0 Done Done Sent false
11882 4311 Acked Write 5 Done Done Sent false
11883 4312 Acked Read 2 Done Done Sent false
11884 4313 Acked Read 1 Done Done Sent false
11885 4314 Acked Write 4 Done Done Sent false
11886 4315 Acked Read 9 Done Done Sent false
11887 4316 Acked Read 6 Done Done Sent false
11888 4317 Acked Read 3 Done Done Sent false
11889 4318 Acked Read 6 Done Done Sent false
11890 4319 Acked Write 1 Done Done Sent false
11891 4320 Acked Write 8 Done Done Sent false
11892 STATES DS:0 DS:1 DS:2 TOTAL
11893 Sent 0 0 301 301
11894 Done 301 301 0 602
11895 Skipped 0 0 0 0
11896 Error 0 0 0 0
11897 Last Flush: Some(JobId(4310)) Some(JobId(4310)) Some(JobId(4019))
11898 Downstairs last five completed: 4019 4018 4017 4016 4015
11899 Upstairs last five completed: 4320 4319 4318 4317 4316
11900 CLIENT: Up:0 ds:301 act:2
119012025-01-28T21:56:49.549ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
119022025-01-28T21:56:50.551ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
119032025-01-28T21:56:51.553ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
119042025-01-28T21:56:52.555ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
11905 ----------------------------------------------------------------
11906 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11907 States: Active Active Offline
11908 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11909 4020 Acked Flush 0 Done Done Sent false
11910 4021 Acked Read 10 Done Done Sent false
11911 4022 Acked Read 6 Done Done Sent false
11912 4023 Acked Write 5 Done Done Sent false
11913 4024 Acked Flush 0 Done Done Sent false
11914 4025 Acked Read 7 Done Done Sent false
11915 4026 Acked Read 9 Done Done Sent false
11916 4027 Acked Flush 0 Done Done Sent false
11917 4028 Acked Read 2 Done Done Sent false
11918 4029 Acked Write 6 Done Done Sent false
11919 4030 Acked Read 3 Done Done Sent false
11920 4031 Acked Write 1 Done Done Sent false
11921 4032 Acked Read 1 Done Done Sent false
11922 4033 Acked Read 8 Done Done Sent false
11923 4034 Acked Read 5 Done Done Sent false
11924 4035 Acked Read 7 Done Done Sent false
11925 4036 Acked Write 9 Done Done Sent false
11926 4037 Acked Write 6 Done Done Sent false
11927 4038 Acked Write 8 Done Done Sent false
11928 4039 Acked Read 8 Done Done Sent false
11929 4040 Acked Write 3 Done Done Sent false
11930 4041 Acked Write 8 Done Done Sent false
11931 4042 Acked Flush 0 Done Done Sent false
11932 4043 Acked Flush 0 Done Done Sent false
11933 4044 Acked Read 10 Done Done Sent false
11934 4045 Acked Write 2 Done Done Sent false
11935 4046 Acked Read 4 Done Done Sent false
11936 4047 Acked Read 2 Done Done Sent false
11937 4048 Acked Write 2 Done Done Sent false
11938 4049 Acked Write 4 Done Done Sent false
11939 4050 Acked Read 1 Done Done Sent false
11940 4051 Acked Write 9 Done Done Sent false
11941 4052 Acked Write 4 Done Done Sent false
11942 4053 Acked Write 6 Done Done Sent false
11943 4054 Acked Write 3 Done Done Sent false
11944 4055 Acked Write 2 Done Done Sent false
11945 4056 Acked Write 4 Done Done Sent false
11946 4057 Acked Flush 0 Done Done Sent false
11947 4058 Acked Write 1 Done Done Sent false
11948 4059 Acked Read 10 Done Done Sent false
11949 4060 Acked Read 2 Done Done Sent false
11950 4061 Acked Read 2 Done Done Sent false
11951 4062 Acked Write 8 Done Done Sent false
11952 4063 Acked Write 2 Done Done Sent false
11953 4064 Acked Write 3 Done Done Sent false
11954 4065 Acked Read 10 Done Done Sent false
11955 4066 Acked Flush 0 Done Done Sent false
11956 4067 Acked Read 7 Done Done Sent false
11957 4068 Acked Write 7 Done Done Sent false
11958 4069 Acked Read 9 Done Done Sent false
11959 4070 Acked Read 4 Done Done Sent false
11960 4071 Acked Read 7 Done Done Sent false
11961 4072 Acked Read 9 Done Done Sent false
11962 4073 Acked Write 10 Done Done Sent false
11963 4074 Acked Read 9 Done Done Sent false
11964 4075 Acked Write 3 Done Done Sent false
11965 4076 Acked Write 5 Done Done Sent false
11966 4077 Acked Write 9 Done Done Sent false
11967 4078 Acked Write 3 Done Done Sent false
11968 4079 Acked Read 7 Done Done Sent false
11969 4080 Acked Read 7 Done Done Sent false
11970 4081 Acked Flush 0 Done Done Sent false
11971 4082 Acked Read 4 Done Done Sent false
11972 4083 Acked Read 7 Done Done Sent false
11973 4084 Acked Write 10 Done Done Sent false
11974 4085 Acked Read 2 Done Done Sent false
11975 4086 Acked Write 4 Done Done Sent false
11976 4087 Acked Write 5 Done Done Sent false
11977 4088 Acked Read 7 Done Done Sent false
11978 4089 Acked Write 2 Done Done Sent false
11979 4090 Acked Read 9 Done Done Sent false
11980 4091 Acked Write 7 Done Done Sent false
11981 4092 Acked Read 2 Done Done Sent false
11982 4093 Acked Write 9 Done Done Sent false
11983 4094 Acked Write 10 Done Done Sent false
11984 4095 Acked Read 5 Done Done Sent false
11985 4096 Acked Flush 0 Done Done Sent false
11986 4097 Acked Write 4 Done Done Sent false
11987 4098 Acked Read 10 Done Done Sent false
11988 4099 Acked Write 4 Done Done Sent false
11989 4100 Acked Read 6 Done Done Sent false
11990 4101 Acked Read 6 Done Done Sent false
11991 4102 Acked Read 3 Done Done Sent false
11992 4103 Acked Read 10 Done Done Sent false
11993 4104 Acked Read 2 Done Done Sent false
11994 4105 Acked Read 9 Done Done Sent false
11995 4106 Acked Read 3 Done Done Sent false
11996 4107 Acked Read 4 Done Done Sent false
11997 4108 Acked Write 1 Done Done Sent false
11998 4109 Acked Read 6 Done Done Sent false
11999 4110 Acked Write 2 Done Done Sent false
12000 4111 Acked Flush 0 Done Done Sent false
12001 4112 Acked Write 9 Done Done Sent false
12002 4113 Acked Read 6 Done Done Sent false
12003 4114 Acked Flush 0 Done Done Sent false
12004 4115 Acked Read 1 Done Done Sent false
12005 4116 Acked Read 7 Done Done Sent false
12006 4117 Acked Read 4 Done Done Sent false
12007 4118 Acked Read 8 Done Done Sent false
12008 4119 Acked Read 10 Done Done Sent false
12009 4120 Acked Write 4 Done Done Sent false
12010 4121 Acked Read 1 Done Done Sent false
12011 4122 Acked Write 8 Done Done Sent false
12012 4123 Acked Read 2 Done Done Sent false
12013 4124 Acked Read 3 Done Done Sent false
12014 4125 Acked Read 7 Done Done Sent false
12015 4126 Acked Read 3 Done Done Sent false
12016 4127 Acked Flush 0 Done Done Sent false
12017 4128 Acked Read 8 Done Done Sent false
12018 4129 Acked Read 7 Done Done Sent false
12019 4130 Acked Write 1 Done Done Sent false
12020 4131 Acked Read 6 Done Done Sent false
12021 4132 Acked Flush 0 Done Done Sent false
12022 4133 Acked Read 8 Done Done Sent false
12023 4134 Acked Flush 0 Done Done Sent false
12024 4135 Acked Write 3 Done Done Sent false
12025 4136 Acked Write 5 Done Done Sent false
12026 4137 Acked Read 1 Done Done Sent false
12027 4138 Acked Read 3 Done Done Sent false
12028 4139 Acked Read 7 Done Done Sent false
12029 4140 Acked Write 5 Done Done Sent false
12030 4141 Acked Write 4 Done Done Sent false
12031 4142 Acked Read 4 Done Done Sent false
12032 4143 Acked Write 1 Done Done Sent false
12033 4144 Acked Write 8 Done Done Sent false
12034 4145 Acked Read 10 Done Done Sent false
12035 4146 Acked Flush 0 Done Done Sent false
12036 4147 Acked Read 9 Done Done Sent false
12037 4148 Acked Flush 0 Done Done Sent false
12038 4149 Acked Read 2 Done Done Sent false
12039 4150 Acked Write 7 Done Done Sent false
12040 4151 Acked Read 2 Done Done Sent false
12041 4152 Acked Read 1 Done Done Sent false
12042 4153 Acked Read 5 Done Done Sent false
12043 4154 Acked Read 3 Done Done Sent false
12044 4155 Acked Read 10 Done Done Sent false
12045 4156 Acked Read 10 Done Done Sent false
12046 4157 Acked Read 1 Done Done Sent false
12047 4158 Acked Write 1 Done Done Sent false
12048 4159 Acked Read 1 Done Done Sent false
12049 4160 Acked Read 10 Done Done Sent false
12050 4161 Acked Flush 0 Done Done Sent false
12051 4162 Acked Write 7 Done Done Sent false
12052 4163 Acked Read 4 Done Done Sent false
12053 4164 Acked Write 9 Done Done Sent false
12054 4165 Acked Write 10 Done Done Sent false
12055 4166 Acked Read 1 Done Done Sent false
12056 4167 Acked Write 3 Done Done Sent false
12057 4168 Acked Read 10 Done Done Sent false
12058 4169 Acked Read 7 Done Done Sent false
12059 4170 Acked Write 5 Done Done Sent false
12060 4171 Acked Write 2 Done Done Sent false
12061 4172 Acked Read 2 Done Done Sent false
12062 4173 Acked Read 10 Done Done Sent false
12063 4174 Acked Read 1 Done Done Sent false
12064 4175 Acked Read 6 Done Done Sent false
12065 4176 Acked Flush 0 Done Done Sent false
12066 4177 Acked Write 10 Done Done Sent false
12067 4178 Acked Write 7 Done Done Sent false
12068 4179 Acked Read 2 Done Done Sent false
12069 4180 Acked Write 5 Done Done Sent false
12070 4181 Acked Flush 0 Done Done Sent false
12071 4182 Acked Write 3 Done Done Sent false
12072 4183 Acked Read 3 Done Done Sent false
12073 4184 Acked Write 6 Done Done Sent false
12074 4185 Acked Read 2 Done Done Sent false
12075 4186 Acked Read 6 Done Done Sent false
12076 4187 Acked Write 5 Done Done Sent false
12077 4188 Acked Read 6 Done Done Sent false
12078 4189 Acked Write 8 Done Done Sent false
12079 4190 Acked Write 9 Done Done Sent false
12080 4191 Acked Write 3 Done Done Sent false
12081 4192 Acked Write 1 Done Done Sent false
12082 4193 Acked Read 9 Done Done Sent false
12083 4194 Acked Read 10 Done Done Sent false
12084 4195 Acked Read 3 Done Done Sent false
12085 4196 Acked Read 5 Done Done Sent false
12086 4197 Acked Write 6 Done Done Sent false
12087 4198 Acked Write 1 Done Done Sent false
12088 4199 Acked Write 2 Done Done Sent false
12089 4200 Acked Write 4 Done Done Sent false
12090 4201 Acked Read 7 Done Done Sent false
12091 4202 Acked Write 1 Done Done Sent false
12092 4203 Acked Read 10 Done Done Sent false
12093 4204 Acked Read 10 Done Done Sent false
12094 4205 Acked Write 4 Done Done Sent false
12095 4206 Acked Write 9 Done Done Sent false
12096 4207 Acked Write 10 Done Done Sent false
12097 4208 Acked Read 7 Done Done Sent false
12098 4209 Acked Read 5 Done Done Sent false
12099 4210 Acked Flush 0 Done Done Sent false
12100 4211 Acked Write 9 Done Done Sent false
12101 4212 Acked Read 4 Done Done Sent false
12102 4213 Acked Read 5 Done Done Sent false
12103 4214 Acked Read 10 Done Done Sent false
12104 4215 Acked Write 2 Done Done Sent false
12105 4216 Acked Read 9 Done Done Sent false
12106 4217 Acked Write 6 Done Done Sent false
12107 4218 Acked Write 3 Done Done Sent false
12108 4219 Acked Read 10 Done Done Sent false
12109 4220 Acked Write 9 Done Done Sent false
12110 4221 Acked Flush 0 Done Done Sent false
12111 4222 Acked Read 5 Done Done Sent false
12112 4223 Acked Read 1 Done Done Sent false
12113 4224 Acked Read 8 Done Done Sent false
12114 4225 Acked Write 3 Done Done Sent false
12115 4226 Acked Read 7 Done Done Sent false
12116 4227 Acked Read 2 Done Done Sent false
12117 4228 Acked Read 7 Done Done Sent false
12118 4229 Acked Read 3 Done Done Sent false
12119 4230 Acked Flush 0 Done Done Sent false
12120 4231 Acked Flush 0 Done Done Sent false
12121 4232 Acked Read 7 Done Done Sent false
12122 4233 Acked Read 7 Done Done Sent false
12123 4234 Acked Write 9 Done Done Sent false
12124 4235 Acked Read 4 Done Done Sent false
12125 4236 Acked Flush 0 Done Done Sent false
12126 4237 Acked Flush 0 Done Done Sent false
12127 4238 Acked Write 3 Done Done Sent false
12128 4239 Acked Read 3 Done Done Sent false
12129 4240 Acked Write 7 Done Done Sent false
12130 4241 Acked Read 5 Done Done Sent false
12131 4242 Acked Write 9 Done Done Sent false
12132 4243 Acked Write 9 Done Done Sent false
12133 4244 Acked Write 4 Done Done Sent false
12134 4245 Acked Write 1 Done Done Sent false
12135 4246 Acked Read 6 Done Done Sent false
12136 4247 Acked Read 1 Done Done Sent false
12137 4248 Acked Flush 0 Done Done Sent false
12138 4249 Acked Read 7 Done Done Sent false
12139 4250 Acked Read 4 Done Done Sent false
12140 4251 Acked Write 9 Done Done Sent false
12141 4252 Acked Write 7 Done Done Sent false
12142 4253 Acked Write 10 Done Done Sent false
12143 4254 Acked Read 8 Done Done Sent false
12144 4255 Acked Read 3 Done Done Sent false
12145 4256 Acked Read 7 Done Done Sent false
12146 4257 Acked Read 2 Done Done Sent false
12147 4258 Acked Write 10 Done Done Sent false
12148 4259 Acked Read 8 Done Done Sent false
12149 4260 Acked Read 2 Done Done Sent false
12150 4261 Acked Write 3 Done Done Sent false
12151 4262 Acked Write 9 Done Done Sent false
12152 4263 Acked Read 8 Done Done Sent false
12153 4264 Acked Write 10 Done Done Sent false
12154 4265 Acked Read 4 Done Done Sent false
12155 4266 Acked Read 4 Done Done Sent false
12156 4267 Acked Flush 0 Done Done Sent false
12157 4268 Acked Read 2 Done Done Sent false
12158 4269 Acked Read 2 Done Done Sent false
12159 4270 Acked Write 8 Done Done Sent false
12160 4271 Acked Write 3 Done Done Sent false
12161 4272 Acked Write 3 Done Done Sent false
12162 4273 Acked Read 5 Done Done Sent false
12163 4274 Acked Write 1 Done Done Sent false
12164 4275 Acked Flush 0 Done Done Sent false
12165 4276 Acked Write 6 Done Done Sent false
12166 4277 Acked Flush 0 Done Done Sent false
12167 4278 Acked Read 8 Done Done Sent false
12168 4279 Acked Read 9 Done Done Sent false
12169 4280 Acked Read 2 Done Done Sent false
12170 4281 Acked Read 5 Done Done Sent false
12171 4282 Acked Read 3 Done Done Sent false
12172 4283 Acked Write 10 Done Done Sent false
12173 4284 Acked Read 1 Done Done Sent false
12174 4285 Acked Read 4 Done Done Sent false
12175 4286 Acked Read 7 Done Done Sent false
12176 4287 Acked Write 9 Done Done Sent false
12177 4288 Acked Write 6 Done Done Sent false
12178 4289 Acked Write 7 Done Done Sent false
12179 4290 Acked Flush 0 Done Done Sent false
12180 4291 Acked Write 7 Done Done Sent false
12181 4292 Acked Read 7 Done Done Sent false
12182 4293 Acked Write 4 Done Done Sent false
12183 4294 Acked Read 2 Done Done Sent false
12184 4295 Acked Read 4 Done Done Sent false
12185 4296 Acked Write 1 Done Done Sent false
12186 4297 Acked Read 7 Done Done Sent false
12187 4298 Acked Read 7 Done Done Sent false
12188 4299 Acked Flush 0 Done Done Sent false
12189 4300 Acked Read 1 Done Done Sent false
12190 4301 Acked Write 6 Done Done Sent false
12191 4302 Acked Read 5 Done Done Sent false
12192 4303 Acked Read 5 Done Done Sent false
12193 4304 Acked Read 7 Done Done Sent false
12194 4305 Acked Read 9 Done Done Sent false
12195 4306 Acked Read 5 Done Done Sent false
12196 4307 Acked Read 9 Done Done Sent false
12197 4308 Acked Write 9 Done Done Sent false
12198 4309 Acked Write 3 Done Done Sent false
12199 4310 Acked Flush 0 Done Done Sent false
12200 4311 Acked Write 5 Done Done Sent false
12201 4312 Acked Read 2 Done Done Sent false
12202 4313 Acked Read 1 Done Done Sent false
12203 4314 Acked Write 4 Done Done Sent false
12204 4315 Acked Read 9 Done Done Sent false
12205 4316 Acked Read 6 Done Done Sent false
12206 4317 Acked Read 3 Done Done Sent false
12207 4318 Acked Read 6 Done Done Sent false
12208 4319 Acked Write 1 Done Done Sent false
12209 4320 Acked Write 8 Done Done Sent false
12210 STATES DS:0 DS:1 DS:2 TOTAL
12211 Sent 0 0 301 301
12212 Done 301 301 0 602
12213 Skipped 0 0 0 0
12214 Error 0 0 0 0
12215 Last Flush: Some(JobId(4310)) Some(JobId(4310)) Some(JobId(4019))
12216 Downstairs last five completed: 4019 4018 4017 4016 4015
12217 Upstairs last five completed: 4320 4319 4318 4317 4316
12218 CLIENT: Up:0 ds:301 act:2
122192025-01-28T21:56:53.557ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
122202025-01-28T21:56:54.559ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
122212025-01-28T21:56:55.560ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
122222025-01-28T21:56:56.355ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
122232025-01-28T21:56:56.355ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37326) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
122242025-01-28T21:56:56.357ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
122252025-01-28T21:56:56.357ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
122262025-01-28T21:56:56.357ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
122272025-01-28T21:56:56.357ZINFOcrucible: send last flush ID to this DS: Some(JobId(4019)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
122282025-01-28T21:56:56.357ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
122292025-01-28T21:56:56.357ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4019)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
122302025-01-28T21:56:56.357ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4019)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
12231 ----------------------------------------------------------------
12232 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:10
12233 States: Active Active Active
12234 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12235 4311 Acked Write 5 Done Done Done true
12236 4312 Acked Read 2 Done Done Done true
12237 4313 Acked Read 1 Done Done Done true
12238 4314 Acked Write 4 Done Done Done true
12239 4315 Acked Read 9 Done Done Done true
12240 4316 Acked Read 6 Done Done Done true
12241 4317 Acked Read 3 Done Done Done true
12242 4318 Acked Read 6 Done Done Done true
12243 4319 Acked Write 1 Done Done Done true
12244 4320 Acked Write 8 Done Done Done true
12245 STATES DS:0 DS:1 DS:2 TOTAL
12246 Sent 0 0 0 0
12247 Done 10 10 10 30
12248 Skipped 0 0 0 0
12249 Error 0 0 0 0
12250 Last Flush: Some(JobId(4310)) Some(JobId(4310)) Some(JobId(4310))
12251 Downstairs last five completed: 4310 4309 4308 4307 4306
12252 Upstairs last five completed: 4320 4319 4318 4317 4316
12253 CLIENT: Up:0 ds:10 act:3
12254 ----------------------------------------------------------------
12255 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12256 Downstairs last five completed: 4321 4320 4319 4318 4317
12257 Upstairs last five completed: 4321 4320 4319 4318 4317
12258 CLIENT: Up:0 ds:0 act:3
12259 Replay: All jobs finished
12260 Main task has work to do, go find it
12261 got dsc Stop(1)
12262 stop 1
12263 Jan 28 21:57:00.610 INFO request completed, [8820] Got stop action so:false kr:false
12264 latency_us: 338, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 6e8a1238-fb37-4d1a-b392-6760247fa15c, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
122652025-01-28T21:57:00.611ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
12266 Jan 28 21:57:00.611 INFO request completed{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:57:00.611590889Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
12267 , {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:57:00.611639305Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
12268 latency_us: 122, response_code: 200, uri: /state/cid/1, method: GET, req_id: 8895498c-08f5-4029-b6d9-36dd25c2b9a1, remote_addr: 127.0.0.1:36665, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:57:00.611710211Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
12269 local_addr: 127.0.0.1:9998
12270 [8820] Exited with: ExitStatus(unix_wait_status(9))
12271 [8820][1] reports Exit
12272 Jan 28 21:57:04.612 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/1, method: GET, req_id: 855efa39-c316-4bfa-bd0b-eae93a2a3b77, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
12273 001/300 Write block 596 len 12288 data: 2 2 3
12274 002/300 Read block 3177 len 28672
12275 003/300 Flush
12276 004/300 Write block 758 len 36864 data: 4 4 4 4 4 4 5 4 5
12277 005/300 Read block 3821 len 4096
12278 006/300 Read block 4023 len 36864
12279 007/300 Read block 2403 len 28672
12280 008/300 Read block 1815 len 36864
12281 009/300 Read block 4733 len 40960
12282 010/300 Write block 2869 len 32768 data: 3 3 3 3 3 3 3 5
12283 011/300 Write block 2287 len 28672 data: 4 4 4 4 4 4 4
12284 012/300 Write block 3698 len 36864 data: 2 3 3 3 3 3 2 2 2
12285 013/300 Read block 552 len 8192
12286 014/300 Read block 4747 len 36864
12287 015/300 Write block 2169 len 4096 data: 6
12288 016/300 Write block 229 len 28672 data: 3 3 4 5 6 5 6
12289 017/300 Read block 2148 len 36864
12290 018/300 Read block 639 len 20480
12291 019/300 Write block 1560 len 12288 data: 3 3 4
12292 020/300 Write block 1707 len 36864 data: 3 4 4 4 4 5 5 5 4
12293 021/300 Flush
12294 022/300 Write block 1452 len 16384 data: 3 3 4 4
12295 023/300 Write block 2098 len 36864 data: 3 4 3 3 3 4 3 3 2
12296 024/300 Write block 4576 len 28672 data: 2 2 2 3 3 3 3
12297 025/300 Read block 3835 len 4096
12298 026/300 Read block 4382 len 24576
12299 027/300 Flush
12300 028/300 Write block 4249 len 32768 data: 3 4 5 5 5 4 4 4
12301 029/300 Write block 3477 len 32768 data: 3 3 3 3 3 3 3 4
12302 030/300 Write block 0 len 16384 data: 2 2 2 2
12303 031/300 Write block 2018 len 20480 data: 3 3 2 2 2
12304 032/300 Read block 2736 len 36864
12305 033/300 Write block 545 len 36864 data: 5 5 7 7 5 5 4 4 4
12306 034/300 Write block 666 len 20480 data: 4 4 4 4 4
12307 035/300 Write block 3606 len 8192 data: 3 3
12308 036/300 Write block 3366 len 20480 data: 2 2 3 3 3
12309 037/300 Read block 2105 len 32768
12310 038/300 Read block 4219 len 12288
12311 039/300 Write block 2084 len 20480 data: 3 3 2 2 2
12312 040/300 Read block 3803 len 16384
12313 041/300 Write block 3492 len 20480 data: 3 3 3 3 2
12314 042/300 Read block 1083 len 40960
12315 043/300 Write block 3366 len 40960 data: 3 3 4 4 4 2 3 3 3 4
12316 044/300 Write block 565 len 16384 data: 3 3 4 4
12317 045/300 Write block 259 len 20480 data: 2 2 2 4 3
12318 046/300 Read block 2349 len 8192
12319 047/300 Read block 1268 len 40960
12320 048/300 Write block 3938 len 16384 data: 3 3 3 3
12321 049/300 Flush
12322 050/300 Write block 3065 len 40960 data: 4 4 4 4 3 2 2 2 2 2
12323 051/300 Read block 3868 len 28672
12324 052/300 Flush
12325 053/300 Write block 754 len 12288 data: 3 4 3
12326 054/300 Read block 1858 len 40960
12327 055/300 Write block 998 len 8192 data: 4 4
12328 056/300 Read block 3499 len 20480
12329 057/300 Write block 327 len 12288 data: 4 4 3
12330 058/300 Write block 1076 len 4096 data: 3
12331 059/300 Read block 2352 len 24576
12332 060/300 Write block 3387 len 40960 data: 4 6 6 5 5 5 5 4 4 4
12333 061/300 Read block 2252 len 32768
12334 062/300 Write block 3884 len 40960 data: 6 5 4 4 4 3 2 2 2 2
12335 063/300 Read block 4766 len 4096
12336 064/300 Write block 4845 len 40960 data: 4 4 4 3 4 5 5 5 6 5
12337 065/300 Read block 1859 len 40960
12338 066/300 Read block 2458 len 12288
12339 067/300 Read block 2891 len 24576
12340 068/300 Write block 4329 len 16384 data: 3 2 2 2
12341 069/300 Read block 3271 len 24576
12342 070/300 Read block 2579 len 32768
12343 071/300 Write block 2212 len 8192 data: 2 3
12344 072/300 Read block 2539 len 40960
12345 073/300 Flush
12346 074/300 Write block 938 len 20480 data: 6 6 6 7 6
12347 075/300 Read block 2784 len 40960
12348 076/300 Write block 4614 len 20480 data: 3 4 3 3 2
12349 077/300 Read block 2932 len 16384
12350 078/300 Flush
12351 079/300 Flush
12352 080/300 Write block 1476 len 32768 data: 2 2 2 3 3 3 4 4
12353 081/300 Write block 740 len 16384 data: 5 5 4 3
12354 082/300 Write block 2467 len 16384 data: 3 3 2 2
12355 083/300 Write block 1412 len 20480 data: 5 4 4 3 2
12356 084/300 Write block 842 len 4096 data: 4
12357 085/300 Write block 2261 len 24576 data: 4 4 3 3 2 3
12358 086/300 Flush
12359 087/300 Write block 3472 len 16384 data: 4 4 3 2
12360 088/300 Read block 1025 len 40960
12361 089/300 Flush
12362 090/300 Write block 4477 len 8192 data: 5 6
12363 091/300 Read block 3501 len 16384
12364 092/300 Read block 692 len 32768
12365 093/300 Write block 481 len 36864 data: 2 2 2 2 2 2 2 2 2
12366 094/300 Read block 1423 len 12288
12367 095/300 Read block 3663 len 12288
12368 096/300 Flush
12369 097/300 Read block 2730 len 40960
12370 098/300 Write block 2212 len 16384 data: 3 4 3 3
12371 099/300 Write block 2341 len 40960 data: 3 2 3 3 3 3 4 3 3 3
12372 100/300 Flush
12373 101/300 Read block 3391 len 28672
12374 102/300 Read block 664 len 20480
12375 103/300 Write block 859 len 12288 data: 2 2 2
12376 104/300 Flush
12377 105/300 Read block 4018 len 28672
12378 106/300 Write block 3595 len 32768 data: 2 2 4 4 5 4 4 4
12379 107/300 Read block 474 len 28672
12380 108/300 Read block 4466 len 24576
12381 109/300 Read block 1952 len 32768
12382 110/300 Write block 3735 len 8192 data: 4 4
12383 111/300 Write block 1446 len 16384 data: 3 3 2 2
12384 112/300 Read block 367 len 20480
12385 113/300 Read block 1169 len 32768
12386 114/300 Read block 3227 len 20480
12387 115/300 Read block 3679 len 32768
12388 116/300 Read block 1834 len 20480
12389 117/300 Read block 3736 len 24576
12390 118/300 Read block 2950 len 32768
12391 119/300 Read block 4169 len 40960
12392 120/300 Read block 3648 len 20480
12393 121/300 Write block 3315 len 32768 data: 5 4 4 4 3 4 4 5
12394 122/300 Read block 4320 len 28672
12395 123/300 Read block 3291 len 8192
12396 124/300 Read block 4671 len 12288
12397 125/300 Write block 4000 len 36864 data: 3 4 4 4 5 5 4 4 4
12398 126/300 Read block 3127 len 12288
12399 127/300 Write block 2729 len 32768 data: 4 5 4 4 5 5 4 3
12400 128/300 Read block 497 len 12288
12401 129/300 Write block 4975 len 36864 data: 2 2 2 2 2 3 5 5 5
12402 130/300 Read block 567 len 24576
12403 131/300 Write block 2831 len 12288 data: 4 4 3
12404 132/300 Write block 673 len 16384 data: 3 2 2 2
12405 133/300 Read block 4224 len 4096
12406 134/300 Write block 4091 len 40960 data: 5 5 4 4 4 4 4 3 3 3
12407 135/300 Flush
12408 136/300 Write block 4959 len 32768 data: 2 2 3 2 2 2 2 2
12409 137/300 Read block 2060 len 28672
12410 138/300 Write block 2594 len 16384 data: 3 3 2 3
12411 139/300 Flush
12412 140/300 Write block 233 len 24576 data: 7 6 7 6 4 5
12413 141/300 Flush
12414 142/300 Read block 1675 len 4096
12415 143/300 Write block 2595 len 4096 data: 4
12416 144/300 Write block 3977 len 32768 data: 3 3 4 4 4 4 4 3
12417 145/300 Write block 4890 len 16384 data: 4 4 4 4
12418 146/300 Read block 735 len 40960
12419 147/300 Write block 4264 len 20480 data: 4 4 4 3 3
12420 148/300 Write block 1973 len 24576 data: 3 5 4 4 4 3
12421 149/300 Write block 1187 len 36864 data: 4 4 3 3 3 3 4 4 4
12422 150/300 Write block 192 len 4096 data: 3
12423 151/300 Read block 2854 len 16384
12424 152/300 Write block 4475 len 24576 data: 3 4 6 7 6 7
12425 153/300 Write block 1344 len 8192 data: 5 5
12426 154/300 Read block 3030 len 36864
12427 155/300 Write block 4804 len 16384 data: 4 4 4 4
12428 156/300 Read block 4974 len 12288
12429 157/300 Read block 220 len 36864
12430 158/300 Read block 3059 len 16384
12431 159/300 Read block 275 len 40960
12432 160/300 Read block 3582 len 28672
12433 161/300 Write block 304 len 28672 data: 6 5 5 3 3 3 3
12434 162/300 Read block 107 len 40960
12435 163/300 Write block 2767 len 32768 data: 2 2 3 3 3 3 4 4
12436 164/300 Read block 3200 len 24576
12437 165/300 Read block 3835 len 40960
12438 166/300 Read block 1572 len 32768
12439 167/300 Write block 1987 len 36864 data: 4 4 3 3 3 2 2 2 2
12440 168/300 Read block 3407 len 20480
12441 169/300 Write block 4372 len 16384 data: 5 5 5 5
12442 170/300 Write block 4775 len 24576 data: 4 3 3 4 4 4
12443 171/300 Read block 2978 len 36864
12444 172/300 Read block 1081 len 16384
12445 173/300 Write block 2081 len 32768 data: 5 5 4 4 4 3 3 3
12446 174/300 Flush
12447 175/300 Flush
12448 176/300 Read block 2840 len 32768
12449 177/300 Read block 1500 len 28672
12450 178/300 Read block 2673 len 40960
12451 179/300 Write block 2647 len 20480 data: 4 4 4 3 3
12452 180/300 Flush
12453 181/300 Flush
12454 182/300 Read block 1242 len 4096
12455 183/300 Flush
12456 184/300 Read block 4113 len 12288
12457 185/300 Flush
12458 186/300 Read block 3371 len 28672
12459 187/300 Read block 3303 len 8192
12460 188/300 Read block 2100 len 8192
12461 189/300 Write block 2309 len 8192 data: 3 3
12462 190/300 Read block 3285 len 8192
12463 191/300 Flush
12464 192/300 Read block 742 len 24576
12465 193/300 Read block 580 len 32768
12466 194/300 Read block 2195 len 32768
12467 195/300 Write block 1172 len 40960 data: 3 3 3 2 2 2 4 6 6 5
12468 196/300 Read block 1610 len 24576
12469 197/300 Write block 3972 len 32768 data: 3 3 3 3 3 4 4 5
12470 198/300 Read block 2422 len 40960
12471 199/300 Write block 204 len 40960 data: 3 3 4 4 5 5 5 5 4 3
12472 200/300 Write block 4963 len 36864 data: 3 3 3 3 3 3 4 4 2
12473 201/300 Read block 1745 len 8192
12474 202/300 Read block 1189 len 20480
12475 203/300 Write block 4348 len 4096 data: 3
12476 204/300 Write block 2538 len 12288 data: 2 2 2
12477 205/300 Flush
12478 206/300 Read block 2940 len 36864
12479 207/300 Write block 3693 len 8192 data: 2 2
12480 208/300 Write block 3943 len 8192 data: 4 4
12481 209/300 Read block 2273 len 24576
12482 210/300 Read block 4663 len 32768
12483 211/300 Write block 1782 len 16384 data: 3 3 3 3
12484 212/300 Read block 372 len 4096
12485 213/300 Read block 3367 len 36864
12486 214/300 Flush
12487 215/300 Write block 684 len 28672 data: 3 3 2 3 4 5 5
12488 216/300 Write block 1176 len 28672 data: 3 3 5 7 7 6 6
12489 217/300 Write block 3762 len 12288 data: 4 4 4
12490 218/300 Write block 1137 len 12288 data: 2 2 2
12491 219/300 Write block 4330 len 40960 data: 3 3 3 2 3 3 4 5 5 5
12492 220/300 Write block 786 len 4096 data: 4
12493 221/300 Read block 3230 len 8192
12494 222/300 Write block 1713 len 24576 data: 6 6 5 5 6 6
12495 223/300 Write block 1964 len 20480 data: 3 3 2 3 4
12496 224/300 Read block 735 len 28672
12497 225/300 Flush
12498 226/300 Read block 530 len 40960
12499 227/300 Write block 319 len 8192 data: 5 4
12500 228/300 Read block 1977 len 24576
12501 229/300 Read block 4122 len 40960
12502 230/300 Write block 1517 len 36864 data: 6 6 6 6 5 5 4 4 4
12503 231/300 Flush
12504 232/300 Read block 1252 len 4096
12505 233/300 Flush
12506 234/300 Read block 4513 len 20480
12507 235/300 Write block 3589 len 36864 data: 4 4 4 4 3 3 3 3 5
12508 236/300 Flush
12509 237/300 Read block 411 len 8192
12510 238/300 Write block 1954 len 40960 data: 4 4 5 5 4 4 4 3 3 3
12511 239/300 Write block 3251 len 24576 data: 5 4 4 5 5 5
12512 240/300 Read block 3340 len 12288
12513 241/300 Write block 621 len 36864 data: 3 2 3 4 4 3 2 3 3
12514 242/300 Read block 4136 len 16384
12515 243/300 Write block 2146 len 32768 data: 4 4 3 4 4 5 5 5
12516 244/300 Read block 3133 len 8192
12517 245/300 Read block 1256 len 24576
12518 246/300 Read block 89 len 4096
12519 247/300 Read block 1390 len 28672
12520 248/300 Write block 4160 len 24576 data: 4 4 4 5 6 6
12521 249/300 Read block 148 len 36864
12522 250/300 Write block 3399 len 32768 data: 4 3 2 2 2 2 2 2
12523 251/300 Read block 960 len 12288
12524 252/300 Write block 2845 len 20480 data: 4 6 6 6 5
12525 253/300 Flush
12526 254/300 Read block 2985 len 40960
12527 255/300 Read block 2586 len 36864
12528 256/300 Flush
12529 257/300 Read block 3145 len 8192
12530 258/300 Write block 873 len 12288 data: 4 3 3
12531 259/300 Flush
12532 260/300 Read block 8 len 32768
12533 261/300 Read block 1278 len 40960
12534 262/300 Read block 4693 len 16384
12535 263/300 Flush
12536 264/300 Write block 3295 len 4096 data: 5
12537 265/300 Read block 3504 len 16384
12538 266/300 Write block 45 len 24576 data: 4 4 3 3 3 2
12539 267/300 Write block 4769 len 20480 data: 5 6 8 7 6
12540 268/300 Read block 4069 len 20480
12541 269/300 Write block 4687 len 4096 data: 4
12542 270/300 Read block 657 len 16384
12543 271/300 Read block 2329 len 28672
12544 272/300 Read block 485 len 32768
12545 273/300 Read block 4083 len 8192
12546 274/300 Read block 2958 len 40960
12547 275/300 Write block 3403 len 16384 data: 3 3 3 3
12548 276/300 Read block 2328 len 4096
12549 277/300 Read block 4705 len 24576
12550 278/300 Read block 3095 len 28672
12551 279/300 Read block 4423 len 24576
12552 280/300 Write block 476 len 16384 data: 6 6 3 3
12553 281/300 Read block 4072 len 20480
12554 282/300 Write block 2033 len 8192 data: 2 2
12555 283/300 Write block 4266 len 20480 data: 5 4 4 3 4
12556 284/300 Write block 3602 len 28672 data: 5 4 4 5 4 4 3
12557 285/300 Read block 3182 len 4096
12558 286/300 Read block 3998 len 36864
12559 287/300 Read block 4696 len 20480
12560 288/300 Write block 3447 len 4096 data: 4
12561 289/300 Flush
12562 290/300 Flush
12563 291/300 Read block 4978 len 12288
12564 292/300 Read block 1162 len 36864
12565 293/300 Write block 3984 len 16384 data: 4 2 2 3
12566 294/300 Write block 3380 len 16384 data: 4 3 2 2
12567 295/300 Flush
12568 296/300 Read block 2446 len 12288
12569 297/300 Read block 2884 len 4096
12570 298/300 Read block 382 len 20480
12571 299/300 Write block 3725 len 40960 data: 5 5 5 6 5 6 5 5 5 4
12572 300/300 Read block 3505 len 28672
12573 301/300 Write block 2379 len 8192 data: 3 3
12574 Main task has work to do, go find it
12575 got dsc Start(1)
12576 start 1
12577 Jan 28 21:57:04.857 INFO request completed, latency_us: 233, response_code: 204, uri: /start/cid/1, method: POST, req_id: d3efc77b-5531-4126-a6f1-27fe638fb614, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
12578 Replay: started 1, returned:Ok(())
12579 ----------------------------------------------------------------
12580 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12581 States: Active Offline Active
12582 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12583 4322 Acked Write 3 Done Sent Done false
12584 4323 Acked Read 7 Done Sent Done false
12585 4324 Acked Flush 0 Done Sent Done false
12586 4325 Acked Write 9 Done Sent Done false
12587 4326 Acked Read 1 Done Sent Done false
12588 4327 Acked Read 9 Done Sent Done false
12589 4328 Acked Read 7 Done Sent Done false
12590 4329 Acked Read 9 Done Sent Done false
12591 4330 Acked Read 10 Done Sent Done false
12592 4331 Acked Write 8 Done Sent Done false
12593 4332 Acked Write 7 Done Sent Done false
12594 4333 Acked Write 9 Done Sent Done false
12595 4334 Acked Read 2 Done Sent Done false
12596 4335 Acked Read 9 Done Sent Done false
12597 4336 Acked Write 1 Done Sent Done false
12598 4337 Acked Write 7 Done Sent Done false
12599 4338 Acked Read 9 Done Sent Done false
12600 4339 Acked Read 5 Done Sent Done false
12601 4340 Acked Write 3 Done Sent Done false
12602 4341 Acked Write 9 Done Sent Done false
12603 4342 Acked Flush 0 Done Sent Done false
12604 4343 Acked Write 4 Done Sent Done false
12605 4344 Acked Write 9 Done Sent Done false
12606 4345 Acked Write 7 Done Sent Done false
12607 4346 Acked Read 1 Done Sent Done false
12608 4347 Acked Read 6 Done Sent Done false
12609 4348 Acked Flush 0 Done Sent Done false
12610 4349 Acked Write 8 Done Sent Done false
12611 4350 Acked Write 8 Done Sent Done false
12612 4351 Acked Write 4 Done Sent Done false
12613 4352 Acked Write 5 Done Sent Done false
12614 4353 Acked Read 9 Done Sent Done false
12615 4354 Acked Write 9 Done Sent Done false
12616 4355 Acked Write 5 Done Sent Done false
12617 4356 Acked Write 2 Done Sent Done false
12618 4357 Acked Write 5 Done Sent Done false
12619 4358 Acked Read 8 Done Sent Done false
12620 4359 Acked Read 3 Done Sent Done false
12621 4360 Acked Write 5 Done Sent Done false
12622 4361 Acked Read 4 Done Sent Done false
12623 4362 Acked Write 5 Done Sent Done false
12624 4363 Acked Read 10 Done Sent Done false
12625 4364 Acked Write 10 Done Sent Done false
12626 4365 Acked Write 4 Done Sent Done false
12627 4366 Acked Write 5 Done Sent Done false
12628 4367 Acked Read 2 Done Sent Done false
12629 4368 Acked Read 10 Done Sent Done false
12630 4369 Acked Write 4 Done Sent Done false
12631 4370 Acked Flush 0 Done Sent Done false
12632 4371 Acked Write 10 Done Sent Done false
12633 4372 Acked Read 7 Done Sent Done false
12634 4373 Acked Flush 0 Done Sent Done false
12635 4374 Acked Write 3 Done Sent Done false
12636 4375 Acked Read 10 Done Sent Done false
12637 4376 Acked Write 2 Done Sent Done false
12638 4377 Acked Read 5 Done Sent Done false
12639 4378 Acked Write 3 Done Sent Done false
12640 4379 Acked Write 1 Done Sent Done false
12641 4380 Acked Read 6 Done Sent Done false
12642 4381 Acked Write 10 Done Sent Done false
12643 4382 Acked Read 8 Done Sent Done false
12644 4383 Acked Write 10 Done Sent Done false
12645 4384 Acked Read 1 Done Sent Done false
12646 4385 Acked Write 10 Done Sent Done false
12647 4386 Acked Read 10 Done Sent Done false
12648 4387 Acked Read 3 Done Sent Done false
12649 4388 Acked Read 6 Done Sent Done false
12650 4389 Acked Write 4 Done Sent Done false
12651 4390 Acked Read 6 Done Sent Done false
12652 4391 Acked Read 8 Done Sent Done false
12653 4392 Acked Write 2 Done Sent Done false
12654 4393 Acked Read 10 Done Sent Done false
12655 4394 Acked Flush 0 Done Sent Done false
12656 4395 Acked Write 5 Done Sent Done false
12657 4396 Acked Read 10 Done Sent Done false
12658 4397 Acked Write 5 Done Sent Done false
12659 4398 Acked Read 4 Done Sent Done false
12660 4399 Acked Flush 0 Done Sent Done false
12661 4400 Acked Flush 0 Done Sent Done false
12662 4401 Acked Write 8 Done Sent Done false
12663 4402 Acked Write 4 Done Sent Done false
12664 4403 Acked Write 4 Done Sent Done false
12665 4404 Acked Write 5 Done Sent Done false
12666 4405 Acked Write 1 Done Sent Done false
12667 4406 Acked Write 6 Done Sent Done false
12668 4407 Acked Flush 0 Done Sent Done false
12669 4408 Acked Write 4 Done Sent Done false
12670 4409 Acked Read 10 Done Sent Done false
12671 4410 Acked Flush 0 Done Sent Done false
12672 4411 Acked Write 2 Done Sent Done false
12673 4412 Acked Read 4 Done Sent Done false
12674 4413 Acked Read 8 Done Sent Done false
12675 4414 Acked Write 9 Done Sent Done false
12676 4415 Acked Read 3 Done Sent Done false
12677 4416 Acked Read 3 Done Sent Done false
12678 4417 Acked Flush 0 Done Sent Done false
12679 4418 Acked Read 10 Done Sent Done false
12680 4419 Acked Write 4 Done Sent Done false
12681 4420 Acked Write 10 Done Sent Done false
12682 4421 Acked Flush 0 Done Sent Done false
12683 4422 Acked Read 7 Done Sent Done false
12684 4423 Acked Read 5 Done Sent Done false
12685 4424 Acked Write 3 Done Sent Done false
12686 4425 Acked Flush 0 Done Sent Done false
12687 4426 Acked Read 7 Done Sent Done false
12688 4427 Acked Write 8 Done Sent Done false
12689 4428 Acked Read 7 Done Sent Done false
12690 4429 Acked Read 6 Done Sent Done false
12691 4430 Acked Read 8 Done Sent Done false
12692 4431 Acked Write 2 Done Sent Done false
12693 4432 Acked Write 4 Done Sent Done false
12694 4433 Acked Read 5 Done Sent Done false
12695 4434 Acked Read 8 Done Sent Done false
12696 4435 Acked Read 5 Done Sent Done false
12697 4436 Acked Read 8 Done Sent Done false
12698 4437 Acked Read 5 Done Sent Done false
12699 4438 Acked Read 6 Done Sent Done false
12700 4439 Acked Read 8 Done Sent Done false
12701 4440 Acked Read 10 Done Sent Done false
12702 4441 Acked Read 5 Done Sent Done false
12703 4442 Acked Write 8 Done Sent Done false
12704 4443 Acked Read 7 Done Sent Done false
12705 4444 Acked Read 2 Done Sent Done false
12706 4445 Acked Read 3 Done Sent Done false
12707 4446 Acked Write 9 Done Sent Done false
12708 4447 Acked Read 3 Done Sent Done false
12709 4448 Acked Write 8 Done Sent Done false
12710 4449 Acked Read 3 Done Sent Done false
12711 4450 Acked Write 9 Done Sent Done false
12712 4451 Acked Read 6 Done Sent Done false
12713 4452 Acked Write 3 Done Sent Done false
12714 4453 Acked Write 4 Done Sent Done false
12715 4454 Acked Read 1 Done Sent Done false
12716 4455 Acked Write 10 Done Sent Done false
12717 4456 Acked Flush 0 Done Sent Done false
12718 4457 Acked Write 8 Done Sent Done false
12719 4458 Acked Read 7 Done Sent Done false
12720 4459 Acked Write 4 Done Sent Done false
12721 4460 Acked Flush 0 Done Sent Done false
12722 4461 Acked Write 6 Done Sent Done false
12723 4462 Acked Flush 0 Done Sent Done false
12724 4463 Acked Read 1 Done Sent Done false
12725 4464 Acked Write 1 Done Sent Done false
12726 4465 Acked Write 8 Done Sent Done false
12727 4466 Acked Write 4 Done Sent Done false
12728 4467 Acked Read 10 Done Sent Done false
12729 4468 Acked Write 5 Done Sent Done false
12730 4469 Acked Write 6 Done Sent Done false
12731 4470 Acked Write 9 Done Sent Done false
12732 4471 Acked Write 1 Done Sent Done false
12733 4472 Acked Read 4 Done Sent Done false
12734 4473 Acked Write 6 Done Sent Done false
12735 4474 Acked Write 2 Done Sent Done false
12736 4475 Acked Read 9 Done Sent Done false
12737 4476 Acked Write 4 Done Sent Done false
12738 4477 Acked Read 3 Done Sent Done false
12739 4478 Acked Read 9 Done Sent Done false
12740 4479 Acked Read 4 Done Sent Done false
12741 4480 Acked Read 10 Done Sent Done false
12742 4481 Acked Read 7 Done Sent Done false
12743 4482 Acked Write 7 Done Sent Done false
12744 4483 Acked Read 10 Done Sent Done false
12745 4484 Acked Write 8 Done Sent Done false
12746 4485 Acked Read 6 Done Sent Done false
12747 4486 Acked Read 10 Done Sent Done false
12748 4487 Acked Read 8 Done Sent Done false
12749 4488 Acked Write 9 Done Sent Done false
12750 4489 Acked Read 5 Done Sent Done false
12751 4490 Acked Write 4 Done Sent Done false
12752 4491 Acked Write 6 Done Sent Done false
12753 4492 Acked Read 9 Done Sent Done false
12754 4493 Acked Read 4 Done Sent Done false
12755 4494 Acked Write 8 Done Sent Done false
12756 4495 Acked Flush 0 Done Sent Done false
12757 4496 Acked Flush 0 Done Sent Done false
12758 4497 Acked Read 8 Done Sent Done false
12759 4498 Acked Read 7 Done Sent Done false
12760 4499 Acked Read 10 Done Sent Done false
12761 4500 Acked Write 5 Done Sent Done false
12762 4501 Acked Flush 0 Done Sent Done false
12763 4502 Acked Flush 0 Done Sent Done false
12764 4503 Acked Read 1 Done Sent Done false
12765 4504 Acked Flush 0 Done Sent Done false
12766 4505 Acked Read 3 Done Sent Done false
12767 4506 Acked Flush 0 Done Sent Done false
12768 4507 Acked Read 7 Done Sent Done false
12769 4508 Acked Read 2 Done Sent Done false
12770 4509 Acked Read 2 Done Sent Done false
12771 4510 Acked Write 2 Done Sent Done false
12772 4511 Acked Read 2 Done Sent Done false
12773 4512 Acked Flush 0 Done Sent Done false
12774 4513 Acked Read 6 Done Sent Done false
12775 4514 Acked Read 8 Done Sent Done false
12776 4515 Acked Read 8 Done Sent Done false
12777 4516 Acked Write 10 Done Sent Done false
12778 4517 Acked Read 6 Done Sent Done false
12779 4518 Acked Write 8 Done Sent Done false
12780 4519 Acked Read 10 Done Sent Done false
12781 4520 Acked Write 10 Done Sent Done false
12782 4521 Acked Write 9 Done Sent Done false
12783 4522 Acked Read 2 Done Sent Done false
12784 4523 Acked Read 5 Done Sent Done false
12785 4524 Acked Write 1 Done Sent Done false
12786 4525 Acked Write 3 Done Sent Done false
12787 4526 Acked Flush 0 Done Sent Done false
12788 4527 Acked Read 9 Done Sent Done false
12789 4528 Acked Write 2 Done Sent Done false
12790 4529 Acked Write 2 Done Sent Done false
12791 4530 Acked Read 6 Done Sent Done false
12792 4531 Acked Read 8 Done Sent Done false
12793 4532 Acked Write 4 Done Sent Done false
12794 4533 Acked Read 1 Done Sent Done false
12795 4534 Acked Read 9 Done Sent Done false
12796 4535 Acked Flush 0 Done Sent Done false
12797 4536 Acked Write 7 Done Sent Done false
12798 4537 Acked Write 7 Done Sent Done false
12799 4538 Acked Write 3 Done Sent Done false
12800 4539 Acked Write 3 Done Sent Done false
12801 4540 Acked Write 10 Done Sent Done false
12802 4541 Acked Write 1 Done Sent Done false
12803 4542 Acked Read 2 Done Sent Done false
12804 4543 Acked Write 6 Done Sent Done false
12805 4544 Acked Write 5 Done Sent Done false
12806 4545 Acked Read 7 Done Sent Done false
12807 4546 Acked Flush 0 Done Sent Done false
12808 4547 Acked Read 10 Done Sent Done false
12809 4548 Acked Write 2 Done Sent Done false
12810 4549 Acked Read 6 Done Sent Done false
12811 4550 Acked Read 10 Done Sent Done false
12812 4551 Acked Write 9 Done Sent Done false
12813 4552 Acked Flush 0 Done Sent Done false
12814 4553 Acked Read 1 Done Sent Done false
12815 4554 Acked Flush 0 Done Sent Done false
12816 4555 Acked Read 5 Done Sent Done false
12817 4556 Acked Write 9 Done Sent Done false
12818 4557 Acked Flush 0 Done Sent Done false
12819 4558 Acked Read 2 Done Sent Done false
12820 4559 Acked Write 10 Done Sent Done false
12821 4560 Acked Write 6 Done Sent Done false
12822 4561 Acked Read 3 Done Sent Done false
12823 4562 Acked Write 9 Done Sent Done false
12824 4563 Acked Read 4 Done Sent Done false
12825 4564 Acked Write 8 Done Sent Done false
12826 4565 Acked Read 2 Done Sent Done false
12827 4566 Acked Read 6 Done Sent Done false
12828 4567 Acked Read 1 Done Sent Done false
12829 4568 Acked Read 7 Done Sent Done false
12830 4569 Acked Write 6 Done Sent Done false
12831 4570 Acked Read 9 Done Sent Done false
12832 4571 Acked Write 8 Done Sent Done false
12833 4572 Acked Read 3 Done Sent Done false
12834 4573 Acked Write 5 Done Sent Done false
12835 4574 Acked Flush 0 Done Sent Done false
12836 4575 Acked Read 10 Done Sent Done false
12837 4576 Acked Read 9 Done Sent Done false
12838 4577 Acked Flush 0 Done Sent Done false
12839 4578 Acked Read 2 Done Sent Done false
12840 4579 Acked Write 3 Done Sent Done false
12841 4580 Acked Flush 0 Done Sent Done false
12842 4581 Acked Read 8 Done Sent Done false
12843 4582 Acked Read 10 Done Sent Done false
12844 4583 Acked Read 4 Done Sent Done false
12845 4584 Acked Flush 0 Done Sent Done false
12846 4585 Acked Write 1 Done Sent Done false
12847 4586 Acked Read 4 Done Sent Done false
12848 4587 Acked Write 6 Done Sent Done false
12849 4588 Acked Write 5 Done Sent Done false
12850 4589 Acked Read 5 Done Sent Done false
12851 4590 Acked Write 1 Done Sent Done false
12852 4591 Acked Read 4 Done Sent Done false
12853 4592 Acked Read 7 Done Sent Done false
12854 4593 Acked Read 8 Done Sent Done false
12855 4594 Acked Read 2 Done Sent Done false
12856 4595 Acked Read 10 Done Sent Done false
12857 4596 Acked Write 4 Done Sent Done false
12858 4597 Acked Read 1 Done Sent Done false
12859 4598 Acked Read 6 Done Sent Done false
12860 4599 Acked Read 7 Done Sent Done false
12861 4600 Acked Read 6 Done Sent Done false
12862 4601 Acked Write 4 Done Sent Done false
12863 4602 Acked Read 5 Done Sent Done false
12864 4603 Acked Write 2 Done Sent Done false
12865 4604 Acked Write 5 Done Sent Done false
12866 4605 Acked Write 7 Done Sent Done false
12867 4606 Acked Read 1 Done Sent Done false
12868 4607 Acked Read 9 Done Sent Done false
12869 4608 Acked Read 5 Done Sent Done false
12870 4609 Acked Write 1 Done Sent Done false
12871 4610 Acked Flush 0 Done Sent Done false
12872 4611 Acked Flush 0 Done Sent Done false
12873 4612 Acked Read 3 Done Sent Done false
12874 4613 Acked Read 9 Done Sent Done false
12875 4614 Acked Write 4 Done Sent Done false
12876 4615 Acked Write 4 Done Sent Done false
12877 4616 Acked Flush 0 Done Sent Done false
12878 4617 Acked Read 3 Done Sent Done false
12879 4618 Acked Read 1 Done Sent Done false
12880 4619 Acked Read 5 Done Sent Done false
12881 4620 Acked Write 10 Done Sent Done false
12882 4621 Acked Read 7 Done Sent Done false
12883 4622 Acked Write 2 Done Sent Done false
12884 STATES DS:0 DS:1 DS:2 TOTAL
12885 Sent 0 301 0 301
12886 Done 301 0 301 602
12887 Skipped 0 0 0 0
12888 Error 0 0 0 0
12889 Last Flush: Some(JobId(4616)) Some(JobId(4321)) Some(JobId(4616))
12890 Downstairs last five completed: 4321 4320 4319 4318 4317
12891 Upstairs last five completed: 4622 4621 4620 4619 4618
12892 CLIENT: Up:0 ds:301 act:2
128932025-01-28T21:57:05.574ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
12894 [8820] got start action
12895 [8820] I am going to restart
12896 Starting downstairs at port 8820
12897 Make output file at "/tmp/dsc/downstairs-8820.txt"
12898 [8820][1] reports Starting
12899 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1214)
12900 [8820][1] reports Running
129012025-01-28T21:57:06.861ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
129022025-01-28T21:57:07.578ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
129032025-01-28T21:57:08.578ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
12904 ----------------------------------------------------------------
12905 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12906 States: Active Offline Active
12907 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12908 4322 Acked Write 3 Done Sent Done false
12909 4323 Acked Read 7 Done Sent Done false
12910 4324 Acked Flush 0 Done Sent Done false
12911 4325 Acked Write 9 Done Sent Done false
12912 4326 Acked Read 1 Done Sent Done false
12913 4327 Acked Read 9 Done Sent Done false
12914 4328 Acked Read 7 Done Sent Done false
12915 4329 Acked Read 9 Done Sent Done false
12916 4330 Acked Read 10 Done Sent Done false
12917 4331 Acked Write 8 Done Sent Done false
12918 4332 Acked Write 7 Done Sent Done false
12919 4333 Acked Write 9 Done Sent Done false
12920 4334 Acked Read 2 Done Sent Done false
12921 4335 Acked Read 9 Done Sent Done false
12922 4336 Acked Write 1 Done Sent Done false
12923 4337 Acked Write 7 Done Sent Done false
12924 4338 Acked Read 9 Done Sent Done false
12925 4339 Acked Read 5 Done Sent Done false
12926 4340 Acked Write 3 Done Sent Done false
12927 4341 Acked Write 9 Done Sent Done false
12928 4342 Acked Flush 0 Done Sent Done false
12929 4343 Acked Write 4 Done Sent Done false
12930 4344 Acked Write 9 Done Sent Done false
12931 4345 Acked Write 7 Done Sent Done false
12932 4346 Acked Read 1 Done Sent Done false
12933 4347 Acked Read 6 Done Sent Done false
12934 4348 Acked Flush 0 Done Sent Done false
12935 4349 Acked Write 8 Done Sent Done false
12936 4350 Acked Write 8 Done Sent Done false
12937 4351 Acked Write 4 Done Sent Done false
12938 4352 Acked Write 5 Done Sent Done false
12939 4353 Acked Read 9 Done Sent Done false
12940 4354 Acked Write 9 Done Sent Done false
12941 4355 Acked Write 5 Done Sent Done false
12942 4356 Acked Write 2 Done Sent Done false
12943 4357 Acked Write 5 Done Sent Done false
12944 4358 Acked Read 8 Done Sent Done false
12945 4359 Acked Read 3 Done Sent Done false
12946 4360 Acked Write 5 Done Sent Done false
12947 4361 Acked Read 4 Done Sent Done false
12948 4362 Acked Write 5 Done Sent Done false
12949 4363 Acked Read 10 Done Sent Done false
12950 4364 Acked Write 10 Done Sent Done false
12951 4365 Acked Write 4 Done Sent Done false
12952 4366 Acked Write 5 Done Sent Done false
12953 4367 Acked Read 2 Done Sent Done false
12954 4368 Acked Read 10 Done Sent Done false
12955 4369 Acked Write 4 Done Sent Done false
12956 4370 Acked Flush 0 Done Sent Done false
12957 4371 Acked Write 10 Done Sent Done false
12958 4372 Acked Read 7 Done Sent Done false
12959 4373 Acked Flush 0 Done Sent Done false
12960 4374 Acked Write 3 Done Sent Done false
12961 4375 Acked Read 10 Done Sent Done false
12962 4376 Acked Write 2 Done Sent Done false
12963 4377 Acked Read 5 Done Sent Done false
12964 4378 Acked Write 3 Done Sent Done false
12965 4379 Acked Write 1 Done Sent Done false
12966 4380 Acked Read 6 Done Sent Done false
12967 4381 Acked Write 10 Done Sent Done false
12968 4382 Acked Read 8 Done Sent Done false
12969 4383 Acked Write 10 Done Sent Done false
12970 4384 Acked Read 1 Done Sent Done false
12971 4385 Acked Write 10 Done Sent Done false
12972 4386 Acked Read 10 Done Sent Done false
12973 4387 Acked Read 3 Done Sent Done false
12974 4388 Acked Read 6 Done Sent Done false
12975 4389 Acked Write 4 Done Sent Done false
12976 4390 Acked Read 6 Done Sent Done false
12977 4391 Acked Read 8 Done Sent Done false
12978 4392 Acked Write 2 Done Sent Done false
12979 4393 Acked Read 10 Done Sent Done false
12980 4394 Acked Flush 0 Done Sent Done false
12981 4395 Acked Write 5 Done Sent Done false
12982 4396 Acked Read 10 Done Sent Done false
12983 4397 Acked Write 5 Done Sent Done false
12984 4398 Acked Read 4 Done Sent Done false
12985 4399 Acked Flush 0 Done Sent Done false
12986 4400 Acked Flush 0 Done Sent Done false
12987 4401 Acked Write 8 Done Sent Done false
12988 4402 Acked Write 4 Done Sent Done false
12989 4403 Acked Write 4 Done Sent Done false
12990 4404 Acked Write 5 Done Sent Done false
12991 4405 Acked Write 1 Done Sent Done false
12992 4406 Acked Write 6 Done Sent Done false
12993 4407 Acked Flush 0 Done Sent Done false
12994 4408 Acked Write 4 Done Sent Done false
12995 4409 Acked Read 10 Done Sent Done false
12996 4410 Acked Flush 0 Done Sent Done false
12997 4411 Acked Write 2 Done Sent Done false
12998 4412 Acked Read 4 Done Sent Done false
12999 4413 Acked Read 8 Done Sent Done false
13000 4414 Acked Write 9 Done Sent Done false
13001 4415 Acked Read 3 Done Sent Done false
13002 4416 Acked Read 3 Done Sent Done false
13003 4417 Acked Flush 0 Done Sent Done false
13004 4418 Acked Read 10 Done Sent Done false
13005 4419 Acked Write 4 Done Sent Done false
13006 4420 Acked Write 10 Done Sent Done false
13007 4421 Acked Flush 0 Done Sent Done false
13008 4422 Acked Read 7 Done Sent Done false
13009 4423 Acked Read 5 Done Sent Done false
13010 4424 Acked Write 3 Done Sent Done false
13011 4425 Acked Flush 0 Done Sent Done false
13012 4426 Acked Read 7 Done Sent Done false
13013 4427 Acked Write 8 Done Sent Done false
13014 4428 Acked Read 7 Done Sent Done false
13015 4429 Acked Read 6 Done Sent Done false
13016 4430 Acked Read 8 Done Sent Done false
13017 4431 Acked Write 2 Done Sent Done false
13018 4432 Acked Write 4 Done Sent Done false
13019 4433 Acked Read 5 Done Sent Done false
13020 4434 Acked Read 8 Done Sent Done false
13021 4435 Acked Read 5 Done Sent Done false
13022 4436 Acked Read 8 Done Sent Done false
13023 4437 Acked Read 5 Done Sent Done false
13024 4438 Acked Read 6 Done Sent Done false
13025 4439 Acked Read 8 Done Sent Done false
13026 4440 Acked Read 10 Done Sent Done false
13027 4441 Acked Read 5 Done Sent Done false
13028 4442 Acked Write 8 Done Sent Done false
13029 4443 Acked Read 7 Done Sent Done false
13030 4444 Acked Read 2 Done Sent Done false
13031 4445 Acked Read 3 Done Sent Done false
13032 4446 Acked Write 9 Done Sent Done false
13033 4447 Acked Read 3 Done Sent Done false
13034 4448 Acked Write 8 Done Sent Done false
13035 4449 Acked Read 3 Done Sent Done false
13036 4450 Acked Write 9 Done Sent Done false
13037 4451 Acked Read 6 Done Sent Done false
13038 4452 Acked Write 3 Done Sent Done false
13039 4453 Acked Write 4 Done Sent Done false
13040 4454 Acked Read 1 Done Sent Done false
13041 4455 Acked Write 10 Done Sent Done false
13042 4456 Acked Flush 0 Done Sent Done false
13043 4457 Acked Write 8 Done Sent Done false
13044 4458 Acked Read 7 Done Sent Done false
13045 4459 Acked Write 4 Done Sent Done false
13046 4460 Acked Flush 0 Done Sent Done false
13047 4461 Acked Write 6 Done Sent Done false
13048 4462 Acked Flush 0 Done Sent Done false
13049 4463 Acked Read 1 Done Sent Done false
13050 4464 Acked Write 1 Done Sent Done false
13051 4465 Acked Write 8 Done Sent Done false
13052 4466 Acked Write 4 Done Sent Done false
13053 4467 Acked Read 10 Done Sent Done false
13054 4468 Acked Write 5 Done Sent Done false
13055 4469 Acked Write 6 Done Sent Done false
13056 4470 Acked Write 9 Done Sent Done false
13057 4471 Acked Write 1 Done Sent Done false
13058 4472 Acked Read 4 Done Sent Done false
13059 4473 Acked Write 6 Done Sent Done false
13060 4474 Acked Write 2 Done Sent Done false
13061 4475 Acked Read 9 Done Sent Done false
13062 4476 Acked Write 4 Done Sent Done false
13063 4477 Acked Read 3 Done Sent Done false
13064 4478 Acked Read 9 Done Sent Done false
13065 4479 Acked Read 4 Done Sent Done false
13066 4480 Acked Read 10 Done Sent Done false
13067 4481 Acked Read 7 Done Sent Done false
13068 4482 Acked Write 7 Done Sent Done false
13069 4483 Acked Read 10 Done Sent Done false
13070 4484 Acked Write 8 Done Sent Done false
13071 4485 Acked Read 6 Done Sent Done false
13072 4486 Acked Read 10 Done Sent Done false
13073 4487 Acked Read 8 Done Sent Done false
13074 4488 Acked Write 9 Done Sent Done false
13075 4489 Acked Read 5 Done Sent Done false
13076 4490 Acked Write 4 Done Sent Done false
13077 4491 Acked Write 6 Done Sent Done false
13078 4492 Acked Read 9 Done Sent Done false
13079 4493 Acked Read 4 Done Sent Done false
13080 4494 Acked Write 8 Done Sent Done false
13081 4495 Acked Flush 0 Done Sent Done false
13082 4496 Acked Flush 0 Done Sent Done false
13083 4497 Acked Read 8 Done Sent Done false
13084 4498 Acked Read 7 Done Sent Done false
13085 4499 Acked Read 10 Done Sent Done false
13086 4500 Acked Write 5 Done Sent Done false
13087 4501 Acked Flush 0 Done Sent Done false
13088 4502 Acked Flush 0 Done Sent Done false
13089 4503 Acked Read 1 Done Sent Done false
13090 4504 Acked Flush 0 Done Sent Done false
13091 4505 Acked Read 3 Done Sent Done false
13092 4506 Acked Flush 0 Done Sent Done false
13093 4507 Acked Read 7 Done Sent Done false
13094 4508 Acked Read 2 Done Sent Done false
13095 4509 Acked Read 2 Done Sent Done false
13096 4510 Acked Write 2 Done Sent Done false
13097 4511 Acked Read 2 Done Sent Done false
13098 4512 Acked Flush 0 Done Sent Done false
13099 4513 Acked Read 6 Done Sent Done false
13100 4514 Acked Read 8 Done Sent Done false
13101 4515 Acked Read 8 Done Sent Done false
13102 4516 Acked Write 10 Done Sent Done false
13103 4517 Acked Read 6 Done Sent Done false
13104 4518 Acked Write 8 Done Sent Done false
13105 4519 Acked Read 10 Done Sent Done false
13106 4520 Acked Write 10 Done Sent Done false
13107 4521 Acked Write 9 Done Sent Done false
13108 4522 Acked Read 2 Done Sent Done false
13109 4523 Acked Read 5 Done Sent Done false
13110 4524 Acked Write 1 Done Sent Done false
13111 4525 Acked Write 3 Done Sent Done false
13112 4526 Acked Flush 0 Done Sent Done false
13113 4527 Acked Read 9 Done Sent Done false
13114 4528 Acked Write 2 Done Sent Done false
13115 4529 Acked Write 2 Done Sent Done false
13116 4530 Acked Read 6 Done Sent Done false
13117 4531 Acked Read 8 Done Sent Done false
13118 4532 Acked Write 4 Done Sent Done false
13119 4533 Acked Read 1 Done Sent Done false
13120 4534 Acked Read 9 Done Sent Done false
13121 4535 Acked Flush 0 Done Sent Done false
13122 4536 Acked Write 7 Done Sent Done false
13123 4537 Acked Write 7 Done Sent Done false
13124 4538 Acked Write 3 Done Sent Done false
13125 4539 Acked Write 3 Done Sent Done false
13126 4540 Acked Write 10 Done Sent Done false
13127 4541 Acked Write 1 Done Sent Done false
13128 4542 Acked Read 2 Done Sent Done false
13129 4543 Acked Write 6 Done Sent Done false
13130 4544 Acked Write 5 Done Sent Done false
13131 4545 Acked Read 7 Done Sent Done false
13132 4546 Acked Flush 0 Done Sent Done false
13133 4547 Acked Read 10 Done Sent Done false
13134 4548 Acked Write 2 Done Sent Done false
13135 4549 Acked Read 6 Done Sent Done false
13136 4550 Acked Read 10 Done Sent Done false
13137 4551 Acked Write 9 Done Sent Done false
13138 4552 Acked Flush 0 Done Sent Done false
13139 4553 Acked Read 1 Done Sent Done false
13140 4554 Acked Flush 0 Done Sent Done false
13141 4555 Acked Read 5 Done Sent Done false
13142 4556 Acked Write 9 Done Sent Done false
13143 4557 Acked Flush 0 Done Sent Done false
13144 4558 Acked Read 2 Done Sent Done false
13145 4559 Acked Write 10 Done Sent Done false
13146 4560 Acked Write 6 Done Sent Done false
13147 4561 Acked Read 3 Done Sent Done false
13148 4562 Acked Write 9 Done Sent Done false
13149 4563 Acked Read 4 Done Sent Done false
13150 4564 Acked Write 8 Done Sent Done false
13151 4565 Acked Read 2 Done Sent Done false
13152 4566 Acked Read 6 Done Sent Done false
13153 4567 Acked Read 1 Done Sent Done false
13154 4568 Acked Read 7 Done Sent Done false
13155 4569 Acked Write 6 Done Sent Done false
13156 4570 Acked Read 9 Done Sent Done false
13157 4571 Acked Write 8 Done Sent Done false
13158 4572 Acked Read 3 Done Sent Done false
13159 4573 Acked Write 5 Done Sent Done false
13160 4574 Acked Flush 0 Done Sent Done false
13161 4575 Acked Read 10 Done Sent Done false
13162 4576 Acked Read 9 Done Sent Done false
13163 4577 Acked Flush 0 Done Sent Done false
13164 4578 Acked Read 2 Done Sent Done false
13165 4579 Acked Write 3 Done Sent Done false
13166 4580 Acked Flush 0 Done Sent Done false
13167 4581 Acked Read 8 Done Sent Done false
13168 4582 Acked Read 10 Done Sent Done false
13169 4583 Acked Read 4 Done Sent Done false
13170 4584 Acked Flush 0 Done Sent Done false
13171 4585 Acked Write 1 Done Sent Done false
13172 4586 Acked Read 4 Done Sent Done false
13173 4587 Acked Write 6 Done Sent Done false
13174 4588 Acked Write 5 Done Sent Done false
13175 4589 Acked Read 5 Done Sent Done false
13176 4590 Acked Write 1 Done Sent Done false
13177 4591 Acked Read 4 Done Sent Done false
13178 4592 Acked Read 7 Done Sent Done false
13179 4593 Acked Read 8 Done Sent Done false
13180 4594 Acked Read 2 Done Sent Done false
13181 4595 Acked Read 10 Done Sent Done false
13182 4596 Acked Write 4 Done Sent Done false
13183 4597 Acked Read 1 Done Sent Done false
13184 4598 Acked Read 6 Done Sent Done false
13185 4599 Acked Read 7 Done Sent Done false
13186 4600 Acked Read 6 Done Sent Done false
13187 4601 Acked Write 4 Done Sent Done false
13188 4602 Acked Read 5 Done Sent Done false
13189 4603 Acked Write 2 Done Sent Done false
13190 4604 Acked Write 5 Done Sent Done false
13191 4605 Acked Write 7 Done Sent Done false
13192 4606 Acked Read 1 Done Sent Done false
13193 4607 Acked Read 9 Done Sent Done false
13194 4608 Acked Read 5 Done Sent Done false
13195 4609 Acked Write 1 Done Sent Done false
13196 4610 Acked Flush 0 Done Sent Done false
13197 4611 Acked Flush 0 Done Sent Done false
13198 4612 Acked Read 3 Done Sent Done false
13199 4613 Acked Read 9 Done Sent Done false
13200 4614 Acked Write 4 Done Sent Done false
13201 4615 Acked Write 4 Done Sent Done false
13202 4616 Acked Flush 0 Done Sent Done false
13203 4617 Acked Read 3 Done Sent Done false
13204 4618 Acked Read 1 Done Sent Done false
13205 4619 Acked Read 5 Done Sent Done false
13206 4620 Acked Write 10 Done Sent Done false
13207 4621 Acked Read 7 Done Sent Done false
13208 4622 Acked Write 2 Done Sent Done false
13209 STATES DS:0 DS:1 DS:2 TOTAL
13210 Sent 0 301 0 301
13211 Done 301 0 301 602
13212 Skipped 0 0 0 0
13213 Error 0 0 0 0
13214 Last Flush: Some(JobId(4616)) Some(JobId(4321)) Some(JobId(4616))
13215 Downstairs last five completed: 4321 4320 4319 4318 4317
13216 Upstairs last five completed: 4622 4621 4620 4619 4618
13217 CLIENT: Up:0 ds:301 act:2
132182025-01-28T21:57:09.579ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
132192025-01-28T21:57:10.580ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
132202025-01-28T21:57:10.612ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
132212025-01-28T21:57:10.612ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59882) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
132222025-01-28T21:57:10.613ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
132232025-01-28T21:57:10.613ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
132242025-01-28T21:57:10.613ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
132252025-01-28T21:57:10.613ZINFOcrucible: send last flush ID to this DS: Some(JobId(4321)) = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
132262025-01-28T21:57:10.613ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
132272025-01-28T21:57:10.613ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4321)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
132282025-01-28T21:57:10.614ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4321)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
13229 ----------------------------------------------------------------
13230 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13231 Downstairs last five completed: 4623 4622 4621 4620 4619
13232 Upstairs last five completed: 4623 4622 4621 4620 4619
13233 CLIENT: Up:0 ds:0 act:3
13234 Replay: All jobs finished
13235 Main task has work to do, go find it
13236 got dsc Stop(2)
13237 stop 2
13238 Jan 28 21:57:12.862 INFO request completed, latency_us: 376, response_code: 204, uri: /stop/cid/2, [8830] Got stop action so:false kr:false
13239 method: POST, req_id: 6cbfc2e2-59ad-4339-a5a7-f17d7453c59e, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
13240 Jan 28 21:57:12.863 INFO request completed, latency_us: 136{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:57:12.863723195Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
13241 , response_code: 200, uri: /state/cid/2, method: GET, req_id: d10fcf76-c6d7-4d38-992a-16ef067f5a87, remote_addr: 127.0.0.1:36665, local_addr: 127.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:57:12.863786241Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
13242 0.0.1:9998
132432025-01-28T21:57:12.863ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
132442025-01-28T21:57:12.863ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
13245 [8830] Exited with: ExitStatus(unix_wait_status(9))
13246 [8830][2] reports Exit
13247 Jan 28 21:57:16.866 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/2, method: GET, req_id: 430995b3-3b6e-467f-b088-9f21cc108992, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
13248 001/300 Write block 4748 len 32768 data: 4 3 3 3 3 4 4 3
13249 002/300 Write block 753 len 32768 data: 3 4 5 4 3 5 5 5
13250 003/300 Flush
13251 004/300 Write block 4544 len 8192 data: 3 4
13252 005/300 Flush
13253 006/300 Write block 2827 len 16384 data: 5 5 4 4
13254 007/300 Read block 4091 len 28672
13255 008/300 Write block 1409 len 24576 data: 5 6 5 6 5 5
13256 009/300 Read block 1748 len 16384
13257 010/300 Read block 2398 len 8192
13258 011/300 Read block 3633 len 32768
13259 012/300 Read block 3575 len 28672
13260 013/300 Read block 2281 len 20480
13261 014/300 Read block 2159 len 8192
13262 015/300 Write block 2325 len 4096 data: 4
13263 016/300 Read block 1062 len 4096
13264 017/300 Read block 3336 len 16384
13265 018/300 Read block 1386 len 12288
13266 019/300 Read block 4660 len 16384
13267 020/300 Read block 2200 len 32768
13268 021/300 Write block 4839 len 32768 data: 3 4 5 5 3 3 5 5
13269 022/300 Read block 2473 len 28672
13270 023/300 Flush
13271 024/300 Read block 1072 len 4096
13272 025/300 Flush
13273 026/300 Read block 59 len 20480
13274 027/300 Write block 1928 len 36864 data: 2 2 2 2 3 3 3 4 4
13275 028/300 Write block 3328 len 40960 data: 5 4 5 4 3 3 3 3 3 4
13276 029/300 Write block 1576 len 20480 data: 3 2 2 3 4
13277 030/300 Write block 2082 len 12288 data: 6 5 5
13278 031/300 Read block 1049 len 24576
13279 032/300 Read block 702 len 36864
13280 033/300 Read block 4422 len 24576
13281 034/300 Read block 4905 len 8192
13282 035/300 Write block 4160 len 32768 data: 5 5 5 6 7 7 6 6
13283 036/300 Write block 4679 len 20480 data: 5 5 5 5 5
13284 037/300 Write block 1312 len 28672 data: 4 4 4 4 4 2 2
13285 038/300 Read block 2109 len 12288
13286 039/300 Write block 2814 len 36864 data: 2 2 2 2 3 3 3 3 2
13287 040/300 Read block 2894 len 12288
13288 041/300 Read block 3401 len 16384
13289 042/300 Write block 1021 len 28672 data: 4 4 4 4 3 3 3
13290 043/300 Write block 1294 len 32768 data: 3 4 5 5 6 5 5 4
13291 044/300 Write block 3682 len 28672 data: 2 2 2 2 2 2 3
13292 045/300 Read block 608 len 32768
13293 046/300 Read block 458 len 20480
13294 047/300 Read block 914 len 28672
13295 048/300 Read block 4248 len 28672
13296 049/300 Write block 2370 len 28672 data: 5 4 3 3 4 2 3
13297 050/300 Write block 3551 len 12288 data: 6 6 5
13298 051/300 Write block 3339 len 32768 data: 3 3 3 3 3 2 2 2
13299 052/300 Read block 712 len 8192
13300 053/300 Read block 4995 len 4096
13301 054/300 Write block 1248 len 12288 data: 2 2 2
13302 055/300 Read block 2100 len 36864
13303 056/300 Read block 3730 len 20480
13304 057/300 Write block 3763 len 40960 data: 5 5 4 4 5 4 4 4 4 3
13305 058/300 Read block 624 len 24576
13306 059/300 Read block 2972 len 28672
13307 060/300 Read block 124 len 24576
13308 061/300 Read block 3236 len 28672
13309 062/300 Read block 3877 len 8192
13310 063/300 Read block 581 len 28672
13311 064/300 Flush
13312 065/300 Flush
13313 066/300 Read block 1841 len 8192
13314 067/300 Read block 912 len 20480
13315 068/300 Write block 1789 len 16384 data: 2 2 2 2
13316 069/300 Read block 3552 len 8192
13317 070/300 Read block 17 len 28672
13318 071/300 Read block 4003 len 36864
13319 072/300 Write block 1965 len 4096 data: 4
13320 073/300 Read block 3295 len 8192
13321 074/300 Write block 376 len 20480 data: 5 5 3 4 3
13322 075/300 Read block 4750 len 4096
13323 076/300 Flush
13324 077/300 Write block 2510 len 36864 data: 4 3 3 4 4 3 3 2 3
13325 078/300 Read block 4377 len 32768
13326 079/300 Read block 777 len 8192
13327 080/300 Read block 2720 len 8192
13328 081/300 Write block 1840 len 28672 data: 2 3 3 4 4 4 4
13329 082/300 Write block 2529 len 8192 data: 3 4
13330 083/300 Read block 2188 len 28672
13331 084/300 Read block 2887 len 8192
13332 085/300 Flush
13333 086/300 Read block 927 len 28672
13334 087/300 Read block 398 len 16384
13335 088/300 Write block 3794 len 12288 data: 4 3 3
13336 089/300 Read block 3 len 40960
13337 090/300 Write block 1949 len 8192 data: 3 3
13338 091/300 Write block 3496 len 4096 data: 3
13339 092/300 Write block 1636 len 8192 data: 5 3
13340 093/300 Write block 1228 len 28672 data: 2 2 3 3 3 2 2
13341 094/300 Write block 2815 len 12288 data: 3 3 3
13342 095/300 Write block 458 len 40960 data: 4 5 6 5 5 3 3 3 3 2
13343 096/300 Read block 4652 len 12288
13344 097/300 Read block 3868 len 32768
13345 098/300 Write block 2445 len 16384 data: 3 3 3 3
13346 099/300 Read block 4677 len 20480
13347 100/300 Write block 1327 len 4096 data: 3
13348 101/300 Read block 3323 len 28672
13349 102/300 Read block 2068 len 20480
13350 103/300 Read block 4052 len 8192
13351 104/300 Read block 2191 len 4096
13352 105/300 Read block 2975 len 16384
13353 106/300 Write block 4357 len 28672 data: 5 5 5 5 4 4 4
13354 107/300 Read block 2486 len 8192
13355 108/300 Read block 1696 len 36864
13356 109/300 Flush
13357 110/300 Read block 994 len 28672
13358 111/300 Read block 1741 len 8192
13359 112/300 Write block 4927 len 8192 data: 5 4
13360 113/300 Flush
13361 114/300 Read block 2274 len 4096
13362 115/300 Read block 1274 len 36864
13363 116/300 Flush
13364 117/300 Write block 4973 len 28672 data: 2 2 3 3 3 3 3
13365 118/300 Write block 4310 len 12288 data: 4 4 4
13366 119/300 Write block 2595 len 8192 data: 5 3
13367 120/300 Read block 2674 len 12288
13368 121/300 Read block 13 len 8192
13369 122/300 Read block 1075 len 12288
13370 123/300 Write block 729 len 24576 data: 4 4 4 4 4 4
13371 124/300 Write block 4930 len 24576 data: 4 3 3 4 4 4
13372 125/300 Read block 3885 len 32768
13373 126/300 Flush
13374 127/300 Read block 668 len 20480
13375 128/300 Read block 547 len 20480
13376 129/300 Write block 2755 len 28672 data: 5 5 5 5 5 5 4
13377 130/300 Read block 3531 len 36864
13378 131/300 Flush
13379 132/300 Read block 3225 len 40960
13380 133/300 Write block 981 len 16384 data: 4 4 3 3
13381 134/300 Write block 2146 len 40960 data: 5 5 4 5 5 6 6 6 6 6
13382 135/300 Write block 2316 len 20480 data: 3 3 3 3 3
13383 136/300 Read block 3921 len 28672
13384 137/300 Write block 4430 len 8192 data: 3 2
13385 138/300 Write block 820 len 16384 data: 2 3 3 2
13386 139/300 Read block 2962 len 36864
13387 140/300 Read block 4029 len 12288
13388 141/300 Write block 1336 len 36864 data: 4 3 3 3 2 2 3 3 6
13389 142/300 Read block 2669 len 32768
13390 143/300 Write block 528 len 20480 data: 3 3 3 3 3
13391 144/300 Read block 3084 len 40960
13392 145/300 Write block 1243 len 16384 data: 3 3 3 2
13393 146/300 Write block 2075 len 28672 data: 3 4 3 4 3 4 6
13394 147/300 Read block 3603 len 28672
13395 148/300 Read block 2121 len 8192
13396 149/300 Read block 2926 len 28672
13397 150/300 Read block 2741 len 16384
13398 151/300 Flush
13399 152/300 Write block 4101 len 12288 data: 4 4 4
13400 153/300 Write block 876 len 32768 data: 3 3 3 2 3 3 3 3
13401 154/300 Read block 2066 len 40960
13402 155/300 Read block 187 len 12288
13403 156/300 Flush
13404 157/300 Read block 1267 len 12288
13405 158/300 Flush
13406 159/300 Write block 345 len 36864 data: 5 5 5 4 6 6 6 3 4
13407 160/300 Write block 237 len 28672 data: 5 6 4 5 5 5 5
13408 161/300 Read block 4850 len 24576
13409 162/300 Write block 1119 len 8192 data: 5 5
13410 163/300 Read block 1780 len 28672
13411 164/300 Read block 4157 len 16384
13412 165/300 Write block 2991 len 4096 data: 5
13413 166/300 Read block 4646 len 20480
13414 167/300 Write block 2820 len 8192 data: 4 4
13415 168/300 Read block 1831 len 28672
13416 169/300 Read block 1854 len 24576
13417 170/300 Flush
13418 171/300 Read block 529 len 4096
13419 172/300 Write block 2350 len 32768 data: 4 3 3 3 3 3 3 2
13420 173/300 Read block 557 len 36864
13421 174/300 Read block 4865 len 12288
13422 175/300 Write block 3925 len 4096 data: 5
13423 176/300 Write block 2614 len 16384 data: 4 6 6 7
13424 177/300 Write block 4959 len 32768 data: 3 3 4 3 4 4 4 4
13425 178/300 Read block 1421 len 4096
13426 179/300 Write block 514 len 4096 data: 2
13427 180/300 Write block 710 len 28672 data: 2 2 2 4 3 3 4
13428 181/300 Flush
13429 182/300 Write block 3153 len 16384 data: 2 2 2 2
13430 183/300 Read block 548 len 12288
13431 184/300 Read block 3911 len 20480
13432 185/300 Write block 3231 len 8192 data: 2 2
13433 186/300 Read block 1483 len 12288
13434 187/300 Write block 3393 len 24576 data: 6 5 5 5 5 4
13435 188/300 Write block 3650 len 20480 data: 4 4 3 2 2
13436 189/300 Read block 4250 len 4096
13437 190/300 Write block 3510 len 12288 data: 2 2 2
13438 191/300 Read block 1918 len 36864
13439 192/300 Read block 3936 len 20480
13440 193/300 Read block 1706 len 4096
13441 194/300 Flush
13442 195/300 Read block 4001 len 16384
13443 196/300 Read block 1998 len 28672
13444 197/300 Write block 4501 len 28672 data: 3 3 3 3 3 3 3
13445 198/300 Write block 40 len 16384 data: 3 4 4 5
13446 199/300 Flush
13447 200/300 Read block 3234 len 12288
13448 201/300 Write block 1251 len 28672 data: 3 3 3 4 4 4 3
13449 202/300 Write block 4484 len 36864 data: 7 6 8 5 5 5 6 4 4
13450 203/300 Read block 3353 len 24576
13451 204/300 Read block 2838 len 4096
13452 205/300 Write block 1270 len 20480 data: 3 3 3 3 3
13453 206/300 Read block 3248 len 32768
13454 207/300 Read block 2149 len 8192
13455 208/300 Write block 327 len 28672 data: 5 5 4 3 2 2 2
13456 209/300 Flush
13457 210/300 Read block 2735 len 8192
13458 211/300 Read block 372 len 4096
13459 212/300 Read block 4384 len 12288
13460 213/300 Read block 818 len 20480
13461 214/300 Write block 282 len 36864 data: 3 3 3 3 2 2 2 2 2
13462 215/300 Read block 13 len 32768
13463 216/300 Write block 2789 len 40960 data: 4 4 3 3 3 3 3 3 3 4
13464 217/300 Write block 1618 len 36864 data: 3 3 3 2 2 2 2 2 3
13465 218/300 Flush
13466 219/300 Write block 3245 len 24576 data: 4 4 3 2 4 4
13467 220/300 Write block 2660 len 20480 data: 7 7 6 5 4
13468 221/300 Read block 2803 len 20480
13469 222/300 Read block 24 len 40960
13470 223/300 Write block 2043 len 40960 data: 2 2 2 2 2 3 3 2 2 3
13471 224/300 Read block 522 len 4096
13472 225/300 Read block 4454 len 24576
13473 226/300 Write block 2291 len 36864 data: 5 5 5 3 3 3 3 2 2
13474 227/300 Write block 3981 len 20480 data: 5 5 5 5 3
13475 228/300 Read block 1910 len 4096
13476 229/300 Flush
13477 230/300 Read block 1708 len 16384
13478 231/300 Read block 2685 len 8192
13479 232/300 Read block 2046 len 12288
13480 233/300 Read block 286 len 40960
13481 234/300 Write block 2190 len 16384 data: 2 2 2 2
13482 235/300 Write block 1233 len 24576 data: 3 3 2 2 2 2
13483 236/300 Write block 2793 len 28672 data: 4 4 4 4 4 5 3
13484 237/300 Read block 3689 len 40960
13485 238/300 Read block 3321 len 8192
13486 239/300 Read block 1954 len 28672
13487 240/300 Flush
13488 241/300 Write block 2950 len 12288 data: 3 2 2
13489 242/300 Flush
13490 243/300 Write block 1620 len 40960 data: 4 3 3 3 3 3 4 3 4 5
13491 244/300 Write block 3937 len 4096 data: 3
13492 245/300 Read block 292 len 32768
13493 246/300 Read block 3961 len 20480
13494 247/300 Read block 4216 len 8192
13495 248/300 Write block 4801 len 40960 data: 3 3 3 5 5 5 5 2 2 2
13496 249/300 Read block 3801 len 36864
13497 250/300 Write block 4576 len 32768 data: 3 3 3 4 4 4 4 3
13498 251/300 Read block 1956 len 36864
13499 252/300 Read block 3798 len 36864
13500 253/300 Write block 687 len 8192 data: 4 5
13501 254/300 Write block 1425 len 36864 data: 4 4 2 2 2 3 3 3 3
13502 255/300 Read block 3686 len 8192
13503 256/300 Read block 3948 len 28672
13504 257/300 Read block 2692 len 12288
13505 258/300 Read block 1085 len 36864
13506 259/300 Read block 2521 len 16384
13507 260/300 Read block 2565 len 4096
13508 261/300 Write block 1140 len 24576 data: 3 3 3 3 3 3
13509 262/300 Write block 2173 len 20480 data: 3 3 3 2 2
13510 263/300 Write block 1573 len 32768 data: 2 3 3 4 3 3 4 5
13511 264/300 Write block 1324 len 4096 data: 3
13512 265/300 Read block 1196 len 40960
13513 266/300 Read block 4788 len 24576
13514 267/300 Read block 600 len 32768
13515 268/300 Write block 476 len 12288 data: 7 7 4
13516 269/300 Read block 4695 len 4096
13517 270/300 Read block 4710 len 36864
13518 271/300 Read block 1180 len 20480
13519 272/300 Read block 3 len 16384
13520 273/300 Read block 2178 len 36864
13521 274/300 Write block 3619 len 8192 data: 3 3
13522 275/300 Read block 1700 len 32768
13523 276/300 Write block 440 len 24576 data: 2 3 3 4 3 3
13524 277/300 Write block 784 len 24576 data: 4 4 5 3 4 4
13525 278/300 Write block 3894 len 4096 data: 2
13526 279/300 Flush
13527 280/300 Write block 4230 len 28672 data: 4 4 4 3 2 3 3
13528 281/300 Read block 4902 len 24576
13529 282/300 Read block 2998 len 4096
13530 283/300 Write block 4621 len 32768 data: 4 3 2 2 2 2 4 4
13531 284/300 Flush
13532 285/300 Read block 1124 len 24576
13533 286/300 Read block 3902 len 8192
13534 287/300 Read block 273 len 20480
13535 288/300 Flush
13536 289/300 Write block 3899 len 20480 data: 4 3 3 3 2
13537 290/300 Read block 3425 len 24576
13538 291/300 Read block 224 len 4096
13539 292/300 Read block 2349 len 16384
13540 293/300 Write block 1614 len 32768 data: 2 3 3 3 4 4 5 4
13541 294/300 Read block 830 len 20480
13542 295/300 Read block 1563 len 36864
13543 296/300 Write block 1770 len 24576 data: 3 3 3 3 4 5
13544 297/300 Write block 4948 len 12288 data: 3 4 5
13545 298/300 Write block 2409 len 12288 data: 4 4 5
13546 299/300 Write block 250 len 36864 data: 3 3 3 2 2 2 2 2 2
13547 300/300 Read block 3871 len 36864
13548 301/300 Read block 458 len 32768
13549 Main task has work to do, go find it
13550 got dsc Start(2)
13551 start 2
13552 Jan 28 21:57:17.107 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/2, method: POST, req_id: 76ac2cb9-5104-48bc-83f7-d51994a0657b, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
13553 Replay: started 2, returned:Ok(())
13554 ----------------------------------------------------------------
13555 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13556 States: Active Active Offline
13557 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13558 4624 Acked Write 8 Done Done Sent false
13559 4625 Acked Write 8 Done Done Sent false
13560 4626 Acked Flush 0 Done Done Sent false
13561 4627 Acked Write 2 Done Done Sent false
13562 4628 Acked Flush 0 Done Done Sent false
13563 4629 Acked Write 4 Done Done Sent false
13564 4630 Acked Read 7 Done Done Sent false
13565 4631 Acked Write 6 Done Done Sent false
13566 4632 Acked Read 4 Done Done Sent false
13567 4633 Acked Read 2 Done Done Sent false
13568 4634 Acked Read 8 Done Done Sent false
13569 4635 Acked Read 7 Done Done Sent false
13570 4636 Acked Read 5 Done Done Sent false
13571 4637 Acked Read 2 Done Done Sent false
13572 4638 Acked Write 1 Done Done Sent false
13573 4639 Acked Read 1 Done Done Sent false
13574 4640 Acked Read 4 Done Done Sent false
13575 4641 Acked Read 3 Done Done Sent false
13576 4642 Acked Read 4 Done Done Sent false
13577 4643 Acked Read 8 Done Done Sent false
13578 4644 Acked Write 8 Done Done Sent false
13579 4645 Acked Read 7 Done Done Sent false
13580 4646 Acked Flush 0 Done Done Sent false
13581 4647 Acked Read 1 Done Done Sent false
13582 4648 Acked Flush 0 Done Done Sent false
13583 4649 Acked Read 5 Done Done Sent false
13584 4650 Acked Write 9 Done Done Sent false
13585 4651 Acked Write 10 Done Done Sent false
13586 4652 Acked Write 5 Done Done Sent false
13587 4653 Acked Write 3 Done Done Sent false
13588 4654 Acked Read 6 Done Done Sent false
13589 4655 Acked Read 9 Done Done Sent false
13590 4656 Acked Read 6 Done Done Sent false
13591 4657 Acked Read 2 Done Done Sent false
13592 4658 Acked Write 8 Done Done Sent false
13593 4659 Acked Write 5 Done Done Sent false
13594 4660 Acked Write 7 Done Done Sent false
13595 4661 Acked Read 3 Done Done Sent false
13596 4662 Acked Write 9 Done Done Sent false
13597 4663 Acked Read 3 Done Done Sent false
13598 4664 Acked Read 4 Done Done Sent false
13599 4665 Acked Write 7 Done Done Sent false
13600 4666 Acked Write 8 Done Done Sent false
13601 4667 Acked Write 7 Done Done Sent false
13602 4668 Acked Read 8 Done Done Sent false
13603 4669 Acked Read 5 Done Done Sent false
13604 4670 Acked Read 7 Done Done Sent false
13605 4671 Acked Read 7 Done Done Sent false
13606 4672 Acked Write 7 Done Done Sent false
13607 4673 Acked Write 3 Done Done Sent false
13608 4674 Acked Write 8 Done Done Sent false
13609 4675 Acked Read 2 Done Done Sent false
13610 4676 Acked Read 1 Done Done Sent false
13611 4677 Acked Write 3 Done Done Sent false
13612 4678 Acked Read 9 Done Done Sent false
13613 4679 Acked Read 5 Done Done Sent false
13614 4680 Acked Write 10 Done Done Sent false
13615 4681 Acked Read 6 Done Done Sent false
13616 4682 Acked Read 7 Done Done Sent false
13617 4683 Acked Read 6 Done Done Sent false
13618 4684 Acked Read 7 Done Done Sent false
13619 4685 Acked Read 2 Done Done Sent false
13620 4686 Acked Read 7 Done Done Sent false
13621 4687 Acked Flush 0 Done Done Sent false
13622 4688 Acked Flush 0 Done Done Sent false
13623 4689 Acked Read 2 Done Done Sent false
13624 4690 Acked Read 5 Done Done Sent false
13625 4691 Acked Write 4 Done Done Sent false
13626 4692 Acked Read 2 Done Done Sent false
13627 4693 Acked Read 7 Done Done Sent false
13628 4694 Acked Read 9 Done Done Sent false
13629 4695 Acked Write 1 Done Done Sent false
13630 4696 Acked Read 2 Done Done Sent false
13631 4697 Acked Write 5 Done Done Sent false
13632 4698 Acked Read 1 Done Done Sent false
13633 4699 Acked Flush 0 Done Done Sent false
13634 4700 Acked Write 9 Done Done Sent false
13635 4701 Acked Read 8 Done Done Sent false
13636 4702 Acked Read 2 Done Done Sent false
13637 4703 Acked Read 2 Done Done Sent false
13638 4704 Acked Write 7 Done Done Sent false
13639 4705 Acked Write 2 Done Done Sent false
13640 4706 Acked Read 7 Done Done Sent false
13641 4707 Acked Read 2 Done Done Sent false
13642 4708 Acked Flush 0 Done Done Sent false
13643 4709 Acked Read 7 Done Done Sent false
13644 4710 Acked Read 4 Done Done Sent false
13645 4711 Acked Write 3 Done Done Sent false
13646 4712 Acked Read 10 Done Done Sent false
13647 4713 Acked Write 2 Done Done Sent false
13648 4714 Acked Write 1 Done Done Sent false
13649 4715 Acked Write 2 Done Done Sent false
13650 4716 Acked Write 7 Done Done Sent false
13651 4717 Acked Write 3 Done Done Sent false
13652 4718 Acked Write 10 Done Done Sent false
13653 4719 Acked Read 3 Done Done Sent false
13654 4720 Acked Read 8 Done Done Sent false
13655 4721 Acked Write 4 Done Done Sent false
13656 4722 Acked Read 5 Done Done Sent false
13657 4723 Acked Write 1 Done Done Sent false
13658 4724 Acked Read 7 Done Done Sent false
13659 4725 Acked Read 5 Done Done Sent false
13660 4726 Acked Read 2 Done Done Sent false
13661 4727 Acked Read 1 Done Done Sent false
13662 4728 Acked Read 4 Done Done Sent false
13663 4729 Acked Write 7 Done Done Sent false
13664 4730 Acked Read 2 Done Done Sent false
13665 4731 Acked Read 9 Done Done Sent false
13666 4732 Acked Flush 0 Done Done Sent false
13667 4733 Acked Read 7 Done Done Sent false
13668 4734 Acked Read 2 Done Done Sent false
13669 4735 Acked Write 2 Done Done Sent false
13670 4736 Acked Flush 0 Done Done Sent false
13671 4737 Acked Read 1 Done Done Sent false
13672 4738 Acked Read 9 Done Done Sent false
13673 4739 Acked Flush 0 Done Done Sent false
13674 4740 Acked Write 7 Done Done Sent false
13675 4741 Acked Write 3 Done Done Sent false
13676 4742 Acked Write 2 Done Done Sent false
13677 4743 Acked Read 3 Done Done Sent false
13678 4744 Acked Read 2 Done Done Sent false
13679 4745 Acked Read 3 Done Done Sent false
13680 4746 Acked Write 6 Done Done Sent false
13681 4747 Acked Write 6 Done Done Sent false
13682 4748 Acked Read 8 Done Done Sent false
13683 4749 Acked Flush 0 Done Done Sent false
13684 4750 Acked Read 5 Done Done Sent false
13685 4751 Acked Read 5 Done Done Sent false
13686 4752 Acked Write 7 Done Done Sent false
13687 4753 Acked Read 9 Done Done Sent false
13688 4754 Acked Flush 0 Done Done Sent false
13689 4755 Acked Read 10 Done Done Sent false
13690 4756 Acked Write 4 Done Done Sent false
13691 4757 Acked Write 10 Done Done Sent false
13692 4758 Acked Write 5 Done Done Sent false
13693 4759 Acked Read 7 Done Done Sent false
13694 4760 Acked Write 2 Done Done Sent false
13695 4761 Acked Write 4 Done Done Sent false
13696 4762 Acked Read 9 Done Done Sent false
13697 4763 Acked Read 3 Done Done Sent false
13698 4764 Acked Write 9 Done Done Sent false
13699 4765 Acked Read 8 Done Done Sent false
13700 4766 Acked Write 5 Done Done Sent false
13701 4767 Acked Read 10 Done Done Sent false
13702 4768 Acked Write 4 Done Done Sent false
13703 4769 Acked Write 7 Done Done Sent false
13704 4770 Acked Read 7 Done Done Sent false
13705 4771 Acked Read 2 Done Done Sent false
13706 4772 Acked Read 7 Done Done Sent false
13707 4773 Acked Read 4 Done Done Sent false
13708 4774 Acked Flush 0 Done Done Sent false
13709 4775 Acked Write 3 Done Done Sent false
13710 4776 Acked Write 8 Done Done Sent false
13711 4777 Acked Read 10 Done Done Sent false
13712 4778 Acked Read 3 Done Done Sent false
13713 4779 Acked Flush 0 Done Done Sent false
13714 4780 Acked Read 3 Done Done Sent false
13715 4781 Acked Flush 0 Done Done Sent false
13716 4782 Acked Write 9 Done Done Sent false
13717 4783 Acked Write 7 Done Done Sent false
13718 4784 Acked Read 6 Done Done Sent false
13719 4785 Acked Write 2 Done Done Sent false
13720 4786 Acked Read 7 Done Done Sent false
13721 4787 Acked Read 4 Done Done Sent false
13722 4788 Acked Write 1 Done Done Sent false
13723 4789 Acked Read 5 Done Done Sent false
13724 4790 Acked Write 2 Done Done Sent false
13725 4791 Acked Read 7 Done Done Sent false
13726 4792 Acked Read 6 Done Done Sent false
13727 4793 Acked Flush 0 Done Done Sent false
13728 4794 Acked Read 1 Done Done Sent false
13729 4795 Acked Write 8 Done Done Sent false
13730 4796 Acked Read 9 Done Done Sent false
13731 4797 Acked Read 3 Done Done Sent false
13732 4798 Acked Write 1 Done Done Sent false
13733 4799 Acked Write 4 Done Done Sent false
13734 4800 Acked Write 8 Done Done Sent false
13735 4801 Acked Read 1 Done Done Sent false
13736 4802 Acked Write 1 Done Done Sent false
13737 4803 Acked Write 7 Done Done Sent false
13738 4804 Acked Flush 0 Done Done Sent false
13739 4805 Acked Write 4 Done Done Sent false
13740 4806 Acked Read 3 Done Done Sent false
13741 4807 Acked Read 5 Done Done Sent false
13742 4808 Acked Write 2 Done Done Sent false
13743 4809 Acked Read 3 Done Done Sent false
13744 4810 Acked Write 6 Done Done Sent false
13745 4811 Acked Write 5 Done Done Sent false
13746 4812 Acked Read 1 Done Done Sent false
13747 4813 Acked Write 3 Done Done Sent false
13748 4814 Acked Read 9 Done Done Sent false
13749 4815 Acked Read 5 Done Done Sent false
13750 4816 Acked Read 1 Done Done Sent false
13751 4817 Acked Flush 0 Done Done Sent false
13752 4818 Acked Read 4 Done Done Sent false
13753 4819 Acked Read 7 Done Done Sent false
13754 4820 Acked Write 7 Done Done Sent false
13755 4821 Acked Write 4 Done Done Sent false
13756 4822 Acked Flush 0 Done Done Sent false
13757 4823 Acked Read 3 Done Done Sent false
13758 4824 Acked Write 7 Done Done Sent false
13759 4825 Acked Write 9 Done Done Sent false
13760 4826 Acked Read 6 Done Done Sent false
13761 4827 Acked Read 1 Done Done Sent false
13762 4828 Acked Write 5 Done Done Sent false
13763 4829 Acked Read 8 Done Done Sent false
13764 4830 Acked Read 2 Done Done Sent false
13765 4831 Acked Write 7 Done Done Sent false
13766 4832 Acked Flush 0 Done Done Sent false
13767 4833 Acked Read 2 Done Done Sent false
13768 4834 Acked Read 1 Done Done Sent false
13769 4835 Acked Read 3 Done Done Sent false
13770 4836 Acked Read 5 Done Done Sent false
13771 4837 Acked Write 9 Done Done Sent false
13772 4838 Acked Read 8 Done Done Sent false
13773 4839 Acked Write 10 Done Done Sent false
13774 4840 Acked Write 9 Done Done Sent false
13775 4841 Acked Flush 0 Done Done Sent false
13776 4842 Acked Write 6 Done Done Sent false
13777 4843 Acked Write 5 Done Done Sent false
13778 4844 Acked Read 5 Done Done Sent false
13779 4845 Acked Read 10 Done Done Sent false
13780 4846 Acked Write 10 Done Done Sent false
13781 4847 Acked Read 1 Done Done Sent false
13782 4848 Acked Read 6 Done Done Sent false
13783 4849 Acked Write 9 Done Done Sent false
13784 4850 Acked Write 5 Done Done Sent false
13785 4851 Acked Read 1 Done Done Sent false
13786 4852 Acked Flush 0 Done Done Sent false
13787 4853 Acked Read 4 Done Done Sent false
13788 4854 Acked Read 2 Done Done Sent false
13789 4855 Acked Read 3 Done Done Sent false
13790 4856 Acked Read 10 Done Done Sent false
13791 4857 Acked Write 4 Done Done Sent false
13792 4858 Acked Write 6 Done Done Sent false
13793 4859 Acked Write 7 Done Done Sent false
13794 4860 Acked Read 10 Done Done Sent false
13795 4861 Acked Read 2 Done Done Sent false
13796 4862 Acked Read 7 Done Done Sent false
13797 4863 Acked Flush 0 Done Done Sent false
13798 4864 Acked Write 3 Done Done Sent false
13799 4865 Acked Flush 0 Done Done Sent false
13800 4866 Acked Write 10 Done Done Sent false
13801 4867 Acked Write 1 Done Done Sent false
13802 4868 Acked Read 8 Done Done Sent false
13803 4869 Acked Read 5 Done Done Sent false
13804 4870 Acked Read 2 Done Done Sent false
13805 4871 Acked Write 10 Done Done Sent false
13806 4872 Acked Read 9 Done Done Sent false
13807 4873 Acked Write 8 Done Done Sent false
13808 4874 Acked Read 9 Done Done Sent false
13809 4875 Acked Read 9 Done Done Sent false
13810 4876 Acked Write 2 Done Done Sent false
13811 4877 Acked Write 9 Done Done Sent false
13812 4878 Acked Read 2 Done Done Sent false
13813 4879 Acked Read 7 Done Done Sent false
13814 4880 Acked Read 3 Done Done Sent false
13815 4881 Acked Read 9 Done Done Sent false
13816 4882 Acked Read 4 Done Done Sent false
13817 4883 Acked Read 1 Done Done Sent false
13818 4884 Acked Write 6 Done Done Sent false
13819 4885 Acked Write 5 Done Done Sent false
13820 4886 Acked Write 8 Done Done Sent false
13821 4887 Acked Write 1 Done Done Sent false
13822 4888 Acked Read 10 Done Done Sent false
13823 4889 Acked Read 6 Done Done Sent false
13824 4890 Acked Read 8 Done Done Sent false
13825 4891 Acked Write 3 Done Done Sent false
13826 4892 Acked Read 1 Done Done Sent false
13827 4893 Acked Read 9 Done Done Sent false
13828 4894 Acked Read 5 Done Done Sent false
13829 4895 Acked Read 4 Done Done Sent false
13830 4896 Acked Read 9 Done Done Sent false
13831 4897 Acked Write 2 Done Done Sent false
13832 4898 Acked Read 8 Done Done Sent false
13833 4899 Acked Write 6 Done Done Sent false
13834 4900 Acked Write 6 Done Done Sent false
13835 4901 Acked Write 1 Done Done Sent false
13836 4902 Acked Flush 0 Done Done Sent false
13837 4903 Acked Write 7 Done Done Sent false
13838 4904 Acked Read 6 Done Done Sent false
13839 4905 Acked Read 1 Done Done Sent false
13840 4906 Acked Write 8 Done Done Sent false
13841 4907 Acked Flush 0 Done Done Sent false
13842 4908 Acked Read 6 Done Done Sent false
13843 4909 Acked Read 2 Done Done Sent false
13844 4910 Acked Read 5 Done Done Sent false
13845 4911 Acked Flush 0 Done Done Sent false
13846 4912 Acked Write 5 Done Done Sent false
13847 4913 Acked Read 6 Done Done Sent false
13848 4914 Acked Read 1 Done Done Sent false
13849 4915 Acked Read 4 Done Done Sent false
13850 4916 Acked Write 8 Done Done Sent false
13851 4917 Acked Read 5 Done Done Sent false
13852 4918 Acked Read 9 Done Done Sent false
13853 4919 Acked Write 6 Done Done Sent false
13854 4920 Acked Write 3 Done Done Sent false
13855 4921 Acked Write 3 Done Done Sent false
13856 4922 Acked Write 9 Done Done Sent false
13857 4923 Acked Read 9 Done Done Sent false
13858 4924 Acked Read 8 Done Done Sent false
13859 STATES DS:0 DS:1 DS:2 TOTAL
13860 Sent 0 0 301 301
13861 Done 301 301 0 602
13862 Skipped 0 0 0 0
13863 Error 0 0 0 0
13864 Last Flush: Some(JobId(4911)) Some(JobId(4911)) Some(JobId(4623))
13865 Downstairs last five completed: 4623 4622 4621 4620 4619
13866 Upstairs last five completed: 4924 4923 4922 4921 4920
13867 CLIENT: Up:0 ds:301 act:2
138682025-01-28T21:57:17.609ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
138692025-01-28T21:57:18.594ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
13870 [8830] got start action
13871 [8830] I am going to restart
13872 Starting downstairs at port 8830
13873 Make output file at "/tmp/dsc/downstairs-8830.txt"
13874 [8830][2] reports Starting
13875 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1215)
13876 [8830][2] reports Running
138772025-01-28T21:57:19.596ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
138782025-01-28T21:57:20.598ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
13879 ----------------------------------------------------------------
13880 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13881 States: Active Active Offline
13882 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13883 4624 Acked Write 8 Done Done Sent false
13884 4625 Acked Write 8 Done Done Sent false
13885 4626 Acked Flush 0 Done Done Sent false
13886 4627 Acked Write 2 Done Done Sent false
13887 4628 Acked Flush 0 Done Done Sent false
13888 4629 Acked Write 4 Done Done Sent false
13889 4630 Acked Read 7 Done Done Sent false
13890 4631 Acked Write 6 Done Done Sent false
13891 4632 Acked Read 4 Done Done Sent false
13892 4633 Acked Read 2 Done Done Sent false
13893 4634 Acked Read 8 Done Done Sent false
13894 4635 Acked Read 7 Done Done Sent false
13895 4636 Acked Read 5 Done Done Sent false
13896 4637 Acked Read 2 Done Done Sent false
13897 4638 Acked Write 1 Done Done Sent false
13898 4639 Acked Read 1 Done Done Sent false
13899 4640 Acked Read 4 Done Done Sent false
13900 4641 Acked Read 3 Done Done Sent false
13901 4642 Acked Read 4 Done Done Sent false
13902 4643 Acked Read 8 Done Done Sent false
13903 4644 Acked Write 8 Done Done Sent false
13904 4645 Acked Read 7 Done Done Sent false
13905 4646 Acked Flush 0 Done Done Sent false
13906 4647 Acked Read 1 Done Done Sent false
13907 4648 Acked Flush 0 Done Done Sent false
13908 4649 Acked Read 5 Done Done Sent false
13909 4650 Acked Write 9 Done Done Sent false
13910 4651 Acked Write 10 Done Done Sent false
13911 4652 Acked Write 5 Done Done Sent false
13912 4653 Acked Write 3 Done Done Sent false
13913 4654 Acked Read 6 Done Done Sent false
13914 4655 Acked Read 9 Done Done Sent false
13915 4656 Acked Read 6 Done Done Sent false
13916 4657 Acked Read 2 Done Done Sent false
13917 4658 Acked Write 8 Done Done Sent false
13918 4659 Acked Write 5 Done Done Sent false
13919 4660 Acked Write 7 Done Done Sent false
13920 4661 Acked Read 3 Done Done Sent false
13921 4662 Acked Write 9 Done Done Sent false
13922 4663 Acked Read 3 Done Done Sent false
13923 4664 Acked Read 4 Done Done Sent false
13924 4665 Acked Write 7 Done Done Sent false
13925 4666 Acked Write 8 Done Done Sent false
13926 4667 Acked Write 7 Done Done Sent false
13927 4668 Acked Read 8 Done Done Sent false
13928 4669 Acked Read 5 Done Done Sent false
13929 4670 Acked Read 7 Done Done Sent false
13930 4671 Acked Read 7 Done Done Sent false
13931 4672 Acked Write 7 Done Done Sent false
13932 4673 Acked Write 3 Done Done Sent false
13933 4674 Acked Write 8 Done Done Sent false
13934 4675 Acked Read 2 Done Done Sent false
13935 4676 Acked Read 1 Done Done Sent false
13936 4677 Acked Write 3 Done Done Sent false
13937 4678 Acked Read 9 Done Done Sent false
13938 4679 Acked Read 5 Done Done Sent false
13939 4680 Acked Write 10 Done Done Sent false
13940 4681 Acked Read 6 Done Done Sent false
13941 4682 Acked Read 7 Done Done Sent false
13942 4683 Acked Read 6 Done Done Sent false
13943 4684 Acked Read 7 Done Done Sent false
13944 4685 Acked Read 2 Done Done Sent false
13945 4686 Acked Read 7 Done Done Sent false
13946 4687 Acked Flush 0 Done Done Sent false
13947 4688 Acked Flush 0 Done Done Sent false
13948 4689 Acked Read 2 Done Done Sent false
13949 4690 Acked Read 5 Done Done Sent false
13950 4691 Acked Write 4 Done Done Sent false
13951 4692 Acked Read 2 Done Done Sent false
13952 4693 Acked Read 7 Done Done Sent false
13953 4694 Acked Read 9 Done Done Sent false
13954 4695 Acked Write 1 Done Done Sent false
13955 4696 Acked Read 2 Done Done Sent false
13956 4697 Acked Write 5 Done Done Sent false
13957 4698 Acked Read 1 Done Done Sent false
13958 4699 Acked Flush 0 Done Done Sent false
13959 4700 Acked Write 9 Done Done Sent false
13960 4701 Acked Read 8 Done Done Sent false
13961 4702 Acked Read 2 Done Done Sent false
13962 4703 Acked Read 2 Done Done Sent false
13963 4704 Acked Write 7 Done Done Sent false
13964 4705 Acked Write 2 Done Done Sent false
13965 4706 Acked Read 7 Done Done Sent false
13966 4707 Acked Read 2 Done Done Sent false
13967 4708 Acked Flush 0 Done Done Sent false
13968 4709 Acked Read 7 Done Done Sent false
13969 4710 Acked Read 4 Done Done Sent false
13970 4711 Acked Write 3 Done Done Sent false
13971 4712 Acked Read 10 Done Done Sent false
13972 4713 Acked Write 2 Done Done Sent false
13973 4714 Acked Write 1 Done Done Sent false
13974 4715 Acked Write 2 Done Done Sent false
13975 4716 Acked Write 7 Done Done Sent false
13976 4717 Acked Write 3 Done Done Sent false
13977 4718 Acked Write 10 Done Done Sent false
13978 4719 Acked Read 3 Done Done Sent false
13979 4720 Acked Read 8 Done Done Sent false
13980 4721 Acked Write 4 Done Done Sent false
13981 4722 Acked Read 5 Done Done Sent false
13982 4723 Acked Write 1 Done Done Sent false
13983 4724 Acked Read 7 Done Done Sent false
13984 4725 Acked Read 5 Done Done Sent false
13985 4726 Acked Read 2 Done Done Sent false
13986 4727 Acked Read 1 Done Done Sent false
13987 4728 Acked Read 4 Done Done Sent false
13988 4729 Acked Write 7 Done Done Sent false
13989 4730 Acked Read 2 Done Done Sent false
13990 4731 Acked Read 9 Done Done Sent false
13991 4732 Acked Flush 0 Done Done Sent false
13992 4733 Acked Read 7 Done Done Sent false
13993 4734 Acked Read 2 Done Done Sent false
13994 4735 Acked Write 2 Done Done Sent false
13995 4736 Acked Flush 0 Done Done Sent false
13996 4737 Acked Read 1 Done Done Sent false
13997 4738 Acked Read 9 Done Done Sent false
13998 4739 Acked Flush 0 Done Done Sent false
13999 4740 Acked Write 7 Done Done Sent false
14000 4741 Acked Write 3 Done Done Sent false
14001 4742 Acked Write 2 Done Done Sent false
14002 4743 Acked Read 3 Done Done Sent false
14003 4744 Acked Read 2 Done Done Sent false
14004 4745 Acked Read 3 Done Done Sent false
14005 4746 Acked Write 6 Done Done Sent false
14006 4747 Acked Write 6 Done Done Sent false
14007 4748 Acked Read 8 Done Done Sent false
14008 4749 Acked Flush 0 Done Done Sent false
14009 4750 Acked Read 5 Done Done Sent false
14010 4751 Acked Read 5 Done Done Sent false
14011 4752 Acked Write 7 Done Done Sent false
14012 4753 Acked Read 9 Done Done Sent false
14013 4754 Acked Flush 0 Done Done Sent false
14014 4755 Acked Read 10 Done Done Sent false
14015 4756 Acked Write 4 Done Done Sent false
14016 4757 Acked Write 10 Done Done Sent false
14017 4758 Acked Write 5 Done Done Sent false
14018 4759 Acked Read 7 Done Done Sent false
14019 4760 Acked Write 2 Done Done Sent false
14020 4761 Acked Write 4 Done Done Sent false
14021 4762 Acked Read 9 Done Done Sent false
14022 4763 Acked Read 3 Done Done Sent false
14023 4764 Acked Write 9 Done Done Sent false
14024 4765 Acked Read 8 Done Done Sent false
14025 4766 Acked Write 5 Done Done Sent false
14026 4767 Acked Read 10 Done Done Sent false
14027 4768 Acked Write 4 Done Done Sent false
14028 4769 Acked Write 7 Done Done Sent false
14029 4770 Acked Read 7 Done Done Sent false
14030 4771 Acked Read 2 Done Done Sent false
14031 4772 Acked Read 7 Done Done Sent false
14032 4773 Acked Read 4 Done Done Sent false
14033 4774 Acked Flush 0 Done Done Sent false
14034 4775 Acked Write 3 Done Done Sent false
14035 4776 Acked Write 8 Done Done Sent false
14036 4777 Acked Read 10 Done Done Sent false
14037 4778 Acked Read 3 Done Done Sent false
14038 4779 Acked Flush 0 Done Done Sent false
14039 4780 Acked Read 3 Done Done Sent false
14040 4781 Acked Flush 0 Done Done Sent false
14041 4782 Acked Write 9 Done Done Sent false
14042 4783 Acked Write 7 Done Done Sent false
14043 4784 Acked Read 6 Done Done Sent false
14044 4785 Acked Write 2 Done Done Sent false
14045 4786 Acked Read 7 Done Done Sent false
14046 4787 Acked Read 4 Done Done Sent false
14047 4788 Acked Write 1 Done Done Sent false
14048 4789 Acked Read 5 Done Done Sent false
14049 4790 Acked Write 2 Done Done Sent false
14050 4791 Acked Read 7 Done Done Sent false
14051 4792 Acked Read 6 Done Done Sent false
14052 4793 Acked Flush 0 Done Done Sent false
14053 4794 Acked Read 1 Done Done Sent false
14054 4795 Acked Write 8 Done Done Sent false
14055 4796 Acked Read 9 Done Done Sent false
14056 4797 Acked Read 3 Done Done Sent false
14057 4798 Acked Write 1 Done Done Sent false
14058 4799 Acked Write 4 Done Done Sent false
14059 4800 Acked Write 8 Done Done Sent false
14060 4801 Acked Read 1 Done Done Sent false
14061 4802 Acked Write 1 Done Done Sent false
14062 4803 Acked Write 7 Done Done Sent false
14063 4804 Acked Flush 0 Done Done Sent false
14064 4805 Acked Write 4 Done Done Sent false
14065 4806 Acked Read 3 Done Done Sent false
14066 4807 Acked Read 5 Done Done Sent false
14067 4808 Acked Write 2 Done Done Sent false
14068 4809 Acked Read 3 Done Done Sent false
14069 4810 Acked Write 6 Done Done Sent false
14070 4811 Acked Write 5 Done Done Sent false
14071 4812 Acked Read 1 Done Done Sent false
14072 4813 Acked Write 3 Done Done Sent false
14073 4814 Acked Read 9 Done Done Sent false
14074 4815 Acked Read 5 Done Done Sent false
14075 4816 Acked Read 1 Done Done Sent false
14076 4817 Acked Flush 0 Done Done Sent false
14077 4818 Acked Read 4 Done Done Sent false
14078 4819 Acked Read 7 Done Done Sent false
14079 4820 Acked Write 7 Done Done Sent false
14080 4821 Acked Write 4 Done Done Sent false
14081 4822 Acked Flush 0 Done Done Sent false
14082 4823 Acked Read 3 Done Done Sent false
14083 4824 Acked Write 7 Done Done Sent false
14084 4825 Acked Write 9 Done Done Sent false
14085 4826 Acked Read 6 Done Done Sent false
14086 4827 Acked Read 1 Done Done Sent false
14087 4828 Acked Write 5 Done Done Sent false
14088 4829 Acked Read 8 Done Done Sent false
14089 4830 Acked Read 2 Done Done Sent false
14090 4831 Acked Write 7 Done Done Sent false
14091 4832 Acked Flush 0 Done Done Sent false
14092 4833 Acked Read 2 Done Done Sent false
14093 4834 Acked Read 1 Done Done Sent false
14094 4835 Acked Read 3 Done Done Sent false
14095 4836 Acked Read 5 Done Done Sent false
14096 4837 Acked Write 9 Done Done Sent false
14097 4838 Acked Read 8 Done Done Sent false
14098 4839 Acked Write 10 Done Done Sent false
14099 4840 Acked Write 9 Done Done Sent false
14100 4841 Acked Flush 0 Done Done Sent false
14101 4842 Acked Write 6 Done Done Sent false
14102 4843 Acked Write 5 Done Done Sent false
14103 4844 Acked Read 5 Done Done Sent false
14104 4845 Acked Read 10 Done Done Sent false
14105 4846 Acked Write 10 Done Done Sent false
14106 4847 Acked Read 1 Done Done Sent false
14107 4848 Acked Read 6 Done Done Sent false
14108 4849 Acked Write 9 Done Done Sent false
14109 4850 Acked Write 5 Done Done Sent false
14110 4851 Acked Read 1 Done Done Sent false
14111 4852 Acked Flush 0 Done Done Sent false
14112 4853 Acked Read 4 Done Done Sent false
14113 4854 Acked Read 2 Done Done Sent false
14114 4855 Acked Read 3 Done Done Sent false
14115 4856 Acked Read 10 Done Done Sent false
14116 4857 Acked Write 4 Done Done Sent false
14117 4858 Acked Write 6 Done Done Sent false
14118 4859 Acked Write 7 Done Done Sent false
14119 4860 Acked Read 10 Done Done Sent false
14120 4861 Acked Read 2 Done Done Sent false
14121 4862 Acked Read 7 Done Done Sent false
14122 4863 Acked Flush 0 Done Done Sent false
14123 4864 Acked Write 3 Done Done Sent false
14124 4865 Acked Flush 0 Done Done Sent false
14125 4866 Acked Write 10 Done Done Sent false
14126 4867 Acked Write 1 Done Done Sent false
14127 4868 Acked Read 8 Done Done Sent false
14128 4869 Acked Read 5 Done Done Sent false
14129 4870 Acked Read 2 Done Done Sent false
14130 4871 Acked Write 10 Done Done Sent false
14131 4872 Acked Read 9 Done Done Sent false
14132 4873 Acked Write 8 Done Done Sent false
14133 4874 Acked Read 9 Done Done Sent false
14134 4875 Acked Read 9 Done Done Sent false
14135 4876 Acked Write 2 Done Done Sent false
14136 4877 Acked Write 9 Done Done Sent false
14137 4878 Acked Read 2 Done Done Sent false
14138 4879 Acked Read 7 Done Done Sent false
14139 4880 Acked Read 3 Done Done Sent false
14140 4881 Acked Read 9 Done Done Sent false
14141 4882 Acked Read 4 Done Done Sent false
14142 4883 Acked Read 1 Done Done Sent false
14143 4884 Acked Write 6 Done Done Sent false
14144 4885 Acked Write 5 Done Done Sent false
14145 4886 Acked Write 8 Done Done Sent false
14146 4887 Acked Write 1 Done Done Sent false
14147 4888 Acked Read 10 Done Done Sent false
14148 4889 Acked Read 6 Done Done Sent false
14149 4890 Acked Read 8 Done Done Sent false
14150 4891 Acked Write 3 Done Done Sent false
14151 4892 Acked Read 1 Done Done Sent false
14152 4893 Acked Read 9 Done Done Sent false
14153 4894 Acked Read 5 Done Done Sent false
14154 4895 Acked Read 4 Done Done Sent false
14155 4896 Acked Read 9 Done Done Sent false
14156 4897 Acked Write 2 Done Done Sent false
14157 4898 Acked Read 8 Done Done Sent false
14158 4899 Acked Write 6 Done Done Sent false
14159 4900 Acked Write 6 Done Done Sent false
14160 4901 Acked Write 1 Done Done Sent false
14161 4902 Acked Flush 0 Done Done Sent false
14162 4903 Acked Write 7 Done Done Sent false
14163 4904 Acked Read 6 Done Done Sent false
14164 4905 Acked Read 1 Done Done Sent false
14165 4906 Acked Write 8 Done Done Sent false
14166 4907 Acked Flush 0 Done Done Sent false
14167 4908 Acked Read 6 Done Done Sent false
14168 4909 Acked Read 2 Done Done Sent false
14169 4910 Acked Read 5 Done Done Sent false
14170 4911 Acked Flush 0 Done Done Sent false
14171 4912 Acked Write 5 Done Done Sent false
14172 4913 Acked Read 6 Done Done Sent false
14173 4914 Acked Read 1 Done Done Sent false
14174 4915 Acked Read 4 Done Done Sent false
14175 4916 Acked Write 8 Done Done Sent false
14176 4917 Acked Read 5 Done Done Sent false
14177 4918 Acked Read 9 Done Done Sent false
14178 4919 Acked Write 6 Done Done Sent false
14179 4920 Acked Write 3 Done Done Sent false
14180 4921 Acked Write 3 Done Done Sent false
14181 4922 Acked Write 9 Done Done Sent false
14182 4923 Acked Read 9 Done Done Sent false
14183 4924 Acked Read 8 Done Done Sent false
14184 STATES DS:0 DS:1 DS:2 TOTAL
14185 Sent 0 0 301 301
14186 Done 301 301 0 602
14187 Skipped 0 0 0 0
14188 Error 0 0 0 0
14189 Last Flush: Some(JobId(4911)) Some(JobId(4911)) Some(JobId(4623))
14190 Downstairs last five completed: 4623 4622 4621 4620 4619
14191 Upstairs last five completed: 4924 4923 4922 4921 4920
14192 CLIENT: Up:0 ds:301 act:2
141932025-01-28T21:57:21.611ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
141942025-01-28T21:57:22.601ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
141952025-01-28T21:57:22.864ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
141962025-01-28T21:57:22.865ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37600) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
141972025-01-28T21:57:22.866ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
141982025-01-28T21:57:22.866ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
141992025-01-28T21:57:22.866ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
142002025-01-28T21:57:22.866ZINFOcrucible: send last flush ID to this DS: Some(JobId(4623)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
142012025-01-28T21:57:22.866ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
142022025-01-28T21:57:22.866ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4623)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
142032025-01-28T21:57:22.866ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4623)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
14204 ----------------------------------------------------------------
14205 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14206 Downstairs last five completed: 4925 4924 4923 4922 4921
14207 Upstairs last five completed: 4925 4924 4923 4922 4921
14208 CLIENT: Up:0 ds:0 act:3
14209 Replay: All jobs finished
14210 Main task has work to do, go find it
14211 got dsc Stop(2)
14212 stop 2
14213 Jan 28 21:57:25.112[8830] Got stop action so:false kr:false
14214 INFO request completed, latency_us: 339, response_code: 204, uri: /stop/cid/2, method: POST, req_id: f3d8185b-db1e-4141-b754-5e8729573a77, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
142152025-01-28T21:57:25.113ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
142162025-01-28T21:57:25.113ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
142172025-01-28T21:57:25.113ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
142182025-01-28T21:57:25.113ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
14219 Jan 28 21:57:25.113 INFO request completed, latency_us: 160, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3d114e3b-a904-405f-b729-a8e81bb81689, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
14220 [8830] Exited with: ExitStatus(unix_wait_status(9))
14221 [8830][2] reports Exit
14222 Jan 28 21:57:29.115 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6ec7eff8-7aca-4e1c-a669-7e2f6191374a, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
14223 001/300 Write block 6 len 36864 data: 4 4 3 2 2 2 2 2 2
14224 002/300 Write block 3594 len 24576 data: 4 4 4 6 5 6
14225 003/300 Write block 465 len 16384 data: 4 4 3 3
14226 004/300 Read block 4712 len 36864
14227 005/300 Read block 4947 len 12288
14228 006/300 Read block 1459 len 36864
14229 007/300 Read block 4101 len 32768
14230 008/300 Read block 1539 len 32768
14231 009/300 Read block 2871 len 20480
14232 010/300 Write block 4843 len 20480 data: 4 4 6 6 5
14233 011/300 Read block 2229 len 24576
14234 012/300 Read block 2875 len 36864
14235 013/300 Read block 121 len 4096
14236 014/300 Write block 1147 len 4096 data: 3
14237 015/300 Read block 4731 len 24576
14238 016/300 Read block 1663 len 32768
14239 017/300 Read block 1536 len 36864
14240 018/300 Write block 3396 len 28672 data: 6 6 5 5 4 3 3
14241 019/300 Write block 2629 len 36864 data: 3 3 3 3 5 4 5 5 5
14242 020/300 Read block 859 len 32768
14243 021/300 Read block 1432 len 8192
14244 022/300 Write block 4448 len 20480 data: 3 4 4 4 4
14245 023/300 Write block 535 len 4096 data: 3
14246 024/300 Read block 767 len 36864
14247 025/300 Read block 3517 len 8192
14248 026/300 Write block 2330 len 16384 data: 5 5 5 5
14249 027/300 Read block 3988 len 40960
14250 028/300 Flush
14251 029/300 Read block 1166 len 36864
14252 030/300 Write block 4286 len 28672 data: 2 2 3 3 3 2 3
14253 031/300 Read block 2367 len 8192
14254 032/300 Write block 2730 len 12288 data: 6 5 5
14255 033/300 Read block 3903 len 12288
14256 034/300 Read block 4338 len 8192
14257 035/300 Write block 4817 len 36864 data: 3 2 2 3 3 3 3 2 2
14258 036/300 Write block 1760 len 40960 data: 2 2 3 3 3 3 3 3 3 3
14259 037/300 Write block 3641 len 8192 data: 3 5
14260 038/300 Read block 4973 len 24576
14261 039/300 Read block 2255 len 8192
14262 040/300 Write block 808 len 36864 data: 2 2 3 3 4 4 4 4 4
14263 041/300 Write block 2967 len 20480 data: 4 4 4 4 5
14264 042/300 Write block 4480 len 4096 data: 8
14265 043/300 Read block 2628 len 24576
14266 044/300 Flush
14267 045/300 Read block 486 len 20480
14268 046/300 Read block 3881 len 24576
14269 047/300 Write block 2784 len 28672 data: 4 3 2 2 2 5 5
14270 048/300 Flush
14271 049/300 Read block 3298 len 20480
14272 050/300 Write block 2515 len 36864 data: 4 4 3 4 3 3 3 3 3
14273 051/300 Write block 4411 len 12288 data: 4 4 4
14274 052/300 Write block 3609 len 28672 data: 3 3 3 3 3 3 3
14275 053/300 Read block 2059 len 36864
14276 054/300 Read block 1615 len 20480
14277 055/300 Write block 2451 len 24576 data: 2 2 2 3 3 3
14278 056/300 Flush
14279 057/300 Read block 3051 len 32768
14280 058/300 Read block 4861 len 20480
14281 059/300 Write block 1013 len 40960 data: 3 4 3 4 3 3 3 3 5 5
14282 060/300 Read block 3097 len 40960
14283 061/300 Read block 3436 len 28672
14284 062/300 Write block 2183 len 12288 data: 2 3 3
14285 063/300 Write block 4131 len 36864 data: 2 2 3 4 4 4 5 4 4
14286 064/300 Read block 3315 len 20480
14287 065/300 Read block 1650 len 24576
14288 066/300 Read block 1122 len 28672
14289 067/300 Read block 232 len 24576
14290 068/300 Read block 1661 len 16384
14291 069/300 Read block 1213 len 8192
14292 070/300 Read block 3015 len 36864
14293 071/300 Write block 616 len 4096 data: 4
14294 072/300 Write block 12 len 24576 data: 3 3 3 2 2 2
14295 073/300 Read block 3199 len 36864
14296 074/300 Write block 3474 len 20480 data: 4 3 2 4 4
14297 075/300 Read block 1570 len 12288
14298 076/300 Write block 4009 len 20480 data: 4 4 4 4 3
14299 077/300 Write block 3934 len 4096 data: 3
14300 078/300 Read block 2087 len 8192
14301 079/300 Read block 2018 len 20480
14302 080/300 Read block 1609 len 4096
14303 081/300 Read block 1314 len 32768
14304 082/300 Read block 798 len 24576
14305 083/300 Write block 4942 len 4096 data: 2
14306 084/300 Read block 1241 len 24576
14307 085/300 Read block 3561 len 40960
14308 086/300 Write block 1011 len 12288 data: 3 3 4
14309 087/300 Read block 3575 len 36864
14310 088/300 Read block 3591 len 20480
14311 089/300 Flush
14312 090/300 Read block 4803 len 24576
14313 091/300 Flush
14314 092/300 Flush
14315 093/300 Write block 4598 len 40960 data: 2 3 3 3 3 4 4 4 4 5
14316 094/300 Write block 4569 len 12288 data: 5 5 5
14317 095/300 Write block 2652 len 4096 data: 2
14318 096/300 Flush
14319 097/300 Read block 3969 len 4096
14320 098/300 Read block 2589 len 4096
14321 099/300 Write block 4504 len 40960 data: 4 4 4 4 3 3 3 3 2 2
14322 100/300 Read block 2872 len 24576
14323 101/300 Read block 4440 len 32768
14324 102/300 Flush
14325 103/300 Write block 4037 len 28672 data: 2 2 2 2 3 3 3
14326 104/300 Write block 1180 len 12288 data: 8 7 7
14327 105/300 Write block 3398 len 32768 data: 6 6 5 4 4 4 4 4
14328 106/300 Read block 1835 len 20480
14329 107/300 Write block 4292 len 12288 data: 4 3 3
14330 108/300 Flush
14331 109/300 Read block 3136 len 4096
14332 110/300 Read block 271 len 36864
14333 111/300 Read block 498 len 12288
14334 112/300 Read block 4938 len 24576
14335 113/300 Write block 4270 len 8192 data: 5 5
14336 114/300 Write block 3223 len 40960 data: 3 4 5 4 4 4 3 3 3 3
14337 115/300 Read block 3790 len 16384
14338 116/300 Read block 2773 len 32768
14339 117/300 Flush
14340 118/300 Write block 1024 len 32768 data: 5 4 4 4 3 4 3 3
14341 119/300 Read block 4996 len 12288
14342 120/300 Read block 3520 len 16384
14343 121/300 Write block 1824 len 28672 data: 4 5 5 4 3 3 3
14344 122/300 Read block 4296 len 32768
14345 123/300 Read block 202 len 8192
14346 124/300 Write block 1882 len 24576 data: 4 4 4 3 2 2
14347 125/300 Read block 4365 len 36864
14348 126/300 Read block 4880 len 4096
14349 127/300 Read block 1860 len 28672
14350 128/300 Read block 4834 len 36864
14351 129/300 Flush
14352 130/300 Flush
14353 131/300 Read block 3101 len 12288
14354 132/300 Write block 2490 len 12288 data: 2 2 2
14355 133/300 Read block 4670 len 28672
14356 134/300 Write block 96 len 4096 data: 4
14357 135/300 Write block 4022 len 36864 data: 3 2 2 2 2 2 3 3 3
14358 136/300 Write block 3221 len 4096 data: 4
14359 137/300 Read block 3177 len 12288
14360 138/300 Write block 1686 len 16384 data: 2 2 2 2
14361 139/300 Read block 1397 len 36864
14362 140/300 Flush
14363 141/300 Read block 4015 len 36864
14364 142/300 Read block 566 len 28672
14365 143/300 Read block 1002 len 4096
14366 144/300 Read block 4596 len 32768
14367 145/300 Write block 4641 len 20480 data: 2 2 2 2 2
14368 146/300 Write block 244 len 28672 data: 4 3 3 3 3 3 4
14369 147/300 Write block 1252 len 16384 data: 4 4 5 5
14370 148/300 Read block 4233 len 20480
14371 149/300 Read block 3835 len 24576
14372 150/300 Read block 3121 len 4096
14373 151/300 Write block 205 len 36864 data: 4 5 5 6 6 6 6 5 4
14374 152/300 Read block 4160 len 36864
14375 153/300 Write block 409 len 36864 data: 3 2 2 2 2 2 2 3 3
14376 154/300 Read block 2932 len 12288
14377 155/300 Write block 1399 len 4096 data: 3
14378 156/300 Read block 4291 len 20480
14379 157/300 Read block 495 len 12288
14380 158/300 Read block 4097 len 40960
14381 159/300 Write block 2735 len 4096 data: 5
14382 160/300 Write block 2343 len 20480 data: 4 4 4 4 5
14383 161/300 Write block 1983 len 28672 data: 5 5 5 4 5 5 4
14384 162/300 Write block 3150 len 40960 data: 2 2 2 3 3 3 3 2 3 3
14385 163/300 Flush
14386 164/300 Read block 2521 len 8192
14387 165/300 Read block 4602 len 40960
14388 166/300 Write block 2752 len 8192 data: 6 5
14389 167/300 Flush
14390 168/300 Write block 3291 len 28672 data: 2 2 4 5 6 4 4
14391 169/300 Write block 521 len 36864 data: 3 3 4 4 3 3 4 4 4
14392 170/300 Read block 2667 len 32768
14393 171/300 Read block 1803 len 8192
14394 172/300 Write block 4131 len 20480 data: 3 3 4 5 5
14395 173/300 Write block 3147 len 16384 data: 3 3 2 3
14396 174/300 Write block 4021 len 40960 data: 3 4 3 3 3 3 3 4 4 4
14397 175/300 Write block 1433 len 24576 data: 4 2 2 2 2 2
14398 176/300 Read block 2100 len 8192
14399 177/300 Flush
14400 178/300 Write block 2727 len 20480 data: 3 4 5 7 6
14401 179/300 Flush
14402 180/300 Read block 3357 len 16384
14403 181/300 Read block 4611 len 32768
14404 182/300 Read block 3398 len 8192
14405 183/300 Flush
14406 184/300 Write block 3891 len 20480 data: 3 3 3 3 4
14407 185/300 Write block 3137 len 24576 data: 4 4 3 4 3 3
14408 186/300 Read block 4841 len 20480
14409 187/300 Write block 3727 len 16384 data: 6 7 6 7
14410 188/300 Write block 32 len 16384 data: 2 2 2 2
14411 189/300 Flush
14412 190/300 Read block 1235 len 40960
14413 191/300 Read block 466 len 36864
14414 192/300 Read block 3158 len 24576
14415 193/300 Write block 2154 len 16384 data: 7 7 5 6
14416 194/300 Write block 281 len 12288 data: 3 4 4
14417 195/300 Read block 3912 len 4096
14418 196/300 Read block 1599 len 28672
14419 197/300 Read block 1468 len 28672
14420 198/300 Write block 58 len 40960 data: 3 3 3 3 5 4 5 4 4 4
14421 199/300 Read block 463 len 24576
14422 200/300 Read block 526 len 28672
14423 201/300 Read block 859 len 16384
14424 202/300 Read block 4841 len 24576
14425 203/300 Write block 640 len 12288 data: 3 2 2
14426 204/300 Read block 458 len 28672
14427 205/300 Flush
14428 206/300 Read block 4393 len 28672
14429 207/300 Read block 4914 len 28672
14430 208/300 Write block 4191 len 40960 data: 4 4 3 3 3 3 4 3 3 3
14431 209/300 Read block 4262 len 16384
14432 210/300 Read block 1342 len 12288
14433 211/300 Read block 4052 len 28672
14434 212/300 Read block 1099 len 12288
14435 213/300 Read block 998 len 12288
14436 214/300 Read block 15 len 20480
14437 215/300 Write block 4667 len 28672 data: 7 4 4 4 4 4 3
14438 216/300 Write block 1647 len 32768 data: 3 2 2 2 2 2 3 3
14439 217/300 Write block 4711 len 8192 data: 3 3
14440 218/300 Write block 124 len 16384 data: 5 5 7 4
14441 219/300 Write block 3631 len 36864 data: 2 2 3 2 4 4 4 4 3
14442 220/300 Read block 2418 len 40960
14443 221/300 Read block 970 len 36864
14444 222/300 Write block 1526 len 24576 data: 3 4 4 3 4 4
14445 223/300 Write block 2507 len 8192 data: 4 4
14446 224/300 Read block 1090 len 40960
14447 225/300 Write block 681 len 36864 data: 3 3 3 4 4 3 5 6 6
14448 226/300 Read block 3761 len 40960
14449 227/300 Write block 4015 len 16384 data: 2 4 5 5
14450 228/300 Read block 3776 len 20480
14451 229/300 Read block 1456 len 12288
14452 230/300 Read block 1245 len 12288
14453 231/300 Flush
14454 232/300 Write block 3402 len 24576 data: 5 5 5 5 4 2
14455 233/300 Write block 2809 len 12288 data: 4 3 3
14456 234/300 Write block 4204 len 36864 data: 3 3 3 3 3 3 3 2 2
14457 235/300 Write block 3905 len 32768 data: 3 4 4 4 5 4 4 4
14458 236/300 Write block 3168 len 28672 data: 2 2 2 2 2 2 2
14459 237/300 Read block 3136 len 36864
14460 238/300 Flush
14461 239/300 Read block 280 len 28672
14462 240/300 Read block 82 len 16384
14463 241/300 Read block 3343 len 4096
14464 242/300 Read block 4539 len 8192
14465 243/300 Write block 2878 len 20480 data: 3 3 3 4 3
14466 244/300 Read block 2903 len 20480
14467 245/300 Write block 2702 len 40960 data: 2 2 2 3 3 3 3 3 4 3
14468 246/300 Read block 4343 len 8192
14469 247/300 Read block 715 len 20480
14470 248/300 Write block 4964 len 8192 data: 5 5
14471 249/300 Read block 2330 len 12288
14472 250/300 Read block 771 len 24576
14473 251/300 Read block 3218 len 40960
14474 252/300 Write block 4090 len 36864 data: 5 6 6 5 5 5 5 5 4
14475 253/300 Read block 2560 len 8192
14476 254/300 Flush
14477 255/300 Read block 3249 len 4096
14478 256/300 Write block 834 len 4096 data: 3
14479 257/300 Read block 614 len 8192
14480 258/300 Flush
14481 259/300 Read block 2859 len 32768
14482 260/300 Read block 4115 len 20480
14483 261/300 Write block 3261 len 28672 data: 2 3 4 4 4 3 3
14484 262/300 Write block 61 len 24576 data: 4 6 5 6 5 5
14485 263/300 Write block 3747 len 4096 data: 2
14486 264/300 Write block 773 len 4096 data: 3
14487 265/300 Read block 3532 len 40960
14488 266/300 Flush
14489 267/300 Read block 3529 len 40960
14490 268/300 Read block 4036 len 36864
14491 269/300 Write block 1043 len 20480 data: 5 6 4 6 6
14492 270/300 Write block 4031 len 24576 data: 3 3 3 3 3 3
14493 271/300 Read block 3679 len 40960
14494 272/300 Read block 1753 len 20480
14495 273/300 Read block 324 len 40960
14496 274/300 Read block 2627 len 28672
14497 275/300 Write block 2433 len 12288 data: 6 5 4
14498 276/300 Read block 2579 len 36864
14499 277/300 Read block 3356 len 28672
14500 278/300 Write block 1095 len 12288 data: 2 2 2
14501 279/300 Read block 2969 len 24576
14502 280/300 Read block 4384 len 32768
14503 281/300 Flush
14504 282/300 Write block 3388 len 36864 data: 7 7 6 6 6 7 6 6 7
14505 283/300 Write block 4027 len 24576 data: 4 5 5 5 4 4
14506 284/300 Write block 4833 len 16384 data: 3 3 3 3
14507 285/300 Write block 1214 len 32768 data: 2 2 2 3 3 3 3 4
14508 286/300 Read block 1387 len 36864
14509 287/300 Flush
14510 288/300 Read block 3886 len 16384
14511 289/300 Flush
14512 290/300 Read block 2644 len 40960
14513 291/300 Read block 982 len 4096
14514 292/300 Read block 3270 len 8192
14515 293/300 Write block 2444 len 32768 data: 4 4 4 4 4 2 2 3
14516 294/300 Write block 4926 len 8192 data: 5 6
14517 295/300 Read block 1997 len 4096
14518 296/300 Write block 2794 len 32768 data: 5 5 5 5 6 4 3 3
14519 297/300 Read block 4371 len 28672
14520 298/300 Write block 2747 len 4096 data: 2
14521 299/300 Write block 3371 len 28672 data: 3 4 4 4 5 4 4
14522 300/300 Read block 146 len 12288
14523 301/300 Read block 2307 len 24576
14524 Main task has work to do, go find it
14525 got dsc Start(2)
14526 start 2
14527 Jan 28 21:57:29.361 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/2, method: POST, req_id: d0b67b11-0ccc-4698-b961-4dbec483c435, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
14528 Replay: started 2, returned:Ok(())
14529 ----------------------------------------------------------------
14530 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14531 States: Active Active Offline
14532 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14533 4926 Acked Write 9 Done Done Sent false
14534 4927 Acked Write 6 Done Done Sent false
14535 4928 Acked Write 4 Done Done Sent false
14536 4929 Acked Read 9 Done Done Sent false
14537 4930 Acked Read 3 Done Done Sent false
14538 4931 Acked Read 9 Done Done Sent false
14539 4932 Acked Read 8 Done Done Sent false
14540 4933 Acked Read 8 Done Done Sent false
14541 4934 Acked Read 5 Done Done Sent false
14542 4935 Acked Write 5 Done Done Sent false
14543 4936 Acked Read 6 Done Done Sent false
14544 4937 Acked Read 9 Done Done Sent false
14545 4938 Acked Read 1 Done Done Sent false
14546 4939 Acked Write 1 Done Done Sent false
14547 4940 Acked Read 6 Done Done Sent false
14548 4941 Acked Read 8 Done Done Sent false
14549 4942 Acked Read 9 Done Done Sent false
14550 4943 Acked Write 7 Done Done Sent false
14551 4944 Acked Write 9 Done Done Sent false
14552 4945 Acked Read 8 Done Done Sent false
14553 4946 Acked Read 2 Done Done Sent false
14554 4947 Acked Write 5 Done Done Sent false
14555 4948 Acked Write 1 Done Done Sent false
14556 4949 Acked Read 9 Done Done Sent false
14557 4950 Acked Read 2 Done Done Sent false
14558 4951 Acked Write 4 Done Done Sent false
14559 4952 Acked Read 10 Done Done Sent false
14560 4953 Acked Flush 0 Done Done Sent false
14561 4954 Acked Read 9 Done Done Sent false
14562 4955 Acked Write 7 Done Done Sent false
14563 4956 Acked Read 2 Done Done Sent false
14564 4957 Acked Write 3 Done Done Sent false
14565 4958 Acked Read 3 Done Done Sent false
14566 4959 Acked Read 2 Done Done Sent false
14567 4960 Acked Write 9 Done Done Sent false
14568 4961 Acked Write 10 Done Done Sent false
14569 4962 Acked Write 2 Done Done Sent false
14570 4963 Acked Read 6 Done Done Sent false
14571 4964 Acked Read 2 Done Done Sent false
14572 4965 Acked Write 9 Done Done Sent false
14573 4966 Acked Write 5 Done Done Sent false
14574 4967 Acked Write 1 Done Done Sent false
14575 4968 Acked Read 6 Done Done Sent false
14576 4969 Acked Flush 0 Done Done Sent false
14577 4970 Acked Read 5 Done Done Sent false
14578 4971 Acked Read 6 Done Done Sent false
14579 4972 Acked Write 7 Done Done Sent false
14580 4973 Acked Flush 0 Done Done Sent false
14581 4974 Acked Read 5 Done Done Sent false
14582 4975 Acked Write 9 Done Done Sent false
14583 4976 Acked Write 3 Done Done Sent false
14584 4977 Acked Write 7 Done Done Sent false
14585 4978 Acked Read 9 Done Done Sent false
14586 4979 Acked Read 5 Done Done Sent false
14587 4980 Acked Write 6 Done Done Sent false
14588 4981 Acked Flush 0 Done Done Sent false
14589 4982 Acked Read 8 Done Done Sent false
14590 4983 Acked Read 5 Done Done Sent false
14591 4984 Acked Write 10 Done Done Sent false
14592 4985 Acked Read 10 Done Done Sent false
14593 4986 Acked Read 7 Done Done Sent false
14594 4987 Acked Write 3 Done Done Sent false
14595 4988 Acked Write 9 Done Done Sent false
14596 4989 Acked Read 5 Done Done Sent false
14597 4990 Acked Read 6 Done Done Sent false
14598 4991 Acked Read 7 Done Done Sent false
14599 4992 Acked Read 6 Done Done Sent false
14600 4993 Acked Read 4 Done Done Sent false
14601 4994 Acked Read 2 Done Done Sent false
14602 4995 Acked Read 9 Done Done Sent false
14603 4996 Acked Write 1 Done Done Sent false
14604 4997 Acked Write 6 Done Done Sent false
14605 4998 Acked Read 9 Done Done Sent false
14606 4999 Acked Write 5 Done Done Sent false
14607 5000 Acked Read 3 Done Done Sent false
14608 5001 Acked Write 5 Done Done Sent false
14609 5002 Acked Write 1 Done Done Sent false
14610 5003 Acked Read 2 Done Done Sent false
14611 5004 Acked Read 5 Done Done Sent false
14612 5005 Acked Read 1 Done Done Sent false
14613 5006 Acked Read 8 Done Done Sent false
14614 5007 Acked Read 6 Done Done Sent false
14615 5008 Acked Write 1 Done Done Sent false
14616 5009 Acked Read 6 Done Done Sent false
14617 5010 Acked Read 10 Done Done Sent false
14618 5011 Acked Write 3 Done Done Sent false
14619 5012 Acked Read 9 Done Done Sent false
14620 5013 Acked Read 5 Done Done Sent false
14621 5014 Acked Flush 0 Done Done Sent false
14622 5015 Acked Read 6 Done Done Sent false
14623 5016 Acked Flush 0 Done Done Sent false
14624 5017 Acked Flush 0 Done Done Sent false
14625 5018 Acked Write 10 Done Done Sent false
14626 5019 Acked Write 3 Done Done Sent false
14627 5020 Acked Write 1 Done Done Sent false
14628 5021 Acked Flush 0 Done Done Sent false
14629 5022 Acked Read 1 Done Done Sent false
14630 5023 Acked Read 1 Done Done Sent false
14631 5024 Acked Write 10 Done Done Sent false
14632 5025 Acked Read 6 Done Done Sent false
14633 5026 Acked Read 8 Done Done Sent false
14634 5027 Acked Flush 0 Done Done Sent false
14635 5028 Acked Write 7 Done Done Sent false
14636 5029 Acked Write 3 Done Done Sent false
14637 5030 Acked Write 8 Done Done Sent false
14638 5031 Acked Read 5 Done Done Sent false
14639 5032 Acked Write 3 Done Done Sent false
14640 5033 Acked Flush 0 Done Done Sent false
14641 5034 Acked Read 1 Done Done Sent false
14642 5035 Acked Read 9 Done Done Sent false
14643 5036 Acked Read 3 Done Done Sent false
14644 5037 Acked Read 6 Done Done Sent false
14645 5038 Acked Write 2 Done Done Sent false
14646 5039 Acked Write 10 Done Done Sent false
14647 5040 Acked Read 4 Done Done Sent false
14648 5041 Acked Read 8 Done Done Sent false
14649 5042 Acked Flush 0 Done Done Sent false
14650 5043 Acked Write 8 Done Done Sent false
14651 5044 Acked Read 3 Done Done Sent false
14652 5045 Acked Read 4 Done Done Sent false
14653 5046 Acked Write 7 Done Done Sent false
14654 5047 Acked Read 8 Done Done Sent false
14655 5048 Acked Read 2 Done Done Sent false
14656 5049 Acked Write 6 Done Done Sent false
14657 5050 Acked Read 9 Done Done Sent false
14658 5051 Acked Read 1 Done Done Sent false
14659 5052 Acked Read 7 Done Done Sent false
14660 5053 Acked Read 9 Done Done Sent false
14661 5054 Acked Flush 0 Done Done Sent false
14662 5055 Acked Flush 0 Done Done Sent false
14663 5056 Acked Read 3 Done Done Sent false
14664 5057 Acked Write 3 Done Done Sent false
14665 5058 Acked Read 7 Done Done Sent false
14666 5059 Acked Write 1 Done Done Sent false
14667 5060 Acked Write 9 Done Done Sent false
14668 5061 Acked Write 1 Done Done Sent false
14669 5062 Acked Read 3 Done Done Sent false
14670 5063 Acked Write 4 Done Done Sent false
14671 5064 Acked Read 9 Done Done Sent false
14672 5065 Acked Flush 0 Done Done Sent false
14673 5066 Acked Read 9 Done Done Sent false
14674 5067 Acked Read 7 Done Done Sent false
14675 5068 Acked Read 1 Done Done Sent false
14676 5069 Acked Read 8 Done Done Sent false
14677 5070 Acked Write 5 Done Done Sent false
14678 5071 Acked Write 7 Done Done Sent false
14679 5072 Acked Write 4 Done Done Sent false
14680 5073 Acked Read 5 Done Done Sent false
14681 5074 Acked Read 6 Done Done Sent false
14682 5075 Acked Read 1 Done Done Sent false
14683 5076 Acked Write 9 Done Done Sent false
14684 5077 Acked Read 9 Done Done Sent false
14685 5078 Acked Write 9 Done Done Sent false
14686 5079 Acked Read 3 Done Done Sent false
14687 5080 Acked Write 1 Done Done Sent false
14688 5081 Acked Read 5 Done Done Sent false
14689 5082 Acked Read 3 Done Done Sent false
14690 5083 Acked Read 10 Done Done Sent false
14691 5084 Acked Write 1 Done Done Sent false
14692 5085 Acked Write 5 Done Done Sent false
14693 5086 Acked Write 7 Done Done Sent false
14694 5087 Acked Write 10 Done Done Sent false
14695 5088 Acked Flush 0 Done Done Sent false
14696 5089 Acked Read 2 Done Done Sent false
14697 5090 Acked Read 10 Done Done Sent false
14698 5091 Acked Write 2 Done Done Sent false
14699 5092 Acked Flush 0 Done Done Sent false
14700 5093 Acked Write 7 Done Done Sent false
14701 5094 Acked Write 9 Done Done Sent false
14702 5095 Acked Read 8 Done Done Sent false
14703 5096 Acked Read 2 Done Done Sent false
14704 5097 Acked Write 5 Done Done Sent false
14705 5098 Acked Write 4 Done Done Sent false
14706 5099 Acked Write 10 Done Done Sent false
14707 5100 Acked Write 6 Done Done Sent false
14708 5101 Acked Read 2 Done Done Sent false
14709 5102 Acked Flush 0 Done Done Sent false
14710 5103 Acked Write 5 Done Done Sent false
14711 5104 Acked Flush 0 Done Done Sent false
14712 5105 Acked Read 4 Done Done Sent false
14713 5106 Acked Read 8 Done Done Sent false
14714 5107 Acked Read 2 Done Done Sent false
14715 5108 Acked Flush 0 Done Done Sent false
14716 5109 Acked Write 5 Done Done Sent false
14717 5110 Acked Write 6 Done Done Sent false
14718 5111 Acked Read 5 Done Done Sent false
14719 5112 Acked Write 4 Done Done Sent false
14720 5113 Acked Write 4 Done Done Sent false
14721 5114 Acked Flush 0 Done Done Sent false
14722 5115 Acked Read 10 Done Done Sent false
14723 5116 Acked Read 9 Done Done Sent false
14724 5117 Acked Read 6 Done Done Sent false
14725 5118 Acked Write 4 Done Done Sent false
14726 5119 Acked Write 3 Done Done Sent false
14727 5120 Acked Read 1 Done Done Sent false
14728 5121 Acked Read 7 Done Done Sent false
14729 5122 Acked Read 7 Done Done Sent false
14730 5123 Acked Write 10 Done Done Sent false
14731 5124 Acked Read 6 Done Done Sent false
14732 5125 Acked Read 7 Done Done Sent false
14733 5126 Acked Read 4 Done Done Sent false
14734 5127 Acked Read 6 Done Done Sent false
14735 5128 Acked Write 3 Done Done Sent false
14736 5129 Acked Read 7 Done Done Sent false
14737 5130 Acked Flush 0 Done Done Sent false
14738 5131 Acked Read 7 Done Done Sent false
14739 5132 Acked Read 7 Done Done Sent false
14740 5133 Acked Write 10 Done Done Sent false
14741 5134 Acked Read 4 Done Done Sent false
14742 5135 Acked Read 3 Done Done Sent false
14743 5136 Acked Read 7 Done Done Sent false
14744 5137 Acked Read 3 Done Done Sent false
14745 5138 Acked Read 3 Done Done Sent false
14746 5139 Acked Read 5 Done Done Sent false
14747 5140 Acked Write 7 Done Done Sent false
14748 5141 Acked Write 8 Done Done Sent false
14749 5142 Acked Write 2 Done Done Sent false
14750 5143 Acked Write 4 Done Done Sent false
14751 5144 Acked Write 9 Done Done Sent false
14752 5145 Acked Read 10 Done Done Sent false
14753 5146 Acked Read 9 Done Done Sent false
14754 5147 Acked Write 6 Done Done Sent false
14755 5148 Acked Write 2 Done Done Sent false
14756 5149 Acked Read 10 Done Done Sent false
14757 5150 Acked Write 9 Done Done Sent false
14758 5151 Acked Read 10 Done Done Sent false
14759 5152 Acked Write 4 Done Done Sent false
14760 5153 Acked Read 5 Done Done Sent false
14761 5154 Acked Read 3 Done Done Sent false
14762 5155 Acked Read 3 Done Done Sent false
14763 5156 Acked Flush 0 Done Done Sent false
14764 5157 Acked Write 6 Done Done Sent false
14765 5158 Acked Write 3 Done Done Sent false
14766 5159 Acked Write 9 Done Done Sent false
14767 5160 Acked Write 8 Done Done Sent false
14768 5161 Acked Write 7 Done Done Sent false
14769 5162 Acked Read 9 Done Done Sent false
14770 5163 Acked Flush 0 Done Done Sent false
14771 5164 Acked Read 7 Done Done Sent false
14772 5165 Acked Read 4 Done Done Sent false
14773 5166 Acked Read 1 Done Done Sent false
14774 5167 Acked Read 2 Done Done Sent false
14775 5168 Acked Write 5 Done Done Sent false
14776 5169 Acked Read 5 Done Done Sent false
14777 5170 Acked Write 10 Done Done Sent false
14778 5171 Acked Read 2 Done Done Sent false
14779 5172 Acked Read 5 Done Done Sent false
14780 5173 Acked Write 2 Done Done Sent false
14781 5174 Acked Read 3 Done Done Sent false
14782 5175 Acked Read 6 Done Done Sent false
14783 5176 Acked Read 10 Done Done Sent false
14784 5177 Acked Write 9 Done Done Sent false
14785 5178 Acked Read 2 Done Done Sent false
14786 5179 Acked Flush 0 Done Done Sent false
14787 5180 Acked Read 1 Done Done Sent false
14788 5181 Acked Write 1 Done Done Sent false
14789 5182 Acked Read 2 Done Done Sent false
14790 5183 Acked Flush 0 Done Done Sent false
14791 5184 Acked Read 8 Done Done Sent false
14792 5185 Acked Read 5 Done Done Sent false
14793 5186 Acked Write 7 Done Done Sent false
14794 5187 Acked Write 6 Done Done Sent false
14795 5188 Acked Write 1 Done Done Sent false
14796 5189 Acked Write 1 Done Done Sent false
14797 5190 Acked Read 10 Done Done Sent false
14798 5191 Acked Flush 0 Done Done Sent false
14799 5192 Acked Read 10 Done Done Sent false
14800 5193 Acked Read 9 Done Done Sent false
14801 5194 Acked Write 5 Done Done Sent false
14802 5195 Acked Write 6 Done Done Sent false
14803 5196 Acked Read 10 Done Done Sent false
14804 5197 Acked Read 5 Done Done Sent false
14805 5198 Acked Read 10 Done Done Sent false
14806 5199 Acked Read 7 Done Done Sent false
14807 5200 Acked Write 3 Done Done Sent false
14808 5201 Acked Read 9 Done Done Sent false
14809 5202 Acked Read 7 Done Done Sent false
14810 5203 Acked Write 3 Done Done Sent false
14811 5204 Acked Read 6 Done Done Sent false
14812 5205 Acked Read 8 Done Done Sent false
14813 5206 Acked Flush 0 Done Done Sent false
14814 5207 Acked Write 9 Done Done Sent false
14815 5208 Acked Write 6 Done Done Sent false
14816 5209 Acked Write 4 Done Done Sent false
14817 5210 Acked Write 8 Done Done Sent false
14818 5211 Acked Read 9 Done Done Sent false
14819 5212 Acked Flush 0 Done Done Sent false
14820 5213 Acked Read 4 Done Done Sent false
14821 5214 Acked Flush 0 Done Done Sent false
14822 5215 Acked Read 10 Done Done Sent false
14823 5216 Acked Read 1 Done Done Sent false
14824 5217 Acked Read 2 Done Done Sent false
14825 5218 Acked Write 8 Done Done Sent false
14826 5219 Acked Write 2 Done Done Sent false
14827 5220 Acked Read 1 Done Done Sent false
14828 5221 Acked Write 8 Done Done Sent false
14829 5222 Acked Read 7 Done Done Sent false
14830 5223 Acked Write 1 Done Done Sent false
14831 5224 Acked Write 7 Done Done Sent false
14832 5225 Acked Read 3 Done Done Sent false
14833 5226 Acked Read 6 Done Done Sent false
14834 STATES DS:0 DS:1 DS:2 TOTAL
14835 Sent 0 0 301 301
14836 Done 301 301 0 602
14837 Skipped 0 0 0 0
14838 Error 0 0 0 0
14839 Last Flush: Some(JobId(5214)) Some(JobId(5214)) Some(JobId(4925))
14840 Downstairs last five completed: 4925 4924 4923 4922 4921
14841 Upstairs last five completed: 5226 5225 5224 5223 5222
14842 CLIENT: Up:0 ds:301 act:2
148432025-01-28T21:57:30.612ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
14844 [8830] got start action
14845 [8830] I am going to restart
14846 Starting downstairs at port 8830
14847 Make output file at "/tmp/dsc/downstairs-8830.txt"
14848 [8830][2] reports Starting
14849 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1216)
14850 [8830][2] reports Running
148512025-01-28T21:57:31.613ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
148522025-01-28T21:57:32.614ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
14853 ----------------------------------------------------------------
14854 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14855 States: Active Active Offline
14856 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14857 4926 Acked Write 9 Done Done Sent false
14858 4927 Acked Write 6 Done Done Sent false
14859 4928 Acked Write 4 Done Done Sent false
14860 4929 Acked Read 9 Done Done Sent false
14861 4930 Acked Read 3 Done Done Sent false
14862 4931 Acked Read 9 Done Done Sent false
14863 4932 Acked Read 8 Done Done Sent false
14864 4933 Acked Read 8 Done Done Sent false
14865 4934 Acked Read 5 Done Done Sent false
14866 4935 Acked Write 5 Done Done Sent false
14867 4936 Acked Read 6 Done Done Sent false
14868 4937 Acked Read 9 Done Done Sent false
14869 4938 Acked Read 1 Done Done Sent false
14870 4939 Acked Write 1 Done Done Sent false
14871 4940 Acked Read 6 Done Done Sent false
14872 4941 Acked Read 8 Done Done Sent false
14873 4942 Acked Read 9 Done Done Sent false
14874 4943 Acked Write 7 Done Done Sent false
14875 4944 Acked Write 9 Done Done Sent false
14876 4945 Acked Read 8 Done Done Sent false
14877 4946 Acked Read 2 Done Done Sent false
14878 4947 Acked Write 5 Done Done Sent false
14879 4948 Acked Write 1 Done Done Sent false
14880 4949 Acked Read 9 Done Done Sent false
14881 4950 Acked Read 2 Done Done Sent false
14882 4951 Acked Write 4 Done Done Sent false
14883 4952 Acked Read 10 Done Done Sent false
14884 4953 Acked Flush 0 Done Done Sent false
14885 4954 Acked Read 9 Done Done Sent false
14886 4955 Acked Write 7 Done Done Sent false
14887 4956 Acked Read 2 Done Done Sent false
14888 4957 Acked Write 3 Done Done Sent false
14889 4958 Acked Read 3 Done Done Sent false
14890 4959 Acked Read 2 Done Done Sent false
14891 4960 Acked Write 9 Done Done Sent false
14892 4961 Acked Write 10 Done Done Sent false
14893 4962 Acked Write 2 Done Done Sent false
14894 4963 Acked Read 6 Done Done Sent false
14895 4964 Acked Read 2 Done Done Sent false
14896 4965 Acked Write 9 Done Done Sent false
14897 4966 Acked Write 5 Done Done Sent false
14898 4967 Acked Write 1 Done Done Sent false
14899 4968 Acked Read 6 Done Done Sent false
14900 4969 Acked Flush 0 Done Done Sent false
14901 4970 Acked Read 5 Done Done Sent false
14902 4971 Acked Read 6 Done Done Sent false
14903 4972 Acked Write 7 Done Done Sent false
14904 4973 Acked Flush 0 Done Done Sent false
14905 4974 Acked Read 5 Done Done Sent false
14906 4975 Acked Write 9 Done Done Sent false
14907 4976 Acked Write 3 Done Done Sent false
14908 4977 Acked Write 7 Done Done Sent false
14909 4978 Acked Read 9 Done Done Sent false
14910 4979 Acked Read 5 Done Done Sent false
14911 4980 Acked Write 6 Done Done Sent false
14912 4981 Acked Flush 0 Done Done Sent false
14913 4982 Acked Read 8 Done Done Sent false
14914 4983 Acked Read 5 Done Done Sent false
14915 4984 Acked Write 10 Done Done Sent false
14916 4985 Acked Read 10 Done Done Sent false
14917 4986 Acked Read 7 Done Done Sent false
14918 4987 Acked Write 3 Done Done Sent false
14919 4988 Acked Write 9 Done Done Sent false
14920 4989 Acked Read 5 Done Done Sent false
14921 4990 Acked Read 6 Done Done Sent false
14922 4991 Acked Read 7 Done Done Sent false
14923 4992 Acked Read 6 Done Done Sent false
14924 4993 Acked Read 4 Done Done Sent false
14925 4994 Acked Read 2 Done Done Sent false
14926 4995 Acked Read 9 Done Done Sent false
14927 4996 Acked Write 1 Done Done Sent false
14928 4997 Acked Write 6 Done Done Sent false
14929 4998 Acked Read 9 Done Done Sent false
14930 4999 Acked Write 5 Done Done Sent false
14931 5000 Acked Read 3 Done Done Sent false
14932 5001 Acked Write 5 Done Done Sent false
14933 5002 Acked Write 1 Done Done Sent false
14934 5003 Acked Read 2 Done Done Sent false
14935 5004 Acked Read 5 Done Done Sent false
14936 5005 Acked Read 1 Done Done Sent false
14937 5006 Acked Read 8 Done Done Sent false
14938 5007 Acked Read 6 Done Done Sent false
14939 5008 Acked Write 1 Done Done Sent false
14940 5009 Acked Read 6 Done Done Sent false
14941 5010 Acked Read 10 Done Done Sent false
14942 5011 Acked Write 3 Done Done Sent false
14943 5012 Acked Read 9 Done Done Sent false
14944 5013 Acked Read 5 Done Done Sent false
14945 5014 Acked Flush 0 Done Done Sent false
14946 5015 Acked Read 6 Done Done Sent false
14947 5016 Acked Flush 0 Done Done Sent false
14948 5017 Acked Flush 0 Done Done Sent false
14949 5018 Acked Write 10 Done Done Sent false
14950 5019 Acked Write 3 Done Done Sent false
14951 5020 Acked Write 1 Done Done Sent false
14952 5021 Acked Flush 0 Done Done Sent false
14953 5022 Acked Read 1 Done Done Sent false
14954 5023 Acked Read 1 Done Done Sent false
14955 5024 Acked Write 10 Done Done Sent false
14956 5025 Acked Read 6 Done Done Sent false
14957 5026 Acked Read 8 Done Done Sent false
14958 5027 Acked Flush 0 Done Done Sent false
14959 5028 Acked Write 7 Done Done Sent false
14960 5029 Acked Write 3 Done Done Sent false
14961 5030 Acked Write 8 Done Done Sent false
14962 5031 Acked Read 5 Done Done Sent false
14963 5032 Acked Write 3 Done Done Sent false
14964 5033 Acked Flush 0 Done Done Sent false
14965 5034 Acked Read 1 Done Done Sent false
14966 5035 Acked Read 9 Done Done Sent false
14967 5036 Acked Read 3 Done Done Sent false
14968 5037 Acked Read 6 Done Done Sent false
14969 5038 Acked Write 2 Done Done Sent false
14970 5039 Acked Write 10 Done Done Sent false
14971 5040 Acked Read 4 Done Done Sent false
14972 5041 Acked Read 8 Done Done Sent false
14973 5042 Acked Flush 0 Done Done Sent false
14974 5043 Acked Write 8 Done Done Sent false
14975 5044 Acked Read 3 Done Done Sent false
14976 5045 Acked Read 4 Done Done Sent false
14977 5046 Acked Write 7 Done Done Sent false
14978 5047 Acked Read 8 Done Done Sent false
14979 5048 Acked Read 2 Done Done Sent false
14980 5049 Acked Write 6 Done Done Sent false
14981 5050 Acked Read 9 Done Done Sent false
14982 5051 Acked Read 1 Done Done Sent false
14983 5052 Acked Read 7 Done Done Sent false
14984 5053 Acked Read 9 Done Done Sent false
14985 5054 Acked Flush 0 Done Done Sent false
14986 5055 Acked Flush 0 Done Done Sent false
14987 5056 Acked Read 3 Done Done Sent false
14988 5057 Acked Write 3 Done Done Sent false
14989 5058 Acked Read 7 Done Done Sent false
14990 5059 Acked Write 1 Done Done Sent false
14991 5060 Acked Write 9 Done Done Sent false
14992 5061 Acked Write 1 Done Done Sent false
14993 5062 Acked Read 3 Done Done Sent false
14994 5063 Acked Write 4 Done Done Sent false
14995 5064 Acked Read 9 Done Done Sent false
14996 5065 Acked Flush 0 Done Done Sent false
14997 5066 Acked Read 9 Done Done Sent false
14998 5067 Acked Read 7 Done Done Sent false
14999 5068 Acked Read 1 Done Done Sent false
15000 5069 Acked Read 8 Done Done Sent false
15001 5070 Acked Write 5 Done Done Sent false
15002 5071 Acked Write 7 Done Done Sent false
15003 5072 Acked Write 4 Done Done Sent false
15004 5073 Acked Read 5 Done Done Sent false
15005 5074 Acked Read 6 Done Done Sent false
15006 5075 Acked Read 1 Done Done Sent false
15007 5076 Acked Write 9 Done Done Sent false
15008 5077 Acked Read 9 Done Done Sent false
15009 5078 Acked Write 9 Done Done Sent false
15010 5079 Acked Read 3 Done Done Sent false
15011 5080 Acked Write 1 Done Done Sent false
15012 5081 Acked Read 5 Done Done Sent false
15013 5082 Acked Read 3 Done Done Sent false
15014 5083 Acked Read 10 Done Done Sent false
15015 5084 Acked Write 1 Done Done Sent false
15016 5085 Acked Write 5 Done Done Sent false
15017 5086 Acked Write 7 Done Done Sent false
15018 5087 Acked Write 10 Done Done Sent false
15019 5088 Acked Flush 0 Done Done Sent false
15020 5089 Acked Read 2 Done Done Sent false
15021 5090 Acked Read 10 Done Done Sent false
15022 5091 Acked Write 2 Done Done Sent false
15023 5092 Acked Flush 0 Done Done Sent false
15024 5093 Acked Write 7 Done Done Sent false
15025 5094 Acked Write 9 Done Done Sent false
15026 5095 Acked Read 8 Done Done Sent false
15027 5096 Acked Read 2 Done Done Sent false
15028 5097 Acked Write 5 Done Done Sent false
15029 5098 Acked Write 4 Done Done Sent false
15030 5099 Acked Write 10 Done Done Sent false
15031 5100 Acked Write 6 Done Done Sent false
15032 5101 Acked Read 2 Done Done Sent false
15033 5102 Acked Flush 0 Done Done Sent false
15034 5103 Acked Write 5 Done Done Sent false
15035 5104 Acked Flush 0 Done Done Sent false
15036 5105 Acked Read 4 Done Done Sent false
15037 5106 Acked Read 8 Done Done Sent false
15038 5107 Acked Read 2 Done Done Sent false
15039 5108 Acked Flush 0 Done Done Sent false
15040 5109 Acked Write 5 Done Done Sent false
15041 5110 Acked Write 6 Done Done Sent false
15042 5111 Acked Read 5 Done Done Sent false
15043 5112 Acked Write 4 Done Done Sent false
15044 5113 Acked Write 4 Done Done Sent false
15045 5114 Acked Flush 0 Done Done Sent false
15046 5115 Acked Read 10 Done Done Sent false
15047 5116 Acked Read 9 Done Done Sent false
15048 5117 Acked Read 6 Done Done Sent false
15049 5118 Acked Write 4 Done Done Sent false
15050 5119 Acked Write 3 Done Done Sent false
15051 5120 Acked Read 1 Done Done Sent false
15052 5121 Acked Read 7 Done Done Sent false
15053 5122 Acked Read 7 Done Done Sent false
15054 5123 Acked Write 10 Done Done Sent false
15055 5124 Acked Read 6 Done Done Sent false
15056 5125 Acked Read 7 Done Done Sent false
15057 5126 Acked Read 4 Done Done Sent false
15058 5127 Acked Read 6 Done Done Sent false
15059 5128 Acked Write 3 Done Done Sent false
15060 5129 Acked Read 7 Done Done Sent false
15061 5130 Acked Flush 0 Done Done Sent false
15062 5131 Acked Read 7 Done Done Sent false
15063 5132 Acked Read 7 Done Done Sent false
15064 5133 Acked Write 10 Done Done Sent false
15065 5134 Acked Read 4 Done Done Sent false
15066 5135 Acked Read 3 Done Done Sent false
15067 5136 Acked Read 7 Done Done Sent false
15068 5137 Acked Read 3 Done Done Sent false
15069 5138 Acked Read 3 Done Done Sent false
15070 5139 Acked Read 5 Done Done Sent false
15071 5140 Acked Write 7 Done Done Sent false
15072 5141 Acked Write 8 Done Done Sent false
15073 5142 Acked Write 2 Done Done Sent false
15074 5143 Acked Write 4 Done Done Sent false
15075 5144 Acked Write 9 Done Done Sent false
15076 5145 Acked Read 10 Done Done Sent false
15077 5146 Acked Read 9 Done Done Sent false
15078 5147 Acked Write 6 Done Done Sent false
15079 5148 Acked Write 2 Done Done Sent false
15080 5149 Acked Read 10 Done Done Sent false
15081 5150 Acked Write 9 Done Done Sent false
15082 5151 Acked Read 10 Done Done Sent false
15083 5152 Acked Write 4 Done Done Sent false
15084 5153 Acked Read 5 Done Done Sent false
15085 5154 Acked Read 3 Done Done Sent false
15086 5155 Acked Read 3 Done Done Sent false
15087 5156 Acked Flush 0 Done Done Sent false
15088 5157 Acked Write 6 Done Done Sent false
15089 5158 Acked Write 3 Done Done Sent false
15090 5159 Acked Write 9 Done Done Sent false
15091 5160 Acked Write 8 Done Done Sent false
15092 5161 Acked Write 7 Done Done Sent false
15093 5162 Acked Read 9 Done Done Sent false
15094 5163 Acked Flush 0 Done Done Sent false
15095 5164 Acked Read 7 Done Done Sent false
15096 5165 Acked Read 4 Done Done Sent false
15097 5166 Acked Read 1 Done Done Sent false
15098 5167 Acked Read 2 Done Done Sent false
15099 5168 Acked Write 5 Done Done Sent false
15100 5169 Acked Read 5 Done Done Sent false
15101 5170 Acked Write 10 Done Done Sent false
15102 5171 Acked Read 2 Done Done Sent false
15103 5172 Acked Read 5 Done Done Sent false
15104 5173 Acked Write 2 Done Done Sent false
15105 5174 Acked Read 3 Done Done Sent false
15106 5175 Acked Read 6 Done Done Sent false
15107 5176 Acked Read 10 Done Done Sent false
15108 5177 Acked Write 9 Done Done Sent false
15109 5178 Acked Read 2 Done Done Sent false
15110 5179 Acked Flush 0 Done Done Sent false
15111 5180 Acked Read 1 Done Done Sent false
15112 5181 Acked Write 1 Done Done Sent false
15113 5182 Acked Read 2 Done Done Sent false
15114 5183 Acked Flush 0 Done Done Sent false
15115 5184 Acked Read 8 Done Done Sent false
15116 5185 Acked Read 5 Done Done Sent false
15117 5186 Acked Write 7 Done Done Sent false
15118 5187 Acked Write 6 Done Done Sent false
15119 5188 Acked Write 1 Done Done Sent false
15120 5189 Acked Write 1 Done Done Sent false
15121 5190 Acked Read 10 Done Done Sent false
15122 5191 Acked Flush 0 Done Done Sent false
15123 5192 Acked Read 10 Done Done Sent false
15124 5193 Acked Read 9 Done Done Sent false
15125 5194 Acked Write 5 Done Done Sent false
15126 5195 Acked Write 6 Done Done Sent false
15127 5196 Acked Read 10 Done Done Sent false
15128 5197 Acked Read 5 Done Done Sent false
15129 5198 Acked Read 10 Done Done Sent false
15130 5199 Acked Read 7 Done Done Sent false
15131 5200 Acked Write 3 Done Done Sent false
15132 5201 Acked Read 9 Done Done Sent false
15133 5202 Acked Read 7 Done Done Sent false
15134 5203 Acked Write 3 Done Done Sent false
15135 5204 Acked Read 6 Done Done Sent false
15136 5205 Acked Read 8 Done Done Sent false
15137 5206 Acked Flush 0 Done Done Sent false
15138 5207 Acked Write 9 Done Done Sent false
15139 5208 Acked Write 6 Done Done Sent false
15140 5209 Acked Write 4 Done Done Sent false
15141 5210 Acked Write 8 Done Done Sent false
15142 5211 Acked Read 9 Done Done Sent false
15143 5212 Acked Flush 0 Done Done Sent false
15144 5213 Acked Read 4 Done Done Sent false
15145 5214 Acked Flush 0 Done Done Sent false
15146 5215 Acked Read 10 Done Done Sent false
15147 5216 Acked Read 1 Done Done Sent false
15148 5217 Acked Read 2 Done Done Sent false
15149 5218 Acked Write 8 Done Done Sent false
15150 5219 Acked Write 2 Done Done Sent false
15151 5220 Acked Read 1 Done Done Sent false
15152 5221 Acked Write 8 Done Done Sent false
15153 5222 Acked Read 7 Done Done Sent false
15154 5223 Acked Write 1 Done Done Sent false
15155 5224 Acked Write 7 Done Done Sent false
15156 5225 Acked Read 3 Done Done Sent false
15157 5226 Acked Read 6 Done Done Sent false
15158 STATES DS:0 DS:1 DS:2 TOTAL
15159 Sent 0 0 301 301
15160 Done 301 301 0 602
15161 Skipped 0 0 0 0
15162 Error 0 0 0 0
15163 Last Flush: Some(JobId(5214)) Some(JobId(5214)) Some(JobId(4925))
15164 Downstairs last five completed: 4925 4924 4923 4922 4921
15165 Upstairs last five completed: 5226 5225 5224 5223 5222
15166 CLIENT: Up:0 ds:301 act:2
151672025-01-28T21:57:33.866ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151682025-01-28T21:57:34.617ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151692025-01-28T21:57:35.114ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151702025-01-28T21:57:35.114ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42970) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151712025-01-28T21:57:35.115ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151722025-01-28T21:57:35.116ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151732025-01-28T21:57:35.116ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151742025-01-28T21:57:35.116ZINFOcrucible: send last flush ID to this DS: Some(JobId(4925)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151752025-01-28T21:57:35.116ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151762025-01-28T21:57:35.116ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4925)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151772025-01-28T21:57:35.116ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4925)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
15178 ----------------------------------------------------------------
15179 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15180 Downstairs last five completed: 5227 5226 5225 5224 5223
15181 Upstairs last five completed: 5227 5226 5225 5224 5223
15182 CLIENT: Up:0 ds:0 act:3
15183 Replay: All jobs finished
15184 Main task has work to do, go find it
15185 got dsc Stop(1)
15186 stop 1
15187 Jan 28 21:57:37.366[8820] Got stop action so:false kr:false
15188 INFO request completed, latency_us: 392, response_code: 204, uri: /stop/cid/1, method: POST, req_id: e5c28b7b-8770-4890-b52e-2733e3c830b4, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
15189 Jan 28 21:57:37.366 INFO request completed, latency_us: 133, response_code: 200, uri: /state/cid/1, method: GET, req_id: db0f136e-9313-4fb4-be67-3f197bff4e30, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
151902025-01-28T21:57:37.367ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151912025-01-28T21:57:37.367ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151922025-01-28T21:57:37.367ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
151932025-01-28T21:57:37.367ZINFOcrucible: client task is exiting = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
15194 [8820] Exited with: ExitStatus(unix_wait_status(9))
15195 [8820][1] reports Exit
15196 Jan 28 21:57:41.369 INFO request completed, latency_us: 234, response_code: 200, uri: /state/cid/1, method: GET, req_id: 955390f7-e699-4163-ae92-aca61daa51a8, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
15197 001/300 Flush
15198 002/300 Write block 4741 len 12288 data: 4 4 4
15199 003/300 Flush
15200 004/300 Write block 4103 len 32768 data: 5 4 4 4 4 4 3 2
15201 005/300 Read block 4801 len 40960
15202 006/300 Write block 254 len 4096 data: 3
15203 007/300 Write block 1869 len 20480 data: 3 3 3 3 2
15204 008/300 Write block 1572 len 40960 data: 2 3 4 4 5 4 4 5 6 4
15205 009/300 Write block 3534 len 40960 data: 5 4 3 4 4 4 4 3 3 3
15206 010/300 Write block 3727 len 8192 data: 7 8
15207 011/300 Write block 217 len 16384 data: 3 2 3 3
15208 012/300 Read block 2176 len 28672
15209 013/300 Write block 3334 len 20480 data: 4 4 4 5 4
15210 014/300 Read block 3553 len 4096
15211 015/300 Write block 937 len 24576 data: 6 7 7 7 8 7
15212 016/300 Write block 371 len 4096 data: 4
15213 017/300 Read block 473 len 20480
15214 018/300 Write block 334 len 4096 data: 2
15215 019/300 Read block 2253 len 4096
15216 020/300 Write block 4585 len 4096 data: 2
15217 021/300 Write block 307 len 28672 data: 4 4 4 4 3 3 3
15218 022/300 Read block 1539 len 28672
15219 023/300 Write block 1910 len 12288 data: 2 2 3
15220 024/300 Read block 653 len 4096
15221 025/300 Read block 3802 len 36864
15222 026/300 Read block 1532 len 36864
15223 027/300 Flush
15224 028/300 Read block 2789 len 40960
15225 029/300 Write block 4764 len 12288 data: 3 3 3
15226 030/300 Read block 1547 len 16384
15227 031/300 Read block 4429 len 20480
15228 032/300 Write block 154 len 24576 data: 4 4 4 4 4 3
15229 033/300 Read block 513 len 36864
15230 034/300 Read block 632 len 32768
15231 035/300 Write block 1749 len 8192 data: 3 3
15232 036/300 Read block 4793 len 4096
15233 037/300 Read block 243 len 24576
15234 038/300 Write block 4690 len 40960 data: 4 4 3 3 5 5 4 5 3 2
15235 039/300 Write block 3708 len 24576 data: 3 3 3 3 3 4
15236 040/300 Write block 1746 len 24576 data: 2 3 3 4 4 3
15237 041/300 Write block 1173 len 40960 data: 4 4 3 4 4 6 8 9 8 8
15238 042/300 Write block 3236 len 40960 data: 2 2 2 3 3 2 2 3 4 5
15239 043/300 Read block 530 len 36864
15240 044/300 Write block 4222 len 20480 data: 2 2 2 2 2
15241 045/300 Read block 3292 len 16384
15242 046/300 Read block 2645 len 28672
15243 047/300 Write block 2133 len 12288 data: 7 7 7
15244 048/300 Write block 4378 len 36864 data: 3 3 4 5 5 5 4 4 3
15245 049/300 Write block 2402 len 16384 data: 2 2 2 2
15246 050/300 Read block 2714 len 16384
15247 051/300 Read block 3509 len 36864
15248 052/300 Read block 4500 len 20480
15249 053/300 Read block 126 len 4096
15250 054/300 Read block 4925 len 4096
15251 055/300 Read block 1950 len 8192
15252 056/300 Read block 4215 len 16384
15253 057/300 Flush
15254 058/300 Read block 2500 len 16384
15255 059/300 Write block 4163 len 8192 data: 7 8
15256 060/300 Flush
15257 061/300 Read block 3911 len 20480
15258 062/300 Read block 3266 len 8192
15259 063/300 Write block 4383 len 36864 data: 6 5 5 4 3 3 3 3 2
15260 064/300 Write block 2854 len 40960 data: 2 2 3 5 5 5 5 6 4 3
15261 065/300 Write block 4764 len 40960 data: 4 4 4 4 4 6 7 9 8 7
15262 066/300 Read block 2209 len 24576
15263 067/300 Read block 3801 len 28672
15264 068/300 Read block 3187 len 28672
15265 069/300 Read block 2419 len 28672
15266 070/300 Read block 1416 len 32768
15267 071/300 Read block 526 len 16384
15268 072/300 Write block 4001 len 24576 data: 5 5 5 6 6 5
15269 073/300 Read block 4530 len 12288
15270 074/300 Read block 4438 len 24576
15271 075/300 Read block 2238 len 8192
15272 076/300 Read block 2744 len 40960
15273 077/300 Write block 979 len 12288 data: 4 4 5
15274 078/300 Read block 2674 len 28672
15275 079/300 Read block 376 len 28672
15276 080/300 Read block 3226 len 24576
15277 081/300 Write block 136 len 4096 data: 3
15278 082/300 Read block 42 len 36864
15279 083/300 Read block 2560 len 12288
15280 084/300 Read block 3897 len 32768
15281 085/300 Write block 1011 len 24576 data: 4 4 5 5 4 5
15282 086/300 Read block 555 len 16384
15283 087/300 Flush
15284 088/300 Read block 603 len 8192
15285 089/300 Write block 2894 len 4096 data: 3
15286 090/300 Read block 2775 len 20480
15287 091/300 Write block 4855 len 32768 data: 4 5 5 4 4 5 5 4
15288 092/300 Write block 4355 len 8192 data: 4 5
15289 093/300 Read block 4115 len 20480
15290 094/300 Write block 51 len 8192 data: 2 2
15291 095/300 Write block 2392 len 28672 data: 3 4 5 5 6 6 6
15292 096/300 Write block 2675 len 32768 data: 5 5 5 6 6 7 6 4
15293 097/300 Write block 2416 len 12288 data: 4 4 2
15294 098/300 Write block 498 len 20480 data: 3 3 3 3 3
15295 099/300 Write block 3656 len 36864 data: 2 2 2 2 2 2 2 3 3
15296 100/300 Write block 1769 len 20480 data: 4 4 4 4 4
15297 101/300 Write block 987 len 36864 data: 2 2 2 2 2 2 2 2 2
15298 102/300 Write block 2382 len 32768 data: 5 5 5 4 4 4 5 5
15299 103/300 Read block 791 len 4096
15300 104/300 Read block 4234 len 16384
15301 105/300 Write block 4578 len 4096 data: 4
15302 106/300 Read block 91 len 20480
15303 107/300 Write block 2038 len 36864 data: 3 3 3 2 2 3 3 3 3
15304 108/300 Read block 1244 len 28672
15305 109/300 Write block 1820 len 8192 data: 4 4
15306 110/300 Write block 3591 len 4096 data: 5
15307 111/300 Read block 1925 len 20480
15308 112/300 Read block 4110 len 12288
15309 113/300 Read block 3933 len 12288
15310 114/300 Write block 2766 len 32768 data: 3 3 3 4 4 4 4 5
15311 115/300 Read block 357 len 16384
15312 116/300 Read block 1334 len 8192
15313 117/300 Write block 64 len 4096 data: 7
15314 118/300 Write block 1251 len 8192 data: 4 5
15315 119/300 Write block 1318 len 20480 data: 3 2 2 3 3
15316 120/300 Write block 3993 len 8192 data: 3 3
15317 121/300 Write block 4426 len 36864 data: 6 5 5 4 4 3 2 3 3
15318 122/300 Read block 1579 len 28672
15319 123/300 Write block 779 len 24576 data: 3 3 2 3 4 5
15320 124/300 Write block 3608 len 8192 data: 4 4
15321 125/300 Read block 525 len 8192
15322 126/300 Read block 4300 len 16384
15323 127/300 Read block 144 len 8192
15324 128/300 Write block 4364 len 24576 data: 5 3 2 2 2 2
15325 129/300 Read block 3518 len 40960
15326 130/300 Read block 2402 len 20480
15327 131/300 Write block 2152 len 36864 data: 7 7 8 8 6 7 6 8 7
15328 132/300 Read block 3090 len 40960
15329 133/300 Write block 100 len 24576 data: 3 3 3 3 3 3
15330 134/300 Read block 3054 len 20480
15331 135/300 Write block 2501 len 12288 data: 4 4 3
15332 136/300 Read block 337 len 20480
15333 137/300 Write block 4323 len 24576 data: 3 3 2 3 3 3
15334 138/300 Write block 1141 len 24576 data: 4 4 4 4 4 3
15335 139/300 Read block 1578 len 4096
15336 140/300 Read block 2027 len 12288
15337 141/300 Read block 2313 len 28672
15338 142/300 Read block 197 len 12288
15339 143/300 Write block 2678 len 36864 data: 7 7 8 7 5 3 3 3 3
15340 144/300 Read block 2846 len 40960
15341 145/300 Read block 3628 len 12288
15342 146/300 Flush
15343 147/300 Write block 1158 len 4096 data: 4
15344 148/300 Write block 1698 len 32768 data: 4 4 5 3 3 4 4 4
15345 149/300 Read block 3513 len 32768
15346 150/300 Write block 4156 len 24576 data: 3 4 4 4 6 6
15347 151/300 Flush
15348 152/300 Read block 1086 len 32768
15349 153/300 Read block 3200 len 16384
15350 154/300 Write block 3325 len 4096 data: 5
15351 155/300 Read block 960 len 32768
15352 156/300 Write block 344 len 24576 data: 4 6 6 6 5 7
15353 157/300 Write block 1101 len 20480 data: 4 3 3 3 6
15354 158/300 Write block 2063 len 32768 data: 2 2 2 2 3 4 5 6
15355 159/300 Read block 1176 len 40960
15356 160/300 Write block 648 len 4096 data: 4
15357 161/300 Write block 3905 len 8192 data: 4 5
15358 162/300 Read block 4565 len 32768
15359 163/300 Flush
15360 164/300 Write block 2806 len 24576 data: 3 3 4 5 4 4
15361 165/300 Read block 1999 len 24576
15362 166/300 Write block 1501 len 12288 data: 2 3 3
15363 167/300 Write block 4984 len 36864 data: 6 6 7 6 6 4 4 2 2
15364 168/300 Write block 1045 len 36864 data: 5 7 7 6 4 4 5 8 8
15365 169/300 Write block 4001 len 40960 data: 6 6 6 7 7 6 5 5 5 5
15366 170/300 Write block 4482 len 28672 data: 9 8 8 7 9 6 6
15367 171/300 Read block 1800 len 16384
15368 172/300 Read block 2987 len 28672
15369 173/300 Read block 567 len 24576
15370 174/300 Write block 1246 len 24576 data: 3 2 3 3 3 5
15371 175/300 Read block 4173 len 20480
15372 176/300 Write block 2662 len 12288 data: 7 6 5
15373 177/300 Write block 6 len 24576 data: 5 5 4 3 3 3
15374 178/300 Write block 3873 len 32768 data: 4 3 3 3 3 4 4 4
15375 179/300 Write block 4319 len 28672 data: 4 3 3 3 4 4 3
15376 180/300 Write block 4328 len 40960 data: 4 4 4 4 4 3 4 4 5 6
15377 181/300 Write block 454 len 24576 data: 3 3 4 4 5 6
15378 182/300 Write block 2573 len 4096 data: 2
15379 183/300 Read block 4300 len 12288
15380 184/300 Read block 2512 len 4096
15381 185/300 Write block 2941 len 36864 data: 5 5 5 4 5 5 4 4 3
15382 186/300 Write block 1302 len 16384 data: 3 3 4 4
15383 187/300 Write block 3830 len 28672 data: 3 3 2 2 3 3 3
15384 188/300 Read block 2772 len 36864
15385 189/300 Write block 652 len 8192 data: 7 7
15386 190/300 Read block 2781 len 16384
15387 191/300 Write block 3633 len 20480 data: 4 3 5 5 5
15388 192/300 Write block 1079 len 16384 data: 5 5 5 5
15389 193/300 Read block 2516 len 8192
15390 194/300 Read block 3134 len 8192
15391 195/300 Write block 203 len 16384 data: 2 4 5 6
15392 196/300 Read block 546 len 8192
15393 197/300 Write block 939 len 20480 data: 8 8 9 8 5
15394 198/300 Read block 1843 len 12288
15395 199/300 Read block 3037 len 40960
15396 200/300 Read block 2101 len 16384
15397 201/300 Write block 3795 len 24576 data: 4 4 3 2 2 4
15398 202/300 Write block 855 len 28672 data: 3 3 3 2 3 3 3
15399 203/300 Read block 3886 len 28672
15400 204/300 Read block 4809 len 12288
15401 205/300 Write block 2564 len 20480 data: 3 3 3 3 2
15402 206/300 Write block 3544 len 20480 data: 4 5 4 5 5
15403 207/300 Read block 3248 len 40960
15404 208/300 Write block 4406 len 32768 data: 4 3 4 4 4 5 5 5
15405 209/300 Read block 522 len 8192
15406 210/300 Write block 2810 len 16384 data: 5 5 3 3
15407 211/300 Write block 4020 len 28672 data: 3 4 5 4 4 4 4
15408 212/300 Read block 1732 len 16384
15409 213/300 Read block 11 len 20480
15410 214/300 Read block 1774 len 32768
15411 215/300 Read block 2615 len 4096
15412 216/300 Read block 891 len 4096
15413 217/300 Read block 735 len 28672
15414 218/300 Write block 300 len 24576 data: 5 6 5 5 7 6
15415 219/300 Write block 4790 len 28672 data: 2 2 4 4 5 5 4
15416 220/300 Flush
15417 221/300 Write block 4753 len 20480 data: 5 5 4 2 2
15418 222/300 Write block 1974 len 36864 data: 6 5 5 5 4 4 5 5 5
15419 223/300 Read block 3159 len 8192
15420 224/300 Read block 4157 len 16384
15421 225/300 Write block 150 len 32768 data: 2 2 4 4 5 5 5 5
15422 226/300 Read block 3646 len 20480
15423 227/300 Read block 3019 len 4096
15424 228/300 Read block 1815 len 16384
15425 229/300 Write block 1448 len 32768 data: 3 3 2 2 4 4 5 5
15426 230/300 Write block 2701 len 24576 data: 2 3 3 3 4 4
15427 231/300 Read block 2970 len 8192
15428 232/300 Read block 1186 len 28672
15429 233/300 Write block 3640 len 40960 data: 3 4 6 5 4 4 4 4 5 4
15430 234/300 Read block 1462 len 40960
15431 235/300 Write block 3418 len 4096 data: 3
15432 236/300 Read block 1412 len 24576
15433 237/300 Read block 4678 len 40960
15434 238/300 Write block 3236 len 16384 data: 3 3 3 4
15435 239/300 Read block 425 len 28672
15436 240/300 Write block 193 len 40960 data: 3 2 2 2 2 2 2 2 2 2
15437 241/300 Read block 2254 len 40960
15438 242/300 Read block 349 len 24576
15439 243/300 Flush
15440 244/300 Write block 1808 len 20480 data: 3 3 3 3 3
15441 245/300 Write block 4496 len 4096 data: 2
15442 246/300 Write block 520 len 20480 data: 2 4 4 5 5
15443 247/300 Read block 4785 len 16384
15444 248/300 Write block 4426 len 28672 data: 7 6 6 5 5 4 3
15445 249/300 Read block 1328 len 8192
15446 250/300 Write block 1249 len 28672 data: 4 4 6 6 5 6 6
15447 251/300 Read block 1931 len 32768
15448 252/300 Read block 4404 len 4096
15449 253/300 Read block 2532 len 8192
15450 254/300 Read block 1787 len 32768
15451 255/300 Read block 2411 len 8192
15452 256/300 Flush
15453 257/300 Read block 2458 len 32768
15454 258/300 Flush
15455 259/300 Read block 2557 len 36864
15456 260/300 Read block 2192 len 12288
15457 261/300 Write block 738 len 4096 data: 5
15458 262/300 Flush
15459 263/300 Read block 2986 len 24576
15460 264/300 Write block 3786 len 32768 data: 4 4 5 5 4 3 3 4
15461 265/300 Write block 4938 len 8192 data: 2 2
15462 266/300 Read block 3554 len 28672
15463 267/300 Write block 3770 len 36864 data: 5 5 4 3 3 3 3 4 3
15464 268/300 Read block 838 len 40960
15465 269/300 Read block 172 len 16384
15466 270/300 Flush
15467 271/300 Write block 2395 len 4096 data: 6
15468 272/300 Read block 3922 len 24576
15469 273/300 Read block 4838 len 12288
15470 274/300 Write block 2170 len 8192 data: 5 4
15471 275/300 Read block 3870 len 8192
15472 276/300 Read block 4883 len 4096
15473 277/300 Write block 2470 len 4096 data: 3
15474 278/300 Flush
15475 279/300 Write block 2283 len 32768 data: 4 3 4 4 5 5 5 5
15476 280/300 Flush
15477 281/300 Write block 652 len 28672 data: 8 8 5 4 3 3 2
15478 282/300 Read block 2381 len 4096
15479 283/300 Write block 500 len 24576 data: 4 4 4 3 2 2
15480 284/300 Flush
15481 285/300 Flush
15482 286/300 Read block 4003 len 4096
15483 287/300 Read block 628 len 40960
15484 288/300 Flush
15485 289/300 Read block 3281 len 32768
15486 290/300 Read block 2562 len 8192
15487 291/300 Read block 2073 len 4096
15488 292/300 Write block 2360 len 24576 data: 2 2 2 2 2 2
15489 293/300 Read block 4003 len 8192
15490 294/300 Flush
15491 295/300 Read block 2791 len 28672
15492 296/300 Read block 1648 len 12288
15493 297/300 Read block 2411 len 12288
15494 298/300 Write block 3221 len 40960 data: 5 4 4 5 6 5 5 5 4 4
15495 299/300 Read block 2220 len 12288
15496 300/300 Read block 636 len 32768
15497 301/300 Flush
15498 Main task has work to do, go find it
15499 got dsc Start(1)
15500 start 1
15501 Jan 28 21:57:41.601 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/1, method: POST, req_id: 16f74a83-28af-4cd6-a404-82eebb1d5732, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
15502 Replay: started 1, returned:Ok(())
15503 ----------------------------------------------------------------
15504 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15505 States: Active Offline Active
15506 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15507 5228 Acked Flush 0 Done Sent Done false
15508 5229 Acked Write 3 Done Sent Done false
15509 5230 Acked Flush 0 Done Sent Done false
15510 5231 Acked Write 8 Done Sent Done false
15511 5232 Acked Read 10 Done Sent Done false
15512 5233 Acked Write 1 Done Sent Done false
15513 5234 Acked Write 5 Done Sent Done false
15514 5235 Acked Write 10 Done Sent Done false
15515 5236 Acked Write 10 Done Sent Done false
15516 5237 Acked Write 2 Done Sent Done false
15517 5238 Acked Write 4 Done Sent Done false
15518 5239 Acked Read 7 Done Sent Done false
15519 5240 Acked Write 5 Done Sent Done false
15520 5241 Acked Read 1 Done Sent Done false
15521 5242 Acked Write 6 Done Sent Done false
15522 5243 Acked Write 1 Done Sent Done false
15523 5244 Acked Read 5 Done Sent Done false
15524 5245 Acked Write 1 Done Sent Done false
15525 5246 Acked Read 1 Done Sent Done false
15526 5247 Acked Write 1 Done Sent Done false
15527 5248 Acked Write 7 Done Sent Done false
15528 5249 Acked Read 7 Done Sent Done false
15529 5250 Acked Write 3 Done Sent Done false
15530 5251 Acked Read 1 Done Sent Done false
15531 5252 Acked Read 9 Done Sent Done false
15532 5253 Acked Read 9 Done Sent Done false
15533 5254 Acked Flush 0 Done Sent Done false
15534 5255 Acked Read 10 Done Sent Done false
15535 5256 Acked Write 3 Done Sent Done false
15536 5257 Acked Read 4 Done Sent Done false
15537 5258 Acked Read 5 Done Sent Done false
15538 5259 Acked Write 6 Done Sent Done false
15539 5260 Acked Read 9 Done Sent Done false
15540 5261 Acked Read 8 Done Sent Done false
15541 5262 Acked Write 2 Done Sent Done false
15542 5263 Acked Read 1 Done Sent Done false
15543 5264 Acked Read 6 Done Sent Done false
15544 5265 Acked Write 10 Done Sent Done false
15545 5266 Acked Write 6 Done Sent Done false
15546 5267 Acked Write 6 Done Sent Done false
15547 5268 Acked Write 10 Done Sent Done false
15548 5269 Acked Write 10 Done Sent Done false
15549 5270 Acked Read 9 Done Sent Done false
15550 5271 Acked Write 5 Done Sent Done false
15551 5272 Acked Read 4 Done Sent Done false
15552 5273 Acked Read 7 Done Sent Done false
15553 5274 Acked Write 3 Done Sent Done false
15554 5275 Acked Write 9 Done Sent Done false
15555 5276 Acked Write 4 Done Sent Done false
15556 5277 Acked Read 4 Done Sent Done false
15557 5278 Acked Read 9 Done Sent Done false
15558 5279 Acked Read 5 Done Sent Done false
15559 5280 Acked Read 1 Done Sent Done false
15560 5281 Acked Read 1 Done Sent Done false
15561 5282 Acked Read 2 Done Sent Done false
15562 5283 Acked Read 4 Done Sent Done false
15563 5284 Acked Flush 0 Done Sent Done false
15564 5285 Acked Read 4 Done Sent Done false
15565 5286 Acked Write 2 Done Sent Done false
15566 5287 Acked Flush 0 Done Sent Done false
15567 5288 Acked Read 5 Done Sent Done false
15568 5289 Acked Read 2 Done Sent Done false
15569 5290 Acked Write 9 Done Sent Done false
15570 5291 Acked Write 10 Done Sent Done false
15571 5292 Acked Write 10 Done Sent Done false
15572 5293 Acked Read 6 Done Sent Done false
15573 5294 Acked Read 7 Done Sent Done false
15574 5295 Acked Read 7 Done Sent Done false
15575 5296 Acked Read 7 Done Sent Done false
15576 5297 Acked Read 8 Done Sent Done false
15577 5298 Acked Read 4 Done Sent Done false
15578 5299 Acked Write 6 Done Sent Done false
15579 5300 Acked Read 3 Done Sent Done false
15580 5301 Acked Read 6 Done Sent Done false
15581 5302 Acked Read 2 Done Sent Done false
15582 5303 Acked Read 10 Done Sent Done false
15583 5304 Acked Write 3 Done Sent Done false
15584 5305 Acked Read 7 Done Sent Done false
15585 5306 Acked Read 7 Done Sent Done false
15586 5307 Acked Read 6 Done Sent Done false
15587 5308 Acked Write 1 Done Sent Done false
15588 5309 Acked Read 9 Done Sent Done false
15589 5310 Acked Read 3 Done Sent Done false
15590 5311 Acked Read 8 Done Sent Done false
15591 5312 Acked Write 6 Done Sent Done false
15592 5313 Acked Read 4 Done Sent Done false
15593 5314 Acked Flush 0 Done Sent Done false
15594 5315 Acked Read 2 Done Sent Done false
15595 5316 Acked Write 1 Done Sent Done false
15596 5317 Acked Read 5 Done Sent Done false
15597 5318 Acked Write 8 Done Sent Done false
15598 5319 Acked Write 2 Done Sent Done false
15599 5320 Acked Read 5 Done Sent Done false
15600 5321 Acked Write 2 Done Sent Done false
15601 5322 Acked Write 7 Done Sent Done false
15602 5323 Acked Write 8 Done Sent Done false
15603 5324 Acked Write 3 Done Sent Done false
15604 5325 Acked Write 5 Done Sent Done false
15605 5326 Acked Write 9 Done Sent Done false
15606 5327 Acked Write 5 Done Sent Done false
15607 5328 Acked Write 9 Done Sent Done false
15608 5329 Acked Write 8 Done Sent Done false
15609 5330 Acked Read 1 Done Sent Done false
15610 5331 Acked Read 4 Done Sent Done false
15611 5332 Acked Write 1 Done Sent Done false
15612 5333 Acked Read 5 Done Sent Done false
15613 5334 Acked Write 9 Done Sent Done false
15614 5335 Acked Read 7 Done Sent Done false
15615 5336 Acked Write 2 Done Sent Done false
15616 5337 Acked Write 1 Done Sent Done false
15617 5338 Acked Read 5 Done Sent Done false
15618 5339 Acked Read 3 Done Sent Done false
15619 5340 Acked Read 3 Done Sent Done false
15620 5341 Acked Write 8 Done Sent Done false
15621 5342 Acked Read 4 Done Sent Done false
15622 5343 Acked Read 2 Done Sent Done false
15623 5344 Acked Write 1 Done Sent Done false
15624 5345 Acked Write 2 Done Sent Done false
15625 5346 Acked Write 5 Done Sent Done false
15626 5347 Acked Write 2 Done Sent Done false
15627 5348 Acked Write 9 Done Sent Done false
15628 5349 Acked Read 7 Done Sent Done false
15629 5350 Acked Write 6 Done Sent Done false
15630 5351 Acked Write 2 Done Sent Done false
15631 5352 Acked Read 2 Done Sent Done false
15632 5353 Acked Read 4 Done Sent Done false
15633 5354 Acked Read 2 Done Sent Done false
15634 5355 Acked Write 6 Done Sent Done false
15635 5356 Acked Read 10 Done Sent Done false
15636 5357 Acked Read 5 Done Sent Done false
15637 5358 Acked Write 9 Done Sent Done false
15638 5359 Acked Read 10 Done Sent Done false
15639 5360 Acked Write 6 Done Sent Done false
15640 5361 Acked Read 5 Done Sent Done false
15641 5362 Acked Write 3 Done Sent Done false
15642 5363 Acked Read 5 Done Sent Done false
15643 5364 Acked Write 6 Done Sent Done false
15644 5365 Acked Write 6 Done Sent Done false
15645 5366 Acked Read 1 Done Sent Done false
15646 5367 Acked Read 3 Done Sent Done false
15647 5368 Acked Read 7 Done Sent Done false
15648 5369 Acked Read 3 Done Sent Done false
15649 5370 Acked Write 9 Done Sent Done false
15650 5371 Acked Read 10 Done Sent Done false
15651 5372 Acked Read 3 Done Sent Done false
15652 5373 Acked Flush 0 Done Sent Done false
15653 5374 Acked Write 1 Done Sent Done false
15654 5375 Acked Write 8 Done Sent Done false
15655 5376 Acked Read 8 Done Sent Done false
15656 5377 Acked Write 6 Done Sent Done false
15657 5378 Acked Flush 0 Done Sent Done false
15658 5379 Acked Read 8 Done Sent Done false
15659 5380 Acked Read 4 Done Sent Done false
15660 5381 Acked Write 1 Done Sent Done false
15661 5382 Acked Read 8 Done Sent Done false
15662 5383 Acked Write 6 Done Sent Done false
15663 5384 Acked Write 5 Done Sent Done false
15664 5385 Acked Write 8 Done Sent Done false
15665 5386 Acked Read 10 Done Sent Done false
15666 5387 Acked Write 1 Done Sent Done false
15667 5388 Acked Write 2 Done Sent Done false
15668 5389 Acked Read 8 Done Sent Done false
15669 5390 Acked Flush 0 Done Sent Done false
15670 5391 Acked Write 6 Done Sent Done false
15671 5392 Acked Read 6 Done Sent Done false
15672 5393 Acked Write 3 Done Sent Done false
15673 5394 Acked Write 9 Done Sent Done false
15674 5395 Acked Write 9 Done Sent Done false
15675 5396 Acked Write 10 Done Sent Done false
15676 5397 Acked Write 7 Done Sent Done false
15677 5398 Acked Read 4 Done Sent Done false
15678 5399 Acked Read 7 Done Sent Done false
15679 5400 Acked Read 6 Done Sent Done false
15680 5401 Acked Write 6 Done Sent Done false
15681 5402 Acked Read 5 Done Sent Done false
15682 5403 Acked Write 3 Done Sent Done false
15683 5404 Acked Write 6 Done Sent Done false
15684 5405 Acked Write 8 Done Sent Done false
15685 5406 Acked Write 7 Done Sent Done false
15686 5407 Acked Write 10 Done Sent Done false
15687 5408 Acked Write 6 Done Sent Done false
15688 5409 Acked Write 1 Done Sent Done false
15689 5410 Acked Read 3 Done Sent Done false
15690 5411 Acked Read 1 Done Sent Done false
15691 5412 Acked Write 9 Done Sent Done false
15692 5413 Acked Write 4 Done Sent Done false
15693 5414 Acked Write 7 Done Sent Done false
15694 5415 Acked Read 9 Done Sent Done false
15695 5416 Acked Write 2 Done Sent Done false
15696 5417 Acked Read 4 Done Sent Done false
15697 5418 Acked Write 5 Done Sent Done false
15698 5419 Acked Write 4 Done Sent Done false
15699 5420 Acked Read 2 Done Sent Done false
15700 5421 Acked Read 2 Done Sent Done false
15701 5422 Acked Write 4 Done Sent Done false
15702 5423 Acked Read 2 Done Sent Done false
15703 5424 Acked Write 5 Done Sent Done false
15704 5425 Acked Read 3 Done Sent Done false
15705 5426 Acked Read 10 Done Sent Done false
15706 5427 Acked Read 4 Done Sent Done false
15707 5428 Acked Write 6 Done Sent Done false
15708 5429 Acked Write 7 Done Sent Done false
15709 5430 Acked Read 7 Done Sent Done false
15710 5431 Acked Read 3 Done Sent Done false
15711 5432 Acked Write 5 Done Sent Done false
15712 5433 Acked Write 5 Done Sent Done false
15713 5434 Acked Read 10 Done Sent Done false
15714 5435 Acked Write 8 Done Sent Done false
15715 5436 Acked Read 2 Done Sent Done false
15716 5437 Acked Write 4 Done Sent Done false
15717 5438 Acked Write 7 Done Sent Done false
15718 5439 Acked Read 4 Done Sent Done false
15719 5440 Acked Read 5 Done Sent Done false
15720 5441 Acked Read 8 Done Sent Done false
15721 5442 Acked Read 1 Done Sent Done false
15722 5443 Acked Read 1 Done Sent Done false
15723 5444 Acked Read 7 Done Sent Done false
15724 5445 Acked Write 6 Done Sent Done false
15725 5446 Acked Write 7 Done Sent Done false
15726 5447 Acked Flush 0 Done Sent Done false
15727 5448 Acked Write 5 Done Sent Done false
15728 5449 Acked Write 9 Done Sent Done false
15729 5450 Acked Read 2 Done Sent Done false
15730 5451 Acked Read 4 Done Sent Done false
15731 5452 Acked Write 8 Done Sent Done false
15732 5453 Acked Read 5 Done Sent Done false
15733 5454 Acked Read 1 Done Sent Done false
15734 5455 Acked Read 4 Done Sent Done false
15735 5456 Acked Write 8 Done Sent Done false
15736 5457 Acked Write 6 Done Sent Done false
15737 5458 Acked Read 2 Done Sent Done false
15738 5459 Acked Read 7 Done Sent Done false
15739 5460 Acked Write 10 Done Sent Done false
15740 5461 Acked Read 10 Done Sent Done false
15741 5462 Acked Write 1 Done Sent Done false
15742 5463 Acked Read 6 Done Sent Done false
15743 5464 Acked Read 10 Done Sent Done false
15744 5465 Acked Write 4 Done Sent Done false
15745 5466 Acked Read 7 Done Sent Done false
15746 5467 Acked Write 10 Done Sent Done false
15747 5468 Acked Read 10 Done Sent Done false
15748 5469 Acked Read 6 Done Sent Done false
15749 5470 Acked Flush 0 Done Sent Done false
15750 5471 Acked Write 5 Done Sent Done false
15751 5472 Acked Write 1 Done Sent Done false
15752 5473 Acked Write 5 Done Sent Done false
15753 5474 Acked Read 4 Done Sent Done false
15754 5475 Acked Write 7 Done Sent Done false
15755 5476 Acked Read 2 Done Sent Done false
15756 5477 Acked Write 7 Done Sent Done false
15757 5478 Acked Read 8 Done Sent Done false
15758 5479 Acked Read 1 Done Sent Done false
15759 5480 Acked Read 2 Done Sent Done false
15760 5481 Acked Read 8 Done Sent Done false
15761 5482 Acked Read 2 Done Sent Done false
15762 5483 Acked Flush 0 Done Sent Done false
15763 5484 Acked Read 8 Done Sent Done false
15764 5485 Acked Flush 0 Done Sent Done false
15765 5486 Acked Read 9 Done Sent Done false
15766 5487 Acked Read 3 Done Sent Done false
15767 5488 Acked Write 1 Done Sent Done false
15768 5489 Acked Flush 0 Done Sent Done false
15769 5490 Acked Read 6 Done Sent Done false
15770 5491 Acked Write 8 Done Sent Done false
15771 5492 Acked Write 2 Done Sent Done false
15772 5493 Acked Read 7 Done Sent Done false
15773 5494 Acked Write 9 Done Sent Done false
15774 5495 Acked Read 10 Done Sent Done false
15775 5496 Acked Read 4 Done Sent Done false
15776 5497 Acked Flush 0 Done Sent Done false
15777 5498 Acked Write 1 Done Sent Done false
15778 5499 Acked Read 6 Done Sent Done false
15779 5500 Acked Read 3 Done Sent Done false
15780 5501 Acked Write 2 Done Sent Done false
15781 5502 Acked Read 2 Done Sent Done false
15782 5503 Acked Read 1 Done Sent Done false
15783 5504 Acked Write 1 Done Sent Done false
15784 5505 Acked Flush 0 Done Sent Done false
15785 5506 Acked Write 8 Done Sent Done false
15786 5507 Acked Flush 0 Done Sent Done false
15787 5508 Acked Write 7 Done Sent Done false
15788 5509 Acked Read 1 Done Sent Done false
15789 5510 Acked Write 6 Done Sent Done false
15790 5511 Acked Flush 0 Done Sent Done false
15791 5512 Acked Flush 0 Done Sent Done false
15792 5513 Acked Read 1 Done Sent Done false
15793 5514 Acked Read 10 Done Sent Done false
15794 5515 Acked Flush 0 Done Sent Done false
15795 5516 Acked Read 8 Done Sent Done false
15796 5517 Acked Read 2 Done Sent Done false
15797 5518 Acked Read 1 Done Sent Done false
15798 5519 Acked Write 6 Done Sent Done false
15799 5520 Acked Read 2 Done Sent Done false
15800 5521 Acked Flush 0 Done Sent Done false
15801 5522 Acked Read 7 Done Sent Done false
15802 5523 Acked Read 3 Done Sent Done false
15803 5524 Acked Read 3 Done Sent Done false
15804 5525 Acked Write 10 Done Sent Done false
15805 5526 Acked Read 3 Done Sent Done false
15806 5527 Acked Read 8 Done Sent Done false
15807 5528 Acked Flush 0 Done Sent Done false
15808 STATES DS:0 DS:1 DS:2 TOTAL
15809 Sent 0 301 0 301
15810 Done 301 0 301 602
15811 Skipped 0 0 0 0
15812 Error 0 0 0 0
15813 Last Flush: Some(JobId(5528)) Some(JobId(5227)) Some(JobId(5528))
15814 Downstairs last five completed: 5227 5226 5225 5224 5223
15815 Upstairs last five completed: 5528 5527 5526 5525 5524
15816 CLIENT: Up:0 ds:301 act:2
158172025-01-28T21:57:42.103ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
158182025-01-28T21:57:42.625ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
15819 [8820] got start action
15820 [8820] I am going to restart
15821 Starting downstairs at port 8820
15822 Make output file at "/tmp/dsc/downstairs-8820.txt"
15823 [8820][1] reports Starting
15824 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1217)
15825 [8820][1] reports Running
158262025-01-28T21:57:43.626ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
158272025-01-28T21:57:44.628ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
15828 ----------------------------------------------------------------
15829 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15830 States: Active Offline Active
15831 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15832 5228 Acked Flush 0 Done Sent Done false
15833 5229 Acked Write 3 Done Sent Done false
15834 5230 Acked Flush 0 Done Sent Done false
15835 5231 Acked Write 8 Done Sent Done false
15836 5232 Acked Read 10 Done Sent Done false
15837 5233 Acked Write 1 Done Sent Done false
15838 5234 Acked Write 5 Done Sent Done false
15839 5235 Acked Write 10 Done Sent Done false
15840 5236 Acked Write 10 Done Sent Done false
15841 5237 Acked Write 2 Done Sent Done false
15842 5238 Acked Write 4 Done Sent Done false
15843 5239 Acked Read 7 Done Sent Done false
15844 5240 Acked Write 5 Done Sent Done false
15845 5241 Acked Read 1 Done Sent Done false
15846 5242 Acked Write 6 Done Sent Done false
15847 5243 Acked Write 1 Done Sent Done false
15848 5244 Acked Read 5 Done Sent Done false
15849 5245 Acked Write 1 Done Sent Done false
15850 5246 Acked Read 1 Done Sent Done false
15851 5247 Acked Write 1 Done Sent Done false
15852 5248 Acked Write 7 Done Sent Done false
15853 5249 Acked Read 7 Done Sent Done false
15854 5250 Acked Write 3 Done Sent Done false
15855 5251 Acked Read 1 Done Sent Done false
15856 5252 Acked Read 9 Done Sent Done false
15857 5253 Acked Read 9 Done Sent Done false
15858 5254 Acked Flush 0 Done Sent Done false
15859 5255 Acked Read 10 Done Sent Done false
15860 5256 Acked Write 3 Done Sent Done false
15861 5257 Acked Read 4 Done Sent Done false
15862 5258 Acked Read 5 Done Sent Done false
15863 5259 Acked Write 6 Done Sent Done false
15864 5260 Acked Read 9 Done Sent Done false
15865 5261 Acked Read 8 Done Sent Done false
15866 5262 Acked Write 2 Done Sent Done false
15867 5263 Acked Read 1 Done Sent Done false
15868 5264 Acked Read 6 Done Sent Done false
15869 5265 Acked Write 10 Done Sent Done false
15870 5266 Acked Write 6 Done Sent Done false
15871 5267 Acked Write 6 Done Sent Done false
15872 5268 Acked Write 10 Done Sent Done false
15873 5269 Acked Write 10 Done Sent Done false
15874 5270 Acked Read 9 Done Sent Done false
15875 5271 Acked Write 5 Done Sent Done false
15876 5272 Acked Read 4 Done Sent Done false
15877 5273 Acked Read 7 Done Sent Done false
15878 5274 Acked Write 3 Done Sent Done false
15879 5275 Acked Write 9 Done Sent Done false
15880 5276 Acked Write 4 Done Sent Done false
15881 5277 Acked Read 4 Done Sent Done false
15882 5278 Acked Read 9 Done Sent Done false
15883 5279 Acked Read 5 Done Sent Done false
15884 5280 Acked Read 1 Done Sent Done false
15885 5281 Acked Read 1 Done Sent Done false
15886 5282 Acked Read 2 Done Sent Done false
15887 5283 Acked Read 4 Done Sent Done false
15888 5284 Acked Flush 0 Done Sent Done false
15889 5285 Acked Read 4 Done Sent Done false
15890 5286 Acked Write 2 Done Sent Done false
15891 5287 Acked Flush 0 Done Sent Done false
15892 5288 Acked Read 5 Done Sent Done false
15893 5289 Acked Read 2 Done Sent Done false
15894 5290 Acked Write 9 Done Sent Done false
15895 5291 Acked Write 10 Done Sent Done false
15896 5292 Acked Write 10 Done Sent Done false
15897 5293 Acked Read 6 Done Sent Done false
15898 5294 Acked Read 7 Done Sent Done false
15899 5295 Acked Read 7 Done Sent Done false
15900 5296 Acked Read 7 Done Sent Done false
15901 5297 Acked Read 8 Done Sent Done false
15902 5298 Acked Read 4 Done Sent Done false
15903 5299 Acked Write 6 Done Sent Done false
15904 5300 Acked Read 3 Done Sent Done false
15905 5301 Acked Read 6 Done Sent Done false
15906 5302 Acked Read 2 Done Sent Done false
15907 5303 Acked Read 10 Done Sent Done false
15908 5304 Acked Write 3 Done Sent Done false
15909 5305 Acked Read 7 Done Sent Done false
15910 5306 Acked Read 7 Done Sent Done false
15911 5307 Acked Read 6 Done Sent Done false
15912 5308 Acked Write 1 Done Sent Done false
15913 5309 Acked Read 9 Done Sent Done false
15914 5310 Acked Read 3 Done Sent Done false
15915 5311 Acked Read 8 Done Sent Done false
15916 5312 Acked Write 6 Done Sent Done false
15917 5313 Acked Read 4 Done Sent Done false
15918 5314 Acked Flush 0 Done Sent Done false
15919 5315 Acked Read 2 Done Sent Done false
15920 5316 Acked Write 1 Done Sent Done false
15921 5317 Acked Read 5 Done Sent Done false
15922 5318 Acked Write 8 Done Sent Done false
15923 5319 Acked Write 2 Done Sent Done false
15924 5320 Acked Read 5 Done Sent Done false
15925 5321 Acked Write 2 Done Sent Done false
15926 5322 Acked Write 7 Done Sent Done false
15927 5323 Acked Write 8 Done Sent Done false
15928 5324 Acked Write 3 Done Sent Done false
15929 5325 Acked Write 5 Done Sent Done false
15930 5326 Acked Write 9 Done Sent Done false
15931 5327 Acked Write 5 Done Sent Done false
15932 5328 Acked Write 9 Done Sent Done false
15933 5329 Acked Write 8 Done Sent Done false
15934 5330 Acked Read 1 Done Sent Done false
15935 5331 Acked Read 4 Done Sent Done false
15936 5332 Acked Write 1 Done Sent Done false
15937 5333 Acked Read 5 Done Sent Done false
15938 5334 Acked Write 9 Done Sent Done false
15939 5335 Acked Read 7 Done Sent Done false
15940 5336 Acked Write 2 Done Sent Done false
15941 5337 Acked Write 1 Done Sent Done false
15942 5338 Acked Read 5 Done Sent Done false
15943 5339 Acked Read 3 Done Sent Done false
15944 5340 Acked Read 3 Done Sent Done false
15945 5341 Acked Write 8 Done Sent Done false
15946 5342 Acked Read 4 Done Sent Done false
15947 5343 Acked Read 2 Done Sent Done false
15948 5344 Acked Write 1 Done Sent Done false
15949 5345 Acked Write 2 Done Sent Done false
15950 5346 Acked Write 5 Done Sent Done false
15951 5347 Acked Write 2 Done Sent Done false
15952 5348 Acked Write 9 Done Sent Done false
15953 5349 Acked Read 7 Done Sent Done false
15954 5350 Acked Write 6 Done Sent Done false
15955 5351 Acked Write 2 Done Sent Done false
15956 5352 Acked Read 2 Done Sent Done false
15957 5353 Acked Read 4 Done Sent Done false
15958 5354 Acked Read 2 Done Sent Done false
15959 5355 Acked Write 6 Done Sent Done false
15960 5356 Acked Read 10 Done Sent Done false
15961 5357 Acked Read 5 Done Sent Done false
15962 5358 Acked Write 9 Done Sent Done false
15963 5359 Acked Read 10 Done Sent Done false
15964 5360 Acked Write 6 Done Sent Done false
15965 5361 Acked Read 5 Done Sent Done false
15966 5362 Acked Write 3 Done Sent Done false
15967 5363 Acked Read 5 Done Sent Done false
15968 5364 Acked Write 6 Done Sent Done false
15969 5365 Acked Write 6 Done Sent Done false
15970 5366 Acked Read 1 Done Sent Done false
15971 5367 Acked Read 3 Done Sent Done false
15972 5368 Acked Read 7 Done Sent Done false
15973 5369 Acked Read 3 Done Sent Done false
15974 5370 Acked Write 9 Done Sent Done false
15975 5371 Acked Read 10 Done Sent Done false
15976 5372 Acked Read 3 Done Sent Done false
15977 5373 Acked Flush 0 Done Sent Done false
15978 5374 Acked Write 1 Done Sent Done false
15979 5375 Acked Write 8 Done Sent Done false
15980 5376 Acked Read 8 Done Sent Done false
15981 5377 Acked Write 6 Done Sent Done false
15982 5378 Acked Flush 0 Done Sent Done false
15983 5379 Acked Read 8 Done Sent Done false
15984 5380 Acked Read 4 Done Sent Done false
15985 5381 Acked Write 1 Done Sent Done false
15986 5382 Acked Read 8 Done Sent Done false
15987 5383 Acked Write 6 Done Sent Done false
15988 5384 Acked Write 5 Done Sent Done false
15989 5385 Acked Write 8 Done Sent Done false
15990 5386 Acked Read 10 Done Sent Done false
15991 5387 Acked Write 1 Done Sent Done false
15992 5388 Acked Write 2 Done Sent Done false
15993 5389 Acked Read 8 Done Sent Done false
15994 5390 Acked Flush 0 Done Sent Done false
15995 5391 Acked Write 6 Done Sent Done false
15996 5392 Acked Read 6 Done Sent Done false
15997 5393 Acked Write 3 Done Sent Done false
15998 5394 Acked Write 9 Done Sent Done false
15999 5395 Acked Write 9 Done Sent Done false
16000 5396 Acked Write 10 Done Sent Done false
16001 5397 Acked Write 7 Done Sent Done false
16002 5398 Acked Read 4 Done Sent Done false
16003 5399 Acked Read 7 Done Sent Done false
16004 5400 Acked Read 6 Done Sent Done false
16005 5401 Acked Write 6 Done Sent Done false
16006 5402 Acked Read 5 Done Sent Done false
16007 5403 Acked Write 3 Done Sent Done false
16008 5404 Acked Write 6 Done Sent Done false
16009 5405 Acked Write 8 Done Sent Done false
16010 5406 Acked Write 7 Done Sent Done false
16011 5407 Acked Write 10 Done Sent Done false
16012 5408 Acked Write 6 Done Sent Done false
16013 5409 Acked Write 1 Done Sent Done false
16014 5410 Acked Read 3 Done Sent Done false
16015 5411 Acked Read 1 Done Sent Done false
16016 5412 Acked Write 9 Done Sent Done false
16017 5413 Acked Write 4 Done Sent Done false
16018 5414 Acked Write 7 Done Sent Done false
16019 5415 Acked Read 9 Done Sent Done false
16020 5416 Acked Write 2 Done Sent Done false
16021 5417 Acked Read 4 Done Sent Done false
16022 5418 Acked Write 5 Done Sent Done false
16023 5419 Acked Write 4 Done Sent Done false
16024 5420 Acked Read 2 Done Sent Done false
16025 5421 Acked Read 2 Done Sent Done false
16026 5422 Acked Write 4 Done Sent Done false
16027 5423 Acked Read 2 Done Sent Done false
16028 5424 Acked Write 5 Done Sent Done false
16029 5425 Acked Read 3 Done Sent Done false
16030 5426 Acked Read 10 Done Sent Done false
16031 5427 Acked Read 4 Done Sent Done false
16032 5428 Acked Write 6 Done Sent Done false
16033 5429 Acked Write 7 Done Sent Done false
16034 5430 Acked Read 7 Done Sent Done false
16035 5431 Acked Read 3 Done Sent Done false
16036 5432 Acked Write 5 Done Sent Done false
16037 5433 Acked Write 5 Done Sent Done false
16038 5434 Acked Read 10 Done Sent Done false
16039 5435 Acked Write 8 Done Sent Done false
16040 5436 Acked Read 2 Done Sent Done false
16041 5437 Acked Write 4 Done Sent Done false
16042 5438 Acked Write 7 Done Sent Done false
16043 5439 Acked Read 4 Done Sent Done false
16044 5440 Acked Read 5 Done Sent Done false
16045 5441 Acked Read 8 Done Sent Done false
16046 5442 Acked Read 1 Done Sent Done false
16047 5443 Acked Read 1 Done Sent Done false
16048 5444 Acked Read 7 Done Sent Done false
16049 5445 Acked Write 6 Done Sent Done false
16050 5446 Acked Write 7 Done Sent Done false
16051 5447 Acked Flush 0 Done Sent Done false
16052 5448 Acked Write 5 Done Sent Done false
16053 5449 Acked Write 9 Done Sent Done false
16054 5450 Acked Read 2 Done Sent Done false
16055 5451 Acked Read 4 Done Sent Done false
16056 5452 Acked Write 8 Done Sent Done false
16057 5453 Acked Read 5 Done Sent Done false
16058 5454 Acked Read 1 Done Sent Done false
16059 5455 Acked Read 4 Done Sent Done false
16060 5456 Acked Write 8 Done Sent Done false
16061 5457 Acked Write 6 Done Sent Done false
16062 5458 Acked Read 2 Done Sent Done false
16063 5459 Acked Read 7 Done Sent Done false
16064 5460 Acked Write 10 Done Sent Done false
16065 5461 Acked Read 10 Done Sent Done false
16066 5462 Acked Write 1 Done Sent Done false
16067 5463 Acked Read 6 Done Sent Done false
16068 5464 Acked Read 10 Done Sent Done false
16069 5465 Acked Write 4 Done Sent Done false
16070 5466 Acked Read 7 Done Sent Done false
16071 5467 Acked Write 10 Done Sent Done false
16072 5468 Acked Read 10 Done Sent Done false
16073 5469 Acked Read 6 Done Sent Done false
16074 5470 Acked Flush 0 Done Sent Done false
16075 5471 Acked Write 5 Done Sent Done false
16076 5472 Acked Write 1 Done Sent Done false
16077 5473 Acked Write 5 Done Sent Done false
16078 5474 Acked Read 4 Done Sent Done false
16079 5475 Acked Write 7 Done Sent Done false
16080 5476 Acked Read 2 Done Sent Done false
16081 5477 Acked Write 7 Done Sent Done false
16082 5478 Acked Read 8 Done Sent Done false
16083 5479 Acked Read 1 Done Sent Done false
16084 5480 Acked Read 2 Done Sent Done false
16085 5481 Acked Read 8 Done Sent Done false
16086 5482 Acked Read 2 Done Sent Done false
16087 5483 Acked Flush 0 Done Sent Done false
16088 5484 Acked Read 8 Done Sent Done false
16089 5485 Acked Flush 0 Done Sent Done false
16090 5486 Acked Read 9 Done Sent Done false
16091 5487 Acked Read 3 Done Sent Done false
16092 5488 Acked Write 1 Done Sent Done false
16093 5489 Acked Flush 0 Done Sent Done false
16094 5490 Acked Read 6 Done Sent Done false
16095 5491 Acked Write 8 Done Sent Done false
16096 5492 Acked Write 2 Done Sent Done false
16097 5493 Acked Read 7 Done Sent Done false
16098 5494 Acked Write 9 Done Sent Done false
16099 5495 Acked Read 10 Done Sent Done false
16100 5496 Acked Read 4 Done Sent Done false
16101 5497 Acked Flush 0 Done Sent Done false
16102 5498 Acked Write 1 Done Sent Done false
16103 5499 Acked Read 6 Done Sent Done false
16104 5500 Acked Read 3 Done Sent Done false
16105 5501 Acked Write 2 Done Sent Done false
16106 5502 Acked Read 2 Done Sent Done false
16107 5503 Acked Read 1 Done Sent Done false
16108 5504 Acked Write 1 Done Sent Done false
16109 5505 Acked Flush 0 Done Sent Done false
16110 5506 Acked Write 8 Done Sent Done false
16111 5507 Acked Flush 0 Done Sent Done false
16112 5508 Acked Write 7 Done Sent Done false
16113 5509 Acked Read 1 Done Sent Done false
16114 5510 Acked Write 6 Done Sent Done false
16115 5511 Acked Flush 0 Done Sent Done false
16116 5512 Acked Flush 0 Done Sent Done false
16117 5513 Acked Read 1 Done Sent Done false
16118 5514 Acked Read 10 Done Sent Done false
16119 5515 Acked Flush 0 Done Sent Done false
16120 5516 Acked Read 8 Done Sent Done false
16121 5517 Acked Read 2 Done Sent Done false
16122 5518 Acked Read 1 Done Sent Done false
16123 5519 Acked Write 6 Done Sent Done false
16124 5520 Acked Read 2 Done Sent Done false
16125 5521 Acked Flush 0 Done Sent Done false
16126 5522 Acked Read 7 Done Sent Done false
16127 5523 Acked Read 3 Done Sent Done false
16128 5524 Acked Read 3 Done Sent Done false
16129 5525 Acked Write 10 Done Sent Done false
16130 5526 Acked Read 3 Done Sent Done false
16131 5527 Acked Read 8 Done Sent Done false
16132 5528 Acked Flush 0 Done Sent Done false
16133 STATES DS:0 DS:1 DS:2 TOTAL
16134 Sent 0 301 0 301
16135 Done 301 0 301 602
16136 Skipped 0 0 0 0
16137 Error 0 0 0 0
16138 Last Flush: Some(JobId(5528)) Some(JobId(5227)) Some(JobId(5528))
16139 Downstairs last five completed: 5227 5226 5225 5224 5223
16140 Upstairs last five completed: 5528 5527 5526 5525 5524
16141 CLIENT: Up:0 ds:301 act:2
161422025-01-28T21:57:46.105ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161432025-01-28T21:57:46.632ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161442025-01-28T21:57:47.368ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161452025-01-28T21:57:47.368ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51819) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161462025-01-28T21:57:47.369ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161472025-01-28T21:57:47.370ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161482025-01-28T21:57:47.370ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161492025-01-28T21:57:47.370ZINFOcrucible: send last flush ID to this DS: Some(JobId(5227)) = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161502025-01-28T21:57:47.370ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161512025-01-28T21:57:47.370ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5227)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161522025-01-28T21:57:47.370ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5227)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
16153 ----------------------------------------------------------------
16154 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16155 Downstairs last five completed: 5528 5527 5526 5525 5524
16156 Upstairs last five completed: 5528 5527 5526 5525 5524
16157 CLIENT: Up:0 ds:0 act:3
16158 Replay: All jobs finished
16159 Main task has work to do, go find it
16160 got dsc Stop(1)
16161 stop 1
16162 [8820] Got stop action so:false kr:false
16163 Jan 28 21:57:49.606 INFO request completed, latency_us: 383, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 30fb7ca3-d609-464b-951c-85767a404f3d, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
16164 Jan 28 21:57:49.607 INFO request completed, latency_us: 130, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4d728fc2-2b47-488f-8792-47f6e8998b22, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
161652025-01-28T21:57:49.607ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161662025-01-28T21:57:49.607ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161672025-01-28T21:57:49.607ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
161682025-01-28T21:57:49.607ZINFOcrucible: client task is exiting = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
16169 [8820] Exited with: ExitStatus(unix_wait_status(9))
16170 [8820][1] reports Exit
16171 Jan 28 21:57:53.609 INFO request completed, latency_us: 202, response_code: 200, uri: /state/cid/1, method: GET, req_id: d05095c4-87af-4eec-b387-384461b7d7e5, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
16172 001/300 Read block 1258 len 4096
16173 002/300 Read block 288 len 12288
16174 003/300 Flush
16175 004/300 Read block 1883 len 4096
16176 005/300 Write block 372 len 20480 data: 4 4 4 4 6
16177 006/300 Read block 4657 len 28672
16178 007/300 Read block 2939 len 40960
16179 008/300 Read block 1765 len 20480
16180 009/300 Read block 4764 len 4096
16181 010/300 Read block 3512 len 24576
16182 011/300 Write block 434 len 8192 data: 2 3
16183 012/300 Read block 2719 len 24576
16184 013/300 Read block 2244 len 40960
16185 014/300 Read block 3492 len 36864
16186 015/300 Flush
16187 016/300 Read block 4735 len 12288
16188 017/300 Write block 4352 len 12288 data: 4 3 3
16189 018/300 Read block 2417 len 4096
16190 019/300 Read block 3920 len 20480
16191 020/300 Write block 4671 len 24576 data: 5 5 4 3 2 3
16192 021/300 Write block 4520 len 28672 data: 3 3 4 5 6 5 5
16193 022/300 Write block 329 len 24576 data: 5 4 3 3 3 3
16194 023/300 Read block 4408 len 16384
16195 024/300 Write block 4124 len 16384 data: 2 2 2 3
16196 025/300 Write block 4056 len 40960 data: 3 3 3 4 4 5 5 5 3 3
16197 026/300 Read block 1939 len 36864
16198 027/300 Read block 3750 len 8192
16199 028/300 Write block 1287 len 4096 data: 3
16200 029/300 Write block 2057 len 12288 data: 3 3 2
16201 030/300 Write block 1955 len 16384 data: 5 6 6 5
16202 031/300 Read block 4067 len 28672
16203 032/300 Flush
16204 033/300 Read block 3034 len 16384
16205 034/300 Read block 2367 len 28672
16206 035/300 Write block 1744 len 24576 data: 2 2 3 4 4 5
16207 036/300 Read block 4973 len 12288
16208 037/300 Read block 780 len 16384
16209 038/300 Read block 884 len 12288
16210 039/300 Write block 3361 len 12288 data: 3 3 2
16211 040/300 Flush
16212 041/300 Write block 984 len 16384 data: 4 3 3 3
16213 042/300 Write block 2399 len 24576 data: 4 4 2 3 3 3
16214 043/300 Write block 4041 len 40960 data: 4 4 4 3 3 3 3 4 4 3
16215 044/300 Read block 4646 len 32768
16216 045/300 Flush
16217 046/300 Read block 1800 len 8192
16218 047/300 Write block 12 len 4096 data: 4
16219 048/300 Read block 3325 len 4096
16220 049/300 Write block 1451 len 40960 data: 3 5 5 6 6 4 4 5 5 4
16221 050/300 Read block 1391 len 24576
16222 051/300 Read block 258 len 12288
16223 052/300 Read block 3992 len 28672
16224 053/300 Read block 4299 len 8192
16225 054/300 Read block 4791 len 20480
16226 055/300 Write block 94 len 24576 data: 3 3 5 4 3 3
16227 056/300 Flush
16228 057/300 Read block 4655 len 8192
16229 058/300 Write block 2947 len 36864 data: 5 5 4 4 3 3 3 3 3
16230 059/300 Write block 2556 len 40960 data: 3 3 3 3 3 3 3 3 4 4
16231 060/300 Read block 2775 len 32768
16232 061/300 Read block 2784 len 36864
16233 062/300 Write block 3048 len 12288 data: 7 6 7
16234 063/300 Write block 3358 len 36864 data: 4 3 3 4 4 3 2 2 4
16235 064/300 Flush
16236 065/300 Read block 4088 len 24576
16237 066/300 Read block 814 len 36864
16238 067/300 Read block 3725 len 8192
16239 068/300 Write block 4579 len 20480 data: 5 5 5 5 4
16240 069/300 Read block 949 len 24576
16241 070/300 Read block 2002 len 36864
16242 071/300 Read block 1667 len 36864
16243 072/300 Read block 971 len 16384
16244 073/300 Read block 3455 len 16384
16245 074/300 Write block 2506 len 28672 data: 3 5 5 4 5 4 4
16246 075/300 Write block 1636 len 40960 data: 6 4 3 3 3 3 3 3 3 2
16247 076/300 Flush
16248 077/300 Read block 1463 len 8192
16249 078/300 Flush
16250 079/300 Read block 3427 len 32768
16251 080/300 Read block 1803 len 4096
16252 081/300 Flush
16253 082/300 Read block 758 len 16384
16254 083/300 Write block 2938 len 40960 data: 5 5 6 6 6 6 5 6 6 6
16255 084/300 Write block 133 len 16384 data: 4 3 2 4
16256 085/300 Read block 2418 len 16384
16257 086/300 Write block 1664 len 40960 data: 6 5 3 3 3 4 4 4 3 3
16258 087/300 Write block 3148 len 28672 data: 4 3 4 3 3 4 4
16259 088/300 Read block 1849 len 20480
16260 089/300 Read block 1457 len 4096
16261 090/300 Write block 2720 len 24576 data: 2 2 2 3 3 3
16262 091/300 Flush
16263 092/300 Flush
16264 093/300 Read block 2621 len 4096
16265 094/300 Read block 2238 len 20480
16266 095/300 Read block 1802 len 20480
16267 096/300 Write block 582 len 40960 data: 2 2 3 3 4 4 4 3 3 3
16268 097/300 Write block 732 len 36864 data: 5 5 5 3 3 5 6 5 6
16269 098/300 Write block 1577 len 24576 data: 5 5 6 7 5 4
16270 099/300 Write block 743 len 24576 data: 4 3 4 3 2 2
16271 100/300 Write block 2385 len 20480 data: 5 5 5 6 6
16272 101/300 Write block 1207 len 12288 data: 6 6 6
16273 102/300 Write block 1825 len 40960 data: 6 6 5 4 4 4 3 3 3 2
16274 103/300 Write block 3452 len 36864 data: 4 2 2 3 3 3 3 3 3
16275 104/300 Read block 4965 len 20480
16276 105/300 Write block 4944 len 20480 data: 4 3 4 5 4
16277 106/300 Read block 4367 len 40960
16278 107/300 Read block 4474 len 24576
16279 108/300 Write block 2899 len 28672 data: 3 3 3 3 3 3 3
16280 109/300 Write block 3000 len 28672 data: 4 4 3 3 3 3 3
16281 110/300 Write block 3667 len 32768 data: 3 5 4 4 4 4 4 4
16282 111/300 Write block 3747 len 28672 data: 3 2 2 2 2 2 2
16283 112/300 Read block 1517 len 20480
16284 113/300 Read block 4698 len 40960
16285 114/300 Read block 1675 len 32768
16286 115/300 Write block 3805 len 16384 data: 2 2 2 2
16287 116/300 Write block 466 len 24576 data: 5 4 4 3 4 4
16288 117/300 Write block 2893 len 12288 data: 2 4 3
16289 118/300 Flush
16290 119/300 Read block 1624 len 40960
16291 120/300 Read block 1495 len 36864
16292 121/300 Read block 1267 len 8192
16293 122/300 Read block 2817 len 24576
16294 123/300 Write block 2979 len 32768 data: 4 3 3 3 3 3 4 4
16295 124/300 Read block 2024 len 4096
16296 125/300 Write block 826 len 20480 data: 3 3 3 3 3
16297 126/300 Write block 4707 len 20480 data: 4 4 4 4 4
16298 127/300 Read block 4378 len 32768
16299 128/300 Write block 4400 len 20480 data: 2 3 5 5 5
16300 129/300 Read block 2871 len 12288
16301 130/300 Read block 3421 len 40960
16302 131/300 Write block 299 len 12288 data: 4 6 7
16303 132/300 Write block 2003 len 24576 data: 3 3 3 3 3 4
16304 133/300 Write block 3283 len 28672 data: 3 3 3 3 3 3 3
16305 134/300 Read block 922 len 36864
16306 135/300 Write block 3103 len 8192 data: 4 4
16307 136/300 Read block 1748 len 8192
16308 137/300 Read block 30 len 36864
16309 138/300 Read block 4302 len 4096
16310 139/300 Flush
16311 140/300 Flush
16312 141/300 Write block 2940 len 4096 data: 7
16313 142/300 Read block 763 len 8192
16314 143/300 Read block 1215 len 16384
16315 144/300 Read block 4338 len 20480
16316 145/300 Read block 4725 len 20480
16317 146/300 Read block 1976 len 16384
16318 147/300 Write block 3094 len 36864 data: 3 4 4 3 3 3 3 4 4
16319 148/300 Flush
16320 149/300 Read block 736 len 12288
16321 150/300 Write block 1966 len 8192 data: 3 4
16322 151/300 Read block 1150 len 4096
16323 152/300 Write block 4462 len 40960 data: 4 4 4 4 4 3 2 3 3 3
16324 153/300 Write block 4597 len 36864 data: 2 3 4 4 4 4 5 5 5
16325 154/300 Read block 1052 len 8192
16326 155/300 Read block 4584 len 32768
16327 156/300 Flush
16328 157/300 Read block 3945 len 40960
16329 158/300 Read block 2711 len 8192
16330 159/300 Read block 65 len 40960
16331 160/300 Write block 3554 len 16384 data: 5 5 5 5
16332 161/300 Write block 1200 len 20480 data: 7 6 7 7 9
16333 162/300 Read block 4509 len 40960
16334 163/300 Write block 2383 len 20480 data: 6 6 6 6 6
16335 164/300 Write block 3667 len 32768 data: 4 6 5 5 5 5 5 5
16336 165/300 Write block 1123 len 24576 data: 4 3 4 3 4 5
16337 166/300 Flush
16338 167/300 Read block 2016 len 40960
16339 168/300 Read block 98 len 40960
16340 169/300 Write block 1177 len 20480 data: 5 7 9 10 9
16341 170/300 Read block 3230 len 28672
16342 171/300 Write block 4229 len 24576 data: 4 5 5 5 4 3
16343 172/300 Write block 3394 len 4096 data: 7
16344 173/300 Write block 1666 len 40960 data: 4 4 4 5 5 5 4 4 3 3
16345 174/300 Read block 4609 len 28672
16346 175/300 Write block 3608 len 8192 data: 5 5
16347 176/300 Write block 262 len 36864 data: 5 4 3 4 4 4 4 3 4
16348 177/300 Write block 901 len 24576 data: 5 6 6 6 6 6
16349 178/300 Read block 2032 len 20480
16350 179/300 Read block 793 len 40960
16351 180/300 Read block 2608 len 20480
16352 181/300 Write block 1039 len 20480 data: 6 5 4 5 6
16353 182/300 Read block 1086 len 4096
16354 183/300 Write block 4232 len 12288 data: 6 5 4
16355 184/300 Write block 112 len 40960 data: 2 2 3 2 2 2 2 3 3 3
16356 185/300 Write block 4559 len 24576 data: 4 4 4 5 5 5
16357 186/300 Write block 3660 len 28672 data: 3 3 3 4 4 4 3
16358 187/300 Read block 4849 len 16384
16359 188/300 Write block 2850 len 20480 data: 5 5 4 3 3
16360 189/300 Read block 2474 len 8192
16361 190/300 Read block 712 len 32768
16362 191/300 Read block 3766 len 16384
16363 192/300 Flush
16364 193/300 Write block 1078 len 4096 data: 4
16365 194/300 Read block 2234 len 24576
16366 195/300 Read block 1588 len 4096
16367 196/300 Write block 1290 len 8192 data: 4 4
16368 197/300 Read block 2330 len 32768
16369 198/300 Read block 1184 len 28672
16370 199/300 Read block 4919 len 12288
16371 200/300 Write block 3402 len 20480 data: 6 6 6 6 5
16372 201/300 Read block 4331 len 32768
16373 202/300 Write block 1359 len 28672 data: 4 3 5 5 5 6 4
16374 203/300 Read block 2173 len 40960
16375 204/300 Write block 3756 len 24576 data: 4 4 4 4 5 4
16376 205/300 Write block 4954 len 8192 data: 4 4
16377 206/300 Read block 3886 len 8192
16378 207/300 Flush
16379 208/300 Write block 233 len 4096 data: 8
16380 209/300 Flush
16381 210/300 Write block 2189 len 24576 data: 2 3 3 3 3 2
16382 211/300 Flush
16383 212/300 Read block 4609 len 32768
16384 213/300 Read block 1133 len 12288
16385 214/300 Write block 2691 len 12288 data: 5 5 5
16386 215/300 Write block 2248 len 8192 data: 4 3
16387 216/300 Write block 2023 len 12288 data: 2 3 3
16388 217/300 Write block 806 len 4096 data: 2
16389 218/300 Write block 4981 len 36864 data: 6 6 6 7 7 8 7 7 5
16390 219/300 Write block 1658 len 32768 data: 4 5 5 5 5 5 7 6
16391 220/300 Read block 1460 len 12288
16392 221/300 Write block 2219 len 40960 data: 3 3 3 2 2 2 2 2 2 3
16393 222/300 Write block 1898 len 12288 data: 3 3 3
16394 223/300 Read block 2895 len 16384
16395 224/300 Write block 1807 len 16384 data: 3 4 4 4
16396 225/300 Write block 2172 len 8192 data: 3 4
16397 226/300 Write block 774 len 24576 data: 4 4 3 2 3 4
16398 227/300 Write block 878 len 36864 data: 4 3 4 4 4 4 3 5 6
16399 228/300 Write block 2311 len 12288 data: 3 3 3
16400 229/300 Read block 4872 len 12288
16401 230/300 Write block 691 len 16384 data: 6 4 4 4
16402 231/300 Read block 756 len 36864
16403 232/300 Write block 1347 len 36864 data: 5 4 3 3 3 3 4 3 4
16404 233/300 Read block 2810 len 8192
16405 234/300 Flush
16406 235/300 Read block 2778 len 20480
16407 236/300 Read block 4307 len 32768
16408 237/300 Read block 4774 len 8192
16409 238/300 Read block 358 len 16384
16410 239/300 Read block 2647 len 20480
16411 240/300 Read block 524 len 40960
16412 241/300 Read block 3938 len 12288
16413 242/300 Flush
16414 243/300 Write block 1080 len 8192 data: 6 6
16415 244/300 Write block 2863 len 24576 data: 4 3 3 3 3 3
16416 245/300 Write block 3539 len 32768 data: 5 5 4 4 4 5 6 5
16417 246/300 Read block 4210 len 24576
16418 247/300 Write block 2420 len 4096 data: 3
16419 248/300 Read block 4459 len 40960
16420 249/300 Read block 3289 len 36864
16421 250/300 Write block 727 len 32768 data: 3 3 5 5 5 6 6 6
16422 251/300 Read block 880 len 16384
16423 252/300 Write block 2054 len 40960 data: 3 3 3 4 4 3 2 2 2 3
16424 253/300 Read block 478 len 36864
16425 254/300 Read block 4425 len 28672
16426 255/300 Write block 3520 len 20480 data: 4 4 4 4 5
16427 256/300 Write block 4134 len 12288 data: 6 6 5
16428 257/300 Write block 1167 len 24576 data: 3 3 3 3 3 4
16429 258/300 Read block 1694 len 32768
16430 259/300 Write block 478 len 12288 data: 5 4 3
16431 260/300 Write block 1267 len 40960 data: 3 4 4 4 4 4 4 4 3 2
16432 261/300 Read block 3147 len 28672
16433 262/300 Read block 3966 len 20480
16434 263/300 Write block 4897 len 4096 data: 3
16435 264/300 Write block 2099 len 28672 data: 5 4 4 4 5 4 4
16436 265/300 Write block 4738 len 16384 data: 5 4 4 5
16437 266/300 Write block 2789 len 24576 data: 6 6 4 4 5 6
16438 267/300 Read block 4414 len 20480
16439 268/300 Write block 2844 len 20480 data: 5 5 7 7 7
16440 269/300 Write block 1144 len 28672 data: 5 5 4 4 3 4 4
16441 270/300 Flush
16442 271/300 Read block 2595 len 16384
16443 272/300 Read block 2251 len 28672
16444 273/300 Write block 396 len 36864 data: 4 6 6 6 6 5 4 4 4
16445 274/300 Write block 241 len 36864 data: 6 6 6 5 4 4 4 4 4
16446 275/300 Read block 2845 len 8192
16447 276/300 Flush
16448 277/300 Write block 2347 len 40960 data: 6 4 4 5 4 4 4 4 4 4
16449 278/300 Write block 4968 len 20480 data: 4 5 5 3 2
16450 279/300 Read block 1346 len 4096
16451 280/300 Write block 1038 len 16384 data: 5 7 6 5
16452 281/300 Write block 3927 len 16384 data: 4 5 3 3
16453 282/300 Read block 4501 len 4096
16454 283/300 Flush
16455 284/300 Read block 3919 len 24576
16456 285/300 Read block 225 len 4096
16457 286/300 Read block 2872 len 12288
16458 287/300 Write block 2001 len 8192 data: 3 2
16459 288/300 Write block 463 len 8192 data: 4 4
16460 289/300 Write block 2710 len 8192 data: 5 4
16461 290/300 Write block 2977 len 24576 data: 3 4 5 4 4 4
16462 291/300 Read block 1163 len 8192
16463 292/300 Write block 1556 len 8192 data: 2 3
16464 293/300 Flush
16465 294/300 Write block 2501 len 8192 data: 5 5
16466 295/300 Read block 1 len 4096
16467 296/300 Write block 1853 len 28672 data: 3 3 3 3 2 2 3
16468 297/300 Read block 3197 len 12288
16469 298/300 Read block 3992 len 36864
16470 299/300 Write block 9 len 32768 data: 4 4 4 5 4 4 3 3
16471 300/300 Read block 4884 len 40960
16472 301/300 Write block 1854 len 32768 data: 4 4 4 3 3 4 3 4
16473 Main task has work to do, go find it
16474 got dsc Start(1)
16475 start 1
16476 Jan 28 21:57:53.840 INFO request completed, latency_us: 284, response_code: 204, uri: /start/cid/1, method: POST, req_id: 09f7f396-7961-4cdf-9f7b-2aae903c1ede, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
16477 Replay: started 1, returned:Ok(())
16478 ----------------------------------------------------------------
16479 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16480 States: Active Offline Active
16481 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16482 5529 Acked Read 1 Done Sent Done false
16483 5530 Acked Read 3 Done Sent Done false
16484 5531 Acked Flush 0 Done Sent Done false
16485 5532 Acked Read 1 Done Sent Done false
16486 5533 Acked Write 5 Done Sent Done false
16487 5534 Acked Read 7 Done Sent Done false
16488 5535 Acked Read 10 Done Sent Done false
16489 5536 Acked Read 5 Done Sent Done false
16490 5537 Acked Read 1 Done Sent Done false
16491 5538 Acked Read 6 Done Sent Done false
16492 5539 Acked Write 2 Done Sent Done false
16493 5540 Acked Read 6 Done Sent Done false
16494 5541 Acked Read 10 Done Sent Done false
16495 5542 Acked Read 9 Done Sent Done false
16496 5543 Acked Flush 0 Done Sent Done false
16497 5544 Acked Read 3 Done Sent Done false
16498 5545 Acked Write 3 Done Sent Done false
16499 5546 Acked Read 1 Done Sent Done false
16500 5547 Acked Read 5 Done Sent Done false
16501 5548 Acked Write 6 Done Sent Done false
16502 5549 Acked Write 7 Done Sent Done false
16503 5550 Acked Write 6 Done Sent Done false
16504 5551 Acked Read 4 Done Sent Done false
16505 5552 Acked Write 4 Done Sent Done false
16506 5553 Acked Write 10 Done Sent Done false
16507 5554 Acked Read 9 Done Sent Done false
16508 5555 Acked Read 2 Done Sent Done false
16509 5556 Acked Write 1 Done Sent Done false
16510 5557 Acked Write 3 Done Sent Done false
16511 5558 Acked Write 4 Done Sent Done false
16512 5559 Acked Read 7 Done Sent Done false
16513 5560 Acked Flush 0 Done Sent Done false
16514 5561 Acked Read 4 Done Sent Done false
16515 5562 Acked Read 7 Done Sent Done false
16516 5563 Acked Write 6 Done Sent Done false
16517 5564 Acked Read 3 Done Sent Done false
16518 5565 Acked Read 4 Done Sent Done false
16519 5566 Acked Read 3 Done Sent Done false
16520 5567 Acked Write 3 Done Sent Done false
16521 5568 Acked Flush 0 Done Sent Done false
16522 5569 Acked Write 4 Done Sent Done false
16523 5570 Acked Write 6 Done Sent Done false
16524 5571 Acked Write 10 Done Sent Done false
16525 5572 Acked Read 8 Done Sent Done false
16526 5573 Acked Flush 0 Done Sent Done false
16527 5574 Acked Read 2 Done Sent Done false
16528 5575 Acked Write 1 Done Sent Done false
16529 5576 Acked Read 1 Done Sent Done false
16530 5577 Acked Write 10 Done Sent Done false
16531 5578 Acked Read 6 Done Sent Done false
16532 5579 Acked Read 3 Done Sent Done false
16533 5580 Acked Read 7 Done Sent Done false
16534 5581 Acked Read 2 Done Sent Done false
16535 5582 Acked Read 5 Done Sent Done false
16536 5583 Acked Write 6 Done Sent Done false
16537 5584 Acked Flush 0 Done Sent Done false
16538 5585 Acked Read 2 Done Sent Done false
16539 5586 Acked Write 9 Done Sent Done false
16540 5587 Acked Write 10 Done Sent Done false
16541 5588 Acked Read 8 Done Sent Done false
16542 5589 Acked Read 9 Done Sent Done false
16543 5590 Acked Write 3 Done Sent Done false
16544 5591 Acked Write 9 Done Sent Done false
16545 5592 Acked Flush 0 Done Sent Done false
16546 5593 Acked Read 6 Done Sent Done false
16547 5594 Acked Read 9 Done Sent Done false
16548 5595 Acked Read 2 Done Sent Done false
16549 5596 Acked Write 5 Done Sent Done false
16550 5597 Acked Read 6 Done Sent Done false
16551 5598 Acked Read 9 Done Sent Done false
16552 5599 Acked Read 9 Done Sent Done false
16553 5600 Acked Read 4 Done Sent Done false
16554 5601 Acked Read 4 Done Sent Done false
16555 5602 Acked Write 7 Done Sent Done false
16556 5603 Acked Write 10 Done Sent Done false
16557 5604 Acked Flush 0 Done Sent Done false
16558 5605 Acked Read 2 Done Sent Done false
16559 5606 Acked Flush 0 Done Sent Done false
16560 5607 Acked Read 8 Done Sent Done false
16561 5608 Acked Read 1 Done Sent Done false
16562 5609 Acked Flush 0 Done Sent Done false
16563 5610 Acked Read 4 Done Sent Done false
16564 5611 Acked Write 10 Done Sent Done false
16565 5612 Acked Write 4 Done Sent Done false
16566 5613 Acked Read 4 Done Sent Done false
16567 5614 Acked Write 10 Done Sent Done false
16568 5615 Acked Write 7 Done Sent Done false
16569 5616 Acked Read 5 Done Sent Done false
16570 5617 Acked Read 1 Done Sent Done false
16571 5618 Acked Write 6 Done Sent Done false
16572 5619 Acked Flush 0 Done Sent Done false
16573 5620 Acked Flush 0 Done Sent Done false
16574 5621 Acked Read 1 Done Sent Done false
16575 5622 Acked Read 5 Done Sent Done false
16576 5623 Acked Read 5 Done Sent Done false
16577 5624 Acked Write 10 Done Sent Done false
16578 5625 Acked Write 9 Done Sent Done false
16579 5626 Acked Write 6 Done Sent Done false
16580 5627 Acked Write 6 Done Sent Done false
16581 5628 Acked Write 5 Done Sent Done false
16582 5629 Acked Write 3 Done Sent Done false
16583 5630 Acked Write 10 Done Sent Done false
16584 5631 Acked Write 9 Done Sent Done false
16585 5632 Acked Read 5 Done Sent Done false
16586 5633 Acked Write 5 Done Sent Done false
16587 5634 Acked Read 10 Done Sent Done false
16588 5635 Acked Read 6 Done Sent Done false
16589 5636 Acked Write 7 Done Sent Done false
16590 5637 Acked Write 7 Done Sent Done false
16591 5638 Acked Write 8 Done Sent Done false
16592 5639 Acked Write 7 Done Sent Done false
16593 5640 Acked Read 5 Done Sent Done false
16594 5641 Acked Read 10 Done Sent Done false
16595 5642 Acked Read 8 Done Sent Done false
16596 5643 Acked Write 4 Done Sent Done false
16597 5644 Acked Write 6 Done Sent Done false
16598 5645 Acked Write 3 Done Sent Done false
16599 5646 Acked Flush 0 Done Sent Done false
16600 5647 Acked Read 10 Done Sent Done false
16601 5648 Acked Read 9 Done Sent Done false
16602 5649 Acked Read 2 Done Sent Done false
16603 5650 Acked Read 6 Done Sent Done false
16604 5651 Acked Write 8 Done Sent Done false
16605 5652 Acked Read 1 Done Sent Done false
16606 5653 Acked Write 5 Done Sent Done false
16607 5654 Acked Write 5 Done Sent Done false
16608 5655 Acked Read 8 Done Sent Done false
16609 5656 Acked Write 5 Done Sent Done false
16610 5657 Acked Read 3 Done Sent Done false
16611 5658 Acked Read 10 Done Sent Done false
16612 5659 Acked Write 3 Done Sent Done false
16613 5660 Acked Write 6 Done Sent Done false
16614 5661 Acked Write 7 Done Sent Done false
16615 5662 Acked Read 9 Done Sent Done false
16616 5663 Acked Write 2 Done Sent Done false
16617 5664 Acked Read 2 Done Sent Done false
16618 5665 Acked Read 9 Done Sent Done false
16619 5666 Acked Read 1 Done Sent Done false
16620 5667 Acked Flush 0 Done Sent Done false
16621 5668 Acked Flush 0 Done Sent Done false
16622 5669 Acked Write 1 Done Sent Done false
16623 5670 Acked Read 2 Done Sent Done false
16624 5671 Acked Read 4 Done Sent Done false
16625 5672 Acked Read 5 Done Sent Done false
16626 5673 Acked Read 5 Done Sent Done false
16627 5674 Acked Read 4 Done Sent Done false
16628 5675 Acked Write 9 Done Sent Done false
16629 5676 Acked Flush 0 Done Sent Done false
16630 5677 Acked Read 3 Done Sent Done false
16631 5678 Acked Write 2 Done Sent Done false
16632 5679 Acked Read 1 Done Sent Done false
16633 5680 Acked Write 10 Done Sent Done false
16634 5681 Acked Write 9 Done Sent Done false
16635 5682 Acked Read 2 Done Sent Done false
16636 5683 Acked Read 8 Done Sent Done false
16637 5684 Acked Flush 0 Done Sent Done false
16638 5685 Acked Read 10 Done Sent Done false
16639 5686 Acked Read 2 Done Sent Done false
16640 5687 Acked Read 10 Done Sent Done false
16641 5688 Acked Write 4 Done Sent Done false
16642 5689 Acked Write 5 Done Sent Done false
16643 5690 Acked Read 10 Done Sent Done false
16644 5691 Acked Write 5 Done Sent Done false
16645 5692 Acked Write 8 Done Sent Done false
16646 5693 Acked Write 6 Done Sent Done false
16647 5694 Acked Flush 0 Done Sent Done false
16648 5695 Acked Read 10 Done Sent Done false
16649 5696 Acked Read 10 Done Sent Done false
16650 5697 Acked Write 5 Done Sent Done false
16651 5698 Acked Read 7 Done Sent Done false
16652 5699 Acked Write 6 Done Sent Done false
16653 5700 Acked Write 1 Done Sent Done false
16654 5701 Acked Write 10 Done Sent Done false
16655 5702 Acked Read 7 Done Sent Done false
16656 5703 Acked Write 2 Done Sent Done false
16657 5704 Acked Write 9 Done Sent Done false
16658 5705 Acked Write 6 Done Sent Done false
16659 5706 Acked Read 5 Done Sent Done false
16660 5707 Acked Read 10 Done Sent Done false
16661 5708 Acked Read 5 Done Sent Done false
16662 5709 Acked Write 5 Done Sent Done false
16663 5710 Acked Read 1 Done Sent Done false
16664 5711 Acked Write 3 Done Sent Done false
16665 5712 Acked Write 10 Done Sent Done false
16666 5713 Acked Write 6 Done Sent Done false
16667 5714 Acked Write 7 Done Sent Done false
16668 5715 Acked Read 4 Done Sent Done false
16669 5716 Acked Write 5 Done Sent Done false
16670 5717 Acked Read 2 Done Sent Done false
16671 5718 Acked Read 8 Done Sent Done false
16672 5719 Acked Read 4 Done Sent Done false
16673 5720 Acked Flush 0 Done Sent Done false
16674 5721 Acked Write 1 Done Sent Done false
16675 5722 Acked Read 6 Done Sent Done false
16676 5723 Acked Read 1 Done Sent Done false
16677 5724 Acked Write 2 Done Sent Done false
16678 5725 Acked Read 8 Done Sent Done false
16679 5726 Acked Read 7 Done Sent Done false
16680 5727 Acked Read 3 Done Sent Done false
16681 5728 Acked Write 5 Done Sent Done false
16682 5729 Acked Read 8 Done Sent Done false
16683 5730 Acked Write 7 Done Sent Done false
16684 5731 Acked Read 10 Done Sent Done false
16685 5732 Acked Write 6 Done Sent Done false
16686 5733 Acked Write 2 Done Sent Done false
16687 5734 Acked Read 2 Done Sent Done false
16688 5735 Acked Flush 0 Done Sent Done false
16689 5736 Acked Write 1 Done Sent Done false
16690 5737 Acked Flush 0 Done Sent Done false
16691 5738 Acked Write 6 Done Sent Done false
16692 5739 Acked Flush 0 Done Sent Done false
16693 5740 Acked Read 8 Done Sent Done false
16694 5741 Acked Read 3 Done Sent Done false
16695 5742 Acked Write 3 Done Sent Done false
16696 5743 Acked Write 2 Done Sent Done false
16697 5744 Acked Write 3 Done Sent Done false
16698 5745 Acked Write 1 Done Sent Done false
16699 5746 Acked Write 9 Done Sent Done false
16700 5747 Acked Write 8 Done Sent Done false
16701 5748 Acked Read 3 Done Sent Done false
16702 5749 Acked Write 10 Done Sent Done false
16703 5750 Acked Write 3 Done Sent Done false
16704 5751 Acked Read 4 Done Sent Done false
16705 5752 Acked Write 4 Done Sent Done false
16706 5753 Acked Write 2 Done Sent Done false
16707 5754 Acked Write 6 Done Sent Done false
16708 5755 Acked Write 9 Done Sent Done false
16709 5756 Acked Write 3 Done Sent Done false
16710 5757 Acked Read 3 Done Sent Done false
16711 5758 Acked Write 4 Done Sent Done false
16712 5759 Acked Read 9 Done Sent Done false
16713 5760 Acked Write 9 Done Sent Done false
16714 5761 Acked Read 2 Done Sent Done false
16715 5762 Acked Flush 0 Done Sent Done false
16716 5763 Acked Read 5 Done Sent Done false
16717 5764 Acked Read 8 Done Sent Done false
16718 5765 Acked Read 2 Done Sent Done false
16719 5766 Acked Read 4 Done Sent Done false
16720 5767 Acked Read 5 Done Sent Done false
16721 5768 Acked Read 10 Done Sent Done false
16722 5769 Acked Read 3 Done Sent Done false
16723 5770 Acked Flush 0 Done Sent Done false
16724 5771 Acked Write 2 Done Sent Done false
16725 5772 Acked Write 6 Done Sent Done false
16726 5773 Acked Write 8 Done Sent Done false
16727 5774 Acked Read 6 Done Sent Done false
16728 5775 Acked Write 1 Done Sent Done false
16729 5776 Acked Read 10 Done Sent Done false
16730 5777 Acked Read 9 Done Sent Done false
16731 5778 Acked Write 8 Done Sent Done false
16732 5779 Acked Read 4 Done Sent Done false
16733 5780 Acked Write 10 Done Sent Done false
16734 5781 Acked Read 9 Done Sent Done false
16735 5782 Acked Read 7 Done Sent Done false
16736 5783 Acked Write 5 Done Sent Done false
16737 5784 Acked Write 3 Done Sent Done false
16738 5785 Acked Write 6 Done Sent Done false
16739 5786 Acked Read 8 Done Sent Done false
16740 5787 Acked Write 3 Done Sent Done false
16741 5788 Acked Write 10 Done Sent Done false
16742 5789 Acked Read 7 Done Sent Done false
16743 5790 Acked Read 5 Done Sent Done false
16744 5791 Acked Write 1 Done Sent Done false
16745 5792 Acked Write 7 Done Sent Done false
16746 5793 Acked Write 4 Done Sent Done false
16747 5794 Acked Write 6 Done Sent Done false
16748 5795 Acked Read 5 Done Sent Done false
16749 5796 Acked Write 5 Done Sent Done false
16750 5797 Acked Write 7 Done Sent Done false
16751 5798 Acked Flush 0 Done Sent Done false
16752 5799 Acked Read 4 Done Sent Done false
16753 5800 Acked Read 7 Done Sent Done false
16754 5801 Acked Write 9 Done Sent Done false
16755 5802 Acked Write 9 Done Sent Done false
16756 5803 Acked Read 2 Done Sent Done false
16757 5804 Acked Flush 0 Done Sent Done false
16758 5805 Acked Write 10 Done Sent Done false
16759 5806 Acked Write 5 Done Sent Done false
16760 5807 Acked Read 1 Done Sent Done false
16761 5808 Acked Write 4 Done Sent Done false
16762 5809 Acked Write 4 Done Sent Done false
16763 5810 Acked Read 1 Done Sent Done false
16764 5811 Acked Flush 0 Done Sent Done false
16765 5812 Acked Read 6 Done Sent Done false
16766 5813 Acked Read 1 Done Sent Done false
16767 5814 Acked Read 3 Done Sent Done false
16768 5815 Acked Write 2 Done Sent Done false
16769 5816 Acked Write 2 Done Sent Done false
16770 5817 Acked Write 2 Done Sent Done false
16771 5818 Acked Write 6 Done Sent Done false
16772 5819 Acked Read 2 Done Sent Done false
16773 5820 Acked Write 2 Done Sent Done false
16774 5821 Acked Flush 0 Done Sent Done false
16775 5822 Acked Write 2 Done Sent Done false
16776 5823 Acked Read 1 Done Sent Done false
16777 5824 Acked Write 7 Done Sent Done false
16778 5825 Acked Read 3 Done Sent Done false
16779 5826 Acked Read 9 Done Sent Done false
16780 5827 Acked Write 8 Done Sent Done false
16781 5828 Acked Read 10 Done Sent Done false
16782 5829 Acked Write 8 Done Sent Done false
16783 STATES DS:0 DS:1 DS:2 TOTAL
16784 Sent 0 301 0 301
16785 Done 301 0 301 602
16786 Skipped 0 0 0 0
16787 Error 0 0 0 0
16788 Last Flush: Some(JobId(5821)) Some(JobId(5528)) Some(JobId(5821))
16789 Downstairs last five completed: 5528 5527 5526 5525 5524
16790 Upstairs last five completed: 5829 5828 5827 5826 5825
16791 CLIENT: Up:0 ds:301 act:2
16792 [8820] got start action
16793 [8820] I am going to restart
16794 Starting downstairs at port 8820
16795 Make output file at "/tmp/dsc/downstairs-8820.txt"
16796 [8820][1] reports Starting
16797 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1218)
16798 [8820][1] reports Running
167992025-01-28T21:57:55.647ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
168002025-01-28T21:57:56.649ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
168012025-01-28T21:57:57.651ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
16802 ----------------------------------------------------------------
16803 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16804 States: Active Offline Active
16805 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16806 5529 Acked Read 1 Done Sent Done false
16807 5530 Acked Read 3 Done Sent Done false
16808 5531 Acked Flush 0 Done Sent Done false
16809 5532 Acked Read 1 Done Sent Done false
16810 5533 Acked Write 5 Done Sent Done false
16811 5534 Acked Read 7 Done Sent Done false
16812 5535 Acked Read 10 Done Sent Done false
16813 5536 Acked Read 5 Done Sent Done false
16814 5537 Acked Read 1 Done Sent Done false
16815 5538 Acked Read 6 Done Sent Done false
16816 5539 Acked Write 2 Done Sent Done false
16817 5540 Acked Read 6 Done Sent Done false
16818 5541 Acked Read 10 Done Sent Done false
16819 5542 Acked Read 9 Done Sent Done false
16820 5543 Acked Flush 0 Done Sent Done false
16821 5544 Acked Read 3 Done Sent Done false
16822 5545 Acked Write 3 Done Sent Done false
16823 5546 Acked Read 1 Done Sent Done false
16824 5547 Acked Read 5 Done Sent Done false
16825 5548 Acked Write 6 Done Sent Done false
16826 5549 Acked Write 7 Done Sent Done false
16827 5550 Acked Write 6 Done Sent Done false
16828 5551 Acked Read 4 Done Sent Done false
16829 5552 Acked Write 4 Done Sent Done false
16830 5553 Acked Write 10 Done Sent Done false
16831 5554 Acked Read 9 Done Sent Done false
16832 5555 Acked Read 2 Done Sent Done false
16833 5556 Acked Write 1 Done Sent Done false
16834 5557 Acked Write 3 Done Sent Done false
16835 5558 Acked Write 4 Done Sent Done false
16836 5559 Acked Read 7 Done Sent Done false
16837 5560 Acked Flush 0 Done Sent Done false
16838 5561 Acked Read 4 Done Sent Done false
16839 5562 Acked Read 7 Done Sent Done false
16840 5563 Acked Write 6 Done Sent Done false
16841 5564 Acked Read 3 Done Sent Done false
16842 5565 Acked Read 4 Done Sent Done false
16843 5566 Acked Read 3 Done Sent Done false
16844 5567 Acked Write 3 Done Sent Done false
16845 5568 Acked Flush 0 Done Sent Done false
16846 5569 Acked Write 4 Done Sent Done false
16847 5570 Acked Write 6 Done Sent Done false
16848 5571 Acked Write 10 Done Sent Done false
16849 5572 Acked Read 8 Done Sent Done false
16850 5573 Acked Flush 0 Done Sent Done false
16851 5574 Acked Read 2 Done Sent Done false
16852 5575 Acked Write 1 Done Sent Done false
16853 5576 Acked Read 1 Done Sent Done false
16854 5577 Acked Write 10 Done Sent Done false
16855 5578 Acked Read 6 Done Sent Done false
16856 5579 Acked Read 3 Done Sent Done false
16857 5580 Acked Read 7 Done Sent Done false
16858 5581 Acked Read 2 Done Sent Done false
16859 5582 Acked Read 5 Done Sent Done false
16860 5583 Acked Write 6 Done Sent Done false
16861 5584 Acked Flush 0 Done Sent Done false
16862 5585 Acked Read 2 Done Sent Done false
16863 5586 Acked Write 9 Done Sent Done false
16864 5587 Acked Write 10 Done Sent Done false
16865 5588 Acked Read 8 Done Sent Done false
16866 5589 Acked Read 9 Done Sent Done false
16867 5590 Acked Write 3 Done Sent Done false
16868 5591 Acked Write 9 Done Sent Done false
16869 5592 Acked Flush 0 Done Sent Done false
16870 5593 Acked Read 6 Done Sent Done false
16871 5594 Acked Read 9 Done Sent Done false
16872 5595 Acked Read 2 Done Sent Done false
16873 5596 Acked Write 5 Done Sent Done false
16874 5597 Acked Read 6 Done Sent Done false
16875 5598 Acked Read 9 Done Sent Done false
16876 5599 Acked Read 9 Done Sent Done false
16877 5600 Acked Read 4 Done Sent Done false
16878 5601 Acked Read 4 Done Sent Done false
16879 5602 Acked Write 7 Done Sent Done false
16880 5603 Acked Write 10 Done Sent Done false
16881 5604 Acked Flush 0 Done Sent Done false
16882 5605 Acked Read 2 Done Sent Done false
16883 5606 Acked Flush 0 Done Sent Done false
16884 5607 Acked Read 8 Done Sent Done false
16885 5608 Acked Read 1 Done Sent Done false
16886 5609 Acked Flush 0 Done Sent Done false
16887 5610 Acked Read 4 Done Sent Done false
16888 5611 Acked Write 10 Done Sent Done false
16889 5612 Acked Write 4 Done Sent Done false
16890 5613 Acked Read 4 Done Sent Done false
16891 5614 Acked Write 10 Done Sent Done false
16892 5615 Acked Write 7 Done Sent Done false
16893 5616 Acked Read 5 Done Sent Done false
16894 5617 Acked Read 1 Done Sent Done false
16895 5618 Acked Write 6 Done Sent Done false
16896 5619 Acked Flush 0 Done Sent Done false
16897 5620 Acked Flush 0 Done Sent Done false
16898 5621 Acked Read 1 Done Sent Done false
16899 5622 Acked Read 5 Done Sent Done false
16900 5623 Acked Read 5 Done Sent Done false
16901 5624 Acked Write 10 Done Sent Done false
16902 5625 Acked Write 9 Done Sent Done false
16903 5626 Acked Write 6 Done Sent Done false
16904 5627 Acked Write 6 Done Sent Done false
16905 5628 Acked Write 5 Done Sent Done false
16906 5629 Acked Write 3 Done Sent Done false
16907 5630 Acked Write 10 Done Sent Done false
16908 5631 Acked Write 9 Done Sent Done false
16909 5632 Acked Read 5 Done Sent Done false
16910 5633 Acked Write 5 Done Sent Done false
16911 5634 Acked Read 10 Done Sent Done false
16912 5635 Acked Read 6 Done Sent Done false
16913 5636 Acked Write 7 Done Sent Done false
16914 5637 Acked Write 7 Done Sent Done false
16915 5638 Acked Write 8 Done Sent Done false
16916 5639 Acked Write 7 Done Sent Done false
16917 5640 Acked Read 5 Done Sent Done false
16918 5641 Acked Read 10 Done Sent Done false
16919 5642 Acked Read 8 Done Sent Done false
16920 5643 Acked Write 4 Done Sent Done false
16921 5644 Acked Write 6 Done Sent Done false
16922 5645 Acked Write 3 Done Sent Done false
16923 5646 Acked Flush 0 Done Sent Done false
16924 5647 Acked Read 10 Done Sent Done false
16925 5648 Acked Read 9 Done Sent Done false
16926 5649 Acked Read 2 Done Sent Done false
16927 5650 Acked Read 6 Done Sent Done false
16928 5651 Acked Write 8 Done Sent Done false
16929 5652 Acked Read 1 Done Sent Done false
16930 5653 Acked Write 5 Done Sent Done false
16931 5654 Acked Write 5 Done Sent Done false
16932 5655 Acked Read 8 Done Sent Done false
16933 5656 Acked Write 5 Done Sent Done false
16934 5657 Acked Read 3 Done Sent Done false
16935 5658 Acked Read 10 Done Sent Done false
16936 5659 Acked Write 3 Done Sent Done false
16937 5660 Acked Write 6 Done Sent Done false
16938 5661 Acked Write 7 Done Sent Done false
16939 5662 Acked Read 9 Done Sent Done false
16940 5663 Acked Write 2 Done Sent Done false
16941 5664 Acked Read 2 Done Sent Done false
16942 5665 Acked Read 9 Done Sent Done false
16943 5666 Acked Read 1 Done Sent Done false
16944 5667 Acked Flush 0 Done Sent Done false
16945 5668 Acked Flush 0 Done Sent Done false
16946 5669 Acked Write 1 Done Sent Done false
16947 5670 Acked Read 2 Done Sent Done false
16948 5671 Acked Read 4 Done Sent Done false
16949 5672 Acked Read 5 Done Sent Done false
16950 5673 Acked Read 5 Done Sent Done false
16951 5674 Acked Read 4 Done Sent Done false
16952 5675 Acked Write 9 Done Sent Done false
16953 5676 Acked Flush 0 Done Sent Done false
16954 5677 Acked Read 3 Done Sent Done false
16955 5678 Acked Write 2 Done Sent Done false
16956 5679 Acked Read 1 Done Sent Done false
16957 5680 Acked Write 10 Done Sent Done false
16958 5681 Acked Write 9 Done Sent Done false
16959 5682 Acked Read 2 Done Sent Done false
16960 5683 Acked Read 8 Done Sent Done false
16961 5684 Acked Flush 0 Done Sent Done false
16962 5685 Acked Read 10 Done Sent Done false
16963 5686 Acked Read 2 Done Sent Done false
16964 5687 Acked Read 10 Done Sent Done false
16965 5688 Acked Write 4 Done Sent Done false
16966 5689 Acked Write 5 Done Sent Done false
16967 5690 Acked Read 10 Done Sent Done false
16968 5691 Acked Write 5 Done Sent Done false
16969 5692 Acked Write 8 Done Sent Done false
16970 5693 Acked Write 6 Done Sent Done false
16971 5694 Acked Flush 0 Done Sent Done false
16972 5695 Acked Read 10 Done Sent Done false
16973 5696 Acked Read 10 Done Sent Done false
16974 5697 Acked Write 5 Done Sent Done false
16975 5698 Acked Read 7 Done Sent Done false
16976 5699 Acked Write 6 Done Sent Done false
16977 5700 Acked Write 1 Done Sent Done false
16978 5701 Acked Write 10 Done Sent Done false
16979 5702 Acked Read 7 Done Sent Done false
16980 5703 Acked Write 2 Done Sent Done false
16981 5704 Acked Write 9 Done Sent Done false
16982 5705 Acked Write 6 Done Sent Done false
16983 5706 Acked Read 5 Done Sent Done false
16984 5707 Acked Read 10 Done Sent Done false
16985 5708 Acked Read 5 Done Sent Done false
16986 5709 Acked Write 5 Done Sent Done false
16987 5710 Acked Read 1 Done Sent Done false
16988 5711 Acked Write 3 Done Sent Done false
16989 5712 Acked Write 10 Done Sent Done false
16990 5713 Acked Write 6 Done Sent Done false
16991 5714 Acked Write 7 Done Sent Done false
16992 5715 Acked Read 4 Done Sent Done false
16993 5716 Acked Write 5 Done Sent Done false
16994 5717 Acked Read 2 Done Sent Done false
16995 5718 Acked Read 8 Done Sent Done false
16996 5719 Acked Read 4 Done Sent Done false
16997 5720 Acked Flush 0 Done Sent Done false
16998 5721 Acked Write 1 Done Sent Done false
16999 5722 Acked Read 6 Done Sent Done false
17000 5723 Acked Read 1 Done Sent Done false
17001 5724 Acked Write 2 Done Sent Done false
17002 5725 Acked Read 8 Done Sent Done false
17003 5726 Acked Read 7 Done Sent Done false
17004 5727 Acked Read 3 Done Sent Done false
17005 5728 Acked Write 5 Done Sent Done false
17006 5729 Acked Read 8 Done Sent Done false
17007 5730 Acked Write 7 Done Sent Done false
17008 5731 Acked Read 10 Done Sent Done false
17009 5732 Acked Write 6 Done Sent Done false
17010 5733 Acked Write 2 Done Sent Done false
17011 5734 Acked Read 2 Done Sent Done false
17012 5735 Acked Flush 0 Done Sent Done false
17013 5736 Acked Write 1 Done Sent Done false
17014 5737 Acked Flush 0 Done Sent Done false
17015 5738 Acked Write 6 Done Sent Done false
17016 5739 Acked Flush 0 Done Sent Done false
17017 5740 Acked Read 8 Done Sent Done false
17018 5741 Acked Read 3 Done Sent Done false
17019 5742 Acked Write 3 Done Sent Done false
17020 5743 Acked Write 2 Done Sent Done false
17021 5744 Acked Write 3 Done Sent Done false
17022 5745 Acked Write 1 Done Sent Done false
17023 5746 Acked Write 9 Done Sent Done false
17024 5747 Acked Write 8 Done Sent Done false
17025 5748 Acked Read 3 Done Sent Done false
17026 5749 Acked Write 10 Done Sent Done false
17027 5750 Acked Write 3 Done Sent Done false
17028 5751 Acked Read 4 Done Sent Done false
17029 5752 Acked Write 4 Done Sent Done false
17030 5753 Acked Write 2 Done Sent Done false
17031 5754 Acked Write 6 Done Sent Done false
17032 5755 Acked Write 9 Done Sent Done false
17033 5756 Acked Write 3 Done Sent Done false
17034 5757 Acked Read 3 Done Sent Done false
17035 5758 Acked Write 4 Done Sent Done false
17036 5759 Acked Read 9 Done Sent Done false
17037 5760 Acked Write 9 Done Sent Done false
17038 5761 Acked Read 2 Done Sent Done false
17039 5762 Acked Flush 0 Done Sent Done false
17040 5763 Acked Read 5 Done Sent Done false
17041 5764 Acked Read 8 Done Sent Done false
17042 5765 Acked Read 2 Done Sent Done false
17043 5766 Acked Read 4 Done Sent Done false
17044 5767 Acked Read 5 Done Sent Done false
17045 5768 Acked Read 10 Done Sent Done false
17046 5769 Acked Read 3 Done Sent Done false
17047 5770 Acked Flush 0 Done Sent Done false
17048 5771 Acked Write 2 Done Sent Done false
17049 5772 Acked Write 6 Done Sent Done false
17050 5773 Acked Write 8 Done Sent Done false
17051 5774 Acked Read 6 Done Sent Done false
17052 5775 Acked Write 1 Done Sent Done false
17053 5776 Acked Read 10 Done Sent Done false
17054 5777 Acked Read 9 Done Sent Done false
17055 5778 Acked Write 8 Done Sent Done false
17056 5779 Acked Read 4 Done Sent Done false
17057 5780 Acked Write 10 Done Sent Done false
17058 5781 Acked Read 9 Done Sent Done false
17059 5782 Acked Read 7 Done Sent Done false
17060 5783 Acked Write 5 Done Sent Done false
17061 5784 Acked Write 3 Done Sent Done false
17062 5785 Acked Write 6 Done Sent Done false
17063 5786 Acked Read 8 Done Sent Done false
17064 5787 Acked Write 3 Done Sent Done false
17065 5788 Acked Write 10 Done Sent Done false
17066 5789 Acked Read 7 Done Sent Done false
17067 5790 Acked Read 5 Done Sent Done false
17068 5791 Acked Write 1 Done Sent Done false
17069 5792 Acked Write 7 Done Sent Done false
17070 5793 Acked Write 4 Done Sent Done false
17071 5794 Acked Write 6 Done Sent Done false
17072 5795 Acked Read 5 Done Sent Done false
17073 5796 Acked Write 5 Done Sent Done false
17074 5797 Acked Write 7 Done Sent Done false
17075 5798 Acked Flush 0 Done Sent Done false
17076 5799 Acked Read 4 Done Sent Done false
17077 5800 Acked Read 7 Done Sent Done false
17078 5801 Acked Write 9 Done Sent Done false
17079 5802 Acked Write 9 Done Sent Done false
17080 5803 Acked Read 2 Done Sent Done false
17081 5804 Acked Flush 0 Done Sent Done false
17082 5805 Acked Write 10 Done Sent Done false
17083 5806 Acked Write 5 Done Sent Done false
17084 5807 Acked Read 1 Done Sent Done false
17085 5808 Acked Write 4 Done Sent Done false
17086 5809 Acked Write 4 Done Sent Done false
17087 5810 Acked Read 1 Done Sent Done false
17088 5811 Acked Flush 0 Done Sent Done false
17089 5812 Acked Read 6 Done Sent Done false
17090 5813 Acked Read 1 Done Sent Done false
17091 5814 Acked Read 3 Done Sent Done false
17092 5815 Acked Write 2 Done Sent Done false
17093 5816 Acked Write 2 Done Sent Done false
17094 5817 Acked Write 2 Done Sent Done false
17095 5818 Acked Write 6 Done Sent Done false
17096 5819 Acked Read 2 Done Sent Done false
17097 5820 Acked Write 2 Done Sent Done false
17098 5821 Acked Flush 0 Done Sent Done false
17099 5822 Acked Write 2 Done Sent Done false
17100 5823 Acked Read 1 Done Sent Done false
17101 5824 Acked Write 7 Done Sent Done false
17102 5825 Acked Read 3 Done Sent Done false
17103 5826 Acked Read 9 Done Sent Done false
17104 5827 Acked Write 8 Done Sent Done false
17105 5828 Acked Read 10 Done Sent Done false
17106 5829 Acked Write 8 Done Sent Done false
17107 STATES DS:0 DS:1 DS:2 TOTAL
17108 Sent 0 301 0 301
17109 Done 301 0 301 602
17110 Skipped 0 0 0 0
17111 Error 0 0 0 0
17112 Last Flush: Some(JobId(5821)) Some(JobId(5528)) Some(JobId(5821))
17113 Downstairs last five completed: 5528 5527 5526 5525 5524
17114 Upstairs last five completed: 5829 5828 5827 5826 5825
17115 CLIENT: Up:0 ds:301 act:2
171162025-01-28T21:57:58.653ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
171172025-01-28T21:57:59.609ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
171182025-01-28T21:57:59.609ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58265) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
171192025-01-28T21:57:59.610ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
171202025-01-28T21:57:59.611ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
171212025-01-28T21:57:59.611ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
171222025-01-28T21:57:59.611ZINFOcrucible: send last flush ID to this DS: Some(JobId(5528)) = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
171232025-01-28T21:57:59.611ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
171242025-01-28T21:57:59.611ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5528)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
171252025-01-28T21:57:59.611ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5528)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
17126 ----------------------------------------------------------------
17127 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17128 Downstairs last five completed: 5830 5829 5828 5827 5826
17129 Upstairs last five completed: 5830 5829 5828 5827 5826
17130 CLIENT: Up:0 ds:0 act:3
17131 Replay: All jobs finished
17132 Main task has work to do, go find it
17133 got dsc Stop(1)
17134 stop 1
17135 [8820] Got stop action so:false kr:false
17136 Jan 28 21:58:01.845 INFO request completed, latency_us: 338, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 5e5842b2-e92b-40ec-83f4-719721d3bb43, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
171372025-01-28T21:58:01.846ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
17138 Jan 28 21:58:01.846 INFO request completed, latency_us: 109, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:58:01.846308372Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
17139 response_code:{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:58:01.846340819Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
17140 200, uri: /state/cid/1, method: GET, req_id: 86fb33d2-715c-49f0-95f0-6800dfd0b7b9, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
171412025-01-28T21:58:01.846ZINFOcrucible: client task is exiting = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
17142 [8820] Exited with: ExitStatus(unix_wait_status(9))
17143 [8820][1] reports Exit
17144 Jan 28 21:58:05.848 INFO request completed, latency_us: 260, response_code: 200, uri: /state/cid/1, method: GET, req_id: b077b80d-494a-47d6-924c-ac8034e2859a, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
17145 001/300 Read block 3882 len 8192
17146 002/300 Write block 2581 len 4096 data: 3
17147 003/300 Flush
17148 004/300 Write block 2899 len 36864 data: 4 4 4 4 4 4 4 2 2
17149 005/300 Write block 1278 len 4096 data: 2
17150 006/300 Flush
17151 007/300 Write block 3100 len 28672 data: 4 5 5 5 5 4 5
17152 008/300 Read block 3521 len 32768
17153 009/300 Write block 245 len 32768 data: 5 5 5 5 5 5 4 4
17154 010/300 Read block 3309 len 28672
17155 011/300 Flush
17156 012/300 Write block 275 len 16384 data: 2 2 2 3
17157 013/300 Write block 3725 len 16384 data: 6 6 8 9
17158 014/300 Write block 2189 len 24576 data: 3 4 4 4 4 3
17159 015/300 Write block 2465 len 4096 data: 3
17160 016/300 Read block 946 len 12288
17161 017/300 Read block 4932 len 28672
17162 018/300 Read block 4858 len 16384
17163 019/300 Write block 4693 len 40960 data: 4 6 6 5 6 4 3 2 2 3
17164 020/300 Write block 1148 len 16384 data: 4 5 5 5
17165 021/300 Flush
17166 022/300 Write block 420 len 8192 data: 3 4
17167 023/300 Read block 3664 len 4096
17168 024/300 Write block 52 len 40960 data: 3 2 2 2 2 3 4 4 4 5
17169 025/300 Flush
17170 026/300 Write block 3888 len 36864 data: 5 4 3 4 4 4 4 5 5
17171 027/300 Write block 3234 len 4096 data: 2
17172 028/300 Write block 160 len 12288 data: 3 3 2
17173 029/300 Write block 1040 len 12288 data: 7 6 6
17174 030/300 Read block 584 len 8192
17175 031/300 Read block 985 len 16384
17176 032/300 Write block 333 len 8192 data: 4 4
17177 033/300 Flush
17178 034/300 Write block 2797 len 32768 data: 6 7 5 4 4 4 4 3
17179 035/300 Write block 4818 len 28672 data: 3 3 4 4 4 4 3
17180 036/300 Write block 705 len 40960 data: 3 2 3 3 3 3 3 3 5 4
17181 037/300 Read block 4907 len 8192
17182 038/300 Read block 1854 len 40960
17183 039/300 Write block 3751 len 40960 data: 3 3 3 3 3 5 5 5 5 6
17184 040/300 Read block 706 len 12288
17185 041/300 Write block 3769 len 40960 data: 5 6 6 5 4 4 4 4 5 4
17186 042/300 Write block 320 len 24576 data: 5 3 4 5 5 4
17187 043/300 Write block 3195 len 16384 data: 4 2 3 3
17188 044/300 Read block 2774 len 24576
17189 045/300 Read block 339 len 20480
17190 046/300 Write block 3486 len 4096 data: 4
17191 047/300 Read block 3288 len 36864
17192 048/300 Write block 1353 len 32768 data: 5 4 5 4 3 4 5 4
17193 049/300 Read block 3837 len 16384
17194 050/300 Write block 1896 len 16384 data: 2 2 4 4
17195 051/300 Read block 951 len 12288
17196 052/300 Write block 3248 len 32768 data: 3 5 5 6 5 5 6 6
17197 053/300 Flush
17198 054/300 Read block 1848 len 20480
17199 055/300 Read block 169 len 12288
17200 056/300 Write block 666 len 24576 data: 5 5 5 5 5 3
17201 057/300 Write block 3501 len 28672 data: 3 2 2 2 2 2 2
17202 058/300 Flush
17203 059/300 Write block 4514 len 20480 data: 2 2 2 3 3
17204 060/300 Write block 157 len 20480 data: 6 5 4 4 4
17205 061/300 Write block 4930 len 40960 data: 5 4 4 5 5 5 4 2 3 3
17206 062/300 Write block 3239 len 28672 data: 5 4 3 3 4 5 6
17207 063/300 Write block 1617 len 4096 data: 4
17208 064/300 Read block 1865 len 36864
17209 065/300 Read block 3253 len 12288
17210 066/300 Write block 777 len 28672 data: 3 4 5 4 3 4 5
17211 067/300 Read block 1137 len 8192
17212 068/300 Read block 1161 len 4096
17213 069/300 Read block 2953 len 12288
17214 070/300 Write block 3304 len 24576 data: 4 5 5 6 5 5
17215 071/300 Read block 3970 len 36864
17216 072/300 Read block 4512 len 36864
17217 073/300 Flush
17218 074/300 Flush
17219 075/300 Flush
17220 076/300 Write block 2150 len 4096 data: 6
17221 077/300 Read block 1506 len 8192
17222 078/300 Write block 798 len 12288 data: 2 2 2
17223 079/300 Write block 1660 len 32768 data: 6 6 6 6 8 7 5 5
17224 080/300 Write block 2371 len 32768 data: 5 4 4 5 3 4 3 3
17225 081/300 Read block 4723 len 12288
17226 082/300 Flush
17227 083/300 Flush
17228 084/300 Read block 2683 len 12288
17229 085/300 Read block 3810 len 40960
17230 086/300 Read block 927 len 36864
17231 087/300 Write block 3414 len 4096 data: 2
17232 088/300 Read block 4964 len 36864
17233 089/300 Read block 2835 len 20480
17234 090/300 Read block 223 len 40960
17235 091/300 Read block 2215 len 36864
17236 092/300 Write block 4195 len 28672 data: 4 4 5 4 4 4 2
17237 093/300 Read block 3079 len 40960
17238 094/300 Flush
17239 095/300 Read block 2438 len 36864
17240 096/300 Read block 2517 len 32768
17241 097/300 Read block 1373 len 24576
17242 098/300 Write block 3907 len 32768 data: 5 5 6 5 5 5 4 3
17243 099/300 Read block 2962 len 16384
17244 100/300 Write block 2477 len 12288 data: 6 5 4
17245 101/300 Write block 4164 len 40960 data: 9 8 7 7 3 3 3 5 5 3
17246 102/300 Read block 164 len 40960
17247 103/300 Read block 4220 len 4096
17248 104/300 Write block 3921 len 40960 data: 3 3 3 3 6 4 5 6 4 4
17249 105/300 Flush
17250 106/300 Write block 1560 len 40960 data: 4 4 5 3 3 3 2 2 2 2
17251 107/300 Read block 4159 len 28672
17252 108/300 Read block 3716 len 40960
17253 109/300 Flush
17254 110/300 Read block 2707 len 28672
17255 111/300 Read block 3630 len 28672
17256 112/300 Read block 1851 len 20480
17257 113/300 Read block 3285 len 36864
17258 114/300 Write block 4187 len 20480 data: 4 3 4 4 5
17259 115/300 Write block 3079 len 16384 data: 5 5 5 5
17260 116/300 Read block 1223 len 16384
17261 117/300 Read block 2164 len 12288
17262 118/300 Read block 4420 len 32768
17263 119/300 Write block 3241 len 32768 data: 4 4 5 6 7 5 4 4
17264 120/300 Read block 1876 len 32768
17265 121/300 Write block 2450 len 40960 data: 3 4 3 3 4 4 4 3 3 4
17266 122/300 Write block 4770 len 40960 data: 8 10 9 8 6 5 4 4 5 5
17267 123/300 Flush
17268 124/300 Read block 4116 len 8192
17269 125/300 Write block 923 len 12288 data: 3 3 2
17270 126/300 Write block 533 len 12288 data: 4 4 4
17271 127/300 Flush
17272 128/300 Read block 2346 len 4096
17273 129/300 Read block 941 len 4096
17274 130/300 Write block 1009 len 40960 data: 3 3 5 5 6 6 5 6 4 4
17275 131/300 Read block 224 len 28672
17276 132/300 Read block 4216 len 36864
17277 133/300 Write block 2153 len 20480 data: 8 9 9 7 8
17278 134/300 Flush
17279 135/300 Read block 4906 len 4096
17280 136/300 Read block 1287 len 36864
17281 137/300 Write block 2244 len 12288 data: 3 4 4
17282 138/300 Write block 3438 len 32768 data: 4 4 4 3 3 4 4 3
17283 139/300 Flush
17284 140/300 Write block 3657 len 32768 data: 3 3 3 4 4 4 5 5
17285 141/300 Write block 1112 len 4096 data: 4
17286 142/300 Read block 4499 len 20480
17287 143/300 Read block 4591 len 36864
17288 144/300 Write block 3918 len 28672 data: 3 3 3 4 4 4 4
17289 145/300 Write block 1275 len 24576 data: 4 3 2 3 2 3
17290 146/300 Read block 2833 len 40960
17291 147/300 Write block 4858 len 36864 data: 5 5 6 6 5 5 4 4 4
17292 148/300 Write block 1698 len 36864 data: 5 5 6 4 4 5 5 5 4
17293 149/300 Write block 2023 len 16384 data: 3 4 4 3
17294 150/300 Write block 3482 len 24576 data: 4 4 5 4 5 4
17295 151/300 Write block 4472 len 40960 data: 3 3 3 4 5 7 8 7 9 9
17296 152/300 Write block 2688 len 28672 data: 6 5 5 6 6 6 5
17297 153/300 Read block 579 len 8192
17298 154/300 Write block 3131 len 32768 data: 3 3 4 4 4 4 5 5
17299 155/300 Write block 1775 len 40960 data: 6 4 4 4 3 3 3 4 4 4
17300 156/300 Write block 947 len 24576 data: 3 3 3 4 4 4
17301 157/300 Flush
17302 158/300 Write block 2145 len 40960 data: 4 6 6 5 6 7 7 8 9 10
17303 159/300 Flush
17304 160/300 Read block 27 len 28672
17305 161/300 Flush
17306 162/300 Write block 149 len 20480 data: 2 3 3 5 5
17307 163/300 Read block 3091 len 20480
17308 164/300 Write block 1163 len 28672 data: 3 2 2 2 4 4 4
17309 165/300 Read block 2309 len 8192
17310 166/300 Read block 2810 len 20480
17311 167/300 Flush
17312 168/300 Write block 2840 len 16384 data: 4 4 4 4
17313 169/300 Read block 2148 len 36864
17314 170/300 Write block 4491 len 4096 data: 5
17315 171/300 Flush
17316 172/300 Write block 2820 len 40960 data: 5 5 3 3 3 3 4 6 6 5
17317 173/300 Read block 876 len 8192
17318 174/300 Read block 2652 len 32768
17319 175/300 Write block 4520 len 32768 data: 4 4 5 6 7 6 6 5
17320 176/300 Read block 968 len 40960
17321 177/300 Read block 856 len 40960
17322 178/300 Flush
17323 179/300 Write block 354 len 36864 data: 4 4 4 4 5 4 6 6 5
17324 180/300 Read block 4022 len 32768
17325 181/300 Read block 3550 len 36864
17326 182/300 Flush
17327 183/300 Write block 4162 len 28672 data: 6 8 10 9 8 8 4
17328 184/300 Write block 91 len 16384 data: 4 3 3 4
17329 185/300 Flush
17330 186/300 Read block 3629 len 8192
17331 187/300 Read block 1563 len 36864
17332 188/300 Read block 1638 len 28672
17333 189/300 Read block 589 len 8192
17334 190/300 Read block 1763 len 36864
17335 191/300 Write block 4631 len 36864 data: 3 3 3 3 3 3 3 3 4
17336 192/300 Write block 2897 len 36864 data: 3 3 5 5 5 5 5 5 5
17337 193/300 Read block 4080 len 24576
17338 194/300 Write block 4866 len 36864 data: 5 4 3 3 3 3 2 3 3
17339 195/300 Read block 3480 len 40960
17340 196/300 Write block 2401 len 36864 data: 3 4 4 4 3 2 3 4 5
17341 197/300 Read block 1329 len 36864
17342 198/300 Flush
17343 199/300 Read block 445 len 8192
17344 200/300 Read block 2291 len 4096
17345 201/300 Read block 1677 len 36864
17346 202/300 Write block 4442 len 8192 data: 2 3
17347 203/300 Write block 4986 len 36864 data: 9 8 8 6 5 3 3 2 2
17348 204/300 Write block 4323 len 36864 data: 5 5 4 4 4 5 5 5 5
17349 205/300 Read block 2693 len 40960
17350 206/300 Read block 2077 len 12288
17351 207/300 Write block 4818 len 32768 data: 4 4 5 5 5 5 4 3
17352 208/300 Write block 3740 len 4096 data: 3
17353 209/300 Read block 1945 len 40960
17354 210/300 Write block 2693 len 24576 data: 7 6 5 5 5 3
17355 211/300 Flush
17356 212/300 Read block 798 len 36864
17357 213/300 Flush
17358 214/300 Flush
17359 215/300 Write block 3153 len 36864 data: 5 5 4 4 3 4 4 3 3
17360 216/300 Write block 978 len 4096 data: 3
17361 217/300 Read block 1532 len 28672
17362 218/300 Read block 1518 len 12288
17363 219/300 Write block 313 len 4096 data: 4
17364 220/300 Read block 3903 len 28672
17365 221/300 Read block 2355 len 8192
17366 222/300 Write block 3498 len 4096 data: 2
17367 223/300 Write block 3215 len 20480 data: 5 4 3 4 4
17368 224/300 Write block 1701 len 32768 data: 5 5 6 6 6 5 4 5
17369 225/300 Read block 3415 len 16384
17370 226/300 Read block 2706 len 12288
17371 227/300 Flush
17372 228/300 Read block 4578 len 36864
17373 229/300 Write block 3078 len 24576 data: 4 6 6 6 6 5
17374 230/300 Read block 783 len 16384
17375 231/300 Write block 3121 len 12288 data: 5 4 2
17376 232/300 Read block 367 len 36864
17377 233/300 Write block 1840 len 40960 data: 3 4 4 5 5 5 5 4 4 4
17378 234/300 Flush
17379 235/300 Flush
17380 236/300 Write block 2994 len 16384 data: 7 7 5 5
17381 237/300 Read block 562 len 40960
17382 238/300 Read block 2578 len 32768
17383 239/300 Flush
17384 240/300 Read block 4443 len 32768
17385 241/300 Write block 4971 len 16384 data: 4 3 3 3
17386 242/300 Flush
17387 243/300 Write block 4153 len 20480 data: 2 2 2 4 5
17388 244/300 Read block 4048 len 28672
17389 245/300 Write block 4564 len 36864 data: 6 7 7 6 6 6 6 6 3
17390 246/300 Read block 8 len 12288
17391 247/300 Read block 2458 len 16384
17392 248/300 Flush
17393 249/300 Read block 2598 len 8192
17394 250/300 Read block 1176 len 36864
17395 251/300 Write block 4625 len 4096 data: 3
17396 252/300 Write block 4470 len 20480 data: 4 4 4 4 4
17397 253/300 Read block 72 len 20480
17398 254/300 Write block 1906 len 20480 data: 4 3 3 3 3
17399 255/300 Flush
17400 256/300 Write block 4382 len 16384 data: 6 7 6 6
17401 257/300 Read block 2005 len 20480
17402 258/300 Write block 3543 len 32768 data: 5 6 7 6 6 6 6 6
17403 259/300 Write block 157 len 8192 data: 7 6
17404 260/300 Read block 3731 len 8192
17405 261/300 Read block 4784 len 40960
17406 262/300 Read block 1474 len 36864
17407 263/300 Read block 2916 len 16384
17408 264/300 Read block 1889 len 36864
17409 265/300 Read block 4961 len 20480
17410 266/300 Read block 865 len 24576
17411 267/300 Write block 4353 len 12288 data: 4 4 5
17412 268/300 Write block 1920 len 32768 data: 5 5 4 4 4 4 3 3
17413 269/300 Read block 2681 len 40960
17414 270/300 Read block 275 len 8192
17415 271/300 Write block 4486 len 8192 data: 10 7
17416 272/300 Read block 3582 len 36864
17417 273/300 Read block 258 len 4096
17418 274/300 Write block 2496 len 4096 data: 4
17419 275/300 Write block 2423 len 8192 data: 4 4
17420 276/300 Write block 3809 len 24576 data: 2 2 2 2 2 2
17421 277/300 Read block 181 len 40960
17422 278/300 Write block 2527 len 40960 data: 3 3 4 5 4 4 4 2 2 2
17423 279/300 Read block 4455 len 24576
17424 280/300 Write block 420 len 28672 data: 4 5 4 4 4 4 4
17425 281/300 Write block 55 len 8192 data: 3 3
17426 282/300 Read block 1917 len 16384
17427 283/300 Read block 3225 len 24576
17428 284/300 Write block 4661 len 12288 data: 6 6 7
17429 285/300 Write block 1925 len 8192 data: 5 4
17430 286/300 Write block 4113 len 36864 data: 4 4 7 6 5 6 5 4 2
17431 287/300 Read block 4633 len 4096
17432 288/300 Write block 2465 len 12288 data: 4 3 4
17433 289/300 Read block 1393 len 16384
17434 290/300 Read block 2517 len 16384
17435 291/300 Write block 1967 len 40960 data: 5 5 5 5 5 4 4 7 6 6
17436 292/300 Read block 4512 len 24576
17437 293/300 Read block 3798 len 20480
17438 294/300 Write block 4253 len 4096 data: 6
17439 295/300 Flush
17440 296/300 Read block 1040 len 24576
17441 297/300 Write block 801 len 12288 data: 2 2 2
17442 298/300 Write block 1552 len 32768 data: 2 2 2 2 3 4 3 2
17443 299/300 Write block 4 len 20480 data: 3 3 6 6 5
17444 300/300 Write block 4658 len 40960 data: 5 6 6 7 7 8 8 8 8 8
17445 301/300 Read block 3102 len 40960
17446 Main task has work to do, go find it
17447 got dsc Start(1)
17448 start 1
17449 Jan 28 21:58:06.090 INFO request completed, latency_us: 236, response_code: 204, uri: /start/cid/1, method: POST, req_id: 33331262-83a3-424a-9def-f18ec076d879, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
17450 Replay: started 1, returned:Ok(())
17451 ----------------------------------------------------------------
17452 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17453 States: Active Offline Active
17454 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17455 5831 Acked Read 2 Done Sent Done false
17456 5832 Acked Write 1 Done Sent Done false
17457 5833 Acked Flush 0 Done Sent Done false
17458 5834 Acked Write 9 Done Sent Done false
17459 5835 Acked Write 1 Done Sent Done false
17460 5836 Acked Flush 0 Done Sent Done false
17461 5837 Acked Write 7 Done Sent Done false
17462 5838 Acked Read 8 Done Sent Done false
17463 5839 Acked Write 8 Done Sent Done false
17464 5840 Acked Read 7 Done Sent Done false
17465 5841 Acked Flush 0 Done Sent Done false
17466 5842 Acked Write 4 Done Sent Done false
17467 5843 Acked Write 4 Done Sent Done false
17468 5844 Acked Write 6 Done Sent Done false
17469 5845 Acked Write 1 Done Sent Done false
17470 5846 Acked Read 3 Done Sent Done false
17471 5847 Acked Read 7 Done Sent Done false
17472 5848 Acked Read 4 Done Sent Done false
17473 5849 Acked Write 10 Done Sent Done false
17474 5850 Acked Write 4 Done Sent Done false
17475 5851 Acked Flush 0 Done Sent Done false
17476 5852 Acked Write 2 Done Sent Done false
17477 5853 Acked Read 1 Done Sent Done false
17478 5854 Acked Write 10 Done Sent Done false
17479 5855 Acked Flush 0 Done Sent Done false
17480 5856 Acked Write 9 Done Sent Done false
17481 5857 Acked Write 1 Done Sent Done false
17482 5858 Acked Write 3 Done Sent Done false
17483 5859 Acked Write 3 Done Sent Done false
17484 5860 Acked Read 2 Done Sent Done false
17485 5861 Acked Read 4 Done Sent Done false
17486 5862 Acked Write 2 Done Sent Done false
17487 5863 Acked Flush 0 Done Sent Done false
17488 5864 Acked Write 8 Done Sent Done false
17489 5865 Acked Write 7 Done Sent Done false
17490 5866 Acked Write 10 Done Sent Done false
17491 5867 Acked Read 2 Done Sent Done false
17492 5868 Acked Read 10 Done Sent Done false
17493 5869 Acked Write 10 Done Sent Done false
17494 5870 Acked Read 3 Done Sent Done false
17495 5871 Acked Write 10 Done Sent Done false
17496 5872 Acked Write 6 Done Sent Done false
17497 5873 Acked Write 4 Done Sent Done false
17498 5874 Acked Read 6 Done Sent Done false
17499 5875 Acked Read 5 Done Sent Done false
17500 5876 Acked Write 1 Done Sent Done false
17501 5877 Acked Read 9 Done Sent Done false
17502 5878 Acked Write 8 Done Sent Done false
17503 5879 Acked Read 4 Done Sent Done false
17504 5880 Acked Write 4 Done Sent Done false
17505 5881 Acked Read 3 Done Sent Done false
17506 5882 Acked Write 8 Done Sent Done false
17507 5883 Acked Flush 0 Done Sent Done false
17508 5884 Acked Read 5 Done Sent Done false
17509 5885 Acked Read 3 Done Sent Done false
17510 5886 Acked Write 6 Done Sent Done false
17511 5887 Acked Write 7 Done Sent Done false
17512 5888 Acked Flush 0 Done Sent Done false
17513 5889 Acked Write 5 Done Sent Done false
17514 5890 Acked Write 5 Done Sent Done false
17515 5891 Acked Write 10 Done Sent Done false
17516 5892 Acked Write 7 Done Sent Done false
17517 5893 Acked Write 1 Done Sent Done false
17518 5894 Acked Read 9 Done Sent Done false
17519 5895 Acked Read 3 Done Sent Done false
17520 5896 Acked Write 7 Done Sent Done false
17521 5897 Acked Read 2 Done Sent Done false
17522 5898 Acked Read 1 Done Sent Done false
17523 5899 Acked Read 3 Done Sent Done false
17524 5900 Acked Write 6 Done Sent Done false
17525 5901 Acked Read 9 Done Sent Done false
17526 5902 Acked Read 9 Done Sent Done false
17527 5903 Acked Flush 0 Done Sent Done false
17528 5904 Acked Flush 0 Done Sent Done false
17529 5905 Acked Flush 0 Done Sent Done false
17530 5906 Acked Write 1 Done Sent Done false
17531 5907 Acked Read 2 Done Sent Done false
17532 5908 Acked Write 3 Done Sent Done false
17533 5909 Acked Write 8 Done Sent Done false
17534 5910 Acked Write 8 Done Sent Done false
17535 5911 Acked Read 3 Done Sent Done false
17536 5912 Acked Flush 0 Done Sent Done false
17537 5913 Acked Flush 0 Done Sent Done false
17538 5914 Acked Read 3 Done Sent Done false
17539 5915 Acked Read 10 Done Sent Done false
17540 5916 Acked Read 9 Done Sent Done false
17541 5917 Acked Write 1 Done Sent Done false
17542 5918 Acked Read 9 Done Sent Done false
17543 5919 Acked Read 5 Done Sent Done false
17544 5920 Acked Read 10 Done Sent Done false
17545 5921 Acked Read 9 Done Sent Done false
17546 5922 Acked Write 7 Done Sent Done false
17547 5923 Acked Read 10 Done Sent Done false
17548 5924 Acked Flush 0 Done Sent Done false
17549 5925 Acked Read 9 Done Sent Done false
17550 5926 Acked Read 8 Done Sent Done false
17551 5927 Acked Read 6 Done Sent Done false
17552 5928 Acked Write 8 Done Sent Done false
17553 5929 Acked Read 4 Done Sent Done false
17554 5930 Acked Write 3 Done Sent Done false
17555 5931 Acked Write 10 Done Sent Done false
17556 5932 Acked Read 10 Done Sent Done false
17557 5933 Acked Read 1 Done Sent Done false
17558 5934 Acked Write 10 Done Sent Done false
17559 5935 Acked Flush 0 Done Sent Done false
17560 5936 Acked Write 10 Done Sent Done false
17561 5937 Acked Read 7 Done Sent Done false
17562 5938 Acked Read 10 Done Sent Done false
17563 5939 Acked Flush 0 Done Sent Done false
17564 5940 Acked Read 7 Done Sent Done false
17565 5941 Acked Read 7 Done Sent Done false
17566 5942 Acked Read 5 Done Sent Done false
17567 5943 Acked Read 9 Done Sent Done false
17568 5944 Acked Write 5 Done Sent Done false
17569 5945 Acked Write 4 Done Sent Done false
17570 5946 Acked Read 4 Done Sent Done false
17571 5947 Acked Read 3 Done Sent Done false
17572 5948 Acked Read 8 Done Sent Done false
17573 5949 Acked Write 8 Done Sent Done false
17574 5950 Acked Read 8 Done Sent Done false
17575 5951 Acked Write 10 Done Sent Done false
17576 5952 Acked Write 10 Done Sent Done false
17577 5953 Acked Flush 0 Done Sent Done false
17578 5954 Acked Read 2 Done Sent Done false
17579 5955 Acked Write 3 Done Sent Done false
17580 5956 Acked Write 3 Done Sent Done false
17581 5957 Acked Flush 0 Done Sent Done false
17582 5958 Acked Read 1 Done Sent Done false
17583 5959 Acked Read 1 Done Sent Done false
17584 5960 Acked Write 10 Done Sent Done false
17585 5961 Acked Read 7 Done Sent Done false
17586 5962 Acked Read 9 Done Sent Done false
17587 5963 Acked Write 5 Done Sent Done false
17588 5964 Acked Flush 0 Done Sent Done false
17589 5965 Acked Read 1 Done Sent Done false
17590 5966 Acked Read 9 Done Sent Done false
17591 5967 Acked Write 3 Done Sent Done false
17592 5968 Acked Write 8 Done Sent Done false
17593 5969 Acked Flush 0 Done Sent Done false
17594 5970 Acked Write 8 Done Sent Done false
17595 5971 Acked Write 1 Done Sent Done false
17596 5972 Acked Read 5 Done Sent Done false
17597 5973 Acked Read 9 Done Sent Done false
17598 5974 Acked Write 7 Done Sent Done false
17599 5975 Acked Write 6 Done Sent Done false
17600 5976 Acked Read 10 Done Sent Done false
17601 5977 Acked Write 9 Done Sent Done false
17602 5978 Acked Write 9 Done Sent Done false
17603 5979 Acked Write 4 Done Sent Done false
17604 5980 Acked Write 6 Done Sent Done false
17605 5981 Acked Write 10 Done Sent Done false
17606 5982 Acked Write 7 Done Sent Done false
17607 5983 Acked Read 2 Done Sent Done false
17608 5984 Acked Write 8 Done Sent Done false
17609 5985 Acked Write 10 Done Sent Done false
17610 5986 Acked Write 6 Done Sent Done false
17611 5987 Acked Flush 0 Done Sent Done false
17612 5988 Acked Write 10 Done Sent Done false
17613 5989 Acked Flush 0 Done Sent Done false
17614 5990 Acked Read 7 Done Sent Done false
17615 5991 Acked Flush 0 Done Sent Done false
17616 5992 Acked Write 5 Done Sent Done false
17617 5993 Acked Read 5 Done Sent Done false
17618 5994 Acked Write 7 Done Sent Done false
17619 5995 Acked Read 2 Done Sent Done false
17620 5996 Acked Read 5 Done Sent Done false
17621 5997 Acked Flush 0 Done Sent Done false
17622 5998 Acked Write 4 Done Sent Done false
17623 5999 Acked Read 9 Done Sent Done false
17624 6000 Acked Write 1 Done Sent Done false
17625 6001 Acked Flush 0 Done Sent Done false
17626 6002 Acked Write 10 Done Sent Done false
17627 6003 Acked Read 2 Done Sent Done false
17628 6004 Acked Read 8 Done Sent Done false
17629 6005 Acked Write 8 Done Sent Done false
17630 6006 Acked Read 10 Done Sent Done false
17631 6007 Acked Read 10 Done Sent Done false
17632 6008 Acked Flush 0 Done Sent Done false
17633 6009 Acked Write 9 Done Sent Done false
17634 6010 Acked Read 8 Done Sent Done false
17635 6011 Acked Read 9 Done Sent Done false
17636 6012 Acked Flush 0 Done Sent Done false
17637 6013 Acked Write 7 Done Sent Done false
17638 6014 Acked Write 4 Done Sent Done false
17639 6015 Acked Flush 0 Done Sent Done false
17640 6016 Acked Read 2 Done Sent Done false
17641 6017 Acked Read 9 Done Sent Done false
17642 6018 Acked Read 7 Done Sent Done false
17643 6019 Acked Read 2 Done Sent Done false
17644 6020 Acked Read 9 Done Sent Done false
17645 6021 Acked Write 9 Done Sent Done false
17646 6022 Acked Write 9 Done Sent Done false
17647 6023 Acked Read 6 Done Sent Done false
17648 6024 Acked Write 9 Done Sent Done false
17649 6025 Acked Read 10 Done Sent Done false
17650 6026 Acked Write 9 Done Sent Done false
17651 6027 Acked Read 9 Done Sent Done false
17652 6028 Acked Flush 0 Done Sent Done false
17653 6029 Acked Read 2 Done Sent Done false
17654 6030 Acked Read 1 Done Sent Done false
17655 6031 Acked Read 9 Done Sent Done false
17656 6032 Acked Write 2 Done Sent Done false
17657 6033 Acked Write 9 Done Sent Done false
17658 6034 Acked Write 9 Done Sent Done false
17659 6035 Acked Read 10 Done Sent Done false
17660 6036 Acked Read 3 Done Sent Done false
17661 6037 Acked Write 8 Done Sent Done false
17662 6038 Acked Write 1 Done Sent Done false
17663 6039 Acked Read 10 Done Sent Done false
17664 6040 Acked Write 6 Done Sent Done false
17665 6041 Acked Flush 0 Done Sent Done false
17666 6042 Acked Read 9 Done Sent Done false
17667 6043 Acked Flush 0 Done Sent Done false
17668 6044 Acked Flush 0 Done Sent Done false
17669 6045 Acked Write 9 Done Sent Done false
17670 6046 Acked Write 1 Done Sent Done false
17671 6047 Acked Read 7 Done Sent Done false
17672 6048 Acked Read 3 Done Sent Done false
17673 6049 Acked Write 1 Done Sent Done false
17674 6050 Acked Read 7 Done Sent Done false
17675 6051 Acked Read 2 Done Sent Done false
17676 6052 Acked Write 1 Done Sent Done false
17677 6053 Acked Write 5 Done Sent Done false
17678 6054 Acked Write 8 Done Sent Done false
17679 6055 Acked Read 4 Done Sent Done false
17680 6056 Acked Read 3 Done Sent Done false
17681 6057 Acked Flush 0 Done Sent Done false
17682 6058 Acked Read 9 Done Sent Done false
17683 6059 Acked Write 6 Done Sent Done false
17684 6060 Acked Read 4 Done Sent Done false
17685 6061 Acked Write 3 Done Sent Done false
17686 6062 Acked Read 9 Done Sent Done false
17687 6063 Acked Write 10 Done Sent Done false
17688 6064 Acked Flush 0 Done Sent Done false
17689 6065 Acked Flush 0 Done Sent Done false
17690 6066 Acked Write 4 Done Sent Done false
17691 6067 Acked Read 10 Done Sent Done false
17692 6068 Acked Read 8 Done Sent Done false
17693 6069 Acked Flush 0 Done Sent Done false
17694 6070 Acked Read 8 Done Sent Done false
17695 6071 Acked Write 4 Done Sent Done false
17696 6072 Acked Flush 0 Done Sent Done false
17697 6073 Acked Write 5 Done Sent Done false
17698 6074 Acked Read 7 Done Sent Done false
17699 6075 Acked Write 9 Done Sent Done false
17700 6076 Acked Read 3 Done Sent Done false
17701 6077 Acked Read 4 Done Sent Done false
17702 6078 Acked Flush 0 Done Sent Done false
17703 6079 Acked Read 2 Done Sent Done false
17704 6080 Acked Read 9 Done Sent Done false
17705 6081 Acked Write 1 Done Sent Done false
17706 6082 Acked Write 5 Done Sent Done false
17707 6083 Acked Read 5 Done Sent Done false
17708 6084 Acked Write 5 Done Sent Done false
17709 6085 Acked Flush 0 Done Sent Done false
17710 6086 Acked Write 4 Done Sent Done false
17711 6087 Acked Read 5 Done Sent Done false
17712 6088 Acked Write 8 Done Sent Done false
17713 6089 Acked Write 2 Done Sent Done false
17714 6090 Acked Read 2 Done Sent Done false
17715 6091 Acked Read 10 Done Sent Done false
17716 6092 Acked Read 9 Done Sent Done false
17717 6093 Acked Read 4 Done Sent Done false
17718 6094 Acked Read 9 Done Sent Done false
17719 6095 Acked Read 5 Done Sent Done false
17720 6096 Acked Read 6 Done Sent Done false
17721 6097 Acked Write 3 Done Sent Done false
17722 6098 Acked Write 8 Done Sent Done false
17723 6099 Acked Read 10 Done Sent Done false
17724 6100 Acked Read 2 Done Sent Done false
17725 6101 Acked Write 2 Done Sent Done false
17726 6102 Acked Read 9 Done Sent Done false
17727 6103 Acked Read 1 Done Sent Done false
17728 6104 Acked Write 1 Done Sent Done false
17729 6105 Acked Write 2 Done Sent Done false
17730 6106 Acked Write 6 Done Sent Done false
17731 6107 Acked Read 10 Done Sent Done false
17732 6108 Acked Write 10 Done Sent Done false
17733 6109 Acked Read 6 Done Sent Done false
17734 6110 Acked Write 7 Done Sent Done false
17735 6111 Acked Write 2 Done Sent Done false
17736 6112 Acked Read 4 Done Sent Done false
17737 6113 Acked Read 6 Done Sent Done false
17738 6114 Acked Write 3 Done Sent Done false
17739 6115 Acked Write 2 Done Sent Done false
17740 6116 Acked Write 9 Done Sent Done false
17741 6117 Acked Read 1 Done Sent Done false
17742 6118 Acked Write 3 Done Sent Done false
17743 6119 Acked Read 4 Done Sent Done false
17744 6120 Acked Read 4 Done Sent Done false
17745 6121 Acked Write 10 Done Sent Done false
17746 6122 Acked Read 6 Done Sent Done false
17747 6123 Acked Read 5 Done Sent Done false
17748 6124 Acked Write 1 Done Sent Done false
17749 6125 Acked Flush 0 Done Sent Done false
17750 6126 Acked Read 6 Done Sent Done false
17751 6127 Acked Write 3 Done Sent Done false
17752 6128 Acked Write 8 Done Sent Done false
17753 6129 Acked Write 5 Done Sent Done false
17754 6130 Acked Write 10 Done Sent Done false
17755 6131 Acked Read 10 Done Sent Done false
17756 STATES DS:0 DS:1 DS:2 TOTAL
17757 Sent 0 301 0 301
17758 Done 301 0 301 602
17759 Skipped 0 0 0 0
17760 Error 0 0 0 0
17761 Last Flush: Some(JobId(6125)) Some(JobId(5830)) Some(JobId(6125))
17762 Downstairs last five completed: 5830 5829 5828 5827 5826
17763 Upstairs last five completed: 6131 6130 6129 6128 6127
17764 CLIENT: Up:0 ds:301 act:2
177652025-01-28T21:58:06.667ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
177662025-01-28T21:58:07.668ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
177672025-01-28T21:58:08.670ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
17768 ----------------------------------------------------------------
17769 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17770 States: Active Offline Active
17771 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17772 5831 Acked Read 2 Done Sent Done false
17773 5832 Acked Write 1 Done Sent Done false
17774 5833 Acked Flush 0 Done Sent Done false
17775 5834 Acked Write 9 Done Sent Done false
17776 5835 Acked Write 1 Done Sent Done false
17777 5836 Acked Flush 0 Done Sent Done false
17778 5837 Acked Write 7 Done Sent Done false
17779 5838 Acked Read 8 Done Sent Done false
17780 5839 Acked Write 8 Done Sent Done false
17781 5840 Acked Read 7 Done Sent Done false
17782 5841 Acked Flush 0 Done Sent Done false
17783 5842 Acked Write 4 Done Sent Done false
17784 5843 Acked Write 4 Done Sent Done false
17785 5844 Acked Write 6 Done Sent Done false
17786 5845 Acked Write 1 Done Sent Done false
17787 5846 Acked Read 3 Done Sent Done false
17788 5847 Acked Read 7 Done Sent Done false
17789 5848 Acked Read 4 Done Sent Done false
17790 5849 Acked Write 10 Done Sent Done false
17791 5850 Acked Write 4 Done Sent Done false
17792 5851 Acked Flush 0 Done Sent Done false
17793 5852 Acked Write 2 Done Sent Done false
17794 5853 Acked Read 1 Done Sent Done false
17795 5854 Acked Write 10 Done Sent Done false
17796 5855 Acked Flush 0 Done Sent Done false
17797 5856 Acked Write 9 Done Sent Done false
17798 5857 Acked Write 1 Done Sent Done false
17799 5858 Acked Write 3 Done Sent Done false
17800 5859 Acked Write 3 Done Sent Done false
17801 5860 Acked Read 2 Done Sent Done false
17802 5861 Acked Read 4 Done Sent Done false
17803 5862 Acked Write 2 Done Sent Done false
17804 5863 Acked Flush 0 Done Sent Done false
17805 5864 Acked Write 8 Done Sent Done false
17806 5865 Acked Write 7 Done Sent Done false
17807 5866 Acked Write 10 Done Sent Done false
17808 5867 Acked Read 2 Done Sent Done false
17809 5868 Acked Read 10 Done Sent Done false
17810 5869 Acked Write 10 Done Sent Done false
17811 5870 Acked Read 3 Done Sent Done false
17812 5871 Acked Write 10 Done Sent Done false
17813 5872 Acked Write 6 Done Sent Done false
17814 5873 Acked Write 4 Done Sent Done false
17815 5874 Acked Read 6 Done Sent Done false
17816 5875 Acked Read 5 Done Sent Done false
17817 5876 Acked Write 1 Done Sent Done false
17818 5877 Acked Read 9 Done Sent Done false
17819 5878 Acked Write 8 Done Sent Done false
17820 5879 Acked Read 4 Done Sent Done false
17821 5880 Acked Write 4 Done Sent Done false
17822 5881 Acked Read 3 Done Sent Done false
17823 5882 Acked Write 8 Done Sent Done false
17824 5883 Acked Flush 0 Done Sent Done false
17825 5884 Acked Read 5 Done Sent Done false
17826 5885 Acked Read 3 Done Sent Done false
17827 5886 Acked Write 6 Done Sent Done false
17828 5887 Acked Write 7 Done Sent Done false
17829 5888 Acked Flush 0 Done Sent Done false
17830 5889 Acked Write 5 Done Sent Done false
17831 5890 Acked Write 5 Done Sent Done false
17832 5891 Acked Write 10 Done Sent Done false
17833 5892 Acked Write 7 Done Sent Done false
17834 5893 Acked Write 1 Done Sent Done false
17835 5894 Acked Read 9 Done Sent Done false
17836 5895 Acked Read 3 Done Sent Done false
17837 5896 Acked Write 7 Done Sent Done false
17838 5897 Acked Read 2 Done Sent Done false
17839 5898 Acked Read 1 Done Sent Done false
17840 5899 Acked Read 3 Done Sent Done false
17841 5900 Acked Write 6 Done Sent Done false
17842 5901 Acked Read 9 Done Sent Done false
17843 5902 Acked Read 9 Done Sent Done false
17844 5903 Acked Flush 0 Done Sent Done false
17845 5904 Acked Flush 0 Done Sent Done false
17846 5905 Acked Flush 0 Done Sent Done false
17847 5906 Acked Write 1 Done Sent Done false
17848 5907 Acked Read 2 Done Sent Done false
17849 5908 Acked Write 3 Done Sent Done false
17850 5909 Acked Write 8 Done Sent Done false
17851 5910 Acked Write 8 Done Sent Done false
17852 5911 Acked Read 3 Done Sent Done false
17853 5912 Acked Flush 0 Done Sent Done false
17854 5913 Acked Flush 0 Done Sent Done false
17855 5914 Acked Read 3 Done Sent Done false
17856 5915 Acked Read 10 Done Sent Done false
17857 5916 Acked Read 9 Done Sent Done false
17858 5917 Acked Write 1 Done Sent Done false
17859 5918 Acked Read 9 Done Sent Done false
17860 5919 Acked Read 5 Done Sent Done false
17861 5920 Acked Read 10 Done Sent Done false
17862 5921 Acked Read 9 Done Sent Done false
17863 5922 Acked Write 7 Done Sent Done false
17864 5923 Acked Read 10 Done Sent Done false
17865 5924 Acked Flush 0 Done Sent Done false
17866 5925 Acked Read 9 Done Sent Done false
17867 5926 Acked Read 8 Done Sent Done false
17868 5927 Acked Read 6 Done Sent Done false
17869 5928 Acked Write 8 Done Sent Done false
17870 5929 Acked Read 4 Done Sent Done false
17871 5930 Acked Write 3 Done Sent Done false
17872 5931 Acked Write 10 Done Sent Done false
17873 5932 Acked Read 10 Done Sent Done false
17874 5933 Acked Read 1 Done Sent Done false
17875 5934 Acked Write 10 Done Sent Done false
17876 5935 Acked Flush 0 Done Sent Done false
17877 5936 Acked Write 10 Done Sent Done false
17878 5937 Acked Read 7 Done Sent Done false
17879 5938 Acked Read 10 Done Sent Done false
17880 5939 Acked Flush 0 Done Sent Done false
17881 5940 Acked Read 7 Done Sent Done false
17882 5941 Acked Read 7 Done Sent Done false
17883 5942 Acked Read 5 Done Sent Done false
17884 5943 Acked Read 9 Done Sent Done false
17885 5944 Acked Write 5 Done Sent Done false
17886 5945 Acked Write 4 Done Sent Done false
17887 5946 Acked Read 4 Done Sent Done false
17888 5947 Acked Read 3 Done Sent Done false
17889 5948 Acked Read 8 Done Sent Done false
17890 5949 Acked Write 8 Done Sent Done false
17891 5950 Acked Read 8 Done Sent Done false
17892 5951 Acked Write 10 Done Sent Done false
17893 5952 Acked Write 10 Done Sent Done false
17894 5953 Acked Flush 0 Done Sent Done false
17895 5954 Acked Read 2 Done Sent Done false
17896 5955 Acked Write 3 Done Sent Done false
17897 5956 Acked Write 3 Done Sent Done false
17898 5957 Acked Flush 0 Done Sent Done false
17899 5958 Acked Read 1 Done Sent Done false
17900 5959 Acked Read 1 Done Sent Done false
17901 5960 Acked Write 10 Done Sent Done false
17902 5961 Acked Read 7 Done Sent Done false
17903 5962 Acked Read 9 Done Sent Done false
17904 5963 Acked Write 5 Done Sent Done false
17905 5964 Acked Flush 0 Done Sent Done false
17906 5965 Acked Read 1 Done Sent Done false
17907 5966 Acked Read 9 Done Sent Done false
17908 5967 Acked Write 3 Done Sent Done false
17909 5968 Acked Write 8 Done Sent Done false
17910 5969 Acked Flush 0 Done Sent Done false
17911 5970 Acked Write 8 Done Sent Done false
17912 5971 Acked Write 1 Done Sent Done false
17913 5972 Acked Read 5 Done Sent Done false
17914 5973 Acked Read 9 Done Sent Done false
17915 5974 Acked Write 7 Done Sent Done false
17916 5975 Acked Write 6 Done Sent Done false
17917 5976 Acked Read 10 Done Sent Done false
17918 5977 Acked Write 9 Done Sent Done false
17919 5978 Acked Write 9 Done Sent Done false
17920 5979 Acked Write 4 Done Sent Done false
17921 5980 Acked Write 6 Done Sent Done false
17922 5981 Acked Write 10 Done Sent Done false
17923 5982 Acked Write 7 Done Sent Done false
17924 5983 Acked Read 2 Done Sent Done false
17925 5984 Acked Write 8 Done Sent Done false
17926 5985 Acked Write 10 Done Sent Done false
17927 5986 Acked Write 6 Done Sent Done false
17928 5987 Acked Flush 0 Done Sent Done false
17929 5988 Acked Write 10 Done Sent Done false
17930 5989 Acked Flush 0 Done Sent Done false
17931 5990 Acked Read 7 Done Sent Done false
17932 5991 Acked Flush 0 Done Sent Done false
17933 5992 Acked Write 5 Done Sent Done false
17934 5993 Acked Read 5 Done Sent Done false
17935 5994 Acked Write 7 Done Sent Done false
17936 5995 Acked Read 2 Done Sent Done false
17937 5996 Acked Read 5 Done Sent Done false
17938 5997 Acked Flush 0 Done Sent Done false
17939 5998 Acked Write 4 Done Sent Done false
17940 5999 Acked Read 9 Done Sent Done false
17941 6000 Acked Write 1 Done Sent Done false
17942 6001 Acked Flush 0 Done Sent Done false
17943 6002 Acked Write 10 Done Sent Done false
17944 6003 Acked Read 2 Done Sent Done false
17945 6004 Acked Read 8 Done Sent Done false
17946 6005 Acked Write 8 Done Sent Done false
17947 6006 Acked Read 10 Done Sent Done false
17948 6007 Acked Read 10 Done Sent Done false
17949 6008 Acked Flush 0 Done Sent Done false
17950 6009 Acked Write 9 Done Sent Done false
17951 6010 Acked Read 8 Done Sent Done false
17952 6011 Acked Read 9 Done Sent Done false
17953 6012 Acked Flush 0 Done Sent Done false
17954 6013 Acked Write 7 Done Sent Done false
17955 6014 Acked Write 4 Done Sent Done false
17956 6015 Acked Flush 0 Done Sent Done false
17957 6016 Acked Read 2 Done Sent Done false
17958 6017 Acked Read 9 Done Sent Done false
17959 6018 Acked Read 7 Done Sent Done false
17960 6019 Acked Read 2 Done Sent Done false
17961 6020 Acked Read 9 Done Sent Done false
17962 6021 Acked Write 9 Done Sent Done false
17963 6022 Acked Write 9 Done Sent Done false
17964 6023 Acked Read 6 Done Sent Done false
17965 6024 Acked Write 9 Done Sent Done false
17966 6025 Acked Read 10 Done Sent Done false
17967 6026 Acked Write 9 Done Sent Done false
17968 6027 Acked Read 9 Done Sent Done false
17969 6028 Acked Flush 0 Done Sent Done false
17970 6029 Acked Read 2 Done Sent Done false
17971 6030 Acked Read 1 Done Sent Done false
17972 6031 Acked Read 9 Done Sent Done false
17973 6032 Acked Write 2 Done Sent Done false
17974 6033 Acked Write 9 Done Sent Done false
17975 6034 Acked Write 9 Done Sent Done false
17976 6035 Acked Read 10 Done Sent Done false
17977 6036 Acked Read 3 Done Sent Done false
17978 6037 Acked Write 8 Done Sent Done false
17979 6038 Acked Write 1 Done Sent Done false
17980 6039 Acked Read 10 Done Sent Done false
17981 6040 Acked Write 6 Done Sent Done false
17982 6041 Acked Flush 0 Done Sent Done false
17983 6042 Acked Read 9 Done Sent Done false
17984 6043 Acked Flush 0 Done Sent Done false
17985 6044 Acked Flush 0 Done Sent Done false
17986 6045 Acked Write 9 Done Sent Done false
17987 6046 Acked Write 1 Done Sent Done false
17988 6047 Acked Read 7 Done Sent Done false
17989 6048 Acked Read 3 Done Sent Done false
17990 6049 Acked Write 1 Done Sent Done false
17991 6050 Acked Read 7 Done Sent Done false
17992 6051 Acked Read 2 Done Sent Done false
17993 6052 Acked Write 1 Done Sent Done false
17994 6053 Acked Write 5 Done Sent Done false
17995 6054 Acked Write 8 Done Sent Done false
17996 6055 Acked Read 4 Done Sent Done false
17997 6056 Acked Read 3 Done Sent Done false
17998 6057 Acked Flush 0 Done Sent Done false
17999 6058 Acked Read 9 Done Sent Done false
18000 6059 Acked Write 6 Done Sent Done false
18001 6060 Acked Read 4 Done Sent Done false
18002 6061 Acked Write 3 Done Sent Done false
18003 6062 Acked Read 9 Done Sent Done false
18004 6063 Acked Write 10 Done Sent Done false
18005 6064 Acked Flush 0 Done Sent Done false
18006 6065 Acked Flush 0 Done Sent Done false
18007 6066 Acked Write 4 Done Sent Done false
18008 6067 Acked Read 10 Done Sent Done false
18009 6068 Acked Read 8 Done Sent Done false
18010 6069 Acked Flush 0 Done Sent Done false
18011 6070 Acked Read 8 Done Sent Done false
18012 6071 Acked Write 4 Done Sent Done false
18013 6072 Acked Flush 0 Done Sent Done false
18014 6073 Acked Write 5 Done Sent Done false
18015 6074 Acked Read 7 Done Sent Done false
18016 6075 Acked Write 9 Done Sent Done false
18017 6076 Acked Read 3 Done Sent Done false
18018 6077 Acked Read 4 Done Sent Done false
18019 6078 Acked Flush 0 Done Sent Done false
18020 6079 Acked Read 2 Done Sent Done false
18021 6080 Acked Read 9 Done Sent Done false
18022 6081 Acked Write 1 Done Sent Done false
18023 6082 Acked Write 5 Done Sent Done false
18024 6083 Acked Read 5 Done Sent Done false
18025 6084 Acked Write 5 Done Sent Done false
18026 6085 Acked Flush 0 Done Sent Done false
18027 6086 Acked Write 4 Done Sent Done false
18028 6087 Acked Read 5 Done Sent Done false
18029 6088 Acked Write 8 Done Sent Done false
18030 6089 Acked Write 2 Done Sent Done false
18031 6090 Acked Read 2 Done Sent Done false
18032 6091 Acked Read 10 Done Sent Done false
18033 6092 Acked Read 9 Done Sent Done false
18034 6093 Acked Read 4 Done Sent Done false
18035 6094 Acked Read 9 Done Sent Done false
18036 6095 Acked Read 5 Done Sent Done false
18037 6096 Acked Read 6 Done Sent Done false
18038 6097 Acked Write 3 Done Sent Done false
18039 6098 Acked Write 8 Done Sent Done false
18040 6099 Acked Read 10 Done Sent Done false
18041 6100 Acked Read 2 Done Sent Done false
18042 6101 Acked Write 2 Done Sent Done false
18043 6102 Acked Read 9 Done Sent Done false
18044 6103 Acked Read 1 Done Sent Done false
18045 6104 Acked Write 1 Done Sent Done false
18046 6105 Acked Write 2 Done Sent Done false
18047 6106 Acked Write 6 Done Sent Done false
18048 6107 Acked Read 10 Done Sent Done false
18049 6108 Acked Write 10 Done Sent Done false
18050 6109 Acked Read 6 Done Sent Done false
18051 6110 Acked Write 7 Done Sent Done false
18052 6111 Acked Write 2 Done Sent Done false
18053 6112 Acked Read 4 Done Sent Done false
18054 6113 Acked Read 6 Done Sent Done false
18055 6114 Acked Write 3 Done Sent Done false
18056 6115 Acked Write 2 Done Sent Done false
18057 6116 Acked Write 9 Done Sent Done false
18058 6117 Acked Read 1 Done Sent Done false
18059 6118 Acked Write 3 Done Sent Done false
18060 6119 Acked Read 4 Done Sent Done false
18061 6120 Acked Read 4 Done Sent Done false
18062 6121 Acked Write 10 Done Sent Done false
18063 6122 Acked Read 6 Done Sent Done false
18064 6123 Acked Read 5 Done Sent Done false
18065 6124 Acked Write 1 Done Sent Done false
18066 6125 Acked Flush 0 Done Sent Done false
18067 6126 Acked Read 6 Done Sent Done false
18068 6127 Acked Write 3 Done Sent Done false
18069 6128 Acked Write 8 Done Sent Done false
18070 6129 Acked Write 5 Done Sent Done false
18071 6130 Acked Write 10 Done Sent Done false
18072 6131 Acked Read 10 Done Sent Done false
18073 STATES DS:0 DS:1 DS:2 TOTAL
18074 Sent 0 301 0 301
18075 Done 301 0 301 602
18076 Skipped 0 0 0 0
18077 Error 0 0 0 0
18078 Last Flush: Some(JobId(6125)) Some(JobId(5830)) Some(JobId(6125))
18079 Downstairs last five completed: 5830 5829 5828 5827 5826
18080 Upstairs last five completed: 6131 6130 6129 6128 6127
18081 CLIENT: Up:0 ds:301 act:2
180822025-01-28T21:58:10.674ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
180832025-01-28T21:58:11.676ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
180842025-01-28T21:58:11.847ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
180852025-01-28T21:58:11.847ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
180862025-01-28T21:58:11.847ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
180872025-01-28T21:58:11.847ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
180882025-01-28T21:58:11.847ZINFOcrucible: client task is exiting = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
18089 [8820] got start action
18090 [8820] I am going to restart
18091 Starting downstairs at port 8820
18092 Make output file at "/tmp/dsc/downstairs-8820.txt"
18093 [8820][1] reports Starting
18094 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1219)
18095 [8820][1] reports Running
180962025-01-28T21:58:12.678ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
180972025-01-28T21:58:13.680ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
18098 ----------------------------------------------------------------
18099 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18100 States: Active Offline Active
18101 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18102 5831 Acked Read 2 Done Sent Done false
18103 5832 Acked Write 1 Done Sent Done false
18104 5833 Acked Flush 0 Done Sent Done false
18105 5834 Acked Write 9 Done Sent Done false
18106 5835 Acked Write 1 Done Sent Done false
18107 5836 Acked Flush 0 Done Sent Done false
18108 5837 Acked Write 7 Done Sent Done false
18109 5838 Acked Read 8 Done Sent Done false
18110 5839 Acked Write 8 Done Sent Done false
18111 5840 Acked Read 7 Done Sent Done false
18112 5841 Acked Flush 0 Done Sent Done false
18113 5842 Acked Write 4 Done Sent Done false
18114 5843 Acked Write 4 Done Sent Done false
18115 5844 Acked Write 6 Done Sent Done false
18116 5845 Acked Write 1 Done Sent Done false
18117 5846 Acked Read 3 Done Sent Done false
18118 5847 Acked Read 7 Done Sent Done false
18119 5848 Acked Read 4 Done Sent Done false
18120 5849 Acked Write 10 Done Sent Done false
18121 5850 Acked Write 4 Done Sent Done false
18122 5851 Acked Flush 0 Done Sent Done false
18123 5852 Acked Write 2 Done Sent Done false
18124 5853 Acked Read 1 Done Sent Done false
18125 5854 Acked Write 10 Done Sent Done false
18126 5855 Acked Flush 0 Done Sent Done false
18127 5856 Acked Write 9 Done Sent Done false
18128 5857 Acked Write 1 Done Sent Done false
18129 5858 Acked Write 3 Done Sent Done false
18130 5859 Acked Write 3 Done Sent Done false
18131 5860 Acked Read 2 Done Sent Done false
18132 5861 Acked Read 4 Done Sent Done false
18133 5862 Acked Write 2 Done Sent Done false
18134 5863 Acked Flush 0 Done Sent Done false
18135 5864 Acked Write 8 Done Sent Done false
18136 5865 Acked Write 7 Done Sent Done false
18137 5866 Acked Write 10 Done Sent Done false
18138 5867 Acked Read 2 Done Sent Done false
18139 5868 Acked Read 10 Done Sent Done false
18140 5869 Acked Write 10 Done Sent Done false
18141 5870 Acked Read 3 Done Sent Done false
18142 5871 Acked Write 10 Done Sent Done false
18143 5872 Acked Write 6 Done Sent Done false
18144 5873 Acked Write 4 Done Sent Done false
18145 5874 Acked Read 6 Done Sent Done false
18146 5875 Acked Read 5 Done Sent Done false
18147 5876 Acked Write 1 Done Sent Done false
18148 5877 Acked Read 9 Done Sent Done false
18149 5878 Acked Write 8 Done Sent Done false
18150 5879 Acked Read 4 Done Sent Done false
18151 5880 Acked Write 4 Done Sent Done false
18152 5881 Acked Read 3 Done Sent Done false
18153 5882 Acked Write 8 Done Sent Done false
18154 5883 Acked Flush 0 Done Sent Done false
18155 5884 Acked Read 5 Done Sent Done false
18156 5885 Acked Read 3 Done Sent Done false
18157 5886 Acked Write 6 Done Sent Done false
18158 5887 Acked Write 7 Done Sent Done false
18159 5888 Acked Flush 0 Done Sent Done false
18160 5889 Acked Write 5 Done Sent Done false
18161 5890 Acked Write 5 Done Sent Done false
18162 5891 Acked Write 10 Done Sent Done false
18163 5892 Acked Write 7 Done Sent Done false
18164 5893 Acked Write 1 Done Sent Done false
18165 5894 Acked Read 9 Done Sent Done false
18166 5895 Acked Read 3 Done Sent Done false
18167 5896 Acked Write 7 Done Sent Done false
18168 5897 Acked Read 2 Done Sent Done false
18169 5898 Acked Read 1 Done Sent Done false
18170 5899 Acked Read 3 Done Sent Done false
18171 5900 Acked Write 6 Done Sent Done false
18172 5901 Acked Read 9 Done Sent Done false
18173 5902 Acked Read 9 Done Sent Done false
18174 5903 Acked Flush 0 Done Sent Done false
18175 5904 Acked Flush 0 Done Sent Done false
18176 5905 Acked Flush 0 Done Sent Done false
18177 5906 Acked Write 1 Done Sent Done false
18178 5907 Acked Read 2 Done Sent Done false
18179 5908 Acked Write 3 Done Sent Done false
18180 5909 Acked Write 8 Done Sent Done false
18181 5910 Acked Write 8 Done Sent Done false
18182 5911 Acked Read 3 Done Sent Done false
18183 5912 Acked Flush 0 Done Sent Done false
18184 5913 Acked Flush 0 Done Sent Done false
18185 5914 Acked Read 3 Done Sent Done false
18186 5915 Acked Read 10 Done Sent Done false
18187 5916 Acked Read 9 Done Sent Done false
18188 5917 Acked Write 1 Done Sent Done false
18189 5918 Acked Read 9 Done Sent Done false
18190 5919 Acked Read 5 Done Sent Done false
18191 5920 Acked Read 10 Done Sent Done false
18192 5921 Acked Read 9 Done Sent Done false
18193 5922 Acked Write 7 Done Sent Done false
18194 5923 Acked Read 10 Done Sent Done false
18195 5924 Acked Flush 0 Done Sent Done false
18196 5925 Acked Read 9 Done Sent Done false
18197 5926 Acked Read 8 Done Sent Done false
18198 5927 Acked Read 6 Done Sent Done false
18199 5928 Acked Write 8 Done Sent Done false
18200 5929 Acked Read 4 Done Sent Done false
18201 5930 Acked Write 3 Done Sent Done false
18202 5931 Acked Write 10 Done Sent Done false
18203 5932 Acked Read 10 Done Sent Done false
18204 5933 Acked Read 1 Done Sent Done false
18205 5934 Acked Write 10 Done Sent Done false
18206 5935 Acked Flush 0 Done Sent Done false
18207 5936 Acked Write 10 Done Sent Done false
18208 5937 Acked Read 7 Done Sent Done false
18209 5938 Acked Read 10 Done Sent Done false
18210 5939 Acked Flush 0 Done Sent Done false
18211 5940 Acked Read 7 Done Sent Done false
18212 5941 Acked Read 7 Done Sent Done false
18213 5942 Acked Read 5 Done Sent Done false
18214 5943 Acked Read 9 Done Sent Done false
18215 5944 Acked Write 5 Done Sent Done false
18216 5945 Acked Write 4 Done Sent Done false
18217 5946 Acked Read 4 Done Sent Done false
18218 5947 Acked Read 3 Done Sent Done false
18219 5948 Acked Read 8 Done Sent Done false
18220 5949 Acked Write 8 Done Sent Done false
18221 5950 Acked Read 8 Done Sent Done false
18222 5951 Acked Write 10 Done Sent Done false
18223 5952 Acked Write 10 Done Sent Done false
18224 5953 Acked Flush 0 Done Sent Done false
18225 5954 Acked Read 2 Done Sent Done false
18226 5955 Acked Write 3 Done Sent Done false
18227 5956 Acked Write 3 Done Sent Done false
18228 5957 Acked Flush 0 Done Sent Done false
18229 5958 Acked Read 1 Done Sent Done false
18230 5959 Acked Read 1 Done Sent Done false
18231 5960 Acked Write 10 Done Sent Done false
18232 5961 Acked Read 7 Done Sent Done false
18233 5962 Acked Read 9 Done Sent Done false
18234 5963 Acked Write 5 Done Sent Done false
18235 5964 Acked Flush 0 Done Sent Done false
18236 5965 Acked Read 1 Done Sent Done false
18237 5966 Acked Read 9 Done Sent Done false
18238 5967 Acked Write 3 Done Sent Done false
18239 5968 Acked Write 8 Done Sent Done false
18240 5969 Acked Flush 0 Done Sent Done false
18241 5970 Acked Write 8 Done Sent Done false
18242 5971 Acked Write 1 Done Sent Done false
18243 5972 Acked Read 5 Done Sent Done false
18244 5973 Acked Read 9 Done Sent Done false
18245 5974 Acked Write 7 Done Sent Done false
18246 5975 Acked Write 6 Done Sent Done false
18247 5976 Acked Read 10 Done Sent Done false
18248 5977 Acked Write 9 Done Sent Done false
18249 5978 Acked Write 9 Done Sent Done false
18250 5979 Acked Write 4 Done Sent Done false
18251 5980 Acked Write 6 Done Sent Done false
18252 5981 Acked Write 10 Done Sent Done false
18253 5982 Acked Write 7 Done Sent Done false
18254 5983 Acked Read 2 Done Sent Done false
18255 5984 Acked Write 8 Done Sent Done false
18256 5985 Acked Write 10 Done Sent Done false
18257 5986 Acked Write 6 Done Sent Done false
18258 5987 Acked Flush 0 Done Sent Done false
18259 5988 Acked Write 10 Done Sent Done false
18260 5989 Acked Flush 0 Done Sent Done false
18261 5990 Acked Read 7 Done Sent Done false
18262 5991 Acked Flush 0 Done Sent Done false
18263 5992 Acked Write 5 Done Sent Done false
18264 5993 Acked Read 5 Done Sent Done false
18265 5994 Acked Write 7 Done Sent Done false
18266 5995 Acked Read 2 Done Sent Done false
18267 5996 Acked Read 5 Done Sent Done false
18268 5997 Acked Flush 0 Done Sent Done false
18269 5998 Acked Write 4 Done Sent Done false
18270 5999 Acked Read 9 Done Sent Done false
18271 6000 Acked Write 1 Done Sent Done false
18272 6001 Acked Flush 0 Done Sent Done false
18273 6002 Acked Write 10 Done Sent Done false
18274 6003 Acked Read 2 Done Sent Done false
18275 6004 Acked Read 8 Done Sent Done false
18276 6005 Acked Write 8 Done Sent Done false
18277 6006 Acked Read 10 Done Sent Done false
18278 6007 Acked Read 10 Done Sent Done false
18279 6008 Acked Flush 0 Done Sent Done false
18280 6009 Acked Write 9 Done Sent Done false
18281 6010 Acked Read 8 Done Sent Done false
18282 6011 Acked Read 9 Done Sent Done false
18283 6012 Acked Flush 0 Done Sent Done false
18284 6013 Acked Write 7 Done Sent Done false
18285 6014 Acked Write 4 Done Sent Done false
18286 6015 Acked Flush 0 Done Sent Done false
18287 6016 Acked Read 2 Done Sent Done false
18288 6017 Acked Read 9 Done Sent Done false
18289 6018 Acked Read 7 Done Sent Done false
18290 6019 Acked Read 2 Done Sent Done false
18291 6020 Acked Read 9 Done Sent Done false
18292 6021 Acked Write 9 Done Sent Done false
18293 6022 Acked Write 9 Done Sent Done false
18294 6023 Acked Read 6 Done Sent Done false
18295 6024 Acked Write 9 Done Sent Done false
18296 6025 Acked Read 10 Done Sent Done false
18297 6026 Acked Write 9 Done Sent Done false
18298 6027 Acked Read 9 Done Sent Done false
18299 6028 Acked Flush 0 Done Sent Done false
18300 6029 Acked Read 2 Done Sent Done false
18301 6030 Acked Read 1 Done Sent Done false
18302 6031 Acked Read 9 Done Sent Done false
18303 6032 Acked Write 2 Done Sent Done false
18304 6033 Acked Write 9 Done Sent Done false
18305 6034 Acked Write 9 Done Sent Done false
18306 6035 Acked Read 10 Done Sent Done false
18307 6036 Acked Read 3 Done Sent Done false
18308 6037 Acked Write 8 Done Sent Done false
18309 6038 Acked Write 1 Done Sent Done false
18310 6039 Acked Read 10 Done Sent Done false
18311 6040 Acked Write 6 Done Sent Done false
18312 6041 Acked Flush 0 Done Sent Done false
18313 6042 Acked Read 9 Done Sent Done false
18314 6043 Acked Flush 0 Done Sent Done false
18315 6044 Acked Flush 0 Done Sent Done false
18316 6045 Acked Write 9 Done Sent Done false
18317 6046 Acked Write 1 Done Sent Done false
18318 6047 Acked Read 7 Done Sent Done false
18319 6048 Acked Read 3 Done Sent Done false
18320 6049 Acked Write 1 Done Sent Done false
18321 6050 Acked Read 7 Done Sent Done false
18322 6051 Acked Read 2 Done Sent Done false
18323 6052 Acked Write 1 Done Sent Done false
18324 6053 Acked Write 5 Done Sent Done false
18325 6054 Acked Write 8 Done Sent Done false
18326 6055 Acked Read 4 Done Sent Done false
18327 6056 Acked Read 3 Done Sent Done false
18328 6057 Acked Flush 0 Done Sent Done false
18329 6058 Acked Read 9 Done Sent Done false
18330 6059 Acked Write 6 Done Sent Done false
18331 6060 Acked Read 4 Done Sent Done false
18332 6061 Acked Write 3 Done Sent Done false
18333 6062 Acked Read 9 Done Sent Done false
18334 6063 Acked Write 10 Done Sent Done false
18335 6064 Acked Flush 0 Done Sent Done false
18336 6065 Acked Flush 0 Done Sent Done false
18337 6066 Acked Write 4 Done Sent Done false
18338 6067 Acked Read 10 Done Sent Done false
18339 6068 Acked Read 8 Done Sent Done false
18340 6069 Acked Flush 0 Done Sent Done false
18341 6070 Acked Read 8 Done Sent Done false
18342 6071 Acked Write 4 Done Sent Done false
18343 6072 Acked Flush 0 Done Sent Done false
18344 6073 Acked Write 5 Done Sent Done false
18345 6074 Acked Read 7 Done Sent Done false
18346 6075 Acked Write 9 Done Sent Done false
18347 6076 Acked Read 3 Done Sent Done false
18348 6077 Acked Read 4 Done Sent Done false
18349 6078 Acked Flush 0 Done Sent Done false
18350 6079 Acked Read 2 Done Sent Done false
18351 6080 Acked Read 9 Done Sent Done false
18352 6081 Acked Write 1 Done Sent Done false
18353 6082 Acked Write 5 Done Sent Done false
18354 6083 Acked Read 5 Done Sent Done false
18355 6084 Acked Write 5 Done Sent Done false
18356 6085 Acked Flush 0 Done Sent Done false
18357 6086 Acked Write 4 Done Sent Done false
18358 6087 Acked Read 5 Done Sent Done false
18359 6088 Acked Write 8 Done Sent Done false
18360 6089 Acked Write 2 Done Sent Done false
18361 6090 Acked Read 2 Done Sent Done false
18362 6091 Acked Read 10 Done Sent Done false
18363 6092 Acked Read 9 Done Sent Done false
18364 6093 Acked Read 4 Done Sent Done false
18365 6094 Acked Read 9 Done Sent Done false
18366 6095 Acked Read 5 Done Sent Done false
18367 6096 Acked Read 6 Done Sent Done false
18368 6097 Acked Write 3 Done Sent Done false
18369 6098 Acked Write 8 Done Sent Done false
18370 6099 Acked Read 10 Done Sent Done false
18371 6100 Acked Read 2 Done Sent Done false
18372 6101 Acked Write 2 Done Sent Done false
18373 6102 Acked Read 9 Done Sent Done false
18374 6103 Acked Read 1 Done Sent Done false
18375 6104 Acked Write 1 Done Sent Done false
18376 6105 Acked Write 2 Done Sent Done false
18377 6106 Acked Write 6 Done Sent Done false
18378 6107 Acked Read 10 Done Sent Done false
18379 6108 Acked Write 10 Done Sent Done false
18380 6109 Acked Read 6 Done Sent Done false
18381 6110 Acked Write 7 Done Sent Done false
18382 6111 Acked Write 2 Done Sent Done false
18383 6112 Acked Read 4 Done Sent Done false
18384 6113 Acked Read 6 Done Sent Done false
18385 6114 Acked Write 3 Done Sent Done false
18386 6115 Acked Write 2 Done Sent Done false
18387 6116 Acked Write 9 Done Sent Done false
18388 6117 Acked Read 1 Done Sent Done false
18389 6118 Acked Write 3 Done Sent Done false
18390 6119 Acked Read 4 Done Sent Done false
18391 6120 Acked Read 4 Done Sent Done false
18392 6121 Acked Write 10 Done Sent Done false
18393 6122 Acked Read 6 Done Sent Done false
18394 6123 Acked Read 5 Done Sent Done false
18395 6124 Acked Write 1 Done Sent Done false
18396 6125 Acked Flush 0 Done Sent Done false
18397 6126 Acked Read 6 Done Sent Done false
18398 6127 Acked Write 3 Done Sent Done false
18399 6128 Acked Write 8 Done Sent Done false
18400 6129 Acked Write 5 Done Sent Done false
18401 6130 Acked Write 10 Done Sent Done false
18402 6131 Acked Read 10 Done Sent Done false
18403 STATES DS:0 DS:1 DS:2 TOTAL
18404 Sent 0 301 0 301
18405 Done 301 0 301 602
18406 Skipped 0 0 0 0
18407 Error 0 0 0 0
18408 Last Flush: Some(JobId(6125)) Some(JobId(5830)) Some(JobId(6125))
18409 Downstairs last five completed: 5830 5829 5828 5827 5826
18410 Upstairs last five completed: 6131 6130 6129 6128 6127
18411 CLIENT: Up:0 ds:301 act:2
184122025-01-28T21:58:15.683ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
184132025-01-28T21:58:16.685ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
184142025-01-28T21:58:17.687ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
18415 ----------------------------------------------------------------
18416 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18417 States: Active Offline Active
18418 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18419 5831 Acked Read 2 Done Sent Done false
18420 5832 Acked Write 1 Done Sent Done false
18421 5833 Acked Flush 0 Done Sent Done false
18422 5834 Acked Write 9 Done Sent Done false
18423 5835 Acked Write 1 Done Sent Done false
18424 5836 Acked Flush 0 Done Sent Done false
18425 5837 Acked Write 7 Done Sent Done false
18426 5838 Acked Read 8 Done Sent Done false
18427 5839 Acked Write 8 Done Sent Done false
18428 5840 Acked Read 7 Done Sent Done false
18429 5841 Acked Flush 0 Done Sent Done false
18430 5842 Acked Write 4 Done Sent Done false
18431 5843 Acked Write 4 Done Sent Done false
18432 5844 Acked Write 6 Done Sent Done false
18433 5845 Acked Write 1 Done Sent Done false
18434 5846 Acked Read 3 Done Sent Done false
18435 5847 Acked Read 7 Done Sent Done false
18436 5848 Acked Read 4 Done Sent Done false
18437 5849 Acked Write 10 Done Sent Done false
18438 5850 Acked Write 4 Done Sent Done false
18439 5851 Acked Flush 0 Done Sent Done false
18440 5852 Acked Write 2 Done Sent Done false
18441 5853 Acked Read 1 Done Sent Done false
18442 5854 Acked Write 10 Done Sent Done false
18443 5855 Acked Flush 0 Done Sent Done false
18444 5856 Acked Write 9 Done Sent Done false
18445 5857 Acked Write 1 Done Sent Done false
18446 5858 Acked Write 3 Done Sent Done false
18447 5859 Acked Write 3 Done Sent Done false
18448 5860 Acked Read 2 Done Sent Done false
18449 5861 Acked Read 4 Done Sent Done false
18450 5862 Acked Write 2 Done Sent Done false
18451 5863 Acked Flush 0 Done Sent Done false
18452 5864 Acked Write 8 Done Sent Done false
18453 5865 Acked Write 7 Done Sent Done false
18454 5866 Acked Write 10 Done Sent Done false
18455 5867 Acked Read 2 Done Sent Done false
18456 5868 Acked Read 10 Done Sent Done false
18457 5869 Acked Write 10 Done Sent Done false
18458 5870 Acked Read 3 Done Sent Done false
18459 5871 Acked Write 10 Done Sent Done false
18460 5872 Acked Write 6 Done Sent Done false
18461 5873 Acked Write 4 Done Sent Done false
18462 5874 Acked Read 6 Done Sent Done false
18463 5875 Acked Read 5 Done Sent Done false
18464 5876 Acked Write 1 Done Sent Done false
18465 5877 Acked Read 9 Done Sent Done false
18466 5878 Acked Write 8 Done Sent Done false
18467 5879 Acked Read 4 Done Sent Done false
18468 5880 Acked Write 4 Done Sent Done false
18469 5881 Acked Read 3 Done Sent Done false
18470 5882 Acked Write 8 Done Sent Done false
18471 5883 Acked Flush 0 Done Sent Done false
18472 5884 Acked Read 5 Done Sent Done false
18473 5885 Acked Read 3 Done Sent Done false
18474 5886 Acked Write 6 Done Sent Done false
18475 5887 Acked Write 7 Done Sent Done false
18476 5888 Acked Flush 0 Done Sent Done false
18477 5889 Acked Write 5 Done Sent Done false
18478 5890 Acked Write 5 Done Sent Done false
18479 5891 Acked Write 10 Done Sent Done false
18480 5892 Acked Write 7 Done Sent Done false
18481 5893 Acked Write 1 Done Sent Done false
18482 5894 Acked Read 9 Done Sent Done false
18483 5895 Acked Read 3 Done Sent Done false
18484 5896 Acked Write 7 Done Sent Done false
18485 5897 Acked Read 2 Done Sent Done false
18486 5898 Acked Read 1 Done Sent Done false
18487 5899 Acked Read 3 Done Sent Done false
18488 5900 Acked Write 6 Done Sent Done false
18489 5901 Acked Read 9 Done Sent Done false
18490 5902 Acked Read 9 Done Sent Done false
18491 5903 Acked Flush 0 Done Sent Done false
18492 5904 Acked Flush 0 Done Sent Done false
18493 5905 Acked Flush 0 Done Sent Done false
18494 5906 Acked Write 1 Done Sent Done false
18495 5907 Acked Read 2 Done Sent Done false
18496 5908 Acked Write 3 Done Sent Done false
18497 5909 Acked Write 8 Done Sent Done false
18498 5910 Acked Write 8 Done Sent Done false
18499 5911 Acked Read 3 Done Sent Done false
18500 5912 Acked Flush 0 Done Sent Done false
18501 5913 Acked Flush 0 Done Sent Done false
18502 5914 Acked Read 3 Done Sent Done false
18503 5915 Acked Read 10 Done Sent Done false
18504 5916 Acked Read 9 Done Sent Done false
18505 5917 Acked Write 1 Done Sent Done false
18506 5918 Acked Read 9 Done Sent Done false
18507 5919 Acked Read 5 Done Sent Done false
18508 5920 Acked Read 10 Done Sent Done false
18509 5921 Acked Read 9 Done Sent Done false
18510 5922 Acked Write 7 Done Sent Done false
18511 5923 Acked Read 10 Done Sent Done false
18512 5924 Acked Flush 0 Done Sent Done false
18513 5925 Acked Read 9 Done Sent Done false
18514 5926 Acked Read 8 Done Sent Done false
18515 5927 Acked Read 6 Done Sent Done false
18516 5928 Acked Write 8 Done Sent Done false
18517 5929 Acked Read 4 Done Sent Done false
18518 5930 Acked Write 3 Done Sent Done false
18519 5931 Acked Write 10 Done Sent Done false
18520 5932 Acked Read 10 Done Sent Done false
18521 5933 Acked Read 1 Done Sent Done false
18522 5934 Acked Write 10 Done Sent Done false
18523 5935 Acked Flush 0 Done Sent Done false
18524 5936 Acked Write 10 Done Sent Done false
18525 5937 Acked Read 7 Done Sent Done false
18526 5938 Acked Read 10 Done Sent Done false
18527 5939 Acked Flush 0 Done Sent Done false
18528 5940 Acked Read 7 Done Sent Done false
18529 5941 Acked Read 7 Done Sent Done false
18530 5942 Acked Read 5 Done Sent Done false
18531 5943 Acked Read 9 Done Sent Done false
18532 5944 Acked Write 5 Done Sent Done false
18533 5945 Acked Write 4 Done Sent Done false
18534 5946 Acked Read 4 Done Sent Done false
18535 5947 Acked Read 3 Done Sent Done false
18536 5948 Acked Read 8 Done Sent Done false
18537 5949 Acked Write 8 Done Sent Done false
18538 5950 Acked Read 8 Done Sent Done false
18539 5951 Acked Write 10 Done Sent Done false
18540 5952 Acked Write 10 Done Sent Done false
18541 5953 Acked Flush 0 Done Sent Done false
18542 5954 Acked Read 2 Done Sent Done false
18543 5955 Acked Write 3 Done Sent Done false
18544 5956 Acked Write 3 Done Sent Done false
18545 5957 Acked Flush 0 Done Sent Done false
18546 5958 Acked Read 1 Done Sent Done false
18547 5959 Acked Read 1 Done Sent Done false
18548 5960 Acked Write 10 Done Sent Done false
18549 5961 Acked Read 7 Done Sent Done false
18550 5962 Acked Read 9 Done Sent Done false
18551 5963 Acked Write 5 Done Sent Done false
18552 5964 Acked Flush 0 Done Sent Done false
18553 5965 Acked Read 1 Done Sent Done false
18554 5966 Acked Read 9 Done Sent Done false
18555 5967 Acked Write 3 Done Sent Done false
18556 5968 Acked Write 8 Done Sent Done false
18557 5969 Acked Flush 0 Done Sent Done false
18558 5970 Acked Write 8 Done Sent Done false
18559 5971 Acked Write 1 Done Sent Done false
18560 5972 Acked Read 5 Done Sent Done false
18561 5973 Acked Read 9 Done Sent Done false
18562 5974 Acked Write 7 Done Sent Done false
18563 5975 Acked Write 6 Done Sent Done false
18564 5976 Acked Read 10 Done Sent Done false
18565 5977 Acked Write 9 Done Sent Done false
18566 5978 Acked Write 9 Done Sent Done false
18567 5979 Acked Write 4 Done Sent Done false
18568 5980 Acked Write 6 Done Sent Done false
18569 5981 Acked Write 10 Done Sent Done false
18570 5982 Acked Write 7 Done Sent Done false
18571 5983 Acked Read 2 Done Sent Done false
18572 5984 Acked Write 8 Done Sent Done false
18573 5985 Acked Write 10 Done Sent Done false
18574 5986 Acked Write 6 Done Sent Done false
18575 5987 Acked Flush 0 Done Sent Done false
18576 5988 Acked Write 10 Done Sent Done false
18577 5989 Acked Flush 0 Done Sent Done false
18578 5990 Acked Read 7 Done Sent Done false
18579 5991 Acked Flush 0 Done Sent Done false
18580 5992 Acked Write 5 Done Sent Done false
18581 5993 Acked Read 5 Done Sent Done false
18582 5994 Acked Write 7 Done Sent Done false
18583 5995 Acked Read 2 Done Sent Done false
18584 5996 Acked Read 5 Done Sent Done false
18585 5997 Acked Flush 0 Done Sent Done false
18586 5998 Acked Write 4 Done Sent Done false
18587 5999 Acked Read 9 Done Sent Done false
18588 6000 Acked Write 1 Done Sent Done false
18589 6001 Acked Flush 0 Done Sent Done false
18590 6002 Acked Write 10 Done Sent Done false
18591 6003 Acked Read 2 Done Sent Done false
18592 6004 Acked Read 8 Done Sent Done false
18593 6005 Acked Write 8 Done Sent Done false
18594 6006 Acked Read 10 Done Sent Done false
18595 6007 Acked Read 10 Done Sent Done false
18596 6008 Acked Flush 0 Done Sent Done false
18597 6009 Acked Write 9 Done Sent Done false
18598 6010 Acked Read 8 Done Sent Done false
18599 6011 Acked Read 9 Done Sent Done false
18600 6012 Acked Flush 0 Done Sent Done false
18601 6013 Acked Write 7 Done Sent Done false
18602 6014 Acked Write 4 Done Sent Done false
18603 6015 Acked Flush 0 Done Sent Done false
18604 6016 Acked Read 2 Done Sent Done false
18605 6017 Acked Read 9 Done Sent Done false
18606 6018 Acked Read 7 Done Sent Done false
18607 6019 Acked Read 2 Done Sent Done false
18608 6020 Acked Read 9 Done Sent Done false
18609 6021 Acked Write 9 Done Sent Done false
18610 6022 Acked Write 9 Done Sent Done false
18611 6023 Acked Read 6 Done Sent Done false
18612 6024 Acked Write 9 Done Sent Done false
18613 6025 Acked Read 10 Done Sent Done false
18614 6026 Acked Write 9 Done Sent Done false
18615 6027 Acked Read 9 Done Sent Done false
18616 6028 Acked Flush 0 Done Sent Done false
18617 6029 Acked Read 2 Done Sent Done false
18618 6030 Acked Read 1 Done Sent Done false
18619 6031 Acked Read 9 Done Sent Done false
18620 6032 Acked Write 2 Done Sent Done false
18621 6033 Acked Write 9 Done Sent Done false
18622 6034 Acked Write 9 Done Sent Done false
18623 6035 Acked Read 10 Done Sent Done false
18624 6036 Acked Read 3 Done Sent Done false
18625 6037 Acked Write 8 Done Sent Done false
18626 6038 Acked Write 1 Done Sent Done false
18627 6039 Acked Read 10 Done Sent Done false
18628 6040 Acked Write 6 Done Sent Done false
18629 6041 Acked Flush 0 Done Sent Done false
18630 6042 Acked Read 9 Done Sent Done false
18631 6043 Acked Flush 0 Done Sent Done false
18632 6044 Acked Flush 0 Done Sent Done false
18633 6045 Acked Write 9 Done Sent Done false
18634 6046 Acked Write 1 Done Sent Done false
18635 6047 Acked Read 7 Done Sent Done false
18636 6048 Acked Read 3 Done Sent Done false
18637 6049 Acked Write 1 Done Sent Done false
18638 6050 Acked Read 7 Done Sent Done false
18639 6051 Acked Read 2 Done Sent Done false
18640 6052 Acked Write 1 Done Sent Done false
18641 6053 Acked Write 5 Done Sent Done false
18642 6054 Acked Write 8 Done Sent Done false
18643 6055 Acked Read 4 Done Sent Done false
18644 6056 Acked Read 3 Done Sent Done false
18645 6057 Acked Flush 0 Done Sent Done false
18646 6058 Acked Read 9 Done Sent Done false
18647 6059 Acked Write 6 Done Sent Done false
18648 6060 Acked Read 4 Done Sent Done false
18649 6061 Acked Write 3 Done Sent Done false
18650 6062 Acked Read 9 Done Sent Done false
18651 6063 Acked Write 10 Done Sent Done false
18652 6064 Acked Flush 0 Done Sent Done false
18653 6065 Acked Flush 0 Done Sent Done false
18654 6066 Acked Write 4 Done Sent Done false
18655 6067 Acked Read 10 Done Sent Done false
18656 6068 Acked Read 8 Done Sent Done false
18657 6069 Acked Flush 0 Done Sent Done false
18658 6070 Acked Read 8 Done Sent Done false
18659 6071 Acked Write 4 Done Sent Done false
18660 6072 Acked Flush 0 Done Sent Done false
18661 6073 Acked Write 5 Done Sent Done false
18662 6074 Acked Read 7 Done Sent Done false
18663 6075 Acked Write 9 Done Sent Done false
18664 6076 Acked Read 3 Done Sent Done false
18665 6077 Acked Read 4 Done Sent Done false
18666 6078 Acked Flush 0 Done Sent Done false
18667 6079 Acked Read 2 Done Sent Done false
18668 6080 Acked Read 9 Done Sent Done false
18669 6081 Acked Write 1 Done Sent Done false
18670 6082 Acked Write 5 Done Sent Done false
18671 6083 Acked Read 5 Done Sent Done false
18672 6084 Acked Write 5 Done Sent Done false
18673 6085 Acked Flush 0 Done Sent Done false
18674 6086 Acked Write 4 Done Sent Done false
18675 6087 Acked Read 5 Done Sent Done false
18676 6088 Acked Write 8 Done Sent Done false
18677 6089 Acked Write 2 Done Sent Done false
18678 6090 Acked Read 2 Done Sent Done false
18679 6091 Acked Read 10 Done Sent Done false
18680 6092 Acked Read 9 Done Sent Done false
18681 6093 Acked Read 4 Done Sent Done false
18682 6094 Acked Read 9 Done Sent Done false
18683 6095 Acked Read 5 Done Sent Done false
18684 6096 Acked Read 6 Done Sent Done false
18685 6097 Acked Write 3 Done Sent Done false
18686 6098 Acked Write 8 Done Sent Done false
18687 6099 Acked Read 10 Done Sent Done false
18688 6100 Acked Read 2 Done Sent Done false
18689 6101 Acked Write 2 Done Sent Done false
18690 6102 Acked Read 9 Done Sent Done false
18691 6103 Acked Read 1 Done Sent Done false
18692 6104 Acked Write 1 Done Sent Done false
18693 6105 Acked Write 2 Done Sent Done false
18694 6106 Acked Write 6 Done Sent Done false
18695 6107 Acked Read 10 Done Sent Done false
18696 6108 Acked Write 10 Done Sent Done false
18697 6109 Acked Read 6 Done Sent Done false
18698 6110 Acked Write 7 Done Sent Done false
18699 6111 Acked Write 2 Done Sent Done false
18700 6112 Acked Read 4 Done Sent Done false
18701 6113 Acked Read 6 Done Sent Done false
18702 6114 Acked Write 3 Done Sent Done false
18703 6115 Acked Write 2 Done Sent Done false
18704 6116 Acked Write 9 Done Sent Done false
18705 6117 Acked Read 1 Done Sent Done false
18706 6118 Acked Write 3 Done Sent Done false
18707 6119 Acked Read 4 Done Sent Done false
18708 6120 Acked Read 4 Done Sent Done false
18709 6121 Acked Write 10 Done Sent Done false
18710 6122 Acked Read 6 Done Sent Done false
18711 6123 Acked Read 5 Done Sent Done false
18712 6124 Acked Write 1 Done Sent Done false
18713 6125 Acked Flush 0 Done Sent Done false
18714 6126 Acked Read 6 Done Sent Done false
18715 6127 Acked Write 3 Done Sent Done false
18716 6128 Acked Write 8 Done Sent Done false
18717 6129 Acked Write 5 Done Sent Done false
18718 6130 Acked Write 10 Done Sent Done false
18719 6131 Acked Read 10 Done Sent Done false
18720 STATES DS:0 DS:1 DS:2 TOTAL
18721 Sent 0 301 0 301
18722 Done 301 0 301 602
18723 Skipped 0 0 0 0
18724 Error 0 0 0 0
18725 Last Flush: Some(JobId(6125)) Some(JobId(5830)) Some(JobId(6125))
18726 Downstairs last five completed: 5830 5829 5828 5827 5826
18727 Upstairs last five completed: 6131 6130 6129 6128 6127
18728 CLIENT: Up:0 ds:301 act:2
187292025-01-28T21:58:18.688ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187302025-01-28T21:58:20.690ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187312025-01-28T21:58:21.691ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187322025-01-28T21:58:21.849ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187332025-01-28T21:58:21.849ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60809) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187342025-01-28T21:58:21.850ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187352025-01-28T21:58:21.850ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187362025-01-28T21:58:21.850ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187372025-01-28T21:58:21.850ZINFOcrucible: send last flush ID to this DS: Some(JobId(5830)) = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187382025-01-28T21:58:21.850ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187392025-01-28T21:58:21.850ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5830)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187402025-01-28T21:58:21.851ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5830)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
18741 ----------------------------------------------------------------
18742 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:6
18743 States: Active Active Active
18744 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18745 6126 Acked Read 6 Done Done Done true
18746 6127 Acked Write 3 Done Done Done true
18747 6128 Acked Write 8 Done Done Done true
18748 6129 Acked Write 5 Done Done Done true
18749 6130 Acked Write 10 Done Done Done true
18750 6131 Acked Read 10 Done Done Done true
18751 STATES DS:0 DS:1 DS:2 TOTAL
18752 Sent 0 0 0 0
18753 Done 6 6 6 18
18754 Skipped 0 0 0 0
18755 Error 0 0 0 0
18756 Last Flush: Some(JobId(6125)) Some(JobId(6125)) Some(JobId(6125))
18757 Downstairs last five completed: 6125 6124 6123 6122 6121
18758 Upstairs last five completed: 6131 6130 6129 6128 6127
18759 CLIENT: Up:0 ds:6 act:3
18760 ----------------------------------------------------------------
18761 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18762 Downstairs last five completed: 6132 6131 6130 6129 6128
18763 Upstairs last five completed: 6132 6131 6130 6129 6128
18764 CLIENT: Up:0 ds:0 act:3
18765 Replay: All jobs finished
18766 Main task has work to do, go find it
18767 got dsc Stop(2)
18768 stop 2
18769 Jan 28 21:58:26.100 [8830] Got stop action so:false kr:false
18770 INFO request completed, latency_us: 372, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 1a5a8164-946b-4609-aafa-3cc371c0f8ca, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
18771 Jan 28 21:58:26.100 INFO request completed, latency_us: 115, response_code: 200, uri: /state/cid/2, method: GET, req_id: 878badcf-853a-42f6-b300-9a35a5a36dfe, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
187722025-01-28T21:58:26.100ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187732025-01-28T21:58:26.100ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187742025-01-28T21:58:26.100ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
187752025-01-28T21:58:26.100ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
18776 [8830] Exited with: ExitStatus(unix_wait_status(9))
18777 [8830][2] reports Exit
18778 Jan 28 21:58:30.102 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/2, method: GET, req_id: ba092c9d-6b20-4d90-87c4-ea635851dbec, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
18779 001/300 Read block 1515 len 32768
18780 002/300 Write block 2243 len 36864 data: 2 4 5 5 4 5 4 4 4
18781 003/300 Write block 879 len 36864 data: 4 5 5 5 5 4 6 7 6
18782 004/300 Write block 2383 len 4096 data: 7
18783 005/300 Flush
18784 006/300 Read block 946 len 24576
18785 007/300 Write block 3330 len 4096 data: 6
18786 008/300 Read block 2436 len 8192
18787 009/300 Read block 2833 len 40960
18788 010/300 Read block 3513 len 16384
18789 011/300 Read block 3844 len 36864
18790 012/300 Write block 1431 len 36864 data: 4 4 5 3 3 3 3 3 2
18791 013/300 Read block 4911 len 24576
18792 014/300 Write block 3714 len 36864 data: 5 4 3 3 4 5 5 5 5
18793 015/300 Write block 3449 len 12288 data: 4 4 4
18794 016/300 Write block 1808 len 28672 data: 5 5 5 4 4 2 2
18795 017/300 Write block 1538 len 8192 data: 3 2
18796 018/300 Read block 3359 len 20480
18797 019/300 Write block 4510 len 24576 data: 4 4 3 3 3 3
18798 020/300 Flush
18799 021/300 Read block 3615 len 4096
18800 022/300 Read block 4992 len 8192
18801 023/300 Read block 1164 len 28672
18802 024/300 Write block 1870 len 16384 data: 4 4 4 3
18803 025/300 Read block 4400 len 8192
18804 026/300 Read block 2946 len 16384
18805 027/300 Read block 2301 len 24576
18806 028/300 Write block 1745 len 12288 data: 3 4 5
18807 029/300 Write block 2477 len 4096 data: 7
18808 030/300 Write block 4076 len 8192 data: 5 4
18809 031/300 Write block 4069 len 24576 data: 3 3 2 3 4 6
18810 032/300 Read block 3402 len 36864
18811 033/300 Read block 3558 len 32768
18812 034/300 Write block 3720 len 20480 data: 6 6 6 5 5
18813 035/300 Write block 637 len 4096 data: 7
18814 036/300 Write block 1772 len 24576 data: 5 5 5 7 5 5
18815 037/300 Read block 4947 len 32768
18816 038/300 Read block 856 len 16384
18817 039/300 Read block 14 len 8192
18818 040/300 Write block 1014 len 36864 data: 7 6 7 5 5 4 4 6 6
18819 041/300 Flush
18820 042/300 Write block 4186 len 24576 data: 4 5 4 5 5 6
18821 043/300 Read block 4612 len 28672
18822 044/300 Read block 2804 len 12288
18823 045/300 Write block 620 len 4096 data: 4
18824 046/300 Flush
18825 047/300 Write block 3763 len 28672 data: 6 6 5 5 6 5 6
18826 048/300 Write block 4168 len 4096 data: 5
18827 049/300 Read block 4702 len 12288
18828 050/300 Read block 950 len 28672
18829 051/300 Read block 1352 len 20480
18830 052/300 Read block 1939 len 20480
18831 053/300 Write block 4895 len 20480 data: 4 3 4 3 3
18832 054/300 Write block 4317 len 32768 data: 4 4 5 4 4 4 6 6
18833 055/300 Write block 4579 len 8192 data: 6 6
18834 056/300 Write block 3359 len 16384 data: 4 4 5 5
18835 057/300 Read block 3710 len 28672
18836 058/300 Read block 1020 len 36864
18837 059/300 Read block 4267 len 40960
18838 060/300 Flush
18839 061/300 Flush
18840 062/300 Read block 4442 len 12288
18841 063/300 Read block 1231 len 24576
18842 064/300 Write block 4338 len 20480 data: 6 6 4 4 4
18843 065/300 Read block 4948 len 8192
18844 066/300 Write block 2467 len 32768 data: 5 4 3 4 4 4 5 6
18845 067/300 Write block 728 len 4096 data: 4
18846 068/300 Write block 3142 len 20480 data: 4 3 3 3 3
18847 069/300 Write block 4753 len 12288 data: 6 6 5
18848 070/300 Write block 4658 len 4096 data: 6
18849 071/300 Write block 3628 len 24576 data: 3 3 3 3 3 5
18850 072/300 Read block 3882 len 8192
18851 073/300 Read block 3092 len 28672
18852 074/300 Write block 4464 len 4096 data: 5
18853 075/300 Read block 1555 len 40960
18854 076/300 Write block 1286 len 40960 data: 3 4 3 4 5 5 4 4 4 5
18855 077/300 Write block 3445 len 28672 data: 4 3 5 4 5 5 5
18856 078/300 Read block 163 len 12288
18857 079/300 Read block 4093 len 32768
18858 080/300 Write block 4525 len 24576 data: 7 7 6 3 3 3
18859 081/300 Read block 2673 len 28672
18860 082/300 Write block 3903 len 8192 data: 3 2
18861 083/300 Flush
18862 084/300 Read block 3714 len 4096
18863 085/300 Write block 1481 len 8192 data: 4 5
18864 086/300 Read block 31 len 32768
18865 087/300 Read block 628 len 8192
18866 088/300 Write block 551 len 12288 data: 5 5 5
18867 089/300 Read block 1689 len 36864
18868 090/300 Read block 1642 len 36864
18869 091/300 Read block 4075 len 8192
18870 092/300 Read block 1184 len 32768
18871 093/300 Write block 706 len 8192 data: 3 4
18872 094/300 Read block 4751 len 24576
18873 095/300 Read block 3488 len 24576
18874 096/300 Write block 4824 len 24576 data: 5 4 3 3 3 3
18875 097/300 Flush
18876 098/300 Read block 974 len 28672
18877 099/300 Read block 1952 len 36864
18878 100/300 Write block 317 len 4096 data: 3
18879 101/300 Write block 4088 len 40960 data: 3 5 6 7 7 6 6 6 6 6
18880 102/300 Read block 2164 len 4096
18881 103/300 Write block 4618 len 12288 data: 3 4 4
18882 104/300 Read block 1786 len 24576
18883 105/300 Write block 4540 len 12288 data: 2 2 3
18884 106/300 Read block 4541 len 36864
18885 107/300 Write block 1699 len 40960 data: 6 7 6 6 7 7 7 6 5 6
18886 108/300 Write block 1316 len 20480 data: 5 3 4 3 3
18887 109/300 Read block 3332 len 40960
18888 110/300 Write block 4134 len 36864 data: 7 7 6 6 5 5 5 5 4
18889 111/300 Write block 19 len 16384 data: 2 2 2 2
18890 112/300 Read block 2133 len 28672
18891 113/300 Read block 2671 len 24576
18892 114/300 Flush
18893 115/300 Flush
18894 116/300 Read block 3499 len 8192
18895 117/300 Write block 817 len 8192 data: 2 2
18896 118/300 Write block 981 len 4096 data: 6
18897 119/300 Read block 4196 len 20480
18898 120/300 Read block 3350 len 4096
18899 121/300 Write block 3943 len 36864 data: 5 5 4 4 4 5 5 4 3
18900 122/300 Read block 207 len 12288
18901 123/300 Flush
18902 124/300 Write block 4260 len 4096 data: 3
18903 125/300 Read block 3056 len 4096
18904 126/300 Read block 2644 len 32768
18905 127/300 Write block 760 len 4096 data: 6
18906 128/300 Write block 3970 len 40960 data: 3 3 4 4 4 4 4 5 5 6
18907 129/300 Write block 1555 len 24576 data: 3 4 5 4 3 5
18908 130/300 Write block 2273 len 24576 data: 3 3 4 5 4 4
18909 131/300 Read block 2655 len 32768
18910 132/300 Read block 2885 len 20480
18911 133/300 Write block 2739 len 4096 data: 3
18912 134/300 Flush
18913 135/300 Read block 4140 len 36864
18914 136/300 Read block 794 len 16384
18915 137/300 Read block 4303 len 40960
18916 138/300 Write block 3750 len 36864 data: 3 4 4 4 4 4 6 6 6
18917 139/300 Read block 4526 len 40960
18918 140/300 Write block 2598 len 28672 data: 3 3 3 3 2 3 3
18919 141/300 Flush
18920 142/300 Read block 123 len 36864
18921 143/300 Read block 4608 len 32768
18922 144/300 Write block 529 len 36864 data: 5 4 4 4 5 5 5 5 4
18923 145/300 Flush
18924 146/300 Write block 154 len 24576 data: 6 6 6 8 7 5
18925 147/300 Write block 2953 len 20480 data: 4 4 4 3 3
18926 148/300 Read block 2571 len 12288
18927 149/300 Flush
18928 150/300 Read block 1556 len 32768
18929 151/300 Write block 2318 len 16384 data: 4 4 4 3
18930 152/300 Write block 3660 len 20480 data: 5 5 5 6 6
18931 153/300 Write block 1014 len 20480 data: 8 7 8 6 6
18932 154/300 Write block 2656 len 40960 data: 2 2 4 6 8 8 8 7 6 4
18933 155/300 Read block 1577 len 8192
18934 156/300 Flush
18935 157/300 Write block 3280 len 8192 data: 2 2
18936 158/300 Read block 3588 len 12288
18937 159/300 Flush
18938 160/300 Write block 688 len 32768 data: 7 7 6 7 5 5 5 3
18939 161/300 Write block 2611 len 32768 data: 3 3 4 5 7 7 8 8
18940 162/300 Write block 4580 len 12288 data: 7 6 6
18941 163/300 Write block 3775 len 16384 data: 5 5 6 5
18942 164/300 Read block 4343 len 20480
18943 165/300 Write block 2093 len 28672 data: 3 3 3 3 3 4 6
18944 166/300 Read block 1674 len 28672
18945 167/300 Flush
18946 168/300 Read block 1588 len 8192
18947 169/300 Read block 12 len 28672
18948 170/300 Read block 1457 len 24576
18949 171/300 Flush
18950 172/300 Read block 4401 len 36864
18951 173/300 Write block 777 len 24576 data: 4 5 6 5 4 5
18952 174/300 Write block 2270 len 4096 data: 2
18953 175/300 Write block 3196 len 40960 data: 3 4 4 4 5 5 5 4 3 2
18954 176/300 Read block 1403 len 20480
18955 177/300 Flush
18956 178/300 Read block 2674 len 40960
18957 179/300 Read block 2623 len 40960
18958 180/300 Read block 4410 len 16384
18959 181/300 Read block 1447 len 16384
18960 182/300 Read block 628 len 8192
18961 183/300 Write block 2661 len 8192 data: 9 9
18962 184/300 Write block 726 len 32768 data: 3 4 5 6 6 6 7 7
18963 185/300 Read block 1801 len 20480
18964 186/300 Write block 3054 len 4096 data: 4
18965 187/300 Read block 3583 len 4096
18966 188/300 Write block 2284 len 32768 data: 4 5 5 6 6 6 6 6
18967 189/300 Write block 1959 len 24576 data: 5 5 4 4 4 4
18968 190/300 Write block 924 len 24576 data: 4 3 3 3 4 3
18969 191/300 Write block 4772 len 20480 data: 10 9 7 6 5
18970 192/300 Read block 1792 len 40960
18971 193/300 Read block 1881 len 28672
18972 194/300 Write block 4539 len 28672 data: 2 3 3 4 3 4 5
18973 195/300 Write block 1923 len 24576 data: 5 5 6 5 4 3
18974 196/300 Write block 3617 len 32768 data: 3 3 4 4 3 3 3 4
18975 197/300 Write block 3263 len 20480 data: 5 5 5 4 4
18976 198/300 Write block 993 len 20480 data: 3 3 3 2 3
18977 199/300 Read block 845 len 28672
18978 200/300 Read block 1555 len 20480
18979 201/300 Write block 3342 len 16384 data: 4 4 3 3
18980 202/300 Read block 1691 len 16384
18981 203/300 Write block 2955 len 24576 data: 5 4 4 3 3 3
18982 204/300 Write block 482 len 8192 data: 3 3
18983 205/300 Write block 2823 len 16384 data: 4 4 4 5
18984 206/300 Read block 4569 len 12288
18985 207/300 Write block 4807 len 28672 data: 6 3 3 3 2 3 3
18986 208/300 Write block 4615 len 16384 data: 5 4 4 4
18987 209/300 Write block 4121 len 32768 data: 3 2 2 3 3 3 4 3
18988 210/300 Read block 2076 len 28672
18989 211/300 Write block 1789 len 16384 data: 3 3 3 3
18990 212/300 Write block 4180 len 8192 data: 3 3
18991 213/300 Read block 1614 len 20480
18992 214/300 Write block 2708 len 36864 data: 4 4 6 5 3 2 2 2 2
18993 215/300 Read block 4562 len 20480
18994 216/300 Read block 3728 len 40960
18995 217/300 Write block 2575 len 4096 data: 3
18996 218/300 Write block 2235 len 20480 data: 3 3 3 3 3
18997 219/300 Write block 1592 len 12288 data: 2 2 2
18998 220/300 Read block 1986 len 36864
18999 221/300 Write block 1465 len 36864 data: 2 2 3 4 4 3 3 3 3
19000 222/300 Read block 749 len 24576
19001 223/300 Read block 2176 len 32768
19002 224/300 Read block 1076 len 12288
19003 225/300 Read block 1742 len 32768
19004 226/300 Read block 4431 len 40960
19005 227/300 Read block 2970 len 12288
19006 228/300 Read block 3186 len 32768
19007 229/300 Read block 1996 len 12288
19008 230/300 Read block 3974 len 12288
19009 231/300 Read block 1841 len 4096
19010 232/300 Read block 2173 len 4096
19011 233/300 Read block 4032 len 12288
19012 234/300 Read block 2405 len 16384
19013 235/300 Read block 4295 len 4096
19014 236/300 Write block 4650 len 32768 data: 2 2 2 3 3 3 3 2
19015 237/300 Flush
19016 238/300 Read block 1164 len 40960
19017 239/300 Read block 3930 len 16384
19018 240/300 Write block 1629 len 20480 data: 6 5 6 5 5
19019 241/300 Read block 318 len 12288
19020 242/300 Flush
19021 243/300 Write block 1664 len 12288 data: 9 8 6
19022 244/300 Read block 4542 len 16384
19023 245/300 Read block 4155 len 4096
19024 246/300 Write block 3411 len 20480 data: 4 4 2 3 2
19025 247/300 Read block 2659 len 40960
19026 248/300 Read block 4203 len 20480
19027 249/300 Read block 4474 len 12288
19028 250/300 Read block 1193 len 36864
19029 251/300 Read block 3419 len 40960
19030 252/300 Write block 1138 len 40960 data: 3 3 4 5 5 5 6 6 5 5
19031 253/300 Read block 232 len 20480
19032 254/300 Write block 1062 len 12288 data: 5 5 5
19033 255/300 Write block 4401 len 12288 data: 4 6 6
19034 256/300 Read block 3157 len 8192
19035 257/300 Flush
19036 258/300 Write block 2143 len 8192 data: 4 5
19037 259/300 Write block 1470 len 28672 data: 4 4 4 4 2 2 3
19038 260/300 Read block 1032 len 28672
19039 261/300 Read block 3634 len 24576
19040 262/300 Flush
19041 263/300 Read block 943 len 20480
19042 264/300 Write block 4659 len 28672 data: 7 7 8 8 9 9 9
19043 265/300 Read block 4873 len 28672
19044 266/300 Write block 4452 len 28672 data: 5 4 4 3 3 4 3
19045 267/300 Write block 263 len 12288 data: 5 4 5
19046 268/300 Flush
19047 269/300 Read block 2400 len 4096
19048 270/300 Flush
19049 271/300 Read block 804 len 8192
19050 272/300 Write block 3640 len 36864 data: 4 5 7 6 5 5 5 5 6
19051 273/300 Read block 2820 len 24576
19052 274/300 Read block 338 len 36864
19053 275/300 Read block 4353 len 32768
19054 276/300 Write block 2918 len 40960 data: 2 2 3 3 3 3 3 3 3 5
19055 277/300 Write block 2918 len 32768 data: 3 3 4 4 4 4 4 4
19056 278/300 Read block 4730 len 24576
19057 279/300 Read block 1763 len 4096
19058 280/300 Read block 1045 len 28672
19059 281/300 Read block 4392 len 32768
19060 282/300 Write block 3079 len 8192 data: 7 7
19061 283/300 Read block 1341 len 4096
19062 284/300 Write block 1465 len 16384 data: 3 3 4 5
19063 285/300 Read block 3325 len 12288
19064 286/300 Flush
19065 287/300 Write block 4600 len 12288 data: 5 5 5
19066 288/300 Read block 2500 len 4096
19067 289/300 Read block 1703 len 4096
19068 290/300 Read block 621 len 20480
19069 291/300 Read block 2134 len 20480
19070 292/300 Read block 3277 len 8192
19071 293/300 Read block 617 len 12288
19072 294/300 Write block 4422 len 20480 data: 4 4 4 4 8
19073 295/300 Write block 2086 len 24576 data: 4 4 4 2 3 3
19074 296/300 Read block 1279 len 24576
19075 297/300 Read block 392 len 40960
19076 298/300 Read block 3118 len 36864
19077 299/300 Write block 2624 len 24576 data: 3 3 3 2 2 4
19078 300/300 Read block 273 len 16384
19079 301/300 Write block 2367 len 32768 data: 3 5 5 6 6 5 5 6
19080 Main task has work to do, go find it
19081 got dsc Start(2)
19082 start 2
19083 Jan 28 21:58:30.338 INFO request completed, latency_us: 301, response_code: 204, uri: /start/cid/2, method: POST, req_id: 9e4fc6f2-feab-4657-9a55-84312c274cc3, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
19084 Replay: started 2, returned:Ok(())
19085 ----------------------------------------------------------------
19086 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19087 States: Active Active Offline
19088 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19089 6133 Acked Read 8 Done Done Sent false
19090 6134 Acked Write 9 Done Done Sent false
19091 6135 Acked Write 9 Done Done Sent false
19092 6136 Acked Write 1 Done Done Sent false
19093 6137 Acked Flush 0 Done Done Sent false
19094 6138 Acked Read 6 Done Done Sent false
19095 6139 Acked Write 1 Done Done Sent false
19096 6140 Acked Read 2 Done Done Sent false
19097 6141 Acked Read 10 Done Done Sent false
19098 6142 Acked Read 4 Done Done Sent false
19099 6143 Acked Read 9 Done Done Sent false
19100 6144 Acked Write 9 Done Done Sent false
19101 6145 Acked Read 6 Done Done Sent false
19102 6146 Acked Write 9 Done Done Sent false
19103 6147 Acked Write 3 Done Done Sent false
19104 6148 Acked Write 7 Done Done Sent false
19105 6149 Acked Write 2 Done Done Sent false
19106 6150 Acked Read 5 Done Done Sent false
19107 6151 Acked Write 6 Done Done Sent false
19108 6152 Acked Flush 0 Done Done Sent false
19109 6153 Acked Read 1 Done Done Sent false
19110 6154 Acked Read 2 Done Done Sent false
19111 6155 Acked Read 7 Done Done Sent false
19112 6156 Acked Write 4 Done Done Sent false
19113 6157 Acked Read 2 Done Done Sent false
19114 6158 Acked Read 4 Done Done Sent false
19115 6159 Acked Read 6 Done Done Sent false
19116 6160 Acked Write 3 Done Done Sent false
19117 6161 Acked Write 1 Done Done Sent false
19118 6162 Acked Write 2 Done Done Sent false
19119 6163 Acked Write 6 Done Done Sent false
19120 6164 Acked Read 9 Done Done Sent false
19121 6165 Acked Read 8 Done Done Sent false
19122 6166 Acked Write 5 Done Done Sent false
19123 6167 Acked Write 1 Done Done Sent false
19124 6168 Acked Write 6 Done Done Sent false
19125 6169 Acked Read 8 Done Done Sent false
19126 6170 Acked Read 4 Done Done Sent false
19127 6171 Acked Read 2 Done Done Sent false
19128 6172 Acked Write 9 Done Done Sent false
19129 6173 Acked Flush 0 Done Done Sent false
19130 6174 Acked Write 6 Done Done Sent false
19131 6175 Acked Read 7 Done Done Sent false
19132 6176 Acked Read 3 Done Done Sent false
19133 6177 Acked Write 1 Done Done Sent false
19134 6178 Acked Flush 0 Done Done Sent false
19135 6179 Acked Write 7 Done Done Sent false
19136 6180 Acked Write 1 Done Done Sent false
19137 6181 Acked Read 3 Done Done Sent false
19138 6182 Acked Read 7 Done Done Sent false
19139 6183 Acked Read 5 Done Done Sent false
19140 6184 Acked Read 5 Done Done Sent false
19141 6185 Acked Write 5 Done Done Sent false
19142 6186 Acked Write 8 Done Done Sent false
19143 6187 Acked Write 2 Done Done Sent false
19144 6188 Acked Write 4 Done Done Sent false
19145 6189 Acked Read 7 Done Done Sent false
19146 6190 Acked Read 9 Done Done Sent false
19147 6191 Acked Read 10 Done Done Sent false
19148 6192 Acked Flush 0 Done Done Sent false
19149 6193 Acked Flush 0 Done Done Sent false
19150 6194 Acked Read 3 Done Done Sent false
19151 6195 Acked Read 6 Done Done Sent false
19152 6196 Acked Write 5 Done Done Sent false
19153 6197 Acked Read 2 Done Done Sent false
19154 6198 Acked Write 8 Done Done Sent false
19155 6199 Acked Write 1 Done Done Sent false
19156 6200 Acked Write 5 Done Done Sent false
19157 6201 Acked Write 3 Done Done Sent false
19158 6202 Acked Write 1 Done Done Sent false
19159 6203 Acked Write 6 Done Done Sent false
19160 6204 Acked Read 2 Done Done Sent false
19161 6205 Acked Read 7 Done Done Sent false
19162 6206 Acked Write 1 Done Done Sent false
19163 6207 Acked Read 10 Done Done Sent false
19164 6208 Acked Write 10 Done Done Sent false
19165 6209 Acked Write 7 Done Done Sent false
19166 6210 Acked Read 3 Done Done Sent false
19167 6211 Acked Read 8 Done Done Sent false
19168 6212 Acked Write 6 Done Done Sent false
19169 6213 Acked Read 7 Done Done Sent false
19170 6214 Acked Write 2 Done Done Sent false
19171 6215 Acked Flush 0 Done Done Sent false
19172 6216 Acked Read 1 Done Done Sent false
19173 6217 Acked Write 2 Done Done Sent false
19174 6218 Acked Read 8 Done Done Sent false
19175 6219 Acked Read 2 Done Done Sent false
19176 6220 Acked Write 3 Done Done Sent false
19177 6221 Acked Read 9 Done Done Sent false
19178 6222 Acked Read 9 Done Done Sent false
19179 6223 Acked Read 2 Done Done Sent false
19180 6224 Acked Read 8 Done Done Sent false
19181 6225 Acked Write 2 Done Done Sent false
19182 6226 Acked Read 6 Done Done Sent false
19183 6227 Acked Read 6 Done Done Sent false
19184 6228 Acked Write 6 Done Done Sent false
19185 6229 Acked Flush 0 Done Done Sent false
19186 6230 Acked Read 7 Done Done Sent false
19187 6231 Acked Read 9 Done Done Sent false
19188 6232 Acked Write 1 Done Done Sent false
19189 6233 Acked Write 10 Done Done Sent false
19190 6234 Acked Read 1 Done Done Sent false
19191 6235 Acked Write 3 Done Done Sent false
19192 6236 Acked Read 6 Done Done Sent false
19193 6237 Acked Write 3 Done Done Sent false
19194 6238 Acked Read 9 Done Done Sent false
19195 6239 Acked Write 10 Done Done Sent false
19196 6240 Acked Write 5 Done Done Sent false
19197 6241 Acked Read 10 Done Done Sent false
19198 6242 Acked Write 9 Done Done Sent false
19199 6243 Acked Write 4 Done Done Sent false
19200 6244 Acked Read 7 Done Done Sent false
19201 6245 Acked Read 6 Done Done Sent false
19202 6246 Acked Flush 0 Done Done Sent false
19203 6247 Acked Flush 0 Done Done Sent false
19204 6248 Acked Read 2 Done Done Sent false
19205 6249 Acked Write 2 Done Done Sent false
19206 6250 Acked Write 1 Done Done Sent false
19207 6251 Acked Read 5 Done Done Sent false
19208 6252 Acked Read 1 Done Done Sent false
19209 6253 Acked Write 9 Done Done Sent false
19210 6254 Acked Read 3 Done Done Sent false
19211 6255 Acked Flush 0 Done Done Sent false
19212 6256 Acked Write 1 Done Done Sent false
19213 6257 Acked Read 1 Done Done Sent false
19214 6258 Acked Read 8 Done Done Sent false
19215 6259 Acked Write 1 Done Done Sent false
19216 6260 Acked Write 10 Done Done Sent false
19217 6261 Acked Write 6 Done Done Sent false
19218 6262 Acked Write 6 Done Done Sent false
19219 6263 Acked Read 8 Done Done Sent false
19220 6264 Acked Read 5 Done Done Sent false
19221 6265 Acked Write 1 Done Done Sent false
19222 6266 Acked Flush 0 Done Done Sent false
19223 6267 Acked Read 9 Done Done Sent false
19224 6268 Acked Read 4 Done Done Sent false
19225 6269 Acked Read 10 Done Done Sent false
19226 6270 Acked Write 9 Done Done Sent false
19227 6271 Acked Read 10 Done Done Sent false
19228 6272 Acked Write 7 Done Done Sent false
19229 6273 Acked Flush 0 Done Done Sent false
19230 6274 Acked Read 9 Done Done Sent false
19231 6275 Acked Read 8 Done Done Sent false
19232 6276 Acked Write 9 Done Done Sent false
19233 6277 Acked Flush 0 Done Done Sent false
19234 6278 Acked Write 6 Done Done Sent false
19235 6279 Acked Write 5 Done Done Sent false
19236 6280 Acked Read 3 Done Done Sent false
19237 6281 Acked Flush 0 Done Done Sent false
19238 6282 Acked Read 8 Done Done Sent false
19239 6283 Acked Write 4 Done Done Sent false
19240 6284 Acked Write 5 Done Done Sent false
19241 6285 Acked Write 5 Done Done Sent false
19242 6286 Acked Write 10 Done Done Sent false
19243 6287 Acked Read 2 Done Done Sent false
19244 6288 Acked Flush 0 Done Done Sent false
19245 6289 Acked Write 2 Done Done Sent false
19246 6290 Acked Read 3 Done Done Sent false
19247 6291 Acked Flush 0 Done Done Sent false
19248 6292 Acked Write 8 Done Done Sent false
19249 6293 Acked Write 8 Done Done Sent false
19250 6294 Acked Write 3 Done Done Sent false
19251 6295 Acked Write 4 Done Done Sent false
19252 6296 Acked Read 5 Done Done Sent false
19253 6297 Acked Write 7 Done Done Sent false
19254 6298 Acked Read 7 Done Done Sent false
19255 6299 Acked Flush 0 Done Done Sent false
19256 6300 Acked Read 2 Done Done Sent false
19257 6301 Acked Read 7 Done Done Sent false
19258 6302 Acked Read 6 Done Done Sent false
19259 6303 Acked Flush 0 Done Done Sent false
19260 6304 Acked Read 9 Done Done Sent false
19261 6305 Acked Write 6 Done Done Sent false
19262 6306 Acked Write 1 Done Done Sent false
19263 6307 Acked Write 10 Done Done Sent false
19264 6308 Acked Read 5 Done Done Sent false
19265 6309 Acked Flush 0 Done Done Sent false
19266 6310 Acked Read 10 Done Done Sent false
19267 6311 Acked Read 10 Done Done Sent false
19268 6312 Acked Read 4 Done Done Sent false
19269 6313 Acked Read 4 Done Done Sent false
19270 6314 Acked Read 2 Done Done Sent false
19271 6315 Acked Write 2 Done Done Sent false
19272 6316 Acked Write 8 Done Done Sent false
19273 6317 Acked Read 5 Done Done Sent false
19274 6318 Acked Write 1 Done Done Sent false
19275 6319 Acked Read 1 Done Done Sent false
19276 6320 Acked Write 8 Done Done Sent false
19277 6321 Acked Write 6 Done Done Sent false
19278 6322 Acked Write 6 Done Done Sent false
19279 6323 Acked Write 5 Done Done Sent false
19280 6324 Acked Read 10 Done Done Sent false
19281 6325 Acked Read 7 Done Done Sent false
19282 6326 Acked Write 7 Done Done Sent false
19283 6327 Acked Write 6 Done Done Sent false
19284 6328 Acked Write 8 Done Done Sent false
19285 6329 Acked Write 5 Done Done Sent false
19286 6330 Acked Write 5 Done Done Sent false
19287 6331 Acked Read 7 Done Done Sent false
19288 6332 Acked Read 5 Done Done Sent false
19289 6333 Acked Write 4 Done Done Sent false
19290 6334 Acked Read 4 Done Done Sent false
19291 6335 Acked Write 6 Done Done Sent false
19292 6336 Acked Write 2 Done Done Sent false
19293 6337 Acked Write 4 Done Done Sent false
19294 6338 Acked Read 3 Done Done Sent false
19295 6339 Acked Write 7 Done Done Sent false
19296 6340 Acked Write 4 Done Done Sent false
19297 6341 Acked Write 8 Done Done Sent false
19298 6342 Acked Read 7 Done Done Sent false
19299 6343 Acked Write 4 Done Done Sent false
19300 6344 Acked Write 2 Done Done Sent false
19301 6345 Acked Read 5 Done Done Sent false
19302 6346 Acked Write 9 Done Done Sent false
19303 6347 Acked Read 5 Done Done Sent false
19304 6348 Acked Read 10 Done Done Sent false
19305 6349 Acked Write 1 Done Done Sent false
19306 6350 Acked Write 5 Done Done Sent false
19307 6351 Acked Write 3 Done Done Sent false
19308 6352 Acked Read 9 Done Done Sent false
19309 6353 Acked Write 9 Done Done Sent false
19310 6354 Acked Read 6 Done Done Sent false
19311 6355 Acked Read 8 Done Done Sent false
19312 6356 Acked Read 3 Done Done Sent false
19313 6357 Acked Read 8 Done Done Sent false
19314 6358 Acked Read 10 Done Done Sent false
19315 6359 Acked Read 3 Done Done Sent false
19316 6360 Acked Read 8 Done Done Sent false
19317 6361 Acked Read 3 Done Done Sent false
19318 6362 Acked Read 3 Done Done Sent false
19319 6363 Acked Read 1 Done Done Sent false
19320 6364 Acked Read 1 Done Done Sent false
19321 6365 Acked Read 3 Done Done Sent false
19322 6366 Acked Read 4 Done Done Sent false
19323 6367 Acked Read 1 Done Done Sent false
19324 6368 Acked Write 8 Done Done Sent false
19325 6369 Acked Flush 0 Done Done Sent false
19326 6370 Acked Read 10 Done Done Sent false
19327 6371 Acked Read 4 Done Done Sent false
19328 6372 Acked Write 5 Done Done Sent false
19329 6373 Acked Read 3 Done Done Sent false
19330 6374 Acked Flush 0 Done Done Sent false
19331 6375 Acked Write 3 Done Done Sent false
19332 6376 Acked Read 4 Done Done Sent false
19333 6377 Acked Read 1 Done Done Sent false
19334 6378 Acked Write 5 Done Done Sent false
19335 6379 Acked Read 10 Done Done Sent false
19336 6380 Acked Read 5 Done Done Sent false
19337 6381 Acked Read 3 Done Done Sent false
19338 6382 Acked Read 9 Done Done Sent false
19339 6383 Acked Read 10 Done Done Sent false
19340 6384 Acked Write 10 Done Done Sent false
19341 6385 Acked Read 5 Done Done Sent false
19342 6386 Acked Write 3 Done Done Sent false
19343 6387 Acked Write 3 Done Done Sent false
19344 6388 Acked Read 2 Done Done Sent false
19345 6389 Acked Flush 0 Done Done Sent false
19346 6390 Acked Write 2 Done Done Sent false
19347 6391 Acked Write 7 Done Done Sent false
19348 6392 Acked Read 7 Done Done Sent false
19349 6393 Acked Read 6 Done Done Sent false
19350 6394 Acked Flush 0 Done Done Sent false
19351 6395 Acked Read 5 Done Done Sent false
19352 6396 Acked Write 7 Done Done Sent false
19353 6397 Acked Read 7 Done Done Sent false
19354 6398 Acked Write 7 Done Done Sent false
19355 6399 Acked Write 3 Done Done Sent false
19356 6400 Acked Flush 0 Done Done Sent false
19357 6401 Acked Read 1 Done Done Sent false
19358 6402 Acked Flush 0 Done Done Sent false
19359 6403 Acked Read 2 Done Done Sent false
19360 6404 Acked Write 9 Done Done Sent false
19361 6405 Acked Read 6 Done Done Sent false
19362 6406 Acked Read 9 Done Done Sent false
19363 6407 Acked Read 8 Done Done Sent false
19364 6408 Acked Write 10 Done Done Sent false
19365 6409 Acked Write 8 Done Done Sent false
19366 6410 Acked Read 6 Done Done Sent false
19367 6411 Acked Read 1 Done Done Sent false
19368 6412 Acked Read 7 Done Done Sent false
19369 6413 Acked Read 8 Done Done Sent false
19370 6414 Acked Write 2 Done Done Sent false
19371 6415 Acked Read 1 Done Done Sent false
19372 6416 Acked Write 4 Done Done Sent false
19373 6417 Acked Read 3 Done Done Sent false
19374 6418 Acked Flush 0 Done Done Sent false
19375 6419 Acked Write 3 Done Done Sent false
19376 6420 Acked Read 1 Done Done Sent false
19377 6421 Acked Read 1 Done Done Sent false
19378 6422 Acked Read 5 Done Done Sent false
19379 6423 Acked Read 5 Done Done Sent false
19380 6424 Acked Read 2 Done Done Sent false
19381 6425 Acked Read 3 Done Done Sent false
19382 6426 Acked Write 5 Done Done Sent false
19383 6427 Acked Write 6 Done Done Sent false
19384 6428 Acked Read 6 Done Done Sent false
19385 6429 Acked Read 10 Done Done Sent false
19386 6430 Acked Read 9 Done Done Sent false
19387 6431 Acked Write 6 Done Done Sent false
19388 6432 Acked Read 4 Done Done Sent false
19389 6433 Acked Write 8 Done Done Sent false
19390 STATES DS:0 DS:1 DS:2 TOTAL
19391 Sent 0 0 301 301
19392 Done 301 301 0 602
19393 Skipped 0 0 0 0
19394 Error 0 0 0 0
19395 Last Flush: Some(JobId(6418)) Some(JobId(6418)) Some(JobId(6132))
19396 Downstairs last five completed: 6132 6131 6130 6129 6128
19397 Upstairs last five completed: 6433 6432 6431 6430 6429
19398 CLIENT: Up:0 ds:301 act:2
193992025-01-28T21:58:30.840ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
194002025-01-28T21:58:31.704ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
19401 [8830] got start action
19402 [8830] I am going to restart
19403 Starting downstairs at port 8830
19404 Make output file at "/tmp/dsc/downstairs-8830.txt"
19405 [8830][2] reports Starting
19406 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1220)
19407 [8830][2] reports Running
194082025-01-28T21:58:32.706ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
194092025-01-28T21:58:33.708ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
19410 ----------------------------------------------------------------
19411 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19412 States: Active Active Offline
19413 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19414 6133 Acked Read 8 Done Done Sent false
19415 6134 Acked Write 9 Done Done Sent false
19416 6135 Acked Write 9 Done Done Sent false
19417 6136 Acked Write 1 Done Done Sent false
19418 6137 Acked Flush 0 Done Done Sent false
19419 6138 Acked Read 6 Done Done Sent false
19420 6139 Acked Write 1 Done Done Sent false
19421 6140 Acked Read 2 Done Done Sent false
19422 6141 Acked Read 10 Done Done Sent false
19423 6142 Acked Read 4 Done Done Sent false
19424 6143 Acked Read 9 Done Done Sent false
19425 6144 Acked Write 9 Done Done Sent false
19426 6145 Acked Read 6 Done Done Sent false
19427 6146 Acked Write 9 Done Done Sent false
19428 6147 Acked Write 3 Done Done Sent false
19429 6148 Acked Write 7 Done Done Sent false
19430 6149 Acked Write 2 Done Done Sent false
19431 6150 Acked Read 5 Done Done Sent false
19432 6151 Acked Write 6 Done Done Sent false
19433 6152 Acked Flush 0 Done Done Sent false
19434 6153 Acked Read 1 Done Done Sent false
19435 6154 Acked Read 2 Done Done Sent false
19436 6155 Acked Read 7 Done Done Sent false
19437 6156 Acked Write 4 Done Done Sent false
19438 6157 Acked Read 2 Done Done Sent false
19439 6158 Acked Read 4 Done Done Sent false
19440 6159 Acked Read 6 Done Done Sent false
19441 6160 Acked Write 3 Done Done Sent false
19442 6161 Acked Write 1 Done Done Sent false
19443 6162 Acked Write 2 Done Done Sent false
19444 6163 Acked Write 6 Done Done Sent false
19445 6164 Acked Read 9 Done Done Sent false
19446 6165 Acked Read 8 Done Done Sent false
19447 6166 Acked Write 5 Done Done Sent false
19448 6167 Acked Write 1 Done Done Sent false
19449 6168 Acked Write 6 Done Done Sent false
19450 6169 Acked Read 8 Done Done Sent false
19451 6170 Acked Read 4 Done Done Sent false
19452 6171 Acked Read 2 Done Done Sent false
19453 6172 Acked Write 9 Done Done Sent false
19454 6173 Acked Flush 0 Done Done Sent false
19455 6174 Acked Write 6 Done Done Sent false
19456 6175 Acked Read 7 Done Done Sent false
19457 6176 Acked Read 3 Done Done Sent false
19458 6177 Acked Write 1 Done Done Sent false
19459 6178 Acked Flush 0 Done Done Sent false
19460 6179 Acked Write 7 Done Done Sent false
19461 6180 Acked Write 1 Done Done Sent false
19462 6181 Acked Read 3 Done Done Sent false
19463 6182 Acked Read 7 Done Done Sent false
19464 6183 Acked Read 5 Done Done Sent false
19465 6184 Acked Read 5 Done Done Sent false
19466 6185 Acked Write 5 Done Done Sent false
19467 6186 Acked Write 8 Done Done Sent false
19468 6187 Acked Write 2 Done Done Sent false
19469 6188 Acked Write 4 Done Done Sent false
19470 6189 Acked Read 7 Done Done Sent false
19471 6190 Acked Read 9 Done Done Sent false
19472 6191 Acked Read 10 Done Done Sent false
19473 6192 Acked Flush 0 Done Done Sent false
19474 6193 Acked Flush 0 Done Done Sent false
19475 6194 Acked Read 3 Done Done Sent false
19476 6195 Acked Read 6 Done Done Sent false
19477 6196 Acked Write 5 Done Done Sent false
19478 6197 Acked Read 2 Done Done Sent false
19479 6198 Acked Write 8 Done Done Sent false
19480 6199 Acked Write 1 Done Done Sent false
19481 6200 Acked Write 5 Done Done Sent false
19482 6201 Acked Write 3 Done Done Sent false
19483 6202 Acked Write 1 Done Done Sent false
19484 6203 Acked Write 6 Done Done Sent false
19485 6204 Acked Read 2 Done Done Sent false
19486 6205 Acked Read 7 Done Done Sent false
19487 6206 Acked Write 1 Done Done Sent false
19488 6207 Acked Read 10 Done Done Sent false
19489 6208 Acked Write 10 Done Done Sent false
19490 6209 Acked Write 7 Done Done Sent false
19491 6210 Acked Read 3 Done Done Sent false
19492 6211 Acked Read 8 Done Done Sent false
19493 6212 Acked Write 6 Done Done Sent false
19494 6213 Acked Read 7 Done Done Sent false
19495 6214 Acked Write 2 Done Done Sent false
19496 6215 Acked Flush 0 Done Done Sent false
19497 6216 Acked Read 1 Done Done Sent false
19498 6217 Acked Write 2 Done Done Sent false
19499 6218 Acked Read 8 Done Done Sent false
19500 6219 Acked Read 2 Done Done Sent false
19501 6220 Acked Write 3 Done Done Sent false
19502 6221 Acked Read 9 Done Done Sent false
19503 6222 Acked Read 9 Done Done Sent false
19504 6223 Acked Read 2 Done Done Sent false
19505 6224 Acked Read 8 Done Done Sent false
19506 6225 Acked Write 2 Done Done Sent false
19507 6226 Acked Read 6 Done Done Sent false
19508 6227 Acked Read 6 Done Done Sent false
19509 6228 Acked Write 6 Done Done Sent false
19510 6229 Acked Flush 0 Done Done Sent false
19511 6230 Acked Read 7 Done Done Sent false
19512 6231 Acked Read 9 Done Done Sent false
19513 6232 Acked Write 1 Done Done Sent false
19514 6233 Acked Write 10 Done Done Sent false
19515 6234 Acked Read 1 Done Done Sent false
19516 6235 Acked Write 3 Done Done Sent false
19517 6236 Acked Read 6 Done Done Sent false
19518 6237 Acked Write 3 Done Done Sent false
19519 6238 Acked Read 9 Done Done Sent false
19520 6239 Acked Write 10 Done Done Sent false
19521 6240 Acked Write 5 Done Done Sent false
19522 6241 Acked Read 10 Done Done Sent false
19523 6242 Acked Write 9 Done Done Sent false
19524 6243 Acked Write 4 Done Done Sent false
19525 6244 Acked Read 7 Done Done Sent false
19526 6245 Acked Read 6 Done Done Sent false
19527 6246 Acked Flush 0 Done Done Sent false
19528 6247 Acked Flush 0 Done Done Sent false
19529 6248 Acked Read 2 Done Done Sent false
19530 6249 Acked Write 2 Done Done Sent false
19531 6250 Acked Write 1 Done Done Sent false
19532 6251 Acked Read 5 Done Done Sent false
19533 6252 Acked Read 1 Done Done Sent false
19534 6253 Acked Write 9 Done Done Sent false
19535 6254 Acked Read 3 Done Done Sent false
19536 6255 Acked Flush 0 Done Done Sent false
19537 6256 Acked Write 1 Done Done Sent false
19538 6257 Acked Read 1 Done Done Sent false
19539 6258 Acked Read 8 Done Done Sent false
19540 6259 Acked Write 1 Done Done Sent false
19541 6260 Acked Write 10 Done Done Sent false
19542 6261 Acked Write 6 Done Done Sent false
19543 6262 Acked Write 6 Done Done Sent false
19544 6263 Acked Read 8 Done Done Sent false
19545 6264 Acked Read 5 Done Done Sent false
19546 6265 Acked Write 1 Done Done Sent false
19547 6266 Acked Flush 0 Done Done Sent false
19548 6267 Acked Read 9 Done Done Sent false
19549 6268 Acked Read 4 Done Done Sent false
19550 6269 Acked Read 10 Done Done Sent false
19551 6270 Acked Write 9 Done Done Sent false
19552 6271 Acked Read 10 Done Done Sent false
19553 6272 Acked Write 7 Done Done Sent false
19554 6273 Acked Flush 0 Done Done Sent false
19555 6274 Acked Read 9 Done Done Sent false
19556 6275 Acked Read 8 Done Done Sent false
19557 6276 Acked Write 9 Done Done Sent false
19558 6277 Acked Flush 0 Done Done Sent false
19559 6278 Acked Write 6 Done Done Sent false
19560 6279 Acked Write 5 Done Done Sent false
19561 6280 Acked Read 3 Done Done Sent false
19562 6281 Acked Flush 0 Done Done Sent false
19563 6282 Acked Read 8 Done Done Sent false
19564 6283 Acked Write 4 Done Done Sent false
19565 6284 Acked Write 5 Done Done Sent false
19566 6285 Acked Write 5 Done Done Sent false
19567 6286 Acked Write 10 Done Done Sent false
19568 6287 Acked Read 2 Done Done Sent false
19569 6288 Acked Flush 0 Done Done Sent false
19570 6289 Acked Write 2 Done Done Sent false
19571 6290 Acked Read 3 Done Done Sent false
19572 6291 Acked Flush 0 Done Done Sent false
19573 6292 Acked Write 8 Done Done Sent false
19574 6293 Acked Write 8 Done Done Sent false
19575 6294 Acked Write 3 Done Done Sent false
19576 6295 Acked Write 4 Done Done Sent false
19577 6296 Acked Read 5 Done Done Sent false
19578 6297 Acked Write 7 Done Done Sent false
19579 6298 Acked Read 7 Done Done Sent false
19580 6299 Acked Flush 0 Done Done Sent false
19581 6300 Acked Read 2 Done Done Sent false
19582 6301 Acked Read 7 Done Done Sent false
19583 6302 Acked Read 6 Done Done Sent false
19584 6303 Acked Flush 0 Done Done Sent false
19585 6304 Acked Read 9 Done Done Sent false
19586 6305 Acked Write 6 Done Done Sent false
19587 6306 Acked Write 1 Done Done Sent false
19588 6307 Acked Write 10 Done Done Sent false
19589 6308 Acked Read 5 Done Done Sent false
19590 6309 Acked Flush 0 Done Done Sent false
19591 6310 Acked Read 10 Done Done Sent false
19592 6311 Acked Read 10 Done Done Sent false
19593 6312 Acked Read 4 Done Done Sent false
19594 6313 Acked Read 4 Done Done Sent false
19595 6314 Acked Read 2 Done Done Sent false
19596 6315 Acked Write 2 Done Done Sent false
19597 6316 Acked Write 8 Done Done Sent false
19598 6317 Acked Read 5 Done Done Sent false
19599 6318 Acked Write 1 Done Done Sent false
19600 6319 Acked Read 1 Done Done Sent false
19601 6320 Acked Write 8 Done Done Sent false
19602 6321 Acked Write 6 Done Done Sent false
19603 6322 Acked Write 6 Done Done Sent false
19604 6323 Acked Write 5 Done Done Sent false
19605 6324 Acked Read 10 Done Done Sent false
19606 6325 Acked Read 7 Done Done Sent false
19607 6326 Acked Write 7 Done Done Sent false
19608 6327 Acked Write 6 Done Done Sent false
19609 6328 Acked Write 8 Done Done Sent false
19610 6329 Acked Write 5 Done Done Sent false
19611 6330 Acked Write 5 Done Done Sent false
19612 6331 Acked Read 7 Done Done Sent false
19613 6332 Acked Read 5 Done Done Sent false
19614 6333 Acked Write 4 Done Done Sent false
19615 6334 Acked Read 4 Done Done Sent false
19616 6335 Acked Write 6 Done Done Sent false
19617 6336 Acked Write 2 Done Done Sent false
19618 6337 Acked Write 4 Done Done Sent false
19619 6338 Acked Read 3 Done Done Sent false
19620 6339 Acked Write 7 Done Done Sent false
19621 6340 Acked Write 4 Done Done Sent false
19622 6341 Acked Write 8 Done Done Sent false
19623 6342 Acked Read 7 Done Done Sent false
19624 6343 Acked Write 4 Done Done Sent false
19625 6344 Acked Write 2 Done Done Sent false
19626 6345 Acked Read 5 Done Done Sent false
19627 6346 Acked Write 9 Done Done Sent false
19628 6347 Acked Read 5 Done Done Sent false
19629 6348 Acked Read 10 Done Done Sent false
19630 6349 Acked Write 1 Done Done Sent false
19631 6350 Acked Write 5 Done Done Sent false
19632 6351 Acked Write 3 Done Done Sent false
19633 6352 Acked Read 9 Done Done Sent false
19634 6353 Acked Write 9 Done Done Sent false
19635 6354 Acked Read 6 Done Done Sent false
19636 6355 Acked Read 8 Done Done Sent false
19637 6356 Acked Read 3 Done Done Sent false
19638 6357 Acked Read 8 Done Done Sent false
19639 6358 Acked Read 10 Done Done Sent false
19640 6359 Acked Read 3 Done Done Sent false
19641 6360 Acked Read 8 Done Done Sent false
19642 6361 Acked Read 3 Done Done Sent false
19643 6362 Acked Read 3 Done Done Sent false
19644 6363 Acked Read 1 Done Done Sent false
19645 6364 Acked Read 1 Done Done Sent false
19646 6365 Acked Read 3 Done Done Sent false
19647 6366 Acked Read 4 Done Done Sent false
19648 6367 Acked Read 1 Done Done Sent false
19649 6368 Acked Write 8 Done Done Sent false
19650 6369 Acked Flush 0 Done Done Sent false
19651 6370 Acked Read 10 Done Done Sent false
19652 6371 Acked Read 4 Done Done Sent false
19653 6372 Acked Write 5 Done Done Sent false
19654 6373 Acked Read 3 Done Done Sent false
19655 6374 Acked Flush 0 Done Done Sent false
19656 6375 Acked Write 3 Done Done Sent false
19657 6376 Acked Read 4 Done Done Sent false
19658 6377 Acked Read 1 Done Done Sent false
19659 6378 Acked Write 5 Done Done Sent false
19660 6379 Acked Read 10 Done Done Sent false
19661 6380 Acked Read 5 Done Done Sent false
19662 6381 Acked Read 3 Done Done Sent false
19663 6382 Acked Read 9 Done Done Sent false
19664 6383 Acked Read 10 Done Done Sent false
19665 6384 Acked Write 10 Done Done Sent false
19666 6385 Acked Read 5 Done Done Sent false
19667 6386 Acked Write 3 Done Done Sent false
19668 6387 Acked Write 3 Done Done Sent false
19669 6388 Acked Read 2 Done Done Sent false
19670 6389 Acked Flush 0 Done Done Sent false
19671 6390 Acked Write 2 Done Done Sent false
19672 6391 Acked Write 7 Done Done Sent false
19673 6392 Acked Read 7 Done Done Sent false
19674 6393 Acked Read 6 Done Done Sent false
19675 6394 Acked Flush 0 Done Done Sent false
19676 6395 Acked Read 5 Done Done Sent false
19677 6396 Acked Write 7 Done Done Sent false
19678 6397 Acked Read 7 Done Done Sent false
19679 6398 Acked Write 7 Done Done Sent false
19680 6399 Acked Write 3 Done Done Sent false
19681 6400 Acked Flush 0 Done Done Sent false
19682 6401 Acked Read 1 Done Done Sent false
19683 6402 Acked Flush 0 Done Done Sent false
19684 6403 Acked Read 2 Done Done Sent false
19685 6404 Acked Write 9 Done Done Sent false
19686 6405 Acked Read 6 Done Done Sent false
19687 6406 Acked Read 9 Done Done Sent false
19688 6407 Acked Read 8 Done Done Sent false
19689 6408 Acked Write 10 Done Done Sent false
19690 6409 Acked Write 8 Done Done Sent false
19691 6410 Acked Read 6 Done Done Sent false
19692 6411 Acked Read 1 Done Done Sent false
19693 6412 Acked Read 7 Done Done Sent false
19694 6413 Acked Read 8 Done Done Sent false
19695 6414 Acked Write 2 Done Done Sent false
19696 6415 Acked Read 1 Done Done Sent false
19697 6416 Acked Write 4 Done Done Sent false
19698 6417 Acked Read 3 Done Done Sent false
19699 6418 Acked Flush 0 Done Done Sent false
19700 6419 Acked Write 3 Done Done Sent false
19701 6420 Acked Read 1 Done Done Sent false
19702 6421 Acked Read 1 Done Done Sent false
19703 6422 Acked Read 5 Done Done Sent false
19704 6423 Acked Read 5 Done Done Sent false
19705 6424 Acked Read 2 Done Done Sent false
19706 6425 Acked Read 3 Done Done Sent false
19707 6426 Acked Write 5 Done Done Sent false
19708 6427 Acked Write 6 Done Done Sent false
19709 6428 Acked Read 6 Done Done Sent false
19710 6429 Acked Read 10 Done Done Sent false
19711 6430 Acked Read 9 Done Done Sent false
19712 6431 Acked Write 6 Done Done Sent false
19713 6432 Acked Read 4 Done Done Sent false
19714 6433 Acked Write 8 Done Done Sent false
19715 STATES DS:0 DS:1 DS:2 TOTAL
19716 Sent 0 0 301 301
19717 Done 301 301 0 602
19718 Skipped 0 0 0 0
19719 Error 0 0 0 0
19720 Last Flush: Some(JobId(6418)) Some(JobId(6418)) Some(JobId(6132))
19721 Downstairs last five completed: 6132 6131 6130 6129 6128
19722 Upstairs last five completed: 6433 6432 6431 6430 6429
19723 CLIENT: Up:0 ds:301 act:2
197242025-01-28T21:58:35.140ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
197252025-01-28T21:58:35.712ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
197262025-01-28T21:58:36.102ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
197272025-01-28T21:58:36.102ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33104) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
197282025-01-28T21:58:36.103ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
197292025-01-28T21:58:36.104ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
197302025-01-28T21:58:36.104ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
197312025-01-28T21:58:36.104ZINFOcrucible: send last flush ID to this DS: Some(JobId(6132)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
197322025-01-28T21:58:36.104ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
197332025-01-28T21:58:36.104ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6132)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
197342025-01-28T21:58:36.104ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6132)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
19735 ----------------------------------------------------------------
19736 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19737 Downstairs last five completed: 6434 6433 6432 6431 6430
19738 Upstairs last five completed: 6434 6433 6432 6431 6430
19739 CLIENT: Up:0 ds:0 act:3
19740 Replay: All jobs finished
19741 Main task has work to do, go find it
19742 got dsc Stop(1)
19743 stop 1
19744 [8820] Got stop action so:false kr:false
19745 Jan 28 21:58:38.343 INFO request completed, latency_us: 376, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 5f9a1a77-d009-4fe9-be1d-e609e61290a0, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
197462025-01-28T21:58:38.344ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
197472025-01-28T21:58:38.344ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
197482025-01-28T21:58:38.344ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
19749 Jan 28 21:58:38.344 INFO request completed, latency_us: 213, response_code{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:58:38.344690053Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
19750 : 200, uri: /state/cid/1, method: GET, req_id: 73d82cd1-5c5b-4d45-9f20-3f84e3156aca, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
19751 [8820] Exited with: ExitStatus(unix_wait_status(9))
19752 [8820][1] reports Exit
19753 Jan 28 21:58:42.346 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/1, method: GET, req_id: b7ddb07b-0618-478a-9121-61e34517ce72, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
19754 001/300 Write block 4848 len 8192 data: 4 5
19755 002/300 Read block 4099 len 8192
19756 003/300 Read block 1932 len 24576
19757 004/300 Write block 4503 len 36864 data: 4 5 5 5 5 4 4 5 5
19758 005/300 Read block 2557 len 16384
19759 006/300 Read block 2005 len 28672
19760 007/300 Write block 4677 len 36864 data: 3 4 6 6 6 6 6 6 6
19761 008/300 Read block 4819 len 24576
19762 009/300 Write block 3509 len 8192 data: 2 3
19763 010/300 Read block 3655 len 24576
19764 011/300 Read block 259 len 32768
19765 012/300 Write block 4964 len 4096 data: 6
19766 013/300 Read block 294 len 24576
19767 014/300 Read block 3419 len 28672
19768 015/300 Read block 768 len 24576
19769 016/300 Flush
19770 017/300 Write block 2549 len 24576 data: 4 4 4 4 4 4
19771 018/300 Write block 9 len 40960 data: 5 5 5 6 5 5 4 4 3 2
19772 019/300 Read block 221 len 4096
19773 020/300 Read block 3694 len 20480
19774 021/300 Read block 879 len 16384
19775 022/300 Write block 1923 len 20480 data: 6 6 7 6 5
19776 023/300 Read block 4665 len 40960
19777 024/300 Write block 3650 len 24576 data: 5 5 4 3 3 2
19778 025/300 Write block 3223 len 40960 data: 5 6 7 6 6 6 5 5 4 4
19779 026/300 Write block 3049 len 24576 data: 7 8 5 5 5 5
19780 027/300 Read block 3246 len 12288
19781 028/300 Write block 2601 len 36864 data: 4 3 4 4 3 2 2 2 2
19782 029/300 Read block 528 len 24576
19783 030/300 Write block 1525 len 40960 data: 5 4 5 5 4 5 5 4 4 4
19784 031/300 Read block 2714 len 8192
19785 032/300 Write block 4573 len 12288 data: 2 2 2
19786 033/300 Read block 2008 len 40960
19787 034/300 Flush
19788 035/300 Read block 844 len 8192
19789 036/300 Read block 2853 len 36864
19790 037/300 Read block 4003 len 20480
19791 038/300 Read block 2105 len 24576
19792 039/300 Read block 3148 len 36864
19793 040/300 Read block 3421 len 36864
19794 041/300 Flush
19795 042/300 Read block 2336 len 36864
19796 043/300 Read block 2245 len 24576
19797 044/300 Write block 1717 len 36864 data: 7 7 6 5 5 4 3 2 2
19798 045/300 Write block 1827 len 12288 data: 6 5 5
19799 046/300 Read block 151 len 24576
19800 047/300 Read block 803 len 12288
19801 048/300 Read block 2863 len 24576
19802 049/300 Flush
19803 050/300 Write block 3031 len 28672 data: 2 2 2 2 2 2 2
19804 051/300 Flush
19805 052/300 Read block 2756 len 36864
19806 053/300 Write block 1094 len 28672 data: 2 3 3 3 2 2 4
19807 054/300 Read block 3941 len 36864
19808 055/300 Read block 1065 len 40960
19809 056/300 Write block 2374 len 40960 data: 7 4 5 4 4 4 4 4 6 8
19810 057/300 Write block 2281 len 16384 data: 3 3 5 5
19811 058/300 Write block 4628 len 40960 data: 5 4 3 4 4 4 4 4 4 4
19812 059/300 Read block 2024 len 16384
19813 060/300 Write block 4602 len 8192 data: 6 6
19814 061/300 Read block 3745 len 40960
19815 062/300 Write block 477 len 28672 data: 8 6 5 4 3 4 4
19816 063/300 Read block 4241 len 28672
19817 064/300 Write block 935 len 32768 data: 4 6 7 8 9 9 10 9
19818 065/300 Write block 2189 len 8192 data: 4 5
19819 066/300 Flush
19820 067/300 Write block 1496 len 4096 data: 2
19821 068/300 Flush
19822 069/300 Write block 2907 len 32768 data: 3 2 3 3 3 3 3 4
19823 070/300 Read block 2943 len 40960
19824 071/300 Read block 4650 len 36864
19825 072/300 Write block 2019 len 12288 data: 4 3 3
19826 073/300 Read block 2354 len 4096
19827 074/300 Write block 2599 len 36864 data: 4 4 5 4 5 5 4 3 3
19828 075/300 Read block 3610 len 36864
19829 076/300 Read block 968 len 8192
19830 077/300 Flush
19831 078/300 Flush
19832 079/300 Read block 4050 len 36864
19833 080/300 Read block 412 len 4096
19834 081/300 Write block 1637 len 24576 data: 5 4 4 4 4 4
19835 082/300 Read block 1499 len 40960
19836 083/300 Write block 2200 len 12288 data: 4 6 6
19837 084/300 Read block 2312 len 20480
19838 085/300 Read block 3237 len 40960
19839 086/300 Read block 2606 len 24576
19840 087/300 Flush
19841 088/300 Write block 369 len 12288 data: 5 4 5
19842 089/300 Read block 3414 len 20480
19843 090/300 Read block 755 len 16384
19844 091/300 Flush
19845 092/300 Write block 3288 len 8192 data: 4 4
19846 093/300 Flush
19847 094/300 Read block 3175 len 40960
19848 095/300 Read block 4066 len 40960
19849 096/300 Read block 4903 len 12288
19850 097/300 Read block 1548 len 40960
19851 098/300 Write block 4513 len 32768 data: 4 4 4 3 4 4 3 5
19852 099/300 Read block 454 len 32768
19853 100/300 Write block 100 len 4096 data: 4
19854 101/300 Read block 3763 len 8192
19855 102/300 Write block 728 len 4096 data: 6
19856 103/300 Read block 2530 len 24576
19857 104/300 Write block 1459 len 4096 data: 6
19858 105/300 Write block 4544 len 36864 data: 5 6 4 3 3 3 4 3 4
19859 106/300 Write block 435 len 28672 data: 4 4 3 2 3 3 4
19860 107/300 Read block 2896 len 32768
19861 108/300 Write block 1389 len 28672 data: 2 2 2 2 2 2 2
19862 109/300 Flush
19863 110/300 Write block 3123 len 8192 data: 3 2
19864 111/300 Read block 3482 len 24576
19865 112/300 Write block 2897 len 28672 data: 4 4 6 6 6 6 6
19866 113/300 Read block 1401 len 32768
19867 114/300 Read block 2823 len 4096
19868 115/300 Read block 2707 len 8192
19869 116/300 Read block 2 len 20480
19870 117/300 Read block 958 len 12288
19871 118/300 Write block 2906 len 4096 data: 3
19872 119/300 Read block 2964 len 8192
19873 120/300 Flush
19874 121/300 Read block 1794 len 24576
19875 122/300 Read block 1882 len 20480
19876 123/300 Write block 957 len 40960 data: 2 2 3 3 3 3 5 5 5 6
19877 124/300 Read block 3139 len 28672
19878 125/300 Read block 4462 len 8192
19879 126/300 Read block 3688 len 24576
19880 127/300 Read block 1161 len 36864
19881 128/300 Read block 2701 len 24576
19882 129/300 Flush
19883 130/300 Read block 4118 len 20480
19884 131/300 Write block 2799 len 28672 data: 6 5 5 5 5 4 3
19885 132/300 Read block 2808 len 40960
19886 133/300 Write block 4194 len 40960 data: 4 5 5 6 5 5 5 3 2 3
19887 134/300 Read block 4186 len 28672
19888 135/300 Write block 2697 len 28672 data: 6 4 2 2 3 4 4
19889 136/300 Read block 856 len 4096
19890 137/300 Read block 2411 len 16384
19891 138/300 Write block 1324 len 12288 data: 4 4 3
19892 139/300 Read block 2465 len 12288
19893 140/300 Write block 549 len 4096 data: 6
19894 141/300 Read block 513 len 16384
19895 142/300 Read block 1069 len 36864
19896 143/300 Read block 1942 len 28672
19897 144/300 Flush
19898 145/300 Read block 3793 len 36864
19899 146/300 Write block 2108 len 16384 data: 3 3 3 2
19900 147/300 Read block 164 len 8192
19901 148/300 Write block 150 len 32768 data: 4 4 6 6 7 7 7 9
19902 149/300 Flush
19903 150/300 Read block 2663 len 24576
19904 151/300 Read block 879 len 12288
19905 152/300 Write block 4102 len 8192 data: 5 6
19906 153/300 Write block 3360 len 36864 data: 5 6 6 4 3 3 5 4 5
19907 154/300 Read block 2137 len 24576
19908 155/300 Read block 3877 len 36864
19909 156/300 Write block 114 len 20480 data: 4 3 3 3 3
19910 157/300 Write block 3899 len 12288 data: 5 4 4
19911 158/300 Write block 4687 len 28672 data: 5 3 3 5 5 4 5
19912 159/300 Write block 1656 len 8192 data: 4 4
19913 160/300 Read block 1035 len 12288
19914 161/300 Write block 511 len 32768 data: 3 3 2 3 2 3 3 3
19915 162/300 Write block 3732 len 12288 data: 6 6 5
19916 163/300 Read block 4699 len 8192
19917 164/300 Write block 953 len 28672 data: 3 3 3 2 3 3 4
19918 165/300 Read block 1902 len 20480
19919 166/300 Flush
19920 167/300 Read block 623 len 40960
19921 168/300 Write block 607 len 28672 data: 5 5 4 4 3 2 2
19922 169/300 Read block 3138 len 36864
19923 170/300 Read block 2267 len 16384
19924 171/300 Write block 110 len 20480 data: 2 2 3 3 5
19925 172/300 Read block 2621 len 40960
19926 173/300 Write block 4545 len 36864 data: 7 5 4 4 4 5 4 5 4
19927 174/300 Read block 4969 len 8192
19928 175/300 Write block 2907 len 32768 data: 4 3 4 4 4 4 4 5
19929 176/300 Read block 1298 len 4096
19930 177/300 Read block 138 len 32768
19931 178/300 Read block 594 len 28672
19932 179/300 Write block 651 len 16384 data: 7 9 9 6
19933 180/300 Read block 493 len 4096
19934 181/300 Write block 3337 len 28672 data: 6 5 4 4 4 5 5
19935 182/300 Read block 2613 len 4096
19936 183/300 Read block 310 len 32768
19937 184/300 Read block 4469 len 4096
19938 185/300 Write block 733 len 12288 data: 8 7 4
19939 186/300 Write block 3695 len 24576 data: 2 2 2 3 4 4
19940 187/300 Read block 4219 len 12288
19941 188/300 Read block 906 len 4096
19942 189/300 Write block 399 len 4096 data: 7
19943 190/300 Write block 1329 len 36864 data: 2 2 3 4 4 4 4 5 4
19944 191/300 Flush
19945 192/300 Read block 2304 len 4096
19946 193/300 Read block 2414 len 8192
19947 194/300 Read block 1562 len 36864
19948 195/300 Write block 3229 len 4096 data: 6
19949 196/300 Read block 2618 len 32768
19950 197/300 Read block 4074 len 8192
19951 198/300 Write block 284 len 16384 data: 4 4 3 3
19952 199/300 Read block 1729 len 32768
19953 200/300 Write block 1432 len 36864 data: 5 6 4 4 4 4 4 3 2
19954 201/300 Read block 858 len 32768
19955 202/300 Write block 1621 len 20480 data: 5 4 4 4 4
19956 203/300 Write block 3980 len 20480 data: 5 6 6 6 6
19957 204/300 Write block 345 len 24576 data: 7 7 7 6 8 7
19958 205/300 Read block 3086 len 36864
19959 206/300 Read block 816 len 12288
19960 207/300 Read block 85 len 12288
19961 208/300 Read block 1468 len 40960
19962 209/300 Write block 4653 len 40960 data: 4 4 4 4 3 7 8 8 9 9
19963 210/300 Write block 2089 len 40960 data: 3 4 4 3 4 4 4 4 4 5
19964 211/300 Read block 4075 len 32768
19965 212/300 Read block 4506 len 20480
19966 213/300 Write block 4064 len 16384 data: 4 4 4 3
19967 214/300 Write block 4560 len 8192 data: 5 5
19968 215/300 Flush
19969 216/300 Write block 1328 len 32768 data: 2 3 3 4 5 5 5 5
19970 217/300 Read block 3074 len 4096
19971 218/300 Write block 24 len 24576 data: 5 4 4 3 3 3
19972 219/300 Write block 935 len 20480 data: 5 7 8 9 10
19973 220/300 Write block 2310 len 12288 data: 4 4 4
19974 221/300 Read block 2062 len 24576
19975 222/300 Read block 2772 len 40960
19976 223/300 Write block 2134 len 4096 data: 8
19977 224/300 Write block 2568 len 20480 data: 3 2 2 3 2
19978 225/300 Write block 4849 len 20480 data: 6 6 6 6 7
19979 226/300 Read block 1147 len 24576
19980 227/300 Flush
19981 228/300 Read block 1046 len 40960
19982 229/300 Read block 1520 len 20480
19983 230/300 Read block 4214 len 8192
19984 231/300 Read block 4278 len 28672
19985 232/300 Write block 3991 len 16384 data: 3 3 4 4
19986 233/300 Write block 2005 len 28672 data: 4 4 4 5 4 5 5
19987 234/300 Write block 2450 len 32768 data: 4 5 4 4 5 5 5 4
19988 235/300 Write block 195 len 36864 data: 3 3 3 3 3 3 3 3 3
19989 236/300 Write block 2690 len 12288 data: 6 7 7
19990 237/300 Read block 3545 len 8192
19991 238/300 Read block 1586 len 24576
19992 239/300 Write block 1043 len 32768 data: 7 7 6 8 8 7 5 5
19993 240/300 Read block 3480 len 12288
19994 241/300 Read block 4924 len 36864
19995 242/300 Read block 2798 len 40960
19996 243/300 Read block 3361 len 36864
19997 244/300 Read block 614 len 20480
19998 245/300 Write block 2549 len 16384 data: 5 5 5 5
19999 246/300 Read block 3772 len 40960
20000 247/300 Flush
20001 248/300 Write block 3079 len 16384 data: 8 8 7 7
20002 249/300 Read block 1331 len 8192
20003 250/300 Read block 831 len 16384
20004 251/300 Read block 1458 len 8192
20005 252/300 Read block 1120 len 20480
20006 253/300 Write block 4616 len 36864 data: 5 5 5 5 5 5 4 3 3
20007 254/300 Read block 825 len 40960
20008 255/300 Read block 3924 len 24576
20009 256/300 Read block 4685 len 28672
20010 257/300 Write block 3869 len 32768 data: 2 4 4 4 5 4 4 4
20011 258/300 Write block 2914 len 36864 data: 6 4 2 2 4 4 5 5 5
20012 259/300 Write block 3713 len 12288 data: 5 6 5
20013 260/300 Read block 2278 len 4096
20014 261/300 Read block 3626 len 8192
20015 262/300 Write block 4646 len 36864 data: 2 2 2 2 3 3 3 5 5
20016 263/300 Write block 4773 len 16384 data: 10 8 7 6
20017 264/300 Read block 4181 len 4096
20018 265/300 Read block 3483 len 24576
20019 266/300 Write block 1095 len 28672 data: 4 4 4 3 3 5 5
20020 267/300 Write block 2128 len 4096 data: 4
20021 268/300 Write block 3623 len 20480 data: 4 5 5 4 3
20022 269/300 Read block 1083 len 36864
20023 270/300 Flush
20024 271/300 Write block 819 len 16384 data: 2 3 4 4
20025 272/300 Write block 3908 len 12288 data: 6 7 6
20026 273/300 Read block 1226 len 36864
20027 274/300 Write block 4835 len 40960 data: 4 4 4 3 4 5 6 6 5 5
20028 275/300 Read block 1380 len 4096
20029 276/300 Read block 3195 len 12288
20030 277/300 Write block 2113 len 24576 data: 2 3 3 3 2 2
20031 278/300 Write block 891 len 24576 data: 3 5 5 2 2 2
20032 279/300 Read block 2217 len 40960
20033 280/300 Flush
20034 281/300 Read block 3729 len 36864
20035 282/300 Flush
20036 283/300 Read block 4726 len 36864
20037 284/300 Write block 29 len 20480 data: 4 3 3 3 3
20038 285/300 Write block 285 len 28672 data: 5 4 4 3 3 3 2
20039 286/300 Read block 850 len 32768
20040 287/300 Flush
20041 288/300 Flush
20042 289/300 Read block 3612 len 4096
20043 290/300 Write block 4736 len 4096 data: 4
20044 291/300 Read block 3512 len 12288
20045 292/300 Write block 120 len 28672 data: 4 4 3 4 6 6 8
20046 293/300 Read block 1229 len 32768
20047 294/300 Read block 1918 len 12288
20048 295/300 Write block 1658 len 28672 data: 5 6 7 7 7 7 10
20049 296/300 Write block 2048 len 12288 data: 4 4 3
20050 297/300 Write block 2774 len 8192 data: 5 4
20051 298/300 Write block 3735 len 4096 data: 5
20052 299/300 Read block 348 len 4096
20053 300/300 Write block 3010 len 16384 data: 2 2 2 3
20054 301/300 Write block 4688 len 8192 data: 4 4
20055 Main task has work to do, go find it
20056 got dsc Start(1)
20057 start 1
20058 Jan 28 21:58:42.592 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/1, method: POST, req_id: 51eabebd-a5ff-4f35-ba36-ce59b8e2585c, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
20059 Replay: started 1, returned:Ok(())
20060 ----------------------------------------------------------------
20061 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20062 States: Active Offline Active
20063 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20064 6435 Acked Write 2 Done Sent Done false
20065 6436 Acked Read 2 Done Sent Done false
20066 6437 Acked Read 6 Done Sent Done false
20067 6438 Acked Write 9 Done Sent Done false
20068 6439 Acked Read 4 Done Sent Done false
20069 6440 Acked Read 7 Done Sent Done false
20070 6441 Acked Write 9 Done Sent Done false
20071 6442 Acked Read 6 Done Sent Done false
20072 6443 Acked Write 2 Done Sent Done false
20073 6444 Acked Read 6 Done Sent Done false
20074 6445 Acked Read 8 Done Sent Done false
20075 6446 Acked Write 1 Done Sent Done false
20076 6447 Acked Read 6 Done Sent Done false
20077 6448 Acked Read 7 Done Sent Done false
20078 6449 Acked Read 6 Done Sent Done false
20079 6450 Acked Flush 0 Done Sent Done false
20080 6451 Acked Write 6 Done Sent Done false
20081 6452 Acked Write 10 Done Sent Done false
20082 6453 Acked Read 1 Done Sent Done false
20083 6454 Acked Read 5 Done Sent Done false
20084 6455 Acked Read 4 Done Sent Done false
20085 6456 Acked Write 5 Done Sent Done false
20086 6457 Acked Read 10 Done Sent Done false
20087 6458 Acked Write 6 Done Sent Done false
20088 6459 Acked Write 10 Done Sent Done false
20089 6460 Acked Write 6 Done Sent Done false
20090 6461 Acked Read 3 Done Sent Done false
20091 6462 Acked Write 9 Done Sent Done false
20092 6463 Acked Read 6 Done Sent Done false
20093 6464 Acked Write 10 Done Sent Done false
20094 6465 Acked Read 2 Done Sent Done false
20095 6466 Acked Write 3 Done Sent Done false
20096 6467 Acked Read 10 Done Sent Done false
20097 6468 Acked Flush 0 Done Sent Done false
20098 6469 Acked Read 2 Done Sent Done false
20099 6470 Acked Read 9 Done Sent Done false
20100 6471 Acked Read 5 Done Sent Done false
20101 6472 Acked Read 6 Done Sent Done false
20102 6473 Acked Read 9 Done Sent Done false
20103 6474 Acked Read 9 Done Sent Done false
20104 6475 Acked Flush 0 Done Sent Done false
20105 6476 Acked Read 9 Done Sent Done false
20106 6477 Acked Read 6 Done Sent Done false
20107 6478 Acked Write 9 Done Sent Done false
20108 6479 Acked Write 3 Done Sent Done false
20109 6480 Acked Read 6 Done Sent Done false
20110 6481 Acked Read 3 Done Sent Done false
20111 6482 Acked Read 6 Done Sent Done false
20112 6483 Acked Flush 0 Done Sent Done false
20113 6484 Acked Write 7 Done Sent Done false
20114 6485 Acked Flush 0 Done Sent Done false
20115 6486 Acked Read 9 Done Sent Done false
20116 6487 Acked Write 7 Done Sent Done false
20117 6488 Acked Read 9 Done Sent Done false
20118 6489 Acked Read 10 Done Sent Done false
20119 6490 Acked Write 10 Done Sent Done false
20120 6491 Acked Write 4 Done Sent Done false
20121 6492 Acked Write 10 Done Sent Done false
20122 6493 Acked Read 4 Done Sent Done false
20123 6494 Acked Write 2 Done Sent Done false
20124 6495 Acked Read 10 Done Sent Done false
20125 6496 Acked Write 7 Done Sent Done false
20126 6497 Acked Read 7 Done Sent Done false
20127 6498 Acked Write 8 Done Sent Done false
20128 6499 Acked Write 2 Done Sent Done false
20129 6500 Acked Flush 0 Done Sent Done false
20130 6501 Acked Write 1 Done Sent Done false
20131 6502 Acked Flush 0 Done Sent Done false
20132 6503 Acked Write 8 Done Sent Done false
20133 6504 Acked Read 10 Done Sent Done false
20134 6505 Acked Read 9 Done Sent Done false
20135 6506 Acked Write 3 Done Sent Done false
20136 6507 Acked Read 1 Done Sent Done false
20137 6508 Acked Write 9 Done Sent Done false
20138 6509 Acked Read 9 Done Sent Done false
20139 6510 Acked Read 2 Done Sent Done false
20140 6511 Acked Flush 0 Done Sent Done false
20141 6512 Acked Flush 0 Done Sent Done false
20142 6513 Acked Read 9 Done Sent Done false
20143 6514 Acked Read 1 Done Sent Done false
20144 6515 Acked Write 6 Done Sent Done false
20145 6516 Acked Read 10 Done Sent Done false
20146 6517 Acked Write 3 Done Sent Done false
20147 6518 Acked Read 5 Done Sent Done false
20148 6519 Acked Read 10 Done Sent Done false
20149 6520 Acked Read 6 Done Sent Done false
20150 6521 Acked Flush 0 Done Sent Done false
20151 6522 Acked Write 3 Done Sent Done false
20152 6523 Acked Read 5 Done Sent Done false
20153 6524 Acked Read 4 Done Sent Done false
20154 6525 Acked Flush 0 Done Sent Done false
20155 6526 Acked Write 2 Done Sent Done false
20156 6527 Acked Flush 0 Done Sent Done false
20157 6528 Acked Read 10 Done Sent Done false
20158 6529 Acked Read 10 Done Sent Done false
20159 6530 Acked Read 3 Done Sent Done false
20160 6531 Acked Read 10 Done Sent Done false
20161 6532 Acked Write 8 Done Sent Done false
20162 6533 Acked Read 8 Done Sent Done false
20163 6534 Acked Write 1 Done Sent Done false
20164 6535 Acked Read 2 Done Sent Done false
20165 6536 Acked Write 1 Done Sent Done false
20166 6537 Acked Read 6 Done Sent Done false
20167 6538 Acked Write 1 Done Sent Done false
20168 6539 Acked Write 9 Done Sent Done false
20169 6540 Acked Write 7 Done Sent Done false
20170 6541 Acked Read 8 Done Sent Done false
20171 6542 Acked Write 7 Done Sent Done false
20172 6543 Acked Flush 0 Done Sent Done false
20173 6544 Acked Write 2 Done Sent Done false
20174 6545 Acked Read 6 Done Sent Done false
20175 6546 Acked Write 7 Done Sent Done false
20176 6547 Acked Read 8 Done Sent Done false
20177 6548 Acked Read 1 Done Sent Done false
20178 6549 Acked Read 2 Done Sent Done false
20179 6550 Acked Read 5 Done Sent Done false
20180 6551 Acked Read 3 Done Sent Done false
20181 6552 Acked Write 1 Done Sent Done false
20182 6553 Acked Read 2 Done Sent Done false
20183 6554 Acked Flush 0 Done Sent Done false
20184 6555 Acked Read 6 Done Sent Done false
20185 6556 Acked Read 5 Done Sent Done false
20186 6557 Acked Write 10 Done Sent Done false
20187 6558 Acked Read 7 Done Sent Done false
20188 6559 Acked Read 2 Done Sent Done false
20189 6560 Acked Read 6 Done Sent Done false
20190 6561 Acked Read 9 Done Sent Done false
20191 6562 Acked Read 6 Done Sent Done false
20192 6563 Acked Flush 0 Done Sent Done false
20193 6564 Acked Read 5 Done Sent Done false
20194 6565 Acked Write 7 Done Sent Done false
20195 6566 Acked Read 10 Done Sent Done false
20196 6567 Acked Write 10 Done Sent Done false
20197 6568 Acked Read 7 Done Sent Done false
20198 6569 Acked Write 7 Done Sent Done false
20199 6570 Acked Read 1 Done Sent Done false
20200 6571 Acked Read 4 Done Sent Done false
20201 6572 Acked Write 3 Done Sent Done false
20202 6573 Acked Read 3 Done Sent Done false
20203 6574 Acked Write 1 Done Sent Done false
20204 6575 Acked Read 4 Done Sent Done false
20205 6576 Acked Read 9 Done Sent Done false
20206 6577 Acked Read 7 Done Sent Done false
20207 6578 Acked Flush 0 Done Sent Done false
20208 6579 Acked Read 9 Done Sent Done false
20209 6580 Acked Write 4 Done Sent Done false
20210 6581 Acked Read 2 Done Sent Done false
20211 6582 Acked Write 8 Done Sent Done false
20212 6583 Acked Flush 0 Done Sent Done false
20213 6584 Acked Read 6 Done Sent Done false
20214 6585 Acked Read 3 Done Sent Done false
20215 6586 Acked Write 2 Done Sent Done false
20216 6587 Acked Write 9 Done Sent Done false
20217 6588 Acked Read 6 Done Sent Done false
20218 6589 Acked Read 9 Done Sent Done false
20219 6590 Acked Write 5 Done Sent Done false
20220 6591 Acked Write 3 Done Sent Done false
20221 6592 Acked Write 7 Done Sent Done false
20222 6593 Acked Write 2 Done Sent Done false
20223 6594 Acked Read 3 Done Sent Done false
20224 6595 Acked Write 8 Done Sent Done false
20225 6596 Acked Write 3 Done Sent Done false
20226 6597 Acked Read 2 Done Sent Done false
20227 6598 Acked Write 7 Done Sent Done false
20228 6599 Acked Read 5 Done Sent Done false
20229 6600 Acked Flush 0 Done Sent Done false
20230 6601 Acked Read 10 Done Sent Done false
20231 6602 Acked Write 7 Done Sent Done false
20232 6603 Acked Read 9 Done Sent Done false
20233 6604 Acked Read 4 Done Sent Done false
20234 6605 Acked Write 5 Done Sent Done false
20235 6606 Acked Read 10 Done Sent Done false
20236 6607 Acked Write 9 Done Sent Done false
20237 6608 Acked Read 2 Done Sent Done false
20238 6609 Acked Write 8 Done Sent Done false
20239 6610 Acked Read 1 Done Sent Done false
20240 6611 Acked Read 8 Done Sent Done false
20241 6612 Acked Read 7 Done Sent Done false
20242 6613 Acked Write 4 Done Sent Done false
20243 6614 Acked Read 1 Done Sent Done false
20244 6615 Acked Write 7 Done Sent Done false
20245 6616 Acked Read 1 Done Sent Done false
20246 6617 Acked Read 8 Done Sent Done false
20247 6618 Acked Read 1 Done Sent Done false
20248 6619 Acked Write 3 Done Sent Done false
20249 6620 Acked Write 6 Done Sent Done false
20250 6621 Acked Read 3 Done Sent Done false
20251 6622 Acked Read 1 Done Sent Done false
20252 6623 Acked Write 1 Done Sent Done false
20253 6624 Acked Write 9 Done Sent Done false
20254 6625 Acked Flush 0 Done Sent Done false
20255 6626 Acked Read 1 Done Sent Done false
20256 6627 Acked Read 2 Done Sent Done false
20257 6628 Acked Read 9 Done Sent Done false
20258 6629 Acked Write 1 Done Sent Done false
20259 6630 Acked Read 8 Done Sent Done false
20260 6631 Acked Read 2 Done Sent Done false
20261 6632 Acked Write 4 Done Sent Done false
20262 6633 Acked Read 8 Done Sent Done false
20263 6634 Acked Write 9 Done Sent Done false
20264 6635 Acked Read 8 Done Sent Done false
20265 6636 Acked Write 5 Done Sent Done false
20266 6637 Acked Write 5 Done Sent Done false
20267 6638 Acked Write 6 Done Sent Done false
20268 6639 Acked Read 9 Done Sent Done false
20269 6640 Acked Read 3 Done Sent Done false
20270 6641 Acked Read 3 Done Sent Done false
20271 6642 Acked Read 10 Done Sent Done false
20272 6643 Acked Write 10 Done Sent Done false
20273 6644 Acked Write 10 Done Sent Done false
20274 6645 Acked Read 8 Done Sent Done false
20275 6646 Acked Read 5 Done Sent Done false
20276 6647 Acked Write 4 Done Sent Done false
20277 6648 Acked Write 2 Done Sent Done false
20278 6649 Acked Flush 0 Done Sent Done false
20279 6650 Acked Write 8 Done Sent Done false
20280 6651 Acked Read 1 Done Sent Done false
20281 6652 Acked Write 6 Done Sent Done false
20282 6653 Acked Write 5 Done Sent Done false
20283 6654 Acked Write 3 Done Sent Done false
20284 6655 Acked Read 6 Done Sent Done false
20285 6656 Acked Read 10 Done Sent Done false
20286 6657 Acked Write 1 Done Sent Done false
20287 6658 Acked Write 5 Done Sent Done false
20288 6659 Acked Write 5 Done Sent Done false
20289 6660 Acked Read 6 Done Sent Done false
20290 6661 Acked Flush 0 Done Sent Done false
20291 6662 Acked Read 10 Done Sent Done false
20292 6663 Acked Read 5 Done Sent Done false
20293 6664 Acked Read 2 Done Sent Done false
20294 6665 Acked Read 7 Done Sent Done false
20295 6666 Acked Write 4 Done Sent Done false
20296 6667 Acked Write 7 Done Sent Done false
20297 6668 Acked Write 8 Done Sent Done false
20298 6669 Acked Write 9 Done Sent Done false
20299 6670 Acked Write 3 Done Sent Done false
20300 6671 Acked Read 2 Done Sent Done false
20301 6672 Acked Read 6 Done Sent Done false
20302 6673 Acked Write 8 Done Sent Done false
20303 6674 Acked Read 3 Done Sent Done false
20304 6675 Acked Read 9 Done Sent Done false
20305 6676 Acked Read 10 Done Sent Done false
20306 6677 Acked Read 9 Done Sent Done false
20307 6678 Acked Read 5 Done Sent Done false
20308 6679 Acked Write 4 Done Sent Done false
20309 6680 Acked Read 10 Done Sent Done false
20310 6681 Acked Flush 0 Done Sent Done false
20311 6682 Acked Write 4 Done Sent Done false
20312 6683 Acked Read 2 Done Sent Done false
20313 6684 Acked Read 4 Done Sent Done false
20314 6685 Acked Read 2 Done Sent Done false
20315 6686 Acked Read 5 Done Sent Done false
20316 6687 Acked Write 9 Done Sent Done false
20317 6688 Acked Read 10 Done Sent Done false
20318 6689 Acked Read 6 Done Sent Done false
20319 6690 Acked Read 7 Done Sent Done false
20320 6691 Acked Write 8 Done Sent Done false
20321 6692 Acked Write 9 Done Sent Done false
20322 6693 Acked Write 3 Done Sent Done false
20323 6694 Acked Read 1 Done Sent Done false
20324 6695 Acked Read 2 Done Sent Done false
20325 6696 Acked Write 9 Done Sent Done false
20326 6697 Acked Write 4 Done Sent Done false
20327 6698 Acked Read 1 Done Sent Done false
20328 6699 Acked Read 6 Done Sent Done false
20329 6700 Acked Write 7 Done Sent Done false
20330 6701 Acked Write 1 Done Sent Done false
20331 6702 Acked Write 5 Done Sent Done false
20332 6703 Acked Read 9 Done Sent Done false
20333 6704 Acked Flush 0 Done Sent Done false
20334 6705 Acked Write 4 Done Sent Done false
20335 6706 Acked Write 3 Done Sent Done false
20336 6707 Acked Read 9 Done Sent Done false
20337 6708 Acked Write 10 Done Sent Done false
20338 6709 Acked Read 1 Done Sent Done false
20339 6710 Acked Read 3 Done Sent Done false
20340 6711 Acked Write 6 Done Sent Done false
20341 6712 Acked Write 6 Done Sent Done false
20342 6713 Acked Read 10 Done Sent Done false
20343 6714 Acked Flush 0 Done Sent Done false
20344 6715 Acked Read 9 Done Sent Done false
20345 6716 Acked Flush 0 Done Sent Done false
20346 6717 Acked Read 9 Done Sent Done false
20347 6718 Acked Write 5 Done Sent Done false
20348 6719 Acked Write 7 Done Sent Done false
20349 6720 Acked Read 8 Done Sent Done false
20350 6721 Acked Flush 0 Done Sent Done false
20351 6722 Acked Flush 0 Done Sent Done false
20352 6723 Acked Read 1 Done Sent Done false
20353 6724 Acked Write 1 Done Sent Done false
20354 6725 Acked Read 3 Done Sent Done false
20355 6726 Acked Write 7 Done Sent Done false
20356 6727 Acked Read 8 Done Sent Done false
20357 6728 Acked Read 3 Done Sent Done false
20358 6729 Acked Write 7 Done Sent Done false
20359 6730 Acked Write 3 Done Sent Done false
20360 6731 Acked Write 2 Done Sent Done false
20361 6732 Acked Write 1 Done Sent Done false
20362 6733 Acked Read 1 Done Sent Done false
20363 6734 Acked Write 4 Done Sent Done false
20364 6735 Acked Write 2 Done Sent Done false
20365 STATES DS:0 DS:1 DS:2 TOTAL
20366 Sent 0 301 0 301
20367 Done 301 0 301 602
20368 Skipped 0 0 0 0
20369 Error 0 0 0 0
20370 Last Flush: Some(JobId(6722)) Some(JobId(6434)) Some(JobId(6722))
20371 Downstairs last five completed: 6434 6433 6432 6431 6430
20372 Upstairs last five completed: 6735 6734 6733 6732 6731
20373 CLIENT: Up:0 ds:301 act:2
203742025-01-28T21:58:43.093ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
203752025-01-28T21:58:43.726ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
20376 [8820] got start action
20377 [8820] I am going to restart
20378 Starting downstairs at port 8820
20379 Make output file at "/tmp/dsc/downstairs-8820.txt"
20380 [8820][1] reports Starting
20381 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1221)
20382 [8820][1] reports Running
203832025-01-28T21:58:45.142ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
203842025-01-28T21:58:45.728ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
20385 ----------------------------------------------------------------
20386 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20387 States: Active Offline Active
20388 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20389 6435 Acked Write 2 Done Sent Done false
20390 6436 Acked Read 2 Done Sent Done false
20391 6437 Acked Read 6 Done Sent Done false
20392 6438 Acked Write 9 Done Sent Done false
20393 6439 Acked Read 4 Done Sent Done false
20394 6440 Acked Read 7 Done Sent Done false
20395 6441 Acked Write 9 Done Sent Done false
20396 6442 Acked Read 6 Done Sent Done false
20397 6443 Acked Write 2 Done Sent Done false
20398 6444 Acked Read 6 Done Sent Done false
20399 6445 Acked Read 8 Done Sent Done false
20400 6446 Acked Write 1 Done Sent Done false
20401 6447 Acked Read 6 Done Sent Done false
20402 6448 Acked Read 7 Done Sent Done false
20403 6449 Acked Read 6 Done Sent Done false
20404 6450 Acked Flush 0 Done Sent Done false
20405 6451 Acked Write 6 Done Sent Done false
20406 6452 Acked Write 10 Done Sent Done false
20407 6453 Acked Read 1 Done Sent Done false
20408 6454 Acked Read 5 Done Sent Done false
20409 6455 Acked Read 4 Done Sent Done false
20410 6456 Acked Write 5 Done Sent Done false
20411 6457 Acked Read 10 Done Sent Done false
20412 6458 Acked Write 6 Done Sent Done false
20413 6459 Acked Write 10 Done Sent Done false
20414 6460 Acked Write 6 Done Sent Done false
20415 6461 Acked Read 3 Done Sent Done false
20416 6462 Acked Write 9 Done Sent Done false
20417 6463 Acked Read 6 Done Sent Done false
20418 6464 Acked Write 10 Done Sent Done false
20419 6465 Acked Read 2 Done Sent Done false
20420 6466 Acked Write 3 Done Sent Done false
20421 6467 Acked Read 10 Done Sent Done false
20422 6468 Acked Flush 0 Done Sent Done false
20423 6469 Acked Read 2 Done Sent Done false
20424 6470 Acked Read 9 Done Sent Done false
20425 6471 Acked Read 5 Done Sent Done false
20426 6472 Acked Read 6 Done Sent Done false
20427 6473 Acked Read 9 Done Sent Done false
20428 6474 Acked Read 9 Done Sent Done false
20429 6475 Acked Flush 0 Done Sent Done false
20430 6476 Acked Read 9 Done Sent Done false
20431 6477 Acked Read 6 Done Sent Done false
20432 6478 Acked Write 9 Done Sent Done false
20433 6479 Acked Write 3 Done Sent Done false
20434 6480 Acked Read 6 Done Sent Done false
20435 6481 Acked Read 3 Done Sent Done false
20436 6482 Acked Read 6 Done Sent Done false
20437 6483 Acked Flush 0 Done Sent Done false
20438 6484 Acked Write 7 Done Sent Done false
20439 6485 Acked Flush 0 Done Sent Done false
20440 6486 Acked Read 9 Done Sent Done false
20441 6487 Acked Write 7 Done Sent Done false
20442 6488 Acked Read 9 Done Sent Done false
20443 6489 Acked Read 10 Done Sent Done false
20444 6490 Acked Write 10 Done Sent Done false
20445 6491 Acked Write 4 Done Sent Done false
20446 6492 Acked Write 10 Done Sent Done false
20447 6493 Acked Read 4 Done Sent Done false
20448 6494 Acked Write 2 Done Sent Done false
20449 6495 Acked Read 10 Done Sent Done false
20450 6496 Acked Write 7 Done Sent Done false
20451 6497 Acked Read 7 Done Sent Done false
20452 6498 Acked Write 8 Done Sent Done false
20453 6499 Acked Write 2 Done Sent Done false
20454 6500 Acked Flush 0 Done Sent Done false
20455 6501 Acked Write 1 Done Sent Done false
20456 6502 Acked Flush 0 Done Sent Done false
20457 6503 Acked Write 8 Done Sent Done false
20458 6504 Acked Read 10 Done Sent Done false
20459 6505 Acked Read 9 Done Sent Done false
20460 6506 Acked Write 3 Done Sent Done false
20461 6507 Acked Read 1 Done Sent Done false
20462 6508 Acked Write 9 Done Sent Done false
20463 6509 Acked Read 9 Done Sent Done false
20464 6510 Acked Read 2 Done Sent Done false
20465 6511 Acked Flush 0 Done Sent Done false
20466 6512 Acked Flush 0 Done Sent Done false
20467 6513 Acked Read 9 Done Sent Done false
20468 6514 Acked Read 1 Done Sent Done false
20469 6515 Acked Write 6 Done Sent Done false
20470 6516 Acked Read 10 Done Sent Done false
20471 6517 Acked Write 3 Done Sent Done false
20472 6518 Acked Read 5 Done Sent Done false
20473 6519 Acked Read 10 Done Sent Done false
20474 6520 Acked Read 6 Done Sent Done false
20475 6521 Acked Flush 0 Done Sent Done false
20476 6522 Acked Write 3 Done Sent Done false
20477 6523 Acked Read 5 Done Sent Done false
20478 6524 Acked Read 4 Done Sent Done false
20479 6525 Acked Flush 0 Done Sent Done false
20480 6526 Acked Write 2 Done Sent Done false
20481 6527 Acked Flush 0 Done Sent Done false
20482 6528 Acked Read 10 Done Sent Done false
20483 6529 Acked Read 10 Done Sent Done false
20484 6530 Acked Read 3 Done Sent Done false
20485 6531 Acked Read 10 Done Sent Done false
20486 6532 Acked Write 8 Done Sent Done false
20487 6533 Acked Read 8 Done Sent Done false
20488 6534 Acked Write 1 Done Sent Done false
20489 6535 Acked Read 2 Done Sent Done false
20490 6536 Acked Write 1 Done Sent Done false
20491 6537 Acked Read 6 Done Sent Done false
20492 6538 Acked Write 1 Done Sent Done false
20493 6539 Acked Write 9 Done Sent Done false
20494 6540 Acked Write 7 Done Sent Done false
20495 6541 Acked Read 8 Done Sent Done false
20496 6542 Acked Write 7 Done Sent Done false
20497 6543 Acked Flush 0 Done Sent Done false
20498 6544 Acked Write 2 Done Sent Done false
20499 6545 Acked Read 6 Done Sent Done false
20500 6546 Acked Write 7 Done Sent Done false
20501 6547 Acked Read 8 Done Sent Done false
20502 6548 Acked Read 1 Done Sent Done false
20503 6549 Acked Read 2 Done Sent Done false
20504 6550 Acked Read 5 Done Sent Done false
20505 6551 Acked Read 3 Done Sent Done false
20506 6552 Acked Write 1 Done Sent Done false
20507 6553 Acked Read 2 Done Sent Done false
20508 6554 Acked Flush 0 Done Sent Done false
20509 6555 Acked Read 6 Done Sent Done false
20510 6556 Acked Read 5 Done Sent Done false
20511 6557 Acked Write 10 Done Sent Done false
20512 6558 Acked Read 7 Done Sent Done false
20513 6559 Acked Read 2 Done Sent Done false
20514 6560 Acked Read 6 Done Sent Done false
20515 6561 Acked Read 9 Done Sent Done false
20516 6562 Acked Read 6 Done Sent Done false
20517 6563 Acked Flush 0 Done Sent Done false
20518 6564 Acked Read 5 Done Sent Done false
20519 6565 Acked Write 7 Done Sent Done false
20520 6566 Acked Read 10 Done Sent Done false
20521 6567 Acked Write 10 Done Sent Done false
20522 6568 Acked Read 7 Done Sent Done false
20523 6569 Acked Write 7 Done Sent Done false
20524 6570 Acked Read 1 Done Sent Done false
20525 6571 Acked Read 4 Done Sent Done false
20526 6572 Acked Write 3 Done Sent Done false
20527 6573 Acked Read 3 Done Sent Done false
20528 6574 Acked Write 1 Done Sent Done false
20529 6575 Acked Read 4 Done Sent Done false
20530 6576 Acked Read 9 Done Sent Done false
20531 6577 Acked Read 7 Done Sent Done false
20532 6578 Acked Flush 0 Done Sent Done false
20533 6579 Acked Read 9 Done Sent Done false
20534 6580 Acked Write 4 Done Sent Done false
20535 6581 Acked Read 2 Done Sent Done false
20536 6582 Acked Write 8 Done Sent Done false
20537 6583 Acked Flush 0 Done Sent Done false
20538 6584 Acked Read 6 Done Sent Done false
20539 6585 Acked Read 3 Done Sent Done false
20540 6586 Acked Write 2 Done Sent Done false
20541 6587 Acked Write 9 Done Sent Done false
20542 6588 Acked Read 6 Done Sent Done false
20543 6589 Acked Read 9 Done Sent Done false
20544 6590 Acked Write 5 Done Sent Done false
20545 6591 Acked Write 3 Done Sent Done false
20546 6592 Acked Write 7 Done Sent Done false
20547 6593 Acked Write 2 Done Sent Done false
20548 6594 Acked Read 3 Done Sent Done false
20549 6595 Acked Write 8 Done Sent Done false
20550 6596 Acked Write 3 Done Sent Done false
20551 6597 Acked Read 2 Done Sent Done false
20552 6598 Acked Write 7 Done Sent Done false
20553 6599 Acked Read 5 Done Sent Done false
20554 6600 Acked Flush 0 Done Sent Done false
20555 6601 Acked Read 10 Done Sent Done false
20556 6602 Acked Write 7 Done Sent Done false
20557 6603 Acked Read 9 Done Sent Done false
20558 6604 Acked Read 4 Done Sent Done false
20559 6605 Acked Write 5 Done Sent Done false
20560 6606 Acked Read 10 Done Sent Done false
20561 6607 Acked Write 9 Done Sent Done false
20562 6608 Acked Read 2 Done Sent Done false
20563 6609 Acked Write 8 Done Sent Done false
20564 6610 Acked Read 1 Done Sent Done false
20565 6611 Acked Read 8 Done Sent Done false
20566 6612 Acked Read 7 Done Sent Done false
20567 6613 Acked Write 4 Done Sent Done false
20568 6614 Acked Read 1 Done Sent Done false
20569 6615 Acked Write 7 Done Sent Done false
20570 6616 Acked Read 1 Done Sent Done false
20571 6617 Acked Read 8 Done Sent Done false
20572 6618 Acked Read 1 Done Sent Done false
20573 6619 Acked Write 3 Done Sent Done false
20574 6620 Acked Write 6 Done Sent Done false
20575 6621 Acked Read 3 Done Sent Done false
20576 6622 Acked Read 1 Done Sent Done false
20577 6623 Acked Write 1 Done Sent Done false
20578 6624 Acked Write 9 Done Sent Done false
20579 6625 Acked Flush 0 Done Sent Done false
20580 6626 Acked Read 1 Done Sent Done false
20581 6627 Acked Read 2 Done Sent Done false
20582 6628 Acked Read 9 Done Sent Done false
20583 6629 Acked Write 1 Done Sent Done false
20584 6630 Acked Read 8 Done Sent Done false
20585 6631 Acked Read 2 Done Sent Done false
20586 6632 Acked Write 4 Done Sent Done false
20587 6633 Acked Read 8 Done Sent Done false
20588 6634 Acked Write 9 Done Sent Done false
20589 6635 Acked Read 8 Done Sent Done false
20590 6636 Acked Write 5 Done Sent Done false
20591 6637 Acked Write 5 Done Sent Done false
20592 6638 Acked Write 6 Done Sent Done false
20593 6639 Acked Read 9 Done Sent Done false
20594 6640 Acked Read 3 Done Sent Done false
20595 6641 Acked Read 3 Done Sent Done false
20596 6642 Acked Read 10 Done Sent Done false
20597 6643 Acked Write 10 Done Sent Done false
20598 6644 Acked Write 10 Done Sent Done false
20599 6645 Acked Read 8 Done Sent Done false
20600 6646 Acked Read 5 Done Sent Done false
20601 6647 Acked Write 4 Done Sent Done false
20602 6648 Acked Write 2 Done Sent Done false
20603 6649 Acked Flush 0 Done Sent Done false
20604 6650 Acked Write 8 Done Sent Done false
20605 6651 Acked Read 1 Done Sent Done false
20606 6652 Acked Write 6 Done Sent Done false
20607 6653 Acked Write 5 Done Sent Done false
20608 6654 Acked Write 3 Done Sent Done false
20609 6655 Acked Read 6 Done Sent Done false
20610 6656 Acked Read 10 Done Sent Done false
20611 6657 Acked Write 1 Done Sent Done false
20612 6658 Acked Write 5 Done Sent Done false
20613 6659 Acked Write 5 Done Sent Done false
20614 6660 Acked Read 6 Done Sent Done false
20615 6661 Acked Flush 0 Done Sent Done false
20616 6662 Acked Read 10 Done Sent Done false
20617 6663 Acked Read 5 Done Sent Done false
20618 6664 Acked Read 2 Done Sent Done false
20619 6665 Acked Read 7 Done Sent Done false
20620 6666 Acked Write 4 Done Sent Done false
20621 6667 Acked Write 7 Done Sent Done false
20622 6668 Acked Write 8 Done Sent Done false
20623 6669 Acked Write 9 Done Sent Done false
20624 6670 Acked Write 3 Done Sent Done false
20625 6671 Acked Read 2 Done Sent Done false
20626 6672 Acked Read 6 Done Sent Done false
20627 6673 Acked Write 8 Done Sent Done false
20628 6674 Acked Read 3 Done Sent Done false
20629 6675 Acked Read 9 Done Sent Done false
20630 6676 Acked Read 10 Done Sent Done false
20631 6677 Acked Read 9 Done Sent Done false
20632 6678 Acked Read 5 Done Sent Done false
20633 6679 Acked Write 4 Done Sent Done false
20634 6680 Acked Read 10 Done Sent Done false
20635 6681 Acked Flush 0 Done Sent Done false
20636 6682 Acked Write 4 Done Sent Done false
20637 6683 Acked Read 2 Done Sent Done false
20638 6684 Acked Read 4 Done Sent Done false
20639 6685 Acked Read 2 Done Sent Done false
20640 6686 Acked Read 5 Done Sent Done false
20641 6687 Acked Write 9 Done Sent Done false
20642 6688 Acked Read 10 Done Sent Done false
20643 6689 Acked Read 6 Done Sent Done false
20644 6690 Acked Read 7 Done Sent Done false
20645 6691 Acked Write 8 Done Sent Done false
20646 6692 Acked Write 9 Done Sent Done false
20647 6693 Acked Write 3 Done Sent Done false
20648 6694 Acked Read 1 Done Sent Done false
20649 6695 Acked Read 2 Done Sent Done false
20650 6696 Acked Write 9 Done Sent Done false
20651 6697 Acked Write 4 Done Sent Done false
20652 6698 Acked Read 1 Done Sent Done false
20653 6699 Acked Read 6 Done Sent Done false
20654 6700 Acked Write 7 Done Sent Done false
20655 6701 Acked Write 1 Done Sent Done false
20656 6702 Acked Write 5 Done Sent Done false
20657 6703 Acked Read 9 Done Sent Done false
20658 6704 Acked Flush 0 Done Sent Done false
20659 6705 Acked Write 4 Done Sent Done false
20660 6706 Acked Write 3 Done Sent Done false
20661 6707 Acked Read 9 Done Sent Done false
20662 6708 Acked Write 10 Done Sent Done false
20663 6709 Acked Read 1 Done Sent Done false
20664 6710 Acked Read 3 Done Sent Done false
20665 6711 Acked Write 6 Done Sent Done false
20666 6712 Acked Write 6 Done Sent Done false
20667 6713 Acked Read 10 Done Sent Done false
20668 6714 Acked Flush 0 Done Sent Done false
20669 6715 Acked Read 9 Done Sent Done false
20670 6716 Acked Flush 0 Done Sent Done false
20671 6717 Acked Read 9 Done Sent Done false
20672 6718 Acked Write 5 Done Sent Done false
20673 6719 Acked Write 7 Done Sent Done false
20674 6720 Acked Read 8 Done Sent Done false
20675 6721 Acked Flush 0 Done Sent Done false
20676 6722 Acked Flush 0 Done Sent Done false
20677 6723 Acked Read 1 Done Sent Done false
20678 6724 Acked Write 1 Done Sent Done false
20679 6725 Acked Read 3 Done Sent Done false
20680 6726 Acked Write 7 Done Sent Done false
20681 6727 Acked Read 8 Done Sent Done false
20682 6728 Acked Read 3 Done Sent Done false
20683 6729 Acked Write 7 Done Sent Done false
20684 6730 Acked Write 3 Done Sent Done false
20685 6731 Acked Write 2 Done Sent Done false
20686 6732 Acked Write 1 Done Sent Done false
20687 6733 Acked Read 1 Done Sent Done false
20688 6734 Acked Write 4 Done Sent Done false
20689 6735 Acked Write 2 Done Sent Done false
20690 STATES DS:0 DS:1 DS:2 TOTAL
20691 Sent 0 301 0 301
20692 Done 301 0 301 602
20693 Skipped 0 0 0 0
20694 Error 0 0 0 0
20695 Last Flush: Some(JobId(6722)) Some(JobId(6434)) Some(JobId(6722))
20696 Downstairs last five completed: 6434 6433 6432 6431 6430
20697 Upstairs last five completed: 6735 6734 6733 6732 6731
20698 CLIENT: Up:0 ds:301 act:2
206992025-01-28T21:58:47.095ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
207002025-01-28T21:58:47.730ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
207012025-01-28T21:58:48.345ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
207022025-01-28T21:58:48.345ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53656) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
207032025-01-28T21:58:48.346ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
207042025-01-28T21:58:48.347ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
207052025-01-28T21:58:48.347ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
207062025-01-28T21:58:48.347ZINFOcrucible: send last flush ID to this DS: Some(JobId(6434)) = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
207072025-01-28T21:58:48.347ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
207082025-01-28T21:58:48.347ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(6434)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
207092025-01-28T21:58:48.347ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(6434)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
20710 ----------------------------------------------------------------
20711 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20712 Downstairs last five completed: 6736 6735 6734 6733 6732
20713 Upstairs last five completed: 6736 6735 6734 6733 6732
20714 CLIENT: Up:0 ds:0 act:3
20715 Replay: All jobs finished
20716 Main task has work to do, go find it
20717 got dsc Stop(2)
20718 stop 2
20719 Jan 28 21:58:50.596 INFO request completed, latency_us: 376, response_code: 204, uri: /stop/cid/2[8830] Got stop action so:false kr:false
20720 , method: POST, req_id: 80f7ee97-42ef-4ada-a3a1-5af9bcecc3c8, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
20721 Jan 28 21:58:50.597 INFO request completed, latency_us: 165, response_code: 200, uri: /state/cid/2, {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:58:50.597647254Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
20722 method: GET, req_id: 570049d3-65ae-4cd8-858c-5f0315348db3, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:58:50.597729068Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
20723 
207242025-01-28T21:58:50.597ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
207252025-01-28T21:58:50.597ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
20726 [8830] Exited with: ExitStatus(unix_wait_status(9))
20727 [8830][2] reports Exit
20728 Jan 28 21:58:54.600 INFO request completed, latency_us: 251, response_code: 200, uri: /state/cid/2, method: GET, req_id: e12d5d66-8b60-4d85-ab6f-56f3d7db9f2b, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
20729 001/300 Read block 2633 len 40960
20730 002/300 Read block 2309 len 20480
20731 003/300 Read block 2094 len 32768
20732 004/300 Read block 1662 len 32768
20733 005/300 Read block 1317 len 32768
20734 006/300 Read block 2796 len 32768
20735 007/300 Write block 2274 len 24576 data: 4 5 6 5 5 4
20736 008/300 Read block 636 len 40960
20737 009/300 Write block 2800 len 8192 data: 6 6
20738 010/300 Write block 2684 len 8192 data: 4 4
20739 011/300 Read block 288 len 8192
20740 012/300 Write block 898 len 16384 data: 2 3 5 6
20741 013/300 Read block 1330 len 4096
20742 014/300 Read block 2622 len 40960
20743 015/300 Write block 4904 len 4096 data: 2
20744 016/300 Read block 4013 len 12288
20745 017/300 Read block 2306 len 4096
20746 018/300 Write block 699 len 20480 data: 3 3 3 3 3
20747 019/300 Read block 3575 len 40960
20748 020/300 Flush
20749 021/300 Write block 3297 len 32768 data: 5 5 5 4 5 5 4 5
20750 022/300 Write block 3324 len 4096 data: 5
20751 023/300 Write block 1308 len 24576 data: 4 4 4 4 5 5
20752 024/300 Write block 2379 len 4096 data: 5
20753 025/300 Write block 2102 len 8192 data: 5 6
20754 026/300 Write block 289 len 28672 data: 4 4 3 2 2 2 2
20755 027/300 Read block 2026 len 24576
20756 028/300 Read block 3720 len 28672
20757 029/300 Read block 3427 len 20480
20758 030/300 Write block 715 len 20480 data: 4 5 2 3 3
20759 031/300 Write block 4483 len 24576 data: 9 9 8 11 8 7
20760 032/300 Read block 4015 len 8192
20761 033/300 Read block 4194 len 16384
20762 034/300 Read block 4668 len 32768
20763 035/300 Read block 2638 len 40960
20764 036/300 Write block 4813 len 28672 data: 4 3 3 3 4 5 5
20765 037/300 Read block 4432 len 4096
20766 038/300 Read block 992 len 40960
20767 039/300 Read block 2406 len 20480
20768 040/300 Read block 2817 len 4096
20769 041/300 Read block 2887 len 12288
20770 042/300 Read block 883 len 8192
20771 043/300 Flush
20772 044/300 Read block 12 len 8192
20773 045/300 Write block 3001 len 32768 data: 5 4 4 4 4 4 3 3
20774 046/300 Write block 4526 len 8192 data: 8 7
20775 047/300 Write block 3024 len 8192 data: 3 4
20776 048/300 Read block 3119 len 4096
20777 049/300 Read block 4391 len 24576
20778 050/300 Flush
20779 051/300 Write block 488 len 36864 data: 3 3 2 4 3 3 3 3 3
20780 052/300 Write block 4071 len 16384 data: 3 4 5 7
20781 053/300 Read block 1884 len 32768
20782 054/300 Write block 1586 len 8192 data: 4 4
20783 055/300 Write block 4299 len 36864 data: 4 4 3 3 3 2 2 3 4
20784 056/300 Read block 2833 len 40960
20785 057/300 Read block 3739 len 4096
20786 058/300 Write block 107 len 20480 data: 2 2 2 3 3
20787 059/300 Write block 1142 len 40960 data: 6 6 7 7 6 6 5 6 6 6
20788 060/300 Write block 943 len 8192 data: 6 5
20789 061/300 Read block 1754 len 36864
20790 062/300 Read block 3268 len 28672
20791 063/300 Write block 703 len 36864 data: 4 3 4 4 5 4 4 4 4
20792 064/300 Flush
20793 065/300 Write block 1291 len 8192 data: 6 5
20794 066/300 Write block 4678 len 32768 data: 5 7 7 7 7 7 7 7
20795 067/300 Read block 1404 len 12288
20796 068/300 Write block 1410 len 32768 data: 7 6 7 6 6 4 3 2
20797 069/300 Read block 749 len 24576
20798 070/300 Read block 4073 len 36864
20799 071/300 Write block 1738 len 24576 data: 4 5 4 3 3 3
20800 072/300 Write block 3910 len 24576 data: 7 6 6 5 4 3
20801 073/300 Write block 2338 len 28672 data: 4 5 5 4 3 5 5
20802 074/300 Write block 330 len 32768 data: 5 4 4 5 5 2 2 2
20803 075/300 Flush
20804 076/300 Read block 681 len 12288
20805 077/300 Write block 4594 len 24576 data: 3 2 2 3 4 5
20806 078/300 Write block 2285 len 16384 data: 6 6 7 7
20807 079/300 Read block 3871 len 12288
20808 080/300 Flush
20809 081/300 Write block 2628 len 36864 data: 3 5 4 4 4 6 5 6 6
20810 082/300 Flush
20811 083/300 Read block 4441 len 36864
20812 084/300 Read block 329 len 8192
20813 085/300 Read block 995 len 28672
20814 086/300 Read block 2617 len 16384
20815 087/300 Read block 954 len 20480
20816 088/300 Read block 212 len 8192
20817 089/300 Read block 2405 len 32768
20818 090/300 Read block 970 len 16384
20819 091/300 Write block 1491 len 28672 data: 5 3 3 2 2 3 2
20820 092/300 Read block 3980 len 40960
20821 093/300 Write block 421 len 28672 data: 6 5 5 5 5 5 3
20822 094/300 Read block 321 len 32768
20823 095/300 Read block 2022 len 24576
20824 096/300 Write block 1545 len 40960 data: 2 2 2 2 2 2 2 3 3 3
20825 097/300 Write block 4929 len 28672 data: 4 6 5 5 6 6 6
20826 098/300 Write block 494 len 16384 data: 4 4 4 2
20827 099/300 Read block 935 len 8192
20828 100/300 Read block 4666 len 20480
20829 101/300 Read block 2177 len 28672
20830 102/300 Write block 4399 len 8192 data: 2 3
20831 103/300 Write block 2243 len 16384 data: 3 5 6 6
20832 104/300 Read block 2669 len 4096
20833 105/300 Flush
20834 106/300 Write block 4226 len 28672 data: 3 3 5 5 6 6 7
20835 107/300 Read block 3854 len 8192
20836 108/300 Write block 2449 len 36864 data: 3 5 6 5 5 6 6 6 5
20837 109/300 Read block 4961 len 36864
20838 110/300 Write block 8 len 12288 data: 6 6 6
20839 111/300 Write block 3980 len 24576 data: 6 7 7 7 7 4
20840 112/300 Write block 710 len 20480 data: 5 5 4 6 5
20841 113/300 Write block 1768 len 12288 data: 4 5 5
20842 114/300 Read block 4972 len 20480
20843 115/300 Read block 1183 len 36864
20844 116/300 Flush
20845 117/300 Read block 1796 len 4096
20846 118/300 Read block 3972 len 28672
20847 119/300 Read block 1976 len 8192
20848 120/300 Write block 730 len 4096 data: 7
20849 121/300 Write block 356 len 4096 data: 5
20850 122/300 Read block 979 len 24576
20851 123/300 Write block 1562 len 28672 data: 6 4 4 4 3 3 3
20852 124/300 Write block 1949 len 32768 data: 4 4 4 4 4 5 6 7
20853 125/300 Flush
20854 126/300 Write block 2896 len 16384 data: 3 5 5 7
20855 127/300 Read block 391 len 24576
20856 128/300 Read block 37 len 32768
20857 129/300 Read block 1749 len 36864
20858 130/300 Flush
20859 131/300 Read block 1598 len 20480
20860 132/300 Write block 2476 len 40960 data: 6 8 6 5 4 5 4 4 3 3
20861 133/300 Flush
20862 134/300 Read block 4793 len 24576
20863 135/300 Write block 2763 len 24576 data: 3 4 4 4 4 4
20864 136/300 Write block 711 len 36864 data: 6 5 7 6 5 6 3 4 4
20865 137/300 Read block 541 len 40960
20866 138/300 Write block 4333 len 12288 data: 4 5 5
20867 139/300 Read block 3091 len 4096
20868 140/300 Read block 246 len 12288
20869 141/300 Write block 4396 len 12288 data: 3 3 3
20870 142/300 Write block 2563 len 16384 data: 4 5 5 4
20871 143/300 Read block 578 len 36864
20872 144/300 Write block 3728 len 8192 data: 10 7
20873 145/300 Write block 855 len 12288 data: 4 4 4
20874 146/300 Flush
20875 147/300 Read block 2764 len 8192
20876 148/300 Read block 4653 len 16384
20877 149/300 Write block 4086 len 28672 data: 5 5 4 6 7 8 8
20878 150/300 Write block 4069 len 12288 data: 4 4 4
20879 151/300 Write block 4540 len 20480 data: 4 4 5 4 6
20880 152/300 Write block 200 len 12288 data: 4 4 4
20881 153/300 Write block 4688 len 28672 data: 5 5 6 6 5 6 7
20882 154/300 Read block 4044 len 12288
20883 155/300 Write block 3928 len 20480 data: 7 5 5 4 3
20884 156/300 Write block 335 len 12288 data: 3 3 3
20885 157/300 Write block 3858 len 4096 data: 2
20886 158/300 Read block 555 len 36864
20887 159/300 Flush
20888 160/300 Read block 1035 len 8192
20889 161/300 Write block 2970 len 8192 data: 5 6
20890 162/300 Write block 72 len 20480 data: 5 4 5 4 4
20891 163/300 Read block 1324 len 24576
20892 164/300 Flush
20893 165/300 Read block 3938 len 36864
20894 166/300 Flush
20895 167/300 Write block 3101 len 24576 data: 6 6 6 6 5 6
20896 168/300 Flush
20897 169/300 Read block 1061 len 4096
20898 170/300 Write block 3133 len 32768 data: 5 5 5 5 6 6 4 5
20899 171/300 Write block 4737 len 24576 data: 5 6 5 5 6 5
20900 172/300 Write block 3703 len 20480 data: 4 3 3 3 2
20901 173/300 Write block 2561 len 8192 data: 4 4
20902 174/300 Read block 1337 len 16384
20903 175/300 Read block 4546 len 28672
20904 176/300 Read block 1053 len 32768
20905 177/300 Flush
20906 178/300 Read block 4017 len 40960
20907 179/300 Read block 3873 len 8192
20908 180/300 Flush
20909 181/300 Read block 619 len 4096
20910 182/300 Write block 493 len 32768 data: 4 5 5 5 3 4 4 5
20911 183/300 Read block 321 len 36864
20912 184/300 Write block 609 len 8192 data: 5 5
20913 185/300 Read block 4844 len 8192
20914 186/300 Write block 2413 len 16384 data: 5 5 4 5
20915 187/300 Read block 2850 len 8192
20916 188/300 Write block 1427 len 24576 data: 3 3 3 4 5 6
20917 189/300 Read block 867 len 28672
20918 190/300 Read block 2757 len 28672
20919 191/300 Write block 239 len 32768 data: 5 6 7 7 7 6 6 6
20920 192/300 Write block 2711 len 16384 data: 6 4 3 3
20921 193/300 Read block 463 len 24576
20922 194/300 Read block 501 len 32768
20923 195/300 Read block 335 len 40960
20924 196/300 Read block 4751 len 32768
20925 197/300 Write block 4932 len 16384 data: 6 7 7 7
20926 198/300 Write block 4033 len 4096 data: 4
20927 199/300 Write block 341 len 12288 data: 5 5 4
20928 200/300 Read block 3123 len 4096
20929 201/300 Read block 4898 len 8192
20930 202/300 Read block 2497 len 12288
20931 203/300 Write block 578 len 28672 data: 3 3 2 2 3 3 4
20932 204/300 Read block 2015 len 20480
20933 205/300 Read block 752 len 32768
20934 206/300 Write block 4394 len 36864 data: 3 3 4 4 4 3 4 5 7
20935 207/300 Read block 2546 len 40960
20936 208/300 Write block 1110 len 32768 data: 5 4 5 3 2 2 3 4
20937 209/300 Read block 4487 len 36864
20938 210/300 Read block 212 len 32768
20939 211/300 Read block 386 len 32768
20940 212/300 Read block 1619 len 12288
20941 213/300 Read block 930 len 20480
20942 214/300 Read block 4161 len 8192
20943 215/300 Flush
20944 216/300 Write block 3505 len 36864 data: 3 3 3 2 3 4 3 3 3
20945 217/300 Flush
20946 218/300 Read block 600 len 8192
20947 219/300 Read block 894 len 20480
20948 220/300 Read block 4969 len 24576
20949 221/300 Read block 3600 len 20480
20950 222/300 Write block 4523 len 40960 data: 7 8 8 9 8 4 4 4 3 4
20951 223/300 Write block 4167 len 36864 data: 9 6 4 4 6 6 4 4 4
20952 224/300 Read block 4954 len 4096
20953 225/300 Write block 160 len 28672 data: 5 5 3 2 2 2 2
20954 226/300 Write block 945 len 4096 data: 5
20955 227/300 Write block 1756 len 12288 data: 2 2 2
20956 228/300 Read block 3702 len 20480
20957 229/300 Write block 1560 len 16384 data: 6 5 7 5
20958 230/300 Read block 3843 len 28672
20959 231/300 Write block 3114 len 40960 data: 5 5 5 5 4 4 4 6 5 4
20960 232/300 Read block 1530 len 16384
20961 233/300 Write block 188 len 16384 data: 3 3 3 3
20962 234/300 Write block 1445 len 8192 data: 3 4
20963 235/300 Write block 1139 len 24576 data: 4 5 6 7 7 8
20964 236/300 Read block 138 len 16384
20965 237/300 Read block 4093 len 16384
20966 238/300 Write block 187 len 12288 data: 3 4 4
20967 239/300 Flush
20968 240/300 Write block 1301 len 40960 data: 5 4 4 5 5 4 3 5 5 5
20969 241/300 Write block 1436 len 32768 data: 5 5 5 4 3 2 3 3
20970 242/300 Write block 4700 len 4096 data: 3
20971 243/300 Flush
20972 244/300 Read block 351 len 36864
20973 245/300 Read block 3026 len 32768
20974 246/300 Read block 1899 len 16384
20975 247/300 Write block 2344 len 4096 data: 6
20976 248/300 Write block 4325 len 40960 data: 5 5 5 6 6 6 6 5 5 6
20977 249/300 Read block 918 len 16384
20978 250/300 Write block 4373 len 16384 data: 6 6 6 5
20979 251/300 Flush
20980 252/300 Write block 2233 len 8192 data: 3 3
20981 253/300 Read block 1283 len 12288
20982 254/300 Flush
20983 255/300 Write block 4708 len 24576 data: 5 5 5 5 4 2
20984 256/300 Read block 2537 len 40960
20985 257/300 Flush
20986 258/300 Read block 3141 len 16384
20987 259/300 Read block 2316 len 4096
20988 260/300 Write block 2226 len 24576 data: 3 3 4 3 3 3
20989 261/300 Read block 508 len 12288
20990 262/300 Write block 404 len 8192 data: 5 3
20991 263/300 Read block 1074 len 8192
20992 264/300 Write block 7 len 4096 data: 7
20993 265/300 Read block 794 len 20480
20994 266/300 Write block 4086 len 12288 data: 6 6 5
20995 267/300 Write block 1047 len 20480 data: 9 8 6 6 6
20996 268/300 Flush
20997 269/300 Read block 609 len 24576
20998 270/300 Flush
20999 271/300 Read block 1115 len 40960
21000 272/300 Read block 4821 len 20480
21001 273/300 Write block 3493 len 20480 data: 4 4 4 4 2
21002 274/300 Write block 2368 len 16384 data: 6 6 7 7
21003 275/300 Flush
21004 276/300 Read block 1205 len 28672
21005 277/300 Write block 1676 len 16384 data: 4 3 3 3
21006 278/300 Read block 2983 len 40960
21007 279/300 Write block 2342 len 4096 data: 4
21008 280/300 Write block 3714 len 24576 data: 7 6 4 4 5 6
21009 281/300 Write block 1858 len 40960 data: 4 5 4 5 4 4 4 4 2 3
21010 282/300 Read block 997 len 12288
21011 283/300 Flush
21012 284/300 Write block 214 len 40960 data: 3 3 3 4 3 4 4 3 4 4
21013 285/300 Read block 3536 len 16384
21014 286/300 Read block 3498 len 36864
21015 287/300 Read block 99 len 8192
21016 288/300 Write block 3753 len 40960 data: 5 5 5 7 7 7 6 7 5 5
21017 289/300 Write block 3023 len 40960 data: 3 4 5 4 2 3 2 2 3 3
21018 290/300 Write block 2666 len 4096 data: 4
21019 291/300 Write block 4208 len 24576 data: 4 4 4 3 3 2
21020 292/300 Read block 2644 len 12288
21021 293/300 Write block 363 len 4096 data: 4
21022 294/300 Write block 2270 len 36864 data: 3 3 3 4 5 6 7 6 6
21023 295/300 Write block 4205 len 8192 data: 4 4
21024 296/300 Write block 2039 len 12288 data: 4 4 3
21025 297/300 Write block 2870 len 36864 data: 4 4 4 4 4 4 6 4 4
21026 298/300 Write block 41 len 32768 data: 5 5 6 4 5 5 4 4
21027 299/300 Read block 519 len 32768
21028 300/300 Read block 1837 len 12288
21029 301/300 Read block 4630 len 40960
21030 Main task has work to do, go find it
21031 got dsc Start(2)
21032 start 2
21033 Jan 28 21:58:54.830 INFO request completed, latency_us: 305, response_code: 204, uri: /start/cid/2, method: POST, req_id: e759ed78-2403-4f66-bdec-36ec45d16596, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
21034 Replay: started 2, returned:Ok(())
21035 ----------------------------------------------------------------
21036 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21037 States: Active Active Offline
21038 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21039 6737 Acked Read 10 Done Done Sent false
21040 6738 Acked Read 5 Done Done Sent false
21041 6739 Acked Read 8 Done Done Sent false
21042 6740 Acked Read 8 Done Done Sent false
21043 6741 Acked Read 8 Done Done Sent false
21044 6742 Acked Read 8 Done Done Sent false
21045 6743 Acked Write 6 Done Done Sent false
21046 6744 Acked Read 10 Done Done Sent false
21047 6745 Acked Write 2 Done Done Sent false
21048 6746 Acked Write 2 Done Done Sent false
21049 6747 Acked Read 2 Done Done Sent false
21050 6748 Acked Write 4 Done Done Sent false
21051 6749 Acked Read 1 Done Done Sent false
21052 6750 Acked Read 10 Done Done Sent false
21053 6751 Acked Write 1 Done Done Sent false
21054 6752 Acked Read 3 Done Done Sent false
21055 6753 Acked Read 1 Done Done Sent false
21056 6754 Acked Write 5 Done Done Sent false
21057 6755 Acked Read 10 Done Done Sent false
21058 6756 Acked Flush 0 Done Done Sent false
21059 6757 Acked Write 8 Done Done Sent false
21060 6758 Acked Write 1 Done Done Sent false
21061 6759 Acked Write 6 Done Done Sent false
21062 6760 Acked Write 1 Done Done Sent false
21063 6761 Acked Write 2 Done Done Sent false
21064 6762 Acked Write 7 Done Done Sent false
21065 6763 Acked Read 6 Done Done Sent false
21066 6764 Acked Read 7 Done Done Sent false
21067 6765 Acked Read 5 Done Done Sent false
21068 6766 Acked Write 5 Done Done Sent false
21069 6767 Acked Write 6 Done Done Sent false
21070 6768 Acked Read 2 Done Done Sent false
21071 6769 Acked Read 4 Done Done Sent false
21072 6770 Acked Read 8 Done Done Sent false
21073 6771 Acked Read 10 Done Done Sent false
21074 6772 Acked Write 7 Done Done Sent false
21075 6773 Acked Read 1 Done Done Sent false
21076 6774 Acked Read 10 Done Done Sent false
21077 6775 Acked Read 5 Done Done Sent false
21078 6776 Acked Read 1 Done Done Sent false
21079 6777 Acked Read 3 Done Done Sent false
21080 6778 Acked Read 2 Done Done Sent false
21081 6779 Acked Flush 0 Done Done Sent false
21082 6780 Acked Read 2 Done Done Sent false
21083 6781 Acked Write 8 Done Done Sent false
21084 6782 Acked Write 2 Done Done Sent false
21085 6783 Acked Write 2 Done Done Sent false
21086 6784 Acked Read 1 Done Done Sent false
21087 6785 Acked Read 6 Done Done Sent false
21088 6786 Acked Flush 0 Done Done Sent false
21089 6787 Acked Write 9 Done Done Sent false
21090 6788 Acked Write 4 Done Done Sent false
21091 6789 Acked Read 8 Done Done Sent false
21092 6790 Acked Write 2 Done Done Sent false
21093 6791 Acked Write 9 Done Done Sent false
21094 6792 Acked Read 10 Done Done Sent false
21095 6793 Acked Read 1 Done Done Sent false
21096 6794 Acked Write 5 Done Done Sent false
21097 6795 Acked Write 10 Done Done Sent false
21098 6796 Acked Write 2 Done Done Sent false
21099 6797 Acked Read 9 Done Done Sent false
21100 6798 Acked Read 7 Done Done Sent false
21101 6799 Acked Write 9 Done Done Sent false
21102 6800 Acked Flush 0 Done Done Sent false
21103 6801 Acked Write 2 Done Done Sent false
21104 6802 Acked Write 8 Done Done Sent false
21105 6803 Acked Read 3 Done Done Sent false
21106 6804 Acked Write 8 Done Done Sent false
21107 6805 Acked Read 6 Done Done Sent false
21108 6806 Acked Read 9 Done Done Sent false
21109 6807 Acked Write 6 Done Done Sent false
21110 6808 Acked Write 6 Done Done Sent false
21111 6809 Acked Write 7 Done Done Sent false
21112 6810 Acked Write 8 Done Done Sent false
21113 6811 Acked Flush 0 Done Done Sent false
21114 6812 Acked Read 3 Done Done Sent false
21115 6813 Acked Write 6 Done Done Sent false
21116 6814 Acked Write 4 Done Done Sent false
21117 6815 Acked Read 3 Done Done Sent false
21118 6816 Acked Flush 0 Done Done Sent false
21119 6817 Acked Write 9 Done Done Sent false
21120 6818 Acked Flush 0 Done Done Sent false
21121 6819 Acked Read 9 Done Done Sent false
21122 6820 Acked Read 2 Done Done Sent false
21123 6821 Acked Read 7 Done Done Sent false
21124 6822 Acked Read 4 Done Done Sent false
21125 6823 Acked Read 5 Done Done Sent false
21126 6824 Acked Read 2 Done Done Sent false
21127 6825 Acked Read 8 Done Done Sent false
21128 6826 Acked Read 4 Done Done Sent false
21129 6827 Acked Write 7 Done Done Sent false
21130 6828 Acked Read 10 Done Done Sent false
21131 6829 Acked Write 7 Done Done Sent false
21132 6830 Acked Read 8 Done Done Sent false
21133 6831 Acked Read 6 Done Done Sent false
21134 6832 Acked Write 10 Done Done Sent false
21135 6833 Acked Write 7 Done Done Sent false
21136 6834 Acked Write 4 Done Done Sent false
21137 6835 Acked Read 2 Done Done Sent false
21138 6836 Acked Read 5 Done Done Sent false
21139 6837 Acked Read 7 Done Done Sent false
21140 6838 Acked Write 2 Done Done Sent false
21141 6839 Acked Write 4 Done Done Sent false
21142 6840 Acked Read 1 Done Done Sent false
21143 6841 Acked Flush 0 Done Done Sent false
21144 6842 Acked Write 7 Done Done Sent false
21145 6843 Acked Read 2 Done Done Sent false
21146 6844 Acked Write 9 Done Done Sent false
21147 6845 Acked Read 9 Done Done Sent false
21148 6846 Acked Write 3 Done Done Sent false
21149 6847 Acked Write 6 Done Done Sent false
21150 6848 Acked Write 5 Done Done Sent false
21151 6849 Acked Write 3 Done Done Sent false
21152 6850 Acked Read 5 Done Done Sent false
21153 6851 Acked Read 9 Done Done Sent false
21154 6852 Acked Flush 0 Done Done Sent false
21155 6853 Acked Read 1 Done Done Sent false
21156 6854 Acked Read 7 Done Done Sent false
21157 6855 Acked Read 2 Done Done Sent false
21158 6856 Acked Write 1 Done Done Sent false
21159 6857 Acked Write 1 Done Done Sent false
21160 6858 Acked Read 6 Done Done Sent false
21161 6859 Acked Write 7 Done Done Sent false
21162 6860 Acked Write 8 Done Done Sent false
21163 6861 Acked Flush 0 Done Done Sent false
21164 6862 Acked Write 4 Done Done Sent false
21165 6863 Acked Read 6 Done Done Sent false
21166 6864 Acked Read 8 Done Done Sent false
21167 6865 Acked Read 9 Done Done Sent false
21168 6866 Acked Flush 0 Done Done Sent false
21169 6867 Acked Read 5 Done Done Sent false
21170 6868 Acked Write 10 Done Done Sent false
21171 6869 Acked Flush 0 Done Done Sent false
21172 6870 Acked Read 6 Done Done Sent false
21173 6871 Acked Write 6 Done Done Sent false
21174 6872 Acked Write 9 Done Done Sent false
21175 6873 Acked Read 10 Done Done Sent false
21176 6874 Acked Write 3 Done Done Sent false
21177 6875 Acked Read 1 Done Done Sent false
21178 6876 Acked Read 3 Done Done Sent false
21179 6877 Acked Write 3 Done Done Sent false
21180 6878 Acked Write 4 Done Done Sent false
21181 6879 Acked Read 9 Done Done Sent false
21182 6880 Acked Write 2 Done Done Sent false
21183 6881 Acked Write 3 Done Done Sent false
21184 6882 Acked Flush 0 Done Done Sent false
21185 6883 Acked Read 2 Done Done Sent false
21186 6884 Acked Read 4 Done Done Sent false
21187 6885 Acked Write 7 Done Done Sent false
21188 6886 Acked Write 3 Done Done Sent false
21189 6887 Acked Write 5 Done Done Sent false
21190 6888 Acked Write 3 Done Done Sent false
21191 6889 Acked Write 7 Done Done Sent false
21192 6890 Acked Read 3 Done Done Sent false
21193 6891 Acked Write 5 Done Done Sent false
21194 6892 Acked Write 3 Done Done Sent false
21195 6893 Acked Write 1 Done Done Sent false
21196 6894 Acked Read 9 Done Done Sent false
21197 6895 Acked Flush 0 Done Done Sent false
21198 6896 Acked Read 2 Done Done Sent false
21199 6897 Acked Write 2 Done Done Sent false
21200 6898 Acked Write 5 Done Done Sent false
21201 6899 Acked Read 6 Done Done Sent false
21202 6900 Acked Flush 0 Done Done Sent false
21203 6901 Acked Read 9 Done Done Sent false
21204 6902 Acked Flush 0 Done Done Sent false
21205 6903 Acked Write 6 Done Done Sent false
21206 6904 Acked Flush 0 Done Done Sent false
21207 6905 Acked Read 1 Done Done Sent false
21208 6906 Acked Write 8 Done Done Sent false
21209 6907 Acked Write 6 Done Done Sent false
21210 6908 Acked Write 5 Done Done Sent false
21211 6909 Acked Write 2 Done Done Sent false
21212 6910 Acked Read 4 Done Done Sent false
21213 6911 Acked Read 7 Done Done Sent false
21214 6912 Acked Read 8 Done Done Sent false
21215 6913 Acked Flush 0 Done Done Sent false
21216 6914 Acked Read 10 Done Done Sent false
21217 6915 Acked Read 2 Done Done Sent false
21218 6916 Acked Flush 0 Done Done Sent false
21219 6917 Acked Read 1 Done Done Sent false
21220 6918 Acked Write 8 Done Done Sent false
21221 6919 Acked Read 9 Done Done Sent false
21222 6920 Acked Write 2 Done Done Sent false
21223 6921 Acked Read 2 Done Done Sent false
21224 6922 Acked Write 4 Done Done Sent false
21225 6923 Acked Read 2 Done Done Sent false
21226 6924 Acked Write 6 Done Done Sent false
21227 6925 Acked Read 7 Done Done Sent false
21228 6926 Acked Read 7 Done Done Sent false
21229 6927 Acked Write 8 Done Done Sent false
21230 6928 Acked Write 4 Done Done Sent false
21231 6929 Acked Read 6 Done Done Sent false
21232 6930 Acked Read 8 Done Done Sent false
21233 6931 Acked Read 10 Done Done Sent false
21234 6932 Acked Read 8 Done Done Sent false
21235 6933 Acked Write 4 Done Done Sent false
21236 6934 Acked Write 1 Done Done Sent false
21237 6935 Acked Write 3 Done Done Sent false
21238 6936 Acked Read 1 Done Done Sent false
21239 6937 Acked Read 2 Done Done Sent false
21240 6938 Acked Read 3 Done Done Sent false
21241 6939 Acked Write 7 Done Done Sent false
21242 6940 Acked Read 5 Done Done Sent false
21243 6941 Acked Read 8 Done Done Sent false
21244 6942 Acked Write 9 Done Done Sent false
21245 6943 Acked Read 10 Done Done Sent false
21246 6944 Acked Write 8 Done Done Sent false
21247 6945 Acked Read 9 Done Done Sent false
21248 6946 Acked Read 8 Done Done Sent false
21249 6947 Acked Read 8 Done Done Sent false
21250 6948 Acked Read 3 Done Done Sent false
21251 6949 Acked Read 5 Done Done Sent false
21252 6950 Acked Read 2 Done Done Sent false
21253 6951 Acked Flush 0 Done Done Sent false
21254 6952 Acked Write 9 Done Done Sent false
21255 6953 Acked Flush 0 Done Done Sent false
21256 6954 Acked Read 2 Done Done Sent false
21257 6955 Acked Read 5 Done Done Sent false
21258 6956 Acked Read 6 Done Done Sent false
21259 6957 Acked Read 5 Done Done Sent false
21260 6958 Acked Write 10 Done Done Sent false
21261 6959 Acked Write 9 Done Done Sent false
21262 6960 Acked Read 1 Done Done Sent false
21263 6961 Acked Write 7 Done Done Sent false
21264 6962 Acked Write 1 Done Done Sent false
21265 6963 Acked Write 3 Done Done Sent false
21266 6964 Acked Read 5 Done Done Sent false
21267 6965 Acked Write 4 Done Done Sent false
21268 6966 Acked Read 7 Done Done Sent false
21269 6967 Acked Write 10 Done Done Sent false
21270 6968 Acked Read 4 Done Done Sent false
21271 6969 Acked Write 4 Done Done Sent false
21272 6970 Acked Write 2 Done Done Sent false
21273 6971 Acked Write 6 Done Done Sent false
21274 6972 Acked Read 4 Done Done Sent false
21275 6973 Acked Read 4 Done Done Sent false
21276 6974 Acked Write 3 Done Done Sent false
21277 6975 Acked Flush 0 Done Done Sent false
21278 6976 Acked Write 10 Done Done Sent false
21279 6977 Acked Write 8 Done Done Sent false
21280 6978 Acked Write 1 Done Done Sent false
21281 6979 Acked Flush 0 Done Done Sent false
21282 6980 Acked Read 9 Done Done Sent false
21283 6981 Acked Read 8 Done Done Sent false
21284 6982 Acked Read 4 Done Done Sent false
21285 6983 Acked Write 1 Done Done Sent false
21286 6984 Acked Write 10 Done Done Sent false
21287 6985 Acked Read 4 Done Done Sent false
21288 6986 Acked Write 4 Done Done Sent false
21289 6987 Acked Flush 0 Done Done Sent false
21290 6988 Acked Write 2 Done Done Sent false
21291 6989 Acked Read 3 Done Done Sent false
21292 6990 Acked Flush 0 Done Done Sent false
21293 6991 Acked Write 6 Done Done Sent false
21294 6992 Acked Read 10 Done Done Sent false
21295 6993 Acked Flush 0 Done Done Sent false
21296 6994 Acked Read 4 Done Done Sent false
21297 6995 Acked Read 1 Done Done Sent false
21298 6996 Acked Write 6 Done Done Sent false
21299 6997 Acked Read 3 Done Done Sent false
21300 6998 Acked Write 2 Done Done Sent false
21301 6999 Acked Read 2 Done Done Sent false
21302 7000 Acked Write 1 Done Done Sent false
21303 7001 Acked Read 5 Done Done Sent false
21304 7002 Acked Write 3 Done Done Sent false
21305 7003 Acked Write 5 Done Done Sent false
21306 7004 Acked Flush 0 Done Done Sent false
21307 7005 Acked Read 6 Done Done Sent false
21308 7006 Acked Flush 0 Done Done Sent false
21309 7007 Acked Read 10 Done Done Sent false
21310 7008 Acked Read 5 Done Done Sent false
21311 7009 Acked Write 5 Done Done Sent false
21312 7010 Acked Write 4 Done Done Sent false
21313 7011 Acked Flush 0 Done Done Sent false
21314 7012 Acked Read 7 Done Done Sent false
21315 7013 Acked Write 4 Done Done Sent false
21316 7014 Acked Read 10 Done Done Sent false
21317 7015 Acked Write 1 Done Done Sent false
21318 7016 Acked Write 6 Done Done Sent false
21319 7017 Acked Write 10 Done Done Sent false
21320 7018 Acked Read 3 Done Done Sent false
21321 7019 Acked Flush 0 Done Done Sent false
21322 7020 Acked Write 10 Done Done Sent false
21323 7021 Acked Read 4 Done Done Sent false
21324 7022 Acked Read 9 Done Done Sent false
21325 7023 Acked Read 2 Done Done Sent false
21326 7024 Acked Write 10 Done Done Sent false
21327 7025 Acked Write 10 Done Done Sent false
21328 7026 Acked Write 1 Done Done Sent false
21329 7027 Acked Write 6 Done Done Sent false
21330 7028 Acked Read 3 Done Done Sent false
21331 7029 Acked Write 1 Done Done Sent false
21332 7030 Acked Write 9 Done Done Sent false
21333 7031 Acked Write 2 Done Done Sent false
21334 7032 Acked Write 3 Done Done Sent false
21335 7033 Acked Write 9 Done Done Sent false
21336 7034 Acked Write 8 Done Done Sent false
21337 7035 Acked Read 8 Done Done Sent false
21338 7036 Acked Read 3 Done Done Sent false
21339 7037 Acked Read 10 Done Done Sent false
21340 STATES DS:0 DS:1 DS:2 TOTAL
21341 Sent 0 0 301 301
21342 Done 301 301 0 602
21343 Skipped 0 0 0 0
21344 Error 0 0 0 0
21345 Last Flush: Some(JobId(7019)) Some(JobId(7019)) Some(JobId(6736))
21346 Downstairs last five completed: 6736 6735 6734 6733 6732
21347 Upstairs last five completed: 7037 7036 7035 7034 7033
21348 CLIENT: Up:0 ds:301 act:2
213492025-01-28T21:58:55.742ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
213502025-01-28T21:58:56.243ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
213512025-01-28T21:58:56.744ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
213522025-01-28T21:58:57.245ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
213532025-01-28T21:58:57.746ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
213542025-01-28T21:58:58.247ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
21355 [8830] got start action
21356 [8830] I am going to restart
21357 Starting downstairs at port 8830
21358 Make output file at "/tmp/dsc/downstairs-8830.txt"
21359 [8830][2] reports Starting
21360 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1222)
21361 [8830][2] reports Running
21362 ----------------------------------------------------------------
21363 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21364 States: Active Active Offline
21365 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21366 6737 Acked Read 10 Done Done Sent false
21367 6738 Acked Read 5 Done Done Sent false
21368 6739 Acked Read 8 Done Done Sent false
21369 6740 Acked Read 8 Done Done Sent false
21370 6741 Acked Read 8 Done Done Sent false
21371 6742 Acked Read 8 Done Done Sent false
21372 6743 Acked Write 6 Done Done Sent false
21373 6744 Acked Read 10 Done Done Sent false
21374 6745 Acked Write 2 Done Done Sent false
21375 6746 Acked Write 2 Done Done Sent false
21376 6747 Acked Read 2 Done Done Sent false
21377 6748 Acked Write 4 Done Done Sent false
21378 6749 Acked Read 1 Done Done Sent false
21379 6750 Acked Read 10 Done Done Sent false
21380 6751 Acked Write 1 Done Done Sent false
21381 6752 Acked Read 3 Done Done Sent false
21382 6753 Acked Read 1 Done Done Sent false
21383 6754 Acked Write 5 Done Done Sent false
21384 6755 Acked Read 10 Done Done Sent false
21385 6756 Acked Flush 0 Done Done Sent false
21386 6757 Acked Write 8 Done Done Sent false
21387 6758 Acked Write 1 Done Done Sent false
21388 6759 Acked Write 6 Done Done Sent false
21389 6760 Acked Write 1 Done Done Sent false
21390 6761 Acked Write 2 Done Done Sent false
21391 6762 Acked Write 7 Done Done Sent false
21392 6763 Acked Read 6 Done Done Sent false
21393 6764 Acked Read 7 Done Done Sent false
21394 6765 Acked Read 5 Done Done Sent false
21395 6766 Acked Write 5 Done Done Sent false
21396 6767 Acked Write 6 Done Done Sent false
21397 6768 Acked Read 2 Done Done Sent false
21398 6769 Acked Read 4 Done Done Sent false
21399 6770 Acked Read 8 Done Done Sent false
21400 6771 Acked Read 10 Done Done Sent false
21401 6772 Acked Write 7 Done Done Sent false
21402 6773 Acked Read 1 Done Done Sent false
21403 6774 Acked Read 10 Done Done Sent false
21404 6775 Acked Read 5 Done Done Sent false
21405 6776 Acked Read 1 Done Done Sent false
21406 6777 Acked Read 3 Done Done Sent false
21407 6778 Acked Read 2 Done Done Sent false
21408 6779 Acked Flush 0 Done Done Sent false
21409 6780 Acked Read 2 Done Done Sent false
21410 6781 Acked Write 8 Done Done Sent false
21411 6782 Acked Write 2 Done Done Sent false
21412 6783 Acked Write 2 Done Done Sent false
21413 6784 Acked Read 1 Done Done Sent false
21414 6785 Acked Read 6 Done Done Sent false
21415 6786 Acked Flush 0 Done Done Sent false
21416 6787 Acked Write 9 Done Done Sent false
21417 6788 Acked Write 4 Done Done Sent false
21418 6789 Acked Read 8 Done Done Sent false
21419 6790 Acked Write 2 Done Done Sent false
21420 6791 Acked Write 9 Done Done Sent false
21421 6792 Acked Read 10 Done Done Sent false
21422 6793 Acked Read 1 Done Done Sent false
21423 6794 Acked Write 5 Done Done Sent false
21424 6795 Acked Write 10 Done Done Sent false
21425 6796 Acked Write 2 Done Done Sent false
21426 6797 Acked Read 9 Done Done Sent false
21427 6798 Acked Read 7 Done Done Sent false
21428 6799 Acked Write 9 Done Done Sent false
21429 6800 Acked Flush 0 Done Done Sent false
21430 6801 Acked Write 2 Done Done Sent false
21431 6802 Acked Write 8 Done Done Sent false
21432 6803 Acked Read 3 Done Done Sent false
21433 6804 Acked Write 8 Done Done Sent false
21434 6805 Acked Read 6 Done Done Sent false
21435 6806 Acked Read 9 Done Done Sent false
21436 6807 Acked Write 6 Done Done Sent false
21437 6808 Acked Write 6 Done Done Sent false
21438 6809 Acked Write 7 Done Done Sent false
21439 6810 Acked Write 8 Done Done Sent false
21440 6811 Acked Flush 0 Done Done Sent false
21441 6812 Acked Read 3 Done Done Sent false
21442 6813 Acked Write 6 Done Done Sent false
21443 6814 Acked Write 4 Done Done Sent false
21444 6815 Acked Read 3 Done Done Sent false
21445 6816 Acked Flush 0 Done Done Sent false
21446 6817 Acked Write 9 Done Done Sent false
21447 6818 Acked Flush 0 Done Done Sent false
21448 6819 Acked Read 9 Done Done Sent false
21449 6820 Acked Read 2 Done Done Sent false
21450 6821 Acked Read 7 Done Done Sent false
21451 6822 Acked Read 4 Done Done Sent false
21452 6823 Acked Read 5 Done Done Sent false
21453 6824 Acked Read 2 Done Done Sent false
21454 6825 Acked Read 8 Done Done Sent false
21455 6826 Acked Read 4 Done Done Sent false
21456 6827 Acked Write 7 Done Done Sent false
21457 6828 Acked Read 10 Done Done Sent false
21458 6829 Acked Write 7 Done Done Sent false
21459 6830 Acked Read 8 Done Done Sent false
21460 6831 Acked Read 6 Done Done Sent false
21461 6832 Acked Write 10 Done Done Sent false
21462 6833 Acked Write 7 Done Done Sent false
21463 6834 Acked Write 4 Done Done Sent false
21464 6835 Acked Read 2 Done Done Sent false
21465 6836 Acked Read 5 Done Done Sent false
21466 6837 Acked Read 7 Done Done Sent false
21467 6838 Acked Write 2 Done Done Sent false
21468 6839 Acked Write 4 Done Done Sent false
21469 6840 Acked Read 1 Done Done Sent false
21470 6841 Acked Flush 0 Done Done Sent false
21471 6842 Acked Write 7 Done Done Sent false
21472 6843 Acked Read 2 Done Done Sent false
21473 6844 Acked Write 9 Done Done Sent false
21474 6845 Acked Read 9 Done Done Sent false
21475 6846 Acked Write 3 Done Done Sent false
21476 6847 Acked Write 6 Done Done Sent false
21477 6848 Acked Write 5 Done Done Sent false
21478 6849 Acked Write 3 Done Done Sent false
21479 6850 Acked Read 5 Done Done Sent false
21480 6851 Acked Read 9 Done Done Sent false
21481 6852 Acked Flush 0 Done Done Sent false
21482 6853 Acked Read 1 Done Done Sent false
21483 6854 Acked Read 7 Done Done Sent false
21484 6855 Acked Read 2 Done Done Sent false
21485 6856 Acked Write 1 Done Done Sent false
21486 6857 Acked Write 1 Done Done Sent false
21487 6858 Acked Read 6 Done Done Sent false
21488 6859 Acked Write 7 Done Done Sent false
21489 6860 Acked Write 8 Done Done Sent false
21490 6861 Acked Flush 0 Done Done Sent false
21491 6862 Acked Write 4 Done Done Sent false
21492 6863 Acked Read 6 Done Done Sent false
21493 6864 Acked Read 8 Done Done Sent false
21494 6865 Acked Read 9 Done Done Sent false
21495 6866 Acked Flush 0 Done Done Sent false
21496 6867 Acked Read 5 Done Done Sent false
21497 6868 Acked Write 10 Done Done Sent false
21498 6869 Acked Flush 0 Done Done Sent false
21499 6870 Acked Read 6 Done Done Sent false
21500 6871 Acked Write 6 Done Done Sent false
21501 6872 Acked Write 9 Done Done Sent false
21502 6873 Acked Read 10 Done Done Sent false
21503 6874 Acked Write 3 Done Done Sent false
21504 6875 Acked Read 1 Done Done Sent false
21505 6876 Acked Read 3 Done Done Sent false
21506 6877 Acked Write 3 Done Done Sent false
21507 6878 Acked Write 4 Done Done Sent false
21508 6879 Acked Read 9 Done Done Sent false
21509 6880 Acked Write 2 Done Done Sent false
21510 6881 Acked Write 3 Done Done Sent false
21511 6882 Acked Flush 0 Done Done Sent false
21512 6883 Acked Read 2 Done Done Sent false
21513 6884 Acked Read 4 Done Done Sent false
21514 6885 Acked Write 7 Done Done Sent false
21515 6886 Acked Write 3 Done Done Sent false
21516 6887 Acked Write 5 Done Done Sent false
21517 6888 Acked Write 3 Done Done Sent false
21518 6889 Acked Write 7 Done Done Sent false
21519 6890 Acked Read 3 Done Done Sent false
21520 6891 Acked Write 5 Done Done Sent false
21521 6892 Acked Write 3 Done Done Sent false
21522 6893 Acked Write 1 Done Done Sent false
21523 6894 Acked Read 9 Done Done Sent false
21524 6895 Acked Flush 0 Done Done Sent false
21525 6896 Acked Read 2 Done Done Sent false
21526 6897 Acked Write 2 Done Done Sent false
21527 6898 Acked Write 5 Done Done Sent false
21528 6899 Acked Read 6 Done Done Sent false
21529 6900 Acked Flush 0 Done Done Sent false
21530 6901 Acked Read 9 Done Done Sent false
21531 6902 Acked Flush 0 Done Done Sent false
21532 6903 Acked Write 6 Done Done Sent false
21533 6904 Acked Flush 0 Done Done Sent false
21534 6905 Acked Read 1 Done Done Sent false
21535 6906 Acked Write 8 Done Done Sent false
21536 6907 Acked Write 6 Done Done Sent false
21537 6908 Acked Write 5 Done Done Sent false
21538 6909 Acked Write 2 Done Done Sent false
21539 6910 Acked Read 4 Done Done Sent false
21540 6911 Acked Read 7 Done Done Sent false
21541 6912 Acked Read 8 Done Done Sent false
21542 6913 Acked Flush 0 Done Done Sent false
21543 6914 Acked Read 10 Done Done Sent false
21544 6915 Acked Read 2 Done Done Sent false
21545 6916 Acked Flush 0 Done Done Sent false
21546 6917 Acked Read 1 Done Done Sent false
21547 6918 Acked Write 8 Done Done Sent false
21548 6919 Acked Read 9 Done Done Sent false
21549 6920 Acked Write 2 Done Done Sent false
21550 6921 Acked Read 2 Done Done Sent false
21551 6922 Acked Write 4 Done Done Sent false
21552 6923 Acked Read 2 Done Done Sent false
21553 6924 Acked Write 6 Done Done Sent false
21554 6925 Acked Read 7 Done Done Sent false
21555 6926 Acked Read 7 Done Done Sent false
21556 6927 Acked Write 8 Done Done Sent false
21557 6928 Acked Write 4 Done Done Sent false
21558 6929 Acked Read 6 Done Done Sent false
21559 6930 Acked Read 8 Done Done Sent false
21560 6931 Acked Read 10 Done Done Sent false
21561 6932 Acked Read 8 Done Done Sent false
21562 6933 Acked Write 4 Done Done Sent false
21563 6934 Acked Write 1 Done Done Sent false
21564 6935 Acked Write 3 Done Done Sent false
21565 6936 Acked Read 1 Done Done Sent false
21566 6937 Acked Read 2 Done Done Sent false
21567 6938 Acked Read 3 Done Done Sent false
21568 6939 Acked Write 7 Done Done Sent false
21569 6940 Acked Read 5 Done Done Sent false
21570 6941 Acked Read 8 Done Done Sent false
21571 6942 Acked Write 9 Done Done Sent false
21572 6943 Acked Read 10 Done Done Sent false
21573 6944 Acked Write 8 Done Done Sent false
21574 6945 Acked Read 9 Done Done Sent false
21575 6946 Acked Read 8 Done Done Sent false
21576 6947 Acked Read 8 Done Done Sent false
21577 6948 Acked Read 3 Done Done Sent false
21578 6949 Acked Read 5 Done Done Sent false
21579 6950 Acked Read 2 Done Done Sent false
21580 6951 Acked Flush 0 Done Done Sent false
21581 6952 Acked Write 9 Done Done Sent false
21582 6953 Acked Flush 0 Done Done Sent false
21583 6954 Acked Read 2 Done Done Sent false
21584 6955 Acked Read 5 Done Done Sent false
21585 6956 Acked Read 6 Done Done Sent false
21586 6957 Acked Read 5 Done Done Sent false
21587 6958 Acked Write 10 Done Done Sent false
21588 6959 Acked Write 9 Done Done Sent false
21589 6960 Acked Read 1 Done Done Sent false
21590 6961 Acked Write 7 Done Done Sent false
21591 6962 Acked Write 1 Done Done Sent false
21592 6963 Acked Write 3 Done Done Sent false
21593 6964 Acked Read 5 Done Done Sent false
21594 6965 Acked Write 4 Done Done Sent false
21595 6966 Acked Read 7 Done Done Sent false
21596 6967 Acked Write 10 Done Done Sent false
21597 6968 Acked Read 4 Done Done Sent false
21598 6969 Acked Write 4 Done Done Sent false
21599 6970 Acked Write 2 Done Done Sent false
21600 6971 Acked Write 6 Done Done Sent false
21601 6972 Acked Read 4 Done Done Sent false
21602 6973 Acked Read 4 Done Done Sent false
21603 6974 Acked Write 3 Done Done Sent false
21604 6975 Acked Flush 0 Done Done Sent false
21605 6976 Acked Write 10 Done Done Sent false
21606 6977 Acked Write 8 Done Done Sent false
21607 6978 Acked Write 1 Done Done Sent false
21608 6979 Acked Flush 0 Done Done Sent false
21609 6980 Acked Read 9 Done Done Sent false
21610 6981 Acked Read 8 Done Done Sent false
21611 6982 Acked Read 4 Done Done Sent false
21612 6983 Acked Write 1 Done Done Sent false
21613 6984 Acked Write 10 Done Done Sent false
21614 6985 Acked Read 4 Done Done Sent false
21615 6986 Acked Write 4 Done Done Sent false
21616 6987 Acked Flush 0 Done Done Sent false
21617 6988 Acked Write 2 Done Done Sent false
21618 6989 Acked Read 3 Done Done Sent false
21619 6990 Acked Flush 0 Done Done Sent false
21620 6991 Acked Write 6 Done Done Sent false
21621 6992 Acked Read 10 Done Done Sent false
21622 6993 Acked Flush 0 Done Done Sent false
21623 6994 Acked Read 4 Done Done Sent false
21624 6995 Acked Read 1 Done Done Sent false
21625 6996 Acked Write 6 Done Done Sent false
21626 6997 Acked Read 3 Done Done Sent false
21627 6998 Acked Write 2 Done Done Sent false
21628 6999 Acked Read 2 Done Done Sent false
21629 7000 Acked Write 1 Done Done Sent false
21630 7001 Acked Read 5 Done Done Sent false
21631 7002 Acked Write 3 Done Done Sent false
21632 7003 Acked Write 5 Done Done Sent false
21633 7004 Acked Flush 0 Done Done Sent false
21634 7005 Acked Read 6 Done Done Sent false
21635 7006 Acked Flush 0 Done Done Sent false
21636 7007 Acked Read 10 Done Done Sent false
21637 7008 Acked Read 5 Done Done Sent false
21638 7009 Acked Write 5 Done Done Sent false
21639 7010 Acked Write 4 Done Done Sent false
21640 7011 Acked Flush 0 Done Done Sent false
21641 7012 Acked Read 7 Done Done Sent false
21642 7013 Acked Write 4 Done Done Sent false
21643 7014 Acked Read 10 Done Done Sent false
21644 7015 Acked Write 1 Done Done Sent false
21645 7016 Acked Write 6 Done Done Sent false
21646 7017 Acked Write 10 Done Done Sent false
21647 7018 Acked Read 3 Done Done Sent false
21648 7019 Acked Flush 0 Done Done Sent false
21649 7020 Acked Write 10 Done Done Sent false
21650 7021 Acked Read 4 Done Done Sent false
21651 7022 Acked Read 9 Done Done Sent false
21652 7023 Acked Read 2 Done Done Sent false
21653 7024 Acked Write 10 Done Done Sent false
21654 7025 Acked Write 10 Done Done Sent false
21655 7026 Acked Write 1 Done Done Sent false
21656 7027 Acked Write 6 Done Done Sent false
21657 7028 Acked Read 3 Done Done Sent false
21658 7029 Acked Write 1 Done Done Sent false
21659 7030 Acked Write 9 Done Done Sent false
21660 7031 Acked Write 2 Done Done Sent false
21661 7032 Acked Write 3 Done Done Sent false
21662 7033 Acked Write 9 Done Done Sent false
21663 7034 Acked Write 8 Done Done Sent false
21664 7035 Acked Read 8 Done Done Sent false
21665 7036 Acked Read 3 Done Done Sent false
21666 7037 Acked Read 10 Done Done Sent false
21667 STATES DS:0 DS:1 DS:2 TOTAL
21668 Sent 0 0 301 301
21669 Done 301 301 0 602
21670 Skipped 0 0 0 0
21671 Error 0 0 0 0
21672 Last Flush: Some(JobId(7019)) Some(JobId(7019)) Some(JobId(6736))
21673 Downstairs last five completed: 6736 6735 6734 6733 6732
21674 Upstairs last five completed: 7037 7036 7035 7034 7033
21675 CLIENT: Up:0 ds:301 act:2
216762025-01-28T21:59:00.146ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
216772025-01-28T21:59:00.599ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
216782025-01-28T21:59:00.599ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58132) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
216792025-01-28T21:59:00.600ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
216802025-01-28T21:59:00.600ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
216812025-01-28T21:59:00.600ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
216822025-01-28T21:59:00.600ZINFOcrucible: send last flush ID to this DS: Some(JobId(6736)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
216832025-01-28T21:59:00.600ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
216842025-01-28T21:59:00.600ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6736)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
216852025-01-28T21:59:00.600ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6736)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
21686 ----------------------------------------------------------------
21687 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21688 Downstairs last five completed: 7038 7037 7036 7035 7034
21689 Upstairs last five completed: 7038 7037 7036 7035 7034
21690 CLIENT: Up:0 ds:0 act:3
21691 Replay: All jobs finished
21692 Main task has work to do, go find it
21693 got dsc Stop(1)
21694 stop 1
21695 Jan 28 21:59:02.836 INFO request completed, latency_us[8820] Got stop action so:false kr:false
21696 : 366, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 51be9b25-d440-4058-952a-267762b0319b, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
216972025-01-28T21:59:02.836ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
216982025-01-28T21:59:02.836ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
21699 Jan 28 21:59:02.836 INFO request completed{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:59:02.836879714Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
21700 , latency_us: 122, response_code: 200, uri: /state/cid/1, method: GET, req_id: baa0bb18-1254-4637-a047-317c329fe191, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
217012025-01-28T21:59:02.836ZINFOcrucible: client task is exiting = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
21702 [8820] Exited with: ExitStatus(unix_wait_status(9))
21703 [8820][1] reports Exit
21704 Jan 28 21:59:06.839 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/1, method: GET, req_id: 7c140c69-7901-4f14-a023-02bdce4144cb, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
21705 001/300 Read block 1732 len 32768
21706 002/300 Write block 4834 len 20480 data: 4 5 5 5 4
21707 003/300 Read block 1649 len 20480
21708 004/300 Write block 1111 len 36864 data: 5 6 4 3 3 4 5 5 6
21709 005/300 Write block 3415 len 32768 data: 3 2 2 4 3 3 4 4
21710 006/300 Write block 600 len 28672 data: 4 4 4 4 5 3 4
21711 007/300 Write block 4052 len 20480 data: 3 3 3 3 4
21712 008/300 Read block 866 len 4096
21713 009/300 Read block 3186 len 12288
21714 010/300 Write block 2779 len 4096 data: 3
21715 011/300 Write block 603 len 40960 data: 5 6 4 5 6 6 6 6 4 3
21716 012/300 Write block 4253 len 12288 data: 7 5 5
21717 013/300 Read block 909 len 8192
21718 014/300 Write block 3602 len 28672 data: 6 5 5 6 5 5 6
21719 015/300 Read block 2549 len 8192
21720 016/300 Read block 3296 len 28672
21721 017/300 Read block 2683 len 4096
21722 018/300 Write block 724 len 12288 data: 4 4 4
21723 019/300 Write block 2118 len 8192 data: 3 3
21724 020/300 Write block 1596 len 8192 data: 2 2
21725 021/300 Read block 4569 len 32768
21726 022/300 Read block 1828 len 12288
21727 023/300 Read block 582 len 32768
21728 024/300 Flush
21729 025/300 Write block 1277 len 40960 data: 3 4 3 4 3 3 3 4 4 4
21730 026/300 Write block 4806 len 36864 data: 6 7 4 4 4 3 4 5 4
21731 027/300 Flush
21732 028/300 Write block 3036 len 4096 data: 3
21733 029/300 Write block 4066 len 36864 data: 5 4 3 5 5 5 5 6 8
21734 030/300 Write block 2429 len 4096 data: 7
21735 031/300 Read block 3703 len 20480
21736 032/300 Write block 4170 len 40960 data: 5 7 7 5 5 5 4 4 4 4
21737 033/300 Write block 3016 len 40960 data: 3 3 2 2 3 4 3 4 5 6
21738 034/300 Read block 372 len 40960
21739 035/300 Read block 919 len 12288
21740 036/300 Write block 375 len 4096 data: 5
21741 037/300 Read block 1752 len 4096
21742 038/300 Write block 1549 len 16384 data: 3 3 3 4
21743 039/300 Flush
21744 040/300 Write block 4368 len 36864 data: 3 3 3 4 6 7 7 7 6
21745 041/300 Write block 1524 len 16384 data: 5 6 5 6
21746 042/300 Flush
21747 043/300 Read block 2334 len 20480
21748 044/300 Read block 118 len 8192
21749 045/300 Read block 1679 len 40960
21750 046/300 Read block 4935 len 32768
21751 047/300 Read block 301 len 32768
21752 048/300 Read block 4780 len 28672
21753 049/300 Write block 847 len 28672 data: 4 4 3 3 3 3 4
21754 050/300 Write block 3647 len 28672 data: 6 7 5 6 6 5 4
21755 051/300 Flush
21756 052/300 Read block 4848 len 12288
21757 053/300 Read block 2276 len 16384
21758 054/300 Read block 763 len 12288
21759 055/300 Read block 4867 len 32768
21760 056/300 Write block 3980 len 12288 data: 7 8 8
21761 057/300 Write block 4734 len 8192 data: 3 3
21762 058/300 Write block 4692 len 40960 data: 6 7 8 7 6 7 5 4 4 3
21763 059/300 Read block 2065 len 24576
21764 060/300 Read block 3635 len 12288
21765 061/300 Flush
21766 062/300 Read block 972 len 28672
21767 063/300 Write block 1615 len 4096 data: 4
21768 064/300 Read block 1202 len 8192
21769 065/300 Flush
21770 066/300 Write block 4736 len 12288 data: 5 6 7
21771 067/300 Write block 4618 len 24576 data: 6 6 6 6 5 4
21772 068/300 Write block 336 len 20480 data: 4 4 2 2 4
21773 069/300 Read block 1603 len 12288
21774 070/300 Write block 3742 len 40960 data: 3 3 2 2 2 4 3 3 4 5
21775 071/300 Read block 1666 len 24576
21776 072/300 Read block 2861 len 4096
21777 073/300 Read block 2934 len 4096
21778 074/300 Read block 3319 len 4096
21779 075/300 Write block 3239 len 20480 data: 6 5 5 5 6
21780 076/300 Read block 1556 len 40960
21781 077/300 Read block 974 len 32768
21782 078/300 Read block 802 len 32768
21783 079/300 Read block 154 len 24576
21784 080/300 Flush
21785 081/300 Flush
21786 082/300 Read block 3834 len 28672
21787 083/300 Flush
21788 084/300 Write block 215 len 32768 data: 4 4 5 4 5 5 4 5
21789 085/300 Write block 4764 len 16384 data: 5 5 5 5
21790 086/300 Write block 1297 len 4096 data: 6
21791 087/300 Read block 2879 len 20480
21792 088/300 Read block 2605 len 28672
21793 089/300 Write block 3470 len 40960 data: 5 5 5 5 5 4 3 5 5 4
21794 090/300 Flush
21795 091/300 Write block 2142 len 16384 data: 4 5 6 5
21796 092/300 Write block 4762 len 12288 data: 2 2 6
21797 093/300 Read block 958 len 28672
21798 094/300 Write block 3922 len 40960 data: 5 5 5 7 5 6 8 6 6 5
21799 095/300 Flush
21800 096/300 Read block 2837 len 40960
21801 097/300 Write block 1850 len 4096 data: 5
21802 098/300 Read block 4277 len 24576
21803 099/300 Write block 657 len 28672 data: 4 3 2 3 3 3 3
21804 100/300 Write block 1698 len 4096 data: 6
21805 101/300 Read block 4417 len 8192
21806 102/300 Read block 3751 len 12288
21807 103/300 Read block 1249 len 32768
21808 104/300 Flush
21809 105/300 Read block 4292 len 20480
21810 106/300 Flush
21811 107/300 Read block 4187 len 24576
21812 108/300 Flush
21813 109/300 Read block 1015 len 28672
21814 110/300 Write block 1205 len 4096 data: 8
21815 111/300 Read block 4477 len 16384
21816 112/300 Write block 869 len 28672 data: 5 5 5 4 5 4 4
21817 113/300 Read block 3968 len 40960
21818 114/300 Read block 4830 len 4096
21819 115/300 Read block 3949 len 8192
21820 116/300 Read block 610 len 12288
21821 117/300 Flush
21822 118/300 Read block 4227 len 16384
21823 119/300 Read block 2115 len 8192
21824 120/300 Read block 1414 len 32768
21825 121/300 Flush
21826 122/300 Read block 801 len 36864
21827 123/300 Flush
21828 124/300 Flush
21829 125/300 Read block 468 len 4096
21830 126/300 Flush
21831 127/300 Read block 4402 len 32768
21832 128/300 Flush
21833 129/300 Write block 4072 len 16384 data: 6 7 9 5
21834 130/300 Write block 719 len 4096 data: 5
21835 131/300 Read block 3610 len 4096
21836 132/300 Read block 2928 len 28672
21837 133/300 Write block 435 len 36864 data: 5 5 4 3 4 4 5 4 5
21838 134/300 Read block 1369 len 12288
21839 135/300 Write block 4611 len 32768 data: 5 4 4 4 6 6 6 7
21840 136/300 Write block 2315 len 36864 data: 3 4 4 5 5 5 4 2 2
21841 137/300 Read block 3802 len 16384
21842 138/300 Read block 1520 len 24576
21843 139/300 Write block 3857 len 4096 data: 2
21844 140/300 Write block 2073 len 36864 data: 4 3 4 5 4 5 4 5 7
21845 141/300 Read block 4210 len 4096
21846 142/300 Read block 728 len 32768
21847 143/300 Read block 190 len 32768
21848 144/300 Write block 3876 len 36864 data: 5 4 5 5 5 4 5 5 7
21849 145/300 Read block 3664 len 36864
21850 146/300 Write block 2754 len 28672 data: 5 6 6 6 6 6 6
21851 147/300 Read block 1236 len 40960
21852 148/300 Write block 4496 len 12288 data: 3 2 2
21853 149/300 Read block 1142 len 20480
21854 150/300 Write block 4657 len 4096 data: 4
21855 151/300 Write block 2995 len 32768 data: 8 6 6 5 3 5 6 5
21856 152/300 Read block 2113 len 24576
21857 153/300 Write block 4080 len 8192 data: 5 6
21858 154/300 Write block 3593 len 28672 data: 4 5 5 5 7 6 7
21859 155/300 Write block 2996 len 4096 data: 7
21860 156/300 Read block 1143 len 16384
21861 157/300 Write block 4382 len 40960 data: 7 8 7 7 5 4 4 4 4 3
21862 158/300 Write block 3393 len 4096 data: 8
21863 159/300 Flush
21864 160/300 Read block 202 len 4096
21865 161/300 Write block 4869 len 16384 data: 4 4 4 3
21866 162/300 Write block 1179 len 8192 data: 10 11
21867 163/300 Write block 4293 len 24576 data: 4 4 3 3 3 4
21868 164/300 Read block 829 len 36864
21869 165/300 Read block 2475 len 16384
21870 166/300 Write block 3661 len 12288 data: 6 6 7
21871 167/300 Read block 1607 len 20480
21872 168/300 Write block 4709 len 8192 data: 6 6
21873 169/300 Read block 1886 len 16384
21874 170/300 Read block 4547 len 32768
21875 171/300 Flush
21876 172/300 Read block 792 len 28672
21877 173/300 Write block 2857 len 28672 data: 6 6 6 6 7 5 5
21878 174/300 Write block 3275 len 20480 data: 5 5 4 2 2
21879 175/300 Read block 2950 len 8192
21880 176/300 Write block 3324 len 40960 data: 6 6 5 4 6 5 7 5 4 4
21881 177/300 Write block 3102 len 36864 data: 7 7 7 6 7 3 3 2 4
21882 178/300 Read block 4257 len 28672
21883 179/300 Write block 3949 len 12288 data: 6 5 4
21884 180/300 Write block 3062 len 36864 data: 5 5 4 5 5 5 5 4 3
21885 181/300 Flush
21886 182/300 Read block 944 len 12288
21887 183/300 Read block 3289 len 8192
21888 184/300 Write block 2430 len 4096 data: 7
21889 185/300 Write block 2385 len 4096 data: 7
21890 186/300 Flush
21891 187/300 Write block 3578 len 12288 data: 4 3 3
21892 188/300 Read block 3973 len 4096
21893 189/300 Read block 4524 len 28672
21894 190/300 Write block 1590 len 36864 data: 2 2 3 3 3 2 3 3 2
21895 191/300 Read block 2768 len 36864
21896 192/300 Read block 1205 len 36864
21897 193/300 Write block 4381 len 40960 data: 6 8 9 8 8 6 5 5 5 5
21898 194/300 Write block 3820 len 16384 data: 6 5 5 5
21899 195/300 Read block 2049 len 16384
21900 196/300 Write block 195 len 36864 data: 4 4 4 4 4 5 5 5 4
21901 197/300 Write block 1338 len 12288 data: 4 4 3
21902 198/300 Read block 4985 len 8192
21903 199/300 Read block 3548 len 8192
21904 200/300 Write block 3154 len 36864 data: 6 5 5 4 5 5 4 4 3
21905 201/300 Read block 692 len 32768
21906 202/300 Write block 1569 len 12288 data: 3 2 2
21907 203/300 Read block 3302 len 40960
21908 204/300 Read block 4620 len 28672
21909 205/300 Write block 2568 len 4096 data: 4
21910 206/300 Write block 2564 len 8192 data: 6 6
21911 207/300 Read block 4125 len 24576
21912 208/300 Read block 358 len 32768
21913 209/300 Read block 4454 len 32768
21914 210/300 Write block 4083 len 32768 data: 7 6 6 7 7 6 7 8
21915 211/300 Write block 4345 len 36864 data: 4 4 4 4 3 3 4 5 5
21916 212/300 Flush
21917 213/300 Read block 2951 len 36864
21918 214/300 Read block 590 len 4096
21919 215/300 Write block 1794 len 32768 data: 4 4 4 5 5 4 2 2
21920 216/300 Read block 2988 len 40960
21921 217/300 Write block 1162 len 4096 data: 3
21922 218/300 Read block 1098 len 36864
21923 219/300 Read block 2039 len 32768
21924 220/300 Write block 2959 len 4096 data: 4
21925 221/300 Read block 1115 len 24576
21926 222/300 Read block 4369 len 28672
21927 223/300 Read block 4533 len 12288
21928 224/300 Read block 2687 len 40960
21929 225/300 Read block 3169 len 28672
21930 226/300 Write block 4645 len 16384 data: 3 3 3 3
21931 227/300 Read block 675 len 24576
21932 228/300 Read block 1079 len 36864
21933 229/300 Read block 3142 len 4096
21934 230/300 Read block 4475 len 16384
21935 231/300 Write block 4203 len 36864 data: 4 4 5 5 4 5 5 5 4
21936 232/300 Write block 662 len 16384 data: 4 4 4 3
21937 233/300 Flush
21938 234/300 Write block 4310 len 8192 data: 5 5
21939 235/300 Read block 517 len 12288
21940 236/300 Write block 3620 len 20480 data: 5 4 4 5 6
21941 237/300 Flush
21942 238/300 Write block 4037 len 12288 data: 3 3 3
21943 239/300 Write block 4137 len 24576 data: 7 6 6 6 6 5
21944 240/300 Write block 3803 len 4096 data: 2
21945 241/300 Read block 620 len 16384
21946 242/300 Read block 94 len 24576
21947 243/300 Write block 1011 len 36864 data: 6 6 7 9 8 9 7 7 5
21948 244/300 Write block 696 len 40960 data: 2 2 2 4 4 4 4 5 4 5
21949 245/300 Read block 1062 len 12288
21950 246/300 Flush
21951 247/300 Write block 3380 len 28672 data: 5 4 3 3 2 4 4
21952 248/300 Read block 3863 len 12288
21953 249/300 Write block 2222 len 32768 data: 3 3 3 3 4 4 5 4
21954 250/300 Write block 2094 len 24576 data: 5 5 5 5 6 7
21955 251/300 Read block 1132 len 40960
21956 252/300 Read block 4283 len 8192
21957 253/300 Read block 4205 len 36864
21958 254/300 Read block 1181 len 32768
21959 255/300 Read block 3359 len 28672
21960 256/300 Write block 2320 len 8192 data: 6 5
21961 257/300 Read block 2170 len 32768
21962 258/300 Write block 2803 len 20480 data: 6 5 4 4 4
21963 259/300 Write block 2734 len 28672 data: 6 6 4 4 4 4 3
21964 260/300 Read block 154 len 28672
21965 261/300 Read block 4643 len 12288
21966 262/300 Flush
21967 263/300 Write block 2237 len 32768 data: 4 4 4 3 3 3 4 6
21968 264/300 Flush
21969 265/300 Flush
21970 266/300 Read block 1979 len 36864
21971 267/300 Write block 3851 len 12288 data: 2 2 2
21972 268/300 Read block 4601 len 28672
21973 269/300 Write block 3457 len 4096 data: 4
21974 270/300 Write block 4345 len 4096 data: 5
21975 271/300 Flush
21976 272/300 Read block 961 len 40960
21977 273/300 Read block 3591 len 16384
21978 274/300 Write block 3250 len 40960 data: 6 7 6 6 7 7 6 5 5 5
21979 275/300 Read block 4940 len 20480
21980 276/300 Write block 1298 len 32768 data: 7 6 6 6 5 5 6 6
21981 277/300 Read block 1586 len 4096
21982 278/300 Read block 3018 len 36864
21983 279/300 Read block 2205 len 4096
21984 280/300 Flush
21985 281/300 Write block 4804 len 28672 data: 6 6 7 8 5 5 5
21986 282/300 Flush
21987 283/300 Read block 3363 len 20480
21988 284/300 Read block 2510 len 20480
21989 285/300 Read block 1042 len 24576
21990 286/300 Flush
21991 287/300 Read block 2917 len 16384
21992 288/300 Flush
21993 289/300 Read block 4390 len 28672
21994 290/300 Write block 4479 len 28672 data: 8 10 10 10 10 10 9
21995 291/300 Read block 2323 len 32768
21996 292/300 Read block 2342 len 12288
21997 293/300 Write block 234 len 8192 data: 7 8
21998 294/300 Write block 4265 len 24576 data: 5 6 5 5 4 6
21999 295/300 Read block 4743 len 28672
22000 296/300 Flush
22001 297/300 Write block 480 len 32768 data: 5 4 5 5 3 3 3 3
22002 298/300 Read block 686 len 20480
22003 299/300 Flush
22004 300/300 Read block 1900 len 40960
22005 301/300 Flush
22006 Main task has work to do, go find it
22007 got dsc Start(1)
22008 start 1
22009 Jan 28 21:59:07.076 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/1, method: POST, req_id: c19abd8f-8a34-4105-be51-4bdbcc9eb04f, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
22010 Replay: started 1, returned:Ok(())
22011 ----------------------------------------------------------------
22012 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22013 States: Active Offline Active
22014 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22015 7039 Acked Read 8 Done Sent Done false
22016 7040 Acked Write 5 Done Sent Done false
22017 7041 Acked Read 5 Done Sent Done false
22018 7042 Acked Write 9 Done Sent Done false
22019 7043 Acked Write 8 Done Sent Done false
22020 7044 Acked Write 7 Done Sent Done false
22021 7045 Acked Write 5 Done Sent Done false
22022 7046 Acked Read 1 Done Sent Done false
22023 7047 Acked Read 3 Done Sent Done false
22024 7048 Acked Write 1 Done Sent Done false
22025 7049 Acked Write 10 Done Sent Done false
22026 7050 Acked Write 3 Done Sent Done false
22027 7051 Acked Read 2 Done Sent Done false
22028 7052 Acked Write 7 Done Sent Done false
22029 7053 Acked Read 2 Done Sent Done false
22030 7054 Acked Read 7 Done Sent Done false
22031 7055 Acked Read 1 Done Sent Done false
22032 7056 Acked Write 3 Done Sent Done false
22033 7057 Acked Write 2 Done Sent Done false
22034 7058 Acked Write 2 Done Sent Done false
22035 7059 Acked Read 8 Done Sent Done false
22036 7060 Acked Read 3 Done Sent Done false
22037 7061 Acked Read 8 Done Sent Done false
22038 7062 Acked Flush 0 Done Sent Done false
22039 7063 Acked Write 10 Done Sent Done false
22040 7064 Acked Write 9 Done Sent Done false
22041 7065 Acked Flush 0 Done Sent Done false
22042 7066 Acked Write 1 Done Sent Done false
22043 7067 Acked Write 9 Done Sent Done false
22044 7068 Acked Write 1 Done Sent Done false
22045 7069 Acked Read 5 Done Sent Done false
22046 7070 Acked Write 10 Done Sent Done false
22047 7071 Acked Write 10 Done Sent Done false
22048 7072 Acked Read 10 Done Sent Done false
22049 7073 Acked Read 3 Done Sent Done false
22050 7074 Acked Write 1 Done Sent Done false
22051 7075 Acked Read 1 Done Sent Done false
22052 7076 Acked Write 4 Done Sent Done false
22053 7077 Acked Flush 0 Done Sent Done false
22054 7078 Acked Write 9 Done Sent Done false
22055 7079 Acked Write 4 Done Sent Done false
22056 7080 Acked Flush 0 Done Sent Done false
22057 7081 Acked Read 5 Done Sent Done false
22058 7082 Acked Read 2 Done Sent Done false
22059 7083 Acked Read 10 Done Sent Done false
22060 7084 Acked Read 8 Done Sent Done false
22061 7085 Acked Read 8 Done Sent Done false
22062 7086 Acked Read 7 Done Sent Done false
22063 7087 Acked Write 7 Done Sent Done false
22064 7088 Acked Write 7 Done Sent Done false
22065 7089 Acked Flush 0 Done Sent Done false
22066 7090 Acked Read 3 Done Sent Done false
22067 7091 Acked Read 4 Done Sent Done false
22068 7092 Acked Read 3 Done Sent Done false
22069 7093 Acked Read 8 Done Sent Done false
22070 7094 Acked Write 3 Done Sent Done false
22071 7095 Acked Write 2 Done Sent Done false
22072 7096 Acked Write 10 Done Sent Done false
22073 7097 Acked Read 6 Done Sent Done false
22074 7098 Acked Read 3 Done Sent Done false
22075 7099 Acked Flush 0 Done Sent Done false
22076 7100 Acked Read 7 Done Sent Done false
22077 7101 Acked Write 1 Done Sent Done false
22078 7102 Acked Read 2 Done Sent Done false
22079 7103 Acked Flush 0 Done Sent Done false
22080 7104 Acked Write 3 Done Sent Done false
22081 7105 Acked Write 6 Done Sent Done false
22082 7106 Acked Write 5 Done Sent Done false
22083 7107 Acked Read 3 Done Sent Done false
22084 7108 Acked Write 10 Done Sent Done false
22085 7109 Acked Read 6 Done Sent Done false
22086 7110 Acked Read 1 Done Sent Done false
22087 7111 Acked Read 1 Done Sent Done false
22088 7112 Acked Read 1 Done Sent Done false
22089 7113 Acked Write 5 Done Sent Done false
22090 7114 Acked Read 10 Done Sent Done false
22091 7115 Acked Read 8 Done Sent Done false
22092 7116 Acked Read 8 Done Sent Done false
22093 7117 Acked Read 6 Done Sent Done false
22094 7118 Acked Flush 0 Done Sent Done false
22095 7119 Acked Flush 0 Done Sent Done false
22096 7120 Acked Read 7 Done Sent Done false
22097 7121 Acked Flush 0 Done Sent Done false
22098 7122 Acked Write 8 Done Sent Done false
22099 7123 Acked Write 4 Done Sent Done false
22100 7124 Acked Write 1 Done Sent Done false
22101 7125 Acked Read 5 Done Sent Done false
22102 7126 Acked Read 7 Done Sent Done false
22103 7127 Acked Write 10 Done Sent Done false
22104 7128 Acked Flush 0 Done Sent Done false
22105 7129 Acked Write 4 Done Sent Done false
22106 7130 Acked Write 3 Done Sent Done false
22107 7131 Acked Read 7 Done Sent Done false
22108 7132 Acked Write 10 Done Sent Done false
22109 7133 Acked Flush 0 Done Sent Done false
22110 7134 Acked Read 10 Done Sent Done false
22111 7135 Acked Write 1 Done Sent Done false
22112 7136 Acked Read 6 Done Sent Done false
22113 7137 Acked Write 7 Done Sent Done false
22114 7138 Acked Write 1 Done Sent Done false
22115 7139 Acked Read 2 Done Sent Done false
22116 7140 Acked Read 3 Done Sent Done false
22117 7141 Acked Read 8 Done Sent Done false
22118 7142 Acked Flush 0 Done Sent Done false
22119 7143 Acked Read 5 Done Sent Done false
22120 7144 Acked Flush 0 Done Sent Done false
22121 7145 Acked Read 6 Done Sent Done false
22122 7146 Acked Flush 0 Done Sent Done false
22123 7147 Acked Read 7 Done Sent Done false
22124 7148 Acked Write 1 Done Sent Done false
22125 7149 Acked Read 4 Done Sent Done false
22126 7150 Acked Write 7 Done Sent Done false
22127 7151 Acked Read 10 Done Sent Done false
22128 7152 Acked Read 1 Done Sent Done false
22129 7153 Acked Read 2 Done Sent Done false
22130 7154 Acked Read 3 Done Sent Done false
22131 7155 Acked Flush 0 Done Sent Done false
22132 7156 Acked Read 4 Done Sent Done false
22133 7157 Acked Read 2 Done Sent Done false
22134 7158 Acked Read 8 Done Sent Done false
22135 7159 Acked Flush 0 Done Sent Done false
22136 7160 Acked Read 9 Done Sent Done false
22137 7161 Acked Flush 0 Done Sent Done false
22138 7162 Acked Flush 0 Done Sent Done false
22139 7163 Acked Read 1 Done Sent Done false
22140 7164 Acked Flush 0 Done Sent Done false
22141 7165 Acked Read 8 Done Sent Done false
22142 7166 Acked Flush 0 Done Sent Done false
22143 7167 Acked Write 4 Done Sent Done false
22144 7168 Acked Write 1 Done Sent Done false
22145 7169 Acked Read 1 Done Sent Done false
22146 7170 Acked Read 7 Done Sent Done false
22147 7171 Acked Write 9 Done Sent Done false
22148 7172 Acked Read 3 Done Sent Done false
22149 7173 Acked Write 8 Done Sent Done false
22150 7174 Acked Write 9 Done Sent Done false
22151 7175 Acked Read 4 Done Sent Done false
22152 7176 Acked Read 6 Done Sent Done false
22153 7177 Acked Write 1 Done Sent Done false
22154 7178 Acked Write 9 Done Sent Done false
22155 7179 Acked Read 1 Done Sent Done false
22156 7180 Acked Read 8 Done Sent Done false
22157 7181 Acked Read 8 Done Sent Done false
22158 7182 Acked Write 9 Done Sent Done false
22159 7183 Acked Read 9 Done Sent Done false
22160 7184 Acked Write 7 Done Sent Done false
22161 7185 Acked Read 10 Done Sent Done false
22162 7186 Acked Write 3 Done Sent Done false
22163 7187 Acked Read 5 Done Sent Done false
22164 7188 Acked Write 1 Done Sent Done false
22165 7189 Acked Write 8 Done Sent Done false
22166 7190 Acked Read 6 Done Sent Done false
22167 7191 Acked Write 2 Done Sent Done false
22168 7192 Acked Write 7 Done Sent Done false
22169 7193 Acked Write 1 Done Sent Done false
22170 7194 Acked Read 4 Done Sent Done false
22171 7195 Acked Write 10 Done Sent Done false
22172 7196 Acked Write 1 Done Sent Done false
22173 7197 Acked Flush 0 Done Sent Done false
22174 7198 Acked Read 1 Done Sent Done false
22175 7199 Acked Write 4 Done Sent Done false
22176 7200 Acked Write 2 Done Sent Done false
22177 7201 Acked Write 6 Done Sent Done false
22178 7202 Acked Read 9 Done Sent Done false
22179 7203 Acked Read 4 Done Sent Done false
22180 7204 Acked Write 3 Done Sent Done false
22181 7205 Acked Read 5 Done Sent Done false
22182 7206 Acked Write 2 Done Sent Done false
22183 7207 Acked Read 4 Done Sent Done false
22184 7208 Acked Read 8 Done Sent Done false
22185 7209 Acked Flush 0 Done Sent Done false
22186 7210 Acked Read 7 Done Sent Done false
22187 7211 Acked Write 7 Done Sent Done false
22188 7212 Acked Write 5 Done Sent Done false
22189 7213 Acked Read 2 Done Sent Done false
22190 7214 Acked Write 10 Done Sent Done false
22191 7215 Acked Write 9 Done Sent Done false
22192 7216 Acked Read 7 Done Sent Done false
22193 7217 Acked Write 3 Done Sent Done false
22194 7218 Acked Write 9 Done Sent Done false
22195 7219 Acked Flush 0 Done Sent Done false
22196 7220 Acked Read 3 Done Sent Done false
22197 7221 Acked Read 2 Done Sent Done false
22198 7222 Acked Write 1 Done Sent Done false
22199 7223 Acked Write 1 Done Sent Done false
22200 7224 Acked Flush 0 Done Sent Done false
22201 7225 Acked Write 3 Done Sent Done false
22202 7226 Acked Read 1 Done Sent Done false
22203 7227 Acked Read 7 Done Sent Done false
22204 7228 Acked Write 9 Done Sent Done false
22205 7229 Acked Read 9 Done Sent Done false
22206 7230 Acked Read 9 Done Sent Done false
22207 7231 Acked Write 10 Done Sent Done false
22208 7232 Acked Write 4 Done Sent Done false
22209 7233 Acked Read 4 Done Sent Done false
22210 7234 Acked Write 9 Done Sent Done false
22211 7235 Acked Write 3 Done Sent Done false
22212 7236 Acked Read 2 Done Sent Done false
22213 7237 Acked Read 2 Done Sent Done false
22214 7238 Acked Write 9 Done Sent Done false
22215 7239 Acked Read 8 Done Sent Done false
22216 7240 Acked Write 3 Done Sent Done false
22217 7241 Acked Read 10 Done Sent Done false
22218 7242 Acked Read 7 Done Sent Done false
22219 7243 Acked Write 1 Done Sent Done false
22220 7244 Acked Write 2 Done Sent Done false
22221 7245 Acked Read 6 Done Sent Done false
22222 7246 Acked Read 8 Done Sent Done false
22223 7247 Acked Read 8 Done Sent Done false
22224 7248 Acked Write 8 Done Sent Done false
22225 7249 Acked Write 9 Done Sent Done false
22226 7250 Acked Flush 0 Done Sent Done false
22227 7251 Acked Read 9 Done Sent Done false
22228 7252 Acked Read 1 Done Sent Done false
22229 7253 Acked Write 8 Done Sent Done false
22230 7254 Acked Read 10 Done Sent Done false
22231 7255 Acked Write 1 Done Sent Done false
22232 7256 Acked Read 9 Done Sent Done false
22233 7257 Acked Read 8 Done Sent Done false
22234 7258 Acked Write 1 Done Sent Done false
22235 7259 Acked Read 6 Done Sent Done false
22236 7260 Acked Read 7 Done Sent Done false
22237 7261 Acked Read 3 Done Sent Done false
22238 7262 Acked Read 10 Done Sent Done false
22239 7263 Acked Read 7 Done Sent Done false
22240 7264 Acked Write 4 Done Sent Done false
22241 7265 Acked Read 6 Done Sent Done false
22242 7266 Acked Read 9 Done Sent Done false
22243 7267 Acked Read 1 Done Sent Done false
22244 7268 Acked Read 4 Done Sent Done false
22245 7269 Acked Write 9 Done Sent Done false
22246 7270 Acked Write 4 Done Sent Done false
22247 7271 Acked Flush 0 Done Sent Done false
22248 7272 Acked Write 2 Done Sent Done false
22249 7273 Acked Read 3 Done Sent Done false
22250 7274 Acked Write 5 Done Sent Done false
22251 7275 Acked Flush 0 Done Sent Done false
22252 7276 Acked Write 3 Done Sent Done false
22253 7277 Acked Write 6 Done Sent Done false
22254 7278 Acked Write 1 Done Sent Done false
22255 7279 Acked Read 4 Done Sent Done false
22256 7280 Acked Read 6 Done Sent Done false
22257 7281 Acked Write 9 Done Sent Done false
22258 7282 Acked Write 10 Done Sent Done false
22259 7283 Acked Read 3 Done Sent Done false
22260 7284 Acked Flush 0 Done Sent Done false
22261 7285 Acked Write 7 Done Sent Done false
22262 7286 Acked Read 3 Done Sent Done false
22263 7287 Acked Write 8 Done Sent Done false
22264 7288 Acked Write 6 Done Sent Done false
22265 7289 Acked Read 10 Done Sent Done false
22266 7290 Acked Read 2 Done Sent Done false
22267 7291 Acked Read 9 Done Sent Done false
22268 7292 Acked Read 8 Done Sent Done false
22269 7293 Acked Read 7 Done Sent Done false
22270 7294 Acked Write 2 Done Sent Done false
22271 7295 Acked Read 8 Done Sent Done false
22272 7296 Acked Write 5 Done Sent Done false
22273 7297 Acked Write 7 Done Sent Done false
22274 7298 Acked Read 7 Done Sent Done false
22275 7299 Acked Read 3 Done Sent Done false
22276 7300 Acked Flush 0 Done Sent Done false
22277 7301 Acked Write 8 Done Sent Done false
22278 7302 Acked Flush 0 Done Sent Done false
22279 7303 Acked Flush 0 Done Sent Done false
22280 7304 Acked Read 9 Done Sent Done false
22281 7305 Acked Write 3 Done Sent Done false
22282 7306 Acked Read 7 Done Sent Done false
22283 7307 Acked Write 1 Done Sent Done false
22284 7308 Acked Write 1 Done Sent Done false
22285 7309 Acked Flush 0 Done Sent Done false
22286 7310 Acked Read 10 Done Sent Done false
22287 7311 Acked Read 4 Done Sent Done false
22288 7312 Acked Write 10 Done Sent Done false
22289 7313 Acked Read 5 Done Sent Done false
22290 7314 Acked Write 8 Done Sent Done false
22291 7315 Acked Read 1 Done Sent Done false
22292 7316 Acked Read 9 Done Sent Done false
22293 7317 Acked Read 1 Done Sent Done false
22294 7318 Acked Flush 0 Done Sent Done false
22295 7319 Acked Write 7 Done Sent Done false
22296 7320 Acked Flush 0 Done Sent Done false
22297 7321 Acked Read 5 Done Sent Done false
22298 7322 Acked Read 5 Done Sent Done false
22299 7323 Acked Read 6 Done Sent Done false
22300 7324 Acked Flush 0 Done Sent Done false
22301 7325 Acked Read 4 Done Sent Done false
22302 7326 Acked Flush 0 Done Sent Done false
22303 7327 Acked Read 7 Done Sent Done false
22304 7328 Acked Write 7 Done Sent Done false
22305 7329 Acked Read 8 Done Sent Done false
22306 7330 Acked Read 3 Done Sent Done false
22307 7331 Acked Write 2 Done Sent Done false
22308 7332 Acked Write 6 Done Sent Done false
22309 7333 Acked Read 7 Done Sent Done false
22310 7334 Acked Flush 0 Done Sent Done false
22311 7335 Acked Write 8 Done Sent Done false
22312 7336 Acked Read 5 Done Sent Done false
22313 7337 Acked Flush 0 Done Sent Done false
22314 7338 Acked Read 10 Done Sent Done false
22315 7339 Acked Flush 0 Done Sent Done false
22316 STATES DS:0 DS:1 DS:2 TOTAL
22317 Sent 0 301 0 301
22318 Done 301 0 301 602
22319 Skipped 0 0 0 0
22320 Error 0 0 0 0
22321 Last Flush: Some(JobId(7339)) Some(JobId(7038)) Some(JobId(7339))
22322 Downstairs last five completed: 7038 7037 7036 7035 7034
22323 Upstairs last five completed: 7339 7338 7337 7336 7335
22324 CLIENT: Up:0 ds:301 act:2
223252025-01-28T21:59:07.764ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
223262025-01-28T21:59:08.765ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
22327 [8820] got start action
22328 [8820] I am going to restart
22329 Starting downstairs at port 8820
22330 Make output file at "/tmp/dsc/downstairs-8820.txt"
22331 [8820][1] reports Starting
22332 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1223)
22333 [8820][1] reports Running
223342025-01-28T21:59:10.147ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
22335 ----------------------------------------------------------------
22336 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22337 States: Active Offline Active
22338 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22339 7039 Acked Read 8 Done Sent Done false
22340 7040 Acked Write 5 Done Sent Done false
22341 7041 Acked Read 5 Done Sent Done false
22342 7042 Acked Write 9 Done Sent Done false
22343 7043 Acked Write 8 Done Sent Done false
22344 7044 Acked Write 7 Done Sent Done false
22345 7045 Acked Write 5 Done Sent Done false
22346 7046 Acked Read 1 Done Sent Done false
22347 7047 Acked Read 3 Done Sent Done false
22348 7048 Acked Write 1 Done Sent Done false
22349 7049 Acked Write 10 Done Sent Done false
22350 7050 Acked Write 3 Done Sent Done false
22351 7051 Acked Read 2 Done Sent Done false
22352 7052 Acked Write 7 Done Sent Done false
22353 7053 Acked Read 2 Done Sent Done false
22354 7054 Acked Read 7 Done Sent Done false
22355 7055 Acked Read 1 Done Sent Done false
22356 7056 Acked Write 3 Done Sent Done false
22357 7057 Acked Write 2 Done Sent Done false
22358 7058 Acked Write 2 Done Sent Done false
22359 7059 Acked Read 8 Done Sent Done false
22360 7060 Acked Read 3 Done Sent Done false
22361 7061 Acked Read 8 Done Sent Done false
22362 7062 Acked Flush 0 Done Sent Done false
22363 7063 Acked Write 10 Done Sent Done false
22364 7064 Acked Write 9 Done Sent Done false
22365 7065 Acked Flush 0 Done Sent Done false
22366 7066 Acked Write 1 Done Sent Done false
22367 7067 Acked Write 9 Done Sent Done false
22368 7068 Acked Write 1 Done Sent Done false
22369 7069 Acked Read 5 Done Sent Done false
22370 7070 Acked Write 10 Done Sent Done false
22371 7071 Acked Write 10 Done Sent Done false
22372 7072 Acked Read 10 Done Sent Done false
22373 7073 Acked Read 3 Done Sent Done false
22374 7074 Acked Write 1 Done Sent Done false
22375 7075 Acked Read 1 Done Sent Done false
22376 7076 Acked Write 4 Done Sent Done false
22377 7077 Acked Flush 0 Done Sent Done false
22378 7078 Acked Write 9 Done Sent Done false
22379 7079 Acked Write 4 Done Sent Done false
22380 7080 Acked Flush 0 Done Sent Done false
22381 7081 Acked Read 5 Done Sent Done false
22382 7082 Acked Read 2 Done Sent Done false
22383 7083 Acked Read 10 Done Sent Done false
22384 7084 Acked Read 8 Done Sent Done false
22385 7085 Acked Read 8 Done Sent Done false
22386 7086 Acked Read 7 Done Sent Done false
22387 7087 Acked Write 7 Done Sent Done false
22388 7088 Acked Write 7 Done Sent Done false
22389 7089 Acked Flush 0 Done Sent Done false
22390 7090 Acked Read 3 Done Sent Done false
22391 7091 Acked Read 4 Done Sent Done false
22392 7092 Acked Read 3 Done Sent Done false
22393 7093 Acked Read 8 Done Sent Done false
22394 7094 Acked Write 3 Done Sent Done false
22395 7095 Acked Write 2 Done Sent Done false
22396 7096 Acked Write 10 Done Sent Done false
22397 7097 Acked Read 6 Done Sent Done false
22398 7098 Acked Read 3 Done Sent Done false
22399 7099 Acked Flush 0 Done Sent Done false
22400 7100 Acked Read 7 Done Sent Done false
22401 7101 Acked Write 1 Done Sent Done false
22402 7102 Acked Read 2 Done Sent Done false
22403 7103 Acked Flush 0 Done Sent Done false
22404 7104 Acked Write 3 Done Sent Done false
22405 7105 Acked Write 6 Done Sent Done false
22406 7106 Acked Write 5 Done Sent Done false
22407 7107 Acked Read 3 Done Sent Done false
22408 7108 Acked Write 10 Done Sent Done false
22409 7109 Acked Read 6 Done Sent Done false
22410 7110 Acked Read 1 Done Sent Done false
22411 7111 Acked Read 1 Done Sent Done false
22412 7112 Acked Read 1 Done Sent Done false
22413 7113 Acked Write 5 Done Sent Done false
22414 7114 Acked Read 10 Done Sent Done false
22415 7115 Acked Read 8 Done Sent Done false
22416 7116 Acked Read 8 Done Sent Done false
22417 7117 Acked Read 6 Done Sent Done false
22418 7118 Acked Flush 0 Done Sent Done false
22419 7119 Acked Flush 0 Done Sent Done false
22420 7120 Acked Read 7 Done Sent Done false
22421 7121 Acked Flush 0 Done Sent Done false
22422 7122 Acked Write 8 Done Sent Done false
22423 7123 Acked Write 4 Done Sent Done false
22424 7124 Acked Write 1 Done Sent Done false
22425 7125 Acked Read 5 Done Sent Done false
22426 7126 Acked Read 7 Done Sent Done false
22427 7127 Acked Write 10 Done Sent Done false
22428 7128 Acked Flush 0 Done Sent Done false
22429 7129 Acked Write 4 Done Sent Done false
22430 7130 Acked Write 3 Done Sent Done false
22431 7131 Acked Read 7 Done Sent Done false
22432 7132 Acked Write 10 Done Sent Done false
22433 7133 Acked Flush 0 Done Sent Done false
22434 7134 Acked Read 10 Done Sent Done false
22435 7135 Acked Write 1 Done Sent Done false
22436 7136 Acked Read 6 Done Sent Done false
22437 7137 Acked Write 7 Done Sent Done false
22438 7138 Acked Write 1 Done Sent Done false
22439 7139 Acked Read 2 Done Sent Done false
22440 7140 Acked Read 3 Done Sent Done false
22441 7141 Acked Read 8 Done Sent Done false
22442 7142 Acked Flush 0 Done Sent Done false
22443 7143 Acked Read 5 Done Sent Done false
22444 7144 Acked Flush 0 Done Sent Done false
22445 7145 Acked Read 6 Done Sent Done false
22446 7146 Acked Flush 0 Done Sent Done false
22447 7147 Acked Read 7 Done Sent Done false
22448 7148 Acked Write 1 Done Sent Done false
22449 7149 Acked Read 4 Done Sent Done false
22450 7150 Acked Write 7 Done Sent Done false
22451 7151 Acked Read 10 Done Sent Done false
22452 7152 Acked Read 1 Done Sent Done false
22453 7153 Acked Read 2 Done Sent Done false
22454 7154 Acked Read 3 Done Sent Done false
22455 7155 Acked Flush 0 Done Sent Done false
22456 7156 Acked Read 4 Done Sent Done false
22457 7157 Acked Read 2 Done Sent Done false
22458 7158 Acked Read 8 Done Sent Done false
22459 7159 Acked Flush 0 Done Sent Done false
22460 7160 Acked Read 9 Done Sent Done false
22461 7161 Acked Flush 0 Done Sent Done false
22462 7162 Acked Flush 0 Done Sent Done false
22463 7163 Acked Read 1 Done Sent Done false
22464 7164 Acked Flush 0 Done Sent Done false
22465 7165 Acked Read 8 Done Sent Done false
22466 7166 Acked Flush 0 Done Sent Done false
22467 7167 Acked Write 4 Done Sent Done false
22468 7168 Acked Write 1 Done Sent Done false
22469 7169 Acked Read 1 Done Sent Done false
22470 7170 Acked Read 7 Done Sent Done false
22471 7171 Acked Write 9 Done Sent Done false
22472 7172 Acked Read 3 Done Sent Done false
22473 7173 Acked Write 8 Done Sent Done false
22474 7174 Acked Write 9 Done Sent Done false
22475 7175 Acked Read 4 Done Sent Done false
22476 7176 Acked Read 6 Done Sent Done false
22477 7177 Acked Write 1 Done Sent Done false
22478 7178 Acked Write 9 Done Sent Done false
22479 7179 Acked Read 1 Done Sent Done false
22480 7180 Acked Read 8 Done Sent Done false
22481 7181 Acked Read 8 Done Sent Done false
22482 7182 Acked Write 9 Done Sent Done false
22483 7183 Acked Read 9 Done Sent Done false
22484 7184 Acked Write 7 Done Sent Done false
22485 7185 Acked Read 10 Done Sent Done false
22486 7186 Acked Write 3 Done Sent Done false
22487 7187 Acked Read 5 Done Sent Done false
22488 7188 Acked Write 1 Done Sent Done false
22489 7189 Acked Write 8 Done Sent Done false
22490 7190 Acked Read 6 Done Sent Done false
22491 7191 Acked Write 2 Done Sent Done false
22492 7192 Acked Write 7 Done Sent Done false
22493 7193 Acked Write 1 Done Sent Done false
22494 7194 Acked Read 4 Done Sent Done false
22495 7195 Acked Write 10 Done Sent Done false
22496 7196 Acked Write 1 Done Sent Done false
22497 7197 Acked Flush 0 Done Sent Done false
22498 7198 Acked Read 1 Done Sent Done false
22499 7199 Acked Write 4 Done Sent Done false
22500 7200 Acked Write 2 Done Sent Done false
22501 7201 Acked Write 6 Done Sent Done false
22502 7202 Acked Read 9 Done Sent Done false
22503 7203 Acked Read 4 Done Sent Done false
22504 7204 Acked Write 3 Done Sent Done false
22505 7205 Acked Read 5 Done Sent Done false
22506 7206 Acked Write 2 Done Sent Done false
22507 7207 Acked Read 4 Done Sent Done false
22508 7208 Acked Read 8 Done Sent Done false
22509 7209 Acked Flush 0 Done Sent Done false
22510 7210 Acked Read 7 Done Sent Done false
22511 7211 Acked Write 7 Done Sent Done false
22512 7212 Acked Write 5 Done Sent Done false
22513 7213 Acked Read 2 Done Sent Done false
22514 7214 Acked Write 10 Done Sent Done false
22515 7215 Acked Write 9 Done Sent Done false
22516 7216 Acked Read 7 Done Sent Done false
22517 7217 Acked Write 3 Done Sent Done false
22518 7218 Acked Write 9 Done Sent Done false
22519 7219 Acked Flush 0 Done Sent Done false
22520 7220 Acked Read 3 Done Sent Done false
22521 7221 Acked Read 2 Done Sent Done false
22522 7222 Acked Write 1 Done Sent Done false
22523 7223 Acked Write 1 Done Sent Done false
22524 7224 Acked Flush 0 Done Sent Done false
22525 7225 Acked Write 3 Done Sent Done false
22526 7226 Acked Read 1 Done Sent Done false
22527 7227 Acked Read 7 Done Sent Done false
22528 7228 Acked Write 9 Done Sent Done false
22529 7229 Acked Read 9 Done Sent Done false
22530 7230 Acked Read 9 Done Sent Done false
22531 7231 Acked Write 10 Done Sent Done false
22532 7232 Acked Write 4 Done Sent Done false
22533 7233 Acked Read 4 Done Sent Done false
22534 7234 Acked Write 9 Done Sent Done false
22535 7235 Acked Write 3 Done Sent Done false
22536 7236 Acked Read 2 Done Sent Done false
22537 7237 Acked Read 2 Done Sent Done false
22538 7238 Acked Write 9 Done Sent Done false
22539 7239 Acked Read 8 Done Sent Done false
22540 7240 Acked Write 3 Done Sent Done false
22541 7241 Acked Read 10 Done Sent Done false
22542 7242 Acked Read 7 Done Sent Done false
22543 7243 Acked Write 1 Done Sent Done false
22544 7244 Acked Write 2 Done Sent Done false
22545 7245 Acked Read 6 Done Sent Done false
22546 7246 Acked Read 8 Done Sent Done false
22547 7247 Acked Read 8 Done Sent Done false
22548 7248 Acked Write 8 Done Sent Done false
22549 7249 Acked Write 9 Done Sent Done false
22550 7250 Acked Flush 0 Done Sent Done false
22551 7251 Acked Read 9 Done Sent Done false
22552 7252 Acked Read 1 Done Sent Done false
22553 7253 Acked Write 8 Done Sent Done false
22554 7254 Acked Read 10 Done Sent Done false
22555 7255 Acked Write 1 Done Sent Done false
22556 7256 Acked Read 9 Done Sent Done false
22557 7257 Acked Read 8 Done Sent Done false
22558 7258 Acked Write 1 Done Sent Done false
22559 7259 Acked Read 6 Done Sent Done false
22560 7260 Acked Read 7 Done Sent Done false
22561 7261 Acked Read 3 Done Sent Done false
22562 7262 Acked Read 10 Done Sent Done false
22563 7263 Acked Read 7 Done Sent Done false
22564 7264 Acked Write 4 Done Sent Done false
22565 7265 Acked Read 6 Done Sent Done false
22566 7266 Acked Read 9 Done Sent Done false
22567 7267 Acked Read 1 Done Sent Done false
22568 7268 Acked Read 4 Done Sent Done false
22569 7269 Acked Write 9 Done Sent Done false
22570 7270 Acked Write 4 Done Sent Done false
22571 7271 Acked Flush 0 Done Sent Done false
22572 7272 Acked Write 2 Done Sent Done false
22573 7273 Acked Read 3 Done Sent Done false
22574 7274 Acked Write 5 Done Sent Done false
22575 7275 Acked Flush 0 Done Sent Done false
22576 7276 Acked Write 3 Done Sent Done false
22577 7277 Acked Write 6 Done Sent Done false
22578 7278 Acked Write 1 Done Sent Done false
22579 7279 Acked Read 4 Done Sent Done false
22580 7280 Acked Read 6 Done Sent Done false
22581 7281 Acked Write 9 Done Sent Done false
22582 7282 Acked Write 10 Done Sent Done false
22583 7283 Acked Read 3 Done Sent Done false
22584 7284 Acked Flush 0 Done Sent Done false
22585 7285 Acked Write 7 Done Sent Done false
22586 7286 Acked Read 3 Done Sent Done false
22587 7287 Acked Write 8 Done Sent Done false
22588 7288 Acked Write 6 Done Sent Done false
22589 7289 Acked Read 10 Done Sent Done false
22590 7290 Acked Read 2 Done Sent Done false
22591 7291 Acked Read 9 Done Sent Done false
22592 7292 Acked Read 8 Done Sent Done false
22593 7293 Acked Read 7 Done Sent Done false
22594 7294 Acked Write 2 Done Sent Done false
22595 7295 Acked Read 8 Done Sent Done false
22596 7296 Acked Write 5 Done Sent Done false
22597 7297 Acked Write 7 Done Sent Done false
22598 7298 Acked Read 7 Done Sent Done false
22599 7299 Acked Read 3 Done Sent Done false
22600 7300 Acked Flush 0 Done Sent Done false
22601 7301 Acked Write 8 Done Sent Done false
22602 7302 Acked Flush 0 Done Sent Done false
22603 7303 Acked Flush 0 Done Sent Done false
22604 7304 Acked Read 9 Done Sent Done false
22605 7305 Acked Write 3 Done Sent Done false
22606 7306 Acked Read 7 Done Sent Done false
22607 7307 Acked Write 1 Done Sent Done false
22608 7308 Acked Write 1 Done Sent Done false
22609 7309 Acked Flush 0 Done Sent Done false
22610 7310 Acked Read 10 Done Sent Done false
22611 7311 Acked Read 4 Done Sent Done false
22612 7312 Acked Write 10 Done Sent Done false
22613 7313 Acked Read 5 Done Sent Done false
22614 7314 Acked Write 8 Done Sent Done false
22615 7315 Acked Read 1 Done Sent Done false
22616 7316 Acked Read 9 Done Sent Done false
22617 7317 Acked Read 1 Done Sent Done false
22618 7318 Acked Flush 0 Done Sent Done false
22619 7319 Acked Write 7 Done Sent Done false
22620 7320 Acked Flush 0 Done Sent Done false
22621 7321 Acked Read 5 Done Sent Done false
22622 7322 Acked Read 5 Done Sent Done false
22623 7323 Acked Read 6 Done Sent Done false
22624 7324 Acked Flush 0 Done Sent Done false
22625 7325 Acked Read 4 Done Sent Done false
22626 7326 Acked Flush 0 Done Sent Done false
22627 7327 Acked Read 7 Done Sent Done false
22628 7328 Acked Write 7 Done Sent Done false
22629 7329 Acked Read 8 Done Sent Done false
22630 7330 Acked Read 3 Done Sent Done false
22631 7331 Acked Write 2 Done Sent Done false
22632 7332 Acked Write 6 Done Sent Done false
22633 7333 Acked Read 7 Done Sent Done false
22634 7334 Acked Flush 0 Done Sent Done false
22635 7335 Acked Write 8 Done Sent Done false
22636 7336 Acked Read 5 Done Sent Done false
22637 7337 Acked Flush 0 Done Sent Done false
22638 7338 Acked Read 10 Done Sent Done false
22639 7339 Acked Flush 0 Done Sent Done false
22640 STATES DS:0 DS:1 DS:2 TOTAL
22641 Sent 0 301 0 301
22642 Done 301 0 301 602
22643 Skipped 0 0 0 0
22644 Error 0 0 0 0
22645 Last Flush: Some(JobId(7339)) Some(JobId(7038)) Some(JobId(7339))
22646 Downstairs last five completed: 7038 7037 7036 7035 7034
22647 Upstairs last five completed: 7339 7338 7337 7336 7335
22648 CLIENT: Up:0 ds:301 act:2
226492025-01-28T21:59:11.770ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
226502025-01-28T21:59:12.771ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
226512025-01-28T21:59:12.838ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
226522025-01-28T21:59:12.838ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51183) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
226532025-01-28T21:59:12.839ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
226542025-01-28T21:59:12.840ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
226552025-01-28T21:59:12.840ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
226562025-01-28T21:59:12.840ZINFOcrucible: send last flush ID to this DS: Some(JobId(7038)) = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
226572025-01-28T21:59:12.840ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
226582025-01-28T21:59:12.840ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7038)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
226592025-01-28T21:59:12.840ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7038)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
22660 ----------------------------------------------------------------
22661 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22662 Downstairs last five completed: 7339 7338 7337 7336 7335
22663 Upstairs last five completed: 7339 7338 7337 7336 7335
22664 CLIENT: Up:0 ds:0 act:3
22665 Replay: All jobs finished
22666 Main task has work to do, go find it
22667 got dsc Stop(0)
22668 stop 0
22669 Jan 28 21:59:15.082 INFO request completed[8810] Got stop action so:false kr:false
22670 , latency_us: 363, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 37d9003e-9e98-46ef-9c37-3bd075ae9953, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
226712025-01-28T21:59:15.082ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
22672 Jan 28 21:59:15.082 INFO request completed, latency_us: 135, response_code: 200, uri: /state/cid/0, method: GET, req_id: 17c4b6ce-95db-45d0-9ccd-126ca1d84771, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-28T21:59:15.082928208Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
22673 9998
226742025-01-28T21:59:15.082ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
226752025-01-28T21:59:15.083ZINFOcrucible: client task is exiting = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
22676 [8810] Exited with: ExitStatus(unix_wait_status(9))
22677 [8810][0] reports Exit
22678 Jan 28 21:59:19.085 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/0, method: GET, req_id: d52c1da9-b7a7-4830-82de-be100aa3a5ab, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
22679 001/300 Write block 2565 len 8192 data: 7 5
22680 002/300 Write block 4363 len 8192 data: 5 6
22681 003/300 Write block 4212 len 40960 data: 4 3 3 3 3 3 3 3 2 2
22682 004/300 Write block 873 len 40960 data: 6 5 5 4 4 5 5 6 6 6
22683 005/300 Read block 1985 len 24576
22684 006/300 Write block 3652 len 16384 data: 6 5 4 3
22685 007/300 Read block 2685 len 32768
22686 008/300 Read block 2389 len 16384
22687 009/300 Write block 2050 len 20480 data: 4 3 4 3 4
22688 010/300 Write block 546 len 20480 data: 6 8 8 7 6
22689 011/300 Read block 955 len 40960
22690 012/300 Flush
22691 013/300 Read block 372 len 12288
22692 014/300 Write block 1462 len 4096 data: 3
22693 015/300 Write block 4622 len 36864 data: 6 5 4 4 3 5 6 5 4
22694 016/300 Write block 2935 len 20480 data: 4 3 4 6 6
22695 017/300 Read block 2217 len 36864
22696 018/300 Write block 1243 len 36864 data: 4 4 4 4 3 4 5 5 7
22697 019/300 Read block 1112 len 28672
22698 020/300 Read block 614 len 20480
22699 021/300 Write block 577 len 12288 data: 3 4 4
22700 022/300 Write block 2253 len 8192 data: 3 2
22701 023/300 Read block 1910 len 32768
22702 024/300 Write block 4248 len 28672 data: 3 4 5 6 6 8 6
22703 025/300 Read block 4825 len 32768
22704 026/300 Read block 4734 len 24576
22705 027/300 Write block 3632 len 32768 data: 4 6 4 6 6 6 5 4
22706 028/300 Read block 4444 len 4096
22707 029/300 Write block 2082 len 40960 data: 7 6 6 5 5 5 5 4 5 5
22708 030/300 Write block 1235 len 20480 data: 3 3 3 3 2
22709 031/300 Read block 4240 len 4096
22710 032/300 Read block 2689 len 36864
22711 033/300 Write block 3081 len 4096 data: 8
22712 034/300 Read block 1999 len 40960
22713 035/300 Write block 2582 len 8192 data: 3 3
22714 036/300 Read block 4740 len 16384
22715 037/300 Write block 2902 len 24576 data: 7 7 6 6 4 5
22716 038/300 Read block 1689 len 28672
22717 039/300 Write block 4308 len 4096 data: 4
22718 040/300 Read block 484 len 40960
22719 041/300 Write block 212 len 20480 data: 6 5 4 5 5
22720 042/300 Flush
22721 043/300 Read block 2191 len 28672
22722 044/300 Read block 3058 len 40960
22723 045/300 Read block 2499 len 8192
22724 046/300 Write block 165 len 4096 data: 3
22725 047/300 Write block 2594 len 16384 data: 4 6 4 4
22726 048/300 Read block 3401 len 36864
22727 049/300 Write block 4551 len 28672 data: 5 6 5 4 5 4 4
22728 050/300 Read block 2457 len 28672
22729 051/300 Write block 2837 len 4096 data: 2
22730 052/300 Write block 2347 len 28672 data: 7 5 5 6 5 5 5
22731 053/300 Read block 3764 len 4096
22732 054/300 Write block 1316 len 24576 data: 6 4 5 4 4 4
22733 055/300 Read block 3279 len 16384
22734 056/300 Write block 3668 len 20480 data: 7 6 6 6 6
22735 057/300 Read block 1687 len 36864
22736 058/300 Write block 4423 len 16384 data: 5 5 5 9
22737 059/300 Read block 4758 len 24576
22738 060/300 Flush
22739 061/300 Write block 2051 len 36864 data: 4 5 4 5 4 4 5 5 4
22740 062/300 Write block 4570 len 8192 data: 7 7
22741 063/300 Write block 3212 len 36864 data: 4 6 6 6 5 4 5 5 5
22742 064/300 Read block 2351 len 40960
22743 065/300 Write block 958 len 16384 data: 4 5 4 4
22744 066/300 Write block 4132 len 24576 data: 4 5 8 8 7 8
22745 067/300 Read block 18 len 16384
22746 068/300 Read block 130 len 36864
22747 069/300 Write block 242 len 24576 data: 8 8 7 7 7 6
22748 070/300 Read block 2593 len 8192
22749 071/300 Write block 2125 len 4096 data: 6
22750 072/300 Read block 2263 len 36864
22751 073/300 Write block 1568 len 40960 data: 4 4 3 3 3 4 5 5 6 6
22752 074/300 Write block 1318 len 24576 data: 6 5 5 5 4 3
22753 075/300 Read block 4205 len 12288
22754 076/300 Read block 128 len 32768
22755 077/300 Write block 2515 len 40960 data: 5 5 4 5 4 4 4 4 4 3
22756 078/300 Write block 193 len 28672 data: 4 3 5 5 5 5 5
22757 079/300 Write block 4594 len 28672 data: 4 3 3 4 5 6 6
22758 080/300 Write block 4705 len 24576 data: 3 3 5 6 7 7
22759 081/300 Read block 4295 len 20480
22760 082/300 Read block 4858 len 20480
22761 083/300 Read block 4653 len 40960
22762 084/300 Write block 1024 len 12288 data: 6 5 5
22763 085/300 Write block 3923 len 4096 data: 6
22764 086/300 Read block 1886 len 16384
22765 087/300 Read block 3456 len 8192
22766 088/300 Write block 3500 len 36864 data: 3 4 3 3 3 4 4 4 3
22767 089/300 Flush
22768 090/300 Write block 2727 len 36864 data: 4 5 6 8 7 6 6 7 7
22769 091/300 Write block 3623 len 8192 data: 6 7
22770 092/300 Write block 4839 len 24576 data: 5 6 7 7 6 6
22771 093/300 Flush
22772 094/300 Write block 567 len 16384 data: 5 5 3 3
22773 095/300 Read block 4747 len 36864
22774 096/300 Read block 1322 len 40960
22775 097/300 Write block 1342 len 36864 data: 4 4 7 6 5 6 5 4 4
22776 098/300 Write block 4142 len 28672 data: 6 3 4 4 3 3 3
22777 099/300 Read block 4170 len 32768
22778 100/300 Read block 4699 len 4096
22779 101/300 Write block 694 len 20480 data: 6 4 3 3 3
22780 102/300 Read block 2134 len 28672
22781 103/300 Write block 902 len 32768 data: 7 7 7 7 7 6 5 3
22782 104/300 Write block 3024 len 36864 data: 6 7 5 3 4 3 3 4 4
22783 105/300 Write block 3680 len 4096 data: 3
22784 106/300 Write block 58 len 40960 data: 5 5 5 6 7 6 8 6 6 5
22785 107/300 Flush
22786 108/300 Write block 2130 len 16384 data: 6 6 6 8
22787 109/300 Flush
22788 110/300 Read block 4372 len 20480
22789 111/300 Write block 3166 len 20480 data: 2 2 3 3 3
22790 112/300 Write block 1029 len 8192 data: 5 4
22791 113/300 Flush
22792 114/300 Flush
22793 115/300 Read block 562 len 4096
22794 116/300 Read block 4435 len 20480
22795 117/300 Read block 4769 len 4096
22796 118/300 Write block 4172 len 20480 data: 8 6 6 6 5
22797 119/300 Write block 4364 len 4096 data: 7
22798 120/300 Write block 952 len 4096 data: 5
22799 121/300 Write block 1697 len 40960 data: 4 7 7 8 7 7 8 8 8 7
22800 122/300 Read block 239 len 16384
22801 123/300 Read block 1920 len 24576
22802 124/300 Flush
22803 125/300 Write block 2106 len 24576 data: 3 4 4 4 4 3
22804 126/300 Read block 266 len 28672
22805 127/300 Read block 3881 len 20480
22806 128/300 Flush
22807 129/300 Read block 4774 len 24576
22808 130/300 Read block 2708 len 40960
22809 131/300 Flush
22810 132/300 Read block 1603 len 4096
22811 133/300 Read block 878 len 16384
22812 134/300 Read block 4582 len 24576
22813 135/300 Write block 3953 len 20480 data: 2 3 3 4 5
22814 136/300 Read block 3367 len 20480
22815 137/300 Read block 3456 len 20480
22816 138/300 Write block 1677 len 20480 data: 4 4 4 3 4
22817 139/300 Write block 4619 len 36864 data: 7 7 7 7 6 5 5 4 6
22818 140/300 Read block 1183 len 36864
22819 141/300 Write block 4770 len 32768 data: 9 11 11 11 9 8 7 5
22820 142/300 Write block 3353 len 32768 data: 3 3 4 5 4 5 5 6
22821 143/300 Write block 1431 len 12288 data: 6 7 7
22822 144/300 Write block 901 len 4096 data: 7
22823 145/300 Flush
22824 146/300 Flush
22825 147/300 Read block 1074 len 8192
22826 148/300 Write block 4357 len 20480 data: 6 6 6 6 5
22827 149/300 Read block 1306 len 12288
22828 150/300 Read block 1011 len 16384
22829 151/300 Write block 41 len 12288 data: 6 6 7
22830 152/300 Write block 3934 len 4096 data: 4
22831 153/300 Read block 1232 len 8192
22832 154/300 Read block 4844 len 40960
22833 155/300 Write block 1903 len 24576 data: 3 5 4 5 4 4
22834 156/300 Write block 2965 len 28672 data: 3 3 5 5 5 6 7
22835 157/300 Write block 3365 len 20480 data: 4 6 5 6 5
22836 158/300 Flush
22837 159/300 Write block 3219 len 16384 data: 6 6 6 5
22838 160/300 Read block 1548 len 12288
22839 161/300 Read block 4039 len 20480
22840 162/300 Write block 3665 len 4096 data: 5
22841 163/300 Write block 2130 len 40960 data: 7 7 7 9 9 8 5 5 6 4
22842 164/300 Write block 1631 len 8192 data: 7 6
22843 165/300 Read block 3852 len 4096
22844 166/300 Read block 3695 len 24576
22845 167/300 Read block 767 len 24576
22846 168/300 Read block 3714 len 4096
22847 169/300 Read block 4844 len 40960
22848 170/300 Read block 2469 len 32768
22849 171/300 Write block 2912 len 20480 data: 5 5 7 5 3
22850 172/300 Write block 3469 len 32768 data: 4 6 6 6 6 6 5 4
22851 173/300 Read block 2262 len 16384
22852 174/300 Read block 2819 len 32768
22853 175/300 Read block 3208 len 16384
22854 176/300 Read block 4724 len 36864
22855 177/300 Write block 1533 len 4096 data: 5
22856 178/300 Write block 1333 len 28672 data: 6 6 6 6 5 5 5
22857 179/300 Read block 3719 len 28672
22858 180/300 Write block 2465 len 4096 data: 5
22859 181/300 Write block 4993 len 12288 data: 3 3 2
22860 182/300 Write block 3210 len 12288 data: 3 4 5
22861 183/300 Read block 3178 len 28672
22862 184/300 Read block 1034 len 4096
22863 185/300 Write block 4153 len 36864 data: 3 3 3 5 6 5 5 7 7
22864 186/300 Write block 1490 len 36864 data: 7 6 4 4 3 3 4 3 2
22865 187/300 Read block 3744 len 40960
22866 188/300 Read block 1023 len 20480
22867 189/300 Write block 2975 len 28672 data: 3 3 4 5 6 5 5
22868 190/300 Write block 625 len 36864 data: 5 4 3 4 4 3 3 3 3
22869 191/300 Read block 1435 len 28672
22870 192/300 Write block 4750 len 16384 data: 4 4 4 7
22871 193/300 Flush
22872 194/300 Read block 2137 len 24576
22873 195/300 Read block 557 len 28672
22874 196/300 Read block 1212 len 20480
22875 197/300 Write block 2522 len 16384 data: 5 5 4 2
22876 198/300 Read block 640 len 24576
22877 199/300 Write block 2883 len 4096 data: 3
22878 200/300 Read block 3661 len 8192
22879 201/300 Read block 2996 len 36864
22880 202/300 Write block 4083 len 24576 data: 8 7 7 8 8 7
22881 203/300 Write block 1481 len 20480 data: 5 6 5 5 5
22882 204/300 Flush
22883 205/300 Write block 4426 len 12288 data: 10 7 7
22884 206/300 Write block 2133 len 24576 data: 10 10 9 6 6 7
22885 207/300 Read block 4302 len 28672
22886 208/300 Read block 4621 len 28672
22887 209/300 Flush
22888 210/300 Write block 1492 len 36864 data: 5 5 4 4 5 4 3 2 2
22889 211/300 Read block 1690 len 16384
22890 212/300 Read block 3724 len 36864
22891 213/300 Read block 4335 len 4096
22892 214/300 Flush
22893 215/300 Write block 943 len 12288 data: 7 6 6
22894 216/300 Write block 4956 len 24576 data: 4 4 2 4 4 5
22895 217/300 Read block 3074 len 16384
22896 218/300 Read block 1775 len 12288
22897 219/300 Read block 3139 len 28672
22898 220/300 Read block 1831 len 20480
22899 221/300 Write block 4351 len 40960 data: 5 6 6 5 6 6 7 7 7 7
22900 222/300 Write block 1089 len 24576 data: 3 3 3 2 2 3
22901 223/300 Flush
22902 224/300 Read block 4042 len 8192
22903 225/300 Write block 4307 len 16384 data: 5 5 4 6
22904 226/300 Read block 2680 len 20480
22905 227/300 Write block 1293 len 32768 data: 5 5 6 6 7 8 7 7
22906 228/300 Flush
22907 229/300 Flush
22908 230/300 Read block 3077 len 24576
22909 231/300 Read block 1693 len 16384
22910 232/300 Read block 2260 len 32768
22911 233/300 Read block 1194 len 32768
22912 234/300 Write block 4550 len 20480 data: 6 6 7 6 5
22913 235/300 Read block 1973 len 36864
22914 236/300 Write block 735 len 20480 data: 5 4 6 7 6
22915 237/300 Read block 1299 len 40960
22916 238/300 Write block 3427 len 32768 data: 2 2 2 3 4 4 5 4
22917 239/300 Read block 2060 len 36864
22918 240/300 Write block 706 len 16384 data: 5 6 5 5
22919 241/300 Write block 1873 len 8192 data: 4 2
22920 242/300 Read block 1865 len 32768
22921 243/300 Write block 4775 len 20480 data: 9 8 6 6 6
22922 244/300 Read block 2479 len 8192
22923 245/300 Write block 1983 len 36864 data: 6 6 6 5 6 6 5 4 4
22924 246/300 Write block 3955 len 12288 data: 4 5 6
22925 247/300 Read block 390 len 40960
22926 248/300 Read block 2501 len 28672
22927 249/300 Write block 1946 len 8192 data: 4 4
22928 250/300 Write block 4336 len 36864 data: 6 7 7 7 5 5 5 3 3
22929 251/300 Write block 3725 len 28672 data: 7 7 9 11 8 8 6
22930 252/300 Flush
22931 253/300 Read block 614 len 20480
22932 254/300 Read block 3386 len 12288
22933 255/300 Read block 1639 len 24576
22934 256/300 Read block 4476 len 28672
22935 257/300 Write block 706 len 24576 data: 6 7 6 6 6 7
22936 258/300 Read block 38 len 16384
22937 259/300 Read block 827 len 20480
22938 260/300 Read block 776 len 36864
22939 261/300 Write block 202 len 24576 data: 6 5 5 6 7 6
22940 262/300 Flush
22941 263/300 Read block 3315 len 16384
22942 264/300 Read block 2708 len 8192
22943 265/300 Read block 493 len 16384
22944 266/300 Read block 1304 len 20480
22945 267/300 Read block 29 len 20480
22946 268/300 Read block 4044 len 24576
22947 269/300 Read block 1625 len 4096
22948 270/300 Read block 4357 len 40960
22949 271/300 Write block 1707 len 28672 data: 6 7 5 5 5 6 7
22950 272/300 Write block 3636 len 24576 data: 7 7 6 5 5 6
22951 273/300 Write block 4772 len 24576 data: 12 12 10 10 9 7
22952 274/300 Write block 73 len 36864 data: 5 6 5 5 5 6 6 6 6
22953 275/300 Write block 694 len 32768 data: 7 5 4 4 4 5 5 5
22954 276/300 Write block 4766 len 16384 data: 6 6 5 7
22955 277/300 Read block 1276 len 12288
22956 278/300 Read block 4082 len 4096
22957 279/300 Write block 4057 len 4096 data: 4
22958 280/300 Read block 3745 len 36864
22959 281/300 Write block 3259 len 24576 data: 6 2 3 4 6 6
22960 282/300 Read block 745 len 24576
22961 283/300 Write block 1230 len 8192 data: 4 4
22962 284/300 Read block 1014 len 36864
22963 285/300 Read block 4197 len 12288
22964 286/300 Read block 2548 len 4096
22965 287/300 Write block 3532 len 32768 data: 6 5 6 5 4 5 5 6
22966 288/300 Write block 3014 len 12288 data: 3 3 4
22967 289/300 Read block 3851 len 24576
22968 290/300 Write block 356 len 16384 data: 6 5 6 5
22969 291/300 Write block 2476 len 32768 data: 7 9 7 6 5 6 5 5
22970 292/300 Write block 1021 len 36864 data: 7 7 5 7 6 6 5 4 6
22971 293/300 Write block 968 len 20480 data: 5 5 5 2 3
22972 294/300 Flush
22973 295/300 Write block 2120 len 4096 data: 3
22974 296/300 Read block 2070 len 24576
22975 297/300 Write block 4148 len 28672 data: 4 3 3 3 3 4 4
22976 298/300 Read block 2189 len 20480
22977 299/300 Read block 1035 len 36864
22978 300/300 Read block 1040 len 24576
22979 301/300 Read block 596 len 36864
22980 Main task has work to do, go find it
22981 got dsc Start(0)
22982 start 0
22983 Jan 28 21:59:19.323 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/0, method: POST, req_id: e2f7a103-56bd-4b8e-9063-a2ca1ebb5c39, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
22984 Replay: started 0, returned:Ok(())
22985 ----------------------------------------------------------------
22986 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22987 States: Offline Active Active
22988 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22989 7340 Acked Write 2 Sent Done Done false
22990 7341 Acked Write 2 Sent Done Done false
22991 7342 Acked Write 10 Sent Done Done false
22992 7343 Acked Write 10 Sent Done Done false
22993 7344 Acked Read 6 Sent Done Done false
22994 7345 Acked Write 4 Sent Done Done false
22995 7346 Acked Read 8 Sent Done Done false
22996 7347 Acked Read 4 Sent Done Done false
22997 7348 Acked Write 5 Sent Done Done false
22998 7349 Acked Write 5 Sent Done Done false
22999 7350 Acked Read 10 Sent Done Done false
23000 7351 Acked Flush 0 Sent Done Done false
23001 7352 Acked Read 3 Sent Done Done false
23002 7353 Acked Write 1 Sent Done Done false
23003 7354 Acked Write 9 Sent Done Done false
23004 7355 Acked Write 5 Sent Done Done false
23005 7356 Acked Read 9 Sent Done Done false
23006 7357 Acked Write 9 Sent Done Done false
23007 7358 Acked Read 7 Sent Done Done false
23008 7359 Acked Read 5 Sent Done Done false
23009 7360 Acked Write 3 Sent Done Done false
23010 7361 Acked Write 2 Sent Done Done false
23011 7362 Acked Read 8 Sent Done Done false
23012 7363 Acked Write 7 Sent Done Done false
23013 7364 Acked Read 8 Sent Done Done false
23014 7365 Acked Read 6 Sent Done Done false
23015 7366 Acked Write 8 Sent Done Done false
23016 7367 Acked Read 1 Sent Done Done false
23017 7368 Acked Write 10 Sent Done Done false
23018 7369 Acked Write 5 Sent Done Done false
23019 7370 Acked Read 1 Sent Done Done false
23020 7371 Acked Read 9 Sent Done Done false
23021 7372 Acked Write 1 Sent Done Done false
23022 7373 Acked Read 10 Sent Done Done false
23023 7374 Acked Write 2 Sent Done Done false
23024 7375 Acked Read 4 Sent Done Done false
23025 7376 Acked Write 6 Sent Done Done false
23026 7377 Acked Read 7 Sent Done Done false
23027 7378 Acked Write 1 Sent Done Done false
23028 7379 Acked Read 10 Sent Done Done false
23029 7380 Acked Write 5 Sent Done Done false
23030 7381 Acked Flush 0 Sent Done Done false
23031 7382 Acked Read 7 Sent Done Done false
23032 7383 Acked Read 10 Sent Done Done false
23033 7384 Acked Read 2 Sent Done Done false
23034 7385 Acked Write 1 Sent Done Done false
23035 7386 Acked Write 4 Sent Done Done false
23036 7387 Acked Read 9 Sent Done Done false
23037 7388 Acked Write 7 Sent Done Done false
23038 7389 Acked Read 7 Sent Done Done false
23039 7390 Acked Write 1 Sent Done Done false
23040 7391 Acked Write 7 Sent Done Done false
23041 7392 Acked Read 1 Sent Done Done false
23042 7393 Acked Write 6 Sent Done Done false
23043 7394 Acked Read 4 Sent Done Done false
23044 7395 Acked Write 5 Sent Done Done false
23045 7396 Acked Read 9 Sent Done Done false
23046 7397 Acked Write 4 Sent Done Done false
23047 7398 Acked Read 6 Sent Done Done false
23048 7399 Acked Flush 0 Sent Done Done false
23049 7400 Acked Write 9 Sent Done Done false
23050 7401 Acked Write 2 Sent Done Done false
23051 7402 Acked Write 9 Sent Done Done false
23052 7403 Acked Read 10 Sent Done Done false
23053 7404 Acked Write 4 Sent Done Done false
23054 7405 Acked Write 6 Sent Done Done false
23055 7406 Acked Read 4 Sent Done Done false
23056 7407 Acked Read 9 Sent Done Done false
23057 7408 Acked Write 6 Sent Done Done false
23058 7409 Acked Read 2 Sent Done Done false
23059 7410 Acked Write 1 Sent Done Done false
23060 7411 Acked Read 9 Sent Done Done false
23061 7412 Acked Write 10 Sent Done Done false
23062 7413 Acked Write 6 Sent Done Done false
23063 7414 Acked Read 3 Sent Done Done false
23064 7415 Acked Read 8 Sent Done Done false
23065 7416 Acked Write 10 Sent Done Done false
23066 7417 Acked Write 7 Sent Done Done false
23067 7418 Acked Write 7 Sent Done Done false
23068 7419 Acked Write 6 Sent Done Done false
23069 7420 Acked Read 5 Sent Done Done false
23070 7421 Acked Read 5 Sent Done Done false
23071 7422 Acked Read 10 Sent Done Done false
23072 7423 Acked Write 3 Sent Done Done false
23073 7424 Acked Write 1 Sent Done Done false
23074 7425 Acked Read 4 Sent Done Done false
23075 7426 Acked Read 2 Sent Done Done false
23076 7427 Acked Write 9 Sent Done Done false
23077 7428 Acked Flush 0 Sent Done Done false
23078 7429 Acked Write 9 Sent Done Done false
23079 7430 Acked Write 2 Sent Done Done false
23080 7431 Acked Write 6 Sent Done Done false
23081 7432 Acked Flush 0 Sent Done Done false
23082 7433 Acked Write 4 Sent Done Done false
23083 7434 Acked Read 9 Sent Done Done false
23084 7435 Acked Read 10 Sent Done Done false
23085 7436 Acked Write 9 Sent Done Done false
23086 7437 Acked Write 7 Sent Done Done false
23087 7438 Acked Read 8 Sent Done Done false
23088 7439 Acked Read 1 Sent Done Done false
23089 7440 Acked Write 5 Sent Done Done false
23090 7441 Acked Read 7 Sent Done Done false
23091 7442 Acked Write 8 Sent Done Done false
23092 7443 Acked Write 9 Sent Done Done false
23093 7444 Acked Write 1 Sent Done Done false
23094 7445 Acked Write 10 Sent Done Done false
23095 7446 Acked Flush 0 Sent Done Done false
23096 7447 Acked Write 4 Sent Done Done false
23097 7448 Acked Flush 0 Sent Done Done false
23098 7449 Acked Read 5 Sent Done Done false
23099 7450 Acked Write 5 Sent Done Done false
23100 7451 Acked Write 2 Sent Done Done false
23101 7452 Acked Flush 0 Sent Done Done false
23102 7453 Acked Flush 0 Sent Done Done false
23103 7454 Acked Read 1 Sent Done Done false
23104 7455 Acked Read 5 Sent Done Done false
23105 7456 Acked Read 1 Sent Done Done false
23106 7457 Acked Write 5 Sent Done Done false
23107 7458 Acked Write 1 Sent Done Done false
23108 7459 Acked Write 1 Sent Done Done false
23109 7460 Acked Write 10 Sent Done Done false
23110 7461 Acked Read 4 Sent Done Done false
23111 7462 Acked Read 6 Sent Done Done false
23112 7463 Acked Flush 0 Sent Done Done false
23113 7464 Acked Write 6 Sent Done Done false
23114 7465 Acked Read 7 Sent Done Done false
23115 7466 Acked Read 5 Sent Done Done false
23116 7467 Acked Flush 0 Sent Done Done false
23117 7468 Acked Read 6 Sent Done Done false
23118 7469 Acked Read 10 Sent Done Done false
23119 7470 Acked Flush 0 Sent Done Done false
23120 7471 Acked Read 1 Sent Done Done false
23121 7472 Acked Read 4 Sent Done Done false
23122 7473 Acked Read 6 Sent Done Done false
23123 7474 Acked Write 5 Sent Done Done false
23124 7475 Acked Read 5 Sent Done Done false
23125 7476 Acked Read 5 Sent Done Done false
23126 7477 Acked Write 5 Sent Done Done false
23127 7478 Acked Write 9 Sent Done Done false
23128 7479 Acked Read 9 Sent Done Done false
23129 7480 Acked Write 8 Sent Done Done false
23130 7481 Acked Write 8 Sent Done Done false
23131 7482 Acked Write 3 Sent Done Done false
23132 7483 Acked Write 1 Sent Done Done false
23133 7484 Acked Flush 0 Sent Done Done false
23134 7485 Acked Flush 0 Sent Done Done false
23135 7486 Acked Read 2 Sent Done Done false
23136 7487 Acked Write 5 Sent Done Done false
23137 7488 Acked Read 3 Sent Done Done false
23138 7489 Acked Read 4 Sent Done Done false
23139 7490 Acked Write 3 Sent Done Done false
23140 7491 Acked Write 1 Sent Done Done false
23141 7492 Acked Read 2 Sent Done Done false
23142 7493 Acked Read 10 Sent Done Done false
23143 7494 Acked Write 6 Sent Done Done false
23144 7495 Acked Write 7 Sent Done Done false
23145 7496 Acked Write 5 Sent Done Done false
23146 7497 Acked Flush 0 Sent Done Done false
23147 7498 Acked Write 4 Sent Done Done false
23148 7499 Acked Read 3 Sent Done Done false
23149 7500 Acked Read 5 Sent Done Done false
23150 7501 Acked Write 1 Sent Done Done false
23151 7502 Acked Write 10 Sent Done Done false
23152 7503 Acked Write 2 Sent Done Done false
23153 7504 Acked Read 1 Sent Done Done false
23154 7505 Acked Read 6 Sent Done Done false
23155 7506 Acked Read 6 Sent Done Done false
23156 7507 Acked Read 1 Sent Done Done false
23157 7508 Acked Read 10 Sent Done Done false
23158 7509 Acked Read 8 Sent Done Done false
23159 7510 Acked Write 5 Sent Done Done false
23160 7511 Acked Write 8 Sent Done Done false
23161 7512 Acked Read 4 Sent Done Done false
23162 7513 Acked Read 8 Sent Done Done false
23163 7514 Acked Read 4 Sent Done Done false
23164 7515 Acked Read 9 Sent Done Done false
23165 7516 Acked Write 1 Sent Done Done false
23166 7517 Acked Write 7 Sent Done Done false
23167 7518 Acked Read 7 Sent Done Done false
23168 7519 Acked Write 1 Sent Done Done false
23169 7520 Acked Write 3 Sent Done Done false
23170 7521 Acked Write 3 Sent Done Done false
23171 7522 Acked Read 7 Sent Done Done false
23172 7523 Acked Read 1 Sent Done Done false
23173 7524 Acked Write 9 Sent Done Done false
23174 7525 Acked Write 9 Sent Done Done false
23175 7526 Acked Read 10 Sent Done Done false
23176 7527 Acked Read 5 Sent Done Done false
23177 7528 Acked Write 7 Sent Done Done false
23178 7529 Acked Write 9 Sent Done Done false
23179 7530 Acked Read 7 Sent Done Done false
23180 7531 Acked Write 4 Sent Done Done false
23181 7532 Acked Flush 0 Sent Done Done false
23182 7533 Acked Read 6 Sent Done Done false
23183 7534 Acked Read 7 Sent Done Done false
23184 7535 Acked Read 5 Sent Done Done false
23185 7536 Acked Write 4 Sent Done Done false
23186 7537 Acked Read 6 Sent Done Done false
23187 7538 Acked Write 1 Sent Done Done false
23188 7539 Acked Read 2 Sent Done Done false
23189 7540 Acked Read 9 Sent Done Done false
23190 7541 Acked Write 6 Sent Done Done false
23191 7542 Acked Write 5 Sent Done Done false
23192 7543 Acked Flush 0 Sent Done Done false
23193 7544 Acked Write 3 Sent Done Done false
23194 7545 Acked Write 6 Sent Done Done false
23195 7546 Acked Read 7 Sent Done Done false
23196 7547 Acked Read 7 Sent Done Done false
23197 7548 Acked Flush 0 Sent Done Done false
23198 7549 Acked Write 9 Sent Done Done false
23199 7550 Acked Read 4 Sent Done Done false
23200 7551 Acked Read 9 Sent Done Done false
23201 7552 Acked Read 1 Sent Done Done false
23202 7553 Acked Flush 0 Sent Done Done false
23203 7554 Acked Write 3 Sent Done Done false
23204 7555 Acked Write 6 Sent Done Done false
23205 7556 Acked Read 4 Sent Done Done false
23206 7557 Acked Read 3 Sent Done Done false
23207 7558 Acked Read 7 Sent Done Done false
23208 7559 Acked Read 5 Sent Done Done false
23209 7560 Acked Write 10 Sent Done Done false
23210 7561 Acked Write 6 Sent Done Done false
23211 7562 Acked Flush 0 Sent Done Done false
23212 7563 Acked Read 2 Sent Done Done false
23213 7564 Acked Write 4 Sent Done Done false
23214 7565 Acked Read 5 Sent Done Done false
23215 7566 Acked Write 8 Sent Done Done false
23216 7567 Acked Flush 0 Sent Done Done false
23217 7568 Acked Flush 0 Sent Done Done false
23218 7569 Acked Read 6 Sent Done Done false
23219 7570 Acked Read 4 Sent Done Done false
23220 7571 Acked Read 8 Sent Done Done false
23221 7572 Acked Read 8 Sent Done Done false
23222 7573 Acked Write 5 Sent Done Done false
23223 7574 Acked Read 9 Sent Done Done false
23224 7575 Acked Write 5 Sent Done Done false
23225 7576 Acked Read 10 Sent Done Done false
23226 7577 Acked Write 8 Sent Done Done false
23227 7578 Acked Read 9 Sent Done Done false
23228 7579 Acked Write 4 Sent Done Done false
23229 7580 Acked Write 2 Sent Done Done false
23230 7581 Acked Read 8 Sent Done Done false
23231 7582 Acked Write 5 Sent Done Done false
23232 7583 Acked Read 2 Sent Done Done false
23233 7584 Acked Write 9 Sent Done Done false
23234 7585 Acked Write 3 Sent Done Done false
23235 7586 Acked Read 10 Sent Done Done false
23236 7587 Acked Read 7 Sent Done Done false
23237 7588 Acked Write 2 Sent Done Done false
23238 7589 Acked Write 9 Sent Done Done false
23239 7590 Acked Write 7 Sent Done Done false
23240 7591 Acked Flush 0 Sent Done Done false
23241 7592 Acked Read 5 Sent Done Done false
23242 7593 Acked Read 3 Sent Done Done false
23243 7594 Acked Read 6 Sent Done Done false
23244 7595 Acked Read 7 Sent Done Done false
23245 7596 Acked Write 6 Sent Done Done false
23246 7597 Acked Read 4 Sent Done Done false
23247 7598 Acked Read 5 Sent Done Done false
23248 7599 Acked Read 9 Sent Done Done false
23249 7600 Acked Write 6 Sent Done Done false
23250 7601 Acked Flush 0 Sent Done Done false
23251 7602 Acked Read 4 Sent Done Done false
23252 7603 Acked Read 2 Sent Done Done false
23253 7604 Acked Read 4 Sent Done Done false
23254 7605 Acked Read 5 Sent Done Done false
23255 7606 Acked Read 5 Sent Done Done false
23256 7607 Acked Read 6 Sent Done Done false
23257 7608 Acked Read 1 Sent Done Done false
23258 7609 Acked Read 10 Sent Done Done false
23259 7610 Acked Write 7 Sent Done Done false
23260 7611 Acked Write 6 Sent Done Done false
23261 7612 Acked Write 6 Sent Done Done false
23262 7613 Acked Write 9 Sent Done Done false
23263 7614 Acked Write 8 Sent Done Done false
23264 7615 Acked Write 4 Sent Done Done false
23265 7616 Acked Read 3 Sent Done Done false
23266 7617 Acked Read 1 Sent Done Done false
23267 7618 Acked Write 1 Sent Done Done false
23268 7619 Acked Read 9 Sent Done Done false
23269 7620 Acked Write 6 Sent Done Done false
23270 7621 Acked Read 6 Sent Done Done false
23271 7622 Acked Write 2 Sent Done Done false
23272 7623 Acked Read 9 Sent Done Done false
23273 7624 Acked Read 3 Sent Done Done false
23274 7625 Acked Read 1 Sent Done Done false
23275 7626 Acked Write 8 Sent Done Done false
23276 7627 Acked Write 3 Sent Done Done false
23277 7628 Acked Read 6 Sent Done Done false
23278 7629 Acked Write 4 Sent Done Done false
23279 7630 Acked Write 8 Sent Done Done false
23280 7631 Acked Write 9 Sent Done Done false
23281 7632 Acked Write 5 Sent Done Done false
23282 7633 Acked Flush 0 Sent Done Done false
23283 7634 Acked Write 1 Sent Done Done false
23284 7635 Acked Read 6 Sent Done Done false
23285 7636 Acked Write 7 Sent Done Done false
23286 7637 Acked Read 5 Sent Done Done false
23287 7638 Acked Read 9 Sent Done Done false
23288 7639 Acked Read 6 Sent Done Done false
23289 7640 Acked Read 9 Sent Done Done false
23290 STATES DS:0 DS:1 DS:2 TOTAL
23291 Sent 301 0 0 301
23292 Done 0 301 301 602
23293 Skipped 0 0 0 0
23294 Error 0 0 0 0
23295 Last Flush: Some(JobId(7339)) Some(JobId(7633)) Some(JobId(7633))
23296 Downstairs last five completed: 7339 7338 7337 7336 7335
23297 Upstairs last five completed: 7640 7639 7638 7637 7636
23298 CLIENT: Up:0 ds:301 act:2
232992025-01-28T21:59:19.825ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
23300 [8810] got start action
23301 [8810] I am going to restart
23302 Starting downstairs at port 8810
23303 Make output file at "/tmp/dsc/downstairs-8810.txt"
23304 [8810][0] reports Starting
23305 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1224)
23306 [8810][0] reports Running
233072025-01-28T21:59:21.104ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
233082025-01-28T21:59:21.783ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
233092025-01-28T21:59:22.785ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
23310 ----------------------------------------------------------------
23311 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23312 States: Offline Active Active
23313 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23314 7340 Acked Write 2 Sent Done Done false
23315 7341 Acked Write 2 Sent Done Done false
23316 7342 Acked Write 10 Sent Done Done false
23317 7343 Acked Write 10 Sent Done Done false
23318 7344 Acked Read 6 Sent Done Done false
23319 7345 Acked Write 4 Sent Done Done false
23320 7346 Acked Read 8 Sent Done Done false
23321 7347 Acked Read 4 Sent Done Done false
23322 7348 Acked Write 5 Sent Done Done false
23323 7349 Acked Write 5 Sent Done Done false
23324 7350 Acked Read 10 Sent Done Done false
23325 7351 Acked Flush 0 Sent Done Done false
23326 7352 Acked Read 3 Sent Done Done false
23327 7353 Acked Write 1 Sent Done Done false
23328 7354 Acked Write 9 Sent Done Done false
23329 7355 Acked Write 5 Sent Done Done false
23330 7356 Acked Read 9 Sent Done Done false
23331 7357 Acked Write 9 Sent Done Done false
23332 7358 Acked Read 7 Sent Done Done false
23333 7359 Acked Read 5 Sent Done Done false
23334 7360 Acked Write 3 Sent Done Done false
23335 7361 Acked Write 2 Sent Done Done false
23336 7362 Acked Read 8 Sent Done Done false
23337 7363 Acked Write 7 Sent Done Done false
23338 7364 Acked Read 8 Sent Done Done false
23339 7365 Acked Read 6 Sent Done Done false
23340 7366 Acked Write 8 Sent Done Done false
23341 7367 Acked Read 1 Sent Done Done false
23342 7368 Acked Write 10 Sent Done Done false
23343 7369 Acked Write 5 Sent Done Done false
23344 7370 Acked Read 1 Sent Done Done false
23345 7371 Acked Read 9 Sent Done Done false
23346 7372 Acked Write 1 Sent Done Done false
23347 7373 Acked Read 10 Sent Done Done false
23348 7374 Acked Write 2 Sent Done Done false
23349 7375 Acked Read 4 Sent Done Done false
23350 7376 Acked Write 6 Sent Done Done false
23351 7377 Acked Read 7 Sent Done Done false
23352 7378 Acked Write 1 Sent Done Done false
23353 7379 Acked Read 10 Sent Done Done false
23354 7380 Acked Write 5 Sent Done Done false
23355 7381 Acked Flush 0 Sent Done Done false
23356 7382 Acked Read 7 Sent Done Done false
23357 7383 Acked Read 10 Sent Done Done false
23358 7384 Acked Read 2 Sent Done Done false
23359 7385 Acked Write 1 Sent Done Done false
23360 7386 Acked Write 4 Sent Done Done false
23361 7387 Acked Read 9 Sent Done Done false
23362 7388 Acked Write 7 Sent Done Done false
23363 7389 Acked Read 7 Sent Done Done false
23364 7390 Acked Write 1 Sent Done Done false
23365 7391 Acked Write 7 Sent Done Done false
23366 7392 Acked Read 1 Sent Done Done false
23367 7393 Acked Write 6 Sent Done Done false
23368 7394 Acked Read 4 Sent Done Done false
23369 7395 Acked Write 5 Sent Done Done false
23370 7396 Acked Read 9 Sent Done Done false
23371 7397 Acked Write 4 Sent Done Done false
23372 7398 Acked Read 6 Sent Done Done false
23373 7399 Acked Flush 0 Sent Done Done false
23374 7400 Acked Write 9 Sent Done Done false
23375 7401 Acked Write 2 Sent Done Done false
23376 7402 Acked Write 9 Sent Done Done false
23377 7403 Acked Read 10 Sent Done Done false
23378 7404 Acked Write 4 Sent Done Done false
23379 7405 Acked Write 6 Sent Done Done false
23380 7406 Acked Read 4 Sent Done Done false
23381 7407 Acked Read 9 Sent Done Done false
23382 7408 Acked Write 6 Sent Done Done false
23383 7409 Acked Read 2 Sent Done Done false
23384 7410 Acked Write 1 Sent Done Done false
23385 7411 Acked Read 9 Sent Done Done false
23386 7412 Acked Write 10 Sent Done Done false
23387 7413 Acked Write 6 Sent Done Done false
23388 7414 Acked Read 3 Sent Done Done false
23389 7415 Acked Read 8 Sent Done Done false
23390 7416 Acked Write 10 Sent Done Done false
23391 7417 Acked Write 7 Sent Done Done false
23392 7418 Acked Write 7 Sent Done Done false
23393 7419 Acked Write 6 Sent Done Done false
23394 7420 Acked Read 5 Sent Done Done false
23395 7421 Acked Read 5 Sent Done Done false
23396 7422 Acked Read 10 Sent Done Done false
23397 7423 Acked Write 3 Sent Done Done false
23398 7424 Acked Write 1 Sent Done Done false
23399 7425 Acked Read 4 Sent Done Done false
23400 7426 Acked Read 2 Sent Done Done false
23401 7427 Acked Write 9 Sent Done Done false
23402 7428 Acked Flush 0 Sent Done Done false
23403 7429 Acked Write 9 Sent Done Done false
23404 7430 Acked Write 2 Sent Done Done false
23405 7431 Acked Write 6 Sent Done Done false
23406 7432 Acked Flush 0 Sent Done Done false
23407 7433 Acked Write 4 Sent Done Done false
23408 7434 Acked Read 9 Sent Done Done false
23409 7435 Acked Read 10 Sent Done Done false
23410 7436 Acked Write 9 Sent Done Done false
23411 7437 Acked Write 7 Sent Done Done false
23412 7438 Acked Read 8 Sent Done Done false
23413 7439 Acked Read 1 Sent Done Done false
23414 7440 Acked Write 5 Sent Done Done false
23415 7441 Acked Read 7 Sent Done Done false
23416 7442 Acked Write 8 Sent Done Done false
23417 7443 Acked Write 9 Sent Done Done false
23418 7444 Acked Write 1 Sent Done Done false
23419 7445 Acked Write 10 Sent Done Done false
23420 7446 Acked Flush 0 Sent Done Done false
23421 7447 Acked Write 4 Sent Done Done false
23422 7448 Acked Flush 0 Sent Done Done false
23423 7449 Acked Read 5 Sent Done Done false
23424 7450 Acked Write 5 Sent Done Done false
23425 7451 Acked Write 2 Sent Done Done false
23426 7452 Acked Flush 0 Sent Done Done false
23427 7453 Acked Flush 0 Sent Done Done false
23428 7454 Acked Read 1 Sent Done Done false
23429 7455 Acked Read 5 Sent Done Done false
23430 7456 Acked Read 1 Sent Done Done false
23431 7457 Acked Write 5 Sent Done Done false
23432 7458 Acked Write 1 Sent Done Done false
23433 7459 Acked Write 1 Sent Done Done false
23434 7460 Acked Write 10 Sent Done Done false
23435 7461 Acked Read 4 Sent Done Done false
23436 7462 Acked Read 6 Sent Done Done false
23437 7463 Acked Flush 0 Sent Done Done false
23438 7464 Acked Write 6 Sent Done Done false
23439 7465 Acked Read 7 Sent Done Done false
23440 7466 Acked Read 5 Sent Done Done false
23441 7467 Acked Flush 0 Sent Done Done false
23442 7468 Acked Read 6 Sent Done Done false
23443 7469 Acked Read 10 Sent Done Done false
23444 7470 Acked Flush 0 Sent Done Done false
23445 7471 Acked Read 1 Sent Done Done false
23446 7472 Acked Read 4 Sent Done Done false
23447 7473 Acked Read 6 Sent Done Done false
23448 7474 Acked Write 5 Sent Done Done false
23449 7475 Acked Read 5 Sent Done Done false
23450 7476 Acked Read 5 Sent Done Done false
23451 7477 Acked Write 5 Sent Done Done false
23452 7478 Acked Write 9 Sent Done Done false
23453 7479 Acked Read 9 Sent Done Done false
23454 7480 Acked Write 8 Sent Done Done false
23455 7481 Acked Write 8 Sent Done Done false
23456 7482 Acked Write 3 Sent Done Done false
23457 7483 Acked Write 1 Sent Done Done false
23458 7484 Acked Flush 0 Sent Done Done false
23459 7485 Acked Flush 0 Sent Done Done false
23460 7486 Acked Read 2 Sent Done Done false
23461 7487 Acked Write 5 Sent Done Done false
23462 7488 Acked Read 3 Sent Done Done false
23463 7489 Acked Read 4 Sent Done Done false
23464 7490 Acked Write 3 Sent Done Done false
23465 7491 Acked Write 1 Sent Done Done false
23466 7492 Acked Read 2 Sent Done Done false
23467 7493 Acked Read 10 Sent Done Done false
23468 7494 Acked Write 6 Sent Done Done false
23469 7495 Acked Write 7 Sent Done Done false
23470 7496 Acked Write 5 Sent Done Done false
23471 7497 Acked Flush 0 Sent Done Done false
23472 7498 Acked Write 4 Sent Done Done false
23473 7499 Acked Read 3 Sent Done Done false
23474 7500 Acked Read 5 Sent Done Done false
23475 7501 Acked Write 1 Sent Done Done false
23476 7502 Acked Write 10 Sent Done Done false
23477 7503 Acked Write 2 Sent Done Done false
23478 7504 Acked Read 1 Sent Done Done false
23479 7505 Acked Read 6 Sent Done Done false
23480 7506 Acked Read 6 Sent Done Done false
23481 7507 Acked Read 1 Sent Done Done false
23482 7508 Acked Read 10 Sent Done Done false
23483 7509 Acked Read 8 Sent Done Done false
23484 7510 Acked Write 5 Sent Done Done false
23485 7511 Acked Write 8 Sent Done Done false
23486 7512 Acked Read 4 Sent Done Done false
23487 7513 Acked Read 8 Sent Done Done false
23488 7514 Acked Read 4 Sent Done Done false
23489 7515 Acked Read 9 Sent Done Done false
23490 7516 Acked Write 1 Sent Done Done false
23491 7517 Acked Write 7 Sent Done Done false
23492 7518 Acked Read 7 Sent Done Done false
23493 7519 Acked Write 1 Sent Done Done false
23494 7520 Acked Write 3 Sent Done Done false
23495 7521 Acked Write 3 Sent Done Done false
23496 7522 Acked Read 7 Sent Done Done false
23497 7523 Acked Read 1 Sent Done Done false
23498 7524 Acked Write 9 Sent Done Done false
23499 7525 Acked Write 9 Sent Done Done false
23500 7526 Acked Read 10 Sent Done Done false
23501 7527 Acked Read 5 Sent Done Done false
23502 7528 Acked Write 7 Sent Done Done false
23503 7529 Acked Write 9 Sent Done Done false
23504 7530 Acked Read 7 Sent Done Done false
23505 7531 Acked Write 4 Sent Done Done false
23506 7532 Acked Flush 0 Sent Done Done false
23507 7533 Acked Read 6 Sent Done Done false
23508 7534 Acked Read 7 Sent Done Done false
23509 7535 Acked Read 5 Sent Done Done false
23510 7536 Acked Write 4 Sent Done Done false
23511 7537 Acked Read 6 Sent Done Done false
23512 7538 Acked Write 1 Sent Done Done false
23513 7539 Acked Read 2 Sent Done Done false
23514 7540 Acked Read 9 Sent Done Done false
23515 7541 Acked Write 6 Sent Done Done false
23516 7542 Acked Write 5 Sent Done Done false
23517 7543 Acked Flush 0 Sent Done Done false
23518 7544 Acked Write 3 Sent Done Done false
23519 7545 Acked Write 6 Sent Done Done false
23520 7546 Acked Read 7 Sent Done Done false
23521 7547 Acked Read 7 Sent Done Done false
23522 7548 Acked Flush 0 Sent Done Done false
23523 7549 Acked Write 9 Sent Done Done false
23524 7550 Acked Read 4 Sent Done Done false
23525 7551 Acked Read 9 Sent Done Done false
23526 7552 Acked Read 1 Sent Done Done false
23527 7553 Acked Flush 0 Sent Done Done false
23528 7554 Acked Write 3 Sent Done Done false
23529 7555 Acked Write 6 Sent Done Done false
23530 7556 Acked Read 4 Sent Done Done false
23531 7557 Acked Read 3 Sent Done Done false
23532 7558 Acked Read 7 Sent Done Done false
23533 7559 Acked Read 5 Sent Done Done false
23534 7560 Acked Write 10 Sent Done Done false
23535 7561 Acked Write 6 Sent Done Done false
23536 7562 Acked Flush 0 Sent Done Done false
23537 7563 Acked Read 2 Sent Done Done false
23538 7564 Acked Write 4 Sent Done Done false
23539 7565 Acked Read 5 Sent Done Done false
23540 7566 Acked Write 8 Sent Done Done false
23541 7567 Acked Flush 0 Sent Done Done false
23542 7568 Acked Flush 0 Sent Done Done false
23543 7569 Acked Read 6 Sent Done Done false
23544 7570 Acked Read 4 Sent Done Done false
23545 7571 Acked Read 8 Sent Done Done false
23546 7572 Acked Read 8 Sent Done Done false
23547 7573 Acked Write 5 Sent Done Done false
23548 7574 Acked Read 9 Sent Done Done false
23549 7575 Acked Write 5 Sent Done Done false
23550 7576 Acked Read 10 Sent Done Done false
23551 7577 Acked Write 8 Sent Done Done false
23552 7578 Acked Read 9 Sent Done Done false
23553 7579 Acked Write 4 Sent Done Done false
23554 7580 Acked Write 2 Sent Done Done false
23555 7581 Acked Read 8 Sent Done Done false
23556 7582 Acked Write 5 Sent Done Done false
23557 7583 Acked Read 2 Sent Done Done false
23558 7584 Acked Write 9 Sent Done Done false
23559 7585 Acked Write 3 Sent Done Done false
23560 7586 Acked Read 10 Sent Done Done false
23561 7587 Acked Read 7 Sent Done Done false
23562 7588 Acked Write 2 Sent Done Done false
23563 7589 Acked Write 9 Sent Done Done false
23564 7590 Acked Write 7 Sent Done Done false
23565 7591 Acked Flush 0 Sent Done Done false
23566 7592 Acked Read 5 Sent Done Done false
23567 7593 Acked Read 3 Sent Done Done false
23568 7594 Acked Read 6 Sent Done Done false
23569 7595 Acked Read 7 Sent Done Done false
23570 7596 Acked Write 6 Sent Done Done false
23571 7597 Acked Read 4 Sent Done Done false
23572 7598 Acked Read 5 Sent Done Done false
23573 7599 Acked Read 9 Sent Done Done false
23574 7600 Acked Write 6 Sent Done Done false
23575 7601 Acked Flush 0 Sent Done Done false
23576 7602 Acked Read 4 Sent Done Done false
23577 7603 Acked Read 2 Sent Done Done false
23578 7604 Acked Read 4 Sent Done Done false
23579 7605 Acked Read 5 Sent Done Done false
23580 7606 Acked Read 5 Sent Done Done false
23581 7607 Acked Read 6 Sent Done Done false
23582 7608 Acked Read 1 Sent Done Done false
23583 7609 Acked Read 10 Sent Done Done false
23584 7610 Acked Write 7 Sent Done Done false
23585 7611 Acked Write 6 Sent Done Done false
23586 7612 Acked Write 6 Sent Done Done false
23587 7613 Acked Write 9 Sent Done Done false
23588 7614 Acked Write 8 Sent Done Done false
23589 7615 Acked Write 4 Sent Done Done false
23590 7616 Acked Read 3 Sent Done Done false
23591 7617 Acked Read 1 Sent Done Done false
23592 7618 Acked Write 1 Sent Done Done false
23593 7619 Acked Read 9 Sent Done Done false
23594 7620 Acked Write 6 Sent Done Done false
23595 7621 Acked Read 6 Sent Done Done false
23596 7622 Acked Write 2 Sent Done Done false
23597 7623 Acked Read 9 Sent Done Done false
23598 7624 Acked Read 3 Sent Done Done false
23599 7625 Acked Read 1 Sent Done Done false
23600 7626 Acked Write 8 Sent Done Done false
23601 7627 Acked Write 3 Sent Done Done false
23602 7628 Acked Read 6 Sent Done Done false
23603 7629 Acked Write 4 Sent Done Done false
23604 7630 Acked Write 8 Sent Done Done false
23605 7631 Acked Write 9 Sent Done Done false
23606 7632 Acked Write 5 Sent Done Done false
23607 7633 Acked Flush 0 Sent Done Done false
23608 7634 Acked Write 1 Sent Done Done false
23609 7635 Acked Read 6 Sent Done Done false
23610 7636 Acked Write 7 Sent Done Done false
23611 7637 Acked Read 5 Sent Done Done false
23612 7638 Acked Read 9 Sent Done Done false
23613 7639 Acked Read 6 Sent Done Done false
23614 7640 Acked Read 9 Sent Done Done false
23615 STATES DS:0 DS:1 DS:2 TOTAL
23616 Sent 301 0 0 301
23617 Done 0 301 301 602
23618 Skipped 0 0 0 0
23619 Error 0 0 0 0
23620 Last Flush: Some(JobId(7339)) Some(JobId(7633)) Some(JobId(7633))
23621 Downstairs last five completed: 7339 7338 7337 7336 7335
23622 Upstairs last five completed: 7640 7639 7638 7637 7636
23623 CLIENT: Up:0 ds:301 act:2
236242025-01-28T21:59:23.827ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
236252025-01-28T21:59:24.788ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
236262025-01-28T21:59:25.084ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
236272025-01-28T21:59:25.084ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42617) = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
236282025-01-28T21:59:25.085ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
236292025-01-28T21:59:25.086ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
236302025-01-28T21:59:25.086ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
236312025-01-28T21:59:25.086ZINFOcrucible: send last flush ID to this DS: Some(JobId(7339)) = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
236322025-01-28T21:59:25.086ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
236332025-01-28T21:59:25.086ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(7339)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
236342025-01-28T21:59:25.086ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(7339)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
23635 ----------------------------------------------------------------
23636 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23637 Downstairs last five completed: 7641 7640 7639 7638 7637
23638 Upstairs last five completed: 7641 7640 7639 7638 7637
23639 CLIENT: Up:0 ds:0 act:3
23640 Replay: All jobs finished
23641 Main task has work to do, go find it
23642 got dsc Stop(0)
23643 stop 0
23644 [8810] Got stop action so:false kr:false
23645 Jan 28 21:59:27.328 INFO request completed, latency_us: 383, response_code: 204, uri: /stop/cid/0, method: POST, req_id: fb0debac-c4a3-4766-afaf-427d43967903, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
236462025-01-28T21:59:27.329ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
236472025-01-28T21:59:27.329ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
236482025-01-28T21:59:27.329ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
23649 Jan 28 21:59:27.329 INFO request completed, latency_us: 135, response_code: 200, uri: /state/cid/0, method: GET, req_id: 11e32023-2ce9-4c97-8efc-ca129237a4d1, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-28T21:59:27.329216055Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
23650 :9998
23651 [8810] Exited with: ExitStatus(unix_wait_status(9))
23652 [8810][0] reports Exit
23653 Jan 28 21:59:31.331 INFO request completed, latency_us: 205, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9cb8ccef-1005-4ae6-85dc-6cd565517fa0, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
23654 001/300 Read block 2635 len 4096
23655 002/300 Read block 3547 len 36864
23656 003/300 Read block 1434 len 24576
23657 004/300 Flush
23658 005/300 Write block 254 len 36864 data: 4 3 3 3 3 3 3 3 6
23659 006/300 Read block 623 len 28672
23660 007/300 Read block 860 len 16384
23661 008/300 Write block 1423 len 24576 data: 3 3 5 5 4 4
23662 009/300 Flush
23663 010/300 Write block 3808 len 12288 data: 3 3 3
23664 011/300 Write block 1532 len 20480 data: 5 6 5 5 4
23665 012/300 Read block 4566 len 16384
23666 013/300 Read block 1549 len 4096
23667 014/300 Flush
23668 015/300 Read block 2967 len 8192
23669 016/300 Read block 394 len 12288
23670 017/300 Read block 409 len 28672
23671 018/300 Write block 1052 len 8192 data: 9 9
23672 019/300 Write block 2759 len 16384 data: 7 7 5 5
23673 020/300 Write block 602 len 8192 data: 5 6
23674 021/300 Read block 1547 len 4096
23675 022/300 Write block 2403 len 28672 data: 5 5 4 3 4 5 6
23676 023/300 Write block 3740 len 28672 data: 4 3 4 4 3 3 3
23677 024/300 Flush
23678 025/300 Write block 4108 len 32768 data: 5 4 3 4 4 5 5 8
23679 026/300 Write block 1560 len 16384 data: 7 6 8 6
23680 027/300 Read block 139 len 28672
23681 028/300 Write block 2091 len 24576 data: 6 4 5 6 6 6
23682 029/300 Read block 3869 len 4096
23683 030/300 Read block 1492 len 32768
23684 031/300 Write block 4259 len 20480 data: 3 4 3 3 4
23685 032/300 Read block 977 len 32768
23686 033/300 Write block 2890 len 28672 data: 2 2 2 3 5 4 4
23687 034/300 Write block 2687 len 20480 data: 4 7 6 7 8
23688 035/300 Read block 2376 len 20480
23689 036/300 Write block 2190 len 28672 data: 6 5 5 5 4 3 3
23690 037/300 Read block 697 len 36864
23691 038/300 Write block 1672 len 24576 data: 5 5 4 4 5 5
23692 039/300 Read block 4786 len 20480
23693 040/300 Write block 4319 len 40960 data: 6 5 5 5 7 7 6 6 6 7
23694 041/300 Write block 3858 len 4096 data: 3
23695 042/300 Write block 4345 len 40960 data: 6 5 5 5 4 4 6 7 7 6
23696 043/300 Write block 4195 len 24576 data: 6 6 7 6 6 6
23697 044/300 Write block 2929 len 8192 data: 6 5
23698 045/300 Read block 85 len 40960
23699 046/300 Read block 974 len 20480
23700 047/300 Read block 4834 len 36864
23701 048/300 Write block 3181 len 36864 data: 5 5 5 4 4 3 4 4 4
23702 049/300 Read block 2846 len 32768
23703 050/300 Read block 832 len 4096
23704 051/300 Read block 2863 len 12288
23705 052/300 Read block 1474 len 12288
23706 053/300 Write block 130 len 4096 data: 3
23707 054/300 Write block 4781 len 20480 data: 4 3 2 2 2
23708 055/300 Write block 1403 len 12288 data: 4 5 5
23709 056/300 Read block 4495 len 16384
23710 057/300 Read block 315 len 4096
23711 058/300 Read block 1533 len 4096
23712 059/300 Read block 2257 len 4096
23713 060/300 Flush
23714 061/300 Flush
23715 062/300 Write block 159 len 32768 data: 6 6 6 4 3 3 4 3
23716 063/300 Read block 3202 len 28672
23717 064/300 Write block 1005 len 8192 data: 5 5
23718 065/300 Read block 1633 len 20480
23719 066/300 Read block 1392 len 32768
23720 067/300 Write block 2482 len 24576 data: 6 6 4 4 2 2
23721 068/300 Read block 2191 len 28672
23722 069/300 Flush
23723 070/300 Read block 2838 len 28672
23724 071/300 Read block 2712 len 16384
23725 072/300 Write block 3328 len 36864 data: 7 6 8 6 5 5 5 5 5
23726 073/300 Write block 4631 len 20480 data: 5 5 5 5 5
23727 074/300 Read block 1555 len 16384
23728 075/300 Read block 590 len 20480
23729 076/300 Read block 3865 len 24576
23730 077/300 Write block 4377 len 20480 data: 4 4 4 5 7
23731 078/300 Write block 4864 len 20480 data: 5 5 6 5 4
23732 079/300 Flush
23733 080/300 Read block 1542 len 20480
23734 081/300 Read block 4567 len 40960
23735 082/300 Write block 4724 len 4096 data: 3
23736 083/300 Flush
23737 084/300 Write block 4203 len 12288 data: 5 5 6
23738 085/300 Read block 654 len 20480
23739 086/300 Flush
23740 087/300 Read block 1915 len 12288
23741 088/300 Write block 2752 len 32768 data: 7 6 6 7 7 7 7 8
23742 089/300 Flush
23743 090/300 Read block 2863 len 8192
23744 091/300 Read block 4897 len 12288
23745 092/300 Write block 1827 len 24576 data: 7 6 6 5 4 4
23746 093/300 Write block 2874 len 24576 data: 5 5 7 5 5 4
23747 094/300 Write block 3595 len 28672 data: 6 6 8 7 8 5 5
23748 095/300 Write block 2060 len 12288 data: 3 3 3
23749 096/300 Write block 4538 len 36864 data: 2 3 5 5 6 5 7 8 6
23750 097/300 Write block 3452 len 20480 data: 5 3 3 4 4
23751 098/300 Write block 171 len 16384 data: 4 5 5 5
23752 099/300 Read block 4105 len 36864
23753 100/300 Write block 636 len 36864 data: 6 8 3 3 4 3 3 2 3
23754 101/300 Write block 1012 len 16384 data: 7 8 10 9
23755 102/300 Write block 51 len 20480 data: 3 4 3 3 4
23756 103/300 Write block 2197 len 24576 data: 4 3 4 5 7 7
23757 104/300 Read block 3133 len 36864
23758 105/300 Write block 921 len 8192 data: 3 3
23759 106/300 Read block 2666 len 32768
23760 107/300 Read block 753 len 12288
23761 108/300 Write block 3855 len 8192 data: 2 2
23762 109/300 Read block 1908 len 4096
23763 110/300 Read block 2365 len 8192
23764 111/300 Write block 1404 len 16384 data: 6 6 3 4
23765 112/300 Flush
23766 113/300 Read block 1363 len 40960
23767 114/300 Write block 3946 len 24576 data: 5 5 6 7 6 5
23768 115/300 Read block 2039 len 24576
23769 116/300 Read block 951 len 24576
23770 117/300 Flush
23771 118/300 Read block 2828 len 16384
23772 119/300 Read block 3332 len 36864
23773 120/300 Write block 4740 len 36864 data: 6 7 6 5 3 3 4 4 5
23774 121/300 Write block 1638 len 32768 data: 5 5 5 5 5 4 4 3
23775 122/300 Write block 2639 len 4096 data: 4
23776 123/300 Read block 1881 len 16384
23777 124/300 Flush
23778 125/300 Read block 1871 len 28672
23779 126/300 Write block 2263 len 24576 data: 4 4 3 4 3 2
23780 127/300 Read block 3071 len 36864
23781 128/300 Read block 1184 len 12288
23782 129/300 Write block 1043 len 16384 data: 8 8 7 9
23783 130/300 Read block 4351 len 24576
23784 131/300 Read block 2823 len 36864
23785 132/300 Read block 2232 len 40960
23786 133/300 Read block 3552 len 8192
23787 134/300 Flush
23788 135/300 Write block 1178 len 12288 data: 8 11 12
23789 136/300 Write block 382 len 32768 data: 4 4 4 4 4 3 4 3
23790 137/300 Read block 2014 len 36864
23791 138/300 Read block 1992 len 12288
23792 139/300 Read block 193 len 32768
23793 140/300 Read block 1341 len 12288
23794 141/300 Flush
23795 142/300 Read block 2350 len 16384
23796 143/300 Read block 1608 len 12288
23797 144/300 Write block 2841 len 16384 data: 5 5 5 6
23798 145/300 Write block 4674 len 32768 data: 4 3 4 4 6 8 8 8
23799 146/300 Write block 4177 len 28672 data: 5 5 5 4 4 3 4
23800 147/300 Flush
23801 148/300 Write block 1173 len 32768 data: 5 5 4 5 6 9 12 13
23802 149/300 Flush
23803 150/300 Write block 2672 len 12288 data: 2 3 4
23804 151/300 Read block 3515 len 16384
23805 152/300 Read block 2275 len 4096
23806 153/300 Read block 3321 len 36864
23807 154/300 Read block 3274 len 28672
23808 155/300 Write block 1458 len 36864 data: 6 7 5 3 4 3 3 4 4
23809 156/300 Read block 94 len 12288
23810 157/300 Read block 927 len 4096
23811 158/300 Write block 3587 len 36864 data: 3 4 5 5 6 5 5 6 7
23812 159/300 Write block 93 len 12288 data: 4 5 4
23813 160/300 Write block 1137 len 8192 data: 3 4
23814 161/300 Write block 1896 len 4096 data: 3
23815 162/300 Read block 3662 len 28672
23816 163/300 Read block 2472 len 28672
23817 164/300 Write block 444 len 28672 data: 4 4 3 3 3 3 2
23818 165/300 Read block 2462 len 4096
23819 166/300 Read block 1438 len 24576
23820 167/300 Write block 4271 len 20480 data: 6 4 3 4 5
23821 168/300 Read block 4089 len 8192
23822 169/300 Read block 3521 len 20480
23823 170/300 Write block 4008 len 16384 data: 6 6 6 5
23824 171/300 Write block 1141 len 16384 data: 7 8 8 9
23825 172/300 Read block 2438 len 32768
23826 173/300 Write block 3069 len 20480 data: 5 4 3 3 3
23827 174/300 Read block 254 len 8192
23828 175/300 Read block 3421 len 24576
23829 176/300 Read block 1422 len 12288
23830 177/300 Write block 597 len 32768 data: 3 4 4 5 5 6 7 7
23831 178/300 Flush
23832 179/300 Write block 2755 len 32768 data: 8 8 8 8 9 8 6 6
23833 180/300 Write block 2495 len 8192 data: 4 5
23834 181/300 Read block 115 len 16384
23835 182/300 Read block 2205 len 4096
23836 183/300 Read block 603 len 40960
23837 184/300 Read block 609 len 24576
23838 185/300 Flush
23839 186/300 Write block 2937 len 4096 data: 5
23840 187/300 Write block 1288 len 40960 data: 4 5 6 7 6 6 6 7 7 8
23841 188/300 Read block 4686 len 28672
23842 189/300 Read block 3776 len 36864
23843 190/300 Read block 3191 len 4096
23844 191/300 Write block 4931 len 40960 data: 6 7 8 8 8 5 3 4 4 2
23845 192/300 Flush
23846 193/300 Write block 2631 len 24576 data: 5 5 7 6 7 7
23847 194/300 Write block 3673 len 4096 data: 6
23848 195/300 Write block 1741 len 32768 data: 4 4 4 3 4 5 6 5
23849 196/300 Flush
23850 197/300 Read block 745 len 32768
23851 198/300 Write block 2082 len 8192 data: 8 7
23852 199/300 Write block 4813 len 12288 data: 6 5 4
23853 200/300 Write block 403 len 16384 data: 5 6 4 3
23854 201/300 Write block 1600 len 8192 data: 2 2
23855 202/300 Read block 3234 len 28672
23856 203/300 Read block 2233 len 12288
23857 204/300 Write block 3706 len 4096 data: 4
23858 205/300 Write block 2537 len 24576 data: 2 3 3 3 2 3
23859 206/300 Read block 655 len 8192
23860 207/300 Write block 1334 len 16384 data: 7 7 7 6
23861 208/300 Read block 4868 len 20480
23862 209/300 Write block 3141 len 4096 data: 4
23863 210/300 Read block 1868 len 32768
23864 211/300 Write block 2416 len 20480 data: 6 5 3 4 4
23865 212/300 Write block 3106 len 40960 data: 8 4 4 3 5 4 5 5 6 6
23866 213/300 Read block 4194 len 8192
23867 214/300 Read block 3385 len 20480
23868 215/300 Flush
23869 216/300 Read block 1801 len 36864
23870 217/300 Write block 1044 len 12288 data: 9 8 10
23871 218/300 Write block 73 len 24576 data: 6 7 6 6 6 7
23872 219/300 Read block 231 len 28672
23873 220/300 Read block 3140 len 36864
23874 221/300 Read block 1361 len 24576
23875 222/300 Read block 3637 len 24576
23876 223/300 Read block 2485 len 8192
23877 224/300 Read block 1181 len 16384
23878 225/300 Write block 2560 len 28672 data: 4 5 5 5 7 8 6
23879 226/300 Read block 440 len 8192
23880 227/300 Flush
23881 228/300 Write block 3949 len 24576 data: 8 7 6 3 3 4
23882 229/300 Write block 417 len 20480 data: 4 3 3 5 7
23883 230/300 Write block 1764 len 24576 data: 4 4 4 4 5 6
23884 231/300 Read block 4773 len 8192
23885 232/300 Read block 3784 len 12288
23886 233/300 Write block 1599 len 4096 data: 2
23887 234/300 Read block 611 len 12288
23888 235/300 Write block 4369 len 8192 data: 4 4
23889 236/300 Write block 2350 len 12288 data: 7 6 6
23890 237/300 Read block 2970 len 12288
23891 238/300 Write block 1396 len 16384 data: 2 2 2 4
23892 239/300 Flush
23893 240/300 Write block 996 len 24576 data: 3 4 5 5 4 5
23894 241/300 Read block 4869 len 28672
23895 242/300 Write block 550 len 36864 data: 7 6 6 6 3 3 3 2 2
23896 243/300 Read block 2554 len 36864
23897 244/300 Write block 2622 len 4096 data: 4
23898 245/300 Flush
23899 246/300 Write block 4177 len 28672 data: 6 6 6 5 5 4 5
23900 247/300 Read block 4369 len 12288
23901 248/300 Read block 2292 len 12288
23902 249/300 Write block 3118 len 36864 data: 5 5 5 7 6 5 3 2 2
23903 250/300 Write block 4268 len 8192 data: 6 5
23904 251/300 Read block 1722 len 24576
23905 252/300 Write block 148 len 12288 data: 2 3 5
23906 253/300 Read block 4859 len 24576
23907 254/300 Read block 1975 len 8192
23908 255/300 Read block 3187 len 16384
23909 256/300 Write block 945 len 12288 data: 7 3 4
23910 257/300 Write block 1763 len 40960 data: 4 5 5 5 5 6 7 6 5 6
23911 258/300 Read block 2714 len 12288
23912 259/300 Read block 3421 len 28672
23913 260/300 Write block 2542 len 32768 data: 4 3 3 3 4 4 5 6
23914 261/300 Read block 2268 len 4096
23915 262/300 Write block 3029 len 24576 data: 4 4 5 5 3 3
23916 263/300 Flush
23917 264/300 Write block 4006 len 40960 data: 7 6 7 7 7 6 5 4 2 3
23918 265/300 Write block 3415 len 40960 data: 4 3 3 5 4 4 5 5 5 4
23919 266/300 Write block 2874 len 20480 data: 6 6 8 6 6
23920 267/300 Flush
23921 268/300 Read block 4608 len 16384
23922 269/300 Read block 4721 len 40960
23923 270/300 Write block 4798 len 12288 data: 5 6 5
23924 271/300 Read block 4488 len 24576
23925 272/300 Read block 4562 len 4096
23926 273/300 Read block 2574 len 20480
23927 274/300 Read block 851 len 12288
23928 275/300 Read block 1514 len 36864
23929 276/300 Read block 341 len 32768
23930 277/300 Read block 3669 len 8192
23931 278/300 Read block 2331 len 20480
23932 279/300 Write block 4640 len 28672 data: 4 3 3 3 3 4 4
23933 280/300 Write block 1879 len 12288 data: 4 4 4
23934 281/300 Read block 1957 len 20480
23935 282/300 Read block 1387 len 20480
23936 283/300 Write block 1884 len 28672 data: 5 4 3 3 2 2 2
23937 284/300 Write block 2293 len 32768 data: 6 4 4 4 4 3 3 2
23938 285/300 Read block 1523 len 4096
23939 286/300 Write block 4297 len 12288 data: 4 5 5
23940 287/300 Flush
23941 288/300 Write block 1256 len 20480 data: 5 4 3 3 3
23942 289/300 Read block 657 len 40960
23943 290/300 Write block 3613 len 24576 data: 4 4 4 3 4 4
23944 291/300 Write block 3255 len 4096 data: 8
23945 292/300 Write block 4183 len 12288 data: 6 4 4
23946 293/300 Write block 1722 len 12288 data: 5 4 3
23947 294/300 Write block 2239 len 16384 data: 5 4 4 4
23948 295/300 Write block 779 len 24576 data: 7 6 5 6 6 6
23949 296/300 Read block 1233 len 12288
23950 297/300 Write block 3877 len 32768 data: 5 6 6 6 5 6 6 8
23951 298/300 Write block 2372 len 24576 data: 6 6 8 5 6 5
23952 299/300 Read block 4916 len 4096
23953 300/300 Write block 1219 len 32768 data: 4 4 5 4 4 3 3 3
23954 301/300 Read block 4085 len 20480
23955 Main task has work to do, go find it
23956 got dsc Start(0)
23957 start 0
23958 Jan 28 21:59:31.558 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/0, method: POST, req_id: f1387409-77f8-47b5-8d82-01fa2c032da6, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
23959 Replay: started 0, returned:Ok(())
23960 ----------------------------------------------------------------
23961 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23962 States: Offline Active Active
23963 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23964 7642 Acked Read 1 Sent Done Done false
23965 7643 Acked Read 9 Sent Done Done false
23966 7644 Acked Read 6 Sent Done Done false
23967 7645 Acked Flush 0 Sent Done Done false
23968 7646 Acked Write 9 Sent Done Done false
23969 7647 Acked Read 7 Sent Done Done false
23970 7648 Acked Read 4 Sent Done Done false
23971 7649 Acked Write 6 Sent Done Done false
23972 7650 Acked Flush 0 Sent Done Done false
23973 7651 Acked Write 3 Sent Done Done false
23974 7652 Acked Write 5 Sent Done Done false
23975 7653 Acked Read 4 Sent Done Done false
23976 7654 Acked Read 1 Sent Done Done false
23977 7655 Acked Flush 0 Sent Done Done false
23978 7656 Acked Read 2 Sent Done Done false
23979 7657 Acked Read 3 Sent Done Done false
23980 7658 Acked Read 7 Sent Done Done false
23981 7659 Acked Write 2 Sent Done Done false
23982 7660 Acked Write 4 Sent Done Done false
23983 7661 Acked Write 2 Sent Done Done false
23984 7662 Acked Read 1 Sent Done Done false
23985 7663 Acked Write 7 Sent Done Done false
23986 7664 Acked Write 7 Sent Done Done false
23987 7665 Acked Flush 0 Sent Done Done false
23988 7666 Acked Write 8 Sent Done Done false
23989 7667 Acked Write 4 Sent Done Done false
23990 7668 Acked Read 7 Sent Done Done false
23991 7669 Acked Write 6 Sent Done Done false
23992 7670 Acked Read 1 Sent Done Done false
23993 7671 Acked Read 8 Sent Done Done false
23994 7672 Acked Write 5 Sent Done Done false
23995 7673 Acked Read 8 Sent Done Done false
23996 7674 Acked Write 7 Sent Done Done false
23997 7675 Acked Write 5 Sent Done Done false
23998 7676 Acked Read 5 Sent Done Done false
23999 7677 Acked Write 7 Sent Done Done false
24000 7678 Acked Read 9 Sent Done Done false
24001 7679 Acked Write 6 Sent Done Done false
24002 7680 Acked Read 5 Sent Done Done false
24003 7681 Acked Write 10 Sent Done Done false
24004 7682 Acked Write 1 Sent Done Done false
24005 7683 Acked Write 10 Sent Done Done false
24006 7684 Acked Write 6 Sent Done Done false
24007 7685 Acked Write 2 Sent Done Done false
24008 7686 Acked Read 10 Sent Done Done false
24009 7687 Acked Read 5 Sent Done Done false
24010 7688 Acked Read 9 Sent Done Done false
24011 7689 Acked Write 9 Sent Done Done false
24012 7690 Acked Read 8 Sent Done Done false
24013 7691 Acked Read 1 Sent Done Done false
24014 7692 Acked Read 3 Sent Done Done false
24015 7693 Acked Read 3 Sent Done Done false
24016 7694 Acked Write 1 Sent Done Done false
24017 7695 Acked Write 5 Sent Done Done false
24018 7696 Acked Write 3 Sent Done Done false
24019 7697 Acked Read 4 Sent Done Done false
24020 7698 Acked Read 1 Sent Done Done false
24021 7699 Acked Read 1 Sent Done Done false
24022 7700 Acked Read 1 Sent Done Done false
24023 7701 Acked Flush 0 Sent Done Done false
24024 7702 Acked Flush 0 Sent Done Done false
24025 7703 Acked Write 8 Sent Done Done false
24026 7704 Acked Read 7 Sent Done Done false
24027 7705 Acked Write 2 Sent Done Done false
24028 7706 Acked Read 5 Sent Done Done false
24029 7707 Acked Read 8 Sent Done Done false
24030 7708 Acked Write 6 Sent Done Done false
24031 7709 Acked Read 7 Sent Done Done false
24032 7710 Acked Flush 0 Sent Done Done false
24033 7711 Acked Read 7 Sent Done Done false
24034 7712 Acked Read 4 Sent Done Done false
24035 7713 Acked Write 9 Sent Done Done false
24036 7714 Acked Write 5 Sent Done Done false
24037 7715 Acked Read 4 Sent Done Done false
24038 7716 Acked Read 5 Sent Done Done false
24039 7717 Acked Read 6 Sent Done Done false
24040 7718 Acked Write 5 Sent Done Done false
24041 7719 Acked Write 5 Sent Done Done false
24042 7720 Acked Flush 0 Sent Done Done false
24043 7721 Acked Read 5 Sent Done Done false
24044 7722 Acked Read 10 Sent Done Done false
24045 7723 Acked Write 1 Sent Done Done false
24046 7724 Acked Flush 0 Sent Done Done false
24047 7725 Acked Write 3 Sent Done Done false
24048 7726 Acked Read 5 Sent Done Done false
24049 7727 Acked Flush 0 Sent Done Done false
24050 7728 Acked Read 3 Sent Done Done false
24051 7729 Acked Write 8 Sent Done Done false
24052 7730 Acked Flush 0 Sent Done Done false
24053 7731 Acked Read 2 Sent Done Done false
24054 7732 Acked Read 3 Sent Done Done false
24055 7733 Acked Write 6 Sent Done Done false
24056 7734 Acked Write 6 Sent Done Done false
24057 7735 Acked Write 7 Sent Done Done false
24058 7736 Acked Write 3 Sent Done Done false
24059 7737 Acked Write 9 Sent Done Done false
24060 7738 Acked Write 5 Sent Done Done false
24061 7739 Acked Write 4 Sent Done Done false
24062 7740 Acked Read 9 Sent Done Done false
24063 7741 Acked Write 9 Sent Done Done false
24064 7742 Acked Write 4 Sent Done Done false
24065 7743 Acked Write 5 Sent Done Done false
24066 7744 Acked Write 6 Sent Done Done false
24067 7745 Acked Read 9 Sent Done Done false
24068 7746 Acked Write 2 Sent Done Done false
24069 7747 Acked Read 8 Sent Done Done false
24070 7748 Acked Read 3 Sent Done Done false
24071 7749 Acked Write 2 Sent Done Done false
24072 7750 Acked Read 1 Sent Done Done false
24073 7751 Acked Read 2 Sent Done Done false
24074 7752 Acked Write 4 Sent Done Done false
24075 7753 Acked Flush 0 Sent Done Done false
24076 7754 Acked Read 10 Sent Done Done false
24077 7755 Acked Write 6 Sent Done Done false
24078 7756 Acked Read 6 Sent Done Done false
24079 7757 Acked Read 6 Sent Done Done false
24080 7758 Acked Flush 0 Sent Done Done false
24081 7759 Acked Read 4 Sent Done Done false
24082 7760 Acked Read 9 Sent Done Done false
24083 7761 Acked Write 9 Sent Done Done false
24084 7762 Acked Write 8 Sent Done Done false
24085 7763 Acked Write 1 Sent Done Done false
24086 7764 Acked Read 4 Sent Done Done false
24087 7765 Acked Flush 0 Sent Done Done false
24088 7766 Acked Read 7 Sent Done Done false
24089 7767 Acked Write 6 Sent Done Done false
24090 7768 Acked Read 9 Sent Done Done false
24091 7769 Acked Read 3 Sent Done Done false
24092 7770 Acked Write 4 Sent Done Done false
24093 7771 Acked Read 6 Sent Done Done false
24094 7772 Acked Read 9 Sent Done Done false
24095 7773 Acked Read 10 Sent Done Done false
24096 7774 Acked Read 2 Sent Done Done false
24097 7775 Acked Flush 0 Sent Done Done false
24098 7776 Acked Write 3 Sent Done Done false
24099 7777 Acked Write 8 Sent Done Done false
24100 7778 Acked Read 9 Sent Done Done false
24101 7779 Acked Read 3 Sent Done Done false
24102 7780 Acked Read 8 Sent Done Done false
24103 7781 Acked Read 3 Sent Done Done false
24104 7782 Acked Flush 0 Sent Done Done false
24105 7783 Acked Read 4 Sent Done Done false
24106 7784 Acked Read 3 Sent Done Done false
24107 7785 Acked Write 4 Sent Done Done false
24108 7786 Acked Write 8 Sent Done Done false
24109 7787 Acked Write 7 Sent Done Done false
24110 7788 Acked Flush 0 Sent Done Done false
24111 7789 Acked Write 8 Sent Done Done false
24112 7790 Acked Flush 0 Sent Done Done false
24113 7791 Acked Write 3 Sent Done Done false
24114 7792 Acked Read 4 Sent Done Done false
24115 7793 Acked Read 1 Sent Done Done false
24116 7794 Acked Read 9 Sent Done Done false
24117 7795 Acked Read 7 Sent Done Done false
24118 7796 Acked Write 9 Sent Done Done false
24119 7797 Acked Read 3 Sent Done Done false
24120 7798 Acked Read 1 Sent Done Done false
24121 7799 Acked Write 9 Sent Done Done false
24122 7800 Acked Write 3 Sent Done Done false
24123 7801 Acked Write 2 Sent Done Done false
24124 7802 Acked Write 1 Sent Done Done false
24125 7803 Acked Read 7 Sent Done Done false
24126 7804 Acked Read 7 Sent Done Done false
24127 7805 Acked Write 7 Sent Done Done false
24128 7806 Acked Read 1 Sent Done Done false
24129 7807 Acked Read 6 Sent Done Done false
24130 7808 Acked Write 5 Sent Done Done false
24131 7809 Acked Read 2 Sent Done Done false
24132 7810 Acked Read 5 Sent Done Done false
24133 7811 Acked Write 4 Sent Done Done false
24134 7812 Acked Write 4 Sent Done Done false
24135 7813 Acked Read 8 Sent Done Done false
24136 7814 Acked Write 5 Sent Done Done false
24137 7815 Acked Read 2 Sent Done Done false
24138 7816 Acked Read 6 Sent Done Done false
24139 7817 Acked Read 3 Sent Done Done false
24140 7818 Acked Write 8 Sent Done Done false
24141 7819 Acked Flush 0 Sent Done Done false
24142 7820 Acked Write 8 Sent Done Done false
24143 7821 Acked Write 2 Sent Done Done false
24144 7822 Acked Read 4 Sent Done Done false
24145 7823 Acked Read 1 Sent Done Done false
24146 7824 Acked Read 10 Sent Done Done false
24147 7825 Acked Read 6 Sent Done Done false
24148 7826 Acked Flush 0 Sent Done Done false
24149 7827 Acked Write 1 Sent Done Done false
24150 7828 Acked Write 10 Sent Done Done false
24151 7829 Acked Read 7 Sent Done Done false
24152 7830 Acked Read 9 Sent Done Done false
24153 7831 Acked Read 1 Sent Done Done false
24154 7832 Acked Write 10 Sent Done Done false
24155 7833 Acked Flush 0 Sent Done Done false
24156 7834 Acked Write 6 Sent Done Done false
24157 7835 Acked Write 1 Sent Done Done false
24158 7836 Acked Write 8 Sent Done Done false
24159 7837 Acked Flush 0 Sent Done Done false
24160 7838 Acked Read 8 Sent Done Done false
24161 7839 Acked Write 2 Sent Done Done false
24162 7840 Acked Write 3 Sent Done Done false
24163 7841 Acked Write 4 Sent Done Done false
24164 7842 Acked Write 2 Sent Done Done false
24165 7843 Acked Read 7 Sent Done Done false
24166 7844 Acked Read 3 Sent Done Done false
24167 7845 Acked Write 1 Sent Done Done false
24168 7846 Acked Write 6 Sent Done Done false
24169 7847 Acked Read 2 Sent Done Done false
24170 7848 Acked Write 4 Sent Done Done false
24171 7849 Acked Read 5 Sent Done Done false
24172 7850 Acked Write 1 Sent Done Done false
24173 7851 Acked Read 8 Sent Done Done false
24174 7852 Acked Write 5 Sent Done Done false
24175 7853 Acked Write 10 Sent Done Done false
24176 7854 Acked Read 2 Sent Done Done false
24177 7855 Acked Read 5 Sent Done Done false
24178 7856 Acked Flush 0 Sent Done Done false
24179 7857 Acked Read 9 Sent Done Done false
24180 7858 Acked Write 3 Sent Done Done false
24181 7859 Acked Write 6 Sent Done Done false
24182 7860 Acked Read 7 Sent Done Done false
24183 7861 Acked Read 9 Sent Done Done false
24184 7862 Acked Read 6 Sent Done Done false
24185 7863 Acked Read 6 Sent Done Done false
24186 7864 Acked Read 2 Sent Done Done false
24187 7865 Acked Read 4 Sent Done Done false
24188 7866 Acked Write 7 Sent Done Done false
24189 7867 Acked Read 2 Sent Done Done false
24190 7868 Acked Flush 0 Sent Done Done false
24191 7869 Acked Write 6 Sent Done Done false
24192 7870 Acked Write 5 Sent Done Done false
24193 7871 Acked Write 6 Sent Done Done false
24194 7872 Acked Read 2 Sent Done Done false
24195 7873 Acked Read 3 Sent Done Done false
24196 7874 Acked Write 1 Sent Done Done false
24197 7875 Acked Read 3 Sent Done Done false
24198 7876 Acked Write 2 Sent Done Done false
24199 7877 Acked Write 3 Sent Done Done false
24200 7878 Acked Read 3 Sent Done Done false
24201 7879 Acked Write 4 Sent Done Done false
24202 7880 Acked Flush 0 Sent Done Done false
24203 7881 Acked Write 6 Sent Done Done false
24204 7882 Acked Read 7 Sent Done Done false
24205 7883 Acked Write 9 Sent Done Done false
24206 7884 Acked Read 9 Sent Done Done false
24207 7885 Acked Write 1 Sent Done Done false
24208 7886 Acked Flush 0 Sent Done Done false
24209 7887 Acked Write 7 Sent Done Done false
24210 7888 Acked Read 3 Sent Done Done false
24211 7889 Acked Read 3 Sent Done Done false
24212 7890 Acked Write 9 Sent Done Done false
24213 7891 Acked Write 2 Sent Done Done false
24214 7892 Acked Read 6 Sent Done Done false
24215 7893 Acked Write 3 Sent Done Done false
24216 7894 Acked Read 6 Sent Done Done false
24217 7895 Acked Read 2 Sent Done Done false
24218 7896 Acked Read 4 Sent Done Done false
24219 7897 Acked Write 3 Sent Done Done false
24220 7898 Acked Write 10 Sent Done Done false
24221 7899 Acked Read 3 Sent Done Done false
24222 7900 Acked Read 7 Sent Done Done false
24223 7901 Acked Write 8 Sent Done Done false
24224 7902 Acked Read 1 Sent Done Done false
24225 7903 Acked Write 6 Sent Done Done false
24226 7904 Acked Flush 0 Sent Done Done false
24227 7905 Acked Write 10 Sent Done Done false
24228 7906 Acked Write 10 Sent Done Done false
24229 7907 Acked Write 5 Sent Done Done false
24230 7908 Acked Flush 0 Sent Done Done false
24231 7909 Acked Read 4 Sent Done Done false
24232 7910 Acked Read 10 Sent Done Done false
24233 7911 Acked Write 3 Sent Done Done false
24234 7912 Acked Read 6 Sent Done Done false
24235 7913 Acked Read 1 Sent Done Done false
24236 7914 Acked Read 5 Sent Done Done false
24237 7915 Acked Read 3 Sent Done Done false
24238 7916 Acked Read 9 Sent Done Done false
24239 7917 Acked Read 8 Sent Done Done false
24240 7918 Acked Read 2 Sent Done Done false
24241 7919 Acked Read 5 Sent Done Done false
24242 7920 Acked Write 7 Sent Done Done false
24243 7921 Acked Write 3 Sent Done Done false
24244 7922 Acked Read 5 Sent Done Done false
24245 7923 Acked Read 5 Sent Done Done false
24246 7924 Acked Write 7 Sent Done Done false
24247 7925 Acked Write 8 Sent Done Done false
24248 7926 Acked Read 1 Sent Done Done false
24249 7927 Acked Write 3 Sent Done Done false
24250 7928 Acked Flush 0 Sent Done Done false
24251 7929 Acked Write 5 Sent Done Done false
24252 7930 Acked Read 10 Sent Done Done false
24253 7931 Acked Write 6 Sent Done Done false
24254 7932 Acked Write 1 Sent Done Done false
24255 7933 Acked Write 3 Sent Done Done false
24256 7934 Acked Write 3 Sent Done Done false
24257 7935 Acked Write 4 Sent Done Done false
24258 7936 Acked Write 6 Sent Done Done false
24259 7937 Acked Read 3 Sent Done Done false
24260 7938 Acked Write 8 Sent Done Done false
24261 7939 Acked Write 6 Sent Done Done false
24262 7940 Acked Read 1 Sent Done Done false
24263 7941 Acked Write 8 Sent Done Done false
24264 7942 Acked Read 5 Sent Done Done false
24265 STATES DS:0 DS:1 DS:2 TOTAL
24266 Sent 301 0 0 301
24267 Done 0 301 301 602
24268 Skipped 0 0 0 0
24269 Error 0 0 0 0
24270 Last Flush: Some(JobId(7641)) Some(JobId(7928)) Some(JobId(7928))
24271 Downstairs last five completed: 7641 7640 7639 7638 7637
24272 Upstairs last five completed: 7942 7941 7940 7939 7938
24273 CLIENT: Up:0 ds:301 act:2
242742025-01-28T21:59:32.060ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
242752025-01-28T21:59:32.801ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
242762025-01-28T21:59:33.803ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
242772025-01-28T21:59:34.803ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
24278 ----------------------------------------------------------------
24279 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24280 States: Offline Active Active
24281 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24282 7642 Acked Read 1 Sent Done Done false
24283 7643 Acked Read 9 Sent Done Done false
24284 7644 Acked Read 6 Sent Done Done false
24285 7645 Acked Flush 0 Sent Done Done false
24286 7646 Acked Write 9 Sent Done Done false
24287 7647 Acked Read 7 Sent Done Done false
24288 7648 Acked Read 4 Sent Done Done false
24289 7649 Acked Write 6 Sent Done Done false
24290 7650 Acked Flush 0 Sent Done Done false
24291 7651 Acked Write 3 Sent Done Done false
24292 7652 Acked Write 5 Sent Done Done false
24293 7653 Acked Read 4 Sent Done Done false
24294 7654 Acked Read 1 Sent Done Done false
24295 7655 Acked Flush 0 Sent Done Done false
24296 7656 Acked Read 2 Sent Done Done false
24297 7657 Acked Read 3 Sent Done Done false
24298 7658 Acked Read 7 Sent Done Done false
24299 7659 Acked Write 2 Sent Done Done false
24300 7660 Acked Write 4 Sent Done Done false
24301 7661 Acked Write 2 Sent Done Done false
24302 7662 Acked Read 1 Sent Done Done false
24303 7663 Acked Write 7 Sent Done Done false
24304 7664 Acked Write 7 Sent Done Done false
24305 7665 Acked Flush 0 Sent Done Done false
24306 7666 Acked Write 8 Sent Done Done false
24307 7667 Acked Write 4 Sent Done Done false
24308 7668 Acked Read 7 Sent Done Done false
24309 7669 Acked Write 6 Sent Done Done false
24310 7670 Acked Read 1 Sent Done Done false
24311 7671 Acked Read 8 Sent Done Done false
24312 7672 Acked Write 5 Sent Done Done false
24313 7673 Acked Read 8 Sent Done Done false
24314 7674 Acked Write 7 Sent Done Done false
24315 7675 Acked Write 5 Sent Done Done false
24316 7676 Acked Read 5 Sent Done Done false
24317 7677 Acked Write 7 Sent Done Done false
24318 7678 Acked Read 9 Sent Done Done false
24319 7679 Acked Write 6 Sent Done Done false
24320 7680 Acked Read 5 Sent Done Done false
24321 7681 Acked Write 10 Sent Done Done false
24322 7682 Acked Write 1 Sent Done Done false
24323 7683 Acked Write 10 Sent Done Done false
24324 7684 Acked Write 6 Sent Done Done false
24325 7685 Acked Write 2 Sent Done Done false
24326 7686 Acked Read 10 Sent Done Done false
24327 7687 Acked Read 5 Sent Done Done false
24328 7688 Acked Read 9 Sent Done Done false
24329 7689 Acked Write 9 Sent Done Done false
24330 7690 Acked Read 8 Sent Done Done false
24331 7691 Acked Read 1 Sent Done Done false
24332 7692 Acked Read 3 Sent Done Done false
24333 7693 Acked Read 3 Sent Done Done false
24334 7694 Acked Write 1 Sent Done Done false
24335 7695 Acked Write 5 Sent Done Done false
24336 7696 Acked Write 3 Sent Done Done false
24337 7697 Acked Read 4 Sent Done Done false
24338 7698 Acked Read 1 Sent Done Done false
24339 7699 Acked Read 1 Sent Done Done false
24340 7700 Acked Read 1 Sent Done Done false
24341 7701 Acked Flush 0 Sent Done Done false
24342 7702 Acked Flush 0 Sent Done Done false
24343 7703 Acked Write 8 Sent Done Done false
24344 7704 Acked Read 7 Sent Done Done false
24345 7705 Acked Write 2 Sent Done Done false
24346 7706 Acked Read 5 Sent Done Done false
24347 7707 Acked Read 8 Sent Done Done false
24348 7708 Acked Write 6 Sent Done Done false
24349 7709 Acked Read 7 Sent Done Done false
24350 7710 Acked Flush 0 Sent Done Done false
24351 7711 Acked Read 7 Sent Done Done false
24352 7712 Acked Read 4 Sent Done Done false
24353 7713 Acked Write 9 Sent Done Done false
24354 7714 Acked Write 5 Sent Done Done false
24355 7715 Acked Read 4 Sent Done Done false
24356 7716 Acked Read 5 Sent Done Done false
24357 7717 Acked Read 6 Sent Done Done false
24358 7718 Acked Write 5 Sent Done Done false
24359 7719 Acked Write 5 Sent Done Done false
24360 7720 Acked Flush 0 Sent Done Done false
24361 7721 Acked Read 5 Sent Done Done false
24362 7722 Acked Read 10 Sent Done Done false
24363 7723 Acked Write 1 Sent Done Done false
24364 7724 Acked Flush 0 Sent Done Done false
24365 7725 Acked Write 3 Sent Done Done false
24366 7726 Acked Read 5 Sent Done Done false
24367 7727 Acked Flush 0 Sent Done Done false
24368 7728 Acked Read 3 Sent Done Done false
24369 7729 Acked Write 8 Sent Done Done false
24370 7730 Acked Flush 0 Sent Done Done false
24371 7731 Acked Read 2 Sent Done Done false
24372 7732 Acked Read 3 Sent Done Done false
24373 7733 Acked Write 6 Sent Done Done false
24374 7734 Acked Write 6 Sent Done Done false
24375 7735 Acked Write 7 Sent Done Done false
24376 7736 Acked Write 3 Sent Done Done false
24377 7737 Acked Write 9 Sent Done Done false
24378 7738 Acked Write 5 Sent Done Done false
24379 7739 Acked Write 4 Sent Done Done false
24380 7740 Acked Read 9 Sent Done Done false
24381 7741 Acked Write 9 Sent Done Done false
24382 7742 Acked Write 4 Sent Done Done false
24383 7743 Acked Write 5 Sent Done Done false
24384 7744 Acked Write 6 Sent Done Done false
24385 7745 Acked Read 9 Sent Done Done false
24386 7746 Acked Write 2 Sent Done Done false
24387 7747 Acked Read 8 Sent Done Done false
24388 7748 Acked Read 3 Sent Done Done false
24389 7749 Acked Write 2 Sent Done Done false
24390 7750 Acked Read 1 Sent Done Done false
24391 7751 Acked Read 2 Sent Done Done false
24392 7752 Acked Write 4 Sent Done Done false
24393 7753 Acked Flush 0 Sent Done Done false
24394 7754 Acked Read 10 Sent Done Done false
24395 7755 Acked Write 6 Sent Done Done false
24396 7756 Acked Read 6 Sent Done Done false
24397 7757 Acked Read 6 Sent Done Done false
24398 7758 Acked Flush 0 Sent Done Done false
24399 7759 Acked Read 4 Sent Done Done false
24400 7760 Acked Read 9 Sent Done Done false
24401 7761 Acked Write 9 Sent Done Done false
24402 7762 Acked Write 8 Sent Done Done false
24403 7763 Acked Write 1 Sent Done Done false
24404 7764 Acked Read 4 Sent Done Done false
24405 7765 Acked Flush 0 Sent Done Done false
24406 7766 Acked Read 7 Sent Done Done false
24407 7767 Acked Write 6 Sent Done Done false
24408 7768 Acked Read 9 Sent Done Done false
24409 7769 Acked Read 3 Sent Done Done false
24410 7770 Acked Write 4 Sent Done Done false
24411 7771 Acked Read 6 Sent Done Done false
24412 7772 Acked Read 9 Sent Done Done false
24413 7773 Acked Read 10 Sent Done Done false
24414 7774 Acked Read 2 Sent Done Done false
24415 7775 Acked Flush 0 Sent Done Done false
24416 7776 Acked Write 3 Sent Done Done false
24417 7777 Acked Write 8 Sent Done Done false
24418 7778 Acked Read 9 Sent Done Done false
24419 7779 Acked Read 3 Sent Done Done false
24420 7780 Acked Read 8 Sent Done Done false
24421 7781 Acked Read 3 Sent Done Done false
24422 7782 Acked Flush 0 Sent Done Done false
24423 7783 Acked Read 4 Sent Done Done false
24424 7784 Acked Read 3 Sent Done Done false
24425 7785 Acked Write 4 Sent Done Done false
24426 7786 Acked Write 8 Sent Done Done false
24427 7787 Acked Write 7 Sent Done Done false
24428 7788 Acked Flush 0 Sent Done Done false
24429 7789 Acked Write 8 Sent Done Done false
24430 7790 Acked Flush 0 Sent Done Done false
24431 7791 Acked Write 3 Sent Done Done false
24432 7792 Acked Read 4 Sent Done Done false
24433 7793 Acked Read 1 Sent Done Done false
24434 7794 Acked Read 9 Sent Done Done false
24435 7795 Acked Read 7 Sent Done Done false
24436 7796 Acked Write 9 Sent Done Done false
24437 7797 Acked Read 3 Sent Done Done false
24438 7798 Acked Read 1 Sent Done Done false
24439 7799 Acked Write 9 Sent Done Done false
24440 7800 Acked Write 3 Sent Done Done false
24441 7801 Acked Write 2 Sent Done Done false
24442 7802 Acked Write 1 Sent Done Done false
24443 7803 Acked Read 7 Sent Done Done false
24444 7804 Acked Read 7 Sent Done Done false
24445 7805 Acked Write 7 Sent Done Done false
24446 7806 Acked Read 1 Sent Done Done false
24447 7807 Acked Read 6 Sent Done Done false
24448 7808 Acked Write 5 Sent Done Done false
24449 7809 Acked Read 2 Sent Done Done false
24450 7810 Acked Read 5 Sent Done Done false
24451 7811 Acked Write 4 Sent Done Done false
24452 7812 Acked Write 4 Sent Done Done false
24453 7813 Acked Read 8 Sent Done Done false
24454 7814 Acked Write 5 Sent Done Done false
24455 7815 Acked Read 2 Sent Done Done false
24456 7816 Acked Read 6 Sent Done Done false
24457 7817 Acked Read 3 Sent Done Done false
24458 7818 Acked Write 8 Sent Done Done false
24459 7819 Acked Flush 0 Sent Done Done false
24460 7820 Acked Write 8 Sent Done Done false
24461 7821 Acked Write 2 Sent Done Done false
24462 7822 Acked Read 4 Sent Done Done false
24463 7823 Acked Read 1 Sent Done Done false
24464 7824 Acked Read 10 Sent Done Done false
24465 7825 Acked Read 6 Sent Done Done false
24466 7826 Acked Flush 0 Sent Done Done false
24467 7827 Acked Write 1 Sent Done Done false
24468 7828 Acked Write 10 Sent Done Done false
24469 7829 Acked Read 7 Sent Done Done false
24470 7830 Acked Read 9 Sent Done Done false
24471 7831 Acked Read 1 Sent Done Done false
24472 7832 Acked Write 10 Sent Done Done false
24473 7833 Acked Flush 0 Sent Done Done false
24474 7834 Acked Write 6 Sent Done Done false
24475 7835 Acked Write 1 Sent Done Done false
24476 7836 Acked Write 8 Sent Done Done false
24477 7837 Acked Flush 0 Sent Done Done false
24478 7838 Acked Read 8 Sent Done Done false
24479 7839 Acked Write 2 Sent Done Done false
24480 7840 Acked Write 3 Sent Done Done false
24481 7841 Acked Write 4 Sent Done Done false
24482 7842 Acked Write 2 Sent Done Done false
24483 7843 Acked Read 7 Sent Done Done false
24484 7844 Acked Read 3 Sent Done Done false
24485 7845 Acked Write 1 Sent Done Done false
24486 7846 Acked Write 6 Sent Done Done false
24487 7847 Acked Read 2 Sent Done Done false
24488 7848 Acked Write 4 Sent Done Done false
24489 7849 Acked Read 5 Sent Done Done false
24490 7850 Acked Write 1 Sent Done Done false
24491 7851 Acked Read 8 Sent Done Done false
24492 7852 Acked Write 5 Sent Done Done false
24493 7853 Acked Write 10 Sent Done Done false
24494 7854 Acked Read 2 Sent Done Done false
24495 7855 Acked Read 5 Sent Done Done false
24496 7856 Acked Flush 0 Sent Done Done false
24497 7857 Acked Read 9 Sent Done Done false
24498 7858 Acked Write 3 Sent Done Done false
24499 7859 Acked Write 6 Sent Done Done false
24500 7860 Acked Read 7 Sent Done Done false
24501 7861 Acked Read 9 Sent Done Done false
24502 7862 Acked Read 6 Sent Done Done false
24503 7863 Acked Read 6 Sent Done Done false
24504 7864 Acked Read 2 Sent Done Done false
24505 7865 Acked Read 4 Sent Done Done false
24506 7866 Acked Write 7 Sent Done Done false
24507 7867 Acked Read 2 Sent Done Done false
24508 7868 Acked Flush 0 Sent Done Done false
24509 7869 Acked Write 6 Sent Done Done false
24510 7870 Acked Write 5 Sent Done Done false
24511 7871 Acked Write 6 Sent Done Done false
24512 7872 Acked Read 2 Sent Done Done false
24513 7873 Acked Read 3 Sent Done Done false
24514 7874 Acked Write 1 Sent Done Done false
24515 7875 Acked Read 3 Sent Done Done false
24516 7876 Acked Write 2 Sent Done Done false
24517 7877 Acked Write 3 Sent Done Done false
24518 7878 Acked Read 3 Sent Done Done false
24519 7879 Acked Write 4 Sent Done Done false
24520 7880 Acked Flush 0 Sent Done Done false
24521 7881 Acked Write 6 Sent Done Done false
24522 7882 Acked Read 7 Sent Done Done false
24523 7883 Acked Write 9 Sent Done Done false
24524 7884 Acked Read 9 Sent Done Done false
24525 7885 Acked Write 1 Sent Done Done false
24526 7886 Acked Flush 0 Sent Done Done false
24527 7887 Acked Write 7 Sent Done Done false
24528 7888 Acked Read 3 Sent Done Done false
24529 7889 Acked Read 3 Sent Done Done false
24530 7890 Acked Write 9 Sent Done Done false
24531 7891 Acked Write 2 Sent Done Done false
24532 7892 Acked Read 6 Sent Done Done false
24533 7893 Acked Write 3 Sent Done Done false
24534 7894 Acked Read 6 Sent Done Done false
24535 7895 Acked Read 2 Sent Done Done false
24536 7896 Acked Read 4 Sent Done Done false
24537 7897 Acked Write 3 Sent Done Done false
24538 7898 Acked Write 10 Sent Done Done false
24539 7899 Acked Read 3 Sent Done Done false
24540 7900 Acked Read 7 Sent Done Done false
24541 7901 Acked Write 8 Sent Done Done false
24542 7902 Acked Read 1 Sent Done Done false
24543 7903 Acked Write 6 Sent Done Done false
24544 7904 Acked Flush 0 Sent Done Done false
24545 7905 Acked Write 10 Sent Done Done false
24546 7906 Acked Write 10 Sent Done Done false
24547 7907 Acked Write 5 Sent Done Done false
24548 7908 Acked Flush 0 Sent Done Done false
24549 7909 Acked Read 4 Sent Done Done false
24550 7910 Acked Read 10 Sent Done Done false
24551 7911 Acked Write 3 Sent Done Done false
24552 7912 Acked Read 6 Sent Done Done false
24553 7913 Acked Read 1 Sent Done Done false
24554 7914 Acked Read 5 Sent Done Done false
24555 7915 Acked Read 3 Sent Done Done false
24556 7916 Acked Read 9 Sent Done Done false
24557 7917 Acked Read 8 Sent Done Done false
24558 7918 Acked Read 2 Sent Done Done false
24559 7919 Acked Read 5 Sent Done Done false
24560 7920 Acked Write 7 Sent Done Done false
24561 7921 Acked Write 3 Sent Done Done false
24562 7922 Acked Read 5 Sent Done Done false
24563 7923 Acked Read 5 Sent Done Done false
24564 7924 Acked Write 7 Sent Done Done false
24565 7925 Acked Write 8 Sent Done Done false
24566 7926 Acked Read 1 Sent Done Done false
24567 7927 Acked Write 3 Sent Done Done false
24568 7928 Acked Flush 0 Sent Done Done false
24569 7929 Acked Write 5 Sent Done Done false
24570 7930 Acked Read 10 Sent Done Done false
24571 7931 Acked Write 6 Sent Done Done false
24572 7932 Acked Write 1 Sent Done Done false
24573 7933 Acked Write 3 Sent Done Done false
24574 7934 Acked Write 3 Sent Done Done false
24575 7935 Acked Write 4 Sent Done Done false
24576 7936 Acked Write 6 Sent Done Done false
24577 7937 Acked Read 3 Sent Done Done false
24578 7938 Acked Write 8 Sent Done Done false
24579 7939 Acked Write 6 Sent Done Done false
24580 7940 Acked Read 1 Sent Done Done false
24581 7941 Acked Write 8 Sent Done Done false
24582 7942 Acked Read 5 Sent Done Done false
24583 STATES DS:0 DS:1 DS:2 TOTAL
24584 Sent 301 0 0 301
24585 Done 0 301 301 602
24586 Skipped 0 0 0 0
24587 Error 0 0 0 0
24588 Last Flush: Some(JobId(7641)) Some(JobId(7928)) Some(JobId(7928))
24589 Downstairs last five completed: 7641 7640 7639 7638 7637
24590 Upstairs last five completed: 7942 7941 7940 7939 7938
24591 CLIENT: Up:0 ds:301 act:2
245922025-01-28T21:59:36.107ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
245932025-01-28T21:59:36.806ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
245942025-01-28T21:59:37.330ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
245952025-01-28T21:59:37.330ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
245962025-01-28T21:59:37.330ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
245972025-01-28T21:59:37.330ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
245982025-01-28T21:59:37.330ZINFOcrucible: client task is exiting = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
245992025-01-28T21:59:37.831ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
246002025-01-28T21:59:38.810ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
24601 [8810] got start action
24602 [8810] I am going to restart
24603 Starting downstairs at port 8810
24604 Make output file at "/tmp/dsc/downstairs-8810.txt"
24605 [8810][0] reports Starting
24606 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1225)
24607 [8810][0] reports Running
24608 ----------------------------------------------------------------
24609 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24610 States: Offline Active Active
24611 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24612 7642 Acked Read 1 Sent Done Done false
24613 7643 Acked Read 9 Sent Done Done false
24614 7644 Acked Read 6 Sent Done Done false
24615 7645 Acked Flush 0 Sent Done Done false
24616 7646 Acked Write 9 Sent Done Done false
24617 7647 Acked Read 7 Sent Done Done false
24618 7648 Acked Read 4 Sent Done Done false
24619 7649 Acked Write 6 Sent Done Done false
24620 7650 Acked Flush 0 Sent Done Done false
24621 7651 Acked Write 3 Sent Done Done false
24622 7652 Acked Write 5 Sent Done Done false
24623 7653 Acked Read 4 Sent Done Done false
24624 7654 Acked Read 1 Sent Done Done false
24625 7655 Acked Flush 0 Sent Done Done false
24626 7656 Acked Read 2 Sent Done Done false
24627 7657 Acked Read 3 Sent Done Done false
24628 7658 Acked Read 7 Sent Done Done false
24629 7659 Acked Write 2 Sent Done Done false
24630 7660 Acked Write 4 Sent Done Done false
24631 7661 Acked Write 2 Sent Done Done false
24632 7662 Acked Read 1 Sent Done Done false
24633 7663 Acked Write 7 Sent Done Done false
24634 7664 Acked Write 7 Sent Done Done false
24635 7665 Acked Flush 0 Sent Done Done false
24636 7666 Acked Write 8 Sent Done Done false
24637 7667 Acked Write 4 Sent Done Done false
24638 7668 Acked Read 7 Sent Done Done false
24639 7669 Acked Write 6 Sent Done Done false
24640 7670 Acked Read 1 Sent Done Done false
24641 7671 Acked Read 8 Sent Done Done false
24642 7672 Acked Write 5 Sent Done Done false
24643 7673 Acked Read 8 Sent Done Done false
24644 7674 Acked Write 7 Sent Done Done false
24645 7675 Acked Write 5 Sent Done Done false
24646 7676 Acked Read 5 Sent Done Done false
24647 7677 Acked Write 7 Sent Done Done false
24648 7678 Acked Read 9 Sent Done Done false
24649 7679 Acked Write 6 Sent Done Done false
24650 7680 Acked Read 5 Sent Done Done false
24651 7681 Acked Write 10 Sent Done Done false
24652 7682 Acked Write 1 Sent Done Done false
24653 7683 Acked Write 10 Sent Done Done false
24654 7684 Acked Write 6 Sent Done Done false
24655 7685 Acked Write 2 Sent Done Done false
24656 7686 Acked Read 10 Sent Done Done false
24657 7687 Acked Read 5 Sent Done Done false
24658 7688 Acked Read 9 Sent Done Done false
24659 7689 Acked Write 9 Sent Done Done false
24660 7690 Acked Read 8 Sent Done Done false
24661 7691 Acked Read 1 Sent Done Done false
24662 7692 Acked Read 3 Sent Done Done false
24663 7693 Acked Read 3 Sent Done Done false
24664 7694 Acked Write 1 Sent Done Done false
24665 7695 Acked Write 5 Sent Done Done false
24666 7696 Acked Write 3 Sent Done Done false
24667 7697 Acked Read 4 Sent Done Done false
24668 7698 Acked Read 1 Sent Done Done false
24669 7699 Acked Read 1 Sent Done Done false
24670 7700 Acked Read 1 Sent Done Done false
24671 7701 Acked Flush 0 Sent Done Done false
24672 7702 Acked Flush 0 Sent Done Done false
24673 7703 Acked Write 8 Sent Done Done false
24674 7704 Acked Read 7 Sent Done Done false
24675 7705 Acked Write 2 Sent Done Done false
24676 7706 Acked Read 5 Sent Done Done false
24677 7707 Acked Read 8 Sent Done Done false
24678 7708 Acked Write 6 Sent Done Done false
24679 7709 Acked Read 7 Sent Done Done false
24680 7710 Acked Flush 0 Sent Done Done false
24681 7711 Acked Read 7 Sent Done Done false
24682 7712 Acked Read 4 Sent Done Done false
24683 7713 Acked Write 9 Sent Done Done false
24684 7714 Acked Write 5 Sent Done Done false
24685 7715 Acked Read 4 Sent Done Done false
24686 7716 Acked Read 5 Sent Done Done false
24687 7717 Acked Read 6 Sent Done Done false
24688 7718 Acked Write 5 Sent Done Done false
24689 7719 Acked Write 5 Sent Done Done false
24690 7720 Acked Flush 0 Sent Done Done false
24691 7721 Acked Read 5 Sent Done Done false
24692 7722 Acked Read 10 Sent Done Done false
24693 7723 Acked Write 1 Sent Done Done false
24694 7724 Acked Flush 0 Sent Done Done false
24695 7725 Acked Write 3 Sent Done Done false
24696 7726 Acked Read 5 Sent Done Done false
24697 7727 Acked Flush 0 Sent Done Done false
24698 7728 Acked Read 3 Sent Done Done false
24699 7729 Acked Write 8 Sent Done Done false
24700 7730 Acked Flush 0 Sent Done Done false
24701 7731 Acked Read 2 Sent Done Done false
24702 7732 Acked Read 3 Sent Done Done false
24703 7733 Acked Write 6 Sent Done Done false
24704 7734 Acked Write 6 Sent Done Done false
24705 7735 Acked Write 7 Sent Done Done false
24706 7736 Acked Write 3 Sent Done Done false
24707 7737 Acked Write 9 Sent Done Done false
24708 7738 Acked Write 5 Sent Done Done false
24709 7739 Acked Write 4 Sent Done Done false
24710 7740 Acked Read 9 Sent Done Done false
24711 7741 Acked Write 9 Sent Done Done false
24712 7742 Acked Write 4 Sent Done Done false
24713 7743 Acked Write 5 Sent Done Done false
24714 7744 Acked Write 6 Sent Done Done false
24715 7745 Acked Read 9 Sent Done Done false
24716 7746 Acked Write 2 Sent Done Done false
24717 7747 Acked Read 8 Sent Done Done false
24718 7748 Acked Read 3 Sent Done Done false
24719 7749 Acked Write 2 Sent Done Done false
24720 7750 Acked Read 1 Sent Done Done false
24721 7751 Acked Read 2 Sent Done Done false
24722 7752 Acked Write 4 Sent Done Done false
24723 7753 Acked Flush 0 Sent Done Done false
24724 7754 Acked Read 10 Sent Done Done false
24725 7755 Acked Write 6 Sent Done Done false
24726 7756 Acked Read 6 Sent Done Done false
24727 7757 Acked Read 6 Sent Done Done false
24728 7758 Acked Flush 0 Sent Done Done false
24729 7759 Acked Read 4 Sent Done Done false
24730 7760 Acked Read 9 Sent Done Done false
24731 7761 Acked Write 9 Sent Done Done false
24732 7762 Acked Write 8 Sent Done Done false
24733 7763 Acked Write 1 Sent Done Done false
24734 7764 Acked Read 4 Sent Done Done false
24735 7765 Acked Flush 0 Sent Done Done false
24736 7766 Acked Read 7 Sent Done Done false
24737 7767 Acked Write 6 Sent Done Done false
24738 7768 Acked Read 9 Sent Done Done false
24739 7769 Acked Read 3 Sent Done Done false
24740 7770 Acked Write 4 Sent Done Done false
24741 7771 Acked Read 6 Sent Done Done false
24742 7772 Acked Read 9 Sent Done Done false
24743 7773 Acked Read 10 Sent Done Done false
24744 7774 Acked Read 2 Sent Done Done false
24745 7775 Acked Flush 0 Sent Done Done false
24746 7776 Acked Write 3 Sent Done Done false
24747 7777 Acked Write 8 Sent Done Done false
24748 7778 Acked Read 9 Sent Done Done false
24749 7779 Acked Read 3 Sent Done Done false
24750 7780 Acked Read 8 Sent Done Done false
24751 7781 Acked Read 3 Sent Done Done false
24752 7782 Acked Flush 0 Sent Done Done false
24753 7783 Acked Read 4 Sent Done Done false
24754 7784 Acked Read 3 Sent Done Done false
24755 7785 Acked Write 4 Sent Done Done false
24756 7786 Acked Write 8 Sent Done Done false
24757 7787 Acked Write 7 Sent Done Done false
24758 7788 Acked Flush 0 Sent Done Done false
24759 7789 Acked Write 8 Sent Done Done false
24760 7790 Acked Flush 0 Sent Done Done false
24761 7791 Acked Write 3 Sent Done Done false
24762 7792 Acked Read 4 Sent Done Done false
24763 7793 Acked Read 1 Sent Done Done false
24764 7794 Acked Read 9 Sent Done Done false
24765 7795 Acked Read 7 Sent Done Done false
24766 7796 Acked Write 9 Sent Done Done false
24767 7797 Acked Read 3 Sent Done Done false
24768 7798 Acked Read 1 Sent Done Done false
24769 7799 Acked Write 9 Sent Done Done false
24770 7800 Acked Write 3 Sent Done Done false
24771 7801 Acked Write 2 Sent Done Done false
24772 7802 Acked Write 1 Sent Done Done false
24773 7803 Acked Read 7 Sent Done Done false
24774 7804 Acked Read 7 Sent Done Done false
24775 7805 Acked Write 7 Sent Done Done false
24776 7806 Acked Read 1 Sent Done Done false
24777 7807 Acked Read 6 Sent Done Done false
24778 7808 Acked Write 5 Sent Done Done false
24779 7809 Acked Read 2 Sent Done Done false
24780 7810 Acked Read 5 Sent Done Done false
24781 7811 Acked Write 4 Sent Done Done false
24782 7812 Acked Write 4 Sent Done Done false
24783 7813 Acked Read 8 Sent Done Done false
24784 7814 Acked Write 5 Sent Done Done false
24785 7815 Acked Read 2 Sent Done Done false
24786 7816 Acked Read 6 Sent Done Done false
24787 7817 Acked Read 3 Sent Done Done false
24788 7818 Acked Write 8 Sent Done Done false
24789 7819 Acked Flush 0 Sent Done Done false
24790 7820 Acked Write 8 Sent Done Done false
24791 7821 Acked Write 2 Sent Done Done false
24792 7822 Acked Read 4 Sent Done Done false
24793 7823 Acked Read 1 Sent Done Done false
24794 7824 Acked Read 10 Sent Done Done false
24795 7825 Acked Read 6 Sent Done Done false
24796 7826 Acked Flush 0 Sent Done Done false
24797 7827 Acked Write 1 Sent Done Done false
24798 7828 Acked Write 10 Sent Done Done false
24799 7829 Acked Read 7 Sent Done Done false
24800 7830 Acked Read 9 Sent Done Done false
24801 7831 Acked Read 1 Sent Done Done false
24802 7832 Acked Write 10 Sent Done Done false
24803 7833 Acked Flush 0 Sent Done Done false
24804 7834 Acked Write 6 Sent Done Done false
24805 7835 Acked Write 1 Sent Done Done false
24806 7836 Acked Write 8 Sent Done Done false
24807 7837 Acked Flush 0 Sent Done Done false
24808 7838 Acked Read 8 Sent Done Done false
24809 7839 Acked Write 2 Sent Done Done false
24810 7840 Acked Write 3 Sent Done Done false
24811 7841 Acked Write 4 Sent Done Done false
24812 7842 Acked Write 2 Sent Done Done false
24813 7843 Acked Read 7 Sent Done Done false
24814 7844 Acked Read 3 Sent Done Done false
24815 7845 Acked Write 1 Sent Done Done false
24816 7846 Acked Write 6 Sent Done Done false
24817 7847 Acked Read 2 Sent Done Done false
24818 7848 Acked Write 4 Sent Done Done false
24819 7849 Acked Read 5 Sent Done Done false
24820 7850 Acked Write 1 Sent Done Done false
24821 7851 Acked Read 8 Sent Done Done false
24822 7852 Acked Write 5 Sent Done Done false
24823 7853 Acked Write 10 Sent Done Done false
24824 7854 Acked Read 2 Sent Done Done false
24825 7855 Acked Read 5 Sent Done Done false
24826 7856 Acked Flush 0 Sent Done Done false
24827 7857 Acked Read 9 Sent Done Done false
24828 7858 Acked Write 3 Sent Done Done false
24829 7859 Acked Write 6 Sent Done Done false
24830 7860 Acked Read 7 Sent Done Done false
24831 7861 Acked Read 9 Sent Done Done false
24832 7862 Acked Read 6 Sent Done Done false
24833 7863 Acked Read 6 Sent Done Done false
24834 7864 Acked Read 2 Sent Done Done false
24835 7865 Acked Read 4 Sent Done Done false
24836 7866 Acked Write 7 Sent Done Done false
24837 7867 Acked Read 2 Sent Done Done false
24838 7868 Acked Flush 0 Sent Done Done false
24839 7869 Acked Write 6 Sent Done Done false
24840 7870 Acked Write 5 Sent Done Done false
24841 7871 Acked Write 6 Sent Done Done false
24842 7872 Acked Read 2 Sent Done Done false
24843 7873 Acked Read 3 Sent Done Done false
24844 7874 Acked Write 1 Sent Done Done false
24845 7875 Acked Read 3 Sent Done Done false
24846 7876 Acked Write 2 Sent Done Done false
24847 7877 Acked Write 3 Sent Done Done false
24848 7878 Acked Read 3 Sent Done Done false
24849 7879 Acked Write 4 Sent Done Done false
24850 7880 Acked Flush 0 Sent Done Done false
24851 7881 Acked Write 6 Sent Done Done false
24852 7882 Acked Read 7 Sent Done Done false
24853 7883 Acked Write 9 Sent Done Done false
24854 7884 Acked Read 9 Sent Done Done false
24855 7885 Acked Write 1 Sent Done Done false
24856 7886 Acked Flush 0 Sent Done Done false
24857 7887 Acked Write 7 Sent Done Done false
24858 7888 Acked Read 3 Sent Done Done false
24859 7889 Acked Read 3 Sent Done Done false
24860 7890 Acked Write 9 Sent Done Done false
24861 7891 Acked Write 2 Sent Done Done false
24862 7892 Acked Read 6 Sent Done Done false
24863 7893 Acked Write 3 Sent Done Done false
24864 7894 Acked Read 6 Sent Done Done false
24865 7895 Acked Read 2 Sent Done Done false
24866 7896 Acked Read 4 Sent Done Done false
24867 7897 Acked Write 3 Sent Done Done false
24868 7898 Acked Write 10 Sent Done Done false
24869 7899 Acked Read 3 Sent Done Done false
24870 7900 Acked Read 7 Sent Done Done false
24871 7901 Acked Write 8 Sent Done Done false
24872 7902 Acked Read 1 Sent Done Done false
24873 7903 Acked Write 6 Sent Done Done false
24874 7904 Acked Flush 0 Sent Done Done false
24875 7905 Acked Write 10 Sent Done Done false
24876 7906 Acked Write 10 Sent Done Done false
24877 7907 Acked Write 5 Sent Done Done false
24878 7908 Acked Flush 0 Sent Done Done false
24879 7909 Acked Read 4 Sent Done Done false
24880 7910 Acked Read 10 Sent Done Done false
24881 7911 Acked Write 3 Sent Done Done false
24882 7912 Acked Read 6 Sent Done Done false
24883 7913 Acked Read 1 Sent Done Done false
24884 7914 Acked Read 5 Sent Done Done false
24885 7915 Acked Read 3 Sent Done Done false
24886 7916 Acked Read 9 Sent Done Done false
24887 7917 Acked Read 8 Sent Done Done false
24888 7918 Acked Read 2 Sent Done Done false
24889 7919 Acked Read 5 Sent Done Done false
24890 7920 Acked Write 7 Sent Done Done false
24891 7921 Acked Write 3 Sent Done Done false
24892 7922 Acked Read 5 Sent Done Done false
24893 7923 Acked Read 5 Sent Done Done false
24894 7924 Acked Write 7 Sent Done Done false
24895 7925 Acked Write 8 Sent Done Done false
24896 7926 Acked Read 1 Sent Done Done false
24897 7927 Acked Write 3 Sent Done Done false
24898 7928 Acked Flush 0 Sent Done Done false
24899 7929 Acked Write 5 Sent Done Done false
24900 7930 Acked Read 10 Sent Done Done false
24901 7931 Acked Write 6 Sent Done Done false
24902 7932 Acked Write 1 Sent Done Done false
24903 7933 Acked Write 3 Sent Done Done false
24904 7934 Acked Write 3 Sent Done Done false
24905 7935 Acked Write 4 Sent Done Done false
24906 7936 Acked Write 6 Sent Done Done false
24907 7937 Acked Read 3 Sent Done Done false
24908 7938 Acked Write 8 Sent Done Done false
24909 7939 Acked Write 6 Sent Done Done false
24910 7940 Acked Read 1 Sent Done Done false
24911 7941 Acked Write 8 Sent Done Done false
24912 7942 Acked Read 5 Sent Done Done false
24913 STATES DS:0 DS:1 DS:2 TOTAL
24914 Sent 301 0 0 301
24915 Done 0 301 301 602
24916 Skipped 0 0 0 0
24917 Error 0 0 0 0
24918 Last Flush: Some(JobId(7641)) Some(JobId(7928)) Some(JobId(7928))
24919 Downstairs last five completed: 7641 7640 7639 7638 7637
24920 Upstairs last five completed: 7942 7941 7940 7939 7938
24921 CLIENT: Up:0 ds:301 act:2
249222025-01-28T21:59:40.064ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
249232025-01-28T21:59:41.108ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
249242025-01-28T21:59:41.814ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
249252025-01-28T21:59:42.814ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
24926 ----------------------------------------------------------------
24927 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24928 States: Offline Active Active
24929 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24930 7642 Acked Read 1 Sent Done Done false
24931 7643 Acked Read 9 Sent Done Done false
24932 7644 Acked Read 6 Sent Done Done false
24933 7645 Acked Flush 0 Sent Done Done false
24934 7646 Acked Write 9 Sent Done Done false
24935 7647 Acked Read 7 Sent Done Done false
24936 7648 Acked Read 4 Sent Done Done false
24937 7649 Acked Write 6 Sent Done Done false
24938 7650 Acked Flush 0 Sent Done Done false
24939 7651 Acked Write 3 Sent Done Done false
24940 7652 Acked Write 5 Sent Done Done false
24941 7653 Acked Read 4 Sent Done Done false
24942 7654 Acked Read 1 Sent Done Done false
24943 7655 Acked Flush 0 Sent Done Done false
24944 7656 Acked Read 2 Sent Done Done false
24945 7657 Acked Read 3 Sent Done Done false
24946 7658 Acked Read 7 Sent Done Done false
24947 7659 Acked Write 2 Sent Done Done false
24948 7660 Acked Write 4 Sent Done Done false
24949 7661 Acked Write 2 Sent Done Done false
24950 7662 Acked Read 1 Sent Done Done false
24951 7663 Acked Write 7 Sent Done Done false
24952 7664 Acked Write 7 Sent Done Done false
24953 7665 Acked Flush 0 Sent Done Done false
24954 7666 Acked Write 8 Sent Done Done false
24955 7667 Acked Write 4 Sent Done Done false
24956 7668 Acked Read 7 Sent Done Done false
24957 7669 Acked Write 6 Sent Done Done false
24958 7670 Acked Read 1 Sent Done Done false
24959 7671 Acked Read 8 Sent Done Done false
24960 7672 Acked Write 5 Sent Done Done false
24961 7673 Acked Read 8 Sent Done Done false
24962 7674 Acked Write 7 Sent Done Done false
24963 7675 Acked Write 5 Sent Done Done false
24964 7676 Acked Read 5 Sent Done Done false
24965 7677 Acked Write 7 Sent Done Done false
24966 7678 Acked Read 9 Sent Done Done false
24967 7679 Acked Write 6 Sent Done Done false
24968 7680 Acked Read 5 Sent Done Done false
24969 7681 Acked Write 10 Sent Done Done false
24970 7682 Acked Write 1 Sent Done Done false
24971 7683 Acked Write 10 Sent Done Done false
24972 7684 Acked Write 6 Sent Done Done false
24973 7685 Acked Write 2 Sent Done Done false
24974 7686 Acked Read 10 Sent Done Done false
24975 7687 Acked Read 5 Sent Done Done false
24976 7688 Acked Read 9 Sent Done Done false
24977 7689 Acked Write 9 Sent Done Done false
24978 7690 Acked Read 8 Sent Done Done false
24979 7691 Acked Read 1 Sent Done Done false
24980 7692 Acked Read 3 Sent Done Done false
24981 7693 Acked Read 3 Sent Done Done false
24982 7694 Acked Write 1 Sent Done Done false
24983 7695 Acked Write 5 Sent Done Done false
24984 7696 Acked Write 3 Sent Done Done false
24985 7697 Acked Read 4 Sent Done Done false
24986 7698 Acked Read 1 Sent Done Done false
24987 7699 Acked Read 1 Sent Done Done false
24988 7700 Acked Read 1 Sent Done Done false
24989 7701 Acked Flush 0 Sent Done Done false
24990 7702 Acked Flush 0 Sent Done Done false
24991 7703 Acked Write 8 Sent Done Done false
24992 7704 Acked Read 7 Sent Done Done false
24993 7705 Acked Write 2 Sent Done Done false
24994 7706 Acked Read 5 Sent Done Done false
24995 7707 Acked Read 8 Sent Done Done false
24996 7708 Acked Write 6 Sent Done Done false
24997 7709 Acked Read 7 Sent Done Done false
24998 7710 Acked Flush 0 Sent Done Done false
24999 7711 Acked Read 7 Sent Done Done false
25000 7712 Acked Read 4 Sent Done Done false
25001 7713 Acked Write 9 Sent Done Done false
25002 7714 Acked Write 5 Sent Done Done false
25003 7715 Acked Read 4 Sent Done Done false
25004 7716 Acked Read 5 Sent Done Done false
25005 7717 Acked Read 6 Sent Done Done false
25006 7718 Acked Write 5 Sent Done Done false
25007 7719 Acked Write 5 Sent Done Done false
25008 7720 Acked Flush 0 Sent Done Done false
25009 7721 Acked Read 5 Sent Done Done false
25010 7722 Acked Read 10 Sent Done Done false
25011 7723 Acked Write 1 Sent Done Done false
25012 7724 Acked Flush 0 Sent Done Done false
25013 7725 Acked Write 3 Sent Done Done false
25014 7726 Acked Read 5 Sent Done Done false
25015 7727 Acked Flush 0 Sent Done Done false
25016 7728 Acked Read 3 Sent Done Done false
25017 7729 Acked Write 8 Sent Done Done false
25018 7730 Acked Flush 0 Sent Done Done false
25019 7731 Acked Read 2 Sent Done Done false
25020 7732 Acked Read 3 Sent Done Done false
25021 7733 Acked Write 6 Sent Done Done false
25022 7734 Acked Write 6 Sent Done Done false
25023 7735 Acked Write 7 Sent Done Done false
25024 7736 Acked Write 3 Sent Done Done false
25025 7737 Acked Write 9 Sent Done Done false
25026 7738 Acked Write 5 Sent Done Done false
25027 7739 Acked Write 4 Sent Done Done false
25028 7740 Acked Read 9 Sent Done Done false
25029 7741 Acked Write 9 Sent Done Done false
25030 7742 Acked Write 4 Sent Done Done false
25031 7743 Acked Write 5 Sent Done Done false
25032 7744 Acked Write 6 Sent Done Done false
25033 7745 Acked Read 9 Sent Done Done false
25034 7746 Acked Write 2 Sent Done Done false
25035 7747 Acked Read 8 Sent Done Done false
25036 7748 Acked Read 3 Sent Done Done false
25037 7749 Acked Write 2 Sent Done Done false
25038 7750 Acked Read 1 Sent Done Done false
25039 7751 Acked Read 2 Sent Done Done false
25040 7752 Acked Write 4 Sent Done Done false
25041 7753 Acked Flush 0 Sent Done Done false
25042 7754 Acked Read 10 Sent Done Done false
25043 7755 Acked Write 6 Sent Done Done false
25044 7756 Acked Read 6 Sent Done Done false
25045 7757 Acked Read 6 Sent Done Done false
25046 7758 Acked Flush 0 Sent Done Done false
25047 7759 Acked Read 4 Sent Done Done false
25048 7760 Acked Read 9 Sent Done Done false
25049 7761 Acked Write 9 Sent Done Done false
25050 7762 Acked Write 8 Sent Done Done false
25051 7763 Acked Write 1 Sent Done Done false
25052 7764 Acked Read 4 Sent Done Done false
25053 7765 Acked Flush 0 Sent Done Done false
25054 7766 Acked Read 7 Sent Done Done false
25055 7767 Acked Write 6 Sent Done Done false
25056 7768 Acked Read 9 Sent Done Done false
25057 7769 Acked Read 3 Sent Done Done false
25058 7770 Acked Write 4 Sent Done Done false
25059 7771 Acked Read 6 Sent Done Done false
25060 7772 Acked Read 9 Sent Done Done false
25061 7773 Acked Read 10 Sent Done Done false
25062 7774 Acked Read 2 Sent Done Done false
25063 7775 Acked Flush 0 Sent Done Done false
25064 7776 Acked Write 3 Sent Done Done false
25065 7777 Acked Write 8 Sent Done Done false
25066 7778 Acked Read 9 Sent Done Done false
25067 7779 Acked Read 3 Sent Done Done false
25068 7780 Acked Read 8 Sent Done Done false
25069 7781 Acked Read 3 Sent Done Done false
25070 7782 Acked Flush 0 Sent Done Done false
25071 7783 Acked Read 4 Sent Done Done false
25072 7784 Acked Read 3 Sent Done Done false
25073 7785 Acked Write 4 Sent Done Done false
25074 7786 Acked Write 8 Sent Done Done false
25075 7787 Acked Write 7 Sent Done Done false
25076 7788 Acked Flush 0 Sent Done Done false
25077 7789 Acked Write 8 Sent Done Done false
25078 7790 Acked Flush 0 Sent Done Done false
25079 7791 Acked Write 3 Sent Done Done false
25080 7792 Acked Read 4 Sent Done Done false
25081 7793 Acked Read 1 Sent Done Done false
25082 7794 Acked Read 9 Sent Done Done false
25083 7795 Acked Read 7 Sent Done Done false
25084 7796 Acked Write 9 Sent Done Done false
25085 7797 Acked Read 3 Sent Done Done false
25086 7798 Acked Read 1 Sent Done Done false
25087 7799 Acked Write 9 Sent Done Done false
25088 7800 Acked Write 3 Sent Done Done false
25089 7801 Acked Write 2 Sent Done Done false
25090 7802 Acked Write 1 Sent Done Done false
25091 7803 Acked Read 7 Sent Done Done false
25092 7804 Acked Read 7 Sent Done Done false
25093 7805 Acked Write 7 Sent Done Done false
25094 7806 Acked Read 1 Sent Done Done false
25095 7807 Acked Read 6 Sent Done Done false
25096 7808 Acked Write 5 Sent Done Done false
25097 7809 Acked Read 2 Sent Done Done false
25098 7810 Acked Read 5 Sent Done Done false
25099 7811 Acked Write 4 Sent Done Done false
25100 7812 Acked Write 4 Sent Done Done false
25101 7813 Acked Read 8 Sent Done Done false
25102 7814 Acked Write 5 Sent Done Done false
25103 7815 Acked Read 2 Sent Done Done false
25104 7816 Acked Read 6 Sent Done Done false
25105 7817 Acked Read 3 Sent Done Done false
25106 7818 Acked Write 8 Sent Done Done false
25107 7819 Acked Flush 0 Sent Done Done false
25108 7820 Acked Write 8 Sent Done Done false
25109 7821 Acked Write 2 Sent Done Done false
25110 7822 Acked Read 4 Sent Done Done false
25111 7823 Acked Read 1 Sent Done Done false
25112 7824 Acked Read 10 Sent Done Done false
25113 7825 Acked Read 6 Sent Done Done false
25114 7826 Acked Flush 0 Sent Done Done false
25115 7827 Acked Write 1 Sent Done Done false
25116 7828 Acked Write 10 Sent Done Done false
25117 7829 Acked Read 7 Sent Done Done false
25118 7830 Acked Read 9 Sent Done Done false
25119 7831 Acked Read 1 Sent Done Done false
25120 7832 Acked Write 10 Sent Done Done false
25121 7833 Acked Flush 0 Sent Done Done false
25122 7834 Acked Write 6 Sent Done Done false
25123 7835 Acked Write 1 Sent Done Done false
25124 7836 Acked Write 8 Sent Done Done false
25125 7837 Acked Flush 0 Sent Done Done false
25126 7838 Acked Read 8 Sent Done Done false
25127 7839 Acked Write 2 Sent Done Done false
25128 7840 Acked Write 3 Sent Done Done false
25129 7841 Acked Write 4 Sent Done Done false
25130 7842 Acked Write 2 Sent Done Done false
25131 7843 Acked Read 7 Sent Done Done false
25132 7844 Acked Read 3 Sent Done Done false
25133 7845 Acked Write 1 Sent Done Done false
25134 7846 Acked Write 6 Sent Done Done false
25135 7847 Acked Read 2 Sent Done Done false
25136 7848 Acked Write 4 Sent Done Done false
25137 7849 Acked Read 5 Sent Done Done false
25138 7850 Acked Write 1 Sent Done Done false
25139 7851 Acked Read 8 Sent Done Done false
25140 7852 Acked Write 5 Sent Done Done false
25141 7853 Acked Write 10 Sent Done Done false
25142 7854 Acked Read 2 Sent Done Done false
25143 7855 Acked Read 5 Sent Done Done false
25144 7856 Acked Flush 0 Sent Done Done false
25145 7857 Acked Read 9 Sent Done Done false
25146 7858 Acked Write 3 Sent Done Done false
25147 7859 Acked Write 6 Sent Done Done false
25148 7860 Acked Read 7 Sent Done Done false
25149 7861 Acked Read 9 Sent Done Done false
25150 7862 Acked Read 6 Sent Done Done false
25151 7863 Acked Read 6 Sent Done Done false
25152 7864 Acked Read 2 Sent Done Done false
25153 7865 Acked Read 4 Sent Done Done false
25154 7866 Acked Write 7 Sent Done Done false
25155 7867 Acked Read 2 Sent Done Done false
25156 7868 Acked Flush 0 Sent Done Done false
25157 7869 Acked Write 6 Sent Done Done false
25158 7870 Acked Write 5 Sent Done Done false
25159 7871 Acked Write 6 Sent Done Done false
25160 7872 Acked Read 2 Sent Done Done false
25161 7873 Acked Read 3 Sent Done Done false
25162 7874 Acked Write 1 Sent Done Done false
25163 7875 Acked Read 3 Sent Done Done false
25164 7876 Acked Write 2 Sent Done Done false
25165 7877 Acked Write 3 Sent Done Done false
25166 7878 Acked Read 3 Sent Done Done false
25167 7879 Acked Write 4 Sent Done Done false
25168 7880 Acked Flush 0 Sent Done Done false
25169 7881 Acked Write 6 Sent Done Done false
25170 7882 Acked Read 7 Sent Done Done false
25171 7883 Acked Write 9 Sent Done Done false
25172 7884 Acked Read 9 Sent Done Done false
25173 7885 Acked Write 1 Sent Done Done false
25174 7886 Acked Flush 0 Sent Done Done false
25175 7887 Acked Write 7 Sent Done Done false
25176 7888 Acked Read 3 Sent Done Done false
25177 7889 Acked Read 3 Sent Done Done false
25178 7890 Acked Write 9 Sent Done Done false
25179 7891 Acked Write 2 Sent Done Done false
25180 7892 Acked Read 6 Sent Done Done false
25181 7893 Acked Write 3 Sent Done Done false
25182 7894 Acked Read 6 Sent Done Done false
25183 7895 Acked Read 2 Sent Done Done false
25184 7896 Acked Read 4 Sent Done Done false
25185 7897 Acked Write 3 Sent Done Done false
25186 7898 Acked Write 10 Sent Done Done false
25187 7899 Acked Read 3 Sent Done Done false
25188 7900 Acked Read 7 Sent Done Done false
25189 7901 Acked Write 8 Sent Done Done false
25190 7902 Acked Read 1 Sent Done Done false
25191 7903 Acked Write 6 Sent Done Done false
25192 7904 Acked Flush 0 Sent Done Done false
25193 7905 Acked Write 10 Sent Done Done false
25194 7906 Acked Write 10 Sent Done Done false
25195 7907 Acked Write 5 Sent Done Done false
25196 7908 Acked Flush 0 Sent Done Done false
25197 7909 Acked Read 4 Sent Done Done false
25198 7910 Acked Read 10 Sent Done Done false
25199 7911 Acked Write 3 Sent Done Done false
25200 7912 Acked Read 6 Sent Done Done false
25201 7913 Acked Read 1 Sent Done Done false
25202 7914 Acked Read 5 Sent Done Done false
25203 7915 Acked Read 3 Sent Done Done false
25204 7916 Acked Read 9 Sent Done Done false
25205 7917 Acked Read 8 Sent Done Done false
25206 7918 Acked Read 2 Sent Done Done false
25207 7919 Acked Read 5 Sent Done Done false
25208 7920 Acked Write 7 Sent Done Done false
25209 7921 Acked Write 3 Sent Done Done false
25210 7922 Acked Read 5 Sent Done Done false
25211 7923 Acked Read 5 Sent Done Done false
25212 7924 Acked Write 7 Sent Done Done false
25213 7925 Acked Write 8 Sent Done Done false
25214 7926 Acked Read 1 Sent Done Done false
25215 7927 Acked Write 3 Sent Done Done false
25216 7928 Acked Flush 0 Sent Done Done false
25217 7929 Acked Write 5 Sent Done Done false
25218 7930 Acked Read 10 Sent Done Done false
25219 7931 Acked Write 6 Sent Done Done false
25220 7932 Acked Write 1 Sent Done Done false
25221 7933 Acked Write 3 Sent Done Done false
25222 7934 Acked Write 3 Sent Done Done false
25223 7935 Acked Write 4 Sent Done Done false
25224 7936 Acked Write 6 Sent Done Done false
25225 7937 Acked Read 3 Sent Done Done false
25226 7938 Acked Write 8 Sent Done Done false
25227 7939 Acked Write 6 Sent Done Done false
25228 7940 Acked Read 1 Sent Done Done false
25229 7941 Acked Write 8 Sent Done Done false
25230 7942 Acked Read 5 Sent Done Done false
25231 STATES DS:0 DS:1 DS:2 TOTAL
25232 Sent 301 0 0 301
25233 Done 0 301 301 602
25234 Skipped 0 0 0 0
25235 Error 0 0 0 0
25236 Last Flush: Some(JobId(7641)) Some(JobId(7928)) Some(JobId(7928))
25237 Downstairs last five completed: 7641 7640 7639 7638 7637
25238 Upstairs last five completed: 7942 7941 7940 7939 7938
25239 CLIENT: Up:0 ds:301 act:2
252402025-01-28T21:59:44.066ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252412025-01-28T21:59:44.818ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252422025-01-28T21:59:46.109ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252432025-01-28T21:59:46.820ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252442025-01-28T21:59:47.331ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252452025-01-28T21:59:47.331ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52739) = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252462025-01-28T21:59:47.332ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252472025-01-28T21:59:47.333ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252482025-01-28T21:59:47.333ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252492025-01-28T21:59:47.333ZINFOcrucible: send last flush ID to this DS: Some(JobId(7641)) = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252502025-01-28T21:59:47.333ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252512025-01-28T21:59:47.333ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(7641)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252522025-01-28T21:59:47.333ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(7641)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
25253 ----------------------------------------------------------------
25254 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:14
25255 States: Active Active Active
25256 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25257 7929 Acked Write 5 Done Done Done true
25258 7930 Acked Read 10 Done Done Done true
25259 7931 Acked Write 6 Done Done Done true
25260 7932 Acked Write 1 Done Done Done true
25261 7933 Acked Write 3 Done Done Done true
25262 7934 Acked Write 3 Done Done Done true
25263 7935 Acked Write 4 Done Done Done true
25264 7936 Acked Write 6 Done Done Done true
25265 7937 Acked Read 3 Done Done Done true
25266 7938 Acked Write 8 Done Done Done true
25267 7939 Acked Write 6 Done Done Done true
25268 7940 Acked Read 1 Done Done Done true
25269 7941 Acked Write 8 Done Done Done true
25270 7942 Acked Read 5 Done Done Done true
25271 STATES DS:0 DS:1 DS:2 TOTAL
25272 Sent 0 0 0 0
25273 Done 14 14 14 42
25274 Skipped 0 0 0 0
25275 Error 0 0 0 0
25276 Last Flush: Some(JobId(7928)) Some(JobId(7928)) Some(JobId(7928))
25277 Downstairs last five completed: 7928 7927 7926 7925 7924
25278 Upstairs last five completed: 7942 7941 7940 7939 7938
25279 CLIENT: Up:0 ds:14 act:3
25280 ----------------------------------------------------------------
25281 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25282 Downstairs last five completed: 7943 7942 7941 7940 7939
25283 Upstairs last five completed: 7943 7942 7941 7940 7939
25284 CLIENT: Up:0 ds:0 act:3
25285 Replay: All jobs finished
25286 Main task has work to do, go find it
25287 got dsc Stop(2)
25288 stop 2
25289 Jan 28 21:59:51.568[8830] Got stop action so:false kr:false
25290 INFO request completed, latency_us: 371, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 40d45336-8aba-42b0-89d1-81dbb7a86e54, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
25291 Jan 28 21:59:51.569 INFO request completed, latency_us: 140, response_code: 200, uri: /state/cid/2, method: GET, req_id: eba24aab-e847-4a76-8d95-f1b9fc88bf09, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
252922025-01-28T21:59:51.569ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252932025-01-28T21:59:51.569ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252942025-01-28T21:59:51.569ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
252952025-01-28T21:59:51.569ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
25296 [8830] Exited with: ExitStatus(unix_wait_status(9))
25297 [8830][2] reports Exit
25298 Jan 28 21:59:55.570 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/2, method: GET, req_id: ab7955f2-bae2-404d-bd3a-11a2631fc7b2, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
25299 001/300 Read block 3407 len 20480
25300 002/300 Read block 2697 len 24576
25301 003/300 Write block 534 len 8192 data: 6 6
25302 004/300 Write block 3125 len 12288 data: 3 3 2
25303 005/300 Read block 2072 len 36864
25304 006/300 Read block 3353 len 20480
25305 007/300 Write block 3207 len 8192 data: 2 2
25306 008/300 Read block 3316 len 28672
25307 009/300 Read block 3141 len 4096
25308 010/300 Write block 1998 len 8192 data: 2 2
25309 011/300 Write block 3769 len 4096 data: 7
25310 012/300 Read block 234 len 12288
25311 013/300 Read block 1141 len 24576
25312 014/300 Write block 677 len 28672 data: 2 3 3 3 4 4 4
25313 015/300 Write block 4472 len 12288 data: 5 5 5
25314 016/300 Read block 3142 len 40960
25315 017/300 Read block 3037 len 40960
25316 018/300 Read block 4131 len 8192
25317 019/300 Read block 595 len 40960
25318 020/300 Read block 4130 len 28672
25319 021/300 Write block 4046 len 24576 data: 4 4 5 5 4 3
25320 022/300 Read block 3492 len 32768
25321 023/300 Write block 4285 len 20480 data: 2 3 3 4 4
25322 024/300 Write block 2399 len 8192 data: 5 5
25323 025/300 Read block 609 len 16384
25324 026/300 Read block 950 len 24576
25325 027/300 Read block 2749 len 24576
25326 028/300 Read block 4243 len 12288
25327 029/300 Read block 3472 len 40960
25328 030/300 Read block 4245 len 28672
25329 031/300 Read block 2371 len 32768
25330 032/300 Write block 1957 len 24576 data: 7 6 6 6 5 5
25331 033/300 Write block 323 len 16384 data: 6 6 5 4
25332 034/300 Read block 1106 len 40960
25333 035/300 Read block 4172 len 16384
25334 036/300 Read block 4168 len 12288
25335 037/300 Read block 1234 len 32768
25336 038/300 Read block 1841 len 20480
25337 039/300 Read block 4281 len 24576
25338 040/300 Write block 2961 len 28672 data: 3 3 3 3 4 4 6
25339 041/300 Write block 1725 len 28672 data: 3 2 2 2 2 2 3
25340 042/300 Read block 1640 len 32768
25341 043/300 Read block 561 len 28672
25342 044/300 Write block 970 len 12288 data: 6 3 4
25343 045/300 Read block 3793 len 24576
25344 046/300 Read block 1647 len 20480
25345 047/300 Write block 952 len 32768 data: 6 4 4 4 3 4 5 6
25346 048/300 Read block 4919 len 20480
25347 049/300 Write block 19 len 36864 data: 3 3 3 3 3 6 5 5 4
25348 050/300 Write block 1468 len 36864 data: 6 5 5 5 5 5 3 3 4
25349 051/300 Read block 260 len 28672
25350 052/300 Write block 685 len 32768 data: 5 4 6 8 8 7 8 6
25351 053/300 Read block 2980 len 4096
25352 054/300 Read block 4455 len 8192
25353 055/300 Read block 1099 len 40960
25354 056/300 Write block 2513 len 16384 data: 5 5 6 6
25355 057/300 Flush
25356 058/300 Write block 3050 len 24576 data: 9 6 6 6 6 6
25357 059/300 Read block 4115 len 4096
25358 060/300 Read block 1208 len 12288
25359 061/300 Read block 2075 len 20480
25360 062/300 Read block 825 len 24576
25361 063/300 Read block 819 len 16384
25362 064/300 Read block 2030 len 32768
25363 065/300 Write block 3987 len 40960 data: 4 4 4 4 4 4 5 5 3 4
25364 066/300 Read block 372 len 36864
25365 067/300 Write block 2373 len 4096 data: 7
25366 068/300 Read block 4794 len 8192
25367 069/300 Write block 3900 len 28672 data: 5 5 4 4 3 5 6
25368 070/300 Read block 4703 len 28672
25369 071/300 Flush
25370 072/300 Read block 2377 len 24576
25371 073/300 Read block 4072 len 24576
25372 074/300 Read block 4434 len 8192
25373 075/300 Read block 3591 len 36864
25374 076/300 Read block 2290 len 40960
25375 077/300 Flush
25376 078/300 Read block 1775 len 12288
25377 079/300 Read block 3435 len 40960
25378 080/300 Read block 3238 len 12288
25379 081/300 Read block 4891 len 32768
25380 082/300 Read block 4148 len 32768
25381 083/300 Flush
25382 084/300 Read block 785 len 12288
25383 085/300 Write block 1281 len 16384 data: 4 4 4 5
25384 086/300 Flush
25385 087/300 Read block 3025 len 8192
25386 088/300 Write block 3424 len 24576 data: 5 4 3 3 3 3
25387 089/300 Read block 1774 len 32768
25388 090/300 Read block 1091 len 24576
25389 091/300 Write block 1076 len 4096 data: 4
25390 092/300 Read block 1339 len 32768
25391 093/300 Read block 2050 len 28672
25392 094/300 Write block 4013 len 8192 data: 5 3
25393 095/300 Write block 4297 len 20480 data: 5 6 6 5 4
25394 096/300 Write block 609 len 8192 data: 7 7
25395 097/300 Write block 1578 len 4096 data: 6
25396 098/300 Read block 3935 len 40960
25397 099/300 Read block 4074 len 4096
25398 100/300 Write block 2779 len 20480 data: 4 3 2 2 2
25399 101/300 Write block 2385 len 4096 data: 8
25400 102/300 Write block 499 len 20480 data: 5 6 5 5 4
25401 103/300 Write block 348 len 36864 data: 7 9 8 7 4 5 5 5 7
25402 104/300 Write block 2549 len 24576 data: 7 6 6 6 5 5
25403 105/300 Read block 3645 len 20480
25404 106/300 Read block 927 len 20480
25405 107/300 Write block 4427 len 28672 data: 8 8 6 6 5 4 4
25406 108/300 Flush
25407 109/300 Write block 361 len 32768 data: 7 6 5 4 4 5 5 5
25408 110/300 Read block 401 len 16384
25409 111/300 Read block 3605 len 32768
25410 112/300 Read block 3168 len 20480
25411 113/300 Read block 1129 len 40960
25412 114/300 Write block 478 len 36864 data: 7 6 6 5 6 6 4 4 4
25413 115/300 Read block 1082 len 32768
25414 116/300 Read block 3531 len 4096
25415 117/300 Flush
25416 118/300 Write block 2631 len 16384 data: 6 6 8 7
25417 119/300 Write block 3168 len 40960 data: 4 4 4 3 3 3 3 3 4 3
25418 120/300 Read block 1229 len 40960
25419 121/300 Flush
25420 122/300 Read block 4472 len 24576
25421 123/300 Write block 2569 len 36864 data: 3 3 4 3 3 3 4 3 3
25422 124/300 Flush
25423 125/300 Write block 3481 len 32768 data: 4 5 5 6 5 6 5 5
25424 126/300 Write block 4111 len 32768 data: 5 5 6 6 9 7 6 7
25425 127/300 Write block 1999 len 8192 data: 3 3
25426 128/300 Read block 3006 len 40960
25427 129/300 Read block 2641 len 32768
25428 130/300 Read block 2135 len 16384
25429 131/300 Write block 393 len 28672 data: 3 3 4 5 7 7 8
25430 132/300 Read block 3052 len 28672
25431 133/300 Read block 1324 len 8192
25432 134/300 Write block 555 len 16384 data: 4 4 3 3
25433 135/300 Flush
25434 136/300 Read block 1322 len 40960
25435 137/300 Read block 1775 len 12288
25436 138/300 Write block 3865 len 24576 data: 4 4 3 3 3 5
25437 139/300 Write block 53 len 12288 data: 4 4 5
25438 140/300 Read block 1426 len 24576
25439 141/300 Read block 208 len 16384
25440 142/300 Read block 318 len 32768
25441 143/300 Write block 2946 len 4096 data: 7
25442 144/300 Read block 1382 len 8192
25443 145/300 Write block 868 len 8192 data: 4 6
25444 146/300 Write block 510 len 24576 data: 2 4 4 3 4 3
25445 147/300 Flush
25446 148/300 Write block 3338 len 20480 data: 6 5 5 5 6
25447 149/300 Write block 3661 len 4096 data: 7
25448 150/300 Read block 1506 len 8192
25449 151/300 Write block 3248 len 40960 data: 5 6 7 8 7 7 8 9 7 6
25450 152/300 Read block 4461 len 36864
25451 153/300 Read block 3582 len 8192
25452 154/300 Write block 1730 len 8192 data: 3 4
25453 155/300 Read block 4309 len 28672
25454 156/300 Write block 3759 len 16384 data: 7 8 6 6
25455 157/300 Read block 4415 len 40960
25456 158/300 Write block 3069 len 4096 data: 6
25457 159/300 Write block 835 len 32768 data: 3 3 3 3 4 4 4 5
25458 160/300 Write block 518 len 40960 data: 4 3 3 5 5 6 6 4 4 5
25459 161/300 Write block 840 len 16384 data: 5 5 6 3
25460 162/300 Write block 617 len 4096 data: 4
25461 163/300 Write block 1775 len 32768 data: 8 6 6 5 4 4 4 5
25462 164/300 Read block 2340 len 12288
25463 165/300 Read block 1092 len 12288
25464 166/300 Read block 2941 len 32768
25465 167/300 Write block 4634 len 24576 data: 6 6 5 5 4 5
25466 168/300 Read block 978 len 32768
25467 169/300 Write block 2224 len 4096 data: 4
25468 170/300 Read block 1325 len 16384
25469 171/300 Read block 745 len 24576
25470 172/300 Write block 4909 len 28672 data: 3 3 3 3 3 3 4
25471 173/300 Read block 3789 len 40960
25472 174/300 Read block 4728 len 24576
25473 175/300 Read block 1188 len 16384
25474 176/300 Read block 3380 len 4096
25475 177/300 Write block 59 len 40960 data: 6 6 7 8 7 9 7 7 6 4
25476 178/300 Read block 3236 len 32768
25477 179/300 Read block 1956 len 4096
25478 180/300 Write block 833 len 12288 data: 3 4 4
25479 181/300 Flush
25480 182/300 Write block 1747 len 28672 data: 7 6 6 5 4 3 3
25481 183/300 Flush
25482 184/300 Read block 1232 len 16384
25483 185/300 Read block 658 len 24576
25484 186/300 Read block 2681 len 8192
25485 187/300 Read block 1246 len 24576
25486 188/300 Read block 348 len 12288
25487 189/300 Write block 1027 len 40960 data: 6 5 7 5 4 4 6 5 4 5
25488 190/300 Flush
25489 191/300 Read block 2574 len 36864
25490 192/300 Read block 2764 len 24576
25491 193/300 Flush
25492 194/300 Read block 2314 len 24576
25493 195/300 Read block 4440 len 28672
25494 196/300 Read block 4410 len 20480
25495 197/300 Write block 3297 len 12288 data: 6 6 6
25496 198/300 Read block 4165 len 40960
25497 199/300 Read block 2101 len 24576
25498 200/300 Read block 616 len 8192
25499 201/300 Read block 3245 len 12288
25500 202/300 Read block 2316 len 36864
25501 203/300 Write block 80 len 20480 data: 7 7 5 4 3
25502 204/300 Write block 2011 len 4096 data: 6
25503 205/300 Write block 3270 len 4096 data: 4
25504 206/300 Write block 3144 len 16384 data: 4 4 4 4
25505 207/300 Write block 2335 len 16384 data: 3 3 5 5
25506 208/300 Write block 2932 len 28672 data: 4 4 5 5 4 6 7
25507 209/300 Write block 2220 len 20480 data: 4 4 4 4 5
25508 210/300 Write block 363 len 12288 data: 6 5 5
25509 211/300 Write block 2032 len 36864 data: 2 3 3 2 3 3 4 5 5
25510 212/300 Read block 1989 len 20480
25511 213/300 Write block 2097 len 36864 data: 6 7 8 5 5 6 7 5 5
25512 214/300 Write block 1672 len 28672 data: 6 6 5 5 6 6 5
25513 215/300 Write block 4643 len 16384 data: 4 4 5 5
25514 216/300 Write block 844 len 8192 data: 3 3
25515 217/300 Write block 2002 len 20480 data: 3 4 4 5 5
25516 218/300 Write block 3476 len 4096 data: 5
25517 219/300 Read block 2109 len 16384
25518 220/300 Read block 3163 len 12288
25519 221/300 Write block 4057 len 12288 data: 5 4 5
25520 222/300 Flush
25521 223/300 Write block 3322 len 28672 data: 6 5 7 7 6 5 8
25522 224/300 Read block 1440 len 4096
25523 225/300 Read block 4467 len 16384
25524 226/300 Read block 644 len 4096
25525 227/300 Write block 142 len 16384 data: 4 4 4 4
25526 228/300 Read block 4119 len 36864
25527 229/300 Read block 1318 len 16384
25528 230/300 Write block 2848 len 8192 data: 8 6
25529 231/300 Write block 4581 len 32768 data: 7 7 5 2 3 2 3 4
25530 232/300 Read block 58 len 20480
25531 233/300 Write block 203 len 12288 data: 6 6 7
25532 234/300 Write block 2631 len 28672 data: 7 7 9 8 8 8 6
25533 235/300 Read block 4428 len 32768
25534 236/300 Write block 1344 len 8192 data: 8 7
25535 237/300 Flush
25536 238/300 Write block 2397 len 28672 data: 7 7 6 6 4 5 6
25537 239/300 Read block 4206 len 20480
25538 240/300 Write block 867 len 16384 data: 3 5 7 6
25539 241/300 Write block 3358 len 36864 data: 6 6 7 7 7 5 4 5 7
25540 242/300 Read block 4222 len 8192
25541 243/300 Read block 1439 len 28672
25542 244/300 Read block 2637 len 36864
25543 245/300 Write block 3430 len 12288 data: 4 5 5
25544 246/300 Flush
25545 247/300 Read block 3673 len 12288
25546 248/300 Read block 1249 len 8192
25547 249/300 Write block 1363 len 20480 data: 6 7 5 4 4
25548 250/300 Write block 1300 len 20480 data: 8 7 6 6 7
25549 251/300 Write block 2370 len 16384 data: 8 8 7 8
25550 252/300 Write block 2425 len 32768 data: 5 5 5 6 8 8 7 6
25551 253/300 Write block 1266 len 32768 data: 3 4 5 5 5 5 5 5
25552 254/300 Read block 4595 len 40960
25553 255/300 Read block 4920 len 20480
25554 256/300 Read block 1031 len 16384
25555 257/300 Read block 3389 len 28672
25556 258/300 Write block 3608 len 4096 data: 7
25557 259/300 Write block 365 len 8192 data: 6 6
25558 260/300 Read block 2043 len 32768
25559 261/300 Flush
25560 262/300 Write block 3846 len 16384 data: 3 3 2 2
25561 263/300 Read block 4389 len 20480
25562 264/300 Read block 3174 len 32768
25563 265/300 Read block 4785 len 40960
25564 266/300 Write block 4479 len 36864 data: 9 11 11 11 11 11 10 12 9
25565 267/300 Write block 2278 len 32768 data: 7 5 3 4 4 6 6 7
25566 268/300 Write block 2821 len 4096 data: 6
25567 269/300 Read block 3779 len 40960
25568 270/300 Write block 2358 len 32768 data: 2 2 3 3 3 3 3 3
25569 271/300 Read block 2053 len 12288
25570 272/300 Write block 1828 len 16384 data: 7 7 6 5
25571 273/300 Flush
25572 274/300 Read block 1270 len 28672
25573 275/300 Read block 2636 len 4096
25574 276/300 Read block 478 len 8192
25575 277/300 Flush
25576 278/300 Read block 2261 len 24576
25577 279/300 Read block 4454 len 28672
25578 280/300 Read block 4614 len 20480
25579 281/300 Read block 804 len 8192
25580 282/300 Write block 3916 len 16384 data: 3 3 4 4
25581 283/300 Read block 608 len 8192
25582 284/300 Read block 4354 len 24576
25583 285/300 Read block 3029 len 12288
25584 286/300 Read block 2404 len 20480
25585 287/300 Read block 3282 len 4096
25586 288/300 Write block 2521 len 20480 data: 5 6 6 5 3
25587 289/300 Read block 2944 len 4096
25588 290/300 Read block 3049 len 40960
25589 291/300 Read block 4804 len 4096
25590 292/300 Read block 1079 len 4096
25591 293/300 Write block 4535 len 28672 data: 4 4 4 3 4 6 6
25592 294/300 Read block 3909 len 40960
25593 295/300 Write block 4510 len 28672 data: 6 6 4 5 5 5 4
25594 296/300 Read block 1186 len 4096
25595 297/300 Write block 3667 len 32768 data: 5 8 7 7 7 7 7 6
25596 298/300 Read block 1777 len 12288
25597 299/300 Write block 762 len 12288 data: 5 5 6
25598 300/300 Write block 1309 len 12288 data: 6 6 5
25599 301/300 Read block 1227 len 40960
25600 Main task has work to do, go find it
25601 got dsc Start(2)
25602 start 2
25603 Jan 28 21:59:55.816 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/2, method: POST, req_id: 7f55f855-f4e2-4a50-ac9a-95c94b8d2b11, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
25604 Replay: started 2, returned:Ok(())
25605 ----------------------------------------------------------------
25606 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25607 States: Active Active Offline
25608 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25609 7944 Acked Read 5 Done Done Sent false
25610 7945 Acked Read 6 Done Done Sent false
25611 7946 Acked Write 2 Done Done Sent false
25612 7947 Acked Write 3 Done Done Sent false
25613 7948 Acked Read 9 Done Done Sent false
25614 7949 Acked Read 5 Done Done Sent false
25615 7950 Acked Write 2 Done Done Sent false
25616 7951 Acked Read 7 Done Done Sent false
25617 7952 Acked Read 1 Done Done Sent false
25618 7953 Acked Write 2 Done Done Sent false
25619 7954 Acked Write 1 Done Done Sent false
25620 7955 Acked Read 3 Done Done Sent false
25621 7956 Acked Read 6 Done Done Sent false
25622 7957 Acked Write 7 Done Done Sent false
25623 7958 Acked Write 3 Done Done Sent false
25624 7959 Acked Read 10 Done Done Sent false
25625 7960 Acked Read 10 Done Done Sent false
25626 7961 Acked Read 2 Done Done Sent false
25627 7962 Acked Read 10 Done Done Sent false
25628 7963 Acked Read 7 Done Done Sent false
25629 7964 Acked Write 6 Done Done Sent false
25630 7965 Acked Read 8 Done Done Sent false
25631 7966 Acked Write 5 Done Done Sent false
25632 7967 Acked Write 2 Done Done Sent false
25633 7968 Acked Read 4 Done Done Sent false
25634 7969 Acked Read 6 Done Done Sent false
25635 7970 Acked Read 6 Done Done Sent false
25636 7971 Acked Read 3 Done Done Sent false
25637 7972 Acked Read 10 Done Done Sent false
25638 7973 Acked Read 7 Done Done Sent false
25639 7974 Acked Read 8 Done Done Sent false
25640 7975 Acked Write 6 Done Done Sent false
25641 7976 Acked Write 4 Done Done Sent false
25642 7977 Acked Read 10 Done Done Sent false
25643 7978 Acked Read 4 Done Done Sent false
25644 7979 Acked Read 3 Done Done Sent false
25645 7980 Acked Read 8 Done Done Sent false
25646 7981 Acked Read 5 Done Done Sent false
25647 7982 Acked Read 6 Done Done Sent false
25648 7983 Acked Write 7 Done Done Sent false
25649 7984 Acked Write 7 Done Done Sent false
25650 7985 Acked Read 8 Done Done Sent false
25651 7986 Acked Read 7 Done Done Sent false
25652 7987 Acked Write 3 Done Done Sent false
25653 7988 Acked Read 6 Done Done Sent false
25654 7989 Acked Read 5 Done Done Sent false
25655 7990 Acked Write 8 Done Done Sent false
25656 7991 Acked Read 5 Done Done Sent false
25657 7992 Acked Write 9 Done Done Sent false
25658 7993 Acked Write 9 Done Done Sent false
25659 7994 Acked Read 7 Done Done Sent false
25660 7995 Acked Write 8 Done Done Sent false
25661 7996 Acked Read 1 Done Done Sent false
25662 7997 Acked Read 2 Done Done Sent false
25663 7998 Acked Read 10 Done Done Sent false
25664 7999 Acked Write 4 Done Done Sent false
25665 8000 Acked Flush 0 Done Done Sent false
25666 8001 Acked Write 6 Done Done Sent false
25667 8002 Acked Read 1 Done Done Sent false
25668 8003 Acked Read 3 Done Done Sent false
25669 8004 Acked Read 5 Done Done Sent false
25670 8005 Acked Read 6 Done Done Sent false
25671 8006 Acked Read 4 Done Done Sent false
25672 8007 Acked Read 8 Done Done Sent false
25673 8008 Acked Write 10 Done Done Sent false
25674 8009 Acked Read 9 Done Done Sent false
25675 8010 Acked Write 1 Done Done Sent false
25676 8011 Acked Read 2 Done Done Sent false
25677 8012 Acked Write 7 Done Done Sent false
25678 8013 Acked Read 7 Done Done Sent false
25679 8014 Acked Flush 0 Done Done Sent false
25680 8015 Acked Read 6 Done Done Sent false
25681 8016 Acked Read 6 Done Done Sent false
25682 8017 Acked Read 2 Done Done Sent false
25683 8018 Acked Read 9 Done Done Sent false
25684 8019 Acked Read 10 Done Done Sent false
25685 8020 Acked Flush 0 Done Done Sent false
25686 8021 Acked Read 3 Done Done Sent false
25687 8022 Acked Read 10 Done Done Sent false
25688 8023 Acked Read 3 Done Done Sent false
25689 8024 Acked Read 8 Done Done Sent false
25690 8025 Acked Read 8 Done Done Sent false
25691 8026 Acked Flush 0 Done Done Sent false
25692 8027 Acked Read 3 Done Done Sent false
25693 8028 Acked Write 4 Done Done Sent false
25694 8029 Acked Flush 0 Done Done Sent false
25695 8030 Acked Read 2 Done Done Sent false
25696 8031 Acked Write 6 Done Done Sent false
25697 8032 Acked Read 8 Done Done Sent false
25698 8033 Acked Read 6 Done Done Sent false
25699 8034 Acked Write 1 Done Done Sent false
25700 8035 Acked Read 8 Done Done Sent false
25701 8036 Acked Read 7 Done Done Sent false
25702 8037 Acked Write 2 Done Done Sent false
25703 8038 Acked Write 5 Done Done Sent false
25704 8039 Acked Write 2 Done Done Sent false
25705 8040 Acked Write 1 Done Done Sent false
25706 8041 Acked Read 10 Done Done Sent false
25707 8042 Acked Read 1 Done Done Sent false
25708 8043 Acked Write 5 Done Done Sent false
25709 8044 Acked Write 1 Done Done Sent false
25710 8045 Acked Write 5 Done Done Sent false
25711 8046 Acked Write 9 Done Done Sent false
25712 8047 Acked Write 6 Done Done Sent false
25713 8048 Acked Read 5 Done Done Sent false
25714 8049 Acked Read 5 Done Done Sent false
25715 8050 Acked Write 7 Done Done Sent false
25716 8051 Acked Flush 0 Done Done Sent false
25717 8052 Acked Write 8 Done Done Sent false
25718 8053 Acked Read 4 Done Done Sent false
25719 8054 Acked Read 8 Done Done Sent false
25720 8055 Acked Read 5 Done Done Sent false
25721 8056 Acked Read 10 Done Done Sent false
25722 8057 Acked Write 9 Done Done Sent false
25723 8058 Acked Read 8 Done Done Sent false
25724 8059 Acked Read 1 Done Done Sent false
25725 8060 Acked Flush 0 Done Done Sent false
25726 8061 Acked Write 4 Done Done Sent false
25727 8062 Acked Write 10 Done Done Sent false
25728 8063 Acked Read 10 Done Done Sent false
25729 8064 Acked Flush 0 Done Done Sent false
25730 8065 Acked Read 6 Done Done Sent false
25731 8066 Acked Write 9 Done Done Sent false
25732 8067 Acked Flush 0 Done Done Sent false
25733 8068 Acked Write 8 Done Done Sent false
25734 8069 Acked Write 8 Done Done Sent false
25735 8070 Acked Write 2 Done Done Sent false
25736 8071 Acked Read 10 Done Done Sent false
25737 8072 Acked Read 8 Done Done Sent false
25738 8073 Acked Read 4 Done Done Sent false
25739 8074 Acked Write 7 Done Done Sent false
25740 8075 Acked Read 7 Done Done Sent false
25741 8076 Acked Read 2 Done Done Sent false
25742 8077 Acked Write 4 Done Done Sent false
25743 8078 Acked Flush 0 Done Done Sent false
25744 8079 Acked Read 10 Done Done Sent false
25745 8080 Acked Read 3 Done Done Sent false
25746 8081 Acked Write 6 Done Done Sent false
25747 8082 Acked Write 3 Done Done Sent false
25748 8083 Acked Read 6 Done Done Sent false
25749 8084 Acked Read 4 Done Done Sent false
25750 8085 Acked Read 8 Done Done Sent false
25751 8086 Acked Write 1 Done Done Sent false
25752 8087 Acked Read 2 Done Done Sent false
25753 8088 Acked Write 2 Done Done Sent false
25754 8089 Acked Write 6 Done Done Sent false
25755 8090 Acked Flush 0 Done Done Sent false
25756 8091 Acked Write 5 Done Done Sent false
25757 8092 Acked Write 1 Done Done Sent false
25758 8093 Acked Read 2 Done Done Sent false
25759 8094 Acked Write 10 Done Done Sent false
25760 8095 Acked Read 9 Done Done Sent false
25761 8096 Acked Read 2 Done Done Sent false
25762 8097 Acked Write 2 Done Done Sent false
25763 8098 Acked Read 7 Done Done Sent false
25764 8099 Acked Write 4 Done Done Sent false
25765 8100 Acked Read 10 Done Done Sent false
25766 8101 Acked Write 1 Done Done Sent false
25767 8102 Acked Write 8 Done Done Sent false
25768 8103 Acked Write 10 Done Done Sent false
25769 8104 Acked Write 4 Done Done Sent false
25770 8105 Acked Write 1 Done Done Sent false
25771 8106 Acked Write 8 Done Done Sent false
25772 8107 Acked Read 3 Done Done Sent false
25773 8108 Acked Read 3 Done Done Sent false
25774 8109 Acked Read 8 Done Done Sent false
25775 8110 Acked Write 6 Done Done Sent false
25776 8111 Acked Read 8 Done Done Sent false
25777 8112 Acked Write 1 Done Done Sent false
25778 8113 Acked Read 4 Done Done Sent false
25779 8114 Acked Read 6 Done Done Sent false
25780 8115 Acked Write 7 Done Done Sent false
25781 8116 Acked Read 10 Done Done Sent false
25782 8117 Acked Read 6 Done Done Sent false
25783 8118 Acked Read 4 Done Done Sent false
25784 8119 Acked Read 1 Done Done Sent false
25785 8120 Acked Write 10 Done Done Sent false
25786 8121 Acked Read 8 Done Done Sent false
25787 8122 Acked Read 1 Done Done Sent false
25788 8123 Acked Write 3 Done Done Sent false
25789 8124 Acked Flush 0 Done Done Sent false
25790 8125 Acked Write 7 Done Done Sent false
25791 8126 Acked Flush 0 Done Done Sent false
25792 8127 Acked Read 4 Done Done Sent false
25793 8128 Acked Read 6 Done Done Sent false
25794 8129 Acked Read 2 Done Done Sent false
25795 8130 Acked Read 6 Done Done Sent false
25796 8131 Acked Read 3 Done Done Sent false
25797 8132 Acked Write 10 Done Done Sent false
25798 8133 Acked Flush 0 Done Done Sent false
25799 8134 Acked Read 9 Done Done Sent false
25800 8135 Acked Read 6 Done Done Sent false
25801 8136 Acked Flush 0 Done Done Sent false
25802 8137 Acked Read 6 Done Done Sent false
25803 8138 Acked Read 7 Done Done Sent false
25804 8139 Acked Read 5 Done Done Sent false
25805 8140 Acked Write 3 Done Done Sent false
25806 8141 Acked Read 10 Done Done Sent false
25807 8142 Acked Read 6 Done Done Sent false
25808 8143 Acked Read 2 Done Done Sent false
25809 8144 Acked Read 3 Done Done Sent false
25810 8145 Acked Read 9 Done Done Sent false
25811 8146 Acked Write 5 Done Done Sent false
25812 8147 Acked Write 1 Done Done Sent false
25813 8148 Acked Write 1 Done Done Sent false
25814 8149 Acked Write 4 Done Done Sent false
25815 8150 Acked Write 4 Done Done Sent false
25816 8151 Acked Write 7 Done Done Sent false
25817 8152 Acked Write 5 Done Done Sent false
25818 8153 Acked Write 3 Done Done Sent false
25819 8154 Acked Write 9 Done Done Sent false
25820 8155 Acked Read 5 Done Done Sent false
25821 8156 Acked Write 9 Done Done Sent false
25822 8157 Acked Write 7 Done Done Sent false
25823 8158 Acked Write 4 Done Done Sent false
25824 8159 Acked Write 2 Done Done Sent false
25825 8160 Acked Write 5 Done Done Sent false
25826 8161 Acked Write 1 Done Done Sent false
25827 8162 Acked Read 4 Done Done Sent false
25828 8163 Acked Read 3 Done Done Sent false
25829 8164 Acked Write 3 Done Done Sent false
25830 8165 Acked Flush 0 Done Done Sent false
25831 8166 Acked Write 7 Done Done Sent false
25832 8167 Acked Read 1 Done Done Sent false
25833 8168 Acked Read 4 Done Done Sent false
25834 8169 Acked Read 1 Done Done Sent false
25835 8170 Acked Write 4 Done Done Sent false
25836 8171 Acked Read 9 Done Done Sent false
25837 8172 Acked Read 4 Done Done Sent false
25838 8173 Acked Write 2 Done Done Sent false
25839 8174 Acked Write 8 Done Done Sent false
25840 8175 Acked Read 5 Done Done Sent false
25841 8176 Acked Write 3 Done Done Sent false
25842 8177 Acked Write 7 Done Done Sent false
25843 8178 Acked Read 8 Done Done Sent false
25844 8179 Acked Write 2 Done Done Sent false
25845 8180 Acked Flush 0 Done Done Sent false
25846 8181 Acked Write 7 Done Done Sent false
25847 8182 Acked Read 5 Done Done Sent false
25848 8183 Acked Write 4 Done Done Sent false
25849 8184 Acked Write 9 Done Done Sent false
25850 8185 Acked Read 2 Done Done Sent false
25851 8186 Acked Read 7 Done Done Sent false
25852 8187 Acked Read 9 Done Done Sent false
25853 8188 Acked Write 3 Done Done Sent false
25854 8189 Acked Flush 0 Done Done Sent false
25855 8190 Acked Read 3 Done Done Sent false
25856 8191 Acked Read 2 Done Done Sent false
25857 8192 Acked Write 5 Done Done Sent false
25858 8193 Acked Write 5 Done Done Sent false
25859 8194 Acked Write 4 Done Done Sent false
25860 8195 Acked Write 8 Done Done Sent false
25861 8196 Acked Write 8 Done Done Sent false
25862 8197 Acked Read 10 Done Done Sent false
25863 8198 Acked Read 5 Done Done Sent false
25864 8199 Acked Read 4 Done Done Sent false
25865 8200 Acked Read 7 Done Done Sent false
25866 8201 Acked Write 1 Done Done Sent false
25867 8202 Acked Write 2 Done Done Sent false
25868 8203 Acked Read 8 Done Done Sent false
25869 8204 Acked Flush 0 Done Done Sent false
25870 8205 Acked Write 4 Done Done Sent false
25871 8206 Acked Read 5 Done Done Sent false
25872 8207 Acked Read 8 Done Done Sent false
25873 8208 Acked Read 10 Done Done Sent false
25874 8209 Acked Write 9 Done Done Sent false
25875 8210 Acked Write 8 Done Done Sent false
25876 8211 Acked Write 1 Done Done Sent false
25877 8212 Acked Read 10 Done Done Sent false
25878 8213 Acked Write 8 Done Done Sent false
25879 8214 Acked Read 3 Done Done Sent false
25880 8215 Acked Write 4 Done Done Sent false
25881 8216 Acked Flush 0 Done Done Sent false
25882 8217 Acked Read 7 Done Done Sent false
25883 8218 Acked Read 1 Done Done Sent false
25884 8219 Acked Read 2 Done Done Sent false
25885 8220 Acked Flush 0 Done Done Sent false
25886 8221 Acked Read 6 Done Done Sent false
25887 8222 Acked Read 7 Done Done Sent false
25888 8223 Acked Read 5 Done Done Sent false
25889 8224 Acked Read 2 Done Done Sent false
25890 8225 Acked Write 4 Done Done Sent false
25891 8226 Acked Read 2 Done Done Sent false
25892 8227 Acked Read 6 Done Done Sent false
25893 8228 Acked Read 3 Done Done Sent false
25894 8229 Acked Read 5 Done Done Sent false
25895 8230 Acked Read 1 Done Done Sent false
25896 8231 Acked Write 5 Done Done Sent false
25897 8232 Acked Read 1 Done Done Sent false
25898 8233 Acked Read 10 Done Done Sent false
25899 8234 Acked Read 1 Done Done Sent false
25900 8235 Acked Read 1 Done Done Sent false
25901 8236 Acked Write 7 Done Done Sent false
25902 8237 Acked Read 10 Done Done Sent false
25903 8238 Acked Write 7 Done Done Sent false
25904 8239 Acked Read 1 Done Done Sent false
25905 8240 Acked Write 8 Done Done Sent false
25906 8241 Acked Read 3 Done Done Sent false
25907 8242 Acked Write 3 Done Done Sent false
25908 8243 Acked Write 3 Done Done Sent false
25909 8244 Acked Read 10 Done Done Sent false
25910 STATES DS:0 DS:1 DS:2 TOTAL
25911 Sent 0 0 301 301
25912 Done 301 301 0 602
25913 Skipped 0 0 0 0
25914 Error 0 0 0 0
25915 Last Flush: Some(JobId(8220)) Some(JobId(8220)) Some(JobId(7943))
25916 Downstairs last five completed: 7943 7942 7941 7940 7939
25917 Upstairs last five completed: 8244 8243 8242 8241 8240
25918 CLIENT: Up:0 ds:301 act:2
259192025-01-28T21:59:56.318ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
259202025-01-28T21:59:56.835ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
259212025-01-28T21:59:57.836ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
259222025-01-28T21:59:58.837ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
25923 ----------------------------------------------------------------
25924 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25925 States: Active Active Offline
25926 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25927 7944 Acked Read 5 Done Done Sent false
25928 7945 Acked Read 6 Done Done Sent false
25929 7946 Acked Write 2 Done Done Sent false
25930 7947 Acked Write 3 Done Done Sent false
25931 7948 Acked Read 9 Done Done Sent false
25932 7949 Acked Read 5 Done Done Sent false
25933 7950 Acked Write 2 Done Done Sent false
25934 7951 Acked Read 7 Done Done Sent false
25935 7952 Acked Read 1 Done Done Sent false
25936 7953 Acked Write 2 Done Done Sent false
25937 7954 Acked Write 1 Done Done Sent false
25938 7955 Acked Read 3 Done Done Sent false
25939 7956 Acked Read 6 Done Done Sent false
25940 7957 Acked Write 7 Done Done Sent false
25941 7958 Acked Write 3 Done Done Sent false
25942 7959 Acked Read 10 Done Done Sent false
25943 7960 Acked Read 10 Done Done Sent false
25944 7961 Acked Read 2 Done Done Sent false
25945 7962 Acked Read 10 Done Done Sent false
25946 7963 Acked Read 7 Done Done Sent false
25947 7964 Acked Write 6 Done Done Sent false
25948 7965 Acked Read 8 Done Done Sent false
25949 7966 Acked Write 5 Done Done Sent false
25950 7967 Acked Write 2 Done Done Sent false
25951 7968 Acked Read 4 Done Done Sent false
25952 7969 Acked Read 6 Done Done Sent false
25953 7970 Acked Read 6 Done Done Sent false
25954 7971 Acked Read 3 Done Done Sent false
25955 7972 Acked Read 10 Done Done Sent false
25956 7973 Acked Read 7 Done Done Sent false
25957 7974 Acked Read 8 Done Done Sent false
25958 7975 Acked Write 6 Done Done Sent false
25959 7976 Acked Write 4 Done Done Sent false
25960 7977 Acked Read 10 Done Done Sent false
25961 7978 Acked Read 4 Done Done Sent false
25962 7979 Acked Read 3 Done Done Sent false
25963 7980 Acked Read 8 Done Done Sent false
25964 7981 Acked Read 5 Done Done Sent false
25965 7982 Acked Read 6 Done Done Sent false
25966 7983 Acked Write 7 Done Done Sent false
25967 7984 Acked Write 7 Done Done Sent false
25968 7985 Acked Read 8 Done Done Sent false
25969 7986 Acked Read 7 Done Done Sent false
25970 7987 Acked Write 3 Done Done Sent false
25971 7988 Acked Read 6 Done Done Sent false
25972 7989 Acked Read 5 Done Done Sent false
25973 7990 Acked Write 8 Done Done Sent false
25974 7991 Acked Read 5 Done Done Sent false
25975 7992 Acked Write 9 Done Done Sent false
25976 7993 Acked Write 9 Done Done Sent false
25977 7994 Acked Read 7 Done Done Sent false
25978 7995 Acked Write 8 Done Done Sent false
25979 7996 Acked Read 1 Done Done Sent false
25980 7997 Acked Read 2 Done Done Sent false
25981 7998 Acked Read 10 Done Done Sent false
25982 7999 Acked Write 4 Done Done Sent false
25983 8000 Acked Flush 0 Done Done Sent false
25984 8001 Acked Write 6 Done Done Sent false
25985 8002 Acked Read 1 Done Done Sent false
25986 8003 Acked Read 3 Done Done Sent false
25987 8004 Acked Read 5 Done Done Sent false
25988 8005 Acked Read 6 Done Done Sent false
25989 8006 Acked Read 4 Done Done Sent false
25990 8007 Acked Read 8 Done Done Sent false
25991 8008 Acked Write 10 Done Done Sent false
25992 8009 Acked Read 9 Done Done Sent false
25993 8010 Acked Write 1 Done Done Sent false
25994 8011 Acked Read 2 Done Done Sent false
25995 8012 Acked Write 7 Done Done Sent false
25996 8013 Acked Read 7 Done Done Sent false
25997 8014 Acked Flush 0 Done Done Sent false
25998 8015 Acked Read 6 Done Done Sent false
25999 8016 Acked Read 6 Done Done Sent false
26000 8017 Acked Read 2 Done Done Sent false
26001 8018 Acked Read 9 Done Done Sent false
26002 8019 Acked Read 10 Done Done Sent false
26003 8020 Acked Flush 0 Done Done Sent false
26004 8021 Acked Read 3 Done Done Sent false
26005 8022 Acked Read 10 Done Done Sent false
26006 8023 Acked Read 3 Done Done Sent false
26007 8024 Acked Read 8 Done Done Sent false
26008 8025 Acked Read 8 Done Done Sent false
26009 8026 Acked Flush 0 Done Done Sent false
26010 8027 Acked Read 3 Done Done Sent false
26011 8028 Acked Write 4 Done Done Sent false
26012 8029 Acked Flush 0 Done Done Sent false
26013 8030 Acked Read 2 Done Done Sent false
26014 8031 Acked Write 6 Done Done Sent false
26015 8032 Acked Read 8 Done Done Sent false
26016 8033 Acked Read 6 Done Done Sent false
26017 8034 Acked Write 1 Done Done Sent false
26018 8035 Acked Read 8 Done Done Sent false
26019 8036 Acked Read 7 Done Done Sent false
26020 8037 Acked Write 2 Done Done Sent false
26021 8038 Acked Write 5 Done Done Sent false
26022 8039 Acked Write 2 Done Done Sent false
26023 8040 Acked Write 1 Done Done Sent false
26024 8041 Acked Read 10 Done Done Sent false
26025 8042 Acked Read 1 Done Done Sent false
26026 8043 Acked Write 5 Done Done Sent false
26027 8044 Acked Write 1 Done Done Sent false
26028 8045 Acked Write 5 Done Done Sent false
26029 8046 Acked Write 9 Done Done Sent false
26030 8047 Acked Write 6 Done Done Sent false
26031 8048 Acked Read 5 Done Done Sent false
26032 8049 Acked Read 5 Done Done Sent false
26033 8050 Acked Write 7 Done Done Sent false
26034 8051 Acked Flush 0 Done Done Sent false
26035 8052 Acked Write 8 Done Done Sent false
26036 8053 Acked Read 4 Done Done Sent false
26037 8054 Acked Read 8 Done Done Sent false
26038 8055 Acked Read 5 Done Done Sent false
26039 8056 Acked Read 10 Done Done Sent false
26040 8057 Acked Write 9 Done Done Sent false
26041 8058 Acked Read 8 Done Done Sent false
26042 8059 Acked Read 1 Done Done Sent false
26043 8060 Acked Flush 0 Done Done Sent false
26044 8061 Acked Write 4 Done Done Sent false
26045 8062 Acked Write 10 Done Done Sent false
26046 8063 Acked Read 10 Done Done Sent false
26047 8064 Acked Flush 0 Done Done Sent false
26048 8065 Acked Read 6 Done Done Sent false
26049 8066 Acked Write 9 Done Done Sent false
26050 8067 Acked Flush 0 Done Done Sent false
26051 8068 Acked Write 8 Done Done Sent false
26052 8069 Acked Write 8 Done Done Sent false
26053 8070 Acked Write 2 Done Done Sent false
26054 8071 Acked Read 10 Done Done Sent false
26055 8072 Acked Read 8 Done Done Sent false
26056 8073 Acked Read 4 Done Done Sent false
26057 8074 Acked Write 7 Done Done Sent false
26058 8075 Acked Read 7 Done Done Sent false
26059 8076 Acked Read 2 Done Done Sent false
26060 8077 Acked Write 4 Done Done Sent false
26061 8078 Acked Flush 0 Done Done Sent false
26062 8079 Acked Read 10 Done Done Sent false
26063 8080 Acked Read 3 Done Done Sent false
26064 8081 Acked Write 6 Done Done Sent false
26065 8082 Acked Write 3 Done Done Sent false
26066 8083 Acked Read 6 Done Done Sent false
26067 8084 Acked Read 4 Done Done Sent false
26068 8085 Acked Read 8 Done Done Sent false
26069 8086 Acked Write 1 Done Done Sent false
26070 8087 Acked Read 2 Done Done Sent false
26071 8088 Acked Write 2 Done Done Sent false
26072 8089 Acked Write 6 Done Done Sent false
26073 8090 Acked Flush 0 Done Done Sent false
26074 8091 Acked Write 5 Done Done Sent false
26075 8092 Acked Write 1 Done Done Sent false
26076 8093 Acked Read 2 Done Done Sent false
26077 8094 Acked Write 10 Done Done Sent false
26078 8095 Acked Read 9 Done Done Sent false
26079 8096 Acked Read 2 Done Done Sent false
26080 8097 Acked Write 2 Done Done Sent false
26081 8098 Acked Read 7 Done Done Sent false
26082 8099 Acked Write 4 Done Done Sent false
26083 8100 Acked Read 10 Done Done Sent false
26084 8101 Acked Write 1 Done Done Sent false
26085 8102 Acked Write 8 Done Done Sent false
26086 8103 Acked Write 10 Done Done Sent false
26087 8104 Acked Write 4 Done Done Sent false
26088 8105 Acked Write 1 Done Done Sent false
26089 8106 Acked Write 8 Done Done Sent false
26090 8107 Acked Read 3 Done Done Sent false
26091 8108 Acked Read 3 Done Done Sent false
26092 8109 Acked Read 8 Done Done Sent false
26093 8110 Acked Write 6 Done Done Sent false
26094 8111 Acked Read 8 Done Done Sent false
26095 8112 Acked Write 1 Done Done Sent false
26096 8113 Acked Read 4 Done Done Sent false
26097 8114 Acked Read 6 Done Done Sent false
26098 8115 Acked Write 7 Done Done Sent false
26099 8116 Acked Read 10 Done Done Sent false
26100 8117 Acked Read 6 Done Done Sent false
26101 8118 Acked Read 4 Done Done Sent false
26102 8119 Acked Read 1 Done Done Sent false
26103 8120 Acked Write 10 Done Done Sent false
26104 8121 Acked Read 8 Done Done Sent false
26105 8122 Acked Read 1 Done Done Sent false
26106 8123 Acked Write 3 Done Done Sent false
26107 8124 Acked Flush 0 Done Done Sent false
26108 8125 Acked Write 7 Done Done Sent false
26109 8126 Acked Flush 0 Done Done Sent false
26110 8127 Acked Read 4 Done Done Sent false
26111 8128 Acked Read 6 Done Done Sent false
26112 8129 Acked Read 2 Done Done Sent false
26113 8130 Acked Read 6 Done Done Sent false
26114 8131 Acked Read 3 Done Done Sent false
26115 8132 Acked Write 10 Done Done Sent false
26116 8133 Acked Flush 0 Done Done Sent false
26117 8134 Acked Read 9 Done Done Sent false
26118 8135 Acked Read 6 Done Done Sent false
26119 8136 Acked Flush 0 Done Done Sent false
26120 8137 Acked Read 6 Done Done Sent false
26121 8138 Acked Read 7 Done Done Sent false
26122 8139 Acked Read 5 Done Done Sent false
26123 8140 Acked Write 3 Done Done Sent false
26124 8141 Acked Read 10 Done Done Sent false
26125 8142 Acked Read 6 Done Done Sent false
26126 8143 Acked Read 2 Done Done Sent false
26127 8144 Acked Read 3 Done Done Sent false
26128 8145 Acked Read 9 Done Done Sent false
26129 8146 Acked Write 5 Done Done Sent false
26130 8147 Acked Write 1 Done Done Sent false
26131 8148 Acked Write 1 Done Done Sent false
26132 8149 Acked Write 4 Done Done Sent false
26133 8150 Acked Write 4 Done Done Sent false
26134 8151 Acked Write 7 Done Done Sent false
26135 8152 Acked Write 5 Done Done Sent false
26136 8153 Acked Write 3 Done Done Sent false
26137 8154 Acked Write 9 Done Done Sent false
26138 8155 Acked Read 5 Done Done Sent false
26139 8156 Acked Write 9 Done Done Sent false
26140 8157 Acked Write 7 Done Done Sent false
26141 8158 Acked Write 4 Done Done Sent false
26142 8159 Acked Write 2 Done Done Sent false
26143 8160 Acked Write 5 Done Done Sent false
26144 8161 Acked Write 1 Done Done Sent false
26145 8162 Acked Read 4 Done Done Sent false
26146 8163 Acked Read 3 Done Done Sent false
26147 8164 Acked Write 3 Done Done Sent false
26148 8165 Acked Flush 0 Done Done Sent false
26149 8166 Acked Write 7 Done Done Sent false
26150 8167 Acked Read 1 Done Done Sent false
26151 8168 Acked Read 4 Done Done Sent false
26152 8169 Acked Read 1 Done Done Sent false
26153 8170 Acked Write 4 Done Done Sent false
26154 8171 Acked Read 9 Done Done Sent false
26155 8172 Acked Read 4 Done Done Sent false
26156 8173 Acked Write 2 Done Done Sent false
26157 8174 Acked Write 8 Done Done Sent false
26158 8175 Acked Read 5 Done Done Sent false
26159 8176 Acked Write 3 Done Done Sent false
26160 8177 Acked Write 7 Done Done Sent false
26161 8178 Acked Read 8 Done Done Sent false
26162 8179 Acked Write 2 Done Done Sent false
26163 8180 Acked Flush 0 Done Done Sent false
26164 8181 Acked Write 7 Done Done Sent false
26165 8182 Acked Read 5 Done Done Sent false
26166 8183 Acked Write 4 Done Done Sent false
26167 8184 Acked Write 9 Done Done Sent false
26168 8185 Acked Read 2 Done Done Sent false
26169 8186 Acked Read 7 Done Done Sent false
26170 8187 Acked Read 9 Done Done Sent false
26171 8188 Acked Write 3 Done Done Sent false
26172 8189 Acked Flush 0 Done Done Sent false
26173 8190 Acked Read 3 Done Done Sent false
26174 8191 Acked Read 2 Done Done Sent false
26175 8192 Acked Write 5 Done Done Sent false
26176 8193 Acked Write 5 Done Done Sent false
26177 8194 Acked Write 4 Done Done Sent false
26178 8195 Acked Write 8 Done Done Sent false
26179 8196 Acked Write 8 Done Done Sent false
26180 8197 Acked Read 10 Done Done Sent false
26181 8198 Acked Read 5 Done Done Sent false
26182 8199 Acked Read 4 Done Done Sent false
26183 8200 Acked Read 7 Done Done Sent false
26184 8201 Acked Write 1 Done Done Sent false
26185 8202 Acked Write 2 Done Done Sent false
26186 8203 Acked Read 8 Done Done Sent false
26187 8204 Acked Flush 0 Done Done Sent false
26188 8205 Acked Write 4 Done Done Sent false
26189 8206 Acked Read 5 Done Done Sent false
26190 8207 Acked Read 8 Done Done Sent false
26191 8208 Acked Read 10 Done Done Sent false
26192 8209 Acked Write 9 Done Done Sent false
26193 8210 Acked Write 8 Done Done Sent false
26194 8211 Acked Write 1 Done Done Sent false
26195 8212 Acked Read 10 Done Done Sent false
26196 8213 Acked Write 8 Done Done Sent false
26197 8214 Acked Read 3 Done Done Sent false
26198 8215 Acked Write 4 Done Done Sent false
26199 8216 Acked Flush 0 Done Done Sent false
26200 8217 Acked Read 7 Done Done Sent false
26201 8218 Acked Read 1 Done Done Sent false
26202 8219 Acked Read 2 Done Done Sent false
26203 8220 Acked Flush 0 Done Done Sent false
26204 8221 Acked Read 6 Done Done Sent false
26205 8222 Acked Read 7 Done Done Sent false
26206 8223 Acked Read 5 Done Done Sent false
26207 8224 Acked Read 2 Done Done Sent false
26208 8225 Acked Write 4 Done Done Sent false
26209 8226 Acked Read 2 Done Done Sent false
26210 8227 Acked Read 6 Done Done Sent false
26211 8228 Acked Read 3 Done Done Sent false
26212 8229 Acked Read 5 Done Done Sent false
26213 8230 Acked Read 1 Done Done Sent false
26214 8231 Acked Write 5 Done Done Sent false
26215 8232 Acked Read 1 Done Done Sent false
26216 8233 Acked Read 10 Done Done Sent false
26217 8234 Acked Read 1 Done Done Sent false
26218 8235 Acked Read 1 Done Done Sent false
26219 8236 Acked Write 7 Done Done Sent false
26220 8237 Acked Read 10 Done Done Sent false
26221 8238 Acked Write 7 Done Done Sent false
26222 8239 Acked Read 1 Done Done Sent false
26223 8240 Acked Write 8 Done Done Sent false
26224 8241 Acked Read 3 Done Done Sent false
26225 8242 Acked Write 3 Done Done Sent false
26226 8243 Acked Write 3 Done Done Sent false
26227 8244 Acked Read 10 Done Done Sent false
26228 STATES DS:0 DS:1 DS:2 TOTAL
26229 Sent 0 0 301 301
26230 Done 301 301 0 602
26231 Skipped 0 0 0 0
26232 Error 0 0 0 0
26233 Last Flush: Some(JobId(8220)) Some(JobId(8220)) Some(JobId(7943))
26234 Downstairs last five completed: 7943 7942 7941 7940 7939
26235 Upstairs last five completed: 8244 8243 8242 8241 8240
26236 CLIENT: Up:0 ds:301 act:2
262372025-01-28T22:00:00.320ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
262382025-01-28T22:00:00.839ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
262392025-01-28T22:00:01.570ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
262402025-01-28T22:00:01.570ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
262412025-01-28T22:00:01.570ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
262422025-01-28T22:00:01.570ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
262432025-01-28T22:00:01.571ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
262442025-01-28T22:00:02.072ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
262452025-01-28T22:00:02.841ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
26246 [8830] got start action
26247 [8830] I am going to restart
26248 Starting downstairs at port 8830
26249 Make output file at "/tmp/dsc/downstairs-8830.txt"
26250 [8830][2] reports Starting
26251 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1226)
26252 [8830][2] reports Running
26253 ----------------------------------------------------------------
26254 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26255 States: Active Active Offline
26256 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26257 7944 Acked Read 5 Done Done Sent false
26258 7945 Acked Read 6 Done Done Sent false
26259 7946 Acked Write 2 Done Done Sent false
26260 7947 Acked Write 3 Done Done Sent false
26261 7948 Acked Read 9 Done Done Sent false
26262 7949 Acked Read 5 Done Done Sent false
26263 7950 Acked Write 2 Done Done Sent false
26264 7951 Acked Read 7 Done Done Sent false
26265 7952 Acked Read 1 Done Done Sent false
26266 7953 Acked Write 2 Done Done Sent false
26267 7954 Acked Write 1 Done Done Sent false
26268 7955 Acked Read 3 Done Done Sent false
26269 7956 Acked Read 6 Done Done Sent false
26270 7957 Acked Write 7 Done Done Sent false
26271 7958 Acked Write 3 Done Done Sent false
26272 7959 Acked Read 10 Done Done Sent false
26273 7960 Acked Read 10 Done Done Sent false
26274 7961 Acked Read 2 Done Done Sent false
26275 7962 Acked Read 10 Done Done Sent false
26276 7963 Acked Read 7 Done Done Sent false
26277 7964 Acked Write 6 Done Done Sent false
26278 7965 Acked Read 8 Done Done Sent false
26279 7966 Acked Write 5 Done Done Sent false
26280 7967 Acked Write 2 Done Done Sent false
26281 7968 Acked Read 4 Done Done Sent false
26282 7969 Acked Read 6 Done Done Sent false
26283 7970 Acked Read 6 Done Done Sent false
26284 7971 Acked Read 3 Done Done Sent false
26285 7972 Acked Read 10 Done Done Sent false
26286 7973 Acked Read 7 Done Done Sent false
26287 7974 Acked Read 8 Done Done Sent false
26288 7975 Acked Write 6 Done Done Sent false
26289 7976 Acked Write 4 Done Done Sent false
26290 7977 Acked Read 10 Done Done Sent false
26291 7978 Acked Read 4 Done Done Sent false
26292 7979 Acked Read 3 Done Done Sent false
26293 7980 Acked Read 8 Done Done Sent false
26294 7981 Acked Read 5 Done Done Sent false
26295 7982 Acked Read 6 Done Done Sent false
26296 7983 Acked Write 7 Done Done Sent false
26297 7984 Acked Write 7 Done Done Sent false
26298 7985 Acked Read 8 Done Done Sent false
26299 7986 Acked Read 7 Done Done Sent false
26300 7987 Acked Write 3 Done Done Sent false
26301 7988 Acked Read 6 Done Done Sent false
26302 7989 Acked Read 5 Done Done Sent false
26303 7990 Acked Write 8 Done Done Sent false
26304 7991 Acked Read 5 Done Done Sent false
26305 7992 Acked Write 9 Done Done Sent false
26306 7993 Acked Write 9 Done Done Sent false
26307 7994 Acked Read 7 Done Done Sent false
26308 7995 Acked Write 8 Done Done Sent false
26309 7996 Acked Read 1 Done Done Sent false
26310 7997 Acked Read 2 Done Done Sent false
26311 7998 Acked Read 10 Done Done Sent false
26312 7999 Acked Write 4 Done Done Sent false
26313 8000 Acked Flush 0 Done Done Sent false
26314 8001 Acked Write 6 Done Done Sent false
26315 8002 Acked Read 1 Done Done Sent false
26316 8003 Acked Read 3 Done Done Sent false
26317 8004 Acked Read 5 Done Done Sent false
26318 8005 Acked Read 6 Done Done Sent false
26319 8006 Acked Read 4 Done Done Sent false
26320 8007 Acked Read 8 Done Done Sent false
26321 8008 Acked Write 10 Done Done Sent false
26322 8009 Acked Read 9 Done Done Sent false
26323 8010 Acked Write 1 Done Done Sent false
26324 8011 Acked Read 2 Done Done Sent false
26325 8012 Acked Write 7 Done Done Sent false
26326 8013 Acked Read 7 Done Done Sent false
26327 8014 Acked Flush 0 Done Done Sent false
26328 8015 Acked Read 6 Done Done Sent false
26329 8016 Acked Read 6 Done Done Sent false
26330 8017 Acked Read 2 Done Done Sent false
26331 8018 Acked Read 9 Done Done Sent false
26332 8019 Acked Read 10 Done Done Sent false
26333 8020 Acked Flush 0 Done Done Sent false
26334 8021 Acked Read 3 Done Done Sent false
26335 8022 Acked Read 10 Done Done Sent false
26336 8023 Acked Read 3 Done Done Sent false
26337 8024 Acked Read 8 Done Done Sent false
26338 8025 Acked Read 8 Done Done Sent false
26339 8026 Acked Flush 0 Done Done Sent false
26340 8027 Acked Read 3 Done Done Sent false
26341 8028 Acked Write 4 Done Done Sent false
26342 8029 Acked Flush 0 Done Done Sent false
26343 8030 Acked Read 2 Done Done Sent false
26344 8031 Acked Write 6 Done Done Sent false
26345 8032 Acked Read 8 Done Done Sent false
26346 8033 Acked Read 6 Done Done Sent false
26347 8034 Acked Write 1 Done Done Sent false
26348 8035 Acked Read 8 Done Done Sent false
26349 8036 Acked Read 7 Done Done Sent false
26350 8037 Acked Write 2 Done Done Sent false
26351 8038 Acked Write 5 Done Done Sent false
26352 8039 Acked Write 2 Done Done Sent false
26353 8040 Acked Write 1 Done Done Sent false
26354 8041 Acked Read 10 Done Done Sent false
26355 8042 Acked Read 1 Done Done Sent false
26356 8043 Acked Write 5 Done Done Sent false
26357 8044 Acked Write 1 Done Done Sent false
26358 8045 Acked Write 5 Done Done Sent false
26359 8046 Acked Write 9 Done Done Sent false
26360 8047 Acked Write 6 Done Done Sent false
26361 8048 Acked Read 5 Done Done Sent false
26362 8049 Acked Read 5 Done Done Sent false
26363 8050 Acked Write 7 Done Done Sent false
26364 8051 Acked Flush 0 Done Done Sent false
26365 8052 Acked Write 8 Done Done Sent false
26366 8053 Acked Read 4 Done Done Sent false
26367 8054 Acked Read 8 Done Done Sent false
26368 8055 Acked Read 5 Done Done Sent false
26369 8056 Acked Read 10 Done Done Sent false
26370 8057 Acked Write 9 Done Done Sent false
26371 8058 Acked Read 8 Done Done Sent false
26372 8059 Acked Read 1 Done Done Sent false
26373 8060 Acked Flush 0 Done Done Sent false
26374 8061 Acked Write 4 Done Done Sent false
26375 8062 Acked Write 10 Done Done Sent false
26376 8063 Acked Read 10 Done Done Sent false
26377 8064 Acked Flush 0 Done Done Sent false
26378 8065 Acked Read 6 Done Done Sent false
26379 8066 Acked Write 9 Done Done Sent false
26380 8067 Acked Flush 0 Done Done Sent false
26381 8068 Acked Write 8 Done Done Sent false
26382 8069 Acked Write 8 Done Done Sent false
26383 8070 Acked Write 2 Done Done Sent false
26384 8071 Acked Read 10 Done Done Sent false
26385 8072 Acked Read 8 Done Done Sent false
26386 8073 Acked Read 4 Done Done Sent false
26387 8074 Acked Write 7 Done Done Sent false
26388 8075 Acked Read 7 Done Done Sent false
26389 8076 Acked Read 2 Done Done Sent false
26390 8077 Acked Write 4 Done Done Sent false
26391 8078 Acked Flush 0 Done Done Sent false
26392 8079 Acked Read 10 Done Done Sent false
26393 8080 Acked Read 3 Done Done Sent false
26394 8081 Acked Write 6 Done Done Sent false
26395 8082 Acked Write 3 Done Done Sent false
26396 8083 Acked Read 6 Done Done Sent false
26397 8084 Acked Read 4 Done Done Sent false
26398 8085 Acked Read 8 Done Done Sent false
26399 8086 Acked Write 1 Done Done Sent false
26400 8087 Acked Read 2 Done Done Sent false
26401 8088 Acked Write 2 Done Done Sent false
26402 8089 Acked Write 6 Done Done Sent false
26403 8090 Acked Flush 0 Done Done Sent false
26404 8091 Acked Write 5 Done Done Sent false
26405 8092 Acked Write 1 Done Done Sent false
26406 8093 Acked Read 2 Done Done Sent false
26407 8094 Acked Write 10 Done Done Sent false
26408 8095 Acked Read 9 Done Done Sent false
26409 8096 Acked Read 2 Done Done Sent false
26410 8097 Acked Write 2 Done Done Sent false
26411 8098 Acked Read 7 Done Done Sent false
26412 8099 Acked Write 4 Done Done Sent false
26413 8100 Acked Read 10 Done Done Sent false
26414 8101 Acked Write 1 Done Done Sent false
26415 8102 Acked Write 8 Done Done Sent false
26416 8103 Acked Write 10 Done Done Sent false
26417 8104 Acked Write 4 Done Done Sent false
26418 8105 Acked Write 1 Done Done Sent false
26419 8106 Acked Write 8 Done Done Sent false
26420 8107 Acked Read 3 Done Done Sent false
26421 8108 Acked Read 3 Done Done Sent false
26422 8109 Acked Read 8 Done Done Sent false
26423 8110 Acked Write 6 Done Done Sent false
26424 8111 Acked Read 8 Done Done Sent false
26425 8112 Acked Write 1 Done Done Sent false
26426 8113 Acked Read 4 Done Done Sent false
26427 8114 Acked Read 6 Done Done Sent false
26428 8115 Acked Write 7 Done Done Sent false
26429 8116 Acked Read 10 Done Done Sent false
26430 8117 Acked Read 6 Done Done Sent false
26431 8118 Acked Read 4 Done Done Sent false
26432 8119 Acked Read 1 Done Done Sent false
26433 8120 Acked Write 10 Done Done Sent false
26434 8121 Acked Read 8 Done Done Sent false
26435 8122 Acked Read 1 Done Done Sent false
26436 8123 Acked Write 3 Done Done Sent false
26437 8124 Acked Flush 0 Done Done Sent false
26438 8125 Acked Write 7 Done Done Sent false
26439 8126 Acked Flush 0 Done Done Sent false
26440 8127 Acked Read 4 Done Done Sent false
26441 8128 Acked Read 6 Done Done Sent false
26442 8129 Acked Read 2 Done Done Sent false
26443 8130 Acked Read 6 Done Done Sent false
26444 8131 Acked Read 3 Done Done Sent false
26445 8132 Acked Write 10 Done Done Sent false
26446 8133 Acked Flush 0 Done Done Sent false
26447 8134 Acked Read 9 Done Done Sent false
26448 8135 Acked Read 6 Done Done Sent false
26449 8136 Acked Flush 0 Done Done Sent false
26450 8137 Acked Read 6 Done Done Sent false
26451 8138 Acked Read 7 Done Done Sent false
26452 8139 Acked Read 5 Done Done Sent false
26453 8140 Acked Write 3 Done Done Sent false
26454 8141 Acked Read 10 Done Done Sent false
26455 8142 Acked Read 6 Done Done Sent false
26456 8143 Acked Read 2 Done Done Sent false
26457 8144 Acked Read 3 Done Done Sent false
26458 8145 Acked Read 9 Done Done Sent false
26459 8146 Acked Write 5 Done Done Sent false
26460 8147 Acked Write 1 Done Done Sent false
26461 8148 Acked Write 1 Done Done Sent false
26462 8149 Acked Write 4 Done Done Sent false
26463 8150 Acked Write 4 Done Done Sent false
26464 8151 Acked Write 7 Done Done Sent false
26465 8152 Acked Write 5 Done Done Sent false
26466 8153 Acked Write 3 Done Done Sent false
26467 8154 Acked Write 9 Done Done Sent false
26468 8155 Acked Read 5 Done Done Sent false
26469 8156 Acked Write 9 Done Done Sent false
26470 8157 Acked Write 7 Done Done Sent false
26471 8158 Acked Write 4 Done Done Sent false
26472 8159 Acked Write 2 Done Done Sent false
26473 8160 Acked Write 5 Done Done Sent false
26474 8161 Acked Write 1 Done Done Sent false
26475 8162 Acked Read 4 Done Done Sent false
26476 8163 Acked Read 3 Done Done Sent false
26477 8164 Acked Write 3 Done Done Sent false
26478 8165 Acked Flush 0 Done Done Sent false
26479 8166 Acked Write 7 Done Done Sent false
26480 8167 Acked Read 1 Done Done Sent false
26481 8168 Acked Read 4 Done Done Sent false
26482 8169 Acked Read 1 Done Done Sent false
26483 8170 Acked Write 4 Done Done Sent false
26484 8171 Acked Read 9 Done Done Sent false
26485 8172 Acked Read 4 Done Done Sent false
26486 8173 Acked Write 2 Done Done Sent false
26487 8174 Acked Write 8 Done Done Sent false
26488 8175 Acked Read 5 Done Done Sent false
26489 8176 Acked Write 3 Done Done Sent false
26490 8177 Acked Write 7 Done Done Sent false
26491 8178 Acked Read 8 Done Done Sent false
26492 8179 Acked Write 2 Done Done Sent false
26493 8180 Acked Flush 0 Done Done Sent false
26494 8181 Acked Write 7 Done Done Sent false
26495 8182 Acked Read 5 Done Done Sent false
26496 8183 Acked Write 4 Done Done Sent false
26497 8184 Acked Write 9 Done Done Sent false
26498 8185 Acked Read 2 Done Done Sent false
26499 8186 Acked Read 7 Done Done Sent false
26500 8187 Acked Read 9 Done Done Sent false
26501 8188 Acked Write 3 Done Done Sent false
26502 8189 Acked Flush 0 Done Done Sent false
26503 8190 Acked Read 3 Done Done Sent false
26504 8191 Acked Read 2 Done Done Sent false
26505 8192 Acked Write 5 Done Done Sent false
26506 8193 Acked Write 5 Done Done Sent false
26507 8194 Acked Write 4 Done Done Sent false
26508 8195 Acked Write 8 Done Done Sent false
26509 8196 Acked Write 8 Done Done Sent false
26510 8197 Acked Read 10 Done Done Sent false
26511 8198 Acked Read 5 Done Done Sent false
26512 8199 Acked Read 4 Done Done Sent false
26513 8200 Acked Read 7 Done Done Sent false
26514 8201 Acked Write 1 Done Done Sent false
26515 8202 Acked Write 2 Done Done Sent false
26516 8203 Acked Read 8 Done Done Sent false
26517 8204 Acked Flush 0 Done Done Sent false
26518 8205 Acked Write 4 Done Done Sent false
26519 8206 Acked Read 5 Done Done Sent false
26520 8207 Acked Read 8 Done Done Sent false
26521 8208 Acked Read 10 Done Done Sent false
26522 8209 Acked Write 9 Done Done Sent false
26523 8210 Acked Write 8 Done Done Sent false
26524 8211 Acked Write 1 Done Done Sent false
26525 8212 Acked Read 10 Done Done Sent false
26526 8213 Acked Write 8 Done Done Sent false
26527 8214 Acked Read 3 Done Done Sent false
26528 8215 Acked Write 4 Done Done Sent false
26529 8216 Acked Flush 0 Done Done Sent false
26530 8217 Acked Read 7 Done Done Sent false
26531 8218 Acked Read 1 Done Done Sent false
26532 8219 Acked Read 2 Done Done Sent false
26533 8220 Acked Flush 0 Done Done Sent false
26534 8221 Acked Read 6 Done Done Sent false
26535 8222 Acked Read 7 Done Done Sent false
26536 8223 Acked Read 5 Done Done Sent false
26537 8224 Acked Read 2 Done Done Sent false
26538 8225 Acked Write 4 Done Done Sent false
26539 8226 Acked Read 2 Done Done Sent false
26540 8227 Acked Read 6 Done Done Sent false
26541 8228 Acked Read 3 Done Done Sent false
26542 8229 Acked Read 5 Done Done Sent false
26543 8230 Acked Read 1 Done Done Sent false
26544 8231 Acked Write 5 Done Done Sent false
26545 8232 Acked Read 1 Done Done Sent false
26546 8233 Acked Read 10 Done Done Sent false
26547 8234 Acked Read 1 Done Done Sent false
26548 8235 Acked Read 1 Done Done Sent false
26549 8236 Acked Write 7 Done Done Sent false
26550 8237 Acked Read 10 Done Done Sent false
26551 8238 Acked Write 7 Done Done Sent false
26552 8239 Acked Read 1 Done Done Sent false
26553 8240 Acked Write 8 Done Done Sent false
26554 8241 Acked Read 3 Done Done Sent false
26555 8242 Acked Write 3 Done Done Sent false
26556 8243 Acked Write 3 Done Done Sent false
26557 8244 Acked Read 10 Done Done Sent false
26558 STATES DS:0 DS:1 DS:2 TOTAL
26559 Sent 0 0 301 301
26560 Done 301 301 0 602
26561 Skipped 0 0 0 0
26562 Error 0 0 0 0
26563 Last Flush: Some(JobId(8220)) Some(JobId(8220)) Some(JobId(7943))
26564 Downstairs last five completed: 7943 7942 7941 7940 7939
26565 Upstairs last five completed: 8244 8243 8242 8241 8240
26566 CLIENT: Up:0 ds:301 act:2
265672025-01-28T22:00:04.322ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
265682025-01-28T22:00:04.844ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
265692025-01-28T22:00:05.845ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
265702025-01-28T22:00:06.847ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
26571 ----------------------------------------------------------------
26572 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26573 States: Active Active Offline
26574 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26575 7944 Acked Read 5 Done Done Sent false
26576 7945 Acked Read 6 Done Done Sent false
26577 7946 Acked Write 2 Done Done Sent false
26578 7947 Acked Write 3 Done Done Sent false
26579 7948 Acked Read 9 Done Done Sent false
26580 7949 Acked Read 5 Done Done Sent false
26581 7950 Acked Write 2 Done Done Sent false
26582 7951 Acked Read 7 Done Done Sent false
26583 7952 Acked Read 1 Done Done Sent false
26584 7953 Acked Write 2 Done Done Sent false
26585 7954 Acked Write 1 Done Done Sent false
26586 7955 Acked Read 3 Done Done Sent false
26587 7956 Acked Read 6 Done Done Sent false
26588 7957 Acked Write 7 Done Done Sent false
26589 7958 Acked Write 3 Done Done Sent false
26590 7959 Acked Read 10 Done Done Sent false
26591 7960 Acked Read 10 Done Done Sent false
26592 7961 Acked Read 2 Done Done Sent false
26593 7962 Acked Read 10 Done Done Sent false
26594 7963 Acked Read 7 Done Done Sent false
26595 7964 Acked Write 6 Done Done Sent false
26596 7965 Acked Read 8 Done Done Sent false
26597 7966 Acked Write 5 Done Done Sent false
26598 7967 Acked Write 2 Done Done Sent false
26599 7968 Acked Read 4 Done Done Sent false
26600 7969 Acked Read 6 Done Done Sent false
26601 7970 Acked Read 6 Done Done Sent false
26602 7971 Acked Read 3 Done Done Sent false
26603 7972 Acked Read 10 Done Done Sent false
26604 7973 Acked Read 7 Done Done Sent false
26605 7974 Acked Read 8 Done Done Sent false
26606 7975 Acked Write 6 Done Done Sent false
26607 7976 Acked Write 4 Done Done Sent false
26608 7977 Acked Read 10 Done Done Sent false
26609 7978 Acked Read 4 Done Done Sent false
26610 7979 Acked Read 3 Done Done Sent false
26611 7980 Acked Read 8 Done Done Sent false
26612 7981 Acked Read 5 Done Done Sent false
26613 7982 Acked Read 6 Done Done Sent false
26614 7983 Acked Write 7 Done Done Sent false
26615 7984 Acked Write 7 Done Done Sent false
26616 7985 Acked Read 8 Done Done Sent false
26617 7986 Acked Read 7 Done Done Sent false
26618 7987 Acked Write 3 Done Done Sent false
26619 7988 Acked Read 6 Done Done Sent false
26620 7989 Acked Read 5 Done Done Sent false
26621 7990 Acked Write 8 Done Done Sent false
26622 7991 Acked Read 5 Done Done Sent false
26623 7992 Acked Write 9 Done Done Sent false
26624 7993 Acked Write 9 Done Done Sent false
26625 7994 Acked Read 7 Done Done Sent false
26626 7995 Acked Write 8 Done Done Sent false
26627 7996 Acked Read 1 Done Done Sent false
26628 7997 Acked Read 2 Done Done Sent false
26629 7998 Acked Read 10 Done Done Sent false
26630 7999 Acked Write 4 Done Done Sent false
26631 8000 Acked Flush 0 Done Done Sent false
26632 8001 Acked Write 6 Done Done Sent false
26633 8002 Acked Read 1 Done Done Sent false
26634 8003 Acked Read 3 Done Done Sent false
26635 8004 Acked Read 5 Done Done Sent false
26636 8005 Acked Read 6 Done Done Sent false
26637 8006 Acked Read 4 Done Done Sent false
26638 8007 Acked Read 8 Done Done Sent false
26639 8008 Acked Write 10 Done Done Sent false
26640 8009 Acked Read 9 Done Done Sent false
26641 8010 Acked Write 1 Done Done Sent false
26642 8011 Acked Read 2 Done Done Sent false
26643 8012 Acked Write 7 Done Done Sent false
26644 8013 Acked Read 7 Done Done Sent false
26645 8014 Acked Flush 0 Done Done Sent false
26646 8015 Acked Read 6 Done Done Sent false
26647 8016 Acked Read 6 Done Done Sent false
26648 8017 Acked Read 2 Done Done Sent false
26649 8018 Acked Read 9 Done Done Sent false
26650 8019 Acked Read 10 Done Done Sent false
26651 8020 Acked Flush 0 Done Done Sent false
26652 8021 Acked Read 3 Done Done Sent false
26653 8022 Acked Read 10 Done Done Sent false
26654 8023 Acked Read 3 Done Done Sent false
26655 8024 Acked Read 8 Done Done Sent false
26656 8025 Acked Read 8 Done Done Sent false
26657 8026 Acked Flush 0 Done Done Sent false
26658 8027 Acked Read 3 Done Done Sent false
26659 8028 Acked Write 4 Done Done Sent false
26660 8029 Acked Flush 0 Done Done Sent false
26661 8030 Acked Read 2 Done Done Sent false
26662 8031 Acked Write 6 Done Done Sent false
26663 8032 Acked Read 8 Done Done Sent false
26664 8033 Acked Read 6 Done Done Sent false
26665 8034 Acked Write 1 Done Done Sent false
26666 8035 Acked Read 8 Done Done Sent false
26667 8036 Acked Read 7 Done Done Sent false
26668 8037 Acked Write 2 Done Done Sent false
26669 8038 Acked Write 5 Done Done Sent false
26670 8039 Acked Write 2 Done Done Sent false
26671 8040 Acked Write 1 Done Done Sent false
26672 8041 Acked Read 10 Done Done Sent false
26673 8042 Acked Read 1 Done Done Sent false
26674 8043 Acked Write 5 Done Done Sent false
26675 8044 Acked Write 1 Done Done Sent false
26676 8045 Acked Write 5 Done Done Sent false
26677 8046 Acked Write 9 Done Done Sent false
26678 8047 Acked Write 6 Done Done Sent false
26679 8048 Acked Read 5 Done Done Sent false
26680 8049 Acked Read 5 Done Done Sent false
26681 8050 Acked Write 7 Done Done Sent false
26682 8051 Acked Flush 0 Done Done Sent false
26683 8052 Acked Write 8 Done Done Sent false
26684 8053 Acked Read 4 Done Done Sent false
26685 8054 Acked Read 8 Done Done Sent false
26686 8055 Acked Read 5 Done Done Sent false
26687 8056 Acked Read 10 Done Done Sent false
26688 8057 Acked Write 9 Done Done Sent false
26689 8058 Acked Read 8 Done Done Sent false
26690 8059 Acked Read 1 Done Done Sent false
26691 8060 Acked Flush 0 Done Done Sent false
26692 8061 Acked Write 4 Done Done Sent false
26693 8062 Acked Write 10 Done Done Sent false
26694 8063 Acked Read 10 Done Done Sent false
26695 8064 Acked Flush 0 Done Done Sent false
26696 8065 Acked Read 6 Done Done Sent false
26697 8066 Acked Write 9 Done Done Sent false
26698 8067 Acked Flush 0 Done Done Sent false
26699 8068 Acked Write 8 Done Done Sent false
26700 8069 Acked Write 8 Done Done Sent false
26701 8070 Acked Write 2 Done Done Sent false
26702 8071 Acked Read 10 Done Done Sent false
26703 8072 Acked Read 8 Done Done Sent false
26704 8073 Acked Read 4 Done Done Sent false
26705 8074 Acked Write 7 Done Done Sent false
26706 8075 Acked Read 7 Done Done Sent false
26707 8076 Acked Read 2 Done Done Sent false
26708 8077 Acked Write 4 Done Done Sent false
26709 8078 Acked Flush 0 Done Done Sent false
26710 8079 Acked Read 10 Done Done Sent false
26711 8080 Acked Read 3 Done Done Sent false
26712 8081 Acked Write 6 Done Done Sent false
26713 8082 Acked Write 3 Done Done Sent false
26714 8083 Acked Read 6 Done Done Sent false
26715 8084 Acked Read 4 Done Done Sent false
26716 8085 Acked Read 8 Done Done Sent false
26717 8086 Acked Write 1 Done Done Sent false
26718 8087 Acked Read 2 Done Done Sent false
26719 8088 Acked Write 2 Done Done Sent false
26720 8089 Acked Write 6 Done Done Sent false
26721 8090 Acked Flush 0 Done Done Sent false
26722 8091 Acked Write 5 Done Done Sent false
26723 8092 Acked Write 1 Done Done Sent false
26724 8093 Acked Read 2 Done Done Sent false
26725 8094 Acked Write 10 Done Done Sent false
26726 8095 Acked Read 9 Done Done Sent false
26727 8096 Acked Read 2 Done Done Sent false
26728 8097 Acked Write 2 Done Done Sent false
26729 8098 Acked Read 7 Done Done Sent false
26730 8099 Acked Write 4 Done Done Sent false
26731 8100 Acked Read 10 Done Done Sent false
26732 8101 Acked Write 1 Done Done Sent false
26733 8102 Acked Write 8 Done Done Sent false
26734 8103 Acked Write 10 Done Done Sent false
26735 8104 Acked Write 4 Done Done Sent false
26736 8105 Acked Write 1 Done Done Sent false
26737 8106 Acked Write 8 Done Done Sent false
26738 8107 Acked Read 3 Done Done Sent false
26739 8108 Acked Read 3 Done Done Sent false
26740 8109 Acked Read 8 Done Done Sent false
26741 8110 Acked Write 6 Done Done Sent false
26742 8111 Acked Read 8 Done Done Sent false
26743 8112 Acked Write 1 Done Done Sent false
26744 8113 Acked Read 4 Done Done Sent false
26745 8114 Acked Read 6 Done Done Sent false
26746 8115 Acked Write 7 Done Done Sent false
26747 8116 Acked Read 10 Done Done Sent false
26748 8117 Acked Read 6 Done Done Sent false
26749 8118 Acked Read 4 Done Done Sent false
26750 8119 Acked Read 1 Done Done Sent false
26751 8120 Acked Write 10 Done Done Sent false
26752 8121 Acked Read 8 Done Done Sent false
26753 8122 Acked Read 1 Done Done Sent false
26754 8123 Acked Write 3 Done Done Sent false
26755 8124 Acked Flush 0 Done Done Sent false
26756 8125 Acked Write 7 Done Done Sent false
26757 8126 Acked Flush 0 Done Done Sent false
26758 8127 Acked Read 4 Done Done Sent false
26759 8128 Acked Read 6 Done Done Sent false
26760 8129 Acked Read 2 Done Done Sent false
26761 8130 Acked Read 6 Done Done Sent false
26762 8131 Acked Read 3 Done Done Sent false
26763 8132 Acked Write 10 Done Done Sent false
26764 8133 Acked Flush 0 Done Done Sent false
26765 8134 Acked Read 9 Done Done Sent false
26766 8135 Acked Read 6 Done Done Sent false
26767 8136 Acked Flush 0 Done Done Sent false
26768 8137 Acked Read 6 Done Done Sent false
26769 8138 Acked Read 7 Done Done Sent false
26770 8139 Acked Read 5 Done Done Sent false
26771 8140 Acked Write 3 Done Done Sent false
26772 8141 Acked Read 10 Done Done Sent false
26773 8142 Acked Read 6 Done Done Sent false
26774 8143 Acked Read 2 Done Done Sent false
26775 8144 Acked Read 3 Done Done Sent false
26776 8145 Acked Read 9 Done Done Sent false
26777 8146 Acked Write 5 Done Done Sent false
26778 8147 Acked Write 1 Done Done Sent false
26779 8148 Acked Write 1 Done Done Sent false
26780 8149 Acked Write 4 Done Done Sent false
26781 8150 Acked Write 4 Done Done Sent false
26782 8151 Acked Write 7 Done Done Sent false
26783 8152 Acked Write 5 Done Done Sent false
26784 8153 Acked Write 3 Done Done Sent false
26785 8154 Acked Write 9 Done Done Sent false
26786 8155 Acked Read 5 Done Done Sent false
26787 8156 Acked Write 9 Done Done Sent false
26788 8157 Acked Write 7 Done Done Sent false
26789 8158 Acked Write 4 Done Done Sent false
26790 8159 Acked Write 2 Done Done Sent false
26791 8160 Acked Write 5 Done Done Sent false
26792 8161 Acked Write 1 Done Done Sent false
26793 8162 Acked Read 4 Done Done Sent false
26794 8163 Acked Read 3 Done Done Sent false
26795 8164 Acked Write 3 Done Done Sent false
26796 8165 Acked Flush 0 Done Done Sent false
26797 8166 Acked Write 7 Done Done Sent false
26798 8167 Acked Read 1 Done Done Sent false
26799 8168 Acked Read 4 Done Done Sent false
26800 8169 Acked Read 1 Done Done Sent false
26801 8170 Acked Write 4 Done Done Sent false
26802 8171 Acked Read 9 Done Done Sent false
26803 8172 Acked Read 4 Done Done Sent false
26804 8173 Acked Write 2 Done Done Sent false
26805 8174 Acked Write 8 Done Done Sent false
26806 8175 Acked Read 5 Done Done Sent false
26807 8176 Acked Write 3 Done Done Sent false
26808 8177 Acked Write 7 Done Done Sent false
26809 8178 Acked Read 8 Done Done Sent false
26810 8179 Acked Write 2 Done Done Sent false
26811 8180 Acked Flush 0 Done Done Sent false
26812 8181 Acked Write 7 Done Done Sent false
26813 8182 Acked Read 5 Done Done Sent false
26814 8183 Acked Write 4 Done Done Sent false
26815 8184 Acked Write 9 Done Done Sent false
26816 8185 Acked Read 2 Done Done Sent false
26817 8186 Acked Read 7 Done Done Sent false
26818 8187 Acked Read 9 Done Done Sent false
26819 8188 Acked Write 3 Done Done Sent false
26820 8189 Acked Flush 0 Done Done Sent false
26821 8190 Acked Read 3 Done Done Sent false
26822 8191 Acked Read 2 Done Done Sent false
26823 8192 Acked Write 5 Done Done Sent false
26824 8193 Acked Write 5 Done Done Sent false
26825 8194 Acked Write 4 Done Done Sent false
26826 8195 Acked Write 8 Done Done Sent false
26827 8196 Acked Write 8 Done Done Sent false
26828 8197 Acked Read 10 Done Done Sent false
26829 8198 Acked Read 5 Done Done Sent false
26830 8199 Acked Read 4 Done Done Sent false
26831 8200 Acked Read 7 Done Done Sent false
26832 8201 Acked Write 1 Done Done Sent false
26833 8202 Acked Write 2 Done Done Sent false
26834 8203 Acked Read 8 Done Done Sent false
26835 8204 Acked Flush 0 Done Done Sent false
26836 8205 Acked Write 4 Done Done Sent false
26837 8206 Acked Read 5 Done Done Sent false
26838 8207 Acked Read 8 Done Done Sent false
26839 8208 Acked Read 10 Done Done Sent false
26840 8209 Acked Write 9 Done Done Sent false
26841 8210 Acked Write 8 Done Done Sent false
26842 8211 Acked Write 1 Done Done Sent false
26843 8212 Acked Read 10 Done Done Sent false
26844 8213 Acked Write 8 Done Done Sent false
26845 8214 Acked Read 3 Done Done Sent false
26846 8215 Acked Write 4 Done Done Sent false
26847 8216 Acked Flush 0 Done Done Sent false
26848 8217 Acked Read 7 Done Done Sent false
26849 8218 Acked Read 1 Done Done Sent false
26850 8219 Acked Read 2 Done Done Sent false
26851 8220 Acked Flush 0 Done Done Sent false
26852 8221 Acked Read 6 Done Done Sent false
26853 8222 Acked Read 7 Done Done Sent false
26854 8223 Acked Read 5 Done Done Sent false
26855 8224 Acked Read 2 Done Done Sent false
26856 8225 Acked Write 4 Done Done Sent false
26857 8226 Acked Read 2 Done Done Sent false
26858 8227 Acked Read 6 Done Done Sent false
26859 8228 Acked Read 3 Done Done Sent false
26860 8229 Acked Read 5 Done Done Sent false
26861 8230 Acked Read 1 Done Done Sent false
26862 8231 Acked Write 5 Done Done Sent false
26863 8232 Acked Read 1 Done Done Sent false
26864 8233 Acked Read 10 Done Done Sent false
26865 8234 Acked Read 1 Done Done Sent false
26866 8235 Acked Read 1 Done Done Sent false
26867 8236 Acked Write 7 Done Done Sent false
26868 8237 Acked Read 10 Done Done Sent false
26869 8238 Acked Write 7 Done Done Sent false
26870 8239 Acked Read 1 Done Done Sent false
26871 8240 Acked Write 8 Done Done Sent false
26872 8241 Acked Read 3 Done Done Sent false
26873 8242 Acked Write 3 Done Done Sent false
26874 8243 Acked Write 3 Done Done Sent false
26875 8244 Acked Read 10 Done Done Sent false
26876 STATES DS:0 DS:1 DS:2 TOTAL
26877 Sent 0 0 301 301
26878 Done 301 301 0 602
26879 Skipped 0 0 0 0
26880 Error 0 0 0 0
26881 Last Flush: Some(JobId(8220)) Some(JobId(8220)) Some(JobId(7943))
26882 Downstairs last five completed: 7943 7942 7941 7940 7939
26883 Upstairs last five completed: 8244 8243 8242 8241 8240
26884 CLIENT: Up:0 ds:301 act:2
268852025-01-28T22:00:08.849ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
268862025-01-28T22:00:09.850ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
268872025-01-28T22:00:10.351ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
268882025-01-28T22:00:10.852ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
268892025-01-28T22:00:11.572ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
268902025-01-28T22:00:11.572ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47153) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
268912025-01-28T22:00:11.573ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
268922025-01-28T22:00:11.574ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
268932025-01-28T22:00:11.574ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
268942025-01-28T22:00:11.574ZINFOcrucible: send last flush ID to this DS: Some(JobId(7943)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
268952025-01-28T22:00:11.574ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
268962025-01-28T22:00:11.574ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7943)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
268972025-01-28T22:00:11.574ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7943)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
26898 ----------------------------------------------------------------
26899 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:24
26900 States: Active Active Active
26901 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26902 8221 Acked Read 6 Done Done Done true
26903 8222 Acked Read 7 Done Done Done true
26904 8223 Acked Read 5 Done Done Done true
26905 8224 Acked Read 2 Done Done Done true
26906 8225 Acked Write 4 Done Done Done true
26907 8226 Acked Read 2 Done Done Done true
26908 8227 Acked Read 6 Done Done Done true
26909 8228 Acked Read 3 Done Done Done true
26910 8229 Acked Read 5 Done Done Done true
26911 8230 Acked Read 1 Done Done Done true
26912 8231 Acked Write 5 Done Done Done true
26913 8232 Acked Read 1 Done Done Done true
26914 8233 Acked Read 10 Done Done Done true
26915 8234 Acked Read 1 Done Done Done true
26916 8235 Acked Read 1 Done Done Done true
26917 8236 Acked Write 7 Done Done Done true
26918 8237 Acked Read 10 Done Done Done true
26919 8238 Acked Write 7 Done Done Done true
26920 8239 Acked Read 1 Done Done Done true
26921 8240 Acked Write 8 Done Done Done true
26922 8241 Acked Read 3 Done Done Done true
26923 8242 Acked Write 3 Done Done Done true
26924 8243 Acked Write 3 Done Done Done true
26925 8244 Acked Read 10 Done Done Done true
26926 STATES DS:0 DS:1 DS:2 TOTAL
26927 Sent 0 0 0 0
26928 Done 24 24 24 72
26929 Skipped 0 0 0 0
26930 Error 0 0 0 0
26931 Last Flush: Some(JobId(8220)) Some(JobId(8220)) Some(JobId(8220))
26932 Downstairs last five completed: 8220 8219 8218 8217 8216
26933 Upstairs last five completed: 8244 8243 8242 8241 8240
26934 CLIENT: Up:0 ds:24 act:3
26935 ----------------------------------------------------------------
26936 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26937 Downstairs last five completed: 8245 8244 8243 8242 8241
26938 Upstairs last five completed: 8245 8244 8243 8242 8241
26939 CLIENT: Up:0 ds:0 act:3
26940 Replay: All jobs finished
26941 Main task has work to do, go find it
26942 got dsc Stop(1)
26943 stop 1
26944 Jan 28 22:00:15.827 INFO request completed, latency_us: 387, response_code: [8820] Got stop action so:false kr:false
26945 204, uri: /stop/cid/1, method: POST, req_id: 1e92a37f-3959-4a48-ae07-bf2c179bd1e1, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
269462025-01-28T22:00:15.828ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
26947 Jan 28 22:00:15.828 INFO request completed, latency_us: 128, response_code: 200, uri: /state/cid/1, method: GET, req_id: d4538417-b4d9-4642-97ce-b6613904581c, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
269482025-01-28T22:00:15.828ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
269492025-01-28T22:00:15.828ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
269502025-01-28T22:00:15.828ZINFOcrucible: client task is exiting = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
26951 [8820] Exited with: ExitStatus(unix_wait_status(9))
26952 [8820][1] reports Exit
26953 Jan 28 22:00:19.830 INFO request completed, latency_us: 256, response_code: 200, uri: /state/cid/1, method: GET, req_id: 74ea3335-2603-4f42-836e-38df3469404d, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
26954 001/300 Read block 3783 len 28672
26955 002/300 Read block 4933 len 12288
26956 003/300 Write block 3079 len 28672 data: 9 9 9 8 6 4 3
26957 004/300 Write block 4381 len 40960 data: 8 9 10 9 9 7 6 6 6 6
26958 005/300 Write block 4741 len 4096 data: 8
26959 006/300 Write block 4271 len 36864 data: 7 5 4 5 6 4 4 3 3
26960 007/300 Write block 632 len 36864 data: 4 4 3 5 7 9 4 4 5
26961 008/300 Write block 4600 len 36864 data: 7 6 7 7 6 6 5 6 4
26962 009/300 Flush
26963 010/300 Write block 1413 len 28672 data: 7 7 5 4 3 2 2
26964 011/300 Read block 920 len 28672
26965 012/300 Write block 1963 len 16384 data: 5 5 5 4
26966 013/300 Read block 3563 len 36864
26967 014/300 Read block 4621 len 40960
26968 015/300 Write block 497 len 4096 data: 4
26969 016/300 Write block 1525 len 36864 data: 7 6 7 6 5 6 6 6 7
26970 017/300 Read block 2070 len 24576
26971 018/300 Read block 715 len 4096
26972 019/300 Read block 188 len 16384
26973 020/300 Read block 1018 len 28672
26974 021/300 Read block 3963 len 24576
26975 022/300 Read block 3998 len 40960
26976 023/300 Read block 977 len 20480
26977 024/300 Flush
26978 025/300 Write block 3835 len 12288 data: 4 4 3
26979 026/300 Write block 414 len 28672 data: 3 3 4 5 4 4 6
26980 027/300 Read block 3958 len 8192
26981 028/300 Read block 4970 len 40960
26982 029/300 Read block 1893 len 40960
26983 030/300 Write block 2137 len 40960 data: 7 8 5 4 3 5 6 7 6 7
26984 031/300 Write block 2267 len 8192 data: 4 3
26985 032/300 Read block 73 len 12288
26986 033/300 Write block 74 len 12288 data: 8 7 7
26987 034/300 Read block 2701 len 36864
26988 035/300 Flush
26989 036/300 Read block 479 len 40960
26990 037/300 Read block 3861 len 36864
26991 038/300 Read block 627 len 36864
26992 039/300 Read block 4234 len 28672
26993 040/300 Write block 4029 len 20480 data: 6 6 5 5 5
26994 041/300 Read block 2616 len 36864
26995 042/300 Flush
26996 043/300 Write block 2204 len 28672 data: 7 6 6 5 5 4 4
26997 044/300 Write block 565 len 8192 data: 4 4
26998 045/300 Write block 762 len 8192 data: 6 6
26999 046/300 Write block 3171 len 24576 data: 4 4 4 4 4 5
27000 047/300 Write block 1821 len 36864 data: 5 2 2 5 7 7 8 8 8
27001 048/300 Flush
27002 049/300 Flush
27003 050/300 Flush
27004 051/300 Read block 3884 len 28672
27005 052/300 Read block 102 len 24576
27006 053/300 Write block 790 len 40960 data: 4 4 4 4 4 3 3 2 3 3
27007 054/300 Read block 485 len 4096
27008 055/300 Write block 2603 len 36864 data: 6 6 5 4 4 3 3 3 4
27009 056/300 Read block 3101 len 36864
27010 057/300 Write block 511 len 40960 data: 5 5 4 5 4 4 4 5 4 4
27011 058/300 Read block 4432 len 4096
27012 059/300 Read block 2522 len 24576
27013 060/300 Flush
27014 061/300 Read block 1880 len 16384
27015 062/300 Write block 3165 len 28672 data: 3 3 3 5 5 5 5
27016 063/300 Read block 670 len 20480
27017 064/300 Read block 1745 len 4096
27018 065/300 Read block 3643 len 40960
27019 066/300 Read block 4362 len 24576
27020 067/300 Flush
27021 068/300 Read block 2057 len 12288
27022 069/300 Read block 1018 len 28672
27023 070/300 Read block 2364 len 36864
27024 071/300 Read block 4499 len 32768
27025 072/300 Write block 3876 len 20480 data: 6 6 7 7 7
27026 073/300 Write block 2446 len 20480 data: 5 5 5 4 6
27027 074/300 Read block 238 len 8192
27028 075/300 Read block 3341 len 16384
27029 076/300 Read block 1297 len 20480
27030 077/300 Read block 166 len 4096
27031 078/300 Write block 4671 len 4096 data: 6
27032 079/300 Write block 3208 len 4096 data: 3
27033 080/300 Read block 2083 len 36864
27034 081/300 Read block 2264 len 4096
27035 082/300 Read block 4273 len 20480
27036 083/300 Write block 2796 len 28672 data: 6 7 8 7 7 7 6
27037 084/300 Read block 917 len 20480
27038 085/300 Write block 3621 len 32768 data: 5 5 7 8 6 5 4 4
27039 086/300 Write block 4063 len 12288 data: 6 5 5
27040 087/300 Write block 4925 len 4096 data: 4
27041 088/300 Read block 3228 len 32768
27042 089/300 Flush
27043 090/300 Read block 1632 len 20480
27044 091/300 Read block 1478 len 8192
27045 092/300 Read block 3024 len 4096
27046 093/300 Read block 4959 len 16384
27047 094/300 Write block 4518 len 28672 data: 5 4 6 5 6 8 9
27048 095/300 Flush
27049 096/300 Read block 2011 len 40960
27050 097/300 Read block 1978 len 32768
27051 098/300 Read block 171 len 8192
27052 099/300 Write block 3839 len 16384 data: 4 4 3 4
27053 100/300 Flush
27054 101/300 Flush
27055 102/300 Write block 1810 len 16384 data: 6 5 5 3
27056 103/300 Read block 2185 len 16384
27057 104/300 Read block 4823 len 40960
27058 105/300 Read block 4852 len 28672
27059 106/300 Read block 1397 len 28672
27060 107/300 Read block 1792 len 40960
27061 108/300 Read block 4845 len 32768
27062 109/300 Write block 79 len 24576 data: 7 8 8 6 5 4
27063 110/300 Write block 4941 len 32768 data: 2 3 3 5 4 5 6 5
27064 111/300 Write block 3419 len 32768 data: 5 5 6 6 6 6 5 4
27065 112/300 Read block 4509 len 12288
27066 113/300 Write block 2409 len 36864 data: 7 5 6 5 6 6 5 7 6
27067 114/300 Write block 1480 len 36864 data: 4 6 7 6 6 6 6 6 7
27068 115/300 Write block 4145 len 40960 data: 5 4 4 5 4 4 4 4 5 5
27069 116/300 Read block 4856 len 36864
27070 117/300 Read block 2584 len 24576
27071 118/300 Write block 2009 len 36864 data: 5 6 7 4 4 3 2 3 3
27072 119/300 Write block 411 len 12288 data: 3 3 3
27073 120/300 Read block 1800 len 12288
27074 121/300 Read block 4431 len 28672
27075 122/300 Read block 538 len 40960
27076 123/300 Write block 3764 len 32768 data: 7 6 6 7 6 8 7 7
27077 124/300 Read block 4458 len 16384
27078 125/300 Write block 3762 len 8192 data: 7 7
27079 126/300 Flush
27080 127/300 Read block 2141 len 12288
27081 128/300 Write block 2506 len 12288 data: 4 6 6
27082 129/300 Write block 263 len 16384 data: 6 5 6 5
27083 130/300 Flush
27084 131/300 Read block 4088 len 16384
27085 132/300 Read block 2928 len 36864
27086 133/300 Read block 77 len 16384
27087 134/300 Read block 2257 len 36864
27088 135/300 Read block 2191 len 28672
27089 136/300 Write block 2466 len 12288 data: 4 6 5
27090 137/300 Read block 2812 len 12288
27091 138/300 Read block 775 len 24576
27092 139/300 Write block 264 len 28672 data: 6 7 6 5 5 4 5
27093 140/300 Write block 1516 len 8192 data: 4 7
27094 141/300 Read block 4854 len 24576
27095 142/300 Read block 4303 len 40960
27096 143/300 Read block 3980 len 16384
27097 144/300 Flush
27098 145/300 Read block 3582 len 28672
27099 146/300 Write block 1535 len 8192 data: 6 5
27100 147/300 Write block 4681 len 20480 data: 9 8 8 8 8
27101 148/300 Read block 3400 len 20480
27102 149/300 Write block 518 len 16384 data: 6 5 5 6
27103 150/300 Write block 141 len 16384 data: 4 5 5 5
27104 151/300 Flush
27105 152/300 Write block 429 len 4096 data: 3
27106 153/300 Write block 1703 len 36864 data: 9 9 9 8 7 8 6 6 6
27107 154/300 Write block 1818 len 24576 data: 4 5 5 6 3 3
27108 155/300 Write block 2404 len 12288 data: 6 5 4
27109 156/300 Write block 524 len 40960 data: 7 5 5 6 5 6 5 5 5 6
27110 157/300 Write block 2884 len 24576 data: 3 4 4 3 2 2
27111 158/300 Read block 2971 len 16384
27112 159/300 Read block 4186 len 8192
27113 160/300 Flush
27114 161/300 Read block 2266 len 28672
27115 162/300 Write block 3839 len 32768 data: 5 5 4 5 4 4 4 4
27116 163/300 Read block 4174 len 28672
27117 164/300 Read block 1730 len 40960
27118 165/300 Write block 2570 len 24576 data: 4 5 4 4 4 5
27119 166/300 Write block 2951 len 12288 data: 4 4 5
27120 167/300 Read block 3639 len 28672
27121 168/300 Flush
27122 169/300 Read block 4658 len 32768
27123 170/300 Write block 3760 len 4096 data: 9
27124 171/300 Read block 3150 len 8192
27125 172/300 Write block 2228 len 4096 data: 6
27126 173/300 Write block 1094 len 36864 data: 4 5 5 5 4 4 6 6 4
27127 174/300 Read block 2647 len 8192
27128 175/300 Write block 994 len 12288 data: 4 4 4
27129 176/300 Write block 1190 len 20480 data: 4 4 4 5 5
27130 177/300 Write block 4220 len 16384 data: 3 3 3 3
27131 178/300 Read block 108 len 4096
27132 179/300 Write block 1014 len 32768 data: 11 10 10 8 8 6 5 8
27133 180/300 Write block 3791 len 20480 data: 4 4 5 5 5
27134 181/300 Write block 843 len 12288 data: 4 4 4
27135 182/300 Read block 4067 len 32768
27136 183/300 Flush
27137 184/300 Read block 4 len 24576
27138 185/300 Read block 3560 len 4096
27139 186/300 Write block 30 len 12288 data: 4 4 4
27140 187/300 Read block 1091 len 32768
27141 188/300 Read block 4477 len 32768
27142 189/300 Read block 3146 len 40960
27143 190/300 Write block 1662 len 40960 data: 8 8 11 9 7 6 5 6 6 6
27144 191/300 Write block 1278 len 16384 data: 5 4 5 5
27145 192/300 Read block 1150 len 16384
27146 193/300 Read block 4490 len 8192
27147 194/300 Read block 4176 len 4096
27148 195/300 Read block 124 len 12288
27149 196/300 Read block 484 len 32768
27150 197/300 Read block 2812 len 12288
27151 198/300 Read block 48 len 32768
27152 199/300 Write block 3479 len 40960 data: 5 4 5 6 6 7 6 7 6 6
27153 200/300 Write block 3866 len 24576 data: 5 4 4 4 6 5
27154 201/300 Flush
27155 202/300 Write block 4968 len 40960 data: 5 6 6 5 4 4 4 4 4 4
27156 203/300 Write block 4995 len 16384 data: 3 2 2 3
27157 204/300 Write block 997 len 32768 data: 5 6 6 5 6 5 6 5
27158 205/300 Write block 4816 len 32768 data: 4 5 6 6 6 6 6 6
27159 206/300 Read block 1396 len 8192
27160 207/300 Write block 68 len 24576 data: 5 5 5 4 6 7
27161 208/300 Write block 2225 len 36864 data: 4 5 5 7 5 4 4 3 4
27162 209/300 Read block 4601 len 20480
27163 210/300 Read block 3588 len 16384
27164 211/300 Read block 890 len 40960
27165 212/300 Write block 2744 len 28672 data: 2 2 2 3 3 3 4
27166 213/300 Read block 398 len 28672
27167 214/300 Read block 3439 len 4096
27168 215/300 Write block 951 len 32768 data: 5 7 5 5 5 4 5 6
27169 216/300 Write block 2214 len 8192 data: 4 4
27170 217/300 Read block 1051 len 20480
27171 218/300 Write block 228 len 16384 data: 3 4 4 5
27172 219/300 Write block 1887 len 28672 data: 4 3 3 3 2 2 2
27173 220/300 Read block 4766 len 8192
27174 221/300 Read block 4923 len 40960
27175 222/300 Write block 432 len 28672 data: 2 2 3 6 6 5 4
27176 223/300 Write block 510 len 12288 data: 3 6 6
27177 224/300 Flush
27178 225/300 Write block 4866 len 16384 data: 7 6 5 5
27179 226/300 Read block 3824 len 36864
27180 227/300 Write block 3366 len 40960 data: 8 6 7 6 5 4 5 5 5 6
27181 228/300 Read block 3550 len 40960
27182 229/300 Read block 3130 len 28672
27183 230/300 Write block 2812 len 28672 data: 4 4 3 4 4 4 4
27184 231/300 Write block 1497 len 20480 data: 5 4 3 3 3
27185 232/300 Write block 2804 len 24576 data: 6 5 5 5 5 6
27186 233/300 Read block 1053 len 8192
27187 234/300 Write block 3750 len 12288 data: 5 6 5
27188 235/300 Read block 708 len 36864
27189 236/300 Read block 901 len 4096
27190 237/300 Write block 1074 len 4096 data: 3
27191 238/300 Read block 1697 len 36864
27192 239/300 Flush
27193 240/300 Read block 3754 len 20480
27194 241/300 Flush
27195 242/300 Read block 3643 len 32768
27196 243/300 Read block 1600 len 28672
27197 244/300 Read block 3653 len 28672
27198 245/300 Write block 1086 len 4096 data: 4
27199 246/300 Write block 4357 len 40960 data: 8 8 8 8 6 5 6 8 4 3
27200 247/300 Read block 3026 len 12288
27201 248/300 Read block 4772 len 28672
27202 249/300 Read block 4990 len 40960
27203 250/300 Flush
27204 251/300 Write block 1599 len 24576 data: 3 3 3 3 4 4
27205 252/300 Read block 1786 len 28672
27206 253/300 Read block 2330 len 40960
27207 254/300 Read block 603 len 24576
27208 255/300 Write block 3754 len 28672 data: 6 6 8 8 8 8 10
27209 256/300 Write block 1832 len 24576 data: 5 4 3 2 2 2
27210 257/300 Flush
27211 258/300 Read block 4804 len 28672
27212 259/300 Write block 4175 len 24576 data: 7 6 7 7 7 6
27213 260/300 Write block 4223 len 28672 data: 4 3 3 4 4 6 6
27214 261/300 Read block 2993 len 8192
27215 262/300 Read block 2608 len 28672
27216 263/300 Read block 1154 len 4096
27217 264/300 Read block 3802 len 12288
27218 265/300 Read block 1618 len 36864
27219 266/300 Read block 2664 len 4096
27220 267/300 Read block 1548 len 32768
27221 268/300 Write block 3046 len 28672 data: 6 6 8 8 10 7 7
27222 269/300 Flush
27223 270/300 Read block 1883 len 32768
27224 271/300 Read block 82 len 28672
27225 272/300 Write block 4888 len 32768 data: 4 4 5 5 5 5 4 5
27226 273/300 Write block 1076 len 28672 data: 5 4 5 6 7 7 6
27227 274/300 Read block 861 len 24576
27228 275/300 Read block 279 len 16384
27229 276/300 Flush
27230 277/300 Read block 876 len 20480
27231 278/300 Write block 4284 len 12288 data: 3 3 4
27232 279/300 Read block 659 len 36864
27233 280/300 Write block 2778 len 16384 data: 3 5 4 3
27234 281/300 Write block 3109 len 28672 data: 4 6 5 6 6 7 7
27235 282/300 Read block 688 len 16384
27236 283/300 Read block 4814 len 32768
27237 284/300 Read block 1113 len 32768
27238 285/300 Read block 2782 len 16384
27239 286/300 Write block 4292 len 4096 data: 5
27240 287/300 Read block 4581 len 12288
27241 288/300 Read block 4720 len 8192
27242 289/300 Read block 273 len 28672
27243 290/300 Write block 3345 len 32768 data: 4 3 2 2 4 3 3 4
27244 291/300 Read block 2646 len 12288
27245 292/300 Write block 2062 len 40960 data: 4 4 3 3 3 4 5 6 7 5
27246 293/300 Read block 486 len 36864
27247 294/300 Write block 3554 len 24576 data: 6 6 6 6 5 6
27248 295/300 Read block 4515 len 16384
27249 296/300 Flush
27250 297/300 Read block 4225 len 36864
27251 298/300 Read block 15 len 12288
27252 299/300 Read block 4761 len 36864
27253 300/300 Read block 4068 len 28672
27254 301/300 Read block 2643 len 12288
27255 Main task has work to do, go find it
27256 got dsc Start(1)
27257 start 1
27258 Jan 28 22:00:20.081 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/1, method: POST, req_id: ac986831-33a3-4f29-a906-6a24231f594f, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
27259 Replay: started 1, returned:Ok(())
27260 ----------------------------------------------------------------
27261 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27262 States: Active Offline Active
27263 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27264 8246 Acked Read 7 Done Sent Done false
27265 8247 Acked Read 3 Done Sent Done false
27266 8248 Acked Write 7 Done Sent Done false
27267 8249 Acked Write 10 Done Sent Done false
27268 8250 Acked Write 1 Done Sent Done false
27269 8251 Acked Write 9 Done Sent Done false
27270 8252 Acked Write 9 Done Sent Done false
27271 8253 Acked Write 9 Done Sent Done false
27272 8254 Acked Flush 0 Done Sent Done false
27273 8255 Acked Write 7 Done Sent Done false
27274 8256 Acked Read 7 Done Sent Done false
27275 8257 Acked Write 4 Done Sent Done false
27276 8258 Acked Read 9 Done Sent Done false
27277 8259 Acked Read 10 Done Sent Done false
27278 8260 Acked Write 1 Done Sent Done false
27279 8261 Acked Write 9 Done Sent Done false
27280 8262 Acked Read 6 Done Sent Done false
27281 8263 Acked Read 1 Done Sent Done false
27282 8264 Acked Read 4 Done Sent Done false
27283 8265 Acked Read 7 Done Sent Done false
27284 8266 Acked Read 6 Done Sent Done false
27285 8267 Acked Read 10 Done Sent Done false
27286 8268 Acked Read 5 Done Sent Done false
27287 8269 Acked Flush 0 Done Sent Done false
27288 8270 Acked Write 3 Done Sent Done false
27289 8271 Acked Write 7 Done Sent Done false
27290 8272 Acked Read 2 Done Sent Done false
27291 8273 Acked Read 10 Done Sent Done false
27292 8274 Acked Read 10 Done Sent Done false
27293 8275 Acked Write 10 Done Sent Done false
27294 8276 Acked Write 2 Done Sent Done false
27295 8277 Acked Read 3 Done Sent Done false
27296 8278 Acked Write 3 Done Sent Done false
27297 8279 Acked Read 9 Done Sent Done false
27298 8280 Acked Flush 0 Done Sent Done false
27299 8281 Acked Read 10 Done Sent Done false
27300 8282 Acked Read 9 Done Sent Done false
27301 8283 Acked Read 9 Done Sent Done false
27302 8284 Acked Read 7 Done Sent Done false
27303 8285 Acked Write 5 Done Sent Done false
27304 8286 Acked Read 9 Done Sent Done false
27305 8287 Acked Flush 0 Done Sent Done false
27306 8288 Acked Write 7 Done Sent Done false
27307 8289 Acked Write 2 Done Sent Done false
27308 8290 Acked Write 2 Done Sent Done false
27309 8291 Acked Write 6 Done Sent Done false
27310 8292 Acked Write 9 Done Sent Done false
27311 8293 Acked Flush 0 Done Sent Done false
27312 8294 Acked Flush 0 Done Sent Done false
27313 8295 Acked Flush 0 Done Sent Done false
27314 8296 Acked Read 7 Done Sent Done false
27315 8297 Acked Read 6 Done Sent Done false
27316 8298 Acked Write 10 Done Sent Done false
27317 8299 Acked Read 1 Done Sent Done false
27318 8300 Acked Write 9 Done Sent Done false
27319 8301 Acked Read 9 Done Sent Done false
27320 8302 Acked Write 10 Done Sent Done false
27321 8303 Acked Read 1 Done Sent Done false
27322 8304 Acked Read 6 Done Sent Done false
27323 8305 Acked Flush 0 Done Sent Done false
27324 8306 Acked Read 4 Done Sent Done false
27325 8307 Acked Write 7 Done Sent Done false
27326 8308 Acked Read 5 Done Sent Done false
27327 8309 Acked Read 1 Done Sent Done false
27328 8310 Acked Read 10 Done Sent Done false
27329 8311 Acked Read 6 Done Sent Done false
27330 8312 Acked Flush 0 Done Sent Done false
27331 8313 Acked Read 3 Done Sent Done false
27332 8314 Acked Read 7 Done Sent Done false
27333 8315 Acked Read 9 Done Sent Done false
27334 8316 Acked Read 8 Done Sent Done false
27335 8317 Acked Write 5 Done Sent Done false
27336 8318 Acked Write 5 Done Sent Done false
27337 8319 Acked Read 2 Done Sent Done false
27338 8320 Acked Read 4 Done Sent Done false
27339 8321 Acked Read 5 Done Sent Done false
27340 8322 Acked Read 1 Done Sent Done false
27341 8323 Acked Write 1 Done Sent Done false
27342 8324 Acked Write 1 Done Sent Done false
27343 8325 Acked Read 9 Done Sent Done false
27344 8326 Acked Read 1 Done Sent Done false
27345 8327 Acked Read 5 Done Sent Done false
27346 8328 Acked Write 7 Done Sent Done false
27347 8329 Acked Read 5 Done Sent Done false
27348 8330 Acked Write 8 Done Sent Done false
27349 8331 Acked Write 3 Done Sent Done false
27350 8332 Acked Write 1 Done Sent Done false
27351 8333 Acked Read 8 Done Sent Done false
27352 8334 Acked Flush 0 Done Sent Done false
27353 8335 Acked Read 5 Done Sent Done false
27354 8336 Acked Read 2 Done Sent Done false
27355 8337 Acked Read 1 Done Sent Done false
27356 8338 Acked Read 4 Done Sent Done false
27357 8339 Acked Write 7 Done Sent Done false
27358 8340 Acked Flush 0 Done Sent Done false
27359 8341 Acked Read 10 Done Sent Done false
27360 8342 Acked Read 8 Done Sent Done false
27361 8343 Acked Read 2 Done Sent Done false
27362 8344 Acked Write 4 Done Sent Done false
27363 8345 Acked Flush 0 Done Sent Done false
27364 8346 Acked Flush 0 Done Sent Done false
27365 8347 Acked Write 4 Done Sent Done false
27366 8348 Acked Read 4 Done Sent Done false
27367 8349 Acked Read 10 Done Sent Done false
27368 8350 Acked Read 7 Done Sent Done false
27369 8351 Acked Read 7 Done Sent Done false
27370 8352 Acked Read 10 Done Sent Done false
27371 8353 Acked Read 8 Done Sent Done false
27372 8354 Acked Write 6 Done Sent Done false
27373 8355 Acked Write 8 Done Sent Done false
27374 8356 Acked Write 8 Done Sent Done false
27375 8357 Acked Read 3 Done Sent Done false
27376 8358 Acked Write 9 Done Sent Done false
27377 8359 Acked Write 9 Done Sent Done false
27378 8360 Acked Write 10 Done Sent Done false
27379 8361 Acked Read 9 Done Sent Done false
27380 8362 Acked Read 6 Done Sent Done false
27381 8363 Acked Write 9 Done Sent Done false
27382 8364 Acked Write 3 Done Sent Done false
27383 8365 Acked Read 3 Done Sent Done false
27384 8366 Acked Read 7 Done Sent Done false
27385 8367 Acked Read 10 Done Sent Done false
27386 8368 Acked Write 8 Done Sent Done false
27387 8369 Acked Read 4 Done Sent Done false
27388 8370 Acked Write 2 Done Sent Done false
27389 8371 Acked Flush 0 Done Sent Done false
27390 8372 Acked Read 3 Done Sent Done false
27391 8373 Acked Write 3 Done Sent Done false
27392 8374 Acked Write 4 Done Sent Done false
27393 8375 Acked Flush 0 Done Sent Done false
27394 8376 Acked Read 4 Done Sent Done false
27395 8377 Acked Read 9 Done Sent Done false
27396 8378 Acked Read 4 Done Sent Done false
27397 8379 Acked Read 9 Done Sent Done false
27398 8380 Acked Read 7 Done Sent Done false
27399 8381 Acked Write 3 Done Sent Done false
27400 8382 Acked Read 3 Done Sent Done false
27401 8383 Acked Read 6 Done Sent Done false
27402 8384 Acked Write 7 Done Sent Done false
27403 8385 Acked Write 2 Done Sent Done false
27404 8386 Acked Read 6 Done Sent Done false
27405 8387 Acked Read 10 Done Sent Done false
27406 8388 Acked Read 4 Done Sent Done false
27407 8389 Acked Flush 0 Done Sent Done false
27408 8390 Acked Read 7 Done Sent Done false
27409 8391 Acked Write 2 Done Sent Done false
27410 8392 Acked Write 5 Done Sent Done false
27411 8393 Acked Read 5 Done Sent Done false
27412 8394 Acked Write 4 Done Sent Done false
27413 8395 Acked Write 4 Done Sent Done false
27414 8396 Acked Flush 0 Done Sent Done false
27415 8397 Acked Write 1 Done Sent Done false
27416 8398 Acked Write 9 Done Sent Done false
27417 8399 Acked Write 6 Done Sent Done false
27418 8400 Acked Write 3 Done Sent Done false
27419 8401 Acked Write 10 Done Sent Done false
27420 8402 Acked Write 6 Done Sent Done false
27421 8403 Acked Read 4 Done Sent Done false
27422 8404 Acked Read 2 Done Sent Done false
27423 8405 Acked Flush 0 Done Sent Done false
27424 8406 Acked Read 7 Done Sent Done false
27425 8407 Acked Write 8 Done Sent Done false
27426 8408 Acked Read 7 Done Sent Done false
27427 8409 Acked Read 10 Done Sent Done false
27428 8410 Acked Write 6 Done Sent Done false
27429 8411 Acked Write 3 Done Sent Done false
27430 8412 Acked Read 7 Done Sent Done false
27431 8413 Acked Flush 0 Done Sent Done false
27432 8414 Acked Read 8 Done Sent Done false
27433 8415 Acked Write 1 Done Sent Done false
27434 8416 Acked Read 2 Done Sent Done false
27435 8417 Acked Write 1 Done Sent Done false
27436 8418 Acked Write 9 Done Sent Done false
27437 8419 Acked Read 2 Done Sent Done false
27438 8420 Acked Write 3 Done Sent Done false
27439 8421 Acked Write 5 Done Sent Done false
27440 8422 Acked Write 4 Done Sent Done false
27441 8423 Acked Read 1 Done Sent Done false
27442 8424 Acked Write 8 Done Sent Done false
27443 8425 Acked Write 5 Done Sent Done false
27444 8426 Acked Write 3 Done Sent Done false
27445 8427 Acked Read 8 Done Sent Done false
27446 8428 Acked Flush 0 Done Sent Done false
27447 8429 Acked Read 6 Done Sent Done false
27448 8430 Acked Read 1 Done Sent Done false
27449 8431 Acked Write 3 Done Sent Done false
27450 8432 Acked Read 8 Done Sent Done false
27451 8433 Acked Read 8 Done Sent Done false
27452 8434 Acked Read 10 Done Sent Done false
27453 8435 Acked Write 10 Done Sent Done false
27454 8436 Acked Write 4 Done Sent Done false
27455 8437 Acked Read 4 Done Sent Done false
27456 8438 Acked Read 2 Done Sent Done false
27457 8439 Acked Read 1 Done Sent Done false
27458 8440 Acked Read 3 Done Sent Done false
27459 8441 Acked Read 8 Done Sent Done false
27460 8442 Acked Read 3 Done Sent Done false
27461 8443 Acked Read 8 Done Sent Done false
27462 8444 Acked Write 10 Done Sent Done false
27463 8445 Acked Write 6 Done Sent Done false
27464 8446 Acked Flush 0 Done Sent Done false
27465 8447 Acked Write 10 Done Sent Done false
27466 8448 Acked Write 4 Done Sent Done false
27467 8449 Acked Write 8 Done Sent Done false
27468 8450 Acked Write 8 Done Sent Done false
27469 8451 Acked Read 2 Done Sent Done false
27470 8452 Acked Write 6 Done Sent Done false
27471 8453 Acked Write 9 Done Sent Done false
27472 8454 Acked Read 5 Done Sent Done false
27473 8455 Acked Read 4 Done Sent Done false
27474 8456 Acked Read 10 Done Sent Done false
27475 8457 Acked Write 7 Done Sent Done false
27476 8458 Acked Read 7 Done Sent Done false
27477 8459 Acked Read 1 Done Sent Done false
27478 8460 Acked Write 8 Done Sent Done false
27479 8461 Acked Write 2 Done Sent Done false
27480 8462 Acked Read 5 Done Sent Done false
27481 8463 Acked Write 4 Done Sent Done false
27482 8464 Acked Write 7 Done Sent Done false
27483 8465 Acked Read 2 Done Sent Done false
27484 8466 Acked Read 10 Done Sent Done false
27485 8467 Acked Write 7 Done Sent Done false
27486 8468 Acked Write 3 Done Sent Done false
27487 8469 Acked Flush 0 Done Sent Done false
27488 8470 Acked Write 4 Done Sent Done false
27489 8471 Acked Read 9 Done Sent Done false
27490 8472 Acked Write 10 Done Sent Done false
27491 8473 Acked Read 10 Done Sent Done false
27492 8474 Acked Read 7 Done Sent Done false
27493 8475 Acked Write 7 Done Sent Done false
27494 8476 Acked Write 5 Done Sent Done false
27495 8477 Acked Write 6 Done Sent Done false
27496 8478 Acked Read 2 Done Sent Done false
27497 8479 Acked Write 3 Done Sent Done false
27498 8480 Acked Read 9 Done Sent Done false
27499 8481 Acked Read 1 Done Sent Done false
27500 8482 Acked Write 1 Done Sent Done false
27501 8483 Acked Read 9 Done Sent Done false
27502 8484 Acked Flush 0 Done Sent Done false
27503 8485 Acked Read 5 Done Sent Done false
27504 8486 Acked Flush 0 Done Sent Done false
27505 8487 Acked Read 8 Done Sent Done false
27506 8488 Acked Read 7 Done Sent Done false
27507 8489 Acked Read 7 Done Sent Done false
27508 8490 Acked Write 1 Done Sent Done false
27509 8491 Acked Write 10 Done Sent Done false
27510 8492 Acked Read 3 Done Sent Done false
27511 8493 Acked Read 7 Done Sent Done false
27512 8494 Acked Read 10 Done Sent Done false
27513 8495 Acked Flush 0 Done Sent Done false
27514 8496 Acked Write 6 Done Sent Done false
27515 8497 Acked Read 7 Done Sent Done false
27516 8498 Acked Read 10 Done Sent Done false
27517 8499 Acked Read 6 Done Sent Done false
27518 8500 Acked Write 7 Done Sent Done false
27519 8501 Acked Write 6 Done Sent Done false
27520 8502 Acked Flush 0 Done Sent Done false
27521 8503 Acked Read 7 Done Sent Done false
27522 8504 Acked Write 6 Done Sent Done false
27523 8505 Acked Write 7 Done Sent Done false
27524 8506 Acked Read 2 Done Sent Done false
27525 8507 Acked Read 7 Done Sent Done false
27526 8508 Acked Read 1 Done Sent Done false
27527 8509 Acked Read 3 Done Sent Done false
27528 8510 Acked Read 9 Done Sent Done false
27529 8511 Acked Read 1 Done Sent Done false
27530 8512 Acked Read 8 Done Sent Done false
27531 8513 Acked Write 7 Done Sent Done false
27532 8514 Acked Flush 0 Done Sent Done false
27533 8515 Acked Read 8 Done Sent Done false
27534 8516 Acked Read 7 Done Sent Done false
27535 8517 Acked Write 8 Done Sent Done false
27536 8518 Acked Write 7 Done Sent Done false
27537 8519 Acked Read 6 Done Sent Done false
27538 8520 Acked Read 4 Done Sent Done false
27539 8521 Acked Flush 0 Done Sent Done false
27540 8522 Acked Read 5 Done Sent Done false
27541 8523 Acked Write 3 Done Sent Done false
27542 8524 Acked Read 9 Done Sent Done false
27543 8525 Acked Write 4 Done Sent Done false
27544 8526 Acked Write 7 Done Sent Done false
27545 8527 Acked Read 4 Done Sent Done false
27546 8528 Acked Read 8 Done Sent Done false
27547 8529 Acked Read 8 Done Sent Done false
27548 8530 Acked Read 4 Done Sent Done false
27549 8531 Acked Write 1 Done Sent Done false
27550 8532 Acked Read 3 Done Sent Done false
27551 8533 Acked Read 2 Done Sent Done false
27552 8534 Acked Read 7 Done Sent Done false
27553 8535 Acked Write 8 Done Sent Done false
27554 8536 Acked Read 3 Done Sent Done false
27555 8537 Acked Write 10 Done Sent Done false
27556 8538 Acked Read 9 Done Sent Done false
27557 8539 Acked Write 6 Done Sent Done false
27558 8540 Acked Read 4 Done Sent Done false
27559 8541 Acked Flush 0 Done Sent Done false
27560 8542 Acked Read 9 Done Sent Done false
27561 8543 Acked Read 3 Done Sent Done false
27562 8544 Acked Read 9 Done Sent Done false
27563 8545 Acked Read 7 Done Sent Done false
27564 8546 Acked Read 3 Done Sent Done false
27565 STATES DS:0 DS:1 DS:2 TOTAL
27566 Sent 0 301 0 301
27567 Done 301 0 301 602
27568 Skipped 0 0 0 0
27569 Error 0 0 0 0
27570 Last Flush: Some(JobId(8541)) Some(JobId(8245)) Some(JobId(8541))
27571 Downstairs last five completed: 8245 8244 8243 8242 8241
27572 Upstairs last five completed: 8546 8545 8544 8543 8542
27573 CLIENT: Up:0 ds:301 act:2
275742025-01-28T22:00:20.866ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27575 [8820] got start action
27576 [8820] I am going to restart
27577 Starting downstairs at port 8820
27578 Make output file at "/tmp/dsc/downstairs-8820.txt"
27579 [8820][1] reports Starting
27580 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1227)
27581 [8820][1] reports Running
275822025-01-28T22:00:22.075ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
275832025-01-28T22:00:22.869ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
275842025-01-28T22:00:23.871ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27585 ----------------------------------------------------------------
27586 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27587 States: Active Offline Active
27588 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27589 8246 Acked Read 7 Done Sent Done false
27590 8247 Acked Read 3 Done Sent Done false
27591 8248 Acked Write 7 Done Sent Done false
27592 8249 Acked Write 10 Done Sent Done false
27593 8250 Acked Write 1 Done Sent Done false
27594 8251 Acked Write 9 Done Sent Done false
27595 8252 Acked Write 9 Done Sent Done false
27596 8253 Acked Write 9 Done Sent Done false
27597 8254 Acked Flush 0 Done Sent Done false
27598 8255 Acked Write 7 Done Sent Done false
27599 8256 Acked Read 7 Done Sent Done false
27600 8257 Acked Write 4 Done Sent Done false
27601 8258 Acked Read 9 Done Sent Done false
27602 8259 Acked Read 10 Done Sent Done false
27603 8260 Acked Write 1 Done Sent Done false
27604 8261 Acked Write 9 Done Sent Done false
27605 8262 Acked Read 6 Done Sent Done false
27606 8263 Acked Read 1 Done Sent Done false
27607 8264 Acked Read 4 Done Sent Done false
27608 8265 Acked Read 7 Done Sent Done false
27609 8266 Acked Read 6 Done Sent Done false
27610 8267 Acked Read 10 Done Sent Done false
27611 8268 Acked Read 5 Done Sent Done false
27612 8269 Acked Flush 0 Done Sent Done false
27613 8270 Acked Write 3 Done Sent Done false
27614 8271 Acked Write 7 Done Sent Done false
27615 8272 Acked Read 2 Done Sent Done false
27616 8273 Acked Read 10 Done Sent Done false
27617 8274 Acked Read 10 Done Sent Done false
27618 8275 Acked Write 10 Done Sent Done false
27619 8276 Acked Write 2 Done Sent Done false
27620 8277 Acked Read 3 Done Sent Done false
27621 8278 Acked Write 3 Done Sent Done false
27622 8279 Acked Read 9 Done Sent Done false
27623 8280 Acked Flush 0 Done Sent Done false
27624 8281 Acked Read 10 Done Sent Done false
27625 8282 Acked Read 9 Done Sent Done false
27626 8283 Acked Read 9 Done Sent Done false
27627 8284 Acked Read 7 Done Sent Done false
27628 8285 Acked Write 5 Done Sent Done false
27629 8286 Acked Read 9 Done Sent Done false
27630 8287 Acked Flush 0 Done Sent Done false
27631 8288 Acked Write 7 Done Sent Done false
27632 8289 Acked Write 2 Done Sent Done false
27633 8290 Acked Write 2 Done Sent Done false
27634 8291 Acked Write 6 Done Sent Done false
27635 8292 Acked Write 9 Done Sent Done false
27636 8293 Acked Flush 0 Done Sent Done false
27637 8294 Acked Flush 0 Done Sent Done false
27638 8295 Acked Flush 0 Done Sent Done false
27639 8296 Acked Read 7 Done Sent Done false
27640 8297 Acked Read 6 Done Sent Done false
27641 8298 Acked Write 10 Done Sent Done false
27642 8299 Acked Read 1 Done Sent Done false
27643 8300 Acked Write 9 Done Sent Done false
27644 8301 Acked Read 9 Done Sent Done false
27645 8302 Acked Write 10 Done Sent Done false
27646 8303 Acked Read 1 Done Sent Done false
27647 8304 Acked Read 6 Done Sent Done false
27648 8305 Acked Flush 0 Done Sent Done false
27649 8306 Acked Read 4 Done Sent Done false
27650 8307 Acked Write 7 Done Sent Done false
27651 8308 Acked Read 5 Done Sent Done false
27652 8309 Acked Read 1 Done Sent Done false
27653 8310 Acked Read 10 Done Sent Done false
27654 8311 Acked Read 6 Done Sent Done false
27655 8312 Acked Flush 0 Done Sent Done false
27656 8313 Acked Read 3 Done Sent Done false
27657 8314 Acked Read 7 Done Sent Done false
27658 8315 Acked Read 9 Done Sent Done false
27659 8316 Acked Read 8 Done Sent Done false
27660 8317 Acked Write 5 Done Sent Done false
27661 8318 Acked Write 5 Done Sent Done false
27662 8319 Acked Read 2 Done Sent Done false
27663 8320 Acked Read 4 Done Sent Done false
27664 8321 Acked Read 5 Done Sent Done false
27665 8322 Acked Read 1 Done Sent Done false
27666 8323 Acked Write 1 Done Sent Done false
27667 8324 Acked Write 1 Done Sent Done false
27668 8325 Acked Read 9 Done Sent Done false
27669 8326 Acked Read 1 Done Sent Done false
27670 8327 Acked Read 5 Done Sent Done false
27671 8328 Acked Write 7 Done Sent Done false
27672 8329 Acked Read 5 Done Sent Done false
27673 8330 Acked Write 8 Done Sent Done false
27674 8331 Acked Write 3 Done Sent Done false
27675 8332 Acked Write 1 Done Sent Done false
27676 8333 Acked Read 8 Done Sent Done false
27677 8334 Acked Flush 0 Done Sent Done false
27678 8335 Acked Read 5 Done Sent Done false
27679 8336 Acked Read 2 Done Sent Done false
27680 8337 Acked Read 1 Done Sent Done false
27681 8338 Acked Read 4 Done Sent Done false
27682 8339 Acked Write 7 Done Sent Done false
27683 8340 Acked Flush 0 Done Sent Done false
27684 8341 Acked Read 10 Done Sent Done false
27685 8342 Acked Read 8 Done Sent Done false
27686 8343 Acked Read 2 Done Sent Done false
27687 8344 Acked Write 4 Done Sent Done false
27688 8345 Acked Flush 0 Done Sent Done false
27689 8346 Acked Flush 0 Done Sent Done false
27690 8347 Acked Write 4 Done Sent Done false
27691 8348 Acked Read 4 Done Sent Done false
27692 8349 Acked Read 10 Done Sent Done false
27693 8350 Acked Read 7 Done Sent Done false
27694 8351 Acked Read 7 Done Sent Done false
27695 8352 Acked Read 10 Done Sent Done false
27696 8353 Acked Read 8 Done Sent Done false
27697 8354 Acked Write 6 Done Sent Done false
27698 8355 Acked Write 8 Done Sent Done false
27699 8356 Acked Write 8 Done Sent Done false
27700 8357 Acked Read 3 Done Sent Done false
27701 8358 Acked Write 9 Done Sent Done false
27702 8359 Acked Write 9 Done Sent Done false
27703 8360 Acked Write 10 Done Sent Done false
27704 8361 Acked Read 9 Done Sent Done false
27705 8362 Acked Read 6 Done Sent Done false
27706 8363 Acked Write 9 Done Sent Done false
27707 8364 Acked Write 3 Done Sent Done false
27708 8365 Acked Read 3 Done Sent Done false
27709 8366 Acked Read 7 Done Sent Done false
27710 8367 Acked Read 10 Done Sent Done false
27711 8368 Acked Write 8 Done Sent Done false
27712 8369 Acked Read 4 Done Sent Done false
27713 8370 Acked Write 2 Done Sent Done false
27714 8371 Acked Flush 0 Done Sent Done false
27715 8372 Acked Read 3 Done Sent Done false
27716 8373 Acked Write 3 Done Sent Done false
27717 8374 Acked Write 4 Done Sent Done false
27718 8375 Acked Flush 0 Done Sent Done false
27719 8376 Acked Read 4 Done Sent Done false
27720 8377 Acked Read 9 Done Sent Done false
27721 8378 Acked Read 4 Done Sent Done false
27722 8379 Acked Read 9 Done Sent Done false
27723 8380 Acked Read 7 Done Sent Done false
27724 8381 Acked Write 3 Done Sent Done false
27725 8382 Acked Read 3 Done Sent Done false
27726 8383 Acked Read 6 Done Sent Done false
27727 8384 Acked Write 7 Done Sent Done false
27728 8385 Acked Write 2 Done Sent Done false
27729 8386 Acked Read 6 Done Sent Done false
27730 8387 Acked Read 10 Done Sent Done false
27731 8388 Acked Read 4 Done Sent Done false
27732 8389 Acked Flush 0 Done Sent Done false
27733 8390 Acked Read 7 Done Sent Done false
27734 8391 Acked Write 2 Done Sent Done false
27735 8392 Acked Write 5 Done Sent Done false
27736 8393 Acked Read 5 Done Sent Done false
27737 8394 Acked Write 4 Done Sent Done false
27738 8395 Acked Write 4 Done Sent Done false
27739 8396 Acked Flush 0 Done Sent Done false
27740 8397 Acked Write 1 Done Sent Done false
27741 8398 Acked Write 9 Done Sent Done false
27742 8399 Acked Write 6 Done Sent Done false
27743 8400 Acked Write 3 Done Sent Done false
27744 8401 Acked Write 10 Done Sent Done false
27745 8402 Acked Write 6 Done Sent Done false
27746 8403 Acked Read 4 Done Sent Done false
27747 8404 Acked Read 2 Done Sent Done false
27748 8405 Acked Flush 0 Done Sent Done false
27749 8406 Acked Read 7 Done Sent Done false
27750 8407 Acked Write 8 Done Sent Done false
27751 8408 Acked Read 7 Done Sent Done false
27752 8409 Acked Read 10 Done Sent Done false
27753 8410 Acked Write 6 Done Sent Done false
27754 8411 Acked Write 3 Done Sent Done false
27755 8412 Acked Read 7 Done Sent Done false
27756 8413 Acked Flush 0 Done Sent Done false
27757 8414 Acked Read 8 Done Sent Done false
27758 8415 Acked Write 1 Done Sent Done false
27759 8416 Acked Read 2 Done Sent Done false
27760 8417 Acked Write 1 Done Sent Done false
27761 8418 Acked Write 9 Done Sent Done false
27762 8419 Acked Read 2 Done Sent Done false
27763 8420 Acked Write 3 Done Sent Done false
27764 8421 Acked Write 5 Done Sent Done false
27765 8422 Acked Write 4 Done Sent Done false
27766 8423 Acked Read 1 Done Sent Done false
27767 8424 Acked Write 8 Done Sent Done false
27768 8425 Acked Write 5 Done Sent Done false
27769 8426 Acked Write 3 Done Sent Done false
27770 8427 Acked Read 8 Done Sent Done false
27771 8428 Acked Flush 0 Done Sent Done false
27772 8429 Acked Read 6 Done Sent Done false
27773 8430 Acked Read 1 Done Sent Done false
27774 8431 Acked Write 3 Done Sent Done false
27775 8432 Acked Read 8 Done Sent Done false
27776 8433 Acked Read 8 Done Sent Done false
27777 8434 Acked Read 10 Done Sent Done false
27778 8435 Acked Write 10 Done Sent Done false
27779 8436 Acked Write 4 Done Sent Done false
27780 8437 Acked Read 4 Done Sent Done false
27781 8438 Acked Read 2 Done Sent Done false
27782 8439 Acked Read 1 Done Sent Done false
27783 8440 Acked Read 3 Done Sent Done false
27784 8441 Acked Read 8 Done Sent Done false
27785 8442 Acked Read 3 Done Sent Done false
27786 8443 Acked Read 8 Done Sent Done false
27787 8444 Acked Write 10 Done Sent Done false
27788 8445 Acked Write 6 Done Sent Done false
27789 8446 Acked Flush 0 Done Sent Done false
27790 8447 Acked Write 10 Done Sent Done false
27791 8448 Acked Write 4 Done Sent Done false
27792 8449 Acked Write 8 Done Sent Done false
27793 8450 Acked Write 8 Done Sent Done false
27794 8451 Acked Read 2 Done Sent Done false
27795 8452 Acked Write 6 Done Sent Done false
27796 8453 Acked Write 9 Done Sent Done false
27797 8454 Acked Read 5 Done Sent Done false
27798 8455 Acked Read 4 Done Sent Done false
27799 8456 Acked Read 10 Done Sent Done false
27800 8457 Acked Write 7 Done Sent Done false
27801 8458 Acked Read 7 Done Sent Done false
27802 8459 Acked Read 1 Done Sent Done false
27803 8460 Acked Write 8 Done Sent Done false
27804 8461 Acked Write 2 Done Sent Done false
27805 8462 Acked Read 5 Done Sent Done false
27806 8463 Acked Write 4 Done Sent Done false
27807 8464 Acked Write 7 Done Sent Done false
27808 8465 Acked Read 2 Done Sent Done false
27809 8466 Acked Read 10 Done Sent Done false
27810 8467 Acked Write 7 Done Sent Done false
27811 8468 Acked Write 3 Done Sent Done false
27812 8469 Acked Flush 0 Done Sent Done false
27813 8470 Acked Write 4 Done Sent Done false
27814 8471 Acked Read 9 Done Sent Done false
27815 8472 Acked Write 10 Done Sent Done false
27816 8473 Acked Read 10 Done Sent Done false
27817 8474 Acked Read 7 Done Sent Done false
27818 8475 Acked Write 7 Done Sent Done false
27819 8476 Acked Write 5 Done Sent Done false
27820 8477 Acked Write 6 Done Sent Done false
27821 8478 Acked Read 2 Done Sent Done false
27822 8479 Acked Write 3 Done Sent Done false
27823 8480 Acked Read 9 Done Sent Done false
27824 8481 Acked Read 1 Done Sent Done false
27825 8482 Acked Write 1 Done Sent Done false
27826 8483 Acked Read 9 Done Sent Done false
27827 8484 Acked Flush 0 Done Sent Done false
27828 8485 Acked Read 5 Done Sent Done false
27829 8486 Acked Flush 0 Done Sent Done false
27830 8487 Acked Read 8 Done Sent Done false
27831 8488 Acked Read 7 Done Sent Done false
27832 8489 Acked Read 7 Done Sent Done false
27833 8490 Acked Write 1 Done Sent Done false
27834 8491 Acked Write 10 Done Sent Done false
27835 8492 Acked Read 3 Done Sent Done false
27836 8493 Acked Read 7 Done Sent Done false
27837 8494 Acked Read 10 Done Sent Done false
27838 8495 Acked Flush 0 Done Sent Done false
27839 8496 Acked Write 6 Done Sent Done false
27840 8497 Acked Read 7 Done Sent Done false
27841 8498 Acked Read 10 Done Sent Done false
27842 8499 Acked Read 6 Done Sent Done false
27843 8500 Acked Write 7 Done Sent Done false
27844 8501 Acked Write 6 Done Sent Done false
27845 8502 Acked Flush 0 Done Sent Done false
27846 8503 Acked Read 7 Done Sent Done false
27847 8504 Acked Write 6 Done Sent Done false
27848 8505 Acked Write 7 Done Sent Done false
27849 8506 Acked Read 2 Done Sent Done false
27850 8507 Acked Read 7 Done Sent Done false
27851 8508 Acked Read 1 Done Sent Done false
27852 8509 Acked Read 3 Done Sent Done false
27853 8510 Acked Read 9 Done Sent Done false
27854 8511 Acked Read 1 Done Sent Done false
27855 8512 Acked Read 8 Done Sent Done false
27856 8513 Acked Write 7 Done Sent Done false
27857 8514 Acked Flush 0 Done Sent Done false
27858 8515 Acked Read 8 Done Sent Done false
27859 8516 Acked Read 7 Done Sent Done false
27860 8517 Acked Write 8 Done Sent Done false
27861 8518 Acked Write 7 Done Sent Done false
27862 8519 Acked Read 6 Done Sent Done false
27863 8520 Acked Read 4 Done Sent Done false
27864 8521 Acked Flush 0 Done Sent Done false
27865 8522 Acked Read 5 Done Sent Done false
27866 8523 Acked Write 3 Done Sent Done false
27867 8524 Acked Read 9 Done Sent Done false
27868 8525 Acked Write 4 Done Sent Done false
27869 8526 Acked Write 7 Done Sent Done false
27870 8527 Acked Read 4 Done Sent Done false
27871 8528 Acked Read 8 Done Sent Done false
27872 8529 Acked Read 8 Done Sent Done false
27873 8530 Acked Read 4 Done Sent Done false
27874 8531 Acked Write 1 Done Sent Done false
27875 8532 Acked Read 3 Done Sent Done false
27876 8533 Acked Read 2 Done Sent Done false
27877 8534 Acked Read 7 Done Sent Done false
27878 8535 Acked Write 8 Done Sent Done false
27879 8536 Acked Read 3 Done Sent Done false
27880 8537 Acked Write 10 Done Sent Done false
27881 8538 Acked Read 9 Done Sent Done false
27882 8539 Acked Write 6 Done Sent Done false
27883 8540 Acked Read 4 Done Sent Done false
27884 8541 Acked Flush 0 Done Sent Done false
27885 8542 Acked Read 9 Done Sent Done false
27886 8543 Acked Read 3 Done Sent Done false
27887 8544 Acked Read 9 Done Sent Done false
27888 8545 Acked Read 7 Done Sent Done false
27889 8546 Acked Read 3 Done Sent Done false
27890 STATES DS:0 DS:1 DS:2 TOTAL
27891 Sent 0 301 0 301
27892 Done 301 0 301 602
27893 Skipped 0 0 0 0
27894 Error 0 0 0 0
27895 Last Flush: Some(JobId(8541)) Some(JobId(8245)) Some(JobId(8541))
27896 Downstairs last five completed: 8245 8244 8243 8242 8241
27897 Upstairs last five completed: 8546 8545 8544 8543 8542
27898 CLIENT: Up:0 ds:301 act:2
278992025-01-28T22:00:24.873ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
279002025-01-28T22:00:25.829ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
279012025-01-28T22:00:25.829ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63108) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
279022025-01-28T22:00:25.830ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
279032025-01-28T22:00:25.831ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
279042025-01-28T22:00:25.831ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
279052025-01-28T22:00:25.831ZINFOcrucible: send last flush ID to this DS: Some(JobId(8245)) = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
279062025-01-28T22:00:25.831ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
279072025-01-28T22:00:25.831ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(8245)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
279082025-01-28T22:00:25.831ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(8245)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27909 ----------------------------------------------------------------
27910 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27911 Downstairs last five completed: 8547 8546 8545 8544 8543
27912 Upstairs last five completed: 8547 8546 8545 8544 8543
27913 CLIENT: Up:0 ds:0 act:3
27914 Replay: All jobs finished
27915 Main task has work to do, go find it
27916 got dsc Stop(1)
27917 stop 1
27918 Jan 28 22:00:28.087 INFO request completed, latency_us: 381, response_code: 204[8820] Got stop action so:false kr:false
27919 , uri: /stop/cid/1, method: POST, req_id: 7b864d06-eba9-49b5-ad4d-cf4b7bec6874, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
27920 Jan 28 22:00:28.087 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-28T22:00:28.087935826Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
27921 INFO request completed, latency_us: 126, response_code: 200, uri: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-28T22:00:28.087997352Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
27922 /state/cid/1, method: GET, req_id: c11d4751-b89c-4e5b-a561-f100d8050ccd, remote_addr: {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-28T22:00:28.08802375Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
27923 127.0.0.1:36665, local_addr: 127.0.0.1:9998
279242025-01-28T22:00:28.088ZINFOcrucible: client task is exiting = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
27925 [8820] Exited with: ExitStatus(unix_wait_status(9))
27926 [8820][1] reports Exit
27927 Jan 28 22:00:32.089 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/1, method: GET, req_id: 484de531-ea74-4a93-ada3-a543b7800f06, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
27928 001/300 Read block 1895 len 28672
27929 002/300 Read block 671 len 16384
27930 003/300 Write block 1254 len 8192 data: 7 7
27931 004/300 Write block 1186 len 36864 data: 5 5 5 4 5 5 5 6 6
27932 005/300 Write block 3326 len 36864 data: 7 6 9 7 9 7 6 6 6
27933 006/300 Write block 4119 len 32768 data: 6 5 4 3 3 4 4 4
27934 007/300 Read block 1254 len 20480
27935 008/300 Read block 425 len 32768
27936 009/300 Write block 1358 len 12288 data: 5 6 5
27937 010/300 Read block 3419 len 28672
27938 011/300 Read block 299 len 28672
27939 012/300 Read block 4712 len 8192
27940 013/300 Read block 484 len 16384
27941 014/300 Flush
27942 015/300 Read block 950 len 20480
27943 016/300 Write block 1021 len 24576 data: 9 8 6 8 7 7
27944 017/300 Read block 1799 len 4096
27945 018/300 Write block 3160 len 32768 data: 5 5 4 4 4 4 4 4
27946 019/300 Write block 628 len 40960 data: 5 5 4 4 5 5 4 6 8 10
27947 020/300 Flush
27948 021/300 Write block 2290 len 16384 data: 7 7 6 7
27949 022/300 Read block 1527 len 40960
27950 023/300 Write block 3802 len 4096 data: 3
27951 024/300 Flush
27952 025/300 Write block 876 len 40960 data: 5 5 6 6 7 7 7 6 5 7
27953 026/300 Read block 4295 len 24576
27954 027/300 Read block 4528 len 12288
27955 028/300 Read block 98 len 8192
27956 029/300 Write block 809 len 32768 data: 3 4 4 5 5 5 5 5
27957 030/300 Read block 2763 len 40960
27958 031/300 Flush
27959 032/300 Read block 3771 len 4096
27960 033/300 Read block 759 len 16384
27961 034/300 Write block 4867 len 4096 data: 7
27962 035/300 Write block 1794 len 28672 data: 5 5 5 6 6 5 3
27963 036/300 Flush
27964 037/300 Flush
27965 038/300 Read block 4633 len 12288
27966 039/300 Read block 1672 len 16384
27967 040/300 Read block 265 len 20480
27968 041/300 Write block 543 len 20480 data: 4 4 6 7 9
27969 042/300 Flush
27970 043/300 Flush
27971 044/300 Write block 1166 len 4096 data: 3
27972 045/300 Read block 2738 len 8192
27973 046/300 Read block 4234 len 8192
27974 047/300 Read block 3275 len 24576
27975 048/300 Write block 2673 len 24576 data: 4 5 6 6 6 8
27976 049/300 Read block 2485 len 8192
27977 050/300 Read block 23 len 8192
27978 051/300 Read block 4199 len 36864
27979 052/300 Read block 3505 len 36864
27980 053/300 Read block 4182 len 16384
27981 054/300 Write block 1676 len 24576 data: 7 7 6 5 4 5
27982 055/300 Read block 4310 len 32768
27983 056/300 Read block 1417 len 8192
27984 057/300 Read block 1961 len 28672
27985 058/300 Write block 4617 len 24576 data: 7 8 8 8 8 8
27986 059/300 Write block 3664 len 4096 data: 7
27987 060/300 Write block 510 len 4096 data: 4
27988 061/300 Read block 1524 len 4096
27989 062/300 Read block 4981 len 40960
27990 063/300 Write block 2958 len 12288 data: 4 5 4
27991 064/300 Read block 1692 len 8192
27992 065/300 Write block 21 len 20480 data: 4 4 4 7 6
27993 066/300 Read block 872 len 32768
27994 067/300 Read block 4254 len 4096
27995 068/300 Flush
27996 069/300 Write block 1103 len 24576 data: 4 4 7 6 5 5
27997 070/300 Write block 1869 len 12288 data: 4 5 5
27998 071/300 Write block 875 len 28672 data: 6 6 6 7 7 8 8
27999 072/300 Flush
28000 073/300 Read block 1865 len 16384
28001 074/300 Write block 3296 len 12288 data: 5 7 7
28002 075/300 Read block 3020 len 24576
28003 076/300 Read block 1857 len 8192
28004 077/300 Write block 477 len 4096 data: 9
28005 078/300 Read block 3831 len 20480
28006 079/300 Read block 4637 len 32768
28007 080/300 Write block 374 len 8192 data: 5 6
28008 081/300 Write block 4740 len 16384 data: 7 9 7 6
28009 082/300 Read block 922 len 20480
28010 083/300 Read block 2025 len 12288
28011 084/300 Read block 2821 len 20480
28012 085/300 Write block 712 len 36864 data: 6 8 7 6 7 4 5 6 3
28013 086/300 Write block 2930 len 20480 data: 6 5 5 5 6
28014 087/300 Read block 4702 len 8192
28015 088/300 Flush
28016 089/300 Write block 2457 len 24576 data: 6 4 5 4 4 3
28017 090/300 Write block 1836 len 40960 data: 3 3 2 2 4 5 5 6 6 6
28018 091/300 Read block 566 len 24576
28019 092/300 Read block 237 len 28672
28020 093/300 Write block 614 len 40960 data: 3 3 5 5 4 4 5 4 3 4
28021 094/300 Write block 1416 len 16384 data: 5 4 3 3
28022 095/300 Read block 1955 len 8192
28023 096/300 Write block 4750 len 8192 data: 5 5
28024 097/300 Write block 3380 len 40960 data: 6 5 4 4 3 5 5 5 8 8
28025 098/300 Write block 2013 len 24576 data: 5 4 3 4 4 4
28026 099/300 Read block 875 len 4096
28027 100/300 Write block 4100 len 12288 data: 4 5 6
28028 101/300 Read block 4933 len 8192
28029 102/300 Read block 771 len 40960
28030 103/300 Write block 1438 len 20480 data: 6 5 4 3 4
28031 104/300 Read block 157 len 8192
28032 105/300 Read block 1239 len 32768
28033 106/300 Read block 1411 len 16384
28034 107/300 Read block 1383 len 28672
28035 108/300 Write block 3337 len 20480 data: 7 7 6 6 6
28036 109/300 Write block 1429 len 20480 data: 4 5 7 8 8
28037 110/300 Flush
28038 111/300 Read block 3114 len 36864
28039 112/300 Read block 2737 len 24576
28040 113/300 Write block 4437 len 24576 data: 4 4 3 3 3 3
28041 114/300 Write block 4127 len 8192 data: 5 4
28042 115/300 Write block 2883 len 16384 data: 4 4 5 5
28043 116/300 Flush
28044 117/300 Read block 2727 len 12288
28045 118/300 Flush
28046 119/300 Write block 802 len 36864 data: 3 3 2 2 3 2 3 4 5
28047 120/300 Write block 2080 len 8192 data: 6 8
28048 121/300 Read block 1364 len 8192
28049 122/300 Read block 4557 len 36864
28050 123/300 Read block 1344 len 4096
28051 124/300 Write block 3591 len 24576 data: 7 6 6 7 8 7
28052 125/300 Read block 4800 len 8192
28053 126/300 Read block 1909 len 28672
28054 127/300 Write block 2938 len 12288 data: 8 7 8
28055 128/300 Write block 613 len 28672 data: 3 4 4 6 6 5 5
28056 129/300 Read block 4355 len 40960
28057 130/300 Write block 351 len 28672 data: 8 5 6 6 6 8 6
28058 131/300 Write block 1874 len 36864 data: 3 3 4 4 4 5 5 5 5
28059 132/300 Write block 373 len 24576 data: 5 6 7 7 6 4
28060 133/300 Read block 1170 len 24576
28061 134/300 Write block 1822 len 16384 data: 4 4 6 8
28062 135/300 Write block 103 len 32768 data: 4 4 4 3 3 3 3 4
28063 136/300 Read block 2872 len 24576
28064 137/300 Write block 4766 len 16384 data: 7 7 6 8
28065 138/300 Read block 2050 len 24576
28066 139/300 Read block 3035 len 32768
28067 140/300 Read block 2581 len 40960
28068 141/300 Write block 2848 len 16384 data: 9 7 6 6
28069 142/300 Write block 4980 len 12288 data: 4 7 7
28070 143/300 Write block 109 len 28672 data: 4 5 4 4 4 6 4
28071 144/300 Read block 4524 len 24576
28072 145/300 Read block 505 len 40960
28073 146/300 Read block 1257 len 36864
28074 147/300 Write block 4941 len 8192 data: 3 4
28075 148/300 Read block 3109 len 20480
28076 149/300 Write block 2948 len 12288 data: 6 5 5
28077 150/300 Read block 4651 len 28672
28078 151/300 Write block 4676 len 12288 data: 5 5 7
28079 152/300 Flush
28080 153/300 Read block 763 len 40960
28081 154/300 Flush
28082 155/300 Write block 3182 len 20480 data: 6 6 5 5 4
28083 156/300 Flush
28084 157/300 Read block 3223 len 36864
28085 158/300 Write block 1041 len 8192 data: 7 7
28086 159/300 Read block 804 len 20480
28087 160/300 Read block 1137 len 4096
28088 161/300 Read block 1357 len 16384
28089 162/300 Read block 338 len 24576
28090 163/300 Write block 4341 len 20480 data: 6 6 4 4 7
28091 164/300 Write block 1796 len 36864 data: 6 7 7 6 4 3 2 3 3
28092 165/300 Read block 2341 len 32768
28093 166/300 Read block 2285 len 12288
28094 167/300 Write block 1818 len 16384 data: 5 6 6 7
28095 168/300 Write block 3851 len 16384 data: 3 3 3 2
28096 169/300 Flush
28097 170/300 Flush
28098 171/300 Read block 1244 len 20480
28099 172/300 Write block 4775 len 16384 data: 11 10 8 7
28100 173/300 Write block 1182 len 40960 data: 9 6 6 6 6 6 6 5 6 6
28101 174/300 Write block 1413 len 32768 data: 8 8 6 6 5 4 4 3
28102 175/300 Read block 669 len 16384
28103 176/300 Read block 4061 len 24576
28104 177/300 Write block 2962 len 24576 data: 4 4 4 5 5 7
28105 178/300 Read block 1695 len 8192
28106 179/300 Read block 1669 len 12288
28107 180/300 Write block 736 len 4096 data: 5
28108 181/300 Write block 1272 len 40960 data: 6 6 5 5 4 4 6 5 6 6
28109 182/300 Write block 2021 len 16384 data: 4 3 4 5
28110 183/300 Read block 2302 len 12288
28111 184/300 Flush
28112 185/300 Read block 1430 len 40960
28113 186/300 Write block 3199 len 28672 data: 5 6 6 6 5 4 3
28114 187/300 Read block 874 len 16384
28115 188/300 Write block 2561 len 4096 data: 6
28116 189/300 Read block 2577 len 16384
28117 190/300 Write block 2519 len 8192 data: 5 5
28118 191/300 Read block 898 len 24576
28119 192/300 Write block 4038 len 4096 data: 4
28120 193/300 Read block 1195 len 36864
28121 194/300 Read block 1014 len 36864
28122 195/300 Read block 1843 len 12288
28123 196/300 Read block 1379 len 36864
28124 197/300 Write block 4870 len 16384 data: 5 5 4 4
28125 198/300 Read block 4633 len 40960
28126 199/300 Read block 1611 len 24576
28127 200/300 Read block 4153 len 32768
28128 201/300 Read block 4450 len 40960
28129 202/300 Write block 1471 len 16384 data: 6 6 6 4
28130 203/300 Flush
28131 204/300 Read block 4172 len 4096
28132 205/300 Read block 4789 len 24576
28133 206/300 Read block 1978 len 8192
28134 207/300 Read block 143 len 4096
28135 208/300 Write block 4176 len 28672 data: 7 8 8 8 7 6 5
28136 209/300 Read block 3715 len 20480
28137 210/300 Read block 2050 len 12288
28138 211/300 Write block 3117 len 4096 data: 6
28139 212/300 Write block 1165 len 20480 data: 3 4 5 5 5
28140 213/300 Read block 996 len 40960
28141 214/300 Read block 270 len 40960
28142 215/300 Write block 1720 len 12288 data: 6 6 6
28143 216/300 Write block 906 len 24576 data: 8 7 6 4 3 4
28144 217/300 Write block 125 len 16384 data: 7 9 5 3
28145 218/300 Read block 2123 len 12288
28146 219/300 Write block 696 len 32768 data: 5 5 5 6 6 6 5 6
28147 220/300 Write block 2506 len 24576 data: 5 7 7 5 6 5
28148 221/300 Write block 2486 len 20480 data: 3 3 2 2 3
28149 222/300 Write block 1628 len 40960 data: 5 7 6 8 7 6 4 4 7 6
28150 223/300 Read block 3209 len 28672
28151 224/300 Write block 2155 len 8192 data: 10 8
28152 225/300 Read block 4235 len 32768
28153 226/300 Flush
28154 227/300 Read block 4683 len 24576
28155 228/300 Read block 4639 len 40960
28156 229/300 Write block 2583 len 4096 data: 4
28157 230/300 Read block 1952 len 20480
28158 231/300 Flush
28159 232/300 Read block 723 len 40960
28160 233/300 Read block 2987 len 36864
28161 234/300 Write block 1757 len 36864 data: 3 3 2 3 3 4 5 6 6
28162 235/300 Read block 4003 len 20480
28163 236/300 Write block 4377 len 32768 data: 5 5 5 6 9 10 11 10
28164 237/300 Read block 3028 len 24576
28165 238/300 Read block 2507 len 40960
28166 239/300 Read block 4824 len 40960
28167 240/300 Write block 1915 len 32768 data: 4 4 5 5 5 6 6 5
28168 241/300 Write block 4349 len 8192 data: 5 5
28169 242/300 Read block 2532 len 4096
28170 243/300 Read block 25 len 16384
28171 244/300 Read block 2983 len 8192
28172 245/300 Write block 836 len 40960 data: 4 4 4 5 6 6 7 5 5 5
28173 246/300 Write block 4742 len 12288 data: 8 7 4
28174 247/300 Write block 1107 len 20480 data: 6 6 5 6 6
28175 248/300 Read block 234 len 16384
28176 249/300 Write block 4472 len 24576 data: 6 6 6 5 6 8
28177 250/300 Read block 4847 len 36864
28178 251/300 Read block 4389 len 28672
28179 252/300 Flush
28180 253/300 Write block 4524 len 8192 data: 10 9
28181 254/300 Read block 2788 len 4096
28182 255/300 Read block 3450 len 24576
28183 256/300 Write block 3244 len 4096 data: 7
28184 257/300 Read block 1007 len 28672
28185 258/300 Write block 3260 len 12288 data: 3 4 5
28186 259/300 Write block 4081 len 28672 data: 7 6 9 8 8 9 9
28187 260/300 Read block 4945 len 12288
28188 261/300 Write block 3806 len 32768 data: 3 3 4 4 4 3 3 3
28189 262/300 Read block 3536 len 16384
28190 263/300 Write block 4894 len 16384 data: 5 6 4 5
28191 264/300 Write block 1559 len 12288 data: 4 8 7
28192 265/300 Write block 1641 len 32768 data: 6 6 5 5 4 2 4 3
28193 266/300 Read block 1126 len 4096
28194 267/300 Read block 33 len 28672
28195 268/300 Flush
28196 269/300 Flush
28197 270/300 Read block 1867 len 16384
28198 271/300 Read block 4606 len 4096
28199 272/300 Write block 3260 len 12288 data: 4 5 6
28200 273/300 Read block 2280 len 8192
28201 274/300 Flush
28202 275/300 Write block 1524 len 24576 data: 6 8 7 8 7 6
28203 276/300 Read block 2441 len 4096
28204 277/300 Write block 3818 len 32768 data: 5 5 7 6 6 6 4 3
28205 278/300 Write block 2552 len 32768 data: 7 6 6 4 4 4 4 4
28206 279/300 Write block 762 len 36864 data: 7 7 7 5 6 4 4 4 4
28207 280/300 Write block 1179 len 20480 data: 13 14 10 10 7
28208 281/300 Write block 3180 len 16384 data: 4 6 7 7
28209 282/300 Read block 2927 len 8192
28210 283/300 Read block 2128 len 36864
28211 284/300 Write block 925 len 8192 data: 4 4
28212 285/300 Read block 3302 len 4096
28213 286/300 Read block 1357 len 28672
28214 287/300 Read block 3519 len 16384
28215 288/300 Write block 3455 len 32768 data: 5 5 5 4 4 4 3 3
28216 289/300 Write block 843 len 28672 data: 6 6 6 2 5 5 4
28217 290/300 Read block 1600 len 24576
28218 291/300 Write block 1549 len 8192 data: 4 4
28219 292/300 Read block 2469 len 20480
28220 293/300 Flush
28221 294/300 Flush
28222 295/300 Read block 2197 len 28672
28223 296/300 Write block 205 len 16384 data: 8 8 7 7
28224 297/300 Write block 751 len 16384 data: 3 3 4 5
28225 298/300 Write block 606 len 12288 data: 6 7 7
28226 299/300 Write block 3357 len 40960 data: 5 7 7 8 8 8 6 5 6 9
28227 300/300 Read block 2961 len 40960
28228 301/300 Read block 4418 len 28672
28229 Main task has work to do, go find it
28230 got dsc Start(1)
28231 start 1
28232 Jan 28 22:00:32.318 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/1, method: POST, req_id: d7452e83-622b-4479-8f4c-d575801c3891, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
28233 Replay: started 1, returned:Ok(())
28234 ----------------------------------------------------------------
28235 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28236 States: Active Offline Active
28237 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28238 8548 Acked Read 7 Done Sent Done false
28239 8549 Acked Read 4 Done Sent Done false
28240 8550 Acked Write 2 Done Sent Done false
28241 8551 Acked Write 9 Done Sent Done false
28242 8552 Acked Write 9 Done Sent Done false
28243 8553 Acked Write 8 Done Sent Done false
28244 8554 Acked Read 5 Done Sent Done false
28245 8555 Acked Read 8 Done Sent Done false
28246 8556 Acked Write 3 Done Sent Done false
28247 8557 Acked Read 7 Done Sent Done false
28248 8558 Acked Read 7 Done Sent Done false
28249 8559 Acked Read 2 Done Sent Done false
28250 8560 Acked Read 4 Done Sent Done false
28251 8561 Acked Flush 0 Done Sent Done false
28252 8562 Acked Read 5 Done Sent Done false
28253 8563 Acked Write 6 Done Sent Done false
28254 8564 Acked Read 1 Done Sent Done false
28255 8565 Acked Write 8 Done Sent Done false
28256 8566 Acked Write 10 Done Sent Done false
28257 8567 Acked Flush 0 Done Sent Done false
28258 8568 Acked Write 4 Done Sent Done false
28259 8569 Acked Read 10 Done Sent Done false
28260 8570 Acked Write 1 Done Sent Done false
28261 8571 Acked Flush 0 Done Sent Done false
28262 8572 Acked Write 10 Done Sent Done false
28263 8573 Acked Read 6 Done Sent Done false
28264 8574 Acked Read 3 Done Sent Done false
28265 8575 Acked Read 2 Done Sent Done false
28266 8576 Acked Write 8 Done Sent Done false
28267 8577 Acked Read 10 Done Sent Done false
28268 8578 Acked Flush 0 Done Sent Done false
28269 8579 Acked Read 1 Done Sent Done false
28270 8580 Acked Read 4 Done Sent Done false
28271 8581 Acked Write 1 Done Sent Done false
28272 8582 Acked Write 7 Done Sent Done false
28273 8583 Acked Flush 0 Done Sent Done false
28274 8584 Acked Flush 0 Done Sent Done false
28275 8585 Acked Read 3 Done Sent Done false
28276 8586 Acked Read 4 Done Sent Done false
28277 8587 Acked Read 5 Done Sent Done false
28278 8588 Acked Write 5 Done Sent Done false
28279 8589 Acked Flush 0 Done Sent Done false
28280 8590 Acked Flush 0 Done Sent Done false
28281 8591 Acked Write 1 Done Sent Done false
28282 8592 Acked Read 2 Done Sent Done false
28283 8593 Acked Read 2 Done Sent Done false
28284 8594 Acked Read 6 Done Sent Done false
28285 8595 Acked Write 6 Done Sent Done false
28286 8596 Acked Read 2 Done Sent Done false
28287 8597 Acked Read 2 Done Sent Done false
28288 8598 Acked Read 9 Done Sent Done false
28289 8599 Acked Read 9 Done Sent Done false
28290 8600 Acked Read 4 Done Sent Done false
28291 8601 Acked Write 6 Done Sent Done false
28292 8602 Acked Read 8 Done Sent Done false
28293 8603 Acked Read 2 Done Sent Done false
28294 8604 Acked Read 7 Done Sent Done false
28295 8605 Acked Write 6 Done Sent Done false
28296 8606 Acked Write 1 Done Sent Done false
28297 8607 Acked Write 1 Done Sent Done false
28298 8608 Acked Read 1 Done Sent Done false
28299 8609 Acked Read 10 Done Sent Done false
28300 8610 Acked Write 3 Done Sent Done false
28301 8611 Acked Read 2 Done Sent Done false
28302 8612 Acked Write 5 Done Sent Done false
28303 8613 Acked Read 8 Done Sent Done false
28304 8614 Acked Read 1 Done Sent Done false
28305 8615 Acked Flush 0 Done Sent Done false
28306 8616 Acked Write 6 Done Sent Done false
28307 8617 Acked Write 3 Done Sent Done false
28308 8618 Acked Write 7 Done Sent Done false
28309 8619 Acked Flush 0 Done Sent Done false
28310 8620 Acked Read 4 Done Sent Done false
28311 8621 Acked Write 3 Done Sent Done false
28312 8622 Acked Read 6 Done Sent Done false
28313 8623 Acked Read 2 Done Sent Done false
28314 8624 Acked Write 1 Done Sent Done false
28315 8625 Acked Read 5 Done Sent Done false
28316 8626 Acked Read 8 Done Sent Done false
28317 8627 Acked Write 2 Done Sent Done false
28318 8628 Acked Write 4 Done Sent Done false
28319 8629 Acked Read 5 Done Sent Done false
28320 8630 Acked Read 3 Done Sent Done false
28321 8631 Acked Read 5 Done Sent Done false
28322 8632 Acked Write 9 Done Sent Done false
28323 8633 Acked Write 5 Done Sent Done false
28324 8634 Acked Read 2 Done Sent Done false
28325 8635 Acked Flush 0 Done Sent Done false
28326 8636 Acked Write 6 Done Sent Done false
28327 8637 Acked Write 10 Done Sent Done false
28328 8638 Acked Read 6 Done Sent Done false
28329 8639 Acked Read 7 Done Sent Done false
28330 8640 Acked Write 10 Done Sent Done false
28331 8641 Acked Write 4 Done Sent Done false
28332 8642 Acked Read 2 Done Sent Done false
28333 8643 Acked Write 2 Done Sent Done false
28334 8644 Acked Write 10 Done Sent Done false
28335 8645 Acked Write 6 Done Sent Done false
28336 8646 Acked Read 1 Done Sent Done false
28337 8647 Acked Write 3 Done Sent Done false
28338 8648 Acked Read 2 Done Sent Done false
28339 8649 Acked Read 10 Done Sent Done false
28340 8650 Acked Write 5 Done Sent Done false
28341 8651 Acked Read 2 Done Sent Done false
28342 8652 Acked Read 8 Done Sent Done false
28343 8653 Acked Read 4 Done Sent Done false
28344 8654 Acked Read 7 Done Sent Done false
28345 8655 Acked Write 5 Done Sent Done false
28346 8656 Acked Write 5 Done Sent Done false
28347 8657 Acked Flush 0 Done Sent Done false
28348 8658 Acked Read 9 Done Sent Done false
28349 8659 Acked Read 6 Done Sent Done false
28350 8660 Acked Write 6 Done Sent Done false
28351 8661 Acked Write 2 Done Sent Done false
28352 8662 Acked Write 4 Done Sent Done false
28353 8663 Acked Flush 0 Done Sent Done false
28354 8664 Acked Read 3 Done Sent Done false
28355 8665 Acked Flush 0 Done Sent Done false
28356 8666 Acked Write 9 Done Sent Done false
28357 8667 Acked Write 2 Done Sent Done false
28358 8668 Acked Read 2 Done Sent Done false
28359 8669 Acked Read 9 Done Sent Done false
28360 8670 Acked Read 1 Done Sent Done false
28361 8671 Acked Write 6 Done Sent Done false
28362 8672 Acked Read 2 Done Sent Done false
28363 8673 Acked Read 7 Done Sent Done false
28364 8674 Acked Write 3 Done Sent Done false
28365 8675 Acked Write 7 Done Sent Done false
28366 8676 Acked Read 10 Done Sent Done false
28367 8677 Acked Write 7 Done Sent Done false
28368 8678 Acked Write 9 Done Sent Done false
28369 8679 Acked Write 6 Done Sent Done false
28370 8680 Acked Read 6 Done Sent Done false
28371 8681 Acked Write 4 Done Sent Done false
28372 8682 Acked Write 8 Done Sent Done false
28373 8683 Acked Read 6 Done Sent Done false
28374 8684 Acked Write 4 Done Sent Done false
28375 8685 Acked Read 6 Done Sent Done false
28376 8686 Acked Read 8 Done Sent Done false
28377 8687 Acked Read 10 Done Sent Done false
28378 8688 Acked Write 4 Done Sent Done false
28379 8689 Acked Write 3 Done Sent Done false
28380 8690 Acked Write 7 Done Sent Done false
28381 8691 Acked Read 6 Done Sent Done false
28382 8692 Acked Read 10 Done Sent Done false
28383 8693 Acked Read 9 Done Sent Done false
28384 8694 Acked Write 2 Done Sent Done false
28385 8695 Acked Read 5 Done Sent Done false
28386 8696 Acked Write 3 Done Sent Done false
28387 8697 Acked Read 7 Done Sent Done false
28388 8698 Acked Write 3 Done Sent Done false
28389 8699 Acked Flush 0 Done Sent Done false
28390 8700 Acked Read 10 Done Sent Done false
28391 8701 Acked Flush 0 Done Sent Done false
28392 8702 Acked Write 5 Done Sent Done false
28393 8703 Acked Flush 0 Done Sent Done false
28394 8704 Acked Read 9 Done Sent Done false
28395 8705 Acked Write 2 Done Sent Done false
28396 8706 Acked Read 5 Done Sent Done false
28397 8707 Acked Read 1 Done Sent Done false
28398 8708 Acked Read 4 Done Sent Done false
28399 8709 Acked Read 6 Done Sent Done false
28400 8710 Acked Write 5 Done Sent Done false
28401 8711 Acked Write 9 Done Sent Done false
28402 8712 Acked Read 8 Done Sent Done false
28403 8713 Acked Read 3 Done Sent Done false
28404 8714 Acked Write 4 Done Sent Done false
28405 8715 Acked Write 4 Done Sent Done false
28406 8716 Acked Flush 0 Done Sent Done false
28407 8717 Acked Flush 0 Done Sent Done false
28408 8718 Acked Read 5 Done Sent Done false
28409 8719 Acked Write 4 Done Sent Done false
28410 8720 Acked Write 10 Done Sent Done false
28411 8721 Acked Write 8 Done Sent Done false
28412 8722 Acked Read 4 Done Sent Done false
28413 8723 Acked Read 6 Done Sent Done false
28414 8724 Acked Write 6 Done Sent Done false
28415 8725 Acked Read 2 Done Sent Done false
28416 8726 Acked Read 3 Done Sent Done false
28417 8727 Acked Write 1 Done Sent Done false
28418 8728 Acked Write 10 Done Sent Done false
28419 8729 Acked Write 4 Done Sent Done false
28420 8730 Acked Read 3 Done Sent Done false
28421 8731 Acked Flush 0 Done Sent Done false
28422 8732 Acked Read 10 Done Sent Done false
28423 8733 Acked Write 7 Done Sent Done false
28424 8734 Acked Read 4 Done Sent Done false
28425 8735 Acked Write 1 Done Sent Done false
28426 8736 Acked Read 4 Done Sent Done false
28427 8737 Acked Write 2 Done Sent Done false
28428 8738 Acked Read 6 Done Sent Done false
28429 8739 Acked Write 1 Done Sent Done false
28430 8740 Acked Read 9 Done Sent Done false
28431 8741 Acked Read 9 Done Sent Done false
28432 8742 Acked Read 3 Done Sent Done false
28433 8743 Acked Read 9 Done Sent Done false
28434 8744 Acked Write 4 Done Sent Done false
28435 8745 Acked Read 10 Done Sent Done false
28436 8746 Acked Read 6 Done Sent Done false
28437 8747 Acked Read 8 Done Sent Done false
28438 8748 Acked Read 10 Done Sent Done false
28439 8749 Acked Write 4 Done Sent Done false
28440 8750 Acked Flush 0 Done Sent Done false
28441 8751 Acked Read 1 Done Sent Done false
28442 8752 Acked Read 6 Done Sent Done false
28443 8753 Acked Read 2 Done Sent Done false
28444 8754 Acked Read 1 Done Sent Done false
28445 8755 Acked Write 7 Done Sent Done false
28446 8756 Acked Read 5 Done Sent Done false
28447 8757 Acked Read 3 Done Sent Done false
28448 8758 Acked Write 1 Done Sent Done false
28449 8759 Acked Write 5 Done Sent Done false
28450 8760 Acked Read 10 Done Sent Done false
28451 8761 Acked Read 10 Done Sent Done false
28452 8762 Acked Write 3 Done Sent Done false
28453 8763 Acked Write 6 Done Sent Done false
28454 8764 Acked Write 4 Done Sent Done false
28455 8765 Acked Read 3 Done Sent Done false
28456 8766 Acked Write 8 Done Sent Done false
28457 8767 Acked Write 6 Done Sent Done false
28458 8768 Acked Write 5 Done Sent Done false
28459 8769 Acked Write 10 Done Sent Done false
28460 8770 Acked Read 7 Done Sent Done false
28461 8771 Acked Write 2 Done Sent Done false
28462 8772 Acked Read 8 Done Sent Done false
28463 8773 Acked Flush 0 Done Sent Done false
28464 8774 Acked Read 6 Done Sent Done false
28465 8775 Acked Read 10 Done Sent Done false
28466 8776 Acked Write 1 Done Sent Done false
28467 8777 Acked Read 5 Done Sent Done false
28468 8778 Acked Flush 0 Done Sent Done false
28469 8779 Acked Read 10 Done Sent Done false
28470 8780 Acked Read 9 Done Sent Done false
28471 8781 Acked Write 9 Done Sent Done false
28472 8782 Acked Read 5 Done Sent Done false
28473 8783 Acked Write 8 Done Sent Done false
28474 8784 Acked Read 6 Done Sent Done false
28475 8785 Acked Read 10 Done Sent Done false
28476 8786 Acked Read 10 Done Sent Done false
28477 8787 Acked Write 8 Done Sent Done false
28478 8788 Acked Write 2 Done Sent Done false
28479 8789 Acked Read 1 Done Sent Done false
28480 8790 Acked Read 4 Done Sent Done false
28481 8791 Acked Read 2 Done Sent Done false
28482 8792 Acked Write 10 Done Sent Done false
28483 8793 Acked Write 3 Done Sent Done false
28484 8794 Acked Write 5 Done Sent Done false
28485 8795 Acked Read 4 Done Sent Done false
28486 8796 Acked Write 6 Done Sent Done false
28487 8797 Acked Read 9 Done Sent Done false
28488 8798 Acked Read 7 Done Sent Done false
28489 8799 Acked Flush 0 Done Sent Done false
28490 8800 Acked Write 2 Done Sent Done false
28491 8801 Acked Read 1 Done Sent Done false
28492 8802 Acked Read 6 Done Sent Done false
28493 8803 Acked Write 1 Done Sent Done false
28494 8804 Acked Read 7 Done Sent Done false
28495 8805 Acked Write 3 Done Sent Done false
28496 8806 Acked Write 7 Done Sent Done false
28497 8807 Acked Read 3 Done Sent Done false
28498 8808 Acked Write 8 Done Sent Done false
28499 8809 Acked Read 4 Done Sent Done false
28500 8810 Acked Write 4 Done Sent Done false
28501 8811 Acked Write 3 Done Sent Done false
28502 8812 Acked Write 8 Done Sent Done false
28503 8813 Acked Read 1 Done Sent Done false
28504 8814 Acked Read 7 Done Sent Done false
28505 8815 Acked Flush 0 Done Sent Done false
28506 8816 Acked Flush 0 Done Sent Done false
28507 8817 Acked Read 4 Done Sent Done false
28508 8818 Acked Read 1 Done Sent Done false
28509 8819 Acked Write 3 Done Sent Done false
28510 8820 Acked Read 2 Done Sent Done false
28511 8821 Acked Flush 0 Done Sent Done false
28512 8822 Acked Write 6 Done Sent Done false
28513 8823 Acked Read 1 Done Sent Done false
28514 8824 Acked Write 8 Done Sent Done false
28515 8825 Acked Write 8 Done Sent Done false
28516 8826 Acked Write 9 Done Sent Done false
28517 8827 Acked Write 5 Done Sent Done false
28518 8828 Acked Write 4 Done Sent Done false
28519 8829 Acked Read 2 Done Sent Done false
28520 8830 Acked Read 9 Done Sent Done false
28521 8831 Acked Write 2 Done Sent Done false
28522 8832 Acked Read 1 Done Sent Done false
28523 8833 Acked Read 7 Done Sent Done false
28524 8834 Acked Read 4 Done Sent Done false
28525 8835 Acked Write 8 Done Sent Done false
28526 8836 Acked Write 7 Done Sent Done false
28527 8837 Acked Read 6 Done Sent Done false
28528 8838 Acked Write 2 Done Sent Done false
28529 8839 Acked Read 5 Done Sent Done false
28530 8840 Acked Flush 0 Done Sent Done false
28531 8841 Acked Flush 0 Done Sent Done false
28532 8842 Acked Read 7 Done Sent Done false
28533 8843 Acked Write 4 Done Sent Done false
28534 8844 Acked Write 4 Done Sent Done false
28535 8845 Acked Write 3 Done Sent Done false
28536 8846 Acked Write 10 Done Sent Done false
28537 8847 Acked Read 10 Done Sent Done false
28538 8848 Acked Read 7 Done Sent Done false
28539 STATES DS:0 DS:1 DS:2 TOTAL
28540 Sent 0 301 0 301
28541 Done 301 0 301 602
28542 Skipped 0 0 0 0
28543 Error 0 0 0 0
28544 Last Flush: Some(JobId(8841)) Some(JobId(8547)) Some(JobId(8841))
28545 Downstairs last five completed: 8547 8546 8545 8544 8543
28546 Upstairs last five completed: 8848 8847 8846 8845 8844
28547 CLIENT: Up:0 ds:301 act:2
285482025-01-28T22:00:32.885ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
285492025-01-28T22:00:33.887ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
28550 [8820] got start action
28551 [8820] I am going to restart
28552 Starting downstairs at port 8820
28553 Make output file at "/tmp/dsc/downstairs-8820.txt"
28554 [8820][1] reports Starting
28555 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1228)
28556 [8820][1] reports Running
285572025-01-28T22:00:34.888ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
285582025-01-28T22:00:35.889ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
28559 ----------------------------------------------------------------
28560 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28561 States: Active Offline Active
28562 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28563 8548 Acked Read 7 Done Sent Done false
28564 8549 Acked Read 4 Done Sent Done false
28565 8550 Acked Write 2 Done Sent Done false
28566 8551 Acked Write 9 Done Sent Done false
28567 8552 Acked Write 9 Done Sent Done false
28568 8553 Acked Write 8 Done Sent Done false
28569 8554 Acked Read 5 Done Sent Done false
28570 8555 Acked Read 8 Done Sent Done false
28571 8556 Acked Write 3 Done Sent Done false
28572 8557 Acked Read 7 Done Sent Done false
28573 8558 Acked Read 7 Done Sent Done false
28574 8559 Acked Read 2 Done Sent Done false
28575 8560 Acked Read 4 Done Sent Done false
28576 8561 Acked Flush 0 Done Sent Done false
28577 8562 Acked Read 5 Done Sent Done false
28578 8563 Acked Write 6 Done Sent Done false
28579 8564 Acked Read 1 Done Sent Done false
28580 8565 Acked Write 8 Done Sent Done false
28581 8566 Acked Write 10 Done Sent Done false
28582 8567 Acked Flush 0 Done Sent Done false
28583 8568 Acked Write 4 Done Sent Done false
28584 8569 Acked Read 10 Done Sent Done false
28585 8570 Acked Write 1 Done Sent Done false
28586 8571 Acked Flush 0 Done Sent Done false
28587 8572 Acked Write 10 Done Sent Done false
28588 8573 Acked Read 6 Done Sent Done false
28589 8574 Acked Read 3 Done Sent Done false
28590 8575 Acked Read 2 Done Sent Done false
28591 8576 Acked Write 8 Done Sent Done false
28592 8577 Acked Read 10 Done Sent Done false
28593 8578 Acked Flush 0 Done Sent Done false
28594 8579 Acked Read 1 Done Sent Done false
28595 8580 Acked Read 4 Done Sent Done false
28596 8581 Acked Write 1 Done Sent Done false
28597 8582 Acked Write 7 Done Sent Done false
28598 8583 Acked Flush 0 Done Sent Done false
28599 8584 Acked Flush 0 Done Sent Done false
28600 8585 Acked Read 3 Done Sent Done false
28601 8586 Acked Read 4 Done Sent Done false
28602 8587 Acked Read 5 Done Sent Done false
28603 8588 Acked Write 5 Done Sent Done false
28604 8589 Acked Flush 0 Done Sent Done false
28605 8590 Acked Flush 0 Done Sent Done false
28606 8591 Acked Write 1 Done Sent Done false
28607 8592 Acked Read 2 Done Sent Done false
28608 8593 Acked Read 2 Done Sent Done false
28609 8594 Acked Read 6 Done Sent Done false
28610 8595 Acked Write 6 Done Sent Done false
28611 8596 Acked Read 2 Done Sent Done false
28612 8597 Acked Read 2 Done Sent Done false
28613 8598 Acked Read 9 Done Sent Done false
28614 8599 Acked Read 9 Done Sent Done false
28615 8600 Acked Read 4 Done Sent Done false
28616 8601 Acked Write 6 Done Sent Done false
28617 8602 Acked Read 8 Done Sent Done false
28618 8603 Acked Read 2 Done Sent Done false
28619 8604 Acked Read 7 Done Sent Done false
28620 8605 Acked Write 6 Done Sent Done false
28621 8606 Acked Write 1 Done Sent Done false
28622 8607 Acked Write 1 Done Sent Done false
28623 8608 Acked Read 1 Done Sent Done false
28624 8609 Acked Read 10 Done Sent Done false
28625 8610 Acked Write 3 Done Sent Done false
28626 8611 Acked Read 2 Done Sent Done false
28627 8612 Acked Write 5 Done Sent Done false
28628 8613 Acked Read 8 Done Sent Done false
28629 8614 Acked Read 1 Done Sent Done false
28630 8615 Acked Flush 0 Done Sent Done false
28631 8616 Acked Write 6 Done Sent Done false
28632 8617 Acked Write 3 Done Sent Done false
28633 8618 Acked Write 7 Done Sent Done false
28634 8619 Acked Flush 0 Done Sent Done false
28635 8620 Acked Read 4 Done Sent Done false
28636 8621 Acked Write 3 Done Sent Done false
28637 8622 Acked Read 6 Done Sent Done false
28638 8623 Acked Read 2 Done Sent Done false
28639 8624 Acked Write 1 Done Sent Done false
28640 8625 Acked Read 5 Done Sent Done false
28641 8626 Acked Read 8 Done Sent Done false
28642 8627 Acked Write 2 Done Sent Done false
28643 8628 Acked Write 4 Done Sent Done false
28644 8629 Acked Read 5 Done Sent Done false
28645 8630 Acked Read 3 Done Sent Done false
28646 8631 Acked Read 5 Done Sent Done false
28647 8632 Acked Write 9 Done Sent Done false
28648 8633 Acked Write 5 Done Sent Done false
28649 8634 Acked Read 2 Done Sent Done false
28650 8635 Acked Flush 0 Done Sent Done false
28651 8636 Acked Write 6 Done Sent Done false
28652 8637 Acked Write 10 Done Sent Done false
28653 8638 Acked Read 6 Done Sent Done false
28654 8639 Acked Read 7 Done Sent Done false
28655 8640 Acked Write 10 Done Sent Done false
28656 8641 Acked Write 4 Done Sent Done false
28657 8642 Acked Read 2 Done Sent Done false
28658 8643 Acked Write 2 Done Sent Done false
28659 8644 Acked Write 10 Done Sent Done false
28660 8645 Acked Write 6 Done Sent Done false
28661 8646 Acked Read 1 Done Sent Done false
28662 8647 Acked Write 3 Done Sent Done false
28663 8648 Acked Read 2 Done Sent Done false
28664 8649 Acked Read 10 Done Sent Done false
28665 8650 Acked Write 5 Done Sent Done false
28666 8651 Acked Read 2 Done Sent Done false
28667 8652 Acked Read 8 Done Sent Done false
28668 8653 Acked Read 4 Done Sent Done false
28669 8654 Acked Read 7 Done Sent Done false
28670 8655 Acked Write 5 Done Sent Done false
28671 8656 Acked Write 5 Done Sent Done false
28672 8657 Acked Flush 0 Done Sent Done false
28673 8658 Acked Read 9 Done Sent Done false
28674 8659 Acked Read 6 Done Sent Done false
28675 8660 Acked Write 6 Done Sent Done false
28676 8661 Acked Write 2 Done Sent Done false
28677 8662 Acked Write 4 Done Sent Done false
28678 8663 Acked Flush 0 Done Sent Done false
28679 8664 Acked Read 3 Done Sent Done false
28680 8665 Acked Flush 0 Done Sent Done false
28681 8666 Acked Write 9 Done Sent Done false
28682 8667 Acked Write 2 Done Sent Done false
28683 8668 Acked Read 2 Done Sent Done false
28684 8669 Acked Read 9 Done Sent Done false
28685 8670 Acked Read 1 Done Sent Done false
28686 8671 Acked Write 6 Done Sent Done false
28687 8672 Acked Read 2 Done Sent Done false
28688 8673 Acked Read 7 Done Sent Done false
28689 8674 Acked Write 3 Done Sent Done false
28690 8675 Acked Write 7 Done Sent Done false
28691 8676 Acked Read 10 Done Sent Done false
28692 8677 Acked Write 7 Done Sent Done false
28693 8678 Acked Write 9 Done Sent Done false
28694 8679 Acked Write 6 Done Sent Done false
28695 8680 Acked Read 6 Done Sent Done false
28696 8681 Acked Write 4 Done Sent Done false
28697 8682 Acked Write 8 Done Sent Done false
28698 8683 Acked Read 6 Done Sent Done false
28699 8684 Acked Write 4 Done Sent Done false
28700 8685 Acked Read 6 Done Sent Done false
28701 8686 Acked Read 8 Done Sent Done false
28702 8687 Acked Read 10 Done Sent Done false
28703 8688 Acked Write 4 Done Sent Done false
28704 8689 Acked Write 3 Done Sent Done false
28705 8690 Acked Write 7 Done Sent Done false
28706 8691 Acked Read 6 Done Sent Done false
28707 8692 Acked Read 10 Done Sent Done false
28708 8693 Acked Read 9 Done Sent Done false
28709 8694 Acked Write 2 Done Sent Done false
28710 8695 Acked Read 5 Done Sent Done false
28711 8696 Acked Write 3 Done Sent Done false
28712 8697 Acked Read 7 Done Sent Done false
28713 8698 Acked Write 3 Done Sent Done false
28714 8699 Acked Flush 0 Done Sent Done false
28715 8700 Acked Read 10 Done Sent Done false
28716 8701 Acked Flush 0 Done Sent Done false
28717 8702 Acked Write 5 Done Sent Done false
28718 8703 Acked Flush 0 Done Sent Done false
28719 8704 Acked Read 9 Done Sent Done false
28720 8705 Acked Write 2 Done Sent Done false
28721 8706 Acked Read 5 Done Sent Done false
28722 8707 Acked Read 1 Done Sent Done false
28723 8708 Acked Read 4 Done Sent Done false
28724 8709 Acked Read 6 Done Sent Done false
28725 8710 Acked Write 5 Done Sent Done false
28726 8711 Acked Write 9 Done Sent Done false
28727 8712 Acked Read 8 Done Sent Done false
28728 8713 Acked Read 3 Done Sent Done false
28729 8714 Acked Write 4 Done Sent Done false
28730 8715 Acked Write 4 Done Sent Done false
28731 8716 Acked Flush 0 Done Sent Done false
28732 8717 Acked Flush 0 Done Sent Done false
28733 8718 Acked Read 5 Done Sent Done false
28734 8719 Acked Write 4 Done Sent Done false
28735 8720 Acked Write 10 Done Sent Done false
28736 8721 Acked Write 8 Done Sent Done false
28737 8722 Acked Read 4 Done Sent Done false
28738 8723 Acked Read 6 Done Sent Done false
28739 8724 Acked Write 6 Done Sent Done false
28740 8725 Acked Read 2 Done Sent Done false
28741 8726 Acked Read 3 Done Sent Done false
28742 8727 Acked Write 1 Done Sent Done false
28743 8728 Acked Write 10 Done Sent Done false
28744 8729 Acked Write 4 Done Sent Done false
28745 8730 Acked Read 3 Done Sent Done false
28746 8731 Acked Flush 0 Done Sent Done false
28747 8732 Acked Read 10 Done Sent Done false
28748 8733 Acked Write 7 Done Sent Done false
28749 8734 Acked Read 4 Done Sent Done false
28750 8735 Acked Write 1 Done Sent Done false
28751 8736 Acked Read 4 Done Sent Done false
28752 8737 Acked Write 2 Done Sent Done false
28753 8738 Acked Read 6 Done Sent Done false
28754 8739 Acked Write 1 Done Sent Done false
28755 8740 Acked Read 9 Done Sent Done false
28756 8741 Acked Read 9 Done Sent Done false
28757 8742 Acked Read 3 Done Sent Done false
28758 8743 Acked Read 9 Done Sent Done false
28759 8744 Acked Write 4 Done Sent Done false
28760 8745 Acked Read 10 Done Sent Done false
28761 8746 Acked Read 6 Done Sent Done false
28762 8747 Acked Read 8 Done Sent Done false
28763 8748 Acked Read 10 Done Sent Done false
28764 8749 Acked Write 4 Done Sent Done false
28765 8750 Acked Flush 0 Done Sent Done false
28766 8751 Acked Read 1 Done Sent Done false
28767 8752 Acked Read 6 Done Sent Done false
28768 8753 Acked Read 2 Done Sent Done false
28769 8754 Acked Read 1 Done Sent Done false
28770 8755 Acked Write 7 Done Sent Done false
28771 8756 Acked Read 5 Done Sent Done false
28772 8757 Acked Read 3 Done Sent Done false
28773 8758 Acked Write 1 Done Sent Done false
28774 8759 Acked Write 5 Done Sent Done false
28775 8760 Acked Read 10 Done Sent Done false
28776 8761 Acked Read 10 Done Sent Done false
28777 8762 Acked Write 3 Done Sent Done false
28778 8763 Acked Write 6 Done Sent Done false
28779 8764 Acked Write 4 Done Sent Done false
28780 8765 Acked Read 3 Done Sent Done false
28781 8766 Acked Write 8 Done Sent Done false
28782 8767 Acked Write 6 Done Sent Done false
28783 8768 Acked Write 5 Done Sent Done false
28784 8769 Acked Write 10 Done Sent Done false
28785 8770 Acked Read 7 Done Sent Done false
28786 8771 Acked Write 2 Done Sent Done false
28787 8772 Acked Read 8 Done Sent Done false
28788 8773 Acked Flush 0 Done Sent Done false
28789 8774 Acked Read 6 Done Sent Done false
28790 8775 Acked Read 10 Done Sent Done false
28791 8776 Acked Write 1 Done Sent Done false
28792 8777 Acked Read 5 Done Sent Done false
28793 8778 Acked Flush 0 Done Sent Done false
28794 8779 Acked Read 10 Done Sent Done false
28795 8780 Acked Read 9 Done Sent Done false
28796 8781 Acked Write 9 Done Sent Done false
28797 8782 Acked Read 5 Done Sent Done false
28798 8783 Acked Write 8 Done Sent Done false
28799 8784 Acked Read 6 Done Sent Done false
28800 8785 Acked Read 10 Done Sent Done false
28801 8786 Acked Read 10 Done Sent Done false
28802 8787 Acked Write 8 Done Sent Done false
28803 8788 Acked Write 2 Done Sent Done false
28804 8789 Acked Read 1 Done Sent Done false
28805 8790 Acked Read 4 Done Sent Done false
28806 8791 Acked Read 2 Done Sent Done false
28807 8792 Acked Write 10 Done Sent Done false
28808 8793 Acked Write 3 Done Sent Done false
28809 8794 Acked Write 5 Done Sent Done false
28810 8795 Acked Read 4 Done Sent Done false
28811 8796 Acked Write 6 Done Sent Done false
28812 8797 Acked Read 9 Done Sent Done false
28813 8798 Acked Read 7 Done Sent Done false
28814 8799 Acked Flush 0 Done Sent Done false
28815 8800 Acked Write 2 Done Sent Done false
28816 8801 Acked Read 1 Done Sent Done false
28817 8802 Acked Read 6 Done Sent Done false
28818 8803 Acked Write 1 Done Sent Done false
28819 8804 Acked Read 7 Done Sent Done false
28820 8805 Acked Write 3 Done Sent Done false
28821 8806 Acked Write 7 Done Sent Done false
28822 8807 Acked Read 3 Done Sent Done false
28823 8808 Acked Write 8 Done Sent Done false
28824 8809 Acked Read 4 Done Sent Done false
28825 8810 Acked Write 4 Done Sent Done false
28826 8811 Acked Write 3 Done Sent Done false
28827 8812 Acked Write 8 Done Sent Done false
28828 8813 Acked Read 1 Done Sent Done false
28829 8814 Acked Read 7 Done Sent Done false
28830 8815 Acked Flush 0 Done Sent Done false
28831 8816 Acked Flush 0 Done Sent Done false
28832 8817 Acked Read 4 Done Sent Done false
28833 8818 Acked Read 1 Done Sent Done false
28834 8819 Acked Write 3 Done Sent Done false
28835 8820 Acked Read 2 Done Sent Done false
28836 8821 Acked Flush 0 Done Sent Done false
28837 8822 Acked Write 6 Done Sent Done false
28838 8823 Acked Read 1 Done Sent Done false
28839 8824 Acked Write 8 Done Sent Done false
28840 8825 Acked Write 8 Done Sent Done false
28841 8826 Acked Write 9 Done Sent Done false
28842 8827 Acked Write 5 Done Sent Done false
28843 8828 Acked Write 4 Done Sent Done false
28844 8829 Acked Read 2 Done Sent Done false
28845 8830 Acked Read 9 Done Sent Done false
28846 8831 Acked Write 2 Done Sent Done false
28847 8832 Acked Read 1 Done Sent Done false
28848 8833 Acked Read 7 Done Sent Done false
28849 8834 Acked Read 4 Done Sent Done false
28850 8835 Acked Write 8 Done Sent Done false
28851 8836 Acked Write 7 Done Sent Done false
28852 8837 Acked Read 6 Done Sent Done false
28853 8838 Acked Write 2 Done Sent Done false
28854 8839 Acked Read 5 Done Sent Done false
28855 8840 Acked Flush 0 Done Sent Done false
28856 8841 Acked Flush 0 Done Sent Done false
28857 8842 Acked Read 7 Done Sent Done false
28858 8843 Acked Write 4 Done Sent Done false
28859 8844 Acked Write 4 Done Sent Done false
28860 8845 Acked Write 3 Done Sent Done false
28861 8846 Acked Write 10 Done Sent Done false
28862 8847 Acked Read 10 Done Sent Done false
28863 8848 Acked Read 7 Done Sent Done false
28864 STATES DS:0 DS:1 DS:2 TOTAL
28865 Sent 0 301 0 301
28866 Done 301 0 301 602
28867 Skipped 0 0 0 0
28868 Error 0 0 0 0
28869 Last Flush: Some(JobId(8841)) Some(JobId(8547)) Some(JobId(8841))
28870 Downstairs last five completed: 8547 8546 8545 8544 8543
28871 Upstairs last five completed: 8848 8847 8846 8845 8844
28872 CLIENT: Up:0 ds:301 act:2
288732025-01-28T22:00:37.078ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
288742025-01-28T22:00:37.892ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
288752025-01-28T22:00:38.089ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
288762025-01-28T22:00:38.089ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48176) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
288772025-01-28T22:00:38.090ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
288782025-01-28T22:00:38.091ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
288792025-01-28T22:00:38.091ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
288802025-01-28T22:00:38.091ZINFOcrucible: send last flush ID to this DS: Some(JobId(8547)) = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
288812025-01-28T22:00:38.091ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
288822025-01-28T22:00:38.091ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(8547)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
288832025-01-28T22:00:38.091ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(8547)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
28884 ----------------------------------------------------------------
28885 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28886 Downstairs last five completed: 8849 8848 8847 8846 8845
28887 Upstairs last five completed: 8849 8848 8847 8846 8845
28888 CLIENT: Up:0 ds:0 act:3
28889 Replay: All jobs finished
28890 Main task has work to do, go find it
28891 got dsc Stop(0)
28892 stop 0
28893 [8810] Got stop action so:false kr:false
28894 Jan 28 22:00:40.323 INFO request completed, latency_us: 365, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 95940a37-7966-40ab-8b41-bfe300f8e3a0, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
288952025-01-28T22:00:40.324ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
28896 Jan 28 22:00:40.324 INFO {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-28T22:00:40.324321111Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
28897 request completed, latency_us: 158, response_code: 200, uri: /state/cid/0, method: GET{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-28T22:00:40.324392026Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
28898 , req_id: d3b273d0-9eb6-45eb-a845-b50fa0cc3683, remote_addr: 127.0.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-28T22:00:40.324421484Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
28899 0.1:36665, local_addr: 127.0.0.1:9998
28900 [8810] Exited with: ExitStatus(unix_wait_status(9))
28901 [8810][0] reports Exit
28902 Jan 28 22:00:44.326 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/0, method: GET, req_id: 3ce35cb6-5104-4c03-bf74-f5a3f51c6935, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
28903 001/300 Flush
28904 002/300 Read block 2386 len 28672
28905 003/300 Read block 840 len 32768
28906 004/300 Read block 4600 len 28672
28907 005/300 Write block 126 len 4096 data: 10
28908 006/300 Write block 2841 len 40960 data: 6 6 6 7 6 8 8 10 8 7
28909 007/300 Read block 2102 len 12288
28910 008/300 Read block 3812 len 8192
28911 009/300 Write block 1530 len 12288 data: 7 7 7
28912 010/300 Write block 1720 len 20480 data: 7 7 7 5 4
28913 011/300 Write block 1068 len 24576 data: 4 4 3 3 3 3
28914 012/300 Read block 3094 len 32768
28915 013/300 Read block 359 len 28672
28916 014/300 Read block 3681 len 16384
28917 015/300 Write block 2986 len 20480 data: 5 3 5 4 4
28918 016/300 Read block 1319 len 16384
28919 017/300 Read block 4635 len 20480
28920 018/300 Write block 991 len 36864 data: 3 3 4 5 5 5 6 7 7
28921 019/300 Write block 1481 len 40960 data: 7 8 7 7 7 7 7 8 7 8
28922 020/300 Flush
28923 021/300 Read block 4025 len 12288
28924 022/300 Write block 1876 len 20480 data: 5 5 5 6 6
28925 023/300 Flush
28926 024/300 Flush
28927 025/300 Read block 3632 len 20480
28928 026/300 Write block 1807 len 24576 data: 4 6 6 7 6 6
28929 027/300 Read block 3574 len 12288
28930 028/300 Flush
28931 029/300 Read block 4219 len 24576
28932 030/300 Write block 2995 len 12288 data: 9 8 7
28933 031/300 Read block 490 len 24576
28934 032/300 Write block 3552 len 24576 data: 7 6 7 7 7 7
28935 033/300 Write block 4075 len 40960 data: 6 6 5 5 5 6 8 7 10 9
28936 034/300 Write block 3820 len 16384 data: 8 7 7 7
28937 035/300 Read block 3207 len 20480
28938 036/300 Write block 3835 len 4096 data: 5
28939 037/300 Read block 3580 len 16384
28940 038/300 Read block 1548 len 24576
28941 039/300 Read block 1598 len 24576
28942 040/300 Write block 2131 len 12288 data: 8 8 11
28943 041/300 Read block 981 len 32768
28944 042/300 Read block 3554 len 40960
28945 043/300 Write block 3770 len 16384 data: 8 8 6 5
28946 044/300 Read block 518 len 16384
28947 045/300 Write block 3289 len 28672 data: 5 2 3 3 5 6 7
28948 046/300 Write block 432 len 32768 data: 3 3 4 7 7 6 5 5
28949 047/300 Write block 3954 len 16384 data: 5 5 6 7
28950 048/300 Flush
28951 049/300 Write block 4705 len 40960 data: 4 4 6 7 8 8 6 5 3 3
28952 050/300 Write block 483 len 40960 data: 7 5 5 5 4 4 4 3 5 4
28953 051/300 Read block 3310 len 32768
28954 052/300 Read block 2909 len 40960
28955 053/300 Read block 4381 len 4096
28956 054/300 Read block 3085 len 40960
28957 055/300 Read block 955 len 28672
28958 056/300 Read block 3095 len 32768
28959 057/300 Read block 2286 len 36864
28960 058/300 Read block 2441 len 40960
28961 059/300 Read block 1936 len 32768
28962 060/300 Read block 2069 len 40960
28963 061/300 Write block 4607 len 36864 data: 7 5 4 4 6 5 5 5 7
28964 062/300 Write block 4828 len 32768 data: 4 4 4 3 3 4 5 6
28965 063/300 Read block 3558 len 28672
28966 064/300 Read block 2730 len 4096
28967 065/300 Read block 3719 len 20480
28968 066/300 Read block 3601 len 20480
28969 067/300 Read block 4486 len 32768
28970 068/300 Write block 2269 len 12288 data: 2 4 4
28971 069/300 Write block 325 len 12288 data: 6 5 6
28972 070/300 Read block 4011 len 16384
28973 071/300 Read block 227 len 24576
28974 072/300 Read block 1375 len 12288
28975 073/300 Write block 4704 len 32768 data: 3 5 5 7 8 9 9 7
28976 074/300 Write block 3018 len 32768 data: 3 3 4 5 4 5 7 8
28977 075/300 Write block 4574 len 32768 data: 3 3 4 4 5 7 8 8
28978 076/300 Read block 1259 len 32768
28979 077/300 Read block 4169 len 40960
28980 078/300 Read block 1473 len 36864
28981 079/300 Read block 3039 len 4096
28982 080/300 Write block 2133 len 12288 data: 12 11 10
28983 081/300 Flush
28984 082/300 Write block 3353 len 8192 data: 4 4
28985 083/300 Write block 713 len 20480 data: 9 8 7 8 5
28986 084/300 Write block 4190 len 12288 data: 6 7 5
28987 085/300 Read block 2091 len 40960
28988 086/300 Read block 1928 len 32768
28989 087/300 Read block 2584 len 8192
28990 088/300 Read block 801 len 12288
28991 089/300 Write block 3153 len 40960 data: 6 7 6 6 5 6 6 6 6 5
28992 090/300 Read block 2225 len 36864
28993 091/300 Flush
28994 092/300 Read block 4130 len 8192
28995 093/300 Write block 52 len 12288 data: 5 5 5
28996 094/300 Read block 2056 len 20480
28997 095/300 Read block 2885 len 20480
28998 096/300 Flush
28999 097/300 Read block 2223 len 8192
29000 098/300 Write block 969 len 8192 data: 6 7
29001 099/300 Flush
29002 100/300 Read block 2380 len 8192
29003 101/300 Write block 2907 len 20480 data: 6 4 5 5 5
29004 102/300 Read block 4907 len 12288
29005 103/300 Read block 1402 len 32768
29006 104/300 Write block 1381 len 36864 data: 4 3 3 4 2 2 2 2 3
29007 105/300 Write block 1611 len 32768 data: 2 2 2 3 5 4 5 5
29008 106/300 Read block 4838 len 28672
29009 107/300 Write block 693 len 40960 data: 6 8 6 6 6 6 7 7 7 6
29010 108/300 Read block 252 len 36864
29011 109/300 Write block 430 len 32768 data: 2 2 4 4 5 8 8 7
29012 110/300 Write block 372 len 8192 data: 5 6
29013 111/300 Read block 4540 len 20480
29014 112/300 Read block 1949 len 24576
29015 113/300 Write block 414 len 32768 data: 4 4 5 6 5 5 7 8
29016 114/300 Write block 2408 len 40960 data: 6 8 6 7 6 7 7 6 8 7
29017 115/300 Write block 4014 len 32768 data: 4 4 5 6 6 4 4 5
29018 116/300 Write block 2438 len 16384 data: 3 2 2 3
29019 117/300 Write block 1649 len 32768 data: 3 3 3 3 4 4 4 5
29020 118/300 Read block 3283 len 32768
29021 119/300 Read block 4900 len 20480
29022 120/300 Write block 819 len 32768 data: 3 4 5 5 3 2 3 4
29023 121/300 Flush
29024 122/300 Write block 525 len 40960 data: 6 6 7 6 7 6 6 6 7 7
29025 123/300 Read block 4067 len 32768
29026 124/300 Write block 4887 len 36864 data: 5 5 5 6 6 6 6 6 7
29027 125/300 Read block 29 len 20480
29028 126/300 Write block 78 len 16384 data: 8 8 9 9
29029 127/300 Read block 220 len 16384
29030 128/300 Read block 2909 len 32768
29031 129/300 Write block 3172 len 32768 data: 5 5 5 5 6 4 4 4
29032 130/300 Write block 2303 len 16384 data: 4 4 4 4
29033 131/300 Write block 4650 len 40960 data: 4 4 4 6 6 5 5 5 8 9
29034 132/300 Read block 312 len 20480
29035 133/300 Read block 4935 len 32768
29036 134/300 Flush
29037 135/300 Read block 3361 len 4096
29038 136/300 Write block 3196 len 36864 data: 4 5 5 6 7 7 7 6 5
29039 137/300 Read block 853 len 8192
29040 138/300 Write block 2891 len 16384 data: 3 3 4 6
29041 139/300 Read block 3141 len 32768
29042 140/300 Read block 1480 len 28672
29043 141/300 Read block 1659 len 16384
29044 142/300 Read block 542 len 40960
29045 143/300 Read block 1846 len 4096
29046 144/300 Read block 3108 len 24576
29047 145/300 Flush
29048 146/300 Write block 2662 len 4096 data: 10
29049 147/300 Write block 1806 len 32768 data: 3 5 7 7 8 7 7 4
29050 148/300 Write block 2635 len 40960 data: 9 9 7 4 5 4 5 5 4 3
29051 149/300 Read block 3417 len 16384
29052 150/300 Read block 2743 len 36864
29053 151/300 Read block 3355 len 32768
29054 152/300 Read block 1864 len 24576
29055 153/300 Write block 3995 len 12288 data: 4 5 4
29056 154/300 Write block 826 len 36864 data: 5 4 4 4 4 3 3 4 5
29057 155/300 Write block 4798 len 40960 data: 6 7 6 4 4 4 7 7 8 9
29058 156/300 Write block 693 len 20480 data: 7 9 7 7 7
29059 157/300 Flush
29060 158/300 Read block 1203 len 8192
29061 159/300 Write block 4402 len 12288 data: 8 7 6
29062 160/300 Write block 4034 len 28672 data: 4 4 4 4 5 4 3
29063 161/300 Write block 3875 len 16384 data: 5 7 7 8
29064 162/300 Write block 1963 len 4096 data: 6
29065 163/300 Read block 158 len 8192
29066 164/300 Flush
29067 165/300 Read block 1484 len 20480
29068 166/300 Read block 113 len 40960
29069 167/300 Read block 4463 len 20480
29070 168/300 Write block 3356 len 12288 data: 6 6 8
29071 169/300 Read block 211 len 16384
29072 170/300 Read block 4445 len 36864
29073 171/300 Write block 1257 len 28672 data: 5 4 4 4 3 3 3
29074 172/300 Read block 2687 len 40960
29075 173/300 Write block 2477 len 20480 data: 10 8 7 6 7
29076 174/300 Flush
29077 175/300 Read block 1450 len 24576
29078 176/300 Read block 232 len 24576
29079 177/300 Read block 1452 len 24576
29080 178/300 Write block 3050 len 20480 data: 11 8 8 7 7
29081 179/300 Write block 4959 len 8192 data: 5 5
29082 180/300 Flush
29083 181/300 Write block 114 len 12288 data: 7 5 4
29084 182/300 Write block 1088 len 32768 data: 3 4 4 4 3 3 5 6
29085 183/300 Write block 3219 len 20480 data: 7 7 7 6 6
29086 184/300 Flush
29087 185/300 Read block 3907 len 16384
29088 186/300 Write block 1787 len 16384 data: 2 2 4 4
29089 187/300 Read block 2631 len 36864
29090 188/300 Read block 1385 len 36864
29091 189/300 Write block 3491 len 32768 data: 4 4 5 5 5 5 3 3
29092 190/300 Write block 3329 len 28672 data: 8 10 8 7 7 7 6
29093 191/300 Write block 3922 len 32768 data: 6 7 6 8 6 7 9 7
29094 192/300 Write block 175 len 36864 data: 5 5 3 3 3 3 4 4 3
29095 193/300 Write block 2820 len 12288 data: 6 7 4
29096 194/300 Write block 3190 len 24576 data: 5 4 5 5 5 5
29097 195/300 Read block 1791 len 8192
29098 196/300 Write block 4489 len 40960 data: 6 7 6 5 3 3 3 4 3 3
29099 197/300 Read block 1278 len 12288
29100 198/300 Read block 3056 len 40960
29101 199/300 Flush
29102 200/300 Write block 1518 len 40960 data: 7 7 7 6 6 5 7 9 8 9
29103 201/300 Read block 1829 len 28672
29104 202/300 Write block 4031 len 12288 data: 6 6 6
29105 203/300 Read block 680 len 24576
29106 204/300 Read block 3249 len 28672
29107 205/300 Read block 2511 len 4096
29108 206/300 Read block 3704 len 28672
29109 207/300 Write block 1760 len 36864 data: 4 4 5 6 7 7 6 6 7
29110 208/300 Read block 164 len 16384
29111 209/300 Read block 3934 len 36864
29112 210/300 Write block 1893 len 32768 data: 3 2 2 4 3 5 5 4
29113 211/300 Write block 892 len 12288 data: 6 6 3
29114 212/300 Read block 2001 len 24576
29115 213/300 Read block 4447 len 36864
29116 214/300 Read block 217 len 12288
29117 215/300 Write block 1311 len 16384 data: 6 6 6 5
29118 216/300 Read block 558 len 40960
29119 217/300 Read block 2378 len 40960
29120 218/300 Read block 1287 len 36864
29121 219/300 Write block 1617 len 8192 data: 6 6
29122 220/300 Read block 4097 len 4096
29123 221/300 Write block 439 len 40960 data: 6 5 6 5 6 5 5 4 4 4
29124 222/300 Read block 4238 len 32768
29125 223/300 Write block 4705 len 36864 data: 6 6 8 9 10 10 8 6 4
29126 224/300 Write block 655 len 16384 data: 5 4 5 4
29127 225/300 Read block 3035 len 36864
29128 226/300 Read block 664 len 24576
29129 227/300 Write block 2475 len 16384 data: 7 8 11 9
29130 228/300 Write block 3557 len 16384 data: 8 6 7 5
29131 229/300 Write block 889 len 28672 data: 6 3 4 7 7 4 3
29132 230/300 Read block 227 len 40960
29133 231/300 Read block 1793 len 20480
29134 232/300 Flush
29135 233/300 Write block 1426 len 16384 data: 6 5 5 5
29136 234/300 Read block 3039 len 36864
29137 235/300 Write block 2062 len 16384 data: 5 5 4 4
29138 236/300 Write block 3032 len 20480 data: 6 4 4 3 4
29139 237/300 Read block 169 len 28672
29140 238/300 Flush
29141 239/300 Write block 3264 len 16384 data: 7 6 5 5
29142 240/300 Write block 1451 len 16384 data: 4 6 6 7
29143 241/300 Write block 498 len 8192 data: 5 6
29144 242/300 Read block 4553 len 32768
29145 243/300 Write block 4968 len 24576 data: 6 7 7 6 5 5
29146 244/300 Read block 4856 len 40960
29147 245/300 Write block 4203 len 12288 data: 6 6 7
29148 246/300 Write block 1520 len 4096 data: 8
29149 247/300 Read block 900 len 20480
29150 248/300 Write block 4062 len 36864 data: 6 7 6 6 6 5 4 6 6
29151 249/300 Write block 4657 len 28672 data: 6 9 10 9 10 10 10
29152 250/300 Read block 87 len 40960
29153 251/300 Read block 3388 len 8192
29154 252/300 Write block 840 len 24576 data: 7 7 8 7 7 7
29155 253/300 Write block 3202 len 4096 data: 8
29156 254/300 Read block 381 len 20480
29157 255/300 Write block 862 len 12288 data: 2 2 3
29158 256/300 Write block 3212 len 40960 data: 6 7 7 7 6 5 6 8 8 8
29159 257/300 Write block 3766 len 4096 data: 7
29160 258/300 Flush
29161 259/300 Read block 76 len 40960
29162 260/300 Read block 306 len 24576
29163 261/300 Flush
29164 262/300 Read block 2717 len 36864
29165 263/300 Write block 2962 len 32768 data: 5 5 5 6 6 8 6 6
29166 264/300 Read block 3800 len 8192
29167 265/300 Read block 322 len 16384
29168 266/300 Read block 3031 len 32768
29169 267/300 Write block 2683 len 40960 data: 4 5 5 4 5 8 7 8 9 8
29170 268/300 Read block 599 len 40960
29171 269/300 Write block 2872 len 12288 data: 5 5 7
29172 270/300 Read block 2144 len 24576
29173 271/300 Read block 3806 len 20480
29174 272/300 Read block 2047 len 4096
29175 273/300 Read block 748 len 20480
29176 274/300 Flush
29177 275/300 Read block 4526 len 4096
29178 276/300 Write block 1843 len 4096 data: 7
29179 277/300 Read block 1206 len 8192
29180 278/300 Read block 387 len 36864
29181 279/300 Read block 3706 len 8192
29182 280/300 Write block 3764 len 20480 data: 8 7 8 8 7
29183 281/300 Read block 3217 len 12288
29184 282/300 Read block 4576 len 32768
29185 283/300 Read block 2861 len 36864
29186 284/300 Read block 4691 len 8192
29187 285/300 Write block 1098 len 36864 data: 5 5 7 7 5 5 5 8 7
29188 286/300 Read block 2239 len 4096
29189 287/300 Read block 958 len 32768
29190 288/300 Flush
29191 289/300 Write block 4507 len 32768 data: 6 5 5 7 7 5 6 6
29192 290/300 Write block 4053 len 20480 data: 4 4 4 5 6
29193 291/300 Read block 1930 len 24576
29194 292/300 Flush
29195 293/300 Write block 269 len 16384 data: 5 6 3 3
29196 294/300 Read block 794 len 8192
29197 295/300 Write block 4466 len 12288 data: 5 4 3
29198 296/300 Write block 480 len 8192 data: 7 6
29199 297/300 Read block 1600 len 24576
29200 298/300 Read block 1370 len 8192
29201 299/300 Flush
29202 300/300 Read block 4545 len 24576
29203 301/300 Read block 944 len 8192
29204 Main task has work to do, go find it
29205 got dsc Start(0)
29206 start 0
29207 Jan 28 22:00:44.575 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/0, method: POST, req_id: a019a232-2930-4064-8183-936508e2477e, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
29208 Replay: started 0, returned:Ok(())
29209 ----------------------------------------------------------------
29210 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29211 States: Offline Active Active
29212 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29213 8850 Acked Flush 0 Sent Done Done false
29214 8851 Acked Read 7 Sent Done Done false
29215 8852 Acked Read 8 Sent Done Done false
29216 8853 Acked Read 7 Sent Done Done false
29217 8854 Acked Write 1 Sent Done Done false
29218 8855 Acked Write 10 Sent Done Done false
29219 8856 Acked Read 3 Sent Done Done false
29220 8857 Acked Read 2 Sent Done Done false
29221 8858 Acked Write 3 Sent Done Done false
29222 8859 Acked Write 5 Sent Done Done false
29223 8860 Acked Write 6 Sent Done Done false
29224 8861 Acked Read 8 Sent Done Done false
29225 8862 Acked Read 7 Sent Done Done false
29226 8863 Acked Read 4 Sent Done Done false
29227 8864 Acked Write 5 Sent Done Done false
29228 8865 Acked Read 4 Sent Done Done false
29229 8866 Acked Read 5 Sent Done Done false
29230 8867 Acked Write 9 Sent Done Done false
29231 8868 Acked Write 10 Sent Done Done false
29232 8869 Acked Flush 0 Sent Done Done false
29233 8870 Acked Read 3 Sent Done Done false
29234 8871 Acked Write 5 Sent Done Done false
29235 8872 Acked Flush 0 Sent Done Done false
29236 8873 Acked Flush 0 Sent Done Done false
29237 8874 Acked Read 5 Sent Done Done false
29238 8875 Acked Write 6 Sent Done Done false
29239 8876 Acked Read 3 Sent Done Done false
29240 8877 Acked Flush 0 Sent Done Done false
29241 8878 Acked Read 6 Sent Done Done false
29242 8879 Acked Write 3 Sent Done Done false
29243 8880 Acked Read 6 Sent Done Done false
29244 8881 Acked Write 6 Sent Done Done false
29245 8882 Acked Write 10 Sent Done Done false
29246 8883 Acked Write 4 Sent Done Done false
29247 8884 Acked Read 5 Sent Done Done false
29248 8885 Acked Write 1 Sent Done Done false
29249 8886 Acked Read 4 Sent Done Done false
29250 8887 Acked Read 6 Sent Done Done false
29251 8888 Acked Read 6 Sent Done Done false
29252 8889 Acked Write 3 Sent Done Done false
29253 8890 Acked Read 8 Sent Done Done false
29254 8891 Acked Read 10 Sent Done Done false
29255 8892 Acked Write 4 Sent Done Done false
29256 8893 Acked Read 4 Sent Done Done false
29257 8894 Acked Write 7 Sent Done Done false
29258 8895 Acked Write 8 Sent Done Done false
29259 8896 Acked Write 4 Sent Done Done false
29260 8897 Acked Flush 0 Sent Done Done false
29261 8898 Acked Write 10 Sent Done Done false
29262 8899 Acked Write 10 Sent Done Done false
29263 8900 Acked Read 8 Sent Done Done false
29264 8901 Acked Read 10 Sent Done Done false
29265 8902 Acked Read 1 Sent Done Done false
29266 8903 Acked Read 10 Sent Done Done false
29267 8904 Acked Read 7 Sent Done Done false
29268 8905 Acked Read 8 Sent Done Done false
29269 8906 Acked Read 9 Sent Done Done false
29270 8907 Acked Read 10 Sent Done Done false
29271 8908 Acked Read 8 Sent Done Done false
29272 8909 Acked Read 10 Sent Done Done false
29273 8910 Acked Write 9 Sent Done Done false
29274 8911 Acked Write 8 Sent Done Done false
29275 8912 Acked Read 7 Sent Done Done false
29276 8913 Acked Read 1 Sent Done Done false
29277 8914 Acked Read 5 Sent Done Done false
29278 8915 Acked Read 5 Sent Done Done false
29279 8916 Acked Read 8 Sent Done Done false
29280 8917 Acked Write 3 Sent Done Done false
29281 8918 Acked Write 3 Sent Done Done false
29282 8919 Acked Read 4 Sent Done Done false
29283 8920 Acked Read 6 Sent Done Done false
29284 8921 Acked Read 3 Sent Done Done false
29285 8922 Acked Write 8 Sent Done Done false
29286 8923 Acked Write 8 Sent Done Done false
29287 8924 Acked Write 8 Sent Done Done false
29288 8925 Acked Read 8 Sent Done Done false
29289 8926 Acked Read 10 Sent Done Done false
29290 8927 Acked Read 9 Sent Done Done false
29291 8928 Acked Read 1 Sent Done Done false
29292 8929 Acked Write 3 Sent Done Done false
29293 8930 Acked Flush 0 Sent Done Done false
29294 8931 Acked Write 2 Sent Done Done false
29295 8932 Acked Write 5 Sent Done Done false
29296 8933 Acked Write 3 Sent Done Done false
29297 8934 Acked Read 10 Sent Done Done false
29298 8935 Acked Read 8 Sent Done Done false
29299 8936 Acked Read 2 Sent Done Done false
29300 8937 Acked Read 3 Sent Done Done false
29301 8938 Acked Write 10 Sent Done Done false
29302 8939 Acked Read 9 Sent Done Done false
29303 8940 Acked Flush 0 Sent Done Done false
29304 8941 Acked Read 2 Sent Done Done false
29305 8942 Acked Write 3 Sent Done Done false
29306 8943 Acked Read 5 Sent Done Done false
29307 8944 Acked Read 5 Sent Done Done false
29308 8945 Acked Flush 0 Sent Done Done false
29309 8946 Acked Read 2 Sent Done Done false
29310 8947 Acked Write 2 Sent Done Done false
29311 8948 Acked Flush 0 Sent Done Done false
29312 8949 Acked Read 2 Sent Done Done false
29313 8950 Acked Write 5 Sent Done Done false
29314 8951 Acked Read 3 Sent Done Done false
29315 8952 Acked Read 8 Sent Done Done false
29316 8953 Acked Write 9 Sent Done Done false
29317 8954 Acked Write 8 Sent Done Done false
29318 8955 Acked Read 7 Sent Done Done false
29319 8956 Acked Write 10 Sent Done Done false
29320 8957 Acked Read 9 Sent Done Done false
29321 8958 Acked Write 8 Sent Done Done false
29322 8959 Acked Write 2 Sent Done Done false
29323 8960 Acked Read 5 Sent Done Done false
29324 8961 Acked Read 6 Sent Done Done false
29325 8962 Acked Write 8 Sent Done Done false
29326 8963 Acked Write 10 Sent Done Done false
29327 8964 Acked Write 8 Sent Done Done false
29328 8965 Acked Write 4 Sent Done Done false
29329 8966 Acked Write 8 Sent Done Done false
29330 8967 Acked Read 8 Sent Done Done false
29331 8968 Acked Read 5 Sent Done Done false
29332 8969 Acked Write 8 Sent Done Done false
29333 8970 Acked Flush 0 Sent Done Done false
29334 8971 Acked Write 10 Sent Done Done false
29335 8972 Acked Read 8 Sent Done Done false
29336 8973 Acked Write 9 Sent Done Done false
29337 8974 Acked Read 5 Sent Done Done false
29338 8975 Acked Write 4 Sent Done Done false
29339 8976 Acked Read 4 Sent Done Done false
29340 8977 Acked Read 8 Sent Done Done false
29341 8978 Acked Write 8 Sent Done Done false
29342 8979 Acked Write 4 Sent Done Done false
29343 8980 Acked Write 10 Sent Done Done false
29344 8981 Acked Read 5 Sent Done Done false
29345 8982 Acked Read 8 Sent Done Done false
29346 8983 Acked Flush 0 Sent Done Done false
29347 8984 Acked Read 1 Sent Done Done false
29348 8985 Acked Write 9 Sent Done Done false
29349 8986 Acked Read 2 Sent Done Done false
29350 8987 Acked Write 4 Sent Done Done false
29351 8988 Acked Read 8 Sent Done Done false
29352 8989 Acked Read 7 Sent Done Done false
29353 8990 Acked Read 4 Sent Done Done false
29354 8991 Acked Read 10 Sent Done Done false
29355 8992 Acked Read 1 Sent Done Done false
29356 8993 Acked Read 6 Sent Done Done false
29357 8994 Acked Flush 0 Sent Done Done false
29358 8995 Acked Write 1 Sent Done Done false
29359 8996 Acked Write 8 Sent Done Done false
29360 8997 Acked Write 10 Sent Done Done false
29361 8998 Acked Read 4 Sent Done Done false
29362 8999 Acked Read 9 Sent Done Done false
29363 9000 Acked Read 8 Sent Done Done false
29364 9001 Acked Read 6 Sent Done Done false
29365 9002 Acked Write 3 Sent Done Done false
29366 9003 Acked Write 9 Sent Done Done false
29367 9004 Acked Write 10 Sent Done Done false
29368 9005 Acked Write 5 Sent Done Done false
29369 9006 Acked Flush 0 Sent Done Done false
29370 9007 Acked Read 2 Sent Done Done false
29371 9008 Acked Write 3 Sent Done Done false
29372 9009 Acked Write 7 Sent Done Done false
29373 9010 Acked Write 4 Sent Done Done false
29374 9011 Acked Write 1 Sent Done Done false
29375 9012 Acked Read 2 Sent Done Done false
29376 9013 Acked Flush 0 Sent Done Done false
29377 9014 Acked Read 5 Sent Done Done false
29378 9015 Acked Read 10 Sent Done Done false
29379 9016 Acked Read 5 Sent Done Done false
29380 9017 Acked Write 3 Sent Done Done false
29381 9018 Acked Read 4 Sent Done Done false
29382 9019 Acked Read 9 Sent Done Done false
29383 9020 Acked Write 7 Sent Done Done false
29384 9021 Acked Read 10 Sent Done Done false
29385 9022 Acked Write 5 Sent Done Done false
29386 9023 Acked Flush 0 Sent Done Done false
29387 9024 Acked Read 6 Sent Done Done false
29388 9025 Acked Read 6 Sent Done Done false
29389 9026 Acked Read 6 Sent Done Done false
29390 9027 Acked Write 5 Sent Done Done false
29391 9028 Acked Write 2 Sent Done Done false
29392 9029 Acked Flush 0 Sent Done Done false
29393 9030 Acked Write 3 Sent Done Done false
29394 9031 Acked Write 8 Sent Done Done false
29395 9032 Acked Write 5 Sent Done Done false
29396 9033 Acked Flush 0 Sent Done Done false
29397 9034 Acked Read 4 Sent Done Done false
29398 9035 Acked Write 4 Sent Done Done false
29399 9036 Acked Read 9 Sent Done Done false
29400 9037 Acked Read 9 Sent Done Done false
29401 9038 Acked Write 8 Sent Done Done false
29402 9039 Acked Write 7 Sent Done Done false
29403 9040 Acked Write 8 Sent Done Done false
29404 9041 Acked Write 9 Sent Done Done false
29405 9042 Acked Write 3 Sent Done Done false
29406 9043 Acked Write 6 Sent Done Done false
29407 9044 Acked Read 2 Sent Done Done false
29408 9045 Acked Write 10 Sent Done Done false
29409 9046 Acked Read 3 Sent Done Done false
29410 9047 Acked Read 10 Sent Done Done false
29411 9048 Acked Flush 0 Sent Done Done false
29412 9049 Acked Write 10 Sent Done Done false
29413 9050 Acked Read 7 Sent Done Done false
29414 9051 Acked Write 3 Sent Done Done false
29415 9052 Acked Read 6 Sent Done Done false
29416 9053 Acked Read 7 Sent Done Done false
29417 9054 Acked Read 1 Sent Done Done false
29418 9055 Acked Read 7 Sent Done Done false
29419 9056 Acked Write 9 Sent Done Done false
29420 9057 Acked Read 4 Sent Done Done false
29421 9058 Acked Read 9 Sent Done Done false
29422 9059 Acked Write 8 Sent Done Done false
29423 9060 Acked Write 3 Sent Done Done false
29424 9061 Acked Read 6 Sent Done Done false
29425 9062 Acked Read 9 Sent Done Done false
29426 9063 Acked Read 3 Sent Done Done false
29427 9064 Acked Write 4 Sent Done Done false
29428 9065 Acked Read 10 Sent Done Done false
29429 9066 Acked Read 10 Sent Done Done false
29430 9067 Acked Read 9 Sent Done Done false
29431 9068 Acked Write 2 Sent Done Done false
29432 9069 Acked Read 1 Sent Done Done false
29433 9070 Acked Write 10 Sent Done Done false
29434 9071 Acked Read 8 Sent Done Done false
29435 9072 Acked Write 9 Sent Done Done false
29436 9073 Acked Write 4 Sent Done Done false
29437 9074 Acked Read 9 Sent Done Done false
29438 9075 Acked Read 6 Sent Done Done false
29439 9076 Acked Write 4 Sent Done Done false
29440 9077 Acked Write 4 Sent Done Done false
29441 9078 Acked Write 7 Sent Done Done false
29442 9079 Acked Read 10 Sent Done Done false
29443 9080 Acked Read 5 Sent Done Done false
29444 9081 Acked Flush 0 Sent Done Done false
29445 9082 Acked Write 4 Sent Done Done false
29446 9083 Acked Read 9 Sent Done Done false
29447 9084 Acked Write 4 Sent Done Done false
29448 9085 Acked Write 5 Sent Done Done false
29449 9086 Acked Read 7 Sent Done Done false
29450 9087 Acked Flush 0 Sent Done Done false
29451 9088 Acked Write 4 Sent Done Done false
29452 9089 Acked Write 4 Sent Done Done false
29453 9090 Acked Write 2 Sent Done Done false
29454 9091 Acked Read 8 Sent Done Done false
29455 9092 Acked Write 6 Sent Done Done false
29456 9093 Acked Read 10 Sent Done Done false
29457 9094 Acked Write 3 Sent Done Done false
29458 9095 Acked Write 1 Sent Done Done false
29459 9096 Acked Read 5 Sent Done Done false
29460 9097 Acked Write 9 Sent Done Done false
29461 9098 Acked Write 7 Sent Done Done false
29462 9099 Acked Read 10 Sent Done Done false
29463 9100 Acked Read 2 Sent Done Done false
29464 9101 Acked Write 6 Sent Done Done false
29465 9102 Acked Write 1 Sent Done Done false
29466 9103 Acked Read 5 Sent Done Done false
29467 9104 Acked Write 3 Sent Done Done false
29468 9105 Acked Write 10 Sent Done Done false
29469 9106 Acked Write 1 Sent Done Done false
29470 9107 Acked Flush 0 Sent Done Done false
29471 9108 Acked Read 10 Sent Done Done false
29472 9109 Acked Read 6 Sent Done Done false
29473 9110 Acked Flush 0 Sent Done Done false
29474 9111 Acked Read 9 Sent Done Done false
29475 9112 Acked Write 8 Sent Done Done false
29476 9113 Acked Read 2 Sent Done Done false
29477 9114 Acked Read 4 Sent Done Done false
29478 9115 Acked Read 8 Sent Done Done false
29479 9116 Acked Write 10 Sent Done Done false
29480 9117 Acked Read 10 Sent Done Done false
29481 9118 Acked Write 3 Sent Done Done false
29482 9119 Acked Read 6 Sent Done Done false
29483 9120 Acked Read 5 Sent Done Done false
29484 9121 Acked Read 1 Sent Done Done false
29485 9122 Acked Read 5 Sent Done Done false
29486 9123 Acked Flush 0 Sent Done Done false
29487 9124 Acked Read 1 Sent Done Done false
29488 9125 Acked Write 1 Sent Done Done false
29489 9126 Acked Read 2 Sent Done Done false
29490 9127 Acked Read 9 Sent Done Done false
29491 9128 Acked Read 2 Sent Done Done false
29492 9129 Acked Write 5 Sent Done Done false
29493 9130 Acked Read 3 Sent Done Done false
29494 9131 Acked Read 8 Sent Done Done false
29495 9132 Acked Read 9 Sent Done Done false
29496 9133 Acked Read 2 Sent Done Done false
29497 9134 Acked Write 9 Sent Done Done false
29498 9135 Acked Read 1 Sent Done Done false
29499 9136 Acked Read 8 Sent Done Done false
29500 9137 Acked Flush 0 Sent Done Done false
29501 9138 Acked Write 8 Sent Done Done false
29502 9139 Acked Write 5 Sent Done Done false
29503 9140 Acked Read 6 Sent Done Done false
29504 9141 Acked Flush 0 Sent Done Done false
29505 9142 Acked Write 4 Sent Done Done false
29506 9143 Acked Read 2 Sent Done Done false
29507 9144 Acked Write 3 Sent Done Done false
29508 9145 Acked Write 2 Sent Done Done false
29509 9146 Acked Read 6 Sent Done Done false
29510 9147 Acked Read 2 Sent Done Done false
29511 9148 Acked Flush 0 Sent Done Done false
29512 9149 Acked Read 6 Sent Done Done false
29513 9150 Acked Read 2 Sent Done Done false
29514 STATES DS:0 DS:1 DS:2 TOTAL
29515 Sent 301 0 0 301
29516 Done 0 301 301 602
29517 Skipped 0 0 0 0
29518 Error 0 0 0 0
29519 Last Flush: Some(JobId(8849)) Some(JobId(9148)) Some(JobId(9148))
29520 Downstairs last five completed: 8849 8848 8847 8846 8845
29521 Upstairs last five completed: 9150 9149 9148 9147 9146
29522 CLIENT: Up:0 ds:301 act:2
295232025-01-28T22:00:45.077ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
295242025-01-28T22:00:45.903ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
295252025-01-28T22:00:47.080ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
295262025-01-28T22:00:47.906ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
29527 ----------------------------------------------------------------
29528 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29529 States: Offline Active Active
29530 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29531 8850 Acked Flush 0 Sent Done Done false
29532 8851 Acked Read 7 Sent Done Done false
29533 8852 Acked Read 8 Sent Done Done false
29534 8853 Acked Read 7 Sent Done Done false
29535 8854 Acked Write 1 Sent Done Done false
29536 8855 Acked Write 10 Sent Done Done false
29537 8856 Acked Read 3 Sent Done Done false
29538 8857 Acked Read 2 Sent Done Done false
29539 8858 Acked Write 3 Sent Done Done false
29540 8859 Acked Write 5 Sent Done Done false
29541 8860 Acked Write 6 Sent Done Done false
29542 8861 Acked Read 8 Sent Done Done false
29543 8862 Acked Read 7 Sent Done Done false
29544 8863 Acked Read 4 Sent Done Done false
29545 8864 Acked Write 5 Sent Done Done false
29546 8865 Acked Read 4 Sent Done Done false
29547 8866 Acked Read 5 Sent Done Done false
29548 8867 Acked Write 9 Sent Done Done false
29549 8868 Acked Write 10 Sent Done Done false
29550 8869 Acked Flush 0 Sent Done Done false
29551 8870 Acked Read 3 Sent Done Done false
29552 8871 Acked Write 5 Sent Done Done false
29553 8872 Acked Flush 0 Sent Done Done false
29554 8873 Acked Flush 0 Sent Done Done false
29555 8874 Acked Read 5 Sent Done Done false
29556 8875 Acked Write 6 Sent Done Done false
29557 8876 Acked Read 3 Sent Done Done false
29558 8877 Acked Flush 0 Sent Done Done false
29559 8878 Acked Read 6 Sent Done Done false
29560 8879 Acked Write 3 Sent Done Done false
29561 8880 Acked Read 6 Sent Done Done false
29562 8881 Acked Write 6 Sent Done Done false
29563 8882 Acked Write 10 Sent Done Done false
29564 8883 Acked Write 4 Sent Done Done false
29565 8884 Acked Read 5 Sent Done Done false
29566 8885 Acked Write 1 Sent Done Done false
29567 8886 Acked Read 4 Sent Done Done false
29568 8887 Acked Read 6 Sent Done Done false
29569 8888 Acked Read 6 Sent Done Done false
29570 8889 Acked Write 3 Sent Done Done false
29571 8890 Acked Read 8 Sent Done Done false
29572 8891 Acked Read 10 Sent Done Done false
29573 8892 Acked Write 4 Sent Done Done false
29574 8893 Acked Read 4 Sent Done Done false
29575 8894 Acked Write 7 Sent Done Done false
29576 8895 Acked Write 8 Sent Done Done false
29577 8896 Acked Write 4 Sent Done Done false
29578 8897 Acked Flush 0 Sent Done Done false
29579 8898 Acked Write 10 Sent Done Done false
29580 8899 Acked Write 10 Sent Done Done false
29581 8900 Acked Read 8 Sent Done Done false
29582 8901 Acked Read 10 Sent Done Done false
29583 8902 Acked Read 1 Sent Done Done false
29584 8903 Acked Read 10 Sent Done Done false
29585 8904 Acked Read 7 Sent Done Done false
29586 8905 Acked Read 8 Sent Done Done false
29587 8906 Acked Read 9 Sent Done Done false
29588 8907 Acked Read 10 Sent Done Done false
29589 8908 Acked Read 8 Sent Done Done false
29590 8909 Acked Read 10 Sent Done Done false
29591 8910 Acked Write 9 Sent Done Done false
29592 8911 Acked Write 8 Sent Done Done false
29593 8912 Acked Read 7 Sent Done Done false
29594 8913 Acked Read 1 Sent Done Done false
29595 8914 Acked Read 5 Sent Done Done false
29596 8915 Acked Read 5 Sent Done Done false
29597 8916 Acked Read 8 Sent Done Done false
29598 8917 Acked Write 3 Sent Done Done false
29599 8918 Acked Write 3 Sent Done Done false
29600 8919 Acked Read 4 Sent Done Done false
29601 8920 Acked Read 6 Sent Done Done false
29602 8921 Acked Read 3 Sent Done Done false
29603 8922 Acked Write 8 Sent Done Done false
29604 8923 Acked Write 8 Sent Done Done false
29605 8924 Acked Write 8 Sent Done Done false
29606 8925 Acked Read 8 Sent Done Done false
29607 8926 Acked Read 10 Sent Done Done false
29608 8927 Acked Read 9 Sent Done Done false
29609 8928 Acked Read 1 Sent Done Done false
29610 8929 Acked Write 3 Sent Done Done false
29611 8930 Acked Flush 0 Sent Done Done false
29612 8931 Acked Write 2 Sent Done Done false
29613 8932 Acked Write 5 Sent Done Done false
29614 8933 Acked Write 3 Sent Done Done false
29615 8934 Acked Read 10 Sent Done Done false
29616 8935 Acked Read 8 Sent Done Done false
29617 8936 Acked Read 2 Sent Done Done false
29618 8937 Acked Read 3 Sent Done Done false
29619 8938 Acked Write 10 Sent Done Done false
29620 8939 Acked Read 9 Sent Done Done false
29621 8940 Acked Flush 0 Sent Done Done false
29622 8941 Acked Read 2 Sent Done Done false
29623 8942 Acked Write 3 Sent Done Done false
29624 8943 Acked Read 5 Sent Done Done false
29625 8944 Acked Read 5 Sent Done Done false
29626 8945 Acked Flush 0 Sent Done Done false
29627 8946 Acked Read 2 Sent Done Done false
29628 8947 Acked Write 2 Sent Done Done false
29629 8948 Acked Flush 0 Sent Done Done false
29630 8949 Acked Read 2 Sent Done Done false
29631 8950 Acked Write 5 Sent Done Done false
29632 8951 Acked Read 3 Sent Done Done false
29633 8952 Acked Read 8 Sent Done Done false
29634 8953 Acked Write 9 Sent Done Done false
29635 8954 Acked Write 8 Sent Done Done false
29636 8955 Acked Read 7 Sent Done Done false
29637 8956 Acked Write 10 Sent Done Done false
29638 8957 Acked Read 9 Sent Done Done false
29639 8958 Acked Write 8 Sent Done Done false
29640 8959 Acked Write 2 Sent Done Done false
29641 8960 Acked Read 5 Sent Done Done false
29642 8961 Acked Read 6 Sent Done Done false
29643 8962 Acked Write 8 Sent Done Done false
29644 8963 Acked Write 10 Sent Done Done false
29645 8964 Acked Write 8 Sent Done Done false
29646 8965 Acked Write 4 Sent Done Done false
29647 8966 Acked Write 8 Sent Done Done false
29648 8967 Acked Read 8 Sent Done Done false
29649 8968 Acked Read 5 Sent Done Done false
29650 8969 Acked Write 8 Sent Done Done false
29651 8970 Acked Flush 0 Sent Done Done false
29652 8971 Acked Write 10 Sent Done Done false
29653 8972 Acked Read 8 Sent Done Done false
29654 8973 Acked Write 9 Sent Done Done false
29655 8974 Acked Read 5 Sent Done Done false
29656 8975 Acked Write 4 Sent Done Done false
29657 8976 Acked Read 4 Sent Done Done false
29658 8977 Acked Read 8 Sent Done Done false
29659 8978 Acked Write 8 Sent Done Done false
29660 8979 Acked Write 4 Sent Done Done false
29661 8980 Acked Write 10 Sent Done Done false
29662 8981 Acked Read 5 Sent Done Done false
29663 8982 Acked Read 8 Sent Done Done false
29664 8983 Acked Flush 0 Sent Done Done false
29665 8984 Acked Read 1 Sent Done Done false
29666 8985 Acked Write 9 Sent Done Done false
29667 8986 Acked Read 2 Sent Done Done false
29668 8987 Acked Write 4 Sent Done Done false
29669 8988 Acked Read 8 Sent Done Done false
29670 8989 Acked Read 7 Sent Done Done false
29671 8990 Acked Read 4 Sent Done Done false
29672 8991 Acked Read 10 Sent Done Done false
29673 8992 Acked Read 1 Sent Done Done false
29674 8993 Acked Read 6 Sent Done Done false
29675 8994 Acked Flush 0 Sent Done Done false
29676 8995 Acked Write 1 Sent Done Done false
29677 8996 Acked Write 8 Sent Done Done false
29678 8997 Acked Write 10 Sent Done Done false
29679 8998 Acked Read 4 Sent Done Done false
29680 8999 Acked Read 9 Sent Done Done false
29681 9000 Acked Read 8 Sent Done Done false
29682 9001 Acked Read 6 Sent Done Done false
29683 9002 Acked Write 3 Sent Done Done false
29684 9003 Acked Write 9 Sent Done Done false
29685 9004 Acked Write 10 Sent Done Done false
29686 9005 Acked Write 5 Sent Done Done false
29687 9006 Acked Flush 0 Sent Done Done false
29688 9007 Acked Read 2 Sent Done Done false
29689 9008 Acked Write 3 Sent Done Done false
29690 9009 Acked Write 7 Sent Done Done false
29691 9010 Acked Write 4 Sent Done Done false
29692 9011 Acked Write 1 Sent Done Done false
29693 9012 Acked Read 2 Sent Done Done false
29694 9013 Acked Flush 0 Sent Done Done false
29695 9014 Acked Read 5 Sent Done Done false
29696 9015 Acked Read 10 Sent Done Done false
29697 9016 Acked Read 5 Sent Done Done false
29698 9017 Acked Write 3 Sent Done Done false
29699 9018 Acked Read 4 Sent Done Done false
29700 9019 Acked Read 9 Sent Done Done false
29701 9020 Acked Write 7 Sent Done Done false
29702 9021 Acked Read 10 Sent Done Done false
29703 9022 Acked Write 5 Sent Done Done false
29704 9023 Acked Flush 0 Sent Done Done false
29705 9024 Acked Read 6 Sent Done Done false
29706 9025 Acked Read 6 Sent Done Done false
29707 9026 Acked Read 6 Sent Done Done false
29708 9027 Acked Write 5 Sent Done Done false
29709 9028 Acked Write 2 Sent Done Done false
29710 9029 Acked Flush 0 Sent Done Done false
29711 9030 Acked Write 3 Sent Done Done false
29712 9031 Acked Write 8 Sent Done Done false
29713 9032 Acked Write 5 Sent Done Done false
29714 9033 Acked Flush 0 Sent Done Done false
29715 9034 Acked Read 4 Sent Done Done false
29716 9035 Acked Write 4 Sent Done Done false
29717 9036 Acked Read 9 Sent Done Done false
29718 9037 Acked Read 9 Sent Done Done false
29719 9038 Acked Write 8 Sent Done Done false
29720 9039 Acked Write 7 Sent Done Done false
29721 9040 Acked Write 8 Sent Done Done false
29722 9041 Acked Write 9 Sent Done Done false
29723 9042 Acked Write 3 Sent Done Done false
29724 9043 Acked Write 6 Sent Done Done false
29725 9044 Acked Read 2 Sent Done Done false
29726 9045 Acked Write 10 Sent Done Done false
29727 9046 Acked Read 3 Sent Done Done false
29728 9047 Acked Read 10 Sent Done Done false
29729 9048 Acked Flush 0 Sent Done Done false
29730 9049 Acked Write 10 Sent Done Done false
29731 9050 Acked Read 7 Sent Done Done false
29732 9051 Acked Write 3 Sent Done Done false
29733 9052 Acked Read 6 Sent Done Done false
29734 9053 Acked Read 7 Sent Done Done false
29735 9054 Acked Read 1 Sent Done Done false
29736 9055 Acked Read 7 Sent Done Done false
29737 9056 Acked Write 9 Sent Done Done false
29738 9057 Acked Read 4 Sent Done Done false
29739 9058 Acked Read 9 Sent Done Done false
29740 9059 Acked Write 8 Sent Done Done false
29741 9060 Acked Write 3 Sent Done Done false
29742 9061 Acked Read 6 Sent Done Done false
29743 9062 Acked Read 9 Sent Done Done false
29744 9063 Acked Read 3 Sent Done Done false
29745 9064 Acked Write 4 Sent Done Done false
29746 9065 Acked Read 10 Sent Done Done false
29747 9066 Acked Read 10 Sent Done Done false
29748 9067 Acked Read 9 Sent Done Done false
29749 9068 Acked Write 2 Sent Done Done false
29750 9069 Acked Read 1 Sent Done Done false
29751 9070 Acked Write 10 Sent Done Done false
29752 9071 Acked Read 8 Sent Done Done false
29753 9072 Acked Write 9 Sent Done Done false
29754 9073 Acked Write 4 Sent Done Done false
29755 9074 Acked Read 9 Sent Done Done false
29756 9075 Acked Read 6 Sent Done Done false
29757 9076 Acked Write 4 Sent Done Done false
29758 9077 Acked Write 4 Sent Done Done false
29759 9078 Acked Write 7 Sent Done Done false
29760 9079 Acked Read 10 Sent Done Done false
29761 9080 Acked Read 5 Sent Done Done false
29762 9081 Acked Flush 0 Sent Done Done false
29763 9082 Acked Write 4 Sent Done Done false
29764 9083 Acked Read 9 Sent Done Done false
29765 9084 Acked Write 4 Sent Done Done false
29766 9085 Acked Write 5 Sent Done Done false
29767 9086 Acked Read 7 Sent Done Done false
29768 9087 Acked Flush 0 Sent Done Done false
29769 9088 Acked Write 4 Sent Done Done false
29770 9089 Acked Write 4 Sent Done Done false
29771 9090 Acked Write 2 Sent Done Done false
29772 9091 Acked Read 8 Sent Done Done false
29773 9092 Acked Write 6 Sent Done Done false
29774 9093 Acked Read 10 Sent Done Done false
29775 9094 Acked Write 3 Sent Done Done false
29776 9095 Acked Write 1 Sent Done Done false
29777 9096 Acked Read 5 Sent Done Done false
29778 9097 Acked Write 9 Sent Done Done false
29779 9098 Acked Write 7 Sent Done Done false
29780 9099 Acked Read 10 Sent Done Done false
29781 9100 Acked Read 2 Sent Done Done false
29782 9101 Acked Write 6 Sent Done Done false
29783 9102 Acked Write 1 Sent Done Done false
29784 9103 Acked Read 5 Sent Done Done false
29785 9104 Acked Write 3 Sent Done Done false
29786 9105 Acked Write 10 Sent Done Done false
29787 9106 Acked Write 1 Sent Done Done false
29788 9107 Acked Flush 0 Sent Done Done false
29789 9108 Acked Read 10 Sent Done Done false
29790 9109 Acked Read 6 Sent Done Done false
29791 9110 Acked Flush 0 Sent Done Done false
29792 9111 Acked Read 9 Sent Done Done false
29793 9112 Acked Write 8 Sent Done Done false
29794 9113 Acked Read 2 Sent Done Done false
29795 9114 Acked Read 4 Sent Done Done false
29796 9115 Acked Read 8 Sent Done Done false
29797 9116 Acked Write 10 Sent Done Done false
29798 9117 Acked Read 10 Sent Done Done false
29799 9118 Acked Write 3 Sent Done Done false
29800 9119 Acked Read 6 Sent Done Done false
29801 9120 Acked Read 5 Sent Done Done false
29802 9121 Acked Read 1 Sent Done Done false
29803 9122 Acked Read 5 Sent Done Done false
29804 9123 Acked Flush 0 Sent Done Done false
29805 9124 Acked Read 1 Sent Done Done false
29806 9125 Acked Write 1 Sent Done Done false
29807 9126 Acked Read 2 Sent Done Done false
29808 9127 Acked Read 9 Sent Done Done false
29809 9128 Acked Read 2 Sent Done Done false
29810 9129 Acked Write 5 Sent Done Done false
29811 9130 Acked Read 3 Sent Done Done false
29812 9131 Acked Read 8 Sent Done Done false
29813 9132 Acked Read 9 Sent Done Done false
29814 9133 Acked Read 2 Sent Done Done false
29815 9134 Acked Write 9 Sent Done Done false
29816 9135 Acked Read 1 Sent Done Done false
29817 9136 Acked Read 8 Sent Done Done false
29818 9137 Acked Flush 0 Sent Done Done false
29819 9138 Acked Write 8 Sent Done Done false
29820 9139 Acked Write 5 Sent Done Done false
29821 9140 Acked Read 6 Sent Done Done false
29822 9141 Acked Flush 0 Sent Done Done false
29823 9142 Acked Write 4 Sent Done Done false
29824 9143 Acked Read 2 Sent Done Done false
29825 9144 Acked Write 3 Sent Done Done false
29826 9145 Acked Write 2 Sent Done Done false
29827 9146 Acked Read 6 Sent Done Done false
29828 9147 Acked Read 2 Sent Done Done false
29829 9148 Acked Flush 0 Sent Done Done false
29830 9149 Acked Read 6 Sent Done Done false
29831 9150 Acked Read 2 Sent Done Done false
29832 STATES DS:0 DS:1 DS:2 TOTAL
29833 Sent 301 0 0 301
29834 Done 0 301 301 602
29835 Skipped 0 0 0 0
29836 Error 0 0 0 0
29837 Last Flush: Some(JobId(8849)) Some(JobId(9148)) Some(JobId(9148))
29838 Downstairs last five completed: 8849 8848 8847 8846 8845
29839 Upstairs last five completed: 9150 9149 9148 9147 9146
29840 CLIENT: Up:0 ds:301 act:2
298412025-01-28T22:00:49.079ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
298422025-01-28T22:00:49.910ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
298432025-01-28T22:00:50.325ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
298442025-01-28T22:00:50.325ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
298452025-01-28T22:00:50.325ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
298462025-01-28T22:00:50.325ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
298472025-01-28T22:00:50.325ZINFOcrucible: client task is exiting = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
298482025-01-28T22:00:50.911ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
298492025-01-28T22:00:52.081ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
29850 ----------------------------------------------------------------
29851 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29852 States: Offline Active Active
29853 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29854 8850 Acked Flush 0 Sent Done Done false
29855 8851 Acked Read 7 Sent Done Done false
29856 8852 Acked Read 8 Sent Done Done false
29857 8853 Acked Read 7 Sent Done Done false
29858 8854 Acked Write 1 Sent Done Done false
29859 8855 Acked Write 10 Sent Done Done false
29860 8856 Acked Read 3 Sent Done Done false
29861 8857 Acked Read 2 Sent Done Done false
29862 8858 Acked Write 3 Sent Done Done false
29863 8859 Acked Write 5 Sent Done Done false
29864 8860 Acked Write 6 Sent Done Done false
29865 8861 Acked Read 8 Sent Done Done false
29866 8862 Acked Read 7 Sent Done Done false
29867 8863 Acked Read 4 Sent Done Done false
29868 8864 Acked Write 5 Sent Done Done false
29869 8865 Acked Read 4 Sent Done Done false
29870 8866 Acked Read 5 Sent Done Done false
29871 8867 Acked Write 9 Sent Done Done false
29872 8868 Acked Write 10 Sent Done Done false
29873 8869 Acked Flush 0 Sent Done Done false
29874 8870 Acked Read 3 Sent Done Done false
29875 8871 Acked Write 5 Sent Done Done false
29876 8872 Acked Flush 0 Sent Done Done false
29877 8873 Acked Flush 0 Sent Done Done false
29878 8874 Acked Read 5 Sent Done Done false
29879 8875 Acked Write 6 Sent Done Done false
29880 8876 Acked Read 3 Sent Done Done false
29881 8877 Acked Flush 0 Sent Done Done false
29882 8878 Acked Read 6 Sent Done Done false
29883 8879 Acked Write 3 Sent Done Done false
29884 8880 Acked Read 6 Sent Done Done false
29885 8881 Acked Write 6 Sent Done Done false
29886 8882 Acked Write 10 Sent Done Done false
29887 8883 Acked Write 4 Sent Done Done false
29888 8884 Acked Read 5 Sent Done Done false
29889 8885 Acked Write 1 Sent Done Done false
29890 8886 Acked Read 4 Sent Done Done false
29891 8887 Acked Read 6 Sent Done Done false
29892 8888 Acked Read 6 Sent Done Done false
29893 8889 Acked Write 3 Sent Done Done false
29894 8890 Acked Read 8 Sent Done Done false
29895 8891 Acked Read 10 Sent Done Done false
29896 8892 Acked Write 4 Sent Done Done false
29897 8893 Acked Read 4 Sent Done Done false
29898 8894 Acked Write 7 Sent Done Done false
29899 8895 Acked Write 8 Sent Done Done false
29900 8896 Acked Write 4 Sent Done Done false
29901 8897 Acked Flush 0 Sent Done Done false
29902 8898 Acked Write 10 Sent Done Done false
29903 8899 Acked Write 10 Sent Done Done false
29904 8900 Acked Read 8 Sent Done Done false
29905 8901 Acked Read 10 Sent Done Done false
29906 8902 Acked Read 1 Sent Done Done false
29907 8903 Acked Read 10 Sent Done Done false
29908 8904 Acked Read 7 Sent Done Done false
29909 8905 Acked Read 8 Sent Done Done false
29910 8906 Acked Read 9 Sent Done Done false
29911 8907 Acked Read 10 Sent Done Done false
29912 8908 Acked Read 8 Sent Done Done false
29913 8909 Acked Read 10 Sent Done Done false
29914 8910 Acked Write 9 Sent Done Done false
29915 8911 Acked Write 8 Sent Done Done false
29916 8912 Acked Read 7 Sent Done Done false
29917 8913 Acked Read 1 Sent Done Done false
29918 8914 Acked Read 5 Sent Done Done false
29919 8915 Acked Read 5 Sent Done Done false
29920 8916 Acked Read 8 Sent Done Done false
29921 8917 Acked Write 3 Sent Done Done false
29922 8918 Acked Write 3 Sent Done Done false
29923 8919 Acked Read 4 Sent Done Done false
29924 8920 Acked Read 6 Sent Done Done false
29925 8921 Acked Read 3 Sent Done Done false
29926 8922 Acked Write 8 Sent Done Done false
29927 8923 Acked Write 8 Sent Done Done false
29928 8924 Acked Write 8 Sent Done Done false
29929 8925 Acked Read 8 Sent Done Done false
29930 8926 Acked Read 10 Sent Done Done false
29931 8927 Acked Read 9 Sent Done Done false
29932 8928 Acked Read 1 Sent Done Done false
29933 8929 Acked Write 3 Sent Done Done false
29934 8930 Acked Flush 0 Sent Done Done false
29935 8931 Acked Write 2 Sent Done Done false
29936 8932 Acked Write 5 Sent Done Done false
29937 8933 Acked Write 3 Sent Done Done false
29938 8934 Acked Read 10 Sent Done Done false
29939 8935 Acked Read 8 Sent Done Done false
29940 8936 Acked Read 2 Sent Done Done false
29941 8937 Acked Read 3 Sent Done Done false
29942 8938 Acked Write 10 Sent Done Done false
29943 8939 Acked Read 9 Sent Done Done false
29944 8940 Acked Flush 0 Sent Done Done false
29945 8941 Acked Read 2 Sent Done Done false
29946 8942 Acked Write 3 Sent Done Done false
29947 8943 Acked Read 5 Sent Done Done false
29948 8944 Acked Read 5 Sent Done Done false
29949 8945 Acked Flush 0 Sent Done Done false
29950 8946 Acked Read 2 Sent Done Done false
29951 8947 Acked Write 2 Sent Done Done false
29952 8948 Acked Flush 0 Sent Done Done false
29953 8949 Acked Read 2 Sent Done Done false
29954 8950 Acked Write 5 Sent Done Done false
29955 8951 Acked Read 3 Sent Done Done false
29956 8952 Acked Read 8 Sent Done Done false
29957 8953 Acked Write 9 Sent Done Done false
29958 8954 Acked Write 8 Sent Done Done false
29959 8955 Acked Read 7 Sent Done Done false
29960 8956 Acked Write 10 Sent Done Done false
29961 8957 Acked Read 9 Sent Done Done false
29962 8958 Acked Write 8 Sent Done Done false
29963 8959 Acked Write 2 Sent Done Done false
29964 8960 Acked Read 5 Sent Done Done false
29965 8961 Acked Read 6 Sent Done Done false
29966 8962 Acked Write 8 Sent Done Done false
29967 8963 Acked Write 10 Sent Done Done false
29968 8964 Acked Write 8 Sent Done Done false
29969 8965 Acked Write 4 Sent Done Done false
29970 8966 Acked Write 8 Sent Done Done false
29971 8967 Acked Read 8 Sent Done Done false
29972 8968 Acked Read 5 Sent Done Done false
29973 8969 Acked Write 8 Sent Done Done false
29974 8970 Acked Flush 0 Sent Done Done false
29975 8971 Acked Write 10 Sent Done Done false
29976 8972 Acked Read 8 Sent Done Done false
29977 8973 Acked Write 9 Sent Done Done false
29978 8974 Acked Read 5 Sent Done Done false
29979 8975 Acked Write 4 Sent Done Done false
29980 8976 Acked Read 4 Sent Done Done false
29981 8977 Acked Read 8 Sent Done Done false
29982 8978 Acked Write 8 Sent Done Done false
29983 8979 Acked Write 4 Sent Done Done false
29984 8980 Acked Write 10 Sent Done Done false
29985 8981 Acked Read 5 Sent Done Done false
29986 8982 Acked Read 8 Sent Done Done false
29987 8983 Acked Flush 0 Sent Done Done false
29988 8984 Acked Read 1 Sent Done Done false
29989 8985 Acked Write 9 Sent Done Done false
29990 8986 Acked Read 2 Sent Done Done false
29991 8987 Acked Write 4 Sent Done Done false
29992 8988 Acked Read 8 Sent Done Done false
29993 8989 Acked Read 7 Sent Done Done false
29994 8990 Acked Read 4 Sent Done Done false
29995 8991 Acked Read 10 Sent Done Done false
29996 8992 Acked Read 1 Sent Done Done false
29997 8993 Acked Read 6 Sent Done Done false
29998 8994 Acked Flush 0 Sent Done Done false
29999 8995 Acked Write 1 Sent Done Done false
30000 8996 Acked Write 8 Sent Done Done false
30001 8997 Acked Write 10 Sent Done Done false
30002 8998 Acked Read 4 Sent Done Done false
30003 8999 Acked Read 9 Sent Done Done false
30004 9000 Acked Read 8 Sent Done Done false
30005 9001 Acked Read 6 Sent Done Done false
30006 9002 Acked Write 3 Sent Done Done false
30007 9003 Acked Write 9 Sent Done Done false
30008 9004 Acked Write 10 Sent Done Done false
30009 9005 Acked Write 5 Sent Done Done false
30010 9006 Acked Flush 0 Sent Done Done false
30011 9007 Acked Read 2 Sent Done Done false
30012 9008 Acked Write 3 Sent Done Done false
30013 9009 Acked Write 7 Sent Done Done false
30014 9010 Acked Write 4 Sent Done Done false
30015 9011 Acked Write 1 Sent Done Done false
30016 9012 Acked Read 2 Sent Done Done false
30017 9013 Acked Flush 0 Sent Done Done false
30018 9014 Acked Read 5 Sent Done Done false
30019 9015 Acked Read 10 Sent Done Done false
30020 9016 Acked Read 5 Sent Done Done false
30021 9017 Acked Write 3 Sent Done Done false
30022 9018 Acked Read 4 Sent Done Done false
30023 9019 Acked Read 9 Sent Done Done false
30024 9020 Acked Write 7 Sent Done Done false
30025 9021 Acked Read 10 Sent Done Done false
30026 9022 Acked Write 5 Sent Done Done false
30027 9023 Acked Flush 0 Sent Done Done false
30028 9024 Acked Read 6 Sent Done Done false
30029 9025 Acked Read 6 Sent Done Done false
30030 9026 Acked Read 6 Sent Done Done false
30031 9027 Acked Write 5 Sent Done Done false
30032 9028 Acked Write 2 Sent Done Done false
30033 9029 Acked Flush 0 Sent Done Done false
30034 9030 Acked Write 3 Sent Done Done false
30035 9031 Acked Write 8 Sent Done Done false
30036 9032 Acked Write 5 Sent Done Done false
30037 9033 Acked Flush 0 Sent Done Done false
30038 9034 Acked Read 4 Sent Done Done false
30039 9035 Acked Write 4 Sent Done Done false
30040 9036 Acked Read 9 Sent Done Done false
30041 9037 Acked Read 9 Sent Done Done false
30042 9038 Acked Write 8 Sent Done Done false
30043 9039 Acked Write 7 Sent Done Done false
30044 9040 Acked Write 8 Sent Done Done false
30045 9041 Acked Write 9 Sent Done Done false
30046 9042 Acked Write 3 Sent Done Done false
30047 9043 Acked Write 6 Sent Done Done false
30048 9044 Acked Read 2 Sent Done Done false
30049 9045 Acked Write 10 Sent Done Done false
30050 9046 Acked Read 3 Sent Done Done false
30051 9047 Acked Read 10 Sent Done Done false
30052 9048 Acked Flush 0 Sent Done Done false
30053 9049 Acked Write 10 Sent Done Done false
30054 9050 Acked Read 7 Sent Done Done false
30055 9051 Acked Write 3 Sent Done Done false
30056 9052 Acked Read 6 Sent Done Done false
30057 9053 Acked Read 7 Sent Done Done false
30058 9054 Acked Read 1 Sent Done Done false
30059 9055 Acked Read 7 Sent Done Done false
30060 9056 Acked Write 9 Sent Done Done false
30061 9057 Acked Read 4 Sent Done Done false
30062 9058 Acked Read 9 Sent Done Done false
30063 9059 Acked Write 8 Sent Done Done false
30064 9060 Acked Write 3 Sent Done Done false
30065 9061 Acked Read 6 Sent Done Done false
30066 9062 Acked Read 9 Sent Done Done false
30067 9063 Acked Read 3 Sent Done Done false
30068 9064 Acked Write 4 Sent Done Done false
30069 9065 Acked Read 10 Sent Done Done false
30070 9066 Acked Read 10 Sent Done Done false
30071 9067 Acked Read 9 Sent Done Done false
30072 9068 Acked Write 2 Sent Done Done false
30073 9069 Acked Read 1 Sent Done Done false
30074 9070 Acked Write 10 Sent Done Done false
30075 9071 Acked Read 8 Sent Done Done false
30076 9072 Acked Write 9 Sent Done Done false
30077 9073 Acked Write 4 Sent Done Done false
30078 9074 Acked Read 9 Sent Done Done false
30079 9075 Acked Read 6 Sent Done Done false
30080 9076 Acked Write 4 Sent Done Done false
30081 9077 Acked Write 4 Sent Done Done false
30082 9078 Acked Write 7 Sent Done Done false
30083 9079 Acked Read 10 Sent Done Done false
30084 9080 Acked Read 5 Sent Done Done false
30085 9081 Acked Flush 0 Sent Done Done false
30086 9082 Acked Write 4 Sent Done Done false
30087 9083 Acked Read 9 Sent Done Done false
30088 9084 Acked Write 4 Sent Done Done false
30089 9085 Acked Write 5 Sent Done Done false
30090 9086 Acked Read 7 Sent Done Done false
30091 9087 Acked Flush 0 Sent Done Done false
30092 9088 Acked Write 4 Sent Done Done false
30093 9089 Acked Write 4 Sent Done Done false
30094 9090 Acked Write 2 Sent Done Done false
30095 9091 Acked Read 8 Sent Done Done false
30096 9092 Acked Write 6 Sent Done Done false
30097 9093 Acked Read 10 Sent Done Done false
30098 9094 Acked Write 3 Sent Done Done false
30099 9095 Acked Write 1 Sent Done Done false
30100 9096 Acked Read 5 Sent Done Done false
30101 9097 Acked Write 9 Sent Done Done false
30102 9098 Acked Write 7 Sent Done Done false
30103 9099 Acked Read 10 Sent Done Done false
30104 9100 Acked Read 2 Sent Done Done false
30105 9101 Acked Write 6 Sent Done Done false
30106 9102 Acked Write 1 Sent Done Done false
30107 9103 Acked Read 5 Sent Done Done false
30108 9104 Acked Write 3 Sent Done Done false
30109 9105 Acked Write 10 Sent Done Done false
30110 9106 Acked Write 1 Sent Done Done false
30111 9107 Acked Flush 0 Sent Done Done false
30112 9108 Acked Read 10 Sent Done Done false
30113 9109 Acked Read 6 Sent Done Done false
30114 9110 Acked Flush 0 Sent Done Done false
30115 9111 Acked Read 9 Sent Done Done false
30116 9112 Acked Write 8 Sent Done Done false
30117 9113 Acked Read 2 Sent Done Done false
30118 9114 Acked Read 4 Sent Done Done false
30119 9115 Acked Read 8 Sent Done Done false
30120 9116 Acked Write 10 Sent Done Done false
30121 9117 Acked Read 10 Sent Done Done false
30122 9118 Acked Write 3 Sent Done Done false
30123 9119 Acked Read 6 Sent Done Done false
30124 9120 Acked Read 5 Sent Done Done false
30125 9121 Acked Read 1 Sent Done Done false
30126 9122 Acked Read 5 Sent Done Done false
30127 9123 Acked Flush 0 Sent Done Done false
30128 9124 Acked Read 1 Sent Done Done false
30129 9125 Acked Write 1 Sent Done Done false
30130 9126 Acked Read 2 Sent Done Done false
30131 9127 Acked Read 9 Sent Done Done false
30132 9128 Acked Read 2 Sent Done Done false
30133 9129 Acked Write 5 Sent Done Done false
30134 9130 Acked Read 3 Sent Done Done false
30135 9131 Acked Read 8 Sent Done Done false
30136 9132 Acked Read 9 Sent Done Done false
30137 9133 Acked Read 2 Sent Done Done false
30138 9134 Acked Write 9 Sent Done Done false
30139 9135 Acked Read 1 Sent Done Done false
30140 9136 Acked Read 8 Sent Done Done false
30141 9137 Acked Flush 0 Sent Done Done false
30142 9138 Acked Write 8 Sent Done Done false
30143 9139 Acked Write 5 Sent Done Done false
30144 9140 Acked Read 6 Sent Done Done false
30145 9141 Acked Flush 0 Sent Done Done false
30146 9142 Acked Write 4 Sent Done Done false
30147 9143 Acked Read 2 Sent Done Done false
30148 9144 Acked Write 3 Sent Done Done false
30149 9145 Acked Write 2 Sent Done Done false
30150 9146 Acked Read 6 Sent Done Done false
30151 9147 Acked Read 2 Sent Done Done false
30152 9148 Acked Flush 0 Sent Done Done false
30153 9149 Acked Read 6 Sent Done Done false
30154 9150 Acked Read 2 Sent Done Done false
30155 STATES DS:0 DS:1 DS:2 TOTAL
30156 Sent 301 0 0 301
30157 Done 0 301 301 602
30158 Skipped 0 0 0 0
30159 Error 0 0 0 0
30160 Last Flush: Some(JobId(8849)) Some(JobId(9148)) Some(JobId(9148))
30161 Downstairs last five completed: 8849 8848 8847 8846 8845
30162 Upstairs last five completed: 9150 9149 9148 9147 9146
30163 CLIENT: Up:0 ds:301 act:2
301642025-01-28T22:00:53.081ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
301652025-01-28T22:00:53.915ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
301662025-01-28T22:00:54.916ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
301672025-01-28T22:00:55.917ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
30168 [8810] got start action
30169 [8810] I am going to restart
30170 Starting downstairs at port 8810
30171 Make output file at "/tmp/dsc/downstairs-8810.txt"
30172 [8810][0] reports Starting
30173 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1229)
30174 [8810][0] reports Running
301752025-01-28T22:00:56.418ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
30176 ----------------------------------------------------------------
30177 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30178 States: Offline Active Active
30179 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30180 8850 Acked Flush 0 Sent Done Done false
30181 8851 Acked Read 7 Sent Done Done false
30182 8852 Acked Read 8 Sent Done Done false
30183 8853 Acked Read 7 Sent Done Done false
30184 8854 Acked Write 1 Sent Done Done false
30185 8855 Acked Write 10 Sent Done Done false
30186 8856 Acked Read 3 Sent Done Done false
30187 8857 Acked Read 2 Sent Done Done false
30188 8858 Acked Write 3 Sent Done Done false
30189 8859 Acked Write 5 Sent Done Done false
30190 8860 Acked Write 6 Sent Done Done false
30191 8861 Acked Read 8 Sent Done Done false
30192 8862 Acked Read 7 Sent Done Done false
30193 8863 Acked Read 4 Sent Done Done false
30194 8864 Acked Write 5 Sent Done Done false
30195 8865 Acked Read 4 Sent Done Done false
30196 8866 Acked Read 5 Sent Done Done false
30197 8867 Acked Write 9 Sent Done Done false
30198 8868 Acked Write 10 Sent Done Done false
30199 8869 Acked Flush 0 Sent Done Done false
30200 8870 Acked Read 3 Sent Done Done false
30201 8871 Acked Write 5 Sent Done Done false
30202 8872 Acked Flush 0 Sent Done Done false
30203 8873 Acked Flush 0 Sent Done Done false
30204 8874 Acked Read 5 Sent Done Done false
30205 8875 Acked Write 6 Sent Done Done false
30206 8876 Acked Read 3 Sent Done Done false
30207 8877 Acked Flush 0 Sent Done Done false
30208 8878 Acked Read 6 Sent Done Done false
30209 8879 Acked Write 3 Sent Done Done false
30210 8880 Acked Read 6 Sent Done Done false
30211 8881 Acked Write 6 Sent Done Done false
30212 8882 Acked Write 10 Sent Done Done false
30213 8883 Acked Write 4 Sent Done Done false
30214 8884 Acked Read 5 Sent Done Done false
30215 8885 Acked Write 1 Sent Done Done false
30216 8886 Acked Read 4 Sent Done Done false
30217 8887 Acked Read 6 Sent Done Done false
30218 8888 Acked Read 6 Sent Done Done false
30219 8889 Acked Write 3 Sent Done Done false
30220 8890 Acked Read 8 Sent Done Done false
30221 8891 Acked Read 10 Sent Done Done false
30222 8892 Acked Write 4 Sent Done Done false
30223 8893 Acked Read 4 Sent Done Done false
30224 8894 Acked Write 7 Sent Done Done false
30225 8895 Acked Write 8 Sent Done Done false
30226 8896 Acked Write 4 Sent Done Done false
30227 8897 Acked Flush 0 Sent Done Done false
30228 8898 Acked Write 10 Sent Done Done false
30229 8899 Acked Write 10 Sent Done Done false
30230 8900 Acked Read 8 Sent Done Done false
30231 8901 Acked Read 10 Sent Done Done false
30232 8902 Acked Read 1 Sent Done Done false
30233 8903 Acked Read 10 Sent Done Done false
30234 8904 Acked Read 7 Sent Done Done false
30235 8905 Acked Read 8 Sent Done Done false
30236 8906 Acked Read 9 Sent Done Done false
30237 8907 Acked Read 10 Sent Done Done false
30238 8908 Acked Read 8 Sent Done Done false
30239 8909 Acked Read 10 Sent Done Done false
30240 8910 Acked Write 9 Sent Done Done false
30241 8911 Acked Write 8 Sent Done Done false
30242 8912 Acked Read 7 Sent Done Done false
30243 8913 Acked Read 1 Sent Done Done false
30244 8914 Acked Read 5 Sent Done Done false
30245 8915 Acked Read 5 Sent Done Done false
30246 8916 Acked Read 8 Sent Done Done false
30247 8917 Acked Write 3 Sent Done Done false
30248 8918 Acked Write 3 Sent Done Done false
30249 8919 Acked Read 4 Sent Done Done false
30250 8920 Acked Read 6 Sent Done Done false
30251 8921 Acked Read 3 Sent Done Done false
30252 8922 Acked Write 8 Sent Done Done false
30253 8923 Acked Write 8 Sent Done Done false
30254 8924 Acked Write 8 Sent Done Done false
30255 8925 Acked Read 8 Sent Done Done false
30256 8926 Acked Read 10 Sent Done Done false
30257 8927 Acked Read 9 Sent Done Done false
30258 8928 Acked Read 1 Sent Done Done false
30259 8929 Acked Write 3 Sent Done Done false
30260 8930 Acked Flush 0 Sent Done Done false
30261 8931 Acked Write 2 Sent Done Done false
30262 8932 Acked Write 5 Sent Done Done false
30263 8933 Acked Write 3 Sent Done Done false
30264 8934 Acked Read 10 Sent Done Done false
30265 8935 Acked Read 8 Sent Done Done false
30266 8936 Acked Read 2 Sent Done Done false
30267 8937 Acked Read 3 Sent Done Done false
30268 8938 Acked Write 10 Sent Done Done false
30269 8939 Acked Read 9 Sent Done Done false
30270 8940 Acked Flush 0 Sent Done Done false
30271 8941 Acked Read 2 Sent Done Done false
30272 8942 Acked Write 3 Sent Done Done false
30273 8943 Acked Read 5 Sent Done Done false
30274 8944 Acked Read 5 Sent Done Done false
30275 8945 Acked Flush 0 Sent Done Done false
30276 8946 Acked Read 2 Sent Done Done false
30277 8947 Acked Write 2 Sent Done Done false
30278 8948 Acked Flush 0 Sent Done Done false
30279 8949 Acked Read 2 Sent Done Done false
30280 8950 Acked Write 5 Sent Done Done false
30281 8951 Acked Read 3 Sent Done Done false
30282 8952 Acked Read 8 Sent Done Done false
30283 8953 Acked Write 9 Sent Done Done false
30284 8954 Acked Write 8 Sent Done Done false
30285 8955 Acked Read 7 Sent Done Done false
30286 8956 Acked Write 10 Sent Done Done false
30287 8957 Acked Read 9 Sent Done Done false
30288 8958 Acked Write 8 Sent Done Done false
30289 8959 Acked Write 2 Sent Done Done false
30290 8960 Acked Read 5 Sent Done Done false
30291 8961 Acked Read 6 Sent Done Done false
30292 8962 Acked Write 8 Sent Done Done false
30293 8963 Acked Write 10 Sent Done Done false
30294 8964 Acked Write 8 Sent Done Done false
30295 8965 Acked Write 4 Sent Done Done false
30296 8966 Acked Write 8 Sent Done Done false
30297 8967 Acked Read 8 Sent Done Done false
30298 8968 Acked Read 5 Sent Done Done false
30299 8969 Acked Write 8 Sent Done Done false
30300 8970 Acked Flush 0 Sent Done Done false
30301 8971 Acked Write 10 Sent Done Done false
30302 8972 Acked Read 8 Sent Done Done false
30303 8973 Acked Write 9 Sent Done Done false
30304 8974 Acked Read 5 Sent Done Done false
30305 8975 Acked Write 4 Sent Done Done false
30306 8976 Acked Read 4 Sent Done Done false
30307 8977 Acked Read 8 Sent Done Done false
30308 8978 Acked Write 8 Sent Done Done false
30309 8979 Acked Write 4 Sent Done Done false
30310 8980 Acked Write 10 Sent Done Done false
30311 8981 Acked Read 5 Sent Done Done false
30312 8982 Acked Read 8 Sent Done Done false
30313 8983 Acked Flush 0 Sent Done Done false
30314 8984 Acked Read 1 Sent Done Done false
30315 8985 Acked Write 9 Sent Done Done false
30316 8986 Acked Read 2 Sent Done Done false
30317 8987 Acked Write 4 Sent Done Done false
30318 8988 Acked Read 8 Sent Done Done false
30319 8989 Acked Read 7 Sent Done Done false
30320 8990 Acked Read 4 Sent Done Done false
30321 8991 Acked Read 10 Sent Done Done false
30322 8992 Acked Read 1 Sent Done Done false
30323 8993 Acked Read 6 Sent Done Done false
30324 8994 Acked Flush 0 Sent Done Done false
30325 8995 Acked Write 1 Sent Done Done false
30326 8996 Acked Write 8 Sent Done Done false
30327 8997 Acked Write 10 Sent Done Done false
30328 8998 Acked Read 4 Sent Done Done false
30329 8999 Acked Read 9 Sent Done Done false
30330 9000 Acked Read 8 Sent Done Done false
30331 9001 Acked Read 6 Sent Done Done false
30332 9002 Acked Write 3 Sent Done Done false
30333 9003 Acked Write 9 Sent Done Done false
30334 9004 Acked Write 10 Sent Done Done false
30335 9005 Acked Write 5 Sent Done Done false
30336 9006 Acked Flush 0 Sent Done Done false
30337 9007 Acked Read 2 Sent Done Done false
30338 9008 Acked Write 3 Sent Done Done false
30339 9009 Acked Write 7 Sent Done Done false
30340 9010 Acked Write 4 Sent Done Done false
30341 9011 Acked Write 1 Sent Done Done false
30342 9012 Acked Read 2 Sent Done Done false
30343 9013 Acked Flush 0 Sent Done Done false
30344 9014 Acked Read 5 Sent Done Done false
30345 9015 Acked Read 10 Sent Done Done false
30346 9016 Acked Read 5 Sent Done Done false
30347 9017 Acked Write 3 Sent Done Done false
30348 9018 Acked Read 4 Sent Done Done false
30349 9019 Acked Read 9 Sent Done Done false
30350 9020 Acked Write 7 Sent Done Done false
30351 9021 Acked Read 10 Sent Done Done false
30352 9022 Acked Write 5 Sent Done Done false
30353 9023 Acked Flush 0 Sent Done Done false
30354 9024 Acked Read 6 Sent Done Done false
30355 9025 Acked Read 6 Sent Done Done false
30356 9026 Acked Read 6 Sent Done Done false
30357 9027 Acked Write 5 Sent Done Done false
30358 9028 Acked Write 2 Sent Done Done false
30359 9029 Acked Flush 0 Sent Done Done false
30360 9030 Acked Write 3 Sent Done Done false
30361 9031 Acked Write 8 Sent Done Done false
30362 9032 Acked Write 5 Sent Done Done false
30363 9033 Acked Flush 0 Sent Done Done false
30364 9034 Acked Read 4 Sent Done Done false
30365 9035 Acked Write 4 Sent Done Done false
30366 9036 Acked Read 9 Sent Done Done false
30367 9037 Acked Read 9 Sent Done Done false
30368 9038 Acked Write 8 Sent Done Done false
30369 9039 Acked Write 7 Sent Done Done false
30370 9040 Acked Write 8 Sent Done Done false
30371 9041 Acked Write 9 Sent Done Done false
30372 9042 Acked Write 3 Sent Done Done false
30373 9043 Acked Write 6 Sent Done Done false
30374 9044 Acked Read 2 Sent Done Done false
30375 9045 Acked Write 10 Sent Done Done false
30376 9046 Acked Read 3 Sent Done Done false
30377 9047 Acked Read 10 Sent Done Done false
30378 9048 Acked Flush 0 Sent Done Done false
30379 9049 Acked Write 10 Sent Done Done false
30380 9050 Acked Read 7 Sent Done Done false
30381 9051 Acked Write 3 Sent Done Done false
30382 9052 Acked Read 6 Sent Done Done false
30383 9053 Acked Read 7 Sent Done Done false
30384 9054 Acked Read 1 Sent Done Done false
30385 9055 Acked Read 7 Sent Done Done false
30386 9056 Acked Write 9 Sent Done Done false
30387 9057 Acked Read 4 Sent Done Done false
30388 9058 Acked Read 9 Sent Done Done false
30389 9059 Acked Write 8 Sent Done Done false
30390 9060 Acked Write 3 Sent Done Done false
30391 9061 Acked Read 6 Sent Done Done false
30392 9062 Acked Read 9 Sent Done Done false
30393 9063 Acked Read 3 Sent Done Done false
30394 9064 Acked Write 4 Sent Done Done false
30395 9065 Acked Read 10 Sent Done Done false
30396 9066 Acked Read 10 Sent Done Done false
30397 9067 Acked Read 9 Sent Done Done false
30398 9068 Acked Write 2 Sent Done Done false
30399 9069 Acked Read 1 Sent Done Done false
30400 9070 Acked Write 10 Sent Done Done false
30401 9071 Acked Read 8 Sent Done Done false
30402 9072 Acked Write 9 Sent Done Done false
30403 9073 Acked Write 4 Sent Done Done false
30404 9074 Acked Read 9 Sent Done Done false
30405 9075 Acked Read 6 Sent Done Done false
30406 9076 Acked Write 4 Sent Done Done false
30407 9077 Acked Write 4 Sent Done Done false
30408 9078 Acked Write 7 Sent Done Done false
30409 9079 Acked Read 10 Sent Done Done false
30410 9080 Acked Read 5 Sent Done Done false
30411 9081 Acked Flush 0 Sent Done Done false
30412 9082 Acked Write 4 Sent Done Done false
30413 9083 Acked Read 9 Sent Done Done false
30414 9084 Acked Write 4 Sent Done Done false
30415 9085 Acked Write 5 Sent Done Done false
30416 9086 Acked Read 7 Sent Done Done false
30417 9087 Acked Flush 0 Sent Done Done false
30418 9088 Acked Write 4 Sent Done Done false
30419 9089 Acked Write 4 Sent Done Done false
30420 9090 Acked Write 2 Sent Done Done false
30421 9091 Acked Read 8 Sent Done Done false
30422 9092 Acked Write 6 Sent Done Done false
30423 9093 Acked Read 10 Sent Done Done false
30424 9094 Acked Write 3 Sent Done Done false
30425 9095 Acked Write 1 Sent Done Done false
30426 9096 Acked Read 5 Sent Done Done false
30427 9097 Acked Write 9 Sent Done Done false
30428 9098 Acked Write 7 Sent Done Done false
30429 9099 Acked Read 10 Sent Done Done false
30430 9100 Acked Read 2 Sent Done Done false
30431 9101 Acked Write 6 Sent Done Done false
30432 9102 Acked Write 1 Sent Done Done false
30433 9103 Acked Read 5 Sent Done Done false
30434 9104 Acked Write 3 Sent Done Done false
30435 9105 Acked Write 10 Sent Done Done false
30436 9106 Acked Write 1 Sent Done Done false
30437 9107 Acked Flush 0 Sent Done Done false
30438 9108 Acked Read 10 Sent Done Done false
30439 9109 Acked Read 6 Sent Done Done false
30440 9110 Acked Flush 0 Sent Done Done false
30441 9111 Acked Read 9 Sent Done Done false
30442 9112 Acked Write 8 Sent Done Done false
30443 9113 Acked Read 2 Sent Done Done false
30444 9114 Acked Read 4 Sent Done Done false
30445 9115 Acked Read 8 Sent Done Done false
30446 9116 Acked Write 10 Sent Done Done false
30447 9117 Acked Read 10 Sent Done Done false
30448 9118 Acked Write 3 Sent Done Done false
30449 9119 Acked Read 6 Sent Done Done false
30450 9120 Acked Read 5 Sent Done Done false
30451 9121 Acked Read 1 Sent Done Done false
30452 9122 Acked Read 5 Sent Done Done false
30453 9123 Acked Flush 0 Sent Done Done false
30454 9124 Acked Read 1 Sent Done Done false
30455 9125 Acked Write 1 Sent Done Done false
30456 9126 Acked Read 2 Sent Done Done false
30457 9127 Acked Read 9 Sent Done Done false
30458 9128 Acked Read 2 Sent Done Done false
30459 9129 Acked Write 5 Sent Done Done false
30460 9130 Acked Read 3 Sent Done Done false
30461 9131 Acked Read 8 Sent Done Done false
30462 9132 Acked Read 9 Sent Done Done false
30463 9133 Acked Read 2 Sent Done Done false
30464 9134 Acked Write 9 Sent Done Done false
30465 9135 Acked Read 1 Sent Done Done false
30466 9136 Acked Read 8 Sent Done Done false
30467 9137 Acked Flush 0 Sent Done Done false
30468 9138 Acked Write 8 Sent Done Done false
30469 9139 Acked Write 5 Sent Done Done false
30470 9140 Acked Read 6 Sent Done Done false
30471 9141 Acked Flush 0 Sent Done Done false
30472 9142 Acked Write 4 Sent Done Done false
30473 9143 Acked Read 2 Sent Done Done false
30474 9144 Acked Write 3 Sent Done Done false
30475 9145 Acked Write 2 Sent Done Done false
30476 9146 Acked Read 6 Sent Done Done false
30477 9147 Acked Read 2 Sent Done Done false
30478 9148 Acked Flush 0 Sent Done Done false
30479 9149 Acked Read 6 Sent Done Done false
30480 9150 Acked Read 2 Sent Done Done false
30481 STATES DS:0 DS:1 DS:2 TOTAL
30482 Sent 301 0 0 301
30483 Done 0 301 301 602
30484 Skipped 0 0 0 0
30485 Error 0 0 0 0
30486 Last Flush: Some(JobId(8849)) Some(JobId(9148)) Some(JobId(9148))
30487 Downstairs last five completed: 8849 8848 8847 8846 8845
30488 Upstairs last five completed: 9150 9149 9148 9147 9146
30489 CLIENT: Up:0 ds:301 act:2
304902025-01-28T22:00:57.083ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
304912025-01-28T22:00:57.920ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
304922025-01-28T22:00:58.921ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
304932025-01-28T22:00:59.922ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
304942025-01-28T22:01:00.326ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
304952025-01-28T22:01:00.326ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53856) = io task client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
304962025-01-28T22:01:00.327ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
304972025-01-28T22:01:00.328ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
304982025-01-28T22:01:00.328ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
304992025-01-28T22:01:00.328ZINFOcrucible: send last flush ID to this DS: Some(JobId(8849)) = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
305002025-01-28T22:01:00.328ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
305012025-01-28T22:01:00.328ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(8849)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
305022025-01-28T22:01:00.328ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(8849)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
30503 ----------------------------------------------------------------
30504 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:2
30505 States: Active Active Active
30506 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30507 9149 Acked Read 6 Done Done Done true
30508 9150 Acked Read 2 Done Done Done true
30509 STATES DS:0 DS:1 DS:2 TOTAL
30510 Sent 0 0 0 0
30511 Done 2 2 2 6
30512 Skipped 0 0 0 0
30513 Error 0 0 0 0
30514 Last Flush: Some(JobId(9148)) Some(JobId(9148)) Some(JobId(9148))
30515 Downstairs last five completed: 9148 9147 9146 9145 9144
30516 Upstairs last five completed: 9150 9149 9148 9147 9146
30517 CLIENT: Up:0 ds:2 act:3
30518 ----------------------------------------------------------------
30519 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30520 Downstairs last five completed: 9151 9150 9149 9148 9147
30521 Upstairs last five completed: 9151 9150 9149 9148 9147
30522 CLIENT: Up:0 ds:0 act:3
30523 Replay: All jobs finished
30524 Main task has work to do, go find it
30525 got dsc Stop(2)
30526 stop 2
30527 [8830] Got stop action so:false kr:false
30528 Jan 28 22:01:04.585 INFO request completed, latency_us: 370, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 896b0837-2fd5-44c1-8134-e78ebeaa30f4, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
305292025-01-28T22:01:04.586ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
30530 Jan 28 22:01:04.586 INFO request completed, latency_us: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-28T22:01:04.586318187Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
30531 110, response_code: 200, uri: /state/cid/2, method: GET, req_id:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-28T22:01:04.586335296Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1202,"session_id":"97f0d889-45eb-4682-bc34-b4ef4ba454a8"}
30532 5819ae9f-b4a5-45c0-ba94-dc07d12fa966, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
305332025-01-28T22:01:04.586ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
30534 [8830] Exited with: ExitStatus(unix_wait_status(9))
30535 [8830][2] reports Exit
30536 Jan 28 22:01:08.588 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4e879b05-9e39-4a61-8105-2225d5289e87, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
30537 001/300 Read block 327 len 16384
30538 002/300 Write block 51 len 40960 data: 4 6 6 6 6 4 4 6 7 7
30539 003/300 Write block 3161 len 12288 data: 7 6 5
30540 004/300 Read block 1143 len 28672
30541 005/300 Read block 3144 len 20480
30542 006/300 Read block 4797 len 32768
30543 007/300 Write block 3696 len 24576 data: 3 3 4 5 5 4
30544 008/300 Read block 2277 len 16384
30545 009/300 Write block 1368 len 32768 data: 5 5 5 4 3 3 3 3
30546 010/300 Flush
30547 011/300 Write block 4983 len 40960 data: 7 8 8 10 9 9 7 6 4 4
30548 012/300 Read block 4899 len 12288
30549 013/300 Write block 4044 len 12288 data: 4 4 5
30550 014/300 Read block 2367 len 16384
30551 015/300 Write block 3127 len 8192 data: 3 2
30552 016/300 Read block 4804 len 20480
30553 017/300 Read block 2888 len 4096
30554 018/300 Write block 527 len 28672 data: 8 7 8 7 7 7 8
30555 019/300 Write block 2073 len 12288 data: 5 4 5
30556 020/300 Flush
30557 021/300 Write block 4911 len 12288 data: 4 4 4
30558 022/300 Read block 2720 len 32768
30559 023/300 Write block 2479 len 4096 data: 8
30560 024/300 Read block 585 len 16384
30561 025/300 Read block 2044 len 40960
30562 026/300 Read block 2763 len 28672
30563 027/300 Write block 2696 len 16384 data: 6 7 5 3
30564 028/300 Write block 3701 len 4096 data: 5
30565 029/300 Write block 4170 len 28672 data: 6 8 9 7 7 8 8
30566 030/300 Read block 3579 len 12288
30567 031/300 Write block 3319 len 16384 data: 4 5 5 7
30568 032/300 Write block 3579 len 20480 data: 4 4 3 3 3
30569 033/300 Write block 245 len 20480 data: 8 8 7 6 6
30570 034/300 Write block 1687 len 36864 data: 3 3 3 2 2 3 3 3 3
30571 035/300 Flush
30572 036/300 Write block 891 len 32768 data: 5 8 8 5 4 3 2 3
30573 037/300 Write block 2882 len 4096 data: 4
30574 038/300 Write block 2751 len 40960 data: 5 8 7 7 9 9 9 9 10 9
30575 039/300 Write block 3159 len 12288 data: 7 7 8
30576 040/300 Read block 589 len 16384
30577 041/300 Write block 312 len 36864 data: 4 5 2 2 3 4 4 6 6
30578 042/300 Read block 2630 len 16384
30579 043/300 Write block 3730 len 28672 data: 9 7 7 7 6 6 5
30580 044/300 Read block 3336 len 36864
30581 045/300 Write block 554 len 32768 data: 4 5 5 4 4 2 2 2
30582 046/300 Read block 4927 len 16384
30583 047/300 Read block 4934 len 40960
30584 048/300 Write block 777 len 40960 data: 5 6 8 7 6 7 7 7 5 6
30585 049/300 Write block 2812 len 40960 data: 5 5 4 5 5 5 5 4 7 8
30586 050/300 Read block 606 len 12288
30587 051/300 Flush
30588 052/300 Flush
30589 053/300 Read block 1857 len 12288
30590 054/300 Read block 1003 len 4096
30591 055/300 Flush
30592 056/300 Write block 4356 len 36864 data: 7 9 9 9 9 7 6 7 9
30593 057/300 Read block 3497 len 4096
30594 058/300 Write block 4708 len 20480 data: 10 11 11 9 7
30595 059/300 Read block 3117 len 20480
30596 060/300 Write block 1424 len 16384 data: 4 6 7 6
30597 061/300 Read block 4108 len 4096
30598 062/300 Write block 3899 len 16384 data: 6 6 6 5
30599 063/300 Write block 3599 len 20480 data: 9 6 6 7 6
30600 064/300 Read block 2392 len 16384
30601 065/300 Write block 3657 len 8192 data: 4 4
30602 066/300 Write block 3958 len 36864 data: 6 6 6 6 5 4 3 3 3
30603 067/300 Read block 1552 len 12288
30604 068/300 Read block 1 len 4096
30605 069/300 Write block 3147 len 40960 data: 5 5 4 5 4 4 7 8 7 7
30606 070/300 Read block 4925 len 40960
30607 071/300 Read block 4347 len 8192
30608 072/300 Read block 4822 len 40960
30609 073/300 Read block 2570 len 28672
30610 074/300 Flush
30611 075/300 Write block 790 len 16384 data: 5 5 5 5
30612 076/300 Flush
30613 077/300 Read block 3635 len 28672
30614 078/300 Flush
30615 079/300 Read block 3410 len 8192
30616 080/300 Read block 3510 len 36864
30617 081/300 Write block 1027 len 16384 data: 7 6 8 6
30618 082/300 Read block 970 len 20480
30619 083/300 Read block 1382 len 32768
30620 084/300 Read block 3636 len 16384
30621 085/300 Write block 4382 len 12288 data: 11 12 11
30622 086/300 Read block 4185 len 20480
30623 087/300 Flush
30624 088/300 Read block 2945 len 8192
30625 089/300 Write block 4179 len 40960 data: 9 8 7 6 7 5 5 5 6 5
30626 090/300 Write block 1597 len 40960 data: 4 3 4 4 4 4 5 5 4 3
30627 091/300 Read block 3244 len 4096
30628 092/300 Write block 385 len 40960 data: 5 5 4 5 4 3 3 3 4 4
30629 093/300 Read block 872 len 28672
30630 094/300 Flush
30631 095/300 Flush
30632 096/300 Write block 3022 len 28672 data: 5 6 8 9 6 4 5
30633 097/300 Read block 2348 len 24576
30634 098/300 Write block 2464 len 24576 data: 3 6 5 7 6 4
30635 099/300 Write block 1282 len 28672 data: 5 5 6 5 5 5 5
30636 100/300 Write block 1441 len 24576 data: 4 5 4 4 4 5
30637 101/300 Read block 3086 len 4096
30638 102/300 Read block 2298 len 16384
30639 103/300 Write block 4295 len 40960 data: 4 4 6 7 7 6 5 4 4 3
30640 104/300 Read block 2969 len 32768
30641 105/300 Read block 4517 len 16384
30642 106/300 Read block 701 len 20480
30643 107/300 Write block 4912 len 32768 data: 5 5 4 5 5 4 4 4
30644 108/300 Write block 3265 len 4096 data: 7
30645 109/300 Read block 4874 len 32768
30646 110/300 Read block 3800 len 28672
30647 111/300 Read block 265 len 24576
30648 112/300 Read block 591 len 28672
30649 113/300 Read block 4884 len 8192
30650 114/300 Flush
30651 115/300 Read block 682 len 4096
30652 116/300 Write block 3778 len 36864 data: 6 4 4 5 5 5 4 4 5
30653 117/300 Write block 2061 len 24576 data: 4 6 6 5 5 4
30654 118/300 Read block 2406 len 36864
30655 119/300 Flush
30656 120/300 Write block 3672 len 24576 data: 8 8 7 4 4 3
30657 121/300 Read block 29 len 8192
30658 122/300 Write block 305 len 16384 data: 7 6 5 5
30659 123/300 Write block 843 len 28672 data: 8 8 8 3 6 6 5
30660 124/300 Read block 3721 len 16384
30661 125/300 Write block 4289 len 40960 data: 5 4 3 6 5 5 5 5 7 8
30662 126/300 Read block 1739 len 20480
30663 127/300 Read block 19 len 28672
30664 128/300 Read block 2105 len 16384
30665 129/300 Write block 3732 len 8192 data: 8 8
30666 130/300 Write block 2089 len 28672 data: 5 6 7 5 6 7 7
30667 131/300 Read block 1353 len 36864
30668 132/300 Read block 1872 len 4096
30669 133/300 Write block 1431 len 36864 data: 8 9 9 5 5 6 6 7 6
30670 134/300 Write block 4678 len 36864 data: 8 9 9 10 9 9 9 9 5
30671 135/300 Read block 942 len 24576
30672 136/300 Read block 2542 len 8192
30673 137/300 Read block 4431 len 4096
30674 138/300 Read block 3952 len 24576
30675 139/300 Write block 3375 len 4096 data: 7
30676 140/300 Read block 1089 len 40960
30677 141/300 Write block 1411 len 24576 data: 7 8 9 9 7 7
30678 142/300 Write block 2631 len 16384 data: 8 8 10 9
30679 143/300 Read block 663 len 12288
30680 144/300 Write block 250 len 12288 data: 6 5 5
30681 145/300 Write block 2460 len 20480 data: 5 5 4 3 4
30682 146/300 Flush
30683 147/300 Read block 3543 len 24576
30684 148/300 Flush
30685 149/300 Read block 4459 len 16384
30686 150/300 Read block 3305 len 24576
30687 151/300 Read block 1543 len 36864
30688 152/300 Read block 2142 len 40960
30689 153/300 Write block 2121 len 32768 data: 3 5 6 7 7 6 6 5
30690 154/300 Write block 2980 len 20480 data: 6 6 5 4 4
30691 155/300 Write block 2655 len 40960 data: 2 3 3 5 7 9 10 11 8 7
30692 156/300 Read block 2279 len 28672
30693 157/300 Read block 4397 len 32768
30694 158/300 Flush
30695 159/300 Read block 3571 len 28672
30696 160/300 Read block 2789 len 8192
30697 161/300 Read block 1584 len 12288
30698 162/300 Read block 3523 len 4096
30699 163/300 Write block 674 len 20480 data: 3 3 3 3 4
30700 164/300 Read block 1185 len 28672
30701 165/300 Read block 4814 len 4096
30702 166/300 Read block 555 len 4096
30703 167/300 Read block 3533 len 4096
30704 168/300 Read block 1980 len 4096
30705 169/300 Write block 3721 len 28672 data: 7 7 6 6 8 8 10
30706 170/300 Read block 826 len 16384
30707 171/300 Write block 921 len 32768 data: 4 4 4 5 5 5 4 5
30708 172/300 Write block 870 len 24576 data: 7 6 5 7 6 7
30709 173/300 Write block 324 len 28672 data: 7 7 6 7 6 6 6
30710 174/300 Write block 1993 len 12288 data: 3 3 3
30711 175/300 Flush
30712 176/300 Read block 2908 len 20480
30713 177/300 Read block 2314 len 24576
30714 178/300 Write block 2647 len 20480 data: 5 5 5 4 4
30715 179/300 Read block 731 len 16384
30716 180/300 Read block 4803 len 40960
30717 181/300 Read block 1474 len 36864
30718 182/300 Write block 3665 len 32768 data: 6 4 6 9 8 8 8 9
30719 183/300 Flush
30720 184/300 Read block 776 len 32768
30721 185/300 Write block 887 len 24576 data: 7 5 7 4 6 9
30722 186/300 Read block 3994 len 36864
30723 187/300 Read block 2918 len 16384
30724 188/300 Flush
30725 189/300 Read block 3465 len 36864
30726 190/300 Read block 2766 len 12288
30727 191/300 Read block 109 len 32768
30728 192/300 Write block 2083 len 8192 data: 8 7
30729 193/300 Write block 2121 len 20480 data: 4 6 7 8 8
30730 194/300 Write block 3777 len 32768 data: 7 7 5 5 6 6 6 5
30731 195/300 Read block 1881 len 16384
30732 196/300 Read block 247 len 28672
30733 197/300 Write block 4377 len 32768 data: 6 6 6 7 10 12 13 12
30734 198/300 Read block 1031 len 32768
30735 199/300 Write block 1878 len 20480 data: 6 7 7 6 6
30736 200/300 Write block 2339 len 28672 data: 6 6 5 5 6 7 5
30737 201/300 Read block 2060 len 4096
30738 202/300 Write block 1832 len 24576 data: 6 5 4 3 4 4
30739 203/300 Write block 1290 len 16384 data: 7 8 7 7
30740 204/300 Write block 4136 len 4096 data: 8
30741 205/300 Read block 3557 len 28672
30742 206/300 Read block 4880 len 16384
30743 207/300 Write block 2292 len 32768 data: 7 8 5 5 5 5 4 4
30744 208/300 Flush
30745 209/300 Read block 181 len 32768
30746 210/300 Write block 25 len 12288 data: 7 6 5
30747 211/300 Write block 3512 len 36864 data: 4 4 4 4 4 5 5 5 5
30748 212/300 Flush
30749 213/300 Read block 4493 len 8192
30750 214/300 Write block 2032 len 32768 data: 3 4 4 3 4 4 5 6
30751 215/300 Read block 1522 len 4096
30752 216/300 Write block 186 len 32768 data: 2 4 5 5 4 4 4 5
30753 217/300 Read block 971 len 28672
30754 218/300 Flush
30755 219/300 Read block 2390 len 16384
30756 220/300 Write block 4556 len 8192 data: 5 5
30757 221/300 Write block 3071 len 8192 data: 4 4
30758 222/300 Read block 4700 len 4096
30759 223/300 Read block 1868 len 32768
30760 224/300 Read block 2735 len 28672
30761 225/300 Read block 4925 len 28672
30762 226/300 Write block 3842 len 40960 data: 6 5 5 5 5 4 3 3 2 4
30763 227/300 Read block 3449 len 24576
30764 228/300 Read block 264 len 20480
30765 229/300 Write block 2760 len 24576 data: 10 7 7 4 5 5
30766 230/300 Write block 4486 len 28672 data: 13 10 8 7 8 7 6
30767 231/300 Read block 4518 len 16384
30768 232/300 Read block 2887 len 8192
30769 233/300 Read block 3952 len 36864
30770 234/300 Write block 1793 len 8192 data: 3 6
30771 235/300 Read block 4546 len 20480
30772 236/300 Read block 2971 len 12288
30773 237/300 Write block 4458 len 32768 data: 4 3 3 3 5 5 6 5
30774 238/300 Write block 547 len 12288 data: 10 9 8
30775 239/300 Read block 1047 len 4096
30776 240/300 Read block 1526 len 12288
30777 241/300 Read block 652 len 32768
30778 242/300 Read block 1494 len 32768
30779 243/300 Read block 2968 len 40960
30780 244/300 Write block 90 len 24576 data: 3 5 4 5 6 5
30781 245/300 Write block 2140 len 28672 data: 5 4 6 7 8 7 8
30782 246/300 Flush
30783 247/300 Write block 3281 len 40960 data: 3 3 4 4 4 4 4 5 6 3
30784 248/300 Write block 3164 len 24576 data: 5 5 5 5 6 6
30785 249/300 Write block 1354 len 16384 data: 5 6 5 4
30786 250/300 Write block 1123 len 40960 data: 5 4 5 4 5 6 5 5 2 2
30787 251/300 Read block 625 len 32768
30788 252/300 Read block 1927 len 16384
30789 253/300 Flush
30790 254/300 Write block 2270 len 36864 data: 5 5 4 5 6 7 8 7 8
30791 255/300 Write block 2128 len 28672 data: 6 4 8 9 9 13 12
30792 256/300 Write block 3259 len 24576 data: 7 5 6 7 7 8
30793 257/300 Read block 4315 len 12288
30794 258/300 Flush
30795 259/300 Flush
30796 260/300 Write block 3949 len 16384 data: 9 8 7 4
30797 261/300 Read block 103 len 20480
30798 262/300 Write block 2333 len 36864 data: 6 5 4 4 6 6 7 7 6
30799 263/300 Write block 1545 len 32768 data: 3 3 3 3 5 5 4 5
30800 264/300 Read block 3368 len 4096
30801 265/300 Read block 4707 len 40960
30802 266/300 Read block 4195 len 24576
30803 267/300 Write block 4272 len 40960 data: 6 5 6 7 5 5 4 4 3 3
30804 268/300 Read block 4238 len 20480
30805 269/300 Read block 935 len 24576
30806 270/300 Read block 1850 len 40960
30807 271/300 Write block 3128 len 20480 data: 3 3 3 4 4
30808 272/300 Flush
30809 273/300 Read block 2765 len 28672
30810 274/300 Write block 4197 len 32768 data: 8 7 7 7 4 3 7 7
30811 275/300 Read block 3933 len 8192
30812 276/300 Read block 4570 len 16384
30813 277/300 Write block 1015 len 16384 data: 11 11 9 9
30814 278/300 Read block 2524 len 32768
30815 279/300 Write block 981 len 12288 data: 7 5 4
30816 280/300 Write block 2722 len 8192 data: 3 4
30817 281/300 Read block 3138 len 12288
30818 282/300 Read block 3696 len 36864
30819 283/300 Read block 1044 len 32768
30820 284/300 Flush
30821 285/300 Read block 3253 len 36864
30822 286/300 Read block 4602 len 32768
30823 287/300 Read block 2165 len 28672
30824 288/300 Write block 805 len 28672 data: 3 4 3 4 5 6 5
30825 289/300 Read block 3100 len 36864
30826 290/300 Flush
30827 291/300 Read block 2489 len 28672
30828 292/300 Write block 2532 len 12288 data: 5 5 3
30829 293/300 Read block 3172 len 36864
30830 294/300 Read block 3036 len 36864
30831 295/300 Read block 3585 len 16384
30832 296/300 Flush
30833 297/300 Write block 2880 len 40960 data: 4 5 5 5 5 6 6 4 3 3
30834 298/300 Write block 2515 len 12288 data: 7 7 5
30835 299/300 Flush
30836 300/300 Read block 2974 len 36864
30837 301/300 Write block 4928 len 40960 data: 5 5 7 7 8 9 9 9 6 4
30838 Main task has work to do, go find it
30839 got dsc Start(2)
30840 start 2
30841 Jan 28 22:01:08.833 INFO request completed, latency_us: 238, response_code: 204, uri: /start/cid/2, method: POST, req_id: 297a2b6b-6b67-4bcd-b8b5-59d4e7e83d1a, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
30842 Replay: started 2, returned:Ok(())
30843 ----------------------------------------------------------------
30844 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30845 States: Active Active Offline
30846 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30847 9152 Acked Read 4 Done Done Sent false
30848 9153 Acked Write 10 Done Done Sent false
30849 9154 Acked Write 3 Done Done Sent false
30850 9155 Acked Read 7 Done Done Sent false
30851 9156 Acked Read 5 Done Done Sent false
30852 9157 Acked Read 8 Done Done Sent false
30853 9158 Acked Write 6 Done Done Sent false
30854 9159 Acked Read 4 Done Done Sent false
30855 9160 Acked Write 8 Done Done Sent false
30856 9161 Acked Flush 0 Done Done Sent false
30857 9162 Acked Write 10 Done Done Sent false
30858 9163 Acked Read 3 Done Done Sent false
30859 9164 Acked Write 3 Done Done Sent false
30860 9165 Acked Read 4 Done Done Sent false
30861 9166 Acked Write 2 Done Done Sent false
30862 9167 Acked Read 5 Done Done Sent false
30863 9168 Acked Read 1 Done Done Sent false
30864 9169 Acked Write 7 Done Done Sent false
30865 9170 Acked Write 3 Done Done Sent false
30866 9171 Acked Flush 0 Done Done Sent false
30867 9172 Acked Write 3 Done Done Sent false
30868 9173 Acked Read 8 Done Done Sent false
30869 9174 Acked Write 1 Done Done Sent false
30870 9175 Acked Read 4 Done Done Sent false
30871 9176 Acked Read 10 Done Done Sent false
30872 9177 Acked Read 7 Done Done Sent false
30873 9178 Acked Write 4 Done Done Sent false
30874 9179 Acked Write 1 Done Done Sent false
30875 9180 Acked Write 7 Done Done Sent false
30876 9181 Acked Read 3 Done Done Sent false
30877 9182 Acked Write 4 Done Done Sent false
30878 9183 Acked Write 5 Done Done Sent false
30879 9184 Acked Write 5 Done Done Sent false
30880 9185 Acked Write 9 Done Done Sent false
30881 9186 Acked Flush 0 Done Done Sent false
30882 9187 Acked Write 8 Done Done Sent false
30883 9188 Acked Write 1 Done Done Sent false
30884 9189 Acked Write 10 Done Done Sent false
30885 9190 Acked Write 3 Done Done Sent false
30886 9191 Acked Read 4 Done Done Sent false
30887 9192 Acked Write 9 Done Done Sent false
30888 9193 Acked Read 4 Done Done Sent false
30889 9194 Acked Write 7 Done Done Sent false
30890 9195 Acked Read 9 Done Done Sent false
30891 9196 Acked Write 8 Done Done Sent false
30892 9197 Acked Read 4 Done Done Sent false
30893 9198 Acked Read 10 Done Done Sent false
30894 9199 Acked Write 10 Done Done Sent false
30895 9200 Acked Write 10 Done Done Sent false
30896 9201 Acked Read 3 Done Done Sent false
30897 9202 Acked Flush 0 Done Done Sent false
30898 9203 Acked Flush 0 Done Done Sent false
30899 9204 Acked Read 3 Done Done Sent false
30900 9205 Acked Read 1 Done Done Sent false
30901 9206 Acked Flush 0 Done Done Sent false
30902 9207 Acked Write 9 Done Done Sent false
30903 9208 Acked Read 1 Done Done Sent false
30904 9209 Acked Write 5 Done Done Sent false
30905 9210 Acked Read 5 Done Done Sent false
30906 9211 Acked Write 4 Done Done Sent false
30907 9212 Acked Read 1 Done Done Sent false
30908 9213 Acked Write 4 Done Done Sent false
30909 9214 Acked Write 5 Done Done Sent false
30910 9215 Acked Read 4 Done Done Sent false
30911 9216 Acked Write 2 Done Done Sent false
30912 9217 Acked Write 9 Done Done Sent false
30913 9218 Acked Read 3 Done Done Sent false
30914 9219 Acked Read 1 Done Done Sent false
30915 9220 Acked Write 10 Done Done Sent false
30916 9221 Acked Read 10 Done Done Sent false
30917 9222 Acked Read 2 Done Done Sent false
30918 9223 Acked Read 10 Done Done Sent false
30919 9224 Acked Read 7 Done Done Sent false
30920 9225 Acked Flush 0 Done Done Sent false
30921 9226 Acked Write 4 Done Done Sent false
30922 9227 Acked Flush 0 Done Done Sent false
30923 9228 Acked Read 7 Done Done Sent false
30924 9229 Acked Flush 0 Done Done Sent false
30925 9230 Acked Read 2 Done Done Sent false
30926 9231 Acked Read 9 Done Done Sent false
30927 9232 Acked Write 4 Done Done Sent false
30928 9233 Acked Read 5 Done Done Sent false
30929 9234 Acked Read 8 Done Done Sent false
30930 9235 Acked Read 4 Done Done Sent false
30931 9236 Acked Write 3 Done Done Sent false
30932 9237 Acked Read 5 Done Done Sent false
30933 9238 Acked Flush 0 Done Done Sent false
30934 9239 Acked Read 2 Done Done Sent false
30935 9240 Acked Write 10 Done Done Sent false
30936 9241 Acked Write 10 Done Done Sent false
30937 9242 Acked Read 1 Done Done Sent false
30938 9243 Acked Write 10 Done Done Sent false
30939 9244 Acked Read 7 Done Done Sent false
30940 9245 Acked Flush 0 Done Done Sent false
30941 9246 Acked Flush 0 Done Done Sent false
30942 9247 Acked Write 7 Done Done Sent false
30943 9248 Acked Read 6 Done Done Sent false
30944 9249 Acked Write 6 Done Done Sent false
30945 9250 Acked Write 7 Done Done Sent false
30946 9251 Acked Write 6 Done Done Sent false
30947 9252 Acked Read 1 Done Done Sent false
30948 9253 Acked Read 4 Done Done Sent false
30949 9254 Acked Write 10 Done Done Sent false
30950 9255 Acked Read 8 Done Done Sent false
30951 9256 Acked Read 4 Done Done Sent false
30952 9257 Acked Read 5 Done Done Sent false
30953 9258 Acked Write 8 Done Done Sent false
30954 9259 Acked Write 1 Done Done Sent false
30955 9260 Acked Read 8 Done Done Sent false
30956 9261 Acked Read 7 Done Done Sent false
30957 9262 Acked Read 6 Done Done Sent false
30958 9263 Acked Read 7 Done Done Sent false
30959 9264 Acked Read 2 Done Done Sent false
30960 9265 Acked Flush 0 Done Done Sent false
30961 9266 Acked Read 1 Done Done Sent false
30962 9267 Acked Write 9 Done Done Sent false
30963 9268 Acked Write 6 Done Done Sent false
30964 9269 Acked Read 9 Done Done Sent false
30965 9270 Acked Flush 0 Done Done Sent false
30966 9271 Acked Write 6 Done Done Sent false
30967 9272 Acked Read 2 Done Done Sent false
30968 9273 Acked Write 4 Done Done Sent false
30969 9274 Acked Write 7 Done Done Sent false
30970 9275 Acked Read 4 Done Done Sent false
30971 9276 Acked Write 10 Done Done Sent false
30972 9277 Acked Read 5 Done Done Sent false
30973 9278 Acked Read 7 Done Done Sent false
30974 9279 Acked Read 4 Done Done Sent false
30975 9280 Acked Write 2 Done Done Sent false
30976 9281 Acked Write 7 Done Done Sent false
30977 9282 Acked Read 9 Done Done Sent false
30978 9283 Acked Read 1 Done Done Sent false
30979 9284 Acked Write 9 Done Done Sent false
30980 9285 Acked Write 9 Done Done Sent false
30981 9286 Acked Read 6 Done Done Sent false
30982 9287 Acked Read 2 Done Done Sent false
30983 9288 Acked Read 1 Done Done Sent false
30984 9289 Acked Read 6 Done Done Sent false
30985 9290 Acked Write 1 Done Done Sent false
30986 9291 Acked Read 10 Done Done Sent false
30987 9292 Acked Write 6 Done Done Sent false
30988 9293 Acked Write 4 Done Done Sent false
30989 9294 Acked Read 3 Done Done Sent false
30990 9295 Acked Write 3 Done Done Sent false
30991 9296 Acked Write 5 Done Done Sent false
30992 9297 Acked Flush 0 Done Done Sent false
30993 9298 Acked Read 6 Done Done Sent false
30994 9299 Acked Flush 0 Done Done Sent false
30995 9300 Acked Read 4 Done Done Sent false
30996 9301 Acked Read 6 Done Done Sent false
30997 9302 Acked Read 9 Done Done Sent false
30998 9303 Acked Read 10 Done Done Sent false
30999 9304 Acked Write 8 Done Done Sent false
31000 9305 Acked Write 5 Done Done Sent false
31001 9306 Acked Write 10 Done Done Sent false
31002 9307 Acked Read 7 Done Done Sent false
31003 9308 Acked Read 8 Done Done Sent false
31004 9309 Acked Flush 0 Done Done Sent false
31005 9310 Acked Read 7 Done Done Sent false
31006 9311 Acked Read 2 Done Done Sent false
31007 9312 Acked Read 3 Done Done Sent false
31008 9313 Acked Read 1 Done Done Sent false
31009 9314 Acked Write 5 Done Done Sent false
31010 9315 Acked Read 7 Done Done Sent false
31011 9316 Acked Read 1 Done Done Sent false
31012 9317 Acked Read 1 Done Done Sent false
31013 9318 Acked Read 1 Done Done Sent false
31014 9319 Acked Read 1 Done Done Sent false
31015 9320 Acked Write 7 Done Done Sent false
31016 9321 Acked Read 4 Done Done Sent false
31017 9322 Acked Write 8 Done Done Sent false
31018 9323 Acked Write 6 Done Done Sent false
31019 9324 Acked Write 7 Done Done Sent false
31020 9325 Acked Write 3 Done Done Sent false
31021 9326 Acked Flush 0 Done Done Sent false
31022 9327 Acked Read 5 Done Done Sent false
31023 9328 Acked Read 6 Done Done Sent false
31024 9329 Acked Write 5 Done Done Sent false
31025 9330 Acked Read 4 Done Done Sent false
31026 9331 Acked Read 10 Done Done Sent false
31027 9332 Acked Read 9 Done Done Sent false
31028 9333 Acked Write 8 Done Done Sent false
31029 9334 Acked Flush 0 Done Done Sent false
31030 9335 Acked Read 8 Done Done Sent false
31031 9336 Acked Write 6 Done Done Sent false
31032 9337 Acked Read 9 Done Done Sent false
31033 9338 Acked Read 4 Done Done Sent false
31034 9339 Acked Flush 0 Done Done Sent false
31035 9340 Acked Read 9 Done Done Sent false
31036 9341 Acked Read 3 Done Done Sent false
31037 9342 Acked Read 8 Done Done Sent false
31038 9343 Acked Write 2 Done Done Sent false
31039 9344 Acked Write 5 Done Done Sent false
31040 9345 Acked Write 8 Done Done Sent false
31041 9346 Acked Read 4 Done Done Sent false
31042 9347 Acked Read 7 Done Done Sent false
31043 9348 Acked Write 8 Done Done Sent false
31044 9349 Acked Read 8 Done Done Sent false
31045 9350 Acked Write 5 Done Done Sent false
31046 9351 Acked Write 7 Done Done Sent false
31047 9352 Acked Read 1 Done Done Sent false
31048 9353 Acked Write 6 Done Done Sent false
31049 9354 Acked Write 4 Done Done Sent false
31050 9355 Acked Write 1 Done Done Sent false
31051 9356 Acked Read 7 Done Done Sent false
31052 9357 Acked Read 4 Done Done Sent false
31053 9358 Acked Write 8 Done Done Sent false
31054 9359 Acked Flush 0 Done Done Sent false
31055 9360 Acked Read 8 Done Done Sent false
31056 9361 Acked Write 3 Done Done Sent false
31057 9362 Acked Write 9 Done Done Sent false
31058 9363 Acked Flush 0 Done Done Sent false
31059 9364 Acked Read 2 Done Done Sent false
31060 9365 Acked Write 8 Done Done Sent false
31061 9366 Acked Read 1 Done Done Sent false
31062 9367 Acked Write 8 Done Done Sent false
31063 9368 Acked Read 7 Done Done Sent false
31064 9369 Acked Flush 0 Done Done Sent false
31065 9370 Acked Read 4 Done Done Sent false
31066 9371 Acked Write 2 Done Done Sent false
31067 9372 Acked Write 2 Done Done Sent false
31068 9373 Acked Read 1 Done Done Sent false
31069 9374 Acked Read 8 Done Done Sent false
31070 9375 Acked Read 7 Done Done Sent false
31071 9376 Acked Read 7 Done Done Sent false
31072 9377 Acked Write 10 Done Done Sent false
31073 9378 Acked Read 6 Done Done Sent false
31074 9379 Acked Read 5 Done Done Sent false
31075 9380 Acked Write 6 Done Done Sent false
31076 9381 Acked Write 7 Done Done Sent false
31077 9382 Acked Read 4 Done Done Sent false
31078 9383 Acked Read 2 Done Done Sent false
31079 9384 Acked Read 9 Done Done Sent false
31080 9385 Acked Write 2 Done Done Sent false
31081 9386 Acked Read 5 Done Done Sent false
31082 9387 Acked Read 3 Done Done Sent false
31083 9388 Acked Write 8 Done Done Sent false
31084 9389 Acked Write 3 Done Done Sent false
31085 9390 Acked Read 1 Done Done Sent false
31086 9391 Acked Read 3 Done Done Sent false
31087 9392 Acked Read 8 Done Done Sent false
31088 9393 Acked Read 8 Done Done Sent false
31089 9394 Acked Read 10 Done Done Sent false
31090 9395 Acked Write 6 Done Done Sent false
31091 9396 Acked Write 7 Done Done Sent false
31092 9397 Acked Flush 0 Done Done Sent false
31093 9398 Acked Write 10 Done Done Sent false
31094 9399 Acked Write 6 Done Done Sent false
31095 9400 Acked Write 4 Done Done Sent false
31096 9401 Acked Write 10 Done Done Sent false
31097 9402 Acked Read 8 Done Done Sent false
31098 9403 Acked Read 4 Done Done Sent false
31099 9404 Acked Flush 0 Done Done Sent false
31100 9405 Acked Write 9 Done Done Sent false
31101 9406 Acked Write 7 Done Done Sent false
31102 9407 Acked Write 6 Done Done Sent false
31103 9408 Acked Read 3 Done Done Sent false
31104 9409 Acked Flush 0 Done Done Sent false
31105 9410 Acked Flush 0 Done Done Sent false
31106 9411 Acked Write 4 Done Done Sent false
31107 9412 Acked Read 5 Done Done Sent false
31108 9413 Acked Write 9 Done Done Sent false
31109 9414 Acked Write 8 Done Done Sent false
31110 9415 Acked Read 1 Done Done Sent false
31111 9416 Acked Read 10 Done Done Sent false
31112 9417 Acked Read 6 Done Done Sent false
31113 9418 Acked Write 10 Done Done Sent false
31114 9419 Acked Read 5 Done Done Sent false
31115 9420 Acked Read 6 Done Done Sent false
31116 9421 Acked Read 10 Done Done Sent false
31117 9422 Acked Write 5 Done Done Sent false
31118 9423 Acked Flush 0 Done Done Sent false
31119 9424 Acked Read 7 Done Done Sent false
31120 9425 Acked Write 8 Done Done Sent false
31121 9426 Acked Read 2 Done Done Sent false
31122 9427 Acked Read 4 Done Done Sent false
31123 9428 Acked Write 4 Done Done Sent false
31124 9429 Acked Read 8 Done Done Sent false
31125 9430 Acked Write 3 Done Done Sent false
31126 9431 Acked Write 2 Done Done Sent false
31127 9432 Acked Read 3 Done Done Sent false
31128 9433 Acked Read 9 Done Done Sent false
31129 9434 Acked Read 8 Done Done Sent false
31130 9435 Acked Flush 0 Done Done Sent false
31131 9436 Acked Read 9 Done Done Sent false
31132 9437 Acked Read 8 Done Done Sent false
31133 9438 Acked Read 7 Done Done Sent false
31134 9439 Acked Write 7 Done Done Sent false
31135 9440 Acked Read 9 Done Done Sent false
31136 9441 Acked Flush 0 Done Done Sent false
31137 9442 Acked Read 7 Done Done Sent false
31138 9443 Acked Write 3 Done Done Sent false
31139 9444 Acked Read 9 Done Done Sent false
31140 9445 Acked Read 9 Done Done Sent false
31141 9446 Acked Read 4 Done Done Sent false
31142 9447 Acked Flush 0 Done Done Sent false
31143 9448 Acked Write 10 Done Done Sent false
31144 9449 Acked Write 3 Done Done Sent false
31145 9450 Acked Flush 0 Done Done Sent false
31146 9451 Acked Read 9 Done Done Sent false
31147 9452 Acked Write 10 Done Done Sent false
31148 STATES DS:0 DS:1 DS:2 TOTAL
31149 Sent 0 0 301 301
31150 Done 301 301 0 602
31151 Skipped 0 0 0 0
31152 Error 0 0 0 0
31153 Last Flush: Some(JobId(9450)) Some(JobId(9450)) Some(JobId(9151))
31154 Downstairs last five completed: 9151 9150 9149 9148 9147
31155 Upstairs last five completed: 9452 9451 9450 9449 9448
31156 CLIENT: Up:0 ds:301 act:2
311572025-01-28T22:01:09.335ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
311582025-01-28T22:01:09.937ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
31159 [8830] got start action
31160 [8830] I am going to restart
31161 Starting downstairs at port 8830
31162 Make output file at "/tmp/dsc/downstairs-8830.txt"
31163 [8830][2] reports Starting
31164 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1230)
31165 [8830][2] reports Running
311662025-01-28T22:01:10.938ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
311672025-01-28T22:01:11.439ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
311682025-01-28T22:01:11.940ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
311692025-01-28T22:01:12.441ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
31170 ----------------------------------------------------------------
31171 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31172 States: Active Active Offline
31173 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31174 9152 Acked Read 4 Done Done Sent false
31175 9153 Acked Write 10 Done Done Sent false
31176 9154 Acked Write 3 Done Done Sent false
31177 9155 Acked Read 7 Done Done Sent false
31178 9156 Acked Read 5 Done Done Sent false
31179 9157 Acked Read 8 Done Done Sent false
31180 9158 Acked Write 6 Done Done Sent false
31181 9159 Acked Read 4 Done Done Sent false
31182 9160 Acked Write 8 Done Done Sent false
31183 9161 Acked Flush 0 Done Done Sent false
31184 9162 Acked Write 10 Done Done Sent false
31185 9163 Acked Read 3 Done Done Sent false
31186 9164 Acked Write 3 Done Done Sent false
31187 9165 Acked Read 4 Done Done Sent false
31188 9166 Acked Write 2 Done Done Sent false
31189 9167 Acked Read 5 Done Done Sent false
31190 9168 Acked Read 1 Done Done Sent false
31191 9169 Acked Write 7 Done Done Sent false
31192 9170 Acked Write 3 Done Done Sent false
31193 9171 Acked Flush 0 Done Done Sent false
31194 9172 Acked Write 3 Done Done Sent false
31195 9173 Acked Read 8 Done Done Sent false
31196 9174 Acked Write 1 Done Done Sent false
31197 9175 Acked Read 4 Done Done Sent false
31198 9176 Acked Read 10 Done Done Sent false
31199 9177 Acked Read 7 Done Done Sent false
31200 9178 Acked Write 4 Done Done Sent false
31201 9179 Acked Write 1 Done Done Sent false
31202 9180 Acked Write 7 Done Done Sent false
31203 9181 Acked Read 3 Done Done Sent false
31204 9182 Acked Write 4 Done Done Sent false
31205 9183 Acked Write 5 Done Done Sent false
31206 9184 Acked Write 5 Done Done Sent false
31207 9185 Acked Write 9 Done Done Sent false
31208 9186 Acked Flush 0 Done Done Sent false
31209 9187 Acked Write 8 Done Done Sent false
31210 9188 Acked Write 1 Done Done Sent false
31211 9189 Acked Write 10 Done Done Sent false
31212 9190 Acked Write 3 Done Done Sent false
31213 9191 Acked Read 4 Done Done Sent false
31214 9192 Acked Write 9 Done Done Sent false
31215 9193 Acked Read 4 Done Done Sent false
31216 9194 Acked Write 7 Done Done Sent false
31217 9195 Acked Read 9 Done Done Sent false
31218 9196 Acked Write 8 Done Done Sent false
31219 9197 Acked Read 4 Done Done Sent false
31220 9198 Acked Read 10 Done Done Sent false
31221 9199 Acked Write 10 Done Done Sent false
31222 9200 Acked Write 10 Done Done Sent false
31223 9201 Acked Read 3 Done Done Sent false
31224 9202 Acked Flush 0 Done Done Sent false
31225 9203 Acked Flush 0 Done Done Sent false
31226 9204 Acked Read 3 Done Done Sent false
31227 9205 Acked Read 1 Done Done Sent false
31228 9206 Acked Flush 0 Done Done Sent false
31229 9207 Acked Write 9 Done Done Sent false
31230 9208 Acked Read 1 Done Done Sent false
31231 9209 Acked Write 5 Done Done Sent false
31232 9210 Acked Read 5 Done Done Sent false
31233 9211 Acked Write 4 Done Done Sent false
31234 9212 Acked Read 1 Done Done Sent false
31235 9213 Acked Write 4 Done Done Sent false
31236 9214 Acked Write 5 Done Done Sent false
31237 9215 Acked Read 4 Done Done Sent false
31238 9216 Acked Write 2 Done Done Sent false
31239 9217 Acked Write 9 Done Done Sent false
31240 9218 Acked Read 3 Done Done Sent false
31241 9219 Acked Read 1 Done Done Sent false
31242 9220 Acked Write 10 Done Done Sent false
31243 9221 Acked Read 10 Done Done Sent false
31244 9222 Acked Read 2 Done Done Sent false
31245 9223 Acked Read 10 Done Done Sent false
31246 9224 Acked Read 7 Done Done Sent false
31247 9225 Acked Flush 0 Done Done Sent false
31248 9226 Acked Write 4 Done Done Sent false
31249 9227 Acked Flush 0 Done Done Sent false
31250 9228 Acked Read 7 Done Done Sent false
31251 9229 Acked Flush 0 Done Done Sent false
31252 9230 Acked Read 2 Done Done Sent false
31253 9231 Acked Read 9 Done Done Sent false
31254 9232 Acked Write 4 Done Done Sent false
31255 9233 Acked Read 5 Done Done Sent false
31256 9234 Acked Read 8 Done Done Sent false
31257 9235 Acked Read 4 Done Done Sent false
31258 9236 Acked Write 3 Done Done Sent false
31259 9237 Acked Read 5 Done Done Sent false
31260 9238 Acked Flush 0 Done Done Sent false
31261 9239 Acked Read 2 Done Done Sent false
31262 9240 Acked Write 10 Done Done Sent false
31263 9241 Acked Write 10 Done Done Sent false
31264 9242 Acked Read 1 Done Done Sent false
31265 9243 Acked Write 10 Done Done Sent false
31266 9244 Acked Read 7 Done Done Sent false
31267 9245 Acked Flush 0 Done Done Sent false
31268 9246 Acked Flush 0 Done Done Sent false
31269 9247 Acked Write 7 Done Done Sent false
31270 9248 Acked Read 6 Done Done Sent false
31271 9249 Acked Write 6 Done Done Sent false
31272 9250 Acked Write 7 Done Done Sent false
31273 9251 Acked Write 6 Done Done Sent false
31274 9252 Acked Read 1 Done Done Sent false
31275 9253 Acked Read 4 Done Done Sent false
31276 9254 Acked Write 10 Done Done Sent false
31277 9255 Acked Read 8 Done Done Sent false
31278 9256 Acked Read 4 Done Done Sent false
31279 9257 Acked Read 5 Done Done Sent false
31280 9258 Acked Write 8 Done Done Sent false
31281 9259 Acked Write 1 Done Done Sent false
31282 9260 Acked Read 8 Done Done Sent false
31283 9261 Acked Read 7 Done Done Sent false
31284 9262 Acked Read 6 Done Done Sent false
31285 9263 Acked Read 7 Done Done Sent false
31286 9264 Acked Read 2 Done Done Sent false
31287 9265 Acked Flush 0 Done Done Sent false
31288 9266 Acked Read 1 Done Done Sent false
31289 9267 Acked Write 9 Done Done Sent false
31290 9268 Acked Write 6 Done Done Sent false
31291 9269 Acked Read 9 Done Done Sent false
31292 9270 Acked Flush 0 Done Done Sent false
31293 9271 Acked Write 6 Done Done Sent false
31294 9272 Acked Read 2 Done Done Sent false
31295 9273 Acked Write 4 Done Done Sent false
31296 9274 Acked Write 7 Done Done Sent false
31297 9275 Acked Read 4 Done Done Sent false
31298 9276 Acked Write 10 Done Done Sent false
31299 9277 Acked Read 5 Done Done Sent false
31300 9278 Acked Read 7 Done Done Sent false
31301 9279 Acked Read 4 Done Done Sent false
31302 9280 Acked Write 2 Done Done Sent false
31303 9281 Acked Write 7 Done Done Sent false
31304 9282 Acked Read 9 Done Done Sent false
31305 9283 Acked Read 1 Done Done Sent false
31306 9284 Acked Write 9 Done Done Sent false
31307 9285 Acked Write 9 Done Done Sent false
31308 9286 Acked Read 6 Done Done Sent false
31309 9287 Acked Read 2 Done Done Sent false
31310 9288 Acked Read 1 Done Done Sent false
31311 9289 Acked Read 6 Done Done Sent false
31312 9290 Acked Write 1 Done Done Sent false
31313 9291 Acked Read 10 Done Done Sent false
31314 9292 Acked Write 6 Done Done Sent false
31315 9293 Acked Write 4 Done Done Sent false
31316 9294 Acked Read 3 Done Done Sent false
31317 9295 Acked Write 3 Done Done Sent false
31318 9296 Acked Write 5 Done Done Sent false
31319 9297 Acked Flush 0 Done Done Sent false
31320 9298 Acked Read 6 Done Done Sent false
31321 9299 Acked Flush 0 Done Done Sent false
31322 9300 Acked Read 4 Done Done Sent false
31323 9301 Acked Read 6 Done Done Sent false
31324 9302 Acked Read 9 Done Done Sent false
31325 9303 Acked Read 10 Done Done Sent false
31326 9304 Acked Write 8 Done Done Sent false
31327 9305 Acked Write 5 Done Done Sent false
31328 9306 Acked Write 10 Done Done Sent false
31329 9307 Acked Read 7 Done Done Sent false
31330 9308 Acked Read 8 Done Done Sent false
31331 9309 Acked Flush 0 Done Done Sent false
31332 9310 Acked Read 7 Done Done Sent false
31333 9311 Acked Read 2 Done Done Sent false
31334 9312 Acked Read 3 Done Done Sent false
31335 9313 Acked Read 1 Done Done Sent false
31336 9314 Acked Write 5 Done Done Sent false
31337 9315 Acked Read 7 Done Done Sent false
31338 9316 Acked Read 1 Done Done Sent false
31339 9317 Acked Read 1 Done Done Sent false
31340 9318 Acked Read 1 Done Done Sent false
31341 9319 Acked Read 1 Done Done Sent false
31342 9320 Acked Write 7 Done Done Sent false
31343 9321 Acked Read 4 Done Done Sent false
31344 9322 Acked Write 8 Done Done Sent false
31345 9323 Acked Write 6 Done Done Sent false
31346 9324 Acked Write 7 Done Done Sent false
31347 9325 Acked Write 3 Done Done Sent false
31348 9326 Acked Flush 0 Done Done Sent false
31349 9327 Acked Read 5 Done Done Sent false
31350 9328 Acked Read 6 Done Done Sent false
31351 9329 Acked Write 5 Done Done Sent false
31352 9330 Acked Read 4 Done Done Sent false
31353 9331 Acked Read 10 Done Done Sent false
31354 9332 Acked Read 9 Done Done Sent false
31355 9333 Acked Write 8 Done Done Sent false
31356 9334 Acked Flush 0 Done Done Sent false
31357 9335 Acked Read 8 Done Done Sent false
31358 9336 Acked Write 6 Done Done Sent false
31359 9337 Acked Read 9 Done Done Sent false
31360 9338 Acked Read 4 Done Done Sent false
31361 9339 Acked Flush 0 Done Done Sent false
31362 9340 Acked Read 9 Done Done Sent false
31363 9341 Acked Read 3 Done Done Sent false
31364 9342 Acked Read 8 Done Done Sent false
31365 9343 Acked Write 2 Done Done Sent false
31366 9344 Acked Write 5 Done Done Sent false
31367 9345 Acked Write 8 Done Done Sent false
31368 9346 Acked Read 4 Done Done Sent false
31369 9347 Acked Read 7 Done Done Sent false
31370 9348 Acked Write 8 Done Done Sent false
31371 9349 Acked Read 8 Done Done Sent false
31372 9350 Acked Write 5 Done Done Sent false
31373 9351 Acked Write 7 Done Done Sent false
31374 9352 Acked Read 1 Done Done Sent false
31375 9353 Acked Write 6 Done Done Sent false
31376 9354 Acked Write 4 Done Done Sent false
31377 9355 Acked Write 1 Done Done Sent false
31378 9356 Acked Read 7 Done Done Sent false
31379 9357 Acked Read 4 Done Done Sent false
31380 9358 Acked Write 8 Done Done Sent false
31381 9359 Acked Flush 0 Done Done Sent false
31382 9360 Acked Read 8 Done Done Sent false
31383 9361 Acked Write 3 Done Done Sent false
31384 9362 Acked Write 9 Done Done Sent false
31385 9363 Acked Flush 0 Done Done Sent false
31386 9364 Acked Read 2 Done Done Sent false
31387 9365 Acked Write 8 Done Done Sent false
31388 9366 Acked Read 1 Done Done Sent false
31389 9367 Acked Write 8 Done Done Sent false
31390 9368 Acked Read 7 Done Done Sent false
31391 9369 Acked Flush 0 Done Done Sent false
31392 9370 Acked Read 4 Done Done Sent false
31393 9371 Acked Write 2 Done Done Sent false
31394 9372 Acked Write 2 Done Done Sent false
31395 9373 Acked Read 1 Done Done Sent false
31396 9374 Acked Read 8 Done Done Sent false
31397 9375 Acked Read 7 Done Done Sent false
31398 9376 Acked Read 7 Done Done Sent false
31399 9377 Acked Write 10 Done Done Sent false
31400 9378 Acked Read 6 Done Done Sent false
31401 9379 Acked Read 5 Done Done Sent false
31402 9380 Acked Write 6 Done Done Sent false
31403 9381 Acked Write 7 Done Done Sent false
31404 9382 Acked Read 4 Done Done Sent false
31405 9383 Acked Read 2 Done Done Sent false
31406 9384 Acked Read 9 Done Done Sent false
31407 9385 Acked Write 2 Done Done Sent false
31408 9386 Acked Read 5 Done Done Sent false
31409 9387 Acked Read 3 Done Done Sent false
31410 9388 Acked Write 8 Done Done Sent false
31411 9389 Acked Write 3 Done Done Sent false
31412 9390 Acked Read 1 Done Done Sent false
31413 9391 Acked Read 3 Done Done Sent false
31414 9392 Acked Read 8 Done Done Sent false
31415 9393 Acked Read 8 Done Done Sent false
31416 9394 Acked Read 10 Done Done Sent false
31417 9395 Acked Write 6 Done Done Sent false
31418 9396 Acked Write 7 Done Done Sent false
31419 9397 Acked Flush 0 Done Done Sent false
31420 9398 Acked Write 10 Done Done Sent false
31421 9399 Acked Write 6 Done Done Sent false
31422 9400 Acked Write 4 Done Done Sent false
31423 9401 Acked Write 10 Done Done Sent false
31424 9402 Acked Read 8 Done Done Sent false
31425 9403 Acked Read 4 Done Done Sent false
31426 9404 Acked Flush 0 Done Done Sent false
31427 9405 Acked Write 9 Done Done Sent false
31428 9406 Acked Write 7 Done Done Sent false
31429 9407 Acked Write 6 Done Done Sent false
31430 9408 Acked Read 3 Done Done Sent false
31431 9409 Acked Flush 0 Done Done Sent false
31432 9410 Acked Flush 0 Done Done Sent false
31433 9411 Acked Write 4 Done Done Sent false
31434 9412 Acked Read 5 Done Done Sent false
31435 9413 Acked Write 9 Done Done Sent false
31436 9414 Acked Write 8 Done Done Sent false
31437 9415 Acked Read 1 Done Done Sent false
31438 9416 Acked Read 10 Done Done Sent false
31439 9417 Acked Read 6 Done Done Sent false
31440 9418 Acked Write 10 Done Done Sent false
31441 9419 Acked Read 5 Done Done Sent false
31442 9420 Acked Read 6 Done Done Sent false
31443 9421 Acked Read 10 Done Done Sent false
31444 9422 Acked Write 5 Done Done Sent false
31445 9423 Acked Flush 0 Done Done Sent false
31446 9424 Acked Read 7 Done Done Sent false
31447 9425 Acked Write 8 Done Done Sent false
31448 9426 Acked Read 2 Done Done Sent false
31449 9427 Acked Read 4 Done Done Sent false
31450 9428 Acked Write 4 Done Done Sent false
31451 9429 Acked Read 8 Done Done Sent false
31452 9430 Acked Write 3 Done Done Sent false
31453 9431 Acked Write 2 Done Done Sent false
31454 9432 Acked Read 3 Done Done Sent false
31455 9433 Acked Read 9 Done Done Sent false
31456 9434 Acked Read 8 Done Done Sent false
31457 9435 Acked Flush 0 Done Done Sent false
31458 9436 Acked Read 9 Done Done Sent false
31459 9437 Acked Read 8 Done Done Sent false
31460 9438 Acked Read 7 Done Done Sent false
31461 9439 Acked Write 7 Done Done Sent false
31462 9440 Acked Read 9 Done Done Sent false
31463 9441 Acked Flush 0 Done Done Sent false
31464 9442 Acked Read 7 Done Done Sent false
31465 9443 Acked Write 3 Done Done Sent false
31466 9444 Acked Read 9 Done Done Sent false
31467 9445 Acked Read 9 Done Done Sent false
31468 9446 Acked Read 4 Done Done Sent false
31469 9447 Acked Flush 0 Done Done Sent false
31470 9448 Acked Write 10 Done Done Sent false
31471 9449 Acked Write 3 Done Done Sent false
31472 9450 Acked Flush 0 Done Done Sent false
31473 9451 Acked Read 9 Done Done Sent false
31474 9452 Acked Write 10 Done Done Sent false
31475 STATES DS:0 DS:1 DS:2 TOTAL
31476 Sent 0 0 301 301
31477 Done 301 301 0 602
31478 Skipped 0 0 0 0
31479 Error 0 0 0 0
31480 Last Flush: Some(JobId(9450)) Some(JobId(9450)) Some(JobId(9151))
31481 Downstairs last five completed: 9151 9150 9149 9148 9147
31482 Upstairs last five completed: 9452 9451 9450 9449 9448
31483 CLIENT: Up:0 ds:301 act:2
314842025-01-28T22:01:13.944ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
314852025-01-28T22:01:14.587ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
314862025-01-28T22:01:14.587ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47351) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
314872025-01-28T22:01:14.588ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
314882025-01-28T22:01:14.588ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
314892025-01-28T22:01:14.588ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
314902025-01-28T22:01:14.588ZINFOcrucible: send last flush ID to this DS: Some(JobId(9151)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
314912025-01-28T22:01:14.588ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
314922025-01-28T22:01:14.588ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9151)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
314932025-01-28T22:01:14.589ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9151)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
31494 ----------------------------------------------------------------
31495 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31496 Downstairs last five completed: 9453 9452 9451 9450 9449
31497 Upstairs last five completed: 9453 9452 9451 9450 9449
31498 CLIENT: Up:0 ds:0 act:3
31499 Replay: All jobs finished
31500 Main task has work to do, go find it
31501 got dsc Stop(2)
31502 stop 2
31503 Jan 28 22:01:16.838 INFO request completed, latency_us: 392, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c5c452fa-760d-4061-ba98-9c7a5efe0db7, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
31504 [8830] Got stop action so:false kr:false
31505 Jan 28 22:01:16.839 INFO request completed, latency_us: 144, response_code: 200, uri: /state/cid/2, method: GET, req_id: aa92c51d-1dce-4d4e-bb37-fdae5c6e5c09, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
315062025-01-28T22:01:16.839ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
315072025-01-28T22:01:16.839ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
315082025-01-28T22:01:16.839ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
315092025-01-28T22:01:16.839ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
31510 [8830] Exited with: ExitStatus(unix_wait_status(9))
31511 [8830][2] reports Exit
31512 Jan 28 22:01:20.841 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/2, method: GET, req_id: 31b80043-c631-4087-9a60-f90a552eb57b, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
31513 001/300 Write block 830 len 24576 data: 5 4 4 5 6 5
31514 002/300 Read block 3483 len 40960
31515 003/300 Write block 4706 len 20480 data: 7 9 11 12 12
31516 004/300 Read block 4906 len 36864
31517 005/300 Read block 4420 len 4096
31518 006/300 Write block 1297 len 40960 data: 9 9 8 9 8 7 7 8 7 5
31519 007/300 Read block 1451 len 40960
31520 008/300 Read block 1387 len 4096
31521 009/300 Flush
31522 010/300 Write block 3676 len 20480 data: 5 4 3 4 4
31523 011/300 Flush
31524 012/300 Flush
31525 013/300 Read block 81 len 28672
31526 014/300 Write block 4253 len 32768 data: 9 7 6 5 3 3 4 5
31527 015/300 Read block 3252 len 12288
31528 016/300 Read block 460 len 36864
31529 017/300 Write block 2495 len 32768 data: 5 6 4 4 4 4 6 6
31530 018/300 Write block 584 len 20480 data: 5 4 5 5 5
31531 019/300 Write block 2024 len 40960 data: 6 5 4 3 3 2 2 2 4 5
31532 020/300 Read block 2661 len 12288
31533 021/300 Flush
31534 022/300 Read block 3524 len 8192
31535 023/300 Read block 1102 len 36864
31536 024/300 Read block 3532 len 32768
31537 025/300 Read block 2005 len 32768
31538 026/300 Write block 1350 len 36864 data: 5 4 4 6 6 7 6 5 6
31539 027/300 Write block 1532 len 16384 data: 8 8 6 7
31540 028/300 Write block 1393 len 20480 data: 3 3 3 3 3
31541 029/300 Flush
31542 030/300 Write block 435 len 40960 data: 9 9 8 6 7 6 7 6 7 6
31543 031/300 Write block 4268 len 8192 data: 7 6
31544 032/300 Write block 2162 len 28672 data: 5 5 4 5 5 5 6
31545 033/300 Read block 3231 len 24576
31546 034/300 Read block 1365 len 32768
31547 035/300 Write block 1738 len 8192 data: 5 6
31548 036/300 Flush
31549 037/300 Read block 2273 len 12288
31550 038/300 Read block 3492 len 40960
31551 039/300 Write block 2390 len 24576 data: 6 3 4 5 6 7
31552 040/300 Read block 795 len 28672
31553 041/300 Write block 3788 len 32768 data: 6 6 5 5 5 6 6 6
31554 042/300 Write block 3768 len 8192 data: 8 9
31555 043/300 Write block 1794 len 40960 data: 7 6 7 8 8 7 5 4 3 4
31556 044/300 Write block 3045 len 24576 data: 6 7 7 9 9 12
31557 045/300 Read block 2212 len 16384
31558 046/300 Write block 532 len 4096 data: 8
31559 047/300 Read block 3519 len 8192
31560 048/300 Write block 3700 len 36864 data: 6 6 4 5 4 4 5 3 4
31561 049/300 Write block 2252 len 20480 data: 4 4 3 2 2
31562 050/300 Read block 1626 len 32768
31563 051/300 Read block 1412 len 4096
31564 052/300 Flush
31565 053/300 Flush
31566 054/300 Read block 399 len 16384
31567 055/300 Read block 4501 len 20480
31568 056/300 Flush
31569 057/300 Read block 3882 len 28672
31570 058/300 Read block 3682 len 12288
31571 059/300 Flush
31572 060/300 Read block 4174 len 36864
31573 061/300 Write block 3245 len 24576 data: 8 6 5 6 7 8
31574 062/300 Read block 4000 len 24576
31575 063/300 Flush
31576 064/300 Read block 1260 len 16384
31577 065/300 Read block 3324 len 8192
31578 066/300 Read block 2032 len 16384
31579 067/300 Read block 4051 len 20480
31580 068/300 Write block 2055 len 8192 data: 5 5
31581 069/300 Flush
31582 070/300 Read block 1914 len 24576
31583 071/300 Read block 4422 len 32768
31584 072/300 Write block 2156 len 12288 data: 9 9 7
31585 073/300 Write block 2255 len 4096 data: 3
31586 074/300 Write block 4851 len 24576 data: 7 7 8 6 5 6
31587 075/300 Read block 4532 len 28672
31588 076/300 Write block 915 len 20480 data: 5 6 6 7 5
31589 077/300 Write block 516 len 20480 data: 5 5 7 6 6
31590 078/300 Read block 464 len 24576
31591 079/300 Read block 814 len 36864
31592 080/300 Write block 2176 len 4096 data: 3
31593 081/300 Write block 2258 len 20480 data: 3 3 3 5 5
31594 082/300 Read block 2153 len 24576
31595 083/300 Read block 2488 len 8192
31596 084/300 Write block 437 len 28672 data: 9 7 8 7 8 7 8
31597 085/300 Write block 4324 len 32768 data: 8 7 7 7 8 7 7 7
31598 086/300 Read block 292 len 32768
31599 087/300 Read block 4940 len 24576
31600 088/300 Read block 2035 len 40960
31601 089/300 Write block 659 len 24576 data: 3 4 4 5 5 5
31602 090/300 Read block 3553 len 20480
31603 091/300 Flush
31604 092/300 Write block 4019 len 40960 data: 5 5 6 6 5 5 5 5 5 6
31605 093/300 Write block 2399 len 36864 data: 7 7 5 6 7 7 6 5 5
31606 094/300 Read block 562 len 40960
31607 095/300 Read block 2295 len 20480
31608 096/300 Flush
31609 097/300 Flush
31610 098/300 Read block 2622 len 12288
31611 099/300 Flush
31612 100/300 Read block 1953 len 8192
31613 101/300 Read block 3482 len 20480
31614 102/300 Read block 3013 len 36864
31615 103/300 Read block 51 len 24576
31616 104/300 Write block 3693 len 8192 data: 3 3
31617 105/300 Flush
31618 106/300 Read block 1509 len 32768
31619 107/300 Read block 3632 len 20480
31620 108/300 Read block 639 len 4096
31621 109/300 Flush
31622 110/300 Write block 208 len 16384 data: 8 7 7 7
31623 111/300 Write block 2928 len 16384 data: 6 7 7 6
31624 112/300 Read block 1134 len 20480
31625 113/300 Write block 190 len 4096 data: 5
31626 114/300 Write block 509 len 32768 data: 3 5 7 7 5 6 5 6
31627 115/300 Write block 4256 len 8192 data: 6 4
31628 116/300 Read block 3353 len 36864
31629 117/300 Write block 349 len 40960 data: 10 9 9 6 7 7 7 9 7 7
31630 118/300 Flush
31631 119/300 Read block 1134 len 28672
31632 120/300 Write block 1158 len 32768 data: 5 4 3 3 4 4 3 4
31633 121/300 Read block 2072 len 12288
31634 122/300 Read block 3112 len 40960
31635 123/300 Read block 1997 len 16384
31636 124/300 Write block 4659 len 28672 data: 11 10 11 11 11 10 10
31637 125/300 Flush
31638 126/300 Read block 781 len 32768
31639 127/300 Write block 1935 len 20480 data: 5 5 3 3 3
31640 128/300 Write block 1081 len 16384 data: 8 7 4 4
31641 129/300 Flush
31642 130/300 Flush
31643 131/300 Flush
31644 132/300 Flush
31645 133/300 Read block 4420 len 16384
31646 134/300 Read block 2316 len 32768
31647 135/300 Flush
31648 136/300 Read block 1741 len 40960
31649 137/300 Read block 1365 len 4096
31650 138/300 Write block 2209 len 8192 data: 5 5
31651 139/300 Write block 699 len 28672 data: 8 8 8 7 7 5 6
31652 140/300 Flush
31653 141/300 Read block 4826 len 12288
31654 142/300 Write block 1288 len 32768 data: 6 6 8 9 8 8 7 8
31655 143/300 Read block 1136 len 28672
31656 144/300 Write block 4864 len 36864 data: 6 6 8 8 6 6 6 6 5
31657 145/300 Write block 2777 len 12288 data: 4 4 6
31658 146/300 Flush
31659 147/300 Write block 281 len 8192 data: 4 5
31660 148/300 Write block 1600 len 24576 data: 5 5 5 6 6 5
31661 149/300 Read block 1304 len 4096
31662 150/300 Write block 3474 len 16384 data: 7 6 6 6
31663 151/300 Read block 31 len 36864
31664 152/300 Read block 4281 len 4096
31665 153/300 Read block 3055 len 12288
31666 154/300 Read block 2606 len 16384
31667 155/300 Read block 1796 len 32768
31668 156/300 Write block 4068 len 20480 data: 5 7 7 6 7
31669 157/300 Write block 481 len 4096 data: 7
31670 158/300 Flush
31671 159/300 Read block 3082 len 20480
31672 160/300 Read block 1844 len 4096
31673 161/300 Write block 478 len 36864 data: 8 7 8 8 7 8 6 6 6
31674 162/300 Read block 2202 len 12288
31675 163/300 Write block 1717 len 16384 data: 8 8 7 8
31676 164/300 Write block 1852 len 4096 data: 3
31677 165/300 Read block 1366 len 16384
31678 166/300 Write block 491 len 8192 data: 6 5
31679 167/300 Write block 3898 len 40960 data: 4 7 7 7 6 5 4 6 7 6
31680 168/300 Read block 2398 len 28672
31681 169/300 Flush
31682 170/300 Read block 1088 len 20480
31683 171/300 Read block 3625 len 40960
31684 172/300 Read block 2577 len 40960
31685 173/300 Read block 1162 len 4096
31686 174/300 Read block 2166 len 40960
31687 175/300 Write block 1221 len 36864 data: 6 5 5 4 4 4 2 3 3
31688 176/300 Write block 2335 len 12288 data: 5 5 7
31689 177/300 Flush
31690 178/300 Write block 3655 len 40960 data: 4 3 5 5 4 6 8 7 8 8
31691 179/300 Write block 4083 len 16384 data: 11 10 9 10
31692 180/300 Write block 284 len 32768 data: 5 6 5 5 4 5 5 4
31693 181/300 Read block 2221 len 16384
31694 182/300 Read block 1851 len 32768
31695 183/300 Write block 1434 len 28672 data: 6 6 7 7 8 7 5
31696 184/300 Read block 2303 len 28672
31697 185/300 Read block 3917 len 4096
31698 186/300 Write block 486 len 8192 data: 7 5
31699 187/300 Read block 2058 len 28672
31700 188/300 Write block 879 len 24576 data: 8 9 9 8 7 6
31701 189/300 Read block 3521 len 4096
31702 190/300 Read block 2303 len 32768
31703 191/300 Write block 3434 len 8192 data: 5 4
31704 192/300 Read block 2426 len 20480
31705 193/300 Write block 1549 len 28672 data: 6 6 5 6 4 4 4
31706 194/300 Write block 1713 len 24576 data: 8 7 6 6 9 9
31707 195/300 Write block 4904 len 16384 data: 3 4 3 3
31708 196/300 Read block 935 len 28672
31709 197/300 Read block 1127 len 28672
31710 198/300 Write block 1941 len 36864 data: 3 3 3 5 4 5 5 3 5
31711 199/300 Flush
31712 200/300 Write block 756 len 32768 data: 5 4 6 6 7 5 8 8
31713 201/300 Write block 3275 len 20480 data: 6 6 5 3 3
31714 202/300 Flush
31715 203/300 Write block 3117 len 8192 data: 7 6
31716 204/300 Read block 2812 len 40960
31717 205/300 Read block 407 len 4096
31718 206/300 Read block 4957 len 36864
31719 207/300 Read block 4516 len 40960
31720 208/300 Flush
31721 209/300 Read block 2188 len 32768
31722 210/300 Write block 1503 len 24576 data: 4 3 4 6 5 5
31723 211/300 Read block 1774 len 32768
31724 212/300 Read block 3864 len 36864
31725 213/300 Write block 4609 len 16384 data: 5 5 7 6
31726 214/300 Write block 4768 len 20480 data: 7 9 10 12 13
31727 215/300 Write block 196 len 12288 data: 6 6 6
31728 216/300 Read block 2121 len 16384
31729 217/300 Read block 4259 len 32768
31730 218/300 Write block 353 len 12288 data: 8 8 8
31731 219/300 Flush
31732 220/300 Read block 3720 len 40960
31733 221/300 Read block 2838 len 4096
31734 222/300 Read block 4323 len 40960
31735 223/300 Write block 4086 len 32768 data: 11 10 8 8 9 9 9 7
31736 224/300 Write block 4088 len 24576 data: 9 9 10 10 10 8
31737 225/300 Write block 2218 len 16384 data: 4 4 5 5
31738 226/300 Read block 2886 len 16384
31739 227/300 Flush
31740 228/300 Write block 149 len 4096 data: 4
31741 229/300 Read block 109 len 28672
31742 230/300 Read block 2823 len 40960
31743 231/300 Flush
31744 232/300 Read block 4533 len 40960
31745 233/300 Read block 2282 len 24576
31746 234/300 Write block 4399 len 12288 data: 4 5 6
31747 235/300 Read block 2922 len 28672
31748 236/300 Read block 1724 len 4096
31749 237/300 Read block 3474 len 8192
31750 238/300 Write block 2897 len 36864 data: 6 6 8 7 7 8 8 7 7
31751 239/300 Read block 2646 len 28672
31752 240/300 Read block 4406 len 32768
31753 241/300 Write block 650 len 40960 data: 5 8 10 10 7 6 5 6 5 4
31754 242/300 Read block 2634 len 36864
31755 243/300 Write block 2921 len 12288 data: 6 6 5
31756 244/300 Write block 3600 len 28672 data: 7 7 8 7 6 7 6
31757 245/300 Write block 4217 len 24576 data: 4 4 4 4 4 4
31758 246/300 Write block 4824 len 36864 data: 6 5 4 4 5 5 5 4 4
31759 247/300 Flush
31760 248/300 Write block 1641 len 40960 data: 7 7 6 6 5 3 5 4 4 4
31761 249/300 Read block 3517 len 40960
31762 250/300 Read block 4363 len 24576
31763 251/300 Read block 4765 len 4096
31764 252/300 Write block 4084 len 36864 data: 11 10 12 11 10 10 11 11 11
31765 253/300 Read block 4663 len 36864
31766 254/300 Read block 2684 len 8192
31767 255/300 Write block 3896 len 24576 data: 6 5 5 8 8 8
31768 256/300 Flush
31769 257/300 Read block 3621 len 24576
31770 258/300 Read block 4425 len 36864
31771 259/300 Write block 963 len 16384 data: 6 6 6 7
31772 260/300 Read block 4117 len 20480
31773 261/300 Read block 379 len 40960
31774 262/300 Write block 2791 len 20480 data: 5 5 6 7 6
31775 263/300 Write block 3857 len 16384 data: 3 4 2 2
31776 264/300 Write block 3702 len 20480 data: 5 6 5 5 6
31777 265/300 Flush
31778 266/300 Read block 2192 len 16384
31779 267/300 Read block 3733 len 4096
31780 268/300 Read block 2504 len 20480
31781 269/300 Read block 3717 len 40960
31782 270/300 Write block 128 len 20480 data: 4 3 4 3 4
31783 271/300 Flush
31784 272/300 Read block 4970 len 16384
31785 273/300 Read block 3484 len 8192
31786 274/300 Read block 4621 len 20480
31787 275/300 Write block 3129 len 20480 data: 4 4 5 5 6
31788 276/300 Write block 3688 len 8192 data: 4 3
31789 277/300 Read block 1835 len 16384
31790 278/300 Write block 1058 len 36864 data: 6 7 6 5 6 6 6 6 5
31791 279/300 Flush
31792 280/300 Write block 4107 len 28672 data: 5 6 5 4 6 6 7
31793 281/300 Flush
31794 282/300 Read block 3954 len 28672
31795 283/300 Flush
31796 284/300 Read block 2275 len 40960
31797 285/300 Read block 3688 len 28672
31798 286/300 Read block 2828 len 40960
31799 287/300 Read block 3975 len 16384
31800 288/300 Flush
31801 289/300 Flush
31802 290/300 Write block 1821 len 16384 data: 8 5 5 7
31803 291/300 Write block 2942 len 12288 data: 7 7 6
31804 292/300 Read block 2271 len 24576
31805 293/300 Write block 950 len 36864 data: 5 6 8 6 6 6 5 6 7
31806 294/300 Read block 3354 len 16384
31807 295/300 Read block 358 len 20480
31808 296/300 Flush
31809 297/300 Flush
31810 298/300 Write block 438 len 32768 data: 8 9 8 9 8 9 7 6
31811 299/300 Read block 4314 len 28672
31812 300/300 Write block 3628 len 16384 data: 5 4 4 4
31813 301/300 Read block 3477 len 12288
31814 Main task has work to do, go find it
31815 got dsc Start(2)
31816 start 2
31817 Jan 28 22:01:21.086 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: 7817f069-00f4-4698-8f8b-47f27c77bb91, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
31818 Replay: started 2, returned:Ok(())
31819 ----------------------------------------------------------------
31820 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31821 States: Active Active Offline
31822 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31823 9454 Acked Write 6 Done Done Sent false
31824 9455 Acked Read 10 Done Done Sent false
31825 9456 Acked Write 5 Done Done Sent false
31826 9457 Acked Read 9 Done Done Sent false
31827 9458 Acked Read 1 Done Done Sent false
31828 9459 Acked Write 10 Done Done Sent false
31829 9460 Acked Read 10 Done Done Sent false
31830 9461 Acked Read 1 Done Done Sent false
31831 9462 Acked Flush 0 Done Done Sent false
31832 9463 Acked Write 5 Done Done Sent false
31833 9464 Acked Flush 0 Done Done Sent false
31834 9465 Acked Flush 0 Done Done Sent false
31835 9466 Acked Read 7 Done Done Sent false
31836 9467 Acked Write 8 Done Done Sent false
31837 9468 Acked Read 3 Done Done Sent false
31838 9469 Acked Read 9 Done Done Sent false
31839 9470 Acked Write 8 Done Done Sent false
31840 9471 Acked Write 5 Done Done Sent false
31841 9472 Acked Write 10 Done Done Sent false
31842 9473 Acked Read 3 Done Done Sent false
31843 9474 Acked Flush 0 Done Done Sent false
31844 9475 Acked Read 2 Done Done Sent false
31845 9476 Acked Read 9 Done Done Sent false
31846 9477 Acked Read 8 Done Done Sent false
31847 9478 Acked Read 8 Done Done Sent false
31848 9479 Acked Write 9 Done Done Sent false
31849 9480 Acked Write 4 Done Done Sent false
31850 9481 Acked Write 5 Done Done Sent false
31851 9482 Acked Flush 0 Done Done Sent false
31852 9483 Acked Write 10 Done Done Sent false
31853 9484 Acked Write 2 Done Done Sent false
31854 9485 Acked Write 7 Done Done Sent false
31855 9486 Acked Read 6 Done Done Sent false
31856 9487 Acked Read 8 Done Done Sent false
31857 9488 Acked Write 2 Done Done Sent false
31858 9489 Acked Flush 0 Done Done Sent false
31859 9490 Acked Read 3 Done Done Sent false
31860 9491 Acked Read 10 Done Done Sent false
31861 9492 Acked Write 6 Done Done Sent false
31862 9493 Acked Read 7 Done Done Sent false
31863 9494 Acked Write 8 Done Done Sent false
31864 9495 Acked Write 2 Done Done Sent false
31865 9496 Acked Write 10 Done Done Sent false
31866 9497 Acked Write 6 Done Done Sent false
31867 9498 Acked Read 4 Done Done Sent false
31868 9499 Acked Write 1 Done Done Sent false
31869 9500 Acked Read 2 Done Done Sent false
31870 9501 Acked Write 9 Done Done Sent false
31871 9502 Acked Write 5 Done Done Sent false
31872 9503 Acked Read 8 Done Done Sent false
31873 9504 Acked Read 1 Done Done Sent false
31874 9505 Acked Flush 0 Done Done Sent false
31875 9506 Acked Flush 0 Done Done Sent false
31876 9507 Acked Read 4 Done Done Sent false
31877 9508 Acked Read 5 Done Done Sent false
31878 9509 Acked Flush 0 Done Done Sent false
31879 9510 Acked Read 7 Done Done Sent false
31880 9511 Acked Read 3 Done Done Sent false
31881 9512 Acked Flush 0 Done Done Sent false
31882 9513 Acked Read 9 Done Done Sent false
31883 9514 Acked Write 6 Done Done Sent false
31884 9515 Acked Read 6 Done Done Sent false
31885 9516 Acked Flush 0 Done Done Sent false
31886 9517 Acked Read 4 Done Done Sent false
31887 9518 Acked Read 2 Done Done Sent false
31888 9519 Acked Read 4 Done Done Sent false
31889 9520 Acked Read 5 Done Done Sent false
31890 9521 Acked Write 2 Done Done Sent false
31891 9522 Acked Flush 0 Done Done Sent false
31892 9523 Acked Read 6 Done Done Sent false
31893 9524 Acked Read 8 Done Done Sent false
31894 9525 Acked Write 3 Done Done Sent false
31895 9526 Acked Write 1 Done Done Sent false
31896 9527 Acked Write 6 Done Done Sent false
31897 9528 Acked Read 7 Done Done Sent false
31898 9529 Acked Write 5 Done Done Sent false
31899 9530 Acked Write 5 Done Done Sent false
31900 9531 Acked Read 6 Done Done Sent false
31901 9532 Acked Read 9 Done Done Sent false
31902 9533 Acked Write 1 Done Done Sent false
31903 9534 Acked Write 5 Done Done Sent false
31904 9535 Acked Read 6 Done Done Sent false
31905 9536 Acked Read 2 Done Done Sent false
31906 9537 Acked Write 7 Done Done Sent false
31907 9538 Acked Write 8 Done Done Sent false
31908 9539 Acked Read 8 Done Done Sent false
31909 9540 Acked Read 6 Done Done Sent false
31910 9541 Acked Read 10 Done Done Sent false
31911 9542 Acked Write 6 Done Done Sent false
31912 9543 Acked Read 5 Done Done Sent false
31913 9544 Acked Flush 0 Done Done Sent false
31914 9545 Acked Write 10 Done Done Sent false
31915 9546 Acked Write 9 Done Done Sent false
31916 9547 Acked Read 10 Done Done Sent false
31917 9548 Acked Read 5 Done Done Sent false
31918 9549 Acked Flush 0 Done Done Sent false
31919 9550 Acked Flush 0 Done Done Sent false
31920 9551 Acked Read 3 Done Done Sent false
31921 9552 Acked Flush 0 Done Done Sent false
31922 9553 Acked Read 2 Done Done Sent false
31923 9554 Acked Read 5 Done Done Sent false
31924 9555 Acked Read 9 Done Done Sent false
31925 9556 Acked Read 6 Done Done Sent false
31926 9557 Acked Write 2 Done Done Sent false
31927 9558 Acked Flush 0 Done Done Sent false
31928 9559 Acked Read 8 Done Done Sent false
31929 9560 Acked Read 5 Done Done Sent false
31930 9561 Acked Read 1 Done Done Sent false
31931 9562 Acked Flush 0 Done Done Sent false
31932 9563 Acked Write 4 Done Done Sent false
31933 9564 Acked Write 4 Done Done Sent false
31934 9565 Acked Read 5 Done Done Sent false
31935 9566 Acked Write 1 Done Done Sent false
31936 9567 Acked Write 8 Done Done Sent false
31937 9568 Acked Write 2 Done Done Sent false
31938 9569 Acked Read 9 Done Done Sent false
31939 9570 Acked Write 10 Done Done Sent false
31940 9571 Acked Flush 0 Done Done Sent false
31941 9572 Acked Read 7 Done Done Sent false
31942 9573 Acked Write 8 Done Done Sent false
31943 9574 Acked Read 3 Done Done Sent false
31944 9575 Acked Read 10 Done Done Sent false
31945 9576 Acked Read 4 Done Done Sent false
31946 9577 Acked Write 7 Done Done Sent false
31947 9578 Acked Flush 0 Done Done Sent false
31948 9579 Acked Read 8 Done Done Sent false
31949 9580 Acked Write 5 Done Done Sent false
31950 9581 Acked Write 4 Done Done Sent false
31951 9582 Acked Flush 0 Done Done Sent false
31952 9583 Acked Flush 0 Done Done Sent false
31953 9584 Acked Flush 0 Done Done Sent false
31954 9585 Acked Flush 0 Done Done Sent false
31955 9586 Acked Read 4 Done Done Sent false
31956 9587 Acked Read 8 Done Done Sent false
31957 9588 Acked Flush 0 Done Done Sent false
31958 9589 Acked Read 10 Done Done Sent false
31959 9590 Acked Read 1 Done Done Sent false
31960 9591 Acked Write 2 Done Done Sent false
31961 9592 Acked Write 7 Done Done Sent false
31962 9593 Acked Flush 0 Done Done Sent false
31963 9594 Acked Read 3 Done Done Sent false
31964 9595 Acked Write 8 Done Done Sent false
31965 9596 Acked Read 7 Done Done Sent false
31966 9597 Acked Write 9 Done Done Sent false
31967 9598 Acked Write 3 Done Done Sent false
31968 9599 Acked Flush 0 Done Done Sent false
31969 9600 Acked Write 2 Done Done Sent false
31970 9601 Acked Write 6 Done Done Sent false
31971 9602 Acked Read 1 Done Done Sent false
31972 9603 Acked Write 4 Done Done Sent false
31973 9604 Acked Read 9 Done Done Sent false
31974 9605 Acked Read 1 Done Done Sent false
31975 9606 Acked Read 3 Done Done Sent false
31976 9607 Acked Read 4 Done Done Sent false
31977 9608 Acked Read 8 Done Done Sent false
31978 9609 Acked Write 5 Done Done Sent false
31979 9610 Acked Write 1 Done Done Sent false
31980 9611 Acked Flush 0 Done Done Sent false
31981 9612 Acked Read 5 Done Done Sent false
31982 9613 Acked Read 1 Done Done Sent false
31983 9614 Acked Write 9 Done Done Sent false
31984 9615 Acked Read 3 Done Done Sent false
31985 9616 Acked Write 4 Done Done Sent false
31986 9617 Acked Write 1 Done Done Sent false
31987 9618 Acked Read 4 Done Done Sent false
31988 9619 Acked Write 2 Done Done Sent false
31989 9620 Acked Write 10 Done Done Sent false
31990 9621 Acked Read 7 Done Done Sent false
31991 9622 Acked Flush 0 Done Done Sent false
31992 9623 Acked Read 5 Done Done Sent false
31993 9624 Acked Read 10 Done Done Sent false
31994 9625 Acked Read 10 Done Done Sent false
31995 9626 Acked Read 1 Done Done Sent false
31996 9627 Acked Read 10 Done Done Sent false
31997 9628 Acked Write 9 Done Done Sent false
31998 9629 Acked Write 3 Done Done Sent false
31999 9630 Acked Flush 0 Done Done Sent false
32000 9631 Acked Write 10 Done Done Sent false
32001 9632 Acked Write 4 Done Done Sent false
32002 9633 Acked Write 8 Done Done Sent false
32003 9634 Acked Read 4 Done Done Sent false
32004 9635 Acked Read 8 Done Done Sent false
32005 9636 Acked Write 7 Done Done Sent false
32006 9637 Acked Read 7 Done Done Sent false
32007 9638 Acked Read 1 Done Done Sent false
32008 9639 Acked Write 2 Done Done Sent false
32009 9640 Acked Read 7 Done Done Sent false
32010 9641 Acked Write 6 Done Done Sent false
32011 9642 Acked Read 1 Done Done Sent false
32012 9643 Acked Read 8 Done Done Sent false
32013 9644 Acked Write 2 Done Done Sent false
32014 9645 Acked Read 5 Done Done Sent false
32015 9646 Acked Write 7 Done Done Sent false
32016 9647 Acked Write 6 Done Done Sent false
32017 9648 Acked Write 4 Done Done Sent false
32018 9649 Acked Read 7 Done Done Sent false
32019 9650 Acked Read 7 Done Done Sent false
32020 9651 Acked Write 9 Done Done Sent false
32021 9652 Acked Flush 0 Done Done Sent false
32022 9653 Acked Write 8 Done Done Sent false
32023 9654 Acked Write 5 Done Done Sent false
32024 9655 Acked Flush 0 Done Done Sent false
32025 9656 Acked Write 2 Done Done Sent false
32026 9657 Acked Read 10 Done Done Sent false
32027 9658 Acked Read 1 Done Done Sent false
32028 9659 Acked Read 9 Done Done Sent false
32029 9660 Acked Read 10 Done Done Sent false
32030 9661 Acked Flush 0 Done Done Sent false
32031 9662 Acked Read 8 Done Done Sent false
32032 9663 Acked Write 6 Done Done Sent false
32033 9664 Acked Read 8 Done Done Sent false
32034 9665 Acked Read 9 Done Done Sent false
32035 9666 Acked Write 4 Done Done Sent false
32036 9667 Acked Write 5 Done Done Sent false
32037 9668 Acked Write 3 Done Done Sent false
32038 9669 Acked Read 4 Done Done Sent false
32039 9670 Acked Read 8 Done Done Sent false
32040 9671 Acked Write 3 Done Done Sent false
32041 9672 Acked Flush 0 Done Done Sent false
32042 9673 Acked Read 10 Done Done Sent false
32043 9674 Acked Read 1 Done Done Sent false
32044 9675 Acked Read 10 Done Done Sent false
32045 9676 Acked Write 8 Done Done Sent false
32046 9677 Acked Write 6 Done Done Sent false
32047 9678 Acked Write 4 Done Done Sent false
32048 9679 Acked Read 4 Done Done Sent false
32049 9680 Acked Flush 0 Done Done Sent false
32050 9681 Acked Write 1 Done Done Sent false
32051 9682 Acked Read 7 Done Done Sent false
32052 9683 Acked Read 10 Done Done Sent false
32053 9684 Acked Flush 0 Done Done Sent false
32054 9685 Acked Read 10 Done Done Sent false
32055 9686 Acked Read 6 Done Done Sent false
32056 9687 Acked Write 3 Done Done Sent false
32057 9688 Acked Read 7 Done Done Sent false
32058 9689 Acked Read 1 Done Done Sent false
32059 9690 Acked Read 2 Done Done Sent false
32060 9691 Acked Write 9 Done Done Sent false
32061 9692 Acked Read 7 Done Done Sent false
32062 9693 Acked Read 8 Done Done Sent false
32063 9694 Acked Write 10 Done Done Sent false
32064 9695 Acked Read 9 Done Done Sent false
32065 9696 Acked Write 3 Done Done Sent false
32066 9697 Acked Write 7 Done Done Sent false
32067 9698 Acked Write 6 Done Done Sent false
32068 9699 Acked Write 9 Done Done Sent false
32069 9700 Acked Flush 0 Done Done Sent false
32070 9701 Acked Write 10 Done Done Sent false
32071 9702 Acked Read 10 Done Done Sent false
32072 9703 Acked Read 6 Done Done Sent false
32073 9704 Acked Read 1 Done Done Sent false
32074 9705 Acked Write 9 Done Done Sent false
32075 9706 Acked Read 9 Done Done Sent false
32076 9707 Acked Read 2 Done Done Sent false
32077 9708 Acked Write 6 Done Done Sent false
32078 9709 Acked Flush 0 Done Done Sent false
32079 9710 Acked Read 6 Done Done Sent false
32080 9711 Acked Read 9 Done Done Sent false
32081 9712 Acked Write 4 Done Done Sent false
32082 9713 Acked Read 5 Done Done Sent false
32083 9714 Acked Read 10 Done Done Sent false
32084 9715 Acked Write 5 Done Done Sent false
32085 9716 Acked Write 4 Done Done Sent false
32086 9717 Acked Write 5 Done Done Sent false
32087 9718 Acked Flush 0 Done Done Sent false
32088 9719 Acked Read 4 Done Done Sent false
32089 9720 Acked Read 1 Done Done Sent false
32090 9721 Acked Read 5 Done Done Sent false
32091 9722 Acked Read 10 Done Done Sent false
32092 9723 Acked Write 5 Done Done Sent false
32093 9724 Acked Flush 0 Done Done Sent false
32094 9725 Acked Read 4 Done Done Sent false
32095 9726 Acked Read 2 Done Done Sent false
32096 9727 Acked Read 5 Done Done Sent false
32097 9728 Acked Write 5 Done Done Sent false
32098 9729 Acked Write 2 Done Done Sent false
32099 9730 Acked Read 4 Done Done Sent false
32100 9731 Acked Write 9 Done Done Sent false
32101 9732 Acked Flush 0 Done Done Sent false
32102 9733 Acked Write 7 Done Done Sent false
32103 9734 Acked Flush 0 Done Done Sent false
32104 9735 Acked Read 7 Done Done Sent false
32105 9736 Acked Flush 0 Done Done Sent false
32106 9737 Acked Read 10 Done Done Sent false
32107 9738 Acked Read 7 Done Done Sent false
32108 9739 Acked Read 10 Done Done Sent false
32109 9740 Acked Read 4 Done Done Sent false
32110 9741 Acked Flush 0 Done Done Sent false
32111 9742 Acked Flush 0 Done Done Sent false
32112 9743 Acked Write 4 Done Done Sent false
32113 9744 Acked Write 3 Done Done Sent false
32114 9745 Acked Read 6 Done Done Sent false
32115 9746 Acked Write 9 Done Done Sent false
32116 9747 Acked Read 4 Done Done Sent false
32117 9748 Acked Read 5 Done Done Sent false
32118 9749 Acked Flush 0 Done Done Sent false
32119 9750 Acked Flush 0 Done Done Sent false
32120 9751 Acked Write 8 Done Done Sent false
32121 9752 Acked Read 7 Done Done Sent false
32122 9753 Acked Write 4 Done Done Sent false
32123 9754 Acked Read 3 Done Done Sent false
32124 STATES DS:0 DS:1 DS:2 TOTAL
32125 Sent 0 0 301 301
32126 Done 301 301 0 602
32127 Skipped 0 0 0 0
32128 Error 0 0 0 0
32129 Last Flush: Some(JobId(9750)) Some(JobId(9750)) Some(JobId(9453))
32130 Downstairs last five completed: 9453 9452 9451 9450 9449
32131 Upstairs last five completed: 9754 9753 9752 9751 9750
32132 CLIENT: Up:0 ds:301 act:2
321332025-01-28T22:01:21.957ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
32134 [8830] got start action
32135 [8830] I am going to restart
32136 Starting downstairs at port 8830
32137 Make output file at "/tmp/dsc/downstairs-8830.txt"
32138 [8830][2] reports Starting
32139 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1231)
32140 [8830][2] reports Running
321412025-01-28T22:01:22.959ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
321422025-01-28T22:01:23.960ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
321432025-01-28T22:01:24.961ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
32144 ----------------------------------------------------------------
32145 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32146 States: Active Active Offline
32147 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32148 9454 Acked Write 6 Done Done Sent false
32149 9455 Acked Read 10 Done Done Sent false
32150 9456 Acked Write 5 Done Done Sent false
32151 9457 Acked Read 9 Done Done Sent false
32152 9458 Acked Read 1 Done Done Sent false
32153 9459 Acked Write 10 Done Done Sent false
32154 9460 Acked Read 10 Done Done Sent false
32155 9461 Acked Read 1 Done Done Sent false
32156 9462 Acked Flush 0 Done Done Sent false
32157 9463 Acked Write 5 Done Done Sent false
32158 9464 Acked Flush 0 Done Done Sent false
32159 9465 Acked Flush 0 Done Done Sent false
32160 9466 Acked Read 7 Done Done Sent false
32161 9467 Acked Write 8 Done Done Sent false
32162 9468 Acked Read 3 Done Done Sent false
32163 9469 Acked Read 9 Done Done Sent false
32164 9470 Acked Write 8 Done Done Sent false
32165 9471 Acked Write 5 Done Done Sent false
32166 9472 Acked Write 10 Done Done Sent false
32167 9473 Acked Read 3 Done Done Sent false
32168 9474 Acked Flush 0 Done Done Sent false
32169 9475 Acked Read 2 Done Done Sent false
32170 9476 Acked Read 9 Done Done Sent false
32171 9477 Acked Read 8 Done Done Sent false
32172 9478 Acked Read 8 Done Done Sent false
32173 9479 Acked Write 9 Done Done Sent false
32174 9480 Acked Write 4 Done Done Sent false
32175 9481 Acked Write 5 Done Done Sent false
32176 9482 Acked Flush 0 Done Done Sent false
32177 9483 Acked Write 10 Done Done Sent false
32178 9484 Acked Write 2 Done Done Sent false
32179 9485 Acked Write 7 Done Done Sent false
32180 9486 Acked Read 6 Done Done Sent false
32181 9487 Acked Read 8 Done Done Sent false
32182 9488 Acked Write 2 Done Done Sent false
32183 9489 Acked Flush 0 Done Done Sent false
32184 9490 Acked Read 3 Done Done Sent false
32185 9491 Acked Read 10 Done Done Sent false
32186 9492 Acked Write 6 Done Done Sent false
32187 9493 Acked Read 7 Done Done Sent false
32188 9494 Acked Write 8 Done Done Sent false
32189 9495 Acked Write 2 Done Done Sent false
32190 9496 Acked Write 10 Done Done Sent false
32191 9497 Acked Write 6 Done Done Sent false
32192 9498 Acked Read 4 Done Done Sent false
32193 9499 Acked Write 1 Done Done Sent false
32194 9500 Acked Read 2 Done Done Sent false
32195 9501 Acked Write 9 Done Done Sent false
32196 9502 Acked Write 5 Done Done Sent false
32197 9503 Acked Read 8 Done Done Sent false
32198 9504 Acked Read 1 Done Done Sent false
32199 9505 Acked Flush 0 Done Done Sent false
32200 9506 Acked Flush 0 Done Done Sent false
32201 9507 Acked Read 4 Done Done Sent false
32202 9508 Acked Read 5 Done Done Sent false
32203 9509 Acked Flush 0 Done Done Sent false
32204 9510 Acked Read 7 Done Done Sent false
32205 9511 Acked Read 3 Done Done Sent false
32206 9512 Acked Flush 0 Done Done Sent false
32207 9513 Acked Read 9 Done Done Sent false
32208 9514 Acked Write 6 Done Done Sent false
32209 9515 Acked Read 6 Done Done Sent false
32210 9516 Acked Flush 0 Done Done Sent false
32211 9517 Acked Read 4 Done Done Sent false
32212 9518 Acked Read 2 Done Done Sent false
32213 9519 Acked Read 4 Done Done Sent false
32214 9520 Acked Read 5 Done Done Sent false
32215 9521 Acked Write 2 Done Done Sent false
32216 9522 Acked Flush 0 Done Done Sent false
32217 9523 Acked Read 6 Done Done Sent false
32218 9524 Acked Read 8 Done Done Sent false
32219 9525 Acked Write 3 Done Done Sent false
32220 9526 Acked Write 1 Done Done Sent false
32221 9527 Acked Write 6 Done Done Sent false
32222 9528 Acked Read 7 Done Done Sent false
32223 9529 Acked Write 5 Done Done Sent false
32224 9530 Acked Write 5 Done Done Sent false
32225 9531 Acked Read 6 Done Done Sent false
32226 9532 Acked Read 9 Done Done Sent false
32227 9533 Acked Write 1 Done Done Sent false
32228 9534 Acked Write 5 Done Done Sent false
32229 9535 Acked Read 6 Done Done Sent false
32230 9536 Acked Read 2 Done Done Sent false
32231 9537 Acked Write 7 Done Done Sent false
32232 9538 Acked Write 8 Done Done Sent false
32233 9539 Acked Read 8 Done Done Sent false
32234 9540 Acked Read 6 Done Done Sent false
32235 9541 Acked Read 10 Done Done Sent false
32236 9542 Acked Write 6 Done Done Sent false
32237 9543 Acked Read 5 Done Done Sent false
32238 9544 Acked Flush 0 Done Done Sent false
32239 9545 Acked Write 10 Done Done Sent false
32240 9546 Acked Write 9 Done Done Sent false
32241 9547 Acked Read 10 Done Done Sent false
32242 9548 Acked Read 5 Done Done Sent false
32243 9549 Acked Flush 0 Done Done Sent false
32244 9550 Acked Flush 0 Done Done Sent false
32245 9551 Acked Read 3 Done Done Sent false
32246 9552 Acked Flush 0 Done Done Sent false
32247 9553 Acked Read 2 Done Done Sent false
32248 9554 Acked Read 5 Done Done Sent false
32249 9555 Acked Read 9 Done Done Sent false
32250 9556 Acked Read 6 Done Done Sent false
32251 9557 Acked Write 2 Done Done Sent false
32252 9558 Acked Flush 0 Done Done Sent false
32253 9559 Acked Read 8 Done Done Sent false
32254 9560 Acked Read 5 Done Done Sent false
32255 9561 Acked Read 1 Done Done Sent false
32256 9562 Acked Flush 0 Done Done Sent false
32257 9563 Acked Write 4 Done Done Sent false
32258 9564 Acked Write 4 Done Done Sent false
32259 9565 Acked Read 5 Done Done Sent false
32260 9566 Acked Write 1 Done Done Sent false
32261 9567 Acked Write 8 Done Done Sent false
32262 9568 Acked Write 2 Done Done Sent false
32263 9569 Acked Read 9 Done Done Sent false
32264 9570 Acked Write 10 Done Done Sent false
32265 9571 Acked Flush 0 Done Done Sent false
32266 9572 Acked Read 7 Done Done Sent false
32267 9573 Acked Write 8 Done Done Sent false
32268 9574 Acked Read 3 Done Done Sent false
32269 9575 Acked Read 10 Done Done Sent false
32270 9576 Acked Read 4 Done Done Sent false
32271 9577 Acked Write 7 Done Done Sent false
32272 9578 Acked Flush 0 Done Done Sent false
32273 9579 Acked Read 8 Done Done Sent false
32274 9580 Acked Write 5 Done Done Sent false
32275 9581 Acked Write 4 Done Done Sent false
32276 9582 Acked Flush 0 Done Done Sent false
32277 9583 Acked Flush 0 Done Done Sent false
32278 9584 Acked Flush 0 Done Done Sent false
32279 9585 Acked Flush 0 Done Done Sent false
32280 9586 Acked Read 4 Done Done Sent false
32281 9587 Acked Read 8 Done Done Sent false
32282 9588 Acked Flush 0 Done Done Sent false
32283 9589 Acked Read 10 Done Done Sent false
32284 9590 Acked Read 1 Done Done Sent false
32285 9591 Acked Write 2 Done Done Sent false
32286 9592 Acked Write 7 Done Done Sent false
32287 9593 Acked Flush 0 Done Done Sent false
32288 9594 Acked Read 3 Done Done Sent false
32289 9595 Acked Write 8 Done Done Sent false
32290 9596 Acked Read 7 Done Done Sent false
32291 9597 Acked Write 9 Done Done Sent false
32292 9598 Acked Write 3 Done Done Sent false
32293 9599 Acked Flush 0 Done Done Sent false
32294 9600 Acked Write 2 Done Done Sent false
32295 9601 Acked Write 6 Done Done Sent false
32296 9602 Acked Read 1 Done Done Sent false
32297 9603 Acked Write 4 Done Done Sent false
32298 9604 Acked Read 9 Done Done Sent false
32299 9605 Acked Read 1 Done Done Sent false
32300 9606 Acked Read 3 Done Done Sent false
32301 9607 Acked Read 4 Done Done Sent false
32302 9608 Acked Read 8 Done Done Sent false
32303 9609 Acked Write 5 Done Done Sent false
32304 9610 Acked Write 1 Done Done Sent false
32305 9611 Acked Flush 0 Done Done Sent false
32306 9612 Acked Read 5 Done Done Sent false
32307 9613 Acked Read 1 Done Done Sent false
32308 9614 Acked Write 9 Done Done Sent false
32309 9615 Acked Read 3 Done Done Sent false
32310 9616 Acked Write 4 Done Done Sent false
32311 9617 Acked Write 1 Done Done Sent false
32312 9618 Acked Read 4 Done Done Sent false
32313 9619 Acked Write 2 Done Done Sent false
32314 9620 Acked Write 10 Done Done Sent false
32315 9621 Acked Read 7 Done Done Sent false
32316 9622 Acked Flush 0 Done Done Sent false
32317 9623 Acked Read 5 Done Done Sent false
32318 9624 Acked Read 10 Done Done Sent false
32319 9625 Acked Read 10 Done Done Sent false
32320 9626 Acked Read 1 Done Done Sent false
32321 9627 Acked Read 10 Done Done Sent false
32322 9628 Acked Write 9 Done Done Sent false
32323 9629 Acked Write 3 Done Done Sent false
32324 9630 Acked Flush 0 Done Done Sent false
32325 9631 Acked Write 10 Done Done Sent false
32326 9632 Acked Write 4 Done Done Sent false
32327 9633 Acked Write 8 Done Done Sent false
32328 9634 Acked Read 4 Done Done Sent false
32329 9635 Acked Read 8 Done Done Sent false
32330 9636 Acked Write 7 Done Done Sent false
32331 9637 Acked Read 7 Done Done Sent false
32332 9638 Acked Read 1 Done Done Sent false
32333 9639 Acked Write 2 Done Done Sent false
32334 9640 Acked Read 7 Done Done Sent false
32335 9641 Acked Write 6 Done Done Sent false
32336 9642 Acked Read 1 Done Done Sent false
32337 9643 Acked Read 8 Done Done Sent false
32338 9644 Acked Write 2 Done Done Sent false
32339 9645 Acked Read 5 Done Done Sent false
32340 9646 Acked Write 7 Done Done Sent false
32341 9647 Acked Write 6 Done Done Sent false
32342 9648 Acked Write 4 Done Done Sent false
32343 9649 Acked Read 7 Done Done Sent false
32344 9650 Acked Read 7 Done Done Sent false
32345 9651 Acked Write 9 Done Done Sent false
32346 9652 Acked Flush 0 Done Done Sent false
32347 9653 Acked Write 8 Done Done Sent false
32348 9654 Acked Write 5 Done Done Sent false
32349 9655 Acked Flush 0 Done Done Sent false
32350 9656 Acked Write 2 Done Done Sent false
32351 9657 Acked Read 10 Done Done Sent false
32352 9658 Acked Read 1 Done Done Sent false
32353 9659 Acked Read 9 Done Done Sent false
32354 9660 Acked Read 10 Done Done Sent false
32355 9661 Acked Flush 0 Done Done Sent false
32356 9662 Acked Read 8 Done Done Sent false
32357 9663 Acked Write 6 Done Done Sent false
32358 9664 Acked Read 8 Done Done Sent false
32359 9665 Acked Read 9 Done Done Sent false
32360 9666 Acked Write 4 Done Done Sent false
32361 9667 Acked Write 5 Done Done Sent false
32362 9668 Acked Write 3 Done Done Sent false
32363 9669 Acked Read 4 Done Done Sent false
32364 9670 Acked Read 8 Done Done Sent false
32365 9671 Acked Write 3 Done Done Sent false
32366 9672 Acked Flush 0 Done Done Sent false
32367 9673 Acked Read 10 Done Done Sent false
32368 9674 Acked Read 1 Done Done Sent false
32369 9675 Acked Read 10 Done Done Sent false
32370 9676 Acked Write 8 Done Done Sent false
32371 9677 Acked Write 6 Done Done Sent false
32372 9678 Acked Write 4 Done Done Sent false
32373 9679 Acked Read 4 Done Done Sent false
32374 9680 Acked Flush 0 Done Done Sent false
32375 9681 Acked Write 1 Done Done Sent false
32376 9682 Acked Read 7 Done Done Sent false
32377 9683 Acked Read 10 Done Done Sent false
32378 9684 Acked Flush 0 Done Done Sent false
32379 9685 Acked Read 10 Done Done Sent false
32380 9686 Acked Read 6 Done Done Sent false
32381 9687 Acked Write 3 Done Done Sent false
32382 9688 Acked Read 7 Done Done Sent false
32383 9689 Acked Read 1 Done Done Sent false
32384 9690 Acked Read 2 Done Done Sent false
32385 9691 Acked Write 9 Done Done Sent false
32386 9692 Acked Read 7 Done Done Sent false
32387 9693 Acked Read 8 Done Done Sent false
32388 9694 Acked Write 10 Done Done Sent false
32389 9695 Acked Read 9 Done Done Sent false
32390 9696 Acked Write 3 Done Done Sent false
32391 9697 Acked Write 7 Done Done Sent false
32392 9698 Acked Write 6 Done Done Sent false
32393 9699 Acked Write 9 Done Done Sent false
32394 9700 Acked Flush 0 Done Done Sent false
32395 9701 Acked Write 10 Done Done Sent false
32396 9702 Acked Read 10 Done Done Sent false
32397 9703 Acked Read 6 Done Done Sent false
32398 9704 Acked Read 1 Done Done Sent false
32399 9705 Acked Write 9 Done Done Sent false
32400 9706 Acked Read 9 Done Done Sent false
32401 9707 Acked Read 2 Done Done Sent false
32402 9708 Acked Write 6 Done Done Sent false
32403 9709 Acked Flush 0 Done Done Sent false
32404 9710 Acked Read 6 Done Done Sent false
32405 9711 Acked Read 9 Done Done Sent false
32406 9712 Acked Write 4 Done Done Sent false
32407 9713 Acked Read 5 Done Done Sent false
32408 9714 Acked Read 10 Done Done Sent false
32409 9715 Acked Write 5 Done Done Sent false
32410 9716 Acked Write 4 Done Done Sent false
32411 9717 Acked Write 5 Done Done Sent false
32412 9718 Acked Flush 0 Done Done Sent false
32413 9719 Acked Read 4 Done Done Sent false
32414 9720 Acked Read 1 Done Done Sent false
32415 9721 Acked Read 5 Done Done Sent false
32416 9722 Acked Read 10 Done Done Sent false
32417 9723 Acked Write 5 Done Done Sent false
32418 9724 Acked Flush 0 Done Done Sent false
32419 9725 Acked Read 4 Done Done Sent false
32420 9726 Acked Read 2 Done Done Sent false
32421 9727 Acked Read 5 Done Done Sent false
32422 9728 Acked Write 5 Done Done Sent false
32423 9729 Acked Write 2 Done Done Sent false
32424 9730 Acked Read 4 Done Done Sent false
32425 9731 Acked Write 9 Done Done Sent false
32426 9732 Acked Flush 0 Done Done Sent false
32427 9733 Acked Write 7 Done Done Sent false
32428 9734 Acked Flush 0 Done Done Sent false
32429 9735 Acked Read 7 Done Done Sent false
32430 9736 Acked Flush 0 Done Done Sent false
32431 9737 Acked Read 10 Done Done Sent false
32432 9738 Acked Read 7 Done Done Sent false
32433 9739 Acked Read 10 Done Done Sent false
32434 9740 Acked Read 4 Done Done Sent false
32435 9741 Acked Flush 0 Done Done Sent false
32436 9742 Acked Flush 0 Done Done Sent false
32437 9743 Acked Write 4 Done Done Sent false
32438 9744 Acked Write 3 Done Done Sent false
32439 9745 Acked Read 6 Done Done Sent false
32440 9746 Acked Write 9 Done Done Sent false
32441 9747 Acked Read 4 Done Done Sent false
32442 9748 Acked Read 5 Done Done Sent false
32443 9749 Acked Flush 0 Done Done Sent false
32444 9750 Acked Flush 0 Done Done Sent false
32445 9751 Acked Write 8 Done Done Sent false
32446 9752 Acked Read 7 Done Done Sent false
32447 9753 Acked Write 4 Done Done Sent false
32448 9754 Acked Read 3 Done Done Sent false
32449 STATES DS:0 DS:1 DS:2 TOTAL
32450 Sent 0 0 301 301
32451 Done 301 301 0 602
32452 Skipped 0 0 0 0
32453 Error 0 0 0 0
32454 Last Flush: Some(JobId(9750)) Some(JobId(9750)) Some(JobId(9453))
32455 Downstairs last five completed: 9453 9452 9451 9450 9449
32456 Upstairs last five completed: 9754 9753 9752 9751 9750
32457 CLIENT: Up:0 ds:301 act:2
324582025-01-28T22:01:25.963ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
324592025-01-28T22:01:26.841ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
324602025-01-28T22:01:26.841ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46817) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
324612025-01-28T22:01:26.842ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
324622025-01-28T22:01:26.843ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
324632025-01-28T22:01:26.843ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
324642025-01-28T22:01:26.843ZINFOcrucible: send last flush ID to this DS: Some(JobId(9453)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
324652025-01-28T22:01:26.843ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
324662025-01-28T22:01:26.843ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9453)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
324672025-01-28T22:01:26.843ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9453)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
32468 ----------------------------------------------------------------
32469 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
32470 Downstairs last five completed: 9755 9754 9753 9752 9751
32471 Upstairs last five completed: 9755 9754 9753 9752 9751
32472 CLIENT: Up:0 ds:0 act:3
32473 Replay: All jobs finished
32474 Main task has work to do, go find it
32475 got dsc Stop(2)
32476 stop 2
32477 [8830] Got stop action so:false kr:false
32478 Jan 28 22:01:29.091 INFO request completed, latency_us: 372, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 30451ad4-7398-4138-bb09-24f0148f62cf, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
324792025-01-28T22:01:29.092ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
324802025-01-28T22:01:29.092ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
324812025-01-28T22:01:29.092ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
324822025-01-28T22:01:29.092ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
32483 Jan 28 22:01:29.092 INFO request completed, latency_us: 154, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4513d802-f14e-4e9b-8001-793788acd096, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
32484 [8830] Exited with: ExitStatus(unix_wait_status(9))
32485 [8830][2] reports Exit
32486 Jan 28 22:01:33.094 INFO request completed, latency_us: 216, response_code: 200, uri: /state/cid/2, method: GET, req_id: 14300ed9-966c-45a7-b1ed-b6153bd1f8fa, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
32487 001/300 Write block 3811 len 28672 data: 4 4 4 3 2 4 4
32488 002/300 Read block 4656 len 16384
32489 003/300 Read block 1167 len 20480
32490 004/300 Read block 4717 len 24576
32491 005/300 Read block 3148 len 24576
32492 006/300 Write block 4732 len 12288 data: 6 5 4
32493 007/300 Write block 3002 len 36864 data: 6 5 5 5 5 4 4 2 3
32494 008/300 Write block 3952 len 32768 data: 5 4 6 6 7 8 7 7
32495 009/300 Read block 3868 len 4096
32496 010/300 Read block 1206 len 28672
32497 011/300 Write block 4808 len 36864 data: 6 6 6 4 5 7 6 5 5
32498 012/300 Read block 29 len 4096
32499 013/300 Read block 2505 len 16384
32500 014/300 Read block 3427 len 40960
32501 015/300 Write block 1693 len 36864 data: 4 4 4 3 5 8 8 9 8
32502 016/300 Read block 4603 len 16384
32503 017/300 Write block 3625 len 4096 data: 7
32504 018/300 Read block 4234 len 40960
32505 019/300 Write block 4260 len 36864 data: 6 4 4 5 5 6 7 6 8
32506 020/300 Write block 2152 len 40960 data: 9 10 11 11 10 10 8 9 8 6
32507 021/300 Write block 4274 len 36864 data: 7 8 6 6 5 5 4 4 3
32508 022/300 Read block 1273 len 36864
32509 023/300 Write block 4970 len 24576 data: 8 7 6 6 5 5
32510 024/300 Flush
32511 025/300 Write block 69 len 16384 data: 6 6 5 7
32512 026/300 Write block 2400 len 16384 data: 8 6 7 8
32513 027/300 Read block 4022 len 24576
32514 028/300 Read block 2136 len 8192
32515 029/300 Read block 2338 len 24576
32516 030/300 Flush
32517 031/300 Flush
32518 032/300 Read block 3214 len 40960
32519 033/300 Write block 3753 len 24576 data: 6 7 7 9 9 9
32520 034/300 Read block 4650 len 40960
32521 035/300 Read block 479 len 12288
32522 036/300 Write block 208 len 40960 data: 9 8 8 8 7 6 5 6 6 6
32523 037/300 Read block 4251 len 16384
32524 038/300 Read block 2819 len 8192
32525 039/300 Write block 273 len 20480 data: 2 2 3 3 3
32526 040/300 Flush
32527 041/300 Write block 4121 len 40960 data: 5 4 4 5 5 5 6 5 3 3
32528 042/300 Flush
32529 043/300 Read block 1989 len 24576
32530 044/300 Write block 488 len 28672 data: 5 5 4 7 6 5 6
32531 045/300 Write block 4008 len 12288 data: 8 8 8
32532 046/300 Read block 3845 len 4096
32533 047/300 Read block 3399 len 24576
32534 048/300 Read block 1048 len 28672
32535 049/300 Read block 891 len 32768
32536 050/300 Read block 4467 len 32768
32537 051/300 Read block 4484 len 20480
32538 052/300 Write block 438 len 8192 data: 9 10
32539 053/300 Read block 4232 len 28672
32540 054/300 Write block 602 len 28672 data: 7 8 8 5 7 8 8
32541 055/300 Read block 255 len 16384
32542 056/300 Read block 938 len 28672
32543 057/300 Read block 274 len 24576
32544 058/300 Read block 4215 len 24576
32545 059/300 Write block 375 len 20480 data: 8 8 7 5 5
32546 060/300 Write block 240 len 36864 data: 7 8 9 9 8 9 9 8 7
32547 061/300 Write block 2087 len 28672 data: 6 6 6 7 8 6 7
32548 062/300 Write block 3023 len 32768 data: 7 9 10 7 5 6 5 5
32549 063/300 Read block 3952 len 20480
32550 064/300 Read block 2354 len 16384
32551 065/300 Flush
32552 066/300 Read block 3761 len 16384
32553 067/300 Write block 4412 len 36864 data: 6 6 5 3 2 3 3 3 3
32554 068/300 Read block 2319 len 12288
32555 069/300 Read block 717 len 32768
32556 070/300 Read block 243 len 24576
32557 071/300 Read block 2274 len 8192
32558 072/300 Read block 240 len 20480
32559 073/300 Write block 3666 len 12288 data: 5 7 10
32560 074/300 Write block 2026 len 4096 data: 5
32561 075/300 Read block 1282 len 16384
32562 076/300 Write block 3429 len 32768 data: 4 5 6 6 6 6 5 5
32563 077/300 Write block 2842 len 12288 data: 7 7 8
32564 078/300 Write block 4083 len 40960 data: 12 12 11 13 12 11 11 12 12 12
32565 079/300 Read block 2760 len 28672
32566 080/300 Flush
32567 081/300 Flush
32568 082/300 Flush
32569 083/300 Flush
32570 084/300 Write block 1581 len 24576 data: 6 5 4 4 4 5
32571 085/300 Write block 748 len 40960 data: 3 2 2 4 4 5 6 6 6 5
32572 086/300 Read block 2622 len 16384
32573 087/300 Read block 4742 len 28672
32574 088/300 Write block 2422 len 32768 data: 3 5 5 6 6 6 7 9
32575 089/300 Read block 3083 len 24576
32576 090/300 Flush
32577 091/300 Read block 1360 len 36864
32578 092/300 Read block 2137 len 28672
32579 093/300 Read block 3252 len 8192
32580 094/300 Write block 2180 len 8192 data: 2 2
32581 095/300 Write block 431 len 24576 data: 3 5 5 6 10 10
32582 096/300 Read block 1889 len 32768
32583 097/300 Read block 3456 len 40960
32584 098/300 Write block 3897 len 40960 data: 6 6 9 9 9 7 6 5 7 8
32585 099/300 Read block 3615 len 40960
32586 100/300 Read block 757 len 36864
32587 101/300 Flush
32588 102/300 Write block 4793 len 32768 data: 5 6 6 5 5 7 8 7
32589 103/300 Flush
32590 104/300 Write block 3911 len 8192 data: 7 7
32591 105/300 Read block 73 len 4096
32592 106/300 Write block 2321 len 32768 data: 6 3 3 3 5 4 4 5
32593 107/300 Write block 4307 len 12288 data: 6 6 5
32594 108/300 Write block 4119 len 4096 data: 7
32595 109/300 Read block 4710 len 40960
32596 110/300 Read block 44 len 24576
32597 111/300 Write block 4783 len 8192 data: 3 3
32598 112/300 Write block 851 len 12288 data: 4 4 5
32599 113/300 Read block 1698 len 4096
32600 114/300 Read block 1742 len 32768
32601 115/300 Read block 287 len 24576
32602 116/300 Write block 545 len 28672 data: 7 8 11 10 9 8 7
32603 117/300 Write block 3441 len 40960 data: 4 4 5 5 5 4 6 5 6 6
32604 118/300 Read block 452 len 32768
32605 119/300 Write block 4031 len 8192 data: 7 7
32606 120/300 Read block 3079 len 24576
32607 121/300 Write block 205 len 20480 data: 9 9 8 10 9
32608 122/300 Write block 144 len 12288 data: 6 5 3
32609 123/300 Write block 4481 len 20480 data: 12 12 12 12 11
32610 124/300 Write block 4830 len 24576 data: 6 5 5 5 6 7
32611 125/300 Read block 2942 len 12288
32612 126/300 Write block 4311 len 4096 data: 6
32613 127/300 Write block 2328 len 40960 data: 6 5 6 6 6 7 6 6 6 8
32614 128/300 Flush
32615 129/300 Read block 4884 len 40960
32616 130/300 Write block 1599 len 12288 data: 5 6 6
32617 131/300 Read block 1818 len 4096
32618 132/300 Write block 4826 len 12288 data: 5 5 6
32619 133/300 Write block 1172 len 8192 data: 5 6
32620 134/300 Read block 2950 len 20480
32621 135/300 Read block 547 len 36864
32622 136/300 Read block 3553 len 32768
32623 137/300 Write block 3754 len 32768 data: 8 8 10 10 10 9 11 7
32624 138/300 Read block 1755 len 28672
32625 139/300 Read block 2429 len 32768
32626 140/300 Read block 3578 len 12288
32627 141/300 Read block 1188 len 20480
32628 142/300 Read block 776 len 20480
32629 143/300 Read block 2258 len 32768
32630 144/300 Read block 1131 len 28672
32631 145/300 Read block 806 len 20480
32632 146/300 Write block 2589 len 32768 data: 2 2 3 3 4 5 7 5
32633 147/300 Write block 4781 len 24576 data: 5 4 4 4 3 2
32634 148/300 Write block 741 len 32768 data: 6 5 5 4 5 4 3 4
32635 149/300 Write block 4616 len 24576 data: 7 8 9 9 9 9
32636 150/300 Read block 2636 len 28672
32637 151/300 Read block 2469 len 24576
32638 152/300 Read block 2141 len 8192
32639 153/300 Read block 2866 len 12288
32640 154/300 Flush
32641 155/300 Read block 2379 len 8192
32642 156/300 Write block 1866 len 24576 data: 3 4 3 5 6 6
32643 157/300 Read block 276 len 12288
32644 158/300 Flush
32645 159/300 Write block 5 len 12288 data: 4 7 8
32646 160/300 Write block 3761 len 4096 data: 8
32647 161/300 Write block 3275 len 32768 data: 7 7 6 4 4 3 4 4
32648 162/300 Write block 867 len 32768 data: 4 6 8 8 7 6 8 7
32649 163/300 Flush
32650 164/300 Read block 0 len 16384
32651 165/300 Read block 964 len 12288
32652 166/300 Read block 1168 len 24576
32653 167/300 Write block 2602 len 4096 data: 5
32654 168/300 Write block 827 len 4096 data: 5
32655 169/300 Read block 4095 len 40960
32656 170/300 Write block 25 len 28672 data: 8 7 6 4 5 5 5
32657 171/300 Write block 2864 len 16384 data: 4 4 4 4
32658 172/300 Write block 1144 len 40960 data: 10 8 7 7 6 7 7 7 5 5
32659 173/300 Read block 261 len 40960
32660 174/300 Read block 709 len 36864
32661 175/300 Read block 4231 len 24576
32662 176/300 Read block 825 len 20480
32663 177/300 Write block 2406 len 24576 data: 6 6 7 9 7 8
32664 178/300 Flush
32665 179/300 Write block 4469 len 4096 data: 4
32666 180/300 Write block 1829 len 24576 data: 9 7 6 7 6 5
32667 181/300 Read block 4365 len 32768
32668 182/300 Read block 2982 len 40960
32669 183/300 Read block 4393 len 12288
32670 184/300 Write block 4438 len 8192 data: 5 4
32671 185/300 Read block 294 len 40960
32672 186/300 Read block 4388 len 8192
32673 187/300 Read block 1856 len 24576
32674 188/300 Flush
32675 189/300 Read block 4747 len 8192
32676 190/300 Read block 4925 len 12288
32677 191/300 Read block 339 len 24576
32678 192/300 Read block 363 len 36864
32679 193/300 Read block 2928 len 12288
32680 194/300 Write block 787 len 12288 data: 4 5 5
32681 195/300 Write block 2519 len 20480 data: 6 6 6 7 7
32682 196/300 Write block 1003 len 8192 data: 7 6
32683 197/300 Read block 2067 len 40960
32684 198/300 Read block 1879 len 32768
32685 199/300 Write block 3116 len 12288 data: 6 8 7
32686 200/300 Read block 4922 len 16384
32687 201/300 Write block 2652 len 32768 data: 3 2 2 3 4 4 6 8
32688 202/300 Write block 3346 len 20480 data: 4 3 3 5 4
32689 203/300 Read block 3248 len 8192
32690 204/300 Write block 2836 len 4096 data: 2
32691 205/300 Write block 4570 len 20480 data: 8 8 4 3 4
32692 206/300 Read block 4062 len 4096
32693 207/300 Write block 2531 len 16384 data: 5 6 6 4
32694 208/300 Flush
32695 209/300 Read block 3318 len 16384
32696 210/300 Read block 2058 len 4096
32697 211/300 Read block 1399 len 36864
32698 212/300 Write block 2215 len 24576 data: 5 3 3 5 5 6
32699 213/300 Write block 1861 len 12288 data: 6 5 5
32700 214/300 Read block 4155 len 16384
32701 215/300 Write block 2604 len 12288 data: 7 6 5
32702 216/300 Read block 3760 len 28672
32703 217/300 Read block 2032 len 8192
32704 218/300 Read block 2119 len 28672
32705 219/300 Write block 3710 len 24576 data: 4 4 4 6 8 7
32706 220/300 Write block 1778 len 20480 data: 6 5 5 5 6
32707 221/300 Read block 1147 len 20480
32708 222/300 Write block 1512 len 12288 data: 2 3 4
32709 223/300 Read block 4043 len 32768
32710 224/300 Read block 837 len 40960
32711 225/300 Write block 2265 len 4096 data: 4
32712 226/300 Write block 2613 len 32768 data: 5 6 8 8 9 9 7 7
32713 227/300 Read block 1218 len 40960
32714 228/300 Read block 1877 len 28672
32715 229/300 Read block 909 len 20480
32716 230/300 Flush
32717 231/300 Write block 557 len 12288 data: 5 5 3
32718 232/300 Read block 4845 len 8192
32719 233/300 Read block 4833 len 16384
32720 234/300 Flush
32721 235/300 Read block 2390 len 4096
32722 236/300 Read block 4044 len 16384
32723 237/300 Write block 3423 len 20480 data: 7 7 6 5 4
32724 238/300 Read block 487 len 40960
32725 239/300 Flush
32726 240/300 Write block 3724 len 28672 data: 7 9 9 11 12 9 10
32727 241/300 Write block 4500 len 20480 data: 3 4 4 5 6
32728 242/300 Flush
32729 243/300 Write block 3154 len 12288 data: 9 8 8
32730 244/300 Write block 2557 len 20480 data: 5 5 5 5 7
32731 245/300 Read block 1148 len 28672
32732 246/300 Write block 56 len 8192 data: 5 5
32733 247/300 Read block 1480 len 8192
32734 248/300 Read block 771 len 8192
32735 249/300 Write block 4898 len 36864 data: 4 4 3 3 2 2 4 5 4
32736 250/300 Read block 4717 len 36864
32737 251/300 Read block 4941 len 8192
32738 252/300 Write block 2701 len 12288 data: 4 5 5
32739 253/300 Write block 2095 len 28672 data: 8 7 7 8 9 6 6
32740 254/300 Write block 2728 len 16384 data: 6 7 9 8
32741 255/300 Write block 3801 len 12288 data: 4 4 3
32742 256/300 Flush
32743 257/300 Read block 4932 len 12288
32744 258/300 Write block 1551 len 12288 data: 6 7 5
32745 259/300 Write block 864 len 4096 data: 4
32746 260/300 Read block 2452 len 16384
32747 261/300 Read block 4607 len 24576
32748 262/300 Write block 4178 len 28672 data: 9 10 9 8 7 8 6
32749 263/300 Write block 1794 len 36864 data: 8 7 8 9 9 8 6 5 4
32750 264/300 Write block 1640 len 12288 data: 6 8 8
32751 265/300 Write block 930 len 28672 data: 4 4 3 4 5 6 8
32752 266/300 Write block 892 len 28672 data: 10 9 6 5 4 3 4
32753 267/300 Read block 2971 len 32768
32754 268/300 Flush
32755 269/300 Read block 4266 len 32768
32756 270/300 Write block 704 len 36864 data: 6 7 7 8 7 7 7 8 7
32757 271/300 Read block 937 len 40960
32758 272/300 Write block 1221 len 24576 data: 7 6 6 5 5 5
32759 273/300 Write block 1679 len 16384 data: 6 5 6 4
32760 274/300 Read block 1249 len 36864
32761 275/300 Write block 2460 len 24576 data: 6 6 5 4 5 7
32762 276/300 Read block 2877 len 28672
32763 277/300 Read block 2943 len 8192
32764 278/300 Write block 2218 len 36864 data: 6 6 7 6 5 5 6 5 6
32765 279/300 Write block 2270 len 20480 data: 6 6 5 6 7
32766 280/300 Write block 443 len 20480 data: 10 8 7 5 5
32767 281/300 Read block 3652 len 8192
32768 282/300 Flush
32769 283/300 Read block 2176 len 32768
32770 284/300 Write block 797 len 12288 data: 3 4 4
32771 285/300 Write block 2040 len 40960 data: 6 4 3 4 4 4 4 3 5 5
32772 286/300 Read block 706 len 32768
32773 287/300 Flush
32774 288/300 Flush
32775 289/300 Flush
32776 290/300 Flush
32777 291/300 Write block 16 len 12288 data: 5 4 3
32778 292/300 Read block 257 len 24576
32779 293/300 Read block 4797 len 40960
32780 294/300 Write block 335 len 24576 data: 4 5 5 3 3 5
32781 295/300 Read block 4703 len 20480
32782 296/300 Write block 1427 len 28672 data: 7 6 6 6 9 10 10
32783 297/300 Read block 2606 len 36864
32784 298/300 Write block 151 len 32768 data: 5 7 7 8 8 8 10 8
32785 299/300 Write block 866 len 16384 data: 3 5 7 9
32786 300/300 Read block 1455 len 12288
32787 301/300 Write block 4430 len 28672 data: 7 6 5 5 4 3 4
32788 Main task has work to do, go find it
32789 got dsc Start(2)
32790 start 2
32791 Jan 28 22:01:33.333 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/2, method: POST, req_id: 7c51a3f8-66bf-4020-a86c-d9e21af1742a, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
32792 Replay: started 2, returned:Ok(())
32793 ----------------------------------------------------------------
32794 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32795 States: Active Active Offline
32796 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32797 9756 Acked Write 7 Done Done Sent false
32798 9757 Acked Read 4 Done Done Sent false
32799 9758 Acked Read 5 Done Done Sent false
32800 9759 Acked Read 6 Done Done Sent false
32801 9760 Acked Read 6 Done Done Sent false
32802 9761 Acked Write 3 Done Done Sent false
32803 9762 Acked Write 9 Done Done Sent false
32804 9763 Acked Write 8 Done Done Sent false
32805 9764 Acked Read 1 Done Done Sent false
32806 9765 Acked Read 7 Done Done Sent false
32807 9766 Acked Write 9 Done Done Sent false
32808 9767 Acked Read 1 Done Done Sent false
32809 9768 Acked Read 4 Done Done Sent false
32810 9769 Acked Read 10 Done Done Sent false
32811 9770 Acked Write 9 Done Done Sent false
32812 9771 Acked Read 4 Done Done Sent false
32813 9772 Acked Write 1 Done Done Sent false
32814 9773 Acked Read 10 Done Done Sent false
32815 9774 Acked Write 9 Done Done Sent false
32816 9775 Acked Write 10 Done Done Sent false
32817 9776 Acked Write 9 Done Done Sent false
32818 9777 Acked Read 9 Done Done Sent false
32819 9778 Acked Write 6 Done Done Sent false
32820 9779 Acked Flush 0 Done Done Sent false
32821 9780 Acked Write 4 Done Done Sent false
32822 9781 Acked Write 4 Done Done Sent false
32823 9782 Acked Read 6 Done Done Sent false
32824 9783 Acked Read 2 Done Done Sent false
32825 9784 Acked Read 6 Done Done Sent false
32826 9785 Acked Flush 0 Done Done Sent false
32827 9786 Acked Flush 0 Done Done Sent false
32828 9787 Acked Read 10 Done Done Sent false
32829 9788 Acked Write 6 Done Done Sent false
32830 9789 Acked Read 10 Done Done Sent false
32831 9790 Acked Read 3 Done Done Sent false
32832 9791 Acked Write 10 Done Done Sent false
32833 9792 Acked Read 4 Done Done Sent false
32834 9793 Acked Read 2 Done Done Sent false
32835 9794 Acked Write 5 Done Done Sent false
32836 9795 Acked Flush 0 Done Done Sent false
32837 9796 Acked Write 10 Done Done Sent false
32838 9797 Acked Flush 0 Done Done Sent false
32839 9798 Acked Read 6 Done Done Sent false
32840 9799 Acked Write 7 Done Done Sent false
32841 9800 Acked Write 3 Done Done Sent false
32842 9801 Acked Read 1 Done Done Sent false
32843 9802 Acked Read 6 Done Done Sent false
32844 9803 Acked Read 7 Done Done Sent false
32845 9804 Acked Read 8 Done Done Sent false
32846 9805 Acked Read 8 Done Done Sent false
32847 9806 Acked Read 5 Done Done Sent false
32848 9807 Acked Write 2 Done Done Sent false
32849 9808 Acked Read 7 Done Done Sent false
32850 9809 Acked Write 7 Done Done Sent false
32851 9810 Acked Read 4 Done Done Sent false
32852 9811 Acked Read 7 Done Done Sent false
32853 9812 Acked Read 6 Done Done Sent false
32854 9813 Acked Read 6 Done Done Sent false
32855 9814 Acked Write 5 Done Done Sent false
32856 9815 Acked Write 9 Done Done Sent false
32857 9816 Acked Write 7 Done Done Sent false
32858 9817 Acked Write 8 Done Done Sent false
32859 9818 Acked Read 5 Done Done Sent false
32860 9819 Acked Read 4 Done Done Sent false
32861 9820 Acked Flush 0 Done Done Sent false
32862 9821 Acked Read 4 Done Done Sent false
32863 9822 Acked Write 9 Done Done Sent false
32864 9823 Acked Read 3 Done Done Sent false
32865 9824 Acked Read 8 Done Done Sent false
32866 9825 Acked Read 6 Done Done Sent false
32867 9826 Acked Read 2 Done Done Sent false
32868 9827 Acked Read 5 Done Done Sent false
32869 9828 Acked Write 3 Done Done Sent false
32870 9829 Acked Write 1 Done Done Sent false
32871 9830 Acked Read 4 Done Done Sent false
32872 9831 Acked Write 8 Done Done Sent false
32873 9832 Acked Write 3 Done Done Sent false
32874 9833 Acked Write 10 Done Done Sent false
32875 9834 Acked Read 7 Done Done Sent false
32876 9835 Acked Flush 0 Done Done Sent false
32877 9836 Acked Flush 0 Done Done Sent false
32878 9837 Acked Flush 0 Done Done Sent false
32879 9838 Acked Flush 0 Done Done Sent false
32880 9839 Acked Write 6 Done Done Sent false
32881 9840 Acked Write 10 Done Done Sent false
32882 9841 Acked Read 4 Done Done Sent false
32883 9842 Acked Read 7 Done Done Sent false
32884 9843 Acked Write 8 Done Done Sent false
32885 9844 Acked Read 6 Done Done Sent false
32886 9845 Acked Flush 0 Done Done Sent false
32887 9846 Acked Read 9 Done Done Sent false
32888 9847 Acked Read 7 Done Done Sent false
32889 9848 Acked Read 2 Done Done Sent false
32890 9849 Acked Write 2 Done Done Sent false
32891 9850 Acked Write 6 Done Done Sent false
32892 9851 Acked Read 8 Done Done Sent false
32893 9852 Acked Read 10 Done Done Sent false
32894 9853 Acked Write 10 Done Done Sent false
32895 9854 Acked Read 10 Done Done Sent false
32896 9855 Acked Read 9 Done Done Sent false
32897 9856 Acked Flush 0 Done Done Sent false
32898 9857 Acked Write 8 Done Done Sent false
32899 9858 Acked Flush 0 Done Done Sent false
32900 9859 Acked Write 2 Done Done Sent false
32901 9860 Acked Read 1 Done Done Sent false
32902 9861 Acked Write 8 Done Done Sent false
32903 9862 Acked Write 3 Done Done Sent false
32904 9863 Acked Write 1 Done Done Sent false
32905 9864 Acked Read 10 Done Done Sent false
32906 9865 Acked Read 6 Done Done Sent false
32907 9866 Acked Write 2 Done Done Sent false
32908 9867 Acked Write 3 Done Done Sent false
32909 9868 Acked Read 1 Done Done Sent false
32910 9869 Acked Read 8 Done Done Sent false
32911 9870 Acked Read 6 Done Done Sent false
32912 9871 Acked Write 7 Done Done Sent false
32913 9872 Acked Write 10 Done Done Sent false
32914 9873 Acked Read 8 Done Done Sent false
32915 9874 Acked Write 2 Done Done Sent false
32916 9875 Acked Read 6 Done Done Sent false
32917 9876 Acked Write 5 Done Done Sent false
32918 9877 Acked Write 3 Done Done Sent false
32919 9878 Acked Write 5 Done Done Sent false
32920 9879 Acked Write 6 Done Done Sent false
32921 9880 Acked Read 3 Done Done Sent false
32922 9881 Acked Write 1 Done Done Sent false
32923 9882 Acked Write 10 Done Done Sent false
32924 9883 Acked Flush 0 Done Done Sent false
32925 9884 Acked Read 10 Done Done Sent false
32926 9885 Acked Write 3 Done Done Sent false
32927 9886 Acked Read 1 Done Done Sent false
32928 9887 Acked Write 3 Done Done Sent false
32929 9888 Acked Write 2 Done Done Sent false
32930 9889 Acked Read 5 Done Done Sent false
32931 9890 Acked Read 9 Done Done Sent false
32932 9891 Acked Read 8 Done Done Sent false
32933 9892 Acked Write 8 Done Done Sent false
32934 9893 Acked Read 7 Done Done Sent false
32935 9894 Acked Read 8 Done Done Sent false
32936 9895 Acked Read 3 Done Done Sent false
32937 9896 Acked Read 5 Done Done Sent false
32938 9897 Acked Read 5 Done Done Sent false
32939 9898 Acked Read 8 Done Done Sent false
32940 9899 Acked Read 7 Done Done Sent false
32941 9900 Acked Read 5 Done Done Sent false
32942 9901 Acked Write 8 Done Done Sent false
32943 9902 Acked Write 6 Done Done Sent false
32944 9903 Acked Write 8 Done Done Sent false
32945 9904 Acked Write 6 Done Done Sent false
32946 9905 Acked Read 7 Done Done Sent false
32947 9906 Acked Read 6 Done Done Sent false
32948 9907 Acked Read 2 Done Done Sent false
32949 9908 Acked Read 3 Done Done Sent false
32950 9909 Acked Flush 0 Done Done Sent false
32951 9910 Acked Read 2 Done Done Sent false
32952 9911 Acked Write 6 Done Done Sent false
32953 9912 Acked Read 3 Done Done Sent false
32954 9913 Acked Flush 0 Done Done Sent false
32955 9914 Acked Write 3 Done Done Sent false
32956 9915 Acked Write 1 Done Done Sent false
32957 9916 Acked Write 8 Done Done Sent false
32958 9917 Acked Write 8 Done Done Sent false
32959 9918 Acked Flush 0 Done Done Sent false
32960 9919 Acked Read 4 Done Done Sent false
32961 9920 Acked Read 3 Done Done Sent false
32962 9921 Acked Read 6 Done Done Sent false
32963 9922 Acked Write 1 Done Done Sent false
32964 9923 Acked Write 1 Done Done Sent false
32965 9924 Acked Read 10 Done Done Sent false
32966 9925 Acked Write 7 Done Done Sent false
32967 9926 Acked Write 4 Done Done Sent false
32968 9927 Acked Write 10 Done Done Sent false
32969 9928 Acked Read 10 Done Done Sent false
32970 9929 Acked Read 9 Done Done Sent false
32971 9930 Acked Read 6 Done Done Sent false
32972 9931 Acked Read 5 Done Done Sent false
32973 9932 Acked Write 6 Done Done Sent false
32974 9933 Acked Flush 0 Done Done Sent false
32975 9934 Acked Write 1 Done Done Sent false
32976 9935 Acked Write 6 Done Done Sent false
32977 9936 Acked Read 8 Done Done Sent false
32978 9937 Acked Read 10 Done Done Sent false
32979 9938 Acked Read 3 Done Done Sent false
32980 9939 Acked Write 2 Done Done Sent false
32981 9940 Acked Read 10 Done Done Sent false
32982 9941 Acked Read 2 Done Done Sent false
32983 9942 Acked Read 6 Done Done Sent false
32984 9943 Acked Flush 0 Done Done Sent false
32985 9944 Acked Read 2 Done Done Sent false
32986 9945 Acked Read 3 Done Done Sent false
32987 9946 Acked Read 6 Done Done Sent false
32988 9947 Acked Read 9 Done Done Sent false
32989 9948 Acked Read 3 Done Done Sent false
32990 9949 Acked Write 3 Done Done Sent false
32991 9950 Acked Write 5 Done Done Sent false
32992 9951 Acked Write 2 Done Done Sent false
32993 9952 Acked Read 10 Done Done Sent false
32994 9953 Acked Read 8 Done Done Sent false
32995 9954 Acked Write 3 Done Done Sent false
32996 9955 Acked Read 4 Done Done Sent false
32997 9956 Acked Write 8 Done Done Sent false
32998 9957 Acked Write 5 Done Done Sent false
32999 9958 Acked Read 2 Done Done Sent false
33000 9959 Acked Write 1 Done Done Sent false
33001 9960 Acked Write 5 Done Done Sent false
33002 9961 Acked Read 1 Done Done Sent false
33003 9962 Acked Write 4 Done Done Sent false
33004 9963 Acked Flush 0 Done Done Sent false
33005 9964 Acked Read 4 Done Done Sent false
33006 9965 Acked Read 1 Done Done Sent false
33007 9966 Acked Read 9 Done Done Sent false
33008 9967 Acked Write 6 Done Done Sent false
33009 9968 Acked Write 3 Done Done Sent false
33010 9969 Acked Read 4 Done Done Sent false
33011 9970 Acked Write 3 Done Done Sent false
33012 9971 Acked Read 7 Done Done Sent false
33013 9972 Acked Read 2 Done Done Sent false
33014 9973 Acked Read 7 Done Done Sent false
33015 9974 Acked Write 6 Done Done Sent false
33016 9975 Acked Write 5 Done Done Sent false
33017 9976 Acked Read 5 Done Done Sent false
33018 9977 Acked Write 3 Done Done Sent false
33019 9978 Acked Read 8 Done Done Sent false
33020 9979 Acked Read 10 Done Done Sent false
33021 9980 Acked Write 1 Done Done Sent false
33022 9981 Acked Write 8 Done Done Sent false
33023 9982 Acked Read 10 Done Done Sent false
33024 9983 Acked Read 7 Done Done Sent false
33025 9984 Acked Read 5 Done Done Sent false
33026 9985 Acked Flush 0 Done Done Sent false
33027 9986 Acked Write 3 Done Done Sent false
33028 9987 Acked Read 2 Done Done Sent false
33029 9988 Acked Read 4 Done Done Sent false
33030 9989 Acked Flush 0 Done Done Sent false
33031 9990 Acked Read 1 Done Done Sent false
33032 9991 Acked Read 4 Done Done Sent false
33033 9992 Acked Write 5 Done Done Sent false
33034 9993 Acked Read 10 Done Done Sent false
33035 9994 Acked Flush 0 Done Done Sent false
33036 9995 Acked Write 7 Done Done Sent false
33037 9996 Acked Write 5 Done Done Sent false
33038 9997 Acked Flush 0 Done Done Sent false
33039 9998 Acked Write 3 Done Done Sent false
33040 9999 Acked Write 5 Done Done Sent false
33041 10000 Acked Read 7 Done Done Sent false
33042 10001 Acked Write 2 Done Done Sent false
33043 10002 Acked Read 2 Done Done Sent false
33044 10003 Acked Read 2 Done Done Sent false
33045 10004 Acked Write 9 Done Done Sent false
33046 10005 Acked Read 9 Done Done Sent false
33047 10006 Acked Read 2 Done Done Sent false
33048 10007 Acked Write 3 Done Done Sent false
33049 10008 Acked Write 7 Done Done Sent false
33050 10009 Acked Write 4 Done Done Sent false
33051 10010 Acked Write 3 Done Done Sent false
33052 10011 Acked Flush 0 Done Done Sent false
33053 10012 Acked Read 3 Done Done Sent false
33054 10013 Acked Write 3 Done Done Sent false
33055 10014 Acked Write 1 Done Done Sent false
33056 10015 Acked Read 4 Done Done Sent false
33057 10016 Acked Read 6 Done Done Sent false
33058 10017 Acked Write 7 Done Done Sent false
33059 10018 Acked Write 9 Done Done Sent false
33060 10019 Acked Write 3 Done Done Sent false
33061 10020 Acked Write 7 Done Done Sent false
33062 10021 Acked Write 7 Done Done Sent false
33063 10022 Acked Read 8 Done Done Sent false
33064 10023 Acked Flush 0 Done Done Sent false
33065 10024 Acked Read 8 Done Done Sent false
33066 10025 Acked Write 9 Done Done Sent false
33067 10026 Acked Read 10 Done Done Sent false
33068 10027 Acked Write 6 Done Done Sent false
33069 10028 Acked Write 4 Done Done Sent false
33070 10029 Acked Read 9 Done Done Sent false
33071 10030 Acked Write 6 Done Done Sent false
33072 10031 Acked Read 7 Done Done Sent false
33073 10032 Acked Read 2 Done Done Sent false
33074 10033 Acked Write 9 Done Done Sent false
33075 10034 Acked Write 5 Done Done Sent false
33076 10035 Acked Write 5 Done Done Sent false
33077 10036 Acked Read 2 Done Done Sent false
33078 10037 Acked Flush 0 Done Done Sent false
33079 10038 Acked Read 8 Done Done Sent false
33080 10039 Acked Write 3 Done Done Sent false
33081 10040 Acked Write 10 Done Done Sent false
33082 10041 Acked Read 8 Done Done Sent false
33083 10042 Acked Flush 0 Done Done Sent false
33084 10043 Acked Flush 0 Done Done Sent false
33085 10044 Acked Flush 0 Done Done Sent false
33086 10045 Acked Flush 0 Done Done Sent false
33087 10046 Acked Write 3 Done Done Sent false
33088 10047 Acked Read 6 Done Done Sent false
33089 10048 Acked Read 10 Done Done Sent false
33090 10049 Acked Write 6 Done Done Sent false
33091 10050 Acked Read 5 Done Done Sent false
33092 10051 Acked Write 7 Done Done Sent false
33093 10052 Acked Read 9 Done Done Sent false
33094 10053 Acked Write 8 Done Done Sent false
33095 10054 Acked Write 4 Done Done Sent false
33096 10055 Acked Read 3 Done Done Sent false
33097 10056 Acked Write 7 Done Done Sent false
33098 STATES DS:0 DS:1 DS:2 TOTAL
33099 Sent 0 0 301 301
33100 Done 301 301 0 602
33101 Skipped 0 0 0 0
33102 Error 0 0 0 0
33103 Last Flush: Some(JobId(10045)) Some(JobId(10045)) Some(JobId(9755))
33104 Downstairs last five completed: 9755 9754 9753 9752 9751
33105 Upstairs last five completed: 10056 10055 10054 10053 10052
33106 CLIENT: Up:0 ds:301 act:2
331072025-01-28T22:01:33.977ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
331082025-01-28T22:01:34.979ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
331092025-01-28T22:01:35.980ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
331102025-01-28T22:01:36.981ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
33111 ----------------------------------------------------------------
33112 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33113 States: Active Active Offline
33114 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33115 9756 Acked Write 7 Done Done Sent false
33116 9757 Acked Read 4 Done Done Sent false
33117 9758 Acked Read 5 Done Done Sent false
33118 9759 Acked Read 6 Done Done Sent false
33119 9760 Acked Read 6 Done Done Sent false
33120 9761 Acked Write 3 Done Done Sent false
33121 9762 Acked Write 9 Done Done Sent false
33122 9763 Acked Write 8 Done Done Sent false
33123 9764 Acked Read 1 Done Done Sent false
33124 9765 Acked Read 7 Done Done Sent false
33125 9766 Acked Write 9 Done Done Sent false
33126 9767 Acked Read 1 Done Done Sent false
33127 9768 Acked Read 4 Done Done Sent false
33128 9769 Acked Read 10 Done Done Sent false
33129 9770 Acked Write 9 Done Done Sent false
33130 9771 Acked Read 4 Done Done Sent false
33131 9772 Acked Write 1 Done Done Sent false
33132 9773 Acked Read 10 Done Done Sent false
33133 9774 Acked Write 9 Done Done Sent false
33134 9775 Acked Write 10 Done Done Sent false
33135 9776 Acked Write 9 Done Done Sent false
33136 9777 Acked Read 9 Done Done Sent false
33137 9778 Acked Write 6 Done Done Sent false
33138 9779 Acked Flush 0 Done Done Sent false
33139 9780 Acked Write 4 Done Done Sent false
33140 9781 Acked Write 4 Done Done Sent false
33141 9782 Acked Read 6 Done Done Sent false
33142 9783 Acked Read 2 Done Done Sent false
33143 9784 Acked Read 6 Done Done Sent false
33144 9785 Acked Flush 0 Done Done Sent false
33145 9786 Acked Flush 0 Done Done Sent false
33146 9787 Acked Read 10 Done Done Sent false
33147 9788 Acked Write 6 Done Done Sent false
33148 9789 Acked Read 10 Done Done Sent false
33149 9790 Acked Read 3 Done Done Sent false
33150 9791 Acked Write 10 Done Done Sent false
33151 9792 Acked Read 4 Done Done Sent false
33152 9793 Acked Read 2 Done Done Sent false
33153 9794 Acked Write 5 Done Done Sent false
33154 9795 Acked Flush 0 Done Done Sent false
33155 9796 Acked Write 10 Done Done Sent false
33156 9797 Acked Flush 0 Done Done Sent false
33157 9798 Acked Read 6 Done Done Sent false
33158 9799 Acked Write 7 Done Done Sent false
33159 9800 Acked Write 3 Done Done Sent false
33160 9801 Acked Read 1 Done Done Sent false
33161 9802 Acked Read 6 Done Done Sent false
33162 9803 Acked Read 7 Done Done Sent false
33163 9804 Acked Read 8 Done Done Sent false
33164 9805 Acked Read 8 Done Done Sent false
33165 9806 Acked Read 5 Done Done Sent false
33166 9807 Acked Write 2 Done Done Sent false
33167 9808 Acked Read 7 Done Done Sent false
33168 9809 Acked Write 7 Done Done Sent false
33169 9810 Acked Read 4 Done Done Sent false
33170 9811 Acked Read 7 Done Done Sent false
33171 9812 Acked Read 6 Done Done Sent false
33172 9813 Acked Read 6 Done Done Sent false
33173 9814 Acked Write 5 Done Done Sent false
33174 9815 Acked Write 9 Done Done Sent false
33175 9816 Acked Write 7 Done Done Sent false
33176 9817 Acked Write 8 Done Done Sent false
33177 9818 Acked Read 5 Done Done Sent false
33178 9819 Acked Read 4 Done Done Sent false
33179 9820 Acked Flush 0 Done Done Sent false
33180 9821 Acked Read 4 Done Done Sent false
33181 9822 Acked Write 9 Done Done Sent false
33182 9823 Acked Read 3 Done Done Sent false
33183 9824 Acked Read 8 Done Done Sent false
33184 9825 Acked Read 6 Done Done Sent false
33185 9826 Acked Read 2 Done Done Sent false
33186 9827 Acked Read 5 Done Done Sent false
33187 9828 Acked Write 3 Done Done Sent false
33188 9829 Acked Write 1 Done Done Sent false
33189 9830 Acked Read 4 Done Done Sent false
33190 9831 Acked Write 8 Done Done Sent false
33191 9832 Acked Write 3 Done Done Sent false
33192 9833 Acked Write 10 Done Done Sent false
33193 9834 Acked Read 7 Done Done Sent false
33194 9835 Acked Flush 0 Done Done Sent false
33195 9836 Acked Flush 0 Done Done Sent false
33196 9837 Acked Flush 0 Done Done Sent false
33197 9838 Acked Flush 0 Done Done Sent false
33198 9839 Acked Write 6 Done Done Sent false
33199 9840 Acked Write 10 Done Done Sent false
33200 9841 Acked Read 4 Done Done Sent false
33201 9842 Acked Read 7 Done Done Sent false
33202 9843 Acked Write 8 Done Done Sent false
33203 9844 Acked Read 6 Done Done Sent false
33204 9845 Acked Flush 0 Done Done Sent false
33205 9846 Acked Read 9 Done Done Sent false
33206 9847 Acked Read 7 Done Done Sent false
33207 9848 Acked Read 2 Done Done Sent false
33208 9849 Acked Write 2 Done Done Sent false
33209 9850 Acked Write 6 Done Done Sent false
33210 9851 Acked Read 8 Done Done Sent false
33211 9852 Acked Read 10 Done Done Sent false
33212 9853 Acked Write 10 Done Done Sent false
33213 9854 Acked Read 10 Done Done Sent false
33214 9855 Acked Read 9 Done Done Sent false
33215 9856 Acked Flush 0 Done Done Sent false
33216 9857 Acked Write 8 Done Done Sent false
33217 9858 Acked Flush 0 Done Done Sent false
33218 9859 Acked Write 2 Done Done Sent false
33219 9860 Acked Read 1 Done Done Sent false
33220 9861 Acked Write 8 Done Done Sent false
33221 9862 Acked Write 3 Done Done Sent false
33222 9863 Acked Write 1 Done Done Sent false
33223 9864 Acked Read 10 Done Done Sent false
33224 9865 Acked Read 6 Done Done Sent false
33225 9866 Acked Write 2 Done Done Sent false
33226 9867 Acked Write 3 Done Done Sent false
33227 9868 Acked Read 1 Done Done Sent false
33228 9869 Acked Read 8 Done Done Sent false
33229 9870 Acked Read 6 Done Done Sent false
33230 9871 Acked Write 7 Done Done Sent false
33231 9872 Acked Write 10 Done Done Sent false
33232 9873 Acked Read 8 Done Done Sent false
33233 9874 Acked Write 2 Done Done Sent false
33234 9875 Acked Read 6 Done Done Sent false
33235 9876 Acked Write 5 Done Done Sent false
33236 9877 Acked Write 3 Done Done Sent false
33237 9878 Acked Write 5 Done Done Sent false
33238 9879 Acked Write 6 Done Done Sent false
33239 9880 Acked Read 3 Done Done Sent false
33240 9881 Acked Write 1 Done Done Sent false
33241 9882 Acked Write 10 Done Done Sent false
33242 9883 Acked Flush 0 Done Done Sent false
33243 9884 Acked Read 10 Done Done Sent false
33244 9885 Acked Write 3 Done Done Sent false
33245 9886 Acked Read 1 Done Done Sent false
33246 9887 Acked Write 3 Done Done Sent false
33247 9888 Acked Write 2 Done Done Sent false
33248 9889 Acked Read 5 Done Done Sent false
33249 9890 Acked Read 9 Done Done Sent false
33250 9891 Acked Read 8 Done Done Sent false
33251 9892 Acked Write 8 Done Done Sent false
33252 9893 Acked Read 7 Done Done Sent false
33253 9894 Acked Read 8 Done Done Sent false
33254 9895 Acked Read 3 Done Done Sent false
33255 9896 Acked Read 5 Done Done Sent false
33256 9897 Acked Read 5 Done Done Sent false
33257 9898 Acked Read 8 Done Done Sent false
33258 9899 Acked Read 7 Done Done Sent false
33259 9900 Acked Read 5 Done Done Sent false
33260 9901 Acked Write 8 Done Done Sent false
33261 9902 Acked Write 6 Done Done Sent false
33262 9903 Acked Write 8 Done Done Sent false
33263 9904 Acked Write 6 Done Done Sent false
33264 9905 Acked Read 7 Done Done Sent false
33265 9906 Acked Read 6 Done Done Sent false
33266 9907 Acked Read 2 Done Done Sent false
33267 9908 Acked Read 3 Done Done Sent false
33268 9909 Acked Flush 0 Done Done Sent false
33269 9910 Acked Read 2 Done Done Sent false
33270 9911 Acked Write 6 Done Done Sent false
33271 9912 Acked Read 3 Done Done Sent false
33272 9913 Acked Flush 0 Done Done Sent false
33273 9914 Acked Write 3 Done Done Sent false
33274 9915 Acked Write 1 Done Done Sent false
33275 9916 Acked Write 8 Done Done Sent false
33276 9917 Acked Write 8 Done Done Sent false
33277 9918 Acked Flush 0 Done Done Sent false
33278 9919 Acked Read 4 Done Done Sent false
33279 9920 Acked Read 3 Done Done Sent false
33280 9921 Acked Read 6 Done Done Sent false
33281 9922 Acked Write 1 Done Done Sent false
33282 9923 Acked Write 1 Done Done Sent false
33283 9924 Acked Read 10 Done Done Sent false
33284 9925 Acked Write 7 Done Done Sent false
33285 9926 Acked Write 4 Done Done Sent false
33286 9927 Acked Write 10 Done Done Sent false
33287 9928 Acked Read 10 Done Done Sent false
33288 9929 Acked Read 9 Done Done Sent false
33289 9930 Acked Read 6 Done Done Sent false
33290 9931 Acked Read 5 Done Done Sent false
33291 9932 Acked Write 6 Done Done Sent false
33292 9933 Acked Flush 0 Done Done Sent false
33293 9934 Acked Write 1 Done Done Sent false
33294 9935 Acked Write 6 Done Done Sent false
33295 9936 Acked Read 8 Done Done Sent false
33296 9937 Acked Read 10 Done Done Sent false
33297 9938 Acked Read 3 Done Done Sent false
33298 9939 Acked Write 2 Done Done Sent false
33299 9940 Acked Read 10 Done Done Sent false
33300 9941 Acked Read 2 Done Done Sent false
33301 9942 Acked Read 6 Done Done Sent false
33302 9943 Acked Flush 0 Done Done Sent false
33303 9944 Acked Read 2 Done Done Sent false
33304 9945 Acked Read 3 Done Done Sent false
33305 9946 Acked Read 6 Done Done Sent false
33306 9947 Acked Read 9 Done Done Sent false
33307 9948 Acked Read 3 Done Done Sent false
33308 9949 Acked Write 3 Done Done Sent false
33309 9950 Acked Write 5 Done Done Sent false
33310 9951 Acked Write 2 Done Done Sent false
33311 9952 Acked Read 10 Done Done Sent false
33312 9953 Acked Read 8 Done Done Sent false
33313 9954 Acked Write 3 Done Done Sent false
33314 9955 Acked Read 4 Done Done Sent false
33315 9956 Acked Write 8 Done Done Sent false
33316 9957 Acked Write 5 Done Done Sent false
33317 9958 Acked Read 2 Done Done Sent false
33318 9959 Acked Write 1 Done Done Sent false
33319 9960 Acked Write 5 Done Done Sent false
33320 9961 Acked Read 1 Done Done Sent false
33321 9962 Acked Write 4 Done Done Sent false
33322 9963 Acked Flush 0 Done Done Sent false
33323 9964 Acked Read 4 Done Done Sent false
33324 9965 Acked Read 1 Done Done Sent false
33325 9966 Acked Read 9 Done Done Sent false
33326 9967 Acked Write 6 Done Done Sent false
33327 9968 Acked Write 3 Done Done Sent false
33328 9969 Acked Read 4 Done Done Sent false
33329 9970 Acked Write 3 Done Done Sent false
33330 9971 Acked Read 7 Done Done Sent false
33331 9972 Acked Read 2 Done Done Sent false
33332 9973 Acked Read 7 Done Done Sent false
33333 9974 Acked Write 6 Done Done Sent false
33334 9975 Acked Write 5 Done Done Sent false
33335 9976 Acked Read 5 Done Done Sent false
33336 9977 Acked Write 3 Done Done Sent false
33337 9978 Acked Read 8 Done Done Sent false
33338 9979 Acked Read 10 Done Done Sent false
33339 9980 Acked Write 1 Done Done Sent false
33340 9981 Acked Write 8 Done Done Sent false
33341 9982 Acked Read 10 Done Done Sent false
33342 9983 Acked Read 7 Done Done Sent false
33343 9984 Acked Read 5 Done Done Sent false
33344 9985 Acked Flush 0 Done Done Sent false
33345 9986 Acked Write 3 Done Done Sent false
33346 9987 Acked Read 2 Done Done Sent false
33347 9988 Acked Read 4 Done Done Sent false
33348 9989 Acked Flush 0 Done Done Sent false
33349 9990 Acked Read 1 Done Done Sent false
33350 9991 Acked Read 4 Done Done Sent false
33351 9992 Acked Write 5 Done Done Sent false
33352 9993 Acked Read 10 Done Done Sent false
33353 9994 Acked Flush 0 Done Done Sent false
33354 9995 Acked Write 7 Done Done Sent false
33355 9996 Acked Write 5 Done Done Sent false
33356 9997 Acked Flush 0 Done Done Sent false
33357 9998 Acked Write 3 Done Done Sent false
33358 9999 Acked Write 5 Done Done Sent false
33359 10000 Acked Read 7 Done Done Sent false
33360 10001 Acked Write 2 Done Done Sent false
33361 10002 Acked Read 2 Done Done Sent false
33362 10003 Acked Read 2 Done Done Sent false
33363 10004 Acked Write 9 Done Done Sent false
33364 10005 Acked Read 9 Done Done Sent false
33365 10006 Acked Read 2 Done Done Sent false
33366 10007 Acked Write 3 Done Done Sent false
33367 10008 Acked Write 7 Done Done Sent false
33368 10009 Acked Write 4 Done Done Sent false
33369 10010 Acked Write 3 Done Done Sent false
33370 10011 Acked Flush 0 Done Done Sent false
33371 10012 Acked Read 3 Done Done Sent false
33372 10013 Acked Write 3 Done Done Sent false
33373 10014 Acked Write 1 Done Done Sent false
33374 10015 Acked Read 4 Done Done Sent false
33375 10016 Acked Read 6 Done Done Sent false
33376 10017 Acked Write 7 Done Done Sent false
33377 10018 Acked Write 9 Done Done Sent false
33378 10019 Acked Write 3 Done Done Sent false
33379 10020 Acked Write 7 Done Done Sent false
33380 10021 Acked Write 7 Done Done Sent false
33381 10022 Acked Read 8 Done Done Sent false
33382 10023 Acked Flush 0 Done Done Sent false
33383 10024 Acked Read 8 Done Done Sent false
33384 10025 Acked Write 9 Done Done Sent false
33385 10026 Acked Read 10 Done Done Sent false
33386 10027 Acked Write 6 Done Done Sent false
33387 10028 Acked Write 4 Done Done Sent false
33388 10029 Acked Read 9 Done Done Sent false
33389 10030 Acked Write 6 Done Done Sent false
33390 10031 Acked Read 7 Done Done Sent false
33391 10032 Acked Read 2 Done Done Sent false
33392 10033 Acked Write 9 Done Done Sent false
33393 10034 Acked Write 5 Done Done Sent false
33394 10035 Acked Write 5 Done Done Sent false
33395 10036 Acked Read 2 Done Done Sent false
33396 10037 Acked Flush 0 Done Done Sent false
33397 10038 Acked Read 8 Done Done Sent false
33398 10039 Acked Write 3 Done Done Sent false
33399 10040 Acked Write 10 Done Done Sent false
33400 10041 Acked Read 8 Done Done Sent false
33401 10042 Acked Flush 0 Done Done Sent false
33402 10043 Acked Flush 0 Done Done Sent false
33403 10044 Acked Flush 0 Done Done Sent false
33404 10045 Acked Flush 0 Done Done Sent false
33405 10046 Acked Write 3 Done Done Sent false
33406 10047 Acked Read 6 Done Done Sent false
33407 10048 Acked Read 10 Done Done Sent false
33408 10049 Acked Write 6 Done Done Sent false
33409 10050 Acked Read 5 Done Done Sent false
33410 10051 Acked Write 7 Done Done Sent false
33411 10052 Acked Read 9 Done Done Sent false
33412 10053 Acked Write 8 Done Done Sent false
33413 10054 Acked Write 4 Done Done Sent false
33414 10055 Acked Read 3 Done Done Sent false
33415 10056 Acked Write 7 Done Done Sent false
33416 STATES DS:0 DS:1 DS:2 TOTAL
33417 Sent 0 0 301 301
33418 Done 301 301 0 602
33419 Skipped 0 0 0 0
33420 Error 0 0 0 0
33421 Last Flush: Some(JobId(10045)) Some(JobId(10045)) Some(JobId(9755))
33422 Downstairs last five completed: 9755 9754 9753 9752 9751
33423 Upstairs last five completed: 10056 10055 10054 10053 10052
33424 CLIENT: Up:0 ds:301 act:2
334252025-01-28T22:01:37.982ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
334262025-01-28T22:01:38.984ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
334272025-01-28T22:01:39.093ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
334282025-01-28T22:01:39.093ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
334292025-01-28T22:01:39.093ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
334302025-01-28T22:01:39.093ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
334312025-01-28T22:01:39.093ZINFOcrucible: client task is exiting = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
334322025-01-28T22:01:39.985ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
334332025-01-28T22:01:40.987ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
33434 [8830] got start action
33435 [8830] I am going to restart
33436 Starting downstairs at port 8830
33437 Make output file at "/tmp/dsc/downstairs-8830.txt"
33438 [8830][2] reports Starting
33439 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1232)
33440 [8830][2] reports Running
33441 ----------------------------------------------------------------
33442 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33443 States: Active Active Offline
33444 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33445 9756 Acked Write 7 Done Done Sent false
33446 9757 Acked Read 4 Done Done Sent false
33447 9758 Acked Read 5 Done Done Sent false
33448 9759 Acked Read 6 Done Done Sent false
33449 9760 Acked Read 6 Done Done Sent false
33450 9761 Acked Write 3 Done Done Sent false
33451 9762 Acked Write 9 Done Done Sent false
33452 9763 Acked Write 8 Done Done Sent false
33453 9764 Acked Read 1 Done Done Sent false
33454 9765 Acked Read 7 Done Done Sent false
33455 9766 Acked Write 9 Done Done Sent false
33456 9767 Acked Read 1 Done Done Sent false
33457 9768 Acked Read 4 Done Done Sent false
33458 9769 Acked Read 10 Done Done Sent false
33459 9770 Acked Write 9 Done Done Sent false
33460 9771 Acked Read 4 Done Done Sent false
33461 9772 Acked Write 1 Done Done Sent false
33462 9773 Acked Read 10 Done Done Sent false
33463 9774 Acked Write 9 Done Done Sent false
33464 9775 Acked Write 10 Done Done Sent false
33465 9776 Acked Write 9 Done Done Sent false
33466 9777 Acked Read 9 Done Done Sent false
33467 9778 Acked Write 6 Done Done Sent false
33468 9779 Acked Flush 0 Done Done Sent false
33469 9780 Acked Write 4 Done Done Sent false
33470 9781 Acked Write 4 Done Done Sent false
33471 9782 Acked Read 6 Done Done Sent false
33472 9783 Acked Read 2 Done Done Sent false
33473 9784 Acked Read 6 Done Done Sent false
33474 9785 Acked Flush 0 Done Done Sent false
33475 9786 Acked Flush 0 Done Done Sent false
33476 9787 Acked Read 10 Done Done Sent false
33477 9788 Acked Write 6 Done Done Sent false
33478 9789 Acked Read 10 Done Done Sent false
33479 9790 Acked Read 3 Done Done Sent false
33480 9791 Acked Write 10 Done Done Sent false
33481 9792 Acked Read 4 Done Done Sent false
33482 9793 Acked Read 2 Done Done Sent false
33483 9794 Acked Write 5 Done Done Sent false
33484 9795 Acked Flush 0 Done Done Sent false
33485 9796 Acked Write 10 Done Done Sent false
33486 9797 Acked Flush 0 Done Done Sent false
33487 9798 Acked Read 6 Done Done Sent false
33488 9799 Acked Write 7 Done Done Sent false
33489 9800 Acked Write 3 Done Done Sent false
33490 9801 Acked Read 1 Done Done Sent false
33491 9802 Acked Read 6 Done Done Sent false
33492 9803 Acked Read 7 Done Done Sent false
33493 9804 Acked Read 8 Done Done Sent false
33494 9805 Acked Read 8 Done Done Sent false
33495 9806 Acked Read 5 Done Done Sent false
33496 9807 Acked Write 2 Done Done Sent false
33497 9808 Acked Read 7 Done Done Sent false
33498 9809 Acked Write 7 Done Done Sent false
33499 9810 Acked Read 4 Done Done Sent false
33500 9811 Acked Read 7 Done Done Sent false
33501 9812 Acked Read 6 Done Done Sent false
33502 9813 Acked Read 6 Done Done Sent false
33503 9814 Acked Write 5 Done Done Sent false
33504 9815 Acked Write 9 Done Done Sent false
33505 9816 Acked Write 7 Done Done Sent false
33506 9817 Acked Write 8 Done Done Sent false
33507 9818 Acked Read 5 Done Done Sent false
33508 9819 Acked Read 4 Done Done Sent false
33509 9820 Acked Flush 0 Done Done Sent false
33510 9821 Acked Read 4 Done Done Sent false
33511 9822 Acked Write 9 Done Done Sent false
33512 9823 Acked Read 3 Done Done Sent false
33513 9824 Acked Read 8 Done Done Sent false
33514 9825 Acked Read 6 Done Done Sent false
33515 9826 Acked Read 2 Done Done Sent false
33516 9827 Acked Read 5 Done Done Sent false
33517 9828 Acked Write 3 Done Done Sent false
33518 9829 Acked Write 1 Done Done Sent false
33519 9830 Acked Read 4 Done Done Sent false
33520 9831 Acked Write 8 Done Done Sent false
33521 9832 Acked Write 3 Done Done Sent false
33522 9833 Acked Write 10 Done Done Sent false
33523 9834 Acked Read 7 Done Done Sent false
33524 9835 Acked Flush 0 Done Done Sent false
33525 9836 Acked Flush 0 Done Done Sent false
33526 9837 Acked Flush 0 Done Done Sent false
33527 9838 Acked Flush 0 Done Done Sent false
33528 9839 Acked Write 6 Done Done Sent false
33529 9840 Acked Write 10 Done Done Sent false
33530 9841 Acked Read 4 Done Done Sent false
33531 9842 Acked Read 7 Done Done Sent false
33532 9843 Acked Write 8 Done Done Sent false
33533 9844 Acked Read 6 Done Done Sent false
33534 9845 Acked Flush 0 Done Done Sent false
33535 9846 Acked Read 9 Done Done Sent false
33536 9847 Acked Read 7 Done Done Sent false
33537 9848 Acked Read 2 Done Done Sent false
33538 9849 Acked Write 2 Done Done Sent false
33539 9850 Acked Write 6 Done Done Sent false
33540 9851 Acked Read 8 Done Done Sent false
33541 9852 Acked Read 10 Done Done Sent false
33542 9853 Acked Write 10 Done Done Sent false
33543 9854 Acked Read 10 Done Done Sent false
33544 9855 Acked Read 9 Done Done Sent false
33545 9856 Acked Flush 0 Done Done Sent false
33546 9857 Acked Write 8 Done Done Sent false
33547 9858 Acked Flush 0 Done Done Sent false
33548 9859 Acked Write 2 Done Done Sent false
33549 9860 Acked Read 1 Done Done Sent false
33550 9861 Acked Write 8 Done Done Sent false
33551 9862 Acked Write 3 Done Done Sent false
33552 9863 Acked Write 1 Done Done Sent false
33553 9864 Acked Read 10 Done Done Sent false
33554 9865 Acked Read 6 Done Done Sent false
33555 9866 Acked Write 2 Done Done Sent false
33556 9867 Acked Write 3 Done Done Sent false
33557 9868 Acked Read 1 Done Done Sent false
33558 9869 Acked Read 8 Done Done Sent false
33559 9870 Acked Read 6 Done Done Sent false
33560 9871 Acked Write 7 Done Done Sent false
33561 9872 Acked Write 10 Done Done Sent false
33562 9873 Acked Read 8 Done Done Sent false
33563 9874 Acked Write 2 Done Done Sent false
33564 9875 Acked Read 6 Done Done Sent false
33565 9876 Acked Write 5 Done Done Sent false
33566 9877 Acked Write 3 Done Done Sent false
33567 9878 Acked Write 5 Done Done Sent false
33568 9879 Acked Write 6 Done Done Sent false
33569 9880 Acked Read 3 Done Done Sent false
33570 9881 Acked Write 1 Done Done Sent false
33571 9882 Acked Write 10 Done Done Sent false
33572 9883 Acked Flush 0 Done Done Sent false
33573 9884 Acked Read 10 Done Done Sent false
33574 9885 Acked Write 3 Done Done Sent false
33575 9886 Acked Read 1 Done Done Sent false
33576 9887 Acked Write 3 Done Done Sent false
33577 9888 Acked Write 2 Done Done Sent false
33578 9889 Acked Read 5 Done Done Sent false
33579 9890 Acked Read 9 Done Done Sent false
33580 9891 Acked Read 8 Done Done Sent false
33581 9892 Acked Write 8 Done Done Sent false
33582 9893 Acked Read 7 Done Done Sent false
33583 9894 Acked Read 8 Done Done Sent false
33584 9895 Acked Read 3 Done Done Sent false
33585 9896 Acked Read 5 Done Done Sent false
33586 9897 Acked Read 5 Done Done Sent false
33587 9898 Acked Read 8 Done Done Sent false
33588 9899 Acked Read 7 Done Done Sent false
33589 9900 Acked Read 5 Done Done Sent false
33590 9901 Acked Write 8 Done Done Sent false
33591 9902 Acked Write 6 Done Done Sent false
33592 9903 Acked Write 8 Done Done Sent false
33593 9904 Acked Write 6 Done Done Sent false
33594 9905 Acked Read 7 Done Done Sent false
33595 9906 Acked Read 6 Done Done Sent false
33596 9907 Acked Read 2 Done Done Sent false
33597 9908 Acked Read 3 Done Done Sent false
33598 9909 Acked Flush 0 Done Done Sent false
33599 9910 Acked Read 2 Done Done Sent false
33600 9911 Acked Write 6 Done Done Sent false
33601 9912 Acked Read 3 Done Done Sent false
33602 9913 Acked Flush 0 Done Done Sent false
33603 9914 Acked Write 3 Done Done Sent false
33604 9915 Acked Write 1 Done Done Sent false
33605 9916 Acked Write 8 Done Done Sent false
33606 9917 Acked Write 8 Done Done Sent false
33607 9918 Acked Flush 0 Done Done Sent false
33608 9919 Acked Read 4 Done Done Sent false
33609 9920 Acked Read 3 Done Done Sent false
33610 9921 Acked Read 6 Done Done Sent false
33611 9922 Acked Write 1 Done Done Sent false
33612 9923 Acked Write 1 Done Done Sent false
33613 9924 Acked Read 10 Done Done Sent false
33614 9925 Acked Write 7 Done Done Sent false
33615 9926 Acked Write 4 Done Done Sent false
33616 9927 Acked Write 10 Done Done Sent false
33617 9928 Acked Read 10 Done Done Sent false
33618 9929 Acked Read 9 Done Done Sent false
33619 9930 Acked Read 6 Done Done Sent false
33620 9931 Acked Read 5 Done Done Sent false
33621 9932 Acked Write 6 Done Done Sent false
33622 9933 Acked Flush 0 Done Done Sent false
33623 9934 Acked Write 1 Done Done Sent false
33624 9935 Acked Write 6 Done Done Sent false
33625 9936 Acked Read 8 Done Done Sent false
33626 9937 Acked Read 10 Done Done Sent false
33627 9938 Acked Read 3 Done Done Sent false
33628 9939 Acked Write 2 Done Done Sent false
33629 9940 Acked Read 10 Done Done Sent false
33630 9941 Acked Read 2 Done Done Sent false
33631 9942 Acked Read 6 Done Done Sent false
33632 9943 Acked Flush 0 Done Done Sent false
33633 9944 Acked Read 2 Done Done Sent false
33634 9945 Acked Read 3 Done Done Sent false
33635 9946 Acked Read 6 Done Done Sent false
33636 9947 Acked Read 9 Done Done Sent false
33637 9948 Acked Read 3 Done Done Sent false
33638 9949 Acked Write 3 Done Done Sent false
33639 9950 Acked Write 5 Done Done Sent false
33640 9951 Acked Write 2 Done Done Sent false
33641 9952 Acked Read 10 Done Done Sent false
33642 9953 Acked Read 8 Done Done Sent false
33643 9954 Acked Write 3 Done Done Sent false
33644 9955 Acked Read 4 Done Done Sent false
33645 9956 Acked Write 8 Done Done Sent false
33646 9957 Acked Write 5 Done Done Sent false
33647 9958 Acked Read 2 Done Done Sent false
33648 9959 Acked Write 1 Done Done Sent false
33649 9960 Acked Write 5 Done Done Sent false
33650 9961 Acked Read 1 Done Done Sent false
33651 9962 Acked Write 4 Done Done Sent false
33652 9963 Acked Flush 0 Done Done Sent false
33653 9964 Acked Read 4 Done Done Sent false
33654 9965 Acked Read 1 Done Done Sent false
33655 9966 Acked Read 9 Done Done Sent false
33656 9967 Acked Write 6 Done Done Sent false
33657 9968 Acked Write 3 Done Done Sent false
33658 9969 Acked Read 4 Done Done Sent false
33659 9970 Acked Write 3 Done Done Sent false
33660 9971 Acked Read 7 Done Done Sent false
33661 9972 Acked Read 2 Done Done Sent false
33662 9973 Acked Read 7 Done Done Sent false
33663 9974 Acked Write 6 Done Done Sent false
33664 9975 Acked Write 5 Done Done Sent false
33665 9976 Acked Read 5 Done Done Sent false
33666 9977 Acked Write 3 Done Done Sent false
33667 9978 Acked Read 8 Done Done Sent false
33668 9979 Acked Read 10 Done Done Sent false
33669 9980 Acked Write 1 Done Done Sent false
33670 9981 Acked Write 8 Done Done Sent false
33671 9982 Acked Read 10 Done Done Sent false
33672 9983 Acked Read 7 Done Done Sent false
33673 9984 Acked Read 5 Done Done Sent false
33674 9985 Acked Flush 0 Done Done Sent false
33675 9986 Acked Write 3 Done Done Sent false
33676 9987 Acked Read 2 Done Done Sent false
33677 9988 Acked Read 4 Done Done Sent false
33678 9989 Acked Flush 0 Done Done Sent false
33679 9990 Acked Read 1 Done Done Sent false
33680 9991 Acked Read 4 Done Done Sent false
33681 9992 Acked Write 5 Done Done Sent false
33682 9993 Acked Read 10 Done Done Sent false
33683 9994 Acked Flush 0 Done Done Sent false
33684 9995 Acked Write 7 Done Done Sent false
33685 9996 Acked Write 5 Done Done Sent false
33686 9997 Acked Flush 0 Done Done Sent false
33687 9998 Acked Write 3 Done Done Sent false
33688 9999 Acked Write 5 Done Done Sent false
33689 10000 Acked Read 7 Done Done Sent false
33690 10001 Acked Write 2 Done Done Sent false
33691 10002 Acked Read 2 Done Done Sent false
33692 10003 Acked Read 2 Done Done Sent false
33693 10004 Acked Write 9 Done Done Sent false
33694 10005 Acked Read 9 Done Done Sent false
33695 10006 Acked Read 2 Done Done Sent false
33696 10007 Acked Write 3 Done Done Sent false
33697 10008 Acked Write 7 Done Done Sent false
33698 10009 Acked Write 4 Done Done Sent false
33699 10010 Acked Write 3 Done Done Sent false
33700 10011 Acked Flush 0 Done Done Sent false
33701 10012 Acked Read 3 Done Done Sent false
33702 10013 Acked Write 3 Done Done Sent false
33703 10014 Acked Write 1 Done Done Sent false
33704 10015 Acked Read 4 Done Done Sent false
33705 10016 Acked Read 6 Done Done Sent false
33706 10017 Acked Write 7 Done Done Sent false
33707 10018 Acked Write 9 Done Done Sent false
33708 10019 Acked Write 3 Done Done Sent false
33709 10020 Acked Write 7 Done Done Sent false
33710 10021 Acked Write 7 Done Done Sent false
33711 10022 Acked Read 8 Done Done Sent false
33712 10023 Acked Flush 0 Done Done Sent false
33713 10024 Acked Read 8 Done Done Sent false
33714 10025 Acked Write 9 Done Done Sent false
33715 10026 Acked Read 10 Done Done Sent false
33716 10027 Acked Write 6 Done Done Sent false
33717 10028 Acked Write 4 Done Done Sent false
33718 10029 Acked Read 9 Done Done Sent false
33719 10030 Acked Write 6 Done Done Sent false
33720 10031 Acked Read 7 Done Done Sent false
33721 10032 Acked Read 2 Done Done Sent false
33722 10033 Acked Write 9 Done Done Sent false
33723 10034 Acked Write 5 Done Done Sent false
33724 10035 Acked Write 5 Done Done Sent false
33725 10036 Acked Read 2 Done Done Sent false
33726 10037 Acked Flush 0 Done Done Sent false
33727 10038 Acked Read 8 Done Done Sent false
33728 10039 Acked Write 3 Done Done Sent false
33729 10040 Acked Write 10 Done Done Sent false
33730 10041 Acked Read 8 Done Done Sent false
33731 10042 Acked Flush 0 Done Done Sent false
33732 10043 Acked Flush 0 Done Done Sent false
33733 10044 Acked Flush 0 Done Done Sent false
33734 10045 Acked Flush 0 Done Done Sent false
33735 10046 Acked Write 3 Done Done Sent false
33736 10047 Acked Read 6 Done Done Sent false
33737 10048 Acked Read 10 Done Done Sent false
33738 10049 Acked Write 6 Done Done Sent false
33739 10050 Acked Read 5 Done Done Sent false
33740 10051 Acked Write 7 Done Done Sent false
33741 10052 Acked Read 9 Done Done Sent false
33742 10053 Acked Write 8 Done Done Sent false
33743 10054 Acked Write 4 Done Done Sent false
33744 10055 Acked Read 3 Done Done Sent false
33745 10056 Acked Write 7 Done Done Sent false
33746 STATES DS:0 DS:1 DS:2 TOTAL
33747 Sent 0 0 301 301
33748 Done 301 301 0 602
33749 Skipped 0 0 0 0
33750 Error 0 0 0 0
33751 Last Flush: Some(JobId(10045)) Some(JobId(10045)) Some(JobId(9755))
33752 Downstairs last five completed: 9755 9754 9753 9752 9751
33753 Upstairs last five completed: 10056 10055 10054 10053 10052
33754 CLIENT: Up:0 ds:301 act:2
337552025-01-28T22:01:41.989ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
337562025-01-28T22:01:42.991ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
337572025-01-28T22:01:43.993ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
337582025-01-28T22:01:44.994ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
33759 ----------------------------------------------------------------
33760 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33761 States: Active Active Offline
33762 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33763 9756 Acked Write 7 Done Done Sent false
33764 9757 Acked Read 4 Done Done Sent false
33765 9758 Acked Read 5 Done Done Sent false
33766 9759 Acked Read 6 Done Done Sent false
33767 9760 Acked Read 6 Done Done Sent false
33768 9761 Acked Write 3 Done Done Sent false
33769 9762 Acked Write 9 Done Done Sent false
33770 9763 Acked Write 8 Done Done Sent false
33771 9764 Acked Read 1 Done Done Sent false
33772 9765 Acked Read 7 Done Done Sent false
33773 9766 Acked Write 9 Done Done Sent false
33774 9767 Acked Read 1 Done Done Sent false
33775 9768 Acked Read 4 Done Done Sent false
33776 9769 Acked Read 10 Done Done Sent false
33777 9770 Acked Write 9 Done Done Sent false
33778 9771 Acked Read 4 Done Done Sent false
33779 9772 Acked Write 1 Done Done Sent false
33780 9773 Acked Read 10 Done Done Sent false
33781 9774 Acked Write 9 Done Done Sent false
33782 9775 Acked Write 10 Done Done Sent false
33783 9776 Acked Write 9 Done Done Sent false
33784 9777 Acked Read 9 Done Done Sent false
33785 9778 Acked Write 6 Done Done Sent false
33786 9779 Acked Flush 0 Done Done Sent false
33787 9780 Acked Write 4 Done Done Sent false
33788 9781 Acked Write 4 Done Done Sent false
33789 9782 Acked Read 6 Done Done Sent false
33790 9783 Acked Read 2 Done Done Sent false
33791 9784 Acked Read 6 Done Done Sent false
33792 9785 Acked Flush 0 Done Done Sent false
33793 9786 Acked Flush 0 Done Done Sent false
33794 9787 Acked Read 10 Done Done Sent false
33795 9788 Acked Write 6 Done Done Sent false
33796 9789 Acked Read 10 Done Done Sent false
33797 9790 Acked Read 3 Done Done Sent false
33798 9791 Acked Write 10 Done Done Sent false
33799 9792 Acked Read 4 Done Done Sent false
33800 9793 Acked Read 2 Done Done Sent false
33801 9794 Acked Write 5 Done Done Sent false
33802 9795 Acked Flush 0 Done Done Sent false
33803 9796 Acked Write 10 Done Done Sent false
33804 9797 Acked Flush 0 Done Done Sent false
33805 9798 Acked Read 6 Done Done Sent false
33806 9799 Acked Write 7 Done Done Sent false
33807 9800 Acked Write 3 Done Done Sent false
33808 9801 Acked Read 1 Done Done Sent false
33809 9802 Acked Read 6 Done Done Sent false
33810 9803 Acked Read 7 Done Done Sent false
33811 9804 Acked Read 8 Done Done Sent false
33812 9805 Acked Read 8 Done Done Sent false
33813 9806 Acked Read 5 Done Done Sent false
33814 9807 Acked Write 2 Done Done Sent false
33815 9808 Acked Read 7 Done Done Sent false
33816 9809 Acked Write 7 Done Done Sent false
33817 9810 Acked Read 4 Done Done Sent false
33818 9811 Acked Read 7 Done Done Sent false
33819 9812 Acked Read 6 Done Done Sent false
33820 9813 Acked Read 6 Done Done Sent false
33821 9814 Acked Write 5 Done Done Sent false
33822 9815 Acked Write 9 Done Done Sent false
33823 9816 Acked Write 7 Done Done Sent false
33824 9817 Acked Write 8 Done Done Sent false
33825 9818 Acked Read 5 Done Done Sent false
33826 9819 Acked Read 4 Done Done Sent false
33827 9820 Acked Flush 0 Done Done Sent false
33828 9821 Acked Read 4 Done Done Sent false
33829 9822 Acked Write 9 Done Done Sent false
33830 9823 Acked Read 3 Done Done Sent false
33831 9824 Acked Read 8 Done Done Sent false
33832 9825 Acked Read 6 Done Done Sent false
33833 9826 Acked Read 2 Done Done Sent false
33834 9827 Acked Read 5 Done Done Sent false
33835 9828 Acked Write 3 Done Done Sent false
33836 9829 Acked Write 1 Done Done Sent false
33837 9830 Acked Read 4 Done Done Sent false
33838 9831 Acked Write 8 Done Done Sent false
33839 9832 Acked Write 3 Done Done Sent false
33840 9833 Acked Write 10 Done Done Sent false
33841 9834 Acked Read 7 Done Done Sent false
33842 9835 Acked Flush 0 Done Done Sent false
33843 9836 Acked Flush 0 Done Done Sent false
33844 9837 Acked Flush 0 Done Done Sent false
33845 9838 Acked Flush 0 Done Done Sent false
33846 9839 Acked Write 6 Done Done Sent false
33847 9840 Acked Write 10 Done Done Sent false
33848 9841 Acked Read 4 Done Done Sent false
33849 9842 Acked Read 7 Done Done Sent false
33850 9843 Acked Write 8 Done Done Sent false
33851 9844 Acked Read 6 Done Done Sent false
33852 9845 Acked Flush 0 Done Done Sent false
33853 9846 Acked Read 9 Done Done Sent false
33854 9847 Acked Read 7 Done Done Sent false
33855 9848 Acked Read 2 Done Done Sent false
33856 9849 Acked Write 2 Done Done Sent false
33857 9850 Acked Write 6 Done Done Sent false
33858 9851 Acked Read 8 Done Done Sent false
33859 9852 Acked Read 10 Done Done Sent false
33860 9853 Acked Write 10 Done Done Sent false
33861 9854 Acked Read 10 Done Done Sent false
33862 9855 Acked Read 9 Done Done Sent false
33863 9856 Acked Flush 0 Done Done Sent false
33864 9857 Acked Write 8 Done Done Sent false
33865 9858 Acked Flush 0 Done Done Sent false
33866 9859 Acked Write 2 Done Done Sent false
33867 9860 Acked Read 1 Done Done Sent false
33868 9861 Acked Write 8 Done Done Sent false
33869 9862 Acked Write 3 Done Done Sent false
33870 9863 Acked Write 1 Done Done Sent false
33871 9864 Acked Read 10 Done Done Sent false
33872 9865 Acked Read 6 Done Done Sent false
33873 9866 Acked Write 2 Done Done Sent false
33874 9867 Acked Write 3 Done Done Sent false
33875 9868 Acked Read 1 Done Done Sent false
33876 9869 Acked Read 8 Done Done Sent false
33877 9870 Acked Read 6 Done Done Sent false
33878 9871 Acked Write 7 Done Done Sent false
33879 9872 Acked Write 10 Done Done Sent false
33880 9873 Acked Read 8 Done Done Sent false
33881 9874 Acked Write 2 Done Done Sent false
33882 9875 Acked Read 6 Done Done Sent false
33883 9876 Acked Write 5 Done Done Sent false
33884 9877 Acked Write 3 Done Done Sent false
33885 9878 Acked Write 5 Done Done Sent false
33886 9879 Acked Write 6 Done Done Sent false
33887 9880 Acked Read 3 Done Done Sent false
33888 9881 Acked Write 1 Done Done Sent false
33889 9882 Acked Write 10 Done Done Sent false
33890 9883 Acked Flush 0 Done Done Sent false
33891 9884 Acked Read 10 Done Done Sent false
33892 9885 Acked Write 3 Done Done Sent false
33893 9886 Acked Read 1 Done Done Sent false
33894 9887 Acked Write 3 Done Done Sent false
33895 9888 Acked Write 2 Done Done Sent false
33896 9889 Acked Read 5 Done Done Sent false
33897 9890 Acked Read 9 Done Done Sent false
33898 9891 Acked Read 8 Done Done Sent false
33899 9892 Acked Write 8 Done Done Sent false
33900 9893 Acked Read 7 Done Done Sent false
33901 9894 Acked Read 8 Done Done Sent false
33902 9895 Acked Read 3 Done Done Sent false
33903 9896 Acked Read 5 Done Done Sent false
33904 9897 Acked Read 5 Done Done Sent false
33905 9898 Acked Read 8 Done Done Sent false
33906 9899 Acked Read 7 Done Done Sent false
33907 9900 Acked Read 5 Done Done Sent false
33908 9901 Acked Write 8 Done Done Sent false
33909 9902 Acked Write 6 Done Done Sent false
33910 9903 Acked Write 8 Done Done Sent false
33911 9904 Acked Write 6 Done Done Sent false
33912 9905 Acked Read 7 Done Done Sent false
33913 9906 Acked Read 6 Done Done Sent false
33914 9907 Acked Read 2 Done Done Sent false
33915 9908 Acked Read 3 Done Done Sent false
33916 9909 Acked Flush 0 Done Done Sent false
33917 9910 Acked Read 2 Done Done Sent false
33918 9911 Acked Write 6 Done Done Sent false
33919 9912 Acked Read 3 Done Done Sent false
33920 9913 Acked Flush 0 Done Done Sent false
33921 9914 Acked Write 3 Done Done Sent false
33922 9915 Acked Write 1 Done Done Sent false
33923 9916 Acked Write 8 Done Done Sent false
33924 9917 Acked Write 8 Done Done Sent false
33925 9918 Acked Flush 0 Done Done Sent false
33926 9919 Acked Read 4 Done Done Sent false
33927 9920 Acked Read 3 Done Done Sent false
33928 9921 Acked Read 6 Done Done Sent false
33929 9922 Acked Write 1 Done Done Sent false
33930 9923 Acked Write 1 Done Done Sent false
33931 9924 Acked Read 10 Done Done Sent false
33932 9925 Acked Write 7 Done Done Sent false
33933 9926 Acked Write 4 Done Done Sent false
33934 9927 Acked Write 10 Done Done Sent false
33935 9928 Acked Read 10 Done Done Sent false
33936 9929 Acked Read 9 Done Done Sent false
33937 9930 Acked Read 6 Done Done Sent false
33938 9931 Acked Read 5 Done Done Sent false
33939 9932 Acked Write 6 Done Done Sent false
33940 9933 Acked Flush 0 Done Done Sent false
33941 9934 Acked Write 1 Done Done Sent false
33942 9935 Acked Write 6 Done Done Sent false
33943 9936 Acked Read 8 Done Done Sent false
33944 9937 Acked Read 10 Done Done Sent false
33945 9938 Acked Read 3 Done Done Sent false
33946 9939 Acked Write 2 Done Done Sent false
33947 9940 Acked Read 10 Done Done Sent false
33948 9941 Acked Read 2 Done Done Sent false
33949 9942 Acked Read 6 Done Done Sent false
33950 9943 Acked Flush 0 Done Done Sent false
33951 9944 Acked Read 2 Done Done Sent false
33952 9945 Acked Read 3 Done Done Sent false
33953 9946 Acked Read 6 Done Done Sent false
33954 9947 Acked Read 9 Done Done Sent false
33955 9948 Acked Read 3 Done Done Sent false
33956 9949 Acked Write 3 Done Done Sent false
33957 9950 Acked Write 5 Done Done Sent false
33958 9951 Acked Write 2 Done Done Sent false
33959 9952 Acked Read 10 Done Done Sent false
33960 9953 Acked Read 8 Done Done Sent false
33961 9954 Acked Write 3 Done Done Sent false
33962 9955 Acked Read 4 Done Done Sent false
33963 9956 Acked Write 8 Done Done Sent false
33964 9957 Acked Write 5 Done Done Sent false
33965 9958 Acked Read 2 Done Done Sent false
33966 9959 Acked Write 1 Done Done Sent false
33967 9960 Acked Write 5 Done Done Sent false
33968 9961 Acked Read 1 Done Done Sent false
33969 9962 Acked Write 4 Done Done Sent false
33970 9963 Acked Flush 0 Done Done Sent false
33971 9964 Acked Read 4 Done Done Sent false
33972 9965 Acked Read 1 Done Done Sent false
33973 9966 Acked Read 9 Done Done Sent false
33974 9967 Acked Write 6 Done Done Sent false
33975 9968 Acked Write 3 Done Done Sent false
33976 9969 Acked Read 4 Done Done Sent false
33977 9970 Acked Write 3 Done Done Sent false
33978 9971 Acked Read 7 Done Done Sent false
33979 9972 Acked Read 2 Done Done Sent false
33980 9973 Acked Read 7 Done Done Sent false
33981 9974 Acked Write 6 Done Done Sent false
33982 9975 Acked Write 5 Done Done Sent false
33983 9976 Acked Read 5 Done Done Sent false
33984 9977 Acked Write 3 Done Done Sent false
33985 9978 Acked Read 8 Done Done Sent false
33986 9979 Acked Read 10 Done Done Sent false
33987 9980 Acked Write 1 Done Done Sent false
33988 9981 Acked Write 8 Done Done Sent false
33989 9982 Acked Read 10 Done Done Sent false
33990 9983 Acked Read 7 Done Done Sent false
33991 9984 Acked Read 5 Done Done Sent false
33992 9985 Acked Flush 0 Done Done Sent false
33993 9986 Acked Write 3 Done Done Sent false
33994 9987 Acked Read 2 Done Done Sent false
33995 9988 Acked Read 4 Done Done Sent false
33996 9989 Acked Flush 0 Done Done Sent false
33997 9990 Acked Read 1 Done Done Sent false
33998 9991 Acked Read 4 Done Done Sent false
33999 9992 Acked Write 5 Done Done Sent false
34000 9993 Acked Read 10 Done Done Sent false
34001 9994 Acked Flush 0 Done Done Sent false
34002 9995 Acked Write 7 Done Done Sent false
34003 9996 Acked Write 5 Done Done Sent false
34004 9997 Acked Flush 0 Done Done Sent false
34005 9998 Acked Write 3 Done Done Sent false
34006 9999 Acked Write 5 Done Done Sent false
34007 10000 Acked Read 7 Done Done Sent false
34008 10001 Acked Write 2 Done Done Sent false
34009 10002 Acked Read 2 Done Done Sent false
34010 10003 Acked Read 2 Done Done Sent false
34011 10004 Acked Write 9 Done Done Sent false
34012 10005 Acked Read 9 Done Done Sent false
34013 10006 Acked Read 2 Done Done Sent false
34014 10007 Acked Write 3 Done Done Sent false
34015 10008 Acked Write 7 Done Done Sent false
34016 10009 Acked Write 4 Done Done Sent false
34017 10010 Acked Write 3 Done Done Sent false
34018 10011 Acked Flush 0 Done Done Sent false
34019 10012 Acked Read 3 Done Done Sent false
34020 10013 Acked Write 3 Done Done Sent false
34021 10014 Acked Write 1 Done Done Sent false
34022 10015 Acked Read 4 Done Done Sent false
34023 10016 Acked Read 6 Done Done Sent false
34024 10017 Acked Write 7 Done Done Sent false
34025 10018 Acked Write 9 Done Done Sent false
34026 10019 Acked Write 3 Done Done Sent false
34027 10020 Acked Write 7 Done Done Sent false
34028 10021 Acked Write 7 Done Done Sent false
34029 10022 Acked Read 8 Done Done Sent false
34030 10023 Acked Flush 0 Done Done Sent false
34031 10024 Acked Read 8 Done Done Sent false
34032 10025 Acked Write 9 Done Done Sent false
34033 10026 Acked Read 10 Done Done Sent false
34034 10027 Acked Write 6 Done Done Sent false
34035 10028 Acked Write 4 Done Done Sent false
34036 10029 Acked Read 9 Done Done Sent false
34037 10030 Acked Write 6 Done Done Sent false
34038 10031 Acked Read 7 Done Done Sent false
34039 10032 Acked Read 2 Done Done Sent false
34040 10033 Acked Write 9 Done Done Sent false
34041 10034 Acked Write 5 Done Done Sent false
34042 10035 Acked Write 5 Done Done Sent false
34043 10036 Acked Read 2 Done Done Sent false
34044 10037 Acked Flush 0 Done Done Sent false
34045 10038 Acked Read 8 Done Done Sent false
34046 10039 Acked Write 3 Done Done Sent false
34047 10040 Acked Write 10 Done Done Sent false
34048 10041 Acked Read 8 Done Done Sent false
34049 10042 Acked Flush 0 Done Done Sent false
34050 10043 Acked Flush 0 Done Done Sent false
34051 10044 Acked Flush 0 Done Done Sent false
34052 10045 Acked Flush 0 Done Done Sent false
34053 10046 Acked Write 3 Done Done Sent false
34054 10047 Acked Read 6 Done Done Sent false
34055 10048 Acked Read 10 Done Done Sent false
34056 10049 Acked Write 6 Done Done Sent false
34057 10050 Acked Read 5 Done Done Sent false
34058 10051 Acked Write 7 Done Done Sent false
34059 10052 Acked Read 9 Done Done Sent false
34060 10053 Acked Write 8 Done Done Sent false
34061 10054 Acked Write 4 Done Done Sent false
34062 10055 Acked Read 3 Done Done Sent false
34063 10056 Acked Write 7 Done Done Sent false
34064 STATES DS:0 DS:1 DS:2 TOTAL
34065 Sent 0 0 301 301
34066 Done 301 301 0 602
34067 Skipped 0 0 0 0
34068 Error 0 0 0 0
34069 Last Flush: Some(JobId(10045)) Some(JobId(10045)) Some(JobId(9755))
34070 Downstairs last five completed: 9755 9754 9753 9752 9751
34071 Upstairs last five completed: 10056 10055 10054 10053 10052
34072 CLIENT: Up:0 ds:301 act:2
340732025-01-28T22:01:45.995ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
340742025-01-28T22:01:46.997ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
340752025-01-28T22:01:47.999ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
340762025-01-28T22:01:49.001ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
340772025-01-28T22:01:49.094ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
340782025-01-28T22:01:49.095ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54805) = io task client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
340792025-01-28T22:01:49.096ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
340802025-01-28T22:01:49.096ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
340812025-01-28T22:01:49.096ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
340822025-01-28T22:01:49.096ZINFOcrucible: send last flush ID to this DS: Some(JobId(9755)) = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
340832025-01-28T22:01:49.096ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
340842025-01-28T22:01:49.096ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9755)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
340852025-01-28T22:01:49.096ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9755)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
34086 ----------------------------------------------------------------
34087 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:11
34088 States: Active Active Active
34089 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34090 10046 Acked Write 3 Done Done Done true
34091 10047 Acked Read 6 Done Done Done true
34092 10048 Acked Read 10 Done Done Done true
34093 10049 Acked Write 6 Done Done Done true
34094 10050 Acked Read 5 Done Done Done true
34095 10051 Acked Write 7 Done Done Done true
34096 10052 Acked Read 9 Done Done Done true
34097 10053 Acked Write 8 Done Done Done true
34098 10054 Acked Write 4 Done Done Done true
34099 10055 Acked Read 3 Done Done Done true
34100 10056 Acked Write 7 Done Done Done true
34101 STATES DS:0 DS:1 DS:2 TOTAL
34102 Sent 0 0 0 0
34103 Done 11 11 11 33
34104 Skipped 0 0 0 0
34105 Error 0 0 0 0
34106 Last Flush: Some(JobId(10045)) Some(JobId(10045)) Some(JobId(10045))
34107 Downstairs last five completed: 10045 10044 10043 10042 10041
34108 Upstairs last five completed: 10056 10055 10054 10053 10052
34109 CLIENT: Up:0 ds:11 act:3
34110 ----------------------------------------------------------------
34111 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
34112 Downstairs last five completed: 10057 10056 10055 10054 10053
34113 Upstairs last five completed: 10057 10056 10055 10054 10053
34114 CLIENT: Up:0 ds:0 act:3
34115 Replay: All jobs finished
34116 Main task has work to do, go find it
34117 got dsc Stop(1)
34118 stop 1
34119 Jan 28 22:01:53.343 INFO [8820] Got stop action so:false kr:false
34120 request completed, latency_us: 410, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 0c555df3-1e6f-4edd-ba73-82fbed943757, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
34121 Jan 28 22:01:53.344 INFO request completed, latency_us: 102, response_code: 200, uri: /state/cid/1, method: GET, req_id: 555ea4ab-fa93-4f31-b7f4-c0be226dfd55, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
341222025-01-28T22:01:53.344ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
341232025-01-28T22:01:53.344ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
341242025-01-28T22:01:53.344ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
341252025-01-28T22:01:53.344ZINFOcrucible: client task is exiting = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
34126 [8820] Exited with: ExitStatus(unix_wait_status(9))
34127 [8820][1] reports Exit
34128 Jan 28 22:01:57.347 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4705bd6f-ab2b-44c6-89ae-d3f6ff0dafdf, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
34129 001/300 Read block 313 len 16384
34130 002/300 Read block 2196 len 20480
34131 003/300 Write block 2367 len 16384 data: 4 7 7 9
34132 004/300 Write block 3883 len 16384 data: 7 9 6 5
34133 005/300 Flush
34134 006/300 Flush
34135 007/300 Read block 1969 len 8192
34136 008/300 Read block 1009 len 24576
34137 009/300 Read block 2279 len 24576
34138 010/300 Read block 2801 len 20480
34139 011/300 Read block 242 len 12288
34140 012/300 Read block 4941 len 32768
34141 013/300 Read block 4065 len 32768
34142 014/300 Read block 2873 len 28672
34143 015/300 Read block 1076 len 32768
34144 016/300 Write block 2468 len 24576 data: 7 5 5 5 5 6
34145 017/300 Read block 3295 len 12288
34146 018/300 Write block 3271 len 20480 data: 4 4 4 4 8
34147 019/300 Write block 2233 len 28672 data: 5 4 4 4 5 5 6
34148 020/300 Write block 2676 len 24576 data: 7 7 9 8 9 8
34149 021/300 Read block 4825 len 32768
34150 022/300 Write block 3060 len 24576 data: 7 7 6 6 5 6
34151 023/300 Write block 4236 len 16384 data: 4 2 2 2
34152 024/300 Write block 2694 len 40960 data: 7 6 7 8 6 4 3 5 6 6
34153 025/300 Read block 1947 len 20480
34154 026/300 Flush
34155 027/300 Read block 3043 len 40960
34156 028/300 Read block 2645 len 4096
34157 029/300 Read block 1073 len 28672
34158 030/300 Write block 1604 len 20480 data: 7 6 4 3 2
34159 031/300 Read block 4198 len 24576
34160 032/300 Flush
34161 033/300 Flush
34162 034/300 Write block 2574 len 12288 data: 5 6 4
34163 035/300 Read block 3266 len 8192
34164 036/300 Write block 483 len 20480 data: 9 7 7 8 6
34165 037/300 Read block 4563 len 28672
34166 038/300 Read block 2250 len 24576
34167 039/300 Read block 2292 len 16384
34168 040/300 Read block 3448 len 4096
34169 041/300 Write block 934 len 28672 data: 6 7 9 9 10 11 10
34170 042/300 Read block 4341 len 20480
34171 043/300 Read block 4757 len 12288
34172 044/300 Read block 1580 len 4096
34173 045/300 Read block 3331 len 20480
34174 046/300 Read block 4964 len 36864
34175 047/300 Read block 2841 len 20480
34176 048/300 Read block 3511 len 12288
34177 049/300 Read block 4771 len 8192
34178 050/300 Write block 4007 len 36864 data: 7 9 9 9 7 6 6 5 5
34179 051/300 Write block 1188 len 32768 data: 7 6 7 7 6 7 7 5
34180 052/300 Flush
34181 053/300 Flush
34182 054/300 Write block 1477 len 32768 data: 3 3 4 5 8 9 8 8
34183 055/300 Read block 4905 len 36864
34184 056/300 Read block 3371 len 4096
34185 057/300 Read block 462 len 36864
34186 058/300 Read block 3774 len 4096
34187 059/300 Read block 313 len 24576
34188 060/300 Write block 916 len 20480 data: 7 7 8 6 3
34189 061/300 Flush
34190 062/300 Write block 1639 len 24576 data: 6 7 9 9 7 7
34191 063/300 Flush
34192 064/300 Read block 4935 len 16384
34193 065/300 Read block 1015 len 36864
34194 066/300 Read block 4739 len 28672
34195 067/300 Read block 4710 len 24576
34196 068/300 Flush
34197 069/300 Read block 754 len 20480
34198 070/300 Write block 2881 len 4096 data: 6
34199 071/300 Write block 1224 len 24576 data: 6 6 6 3 4 4
34200 072/300 Read block 2557 len 12288
34201 073/300 Write block 4352 len 16384 data: 8 8 7 7
34202 074/300 Write block 4026 len 32768 data: 6 6 7 7 7 8 8 7
34203 075/300 Read block 515 len 40960
34204 076/300 Read block 3006 len 4096
34205 077/300 Read block 4913 len 16384
34206 078/300 Flush
34207 079/300 Write block 652 len 28672 data: 11 11 8 7 6 7 6
34208 080/300 Write block 4177 len 12288 data: 9 10 11
34209 081/300 Read block 3542 len 8192
34210 082/300 Write block 1414 len 12288 data: 10 8 8
34211 083/300 Flush
34212 084/300 Read block 512 len 24576
34213 085/300 Flush
34214 086/300 Write block 4291 len 28672 data: 4 7 6 6 6 6 8
34215 087/300 Write block 3730 len 20480 data: 11 8 9 9 7
34216 088/300 Flush
34217 089/300 Flush
34218 090/300 Write block 4204 len 16384 data: 8 8 6 5
34219 091/300 Write block 1086 len 28672 data: 5 4 4 5 5 5 4
34220 092/300 Read block 2550 len 36864
34221 093/300 Write block 3882 len 4096 data: 7
34222 094/300 Write block 1111 len 32768 data: 7 7 5 4 4 5 6 6
34223 095/300 Write block 2998 len 24576 data: 6 4 6 7 7 6
34224 096/300 Write block 1237 len 40960 data: 4 4 3 3 2 3 5 5 5 5
34225 097/300 Read block 177 len 4096
34226 098/300 Read block 4118 len 36864
34227 099/300 Flush
34228 100/300 Write block 585 len 4096 data: 5
34229 101/300 Read block 4365 len 32768
34230 102/300 Write block 3150 len 8192 data: 6 5
34231 103/300 Read block 4566 len 20480
34232 104/300 Read block 2247 len 40960
34233 105/300 Write block 2977 len 24576 data: 5 6 7 7 7 6
34234 106/300 Read block 1152 len 12288
34235 107/300 Read block 247 len 36864
34236 108/300 Flush
34237 109/300 Read block 2333 len 32768
34238 110/300 Write block 1269 len 8192 data: 6 6
34239 111/300 Write block 3175 len 24576 data: 6 7 5 5 5 5
34240 112/300 Read block 4987 len 12288
34241 113/300 Write block 1279 len 8192 data: 6 7
34242 114/300 Read block 1478 len 36864
34243 115/300 Write block 3465 len 16384 data: 2 3 3 3
34244 116/300 Write block 4247 len 16384 data: 4 4 5 6
34245 117/300 Write block 1480 len 20480 data: 6 9 10 9 9
34246 118/300 Read block 4706 len 36864
34247 119/300 Write block 2088 len 8192 data: 7 7
34248 120/300 Read block 118 len 36864
34249 121/300 Read block 1539 len 24576
34250 122/300 Read block 4069 len 16384
34251 123/300 Read block 2168 len 16384
34252 124/300 Flush
34253 125/300 Read block 914 len 8192
34254 126/300 Read block 4374 len 36864
34255 127/300 Flush
34256 128/300 Write block 1724 len 32768 data: 5 4 3 3 3 3 4 5
34257 129/300 Flush
34258 130/300 Read block 3938 len 16384
34259 131/300 Read block 1323 len 40960
34260 132/300 Write block 1270 len 32768 data: 7 6 7 7 6 6 5 5
34261 133/300 Write block 3507 len 8192 data: 5 4
34262 134/300 Write block 2106 len 36864 data: 4 5 5 5 5 4 2 3 4
34263 135/300 Read block 3110 len 40960
34264 136/300 Flush
34265 137/300 Write block 895 len 40960 data: 6 5 4 5 4 6 8 8 8 8
34266 138/300 Read block 2312 len 16384
34267 139/300 Read block 2171 len 28672
34268 140/300 Write block 519 len 36864 data: 7 7 7 6 7 8 7 7 9
34269 141/300 Read block 353 len 28672
34270 142/300 Read block 916 len 16384
34271 143/300 Read block 3039 len 40960
34272 144/300 Read block 4703 len 8192
34273 145/300 Write block 4226 len 32768 data: 5 5 7 7 7 7 8 6
34274 146/300 Read block 87 len 16384
34275 147/300 Read block 666 len 16384
34276 148/300 Read block 1688 len 32768
34277 149/300 Flush
34278 150/300 Flush
34279 151/300 Flush
34280 152/300 Read block 1222 len 4096
34281 153/300 Write block 2838 len 4096 data: 2
34282 154/300 Write block 69 len 32768 data: 7 7 6 8 8 9 8 8
34283 155/300 Read block 3710 len 12288
34284 156/300 Read block 3739 len 24576
34285 157/300 Write block 3662 len 16384 data: 8 9 9 7
34286 158/300 Read block 685 len 36864
34287 159/300 Flush
34288 160/300 Read block 4031 len 24576
34289 161/300 Read block 4853 len 24576
34290 162/300 Write block 3504 len 24576 data: 4 5 5 6 5 4
34291 163/300 Read block 3343 len 16384
34292 164/300 Read block 2081 len 20480
34293 165/300 Read block 4430 len 12288
34294 166/300 Read block 1523 len 20480
34295 167/300 Write block 849 len 32768 data: 6 4 5 5 6 3 5 5
34296 168/300 Write block 3227 len 40960 data: 7 7 7 6 5 5 2 3 2 4
34297 169/300 Read block 4362 len 24576
34298 170/300 Read block 4899 len 12288
34299 171/300 Write block 1777 len 4096 data: 7
34300 172/300 Read block 1302 len 40960
34301 173/300 Write block 1818 len 28672 data: 6 7 7 9 6 6 8
34302 174/300 Write block 1557 len 20480 data: 6 5 5 9 8
34303 175/300 Write block 103 len 24576 data: 5 5 5 4 4 4
34304 176/300 Read block 4458 len 4096
34305 177/300 Read block 3189 len 40960
34306 178/300 Read block 3998 len 4096
34307 179/300 Write block 2730 len 32768 data: 10 9 7 7 8 8 5 5
34308 180/300 Read block 3709 len 20480
34309 181/300 Read block 1132 len 24576
34310 182/300 Write block 2343 len 36864 data: 7 8 6 5 8 6 6 8 7
34311 183/300 Write block 3833 len 36864 data: 3 4 6 5 4 3 6 6 5
34312 184/300 Read block 4141 len 28672
34313 185/300 Read block 597 len 12288
34314 186/300 Read block 2890 len 28672
34315 187/300 Read block 3533 len 40960
34316 188/300 Read block 2676 len 4096
34317 189/300 Write block 2049 len 28672 data: 6 5 5 6 5 6 6
34318 190/300 Read block 2160 len 12288
34319 191/300 Write block 3165 len 12288 data: 6 6 6
34320 192/300 Write block 3792 len 8192 data: 6 7
34321 193/300 Read block 505 len 16384
34322 194/300 Read block 3834 len 32768
34323 195/300 Write block 1615 len 16384 data: 6 5 7 7
34324 196/300 Write block 3185 len 28672 data: 6 5 5 5 5 6 5
34325 197/300 Read block 1098 len 8192
34326 198/300 Write block 889 len 12288 data: 8 5 7
34327 199/300 Read block 1991 len 36864
34328 200/300 Read block 2893 len 20480
34329 201/300 Read block 4238 len 36864
34330 202/300 Read block 3734 len 36864
34331 203/300 Read block 4334 len 12288
34332 204/300 Read block 751 len 28672
34333 205/300 Write block 4515 len 8192 data: 6 5
34334 206/300 Write block 3745 len 24576 data: 4 4 5 4 4 6
34335 207/300 Write block 2078 len 32768 data: 6 5 7 9 9 9 8 6
34336 208/300 Read block 4263 len 28672
34337 209/300 Read block 1548 len 32768
34338 210/300 Write block 1017 len 24576 data: 10 10 7 6 10 9
34339 211/300 Write block 4252 len 32768 data: 7 10 8 7 7 5 4 5
34340 212/300 Write block 547 len 36864 data: 12 11 10 9 8 7 7 5 6
34341 213/300 Write block 1268 len 4096 data: 6
34342 214/300 Write block 4319 len 12288 data: 7 6 6
34343 215/300 Read block 1719 len 24576
34344 216/300 Write block 2784 len 32768 data: 5 4 3 3 3 7 7 6
34345 217/300 Read block 3714 len 8192
34346 218/300 Read block 3926 len 40960
34347 219/300 Read block 2531 len 12288
34348 220/300 Write block 387 len 36864 data: 5 6 5 4 4 4 5 5 5
34349 221/300 Read block 1761 len 40960
34350 222/300 Read block 1397 len 36864
34351 223/300 Read block 648 len 4096
34352 224/300 Flush
34353 225/300 Read block 1926 len 8192
34354 226/300 Write block 573 len 16384 data: 3 4 3 3
34355 227/300 Write block 4894 len 12288 data: 7 8 5
34356 228/300 Read block 3450 len 32768
34357 229/300 Flush
34358 230/300 Read block 1235 len 24576
34359 231/300 Write block 381 len 16384 data: 3 5 5 5
34360 232/300 Read block 2799 len 40960
34361 233/300 Write block 4359 len 32768 data: 10 10 8 7 8 10 5 4
34362 234/300 Read block 4341 len 32768
34363 235/300 Write block 4417 len 4096 data: 4
34364 236/300 Write block 4407 len 12288 data: 4 5 5
34365 237/300 Write block 2881 len 40960 data: 7 6 6 6 7 7 5 4 4 3
34366 238/300 Write block 2226 len 40960 data: 7 6 8 6 5 5 4 6 5 5
34367 239/300 Write block 1361 len 12288 data: 6 6 7
34368 240/300 Read block 433 len 36864
34369 241/300 Write block 4993 len 28672 data: 4 4 4 3 3 4 2
34370 242/300 Read block 2192 len 4096
34371 243/300 Read block 372 len 16384
34372 244/300 Read block 3438 len 40960
34373 245/300 Read block 2092 len 8192
34374 246/300 Write block 690 len 20480 data: 8 9 7 8 10
34375 247/300 Flush
34376 248/300 Write block 3499 len 12288 data: 3 4 5
34377 249/300 Read block 4069 len 4096
34378 250/300 Flush
34379 251/300 Write block 3879 len 40960 data: 8 8 6 8 8 10 7 6 5 6
34380 252/300 Flush
34381 253/300 Write block 1053 len 20480 data: 10 8 7 6 7
34382 254/300 Read block 1423 len 8192
34383 255/300 Read block 1812 len 4096
34384 256/300 Write block 1373 len 40960 data: 4 4 4 3 3 4 4 4 5 4
34385 257/300 Write block 4295 len 28672 data: 7 7 9 9 8 7 6
34386 258/300 Write block 19 len 16384 data: 4 4 5 5
34387 259/300 Write block 4455 len 28672 data: 4 4 5 5 4 4 4
34388 260/300 Read block 2987 len 24576
34389 261/300 Write block 4392 len 12288 data: 2 3 4
34390 262/300 Write block 788 len 40960 data: 6 6 6 6 6 6 5 4 4 4
34391 263/300 Read block 2588 len 32768
34392 264/300 Write block 1169 len 40960 data: 6 4 4 6 7 6 5 6 7 10
34393 265/300 Read block 4417 len 4096
34394 266/300 Write block 4698 len 4096 data: 6
34395 267/300 Read block 108 len 32768
34396 268/300 Read block 1638 len 24576
34397 269/300 Read block 4141 len 36864
34398 270/300 Read block 4363 len 24576
34399 271/300 Read block 887 len 28672
34400 272/300 Write block 3185 len 4096 data: 7
34401 273/300 Read block 2853 len 40960
34402 274/300 Write block 1097 len 36864 data: 6 6 6 8 8 6 6 6 9
34403 275/300 Read block 2766 len 8192
34404 276/300 Write block 4490 len 24576 data: 9 8 7 4 4 4
34405 277/300 Read block 63 len 4096
34406 278/300 Read block 1306 len 28672
34407 279/300 Read block 2807 len 16384
34408 280/300 Write block 1432 len 20480 data: 11 11 7 7 8
34409 281/300 Read block 4947 len 28672
34410 282/300 Write block 102 len 28672 data: 4 6 6 6 5 5 5
34411 283/300 Flush
34412 284/300 Read block 4117 len 16384
34413 285/300 Read block 3532 len 8192
34414 286/300 Read block 3661 len 12288
34415 287/300 Flush
34416 288/300 Read block 1454 len 12288
34417 289/300 Read block 303 len 4096
34418 290/300 Read block 2598 len 24576
34419 291/300 Write block 2462 len 12288 data: 6 5 6
34420 292/300 Write block 741 len 16384 data: 7 6 6 5
34421 293/300 Read block 761 len 32768
34422 294/300 Read block 4242 len 12288
34423 295/300 Read block 2525 len 24576
34424 296/300 Write block 3041 len 32768 data: 4 5 4 6 7 8 8 10
34425 297/300 Read block 1569 len 20480
34426 298/300 Write block 1042 len 20480 data: 8 9 10 9 11
34427 299/300 Read block 4550 len 24576
34428 300/300 Write block 1367 len 32768 data: 5 6 6 6 5 4 5 5
34429 301/300 Write block 4623 len 20480 data: 7 6 6 5 7
34430 Main task has work to do, go find it
34431 got dsc Start(1)
34432 start 1
34433 Jan 28 22:01:57.593 INFO request completed, latency_us: 283, response_code: 204, uri: /start/cid/1, method: POST, req_id: 36269b9a-c145-4d4f-a4b3-6b3e78703ed6, remote_addr: 127.0.0.1:36665, local_addr: 127.0.0.1:9998
34434 Replay: started 1, returned:Ok(())
34435 ----------------------------------------------------------------
34436 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34437 States: Active Offline Active
34438 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34439 10058 Acked Read 4 Done Sent Done false
34440 10059 Acked Read 5 Done Sent Done false
34441 10060 Acked Write 4 Done Sent Done false
34442 10061 Acked Write 4 Done Sent Done false
34443 10062 Acked Flush 0 Done Sent Done false
34444 10063 Acked Flush 0 Done Sent Done false
34445 10064 Acked Read 2 Done Sent Done false
34446 10065 Acked Read 6 Done Sent Done false
34447 10066 Acked Read 6 Done Sent Done false
34448 10067 Acked Read 5 Done Sent Done false
34449 10068 Acked Read 3 Done Sent Done false
34450 10069 Acked Read 8 Done Sent Done false
34451 10070 Acked Read 8 Done Sent Done false
34452 10071 Acked Read 7 Done Sent Done false
34453 10072 Acked Read 8 Done Sent Done false
34454 10073 Acked Write 6 Done Sent Done false
34455 10074 Acked Read 3 Done Sent Done false
34456 10075 Acked Write 5 Done Sent Done false
34457 10076 Acked Write 7 Done Sent Done false
34458 10077 Acked Write 6 Done Sent Done false
34459 10078 Acked Read 8 Done Sent Done false
34460 10079 Acked Write 6 Done Sent Done false
34461 10080 Acked Write 4 Done Sent Done false
34462 10081 Acked Write 10 Done Sent Done false
34463 10082 Acked Read 5 Done Sent Done false
34464 10083 Acked Flush 0 Done Sent Done false
34465 10084 Acked Read 10 Done Sent Done false
34466 10085 Acked Read 1 Done Sent Done false
34467 10086 Acked Read 7 Done Sent Done false
34468 10087 Acked Write 5 Done Sent Done false
34469 10088 Acked Read 6 Done Sent Done false
34470 10089 Acked Flush 0 Done Sent Done false
34471 10090 Acked Flush 0 Done Sent Done false
34472 10091 Acked Write 3 Done Sent Done false
34473 10092 Acked Read 2 Done Sent Done false
34474 10093 Acked Write 5 Done Sent Done false
34475 10094 Acked Read 7 Done Sent Done false
34476 10095 Acked Read 6 Done Sent Done false
34477 10096 Acked Read 4 Done Sent Done false
34478 10097 Acked Read 1 Done Sent Done false
34479 10098 Acked Write 7 Done Sent Done false
34480 10099 Acked Read 5 Done Sent Done false
34481 10100 Acked Read 3 Done Sent Done false
34482 10101 Acked Read 1 Done Sent Done false
34483 10102 Acked Read 5 Done Sent Done false
34484 10103 Acked Read 9 Done Sent Done false
34485 10104 Acked Read 5 Done Sent Done false
34486 10105 Acked Read 3 Done Sent Done false
34487 10106 Acked Read 2 Done Sent Done false
34488 10107 Acked Write 9 Done Sent Done false
34489 10108 Acked Write 8 Done Sent Done false
34490 10109 Acked Flush 0 Done Sent Done false
34491 10110 Acked Flush 0 Done Sent Done false
34492 10111 Acked Write 8 Done Sent Done false
34493 10112 Acked Read 9 Done Sent Done false
34494 10113 Acked Read 1 Done Sent Done false
34495 10114 Acked Read 9 Done Sent Done false
34496 10115 Acked Read 1 Done Sent Done false
34497 10116 Acked Read 6 Done Sent Done false
34498 10117 Acked Write 5 Done Sent Done false
34499 10118 Acked Flush 0 Done Sent Done false
34500 10119 Acked Write 6 Done Sent Done false
34501 10120 Acked Flush 0 Done Sent Done false
34502 10121 Acked Read 4 Done Sent Done false
34503 10122 Acked Read 9 Done Sent Done false
34504 10123 Acked Read 7 Done Sent Done false
34505 10124 Acked Read 6 Done Sent Done false
34506 10125 Acked Flush 0 Done Sent Done false
34507 10126 Acked Read 5 Done Sent Done false
34508 10127 Acked Write 1 Done Sent Done false
34509 10128 Acked Write 6 Done Sent Done false
34510 10129 Acked Read 3 Done Sent Done false
34511 10130 Acked Write 4 Done Sent Done false
34512 10131 Acked Write 8 Done Sent Done false
34513 10132 Acked Read 10 Done Sent Done false
34514 10133 Acked Read 1 Done Sent Done false
34515 10134 Acked Read 4 Done Sent Done false
34516 10135 Acked Flush 0 Done Sent Done false
34517 10136 Acked Write 7 Done Sent Done false
34518 10137 Acked Write 3 Done Sent Done false
34519 10138 Acked Read 2 Done Sent Done false
34520 10139 Acked Write 3 Done Sent Done false
34521 10140 Acked Flush 0 Done Sent Done false
34522 10141 Acked Read 6 Done Sent Done false
34523 10142 Acked Flush 0 Done Sent Done false
34524 10143 Acked Write 7 Done Sent Done false
34525 10144 Acked Write 5 Done Sent Done false
34526 10145 Acked Flush 0 Done Sent Done false
34527 10146 Acked Flush 0 Done Sent Done false
34528 10147 Acked Write 4 Done Sent Done false
34529 10148 Acked Write 7 Done Sent Done false
34530 10149 Acked Read 9 Done Sent Done false
34531 10150 Acked Write 1 Done Sent Done false
34532 10151 Acked Write 8 Done Sent Done false
34533 10152 Acked Write 6 Done Sent Done false
34534 10153 Acked Write 10 Done Sent Done false
34535 10154 Acked Read 1 Done Sent Done false
34536 10155 Acked Read 9 Done Sent Done false
34537 10156 Acked Flush 0 Done Sent Done false
34538 10157 Acked Write 1 Done Sent Done false
34539 10158 Acked Read 8 Done Sent Done false
34540 10159 Acked Write 2 Done Sent Done false
34541 10160 Acked Read 5 Done Sent Done false
34542 10161 Acked Read 10 Done Sent Done false
34543 10162 Acked Write 6 Done Sent Done false
34544 10163 Acked Read 3 Done Sent Done false
34545 10164 Acked Read 9 Done Sent Done false
34546 10165 Acked Flush 0 Done Sent Done false
34547 10166 Acked Read 8 Done Sent Done false
34548 10167 Acked Write 2 Done Sent Done false
34549 10168 Acked Write 6 Done Sent Done false
34550 10169 Acked Read 3 Done Sent Done false
34551 10170 Acked Write 2 Done Sent Done false
34552 10171 Acked Read 9 Done Sent Done false
34553 10172 Acked Write 4 Done Sent Done false
34554 10173 Acked Write 4 Done Sent Done false
34555 10174 Acked Write 5 Done Sent Done false
34556 10175 Acked Read 9 Done Sent Done false
34557 10176 Acked Write 2 Done Sent Done false
34558 10177 Acked Read 9 Done Sent Done false
34559 10178 Acked Read 6 Done Sent Done false
34560 10179 Acked Read 4 Done Sent Done false
34561 10180 Acked Read 4 Done Sent Done false
34562 10181 Acked Flush 0 Done Sent Done false
34563 10182 Acked Read 2 Done Sent Done false
34564 10183 Acked Read 9 Done Sent Done false
34565 10184 Acked Flush 0 Done Sent Done false
34566 10185 Acked Write 8 Done Sent Done false
34567 10186 Acked Flush 0 Done Sent Done false
34568 10187 Acked Read 4 Done Sent Done false
34569 10188 Acked Read 10 Done Sent Done false
34570 10189 Acked Write 8 Done Sent Done false
34571 10190 Acked Write 2 Done Sent Done false
34572 10191 Acked Write 9 Done Sent Done false
34573 10192 Acked Read 10 Done Sent Done false
34574 10193 Acked Flush 0 Done Sent Done false
34575 10194 Acked Write 10 Done Sent Done false
34576 10195 Acked Read 4 Done Sent Done false
34577 10196 Acked Read 7 Done Sent Done false
34578 10197 Acked Write 9 Done Sent Done false
34579 10198 Acked Read 7 Done Sent Done false
34580 10199 Acked Read 4 Done Sent Done false
34581 10200 Acked Read 10 Done Sent Done false
34582 10201 Acked Read 2 Done Sent Done false
34583 10202 Acked Write 8 Done Sent Done false
34584 10203 Acked Read 4 Done Sent Done false
34585 10204 Acked Read 4 Done Sent Done false
34586 10205 Acked Read 8 Done Sent Done false
34587 10206 Acked Flush 0 Done Sent Done false
34588 10207 Acked Flush 0 Done Sent Done false
34589 10208 Acked Flush 0 Done Sent Done false
34590 10209 Acked Read 1 Done Sent Done false
34591 10210 Acked Write 1 Done Sent Done false
34592 10211 Acked Write 8 Done Sent Done false
34593 10212 Acked Read 3 Done Sent Done false
34594 10213 Acked Read 6 Done Sent Done false
34595 10214 Acked Write 4 Done Sent Done false
34596 10215 Acked Read 9 Done Sent Done false
34597 10216 Acked Flush 0 Done Sent Done false
34598 10217 Acked Read 6 Done Sent Done false
34599 10218 Acked Read 6 Done Sent Done false
34600 10219 Acked Write 6 Done Sent Done false
34601 10220 Acked Read 4 Done Sent Done false
34602 10221 Acked Read 5 Done Sent Done false
34603 10222 Acked Read 3 Done Sent Done false
34604 10223 Acked Read 5 Done Sent Done false
34605 10224 Acked Write 8 Done Sent Done false
34606 10225 Acked Write 10 Done Sent Done false
34607 10226 Acked Read 6 Done Sent Done false
34608 10227 Acked Read 3 Done Sent Done false
34609 10228 Acked Write 1 Done Sent Done false
34610 10229 Acked Read 10 Done Sent Done false
34611 10230 Acked Write 7 Done Sent Done false
34612 10231 Acked Write 5 Done Sent Done false
34613 10232 Acked Write 6 Done Sent Done false
34614 10233 Acked Read 1 Done Sent Done false
34615 10234 Acked Read 10 Done Sent Done false
34616 10235 Acked Read 1 Done Sent Done false
34617 10236 Acked Write 8 Done Sent Done false
34618 10237 Acked Read 5 Done Sent Done false
34619 10238 Acked Read 6 Done Sent Done false
34620 10239 Acked Write 9 Done Sent Done false
34621 10240 Acked Write 9 Done Sent Done false
34622 10241 Acked Read 7 Done Sent Done false
34623 10242 Acked Read 3 Done Sent Done false
34624 10243 Acked Read 7 Done Sent Done false
34625 10244 Acked Read 10 Done Sent Done false
34626 10245 Acked Read 1 Done Sent Done false
34627 10246 Acked Write 7 Done Sent Done false
34628 10247 Acked Read 3 Done Sent Done false
34629 10248 Acked Write 3 Done Sent Done false
34630 10249 Acked Write 2 Done Sent Done false
34631 10250 Acked Read 4 Done Sent Done false
34632 10251 Acked Read 8 Done Sent Done false
34633 10252 Acked Write 4 Done Sent Done false
34634 10253 Acked Write 7 Done Sent Done false
34635 10254 Acked Read 2 Done Sent Done false
34636 10255 Acked Write 3 Done Sent Done false
34637 10256 Acked Read 9 Done Sent Done false
34638 10257 Acked Read 5 Done Sent Done false
34639 10258 Acked Read 9 Done Sent Done false
34640 10259 Acked Read 9 Done Sent Done false
34641 10260 Acked Read 3 Done Sent Done false
34642 10261 Acked Read 7 Done Sent Done false
34643 10262 Acked Write 2 Done Sent Done false
34644 10263 Acked Write 6 Done Sent Done false
34645 10264 Acked Write 8 Done Sent Done false
34646 10265 Acked Read 7 Done Sent Done false
34647 10266 Acked Read 8 Done Sent Done false
34648 10267 Acked Write 6 Done Sent Done false
34649 10268 Acked Write 8 Done Sent Done false
34650 10269 Acked Write 9 Done Sent Done false
34651 10270 Acked Write 1 Done Sent Done false
34652 10271 Acked Write 3 Done Sent Done false
34653 10272 Acked Read 6 Done Sent Done false
34654 10273 Acked Write 8 Done Sent Done false
34655 10274 Acked Read 2 Done Sent Done false
34656 10275 Acked Read 10 Done Sent Done false
34657 10276 Acked Read 3 Done Sent Done false
34658 10277 Acked Write 9 Done Sent Done false
34659 10278 Acked Read 10 Done Sent Done false
34660 10279 Acked Read 9 Done Sent Done false
34661 10280 Acked Read 1 Done Sent Done false
34662 10281 Acked Flush 0 Done Sent Done false
34663 10282 Acked Read 2 Done Sent Done false
34664 10283 Acked Write 4 Done Sent Done false
34665 10284 Acked Write 3 Done Sent Done false
34666 10285 Acked Read 8 Done Sent Done false
34667 10286 Acked Flush 0 Done Sent Done false
34668 10287 Acked Read 6 Done Sent Done false
34669 10288 Acked Write 4 Done Sent Done false
34670 10289 Acked Read 10 Done Sent Done false
34671 10290 Acked Write 8 Done Sent Done false
34672 10291 Acked Read 8 Done Sent Done false
34673 10292 Acked Write 1 Done Sent Done false
34674 10293 Acked Write 3 Done Sent Done false
34675 10294 Acked Write 10 Done Sent Done false
34676 10295 Acked Write 10 Done Sent Done false
34677 10296 Acked Write 3 Done Sent Done false
34678 10297 Acked Read 9 Done Sent Done false
34679 10298 Acked Write 7 Done Sent Done false
34680 10299 Acked Read 1 Done Sent Done false
34681 10300 Acked Read 4 Done Sent Done false
34682 10301 Acked Read 10 Done Sent Done false
34683 10302 Acked Read 2 Done Sent Done false
34684 10303 Acked Write 5 Done Sent Done false
34685 10304 Acked Flush 0 Done Sent Done false
34686 10305 Acked Write 3 Done Sent Done false
34687 10306 Acked Read 1 Done Sent Done false
34688 10307 Acked Flush 0 Done Sent Done false
34689 10308 Acked Write 10 Done Sent Done false
34690 10309 Acked Flush 0 Done Sent Done false
34691 10310 Acked Write 5 Done Sent Done false
34692 10311 Acked Read 2 Done Sent Done false
34693 10312 Acked Read 1 Done Sent Done false
34694 10313 Acked Write 10 Done Sent Done false
34695 10314 Acked Write 7 Done Sent Done false
34696 10315 Acked Write 4 Done Sent Done false
34697 10316 Acked Write 7 Done Sent Done false
34698 10317 Acked Read 6 Done Sent Done false
34699 10318 Acked Write 3 Done Sent Done false
34700 10319 Acked Write 10 Done Sent Done false
34701 10320 Acked Read 8 Done Sent Done false
34702 10321 Acked Write 10 Done Sent Done false
34703 10322 Acked Read 1 Done Sent Done false
34704 10323 Acked Write 1 Done Sent Done false
34705 10324 Acked Read 8 Done Sent Done false
34706 10325 Acked Read 6 Done Sent Done false
34707 10326 Acked Read 9 Done Sent Done false
34708 10327 Acked Read 6 Done Sent Done false
34709 10328 Acked Read 7 Done Sent Done false
34710 10329 Acked Write 1 Done Sent Done false
34711 10330 Acked Read 10 Done Sent Done false
34712 10331 Acked Write 9 Done Sent Done false
34713 10332 Acked Read 2 Done Sent Done false
34714 10333 Acked Write 6 Done Sent Done false
34715 10334 Acked Read 1 Done Sent Done false
34716 10335 Acked Read 7 Done Sent Done false
34717 10336 Acked Read 4 Done Sent Done false
34718 10337 Acked Write 5 Done Sent Done false
34719 10338 Acked Read 7 Done Sent Done false
34720 10339 Acked Write 7 Done Sent Done false
34721 10340 Acked Flush 0 Done Sent Done false
34722 10341 Acked Read 4 Done Sent Done false
34723 10342 Acked Read 2 Done Sent Done false
34724 10343 Acked Read 3 Done Sent Done false
34725 10344 Acked Flush 0 Done Sent Done false
34726 10345 Acked Read 3 Done Sent Done false
34727 10346 Acked Read 1 Done Sent Done false
34728 10347 Acked Read 6 Done Sent Done false
34729 10348 Acked Write 3 Done Sent Done false
34730 10349 Acked Write 4 Done Sent Done false
34731 10350 Acked Read 8 Done Sent Done false
34732 10351 Acked Read 3 Done Sent Done false
34733 10352 Acked Read 6 Done Sent Done false
34734 10353 Acked Write 8 Done Sent Done false
34735 10354 Acked Read 5 Done Sent Done false
34736 10355 Acked Write 5 Done Sent Done false
34737 10356 Acked Read 6 Done Sent Done false
34738 10357 Acked Write 8 Done Sent Done false
34739 10358 Acked Write 5 Done Sent Done false
34740 STATES DS:0 DS:1 DS:2 TOTAL
34741 Sent 0 301 0 301
34742 Done 301 0 301 602
34743 Skipped 0 0 0 0
34744 Error 0 0 0 0
34745 Last Flush: Some(JobId(10344)) Some(JobId(10057)) Some(JobId(10344))
34746 Downstairs last five completed: 10057 10056 10055 10054 10053
34747 Upstairs last five completed: 10358 10357 10356 10355 10354
34748 CLIENT: Up:0 ds:301 act:2
347492025-01-28T22:01:58.095ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
347502025-01-28T22:01:59.018ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
347512025-01-28T22:02:00.019ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
347522025-01-28T22:02:01.020ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
34753 [8820] got start action
34754 [8820] I am going to restart
34755 Starting downstairs at port 8820
34756 Make output file at "/tmp/dsc/downstairs-8820.txt"
34757 [8820][1] reports Starting
34758 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1233)
34759 [8820][1] reports Running
34760 ----------------------------------------------------------------
34761 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34762 States: Active Offline Active
34763 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34764 10058 Acked Read 4 Done Sent Done false
34765 10059 Acked Read 5 Done Sent Done false
34766 10060 Acked Write 4 Done Sent Done false
34767 10061 Acked Write 4 Done Sent Done false
34768 10062 Acked Flush 0 Done Sent Done false
34769 10063 Acked Flush 0 Done Sent Done false
34770 10064 Acked Read 2 Done Sent Done false
34771 10065 Acked Read 6 Done Sent Done false
34772 10066 Acked Read 6 Done Sent Done false
34773 10067 Acked Read 5 Done Sent Done false
34774 10068 Acked Read 3 Done Sent Done false
34775 10069 Acked Read 8 Done Sent Done false
34776 10070 Acked Read 8 Done Sent Done false
34777 10071 Acked Read 7 Done Sent Done false
34778 10072 Acked Read 8 Done Sent Done false
34779 10073 Acked Write 6 Done Sent Done false
34780 10074 Acked Read 3 Done Sent Done false
34781 10075 Acked Write 5 Done Sent Done false
34782 10076 Acked Write 7 Done Sent Done false
34783 10077 Acked Write 6 Done Sent Done false
34784 10078 Acked Read 8 Done Sent Done false
34785 10079 Acked Write 6 Done Sent Done false
34786 10080 Acked Write 4 Done Sent Done false
34787 10081 Acked Write 10 Done Sent Done false
34788 10082 Acked Read 5 Done Sent Done false
34789 10083 Acked Flush 0 Done Sent Done false
34790 10084 Acked Read 10 Done Sent Done false
34791 10085 Acked Read 1 Done Sent Done false
34792 10086 Acked Read 7 Done Sent Done false
34793 10087 Acked Write 5 Done Sent Done false
34794 10088 Acked Read 6 Done Sent Done false
34795 10089 Acked Flush 0 Done Sent Done false
34796 10090 Acked Flush 0 Done Sent Done false
34797 10091 Acked Write 3 Done Sent Done false
34798 10092 Acked Read 2 Done Sent Done false
34799 10093 Acked Write 5 Done Sent Done false
34800 10094 Acked Read 7 Done Sent Done false
34801 10095 Acked Read 6 Done Sent Done false
34802 10096 Acked Read 4 Done Sent Done false
34803 10097 Acked Read 1 Done Sent Done false
34804 10098 Acked Write 7 Done Sent Done false
34805 10099 Acked Read 5 Done Sent Done false
34806 10100 Acked Read 3 Done Sent Done false
34807 10101 Acked Read 1 Done Sent Done false
34808 10102 Acked Read 5 Done Sent Done false
34809 10103 Acked Read 9 Done Sent Done false
34810 10104 Acked Read 5 Done Sent Done false
34811 10105 Acked Read 3 Done Sent Done false
34812 10106 Acked Read 2 Done Sent Done false
34813 10107 Acked Write 9 Done Sent Done false
34814 10108 Acked Write 8 Done Sent Done false
34815 10109 Acked Flush 0 Done Sent Done false
34816 10110 Acked Flush 0 Done Sent Done false
34817 10111 Acked Write 8 Done Sent Done false
34818 10112 Acked Read 9 Done Sent Done false
34819 10113 Acked Read 1 Done Sent Done false
34820 10114 Acked Read 9 Done Sent Done false
34821 10115 Acked Read 1 Done Sent Done false
34822 10116 Acked Read 6 Done Sent Done false
34823 10117 Acked Write 5 Done Sent Done false
34824 10118 Acked Flush 0 Done Sent Done false
34825 10119 Acked Write 6 Done Sent Done false
34826 10120 Acked Flush 0 Done Sent Done false
34827 10121 Acked Read 4 Done Sent Done false
34828 10122 Acked Read 9 Done Sent Done false
34829 10123 Acked Read 7 Done Sent Done false
34830 10124 Acked Read 6 Done Sent Done false
34831 10125 Acked Flush 0 Done Sent Done false
34832 10126 Acked Read 5 Done Sent Done false
34833 10127 Acked Write 1 Done Sent Done false
34834 10128 Acked Write 6 Done Sent Done false
34835 10129 Acked Read 3 Done Sent Done false
34836 10130 Acked Write 4 Done Sent Done false
34837 10131 Acked Write 8 Done Sent Done false
34838 10132 Acked Read 10 Done Sent Done false
34839 10133 Acked Read 1 Done Sent Done false
34840 10134 Acked Read 4 Done Sent Done false
34841 10135 Acked Flush 0 Done Sent Done false
34842 10136 Acked Write 7 Done Sent Done false
34843 10137 Acked Write 3 Done Sent Done false
34844 10138 Acked Read 2 Done Sent Done false
34845 10139 Acked Write 3 Done Sent Done false
34846 10140 Acked Flush 0 Done Sent Done false
34847 10141 Acked Read 6 Done Sent Done false
34848 10142 Acked Flush 0 Done Sent Done false
34849 10143 Acked Write 7 Done Sent Done false
34850 10144 Acked Write 5 Done Sent Done false
34851 10145 Acked Flush 0 Done Sent Done false
34852 10146 Acked Flush 0 Done Sent Done false
34853 10147 Acked Write 4 Done Sent Done false
34854 10148 Acked Write 7 Done Sent Done false
34855 10149 Acked Read 9 Done Sent Done false
34856 10150 Acked Write 1 Done Sent Done false
34857 10151 Acked Write 8 Done Sent Done false
34858 10152 Acked Write 6 Done Sent Done false
34859 10153 Acked Write 10 Done Sent Done false
34860 10154 Acked Read 1 Done Sent Done false
34861 10155 Acked Read 9 Done Sent Done false
34862 10156 Acked Flush 0 Done Sent Done false
34863 10157 Acked Write 1 Done Sent Done false
34864 10158 Acked Read 8 Done Sent Done false
34865 10159 Acked Write 2 Done Sent Done false
34866 10160 Acked Read 5 Done Sent Done false
34867 10161 Acked Read 10 Done Sent Done false
34868 10162 Acked Write 6 Done Sent Done false
34869 10163 Acked Read 3 Done Sent Done false
34870 10164 Acked Read 9 Done Sent Done false
34871 10165 Acked Flush 0 Done Sent Done false
34872 10166 Acked Read 8 Done Sent Done false
34873 10167 Acked Write 2 Done Sent Done false
34874 10168 Acked Write 6 Done Sent Done false
34875 10169 Acked Read 3 Done Sent Done false
34876 10170 Acked Write 2 Done Sent Done false
34877 10171 Acked Read 9 Done Sent Done false
34878 10172 Acked Write 4 Done Sent Done false
34879 10173 Acked Write 4 Done Sent Done false
34880 10174 Acked Write 5 Done Sent Done false
34881 10175 Acked Read 9 Done Sent Done false
34882 10176 Acked Write 2 Done Sent Done false
34883 10177 Acked Read 9 Done Sent Done false
34884 10178 Acked Read 6 Done Sent Done false
34885 10179 Acked Read 4 Done Sent Done false
34886 10180 Acked Read 4 Done Sent Done false
34887 10181 Acked Flush 0 Done Sent Done false
34888 10182 Acked Read 2 Done Sent Done false
34889 10183 Acked Read 9 Done Sent Done false
34890 10184 Acked Flush 0 Done Sent Done false
34891 10185 Acked Write 8 Done Sent Done false
34892 10186 Acked Flush 0 Done Sent Done false
34893 10187 Acked Read 4 Done Sent Done false
34894 10188 Acked Read 10 Done Sent Done false
34895 10189 Acked Write 8 Done Sent Done false
34896 10190 Acked Write 2 Done Sent Done false
34897 10191 Acked Write 9 Done Sent Done false
34898 10192 Acked Read 10 Done Sent Done false
34899 10193 Acked Flush 0 Done Sent Done false
34900 10194 Acked Write 10 Done Sent Done false
34901 10195 Acked Read 4 Done Sent Done false
34902 10196 Acked Read 7 Done Sent Done false
34903 10197 Acked Write 9 Done Sent Done false
34904 10198 Acked Read 7 Done Sent Done false
34905 10199 Acked Read 4 Done Sent Done false
34906 10200 Acked Read 10 Done Sent Done false
34907 10201 Acked Read 2 Done Sent Done false
34908 10202 Acked Write 8 Done Sent Done false
34909 10203 Acked Read 4 Done Sent Done false
34910 10204 Acked Read 4 Done Sent Done false
34911 10205 Acked Read 8 Done Sent Done false
34912 10206 Acked Flush 0 Done Sent Done false
34913 10207 Acked Flush 0 Done Sent Done false
34914 10208 Acked Flush 0 Done Sent Done false
34915 10209 Acked Read 1 Done Sent Done false
34916 10210 Acked Write 1 Done Sent Done false
34917 10211 Acked Write 8 Done Sent Done false
34918 10212 Acked Read 3 Done Sent Done false
34919 10213 Acked Read 6 Done Sent Done false
34920 10214 Acked Write 4 Done Sent Done false
34921 10215 Acked Read 9 Done Sent Done false
34922 10216 Acked Flush 0 Done Sent Done false
34923 10217 Acked Read 6 Done Sent Done false
34924 10218 Acked Read 6 Done Sent Done false
34925 10219 Acked Write 6 Done Sent Done false
34926 10220 Acked Read 4 Done Sent Done false
34927 10221 Acked Read 5 Done Sent Done false
34928 10222 Acked Read 3 Done Sent Done false
34929 10223 Acked Read 5 Done Sent Done false
34930 10224 Acked Write 8 Done Sent Done false
34931 10225 Acked Write 10 Done Sent Done false
34932 10226 Acked Read 6 Done Sent Done false
34933 10227 Acked Read 3 Done Sent Done false
34934 10228 Acked Write 1 Done Sent Done false
34935 10229 Acked Read 10 Done Sent Done false
34936 10230 Acked Write 7 Done Sent Done false
34937 10231 Acked Write 5 Done Sent Done false
34938 10232 Acked Write 6 Done Sent Done false
34939 10233 Acked Read 1 Done Sent Done false
34940 10234 Acked Read 10 Done Sent Done false
34941 10235 Acked Read 1 Done Sent Done false
34942 10236 Acked Write 8 Done Sent Done false
34943 10237 Acked Read 5 Done Sent Done false
34944 10238 Acked Read 6 Done Sent Done false
34945 10239 Acked Write 9 Done Sent Done false
34946 10240 Acked Write 9 Done Sent Done false
34947 10241 Acked Read 7 Done Sent Done false
34948 10242 Acked Read 3 Done Sent Done false
34949 10243 Acked Read 7 Done Sent Done false
34950 10244 Acked Read 10 Done Sent Done false
34951 10245 Acked Read 1 Done Sent Done false
34952 10246 Acked Write 7 Done Sent Done false
34953 10247 Acked Read 3 Done Sent Done false
34954 10248 Acked Write 3 Done Sent Done false
34955 10249 Acked Write 2 Done Sent Done false
34956 10250 Acked Read 4 Done Sent Done false
34957 10251 Acked Read 8 Done Sent Done false
34958 10252 Acked Write 4 Done Sent Done false
34959 10253 Acked Write 7 Done Sent Done false
34960 10254 Acked Read 2 Done Sent Done false
34961 10255 Acked Write 3 Done Sent Done false
34962 10256 Acked Read 9 Done Sent Done false
34963 10257 Acked Read 5 Done Sent Done false
34964 10258 Acked Read 9 Done Sent Done false
34965 10259 Acked Read 9 Done Sent Done false
34966 10260 Acked Read 3 Done Sent Done false
34967 10261 Acked Read 7 Done Sent Done false
34968 10262 Acked Write 2 Done Sent Done false
34969 10263 Acked Write 6 Done Sent Done false
34970 10264 Acked Write 8 Done Sent Done false
34971 10265 Acked Read 7 Done Sent Done false
34972 10266 Acked Read 8 Done Sent Done false
34973 10267 Acked Write 6 Done Sent Done false
34974 10268 Acked Write 8 Done Sent Done false
34975 10269 Acked Write 9 Done Sent Done false
34976 10270 Acked Write 1 Done Sent Done false
34977 10271 Acked Write 3 Done Sent Done false
34978 10272 Acked Read 6 Done Sent Done false
34979 10273 Acked Write 8 Done Sent Done false
34980 10274 Acked Read 2 Done Sent Done false
34981 10275 Acked Read 10 Done Sent Done false
34982 10276 Acked Read 3 Done Sent Done false
34983 10277 Acked Write 9 Done Sent Done false
34984 10278 Acked Read 10 Done Sent Done false
34985 10279 Acked Read 9 Done Sent Done false
34986 10280 Acked Read 1 Done Sent Done false
34987 10281 Acked Flush 0 Done Sent Done false
34988 10282 Acked Read 2 Done Sent Done false
34989 10283 Acked Write 4 Done Sent Done false
34990 10284 Acked Write 3 Done Sent Done false
34991 10285 Acked Read 8 Done Sent Done false
34992 10286 Acked Flush 0 Done Sent Done false
34993 10287 Acked Read 6 Done Sent Done false
34994 10288 Acked Write 4 Done Sent Done false
34995 10289 Acked Read 10 Done Sent Done false
34996 10290 Acked Write 8 Done Sent Done false
34997 10291 Acked Read 8 Done Sent Done false
34998 10292 Acked Write 1 Done Sent Done false
34999 10293 Acked Write 3 Done Sent Done false
35000 10294 Acked Write 10 Done Sent Done false
35001 10295 Acked Write 10 Done Sent Done false
35002 10296 Acked Write 3 Done Sent Done false
35003 10297 Acked Read 9 Done Sent Done false
35004 10298 Acked Write 7 Done Sent Done false
35005 10299 Acked Read 1 Done Sent Done false
35006 10300 Acked Read 4 Done Sent Done false
35007 10301 Acked Read 10 Done Sent Done false
35008 10302 Acked Read 2 Done Sent Done false
35009 10303 Acked Write 5 Done Sent Done false
35010 10304 Acked Flush 0 Done Sent Done false
35011 10305 Acked Write 3 Done Sent Done false
35012 10306 Acked Read 1 Done Sent Done false
35013 10307 Acked Flush 0 Done Sent Done false
35014 10308 Acked Write 10 Done Sent Done false
35015 10309 Acked Flush 0 Done Sent Done false
35016 10310 Acked Write 5 Done Sent Done false
35017 10311 Acked Read 2 Done Sent Done false
35018 10312 Acked Read 1 Done Sent Done false
35019 10313 Acked Write 10 Done Sent Done false
35020 10314 Acked Write 7 Done Sent Done false
35021 10315 Acked Write 4 Done Sent Done false
35022 10316 Acked Write 7 Done Sent Done false
35023 10317 Acked Read 6 Done Sent Done false
35024 10318 Acked Write 3 Done Sent Done false
35025 10319 Acked Write 10 Done Sent Done false
35026 10320 Acked Read 8 Done Sent Done false
35027 10321 Acked Write 10 Done Sent Done false
35028 10322 Acked Read 1 Done Sent Done false
35029 10323 Acked Write 1 Done Sent Done false
35030 10324 Acked Read 8 Done Sent Done false
35031 10325 Acked Read 6 Done Sent Done false
35032 10326 Acked Read 9 Done Sent Done false
35033 10327 Acked Read 6 Done Sent Done false
35034 10328 Acked Read 7 Done Sent Done false
35035 10329 Acked Write 1 Done Sent Done false
35036 10330 Acked Read 10 Done Sent Done false
35037 10331 Acked Write 9 Done Sent Done false
35038 10332 Acked Read 2 Done Sent Done false
35039 10333 Acked Write 6 Done Sent Done false
35040 10334 Acked Read 1 Done Sent Done false
35041 10335 Acked Read 7 Done Sent Done false
35042 10336 Acked Read 4 Done Sent Done false
35043 10337 Acked Write 5 Done Sent Done false
35044 10338 Acked Read 7 Done Sent Done false
35045 10339 Acked Write 7 Done Sent Done false
35046 10340 Acked Flush 0 Done Sent Done false
35047 10341 Acked Read 4 Done Sent Done false
35048 10342 Acked Read 2 Done Sent Done false
35049 10343 Acked Read 3 Done Sent Done false
35050 10344 Acked Flush 0 Done Sent Done false
35051 10345 Acked Read 3 Done Sent Done false
35052 10346 Acked Read 1 Done Sent Done false
35053 10347 Acked Read 6 Done Sent Done false
35054 10348 Acked Write 3 Done Sent Done false
35055 10349 Acked Write 4 Done Sent Done false
35056 10350 Acked Read 8 Done Sent Done false
35057 10351 Acked Read 3 Done Sent Done false
35058 10352 Acked Read 6 Done Sent Done false
35059 10353 Acked Write 8 Done Sent Done false
35060 10354 Acked Read 5 Done Sent Done false
35061 10355 Acked Write 5 Done Sent Done false
35062 10356 Acked Read 6 Done Sent Done false
35063 10357 Acked Write 8 Done Sent Done false
35064 10358 Acked Write 5 Done Sent Done false
35065 STATES DS:0 DS:1 DS:2 TOTAL
35066 Sent 0 301 0 301
35067 Done 301 0 301 602
35068 Skipped 0 0 0 0
35069 Error 0 0 0 0
35070 Last Flush: Some(JobId(10344)) Some(JobId(10057)) Some(JobId(10344))
35071 Downstairs last five completed: 10057 10056 10055 10054 10053
35072 Upstairs last five completed: 10358 10357 10356 10355 10354
35073 CLIENT: Up:0 ds:301 act:2
350742025-01-28T22:02:02.096ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
350752025-01-28T22:02:03.022ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
350762025-01-28T22:02:03.345ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
350772025-01-28T22:02:03.345ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54988) = io task client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
350782025-01-28T22:02:03.346ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
350792025-01-28T22:02:03.346ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
350802025-01-28T22:02:03.346ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
350812025-01-28T22:02:03.347ZINFOcrucible: send last flush ID to this DS: Some(JobId(10057)) = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
350822025-01-28T22:02:03.347ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
350832025-01-28T22:02:03.347ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(10057)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
350842025-01-28T22:02:03.347ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(10057)) = downstairs session_id = 97f0d889-45eb-4682-bc34-b4ef4ba454a8
35085 ----------------------------------------------------------------
35086 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35087 Downstairs last five completed: 10359 10358 10357 10356 10355
35088 Upstairs last five completed: 10359 10358 10357 10356 10355
35089 CLIENT: Up:0 ds:0 act:3
35090 Replay: All jobs finished
35091 Test replay has completed
35092 Wrote out file "/tmp/test_replay_verify.log"
35093 CLIENT: Tests done. All submitted work has been ACK'd
35094 ----------------------------------------------------------------
35095 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35096 Downstairs last five completed: 10359 10358 10357 10356 10355
35097 Upstairs last five completed: 10359 10358 10357 10356 10355
35098 CLIENT: Up:0 ds:0 act:3
35099 CLIENT: All jobs finished, all DS active.
35100 Replays:30 time: 7:44
35101 Do final verify
35102 Jan 28 22:02:05.673 INFO accepted connection, remote_addr: 127.0.0.1:37719, local_addr: 127.0.0.1:9998
35103 Jan 28 22:02:05.673 INFO request completed, latency_us: 218, response_code: 200, uri: /regioninfo, method: GET, req_id: 42200aad-205a-42ec-b085-0b79c6395844, remote_addr: 127.0.0.1:37719, local_addr: 127.0.0.1:9998
351042025-01-28T22:02:05.674ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
35105 Jan 28 22:02:05.674 INFO request completed, latency_us: 100, response_code: 200, uri: /regioncount, method: GET, req_id: 732f46b6-bafd-4198-8c94-9e03e6625e67, remote_addr: 127.0.0.1:37719, local_addr: 127.0.0.1:9998
351062025-01-28T22:02:05.675ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
35107 Jan 28 22:02:05.675 INFO request completed, latency_us: 114, response_code: 200, uri: /port/cid/0, method: GET, req_id: cdc3d4e2-c2f8-4064-909b-30880800b77b, remote_addr: 127.0.0.1:37719, local_addr: 127.0.0.1:9998
35108 Jan 28 22:02:05.676 INFO request completed, latency_us: 104, response_code: 200, uri: /port/cid/1, method: GET, req_id: cc4787c6-8dad-4aba-bf59-d173f0220cff, remote_addr: 127.0.0.1:37719, local_addr: 127.0.0.1:9998
35109 Jan 28 22:02:05.676 INFO request completed, latency_us: 103, response_code: 200, uri: /port/cid/2, method: GET, req_id: 9d94a480-7f71-4819-a645-fa24349bfd3c, remote_addr: 127.0.0.1:37719, local_addr: 127.0.0.1:9998
351102025-01-28T22:02:05.676ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
351112025-01-28T22:02:05.677ZINFOcrucible: Upstairs starts
351122025-01-28T22:02:05.677ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "7817632486f84ab6865c5ce71e16196ec0c710fc", git_commit_timestamp: "2025-01-28T21:38:04.000000000Z", git_branch: "mkeeter/early-io-rejection", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
351132025-01-28T22:02:05.677ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
351142025-01-28T22:02:05.677ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
351152025-01-28T22:02:05.677ZINFOcrucible: Crucible 9ee7e0aa-cb7e-4b1e-807c-b153cafc0c4d has session id: d24a6fbb-5822-452e-92ce-9a811724c7cb session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351162025-01-28T22:02:05.677ZINFOcrucible: Upstairs opts: Upstairs UUID: 9ee7e0aa-cb7e-4b1e-807c-b153cafc0c4d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351172025-01-28T22:02:05.677ZINFOcrucible: Crucible stats registered with UUID: 9ee7e0aa-cb7e-4b1e-807c-b153cafc0c4d session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351182025-01-28T22:02:05.677ZWARNcrucible: could not find Downstairs address for Nexus session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351192025-01-28T22:02:05.678ZINFOcrucible: Downstairs in volume = 3 task = crutest
351202025-01-28T22:02:05.678ZINFOcrucible: The guest has requested activation with gen:3
351212025-01-28T22:02:05.678ZINFOcrucible: 9ee7e0aa-cb7e-4b1e-807c-b153cafc0c4d active request set session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351222025-01-28T22:02:05.678ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351232025-01-28T22:02:05.678ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351242025-01-28T22:02:05.678ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351252025-01-28T22:02:05.678ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351262025-01-28T22:02:05.678ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351272025-01-28T22:02:05.678ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351282025-01-28T22:02:05.679ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351292025-01-28T22:02:05.679ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351302025-01-28T22:02:05.679ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351312025-01-28T22:02:05.679ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36661) = io task client = 2 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351322025-01-28T22:02:05.679ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43095) = io task client = 1 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351332025-01-28T22:02:05.679ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51085) = io task client = 0 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351342025-01-28T22:02:05.679ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351352025-01-28T22:02:05.679ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351362025-01-28T22:02:05.679ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351372025-01-28T22:02:05.680ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351382025-01-28T22:02:05.680ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351392025-01-28T22:02:05.680ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351402025-01-28T22:02:05.681ZINFOcrucible: Waiting for more clients to be ready session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351412025-01-28T22:02:05.681ZINFOcrucible: Waiting for more clients to be ready session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351422025-01-28T22:02:05.681ZINFOcrucible: [0]R flush_numbers[0..12]: [988, 988, 939, 985, 963, 984, 985, 990, 983, 979, 990, 988] = downstairs session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351432025-01-28T22:02:05.681ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351442025-01-28T22:02:05.681ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351452025-01-28T22:02:05.681ZINFOcrucible: [1]R flush_numbers[0..12]: [988, 988, 939, 985, 963, 984, 985, 990, 983, 979, 990, 988] = downstairs session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351462025-01-28T22:02:05.681ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351472025-01-28T22:02:05.681ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351482025-01-28T22:02:05.681ZINFOcrucible: [2]R flush_numbers[0..12]: [988, 988, 939, 985, 963, 984, 985, 990, 983, 979, 990, 988] = downstairs session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351492025-01-28T22:02:05.681ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351502025-01-28T22:02:05.681ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
35151 {"msg":"Max found gen is 3","v":0,"name":"crucible","level":30,"time"Wait for a query_work_queue command to finish before sending IO
35152 :"2025-01-28T22:02:05.681576635Z","hostname":"w-01JJQEEM1S2E5C1YW6XFMRX41Q","pid":1238,"":"downstairs","":"downstairs","session_id":"d24a6fbb-5822-452e-92ce-9a811724c7cb"}
351532025-01-28T22:02:05.681ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351542025-01-28T22:02:05.681ZINFOcrucible: Next flush: 991 = downstairs session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351552025-01-28T22:02:05.681ZINFOcrucible: All extents match = downstairs session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351562025-01-28T22:02:05.681ZINFOcrucible: No downstairs reconciliation required session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351572025-01-28T22:02:05.681ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351582025-01-28T22:02:05.681ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351592025-01-28T22:02:05.681ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351602025-01-28T22:02:05.681ZINFOcrucible: All required reconciliation work is completed session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351612025-01-28T22:02:05.681ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351622025-01-28T22:02:05.681ZINFOcrucible: 9ee7e0aa-cb7e-4b1e-807c-b153cafc0c4d is now active with session: d24a6fbb-5822-452e-92ce-9a811724c7cb session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351632025-01-28T22:02:05.681ZINFOcrucible: Set Active after no reconciliation session_id = d24a6fbb-5822-452e-92ce-9a811724c7cb
351642025-01-28T22:02:05.681ZINFOcrucible: The guest has finished waiting for activation with:3
35165 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
35166 Loading write count information from file "/tmp/test_replay_verify.log"
35167 Read and Verify all blocks (0..5000 range:false)
35168 Wrote out file "/tmp/test_replay_verify.log"
35169 Verify test completed
35170 Wrote out file "/tmp/test_replay_verify.log"
35171 CLIENT: Tests done. All submitted work has been ACK'd
35172 ----------------------------------------------------------------
35173 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
35174 States: Active Active Active
35175 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35176 1000 Acked Read 256 Done Done Done false
35177 1001 Acked Read 256 Done Done Done false
35178 1002 Acked Read 256 Done Done Done false
35179 1003 Acked Read 256 Done Done Done false
35180 1004 Acked Read 256 Done Done Done false
35181 1005 Acked Read 256 Done Done Done false
35182 1006 Acked Read 256 Done Done Done false
35183 1007 Acked Read 256 Done Done Done false
35184 1008 Acked Read 256 Done Done Done false
35185 1009 Acked Read 256 Done Done Done false
35186 1010 Acked Read 256 Done Done Done false
35187 1011 Acked Read 256 Done Done Done false
35188 1012 Acked Read 256 Done Done Done false
35189 1013 Acked Read 256 Done Done Done false
35190 1014 Acked Read 256 Done Done Done false
35191 1015 Acked Read 256 Done Done Done false
35192 1016 Acked Read 256 Done Done Done false
35193 1017 Acked Read 256 Done Done Done false
35194 1018 Acked Read 256 Done Done Done false
35195 1019 Acked Read 136 Done Done Done false
35196 STATES DS:0 DS:1 DS:2 TOTAL
35197 Sent 0 0 0 0
35198 Done 20 20 20 60
35199 Skipped 0 0 0 0
35200 Error 0 0 0 0
35201 Last Flush: None None None
35202 Downstairs last five completed:
35203 Upstairs last five completed: 1019 1018 1017 1016 1015
35204 CLIENT: Up:0 ds:20 act:3
35205 ----------------------------------------------------------------
35206 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
35207 Downstairs last five completed: 1020 1019 1018 1017 1016
35208 Upstairs last five completed: 1020 1019 1018 1017 1016
35209 CLIENT: Up:0 ds:0 act:3
35210 CLIENT: All crucible jobs finished, exiting program
35211 Jan 28 22:02:10.153 INFO accepted connection, remote_addr: 127.0.0.1:48673, local_addr: 127.0.0.1:9998
35212 Main task has work to do, go find it
35213 got dsc Shutdown
35214 Shutdown
35215 Shut it down
35216 Jan 28 22:02:10.154[8810] Disable keep_running
35217 [8810] Got stop action so:false kr:false
35218 INFO request completed, latency_us: 151, response_code: 204, uri:[8820] Disable keep_running
35219 [8820] Got stop action so:false kr:false
35220 /shutdown, method: POST, req_id: 6ee74444-2b78-4957-ad90-d55d272e19b4, remote_addr: 127.0.0.1:48673, local_addr: 127.0.0.1:9998
35221 [8830] Disable keep_running
35222 [8830] Got stop action so:false kr:false
35223 [8830] Exited with: ExitStatus(unix_wait_status(9))
35224 [8830][2] reports Exit
35225 [8820] Exited with: ExitStatus(unix_wait_status(9))
35226 [8820][1] reports Exit
35227 [8810] Exited with: ExitStatus(unix_wait_status(9))
35228 [8810][0] reports Exit
35229 January 28, 2025 at 10:02:15 PM UTC Test ends with 0