Max level shown:
1 starting February 7, 2025 at 08:07:07 PM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.25925976
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.020914376
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.022649208
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start ds: 8820
14 start ds: 8830
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8810
17 Starting downstairs at port 8820
18 Make output file at "/tmp/dsc/downstairs-8810.txt"
19 Make output file at "/tmp/dsc/downstairs-8820.txt"
20 Starting downstairs at port 8830
21 Make output file at "/tmp/dsc/downstairs-8830.txt"
22 [8810][0] initial start wait reports Starting
23 [8820][1] initial start wait reports Starting
24 [8830][2] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1199)
26 [8810][0] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1200)
28 [8830][2] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1201)
30 [8820][1] initial start wait reports Running
31 All downstairs are running
32 Feb 07 20:07:07.718 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Feb 07 20:07:12.781 INFO accepted connection, remote_addr: 127.0.0.1:51034, local_addr: 127.0.0.1:9998
36 Feb 07 20:07:12.784 INFO request completed, latency_us: 553, response_code: 200, uri: /regioninfo, method: GET, req_id: e748de2a-ccab-4b57-b456-5827d15301fa, remote_addr: 127.0.0.1:51034, local_addr: 127.0.0.1:9998
37 Feb 07 20:07:12.785 INFO request completed, latency_us: 97, response_code: 200, uri: /regioncount, method: GET, req_id: 72115c54-40d5-44bd-af3a-e01d0807d06d, remote_addr: 127.0.0.1:51034, local_addr: 127.0.0.1:9998
382025-02-07T20:07:12.785ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-02-07T20:07:12.785ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Feb 07 20:07:12.786 INFO request completed, latency_us: 113, response_code: 200, uri: /port/cid/0, method: GET, req_id: ab6b297d-ff0f-4d71-97ec-cee671967534, remote_addr: 127.0.0.1:51034, local_addr: 127.0.0.1:9998
41 Feb 07 20:07:12.786 INFO request completed, latency_us: 90, response_code: 200, uri: /port/cid/1, method: GET, req_id: 225bf31a-0249-4fc1-8cdf-f28470f5eabe, remote_addr: 127.0.0.1:51034, local_addr: 127.0.0.1:9998
42 Feb 07 20:07:12.787 INFO request completed, latency_us: 110, response_code: 200, uri: /port/cid/2, method: GET, req_id: 135e16cc-b8c8-4e82-8031-5586c1d636e7, remote_addr: 127.0.0.1:51034, local_addr: 127.0.0.1:9998
432025-02-07T20:07:12.787ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-02-07T20:07:12.788ZINFOcrucible: Upstairs starts
452025-02-07T20:07:12.788ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "764da36f6a4255a5834591b69c6e5591e6ce3a04", git_commit_timestamp: "2025-02-07T19:51:46.000000000Z", git_branch: "alan/agent-needs-workers", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-02-07T20:07:12.788ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-02-07T20:07:12.788ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-02-07T20:07:12.789ZINFOcrucible: Crucible e68996f1-47bb-4e67-9fc8-93760a124c2c has session id: 767f2d0a-4cf4-419c-ba1d-d4c07b976153 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
492025-02-07T20:07:12.789ZINFOcrucible: Upstairs opts: Upstairs UUID: e68996f1-47bb-4e67-9fc8-93760a124c2c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
502025-02-07T20:07:12.789ZINFOcrucible: Crucible stats registered with UUID: e68996f1-47bb-4e67-9fc8-93760a124c2c session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
512025-02-07T20:07:12.789ZWARNcrucible: could not find Downstairs address for Nexus session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
522025-02-07T20:07:12.790ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-02-07T20:07:12.790ZINFOcrucible: The guest has requested activation with gen:1
542025-02-07T20:07:12.790ZINFOcrucible: e68996f1-47bb-4e67-9fc8-93760a124c2c active request set session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
552025-02-07T20:07:12.790ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
562025-02-07T20:07:12.790ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
572025-02-07T20:07:12.790ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
582025-02-07T20:07:12.790ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
592025-02-07T20:07:12.790ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
602025-02-07T20:07:12.790ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
612025-02-07T20:07:12.791ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
622025-02-07T20:07:12.791ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
632025-02-07T20:07:12.791ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
642025-02-07T20:07:12.791ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63931) = io task client = 0 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
652025-02-07T20:07:12.791ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59642) = io task client = 1 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
662025-02-07T20:07:12.791ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:40907) = io task client = 2 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
672025-02-07T20:07:12.792ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
682025-02-07T20:07:12.792ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
692025-02-07T20:07:12.792ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
702025-02-07T20:07:12.793ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
712025-02-07T20:07:12.793ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
722025-02-07T20:07:12.794ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
732025-02-07T20:07:12.794ZINFOcrucible: Waiting for more clients to be ready session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
742025-02-07T20:07:12.794ZINFOcrucible: Waiting for more clients to be ready session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
752025-02-07T20:07:12.795ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
762025-02-07T20:07:12.795ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
772025-02-07T20:07:12.795ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
782025-02-07T20:07:12.795ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
792025-02-07T20:07:12.795ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
802025-02-07T20:07:12.795ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
812025-02-07T20:07:12.795ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
822025-02-07T20:07:12.795ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
832025-02-07T20:07:12.795ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
842025-02-07T20:07:12.795ZINFOcrucible: Max found gen is 1 = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
852025-02-07T20:07:12.795ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
862025-02-07T20:07:12.795ZINFOcrucible: Next flush: 1 = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
872025-02-07T20:07:12.795ZINFOcrucible: All extents match = downstairs session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
882025-02-07T20:07:12.795ZINFOcrucible: No downstairs reconciliation required session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
892025-02-07T20:07:12.795ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
902025-02-07T20:07:12.795ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
912025-02-07T20:07:12.795ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
922025-02-07T20:07:12.795ZINFOcrucible: All required reconciliation work is completed session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
93 {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Activate successful
94 ,Wait for a query_work_queue command to finish before sending IO
95 "time":"2025-02-07T20:07:12.795405281Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1205,"session_id":"767f2d0a-4cf4-419c-ba1d-d4c07b976153"}
962025-02-07T20:07:12.795ZINFOcrucible: e68996f1-47bb-4e67-9fc8-93760a124c2c is now active with session: 767f2d0a-4cf4-419c-ba1d-d4c07b976153 session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
972025-02-07T20:07:12.795ZINFOcrucible: Set Active after no reconciliation session_id = 767f2d0a-4cf4-419c-ba1d-d4c07b976153
982025-02-07T20:07:12.795ZINFOcrucible: The guest has finished waiting for activation with:1
99 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
100 Fill test
101 Read and Verify all blocks (0..5000 range:false)
102 Wrote out file "/tmp/test_replay_verify.log"
103 CLIENT: Tests done. All submitted work has been ACK'd
104 ----------------------------------------------------------------
105 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
106 States: Active Active Active
107 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
108 1021 Acked Read 256 Done Done Done false
109 1022 Acked Read 256 Done Done Done false
110 1023 Acked Read 256 Done Done Done false
111 1024 Acked Read 256 Done Done Done false
112 1025 Acked Read 256 Done Done Done false
113 1026 Acked Read 256 Done Done Done false
114 1027 Acked Read 256 Done Done Done false
115 1028 Acked Read 256 Done Done Done false
116 1029 Acked Read 256 Done Done Done false
117 1030 Acked Read 256 Done Done Done false
118 1031 Acked Read 256 Done Done Done false
119 1032 Acked Read 256 Done Done Done false
120 1033 Acked Read 256 Done Done Done false
121 1034 Acked Read 256 Done Done Done false
122 1035 Acked Read 256 Done Done Done false
123 1036 Acked Read 256 Done Done Done false
124 1037 Acked Read 256 Done Done Done false
125 1038 Acked Read 136 Done Done Done false
126 1039 Acked Read 256 Done Done Done false
127 1040 Acked Read 256 Done Done Done false
128 STATES DS:0 DS:1 DS:2 TOTAL
129 Sent 0 0 0 0
130 Done 20 20 20 60
131 Skipped 0 0 0 0
132 Error 0 0 0 0
133 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
134 Downstairs last five completed: 1020 1019 1018 1017 1016
135 Upstairs last five completed: 1040 1039 1038 1037 1036
136 CLIENT: Up:0 ds:20 act:3
137 ----------------------------------------------------------------
138 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
139 Downstairs last five completed: 1041 1040 1039 1038 1037
140 Upstairs last five completed: 1041 1040 1039 1038 1037
141 CLIENT: Up:0 ds:0 act:3
142 CLIENT: All crucible jobs finished, exiting program
143 Replay loop starts now February 7, 2025 at 08:07:17 PM UTC
144 Feb 07 20:07:17.430 INFO accepted connection, remote_addr: 127.0.0.1:58509, local_addr: 127.0.0.1:9998
145 Feb 07 20:07:17.431 INFO request completed, latency_us: 171, response_code: 200, uri: /regioninfo, method: GET, req_id: 1122a221-fc1f-4636-8814-57b6a10bdecc, remote_addr: 127.0.0.1:58509, local_addr: 127.0.0.1:9998
1462025-02-07T20:07:17.431ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
147 Feb 07 20:07:17.432 INFO request completed, latency_us: 96, response_code: 200, uri: /regioncount, method: GET, req_id: 7c0b7fdf-4f15-4e20-90dd-31323732d9d2, remote_addr: 127.0.0.1:58509, local_addr: 127.0.0.1:9998
1482025-02-07T20:07:17.432ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
149 Feb 07 20:07:17.432 INFO request completed, latency_us: 113, response_code: 200, uri: /port/cid/0, method: GET, req_id: 83e1a226-5bce-4623-a570-517af5d2a589, remote_addr: 127.0.0.1:58509, local_addr: 127.0.0.1:9998
150 Feb 07 20:07:17.433 INFO request completed, latency_us: 97, response_code: 200, uri: /port/cid/1, method: GET, req_id: a56f6526-de7b-4a08-90ec-f3526f07d41a, remote_addr: 127.0.0.1:58509, local_addr: 127.0.0.1:9998
151 Feb 07 20:07:17.433 INFO request completed, latency_us: 90, response_code: 200, uri: /port/cid/2, method: GET, req_id: d3504c7b-87d6-4b83-bace-778da8935514, remote_addr: 127.0.0.1:58509, local_addr: 127.0.0.1:9998
1522025-02-07T20:07:17.434ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1532025-02-07T20:07:17.434ZINFOcrucible: Upstairs starts
1542025-02-07T20:07:17.434ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "764da36f6a4255a5834591b69c6e5591e6ce3a04", git_commit_timestamp: "2025-02-07T19:51:46.000000000Z", git_branch: "alan/agent-needs-workers", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1552025-02-07T20:07:17.434ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1562025-02-07T20:07:17.434ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1572025-02-07T20:07:17.434ZINFOcrucible: Crucible 43e4c6d6-54a8-4338-843a-8d17e2f61a69 has session id: 8f7ba378-6b82-4db6-a812-9286715f4b09 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1582025-02-07T20:07:17.434ZINFOcrucible: Upstairs opts: Upstairs UUID: 43e4c6d6-54a8-4338-843a-8d17e2f61a69, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1592025-02-07T20:07:17.434ZINFOcrucible: Crucible stats registered with UUID: 43e4c6d6-54a8-4338-843a-8d17e2f61a69 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1602025-02-07T20:07:17.434ZWARNcrucible: could not find Downstairs address for Nexus session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1612025-02-07T20:07:17.435ZINFOcrucible: Downstairs in volume = 3 task = crutest
1622025-02-07T20:07:17.436ZINFOcrucible: The guest has requested activation with gen:2
1632025-02-07T20:07:17.436ZINFOcrucible: 43e4c6d6-54a8-4338-843a-8d17e2f61a69 active request set session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1642025-02-07T20:07:17.436ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1652025-02-07T20:07:17.436ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1662025-02-07T20:07:17.436ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1672025-02-07T20:07:17.436ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1682025-02-07T20:07:17.436ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1692025-02-07T20:07:17.436ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1702025-02-07T20:07:17.436ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1712025-02-07T20:07:17.436ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1722025-02-07T20:07:17.436ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1732025-02-07T20:07:17.436ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55854) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1742025-02-07T20:07:17.436ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43753) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1752025-02-07T20:07:17.436ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48828) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1762025-02-07T20:07:17.437ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1772025-02-07T20:07:17.437ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1782025-02-07T20:07:17.437ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1792025-02-07T20:07:17.437ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1802025-02-07T20:07:17.437ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1812025-02-07T20:07:17.437ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1822025-02-07T20:07:17.438ZINFOcrucible: Waiting for more clients to be ready session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1832025-02-07T20:07:17.438ZINFOcrucible: Waiting for more clients to be ready session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1842025-02-07T20:07:17.438ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1852025-02-07T20:07:17.438ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1862025-02-07T20:07:17.438ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1872025-02-07T20:07:17.438ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1882025-02-07T20:07:17.438ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1892025-02-07T20:07:17.438ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1902025-02-07T20:07:17.438ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1912025-02-07T20:07:17.438ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
192 {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-07T20:07:17.438990768Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":Activate successful
193 1209Wait for a query_work_queue command to finish before sending IO
194 ,"":"downstairs","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
1952025-02-07T20:07:17.439ZINFOcrucible: Max found gen is 2 = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1962025-02-07T20:07:17.439ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1972025-02-07T20:07:17.439ZINFOcrucible: Next flush: 2 = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1982025-02-07T20:07:17.439ZINFOcrucible: All extents match = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1992025-02-07T20:07:17.439ZINFOcrucible: No downstairs reconciliation required session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2002025-02-07T20:07:17.439ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2012025-02-07T20:07:17.439ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2022025-02-07T20:07:17.439ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2032025-02-07T20:07:17.439ZINFOcrucible: All required reconciliation work is completed session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2042025-02-07T20:07:17.439ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2052025-02-07T20:07:17.439ZINFOcrucible: 43e4c6d6-54a8-4338-843a-8d17e2f61a69 is now active with session: 8f7ba378-6b82-4db6-a812-9286715f4b09 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2062025-02-07T20:07:17.439ZINFOcrucible: Set Active after no reconciliation session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2072025-02-07T20:07:17.439ZINFOcrucible: The guest has finished waiting for activation with:2
208 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
209 Loading write count information from file "/tmp/test_replay_verify.log"
210 Feb 07 20:07:17.471 INFO accepted connection, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
211 Main task has work to do, go find it
212 got dsc Stop(0)
213 stop 0
214 Feb 07 20:07:17.472 INFO request completed, latency_us: 170, response_code: 204, uri: /stop/cid/0, method: POST, req_id:[8810] Got stop action so:false kr:false
215 8181f7a8-cafb-448d-bb16-86e2afdd6854, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
2162025-02-07T20:07:17.473ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2172025-02-07T20:07:17.473ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2182025-02-07T20:07:17.473ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2192025-02-07T20:07:17.473ZINFOcrucible: client task is exiting = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220 Feb 07 20:07:17.473 INFO request completed, latency_us: 157, response_code: 200, uri: /state/cid/0, method: GET, req_id: c70cf8bf-9cfe-46df-b8a1-bd636e2a420b, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
221 [8810] Exited with: ExitStatus(unix_wait_status(9))
222 [8810][0] reports Exit
223 Feb 07 20:07:21.475 INFO request completed, latency_us: 236, response_code: 200, uri: /state/cid/0, method: GET, req_id: 41b85076-10eb-4212-bdbd-5f412e46f5bc, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
224 001/300 Flush
225 002/300 Read block 3327 len 4096
226 003/300 Read block 3881 len 32768
227 004/300 Read block 12 len 36864
228 005/300 Read block 3028 len 4096
229 006/300 Write block 1649 len 36864 data: 2 2 2 2 2 2 2 2 2
230 007/300 Read block 2391 len 36864
231 008/300 Write block 3210 len 16384 data: 2 2 2 2
232 009/300 Write block 2977 len 28672 data: 2 2 2 2 2 2 2
233 010/300 Read block 4451 len 8192
234 011/300 Write block 3832 len 20480 data: 2 2 2 2 2
235 012/300 Read block 3953 len 40960
236 013/300 Read block 2566 len 28672
237 014/300 Write block 2446 len 28672 data: 2 2 2 2 2 2 2
238 015/300 Write block 3337 len 32768 data: 2 2 2 2 2 2 2 2
239 016/300 Write block 3501 len 8192 data: 2 2
240 017/300 Write block 2222 len 36864 data: 2 2 2 2 2 2 2 2 2
241 018/300 Flush
242 019/300 Write block 2397 len 20480 data: 2 2 2 2 2
243 020/300 Read block 144 len 40960
244 021/300 Read block 4393 len 32768
245 022/300 Read block 2396 len 40960
246 023/300 Read block 2612 len 36864
247 024/300 Write block 1235 len 4096 data: 2
248 025/300 Read block 2533 len 20480
249 026/300 Read block 1866 len 16384
250 027/300 Write block 753 len 8192 data: 2 2
251 028/300 Read block 676 len 4096
252 029/300 Read block 4060 len 8192
253 030/300 Read block 219 len 12288
254 031/300 Read block 2879 len 8192
255 032/300 Read block 700 len 28672
256 033/300 Write block 4326 len 12288 data: 2 2 2
257 034/300 Write block 4082 len 16384 data: 2 2 2 2
258 035/300 Write block 1261 len 28672 data: 2 2 2 2 2 2 2
259 036/300 Read block 1029 len 20480
260 037/300 Write block 3248 len 36864 data: 2 2 2 2 2 2 2 2 2
261 038/300 Flush
262 039/300 Write block 3204 len 12288 data: 2 2 2
263 040/300 Flush
264 041/300 Flush
265 042/300 Write block 113 len 36864 data: 2 2 2 2 2 2 2 2 2
266 043/300 Read block 607 len 36864
267 044/300 Write block 4189 len 20480 data: 2 2 2 2 2
268 045/300 Read block 2677 len 28672
269 046/300 Read block 4952 len 12288
270 047/300 Read block 2681 len 20480
271 048/300 Read block 3935 len 36864
272 049/300 Read block 3958 len 8192
273 050/300 Write block 2868 len 16384 data: 2 2 2 2
274 051/300 Write block 2347 len 40960 data: 2 2 2 2 2 2 2 2 2 2
275 052/300 Read block 564 len 28672
276 053/300 Write block 3410 len 8192 data: 2 2
277 054/300 Write block 294 len 12288 data: 2 2 2
278 055/300 Write block 3991 len 24576 data: 2 2 2 2 2 2
279 056/300 Write block 4602 len 40960 data: 2 2 2 2 2 2 2 2 2 2
280 057/300 Write block 384 len 8192 data: 2 2
281 058/300 Flush
282 059/300 Write block 2815 len 28672 data: 2 2 2 2 2 2 2
283 060/300 Read block 3268 len 16384
284 061/300 Write block 1821 len 32768 data: 2 2 2 2 2 2 2 2
285 062/300 Flush
286 063/300 Read block 2208 len 4096
287 064/300 Read block 4389 len 36864
288 065/300 Read block 4725 len 36864
289 066/300 Read block 29 len 32768
290 067/300 Flush
291 068/300 Read block 4197 len 16384
292 069/300 Read block 992 len 40960
293 070/300 Read block 2239 len 24576
294 071/300 Write block 2440 len 16384 data: 2 2 2 2
295 072/300 Read block 480 len 16384
296 073/300 Write block 116 len 8192 data: 3 3
297 074/300 Read block 536 len 12288
298 075/300 Write block 95 len 36864 data: 2 2 2 2 2 2 2 2 2
299 076/300 Write block 361 len 12288 data: 2 2 2
300 077/300 Write block 31 len 36864 data: 2 2 2 2 2 2 2 2 2
301 078/300 Read block 1027 len 24576
302 079/300 Flush
303 080/300 Flush
304 081/300 Read block 3622 len 8192
305 082/300 Read block 4155 len 36864
306 083/300 Read block 3859 len 28672
307 084/300 Read block 1384 len 4096
308 085/300 Read block 4751 len 20480
309 086/300 Write block 4514 len 40960 data: 2 2 2 2 2 2 2 2 2 2
310 087/300 Read block 409 len 8192
311 088/300 Read block 4173 len 20480
312 089/300 Read block 3496 len 8192
313 090/300 Read block 2484 len 4096
314 091/300 Read block 3511 len 32768
315 092/300 Read block 2899 len 8192
316 093/300 Read block 1612 len 40960
317 094/300 Flush
318 095/300 Write block 1979 len 28672 data: 2 2 2 2 2 2 2
319 096/300 Write block 3105 len 28672 data: 2 2 2 2 2 2 2
320 097/300 Write block 4149 len 32768 data: 2 2 2 2 2 2 2 2
321 098/300 Write block 4493 len 40960 data: 2 2 2 2 2 2 2 2 2 2
322 099/300 Read block 662 len 40960
323 100/300 Write block 4815 len 16384 data: 2 2 2 2
324 101/300 Read block 1193 len 4096
325 102/300 Read block 1549 len 24576
326 103/300 Read block 4034 len 4096
327 104/300 Write block 1161 len 20480 data: 2 2 2 2 2
328 105/300 Write block 4338 len 24576 data: 2 2 2 2 2 2
329 106/300 Write block 1108 len 40960 data: 2 2 2 2 2 2 2 2 2 2
330 107/300 Write block 1049 len 40960 data: 2 2 2 2 2 2 2 2 2 2
331 108/300 Write block 2177 len 20480 data: 2 2 2 2 2
332 109/300 Read block 2612 len 12288
333 110/300 Write block 4502 len 36864 data: 3 2 2 2 2 2 2 2 2
334 111/300 Write block 3392 len 32768 data: 2 2 2 2 2 2 2 2
335 112/300 Read block 1541 len 20480
336 113/300 Write block 181 len 12288 data: 2 2 2
337 114/300 Write block 1344 len 12288 data: 2 2 2
338 115/300 Read block 3259 len 40960
339 116/300 Write block 2075 len 16384 data: 2 2 2 2
340 117/300 Write block 4507 len 20480 data: 3 3 3 3 2
341 118/300 Read block 2999 len 4096
342 119/300 Read block 2238 len 20480
343 120/300 Read block 4950 len 40960
344 121/300 Read block 1591 len 4096
345 122/300 Write block 4944 len 28672 data: 2 2 2 2 2 2 2
346 123/300 Read block 3634 len 8192
347 124/300 Write block 4618 len 12288 data: 2 2 2
348 125/300 Read block 1638 len 20480
349 126/300 Write block 1001 len 24576 data: 2 2 2 2 2 2
350 127/300 Write block 1320 len 28672 data: 2 2 2 2 2 2 2
351 128/300 Read block 2596 len 40960
352 129/300 Read block 3462 len 40960
353 130/300 Read block 1517 len 8192
354 131/300 Read block 3057 len 24576
355 132/300 Read block 4677 len 40960
356 133/300 Flush
357 134/300 Write block 3426 len 32768 data: 2 2 2 2 2 2 2 2
358 135/300 Read block 3979 len 40960
359 136/300 Read block 755 len 24576
360 137/300 Write block 2497 len 12288 data: 2 2 2
361 138/300 Read block 3281 len 40960
362 139/300 Write block 2019 len 40960 data: 2 2 2 2 2 2 2 2 2 2
363 140/300 Write block 4794 len 12288 data: 2 2 2
364 141/300 Read block 3199 len 24576
365 142/300 Read block 3886 len 16384
366 143/300 Read block 4383 len 32768
367 144/300 Write block 1211 len 28672 data: 2 2 2 2 2 2 2
368 145/300 Read block 2054 len 40960
369 146/300 Write block 1863 len 24576 data: 2 2 2 2 2 2
370 147/300 Write block 902 len 20480 data: 2 2 2 2 2
371 148/300 Read block 2399 len 32768
372 149/300 Read block 4687 len 28672
373 150/300 Read block 766 len 28672
374 151/300 Write block 4966 len 8192 data: 2 2
375 152/300 Write block 93 len 4096 data: 2
376 153/300 Write block 813 len 36864 data: 2 2 2 2 2 2 2 2 2
377 154/300 Write block 566 len 12288 data: 2 2 2
378 155/300 Read block 2463 len 40960
379 156/300 Read block 4855 len 28672
380 157/300 Write block 2308 len 28672 data: 2 2 2 2 2 2 2
381 158/300 Read block 2532 len 40960
382 159/300 Read block 3639 len 12288
383 160/300 Read block 1285 len 4096
384 161/300 Read block 488 len 36864
385 162/300 Write block 4732 len 16384 data: 2 2 2 2
386 163/300 Read block 1231 len 36864
387 164/300 Read block 2407 len 12288
388 165/300 Read block 1423 len 12288
389 166/300 Read block 1643 len 40960
390 167/300 Write block 54 len 8192 data: 2 2
391 168/300 Read block 3196 len 24576
392 169/300 Read block 2997 len 16384
393 170/300 Write block 3778 len 16384 data: 2 2 2 2
394 171/300 Write block 3812 len 36864 data: 2 2 2 2 2 2 2 2 2
395 172/300 Write block 1612 len 20480 data: 2 2 2 2 2
396 173/300 Read block 4503 len 4096
397 174/300 Read block 3030 len 32768
398 175/300 Read block 1188 len 32768
399 176/300 Flush
400 177/300 Flush
401 178/300 Read block 925 len 40960
402 179/300 Write block 4407 len 24576 data: 2 2 2 2 2 2
403 180/300 Write block 2601 len 4096 data: 2
404 181/300 Read block 2084 len 4096
405 182/300 Read block 3338 len 28672
406 183/300 Write block 677 len 40960 data: 2 2 2 2 2 2 2 2 2 2
407 184/300 Read block 4562 len 28672
408 185/300 Read block 2209 len 16384
409 186/300 Write block 4539 len 12288 data: 2 2 2
410 187/300 Read block 4613 len 28672
411 188/300 Read block 153 len 16384
412 189/300 Write block 3102 len 4096 data: 2
413 190/300 Write block 2294 len 24576 data: 2 2 2 2 2 2
414 191/300 Read block 4109 len 20480
415 192/300 Read block 539 len 32768
416 193/300 Read block 4247 len 4096
417 194/300 Write block 3732 len 16384 data: 2 2 2 2
418 195/300 Flush
419 196/300 Write block 2673 len 28672 data: 2 2 2 2 2 2 2
420 197/300 Read block 553 len 24576
421 198/300 Write block 2737 len 36864 data: 2 2 2 2 2 2 2 2 2
422 199/300 Write block 3444 len 32768 data: 2 2 2 2 2 2 2 2
423 200/300 Flush
424 201/300 Write block 1186 len 28672 data: 2 2 2 2 2 2 2
425 202/300 Read block 26 len 24576
426 203/300 Flush
427 204/300 Write block 3506 len 4096 data: 2
428 205/300 Write block 3351 len 32768 data: 2 2 2 2 2 2 2 2
429 206/300 Read block 4427 len 28672
430 207/300 Read block 3664 len 28672
431 208/300 Flush
432 209/300 Read block 2454 len 28672
433 210/300 Read block 926 len 32768
434 211/300 Write block 4648 len 16384 data: 2 2 2 2
435 212/300 Read block 3151 len 12288
436 213/300 Write block 4680 len 24576 data: 2 2 2 2 2 2
437 214/300 Read block 2162 len 8192
438 215/300 Write block 2363 len 8192 data: 2 2
439 216/300 Write block 2802 len 32768 data: 2 2 2 2 2 2 2 2
440 217/300 Write block 3246 len 40960 data: 2 2 3 3 3 3 3 3 3 3
441 218/300 Read block 3089 len 28672
442 219/300 Read block 251 len 36864
443 220/300 Read block 878 len 12288
444 221/300 Write block 861 len 12288 data: 2 2 2
445 222/300 Write block 4734 len 28672 data: 3 3 2 2 2 2 2
446 223/300 Flush
447 224/300 Read block 2406 len 8192
448 225/300 Read block 3379 len 32768
449 226/300 Read block 2445 len 24576
450 227/300 Write block 4668 len 16384 data: 2 2 2 2
451 228/300 Write block 4502 len 16384 data: 4 3 3 3
452 229/300 Read block 4694 len 20480
453 230/300 Read block 2608 len 16384
454 231/300 Read block 4250 len 24576
455 232/300 Write block 1336 len 20480 data: 2 2 2 2 2
456 233/300 Read block 1158 len 12288
457 234/300 Read block 2135 len 24576
458 235/300 Write block 4422 len 12288 data: 2 2 2
459 236/300 Flush
460 237/300 Read block 746 len 20480
461 238/300 Write block 4158 len 12288 data: 2 2 2
462 239/300 Read block 2166 len 8192
463 240/300 Write block 3673 len 40960 data: 2 2 2 2 2 2 2 2 2 2
464 241/300 Read block 2235 len 24576
465 242/300 Write block 4059 len 4096 data: 2
466 243/300 Flush
467 244/300 Read block 995 len 4096
468 245/300 Flush
469 246/300 Read block 411 len 40960
470 247/300 Write block 299 len 32768 data: 2 2 2 2 2 2 2 2
471 248/300 Write block 3931 len 20480 data: 2 2 2 2 2
472 249/300 Flush
473 250/300 Read block 3388 len 28672
474 251/300 Write block 1195 len 28672 data: 2 2 2 2 2 2 2
475 252/300 Write block 2579 len 32768 data: 2 2 2 2 2 2 2 2
476 253/300 Write block 1421 len 4096 data: 2
477 254/300 Write block 4265 len 8192 data: 2 2
478 255/300 Write block 3329 len 32768 data: 2 2 2 2 2 2 2 2
479 256/300 Write block 3145 len 32768 data: 2 2 2 2 2 2 2 2
480 257/300 Read block 1874 len 12288
481 258/300 Read block 2319 len 8192
482 259/300 Read block 4948 len 8192
483 260/300 Write block 2801 len 12288 data: 2 3 3
484 261/300 Read block 3939 len 36864
485 262/300 Read block 1131 len 12288
486 263/300 Write block 3695 len 8192 data: 2 2
487 264/300 Write block 284 len 24576 data: 2 2 2 2 2 2
488 265/300 Read block 2936 len 32768
489 266/300 Write block 3720 len 12288 data: 2 2 2
490 267/300 Write block 12 len 16384 data: 2 2 2 2
491 268/300 Read block 723 len 4096
492 269/300 Flush
493 270/300 Read block 4172 len 4096
494 271/300 Read block 1191 len 12288
495 272/300 Read block 1799 len 20480
496 273/300 Write block 110 len 32768 data: 2 2 2 3 3 3 4 4
497 274/300 Write block 1179 len 28672 data: 2 2 2 2 2 2 2
498 275/300 Read block 2714 len 24576
499 276/300 Write block 706 len 32768 data: 2 2 2 2 2 2 2 2
500 277/300 Flush
501 278/300 Read block 4594 len 40960
502 279/300 Flush
503 280/300 Flush
504 281/300 Flush
505 282/300 Flush
506 283/300 Write block 2858 len 20480 data: 2 2 2 2 2
507 284/300 Write block 2767 len 24576 data: 2 2 2 2 2 2
508 285/300 Read block 620 len 36864
509 286/300 Write block 1145 len 20480 data: 2 2 2 2 2
510 287/300 Flush
511 288/300 Write block 4724 len 40960 data: 2 2 2 2 2 2 2 2 3 3
512 289/300 Read block 3945 len 32768
513 290/300 Write block 2308 len 32768 data: 3 3 3 3 3 3 3 2
514 291/300 Read block 3575 len 24576
515 292/300 Write block 4741 len 4096 data: 2
516 293/300 Write block 3125 len 36864 data: 2 2 2 2 2 2 2 2 2
517 294/300 Read block 1760 len 4096
518 295/300 Read block 2293 len 16384
519 296/300 Read block 3829 len 8192
520 297/300 Write block 1701 len 24576 data: 2 2 2 2 2 2
521 298/300 Flush
522 299/300 Write block 2642 len 4096 data: 2
523 300/300 Read block 1686 len 4096
524 301/300 Read block 3451 len 12288
525 Main task has work to do, go find it
526 got dsc Start(0)
527 start 0
528 Feb 07 20:07:21.719 INFO request completed, latency_us: 234, response_code: 204, uri: /start/cid/0, method: POST, req_id: 49e08cbd-28bf-40a7-8eaf-1ea5d7dd34c1, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
529 Replay: started 0, returned:Ok(())
530 ----------------------------------------------------------------
531 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
532 States: Offline Active Active
533 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
534 1000 Acked Flush 0 Sent Done Done false
535 1001 Acked Read 1 Sent Done Done false
536 1002 Acked Read 8 Sent Done Done false
537 1003 Acked Read 9 Sent Done Done false
538 1004 Acked Read 1 Sent Done Done false
539 1005 Acked Write 9 Sent Done Done false
540 1006 Acked Read 9 Sent Done Done false
541 1007 Acked Write 4 Sent Done Done false
542 1008 Acked Write 7 Sent Done Done false
543 1009 Acked Read 2 Sent Done Done false
544 1010 Acked Write 5 Sent Done Done false
545 1011 Acked Read 10 Sent Done Done false
546 1012 Acked Read 7 Sent Done Done false
547 1013 Acked Write 7 Sent Done Done false
548 1014 Acked Write 8 Sent Done Done false
549 1015 Acked Write 2 Sent Done Done false
550 1016 Acked Write 9 Sent Done Done false
551 1017 Acked Flush 0 Sent Done Done false
552 1018 Acked Write 5 Sent Done Done false
553 1019 Acked Read 10 Sent Done Done false
554 1020 Acked Read 8 Sent Done Done false
555 1021 Acked Read 10 Sent Done Done false
556 1022 Acked Read 9 Sent Done Done false
557 1023 Acked Write 1 Sent Done Done false
558 1024 Acked Read 5 Sent Done Done false
559 1025 Acked Read 4 Sent Done Done false
560 1026 Acked Write 2 Sent Done Done false
561 1027 Acked Read 1 Sent Done Done false
562 1028 Acked Read 2 Sent Done Done false
563 1029 Acked Read 3 Sent Done Done false
564 1030 Acked Read 2 Sent Done Done false
565 1031 Acked Read 7 Sent Done Done false
566 1032 Acked Write 3 Sent Done Done false
567 1033 Acked Write 4 Sent Done Done false
568 1034 Acked Write 7 Sent Done Done false
569 1035 Acked Read 5 Sent Done Done false
570 1036 Acked Write 9 Sent Done Done false
571 1037 Acked Flush 0 Sent Done Done false
572 1038 Acked Write 3 Sent Done Done false
573 1039 Acked Flush 0 Sent Done Done false
574 1040 Acked Flush 0 Sent Done Done false
575 1041 Acked Write 9 Sent Done Done false
576 1042 Acked Read 9 Sent Done Done false
577 1043 Acked Write 5 Sent Done Done false
578 1044 Acked Read 7 Sent Done Done false
579 1045 Acked Read 3 Sent Done Done false
580 1046 Acked Read 5 Sent Done Done false
581 1047 Acked Read 9 Sent Done Done false
582 1048 Acked Read 2 Sent Done Done false
583 1049 Acked Write 4 Sent Done Done false
584 1050 Acked Write 10 Sent Done Done false
585 1051 Acked Read 7 Sent Done Done false
586 1052 Acked Write 2 Sent Done Done false
587 1053 Acked Write 3 Sent Done Done false
588 1054 Acked Write 6 Sent Done Done false
589 1055 Acked Write 10 Sent Done Done false
590 1056 Acked Write 2 Sent Done Done false
591 1057 Acked Flush 0 Sent Done Done false
592 1058 Acked Write 7 Sent Done Done false
593 1059 Acked Read 4 Sent Done Done false
594 1060 Acked Write 8 Sent Done Done false
595 1061 Acked Flush 0 Sent Done Done false
596 1062 Acked Read 1 Sent Done Done false
597 1063 Acked Read 9 Sent Done Done false
598 1064 Acked Read 9 Sent Done Done false
599 1065 Acked Read 8 Sent Done Done false
600 1066 Acked Flush 0 Sent Done Done false
601 1067 Acked Read 4 Sent Done Done false
602 1068 Acked Read 10 Sent Done Done false
603 1069 Acked Read 6 Sent Done Done false
604 1070 Acked Write 4 Sent Done Done false
605 1071 Acked Read 4 Sent Done Done false
606 1072 Acked Write 2 Sent Done Done false
607 1073 Acked Read 3 Sent Done Done false
608 1074 Acked Write 9 Sent Done Done false
609 1075 Acked Write 3 Sent Done Done false
610 1076 Acked Write 9 Sent Done Done false
611 1077 Acked Read 6 Sent Done Done false
612 1078 Acked Flush 0 Sent Done Done false
613 1079 Acked Flush 0 Sent Done Done false
614 1080 Acked Read 2 Sent Done Done false
615 1081 Acked Read 9 Sent Done Done false
616 1082 Acked Read 7 Sent Done Done false
617 1083 Acked Read 1 Sent Done Done false
618 1084 Acked Read 5 Sent Done Done false
619 1085 Acked Write 10 Sent Done Done false
620 1086 Acked Read 2 Sent Done Done false
621 1087 Acked Read 5 Sent Done Done false
622 1088 Acked Read 2 Sent Done Done false
623 1089 Acked Read 1 Sent Done Done false
624 1090 Acked Read 8 Sent Done Done false
625 1091 Acked Read 2 Sent Done Done false
626 1092 Acked Read 10 Sent Done Done false
627 1093 Acked Flush 0 Sent Done Done false
628 1094 Acked Write 7 Sent Done Done false
629 1095 Acked Write 7 Sent Done Done false
630 1096 Acked Write 8 Sent Done Done false
631 1097 Acked Write 10 Sent Done Done false
632 1098 Acked Read 10 Sent Done Done false
633 1099 Acked Write 4 Sent Done Done false
634 1100 Acked Read 1 Sent Done Done false
635 1101 Acked Read 6 Sent Done Done false
636 1102 Acked Read 1 Sent Done Done false
637 1103 Acked Write 5 Sent Done Done false
638 1104 Acked Write 6 Sent Done Done false
639 1105 Acked Write 10 Sent Done Done false
640 1106 Acked Write 10 Sent Done Done false
641 1107 Acked Write 5 Sent Done Done false
642 1108 Acked Read 3 Sent Done Done false
643 1109 Acked Write 9 Sent Done Done false
644 1110 Acked Write 8 Sent Done Done false
645 1111 Acked Read 5 Sent Done Done false
646 1112 Acked Write 3 Sent Done Done false
647 1113 Acked Write 3 Sent Done Done false
648 1114 Acked Read 10 Sent Done Done false
649 1115 Acked Write 4 Sent Done Done false
650 1116 Acked Write 5 Sent Done Done false
651 1117 Acked Read 1 Sent Done Done false
652 1118 Acked Read 5 Sent Done Done false
653 1119 Acked Read 10 Sent Done Done false
654 1120 Acked Read 1 Sent Done Done false
655 1121 Acked Write 7 Sent Done Done false
656 1122 Acked Read 2 Sent Done Done false
657 1123 Acked Write 3 Sent Done Done false
658 1124 Acked Read 5 Sent Done Done false
659 1125 Acked Write 6 Sent Done Done false
660 1126 Acked Write 7 Sent Done Done false
661 1127 Acked Read 10 Sent Done Done false
662 1128 Acked Read 10 Sent Done Done false
663 1129 Acked Read 2 Sent Done Done false
664 1130 Acked Read 6 Sent Done Done false
665 1131 Acked Read 10 Sent Done Done false
666 1132 Acked Flush 0 Sent Done Done false
667 1133 Acked Write 8 Sent Done Done false
668 1134 Acked Read 10 Sent Done Done false
669 1135 Acked Read 6 Sent Done Done false
670 1136 Acked Write 3 Sent Done Done false
671 1137 Acked Read 10 Sent Done Done false
672 1138 Acked Write 10 Sent Done Done false
673 1139 Acked Write 3 Sent Done Done false
674 1140 Acked Read 6 Sent Done Done false
675 1141 Acked Read 4 Sent Done Done false
676 1142 Acked Read 8 Sent Done Done false
677 1143 Acked Write 7 Sent Done Done false
678 1144 Acked Read 10 Sent Done Done false
679 1145 Acked Write 6 Sent Done Done false
680 1146 Acked Write 5 Sent Done Done false
681 1147 Acked Read 8 Sent Done Done false
682 1148 Acked Read 7 Sent Done Done false
683 1149 Acked Read 7 Sent Done Done false
684 1150 Acked Write 2 Sent Done Done false
685 1151 Acked Write 1 Sent Done Done false
686 1152 Acked Write 9 Sent Done Done false
687 1153 Acked Write 3 Sent Done Done false
688 1154 Acked Read 10 Sent Done Done false
689 1155 Acked Read 7 Sent Done Done false
690 1156 Acked Write 7 Sent Done Done false
691 1157 Acked Read 10 Sent Done Done false
692 1158 Acked Read 3 Sent Done Done false
693 1159 Acked Read 1 Sent Done Done false
694 1160 Acked Read 9 Sent Done Done false
695 1161 Acked Write 4 Sent Done Done false
696 1162 Acked Read 9 Sent Done Done false
697 1163 Acked Read 3 Sent Done Done false
698 1164 Acked Read 3 Sent Done Done false
699 1165 Acked Read 10 Sent Done Done false
700 1166 Acked Write 2 Sent Done Done false
701 1167 Acked Read 6 Sent Done Done false
702 1168 Acked Read 4 Sent Done Done false
703 1169 Acked Write 4 Sent Done Done false
704 1170 Acked Write 9 Sent Done Done false
705 1171 Acked Write 5 Sent Done Done false
706 1172 Acked Read 1 Sent Done Done false
707 1173 Acked Read 8 Sent Done Done false
708 1174 Acked Read 8 Sent Done Done false
709 1175 Acked Flush 0 Sent Done Done false
710 1176 Acked Flush 0 Sent Done Done false
711 1177 Acked Read 10 Sent Done Done false
712 1178 Acked Write 6 Sent Done Done false
713 1179 Acked Write 1 Sent Done Done false
714 1180 Acked Read 1 Sent Done Done false
715 1181 Acked Read 7 Sent Done Done false
716 1182 Acked Write 10 Sent Done Done false
717 1183 Acked Read 7 Sent Done Done false
718 1184 Acked Read 4 Sent Done Done false
719 1185 Acked Write 3 Sent Done Done false
720 1186 Acked Read 7 Sent Done Done false
721 1187 Acked Read 4 Sent Done Done false
722 1188 Acked Write 1 Sent Done Done false
723 1189 Acked Write 6 Sent Done Done false
724 1190 Acked Read 5 Sent Done Done false
725 1191 Acked Read 8 Sent Done Done false
726 1192 Acked Read 1 Sent Done Done false
727 1193 Acked Write 4 Sent Done Done false
728 1194 Acked Flush 0 Sent Done Done false
729 1195 Acked Write 7 Sent Done Done false
730 1196 Acked Read 6 Sent Done Done false
731 1197 Acked Write 9 Sent Done Done false
732 1198 Acked Write 8 Sent Done Done false
733 1199 Acked Flush 0 Sent Done Done false
734 1200 Acked Write 7 Sent Done Done false
735 1201 Acked Read 6 Sent Done Done false
736 1202 Acked Flush 0 Sent Done Done false
737 1203 Acked Write 1 Sent Done Done false
738 1204 Acked Write 8 Sent Done Done false
739 1205 Acked Read 7 Sent Done Done false
740 1206 Acked Read 7 Sent Done Done false
741 1207 Acked Flush 0 Sent Done Done false
742 1208 Acked Read 7 Sent Done Done false
743 1209 Acked Read 8 Sent Done Done false
744 1210 Acked Write 4 Sent Done Done false
745 1211 Acked Read 3 Sent Done Done false
746 1212 Acked Write 6 Sent Done Done false
747 1213 Acked Read 2 Sent Done Done false
748 1214 Acked Write 2 Sent Done Done false
749 1215 Acked Write 8 Sent Done Done false
750 1216 Acked Write 10 Sent Done Done false
751 1217 Acked Read 7 Sent Done Done false
752 1218 Acked Read 9 Sent Done Done false
753 1219 Acked Read 3 Sent Done Done false
754 1220 Acked Write 3 Sent Done Done false
755 1221 Acked Write 7 Sent Done Done false
756 1222 Acked Flush 0 Sent Done Done false
757 1223 Acked Read 2 Sent Done Done false
758 1224 Acked Read 8 Sent Done Done false
759 1225 Acked Read 6 Sent Done Done false
760 1226 Acked Write 4 Sent Done Done false
761 1227 Acked Write 4 Sent Done Done false
762 1228 Acked Read 5 Sent Done Done false
763 1229 Acked Read 4 Sent Done Done false
764 1230 Acked Read 6 Sent Done Done false
765 1231 Acked Write 5 Sent Done Done false
766 1232 Acked Read 3 Sent Done Done false
767 1233 Acked Read 6 Sent Done Done false
768 1234 Acked Write 3 Sent Done Done false
769 1235 Acked Flush 0 Sent Done Done false
770 1236 Acked Read 5 Sent Done Done false
771 1237 Acked Write 3 Sent Done Done false
772 1238 Acked Read 2 Sent Done Done false
773 1239 Acked Write 10 Sent Done Done false
774 1240 Acked Read 6 Sent Done Done false
775 1241 Acked Write 1 Sent Done Done false
776 1242 Acked Flush 0 Sent Done Done false
777 1243 Acked Read 1 Sent Done Done false
778 1244 Acked Flush 0 Sent Done Done false
779 1245 Acked Read 10 Sent Done Done false
780 1246 Acked Write 8 Sent Done Done false
781 1247 Acked Write 5 Sent Done Done false
782 1248 Acked Flush 0 Sent Done Done false
783 1249 Acked Read 7 Sent Done Done false
784 1250 Acked Write 7 Sent Done Done false
785 1251 Acked Write 8 Sent Done Done false
786 1252 Acked Write 1 Sent Done Done false
787 1253 Acked Write 2 Sent Done Done false
788 1254 Acked Write 8 Sent Done Done false
789 1255 Acked Write 8 Sent Done Done false
790 1256 Acked Read 3 Sent Done Done false
791 1257 Acked Read 2 Sent Done Done false
792 1258 Acked Read 2 Sent Done Done false
793 1259 Acked Write 3 Sent Done Done false
794 1260 Acked Read 9 Sent Done Done false
795 1261 Acked Read 3 Sent Done Done false
796 1262 Acked Write 2 Sent Done Done false
797 1263 Acked Write 6 Sent Done Done false
798 1264 Acked Read 8 Sent Done Done false
799 1265 Acked Write 3 Sent Done Done false
800 1266 Acked Write 4 Sent Done Done false
801 1267 Acked Read 1 Sent Done Done false
802 1268 Acked Flush 0 Sent Done Done false
803 1269 Acked Read 1 Sent Done Done false
804 1270 Acked Read 3 Sent Done Done false
805 1271 Acked Read 5 Sent Done Done false
806 1272 Acked Write 8 Sent Done Done false
807 1273 Acked Write 7 Sent Done Done false
808 1274 Acked Read 6 Sent Done Done false
809 1275 Acked Write 8 Sent Done Done false
810 1276 Acked Flush 0 Sent Done Done false
811 1277 Acked Read 10 Sent Done Done false
812 1278 Acked Flush 0 Sent Done Done false
813 1279 Acked Flush 0 Sent Done Done false
814 1280 Acked Flush 0 Sent Done Done false
815 1281 Acked Flush 0 Sent Done Done false
816 1282 Acked Write 5 Sent Done Done false
817 1283 Acked Write 6 Sent Done Done false
818 1284 Acked Read 9 Sent Done Done false
819 1285 Acked Write 5 Sent Done Done false
820 1286 Acked Flush 0 Sent Done Done false
821 1287 Acked Write 10 Sent Done Done false
822 1288 Acked Read 8 Sent Done Done false
823 1289 Acked Write 8 Sent Done Done false
824 1290 Acked Read 6 Sent Done Done false
825 1291 Acked Write 1 Sent Done Done false
826 1292 Acked Write 9 Sent Done Done false
827 1293 Acked Read 1 Sent Done Done false
828 1294 Acked Read 4 Sent Done Done false
829 1295 Acked Read 2 Sent Done Done false
830 1296 Acked Write 6 Sent Done Done false
831 1297 Acked Flush 0 Sent Done Done false
832 1298 Acked Write 1 Sent Done Done false
833 1299 Acked Read 1 Sent Done Done false
834 1300 Acked Read 3 Sent Done Done false
835 STATES DS:0 DS:1 DS:2 TOTAL
836 Sent 301 0 0 301
837 Done 0 301 301 602
838 Skipped 0 0 0 0
839 Error 0 0 0 0
840 Last Flush: None Some(JobId(1297)) Some(JobId(1297))
841 Downstairs last five completed:
842 Upstairs last five completed: 1300 1299 1298 1297 1296
843 CLIENT: Up:0 ds:301 act:2
8442025-02-07T20:07:22.222ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
8452025-02-07T20:07:22.941ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
846 [8810] got start action
847 [8810] I am going to restart
848 Starting downstairs at port 8810
849 Make output file at "/tmp/dsc/downstairs-8810.txt"
850 [8810][0] reports Starting
851 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1210)
852 [8810][0] reports Running
8532025-02-07T20:07:23.943ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
8542025-02-07T20:07:24.944ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
855 ----------------------------------------------------------------
856 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
857 States: Offline Active Active
858 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
859 1000 Acked Flush 0 Sent Done Done false
860 1001 Acked Read 1 Sent Done Done false
861 1002 Acked Read 8 Sent Done Done false
862 1003 Acked Read 9 Sent Done Done false
863 1004 Acked Read 1 Sent Done Done false
864 1005 Acked Write 9 Sent Done Done false
865 1006 Acked Read 9 Sent Done Done false
866 1007 Acked Write 4 Sent Done Done false
867 1008 Acked Write 7 Sent Done Done false
868 1009 Acked Read 2 Sent Done Done false
869 1010 Acked Write 5 Sent Done Done false
870 1011 Acked Read 10 Sent Done Done false
871 1012 Acked Read 7 Sent Done Done false
872 1013 Acked Write 7 Sent Done Done false
873 1014 Acked Write 8 Sent Done Done false
874 1015 Acked Write 2 Sent Done Done false
875 1016 Acked Write 9 Sent Done Done false
876 1017 Acked Flush 0 Sent Done Done false
877 1018 Acked Write 5 Sent Done Done false
878 1019 Acked Read 10 Sent Done Done false
879 1020 Acked Read 8 Sent Done Done false
880 1021 Acked Read 10 Sent Done Done false
881 1022 Acked Read 9 Sent Done Done false
882 1023 Acked Write 1 Sent Done Done false
883 1024 Acked Read 5 Sent Done Done false
884 1025 Acked Read 4 Sent Done Done false
885 1026 Acked Write 2 Sent Done Done false
886 1027 Acked Read 1 Sent Done Done false
887 1028 Acked Read 2 Sent Done Done false
888 1029 Acked Read 3 Sent Done Done false
889 1030 Acked Read 2 Sent Done Done false
890 1031 Acked Read 7 Sent Done Done false
891 1032 Acked Write 3 Sent Done Done false
892 1033 Acked Write 4 Sent Done Done false
893 1034 Acked Write 7 Sent Done Done false
894 1035 Acked Read 5 Sent Done Done false
895 1036 Acked Write 9 Sent Done Done false
896 1037 Acked Flush 0 Sent Done Done false
897 1038 Acked Write 3 Sent Done Done false
898 1039 Acked Flush 0 Sent Done Done false
899 1040 Acked Flush 0 Sent Done Done false
900 1041 Acked Write 9 Sent Done Done false
901 1042 Acked Read 9 Sent Done Done false
902 1043 Acked Write 5 Sent Done Done false
903 1044 Acked Read 7 Sent Done Done false
904 1045 Acked Read 3 Sent Done Done false
905 1046 Acked Read 5 Sent Done Done false
906 1047 Acked Read 9 Sent Done Done false
907 1048 Acked Read 2 Sent Done Done false
908 1049 Acked Write 4 Sent Done Done false
909 1050 Acked Write 10 Sent Done Done false
910 1051 Acked Read 7 Sent Done Done false
911 1052 Acked Write 2 Sent Done Done false
912 1053 Acked Write 3 Sent Done Done false
913 1054 Acked Write 6 Sent Done Done false
914 1055 Acked Write 10 Sent Done Done false
915 1056 Acked Write 2 Sent Done Done false
916 1057 Acked Flush 0 Sent Done Done false
917 1058 Acked Write 7 Sent Done Done false
918 1059 Acked Read 4 Sent Done Done false
919 1060 Acked Write 8 Sent Done Done false
920 1061 Acked Flush 0 Sent Done Done false
921 1062 Acked Read 1 Sent Done Done false
922 1063 Acked Read 9 Sent Done Done false
923 1064 Acked Read 9 Sent Done Done false
924 1065 Acked Read 8 Sent Done Done false
925 1066 Acked Flush 0 Sent Done Done false
926 1067 Acked Read 4 Sent Done Done false
927 1068 Acked Read 10 Sent Done Done false
928 1069 Acked Read 6 Sent Done Done false
929 1070 Acked Write 4 Sent Done Done false
930 1071 Acked Read 4 Sent Done Done false
931 1072 Acked Write 2 Sent Done Done false
932 1073 Acked Read 3 Sent Done Done false
933 1074 Acked Write 9 Sent Done Done false
934 1075 Acked Write 3 Sent Done Done false
935 1076 Acked Write 9 Sent Done Done false
936 1077 Acked Read 6 Sent Done Done false
937 1078 Acked Flush 0 Sent Done Done false
938 1079 Acked Flush 0 Sent Done Done false
939 1080 Acked Read 2 Sent Done Done false
940 1081 Acked Read 9 Sent Done Done false
941 1082 Acked Read 7 Sent Done Done false
942 1083 Acked Read 1 Sent Done Done false
943 1084 Acked Read 5 Sent Done Done false
944 1085 Acked Write 10 Sent Done Done false
945 1086 Acked Read 2 Sent Done Done false
946 1087 Acked Read 5 Sent Done Done false
947 1088 Acked Read 2 Sent Done Done false
948 1089 Acked Read 1 Sent Done Done false
949 1090 Acked Read 8 Sent Done Done false
950 1091 Acked Read 2 Sent Done Done false
951 1092 Acked Read 10 Sent Done Done false
952 1093 Acked Flush 0 Sent Done Done false
953 1094 Acked Write 7 Sent Done Done false
954 1095 Acked Write 7 Sent Done Done false
955 1096 Acked Write 8 Sent Done Done false
956 1097 Acked Write 10 Sent Done Done false
957 1098 Acked Read 10 Sent Done Done false
958 1099 Acked Write 4 Sent Done Done false
959 1100 Acked Read 1 Sent Done Done false
960 1101 Acked Read 6 Sent Done Done false
961 1102 Acked Read 1 Sent Done Done false
962 1103 Acked Write 5 Sent Done Done false
963 1104 Acked Write 6 Sent Done Done false
964 1105 Acked Write 10 Sent Done Done false
965 1106 Acked Write 10 Sent Done Done false
966 1107 Acked Write 5 Sent Done Done false
967 1108 Acked Read 3 Sent Done Done false
968 1109 Acked Write 9 Sent Done Done false
969 1110 Acked Write 8 Sent Done Done false
970 1111 Acked Read 5 Sent Done Done false
971 1112 Acked Write 3 Sent Done Done false
972 1113 Acked Write 3 Sent Done Done false
973 1114 Acked Read 10 Sent Done Done false
974 1115 Acked Write 4 Sent Done Done false
975 1116 Acked Write 5 Sent Done Done false
976 1117 Acked Read 1 Sent Done Done false
977 1118 Acked Read 5 Sent Done Done false
978 1119 Acked Read 10 Sent Done Done false
979 1120 Acked Read 1 Sent Done Done false
980 1121 Acked Write 7 Sent Done Done false
981 1122 Acked Read 2 Sent Done Done false
982 1123 Acked Write 3 Sent Done Done false
983 1124 Acked Read 5 Sent Done Done false
984 1125 Acked Write 6 Sent Done Done false
985 1126 Acked Write 7 Sent Done Done false
986 1127 Acked Read 10 Sent Done Done false
987 1128 Acked Read 10 Sent Done Done false
988 1129 Acked Read 2 Sent Done Done false
989 1130 Acked Read 6 Sent Done Done false
990 1131 Acked Read 10 Sent Done Done false
991 1132 Acked Flush 0 Sent Done Done false
992 1133 Acked Write 8 Sent Done Done false
993 1134 Acked Read 10 Sent Done Done false
994 1135 Acked Read 6 Sent Done Done false
995 1136 Acked Write 3 Sent Done Done false
996 1137 Acked Read 10 Sent Done Done false
997 1138 Acked Write 10 Sent Done Done false
998 1139 Acked Write 3 Sent Done Done false
999 1140 Acked Read 6 Sent Done Done false
1000 1141 Acked Read 4 Sent Done Done false
1001 1142 Acked Read 8 Sent Done Done false
1002 1143 Acked Write 7 Sent Done Done false
1003 1144 Acked Read 10 Sent Done Done false
1004 1145 Acked Write 6 Sent Done Done false
1005 1146 Acked Write 5 Sent Done Done false
1006 1147 Acked Read 8 Sent Done Done false
1007 1148 Acked Read 7 Sent Done Done false
1008 1149 Acked Read 7 Sent Done Done false
1009 1150 Acked Write 2 Sent Done Done false
1010 1151 Acked Write 1 Sent Done Done false
1011 1152 Acked Write 9 Sent Done Done false
1012 1153 Acked Write 3 Sent Done Done false
1013 1154 Acked Read 10 Sent Done Done false
1014 1155 Acked Read 7 Sent Done Done false
1015 1156 Acked Write 7 Sent Done Done false
1016 1157 Acked Read 10 Sent Done Done false
1017 1158 Acked Read 3 Sent Done Done false
1018 1159 Acked Read 1 Sent Done Done false
1019 1160 Acked Read 9 Sent Done Done false
1020 1161 Acked Write 4 Sent Done Done false
1021 1162 Acked Read 9 Sent Done Done false
1022 1163 Acked Read 3 Sent Done Done false
1023 1164 Acked Read 3 Sent Done Done false
1024 1165 Acked Read 10 Sent Done Done false
1025 1166 Acked Write 2 Sent Done Done false
1026 1167 Acked Read 6 Sent Done Done false
1027 1168 Acked Read 4 Sent Done Done false
1028 1169 Acked Write 4 Sent Done Done false
1029 1170 Acked Write 9 Sent Done Done false
1030 1171 Acked Write 5 Sent Done Done false
1031 1172 Acked Read 1 Sent Done Done false
1032 1173 Acked Read 8 Sent Done Done false
1033 1174 Acked Read 8 Sent Done Done false
1034 1175 Acked Flush 0 Sent Done Done false
1035 1176 Acked Flush 0 Sent Done Done false
1036 1177 Acked Read 10 Sent Done Done false
1037 1178 Acked Write 6 Sent Done Done false
1038 1179 Acked Write 1 Sent Done Done false
1039 1180 Acked Read 1 Sent Done Done false
1040 1181 Acked Read 7 Sent Done Done false
1041 1182 Acked Write 10 Sent Done Done false
1042 1183 Acked Read 7 Sent Done Done false
1043 1184 Acked Read 4 Sent Done Done false
1044 1185 Acked Write 3 Sent Done Done false
1045 1186 Acked Read 7 Sent Done Done false
1046 1187 Acked Read 4 Sent Done Done false
1047 1188 Acked Write 1 Sent Done Done false
1048 1189 Acked Write 6 Sent Done Done false
1049 1190 Acked Read 5 Sent Done Done false
1050 1191 Acked Read 8 Sent Done Done false
1051 1192 Acked Read 1 Sent Done Done false
1052 1193 Acked Write 4 Sent Done Done false
1053 1194 Acked Flush 0 Sent Done Done false
1054 1195 Acked Write 7 Sent Done Done false
1055 1196 Acked Read 6 Sent Done Done false
1056 1197 Acked Write 9 Sent Done Done false
1057 1198 Acked Write 8 Sent Done Done false
1058 1199 Acked Flush 0 Sent Done Done false
1059 1200 Acked Write 7 Sent Done Done false
1060 1201 Acked Read 6 Sent Done Done false
1061 1202 Acked Flush 0 Sent Done Done false
1062 1203 Acked Write 1 Sent Done Done false
1063 1204 Acked Write 8 Sent Done Done false
1064 1205 Acked Read 7 Sent Done Done false
1065 1206 Acked Read 7 Sent Done Done false
1066 1207 Acked Flush 0 Sent Done Done false
1067 1208 Acked Read 7 Sent Done Done false
1068 1209 Acked Read 8 Sent Done Done false
1069 1210 Acked Write 4 Sent Done Done false
1070 1211 Acked Read 3 Sent Done Done false
1071 1212 Acked Write 6 Sent Done Done false
1072 1213 Acked Read 2 Sent Done Done false
1073 1214 Acked Write 2 Sent Done Done false
1074 1215 Acked Write 8 Sent Done Done false
1075 1216 Acked Write 10 Sent Done Done false
1076 1217 Acked Read 7 Sent Done Done false
1077 1218 Acked Read 9 Sent Done Done false
1078 1219 Acked Read 3 Sent Done Done false
1079 1220 Acked Write 3 Sent Done Done false
1080 1221 Acked Write 7 Sent Done Done false
1081 1222 Acked Flush 0 Sent Done Done false
1082 1223 Acked Read 2 Sent Done Done false
1083 1224 Acked Read 8 Sent Done Done false
1084 1225 Acked Read 6 Sent Done Done false
1085 1226 Acked Write 4 Sent Done Done false
1086 1227 Acked Write 4 Sent Done Done false
1087 1228 Acked Read 5 Sent Done Done false
1088 1229 Acked Read 4 Sent Done Done false
1089 1230 Acked Read 6 Sent Done Done false
1090 1231 Acked Write 5 Sent Done Done false
1091 1232 Acked Read 3 Sent Done Done false
1092 1233 Acked Read 6 Sent Done Done false
1093 1234 Acked Write 3 Sent Done Done false
1094 1235 Acked Flush 0 Sent Done Done false
1095 1236 Acked Read 5 Sent Done Done false
1096 1237 Acked Write 3 Sent Done Done false
1097 1238 Acked Read 2 Sent Done Done false
1098 1239 Acked Write 10 Sent Done Done false
1099 1240 Acked Read 6 Sent Done Done false
1100 1241 Acked Write 1 Sent Done Done false
1101 1242 Acked Flush 0 Sent Done Done false
1102 1243 Acked Read 1 Sent Done Done false
1103 1244 Acked Flush 0 Sent Done Done false
1104 1245 Acked Read 10 Sent Done Done false
1105 1246 Acked Write 8 Sent Done Done false
1106 1247 Acked Write 5 Sent Done Done false
1107 1248 Acked Flush 0 Sent Done Done false
1108 1249 Acked Read 7 Sent Done Done false
1109 1250 Acked Write 7 Sent Done Done false
1110 1251 Acked Write 8 Sent Done Done false
1111 1252 Acked Write 1 Sent Done Done false
1112 1253 Acked Write 2 Sent Done Done false
1113 1254 Acked Write 8 Sent Done Done false
1114 1255 Acked Write 8 Sent Done Done false
1115 1256 Acked Read 3 Sent Done Done false
1116 1257 Acked Read 2 Sent Done Done false
1117 1258 Acked Read 2 Sent Done Done false
1118 1259 Acked Write 3 Sent Done Done false
1119 1260 Acked Read 9 Sent Done Done false
1120 1261 Acked Read 3 Sent Done Done false
1121 1262 Acked Write 2 Sent Done Done false
1122 1263 Acked Write 6 Sent Done Done false
1123 1264 Acked Read 8 Sent Done Done false
1124 1265 Acked Write 3 Sent Done Done false
1125 1266 Acked Write 4 Sent Done Done false
1126 1267 Acked Read 1 Sent Done Done false
1127 1268 Acked Flush 0 Sent Done Done false
1128 1269 Acked Read 1 Sent Done Done false
1129 1270 Acked Read 3 Sent Done Done false
1130 1271 Acked Read 5 Sent Done Done false
1131 1272 Acked Write 8 Sent Done Done false
1132 1273 Acked Write 7 Sent Done Done false
1133 1274 Acked Read 6 Sent Done Done false
1134 1275 Acked Write 8 Sent Done Done false
1135 1276 Acked Flush 0 Sent Done Done false
1136 1277 Acked Read 10 Sent Done Done false
1137 1278 Acked Flush 0 Sent Done Done false
1138 1279 Acked Flush 0 Sent Done Done false
1139 1280 Acked Flush 0 Sent Done Done false
1140 1281 Acked Flush 0 Sent Done Done false
1141 1282 Acked Write 5 Sent Done Done false
1142 1283 Acked Write 6 Sent Done Done false
1143 1284 Acked Read 9 Sent Done Done false
1144 1285 Acked Write 5 Sent Done Done false
1145 1286 Acked Flush 0 Sent Done Done false
1146 1287 Acked Write 10 Sent Done Done false
1147 1288 Acked Read 8 Sent Done Done false
1148 1289 Acked Write 8 Sent Done Done false
1149 1290 Acked Read 6 Sent Done Done false
1150 1291 Acked Write 1 Sent Done Done false
1151 1292 Acked Write 9 Sent Done Done false
1152 1293 Acked Read 1 Sent Done Done false
1153 1294 Acked Read 4 Sent Done Done false
1154 1295 Acked Read 2 Sent Done Done false
1155 1296 Acked Write 6 Sent Done Done false
1156 1297 Acked Flush 0 Sent Done Done false
1157 1298 Acked Write 1 Sent Done Done false
1158 1299 Acked Read 1 Sent Done Done false
1159 1300 Acked Read 3 Sent Done Done false
1160 STATES DS:0 DS:1 DS:2 TOTAL
1161 Sent 301 0 0 301
1162 Done 0 301 301 602
1163 Skipped 0 0 0 0
1164 Error 0 0 0 0
1165 Last Flush: None Some(JobId(1297)) Some(JobId(1297))
1166 Downstairs last five completed:
1167 Upstairs last five completed: 1300 1299 1298 1297 1296
1168 CLIENT: Up:0 ds:301 act:2
11692025-02-07T20:07:26.223ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11702025-02-07T20:07:26.947ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11712025-02-07T20:07:27.475ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11722025-02-07T20:07:27.475ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56914) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11732025-02-07T20:07:27.476ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11742025-02-07T20:07:27.476ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11752025-02-07T20:07:27.476ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11762025-02-07T20:07:27.476ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11772025-02-07T20:07:27.476ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11782025-02-07T20:07:27.476ZINFOcrucible: [0] client re-new 301 jobs since flush None = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11792025-02-07T20:07:27.477ZINFOcrucible: [0] Replayed 301 jobs since flush: None = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1180 ----------------------------------------------------------------
1181 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1182 Downstairs last five completed: 1301 1300 1299 1298 1297
1183 Upstairs last five completed: 1301 1300 1299 1298 1297
1184 CLIENT: Up:0 ds:0 act:3
1185 Replay: All jobs finished
1186 Main task has work to do, go find it
1187 got dsc Stop(2)
1188 stop 2
1189 Feb 07 20:07:29.725 INFO request completed, latency_us: 367, response_code: 204, uri:[8830] Got stop action so:false kr:false
1190 /stop/cid/2, method: POST, req_id: 3dc4b88e-2bb5-4df3-826d-cbf06bbc36e2, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
11912025-02-07T20:07:29.725ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11922025-02-07T20:07:29.725ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11932025-02-07T20:07:29.725ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1194 Feb 07 20:07:29.725 INFO request completed, latency_us: 147, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1e7e85e1-66b8-4f09-bac1-71b2af38d25b, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
11952025-02-07T20:07:29.725ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1196 [8830] Exited with: ExitStatus(unix_wait_status(9))
1197 [8830][2] reports Exit
1198 Feb 07 20:07:33.728 INFO request completed, latency_us: 325, response_code: 200, uri: /state/cid/2, method: GET, req_id: fa693631-aa64-4b01-b018-96220bbebf3a, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
1199 001/300 Write block 1761 len 4096 data: 2
1200 002/300 Read block 3218 len 12288
1201 003/300 Write block 1524 len 32768 data: 2 2 2 2 2 2 2 2
1202 004/300 Flush
1203 005/300 Write block 447 len 16384 data: 2 2 2 2
1204 006/300 Read block 832 len 4096
1205 007/300 Read block 3041 len 40960
1206 008/300 Read block 3265 len 40960
1207 009/300 Write block 607 len 32768 data: 2 2 2 2 2 2 2 2
1208 010/300 Read block 4317 len 32768
1209 011/300 Read block 1944 len 32768
1210 012/300 Write block 1167 len 12288 data: 2 2 2
1211 013/300 Read block 1685 len 32768
1212 014/300 Write block 2162 len 24576 data: 2 2 2 2 2 2
1213 015/300 Read block 3600 len 12288
1214 016/300 Read block 4470 len 40960
1215 017/300 Write block 3147 len 8192 data: 3 3
1216 018/300 Flush
1217 019/300 Write block 3570 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1218 020/300 Read block 349 len 12288
1219 021/300 Write block 2303 len 40960 data: 2 2 2 2 2 4 4 4 4 4
1220 022/300 Write block 3775 len 4096 data: 2
1221 023/300 Write block 998 len 40960 data: 2 2 2 3 3 3 3 3 3 2
1222 024/300 Read block 2240 len 32768
1223 025/300 Write block 162 len 16384 data: 2 2 2 2
1224 026/300 Flush
1225 027/300 Read block 3337 len 36864
1226 028/300 Read block 345 len 28672
1227 029/300 Read block 264 len 12288
1228 030/300 Write block 3958 len 8192 data: 2 2
1229 031/300 Write block 4123 len 36864 data: 2 2 2 2 2 2 2 2 2
1230 032/300 Write block 3718 len 32768 data: 2 2 3 3 3 2 2 2
1231 033/300 Read block 346 len 4096
1232 034/300 Write block 1918 len 8192 data: 2 2
1233 035/300 Read block 4696 len 28672
1234 036/300 Write block 4061 len 32768 data: 2 2 2 2 2 2 2 2
1235 037/300 Read block 1261 len 36864
1236 038/300 Read block 3836 len 36864
1237 039/300 Read block 3125 len 28672
1238 040/300 Write block 1651 len 36864 data: 3 3 3 3 3 3 3 2 2
1239 041/300 Flush
1240 042/300 Write block 3444 len 16384 data: 3 3 3 3
1241 043/300 Read block 844 len 32768
1242 044/300 Write block 144 len 12288 data: 2 2 2
1243 045/300 Read block 1189 len 36864
1244 046/300 Read block 2110 len 24576
1245 047/300 Read block 4267 len 4096
1246 048/300 Write block 1385 len 32768 data: 2 2 2 2 2 2 2 2
1247 049/300 Read block 4728 len 32768
1248 050/300 Read block 235 len 8192
1249 051/300 Read block 3943 len 12288
1250 052/300 Read block 2110 len 36864
1251 053/300 Read block 788 len 16384
1252 054/300 Read block 1959 len 20480
1253 055/300 Read block 2143 len 16384
1254 056/300 Read block 2753 len 12288
1255 057/300 Write block 4404 len 20480 data: 2 2 2 3 3
1256 058/300 Read block 1440 len 12288
1257 059/300 Read block 2668 len 20480
1258 060/300 Flush
1259 061/300 Read block 4735 len 24576
1260 062/300 Read block 666 len 12288
1261 063/300 Read block 1961 len 20480
1262 064/300 Write block 419 len 12288 data: 2 2 2
1263 065/300 Write block 3257 len 4096 data: 2
1264 066/300 Read block 3149 len 32768
1265 067/300 Write block 2514 len 16384 data: 2 2 2 2
1266 068/300 Read block 299 len 28672
1267 069/300 Write block 622 len 12288 data: 2 2 2
1268 070/300 Write block 1152 len 28672 data: 2 2 2 2 2 2 2
1269 071/300 Read block 2445 len 8192
1270 072/300 Write block 1106 len 24576 data: 2 2 3 3 3 3
1271 073/300 Flush
1272 074/300 Write block 3681 len 40960 data: 3 3 2 2 2 2 2 2 2 2
1273 075/300 Write block 2429 len 28672 data: 2 2 2 2 2 2 2
1274 076/300 Write block 4835 len 28672 data: 2 2 2 2 2 2 2
1275 077/300 Read block 3205 len 8192
1276 078/300 Write block 1275 len 16384 data: 2 2 2 2
1277 079/300 Write block 4865 len 12288 data: 2 2 2
1278 080/300 Read block 1461 len 20480
1279 081/300 Write block 1295 len 36864 data: 2 2 2 2 2 2 2 2 2
1280 082/300 Write block 3037 len 36864 data: 2 2 2 2 2 2 2 2 2
1281 083/300 Write block 2729 len 20480 data: 2 2 2 2 2
1282 084/300 Write block 4747 len 24576 data: 2 2 2 2 2 2
1283 085/300 Read block 4656 len 28672
1284 086/300 Read block 1205 len 16384
1285 087/300 Flush
1286 088/300 Write block 3662 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1287 089/300 Write block 758 len 8192 data: 2 2
1288 090/300 Flush
1289 091/300 Read block 4713 len 8192
1290 092/300 Read block 2406 len 12288
1291 093/300 Read block 2240 len 4096
1292 094/300 Write block 3677 len 4096 data: 3
1293 095/300 Read block 3616 len 20480
1294 096/300 Flush
1295 097/300 Read block 552 len 24576
1296 098/300 Read block 1746 len 40960
1297 099/300 Read block 3261 len 32768
1298 100/300 Read block 874 len 4096
1299 101/300 Write block 1257 len 12288 data: 2 2 2
1300 102/300 Write block 3118 len 40960 data: 2 2 2 2 2 2 2 3 3 3
1301 103/300 Read block 4758 len 36864
1302 104/300 Read block 1216 len 36864
1303 105/300 Read block 2019 len 36864
1304 106/300 Write block 2753 len 32768 data: 2 2 2 2 2 2 2 2
1305 107/300 Flush
1306 108/300 Read block 3403 len 4096
1307 109/300 Read block 1199 len 28672
1308 110/300 Read block 4256 len 12288
1309 111/300 Read block 833 len 24576
1310 112/300 Read block 4095 len 8192
1311 113/300 Read block 2534 len 40960
1312 114/300 Write block 1282 len 20480 data: 2 2 2 2 2
1313 115/300 Write block 791 len 36864 data: 2 2 2 2 2 2 2 2 2
1314 116/300 Write block 2452 len 28672 data: 3 2 2 2 2 2 2
1315 117/300 Read block 4146 len 28672
1316 118/300 Write block 3075 len 12288 data: 2 2 2
1317 119/300 Write block 1771 len 4096 data: 2
1318 120/300 Read block 3202 len 28672
1319 121/300 Read block 3433 len 20480
1320 122/300 Read block 2878 len 28672
1321 123/300 Write block 1266 len 4096 data: 3
1322 124/300 Read block 2079 len 24576
1323 125/300 Read block 917 len 36864
1324 126/300 Read block 3217 len 16384
1325 127/300 Read block 1119 len 20480
1326 128/300 Flush
1327 129/300 Write block 4886 len 20480 data: 2 2 2 2 2
1328 130/300 Write block 1684 len 32768 data: 2 2 2 2 2 2 2 2
1329 131/300 Read block 4908 len 16384
1330 132/300 Write block 4469 len 16384 data: 2 2 2 2
1331 133/300 Read block 2789 len 16384
1332 134/300 Read block 1746 len 12288
1333 135/300 Read block 256 len 8192
1334 136/300 Flush
1335 137/300 Write block 3883 len 24576 data: 2 2 2 2 2 2
1336 138/300 Read block 197 len 8192
1337 139/300 Read block 1979 len 36864
1338 140/300 Write block 886 len 28672 data: 2 2 2 2 2 2 2
1339 141/300 Read block 2212 len 24576
1340 142/300 Read block 3417 len 32768
1341 143/300 Write block 4040 len 28672 data: 2 2 2 2 2 2 2
1342 144/300 Write block 3600 len 8192 data: 2 2
1343 145/300 Read block 1429 len 40960
1344 146/300 Flush
1345 147/300 Write block 4895 len 20480 data: 2 2 2 2 2
1346 148/300 Read block 1434 len 12288
1347 149/300 Write block 1566 len 4096 data: 2
1348 150/300 Read block 3509 len 16384
1349 151/300 Read block 2 len 24576
1350 152/300 Read block 982 len 24576
1351 153/300 Write block 660 len 36864 data: 2 2 2 2 2 2 2 2 2
1352 154/300 Write block 2261 len 28672 data: 2 2 2 2 2 2 2
1353 155/300 Read block 1076 len 20480
1354 156/300 Flush
1355 157/300 Read block 3191 len 8192
1356 158/300 Write block 4920 len 8192 data: 2 2
1357 159/300 Flush
1358 160/300 Write block 71 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1359 161/300 Read block 4121 len 32768
1360 162/300 Read block 1619 len 16384
1361 163/300 Read block 2522 len 16384
1362 164/300 Read block 3753 len 16384
1363 165/300 Flush
1364 166/300 Read block 4390 len 20480
1365 167/300 Read block 1472 len 32768
1366 168/300 Flush
1367 169/300 Read block 3087 len 28672
1368 170/300 Write block 547 len 8192 data: 2 2
1369 171/300 Write block 2457 len 20480 data: 3 3 2 2 2
1370 172/300 Write block 993 len 12288 data: 2 2 2
1371 173/300 Read block 110 len 4096
1372 174/300 Write block 4295 len 8192 data: 2 2
1373 175/300 Read block 1833 len 12288
1374 176/300 Read block 3846 len 28672
1375 177/300 Read block 2909 len 4096
1376 178/300 Write block 4357 len 16384 data: 2 2 2 2
1377 179/300 Read block 2587 len 8192
1378 180/300 Write block 4832 len 36864 data: 2 2 2 3 3 3 3 3 3
1379 181/300 Read block 1682 len 20480
1380 182/300 Write block 4295 len 16384 data: 3 3 2 2
1381 183/300 Write block 4475 len 24576 data: 2 2 2 2 2 2
1382 184/300 Read block 2453 len 24576
1383 185/300 Write block 1740 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1384 186/300 Read block 3321 len 12288
1385 187/300 Write block 1831 len 32768 data: 2 2 2 2 2 2 2 2
1386 188/300 Write block 1574 len 36864 data: 2 2 2 2 2 2 2 2 2
1387 189/300 Read block 326 len 36864
1388 190/300 Read block 3898 len 12288
1389 191/300 Read block 4081 len 32768
1390 192/300 Write block 2713 len 4096 data: 2
1391 193/300 Read block 315 len 8192
1392 194/300 Write block 334 len 16384 data: 2 2 2 2
1393 195/300 Read block 4675 len 36864
1394 196/300 Write block 3414 len 16384 data: 2 2 2 2
1395 197/300 Read block 783 len 12288
1396 198/300 Write block 3912 len 32768 data: 2 2 2 2 2 2 2 2
1397 199/300 Read block 758 len 36864
1398 200/300 Flush
1399 201/300 Read block 4047 len 12288
1400 202/300 Write block 3732 len 16384 data: 3 3 3 3
1401 203/300 Read block 1277 len 8192
1402 204/300 Read block 3311 len 28672
1403 205/300 Read block 1641 len 40960
1404 206/300 Read block 2074 len 16384
1405 207/300 Flush
1406 208/300 Read block 1180 len 24576
1407 209/300 Read block 3934 len 28672
1408 210/300 Write block 239 len 28672 data: 2 2 2 2 2 2 2
1409 211/300 Read block 3916 len 24576
1410 212/300 Flush
1411 213/300 Write block 3474 len 20480 data: 2 2 2 2 2
1412 214/300 Write block 4258 len 24576 data: 2 2 2 2 2 2
1413 215/300 Write block 2375 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1414 216/300 Read block 3639 len 4096
1415 217/300 Read block 2972 len 36864
1416 218/300 Write block 4493 len 12288 data: 3 3 3
1417 219/300 Write block 1159 len 8192 data: 2 2
1418 220/300 Read block 1801 len 16384
1419 221/300 Write block 1987 len 12288 data: 2 2 2
1420 222/300 Write block 2588 len 8192 data: 2 2
1421 223/300 Read block 703 len 12288
1422 224/300 Read block 126 len 8192
1423 225/300 Write block 1959 len 8192 data: 2 2
1424 226/300 Flush
1425 227/300 Write block 536 len 32768 data: 2 2 2 2 2 2 2 2
1426 228/300 Read block 266 len 36864
1427 229/300 Read block 277 len 12288
1428 230/300 Write block 2313 len 8192 data: 4 4
1429 231/300 Read block 1784 len 12288
1430 232/300 Read block 4376 len 40960
1431 233/300 Read block 3974 len 8192
1432 234/300 Read block 3906 len 4096
1433 235/300 Write block 3926 len 12288 data: 2 2 2
1434 236/300 Write block 2555 len 12288 data: 2 2 2
1435 237/300 Read block 2531 len 4096
1436 238/300 Read block 916 len 4096
1437 239/300 Read block 377 len 36864
1438 240/300 Read block 1271 len 28672
1439 241/300 Read block 2735 len 32768
1440 242/300 Read block 1844 len 32768
1441 243/300 Write block 3471 len 4096 data: 2
1442 244/300 Read block 2672 len 24576
1443 245/300 Write block 2140 len 36864 data: 2 2 2 2 2 2 2 2 2
1444 246/300 Read block 675 len 12288
1445 247/300 Read block 4002 len 12288
1446 248/300 Write block 1980 len 36864 data: 3 3 3 3 3 3 2 3 3
1447 249/300 Write block 1886 len 28672 data: 2 2 2 2 2 2 2
1448 250/300 Write block 2382 len 36864 data: 3 3 3 2 2 2 2 2 2
1449 251/300 Write block 2860 len 20480 data: 3 3 3 2 2
1450 252/300 Write block 4062 len 40960 data: 3 3 3 3 3 3 3 2 2 2
1451 253/300 Write block 2312 len 24576 data: 5 5 5 3 2 2
1452 254/300 Write block 4103 len 12288 data: 2 2 2
1453 255/300 Write block 20 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1454 256/300 Read block 431 len 4096
1455 257/300 Read block 728 len 4096
1456 258/300 Read block 4764 len 36864
1457 259/300 Read block 4410 len 24576
1458 260/300 Read block 336 len 36864
1459 261/300 Read block 2132 len 8192
1460 262/300 Read block 1828 len 4096
1461 263/300 Flush
1462 264/300 Write block 583 len 16384 data: 2 2 2 2
1463 265/300 Read block 4268 len 4096
1464 266/300 Write block 1506 len 24576 data: 2 2 2 2 2 2
1465 267/300 Write block 4328 len 20480 data: 3 2 2 2 2
1466 268/300 Flush
1467 269/300 Write block 1247 len 12288 data: 2 2 2
1468 270/300 Write block 2793 len 40960 data: 2 2 2 2 2 2 2 2 3 4
1469 271/300 Read block 1256 len 32768
1470 272/300 Write block 4222 len 8192 data: 2 2
1471 273/300 Flush
1472 274/300 Read block 630 len 40960
1473 275/300 Write block 1792 len 32768 data: 2 2 2 2 2 2 2 2
1474 276/300 Read block 4840 len 8192
1475 277/300 Write block 822 len 20480 data: 2 2 2 2 2
1476 278/300 Write block 2953 len 20480 data: 2 2 2 2 2
1477 279/300 Write block 1461 len 8192 data: 2 2
1478 280/300 Write block 3859 len 8192 data: 2 2
1479 281/300 Write block 3196 len 20480 data: 2 2 2 2 2
1480 282/300 Read block 1742 len 28672
1481 283/300 Read block 476 len 4096
1482 284/300 Write block 2338 len 20480 data: 2 2 2 2 2
1483 285/300 Write block 4325 len 4096 data: 2
1484 286/300 Read block 4253 len 28672
1485 287/300 Write block 1084 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1486 288/300 Write block 3459 len 8192 data: 2 2
1487 289/300 Write block 2078 len 40960 data: 3 2 2 2 2 2 2 2 2 2
1488 290/300 Read block 2512 len 8192
1489 291/300 Write block 3946 len 12288 data: 2 2 2
1490 292/300 Write block 3746 len 32768 data: 2 2 2 2 2 2 2 2
1491 293/300 Read block 3640 len 20480
1492 294/300 Write block 3412 len 24576 data: 2 2 3 3 3 3
1493 295/300 Read block 1665 len 4096
1494 296/300 Write block 4331 len 24576 data: 3 3 2 2 2 2
1495 297/300 Write block 3213 len 36864 data: 3 2 2 2 2 2 2 2 2
1496 298/300 Write block 2942 len 32768 data: 2 2 2 2 2 2 2 2
1497 299/300 Read block 3721 len 28672
1498 300/300 Read block 4953 len 16384
1499 301/300 Read block 3438 len 12288
1500 Main task has work to do, go find it
1501 got dsc Start(2)
1502 start 2
1503 Feb 07 20:07:33.967 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/2, method: POST, req_id: 46609f73-e96d-4082-aca7-c199d1372acb, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
1504 Replay: started 2, returned:Ok(())
1505 ----------------------------------------------------------------
1506 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1507 States: Active Active Offline
1508 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1509 1302 Acked Write 1 Done Done Sent false
1510 1303 Acked Read 3 Done Done Sent false
1511 1304 Acked Write 8 Done Done Sent false
1512 1305 Acked Flush 0 Done Done Sent false
1513 1306 Acked Write 4 Done Done Sent false
1514 1307 Acked Read 1 Done Done Sent false
1515 1308 Acked Read 10 Done Done Sent false
1516 1309 Acked Read 10 Done Done Sent false
1517 1310 Acked Write 8 Done Done Sent false
1518 1311 Acked Read 8 Done Done Sent false
1519 1312 Acked Read 8 Done Done Sent false
1520 1313 Acked Write 3 Done Done Sent false
1521 1314 Acked Read 8 Done Done Sent false
1522 1315 Acked Write 6 Done Done Sent false
1523 1316 Acked Read 3 Done Done Sent false
1524 1317 Acked Read 10 Done Done Sent false
1525 1318 Acked Write 2 Done Done Sent false
1526 1319 Acked Flush 0 Done Done Sent false
1527 1320 Acked Write 10 Done Done Sent false
1528 1321 Acked Read 3 Done Done Sent false
1529 1322 Acked Write 10 Done Done Sent false
1530 1323 Acked Write 1 Done Done Sent false
1531 1324 Acked Write 10 Done Done Sent false
1532 1325 Acked Read 8 Done Done Sent false
1533 1326 Acked Write 4 Done Done Sent false
1534 1327 Acked Flush 0 Done Done Sent false
1535 1328 Acked Read 9 Done Done Sent false
1536 1329 Acked Read 7 Done Done Sent false
1537 1330 Acked Read 3 Done Done Sent false
1538 1331 Acked Write 2 Done Done Sent false
1539 1332 Acked Write 9 Done Done Sent false
1540 1333 Acked Write 8 Done Done Sent false
1541 1334 Acked Read 1 Done Done Sent false
1542 1335 Acked Write 2 Done Done Sent false
1543 1336 Acked Read 7 Done Done Sent false
1544 1337 Acked Write 8 Done Done Sent false
1545 1338 Acked Read 9 Done Done Sent false
1546 1339 Acked Read 9 Done Done Sent false
1547 1340 Acked Read 7 Done Done Sent false
1548 1341 Acked Write 9 Done Done Sent false
1549 1342 Acked Flush 0 Done Done Sent false
1550 1343 Acked Write 4 Done Done Sent false
1551 1344 Acked Read 8 Done Done Sent false
1552 1345 Acked Write 3 Done Done Sent false
1553 1346 Acked Read 9 Done Done Sent false
1554 1347 Acked Read 6 Done Done Sent false
1555 1348 Acked Read 1 Done Done Sent false
1556 1349 Acked Write 8 Done Done Sent false
1557 1350 Acked Read 8 Done Done Sent false
1558 1351 Acked Read 2 Done Done Sent false
1559 1352 Acked Read 3 Done Done Sent false
1560 1353 Acked Read 9 Done Done Sent false
1561 1354 Acked Read 4 Done Done Sent false
1562 1355 Acked Read 5 Done Done Sent false
1563 1356 Acked Read 4 Done Done Sent false
1564 1357 Acked Read 3 Done Done Sent false
1565 1358 Acked Write 5 Done Done Sent false
1566 1359 Acked Read 3 Done Done Sent false
1567 1360 Acked Read 5 Done Done Sent false
1568 1361 Acked Flush 0 Done Done Sent false
1569 1362 Acked Read 6 Done Done Sent false
1570 1363 Acked Read 3 Done Done Sent false
1571 1364 Acked Read 5 Done Done Sent false
1572 1365 Acked Write 3 Done Done Sent false
1573 1366 Acked Write 1 Done Done Sent false
1574 1367 Acked Read 8 Done Done Sent false
1575 1368 Acked Write 4 Done Done Sent false
1576 1369 Acked Read 7 Done Done Sent false
1577 1370 Acked Write 3 Done Done Sent false
1578 1371 Acked Write 7 Done Done Sent false
1579 1372 Acked Read 2 Done Done Sent false
1580 1373 Acked Write 6 Done Done Sent false
1581 1374 Acked Flush 0 Done Done Sent false
1582 1375 Acked Write 10 Done Done Sent false
1583 1376 Acked Write 7 Done Done Sent false
1584 1377 Acked Write 7 Done Done Sent false
1585 1378 Acked Read 2 Done Done Sent false
1586 1379 Acked Write 4 Done Done Sent false
1587 1380 Acked Write 3 Done Done Sent false
1588 1381 Acked Read 5 Done Done Sent false
1589 1382 Acked Write 9 Done Done Sent false
1590 1383 Acked Write 9 Done Done Sent false
1591 1384 Acked Write 5 Done Done Sent false
1592 1385 Acked Write 6 Done Done Sent false
1593 1386 Acked Read 7 Done Done Sent false
1594 1387 Acked Read 4 Done Done Sent false
1595 1388 Acked Flush 0 Done Done Sent false
1596 1389 Acked Write 10 Done Done Sent false
1597 1390 Acked Write 2 Done Done Sent false
1598 1391 Acked Flush 0 Done Done Sent false
1599 1392 Acked Read 2 Done Done Sent false
1600 1393 Acked Read 3 Done Done Sent false
1601 1394 Acked Read 1 Done Done Sent false
1602 1395 Acked Write 1 Done Done Sent false
1603 1396 Acked Read 5 Done Done Sent false
1604 1397 Acked Flush 0 Done Done Sent false
1605 1398 Acked Read 6 Done Done Sent false
1606 1399 Acked Read 10 Done Done Sent false
1607 1400 Acked Read 8 Done Done Sent false
1608 1401 Acked Read 1 Done Done Sent false
1609 1402 Acked Write 3 Done Done Sent false
1610 1403 Acked Write 10 Done Done Sent false
1611 1404 Acked Read 9 Done Done Sent false
1612 1405 Acked Read 9 Done Done Sent false
1613 1406 Acked Read 9 Done Done Sent false
1614 1407 Acked Write 8 Done Done Sent false
1615 1408 Acked Flush 0 Done Done Sent false
1616 1409 Acked Read 1 Done Done Sent false
1617 1410 Acked Read 7 Done Done Sent false
1618 1411 Acked Read 3 Done Done Sent false
1619 1412 Acked Read 6 Done Done Sent false
1620 1413 Acked Read 2 Done Done Sent false
1621 1414 Acked Read 10 Done Done Sent false
1622 1415 Acked Write 5 Done Done Sent false
1623 1416 Acked Write 9 Done Done Sent false
1624 1417 Acked Write 7 Done Done Sent false
1625 1418 Acked Read 7 Done Done Sent false
1626 1419 Acked Write 3 Done Done Sent false
1627 1420 Acked Write 1 Done Done Sent false
1628 1421 Acked Read 7 Done Done Sent false
1629 1422 Acked Read 5 Done Done Sent false
1630 1423 Acked Read 7 Done Done Sent false
1631 1424 Acked Write 1 Done Done Sent false
1632 1425 Acked Read 6 Done Done Sent false
1633 1426 Acked Read 9 Done Done Sent false
1634 1427 Acked Read 4 Done Done Sent false
1635 1428 Acked Read 5 Done Done Sent false
1636 1429 Acked Flush 0 Done Done Sent false
1637 1430 Acked Write 5 Done Done Sent false
1638 1431 Acked Write 8 Done Done Sent false
1639 1432 Acked Read 4 Done Done Sent false
1640 1433 Acked Write 4 Done Done Sent false
1641 1434 Acked Read 4 Done Done Sent false
1642 1435 Acked Read 3 Done Done Sent false
1643 1436 Acked Read 2 Done Done Sent false
1644 1437 Acked Flush 0 Done Done Sent false
1645 1438 Acked Write 6 Done Done Sent false
1646 1439 Acked Read 2 Done Done Sent false
1647 1440 Acked Read 9 Done Done Sent false
1648 1441 Acked Write 7 Done Done Sent false
1649 1442 Acked Read 6 Done Done Sent false
1650 1443 Acked Read 8 Done Done Sent false
1651 1444 Acked Write 7 Done Done Sent false
1652 1445 Acked Write 2 Done Done Sent false
1653 1446 Acked Read 10 Done Done Sent false
1654 1447 Acked Flush 0 Done Done Sent false
1655 1448 Acked Write 5 Done Done Sent false
1656 1449 Acked Read 3 Done Done Sent false
1657 1450 Acked Write 1 Done Done Sent false
1658 1451 Acked Read 4 Done Done Sent false
1659 1452 Acked Read 6 Done Done Sent false
1660 1453 Acked Read 6 Done Done Sent false
1661 1454 Acked Write 9 Done Done Sent false
1662 1455 Acked Write 7 Done Done Sent false
1663 1456 Acked Read 5 Done Done Sent false
1664 1457 Acked Flush 0 Done Done Sent false
1665 1458 Acked Read 2 Done Done Sent false
1666 1459 Acked Write 2 Done Done Sent false
1667 1460 Acked Flush 0 Done Done Sent false
1668 1461 Acked Write 10 Done Done Sent false
1669 1462 Acked Read 8 Done Done Sent false
1670 1463 Acked Read 4 Done Done Sent false
1671 1464 Acked Read 4 Done Done Sent false
1672 1465 Acked Read 4 Done Done Sent false
1673 1466 Acked Flush 0 Done Done Sent false
1674 1467 Acked Read 5 Done Done Sent false
1675 1468 Acked Read 8 Done Done Sent false
1676 1469 Acked Flush 0 Done Done Sent false
1677 1470 Acked Read 7 Done Done Sent false
1678 1471 Acked Write 2 Done Done Sent false
1679 1472 Acked Write 5 Done Done Sent false
1680 1473 Acked Write 3 Done Done Sent false
1681 1474 Acked Read 1 Done Done Sent false
1682 1475 Acked Write 2 Done Done Sent false
1683 1476 Acked Read 3 Done Done Sent false
1684 1477 Acked Read 7 Done Done Sent false
1685 1478 Acked Read 1 Done Done Sent false
1686 1479 Acked Write 4 Done Done Sent false
1687 1480 Acked Read 2 Done Done Sent false
1688 1481 Acked Write 9 Done Done Sent false
1689 1482 Acked Read 5 Done Done Sent false
1690 1483 Acked Write 4 Done Done Sent false
1691 1484 Acked Write 6 Done Done Sent false
1692 1485 Acked Read 6 Done Done Sent false
1693 1486 Acked Write 10 Done Done Sent false
1694 1487 Acked Read 3 Done Done Sent false
1695 1488 Acked Write 8 Done Done Sent false
1696 1489 Acked Write 9 Done Done Sent false
1697 1490 Acked Read 9 Done Done Sent false
1698 1491 Acked Read 3 Done Done Sent false
1699 1492 Acked Read 8 Done Done Sent false
1700 1493 Acked Write 1 Done Done Sent false
1701 1494 Acked Read 2 Done Done Sent false
1702 1495 Acked Write 4 Done Done Sent false
1703 1496 Acked Read 9 Done Done Sent false
1704 1497 Acked Write 4 Done Done Sent false
1705 1498 Acked Read 3 Done Done Sent false
1706 1499 Acked Write 8 Done Done Sent false
1707 1500 Acked Read 9 Done Done Sent false
1708 1501 Acked Flush 0 Done Done Sent false
1709 1502 Acked Read 3 Done Done Sent false
1710 1503 Acked Write 4 Done Done Sent false
1711 1504 Acked Read 2 Done Done Sent false
1712 1505 Acked Read 7 Done Done Sent false
1713 1506 Acked Read 10 Done Done Sent false
1714 1507 Acked Read 4 Done Done Sent false
1715 1508 Acked Flush 0 Done Done Sent false
1716 1509 Acked Read 6 Done Done Sent false
1717 1510 Acked Read 7 Done Done Sent false
1718 1511 Acked Write 7 Done Done Sent false
1719 1512 Acked Read 6 Done Done Sent false
1720 1513 Acked Flush 0 Done Done Sent false
1721 1514 Acked Write 5 Done Done Sent false
1722 1515 Acked Write 6 Done Done Sent false
1723 1516 Acked Write 10 Done Done Sent false
1724 1517 Acked Read 1 Done Done Sent false
1725 1518 Acked Read 9 Done Done Sent false
1726 1519 Acked Write 3 Done Done Sent false
1727 1520 Acked Write 2 Done Done Sent false
1728 1521 Acked Read 4 Done Done Sent false
1729 1522 Acked Write 3 Done Done Sent false
1730 1523 Acked Write 2 Done Done Sent false
1731 1524 Acked Read 3 Done Done Sent false
1732 1525 Acked Read 2 Done Done Sent false
1733 1526 Acked Write 2 Done Done Sent false
1734 1527 Acked Flush 0 Done Done Sent false
1735 1528 Acked Write 8 Done Done Sent false
1736 1529 Acked Read 9 Done Done Sent false
1737 1530 Acked Read 3 Done Done Sent false
1738 1531 Acked Write 2 Done Done Sent false
1739 1532 Acked Read 3 Done Done Sent false
1740 1533 Acked Read 10 Done Done Sent false
1741 1534 Acked Read 2 Done Done Sent false
1742 1535 Acked Read 1 Done Done Sent false
1743 1536 Acked Write 3 Done Done Sent false
1744 1537 Acked Write 3 Done Done Sent false
1745 1538 Acked Read 1 Done Done Sent false
1746 1539 Acked Read 1 Done Done Sent false
1747 1540 Acked Read 9 Done Done Sent false
1748 1541 Acked Read 7 Done Done Sent false
1749 1542 Acked Read 8 Done Done Sent false
1750 1543 Acked Read 8 Done Done Sent false
1751 1544 Acked Write 1 Done Done Sent false
1752 1545 Acked Read 6 Done Done Sent false
1753 1546 Acked Write 9 Done Done Sent false
1754 1547 Acked Read 3 Done Done Sent false
1755 1548 Acked Read 3 Done Done Sent false
1756 1549 Acked Write 9 Done Done Sent false
1757 1550 Acked Write 7 Done Done Sent false
1758 1551 Acked Write 9 Done Done Sent false
1759 1552 Acked Write 5 Done Done Sent false
1760 1553 Acked Write 10 Done Done Sent false
1761 1554 Acked Write 6 Done Done Sent false
1762 1555 Acked Write 3 Done Done Sent false
1763 1556 Acked Write 10 Done Done Sent false
1764 1557 Acked Read 1 Done Done Sent false
1765 1558 Acked Read 1 Done Done Sent false
1766 1559 Acked Read 9 Done Done Sent false
1767 1560 Acked Read 6 Done Done Sent false
1768 1561 Acked Read 9 Done Done Sent false
1769 1562 Acked Read 2 Done Done Sent false
1770 1563 Acked Read 1 Done Done Sent false
1771 1564 Acked Flush 0 Done Done Sent false
1772 1565 Acked Write 4 Done Done Sent false
1773 1566 Acked Read 1 Done Done Sent false
1774 1567 Acked Write 6 Done Done Sent false
1775 1568 Acked Write 5 Done Done Sent false
1776 1569 Acked Flush 0 Done Done Sent false
1777 1570 Acked Write 3 Done Done Sent false
1778 1571 Acked Write 10 Done Done Sent false
1779 1572 Acked Read 8 Done Done Sent false
1780 1573 Acked Write 2 Done Done Sent false
1781 1574 Acked Flush 0 Done Done Sent false
1782 1575 Acked Read 10 Done Done Sent false
1783 1576 Acked Write 8 Done Done Sent false
1784 1577 Acked Read 2 Done Done Sent false
1785 1578 Acked Write 5 Done Done Sent false
1786 1579 Acked Write 5 Done Done Sent false
1787 1580 Acked Write 2 Done Done Sent false
1788 1581 Acked Write 2 Done Done Sent false
1789 1582 Acked Write 5 Done Done Sent false
1790 1583 Acked Read 7 Done Done Sent false
1791 1584 Acked Read 1 Done Done Sent false
1792 1585 Acked Write 5 Done Done Sent false
1793 1586 Acked Write 1 Done Done Sent false
1794 1587 Acked Read 7 Done Done Sent false
1795 1588 Acked Write 10 Done Done Sent false
1796 1589 Acked Write 2 Done Done Sent false
1797 1590 Acked Write 10 Done Done Sent false
1798 1591 Acked Read 2 Done Done Sent false
1799 1592 Acked Write 3 Done Done Sent false
1800 1593 Acked Write 8 Done Done Sent false
1801 1594 Acked Read 5 Done Done Sent false
1802 1595 Acked Write 6 Done Done Sent false
1803 1596 Acked Read 1 Done Done Sent false
1804 1597 Acked Write 6 Done Done Sent false
1805 1598 Acked Write 9 Done Done Sent false
1806 1599 Acked Write 8 Done Done Sent false
1807 1600 Acked Read 7 Done Done Sent false
1808 1601 Acked Read 4 Done Done Sent false
1809 1602 Acked Read 3 Done Done Sent false
1810 STATES DS:0 DS:1 DS:2 TOTAL
1811 Sent 0 0 301 301
1812 Done 301 301 0 602
1813 Skipped 0 0 0 0
1814 Error 0 0 0 0
1815 Last Flush: Some(JobId(1574)) Some(JobId(1574)) Some(JobId(1301))
1816 Downstairs last five completed: 1301 1300 1299 1298 1297
1817 Upstairs last five completed: 1602 1601 1600 1599 1598
1818 CLIENT: Up:0 ds:301 act:2
18192025-02-07T20:07:34.956ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
18202025-02-07T20:07:35.958ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
18212025-02-07T20:07:36.960ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
1822 ----------------------------------------------------------------
1823 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1824 States: Active Active Offline
1825 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1826 1302 Acked Write 1 Done Done Sent false
1827 1303 Acked Read 3 Done Done Sent false
1828 1304 Acked Write 8 Done Done Sent false
1829 1305 Acked Flush 0 Done Done Sent false
1830 1306 Acked Write 4 Done Done Sent false
1831 1307 Acked Read 1 Done Done Sent false
1832 1308 Acked Read 10 Done Done Sent false
1833 1309 Acked Read 10 Done Done Sent false
1834 1310 Acked Write 8 Done Done Sent false
1835 1311 Acked Read 8 Done Done Sent false
1836 1312 Acked Read 8 Done Done Sent false
1837 1313 Acked Write 3 Done Done Sent false
1838 1314 Acked Read 8 Done Done Sent false
1839 1315 Acked Write 6 Done Done Sent false
1840 1316 Acked Read 3 Done Done Sent false
1841 1317 Acked Read 10 Done Done Sent false
1842 1318 Acked Write 2 Done Done Sent false
1843 1319 Acked Flush 0 Done Done Sent false
1844 1320 Acked Write 10 Done Done Sent false
1845 1321 Acked Read 3 Done Done Sent false
1846 1322 Acked Write 10 Done Done Sent false
1847 1323 Acked Write 1 Done Done Sent false
1848 1324 Acked Write 10 Done Done Sent false
1849 1325 Acked Read 8 Done Done Sent false
1850 1326 Acked Write 4 Done Done Sent false
1851 1327 Acked Flush 0 Done Done Sent false
1852 1328 Acked Read 9 Done Done Sent false
1853 1329 Acked Read 7 Done Done Sent false
1854 1330 Acked Read 3 Done Done Sent false
1855 1331 Acked Write 2 Done Done Sent false
1856 1332 Acked Write 9 Done Done Sent false
1857 1333 Acked Write 8 Done Done Sent false
1858 1334 Acked Read 1 Done Done Sent false
1859 1335 Acked Write 2 Done Done Sent false
1860 1336 Acked Read 7 Done Done Sent false
1861 1337 Acked Write 8 Done Done Sent false
1862 1338 Acked Read 9 Done Done Sent false
1863 1339 Acked Read 9 Done Done Sent false
1864 1340 Acked Read 7 Done Done Sent false
1865 1341 Acked Write 9 Done Done Sent false
1866 1342 Acked Flush 0 Done Done Sent false
1867 1343 Acked Write 4 Done Done Sent false
1868 1344 Acked Read 8 Done Done Sent false
1869 1345 Acked Write 3 Done Done Sent false
1870 1346 Acked Read 9 Done Done Sent false
1871 1347 Acked Read 6 Done Done Sent false
1872 1348 Acked Read 1 Done Done Sent false
1873 1349 Acked Write 8 Done Done Sent false
1874 1350 Acked Read 8 Done Done Sent false
1875 1351 Acked Read 2 Done Done Sent false
1876 1352 Acked Read 3 Done Done Sent false
1877 1353 Acked Read 9 Done Done Sent false
1878 1354 Acked Read 4 Done Done Sent false
1879 1355 Acked Read 5 Done Done Sent false
1880 1356 Acked Read 4 Done Done Sent false
1881 1357 Acked Read 3 Done Done Sent false
1882 1358 Acked Write 5 Done Done Sent false
1883 1359 Acked Read 3 Done Done Sent false
1884 1360 Acked Read 5 Done Done Sent false
1885 1361 Acked Flush 0 Done Done Sent false
1886 1362 Acked Read 6 Done Done Sent false
1887 1363 Acked Read 3 Done Done Sent false
1888 1364 Acked Read 5 Done Done Sent false
1889 1365 Acked Write 3 Done Done Sent false
1890 1366 Acked Write 1 Done Done Sent false
1891 1367 Acked Read 8 Done Done Sent false
1892 1368 Acked Write 4 Done Done Sent false
1893 1369 Acked Read 7 Done Done Sent false
1894 1370 Acked Write 3 Done Done Sent false
1895 1371 Acked Write 7 Done Done Sent false
1896 1372 Acked Read 2 Done Done Sent false
1897 1373 Acked Write 6 Done Done Sent false
1898 1374 Acked Flush 0 Done Done Sent false
1899 1375 Acked Write 10 Done Done Sent false
1900 1376 Acked Write 7 Done Done Sent false
1901 1377 Acked Write 7 Done Done Sent false
1902 1378 Acked Read 2 Done Done Sent false
1903 1379 Acked Write 4 Done Done Sent false
1904 1380 Acked Write 3 Done Done Sent false
1905 1381 Acked Read 5 Done Done Sent false
1906 1382 Acked Write 9 Done Done Sent false
1907 1383 Acked Write 9 Done Done Sent false
1908 1384 Acked Write 5 Done Done Sent false
1909 1385 Acked Write 6 Done Done Sent false
1910 1386 Acked Read 7 Done Done Sent false
1911 1387 Acked Read 4 Done Done Sent false
1912 1388 Acked Flush 0 Done Done Sent false
1913 1389 Acked Write 10 Done Done Sent false
1914 1390 Acked Write 2 Done Done Sent false
1915 1391 Acked Flush 0 Done Done Sent false
1916 1392 Acked Read 2 Done Done Sent false
1917 1393 Acked Read 3 Done Done Sent false
1918 1394 Acked Read 1 Done Done Sent false
1919 1395 Acked Write 1 Done Done Sent false
1920 1396 Acked Read 5 Done Done Sent false
1921 1397 Acked Flush 0 Done Done Sent false
1922 1398 Acked Read 6 Done Done Sent false
1923 1399 Acked Read 10 Done Done Sent false
1924 1400 Acked Read 8 Done Done Sent false
1925 1401 Acked Read 1 Done Done Sent false
1926 1402 Acked Write 3 Done Done Sent false
1927 1403 Acked Write 10 Done Done Sent false
1928 1404 Acked Read 9 Done Done Sent false
1929 1405 Acked Read 9 Done Done Sent false
1930 1406 Acked Read 9 Done Done Sent false
1931 1407 Acked Write 8 Done Done Sent false
1932 1408 Acked Flush 0 Done Done Sent false
1933 1409 Acked Read 1 Done Done Sent false
1934 1410 Acked Read 7 Done Done Sent false
1935 1411 Acked Read 3 Done Done Sent false
1936 1412 Acked Read 6 Done Done Sent false
1937 1413 Acked Read 2 Done Done Sent false
1938 1414 Acked Read 10 Done Done Sent false
1939 1415 Acked Write 5 Done Done Sent false
1940 1416 Acked Write 9 Done Done Sent false
1941 1417 Acked Write 7 Done Done Sent false
1942 1418 Acked Read 7 Done Done Sent false
1943 1419 Acked Write 3 Done Done Sent false
1944 1420 Acked Write 1 Done Done Sent false
1945 1421 Acked Read 7 Done Done Sent false
1946 1422 Acked Read 5 Done Done Sent false
1947 1423 Acked Read 7 Done Done Sent false
1948 1424 Acked Write 1 Done Done Sent false
1949 1425 Acked Read 6 Done Done Sent false
1950 1426 Acked Read 9 Done Done Sent false
1951 1427 Acked Read 4 Done Done Sent false
1952 1428 Acked Read 5 Done Done Sent false
1953 1429 Acked Flush 0 Done Done Sent false
1954 1430 Acked Write 5 Done Done Sent false
1955 1431 Acked Write 8 Done Done Sent false
1956 1432 Acked Read 4 Done Done Sent false
1957 1433 Acked Write 4 Done Done Sent false
1958 1434 Acked Read 4 Done Done Sent false
1959 1435 Acked Read 3 Done Done Sent false
1960 1436 Acked Read 2 Done Done Sent false
1961 1437 Acked Flush 0 Done Done Sent false
1962 1438 Acked Write 6 Done Done Sent false
1963 1439 Acked Read 2 Done Done Sent false
1964 1440 Acked Read 9 Done Done Sent false
1965 1441 Acked Write 7 Done Done Sent false
1966 1442 Acked Read 6 Done Done Sent false
1967 1443 Acked Read 8 Done Done Sent false
1968 1444 Acked Write 7 Done Done Sent false
1969 1445 Acked Write 2 Done Done Sent false
1970 1446 Acked Read 10 Done Done Sent false
1971 1447 Acked Flush 0 Done Done Sent false
1972 1448 Acked Write 5 Done Done Sent false
1973 1449 Acked Read 3 Done Done Sent false
1974 1450 Acked Write 1 Done Done Sent false
1975 1451 Acked Read 4 Done Done Sent false
1976 1452 Acked Read 6 Done Done Sent false
1977 1453 Acked Read 6 Done Done Sent false
1978 1454 Acked Write 9 Done Done Sent false
1979 1455 Acked Write 7 Done Done Sent false
1980 1456 Acked Read 5 Done Done Sent false
1981 1457 Acked Flush 0 Done Done Sent false
1982 1458 Acked Read 2 Done Done Sent false
1983 1459 Acked Write 2 Done Done Sent false
1984 1460 Acked Flush 0 Done Done Sent false
1985 1461 Acked Write 10 Done Done Sent false
1986 1462 Acked Read 8 Done Done Sent false
1987 1463 Acked Read 4 Done Done Sent false
1988 1464 Acked Read 4 Done Done Sent false
1989 1465 Acked Read 4 Done Done Sent false
1990 1466 Acked Flush 0 Done Done Sent false
1991 1467 Acked Read 5 Done Done Sent false
1992 1468 Acked Read 8 Done Done Sent false
1993 1469 Acked Flush 0 Done Done Sent false
1994 1470 Acked Read 7 Done Done Sent false
1995 1471 Acked Write 2 Done Done Sent false
1996 1472 Acked Write 5 Done Done Sent false
1997 1473 Acked Write 3 Done Done Sent false
1998 1474 Acked Read 1 Done Done Sent false
1999 1475 Acked Write 2 Done Done Sent false
2000 1476 Acked Read 3 Done Done Sent false
2001 1477 Acked Read 7 Done Done Sent false
2002 1478 Acked Read 1 Done Done Sent false
2003 1479 Acked Write 4 Done Done Sent false
2004 1480 Acked Read 2 Done Done Sent false
2005 1481 Acked Write 9 Done Done Sent false
2006 1482 Acked Read 5 Done Done Sent false
2007 1483 Acked Write 4 Done Done Sent false
2008 1484 Acked Write 6 Done Done Sent false
2009 1485 Acked Read 6 Done Done Sent false
2010 1486 Acked Write 10 Done Done Sent false
2011 1487 Acked Read 3 Done Done Sent false
2012 1488 Acked Write 8 Done Done Sent false
2013 1489 Acked Write 9 Done Done Sent false
2014 1490 Acked Read 9 Done Done Sent false
2015 1491 Acked Read 3 Done Done Sent false
2016 1492 Acked Read 8 Done Done Sent false
2017 1493 Acked Write 1 Done Done Sent false
2018 1494 Acked Read 2 Done Done Sent false
2019 1495 Acked Write 4 Done Done Sent false
2020 1496 Acked Read 9 Done Done Sent false
2021 1497 Acked Write 4 Done Done Sent false
2022 1498 Acked Read 3 Done Done Sent false
2023 1499 Acked Write 8 Done Done Sent false
2024 1500 Acked Read 9 Done Done Sent false
2025 1501 Acked Flush 0 Done Done Sent false
2026 1502 Acked Read 3 Done Done Sent false
2027 1503 Acked Write 4 Done Done Sent false
2028 1504 Acked Read 2 Done Done Sent false
2029 1505 Acked Read 7 Done Done Sent false
2030 1506 Acked Read 10 Done Done Sent false
2031 1507 Acked Read 4 Done Done Sent false
2032 1508 Acked Flush 0 Done Done Sent false
2033 1509 Acked Read 6 Done Done Sent false
2034 1510 Acked Read 7 Done Done Sent false
2035 1511 Acked Write 7 Done Done Sent false
2036 1512 Acked Read 6 Done Done Sent false
2037 1513 Acked Flush 0 Done Done Sent false
2038 1514 Acked Write 5 Done Done Sent false
2039 1515 Acked Write 6 Done Done Sent false
2040 1516 Acked Write 10 Done Done Sent false
2041 1517 Acked Read 1 Done Done Sent false
2042 1518 Acked Read 9 Done Done Sent false
2043 1519 Acked Write 3 Done Done Sent false
2044 1520 Acked Write 2 Done Done Sent false
2045 1521 Acked Read 4 Done Done Sent false
2046 1522 Acked Write 3 Done Done Sent false
2047 1523 Acked Write 2 Done Done Sent false
2048 1524 Acked Read 3 Done Done Sent false
2049 1525 Acked Read 2 Done Done Sent false
2050 1526 Acked Write 2 Done Done Sent false
2051 1527 Acked Flush 0 Done Done Sent false
2052 1528 Acked Write 8 Done Done Sent false
2053 1529 Acked Read 9 Done Done Sent false
2054 1530 Acked Read 3 Done Done Sent false
2055 1531 Acked Write 2 Done Done Sent false
2056 1532 Acked Read 3 Done Done Sent false
2057 1533 Acked Read 10 Done Done Sent false
2058 1534 Acked Read 2 Done Done Sent false
2059 1535 Acked Read 1 Done Done Sent false
2060 1536 Acked Write 3 Done Done Sent false
2061 1537 Acked Write 3 Done Done Sent false
2062 1538 Acked Read 1 Done Done Sent false
2063 1539 Acked Read 1 Done Done Sent false
2064 1540 Acked Read 9 Done Done Sent false
2065 1541 Acked Read 7 Done Done Sent false
2066 1542 Acked Read 8 Done Done Sent false
2067 1543 Acked Read 8 Done Done Sent false
2068 1544 Acked Write 1 Done Done Sent false
2069 1545 Acked Read 6 Done Done Sent false
2070 1546 Acked Write 9 Done Done Sent false
2071 1547 Acked Read 3 Done Done Sent false
2072 1548 Acked Read 3 Done Done Sent false
2073 1549 Acked Write 9 Done Done Sent false
2074 1550 Acked Write 7 Done Done Sent false
2075 1551 Acked Write 9 Done Done Sent false
2076 1552 Acked Write 5 Done Done Sent false
2077 1553 Acked Write 10 Done Done Sent false
2078 1554 Acked Write 6 Done Done Sent false
2079 1555 Acked Write 3 Done Done Sent false
2080 1556 Acked Write 10 Done Done Sent false
2081 1557 Acked Read 1 Done Done Sent false
2082 1558 Acked Read 1 Done Done Sent false
2083 1559 Acked Read 9 Done Done Sent false
2084 1560 Acked Read 6 Done Done Sent false
2085 1561 Acked Read 9 Done Done Sent false
2086 1562 Acked Read 2 Done Done Sent false
2087 1563 Acked Read 1 Done Done Sent false
2088 1564 Acked Flush 0 Done Done Sent false
2089 1565 Acked Write 4 Done Done Sent false
2090 1566 Acked Read 1 Done Done Sent false
2091 1567 Acked Write 6 Done Done Sent false
2092 1568 Acked Write 5 Done Done Sent false
2093 1569 Acked Flush 0 Done Done Sent false
2094 1570 Acked Write 3 Done Done Sent false
2095 1571 Acked Write 10 Done Done Sent false
2096 1572 Acked Read 8 Done Done Sent false
2097 1573 Acked Write 2 Done Done Sent false
2098 1574 Acked Flush 0 Done Done Sent false
2099 1575 Acked Read 10 Done Done Sent false
2100 1576 Acked Write 8 Done Done Sent false
2101 1577 Acked Read 2 Done Done Sent false
2102 1578 Acked Write 5 Done Done Sent false
2103 1579 Acked Write 5 Done Done Sent false
2104 1580 Acked Write 2 Done Done Sent false
2105 1581 Acked Write 2 Done Done Sent false
2106 1582 Acked Write 5 Done Done Sent false
2107 1583 Acked Read 7 Done Done Sent false
2108 1584 Acked Read 1 Done Done Sent false
2109 1585 Acked Write 5 Done Done Sent false
2110 1586 Acked Write 1 Done Done Sent false
2111 1587 Acked Read 7 Done Done Sent false
2112 1588 Acked Write 10 Done Done Sent false
2113 1589 Acked Write 2 Done Done Sent false
2114 1590 Acked Write 10 Done Done Sent false
2115 1591 Acked Read 2 Done Done Sent false
2116 1592 Acked Write 3 Done Done Sent false
2117 1593 Acked Write 8 Done Done Sent false
2118 1594 Acked Read 5 Done Done Sent false
2119 1595 Acked Write 6 Done Done Sent false
2120 1596 Acked Read 1 Done Done Sent false
2121 1597 Acked Write 6 Done Done Sent false
2122 1598 Acked Write 9 Done Done Sent false
2123 1599 Acked Write 8 Done Done Sent false
2124 1600 Acked Read 7 Done Done Sent false
2125 1601 Acked Read 4 Done Done Sent false
2126 1602 Acked Read 3 Done Done Sent false
2127 STATES DS:0 DS:1 DS:2 TOTAL
2128 Sent 0 0 301 301
2129 Done 301 301 0 602
2130 Skipped 0 0 0 0
2131 Error 0 0 0 0
2132 Last Flush: Some(JobId(1574)) Some(JobId(1574)) Some(JobId(1301))
2133 Downstairs last five completed: 1301 1300 1299 1298 1297
2134 Upstairs last five completed: 1602 1601 1600 1599 1598
2135 CLIENT: Up:0 ds:301 act:2
21362025-02-07T20:07:38.963ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
21372025-02-07T20:07:39.727ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
21382025-02-07T20:07:39.728ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
21392025-02-07T20:07:39.728ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
21402025-02-07T20:07:39.728ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
21412025-02-07T20:07:39.728ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2142 [8830] got start action
2143 [8830] I am going to restart
2144 Starting downstairs at port 8830
2145 Make output file at "/tmp/dsc/downstairs-8830.txt"
2146 [8830][2] reports Starting
2147 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1211)
2148 [8830][2] reports Running
21492025-02-07T20:07:40.229ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
21502025-02-07T20:07:40.965ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
21512025-02-07T20:07:41.967ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2152 ----------------------------------------------------------------
2153 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2154 States: Active Active Offline
2155 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2156 1302 Acked Write 1 Done Done Sent false
2157 1303 Acked Read 3 Done Done Sent false
2158 1304 Acked Write 8 Done Done Sent false
2159 1305 Acked Flush 0 Done Done Sent false
2160 1306 Acked Write 4 Done Done Sent false
2161 1307 Acked Read 1 Done Done Sent false
2162 1308 Acked Read 10 Done Done Sent false
2163 1309 Acked Read 10 Done Done Sent false
2164 1310 Acked Write 8 Done Done Sent false
2165 1311 Acked Read 8 Done Done Sent false
2166 1312 Acked Read 8 Done Done Sent false
2167 1313 Acked Write 3 Done Done Sent false
2168 1314 Acked Read 8 Done Done Sent false
2169 1315 Acked Write 6 Done Done Sent false
2170 1316 Acked Read 3 Done Done Sent false
2171 1317 Acked Read 10 Done Done Sent false
2172 1318 Acked Write 2 Done Done Sent false
2173 1319 Acked Flush 0 Done Done Sent false
2174 1320 Acked Write 10 Done Done Sent false
2175 1321 Acked Read 3 Done Done Sent false
2176 1322 Acked Write 10 Done Done Sent false
2177 1323 Acked Write 1 Done Done Sent false
2178 1324 Acked Write 10 Done Done Sent false
2179 1325 Acked Read 8 Done Done Sent false
2180 1326 Acked Write 4 Done Done Sent false
2181 1327 Acked Flush 0 Done Done Sent false
2182 1328 Acked Read 9 Done Done Sent false
2183 1329 Acked Read 7 Done Done Sent false
2184 1330 Acked Read 3 Done Done Sent false
2185 1331 Acked Write 2 Done Done Sent false
2186 1332 Acked Write 9 Done Done Sent false
2187 1333 Acked Write 8 Done Done Sent false
2188 1334 Acked Read 1 Done Done Sent false
2189 1335 Acked Write 2 Done Done Sent false
2190 1336 Acked Read 7 Done Done Sent false
2191 1337 Acked Write 8 Done Done Sent false
2192 1338 Acked Read 9 Done Done Sent false
2193 1339 Acked Read 9 Done Done Sent false
2194 1340 Acked Read 7 Done Done Sent false
2195 1341 Acked Write 9 Done Done Sent false
2196 1342 Acked Flush 0 Done Done Sent false
2197 1343 Acked Write 4 Done Done Sent false
2198 1344 Acked Read 8 Done Done Sent false
2199 1345 Acked Write 3 Done Done Sent false
2200 1346 Acked Read 9 Done Done Sent false
2201 1347 Acked Read 6 Done Done Sent false
2202 1348 Acked Read 1 Done Done Sent false
2203 1349 Acked Write 8 Done Done Sent false
2204 1350 Acked Read 8 Done Done Sent false
2205 1351 Acked Read 2 Done Done Sent false
2206 1352 Acked Read 3 Done Done Sent false
2207 1353 Acked Read 9 Done Done Sent false
2208 1354 Acked Read 4 Done Done Sent false
2209 1355 Acked Read 5 Done Done Sent false
2210 1356 Acked Read 4 Done Done Sent false
2211 1357 Acked Read 3 Done Done Sent false
2212 1358 Acked Write 5 Done Done Sent false
2213 1359 Acked Read 3 Done Done Sent false
2214 1360 Acked Read 5 Done Done Sent false
2215 1361 Acked Flush 0 Done Done Sent false
2216 1362 Acked Read 6 Done Done Sent false
2217 1363 Acked Read 3 Done Done Sent false
2218 1364 Acked Read 5 Done Done Sent false
2219 1365 Acked Write 3 Done Done Sent false
2220 1366 Acked Write 1 Done Done Sent false
2221 1367 Acked Read 8 Done Done Sent false
2222 1368 Acked Write 4 Done Done Sent false
2223 1369 Acked Read 7 Done Done Sent false
2224 1370 Acked Write 3 Done Done Sent false
2225 1371 Acked Write 7 Done Done Sent false
2226 1372 Acked Read 2 Done Done Sent false
2227 1373 Acked Write 6 Done Done Sent false
2228 1374 Acked Flush 0 Done Done Sent false
2229 1375 Acked Write 10 Done Done Sent false
2230 1376 Acked Write 7 Done Done Sent false
2231 1377 Acked Write 7 Done Done Sent false
2232 1378 Acked Read 2 Done Done Sent false
2233 1379 Acked Write 4 Done Done Sent false
2234 1380 Acked Write 3 Done Done Sent false
2235 1381 Acked Read 5 Done Done Sent false
2236 1382 Acked Write 9 Done Done Sent false
2237 1383 Acked Write 9 Done Done Sent false
2238 1384 Acked Write 5 Done Done Sent false
2239 1385 Acked Write 6 Done Done Sent false
2240 1386 Acked Read 7 Done Done Sent false
2241 1387 Acked Read 4 Done Done Sent false
2242 1388 Acked Flush 0 Done Done Sent false
2243 1389 Acked Write 10 Done Done Sent false
2244 1390 Acked Write 2 Done Done Sent false
2245 1391 Acked Flush 0 Done Done Sent false
2246 1392 Acked Read 2 Done Done Sent false
2247 1393 Acked Read 3 Done Done Sent false
2248 1394 Acked Read 1 Done Done Sent false
2249 1395 Acked Write 1 Done Done Sent false
2250 1396 Acked Read 5 Done Done Sent false
2251 1397 Acked Flush 0 Done Done Sent false
2252 1398 Acked Read 6 Done Done Sent false
2253 1399 Acked Read 10 Done Done Sent false
2254 1400 Acked Read 8 Done Done Sent false
2255 1401 Acked Read 1 Done Done Sent false
2256 1402 Acked Write 3 Done Done Sent false
2257 1403 Acked Write 10 Done Done Sent false
2258 1404 Acked Read 9 Done Done Sent false
2259 1405 Acked Read 9 Done Done Sent false
2260 1406 Acked Read 9 Done Done Sent false
2261 1407 Acked Write 8 Done Done Sent false
2262 1408 Acked Flush 0 Done Done Sent false
2263 1409 Acked Read 1 Done Done Sent false
2264 1410 Acked Read 7 Done Done Sent false
2265 1411 Acked Read 3 Done Done Sent false
2266 1412 Acked Read 6 Done Done Sent false
2267 1413 Acked Read 2 Done Done Sent false
2268 1414 Acked Read 10 Done Done Sent false
2269 1415 Acked Write 5 Done Done Sent false
2270 1416 Acked Write 9 Done Done Sent false
2271 1417 Acked Write 7 Done Done Sent false
2272 1418 Acked Read 7 Done Done Sent false
2273 1419 Acked Write 3 Done Done Sent false
2274 1420 Acked Write 1 Done Done Sent false
2275 1421 Acked Read 7 Done Done Sent false
2276 1422 Acked Read 5 Done Done Sent false
2277 1423 Acked Read 7 Done Done Sent false
2278 1424 Acked Write 1 Done Done Sent false
2279 1425 Acked Read 6 Done Done Sent false
2280 1426 Acked Read 9 Done Done Sent false
2281 1427 Acked Read 4 Done Done Sent false
2282 1428 Acked Read 5 Done Done Sent false
2283 1429 Acked Flush 0 Done Done Sent false
2284 1430 Acked Write 5 Done Done Sent false
2285 1431 Acked Write 8 Done Done Sent false
2286 1432 Acked Read 4 Done Done Sent false
2287 1433 Acked Write 4 Done Done Sent false
2288 1434 Acked Read 4 Done Done Sent false
2289 1435 Acked Read 3 Done Done Sent false
2290 1436 Acked Read 2 Done Done Sent false
2291 1437 Acked Flush 0 Done Done Sent false
2292 1438 Acked Write 6 Done Done Sent false
2293 1439 Acked Read 2 Done Done Sent false
2294 1440 Acked Read 9 Done Done Sent false
2295 1441 Acked Write 7 Done Done Sent false
2296 1442 Acked Read 6 Done Done Sent false
2297 1443 Acked Read 8 Done Done Sent false
2298 1444 Acked Write 7 Done Done Sent false
2299 1445 Acked Write 2 Done Done Sent false
2300 1446 Acked Read 10 Done Done Sent false
2301 1447 Acked Flush 0 Done Done Sent false
2302 1448 Acked Write 5 Done Done Sent false
2303 1449 Acked Read 3 Done Done Sent false
2304 1450 Acked Write 1 Done Done Sent false
2305 1451 Acked Read 4 Done Done Sent false
2306 1452 Acked Read 6 Done Done Sent false
2307 1453 Acked Read 6 Done Done Sent false
2308 1454 Acked Write 9 Done Done Sent false
2309 1455 Acked Write 7 Done Done Sent false
2310 1456 Acked Read 5 Done Done Sent false
2311 1457 Acked Flush 0 Done Done Sent false
2312 1458 Acked Read 2 Done Done Sent false
2313 1459 Acked Write 2 Done Done Sent false
2314 1460 Acked Flush 0 Done Done Sent false
2315 1461 Acked Write 10 Done Done Sent false
2316 1462 Acked Read 8 Done Done Sent false
2317 1463 Acked Read 4 Done Done Sent false
2318 1464 Acked Read 4 Done Done Sent false
2319 1465 Acked Read 4 Done Done Sent false
2320 1466 Acked Flush 0 Done Done Sent false
2321 1467 Acked Read 5 Done Done Sent false
2322 1468 Acked Read 8 Done Done Sent false
2323 1469 Acked Flush 0 Done Done Sent false
2324 1470 Acked Read 7 Done Done Sent false
2325 1471 Acked Write 2 Done Done Sent false
2326 1472 Acked Write 5 Done Done Sent false
2327 1473 Acked Write 3 Done Done Sent false
2328 1474 Acked Read 1 Done Done Sent false
2329 1475 Acked Write 2 Done Done Sent false
2330 1476 Acked Read 3 Done Done Sent false
2331 1477 Acked Read 7 Done Done Sent false
2332 1478 Acked Read 1 Done Done Sent false
2333 1479 Acked Write 4 Done Done Sent false
2334 1480 Acked Read 2 Done Done Sent false
2335 1481 Acked Write 9 Done Done Sent false
2336 1482 Acked Read 5 Done Done Sent false
2337 1483 Acked Write 4 Done Done Sent false
2338 1484 Acked Write 6 Done Done Sent false
2339 1485 Acked Read 6 Done Done Sent false
2340 1486 Acked Write 10 Done Done Sent false
2341 1487 Acked Read 3 Done Done Sent false
2342 1488 Acked Write 8 Done Done Sent false
2343 1489 Acked Write 9 Done Done Sent false
2344 1490 Acked Read 9 Done Done Sent false
2345 1491 Acked Read 3 Done Done Sent false
2346 1492 Acked Read 8 Done Done Sent false
2347 1493 Acked Write 1 Done Done Sent false
2348 1494 Acked Read 2 Done Done Sent false
2349 1495 Acked Write 4 Done Done Sent false
2350 1496 Acked Read 9 Done Done Sent false
2351 1497 Acked Write 4 Done Done Sent false
2352 1498 Acked Read 3 Done Done Sent false
2353 1499 Acked Write 8 Done Done Sent false
2354 1500 Acked Read 9 Done Done Sent false
2355 1501 Acked Flush 0 Done Done Sent false
2356 1502 Acked Read 3 Done Done Sent false
2357 1503 Acked Write 4 Done Done Sent false
2358 1504 Acked Read 2 Done Done Sent false
2359 1505 Acked Read 7 Done Done Sent false
2360 1506 Acked Read 10 Done Done Sent false
2361 1507 Acked Read 4 Done Done Sent false
2362 1508 Acked Flush 0 Done Done Sent false
2363 1509 Acked Read 6 Done Done Sent false
2364 1510 Acked Read 7 Done Done Sent false
2365 1511 Acked Write 7 Done Done Sent false
2366 1512 Acked Read 6 Done Done Sent false
2367 1513 Acked Flush 0 Done Done Sent false
2368 1514 Acked Write 5 Done Done Sent false
2369 1515 Acked Write 6 Done Done Sent false
2370 1516 Acked Write 10 Done Done Sent false
2371 1517 Acked Read 1 Done Done Sent false
2372 1518 Acked Read 9 Done Done Sent false
2373 1519 Acked Write 3 Done Done Sent false
2374 1520 Acked Write 2 Done Done Sent false
2375 1521 Acked Read 4 Done Done Sent false
2376 1522 Acked Write 3 Done Done Sent false
2377 1523 Acked Write 2 Done Done Sent false
2378 1524 Acked Read 3 Done Done Sent false
2379 1525 Acked Read 2 Done Done Sent false
2380 1526 Acked Write 2 Done Done Sent false
2381 1527 Acked Flush 0 Done Done Sent false
2382 1528 Acked Write 8 Done Done Sent false
2383 1529 Acked Read 9 Done Done Sent false
2384 1530 Acked Read 3 Done Done Sent false
2385 1531 Acked Write 2 Done Done Sent false
2386 1532 Acked Read 3 Done Done Sent false
2387 1533 Acked Read 10 Done Done Sent false
2388 1534 Acked Read 2 Done Done Sent false
2389 1535 Acked Read 1 Done Done Sent false
2390 1536 Acked Write 3 Done Done Sent false
2391 1537 Acked Write 3 Done Done Sent false
2392 1538 Acked Read 1 Done Done Sent false
2393 1539 Acked Read 1 Done Done Sent false
2394 1540 Acked Read 9 Done Done Sent false
2395 1541 Acked Read 7 Done Done Sent false
2396 1542 Acked Read 8 Done Done Sent false
2397 1543 Acked Read 8 Done Done Sent false
2398 1544 Acked Write 1 Done Done Sent false
2399 1545 Acked Read 6 Done Done Sent false
2400 1546 Acked Write 9 Done Done Sent false
2401 1547 Acked Read 3 Done Done Sent false
2402 1548 Acked Read 3 Done Done Sent false
2403 1549 Acked Write 9 Done Done Sent false
2404 1550 Acked Write 7 Done Done Sent false
2405 1551 Acked Write 9 Done Done Sent false
2406 1552 Acked Write 5 Done Done Sent false
2407 1553 Acked Write 10 Done Done Sent false
2408 1554 Acked Write 6 Done Done Sent false
2409 1555 Acked Write 3 Done Done Sent false
2410 1556 Acked Write 10 Done Done Sent false
2411 1557 Acked Read 1 Done Done Sent false
2412 1558 Acked Read 1 Done Done Sent false
2413 1559 Acked Read 9 Done Done Sent false
2414 1560 Acked Read 6 Done Done Sent false
2415 1561 Acked Read 9 Done Done Sent false
2416 1562 Acked Read 2 Done Done Sent false
2417 1563 Acked Read 1 Done Done Sent false
2418 1564 Acked Flush 0 Done Done Sent false
2419 1565 Acked Write 4 Done Done Sent false
2420 1566 Acked Read 1 Done Done Sent false
2421 1567 Acked Write 6 Done Done Sent false
2422 1568 Acked Write 5 Done Done Sent false
2423 1569 Acked Flush 0 Done Done Sent false
2424 1570 Acked Write 3 Done Done Sent false
2425 1571 Acked Write 10 Done Done Sent false
2426 1572 Acked Read 8 Done Done Sent false
2427 1573 Acked Write 2 Done Done Sent false
2428 1574 Acked Flush 0 Done Done Sent false
2429 1575 Acked Read 10 Done Done Sent false
2430 1576 Acked Write 8 Done Done Sent false
2431 1577 Acked Read 2 Done Done Sent false
2432 1578 Acked Write 5 Done Done Sent false
2433 1579 Acked Write 5 Done Done Sent false
2434 1580 Acked Write 2 Done Done Sent false
2435 1581 Acked Write 2 Done Done Sent false
2436 1582 Acked Write 5 Done Done Sent false
2437 1583 Acked Read 7 Done Done Sent false
2438 1584 Acked Read 1 Done Done Sent false
2439 1585 Acked Write 5 Done Done Sent false
2440 1586 Acked Write 1 Done Done Sent false
2441 1587 Acked Read 7 Done Done Sent false
2442 1588 Acked Write 10 Done Done Sent false
2443 1589 Acked Write 2 Done Done Sent false
2444 1590 Acked Write 10 Done Done Sent false
2445 1591 Acked Read 2 Done Done Sent false
2446 1592 Acked Write 3 Done Done Sent false
2447 1593 Acked Write 8 Done Done Sent false
2448 1594 Acked Read 5 Done Done Sent false
2449 1595 Acked Write 6 Done Done Sent false
2450 1596 Acked Read 1 Done Done Sent false
2451 1597 Acked Write 6 Done Done Sent false
2452 1598 Acked Write 9 Done Done Sent false
2453 1599 Acked Write 8 Done Done Sent false
2454 1600 Acked Read 7 Done Done Sent false
2455 1601 Acked Read 4 Done Done Sent false
2456 1602 Acked Read 3 Done Done Sent false
2457 STATES DS:0 DS:1 DS:2 TOTAL
2458 Sent 0 0 301 301
2459 Done 301 301 0 602
2460 Skipped 0 0 0 0
2461 Error 0 0 0 0
2462 Last Flush: Some(JobId(1574)) Some(JobId(1574)) Some(JobId(1301))
2463 Downstairs last five completed: 1301 1300 1299 1298 1297
2464 Upstairs last five completed: 1602 1601 1600 1599 1598
2465 CLIENT: Up:0 ds:301 act:2
24662025-02-07T20:07:42.980ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
24672025-02-07T20:07:43.971ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
24682025-02-07T20:07:44.973ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2469 ----------------------------------------------------------------
2470 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2471 States: Active Active Offline
2472 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2473 1302 Acked Write 1 Done Done Sent false
2474 1303 Acked Read 3 Done Done Sent false
2475 1304 Acked Write 8 Done Done Sent false
2476 1305 Acked Flush 0 Done Done Sent false
2477 1306 Acked Write 4 Done Done Sent false
2478 1307 Acked Read 1 Done Done Sent false
2479 1308 Acked Read 10 Done Done Sent false
2480 1309 Acked Read 10 Done Done Sent false
2481 1310 Acked Write 8 Done Done Sent false
2482 1311 Acked Read 8 Done Done Sent false
2483 1312 Acked Read 8 Done Done Sent false
2484 1313 Acked Write 3 Done Done Sent false
2485 1314 Acked Read 8 Done Done Sent false
2486 1315 Acked Write 6 Done Done Sent false
2487 1316 Acked Read 3 Done Done Sent false
2488 1317 Acked Read 10 Done Done Sent false
2489 1318 Acked Write 2 Done Done Sent false
2490 1319 Acked Flush 0 Done Done Sent false
2491 1320 Acked Write 10 Done Done Sent false
2492 1321 Acked Read 3 Done Done Sent false
2493 1322 Acked Write 10 Done Done Sent false
2494 1323 Acked Write 1 Done Done Sent false
2495 1324 Acked Write 10 Done Done Sent false
2496 1325 Acked Read 8 Done Done Sent false
2497 1326 Acked Write 4 Done Done Sent false
2498 1327 Acked Flush 0 Done Done Sent false
2499 1328 Acked Read 9 Done Done Sent false
2500 1329 Acked Read 7 Done Done Sent false
2501 1330 Acked Read 3 Done Done Sent false
2502 1331 Acked Write 2 Done Done Sent false
2503 1332 Acked Write 9 Done Done Sent false
2504 1333 Acked Write 8 Done Done Sent false
2505 1334 Acked Read 1 Done Done Sent false
2506 1335 Acked Write 2 Done Done Sent false
2507 1336 Acked Read 7 Done Done Sent false
2508 1337 Acked Write 8 Done Done Sent false
2509 1338 Acked Read 9 Done Done Sent false
2510 1339 Acked Read 9 Done Done Sent false
2511 1340 Acked Read 7 Done Done Sent false
2512 1341 Acked Write 9 Done Done Sent false
2513 1342 Acked Flush 0 Done Done Sent false
2514 1343 Acked Write 4 Done Done Sent false
2515 1344 Acked Read 8 Done Done Sent false
2516 1345 Acked Write 3 Done Done Sent false
2517 1346 Acked Read 9 Done Done Sent false
2518 1347 Acked Read 6 Done Done Sent false
2519 1348 Acked Read 1 Done Done Sent false
2520 1349 Acked Write 8 Done Done Sent false
2521 1350 Acked Read 8 Done Done Sent false
2522 1351 Acked Read 2 Done Done Sent false
2523 1352 Acked Read 3 Done Done Sent false
2524 1353 Acked Read 9 Done Done Sent false
2525 1354 Acked Read 4 Done Done Sent false
2526 1355 Acked Read 5 Done Done Sent false
2527 1356 Acked Read 4 Done Done Sent false
2528 1357 Acked Read 3 Done Done Sent false
2529 1358 Acked Write 5 Done Done Sent false
2530 1359 Acked Read 3 Done Done Sent false
2531 1360 Acked Read 5 Done Done Sent false
2532 1361 Acked Flush 0 Done Done Sent false
2533 1362 Acked Read 6 Done Done Sent false
2534 1363 Acked Read 3 Done Done Sent false
2535 1364 Acked Read 5 Done Done Sent false
2536 1365 Acked Write 3 Done Done Sent false
2537 1366 Acked Write 1 Done Done Sent false
2538 1367 Acked Read 8 Done Done Sent false
2539 1368 Acked Write 4 Done Done Sent false
2540 1369 Acked Read 7 Done Done Sent false
2541 1370 Acked Write 3 Done Done Sent false
2542 1371 Acked Write 7 Done Done Sent false
2543 1372 Acked Read 2 Done Done Sent false
2544 1373 Acked Write 6 Done Done Sent false
2545 1374 Acked Flush 0 Done Done Sent false
2546 1375 Acked Write 10 Done Done Sent false
2547 1376 Acked Write 7 Done Done Sent false
2548 1377 Acked Write 7 Done Done Sent false
2549 1378 Acked Read 2 Done Done Sent false
2550 1379 Acked Write 4 Done Done Sent false
2551 1380 Acked Write 3 Done Done Sent false
2552 1381 Acked Read 5 Done Done Sent false
2553 1382 Acked Write 9 Done Done Sent false
2554 1383 Acked Write 9 Done Done Sent false
2555 1384 Acked Write 5 Done Done Sent false
2556 1385 Acked Write 6 Done Done Sent false
2557 1386 Acked Read 7 Done Done Sent false
2558 1387 Acked Read 4 Done Done Sent false
2559 1388 Acked Flush 0 Done Done Sent false
2560 1389 Acked Write 10 Done Done Sent false
2561 1390 Acked Write 2 Done Done Sent false
2562 1391 Acked Flush 0 Done Done Sent false
2563 1392 Acked Read 2 Done Done Sent false
2564 1393 Acked Read 3 Done Done Sent false
2565 1394 Acked Read 1 Done Done Sent false
2566 1395 Acked Write 1 Done Done Sent false
2567 1396 Acked Read 5 Done Done Sent false
2568 1397 Acked Flush 0 Done Done Sent false
2569 1398 Acked Read 6 Done Done Sent false
2570 1399 Acked Read 10 Done Done Sent false
2571 1400 Acked Read 8 Done Done Sent false
2572 1401 Acked Read 1 Done Done Sent false
2573 1402 Acked Write 3 Done Done Sent false
2574 1403 Acked Write 10 Done Done Sent false
2575 1404 Acked Read 9 Done Done Sent false
2576 1405 Acked Read 9 Done Done Sent false
2577 1406 Acked Read 9 Done Done Sent false
2578 1407 Acked Write 8 Done Done Sent false
2579 1408 Acked Flush 0 Done Done Sent false
2580 1409 Acked Read 1 Done Done Sent false
2581 1410 Acked Read 7 Done Done Sent false
2582 1411 Acked Read 3 Done Done Sent false
2583 1412 Acked Read 6 Done Done Sent false
2584 1413 Acked Read 2 Done Done Sent false
2585 1414 Acked Read 10 Done Done Sent false
2586 1415 Acked Write 5 Done Done Sent false
2587 1416 Acked Write 9 Done Done Sent false
2588 1417 Acked Write 7 Done Done Sent false
2589 1418 Acked Read 7 Done Done Sent false
2590 1419 Acked Write 3 Done Done Sent false
2591 1420 Acked Write 1 Done Done Sent false
2592 1421 Acked Read 7 Done Done Sent false
2593 1422 Acked Read 5 Done Done Sent false
2594 1423 Acked Read 7 Done Done Sent false
2595 1424 Acked Write 1 Done Done Sent false
2596 1425 Acked Read 6 Done Done Sent false
2597 1426 Acked Read 9 Done Done Sent false
2598 1427 Acked Read 4 Done Done Sent false
2599 1428 Acked Read 5 Done Done Sent false
2600 1429 Acked Flush 0 Done Done Sent false
2601 1430 Acked Write 5 Done Done Sent false
2602 1431 Acked Write 8 Done Done Sent false
2603 1432 Acked Read 4 Done Done Sent false
2604 1433 Acked Write 4 Done Done Sent false
2605 1434 Acked Read 4 Done Done Sent false
2606 1435 Acked Read 3 Done Done Sent false
2607 1436 Acked Read 2 Done Done Sent false
2608 1437 Acked Flush 0 Done Done Sent false
2609 1438 Acked Write 6 Done Done Sent false
2610 1439 Acked Read 2 Done Done Sent false
2611 1440 Acked Read 9 Done Done Sent false
2612 1441 Acked Write 7 Done Done Sent false
2613 1442 Acked Read 6 Done Done Sent false
2614 1443 Acked Read 8 Done Done Sent false
2615 1444 Acked Write 7 Done Done Sent false
2616 1445 Acked Write 2 Done Done Sent false
2617 1446 Acked Read 10 Done Done Sent false
2618 1447 Acked Flush 0 Done Done Sent false
2619 1448 Acked Write 5 Done Done Sent false
2620 1449 Acked Read 3 Done Done Sent false
2621 1450 Acked Write 1 Done Done Sent false
2622 1451 Acked Read 4 Done Done Sent false
2623 1452 Acked Read 6 Done Done Sent false
2624 1453 Acked Read 6 Done Done Sent false
2625 1454 Acked Write 9 Done Done Sent false
2626 1455 Acked Write 7 Done Done Sent false
2627 1456 Acked Read 5 Done Done Sent false
2628 1457 Acked Flush 0 Done Done Sent false
2629 1458 Acked Read 2 Done Done Sent false
2630 1459 Acked Write 2 Done Done Sent false
2631 1460 Acked Flush 0 Done Done Sent false
2632 1461 Acked Write 10 Done Done Sent false
2633 1462 Acked Read 8 Done Done Sent false
2634 1463 Acked Read 4 Done Done Sent false
2635 1464 Acked Read 4 Done Done Sent false
2636 1465 Acked Read 4 Done Done Sent false
2637 1466 Acked Flush 0 Done Done Sent false
2638 1467 Acked Read 5 Done Done Sent false
2639 1468 Acked Read 8 Done Done Sent false
2640 1469 Acked Flush 0 Done Done Sent false
2641 1470 Acked Read 7 Done Done Sent false
2642 1471 Acked Write 2 Done Done Sent false
2643 1472 Acked Write 5 Done Done Sent false
2644 1473 Acked Write 3 Done Done Sent false
2645 1474 Acked Read 1 Done Done Sent false
2646 1475 Acked Write 2 Done Done Sent false
2647 1476 Acked Read 3 Done Done Sent false
2648 1477 Acked Read 7 Done Done Sent false
2649 1478 Acked Read 1 Done Done Sent false
2650 1479 Acked Write 4 Done Done Sent false
2651 1480 Acked Read 2 Done Done Sent false
2652 1481 Acked Write 9 Done Done Sent false
2653 1482 Acked Read 5 Done Done Sent false
2654 1483 Acked Write 4 Done Done Sent false
2655 1484 Acked Write 6 Done Done Sent false
2656 1485 Acked Read 6 Done Done Sent false
2657 1486 Acked Write 10 Done Done Sent false
2658 1487 Acked Read 3 Done Done Sent false
2659 1488 Acked Write 8 Done Done Sent false
2660 1489 Acked Write 9 Done Done Sent false
2661 1490 Acked Read 9 Done Done Sent false
2662 1491 Acked Read 3 Done Done Sent false
2663 1492 Acked Read 8 Done Done Sent false
2664 1493 Acked Write 1 Done Done Sent false
2665 1494 Acked Read 2 Done Done Sent false
2666 1495 Acked Write 4 Done Done Sent false
2667 1496 Acked Read 9 Done Done Sent false
2668 1497 Acked Write 4 Done Done Sent false
2669 1498 Acked Read 3 Done Done Sent false
2670 1499 Acked Write 8 Done Done Sent false
2671 1500 Acked Read 9 Done Done Sent false
2672 1501 Acked Flush 0 Done Done Sent false
2673 1502 Acked Read 3 Done Done Sent false
2674 1503 Acked Write 4 Done Done Sent false
2675 1504 Acked Read 2 Done Done Sent false
2676 1505 Acked Read 7 Done Done Sent false
2677 1506 Acked Read 10 Done Done Sent false
2678 1507 Acked Read 4 Done Done Sent false
2679 1508 Acked Flush 0 Done Done Sent false
2680 1509 Acked Read 6 Done Done Sent false
2681 1510 Acked Read 7 Done Done Sent false
2682 1511 Acked Write 7 Done Done Sent false
2683 1512 Acked Read 6 Done Done Sent false
2684 1513 Acked Flush 0 Done Done Sent false
2685 1514 Acked Write 5 Done Done Sent false
2686 1515 Acked Write 6 Done Done Sent false
2687 1516 Acked Write 10 Done Done Sent false
2688 1517 Acked Read 1 Done Done Sent false
2689 1518 Acked Read 9 Done Done Sent false
2690 1519 Acked Write 3 Done Done Sent false
2691 1520 Acked Write 2 Done Done Sent false
2692 1521 Acked Read 4 Done Done Sent false
2693 1522 Acked Write 3 Done Done Sent false
2694 1523 Acked Write 2 Done Done Sent false
2695 1524 Acked Read 3 Done Done Sent false
2696 1525 Acked Read 2 Done Done Sent false
2697 1526 Acked Write 2 Done Done Sent false
2698 1527 Acked Flush 0 Done Done Sent false
2699 1528 Acked Write 8 Done Done Sent false
2700 1529 Acked Read 9 Done Done Sent false
2701 1530 Acked Read 3 Done Done Sent false
2702 1531 Acked Write 2 Done Done Sent false
2703 1532 Acked Read 3 Done Done Sent false
2704 1533 Acked Read 10 Done Done Sent false
2705 1534 Acked Read 2 Done Done Sent false
2706 1535 Acked Read 1 Done Done Sent false
2707 1536 Acked Write 3 Done Done Sent false
2708 1537 Acked Write 3 Done Done Sent false
2709 1538 Acked Read 1 Done Done Sent false
2710 1539 Acked Read 1 Done Done Sent false
2711 1540 Acked Read 9 Done Done Sent false
2712 1541 Acked Read 7 Done Done Sent false
2713 1542 Acked Read 8 Done Done Sent false
2714 1543 Acked Read 8 Done Done Sent false
2715 1544 Acked Write 1 Done Done Sent false
2716 1545 Acked Read 6 Done Done Sent false
2717 1546 Acked Write 9 Done Done Sent false
2718 1547 Acked Read 3 Done Done Sent false
2719 1548 Acked Read 3 Done Done Sent false
2720 1549 Acked Write 9 Done Done Sent false
2721 1550 Acked Write 7 Done Done Sent false
2722 1551 Acked Write 9 Done Done Sent false
2723 1552 Acked Write 5 Done Done Sent false
2724 1553 Acked Write 10 Done Done Sent false
2725 1554 Acked Write 6 Done Done Sent false
2726 1555 Acked Write 3 Done Done Sent false
2727 1556 Acked Write 10 Done Done Sent false
2728 1557 Acked Read 1 Done Done Sent false
2729 1558 Acked Read 1 Done Done Sent false
2730 1559 Acked Read 9 Done Done Sent false
2731 1560 Acked Read 6 Done Done Sent false
2732 1561 Acked Read 9 Done Done Sent false
2733 1562 Acked Read 2 Done Done Sent false
2734 1563 Acked Read 1 Done Done Sent false
2735 1564 Acked Flush 0 Done Done Sent false
2736 1565 Acked Write 4 Done Done Sent false
2737 1566 Acked Read 1 Done Done Sent false
2738 1567 Acked Write 6 Done Done Sent false
2739 1568 Acked Write 5 Done Done Sent false
2740 1569 Acked Flush 0 Done Done Sent false
2741 1570 Acked Write 3 Done Done Sent false
2742 1571 Acked Write 10 Done Done Sent false
2743 1572 Acked Read 8 Done Done Sent false
2744 1573 Acked Write 2 Done Done Sent false
2745 1574 Acked Flush 0 Done Done Sent false
2746 1575 Acked Read 10 Done Done Sent false
2747 1576 Acked Write 8 Done Done Sent false
2748 1577 Acked Read 2 Done Done Sent false
2749 1578 Acked Write 5 Done Done Sent false
2750 1579 Acked Write 5 Done Done Sent false
2751 1580 Acked Write 2 Done Done Sent false
2752 1581 Acked Write 2 Done Done Sent false
2753 1582 Acked Write 5 Done Done Sent false
2754 1583 Acked Read 7 Done Done Sent false
2755 1584 Acked Read 1 Done Done Sent false
2756 1585 Acked Write 5 Done Done Sent false
2757 1586 Acked Write 1 Done Done Sent false
2758 1587 Acked Read 7 Done Done Sent false
2759 1588 Acked Write 10 Done Done Sent false
2760 1589 Acked Write 2 Done Done Sent false
2761 1590 Acked Write 10 Done Done Sent false
2762 1591 Acked Read 2 Done Done Sent false
2763 1592 Acked Write 3 Done Done Sent false
2764 1593 Acked Write 8 Done Done Sent false
2765 1594 Acked Read 5 Done Done Sent false
2766 1595 Acked Write 6 Done Done Sent false
2767 1596 Acked Read 1 Done Done Sent false
2768 1597 Acked Write 6 Done Done Sent false
2769 1598 Acked Write 9 Done Done Sent false
2770 1599 Acked Write 8 Done Done Sent false
2771 1600 Acked Read 7 Done Done Sent false
2772 1601 Acked Read 4 Done Done Sent false
2773 1602 Acked Read 3 Done Done Sent false
2774 STATES DS:0 DS:1 DS:2 TOTAL
2775 Sent 0 0 301 301
2776 Done 301 301 0 602
2777 Skipped 0 0 0 0
2778 Error 0 0 0 0
2779 Last Flush: Some(JobId(1574)) Some(JobId(1574)) Some(JobId(1301))
2780 Downstairs last five completed: 1301 1300 1299 1298 1297
2781 Upstairs last five completed: 1602 1601 1600 1599 1598
2782 CLIENT: Up:0 ds:301 act:2
27832025-02-07T20:07:46.977ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27842025-02-07T20:07:47.980ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27852025-02-07T20:07:48.980ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27862025-02-07T20:07:49.729ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27872025-02-07T20:07:49.729ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62615) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27882025-02-07T20:07:49.731ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27892025-02-07T20:07:49.731ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27902025-02-07T20:07:49.731ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27912025-02-07T20:07:49.731ZINFOcrucible: send last flush ID to this DS: Some(JobId(1301)) = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27922025-02-07T20:07:49.731ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27932025-02-07T20:07:49.731ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1301)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27942025-02-07T20:07:49.731ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1301)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2795 ----------------------------------------------------------------
2796 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:28
2797 States: Active Active Active
2798 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2799 1575 Acked Read 10 Done Done Done true
2800 1576 Acked Write 8 Done Done Done true
2801 1577 Acked Read 2 Done Done Done true
2802 1578 Acked Write 5 Done Done Done true
2803 1579 Acked Write 5 Done Done Done true
2804 1580 Acked Write 2 Done Done Done true
2805 1581 Acked Write 2 Done Done Done true
2806 1582 Acked Write 5 Done Done Done true
2807 1583 Acked Read 7 Done Done Done true
2808 1584 Acked Read 1 Done Done Done true
2809 1585 Acked Write 5 Done Done Done true
2810 1586 Acked Write 1 Done Done Done true
2811 1587 Acked Read 7 Done Done Done true
2812 1588 Acked Write 10 Done Done Done true
2813 1589 Acked Write 2 Done Done Done true
2814 1590 Acked Write 10 Done Done Done true
2815 1591 Acked Read 2 Done Done Done true
2816 1592 Acked Write 3 Done Done Done true
2817 1593 Acked Write 8 Done Done Done true
2818 1594 Acked Read 5 Done Done Done true
2819 1595 Acked Write 6 Done Done Done true
2820 1596 Acked Read 1 Done Done Done true
2821 1597 Acked Write 6 Done Done Done true
2822 1598 Acked Write 9 Done Done Done true
2823 1599 Acked Write 8 Done Done Done true
2824 1600 Acked Read 7 Done Done Done true
2825 1601 Acked Read 4 Done Done Done true
2826 1602 Acked Read 3 Done Done Done true
2827 STATES DS:0 DS:1 DS:2 TOTAL
2828 Sent 0 0 0 0
2829 Done 28 28 28 84
2830 Skipped 0 0 0 0
2831 Error 0 0 0 0
2832 Last Flush: Some(JobId(1574)) Some(JobId(1574)) Some(JobId(1574))
2833 Downstairs last five completed: 1574 1573 1572 1571 1570
2834 Upstairs last five completed: 1602 1601 1600 1599 1598
2835 CLIENT: Up:0 ds:28 act:3
2836 ----------------------------------------------------------------
2837 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2838 Downstairs last five completed: 1603 1602 1601 1600 1599
2839 Upstairs last five completed: 1603 1602 1601 1600 1599
2840 CLIENT: Up:0 ds:0 act:3
2841 Replay: All jobs finished
2842 Main task has work to do, go find it
2843 got dsc Stop(2)
2844 stop 2
2845 Feb 07 20:07:53.978 INFO request completed[8830] Got stop action so:false kr:false
2846 , latency_us: 370, response_code: 204, uri: /stop/cid/2, method: POST, req_id: b49a3632-6c08-4230-a0ca-75af16f4082e, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
28472025-02-07T20:07:53.979ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
28482025-02-07T20:07:53.979ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
28492025-02-07T20:07:53.979ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
28502025-02-07T20:07:53.979ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
2851 Feb 07 20:07:53.979 INFO request completed, latency_us: 152, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3c3ef170-5d3d-455b-bae6-77583a90c8d7, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
2852 [8830] Exited with: ExitStatus(unix_wait_status(9))
2853 [8830][2] reports Exit
2854 Feb 07 20:07:57.981 INFO request completed, latency_us: 244, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0e3f4f22-33d8-4076-93cd-8b0ed1ae3e92, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
2855 001/300 Read block 4479 len 36864
2856 002/300 Write block 4520 len 40960 data: 3 3 3 3 2 2 2 2 2 2
2857 003/300 Read block 3253 len 32768
2858 004/300 Write block 4120 len 32768 data: 2 2 2 3 3 3 3 3
2859 005/300 Read block 4717 len 16384
2860 006/300 Read block 2482 len 4096
2861 007/300 Write block 2559 len 32768 data: 2 2 2 2 2 2 2 2
2862 008/300 Write block 2273 len 28672 data: 2 2 2 2 2 2 2
2863 009/300 Write block 1799 len 36864 data: 3 2 2 2 2 2 2 2 2
2864 010/300 Write block 3424 len 8192 data: 2 2
2865 011/300 Write block 4943 len 36864 data: 2 3 3 3 3 3 3 3 2
2866 012/300 Flush
2867 013/300 Write block 2952 len 24576 data: 2 3 3 3 3 3
2868 014/300 Read block 2238 len 32768
2869 015/300 Read block 938 len 28672
2870 016/300 Read block 4180 len 28672
2871 017/300 Flush
2872 018/300 Read block 3012 len 36864
2873 019/300 Read block 4904 len 32768
2874 020/300 Write block 4488 len 4096 data: 2
2875 021/300 Write block 4832 len 12288 data: 3 3 3
2876 022/300 Read block 4392 len 12288
2877 023/300 Read block 826 len 20480
2878 024/300 Flush
2879 025/300 Read block 4581 len 4096
2880 026/300 Read block 2239 len 8192
2881 027/300 Flush
2882 028/300 Read block 44 len 20480
2883 029/300 Write block 4897 len 8192 data: 3 3
2884 030/300 Write block 4882 len 40960 data: 2 2 2 2 3 3 3 3 3 2
2885 031/300 Flush
2886 032/300 Read block 2963 len 8192
2887 033/300 Write block 4622 len 12288 data: 2 2 2
2888 034/300 Flush
2889 035/300 Read block 1992 len 28672
2890 036/300 Read block 1382 len 8192
2891 037/300 Read block 1097 len 40960
2892 038/300 Read block 2361 len 4096
2893 039/300 Write block 2235 len 4096 data: 2
2894 040/300 Flush
2895 041/300 Write block 4256 len 16384 data: 2 2 3 3
2896 042/300 Write block 523 len 28672 data: 2 2 2 2 2 2 2
2897 043/300 Write block 2335 len 4096 data: 2
2898 044/300 Write block 1557 len 24576 data: 2 2 2 2 2 2
2899 045/300 Read block 903 len 8192
2900 046/300 Read block 3955 len 20480
2901 047/300 Write block 4911 len 40960 data: 2 2 2 2 2 2 2 2 2 3
2902 048/300 Write block 751 len 20480 data: 2 2 3 3 2
2903 049/300 Write block 4497 len 32768 data: 3 3 3 3 3 5 4 4
2904 050/300 Write block 3323 len 20480 data: 2 2 2 2 2
2905 051/300 Write block 2856 len 4096 data: 2
2906 052/300 Write block 1782 len 32768 data: 2 2 2 2 2 2 2 2
2907 053/300 Write block 2120 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2908 054/300 Write block 3394 len 36864 data: 3 3 3 3 3 3 2 2 2
2909 055/300 Write block 1658 len 4096 data: 3
2910 056/300 Read block 3325 len 36864
2911 057/300 Write block 2186 len 36864 data: 2 2 2 2 2 2 2 2 2
2912 058/300 Write block 3969 len 20480 data: 2 2 2 2 2
2913 059/300 Write block 1213 len 4096 data: 3
2914 060/300 Write block 3361 len 20480 data: 2 2 2 2 2
2915 061/300 Read block 2696 len 28672
2916 062/300 Write block 3203 len 20480 data: 2 3 3 3 2
2917 063/300 Read block 670 len 36864
2918 064/300 Read block 4242 len 16384
2919 065/300 Flush
2920 066/300 Write block 2098 len 16384 data: 2 2 2 2
2921 067/300 Write block 1319 len 28672 data: 2 3 3 3 3 3 3
2922 068/300 Read block 3529 len 16384
2923 069/300 Write block 3877 len 4096 data: 2
2924 070/300 Write block 2939 len 4096 data: 2
2925 071/300 Read block 4581 len 36864
2926 072/300 Read block 1669 len 12288
2927 073/300 Read block 4133 len 36864
2928 074/300 Write block 993 len 40960 data: 3 3 3 2 2 3 3 3 4 4
2929 075/300 Write block 4863 len 8192 data: 2 2
2930 076/300 Write block 4799 len 32768 data: 2 2 2 2 2 2 2 2
2931 077/300 Write block 424 len 28672 data: 2 2 2 2 2 2 2
2932 078/300 Flush
2933 079/300 Flush
2934 080/300 Write block 3556 len 4096 data: 2
2935 081/300 Read block 4958 len 12288
2936 082/300 Read block 96 len 28672
2937 083/300 Write block 492 len 12288 data: 2 2 2
2938 084/300 Read block 1813 len 4096
2939 085/300 Read block 3415 len 20480
2940 086/300 Flush
2941 087/300 Write block 4493 len 40960 data: 4 4 4 3 4 4 4 4 4 6
2942 088/300 Read block 4718 len 28672
2943 089/300 Read block 269 len 32768
2944 090/300 Read block 1504 len 12288
2945 091/300 Flush
2946 092/300 Write block 4784 len 32768 data: 2 2 2 2 2 2 2 2
2947 093/300 Read block 1552 len 20480
2948 094/300 Write block 4613 len 8192 data: 2 2
2949 095/300 Flush
2950 096/300 Flush
2951 097/300 Write block 1230 len 8192 data: 2 2
2952 098/300 Read block 4770 len 16384
2953 099/300 Write block 4225 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2954 100/300 Read block 2700 len 12288
2955 101/300 Read block 763 len 16384
2956 102/300 Read block 4838 len 20480
2957 103/300 Read block 600 len 20480
2958 104/300 Write block 3520 len 24576 data: 2 2 2 2 2 2
2959 105/300 Read block 691 len 16384
2960 106/300 Write block 3366 len 4096 data: 2
2961 107/300 Read block 842 len 36864
2962 108/300 Write block 214 len 12288 data: 2 2 2
2963 109/300 Read block 1880 len 36864
2964 110/300 Read block 3562 len 12288
2965 111/300 Read block 1596 len 24576
2966 112/300 Read block 4385 len 32768
2967 113/300 Flush
2968 114/300 Read block 3153 len 12288
2969 115/300 Read block 431 len 16384
2970 116/300 Write block 4031 len 24576 data: 2 2 2 2 2 2
2971 117/300 Read block 1515 len 24576
2972 118/300 Read block 1802 len 16384
2973 119/300 Read block 585 len 24576
2974 120/300 Write block 2341 len 32768 data: 3 3 2 2 2 2 3 3
2975 121/300 Write block 1111 len 24576 data: 4 3 3 3 3 3
2976 122/300 Read block 4038 len 40960
2977 123/300 Read block 3516 len 24576
2978 124/300 Flush
2979 125/300 Write block 1045 len 40960 data: 2 2 2 2 3 3 3 3 3 3
2980 126/300 Read block 4553 len 4096
2981 127/300 Read block 1106 len 8192
2982 128/300 Read block 2803 len 20480
2983 129/300 Read block 3362 len 28672
2984 130/300 Read block 812 len 12288
2985 131/300 Flush
2986 132/300 Read block 4513 len 36864
2987 133/300 Read block 1486 len 4096
2988 134/300 Write block 2623 len 28672 data: 2 2 2 2 2 2 2
2989 135/300 Write block 627 len 32768 data: 2 2 2 2 2 2 2 2
2990 136/300 Flush
2991 137/300 Write block 3805 len 36864 data: 2 2 2 2 2 2 2 3 3
2992 138/300 Write block 725 len 4096 data: 2
2993 139/300 Write block 2078 len 20480 data: 4 3 3 3 3
2994 140/300 Write block 2910 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2995 141/300 Read block 4471 len 20480
2996 142/300 Write block 4504 len 36864 data: 5 4 3 4 4 4 4 3 2
2997 143/300 Read block 597 len 12288
2998 144/300 Read block 4338 len 36864
2999 145/300 Read block 742 len 20480
3000 146/300 Read block 1385 len 4096
3001 147/300 Read block 3292 len 24576
3002 148/300 Write block 3696 len 8192 data: 3 2
3003 149/300 Write block 127 len 28672 data: 2 2 2 2 2 2 2
3004 150/300 Write block 3767 len 20480 data: 2 2 2 2 2
3005 151/300 Write block 1811 len 36864 data: 2 2 2 2 2 2 2 2 2
3006 152/300 Write block 954 len 12288 data: 2 2 2
3007 153/300 Flush
3008 154/300 Read block 3606 len 40960
3009 155/300 Read block 4868 len 4096
3010 156/300 Write block 2795 len 40960 data: 3 3 3 3 3 3 4 5 4 3
3011 157/300 Write block 612 len 24576 data: 3 3 3 2 2 2
3012 158/300 Read block 4036 len 16384
3013 159/300 Read block 2871 len 28672
3014 160/300 Read block 2146 len 4096
3015 161/300 Read block 2035 len 4096
3016 162/300 Read block 2144 len 28672
3017 163/300 Read block 1272 len 24576
3018 164/300 Read block 3456 len 4096
3019 165/300 Flush
3020 166/300 Read block 3779 len 4096
3021 167/300 Read block 1875 len 20480
3022 168/300 Read block 3794 len 8192
3023 169/300 Read block 4455 len 4096
3024 170/300 Read block 2552 len 16384
3025 171/300 Read block 3601 len 40960
3026 172/300 Read block 889 len 40960
3027 173/300 Read block 2605 len 32768
3028 174/300 Read block 181 len 12288
3029 175/300 Write block 2802 len 32768 data: 6 5 4 3 3 3 3 3
3030 176/300 Write block 4747 len 4096 data: 3
3031 177/300 Flush
3032 178/300 Flush
3033 179/300 Read block 726 len 28672
3034 180/300 Read block 1687 len 32768
3035 181/300 Flush
3036 182/300 Flush
3037 183/300 Read block 3101 len 20480
3038 184/300 Read block 3188 len 12288
3039 185/300 Write block 4246 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3040 186/300 Read block 1924 len 24576
3041 187/300 Read block 2688 len 40960
3042 188/300 Write block 2123 len 32768 data: 3 3 3 3 3 3 3 2
3043 189/300 Read block 3532 len 28672
3044 190/300 Write block 1218 len 32768 data: 2 2 2 2 2 2 2 2
3045 191/300 Write block 455 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3046 192/300 Read block 807 len 8192
3047 193/300 Flush
3048 194/300 Write block 287 len 16384 data: 3 3 3 2
3049 195/300 Flush
3050 196/300 Read block 2074 len 32768
3051 197/300 Read block 3517 len 24576
3052 198/300 Write block 1651 len 12288 data: 4 4 4
3053 199/300 Write block 4901 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3054 200/300 Read block 3421 len 8192
3055 201/300 Write block 269 len 36864 data: 2 2 2 2 2 2 2 2 2
3056 202/300 Flush
3057 203/300 Write block 3274 len 28672 data: 2 2 2 2 2 2 2
3058 204/300 Read block 871 len 36864
3059 205/300 Read block 800 len 12288
3060 206/300 Write block 4857 len 32768 data: 2 2 2 2 2 2 3 3
3061 207/300 Write block 2160 len 32768 data: 2 2 3 3 3 3 3 3
3062 208/300 Flush
3063 209/300 Read block 1828 len 12288
3064 210/300 Read block 42 len 24576
3065 211/300 Read block 4995 len 20480
3066 212/300 Write block 2565 len 8192 data: 3 3
3067 213/300 Flush
3068 214/300 Read block 3546 len 8192
3069 215/300 Write block 3398 len 36864 data: 4 4 3 3 3 2 2 2 2
3070 216/300 Read block 3949 len 40960
3071 217/300 Write block 2791 len 40960 data: 2 2 3 3 4 4 4 4 4 4
3072 218/300 Read block 2124 len 12288
3073 219/300 Read block 918 len 16384
3074 220/300 Write block 4585 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3075 221/300 Read block 373 len 40960
3076 222/300 Write block 3542 len 16384 data: 2 2 2 2
3077 223/300 Write block 1986 len 40960 data: 3 4 4 3 2 2 2 2 2 2
3078 224/300 Read block 868 len 12288
3079 225/300 Write block 3394 len 32768 data: 4 4 4 4 5 5 4 4
3080 226/300 Read block 329 len 16384
3081 227/300 Read block 1558 len 16384
3082 228/300 Write block 4869 len 28672 data: 2 2 2 2 2 2 2
3083 229/300 Write block 2980 len 32768 data: 3 3 3 3 2 2 2 2
3084 230/300 Write block 2707 len 32768 data: 2 2 2 2 2 2 3 2
3085 231/300 Read block 2288 len 4096
3086 232/300 Write block 3719 len 8192 data: 3 4
3087 233/300 Write block 4108 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3088 234/300 Read block 1 len 40960
3089 235/300 Read block 2899 len 28672
3090 236/300 Write block 184 len 36864 data: 2 2 2 2 2 2 2 2 2
3091 237/300 Read block 2321 len 20480
3092 238/300 Write block 1034 len 20480 data: 2 2 2 2 2
3093 239/300 Write block 3562 len 4096 data: 2
3094 240/300 Write block 2592 len 24576 data: 2 2 2 2 2 2
3095 241/300 Read block 2433 len 28672
3096 242/300 Read block 1564 len 28672
3097 243/300 Write block 304 len 36864 data: 3 3 3 2 2 2 2 2 2
3098 244/300 Read block 2279 len 4096
3099 245/300 Flush
3100 246/300 Read block 3480 len 36864
3101 247/300 Write block 3622 len 8192 data: 2 2
3102 248/300 Read block 3450 len 24576
3103 249/300 Read block 3773 len 16384
3104 250/300 Read block 4256 len 40960
3105 251/300 Write block 158 len 24576 data: 2 2 2 2 3 3
3106 252/300 Read block 1989 len 4096
3107 253/300 Flush
3108 254/300 Write block 167 len 16384 data: 2 2 2 2
3109 255/300 Write block 587 len 12288 data: 2 2 2
3110 256/300 Write block 497 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3111 257/300 Write block 366 len 4096 data: 2
3112 258/300 Read block 1234 len 40960
3113 259/300 Read block 2050 len 12288
3114 260/300 Read block 4012 len 32768
3115 261/300 Write block 1127 len 16384 data: 2 2 2 2
3116 262/300 Read block 3611 len 32768
3117 263/300 Write block 2958 len 20480 data: 2 2 2 2 2
3118 264/300 Write block 3089 len 12288 data: 2 2 2
3119 265/300 Read block 1898 len 20480
3120 266/300 Read block 1001 len 40960
3121 267/300 Write block 311 len 20480 data: 3 3 2 2 2
3122 268/300 Flush
3123 269/300 Write block 3168 len 24576 data: 2 2 2 2 2 2
3124 270/300 Flush
3125 271/300 Read block 21 len 8192
3126 272/300 Read block 838 len 40960
3127 273/300 Read block 838 len 32768
3128 274/300 Flush
3129 275/300 Write block 3918 len 28672 data: 3 3 2 2 2 2 2
3130 276/300 Read block 3071 len 32768
3131 277/300 Read block 2712 len 32768
3132 278/300 Write block 2820 len 20480 data: 3 3 2 2 2
3133 279/300 Read block 1013 len 36864
3134 280/300 Write block 2710 len 24576 data: 3 3 3 4 3 2
3135 281/300 Read block 308 len 40960
3136 282/300 Write block 1921 len 16384 data: 2 2 2 2
3137 283/300 Read block 4892 len 4096
3138 284/300 Write block 2975 len 4096 data: 2
3139 285/300 Read block 444 len 28672
3140 286/300 Write block 4415 len 36864 data: 2 2 2 2 2 2 2 3 3
3141 287/300 Write block 2425 len 8192 data: 2 2
3142 288/300 Read block 1282 len 32768
3143 289/300 Write block 3749 len 16384 data: 3 3 3 3
3144 290/300 Write block 1279 len 8192 data: 2 2
3145 291/300 Write block 2685 len 4096 data: 2
3146 292/300 Flush
3147 293/300 Read block 471 len 16384
3148 294/300 Read block 1335 len 40960
3149 295/300 Read block 3920 len 12288
3150 296/300 Write block 4274 len 12288 data: 2 2 2
3151 297/300 Flush
3152 298/300 Write block 3842 len 32768 data: 2 2 2 2 2 2 2 2
3153 299/300 Read block 357 len 12288
3154 300/300 Read block 2588 len 4096
3155 301/300 Read block 3017 len 12288
3156 Main task has work to do, go find it
3157 got dsc Start(2)
3158 start 2
3159 Feb 07 20:07:58.220 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/2, method: POST, req_id: f2954699-f4ae-416c-af01-e9ab5e6cd787, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
3160 Replay: started 2, returned:Ok(())
3161 ----------------------------------------------------------------
3162 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3163 States: Active Active Offline
3164 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3165 1604 Acked Read 9 Done Done Sent false
3166 1605 Acked Write 10 Done Done Sent false
3167 1606 Acked Read 8 Done Done Sent false
3168 1607 Acked Write 8 Done Done Sent false
3169 1608 Acked Read 4 Done Done Sent false
3170 1609 Acked Read 1 Done Done Sent false
3171 1610 Acked Write 8 Done Done Sent false
3172 1611 Acked Write 7 Done Done Sent false
3173 1612 Acked Write 9 Done Done Sent false
3174 1613 Acked Write 2 Done Done Sent false
3175 1614 Acked Write 9 Done Done Sent false
3176 1615 Acked Flush 0 Done Done Sent false
3177 1616 Acked Write 6 Done Done Sent false
3178 1617 Acked Read 8 Done Done Sent false
3179 1618 Acked Read 7 Done Done Sent false
3180 1619 Acked Read 7 Done Done Sent false
3181 1620 Acked Flush 0 Done Done Sent false
3182 1621 Acked Read 9 Done Done Sent false
3183 1622 Acked Read 8 Done Done Sent false
3184 1623 Acked Write 1 Done Done Sent false
3185 1624 Acked Write 3 Done Done Sent false
3186 1625 Acked Read 3 Done Done Sent false
3187 1626 Acked Read 5 Done Done Sent false
3188 1627 Acked Flush 0 Done Done Sent false
3189 1628 Acked Read 1 Done Done Sent false
3190 1629 Acked Read 2 Done Done Sent false
3191 1630 Acked Flush 0 Done Done Sent false
3192 1631 Acked Read 5 Done Done Sent false
3193 1632 Acked Write 2 Done Done Sent false
3194 1633 Acked Write 10 Done Done Sent false
3195 1634 Acked Flush 0 Done Done Sent false
3196 1635 Acked Read 2 Done Done Sent false
3197 1636 Acked Write 3 Done Done Sent false
3198 1637 Acked Flush 0 Done Done Sent false
3199 1638 Acked Read 7 Done Done Sent false
3200 1639 Acked Read 2 Done Done Sent false
3201 1640 Acked Read 10 Done Done Sent false
3202 1641 Acked Read 1 Done Done Sent false
3203 1642 Acked Write 1 Done Done Sent false
3204 1643 Acked Flush 0 Done Done Sent false
3205 1644 Acked Write 4 Done Done Sent false
3206 1645 Acked Write 7 Done Done Sent false
3207 1646 Acked Write 1 Done Done Sent false
3208 1647 Acked Write 6 Done Done Sent false
3209 1648 Acked Read 2 Done Done Sent false
3210 1649 Acked Read 5 Done Done Sent false
3211 1650 Acked Write 10 Done Done Sent false
3212 1651 Acked Write 5 Done Done Sent false
3213 1652 Acked Write 8 Done Done Sent false
3214 1653 Acked Write 5 Done Done Sent false
3215 1654 Acked Write 1 Done Done Sent false
3216 1655 Acked Write 8 Done Done Sent false
3217 1656 Acked Write 10 Done Done Sent false
3218 1657 Acked Write 9 Done Done Sent false
3219 1658 Acked Write 1 Done Done Sent false
3220 1659 Acked Read 9 Done Done Sent false
3221 1660 Acked Write 9 Done Done Sent false
3222 1661 Acked Write 5 Done Done Sent false
3223 1662 Acked Write 1 Done Done Sent false
3224 1663 Acked Write 5 Done Done Sent false
3225 1664 Acked Read 7 Done Done Sent false
3226 1665 Acked Write 5 Done Done Sent false
3227 1666 Acked Read 9 Done Done Sent false
3228 1667 Acked Read 4 Done Done Sent false
3229 1668 Acked Flush 0 Done Done Sent false
3230 1669 Acked Write 4 Done Done Sent false
3231 1670 Acked Write 7 Done Done Sent false
3232 1671 Acked Read 4 Done Done Sent false
3233 1672 Acked Write 1 Done Done Sent false
3234 1673 Acked Write 1 Done Done Sent false
3235 1674 Acked Read 9 Done Done Sent false
3236 1675 Acked Read 3 Done Done Sent false
3237 1676 Acked Read 9 Done Done Sent false
3238 1677 Acked Write 10 Done Done Sent false
3239 1678 Acked Write 2 Done Done Sent false
3240 1679 Acked Write 8 Done Done Sent false
3241 1680 Acked Write 7 Done Done Sent false
3242 1681 Acked Flush 0 Done Done Sent false
3243 1682 Acked Flush 0 Done Done Sent false
3244 1683 Acked Write 1 Done Done Sent false
3245 1684 Acked Read 3 Done Done Sent false
3246 1685 Acked Read 7 Done Done Sent false
3247 1686 Acked Write 3 Done Done Sent false
3248 1687 Acked Read 1 Done Done Sent false
3249 1688 Acked Read 5 Done Done Sent false
3250 1689 Acked Flush 0 Done Done Sent false
3251 1690 Acked Write 10 Done Done Sent false
3252 1691 Acked Read 7 Done Done Sent false
3253 1692 Acked Read 8 Done Done Sent false
3254 1693 Acked Read 3 Done Done Sent false
3255 1694 Acked Flush 0 Done Done Sent false
3256 1695 Acked Write 8 Done Done Sent false
3257 1696 Acked Read 5 Done Done Sent false
3258 1697 Acked Write 2 Done Done Sent false
3259 1698 Acked Flush 0 Done Done Sent false
3260 1699 Acked Flush 0 Done Done Sent false
3261 1700 Acked Write 2 Done Done Sent false
3262 1701 Acked Read 4 Done Done Sent false
3263 1702 Acked Write 10 Done Done Sent false
3264 1703 Acked Read 3 Done Done Sent false
3265 1704 Acked Read 4 Done Done Sent false
3266 1705 Acked Read 5 Done Done Sent false
3267 1706 Acked Read 5 Done Done Sent false
3268 1707 Acked Write 6 Done Done Sent false
3269 1708 Acked Read 4 Done Done Sent false
3270 1709 Acked Write 1 Done Done Sent false
3271 1710 Acked Read 9 Done Done Sent false
3272 1711 Acked Write 3 Done Done Sent false
3273 1712 Acked Read 9 Done Done Sent false
3274 1713 Acked Read 3 Done Done Sent false
3275 1714 Acked Read 6 Done Done Sent false
3276 1715 Acked Read 8 Done Done Sent false
3277 1716 Acked Flush 0 Done Done Sent false
3278 1717 Acked Read 3 Done Done Sent false
3279 1718 Acked Read 4 Done Done Sent false
3280 1719 Acked Write 6 Done Done Sent false
3281 1720 Acked Read 6 Done Done Sent false
3282 1721 Acked Read 4 Done Done Sent false
3283 1722 Acked Read 6 Done Done Sent false
3284 1723 Acked Write 8 Done Done Sent false
3285 1724 Acked Write 6 Done Done Sent false
3286 1725 Acked Read 10 Done Done Sent false
3287 1726 Acked Read 6 Done Done Sent false
3288 1727 Acked Flush 0 Done Done Sent false
3289 1728 Acked Write 10 Done Done Sent false
3290 1729 Acked Read 1 Done Done Sent false
3291 1730 Acked Read 2 Done Done Sent false
3292 1731 Acked Read 5 Done Done Sent false
3293 1732 Acked Read 7 Done Done Sent false
3294 1733 Acked Read 3 Done Done Sent false
3295 1734 Acked Flush 0 Done Done Sent false
3296 1735 Acked Read 9 Done Done Sent false
3297 1736 Acked Read 1 Done Done Sent false
3298 1737 Acked Write 7 Done Done Sent false
3299 1738 Acked Write 8 Done Done Sent false
3300 1739 Acked Flush 0 Done Done Sent false
3301 1740 Acked Write 9 Done Done Sent false
3302 1741 Acked Write 1 Done Done Sent false
3303 1742 Acked Write 5 Done Done Sent false
3304 1743 Acked Write 10 Done Done Sent false
3305 1744 Acked Read 5 Done Done Sent false
3306 1745 Acked Write 9 Done Done Sent false
3307 1746 Acked Read 3 Done Done Sent false
3308 1747 Acked Read 9 Done Done Sent false
3309 1748 Acked Read 5 Done Done Sent false
3310 1749 Acked Read 1 Done Done Sent false
3311 1750 Acked Read 6 Done Done Sent false
3312 1751 Acked Write 2 Done Done Sent false
3313 1752 Acked Write 7 Done Done Sent false
3314 1753 Acked Write 5 Done Done Sent false
3315 1754 Acked Write 9 Done Done Sent false
3316 1755 Acked Write 3 Done Done Sent false
3317 1756 Acked Flush 0 Done Done Sent false
3318 1757 Acked Read 10 Done Done Sent false
3319 1758 Acked Read 1 Done Done Sent false
3320 1759 Acked Write 10 Done Done Sent false
3321 1760 Acked Write 6 Done Done Sent false
3322 1761 Acked Read 4 Done Done Sent false
3323 1762 Acked Read 7 Done Done Sent false
3324 1763 Acked Read 1 Done Done Sent false
3325 1764 Acked Read 1 Done Done Sent false
3326 1765 Acked Read 7 Done Done Sent false
3327 1766 Acked Read 6 Done Done Sent false
3328 1767 Acked Read 1 Done Done Sent false
3329 1768 Acked Flush 0 Done Done Sent false
3330 1769 Acked Read 1 Done Done Sent false
3331 1770 Acked Read 5 Done Done Sent false
3332 1771 Acked Read 2 Done Done Sent false
3333 1772 Acked Read 1 Done Done Sent false
3334 1773 Acked Read 4 Done Done Sent false
3335 1774 Acked Read 10 Done Done Sent false
3336 1775 Acked Read 10 Done Done Sent false
3337 1776 Acked Read 8 Done Done Sent false
3338 1777 Acked Read 3 Done Done Sent false
3339 1778 Acked Write 8 Done Done Sent false
3340 1779 Acked Write 1 Done Done Sent false
3341 1780 Acked Flush 0 Done Done Sent false
3342 1781 Acked Flush 0 Done Done Sent false
3343 1782 Acked Read 7 Done Done Sent false
3344 1783 Acked Read 8 Done Done Sent false
3345 1784 Acked Flush 0 Done Done Sent false
3346 1785 Acked Flush 0 Done Done Sent false
3347 1786 Acked Read 5 Done Done Sent false
3348 1787 Acked Read 3 Done Done Sent false
3349 1788 Acked Write 10 Done Done Sent false
3350 1789 Acked Read 6 Done Done Sent false
3351 1790 Acked Read 10 Done Done Sent false
3352 1791 Acked Write 8 Done Done Sent false
3353 1792 Acked Read 7 Done Done Sent false
3354 1793 Acked Write 8 Done Done Sent false
3355 1794 Acked Write 10 Done Done Sent false
3356 1795 Acked Read 2 Done Done Sent false
3357 1796 Acked Flush 0 Done Done Sent false
3358 1797 Acked Write 4 Done Done Sent false
3359 1798 Acked Flush 0 Done Done Sent false
3360 1799 Acked Read 8 Done Done Sent false
3361 1800 Acked Read 6 Done Done Sent false
3362 1801 Acked Write 3 Done Done Sent false
3363 1802 Acked Write 10 Done Done Sent false
3364 1803 Acked Read 2 Done Done Sent false
3365 1804 Acked Write 9 Done Done Sent false
3366 1805 Acked Flush 0 Done Done Sent false
3367 1806 Acked Write 7 Done Done Sent false
3368 1807 Acked Read 9 Done Done Sent false
3369 1808 Acked Read 3 Done Done Sent false
3370 1809 Acked Write 8 Done Done Sent false
3371 1810 Acked Write 8 Done Done Sent false
3372 1811 Acked Flush 0 Done Done Sent false
3373 1812 Acked Read 3 Done Done Sent false
3374 1813 Acked Read 6 Done Done Sent false
3375 1814 Acked Read 5 Done Done Sent false
3376 1815 Acked Write 2 Done Done Sent false
3377 1816 Acked Flush 0 Done Done Sent false
3378 1817 Acked Read 2 Done Done Sent false
3379 1818 Acked Write 9 Done Done Sent false
3380 1819 Acked Read 10 Done Done Sent false
3381 1820 Acked Write 10 Done Done Sent false
3382 1821 Acked Read 3 Done Done Sent false
3383 1822 Acked Read 4 Done Done Sent false
3384 1823 Acked Write 10 Done Done Sent false
3385 1824 Acked Read 10 Done Done Sent false
3386 1825 Acked Write 4 Done Done Sent false
3387 1826 Acked Write 10 Done Done Sent false
3388 1827 Acked Read 3 Done Done Sent false
3389 1828 Acked Write 8 Done Done Sent false
3390 1829 Acked Read 4 Done Done Sent false
3391 1830 Acked Read 4 Done Done Sent false
3392 1831 Acked Write 7 Done Done Sent false
3393 1832 Acked Write 8 Done Done Sent false
3394 1833 Acked Write 8 Done Done Sent false
3395 1834 Acked Read 1 Done Done Sent false
3396 1835 Acked Write 2 Done Done Sent false
3397 1836 Acked Write 10 Done Done Sent false
3398 1837 Acked Read 10 Done Done Sent false
3399 1838 Acked Read 7 Done Done Sent false
3400 1839 Acked Write 9 Done Done Sent false
3401 1840 Acked Read 5 Done Done Sent false
3402 1841 Acked Write 5 Done Done Sent false
3403 1842 Acked Write 1 Done Done Sent false
3404 1843 Acked Write 6 Done Done Sent false
3405 1844 Acked Read 7 Done Done Sent false
3406 1845 Acked Read 7 Done Done Sent false
3407 1846 Acked Write 9 Done Done Sent false
3408 1847 Acked Read 1 Done Done Sent false
3409 1848 Acked Flush 0 Done Done Sent false
3410 1849 Acked Read 9 Done Done Sent false
3411 1850 Acked Write 2 Done Done Sent false
3412 1851 Acked Read 6 Done Done Sent false
3413 1852 Acked Read 4 Done Done Sent false
3414 1853 Acked Read 10 Done Done Sent false
3415 1854 Acked Write 6 Done Done Sent false
3416 1855 Acked Read 1 Done Done Sent false
3417 1856 Acked Flush 0 Done Done Sent false
3418 1857 Acked Write 4 Done Done Sent false
3419 1858 Acked Write 3 Done Done Sent false
3420 1859 Acked Write 10 Done Done Sent false
3421 1860 Acked Write 1 Done Done Sent false
3422 1861 Acked Read 10 Done Done Sent false
3423 1862 Acked Read 3 Done Done Sent false
3424 1863 Acked Read 8 Done Done Sent false
3425 1864 Acked Write 4 Done Done Sent false
3426 1865 Acked Read 8 Done Done Sent false
3427 1866 Acked Write 5 Done Done Sent false
3428 1867 Acked Write 3 Done Done Sent false
3429 1868 Acked Read 5 Done Done Sent false
3430 1869 Acked Read 10 Done Done Sent false
3431 1870 Acked Write 5 Done Done Sent false
3432 1871 Acked Flush 0 Done Done Sent false
3433 1872 Acked Write 6 Done Done Sent false
3434 1873 Acked Flush 0 Done Done Sent false
3435 1874 Acked Read 2 Done Done Sent false
3436 1875 Acked Read 10 Done Done Sent false
3437 1876 Acked Read 8 Done Done Sent false
3438 1877 Acked Flush 0 Done Done Sent false
3439 1878 Acked Write 7 Done Done Sent false
3440 1879 Acked Read 8 Done Done Sent false
3441 1880 Acked Read 8 Done Done Sent false
3442 1881 Acked Write 5 Done Done Sent false
3443 1882 Acked Read 9 Done Done Sent false
3444 1883 Acked Write 6 Done Done Sent false
3445 1884 Acked Read 10 Done Done Sent false
3446 1885 Acked Write 4 Done Done Sent false
3447 1886 Acked Read 1 Done Done Sent false
3448 1887 Acked Write 1 Done Done Sent false
3449 1888 Acked Read 7 Done Done Sent false
3450 1889 Acked Write 9 Done Done Sent false
3451 1890 Acked Write 2 Done Done Sent false
3452 1891 Acked Read 8 Done Done Sent false
3453 1892 Acked Write 4 Done Done Sent false
3454 1893 Acked Write 2 Done Done Sent false
3455 1894 Acked Write 1 Done Done Sent false
3456 1895 Acked Flush 0 Done Done Sent false
3457 1896 Acked Read 4 Done Done Sent false
3458 1897 Acked Read 10 Done Done Sent false
3459 1898 Acked Read 3 Done Done Sent false
3460 1899 Acked Write 3 Done Done Sent false
3461 1900 Acked Flush 0 Done Done Sent false
3462 1901 Acked Write 8 Done Done Sent false
3463 1902 Acked Read 3 Done Done Sent false
3464 1903 Acked Read 1 Done Done Sent false
3465 1904 Acked Read 3 Done Done Sent false
3466 STATES DS:0 DS:1 DS:2 TOTAL
3467 Sent 0 0 301 301
3468 Done 301 301 0 602
3469 Skipped 0 0 0 0
3470 Error 0 0 0 0
3471 Last Flush: Some(JobId(1900)) Some(JobId(1900)) Some(JobId(1603))
3472 Downstairs last five completed: 1603 1602 1601 1600 1599
3473 Upstairs last five completed: 1904 1903 1902 1901 1900
3474 CLIENT: Up:0 ds:301 act:2
34752025-02-07T20:07:58.998ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
3476 [8830] got start action
3477 [8830] I am going to restart
3478 Starting downstairs at port 8830
3479 Make output file at "/tmp/dsc/downstairs-8830.txt"
3480 [8830][2] reports Starting
3481 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1212)
3482 [8830][2] reports Running
34832025-02-07T20:07:59.999ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
34842025-02-07T20:08:01.001ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
34852025-02-07T20:08:02.003ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
3486 ----------------------------------------------------------------
3487 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3488 States: Active Active Offline
3489 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3490 1604 Acked Read 9 Done Done Sent false
3491 1605 Acked Write 10 Done Done Sent false
3492 1606 Acked Read 8 Done Done Sent false
3493 1607 Acked Write 8 Done Done Sent false
3494 1608 Acked Read 4 Done Done Sent false
3495 1609 Acked Read 1 Done Done Sent false
3496 1610 Acked Write 8 Done Done Sent false
3497 1611 Acked Write 7 Done Done Sent false
3498 1612 Acked Write 9 Done Done Sent false
3499 1613 Acked Write 2 Done Done Sent false
3500 1614 Acked Write 9 Done Done Sent false
3501 1615 Acked Flush 0 Done Done Sent false
3502 1616 Acked Write 6 Done Done Sent false
3503 1617 Acked Read 8 Done Done Sent false
3504 1618 Acked Read 7 Done Done Sent false
3505 1619 Acked Read 7 Done Done Sent false
3506 1620 Acked Flush 0 Done Done Sent false
3507 1621 Acked Read 9 Done Done Sent false
3508 1622 Acked Read 8 Done Done Sent false
3509 1623 Acked Write 1 Done Done Sent false
3510 1624 Acked Write 3 Done Done Sent false
3511 1625 Acked Read 3 Done Done Sent false
3512 1626 Acked Read 5 Done Done Sent false
3513 1627 Acked Flush 0 Done Done Sent false
3514 1628 Acked Read 1 Done Done Sent false
3515 1629 Acked Read 2 Done Done Sent false
3516 1630 Acked Flush 0 Done Done Sent false
3517 1631 Acked Read 5 Done Done Sent false
3518 1632 Acked Write 2 Done Done Sent false
3519 1633 Acked Write 10 Done Done Sent false
3520 1634 Acked Flush 0 Done Done Sent false
3521 1635 Acked Read 2 Done Done Sent false
3522 1636 Acked Write 3 Done Done Sent false
3523 1637 Acked Flush 0 Done Done Sent false
3524 1638 Acked Read 7 Done Done Sent false
3525 1639 Acked Read 2 Done Done Sent false
3526 1640 Acked Read 10 Done Done Sent false
3527 1641 Acked Read 1 Done Done Sent false
3528 1642 Acked Write 1 Done Done Sent false
3529 1643 Acked Flush 0 Done Done Sent false
3530 1644 Acked Write 4 Done Done Sent false
3531 1645 Acked Write 7 Done Done Sent false
3532 1646 Acked Write 1 Done Done Sent false
3533 1647 Acked Write 6 Done Done Sent false
3534 1648 Acked Read 2 Done Done Sent false
3535 1649 Acked Read 5 Done Done Sent false
3536 1650 Acked Write 10 Done Done Sent false
3537 1651 Acked Write 5 Done Done Sent false
3538 1652 Acked Write 8 Done Done Sent false
3539 1653 Acked Write 5 Done Done Sent false
3540 1654 Acked Write 1 Done Done Sent false
3541 1655 Acked Write 8 Done Done Sent false
3542 1656 Acked Write 10 Done Done Sent false
3543 1657 Acked Write 9 Done Done Sent false
3544 1658 Acked Write 1 Done Done Sent false
3545 1659 Acked Read 9 Done Done Sent false
3546 1660 Acked Write 9 Done Done Sent false
3547 1661 Acked Write 5 Done Done Sent false
3548 1662 Acked Write 1 Done Done Sent false
3549 1663 Acked Write 5 Done Done Sent false
3550 1664 Acked Read 7 Done Done Sent false
3551 1665 Acked Write 5 Done Done Sent false
3552 1666 Acked Read 9 Done Done Sent false
3553 1667 Acked Read 4 Done Done Sent false
3554 1668 Acked Flush 0 Done Done Sent false
3555 1669 Acked Write 4 Done Done Sent false
3556 1670 Acked Write 7 Done Done Sent false
3557 1671 Acked Read 4 Done Done Sent false
3558 1672 Acked Write 1 Done Done Sent false
3559 1673 Acked Write 1 Done Done Sent false
3560 1674 Acked Read 9 Done Done Sent false
3561 1675 Acked Read 3 Done Done Sent false
3562 1676 Acked Read 9 Done Done Sent false
3563 1677 Acked Write 10 Done Done Sent false
3564 1678 Acked Write 2 Done Done Sent false
3565 1679 Acked Write 8 Done Done Sent false
3566 1680 Acked Write 7 Done Done Sent false
3567 1681 Acked Flush 0 Done Done Sent false
3568 1682 Acked Flush 0 Done Done Sent false
3569 1683 Acked Write 1 Done Done Sent false
3570 1684 Acked Read 3 Done Done Sent false
3571 1685 Acked Read 7 Done Done Sent false
3572 1686 Acked Write 3 Done Done Sent false
3573 1687 Acked Read 1 Done Done Sent false
3574 1688 Acked Read 5 Done Done Sent false
3575 1689 Acked Flush 0 Done Done Sent false
3576 1690 Acked Write 10 Done Done Sent false
3577 1691 Acked Read 7 Done Done Sent false
3578 1692 Acked Read 8 Done Done Sent false
3579 1693 Acked Read 3 Done Done Sent false
3580 1694 Acked Flush 0 Done Done Sent false
3581 1695 Acked Write 8 Done Done Sent false
3582 1696 Acked Read 5 Done Done Sent false
3583 1697 Acked Write 2 Done Done Sent false
3584 1698 Acked Flush 0 Done Done Sent false
3585 1699 Acked Flush 0 Done Done Sent false
3586 1700 Acked Write 2 Done Done Sent false
3587 1701 Acked Read 4 Done Done Sent false
3588 1702 Acked Write 10 Done Done Sent false
3589 1703 Acked Read 3 Done Done Sent false
3590 1704 Acked Read 4 Done Done Sent false
3591 1705 Acked Read 5 Done Done Sent false
3592 1706 Acked Read 5 Done Done Sent false
3593 1707 Acked Write 6 Done Done Sent false
3594 1708 Acked Read 4 Done Done Sent false
3595 1709 Acked Write 1 Done Done Sent false
3596 1710 Acked Read 9 Done Done Sent false
3597 1711 Acked Write 3 Done Done Sent false
3598 1712 Acked Read 9 Done Done Sent false
3599 1713 Acked Read 3 Done Done Sent false
3600 1714 Acked Read 6 Done Done Sent false
3601 1715 Acked Read 8 Done Done Sent false
3602 1716 Acked Flush 0 Done Done Sent false
3603 1717 Acked Read 3 Done Done Sent false
3604 1718 Acked Read 4 Done Done Sent false
3605 1719 Acked Write 6 Done Done Sent false
3606 1720 Acked Read 6 Done Done Sent false
3607 1721 Acked Read 4 Done Done Sent false
3608 1722 Acked Read 6 Done Done Sent false
3609 1723 Acked Write 8 Done Done Sent false
3610 1724 Acked Write 6 Done Done Sent false
3611 1725 Acked Read 10 Done Done Sent false
3612 1726 Acked Read 6 Done Done Sent false
3613 1727 Acked Flush 0 Done Done Sent false
3614 1728 Acked Write 10 Done Done Sent false
3615 1729 Acked Read 1 Done Done Sent false
3616 1730 Acked Read 2 Done Done Sent false
3617 1731 Acked Read 5 Done Done Sent false
3618 1732 Acked Read 7 Done Done Sent false
3619 1733 Acked Read 3 Done Done Sent false
3620 1734 Acked Flush 0 Done Done Sent false
3621 1735 Acked Read 9 Done Done Sent false
3622 1736 Acked Read 1 Done Done Sent false
3623 1737 Acked Write 7 Done Done Sent false
3624 1738 Acked Write 8 Done Done Sent false
3625 1739 Acked Flush 0 Done Done Sent false
3626 1740 Acked Write 9 Done Done Sent false
3627 1741 Acked Write 1 Done Done Sent false
3628 1742 Acked Write 5 Done Done Sent false
3629 1743 Acked Write 10 Done Done Sent false
3630 1744 Acked Read 5 Done Done Sent false
3631 1745 Acked Write 9 Done Done Sent false
3632 1746 Acked Read 3 Done Done Sent false
3633 1747 Acked Read 9 Done Done Sent false
3634 1748 Acked Read 5 Done Done Sent false
3635 1749 Acked Read 1 Done Done Sent false
3636 1750 Acked Read 6 Done Done Sent false
3637 1751 Acked Write 2 Done Done Sent false
3638 1752 Acked Write 7 Done Done Sent false
3639 1753 Acked Write 5 Done Done Sent false
3640 1754 Acked Write 9 Done Done Sent false
3641 1755 Acked Write 3 Done Done Sent false
3642 1756 Acked Flush 0 Done Done Sent false
3643 1757 Acked Read 10 Done Done Sent false
3644 1758 Acked Read 1 Done Done Sent false
3645 1759 Acked Write 10 Done Done Sent false
3646 1760 Acked Write 6 Done Done Sent false
3647 1761 Acked Read 4 Done Done Sent false
3648 1762 Acked Read 7 Done Done Sent false
3649 1763 Acked Read 1 Done Done Sent false
3650 1764 Acked Read 1 Done Done Sent false
3651 1765 Acked Read 7 Done Done Sent false
3652 1766 Acked Read 6 Done Done Sent false
3653 1767 Acked Read 1 Done Done Sent false
3654 1768 Acked Flush 0 Done Done Sent false
3655 1769 Acked Read 1 Done Done Sent false
3656 1770 Acked Read 5 Done Done Sent false
3657 1771 Acked Read 2 Done Done Sent false
3658 1772 Acked Read 1 Done Done Sent false
3659 1773 Acked Read 4 Done Done Sent false
3660 1774 Acked Read 10 Done Done Sent false
3661 1775 Acked Read 10 Done Done Sent false
3662 1776 Acked Read 8 Done Done Sent false
3663 1777 Acked Read 3 Done Done Sent false
3664 1778 Acked Write 8 Done Done Sent false
3665 1779 Acked Write 1 Done Done Sent false
3666 1780 Acked Flush 0 Done Done Sent false
3667 1781 Acked Flush 0 Done Done Sent false
3668 1782 Acked Read 7 Done Done Sent false
3669 1783 Acked Read 8 Done Done Sent false
3670 1784 Acked Flush 0 Done Done Sent false
3671 1785 Acked Flush 0 Done Done Sent false
3672 1786 Acked Read 5 Done Done Sent false
3673 1787 Acked Read 3 Done Done Sent false
3674 1788 Acked Write 10 Done Done Sent false
3675 1789 Acked Read 6 Done Done Sent false
3676 1790 Acked Read 10 Done Done Sent false
3677 1791 Acked Write 8 Done Done Sent false
3678 1792 Acked Read 7 Done Done Sent false
3679 1793 Acked Write 8 Done Done Sent false
3680 1794 Acked Write 10 Done Done Sent false
3681 1795 Acked Read 2 Done Done Sent false
3682 1796 Acked Flush 0 Done Done Sent false
3683 1797 Acked Write 4 Done Done Sent false
3684 1798 Acked Flush 0 Done Done Sent false
3685 1799 Acked Read 8 Done Done Sent false
3686 1800 Acked Read 6 Done Done Sent false
3687 1801 Acked Write 3 Done Done Sent false
3688 1802 Acked Write 10 Done Done Sent false
3689 1803 Acked Read 2 Done Done Sent false
3690 1804 Acked Write 9 Done Done Sent false
3691 1805 Acked Flush 0 Done Done Sent false
3692 1806 Acked Write 7 Done Done Sent false
3693 1807 Acked Read 9 Done Done Sent false
3694 1808 Acked Read 3 Done Done Sent false
3695 1809 Acked Write 8 Done Done Sent false
3696 1810 Acked Write 8 Done Done Sent false
3697 1811 Acked Flush 0 Done Done Sent false
3698 1812 Acked Read 3 Done Done Sent false
3699 1813 Acked Read 6 Done Done Sent false
3700 1814 Acked Read 5 Done Done Sent false
3701 1815 Acked Write 2 Done Done Sent false
3702 1816 Acked Flush 0 Done Done Sent false
3703 1817 Acked Read 2 Done Done Sent false
3704 1818 Acked Write 9 Done Done Sent false
3705 1819 Acked Read 10 Done Done Sent false
3706 1820 Acked Write 10 Done Done Sent false
3707 1821 Acked Read 3 Done Done Sent false
3708 1822 Acked Read 4 Done Done Sent false
3709 1823 Acked Write 10 Done Done Sent false
3710 1824 Acked Read 10 Done Done Sent false
3711 1825 Acked Write 4 Done Done Sent false
3712 1826 Acked Write 10 Done Done Sent false
3713 1827 Acked Read 3 Done Done Sent false
3714 1828 Acked Write 8 Done Done Sent false
3715 1829 Acked Read 4 Done Done Sent false
3716 1830 Acked Read 4 Done Done Sent false
3717 1831 Acked Write 7 Done Done Sent false
3718 1832 Acked Write 8 Done Done Sent false
3719 1833 Acked Write 8 Done Done Sent false
3720 1834 Acked Read 1 Done Done Sent false
3721 1835 Acked Write 2 Done Done Sent false
3722 1836 Acked Write 10 Done Done Sent false
3723 1837 Acked Read 10 Done Done Sent false
3724 1838 Acked Read 7 Done Done Sent false
3725 1839 Acked Write 9 Done Done Sent false
3726 1840 Acked Read 5 Done Done Sent false
3727 1841 Acked Write 5 Done Done Sent false
3728 1842 Acked Write 1 Done Done Sent false
3729 1843 Acked Write 6 Done Done Sent false
3730 1844 Acked Read 7 Done Done Sent false
3731 1845 Acked Read 7 Done Done Sent false
3732 1846 Acked Write 9 Done Done Sent false
3733 1847 Acked Read 1 Done Done Sent false
3734 1848 Acked Flush 0 Done Done Sent false
3735 1849 Acked Read 9 Done Done Sent false
3736 1850 Acked Write 2 Done Done Sent false
3737 1851 Acked Read 6 Done Done Sent false
3738 1852 Acked Read 4 Done Done Sent false
3739 1853 Acked Read 10 Done Done Sent false
3740 1854 Acked Write 6 Done Done Sent false
3741 1855 Acked Read 1 Done Done Sent false
3742 1856 Acked Flush 0 Done Done Sent false
3743 1857 Acked Write 4 Done Done Sent false
3744 1858 Acked Write 3 Done Done Sent false
3745 1859 Acked Write 10 Done Done Sent false
3746 1860 Acked Write 1 Done Done Sent false
3747 1861 Acked Read 10 Done Done Sent false
3748 1862 Acked Read 3 Done Done Sent false
3749 1863 Acked Read 8 Done Done Sent false
3750 1864 Acked Write 4 Done Done Sent false
3751 1865 Acked Read 8 Done Done Sent false
3752 1866 Acked Write 5 Done Done Sent false
3753 1867 Acked Write 3 Done Done Sent false
3754 1868 Acked Read 5 Done Done Sent false
3755 1869 Acked Read 10 Done Done Sent false
3756 1870 Acked Write 5 Done Done Sent false
3757 1871 Acked Flush 0 Done Done Sent false
3758 1872 Acked Write 6 Done Done Sent false
3759 1873 Acked Flush 0 Done Done Sent false
3760 1874 Acked Read 2 Done Done Sent false
3761 1875 Acked Read 10 Done Done Sent false
3762 1876 Acked Read 8 Done Done Sent false
3763 1877 Acked Flush 0 Done Done Sent false
3764 1878 Acked Write 7 Done Done Sent false
3765 1879 Acked Read 8 Done Done Sent false
3766 1880 Acked Read 8 Done Done Sent false
3767 1881 Acked Write 5 Done Done Sent false
3768 1882 Acked Read 9 Done Done Sent false
3769 1883 Acked Write 6 Done Done Sent false
3770 1884 Acked Read 10 Done Done Sent false
3771 1885 Acked Write 4 Done Done Sent false
3772 1886 Acked Read 1 Done Done Sent false
3773 1887 Acked Write 1 Done Done Sent false
3774 1888 Acked Read 7 Done Done Sent false
3775 1889 Acked Write 9 Done Done Sent false
3776 1890 Acked Write 2 Done Done Sent false
3777 1891 Acked Read 8 Done Done Sent false
3778 1892 Acked Write 4 Done Done Sent false
3779 1893 Acked Write 2 Done Done Sent false
3780 1894 Acked Write 1 Done Done Sent false
3781 1895 Acked Flush 0 Done Done Sent false
3782 1896 Acked Read 4 Done Done Sent false
3783 1897 Acked Read 10 Done Done Sent false
3784 1898 Acked Read 3 Done Done Sent false
3785 1899 Acked Write 3 Done Done Sent false
3786 1900 Acked Flush 0 Done Done Sent false
3787 1901 Acked Write 8 Done Done Sent false
3788 1902 Acked Read 3 Done Done Sent false
3789 1903 Acked Read 1 Done Done Sent false
3790 1904 Acked Read 3 Done Done Sent false
3791 STATES DS:0 DS:1 DS:2 TOTAL
3792 Sent 0 0 301 301
3793 Done 301 301 0 602
3794 Skipped 0 0 0 0
3795 Error 0 0 0 0
3796 Last Flush: Some(JobId(1900)) Some(JobId(1900)) Some(JobId(1603))
3797 Downstairs last five completed: 1603 1602 1601 1600 1599
3798 Upstairs last five completed: 1904 1903 1902 1901 1900
3799 CLIENT: Up:0 ds:301 act:2
38002025-02-07T20:08:03.005ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
38012025-02-07T20:08:03.980ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
38022025-02-07T20:08:03.980ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52030) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
38032025-02-07T20:08:03.981ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
38042025-02-07T20:08:03.982ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
38052025-02-07T20:08:03.982ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
38062025-02-07T20:08:03.982ZINFOcrucible: send last flush ID to this DS: Some(JobId(1603)) = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
38072025-02-07T20:08:03.982ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
38082025-02-07T20:08:03.982ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1603)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
38092025-02-07T20:08:03.982ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1603)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
3810 ----------------------------------------------------------------
3811 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3812 Downstairs last five completed: 1905 1904 1903 1902 1901
3813 Upstairs last five completed: 1905 1904 1903 1902 1901
3814 CLIENT: Up:0 ds:0 act:3
3815 Replay: All jobs finished
3816 Main task has work to do, go find it
3817 got dsc Stop(0)
3818 stop 0
3819 Feb 07 20:08:06.225 INFO request completed, latency_us: 328, response_code: 204, [8810] Got stop action so:false kr:false
3820 uri: /stop/cid/0, method: POST, req_id: 84759e2b-3c53-45b3-a54b-d195ff822425, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
38212025-02-07T20:08:06.226ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
38222025-02-07T20:08:06.226ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
38232025-02-07T20:08:06.226ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
38242025-02-07T20:08:06.226ZINFOcrucible: client task is exiting = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
3825 Feb 07 20:08:06.226 INFO request completed, latency_us: 166, response_code: 200, uri: /state/cid/0, method: GET, req_id: 26bcb0af-c333-44cf-995c-fba9a79e34c4, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
3826 [8810] Exited with: ExitStatus(unix_wait_status(9))
3827 [8810][0] reports Exit
3828 Feb 07 20:08:10.228 INFO request completed, latency_us: 243, response_code: 200, uri: /state/cid/0, method: GET, req_id: 908fde99-59c2-4279-91b4-cc0a4cdb9543, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
3829 001/300 Flush
3830 002/300 Write block 1001 len 32768 data: 5 5 4 4 4 4 3 2
3831 003/300 Write block 2512 len 4096 data: 2
3832 004/300 Read block 3639 len 40960
3833 005/300 Write block 2537 len 28672 data: 2 2 2 2 2 2 2
3834 006/300 Write block 3600 len 8192 data: 3 3
3835 007/300 Write block 3494 len 28672 data: 2 2 2 2 2 2 2
3836 008/300 Read block 3932 len 32768
3837 009/300 Write block 882 len 20480 data: 2 2 2 2 3
3838 010/300 Read block 2963 len 36864
3839 011/300 Flush
3840 012/300 Read block 4677 len 12288
3841 013/300 Flush
3842 014/300 Read block 118 len 12288
3843 015/300 Read block 4203 len 16384
3844 016/300 Read block 1066 len 8192
3845 017/300 Flush
3846 018/300 Write block 4454 len 32768 data: 2 2 2 2 2 2 2 2
3847 019/300 Read block 4197 len 20480
3848 020/300 Read block 3913 len 28672
3849 021/300 Write block 3412 len 20480 data: 3 3 4 4 4
3850 022/300 Read block 705 len 24576
3851 023/300 Read block 4061 len 28672
3852 024/300 Flush
3853 025/300 Read block 3960 len 20480
3854 026/300 Read block 720 len 12288
3855 027/300 Write block 4620 len 36864 data: 3 2 3 3 3 2 2 2 2
3856 028/300 Write block 4416 len 8192 data: 3 3
3857 029/300 Flush
3858 030/300 Write block 4650 len 36864 data: 3 3 2 2 2 2 2 2 2
3859 031/300 Write block 846 len 28672 data: 2 2 2 2 2 2 2
3860 032/300 Write block 1845 len 20480 data: 2 2 2 2 2
3861 033/300 Read block 1141 len 4096
3862 034/300 Read block 1467 len 32768
3863 035/300 Read block 2147 len 20480
3864 036/300 Flush
3865 037/300 Write block 1781 len 4096 data: 2
3866 038/300 Write block 1998 len 8192 data: 2 2
3867 039/300 Write block 4052 len 20480 data: 2 2 2 2 2
3868 040/300 Write block 719 len 28672 data: 2 2 2 2 2 2 3
3869 041/300 Read block 185 len 4096
3870 042/300 Read block 676 len 28672
3871 043/300 Read block 4367 len 4096
3872 044/300 Read block 4309 len 28672
3873 045/300 Write block 1709 len 16384 data: 2 2 2 2
3874 046/300 Flush
3875 047/300 Write block 2108 len 32768 data: 2 2 2 2 2 2 2 2
3876 048/300 Read block 242 len 40960
3877 049/300 Read block 4951 len 12288
3878 050/300 Read block 396 len 16384
3879 051/300 Read block 4564 len 12288
3880 052/300 Read block 2110 len 40960
3881 053/300 Write block 4049 len 4096 data: 2
3882 054/300 Write block 2968 len 28672 data: 2 2 2 2 2 2 2
3883 055/300 Read block 627 len 4096
3884 056/300 Write block 4458 len 40960 data: 3 3 3 3 2 2 2 2 2 2
3885 057/300 Read block 3058 len 8192
3886 058/300 Write block 1330 len 36864 data: 2 2 2 2 2 2 3 3 3
3887 059/300 Write block 1229 len 36864 data: 2 3 3 2 2 2 3 2 2
3888 060/300 Write block 910 len 12288 data: 2 2 2
3889 061/300 Read block 1974 len 8192
3890 062/300 Read block 1902 len 28672
3891 063/300 Read block 454 len 24576
3892 064/300 Flush
3893 065/300 Flush
3894 066/300 Read block 2499 len 28672
3895 067/300 Write block 3433 len 36864 data: 3 2 2 2 2 2 2 2 2
3896 068/300 Flush
3897 069/300 Write block 1171 len 32768 data: 2 2 2 2 2 2 2 2
3898 070/300 Flush
3899 071/300 Read block 3027 len 40960
3900 072/300 Read block 1892 len 28672
3901 073/300 Read block 4192 len 16384
3902 074/300 Read block 104 len 16384
3903 075/300 Read block 4165 len 28672
3904 076/300 Read block 1954 len 24576
3905 077/300 Read block 4423 len 36864
3906 078/300 Write block 229 len 32768 data: 2 2 2 2 2 2 2 2
3907 079/300 Read block 1504 len 4096
3908 080/300 Flush
3909 081/300 Write block 1569 len 16384 data: 2 2 2 2
3910 082/300 Flush
3911 083/300 Write block 4606 len 28672 data: 3 3 3 3 3 3 2
3912 084/300 Write block 1489 len 28672 data: 2 2 2 2 2 2 2
3913 085/300 Write block 4475 len 16384 data: 3 3 3 3
3914 086/300 Read block 1084 len 32768
3915 087/300 Flush
3916 088/300 Read block 417 len 36864
3917 089/300 Write block 1631 len 36864 data: 2 2 2 2 2 2 2 2 2
3918 090/300 Flush
3919 091/300 Read block 2933 len 4096
3920 092/300 Read block 4977 len 20480
3921 093/300 Write block 2436 len 16384 data: 2 2 2 2
3922 094/300 Read block 3657 len 36864
3923 095/300 Write block 3855 len 4096 data: 2
3924 096/300 Read block 2757 len 16384
3925 097/300 Write block 700 len 32768 data: 2 2 2 2 2 2 3 3
3926 098/300 Write block 3754 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3927 099/300 Read block 3750 len 8192
3928 100/300 Read block 3980 len 4096
3929 101/300 Write block 754 len 12288 data: 4 3 2
3930 102/300 Read block 3194 len 12288
3931 103/300 Write block 722 len 20480 data: 3 3 3 4 2
3932 104/300 Read block 2550 len 28672
3933 105/300 Read block 2822 len 20480
3934 106/300 Write block 2358 len 16384 data: 2 2 2 2
3935 107/300 Write block 3433 len 28672 data: 4 3 3 3 3 3 3
3936 108/300 Write block 1622 len 12288 data: 2 2 2
3937 109/300 Write block 2601 len 28672 data: 3 2 2 2 2 2 2
3938 110/300 Read block 3949 len 20480
3939 111/300 Write block 3242 len 4096 data: 2
3940 112/300 Flush
3941 113/300 Write block 4848 len 36864 data: 2 2 2 2 2 2 2 2 2
3942 114/300 Write block 1311 len 16384 data: 2 2 2 2
3943 115/300 Read block 2233 len 32768
3944 116/300 Read block 682 len 28672
3945 117/300 Read block 1539 len 40960
3946 118/300 Write block 1630 len 16384 data: 2 3 3 3
3947 119/300 Flush
3948 120/300 Write block 1275 len 40960 data: 3 3 3 3 3 3 2 3 3 3
3949 121/300 Write block 1931 len 32768 data: 2 2 2 2 2 2 2 2
3950 122/300 Write block 2566 len 32768 data: 4 2 2 2 2 2 2 2
3951 123/300 Read block 1726 len 32768
3952 124/300 Flush
3953 125/300 Write block 1154 len 28672 data: 3 3 3 3 3 3 3
3954 126/300 Flush
3955 127/300 Read block 1759 len 40960
3956 128/300 Write block 4620 len 4096 data: 4
3957 129/300 Write block 2017 len 12288 data: 2 2 3
3958 130/300 Write block 4460 len 28672 data: 4 4 3 3 3 3 3
3959 131/300 Read block 2632 len 4096
3960 132/300 Read block 535 len 40960
3961 133/300 Read block 519 len 32768
3962 134/300 Read block 724 len 4096
3963 135/300 Read block 1558 len 8192
3964 136/300 Flush
3965 137/300 Flush
3966 138/300 Read block 1703 len 8192
3967 139/300 Read block 2323 len 20480
3968 140/300 Read block 386 len 28672
3969 141/300 Read block 3639 len 12288
3970 142/300 Read block 453 len 36864
3971 143/300 Read block 1166 len 12288
3972 144/300 Read block 871 len 32768
3973 145/300 Read block 1595 len 24576
3974 146/300 Read block 1786 len 36864
3975 147/300 Write block 3851 len 32768 data: 2 2 2 2 3 2 2 2
3976 148/300 Read block 998 len 4096
3977 149/300 Read block 2216 len 20480
3978 150/300 Read block 3743 len 24576
3979 151/300 Write block 2735 len 8192 data: 2 2
3980 152/300 Write block 3251 len 40960 data: 4 4 4 4 4 3 3 2 2 2
3981 153/300 Flush
3982 154/300 Flush
3983 155/300 Read block 4103 len 20480
3984 156/300 Read block 788 len 20480
3985 157/300 Write block 3269 len 28672 data: 2 2 2 2 2 3 3
3986 158/300 Read block 2612 len 20480
3987 159/300 Write block 3228 len 32768 data: 2 2 2 2 2 2 2 2
3988 160/300 Write block 4836 len 4096 data: 4
3989 161/300 Read block 4621 len 8192
3990 162/300 Write block 1690 len 28672 data: 3 3 2 2 2 2 2
3991 163/300 Write block 560 len 20480 data: 2 2 2 2 2
3992 164/300 Read block 148 len 4096
3993 165/300 Read block 965 len 32768
3994 166/300 Write block 3863 len 24576 data: 2 2 2 2 2 2
3995 167/300 Read block 3960 len 36864
3996 168/300 Write block 3132 len 40960 data: 3 3 2 2 2 2 2 2 2 2
3997 169/300 Write block 1977 len 8192 data: 2 2
3998 170/300 Read block 73 len 16384
3999 171/300 Write block 670 len 36864 data: 2 2 2 2 2 2 2 3 3
4000 172/300 Read block 3017 len 32768
4001 173/300 Read block 3459 len 16384
4002 174/300 Write block 1713 len 24576 data: 2 2 2 2 2 2
4003 175/300 Write block 1396 len 12288 data: 2 2 2
4004 176/300 Write block 2704 len 20480 data: 2 2 2 3 3
4005 177/300 Write block 2170 len 28672 data: 2 2 2 2 2 2 2
4006 178/300 Write block 3259 len 36864 data: 3 3 2 2 2 2 2 2 2
4007 179/300 Read block 375 len 32768
4008 180/300 Read block 1762 len 20480
4009 181/300 Write block 4375 len 4096 data: 2
4010 182/300 Read block 3926 len 36864
4011 183/300 Write block 2034 len 8192 data: 2 2
4012 184/300 Read block 4285 len 8192
4013 185/300 Write block 3065 len 4096 data: 2
4014 186/300 Read block 4752 len 16384
4015 187/300 Write block 2704 len 20480 data: 3 3 3 4 4
4016 188/300 Flush
4017 189/300 Write block 4047 len 12288 data: 2 2 3
4018 190/300 Write block 3897 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4019 191/300 Write block 4662 len 36864 data: 2 2 2 2 2 2 3 3 3
4020 192/300 Read block 1974 len 32768
4021 193/300 Read block 4688 len 4096
4022 194/300 Read block 3322 len 24576
4023 195/300 Read block 478 len 32768
4024 196/300 Read block 4366 len 20480
4025 197/300 Read block 1039 len 40960
4026 198/300 Write block 3875 len 12288 data: 2 2 3
4027 199/300 Write block 3167 len 28672 data: 2 3 3 3 3 3 3
4028 200/300 Write block 4390 len 8192 data: 2 2
4029 201/300 Write block 4219 len 24576 data: 2 2 2 3 3 2
4030 202/300 Read block 4865 len 28672
4031 203/300 Read block 3605 len 32768
4032 204/300 Flush
4033 205/300 Write block 617 len 40960 data: 3 2 2 2 2 3 3 3 2 2
4034 206/300 Read block 3717 len 4096
4035 207/300 Read block 4950 len 24576
4036 208/300 Write block 1037 len 4096 data: 3
4037 209/300 Read block 3223 len 32768
4038 210/300 Read block 468 len 8192
4039 211/300 Write block 4071 len 36864 data: 3 2 2 2 2 2 2 2 2
4040 212/300 Read block 2046 len 4096
4041 213/300 Read block 4663 len 20480
4042 214/300 Write block 1764 len 20480 data: 2 2 2 2 2
4043 215/300 Read block 4618 len 28672
4044 216/300 Write block 4666 len 4096 data: 3
4045 217/300 Write block 2341 len 24576 data: 4 4 3 3 3 3
4046 218/300 Write block 721 len 8192 data: 3 4
4047 219/300 Read block 4696 len 36864
4048 220/300 Read block 2214 len 12288
4049 221/300 Read block 4872 len 20480
4050 222/300 Read block 1825 len 20480
4051 223/300 Read block 3341 len 32768
4052 224/300 Read block 2847 len 12288
4053 225/300 Flush
4054 226/300 Read block 523 len 32768
4055 227/300 Read block 299 len 4096
4056 228/300 Write block 1296 len 40960 data: 3 3 3 3 3 3 3 3 2 2
4057 229/300 Read block 3204 len 40960
4058 230/300 Read block 2680 len 24576
4059 231/300 Write block 3716 len 8192 data: 2 2
4060 232/300 Read block 517 len 28672
4061 233/300 Write block 2267 len 24576 data: 3 2 2 2 2 2
4062 234/300 Read block 977 len 36864
4063 235/300 Write block 3225 len 8192 data: 2 2
4064 236/300 Read block 2201 len 24576
4065 237/300 Write block 3333 len 36864 data: 3 3 3 3 3 3 3 3 3
4066 238/300 Write block 3365 len 16384 data: 3 3 2 2
4067 239/300 Write block 3156 len 24576 data: 2 2 2 2 2 2
4068 240/300 Write block 4987 len 28672 data: 2 2 2 2 2 2 2
4069 241/300 Write block 2200 len 16384 data: 2 2 2 2
4070 242/300 Read block 596 len 8192
4071 243/300 Read block 618 len 12288
4072 244/300 Flush
4073 245/300 Read block 3087 len 16384
4074 246/300 Read block 2951 len 40960
4075 247/300 Read block 353 len 16384
4076 248/300 Read block 1139 len 20480
4077 249/300 Read block 4323 len 40960
4078 250/300 Write block 976 len 16384 data: 2 2 2 2
4079 251/300 Write block 4042 len 16384 data: 3 3 3 3
4080 252/300 Read block 1929 len 36864
4081 253/300 Write block 1069 len 16384 data: 2 2 2 2
4082 254/300 Write block 4471 len 40960 data: 3 3 2 2 4 4 4 4 3 3
4083 255/300 Read block 3037 len 36864
4084 256/300 Read block 940 len 36864
4085 257/300 Read block 2661 len 16384
4086 258/300 Write block 3701 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4087 259/300 Read block 4484 len 40960
4088 260/300 Write block 3423 len 40960 data: 2 3 3 3 3 3 3 3 3 3
4089 261/300 Read block 4726 len 12288
4090 262/300 Write block 2760 len 16384 data: 3 2 2 2
4091 263/300 Write block 3365 len 12288 data: 4 4 3
4092 264/300 Read block 35 len 12288
4093 265/300 Read block 2670 len 36864
4094 266/300 Write block 3473 len 28672 data: 2 3 3 3 3 3 2
4095 267/300 Flush
4096 268/300 Read block 4817 len 28672
4097 269/300 Write block 2395 len 8192 data: 2 2
4098 270/300 Write block 4729 len 24576 data: 3 3 3 4 4 4
4099 271/300 Write block 4848 len 40960 data: 3 3 3 3 3 3 3 3 3 3
4100 272/300 Read block 4323 len 32768
4101 273/300 Read block 208 len 40960
4102 274/300 Read block 2356 len 20480
4103 275/300 Write block 1658 len 12288 data: 4 3 2
4104 276/300 Read block 3857 len 40960
4105 277/300 Read block 2089 len 24576
4106 278/300 Read block 4814 len 32768
4107 279/300 Read block 2746 len 40960
4108 280/300 Read block 4930 len 20480
4109 281/300 Flush
4110 282/300 Write block 1077 len 24576 data: 2 2 2 2 2 2
4111 283/300 Read block 2666 len 20480
4112 284/300 Write block 1678 len 28672 data: 2 2 2 2 2 2 3
4113 285/300 Write block 1618 len 16384 data: 2 2 2 2
4114 286/300 Write block 1755 len 28672 data: 2 2 2 2 2 2 3
4115 287/300 Write block 2377 len 8192 data: 3 3
4116 288/300 Read block 3598 len 28672
4117 289/300 Write block 3995 len 24576 data: 3 3 2 2 2 2
4118 290/300 Write block 1056 len 4096 data: 3
4119 291/300 Write block 561 len 16384 data: 3 3 3 3
4120 292/300 Read block 4568 len 12288
4121 293/300 Read block 3680 len 40960
4122 294/300 Read block 4658 len 40960
4123 295/300 Flush
4124 296/300 Read block 2775 len 28672
4125 297/300 Write block 4902 len 4096 data: 3
4126 298/300 Read block 4062 len 16384
4127 299/300 Read block 1724 len 12288
4128 300/300 Read block 544 len 4096
4129 301/300 Read block 2533 len 32768
4130 Main task has work to do, go find it
4131 got dsc Start(0)
4132 start 0
4133 Feb 07 20:08:10.473 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/0, method: POST, req_id: 4e93dcf9-6eb7-4b23-97a8-4f5e7c82bda7, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
4134 Replay: started 0, returned:Ok(())
4135 ----------------------------------------------------------------
4136 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4137 States: Offline Active Active
4138 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4139 1906 Acked Flush 0 Sent Done Done false
4140 1907 Acked Write 8 Sent Done Done false
4141 1908 Acked Write 1 Sent Done Done false
4142 1909 Acked Read 10 Sent Done Done false
4143 1910 Acked Write 7 Sent Done Done false
4144 1911 Acked Write 2 Sent Done Done false
4145 1912 Acked Write 7 Sent Done Done false
4146 1913 Acked Read 8 Sent Done Done false
4147 1914 Acked Write 5 Sent Done Done false
4148 1915 Acked Read 9 Sent Done Done false
4149 1916 Acked Flush 0 Sent Done Done false
4150 1917 Acked Read 3 Sent Done Done false
4151 1918 Acked Flush 0 Sent Done Done false
4152 1919 Acked Read 3 Sent Done Done false
4153 1920 Acked Read 4 Sent Done Done false
4154 1921 Acked Read 2 Sent Done Done false
4155 1922 Acked Flush 0 Sent Done Done false
4156 1923 Acked Write 8 Sent Done Done false
4157 1924 Acked Read 5 Sent Done Done false
4158 1925 Acked Read 7 Sent Done Done false
4159 1926 Acked Write 5 Sent Done Done false
4160 1927 Acked Read 6 Sent Done Done false
4161 1928 Acked Read 7 Sent Done Done false
4162 1929 Acked Flush 0 Sent Done Done false
4163 1930 Acked Read 5 Sent Done Done false
4164 1931 Acked Read 3 Sent Done Done false
4165 1932 Acked Write 9 Sent Done Done false
4166 1933 Acked Write 2 Sent Done Done false
4167 1934 Acked Flush 0 Sent Done Done false
4168 1935 Acked Write 9 Sent Done Done false
4169 1936 Acked Write 7 Sent Done Done false
4170 1937 Acked Write 5 Sent Done Done false
4171 1938 Acked Read 1 Sent Done Done false
4172 1939 Acked Read 8 Sent Done Done false
4173 1940 Acked Read 5 Sent Done Done false
4174 1941 Acked Flush 0 Sent Done Done false
4175 1942 Acked Write 1 Sent Done Done false
4176 1943 Acked Write 2 Sent Done Done false
4177 1944 Acked Write 5 Sent Done Done false
4178 1945 Acked Write 7 Sent Done Done false
4179 1946 Acked Read 1 Sent Done Done false
4180 1947 Acked Read 7 Sent Done Done false
4181 1948 Acked Read 1 Sent Done Done false
4182 1949 Acked Read 7 Sent Done Done false
4183 1950 Acked Write 4 Sent Done Done false
4184 1951 Acked Flush 0 Sent Done Done false
4185 1952 Acked Write 8 Sent Done Done false
4186 1953 Acked Read 10 Sent Done Done false
4187 1954 Acked Read 3 Sent Done Done false
4188 1955 Acked Read 4 Sent Done Done false
4189 1956 Acked Read 3 Sent Done Done false
4190 1957 Acked Read 10 Sent Done Done false
4191 1958 Acked Write 1 Sent Done Done false
4192 1959 Acked Write 7 Sent Done Done false
4193 1960 Acked Read 1 Sent Done Done false
4194 1961 Acked Write 10 Sent Done Done false
4195 1962 Acked Read 2 Sent Done Done false
4196 1963 Acked Write 9 Sent Done Done false
4197 1964 Acked Write 9 Sent Done Done false
4198 1965 Acked Write 3 Sent Done Done false
4199 1966 Acked Read 2 Sent Done Done false
4200 1967 Acked Read 7 Sent Done Done false
4201 1968 Acked Read 6 Sent Done Done false
4202 1969 Acked Flush 0 Sent Done Done false
4203 1970 Acked Flush 0 Sent Done Done false
4204 1971 Acked Read 7 Sent Done Done false
4205 1972 Acked Write 9 Sent Done Done false
4206 1973 Acked Flush 0 Sent Done Done false
4207 1974 Acked Write 8 Sent Done Done false
4208 1975 Acked Flush 0 Sent Done Done false
4209 1976 Acked Read 10 Sent Done Done false
4210 1977 Acked Read 7 Sent Done Done false
4211 1978 Acked Read 4 Sent Done Done false
4212 1979 Acked Read 4 Sent Done Done false
4213 1980 Acked Read 7 Sent Done Done false
4214 1981 Acked Read 6 Sent Done Done false
4215 1982 Acked Read 9 Sent Done Done false
4216 1983 Acked Write 8 Sent Done Done false
4217 1984 Acked Read 1 Sent Done Done false
4218 1985 Acked Flush 0 Sent Done Done false
4219 1986 Acked Write 4 Sent Done Done false
4220 1987 Acked Flush 0 Sent Done Done false
4221 1988 Acked Write 7 Sent Done Done false
4222 1989 Acked Write 7 Sent Done Done false
4223 1990 Acked Write 4 Sent Done Done false
4224 1991 Acked Read 8 Sent Done Done false
4225 1992 Acked Flush 0 Sent Done Done false
4226 1993 Acked Read 9 Sent Done Done false
4227 1994 Acked Write 9 Sent Done Done false
4228 1995 Acked Flush 0 Sent Done Done false
4229 1996 Acked Read 1 Sent Done Done false
4230 1997 Acked Read 5 Sent Done Done false
4231 1998 Acked Write 4 Sent Done Done false
4232 1999 Acked Read 9 Sent Done Done false
4233 2000 Acked Write 1 Sent Done Done false
4234 2001 Acked Read 4 Sent Done Done false
4235 2002 Acked Write 8 Sent Done Done false
4236 2003 Acked Write 10 Sent Done Done false
4237 2004 Acked Read 2 Sent Done Done false
4238 2005 Acked Read 1 Sent Done Done false
4239 2006 Acked Write 3 Sent Done Done false
4240 2007 Acked Read 3 Sent Done Done false
4241 2008 Acked Write 5 Sent Done Done false
4242 2009 Acked Read 7 Sent Done Done false
4243 2010 Acked Read 5 Sent Done Done false
4244 2011 Acked Write 4 Sent Done Done false
4245 2012 Acked Write 7 Sent Done Done false
4246 2013 Acked Write 3 Sent Done Done false
4247 2014 Acked Write 7 Sent Done Done false
4248 2015 Acked Read 5 Sent Done Done false
4249 2016 Acked Write 1 Sent Done Done false
4250 2017 Acked Flush 0 Sent Done Done false
4251 2018 Acked Write 9 Sent Done Done false
4252 2019 Acked Write 4 Sent Done Done false
4253 2020 Acked Read 8 Sent Done Done false
4254 2021 Acked Read 7 Sent Done Done false
4255 2022 Acked Read 10 Sent Done Done false
4256 2023 Acked Write 4 Sent Done Done false
4257 2024 Acked Flush 0 Sent Done Done false
4258 2025 Acked Write 10 Sent Done Done false
4259 2026 Acked Write 8 Sent Done Done false
4260 2027 Acked Write 8 Sent Done Done false
4261 2028 Acked Read 8 Sent Done Done false
4262 2029 Acked Flush 0 Sent Done Done false
4263 2030 Acked Write 7 Sent Done Done false
4264 2031 Acked Flush 0 Sent Done Done false
4265 2032 Acked Read 10 Sent Done Done false
4266 2033 Acked Write 1 Sent Done Done false
4267 2034 Acked Write 3 Sent Done Done false
4268 2035 Acked Write 7 Sent Done Done false
4269 2036 Acked Read 1 Sent Done Done false
4270 2037 Acked Read 10 Sent Done Done false
4271 2038 Acked Read 8 Sent Done Done false
4272 2039 Acked Read 1 Sent Done Done false
4273 2040 Acked Read 2 Sent Done Done false
4274 2041 Acked Flush 0 Sent Done Done false
4275 2042 Acked Flush 0 Sent Done Done false
4276 2043 Acked Read 2 Sent Done Done false
4277 2044 Acked Read 5 Sent Done Done false
4278 2045 Acked Read 7 Sent Done Done false
4279 2046 Acked Read 3 Sent Done Done false
4280 2047 Acked Read 9 Sent Done Done false
4281 2048 Acked Read 3 Sent Done Done false
4282 2049 Acked Read 8 Sent Done Done false
4283 2050 Acked Read 6 Sent Done Done false
4284 2051 Acked Read 9 Sent Done Done false
4285 2052 Acked Write 8 Sent Done Done false
4286 2053 Acked Read 1 Sent Done Done false
4287 2054 Acked Read 5 Sent Done Done false
4288 2055 Acked Read 6 Sent Done Done false
4289 2056 Acked Write 2 Sent Done Done false
4290 2057 Acked Write 10 Sent Done Done false
4291 2058 Acked Flush 0 Sent Done Done false
4292 2059 Acked Flush 0 Sent Done Done false
4293 2060 Acked Read 5 Sent Done Done false
4294 2061 Acked Read 5 Sent Done Done false
4295 2062 Acked Write 7 Sent Done Done false
4296 2063 Acked Read 5 Sent Done Done false
4297 2064 Acked Write 8 Sent Done Done false
4298 2065 Acked Write 1 Sent Done Done false
4299 2066 Acked Read 2 Sent Done Done false
4300 2067 Acked Write 7 Sent Done Done false
4301 2068 Acked Write 5 Sent Done Done false
4302 2069 Acked Read 1 Sent Done Done false
4303 2070 Acked Read 8 Sent Done Done false
4304 2071 Acked Write 6 Sent Done Done false
4305 2072 Acked Read 9 Sent Done Done false
4306 2073 Acked Write 10 Sent Done Done false
4307 2074 Acked Write 2 Sent Done Done false
4308 2075 Acked Read 4 Sent Done Done false
4309 2076 Acked Write 9 Sent Done Done false
4310 2077 Acked Read 8 Sent Done Done false
4311 2078 Acked Read 4 Sent Done Done false
4312 2079 Acked Write 6 Sent Done Done false
4313 2080 Acked Write 3 Sent Done Done false
4314 2081 Acked Write 5 Sent Done Done false
4315 2082 Acked Write 7 Sent Done Done false
4316 2083 Acked Write 9 Sent Done Done false
4317 2084 Acked Read 8 Sent Done Done false
4318 2085 Acked Read 5 Sent Done Done false
4319 2086 Acked Write 1 Sent Done Done false
4320 2087 Acked Read 9 Sent Done Done false
4321 2088 Acked Write 2 Sent Done Done false
4322 2089 Acked Read 2 Sent Done Done false
4323 2090 Acked Write 1 Sent Done Done false
4324 2091 Acked Read 4 Sent Done Done false
4325 2092 Acked Write 5 Sent Done Done false
4326 2093 Acked Flush 0 Sent Done Done false
4327 2094 Acked Write 3 Sent Done Done false
4328 2095 Acked Write 10 Sent Done Done false
4329 2096 Acked Write 9 Sent Done Done false
4330 2097 Acked Read 8 Sent Done Done false
4331 2098 Acked Read 1 Sent Done Done false
4332 2099 Acked Read 6 Sent Done Done false
4333 2100 Acked Read 8 Sent Done Done false
4334 2101 Acked Read 5 Sent Done Done false
4335 2102 Acked Read 10 Sent Done Done false
4336 2103 Acked Write 3 Sent Done Done false
4337 2104 Acked Write 7 Sent Done Done false
4338 2105 Acked Write 2 Sent Done Done false
4339 2106 Acked Write 6 Sent Done Done false
4340 2107 Acked Read 7 Sent Done Done false
4341 2108 Acked Read 8 Sent Done Done false
4342 2109 Acked Flush 0 Sent Done Done false
4343 2110 Acked Write 10 Sent Done Done false
4344 2111 Acked Read 1 Sent Done Done false
4345 2112 Acked Read 6 Sent Done Done false
4346 2113 Acked Write 1 Sent Done Done false
4347 2114 Acked Read 8 Sent Done Done false
4348 2115 Acked Read 2 Sent Done Done false
4349 2116 Acked Write 9 Sent Done Done false
4350 2117 Acked Read 1 Sent Done Done false
4351 2118 Acked Read 5 Sent Done Done false
4352 2119 Acked Write 5 Sent Done Done false
4353 2120 Acked Read 7 Sent Done Done false
4354 2121 Acked Write 1 Sent Done Done false
4355 2122 Acked Write 6 Sent Done Done false
4356 2123 Acked Write 2 Sent Done Done false
4357 2124 Acked Read 9 Sent Done Done false
4358 2125 Acked Read 3 Sent Done Done false
4359 2126 Acked Read 5 Sent Done Done false
4360 2127 Acked Read 5 Sent Done Done false
4361 2128 Acked Read 8 Sent Done Done false
4362 2129 Acked Read 3 Sent Done Done false
4363 2130 Acked Flush 0 Sent Done Done false
4364 2131 Acked Read 8 Sent Done Done false
4365 2132 Acked Read 1 Sent Done Done false
4366 2133 Acked Write 10 Sent Done Done false
4367 2134 Acked Read 10 Sent Done Done false
4368 2135 Acked Read 6 Sent Done Done false
4369 2136 Acked Write 2 Sent Done Done false
4370 2137 Acked Read 7 Sent Done Done false
4371 2138 Acked Write 6 Sent Done Done false
4372 2139 Acked Read 9 Sent Done Done false
4373 2140 Acked Write 2 Sent Done Done false
4374 2141 Acked Read 6 Sent Done Done false
4375 2142 Acked Write 9 Sent Done Done false
4376 2143 Acked Write 4 Sent Done Done false
4377 2144 Acked Write 6 Sent Done Done false
4378 2145 Acked Write 7 Sent Done Done false
4379 2146 Acked Write 4 Sent Done Done false
4380 2147 Acked Read 2 Sent Done Done false
4381 2148 Acked Read 3 Sent Done Done false
4382 2149 Acked Flush 0 Sent Done Done false
4383 2150 Acked Read 4 Sent Done Done false
4384 2151 Acked Read 10 Sent Done Done false
4385 2152 Acked Read 4 Sent Done Done false
4386 2153 Acked Read 5 Sent Done Done false
4387 2154 Acked Read 10 Sent Done Done false
4388 2155 Acked Write 4 Sent Done Done false
4389 2156 Acked Write 4 Sent Done Done false
4390 2157 Acked Read 9 Sent Done Done false
4391 2158 Acked Write 4 Sent Done Done false
4392 2159 Acked Write 10 Sent Done Done false
4393 2160 Acked Read 9 Sent Done Done false
4394 2161 Acked Read 9 Sent Done Done false
4395 2162 Acked Read 4 Sent Done Done false
4396 2163 Acked Write 10 Sent Done Done false
4397 2164 Acked Read 10 Sent Done Done false
4398 2165 Acked Write 10 Sent Done Done false
4399 2166 Acked Read 3 Sent Done Done false
4400 2167 Acked Write 4 Sent Done Done false
4401 2168 Acked Write 3 Sent Done Done false
4402 2169 Acked Read 3 Sent Done Done false
4403 2170 Acked Read 9 Sent Done Done false
4404 2171 Acked Write 7 Sent Done Done false
4405 2172 Acked Flush 0 Sent Done Done false
4406 2173 Acked Read 7 Sent Done Done false
4407 2174 Acked Write 2 Sent Done Done false
4408 2175 Acked Write 6 Sent Done Done false
4409 2176 Acked Write 10 Sent Done Done false
4410 2177 Acked Read 8 Sent Done Done false
4411 2178 Acked Read 10 Sent Done Done false
4412 2179 Acked Read 5 Sent Done Done false
4413 2180 Acked Write 3 Sent Done Done false
4414 2181 Acked Read 10 Sent Done Done false
4415 2182 Acked Read 6 Sent Done Done false
4416 2183 Acked Read 8 Sent Done Done false
4417 2184 Acked Read 10 Sent Done Done false
4418 2185 Acked Read 5 Sent Done Done false
4419 2186 Acked Flush 0 Sent Done Done false
4420 2187 Acked Write 6 Sent Done Done false
4421 2188 Acked Read 5 Sent Done Done false
4422 2189 Acked Write 7 Sent Done Done false
4423 2190 Acked Write 4 Sent Done Done false
4424 2191 Acked Write 7 Sent Done Done false
4425 2192 Acked Write 2 Sent Done Done false
4426 2193 Acked Read 7 Sent Done Done false
4427 2194 Acked Write 6 Sent Done Done false
4428 2195 Acked Write 1 Sent Done Done false
4429 2196 Acked Write 4 Sent Done Done false
4430 2197 Acked Read 3 Sent Done Done false
4431 2198 Acked Read 10 Sent Done Done false
4432 2199 Acked Read 10 Sent Done Done false
4433 2200 Acked Flush 0 Sent Done Done false
4434 2201 Acked Read 7 Sent Done Done false
4435 2202 Acked Write 1 Sent Done Done false
4436 2203 Acked Read 4 Sent Done Done false
4437 2204 Acked Read 3 Sent Done Done false
4438 2205 Acked Read 1 Sent Done Done false
4439 2206 Acked Read 8 Sent Done Done false
4440 STATES DS:0 DS:1 DS:2 TOTAL
4441 Sent 301 0 0 301
4442 Done 0 301 301 602
4443 Skipped 0 0 0 0
4444 Error 0 0 0 0
4445 Last Flush: Some(JobId(1905)) Some(JobId(2200)) Some(JobId(2200))
4446 Downstairs last five completed: 1905 1904 1903 1902 1901
4447 Upstairs last five completed: 2206 2205 2204 2203 2202
4448 CLIENT: Up:0 ds:301 act:2
44492025-02-07T20:08:11.019ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
44502025-02-07T20:08:12.020ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
44512025-02-07T20:08:13.022ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
4452 [8810] got start action
4453 [8810] I am going to restart
4454 Starting downstairs at port 8810
4455 Make output file at "/tmp/dsc/downstairs-8810.txt"
4456 [8810][0] reports Starting
4457 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1213)
4458 [8810][0] reports Running
4459 ----------------------------------------------------------------
4460 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4461 States: Offline Active Active
4462 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4463 1906 Acked Flush 0 Sent Done Done false
4464 1907 Acked Write 8 Sent Done Done false
4465 1908 Acked Write 1 Sent Done Done false
4466 1909 Acked Read 10 Sent Done Done false
4467 1910 Acked Write 7 Sent Done Done false
4468 1911 Acked Write 2 Sent Done Done false
4469 1912 Acked Write 7 Sent Done Done false
4470 1913 Acked Read 8 Sent Done Done false
4471 1914 Acked Write 5 Sent Done Done false
4472 1915 Acked Read 9 Sent Done Done false
4473 1916 Acked Flush 0 Sent Done Done false
4474 1917 Acked Read 3 Sent Done Done false
4475 1918 Acked Flush 0 Sent Done Done false
4476 1919 Acked Read 3 Sent Done Done false
4477 1920 Acked Read 4 Sent Done Done false
4478 1921 Acked Read 2 Sent Done Done false
4479 1922 Acked Flush 0 Sent Done Done false
4480 1923 Acked Write 8 Sent Done Done false
4481 1924 Acked Read 5 Sent Done Done false
4482 1925 Acked Read 7 Sent Done Done false
4483 1926 Acked Write 5 Sent Done Done false
4484 1927 Acked Read 6 Sent Done Done false
4485 1928 Acked Read 7 Sent Done Done false
4486 1929 Acked Flush 0 Sent Done Done false
4487 1930 Acked Read 5 Sent Done Done false
4488 1931 Acked Read 3 Sent Done Done false
4489 1932 Acked Write 9 Sent Done Done false
4490 1933 Acked Write 2 Sent Done Done false
4491 1934 Acked Flush 0 Sent Done Done false
4492 1935 Acked Write 9 Sent Done Done false
4493 1936 Acked Write 7 Sent Done Done false
4494 1937 Acked Write 5 Sent Done Done false
4495 1938 Acked Read 1 Sent Done Done false
4496 1939 Acked Read 8 Sent Done Done false
4497 1940 Acked Read 5 Sent Done Done false
4498 1941 Acked Flush 0 Sent Done Done false
4499 1942 Acked Write 1 Sent Done Done false
4500 1943 Acked Write 2 Sent Done Done false
4501 1944 Acked Write 5 Sent Done Done false
4502 1945 Acked Write 7 Sent Done Done false
4503 1946 Acked Read 1 Sent Done Done false
4504 1947 Acked Read 7 Sent Done Done false
4505 1948 Acked Read 1 Sent Done Done false
4506 1949 Acked Read 7 Sent Done Done false
4507 1950 Acked Write 4 Sent Done Done false
4508 1951 Acked Flush 0 Sent Done Done false
4509 1952 Acked Write 8 Sent Done Done false
4510 1953 Acked Read 10 Sent Done Done false
4511 1954 Acked Read 3 Sent Done Done false
4512 1955 Acked Read 4 Sent Done Done false
4513 1956 Acked Read 3 Sent Done Done false
4514 1957 Acked Read 10 Sent Done Done false
4515 1958 Acked Write 1 Sent Done Done false
4516 1959 Acked Write 7 Sent Done Done false
4517 1960 Acked Read 1 Sent Done Done false
4518 1961 Acked Write 10 Sent Done Done false
4519 1962 Acked Read 2 Sent Done Done false
4520 1963 Acked Write 9 Sent Done Done false
4521 1964 Acked Write 9 Sent Done Done false
4522 1965 Acked Write 3 Sent Done Done false
4523 1966 Acked Read 2 Sent Done Done false
4524 1967 Acked Read 7 Sent Done Done false
4525 1968 Acked Read 6 Sent Done Done false
4526 1969 Acked Flush 0 Sent Done Done false
4527 1970 Acked Flush 0 Sent Done Done false
4528 1971 Acked Read 7 Sent Done Done false
4529 1972 Acked Write 9 Sent Done Done false
4530 1973 Acked Flush 0 Sent Done Done false
4531 1974 Acked Write 8 Sent Done Done false
4532 1975 Acked Flush 0 Sent Done Done false
4533 1976 Acked Read 10 Sent Done Done false
4534 1977 Acked Read 7 Sent Done Done false
4535 1978 Acked Read 4 Sent Done Done false
4536 1979 Acked Read 4 Sent Done Done false
4537 1980 Acked Read 7 Sent Done Done false
4538 1981 Acked Read 6 Sent Done Done false
4539 1982 Acked Read 9 Sent Done Done false
4540 1983 Acked Write 8 Sent Done Done false
4541 1984 Acked Read 1 Sent Done Done false
4542 1985 Acked Flush 0 Sent Done Done false
4543 1986 Acked Write 4 Sent Done Done false
4544 1987 Acked Flush 0 Sent Done Done false
4545 1988 Acked Write 7 Sent Done Done false
4546 1989 Acked Write 7 Sent Done Done false
4547 1990 Acked Write 4 Sent Done Done false
4548 1991 Acked Read 8 Sent Done Done false
4549 1992 Acked Flush 0 Sent Done Done false
4550 1993 Acked Read 9 Sent Done Done false
4551 1994 Acked Write 9 Sent Done Done false
4552 1995 Acked Flush 0 Sent Done Done false
4553 1996 Acked Read 1 Sent Done Done false
4554 1997 Acked Read 5 Sent Done Done false
4555 1998 Acked Write 4 Sent Done Done false
4556 1999 Acked Read 9 Sent Done Done false
4557 2000 Acked Write 1 Sent Done Done false
4558 2001 Acked Read 4 Sent Done Done false
4559 2002 Acked Write 8 Sent Done Done false
4560 2003 Acked Write 10 Sent Done Done false
4561 2004 Acked Read 2 Sent Done Done false
4562 2005 Acked Read 1 Sent Done Done false
4563 2006 Acked Write 3 Sent Done Done false
4564 2007 Acked Read 3 Sent Done Done false
4565 2008 Acked Write 5 Sent Done Done false
4566 2009 Acked Read 7 Sent Done Done false
4567 2010 Acked Read 5 Sent Done Done false
4568 2011 Acked Write 4 Sent Done Done false
4569 2012 Acked Write 7 Sent Done Done false
4570 2013 Acked Write 3 Sent Done Done false
4571 2014 Acked Write 7 Sent Done Done false
4572 2015 Acked Read 5 Sent Done Done false
4573 2016 Acked Write 1 Sent Done Done false
4574 2017 Acked Flush 0 Sent Done Done false
4575 2018 Acked Write 9 Sent Done Done false
4576 2019 Acked Write 4 Sent Done Done false
4577 2020 Acked Read 8 Sent Done Done false
4578 2021 Acked Read 7 Sent Done Done false
4579 2022 Acked Read 10 Sent Done Done false
4580 2023 Acked Write 4 Sent Done Done false
4581 2024 Acked Flush 0 Sent Done Done false
4582 2025 Acked Write 10 Sent Done Done false
4583 2026 Acked Write 8 Sent Done Done false
4584 2027 Acked Write 8 Sent Done Done false
4585 2028 Acked Read 8 Sent Done Done false
4586 2029 Acked Flush 0 Sent Done Done false
4587 2030 Acked Write 7 Sent Done Done false
4588 2031 Acked Flush 0 Sent Done Done false
4589 2032 Acked Read 10 Sent Done Done false
4590 2033 Acked Write 1 Sent Done Done false
4591 2034 Acked Write 3 Sent Done Done false
4592 2035 Acked Write 7 Sent Done Done false
4593 2036 Acked Read 1 Sent Done Done false
4594 2037 Acked Read 10 Sent Done Done false
4595 2038 Acked Read 8 Sent Done Done false
4596 2039 Acked Read 1 Sent Done Done false
4597 2040 Acked Read 2 Sent Done Done false
4598 2041 Acked Flush 0 Sent Done Done false
4599 2042 Acked Flush 0 Sent Done Done false
4600 2043 Acked Read 2 Sent Done Done false
4601 2044 Acked Read 5 Sent Done Done false
4602 2045 Acked Read 7 Sent Done Done false
4603 2046 Acked Read 3 Sent Done Done false
4604 2047 Acked Read 9 Sent Done Done false
4605 2048 Acked Read 3 Sent Done Done false
4606 2049 Acked Read 8 Sent Done Done false
4607 2050 Acked Read 6 Sent Done Done false
4608 2051 Acked Read 9 Sent Done Done false
4609 2052 Acked Write 8 Sent Done Done false
4610 2053 Acked Read 1 Sent Done Done false
4611 2054 Acked Read 5 Sent Done Done false
4612 2055 Acked Read 6 Sent Done Done false
4613 2056 Acked Write 2 Sent Done Done false
4614 2057 Acked Write 10 Sent Done Done false
4615 2058 Acked Flush 0 Sent Done Done false
4616 2059 Acked Flush 0 Sent Done Done false
4617 2060 Acked Read 5 Sent Done Done false
4618 2061 Acked Read 5 Sent Done Done false
4619 2062 Acked Write 7 Sent Done Done false
4620 2063 Acked Read 5 Sent Done Done false
4621 2064 Acked Write 8 Sent Done Done false
4622 2065 Acked Write 1 Sent Done Done false
4623 2066 Acked Read 2 Sent Done Done false
4624 2067 Acked Write 7 Sent Done Done false
4625 2068 Acked Write 5 Sent Done Done false
4626 2069 Acked Read 1 Sent Done Done false
4627 2070 Acked Read 8 Sent Done Done false
4628 2071 Acked Write 6 Sent Done Done false
4629 2072 Acked Read 9 Sent Done Done false
4630 2073 Acked Write 10 Sent Done Done false
4631 2074 Acked Write 2 Sent Done Done false
4632 2075 Acked Read 4 Sent Done Done false
4633 2076 Acked Write 9 Sent Done Done false
4634 2077 Acked Read 8 Sent Done Done false
4635 2078 Acked Read 4 Sent Done Done false
4636 2079 Acked Write 6 Sent Done Done false
4637 2080 Acked Write 3 Sent Done Done false
4638 2081 Acked Write 5 Sent Done Done false
4639 2082 Acked Write 7 Sent Done Done false
4640 2083 Acked Write 9 Sent Done Done false
4641 2084 Acked Read 8 Sent Done Done false
4642 2085 Acked Read 5 Sent Done Done false
4643 2086 Acked Write 1 Sent Done Done false
4644 2087 Acked Read 9 Sent Done Done false
4645 2088 Acked Write 2 Sent Done Done false
4646 2089 Acked Read 2 Sent Done Done false
4647 2090 Acked Write 1 Sent Done Done false
4648 2091 Acked Read 4 Sent Done Done false
4649 2092 Acked Write 5 Sent Done Done false
4650 2093 Acked Flush 0 Sent Done Done false
4651 2094 Acked Write 3 Sent Done Done false
4652 2095 Acked Write 10 Sent Done Done false
4653 2096 Acked Write 9 Sent Done Done false
4654 2097 Acked Read 8 Sent Done Done false
4655 2098 Acked Read 1 Sent Done Done false
4656 2099 Acked Read 6 Sent Done Done false
4657 2100 Acked Read 8 Sent Done Done false
4658 2101 Acked Read 5 Sent Done Done false
4659 2102 Acked Read 10 Sent Done Done false
4660 2103 Acked Write 3 Sent Done Done false
4661 2104 Acked Write 7 Sent Done Done false
4662 2105 Acked Write 2 Sent Done Done false
4663 2106 Acked Write 6 Sent Done Done false
4664 2107 Acked Read 7 Sent Done Done false
4665 2108 Acked Read 8 Sent Done Done false
4666 2109 Acked Flush 0 Sent Done Done false
4667 2110 Acked Write 10 Sent Done Done false
4668 2111 Acked Read 1 Sent Done Done false
4669 2112 Acked Read 6 Sent Done Done false
4670 2113 Acked Write 1 Sent Done Done false
4671 2114 Acked Read 8 Sent Done Done false
4672 2115 Acked Read 2 Sent Done Done false
4673 2116 Acked Write 9 Sent Done Done false
4674 2117 Acked Read 1 Sent Done Done false
4675 2118 Acked Read 5 Sent Done Done false
4676 2119 Acked Write 5 Sent Done Done false
4677 2120 Acked Read 7 Sent Done Done false
4678 2121 Acked Write 1 Sent Done Done false
4679 2122 Acked Write 6 Sent Done Done false
4680 2123 Acked Write 2 Sent Done Done false
4681 2124 Acked Read 9 Sent Done Done false
4682 2125 Acked Read 3 Sent Done Done false
4683 2126 Acked Read 5 Sent Done Done false
4684 2127 Acked Read 5 Sent Done Done false
4685 2128 Acked Read 8 Sent Done Done false
4686 2129 Acked Read 3 Sent Done Done false
4687 2130 Acked Flush 0 Sent Done Done false
4688 2131 Acked Read 8 Sent Done Done false
4689 2132 Acked Read 1 Sent Done Done false
4690 2133 Acked Write 10 Sent Done Done false
4691 2134 Acked Read 10 Sent Done Done false
4692 2135 Acked Read 6 Sent Done Done false
4693 2136 Acked Write 2 Sent Done Done false
4694 2137 Acked Read 7 Sent Done Done false
4695 2138 Acked Write 6 Sent Done Done false
4696 2139 Acked Read 9 Sent Done Done false
4697 2140 Acked Write 2 Sent Done Done false
4698 2141 Acked Read 6 Sent Done Done false
4699 2142 Acked Write 9 Sent Done Done false
4700 2143 Acked Write 4 Sent Done Done false
4701 2144 Acked Write 6 Sent Done Done false
4702 2145 Acked Write 7 Sent Done Done false
4703 2146 Acked Write 4 Sent Done Done false
4704 2147 Acked Read 2 Sent Done Done false
4705 2148 Acked Read 3 Sent Done Done false
4706 2149 Acked Flush 0 Sent Done Done false
4707 2150 Acked Read 4 Sent Done Done false
4708 2151 Acked Read 10 Sent Done Done false
4709 2152 Acked Read 4 Sent Done Done false
4710 2153 Acked Read 5 Sent Done Done false
4711 2154 Acked Read 10 Sent Done Done false
4712 2155 Acked Write 4 Sent Done Done false
4713 2156 Acked Write 4 Sent Done Done false
4714 2157 Acked Read 9 Sent Done Done false
4715 2158 Acked Write 4 Sent Done Done false
4716 2159 Acked Write 10 Sent Done Done false
4717 2160 Acked Read 9 Sent Done Done false
4718 2161 Acked Read 9 Sent Done Done false
4719 2162 Acked Read 4 Sent Done Done false
4720 2163 Acked Write 10 Sent Done Done false
4721 2164 Acked Read 10 Sent Done Done false
4722 2165 Acked Write 10 Sent Done Done false
4723 2166 Acked Read 3 Sent Done Done false
4724 2167 Acked Write 4 Sent Done Done false
4725 2168 Acked Write 3 Sent Done Done false
4726 2169 Acked Read 3 Sent Done Done false
4727 2170 Acked Read 9 Sent Done Done false
4728 2171 Acked Write 7 Sent Done Done false
4729 2172 Acked Flush 0 Sent Done Done false
4730 2173 Acked Read 7 Sent Done Done false
4731 2174 Acked Write 2 Sent Done Done false
4732 2175 Acked Write 6 Sent Done Done false
4733 2176 Acked Write 10 Sent Done Done false
4734 2177 Acked Read 8 Sent Done Done false
4735 2178 Acked Read 10 Sent Done Done false
4736 2179 Acked Read 5 Sent Done Done false
4737 2180 Acked Write 3 Sent Done Done false
4738 2181 Acked Read 10 Sent Done Done false
4739 2182 Acked Read 6 Sent Done Done false
4740 2183 Acked Read 8 Sent Done Done false
4741 2184 Acked Read 10 Sent Done Done false
4742 2185 Acked Read 5 Sent Done Done false
4743 2186 Acked Flush 0 Sent Done Done false
4744 2187 Acked Write 6 Sent Done Done false
4745 2188 Acked Read 5 Sent Done Done false
4746 2189 Acked Write 7 Sent Done Done false
4747 2190 Acked Write 4 Sent Done Done false
4748 2191 Acked Write 7 Sent Done Done false
4749 2192 Acked Write 2 Sent Done Done false
4750 2193 Acked Read 7 Sent Done Done false
4751 2194 Acked Write 6 Sent Done Done false
4752 2195 Acked Write 1 Sent Done Done false
4753 2196 Acked Write 4 Sent Done Done false
4754 2197 Acked Read 3 Sent Done Done false
4755 2198 Acked Read 10 Sent Done Done false
4756 2199 Acked Read 10 Sent Done Done false
4757 2200 Acked Flush 0 Sent Done Done false
4758 2201 Acked Read 7 Sent Done Done false
4759 2202 Acked Write 1 Sent Done Done false
4760 2203 Acked Read 4 Sent Done Done false
4761 2204 Acked Read 3 Sent Done Done false
4762 2205 Acked Read 1 Sent Done Done false
4763 2206 Acked Read 8 Sent Done Done false
4764 STATES DS:0 DS:1 DS:2 TOTAL
4765 Sent 301 0 0 301
4766 Done 0 301 301 602
4767 Skipped 0 0 0 0
4768 Error 0 0 0 0
4769 Last Flush: Some(JobId(1905)) Some(JobId(2200)) Some(JobId(2200))
4770 Downstairs last five completed: 1905 1904 1903 1902 1901
4771 Upstairs last five completed: 2206 2205 2204 2203 2202
4772 CLIENT: Up:0 ds:301 act:2
47732025-02-07T20:08:15.025ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
47742025-02-07T20:08:16.027ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
47752025-02-07T20:08:16.227ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
47762025-02-07T20:08:16.227ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41407) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
47772025-02-07T20:08:16.228ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
47782025-02-07T20:08:16.229ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
47792025-02-07T20:08:16.229ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
47802025-02-07T20:08:16.229ZINFOcrucible: send last flush ID to this DS: Some(JobId(1905)) = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
47812025-02-07T20:08:16.229ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
47822025-02-07T20:08:16.229ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(1905)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
47832025-02-07T20:08:16.229ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(1905)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
4784 ----------------------------------------------------------------
4785 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4786 Downstairs last five completed: 2207 2206 2205 2204 2203
4787 Upstairs last five completed: 2207 2206 2205 2204 2203
4788 CLIENT: Up:0 ds:0 act:3
4789 Replay: All jobs finished
4790 Main task has work to do, go find it
4791 got dsc Stop(1)
4792 stop 1
4793 Feb 07 20:08:18.479 INFO request completed, latency_us: 344, response_code: 204[8820] Got stop action so:false kr:false
4794 , uri: /stop/cid/1, method: POST, req_id: 4ec3064c-c9c8-449d-8af7-7528417431b6, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
47952025-02-07T20:08:18.480ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
47962025-02-07T20:08:18.480ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
47972025-02-07T20:08:18.480ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
4798 Feb 07 20:08:18.480 INFO request completed, latency_us: 150, response_code: 200, uri: /state/cid/1, method: GET, req_id: 9a8ba998-eed3-4275-9bee-8c204c02e223, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
47992025-02-07T20:08:18.480ZINFOcrucible: client task is exiting = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
4800 [8820] Exited with: ExitStatus(unix_wait_status(9))
4801 [8820][1] reports Exit
4802 Feb 07 20:08:22.482 INFO request completed, latency_us: 380, response_code: 200, uri: /state/cid/1, method: GET, req_id: 56c5b6b6-56ef-4921-b937-376b3e3c45d9, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
4803 001/300 Write block 2894 len 4096 data: 2
4804 002/300 Read block 2190 len 24576
4805 003/300 Flush
4806 004/300 Write block 4880 len 40960 data: 2 2 3 3 3 3 4 4 4 4
4807 005/300 Read block 631 len 24576
4808 006/300 Read block 1690 len 12288
4809 007/300 Read block 4720 len 8192
4810 008/300 Read block 3684 len 28672
4811 009/300 Write block 1121 len 12288 data: 2 2 2
4812 010/300 Write block 1202 len 40960 data: 2 2 2 2 2 2 2 2 2 3
4813 011/300 Write block 4513 len 32768 data: 2 3 3 3 3 3 3 4
4814 012/300 Read block 4038 len 12288
4815 013/300 Write block 2203 len 20480 data: 3 2 2 2 2
4816 014/300 Write block 3599 len 12288 data: 2 4 4
4817 015/300 Flush
4818 016/300 Read block 1149 len 8192
4819 017/300 Read block 116 len 40960
4820 018/300 Flush
4821 019/300 Read block 3507 len 16384
4822 020/300 Read block 1803 len 4096
4823 021/300 Write block 3888 len 28672 data: 3 2 2 2 2 2 2
4824 022/300 Read block 3318 len 36864
4825 023/300 Write block 1938 len 4096 data: 3
4826 024/300 Read block 1982 len 28672
4827 025/300 Write block 4483 len 12288 data: 2 2 2
4828 026/300 Read block 4107 len 36864
4829 027/300 Read block 2119 len 40960
4830 028/300 Write block 2004 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4831 029/300 Write block 3227 len 36864 data: 2 3 3 3 3 3 3 3 3
4832 030/300 Write block 336 len 32768 data: 3 3 2 2 2 2 2 2
4833 031/300 Flush
4834 032/300 Write block 882 len 32768 data: 3 3 3 3 4 3 3 3
4835 033/300 Read block 3057 len 16384
4836 034/300 Read block 2484 len 24576
4837 035/300 Read block 1191 len 28672
4838 036/300 Read block 2048 len 4096
4839 037/300 Write block 432 len 12288 data: 2 2 2
4840 038/300 Read block 4870 len 24576
4841 039/300 Write block 166 len 36864 data: 2 3 3 3 3 2 2 2 2
4842 040/300 Read block 3146 len 40960
4843 041/300 Read block 4619 len 16384
4844 042/300 Read block 436 len 36864
4845 043/300 Write block 4193 len 28672 data: 3 2 2 2 2 2 2
4846 044/300 Write block 3947 len 16384 data: 3 3 2 2
4847 045/300 Read block 1320 len 4096
4848 046/300 Read block 2219 len 16384
4849 047/300 Write block 264 len 28672 data: 2 2 2 2 2 3 3
4850 048/300 Write block 1000 len 12288 data: 4 6 6
4851 049/300 Read block 2187 len 36864
4852 050/300 Read block 3348 len 32768
4853 051/300 Write block 3797 len 32768 data: 2 2 2 2 2 2 2 2
4854 052/300 Read block 3593 len 24576
4855 053/300 Write block 2515 len 40960 data: 3 3 3 2 2 2 2 2 2 2
4856 054/300 Read block 3605 len 8192
4857 055/300 Write block 582 len 24576 data: 2 3 3 3 3 3
4858 056/300 Read block 4153 len 4096
4859 057/300 Read block 348 len 16384
4860 058/300 Flush
4861 059/300 Write block 3048 len 20480 data: 2 2 2 2 2
4862 060/300 Flush
4863 061/300 Read block 4658 len 4096
4864 062/300 Read block 3726 len 8192
4865 063/300 Write block 4016 len 16384 data: 2 2 2 2
4866 064/300 Write block 2368 len 4096 data: 2
4867 065/300 Read block 2433 len 40960
4868 066/300 Write block 4437 len 32768 data: 2 2 2 2 2 2 2 2
4869 067/300 Write block 4849 len 20480 data: 4 4 4 4 4
4870 068/300 Write block 3770 len 20480 data: 3 3 2 2 2
4871 069/300 Flush
4872 070/300 Read block 4961 len 16384
4873 071/300 Read block 3193 len 8192
4874 072/300 Read block 4837 len 20480
4875 073/300 Write block 3406 len 32768 data: 3 2 2 2 3 3 4 4
4876 074/300 Flush
4877 075/300 Write block 4732 len 20480 data: 5 5 5 4 3
4878 076/300 Write block 1904 len 24576 data: 2 2 2 2 2 2
4879 077/300 Write block 3185 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4880 078/300 Write block 3001 len 28672 data: 2 2 2 2 2 2 2
4881 079/300 Write block 4899 len 40960 data: 3 2 3 4 3 3 3 3 3 3
4882 080/300 Read block 748 len 32768
4883 081/300 Flush
4884 082/300 Flush
4885 083/300 Write block 3359 len 16384 data: 2 2 3 3
4886 084/300 Read block 4568 len 40960
4887 085/300 Flush
4888 086/300 Read block 1728 len 20480
4889 087/300 Read block 2456 len 28672
4890 088/300 Read block 743 len 32768
4891 089/300 Write block 3100 len 4096 data: 2
4892 090/300 Write block 3486 len 4096 data: 2
4893 091/300 Write block 4365 len 16384 data: 2 2 2 2
4894 092/300 Write block 3452 len 36864 data: 2 2 2 2 2 2 2 3 3
4895 093/300 Write block 4973 len 8192 data: 2 2
4896 094/300 Read block 74 len 40960
4897 095/300 Read block 1984 len 8192
4898 096/300 Flush
4899 097/300 Read block 4582 len 40960
4900 098/300 Read block 3988 len 4096
4901 099/300 Read block 4476 len 40960
4902 100/300 Read block 2023 len 8192
4903 101/300 Write block 3009 len 28672 data: 2 2 2 2 2 2 2
4904 102/300 Write block 4955 len 20480 data: 2 2 2 2 2
4905 103/300 Read block 2638 len 4096
4906 104/300 Write block 4743 len 12288 data: 2 2 2
4907 105/300 Read block 1189 len 12288
4908 106/300 Write block 3311 len 36864 data: 2 2 2 2 2 2 2 2 2
4909 107/300 Read block 3120 len 32768
4910 108/300 Write block 2670 len 40960 data: 2 2 2 3 3 3 3 3 3 3
4911 109/300 Read block 1286 len 36864
4912 110/300 Flush
4913 111/300 Write block 361 len 36864 data: 3 3 3 2 2 3 2 2 2
4914 112/300 Read block 1714 len 36864
4915 113/300 Read block 3556 len 8192
4916 114/300 Read block 3117 len 32768
4917 115/300 Read block 4055 len 20480
4918 116/300 Write block 2431 len 32768 data: 3 3 3 3 3 3 3 3
4919 117/300 Read block 97 len 8192
4920 118/300 Write block 675 len 16384 data: 3 3 4 4
4921 119/300 Write block 3669 len 20480 data: 3 3 3 2 3
4922 120/300 Write block 2696 len 4096 data: 2
4923 121/300 Read block 2180 len 36864
4924 122/300 Read block 1920 len 4096
4925 123/300 Write block 2918 len 28672 data: 3 3 2 2 2 2 2
4926 124/300 Flush
4927 125/300 Read block 2294 len 8192
4928 126/300 Write block 1892 len 36864 data: 3 2 2 2 2 2 2 2 2
4929 127/300 Write block 2580 len 8192 data: 3 3
4930 128/300 Read block 1464 len 8192
4931 129/300 Read block 2400 len 8192
4932 130/300 Read block 4484 len 36864
4933 131/300 Read block 3949 len 12288
4934 132/300 Write block 3488 len 36864 data: 2 2 2 2 2 2 3 3 3
4935 133/300 Read block 780 len 40960
4936 134/300 Write block 197 len 16384 data: 2 2 2 2
4937 135/300 Read block 1247 len 32768
4938 136/300 Flush
4939 137/300 Read block 1844 len 40960
4940 138/300 Write block 4829 len 32768 data: 2 2 2 4 4 4 4 5
4941 139/300 Read block 4511 len 12288
4942 140/300 Read block 1546 len 12288
4943 141/300 Write block 858 len 12288 data: 2 2 2
4944 142/300 Write block 4898 len 36864 data: 4 4 3 4 5 4 4 4 4
4945 143/300 Read block 3661 len 32768
4946 144/300 Read block 1408 len 16384
4947 145/300 Write block 2392 len 40960 data: 2 2 2 3 3 3 3 3 3 3
4948 146/300 Read block 4876 len 20480
4949 147/300 Write block 2623 len 24576 data: 3 3 3 3 3 3
4950 148/300 Read block 2390 len 16384
4951 149/300 Read block 1520 len 28672
4952 150/300 Read block 2862 len 28672
4953 151/300 Read block 4173 len 8192
4954 152/300 Write block 4565 len 32768 data: 2 2 2 2 2 2 2 2
4955 153/300 Write block 4898 len 8192 data: 5 5
4956 154/300 Read block 1084 len 32768
4957 155/300 Write block 3997 len 4096 data: 3
4958 156/300 Read block 4740 len 24576
4959 157/300 Read block 3196 len 8192
4960 158/300 Read block 3472 len 40960
4961 159/300 Read block 1867 len 32768
4962 160/300 Read block 1554 len 24576
4963 161/300 Read block 1532 len 4096
4964 162/300 Read block 3733 len 16384
4965 163/300 Write block 813 len 4096 data: 3
4966 164/300 Read block 2505 len 40960
4967 165/300 Write block 2457 len 20480 data: 4 4 3 3 3
4968 166/300 Write block 1848 len 8192 data: 3 3
4969 167/300 Read block 1297 len 8192
4970 168/300 Write block 4895 len 12288 data: 3 3 4
4971 169/300 Write block 2085 len 32768 data: 3 3 3 2 2 2 2 2
4972 170/300 Read block 3419 len 20480
4973 171/300 Write block 992 len 16384 data: 2 4 4 4
4974 172/300 Read block 4324 len 20480
4975 173/300 Read block 3549 len 20480
4976 174/300 Read block 4888 len 12288
4977 175/300 Write block 1179 len 16384 data: 3 3 3 3
4978 176/300 Read block 2220 len 40960
4979 177/300 Write block 4024 len 4096 data: 2
4980 178/300 Read block 2117 len 28672
4981 179/300 Read block 1519 len 40960
4982 180/300 Flush
4983 181/300 Write block 1474 len 32768 data: 2 2 2 2 2 2 2 2
4984 182/300 Read block 320 len 40960
4985 183/300 Write block 2832 len 12288 data: 2 2 2
4986 184/300 Write block 1718 len 28672 data: 3 2 2 2 2 2 2
4987 185/300 Flush
4988 186/300 Read block 1877 len 24576
4989 187/300 Write block 3475 len 40960 data: 4 4 4 4 3 2 2 2 2 2
4990 188/300 Read block 3539 len 8192
4991 189/300 Read block 787 len 8192
4992 190/300 Read block 2177 len 36864
4993 191/300 Read block 318 len 24576
4994 192/300 Write block 3758 len 4096 data: 3
4995 193/300 Read block 2128 len 8192
4996 194/300 Write block 4786 len 40960 data: 3 3 3 3 3 3 2 2 3 3
4997 195/300 Read block 3264 len 36864
4998 196/300 Flush
4999 197/300 Read block 2573 len 36864
5000 198/300 Write block 4177 len 12288 data: 2 2 2
5001 199/300 Read block 90 len 36864
5002 200/300 Read block 4942 len 32768
5003 201/300 Write block 1017 len 24576 data: 2 2 2 2 2 2
5004 202/300 Read block 348 len 20480
5005 203/300 Write block 4060 len 12288 data: 2 3 4
5006 204/300 Read block 2870 len 12288
5007 205/300 Read block 1049 len 24576
5008 206/300 Read block 3244 len 4096
5009 207/300 Write block 3133 len 40960 data: 4 3 3 3 3 3 3 3 3 2
5010 208/300 Read block 4258 len 16384
5011 209/300 Read block 1305 len 4096
5012 210/300 Write block 2513 len 20480 data: 2 3 4 4 4
5013 211/300 Write block 1795 len 4096 data: 3
5014 212/300 Read block 2535 len 4096
5015 213/300 Write block 1538 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5016 214/300 Write block 3301 len 12288 data: 2 2 2
5017 215/300 Read block 4735 len 20480
5018 216/300 Read block 2874 len 16384
5019 217/300 Write block 2204 len 28672 data: 3 3 3 3 2 2 2
5020 218/300 Write block 2381 len 16384 data: 3 4 4 4
5021 219/300 Read block 3723 len 4096
5022 220/300 Write block 3660 len 40960 data: 2 2 3 3 3 3 3 3 3 4
5023 221/300 Flush
5024 222/300 Read block 4149 len 4096
5025 223/300 Read block 3387 len 20480
5026 224/300 Write block 2259 len 24576 data: 2 2 3 3 3 3
5027 225/300 Write block 2053 len 16384 data: 2 2 2 2
5028 226/300 Read block 3219 len 4096
5029 227/300 Read block 871 len 4096
5030 228/300 Read block 3541 len 28672
5031 229/300 Read block 2465 len 32768
5032 230/300 Write block 3476 len 32768 data: 5 5 5 4 3 3 3 3
5033 231/300 Read block 297 len 8192
5034 232/300 Read block 2791 len 12288
5035 233/300 Read block 3422 len 40960
5036 234/300 Write block 2250 len 20480 data: 2 2 2 2 2
5037 235/300 Write block 2860 len 36864 data: 4 4 4 3 3 2 2 2 3
5038 236/300 Write block 828 len 32768 data: 2 2 2 2 2 2 2 2
5039 237/300 Read block 1499 len 28672
5040 238/300 Read block 4447 len 24576
5041 239/300 Read block 2906 len 40960
5042 240/300 Read block 4389 len 4096
5043 241/300 Write block 1121 len 20480 data: 3 3 3 2 2
5044 242/300 Write block 1713 len 4096 data: 3
5045 243/300 Write block 3817 len 12288 data: 3 3 3
5046 244/300 Write block 2593 len 32768 data: 3 3 3 3 3 2 2 2
5047 245/300 Read block 947 len 16384
5048 246/300 Read block 1763 len 32768
5049 247/300 Read block 1294 len 24576
5050 248/300 Write block 2519 len 40960 data: 3 3 3 3 3 3 2 2 2 2
5051 249/300 Flush
5052 250/300 Write block 479 len 8192 data: 2 2
5053 251/300 Flush
5054 252/300 Write block 4374 len 24576 data: 2 3 2 2 2 2
5055 253/300 Read block 2178 len 36864
5056 254/300 Read block 403 len 36864
5057 255/300 Write block 1971 len 32768 data: 2 2 2 2 2 2 3 3
5058 256/300 Read block 2689 len 32768
5059 257/300 Read block 859 len 28672
5060 258/300 Read block 2655 len 12288
5061 259/300 Write block 3260 len 16384 data: 4 3 3 3
5062 260/300 Write block 187 len 12288 data: 3 3 3
5063 261/300 Write block 1999 len 4096 data: 3
5064 262/300 Read block 4391 len 40960
5065 263/300 Read block 4604 len 8192
5066 264/300 Write block 1274 len 24576 data: 2 4 4 4 4 4
5067 265/300 Read block 28 len 16384
5068 266/300 Write block 3333 len 4096 data: 4
5069 267/300 Read block 3731 len 12288
5070 268/300 Write block 2155 len 20480 data: 2 2 2 2 2
5071 269/300 Read block 1184 len 36864
5072 270/300 Flush
5073 271/300 Read block 3706 len 16384
5074 272/300 Read block 1389 len 32768
5075 273/300 Read block 414 len 16384
5076 274/300 Read block 284 len 8192
5077 275/300 Read block 2523 len 32768
5078 276/300 Read block 4008 len 8192
5079 277/300 Read block 1321 len 16384
5080 278/300 Read block 4767 len 4096
5081 279/300 Read block 347 len 4096
5082 280/300 Write block 3510 len 28672 data: 2 2 2 2 2 2 2
5083 281/300 Read block 3676 len 16384
5084 282/300 Read block 1774 len 36864
5085 283/300 Read block 4847 len 8192
5086 284/300 Write block 4273 len 20480 data: 2 3 3 3 2
5087 285/300 Write block 4953 len 4096 data: 2
5088 286/300 Read block 874 len 20480
5089 287/300 Read block 4197 len 32768
5090 288/300 Read block 447 len 28672
5091 289/300 Write block 2801 len 8192 data: 5 7
5092 290/300 Flush
5093 291/300 Write block 75 len 20480 data: 3 3 3 3 3
5094 292/300 Read block 4758 len 4096
5095 293/300 Read block 404 len 8192
5096 294/300 Read block 3765 len 16384
5097 295/300 Read block 3240 len 4096
5098 296/300 Read block 2587 len 16384
5099 297/300 Read block 4301 len 24576
5100 298/300 Read block 992 len 32768
5101 299/300 Write block 527 len 8192 data: 3 3
5102 300/300 Read block 674 len 28672
5103 301/300 Write block 1390 len 28672 data: 3 3 3 2 2 2 3
5104 Main task has work to do, go find it
5105 got dsc Start(1)
5106 start 1
5107 Feb 07 20:08:22.726 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/1, method: POST, req_id: bf1696ca-0b37-4598-a0ad-a6148b5d389e, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
5108 Replay: started 1, returned:Ok(())
5109 ----------------------------------------------------------------
5110 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5111 States: Active Offline Active
5112 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5113 2208 Acked Write 1 Done Sent Done false
5114 2209 Acked Read 6 Done Sent Done false
5115 2210 Acked Flush 0 Done Sent Done false
5116 2211 Acked Write 10 Done Sent Done false
5117 2212 Acked Read 6 Done Sent Done false
5118 2213 Acked Read 3 Done Sent Done false
5119 2214 Acked Read 2 Done Sent Done false
5120 2215 Acked Read 7 Done Sent Done false
5121 2216 Acked Write 3 Done Sent Done false
5122 2217 Acked Write 10 Done Sent Done false
5123 2218 Acked Write 8 Done Sent Done false
5124 2219 Acked Read 3 Done Sent Done false
5125 2220 Acked Write 5 Done Sent Done false
5126 2221 Acked Write 3 Done Sent Done false
5127 2222 Acked Flush 0 Done Sent Done false
5128 2223 Acked Read 2 Done Sent Done false
5129 2224 Acked Read 10 Done Sent Done false
5130 2225 Acked Flush 0 Done Sent Done false
5131 2226 Acked Read 4 Done Sent Done false
5132 2227 Acked Read 1 Done Sent Done false
5133 2228 Acked Write 7 Done Sent Done false
5134 2229 Acked Read 9 Done Sent Done false
5135 2230 Acked Write 1 Done Sent Done false
5136 2231 Acked Read 7 Done Sent Done false
5137 2232 Acked Write 3 Done Sent Done false
5138 2233 Acked Read 9 Done Sent Done false
5139 2234 Acked Read 10 Done Sent Done false
5140 2235 Acked Write 10 Done Sent Done false
5141 2236 Acked Write 9 Done Sent Done false
5142 2237 Acked Write 8 Done Sent Done false
5143 2238 Acked Flush 0 Done Sent Done false
5144 2239 Acked Write 8 Done Sent Done false
5145 2240 Acked Read 4 Done Sent Done false
5146 2241 Acked Read 6 Done Sent Done false
5147 2242 Acked Read 7 Done Sent Done false
5148 2243 Acked Read 1 Done Sent Done false
5149 2244 Acked Write 3 Done Sent Done false
5150 2245 Acked Read 6 Done Sent Done false
5151 2246 Acked Write 9 Done Sent Done false
5152 2247 Acked Read 10 Done Sent Done false
5153 2248 Acked Read 4 Done Sent Done false
5154 2249 Acked Read 9 Done Sent Done false
5155 2250 Acked Write 7 Done Sent Done false
5156 2251 Acked Write 4 Done Sent Done false
5157 2252 Acked Read 1 Done Sent Done false
5158 2253 Acked Read 4 Done Sent Done false
5159 2254 Acked Write 7 Done Sent Done false
5160 2255 Acked Write 3 Done Sent Done false
5161 2256 Acked Read 9 Done Sent Done false
5162 2257 Acked Read 8 Done Sent Done false
5163 2258 Acked Write 8 Done Sent Done false
5164 2259 Acked Read 6 Done Sent Done false
5165 2260 Acked Write 10 Done Sent Done false
5166 2261 Acked Read 2 Done Sent Done false
5167 2262 Acked Write 6 Done Sent Done false
5168 2263 Acked Read 1 Done Sent Done false
5169 2264 Acked Read 4 Done Sent Done false
5170 2265 Acked Flush 0 Done Sent Done false
5171 2266 Acked Write 5 Done Sent Done false
5172 2267 Acked Flush 0 Done Sent Done false
5173 2268 Acked Read 1 Done Sent Done false
5174 2269 Acked Read 2 Done Sent Done false
5175 2270 Acked Write 4 Done Sent Done false
5176 2271 Acked Write 1 Done Sent Done false
5177 2272 Acked Read 10 Done Sent Done false
5178 2273 Acked Write 8 Done Sent Done false
5179 2274 Acked Write 5 Done Sent Done false
5180 2275 Acked Write 5 Done Sent Done false
5181 2276 Acked Flush 0 Done Sent Done false
5182 2277 Acked Read 4 Done Sent Done false
5183 2278 Acked Read 2 Done Sent Done false
5184 2279 Acked Read 5 Done Sent Done false
5185 2280 Acked Write 8 Done Sent Done false
5186 2281 Acked Flush 0 Done Sent Done false
5187 2282 Acked Write 5 Done Sent Done false
5188 2283 Acked Write 6 Done Sent Done false
5189 2284 Acked Write 10 Done Sent Done false
5190 2285 Acked Write 7 Done Sent Done false
5191 2286 Acked Write 10 Done Sent Done false
5192 2287 Acked Read 8 Done Sent Done false
5193 2288 Acked Flush 0 Done Sent Done false
5194 2289 Acked Flush 0 Done Sent Done false
5195 2290 Acked Write 4 Done Sent Done false
5196 2291 Acked Read 10 Done Sent Done false
5197 2292 Acked Flush 0 Done Sent Done false
5198 2293 Acked Read 5 Done Sent Done false
5199 2294 Acked Read 7 Done Sent Done false
5200 2295 Acked Read 8 Done Sent Done false
5201 2296 Acked Write 1 Done Sent Done false
5202 2297 Acked Write 1 Done Sent Done false
5203 2298 Acked Write 4 Done Sent Done false
5204 2299 Acked Write 9 Done Sent Done false
5205 2300 Acked Write 2 Done Sent Done false
5206 2301 Acked Read 10 Done Sent Done false
5207 2302 Acked Read 2 Done Sent Done false
5208 2303 Acked Flush 0 Done Sent Done false
5209 2304 Acked Read 10 Done Sent Done false
5210 2305 Acked Read 1 Done Sent Done false
5211 2306 Acked Read 10 Done Sent Done false
5212 2307 Acked Read 2 Done Sent Done false
5213 2308 Acked Write 7 Done Sent Done false
5214 2309 Acked Write 5 Done Sent Done false
5215 2310 Acked Read 1 Done Sent Done false
5216 2311 Acked Write 3 Done Sent Done false
5217 2312 Acked Read 3 Done Sent Done false
5218 2313 Acked Write 9 Done Sent Done false
5219 2314 Acked Read 8 Done Sent Done false
5220 2315 Acked Write 10 Done Sent Done false
5221 2316 Acked Read 9 Done Sent Done false
5222 2317 Acked Flush 0 Done Sent Done false
5223 2318 Acked Write 9 Done Sent Done false
5224 2319 Acked Read 9 Done Sent Done false
5225 2320 Acked Read 2 Done Sent Done false
5226 2321 Acked Read 8 Done Sent Done false
5227 2322 Acked Read 5 Done Sent Done false
5228 2323 Acked Write 8 Done Sent Done false
5229 2324 Acked Read 2 Done Sent Done false
5230 2325 Acked Write 4 Done Sent Done false
5231 2326 Acked Write 5 Done Sent Done false
5232 2327 Acked Write 1 Done Sent Done false
5233 2328 Acked Read 9 Done Sent Done false
5234 2329 Acked Read 1 Done Sent Done false
5235 2330 Acked Write 7 Done Sent Done false
5236 2331 Acked Flush 0 Done Sent Done false
5237 2332 Acked Read 2 Done Sent Done false
5238 2333 Acked Write 9 Done Sent Done false
5239 2334 Acked Write 2 Done Sent Done false
5240 2335 Acked Read 2 Done Sent Done false
5241 2336 Acked Read 2 Done Sent Done false
5242 2337 Acked Read 9 Done Sent Done false
5243 2338 Acked Read 3 Done Sent Done false
5244 2339 Acked Write 9 Done Sent Done false
5245 2340 Acked Read 10 Done Sent Done false
5246 2341 Acked Write 4 Done Sent Done false
5247 2342 Acked Read 8 Done Sent Done false
5248 2343 Acked Flush 0 Done Sent Done false
5249 2344 Acked Read 10 Done Sent Done false
5250 2345 Acked Write 8 Done Sent Done false
5251 2346 Acked Read 3 Done Sent Done false
5252 2347 Acked Read 3 Done Sent Done false
5253 2348 Acked Write 3 Done Sent Done false
5254 2349 Acked Write 9 Done Sent Done false
5255 2350 Acked Read 8 Done Sent Done false
5256 2351 Acked Read 4 Done Sent Done false
5257 2352 Acked Write 10 Done Sent Done false
5258 2353 Acked Read 5 Done Sent Done false
5259 2354 Acked Write 6 Done Sent Done false
5260 2355 Acked Read 4 Done Sent Done false
5261 2356 Acked Read 7 Done Sent Done false
5262 2357 Acked Read 7 Done Sent Done false
5263 2358 Acked Read 2 Done Sent Done false
5264 2359 Acked Write 8 Done Sent Done false
5265 2360 Acked Write 2 Done Sent Done false
5266 2361 Acked Read 8 Done Sent Done false
5267 2362 Acked Write 1 Done Sent Done false
5268 2363 Acked Read 6 Done Sent Done false
5269 2364 Acked Read 2 Done Sent Done false
5270 2365 Acked Read 10 Done Sent Done false
5271 2366 Acked Read 8 Done Sent Done false
5272 2367 Acked Read 6 Done Sent Done false
5273 2368 Acked Read 1 Done Sent Done false
5274 2369 Acked Read 4 Done Sent Done false
5275 2370 Acked Write 1 Done Sent Done false
5276 2371 Acked Read 10 Done Sent Done false
5277 2372 Acked Write 5 Done Sent Done false
5278 2373 Acked Write 2 Done Sent Done false
5279 2374 Acked Read 2 Done Sent Done false
5280 2375 Acked Write 3 Done Sent Done false
5281 2376 Acked Write 8 Done Sent Done false
5282 2377 Acked Read 5 Done Sent Done false
5283 2378 Acked Write 4 Done Sent Done false
5284 2379 Acked Read 5 Done Sent Done false
5285 2380 Acked Read 5 Done Sent Done false
5286 2381 Acked Read 3 Done Sent Done false
5287 2382 Acked Write 4 Done Sent Done false
5288 2383 Acked Read 10 Done Sent Done false
5289 2384 Acked Write 1 Done Sent Done false
5290 2385 Acked Read 7 Done Sent Done false
5291 2386 Acked Read 10 Done Sent Done false
5292 2387 Acked Flush 0 Done Sent Done false
5293 2388 Acked Write 8 Done Sent Done false
5294 2389 Acked Read 10 Done Sent Done false
5295 2390 Acked Write 3 Done Sent Done false
5296 2391 Acked Write 7 Done Sent Done false
5297 2392 Acked Flush 0 Done Sent Done false
5298 2393 Acked Read 6 Done Sent Done false
5299 2394 Acked Write 10 Done Sent Done false
5300 2395 Acked Read 2 Done Sent Done false
5301 2396 Acked Read 2 Done Sent Done false
5302 2397 Acked Read 9 Done Sent Done false
5303 2398 Acked Read 6 Done Sent Done false
5304 2399 Acked Write 1 Done Sent Done false
5305 2400 Acked Read 2 Done Sent Done false
5306 2401 Acked Write 10 Done Sent Done false
5307 2402 Acked Read 9 Done Sent Done false
5308 2403 Acked Flush 0 Done Sent Done false
5309 2404 Acked Read 9 Done Sent Done false
5310 2405 Acked Write 3 Done Sent Done false
5311 2406 Acked Read 9 Done Sent Done false
5312 2407 Acked Read 8 Done Sent Done false
5313 2408 Acked Write 6 Done Sent Done false
5314 2409 Acked Read 5 Done Sent Done false
5315 2410 Acked Write 3 Done Sent Done false
5316 2411 Acked Read 3 Done Sent Done false
5317 2412 Acked Read 6 Done Sent Done false
5318 2413 Acked Read 1 Done Sent Done false
5319 2414 Acked Write 10 Done Sent Done false
5320 2415 Acked Read 4 Done Sent Done false
5321 2416 Acked Read 1 Done Sent Done false
5322 2417 Acked Write 5 Done Sent Done false
5323 2418 Acked Write 1 Done Sent Done false
5324 2419 Acked Read 1 Done Sent Done false
5325 2420 Acked Write 10 Done Sent Done false
5326 2421 Acked Write 3 Done Sent Done false
5327 2422 Acked Read 5 Done Sent Done false
5328 2423 Acked Read 4 Done Sent Done false
5329 2424 Acked Write 7 Done Sent Done false
5330 2425 Acked Write 4 Done Sent Done false
5331 2426 Acked Read 1 Done Sent Done false
5332 2427 Acked Write 10 Done Sent Done false
5333 2428 Acked Flush 0 Done Sent Done false
5334 2429 Acked Read 1 Done Sent Done false
5335 2430 Acked Read 5 Done Sent Done false
5336 2431 Acked Write 6 Done Sent Done false
5337 2432 Acked Write 4 Done Sent Done false
5338 2433 Acked Read 1 Done Sent Done false
5339 2434 Acked Read 1 Done Sent Done false
5340 2435 Acked Read 7 Done Sent Done false
5341 2436 Acked Read 8 Done Sent Done false
5342 2437 Acked Write 8 Done Sent Done false
5343 2438 Acked Read 2 Done Sent Done false
5344 2439 Acked Read 3 Done Sent Done false
5345 2440 Acked Read 10 Done Sent Done false
5346 2441 Acked Write 5 Done Sent Done false
5347 2442 Acked Write 9 Done Sent Done false
5348 2443 Acked Write 8 Done Sent Done false
5349 2444 Acked Read 7 Done Sent Done false
5350 2445 Acked Read 6 Done Sent Done false
5351 2446 Acked Read 10 Done Sent Done false
5352 2447 Acked Read 1 Done Sent Done false
5353 2448 Acked Write 5 Done Sent Done false
5354 2449 Acked Write 1 Done Sent Done false
5355 2450 Acked Write 3 Done Sent Done false
5356 2451 Acked Write 8 Done Sent Done false
5357 2452 Acked Read 4 Done Sent Done false
5358 2453 Acked Read 8 Done Sent Done false
5359 2454 Acked Read 6 Done Sent Done false
5360 2455 Acked Write 10 Done Sent Done false
5361 2456 Acked Flush 0 Done Sent Done false
5362 2457 Acked Write 2 Done Sent Done false
5363 2458 Acked Flush 0 Done Sent Done false
5364 2459 Acked Write 6 Done Sent Done false
5365 2460 Acked Read 9 Done Sent Done false
5366 2461 Acked Read 9 Done Sent Done false
5367 2462 Acked Write 8 Done Sent Done false
5368 2463 Acked Read 8 Done Sent Done false
5369 2464 Acked Read 7 Done Sent Done false
5370 2465 Acked Read 3 Done Sent Done false
5371 2466 Acked Write 4 Done Sent Done false
5372 2467 Acked Write 3 Done Sent Done false
5373 2468 Acked Write 1 Done Sent Done false
5374 2469 Acked Read 10 Done Sent Done false
5375 2470 Acked Read 2 Done Sent Done false
5376 2471 Acked Write 6 Done Sent Done false
5377 2472 Acked Read 4 Done Sent Done false
5378 2473 Acked Write 1 Done Sent Done false
5379 2474 Acked Read 3 Done Sent Done false
5380 2475 Acked Write 5 Done Sent Done false
5381 2476 Acked Read 9 Done Sent Done false
5382 2477 Acked Flush 0 Done Sent Done false
5383 2478 Acked Read 4 Done Sent Done false
5384 2479 Acked Read 8 Done Sent Done false
5385 2480 Acked Read 4 Done Sent Done false
5386 2481 Acked Read 2 Done Sent Done false
5387 2482 Acked Read 8 Done Sent Done false
5388 2483 Acked Read 2 Done Sent Done false
5389 2484 Acked Read 4 Done Sent Done false
5390 2485 Acked Read 1 Done Sent Done false
5391 2486 Acked Read 1 Done Sent Done false
5392 2487 Acked Write 7 Done Sent Done false
5393 2488 Acked Read 4 Done Sent Done false
5394 2489 Acked Read 9 Done Sent Done false
5395 2490 Acked Read 2 Done Sent Done false
5396 2491 Acked Write 5 Done Sent Done false
5397 2492 Acked Write 1 Done Sent Done false
5398 2493 Acked Read 5 Done Sent Done false
5399 2494 Acked Read 8 Done Sent Done false
5400 2495 Acked Read 7 Done Sent Done false
5401 2496 Acked Write 2 Done Sent Done false
5402 2497 Acked Flush 0 Done Sent Done false
5403 2498 Acked Write 5 Done Sent Done false
5404 2499 Acked Read 1 Done Sent Done false
5405 2500 Acked Read 2 Done Sent Done false
5406 2501 Acked Read 4 Done Sent Done false
5407 2502 Acked Read 1 Done Sent Done false
5408 2503 Acked Read 4 Done Sent Done false
5409 2504 Acked Read 6 Done Sent Done false
5410 2505 Acked Read 8 Done Sent Done false
5411 2506 Acked Write 2 Done Sent Done false
5412 2507 Acked Read 7 Done Sent Done false
5413 2508 Acked Write 7 Done Sent Done false
5414 STATES DS:0 DS:1 DS:2 TOTAL
5415 Sent 0 301 0 301
5416 Done 301 0 301 602
5417 Skipped 0 0 0 0
5418 Error 0 0 0 0
5419 Last Flush: Some(JobId(2497)) Some(JobId(2207)) Some(JobId(2497))
5420 Downstairs last five completed: 2207 2206 2205 2204 2203
5421 Upstairs last five completed: 2508 2507 2506 2505 2504
5422 CLIENT: Up:0 ds:301 act:2
54232025-02-07T20:08:23.228ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
54242025-02-07T20:08:24.486ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
54252025-02-07T20:08:25.040ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
54262025-02-07T20:08:26.041ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
5427 ----------------------------------------------------------------
5428 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5429 States: Active Offline Active
5430 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5431 2208 Acked Write 1 Done Sent Done false
5432 2209 Acked Read 6 Done Sent Done false
5433 2210 Acked Flush 0 Done Sent Done false
5434 2211 Acked Write 10 Done Sent Done false
5435 2212 Acked Read 6 Done Sent Done false
5436 2213 Acked Read 3 Done Sent Done false
5437 2214 Acked Read 2 Done Sent Done false
5438 2215 Acked Read 7 Done Sent Done false
5439 2216 Acked Write 3 Done Sent Done false
5440 2217 Acked Write 10 Done Sent Done false
5441 2218 Acked Write 8 Done Sent Done false
5442 2219 Acked Read 3 Done Sent Done false
5443 2220 Acked Write 5 Done Sent Done false
5444 2221 Acked Write 3 Done Sent Done false
5445 2222 Acked Flush 0 Done Sent Done false
5446 2223 Acked Read 2 Done Sent Done false
5447 2224 Acked Read 10 Done Sent Done false
5448 2225 Acked Flush 0 Done Sent Done false
5449 2226 Acked Read 4 Done Sent Done false
5450 2227 Acked Read 1 Done Sent Done false
5451 2228 Acked Write 7 Done Sent Done false
5452 2229 Acked Read 9 Done Sent Done false
5453 2230 Acked Write 1 Done Sent Done false
5454 2231 Acked Read 7 Done Sent Done false
5455 2232 Acked Write 3 Done Sent Done false
5456 2233 Acked Read 9 Done Sent Done false
5457 2234 Acked Read 10 Done Sent Done false
5458 2235 Acked Write 10 Done Sent Done false
5459 2236 Acked Write 9 Done Sent Done false
5460 2237 Acked Write 8 Done Sent Done false
5461 2238 Acked Flush 0 Done Sent Done false
5462 2239 Acked Write 8 Done Sent Done false
5463 2240 Acked Read 4 Done Sent Done false
5464 2241 Acked Read 6 Done Sent Done false
5465 2242 Acked Read 7 Done Sent Done false
5466 2243 Acked Read 1 Done Sent Done false
5467 2244 Acked Write 3 Done Sent Done false
5468 2245 Acked Read 6 Done Sent Done false
5469 2246 Acked Write 9 Done Sent Done false
5470 2247 Acked Read 10 Done Sent Done false
5471 2248 Acked Read 4 Done Sent Done false
5472 2249 Acked Read 9 Done Sent Done false
5473 2250 Acked Write 7 Done Sent Done false
5474 2251 Acked Write 4 Done Sent Done false
5475 2252 Acked Read 1 Done Sent Done false
5476 2253 Acked Read 4 Done Sent Done false
5477 2254 Acked Write 7 Done Sent Done false
5478 2255 Acked Write 3 Done Sent Done false
5479 2256 Acked Read 9 Done Sent Done false
5480 2257 Acked Read 8 Done Sent Done false
5481 2258 Acked Write 8 Done Sent Done false
5482 2259 Acked Read 6 Done Sent Done false
5483 2260 Acked Write 10 Done Sent Done false
5484 2261 Acked Read 2 Done Sent Done false
5485 2262 Acked Write 6 Done Sent Done false
5486 2263 Acked Read 1 Done Sent Done false
5487 2264 Acked Read 4 Done Sent Done false
5488 2265 Acked Flush 0 Done Sent Done false
5489 2266 Acked Write 5 Done Sent Done false
5490 2267 Acked Flush 0 Done Sent Done false
5491 2268 Acked Read 1 Done Sent Done false
5492 2269 Acked Read 2 Done Sent Done false
5493 2270 Acked Write 4 Done Sent Done false
5494 2271 Acked Write 1 Done Sent Done false
5495 2272 Acked Read 10 Done Sent Done false
5496 2273 Acked Write 8 Done Sent Done false
5497 2274 Acked Write 5 Done Sent Done false
5498 2275 Acked Write 5 Done Sent Done false
5499 2276 Acked Flush 0 Done Sent Done false
5500 2277 Acked Read 4 Done Sent Done false
5501 2278 Acked Read 2 Done Sent Done false
5502 2279 Acked Read 5 Done Sent Done false
5503 2280 Acked Write 8 Done Sent Done false
5504 2281 Acked Flush 0 Done Sent Done false
5505 2282 Acked Write 5 Done Sent Done false
5506 2283 Acked Write 6 Done Sent Done false
5507 2284 Acked Write 10 Done Sent Done false
5508 2285 Acked Write 7 Done Sent Done false
5509 2286 Acked Write 10 Done Sent Done false
5510 2287 Acked Read 8 Done Sent Done false
5511 2288 Acked Flush 0 Done Sent Done false
5512 2289 Acked Flush 0 Done Sent Done false
5513 2290 Acked Write 4 Done Sent Done false
5514 2291 Acked Read 10 Done Sent Done false
5515 2292 Acked Flush 0 Done Sent Done false
5516 2293 Acked Read 5 Done Sent Done false
5517 2294 Acked Read 7 Done Sent Done false
5518 2295 Acked Read 8 Done Sent Done false
5519 2296 Acked Write 1 Done Sent Done false
5520 2297 Acked Write 1 Done Sent Done false
5521 2298 Acked Write 4 Done Sent Done false
5522 2299 Acked Write 9 Done Sent Done false
5523 2300 Acked Write 2 Done Sent Done false
5524 2301 Acked Read 10 Done Sent Done false
5525 2302 Acked Read 2 Done Sent Done false
5526 2303 Acked Flush 0 Done Sent Done false
5527 2304 Acked Read 10 Done Sent Done false
5528 2305 Acked Read 1 Done Sent Done false
5529 2306 Acked Read 10 Done Sent Done false
5530 2307 Acked Read 2 Done Sent Done false
5531 2308 Acked Write 7 Done Sent Done false
5532 2309 Acked Write 5 Done Sent Done false
5533 2310 Acked Read 1 Done Sent Done false
5534 2311 Acked Write 3 Done Sent Done false
5535 2312 Acked Read 3 Done Sent Done false
5536 2313 Acked Write 9 Done Sent Done false
5537 2314 Acked Read 8 Done Sent Done false
5538 2315 Acked Write 10 Done Sent Done false
5539 2316 Acked Read 9 Done Sent Done false
5540 2317 Acked Flush 0 Done Sent Done false
5541 2318 Acked Write 9 Done Sent Done false
5542 2319 Acked Read 9 Done Sent Done false
5543 2320 Acked Read 2 Done Sent Done false
5544 2321 Acked Read 8 Done Sent Done false
5545 2322 Acked Read 5 Done Sent Done false
5546 2323 Acked Write 8 Done Sent Done false
5547 2324 Acked Read 2 Done Sent Done false
5548 2325 Acked Write 4 Done Sent Done false
5549 2326 Acked Write 5 Done Sent Done false
5550 2327 Acked Write 1 Done Sent Done false
5551 2328 Acked Read 9 Done Sent Done false
5552 2329 Acked Read 1 Done Sent Done false
5553 2330 Acked Write 7 Done Sent Done false
5554 2331 Acked Flush 0 Done Sent Done false
5555 2332 Acked Read 2 Done Sent Done false
5556 2333 Acked Write 9 Done Sent Done false
5557 2334 Acked Write 2 Done Sent Done false
5558 2335 Acked Read 2 Done Sent Done false
5559 2336 Acked Read 2 Done Sent Done false
5560 2337 Acked Read 9 Done Sent Done false
5561 2338 Acked Read 3 Done Sent Done false
5562 2339 Acked Write 9 Done Sent Done false
5563 2340 Acked Read 10 Done Sent Done false
5564 2341 Acked Write 4 Done Sent Done false
5565 2342 Acked Read 8 Done Sent Done false
5566 2343 Acked Flush 0 Done Sent Done false
5567 2344 Acked Read 10 Done Sent Done false
5568 2345 Acked Write 8 Done Sent Done false
5569 2346 Acked Read 3 Done Sent Done false
5570 2347 Acked Read 3 Done Sent Done false
5571 2348 Acked Write 3 Done Sent Done false
5572 2349 Acked Write 9 Done Sent Done false
5573 2350 Acked Read 8 Done Sent Done false
5574 2351 Acked Read 4 Done Sent Done false
5575 2352 Acked Write 10 Done Sent Done false
5576 2353 Acked Read 5 Done Sent Done false
5577 2354 Acked Write 6 Done Sent Done false
5578 2355 Acked Read 4 Done Sent Done false
5579 2356 Acked Read 7 Done Sent Done false
5580 2357 Acked Read 7 Done Sent Done false
5581 2358 Acked Read 2 Done Sent Done false
5582 2359 Acked Write 8 Done Sent Done false
5583 2360 Acked Write 2 Done Sent Done false
5584 2361 Acked Read 8 Done Sent Done false
5585 2362 Acked Write 1 Done Sent Done false
5586 2363 Acked Read 6 Done Sent Done false
5587 2364 Acked Read 2 Done Sent Done false
5588 2365 Acked Read 10 Done Sent Done false
5589 2366 Acked Read 8 Done Sent Done false
5590 2367 Acked Read 6 Done Sent Done false
5591 2368 Acked Read 1 Done Sent Done false
5592 2369 Acked Read 4 Done Sent Done false
5593 2370 Acked Write 1 Done Sent Done false
5594 2371 Acked Read 10 Done Sent Done false
5595 2372 Acked Write 5 Done Sent Done false
5596 2373 Acked Write 2 Done Sent Done false
5597 2374 Acked Read 2 Done Sent Done false
5598 2375 Acked Write 3 Done Sent Done false
5599 2376 Acked Write 8 Done Sent Done false
5600 2377 Acked Read 5 Done Sent Done false
5601 2378 Acked Write 4 Done Sent Done false
5602 2379 Acked Read 5 Done Sent Done false
5603 2380 Acked Read 5 Done Sent Done false
5604 2381 Acked Read 3 Done Sent Done false
5605 2382 Acked Write 4 Done Sent Done false
5606 2383 Acked Read 10 Done Sent Done false
5607 2384 Acked Write 1 Done Sent Done false
5608 2385 Acked Read 7 Done Sent Done false
5609 2386 Acked Read 10 Done Sent Done false
5610 2387 Acked Flush 0 Done Sent Done false
5611 2388 Acked Write 8 Done Sent Done false
5612 2389 Acked Read 10 Done Sent Done false
5613 2390 Acked Write 3 Done Sent Done false
5614 2391 Acked Write 7 Done Sent Done false
5615 2392 Acked Flush 0 Done Sent Done false
5616 2393 Acked Read 6 Done Sent Done false
5617 2394 Acked Write 10 Done Sent Done false
5618 2395 Acked Read 2 Done Sent Done false
5619 2396 Acked Read 2 Done Sent Done false
5620 2397 Acked Read 9 Done Sent Done false
5621 2398 Acked Read 6 Done Sent Done false
5622 2399 Acked Write 1 Done Sent Done false
5623 2400 Acked Read 2 Done Sent Done false
5624 2401 Acked Write 10 Done Sent Done false
5625 2402 Acked Read 9 Done Sent Done false
5626 2403 Acked Flush 0 Done Sent Done false
5627 2404 Acked Read 9 Done Sent Done false
5628 2405 Acked Write 3 Done Sent Done false
5629 2406 Acked Read 9 Done Sent Done false
5630 2407 Acked Read 8 Done Sent Done false
5631 2408 Acked Write 6 Done Sent Done false
5632 2409 Acked Read 5 Done Sent Done false
5633 2410 Acked Write 3 Done Sent Done false
5634 2411 Acked Read 3 Done Sent Done false
5635 2412 Acked Read 6 Done Sent Done false
5636 2413 Acked Read 1 Done Sent Done false
5637 2414 Acked Write 10 Done Sent Done false
5638 2415 Acked Read 4 Done Sent Done false
5639 2416 Acked Read 1 Done Sent Done false
5640 2417 Acked Write 5 Done Sent Done false
5641 2418 Acked Write 1 Done Sent Done false
5642 2419 Acked Read 1 Done Sent Done false
5643 2420 Acked Write 10 Done Sent Done false
5644 2421 Acked Write 3 Done Sent Done false
5645 2422 Acked Read 5 Done Sent Done false
5646 2423 Acked Read 4 Done Sent Done false
5647 2424 Acked Write 7 Done Sent Done false
5648 2425 Acked Write 4 Done Sent Done false
5649 2426 Acked Read 1 Done Sent Done false
5650 2427 Acked Write 10 Done Sent Done false
5651 2428 Acked Flush 0 Done Sent Done false
5652 2429 Acked Read 1 Done Sent Done false
5653 2430 Acked Read 5 Done Sent Done false
5654 2431 Acked Write 6 Done Sent Done false
5655 2432 Acked Write 4 Done Sent Done false
5656 2433 Acked Read 1 Done Sent Done false
5657 2434 Acked Read 1 Done Sent Done false
5658 2435 Acked Read 7 Done Sent Done false
5659 2436 Acked Read 8 Done Sent Done false
5660 2437 Acked Write 8 Done Sent Done false
5661 2438 Acked Read 2 Done Sent Done false
5662 2439 Acked Read 3 Done Sent Done false
5663 2440 Acked Read 10 Done Sent Done false
5664 2441 Acked Write 5 Done Sent Done false
5665 2442 Acked Write 9 Done Sent Done false
5666 2443 Acked Write 8 Done Sent Done false
5667 2444 Acked Read 7 Done Sent Done false
5668 2445 Acked Read 6 Done Sent Done false
5669 2446 Acked Read 10 Done Sent Done false
5670 2447 Acked Read 1 Done Sent Done false
5671 2448 Acked Write 5 Done Sent Done false
5672 2449 Acked Write 1 Done Sent Done false
5673 2450 Acked Write 3 Done Sent Done false
5674 2451 Acked Write 8 Done Sent Done false
5675 2452 Acked Read 4 Done Sent Done false
5676 2453 Acked Read 8 Done Sent Done false
5677 2454 Acked Read 6 Done Sent Done false
5678 2455 Acked Write 10 Done Sent Done false
5679 2456 Acked Flush 0 Done Sent Done false
5680 2457 Acked Write 2 Done Sent Done false
5681 2458 Acked Flush 0 Done Sent Done false
5682 2459 Acked Write 6 Done Sent Done false
5683 2460 Acked Read 9 Done Sent Done false
5684 2461 Acked Read 9 Done Sent Done false
5685 2462 Acked Write 8 Done Sent Done false
5686 2463 Acked Read 8 Done Sent Done false
5687 2464 Acked Read 7 Done Sent Done false
5688 2465 Acked Read 3 Done Sent Done false
5689 2466 Acked Write 4 Done Sent Done false
5690 2467 Acked Write 3 Done Sent Done false
5691 2468 Acked Write 1 Done Sent Done false
5692 2469 Acked Read 10 Done Sent Done false
5693 2470 Acked Read 2 Done Sent Done false
5694 2471 Acked Write 6 Done Sent Done false
5695 2472 Acked Read 4 Done Sent Done false
5696 2473 Acked Write 1 Done Sent Done false
5697 2474 Acked Read 3 Done Sent Done false
5698 2475 Acked Write 5 Done Sent Done false
5699 2476 Acked Read 9 Done Sent Done false
5700 2477 Acked Flush 0 Done Sent Done false
5701 2478 Acked Read 4 Done Sent Done false
5702 2479 Acked Read 8 Done Sent Done false
5703 2480 Acked Read 4 Done Sent Done false
5704 2481 Acked Read 2 Done Sent Done false
5705 2482 Acked Read 8 Done Sent Done false
5706 2483 Acked Read 2 Done Sent Done false
5707 2484 Acked Read 4 Done Sent Done false
5708 2485 Acked Read 1 Done Sent Done false
5709 2486 Acked Read 1 Done Sent Done false
5710 2487 Acked Write 7 Done Sent Done false
5711 2488 Acked Read 4 Done Sent Done false
5712 2489 Acked Read 9 Done Sent Done false
5713 2490 Acked Read 2 Done Sent Done false
5714 2491 Acked Write 5 Done Sent Done false
5715 2492 Acked Write 1 Done Sent Done false
5716 2493 Acked Read 5 Done Sent Done false
5717 2494 Acked Read 8 Done Sent Done false
5718 2495 Acked Read 7 Done Sent Done false
5719 2496 Acked Write 2 Done Sent Done false
5720 2497 Acked Flush 0 Done Sent Done false
5721 2498 Acked Write 5 Done Sent Done false
5722 2499 Acked Read 1 Done Sent Done false
5723 2500 Acked Read 2 Done Sent Done false
5724 2501 Acked Read 4 Done Sent Done false
5725 2502 Acked Read 1 Done Sent Done false
5726 2503 Acked Read 4 Done Sent Done false
5727 2504 Acked Read 6 Done Sent Done false
5728 2505 Acked Read 8 Done Sent Done false
5729 2506 Acked Write 2 Done Sent Done false
5730 2507 Acked Read 7 Done Sent Done false
5731 2508 Acked Write 7 Done Sent Done false
5732 STATES DS:0 DS:1 DS:2 TOTAL
5733 Sent 0 301 0 301
5734 Done 301 0 301 602
5735 Skipped 0 0 0 0
5736 Error 0 0 0 0
5737 Last Flush: Some(JobId(2497)) Some(JobId(2207)) Some(JobId(2497))
5738 Downstairs last five completed: 2207 2206 2205 2204 2203
5739 Upstairs last five completed: 2508 2507 2506 2505 2504
5740 CLIENT: Up:0 ds:301 act:2
57412025-02-07T20:08:27.231ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
57422025-02-07T20:08:28.045ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
57432025-02-07T20:08:28.481ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
57442025-02-07T20:08:28.481ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
57452025-02-07T20:08:28.481ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
57462025-02-07T20:08:28.481ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
57472025-02-07T20:08:28.481ZINFOcrucible: client task is exiting = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
5748 [8820] got start action
5749 [8820] I am going to restart
5750 Starting downstairs at port 8820
5751 Make output file at "/tmp/dsc/downstairs-8820.txt"
5752 [8820][1] reports Starting
5753 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1214)
5754 [8820][1] reports Running
57552025-02-07T20:08:29.487ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
57562025-02-07T20:08:30.048ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
5757 ----------------------------------------------------------------
5758 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5759 States: Active Offline Active
5760 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5761 2208 Acked Write 1 Done Sent Done false
5762 2209 Acked Read 6 Done Sent Done false
5763 2210 Acked Flush 0 Done Sent Done false
5764 2211 Acked Write 10 Done Sent Done false
5765 2212 Acked Read 6 Done Sent Done false
5766 2213 Acked Read 3 Done Sent Done false
5767 2214 Acked Read 2 Done Sent Done false
5768 2215 Acked Read 7 Done Sent Done false
5769 2216 Acked Write 3 Done Sent Done false
5770 2217 Acked Write 10 Done Sent Done false
5771 2218 Acked Write 8 Done Sent Done false
5772 2219 Acked Read 3 Done Sent Done false
5773 2220 Acked Write 5 Done Sent Done false
5774 2221 Acked Write 3 Done Sent Done false
5775 2222 Acked Flush 0 Done Sent Done false
5776 2223 Acked Read 2 Done Sent Done false
5777 2224 Acked Read 10 Done Sent Done false
5778 2225 Acked Flush 0 Done Sent Done false
5779 2226 Acked Read 4 Done Sent Done false
5780 2227 Acked Read 1 Done Sent Done false
5781 2228 Acked Write 7 Done Sent Done false
5782 2229 Acked Read 9 Done Sent Done false
5783 2230 Acked Write 1 Done Sent Done false
5784 2231 Acked Read 7 Done Sent Done false
5785 2232 Acked Write 3 Done Sent Done false
5786 2233 Acked Read 9 Done Sent Done false
5787 2234 Acked Read 10 Done Sent Done false
5788 2235 Acked Write 10 Done Sent Done false
5789 2236 Acked Write 9 Done Sent Done false
5790 2237 Acked Write 8 Done Sent Done false
5791 2238 Acked Flush 0 Done Sent Done false
5792 2239 Acked Write 8 Done Sent Done false
5793 2240 Acked Read 4 Done Sent Done false
5794 2241 Acked Read 6 Done Sent Done false
5795 2242 Acked Read 7 Done Sent Done false
5796 2243 Acked Read 1 Done Sent Done false
5797 2244 Acked Write 3 Done Sent Done false
5798 2245 Acked Read 6 Done Sent Done false
5799 2246 Acked Write 9 Done Sent Done false
5800 2247 Acked Read 10 Done Sent Done false
5801 2248 Acked Read 4 Done Sent Done false
5802 2249 Acked Read 9 Done Sent Done false
5803 2250 Acked Write 7 Done Sent Done false
5804 2251 Acked Write 4 Done Sent Done false
5805 2252 Acked Read 1 Done Sent Done false
5806 2253 Acked Read 4 Done Sent Done false
5807 2254 Acked Write 7 Done Sent Done false
5808 2255 Acked Write 3 Done Sent Done false
5809 2256 Acked Read 9 Done Sent Done false
5810 2257 Acked Read 8 Done Sent Done false
5811 2258 Acked Write 8 Done Sent Done false
5812 2259 Acked Read 6 Done Sent Done false
5813 2260 Acked Write 10 Done Sent Done false
5814 2261 Acked Read 2 Done Sent Done false
5815 2262 Acked Write 6 Done Sent Done false
5816 2263 Acked Read 1 Done Sent Done false
5817 2264 Acked Read 4 Done Sent Done false
5818 2265 Acked Flush 0 Done Sent Done false
5819 2266 Acked Write 5 Done Sent Done false
5820 2267 Acked Flush 0 Done Sent Done false
5821 2268 Acked Read 1 Done Sent Done false
5822 2269 Acked Read 2 Done Sent Done false
5823 2270 Acked Write 4 Done Sent Done false
5824 2271 Acked Write 1 Done Sent Done false
5825 2272 Acked Read 10 Done Sent Done false
5826 2273 Acked Write 8 Done Sent Done false
5827 2274 Acked Write 5 Done Sent Done false
5828 2275 Acked Write 5 Done Sent Done false
5829 2276 Acked Flush 0 Done Sent Done false
5830 2277 Acked Read 4 Done Sent Done false
5831 2278 Acked Read 2 Done Sent Done false
5832 2279 Acked Read 5 Done Sent Done false
5833 2280 Acked Write 8 Done Sent Done false
5834 2281 Acked Flush 0 Done Sent Done false
5835 2282 Acked Write 5 Done Sent Done false
5836 2283 Acked Write 6 Done Sent Done false
5837 2284 Acked Write 10 Done Sent Done false
5838 2285 Acked Write 7 Done Sent Done false
5839 2286 Acked Write 10 Done Sent Done false
5840 2287 Acked Read 8 Done Sent Done false
5841 2288 Acked Flush 0 Done Sent Done false
5842 2289 Acked Flush 0 Done Sent Done false
5843 2290 Acked Write 4 Done Sent Done false
5844 2291 Acked Read 10 Done Sent Done false
5845 2292 Acked Flush 0 Done Sent Done false
5846 2293 Acked Read 5 Done Sent Done false
5847 2294 Acked Read 7 Done Sent Done false
5848 2295 Acked Read 8 Done Sent Done false
5849 2296 Acked Write 1 Done Sent Done false
5850 2297 Acked Write 1 Done Sent Done false
5851 2298 Acked Write 4 Done Sent Done false
5852 2299 Acked Write 9 Done Sent Done false
5853 2300 Acked Write 2 Done Sent Done false
5854 2301 Acked Read 10 Done Sent Done false
5855 2302 Acked Read 2 Done Sent Done false
5856 2303 Acked Flush 0 Done Sent Done false
5857 2304 Acked Read 10 Done Sent Done false
5858 2305 Acked Read 1 Done Sent Done false
5859 2306 Acked Read 10 Done Sent Done false
5860 2307 Acked Read 2 Done Sent Done false
5861 2308 Acked Write 7 Done Sent Done false
5862 2309 Acked Write 5 Done Sent Done false
5863 2310 Acked Read 1 Done Sent Done false
5864 2311 Acked Write 3 Done Sent Done false
5865 2312 Acked Read 3 Done Sent Done false
5866 2313 Acked Write 9 Done Sent Done false
5867 2314 Acked Read 8 Done Sent Done false
5868 2315 Acked Write 10 Done Sent Done false
5869 2316 Acked Read 9 Done Sent Done false
5870 2317 Acked Flush 0 Done Sent Done false
5871 2318 Acked Write 9 Done Sent Done false
5872 2319 Acked Read 9 Done Sent Done false
5873 2320 Acked Read 2 Done Sent Done false
5874 2321 Acked Read 8 Done Sent Done false
5875 2322 Acked Read 5 Done Sent Done false
5876 2323 Acked Write 8 Done Sent Done false
5877 2324 Acked Read 2 Done Sent Done false
5878 2325 Acked Write 4 Done Sent Done false
5879 2326 Acked Write 5 Done Sent Done false
5880 2327 Acked Write 1 Done Sent Done false
5881 2328 Acked Read 9 Done Sent Done false
5882 2329 Acked Read 1 Done Sent Done false
5883 2330 Acked Write 7 Done Sent Done false
5884 2331 Acked Flush 0 Done Sent Done false
5885 2332 Acked Read 2 Done Sent Done false
5886 2333 Acked Write 9 Done Sent Done false
5887 2334 Acked Write 2 Done Sent Done false
5888 2335 Acked Read 2 Done Sent Done false
5889 2336 Acked Read 2 Done Sent Done false
5890 2337 Acked Read 9 Done Sent Done false
5891 2338 Acked Read 3 Done Sent Done false
5892 2339 Acked Write 9 Done Sent Done false
5893 2340 Acked Read 10 Done Sent Done false
5894 2341 Acked Write 4 Done Sent Done false
5895 2342 Acked Read 8 Done Sent Done false
5896 2343 Acked Flush 0 Done Sent Done false
5897 2344 Acked Read 10 Done Sent Done false
5898 2345 Acked Write 8 Done Sent Done false
5899 2346 Acked Read 3 Done Sent Done false
5900 2347 Acked Read 3 Done Sent Done false
5901 2348 Acked Write 3 Done Sent Done false
5902 2349 Acked Write 9 Done Sent Done false
5903 2350 Acked Read 8 Done Sent Done false
5904 2351 Acked Read 4 Done Sent Done false
5905 2352 Acked Write 10 Done Sent Done false
5906 2353 Acked Read 5 Done Sent Done false
5907 2354 Acked Write 6 Done Sent Done false
5908 2355 Acked Read 4 Done Sent Done false
5909 2356 Acked Read 7 Done Sent Done false
5910 2357 Acked Read 7 Done Sent Done false
5911 2358 Acked Read 2 Done Sent Done false
5912 2359 Acked Write 8 Done Sent Done false
5913 2360 Acked Write 2 Done Sent Done false
5914 2361 Acked Read 8 Done Sent Done false
5915 2362 Acked Write 1 Done Sent Done false
5916 2363 Acked Read 6 Done Sent Done false
5917 2364 Acked Read 2 Done Sent Done false
5918 2365 Acked Read 10 Done Sent Done false
5919 2366 Acked Read 8 Done Sent Done false
5920 2367 Acked Read 6 Done Sent Done false
5921 2368 Acked Read 1 Done Sent Done false
5922 2369 Acked Read 4 Done Sent Done false
5923 2370 Acked Write 1 Done Sent Done false
5924 2371 Acked Read 10 Done Sent Done false
5925 2372 Acked Write 5 Done Sent Done false
5926 2373 Acked Write 2 Done Sent Done false
5927 2374 Acked Read 2 Done Sent Done false
5928 2375 Acked Write 3 Done Sent Done false
5929 2376 Acked Write 8 Done Sent Done false
5930 2377 Acked Read 5 Done Sent Done false
5931 2378 Acked Write 4 Done Sent Done false
5932 2379 Acked Read 5 Done Sent Done false
5933 2380 Acked Read 5 Done Sent Done false
5934 2381 Acked Read 3 Done Sent Done false
5935 2382 Acked Write 4 Done Sent Done false
5936 2383 Acked Read 10 Done Sent Done false
5937 2384 Acked Write 1 Done Sent Done false
5938 2385 Acked Read 7 Done Sent Done false
5939 2386 Acked Read 10 Done Sent Done false
5940 2387 Acked Flush 0 Done Sent Done false
5941 2388 Acked Write 8 Done Sent Done false
5942 2389 Acked Read 10 Done Sent Done false
5943 2390 Acked Write 3 Done Sent Done false
5944 2391 Acked Write 7 Done Sent Done false
5945 2392 Acked Flush 0 Done Sent Done false
5946 2393 Acked Read 6 Done Sent Done false
5947 2394 Acked Write 10 Done Sent Done false
5948 2395 Acked Read 2 Done Sent Done false
5949 2396 Acked Read 2 Done Sent Done false
5950 2397 Acked Read 9 Done Sent Done false
5951 2398 Acked Read 6 Done Sent Done false
5952 2399 Acked Write 1 Done Sent Done false
5953 2400 Acked Read 2 Done Sent Done false
5954 2401 Acked Write 10 Done Sent Done false
5955 2402 Acked Read 9 Done Sent Done false
5956 2403 Acked Flush 0 Done Sent Done false
5957 2404 Acked Read 9 Done Sent Done false
5958 2405 Acked Write 3 Done Sent Done false
5959 2406 Acked Read 9 Done Sent Done false
5960 2407 Acked Read 8 Done Sent Done false
5961 2408 Acked Write 6 Done Sent Done false
5962 2409 Acked Read 5 Done Sent Done false
5963 2410 Acked Write 3 Done Sent Done false
5964 2411 Acked Read 3 Done Sent Done false
5965 2412 Acked Read 6 Done Sent Done false
5966 2413 Acked Read 1 Done Sent Done false
5967 2414 Acked Write 10 Done Sent Done false
5968 2415 Acked Read 4 Done Sent Done false
5969 2416 Acked Read 1 Done Sent Done false
5970 2417 Acked Write 5 Done Sent Done false
5971 2418 Acked Write 1 Done Sent Done false
5972 2419 Acked Read 1 Done Sent Done false
5973 2420 Acked Write 10 Done Sent Done false
5974 2421 Acked Write 3 Done Sent Done false
5975 2422 Acked Read 5 Done Sent Done false
5976 2423 Acked Read 4 Done Sent Done false
5977 2424 Acked Write 7 Done Sent Done false
5978 2425 Acked Write 4 Done Sent Done false
5979 2426 Acked Read 1 Done Sent Done false
5980 2427 Acked Write 10 Done Sent Done false
5981 2428 Acked Flush 0 Done Sent Done false
5982 2429 Acked Read 1 Done Sent Done false
5983 2430 Acked Read 5 Done Sent Done false
5984 2431 Acked Write 6 Done Sent Done false
5985 2432 Acked Write 4 Done Sent Done false
5986 2433 Acked Read 1 Done Sent Done false
5987 2434 Acked Read 1 Done Sent Done false
5988 2435 Acked Read 7 Done Sent Done false
5989 2436 Acked Read 8 Done Sent Done false
5990 2437 Acked Write 8 Done Sent Done false
5991 2438 Acked Read 2 Done Sent Done false
5992 2439 Acked Read 3 Done Sent Done false
5993 2440 Acked Read 10 Done Sent Done false
5994 2441 Acked Write 5 Done Sent Done false
5995 2442 Acked Write 9 Done Sent Done false
5996 2443 Acked Write 8 Done Sent Done false
5997 2444 Acked Read 7 Done Sent Done false
5998 2445 Acked Read 6 Done Sent Done false
5999 2446 Acked Read 10 Done Sent Done false
6000 2447 Acked Read 1 Done Sent Done false
6001 2448 Acked Write 5 Done Sent Done false
6002 2449 Acked Write 1 Done Sent Done false
6003 2450 Acked Write 3 Done Sent Done false
6004 2451 Acked Write 8 Done Sent Done false
6005 2452 Acked Read 4 Done Sent Done false
6006 2453 Acked Read 8 Done Sent Done false
6007 2454 Acked Read 6 Done Sent Done false
6008 2455 Acked Write 10 Done Sent Done false
6009 2456 Acked Flush 0 Done Sent Done false
6010 2457 Acked Write 2 Done Sent Done false
6011 2458 Acked Flush 0 Done Sent Done false
6012 2459 Acked Write 6 Done Sent Done false
6013 2460 Acked Read 9 Done Sent Done false
6014 2461 Acked Read 9 Done Sent Done false
6015 2462 Acked Write 8 Done Sent Done false
6016 2463 Acked Read 8 Done Sent Done false
6017 2464 Acked Read 7 Done Sent Done false
6018 2465 Acked Read 3 Done Sent Done false
6019 2466 Acked Write 4 Done Sent Done false
6020 2467 Acked Write 3 Done Sent Done false
6021 2468 Acked Write 1 Done Sent Done false
6022 2469 Acked Read 10 Done Sent Done false
6023 2470 Acked Read 2 Done Sent Done false
6024 2471 Acked Write 6 Done Sent Done false
6025 2472 Acked Read 4 Done Sent Done false
6026 2473 Acked Write 1 Done Sent Done false
6027 2474 Acked Read 3 Done Sent Done false
6028 2475 Acked Write 5 Done Sent Done false
6029 2476 Acked Read 9 Done Sent Done false
6030 2477 Acked Flush 0 Done Sent Done false
6031 2478 Acked Read 4 Done Sent Done false
6032 2479 Acked Read 8 Done Sent Done false
6033 2480 Acked Read 4 Done Sent Done false
6034 2481 Acked Read 2 Done Sent Done false
6035 2482 Acked Read 8 Done Sent Done false
6036 2483 Acked Read 2 Done Sent Done false
6037 2484 Acked Read 4 Done Sent Done false
6038 2485 Acked Read 1 Done Sent Done false
6039 2486 Acked Read 1 Done Sent Done false
6040 2487 Acked Write 7 Done Sent Done false
6041 2488 Acked Read 4 Done Sent Done false
6042 2489 Acked Read 9 Done Sent Done false
6043 2490 Acked Read 2 Done Sent Done false
6044 2491 Acked Write 5 Done Sent Done false
6045 2492 Acked Write 1 Done Sent Done false
6046 2493 Acked Read 5 Done Sent Done false
6047 2494 Acked Read 8 Done Sent Done false
6048 2495 Acked Read 7 Done Sent Done false
6049 2496 Acked Write 2 Done Sent Done false
6050 2497 Acked Flush 0 Done Sent Done false
6051 2498 Acked Write 5 Done Sent Done false
6052 2499 Acked Read 1 Done Sent Done false
6053 2500 Acked Read 2 Done Sent Done false
6054 2501 Acked Read 4 Done Sent Done false
6055 2502 Acked Read 1 Done Sent Done false
6056 2503 Acked Read 4 Done Sent Done false
6057 2504 Acked Read 6 Done Sent Done false
6058 2505 Acked Read 8 Done Sent Done false
6059 2506 Acked Write 2 Done Sent Done false
6060 2507 Acked Read 7 Done Sent Done false
6061 2508 Acked Write 7 Done Sent Done false
6062 STATES DS:0 DS:1 DS:2 TOTAL
6063 Sent 0 301 0 301
6064 Done 301 0 301 602
6065 Skipped 0 0 0 0
6066 Error 0 0 0 0
6067 Last Flush: Some(JobId(2497)) Some(JobId(2207)) Some(JobId(2497))
6068 Downstairs last five completed: 2207 2206 2205 2204 2203
6069 Upstairs last five completed: 2508 2507 2506 2505 2504
6070 CLIENT: Up:0 ds:301 act:2
60712025-02-07T20:08:32.052ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
60722025-02-07T20:08:33.054ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
60732025-02-07T20:08:33.555ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
60742025-02-07T20:08:34.488ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
6075 ----------------------------------------------------------------
6076 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6077 States: Active Offline Active
6078 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6079 2208 Acked Write 1 Done Sent Done false
6080 2209 Acked Read 6 Done Sent Done false
6081 2210 Acked Flush 0 Done Sent Done false
6082 2211 Acked Write 10 Done Sent Done false
6083 2212 Acked Read 6 Done Sent Done false
6084 2213 Acked Read 3 Done Sent Done false
6085 2214 Acked Read 2 Done Sent Done false
6086 2215 Acked Read 7 Done Sent Done false
6087 2216 Acked Write 3 Done Sent Done false
6088 2217 Acked Write 10 Done Sent Done false
6089 2218 Acked Write 8 Done Sent Done false
6090 2219 Acked Read 3 Done Sent Done false
6091 2220 Acked Write 5 Done Sent Done false
6092 2221 Acked Write 3 Done Sent Done false
6093 2222 Acked Flush 0 Done Sent Done false
6094 2223 Acked Read 2 Done Sent Done false
6095 2224 Acked Read 10 Done Sent Done false
6096 2225 Acked Flush 0 Done Sent Done false
6097 2226 Acked Read 4 Done Sent Done false
6098 2227 Acked Read 1 Done Sent Done false
6099 2228 Acked Write 7 Done Sent Done false
6100 2229 Acked Read 9 Done Sent Done false
6101 2230 Acked Write 1 Done Sent Done false
6102 2231 Acked Read 7 Done Sent Done false
6103 2232 Acked Write 3 Done Sent Done false
6104 2233 Acked Read 9 Done Sent Done false
6105 2234 Acked Read 10 Done Sent Done false
6106 2235 Acked Write 10 Done Sent Done false
6107 2236 Acked Write 9 Done Sent Done false
6108 2237 Acked Write 8 Done Sent Done false
6109 2238 Acked Flush 0 Done Sent Done false
6110 2239 Acked Write 8 Done Sent Done false
6111 2240 Acked Read 4 Done Sent Done false
6112 2241 Acked Read 6 Done Sent Done false
6113 2242 Acked Read 7 Done Sent Done false
6114 2243 Acked Read 1 Done Sent Done false
6115 2244 Acked Write 3 Done Sent Done false
6116 2245 Acked Read 6 Done Sent Done false
6117 2246 Acked Write 9 Done Sent Done false
6118 2247 Acked Read 10 Done Sent Done false
6119 2248 Acked Read 4 Done Sent Done false
6120 2249 Acked Read 9 Done Sent Done false
6121 2250 Acked Write 7 Done Sent Done false
6122 2251 Acked Write 4 Done Sent Done false
6123 2252 Acked Read 1 Done Sent Done false
6124 2253 Acked Read 4 Done Sent Done false
6125 2254 Acked Write 7 Done Sent Done false
6126 2255 Acked Write 3 Done Sent Done false
6127 2256 Acked Read 9 Done Sent Done false
6128 2257 Acked Read 8 Done Sent Done false
6129 2258 Acked Write 8 Done Sent Done false
6130 2259 Acked Read 6 Done Sent Done false
6131 2260 Acked Write 10 Done Sent Done false
6132 2261 Acked Read 2 Done Sent Done false
6133 2262 Acked Write 6 Done Sent Done false
6134 2263 Acked Read 1 Done Sent Done false
6135 2264 Acked Read 4 Done Sent Done false
6136 2265 Acked Flush 0 Done Sent Done false
6137 2266 Acked Write 5 Done Sent Done false
6138 2267 Acked Flush 0 Done Sent Done false
6139 2268 Acked Read 1 Done Sent Done false
6140 2269 Acked Read 2 Done Sent Done false
6141 2270 Acked Write 4 Done Sent Done false
6142 2271 Acked Write 1 Done Sent Done false
6143 2272 Acked Read 10 Done Sent Done false
6144 2273 Acked Write 8 Done Sent Done false
6145 2274 Acked Write 5 Done Sent Done false
6146 2275 Acked Write 5 Done Sent Done false
6147 2276 Acked Flush 0 Done Sent Done false
6148 2277 Acked Read 4 Done Sent Done false
6149 2278 Acked Read 2 Done Sent Done false
6150 2279 Acked Read 5 Done Sent Done false
6151 2280 Acked Write 8 Done Sent Done false
6152 2281 Acked Flush 0 Done Sent Done false
6153 2282 Acked Write 5 Done Sent Done false
6154 2283 Acked Write 6 Done Sent Done false
6155 2284 Acked Write 10 Done Sent Done false
6156 2285 Acked Write 7 Done Sent Done false
6157 2286 Acked Write 10 Done Sent Done false
6158 2287 Acked Read 8 Done Sent Done false
6159 2288 Acked Flush 0 Done Sent Done false
6160 2289 Acked Flush 0 Done Sent Done false
6161 2290 Acked Write 4 Done Sent Done false
6162 2291 Acked Read 10 Done Sent Done false
6163 2292 Acked Flush 0 Done Sent Done false
6164 2293 Acked Read 5 Done Sent Done false
6165 2294 Acked Read 7 Done Sent Done false
6166 2295 Acked Read 8 Done Sent Done false
6167 2296 Acked Write 1 Done Sent Done false
6168 2297 Acked Write 1 Done Sent Done false
6169 2298 Acked Write 4 Done Sent Done false
6170 2299 Acked Write 9 Done Sent Done false
6171 2300 Acked Write 2 Done Sent Done false
6172 2301 Acked Read 10 Done Sent Done false
6173 2302 Acked Read 2 Done Sent Done false
6174 2303 Acked Flush 0 Done Sent Done false
6175 2304 Acked Read 10 Done Sent Done false
6176 2305 Acked Read 1 Done Sent Done false
6177 2306 Acked Read 10 Done Sent Done false
6178 2307 Acked Read 2 Done Sent Done false
6179 2308 Acked Write 7 Done Sent Done false
6180 2309 Acked Write 5 Done Sent Done false
6181 2310 Acked Read 1 Done Sent Done false
6182 2311 Acked Write 3 Done Sent Done false
6183 2312 Acked Read 3 Done Sent Done false
6184 2313 Acked Write 9 Done Sent Done false
6185 2314 Acked Read 8 Done Sent Done false
6186 2315 Acked Write 10 Done Sent Done false
6187 2316 Acked Read 9 Done Sent Done false
6188 2317 Acked Flush 0 Done Sent Done false
6189 2318 Acked Write 9 Done Sent Done false
6190 2319 Acked Read 9 Done Sent Done false
6191 2320 Acked Read 2 Done Sent Done false
6192 2321 Acked Read 8 Done Sent Done false
6193 2322 Acked Read 5 Done Sent Done false
6194 2323 Acked Write 8 Done Sent Done false
6195 2324 Acked Read 2 Done Sent Done false
6196 2325 Acked Write 4 Done Sent Done false
6197 2326 Acked Write 5 Done Sent Done false
6198 2327 Acked Write 1 Done Sent Done false
6199 2328 Acked Read 9 Done Sent Done false
6200 2329 Acked Read 1 Done Sent Done false
6201 2330 Acked Write 7 Done Sent Done false
6202 2331 Acked Flush 0 Done Sent Done false
6203 2332 Acked Read 2 Done Sent Done false
6204 2333 Acked Write 9 Done Sent Done false
6205 2334 Acked Write 2 Done Sent Done false
6206 2335 Acked Read 2 Done Sent Done false
6207 2336 Acked Read 2 Done Sent Done false
6208 2337 Acked Read 9 Done Sent Done false
6209 2338 Acked Read 3 Done Sent Done false
6210 2339 Acked Write 9 Done Sent Done false
6211 2340 Acked Read 10 Done Sent Done false
6212 2341 Acked Write 4 Done Sent Done false
6213 2342 Acked Read 8 Done Sent Done false
6214 2343 Acked Flush 0 Done Sent Done false
6215 2344 Acked Read 10 Done Sent Done false
6216 2345 Acked Write 8 Done Sent Done false
6217 2346 Acked Read 3 Done Sent Done false
6218 2347 Acked Read 3 Done Sent Done false
6219 2348 Acked Write 3 Done Sent Done false
6220 2349 Acked Write 9 Done Sent Done false
6221 2350 Acked Read 8 Done Sent Done false
6222 2351 Acked Read 4 Done Sent Done false
6223 2352 Acked Write 10 Done Sent Done false
6224 2353 Acked Read 5 Done Sent Done false
6225 2354 Acked Write 6 Done Sent Done false
6226 2355 Acked Read 4 Done Sent Done false
6227 2356 Acked Read 7 Done Sent Done false
6228 2357 Acked Read 7 Done Sent Done false
6229 2358 Acked Read 2 Done Sent Done false
6230 2359 Acked Write 8 Done Sent Done false
6231 2360 Acked Write 2 Done Sent Done false
6232 2361 Acked Read 8 Done Sent Done false
6233 2362 Acked Write 1 Done Sent Done false
6234 2363 Acked Read 6 Done Sent Done false
6235 2364 Acked Read 2 Done Sent Done false
6236 2365 Acked Read 10 Done Sent Done false
6237 2366 Acked Read 8 Done Sent Done false
6238 2367 Acked Read 6 Done Sent Done false
6239 2368 Acked Read 1 Done Sent Done false
6240 2369 Acked Read 4 Done Sent Done false
6241 2370 Acked Write 1 Done Sent Done false
6242 2371 Acked Read 10 Done Sent Done false
6243 2372 Acked Write 5 Done Sent Done false
6244 2373 Acked Write 2 Done Sent Done false
6245 2374 Acked Read 2 Done Sent Done false
6246 2375 Acked Write 3 Done Sent Done false
6247 2376 Acked Write 8 Done Sent Done false
6248 2377 Acked Read 5 Done Sent Done false
6249 2378 Acked Write 4 Done Sent Done false
6250 2379 Acked Read 5 Done Sent Done false
6251 2380 Acked Read 5 Done Sent Done false
6252 2381 Acked Read 3 Done Sent Done false
6253 2382 Acked Write 4 Done Sent Done false
6254 2383 Acked Read 10 Done Sent Done false
6255 2384 Acked Write 1 Done Sent Done false
6256 2385 Acked Read 7 Done Sent Done false
6257 2386 Acked Read 10 Done Sent Done false
6258 2387 Acked Flush 0 Done Sent Done false
6259 2388 Acked Write 8 Done Sent Done false
6260 2389 Acked Read 10 Done Sent Done false
6261 2390 Acked Write 3 Done Sent Done false
6262 2391 Acked Write 7 Done Sent Done false
6263 2392 Acked Flush 0 Done Sent Done false
6264 2393 Acked Read 6 Done Sent Done false
6265 2394 Acked Write 10 Done Sent Done false
6266 2395 Acked Read 2 Done Sent Done false
6267 2396 Acked Read 2 Done Sent Done false
6268 2397 Acked Read 9 Done Sent Done false
6269 2398 Acked Read 6 Done Sent Done false
6270 2399 Acked Write 1 Done Sent Done false
6271 2400 Acked Read 2 Done Sent Done false
6272 2401 Acked Write 10 Done Sent Done false
6273 2402 Acked Read 9 Done Sent Done false
6274 2403 Acked Flush 0 Done Sent Done false
6275 2404 Acked Read 9 Done Sent Done false
6276 2405 Acked Write 3 Done Sent Done false
6277 2406 Acked Read 9 Done Sent Done false
6278 2407 Acked Read 8 Done Sent Done false
6279 2408 Acked Write 6 Done Sent Done false
6280 2409 Acked Read 5 Done Sent Done false
6281 2410 Acked Write 3 Done Sent Done false
6282 2411 Acked Read 3 Done Sent Done false
6283 2412 Acked Read 6 Done Sent Done false
6284 2413 Acked Read 1 Done Sent Done false
6285 2414 Acked Write 10 Done Sent Done false
6286 2415 Acked Read 4 Done Sent Done false
6287 2416 Acked Read 1 Done Sent Done false
6288 2417 Acked Write 5 Done Sent Done false
6289 2418 Acked Write 1 Done Sent Done false
6290 2419 Acked Read 1 Done Sent Done false
6291 2420 Acked Write 10 Done Sent Done false
6292 2421 Acked Write 3 Done Sent Done false
6293 2422 Acked Read 5 Done Sent Done false
6294 2423 Acked Read 4 Done Sent Done false
6295 2424 Acked Write 7 Done Sent Done false
6296 2425 Acked Write 4 Done Sent Done false
6297 2426 Acked Read 1 Done Sent Done false
6298 2427 Acked Write 10 Done Sent Done false
6299 2428 Acked Flush 0 Done Sent Done false
6300 2429 Acked Read 1 Done Sent Done false
6301 2430 Acked Read 5 Done Sent Done false
6302 2431 Acked Write 6 Done Sent Done false
6303 2432 Acked Write 4 Done Sent Done false
6304 2433 Acked Read 1 Done Sent Done false
6305 2434 Acked Read 1 Done Sent Done false
6306 2435 Acked Read 7 Done Sent Done false
6307 2436 Acked Read 8 Done Sent Done false
6308 2437 Acked Write 8 Done Sent Done false
6309 2438 Acked Read 2 Done Sent Done false
6310 2439 Acked Read 3 Done Sent Done false
6311 2440 Acked Read 10 Done Sent Done false
6312 2441 Acked Write 5 Done Sent Done false
6313 2442 Acked Write 9 Done Sent Done false
6314 2443 Acked Write 8 Done Sent Done false
6315 2444 Acked Read 7 Done Sent Done false
6316 2445 Acked Read 6 Done Sent Done false
6317 2446 Acked Read 10 Done Sent Done false
6318 2447 Acked Read 1 Done Sent Done false
6319 2448 Acked Write 5 Done Sent Done false
6320 2449 Acked Write 1 Done Sent Done false
6321 2450 Acked Write 3 Done Sent Done false
6322 2451 Acked Write 8 Done Sent Done false
6323 2452 Acked Read 4 Done Sent Done false
6324 2453 Acked Read 8 Done Sent Done false
6325 2454 Acked Read 6 Done Sent Done false
6326 2455 Acked Write 10 Done Sent Done false
6327 2456 Acked Flush 0 Done Sent Done false
6328 2457 Acked Write 2 Done Sent Done false
6329 2458 Acked Flush 0 Done Sent Done false
6330 2459 Acked Write 6 Done Sent Done false
6331 2460 Acked Read 9 Done Sent Done false
6332 2461 Acked Read 9 Done Sent Done false
6333 2462 Acked Write 8 Done Sent Done false
6334 2463 Acked Read 8 Done Sent Done false
6335 2464 Acked Read 7 Done Sent Done false
6336 2465 Acked Read 3 Done Sent Done false
6337 2466 Acked Write 4 Done Sent Done false
6338 2467 Acked Write 3 Done Sent Done false
6339 2468 Acked Write 1 Done Sent Done false
6340 2469 Acked Read 10 Done Sent Done false
6341 2470 Acked Read 2 Done Sent Done false
6342 2471 Acked Write 6 Done Sent Done false
6343 2472 Acked Read 4 Done Sent Done false
6344 2473 Acked Write 1 Done Sent Done false
6345 2474 Acked Read 3 Done Sent Done false
6346 2475 Acked Write 5 Done Sent Done false
6347 2476 Acked Read 9 Done Sent Done false
6348 2477 Acked Flush 0 Done Sent Done false
6349 2478 Acked Read 4 Done Sent Done false
6350 2479 Acked Read 8 Done Sent Done false
6351 2480 Acked Read 4 Done Sent Done false
6352 2481 Acked Read 2 Done Sent Done false
6353 2482 Acked Read 8 Done Sent Done false
6354 2483 Acked Read 2 Done Sent Done false
6355 2484 Acked Read 4 Done Sent Done false
6356 2485 Acked Read 1 Done Sent Done false
6357 2486 Acked Read 1 Done Sent Done false
6358 2487 Acked Write 7 Done Sent Done false
6359 2488 Acked Read 4 Done Sent Done false
6360 2489 Acked Read 9 Done Sent Done false
6361 2490 Acked Read 2 Done Sent Done false
6362 2491 Acked Write 5 Done Sent Done false
6363 2492 Acked Write 1 Done Sent Done false
6364 2493 Acked Read 5 Done Sent Done false
6365 2494 Acked Read 8 Done Sent Done false
6366 2495 Acked Read 7 Done Sent Done false
6367 2496 Acked Write 2 Done Sent Done false
6368 2497 Acked Flush 0 Done Sent Done false
6369 2498 Acked Write 5 Done Sent Done false
6370 2499 Acked Read 1 Done Sent Done false
6371 2500 Acked Read 2 Done Sent Done false
6372 2501 Acked Read 4 Done Sent Done false
6373 2502 Acked Read 1 Done Sent Done false
6374 2503 Acked Read 4 Done Sent Done false
6375 2504 Acked Read 6 Done Sent Done false
6376 2505 Acked Read 8 Done Sent Done false
6377 2506 Acked Write 2 Done Sent Done false
6378 2507 Acked Read 7 Done Sent Done false
6379 2508 Acked Write 7 Done Sent Done false
6380 STATES DS:0 DS:1 DS:2 TOTAL
6381 Sent 0 301 0 301
6382 Done 301 0 301 602
6383 Skipped 0 0 0 0
6384 Error 0 0 0 0
6385 Last Flush: Some(JobId(2497)) Some(JobId(2207)) Some(JobId(2497))
6386 Downstairs last five completed: 2207 2206 2205 2204 2203
6387 Upstairs last five completed: 2508 2507 2506 2505 2504
6388 CLIENT: Up:0 ds:301 act:2
63892025-02-07T20:08:35.235ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
63902025-02-07T20:08:36.060ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
63912025-02-07T20:08:37.062ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
63922025-02-07T20:08:38.064ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
63932025-02-07T20:08:38.482ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
63942025-02-07T20:08:38.482ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56967) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
63952025-02-07T20:08:38.483ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
63962025-02-07T20:08:38.484ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
63972025-02-07T20:08:38.484ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
63982025-02-07T20:08:38.484ZINFOcrucible: send last flush ID to this DS: Some(JobId(2207)) = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
63992025-02-07T20:08:38.484ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
64002025-02-07T20:08:38.484ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2207)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
64012025-02-07T20:08:38.484ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2207)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
6402 ----------------------------------------------------------------
6403 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:11
6404 States: Active Active Active
6405 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6406 2498 Acked Write 5 Done Done Done true
6407 2499 Acked Read 1 Done Done Done true
6408 2500 Acked Read 2 Done Done Done true
6409 2501 Acked Read 4 Done Done Done true
6410 2502 Acked Read 1 Done Done Done true
6411 2503 Acked Read 4 Done Done Done true
6412 2504 Acked Read 6 Done Done Done true
6413 2505 Acked Read 8 Done Done Done true
6414 2506 Acked Write 2 Done Done Done true
6415 2507 Acked Read 7 Done Done Done true
6416 2508 Acked Write 7 Done Done Done true
6417 STATES DS:0 DS:1 DS:2 TOTAL
6418 Sent 0 0 0 0
6419 Done 11 11 11 33
6420 Skipped 0 0 0 0
6421 Error 0 0 0 0
6422 Last Flush: Some(JobId(2497)) Some(JobId(2497)) Some(JobId(2497))
6423 Downstairs last five completed: 2497 2496 2495 2494 2493
6424 Upstairs last five completed: 2508 2507 2506 2505 2504
6425 CLIENT: Up:0 ds:11 act:3
6426 ----------------------------------------------------------------
6427 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6428 Downstairs last five completed: 2509 2508 2507 2506 2505
6429 Upstairs last five completed: 2509 2508 2507 2506 2505
6430 CLIENT: Up:0 ds:0 act:3
6431 Replay: All jobs finished
6432 Main task has work to do, go find it
6433 got dsc Stop(0)
6434 stop 0
6435 Feb 07 20:08:42.738 INFO [8810] Got stop action so:false kr:false
6436 request completed, latency_us: 323, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 8e4df38a-c26b-420e-8781-f2064e1939fe, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
64372025-02-07T20:08:42.739ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
64382025-02-07T20:08:42.739ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
6439 Feb 07 20:08:42.739 INFO request completed, latency_us: 154, response_code: 200, uri: /state/cid/0, method: GET, req_id: 103f1274-cc5b-4336-a02a-c512f62e6d5a, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
64402025-02-07T20:08:42.739ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
64412025-02-07T20:08:42.739ZINFOcrucible: client task is exiting = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
6442 [8810] Exited with: ExitStatus(unix_wait_status(9))
6443 [8810][0] reports Exit
6444 Feb 07 20:08:46.741 INFO request completed, latency_us: 322, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2bf07ec7-95de-4cd2-9138-07d5a1aa64d2, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
6445 001/300 Write block 1487 len 8192 data: 2 2
6446 002/300 Write block 4672 len 32768 data: 2 2 2 2 2 2 2 2
6447 003/300 Write block 3799 len 12288 data: 3 3 3
6448 004/300 Write block 4666 len 32768 data: 4 3 4 4 4 3 3 3
6449 005/300 Read block 3269 len 28672
6450 006/300 Read block 1314 len 32768
6451 007/300 Read block 2580 len 8192
6452 008/300 Write block 3815 len 20480 data: 3 3 4 4 4
6453 009/300 Read block 3039 len 36864
6454 010/300 Write block 422 len 4096 data: 2
6455 011/300 Write block 3341 len 8192 data: 4 3
6456 012/300 Read block 1447 len 24576
6457 013/300 Write block 3840 len 28672 data: 2 2 3 3 3 3 3
6458 014/300 Write block 505 len 40960 data: 3 3 2 2 2 2 2 2 2 2
6459 015/300 Read block 2766 len 4096
6460 016/300 Write block 4472 len 4096 data: 4
6461 017/300 Read block 1789 len 24576
6462 018/300 Write block 4990 len 24576 data: 3 3 3 3 2 2
6463 019/300 Write block 3659 len 32768 data: 2 3 3 4 4 4 4 4
6464 020/300 Read block 4964 len 40960
6465 021/300 Flush
6466 022/300 Write block 1492 len 20480 data: 3 3 3 3 2
6467 023/300 Read block 3247 len 40960
6468 024/300 Flush
6469 025/300 Write block 642 len 16384 data: 2 2 2 2
6470 026/300 Write block 3072 len 8192 data: 2 2
6471 027/300 Read block 1547 len 4096
6472 028/300 Read block 3460 len 8192
6473 029/300 Write block 695 len 20480 data: 2 2 2 2 2
6474 030/300 Read block 4731 len 32768
6475 031/300 Write block 1503 len 36864 data: 2 2 2 3 3 3 3 3 3
6476 032/300 Read block 3339 len 32768
6477 033/300 Write block 2976 len 32768 data: 2 3 3 3 4 4 4 4
6478 034/300 Read block 4555 len 40960
6479 035/300 Read block 37 len 12288
6480 036/300 Write block 1609 len 12288 data: 2 2 2
6481 037/300 Read block 3747 len 40960
6482 038/300 Write block 1274 len 12288 data: 3 5 5
6483 039/300 Write block 4499 len 28672 data: 5 5 5 7 5 6 5
6484 040/300 Read block 1569 len 12288
6485 041/300 Read block 4483 len 28672
6486 042/300 Read block 1293 len 12288
6487 043/300 Read block 4435 len 12288
6488 044/300 Write block 683 len 36864 data: 3 3 3 3 2 2 2 2 2
6489 045/300 Read block 4615 len 4096
6490 046/300 Read block 3958 len 28672
6491 047/300 Write block 4801 len 32768 data: 3 3 3 3 3 3 2 2
6492 048/300 Write block 1767 len 20480 data: 3 3 2 2 3
6493 049/300 Write block 897 len 36864 data: 2 2 2 2 2 3 3 3 3
6494 050/300 Write block 3073 len 24576 data: 3 2 3 3 3 2
6495 051/300 Read block 1422 len 28672
6496 052/300 Read block 66 len 28672
6497 053/300 Write block 1277 len 12288 data: 5 5 5
6498 054/300 Read block 617 len 16384
6499 055/300 Read block 2028 len 36864
6500 056/300 Read block 4607 len 28672
6501 057/300 Read block 2845 len 8192
6502 058/300 Write block 1520 len 28672 data: 2 2 2 2 3 3 3
6503 059/300 Read block 1358 len 4096
6504 060/300 Read block 3422 len 28672
6505 061/300 Write block 1580 len 40960 data: 3 3 3 2 2 2 2 2 2 2
6506 062/300 Write block 4744 len 20480 data: 3 3 2 4 3
6507 063/300 Read block 833 len 24576
6508 064/300 Read block 827 len 12288
6509 065/300 Read block 1412 len 8192
6510 066/300 Write block 2012 len 36864 data: 3 3 2 2 2 3 3 4 3
6511 067/300 Read block 2923 len 28672
6512 068/300 Read block 2711 len 24576
6513 069/300 Flush
6514 070/300 Write block 3815 len 24576 data: 4 4 5 5 5 3
6515 071/300 Read block 4512 len 32768
6516 072/300 Read block 2373 len 16384
6517 073/300 Write block 4673 len 4096 data: 4
6518 074/300 Write block 4376 len 32768 data: 3 3 3 3 2 2 2 2
6519 075/300 Write block 531 len 8192 data: 2 2
6520 076/300 Read block 2229 len 20480
6521 077/300 Read block 1389 len 24576
6522 078/300 Write block 4787 len 16384 data: 4 4 4 4
6523 079/300 Write block 4007 len 36864 data: 2 2 2 2 2 2 2 2 2
6524 080/300 Read block 3435 len 16384
6525 081/300 Read block 3475 len 32768
6526 082/300 Read block 4637 len 16384
6527 083/300 Read block 4802 len 4096
6528 084/300 Write block 1850 len 32768 data: 2 2 2 2 2 2 2 2
6529 085/300 Write block 3164 len 16384 data: 2 2 2 3
6530 086/300 Write block 2505 len 28672 data: 2 2 2 2 2 2 2
6531 087/300 Write block 4684 len 20480 data: 3 3 2 2 2
6532 088/300 Read block 2852 len 36864
6533 089/300 Read block 1520 len 36864
6534 090/300 Read block 114 len 36864
6535 091/300 Write block 2755 len 8192 data: 3 3
6536 092/300 Read block 4928 len 36864
6537 093/300 Write block 1581 len 24576 data: 4 4 3 3 3 3
6538 094/300 Write block 1680 len 36864 data: 3 3 3 3 4 3 3 3 3
6539 095/300 Read block 1151 len 28672
6540 096/300 Read block 923 len 4096
6541 097/300 Read block 4740 len 24576
6542 098/300 Flush
6543 099/300 Flush
6544 100/300 Read block 2448 len 24576
6545 101/300 Write block 1417 len 24576 data: 2 2 2 2 3 2
6546 102/300 Flush
6547 103/300 Read block 2366 len 20480
6548 104/300 Read block 3142 len 40960
6549 105/300 Write block 3912 len 20480 data: 3 3 3 3 3
6550 106/300 Read block 2822 len 24576
6551 107/300 Read block 4362 len 40960
6552 108/300 Write block 1389 len 20480 data: 3 4 4 4 3
6553 109/300 Flush
6554 110/300 Read block 3528 len 16384
6555 111/300 Read block 116 len 24576
6556 112/300 Flush
6557 113/300 Read block 1253 len 4096
6558 114/300 Write block 2056 len 4096 data: 3
6559 115/300 Flush
6560 116/300 Read block 2846 len 36864
6561 117/300 Write block 3797 len 36864 data: 3 3 4 4 4 3 3 3 3
6562 118/300 Read block 3779 len 8192
6563 119/300 Read block 187 len 32768
6564 120/300 Write block 1454 len 36864 data: 2 2 2 2 2 2 2 3 3
6565 121/300 Flush
6566 122/300 Read block 4693 len 32768
6567 123/300 Read block 3673 len 8192
6568 124/300 Read block 4011 len 40960
6569 125/300 Flush
6570 126/300 Read block 2042 len 32768
6571 127/300 Read block 1473 len 36864
6572 128/300 Read block 14 len 32768
6573 129/300 Write block 4742 len 40960 data: 2 3 4 4 3 5 4 3 3 3
6574 130/300 Read block 3526 len 32768
6575 131/300 Write block 2028 len 12288 data: 3 2 2
6576 132/300 Read block 3750 len 24576
6577 133/300 Read block 54 len 24576
6578 134/300 Write block 2188 len 36864 data: 3 3 3 3 3 3 3 2 2
6579 135/300 Read block 3736 len 24576
6580 136/300 Read block 4773 len 24576
6581 137/300 Read block 2806 len 24576
6582 138/300 Write block 29 len 32768 data: 3 2 3 3 3 3 3 3
6583 139/300 Write block 946 len 40960 data: 2 2 2 2 2 2 2 2 3 3
6584 140/300 Read block 3781 len 28672
6585 141/300 Write block 197 len 24576 data: 3 3 3 3 2 2
6586 142/300 Write block 401 len 4096 data: 2
6587 143/300 Read block 1781 len 32768
6588 144/300 Read block 1169 len 32768
6589 145/300 Write block 2999 len 36864 data: 2 2 3 3 3 3 3 3 3
6590 146/300 Read block 4350 len 12288
6591 147/300 Read block 2316 len 20480
6592 148/300 Read block 331 len 40960
6593 149/300 Read block 3891 len 12288
6594 150/300 Flush
6595 151/300 Read block 380 len 28672
6596 152/300 Write block 2233 len 32768 data: 2 2 3 2 2 2 2 2
6597 153/300 Read block 2578 len 4096
6598 154/300 Read block 2065 len 28672
6599 155/300 Read block 66 len 4096
6600 156/300 Write block 3865 len 12288 data: 3 3 3
6601 157/300 Read block 265 len 20480
6602 158/300 Read block 3785 len 40960
6603 159/300 Write block 4200 len 8192 data: 2 2
6604 160/300 Write block 3528 len 36864 data: 2 2 2 2 2 2 2 2 2
6605 161/300 Write block 1019 len 20480 data: 3 3 3 3 2
6606 162/300 Read block 4346 len 8192
6607 163/300 Write block 3052 len 12288 data: 3 2 2
6608 164/300 Write block 2774 len 20480 data: 2 2 2 2 2
6609 165/300 Read block 3990 len 16384
6610 166/300 Read block 2374 len 12288
6611 167/300 Write block 4703 len 24576 data: 2 2 2 2 2 2
6612 168/300 Write block 2863 len 12288 data: 4 4 3
6613 169/300 Flush
6614 170/300 Read block 1375 len 8192
6615 171/300 Read block 3865 len 8192
6616 172/300 Flush
6617 173/300 Flush
6618 174/300 Write block 2862 len 4096 data: 5
6619 175/300 Write block 3079 len 32768 data: 2 2 2 2 2 2 2 2
6620 176/300 Read block 2500 len 32768
6621 177/300 Read block 615 len 20480
6622 178/300 Write block 4959 len 36864 data: 3 2 2 2 2 2 2 3 3
6623 179/300 Read block 931 len 32768
6624 180/300 Flush
6625 181/300 Read block 4073 len 40960
6626 182/300 Read block 3985 len 36864
6627 183/300 Read block 1295 len 36864
6628 184/300 Read block 3298 len 4096
6629 185/300 Write block 1834 len 28672 data: 3 3 3 3 3 2 2
6630 186/300 Read block 916 len 40960
6631 187/300 Write block 266 len 40960 data: 3 3 3 4 4 3 3 3 3 3
6632 188/300 Write block 1345 len 12288 data: 3 3 2
6633 189/300 Write block 2287 len 24576 data: 2 2 2 2 2 2
6634 190/300 Read block 1509 len 12288
6635 191/300 Write block 278 len 20480 data: 2 2 2 2 2
6636 192/300 Read block 4809 len 28672
6637 193/300 Read block 1032 len 20480
6638 194/300 Write block 1639 len 16384 data: 3 2 2 2
6639 195/300 Flush
6640 196/300 Read block 4217 len 8192
6641 197/300 Write block 1517 len 40960 data: 2 2 2 3 3 3 3 4 4 4
6642 198/300 Read block 4889 len 24576
6643 199/300 Flush
6644 200/300 Read block 2953 len 20480
6645 201/300 Read block 3949 len 4096
6646 202/300 Read block 2063 len 4096
6647 203/300 Write block 645 len 8192 data: 3 2
6648 204/300 Write block 2832 len 20480 data: 3 3 3 2 2
6649 205/300 Write block 3193 len 4096 data: 3
6650 206/300 Read block 1978 len 28672
6651 207/300 Read block 4196 len 40960
6652 208/300 Read block 447 len 12288
6653 209/300 Flush
6654 210/300 Write block 1051 len 28672 data: 4 4 4 4 3 4 3
6655 211/300 Write block 1918 len 8192 data: 3 3
6656 212/300 Write block 4509 len 12288 data: 5 5 4
6657 213/300 Read block 3320 len 16384
6658 214/300 Write block 2017 len 4096 data: 4
6659 215/300 Read block 655 len 28672
6660 216/300 Read block 3557 len 8192
6661 217/300 Read block 1361 len 28672
6662 218/300 Read block 2008 len 4096
6663 219/300 Read block 284 len 12288
6664 220/300 Write block 643 len 28672 data: 3 3 4 3 2 2 2
6665 221/300 Read block 4590 len 16384
6666 222/300 Write block 256 len 32768 data: 2 2 2 2 2 2 2 2
6667 223/300 Read block 3010 len 4096
6668 224/300 Read block 4726 len 40960
6669 225/300 Read block 2323 len 16384
6670 226/300 Read block 2268 len 20480
6671 227/300 Flush
6672 228/300 Write block 709 len 40960 data: 3 3 3 3 3 2 2 2 2 2
6673 229/300 Write block 3300 len 36864 data: 2 3 3 3 2 2 2 2 2
6674 230/300 Write block 3453 len 8192 data: 3 3
6675 231/300 Write block 582 len 8192 data: 3 4
6676 232/300 Read block 3578 len 12288
6677 233/300 Write block 4801 len 36864 data: 4 4 4 4 4 4 3 3 2
6678 234/300 Read block 4131 len 16384
6679 235/300 Write block 3654 len 8192 data: 2 2
6680 236/300 Write block 1223 len 20480 data: 3 3 3 2 2
6681 237/300 Flush
6682 238/300 Flush
6683 239/300 Flush
6684 240/300 Flush
6685 241/300 Write block 1321 len 20480 data: 4 4 4 4 4
6686 242/300 Read block 3547 len 16384
6687 243/300 Read block 490 len 36864
6688 244/300 Write block 759 len 20480 data: 3 2 2 2 2
6689 245/300 Write block 2072 len 16384 data: 2 2 2 3
6690 246/300 Write block 1037 len 32768 data: 4 3 2 2 2 2 2 2
6691 247/300 Read block 4842 len 4096
6692 248/300 Read block 4287 len 12288
6693 249/300 Write block 369 len 12288 data: 3 2 2
6694 250/300 Write block 3272 len 40960 data: 3 3 4 4 3 3 3 3 3 2
6695 251/300 Write block 974 len 40960 data: 2 2 3 3 3 3 2 2 2 2
6696 252/300 Flush
6697 253/300 Read block 1016 len 40960
6698 254/300 Write block 4536 len 4096 data: 2
6699 255/300 Flush
6700 256/300 Read block 4062 len 24576
6701 257/300 Read block 893 len 32768
6702 258/300 Read block 3129 len 36864
6703 259/300 Write block 1446 len 16384 data: 2 2 2 2
6704 260/300 Read block 3628 len 8192
6705 261/300 Write block 4461 len 24576 data: 5 4 4 4 4 4
6706 262/300 Write block 2164 len 4096 data: 4
6707 263/300 Read block 1774 len 28672
6708 264/300 Write block 1856 len 24576 data: 3 3 2 2 2 2
6709 265/300 Read block 2280 len 16384
6710 266/300 Read block 2573 len 28672
6711 267/300 Write block 683 len 20480 data: 4 4 4 4 3
6712 268/300 Write block 4647 len 36864 data: 2 3 3 4 4 3 3 3 3
6713 269/300 Read block 42 len 12288
6714 270/300 Read block 4559 len 24576
6715 271/300 Write block 827 len 8192 data: 2 3
6716 272/300 Write block 3049 len 20480 data: 3 3 3 4 3
6717 273/300 Read block 4444 len 8192
6718 274/300 Write block 4119 len 16384 data: 2 3 3 3
6719 275/300 Read block 2819 len 16384
6720 276/300 Read block 2990 len 36864
6721 277/300 Write block 618 len 24576 data: 3 3 3 3 4 4
6722 278/300 Write block 1533 len 8192 data: 2 2
6723 279/300 Read block 517 len 12288
6724 280/300 Write block 1023 len 36864 data: 3 2 2 2 2 2 2 2 2
6725 281/300 Write block 3022 len 4096 data: 2
6726 282/300 Read block 607 len 8192
6727 283/300 Write block 1120 len 4096 data: 2
6728 284/300 Write block 2536 len 36864 data: 2 3 3 3 3 3 3 3 2
6729 285/300 Write block 1160 len 8192 data: 4 3
6730 286/300 Write block 3380 len 16384 data: 2 2 2 2
6731 287/300 Read block 719 len 16384
6732 288/300 Read block 3942 len 32768
6733 289/300 Read block 2526 len 20480
6734 290/300 Read block 4521 len 24576
6735 291/300 Read block 125 len 32768
6736 292/300 Write block 1658 len 20480 data: 5 4 3 2 2
6737 293/300 Read block 1910 len 32768
6738 294/300 Flush
6739 295/300 Write block 762 len 8192 data: 3 3
6740 296/300 Read block 1786 len 20480
6741 297/300 Flush
6742 298/300 Write block 4975 len 4096 data: 2
6743 299/300 Read block 1806 len 4096
6744 300/300 Read block 3428 len 8192
6745 301/300 Read block 1800 len 12288
6746 Main task has work to do, go find it
6747 got dsc Start(0)
6748 start 0
6749 Feb 07 20:08:46.985 INFO request completed, latency_us: 262, response_code: 204, uri: /start/cid/0, method: POST, req_id: cba8c031-3f6a-4153-8be6-6c0212c7882a, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
6750 Replay: started 0, returned:Ok(())
6751 ----------------------------------------------------------------
6752 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6753 States: Offline Active Active
6754 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6755 2510 Acked Write 2 Sent Done Done false
6756 2511 Acked Write 8 Sent Done Done false
6757 2512 Acked Write 3 Sent Done Done false
6758 2513 Acked Write 8 Sent Done Done false
6759 2514 Acked Read 7 Sent Done Done false
6760 2515 Acked Read 8 Sent Done Done false
6761 2516 Acked Read 2 Sent Done Done false
6762 2517 Acked Write 5 Sent Done Done false
6763 2518 Acked Read 9 Sent Done Done false
6764 2519 Acked Write 1 Sent Done Done false
6765 2520 Acked Write 2 Sent Done Done false
6766 2521 Acked Read 6 Sent Done Done false
6767 2522 Acked Write 7 Sent Done Done false
6768 2523 Acked Write 10 Sent Done Done false
6769 2524 Acked Read 1 Sent Done Done false
6770 2525 Acked Write 1 Sent Done Done false
6771 2526 Acked Read 6 Sent Done Done false
6772 2527 Acked Write 6 Sent Done Done false
6773 2528 Acked Write 8 Sent Done Done false
6774 2529 Acked Read 10 Sent Done Done false
6775 2530 Acked Flush 0 Sent Done Done false
6776 2531 Acked Write 5 Sent Done Done false
6777 2532 Acked Read 10 Sent Done Done false
6778 2533 Acked Flush 0 Sent Done Done false
6779 2534 Acked Write 4 Sent Done Done false
6780 2535 Acked Write 2 Sent Done Done false
6781 2536 Acked Read 1 Sent Done Done false
6782 2537 Acked Read 2 Sent Done Done false
6783 2538 Acked Write 5 Sent Done Done false
6784 2539 Acked Read 8 Sent Done Done false
6785 2540 Acked Write 9 Sent Done Done false
6786 2541 Acked Read 8 Sent Done Done false
6787 2542 Acked Write 8 Sent Done Done false
6788 2543 Acked Read 10 Sent Done Done false
6789 2544 Acked Read 3 Sent Done Done false
6790 2545 Acked Write 3 Sent Done Done false
6791 2546 Acked Read 10 Sent Done Done false
6792 2547 Acked Write 3 Sent Done Done false
6793 2548 Acked Write 7 Sent Done Done false
6794 2549 Acked Read 3 Sent Done Done false
6795 2550 Acked Read 7 Sent Done Done false
6796 2551 Acked Read 3 Sent Done Done false
6797 2552 Acked Read 3 Sent Done Done false
6798 2553 Acked Write 9 Sent Done Done false
6799 2554 Acked Read 1 Sent Done Done false
6800 2555 Acked Read 7 Sent Done Done false
6801 2556 Acked Write 8 Sent Done Done false
6802 2557 Acked Write 5 Sent Done Done false
6803 2558 Acked Write 9 Sent Done Done false
6804 2559 Acked Write 6 Sent Done Done false
6805 2560 Acked Read 7 Sent Done Done false
6806 2561 Acked Read 7 Sent Done Done false
6807 2562 Acked Write 3 Sent Done Done false
6808 2563 Acked Read 4 Sent Done Done false
6809 2564 Acked Read 9 Sent Done Done false
6810 2565 Acked Read 7 Sent Done Done false
6811 2566 Acked Read 2 Sent Done Done false
6812 2567 Acked Write 7 Sent Done Done false
6813 2568 Acked Read 1 Sent Done Done false
6814 2569 Acked Read 7 Sent Done Done false
6815 2570 Acked Write 10 Sent Done Done false
6816 2571 Acked Write 5 Sent Done Done false
6817 2572 Acked Read 6 Sent Done Done false
6818 2573 Acked Read 3 Sent Done Done false
6819 2574 Acked Read 2 Sent Done Done false
6820 2575 Acked Write 9 Sent Done Done false
6821 2576 Acked Read 7 Sent Done Done false
6822 2577 Acked Read 6 Sent Done Done false
6823 2578 Acked Flush 0 Sent Done Done false
6824 2579 Acked Write 6 Sent Done Done false
6825 2580 Acked Read 8 Sent Done Done false
6826 2581 Acked Read 4 Sent Done Done false
6827 2582 Acked Write 1 Sent Done Done false
6828 2583 Acked Write 8 Sent Done Done false
6829 2584 Acked Write 2 Sent Done Done false
6830 2585 Acked Read 5 Sent Done Done false
6831 2586 Acked Read 6 Sent Done Done false
6832 2587 Acked Write 4 Sent Done Done false
6833 2588 Acked Write 9 Sent Done Done false
6834 2589 Acked Read 4 Sent Done Done false
6835 2590 Acked Read 8 Sent Done Done false
6836 2591 Acked Read 4 Sent Done Done false
6837 2592 Acked Read 1 Sent Done Done false
6838 2593 Acked Write 8 Sent Done Done false
6839 2594 Acked Write 4 Sent Done Done false
6840 2595 Acked Write 7 Sent Done Done false
6841 2596 Acked Write 5 Sent Done Done false
6842 2597 Acked Read 9 Sent Done Done false
6843 2598 Acked Read 9 Sent Done Done false
6844 2599 Acked Read 9 Sent Done Done false
6845 2600 Acked Write 2 Sent Done Done false
6846 2601 Acked Read 9 Sent Done Done false
6847 2602 Acked Write 6 Sent Done Done false
6848 2603 Acked Write 9 Sent Done Done false
6849 2604 Acked Read 7 Sent Done Done false
6850 2605 Acked Read 1 Sent Done Done false
6851 2606 Acked Read 6 Sent Done Done false
6852 2607 Acked Flush 0 Sent Done Done false
6853 2608 Acked Flush 0 Sent Done Done false
6854 2609 Acked Read 6 Sent Done Done false
6855 2610 Acked Write 6 Sent Done Done false
6856 2611 Acked Flush 0 Sent Done Done false
6857 2612 Acked Read 5 Sent Done Done false
6858 2613 Acked Read 10 Sent Done Done false
6859 2614 Acked Write 5 Sent Done Done false
6860 2615 Acked Read 6 Sent Done Done false
6861 2616 Acked Read 10 Sent Done Done false
6862 2617 Acked Write 5 Sent Done Done false
6863 2618 Acked Flush 0 Sent Done Done false
6864 2619 Acked Read 4 Sent Done Done false
6865 2620 Acked Read 6 Sent Done Done false
6866 2621 Acked Flush 0 Sent Done Done false
6867 2622 Acked Read 1 Sent Done Done false
6868 2623 Acked Write 1 Sent Done Done false
6869 2624 Acked Flush 0 Sent Done Done false
6870 2625 Acked Read 9 Sent Done Done false
6871 2626 Acked Write 9 Sent Done Done false
6872 2627 Acked Read 2 Sent Done Done false
6873 2628 Acked Read 8 Sent Done Done false
6874 2629 Acked Write 9 Sent Done Done false
6875 2630 Acked Flush 0 Sent Done Done false
6876 2631 Acked Read 8 Sent Done Done false
6877 2632 Acked Read 2 Sent Done Done false
6878 2633 Acked Read 10 Sent Done Done false
6879 2634 Acked Flush 0 Sent Done Done false
6880 2635 Acked Read 8 Sent Done Done false
6881 2636 Acked Read 9 Sent Done Done false
6882 2637 Acked Read 8 Sent Done Done false
6883 2638 Acked Write 10 Sent Done Done false
6884 2639 Acked Read 8 Sent Done Done false
6885 2640 Acked Write 3 Sent Done Done false
6886 2641 Acked Read 6 Sent Done Done false
6887 2642 Acked Read 6 Sent Done Done false
6888 2643 Acked Write 9 Sent Done Done false
6889 2644 Acked Read 6 Sent Done Done false
6890 2645 Acked Read 6 Sent Done Done false
6891 2646 Acked Read 6 Sent Done Done false
6892 2647 Acked Write 8 Sent Done Done false
6893 2648 Acked Write 10 Sent Done Done false
6894 2649 Acked Read 7 Sent Done Done false
6895 2650 Acked Write 6 Sent Done Done false
6896 2651 Acked Write 1 Sent Done Done false
6897 2652 Acked Read 8 Sent Done Done false
6898 2653 Acked Read 8 Sent Done Done false
6899 2654 Acked Write 9 Sent Done Done false
6900 2655 Acked Read 3 Sent Done Done false
6901 2656 Acked Read 5 Sent Done Done false
6902 2657 Acked Read 10 Sent Done Done false
6903 2658 Acked Read 3 Sent Done Done false
6904 2659 Acked Flush 0 Sent Done Done false
6905 2660 Acked Read 7 Sent Done Done false
6906 2661 Acked Write 8 Sent Done Done false
6907 2662 Acked Read 1 Sent Done Done false
6908 2663 Acked Read 7 Sent Done Done false
6909 2664 Acked Read 1 Sent Done Done false
6910 2665 Acked Write 3 Sent Done Done false
6911 2666 Acked Read 5 Sent Done Done false
6912 2667 Acked Read 10 Sent Done Done false
6913 2668 Acked Write 2 Sent Done Done false
6914 2669 Acked Write 9 Sent Done Done false
6915 2670 Acked Write 5 Sent Done Done false
6916 2671 Acked Read 2 Sent Done Done false
6917 2672 Acked Write 3 Sent Done Done false
6918 2673 Acked Write 5 Sent Done Done false
6919 2674 Acked Read 4 Sent Done Done false
6920 2675 Acked Read 3 Sent Done Done false
6921 2676 Acked Write 6 Sent Done Done false
6922 2677 Acked Write 3 Sent Done Done false
6923 2678 Acked Flush 0 Sent Done Done false
6924 2679 Acked Read 2 Sent Done Done false
6925 2680 Acked Read 2 Sent Done Done false
6926 2681 Acked Flush 0 Sent Done Done false
6927 2682 Acked Flush 0 Sent Done Done false
6928 2683 Acked Write 1 Sent Done Done false
6929 2684 Acked Write 8 Sent Done Done false
6930 2685 Acked Read 8 Sent Done Done false
6931 2686 Acked Read 5 Sent Done Done false
6932 2687 Acked Write 9 Sent Done Done false
6933 2688 Acked Read 8 Sent Done Done false
6934 2689 Acked Flush 0 Sent Done Done false
6935 2690 Acked Read 10 Sent Done Done false
6936 2691 Acked Read 9 Sent Done Done false
6937 2692 Acked Read 9 Sent Done Done false
6938 2693 Acked Read 1 Sent Done Done false
6939 2694 Acked Write 7 Sent Done Done false
6940 2695 Acked Read 10 Sent Done Done false
6941 2696 Acked Write 10 Sent Done Done false
6942 2697 Acked Write 3 Sent Done Done false
6943 2698 Acked Write 6 Sent Done Done false
6944 2699 Acked Read 3 Sent Done Done false
6945 2700 Acked Write 5 Sent Done Done false
6946 2701 Acked Read 7 Sent Done Done false
6947 2702 Acked Read 5 Sent Done Done false
6948 2703 Acked Write 4 Sent Done Done false
6949 2704 Acked Flush 0 Sent Done Done false
6950 2705 Acked Read 2 Sent Done Done false
6951 2706 Acked Write 10 Sent Done Done false
6952 2707 Acked Read 6 Sent Done Done false
6953 2708 Acked Flush 0 Sent Done Done false
6954 2709 Acked Read 5 Sent Done Done false
6955 2710 Acked Read 1 Sent Done Done false
6956 2711 Acked Read 1 Sent Done Done false
6957 2712 Acked Write 2 Sent Done Done false
6958 2713 Acked Write 5 Sent Done Done false
6959 2714 Acked Write 1 Sent Done Done false
6960 2715 Acked Read 7 Sent Done Done false
6961 2716 Acked Read 10 Sent Done Done false
6962 2717 Acked Read 3 Sent Done Done false
6963 2718 Acked Flush 0 Sent Done Done false
6964 2719 Acked Write 7 Sent Done Done false
6965 2720 Acked Write 2 Sent Done Done false
6966 2721 Acked Write 3 Sent Done Done false
6967 2722 Acked Read 4 Sent Done Done false
6968 2723 Acked Write 1 Sent Done Done false
6969 2724 Acked Read 7 Sent Done Done false
6970 2725 Acked Read 2 Sent Done Done false
6971 2726 Acked Read 7 Sent Done Done false
6972 2727 Acked Read 1 Sent Done Done false
6973 2728 Acked Read 3 Sent Done Done false
6974 2729 Acked Write 7 Sent Done Done false
6975 2730 Acked Read 4 Sent Done Done false
6976 2731 Acked Write 8 Sent Done Done false
6977 2732 Acked Read 1 Sent Done Done false
6978 2733 Acked Read 10 Sent Done Done false
6979 2734 Acked Read 4 Sent Done Done false
6980 2735 Acked Read 5 Sent Done Done false
6981 2736 Acked Flush 0 Sent Done Done false
6982 2737 Acked Write 10 Sent Done Done false
6983 2738 Acked Write 9 Sent Done Done false
6984 2739 Acked Write 2 Sent Done Done false
6985 2740 Acked Write 2 Sent Done Done false
6986 2741 Acked Read 3 Sent Done Done false
6987 2742 Acked Write 9 Sent Done Done false
6988 2743 Acked Read 4 Sent Done Done false
6989 2744 Acked Write 2 Sent Done Done false
6990 2745 Acked Write 5 Sent Done Done false
6991 2746 Acked Flush 0 Sent Done Done false
6992 2747 Acked Flush 0 Sent Done Done false
6993 2748 Acked Flush 0 Sent Done Done false
6994 2749 Acked Flush 0 Sent Done Done false
6995 2750 Acked Write 5 Sent Done Done false
6996 2751 Acked Read 4 Sent Done Done false
6997 2752 Acked Read 9 Sent Done Done false
6998 2753 Acked Write 5 Sent Done Done false
6999 2754 Acked Write 4 Sent Done Done false
7000 2755 Acked Write 8 Sent Done Done false
7001 2756 Acked Read 1 Sent Done Done false
7002 2757 Acked Read 3 Sent Done Done false
7003 2758 Acked Write 3 Sent Done Done false
7004 2759 Acked Write 10 Sent Done Done false
7005 2760 Acked Write 10 Sent Done Done false
7006 2761 Acked Flush 0 Sent Done Done false
7007 2762 Acked Read 10 Sent Done Done false
7008 2763 Acked Write 1 Sent Done Done false
7009 2764 Acked Flush 0 Sent Done Done false
7010 2765 Acked Read 6 Sent Done Done false
7011 2766 Acked Read 8 Sent Done Done false
7012 2767 Acked Read 9 Sent Done Done false
7013 2768 Acked Write 4 Sent Done Done false
7014 2769 Acked Read 2 Sent Done Done false
7015 2770 Acked Write 6 Sent Done Done false
7016 2771 Acked Write 1 Sent Done Done false
7017 2772 Acked Read 7 Sent Done Done false
7018 2773 Acked Write 6 Sent Done Done false
7019 2774 Acked Read 4 Sent Done Done false
7020 2775 Acked Read 7 Sent Done Done false
7021 2776 Acked Write 5 Sent Done Done false
7022 2777 Acked Write 9 Sent Done Done false
7023 2778 Acked Read 3 Sent Done Done false
7024 2779 Acked Read 6 Sent Done Done false
7025 2780 Acked Write 2 Sent Done Done false
7026 2781 Acked Write 5 Sent Done Done false
7027 2782 Acked Read 2 Sent Done Done false
7028 2783 Acked Write 4 Sent Done Done false
7029 2784 Acked Read 4 Sent Done Done false
7030 2785 Acked Read 9 Sent Done Done false
7031 2786 Acked Write 6 Sent Done Done false
7032 2787 Acked Write 2 Sent Done Done false
7033 2788 Acked Read 3 Sent Done Done false
7034 2789 Acked Write 9 Sent Done Done false
7035 2790 Acked Write 1 Sent Done Done false
7036 2791 Acked Read 2 Sent Done Done false
7037 2792 Acked Write 1 Sent Done Done false
7038 2793 Acked Write 9 Sent Done Done false
7039 2794 Acked Write 2 Sent Done Done false
7040 2795 Acked Write 4 Sent Done Done false
7041 2796 Acked Read 4 Sent Done Done false
7042 2797 Acked Read 8 Sent Done Done false
7043 2798 Acked Read 5 Sent Done Done false
7044 2799 Acked Read 6 Sent Done Done false
7045 2800 Acked Read 8 Sent Done Done false
7046 2801 Acked Write 5 Sent Done Done false
7047 2802 Acked Read 8 Sent Done Done false
7048 2803 Acked Flush 0 Sent Done Done false
7049 2804 Acked Write 2 Sent Done Done false
7050 2805 Acked Read 5 Sent Done Done false
7051 2806 Acked Flush 0 Sent Done Done false
7052 2807 Acked Write 1 Sent Done Done false
7053 2808 Acked Read 1 Sent Done Done false
7054 2809 Acked Read 2 Sent Done Done false
7055 2810 Acked Read 3 Sent Done Done false
7056 STATES DS:0 DS:1 DS:2 TOTAL
7057 Sent 301 0 0 301
7058 Done 0 301 301 602
7059 Skipped 0 0 0 0
7060 Error 0 0 0 0
7061 Last Flush: Some(JobId(2509)) Some(JobId(2806)) Some(JobId(2806))
7062 Downstairs last five completed: 2509 2508 2507 2506 2505
7063 Upstairs last five completed: 2810 2809 2808 2807 2806
7064 CLIENT: Up:0 ds:301 act:2
70652025-02-07T20:08:47.487ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
70662025-02-07T20:08:48.082ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
70672025-02-07T20:08:49.491ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
70682025-02-07T20:08:50.086ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
7069 ----------------------------------------------------------------
7070 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7071 States: Offline Active Active
7072 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7073 2510 Acked Write 2 Sent Done Done false
7074 2511 Acked Write 8 Sent Done Done false
7075 2512 Acked Write 3 Sent Done Done false
7076 2513 Acked Write 8 Sent Done Done false
7077 2514 Acked Read 7 Sent Done Done false
7078 2515 Acked Read 8 Sent Done Done false
7079 2516 Acked Read 2 Sent Done Done false
7080 2517 Acked Write 5 Sent Done Done false
7081 2518 Acked Read 9 Sent Done Done false
7082 2519 Acked Write 1 Sent Done Done false
7083 2520 Acked Write 2 Sent Done Done false
7084 2521 Acked Read 6 Sent Done Done false
7085 2522 Acked Write 7 Sent Done Done false
7086 2523 Acked Write 10 Sent Done Done false
7087 2524 Acked Read 1 Sent Done Done false
7088 2525 Acked Write 1 Sent Done Done false
7089 2526 Acked Read 6 Sent Done Done false
7090 2527 Acked Write 6 Sent Done Done false
7091 2528 Acked Write 8 Sent Done Done false
7092 2529 Acked Read 10 Sent Done Done false
7093 2530 Acked Flush 0 Sent Done Done false
7094 2531 Acked Write 5 Sent Done Done false
7095 2532 Acked Read 10 Sent Done Done false
7096 2533 Acked Flush 0 Sent Done Done false
7097 2534 Acked Write 4 Sent Done Done false
7098 2535 Acked Write 2 Sent Done Done false
7099 2536 Acked Read 1 Sent Done Done false
7100 2537 Acked Read 2 Sent Done Done false
7101 2538 Acked Write 5 Sent Done Done false
7102 2539 Acked Read 8 Sent Done Done false
7103 2540 Acked Write 9 Sent Done Done false
7104 2541 Acked Read 8 Sent Done Done false
7105 2542 Acked Write 8 Sent Done Done false
7106 2543 Acked Read 10 Sent Done Done false
7107 2544 Acked Read 3 Sent Done Done false
7108 2545 Acked Write 3 Sent Done Done false
7109 2546 Acked Read 10 Sent Done Done false
7110 2547 Acked Write 3 Sent Done Done false
7111 2548 Acked Write 7 Sent Done Done false
7112 2549 Acked Read 3 Sent Done Done false
7113 2550 Acked Read 7 Sent Done Done false
7114 2551 Acked Read 3 Sent Done Done false
7115 2552 Acked Read 3 Sent Done Done false
7116 2553 Acked Write 9 Sent Done Done false
7117 2554 Acked Read 1 Sent Done Done false
7118 2555 Acked Read 7 Sent Done Done false
7119 2556 Acked Write 8 Sent Done Done false
7120 2557 Acked Write 5 Sent Done Done false
7121 2558 Acked Write 9 Sent Done Done false
7122 2559 Acked Write 6 Sent Done Done false
7123 2560 Acked Read 7 Sent Done Done false
7124 2561 Acked Read 7 Sent Done Done false
7125 2562 Acked Write 3 Sent Done Done false
7126 2563 Acked Read 4 Sent Done Done false
7127 2564 Acked Read 9 Sent Done Done false
7128 2565 Acked Read 7 Sent Done Done false
7129 2566 Acked Read 2 Sent Done Done false
7130 2567 Acked Write 7 Sent Done Done false
7131 2568 Acked Read 1 Sent Done Done false
7132 2569 Acked Read 7 Sent Done Done false
7133 2570 Acked Write 10 Sent Done Done false
7134 2571 Acked Write 5 Sent Done Done false
7135 2572 Acked Read 6 Sent Done Done false
7136 2573 Acked Read 3 Sent Done Done false
7137 2574 Acked Read 2 Sent Done Done false
7138 2575 Acked Write 9 Sent Done Done false
7139 2576 Acked Read 7 Sent Done Done false
7140 2577 Acked Read 6 Sent Done Done false
7141 2578 Acked Flush 0 Sent Done Done false
7142 2579 Acked Write 6 Sent Done Done false
7143 2580 Acked Read 8 Sent Done Done false
7144 2581 Acked Read 4 Sent Done Done false
7145 2582 Acked Write 1 Sent Done Done false
7146 2583 Acked Write 8 Sent Done Done false
7147 2584 Acked Write 2 Sent Done Done false
7148 2585 Acked Read 5 Sent Done Done false
7149 2586 Acked Read 6 Sent Done Done false
7150 2587 Acked Write 4 Sent Done Done false
7151 2588 Acked Write 9 Sent Done Done false
7152 2589 Acked Read 4 Sent Done Done false
7153 2590 Acked Read 8 Sent Done Done false
7154 2591 Acked Read 4 Sent Done Done false
7155 2592 Acked Read 1 Sent Done Done false
7156 2593 Acked Write 8 Sent Done Done false
7157 2594 Acked Write 4 Sent Done Done false
7158 2595 Acked Write 7 Sent Done Done false
7159 2596 Acked Write 5 Sent Done Done false
7160 2597 Acked Read 9 Sent Done Done false
7161 2598 Acked Read 9 Sent Done Done false
7162 2599 Acked Read 9 Sent Done Done false
7163 2600 Acked Write 2 Sent Done Done false
7164 2601 Acked Read 9 Sent Done Done false
7165 2602 Acked Write 6 Sent Done Done false
7166 2603 Acked Write 9 Sent Done Done false
7167 2604 Acked Read 7 Sent Done Done false
7168 2605 Acked Read 1 Sent Done Done false
7169 2606 Acked Read 6 Sent Done Done false
7170 2607 Acked Flush 0 Sent Done Done false
7171 2608 Acked Flush 0 Sent Done Done false
7172 2609 Acked Read 6 Sent Done Done false
7173 2610 Acked Write 6 Sent Done Done false
7174 2611 Acked Flush 0 Sent Done Done false
7175 2612 Acked Read 5 Sent Done Done false
7176 2613 Acked Read 10 Sent Done Done false
7177 2614 Acked Write 5 Sent Done Done false
7178 2615 Acked Read 6 Sent Done Done false
7179 2616 Acked Read 10 Sent Done Done false
7180 2617 Acked Write 5 Sent Done Done false
7181 2618 Acked Flush 0 Sent Done Done false
7182 2619 Acked Read 4 Sent Done Done false
7183 2620 Acked Read 6 Sent Done Done false
7184 2621 Acked Flush 0 Sent Done Done false
7185 2622 Acked Read 1 Sent Done Done false
7186 2623 Acked Write 1 Sent Done Done false
7187 2624 Acked Flush 0 Sent Done Done false
7188 2625 Acked Read 9 Sent Done Done false
7189 2626 Acked Write 9 Sent Done Done false
7190 2627 Acked Read 2 Sent Done Done false
7191 2628 Acked Read 8 Sent Done Done false
7192 2629 Acked Write 9 Sent Done Done false
7193 2630 Acked Flush 0 Sent Done Done false
7194 2631 Acked Read 8 Sent Done Done false
7195 2632 Acked Read 2 Sent Done Done false
7196 2633 Acked Read 10 Sent Done Done false
7197 2634 Acked Flush 0 Sent Done Done false
7198 2635 Acked Read 8 Sent Done Done false
7199 2636 Acked Read 9 Sent Done Done false
7200 2637 Acked Read 8 Sent Done Done false
7201 2638 Acked Write 10 Sent Done Done false
7202 2639 Acked Read 8 Sent Done Done false
7203 2640 Acked Write 3 Sent Done Done false
7204 2641 Acked Read 6 Sent Done Done false
7205 2642 Acked Read 6 Sent Done Done false
7206 2643 Acked Write 9 Sent Done Done false
7207 2644 Acked Read 6 Sent Done Done false
7208 2645 Acked Read 6 Sent Done Done false
7209 2646 Acked Read 6 Sent Done Done false
7210 2647 Acked Write 8 Sent Done Done false
7211 2648 Acked Write 10 Sent Done Done false
7212 2649 Acked Read 7 Sent Done Done false
7213 2650 Acked Write 6 Sent Done Done false
7214 2651 Acked Write 1 Sent Done Done false
7215 2652 Acked Read 8 Sent Done Done false
7216 2653 Acked Read 8 Sent Done Done false
7217 2654 Acked Write 9 Sent Done Done false
7218 2655 Acked Read 3 Sent Done Done false
7219 2656 Acked Read 5 Sent Done Done false
7220 2657 Acked Read 10 Sent Done Done false
7221 2658 Acked Read 3 Sent Done Done false
7222 2659 Acked Flush 0 Sent Done Done false
7223 2660 Acked Read 7 Sent Done Done false
7224 2661 Acked Write 8 Sent Done Done false
7225 2662 Acked Read 1 Sent Done Done false
7226 2663 Acked Read 7 Sent Done Done false
7227 2664 Acked Read 1 Sent Done Done false
7228 2665 Acked Write 3 Sent Done Done false
7229 2666 Acked Read 5 Sent Done Done false
7230 2667 Acked Read 10 Sent Done Done false
7231 2668 Acked Write 2 Sent Done Done false
7232 2669 Acked Write 9 Sent Done Done false
7233 2670 Acked Write 5 Sent Done Done false
7234 2671 Acked Read 2 Sent Done Done false
7235 2672 Acked Write 3 Sent Done Done false
7236 2673 Acked Write 5 Sent Done Done false
7237 2674 Acked Read 4 Sent Done Done false
7238 2675 Acked Read 3 Sent Done Done false
7239 2676 Acked Write 6 Sent Done Done false
7240 2677 Acked Write 3 Sent Done Done false
7241 2678 Acked Flush 0 Sent Done Done false
7242 2679 Acked Read 2 Sent Done Done false
7243 2680 Acked Read 2 Sent Done Done false
7244 2681 Acked Flush 0 Sent Done Done false
7245 2682 Acked Flush 0 Sent Done Done false
7246 2683 Acked Write 1 Sent Done Done false
7247 2684 Acked Write 8 Sent Done Done false
7248 2685 Acked Read 8 Sent Done Done false
7249 2686 Acked Read 5 Sent Done Done false
7250 2687 Acked Write 9 Sent Done Done false
7251 2688 Acked Read 8 Sent Done Done false
7252 2689 Acked Flush 0 Sent Done Done false
7253 2690 Acked Read 10 Sent Done Done false
7254 2691 Acked Read 9 Sent Done Done false
7255 2692 Acked Read 9 Sent Done Done false
7256 2693 Acked Read 1 Sent Done Done false
7257 2694 Acked Write 7 Sent Done Done false
7258 2695 Acked Read 10 Sent Done Done false
7259 2696 Acked Write 10 Sent Done Done false
7260 2697 Acked Write 3 Sent Done Done false
7261 2698 Acked Write 6 Sent Done Done false
7262 2699 Acked Read 3 Sent Done Done false
7263 2700 Acked Write 5 Sent Done Done false
7264 2701 Acked Read 7 Sent Done Done false
7265 2702 Acked Read 5 Sent Done Done false
7266 2703 Acked Write 4 Sent Done Done false
7267 2704 Acked Flush 0 Sent Done Done false
7268 2705 Acked Read 2 Sent Done Done false
7269 2706 Acked Write 10 Sent Done Done false
7270 2707 Acked Read 6 Sent Done Done false
7271 2708 Acked Flush 0 Sent Done Done false
7272 2709 Acked Read 5 Sent Done Done false
7273 2710 Acked Read 1 Sent Done Done false
7274 2711 Acked Read 1 Sent Done Done false
7275 2712 Acked Write 2 Sent Done Done false
7276 2713 Acked Write 5 Sent Done Done false
7277 2714 Acked Write 1 Sent Done Done false
7278 2715 Acked Read 7 Sent Done Done false
7279 2716 Acked Read 10 Sent Done Done false
7280 2717 Acked Read 3 Sent Done Done false
7281 2718 Acked Flush 0 Sent Done Done false
7282 2719 Acked Write 7 Sent Done Done false
7283 2720 Acked Write 2 Sent Done Done false
7284 2721 Acked Write 3 Sent Done Done false
7285 2722 Acked Read 4 Sent Done Done false
7286 2723 Acked Write 1 Sent Done Done false
7287 2724 Acked Read 7 Sent Done Done false
7288 2725 Acked Read 2 Sent Done Done false
7289 2726 Acked Read 7 Sent Done Done false
7290 2727 Acked Read 1 Sent Done Done false
7291 2728 Acked Read 3 Sent Done Done false
7292 2729 Acked Write 7 Sent Done Done false
7293 2730 Acked Read 4 Sent Done Done false
7294 2731 Acked Write 8 Sent Done Done false
7295 2732 Acked Read 1 Sent Done Done false
7296 2733 Acked Read 10 Sent Done Done false
7297 2734 Acked Read 4 Sent Done Done false
7298 2735 Acked Read 5 Sent Done Done false
7299 2736 Acked Flush 0 Sent Done Done false
7300 2737 Acked Write 10 Sent Done Done false
7301 2738 Acked Write 9 Sent Done Done false
7302 2739 Acked Write 2 Sent Done Done false
7303 2740 Acked Write 2 Sent Done Done false
7304 2741 Acked Read 3 Sent Done Done false
7305 2742 Acked Write 9 Sent Done Done false
7306 2743 Acked Read 4 Sent Done Done false
7307 2744 Acked Write 2 Sent Done Done false
7308 2745 Acked Write 5 Sent Done Done false
7309 2746 Acked Flush 0 Sent Done Done false
7310 2747 Acked Flush 0 Sent Done Done false
7311 2748 Acked Flush 0 Sent Done Done false
7312 2749 Acked Flush 0 Sent Done Done false
7313 2750 Acked Write 5 Sent Done Done false
7314 2751 Acked Read 4 Sent Done Done false
7315 2752 Acked Read 9 Sent Done Done false
7316 2753 Acked Write 5 Sent Done Done false
7317 2754 Acked Write 4 Sent Done Done false
7318 2755 Acked Write 8 Sent Done Done false
7319 2756 Acked Read 1 Sent Done Done false
7320 2757 Acked Read 3 Sent Done Done false
7321 2758 Acked Write 3 Sent Done Done false
7322 2759 Acked Write 10 Sent Done Done false
7323 2760 Acked Write 10 Sent Done Done false
7324 2761 Acked Flush 0 Sent Done Done false
7325 2762 Acked Read 10 Sent Done Done false
7326 2763 Acked Write 1 Sent Done Done false
7327 2764 Acked Flush 0 Sent Done Done false
7328 2765 Acked Read 6 Sent Done Done false
7329 2766 Acked Read 8 Sent Done Done false
7330 2767 Acked Read 9 Sent Done Done false
7331 2768 Acked Write 4 Sent Done Done false
7332 2769 Acked Read 2 Sent Done Done false
7333 2770 Acked Write 6 Sent Done Done false
7334 2771 Acked Write 1 Sent Done Done false
7335 2772 Acked Read 7 Sent Done Done false
7336 2773 Acked Write 6 Sent Done Done false
7337 2774 Acked Read 4 Sent Done Done false
7338 2775 Acked Read 7 Sent Done Done false
7339 2776 Acked Write 5 Sent Done Done false
7340 2777 Acked Write 9 Sent Done Done false
7341 2778 Acked Read 3 Sent Done Done false
7342 2779 Acked Read 6 Sent Done Done false
7343 2780 Acked Write 2 Sent Done Done false
7344 2781 Acked Write 5 Sent Done Done false
7345 2782 Acked Read 2 Sent Done Done false
7346 2783 Acked Write 4 Sent Done Done false
7347 2784 Acked Read 4 Sent Done Done false
7348 2785 Acked Read 9 Sent Done Done false
7349 2786 Acked Write 6 Sent Done Done false
7350 2787 Acked Write 2 Sent Done Done false
7351 2788 Acked Read 3 Sent Done Done false
7352 2789 Acked Write 9 Sent Done Done false
7353 2790 Acked Write 1 Sent Done Done false
7354 2791 Acked Read 2 Sent Done Done false
7355 2792 Acked Write 1 Sent Done Done false
7356 2793 Acked Write 9 Sent Done Done false
7357 2794 Acked Write 2 Sent Done Done false
7358 2795 Acked Write 4 Sent Done Done false
7359 2796 Acked Read 4 Sent Done Done false
7360 2797 Acked Read 8 Sent Done Done false
7361 2798 Acked Read 5 Sent Done Done false
7362 2799 Acked Read 6 Sent Done Done false
7363 2800 Acked Read 8 Sent Done Done false
7364 2801 Acked Write 5 Sent Done Done false
7365 2802 Acked Read 8 Sent Done Done false
7366 2803 Acked Flush 0 Sent Done Done false
7367 2804 Acked Write 2 Sent Done Done false
7368 2805 Acked Read 5 Sent Done Done false
7369 2806 Acked Flush 0 Sent Done Done false
7370 2807 Acked Write 1 Sent Done Done false
7371 2808 Acked Read 1 Sent Done Done false
7372 2809 Acked Read 2 Sent Done Done false
7373 2810 Acked Read 3 Sent Done Done false
7374 STATES DS:0 DS:1 DS:2 TOTAL
7375 Sent 301 0 0 301
7376 Done 0 301 301 602
7377 Skipped 0 0 0 0
7378 Error 0 0 0 0
7379 Last Flush: Some(JobId(2509)) Some(JobId(2806)) Some(JobId(2806))
7380 Downstairs last five completed: 2509 2508 2507 2506 2505
7381 Upstairs last five completed: 2810 2809 2808 2807 2806
7382 CLIENT: Up:0 ds:301 act:2
73832025-02-07T20:08:51.489ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
73842025-02-07T20:08:52.090ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
73852025-02-07T20:08:52.740ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
73862025-02-07T20:08:52.740ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
73872025-02-07T20:08:52.740ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
73882025-02-07T20:08:52.741ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
73892025-02-07T20:08:52.741ZINFOcrucible: client task is exiting = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
73902025-02-07T20:08:53.242ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
73912025-02-07T20:08:54.491ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
7392 ----------------------------------------------------------------
7393 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7394 States: Offline Active Active
7395 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7396 2510 Acked Write 2 Sent Done Done false
7397 2511 Acked Write 8 Sent Done Done false
7398 2512 Acked Write 3 Sent Done Done false
7399 2513 Acked Write 8 Sent Done Done false
7400 2514 Acked Read 7 Sent Done Done false
7401 2515 Acked Read 8 Sent Done Done false
7402 2516 Acked Read 2 Sent Done Done false
7403 2517 Acked Write 5 Sent Done Done false
7404 2518 Acked Read 9 Sent Done Done false
7405 2519 Acked Write 1 Sent Done Done false
7406 2520 Acked Write 2 Sent Done Done false
7407 2521 Acked Read 6 Sent Done Done false
7408 2522 Acked Write 7 Sent Done Done false
7409 2523 Acked Write 10 Sent Done Done false
7410 2524 Acked Read 1 Sent Done Done false
7411 2525 Acked Write 1 Sent Done Done false
7412 2526 Acked Read 6 Sent Done Done false
7413 2527 Acked Write 6 Sent Done Done false
7414 2528 Acked Write 8 Sent Done Done false
7415 2529 Acked Read 10 Sent Done Done false
7416 2530 Acked Flush 0 Sent Done Done false
7417 2531 Acked Write 5 Sent Done Done false
7418 2532 Acked Read 10 Sent Done Done false
7419 2533 Acked Flush 0 Sent Done Done false
7420 2534 Acked Write 4 Sent Done Done false
7421 2535 Acked Write 2 Sent Done Done false
7422 2536 Acked Read 1 Sent Done Done false
7423 2537 Acked Read 2 Sent Done Done false
7424 2538 Acked Write 5 Sent Done Done false
7425 2539 Acked Read 8 Sent Done Done false
7426 2540 Acked Write 9 Sent Done Done false
7427 2541 Acked Read 8 Sent Done Done false
7428 2542 Acked Write 8 Sent Done Done false
7429 2543 Acked Read 10 Sent Done Done false
7430 2544 Acked Read 3 Sent Done Done false
7431 2545 Acked Write 3 Sent Done Done false
7432 2546 Acked Read 10 Sent Done Done false
7433 2547 Acked Write 3 Sent Done Done false
7434 2548 Acked Write 7 Sent Done Done false
7435 2549 Acked Read 3 Sent Done Done false
7436 2550 Acked Read 7 Sent Done Done false
7437 2551 Acked Read 3 Sent Done Done false
7438 2552 Acked Read 3 Sent Done Done false
7439 2553 Acked Write 9 Sent Done Done false
7440 2554 Acked Read 1 Sent Done Done false
7441 2555 Acked Read 7 Sent Done Done false
7442 2556 Acked Write 8 Sent Done Done false
7443 2557 Acked Write 5 Sent Done Done false
7444 2558 Acked Write 9 Sent Done Done false
7445 2559 Acked Write 6 Sent Done Done false
7446 2560 Acked Read 7 Sent Done Done false
7447 2561 Acked Read 7 Sent Done Done false
7448 2562 Acked Write 3 Sent Done Done false
7449 2563 Acked Read 4 Sent Done Done false
7450 2564 Acked Read 9 Sent Done Done false
7451 2565 Acked Read 7 Sent Done Done false
7452 2566 Acked Read 2 Sent Done Done false
7453 2567 Acked Write 7 Sent Done Done false
7454 2568 Acked Read 1 Sent Done Done false
7455 2569 Acked Read 7 Sent Done Done false
7456 2570 Acked Write 10 Sent Done Done false
7457 2571 Acked Write 5 Sent Done Done false
7458 2572 Acked Read 6 Sent Done Done false
7459 2573 Acked Read 3 Sent Done Done false
7460 2574 Acked Read 2 Sent Done Done false
7461 2575 Acked Write 9 Sent Done Done false
7462 2576 Acked Read 7 Sent Done Done false
7463 2577 Acked Read 6 Sent Done Done false
7464 2578 Acked Flush 0 Sent Done Done false
7465 2579 Acked Write 6 Sent Done Done false
7466 2580 Acked Read 8 Sent Done Done false
7467 2581 Acked Read 4 Sent Done Done false
7468 2582 Acked Write 1 Sent Done Done false
7469 2583 Acked Write 8 Sent Done Done false
7470 2584 Acked Write 2 Sent Done Done false
7471 2585 Acked Read 5 Sent Done Done false
7472 2586 Acked Read 6 Sent Done Done false
7473 2587 Acked Write 4 Sent Done Done false
7474 2588 Acked Write 9 Sent Done Done false
7475 2589 Acked Read 4 Sent Done Done false
7476 2590 Acked Read 8 Sent Done Done false
7477 2591 Acked Read 4 Sent Done Done false
7478 2592 Acked Read 1 Sent Done Done false
7479 2593 Acked Write 8 Sent Done Done false
7480 2594 Acked Write 4 Sent Done Done false
7481 2595 Acked Write 7 Sent Done Done false
7482 2596 Acked Write 5 Sent Done Done false
7483 2597 Acked Read 9 Sent Done Done false
7484 2598 Acked Read 9 Sent Done Done false
7485 2599 Acked Read 9 Sent Done Done false
7486 2600 Acked Write 2 Sent Done Done false
7487 2601 Acked Read 9 Sent Done Done false
7488 2602 Acked Write 6 Sent Done Done false
7489 2603 Acked Write 9 Sent Done Done false
7490 2604 Acked Read 7 Sent Done Done false
7491 2605 Acked Read 1 Sent Done Done false
7492 2606 Acked Read 6 Sent Done Done false
7493 2607 Acked Flush 0 Sent Done Done false
7494 2608 Acked Flush 0 Sent Done Done false
7495 2609 Acked Read 6 Sent Done Done false
7496 2610 Acked Write 6 Sent Done Done false
7497 2611 Acked Flush 0 Sent Done Done false
7498 2612 Acked Read 5 Sent Done Done false
7499 2613 Acked Read 10 Sent Done Done false
7500 2614 Acked Write 5 Sent Done Done false
7501 2615 Acked Read 6 Sent Done Done false
7502 2616 Acked Read 10 Sent Done Done false
7503 2617 Acked Write 5 Sent Done Done false
7504 2618 Acked Flush 0 Sent Done Done false
7505 2619 Acked Read 4 Sent Done Done false
7506 2620 Acked Read 6 Sent Done Done false
7507 2621 Acked Flush 0 Sent Done Done false
7508 2622 Acked Read 1 Sent Done Done false
7509 2623 Acked Write 1 Sent Done Done false
7510 2624 Acked Flush 0 Sent Done Done false
7511 2625 Acked Read 9 Sent Done Done false
7512 2626 Acked Write 9 Sent Done Done false
7513 2627 Acked Read 2 Sent Done Done false
7514 2628 Acked Read 8 Sent Done Done false
7515 2629 Acked Write 9 Sent Done Done false
7516 2630 Acked Flush 0 Sent Done Done false
7517 2631 Acked Read 8 Sent Done Done false
7518 2632 Acked Read 2 Sent Done Done false
7519 2633 Acked Read 10 Sent Done Done false
7520 2634 Acked Flush 0 Sent Done Done false
7521 2635 Acked Read 8 Sent Done Done false
7522 2636 Acked Read 9 Sent Done Done false
7523 2637 Acked Read 8 Sent Done Done false
7524 2638 Acked Write 10 Sent Done Done false
7525 2639 Acked Read 8 Sent Done Done false
7526 2640 Acked Write 3 Sent Done Done false
7527 2641 Acked Read 6 Sent Done Done false
7528 2642 Acked Read 6 Sent Done Done false
7529 2643 Acked Write 9 Sent Done Done false
7530 2644 Acked Read 6 Sent Done Done false
7531 2645 Acked Read 6 Sent Done Done false
7532 2646 Acked Read 6 Sent Done Done false
7533 2647 Acked Write 8 Sent Done Done false
7534 2648 Acked Write 10 Sent Done Done false
7535 2649 Acked Read 7 Sent Done Done false
7536 2650 Acked Write 6 Sent Done Done false
7537 2651 Acked Write 1 Sent Done Done false
7538 2652 Acked Read 8 Sent Done Done false
7539 2653 Acked Read 8 Sent Done Done false
7540 2654 Acked Write 9 Sent Done Done false
7541 2655 Acked Read 3 Sent Done Done false
7542 2656 Acked Read 5 Sent Done Done false
7543 2657 Acked Read 10 Sent Done Done false
7544 2658 Acked Read 3 Sent Done Done false
7545 2659 Acked Flush 0 Sent Done Done false
7546 2660 Acked Read 7 Sent Done Done false
7547 2661 Acked Write 8 Sent Done Done false
7548 2662 Acked Read 1 Sent Done Done false
7549 2663 Acked Read 7 Sent Done Done false
7550 2664 Acked Read 1 Sent Done Done false
7551 2665 Acked Write 3 Sent Done Done false
7552 2666 Acked Read 5 Sent Done Done false
7553 2667 Acked Read 10 Sent Done Done false
7554 2668 Acked Write 2 Sent Done Done false
7555 2669 Acked Write 9 Sent Done Done false
7556 2670 Acked Write 5 Sent Done Done false
7557 2671 Acked Read 2 Sent Done Done false
7558 2672 Acked Write 3 Sent Done Done false
7559 2673 Acked Write 5 Sent Done Done false
7560 2674 Acked Read 4 Sent Done Done false
7561 2675 Acked Read 3 Sent Done Done false
7562 2676 Acked Write 6 Sent Done Done false
7563 2677 Acked Write 3 Sent Done Done false
7564 2678 Acked Flush 0 Sent Done Done false
7565 2679 Acked Read 2 Sent Done Done false
7566 2680 Acked Read 2 Sent Done Done false
7567 2681 Acked Flush 0 Sent Done Done false
7568 2682 Acked Flush 0 Sent Done Done false
7569 2683 Acked Write 1 Sent Done Done false
7570 2684 Acked Write 8 Sent Done Done false
7571 2685 Acked Read 8 Sent Done Done false
7572 2686 Acked Read 5 Sent Done Done false
7573 2687 Acked Write 9 Sent Done Done false
7574 2688 Acked Read 8 Sent Done Done false
7575 2689 Acked Flush 0 Sent Done Done false
7576 2690 Acked Read 10 Sent Done Done false
7577 2691 Acked Read 9 Sent Done Done false
7578 2692 Acked Read 9 Sent Done Done false
7579 2693 Acked Read 1 Sent Done Done false
7580 2694 Acked Write 7 Sent Done Done false
7581 2695 Acked Read 10 Sent Done Done false
7582 2696 Acked Write 10 Sent Done Done false
7583 2697 Acked Write 3 Sent Done Done false
7584 2698 Acked Write 6 Sent Done Done false
7585 2699 Acked Read 3 Sent Done Done false
7586 2700 Acked Write 5 Sent Done Done false
7587 2701 Acked Read 7 Sent Done Done false
7588 2702 Acked Read 5 Sent Done Done false
7589 2703 Acked Write 4 Sent Done Done false
7590 2704 Acked Flush 0 Sent Done Done false
7591 2705 Acked Read 2 Sent Done Done false
7592 2706 Acked Write 10 Sent Done Done false
7593 2707 Acked Read 6 Sent Done Done false
7594 2708 Acked Flush 0 Sent Done Done false
7595 2709 Acked Read 5 Sent Done Done false
7596 2710 Acked Read 1 Sent Done Done false
7597 2711 Acked Read 1 Sent Done Done false
7598 2712 Acked Write 2 Sent Done Done false
7599 2713 Acked Write 5 Sent Done Done false
7600 2714 Acked Write 1 Sent Done Done false
7601 2715 Acked Read 7 Sent Done Done false
7602 2716 Acked Read 10 Sent Done Done false
7603 2717 Acked Read 3 Sent Done Done false
7604 2718 Acked Flush 0 Sent Done Done false
7605 2719 Acked Write 7 Sent Done Done false
7606 2720 Acked Write 2 Sent Done Done false
7607 2721 Acked Write 3 Sent Done Done false
7608 2722 Acked Read 4 Sent Done Done false
7609 2723 Acked Write 1 Sent Done Done false
7610 2724 Acked Read 7 Sent Done Done false
7611 2725 Acked Read 2 Sent Done Done false
7612 2726 Acked Read 7 Sent Done Done false
7613 2727 Acked Read 1 Sent Done Done false
7614 2728 Acked Read 3 Sent Done Done false
7615 2729 Acked Write 7 Sent Done Done false
7616 2730 Acked Read 4 Sent Done Done false
7617 2731 Acked Write 8 Sent Done Done false
7618 2732 Acked Read 1 Sent Done Done false
7619 2733 Acked Read 10 Sent Done Done false
7620 2734 Acked Read 4 Sent Done Done false
7621 2735 Acked Read 5 Sent Done Done false
7622 2736 Acked Flush 0 Sent Done Done false
7623 2737 Acked Write 10 Sent Done Done false
7624 2738 Acked Write 9 Sent Done Done false
7625 2739 Acked Write 2 Sent Done Done false
7626 2740 Acked Write 2 Sent Done Done false
7627 2741 Acked Read 3 Sent Done Done false
7628 2742 Acked Write 9 Sent Done Done false
7629 2743 Acked Read 4 Sent Done Done false
7630 2744 Acked Write 2 Sent Done Done false
7631 2745 Acked Write 5 Sent Done Done false
7632 2746 Acked Flush 0 Sent Done Done false
7633 2747 Acked Flush 0 Sent Done Done false
7634 2748 Acked Flush 0 Sent Done Done false
7635 2749 Acked Flush 0 Sent Done Done false
7636 2750 Acked Write 5 Sent Done Done false
7637 2751 Acked Read 4 Sent Done Done false
7638 2752 Acked Read 9 Sent Done Done false
7639 2753 Acked Write 5 Sent Done Done false
7640 2754 Acked Write 4 Sent Done Done false
7641 2755 Acked Write 8 Sent Done Done false
7642 2756 Acked Read 1 Sent Done Done false
7643 2757 Acked Read 3 Sent Done Done false
7644 2758 Acked Write 3 Sent Done Done false
7645 2759 Acked Write 10 Sent Done Done false
7646 2760 Acked Write 10 Sent Done Done false
7647 2761 Acked Flush 0 Sent Done Done false
7648 2762 Acked Read 10 Sent Done Done false
7649 2763 Acked Write 1 Sent Done Done false
7650 2764 Acked Flush 0 Sent Done Done false
7651 2765 Acked Read 6 Sent Done Done false
7652 2766 Acked Read 8 Sent Done Done false
7653 2767 Acked Read 9 Sent Done Done false
7654 2768 Acked Write 4 Sent Done Done false
7655 2769 Acked Read 2 Sent Done Done false
7656 2770 Acked Write 6 Sent Done Done false
7657 2771 Acked Write 1 Sent Done Done false
7658 2772 Acked Read 7 Sent Done Done false
7659 2773 Acked Write 6 Sent Done Done false
7660 2774 Acked Read 4 Sent Done Done false
7661 2775 Acked Read 7 Sent Done Done false
7662 2776 Acked Write 5 Sent Done Done false
7663 2777 Acked Write 9 Sent Done Done false
7664 2778 Acked Read 3 Sent Done Done false
7665 2779 Acked Read 6 Sent Done Done false
7666 2780 Acked Write 2 Sent Done Done false
7667 2781 Acked Write 5 Sent Done Done false
7668 2782 Acked Read 2 Sent Done Done false
7669 2783 Acked Write 4 Sent Done Done false
7670 2784 Acked Read 4 Sent Done Done false
7671 2785 Acked Read 9 Sent Done Done false
7672 2786 Acked Write 6 Sent Done Done false
7673 2787 Acked Write 2 Sent Done Done false
7674 2788 Acked Read 3 Sent Done Done false
7675 2789 Acked Write 9 Sent Done Done false
7676 2790 Acked Write 1 Sent Done Done false
7677 2791 Acked Read 2 Sent Done Done false
7678 2792 Acked Write 1 Sent Done Done false
7679 2793 Acked Write 9 Sent Done Done false
7680 2794 Acked Write 2 Sent Done Done false
7681 2795 Acked Write 4 Sent Done Done false
7682 2796 Acked Read 4 Sent Done Done false
7683 2797 Acked Read 8 Sent Done Done false
7684 2798 Acked Read 5 Sent Done Done false
7685 2799 Acked Read 6 Sent Done Done false
7686 2800 Acked Read 8 Sent Done Done false
7687 2801 Acked Write 5 Sent Done Done false
7688 2802 Acked Read 8 Sent Done Done false
7689 2803 Acked Flush 0 Sent Done Done false
7690 2804 Acked Write 2 Sent Done Done false
7691 2805 Acked Read 5 Sent Done Done false
7692 2806 Acked Flush 0 Sent Done Done false
7693 2807 Acked Write 1 Sent Done Done false
7694 2808 Acked Read 1 Sent Done Done false
7695 2809 Acked Read 2 Sent Done Done false
7696 2810 Acked Read 3 Sent Done Done false
7697 STATES DS:0 DS:1 DS:2 TOTAL
7698 Sent 301 0 0 301
7699 Done 0 301 301 602
7700 Skipped 0 0 0 0
7701 Error 0 0 0 0
7702 Last Flush: Some(JobId(2509)) Some(JobId(2806)) Some(JobId(2806))
7703 Downstairs last five completed: 2509 2508 2507 2506 2505
7704 Upstairs last five completed: 2810 2809 2808 2807 2806
7705 CLIENT: Up:0 ds:301 act:2
77062025-02-07T20:08:55.491ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
77072025-02-07T20:08:56.095ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
7708 [8810] got start action
7709 [8810] I am going to restart
7710 Starting downstairs at port 8810
7711 Make output file at "/tmp/dsc/downstairs-8810.txt"
7712 [8810][0] reports Starting
7713 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1215)
7714 [8810][0] reports Running
77152025-02-07T20:08:57.097ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
77162025-02-07T20:08:58.099ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
7717 ----------------------------------------------------------------
7718 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7719 States: Offline Active Active
7720 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7721 2510 Acked Write 2 Sent Done Done false
7722 2511 Acked Write 8 Sent Done Done false
7723 2512 Acked Write 3 Sent Done Done false
7724 2513 Acked Write 8 Sent Done Done false
7725 2514 Acked Read 7 Sent Done Done false
7726 2515 Acked Read 8 Sent Done Done false
7727 2516 Acked Read 2 Sent Done Done false
7728 2517 Acked Write 5 Sent Done Done false
7729 2518 Acked Read 9 Sent Done Done false
7730 2519 Acked Write 1 Sent Done Done false
7731 2520 Acked Write 2 Sent Done Done false
7732 2521 Acked Read 6 Sent Done Done false
7733 2522 Acked Write 7 Sent Done Done false
7734 2523 Acked Write 10 Sent Done Done false
7735 2524 Acked Read 1 Sent Done Done false
7736 2525 Acked Write 1 Sent Done Done false
7737 2526 Acked Read 6 Sent Done Done false
7738 2527 Acked Write 6 Sent Done Done false
7739 2528 Acked Write 8 Sent Done Done false
7740 2529 Acked Read 10 Sent Done Done false
7741 2530 Acked Flush 0 Sent Done Done false
7742 2531 Acked Write 5 Sent Done Done false
7743 2532 Acked Read 10 Sent Done Done false
7744 2533 Acked Flush 0 Sent Done Done false
7745 2534 Acked Write 4 Sent Done Done false
7746 2535 Acked Write 2 Sent Done Done false
7747 2536 Acked Read 1 Sent Done Done false
7748 2537 Acked Read 2 Sent Done Done false
7749 2538 Acked Write 5 Sent Done Done false
7750 2539 Acked Read 8 Sent Done Done false
7751 2540 Acked Write 9 Sent Done Done false
7752 2541 Acked Read 8 Sent Done Done false
7753 2542 Acked Write 8 Sent Done Done false
7754 2543 Acked Read 10 Sent Done Done false
7755 2544 Acked Read 3 Sent Done Done false
7756 2545 Acked Write 3 Sent Done Done false
7757 2546 Acked Read 10 Sent Done Done false
7758 2547 Acked Write 3 Sent Done Done false
7759 2548 Acked Write 7 Sent Done Done false
7760 2549 Acked Read 3 Sent Done Done false
7761 2550 Acked Read 7 Sent Done Done false
7762 2551 Acked Read 3 Sent Done Done false
7763 2552 Acked Read 3 Sent Done Done false
7764 2553 Acked Write 9 Sent Done Done false
7765 2554 Acked Read 1 Sent Done Done false
7766 2555 Acked Read 7 Sent Done Done false
7767 2556 Acked Write 8 Sent Done Done false
7768 2557 Acked Write 5 Sent Done Done false
7769 2558 Acked Write 9 Sent Done Done false
7770 2559 Acked Write 6 Sent Done Done false
7771 2560 Acked Read 7 Sent Done Done false
7772 2561 Acked Read 7 Sent Done Done false
7773 2562 Acked Write 3 Sent Done Done false
7774 2563 Acked Read 4 Sent Done Done false
7775 2564 Acked Read 9 Sent Done Done false
7776 2565 Acked Read 7 Sent Done Done false
7777 2566 Acked Read 2 Sent Done Done false
7778 2567 Acked Write 7 Sent Done Done false
7779 2568 Acked Read 1 Sent Done Done false
7780 2569 Acked Read 7 Sent Done Done false
7781 2570 Acked Write 10 Sent Done Done false
7782 2571 Acked Write 5 Sent Done Done false
7783 2572 Acked Read 6 Sent Done Done false
7784 2573 Acked Read 3 Sent Done Done false
7785 2574 Acked Read 2 Sent Done Done false
7786 2575 Acked Write 9 Sent Done Done false
7787 2576 Acked Read 7 Sent Done Done false
7788 2577 Acked Read 6 Sent Done Done false
7789 2578 Acked Flush 0 Sent Done Done false
7790 2579 Acked Write 6 Sent Done Done false
7791 2580 Acked Read 8 Sent Done Done false
7792 2581 Acked Read 4 Sent Done Done false
7793 2582 Acked Write 1 Sent Done Done false
7794 2583 Acked Write 8 Sent Done Done false
7795 2584 Acked Write 2 Sent Done Done false
7796 2585 Acked Read 5 Sent Done Done false
7797 2586 Acked Read 6 Sent Done Done false
7798 2587 Acked Write 4 Sent Done Done false
7799 2588 Acked Write 9 Sent Done Done false
7800 2589 Acked Read 4 Sent Done Done false
7801 2590 Acked Read 8 Sent Done Done false
7802 2591 Acked Read 4 Sent Done Done false
7803 2592 Acked Read 1 Sent Done Done false
7804 2593 Acked Write 8 Sent Done Done false
7805 2594 Acked Write 4 Sent Done Done false
7806 2595 Acked Write 7 Sent Done Done false
7807 2596 Acked Write 5 Sent Done Done false
7808 2597 Acked Read 9 Sent Done Done false
7809 2598 Acked Read 9 Sent Done Done false
7810 2599 Acked Read 9 Sent Done Done false
7811 2600 Acked Write 2 Sent Done Done false
7812 2601 Acked Read 9 Sent Done Done false
7813 2602 Acked Write 6 Sent Done Done false
7814 2603 Acked Write 9 Sent Done Done false
7815 2604 Acked Read 7 Sent Done Done false
7816 2605 Acked Read 1 Sent Done Done false
7817 2606 Acked Read 6 Sent Done Done false
7818 2607 Acked Flush 0 Sent Done Done false
7819 2608 Acked Flush 0 Sent Done Done false
7820 2609 Acked Read 6 Sent Done Done false
7821 2610 Acked Write 6 Sent Done Done false
7822 2611 Acked Flush 0 Sent Done Done false
7823 2612 Acked Read 5 Sent Done Done false
7824 2613 Acked Read 10 Sent Done Done false
7825 2614 Acked Write 5 Sent Done Done false
7826 2615 Acked Read 6 Sent Done Done false
7827 2616 Acked Read 10 Sent Done Done false
7828 2617 Acked Write 5 Sent Done Done false
7829 2618 Acked Flush 0 Sent Done Done false
7830 2619 Acked Read 4 Sent Done Done false
7831 2620 Acked Read 6 Sent Done Done false
7832 2621 Acked Flush 0 Sent Done Done false
7833 2622 Acked Read 1 Sent Done Done false
7834 2623 Acked Write 1 Sent Done Done false
7835 2624 Acked Flush 0 Sent Done Done false
7836 2625 Acked Read 9 Sent Done Done false
7837 2626 Acked Write 9 Sent Done Done false
7838 2627 Acked Read 2 Sent Done Done false
7839 2628 Acked Read 8 Sent Done Done false
7840 2629 Acked Write 9 Sent Done Done false
7841 2630 Acked Flush 0 Sent Done Done false
7842 2631 Acked Read 8 Sent Done Done false
7843 2632 Acked Read 2 Sent Done Done false
7844 2633 Acked Read 10 Sent Done Done false
7845 2634 Acked Flush 0 Sent Done Done false
7846 2635 Acked Read 8 Sent Done Done false
7847 2636 Acked Read 9 Sent Done Done false
7848 2637 Acked Read 8 Sent Done Done false
7849 2638 Acked Write 10 Sent Done Done false
7850 2639 Acked Read 8 Sent Done Done false
7851 2640 Acked Write 3 Sent Done Done false
7852 2641 Acked Read 6 Sent Done Done false
7853 2642 Acked Read 6 Sent Done Done false
7854 2643 Acked Write 9 Sent Done Done false
7855 2644 Acked Read 6 Sent Done Done false
7856 2645 Acked Read 6 Sent Done Done false
7857 2646 Acked Read 6 Sent Done Done false
7858 2647 Acked Write 8 Sent Done Done false
7859 2648 Acked Write 10 Sent Done Done false
7860 2649 Acked Read 7 Sent Done Done false
7861 2650 Acked Write 6 Sent Done Done false
7862 2651 Acked Write 1 Sent Done Done false
7863 2652 Acked Read 8 Sent Done Done false
7864 2653 Acked Read 8 Sent Done Done false
7865 2654 Acked Write 9 Sent Done Done false
7866 2655 Acked Read 3 Sent Done Done false
7867 2656 Acked Read 5 Sent Done Done false
7868 2657 Acked Read 10 Sent Done Done false
7869 2658 Acked Read 3 Sent Done Done false
7870 2659 Acked Flush 0 Sent Done Done false
7871 2660 Acked Read 7 Sent Done Done false
7872 2661 Acked Write 8 Sent Done Done false
7873 2662 Acked Read 1 Sent Done Done false
7874 2663 Acked Read 7 Sent Done Done false
7875 2664 Acked Read 1 Sent Done Done false
7876 2665 Acked Write 3 Sent Done Done false
7877 2666 Acked Read 5 Sent Done Done false
7878 2667 Acked Read 10 Sent Done Done false
7879 2668 Acked Write 2 Sent Done Done false
7880 2669 Acked Write 9 Sent Done Done false
7881 2670 Acked Write 5 Sent Done Done false
7882 2671 Acked Read 2 Sent Done Done false
7883 2672 Acked Write 3 Sent Done Done false
7884 2673 Acked Write 5 Sent Done Done false
7885 2674 Acked Read 4 Sent Done Done false
7886 2675 Acked Read 3 Sent Done Done false
7887 2676 Acked Write 6 Sent Done Done false
7888 2677 Acked Write 3 Sent Done Done false
7889 2678 Acked Flush 0 Sent Done Done false
7890 2679 Acked Read 2 Sent Done Done false
7891 2680 Acked Read 2 Sent Done Done false
7892 2681 Acked Flush 0 Sent Done Done false
7893 2682 Acked Flush 0 Sent Done Done false
7894 2683 Acked Write 1 Sent Done Done false
7895 2684 Acked Write 8 Sent Done Done false
7896 2685 Acked Read 8 Sent Done Done false
7897 2686 Acked Read 5 Sent Done Done false
7898 2687 Acked Write 9 Sent Done Done false
7899 2688 Acked Read 8 Sent Done Done false
7900 2689 Acked Flush 0 Sent Done Done false
7901 2690 Acked Read 10 Sent Done Done false
7902 2691 Acked Read 9 Sent Done Done false
7903 2692 Acked Read 9 Sent Done Done false
7904 2693 Acked Read 1 Sent Done Done false
7905 2694 Acked Write 7 Sent Done Done false
7906 2695 Acked Read 10 Sent Done Done false
7907 2696 Acked Write 10 Sent Done Done false
7908 2697 Acked Write 3 Sent Done Done false
7909 2698 Acked Write 6 Sent Done Done false
7910 2699 Acked Read 3 Sent Done Done false
7911 2700 Acked Write 5 Sent Done Done false
7912 2701 Acked Read 7 Sent Done Done false
7913 2702 Acked Read 5 Sent Done Done false
7914 2703 Acked Write 4 Sent Done Done false
7915 2704 Acked Flush 0 Sent Done Done false
7916 2705 Acked Read 2 Sent Done Done false
7917 2706 Acked Write 10 Sent Done Done false
7918 2707 Acked Read 6 Sent Done Done false
7919 2708 Acked Flush 0 Sent Done Done false
7920 2709 Acked Read 5 Sent Done Done false
7921 2710 Acked Read 1 Sent Done Done false
7922 2711 Acked Read 1 Sent Done Done false
7923 2712 Acked Write 2 Sent Done Done false
7924 2713 Acked Write 5 Sent Done Done false
7925 2714 Acked Write 1 Sent Done Done false
7926 2715 Acked Read 7 Sent Done Done false
7927 2716 Acked Read 10 Sent Done Done false
7928 2717 Acked Read 3 Sent Done Done false
7929 2718 Acked Flush 0 Sent Done Done false
7930 2719 Acked Write 7 Sent Done Done false
7931 2720 Acked Write 2 Sent Done Done false
7932 2721 Acked Write 3 Sent Done Done false
7933 2722 Acked Read 4 Sent Done Done false
7934 2723 Acked Write 1 Sent Done Done false
7935 2724 Acked Read 7 Sent Done Done false
7936 2725 Acked Read 2 Sent Done Done false
7937 2726 Acked Read 7 Sent Done Done false
7938 2727 Acked Read 1 Sent Done Done false
7939 2728 Acked Read 3 Sent Done Done false
7940 2729 Acked Write 7 Sent Done Done false
7941 2730 Acked Read 4 Sent Done Done false
7942 2731 Acked Write 8 Sent Done Done false
7943 2732 Acked Read 1 Sent Done Done false
7944 2733 Acked Read 10 Sent Done Done false
7945 2734 Acked Read 4 Sent Done Done false
7946 2735 Acked Read 5 Sent Done Done false
7947 2736 Acked Flush 0 Sent Done Done false
7948 2737 Acked Write 10 Sent Done Done false
7949 2738 Acked Write 9 Sent Done Done false
7950 2739 Acked Write 2 Sent Done Done false
7951 2740 Acked Write 2 Sent Done Done false
7952 2741 Acked Read 3 Sent Done Done false
7953 2742 Acked Write 9 Sent Done Done false
7954 2743 Acked Read 4 Sent Done Done false
7955 2744 Acked Write 2 Sent Done Done false
7956 2745 Acked Write 5 Sent Done Done false
7957 2746 Acked Flush 0 Sent Done Done false
7958 2747 Acked Flush 0 Sent Done Done false
7959 2748 Acked Flush 0 Sent Done Done false
7960 2749 Acked Flush 0 Sent Done Done false
7961 2750 Acked Write 5 Sent Done Done false
7962 2751 Acked Read 4 Sent Done Done false
7963 2752 Acked Read 9 Sent Done Done false
7964 2753 Acked Write 5 Sent Done Done false
7965 2754 Acked Write 4 Sent Done Done false
7966 2755 Acked Write 8 Sent Done Done false
7967 2756 Acked Read 1 Sent Done Done false
7968 2757 Acked Read 3 Sent Done Done false
7969 2758 Acked Write 3 Sent Done Done false
7970 2759 Acked Write 10 Sent Done Done false
7971 2760 Acked Write 10 Sent Done Done false
7972 2761 Acked Flush 0 Sent Done Done false
7973 2762 Acked Read 10 Sent Done Done false
7974 2763 Acked Write 1 Sent Done Done false
7975 2764 Acked Flush 0 Sent Done Done false
7976 2765 Acked Read 6 Sent Done Done false
7977 2766 Acked Read 8 Sent Done Done false
7978 2767 Acked Read 9 Sent Done Done false
7979 2768 Acked Write 4 Sent Done Done false
7980 2769 Acked Read 2 Sent Done Done false
7981 2770 Acked Write 6 Sent Done Done false
7982 2771 Acked Write 1 Sent Done Done false
7983 2772 Acked Read 7 Sent Done Done false
7984 2773 Acked Write 6 Sent Done Done false
7985 2774 Acked Read 4 Sent Done Done false
7986 2775 Acked Read 7 Sent Done Done false
7987 2776 Acked Write 5 Sent Done Done false
7988 2777 Acked Write 9 Sent Done Done false
7989 2778 Acked Read 3 Sent Done Done false
7990 2779 Acked Read 6 Sent Done Done false
7991 2780 Acked Write 2 Sent Done Done false
7992 2781 Acked Write 5 Sent Done Done false
7993 2782 Acked Read 2 Sent Done Done false
7994 2783 Acked Write 4 Sent Done Done false
7995 2784 Acked Read 4 Sent Done Done false
7996 2785 Acked Read 9 Sent Done Done false
7997 2786 Acked Write 6 Sent Done Done false
7998 2787 Acked Write 2 Sent Done Done false
7999 2788 Acked Read 3 Sent Done Done false
8000 2789 Acked Write 9 Sent Done Done false
8001 2790 Acked Write 1 Sent Done Done false
8002 2791 Acked Read 2 Sent Done Done false
8003 2792 Acked Write 1 Sent Done Done false
8004 2793 Acked Write 9 Sent Done Done false
8005 2794 Acked Write 2 Sent Done Done false
8006 2795 Acked Write 4 Sent Done Done false
8007 2796 Acked Read 4 Sent Done Done false
8008 2797 Acked Read 8 Sent Done Done false
8009 2798 Acked Read 5 Sent Done Done false
8010 2799 Acked Read 6 Sent Done Done false
8011 2800 Acked Read 8 Sent Done Done false
8012 2801 Acked Write 5 Sent Done Done false
8013 2802 Acked Read 8 Sent Done Done false
8014 2803 Acked Flush 0 Sent Done Done false
8015 2804 Acked Write 2 Sent Done Done false
8016 2805 Acked Read 5 Sent Done Done false
8017 2806 Acked Flush 0 Sent Done Done false
8018 2807 Acked Write 1 Sent Done Done false
8019 2808 Acked Read 1 Sent Done Done false
8020 2809 Acked Read 2 Sent Done Done false
8021 2810 Acked Read 3 Sent Done Done false
8022 STATES DS:0 DS:1 DS:2 TOTAL
8023 Sent 301 0 0 301
8024 Done 0 301 301 602
8025 Skipped 0 0 0 0
8026 Error 0 0 0 0
8027 Last Flush: Some(JobId(2509)) Some(JobId(2806)) Some(JobId(2806))
8028 Downstairs last five completed: 2509 2508 2507 2506 2505
8029 Upstairs last five completed: 2810 2809 2808 2807 2806
8030 CLIENT: Up:0 ds:301 act:2
80312025-02-07T20:08:59.494ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80322025-02-07T20:09:00.102ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80332025-02-07T20:09:01.103ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80342025-02-07T20:09:02.105ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80352025-02-07T20:09:02.742ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80362025-02-07T20:09:02.742ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41055) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80372025-02-07T20:09:02.743ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80382025-02-07T20:09:02.744ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80392025-02-07T20:09:02.744ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80402025-02-07T20:09:02.744ZINFOcrucible: send last flush ID to this DS: Some(JobId(2509)) = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80412025-02-07T20:09:02.744ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80422025-02-07T20:09:02.744ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2509)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80432025-02-07T20:09:02.744ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2509)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
8044 ----------------------------------------------------------------
8045 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:4
8046 States: Active Active Active
8047 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8048 2807 Acked Write 1 Done Done Done true
8049 2808 Acked Read 1 Done Done Done true
8050 2809 Acked Read 2 Done Done Done true
8051 2810 Acked Read 3 Done Done Done true
8052 STATES DS:0 DS:1 DS:2 TOTAL
8053 Sent 0 0 0 0
8054 Done 4 4 4 12
8055 Skipped 0 0 0 0
8056 Error 0 0 0 0
8057 Last Flush: Some(JobId(2806)) Some(JobId(2806)) Some(JobId(2806))
8058 Downstairs last five completed: 2806 2805 2804 2803 2802
8059 Upstairs last five completed: 2810 2809 2808 2807 2806
8060 CLIENT: Up:0 ds:4 act:3
8061 ----------------------------------------------------------------
8062 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8063 Downstairs last five completed: 2811 2810 2809 2808 2807
8064 Upstairs last five completed: 2811 2810 2809 2808 2807
8065 CLIENT: Up:0 ds:0 act:3
8066 Replay: All jobs finished
8067 Main task has work to do, go find it
8068 got dsc Stop(0)
8069 stop 0
8070 Feb 07 20:09:06.998 INFO request completed[8810] Got stop action so:false kr:false
8071 , latency_us: 369, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 87e6b779-93ad-4aac-8e64-f59a2d686140, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
8072 Feb 07 20:09:06.998 INFO request completed, latency_us: 127, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4480bfe0-46c9-4ba2-ac51-23c99a695403, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
80732025-02-07T20:09:06.999ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80742025-02-07T20:09:06.999ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80752025-02-07T20:09:06.999ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
80762025-02-07T20:09:06.999ZINFOcrucible: client task is exiting = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
8077 [8810] Exited with: ExitStatus(unix_wait_status(9))
8078 [8810][0] reports Exit
8079 Feb 07 20:09:11.001 INFO request completed, latency_us: 216, response_code: 200, uri: /state/cid/0, method: GET, req_id: df63a2fb-35d2-4f2e-976b-95e586f7d263, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
8080 001/300 Read block 3903 len 20480
8081 002/300 Write block 4817 len 36864 data: 3 3 2 2 2 2 2 2 2
8082 003/300 Flush
8083 004/300 Flush
8084 005/300 Write block 2392 len 4096 data: 3
8085 006/300 Write block 1347 len 20480 data: 3 2 2 2 2
8086 007/300 Write block 4022 len 36864 data: 2 2 3 2 2 2 2 2 2
8087 008/300 Read block 3477 len 28672
8088 009/300 Read block 3495 len 16384
8089 010/300 Read block 3956 len 8192
8090 011/300 Write block 11 len 28672 data: 2 3 3 3 3 2 2
8091 012/300 Read block 1930 len 24576
8092 013/300 Read block 3512 len 16384
8093 014/300 Write block 3078 len 16384 data: 3 3 3 3
8094 015/300 Read block 214 len 8192
8095 016/300 Read block 239 len 36864
8096 017/300 Write block 4810 len 36864 data: 2 2 2 2 2 3 3 4 4
8097 018/300 Write block 3753 len 40960 data: 3 3 3 3 3 4 3 3 3 3
8098 019/300 Write block 312 len 28672 data: 4 3 3 3 2 2 2
8099 020/300 Write block 810 len 16384 data: 2 2 2 4
8100 021/300 Write block 379 len 16384 data: 2 2 2 2
8101 022/300 Write block 2203 len 36864 data: 4 4 4 4 4 3 3 3 2
8102 023/300 Read block 1111 len 8192
8103 024/300 Read block 3691 len 8192
8104 025/300 Read block 3878 len 24576
8105 026/300 Read block 426 len 20480
8106 027/300 Write block 2237 len 28672 data: 3 3 3 3 2 2 2
8107 028/300 Read block 2636 len 12288
8108 029/300 Write block 4788 len 24576 data: 5 5 5 4 3 3
8109 030/300 Read block 3131 len 40960
8110 031/300 Read block 1003 len 36864
8111 032/300 Read block 457 len 28672
8112 033/300 Read block 1712 len 32768
8113 034/300 Read block 3393 len 20480
8114 035/300 Write block 3911 len 8192 data: 2 4
8115 036/300 Read block 1630 len 40960
8116 037/300 Read block 2473 len 16384
8117 038/300 Read block 131 len 20480
8118 039/300 Write block 4754 len 16384 data: 2 2 2 2
8119 040/300 Read block 2723 len 28672
8120 041/300 Flush
8121 042/300 Flush
8122 043/300 Read block 3948 len 20480
8123 044/300 Read block 1577 len 36864
8124 045/300 Write block 4922 len 24576 data: 2 2 2 2 2 2
8125 046/300 Read block 3522 len 20480
8126 047/300 Read block 1497 len 28672
8127 048/300 Write block 4166 len 16384 data: 2 2 2 2
8128 049/300 Read block 1701 len 4096
8129 050/300 Write block 3755 len 8192 data: 4 4
8130 051/300 Write block 618 len 4096 data: 4
8131 052/300 Read block 56 len 40960
8132 053/300 Read block 357 len 40960
8133 054/300 Read block 2249 len 32768
8134 055/300 Write block 3366 len 36864 data: 5 4 3 2 2 2 2 2 2
8135 056/300 Flush
8136 057/300 Read block 3893 len 12288
8137 058/300 Read block 2471 len 24576
8138 059/300 Flush
8139 060/300 Write block 3324 len 16384 data: 3 3 3 3
8140 061/300 Write block 4890 len 40960 data: 4 3 2 2 2 4 4 5 6 6
8141 062/300 Read block 3878 len 4096
8142 063/300 Read block 1393 len 12288
8143 064/300 Write block 3303 len 28672 data: 4 3 3 3 3 3 2
8144 065/300 Write block 3634 len 28672 data: 2 2 2 2 2 2 2
8145 066/300 Read block 3406 len 32768
8146 067/300 Read block 1890 len 40960
8147 068/300 Read block 3339 len 32768
8148 069/300 Write block 635 len 12288 data: 2 2 2
8149 070/300 Read block 2543 len 40960
8150 071/300 Read block 2532 len 20480
8151 072/300 Write block 3248 len 28672 data: 4 4 4 5 5 5 5
8152 073/300 Flush
8153 074/300 Write block 4089 len 24576 data: 2 2 2 2 2 2
8154 075/300 Flush
8155 076/300 Read block 1410 len 28672
8156 077/300 Write block 990 len 12288 data: 2 2 3
8157 078/300 Write block 3105 len 40960 data: 3 3 3 3 3 3 3 2 2 2
8158 079/300 Read block 2289 len 8192
8159 080/300 Read block 419 len 40960
8160 081/300 Write block 4738 len 24576 data: 3 3 3 3 3 4
8161 082/300 Read block 1749 len 8192
8162 083/300 Read block 1914 len 12288
8163 084/300 Read block 2584 len 28672
8164 085/300 Flush
8165 086/300 Read block 1117 len 36864
8166 087/300 Read block 1233 len 8192
8167 088/300 Read block 2359 len 20480
8168 089/300 Read block 3692 len 16384
8169 090/300 Read block 4851 len 28672
8170 091/300 Write block 2463 len 32768 data: 2 2 2 2 2 2 2 2
8171 092/300 Write block 4310 len 4096 data: 2
8172 093/300 Write block 3569 len 36864 data: 2 3 3 3 3 3 3 3 3
8173 094/300 Flush
8174 095/300 Flush
8175 096/300 Read block 1709 len 28672
8176 097/300 Write block 2367 len 36864 data: 2 3 2 2 2 2 2 2 3
8177 098/300 Read block 4535 len 8192
8178 099/300 Write block 1283 len 24576 data: 4 4 3 3 2 2
8179 100/300 Flush
8180 101/300 Flush
8181 102/300 Write block 1496 len 32768 data: 3 2 2 2 2 2 2 3
8182 103/300 Read block 3543 len 4096
8183 104/300 Read block 3901 len 32768
8184 105/300 Flush
8185 106/300 Write block 1251 len 16384 data: 2 2 2 2
8186 107/300 Read block 1053 len 32768
8187 108/300 Write block 1292 len 12288 data: 2 2 2
8188 109/300 Read block 1496 len 32768
8189 110/300 Write block 2602 len 36864 data: 3 3 3 3 3 3 2 2 2
8190 111/300 Read block 2533 len 16384
8191 112/300 Flush
8192 113/300 Flush
8193 114/300 Read block 486 len 12288
8194 115/300 Write block 4191 len 12288 data: 3 3 4
8195 116/300 Flush
8196 117/300 Write block 1057 len 20480 data: 4 3 2 2 2
8197 118/300 Read block 2921 len 40960
8198 119/300 Write block 1223 len 32768 data: 4 4 4 3 3 2 3 4
8199 120/300 Read block 2137 len 28672
8200 121/300 Write block 2680 len 20480 data: 2 2 2 2 2
8201 122/300 Write block 4855 len 24576 data: 4 4 4 3 3 3
8202 123/300 Read block 199 len 32768
8203 124/300 Read block 4973 len 40960
8204 125/300 Flush
8205 126/300 Write block 792 len 12288 data: 3 3 3
8206 127/300 Read block 853 len 4096
8207 128/300 Write block 1539 len 36864 data: 3 3 3 3 3 3 3 3 3
8208 129/300 Read block 4103 len 40960
8209 130/300 Read block 286 len 24576
8210 131/300 Write block 3833 len 32768 data: 3 3 3 3 2 2 2 3
8211 132/300 Write block 306 len 12288 data: 4 3 3
8212 133/300 Write block 2666 len 16384 data: 2 2 2 2
8213 134/300 Write block 1991 len 16384 data: 3 3 3 3
8214 135/300 Flush
8215 136/300 Write block 2026 len 24576 data: 3 3 4 3 3 2
8216 137/300 Write block 335 len 4096 data: 3
8217 138/300 Read block 2696 len 40960
8218 139/300 Flush
8219 140/300 Write block 3283 len 4096 data: 2
8220 141/300 Read block 2117 len 24576
8221 142/300 Write block 1379 len 4096 data: 2
8222 143/300 Write block 4636 len 20480 data: 2 2 2 2 2
8223 144/300 Write block 1486 len 36864 data: 2 3 3 3 3 3 4 4 4
8224 145/300 Read block 4879 len 36864
8225 146/300 Write block 2099 len 24576 data: 3 3 3 2 2 2
8226 147/300 Read block 2002 len 12288
8227 148/300 Read block 2643 len 28672
8228 149/300 Read block 56 len 36864
8229 150/300 Write block 3739 len 24576 data: 2 2 2 2 2 2
8230 151/300 Flush
8231 152/300 Write block 4383 len 12288 data: 3 2 2
8232 153/300 Write block 4394 len 28672 data: 2 2 2 2 2 2 2
8233 154/300 Write block 4339 len 20480 data: 3 3 3 3 3
8234 155/300 Write block 4147 len 28672 data: 2 2 3 3 3 3 3
8235 156/300 Write block 4344 len 16384 data: 2 2 2 2
8236 157/300 Flush
8237 158/300 Read block 3411 len 12288
8238 159/300 Flush
8239 160/300 Flush
8240 161/300 Read block 869 len 20480
8241 162/300 Read block 425 len 12288
8242 163/300 Read block 4354 len 4096
8243 164/300 Write block 1235 len 32768 data: 4 3 3 2 2 2 2 2
8244 165/300 Read block 4174 len 36864
8245 166/300 Flush
8246 167/300 Read block 3306 len 32768
8247 168/300 Write block 4147 len 8192 data: 3 3
8248 169/300 Write block 1639 len 28672 data: 4 3 3 3 2 2 2
8249 170/300 Flush
8250 171/300 Flush
8251 172/300 Write block 1195 len 8192 data: 3 3
8252 173/300 Flush
8253 174/300 Flush
8254 175/300 Flush
8255 176/300 Write block 4214 len 36864 data: 2 2 2 2 2 3 3 3 4
8256 177/300 Write block 4370 len 24576 data: 2 2 2 2 3 4
8257 178/300 Write block 1601 len 12288 data: 2 2 2
8258 179/300 Write block 4783 len 20480 data: 2 3 3 4 5
8259 180/300 Read block 3862 len 28672
8260 181/300 Write block 1928 len 36864 data: 2 2 2 3 3 3 3 3 3
8261 182/300 Read block 124 len 28672
8262 183/300 Write block 3565 len 36864 data: 2 2 2 2 3 4 4 4 4
8263 184/300 Write block 3325 len 40960 data: 4 4 4 2 3 3 3 3 5 4
8264 185/300 Write block 983 len 28672 data: 3 2 2 2 2 2 2
8265 186/300 Flush
8266 187/300 Read block 3714 len 12288
8267 188/300 Read block 4412 len 40960
8268 189/300 Write block 4878 len 24576 data: 2 2 3 3 4 4
8269 190/300 Read block 3131 len 24576
8270 191/300 Read block 4178 len 12288
8271 192/300 Write block 560 len 12288 data: 3 4 4
8272 193/300 Read block 792 len 24576
8273 194/300 Write block 183 len 24576 data: 3 3 3 3 4 4
8274 195/300 Write block 4289 len 16384 data: 2 2 2 2
8275 196/300 Read block 4543 len 16384
8276 197/300 Read block 1324 len 16384
8277 198/300 Flush
8278 199/300 Read block 1225 len 28672
8279 200/300 Flush
8280 201/300 Write block 4562 len 16384 data: 2 2 2 3
8281 202/300 Write block 159 len 20480 data: 3 3 3 4 4
8282 203/300 Read block 4231 len 4096
8283 204/300 Write block 2616 len 36864 data: 2 2 2 2 2 2 2 4 4
8284 205/300 Read block 4884 len 12288
8285 206/300 Read block 516 len 28672
8286 207/300 Write block 713 len 36864 data: 4 3 3 3 3 3 3 3 4
8287 208/300 Read block 3213 len 28672
8288 209/300 Read block 647 len 16384
8289 210/300 Write block 961 len 4096 data: 2
8290 211/300 Flush
8291 212/300 Write block 4954 len 40960 data: 2 3 3 3 3 4 3 3 3 3
8292 213/300 Read block 2938 len 8192
8293 214/300 Read block 1506 len 36864
8294 215/300 Write block 4799 len 20480 data: 3 3 5 5 5
8295 216/300 Read block 1226 len 8192
8296 217/300 Write block 3607 len 12288 data: 2 2 2
8297 218/300 Read block 1152 len 24576
8298 219/300 Read block 3239 len 20480
8299 220/300 Read block 2052 len 16384
8300 221/300 Read block 4381 len 8192
8301 222/300 Write block 2949 len 20480 data: 3 2 2 3 4
8302 223/300 Read block 3430 len 20480
8303 224/300 Flush
8304 225/300 Read block 1254 len 20480
8305 226/300 Write block 3045 len 8192 data: 3 2
8306 227/300 Read block 4213 len 32768
8307 228/300 Write block 928 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8308 229/300 Write block 3611 len 36864 data: 2 2 2 2 2 2 2 2 2
8309 230/300 Read block 141 len 24576
8310 231/300 Write block 4245 len 32768 data: 2 3 3 3 3 3 3 3
8311 232/300 Read block 3621 len 36864
8312 233/300 Read block 506 len 12288
8313 234/300 Read block 4630 len 20480
8314 235/300 Read block 745 len 40960
8315 236/300 Read block 2888 len 28672
8316 237/300 Read block 4977 len 16384
8317 238/300 Read block 3406 len 20480
8318 239/300 Read block 4941 len 36864
8319 240/300 Write block 2968 len 8192 data: 3 3
8320 241/300 Read block 2803 len 32768
8321 242/300 Read block 3545 len 12288
8322 243/300 Read block 2550 len 4096
8323 244/300 Read block 4958 len 40960
8324 245/300 Read block 3510 len 8192
8325 246/300 Read block 2144 len 16384
8326 247/300 Write block 2177 len 8192 data: 3 3
8327 248/300 Write block 528 len 8192 data: 4 3
8328 249/300 Read block 4726 len 40960
8329 250/300 Read block 4293 len 36864
8330 251/300 Write block 1132 len 36864 data: 2 2 2 2 2 2 2 2 2
8331 252/300 Write block 2266 len 40960 data: 3 4 3 3 3 3 3 3 3 3
8332 253/300 Write block 1906 len 40960 data: 3 3 3 3 2 2 2 2 2 2
8333 254/300 Write block 3270 len 32768 data: 3 3 4 4 5 5 4 4
8334 255/300 Write block 4829 len 12288 data: 3 3 3
8335 256/300 Write block 4979 len 8192 data: 2 2
8336 257/300 Write block 2647 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8337 258/300 Write block 78 len 16384 data: 4 4 3 2
8338 259/300 Read block 4478 len 8192
8339 260/300 Read block 4030 len 36864
8340 261/300 Read block 3422 len 12288
8341 262/300 Read block 665 len 40960
8342 263/300 Write block 2288 len 40960 data: 3 3 3 3 3 2 3 3 3 3
8343 264/300 Write block 2933 len 32768 data: 2 2 2 2 2 2 3 2
8344 265/300 Write block 3140 len 28672 data: 4 4 3 2 2 3 3
8345 266/300 Flush
8346 267/300 Flush
8347 268/300 Read block 4708 len 20480
8348 269/300 Write block 3320 len 20480 data: 2 2 2 3 4
8349 270/300 Read block 2209 len 12288
8350 271/300 Flush
8351 272/300 Write block 171 len 12288 data: 3 3 3
8352 273/300 Read block 2386 len 4096
8353 274/300 Flush
8354 275/300 Read block 4223 len 32768
8355 276/300 Write block 2908 len 8192 data: 2 2
8356 277/300 Read block 2309 len 4096
8357 278/300 Write block 448 len 32768 data: 3 3 3 2 2 2 2 3
8358 279/300 Write block 1364 len 20480 data: 2 2 2 2 2
8359 280/300 Write block 1270 len 36864 data: 2 2 2 2 4 6 6 6 6
8360 281/300 Write block 4736 len 4096 data: 4
8361 282/300 Write block 1101 len 24576 data: 2 2 2 2 2 3
8362 283/300 Read block 3525 len 24576
8363 284/300 Write block 64 len 24576 data: 2 2 2 2 2 2
8364 285/300 Write block 1457 len 16384 data: 3 3 3 3
8365 286/300 Read block 4436 len 40960
8366 287/300 Flush
8367 288/300 Read block 1002 len 32768
8368 289/300 Write block 755 len 32768 data: 4 3 2 3 4 3 3 4
8369 290/300 Write block 3196 len 40960 data: 3 3 3 3 3 2 2 3 4 4
8370 291/300 Flush
8371 292/300 Read block 531 len 4096
8372 293/300 Write block 855 len 4096 data: 2
8373 294/300 Write block 2366 len 12288 data: 2 3 4
8374 295/300 Write block 2440 len 40960 data: 3 3 3 3 2 2 3 3 3 3
8375 296/300 Flush
8376 297/300 Read block 4263 len 4096
8377 298/300 Write block 4902 len 32768 data: 6 5 5 5 5 4 4 3
8378 299/300 Write block 148 len 28672 data: 2 2 2 2 2 2 2
8379 300/300 Write block 4487 len 36864 data: 2 3 2 2 2 2 5 5 5
8380 301/300 Read block 1659 len 12288
8381 Main task has work to do, go find it
8382 got dsc Start(0)
8383 start 0
8384 Feb 07 20:09:11.239 INFO request completed, latency_us: 198, response_code: 204, uri: /start/cid/0, method: POST, req_id: efba3f26-6fb1-4eb2-91bf-b17b5b3b27b7, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
8385 Replay: started 0, returned:Ok(())
8386 ----------------------------------------------------------------
8387 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8388 States: Offline Active Active
8389 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8390 2812 Acked Read 5 Sent Done Done false
8391 2813 Acked Write 9 Sent Done Done false
8392 2814 Acked Flush 0 Sent Done Done false
8393 2815 Acked Flush 0 Sent Done Done false
8394 2816 Acked Write 1 Sent Done Done false
8395 2817 Acked Write 5 Sent Done Done false
8396 2818 Acked Write 9 Sent Done Done false
8397 2819 Acked Read 7 Sent Done Done false
8398 2820 Acked Read 4 Sent Done Done false
8399 2821 Acked Read 2 Sent Done Done false
8400 2822 Acked Write 7 Sent Done Done false
8401 2823 Acked Read 6 Sent Done Done false
8402 2824 Acked Read 4 Sent Done Done false
8403 2825 Acked Write 4 Sent Done Done false
8404 2826 Acked Read 2 Sent Done Done false
8405 2827 Acked Read 9 Sent Done Done false
8406 2828 Acked Write 9 Sent Done Done false
8407 2829 Acked Write 10 Sent Done Done false
8408 2830 Acked Write 7 Sent Done Done false
8409 2831 Acked Write 4 Sent Done Done false
8410 2832 Acked Write 4 Sent Done Done false
8411 2833 Acked Write 9 Sent Done Done false
8412 2834 Acked Read 2 Sent Done Done false
8413 2835 Acked Read 2 Sent Done Done false
8414 2836 Acked Read 6 Sent Done Done false
8415 2837 Acked Read 5 Sent Done Done false
8416 2838 Acked Write 7 Sent Done Done false
8417 2839 Acked Read 3 Sent Done Done false
8418 2840 Acked Write 6 Sent Done Done false
8419 2841 Acked Read 10 Sent Done Done false
8420 2842 Acked Read 9 Sent Done Done false
8421 2843 Acked Read 7 Sent Done Done false
8422 2844 Acked Read 8 Sent Done Done false
8423 2845 Acked Read 5 Sent Done Done false
8424 2846 Acked Write 2 Sent Done Done false
8425 2847 Acked Read 10 Sent Done Done false
8426 2848 Acked Read 4 Sent Done Done false
8427 2849 Acked Read 5 Sent Done Done false
8428 2850 Acked Write 4 Sent Done Done false
8429 2851 Acked Read 7 Sent Done Done false
8430 2852 Acked Flush 0 Sent Done Done false
8431 2853 Acked Flush 0 Sent Done Done false
8432 2854 Acked Read 5 Sent Done Done false
8433 2855 Acked Read 9 Sent Done Done false
8434 2856 Acked Write 6 Sent Done Done false
8435 2857 Acked Read 5 Sent Done Done false
8436 2858 Acked Read 7 Sent Done Done false
8437 2859 Acked Write 4 Sent Done Done false
8438 2860 Acked Read 1 Sent Done Done false
8439 2861 Acked Write 2 Sent Done Done false
8440 2862 Acked Write 1 Sent Done Done false
8441 2863 Acked Read 10 Sent Done Done false
8442 2864 Acked Read 10 Sent Done Done false
8443 2865 Acked Read 8 Sent Done Done false
8444 2866 Acked Write 9 Sent Done Done false
8445 2867 Acked Flush 0 Sent Done Done false
8446 2868 Acked Read 3 Sent Done Done false
8447 2869 Acked Read 6 Sent Done Done false
8448 2870 Acked Flush 0 Sent Done Done false
8449 2871 Acked Write 4 Sent Done Done false
8450 2872 Acked Write 10 Sent Done Done false
8451 2873 Acked Read 1 Sent Done Done false
8452 2874 Acked Read 3 Sent Done Done false
8453 2875 Acked Write 7 Sent Done Done false
8454 2876 Acked Write 7 Sent Done Done false
8455 2877 Acked Read 8 Sent Done Done false
8456 2878 Acked Read 10 Sent Done Done false
8457 2879 Acked Read 8 Sent Done Done false
8458 2880 Acked Write 3 Sent Done Done false
8459 2881 Acked Read 10 Sent Done Done false
8460 2882 Acked Read 5 Sent Done Done false
8461 2883 Acked Write 7 Sent Done Done false
8462 2884 Acked Flush 0 Sent Done Done false
8463 2885 Acked Write 6 Sent Done Done false
8464 2886 Acked Flush 0 Sent Done Done false
8465 2887 Acked Read 7 Sent Done Done false
8466 2888 Acked Write 3 Sent Done Done false
8467 2889 Acked Write 10 Sent Done Done false
8468 2890 Acked Read 2 Sent Done Done false
8469 2891 Acked Read 10 Sent Done Done false
8470 2892 Acked Write 6 Sent Done Done false
8471 2893 Acked Read 2 Sent Done Done false
8472 2894 Acked Read 3 Sent Done Done false
8473 2895 Acked Read 7 Sent Done Done false
8474 2896 Acked Flush 0 Sent Done Done false
8475 2897 Acked Read 9 Sent Done Done false
8476 2898 Acked Read 2 Sent Done Done false
8477 2899 Acked Read 5 Sent Done Done false
8478 2900 Acked Read 4 Sent Done Done false
8479 2901 Acked Read 7 Sent Done Done false
8480 2902 Acked Write 8 Sent Done Done false
8481 2903 Acked Write 1 Sent Done Done false
8482 2904 Acked Write 9 Sent Done Done false
8483 2905 Acked Flush 0 Sent Done Done false
8484 2906 Acked Flush 0 Sent Done Done false
8485 2907 Acked Read 7 Sent Done Done false
8486 2908 Acked Write 9 Sent Done Done false
8487 2909 Acked Read 2 Sent Done Done false
8488 2910 Acked Write 6 Sent Done Done false
8489 2911 Acked Flush 0 Sent Done Done false
8490 2912 Acked Flush 0 Sent Done Done false
8491 2913 Acked Write 8 Sent Done Done false
8492 2914 Acked Read 1 Sent Done Done false
8493 2915 Acked Read 8 Sent Done Done false
8494 2916 Acked Flush 0 Sent Done Done false
8495 2917 Acked Write 4 Sent Done Done false
8496 2918 Acked Read 8 Sent Done Done false
8497 2919 Acked Write 3 Sent Done Done false
8498 2920 Acked Read 8 Sent Done Done false
8499 2921 Acked Write 9 Sent Done Done false
8500 2922 Acked Read 4 Sent Done Done false
8501 2923 Acked Flush 0 Sent Done Done false
8502 2924 Acked Flush 0 Sent Done Done false
8503 2925 Acked Read 3 Sent Done Done false
8504 2926 Acked Write 3 Sent Done Done false
8505 2927 Acked Flush 0 Sent Done Done false
8506 2928 Acked Write 5 Sent Done Done false
8507 2929 Acked Read 10 Sent Done Done false
8508 2930 Acked Write 8 Sent Done Done false
8509 2931 Acked Read 7 Sent Done Done false
8510 2932 Acked Write 5 Sent Done Done false
8511 2933 Acked Write 6 Sent Done Done false
8512 2934 Acked Read 8 Sent Done Done false
8513 2935 Acked Read 10 Sent Done Done false
8514 2936 Acked Flush 0 Sent Done Done false
8515 2937 Acked Write 3 Sent Done Done false
8516 2938 Acked Read 1 Sent Done Done false
8517 2939 Acked Write 9 Sent Done Done false
8518 2940 Acked Read 10 Sent Done Done false
8519 2941 Acked Read 6 Sent Done Done false
8520 2942 Acked Write 8 Sent Done Done false
8521 2943 Acked Write 3 Sent Done Done false
8522 2944 Acked Write 4 Sent Done Done false
8523 2945 Acked Write 4 Sent Done Done false
8524 2946 Acked Flush 0 Sent Done Done false
8525 2947 Acked Write 6 Sent Done Done false
8526 2948 Acked Write 1 Sent Done Done false
8527 2949 Acked Read 10 Sent Done Done false
8528 2950 Acked Flush 0 Sent Done Done false
8529 2951 Acked Write 1 Sent Done Done false
8530 2952 Acked Read 6 Sent Done Done false
8531 2953 Acked Write 1 Sent Done Done false
8532 2954 Acked Write 5 Sent Done Done false
8533 2955 Acked Write 9 Sent Done Done false
8534 2956 Acked Read 9 Sent Done Done false
8535 2957 Acked Write 6 Sent Done Done false
8536 2958 Acked Read 3 Sent Done Done false
8537 2959 Acked Read 7 Sent Done Done false
8538 2960 Acked Read 9 Sent Done Done false
8539 2961 Acked Write 6 Sent Done Done false
8540 2962 Acked Flush 0 Sent Done Done false
8541 2963 Acked Write 3 Sent Done Done false
8542 2964 Acked Write 7 Sent Done Done false
8543 2965 Acked Write 5 Sent Done Done false
8544 2966 Acked Write 7 Sent Done Done false
8545 2967 Acked Write 4 Sent Done Done false
8546 2968 Acked Flush 0 Sent Done Done false
8547 2969 Acked Read 3 Sent Done Done false
8548 2970 Acked Flush 0 Sent Done Done false
8549 2971 Acked Flush 0 Sent Done Done false
8550 2972 Acked Read 5 Sent Done Done false
8551 2973 Acked Read 3 Sent Done Done false
8552 2974 Acked Read 1 Sent Done Done false
8553 2975 Acked Write 8 Sent Done Done false
8554 2976 Acked Read 9 Sent Done Done false
8555 2977 Acked Flush 0 Sent Done Done false
8556 2978 Acked Read 8 Sent Done Done false
8557 2979 Acked Write 2 Sent Done Done false
8558 2980 Acked Write 7 Sent Done Done false
8559 2981 Acked Flush 0 Sent Done Done false
8560 2982 Acked Flush 0 Sent Done Done false
8561 2983 Acked Write 2 Sent Done Done false
8562 2984 Acked Flush 0 Sent Done Done false
8563 2985 Acked Flush 0 Sent Done Done false
8564 2986 Acked Flush 0 Sent Done Done false
8565 2987 Acked Write 9 Sent Done Done false
8566 2988 Acked Write 6 Sent Done Done false
8567 2989 Acked Write 3 Sent Done Done false
8568 2990 Acked Write 5 Sent Done Done false
8569 2991 Acked Read 7 Sent Done Done false
8570 2992 Acked Write 9 Sent Done Done false
8571 2993 Acked Read 7 Sent Done Done false
8572 2994 Acked Write 9 Sent Done Done false
8573 2995 Acked Write 10 Sent Done Done false
8574 2996 Acked Write 7 Sent Done Done false
8575 2997 Acked Flush 0 Sent Done Done false
8576 2998 Acked Read 3 Sent Done Done false
8577 2999 Acked Read 10 Sent Done Done false
8578 3000 Acked Write 6 Sent Done Done false
8579 3001 Acked Read 6 Sent Done Done false
8580 3002 Acked Read 3 Sent Done Done false
8581 3003 Acked Write 3 Sent Done Done false
8582 3004 Acked Read 6 Sent Done Done false
8583 3005 Acked Write 6 Sent Done Done false
8584 3006 Acked Write 4 Sent Done Done false
8585 3007 Acked Read 4 Sent Done Done false
8586 3008 Acked Read 4 Sent Done Done false
8587 3009 Acked Flush 0 Sent Done Done false
8588 3010 Acked Read 7 Sent Done Done false
8589 3011 Acked Flush 0 Sent Done Done false
8590 3012 Acked Write 4 Sent Done Done false
8591 3013 Acked Write 5 Sent Done Done false
8592 3014 Acked Read 1 Sent Done Done false
8593 3015 Acked Write 9 Sent Done Done false
8594 3016 Acked Read 3 Sent Done Done false
8595 3017 Acked Read 7 Sent Done Done false
8596 3018 Acked Write 9 Sent Done Done false
8597 3019 Acked Read 7 Sent Done Done false
8598 3020 Acked Read 4 Sent Done Done false
8599 3021 Acked Write 1 Sent Done Done false
8600 3022 Acked Flush 0 Sent Done Done false
8601 3023 Acked Write 10 Sent Done Done false
8602 3024 Acked Read 2 Sent Done Done false
8603 3025 Acked Read 9 Sent Done Done false
8604 3026 Acked Write 5 Sent Done Done false
8605 3027 Acked Read 2 Sent Done Done false
8606 3028 Acked Write 3 Sent Done Done false
8607 3029 Acked Read 6 Sent Done Done false
8608 3030 Acked Read 5 Sent Done Done false
8609 3031 Acked Read 4 Sent Done Done false
8610 3032 Acked Read 2 Sent Done Done false
8611 3033 Acked Write 5 Sent Done Done false
8612 3034 Acked Read 5 Sent Done Done false
8613 3035 Acked Flush 0 Sent Done Done false
8614 3036 Acked Read 5 Sent Done Done false
8615 3037 Acked Write 2 Sent Done Done false
8616 3038 Acked Read 8 Sent Done Done false
8617 3039 Acked Write 10 Sent Done Done false
8618 3040 Acked Write 9 Sent Done Done false
8619 3041 Acked Read 6 Sent Done Done false
8620 3042 Acked Write 8 Sent Done Done false
8621 3043 Acked Read 9 Sent Done Done false
8622 3044 Acked Read 3 Sent Done Done false
8623 3045 Acked Read 5 Sent Done Done false
8624 3046 Acked Read 10 Sent Done Done false
8625 3047 Acked Read 7 Sent Done Done false
8626 3048 Acked Read 4 Sent Done Done false
8627 3049 Acked Read 5 Sent Done Done false
8628 3050 Acked Read 9 Sent Done Done false
8629 3051 Acked Write 2 Sent Done Done false
8630 3052 Acked Read 8 Sent Done Done false
8631 3053 Acked Read 3 Sent Done Done false
8632 3054 Acked Read 1 Sent Done Done false
8633 3055 Acked Read 10 Sent Done Done false
8634 3056 Acked Read 2 Sent Done Done false
8635 3057 Acked Read 4 Sent Done Done false
8636 3058 Acked Write 2 Sent Done Done false
8637 3059 Acked Write 2 Sent Done Done false
8638 3060 Acked Read 10 Sent Done Done false
8639 3061 Acked Read 9 Sent Done Done false
8640 3062 Acked Write 9 Sent Done Done false
8641 3063 Acked Write 10 Sent Done Done false
8642 3064 Acked Write 10 Sent Done Done false
8643 3065 Acked Write 8 Sent Done Done false
8644 3066 Acked Write 3 Sent Done Done false
8645 3067 Acked Write 2 Sent Done Done false
8646 3068 Acked Write 10 Sent Done Done false
8647 3069 Acked Write 4 Sent Done Done false
8648 3070 Acked Read 2 Sent Done Done false
8649 3071 Acked Read 9 Sent Done Done false
8650 3072 Acked Read 3 Sent Done Done false
8651 3073 Acked Read 10 Sent Done Done false
8652 3074 Acked Write 10 Sent Done Done false
8653 3075 Acked Write 8 Sent Done Done false
8654 3076 Acked Write 7 Sent Done Done false
8655 3077 Acked Flush 0 Sent Done Done false
8656 3078 Acked Flush 0 Sent Done Done false
8657 3079 Acked Read 5 Sent Done Done false
8658 3080 Acked Write 5 Sent Done Done false
8659 3081 Acked Read 3 Sent Done Done false
8660 3082 Acked Flush 0 Sent Done Done false
8661 3083 Acked Write 3 Sent Done Done false
8662 3084 Acked Read 1 Sent Done Done false
8663 3085 Acked Flush 0 Sent Done Done false
8664 3086 Acked Read 8 Sent Done Done false
8665 3087 Acked Write 2 Sent Done Done false
8666 3088 Acked Read 1 Sent Done Done false
8667 3089 Acked Write 8 Sent Done Done false
8668 3090 Acked Write 5 Sent Done Done false
8669 3091 Acked Write 9 Sent Done Done false
8670 3092 Acked Write 1 Sent Done Done false
8671 3093 Acked Write 6 Sent Done Done false
8672 3094 Acked Read 6 Sent Done Done false
8673 3095 Acked Write 6 Sent Done Done false
8674 3096 Acked Write 4 Sent Done Done false
8675 3097 Acked Read 10 Sent Done Done false
8676 3098 Acked Flush 0 Sent Done Done false
8677 3099 Acked Read 8 Sent Done Done false
8678 3100 Acked Write 8 Sent Done Done false
8679 3101 Acked Write 10 Sent Done Done false
8680 3102 Acked Flush 0 Sent Done Done false
8681 3103 Acked Read 1 Sent Done Done false
8682 3104 Acked Write 1 Sent Done Done false
8683 3105 Acked Write 3 Sent Done Done false
8684 3106 Acked Write 10 Sent Done Done false
8685 3107 Acked Flush 0 Sent Done Done false
8686 3108 Acked Read 1 Sent Done Done false
8687 3109 Acked Write 8 Sent Done Done false
8688 3110 Acked Write 7 Sent Done Done false
8689 3111 Acked Write 9 Sent Done Done false
8690 3112 Acked Read 3 Sent Done Done false
8691 STATES DS:0 DS:1 DS:2 TOTAL
8692 Sent 301 0 0 301
8693 Done 0 301 301 602
8694 Skipped 0 0 0 0
8695 Error 0 0 0 0
8696 Last Flush: Some(JobId(2811)) Some(JobId(3107)) Some(JobId(3107))
8697 Downstairs last five completed: 2811 2810 2809 2808 2807
8698 Upstairs last five completed: 3112 3111 3110 3109 3108
8699 CLIENT: Up:0 ds:301 act:2
87002025-02-07T20:09:12.123ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
87012025-02-07T20:09:13.124ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
87022025-02-07T20:09:14.496ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
8703 [8810] got start action
8704 [8810] I am going to restart
8705 Starting downstairs at port 8810
8706 Make output file at "/tmp/dsc/downstairs-8810.txt"
8707 [8810][0] reports Starting
8708 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1216)
8709 [8810][0] reports Running
87102025-02-07T20:09:15.126ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
8711 ----------------------------------------------------------------
8712 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8713 States: Offline Active Active
8714 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8715 2812 Acked Read 5 Sent Done Done false
8716 2813 Acked Write 9 Sent Done Done false
8717 2814 Acked Flush 0 Sent Done Done false
8718 2815 Acked Flush 0 Sent Done Done false
8719 2816 Acked Write 1 Sent Done Done false
8720 2817 Acked Write 5 Sent Done Done false
8721 2818 Acked Write 9 Sent Done Done false
8722 2819 Acked Read 7 Sent Done Done false
8723 2820 Acked Read 4 Sent Done Done false
8724 2821 Acked Read 2 Sent Done Done false
8725 2822 Acked Write 7 Sent Done Done false
8726 2823 Acked Read 6 Sent Done Done false
8727 2824 Acked Read 4 Sent Done Done false
8728 2825 Acked Write 4 Sent Done Done false
8729 2826 Acked Read 2 Sent Done Done false
8730 2827 Acked Read 9 Sent Done Done false
8731 2828 Acked Write 9 Sent Done Done false
8732 2829 Acked Write 10 Sent Done Done false
8733 2830 Acked Write 7 Sent Done Done false
8734 2831 Acked Write 4 Sent Done Done false
8735 2832 Acked Write 4 Sent Done Done false
8736 2833 Acked Write 9 Sent Done Done false
8737 2834 Acked Read 2 Sent Done Done false
8738 2835 Acked Read 2 Sent Done Done false
8739 2836 Acked Read 6 Sent Done Done false
8740 2837 Acked Read 5 Sent Done Done false
8741 2838 Acked Write 7 Sent Done Done false
8742 2839 Acked Read 3 Sent Done Done false
8743 2840 Acked Write 6 Sent Done Done false
8744 2841 Acked Read 10 Sent Done Done false
8745 2842 Acked Read 9 Sent Done Done false
8746 2843 Acked Read 7 Sent Done Done false
8747 2844 Acked Read 8 Sent Done Done false
8748 2845 Acked Read 5 Sent Done Done false
8749 2846 Acked Write 2 Sent Done Done false
8750 2847 Acked Read 10 Sent Done Done false
8751 2848 Acked Read 4 Sent Done Done false
8752 2849 Acked Read 5 Sent Done Done false
8753 2850 Acked Write 4 Sent Done Done false
8754 2851 Acked Read 7 Sent Done Done false
8755 2852 Acked Flush 0 Sent Done Done false
8756 2853 Acked Flush 0 Sent Done Done false
8757 2854 Acked Read 5 Sent Done Done false
8758 2855 Acked Read 9 Sent Done Done false
8759 2856 Acked Write 6 Sent Done Done false
8760 2857 Acked Read 5 Sent Done Done false
8761 2858 Acked Read 7 Sent Done Done false
8762 2859 Acked Write 4 Sent Done Done false
8763 2860 Acked Read 1 Sent Done Done false
8764 2861 Acked Write 2 Sent Done Done false
8765 2862 Acked Write 1 Sent Done Done false
8766 2863 Acked Read 10 Sent Done Done false
8767 2864 Acked Read 10 Sent Done Done false
8768 2865 Acked Read 8 Sent Done Done false
8769 2866 Acked Write 9 Sent Done Done false
8770 2867 Acked Flush 0 Sent Done Done false
8771 2868 Acked Read 3 Sent Done Done false
8772 2869 Acked Read 6 Sent Done Done false
8773 2870 Acked Flush 0 Sent Done Done false
8774 2871 Acked Write 4 Sent Done Done false
8775 2872 Acked Write 10 Sent Done Done false
8776 2873 Acked Read 1 Sent Done Done false
8777 2874 Acked Read 3 Sent Done Done false
8778 2875 Acked Write 7 Sent Done Done false
8779 2876 Acked Write 7 Sent Done Done false
8780 2877 Acked Read 8 Sent Done Done false
8781 2878 Acked Read 10 Sent Done Done false
8782 2879 Acked Read 8 Sent Done Done false
8783 2880 Acked Write 3 Sent Done Done false
8784 2881 Acked Read 10 Sent Done Done false
8785 2882 Acked Read 5 Sent Done Done false
8786 2883 Acked Write 7 Sent Done Done false
8787 2884 Acked Flush 0 Sent Done Done false
8788 2885 Acked Write 6 Sent Done Done false
8789 2886 Acked Flush 0 Sent Done Done false
8790 2887 Acked Read 7 Sent Done Done false
8791 2888 Acked Write 3 Sent Done Done false
8792 2889 Acked Write 10 Sent Done Done false
8793 2890 Acked Read 2 Sent Done Done false
8794 2891 Acked Read 10 Sent Done Done false
8795 2892 Acked Write 6 Sent Done Done false
8796 2893 Acked Read 2 Sent Done Done false
8797 2894 Acked Read 3 Sent Done Done false
8798 2895 Acked Read 7 Sent Done Done false
8799 2896 Acked Flush 0 Sent Done Done false
8800 2897 Acked Read 9 Sent Done Done false
8801 2898 Acked Read 2 Sent Done Done false
8802 2899 Acked Read 5 Sent Done Done false
8803 2900 Acked Read 4 Sent Done Done false
8804 2901 Acked Read 7 Sent Done Done false
8805 2902 Acked Write 8 Sent Done Done false
8806 2903 Acked Write 1 Sent Done Done false
8807 2904 Acked Write 9 Sent Done Done false
8808 2905 Acked Flush 0 Sent Done Done false
8809 2906 Acked Flush 0 Sent Done Done false
8810 2907 Acked Read 7 Sent Done Done false
8811 2908 Acked Write 9 Sent Done Done false
8812 2909 Acked Read 2 Sent Done Done false
8813 2910 Acked Write 6 Sent Done Done false
8814 2911 Acked Flush 0 Sent Done Done false
8815 2912 Acked Flush 0 Sent Done Done false
8816 2913 Acked Write 8 Sent Done Done false
8817 2914 Acked Read 1 Sent Done Done false
8818 2915 Acked Read 8 Sent Done Done false
8819 2916 Acked Flush 0 Sent Done Done false
8820 2917 Acked Write 4 Sent Done Done false
8821 2918 Acked Read 8 Sent Done Done false
8822 2919 Acked Write 3 Sent Done Done false
8823 2920 Acked Read 8 Sent Done Done false
8824 2921 Acked Write 9 Sent Done Done false
8825 2922 Acked Read 4 Sent Done Done false
8826 2923 Acked Flush 0 Sent Done Done false
8827 2924 Acked Flush 0 Sent Done Done false
8828 2925 Acked Read 3 Sent Done Done false
8829 2926 Acked Write 3 Sent Done Done false
8830 2927 Acked Flush 0 Sent Done Done false
8831 2928 Acked Write 5 Sent Done Done false
8832 2929 Acked Read 10 Sent Done Done false
8833 2930 Acked Write 8 Sent Done Done false
8834 2931 Acked Read 7 Sent Done Done false
8835 2932 Acked Write 5 Sent Done Done false
8836 2933 Acked Write 6 Sent Done Done false
8837 2934 Acked Read 8 Sent Done Done false
8838 2935 Acked Read 10 Sent Done Done false
8839 2936 Acked Flush 0 Sent Done Done false
8840 2937 Acked Write 3 Sent Done Done false
8841 2938 Acked Read 1 Sent Done Done false
8842 2939 Acked Write 9 Sent Done Done false
8843 2940 Acked Read 10 Sent Done Done false
8844 2941 Acked Read 6 Sent Done Done false
8845 2942 Acked Write 8 Sent Done Done false
8846 2943 Acked Write 3 Sent Done Done false
8847 2944 Acked Write 4 Sent Done Done false
8848 2945 Acked Write 4 Sent Done Done false
8849 2946 Acked Flush 0 Sent Done Done false
8850 2947 Acked Write 6 Sent Done Done false
8851 2948 Acked Write 1 Sent Done Done false
8852 2949 Acked Read 10 Sent Done Done false
8853 2950 Acked Flush 0 Sent Done Done false
8854 2951 Acked Write 1 Sent Done Done false
8855 2952 Acked Read 6 Sent Done Done false
8856 2953 Acked Write 1 Sent Done Done false
8857 2954 Acked Write 5 Sent Done Done false
8858 2955 Acked Write 9 Sent Done Done false
8859 2956 Acked Read 9 Sent Done Done false
8860 2957 Acked Write 6 Sent Done Done false
8861 2958 Acked Read 3 Sent Done Done false
8862 2959 Acked Read 7 Sent Done Done false
8863 2960 Acked Read 9 Sent Done Done false
8864 2961 Acked Write 6 Sent Done Done false
8865 2962 Acked Flush 0 Sent Done Done false
8866 2963 Acked Write 3 Sent Done Done false
8867 2964 Acked Write 7 Sent Done Done false
8868 2965 Acked Write 5 Sent Done Done false
8869 2966 Acked Write 7 Sent Done Done false
8870 2967 Acked Write 4 Sent Done Done false
8871 2968 Acked Flush 0 Sent Done Done false
8872 2969 Acked Read 3 Sent Done Done false
8873 2970 Acked Flush 0 Sent Done Done false
8874 2971 Acked Flush 0 Sent Done Done false
8875 2972 Acked Read 5 Sent Done Done false
8876 2973 Acked Read 3 Sent Done Done false
8877 2974 Acked Read 1 Sent Done Done false
8878 2975 Acked Write 8 Sent Done Done false
8879 2976 Acked Read 9 Sent Done Done false
8880 2977 Acked Flush 0 Sent Done Done false
8881 2978 Acked Read 8 Sent Done Done false
8882 2979 Acked Write 2 Sent Done Done false
8883 2980 Acked Write 7 Sent Done Done false
8884 2981 Acked Flush 0 Sent Done Done false
8885 2982 Acked Flush 0 Sent Done Done false
8886 2983 Acked Write 2 Sent Done Done false
8887 2984 Acked Flush 0 Sent Done Done false
8888 2985 Acked Flush 0 Sent Done Done false
8889 2986 Acked Flush 0 Sent Done Done false
8890 2987 Acked Write 9 Sent Done Done false
8891 2988 Acked Write 6 Sent Done Done false
8892 2989 Acked Write 3 Sent Done Done false
8893 2990 Acked Write 5 Sent Done Done false
8894 2991 Acked Read 7 Sent Done Done false
8895 2992 Acked Write 9 Sent Done Done false
8896 2993 Acked Read 7 Sent Done Done false
8897 2994 Acked Write 9 Sent Done Done false
8898 2995 Acked Write 10 Sent Done Done false
8899 2996 Acked Write 7 Sent Done Done false
8900 2997 Acked Flush 0 Sent Done Done false
8901 2998 Acked Read 3 Sent Done Done false
8902 2999 Acked Read 10 Sent Done Done false
8903 3000 Acked Write 6 Sent Done Done false
8904 3001 Acked Read 6 Sent Done Done false
8905 3002 Acked Read 3 Sent Done Done false
8906 3003 Acked Write 3 Sent Done Done false
8907 3004 Acked Read 6 Sent Done Done false
8908 3005 Acked Write 6 Sent Done Done false
8909 3006 Acked Write 4 Sent Done Done false
8910 3007 Acked Read 4 Sent Done Done false
8911 3008 Acked Read 4 Sent Done Done false
8912 3009 Acked Flush 0 Sent Done Done false
8913 3010 Acked Read 7 Sent Done Done false
8914 3011 Acked Flush 0 Sent Done Done false
8915 3012 Acked Write 4 Sent Done Done false
8916 3013 Acked Write 5 Sent Done Done false
8917 3014 Acked Read 1 Sent Done Done false
8918 3015 Acked Write 9 Sent Done Done false
8919 3016 Acked Read 3 Sent Done Done false
8920 3017 Acked Read 7 Sent Done Done false
8921 3018 Acked Write 9 Sent Done Done false
8922 3019 Acked Read 7 Sent Done Done false
8923 3020 Acked Read 4 Sent Done Done false
8924 3021 Acked Write 1 Sent Done Done false
8925 3022 Acked Flush 0 Sent Done Done false
8926 3023 Acked Write 10 Sent Done Done false
8927 3024 Acked Read 2 Sent Done Done false
8928 3025 Acked Read 9 Sent Done Done false
8929 3026 Acked Write 5 Sent Done Done false
8930 3027 Acked Read 2 Sent Done Done false
8931 3028 Acked Write 3 Sent Done Done false
8932 3029 Acked Read 6 Sent Done Done false
8933 3030 Acked Read 5 Sent Done Done false
8934 3031 Acked Read 4 Sent Done Done false
8935 3032 Acked Read 2 Sent Done Done false
8936 3033 Acked Write 5 Sent Done Done false
8937 3034 Acked Read 5 Sent Done Done false
8938 3035 Acked Flush 0 Sent Done Done false
8939 3036 Acked Read 5 Sent Done Done false
8940 3037 Acked Write 2 Sent Done Done false
8941 3038 Acked Read 8 Sent Done Done false
8942 3039 Acked Write 10 Sent Done Done false
8943 3040 Acked Write 9 Sent Done Done false
8944 3041 Acked Read 6 Sent Done Done false
8945 3042 Acked Write 8 Sent Done Done false
8946 3043 Acked Read 9 Sent Done Done false
8947 3044 Acked Read 3 Sent Done Done false
8948 3045 Acked Read 5 Sent Done Done false
8949 3046 Acked Read 10 Sent Done Done false
8950 3047 Acked Read 7 Sent Done Done false
8951 3048 Acked Read 4 Sent Done Done false
8952 3049 Acked Read 5 Sent Done Done false
8953 3050 Acked Read 9 Sent Done Done false
8954 3051 Acked Write 2 Sent Done Done false
8955 3052 Acked Read 8 Sent Done Done false
8956 3053 Acked Read 3 Sent Done Done false
8957 3054 Acked Read 1 Sent Done Done false
8958 3055 Acked Read 10 Sent Done Done false
8959 3056 Acked Read 2 Sent Done Done false
8960 3057 Acked Read 4 Sent Done Done false
8961 3058 Acked Write 2 Sent Done Done false
8962 3059 Acked Write 2 Sent Done Done false
8963 3060 Acked Read 10 Sent Done Done false
8964 3061 Acked Read 9 Sent Done Done false
8965 3062 Acked Write 9 Sent Done Done false
8966 3063 Acked Write 10 Sent Done Done false
8967 3064 Acked Write 10 Sent Done Done false
8968 3065 Acked Write 8 Sent Done Done false
8969 3066 Acked Write 3 Sent Done Done false
8970 3067 Acked Write 2 Sent Done Done false
8971 3068 Acked Write 10 Sent Done Done false
8972 3069 Acked Write 4 Sent Done Done false
8973 3070 Acked Read 2 Sent Done Done false
8974 3071 Acked Read 9 Sent Done Done false
8975 3072 Acked Read 3 Sent Done Done false
8976 3073 Acked Read 10 Sent Done Done false
8977 3074 Acked Write 10 Sent Done Done false
8978 3075 Acked Write 8 Sent Done Done false
8979 3076 Acked Write 7 Sent Done Done false
8980 3077 Acked Flush 0 Sent Done Done false
8981 3078 Acked Flush 0 Sent Done Done false
8982 3079 Acked Read 5 Sent Done Done false
8983 3080 Acked Write 5 Sent Done Done false
8984 3081 Acked Read 3 Sent Done Done false
8985 3082 Acked Flush 0 Sent Done Done false
8986 3083 Acked Write 3 Sent Done Done false
8987 3084 Acked Read 1 Sent Done Done false
8988 3085 Acked Flush 0 Sent Done Done false
8989 3086 Acked Read 8 Sent Done Done false
8990 3087 Acked Write 2 Sent Done Done false
8991 3088 Acked Read 1 Sent Done Done false
8992 3089 Acked Write 8 Sent Done Done false
8993 3090 Acked Write 5 Sent Done Done false
8994 3091 Acked Write 9 Sent Done Done false
8995 3092 Acked Write 1 Sent Done Done false
8996 3093 Acked Write 6 Sent Done Done false
8997 3094 Acked Read 6 Sent Done Done false
8998 3095 Acked Write 6 Sent Done Done false
8999 3096 Acked Write 4 Sent Done Done false
9000 3097 Acked Read 10 Sent Done Done false
9001 3098 Acked Flush 0 Sent Done Done false
9002 3099 Acked Read 8 Sent Done Done false
9003 3100 Acked Write 8 Sent Done Done false
9004 3101 Acked Write 10 Sent Done Done false
9005 3102 Acked Flush 0 Sent Done Done false
9006 3103 Acked Read 1 Sent Done Done false
9007 3104 Acked Write 1 Sent Done Done false
9008 3105 Acked Write 3 Sent Done Done false
9009 3106 Acked Write 10 Sent Done Done false
9010 3107 Acked Flush 0 Sent Done Done false
9011 3108 Acked Read 1 Sent Done Done false
9012 3109 Acked Write 8 Sent Done Done false
9013 3110 Acked Write 7 Sent Done Done false
9014 3111 Acked Write 9 Sent Done Done false
9015 3112 Acked Read 3 Sent Done Done false
9016 STATES DS:0 DS:1 DS:2 TOTAL
9017 Sent 301 0 0 301
9018 Done 0 301 301 602
9019 Skipped 0 0 0 0
9020 Error 0 0 0 0
9021 Last Flush: Some(JobId(2811)) Some(JobId(3107)) Some(JobId(3107))
9022 Downstairs last five completed: 2811 2810 2809 2808 2807
9023 Upstairs last five completed: 3112 3111 3110 3109 3108
9024 CLIENT: Up:0 ds:301 act:2
90252025-02-07T20:09:16.127ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
90262025-02-07T20:09:17.000ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
90272025-02-07T20:09:17.000ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63371) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
90282025-02-07T20:09:17.001ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
90292025-02-07T20:09:17.002ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
90302025-02-07T20:09:17.002ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
90312025-02-07T20:09:17.002ZINFOcrucible: send last flush ID to this DS: Some(JobId(2811)) = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
90322025-02-07T20:09:17.002ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
90332025-02-07T20:09:17.002ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2811)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
90342025-02-07T20:09:17.002ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2811)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
9035 ----------------------------------------------------------------
9036 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9037 Downstairs last five completed: 3113 3112 3111 3110 3109
9038 Upstairs last five completed: 3113 3112 3111 3110 3109
9039 CLIENT: Up:0 ds:0 act:3
9040 Replay: All jobs finished
9041 Main task has work to do, go find it
9042 got dsc Stop(2)
9043 stop 2
9044 [8830] Got stop action so:false kr:false
9045 Feb 07 20:09:19.246 INFO request completed, latency_us: 330, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 4dd303e5-3a6e-415b-a9ae-077680a3170c, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
90462025-02-07T20:09:19.246ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
90472025-02-07T20:09:19.246ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
90482025-02-07T20:09:19.246ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
90492025-02-07T20:09:19.246ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
9050 Feb 07 20:09:19.247 INFO request completed, latency_us: 163, response_code: 200, uri: /state/cid/2, method: GET, req_id: 21089fc1-d709-486f-aa0c-24c51a463e77, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
9051 [8830] Exited with: ExitStatus(unix_wait_status(9))
9052 [8830][2] reports Exit
9053 Feb 07 20:09:23.249 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/2, method: GET, req_id: 60d42102-9eb7-47a0-bd46-28d67b1981a3, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
9054 001/300 Write block 428 len 4096 data: 3
9055 002/300 Write block 591 len 8192 data: 2 2
9056 003/300 Read block 3056 len 40960
9057 004/300 Write block 3763 len 36864 data: 3 2 2 2 3 3 3 4 4
9058 005/300 Read block 652 len 20480
9059 006/300 Write block 3044 len 36864 data: 3 4 3 2 3 4 4 4 5
9060 007/300 Read block 4528 len 12288
9061 008/300 Read block 3633 len 12288
9062 009/300 Read block 3445 len 24576
9063 010/300 Read block 867 len 20480
9064 011/300 Flush
9065 012/300 Write block 762 len 4096 data: 5
9066 013/300 Read block 2885 len 16384
9067 014/300 Read block 3200 len 24576
9068 015/300 Write block 1220 len 40960 data: 3 3 3 5 5 5 4 4 3 4
9069 016/300 Write block 4960 len 24576 data: 4 4 4 4 3 3
9070 017/300 Write block 2960 len 32768 data: 3 3 3 2 2 2 2 2
9071 018/300 Read block 4055 len 28672
9072 019/300 Read block 4818 len 24576
9073 020/300 Read block 4921 len 12288
9074 021/300 Read block 2628 len 12288
9075 022/300 Read block 979 len 8192
9076 023/300 Read block 1413 len 24576
9077 024/300 Read block 4939 len 32768
9078 025/300 Write block 999 len 24576 data: 4 5 7 7 5 5
9079 026/300 Read block 280 len 20480
9080 027/300 Write block 2155 len 32768 data: 3 3 3 3 3 3 3 4
9081 028/300 Write block 4330 len 28672 data: 3 4 4 3 3 3 3
9082 029/300 Read block 2819 len 32768
9083 030/300 Read block 4582 len 24576
9084 031/300 Read block 1750 len 12288
9085 032/300 Read block 4930 len 12288
9086 033/300 Read block 586 len 40960
9087 034/300 Write block 366 len 36864 data: 4 3 3 4 3 3 2 2 2
9088 035/300 Write block 1316 len 12288 data: 2 2 2
9089 036/300 Write block 118 len 40960 data: 3 3 3 3 2 2 2 2 2 3
9090 037/300 Write block 3126 len 24576 data: 4 4 3 3 3 3
9091 038/300 Read block 550 len 8192
9092 039/300 Write block 796 len 40960 data: 3 3 3 3 2 2 2 2 2 2
9093 040/300 Read block 878 len 28672
9094 041/300 Write block 4144 len 12288 data: 2 2 2
9095 042/300 Write block 1674 len 20480 data: 2 2 2 2 3
9096 043/300 Read block 2788 len 16384
9097 044/300 Read block 3949 len 28672
9098 045/300 Write block 1645 len 36864 data: 3 2 2 2 3 3 5 5 5
9099 046/300 Write block 4412 len 40960 data: 3 2 2 3 4 4 3 3 3 3
9100 047/300 Write block 2586 len 4096 data: 3
9101 048/300 Write block 3258 len 24576 data: 3 4 5 4 4 4
9102 049/300 Write block 4514 len 4096 data: 4
9103 050/300 Write block 1828 len 4096 data: 3
9104 051/300 Read block 3128 len 8192
9105 052/300 Read block 721 len 16384
9106 053/300 Read block 2827 len 8192
9107 054/300 Flush
9108 055/300 Read block 3254 len 4096
9109 056/300 Read block 4843 len 32768
9110 057/300 Read block 4219 len 8192
9111 058/300 Flush
9112 059/300 Read block 1161 len 36864
9113 060/300 Read block 1577 len 40960
9114 061/300 Write block 4171 len 24576 data: 2 2 2 2 2 2
9115 062/300 Write block 3540 len 12288 data: 2 2 3
9116 063/300 Write block 4771 len 12288 data: 2 2 2
9117 064/300 Read block 2092 len 12288
9118 065/300 Write block 0 len 32768 data: 2 2 2 2 2 2 2 2
9119 066/300 Write block 3659 len 32768 data: 3 4 4 5 5 5 5 5
9120 067/300 Read block 997 len 20480
9121 068/300 Write block 1071 len 12288 data: 3 3 2
9122 069/300 Write block 678 len 16384 data: 5 3 3 3
9123 070/300 Read block 1064 len 16384
9124 071/300 Write block 1504 len 20480 data: 3 3 4 4 4
9125 072/300 Read block 1967 len 36864
9126 073/300 Write block 1438 len 12288 data: 2 2 2
9127 074/300 Read block 1888 len 40960
9128 075/300 Write block 627 len 28672 data: 3 3 3 3 3 3 3
9129 076/300 Flush
9130 077/300 Read block 1912 len 8192
9131 078/300 Write block 4761 len 12288 data: 2 2 2
9132 079/300 Read block 1774 len 28672
9133 080/300 Write block 4902 len 28672 data: 7 6 6 6 6 5 5
9134 081/300 Read block 4960 len 40960
9135 082/300 Write block 4157 len 36864 data: 2 3 3 3 2 2 2 2 2
9136 083/300 Write block 3865 len 28672 data: 4 4 4 3 2 2 2
9137 084/300 Write block 828 len 8192 data: 4 3
9138 085/300 Read block 3560 len 16384
9139 086/300 Read block 4708 len 24576
9140 087/300 Read block 3405 len 20480
9141 088/300 Flush
9142 089/300 Write block 4742 len 16384 data: 4 5 5 5
9143 090/300 Read block 4432 len 12288
9144 091/300 Read block 2724 len 8192
9145 092/300 Read block 3684 len 4096
9146 093/300 Write block 1487 len 40960 data: 4 4 4 4 4 5 5 5 4 4
9147 094/300 Read block 2916 len 16384
9148 095/300 Read block 3414 len 36864
9149 096/300 Write block 2298 len 24576 data: 3 3 2 2 2 3
9150 097/300 Read block 1586 len 24576
9151 098/300 Read block 2528 len 16384
9152 099/300 Flush
9153 100/300 Read block 2359 len 16384
9154 101/300 Write block 2889 len 4096 data: 2
9155 102/300 Read block 1537 len 32768
9156 103/300 Read block 64 len 4096
9157 104/300 Read block 458 len 40960
9158 105/300 Write block 1878 len 32768 data: 2 2 2 2 2 2 2 2
9159 106/300 Flush
9160 107/300 Read block 1276 len 12288
9161 108/300 Write block 9 len 20480 data: 2 2 3 4 4
9162 109/300 Read block 191 len 36864
9163 110/300 Read block 3056 len 8192
9164 111/300 Read block 3821 len 16384
9165 112/300 Write block 39 len 16384 data: 3 2 2 2
9166 113/300 Flush
9167 114/300 Write block 866 len 4096 data: 2
9168 115/300 Write block 3483 len 40960 data: 4 3 2 3 2 3 3 3 3 3
9169 116/300 Write block 2249 len 20480 data: 2 3 3 3 3
9170 117/300 Write block 4662 len 8192 data: 3 3
9171 118/300 Write block 1074 len 28672 data: 2 2 2 3 3 3 3
9172 119/300 Read block 4123 len 24576
9173 120/300 Write block 4735 len 32768 data: 5 5 3 4 4 4 4 5
9174 121/300 Flush
9175 122/300 Read block 3085 len 4096
9176 123/300 Write block 1218 len 12288 data: 3 3 4
9177 124/300 Write block 1350 len 36864 data: 3 3 2 2 2 2 2 2 2
9178 125/300 Write block 357 len 4096 data: 2
9179 126/300 Write block 3167 len 12288 data: 4 4 4
9180 127/300 Write block 418 len 36864 data: 2 3 3 3 3 2 3 3 3
9181 128/300 Write block 3966 len 8192 data: 2 2
9182 129/300 Write block 4449 len 40960 data: 2 2 2 2 2 3 3 3 3 4
9183 130/300 Write block 2689 len 16384 data: 2 2 2 2
9184 131/300 Flush
9185 132/300 Read block 2865 len 8192
9186 133/300 Write block 3595 len 4096 data: 2
9187 134/300 Read block 765 len 16384
9188 135/300 Read block 3197 len 28672
9189 136/300 Read block 4340 len 12288
9190 137/300 Read block 493 len 24576
9191 138/300 Read block 830 len 32768
9192 139/300 Read block 3500 len 12288
9193 140/300 Read block 2365 len 24576
9194 141/300 Read block 4844 len 20480
9195 142/300 Read block 1027 len 4096
9196 143/300 Read block 1039 len 28672
9197 144/300 Read block 2991 len 32768
9198 145/300 Write block 627 len 8192 data: 4 4
9199 146/300 Read block 3266 len 28672
9200 147/300 Flush
9201 148/300 Write block 3125 len 12288 data: 4 5 5
9202 149/300 Read block 626 len 8192
9203 150/300 Read block 3290 len 36864
9204 151/300 Write block 2362 len 28672 data: 2 3 3 2 3 4 5
9205 152/300 Read block 1837 len 36864
9206 153/300 Read block 4056 len 36864
9207 154/300 Read block 2365 len 8192
9208 155/300 Read block 4782 len 24576
9209 156/300 Write block 4549 len 32768 data: 2 2 2 2 2 2 2 2
9210 157/300 Read block 559 len 4096
9211 158/300 Read block 2660 len 36864
9212 159/300 Write block 1349 len 20480 data: 3 4 4 3 3
9213 160/300 Flush
9214 161/300 Write block 530 len 32768 data: 2 3 3 2 2 2 3 3
9215 162/300 Write block 2372 len 36864 data: 3 3 3 4 3 4 4 3 3
9216 163/300 Read block 954 len 4096
9217 164/300 Write block 840 len 12288 data: 2 2 2
9218 165/300 Read block 1748 len 20480
9219 166/300 Write block 3182 len 12288 data: 2 2 2
9220 167/300 Read block 2075 len 24576
9221 168/300 Write block 1756 len 32768 data: 3 3 3 3 3 4 2 2
9222 169/300 Read block 712 len 4096
9223 170/300 Write block 3522 len 8192 data: 3 3
9224 171/300 Write block 1327 len 4096 data: 2
9225 172/300 Write block 4028 len 24576 data: 3 3 3 3 3 3
9226 173/300 Read block 2134 len 12288
9227 174/300 Read block 344 len 4096
9228 175/300 Read block 1129 len 20480
9229 176/300 Read block 951 len 16384
9230 177/300 Write block 4051 len 20480 data: 2 3 3 3 3
9231 178/300 Read block 1951 len 28672
9232 179/300 Read block 4887 len 20480
9233 180/300 Read block 2739 len 16384
9234 181/300 Read block 4470 len 32768
9235 182/300 Write block 311 len 40960 data: 4 5 4 4 4 3 3 3 2 2
9236 183/300 Read block 3551 len 36864
9237 184/300 Write block 3088 len 40960 data: 2 3 3 3 2 2 2 2 2 2
9238 185/300 Read block 4301 len 32768
9239 186/300 Write block 3079 len 24576 data: 4 4 4 3 3 3
9240 187/300 Read block 794 len 4096
9241 188/300 Write block 4191 len 32768 data: 4 4 5 3 3 3 3 3
9242 189/300 Flush
9243 190/300 Read block 3574 len 4096
9244 191/300 Read block 3760 len 20480
9245 192/300 Flush
9246 193/300 Read block 1465 len 20480
9247 194/300 Write block 4007 len 20480 data: 3 3 3 3 3
9248 195/300 Read block 1794 len 16384
9249 196/300 Flush
9250 197/300 Write block 2617 len 36864 data: 3 3 3 3 3 3 5 5 4
9251 198/300 Read block 717 len 12288
9252 199/300 Write block 3004 len 12288 data: 4 4 4
9253 200/300 Write block 4822 len 40960 data: 3 3 3 3 2 2 2 4 4 4
9254 201/300 Read block 2670 len 16384
9255 202/300 Write block 2363 len 36864 data: 4 4 3 4 5 6 3 3 3
9256 203/300 Read block 2803 len 4096
9257 204/300 Read block 3789 len 20480
9258 205/300 Read block 3179 len 36864
9259 206/300 Write block 3853 len 12288 data: 3 3 4
9260 207/300 Write block 1183 len 4096 data: 3
9261 208/300 Read block 4331 len 32768
9262 209/300 Read block 1069 len 24576
9263 210/300 Write block 4264 len 40960 data: 2 3 3 2 2 2 2 2 2 3
9264 211/300 Read block 2735 len 12288
9265 212/300 Write block 996 len 4096 data: 3
9266 213/300 Flush
9267 214/300 Read block 1977 len 8192
9268 215/300 Flush
9269 216/300 Write block 1296 len 36864 data: 4 4 4 4 4 4 4 4 3
9270 217/300 Read block 4311 len 12288
9271 218/300 Read block 1354 len 16384
9272 219/300 Flush
9273 220/300 Read block 2055 len 16384
9274 221/300 Read block 4833 len 12288
9275 222/300 Write block 429 len 36864 data: 3 3 2 3 3 3 2 2 2
9276 223/300 Read block 2705 len 8192
9277 224/300 Flush
9278 225/300 Read block 4681 len 20480
9279 226/300 Write block 2316 len 20480 data: 3 3 2 2 2
9280 227/300 Read block 1386 len 16384
9281 228/300 Write block 4307 len 16384 data: 2 2 2 3
9282 229/300 Read block 4063 len 16384
9283 230/300 Read block 950 len 8192
9284 231/300 Write block 612 len 4096 data: 4
9285 232/300 Flush
9286 233/300 Write block 942 len 20480 data: 2 2 2 2 3
9287 234/300 Write block 4697 len 24576 data: 2 2 2 2 2 2
9288 235/300 Read block 669 len 40960
9289 236/300 Write block 4166 len 20480 data: 3 3 3 3 2
9290 237/300 Read block 1341 len 40960
9291 238/300 Flush
9292 239/300 Write block 3347 len 20480 data: 2 2 2 2 3
9293 240/300 Write block 4994 len 20480 data: 3 3 2 2 2
9294 241/300 Write block 4124 len 8192 data: 4 4
9295 242/300 Read block 617 len 4096
9296 243/300 Flush
9297 244/300 Flush
9298 245/300 Read block 1164 len 32768
9299 246/300 Flush
9300 247/300 Read block 221 len 16384
9301 248/300 Read block 4466 len 12288
9302 249/300 Read block 1200 len 36864
9303 250/300 Write block 3380 len 20480 data: 3 3 3 3 2
9304 251/300 Write block 3984 len 4096 data: 2
9305 252/300 Write block 2774 len 36864 data: 3 3 3 3 3 2 2 2 2
9306 253/300 Write block 1781 len 28672 data: 3 3 3 3 3 3 3
9307 254/300 Read block 2096 len 12288
9308 255/300 Read block 4733 len 20480
9309 256/300 Read block 1076 len 32768
9310 257/300 Write block 2116 len 32768 data: 2 2 2 2 3 3 3 4
9311 258/300 Read block 3409 len 36864
9312 259/300 Read block 2002 len 12288
9313 260/300 Write block 724 len 32768 data: 4 5 3 2 2 2 2 2
9314 261/300 Flush
9315 262/300 Write block 3775 len 12288 data: 3 2 2
9316 263/300 Read block 3584 len 12288
9317 264/300 Write block 1083 len 24576 data: 2 3 3 3 3 3
9318 265/300 Read block 3857 len 4096
9319 266/300 Write block 332 len 36864 data: 2 2 3 4 4 4 3 3 3
9320 267/300 Write block 331 len 20480 data: 2 3 3 4 5
9321 268/300 Write block 1871 len 8192 data: 2 2
9322 269/300 Read block 2571 len 40960
9323 270/300 Read block 1105 len 20480
9324 271/300 Read block 1347 len 40960
9325 272/300 Flush
9326 273/300 Read block 2163 len 4096
9327 274/300 Read block 1169 len 20480
9328 275/300 Write block 4083 len 24576 data: 3 3 3 2 2 2
9329 276/300 Write block 3148 len 24576 data: 4 3 3 3 3 2
9330 277/300 Read block 1546 len 20480
9331 278/300 Flush
9332 279/300 Read block 1830 len 12288
9333 280/300 Write block 1472 len 36864 data: 2 2 3 3 3 3 3 3 3
9334 281/300 Write block 2978 len 20480 data: 4 4 5 5 5
9335 282/300 Read block 2487 len 12288
9336 283/300 Flush
9337 284/300 Read block 2174 len 20480
9338 285/300 Write block 1750 len 20480 data: 2 2 2 2 2
9339 286/300 Write block 1190 len 16384 data: 3 3 3 2
9340 287/300 Flush
9341 288/300 Write block 4134 len 36864 data: 2 2 2 2 2 2 2 2 2
9342 289/300 Write block 2735 len 24576 data: 3 3 3 3 3 3
9343 290/300 Read block 808 len 24576
9344 291/300 Write block 2288 len 24576 data: 4 4 4 4 4 3
9345 292/300 Read block 4641 len 8192
9346 293/300 Write block 3656 len 32768 data: 2 2 2 4 5 5 6 6
9347 294/300 Write block 2830 len 20480 data: 2 2 4 4 4
9348 295/300 Write block 1068 len 4096 data: 2
9349 296/300 Write block 2192 len 32768 data: 4 4 4 3 3 2 2 2
9350 297/300 Write block 4960 len 40960 data: 5 5 5 5 4 4 4 4 2 2
9351 298/300 Write block 3654 len 36864 data: 3 3 3 3 3 5 6 6 7
9352 299/300 Read block 3680 len 16384
9353 300/300 Read block 2739 len 40960
9354 301/300 Write block 1411 len 20480 data: 2 2 2 2 2
9355 Main task has work to do, go find it
9356 got dsc Start(2)
9357 start 2
9358 Feb 07 20:09:23.482 INFO request completed, latency_us: 227, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5ca5f23d-a2a5-4044-82df-8d97b5ae5984, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
9359 Replay: started 2, returned:Ok(())
9360 ----------------------------------------------------------------
9361 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9362 States: Active Active Offline
9363 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9364 3114 Acked Write 1 Done Done Sent false
9365 3115 Acked Write 2 Done Done Sent false
9366 3116 Acked Read 10 Done Done Sent false
9367 3117 Acked Write 9 Done Done Sent false
9368 3118 Acked Read 5 Done Done Sent false
9369 3119 Acked Write 9 Done Done Sent false
9370 3120 Acked Read 3 Done Done Sent false
9371 3121 Acked Read 3 Done Done Sent false
9372 3122 Acked Read 6 Done Done Sent false
9373 3123 Acked Read 5 Done Done Sent false
9374 3124 Acked Flush 0 Done Done Sent false
9375 3125 Acked Write 1 Done Done Sent false
9376 3126 Acked Read 4 Done Done Sent false
9377 3127 Acked Read 6 Done Done Sent false
9378 3128 Acked Write 10 Done Done Sent false
9379 3129 Acked Write 6 Done Done Sent false
9380 3130 Acked Write 8 Done Done Sent false
9381 3131 Acked Read 7 Done Done Sent false
9382 3132 Acked Read 6 Done Done Sent false
9383 3133 Acked Read 3 Done Done Sent false
9384 3134 Acked Read 3 Done Done Sent false
9385 3135 Acked Read 2 Done Done Sent false
9386 3136 Acked Read 6 Done Done Sent false
9387 3137 Acked Read 8 Done Done Sent false
9388 3138 Acked Write 6 Done Done Sent false
9389 3139 Acked Read 5 Done Done Sent false
9390 3140 Acked Write 8 Done Done Sent false
9391 3141 Acked Write 7 Done Done Sent false
9392 3142 Acked Read 8 Done Done Sent false
9393 3143 Acked Read 6 Done Done Sent false
9394 3144 Acked Read 3 Done Done Sent false
9395 3145 Acked Read 3 Done Done Sent false
9396 3146 Acked Read 10 Done Done Sent false
9397 3147 Acked Write 9 Done Done Sent false
9398 3148 Acked Write 3 Done Done Sent false
9399 3149 Acked Write 10 Done Done Sent false
9400 3150 Acked Write 6 Done Done Sent false
9401 3151 Acked Read 2 Done Done Sent false
9402 3152 Acked Write 10 Done Done Sent false
9403 3153 Acked Read 7 Done Done Sent false
9404 3154 Acked Write 3 Done Done Sent false
9405 3155 Acked Write 5 Done Done Sent false
9406 3156 Acked Read 4 Done Done Sent false
9407 3157 Acked Read 7 Done Done Sent false
9408 3158 Acked Write 9 Done Done Sent false
9409 3159 Acked Write 10 Done Done Sent false
9410 3160 Acked Write 1 Done Done Sent false
9411 3161 Acked Write 6 Done Done Sent false
9412 3162 Acked Write 1 Done Done Sent false
9413 3163 Acked Write 1 Done Done Sent false
9414 3164 Acked Read 2 Done Done Sent false
9415 3165 Acked Read 4 Done Done Sent false
9416 3166 Acked Read 2 Done Done Sent false
9417 3167 Acked Flush 0 Done Done Sent false
9418 3168 Acked Read 1 Done Done Sent false
9419 3169 Acked Read 8 Done Done Sent false
9420 3170 Acked Read 2 Done Done Sent false
9421 3171 Acked Flush 0 Done Done Sent false
9422 3172 Acked Read 9 Done Done Sent false
9423 3173 Acked Read 10 Done Done Sent false
9424 3174 Acked Write 6 Done Done Sent false
9425 3175 Acked Write 3 Done Done Sent false
9426 3176 Acked Write 3 Done Done Sent false
9427 3177 Acked Read 3 Done Done Sent false
9428 3178 Acked Write 8 Done Done Sent false
9429 3179 Acked Write 8 Done Done Sent false
9430 3180 Acked Read 5 Done Done Sent false
9431 3181 Acked Write 3 Done Done Sent false
9432 3182 Acked Write 4 Done Done Sent false
9433 3183 Acked Read 4 Done Done Sent false
9434 3184 Acked Write 5 Done Done Sent false
9435 3185 Acked Read 9 Done Done Sent false
9436 3186 Acked Write 3 Done Done Sent false
9437 3187 Acked Read 10 Done Done Sent false
9438 3188 Acked Write 7 Done Done Sent false
9439 3189 Acked Flush 0 Done Done Sent false
9440 3190 Acked Read 2 Done Done Sent false
9441 3191 Acked Write 3 Done Done Sent false
9442 3192 Acked Read 7 Done Done Sent false
9443 3193 Acked Write 7 Done Done Sent false
9444 3194 Acked Read 10 Done Done Sent false
9445 3195 Acked Write 9 Done Done Sent false
9446 3196 Acked Write 7 Done Done Sent false
9447 3197 Acked Write 2 Done Done Sent false
9448 3198 Acked Read 4 Done Done Sent false
9449 3199 Acked Read 6 Done Done Sent false
9450 3200 Acked Read 5 Done Done Sent false
9451 3201 Acked Flush 0 Done Done Sent false
9452 3202 Acked Write 4 Done Done Sent false
9453 3203 Acked Read 3 Done Done Sent false
9454 3204 Acked Read 2 Done Done Sent false
9455 3205 Acked Read 1 Done Done Sent false
9456 3206 Acked Write 10 Done Done Sent false
9457 3207 Acked Read 4 Done Done Sent false
9458 3208 Acked Read 9 Done Done Sent false
9459 3209 Acked Write 6 Done Done Sent false
9460 3210 Acked Read 6 Done Done Sent false
9461 3211 Acked Read 4 Done Done Sent false
9462 3212 Acked Flush 0 Done Done Sent false
9463 3213 Acked Read 4 Done Done Sent false
9464 3214 Acked Write 1 Done Done Sent false
9465 3215 Acked Read 8 Done Done Sent false
9466 3216 Acked Read 1 Done Done Sent false
9467 3217 Acked Read 10 Done Done Sent false
9468 3218 Acked Write 8 Done Done Sent false
9469 3219 Acked Flush 0 Done Done Sent false
9470 3220 Acked Read 3 Done Done Sent false
9471 3221 Acked Write 5 Done Done Sent false
9472 3222 Acked Read 9 Done Done Sent false
9473 3223 Acked Read 2 Done Done Sent false
9474 3224 Acked Read 4 Done Done Sent false
9475 3225 Acked Write 4 Done Done Sent false
9476 3226 Acked Flush 0 Done Done Sent false
9477 3227 Acked Write 1 Done Done Sent false
9478 3228 Acked Write 10 Done Done Sent false
9479 3229 Acked Write 5 Done Done Sent false
9480 3230 Acked Write 2 Done Done Sent false
9481 3231 Acked Write 7 Done Done Sent false
9482 3232 Acked Read 6 Done Done Sent false
9483 3233 Acked Write 8 Done Done Sent false
9484 3234 Acked Flush 0 Done Done Sent false
9485 3235 Acked Read 1 Done Done Sent false
9486 3236 Acked Write 3 Done Done Sent false
9487 3237 Acked Write 9 Done Done Sent false
9488 3238 Acked Write 1 Done Done Sent false
9489 3239 Acked Write 3 Done Done Sent false
9490 3240 Acked Write 9 Done Done Sent false
9491 3241 Acked Write 2 Done Done Sent false
9492 3242 Acked Write 10 Done Done Sent false
9493 3243 Acked Write 4 Done Done Sent false
9494 3244 Acked Flush 0 Done Done Sent false
9495 3245 Acked Read 2 Done Done Sent false
9496 3246 Acked Write 1 Done Done Sent false
9497 3247 Acked Read 4 Done Done Sent false
9498 3248 Acked Read 7 Done Done Sent false
9499 3249 Acked Read 3 Done Done Sent false
9500 3250 Acked Read 6 Done Done Sent false
9501 3251 Acked Read 8 Done Done Sent false
9502 3252 Acked Read 3 Done Done Sent false
9503 3253 Acked Read 6 Done Done Sent false
9504 3254 Acked Read 5 Done Done Sent false
9505 3255 Acked Read 1 Done Done Sent false
9506 3256 Acked Read 7 Done Done Sent false
9507 3257 Acked Read 8 Done Done Sent false
9508 3258 Acked Write 2 Done Done Sent false
9509 3259 Acked Read 7 Done Done Sent false
9510 3260 Acked Flush 0 Done Done Sent false
9511 3261 Acked Write 3 Done Done Sent false
9512 3262 Acked Read 2 Done Done Sent false
9513 3263 Acked Read 9 Done Done Sent false
9514 3264 Acked Write 7 Done Done Sent false
9515 3265 Acked Read 9 Done Done Sent false
9516 3266 Acked Read 9 Done Done Sent false
9517 3267 Acked Read 2 Done Done Sent false
9518 3268 Acked Read 6 Done Done Sent false
9519 3269 Acked Write 8 Done Done Sent false
9520 3270 Acked Read 1 Done Done Sent false
9521 3271 Acked Read 9 Done Done Sent false
9522 3272 Acked Write 5 Done Done Sent false
9523 3273 Acked Flush 0 Done Done Sent false
9524 3274 Acked Write 8 Done Done Sent false
9525 3275 Acked Write 9 Done Done Sent false
9526 3276 Acked Read 1 Done Done Sent false
9527 3277 Acked Write 3 Done Done Sent false
9528 3278 Acked Read 5 Done Done Sent false
9529 3279 Acked Write 3 Done Done Sent false
9530 3280 Acked Read 6 Done Done Sent false
9531 3281 Acked Write 8 Done Done Sent false
9532 3282 Acked Read 1 Done Done Sent false
9533 3283 Acked Write 2 Done Done Sent false
9534 3284 Acked Write 1 Done Done Sent false
9535 3285 Acked Write 6 Done Done Sent false
9536 3286 Acked Read 3 Done Done Sent false
9537 3287 Acked Read 1 Done Done Sent false
9538 3288 Acked Read 5 Done Done Sent false
9539 3289 Acked Read 4 Done Done Sent false
9540 3290 Acked Write 5 Done Done Sent false
9541 3291 Acked Read 7 Done Done Sent false
9542 3292 Acked Read 5 Done Done Sent false
9543 3293 Acked Read 4 Done Done Sent false
9544 3294 Acked Read 8 Done Done Sent false
9545 3295 Acked Write 10 Done Done Sent false
9546 3296 Acked Read 9 Done Done Sent false
9547 3297 Acked Write 10 Done Done Sent false
9548 3298 Acked Read 8 Done Done Sent false
9549 3299 Acked Write 6 Done Done Sent false
9550 3300 Acked Read 1 Done Done Sent false
9551 3301 Acked Write 8 Done Done Sent false
9552 3302 Acked Flush 0 Done Done Sent false
9553 3303 Acked Read 1 Done Done Sent false
9554 3304 Acked Read 5 Done Done Sent false
9555 3305 Acked Flush 0 Done Done Sent false
9556 3306 Acked Read 5 Done Done Sent false
9557 3307 Acked Write 5 Done Done Sent false
9558 3308 Acked Read 4 Done Done Sent false
9559 3309 Acked Flush 0 Done Done Sent false
9560 3310 Acked Write 9 Done Done Sent false
9561 3311 Acked Read 3 Done Done Sent false
9562 3312 Acked Write 3 Done Done Sent false
9563 3313 Acked Write 10 Done Done Sent false
9564 3314 Acked Read 4 Done Done Sent false
9565 3315 Acked Write 9 Done Done Sent false
9566 3316 Acked Read 1 Done Done Sent false
9567 3317 Acked Read 5 Done Done Sent false
9568 3318 Acked Read 9 Done Done Sent false
9569 3319 Acked Write 3 Done Done Sent false
9570 3320 Acked Write 1 Done Done Sent false
9571 3321 Acked Read 8 Done Done Sent false
9572 3322 Acked Read 6 Done Done Sent false
9573 3323 Acked Write 10 Done Done Sent false
9574 3324 Acked Read 3 Done Done Sent false
9575 3325 Acked Write 1 Done Done Sent false
9576 3326 Acked Flush 0 Done Done Sent false
9577 3327 Acked Read 2 Done Done Sent false
9578 3328 Acked Flush 0 Done Done Sent false
9579 3329 Acked Write 9 Done Done Sent false
9580 3330 Acked Read 3 Done Done Sent false
9581 3331 Acked Read 4 Done Done Sent false
9582 3332 Acked Flush 0 Done Done Sent false
9583 3333 Acked Read 4 Done Done Sent false
9584 3334 Acked Read 3 Done Done Sent false
9585 3335 Acked Write 9 Done Done Sent false
9586 3336 Acked Read 2 Done Done Sent false
9587 3337 Acked Flush 0 Done Done Sent false
9588 3338 Acked Read 5 Done Done Sent false
9589 3339 Acked Write 5 Done Done Sent false
9590 3340 Acked Read 4 Done Done Sent false
9591 3341 Acked Write 4 Done Done Sent false
9592 3342 Acked Read 4 Done Done Sent false
9593 3343 Acked Read 2 Done Done Sent false
9594 3344 Acked Write 1 Done Done Sent false
9595 3345 Acked Flush 0 Done Done Sent false
9596 3346 Acked Write 5 Done Done Sent false
9597 3347 Acked Write 6 Done Done Sent false
9598 3348 Acked Read 10 Done Done Sent false
9599 3349 Acked Write 5 Done Done Sent false
9600 3350 Acked Read 10 Done Done Sent false
9601 3351 Acked Flush 0 Done Done Sent false
9602 3352 Acked Write 5 Done Done Sent false
9603 3353 Acked Write 5 Done Done Sent false
9604 3354 Acked Write 2 Done Done Sent false
9605 3355 Acked Read 1 Done Done Sent false
9606 3356 Acked Flush 0 Done Done Sent false
9607 3357 Acked Flush 0 Done Done Sent false
9608 3358 Acked Read 8 Done Done Sent false
9609 3359 Acked Flush 0 Done Done Sent false
9610 3360 Acked Read 4 Done Done Sent false
9611 3361 Acked Read 3 Done Done Sent false
9612 3362 Acked Read 9 Done Done Sent false
9613 3363 Acked Write 5 Done Done Sent false
9614 3364 Acked Write 1 Done Done Sent false
9615 3365 Acked Write 9 Done Done Sent false
9616 3366 Acked Write 7 Done Done Sent false
9617 3367 Acked Read 3 Done Done Sent false
9618 3368 Acked Read 5 Done Done Sent false
9619 3369 Acked Read 8 Done Done Sent false
9620 3370 Acked Write 8 Done Done Sent false
9621 3371 Acked Read 9 Done Done Sent false
9622 3372 Acked Read 3 Done Done Sent false
9623 3373 Acked Write 8 Done Done Sent false
9624 3374 Acked Flush 0 Done Done Sent false
9625 3375 Acked Write 3 Done Done Sent false
9626 3376 Acked Read 3 Done Done Sent false
9627 3377 Acked Write 6 Done Done Sent false
9628 3378 Acked Read 1 Done Done Sent false
9629 3379 Acked Write 9 Done Done Sent false
9630 3380 Acked Write 5 Done Done Sent false
9631 3381 Acked Write 2 Done Done Sent false
9632 3382 Acked Read 10 Done Done Sent false
9633 3383 Acked Read 5 Done Done Sent false
9634 3384 Acked Read 10 Done Done Sent false
9635 3385 Acked Flush 0 Done Done Sent false
9636 3386 Acked Read 1 Done Done Sent false
9637 3387 Acked Read 5 Done Done Sent false
9638 3388 Acked Write 6 Done Done Sent false
9639 3389 Acked Write 6 Done Done Sent false
9640 3390 Acked Read 5 Done Done Sent false
9641 3391 Acked Flush 0 Done Done Sent false
9642 3392 Acked Read 3 Done Done Sent false
9643 3393 Acked Write 9 Done Done Sent false
9644 3394 Acked Write 5 Done Done Sent false
9645 3395 Acked Read 3 Done Done Sent false
9646 3396 Acked Flush 0 Done Done Sent false
9647 3397 Acked Read 5 Done Done Sent false
9648 3398 Acked Write 5 Done Done Sent false
9649 3399 Acked Write 4 Done Done Sent false
9650 3400 Acked Flush 0 Done Done Sent false
9651 3401 Acked Write 9 Done Done Sent false
9652 3402 Acked Write 6 Done Done Sent false
9653 3403 Acked Read 6 Done Done Sent false
9654 3404 Acked Write 6 Done Done Sent false
9655 3405 Acked Read 2 Done Done Sent false
9656 3406 Acked Write 8 Done Done Sent false
9657 3407 Acked Write 5 Done Done Sent false
9658 3408 Acked Write 1 Done Done Sent false
9659 3409 Acked Write 8 Done Done Sent false
9660 3410 Acked Write 10 Done Done Sent false
9661 3411 Acked Write 9 Done Done Sent false
9662 3412 Acked Read 4 Done Done Sent false
9663 3413 Acked Read 10 Done Done Sent false
9664 3414 Acked Write 5 Done Done Sent false
9665 STATES DS:0 DS:1 DS:2 TOTAL
9666 Sent 0 0 301 301
9667 Done 301 301 0 602
9668 Skipped 0 0 0 0
9669 Error 0 0 0 0
9670 Last Flush: Some(JobId(3400)) Some(JobId(3400)) Some(JobId(3113))
9671 Downstairs last five completed: 3113 3112 3111 3110 3109
9672 Upstairs last five completed: 3414 3413 3412 3411 3410
9673 CLIENT: Up:0 ds:301 act:2
96742025-02-07T20:09:24.142ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
96752025-02-07T20:09:25.144ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
96762025-02-07T20:09:26.146ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
9677 ----------------------------------------------------------------
9678 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9679 States: Active Active Offline
9680 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9681 3114 Acked Write 1 Done Done Sent false
9682 3115 Acked Write 2 Done Done Sent false
9683 3116 Acked Read 10 Done Done Sent false
9684 3117 Acked Write 9 Done Done Sent false
9685 3118 Acked Read 5 Done Done Sent false
9686 3119 Acked Write 9 Done Done Sent false
9687 3120 Acked Read 3 Done Done Sent false
9688 3121 Acked Read 3 Done Done Sent false
9689 3122 Acked Read 6 Done Done Sent false
9690 3123 Acked Read 5 Done Done Sent false
9691 3124 Acked Flush 0 Done Done Sent false
9692 3125 Acked Write 1 Done Done Sent false
9693 3126 Acked Read 4 Done Done Sent false
9694 3127 Acked Read 6 Done Done Sent false
9695 3128 Acked Write 10 Done Done Sent false
9696 3129 Acked Write 6 Done Done Sent false
9697 3130 Acked Write 8 Done Done Sent false
9698 3131 Acked Read 7 Done Done Sent false
9699 3132 Acked Read 6 Done Done Sent false
9700 3133 Acked Read 3 Done Done Sent false
9701 3134 Acked Read 3 Done Done Sent false
9702 3135 Acked Read 2 Done Done Sent false
9703 3136 Acked Read 6 Done Done Sent false
9704 3137 Acked Read 8 Done Done Sent false
9705 3138 Acked Write 6 Done Done Sent false
9706 3139 Acked Read 5 Done Done Sent false
9707 3140 Acked Write 8 Done Done Sent false
9708 3141 Acked Write 7 Done Done Sent false
9709 3142 Acked Read 8 Done Done Sent false
9710 3143 Acked Read 6 Done Done Sent false
9711 3144 Acked Read 3 Done Done Sent false
9712 3145 Acked Read 3 Done Done Sent false
9713 3146 Acked Read 10 Done Done Sent false
9714 3147 Acked Write 9 Done Done Sent false
9715 3148 Acked Write 3 Done Done Sent false
9716 3149 Acked Write 10 Done Done Sent false
9717 3150 Acked Write 6 Done Done Sent false
9718 3151 Acked Read 2 Done Done Sent false
9719 3152 Acked Write 10 Done Done Sent false
9720 3153 Acked Read 7 Done Done Sent false
9721 3154 Acked Write 3 Done Done Sent false
9722 3155 Acked Write 5 Done Done Sent false
9723 3156 Acked Read 4 Done Done Sent false
9724 3157 Acked Read 7 Done Done Sent false
9725 3158 Acked Write 9 Done Done Sent false
9726 3159 Acked Write 10 Done Done Sent false
9727 3160 Acked Write 1 Done Done Sent false
9728 3161 Acked Write 6 Done Done Sent false
9729 3162 Acked Write 1 Done Done Sent false
9730 3163 Acked Write 1 Done Done Sent false
9731 3164 Acked Read 2 Done Done Sent false
9732 3165 Acked Read 4 Done Done Sent false
9733 3166 Acked Read 2 Done Done Sent false
9734 3167 Acked Flush 0 Done Done Sent false
9735 3168 Acked Read 1 Done Done Sent false
9736 3169 Acked Read 8 Done Done Sent false
9737 3170 Acked Read 2 Done Done Sent false
9738 3171 Acked Flush 0 Done Done Sent false
9739 3172 Acked Read 9 Done Done Sent false
9740 3173 Acked Read 10 Done Done Sent false
9741 3174 Acked Write 6 Done Done Sent false
9742 3175 Acked Write 3 Done Done Sent false
9743 3176 Acked Write 3 Done Done Sent false
9744 3177 Acked Read 3 Done Done Sent false
9745 3178 Acked Write 8 Done Done Sent false
9746 3179 Acked Write 8 Done Done Sent false
9747 3180 Acked Read 5 Done Done Sent false
9748 3181 Acked Write 3 Done Done Sent false
9749 3182 Acked Write 4 Done Done Sent false
9750 3183 Acked Read 4 Done Done Sent false
9751 3184 Acked Write 5 Done Done Sent false
9752 3185 Acked Read 9 Done Done Sent false
9753 3186 Acked Write 3 Done Done Sent false
9754 3187 Acked Read 10 Done Done Sent false
9755 3188 Acked Write 7 Done Done Sent false
9756 3189 Acked Flush 0 Done Done Sent false
9757 3190 Acked Read 2 Done Done Sent false
9758 3191 Acked Write 3 Done Done Sent false
9759 3192 Acked Read 7 Done Done Sent false
9760 3193 Acked Write 7 Done Done Sent false
9761 3194 Acked Read 10 Done Done Sent false
9762 3195 Acked Write 9 Done Done Sent false
9763 3196 Acked Write 7 Done Done Sent false
9764 3197 Acked Write 2 Done Done Sent false
9765 3198 Acked Read 4 Done Done Sent false
9766 3199 Acked Read 6 Done Done Sent false
9767 3200 Acked Read 5 Done Done Sent false
9768 3201 Acked Flush 0 Done Done Sent false
9769 3202 Acked Write 4 Done Done Sent false
9770 3203 Acked Read 3 Done Done Sent false
9771 3204 Acked Read 2 Done Done Sent false
9772 3205 Acked Read 1 Done Done Sent false
9773 3206 Acked Write 10 Done Done Sent false
9774 3207 Acked Read 4 Done Done Sent false
9775 3208 Acked Read 9 Done Done Sent false
9776 3209 Acked Write 6 Done Done Sent false
9777 3210 Acked Read 6 Done Done Sent false
9778 3211 Acked Read 4 Done Done Sent false
9779 3212 Acked Flush 0 Done Done Sent false
9780 3213 Acked Read 4 Done Done Sent false
9781 3214 Acked Write 1 Done Done Sent false
9782 3215 Acked Read 8 Done Done Sent false
9783 3216 Acked Read 1 Done Done Sent false
9784 3217 Acked Read 10 Done Done Sent false
9785 3218 Acked Write 8 Done Done Sent false
9786 3219 Acked Flush 0 Done Done Sent false
9787 3220 Acked Read 3 Done Done Sent false
9788 3221 Acked Write 5 Done Done Sent false
9789 3222 Acked Read 9 Done Done Sent false
9790 3223 Acked Read 2 Done Done Sent false
9791 3224 Acked Read 4 Done Done Sent false
9792 3225 Acked Write 4 Done Done Sent false
9793 3226 Acked Flush 0 Done Done Sent false
9794 3227 Acked Write 1 Done Done Sent false
9795 3228 Acked Write 10 Done Done Sent false
9796 3229 Acked Write 5 Done Done Sent false
9797 3230 Acked Write 2 Done Done Sent false
9798 3231 Acked Write 7 Done Done Sent false
9799 3232 Acked Read 6 Done Done Sent false
9800 3233 Acked Write 8 Done Done Sent false
9801 3234 Acked Flush 0 Done Done Sent false
9802 3235 Acked Read 1 Done Done Sent false
9803 3236 Acked Write 3 Done Done Sent false
9804 3237 Acked Write 9 Done Done Sent false
9805 3238 Acked Write 1 Done Done Sent false
9806 3239 Acked Write 3 Done Done Sent false
9807 3240 Acked Write 9 Done Done Sent false
9808 3241 Acked Write 2 Done Done Sent false
9809 3242 Acked Write 10 Done Done Sent false
9810 3243 Acked Write 4 Done Done Sent false
9811 3244 Acked Flush 0 Done Done Sent false
9812 3245 Acked Read 2 Done Done Sent false
9813 3246 Acked Write 1 Done Done Sent false
9814 3247 Acked Read 4 Done Done Sent false
9815 3248 Acked Read 7 Done Done Sent false
9816 3249 Acked Read 3 Done Done Sent false
9817 3250 Acked Read 6 Done Done Sent false
9818 3251 Acked Read 8 Done Done Sent false
9819 3252 Acked Read 3 Done Done Sent false
9820 3253 Acked Read 6 Done Done Sent false
9821 3254 Acked Read 5 Done Done Sent false
9822 3255 Acked Read 1 Done Done Sent false
9823 3256 Acked Read 7 Done Done Sent false
9824 3257 Acked Read 8 Done Done Sent false
9825 3258 Acked Write 2 Done Done Sent false
9826 3259 Acked Read 7 Done Done Sent false
9827 3260 Acked Flush 0 Done Done Sent false
9828 3261 Acked Write 3 Done Done Sent false
9829 3262 Acked Read 2 Done Done Sent false
9830 3263 Acked Read 9 Done Done Sent false
9831 3264 Acked Write 7 Done Done Sent false
9832 3265 Acked Read 9 Done Done Sent false
9833 3266 Acked Read 9 Done Done Sent false
9834 3267 Acked Read 2 Done Done Sent false
9835 3268 Acked Read 6 Done Done Sent false
9836 3269 Acked Write 8 Done Done Sent false
9837 3270 Acked Read 1 Done Done Sent false
9838 3271 Acked Read 9 Done Done Sent false
9839 3272 Acked Write 5 Done Done Sent false
9840 3273 Acked Flush 0 Done Done Sent false
9841 3274 Acked Write 8 Done Done Sent false
9842 3275 Acked Write 9 Done Done Sent false
9843 3276 Acked Read 1 Done Done Sent false
9844 3277 Acked Write 3 Done Done Sent false
9845 3278 Acked Read 5 Done Done Sent false
9846 3279 Acked Write 3 Done Done Sent false
9847 3280 Acked Read 6 Done Done Sent false
9848 3281 Acked Write 8 Done Done Sent false
9849 3282 Acked Read 1 Done Done Sent false
9850 3283 Acked Write 2 Done Done Sent false
9851 3284 Acked Write 1 Done Done Sent false
9852 3285 Acked Write 6 Done Done Sent false
9853 3286 Acked Read 3 Done Done Sent false
9854 3287 Acked Read 1 Done Done Sent false
9855 3288 Acked Read 5 Done Done Sent false
9856 3289 Acked Read 4 Done Done Sent false
9857 3290 Acked Write 5 Done Done Sent false
9858 3291 Acked Read 7 Done Done Sent false
9859 3292 Acked Read 5 Done Done Sent false
9860 3293 Acked Read 4 Done Done Sent false
9861 3294 Acked Read 8 Done Done Sent false
9862 3295 Acked Write 10 Done Done Sent false
9863 3296 Acked Read 9 Done Done Sent false
9864 3297 Acked Write 10 Done Done Sent false
9865 3298 Acked Read 8 Done Done Sent false
9866 3299 Acked Write 6 Done Done Sent false
9867 3300 Acked Read 1 Done Done Sent false
9868 3301 Acked Write 8 Done Done Sent false
9869 3302 Acked Flush 0 Done Done Sent false
9870 3303 Acked Read 1 Done Done Sent false
9871 3304 Acked Read 5 Done Done Sent false
9872 3305 Acked Flush 0 Done Done Sent false
9873 3306 Acked Read 5 Done Done Sent false
9874 3307 Acked Write 5 Done Done Sent false
9875 3308 Acked Read 4 Done Done Sent false
9876 3309 Acked Flush 0 Done Done Sent false
9877 3310 Acked Write 9 Done Done Sent false
9878 3311 Acked Read 3 Done Done Sent false
9879 3312 Acked Write 3 Done Done Sent false
9880 3313 Acked Write 10 Done Done Sent false
9881 3314 Acked Read 4 Done Done Sent false
9882 3315 Acked Write 9 Done Done Sent false
9883 3316 Acked Read 1 Done Done Sent false
9884 3317 Acked Read 5 Done Done Sent false
9885 3318 Acked Read 9 Done Done Sent false
9886 3319 Acked Write 3 Done Done Sent false
9887 3320 Acked Write 1 Done Done Sent false
9888 3321 Acked Read 8 Done Done Sent false
9889 3322 Acked Read 6 Done Done Sent false
9890 3323 Acked Write 10 Done Done Sent false
9891 3324 Acked Read 3 Done Done Sent false
9892 3325 Acked Write 1 Done Done Sent false
9893 3326 Acked Flush 0 Done Done Sent false
9894 3327 Acked Read 2 Done Done Sent false
9895 3328 Acked Flush 0 Done Done Sent false
9896 3329 Acked Write 9 Done Done Sent false
9897 3330 Acked Read 3 Done Done Sent false
9898 3331 Acked Read 4 Done Done Sent false
9899 3332 Acked Flush 0 Done Done Sent false
9900 3333 Acked Read 4 Done Done Sent false
9901 3334 Acked Read 3 Done Done Sent false
9902 3335 Acked Write 9 Done Done Sent false
9903 3336 Acked Read 2 Done Done Sent false
9904 3337 Acked Flush 0 Done Done Sent false
9905 3338 Acked Read 5 Done Done Sent false
9906 3339 Acked Write 5 Done Done Sent false
9907 3340 Acked Read 4 Done Done Sent false
9908 3341 Acked Write 4 Done Done Sent false
9909 3342 Acked Read 4 Done Done Sent false
9910 3343 Acked Read 2 Done Done Sent false
9911 3344 Acked Write 1 Done Done Sent false
9912 3345 Acked Flush 0 Done Done Sent false
9913 3346 Acked Write 5 Done Done Sent false
9914 3347 Acked Write 6 Done Done Sent false
9915 3348 Acked Read 10 Done Done Sent false
9916 3349 Acked Write 5 Done Done Sent false
9917 3350 Acked Read 10 Done Done Sent false
9918 3351 Acked Flush 0 Done Done Sent false
9919 3352 Acked Write 5 Done Done Sent false
9920 3353 Acked Write 5 Done Done Sent false
9921 3354 Acked Write 2 Done Done Sent false
9922 3355 Acked Read 1 Done Done Sent false
9923 3356 Acked Flush 0 Done Done Sent false
9924 3357 Acked Flush 0 Done Done Sent false
9925 3358 Acked Read 8 Done Done Sent false
9926 3359 Acked Flush 0 Done Done Sent false
9927 3360 Acked Read 4 Done Done Sent false
9928 3361 Acked Read 3 Done Done Sent false
9929 3362 Acked Read 9 Done Done Sent false
9930 3363 Acked Write 5 Done Done Sent false
9931 3364 Acked Write 1 Done Done Sent false
9932 3365 Acked Write 9 Done Done Sent false
9933 3366 Acked Write 7 Done Done Sent false
9934 3367 Acked Read 3 Done Done Sent false
9935 3368 Acked Read 5 Done Done Sent false
9936 3369 Acked Read 8 Done Done Sent false
9937 3370 Acked Write 8 Done Done Sent false
9938 3371 Acked Read 9 Done Done Sent false
9939 3372 Acked Read 3 Done Done Sent false
9940 3373 Acked Write 8 Done Done Sent false
9941 3374 Acked Flush 0 Done Done Sent false
9942 3375 Acked Write 3 Done Done Sent false
9943 3376 Acked Read 3 Done Done Sent false
9944 3377 Acked Write 6 Done Done Sent false
9945 3378 Acked Read 1 Done Done Sent false
9946 3379 Acked Write 9 Done Done Sent false
9947 3380 Acked Write 5 Done Done Sent false
9948 3381 Acked Write 2 Done Done Sent false
9949 3382 Acked Read 10 Done Done Sent false
9950 3383 Acked Read 5 Done Done Sent false
9951 3384 Acked Read 10 Done Done Sent false
9952 3385 Acked Flush 0 Done Done Sent false
9953 3386 Acked Read 1 Done Done Sent false
9954 3387 Acked Read 5 Done Done Sent false
9955 3388 Acked Write 6 Done Done Sent false
9956 3389 Acked Write 6 Done Done Sent false
9957 3390 Acked Read 5 Done Done Sent false
9958 3391 Acked Flush 0 Done Done Sent false
9959 3392 Acked Read 3 Done Done Sent false
9960 3393 Acked Write 9 Done Done Sent false
9961 3394 Acked Write 5 Done Done Sent false
9962 3395 Acked Read 3 Done Done Sent false
9963 3396 Acked Flush 0 Done Done Sent false
9964 3397 Acked Read 5 Done Done Sent false
9965 3398 Acked Write 5 Done Done Sent false
9966 3399 Acked Write 4 Done Done Sent false
9967 3400 Acked Flush 0 Done Done Sent false
9968 3401 Acked Write 9 Done Done Sent false
9969 3402 Acked Write 6 Done Done Sent false
9970 3403 Acked Read 6 Done Done Sent false
9971 3404 Acked Write 6 Done Done Sent false
9972 3405 Acked Read 2 Done Done Sent false
9973 3406 Acked Write 8 Done Done Sent false
9974 3407 Acked Write 5 Done Done Sent false
9975 3408 Acked Write 1 Done Done Sent false
9976 3409 Acked Write 8 Done Done Sent false
9977 3410 Acked Write 10 Done Done Sent false
9978 3411 Acked Write 9 Done Done Sent false
9979 3412 Acked Read 4 Done Done Sent false
9980 3413 Acked Read 10 Done Done Sent false
9981 3414 Acked Write 5 Done Done Sent false
9982 STATES DS:0 DS:1 DS:2 TOTAL
9983 Sent 0 0 301 301
9984 Done 301 301 0 602
9985 Skipped 0 0 0 0
9986 Error 0 0 0 0
9987 Last Flush: Some(JobId(3400)) Some(JobId(3400)) Some(JobId(3113))
9988 Downstairs last five completed: 3113 3112 3111 3110 3109
9989 Upstairs last five completed: 3414 3413 3412 3411 3410
9990 CLIENT: Up:0 ds:301 act:2
99912025-02-07T20:09:28.150ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
99922025-02-07T20:09:29.152ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
99932025-02-07T20:09:29.248ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
99942025-02-07T20:09:29.248ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
99952025-02-07T20:09:29.248ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
99962025-02-07T20:09:29.248ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
99972025-02-07T20:09:29.248ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
99982025-02-07T20:09:30.154ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
99992025-02-07T20:09:31.156ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
10000 [8830] got start action
10001 [8830] I am going to restart
10002 Starting downstairs at port 8830
10003 Make output file at "/tmp/dsc/downstairs-8830.txt"
10004 [8830][2] reports Starting
10005 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1217)
10006 [8830][2] reports Running
10007 ----------------------------------------------------------------
10008 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10009 States: Active Active Offline
10010 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10011 3114 Acked Write 1 Done Done Sent false
10012 3115 Acked Write 2 Done Done Sent false
10013 3116 Acked Read 10 Done Done Sent false
10014 3117 Acked Write 9 Done Done Sent false
10015 3118 Acked Read 5 Done Done Sent false
10016 3119 Acked Write 9 Done Done Sent false
10017 3120 Acked Read 3 Done Done Sent false
10018 3121 Acked Read 3 Done Done Sent false
10019 3122 Acked Read 6 Done Done Sent false
10020 3123 Acked Read 5 Done Done Sent false
10021 3124 Acked Flush 0 Done Done Sent false
10022 3125 Acked Write 1 Done Done Sent false
10023 3126 Acked Read 4 Done Done Sent false
10024 3127 Acked Read 6 Done Done Sent false
10025 3128 Acked Write 10 Done Done Sent false
10026 3129 Acked Write 6 Done Done Sent false
10027 3130 Acked Write 8 Done Done Sent false
10028 3131 Acked Read 7 Done Done Sent false
10029 3132 Acked Read 6 Done Done Sent false
10030 3133 Acked Read 3 Done Done Sent false
10031 3134 Acked Read 3 Done Done Sent false
10032 3135 Acked Read 2 Done Done Sent false
10033 3136 Acked Read 6 Done Done Sent false
10034 3137 Acked Read 8 Done Done Sent false
10035 3138 Acked Write 6 Done Done Sent false
10036 3139 Acked Read 5 Done Done Sent false
10037 3140 Acked Write 8 Done Done Sent false
10038 3141 Acked Write 7 Done Done Sent false
10039 3142 Acked Read 8 Done Done Sent false
10040 3143 Acked Read 6 Done Done Sent false
10041 3144 Acked Read 3 Done Done Sent false
10042 3145 Acked Read 3 Done Done Sent false
10043 3146 Acked Read 10 Done Done Sent false
10044 3147 Acked Write 9 Done Done Sent false
10045 3148 Acked Write 3 Done Done Sent false
10046 3149 Acked Write 10 Done Done Sent false
10047 3150 Acked Write 6 Done Done Sent false
10048 3151 Acked Read 2 Done Done Sent false
10049 3152 Acked Write 10 Done Done Sent false
10050 3153 Acked Read 7 Done Done Sent false
10051 3154 Acked Write 3 Done Done Sent false
10052 3155 Acked Write 5 Done Done Sent false
10053 3156 Acked Read 4 Done Done Sent false
10054 3157 Acked Read 7 Done Done Sent false
10055 3158 Acked Write 9 Done Done Sent false
10056 3159 Acked Write 10 Done Done Sent false
10057 3160 Acked Write 1 Done Done Sent false
10058 3161 Acked Write 6 Done Done Sent false
10059 3162 Acked Write 1 Done Done Sent false
10060 3163 Acked Write 1 Done Done Sent false
10061 3164 Acked Read 2 Done Done Sent false
10062 3165 Acked Read 4 Done Done Sent false
10063 3166 Acked Read 2 Done Done Sent false
10064 3167 Acked Flush 0 Done Done Sent false
10065 3168 Acked Read 1 Done Done Sent false
10066 3169 Acked Read 8 Done Done Sent false
10067 3170 Acked Read 2 Done Done Sent false
10068 3171 Acked Flush 0 Done Done Sent false
10069 3172 Acked Read 9 Done Done Sent false
10070 3173 Acked Read 10 Done Done Sent false
10071 3174 Acked Write 6 Done Done Sent false
10072 3175 Acked Write 3 Done Done Sent false
10073 3176 Acked Write 3 Done Done Sent false
10074 3177 Acked Read 3 Done Done Sent false
10075 3178 Acked Write 8 Done Done Sent false
10076 3179 Acked Write 8 Done Done Sent false
10077 3180 Acked Read 5 Done Done Sent false
10078 3181 Acked Write 3 Done Done Sent false
10079 3182 Acked Write 4 Done Done Sent false
10080 3183 Acked Read 4 Done Done Sent false
10081 3184 Acked Write 5 Done Done Sent false
10082 3185 Acked Read 9 Done Done Sent false
10083 3186 Acked Write 3 Done Done Sent false
10084 3187 Acked Read 10 Done Done Sent false
10085 3188 Acked Write 7 Done Done Sent false
10086 3189 Acked Flush 0 Done Done Sent false
10087 3190 Acked Read 2 Done Done Sent false
10088 3191 Acked Write 3 Done Done Sent false
10089 3192 Acked Read 7 Done Done Sent false
10090 3193 Acked Write 7 Done Done Sent false
10091 3194 Acked Read 10 Done Done Sent false
10092 3195 Acked Write 9 Done Done Sent false
10093 3196 Acked Write 7 Done Done Sent false
10094 3197 Acked Write 2 Done Done Sent false
10095 3198 Acked Read 4 Done Done Sent false
10096 3199 Acked Read 6 Done Done Sent false
10097 3200 Acked Read 5 Done Done Sent false
10098 3201 Acked Flush 0 Done Done Sent false
10099 3202 Acked Write 4 Done Done Sent false
10100 3203 Acked Read 3 Done Done Sent false
10101 3204 Acked Read 2 Done Done Sent false
10102 3205 Acked Read 1 Done Done Sent false
10103 3206 Acked Write 10 Done Done Sent false
10104 3207 Acked Read 4 Done Done Sent false
10105 3208 Acked Read 9 Done Done Sent false
10106 3209 Acked Write 6 Done Done Sent false
10107 3210 Acked Read 6 Done Done Sent false
10108 3211 Acked Read 4 Done Done Sent false
10109 3212 Acked Flush 0 Done Done Sent false
10110 3213 Acked Read 4 Done Done Sent false
10111 3214 Acked Write 1 Done Done Sent false
10112 3215 Acked Read 8 Done Done Sent false
10113 3216 Acked Read 1 Done Done Sent false
10114 3217 Acked Read 10 Done Done Sent false
10115 3218 Acked Write 8 Done Done Sent false
10116 3219 Acked Flush 0 Done Done Sent false
10117 3220 Acked Read 3 Done Done Sent false
10118 3221 Acked Write 5 Done Done Sent false
10119 3222 Acked Read 9 Done Done Sent false
10120 3223 Acked Read 2 Done Done Sent false
10121 3224 Acked Read 4 Done Done Sent false
10122 3225 Acked Write 4 Done Done Sent false
10123 3226 Acked Flush 0 Done Done Sent false
10124 3227 Acked Write 1 Done Done Sent false
10125 3228 Acked Write 10 Done Done Sent false
10126 3229 Acked Write 5 Done Done Sent false
10127 3230 Acked Write 2 Done Done Sent false
10128 3231 Acked Write 7 Done Done Sent false
10129 3232 Acked Read 6 Done Done Sent false
10130 3233 Acked Write 8 Done Done Sent false
10131 3234 Acked Flush 0 Done Done Sent false
10132 3235 Acked Read 1 Done Done Sent false
10133 3236 Acked Write 3 Done Done Sent false
10134 3237 Acked Write 9 Done Done Sent false
10135 3238 Acked Write 1 Done Done Sent false
10136 3239 Acked Write 3 Done Done Sent false
10137 3240 Acked Write 9 Done Done Sent false
10138 3241 Acked Write 2 Done Done Sent false
10139 3242 Acked Write 10 Done Done Sent false
10140 3243 Acked Write 4 Done Done Sent false
10141 3244 Acked Flush 0 Done Done Sent false
10142 3245 Acked Read 2 Done Done Sent false
10143 3246 Acked Write 1 Done Done Sent false
10144 3247 Acked Read 4 Done Done Sent false
10145 3248 Acked Read 7 Done Done Sent false
10146 3249 Acked Read 3 Done Done Sent false
10147 3250 Acked Read 6 Done Done Sent false
10148 3251 Acked Read 8 Done Done Sent false
10149 3252 Acked Read 3 Done Done Sent false
10150 3253 Acked Read 6 Done Done Sent false
10151 3254 Acked Read 5 Done Done Sent false
10152 3255 Acked Read 1 Done Done Sent false
10153 3256 Acked Read 7 Done Done Sent false
10154 3257 Acked Read 8 Done Done Sent false
10155 3258 Acked Write 2 Done Done Sent false
10156 3259 Acked Read 7 Done Done Sent false
10157 3260 Acked Flush 0 Done Done Sent false
10158 3261 Acked Write 3 Done Done Sent false
10159 3262 Acked Read 2 Done Done Sent false
10160 3263 Acked Read 9 Done Done Sent false
10161 3264 Acked Write 7 Done Done Sent false
10162 3265 Acked Read 9 Done Done Sent false
10163 3266 Acked Read 9 Done Done Sent false
10164 3267 Acked Read 2 Done Done Sent false
10165 3268 Acked Read 6 Done Done Sent false
10166 3269 Acked Write 8 Done Done Sent false
10167 3270 Acked Read 1 Done Done Sent false
10168 3271 Acked Read 9 Done Done Sent false
10169 3272 Acked Write 5 Done Done Sent false
10170 3273 Acked Flush 0 Done Done Sent false
10171 3274 Acked Write 8 Done Done Sent false
10172 3275 Acked Write 9 Done Done Sent false
10173 3276 Acked Read 1 Done Done Sent false
10174 3277 Acked Write 3 Done Done Sent false
10175 3278 Acked Read 5 Done Done Sent false
10176 3279 Acked Write 3 Done Done Sent false
10177 3280 Acked Read 6 Done Done Sent false
10178 3281 Acked Write 8 Done Done Sent false
10179 3282 Acked Read 1 Done Done Sent false
10180 3283 Acked Write 2 Done Done Sent false
10181 3284 Acked Write 1 Done Done Sent false
10182 3285 Acked Write 6 Done Done Sent false
10183 3286 Acked Read 3 Done Done Sent false
10184 3287 Acked Read 1 Done Done Sent false
10185 3288 Acked Read 5 Done Done Sent false
10186 3289 Acked Read 4 Done Done Sent false
10187 3290 Acked Write 5 Done Done Sent false
10188 3291 Acked Read 7 Done Done Sent false
10189 3292 Acked Read 5 Done Done Sent false
10190 3293 Acked Read 4 Done Done Sent false
10191 3294 Acked Read 8 Done Done Sent false
10192 3295 Acked Write 10 Done Done Sent false
10193 3296 Acked Read 9 Done Done Sent false
10194 3297 Acked Write 10 Done Done Sent false
10195 3298 Acked Read 8 Done Done Sent false
10196 3299 Acked Write 6 Done Done Sent false
10197 3300 Acked Read 1 Done Done Sent false
10198 3301 Acked Write 8 Done Done Sent false
10199 3302 Acked Flush 0 Done Done Sent false
10200 3303 Acked Read 1 Done Done Sent false
10201 3304 Acked Read 5 Done Done Sent false
10202 3305 Acked Flush 0 Done Done Sent false
10203 3306 Acked Read 5 Done Done Sent false
10204 3307 Acked Write 5 Done Done Sent false
10205 3308 Acked Read 4 Done Done Sent false
10206 3309 Acked Flush 0 Done Done Sent false
10207 3310 Acked Write 9 Done Done Sent false
10208 3311 Acked Read 3 Done Done Sent false
10209 3312 Acked Write 3 Done Done Sent false
10210 3313 Acked Write 10 Done Done Sent false
10211 3314 Acked Read 4 Done Done Sent false
10212 3315 Acked Write 9 Done Done Sent false
10213 3316 Acked Read 1 Done Done Sent false
10214 3317 Acked Read 5 Done Done Sent false
10215 3318 Acked Read 9 Done Done Sent false
10216 3319 Acked Write 3 Done Done Sent false
10217 3320 Acked Write 1 Done Done Sent false
10218 3321 Acked Read 8 Done Done Sent false
10219 3322 Acked Read 6 Done Done Sent false
10220 3323 Acked Write 10 Done Done Sent false
10221 3324 Acked Read 3 Done Done Sent false
10222 3325 Acked Write 1 Done Done Sent false
10223 3326 Acked Flush 0 Done Done Sent false
10224 3327 Acked Read 2 Done Done Sent false
10225 3328 Acked Flush 0 Done Done Sent false
10226 3329 Acked Write 9 Done Done Sent false
10227 3330 Acked Read 3 Done Done Sent false
10228 3331 Acked Read 4 Done Done Sent false
10229 3332 Acked Flush 0 Done Done Sent false
10230 3333 Acked Read 4 Done Done Sent false
10231 3334 Acked Read 3 Done Done Sent false
10232 3335 Acked Write 9 Done Done Sent false
10233 3336 Acked Read 2 Done Done Sent false
10234 3337 Acked Flush 0 Done Done Sent false
10235 3338 Acked Read 5 Done Done Sent false
10236 3339 Acked Write 5 Done Done Sent false
10237 3340 Acked Read 4 Done Done Sent false
10238 3341 Acked Write 4 Done Done Sent false
10239 3342 Acked Read 4 Done Done Sent false
10240 3343 Acked Read 2 Done Done Sent false
10241 3344 Acked Write 1 Done Done Sent false
10242 3345 Acked Flush 0 Done Done Sent false
10243 3346 Acked Write 5 Done Done Sent false
10244 3347 Acked Write 6 Done Done Sent false
10245 3348 Acked Read 10 Done Done Sent false
10246 3349 Acked Write 5 Done Done Sent false
10247 3350 Acked Read 10 Done Done Sent false
10248 3351 Acked Flush 0 Done Done Sent false
10249 3352 Acked Write 5 Done Done Sent false
10250 3353 Acked Write 5 Done Done Sent false
10251 3354 Acked Write 2 Done Done Sent false
10252 3355 Acked Read 1 Done Done Sent false
10253 3356 Acked Flush 0 Done Done Sent false
10254 3357 Acked Flush 0 Done Done Sent false
10255 3358 Acked Read 8 Done Done Sent false
10256 3359 Acked Flush 0 Done Done Sent false
10257 3360 Acked Read 4 Done Done Sent false
10258 3361 Acked Read 3 Done Done Sent false
10259 3362 Acked Read 9 Done Done Sent false
10260 3363 Acked Write 5 Done Done Sent false
10261 3364 Acked Write 1 Done Done Sent false
10262 3365 Acked Write 9 Done Done Sent false
10263 3366 Acked Write 7 Done Done Sent false
10264 3367 Acked Read 3 Done Done Sent false
10265 3368 Acked Read 5 Done Done Sent false
10266 3369 Acked Read 8 Done Done Sent false
10267 3370 Acked Write 8 Done Done Sent false
10268 3371 Acked Read 9 Done Done Sent false
10269 3372 Acked Read 3 Done Done Sent false
10270 3373 Acked Write 8 Done Done Sent false
10271 3374 Acked Flush 0 Done Done Sent false
10272 3375 Acked Write 3 Done Done Sent false
10273 3376 Acked Read 3 Done Done Sent false
10274 3377 Acked Write 6 Done Done Sent false
10275 3378 Acked Read 1 Done Done Sent false
10276 3379 Acked Write 9 Done Done Sent false
10277 3380 Acked Write 5 Done Done Sent false
10278 3381 Acked Write 2 Done Done Sent false
10279 3382 Acked Read 10 Done Done Sent false
10280 3383 Acked Read 5 Done Done Sent false
10281 3384 Acked Read 10 Done Done Sent false
10282 3385 Acked Flush 0 Done Done Sent false
10283 3386 Acked Read 1 Done Done Sent false
10284 3387 Acked Read 5 Done Done Sent false
10285 3388 Acked Write 6 Done Done Sent false
10286 3389 Acked Write 6 Done Done Sent false
10287 3390 Acked Read 5 Done Done Sent false
10288 3391 Acked Flush 0 Done Done Sent false
10289 3392 Acked Read 3 Done Done Sent false
10290 3393 Acked Write 9 Done Done Sent false
10291 3394 Acked Write 5 Done Done Sent false
10292 3395 Acked Read 3 Done Done Sent false
10293 3396 Acked Flush 0 Done Done Sent false
10294 3397 Acked Read 5 Done Done Sent false
10295 3398 Acked Write 5 Done Done Sent false
10296 3399 Acked Write 4 Done Done Sent false
10297 3400 Acked Flush 0 Done Done Sent false
10298 3401 Acked Write 9 Done Done Sent false
10299 3402 Acked Write 6 Done Done Sent false
10300 3403 Acked Read 6 Done Done Sent false
10301 3404 Acked Write 6 Done Done Sent false
10302 3405 Acked Read 2 Done Done Sent false
10303 3406 Acked Write 8 Done Done Sent false
10304 3407 Acked Write 5 Done Done Sent false
10305 3408 Acked Write 1 Done Done Sent false
10306 3409 Acked Write 8 Done Done Sent false
10307 3410 Acked Write 10 Done Done Sent false
10308 3411 Acked Write 9 Done Done Sent false
10309 3412 Acked Read 4 Done Done Sent false
10310 3413 Acked Read 10 Done Done Sent false
10311 3414 Acked Write 5 Done Done Sent false
10312 STATES DS:0 DS:1 DS:2 TOTAL
10313 Sent 0 0 301 301
10314 Done 301 301 0 602
10315 Skipped 0 0 0 0
10316 Error 0 0 0 0
10317 Last Flush: Some(JobId(3400)) Some(JobId(3400)) Some(JobId(3113))
10318 Downstairs last five completed: 3113 3112 3111 3110 3109
10319 Upstairs last five completed: 3414 3413 3412 3411 3410
10320 CLIENT: Up:0 ds:301 act:2
103212025-02-07T20:09:32.505ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
103222025-02-07T20:09:33.159ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
103232025-02-07T20:09:34.161ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
103242025-02-07T20:09:35.163ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
10325 ----------------------------------------------------------------
10326 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10327 States: Active Active Offline
10328 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10329 3114 Acked Write 1 Done Done Sent false
10330 3115 Acked Write 2 Done Done Sent false
10331 3116 Acked Read 10 Done Done Sent false
10332 3117 Acked Write 9 Done Done Sent false
10333 3118 Acked Read 5 Done Done Sent false
10334 3119 Acked Write 9 Done Done Sent false
10335 3120 Acked Read 3 Done Done Sent false
10336 3121 Acked Read 3 Done Done Sent false
10337 3122 Acked Read 6 Done Done Sent false
10338 3123 Acked Read 5 Done Done Sent false
10339 3124 Acked Flush 0 Done Done Sent false
10340 3125 Acked Write 1 Done Done Sent false
10341 3126 Acked Read 4 Done Done Sent false
10342 3127 Acked Read 6 Done Done Sent false
10343 3128 Acked Write 10 Done Done Sent false
10344 3129 Acked Write 6 Done Done Sent false
10345 3130 Acked Write 8 Done Done Sent false
10346 3131 Acked Read 7 Done Done Sent false
10347 3132 Acked Read 6 Done Done Sent false
10348 3133 Acked Read 3 Done Done Sent false
10349 3134 Acked Read 3 Done Done Sent false
10350 3135 Acked Read 2 Done Done Sent false
10351 3136 Acked Read 6 Done Done Sent false
10352 3137 Acked Read 8 Done Done Sent false
10353 3138 Acked Write 6 Done Done Sent false
10354 3139 Acked Read 5 Done Done Sent false
10355 3140 Acked Write 8 Done Done Sent false
10356 3141 Acked Write 7 Done Done Sent false
10357 3142 Acked Read 8 Done Done Sent false
10358 3143 Acked Read 6 Done Done Sent false
10359 3144 Acked Read 3 Done Done Sent false
10360 3145 Acked Read 3 Done Done Sent false
10361 3146 Acked Read 10 Done Done Sent false
10362 3147 Acked Write 9 Done Done Sent false
10363 3148 Acked Write 3 Done Done Sent false
10364 3149 Acked Write 10 Done Done Sent false
10365 3150 Acked Write 6 Done Done Sent false
10366 3151 Acked Read 2 Done Done Sent false
10367 3152 Acked Write 10 Done Done Sent false
10368 3153 Acked Read 7 Done Done Sent false
10369 3154 Acked Write 3 Done Done Sent false
10370 3155 Acked Write 5 Done Done Sent false
10371 3156 Acked Read 4 Done Done Sent false
10372 3157 Acked Read 7 Done Done Sent false
10373 3158 Acked Write 9 Done Done Sent false
10374 3159 Acked Write 10 Done Done Sent false
10375 3160 Acked Write 1 Done Done Sent false
10376 3161 Acked Write 6 Done Done Sent false
10377 3162 Acked Write 1 Done Done Sent false
10378 3163 Acked Write 1 Done Done Sent false
10379 3164 Acked Read 2 Done Done Sent false
10380 3165 Acked Read 4 Done Done Sent false
10381 3166 Acked Read 2 Done Done Sent false
10382 3167 Acked Flush 0 Done Done Sent false
10383 3168 Acked Read 1 Done Done Sent false
10384 3169 Acked Read 8 Done Done Sent false
10385 3170 Acked Read 2 Done Done Sent false
10386 3171 Acked Flush 0 Done Done Sent false
10387 3172 Acked Read 9 Done Done Sent false
10388 3173 Acked Read 10 Done Done Sent false
10389 3174 Acked Write 6 Done Done Sent false
10390 3175 Acked Write 3 Done Done Sent false
10391 3176 Acked Write 3 Done Done Sent false
10392 3177 Acked Read 3 Done Done Sent false
10393 3178 Acked Write 8 Done Done Sent false
10394 3179 Acked Write 8 Done Done Sent false
10395 3180 Acked Read 5 Done Done Sent false
10396 3181 Acked Write 3 Done Done Sent false
10397 3182 Acked Write 4 Done Done Sent false
10398 3183 Acked Read 4 Done Done Sent false
10399 3184 Acked Write 5 Done Done Sent false
10400 3185 Acked Read 9 Done Done Sent false
10401 3186 Acked Write 3 Done Done Sent false
10402 3187 Acked Read 10 Done Done Sent false
10403 3188 Acked Write 7 Done Done Sent false
10404 3189 Acked Flush 0 Done Done Sent false
10405 3190 Acked Read 2 Done Done Sent false
10406 3191 Acked Write 3 Done Done Sent false
10407 3192 Acked Read 7 Done Done Sent false
10408 3193 Acked Write 7 Done Done Sent false
10409 3194 Acked Read 10 Done Done Sent false
10410 3195 Acked Write 9 Done Done Sent false
10411 3196 Acked Write 7 Done Done Sent false
10412 3197 Acked Write 2 Done Done Sent false
10413 3198 Acked Read 4 Done Done Sent false
10414 3199 Acked Read 6 Done Done Sent false
10415 3200 Acked Read 5 Done Done Sent false
10416 3201 Acked Flush 0 Done Done Sent false
10417 3202 Acked Write 4 Done Done Sent false
10418 3203 Acked Read 3 Done Done Sent false
10419 3204 Acked Read 2 Done Done Sent false
10420 3205 Acked Read 1 Done Done Sent false
10421 3206 Acked Write 10 Done Done Sent false
10422 3207 Acked Read 4 Done Done Sent false
10423 3208 Acked Read 9 Done Done Sent false
10424 3209 Acked Write 6 Done Done Sent false
10425 3210 Acked Read 6 Done Done Sent false
10426 3211 Acked Read 4 Done Done Sent false
10427 3212 Acked Flush 0 Done Done Sent false
10428 3213 Acked Read 4 Done Done Sent false
10429 3214 Acked Write 1 Done Done Sent false
10430 3215 Acked Read 8 Done Done Sent false
10431 3216 Acked Read 1 Done Done Sent false
10432 3217 Acked Read 10 Done Done Sent false
10433 3218 Acked Write 8 Done Done Sent false
10434 3219 Acked Flush 0 Done Done Sent false
10435 3220 Acked Read 3 Done Done Sent false
10436 3221 Acked Write 5 Done Done Sent false
10437 3222 Acked Read 9 Done Done Sent false
10438 3223 Acked Read 2 Done Done Sent false
10439 3224 Acked Read 4 Done Done Sent false
10440 3225 Acked Write 4 Done Done Sent false
10441 3226 Acked Flush 0 Done Done Sent false
10442 3227 Acked Write 1 Done Done Sent false
10443 3228 Acked Write 10 Done Done Sent false
10444 3229 Acked Write 5 Done Done Sent false
10445 3230 Acked Write 2 Done Done Sent false
10446 3231 Acked Write 7 Done Done Sent false
10447 3232 Acked Read 6 Done Done Sent false
10448 3233 Acked Write 8 Done Done Sent false
10449 3234 Acked Flush 0 Done Done Sent false
10450 3235 Acked Read 1 Done Done Sent false
10451 3236 Acked Write 3 Done Done Sent false
10452 3237 Acked Write 9 Done Done Sent false
10453 3238 Acked Write 1 Done Done Sent false
10454 3239 Acked Write 3 Done Done Sent false
10455 3240 Acked Write 9 Done Done Sent false
10456 3241 Acked Write 2 Done Done Sent false
10457 3242 Acked Write 10 Done Done Sent false
10458 3243 Acked Write 4 Done Done Sent false
10459 3244 Acked Flush 0 Done Done Sent false
10460 3245 Acked Read 2 Done Done Sent false
10461 3246 Acked Write 1 Done Done Sent false
10462 3247 Acked Read 4 Done Done Sent false
10463 3248 Acked Read 7 Done Done Sent false
10464 3249 Acked Read 3 Done Done Sent false
10465 3250 Acked Read 6 Done Done Sent false
10466 3251 Acked Read 8 Done Done Sent false
10467 3252 Acked Read 3 Done Done Sent false
10468 3253 Acked Read 6 Done Done Sent false
10469 3254 Acked Read 5 Done Done Sent false
10470 3255 Acked Read 1 Done Done Sent false
10471 3256 Acked Read 7 Done Done Sent false
10472 3257 Acked Read 8 Done Done Sent false
10473 3258 Acked Write 2 Done Done Sent false
10474 3259 Acked Read 7 Done Done Sent false
10475 3260 Acked Flush 0 Done Done Sent false
10476 3261 Acked Write 3 Done Done Sent false
10477 3262 Acked Read 2 Done Done Sent false
10478 3263 Acked Read 9 Done Done Sent false
10479 3264 Acked Write 7 Done Done Sent false
10480 3265 Acked Read 9 Done Done Sent false
10481 3266 Acked Read 9 Done Done Sent false
10482 3267 Acked Read 2 Done Done Sent false
10483 3268 Acked Read 6 Done Done Sent false
10484 3269 Acked Write 8 Done Done Sent false
10485 3270 Acked Read 1 Done Done Sent false
10486 3271 Acked Read 9 Done Done Sent false
10487 3272 Acked Write 5 Done Done Sent false
10488 3273 Acked Flush 0 Done Done Sent false
10489 3274 Acked Write 8 Done Done Sent false
10490 3275 Acked Write 9 Done Done Sent false
10491 3276 Acked Read 1 Done Done Sent false
10492 3277 Acked Write 3 Done Done Sent false
10493 3278 Acked Read 5 Done Done Sent false
10494 3279 Acked Write 3 Done Done Sent false
10495 3280 Acked Read 6 Done Done Sent false
10496 3281 Acked Write 8 Done Done Sent false
10497 3282 Acked Read 1 Done Done Sent false
10498 3283 Acked Write 2 Done Done Sent false
10499 3284 Acked Write 1 Done Done Sent false
10500 3285 Acked Write 6 Done Done Sent false
10501 3286 Acked Read 3 Done Done Sent false
10502 3287 Acked Read 1 Done Done Sent false
10503 3288 Acked Read 5 Done Done Sent false
10504 3289 Acked Read 4 Done Done Sent false
10505 3290 Acked Write 5 Done Done Sent false
10506 3291 Acked Read 7 Done Done Sent false
10507 3292 Acked Read 5 Done Done Sent false
10508 3293 Acked Read 4 Done Done Sent false
10509 3294 Acked Read 8 Done Done Sent false
10510 3295 Acked Write 10 Done Done Sent false
10511 3296 Acked Read 9 Done Done Sent false
10512 3297 Acked Write 10 Done Done Sent false
10513 3298 Acked Read 8 Done Done Sent false
10514 3299 Acked Write 6 Done Done Sent false
10515 3300 Acked Read 1 Done Done Sent false
10516 3301 Acked Write 8 Done Done Sent false
10517 3302 Acked Flush 0 Done Done Sent false
10518 3303 Acked Read 1 Done Done Sent false
10519 3304 Acked Read 5 Done Done Sent false
10520 3305 Acked Flush 0 Done Done Sent false
10521 3306 Acked Read 5 Done Done Sent false
10522 3307 Acked Write 5 Done Done Sent false
10523 3308 Acked Read 4 Done Done Sent false
10524 3309 Acked Flush 0 Done Done Sent false
10525 3310 Acked Write 9 Done Done Sent false
10526 3311 Acked Read 3 Done Done Sent false
10527 3312 Acked Write 3 Done Done Sent false
10528 3313 Acked Write 10 Done Done Sent false
10529 3314 Acked Read 4 Done Done Sent false
10530 3315 Acked Write 9 Done Done Sent false
10531 3316 Acked Read 1 Done Done Sent false
10532 3317 Acked Read 5 Done Done Sent false
10533 3318 Acked Read 9 Done Done Sent false
10534 3319 Acked Write 3 Done Done Sent false
10535 3320 Acked Write 1 Done Done Sent false
10536 3321 Acked Read 8 Done Done Sent false
10537 3322 Acked Read 6 Done Done Sent false
10538 3323 Acked Write 10 Done Done Sent false
10539 3324 Acked Read 3 Done Done Sent false
10540 3325 Acked Write 1 Done Done Sent false
10541 3326 Acked Flush 0 Done Done Sent false
10542 3327 Acked Read 2 Done Done Sent false
10543 3328 Acked Flush 0 Done Done Sent false
10544 3329 Acked Write 9 Done Done Sent false
10545 3330 Acked Read 3 Done Done Sent false
10546 3331 Acked Read 4 Done Done Sent false
10547 3332 Acked Flush 0 Done Done Sent false
10548 3333 Acked Read 4 Done Done Sent false
10549 3334 Acked Read 3 Done Done Sent false
10550 3335 Acked Write 9 Done Done Sent false
10551 3336 Acked Read 2 Done Done Sent false
10552 3337 Acked Flush 0 Done Done Sent false
10553 3338 Acked Read 5 Done Done Sent false
10554 3339 Acked Write 5 Done Done Sent false
10555 3340 Acked Read 4 Done Done Sent false
10556 3341 Acked Write 4 Done Done Sent false
10557 3342 Acked Read 4 Done Done Sent false
10558 3343 Acked Read 2 Done Done Sent false
10559 3344 Acked Write 1 Done Done Sent false
10560 3345 Acked Flush 0 Done Done Sent false
10561 3346 Acked Write 5 Done Done Sent false
10562 3347 Acked Write 6 Done Done Sent false
10563 3348 Acked Read 10 Done Done Sent false
10564 3349 Acked Write 5 Done Done Sent false
10565 3350 Acked Read 10 Done Done Sent false
10566 3351 Acked Flush 0 Done Done Sent false
10567 3352 Acked Write 5 Done Done Sent false
10568 3353 Acked Write 5 Done Done Sent false
10569 3354 Acked Write 2 Done Done Sent false
10570 3355 Acked Read 1 Done Done Sent false
10571 3356 Acked Flush 0 Done Done Sent false
10572 3357 Acked Flush 0 Done Done Sent false
10573 3358 Acked Read 8 Done Done Sent false
10574 3359 Acked Flush 0 Done Done Sent false
10575 3360 Acked Read 4 Done Done Sent false
10576 3361 Acked Read 3 Done Done Sent false
10577 3362 Acked Read 9 Done Done Sent false
10578 3363 Acked Write 5 Done Done Sent false
10579 3364 Acked Write 1 Done Done Sent false
10580 3365 Acked Write 9 Done Done Sent false
10581 3366 Acked Write 7 Done Done Sent false
10582 3367 Acked Read 3 Done Done Sent false
10583 3368 Acked Read 5 Done Done Sent false
10584 3369 Acked Read 8 Done Done Sent false
10585 3370 Acked Write 8 Done Done Sent false
10586 3371 Acked Read 9 Done Done Sent false
10587 3372 Acked Read 3 Done Done Sent false
10588 3373 Acked Write 8 Done Done Sent false
10589 3374 Acked Flush 0 Done Done Sent false
10590 3375 Acked Write 3 Done Done Sent false
10591 3376 Acked Read 3 Done Done Sent false
10592 3377 Acked Write 6 Done Done Sent false
10593 3378 Acked Read 1 Done Done Sent false
10594 3379 Acked Write 9 Done Done Sent false
10595 3380 Acked Write 5 Done Done Sent false
10596 3381 Acked Write 2 Done Done Sent false
10597 3382 Acked Read 10 Done Done Sent false
10598 3383 Acked Read 5 Done Done Sent false
10599 3384 Acked Read 10 Done Done Sent false
10600 3385 Acked Flush 0 Done Done Sent false
10601 3386 Acked Read 1 Done Done Sent false
10602 3387 Acked Read 5 Done Done Sent false
10603 3388 Acked Write 6 Done Done Sent false
10604 3389 Acked Write 6 Done Done Sent false
10605 3390 Acked Read 5 Done Done Sent false
10606 3391 Acked Flush 0 Done Done Sent false
10607 3392 Acked Read 3 Done Done Sent false
10608 3393 Acked Write 9 Done Done Sent false
10609 3394 Acked Write 5 Done Done Sent false
10610 3395 Acked Read 3 Done Done Sent false
10611 3396 Acked Flush 0 Done Done Sent false
10612 3397 Acked Read 5 Done Done Sent false
10613 3398 Acked Write 5 Done Done Sent false
10614 3399 Acked Write 4 Done Done Sent false
10615 3400 Acked Flush 0 Done Done Sent false
10616 3401 Acked Write 9 Done Done Sent false
10617 3402 Acked Write 6 Done Done Sent false
10618 3403 Acked Read 6 Done Done Sent false
10619 3404 Acked Write 6 Done Done Sent false
10620 3405 Acked Read 2 Done Done Sent false
10621 3406 Acked Write 8 Done Done Sent false
10622 3407 Acked Write 5 Done Done Sent false
10623 3408 Acked Write 1 Done Done Sent false
10624 3409 Acked Write 8 Done Done Sent false
10625 3410 Acked Write 10 Done Done Sent false
10626 3411 Acked Write 9 Done Done Sent false
10627 3412 Acked Read 4 Done Done Sent false
10628 3413 Acked Read 10 Done Done Sent false
10629 3414 Acked Write 5 Done Done Sent false
10630 STATES DS:0 DS:1 DS:2 TOTAL
10631 Sent 0 0 301 301
10632 Done 301 301 0 602
10633 Skipped 0 0 0 0
10634 Error 0 0 0 0
10635 Last Flush: Some(JobId(3400)) Some(JobId(3400)) Some(JobId(3113))
10636 Downstairs last five completed: 3113 3112 3111 3110 3109
10637 Upstairs last five completed: 3414 3413 3412 3411 3410
10638 CLIENT: Up:0 ds:301 act:2
106392025-02-07T20:09:36.164ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106402025-02-07T20:09:37.506ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106412025-02-07T20:09:38.167ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106422025-02-07T20:09:39.169ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106432025-02-07T20:09:39.249ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106442025-02-07T20:09:39.249ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60960) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106452025-02-07T20:09:39.250ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106462025-02-07T20:09:39.251ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106472025-02-07T20:09:39.251ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106482025-02-07T20:09:39.251ZINFOcrucible: send last flush ID to this DS: Some(JobId(3113)) = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106492025-02-07T20:09:39.251ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106502025-02-07T20:09:39.251ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3113)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106512025-02-07T20:09:39.251ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3113)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
10652 ----------------------------------------------------------------
10653 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:14
10654 States: Active Active Active
10655 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10656 3401 Acked Write 9 Done Done Done true
10657 3402 Acked Write 6 Done Done Done true
10658 3403 Acked Read 6 Done Done Done true
10659 3404 Acked Write 6 Done Done Done true
10660 3405 Acked Read 2 Done Done Done true
10661 3406 Acked Write 8 Done Done Done true
10662 3407 Acked Write 5 Done Done Done true
10663 3408 Acked Write 1 Done Done Done true
10664 3409 Acked Write 8 Done Done Done true
10665 3410 Acked Write 10 Done Done Done true
10666 3411 Acked Write 9 Done Done Done true
10667 3412 Acked Read 4 Done Done Done true
10668 3413 Acked Read 10 Done Done Done true
10669 3414 Acked Write 5 Done Done Done true
10670 STATES DS:0 DS:1 DS:2 TOTAL
10671 Sent 0 0 0 0
10672 Done 14 14 14 42
10673 Skipped 0 0 0 0
10674 Error 0 0 0 0
10675 Last Flush: Some(JobId(3400)) Some(JobId(3400)) Some(JobId(3400))
10676 Downstairs last five completed: 3400 3399 3398 3397 3396
10677 Upstairs last five completed: 3414 3413 3412 3411 3410
10678 CLIENT: Up:0 ds:14 act:3
10679 ----------------------------------------------------------------
10680 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10681 Downstairs last five completed: 3415 3414 3413 3412 3411
10682 Upstairs last five completed: 3415 3414 3413 3412 3411
10683 CLIENT: Up:0 ds:0 act:3
10684 Replay: All jobs finished
10685 Main task has work to do, go find it
10686 got dsc Stop(0)
10687 stop 0
10688 Feb 07 20:09:43.494 INFO request completed[8810] Got stop action so:false kr:false
10689 , latency_us: 334, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 42f1f162-6be4-471a-9f0a-44fbbfd23c11, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
106902025-02-07T20:09:43.495ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106912025-02-07T20:09:43.495ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
106922025-02-07T20:09:43.495ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
10693 Feb 07 20:09:43.495{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-07T20:09:43.495499953Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
10694 INFO request completed, latency_us: 153, response_code: 200, uri: /state/cid/0, method: GET, req_id: 13684425-96b3-43b3-9802-87571a3d2782, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
10695 [8810] Exited with: ExitStatus(unix_wait_status(9))
10696 [8810][0] reports Exit
10697 Feb 07 20:09:47.497 INFO request completed, latency_us: 281, response_code: 200, uri: /state/cid/0, method: GET, req_id: 14a69af9-6291-4e5c-958a-d18dbafc097c, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
10698 001/300 Flush
10699 002/300 Read block 3631 len 4096
10700 003/300 Write block 929 len 8192 data: 3 3
10701 004/300 Read block 2510 len 12288
10702 005/300 Write block 3054 len 8192 data: 3 2
10703 006/300 Write block 3998 len 28672 data: 3 3 3 2 2 2 2
10704 007/300 Read block 340 len 16384
10705 008/300 Read block 509 len 16384
10706 009/300 Write block 2660 len 32768 data: 2 2 2 2 2 2 3 3
10707 010/300 Flush
10708 011/300 Write block 2927 len 40960 data: 2 2 2 2 2 2 3 3 3 3
10709 012/300 Write block 4646 len 24576 data: 2 3 4 4 5 5
10710 013/300 Read block 2038 len 12288
10711 014/300 Write block 3153 len 28672 data: 3 2 2 3 3 3 3
10712 015/300 Write block 2782 len 32768 data: 3 2 2 2 2 2 2 2
10713 016/300 Write block 1811 len 16384 data: 3 3 3 3
10714 017/300 Write block 222 len 20480 data: 2 2 2 2 2
10715 018/300 Read block 4052 len 12288
10716 019/300 Read block 1223 len 8192
10717 020/300 Write block 2952 len 20480 data: 4 5 4 4 4
10718 021/300 Write block 2668 len 36864 data: 3 3 3 3 3 4 4 4 4
10719 022/300 Write block 556 len 40960 data: 2 2 2 2 4 5 5 4 4 2
10720 023/300 Read block 3857 len 40960
10721 024/300 Write block 4966 len 24576 data: 5 5 3 3 2 2
10722 025/300 Read block 1809 len 40960
10723 026/300 Write block 3651 len 4096 data: 2
10724 027/300 Read block 1390 len 4096
10725 028/300 Write block 3072 len 4096 data: 3
10726 029/300 Write block 2175 len 8192 data: 3 3
10727 030/300 Write block 1044 len 16384 data: 3 3 3 3
10728 031/300 Write block 3418 len 20480 data: 2 2 2 2 2
10729 032/300 Read block 3348 len 20480
10730 033/300 Write block 3477 len 28672 data: 6 6 5 4 4 4 5
10731 034/300 Read block 4428 len 4096
10732 035/300 Write block 3744 len 32768 data: 3 2 3 3 3 4 4 4
10733 036/300 Read block 1945 len 24576
10734 037/300 Read block 1078 len 20480
10735 038/300 Flush
10736 039/300 Flush
10737 040/300 Read block 3968 len 32768
10738 041/300 Write block 4467 len 4096 data: 3
10739 042/300 Read block 1340 len 16384
10740 043/300 Write block 462 len 8192 data: 3 3
10741 044/300 Write block 1329 len 12288 data: 2 3 3
10742 045/300 Write block 2314 len 28672 data: 6 4 4 4 3 3 3
10743 046/300 Read block 1111 len 24576
10744 047/300 Read block 4015 len 24576
10745 048/300 Read block 2381 len 36864
10746 049/300 Read block 3522 len 28672
10747 050/300 Read block 4315 len 12288
10748 051/300 Write block 498 len 32768 data: 3 3 3 3 3 3 3 4
10749 052/300 Read block 4526 len 8192
10750 053/300 Read block 1559 len 36864
10751 054/300 Read block 1157 len 32768
10752 055/300 Read block 1964 len 32768
10753 056/300 Write block 4301 len 4096 data: 2
10754 057/300 Read block 2350 len 36864
10755 058/300 Write block 3669 len 4096 data: 5
10756 059/300 Write block 2432 len 4096 data: 4
10757 060/300 Write block 2395 len 36864 data: 4 4 4 4 4 4 4 2 2
10758 061/300 Read block 399 len 36864
10759 062/300 Write block 3238 len 28672 data: 2 2 2 2 3 2 2
10760 063/300 Read block 2152 len 4096
10761 064/300 Write block 2311 len 4096 data: 5
10762 065/300 Read block 3033 len 12288
10763 066/300 Read block 4156 len 20480
10764 067/300 Read block 4546 len 36864
10765 068/300 Write block 409 len 16384 data: 2 2 2 2
10766 069/300 Read block 2537 len 4096
10767 070/300 Read block 3208 len 28672
10768 071/300 Read block 80 len 16384
10769 072/300 Read block 2187 len 4096
10770 073/300 Read block 2619 len 8192
10771 074/300 Flush
10772 075/300 Read block 4832 len 12288
10773 076/300 Read block 1443 len 4096
10774 077/300 Write block 729 len 40960 data: 3 3 3 2 2 2 2 2 2 2
10775 078/300 Flush
10776 079/300 Write block 4511 len 20480 data: 5 3 3 5 4
10777 080/300 Read block 4042 len 24576
10778 081/300 Write block 810 len 40960 data: 3 3 3 5 3 3 3 3 3 3
10779 082/300 Read block 4279 len 20480
10780 083/300 Read block 741 len 20480
10781 084/300 Read block 1722 len 16384
10782 085/300 Write block 457 len 16384 data: 3 3 3 3
10783 086/300 Flush
10784 087/300 Read block 1727 len 28672
10785 088/300 Read block 2274 len 12288
10786 089/300 Write block 1242 len 4096 data: 3
10787 090/300 Read block 3206 len 20480
10788 091/300 Flush
10789 092/300 Write block 3100 len 4096 data: 3
10790 093/300 Write block 2813 len 16384 data: 2 2 3 3
10791 094/300 Write block 1454 len 24576 data: 3 3 3 4 4 4
10792 095/300 Write block 3774 len 40960 data: 3 4 3 3 3 3 3 3 2 2
10793 096/300 Read block 2574 len 32768
10794 097/300 Flush
10795 098/300 Read block 4899 len 8192
10796 099/300 Read block 2744 len 28672
10797 100/300 Read block 713 len 28672
10798 101/300 Write block 1057 len 28672 data: 5 4 3 3 3 2 2
10799 102/300 Flush
10800 103/300 Read block 225 len 20480
10801 104/300 Read block 787 len 20480
10802 105/300 Write block 274 len 40960 data: 4 4 3 3 3 3 3 3 3 2
10803 106/300 Read block 2401 len 28672
10804 107/300 Read block 987 len 16384
10805 108/300 Read block 4353 len 8192
10806 109/300 Read block 1725 len 36864
10807 110/300 Write block 4949 len 40960 data: 4 4 3 2 3 3 4 4 4 4
10808 111/300 Write block 3298 len 32768 data: 2 2 3 4 4 5 4 4
10809 112/300 Write block 4908 len 36864 data: 6 4 3 3 3 3 3 3 3
10810 113/300 Flush
10811 114/300 Read block 2844 len 12288
10812 115/300 Read block 2679 len 24576
10813 116/300 Flush
10814 117/300 Flush
10815 118/300 Read block 3780 len 4096
10816 119/300 Read block 1056 len 20480
10817 120/300 Read block 670 len 12288
10818 121/300 Read block 3877 len 28672
10819 122/300 Write block 1947 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10820 123/300 Read block 4505 len 24576
10821 124/300 Write block 2079 len 36864 data: 4 4 4 4 3 3 4 4 4
10822 125/300 Read block 867 len 24576
10823 126/300 Read block 3969 len 20480
10824 127/300 Read block 4609 len 24576
10825 128/300 Read block 2532 len 40960
10826 129/300 Flush
10827 130/300 Write block 255 len 32768 data: 2 3 3 3 3 3 3 3
10828 131/300 Flush
10829 132/300 Read block 1606 len 12288
10830 133/300 Read block 4841 len 8192
10831 134/300 Read block 779 len 24576
10832 135/300 Write block 4276 len 16384 data: 4 3 2 2
10833 136/300 Flush
10834 137/300 Read block 2797 len 12288
10835 138/300 Read block 1837 len 20480
10836 139/300 Write block 4217 len 8192 data: 3 3
10837 140/300 Write block 1777 len 16384 data: 2 2 2 2
10838 141/300 Read block 4119 len 8192
10839 142/300 Flush
10840 143/300 Write block 4236 len 24576 data: 2 2 2 2 2 2
10841 144/300 Read block 2418 len 28672
10842 145/300 Read block 181 len 12288
10843 146/300 Read block 6 len 24576
10844 147/300 Write block 2971 len 36864 data: 3 3 3 3 3 3 4 5 5
10845 148/300 Read block 2130 len 24576
10846 149/300 Write block 3106 len 36864 data: 4 4 4 4 4 4 3 3 3
10847 150/300 Write block 1860 len 40960 data: 3 3 2 3 3 3 3 3 3 2
10848 151/300 Write block 1246 len 24576 data: 2 3 3 3 2 3
10849 152/300 Read block 188 len 28672
10850 153/300 Flush
10851 154/300 Flush
10852 155/300 Read block 4712 len 32768
10853 156/300 Write block 3905 len 16384 data: 3 3 2 2
10854 157/300 Write block 3681 len 40960 data: 4 4 3 3 3 3 3 3 3 3
10855 158/300 Write block 4325 len 8192 data: 3 3
10856 159/300 Write block 1466 len 20480 data: 2 2 2 2 2
10857 160/300 Read block 3069 len 8192
10858 161/300 Read block 146 len 12288
10859 162/300 Write block 648 len 32768 data: 3 3 2 2 2 2 2 2
10860 163/300 Read block 4311 len 32768
10861 164/300 Write block 3304 len 20480 data: 5 5 4 4 4
10862 165/300 Write block 4987 len 4096 data: 3
10863 166/300 Write block 3826 len 16384 data: 2 2 2 2
10864 167/300 Write block 4552 len 32768 data: 3 3 3 3 3 2 2 2
10865 168/300 Write block 1187 len 12288 data: 3 3 3
10866 169/300 Write block 3746 len 36864 data: 4 4 4 5 5 5 4 4 4
10867 170/300 Write block 3778 len 4096 data: 4
10868 171/300 Write block 309 len 12288 data: 3 3 5
10869 172/300 Write block 2045 len 12288 data: 2 2 2
10870 173/300 Read block 3789 len 8192
10871 174/300 Flush
10872 175/300 Flush
10873 176/300 Read block 1191 len 4096
10874 177/300 Read block 3792 len 12288
10875 178/300 Write block 3261 len 36864 data: 5 5 5 3 3 3 3 2 3
10876 179/300 Write block 1823 len 16384 data: 3 3 3 3
10877 180/300 Read block 47 len 16384
10878 181/300 Read block 4703 len 4096
10879 182/300 Write block 987 len 16384 data: 3 3 3 3
10880 183/300 Write block 4462 len 28672 data: 5 5 5 5 5 4 2
10881 184/300 Read block 640 len 8192
10882 185/300 Write block 4751 len 24576 data: 4 3 2 3 3 3
10883 186/300 Write block 3470 len 12288 data: 2 3 2
10884 187/300 Read block 2248 len 40960
10885 188/300 Read block 476 len 24576
10886 189/300 Flush
10887 190/300 Write block 1829 len 24576 data: 2 2 3 3 3 4
10888 191/300 Read block 489 len 36864
10889 192/300 Write block 3334 len 4096 data: 5
10890 193/300 Write block 541 len 16384 data: 3 3 3 2
10891 194/300 Read block 1540 len 40960
10892 195/300 Read block 2774 len 12288
10893 196/300 Write block 4691 len 24576 data: 2 2 2 2 2 2
10894 197/300 Read block 1292 len 4096
10895 198/300 Read block 2894 len 40960
10896 199/300 Write block 2156 len 4096 data: 4
10897 200/300 Read block 2783 len 28672
10898 201/300 Write block 1859 len 8192 data: 3 4
10899 202/300 Write block 1919 len 28672 data: 4 2 3 3 3 3 2
10900 203/300 Read block 3272 len 40960
10901 204/300 Flush
10902 205/300 Read block 3878 len 40960
10903 206/300 Read block 2165 len 32768
10904 207/300 Read block 3630 len 24576
10905 208/300 Flush
10906 209/300 Write block 1213 len 24576 data: 4 3 3 3 3 4
10907 210/300 Read block 1553 len 36864
10908 211/300 Flush
10909 212/300 Write block 3692 len 32768 data: 2 2 2 3 4 3 2 2
10910 213/300 Write block 1326 len 16384 data: 3 3 2 3
10911 214/300 Read block 3591 len 28672
10912 215/300 Read block 2061 len 36864
10913 216/300 Write block 1213 len 8192 data: 5 4
10914 217/300 Read block 2093 len 20480
10915 218/300 Flush
10916 219/300 Write block 4511 len 36864 data: 6 4 4 6 5 4 4 4 4
10917 220/300 Read block 4109 len 12288
10918 221/300 Write block 4182 len 36864 data: 2 2 2 2 2 2 2 3 3
10919 222/300 Read block 188 len 4096
10920 223/300 Write block 2744 len 32768 data: 3 3 2 2 2 2 2 2
10921 224/300 Read block 4299 len 40960
10922 225/300 Write block 4986 len 20480 data: 2 4 3 3 4
10923 226/300 Write block 3720 len 12288 data: 5 4 4
10924 227/300 Read block 649 len 8192
10925 228/300 Read block 619 len 36864
10926 229/300 Write block 3052 len 20480 data: 6 4 4 3 2
10927 230/300 Write block 3538 len 24576 data: 2 2 3 3 4 3
10928 231/300 Read block 1778 len 20480
10929 232/300 Write block 3649 len 16384 data: 2 2 3 2
10930 233/300 Read block 4993 len 8192
10931 234/300 Write block 1121 len 8192 data: 4 4
10932 235/300 Read block 2804 len 28672
10933 236/300 Write block 1332 len 12288 data: 3 3 3
10934 237/300 Read block 2498 len 32768
10935 238/300 Write block 1501 len 12288 data: 3 3 4
10936 239/300 Read block 493 len 12288
10937 240/300 Read block 22 len 20480
10938 241/300 Write block 770 len 8192 data: 2 2
10939 242/300 Write block 1829 len 28672 data: 3 3 4 4 4 5 4
10940 243/300 Write block 516 len 32768 data: 2 2 2 2 2 2 2 3
10941 244/300 Write block 1866 len 4096 data: 4
10942 245/300 Write block 597 len 20480 data: 2 2 2 2 2
10943 246/300 Write block 1939 len 28672 data: 2 2 2 2 2 2 2
10944 247/300 Flush
10945 248/300 Write block 1814 len 12288 data: 4 3 3
10946 249/300 Read block 3857 len 8192
10947 250/300 Read block 1628 len 16384
10948 251/300 Write block 1013 len 40960 data: 2 2 2 2 3 3 4 4 4 4
10949 252/300 Flush
10950 253/300 Write block 3947 len 40960 data: 4 4 3 3 2 2 2 2 2 2
10951 254/300 Read block 2062 len 24576
10952 255/300 Write block 210 len 24576 data: 2 2 2 2 3 3
10953 256/300 Write block 4720 len 32768 data: 2 2 2 2 3 3 3 3
10954 257/300 Read block 3904 len 16384
10955 258/300 Write block 1169 len 8192 data: 3 2
10956 259/300 Write block 1433 len 40960 data: 2 2 2 2 2 3 3 3 2 2
10957 260/300 Write block 334 len 28672 data: 5 6 5 5 4 4 4
10958 261/300 Read block 417 len 32768
10959 262/300 Read block 4962 len 20480
10960 263/300 Flush
10961 264/300 Read block 4752 len 40960
10962 265/300 Flush
10963 266/300 Write block 3162 len 20480 data: 2 2 3 3 3
10964 267/300 Flush
10965 268/300 Write block 1585 len 36864 data: 4 4 3 3 3 2 2 2 2
10966 269/300 Write block 1706 len 36864 data: 3 2 2 3 3 3 3 4 3
10967 270/300 Read block 2927 len 12288
10968 271/300 Read block 2550 len 36864
10969 272/300 Flush
10970 273/300 Write block 1288 len 20480 data: 3 2 2 2 3
10971 274/300 Read block 3614 len 36864
10972 275/300 Read block 2785 len 16384
10973 276/300 Read block 1208 len 40960
10974 277/300 Read block 4864 len 24576
10975 278/300 Read block 3780 len 12288
10976 279/300 Flush
10977 280/300 Read block 1029 len 36864
10978 281/300 Write block 698 len 4096 data: 3
10979 282/300 Write block 252 len 24576 data: 2 2 2 3 4 4
10980 283/300 Write block 2617 len 4096 data: 4
10981 284/300 Read block 2001 len 40960
10982 285/300 Write block 1523 len 28672 data: 4 5 5 5 3 3 3
10983 286/300 Flush
10984 287/300 Read block 1744 len 32768
10985 288/300 Read block 2911 len 16384
10986 289/300 Write block 3087 len 8192 data: 2 3
10987 290/300 Write block 1897 len 24576 data: 3 3 3 3 2 2
10988 291/300 Read block 3300 len 4096
10989 292/300 Write block 2002 len 32768 data: 2 2 3 3 3 3 3 3
10990 293/300 Write block 472 len 28672 data: 2 2 2 2 2 2 2
10991 294/300 Read block 2085 len 32768
10992 295/300 Read block 3359 len 40960
10993 296/300 Read block 1730 len 16384
10994 297/300 Write block 3298 len 40960 data: 3 3 4 5 5 6 6 6 5 5
10995 298/300 Read block 2610 len 24576
10996 299/300 Read block 4264 len 8192
10997 300/300 Read block 356 len 36864
10998 301/300 Write block 298 len 24576 data: 2 3 3 3 3 3
10999 Main task has work to do, go find it
11000 got dsc Start(0)
11001 start 0
11002 Feb 07 20:09:47.732 INFO request completed, latency_us: 276, response_code: 204, uri: /start/cid/0, method: POST, req_id: 28a9cde6-450b-4b99-9cb0-4e27e683ab7d, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
11003 Replay: started 0, returned:Ok(())
11004 ----------------------------------------------------------------
11005 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11006 States: Offline Active Active
11007 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11008 3416 Acked Flush 0 Sent Done Done false
11009 3417 Acked Read 1 Sent Done Done false
11010 3418 Acked Write 2 Sent Done Done false
11011 3419 Acked Read 3 Sent Done Done false
11012 3420 Acked Write 2 Sent Done Done false
11013 3421 Acked Write 7 Sent Done Done false
11014 3422 Acked Read 4 Sent Done Done false
11015 3423 Acked Read 4 Sent Done Done false
11016 3424 Acked Write 8 Sent Done Done false
11017 3425 Acked Flush 0 Sent Done Done false
11018 3426 Acked Write 10 Sent Done Done false
11019 3427 Acked Write 6 Sent Done Done false
11020 3428 Acked Read 3 Sent Done Done false
11021 3429 Acked Write 7 Sent Done Done false
11022 3430 Acked Write 8 Sent Done Done false
11023 3431 Acked Write 4 Sent Done Done false
11024 3432 Acked Write 5 Sent Done Done false
11025 3433 Acked Read 3 Sent Done Done false
11026 3434 Acked Read 2 Sent Done Done false
11027 3435 Acked Write 5 Sent Done Done false
11028 3436 Acked Write 9 Sent Done Done false
11029 3437 Acked Write 10 Sent Done Done false
11030 3438 Acked Read 10 Sent Done Done false
11031 3439 Acked Write 6 Sent Done Done false
11032 3440 Acked Read 10 Sent Done Done false
11033 3441 Acked Write 1 Sent Done Done false
11034 3442 Acked Read 1 Sent Done Done false
11035 3443 Acked Write 1 Sent Done Done false
11036 3444 Acked Write 2 Sent Done Done false
11037 3445 Acked Write 4 Sent Done Done false
11038 3446 Acked Write 5 Sent Done Done false
11039 3447 Acked Read 5 Sent Done Done false
11040 3448 Acked Write 7 Sent Done Done false
11041 3449 Acked Read 1 Sent Done Done false
11042 3450 Acked Write 8 Sent Done Done false
11043 3451 Acked Read 6 Sent Done Done false
11044 3452 Acked Read 5 Sent Done Done false
11045 3453 Acked Flush 0 Sent Done Done false
11046 3454 Acked Flush 0 Sent Done Done false
11047 3455 Acked Read 8 Sent Done Done false
11048 3456 Acked Write 1 Sent Done Done false
11049 3457 Acked Read 4 Sent Done Done false
11050 3458 Acked Write 2 Sent Done Done false
11051 3459 Acked Write 3 Sent Done Done false
11052 3460 Acked Write 7 Sent Done Done false
11053 3461 Acked Read 6 Sent Done Done false
11054 3462 Acked Read 6 Sent Done Done false
11055 3463 Acked Read 9 Sent Done Done false
11056 3464 Acked Read 7 Sent Done Done false
11057 3465 Acked Read 3 Sent Done Done false
11058 3466 Acked Write 8 Sent Done Done false
11059 3467 Acked Read 2 Sent Done Done false
11060 3468 Acked Read 9 Sent Done Done false
11061 3469 Acked Read 8 Sent Done Done false
11062 3470 Acked Read 8 Sent Done Done false
11063 3471 Acked Write 1 Sent Done Done false
11064 3472 Acked Read 9 Sent Done Done false
11065 3473 Acked Write 1 Sent Done Done false
11066 3474 Acked Write 1 Sent Done Done false
11067 3475 Acked Write 9 Sent Done Done false
11068 3476 Acked Read 9 Sent Done Done false
11069 3477 Acked Write 7 Sent Done Done false
11070 3478 Acked Read 1 Sent Done Done false
11071 3479 Acked Write 1 Sent Done Done false
11072 3480 Acked Read 3 Sent Done Done false
11073 3481 Acked Read 5 Sent Done Done false
11074 3482 Acked Read 9 Sent Done Done false
11075 3483 Acked Write 4 Sent Done Done false
11076 3484 Acked Read 1 Sent Done Done false
11077 3485 Acked Read 7 Sent Done Done false
11078 3486 Acked Read 4 Sent Done Done false
11079 3487 Acked Read 1 Sent Done Done false
11080 3488 Acked Read 2 Sent Done Done false
11081 3489 Acked Flush 0 Sent Done Done false
11082 3490 Acked Read 3 Sent Done Done false
11083 3491 Acked Read 1 Sent Done Done false
11084 3492 Acked Write 10 Sent Done Done false
11085 3493 Acked Flush 0 Sent Done Done false
11086 3494 Acked Write 5 Sent Done Done false
11087 3495 Acked Read 6 Sent Done Done false
11088 3496 Acked Write 10 Sent Done Done false
11089 3497 Acked Read 5 Sent Done Done false
11090 3498 Acked Read 5 Sent Done Done false
11091 3499 Acked Read 4 Sent Done Done false
11092 3500 Acked Write 4 Sent Done Done false
11093 3501 Acked Flush 0 Sent Done Done false
11094 3502 Acked Read 7 Sent Done Done false
11095 3503 Acked Read 3 Sent Done Done false
11096 3504 Acked Write 1 Sent Done Done false
11097 3505 Acked Read 5 Sent Done Done false
11098 3506 Acked Flush 0 Sent Done Done false
11099 3507 Acked Write 1 Sent Done Done false
11100 3508 Acked Write 4 Sent Done Done false
11101 3509 Acked Write 6 Sent Done Done false
11102 3510 Acked Write 10 Sent Done Done false
11103 3511 Acked Read 8 Sent Done Done false
11104 3512 Acked Flush 0 Sent Done Done false
11105 3513 Acked Read 2 Sent Done Done false
11106 3514 Acked Read 7 Sent Done Done false
11107 3515 Acked Read 7 Sent Done Done false
11108 3516 Acked Write 7 Sent Done Done false
11109 3517 Acked Flush 0 Sent Done Done false
11110 3518 Acked Read 5 Sent Done Done false
11111 3519 Acked Read 5 Sent Done Done false
11112 3520 Acked Write 10 Sent Done Done false
11113 3521 Acked Read 7 Sent Done Done false
11114 3522 Acked Read 4 Sent Done Done false
11115 3523 Acked Read 2 Sent Done Done false
11116 3524 Acked Read 9 Sent Done Done false
11117 3525 Acked Write 10 Sent Done Done false
11118 3526 Acked Write 8 Sent Done Done false
11119 3527 Acked Write 9 Sent Done Done false
11120 3528 Acked Flush 0 Sent Done Done false
11121 3529 Acked Read 3 Sent Done Done false
11122 3530 Acked Read 6 Sent Done Done false
11123 3531 Acked Flush 0 Sent Done Done false
11124 3532 Acked Flush 0 Sent Done Done false
11125 3533 Acked Read 1 Sent Done Done false
11126 3534 Acked Read 5 Sent Done Done false
11127 3535 Acked Read 3 Sent Done Done false
11128 3536 Acked Read 7 Sent Done Done false
11129 3537 Acked Write 10 Sent Done Done false
11130 3538 Acked Read 6 Sent Done Done false
11131 3539 Acked Write 9 Sent Done Done false
11132 3540 Acked Read 6 Sent Done Done false
11133 3541 Acked Read 5 Sent Done Done false
11134 3542 Acked Read 6 Sent Done Done false
11135 3543 Acked Read 10 Sent Done Done false
11136 3544 Acked Flush 0 Sent Done Done false
11137 3545 Acked Write 8 Sent Done Done false
11138 3546 Acked Flush 0 Sent Done Done false
11139 3547 Acked Read 3 Sent Done Done false
11140 3548 Acked Read 2 Sent Done Done false
11141 3549 Acked Read 6 Sent Done Done false
11142 3550 Acked Write 4 Sent Done Done false
11143 3551 Acked Flush 0 Sent Done Done false
11144 3552 Acked Read 3 Sent Done Done false
11145 3553 Acked Read 5 Sent Done Done false
11146 3554 Acked Write 2 Sent Done Done false
11147 3555 Acked Write 4 Sent Done Done false
11148 3556 Acked Read 2 Sent Done Done false
11149 3557 Acked Flush 0 Sent Done Done false
11150 3558 Acked Write 6 Sent Done Done false
11151 3559 Acked Read 7 Sent Done Done false
11152 3560 Acked Read 3 Sent Done Done false
11153 3561 Acked Read 6 Sent Done Done false
11154 3562 Acked Write 9 Sent Done Done false
11155 3563 Acked Read 6 Sent Done Done false
11156 3564 Acked Write 9 Sent Done Done false
11157 3565 Acked Write 10 Sent Done Done false
11158 3566 Acked Write 6 Sent Done Done false
11159 3567 Acked Read 7 Sent Done Done false
11160 3568 Acked Flush 0 Sent Done Done false
11161 3569 Acked Flush 0 Sent Done Done false
11162 3570 Acked Read 8 Sent Done Done false
11163 3571 Acked Write 4 Sent Done Done false
11164 3572 Acked Write 10 Sent Done Done false
11165 3573 Acked Write 2 Sent Done Done false
11166 3574 Acked Write 5 Sent Done Done false
11167 3575 Acked Read 2 Sent Done Done false
11168 3576 Acked Read 3 Sent Done Done false
11169 3577 Acked Write 8 Sent Done Done false
11170 3578 Acked Read 8 Sent Done Done false
11171 3579 Acked Write 5 Sent Done Done false
11172 3580 Acked Write 1 Sent Done Done false
11173 3581 Acked Write 4 Sent Done Done false
11174 3582 Acked Write 8 Sent Done Done false
11175 3583 Acked Write 3 Sent Done Done false
11176 3584 Acked Write 9 Sent Done Done false
11177 3585 Acked Write 1 Sent Done Done false
11178 3586 Acked Write 3 Sent Done Done false
11179 3587 Acked Write 3 Sent Done Done false
11180 3588 Acked Read 2 Sent Done Done false
11181 3589 Acked Flush 0 Sent Done Done false
11182 3590 Acked Flush 0 Sent Done Done false
11183 3591 Acked Read 1 Sent Done Done false
11184 3592 Acked Read 3 Sent Done Done false
11185 3593 Acked Write 9 Sent Done Done false
11186 3594 Acked Write 4 Sent Done Done false
11187 3595 Acked Read 4 Sent Done Done false
11188 3596 Acked Read 1 Sent Done Done false
11189 3597 Acked Write 4 Sent Done Done false
11190 3598 Acked Write 7 Sent Done Done false
11191 3599 Acked Read 2 Sent Done Done false
11192 3600 Acked Write 6 Sent Done Done false
11193 3601 Acked Write 3 Sent Done Done false
11194 3602 Acked Read 10 Sent Done Done false
11195 3603 Acked Read 6 Sent Done Done false
11196 3604 Acked Flush 0 Sent Done Done false
11197 3605 Acked Write 6 Sent Done Done false
11198 3606 Acked Read 9 Sent Done Done false
11199 3607 Acked Write 1 Sent Done Done false
11200 3608 Acked Write 4 Sent Done Done false
11201 3609 Acked Read 10 Sent Done Done false
11202 3610 Acked Read 3 Sent Done Done false
11203 3611 Acked Write 6 Sent Done Done false
11204 3612 Acked Read 1 Sent Done Done false
11205 3613 Acked Read 10 Sent Done Done false
11206 3614 Acked Write 1 Sent Done Done false
11207 3615 Acked Read 7 Sent Done Done false
11208 3616 Acked Write 2 Sent Done Done false
11209 3617 Acked Write 7 Sent Done Done false
11210 3618 Acked Read 10 Sent Done Done false
11211 3619 Acked Flush 0 Sent Done Done false
11212 3620 Acked Read 10 Sent Done Done false
11213 3621 Acked Read 8 Sent Done Done false
11214 3622 Acked Read 6 Sent Done Done false
11215 3623 Acked Flush 0 Sent Done Done false
11216 3624 Acked Write 6 Sent Done Done false
11217 3625 Acked Read 9 Sent Done Done false
11218 3626 Acked Flush 0 Sent Done Done false
11219 3627 Acked Write 8 Sent Done Done false
11220 3628 Acked Write 4 Sent Done Done false
11221 3629 Acked Read 7 Sent Done Done false
11222 3630 Acked Read 9 Sent Done Done false
11223 3631 Acked Write 2 Sent Done Done false
11224 3632 Acked Read 5 Sent Done Done false
11225 3633 Acked Flush 0 Sent Done Done false
11226 3634 Acked Write 9 Sent Done Done false
11227 3635 Acked Read 3 Sent Done Done false
11228 3636 Acked Write 9 Sent Done Done false
11229 3637 Acked Read 1 Sent Done Done false
11230 3638 Acked Write 8 Sent Done Done false
11231 3639 Acked Read 10 Sent Done Done false
11232 3640 Acked Write 5 Sent Done Done false
11233 3641 Acked Write 3 Sent Done Done false
11234 3642 Acked Read 2 Sent Done Done false
11235 3643 Acked Read 9 Sent Done Done false
11236 3644 Acked Write 5 Sent Done Done false
11237 3645 Acked Write 6 Sent Done Done false
11238 3646 Acked Read 5 Sent Done Done false
11239 3647 Acked Write 4 Sent Done Done false
11240 3648 Acked Read 2 Sent Done Done false
11241 3649 Acked Write 2 Sent Done Done false
11242 3650 Acked Read 7 Sent Done Done false
11243 3651 Acked Write 3 Sent Done Done false
11244 3652 Acked Read 8 Sent Done Done false
11245 3653 Acked Write 3 Sent Done Done false
11246 3654 Acked Read 3 Sent Done Done false
11247 3655 Acked Read 5 Sent Done Done false
11248 3656 Acked Write 2 Sent Done Done false
11249 3657 Acked Write 7 Sent Done Done false
11250 3658 Acked Write 8 Sent Done Done false
11251 3659 Acked Write 1 Sent Done Done false
11252 3660 Acked Write 5 Sent Done Done false
11253 3661 Acked Write 7 Sent Done Done false
11254 3662 Acked Flush 0 Sent Done Done false
11255 3663 Acked Write 3 Sent Done Done false
11256 3664 Acked Read 2 Sent Done Done false
11257 3665 Acked Read 4 Sent Done Done false
11258 3666 Acked Write 10 Sent Done Done false
11259 3667 Acked Flush 0 Sent Done Done false
11260 3668 Acked Write 10 Sent Done Done false
11261 3669 Acked Read 6 Sent Done Done false
11262 3670 Acked Write 6 Sent Done Done false
11263 3671 Acked Write 8 Sent Done Done false
11264 3672 Acked Read 4 Sent Done Done false
11265 3673 Acked Write 2 Sent Done Done false
11266 3674 Acked Write 10 Sent Done Done false
11267 3675 Acked Write 7 Sent Done Done false
11268 3676 Acked Read 8 Sent Done Done false
11269 3677 Acked Read 5 Sent Done Done false
11270 3678 Acked Flush 0 Sent Done Done false
11271 3679 Acked Read 10 Sent Done Done false
11272 3680 Acked Flush 0 Sent Done Done false
11273 3681 Acked Write 5 Sent Done Done false
11274 3682 Acked Flush 0 Sent Done Done false
11275 3683 Acked Write 9 Sent Done Done false
11276 3684 Acked Write 9 Sent Done Done false
11277 3685 Acked Read 3 Sent Done Done false
11278 3686 Acked Read 9 Sent Done Done false
11279 3687 Acked Flush 0 Sent Done Done false
11280 3688 Acked Write 5 Sent Done Done false
11281 3689 Acked Read 9 Sent Done Done false
11282 3690 Acked Read 4 Sent Done Done false
11283 3691 Acked Read 10 Sent Done Done false
11284 3692 Acked Read 6 Sent Done Done false
11285 3693 Acked Read 3 Sent Done Done false
11286 3694 Acked Flush 0 Sent Done Done false
11287 3695 Acked Read 9 Sent Done Done false
11288 3696 Acked Write 1 Sent Done Done false
11289 3697 Acked Write 6 Sent Done Done false
11290 3698 Acked Write 1 Sent Done Done false
11291 3699 Acked Read 10 Sent Done Done false
11292 3700 Acked Write 7 Sent Done Done false
11293 3701 Acked Flush 0 Sent Done Done false
11294 3702 Acked Read 8 Sent Done Done false
11295 3703 Acked Read 4 Sent Done Done false
11296 3704 Acked Write 2 Sent Done Done false
11297 3705 Acked Write 6 Sent Done Done false
11298 3706 Acked Read 1 Sent Done Done false
11299 3707 Acked Write 8 Sent Done Done false
11300 3708 Acked Write 7 Sent Done Done false
11301 3709 Acked Read 8 Sent Done Done false
11302 3710 Acked Read 10 Sent Done Done false
11303 3711 Acked Read 4 Sent Done Done false
11304 3712 Acked Write 10 Sent Done Done false
11305 3713 Acked Read 6 Sent Done Done false
11306 3714 Acked Read 2 Sent Done Done false
11307 3715 Acked Read 9 Sent Done Done false
11308 3716 Acked Write 6 Sent Done Done false
11309 STATES DS:0 DS:1 DS:2 TOTAL
11310 Sent 301 0 0 301
11311 Done 0 301 301 602
11312 Skipped 0 0 0 0
11313 Error 0 0 0 0
11314 Last Flush: Some(JobId(3415)) Some(JobId(3701)) Some(JobId(3701))
11315 Downstairs last five completed: 3415 3414 3413 3412 3411
11316 Upstairs last five completed: 3716 3715 3714 3713 3712
11317 CLIENT: Up:0 ds:301 act:2
113182025-02-07T20:09:48.234ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
113192025-02-07T20:09:49.184ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
113202025-02-07T20:09:50.186ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
113212025-02-07T20:09:51.188ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11322 ----------------------------------------------------------------
11323 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11324 States: Offline Active Active
11325 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11326 3416 Acked Flush 0 Sent Done Done false
11327 3417 Acked Read 1 Sent Done Done false
11328 3418 Acked Write 2 Sent Done Done false
11329 3419 Acked Read 3 Sent Done Done false
11330 3420 Acked Write 2 Sent Done Done false
11331 3421 Acked Write 7 Sent Done Done false
11332 3422 Acked Read 4 Sent Done Done false
11333 3423 Acked Read 4 Sent Done Done false
11334 3424 Acked Write 8 Sent Done Done false
11335 3425 Acked Flush 0 Sent Done Done false
11336 3426 Acked Write 10 Sent Done Done false
11337 3427 Acked Write 6 Sent Done Done false
11338 3428 Acked Read 3 Sent Done Done false
11339 3429 Acked Write 7 Sent Done Done false
11340 3430 Acked Write 8 Sent Done Done false
11341 3431 Acked Write 4 Sent Done Done false
11342 3432 Acked Write 5 Sent Done Done false
11343 3433 Acked Read 3 Sent Done Done false
11344 3434 Acked Read 2 Sent Done Done false
11345 3435 Acked Write 5 Sent Done Done false
11346 3436 Acked Write 9 Sent Done Done false
11347 3437 Acked Write 10 Sent Done Done false
11348 3438 Acked Read 10 Sent Done Done false
11349 3439 Acked Write 6 Sent Done Done false
11350 3440 Acked Read 10 Sent Done Done false
11351 3441 Acked Write 1 Sent Done Done false
11352 3442 Acked Read 1 Sent Done Done false
11353 3443 Acked Write 1 Sent Done Done false
11354 3444 Acked Write 2 Sent Done Done false
11355 3445 Acked Write 4 Sent Done Done false
11356 3446 Acked Write 5 Sent Done Done false
11357 3447 Acked Read 5 Sent Done Done false
11358 3448 Acked Write 7 Sent Done Done false
11359 3449 Acked Read 1 Sent Done Done false
11360 3450 Acked Write 8 Sent Done Done false
11361 3451 Acked Read 6 Sent Done Done false
11362 3452 Acked Read 5 Sent Done Done false
11363 3453 Acked Flush 0 Sent Done Done false
11364 3454 Acked Flush 0 Sent Done Done false
11365 3455 Acked Read 8 Sent Done Done false
11366 3456 Acked Write 1 Sent Done Done false
11367 3457 Acked Read 4 Sent Done Done false
11368 3458 Acked Write 2 Sent Done Done false
11369 3459 Acked Write 3 Sent Done Done false
11370 3460 Acked Write 7 Sent Done Done false
11371 3461 Acked Read 6 Sent Done Done false
11372 3462 Acked Read 6 Sent Done Done false
11373 3463 Acked Read 9 Sent Done Done false
11374 3464 Acked Read 7 Sent Done Done false
11375 3465 Acked Read 3 Sent Done Done false
11376 3466 Acked Write 8 Sent Done Done false
11377 3467 Acked Read 2 Sent Done Done false
11378 3468 Acked Read 9 Sent Done Done false
11379 3469 Acked Read 8 Sent Done Done false
11380 3470 Acked Read 8 Sent Done Done false
11381 3471 Acked Write 1 Sent Done Done false
11382 3472 Acked Read 9 Sent Done Done false
11383 3473 Acked Write 1 Sent Done Done false
11384 3474 Acked Write 1 Sent Done Done false
11385 3475 Acked Write 9 Sent Done Done false
11386 3476 Acked Read 9 Sent Done Done false
11387 3477 Acked Write 7 Sent Done Done false
11388 3478 Acked Read 1 Sent Done Done false
11389 3479 Acked Write 1 Sent Done Done false
11390 3480 Acked Read 3 Sent Done Done false
11391 3481 Acked Read 5 Sent Done Done false
11392 3482 Acked Read 9 Sent Done Done false
11393 3483 Acked Write 4 Sent Done Done false
11394 3484 Acked Read 1 Sent Done Done false
11395 3485 Acked Read 7 Sent Done Done false
11396 3486 Acked Read 4 Sent Done Done false
11397 3487 Acked Read 1 Sent Done Done false
11398 3488 Acked Read 2 Sent Done Done false
11399 3489 Acked Flush 0 Sent Done Done false
11400 3490 Acked Read 3 Sent Done Done false
11401 3491 Acked Read 1 Sent Done Done false
11402 3492 Acked Write 10 Sent Done Done false
11403 3493 Acked Flush 0 Sent Done Done false
11404 3494 Acked Write 5 Sent Done Done false
11405 3495 Acked Read 6 Sent Done Done false
11406 3496 Acked Write 10 Sent Done Done false
11407 3497 Acked Read 5 Sent Done Done false
11408 3498 Acked Read 5 Sent Done Done false
11409 3499 Acked Read 4 Sent Done Done false
11410 3500 Acked Write 4 Sent Done Done false
11411 3501 Acked Flush 0 Sent Done Done false
11412 3502 Acked Read 7 Sent Done Done false
11413 3503 Acked Read 3 Sent Done Done false
11414 3504 Acked Write 1 Sent Done Done false
11415 3505 Acked Read 5 Sent Done Done false
11416 3506 Acked Flush 0 Sent Done Done false
11417 3507 Acked Write 1 Sent Done Done false
11418 3508 Acked Write 4 Sent Done Done false
11419 3509 Acked Write 6 Sent Done Done false
11420 3510 Acked Write 10 Sent Done Done false
11421 3511 Acked Read 8 Sent Done Done false
11422 3512 Acked Flush 0 Sent Done Done false
11423 3513 Acked Read 2 Sent Done Done false
11424 3514 Acked Read 7 Sent Done Done false
11425 3515 Acked Read 7 Sent Done Done false
11426 3516 Acked Write 7 Sent Done Done false
11427 3517 Acked Flush 0 Sent Done Done false
11428 3518 Acked Read 5 Sent Done Done false
11429 3519 Acked Read 5 Sent Done Done false
11430 3520 Acked Write 10 Sent Done Done false
11431 3521 Acked Read 7 Sent Done Done false
11432 3522 Acked Read 4 Sent Done Done false
11433 3523 Acked Read 2 Sent Done Done false
11434 3524 Acked Read 9 Sent Done Done false
11435 3525 Acked Write 10 Sent Done Done false
11436 3526 Acked Write 8 Sent Done Done false
11437 3527 Acked Write 9 Sent Done Done false
11438 3528 Acked Flush 0 Sent Done Done false
11439 3529 Acked Read 3 Sent Done Done false
11440 3530 Acked Read 6 Sent Done Done false
11441 3531 Acked Flush 0 Sent Done Done false
11442 3532 Acked Flush 0 Sent Done Done false
11443 3533 Acked Read 1 Sent Done Done false
11444 3534 Acked Read 5 Sent Done Done false
11445 3535 Acked Read 3 Sent Done Done false
11446 3536 Acked Read 7 Sent Done Done false
11447 3537 Acked Write 10 Sent Done Done false
11448 3538 Acked Read 6 Sent Done Done false
11449 3539 Acked Write 9 Sent Done Done false
11450 3540 Acked Read 6 Sent Done Done false
11451 3541 Acked Read 5 Sent Done Done false
11452 3542 Acked Read 6 Sent Done Done false
11453 3543 Acked Read 10 Sent Done Done false
11454 3544 Acked Flush 0 Sent Done Done false
11455 3545 Acked Write 8 Sent Done Done false
11456 3546 Acked Flush 0 Sent Done Done false
11457 3547 Acked Read 3 Sent Done Done false
11458 3548 Acked Read 2 Sent Done Done false
11459 3549 Acked Read 6 Sent Done Done false
11460 3550 Acked Write 4 Sent Done Done false
11461 3551 Acked Flush 0 Sent Done Done false
11462 3552 Acked Read 3 Sent Done Done false
11463 3553 Acked Read 5 Sent Done Done false
11464 3554 Acked Write 2 Sent Done Done false
11465 3555 Acked Write 4 Sent Done Done false
11466 3556 Acked Read 2 Sent Done Done false
11467 3557 Acked Flush 0 Sent Done Done false
11468 3558 Acked Write 6 Sent Done Done false
11469 3559 Acked Read 7 Sent Done Done false
11470 3560 Acked Read 3 Sent Done Done false
11471 3561 Acked Read 6 Sent Done Done false
11472 3562 Acked Write 9 Sent Done Done false
11473 3563 Acked Read 6 Sent Done Done false
11474 3564 Acked Write 9 Sent Done Done false
11475 3565 Acked Write 10 Sent Done Done false
11476 3566 Acked Write 6 Sent Done Done false
11477 3567 Acked Read 7 Sent Done Done false
11478 3568 Acked Flush 0 Sent Done Done false
11479 3569 Acked Flush 0 Sent Done Done false
11480 3570 Acked Read 8 Sent Done Done false
11481 3571 Acked Write 4 Sent Done Done false
11482 3572 Acked Write 10 Sent Done Done false
11483 3573 Acked Write 2 Sent Done Done false
11484 3574 Acked Write 5 Sent Done Done false
11485 3575 Acked Read 2 Sent Done Done false
11486 3576 Acked Read 3 Sent Done Done false
11487 3577 Acked Write 8 Sent Done Done false
11488 3578 Acked Read 8 Sent Done Done false
11489 3579 Acked Write 5 Sent Done Done false
11490 3580 Acked Write 1 Sent Done Done false
11491 3581 Acked Write 4 Sent Done Done false
11492 3582 Acked Write 8 Sent Done Done false
11493 3583 Acked Write 3 Sent Done Done false
11494 3584 Acked Write 9 Sent Done Done false
11495 3585 Acked Write 1 Sent Done Done false
11496 3586 Acked Write 3 Sent Done Done false
11497 3587 Acked Write 3 Sent Done Done false
11498 3588 Acked Read 2 Sent Done Done false
11499 3589 Acked Flush 0 Sent Done Done false
11500 3590 Acked Flush 0 Sent Done Done false
11501 3591 Acked Read 1 Sent Done Done false
11502 3592 Acked Read 3 Sent Done Done false
11503 3593 Acked Write 9 Sent Done Done false
11504 3594 Acked Write 4 Sent Done Done false
11505 3595 Acked Read 4 Sent Done Done false
11506 3596 Acked Read 1 Sent Done Done false
11507 3597 Acked Write 4 Sent Done Done false
11508 3598 Acked Write 7 Sent Done Done false
11509 3599 Acked Read 2 Sent Done Done false
11510 3600 Acked Write 6 Sent Done Done false
11511 3601 Acked Write 3 Sent Done Done false
11512 3602 Acked Read 10 Sent Done Done false
11513 3603 Acked Read 6 Sent Done Done false
11514 3604 Acked Flush 0 Sent Done Done false
11515 3605 Acked Write 6 Sent Done Done false
11516 3606 Acked Read 9 Sent Done Done false
11517 3607 Acked Write 1 Sent Done Done false
11518 3608 Acked Write 4 Sent Done Done false
11519 3609 Acked Read 10 Sent Done Done false
11520 3610 Acked Read 3 Sent Done Done false
11521 3611 Acked Write 6 Sent Done Done false
11522 3612 Acked Read 1 Sent Done Done false
11523 3613 Acked Read 10 Sent Done Done false
11524 3614 Acked Write 1 Sent Done Done false
11525 3615 Acked Read 7 Sent Done Done false
11526 3616 Acked Write 2 Sent Done Done false
11527 3617 Acked Write 7 Sent Done Done false
11528 3618 Acked Read 10 Sent Done Done false
11529 3619 Acked Flush 0 Sent Done Done false
11530 3620 Acked Read 10 Sent Done Done false
11531 3621 Acked Read 8 Sent Done Done false
11532 3622 Acked Read 6 Sent Done Done false
11533 3623 Acked Flush 0 Sent Done Done false
11534 3624 Acked Write 6 Sent Done Done false
11535 3625 Acked Read 9 Sent Done Done false
11536 3626 Acked Flush 0 Sent Done Done false
11537 3627 Acked Write 8 Sent Done Done false
11538 3628 Acked Write 4 Sent Done Done false
11539 3629 Acked Read 7 Sent Done Done false
11540 3630 Acked Read 9 Sent Done Done false
11541 3631 Acked Write 2 Sent Done Done false
11542 3632 Acked Read 5 Sent Done Done false
11543 3633 Acked Flush 0 Sent Done Done false
11544 3634 Acked Write 9 Sent Done Done false
11545 3635 Acked Read 3 Sent Done Done false
11546 3636 Acked Write 9 Sent Done Done false
11547 3637 Acked Read 1 Sent Done Done false
11548 3638 Acked Write 8 Sent Done Done false
11549 3639 Acked Read 10 Sent Done Done false
11550 3640 Acked Write 5 Sent Done Done false
11551 3641 Acked Write 3 Sent Done Done false
11552 3642 Acked Read 2 Sent Done Done false
11553 3643 Acked Read 9 Sent Done Done false
11554 3644 Acked Write 5 Sent Done Done false
11555 3645 Acked Write 6 Sent Done Done false
11556 3646 Acked Read 5 Sent Done Done false
11557 3647 Acked Write 4 Sent Done Done false
11558 3648 Acked Read 2 Sent Done Done false
11559 3649 Acked Write 2 Sent Done Done false
11560 3650 Acked Read 7 Sent Done Done false
11561 3651 Acked Write 3 Sent Done Done false
11562 3652 Acked Read 8 Sent Done Done false
11563 3653 Acked Write 3 Sent Done Done false
11564 3654 Acked Read 3 Sent Done Done false
11565 3655 Acked Read 5 Sent Done Done false
11566 3656 Acked Write 2 Sent Done Done false
11567 3657 Acked Write 7 Sent Done Done false
11568 3658 Acked Write 8 Sent Done Done false
11569 3659 Acked Write 1 Sent Done Done false
11570 3660 Acked Write 5 Sent Done Done false
11571 3661 Acked Write 7 Sent Done Done false
11572 3662 Acked Flush 0 Sent Done Done false
11573 3663 Acked Write 3 Sent Done Done false
11574 3664 Acked Read 2 Sent Done Done false
11575 3665 Acked Read 4 Sent Done Done false
11576 3666 Acked Write 10 Sent Done Done false
11577 3667 Acked Flush 0 Sent Done Done false
11578 3668 Acked Write 10 Sent Done Done false
11579 3669 Acked Read 6 Sent Done Done false
11580 3670 Acked Write 6 Sent Done Done false
11581 3671 Acked Write 8 Sent Done Done false
11582 3672 Acked Read 4 Sent Done Done false
11583 3673 Acked Write 2 Sent Done Done false
11584 3674 Acked Write 10 Sent Done Done false
11585 3675 Acked Write 7 Sent Done Done false
11586 3676 Acked Read 8 Sent Done Done false
11587 3677 Acked Read 5 Sent Done Done false
11588 3678 Acked Flush 0 Sent Done Done false
11589 3679 Acked Read 10 Sent Done Done false
11590 3680 Acked Flush 0 Sent Done Done false
11591 3681 Acked Write 5 Sent Done Done false
11592 3682 Acked Flush 0 Sent Done Done false
11593 3683 Acked Write 9 Sent Done Done false
11594 3684 Acked Write 9 Sent Done Done false
11595 3685 Acked Read 3 Sent Done Done false
11596 3686 Acked Read 9 Sent Done Done false
11597 3687 Acked Flush 0 Sent Done Done false
11598 3688 Acked Write 5 Sent Done Done false
11599 3689 Acked Read 9 Sent Done Done false
11600 3690 Acked Read 4 Sent Done Done false
11601 3691 Acked Read 10 Sent Done Done false
11602 3692 Acked Read 6 Sent Done Done false
11603 3693 Acked Read 3 Sent Done Done false
11604 3694 Acked Flush 0 Sent Done Done false
11605 3695 Acked Read 9 Sent Done Done false
11606 3696 Acked Write 1 Sent Done Done false
11607 3697 Acked Write 6 Sent Done Done false
11608 3698 Acked Write 1 Sent Done Done false
11609 3699 Acked Read 10 Sent Done Done false
11610 3700 Acked Write 7 Sent Done Done false
11611 3701 Acked Flush 0 Sent Done Done false
11612 3702 Acked Read 8 Sent Done Done false
11613 3703 Acked Read 4 Sent Done Done false
11614 3704 Acked Write 2 Sent Done Done false
11615 3705 Acked Write 6 Sent Done Done false
11616 3706 Acked Read 1 Sent Done Done false
11617 3707 Acked Write 8 Sent Done Done false
11618 3708 Acked Write 7 Sent Done Done false
11619 3709 Acked Read 8 Sent Done Done false
11620 3710 Acked Read 10 Sent Done Done false
11621 3711 Acked Read 4 Sent Done Done false
11622 3712 Acked Write 10 Sent Done Done false
11623 3713 Acked Read 6 Sent Done Done false
11624 3714 Acked Read 2 Sent Done Done false
11625 3715 Acked Read 9 Sent Done Done false
11626 3716 Acked Write 6 Sent Done Done false
11627 STATES DS:0 DS:1 DS:2 TOTAL
11628 Sent 301 0 0 301
11629 Done 0 301 301 602
11630 Skipped 0 0 0 0
11631 Error 0 0 0 0
11632 Last Flush: Some(JobId(3415)) Some(JobId(3701)) Some(JobId(3701))
11633 Downstairs last five completed: 3415 3414 3413 3412 3411
11634 Upstairs last five completed: 3716 3715 3714 3713 3712
11635 CLIENT: Up:0 ds:301 act:2
116362025-02-07T20:09:52.236ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
116372025-02-07T20:09:53.192ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
116382025-02-07T20:09:53.496ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
116392025-02-07T20:09:53.496ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
116402025-02-07T20:09:53.496ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
116412025-02-07T20:09:53.496ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
116422025-02-07T20:09:53.496ZINFOcrucible: client task is exiting = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11643 [8810] got start action
11644 [8810] I am going to restart
11645 Starting downstairs at port 8810
11646 Make output file at "/tmp/dsc/downstairs-8810.txt"
11647 [8810][0] reports Starting
11648 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1218)
11649 [8810][0] reports Running
116502025-02-07T20:09:54.194ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
116512025-02-07T20:09:55.196ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11652 ----------------------------------------------------------------
11653 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11654 States: Offline Active Active
11655 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11656 3416 Acked Flush 0 Sent Done Done false
11657 3417 Acked Read 1 Sent Done Done false
11658 3418 Acked Write 2 Sent Done Done false
11659 3419 Acked Read 3 Sent Done Done false
11660 3420 Acked Write 2 Sent Done Done false
11661 3421 Acked Write 7 Sent Done Done false
11662 3422 Acked Read 4 Sent Done Done false
11663 3423 Acked Read 4 Sent Done Done false
11664 3424 Acked Write 8 Sent Done Done false
11665 3425 Acked Flush 0 Sent Done Done false
11666 3426 Acked Write 10 Sent Done Done false
11667 3427 Acked Write 6 Sent Done Done false
11668 3428 Acked Read 3 Sent Done Done false
11669 3429 Acked Write 7 Sent Done Done false
11670 3430 Acked Write 8 Sent Done Done false
11671 3431 Acked Write 4 Sent Done Done false
11672 3432 Acked Write 5 Sent Done Done false
11673 3433 Acked Read 3 Sent Done Done false
11674 3434 Acked Read 2 Sent Done Done false
11675 3435 Acked Write 5 Sent Done Done false
11676 3436 Acked Write 9 Sent Done Done false
11677 3437 Acked Write 10 Sent Done Done false
11678 3438 Acked Read 10 Sent Done Done false
11679 3439 Acked Write 6 Sent Done Done false
11680 3440 Acked Read 10 Sent Done Done false
11681 3441 Acked Write 1 Sent Done Done false
11682 3442 Acked Read 1 Sent Done Done false
11683 3443 Acked Write 1 Sent Done Done false
11684 3444 Acked Write 2 Sent Done Done false
11685 3445 Acked Write 4 Sent Done Done false
11686 3446 Acked Write 5 Sent Done Done false
11687 3447 Acked Read 5 Sent Done Done false
11688 3448 Acked Write 7 Sent Done Done false
11689 3449 Acked Read 1 Sent Done Done false
11690 3450 Acked Write 8 Sent Done Done false
11691 3451 Acked Read 6 Sent Done Done false
11692 3452 Acked Read 5 Sent Done Done false
11693 3453 Acked Flush 0 Sent Done Done false
11694 3454 Acked Flush 0 Sent Done Done false
11695 3455 Acked Read 8 Sent Done Done false
11696 3456 Acked Write 1 Sent Done Done false
11697 3457 Acked Read 4 Sent Done Done false
11698 3458 Acked Write 2 Sent Done Done false
11699 3459 Acked Write 3 Sent Done Done false
11700 3460 Acked Write 7 Sent Done Done false
11701 3461 Acked Read 6 Sent Done Done false
11702 3462 Acked Read 6 Sent Done Done false
11703 3463 Acked Read 9 Sent Done Done false
11704 3464 Acked Read 7 Sent Done Done false
11705 3465 Acked Read 3 Sent Done Done false
11706 3466 Acked Write 8 Sent Done Done false
11707 3467 Acked Read 2 Sent Done Done false
11708 3468 Acked Read 9 Sent Done Done false
11709 3469 Acked Read 8 Sent Done Done false
11710 3470 Acked Read 8 Sent Done Done false
11711 3471 Acked Write 1 Sent Done Done false
11712 3472 Acked Read 9 Sent Done Done false
11713 3473 Acked Write 1 Sent Done Done false
11714 3474 Acked Write 1 Sent Done Done false
11715 3475 Acked Write 9 Sent Done Done false
11716 3476 Acked Read 9 Sent Done Done false
11717 3477 Acked Write 7 Sent Done Done false
11718 3478 Acked Read 1 Sent Done Done false
11719 3479 Acked Write 1 Sent Done Done false
11720 3480 Acked Read 3 Sent Done Done false
11721 3481 Acked Read 5 Sent Done Done false
11722 3482 Acked Read 9 Sent Done Done false
11723 3483 Acked Write 4 Sent Done Done false
11724 3484 Acked Read 1 Sent Done Done false
11725 3485 Acked Read 7 Sent Done Done false
11726 3486 Acked Read 4 Sent Done Done false
11727 3487 Acked Read 1 Sent Done Done false
11728 3488 Acked Read 2 Sent Done Done false
11729 3489 Acked Flush 0 Sent Done Done false
11730 3490 Acked Read 3 Sent Done Done false
11731 3491 Acked Read 1 Sent Done Done false
11732 3492 Acked Write 10 Sent Done Done false
11733 3493 Acked Flush 0 Sent Done Done false
11734 3494 Acked Write 5 Sent Done Done false
11735 3495 Acked Read 6 Sent Done Done false
11736 3496 Acked Write 10 Sent Done Done false
11737 3497 Acked Read 5 Sent Done Done false
11738 3498 Acked Read 5 Sent Done Done false
11739 3499 Acked Read 4 Sent Done Done false
11740 3500 Acked Write 4 Sent Done Done false
11741 3501 Acked Flush 0 Sent Done Done false
11742 3502 Acked Read 7 Sent Done Done false
11743 3503 Acked Read 3 Sent Done Done false
11744 3504 Acked Write 1 Sent Done Done false
11745 3505 Acked Read 5 Sent Done Done false
11746 3506 Acked Flush 0 Sent Done Done false
11747 3507 Acked Write 1 Sent Done Done false
11748 3508 Acked Write 4 Sent Done Done false
11749 3509 Acked Write 6 Sent Done Done false
11750 3510 Acked Write 10 Sent Done Done false
11751 3511 Acked Read 8 Sent Done Done false
11752 3512 Acked Flush 0 Sent Done Done false
11753 3513 Acked Read 2 Sent Done Done false
11754 3514 Acked Read 7 Sent Done Done false
11755 3515 Acked Read 7 Sent Done Done false
11756 3516 Acked Write 7 Sent Done Done false
11757 3517 Acked Flush 0 Sent Done Done false
11758 3518 Acked Read 5 Sent Done Done false
11759 3519 Acked Read 5 Sent Done Done false
11760 3520 Acked Write 10 Sent Done Done false
11761 3521 Acked Read 7 Sent Done Done false
11762 3522 Acked Read 4 Sent Done Done false
11763 3523 Acked Read 2 Sent Done Done false
11764 3524 Acked Read 9 Sent Done Done false
11765 3525 Acked Write 10 Sent Done Done false
11766 3526 Acked Write 8 Sent Done Done false
11767 3527 Acked Write 9 Sent Done Done false
11768 3528 Acked Flush 0 Sent Done Done false
11769 3529 Acked Read 3 Sent Done Done false
11770 3530 Acked Read 6 Sent Done Done false
11771 3531 Acked Flush 0 Sent Done Done false
11772 3532 Acked Flush 0 Sent Done Done false
11773 3533 Acked Read 1 Sent Done Done false
11774 3534 Acked Read 5 Sent Done Done false
11775 3535 Acked Read 3 Sent Done Done false
11776 3536 Acked Read 7 Sent Done Done false
11777 3537 Acked Write 10 Sent Done Done false
11778 3538 Acked Read 6 Sent Done Done false
11779 3539 Acked Write 9 Sent Done Done false
11780 3540 Acked Read 6 Sent Done Done false
11781 3541 Acked Read 5 Sent Done Done false
11782 3542 Acked Read 6 Sent Done Done false
11783 3543 Acked Read 10 Sent Done Done false
11784 3544 Acked Flush 0 Sent Done Done false
11785 3545 Acked Write 8 Sent Done Done false
11786 3546 Acked Flush 0 Sent Done Done false
11787 3547 Acked Read 3 Sent Done Done false
11788 3548 Acked Read 2 Sent Done Done false
11789 3549 Acked Read 6 Sent Done Done false
11790 3550 Acked Write 4 Sent Done Done false
11791 3551 Acked Flush 0 Sent Done Done false
11792 3552 Acked Read 3 Sent Done Done false
11793 3553 Acked Read 5 Sent Done Done false
11794 3554 Acked Write 2 Sent Done Done false
11795 3555 Acked Write 4 Sent Done Done false
11796 3556 Acked Read 2 Sent Done Done false
11797 3557 Acked Flush 0 Sent Done Done false
11798 3558 Acked Write 6 Sent Done Done false
11799 3559 Acked Read 7 Sent Done Done false
11800 3560 Acked Read 3 Sent Done Done false
11801 3561 Acked Read 6 Sent Done Done false
11802 3562 Acked Write 9 Sent Done Done false
11803 3563 Acked Read 6 Sent Done Done false
11804 3564 Acked Write 9 Sent Done Done false
11805 3565 Acked Write 10 Sent Done Done false
11806 3566 Acked Write 6 Sent Done Done false
11807 3567 Acked Read 7 Sent Done Done false
11808 3568 Acked Flush 0 Sent Done Done false
11809 3569 Acked Flush 0 Sent Done Done false
11810 3570 Acked Read 8 Sent Done Done false
11811 3571 Acked Write 4 Sent Done Done false
11812 3572 Acked Write 10 Sent Done Done false
11813 3573 Acked Write 2 Sent Done Done false
11814 3574 Acked Write 5 Sent Done Done false
11815 3575 Acked Read 2 Sent Done Done false
11816 3576 Acked Read 3 Sent Done Done false
11817 3577 Acked Write 8 Sent Done Done false
11818 3578 Acked Read 8 Sent Done Done false
11819 3579 Acked Write 5 Sent Done Done false
11820 3580 Acked Write 1 Sent Done Done false
11821 3581 Acked Write 4 Sent Done Done false
11822 3582 Acked Write 8 Sent Done Done false
11823 3583 Acked Write 3 Sent Done Done false
11824 3584 Acked Write 9 Sent Done Done false
11825 3585 Acked Write 1 Sent Done Done false
11826 3586 Acked Write 3 Sent Done Done false
11827 3587 Acked Write 3 Sent Done Done false
11828 3588 Acked Read 2 Sent Done Done false
11829 3589 Acked Flush 0 Sent Done Done false
11830 3590 Acked Flush 0 Sent Done Done false
11831 3591 Acked Read 1 Sent Done Done false
11832 3592 Acked Read 3 Sent Done Done false
11833 3593 Acked Write 9 Sent Done Done false
11834 3594 Acked Write 4 Sent Done Done false
11835 3595 Acked Read 4 Sent Done Done false
11836 3596 Acked Read 1 Sent Done Done false
11837 3597 Acked Write 4 Sent Done Done false
11838 3598 Acked Write 7 Sent Done Done false
11839 3599 Acked Read 2 Sent Done Done false
11840 3600 Acked Write 6 Sent Done Done false
11841 3601 Acked Write 3 Sent Done Done false
11842 3602 Acked Read 10 Sent Done Done false
11843 3603 Acked Read 6 Sent Done Done false
11844 3604 Acked Flush 0 Sent Done Done false
11845 3605 Acked Write 6 Sent Done Done false
11846 3606 Acked Read 9 Sent Done Done false
11847 3607 Acked Write 1 Sent Done Done false
11848 3608 Acked Write 4 Sent Done Done false
11849 3609 Acked Read 10 Sent Done Done false
11850 3610 Acked Read 3 Sent Done Done false
11851 3611 Acked Write 6 Sent Done Done false
11852 3612 Acked Read 1 Sent Done Done false
11853 3613 Acked Read 10 Sent Done Done false
11854 3614 Acked Write 1 Sent Done Done false
11855 3615 Acked Read 7 Sent Done Done false
11856 3616 Acked Write 2 Sent Done Done false
11857 3617 Acked Write 7 Sent Done Done false
11858 3618 Acked Read 10 Sent Done Done false
11859 3619 Acked Flush 0 Sent Done Done false
11860 3620 Acked Read 10 Sent Done Done false
11861 3621 Acked Read 8 Sent Done Done false
11862 3622 Acked Read 6 Sent Done Done false
11863 3623 Acked Flush 0 Sent Done Done false
11864 3624 Acked Write 6 Sent Done Done false
11865 3625 Acked Read 9 Sent Done Done false
11866 3626 Acked Flush 0 Sent Done Done false
11867 3627 Acked Write 8 Sent Done Done false
11868 3628 Acked Write 4 Sent Done Done false
11869 3629 Acked Read 7 Sent Done Done false
11870 3630 Acked Read 9 Sent Done Done false
11871 3631 Acked Write 2 Sent Done Done false
11872 3632 Acked Read 5 Sent Done Done false
11873 3633 Acked Flush 0 Sent Done Done false
11874 3634 Acked Write 9 Sent Done Done false
11875 3635 Acked Read 3 Sent Done Done false
11876 3636 Acked Write 9 Sent Done Done false
11877 3637 Acked Read 1 Sent Done Done false
11878 3638 Acked Write 8 Sent Done Done false
11879 3639 Acked Read 10 Sent Done Done false
11880 3640 Acked Write 5 Sent Done Done false
11881 3641 Acked Write 3 Sent Done Done false
11882 3642 Acked Read 2 Sent Done Done false
11883 3643 Acked Read 9 Sent Done Done false
11884 3644 Acked Write 5 Sent Done Done false
11885 3645 Acked Write 6 Sent Done Done false
11886 3646 Acked Read 5 Sent Done Done false
11887 3647 Acked Write 4 Sent Done Done false
11888 3648 Acked Read 2 Sent Done Done false
11889 3649 Acked Write 2 Sent Done Done false
11890 3650 Acked Read 7 Sent Done Done false
11891 3651 Acked Write 3 Sent Done Done false
11892 3652 Acked Read 8 Sent Done Done false
11893 3653 Acked Write 3 Sent Done Done false
11894 3654 Acked Read 3 Sent Done Done false
11895 3655 Acked Read 5 Sent Done Done false
11896 3656 Acked Write 2 Sent Done Done false
11897 3657 Acked Write 7 Sent Done Done false
11898 3658 Acked Write 8 Sent Done Done false
11899 3659 Acked Write 1 Sent Done Done false
11900 3660 Acked Write 5 Sent Done Done false
11901 3661 Acked Write 7 Sent Done Done false
11902 3662 Acked Flush 0 Sent Done Done false
11903 3663 Acked Write 3 Sent Done Done false
11904 3664 Acked Read 2 Sent Done Done false
11905 3665 Acked Read 4 Sent Done Done false
11906 3666 Acked Write 10 Sent Done Done false
11907 3667 Acked Flush 0 Sent Done Done false
11908 3668 Acked Write 10 Sent Done Done false
11909 3669 Acked Read 6 Sent Done Done false
11910 3670 Acked Write 6 Sent Done Done false
11911 3671 Acked Write 8 Sent Done Done false
11912 3672 Acked Read 4 Sent Done Done false
11913 3673 Acked Write 2 Sent Done Done false
11914 3674 Acked Write 10 Sent Done Done false
11915 3675 Acked Write 7 Sent Done Done false
11916 3676 Acked Read 8 Sent Done Done false
11917 3677 Acked Read 5 Sent Done Done false
11918 3678 Acked Flush 0 Sent Done Done false
11919 3679 Acked Read 10 Sent Done Done false
11920 3680 Acked Flush 0 Sent Done Done false
11921 3681 Acked Write 5 Sent Done Done false
11922 3682 Acked Flush 0 Sent Done Done false
11923 3683 Acked Write 9 Sent Done Done false
11924 3684 Acked Write 9 Sent Done Done false
11925 3685 Acked Read 3 Sent Done Done false
11926 3686 Acked Read 9 Sent Done Done false
11927 3687 Acked Flush 0 Sent Done Done false
11928 3688 Acked Write 5 Sent Done Done false
11929 3689 Acked Read 9 Sent Done Done false
11930 3690 Acked Read 4 Sent Done Done false
11931 3691 Acked Read 10 Sent Done Done false
11932 3692 Acked Read 6 Sent Done Done false
11933 3693 Acked Read 3 Sent Done Done false
11934 3694 Acked Flush 0 Sent Done Done false
11935 3695 Acked Read 9 Sent Done Done false
11936 3696 Acked Write 1 Sent Done Done false
11937 3697 Acked Write 6 Sent Done Done false
11938 3698 Acked Write 1 Sent Done Done false
11939 3699 Acked Read 10 Sent Done Done false
11940 3700 Acked Write 7 Sent Done Done false
11941 3701 Acked Flush 0 Sent Done Done false
11942 3702 Acked Read 8 Sent Done Done false
11943 3703 Acked Read 4 Sent Done Done false
11944 3704 Acked Write 2 Sent Done Done false
11945 3705 Acked Write 6 Sent Done Done false
11946 3706 Acked Read 1 Sent Done Done false
11947 3707 Acked Write 8 Sent Done Done false
11948 3708 Acked Write 7 Sent Done Done false
11949 3709 Acked Read 8 Sent Done Done false
11950 3710 Acked Read 10 Sent Done Done false
11951 3711 Acked Read 4 Sent Done Done false
11952 3712 Acked Write 10 Sent Done Done false
11953 3713 Acked Read 6 Sent Done Done false
11954 3714 Acked Read 2 Sent Done Done false
11955 3715 Acked Read 9 Sent Done Done false
11956 3716 Acked Write 6 Sent Done Done false
11957 STATES DS:0 DS:1 DS:2 TOTAL
11958 Sent 301 0 0 301
11959 Done 0 301 301 602
11960 Skipped 0 0 0 0
11961 Error 0 0 0 0
11962 Last Flush: Some(JobId(3415)) Some(JobId(3701)) Some(JobId(3701))
11963 Downstairs last five completed: 3415 3414 3413 3412 3411
11964 Upstairs last five completed: 3716 3715 3714 3713 3712
11965 CLIENT: Up:0 ds:301 act:2
119662025-02-07T20:09:56.238ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
119672025-02-07T20:09:57.200ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
119682025-02-07T20:09:58.202ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
119692025-02-07T20:09:59.203ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
11970 ----------------------------------------------------------------
11971 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11972 States: Offline Active Active
11973 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11974 3416 Acked Flush 0 Sent Done Done false
11975 3417 Acked Read 1 Sent Done Done false
11976 3418 Acked Write 2 Sent Done Done false
11977 3419 Acked Read 3 Sent Done Done false
11978 3420 Acked Write 2 Sent Done Done false
11979 3421 Acked Write 7 Sent Done Done false
11980 3422 Acked Read 4 Sent Done Done false
11981 3423 Acked Read 4 Sent Done Done false
11982 3424 Acked Write 8 Sent Done Done false
11983 3425 Acked Flush 0 Sent Done Done false
11984 3426 Acked Write 10 Sent Done Done false
11985 3427 Acked Write 6 Sent Done Done false
11986 3428 Acked Read 3 Sent Done Done false
11987 3429 Acked Write 7 Sent Done Done false
11988 3430 Acked Write 8 Sent Done Done false
11989 3431 Acked Write 4 Sent Done Done false
11990 3432 Acked Write 5 Sent Done Done false
11991 3433 Acked Read 3 Sent Done Done false
11992 3434 Acked Read 2 Sent Done Done false
11993 3435 Acked Write 5 Sent Done Done false
11994 3436 Acked Write 9 Sent Done Done false
11995 3437 Acked Write 10 Sent Done Done false
11996 3438 Acked Read 10 Sent Done Done false
11997 3439 Acked Write 6 Sent Done Done false
11998 3440 Acked Read 10 Sent Done Done false
11999 3441 Acked Write 1 Sent Done Done false
12000 3442 Acked Read 1 Sent Done Done false
12001 3443 Acked Write 1 Sent Done Done false
12002 3444 Acked Write 2 Sent Done Done false
12003 3445 Acked Write 4 Sent Done Done false
12004 3446 Acked Write 5 Sent Done Done false
12005 3447 Acked Read 5 Sent Done Done false
12006 3448 Acked Write 7 Sent Done Done false
12007 3449 Acked Read 1 Sent Done Done false
12008 3450 Acked Write 8 Sent Done Done false
12009 3451 Acked Read 6 Sent Done Done false
12010 3452 Acked Read 5 Sent Done Done false
12011 3453 Acked Flush 0 Sent Done Done false
12012 3454 Acked Flush 0 Sent Done Done false
12013 3455 Acked Read 8 Sent Done Done false
12014 3456 Acked Write 1 Sent Done Done false
12015 3457 Acked Read 4 Sent Done Done false
12016 3458 Acked Write 2 Sent Done Done false
12017 3459 Acked Write 3 Sent Done Done false
12018 3460 Acked Write 7 Sent Done Done false
12019 3461 Acked Read 6 Sent Done Done false
12020 3462 Acked Read 6 Sent Done Done false
12021 3463 Acked Read 9 Sent Done Done false
12022 3464 Acked Read 7 Sent Done Done false
12023 3465 Acked Read 3 Sent Done Done false
12024 3466 Acked Write 8 Sent Done Done false
12025 3467 Acked Read 2 Sent Done Done false
12026 3468 Acked Read 9 Sent Done Done false
12027 3469 Acked Read 8 Sent Done Done false
12028 3470 Acked Read 8 Sent Done Done false
12029 3471 Acked Write 1 Sent Done Done false
12030 3472 Acked Read 9 Sent Done Done false
12031 3473 Acked Write 1 Sent Done Done false
12032 3474 Acked Write 1 Sent Done Done false
12033 3475 Acked Write 9 Sent Done Done false
12034 3476 Acked Read 9 Sent Done Done false
12035 3477 Acked Write 7 Sent Done Done false
12036 3478 Acked Read 1 Sent Done Done false
12037 3479 Acked Write 1 Sent Done Done false
12038 3480 Acked Read 3 Sent Done Done false
12039 3481 Acked Read 5 Sent Done Done false
12040 3482 Acked Read 9 Sent Done Done false
12041 3483 Acked Write 4 Sent Done Done false
12042 3484 Acked Read 1 Sent Done Done false
12043 3485 Acked Read 7 Sent Done Done false
12044 3486 Acked Read 4 Sent Done Done false
12045 3487 Acked Read 1 Sent Done Done false
12046 3488 Acked Read 2 Sent Done Done false
12047 3489 Acked Flush 0 Sent Done Done false
12048 3490 Acked Read 3 Sent Done Done false
12049 3491 Acked Read 1 Sent Done Done false
12050 3492 Acked Write 10 Sent Done Done false
12051 3493 Acked Flush 0 Sent Done Done false
12052 3494 Acked Write 5 Sent Done Done false
12053 3495 Acked Read 6 Sent Done Done false
12054 3496 Acked Write 10 Sent Done Done false
12055 3497 Acked Read 5 Sent Done Done false
12056 3498 Acked Read 5 Sent Done Done false
12057 3499 Acked Read 4 Sent Done Done false
12058 3500 Acked Write 4 Sent Done Done false
12059 3501 Acked Flush 0 Sent Done Done false
12060 3502 Acked Read 7 Sent Done Done false
12061 3503 Acked Read 3 Sent Done Done false
12062 3504 Acked Write 1 Sent Done Done false
12063 3505 Acked Read 5 Sent Done Done false
12064 3506 Acked Flush 0 Sent Done Done false
12065 3507 Acked Write 1 Sent Done Done false
12066 3508 Acked Write 4 Sent Done Done false
12067 3509 Acked Write 6 Sent Done Done false
12068 3510 Acked Write 10 Sent Done Done false
12069 3511 Acked Read 8 Sent Done Done false
12070 3512 Acked Flush 0 Sent Done Done false
12071 3513 Acked Read 2 Sent Done Done false
12072 3514 Acked Read 7 Sent Done Done false
12073 3515 Acked Read 7 Sent Done Done false
12074 3516 Acked Write 7 Sent Done Done false
12075 3517 Acked Flush 0 Sent Done Done false
12076 3518 Acked Read 5 Sent Done Done false
12077 3519 Acked Read 5 Sent Done Done false
12078 3520 Acked Write 10 Sent Done Done false
12079 3521 Acked Read 7 Sent Done Done false
12080 3522 Acked Read 4 Sent Done Done false
12081 3523 Acked Read 2 Sent Done Done false
12082 3524 Acked Read 9 Sent Done Done false
12083 3525 Acked Write 10 Sent Done Done false
12084 3526 Acked Write 8 Sent Done Done false
12085 3527 Acked Write 9 Sent Done Done false
12086 3528 Acked Flush 0 Sent Done Done false
12087 3529 Acked Read 3 Sent Done Done false
12088 3530 Acked Read 6 Sent Done Done false
12089 3531 Acked Flush 0 Sent Done Done false
12090 3532 Acked Flush 0 Sent Done Done false
12091 3533 Acked Read 1 Sent Done Done false
12092 3534 Acked Read 5 Sent Done Done false
12093 3535 Acked Read 3 Sent Done Done false
12094 3536 Acked Read 7 Sent Done Done false
12095 3537 Acked Write 10 Sent Done Done false
12096 3538 Acked Read 6 Sent Done Done false
12097 3539 Acked Write 9 Sent Done Done false
12098 3540 Acked Read 6 Sent Done Done false
12099 3541 Acked Read 5 Sent Done Done false
12100 3542 Acked Read 6 Sent Done Done false
12101 3543 Acked Read 10 Sent Done Done false
12102 3544 Acked Flush 0 Sent Done Done false
12103 3545 Acked Write 8 Sent Done Done false
12104 3546 Acked Flush 0 Sent Done Done false
12105 3547 Acked Read 3 Sent Done Done false
12106 3548 Acked Read 2 Sent Done Done false
12107 3549 Acked Read 6 Sent Done Done false
12108 3550 Acked Write 4 Sent Done Done false
12109 3551 Acked Flush 0 Sent Done Done false
12110 3552 Acked Read 3 Sent Done Done false
12111 3553 Acked Read 5 Sent Done Done false
12112 3554 Acked Write 2 Sent Done Done false
12113 3555 Acked Write 4 Sent Done Done false
12114 3556 Acked Read 2 Sent Done Done false
12115 3557 Acked Flush 0 Sent Done Done false
12116 3558 Acked Write 6 Sent Done Done false
12117 3559 Acked Read 7 Sent Done Done false
12118 3560 Acked Read 3 Sent Done Done false
12119 3561 Acked Read 6 Sent Done Done false
12120 3562 Acked Write 9 Sent Done Done false
12121 3563 Acked Read 6 Sent Done Done false
12122 3564 Acked Write 9 Sent Done Done false
12123 3565 Acked Write 10 Sent Done Done false
12124 3566 Acked Write 6 Sent Done Done false
12125 3567 Acked Read 7 Sent Done Done false
12126 3568 Acked Flush 0 Sent Done Done false
12127 3569 Acked Flush 0 Sent Done Done false
12128 3570 Acked Read 8 Sent Done Done false
12129 3571 Acked Write 4 Sent Done Done false
12130 3572 Acked Write 10 Sent Done Done false
12131 3573 Acked Write 2 Sent Done Done false
12132 3574 Acked Write 5 Sent Done Done false
12133 3575 Acked Read 2 Sent Done Done false
12134 3576 Acked Read 3 Sent Done Done false
12135 3577 Acked Write 8 Sent Done Done false
12136 3578 Acked Read 8 Sent Done Done false
12137 3579 Acked Write 5 Sent Done Done false
12138 3580 Acked Write 1 Sent Done Done false
12139 3581 Acked Write 4 Sent Done Done false
12140 3582 Acked Write 8 Sent Done Done false
12141 3583 Acked Write 3 Sent Done Done false
12142 3584 Acked Write 9 Sent Done Done false
12143 3585 Acked Write 1 Sent Done Done false
12144 3586 Acked Write 3 Sent Done Done false
12145 3587 Acked Write 3 Sent Done Done false
12146 3588 Acked Read 2 Sent Done Done false
12147 3589 Acked Flush 0 Sent Done Done false
12148 3590 Acked Flush 0 Sent Done Done false
12149 3591 Acked Read 1 Sent Done Done false
12150 3592 Acked Read 3 Sent Done Done false
12151 3593 Acked Write 9 Sent Done Done false
12152 3594 Acked Write 4 Sent Done Done false
12153 3595 Acked Read 4 Sent Done Done false
12154 3596 Acked Read 1 Sent Done Done false
12155 3597 Acked Write 4 Sent Done Done false
12156 3598 Acked Write 7 Sent Done Done false
12157 3599 Acked Read 2 Sent Done Done false
12158 3600 Acked Write 6 Sent Done Done false
12159 3601 Acked Write 3 Sent Done Done false
12160 3602 Acked Read 10 Sent Done Done false
12161 3603 Acked Read 6 Sent Done Done false
12162 3604 Acked Flush 0 Sent Done Done false
12163 3605 Acked Write 6 Sent Done Done false
12164 3606 Acked Read 9 Sent Done Done false
12165 3607 Acked Write 1 Sent Done Done false
12166 3608 Acked Write 4 Sent Done Done false
12167 3609 Acked Read 10 Sent Done Done false
12168 3610 Acked Read 3 Sent Done Done false
12169 3611 Acked Write 6 Sent Done Done false
12170 3612 Acked Read 1 Sent Done Done false
12171 3613 Acked Read 10 Sent Done Done false
12172 3614 Acked Write 1 Sent Done Done false
12173 3615 Acked Read 7 Sent Done Done false
12174 3616 Acked Write 2 Sent Done Done false
12175 3617 Acked Write 7 Sent Done Done false
12176 3618 Acked Read 10 Sent Done Done false
12177 3619 Acked Flush 0 Sent Done Done false
12178 3620 Acked Read 10 Sent Done Done false
12179 3621 Acked Read 8 Sent Done Done false
12180 3622 Acked Read 6 Sent Done Done false
12181 3623 Acked Flush 0 Sent Done Done false
12182 3624 Acked Write 6 Sent Done Done false
12183 3625 Acked Read 9 Sent Done Done false
12184 3626 Acked Flush 0 Sent Done Done false
12185 3627 Acked Write 8 Sent Done Done false
12186 3628 Acked Write 4 Sent Done Done false
12187 3629 Acked Read 7 Sent Done Done false
12188 3630 Acked Read 9 Sent Done Done false
12189 3631 Acked Write 2 Sent Done Done false
12190 3632 Acked Read 5 Sent Done Done false
12191 3633 Acked Flush 0 Sent Done Done false
12192 3634 Acked Write 9 Sent Done Done false
12193 3635 Acked Read 3 Sent Done Done false
12194 3636 Acked Write 9 Sent Done Done false
12195 3637 Acked Read 1 Sent Done Done false
12196 3638 Acked Write 8 Sent Done Done false
12197 3639 Acked Read 10 Sent Done Done false
12198 3640 Acked Write 5 Sent Done Done false
12199 3641 Acked Write 3 Sent Done Done false
12200 3642 Acked Read 2 Sent Done Done false
12201 3643 Acked Read 9 Sent Done Done false
12202 3644 Acked Write 5 Sent Done Done false
12203 3645 Acked Write 6 Sent Done Done false
12204 3646 Acked Read 5 Sent Done Done false
12205 3647 Acked Write 4 Sent Done Done false
12206 3648 Acked Read 2 Sent Done Done false
12207 3649 Acked Write 2 Sent Done Done false
12208 3650 Acked Read 7 Sent Done Done false
12209 3651 Acked Write 3 Sent Done Done false
12210 3652 Acked Read 8 Sent Done Done false
12211 3653 Acked Write 3 Sent Done Done false
12212 3654 Acked Read 3 Sent Done Done false
12213 3655 Acked Read 5 Sent Done Done false
12214 3656 Acked Write 2 Sent Done Done false
12215 3657 Acked Write 7 Sent Done Done false
12216 3658 Acked Write 8 Sent Done Done false
12217 3659 Acked Write 1 Sent Done Done false
12218 3660 Acked Write 5 Sent Done Done false
12219 3661 Acked Write 7 Sent Done Done false
12220 3662 Acked Flush 0 Sent Done Done false
12221 3663 Acked Write 3 Sent Done Done false
12222 3664 Acked Read 2 Sent Done Done false
12223 3665 Acked Read 4 Sent Done Done false
12224 3666 Acked Write 10 Sent Done Done false
12225 3667 Acked Flush 0 Sent Done Done false
12226 3668 Acked Write 10 Sent Done Done false
12227 3669 Acked Read 6 Sent Done Done false
12228 3670 Acked Write 6 Sent Done Done false
12229 3671 Acked Write 8 Sent Done Done false
12230 3672 Acked Read 4 Sent Done Done false
12231 3673 Acked Write 2 Sent Done Done false
12232 3674 Acked Write 10 Sent Done Done false
12233 3675 Acked Write 7 Sent Done Done false
12234 3676 Acked Read 8 Sent Done Done false
12235 3677 Acked Read 5 Sent Done Done false
12236 3678 Acked Flush 0 Sent Done Done false
12237 3679 Acked Read 10 Sent Done Done false
12238 3680 Acked Flush 0 Sent Done Done false
12239 3681 Acked Write 5 Sent Done Done false
12240 3682 Acked Flush 0 Sent Done Done false
12241 3683 Acked Write 9 Sent Done Done false
12242 3684 Acked Write 9 Sent Done Done false
12243 3685 Acked Read 3 Sent Done Done false
12244 3686 Acked Read 9 Sent Done Done false
12245 3687 Acked Flush 0 Sent Done Done false
12246 3688 Acked Write 5 Sent Done Done false
12247 3689 Acked Read 9 Sent Done Done false
12248 3690 Acked Read 4 Sent Done Done false
12249 3691 Acked Read 10 Sent Done Done false
12250 3692 Acked Read 6 Sent Done Done false
12251 3693 Acked Read 3 Sent Done Done false
12252 3694 Acked Flush 0 Sent Done Done false
12253 3695 Acked Read 9 Sent Done Done false
12254 3696 Acked Write 1 Sent Done Done false
12255 3697 Acked Write 6 Sent Done Done false
12256 3698 Acked Write 1 Sent Done Done false
12257 3699 Acked Read 10 Sent Done Done false
12258 3700 Acked Write 7 Sent Done Done false
12259 3701 Acked Flush 0 Sent Done Done false
12260 3702 Acked Read 8 Sent Done Done false
12261 3703 Acked Read 4 Sent Done Done false
12262 3704 Acked Write 2 Sent Done Done false
12263 3705 Acked Write 6 Sent Done Done false
12264 3706 Acked Read 1 Sent Done Done false
12265 3707 Acked Write 8 Sent Done Done false
12266 3708 Acked Write 7 Sent Done Done false
12267 3709 Acked Read 8 Sent Done Done false
12268 3710 Acked Read 10 Sent Done Done false
12269 3711 Acked Read 4 Sent Done Done false
12270 3712 Acked Write 10 Sent Done Done false
12271 3713 Acked Read 6 Sent Done Done false
12272 3714 Acked Read 2 Sent Done Done false
12273 3715 Acked Read 9 Sent Done Done false
12274 3716 Acked Write 6 Sent Done Done false
12275 STATES DS:0 DS:1 DS:2 TOTAL
12276 Sent 301 0 0 301
12277 Done 0 301 301 602
12278 Skipped 0 0 0 0
12279 Error 0 0 0 0
12280 Last Flush: Some(JobId(3415)) Some(JobId(3701)) Some(JobId(3701))
12281 Downstairs last five completed: 3415 3414 3413 3412 3411
12282 Upstairs last five completed: 3716 3715 3714 3713 3712
12283 CLIENT: Up:0 ds:301 act:2
122842025-02-07T20:10:00.241ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
122852025-02-07T20:10:01.207ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
122862025-02-07T20:10:02.208ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
122872025-02-07T20:10:03.209ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
122882025-02-07T20:10:03.497ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
122892025-02-07T20:10:03.497ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34473) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
122902025-02-07T20:10:03.498ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
122912025-02-07T20:10:03.499ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
122922025-02-07T20:10:03.499ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
122932025-02-07T20:10:03.499ZINFOcrucible: send last flush ID to this DS: Some(JobId(3415)) = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
122942025-02-07T20:10:03.499ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
122952025-02-07T20:10:03.499ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(3415)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
122962025-02-07T20:10:03.499ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(3415)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
12297 ----------------------------------------------------------------
12298 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:15
12299 States: Active Active Active
12300 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12301 3702 Acked Read 8 Done Done Done true
12302 3703 Acked Read 4 Done Done Done true
12303 3704 Acked Write 2 Done Done Done true
12304 3705 Acked Write 6 Done Done Done true
12305 3706 Acked Read 1 Done Done Done true
12306 3707 Acked Write 8 Done Done Done true
12307 3708 Acked Write 7 Done Done Done true
12308 3709 Acked Read 8 Done Done Done true
12309 3710 Acked Read 10 Done Done Done true
12310 3711 Acked Read 4 Done Done Done true
12311 3712 Acked Write 10 Done Done Done true
12312 3713 Acked Read 6 Done Done Done true
12313 3714 Acked Read 2 Done Done Done true
12314 3715 Acked Read 9 Done Done Done true
12315 3716 Acked Write 6 Done Done Done true
12316 STATES DS:0 DS:1 DS:2 TOTAL
12317 Sent 0 0 0 0
12318 Done 15 15 15 45
12319 Skipped 0 0 0 0
12320 Error 0 0 0 0
12321 Last Flush: Some(JobId(3701)) Some(JobId(3701)) Some(JobId(3701))
12322 Downstairs last five completed: 3701 3700 3699 3698 3697
12323 Upstairs last five completed: 3716 3715 3714 3713 3712
12324 CLIENT: Up:0 ds:15 act:3
12325 ----------------------------------------------------------------
12326 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12327 Downstairs last five completed: 3717 3716 3715 3714 3713
12328 Upstairs last five completed: 3717 3716 3715 3714 3713
12329 CLIENT: Up:0 ds:0 act:3
12330 Replay: All jobs finished
12331 Main task has work to do, go find it
12332 got dsc Stop(2)
12333 stop 2
12334 Feb 07 20:10:07.744 INFO request completed, latency_us: 350, response_code: 204, uri: /stop/cid/2, method[8830] Got stop action so:false kr:false
12335 : POST, req_id: 8d321794-ae96-4cfb-b37f-c397873c0e60, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
123362025-02-07T20:10:07.745ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
123372025-02-07T20:10:07.745ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
123382025-02-07T20:10:07.745ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
12339 Feb 07 20:10:07.745 INFO request completed, latency_us: 113, response_code: 200, uri: /state/cid/2, method: GET, req_id: 809e27a5-7b87-434f-8439-2a89d24096e5, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
123402025-02-07T20:10:07.745ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
12341 [8830] Exited with: ExitStatus(unix_wait_status(9))
12342 [8830][2] reports Exit
12343 Feb 07 20:10:11.747 INFO request completed, latency_us: 457, response_code: 200, uri: /state/cid/2, method: GET, req_id: cb44ff72-e7ed-4c5d-a918-4930e8238b83, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
12344 001/300 Flush
12345 002/300 Read block 4131 len 16384
12346 003/300 Read block 2441 len 8192
12347 004/300 Read block 375 len 24576
12348 005/300 Write block 3648 len 32768 data: 2 3 3 4 3 2 4 4
12349 006/300 Read block 252 len 20480
12350 007/300 Flush
12351 008/300 Read block 1297 len 28672
12352 009/300 Write block 3392 len 32768 data: 3 3 5 5 5 5 6 6
12353 010/300 Read block 3259 len 40960
12354 011/300 Read block 2344 len 32768
12355 012/300 Read block 2934 len 4096
12356 013/300 Read block 4249 len 36864
12357 014/300 Read block 4818 len 32768
12358 015/300 Read block 4263 len 24576
12359 016/300 Write block 2323 len 36864 data: 2 2 2 2 2 2 2 2 2
12360 017/300 Read block 3165 len 28672
12361 018/300 Read block 2592 len 24576
12362 019/300 Read block 3484 len 16384
12363 020/300 Read block 2951 len 20480
12364 021/300 Write block 2091 len 28672 data: 3 3 2 2 2 2 2
12365 022/300 Read block 2857 len 40960
12366 023/300 Read block 2157 len 32768
12367 024/300 Read block 3189 len 16384
12368 025/300 Write block 4566 len 24576 data: 3 3 3 3 3 3
12369 026/300 Write block 4840 len 20480 data: 4 3 2 2 2
12370 027/300 Read block 4907 len 28672
12371 028/300 Write block 3281 len 12288 data: 3 2 3
12372 029/300 Read block 2434 len 12288
12373 030/300 Write block 4615 len 32768 data: 2 2 2 3 3 5 3 4
12374 031/300 Read block 1506 len 12288
12375 032/300 Read block 2684 len 16384
12376 033/300 Write block 1099 len 28672 data: 2 2 3 3 3 3 3
12377 034/300 Read block 1382 len 40960
12378 035/300 Read block 1471 len 40960
12379 036/300 Read block 1908 len 20480
12380 037/300 Read block 1435 len 36864
12381 038/300 Read block 618 len 24576
12382 039/300 Write block 4330 len 12288 data: 4 5 5
12383 040/300 Write block 2086 len 24576 data: 5 5 3 3 3 4
12384 041/300 Read block 1179 len 4096
12385 042/300 Write block 2784 len 4096 data: 3
12386 043/300 Write block 108 len 16384 data: 2 2 3 3
12387 044/300 Write block 3331 len 28672 data: 4 4 6 6 4 4 4
12388 045/300 Read block 1877 len 20480
12389 046/300 Read block 827 len 8192
12390 047/300 Write block 1810 len 32768 data: 2 4 4 4 5 4 4 3
12391 048/300 Write block 3691 len 8192 data: 2 3
12392 049/300 Read block 4941 len 4096
12393 050/300 Read block 670 len 28672
12394 051/300 Write block 4695 len 12288 data: 3 3 3
12395 052/300 Read block 1186 len 4096
12396 053/300 Read block 2076 len 8192
12397 054/300 Flush
12398 055/300 Flush
12399 056/300 Write block 2645 len 12288 data: 2 2 3
12400 057/300 Flush
12401 058/300 Write block 939 len 20480 data: 2 2 2 3 3
12402 059/300 Read block 2626 len 16384
12403 060/300 Read block 3988 len 4096
12404 061/300 Read block 1143 len 12288
12405 062/300 Write block 3159 len 28672 data: 4 3 3 3 3 4 4
12406 063/300 Write block 2801 len 40960 data: 6 8 6 5 4 4 4 4 4 2
12407 064/300 Read block 909 len 16384
12408 065/300 Write block 3424 len 32768 data: 4 4 4 4 4 4 4 4
12409 066/300 Read block 4506 len 16384
12410 067/300 Read block 20 len 16384
12411 068/300 Write block 3671 len 28672 data: 4 3 4 3 3 3 4
12412 069/300 Flush
12413 070/300 Write block 1834 len 8192 data: 6 5
12414 071/300 Write block 2997 len 36864 data: 2 2 3 3 4 4 4 5 5
12415 072/300 Read block 474 len 32768
12416 073/300 Write block 81 len 20480 data: 3 2 2 2 2
12417 074/300 Flush
12418 075/300 Write block 3643 len 32768 data: 2 2 2 2 2 3 4 4
12419 076/300 Read block 1756 len 16384
12420 077/300 Write block 4217 len 16384 data: 4 4 4 4
12421 078/300 Write block 474 len 8192 data: 3 3
12422 079/300 Flush
12423 080/300 Write block 1980 len 20480 data: 4 4 4 4 4
12424 081/300 Read block 3742 len 8192
12425 082/300 Read block 3489 len 28672
12426 083/300 Read block 3190 len 4096
12427 084/300 Read block 2216 len 40960
12428 085/300 Read block 1019 len 28672
12429 086/300 Read block 3903 len 24576
12430 087/300 Write block 198 len 40960 data: 4 4 4 3 3 2 2 2 2 2
12431 088/300 Read block 1029 len 8192
12432 089/300 Write block 739 len 28672 data: 2 2 2 2 2 2 2
12433 090/300 Write block 3729 len 32768 data: 2 2 2 4 4 4 4 2
12434 091/300 Read block 3197 len 16384
12435 092/300 Read block 2596 len 8192
12436 093/300 Write block 460 len 36864 data: 4 3 4 4 3 2 2 2 2
12437 094/300 Write block 4841 len 24576 data: 4 3 3 3 2 2
12438 095/300 Write block 477 len 16384 data: 3 3 3 3
12439 096/300 Write block 1809 len 40960 data: 2 3 5 5 5 6 5 5 4 3
12440 097/300 Read block 4322 len 12288
12441 098/300 Read block 4064 len 20480
12442 099/300 Read block 2454 len 36864
12443 100/300 Read block 139 len 4096
12444 101/300 Read block 4277 len 28672
12445 102/300 Read block 3890 len 24576
12446 103/300 Read block 2139 len 8192
12447 104/300 Read block 2226 len 4096
12448 105/300 Write block 2167 len 20480 data: 4 2 2 3 3
12449 106/300 Read block 1862 len 32768
12450 107/300 Read block 3755 len 20480
12451 108/300 Write block 3564 len 4096 data: 2
12452 109/300 Write block 2070 len 36864 data: 2 2 3 3 3 4 3 3 5
12453 110/300 Write block 150 len 32768 data: 3 3 3 3 3 2 2 2
12454 111/300 Write block 4207 len 28672 data: 2 2 2 2 2 2 2
12455 112/300 Read block 154 len 32768
12456 113/300 Read block 1467 len 36864
12457 114/300 Write block 2129 len 16384 data: 4 3 2 2
12458 115/300 Read block 1237 len 36864
12459 116/300 Read block 3312 len 24576
12460 117/300 Read block 2772 len 28672
12461 118/300 Read block 4978 len 12288
12462 119/300 Write block 1070 len 36864 data: 3 4 4 3 3 3 3 4 4
12463 120/300 Write block 2800 len 36864 data: 5 7 9 7 6 5 5 5 5
12464 121/300 Flush
12465 122/300 Read block 4450 len 32768
12466 123/300 Write block 4071 len 4096 data: 4
12467 124/300 Write block 3774 len 24576 data: 4 5 4 4 5 4
12468 125/300 Write block 233 len 16384 data: 3 3 3 3
12469 126/300 Write block 3403 len 36864 data: 3 3 3 4 3 3 3 4 4
12470 127/300 Write block 3653 len 16384 data: 3 5 5 4
12471 128/300 Write block 2861 len 40960 data: 5 6 5 5 4 3 3 4 3 3
12472 129/300 Write block 3923 len 28672 data: 3 3 2 3 3 3 2
12473 130/300 Read block 1906 len 28672
12474 131/300 Read block 3607 len 36864
12475 132/300 Read block 2568 len 16384
12476 133/300 Write block 3443 len 20480 data: 2 4 4 4 4
12477 134/300 Read block 1296 len 8192
12478 135/300 Write block 3708 len 32768 data: 3 3 3 2 2 2 2 2
12479 136/300 Read block 4564 len 16384
12480 137/300 Read block 3037 len 28672
12481 138/300 Read block 785 len 20480
12482 139/300 Flush
12483 140/300 Flush
12484 141/300 Write block 913 len 8192 data: 2 2
12485 142/300 Read block 931 len 24576
12486 143/300 Write block 3993 len 24576 data: 3 3 4 4 4 4
12487 144/300 Write block 3038 len 36864 data: 3 3 3 3 3 3 4 5 4
12488 145/300 Read block 4863 len 36864
12489 146/300 Read block 1362 len 24576
12490 147/300 Read block 4713 len 32768
12491 148/300 Flush
12492 149/300 Write block 3677 len 16384 data: 5 3 3 3
12493 150/300 Write block 3173 len 24576 data: 4 2 2 2 2 2
12494 151/300 Read block 1959 len 24576
12495 152/300 Write block 4610 len 28672 data: 4 4 3 3 3 3 3
12496 153/300 Read block 3967 len 28672
12497 154/300 Write block 1110 len 16384 data: 4 5 4 4
12498 155/300 Write block 1887 len 40960 data: 3 3 3 3 3 4 3 3 3 3
12499 156/300 Read block 3514 len 28672
12500 157/300 Read block 1791 len 32768
12501 158/300 Read block 4318 len 32768
12502 159/300 Write block 3666 len 12288 data: 6 4 4
12503 160/300 Write block 1693 len 20480 data: 3 3 3 3 2
12504 161/300 Flush
12505 162/300 Read block 3194 len 28672
12506 163/300 Read block 374 len 20480
12507 164/300 Read block 917 len 12288
12508 165/300 Read block 2915 len 36864
12509 166/300 Write block 4974 len 36864 data: 3 3 2 2 2 3 3 2 2
12510 167/300 Read block 3640 len 24576
12511 168/300 Read block 3452 len 20480
12512 169/300 Read block 2169 len 32768
12513 170/300 Read block 3151 len 12288
12514 171/300 Write block 4836 len 16384 data: 6 4 4 4
12515 172/300 Read block 650 len 24576
12516 173/300 Write block 3657 len 16384 data: 4 4 6 7
12517 174/300 Read block 2212 len 16384
12518 175/300 Flush
12519 176/300 Write block 2630 len 12288 data: 2 2 2
12520 177/300 Read block 2440 len 20480
12521 178/300 Write block 4248 len 4096 data: 4
12522 179/300 Read block 3729 len 36864
12523 180/300 Read block 4128 len 8192
12524 181/300 Read block 4286 len 40960
12525 182/300 Write block 3578 len 4096 data: 3
12526 183/300 Write block 39 len 28672 data: 4 3 3 3 2 2 2
12527 184/300 Read block 2914 len 4096
12528 185/300 Write block 3733 len 36864 data: 5 5 5 3 2 2 3 3 3
12529 186/300 Read block 946 len 40960
12530 187/300 Read block 2283 len 24576
12531 188/300 Read block 3598 len 12288
12532 189/300 Write block 4524 len 16384 data: 3 3 3 3
12533 190/300 Read block 3125 len 24576
12534 191/300 Read block 371 len 36864
12535 192/300 Read block 2598 len 16384
12536 193/300 Write block 4748 len 40960 data: 5 4 4 5 4 3 4 4 4 3
12537 194/300 Read block 4597 len 16384
12538 195/300 Read block 139 len 12288
12539 196/300 Flush
12540 197/300 Write block 1040 len 36864 data: 3 3 3 3 4 4 4 4 3
12541 198/300 Read block 1637 len 32768
12542 199/300 Write block 4506 len 32768 data: 4 5 5 6 6 7 5 5
12543 200/300 Read block 4815 len 4096
12544 201/300 Write block 4024 len 16384 data: 4 3 3 3
12545 202/300 Read block 4331 len 20480
12546 203/300 Write block 4124 len 32768 data: 5 5 4 4 3 3 3 3
12547 204/300 Write block 4313 len 4096 data: 2
12548 205/300 Flush
12549 206/300 Flush
12550 207/300 Read block 531 len 28672
12551 208/300 Write block 859 len 8192 data: 3 3
12552 209/300 Flush
12553 210/300 Read block 74 len 20480
12554 211/300 Write block 1536 len 32768 data: 2 2 3 4 4 4 4 4
12555 212/300 Read block 55 len 20480
12556 213/300 Read block 1724 len 40960
12557 214/300 Write block 3564 len 32768 data: 3 3 3 3 3 4 5 5
12558 215/300 Read block 3522 len 16384
12559 216/300 Read block 2529 len 24576
12560 217/300 Read block 2137 len 12288
12561 218/300 Write block 4689 len 28672 data: 2 2 3 3 3 3 4
12562 219/300 Read block 4769 len 20480
12563 220/300 Read block 2092 len 24576
12564 221/300 Write block 3982 len 32768 data: 2 2 3 2 2 2 2 2
12565 222/300 Write block 979 len 28672 data: 4 3 3 3 4 3 3
12566 223/300 Write block 4773 len 32768 data: 3 2 2 2 2 2 2 2
12567 224/300 Write block 407 len 8192 data: 2 2
12568 225/300 Read block 4153 len 20480
12569 226/300 Write block 3834 len 20480 data: 4 4 4 3 3
12570 227/300 Write block 3218 len 36864 data: 3 3 3 3 2 2 2 3 3
12571 228/300 Read block 3388 len 24576
12572 229/300 Read block 4023 len 16384
12573 230/300 Write block 2507 len 8192 data: 3 3
12574 231/300 Read block 1617 len 20480
12575 232/300 Write block 1645 len 16384 data: 4 3 3 3
12576 233/300 Read block 3333 len 24576
12577 234/300 Write block 3792 len 24576 data: 2 2 2 2 2 4
12578 235/300 Read block 4347 len 8192
12579 236/300 Flush
12580 237/300 Flush
12581 238/300 Read block 807 len 16384
12582 239/300 Read block 2955 len 8192
12583 240/300 Write block 839 len 4096 data: 2
12584 241/300 Write block 3498 len 4096 data: 3
12585 242/300 Write block 1631 len 12288 data: 4 4 4
12586 243/300 Read block 122 len 4096
12587 244/300 Write block 2692 len 36864 data: 3 2 2 2 3 2 2 2 2
12588 245/300 Write block 2754 len 12288 data: 3 4 4
12589 246/300 Write block 2686 len 40960 data: 2 2 2 3 3 3 4 3 3 3
12590 247/300 Flush
12591 248/300 Read block 4119 len 24576
12592 249/300 Read block 2191 len 12288
12593 250/300 Write block 2931 len 28672 data: 3 3 4 4 4 4 3
12594 251/300 Read block 3641 len 40960
12595 252/300 Write block 381 len 4096 data: 3
12596 253/300 Read block 1401 len 24576
12597 254/300 Flush
12598 255/300 Write block 1341 len 20480 data: 2 2 2 3 4
12599 256/300 Flush
12600 257/300 Read block 2503 len 32768
12601 258/300 Write block 3243 len 24576 data: 3 3 2 3 3 5
12602 259/300 Read block 4862 len 16384
12603 260/300 Read block 574 len 20480
12604 261/300 Write block 592 len 40960 data: 3 2 2 2 2 3 3 3 3 3
12605 262/300 Write block 1434 len 32768 data: 3 3 3 3 4 4 4 3
12606 263/300 Write block 3696 len 32768 data: 5 4 3 3 2 3 3 3
12607 264/300 Read block 4629 len 8192
12608 265/300 Write block 4422 len 24576 data: 4 4 3 2 2 2
12609 266/300 Read block 3814 len 24576
12610 267/300 Write block 1606 len 32768 data: 2 2 2 3 3 3 3 3
12611 268/300 Flush
12612 269/300 Flush
12613 270/300 Read block 251 len 12288
12614 271/300 Read block 2675 len 24576
12615 272/300 Read block 312 len 8192
12616 273/300 Read block 2962 len 28672
12617 274/300 Read block 4764 len 40960
12618 275/300 Read block 1242 len 12288
12619 276/300 Flush
12620 277/300 Write block 3580 len 40960 data: 2 2 2 2 2 2 2 2 2 2
12621 278/300 Read block 2766 len 36864
12622 279/300 Read block 393 len 24576
12623 280/300 Read block 2226 len 32768
12624 281/300 Read block 252 len 16384
12625 282/300 Flush
12626 283/300 Read block 1422 len 36864
12627 284/300 Read block 3713 len 8192
12628 285/300 Read block 4494 len 24576
12629 286/300 Read block 430 len 20480
12630 287/300 Flush
12631 288/300 Write block 493 len 40960 data: 3 3 2 2 3 4 4 4 4 4
12632 289/300 Read block 3573 len 24576
12633 290/300 Read block 3618 len 28672
12634 291/300 Write block 4924 len 16384 data: 3 3 3 3
12635 292/300 Write block 121 len 12288 data: 4 3 3
12636 293/300 Write block 3246 len 4096 data: 4
12637 294/300 Flush
12638 295/300 Write block 4593 len 24576 data: 3 3 2 2 2 2
12639 296/300 Read block 18 len 32768
12640 297/300 Read block 3159 len 40960
12641 298/300 Read block 2293 len 40960
12642 299/300 Read block 101 len 36864
12643 300/300 Write block 3378 len 32768 data: 2 2 4 4 4 4 3 2
12644 301/300 Read block 3121 len 28672
12645 Main task has work to do, go find it
12646 got dsc Start(2)
12647 start 2
12648 Feb 07 20:10:12.002 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/2, method: POST, req_id: 132220f5-c76b-4eff-b4c0-59eaaf28bc8b, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
12649 Replay: started 2, returned:Ok(())
12650 ----------------------------------------------------------------
12651 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12652 States: Active Active Offline
12653 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12654 3718 Acked Flush 0 Done Done Sent false
12655 3719 Acked Read 4 Done Done Sent false
12656 3720 Acked Read 2 Done Done Sent false
12657 3721 Acked Read 6 Done Done Sent false
12658 3722 Acked Write 8 Done Done Sent false
12659 3723 Acked Read 5 Done Done Sent false
12660 3724 Acked Flush 0 Done Done Sent false
12661 3725 Acked Read 7 Done Done Sent false
12662 3726 Acked Write 8 Done Done Sent false
12663 3727 Acked Read 10 Done Done Sent false
12664 3728 Acked Read 8 Done Done Sent false
12665 3729 Acked Read 1 Done Done Sent false
12666 3730 Acked Read 9 Done Done Sent false
12667 3731 Acked Read 8 Done Done Sent false
12668 3732 Acked Read 6 Done Done Sent false
12669 3733 Acked Write 9 Done Done Sent false
12670 3734 Acked Read 7 Done Done Sent false
12671 3735 Acked Read 6 Done Done Sent false
12672 3736 Acked Read 4 Done Done Sent false
12673 3737 Acked Read 5 Done Done Sent false
12674 3738 Acked Write 7 Done Done Sent false
12675 3739 Acked Read 10 Done Done Sent false
12676 3740 Acked Read 8 Done Done Sent false
12677 3741 Acked Read 4 Done Done Sent false
12678 3742 Acked Write 6 Done Done Sent false
12679 3743 Acked Write 5 Done Done Sent false
12680 3744 Acked Read 7 Done Done Sent false
12681 3745 Acked Write 3 Done Done Sent false
12682 3746 Acked Read 3 Done Done Sent false
12683 3747 Acked Write 8 Done Done Sent false
12684 3748 Acked Read 3 Done Done Sent false
12685 3749 Acked Read 4 Done Done Sent false
12686 3750 Acked Write 7 Done Done Sent false
12687 3751 Acked Read 10 Done Done Sent false
12688 3752 Acked Read 10 Done Done Sent false
12689 3753 Acked Read 5 Done Done Sent false
12690 3754 Acked Read 9 Done Done Sent false
12691 3755 Acked Read 6 Done Done Sent false
12692 3756 Acked Write 3 Done Done Sent false
12693 3757 Acked Write 6 Done Done Sent false
12694 3758 Acked Read 1 Done Done Sent false
12695 3759 Acked Write 1 Done Done Sent false
12696 3760 Acked Write 4 Done Done Sent false
12697 3761 Acked Write 7 Done Done Sent false
12698 3762 Acked Read 5 Done Done Sent false
12699 3763 Acked Read 2 Done Done Sent false
12700 3764 Acked Write 8 Done Done Sent false
12701 3765 Acked Write 2 Done Done Sent false
12702 3766 Acked Read 1 Done Done Sent false
12703 3767 Acked Read 7 Done Done Sent false
12704 3768 Acked Write 3 Done Done Sent false
12705 3769 Acked Read 1 Done Done Sent false
12706 3770 Acked Read 2 Done Done Sent false
12707 3771 Acked Flush 0 Done Done Sent false
12708 3772 Acked Flush 0 Done Done Sent false
12709 3773 Acked Write 3 Done Done Sent false
12710 3774 Acked Flush 0 Done Done Sent false
12711 3775 Acked Write 5 Done Done Sent false
12712 3776 Acked Read 4 Done Done Sent false
12713 3777 Acked Read 1 Done Done Sent false
12714 3778 Acked Read 3 Done Done Sent false
12715 3779 Acked Write 7 Done Done Sent false
12716 3780 Acked Write 10 Done Done Sent false
12717 3781 Acked Read 4 Done Done Sent false
12718 3782 Acked Write 8 Done Done Sent false
12719 3783 Acked Read 4 Done Done Sent false
12720 3784 Acked Read 4 Done Done Sent false
12721 3785 Acked Write 7 Done Done Sent false
12722 3786 Acked Flush 0 Done Done Sent false
12723 3787 Acked Write 2 Done Done Sent false
12724 3788 Acked Write 9 Done Done Sent false
12725 3789 Acked Read 8 Done Done Sent false
12726 3790 Acked Write 5 Done Done Sent false
12727 3791 Acked Flush 0 Done Done Sent false
12728 3792 Acked Write 8 Done Done Sent false
12729 3793 Acked Read 4 Done Done Sent false
12730 3794 Acked Write 4 Done Done Sent false
12731 3795 Acked Write 2 Done Done Sent false
12732 3796 Acked Flush 0 Done Done Sent false
12733 3797 Acked Write 5 Done Done Sent false
12734 3798 Acked Read 2 Done Done Sent false
12735 3799 Acked Read 7 Done Done Sent false
12736 3800 Acked Read 1 Done Done Sent false
12737 3801 Acked Read 10 Done Done Sent false
12738 3802 Acked Read 7 Done Done Sent false
12739 3803 Acked Read 6 Done Done Sent false
12740 3804 Acked Write 10 Done Done Sent false
12741 3805 Acked Read 2 Done Done Sent false
12742 3806 Acked Write 7 Done Done Sent false
12743 3807 Acked Write 8 Done Done Sent false
12744 3808 Acked Read 4 Done Done Sent false
12745 3809 Acked Read 2 Done Done Sent false
12746 3810 Acked Write 9 Done Done Sent false
12747 3811 Acked Write 6 Done Done Sent false
12748 3812 Acked Write 4 Done Done Sent false
12749 3813 Acked Write 10 Done Done Sent false
12750 3814 Acked Read 3 Done Done Sent false
12751 3815 Acked Read 5 Done Done Sent false
12752 3816 Acked Read 9 Done Done Sent false
12753 3817 Acked Read 1 Done Done Sent false
12754 3818 Acked Read 7 Done Done Sent false
12755 3819 Acked Read 6 Done Done Sent false
12756 3820 Acked Read 2 Done Done Sent false
12757 3821 Acked Read 1 Done Done Sent false
12758 3822 Acked Write 5 Done Done Sent false
12759 3823 Acked Read 8 Done Done Sent false
12760 3824 Acked Read 5 Done Done Sent false
12761 3825 Acked Write 1 Done Done Sent false
12762 3826 Acked Write 9 Done Done Sent false
12763 3827 Acked Write 8 Done Done Sent false
12764 3828 Acked Write 7 Done Done Sent false
12765 3829 Acked Read 8 Done Done Sent false
12766 3830 Acked Read 9 Done Done Sent false
12767 3831 Acked Write 4 Done Done Sent false
12768 3832 Acked Read 9 Done Done Sent false
12769 3833 Acked Read 6 Done Done Sent false
12770 3834 Acked Read 7 Done Done Sent false
12771 3835 Acked Read 3 Done Done Sent false
12772 3836 Acked Write 9 Done Done Sent false
12773 3837 Acked Write 9 Done Done Sent false
12774 3838 Acked Flush 0 Done Done Sent false
12775 3839 Acked Read 8 Done Done Sent false
12776 3840 Acked Write 1 Done Done Sent false
12777 3841 Acked Write 6 Done Done Sent false
12778 3842 Acked Write 4 Done Done Sent false
12779 3843 Acked Write 9 Done Done Sent false
12780 3844 Acked Write 4 Done Done Sent false
12781 3845 Acked Write 10 Done Done Sent false
12782 3846 Acked Write 7 Done Done Sent false
12783 3847 Acked Read 7 Done Done Sent false
12784 3848 Acked Read 9 Done Done Sent false
12785 3849 Acked Read 4 Done Done Sent false
12786 3850 Acked Write 5 Done Done Sent false
12787 3851 Acked Read 2 Done Done Sent false
12788 3852 Acked Write 8 Done Done Sent false
12789 3853 Acked Read 4 Done Done Sent false
12790 3854 Acked Read 7 Done Done Sent false
12791 3855 Acked Read 5 Done Done Sent false
12792 3856 Acked Flush 0 Done Done Sent false
12793 3857 Acked Flush 0 Done Done Sent false
12794 3858 Acked Write 2 Done Done Sent false
12795 3859 Acked Read 6 Done Done Sent false
12796 3860 Acked Write 6 Done Done Sent false
12797 3861 Acked Write 9 Done Done Sent false
12798 3862 Acked Read 9 Done Done Sent false
12799 3863 Acked Read 6 Done Done Sent false
12800 3864 Acked Read 8 Done Done Sent false
12801 3865 Acked Flush 0 Done Done Sent false
12802 3866 Acked Write 4 Done Done Sent false
12803 3867 Acked Write 6 Done Done Sent false
12804 3868 Acked Read 6 Done Done Sent false
12805 3869 Acked Write 7 Done Done Sent false
12806 3870 Acked Read 7 Done Done Sent false
12807 3871 Acked Write 4 Done Done Sent false
12808 3872 Acked Write 10 Done Done Sent false
12809 3873 Acked Read 7 Done Done Sent false
12810 3874 Acked Read 8 Done Done Sent false
12811 3875 Acked Read 8 Done Done Sent false
12812 3876 Acked Write 3 Done Done Sent false
12813 3877 Acked Write 5 Done Done Sent false
12814 3878 Acked Flush 0 Done Done Sent false
12815 3879 Acked Read 7 Done Done Sent false
12816 3880 Acked Read 5 Done Done Sent false
12817 3881 Acked Read 3 Done Done Sent false
12818 3882 Acked Read 9 Done Done Sent false
12819 3883 Acked Write 9 Done Done Sent false
12820 3884 Acked Read 6 Done Done Sent false
12821 3885 Acked Read 5 Done Done Sent false
12822 3886 Acked Read 8 Done Done Sent false
12823 3887 Acked Read 3 Done Done Sent false
12824 3888 Acked Write 4 Done Done Sent false
12825 3889 Acked Read 6 Done Done Sent false
12826 3890 Acked Write 4 Done Done Sent false
12827 3891 Acked Read 4 Done Done Sent false
12828 3892 Acked Flush 0 Done Done Sent false
12829 3893 Acked Write 3 Done Done Sent false
12830 3894 Acked Read 5 Done Done Sent false
12831 3895 Acked Write 1 Done Done Sent false
12832 3896 Acked Read 9 Done Done Sent false
12833 3897 Acked Read 2 Done Done Sent false
12834 3898 Acked Read 10 Done Done Sent false
12835 3899 Acked Write 1 Done Done Sent false
12836 3900 Acked Write 7 Done Done Sent false
12837 3901 Acked Read 1 Done Done Sent false
12838 3902 Acked Write 9 Done Done Sent false
12839 3903 Acked Read 10 Done Done Sent false
12840 3904 Acked Read 6 Done Done Sent false
12841 3905 Acked Read 3 Done Done Sent false
12842 3906 Acked Write 4 Done Done Sent false
12843 3907 Acked Read 6 Done Done Sent false
12844 3908 Acked Read 9 Done Done Sent false
12845 3909 Acked Read 4 Done Done Sent false
12846 3910 Acked Write 10 Done Done Sent false
12847 3911 Acked Read 4 Done Done Sent false
12848 3912 Acked Read 3 Done Done Sent false
12849 3913 Acked Flush 0 Done Done Sent false
12850 3914 Acked Write 9 Done Done Sent false
12851 3915 Acked Read 8 Done Done Sent false
12852 3916 Acked Write 8 Done Done Sent false
12853 3917 Acked Read 1 Done Done Sent false
12854 3918 Acked Write 4 Done Done Sent false
12855 3919 Acked Read 5 Done Done Sent false
12856 3920 Acked Write 8 Done Done Sent false
12857 3921 Acked Write 1 Done Done Sent false
12858 3922 Acked Flush 0 Done Done Sent false
12859 3923 Acked Flush 0 Done Done Sent false
12860 3924 Acked Read 7 Done Done Sent false
12861 3925 Acked Write 2 Done Done Sent false
12862 3926 Acked Flush 0 Done Done Sent false
12863 3927 Acked Read 5 Done Done Sent false
12864 3928 Acked Write 8 Done Done Sent false
12865 3929 Acked Read 5 Done Done Sent false
12866 3930 Acked Read 10 Done Done Sent false
12867 3931 Acked Write 8 Done Done Sent false
12868 3932 Acked Read 4 Done Done Sent false
12869 3933 Acked Read 6 Done Done Sent false
12870 3934 Acked Read 3 Done Done Sent false
12871 3935 Acked Write 7 Done Done Sent false
12872 3936 Acked Read 5 Done Done Sent false
12873 3937 Acked Read 6 Done Done Sent false
12874 3938 Acked Write 8 Done Done Sent false
12875 3939 Acked Write 7 Done Done Sent false
12876 3940 Acked Write 8 Done Done Sent false
12877 3941 Acked Write 2 Done Done Sent false
12878 3942 Acked Read 5 Done Done Sent false
12879 3943 Acked Write 5 Done Done Sent false
12880 3944 Acked Write 9 Done Done Sent false
12881 3945 Acked Read 6 Done Done Sent false
12882 3946 Acked Read 4 Done Done Sent false
12883 3947 Acked Write 2 Done Done Sent false
12884 3948 Acked Read 5 Done Done Sent false
12885 3949 Acked Write 4 Done Done Sent false
12886 3950 Acked Read 6 Done Done Sent false
12887 3951 Acked Write 6 Done Done Sent false
12888 3952 Acked Read 2 Done Done Sent false
12889 3953 Acked Flush 0 Done Done Sent false
12890 3954 Acked Flush 0 Done Done Sent false
12891 3955 Acked Read 4 Done Done Sent false
12892 3956 Acked Read 2 Done Done Sent false
12893 3957 Acked Write 1 Done Done Sent false
12894 3958 Acked Write 1 Done Done Sent false
12895 3959 Acked Write 3 Done Done Sent false
12896 3960 Acked Read 1 Done Done Sent false
12897 3961 Acked Write 9 Done Done Sent false
12898 3962 Acked Write 3 Done Done Sent false
12899 3963 Acked Write 10 Done Done Sent false
12900 3964 Acked Flush 0 Done Done Sent false
12901 3965 Acked Read 6 Done Done Sent false
12902 3966 Acked Read 3 Done Done Sent false
12903 3967 Acked Write 7 Done Done Sent false
12904 3968 Acked Read 10 Done Done Sent false
12905 3969 Acked Write 1 Done Done Sent false
12906 3970 Acked Read 6 Done Done Sent false
12907 3971 Acked Flush 0 Done Done Sent false
12908 3972 Acked Write 5 Done Done Sent false
12909 3973 Acked Flush 0 Done Done Sent false
12910 3974 Acked Read 8 Done Done Sent false
12911 3975 Acked Write 6 Done Done Sent false
12912 3976 Acked Read 4 Done Done Sent false
12913 3977 Acked Read 5 Done Done Sent false
12914 3978 Acked Write 10 Done Done Sent false
12915 3979 Acked Write 8 Done Done Sent false
12916 3980 Acked Write 8 Done Done Sent false
12917 3981 Acked Read 2 Done Done Sent false
12918 3982 Acked Write 6 Done Done Sent false
12919 3983 Acked Read 6 Done Done Sent false
12920 3984 Acked Write 8 Done Done Sent false
12921 3985 Acked Flush 0 Done Done Sent false
12922 3986 Acked Flush 0 Done Done Sent false
12923 3987 Acked Read 3 Done Done Sent false
12924 3988 Acked Read 6 Done Done Sent false
12925 3989 Acked Read 2 Done Done Sent false
12926 3990 Acked Read 7 Done Done Sent false
12927 3991 Acked Read 10 Done Done Sent false
12928 3992 Acked Read 3 Done Done Sent false
12929 3993 Acked Flush 0 Done Done Sent false
12930 3994 Acked Write 10 Done Done Sent false
12931 3995 Acked Read 9 Done Done Sent false
12932 3996 Acked Read 6 Done Done Sent false
12933 3997 Acked Read 8 Done Done Sent false
12934 3998 Acked Read 4 Done Done Sent false
12935 3999 Acked Flush 0 Done Done Sent false
12936 4000 Acked Read 9 Done Done Sent false
12937 4001 Acked Read 2 Done Done Sent false
12938 4002 Acked Read 6 Done Done Sent false
12939 4003 Acked Read 5 Done Done Sent false
12940 4004 Acked Flush 0 Done Done Sent false
12941 4005 Acked Write 10 Done Done Sent false
12942 4006 Acked Read 6 Done Done Sent false
12943 4007 Acked Read 7 Done Done Sent false
12944 4008 Acked Write 4 Done Done Sent false
12945 4009 Acked Write 3 Done Done Sent false
12946 4010 Acked Write 1 Done Done Sent false
12947 4011 Acked Flush 0 Done Done Sent false
12948 4012 Acked Write 6 Done Done Sent false
12949 4013 Acked Read 8 Done Done Sent false
12950 4014 Acked Read 10 Done Done Sent false
12951 4015 Acked Read 10 Done Done Sent false
12952 4016 Acked Read 9 Done Done Sent false
12953 4017 Acked Write 8 Done Done Sent false
12954 4018 Acked Read 7 Done Done Sent false
12955 STATES DS:0 DS:1 DS:2 TOTAL
12956 Sent 0 0 301 301
12957 Done 301 301 0 602
12958 Skipped 0 0 0 0
12959 Error 0 0 0 0
12960 Last Flush: Some(JobId(4011)) Some(JobId(4011)) Some(JobId(3717))
12961 Downstairs last five completed: 3717 3716 3715 3714 3713
12962 Upstairs last five completed: 4018 4017 4016 4015 4014
12963 CLIENT: Up:0 ds:301 act:2
129642025-02-07T20:10:12.505ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
129652025-02-07T20:10:13.225ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
129662025-02-07T20:10:14.227ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
129672025-02-07T20:10:15.228ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
12968 [8830] got start action
12969 [8830] I am going to restart
12970 Starting downstairs at port 8830
12971 Make output file at "/tmp/dsc/downstairs-8830.txt"
12972 [8830][2] reports Starting
12973 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1219)
12974 [8830][2] reports Running
12975 ----------------------------------------------------------------
12976 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12977 States: Active Active Offline
12978 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12979 3718 Acked Flush 0 Done Done Sent false
12980 3719 Acked Read 4 Done Done Sent false
12981 3720 Acked Read 2 Done Done Sent false
12982 3721 Acked Read 6 Done Done Sent false
12983 3722 Acked Write 8 Done Done Sent false
12984 3723 Acked Read 5 Done Done Sent false
12985 3724 Acked Flush 0 Done Done Sent false
12986 3725 Acked Read 7 Done Done Sent false
12987 3726 Acked Write 8 Done Done Sent false
12988 3727 Acked Read 10 Done Done Sent false
12989 3728 Acked Read 8 Done Done Sent false
12990 3729 Acked Read 1 Done Done Sent false
12991 3730 Acked Read 9 Done Done Sent false
12992 3731 Acked Read 8 Done Done Sent false
12993 3732 Acked Read 6 Done Done Sent false
12994 3733 Acked Write 9 Done Done Sent false
12995 3734 Acked Read 7 Done Done Sent false
12996 3735 Acked Read 6 Done Done Sent false
12997 3736 Acked Read 4 Done Done Sent false
12998 3737 Acked Read 5 Done Done Sent false
12999 3738 Acked Write 7 Done Done Sent false
13000 3739 Acked Read 10 Done Done Sent false
13001 3740 Acked Read 8 Done Done Sent false
13002 3741 Acked Read 4 Done Done Sent false
13003 3742 Acked Write 6 Done Done Sent false
13004 3743 Acked Write 5 Done Done Sent false
13005 3744 Acked Read 7 Done Done Sent false
13006 3745 Acked Write 3 Done Done Sent false
13007 3746 Acked Read 3 Done Done Sent false
13008 3747 Acked Write 8 Done Done Sent false
13009 3748 Acked Read 3 Done Done Sent false
13010 3749 Acked Read 4 Done Done Sent false
13011 3750 Acked Write 7 Done Done Sent false
13012 3751 Acked Read 10 Done Done Sent false
13013 3752 Acked Read 10 Done Done Sent false
13014 3753 Acked Read 5 Done Done Sent false
13015 3754 Acked Read 9 Done Done Sent false
13016 3755 Acked Read 6 Done Done Sent false
13017 3756 Acked Write 3 Done Done Sent false
13018 3757 Acked Write 6 Done Done Sent false
13019 3758 Acked Read 1 Done Done Sent false
13020 3759 Acked Write 1 Done Done Sent false
13021 3760 Acked Write 4 Done Done Sent false
13022 3761 Acked Write 7 Done Done Sent false
13023 3762 Acked Read 5 Done Done Sent false
13024 3763 Acked Read 2 Done Done Sent false
13025 3764 Acked Write 8 Done Done Sent false
13026 3765 Acked Write 2 Done Done Sent false
13027 3766 Acked Read 1 Done Done Sent false
13028 3767 Acked Read 7 Done Done Sent false
13029 3768 Acked Write 3 Done Done Sent false
13030 3769 Acked Read 1 Done Done Sent false
13031 3770 Acked Read 2 Done Done Sent false
13032 3771 Acked Flush 0 Done Done Sent false
13033 3772 Acked Flush 0 Done Done Sent false
13034 3773 Acked Write 3 Done Done Sent false
13035 3774 Acked Flush 0 Done Done Sent false
13036 3775 Acked Write 5 Done Done Sent false
13037 3776 Acked Read 4 Done Done Sent false
13038 3777 Acked Read 1 Done Done Sent false
13039 3778 Acked Read 3 Done Done Sent false
13040 3779 Acked Write 7 Done Done Sent false
13041 3780 Acked Write 10 Done Done Sent false
13042 3781 Acked Read 4 Done Done Sent false
13043 3782 Acked Write 8 Done Done Sent false
13044 3783 Acked Read 4 Done Done Sent false
13045 3784 Acked Read 4 Done Done Sent false
13046 3785 Acked Write 7 Done Done Sent false
13047 3786 Acked Flush 0 Done Done Sent false
13048 3787 Acked Write 2 Done Done Sent false
13049 3788 Acked Write 9 Done Done Sent false
13050 3789 Acked Read 8 Done Done Sent false
13051 3790 Acked Write 5 Done Done Sent false
13052 3791 Acked Flush 0 Done Done Sent false
13053 3792 Acked Write 8 Done Done Sent false
13054 3793 Acked Read 4 Done Done Sent false
13055 3794 Acked Write 4 Done Done Sent false
13056 3795 Acked Write 2 Done Done Sent false
13057 3796 Acked Flush 0 Done Done Sent false
13058 3797 Acked Write 5 Done Done Sent false
13059 3798 Acked Read 2 Done Done Sent false
13060 3799 Acked Read 7 Done Done Sent false
13061 3800 Acked Read 1 Done Done Sent false
13062 3801 Acked Read 10 Done Done Sent false
13063 3802 Acked Read 7 Done Done Sent false
13064 3803 Acked Read 6 Done Done Sent false
13065 3804 Acked Write 10 Done Done Sent false
13066 3805 Acked Read 2 Done Done Sent false
13067 3806 Acked Write 7 Done Done Sent false
13068 3807 Acked Write 8 Done Done Sent false
13069 3808 Acked Read 4 Done Done Sent false
13070 3809 Acked Read 2 Done Done Sent false
13071 3810 Acked Write 9 Done Done Sent false
13072 3811 Acked Write 6 Done Done Sent false
13073 3812 Acked Write 4 Done Done Sent false
13074 3813 Acked Write 10 Done Done Sent false
13075 3814 Acked Read 3 Done Done Sent false
13076 3815 Acked Read 5 Done Done Sent false
13077 3816 Acked Read 9 Done Done Sent false
13078 3817 Acked Read 1 Done Done Sent false
13079 3818 Acked Read 7 Done Done Sent false
13080 3819 Acked Read 6 Done Done Sent false
13081 3820 Acked Read 2 Done Done Sent false
13082 3821 Acked Read 1 Done Done Sent false
13083 3822 Acked Write 5 Done Done Sent false
13084 3823 Acked Read 8 Done Done Sent false
13085 3824 Acked Read 5 Done Done Sent false
13086 3825 Acked Write 1 Done Done Sent false
13087 3826 Acked Write 9 Done Done Sent false
13088 3827 Acked Write 8 Done Done Sent false
13089 3828 Acked Write 7 Done Done Sent false
13090 3829 Acked Read 8 Done Done Sent false
13091 3830 Acked Read 9 Done Done Sent false
13092 3831 Acked Write 4 Done Done Sent false
13093 3832 Acked Read 9 Done Done Sent false
13094 3833 Acked Read 6 Done Done Sent false
13095 3834 Acked Read 7 Done Done Sent false
13096 3835 Acked Read 3 Done Done Sent false
13097 3836 Acked Write 9 Done Done Sent false
13098 3837 Acked Write 9 Done Done Sent false
13099 3838 Acked Flush 0 Done Done Sent false
13100 3839 Acked Read 8 Done Done Sent false
13101 3840 Acked Write 1 Done Done Sent false
13102 3841 Acked Write 6 Done Done Sent false
13103 3842 Acked Write 4 Done Done Sent false
13104 3843 Acked Write 9 Done Done Sent false
13105 3844 Acked Write 4 Done Done Sent false
13106 3845 Acked Write 10 Done Done Sent false
13107 3846 Acked Write 7 Done Done Sent false
13108 3847 Acked Read 7 Done Done Sent false
13109 3848 Acked Read 9 Done Done Sent false
13110 3849 Acked Read 4 Done Done Sent false
13111 3850 Acked Write 5 Done Done Sent false
13112 3851 Acked Read 2 Done Done Sent false
13113 3852 Acked Write 8 Done Done Sent false
13114 3853 Acked Read 4 Done Done Sent false
13115 3854 Acked Read 7 Done Done Sent false
13116 3855 Acked Read 5 Done Done Sent false
13117 3856 Acked Flush 0 Done Done Sent false
13118 3857 Acked Flush 0 Done Done Sent false
13119 3858 Acked Write 2 Done Done Sent false
13120 3859 Acked Read 6 Done Done Sent false
13121 3860 Acked Write 6 Done Done Sent false
13122 3861 Acked Write 9 Done Done Sent false
13123 3862 Acked Read 9 Done Done Sent false
13124 3863 Acked Read 6 Done Done Sent false
13125 3864 Acked Read 8 Done Done Sent false
13126 3865 Acked Flush 0 Done Done Sent false
13127 3866 Acked Write 4 Done Done Sent false
13128 3867 Acked Write 6 Done Done Sent false
13129 3868 Acked Read 6 Done Done Sent false
13130 3869 Acked Write 7 Done Done Sent false
13131 3870 Acked Read 7 Done Done Sent false
13132 3871 Acked Write 4 Done Done Sent false
13133 3872 Acked Write 10 Done Done Sent false
13134 3873 Acked Read 7 Done Done Sent false
13135 3874 Acked Read 8 Done Done Sent false
13136 3875 Acked Read 8 Done Done Sent false
13137 3876 Acked Write 3 Done Done Sent false
13138 3877 Acked Write 5 Done Done Sent false
13139 3878 Acked Flush 0 Done Done Sent false
13140 3879 Acked Read 7 Done Done Sent false
13141 3880 Acked Read 5 Done Done Sent false
13142 3881 Acked Read 3 Done Done Sent false
13143 3882 Acked Read 9 Done Done Sent false
13144 3883 Acked Write 9 Done Done Sent false
13145 3884 Acked Read 6 Done Done Sent false
13146 3885 Acked Read 5 Done Done Sent false
13147 3886 Acked Read 8 Done Done Sent false
13148 3887 Acked Read 3 Done Done Sent false
13149 3888 Acked Write 4 Done Done Sent false
13150 3889 Acked Read 6 Done Done Sent false
13151 3890 Acked Write 4 Done Done Sent false
13152 3891 Acked Read 4 Done Done Sent false
13153 3892 Acked Flush 0 Done Done Sent false
13154 3893 Acked Write 3 Done Done Sent false
13155 3894 Acked Read 5 Done Done Sent false
13156 3895 Acked Write 1 Done Done Sent false
13157 3896 Acked Read 9 Done Done Sent false
13158 3897 Acked Read 2 Done Done Sent false
13159 3898 Acked Read 10 Done Done Sent false
13160 3899 Acked Write 1 Done Done Sent false
13161 3900 Acked Write 7 Done Done Sent false
13162 3901 Acked Read 1 Done Done Sent false
13163 3902 Acked Write 9 Done Done Sent false
13164 3903 Acked Read 10 Done Done Sent false
13165 3904 Acked Read 6 Done Done Sent false
13166 3905 Acked Read 3 Done Done Sent false
13167 3906 Acked Write 4 Done Done Sent false
13168 3907 Acked Read 6 Done Done Sent false
13169 3908 Acked Read 9 Done Done Sent false
13170 3909 Acked Read 4 Done Done Sent false
13171 3910 Acked Write 10 Done Done Sent false
13172 3911 Acked Read 4 Done Done Sent false
13173 3912 Acked Read 3 Done Done Sent false
13174 3913 Acked Flush 0 Done Done Sent false
13175 3914 Acked Write 9 Done Done Sent false
13176 3915 Acked Read 8 Done Done Sent false
13177 3916 Acked Write 8 Done Done Sent false
13178 3917 Acked Read 1 Done Done Sent false
13179 3918 Acked Write 4 Done Done Sent false
13180 3919 Acked Read 5 Done Done Sent false
13181 3920 Acked Write 8 Done Done Sent false
13182 3921 Acked Write 1 Done Done Sent false
13183 3922 Acked Flush 0 Done Done Sent false
13184 3923 Acked Flush 0 Done Done Sent false
13185 3924 Acked Read 7 Done Done Sent false
13186 3925 Acked Write 2 Done Done Sent false
13187 3926 Acked Flush 0 Done Done Sent false
13188 3927 Acked Read 5 Done Done Sent false
13189 3928 Acked Write 8 Done Done Sent false
13190 3929 Acked Read 5 Done Done Sent false
13191 3930 Acked Read 10 Done Done Sent false
13192 3931 Acked Write 8 Done Done Sent false
13193 3932 Acked Read 4 Done Done Sent false
13194 3933 Acked Read 6 Done Done Sent false
13195 3934 Acked Read 3 Done Done Sent false
13196 3935 Acked Write 7 Done Done Sent false
13197 3936 Acked Read 5 Done Done Sent false
13198 3937 Acked Read 6 Done Done Sent false
13199 3938 Acked Write 8 Done Done Sent false
13200 3939 Acked Write 7 Done Done Sent false
13201 3940 Acked Write 8 Done Done Sent false
13202 3941 Acked Write 2 Done Done Sent false
13203 3942 Acked Read 5 Done Done Sent false
13204 3943 Acked Write 5 Done Done Sent false
13205 3944 Acked Write 9 Done Done Sent false
13206 3945 Acked Read 6 Done Done Sent false
13207 3946 Acked Read 4 Done Done Sent false
13208 3947 Acked Write 2 Done Done Sent false
13209 3948 Acked Read 5 Done Done Sent false
13210 3949 Acked Write 4 Done Done Sent false
13211 3950 Acked Read 6 Done Done Sent false
13212 3951 Acked Write 6 Done Done Sent false
13213 3952 Acked Read 2 Done Done Sent false
13214 3953 Acked Flush 0 Done Done Sent false
13215 3954 Acked Flush 0 Done Done Sent false
13216 3955 Acked Read 4 Done Done Sent false
13217 3956 Acked Read 2 Done Done Sent false
13218 3957 Acked Write 1 Done Done Sent false
13219 3958 Acked Write 1 Done Done Sent false
13220 3959 Acked Write 3 Done Done Sent false
13221 3960 Acked Read 1 Done Done Sent false
13222 3961 Acked Write 9 Done Done Sent false
13223 3962 Acked Write 3 Done Done Sent false
13224 3963 Acked Write 10 Done Done Sent false
13225 3964 Acked Flush 0 Done Done Sent false
13226 3965 Acked Read 6 Done Done Sent false
13227 3966 Acked Read 3 Done Done Sent false
13228 3967 Acked Write 7 Done Done Sent false
13229 3968 Acked Read 10 Done Done Sent false
13230 3969 Acked Write 1 Done Done Sent false
13231 3970 Acked Read 6 Done Done Sent false
13232 3971 Acked Flush 0 Done Done Sent false
13233 3972 Acked Write 5 Done Done Sent false
13234 3973 Acked Flush 0 Done Done Sent false
13235 3974 Acked Read 8 Done Done Sent false
13236 3975 Acked Write 6 Done Done Sent false
13237 3976 Acked Read 4 Done Done Sent false
13238 3977 Acked Read 5 Done Done Sent false
13239 3978 Acked Write 10 Done Done Sent false
13240 3979 Acked Write 8 Done Done Sent false
13241 3980 Acked Write 8 Done Done Sent false
13242 3981 Acked Read 2 Done Done Sent false
13243 3982 Acked Write 6 Done Done Sent false
13244 3983 Acked Read 6 Done Done Sent false
13245 3984 Acked Write 8 Done Done Sent false
13246 3985 Acked Flush 0 Done Done Sent false
13247 3986 Acked Flush 0 Done Done Sent false
13248 3987 Acked Read 3 Done Done Sent false
13249 3988 Acked Read 6 Done Done Sent false
13250 3989 Acked Read 2 Done Done Sent false
13251 3990 Acked Read 7 Done Done Sent false
13252 3991 Acked Read 10 Done Done Sent false
13253 3992 Acked Read 3 Done Done Sent false
13254 3993 Acked Flush 0 Done Done Sent false
13255 3994 Acked Write 10 Done Done Sent false
13256 3995 Acked Read 9 Done Done Sent false
13257 3996 Acked Read 6 Done Done Sent false
13258 3997 Acked Read 8 Done Done Sent false
13259 3998 Acked Read 4 Done Done Sent false
13260 3999 Acked Flush 0 Done Done Sent false
13261 4000 Acked Read 9 Done Done Sent false
13262 4001 Acked Read 2 Done Done Sent false
13263 4002 Acked Read 6 Done Done Sent false
13264 4003 Acked Read 5 Done Done Sent false
13265 4004 Acked Flush 0 Done Done Sent false
13266 4005 Acked Write 10 Done Done Sent false
13267 4006 Acked Read 6 Done Done Sent false
13268 4007 Acked Read 7 Done Done Sent false
13269 4008 Acked Write 4 Done Done Sent false
13270 4009 Acked Write 3 Done Done Sent false
13271 4010 Acked Write 1 Done Done Sent false
13272 4011 Acked Flush 0 Done Done Sent false
13273 4012 Acked Write 6 Done Done Sent false
13274 4013 Acked Read 8 Done Done Sent false
13275 4014 Acked Read 10 Done Done Sent false
13276 4015 Acked Read 10 Done Done Sent false
13277 4016 Acked Read 9 Done Done Sent false
13278 4017 Acked Write 8 Done Done Sent false
13279 4018 Acked Read 7 Done Done Sent false
13280 STATES DS:0 DS:1 DS:2 TOTAL
13281 Sent 0 0 301 301
13282 Done 301 301 0 602
13283 Skipped 0 0 0 0
13284 Error 0 0 0 0
13285 Last Flush: Some(JobId(4011)) Some(JobId(4011)) Some(JobId(3717))
13286 Downstairs last five completed: 3717 3716 3715 3714 3713
13287 Upstairs last five completed: 4018 4017 4016 4015 4014
13288 CLIENT: Up:0 ds:301 act:2
132892025-02-07T20:10:16.506ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
132902025-02-07T20:10:17.231ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
132912025-02-07T20:10:17.746ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
132922025-02-07T20:10:17.746ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:40748) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
132932025-02-07T20:10:17.747ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
132942025-02-07T20:10:17.748ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
132952025-02-07T20:10:17.748ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
132962025-02-07T20:10:17.748ZINFOcrucible: send last flush ID to this DS: Some(JobId(3717)) = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
132972025-02-07T20:10:17.748ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
132982025-02-07T20:10:17.748ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3717)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
132992025-02-07T20:10:17.748ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3717)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
13300 ----------------------------------------------------------------
13301 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13302 Downstairs last five completed: 4019 4018 4017 4016 4015
13303 Upstairs last five completed: 4019 4018 4017 4016 4015
13304 CLIENT: Up:0 ds:0 act:3
13305 Replay: All jobs finished
13306 Main task has work to do, go find it
13307 got dsc Stop(2)
13308 stop 2
13309 [8830] Got stop action so:false kr:false
13310 Feb 07 20:10:20.007 INFO request completed, latency_us: 345, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 111dd330-5a92-4333-a83a-39f8d7d59d8a, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
133112025-02-07T20:10:20.008ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
133122025-02-07T20:10:20.008ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
133132025-02-07T20:10:20.008ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
133142025-02-07T20:10:20.008ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
13315 Feb 07 20:10:20.008 INFO request completed, latency_us: 176, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2c0f7dec-7c80-4a3d-8609-c6eaf6fa2987, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
13316 [8830] Exited with: ExitStatus(unix_wait_status(9))
13317 [8830][2] reports Exit
13318 Feb 07 20:10:24.011 INFO request completed, latency_us: 221, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9ff882b8-860a-4d93-8352-eeaae3a4cc8e, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
13319 001/300 Read block 1072 len 20480
13320 002/300 Read block 1160 len 20480
13321 003/300 Write block 3664 len 20480 data: 6 6 7 5 5
13322 004/300 Read block 4776 len 28672
13323 005/300 Read block 3517 len 32768
13324 006/300 Write block 1894 len 40960 data: 4 4 4 4 4 4 4 3 3 2
13325 007/300 Read block 4387 len 40960
13326 008/300 Write block 613 len 8192 data: 4 4
13327 009/300 Write block 4618 len 36864 data: 4 4 6 4 5 4 4 3 3
13328 010/300 Write block 4636 len 16384 data: 3 3 3 3
13329 011/300 Read block 2998 len 16384
13330 012/300 Write block 4322 len 28672 data: 2 2 2 4 4 3 4
13331 013/300 Read block 2811 len 28672
13332 014/300 Flush
13333 015/300 Flush
13334 016/300 Read block 4650 len 12288
13335 017/300 Write block 810 len 28672 data: 4 4 4 6 4 4 4
13336 018/300 Write block 1187 len 4096 data: 4
13337 019/300 Write block 3314 len 36864 data: 3 3 3 3 3 3 3 3 3
13338 020/300 Read block 235 len 20480
13339 021/300 Read block 3559 len 16384
13340 022/300 Read block 3592 len 36864
13341 023/300 Read block 4086 len 12288
13342 024/300 Read block 4588 len 4096
13343 025/300 Write block 3908 len 16384 data: 3 2 2 3
13344 026/300 Read block 2281 len 20480
13345 027/300 Read block 2975 len 40960
13346 028/300 Read block 1296 len 24576
13347 029/300 Read block 2394 len 20480
13348 030/300 Read block 4118 len 40960
13349 031/300 Read block 2802 len 16384
13350 032/300 Write block 2266 len 40960 data: 4 5 4 4 4 4 4 4 4 4
13351 033/300 Write block 1195 len 4096 data: 4
13352 034/300 Read block 1047 len 8192
13353 035/300 Write block 1400 len 8192 data: 2 2
13354 036/300 Read block 4431 len 12288
13355 037/300 Read block 375 len 24576
13356 038/300 Write block 3696 len 40960 data: 6 5 4 4 3 4 4 4 3 3
13357 039/300 Read block 811 len 40960
13358 040/300 Write block 3881 len 12288 data: 2 2 3
13359 041/300 Read block 1854 len 36864
13360 042/300 Flush
13361 043/300 Read block 3568 len 8192
13362 044/300 Read block 876 len 12288
13363 045/300 Write block 450 len 36864 data: 4 3 3 3 3 4 3 4 4
13364 046/300 Write block 2273 len 32768 data: 5 5 5 3 3 3 3 2
13365 047/300 Write block 2020 len 40960 data: 4 3 3 3 3 3 4 4 5 4
13366 048/300 Flush
13367 049/300 Read block 4037 len 32768
13368 050/300 Write block 466 len 32768 data: 3 3 3 2 2 2 3 3
13369 051/300 Write block 3037 len 16384 data: 3 4 4 4
13370 052/300 Read block 4022 len 28672
13371 053/300 Write block 1715 len 20480 data: 3 3 3 4 3
13372 054/300 Read block 1852 len 28672
13373 055/300 Read block 79 len 28672
13374 056/300 Write block 3208 len 28672 data: 2 2 3 3 3 4 3
13375 057/300 Write block 4037 len 12288 data: 2 2 2
13376 058/300 Write block 3684 len 4096 data: 4
13377 059/300 Write block 1094 len 20480 data: 2 2 2 2 2
13378 060/300 Write block 3589 len 36864 data: 3 2 2 2 2 2 3 2 2
13379 061/300 Read block 1596 len 40960
13380 062/300 Write block 1657 len 28672 data: 4 6 5 4 3 3 2
13381 063/300 Write block 3576 len 24576 data: 4 4 4 3 3 3
13382 064/300 Flush
13383 065/300 Read block 2010 len 20480
13384 066/300 Write block 329 len 28672 data: 2 2 3 4 4 6 7
13385 067/300 Flush
13386 068/300 Read block 3054 len 4096
13387 069/300 Write block 2990 len 28672 data: 2 2 2 2 2 2 2
13388 070/300 Write block 4527 len 32768 data: 4 3 3 2 2 2 2 2
13389 071/300 Read block 1216 len 36864
13390 072/300 Write block 3041 len 16384 data: 4 4 4 5
13391 073/300 Read block 1156 len 40960
13392 074/300 Read block 1494 len 28672
13393 075/300 Read block 151 len 8192
13394 076/300 Read block 4922 len 36864
13395 077/300 Write block 4299 len 36864 data: 2 2 3 2 2 2 2 2 3
13396 078/300 Write block 630 len 8192 data: 4 4
13397 079/300 Read block 2985 len 8192
13398 080/300 Write block 2967 len 20480 data: 3 4 4 3 4
13399 081/300 Write block 236 len 20480 data: 4 2 2 3 3
13400 082/300 Flush
13401 083/300 Read block 2745 len 12288
13402 084/300 Read block 3803 len 16384
13403 085/300 Write block 1848 len 16384 data: 4 4 3 3
13404 086/300 Write block 57 len 20480 data: 2 2 2 2 2
13405 087/300 Flush
13406 088/300 Read block 4621 len 8192
13407 089/300 Write block 2376 len 12288 data: 4 5 5
13408 090/300 Flush
13409 091/300 Read block 3330 len 20480
13410 092/300 Read block 3516 len 32768
13411 093/300 Flush
13412 094/300 Read block 2494 len 12288
13413 095/300 Read block 2116 len 8192
13414 096/300 Write block 51 len 12288 data: 2 2 2
13415 097/300 Read block 1635 len 36864
13416 098/300 Read block 4472 len 36864
13417 099/300 Write block 2377 len 12288 data: 6 6 4
13418 100/300 Write block 43 len 36864 data: 3 3 3 2 2 2 2 2 3
13419 101/300 Write block 2198 len 36864 data: 3 3 3 3 3 5 5 5 5
13420 102/300 Read block 806 len 36864
13421 103/300 Read block 3985 len 24576
13422 104/300 Write block 4832 len 20480 data: 5 5 5 5 7
13423 105/300 Write block 2590 len 40960 data: 2 2 3 4 4 4 4 4 3 3
13424 106/300 Read block 1492 len 24576
13425 107/300 Read block 3857 len 12288
13426 108/300 Write block 1903 len 20480 data: 3 3 3 4 4
13427 109/300 Write block 4465 len 32768 data: 6 6 5 3 3 3 4 5
13428 110/300 Flush
13429 111/300 Write block 57 len 12288 data: 3 3 3
13430 112/300 Write block 334 len 28672 data: 7 8 6 6 5 5 5
13431 113/300 Write block 2810 len 36864 data: 3 2 2 3 3 4 4 3 3
13432 114/300 Write block 1917 len 4096 data: 2
13433 115/300 Write block 2152 len 20480 data: 2 2 2 4 5
13434 116/300 Write block 2447 len 28672 data: 4 4 4 3 3 4 3
13435 117/300 Write block 3296 len 20480 data: 2 2 4 4 5
13436 118/300 Read block 1302 len 32768
13437 119/300 Write block 2163 len 16384 data: 4 5 4 4
13438 120/300 Read block 4355 len 40960
13439 121/300 Write block 4506 len 32768 data: 5 6 6 7 7 8 6 6
13440 122/300 Write block 2591 len 4096 data: 3
13441 123/300 Read block 2223 len 40960
13442 124/300 Write block 4431 len 8192 data: 2 2
13443 125/300 Write block 4223 len 28672 data: 4 3 3 3 3 3 3
13444 126/300 Read block 3853 len 24576
13445 127/300 Read block 3316 len 24576
13446 128/300 Read block 3175 len 4096
13447 129/300 Flush
13448 130/300 Write block 1409 len 16384 data: 2 2 3 3
13449 131/300 Read block 1669 len 8192
13450 132/300 Read block 3619 len 12288
13451 133/300 Write block 3309 len 32768 data: 3 2 3 3 3 4 4 4
13452 134/300 Write block 3058 len 8192 data: 2 2
13453 135/300 Write block 4405 len 4096 data: 3
13454 136/300 Write block 3883 len 4096 data: 4
13455 137/300 Read block 3364 len 24576
13456 138/300 Read block 767 len 24576
13457 139/300 Write block 1222 len 24576 data: 4 6 6 6 5 5
13458 140/300 Flush
13459 141/300 Write block 3924 len 24576 data: 4 3 4 4 4 3
13460 142/300 Read block 2370 len 40960
13461 143/300 Read block 1686 len 40960
13462 144/300 Write block 603 len 4096 data: 2
13463 145/300 Write block 4167 len 20480 data: 4 4 4 3 3
13464 146/300 Read block 1316 len 12288
13465 147/300 Write block 319 len 4096 data: 3
13466 148/300 Write block 2681 len 40960 data: 3 3 3 3 3 3 3 3 4 4
13467 149/300 Write block 612 len 28672 data: 5 5 5 3 3 4 5
13468 150/300 Flush
13469 151/300 Read block 1276 len 24576
13470 152/300 Read block 2680 len 28672
13471 153/300 Read block 557 len 8192
13472 154/300 Read block 2742 len 40960
13473 155/300 Flush
13474 156/300 Read block 2160 len 28672
13475 157/300 Read block 4614 len 40960
13476 158/300 Write block 2464 len 8192 data: 3 3
13477 159/300 Read block 3487 len 20480
13478 160/300 Read block 1656 len 40960
13479 161/300 Read block 4318 len 4096
13480 162/300 Read block 1338 len 8192
13481 163/300 Write block 3777 len 4096 data: 5
13482 164/300 Read block 2419 len 36864
13483 165/300 Flush
13484 166/300 Write block 761 len 36864 data: 4 6 4 2 2 2 2 2 2
13485 167/300 Write block 2124 len 20480 data: 4 4 4 4 4
13486 168/300 Read block 655 len 40960
13487 169/300 Write block 3535 len 36864 data: 3 3 2 3 3 4 4 5 4
13488 170/300 Write block 1330 len 12288 data: 4 4 4
13489 171/300 Read block 2789 len 16384
13490 172/300 Read block 1333 len 40960
13491 173/300 Write block 3777 len 20480 data: 6 6 5 4 4
13492 174/300 Flush
13493 175/300 Read block 740 len 32768
13494 176/300 Read block 1907 len 8192
13495 177/300 Flush
13496 178/300 Write block 965 len 16384 data: 2 2 2 2
13497 179/300 Write block 2032 len 20480 data: 2 2 3 3 2
13498 180/300 Write block 2565 len 36864 data: 4 5 3 3 3 3 3 3 3
13499 181/300 Write block 570 len 24576 data: 2 2 2 2 2 2
13500 182/300 Read block 616 len 20480
13501 183/300 Read block 1824 len 16384
13502 184/300 Write block 2985 len 32768 data: 3 3 3 2 2 3 3 3
13503 185/300 Write block 2319 len 20480 data: 4 4 2 2 3
13504 186/300 Flush
13505 187/300 Write block 4596 len 32768 data: 3 3 3 2 2 2 3 3
13506 188/300 Read block 1996 len 40960
13507 189/300 Write block 2784 len 4096 data: 4
13508 190/300 Write block 4733 len 12288 data: 6 6 6
13509 191/300 Read block 733 len 12288
13510 192/300 Write block 4474 len 36864 data: 3 5 5 5 5 4 4 2 2
13511 193/300 Read block 4218 len 12288
13512 194/300 Read block 514 len 36864
13513 195/300 Write block 2999 len 24576 data: 4 4 5 5 5 6
13514 196/300 Read block 2548 len 16384
13515 197/300 Flush
13516 198/300 Write block 216 len 32768 data: 3 2 2 2 2 2 3 3
13517 199/300 Read block 1266 len 12288
13518 200/300 Read block 4185 len 32768
13519 201/300 Write block 2424 len 32768 data: 2 3 3 2 2 3 3 4
13520 202/300 Read block 432 len 12288
13521 203/300 Read block 1911 len 28672
13522 204/300 Read block 1502 len 36864
13523 205/300 Write block 4116 len 12288 data: 3 3 2
13524 206/300 Write block 1655 len 16384 data: 4 4 5 7
13525 207/300 Write block 4847 len 12288 data: 2 4 5
13526 208/300 Write block 310 len 28672 data: 4 6 6 5 5 5 4
13527 209/300 Read block 4794 len 4096
13528 210/300 Write block 4292 len 32768 data: 3 2 2 4 4 3 3 3
13529 211/300 Read block 902 len 4096
13530 212/300 Read block 1592 len 8192
13531 213/300 Write block 3384 len 32768 data: 4 3 2 2 2 2 2 2
13532 214/300 Write block 530 len 28672 data: 3 4 4 3 3 3 4
13533 215/300 Read block 444 len 16384
13534 216/300 Write block 3686 len 8192 data: 4 4
13535 217/300 Write block 246 len 32768 data: 2 2 2 2 2 2 3 3
13536 218/300 Write block 4176 len 20480 data: 3 3 3 3 2
13537 219/300 Read block 3823 len 4096
13538 220/300 Read block 4706 len 20480
13539 221/300 Read block 3690 len 28672
13540 222/300 Read block 2341 len 24576
13541 223/300 Read block 2310 len 28672
13542 224/300 Flush
13543 225/300 Read block 1874 len 20480
13544 226/300 Read block 3207 len 4096
13545 227/300 Flush
13546 228/300 Read block 3334 len 24576
13547 229/300 Flush
13548 230/300 Read block 1939 len 12288
13549 231/300 Write block 1419 len 28672 data: 3 3 4 3 2 2 2
13550 232/300 Read block 3785 len 16384
13551 233/300 Read block 208 len 32768
13552 234/300 Read block 1420 len 32768
13553 235/300 Read block 2301 len 4096
13554 236/300 Write block 704 len 20480 data: 3 3 4 4 3
13555 237/300 Read block 1268 len 16384
13556 238/300 Write block 2003 len 12288 data: 3 4 4
13557 239/300 Read block 751 len 32768
13558 240/300 Read block 3427 len 24576
13559 241/300 Write block 825 len 28672 data: 3 3 3 5 4 3 3
13560 242/300 Read block 4362 len 16384
13561 243/300 Read block 335 len 8192
13562 244/300 Write block 4029 len 28672 data: 4 4 4 4 4 3 3
13563 245/300 Read block 2734 len 32768
13564 246/300 Read block 1316 len 40960
13565 247/300 Read block 3302 len 20480
13566 248/300 Read block 1371 len 12288
13567 249/300 Read block 676 len 4096
13568 250/300 Read block 227 len 36864
13569 251/300 Write block 1072 len 16384 data: 5 4 4 4
13570 252/300 Write block 416 len 36864 data: 2 2 3 4 4 4 4 3 4
13571 253/300 Read block 3886 len 24576
13572 254/300 Write block 3047 len 24576 data: 3 4 5 5 5 7
13573 255/300 Write block 1397 len 24576 data: 3 3 2 3 3 2
13574 256/300 Read block 1454 len 20480
13575 257/300 Write block 528 len 28672 data: 5 4 4 5 5 4 4
13576 258/300 Flush
13577 259/300 Write block 986 len 28672 data: 3 4 4 4 4 3 4
13578 260/300 Read block 999 len 4096
13579 261/300 Write block 2742 len 28672 data: 3 3 4 4 3 3 3
13580 262/300 Read block 2664 len 8192
13581 263/300 Write block 1320 len 32768 data: 4 5 5 5 5 5 4 4
13582 264/300 Read block 3906 len 12288
13583 265/300 Write block 2333 len 36864 data: 2 2 3 2 2 3 3 3 5
13584 266/300 Read block 3713 len 32768
13585 267/300 Flush
13586 268/300 Read block 1830 len 16384
13587 269/300 Read block 1441 len 32768
13588 270/300 Flush
13589 271/300 Flush
13590 272/300 Read block 1081 len 24576
13591 273/300 Read block 560 len 28672
13592 274/300 Read block 4683 len 16384
13593 275/300 Read block 1888 len 8192
13594 276/300 Read block 3022 len 32768
13595 277/300 Read block 2697 len 8192
13596 278/300 Write block 1573 len 36864 data: 2 3 3 3 3 3 3 4 5
13597 279/300 Flush
13598 280/300 Write block 1207 len 16384 data: 3 3 3 3
13599 281/300 Write block 2006 len 20480 data: 4 4 4 4 3
13600 282/300 Read block 3570 len 24576
13601 283/300 Write block 4315 len 24576 data: 2 2 2 2 2 2
13602 284/300 Write block 4140 len 36864 data: 3 3 3 2 3 3 3 4 4
13603 285/300 Read block 2028 len 24576
13604 286/300 Read block 2249 len 28672
13605 287/300 Read block 4439 len 24576
13606 288/300 Read block 4817 len 36864
13607 289/300 Read block 2025 len 24576
13608 290/300 Read block 2986 len 16384
13609 291/300 Write block 3061 len 12288 data: 2 2 2
13610 292/300 Read block 1190 len 12288
13611 293/300 Read block 1917 len 36864
13612 294/300 Read block 3763 len 20480
13613 295/300 Read block 1945 len 28672
13614 296/300 Read block 442 len 40960
13615 297/300 Flush
13616 298/300 Read block 3088 len 32768
13617 299/300 Flush
13618 300/300 Read block 4604 len 4096
13619 301/300 Read block 1617 len 20480
13620 Main task has work to do, go find it
13621 got dsc Start(2)
13622 start 2
13623 Feb 07 20:10:24.257 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/2, method: POST, req_id: 7e76a6c1-6b93-48a0-bb0b-f49837852a0b, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
13624 Replay: started 2, returned:Ok(())
13625 ----------------------------------------------------------------
13626 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13627 States: Active Active Offline
13628 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13629 4020 Acked Read 5 Done Done Sent false
13630 4021 Acked Read 5 Done Done Sent false
13631 4022 Acked Write 5 Done Done Sent false
13632 4023 Acked Read 7 Done Done Sent false
13633 4024 Acked Read 8 Done Done Sent false
13634 4025 Acked Write 10 Done Done Sent false
13635 4026 Acked Read 10 Done Done Sent false
13636 4027 Acked Write 2 Done Done Sent false
13637 4028 Acked Write 9 Done Done Sent false
13638 4029 Acked Write 4 Done Done Sent false
13639 4030 Acked Read 4 Done Done Sent false
13640 4031 Acked Write 7 Done Done Sent false
13641 4032 Acked Read 7 Done Done Sent false
13642 4033 Acked Flush 0 Done Done Sent false
13643 4034 Acked Flush 0 Done Done Sent false
13644 4035 Acked Read 3 Done Done Sent false
13645 4036 Acked Write 7 Done Done Sent false
13646 4037 Acked Write 1 Done Done Sent false
13647 4038 Acked Write 9 Done Done Sent false
13648 4039 Acked Read 5 Done Done Sent false
13649 4040 Acked Read 4 Done Done Sent false
13650 4041 Acked Read 9 Done Done Sent false
13651 4042 Acked Read 3 Done Done Sent false
13652 4043 Acked Read 1 Done Done Sent false
13653 4044 Acked Write 4 Done Done Sent false
13654 4045 Acked Read 5 Done Done Sent false
13655 4046 Acked Read 10 Done Done Sent false
13656 4047 Acked Read 6 Done Done Sent false
13657 4048 Acked Read 5 Done Done Sent false
13658 4049 Acked Read 10 Done Done Sent false
13659 4050 Acked Read 4 Done Done Sent false
13660 4051 Acked Write 10 Done Done Sent false
13661 4052 Acked Write 1 Done Done Sent false
13662 4053 Acked Read 2 Done Done Sent false
13663 4054 Acked Write 2 Done Done Sent false
13664 4055 Acked Read 3 Done Done Sent false
13665 4056 Acked Read 6 Done Done Sent false
13666 4057 Acked Write 10 Done Done Sent false
13667 4058 Acked Read 10 Done Done Sent false
13668 4059 Acked Write 3 Done Done Sent false
13669 4060 Acked Read 9 Done Done Sent false
13670 4061 Acked Flush 0 Done Done Sent false
13671 4062 Acked Read 2 Done Done Sent false
13672 4063 Acked Read 3 Done Done Sent false
13673 4064 Acked Write 9 Done Done Sent false
13674 4065 Acked Write 8 Done Done Sent false
13675 4066 Acked Write 10 Done Done Sent false
13676 4067 Acked Flush 0 Done Done Sent false
13677 4068 Acked Read 8 Done Done Sent false
13678 4069 Acked Write 8 Done Done Sent false
13679 4070 Acked Write 4 Done Done Sent false
13680 4071 Acked Read 7 Done Done Sent false
13681 4072 Acked Write 5 Done Done Sent false
13682 4073 Acked Read 7 Done Done Sent false
13683 4074 Acked Read 7 Done Done Sent false
13684 4075 Acked Write 7 Done Done Sent false
13685 4076 Acked Write 3 Done Done Sent false
13686 4077 Acked Write 1 Done Done Sent false
13687 4078 Acked Write 5 Done Done Sent false
13688 4079 Acked Write 9 Done Done Sent false
13689 4080 Acked Read 10 Done Done Sent false
13690 4081 Acked Write 7 Done Done Sent false
13691 4082 Acked Write 6 Done Done Sent false
13692 4083 Acked Flush 0 Done Done Sent false
13693 4084 Acked Read 5 Done Done Sent false
13694 4085 Acked Write 7 Done Done Sent false
13695 4086 Acked Flush 0 Done Done Sent false
13696 4087 Acked Read 1 Done Done Sent false
13697 4088 Acked Write 7 Done Done Sent false
13698 4089 Acked Write 8 Done Done Sent false
13699 4090 Acked Read 9 Done Done Sent false
13700 4091 Acked Write 4 Done Done Sent false
13701 4092 Acked Read 10 Done Done Sent false
13702 4093 Acked Read 7 Done Done Sent false
13703 4094 Acked Read 2 Done Done Sent false
13704 4095 Acked Read 9 Done Done Sent false
13705 4096 Acked Write 9 Done Done Sent false
13706 4097 Acked Write 2 Done Done Sent false
13707 4098 Acked Read 2 Done Done Sent false
13708 4099 Acked Write 5 Done Done Sent false
13709 4100 Acked Write 5 Done Done Sent false
13710 4101 Acked Flush 0 Done Done Sent false
13711 4102 Acked Read 3 Done Done Sent false
13712 4103 Acked Read 4 Done Done Sent false
13713 4104 Acked Write 4 Done Done Sent false
13714 4105 Acked Write 5 Done Done Sent false
13715 4106 Acked Flush 0 Done Done Sent false
13716 4107 Acked Read 2 Done Done Sent false
13717 4108 Acked Write 3 Done Done Sent false
13718 4109 Acked Flush 0 Done Done Sent false
13719 4110 Acked Read 5 Done Done Sent false
13720 4111 Acked Read 8 Done Done Sent false
13721 4112 Acked Flush 0 Done Done Sent false
13722 4113 Acked Read 3 Done Done Sent false
13723 4114 Acked Read 2 Done Done Sent false
13724 4115 Acked Write 3 Done Done Sent false
13725 4116 Acked Read 9 Done Done Sent false
13726 4117 Acked Read 9 Done Done Sent false
13727 4118 Acked Write 3 Done Done Sent false
13728 4119 Acked Write 9 Done Done Sent false
13729 4120 Acked Write 9 Done Done Sent false
13730 4121 Acked Read 9 Done Done Sent false
13731 4122 Acked Read 6 Done Done Sent false
13732 4123 Acked Write 5 Done Done Sent false
13733 4124 Acked Write 10 Done Done Sent false
13734 4125 Acked Read 6 Done Done Sent false
13735 4126 Acked Read 3 Done Done Sent false
13736 4127 Acked Write 5 Done Done Sent false
13737 4128 Acked Write 8 Done Done Sent false
13738 4129 Acked Flush 0 Done Done Sent false
13739 4130 Acked Write 3 Done Done Sent false
13740 4131 Acked Write 7 Done Done Sent false
13741 4132 Acked Write 9 Done Done Sent false
13742 4133 Acked Write 1 Done Done Sent false
13743 4134 Acked Write 5 Done Done Sent false
13744 4135 Acked Write 7 Done Done Sent false
13745 4136 Acked Write 5 Done Done Sent false
13746 4137 Acked Read 8 Done Done Sent false
13747 4138 Acked Write 4 Done Done Sent false
13748 4139 Acked Read 10 Done Done Sent false
13749 4140 Acked Write 8 Done Done Sent false
13750 4141 Acked Write 1 Done Done Sent false
13751 4142 Acked Read 10 Done Done Sent false
13752 4143 Acked Write 2 Done Done Sent false
13753 4144 Acked Write 7 Done Done Sent false
13754 4145 Acked Read 6 Done Done Sent false
13755 4146 Acked Read 6 Done Done Sent false
13756 4147 Acked Read 1 Done Done Sent false
13757 4148 Acked Flush 0 Done Done Sent false
13758 4149 Acked Write 4 Done Done Sent false
13759 4150 Acked Read 2 Done Done Sent false
13760 4151 Acked Read 3 Done Done Sent false
13761 4152 Acked Write 8 Done Done Sent false
13762 4153 Acked Write 2 Done Done Sent false
13763 4154 Acked Write 1 Done Done Sent false
13764 4155 Acked Write 1 Done Done Sent false
13765 4156 Acked Read 6 Done Done Sent false
13766 4157 Acked Read 6 Done Done Sent false
13767 4158 Acked Write 6 Done Done Sent false
13768 4159 Acked Flush 0 Done Done Sent false
13769 4160 Acked Write 6 Done Done Sent false
13770 4161 Acked Read 10 Done Done Sent false
13771 4162 Acked Read 10 Done Done Sent false
13772 4163 Acked Write 1 Done Done Sent false
13773 4164 Acked Write 5 Done Done Sent false
13774 4165 Acked Read 3 Done Done Sent false
13775 4166 Acked Write 1 Done Done Sent false
13776 4167 Acked Write 10 Done Done Sent false
13777 4168 Acked Write 7 Done Done Sent false
13778 4169 Acked Flush 0 Done Done Sent false
13779 4170 Acked Read 6 Done Done Sent false
13780 4171 Acked Read 7 Done Done Sent false
13781 4172 Acked Read 2 Done Done Sent false
13782 4173 Acked Read 10 Done Done Sent false
13783 4174 Acked Flush 0 Done Done Sent false
13784 4175 Acked Read 7 Done Done Sent false
13785 4176 Acked Read 10 Done Done Sent false
13786 4177 Acked Write 2 Done Done Sent false
13787 4178 Acked Read 5 Done Done Sent false
13788 4179 Acked Read 10 Done Done Sent false
13789 4180 Acked Read 1 Done Done Sent false
13790 4181 Acked Read 2 Done Done Sent false
13791 4182 Acked Write 1 Done Done Sent false
13792 4183 Acked Read 9 Done Done Sent false
13793 4184 Acked Flush 0 Done Done Sent false
13794 4185 Acked Write 9 Done Done Sent false
13795 4186 Acked Write 5 Done Done Sent false
13796 4187 Acked Read 10 Done Done Sent false
13797 4188 Acked Write 9 Done Done Sent false
13798 4189 Acked Write 3 Done Done Sent false
13799 4190 Acked Read 4 Done Done Sent false
13800 4191 Acked Read 10 Done Done Sent false
13801 4192 Acked Write 5 Done Done Sent false
13802 4193 Acked Flush 0 Done Done Sent false
13803 4194 Acked Read 8 Done Done Sent false
13804 4195 Acked Read 2 Done Done Sent false
13805 4196 Acked Flush 0 Done Done Sent false
13806 4197 Acked Write 4 Done Done Sent false
13807 4198 Acked Write 5 Done Done Sent false
13808 4199 Acked Write 9 Done Done Sent false
13809 4200 Acked Write 6 Done Done Sent false
13810 4201 Acked Read 5 Done Done Sent false
13811 4202 Acked Read 4 Done Done Sent false
13812 4203 Acked Write 8 Done Done Sent false
13813 4204 Acked Write 5 Done Done Sent false
13814 4205 Acked Flush 0 Done Done Sent false
13815 4206 Acked Write 8 Done Done Sent false
13816 4207 Acked Read 10 Done Done Sent false
13817 4208 Acked Write 1 Done Done Sent false
13818 4209 Acked Write 3 Done Done Sent false
13819 4210 Acked Read 3 Done Done Sent false
13820 4211 Acked Write 9 Done Done Sent false
13821 4212 Acked Read 3 Done Done Sent false
13822 4213 Acked Read 9 Done Done Sent false
13823 4214 Acked Write 6 Done Done Sent false
13824 4215 Acked Read 4 Done Done Sent false
13825 4216 Acked Flush 0 Done Done Sent false
13826 4217 Acked Write 8 Done Done Sent false
13827 4218 Acked Read 3 Done Done Sent false
13828 4219 Acked Read 8 Done Done Sent false
13829 4220 Acked Write 8 Done Done Sent false
13830 4221 Acked Read 3 Done Done Sent false
13831 4222 Acked Read 7 Done Done Sent false
13832 4223 Acked Read 9 Done Done Sent false
13833 4224 Acked Write 3 Done Done Sent false
13834 4225 Acked Write 4 Done Done Sent false
13835 4226 Acked Write 3 Done Done Sent false
13836 4227 Acked Write 7 Done Done Sent false
13837 4228 Acked Read 1 Done Done Sent false
13838 4229 Acked Write 8 Done Done Sent false
13839 4230 Acked Read 1 Done Done Sent false
13840 4231 Acked Read 2 Done Done Sent false
13841 4232 Acked Write 8 Done Done Sent false
13842 4233 Acked Write 7 Done Done Sent false
13843 4234 Acked Read 4 Done Done Sent false
13844 4235 Acked Write 2 Done Done Sent false
13845 4236 Acked Write 8 Done Done Sent false
13846 4237 Acked Write 5 Done Done Sent false
13847 4238 Acked Read 1 Done Done Sent false
13848 4239 Acked Read 5 Done Done Sent false
13849 4240 Acked Read 7 Done Done Sent false
13850 4241 Acked Read 6 Done Done Sent false
13851 4242 Acked Read 7 Done Done Sent false
13852 4243 Acked Flush 0 Done Done Sent false
13853 4244 Acked Read 5 Done Done Sent false
13854 4245 Acked Read 1 Done Done Sent false
13855 4246 Acked Flush 0 Done Done Sent false
13856 4247 Acked Read 6 Done Done Sent false
13857 4248 Acked Flush 0 Done Done Sent false
13858 4249 Acked Read 3 Done Done Sent false
13859 4250 Acked Write 7 Done Done Sent false
13860 4251 Acked Read 4 Done Done Sent false
13861 4252 Acked Read 8 Done Done Sent false
13862 4253 Acked Read 8 Done Done Sent false
13863 4254 Acked Read 1 Done Done Sent false
13864 4255 Acked Write 5 Done Done Sent false
13865 4256 Acked Read 4 Done Done Sent false
13866 4257 Acked Write 3 Done Done Sent false
13867 4258 Acked Read 8 Done Done Sent false
13868 4259 Acked Read 6 Done Done Sent false
13869 4260 Acked Write 7 Done Done Sent false
13870 4261 Acked Read 4 Done Done Sent false
13871 4262 Acked Read 2 Done Done Sent false
13872 4263 Acked Write 7 Done Done Sent false
13873 4264 Acked Read 8 Done Done Sent false
13874 4265 Acked Read 10 Done Done Sent false
13875 4266 Acked Read 5 Done Done Sent false
13876 4267 Acked Read 3 Done Done Sent false
13877 4268 Acked Read 1 Done Done Sent false
13878 4269 Acked Read 9 Done Done Sent false
13879 4270 Acked Write 4 Done Done Sent false
13880 4271 Acked Write 9 Done Done Sent false
13881 4272 Acked Read 6 Done Done Sent false
13882 4273 Acked Write 6 Done Done Sent false
13883 4274 Acked Write 6 Done Done Sent false
13884 4275 Acked Read 5 Done Done Sent false
13885 4276 Acked Write 7 Done Done Sent false
13886 4277 Acked Flush 0 Done Done Sent false
13887 4278 Acked Write 7 Done Done Sent false
13888 4279 Acked Read 1 Done Done Sent false
13889 4280 Acked Write 7 Done Done Sent false
13890 4281 Acked Read 2 Done Done Sent false
13891 4282 Acked Write 8 Done Done Sent false
13892 4283 Acked Read 3 Done Done Sent false
13893 4284 Acked Write 9 Done Done Sent false
13894 4285 Acked Read 8 Done Done Sent false
13895 4286 Acked Flush 0 Done Done Sent false
13896 4287 Acked Read 4 Done Done Sent false
13897 4288 Acked Read 8 Done Done Sent false
13898 4289 Acked Flush 0 Done Done Sent false
13899 4290 Acked Flush 0 Done Done Sent false
13900 4291 Acked Read 6 Done Done Sent false
13901 4292 Acked Read 7 Done Done Sent false
13902 4293 Acked Read 4 Done Done Sent false
13903 4294 Acked Read 2 Done Done Sent false
13904 4295 Acked Read 8 Done Done Sent false
13905 4296 Acked Read 2 Done Done Sent false
13906 4297 Acked Write 9 Done Done Sent false
13907 4298 Acked Flush 0 Done Done Sent false
13908 4299 Acked Write 4 Done Done Sent false
13909 4300 Acked Write 5 Done Done Sent false
13910 4301 Acked Read 6 Done Done Sent false
13911 4302 Acked Write 6 Done Done Sent false
13912 4303 Acked Write 9 Done Done Sent false
13913 4304 Acked Read 6 Done Done Sent false
13914 4305 Acked Read 7 Done Done Sent false
13915 4306 Acked Read 6 Done Done Sent false
13916 4307 Acked Read 9 Done Done Sent false
13917 4308 Acked Read 6 Done Done Sent false
13918 4309 Acked Read 4 Done Done Sent false
13919 4310 Acked Write 3 Done Done Sent false
13920 4311 Acked Read 3 Done Done Sent false
13921 4312 Acked Read 9 Done Done Sent false
13922 4313 Acked Read 5 Done Done Sent false
13923 4314 Acked Read 7 Done Done Sent false
13924 4315 Acked Read 10 Done Done Sent false
13925 4316 Acked Flush 0 Done Done Sent false
13926 4317 Acked Read 8 Done Done Sent false
13927 4318 Acked Flush 0 Done Done Sent false
13928 4319 Acked Read 1 Done Done Sent false
13929 4320 Acked Read 5 Done Done Sent false
13930 STATES DS:0 DS:1 DS:2 TOTAL
13931 Sent 0 0 301 301
13932 Done 301 301 0 602
13933 Skipped 0 0 0 0
13934 Error 0 0 0 0
13935 Last Flush: Some(JobId(4318)) Some(JobId(4318)) Some(JobId(4019))
13936 Downstairs last five completed: 4019 4018 4017 4016 4015
13937 Upstairs last five completed: 4320 4319 4318 4317 4316
13938 CLIENT: Up:0 ds:301 act:2
139392025-02-07T20:10:25.246ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
13940 [8830] got start action
13941 [8830] I am going to restart
13942 Starting downstairs at port 8830
13943 Make output file at "/tmp/dsc/downstairs-8830.txt"
13944 [8830][2] reports Starting
13945 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1220)
13946 [8830][2] reports Running
139472025-02-07T20:10:26.248ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
139482025-02-07T20:10:27.250ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
139492025-02-07T20:10:28.251ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
13950 ----------------------------------------------------------------
13951 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13952 States: Active Active Offline
13953 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13954 4020 Acked Read 5 Done Done Sent false
13955 4021 Acked Read 5 Done Done Sent false
13956 4022 Acked Write 5 Done Done Sent false
13957 4023 Acked Read 7 Done Done Sent false
13958 4024 Acked Read 8 Done Done Sent false
13959 4025 Acked Write 10 Done Done Sent false
13960 4026 Acked Read 10 Done Done Sent false
13961 4027 Acked Write 2 Done Done Sent false
13962 4028 Acked Write 9 Done Done Sent false
13963 4029 Acked Write 4 Done Done Sent false
13964 4030 Acked Read 4 Done Done Sent false
13965 4031 Acked Write 7 Done Done Sent false
13966 4032 Acked Read 7 Done Done Sent false
13967 4033 Acked Flush 0 Done Done Sent false
13968 4034 Acked Flush 0 Done Done Sent false
13969 4035 Acked Read 3 Done Done Sent false
13970 4036 Acked Write 7 Done Done Sent false
13971 4037 Acked Write 1 Done Done Sent false
13972 4038 Acked Write 9 Done Done Sent false
13973 4039 Acked Read 5 Done Done Sent false
13974 4040 Acked Read 4 Done Done Sent false
13975 4041 Acked Read 9 Done Done Sent false
13976 4042 Acked Read 3 Done Done Sent false
13977 4043 Acked Read 1 Done Done Sent false
13978 4044 Acked Write 4 Done Done Sent false
13979 4045 Acked Read 5 Done Done Sent false
13980 4046 Acked Read 10 Done Done Sent false
13981 4047 Acked Read 6 Done Done Sent false
13982 4048 Acked Read 5 Done Done Sent false
13983 4049 Acked Read 10 Done Done Sent false
13984 4050 Acked Read 4 Done Done Sent false
13985 4051 Acked Write 10 Done Done Sent false
13986 4052 Acked Write 1 Done Done Sent false
13987 4053 Acked Read 2 Done Done Sent false
13988 4054 Acked Write 2 Done Done Sent false
13989 4055 Acked Read 3 Done Done Sent false
13990 4056 Acked Read 6 Done Done Sent false
13991 4057 Acked Write 10 Done Done Sent false
13992 4058 Acked Read 10 Done Done Sent false
13993 4059 Acked Write 3 Done Done Sent false
13994 4060 Acked Read 9 Done Done Sent false
13995 4061 Acked Flush 0 Done Done Sent false
13996 4062 Acked Read 2 Done Done Sent false
13997 4063 Acked Read 3 Done Done Sent false
13998 4064 Acked Write 9 Done Done Sent false
13999 4065 Acked Write 8 Done Done Sent false
14000 4066 Acked Write 10 Done Done Sent false
14001 4067 Acked Flush 0 Done Done Sent false
14002 4068 Acked Read 8 Done Done Sent false
14003 4069 Acked Write 8 Done Done Sent false
14004 4070 Acked Write 4 Done Done Sent false
14005 4071 Acked Read 7 Done Done Sent false
14006 4072 Acked Write 5 Done Done Sent false
14007 4073 Acked Read 7 Done Done Sent false
14008 4074 Acked Read 7 Done Done Sent false
14009 4075 Acked Write 7 Done Done Sent false
14010 4076 Acked Write 3 Done Done Sent false
14011 4077 Acked Write 1 Done Done Sent false
14012 4078 Acked Write 5 Done Done Sent false
14013 4079 Acked Write 9 Done Done Sent false
14014 4080 Acked Read 10 Done Done Sent false
14015 4081 Acked Write 7 Done Done Sent false
14016 4082 Acked Write 6 Done Done Sent false
14017 4083 Acked Flush 0 Done Done Sent false
14018 4084 Acked Read 5 Done Done Sent false
14019 4085 Acked Write 7 Done Done Sent false
14020 4086 Acked Flush 0 Done Done Sent false
14021 4087 Acked Read 1 Done Done Sent false
14022 4088 Acked Write 7 Done Done Sent false
14023 4089 Acked Write 8 Done Done Sent false
14024 4090 Acked Read 9 Done Done Sent false
14025 4091 Acked Write 4 Done Done Sent false
14026 4092 Acked Read 10 Done Done Sent false
14027 4093 Acked Read 7 Done Done Sent false
14028 4094 Acked Read 2 Done Done Sent false
14029 4095 Acked Read 9 Done Done Sent false
14030 4096 Acked Write 9 Done Done Sent false
14031 4097 Acked Write 2 Done Done Sent false
14032 4098 Acked Read 2 Done Done Sent false
14033 4099 Acked Write 5 Done Done Sent false
14034 4100 Acked Write 5 Done Done Sent false
14035 4101 Acked Flush 0 Done Done Sent false
14036 4102 Acked Read 3 Done Done Sent false
14037 4103 Acked Read 4 Done Done Sent false
14038 4104 Acked Write 4 Done Done Sent false
14039 4105 Acked Write 5 Done Done Sent false
14040 4106 Acked Flush 0 Done Done Sent false
14041 4107 Acked Read 2 Done Done Sent false
14042 4108 Acked Write 3 Done Done Sent false
14043 4109 Acked Flush 0 Done Done Sent false
14044 4110 Acked Read 5 Done Done Sent false
14045 4111 Acked Read 8 Done Done Sent false
14046 4112 Acked Flush 0 Done Done Sent false
14047 4113 Acked Read 3 Done Done Sent false
14048 4114 Acked Read 2 Done Done Sent false
14049 4115 Acked Write 3 Done Done Sent false
14050 4116 Acked Read 9 Done Done Sent false
14051 4117 Acked Read 9 Done Done Sent false
14052 4118 Acked Write 3 Done Done Sent false
14053 4119 Acked Write 9 Done Done Sent false
14054 4120 Acked Write 9 Done Done Sent false
14055 4121 Acked Read 9 Done Done Sent false
14056 4122 Acked Read 6 Done Done Sent false
14057 4123 Acked Write 5 Done Done Sent false
14058 4124 Acked Write 10 Done Done Sent false
14059 4125 Acked Read 6 Done Done Sent false
14060 4126 Acked Read 3 Done Done Sent false
14061 4127 Acked Write 5 Done Done Sent false
14062 4128 Acked Write 8 Done Done Sent false
14063 4129 Acked Flush 0 Done Done Sent false
14064 4130 Acked Write 3 Done Done Sent false
14065 4131 Acked Write 7 Done Done Sent false
14066 4132 Acked Write 9 Done Done Sent false
14067 4133 Acked Write 1 Done Done Sent false
14068 4134 Acked Write 5 Done Done Sent false
14069 4135 Acked Write 7 Done Done Sent false
14070 4136 Acked Write 5 Done Done Sent false
14071 4137 Acked Read 8 Done Done Sent false
14072 4138 Acked Write 4 Done Done Sent false
14073 4139 Acked Read 10 Done Done Sent false
14074 4140 Acked Write 8 Done Done Sent false
14075 4141 Acked Write 1 Done Done Sent false
14076 4142 Acked Read 10 Done Done Sent false
14077 4143 Acked Write 2 Done Done Sent false
14078 4144 Acked Write 7 Done Done Sent false
14079 4145 Acked Read 6 Done Done Sent false
14080 4146 Acked Read 6 Done Done Sent false
14081 4147 Acked Read 1 Done Done Sent false
14082 4148 Acked Flush 0 Done Done Sent false
14083 4149 Acked Write 4 Done Done Sent false
14084 4150 Acked Read 2 Done Done Sent false
14085 4151 Acked Read 3 Done Done Sent false
14086 4152 Acked Write 8 Done Done Sent false
14087 4153 Acked Write 2 Done Done Sent false
14088 4154 Acked Write 1 Done Done Sent false
14089 4155 Acked Write 1 Done Done Sent false
14090 4156 Acked Read 6 Done Done Sent false
14091 4157 Acked Read 6 Done Done Sent false
14092 4158 Acked Write 6 Done Done Sent false
14093 4159 Acked Flush 0 Done Done Sent false
14094 4160 Acked Write 6 Done Done Sent false
14095 4161 Acked Read 10 Done Done Sent false
14096 4162 Acked Read 10 Done Done Sent false
14097 4163 Acked Write 1 Done Done Sent false
14098 4164 Acked Write 5 Done Done Sent false
14099 4165 Acked Read 3 Done Done Sent false
14100 4166 Acked Write 1 Done Done Sent false
14101 4167 Acked Write 10 Done Done Sent false
14102 4168 Acked Write 7 Done Done Sent false
14103 4169 Acked Flush 0 Done Done Sent false
14104 4170 Acked Read 6 Done Done Sent false
14105 4171 Acked Read 7 Done Done Sent false
14106 4172 Acked Read 2 Done Done Sent false
14107 4173 Acked Read 10 Done Done Sent false
14108 4174 Acked Flush 0 Done Done Sent false
14109 4175 Acked Read 7 Done Done Sent false
14110 4176 Acked Read 10 Done Done Sent false
14111 4177 Acked Write 2 Done Done Sent false
14112 4178 Acked Read 5 Done Done Sent false
14113 4179 Acked Read 10 Done Done Sent false
14114 4180 Acked Read 1 Done Done Sent false
14115 4181 Acked Read 2 Done Done Sent false
14116 4182 Acked Write 1 Done Done Sent false
14117 4183 Acked Read 9 Done Done Sent false
14118 4184 Acked Flush 0 Done Done Sent false
14119 4185 Acked Write 9 Done Done Sent false
14120 4186 Acked Write 5 Done Done Sent false
14121 4187 Acked Read 10 Done Done Sent false
14122 4188 Acked Write 9 Done Done Sent false
14123 4189 Acked Write 3 Done Done Sent false
14124 4190 Acked Read 4 Done Done Sent false
14125 4191 Acked Read 10 Done Done Sent false
14126 4192 Acked Write 5 Done Done Sent false
14127 4193 Acked Flush 0 Done Done Sent false
14128 4194 Acked Read 8 Done Done Sent false
14129 4195 Acked Read 2 Done Done Sent false
14130 4196 Acked Flush 0 Done Done Sent false
14131 4197 Acked Write 4 Done Done Sent false
14132 4198 Acked Write 5 Done Done Sent false
14133 4199 Acked Write 9 Done Done Sent false
14134 4200 Acked Write 6 Done Done Sent false
14135 4201 Acked Read 5 Done Done Sent false
14136 4202 Acked Read 4 Done Done Sent false
14137 4203 Acked Write 8 Done Done Sent false
14138 4204 Acked Write 5 Done Done Sent false
14139 4205 Acked Flush 0 Done Done Sent false
14140 4206 Acked Write 8 Done Done Sent false
14141 4207 Acked Read 10 Done Done Sent false
14142 4208 Acked Write 1 Done Done Sent false
14143 4209 Acked Write 3 Done Done Sent false
14144 4210 Acked Read 3 Done Done Sent false
14145 4211 Acked Write 9 Done Done Sent false
14146 4212 Acked Read 3 Done Done Sent false
14147 4213 Acked Read 9 Done Done Sent false
14148 4214 Acked Write 6 Done Done Sent false
14149 4215 Acked Read 4 Done Done Sent false
14150 4216 Acked Flush 0 Done Done Sent false
14151 4217 Acked Write 8 Done Done Sent false
14152 4218 Acked Read 3 Done Done Sent false
14153 4219 Acked Read 8 Done Done Sent false
14154 4220 Acked Write 8 Done Done Sent false
14155 4221 Acked Read 3 Done Done Sent false
14156 4222 Acked Read 7 Done Done Sent false
14157 4223 Acked Read 9 Done Done Sent false
14158 4224 Acked Write 3 Done Done Sent false
14159 4225 Acked Write 4 Done Done Sent false
14160 4226 Acked Write 3 Done Done Sent false
14161 4227 Acked Write 7 Done Done Sent false
14162 4228 Acked Read 1 Done Done Sent false
14163 4229 Acked Write 8 Done Done Sent false
14164 4230 Acked Read 1 Done Done Sent false
14165 4231 Acked Read 2 Done Done Sent false
14166 4232 Acked Write 8 Done Done Sent false
14167 4233 Acked Write 7 Done Done Sent false
14168 4234 Acked Read 4 Done Done Sent false
14169 4235 Acked Write 2 Done Done Sent false
14170 4236 Acked Write 8 Done Done Sent false
14171 4237 Acked Write 5 Done Done Sent false
14172 4238 Acked Read 1 Done Done Sent false
14173 4239 Acked Read 5 Done Done Sent false
14174 4240 Acked Read 7 Done Done Sent false
14175 4241 Acked Read 6 Done Done Sent false
14176 4242 Acked Read 7 Done Done Sent false
14177 4243 Acked Flush 0 Done Done Sent false
14178 4244 Acked Read 5 Done Done Sent false
14179 4245 Acked Read 1 Done Done Sent false
14180 4246 Acked Flush 0 Done Done Sent false
14181 4247 Acked Read 6 Done Done Sent false
14182 4248 Acked Flush 0 Done Done Sent false
14183 4249 Acked Read 3 Done Done Sent false
14184 4250 Acked Write 7 Done Done Sent false
14185 4251 Acked Read 4 Done Done Sent false
14186 4252 Acked Read 8 Done Done Sent false
14187 4253 Acked Read 8 Done Done Sent false
14188 4254 Acked Read 1 Done Done Sent false
14189 4255 Acked Write 5 Done Done Sent false
14190 4256 Acked Read 4 Done Done Sent false
14191 4257 Acked Write 3 Done Done Sent false
14192 4258 Acked Read 8 Done Done Sent false
14193 4259 Acked Read 6 Done Done Sent false
14194 4260 Acked Write 7 Done Done Sent false
14195 4261 Acked Read 4 Done Done Sent false
14196 4262 Acked Read 2 Done Done Sent false
14197 4263 Acked Write 7 Done Done Sent false
14198 4264 Acked Read 8 Done Done Sent false
14199 4265 Acked Read 10 Done Done Sent false
14200 4266 Acked Read 5 Done Done Sent false
14201 4267 Acked Read 3 Done Done Sent false
14202 4268 Acked Read 1 Done Done Sent false
14203 4269 Acked Read 9 Done Done Sent false
14204 4270 Acked Write 4 Done Done Sent false
14205 4271 Acked Write 9 Done Done Sent false
14206 4272 Acked Read 6 Done Done Sent false
14207 4273 Acked Write 6 Done Done Sent false
14208 4274 Acked Write 6 Done Done Sent false
14209 4275 Acked Read 5 Done Done Sent false
14210 4276 Acked Write 7 Done Done Sent false
14211 4277 Acked Flush 0 Done Done Sent false
14212 4278 Acked Write 7 Done Done Sent false
14213 4279 Acked Read 1 Done Done Sent false
14214 4280 Acked Write 7 Done Done Sent false
14215 4281 Acked Read 2 Done Done Sent false
14216 4282 Acked Write 8 Done Done Sent false
14217 4283 Acked Read 3 Done Done Sent false
14218 4284 Acked Write 9 Done Done Sent false
14219 4285 Acked Read 8 Done Done Sent false
14220 4286 Acked Flush 0 Done Done Sent false
14221 4287 Acked Read 4 Done Done Sent false
14222 4288 Acked Read 8 Done Done Sent false
14223 4289 Acked Flush 0 Done Done Sent false
14224 4290 Acked Flush 0 Done Done Sent false
14225 4291 Acked Read 6 Done Done Sent false
14226 4292 Acked Read 7 Done Done Sent false
14227 4293 Acked Read 4 Done Done Sent false
14228 4294 Acked Read 2 Done Done Sent false
14229 4295 Acked Read 8 Done Done Sent false
14230 4296 Acked Read 2 Done Done Sent false
14231 4297 Acked Write 9 Done Done Sent false
14232 4298 Acked Flush 0 Done Done Sent false
14233 4299 Acked Write 4 Done Done Sent false
14234 4300 Acked Write 5 Done Done Sent false
14235 4301 Acked Read 6 Done Done Sent false
14236 4302 Acked Write 6 Done Done Sent false
14237 4303 Acked Write 9 Done Done Sent false
14238 4304 Acked Read 6 Done Done Sent false
14239 4305 Acked Read 7 Done Done Sent false
14240 4306 Acked Read 6 Done Done Sent false
14241 4307 Acked Read 9 Done Done Sent false
14242 4308 Acked Read 6 Done Done Sent false
14243 4309 Acked Read 4 Done Done Sent false
14244 4310 Acked Write 3 Done Done Sent false
14245 4311 Acked Read 3 Done Done Sent false
14246 4312 Acked Read 9 Done Done Sent false
14247 4313 Acked Read 5 Done Done Sent false
14248 4314 Acked Read 7 Done Done Sent false
14249 4315 Acked Read 10 Done Done Sent false
14250 4316 Acked Flush 0 Done Done Sent false
14251 4317 Acked Read 8 Done Done Sent false
14252 4318 Acked Flush 0 Done Done Sent false
14253 4319 Acked Read 1 Done Done Sent false
14254 4320 Acked Read 5 Done Done Sent false
14255 STATES DS:0 DS:1 DS:2 TOTAL
14256 Sent 0 0 301 301
14257 Done 301 301 0 602
14258 Skipped 0 0 0 0
14259 Error 0 0 0 0
14260 Last Flush: Some(JobId(4318)) Some(JobId(4318)) Some(JobId(4019))
14261 Downstairs last five completed: 4019 4018 4017 4016 4015
14262 Upstairs last five completed: 4320 4319 4318 4317 4316
14263 CLIENT: Up:0 ds:301 act:2
142642025-02-07T20:10:29.253ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
142652025-02-07T20:10:30.009ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
142662025-02-07T20:10:30.009ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34195) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
142672025-02-07T20:10:30.010ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
142682025-02-07T20:10:30.011ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
142692025-02-07T20:10:30.011ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
142702025-02-07T20:10:30.011ZINFOcrucible: send last flush ID to this DS: Some(JobId(4019)) = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
142712025-02-07T20:10:30.011ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
142722025-02-07T20:10:30.011ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4019)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
142732025-02-07T20:10:30.011ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4019)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
14274 ----------------------------------------------------------------
14275 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14276 Downstairs last five completed: 4321 4320 4319 4318 4317
14277 Upstairs last five completed: 4321 4320 4319 4318 4317
14278 CLIENT: Up:0 ds:0 act:3
14279 Replay: All jobs finished
14280 Main task has work to do, go find it
14281 got dsc Stop(0)
14282 stop 0
14283 Feb 07 20:10:32.263 INFO [8810] Got stop action so:false kr:false
14284 request completed, latency_us: 371, response_code: 204, uri: /stop/cid/0, method: POST, req_id: d88975ab-0fce-4b58-ab7f-86e6b3791b98, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
142852025-02-07T20:10:32.263ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
142862025-02-07T20:10:32.264ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
142872025-02-07T20:10:32.264ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
14288 Feb 07 20:10:32.264 INFO request completed, latency_us: 147, response_code: 200, uri: /state/cid/0, method: GET, req_id: 851a2433-5aaf-4033-8481-a58422324ee9, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
142892025-02-07T20:10:32.264ZINFOcrucible: client task is exiting = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
14290 [8810] Exited with: ExitStatus(unix_wait_status(9))
14291 [8810][0] reports Exit
14292 Feb 07 20:10:36.266 INFO request completed, latency_us: 224, response_code: 200, uri: /state/cid/0, method: GET, req_id: d3ea2da7-3d89-4f9a-b46f-3a3cc893f866, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
14293 001/300 Read block 3574 len 4096
14294 002/300 Write block 1448 len 36864 data: 3 3 2 2 2 2 4 4 4
14295 003/300 Write block 4535 len 36864 data: 2 3 2 2 3 3 3 2 2
14296 004/300 Write block 3480 len 32768 data: 5 5 5 6 4 3 4 3
14297 005/300 Write block 3155 len 24576 data: 3 4 4 4 5 4
14298 006/300 Read block 318 len 12288
14299 007/300 Flush
14300 008/300 Write block 2600 len 36864 data: 3 4 4 4 4 4 4 4 3
14301 009/300 Write block 3405 len 12288 data: 4 5 4
14302 010/300 Write block 1673 len 32768 data: 2 3 3 3 3 4 3 4
14303 011/300 Read block 1216 len 40960
14304 012/300 Read block 2413 len 20480
14305 013/300 Flush
14306 014/300 Write block 1021 len 36864 data: 5 5 4 3 3 3 3 3 3
14307 015/300 Write block 1550 len 40960 data: 2 2 2 2 2 2 2 3 3 3
14308 016/300 Read block 170 len 40960
14309 017/300 Read block 3273 len 12288
14310 018/300 Read block 2336 len 12288
14311 019/300 Flush
14312 020/300 Flush
14313 021/300 Read block 4148 len 36864
14314 022/300 Read block 3697 len 40960
14315 023/300 Read block 3449 len 36864
14316 024/300 Write block 2169 len 40960 data: 3 4 4 3 3 3 4 4 4 4
14317 025/300 Write block 2478 len 4096 data: 2
14318 026/300 Write block 3512 len 4096 data: 3
14319 027/300 Read block 2077 len 36864
14320 028/300 Write block 4408 len 32768 data: 4 3 3 3 4 3 3 4
14321 029/300 Flush
14322 030/300 Write block 3392 len 12288 data: 4 4 6
14323 031/300 Write block 2545 len 8192 data: 2 2
14324 032/300 Write block 2970 len 4096 data: 4
14325 033/300 Write block 1112 len 28672 data: 5 5 4 4 4 3 2
14326 034/300 Read block 694 len 32768
14327 035/300 Read block 2310 len 36864
14328 036/300 Write block 829 len 40960 data: 5 4 4 3 3 3 3 2 2 2
14329 037/300 Read block 2610 len 16384
14330 038/300 Flush
14331 039/300 Write block 1783 len 4096 data: 4
14332 040/300 Read block 3037 len 4096
14333 041/300 Read block 3040 len 24576
14334 042/300 Read block 3419 len 28672
14335 043/300 Read block 471 len 24576
14336 044/300 Flush
14337 045/300 Write block 3463 len 12288 data: 2 2 2
14338 046/300 Flush
14339 047/300 Write block 1913 len 28672 data: 3 3 3 2 3 4 5
14340 048/300 Flush
14341 049/300 Read block 1153 len 40960
14342 050/300 Write block 1231 len 20480 data: 4 3 3 3 5
14343 051/300 Read block 1393 len 32768
14344 052/300 Read block 4270 len 28672
14345 053/300 Write block 1096 len 4096 data: 3
14346 054/300 Read block 2463 len 24576
14347 055/300 Read block 2695 len 28672
14348 056/300 Read block 4427 len 12288
14349 057/300 Write block 1853 len 28672 data: 3 3 3 4 4 3 4
14350 058/300 Read block 1678 len 32768
14351 059/300 Write block 1024 len 24576 data: 4 4 4 4 4 4
14352 060/300 Write block 3786 len 12288 data: 2 2 2
14353 061/300 Read block 482 len 36864
14354 062/300 Read block 1608 len 32768
14355 063/300 Write block 3128 len 24576 data: 4 4 4 4 4 5
14356 064/300 Flush
14357 065/300 Read block 2742 len 28672
14358 066/300 Read block 1826 len 40960
14359 067/300 Write block 2169 len 32768 data: 4 5 5 4 4 4 5 5
14360 068/300 Write block 1763 len 32768 data: 3 3 3 3 4 4 3 3
14361 069/300 Flush
14362 070/300 Flush
14363 071/300 Read block 1145 len 4096
14364 072/300 Read block 1188 len 8192
14365 073/300 Read block 287 len 24576
14366 074/300 Read block 3448 len 24576
14367 075/300 Write block 2588 len 12288 data: 3 3 3
14368 076/300 Read block 4930 len 16384
14369 077/300 Read block 3798 len 20480
14370 078/300 Read block 4356 len 32768
14371 079/300 Read block 4502 len 8192
14372 080/300 Write block 1034 len 40960 data: 3 3 3 5 4 3 4 4 4 4
14373 081/300 Write block 775 len 16384 data: 2 2 2 2
14374 082/300 Read block 579 len 36864
14375 083/300 Read block 2646 len 32768
14376 084/300 Flush
14377 085/300 Read block 1708 len 32768
14378 086/300 Read block 3659 len 16384
14379 087/300 Read block 2725 len 24576
14380 088/300 Write block 2712 len 36864 data: 4 5 4 3 2 2 2 2 2
14381 089/300 Read block 4097 len 8192
14382 090/300 Flush
14383 091/300 Write block 3539 len 40960 data: 4 5 5 6 5 3 3 2 2 2
14384 092/300 Read block 976 len 36864
14385 093/300 Read block 89 len 16384
14386 094/300 Read block 2608 len 36864
14387 095/300 Read block 3230 len 16384
14388 096/300 Write block 3809 len 40960 data: 3 3 3 4 4 3 5 5 6 6
14389 097/300 Write block 4340 len 32768 data: 4 4 4 4 3 3 3 3
14390 098/300 Read block 657 len 4096
14391 099/300 Write block 1993 len 32768 data: 4 4 3 2 2 3 4 2
14392 100/300 Read block 3038 len 8192
14393 101/300 Read block 733 len 20480
14394 102/300 Write block 806 len 8192 data: 2 2
14395 103/300 Read block 3155 len 4096
14396 104/300 Read block 2626 len 28672
14397 105/300 Read block 3629 len 24576
14398 106/300 Write block 2525 len 24576 data: 3 3 3 3 2 2
14399 107/300 Flush
14400 108/300 Read block 1603 len 24576
14401 109/300 Write block 4911 len 28672 data: 4 4 4 4 4 4 3
14402 110/300 Read block 1960 len 28672
14403 111/300 Read block 1900 len 4096
14404 112/300 Read block 570 len 8192
14405 113/300 Write block 3607 len 40960 data: 3 3 3 2 3 3 3 3 3 3
14406 114/300 Write block 2064 len 24576 data: 2 2 2 2 2 2
14407 115/300 Write block 2450 len 24576 data: 4 4 5 4 3 3
14408 116/300 Write block 4465 len 16384 data: 7 7 6 4
14409 117/300 Write block 770 len 36864 data: 3 3 2 2 2 3 3 3 3
14410 118/300 Write block 2068 len 8192 data: 3 3
14411 119/300 Read block 1302 len 32768
14412 120/300 Read block 1863 len 24576
14413 121/300 Read block 1928 len 32768
14414 122/300 Write block 3568 len 24576 data: 4 5 6 6 5 5
14415 123/300 Read block 860 len 20480
14416 124/300 Read block 4780 len 20480
14417 125/300 Read block 3055 len 28672
14418 126/300 Read block 2603 len 28672
14419 127/300 Flush
14420 128/300 Write block 4859 len 4096 data: 4
14421 129/300 Read block 3359 len 16384
14422 130/300 Read block 829 len 16384
14423 131/300 Read block 4198 len 40960
14424 132/300 Read block 3159 len 40960
14425 133/300 Read block 856 len 12288
14426 134/300 Write block 744 len 36864 data: 3 3 2 2 2 2 2 3 3
14427 135/300 Flush
14428 136/300 Flush
14429 137/300 Read block 2551 len 36864
14430 138/300 Flush
14431 139/300 Write block 2271 len 8192 data: 5 5
14432 140/300 Read block 126 len 16384
14433 141/300 Write block 2017 len 20480 data: 5 4 5 5 4
14434 142/300 Read block 2716 len 40960
14435 143/300 Read block 1308 len 28672
14436 144/300 Read block 862 len 16384
14437 145/300 Read block 485 len 8192
14438 146/300 Read block 4745 len 28672
14439 147/300 Read block 3757 len 12288
14440 148/300 Flush
14441 149/300 Read block 102 len 20480
14442 150/300 Write block 4037 len 36864 data: 3 3 3 3 3 4 4 4 4
14443 151/300 Read block 261 len 28672
14444 152/300 Write block 317 len 28672 data: 4 4 4 3 2 2 2
14445 153/300 Read block 713 len 36864
14446 154/300 Read block 3914 len 20480
14447 155/300 Read block 2359 len 4096
14448 156/300 Read block 4840 len 28672
14449 157/300 Read block 2197 len 32768
14450 158/300 Read block 1181 len 12288
14451 159/300 Read block 1233 len 28672
14452 160/300 Read block 1564 len 4096
14453 161/300 Write block 2915 len 12288 data: 3 3 3
14454 162/300 Read block 1992 len 24576
14455 163/300 Write block 2053 len 16384 data: 3 3 3 4
14456 164/300 Write block 842 len 20480 data: 3 2 2 2 3
14457 165/300 Write block 2150 len 12288 data: 2 2 3
14458 166/300 Write block 1683 len 32768 data: 4 5 4 4 4 4 3 4
14459 167/300 Flush
14460 168/300 Read block 4869 len 20480
14461 169/300 Read block 4870 len 12288
14462 170/300 Flush
14463 171/300 Write block 4521 len 8192 data: 4 4
14464 172/300 Read block 1257 len 12288
14465 173/300 Read block 697 len 4096
14466 174/300 Read block 653 len 28672
14467 175/300 Flush
14468 176/300 Read block 1418 len 24576
14469 177/300 Write block 1901 len 40960 data: 4 4 4 4 4 5 5 4 4 3
14470 178/300 Read block 3591 len 36864
14471 179/300 Write block 4737 len 28672 data: 4 5 5 5 5 6 6
14472 180/300 Read block 3129 len 20480
14473 181/300 Read block 2774 len 4096
14474 182/300 Read block 1436 len 8192
14475 183/300 Write block 1072 len 28672 data: 6 5 5 5 4 5 5
14476 184/300 Read block 2647 len 28672
14477 185/300 Read block 2669 len 32768
14478 186/300 Read block 4234 len 4096
14479 187/300 Write block 1263 len 12288 data: 3 3 3
14480 188/300 Read block 1812 len 20480
14481 189/300 Write block 4688 len 16384 data: 3 3 3 4
14482 190/300 Read block 2892 len 28672
14483 191/300 Read block 3908 len 20480
14484 192/300 Read block 3559 len 4096
14485 193/300 Read block 4043 len 20480
14486 194/300 Read block 1798 len 28672
14487 195/300 Read block 796 len 8192
14488 196/300 Read block 4608 len 32768
14489 197/300 Read block 625 len 8192
14490 198/300 Read block 2072 len 24576
14491 199/300 Read block 768 len 4096
14492 200/300 Write block 1997 len 40960 data: 3 4 5 3 2 3 4 5 5 5
14493 201/300 Read block 2955 len 24576
14494 202/300 Read block 3973 len 40960
14495 203/300 Read block 946 len 4096
14496 204/300 Write block 2823 len 40960 data: 3 3 2 2 2 2 2 3 3 5
14497 205/300 Read block 2587 len 28672
14498 206/300 Write block 3275 len 20480 data: 6 5 5 4 4
14499 207/300 Read block 1196 len 32768
14500 208/300 Write block 1343 len 16384 data: 3 4 5 4
14501 209/300 Write block 2910 len 32768 data: 3 3 3 3 3 4 4 4
14502 210/300 Write block 1471 len 20480 data: 2 3 3 4 4
14503 211/300 Read block 2820 len 4096
14504 212/300 Read block 1487 len 40960
14505 213/300 Write block 203 len 16384 data: 3 3 3 3
14506 214/300 Read block 3551 len 8192
14507 215/300 Read block 1452 len 24576
14508 216/300 Write block 4817 len 12288 data: 5 5 3
14509 217/300 Read block 4002 len 16384
14510 218/300 Read block 3641 len 32768
14511 219/300 Flush
14512 220/300 Read block 412 len 28672
14513 221/300 Read block 571 len 28672
14514 222/300 Write block 2470 len 28672 data: 3 2 2 2 2 2 2
14515 223/300 Write block 1996 len 16384 data: 3 4 5 6
14516 224/300 Flush
14517 225/300 Write block 3955 len 24576 data: 3 3 2 3 3 2
14518 226/300 Read block 1532 len 8192
14519 227/300 Read block 2592 len 4096
14520 228/300 Read block 2585 len 32768
14521 229/300 Write block 4436 len 20480 data: 2 3 3 3 3
14522 230/300 Write block 2597 len 36864 data: 5 4 4 4 5 5 5 5 5
14523 231/300 Write block 4938 len 32768 data: 2 2 2 2 2 3 4 4
14524 232/300 Read block 2290 len 32768
14525 233/300 Read block 54 len 36864
14526 234/300 Read block 1224 len 8192
14527 235/300 Write block 1368 len 16384 data: 3 2 2 2
14528 236/300 Read block 79 len 4096
14529 237/300 Read block 449 len 40960
14530 238/300 Write block 3784 len 20480 data: 2 2 3 3 3
14531 239/300 Read block 2159 len 24576
14532 240/300 Read block 814 len 36864
14533 241/300 Read block 3761 len 16384
14534 242/300 Write block 1200 len 12288 data: 3 3 3
14535 243/300 Read block 2081 len 24576
14536 244/300 Write block 4349 len 36864 data: 2 2 2 2 2 2 2 2 3
14537 245/300 Read block 953 len 20480
14538 246/300 Read block 297 len 12288
14539 247/300 Read block 2386 len 32768
14540 248/300 Read block 3123 len 28672
14541 249/300 Flush
14542 250/300 Flush
14543 251/300 Read block 2160 len 12288
14544 252/300 Read block 35 len 40960
14545 253/300 Read block 2368 len 8192
14546 254/300 Read block 867 len 4096
14547 255/300 Read block 3048 len 20480
14548 256/300 Read block 1238 len 36864
14549 257/300 Write block 638 len 40960 data: 2 2 2 2 3 4 4 5 4 3
14550 258/300 Read block 4206 len 36864
14551 259/300 Read block 589 len 12288
14552 260/300 Read block 611 len 32768
14553 261/300 Read block 3916 len 36864
14554 262/300 Write block 289 len 20480 data: 4 3 2 2 2
14555 263/300 Write block 361 len 20480 data: 4 4 4 3 3
14556 264/300 Read block 52 len 32768
14557 265/300 Write block 4610 len 32768 data: 5 5 4 4 4 4 4 3
14558 266/300 Read block 1844 len 20480
14559 267/300 Write block 65 len 20480 data: 3 3 3 3 3
14560 268/300 Write block 1641 len 4096 data: 4
14561 269/300 Read block 913 len 12288
14562 270/300 Write block 3055 len 12288 data: 4 3 2
14563 271/300 Read block 3912 len 40960
14564 272/300 Read block 4984 len 4096
14565 273/300 Flush
14566 274/300 Write block 3208 len 36864 data: 3 3 4 4 4 5 4 3 3
14567 275/300 Read block 1758 len 32768
14568 276/300 Flush
14569 277/300 Flush
14570 278/300 Read block 2505 len 24576
14571 279/300 Read block 2335 len 20480
14572 280/300 Read block 3446 len 36864
14573 281/300 Write block 4643 len 20480 data: 2 2 2 3 4
14574 282/300 Write block 2208 len 20480 data: 4 4 4 3 2
14575 283/300 Write block 4623 len 20480 data: 5 5 4 4 3
14576 284/300 Read block 518 len 32768
14577 285/300 Read block 2830 len 20480
14578 286/300 Read block 2786 len 36864
14579 287/300 Flush
14580 288/300 Write block 4466 len 20480 data: 8 7 5 4 4
14581 289/300 Read block 691 len 20480
14582 290/300 Read block 3891 len 12288
14583 291/300 Write block 96 len 40960 data: 3 3 3 3 3 3 3 3 2 2
14584 292/300 Read block 1970 len 28672
14585 293/300 Write block 4277 len 20480 data: 4 3 3 2 2
14586 294/300 Read block 4680 len 4096
14587 295/300 Write block 2121 len 32768 data: 4 4 5 5 5 5 5 5
14588 296/300 Read block 1783 len 40960
14589 297/300 Read block 3360 len 12288
14590 298/300 Write block 4088 len 28672 data: 3 3 3 3 3 3 3
14591 299/300 Read block 338 len 4096
14592 300/300 Read block 3683 len 32768
14593 301/300 Read block 2227 len 16384
14594 Main task has work to do, go find it
14595 got dsc Start(0)
14596 start 0
14597 Feb 07 20:10:36.523 INFO request completed, latency_us: 201, response_code: 204, uri: /start/cid/0, method: POST, req_id: 2fa68a2a-1521-44a9-9186-485e2539e694, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
14598 Replay: started 0, returned:Ok(())
14599 ----------------------------------------------------------------
14600 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14601 States: Offline Active Active
14602 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14603 4322 Acked Read 1 Sent Done Done false
14604 4323 Acked Write 9 Sent Done Done false
14605 4324 Acked Write 9 Sent Done Done false
14606 4325 Acked Write 8 Sent Done Done false
14607 4326 Acked Write 6 Sent Done Done false
14608 4327 Acked Read 3 Sent Done Done false
14609 4328 Acked Flush 0 Sent Done Done false
14610 4329 Acked Write 9 Sent Done Done false
14611 4330 Acked Write 3 Sent Done Done false
14612 4331 Acked Write 8 Sent Done Done false
14613 4332 Acked Read 10 Sent Done Done false
14614 4333 Acked Read 5 Sent Done Done false
14615 4334 Acked Flush 0 Sent Done Done false
14616 4335 Acked Write 9 Sent Done Done false
14617 4336 Acked Write 10 Sent Done Done false
14618 4337 Acked Read 10 Sent Done Done false
14619 4338 Acked Read 3 Sent Done Done false
14620 4339 Acked Read 3 Sent Done Done false
14621 4340 Acked Flush 0 Sent Done Done false
14622 4341 Acked Flush 0 Sent Done Done false
14623 4342 Acked Read 9 Sent Done Done false
14624 4343 Acked Read 10 Sent Done Done false
14625 4344 Acked Read 9 Sent Done Done false
14626 4345 Acked Write 10 Sent Done Done false
14627 4346 Acked Write 1 Sent Done Done false
14628 4347 Acked Write 1 Sent Done Done false
14629 4348 Acked Read 9 Sent Done Done false
14630 4349 Acked Write 8 Sent Done Done false
14631 4350 Acked Flush 0 Sent Done Done false
14632 4351 Acked Write 3 Sent Done Done false
14633 4352 Acked Write 2 Sent Done Done false
14634 4353 Acked Write 1 Sent Done Done false
14635 4354 Acked Write 7 Sent Done Done false
14636 4355 Acked Read 8 Sent Done Done false
14637 4356 Acked Read 9 Sent Done Done false
14638 4357 Acked Write 10 Sent Done Done false
14639 4358 Acked Read 4 Sent Done Done false
14640 4359 Acked Flush 0 Sent Done Done false
14641 4360 Acked Write 1 Sent Done Done false
14642 4361 Acked Read 1 Sent Done Done false
14643 4362 Acked Read 6 Sent Done Done false
14644 4363 Acked Read 7 Sent Done Done false
14645 4364 Acked Read 6 Sent Done Done false
14646 4365 Acked Flush 0 Sent Done Done false
14647 4366 Acked Write 3 Sent Done Done false
14648 4367 Acked Flush 0 Sent Done Done false
14649 4368 Acked Write 7 Sent Done Done false
14650 4369 Acked Flush 0 Sent Done Done false
14651 4370 Acked Read 10 Sent Done Done false
14652 4371 Acked Write 5 Sent Done Done false
14653 4372 Acked Read 8 Sent Done Done false
14654 4373 Acked Read 7 Sent Done Done false
14655 4374 Acked Write 1 Sent Done Done false
14656 4375 Acked Read 6 Sent Done Done false
14657 4376 Acked Read 7 Sent Done Done false
14658 4377 Acked Read 3 Sent Done Done false
14659 4378 Acked Write 7 Sent Done Done false
14660 4379 Acked Read 8 Sent Done Done false
14661 4380 Acked Write 6 Sent Done Done false
14662 4381 Acked Write 3 Sent Done Done false
14663 4382 Acked Read 9 Sent Done Done false
14664 4383 Acked Read 8 Sent Done Done false
14665 4384 Acked Write 6 Sent Done Done false
14666 4385 Acked Flush 0 Sent Done Done false
14667 4386 Acked Read 7 Sent Done Done false
14668 4387 Acked Read 10 Sent Done Done false
14669 4388 Acked Write 8 Sent Done Done false
14670 4389 Acked Write 8 Sent Done Done false
14671 4390 Acked Flush 0 Sent Done Done false
14672 4391 Acked Flush 0 Sent Done Done false
14673 4392 Acked Read 1 Sent Done Done false
14674 4393 Acked Read 2 Sent Done Done false
14675 4394 Acked Read 6 Sent Done Done false
14676 4395 Acked Read 6 Sent Done Done false
14677 4396 Acked Write 3 Sent Done Done false
14678 4397 Acked Read 4 Sent Done Done false
14679 4398 Acked Read 5 Sent Done Done false
14680 4399 Acked Read 8 Sent Done Done false
14681 4400 Acked Read 2 Sent Done Done false
14682 4401 Acked Write 10 Sent Done Done false
14683 4402 Acked Write 4 Sent Done Done false
14684 4403 Acked Read 9 Sent Done Done false
14685 4404 Acked Read 8 Sent Done Done false
14686 4405 Acked Flush 0 Sent Done Done false
14687 4406 Acked Read 8 Sent Done Done false
14688 4407 Acked Read 4 Sent Done Done false
14689 4408 Acked Read 6 Sent Done Done false
14690 4409 Acked Write 9 Sent Done Done false
14691 4410 Acked Read 2 Sent Done Done false
14692 4411 Acked Flush 0 Sent Done Done false
14693 4412 Acked Write 10 Sent Done Done false
14694 4413 Acked Read 9 Sent Done Done false
14695 4414 Acked Read 4 Sent Done Done false
14696 4415 Acked Read 9 Sent Done Done false
14697 4416 Acked Read 4 Sent Done Done false
14698 4417 Acked Write 10 Sent Done Done false
14699 4418 Acked Write 8 Sent Done Done false
14700 4419 Acked Read 1 Sent Done Done false
14701 4420 Acked Write 8 Sent Done Done false
14702 4421 Acked Read 2 Sent Done Done false
14703 4422 Acked Read 5 Sent Done Done false
14704 4423 Acked Write 2 Sent Done Done false
14705 4424 Acked Read 1 Sent Done Done false
14706 4425 Acked Read 7 Sent Done Done false
14707 4426 Acked Read 6 Sent Done Done false
14708 4427 Acked Write 6 Sent Done Done false
14709 4428 Acked Flush 0 Sent Done Done false
14710 4429 Acked Read 6 Sent Done Done false
14711 4430 Acked Write 7 Sent Done Done false
14712 4431 Acked Read 7 Sent Done Done false
14713 4432 Acked Read 1 Sent Done Done false
14714 4433 Acked Read 2 Sent Done Done false
14715 4434 Acked Write 10 Sent Done Done false
14716 4435 Acked Write 6 Sent Done Done false
14717 4436 Acked Write 6 Sent Done Done false
14718 4437 Acked Write 4 Sent Done Done false
14719 4438 Acked Write 9 Sent Done Done false
14720 4439 Acked Write 2 Sent Done Done false
14721 4440 Acked Read 8 Sent Done Done false
14722 4441 Acked Read 6 Sent Done Done false
14723 4442 Acked Read 8 Sent Done Done false
14724 4443 Acked Write 6 Sent Done Done false
14725 4444 Acked Read 5 Sent Done Done false
14726 4445 Acked Read 5 Sent Done Done false
14727 4446 Acked Read 7 Sent Done Done false
14728 4447 Acked Read 7 Sent Done Done false
14729 4448 Acked Flush 0 Sent Done Done false
14730 4449 Acked Write 1 Sent Done Done false
14731 4450 Acked Read 4 Sent Done Done false
14732 4451 Acked Read 4 Sent Done Done false
14733 4452 Acked Read 10 Sent Done Done false
14734 4453 Acked Read 10 Sent Done Done false
14735 4454 Acked Read 3 Sent Done Done false
14736 4455 Acked Write 9 Sent Done Done false
14737 4456 Acked Flush 0 Sent Done Done false
14738 4457 Acked Flush 0 Sent Done Done false
14739 4458 Acked Read 9 Sent Done Done false
14740 4459 Acked Flush 0 Sent Done Done false
14741 4460 Acked Write 2 Sent Done Done false
14742 4461 Acked Read 4 Sent Done Done false
14743 4462 Acked Write 5 Sent Done Done false
14744 4463 Acked Read 10 Sent Done Done false
14745 4464 Acked Read 7 Sent Done Done false
14746 4465 Acked Read 4 Sent Done Done false
14747 4466 Acked Read 2 Sent Done Done false
14748 4467 Acked Read 7 Sent Done Done false
14749 4468 Acked Read 3 Sent Done Done false
14750 4469 Acked Flush 0 Sent Done Done false
14751 4470 Acked Read 5 Sent Done Done false
14752 4471 Acked Write 9 Sent Done Done false
14753 4472 Acked Read 7 Sent Done Done false
14754 4473 Acked Write 7 Sent Done Done false
14755 4474 Acked Read 9 Sent Done Done false
14756 4475 Acked Read 5 Sent Done Done false
14757 4476 Acked Read 1 Sent Done Done false
14758 4477 Acked Read 7 Sent Done Done false
14759 4478 Acked Read 8 Sent Done Done false
14760 4479 Acked Read 3 Sent Done Done false
14761 4480 Acked Read 7 Sent Done Done false
14762 4481 Acked Read 1 Sent Done Done false
14763 4482 Acked Write 3 Sent Done Done false
14764 4483 Acked Read 6 Sent Done Done false
14765 4484 Acked Write 4 Sent Done Done false
14766 4485 Acked Write 5 Sent Done Done false
14767 4486 Acked Write 3 Sent Done Done false
14768 4487 Acked Write 8 Sent Done Done false
14769 4488 Acked Flush 0 Sent Done Done false
14770 4489 Acked Read 5 Sent Done Done false
14771 4490 Acked Read 3 Sent Done Done false
14772 4491 Acked Flush 0 Sent Done Done false
14773 4492 Acked Write 2 Sent Done Done false
14774 4493 Acked Read 3 Sent Done Done false
14775 4494 Acked Read 1 Sent Done Done false
14776 4495 Acked Read 7 Sent Done Done false
14777 4496 Acked Flush 0 Sent Done Done false
14778 4497 Acked Read 6 Sent Done Done false
14779 4498 Acked Write 10 Sent Done Done false
14780 4499 Acked Read 9 Sent Done Done false
14781 4500 Acked Write 7 Sent Done Done false
14782 4501 Acked Read 5 Sent Done Done false
14783 4502 Acked Read 1 Sent Done Done false
14784 4503 Acked Read 2 Sent Done Done false
14785 4504 Acked Write 7 Sent Done Done false
14786 4505 Acked Read 7 Sent Done Done false
14787 4506 Acked Read 8 Sent Done Done false
14788 4507 Acked Read 1 Sent Done Done false
14789 4508 Acked Write 3 Sent Done Done false
14790 4509 Acked Read 5 Sent Done Done false
14791 4510 Acked Write 4 Sent Done Done false
14792 4511 Acked Read 7 Sent Done Done false
14793 4512 Acked Read 5 Sent Done Done false
14794 4513 Acked Read 1 Sent Done Done false
14795 4514 Acked Read 5 Sent Done Done false
14796 4515 Acked Read 7 Sent Done Done false
14797 4516 Acked Read 2 Sent Done Done false
14798 4517 Acked Read 8 Sent Done Done false
14799 4518 Acked Read 2 Sent Done Done false
14800 4519 Acked Read 6 Sent Done Done false
14801 4520 Acked Read 1 Sent Done Done false
14802 4521 Acked Write 10 Sent Done Done false
14803 4522 Acked Read 6 Sent Done Done false
14804 4523 Acked Read 10 Sent Done Done false
14805 4524 Acked Read 1 Sent Done Done false
14806 4525 Acked Write 10 Sent Done Done false
14807 4526 Acked Read 7 Sent Done Done false
14808 4527 Acked Write 5 Sent Done Done false
14809 4528 Acked Read 8 Sent Done Done false
14810 4529 Acked Write 4 Sent Done Done false
14811 4530 Acked Write 8 Sent Done Done false
14812 4531 Acked Write 5 Sent Done Done false
14813 4532 Acked Read 1 Sent Done Done false
14814 4533 Acked Read 10 Sent Done Done false
14815 4534 Acked Write 4 Sent Done Done false
14816 4535 Acked Read 2 Sent Done Done false
14817 4536 Acked Read 6 Sent Done Done false
14818 4537 Acked Write 3 Sent Done Done false
14819 4538 Acked Read 4 Sent Done Done false
14820 4539 Acked Read 8 Sent Done Done false
14821 4540 Acked Flush 0 Sent Done Done false
14822 4541 Acked Read 7 Sent Done Done false
14823 4542 Acked Read 7 Sent Done Done false
14824 4543 Acked Write 7 Sent Done Done false
14825 4544 Acked Write 4 Sent Done Done false
14826 4545 Acked Flush 0 Sent Done Done false
14827 4546 Acked Write 6 Sent Done Done false
14828 4547 Acked Read 2 Sent Done Done false
14829 4548 Acked Read 1 Sent Done Done false
14830 4549 Acked Read 8 Sent Done Done false
14831 4550 Acked Write 5 Sent Done Done false
14832 4551 Acked Write 9 Sent Done Done false
14833 4552 Acked Write 8 Sent Done Done false
14834 4553 Acked Read 8 Sent Done Done false
14835 4554 Acked Read 9 Sent Done Done false
14836 4555 Acked Read 2 Sent Done Done false
14837 4556 Acked Write 4 Sent Done Done false
14838 4557 Acked Read 1 Sent Done Done false
14839 4558 Acked Read 10 Sent Done Done false
14840 4559 Acked Write 5 Sent Done Done false
14841 4560 Acked Read 6 Sent Done Done false
14842 4561 Acked Read 9 Sent Done Done false
14843 4562 Acked Read 4 Sent Done Done false
14844 4563 Acked Write 3 Sent Done Done false
14845 4564 Acked Read 6 Sent Done Done false
14846 4565 Acked Write 9 Sent Done Done false
14847 4566 Acked Read 5 Sent Done Done false
14848 4567 Acked Read 3 Sent Done Done false
14849 4568 Acked Read 8 Sent Done Done false
14850 4569 Acked Read 7 Sent Done Done false
14851 4570 Acked Flush 0 Sent Done Done false
14852 4571 Acked Flush 0 Sent Done Done false
14853 4572 Acked Read 3 Sent Done Done false
14854 4573 Acked Read 10 Sent Done Done false
14855 4574 Acked Read 2 Sent Done Done false
14856 4575 Acked Read 1 Sent Done Done false
14857 4576 Acked Read 5 Sent Done Done false
14858 4577 Acked Read 9 Sent Done Done false
14859 4578 Acked Write 10 Sent Done Done false
14860 4579 Acked Read 9 Sent Done Done false
14861 4580 Acked Read 3 Sent Done Done false
14862 4581 Acked Read 8 Sent Done Done false
14863 4582 Acked Read 9 Sent Done Done false
14864 4583 Acked Write 5 Sent Done Done false
14865 4584 Acked Write 5 Sent Done Done false
14866 4585 Acked Read 8 Sent Done Done false
14867 4586 Acked Write 8 Sent Done Done false
14868 4587 Acked Read 5 Sent Done Done false
14869 4588 Acked Write 5 Sent Done Done false
14870 4589 Acked Write 1 Sent Done Done false
14871 4590 Acked Read 3 Sent Done Done false
14872 4591 Acked Write 3 Sent Done Done false
14873 4592 Acked Read 10 Sent Done Done false
14874 4593 Acked Read 1 Sent Done Done false
14875 4594 Acked Flush 0 Sent Done Done false
14876 4595 Acked Write 9 Sent Done Done false
14877 4596 Acked Read 8 Sent Done Done false
14878 4597 Acked Flush 0 Sent Done Done false
14879 4598 Acked Flush 0 Sent Done Done false
14880 4599 Acked Read 6 Sent Done Done false
14881 4600 Acked Read 5 Sent Done Done false
14882 4601 Acked Read 9 Sent Done Done false
14883 4602 Acked Write 5 Sent Done Done false
14884 4603 Acked Write 5 Sent Done Done false
14885 4604 Acked Write 5 Sent Done Done false
14886 4605 Acked Read 8 Sent Done Done false
14887 4606 Acked Read 5 Sent Done Done false
14888 4607 Acked Read 9 Sent Done Done false
14889 4608 Acked Flush 0 Sent Done Done false
14890 4609 Acked Write 5 Sent Done Done false
14891 4610 Acked Read 5 Sent Done Done false
14892 4611 Acked Read 3 Sent Done Done false
14893 4612 Acked Write 10 Sent Done Done false
14894 4613 Acked Read 7 Sent Done Done false
14895 4614 Acked Write 5 Sent Done Done false
14896 4615 Acked Read 1 Sent Done Done false
14897 4616 Acked Write 8 Sent Done Done false
14898 4617 Acked Read 10 Sent Done Done false
14899 4618 Acked Read 3 Sent Done Done false
14900 4619 Acked Write 7 Sent Done Done false
14901 4620 Acked Read 1 Sent Done Done false
14902 4621 Acked Read 8 Sent Done Done false
14903 4622 Acked Read 4 Sent Done Done false
14904 STATES DS:0 DS:1 DS:2 TOTAL
14905 Sent 301 0 0 301
14906 Done 0 301 301 602
14907 Skipped 0 0 0 0
14908 Error 0 0 0 0
14909 Last Flush: Some(JobId(4321)) Some(JobId(4608)) Some(JobId(4608))
14910 Downstairs last five completed: 4321 4320 4319 4318 4317
14911 Upstairs last five completed: 4622 4621 4620 4619 4618
14912 CLIENT: Up:0 ds:301 act:2
149132025-02-07T20:10:37.265ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
149142025-02-07T20:10:38.266ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
14915 [8810] got start action
14916 [8810] I am going to restart
14917 Starting downstairs at port 8810
14918 Make output file at "/tmp/dsc/downstairs-8810.txt"
14919 [8810][0] reports Starting
14920 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1221)
14921 [8810][0] reports Running
149222025-02-07T20:10:39.268ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
149232025-02-07T20:10:40.512ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
14924 ----------------------------------------------------------------
14925 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14926 States: Offline Active Active
14927 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14928 4322 Acked Read 1 Sent Done Done false
14929 4323 Acked Write 9 Sent Done Done false
14930 4324 Acked Write 9 Sent Done Done false
14931 4325 Acked Write 8 Sent Done Done false
14932 4326 Acked Write 6 Sent Done Done false
14933 4327 Acked Read 3 Sent Done Done false
14934 4328 Acked Flush 0 Sent Done Done false
14935 4329 Acked Write 9 Sent Done Done false
14936 4330 Acked Write 3 Sent Done Done false
14937 4331 Acked Write 8 Sent Done Done false
14938 4332 Acked Read 10 Sent Done Done false
14939 4333 Acked Read 5 Sent Done Done false
14940 4334 Acked Flush 0 Sent Done Done false
14941 4335 Acked Write 9 Sent Done Done false
14942 4336 Acked Write 10 Sent Done Done false
14943 4337 Acked Read 10 Sent Done Done false
14944 4338 Acked Read 3 Sent Done Done false
14945 4339 Acked Read 3 Sent Done Done false
14946 4340 Acked Flush 0 Sent Done Done false
14947 4341 Acked Flush 0 Sent Done Done false
14948 4342 Acked Read 9 Sent Done Done false
14949 4343 Acked Read 10 Sent Done Done false
14950 4344 Acked Read 9 Sent Done Done false
14951 4345 Acked Write 10 Sent Done Done false
14952 4346 Acked Write 1 Sent Done Done false
14953 4347 Acked Write 1 Sent Done Done false
14954 4348 Acked Read 9 Sent Done Done false
14955 4349 Acked Write 8 Sent Done Done false
14956 4350 Acked Flush 0 Sent Done Done false
14957 4351 Acked Write 3 Sent Done Done false
14958 4352 Acked Write 2 Sent Done Done false
14959 4353 Acked Write 1 Sent Done Done false
14960 4354 Acked Write 7 Sent Done Done false
14961 4355 Acked Read 8 Sent Done Done false
14962 4356 Acked Read 9 Sent Done Done false
14963 4357 Acked Write 10 Sent Done Done false
14964 4358 Acked Read 4 Sent Done Done false
14965 4359 Acked Flush 0 Sent Done Done false
14966 4360 Acked Write 1 Sent Done Done false
14967 4361 Acked Read 1 Sent Done Done false
14968 4362 Acked Read 6 Sent Done Done false
14969 4363 Acked Read 7 Sent Done Done false
14970 4364 Acked Read 6 Sent Done Done false
14971 4365 Acked Flush 0 Sent Done Done false
14972 4366 Acked Write 3 Sent Done Done false
14973 4367 Acked Flush 0 Sent Done Done false
14974 4368 Acked Write 7 Sent Done Done false
14975 4369 Acked Flush 0 Sent Done Done false
14976 4370 Acked Read 10 Sent Done Done false
14977 4371 Acked Write 5 Sent Done Done false
14978 4372 Acked Read 8 Sent Done Done false
14979 4373 Acked Read 7 Sent Done Done false
14980 4374 Acked Write 1 Sent Done Done false
14981 4375 Acked Read 6 Sent Done Done false
14982 4376 Acked Read 7 Sent Done Done false
14983 4377 Acked Read 3 Sent Done Done false
14984 4378 Acked Write 7 Sent Done Done false
14985 4379 Acked Read 8 Sent Done Done false
14986 4380 Acked Write 6 Sent Done Done false
14987 4381 Acked Write 3 Sent Done Done false
14988 4382 Acked Read 9 Sent Done Done false
14989 4383 Acked Read 8 Sent Done Done false
14990 4384 Acked Write 6 Sent Done Done false
14991 4385 Acked Flush 0 Sent Done Done false
14992 4386 Acked Read 7 Sent Done Done false
14993 4387 Acked Read 10 Sent Done Done false
14994 4388 Acked Write 8 Sent Done Done false
14995 4389 Acked Write 8 Sent Done Done false
14996 4390 Acked Flush 0 Sent Done Done false
14997 4391 Acked Flush 0 Sent Done Done false
14998 4392 Acked Read 1 Sent Done Done false
14999 4393 Acked Read 2 Sent Done Done false
15000 4394 Acked Read 6 Sent Done Done false
15001 4395 Acked Read 6 Sent Done Done false
15002 4396 Acked Write 3 Sent Done Done false
15003 4397 Acked Read 4 Sent Done Done false
15004 4398 Acked Read 5 Sent Done Done false
15005 4399 Acked Read 8 Sent Done Done false
15006 4400 Acked Read 2 Sent Done Done false
15007 4401 Acked Write 10 Sent Done Done false
15008 4402 Acked Write 4 Sent Done Done false
15009 4403 Acked Read 9 Sent Done Done false
15010 4404 Acked Read 8 Sent Done Done false
15011 4405 Acked Flush 0 Sent Done Done false
15012 4406 Acked Read 8 Sent Done Done false
15013 4407 Acked Read 4 Sent Done Done false
15014 4408 Acked Read 6 Sent Done Done false
15015 4409 Acked Write 9 Sent Done Done false
15016 4410 Acked Read 2 Sent Done Done false
15017 4411 Acked Flush 0 Sent Done Done false
15018 4412 Acked Write 10 Sent Done Done false
15019 4413 Acked Read 9 Sent Done Done false
15020 4414 Acked Read 4 Sent Done Done false
15021 4415 Acked Read 9 Sent Done Done false
15022 4416 Acked Read 4 Sent Done Done false
15023 4417 Acked Write 10 Sent Done Done false
15024 4418 Acked Write 8 Sent Done Done false
15025 4419 Acked Read 1 Sent Done Done false
15026 4420 Acked Write 8 Sent Done Done false
15027 4421 Acked Read 2 Sent Done Done false
15028 4422 Acked Read 5 Sent Done Done false
15029 4423 Acked Write 2 Sent Done Done false
15030 4424 Acked Read 1 Sent Done Done false
15031 4425 Acked Read 7 Sent Done Done false
15032 4426 Acked Read 6 Sent Done Done false
15033 4427 Acked Write 6 Sent Done Done false
15034 4428 Acked Flush 0 Sent Done Done false
15035 4429 Acked Read 6 Sent Done Done false
15036 4430 Acked Write 7 Sent Done Done false
15037 4431 Acked Read 7 Sent Done Done false
15038 4432 Acked Read 1 Sent Done Done false
15039 4433 Acked Read 2 Sent Done Done false
15040 4434 Acked Write 10 Sent Done Done false
15041 4435 Acked Write 6 Sent Done Done false
15042 4436 Acked Write 6 Sent Done Done false
15043 4437 Acked Write 4 Sent Done Done false
15044 4438 Acked Write 9 Sent Done Done false
15045 4439 Acked Write 2 Sent Done Done false
15046 4440 Acked Read 8 Sent Done Done false
15047 4441 Acked Read 6 Sent Done Done false
15048 4442 Acked Read 8 Sent Done Done false
15049 4443 Acked Write 6 Sent Done Done false
15050 4444 Acked Read 5 Sent Done Done false
15051 4445 Acked Read 5 Sent Done Done false
15052 4446 Acked Read 7 Sent Done Done false
15053 4447 Acked Read 7 Sent Done Done false
15054 4448 Acked Flush 0 Sent Done Done false
15055 4449 Acked Write 1 Sent Done Done false
15056 4450 Acked Read 4 Sent Done Done false
15057 4451 Acked Read 4 Sent Done Done false
15058 4452 Acked Read 10 Sent Done Done false
15059 4453 Acked Read 10 Sent Done Done false
15060 4454 Acked Read 3 Sent Done Done false
15061 4455 Acked Write 9 Sent Done Done false
15062 4456 Acked Flush 0 Sent Done Done false
15063 4457 Acked Flush 0 Sent Done Done false
15064 4458 Acked Read 9 Sent Done Done false
15065 4459 Acked Flush 0 Sent Done Done false
15066 4460 Acked Write 2 Sent Done Done false
15067 4461 Acked Read 4 Sent Done Done false
15068 4462 Acked Write 5 Sent Done Done false
15069 4463 Acked Read 10 Sent Done Done false
15070 4464 Acked Read 7 Sent Done Done false
15071 4465 Acked Read 4 Sent Done Done false
15072 4466 Acked Read 2 Sent Done Done false
15073 4467 Acked Read 7 Sent Done Done false
15074 4468 Acked Read 3 Sent Done Done false
15075 4469 Acked Flush 0 Sent Done Done false
15076 4470 Acked Read 5 Sent Done Done false
15077 4471 Acked Write 9 Sent Done Done false
15078 4472 Acked Read 7 Sent Done Done false
15079 4473 Acked Write 7 Sent Done Done false
15080 4474 Acked Read 9 Sent Done Done false
15081 4475 Acked Read 5 Sent Done Done false
15082 4476 Acked Read 1 Sent Done Done false
15083 4477 Acked Read 7 Sent Done Done false
15084 4478 Acked Read 8 Sent Done Done false
15085 4479 Acked Read 3 Sent Done Done false
15086 4480 Acked Read 7 Sent Done Done false
15087 4481 Acked Read 1 Sent Done Done false
15088 4482 Acked Write 3 Sent Done Done false
15089 4483 Acked Read 6 Sent Done Done false
15090 4484 Acked Write 4 Sent Done Done false
15091 4485 Acked Write 5 Sent Done Done false
15092 4486 Acked Write 3 Sent Done Done false
15093 4487 Acked Write 8 Sent Done Done false
15094 4488 Acked Flush 0 Sent Done Done false
15095 4489 Acked Read 5 Sent Done Done false
15096 4490 Acked Read 3 Sent Done Done false
15097 4491 Acked Flush 0 Sent Done Done false
15098 4492 Acked Write 2 Sent Done Done false
15099 4493 Acked Read 3 Sent Done Done false
15100 4494 Acked Read 1 Sent Done Done false
15101 4495 Acked Read 7 Sent Done Done false
15102 4496 Acked Flush 0 Sent Done Done false
15103 4497 Acked Read 6 Sent Done Done false
15104 4498 Acked Write 10 Sent Done Done false
15105 4499 Acked Read 9 Sent Done Done false
15106 4500 Acked Write 7 Sent Done Done false
15107 4501 Acked Read 5 Sent Done Done false
15108 4502 Acked Read 1 Sent Done Done false
15109 4503 Acked Read 2 Sent Done Done false
15110 4504 Acked Write 7 Sent Done Done false
15111 4505 Acked Read 7 Sent Done Done false
15112 4506 Acked Read 8 Sent Done Done false
15113 4507 Acked Read 1 Sent Done Done false
15114 4508 Acked Write 3 Sent Done Done false
15115 4509 Acked Read 5 Sent Done Done false
15116 4510 Acked Write 4 Sent Done Done false
15117 4511 Acked Read 7 Sent Done Done false
15118 4512 Acked Read 5 Sent Done Done false
15119 4513 Acked Read 1 Sent Done Done false
15120 4514 Acked Read 5 Sent Done Done false
15121 4515 Acked Read 7 Sent Done Done false
15122 4516 Acked Read 2 Sent Done Done false
15123 4517 Acked Read 8 Sent Done Done false
15124 4518 Acked Read 2 Sent Done Done false
15125 4519 Acked Read 6 Sent Done Done false
15126 4520 Acked Read 1 Sent Done Done false
15127 4521 Acked Write 10 Sent Done Done false
15128 4522 Acked Read 6 Sent Done Done false
15129 4523 Acked Read 10 Sent Done Done false
15130 4524 Acked Read 1 Sent Done Done false
15131 4525 Acked Write 10 Sent Done Done false
15132 4526 Acked Read 7 Sent Done Done false
15133 4527 Acked Write 5 Sent Done Done false
15134 4528 Acked Read 8 Sent Done Done false
15135 4529 Acked Write 4 Sent Done Done false
15136 4530 Acked Write 8 Sent Done Done false
15137 4531 Acked Write 5 Sent Done Done false
15138 4532 Acked Read 1 Sent Done Done false
15139 4533 Acked Read 10 Sent Done Done false
15140 4534 Acked Write 4 Sent Done Done false
15141 4535 Acked Read 2 Sent Done Done false
15142 4536 Acked Read 6 Sent Done Done false
15143 4537 Acked Write 3 Sent Done Done false
15144 4538 Acked Read 4 Sent Done Done false
15145 4539 Acked Read 8 Sent Done Done false
15146 4540 Acked Flush 0 Sent Done Done false
15147 4541 Acked Read 7 Sent Done Done false
15148 4542 Acked Read 7 Sent Done Done false
15149 4543 Acked Write 7 Sent Done Done false
15150 4544 Acked Write 4 Sent Done Done false
15151 4545 Acked Flush 0 Sent Done Done false
15152 4546 Acked Write 6 Sent Done Done false
15153 4547 Acked Read 2 Sent Done Done false
15154 4548 Acked Read 1 Sent Done Done false
15155 4549 Acked Read 8 Sent Done Done false
15156 4550 Acked Write 5 Sent Done Done false
15157 4551 Acked Write 9 Sent Done Done false
15158 4552 Acked Write 8 Sent Done Done false
15159 4553 Acked Read 8 Sent Done Done false
15160 4554 Acked Read 9 Sent Done Done false
15161 4555 Acked Read 2 Sent Done Done false
15162 4556 Acked Write 4 Sent Done Done false
15163 4557 Acked Read 1 Sent Done Done false
15164 4558 Acked Read 10 Sent Done Done false
15165 4559 Acked Write 5 Sent Done Done false
15166 4560 Acked Read 6 Sent Done Done false
15167 4561 Acked Read 9 Sent Done Done false
15168 4562 Acked Read 4 Sent Done Done false
15169 4563 Acked Write 3 Sent Done Done false
15170 4564 Acked Read 6 Sent Done Done false
15171 4565 Acked Write 9 Sent Done Done false
15172 4566 Acked Read 5 Sent Done Done false
15173 4567 Acked Read 3 Sent Done Done false
15174 4568 Acked Read 8 Sent Done Done false
15175 4569 Acked Read 7 Sent Done Done false
15176 4570 Acked Flush 0 Sent Done Done false
15177 4571 Acked Flush 0 Sent Done Done false
15178 4572 Acked Read 3 Sent Done Done false
15179 4573 Acked Read 10 Sent Done Done false
15180 4574 Acked Read 2 Sent Done Done false
15181 4575 Acked Read 1 Sent Done Done false
15182 4576 Acked Read 5 Sent Done Done false
15183 4577 Acked Read 9 Sent Done Done false
15184 4578 Acked Write 10 Sent Done Done false
15185 4579 Acked Read 9 Sent Done Done false
15186 4580 Acked Read 3 Sent Done Done false
15187 4581 Acked Read 8 Sent Done Done false
15188 4582 Acked Read 9 Sent Done Done false
15189 4583 Acked Write 5 Sent Done Done false
15190 4584 Acked Write 5 Sent Done Done false
15191 4585 Acked Read 8 Sent Done Done false
15192 4586 Acked Write 8 Sent Done Done false
15193 4587 Acked Read 5 Sent Done Done false
15194 4588 Acked Write 5 Sent Done Done false
15195 4589 Acked Write 1 Sent Done Done false
15196 4590 Acked Read 3 Sent Done Done false
15197 4591 Acked Write 3 Sent Done Done false
15198 4592 Acked Read 10 Sent Done Done false
15199 4593 Acked Read 1 Sent Done Done false
15200 4594 Acked Flush 0 Sent Done Done false
15201 4595 Acked Write 9 Sent Done Done false
15202 4596 Acked Read 8 Sent Done Done false
15203 4597 Acked Flush 0 Sent Done Done false
15204 4598 Acked Flush 0 Sent Done Done false
15205 4599 Acked Read 6 Sent Done Done false
15206 4600 Acked Read 5 Sent Done Done false
15207 4601 Acked Read 9 Sent Done Done false
15208 4602 Acked Write 5 Sent Done Done false
15209 4603 Acked Write 5 Sent Done Done false
15210 4604 Acked Write 5 Sent Done Done false
15211 4605 Acked Read 8 Sent Done Done false
15212 4606 Acked Read 5 Sent Done Done false
15213 4607 Acked Read 9 Sent Done Done false
15214 4608 Acked Flush 0 Sent Done Done false
15215 4609 Acked Write 5 Sent Done Done false
15216 4610 Acked Read 5 Sent Done Done false
15217 4611 Acked Read 3 Sent Done Done false
15218 4612 Acked Write 10 Sent Done Done false
15219 4613 Acked Read 7 Sent Done Done false
15220 4614 Acked Write 5 Sent Done Done false
15221 4615 Acked Read 1 Sent Done Done false
15222 4616 Acked Write 8 Sent Done Done false
15223 4617 Acked Read 10 Sent Done Done false
15224 4618 Acked Read 3 Sent Done Done false
15225 4619 Acked Write 7 Sent Done Done false
15226 4620 Acked Read 1 Sent Done Done false
15227 4621 Acked Read 8 Sent Done Done false
15228 4622 Acked Read 4 Sent Done Done false
15229 STATES DS:0 DS:1 DS:2 TOTAL
15230 Sent 301 0 0 301
15231 Done 0 301 301 602
15232 Skipped 0 0 0 0
15233 Error 0 0 0 0
15234 Last Flush: Some(JobId(4321)) Some(JobId(4608)) Some(JobId(4608))
15235 Downstairs last five completed: 4321 4320 4319 4318 4317
15236 Upstairs last five completed: 4622 4621 4620 4619 4618
15237 CLIENT: Up:0 ds:301 act:2
152382025-02-07T20:10:41.272ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
152392025-02-07T20:10:42.265ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
152402025-02-07T20:10:42.266ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41169) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
152412025-02-07T20:10:42.267ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
152422025-02-07T20:10:42.267ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
152432025-02-07T20:10:42.267ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
152442025-02-07T20:10:42.267ZINFOcrucible: send last flush ID to this DS: Some(JobId(4321)) = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
152452025-02-07T20:10:42.267ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
152462025-02-07T20:10:42.267ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4321)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
152472025-02-07T20:10:42.267ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4321)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
15248 ----------------------------------------------------------------
15249 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15250 Downstairs last five completed: 4623 4622 4621 4620 4619
15251 Upstairs last five completed: 4623 4622 4621 4620 4619
15252 CLIENT: Up:0 ds:0 act:3
15253 Replay: All jobs finished
15254 Main task has work to do, go find it
15255 got dsc Stop(2)
15256 stop 2
15257 Feb 07 20:10:44.528 [8830] Got stop action so:false kr:false
15258 INFO request completed, latency_us: 400, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 238e632f-a4a4-4bb9-a884-f257a55b59b5, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
152592025-02-07T20:10:44.529ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
152602025-02-07T20:10:44.529ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
152612025-02-07T20:10:44.529ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
152622025-02-07T20:10:44.529ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
15263 Feb 07 20:10:44.529 INFO request completed, latency_us: 173, response_code: 200, uri: /state/cid/2, method: GET, req_id: a9ea759d-bd37-4dd5-9faa-5be61341db17, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
15264 [8830] Exited with: ExitStatus(unix_wait_status(9))
15265 [8830][2] reports Exit
15266 Feb 07 20:10:48.531 INFO request completed, latency_us: 233, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9bae7c6a-a304-4859-afbc-1500ce21ed81, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
15267 001/300 Write block 3897 len 28672 data: 3 3 3 3 3 3 3
15268 002/300 Read block 3493 len 24576
15269 003/300 Write block 1787 len 16384 data: 4 3 3 2
15270 004/300 Write block 830 len 12288 data: 5 5 4
15271 005/300 Write block 712 len 32768 data: 4 5 4 4 4 4 4 4
15272 006/300 Write block 3895 len 8192 data: 2 2
15273 007/300 Write block 496 len 32768 data: 3 4 5 5 5 5 5 4
15274 008/300 Flush
15275 009/300 Read block 737 len 36864
15276 010/300 Read block 2643 len 12288
15277 011/300 Read block 4811 len 12288
15278 012/300 Write block 3944 len 24576 data: 2 2 3 5 5 4
15279 013/300 Write block 3442 len 20480 data: 2 3 5 5 5
15280 014/300 Write block 4478 len 28672 data: 6 5 5 3 3 3 3
15281 015/300 Read block 1849 len 40960
15282 016/300 Write block 654 len 36864 data: 3 3 2 2 2 2 3 3 3
15283 017/300 Flush
15284 018/300 Write block 4793 len 36864 data: 4 4 4 3 2 2 4 4 6
15285 019/300 Read block 4338 len 4096
15286 020/300 Read block 4914 len 24576
15287 021/300 Write block 466 len 40960 data: 4 4 4 3 3 3 4 4 4 4
15288 022/300 Flush
15289 023/300 Write block 2661 len 20480 data: 3 3 3 3 3
15290 024/300 Write block 1493 len 20480 data: 6 6 5 5 3
15291 025/300 Flush
15292 026/300 Write block 3617 len 28672 data: 3 3 3 2 2 3 3
15293 027/300 Read block 4071 len 24576
15294 028/300 Write block 1359 len 28672 data: 2 2 2 2 2 3 3
15295 029/300 Read block 2831 len 12288
15296 030/300 Write block 4196 len 20480 data: 4 4 4 3 3
15297 031/300 Read block 3873 len 24576
15298 032/300 Read block 2763 len 20480
15299 033/300 Write block 209 len 8192 data: 2 3
15300 034/300 Read block 325 len 32768
15301 035/300 Read block 4870 len 4096
15302 036/300 Flush
15303 037/300 Read block 765 len 24576
15304 038/300 Read block 2200 len 40960
15305 039/300 Write block 4537 len 20480 data: 3 3 4 4 4
15306 040/300 Flush
15307 041/300 Read block 3325 len 12288
15308 042/300 Read block 2057 len 8192
15309 043/300 Read block 3277 len 12288
15310 044/300 Write block 1643 len 4096 data: 3
15311 045/300 Write block 3702 len 28672 data: 5 5 4 4 3 3 4
15312 046/300 Write block 1995 len 40960 data: 4 4 5 6 7 4 3 4 5 6
15313 047/300 Write block 3811 len 24576 data: 4 5 5 4 6 6
15314 048/300 Read block 532 len 4096
15315 049/300 Read block 3772 len 8192
15316 050/300 Write block 886 len 8192 data: 5 4
15317 051/300 Write block 1624 len 20480 data: 3 2 2 2 2
15318 052/300 Read block 3210 len 32768
15319 053/300 Flush
15320 054/300 Write block 3916 len 20480 data: 4 3 4 4 3
15321 055/300 Read block 2217 len 28672
15322 056/300 Write block 822 len 20480 data: 3 3 3 4 4
15323 057/300 Write block 4450 len 8192 data: 3 3
15324 058/300 Read block 2137 len 28672
15325 059/300 Read block 4512 len 32768
15326 060/300 Write block 4500 len 12288 data: 6 6 8
15327 061/300 Read block 1575 len 8192
15328 062/300 Write block 4520 len 8192 data: 5 5
15329 063/300 Read block 398 len 36864
15330 064/300 Read block 1267 len 40960
15331 065/300 Write block 2191 len 40960 data: 4 5 5 5 4 4 3 4 4 4
15332 066/300 Read block 2900 len 36864
15333 067/300 Read block 2560 len 36864
15334 068/300 Read block 1294 len 24576
15335 069/300 Write block 4863 len 32768 data: 4 4 3 3 3 2 3 3
15336 070/300 Write block 1860 len 40960 data: 5 4 3 4 4 4 5 4 4 3
15337 071/300 Write block 2273 len 24576 data: 6 6 6 4 4 4
15338 072/300 Write block 1508 len 32768 data: 5 4 4 4 2 2 2 2
15339 073/300 Read block 4569 len 32768
15340 074/300 Read block 4764 len 16384
15341 075/300 Read block 853 len 28672
15342 076/300 Read block 1500 len 8192
15343 077/300 Write block 1316 len 28672 data: 3 3 3 3 5 6 6
15344 078/300 Write block 4252 len 8192 data: 4 3
15345 079/300 Read block 2226 len 8192
15346 080/300 Write block 4942 len 20480 data: 3 4 5 5 4
15347 081/300 Read block 2487 len 28672
15348 082/300 Write block 1906 len 40960 data: 6 6 5 5 4 3 3 4 4 4
15349 083/300 Read block 3372 len 16384
15350 084/300 Read block 4906 len 36864
15351 085/300 Read block 937 len 32768
15352 086/300 Write block 4326 len 4096 data: 5
15353 087/300 Write block 1074 len 4096 data: 6
15354 088/300 Write block 2753 len 36864 data: 3 4 5 5 3 3 3 4 3
15355 089/300 Write block 3849 len 12288 data: 3 2 3
15356 090/300 Write block 3996 len 40960 data: 5 5 5 4 4 3 3 3 3 2
15357 091/300 Read block 4413 len 12288
15358 092/300 Write block 221 len 4096 data: 3
15359 093/300 Write block 546 len 24576 data: 2 3 3 2 2 2
15360 094/300 Write block 4495 len 40960 data: 6 4 5 5 6 7 7 9 6 7
15361 095/300 Read block 1919 len 36864
15362 096/300 Write block 4957 len 24576 data: 5 5 5 6 6 6
15363 097/300 Read block 2420 len 16384
15364 098/300 Flush
15365 099/300 Write block 1293 len 40960 data: 3 3 3 5 5 5 5 5 5 5
15366 100/300 Write block 1598 len 36864 data: 2 2 2 3 3 3 2 2 3
15367 101/300 Flush
15368 102/300 Write block 1019 len 20480 data: 5 5 6 6 5
15369 103/300 Flush
15370 104/300 Read block 4897 len 32768
15371 105/300 Write block 4489 len 12288 data: 3 3 3
15372 106/300 Read block 2686 len 32768
15373 107/300 Write block 2956 len 40960 data: 5 4 3 3 4 4 4 3 3 3
15374 108/300 Flush
15375 109/300 Write block 3432 len 32768 data: 4 5 4 4 4 4 4 4
15376 110/300 Read block 1172 len 12288
15377 111/300 Read block 2672 len 16384
15378 112/300 Flush
15379 113/300 Write block 1843 len 36864 data: 2 2 3 3 3 5 5 4 4
15380 114/300 Write block 2887 len 8192 data: 2 2
15381 115/300 Write block 1322 len 40960 data: 7 6 6 6 5 5 3 4 5 5
15382 116/300 Write block 182 len 20480 data: 3 4 4 4 4
15383 117/300 Flush
15384 118/300 Read block 573 len 8192
15385 119/300 Read block 676 len 20480
15386 120/300 Read block 4942 len 20480
15387 121/300 Write block 4594 len 4096 data: 4
15388 122/300 Write block 2857 len 20480 data: 2 3 3 5 6
15389 123/300 Write block 4132 len 8192 data: 2 2
15390 124/300 Read block 920 len 8192
15391 125/300 Read block 2707 len 8192
15392 126/300 Write block 4143 len 8192 data: 3 4
15393 127/300 Flush
15394 128/300 Write block 279 len 8192 data: 4 4
15395 129/300 Flush
15396 130/300 Write block 2900 len 40960 data: 2 2 2 2 2 2 2 2 3 3
15397 131/300 Write block 4979 len 24576 data: 4 4 3 3 2 2
15398 132/300 Write block 4743 len 20480 data: 7 6 6 4 6
15399 133/300 Read block 169 len 28672
15400 134/300 Write block 2340 len 12288 data: 4 6 5
15401 135/300 Flush
15402 136/300 Write block 2906 len 20480 data: 3 3 4 4 4
15403 137/300 Read block 470 len 28672
15404 138/300 Read block 3108 len 16384
15405 139/300 Write block 3520 len 24576 data: 3 3 4 4 3 3
15406 140/300 Read block 3205 len 20480
15407 141/300 Read block 586 len 12288
15408 142/300 Read block 811 len 12288
15409 143/300 Read block 310 len 28672
15410 144/300 Write block 2357 len 40960 data: 2 3 3 3 3 3 5 5 4 5
15411 145/300 Write block 4720 len 16384 data: 3 3 3 3
15412 146/300 Write block 2051 len 28672 data: 2 2 4 4 4 5 2
15413 147/300 Read block 3608 len 40960
15414 148/300 Write block 1739 len 8192 data: 2 3
15415 149/300 Read block 1285 len 4096
15416 150/300 Read block 2073 len 8192
15417 151/300 Read block 607 len 16384
15418 152/300 Write block 4031 len 8192 data: 5 5
15419 153/300 Write block 172 len 32768 data: 4 4 3 2 2 2 2 2
15420 154/300 Read block 4389 len 4096
15421 155/300 Flush
15422 156/300 Read block 1171 len 24576
15423 157/300 Write block 754 len 28672 data: 5 5 4 3 4 5 4
15424 158/300 Read block 1837 len 20480
15425 159/300 Write block 3346 len 28672 data: 2 3 3 3 3 4 3
15426 160/300 Read block 1036 len 4096
15427 161/300 Read block 2880 len 40960
15428 162/300 Read block 527 len 36864
15429 163/300 Read block 1778 len 28672
15430 164/300 Write block 1691 len 40960 data: 4 3 4 4 4 4 3 2 2 2
15431 165/300 Read block 2951 len 16384
15432 166/300 Read block 1590 len 12288
15433 167/300 Read block 17 len 32768
15434 168/300 Write block 70 len 24576 data: 2 3 3 3 3 4
15435 169/300 Read block 1597 len 28672
15436 170/300 Flush
15437 171/300 Write block 497 len 12288 data: 5 6 6
15438 172/300 Read block 1270 len 16384
15439 173/300 Read block 705 len 16384
15440 174/300 Write block 3129 len 4096 data: 5
15441 175/300 Flush
15442 176/300 Write block 2236 len 40960 data: 3 4 4 4 4 3 3 3 2 2
15443 177/300 Write block 2703 len 4096 data: 2
15444 178/300 Read block 2650 len 24576
15445 179/300 Write block 3374 len 20480 data: 3 2 2 2 3
15446 180/300 Write block 920 len 12288 data: 2 2 2
15447 181/300 Write block 1129 len 12288 data: 3 3 2
15448 182/300 Write block 2261 len 12288 data: 4 4 4
15449 183/300 Read block 53 len 36864
15450 184/300 Write block 4893 len 8192 data: 3 3
15451 185/300 Write block 2564 len 16384 data: 3 5 6 4
15452 186/300 Read block 1033 len 32768
15453 187/300 Read block 987 len 28672
15454 188/300 Write block 4198 len 8192 data: 5 4
15455 189/300 Read block 3823 len 24576
15456 190/300 Read block 3970 len 8192
15457 191/300 Read block 1447 len 16384
15458 192/300 Write block 278 len 4096 data: 4
15459 193/300 Write block 4961 len 40960 data: 7 7 6 5 5 6 6 4 4 3
15460 194/300 Read block 2100 len 40960
15461 195/300 Read block 4889 len 28672
15462 196/300 Read block 1141 len 12288
15463 197/300 Flush
15464 198/300 Write block 898 len 36864 data: 3 3 3 3 4 4 4 4 3
15465 199/300 Write block 2037 len 24576 data: 2 2 2 2 2 2
15466 200/300 Read block 582 len 4096
15467 201/300 Write block 408 len 24576 data: 3 3 3 3 3 2
15468 202/300 Write block 3708 len 12288 data: 5 4 4
15469 203/300 Read block 4938 len 20480
15470 204/300 Write block 4469 len 32768 data: 5 5 5 6 3 4 6 6
15471 205/300 Read block 1455 len 40960
15472 206/300 Read block 942 len 4096
15473 207/300 Write block 3773 len 40960 data: 3 5 6 5 7 7 6 5 5 3
15474 208/300 Flush
15475 209/300 Read block 2329 len 12288
15476 210/300 Write block 766 len 36864 data: 3 3 3 3 4 4 3 3 3
15477 211/300 Write block 3049 len 28672 data: 6 6 6 8 5 5 5
15478 212/300 Read block 4493 len 24576
15479 213/300 Read block 635 len 36864
15480 214/300 Write block 4611 len 4096 data: 6
15481 215/300 Read block 1404 len 20480
15482 216/300 Write block 2212 len 28672 data: 3 2 2 2 2 2 2
15483 217/300 Read block 2793 len 28672
15484 218/300 Read block 2152 len 24576
15485 219/300 Read block 1548 len 36864
15486 220/300 Write block 1589 len 32768 data: 4 3 3 3 3 2 2 2
15487 221/300 Write block 4082 len 40960 data: 3 4 4 4 3 3 4 4 4 4
15488 222/300 Read block 2459 len 16384
15489 223/300 Read block 1876 len 28672
15490 224/300 Flush
15491 225/300 Flush
15492 226/300 Write block 1083 len 16384 data: 3 4 4 4
15493 227/300 Read block 3768 len 16384
15494 228/300 Read block 429 len 16384
15495 229/300 Read block 4573 len 8192
15496 230/300 Write block 92 len 28672 data: 2 3 2 3 4 4 4
15497 231/300 Write block 2632 len 40960 data: 3 2 2 2 2 2 2 2 2 2
15498 232/300 Read block 3867 len 40960
15499 233/300 Read block 2956 len 16384
15500 234/300 Write block 3772 len 32768 data: 3 4 6 7 6 8 8 7
15501 235/300 Read block 1102 len 20480
15502 236/300 Write block 2164 len 12288 data: 6 5 5
15503 237/300 Write block 4984 len 24576 data: 3 2 3 5 4 4
15504 238/300 Write block 1775 len 20480 data: 2 2 3 3 3
15505 239/300 Write block 1925 len 12288 data: 3 2 2
15506 240/300 Write block 3203 len 40960 data: 4 5 5 4 3 4 4 5 5 5
15507 241/300 Read block 48 len 4096
15508 242/300 Read block 2228 len 24576
15509 243/300 Write block 4095 len 24576 data: 2 2 2 2 2 2
15510 244/300 Read block 1585 len 20480
15511 245/300 Write block 3349 len 24576 data: 4 4 5 4 3 3
15512 246/300 Flush
15513 247/300 Flush
15514 248/300 Read block 338 len 32768
15515 249/300 Write block 1695 len 40960 data: 5 5 4 3 3 3 3 3 3 3
15516 250/300 Read block 1987 len 16384
15517 251/300 Read block 1195 len 12288
15518 252/300 Flush
15519 253/300 Flush
15520 254/300 Read block 1001 len 24576
15521 255/300 Write block 1644 len 40960 data: 3 5 4 4 4 4 4 6 6 6
15522 256/300 Read block 4399 len 16384
15523 257/300 Read block 388 len 28672
15524 258/300 Write block 1893 len 8192 data: 4 5
15525 259/300 Write block 3963 len 12288 data: 2 2 2
15526 260/300 Write block 2839 len 36864 data: 2 2 2 2 2 2 2 2 2
15527 261/300 Write block 574 len 32768 data: 3 3 2 2 2 2 2 2
15528 262/300 Read block 2955 len 24576
15529 263/300 Write block 3151 len 32768 data: 4 4 4 3 4 5 5 5
15530 264/300 Read block 1127 len 20480
15531 265/300 Read block 1090 len 16384
15532 266/300 Write block 3139 len 32768 data: 4 5 5 4 3 3 4 4
15533 267/300 Read block 3767 len 24576
15534 268/300 Write block 1101 len 40960 data: 4 4 4 4 4 4 3 4 4 5
15535 269/300 Flush
15536 270/300 Flush
15537 271/300 Read block 2038 len 40960
15538 272/300 Read block 995 len 4096
15539 273/300 Write block 3854 len 28672 data: 4 5 3 3 3 3 3
15540 274/300 Flush
15541 275/300 Read block 4407 len 40960
15542 276/300 Write block 3359 len 16384 data: 3 3 4 4
15543 277/300 Write block 2807 len 32768 data: 6 6 5 4 3 3 4 4
15544 278/300 Read block 1818 len 8192
15545 279/300 Read block 4592 len 12288
15546 280/300 Write block 905 len 12288 data: 5 4 2
15547 281/300 Write block 4098 len 8192 data: 3 3
15548 282/300 Write block 1014 len 16384 data: 3 3 3 4
15549 283/300 Flush
15550 284/300 Write block 1433 len 32768 data: 3 4 4 4 4 5 5 5
15551 285/300 Read block 4707 len 24576
15552 286/300 Read block 852 len 4096
15553 287/300 Write block 4764 len 4096 data: 2
15554 288/300 Read block 515 len 20480
15555 289/300 Write block 4581 len 16384 data: 2 2 2 2
15556 290/300 Write block 3681 len 32768 data: 5 5 4 5 4 5 5 4
15557 291/300 Write block 4588 len 8192 data: 3 3
15558 292/300 Read block 3985 len 4096
15559 293/300 Write block 1776 len 32768 data: 3 4 4 4 3 4 4 5
15560 294/300 Write block 2346 len 20480 data: 4 4 4 3 3
15561 295/300 Read block 1840 len 28672
15562 296/300 Read block 2360 len 4096
15563 297/300 Read block 569 len 12288
15564 298/300 Write block 445 len 32768 data: 2 2 3 4 4 5 4 4
15565 299/300 Read block 2706 len 36864
15566 300/300 Write block 3062 len 20480 data: 3 3 2 3 2
15567 301/300 Write block 777 len 12288 data: 4 4 2
15568 Main task has work to do, go find it
15569 got dsc Start(2)
15570 start 2
15571 Feb 07 20:10:48.762 INFO request completed, latency_us: 273, response_code: 204, uri: /start/cid/2, method: POST, req_id: a3014129-3c18-4e97-acf2-327156ffd400, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
15572 Replay: started 2, returned:Ok(())
15573 ----------------------------------------------------------------
15574 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15575 States: Active Active Offline
15576 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15577 4624 Acked Write 7 Done Done Sent false
15578 4625 Acked Read 6 Done Done Sent false
15579 4626 Acked Write 4 Done Done Sent false
15580 4627 Acked Write 3 Done Done Sent false
15581 4628 Acked Write 8 Done Done Sent false
15582 4629 Acked Write 2 Done Done Sent false
15583 4630 Acked Write 8 Done Done Sent false
15584 4631 Acked Flush 0 Done Done Sent false
15585 4632 Acked Read 9 Done Done Sent false
15586 4633 Acked Read 3 Done Done Sent false
15587 4634 Acked Read 3 Done Done Sent false
15588 4635 Acked Write 6 Done Done Sent false
15589 4636 Acked Write 5 Done Done Sent false
15590 4637 Acked Write 7 Done Done Sent false
15591 4638 Acked Read 10 Done Done Sent false
15592 4639 Acked Write 9 Done Done Sent false
15593 4640 Acked Flush 0 Done Done Sent false
15594 4641 Acked Write 9 Done Done Sent false
15595 4642 Acked Read 1 Done Done Sent false
15596 4643 Acked Read 6 Done Done Sent false
15597 4644 Acked Write 10 Done Done Sent false
15598 4645 Acked Flush 0 Done Done Sent false
15599 4646 Acked Write 5 Done Done Sent false
15600 4647 Acked Write 5 Done Done Sent false
15601 4648 Acked Flush 0 Done Done Sent false
15602 4649 Acked Write 7 Done Done Sent false
15603 4650 Acked Read 6 Done Done Sent false
15604 4651 Acked Write 7 Done Done Sent false
15605 4652 Acked Read 3 Done Done Sent false
15606 4653 Acked Write 5 Done Done Sent false
15607 4654 Acked Read 6 Done Done Sent false
15608 4655 Acked Read 5 Done Done Sent false
15609 4656 Acked Write 2 Done Done Sent false
15610 4657 Acked Read 8 Done Done Sent false
15611 4658 Acked Read 1 Done Done Sent false
15612 4659 Acked Flush 0 Done Done Sent false
15613 4660 Acked Read 6 Done Done Sent false
15614 4661 Acked Read 10 Done Done Sent false
15615 4662 Acked Write 5 Done Done Sent false
15616 4663 Acked Flush 0 Done Done Sent false
15617 4664 Acked Read 3 Done Done Sent false
15618 4665 Acked Read 2 Done Done Sent false
15619 4666 Acked Read 3 Done Done Sent false
15620 4667 Acked Write 1 Done Done Sent false
15621 4668 Acked Write 7 Done Done Sent false
15622 4669 Acked Write 10 Done Done Sent false
15623 4670 Acked Write 6 Done Done Sent false
15624 4671 Acked Read 1 Done Done Sent false
15625 4672 Acked Read 2 Done Done Sent false
15626 4673 Acked Write 2 Done Done Sent false
15627 4674 Acked Write 5 Done Done Sent false
15628 4675 Acked Read 8 Done Done Sent false
15629 4676 Acked Flush 0 Done Done Sent false
15630 4677 Acked Write 5 Done Done Sent false
15631 4678 Acked Read 7 Done Done Sent false
15632 4679 Acked Write 5 Done Done Sent false
15633 4680 Acked Write 2 Done Done Sent false
15634 4681 Acked Read 7 Done Done Sent false
15635 4682 Acked Read 8 Done Done Sent false
15636 4683 Acked Write 3 Done Done Sent false
15637 4684 Acked Read 2 Done Done Sent false
15638 4685 Acked Write 2 Done Done Sent false
15639 4686 Acked Read 9 Done Done Sent false
15640 4687 Acked Read 10 Done Done Sent false
15641 4688 Acked Write 10 Done Done Sent false
15642 4689 Acked Read 9 Done Done Sent false
15643 4690 Acked Read 9 Done Done Sent false
15644 4691 Acked Read 6 Done Done Sent false
15645 4692 Acked Write 8 Done Done Sent false
15646 4693 Acked Write 10 Done Done Sent false
15647 4694 Acked Write 6 Done Done Sent false
15648 4695 Acked Write 8 Done Done Sent false
15649 4696 Acked Read 8 Done Done Sent false
15650 4697 Acked Read 4 Done Done Sent false
15651 4698 Acked Read 7 Done Done Sent false
15652 4699 Acked Read 2 Done Done Sent false
15653 4700 Acked Write 7 Done Done Sent false
15654 4701 Acked Write 2 Done Done Sent false
15655 4702 Acked Read 2 Done Done Sent false
15656 4703 Acked Write 5 Done Done Sent false
15657 4704 Acked Read 7 Done Done Sent false
15658 4705 Acked Write 10 Done Done Sent false
15659 4706 Acked Read 4 Done Done Sent false
15660 4707 Acked Read 9 Done Done Sent false
15661 4708 Acked Read 8 Done Done Sent false
15662 4709 Acked Write 1 Done Done Sent false
15663 4710 Acked Write 1 Done Done Sent false
15664 4711 Acked Write 9 Done Done Sent false
15665 4712 Acked Write 3 Done Done Sent false
15666 4713 Acked Write 10 Done Done Sent false
15667 4714 Acked Read 3 Done Done Sent false
15668 4715 Acked Write 1 Done Done Sent false
15669 4716 Acked Write 6 Done Done Sent false
15670 4717 Acked Write 10 Done Done Sent false
15671 4718 Acked Read 9 Done Done Sent false
15672 4719 Acked Write 6 Done Done Sent false
15673 4720 Acked Read 4 Done Done Sent false
15674 4721 Acked Flush 0 Done Done Sent false
15675 4722 Acked Write 10 Done Done Sent false
15676 4723 Acked Write 9 Done Done Sent false
15677 4724 Acked Flush 0 Done Done Sent false
15678 4725 Acked Write 5 Done Done Sent false
15679 4726 Acked Flush 0 Done Done Sent false
15680 4727 Acked Read 8 Done Done Sent false
15681 4728 Acked Write 3 Done Done Sent false
15682 4729 Acked Read 8 Done Done Sent false
15683 4730 Acked Write 10 Done Done Sent false
15684 4731 Acked Flush 0 Done Done Sent false
15685 4732 Acked Write 8 Done Done Sent false
15686 4733 Acked Read 3 Done Done Sent false
15687 4734 Acked Read 4 Done Done Sent false
15688 4735 Acked Flush 0 Done Done Sent false
15689 4736 Acked Write 9 Done Done Sent false
15690 4737 Acked Write 2 Done Done Sent false
15691 4738 Acked Write 10 Done Done Sent false
15692 4739 Acked Write 5 Done Done Sent false
15693 4740 Acked Flush 0 Done Done Sent false
15694 4741 Acked Read 2 Done Done Sent false
15695 4742 Acked Read 5 Done Done Sent false
15696 4743 Acked Read 5 Done Done Sent false
15697 4744 Acked Write 1 Done Done Sent false
15698 4745 Acked Write 5 Done Done Sent false
15699 4746 Acked Write 2 Done Done Sent false
15700 4747 Acked Read 2 Done Done Sent false
15701 4748 Acked Read 2 Done Done Sent false
15702 4749 Acked Write 2 Done Done Sent false
15703 4750 Acked Flush 0 Done Done Sent false
15704 4751 Acked Write 2 Done Done Sent false
15705 4752 Acked Flush 0 Done Done Sent false
15706 4753 Acked Write 10 Done Done Sent false
15707 4754 Acked Write 6 Done Done Sent false
15708 4755 Acked Write 5 Done Done Sent false
15709 4756 Acked Read 7 Done Done Sent false
15710 4757 Acked Write 3 Done Done Sent false
15711 4758 Acked Flush 0 Done Done Sent false
15712 4759 Acked Write 5 Done Done Sent false
15713 4760 Acked Read 7 Done Done Sent false
15714 4761 Acked Read 4 Done Done Sent false
15715 4762 Acked Write 6 Done Done Sent false
15716 4763 Acked Read 5 Done Done Sent false
15717 4764 Acked Read 3 Done Done Sent false
15718 4765 Acked Read 3 Done Done Sent false
15719 4766 Acked Read 7 Done Done Sent false
15720 4767 Acked Write 10 Done Done Sent false
15721 4768 Acked Write 4 Done Done Sent false
15722 4769 Acked Write 7 Done Done Sent false
15723 4770 Acked Read 10 Done Done Sent false
15724 4771 Acked Write 2 Done Done Sent false
15725 4772 Acked Read 1 Done Done Sent false
15726 4773 Acked Read 2 Done Done Sent false
15727 4774 Acked Read 4 Done Done Sent false
15728 4775 Acked Write 2 Done Done Sent false
15729 4776 Acked Write 8 Done Done Sent false
15730 4777 Acked Read 1 Done Done Sent false
15731 4778 Acked Flush 0 Done Done Sent false
15732 4779 Acked Read 6 Done Done Sent false
15733 4780 Acked Write 7 Done Done Sent false
15734 4781 Acked Read 5 Done Done Sent false
15735 4782 Acked Write 7 Done Done Sent false
15736 4783 Acked Read 1 Done Done Sent false
15737 4784 Acked Read 10 Done Done Sent false
15738 4785 Acked Read 9 Done Done Sent false
15739 4786 Acked Read 7 Done Done Sent false
15740 4787 Acked Write 10 Done Done Sent false
15741 4788 Acked Read 4 Done Done Sent false
15742 4789 Acked Read 3 Done Done Sent false
15743 4790 Acked Read 8 Done Done Sent false
15744 4791 Acked Write 6 Done Done Sent false
15745 4792 Acked Read 7 Done Done Sent false
15746 4793 Acked Flush 0 Done Done Sent false
15747 4794 Acked Write 3 Done Done Sent false
15748 4795 Acked Read 4 Done Done Sent false
15749 4796 Acked Read 4 Done Done Sent false
15750 4797 Acked Write 1 Done Done Sent false
15751 4798 Acked Flush 0 Done Done Sent false
15752 4799 Acked Write 10 Done Done Sent false
15753 4800 Acked Write 1 Done Done Sent false
15754 4801 Acked Read 6 Done Done Sent false
15755 4802 Acked Write 5 Done Done Sent false
15756 4803 Acked Write 3 Done Done Sent false
15757 4804 Acked Write 3 Done Done Sent false
15758 4805 Acked Write 3 Done Done Sent false
15759 4806 Acked Read 9 Done Done Sent false
15760 4807 Acked Write 2 Done Done Sent false
15761 4808 Acked Write 4 Done Done Sent false
15762 4809 Acked Read 8 Done Done Sent false
15763 4810 Acked Read 7 Done Done Sent false
15764 4811 Acked Write 2 Done Done Sent false
15765 4812 Acked Read 6 Done Done Sent false
15766 4813 Acked Read 2 Done Done Sent false
15767 4814 Acked Read 4 Done Done Sent false
15768 4815 Acked Write 1 Done Done Sent false
15769 4816 Acked Write 10 Done Done Sent false
15770 4817 Acked Read 10 Done Done Sent false
15771 4818 Acked Read 7 Done Done Sent false
15772 4819 Acked Read 3 Done Done Sent false
15773 4820 Acked Flush 0 Done Done Sent false
15774 4821 Acked Write 9 Done Done Sent false
15775 4822 Acked Write 6 Done Done Sent false
15776 4823 Acked Read 1 Done Done Sent false
15777 4824 Acked Write 6 Done Done Sent false
15778 4825 Acked Write 3 Done Done Sent false
15779 4826 Acked Read 5 Done Done Sent false
15780 4827 Acked Write 8 Done Done Sent false
15781 4828 Acked Read 10 Done Done Sent false
15782 4829 Acked Read 1 Done Done Sent false
15783 4830 Acked Write 10 Done Done Sent false
15784 4831 Acked Flush 0 Done Done Sent false
15785 4832 Acked Read 3 Done Done Sent false
15786 4833 Acked Write 9 Done Done Sent false
15787 4834 Acked Write 7 Done Done Sent false
15788 4835 Acked Read 6 Done Done Sent false
15789 4836 Acked Read 9 Done Done Sent false
15790 4837 Acked Write 1 Done Done Sent false
15791 4838 Acked Read 5 Done Done Sent false
15792 4839 Acked Write 7 Done Done Sent false
15793 4840 Acked Read 7 Done Done Sent false
15794 4841 Acked Read 6 Done Done Sent false
15795 4842 Acked Read 9 Done Done Sent false
15796 4843 Acked Write 8 Done Done Sent false
15797 4844 Acked Write 10 Done Done Sent false
15798 4845 Acked Read 4 Done Done Sent false
15799 4846 Acked Read 7 Done Done Sent false
15800 4847 Acked Flush 0 Done Done Sent false
15801 4848 Acked Flush 0 Done Done Sent false
15802 4849 Acked Write 4 Done Done Sent false
15803 4850 Acked Read 4 Done Done Sent false
15804 4851 Acked Read 4 Done Done Sent false
15805 4852 Acked Read 2 Done Done Sent false
15806 4853 Acked Write 7 Done Done Sent false
15807 4854 Acked Write 10 Done Done Sent false
15808 4855 Acked Read 10 Done Done Sent false
15809 4856 Acked Read 4 Done Done Sent false
15810 4857 Acked Write 8 Done Done Sent false
15811 4858 Acked Read 5 Done Done Sent false
15812 4859 Acked Write 3 Done Done Sent false
15813 4860 Acked Write 6 Done Done Sent false
15814 4861 Acked Write 5 Done Done Sent false
15815 4862 Acked Write 3 Done Done Sent false
15816 4863 Acked Write 10 Done Done Sent false
15817 4864 Acked Read 1 Done Done Sent false
15818 4865 Acked Read 6 Done Done Sent false
15819 4866 Acked Write 6 Done Done Sent false
15820 4867 Acked Read 5 Done Done Sent false
15821 4868 Acked Write 6 Done Done Sent false
15822 4869 Acked Flush 0 Done Done Sent false
15823 4870 Acked Flush 0 Done Done Sent false
15824 4871 Acked Read 8 Done Done Sent false
15825 4872 Acked Write 10 Done Done Sent false
15826 4873 Acked Read 4 Done Done Sent false
15827 4874 Acked Read 3 Done Done Sent false
15828 4875 Acked Flush 0 Done Done Sent false
15829 4876 Acked Flush 0 Done Done Sent false
15830 4877 Acked Read 6 Done Done Sent false
15831 4878 Acked Write 10 Done Done Sent false
15832 4879 Acked Read 4 Done Done Sent false
15833 4880 Acked Read 7 Done Done Sent false
15834 4881 Acked Write 2 Done Done Sent false
15835 4882 Acked Write 3 Done Done Sent false
15836 4883 Acked Write 9 Done Done Sent false
15837 4884 Acked Write 8 Done Done Sent false
15838 4885 Acked Read 6 Done Done Sent false
15839 4886 Acked Write 8 Done Done Sent false
15840 4887 Acked Read 5 Done Done Sent false
15841 4888 Acked Read 4 Done Done Sent false
15842 4889 Acked Write 8 Done Done Sent false
15843 4890 Acked Read 6 Done Done Sent false
15844 4891 Acked Write 10 Done Done Sent false
15845 4892 Acked Flush 0 Done Done Sent false
15846 4893 Acked Flush 0 Done Done Sent false
15847 4894 Acked Read 10 Done Done Sent false
15848 4895 Acked Read 1 Done Done Sent false
15849 4896 Acked Write 7 Done Done Sent false
15850 4897 Acked Flush 0 Done Done Sent false
15851 4898 Acked Read 10 Done Done Sent false
15852 4899 Acked Write 4 Done Done Sent false
15853 4900 Acked Write 8 Done Done Sent false
15854 4901 Acked Read 2 Done Done Sent false
15855 4902 Acked Read 3 Done Done Sent false
15856 4903 Acked Write 3 Done Done Sent false
15857 4904 Acked Write 2 Done Done Sent false
15858 4905 Acked Write 4 Done Done Sent false
15859 4906 Acked Flush 0 Done Done Sent false
15860 4907 Acked Write 8 Done Done Sent false
15861 4908 Acked Read 6 Done Done Sent false
15862 4909 Acked Read 1 Done Done Sent false
15863 4910 Acked Write 1 Done Done Sent false
15864 4911 Acked Read 5 Done Done Sent false
15865 4912 Acked Write 4 Done Done Sent false
15866 4913 Acked Write 8 Done Done Sent false
15867 4914 Acked Write 2 Done Done Sent false
15868 4915 Acked Read 1 Done Done Sent false
15869 4916 Acked Write 8 Done Done Sent false
15870 4917 Acked Write 5 Done Done Sent false
15871 4918 Acked Read 7 Done Done Sent false
15872 4919 Acked Read 1 Done Done Sent false
15873 4920 Acked Read 3 Done Done Sent false
15874 4921 Acked Write 8 Done Done Sent false
15875 4922 Acked Read 9 Done Done Sent false
15876 4923 Acked Write 5 Done Done Sent false
15877 4924 Acked Write 3 Done Done Sent false
15878 STATES DS:0 DS:1 DS:2 TOTAL
15879 Sent 0 0 301 301
15880 Done 301 301 0 602
15881 Skipped 0 0 0 0
15882 Error 0 0 0 0
15883 Last Flush: Some(JobId(4906)) Some(JobId(4906)) Some(JobId(4623))
15884 Downstairs last five completed: 4623 4622 4621 4620 4619
15885 Upstairs last five completed: 4924 4923 4922 4921 4920
15886 CLIENT: Up:0 ds:301 act:2
158872025-02-07T20:10:49.284ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
158882025-02-07T20:10:50.286ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
158892025-02-07T20:10:51.288ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
15890 ----------------------------------------------------------------
15891 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15892 States: Active Active Offline
15893 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15894 4624 Acked Write 7 Done Done Sent false
15895 4625 Acked Read 6 Done Done Sent false
15896 4626 Acked Write 4 Done Done Sent false
15897 4627 Acked Write 3 Done Done Sent false
15898 4628 Acked Write 8 Done Done Sent false
15899 4629 Acked Write 2 Done Done Sent false
15900 4630 Acked Write 8 Done Done Sent false
15901 4631 Acked Flush 0 Done Done Sent false
15902 4632 Acked Read 9 Done Done Sent false
15903 4633 Acked Read 3 Done Done Sent false
15904 4634 Acked Read 3 Done Done Sent false
15905 4635 Acked Write 6 Done Done Sent false
15906 4636 Acked Write 5 Done Done Sent false
15907 4637 Acked Write 7 Done Done Sent false
15908 4638 Acked Read 10 Done Done Sent false
15909 4639 Acked Write 9 Done Done Sent false
15910 4640 Acked Flush 0 Done Done Sent false
15911 4641 Acked Write 9 Done Done Sent false
15912 4642 Acked Read 1 Done Done Sent false
15913 4643 Acked Read 6 Done Done Sent false
15914 4644 Acked Write 10 Done Done Sent false
15915 4645 Acked Flush 0 Done Done Sent false
15916 4646 Acked Write 5 Done Done Sent false
15917 4647 Acked Write 5 Done Done Sent false
15918 4648 Acked Flush 0 Done Done Sent false
15919 4649 Acked Write 7 Done Done Sent false
15920 4650 Acked Read 6 Done Done Sent false
15921 4651 Acked Write 7 Done Done Sent false
15922 4652 Acked Read 3 Done Done Sent false
15923 4653 Acked Write 5 Done Done Sent false
15924 4654 Acked Read 6 Done Done Sent false
15925 4655 Acked Read 5 Done Done Sent false
15926 4656 Acked Write 2 Done Done Sent false
15927 4657 Acked Read 8 Done Done Sent false
15928 4658 Acked Read 1 Done Done Sent false
15929 4659 Acked Flush 0 Done Done Sent false
15930 4660 Acked Read 6 Done Done Sent false
15931 4661 Acked Read 10 Done Done Sent false
15932 4662 Acked Write 5 Done Done Sent false
15933 4663 Acked Flush 0 Done Done Sent false
15934 4664 Acked Read 3 Done Done Sent false
15935 4665 Acked Read 2 Done Done Sent false
15936 4666 Acked Read 3 Done Done Sent false
15937 4667 Acked Write 1 Done Done Sent false
15938 4668 Acked Write 7 Done Done Sent false
15939 4669 Acked Write 10 Done Done Sent false
15940 4670 Acked Write 6 Done Done Sent false
15941 4671 Acked Read 1 Done Done Sent false
15942 4672 Acked Read 2 Done Done Sent false
15943 4673 Acked Write 2 Done Done Sent false
15944 4674 Acked Write 5 Done Done Sent false
15945 4675 Acked Read 8 Done Done Sent false
15946 4676 Acked Flush 0 Done Done Sent false
15947 4677 Acked Write 5 Done Done Sent false
15948 4678 Acked Read 7 Done Done Sent false
15949 4679 Acked Write 5 Done Done Sent false
15950 4680 Acked Write 2 Done Done Sent false
15951 4681 Acked Read 7 Done Done Sent false
15952 4682 Acked Read 8 Done Done Sent false
15953 4683 Acked Write 3 Done Done Sent false
15954 4684 Acked Read 2 Done Done Sent false
15955 4685 Acked Write 2 Done Done Sent false
15956 4686 Acked Read 9 Done Done Sent false
15957 4687 Acked Read 10 Done Done Sent false
15958 4688 Acked Write 10 Done Done Sent false
15959 4689 Acked Read 9 Done Done Sent false
15960 4690 Acked Read 9 Done Done Sent false
15961 4691 Acked Read 6 Done Done Sent false
15962 4692 Acked Write 8 Done Done Sent false
15963 4693 Acked Write 10 Done Done Sent false
15964 4694 Acked Write 6 Done Done Sent false
15965 4695 Acked Write 8 Done Done Sent false
15966 4696 Acked Read 8 Done Done Sent false
15967 4697 Acked Read 4 Done Done Sent false
15968 4698 Acked Read 7 Done Done Sent false
15969 4699 Acked Read 2 Done Done Sent false
15970 4700 Acked Write 7 Done Done Sent false
15971 4701 Acked Write 2 Done Done Sent false
15972 4702 Acked Read 2 Done Done Sent false
15973 4703 Acked Write 5 Done Done Sent false
15974 4704 Acked Read 7 Done Done Sent false
15975 4705 Acked Write 10 Done Done Sent false
15976 4706 Acked Read 4 Done Done Sent false
15977 4707 Acked Read 9 Done Done Sent false
15978 4708 Acked Read 8 Done Done Sent false
15979 4709 Acked Write 1 Done Done Sent false
15980 4710 Acked Write 1 Done Done Sent false
15981 4711 Acked Write 9 Done Done Sent false
15982 4712 Acked Write 3 Done Done Sent false
15983 4713 Acked Write 10 Done Done Sent false
15984 4714 Acked Read 3 Done Done Sent false
15985 4715 Acked Write 1 Done Done Sent false
15986 4716 Acked Write 6 Done Done Sent false
15987 4717 Acked Write 10 Done Done Sent false
15988 4718 Acked Read 9 Done Done Sent false
15989 4719 Acked Write 6 Done Done Sent false
15990 4720 Acked Read 4 Done Done Sent false
15991 4721 Acked Flush 0 Done Done Sent false
15992 4722 Acked Write 10 Done Done Sent false
15993 4723 Acked Write 9 Done Done Sent false
15994 4724 Acked Flush 0 Done Done Sent false
15995 4725 Acked Write 5 Done Done Sent false
15996 4726 Acked Flush 0 Done Done Sent false
15997 4727 Acked Read 8 Done Done Sent false
15998 4728 Acked Write 3 Done Done Sent false
15999 4729 Acked Read 8 Done Done Sent false
16000 4730 Acked Write 10 Done Done Sent false
16001 4731 Acked Flush 0 Done Done Sent false
16002 4732 Acked Write 8 Done Done Sent false
16003 4733 Acked Read 3 Done Done Sent false
16004 4734 Acked Read 4 Done Done Sent false
16005 4735 Acked Flush 0 Done Done Sent false
16006 4736 Acked Write 9 Done Done Sent false
16007 4737 Acked Write 2 Done Done Sent false
16008 4738 Acked Write 10 Done Done Sent false
16009 4739 Acked Write 5 Done Done Sent false
16010 4740 Acked Flush 0 Done Done Sent false
16011 4741 Acked Read 2 Done Done Sent false
16012 4742 Acked Read 5 Done Done Sent false
16013 4743 Acked Read 5 Done Done Sent false
16014 4744 Acked Write 1 Done Done Sent false
16015 4745 Acked Write 5 Done Done Sent false
16016 4746 Acked Write 2 Done Done Sent false
16017 4747 Acked Read 2 Done Done Sent false
16018 4748 Acked Read 2 Done Done Sent false
16019 4749 Acked Write 2 Done Done Sent false
16020 4750 Acked Flush 0 Done Done Sent false
16021 4751 Acked Write 2 Done Done Sent false
16022 4752 Acked Flush 0 Done Done Sent false
16023 4753 Acked Write 10 Done Done Sent false
16024 4754 Acked Write 6 Done Done Sent false
16025 4755 Acked Write 5 Done Done Sent false
16026 4756 Acked Read 7 Done Done Sent false
16027 4757 Acked Write 3 Done Done Sent false
16028 4758 Acked Flush 0 Done Done Sent false
16029 4759 Acked Write 5 Done Done Sent false
16030 4760 Acked Read 7 Done Done Sent false
16031 4761 Acked Read 4 Done Done Sent false
16032 4762 Acked Write 6 Done Done Sent false
16033 4763 Acked Read 5 Done Done Sent false
16034 4764 Acked Read 3 Done Done Sent false
16035 4765 Acked Read 3 Done Done Sent false
16036 4766 Acked Read 7 Done Done Sent false
16037 4767 Acked Write 10 Done Done Sent false
16038 4768 Acked Write 4 Done Done Sent false
16039 4769 Acked Write 7 Done Done Sent false
16040 4770 Acked Read 10 Done Done Sent false
16041 4771 Acked Write 2 Done Done Sent false
16042 4772 Acked Read 1 Done Done Sent false
16043 4773 Acked Read 2 Done Done Sent false
16044 4774 Acked Read 4 Done Done Sent false
16045 4775 Acked Write 2 Done Done Sent false
16046 4776 Acked Write 8 Done Done Sent false
16047 4777 Acked Read 1 Done Done Sent false
16048 4778 Acked Flush 0 Done Done Sent false
16049 4779 Acked Read 6 Done Done Sent false
16050 4780 Acked Write 7 Done Done Sent false
16051 4781 Acked Read 5 Done Done Sent false
16052 4782 Acked Write 7 Done Done Sent false
16053 4783 Acked Read 1 Done Done Sent false
16054 4784 Acked Read 10 Done Done Sent false
16055 4785 Acked Read 9 Done Done Sent false
16056 4786 Acked Read 7 Done Done Sent false
16057 4787 Acked Write 10 Done Done Sent false
16058 4788 Acked Read 4 Done Done Sent false
16059 4789 Acked Read 3 Done Done Sent false
16060 4790 Acked Read 8 Done Done Sent false
16061 4791 Acked Write 6 Done Done Sent false
16062 4792 Acked Read 7 Done Done Sent false
16063 4793 Acked Flush 0 Done Done Sent false
16064 4794 Acked Write 3 Done Done Sent false
16065 4795 Acked Read 4 Done Done Sent false
16066 4796 Acked Read 4 Done Done Sent false
16067 4797 Acked Write 1 Done Done Sent false
16068 4798 Acked Flush 0 Done Done Sent false
16069 4799 Acked Write 10 Done Done Sent false
16070 4800 Acked Write 1 Done Done Sent false
16071 4801 Acked Read 6 Done Done Sent false
16072 4802 Acked Write 5 Done Done Sent false
16073 4803 Acked Write 3 Done Done Sent false
16074 4804 Acked Write 3 Done Done Sent false
16075 4805 Acked Write 3 Done Done Sent false
16076 4806 Acked Read 9 Done Done Sent false
16077 4807 Acked Write 2 Done Done Sent false
16078 4808 Acked Write 4 Done Done Sent false
16079 4809 Acked Read 8 Done Done Sent false
16080 4810 Acked Read 7 Done Done Sent false
16081 4811 Acked Write 2 Done Done Sent false
16082 4812 Acked Read 6 Done Done Sent false
16083 4813 Acked Read 2 Done Done Sent false
16084 4814 Acked Read 4 Done Done Sent false
16085 4815 Acked Write 1 Done Done Sent false
16086 4816 Acked Write 10 Done Done Sent false
16087 4817 Acked Read 10 Done Done Sent false
16088 4818 Acked Read 7 Done Done Sent false
16089 4819 Acked Read 3 Done Done Sent false
16090 4820 Acked Flush 0 Done Done Sent false
16091 4821 Acked Write 9 Done Done Sent false
16092 4822 Acked Write 6 Done Done Sent false
16093 4823 Acked Read 1 Done Done Sent false
16094 4824 Acked Write 6 Done Done Sent false
16095 4825 Acked Write 3 Done Done Sent false
16096 4826 Acked Read 5 Done Done Sent false
16097 4827 Acked Write 8 Done Done Sent false
16098 4828 Acked Read 10 Done Done Sent false
16099 4829 Acked Read 1 Done Done Sent false
16100 4830 Acked Write 10 Done Done Sent false
16101 4831 Acked Flush 0 Done Done Sent false
16102 4832 Acked Read 3 Done Done Sent false
16103 4833 Acked Write 9 Done Done Sent false
16104 4834 Acked Write 7 Done Done Sent false
16105 4835 Acked Read 6 Done Done Sent false
16106 4836 Acked Read 9 Done Done Sent false
16107 4837 Acked Write 1 Done Done Sent false
16108 4838 Acked Read 5 Done Done Sent false
16109 4839 Acked Write 7 Done Done Sent false
16110 4840 Acked Read 7 Done Done Sent false
16111 4841 Acked Read 6 Done Done Sent false
16112 4842 Acked Read 9 Done Done Sent false
16113 4843 Acked Write 8 Done Done Sent false
16114 4844 Acked Write 10 Done Done Sent false
16115 4845 Acked Read 4 Done Done Sent false
16116 4846 Acked Read 7 Done Done Sent false
16117 4847 Acked Flush 0 Done Done Sent false
16118 4848 Acked Flush 0 Done Done Sent false
16119 4849 Acked Write 4 Done Done Sent false
16120 4850 Acked Read 4 Done Done Sent false
16121 4851 Acked Read 4 Done Done Sent false
16122 4852 Acked Read 2 Done Done Sent false
16123 4853 Acked Write 7 Done Done Sent false
16124 4854 Acked Write 10 Done Done Sent false
16125 4855 Acked Read 10 Done Done Sent false
16126 4856 Acked Read 4 Done Done Sent false
16127 4857 Acked Write 8 Done Done Sent false
16128 4858 Acked Read 5 Done Done Sent false
16129 4859 Acked Write 3 Done Done Sent false
16130 4860 Acked Write 6 Done Done Sent false
16131 4861 Acked Write 5 Done Done Sent false
16132 4862 Acked Write 3 Done Done Sent false
16133 4863 Acked Write 10 Done Done Sent false
16134 4864 Acked Read 1 Done Done Sent false
16135 4865 Acked Read 6 Done Done Sent false
16136 4866 Acked Write 6 Done Done Sent false
16137 4867 Acked Read 5 Done Done Sent false
16138 4868 Acked Write 6 Done Done Sent false
16139 4869 Acked Flush 0 Done Done Sent false
16140 4870 Acked Flush 0 Done Done Sent false
16141 4871 Acked Read 8 Done Done Sent false
16142 4872 Acked Write 10 Done Done Sent false
16143 4873 Acked Read 4 Done Done Sent false
16144 4874 Acked Read 3 Done Done Sent false
16145 4875 Acked Flush 0 Done Done Sent false
16146 4876 Acked Flush 0 Done Done Sent false
16147 4877 Acked Read 6 Done Done Sent false
16148 4878 Acked Write 10 Done Done Sent false
16149 4879 Acked Read 4 Done Done Sent false
16150 4880 Acked Read 7 Done Done Sent false
16151 4881 Acked Write 2 Done Done Sent false
16152 4882 Acked Write 3 Done Done Sent false
16153 4883 Acked Write 9 Done Done Sent false
16154 4884 Acked Write 8 Done Done Sent false
16155 4885 Acked Read 6 Done Done Sent false
16156 4886 Acked Write 8 Done Done Sent false
16157 4887 Acked Read 5 Done Done Sent false
16158 4888 Acked Read 4 Done Done Sent false
16159 4889 Acked Write 8 Done Done Sent false
16160 4890 Acked Read 6 Done Done Sent false
16161 4891 Acked Write 10 Done Done Sent false
16162 4892 Acked Flush 0 Done Done Sent false
16163 4893 Acked Flush 0 Done Done Sent false
16164 4894 Acked Read 10 Done Done Sent false
16165 4895 Acked Read 1 Done Done Sent false
16166 4896 Acked Write 7 Done Done Sent false
16167 4897 Acked Flush 0 Done Done Sent false
16168 4898 Acked Read 10 Done Done Sent false
16169 4899 Acked Write 4 Done Done Sent false
16170 4900 Acked Write 8 Done Done Sent false
16171 4901 Acked Read 2 Done Done Sent false
16172 4902 Acked Read 3 Done Done Sent false
16173 4903 Acked Write 3 Done Done Sent false
16174 4904 Acked Write 2 Done Done Sent false
16175 4905 Acked Write 4 Done Done Sent false
16176 4906 Acked Flush 0 Done Done Sent false
16177 4907 Acked Write 8 Done Done Sent false
16178 4908 Acked Read 6 Done Done Sent false
16179 4909 Acked Read 1 Done Done Sent false
16180 4910 Acked Write 1 Done Done Sent false
16181 4911 Acked Read 5 Done Done Sent false
16182 4912 Acked Write 4 Done Done Sent false
16183 4913 Acked Write 8 Done Done Sent false
16184 4914 Acked Write 2 Done Done Sent false
16185 4915 Acked Read 1 Done Done Sent false
16186 4916 Acked Write 8 Done Done Sent false
16187 4917 Acked Write 5 Done Done Sent false
16188 4918 Acked Read 7 Done Done Sent false
16189 4919 Acked Read 1 Done Done Sent false
16190 4920 Acked Read 3 Done Done Sent false
16191 4921 Acked Write 8 Done Done Sent false
16192 4922 Acked Read 9 Done Done Sent false
16193 4923 Acked Write 5 Done Done Sent false
16194 4924 Acked Write 3 Done Done Sent false
16195 STATES DS:0 DS:1 DS:2 TOTAL
16196 Sent 0 0 301 301
16197 Done 301 301 0 602
16198 Skipped 0 0 0 0
16199 Error 0 0 0 0
16200 Last Flush: Some(JobId(4906)) Some(JobId(4906)) Some(JobId(4623))
16201 Downstairs last five completed: 4623 4622 4621 4620 4619
16202 Upstairs last five completed: 4924 4923 4922 4921 4920
16203 CLIENT: Up:0 ds:301 act:2
162042025-02-07T20:10:53.292ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
162052025-02-07T20:10:54.293ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
162062025-02-07T20:10:54.530ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
162072025-02-07T20:10:54.530ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
162082025-02-07T20:10:54.530ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
162092025-02-07T20:10:54.530ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
162102025-02-07T20:10:54.530ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
162112025-02-07T20:10:55.294ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
162122025-02-07T20:10:56.296ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
16213 ----------------------------------------------------------------
16214 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16215 States: Active Active Offline
16216 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16217 4624 Acked Write 7 Done Done Sent false
16218 4625 Acked Read 6 Done Done Sent false
16219 4626 Acked Write 4 Done Done Sent false
16220 4627 Acked Write 3 Done Done Sent false
16221 4628 Acked Write 8 Done Done Sent false
16222 4629 Acked Write 2 Done Done Sent false
16223 4630 Acked Write 8 Done Done Sent false
16224 4631 Acked Flush 0 Done Done Sent false
16225 4632 Acked Read 9 Done Done Sent false
16226 4633 Acked Read 3 Done Done Sent false
16227 4634 Acked Read 3 Done Done Sent false
16228 4635 Acked Write 6 Done Done Sent false
16229 4636 Acked Write 5 Done Done Sent false
16230 4637 Acked Write 7 Done Done Sent false
16231 4638 Acked Read 10 Done Done Sent false
16232 4639 Acked Write 9 Done Done Sent false
16233 4640 Acked Flush 0 Done Done Sent false
16234 4641 Acked Write 9 Done Done Sent false
16235 4642 Acked Read 1 Done Done Sent false
16236 4643 Acked Read 6 Done Done Sent false
16237 4644 Acked Write 10 Done Done Sent false
16238 4645 Acked Flush 0 Done Done Sent false
16239 4646 Acked Write 5 Done Done Sent false
16240 4647 Acked Write 5 Done Done Sent false
16241 4648 Acked Flush 0 Done Done Sent false
16242 4649 Acked Write 7 Done Done Sent false
16243 4650 Acked Read 6 Done Done Sent false
16244 4651 Acked Write 7 Done Done Sent false
16245 4652 Acked Read 3 Done Done Sent false
16246 4653 Acked Write 5 Done Done Sent false
16247 4654 Acked Read 6 Done Done Sent false
16248 4655 Acked Read 5 Done Done Sent false
16249 4656 Acked Write 2 Done Done Sent false
16250 4657 Acked Read 8 Done Done Sent false
16251 4658 Acked Read 1 Done Done Sent false
16252 4659 Acked Flush 0 Done Done Sent false
16253 4660 Acked Read 6 Done Done Sent false
16254 4661 Acked Read 10 Done Done Sent false
16255 4662 Acked Write 5 Done Done Sent false
16256 4663 Acked Flush 0 Done Done Sent false
16257 4664 Acked Read 3 Done Done Sent false
16258 4665 Acked Read 2 Done Done Sent false
16259 4666 Acked Read 3 Done Done Sent false
16260 4667 Acked Write 1 Done Done Sent false
16261 4668 Acked Write 7 Done Done Sent false
16262 4669 Acked Write 10 Done Done Sent false
16263 4670 Acked Write 6 Done Done Sent false
16264 4671 Acked Read 1 Done Done Sent false
16265 4672 Acked Read 2 Done Done Sent false
16266 4673 Acked Write 2 Done Done Sent false
16267 4674 Acked Write 5 Done Done Sent false
16268 4675 Acked Read 8 Done Done Sent false
16269 4676 Acked Flush 0 Done Done Sent false
16270 4677 Acked Write 5 Done Done Sent false
16271 4678 Acked Read 7 Done Done Sent false
16272 4679 Acked Write 5 Done Done Sent false
16273 4680 Acked Write 2 Done Done Sent false
16274 4681 Acked Read 7 Done Done Sent false
16275 4682 Acked Read 8 Done Done Sent false
16276 4683 Acked Write 3 Done Done Sent false
16277 4684 Acked Read 2 Done Done Sent false
16278 4685 Acked Write 2 Done Done Sent false
16279 4686 Acked Read 9 Done Done Sent false
16280 4687 Acked Read 10 Done Done Sent false
16281 4688 Acked Write 10 Done Done Sent false
16282 4689 Acked Read 9 Done Done Sent false
16283 4690 Acked Read 9 Done Done Sent false
16284 4691 Acked Read 6 Done Done Sent false
16285 4692 Acked Write 8 Done Done Sent false
16286 4693 Acked Write 10 Done Done Sent false
16287 4694 Acked Write 6 Done Done Sent false
16288 4695 Acked Write 8 Done Done Sent false
16289 4696 Acked Read 8 Done Done Sent false
16290 4697 Acked Read 4 Done Done Sent false
16291 4698 Acked Read 7 Done Done Sent false
16292 4699 Acked Read 2 Done Done Sent false
16293 4700 Acked Write 7 Done Done Sent false
16294 4701 Acked Write 2 Done Done Sent false
16295 4702 Acked Read 2 Done Done Sent false
16296 4703 Acked Write 5 Done Done Sent false
16297 4704 Acked Read 7 Done Done Sent false
16298 4705 Acked Write 10 Done Done Sent false
16299 4706 Acked Read 4 Done Done Sent false
16300 4707 Acked Read 9 Done Done Sent false
16301 4708 Acked Read 8 Done Done Sent false
16302 4709 Acked Write 1 Done Done Sent false
16303 4710 Acked Write 1 Done Done Sent false
16304 4711 Acked Write 9 Done Done Sent false
16305 4712 Acked Write 3 Done Done Sent false
16306 4713 Acked Write 10 Done Done Sent false
16307 4714 Acked Read 3 Done Done Sent false
16308 4715 Acked Write 1 Done Done Sent false
16309 4716 Acked Write 6 Done Done Sent false
16310 4717 Acked Write 10 Done Done Sent false
16311 4718 Acked Read 9 Done Done Sent false
16312 4719 Acked Write 6 Done Done Sent false
16313 4720 Acked Read 4 Done Done Sent false
16314 4721 Acked Flush 0 Done Done Sent false
16315 4722 Acked Write 10 Done Done Sent false
16316 4723 Acked Write 9 Done Done Sent false
16317 4724 Acked Flush 0 Done Done Sent false
16318 4725 Acked Write 5 Done Done Sent false
16319 4726 Acked Flush 0 Done Done Sent false
16320 4727 Acked Read 8 Done Done Sent false
16321 4728 Acked Write 3 Done Done Sent false
16322 4729 Acked Read 8 Done Done Sent false
16323 4730 Acked Write 10 Done Done Sent false
16324 4731 Acked Flush 0 Done Done Sent false
16325 4732 Acked Write 8 Done Done Sent false
16326 4733 Acked Read 3 Done Done Sent false
16327 4734 Acked Read 4 Done Done Sent false
16328 4735 Acked Flush 0 Done Done Sent false
16329 4736 Acked Write 9 Done Done Sent false
16330 4737 Acked Write 2 Done Done Sent false
16331 4738 Acked Write 10 Done Done Sent false
16332 4739 Acked Write 5 Done Done Sent false
16333 4740 Acked Flush 0 Done Done Sent false
16334 4741 Acked Read 2 Done Done Sent false
16335 4742 Acked Read 5 Done Done Sent false
16336 4743 Acked Read 5 Done Done Sent false
16337 4744 Acked Write 1 Done Done Sent false
16338 4745 Acked Write 5 Done Done Sent false
16339 4746 Acked Write 2 Done Done Sent false
16340 4747 Acked Read 2 Done Done Sent false
16341 4748 Acked Read 2 Done Done Sent false
16342 4749 Acked Write 2 Done Done Sent false
16343 4750 Acked Flush 0 Done Done Sent false
16344 4751 Acked Write 2 Done Done Sent false
16345 4752 Acked Flush 0 Done Done Sent false
16346 4753 Acked Write 10 Done Done Sent false
16347 4754 Acked Write 6 Done Done Sent false
16348 4755 Acked Write 5 Done Done Sent false
16349 4756 Acked Read 7 Done Done Sent false
16350 4757 Acked Write 3 Done Done Sent false
16351 4758 Acked Flush 0 Done Done Sent false
16352 4759 Acked Write 5 Done Done Sent false
16353 4760 Acked Read 7 Done Done Sent false
16354 4761 Acked Read 4 Done Done Sent false
16355 4762 Acked Write 6 Done Done Sent false
16356 4763 Acked Read 5 Done Done Sent false
16357 4764 Acked Read 3 Done Done Sent false
16358 4765 Acked Read 3 Done Done Sent false
16359 4766 Acked Read 7 Done Done Sent false
16360 4767 Acked Write 10 Done Done Sent false
16361 4768 Acked Write 4 Done Done Sent false
16362 4769 Acked Write 7 Done Done Sent false
16363 4770 Acked Read 10 Done Done Sent false
16364 4771 Acked Write 2 Done Done Sent false
16365 4772 Acked Read 1 Done Done Sent false
16366 4773 Acked Read 2 Done Done Sent false
16367 4774 Acked Read 4 Done Done Sent false
16368 4775 Acked Write 2 Done Done Sent false
16369 4776 Acked Write 8 Done Done Sent false
16370 4777 Acked Read 1 Done Done Sent false
16371 4778 Acked Flush 0 Done Done Sent false
16372 4779 Acked Read 6 Done Done Sent false
16373 4780 Acked Write 7 Done Done Sent false
16374 4781 Acked Read 5 Done Done Sent false
16375 4782 Acked Write 7 Done Done Sent false
16376 4783 Acked Read 1 Done Done Sent false
16377 4784 Acked Read 10 Done Done Sent false
16378 4785 Acked Read 9 Done Done Sent false
16379 4786 Acked Read 7 Done Done Sent false
16380 4787 Acked Write 10 Done Done Sent false
16381 4788 Acked Read 4 Done Done Sent false
16382 4789 Acked Read 3 Done Done Sent false
16383 4790 Acked Read 8 Done Done Sent false
16384 4791 Acked Write 6 Done Done Sent false
16385 4792 Acked Read 7 Done Done Sent false
16386 4793 Acked Flush 0 Done Done Sent false
16387 4794 Acked Write 3 Done Done Sent false
16388 4795 Acked Read 4 Done Done Sent false
16389 4796 Acked Read 4 Done Done Sent false
16390 4797 Acked Write 1 Done Done Sent false
16391 4798 Acked Flush 0 Done Done Sent false
16392 4799 Acked Write 10 Done Done Sent false
16393 4800 Acked Write 1 Done Done Sent false
16394 4801 Acked Read 6 Done Done Sent false
16395 4802 Acked Write 5 Done Done Sent false
16396 4803 Acked Write 3 Done Done Sent false
16397 4804 Acked Write 3 Done Done Sent false
16398 4805 Acked Write 3 Done Done Sent false
16399 4806 Acked Read 9 Done Done Sent false
16400 4807 Acked Write 2 Done Done Sent false
16401 4808 Acked Write 4 Done Done Sent false
16402 4809 Acked Read 8 Done Done Sent false
16403 4810 Acked Read 7 Done Done Sent false
16404 4811 Acked Write 2 Done Done Sent false
16405 4812 Acked Read 6 Done Done Sent false
16406 4813 Acked Read 2 Done Done Sent false
16407 4814 Acked Read 4 Done Done Sent false
16408 4815 Acked Write 1 Done Done Sent false
16409 4816 Acked Write 10 Done Done Sent false
16410 4817 Acked Read 10 Done Done Sent false
16411 4818 Acked Read 7 Done Done Sent false
16412 4819 Acked Read 3 Done Done Sent false
16413 4820 Acked Flush 0 Done Done Sent false
16414 4821 Acked Write 9 Done Done Sent false
16415 4822 Acked Write 6 Done Done Sent false
16416 4823 Acked Read 1 Done Done Sent false
16417 4824 Acked Write 6 Done Done Sent false
16418 4825 Acked Write 3 Done Done Sent false
16419 4826 Acked Read 5 Done Done Sent false
16420 4827 Acked Write 8 Done Done Sent false
16421 4828 Acked Read 10 Done Done Sent false
16422 4829 Acked Read 1 Done Done Sent false
16423 4830 Acked Write 10 Done Done Sent false
16424 4831 Acked Flush 0 Done Done Sent false
16425 4832 Acked Read 3 Done Done Sent false
16426 4833 Acked Write 9 Done Done Sent false
16427 4834 Acked Write 7 Done Done Sent false
16428 4835 Acked Read 6 Done Done Sent false
16429 4836 Acked Read 9 Done Done Sent false
16430 4837 Acked Write 1 Done Done Sent false
16431 4838 Acked Read 5 Done Done Sent false
16432 4839 Acked Write 7 Done Done Sent false
16433 4840 Acked Read 7 Done Done Sent false
16434 4841 Acked Read 6 Done Done Sent false
16435 4842 Acked Read 9 Done Done Sent false
16436 4843 Acked Write 8 Done Done Sent false
16437 4844 Acked Write 10 Done Done Sent false
16438 4845 Acked Read 4 Done Done Sent false
16439 4846 Acked Read 7 Done Done Sent false
16440 4847 Acked Flush 0 Done Done Sent false
16441 4848 Acked Flush 0 Done Done Sent false
16442 4849 Acked Write 4 Done Done Sent false
16443 4850 Acked Read 4 Done Done Sent false
16444 4851 Acked Read 4 Done Done Sent false
16445 4852 Acked Read 2 Done Done Sent false
16446 4853 Acked Write 7 Done Done Sent false
16447 4854 Acked Write 10 Done Done Sent false
16448 4855 Acked Read 10 Done Done Sent false
16449 4856 Acked Read 4 Done Done Sent false
16450 4857 Acked Write 8 Done Done Sent false
16451 4858 Acked Read 5 Done Done Sent false
16452 4859 Acked Write 3 Done Done Sent false
16453 4860 Acked Write 6 Done Done Sent false
16454 4861 Acked Write 5 Done Done Sent false
16455 4862 Acked Write 3 Done Done Sent false
16456 4863 Acked Write 10 Done Done Sent false
16457 4864 Acked Read 1 Done Done Sent false
16458 4865 Acked Read 6 Done Done Sent false
16459 4866 Acked Write 6 Done Done Sent false
16460 4867 Acked Read 5 Done Done Sent false
16461 4868 Acked Write 6 Done Done Sent false
16462 4869 Acked Flush 0 Done Done Sent false
16463 4870 Acked Flush 0 Done Done Sent false
16464 4871 Acked Read 8 Done Done Sent false
16465 4872 Acked Write 10 Done Done Sent false
16466 4873 Acked Read 4 Done Done Sent false
16467 4874 Acked Read 3 Done Done Sent false
16468 4875 Acked Flush 0 Done Done Sent false
16469 4876 Acked Flush 0 Done Done Sent false
16470 4877 Acked Read 6 Done Done Sent false
16471 4878 Acked Write 10 Done Done Sent false
16472 4879 Acked Read 4 Done Done Sent false
16473 4880 Acked Read 7 Done Done Sent false
16474 4881 Acked Write 2 Done Done Sent false
16475 4882 Acked Write 3 Done Done Sent false
16476 4883 Acked Write 9 Done Done Sent false
16477 4884 Acked Write 8 Done Done Sent false
16478 4885 Acked Read 6 Done Done Sent false
16479 4886 Acked Write 8 Done Done Sent false
16480 4887 Acked Read 5 Done Done Sent false
16481 4888 Acked Read 4 Done Done Sent false
16482 4889 Acked Write 8 Done Done Sent false
16483 4890 Acked Read 6 Done Done Sent false
16484 4891 Acked Write 10 Done Done Sent false
16485 4892 Acked Flush 0 Done Done Sent false
16486 4893 Acked Flush 0 Done Done Sent false
16487 4894 Acked Read 10 Done Done Sent false
16488 4895 Acked Read 1 Done Done Sent false
16489 4896 Acked Write 7 Done Done Sent false
16490 4897 Acked Flush 0 Done Done Sent false
16491 4898 Acked Read 10 Done Done Sent false
16492 4899 Acked Write 4 Done Done Sent false
16493 4900 Acked Write 8 Done Done Sent false
16494 4901 Acked Read 2 Done Done Sent false
16495 4902 Acked Read 3 Done Done Sent false
16496 4903 Acked Write 3 Done Done Sent false
16497 4904 Acked Write 2 Done Done Sent false
16498 4905 Acked Write 4 Done Done Sent false
16499 4906 Acked Flush 0 Done Done Sent false
16500 4907 Acked Write 8 Done Done Sent false
16501 4908 Acked Read 6 Done Done Sent false
16502 4909 Acked Read 1 Done Done Sent false
16503 4910 Acked Write 1 Done Done Sent false
16504 4911 Acked Read 5 Done Done Sent false
16505 4912 Acked Write 4 Done Done Sent false
16506 4913 Acked Write 8 Done Done Sent false
16507 4914 Acked Write 2 Done Done Sent false
16508 4915 Acked Read 1 Done Done Sent false
16509 4916 Acked Write 8 Done Done Sent false
16510 4917 Acked Write 5 Done Done Sent false
16511 4918 Acked Read 7 Done Done Sent false
16512 4919 Acked Read 1 Done Done Sent false
16513 4920 Acked Read 3 Done Done Sent false
16514 4921 Acked Write 8 Done Done Sent false
16515 4922 Acked Read 9 Done Done Sent false
16516 4923 Acked Write 5 Done Done Sent false
16517 4924 Acked Write 3 Done Done Sent false
16518 STATES DS:0 DS:1 DS:2 TOTAL
16519 Sent 0 0 301 301
16520 Done 301 301 0 602
16521 Skipped 0 0 0 0
16522 Error 0 0 0 0
16523 Last Flush: Some(JobId(4906)) Some(JobId(4906)) Some(JobId(4623))
16524 Downstairs last five completed: 4623 4622 4621 4620 4619
16525 Upstairs last five completed: 4924 4923 4922 4921 4920
16526 CLIENT: Up:0 ds:301 act:2
165272025-02-07T20:10:57.771ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
165282025-02-07T20:10:58.299ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
165292025-02-07T20:10:59.301ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
165302025-02-07T20:11:00.301ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
16531 [8830] got start action
16532 [8830] I am going to restart
16533 Starting downstairs at port 8830
16534 Make output file at "/tmp/dsc/downstairs-8830.txt"
16535 [8830][2] reports Starting
16536 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1222)
16537 [8830][2] reports Running
16538 ----------------------------------------------------------------
16539 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16540 States: Active Active Offline
16541 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16542 4624 Acked Write 7 Done Done Sent false
16543 4625 Acked Read 6 Done Done Sent false
16544 4626 Acked Write 4 Done Done Sent false
16545 4627 Acked Write 3 Done Done Sent false
16546 4628 Acked Write 8 Done Done Sent false
16547 4629 Acked Write 2 Done Done Sent false
16548 4630 Acked Write 8 Done Done Sent false
16549 4631 Acked Flush 0 Done Done Sent false
16550 4632 Acked Read 9 Done Done Sent false
16551 4633 Acked Read 3 Done Done Sent false
16552 4634 Acked Read 3 Done Done Sent false
16553 4635 Acked Write 6 Done Done Sent false
16554 4636 Acked Write 5 Done Done Sent false
16555 4637 Acked Write 7 Done Done Sent false
16556 4638 Acked Read 10 Done Done Sent false
16557 4639 Acked Write 9 Done Done Sent false
16558 4640 Acked Flush 0 Done Done Sent false
16559 4641 Acked Write 9 Done Done Sent false
16560 4642 Acked Read 1 Done Done Sent false
16561 4643 Acked Read 6 Done Done Sent false
16562 4644 Acked Write 10 Done Done Sent false
16563 4645 Acked Flush 0 Done Done Sent false
16564 4646 Acked Write 5 Done Done Sent false
16565 4647 Acked Write 5 Done Done Sent false
16566 4648 Acked Flush 0 Done Done Sent false
16567 4649 Acked Write 7 Done Done Sent false
16568 4650 Acked Read 6 Done Done Sent false
16569 4651 Acked Write 7 Done Done Sent false
16570 4652 Acked Read 3 Done Done Sent false
16571 4653 Acked Write 5 Done Done Sent false
16572 4654 Acked Read 6 Done Done Sent false
16573 4655 Acked Read 5 Done Done Sent false
16574 4656 Acked Write 2 Done Done Sent false
16575 4657 Acked Read 8 Done Done Sent false
16576 4658 Acked Read 1 Done Done Sent false
16577 4659 Acked Flush 0 Done Done Sent false
16578 4660 Acked Read 6 Done Done Sent false
16579 4661 Acked Read 10 Done Done Sent false
16580 4662 Acked Write 5 Done Done Sent false
16581 4663 Acked Flush 0 Done Done Sent false
16582 4664 Acked Read 3 Done Done Sent false
16583 4665 Acked Read 2 Done Done Sent false
16584 4666 Acked Read 3 Done Done Sent false
16585 4667 Acked Write 1 Done Done Sent false
16586 4668 Acked Write 7 Done Done Sent false
16587 4669 Acked Write 10 Done Done Sent false
16588 4670 Acked Write 6 Done Done Sent false
16589 4671 Acked Read 1 Done Done Sent false
16590 4672 Acked Read 2 Done Done Sent false
16591 4673 Acked Write 2 Done Done Sent false
16592 4674 Acked Write 5 Done Done Sent false
16593 4675 Acked Read 8 Done Done Sent false
16594 4676 Acked Flush 0 Done Done Sent false
16595 4677 Acked Write 5 Done Done Sent false
16596 4678 Acked Read 7 Done Done Sent false
16597 4679 Acked Write 5 Done Done Sent false
16598 4680 Acked Write 2 Done Done Sent false
16599 4681 Acked Read 7 Done Done Sent false
16600 4682 Acked Read 8 Done Done Sent false
16601 4683 Acked Write 3 Done Done Sent false
16602 4684 Acked Read 2 Done Done Sent false
16603 4685 Acked Write 2 Done Done Sent false
16604 4686 Acked Read 9 Done Done Sent false
16605 4687 Acked Read 10 Done Done Sent false
16606 4688 Acked Write 10 Done Done Sent false
16607 4689 Acked Read 9 Done Done Sent false
16608 4690 Acked Read 9 Done Done Sent false
16609 4691 Acked Read 6 Done Done Sent false
16610 4692 Acked Write 8 Done Done Sent false
16611 4693 Acked Write 10 Done Done Sent false
16612 4694 Acked Write 6 Done Done Sent false
16613 4695 Acked Write 8 Done Done Sent false
16614 4696 Acked Read 8 Done Done Sent false
16615 4697 Acked Read 4 Done Done Sent false
16616 4698 Acked Read 7 Done Done Sent false
16617 4699 Acked Read 2 Done Done Sent false
16618 4700 Acked Write 7 Done Done Sent false
16619 4701 Acked Write 2 Done Done Sent false
16620 4702 Acked Read 2 Done Done Sent false
16621 4703 Acked Write 5 Done Done Sent false
16622 4704 Acked Read 7 Done Done Sent false
16623 4705 Acked Write 10 Done Done Sent false
16624 4706 Acked Read 4 Done Done Sent false
16625 4707 Acked Read 9 Done Done Sent false
16626 4708 Acked Read 8 Done Done Sent false
16627 4709 Acked Write 1 Done Done Sent false
16628 4710 Acked Write 1 Done Done Sent false
16629 4711 Acked Write 9 Done Done Sent false
16630 4712 Acked Write 3 Done Done Sent false
16631 4713 Acked Write 10 Done Done Sent false
16632 4714 Acked Read 3 Done Done Sent false
16633 4715 Acked Write 1 Done Done Sent false
16634 4716 Acked Write 6 Done Done Sent false
16635 4717 Acked Write 10 Done Done Sent false
16636 4718 Acked Read 9 Done Done Sent false
16637 4719 Acked Write 6 Done Done Sent false
16638 4720 Acked Read 4 Done Done Sent false
16639 4721 Acked Flush 0 Done Done Sent false
16640 4722 Acked Write 10 Done Done Sent false
16641 4723 Acked Write 9 Done Done Sent false
16642 4724 Acked Flush 0 Done Done Sent false
16643 4725 Acked Write 5 Done Done Sent false
16644 4726 Acked Flush 0 Done Done Sent false
16645 4727 Acked Read 8 Done Done Sent false
16646 4728 Acked Write 3 Done Done Sent false
16647 4729 Acked Read 8 Done Done Sent false
16648 4730 Acked Write 10 Done Done Sent false
16649 4731 Acked Flush 0 Done Done Sent false
16650 4732 Acked Write 8 Done Done Sent false
16651 4733 Acked Read 3 Done Done Sent false
16652 4734 Acked Read 4 Done Done Sent false
16653 4735 Acked Flush 0 Done Done Sent false
16654 4736 Acked Write 9 Done Done Sent false
16655 4737 Acked Write 2 Done Done Sent false
16656 4738 Acked Write 10 Done Done Sent false
16657 4739 Acked Write 5 Done Done Sent false
16658 4740 Acked Flush 0 Done Done Sent false
16659 4741 Acked Read 2 Done Done Sent false
16660 4742 Acked Read 5 Done Done Sent false
16661 4743 Acked Read 5 Done Done Sent false
16662 4744 Acked Write 1 Done Done Sent false
16663 4745 Acked Write 5 Done Done Sent false
16664 4746 Acked Write 2 Done Done Sent false
16665 4747 Acked Read 2 Done Done Sent false
16666 4748 Acked Read 2 Done Done Sent false
16667 4749 Acked Write 2 Done Done Sent false
16668 4750 Acked Flush 0 Done Done Sent false
16669 4751 Acked Write 2 Done Done Sent false
16670 4752 Acked Flush 0 Done Done Sent false
16671 4753 Acked Write 10 Done Done Sent false
16672 4754 Acked Write 6 Done Done Sent false
16673 4755 Acked Write 5 Done Done Sent false
16674 4756 Acked Read 7 Done Done Sent false
16675 4757 Acked Write 3 Done Done Sent false
16676 4758 Acked Flush 0 Done Done Sent false
16677 4759 Acked Write 5 Done Done Sent false
16678 4760 Acked Read 7 Done Done Sent false
16679 4761 Acked Read 4 Done Done Sent false
16680 4762 Acked Write 6 Done Done Sent false
16681 4763 Acked Read 5 Done Done Sent false
16682 4764 Acked Read 3 Done Done Sent false
16683 4765 Acked Read 3 Done Done Sent false
16684 4766 Acked Read 7 Done Done Sent false
16685 4767 Acked Write 10 Done Done Sent false
16686 4768 Acked Write 4 Done Done Sent false
16687 4769 Acked Write 7 Done Done Sent false
16688 4770 Acked Read 10 Done Done Sent false
16689 4771 Acked Write 2 Done Done Sent false
16690 4772 Acked Read 1 Done Done Sent false
16691 4773 Acked Read 2 Done Done Sent false
16692 4774 Acked Read 4 Done Done Sent false
16693 4775 Acked Write 2 Done Done Sent false
16694 4776 Acked Write 8 Done Done Sent false
16695 4777 Acked Read 1 Done Done Sent false
16696 4778 Acked Flush 0 Done Done Sent false
16697 4779 Acked Read 6 Done Done Sent false
16698 4780 Acked Write 7 Done Done Sent false
16699 4781 Acked Read 5 Done Done Sent false
16700 4782 Acked Write 7 Done Done Sent false
16701 4783 Acked Read 1 Done Done Sent false
16702 4784 Acked Read 10 Done Done Sent false
16703 4785 Acked Read 9 Done Done Sent false
16704 4786 Acked Read 7 Done Done Sent false
16705 4787 Acked Write 10 Done Done Sent false
16706 4788 Acked Read 4 Done Done Sent false
16707 4789 Acked Read 3 Done Done Sent false
16708 4790 Acked Read 8 Done Done Sent false
16709 4791 Acked Write 6 Done Done Sent false
16710 4792 Acked Read 7 Done Done Sent false
16711 4793 Acked Flush 0 Done Done Sent false
16712 4794 Acked Write 3 Done Done Sent false
16713 4795 Acked Read 4 Done Done Sent false
16714 4796 Acked Read 4 Done Done Sent false
16715 4797 Acked Write 1 Done Done Sent false
16716 4798 Acked Flush 0 Done Done Sent false
16717 4799 Acked Write 10 Done Done Sent false
16718 4800 Acked Write 1 Done Done Sent false
16719 4801 Acked Read 6 Done Done Sent false
16720 4802 Acked Write 5 Done Done Sent false
16721 4803 Acked Write 3 Done Done Sent false
16722 4804 Acked Write 3 Done Done Sent false
16723 4805 Acked Write 3 Done Done Sent false
16724 4806 Acked Read 9 Done Done Sent false
16725 4807 Acked Write 2 Done Done Sent false
16726 4808 Acked Write 4 Done Done Sent false
16727 4809 Acked Read 8 Done Done Sent false
16728 4810 Acked Read 7 Done Done Sent false
16729 4811 Acked Write 2 Done Done Sent false
16730 4812 Acked Read 6 Done Done Sent false
16731 4813 Acked Read 2 Done Done Sent false
16732 4814 Acked Read 4 Done Done Sent false
16733 4815 Acked Write 1 Done Done Sent false
16734 4816 Acked Write 10 Done Done Sent false
16735 4817 Acked Read 10 Done Done Sent false
16736 4818 Acked Read 7 Done Done Sent false
16737 4819 Acked Read 3 Done Done Sent false
16738 4820 Acked Flush 0 Done Done Sent false
16739 4821 Acked Write 9 Done Done Sent false
16740 4822 Acked Write 6 Done Done Sent false
16741 4823 Acked Read 1 Done Done Sent false
16742 4824 Acked Write 6 Done Done Sent false
16743 4825 Acked Write 3 Done Done Sent false
16744 4826 Acked Read 5 Done Done Sent false
16745 4827 Acked Write 8 Done Done Sent false
16746 4828 Acked Read 10 Done Done Sent false
16747 4829 Acked Read 1 Done Done Sent false
16748 4830 Acked Write 10 Done Done Sent false
16749 4831 Acked Flush 0 Done Done Sent false
16750 4832 Acked Read 3 Done Done Sent false
16751 4833 Acked Write 9 Done Done Sent false
16752 4834 Acked Write 7 Done Done Sent false
16753 4835 Acked Read 6 Done Done Sent false
16754 4836 Acked Read 9 Done Done Sent false
16755 4837 Acked Write 1 Done Done Sent false
16756 4838 Acked Read 5 Done Done Sent false
16757 4839 Acked Write 7 Done Done Sent false
16758 4840 Acked Read 7 Done Done Sent false
16759 4841 Acked Read 6 Done Done Sent false
16760 4842 Acked Read 9 Done Done Sent false
16761 4843 Acked Write 8 Done Done Sent false
16762 4844 Acked Write 10 Done Done Sent false
16763 4845 Acked Read 4 Done Done Sent false
16764 4846 Acked Read 7 Done Done Sent false
16765 4847 Acked Flush 0 Done Done Sent false
16766 4848 Acked Flush 0 Done Done Sent false
16767 4849 Acked Write 4 Done Done Sent false
16768 4850 Acked Read 4 Done Done Sent false
16769 4851 Acked Read 4 Done Done Sent false
16770 4852 Acked Read 2 Done Done Sent false
16771 4853 Acked Write 7 Done Done Sent false
16772 4854 Acked Write 10 Done Done Sent false
16773 4855 Acked Read 10 Done Done Sent false
16774 4856 Acked Read 4 Done Done Sent false
16775 4857 Acked Write 8 Done Done Sent false
16776 4858 Acked Read 5 Done Done Sent false
16777 4859 Acked Write 3 Done Done Sent false
16778 4860 Acked Write 6 Done Done Sent false
16779 4861 Acked Write 5 Done Done Sent false
16780 4862 Acked Write 3 Done Done Sent false
16781 4863 Acked Write 10 Done Done Sent false
16782 4864 Acked Read 1 Done Done Sent false
16783 4865 Acked Read 6 Done Done Sent false
16784 4866 Acked Write 6 Done Done Sent false
16785 4867 Acked Read 5 Done Done Sent false
16786 4868 Acked Write 6 Done Done Sent false
16787 4869 Acked Flush 0 Done Done Sent false
16788 4870 Acked Flush 0 Done Done Sent false
16789 4871 Acked Read 8 Done Done Sent false
16790 4872 Acked Write 10 Done Done Sent false
16791 4873 Acked Read 4 Done Done Sent false
16792 4874 Acked Read 3 Done Done Sent false
16793 4875 Acked Flush 0 Done Done Sent false
16794 4876 Acked Flush 0 Done Done Sent false
16795 4877 Acked Read 6 Done Done Sent false
16796 4878 Acked Write 10 Done Done Sent false
16797 4879 Acked Read 4 Done Done Sent false
16798 4880 Acked Read 7 Done Done Sent false
16799 4881 Acked Write 2 Done Done Sent false
16800 4882 Acked Write 3 Done Done Sent false
16801 4883 Acked Write 9 Done Done Sent false
16802 4884 Acked Write 8 Done Done Sent false
16803 4885 Acked Read 6 Done Done Sent false
16804 4886 Acked Write 8 Done Done Sent false
16805 4887 Acked Read 5 Done Done Sent false
16806 4888 Acked Read 4 Done Done Sent false
16807 4889 Acked Write 8 Done Done Sent false
16808 4890 Acked Read 6 Done Done Sent false
16809 4891 Acked Write 10 Done Done Sent false
16810 4892 Acked Flush 0 Done Done Sent false
16811 4893 Acked Flush 0 Done Done Sent false
16812 4894 Acked Read 10 Done Done Sent false
16813 4895 Acked Read 1 Done Done Sent false
16814 4896 Acked Write 7 Done Done Sent false
16815 4897 Acked Flush 0 Done Done Sent false
16816 4898 Acked Read 10 Done Done Sent false
16817 4899 Acked Write 4 Done Done Sent false
16818 4900 Acked Write 8 Done Done Sent false
16819 4901 Acked Read 2 Done Done Sent false
16820 4902 Acked Read 3 Done Done Sent false
16821 4903 Acked Write 3 Done Done Sent false
16822 4904 Acked Write 2 Done Done Sent false
16823 4905 Acked Write 4 Done Done Sent false
16824 4906 Acked Flush 0 Done Done Sent false
16825 4907 Acked Write 8 Done Done Sent false
16826 4908 Acked Read 6 Done Done Sent false
16827 4909 Acked Read 1 Done Done Sent false
16828 4910 Acked Write 1 Done Done Sent false
16829 4911 Acked Read 5 Done Done Sent false
16830 4912 Acked Write 4 Done Done Sent false
16831 4913 Acked Write 8 Done Done Sent false
16832 4914 Acked Write 2 Done Done Sent false
16833 4915 Acked Read 1 Done Done Sent false
16834 4916 Acked Write 8 Done Done Sent false
16835 4917 Acked Write 5 Done Done Sent false
16836 4918 Acked Read 7 Done Done Sent false
16837 4919 Acked Read 1 Done Done Sent false
16838 4920 Acked Read 3 Done Done Sent false
16839 4921 Acked Write 8 Done Done Sent false
16840 4922 Acked Read 9 Done Done Sent false
16841 4923 Acked Write 5 Done Done Sent false
16842 4924 Acked Write 3 Done Done Sent false
16843 STATES DS:0 DS:1 DS:2 TOTAL
16844 Sent 0 0 301 301
16845 Done 301 301 0 602
16846 Skipped 0 0 0 0
16847 Error 0 0 0 0
16848 Last Flush: Some(JobId(4906)) Some(JobId(4906)) Some(JobId(4623))
16849 Downstairs last five completed: 4623 4622 4621 4620 4619
16850 Upstairs last five completed: 4924 4923 4922 4921 4920
16851 CLIENT: Up:0 ds:301 act:2
168522025-02-07T20:11:01.303ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
168532025-02-07T20:11:02.773ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
168542025-02-07T20:11:03.305ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
168552025-02-07T20:11:04.306ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
168562025-02-07T20:11:04.532ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
168572025-02-07T20:11:04.532ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49159) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
168582025-02-07T20:11:04.533ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
168592025-02-07T20:11:04.534ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
168602025-02-07T20:11:04.534ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
168612025-02-07T20:11:04.534ZINFOcrucible: send last flush ID to this DS: Some(JobId(4623)) = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
168622025-02-07T20:11:04.534ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
168632025-02-07T20:11:04.534ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4623)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
168642025-02-07T20:11:04.534ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4623)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
16865 ----------------------------------------------------------------
16866 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:18
16867 States: Active Active Active
16868 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16869 4907 Acked Write 8 Done Done Done true
16870 4908 Acked Read 6 Done Done Done true
16871 4909 Acked Read 1 Done Done Done true
16872 4910 Acked Write 1 Done Done Done true
16873 4911 Acked Read 5 Done Done Done true
16874 4912 Acked Write 4 Done Done Done true
16875 4913 Acked Write 8 Done Done Done true
16876 4914 Acked Write 2 Done Done Done true
16877 4915 Acked Read 1 Done Done Done true
16878 4916 Acked Write 8 Done Done Done true
16879 4917 Acked Write 5 Done Done Done true
16880 4918 Acked Read 7 Done Done Done true
16881 4919 Acked Read 1 Done Done Done true
16882 4920 Acked Read 3 Done Done Done true
16883 4921 Acked Write 8 Done Done Done true
16884 4922 Acked Read 9 Done Done Done true
16885 4923 Acked Write 5 Done Done Done true
16886 4924 Acked Write 3 Done Done Done true
16887 STATES DS:0 DS:1 DS:2 TOTAL
16888 Sent 0 0 0 0
16889 Done 18 18 18 54
16890 Skipped 0 0 0 0
16891 Error 0 0 0 0
16892 Last Flush: Some(JobId(4906)) Some(JobId(4906)) Some(JobId(4906))
16893 Downstairs last five completed: 4906 4905 4904 4903 4902
16894 Upstairs last five completed: 4924 4923 4922 4921 4920
16895 CLIENT: Up:0 ds:18 act:3
16896 ----------------------------------------------------------------
16897 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16898 Downstairs last five completed: 4925 4924 4923 4922 4921
16899 Upstairs last five completed: 4925 4924 4923 4922 4921
16900 CLIENT: Up:0 ds:0 act:3
16901 Replay: All jobs finished
16902 Main task has work to do, go find it
16903 got dsc Stop(0)
16904 stop 0
16905 Feb 07 20:11:08.772 INFO request completed, latency_us: 372, response_code: 204, uri: /stop/cid/0, [8810] Got stop action so:false kr:false
16906 method: POST, req_id: 9c1226f7-0a73-4459-bb1f-22eb9f9390b8, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
169072025-02-07T20:11:08.773ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
169082025-02-07T20:11:08.773ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
169092025-02-07T20:11:08.773ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
169102025-02-07T20:11:08.773ZINFOcrucible: client task is exiting = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
16911 Feb 07 20:11:08.773 INFO request completed, latency_us: 147, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5944832e-5ee3-4fa2-a468-6af51b4413f3, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
16912 [8810] Exited with: ExitStatus(unix_wait_status(9))
16913 [8810][0] reports Exit
16914 Feb 07 20:11:12.776 INFO request completed, latency_us: 313, response_code: 200, uri: /state/cid/0, method: GET, req_id: f44d032a-8855-48e4-8709-2ee28bd71f0d, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
16915 001/300 Write block 185 len 8192 data: 5 5
16916 002/300 Write block 1113 len 36864 data: 6 5 5 5 4 3 2 3 5
16917 003/300 Write block 3930 len 40960 data: 2 3 3 3 3 3 2 2 2 2
16918 004/300 Read block 2185 len 4096
16919 005/300 Flush
16920 006/300 Write block 2539 len 4096 data: 4
16921 007/300 Flush
16922 008/300 Write block 1219 len 24576 data: 4 5 4 5 7 7
16923 009/300 Read block 755 len 20480
16924 010/300 Read block 3426 len 4096
16925 011/300 Flush
16926 012/300 Write block 1301 len 36864 data: 6 6 5 4 3 2 2 2 2
16927 013/300 Write block 3883 len 32768 data: 5 3 3 3 3 4 3 3
16928 014/300 Flush
16929 015/300 Read block 1724 len 36864
16930 016/300 Write block 2682 len 32768 data: 4 4 4 4 4 4 4 5
16931 017/300 Write block 1239 len 20480 data: 3 3 3 4 2
16932 018/300 Write block 1911 len 28672 data: 4 4 5 5 5 3 4
16933 019/300 Read block 4702 len 12288
16934 020/300 Write block 3818 len 32768 data: 7 6 4 2 2 2 2 2
16935 021/300 Write block 3253 len 16384 data: 6 6 5 4
16936 022/300 Write block 2035 len 16384 data: 4 3 3 3
16937 023/300 Read block 312 len 40960
16938 024/300 Read block 1200 len 36864
16939 025/300 Read block 1437 len 32768
16940 026/300 Write block 1835 len 16384 data: 6 4 4 4
16941 027/300 Read block 3891 len 4096
16942 028/300 Read block 844 len 24576
16943 029/300 Read block 2866 len 36864
16944 030/300 Write block 3960 len 32768 data: 3 2 2 3 3 3 3 3
16945 031/300 Flush
16946 032/300 Write block 1484 len 32768 data: 2 2 3 5 5 5 5 5
16947 033/300 Read block 1210 len 4096
16948 034/300 Read block 658 len 4096
16949 035/300 Write block 3209 len 16384 data: 5 6 6 6
16950 036/300 Read block 2666 len 16384
16951 037/300 Write block 812 len 20480 data: 5 7 5 5 5
16952 038/300 Read block 2895 len 12288
16953 039/300 Read block 3151 len 24576
16954 040/300 Write block 3674 len 32768 data: 4 4 4 6 4 4 4 6
16955 041/300 Read block 910 len 16384
16956 042/300 Read block 1960 len 28672
16957 043/300 Read block 2628 len 12288
16958 044/300 Read block 3418 len 4096
16959 045/300 Read block 2553 len 4096
16960 046/300 Write block 3497 len 12288 data: 3 4 3
16961 047/300 Write block 1855 len 20480 data: 4 5 5 4 5
16962 048/300 Read block 367 len 20480
16963 049/300 Read block 1644 len 8192
16964 050/300 Read block 4027 len 16384
16965 051/300 Read block 108 len 8192
16966 052/300 Write block 3830 len 8192 data: 2 2
16967 053/300 Write block 4496 len 16384 data: 5 6 6 7
16968 054/300 Read block 806 len 4096
16969 055/300 Read block 1029 len 16384
16970 056/300 Read block 4800 len 4096
16971 057/300 Write block 1439 len 32768 data: 6 6 4 3 2 2 2 3
16972 058/300 Flush
16973 059/300 Write block 4492 len 28672 data: 3 6 6 7 6 7 7
16974 060/300 Write block 4250 len 36864 data: 4 4 5 4 3 3 3 3 4
16975 061/300 Read block 4989 len 20480
16976 062/300 Read block 4853 len 16384
16977 063/300 Read block 2157 len 12288
16978 064/300 Read block 603 len 16384
16979 065/300 Read block 2238 len 4096
16980 066/300 Write block 434 len 28672 data: 4 3 3 3 2 2 2
16981 067/300 Write block 891 len 32768 data: 3 3 2 2 2 2 3 4
16982 068/300 Flush
16983 069/300 Flush
16984 070/300 Write block 121 len 8192 data: 5 4
16985 071/300 Write block 4831 len 32768 data: 5 6 6 6 6 8 5 5
16986 072/300 Write block 3816 len 8192 data: 7 7
16987 073/300 Read block 2368 len 32768
16988 074/300 Write block 2355 len 40960 data: 3 3 3 4 4 4 4 4 6 6
16989 075/300 Read block 969 len 36864
16990 076/300 Flush
16991 077/300 Write block 1772 len 8192 data: 2 2
16992 078/300 Read block 1141 len 40960
16993 079/300 Read block 3461 len 20480
16994 080/300 Write block 2803 len 36864 data: 8 7 6 6 7 7 6 5 4
16995 081/300 Read block 1723 len 28672
16996 082/300 Flush
16997 083/300 Read block 940 len 32768
16998 084/300 Read block 670 len 12288
16999 085/300 Flush
17000 086/300 Read block 2388 len 8192
17001 087/300 Write block 1454 len 20480 data: 5 5 5 5 5
17002 088/300 Read block 1987 len 8192
17003 089/300 Read block 4737 len 36864
17004 090/300 Write block 1864 len 40960 data: 5 5 6 5 5 4 2 3 3 2
17005 091/300 Write block 3678 len 12288 data: 5 5 5
17006 092/300 Write block 4678 len 32768 data: 3 3 3 3 3 3 4 4
17007 093/300 Read block 1937 len 20480
17008 094/300 Read block 4053 len 20480
17009 095/300 Flush
17010 096/300 Write block 4829 len 24576 data: 5 5 6 7 7 7
17011 097/300 Read block 3866 len 24576
17012 098/300 Read block 2691 len 20480
17013 099/300 Read block 365 len 4096
17014 100/300 Read block 1889 len 28672
17015 101/300 Write block 2132 len 36864 data: 3 2 2 2 2 2 2 2 3
17016 102/300 Flush
17017 103/300 Write block 2863 len 4096 data: 6
17018 104/300 Read block 4403 len 28672
17019 105/300 Write block 464 len 40960 data: 4 3 5 5 5 4 4 4 5 5
17020 106/300 Write block 4467 len 28672 data: 8 6 6 6 6 7 4
17021 107/300 Write block 1314 len 20480 data: 3 2 4 4 4
17022 108/300 Read block 1306 len 16384
17023 109/300 Write block 444 len 20480 data: 2 3 3 4 5
17024 110/300 Read block 3976 len 8192
17025 111/300 Write block 1827 len 36864 data: 3 4 4 4 5 5 5 7 7
17026 112/300 Read block 1463 len 32768
17027 113/300 Read block 3008 len 20480
17028 114/300 Read block 1219 len 20480
17029 115/300 Read block 3987 len 12288
17030 116/300 Read block 2638 len 28672
17031 117/300 Write block 2800 len 8192 data: 6 8
17032 118/300 Flush
17033 119/300 Flush
17034 120/300 Write block 38 len 32768 data: 3 5 4 4 4 4 4 4
17035 121/300 Read block 2752 len 40960
17036 122/300 Read block 107 len 32768
17037 123/300 Write block 1845 len 40960 data: 4 4 4 6 6 5 5 3 4 4
17038 124/300 Read block 1651 len 8192
17039 125/300 Write block 3759 len 12288 data: 4 4 4
17040 126/300 Write block 4830 len 12288 data: 6 7 8
17041 127/300 Write block 412 len 36864 data: 4 3 2 2 3 3 4 5 5
17042 128/300 Write block 2770 len 12288 data: 3 3 3
17043 129/300 Write block 285 len 12288 data: 3 3 4
17044 130/300 Read block 2900 len 24576
17045 131/300 Write block 2102 len 20480 data: 3 3 3 2 2
17046 132/300 Write block 1791 len 16384 data: 2 3 3 3
17047 133/300 Write block 1434 len 28672 data: 5 5 5 5 6 7 7
17048 134/300 Read block 1091 len 16384
17049 135/300 Flush
17050 136/300 Read block 3004 len 16384
17051 137/300 Write block 2484 len 4096 data: 2
17052 138/300 Read block 3192 len 4096
17053 139/300 Write block 2717 len 4096 data: 3
17054 140/300 Read block 2135 len 12288
17055 141/300 Write block 973 len 28672 data: 2 3 3 4 4 4 5
17056 142/300 Read block 3038 len 24576
17057 143/300 Read block 1278 len 12288
17058 144/300 Write block 3986 len 40960 data: 3 3 3 3 2 3 3 4 4 5
17059 145/300 Read block 4608 len 24576
17060 146/300 Read block 3549 len 12288
17061 147/300 Read block 1737 len 24576
17062 148/300 Read block 1664 len 20480
17063 149/300 Write block 3111 len 24576 data: 5 4 4 4 2 2
17064 150/300 Read block 4179 len 40960
17065 151/300 Read block 2435 len 16384
17066 152/300 Flush
17067 153/300 Write block 4404 len 28672 data: 3 4 3 4 5 4 4
17068 154/300 Read block 167 len 8192
17069 155/300 Write block 3943 len 24576 data: 2 3 3 4 6 6
17070 156/300 Write block 1359 len 4096 data: 3
17071 157/300 Read block 2645 len 4096
17072 158/300 Flush
17073 159/300 Read block 2091 len 32768
17074 160/300 Write block 2896 len 28672 data: 2 2 2 2 3 3 3
17075 161/300 Write block 319 len 40960 data: 5 4 3 3 3 2 2 2 2 2
17076 162/300 Read block 4106 len 28672
17077 163/300 Write block 1019 len 16384 data: 6 6 7 7
17078 164/300 Read block 3987 len 28672
17079 165/300 Write block 1948 len 36864 data: 3 3 3 3 3 3 3 3 3
17080 166/300 Write block 3127 len 32768 data: 6 5 6 5 5 5 6 4
17081 167/300 Write block 20 len 24576 data: 3 3 3 3 3 3
17082 168/300 Read block 176 len 24576
17083 169/300 Flush
17084 170/300 Read block 1025 len 12288
17085 171/300 Read block 1131 len 12288
17086 172/300 Write block 3910 len 20480 data: 3 4 5 4 4
17087 173/300 Write block 2682 len 40960 data: 5 5 5 5 5 5 5 6 5 4
17088 174/300 Read block 1826 len 32768
17089 175/300 Flush
17090 176/300 Write block 4668 len 32768 data: 5 5 5 4 4 5 3 3
17091 177/300 Write block 3264 len 16384 data: 4 4 4 4
17092 178/300 Write block 2401 len 4096 data: 5
17093 179/300 Write block 2201 len 8192 data: 4 4
17094 180/300 Read block 1360 len 32768
17095 181/300 Write block 997 len 20480 data: 3 4 5 6 8
17096 182/300 Write block 1027 len 16384 data: 5 5 5 3
17097 183/300 Write block 1504 len 40960 data: 4 4 5 5 6 5 5 5 3 3
17098 184/300 Write block 2176 len 24576 data: 6 5 5 3 3 3
17099 185/300 Read block 2111 len 8192
17100 186/300 Read block 1788 len 4096
17101 187/300 Read block 265 len 12288
17102 188/300 Flush
17103 189/300 Write block 734 len 24576 data: 3 3 3 3 3 3
17104 190/300 Write block 4054 len 4096 data: 4
17105 191/300 Read block 2343 len 8192
17106 192/300 Write block 2257 len 32768 data: 2 2 3 3 5 5 5 4
17107 193/300 Read block 3854 len 4096
17108 194/300 Read block 3128 len 16384
17109 195/300 Read block 1790 len 8192
17110 196/300 Write block 3950 len 4096 data: 4
17111 197/300 Write block 4812 len 36864 data: 3 3 3 4 4 6 6 4 3
17112 198/300 Flush
17113 199/300 Read block 3616 len 4096
17114 200/300 Write block 4589 len 28672 data: 4 3 3 3 4 5 3
17115 201/300 Write block 2281 len 40960 data: 2 2 2 2 2 2 3 5 5 5
17116 202/300 Write block 4063 len 32768 data: 4 4 4 4 4 4 3 3
17117 203/300 Read block 2341 len 8192
17118 204/300 Read block 4055 len 12288
17119 205/300 Write block 1694 len 16384 data: 5 6 6 5
17120 206/300 Read block 1058 len 8192
17121 207/300 Write block 2463 len 12288 data: 3 4 4
17122 208/300 Flush
17123 209/300 Read block 1355 len 24576
17124 210/300 Read block 4121 len 8192
17125 211/300 Read block 2128 len 28672
17126 212/300 Read block 3937 len 40960
17127 213/300 Write block 3615 len 20480 data: 4 4 4 4 4
17128 214/300 Read block 345 len 20480
17129 215/300 Write block 4797 len 8192 data: 3 3
17130 216/300 Write block 1693 len 12288 data: 5 6 7
17131 217/300 Read block 1818 len 16384
17132 218/300 Read block 4454 len 8192
17133 219/300 Write block 457 len 32768 data: 5 5 4 5 4 5 5 5
17134 220/300 Write block 4372 len 28672 data: 3 3 4 5 4 4 4
17135 221/300 Read block 3256 len 32768
17136 222/300 Write block 770 len 4096 data: 5
17137 223/300 Read block 4485 len 12288
17138 224/300 Write block 2501 len 32768 data: 2 2 2 2 3 3 4 4
17139 225/300 Read block 2010 len 20480
17140 226/300 Write block 3716 len 36864 data: 3 3 3 4 6 5 5 3 3
17141 227/300 Flush
17142 228/300 Write block 131 len 16384 data: 3 3 3 2
17143 229/300 Write block 4490 len 12288 data: 4 4 4
17144 230/300 Read block 1432 len 36864
17145 231/300 Write block 65 len 12288 data: 4 4 4
17146 232/300 Write block 24 len 4096 data: 4
17147 233/300 Write block 3136 len 32768 data: 4 4 4 5 6 6 5 4
17148 234/300 Read block 90 len 16384
17149 235/300 Read block 3510 len 8192
17150 236/300 Read block 4951 len 4096
17151 237/300 Read block 2942 len 32768
17152 238/300 Read block 4329 len 24576
17153 239/300 Read block 2736 len 20480
17154 240/300 Read block 171 len 36864
17155 241/300 Read block 2990 len 32768
17156 242/300 Read block 3312 len 24576
17157 243/300 Read block 2729 len 24576
17158 244/300 Read block 2621 len 24576
17159 245/300 Write block 4913 len 8192 data: 5 5
17160 246/300 Read block 3539 len 36864
17161 247/300 Read block 2189 len 12288
17162 248/300 Flush
17163 249/300 Flush
17164 250/300 Read block 2585 len 12288
17165 251/300 Write block 1967 len 40960 data: 2 2 2 2 3 3 3 3 3 3
17166 252/300 Read block 2933 len 40960
17167 253/300 Read block 1743 len 24576
17168 254/300 Write block 3163 len 32768 data: 4 5 5 4 5 5 5 4
17169 255/300 Flush
17170 256/300 Write block 1058 len 24576 data: 5 4 4 4 3 3
17171 257/300 Read block 3254 len 40960
17172 258/300 Write block 839 len 28672 data: 3 3 3 4 3 3 3
17173 259/300 Read block 1266 len 20480
17174 260/300 Read block 2564 len 28672
17175 261/300 Read block 3491 len 20480
17176 262/300 Write block 4200 len 16384 data: 4 3 2 2
17177 263/300 Write block 4841 len 20480 data: 5 4 4 4 3
17178 264/300 Read block 4710 len 24576
17179 265/300 Read block 482 len 24576
17180 266/300 Flush
17181 267/300 Write block 3903 len 20480 data: 4 3 4 4 3
17182 268/300 Write block 3116 len 16384 data: 3 2 3 3
17183 269/300 Write block 4215 len 36864 data: 3 3 5 5 5 5 4 5 5
17184 270/300 Read block 82 len 36864
17185 271/300 Write block 4817 len 24576 data: 7 7 5 4 3 4
17186 272/300 Read block 4042 len 36864
17187 273/300 Read block 1680 len 28672
17188 274/300 Write block 1000 len 12288 data: 7 9 8
17189 275/300 Read block 721 len 8192
17190 276/300 Read block 4601 len 24576
17191 277/300 Read block 498 len 4096
17192 278/300 Read block 2816 len 12288
17193 279/300 Read block 2207 len 12288
17194 280/300 Read block 1580 len 36864
17195 281/300 Read block 944 len 24576
17196 282/300 Write block 1999 len 12288 data: 8 5 4
17197 283/300 Read block 4054 len 32768
17198 284/300 Write block 1844 len 8192 data: 3 5
17199 285/300 Read block 2630 len 28672
17200 286/300 Write block 4806 len 4096 data: 5
17201 287/300 Read block 4746 len 40960
17202 288/300 Read block 3983 len 28672
17203 289/300 Read block 4507 len 16384
17204 290/300 Write block 2453 len 24576 data: 5 4 4 3 5 5
17205 291/300 Read block 1119 len 8192
17206 292/300 Write block 644 len 40960 data: 5 6 5 4 4 4 3 3 3 3
17207 293/300 Read block 1752 len 28672
17208 294/300 Read block 4863 len 8192
17209 295/300 Read block 174 len 40960
17210 296/300 Write block 484 len 32768 data: 2 2 2 2 2 2 2 2
17211 297/300 Write block 2805 len 8192 data: 7 7
17212 298/300 Read block 839 len 16384
17213 299/300 Read block 2179 len 32768
17214 300/300 Write block 2169 len 40960 data: 5 6 6 5 5 5 6 7 6 6
17215 301/300 Write block 4060 len 8192 data: 3 4
17216 Main task has work to do, go find it
17217 got dsc Start(0)
17218 start 0
17219 Feb 07 20:11:13.010 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/0, method: POST, req_id: 8220e791-c5c3-4951-ac1f-86dc29f402bc, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
17220 Replay: started 0, returned:Ok(())
17221 ----------------------------------------------------------------
17222 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17223 States: Offline Active Active
17224 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17225 4926 Acked Write 2 Sent Done Done false
17226 4927 Acked Write 9 Sent Done Done false
17227 4928 Acked Write 10 Sent Done Done false
17228 4929 Acked Read 1 Sent Done Done false
17229 4930 Acked Flush 0 Sent Done Done false
17230 4931 Acked Write 1 Sent Done Done false
17231 4932 Acked Flush 0 Sent Done Done false
17232 4933 Acked Write 6 Sent Done Done false
17233 4934 Acked Read 5 Sent Done Done false
17234 4935 Acked Read 1 Sent Done Done false
17235 4936 Acked Flush 0 Sent Done Done false
17236 4937 Acked Write 9 Sent Done Done false
17237 4938 Acked Write 8 Sent Done Done false
17238 4939 Acked Flush 0 Sent Done Done false
17239 4940 Acked Read 9 Sent Done Done false
17240 4941 Acked Write 8 Sent Done Done false
17241 4942 Acked Write 5 Sent Done Done false
17242 4943 Acked Write 7 Sent Done Done false
17243 4944 Acked Read 3 Sent Done Done false
17244 4945 Acked Write 8 Sent Done Done false
17245 4946 Acked Write 4 Sent Done Done false
17246 4947 Acked Write 4 Sent Done Done false
17247 4948 Acked Read 10 Sent Done Done false
17248 4949 Acked Read 9 Sent Done Done false
17249 4950 Acked Read 8 Sent Done Done false
17250 4951 Acked Write 4 Sent Done Done false
17251 4952 Acked Read 1 Sent Done Done false
17252 4953 Acked Read 6 Sent Done Done false
17253 4954 Acked Read 9 Sent Done Done false
17254 4955 Acked Write 8 Sent Done Done false
17255 4956 Acked Flush 0 Sent Done Done false
17256 4957 Acked Write 8 Sent Done Done false
17257 4958 Acked Read 1 Sent Done Done false
17258 4959 Acked Read 1 Sent Done Done false
17259 4960 Acked Write 4 Sent Done Done false
17260 4961 Acked Read 4 Sent Done Done false
17261 4962 Acked Write 5 Sent Done Done false
17262 4963 Acked Read 3 Sent Done Done false
17263 4964 Acked Read 6 Sent Done Done false
17264 4965 Acked Write 8 Sent Done Done false
17265 4966 Acked Read 4 Sent Done Done false
17266 4967 Acked Read 7 Sent Done Done false
17267 4968 Acked Read 3 Sent Done Done false
17268 4969 Acked Read 1 Sent Done Done false
17269 4970 Acked Read 1 Sent Done Done false
17270 4971 Acked Write 3 Sent Done Done false
17271 4972 Acked Write 5 Sent Done Done false
17272 4973 Acked Read 5 Sent Done Done false
17273 4974 Acked Read 2 Sent Done Done false
17274 4975 Acked Read 4 Sent Done Done false
17275 4976 Acked Read 2 Sent Done Done false
17276 4977 Acked Write 2 Sent Done Done false
17277 4978 Acked Write 4 Sent Done Done false
17278 4979 Acked Read 1 Sent Done Done false
17279 4980 Acked Read 4 Sent Done Done false
17280 4981 Acked Read 1 Sent Done Done false
17281 4982 Acked Write 8 Sent Done Done false
17282 4983 Acked Flush 0 Sent Done Done false
17283 4984 Acked Write 7 Sent Done Done false
17284 4985 Acked Write 9 Sent Done Done false
17285 4986 Acked Read 5 Sent Done Done false
17286 4987 Acked Read 4 Sent Done Done false
17287 4988 Acked Read 3 Sent Done Done false
17288 4989 Acked Read 4 Sent Done Done false
17289 4990 Acked Read 1 Sent Done Done false
17290 4991 Acked Write 7 Sent Done Done false
17291 4992 Acked Write 8 Sent Done Done false
17292 4993 Acked Flush 0 Sent Done Done false
17293 4994 Acked Flush 0 Sent Done Done false
17294 4995 Acked Write 2 Sent Done Done false
17295 4996 Acked Write 8 Sent Done Done false
17296 4997 Acked Write 2 Sent Done Done false
17297 4998 Acked Read 8 Sent Done Done false
17298 4999 Acked Write 10 Sent Done Done false
17299 5000 Acked Read 9 Sent Done Done false
17300 5001 Acked Flush 0 Sent Done Done false
17301 5002 Acked Write 2 Sent Done Done false
17302 5003 Acked Read 10 Sent Done Done false
17303 5004 Acked Read 5 Sent Done Done false
17304 5005 Acked Write 9 Sent Done Done false
17305 5006 Acked Read 7 Sent Done Done false
17306 5007 Acked Flush 0 Sent Done Done false
17307 5008 Acked Read 8 Sent Done Done false
17308 5009 Acked Read 3 Sent Done Done false
17309 5010 Acked Flush 0 Sent Done Done false
17310 5011 Acked Read 2 Sent Done Done false
17311 5012 Acked Write 5 Sent Done Done false
17312 5013 Acked Read 2 Sent Done Done false
17313 5014 Acked Read 9 Sent Done Done false
17314 5015 Acked Write 10 Sent Done Done false
17315 5016 Acked Write 3 Sent Done Done false
17316 5017 Acked Write 8 Sent Done Done false
17317 5018 Acked Read 5 Sent Done Done false
17318 5019 Acked Read 5 Sent Done Done false
17319 5020 Acked Flush 0 Sent Done Done false
17320 5021 Acked Write 6 Sent Done Done false
17321 5022 Acked Read 6 Sent Done Done false
17322 5023 Acked Read 5 Sent Done Done false
17323 5024 Acked Read 1 Sent Done Done false
17324 5025 Acked Read 7 Sent Done Done false
17325 5026 Acked Write 9 Sent Done Done false
17326 5027 Acked Flush 0 Sent Done Done false
17327 5028 Acked Write 1 Sent Done Done false
17328 5029 Acked Read 7 Sent Done Done false
17329 5030 Acked Write 10 Sent Done Done false
17330 5031 Acked Write 7 Sent Done Done false
17331 5032 Acked Write 5 Sent Done Done false
17332 5033 Acked Read 4 Sent Done Done false
17333 5034 Acked Write 5 Sent Done Done false
17334 5035 Acked Read 2 Sent Done Done false
17335 5036 Acked Write 9 Sent Done Done false
17336 5037 Acked Read 8 Sent Done Done false
17337 5038 Acked Read 5 Sent Done Done false
17338 5039 Acked Read 5 Sent Done Done false
17339 5040 Acked Read 3 Sent Done Done false
17340 5041 Acked Read 7 Sent Done Done false
17341 5042 Acked Write 2 Sent Done Done false
17342 5043 Acked Flush 0 Sent Done Done false
17343 5044 Acked Flush 0 Sent Done Done false
17344 5045 Acked Write 8 Sent Done Done false
17345 5046 Acked Read 10 Sent Done Done false
17346 5047 Acked Read 8 Sent Done Done false
17347 5048 Acked Write 10 Sent Done Done false
17348 5049 Acked Read 2 Sent Done Done false
17349 5050 Acked Write 3 Sent Done Done false
17350 5051 Acked Write 3 Sent Done Done false
17351 5052 Acked Write 9 Sent Done Done false
17352 5053 Acked Write 3 Sent Done Done false
17353 5054 Acked Write 3 Sent Done Done false
17354 5055 Acked Read 6 Sent Done Done false
17355 5056 Acked Write 5 Sent Done Done false
17356 5057 Acked Write 4 Sent Done Done false
17357 5058 Acked Write 7 Sent Done Done false
17358 5059 Acked Read 4 Sent Done Done false
17359 5060 Acked Flush 0 Sent Done Done false
17360 5061 Acked Read 4 Sent Done Done false
17361 5062 Acked Write 1 Sent Done Done false
17362 5063 Acked Read 1 Sent Done Done false
17363 5064 Acked Write 1 Sent Done Done false
17364 5065 Acked Read 3 Sent Done Done false
17365 5066 Acked Write 7 Sent Done Done false
17366 5067 Acked Read 6 Sent Done Done false
17367 5068 Acked Read 3 Sent Done Done false
17368 5069 Acked Write 10 Sent Done Done false
17369 5070 Acked Read 6 Sent Done Done false
17370 5071 Acked Read 3 Sent Done Done false
17371 5072 Acked Read 6 Sent Done Done false
17372 5073 Acked Read 5 Sent Done Done false
17373 5074 Acked Write 6 Sent Done Done false
17374 5075 Acked Read 10 Sent Done Done false
17375 5076 Acked Read 4 Sent Done Done false
17376 5077 Acked Flush 0 Sent Done Done false
17377 5078 Acked Write 7 Sent Done Done false
17378 5079 Acked Read 2 Sent Done Done false
17379 5080 Acked Write 6 Sent Done Done false
17380 5081 Acked Write 1 Sent Done Done false
17381 5082 Acked Read 1 Sent Done Done false
17382 5083 Acked Flush 0 Sent Done Done false
17383 5084 Acked Read 8 Sent Done Done false
17384 5085 Acked Write 7 Sent Done Done false
17385 5086 Acked Write 10 Sent Done Done false
17386 5087 Acked Read 7 Sent Done Done false
17387 5088 Acked Write 4 Sent Done Done false
17388 5089 Acked Read 7 Sent Done Done false
17389 5090 Acked Write 9 Sent Done Done false
17390 5091 Acked Write 8 Sent Done Done false
17391 5092 Acked Write 6 Sent Done Done false
17392 5093 Acked Read 6 Sent Done Done false
17393 5094 Acked Flush 0 Sent Done Done false
17394 5095 Acked Read 3 Sent Done Done false
17395 5096 Acked Read 3 Sent Done Done false
17396 5097 Acked Write 5 Sent Done Done false
17397 5098 Acked Write 10 Sent Done Done false
17398 5099 Acked Read 8 Sent Done Done false
17399 5100 Acked Flush 0 Sent Done Done false
17400 5101 Acked Write 8 Sent Done Done false
17401 5102 Acked Write 4 Sent Done Done false
17402 5103 Acked Write 1 Sent Done Done false
17403 5104 Acked Write 2 Sent Done Done false
17404 5105 Acked Read 8 Sent Done Done false
17405 5106 Acked Write 5 Sent Done Done false
17406 5107 Acked Write 4 Sent Done Done false
17407 5108 Acked Write 10 Sent Done Done false
17408 5109 Acked Write 6 Sent Done Done false
17409 5110 Acked Read 2 Sent Done Done false
17410 5111 Acked Read 1 Sent Done Done false
17411 5112 Acked Read 3 Sent Done Done false
17412 5113 Acked Flush 0 Sent Done Done false
17413 5114 Acked Write 6 Sent Done Done false
17414 5115 Acked Write 1 Sent Done Done false
17415 5116 Acked Read 2 Sent Done Done false
17416 5117 Acked Write 8 Sent Done Done false
17417 5118 Acked Read 1 Sent Done Done false
17418 5119 Acked Read 4 Sent Done Done false
17419 5120 Acked Read 2 Sent Done Done false
17420 5121 Acked Write 1 Sent Done Done false
17421 5122 Acked Write 9 Sent Done Done false
17422 5123 Acked Flush 0 Sent Done Done false
17423 5124 Acked Read 1 Sent Done Done false
17424 5125 Acked Write 7 Sent Done Done false
17425 5126 Acked Write 10 Sent Done Done false
17426 5127 Acked Write 8 Sent Done Done false
17427 5128 Acked Read 2 Sent Done Done false
17428 5129 Acked Read 3 Sent Done Done false
17429 5130 Acked Write 4 Sent Done Done false
17430 5131 Acked Read 2 Sent Done Done false
17431 5132 Acked Write 3 Sent Done Done false
17432 5133 Acked Flush 0 Sent Done Done false
17433 5134 Acked Read 6 Sent Done Done false
17434 5135 Acked Read 2 Sent Done Done false
17435 5136 Acked Read 7 Sent Done Done false
17436 5137 Acked Read 10 Sent Done Done false
17437 5138 Acked Write 5 Sent Done Done false
17438 5139 Acked Read 5 Sent Done Done false
17439 5140 Acked Write 2 Sent Done Done false
17440 5141 Acked Write 3 Sent Done Done false
17441 5142 Acked Read 4 Sent Done Done false
17442 5143 Acked Read 2 Sent Done Done false
17443 5144 Acked Write 8 Sent Done Done false
17444 5145 Acked Write 7 Sent Done Done false
17445 5146 Acked Read 8 Sent Done Done false
17446 5147 Acked Write 1 Sent Done Done false
17447 5148 Acked Read 3 Sent Done Done false
17448 5149 Acked Write 8 Sent Done Done false
17449 5150 Acked Read 5 Sent Done Done false
17450 5151 Acked Write 9 Sent Done Done false
17451 5152 Acked Flush 0 Sent Done Done false
17452 5153 Acked Write 4 Sent Done Done false
17453 5154 Acked Write 3 Sent Done Done false
17454 5155 Acked Read 9 Sent Done Done false
17455 5156 Acked Write 3 Sent Done Done false
17456 5157 Acked Write 1 Sent Done Done false
17457 5158 Acked Write 8 Sent Done Done false
17458 5159 Acked Read 4 Sent Done Done false
17459 5160 Acked Read 2 Sent Done Done false
17460 5161 Acked Read 1 Sent Done Done false
17461 5162 Acked Read 8 Sent Done Done false
17462 5163 Acked Read 6 Sent Done Done false
17463 5164 Acked Read 5 Sent Done Done false
17464 5165 Acked Read 9 Sent Done Done false
17465 5166 Acked Read 8 Sent Done Done false
17466 5167 Acked Read 6 Sent Done Done false
17467 5168 Acked Read 6 Sent Done Done false
17468 5169 Acked Read 6 Sent Done Done false
17469 5170 Acked Write 2 Sent Done Done false
17470 5171 Acked Read 9 Sent Done Done false
17471 5172 Acked Read 3 Sent Done Done false
17472 5173 Acked Flush 0 Sent Done Done false
17473 5174 Acked Flush 0 Sent Done Done false
17474 5175 Acked Read 3 Sent Done Done false
17475 5176 Acked Write 10 Sent Done Done false
17476 5177 Acked Read 10 Sent Done Done false
17477 5178 Acked Read 6 Sent Done Done false
17478 5179 Acked Write 8 Sent Done Done false
17479 5180 Acked Flush 0 Sent Done Done false
17480 5181 Acked Write 6 Sent Done Done false
17481 5182 Acked Read 10 Sent Done Done false
17482 5183 Acked Write 7 Sent Done Done false
17483 5184 Acked Read 5 Sent Done Done false
17484 5185 Acked Read 7 Sent Done Done false
17485 5186 Acked Read 5 Sent Done Done false
17486 5187 Acked Write 4 Sent Done Done false
17487 5188 Acked Write 5 Sent Done Done false
17488 5189 Acked Read 6 Sent Done Done false
17489 5190 Acked Read 6 Sent Done Done false
17490 5191 Acked Flush 0 Sent Done Done false
17491 5192 Acked Write 5 Sent Done Done false
17492 5193 Acked Write 4 Sent Done Done false
17493 5194 Acked Write 9 Sent Done Done false
17494 5195 Acked Read 9 Sent Done Done false
17495 5196 Acked Write 6 Sent Done Done false
17496 5197 Acked Read 9 Sent Done Done false
17497 5198 Acked Read 7 Sent Done Done false
17498 5199 Acked Write 3 Sent Done Done false
17499 5200 Acked Read 2 Sent Done Done false
17500 5201 Acked Read 6 Sent Done Done false
17501 5202 Acked Read 1 Sent Done Done false
17502 5203 Acked Read 3 Sent Done Done false
17503 5204 Acked Read 3 Sent Done Done false
17504 5205 Acked Read 9 Sent Done Done false
17505 5206 Acked Read 6 Sent Done Done false
17506 5207 Acked Write 3 Sent Done Done false
17507 5208 Acked Read 8 Sent Done Done false
17508 5209 Acked Write 2 Sent Done Done false
17509 5210 Acked Read 7 Sent Done Done false
17510 5211 Acked Write 1 Sent Done Done false
17511 5212 Acked Read 10 Sent Done Done false
17512 5213 Acked Read 7 Sent Done Done false
17513 5214 Acked Read 4 Sent Done Done false
17514 5215 Acked Write 6 Sent Done Done false
17515 5216 Acked Read 2 Sent Done Done false
17516 5217 Acked Write 10 Sent Done Done false
17517 5218 Acked Read 7 Sent Done Done false
17518 5219 Acked Read 2 Sent Done Done false
17519 5220 Acked Read 10 Sent Done Done false
17520 5221 Acked Write 8 Sent Done Done false
17521 5222 Acked Write 2 Sent Done Done false
17522 5223 Acked Read 4 Sent Done Done false
17523 5224 Acked Read 8 Sent Done Done false
17524 5225 Acked Write 10 Sent Done Done false
17525 5226 Acked Write 2 Sent Done Done false
17526 STATES DS:0 DS:1 DS:2 TOTAL
17527 Sent 301 0 0 301
17528 Done 0 301 301 602
17529 Skipped 0 0 0 0
17530 Error 0 0 0 0
17531 Last Flush: Some(JobId(4925)) Some(JobId(5191)) Some(JobId(5191))
17532 Downstairs last five completed: 4925 4924 4923 4922 4921
17533 Upstairs last five completed: 5226 5225 5224 5223 5222
17534 CLIENT: Up:0 ds:301 act:2
175352025-02-07T20:11:13.512ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
175362025-02-07T20:11:14.323ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
17537 [8810] got start action
17538 [8810] I am going to restart
17539 Starting downstairs at port 8810
17540 Make output file at "/tmp/dsc/downstairs-8810.txt"
17541 [8810][0] reports Starting
17542 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1223)
17543 [8810][0] reports Running
175442025-02-07T20:11:15.325ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
175452025-02-07T20:11:16.327ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
175462025-02-07T20:11:16.828ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
17547 ----------------------------------------------------------------
17548 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17549 States: Offline Active Active
17550 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17551 4926 Acked Write 2 Sent Done Done false
17552 4927 Acked Write 9 Sent Done Done false
17553 4928 Acked Write 10 Sent Done Done false
17554 4929 Acked Read 1 Sent Done Done false
17555 4930 Acked Flush 0 Sent Done Done false
17556 4931 Acked Write 1 Sent Done Done false
17557 4932 Acked Flush 0 Sent Done Done false
17558 4933 Acked Write 6 Sent Done Done false
17559 4934 Acked Read 5 Sent Done Done false
17560 4935 Acked Read 1 Sent Done Done false
17561 4936 Acked Flush 0 Sent Done Done false
17562 4937 Acked Write 9 Sent Done Done false
17563 4938 Acked Write 8 Sent Done Done false
17564 4939 Acked Flush 0 Sent Done Done false
17565 4940 Acked Read 9 Sent Done Done false
17566 4941 Acked Write 8 Sent Done Done false
17567 4942 Acked Write 5 Sent Done Done false
17568 4943 Acked Write 7 Sent Done Done false
17569 4944 Acked Read 3 Sent Done Done false
17570 4945 Acked Write 8 Sent Done Done false
17571 4946 Acked Write 4 Sent Done Done false
17572 4947 Acked Write 4 Sent Done Done false
17573 4948 Acked Read 10 Sent Done Done false
17574 4949 Acked Read 9 Sent Done Done false
17575 4950 Acked Read 8 Sent Done Done false
17576 4951 Acked Write 4 Sent Done Done false
17577 4952 Acked Read 1 Sent Done Done false
17578 4953 Acked Read 6 Sent Done Done false
17579 4954 Acked Read 9 Sent Done Done false
17580 4955 Acked Write 8 Sent Done Done false
17581 4956 Acked Flush 0 Sent Done Done false
17582 4957 Acked Write 8 Sent Done Done false
17583 4958 Acked Read 1 Sent Done Done false
17584 4959 Acked Read 1 Sent Done Done false
17585 4960 Acked Write 4 Sent Done Done false
17586 4961 Acked Read 4 Sent Done Done false
17587 4962 Acked Write 5 Sent Done Done false
17588 4963 Acked Read 3 Sent Done Done false
17589 4964 Acked Read 6 Sent Done Done false
17590 4965 Acked Write 8 Sent Done Done false
17591 4966 Acked Read 4 Sent Done Done false
17592 4967 Acked Read 7 Sent Done Done false
17593 4968 Acked Read 3 Sent Done Done false
17594 4969 Acked Read 1 Sent Done Done false
17595 4970 Acked Read 1 Sent Done Done false
17596 4971 Acked Write 3 Sent Done Done false
17597 4972 Acked Write 5 Sent Done Done false
17598 4973 Acked Read 5 Sent Done Done false
17599 4974 Acked Read 2 Sent Done Done false
17600 4975 Acked Read 4 Sent Done Done false
17601 4976 Acked Read 2 Sent Done Done false
17602 4977 Acked Write 2 Sent Done Done false
17603 4978 Acked Write 4 Sent Done Done false
17604 4979 Acked Read 1 Sent Done Done false
17605 4980 Acked Read 4 Sent Done Done false
17606 4981 Acked Read 1 Sent Done Done false
17607 4982 Acked Write 8 Sent Done Done false
17608 4983 Acked Flush 0 Sent Done Done false
17609 4984 Acked Write 7 Sent Done Done false
17610 4985 Acked Write 9 Sent Done Done false
17611 4986 Acked Read 5 Sent Done Done false
17612 4987 Acked Read 4 Sent Done Done false
17613 4988 Acked Read 3 Sent Done Done false
17614 4989 Acked Read 4 Sent Done Done false
17615 4990 Acked Read 1 Sent Done Done false
17616 4991 Acked Write 7 Sent Done Done false
17617 4992 Acked Write 8 Sent Done Done false
17618 4993 Acked Flush 0 Sent Done Done false
17619 4994 Acked Flush 0 Sent Done Done false
17620 4995 Acked Write 2 Sent Done Done false
17621 4996 Acked Write 8 Sent Done Done false
17622 4997 Acked Write 2 Sent Done Done false
17623 4998 Acked Read 8 Sent Done Done false
17624 4999 Acked Write 10 Sent Done Done false
17625 5000 Acked Read 9 Sent Done Done false
17626 5001 Acked Flush 0 Sent Done Done false
17627 5002 Acked Write 2 Sent Done Done false
17628 5003 Acked Read 10 Sent Done Done false
17629 5004 Acked Read 5 Sent Done Done false
17630 5005 Acked Write 9 Sent Done Done false
17631 5006 Acked Read 7 Sent Done Done false
17632 5007 Acked Flush 0 Sent Done Done false
17633 5008 Acked Read 8 Sent Done Done false
17634 5009 Acked Read 3 Sent Done Done false
17635 5010 Acked Flush 0 Sent Done Done false
17636 5011 Acked Read 2 Sent Done Done false
17637 5012 Acked Write 5 Sent Done Done false
17638 5013 Acked Read 2 Sent Done Done false
17639 5014 Acked Read 9 Sent Done Done false
17640 5015 Acked Write 10 Sent Done Done false
17641 5016 Acked Write 3 Sent Done Done false
17642 5017 Acked Write 8 Sent Done Done false
17643 5018 Acked Read 5 Sent Done Done false
17644 5019 Acked Read 5 Sent Done Done false
17645 5020 Acked Flush 0 Sent Done Done false
17646 5021 Acked Write 6 Sent Done Done false
17647 5022 Acked Read 6 Sent Done Done false
17648 5023 Acked Read 5 Sent Done Done false
17649 5024 Acked Read 1 Sent Done Done false
17650 5025 Acked Read 7 Sent Done Done false
17651 5026 Acked Write 9 Sent Done Done false
17652 5027 Acked Flush 0 Sent Done Done false
17653 5028 Acked Write 1 Sent Done Done false
17654 5029 Acked Read 7 Sent Done Done false
17655 5030 Acked Write 10 Sent Done Done false
17656 5031 Acked Write 7 Sent Done Done false
17657 5032 Acked Write 5 Sent Done Done false
17658 5033 Acked Read 4 Sent Done Done false
17659 5034 Acked Write 5 Sent Done Done false
17660 5035 Acked Read 2 Sent Done Done false
17661 5036 Acked Write 9 Sent Done Done false
17662 5037 Acked Read 8 Sent Done Done false
17663 5038 Acked Read 5 Sent Done Done false
17664 5039 Acked Read 5 Sent Done Done false
17665 5040 Acked Read 3 Sent Done Done false
17666 5041 Acked Read 7 Sent Done Done false
17667 5042 Acked Write 2 Sent Done Done false
17668 5043 Acked Flush 0 Sent Done Done false
17669 5044 Acked Flush 0 Sent Done Done false
17670 5045 Acked Write 8 Sent Done Done false
17671 5046 Acked Read 10 Sent Done Done false
17672 5047 Acked Read 8 Sent Done Done false
17673 5048 Acked Write 10 Sent Done Done false
17674 5049 Acked Read 2 Sent Done Done false
17675 5050 Acked Write 3 Sent Done Done false
17676 5051 Acked Write 3 Sent Done Done false
17677 5052 Acked Write 9 Sent Done Done false
17678 5053 Acked Write 3 Sent Done Done false
17679 5054 Acked Write 3 Sent Done Done false
17680 5055 Acked Read 6 Sent Done Done false
17681 5056 Acked Write 5 Sent Done Done false
17682 5057 Acked Write 4 Sent Done Done false
17683 5058 Acked Write 7 Sent Done Done false
17684 5059 Acked Read 4 Sent Done Done false
17685 5060 Acked Flush 0 Sent Done Done false
17686 5061 Acked Read 4 Sent Done Done false
17687 5062 Acked Write 1 Sent Done Done false
17688 5063 Acked Read 1 Sent Done Done false
17689 5064 Acked Write 1 Sent Done Done false
17690 5065 Acked Read 3 Sent Done Done false
17691 5066 Acked Write 7 Sent Done Done false
17692 5067 Acked Read 6 Sent Done Done false
17693 5068 Acked Read 3 Sent Done Done false
17694 5069 Acked Write 10 Sent Done Done false
17695 5070 Acked Read 6 Sent Done Done false
17696 5071 Acked Read 3 Sent Done Done false
17697 5072 Acked Read 6 Sent Done Done false
17698 5073 Acked Read 5 Sent Done Done false
17699 5074 Acked Write 6 Sent Done Done false
17700 5075 Acked Read 10 Sent Done Done false
17701 5076 Acked Read 4 Sent Done Done false
17702 5077 Acked Flush 0 Sent Done Done false
17703 5078 Acked Write 7 Sent Done Done false
17704 5079 Acked Read 2 Sent Done Done false
17705 5080 Acked Write 6 Sent Done Done false
17706 5081 Acked Write 1 Sent Done Done false
17707 5082 Acked Read 1 Sent Done Done false
17708 5083 Acked Flush 0 Sent Done Done false
17709 5084 Acked Read 8 Sent Done Done false
17710 5085 Acked Write 7 Sent Done Done false
17711 5086 Acked Write 10 Sent Done Done false
17712 5087 Acked Read 7 Sent Done Done false
17713 5088 Acked Write 4 Sent Done Done false
17714 5089 Acked Read 7 Sent Done Done false
17715 5090 Acked Write 9 Sent Done Done false
17716 5091 Acked Write 8 Sent Done Done false
17717 5092 Acked Write 6 Sent Done Done false
17718 5093 Acked Read 6 Sent Done Done false
17719 5094 Acked Flush 0 Sent Done Done false
17720 5095 Acked Read 3 Sent Done Done false
17721 5096 Acked Read 3 Sent Done Done false
17722 5097 Acked Write 5 Sent Done Done false
17723 5098 Acked Write 10 Sent Done Done false
17724 5099 Acked Read 8 Sent Done Done false
17725 5100 Acked Flush 0 Sent Done Done false
17726 5101 Acked Write 8 Sent Done Done false
17727 5102 Acked Write 4 Sent Done Done false
17728 5103 Acked Write 1 Sent Done Done false
17729 5104 Acked Write 2 Sent Done Done false
17730 5105 Acked Read 8 Sent Done Done false
17731 5106 Acked Write 5 Sent Done Done false
17732 5107 Acked Write 4 Sent Done Done false
17733 5108 Acked Write 10 Sent Done Done false
17734 5109 Acked Write 6 Sent Done Done false
17735 5110 Acked Read 2 Sent Done Done false
17736 5111 Acked Read 1 Sent Done Done false
17737 5112 Acked Read 3 Sent Done Done false
17738 5113 Acked Flush 0 Sent Done Done false
17739 5114 Acked Write 6 Sent Done Done false
17740 5115 Acked Write 1 Sent Done Done false
17741 5116 Acked Read 2 Sent Done Done false
17742 5117 Acked Write 8 Sent Done Done false
17743 5118 Acked Read 1 Sent Done Done false
17744 5119 Acked Read 4 Sent Done Done false
17745 5120 Acked Read 2 Sent Done Done false
17746 5121 Acked Write 1 Sent Done Done false
17747 5122 Acked Write 9 Sent Done Done false
17748 5123 Acked Flush 0 Sent Done Done false
17749 5124 Acked Read 1 Sent Done Done false
17750 5125 Acked Write 7 Sent Done Done false
17751 5126 Acked Write 10 Sent Done Done false
17752 5127 Acked Write 8 Sent Done Done false
17753 5128 Acked Read 2 Sent Done Done false
17754 5129 Acked Read 3 Sent Done Done false
17755 5130 Acked Write 4 Sent Done Done false
17756 5131 Acked Read 2 Sent Done Done false
17757 5132 Acked Write 3 Sent Done Done false
17758 5133 Acked Flush 0 Sent Done Done false
17759 5134 Acked Read 6 Sent Done Done false
17760 5135 Acked Read 2 Sent Done Done false
17761 5136 Acked Read 7 Sent Done Done false
17762 5137 Acked Read 10 Sent Done Done false
17763 5138 Acked Write 5 Sent Done Done false
17764 5139 Acked Read 5 Sent Done Done false
17765 5140 Acked Write 2 Sent Done Done false
17766 5141 Acked Write 3 Sent Done Done false
17767 5142 Acked Read 4 Sent Done Done false
17768 5143 Acked Read 2 Sent Done Done false
17769 5144 Acked Write 8 Sent Done Done false
17770 5145 Acked Write 7 Sent Done Done false
17771 5146 Acked Read 8 Sent Done Done false
17772 5147 Acked Write 1 Sent Done Done false
17773 5148 Acked Read 3 Sent Done Done false
17774 5149 Acked Write 8 Sent Done Done false
17775 5150 Acked Read 5 Sent Done Done false
17776 5151 Acked Write 9 Sent Done Done false
17777 5152 Acked Flush 0 Sent Done Done false
17778 5153 Acked Write 4 Sent Done Done false
17779 5154 Acked Write 3 Sent Done Done false
17780 5155 Acked Read 9 Sent Done Done false
17781 5156 Acked Write 3 Sent Done Done false
17782 5157 Acked Write 1 Sent Done Done false
17783 5158 Acked Write 8 Sent Done Done false
17784 5159 Acked Read 4 Sent Done Done false
17785 5160 Acked Read 2 Sent Done Done false
17786 5161 Acked Read 1 Sent Done Done false
17787 5162 Acked Read 8 Sent Done Done false
17788 5163 Acked Read 6 Sent Done Done false
17789 5164 Acked Read 5 Sent Done Done false
17790 5165 Acked Read 9 Sent Done Done false
17791 5166 Acked Read 8 Sent Done Done false
17792 5167 Acked Read 6 Sent Done Done false
17793 5168 Acked Read 6 Sent Done Done false
17794 5169 Acked Read 6 Sent Done Done false
17795 5170 Acked Write 2 Sent Done Done false
17796 5171 Acked Read 9 Sent Done Done false
17797 5172 Acked Read 3 Sent Done Done false
17798 5173 Acked Flush 0 Sent Done Done false
17799 5174 Acked Flush 0 Sent Done Done false
17800 5175 Acked Read 3 Sent Done Done false
17801 5176 Acked Write 10 Sent Done Done false
17802 5177 Acked Read 10 Sent Done Done false
17803 5178 Acked Read 6 Sent Done Done false
17804 5179 Acked Write 8 Sent Done Done false
17805 5180 Acked Flush 0 Sent Done Done false
17806 5181 Acked Write 6 Sent Done Done false
17807 5182 Acked Read 10 Sent Done Done false
17808 5183 Acked Write 7 Sent Done Done false
17809 5184 Acked Read 5 Sent Done Done false
17810 5185 Acked Read 7 Sent Done Done false
17811 5186 Acked Read 5 Sent Done Done false
17812 5187 Acked Write 4 Sent Done Done false
17813 5188 Acked Write 5 Sent Done Done false
17814 5189 Acked Read 6 Sent Done Done false
17815 5190 Acked Read 6 Sent Done Done false
17816 5191 Acked Flush 0 Sent Done Done false
17817 5192 Acked Write 5 Sent Done Done false
17818 5193 Acked Write 4 Sent Done Done false
17819 5194 Acked Write 9 Sent Done Done false
17820 5195 Acked Read 9 Sent Done Done false
17821 5196 Acked Write 6 Sent Done Done false
17822 5197 Acked Read 9 Sent Done Done false
17823 5198 Acked Read 7 Sent Done Done false
17824 5199 Acked Write 3 Sent Done Done false
17825 5200 Acked Read 2 Sent Done Done false
17826 5201 Acked Read 6 Sent Done Done false
17827 5202 Acked Read 1 Sent Done Done false
17828 5203 Acked Read 3 Sent Done Done false
17829 5204 Acked Read 3 Sent Done Done false
17830 5205 Acked Read 9 Sent Done Done false
17831 5206 Acked Read 6 Sent Done Done false
17832 5207 Acked Write 3 Sent Done Done false
17833 5208 Acked Read 8 Sent Done Done false
17834 5209 Acked Write 2 Sent Done Done false
17835 5210 Acked Read 7 Sent Done Done false
17836 5211 Acked Write 1 Sent Done Done false
17837 5212 Acked Read 10 Sent Done Done false
17838 5213 Acked Read 7 Sent Done Done false
17839 5214 Acked Read 4 Sent Done Done false
17840 5215 Acked Write 6 Sent Done Done false
17841 5216 Acked Read 2 Sent Done Done false
17842 5217 Acked Write 10 Sent Done Done false
17843 5218 Acked Read 7 Sent Done Done false
17844 5219 Acked Read 2 Sent Done Done false
17845 5220 Acked Read 10 Sent Done Done false
17846 5221 Acked Write 8 Sent Done Done false
17847 5222 Acked Write 2 Sent Done Done false
17848 5223 Acked Read 4 Sent Done Done false
17849 5224 Acked Read 8 Sent Done Done false
17850 5225 Acked Write 10 Sent Done Done false
17851 5226 Acked Write 2 Sent Done Done false
17852 STATES DS:0 DS:1 DS:2 TOTAL
17853 Sent 301 0 0 301
17854 Done 0 301 301 602
17855 Skipped 0 0 0 0
17856 Error 0 0 0 0
17857 Last Flush: Some(JobId(4925)) Some(JobId(5191)) Some(JobId(5191))
17858 Downstairs last five completed: 4925 4924 4923 4922 4921
17859 Upstairs last five completed: 5226 5225 5224 5223 5222
17860 CLIENT: Up:0 ds:301 act:2
178612025-02-07T20:11:17.514ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
178622025-02-07T20:11:18.330ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
178632025-02-07T20:11:18.774ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
178642025-02-07T20:11:18.774ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63922) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
178652025-02-07T20:11:18.775ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
178662025-02-07T20:11:18.775ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
178672025-02-07T20:11:18.775ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
178682025-02-07T20:11:18.775ZINFOcrucible: send last flush ID to this DS: Some(JobId(4925)) = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
178692025-02-07T20:11:18.776ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
178702025-02-07T20:11:18.776ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4925)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
178712025-02-07T20:11:18.776ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4925)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
17872 ----------------------------------------------------------------
17873 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17874 Downstairs last five completed: 5227 5226 5225 5224 5223
17875 Upstairs last five completed: 5227 5226 5225 5224 5223
17876 CLIENT: Up:0 ds:0 act:3
17877 Replay: All jobs finished
17878 Main task has work to do, go find it
17879 got dsc Stop(2)
17880 stop 2
17881 Feb 07 20:11:21.015 INFO request completed, latency_us[8830] Got stop action so:false kr:false
17882 : 382, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 0081f8d6-5c61-487c-9ad2-f482b1b8bbc8, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
17883 Feb 07 20:11:21.016 INFO request completed, latency_us: 125, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7964a393-ae1a-40b1-bad7-46b2dc5412b1, {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:11:21.016434171Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"2","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
17884 remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:11:21.016510205Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"2","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
17885 
178862025-02-07T20:11:21.016ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
178872025-02-07T20:11:21.016ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
17888 [8830] Exited with: ExitStatus(unix_wait_status(9))
17889 [8830][2] reports Exit
17890 Feb 07 20:11:25.017 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/2, method: GET, req_id: 73b304b0-aca3-4fcf-b5cc-7c1e5fe4a279, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
17891 001/300 Read block 1191 len 36864
17892 002/300 Read block 2929 len 16384
17893 003/300 Read block 3745 len 28672
17894 004/300 Write block 3069 len 28672 data: 2 2 2 4 4 3 4
17895 005/300 Read block 4775 len 20480
17896 006/300 Read block 196 len 12288
17897 007/300 Read block 2666 len 32768
17898 008/300 Write block 3007 len 36864 data: 4 2 3 3 3 3 3 3 3
17899 009/300 Write block 3883 len 4096 data: 6
17900 010/300 Read block 2612 len 8192
17901 011/300 Read block 3838 len 12288
17902 012/300 Read block 549 len 24576
17903 013/300 Read block 394 len 12288
17904 014/300 Read block 3530 len 40960
17905 015/300 Read block 2866 len 4096
17906 016/300 Write block 4027 len 24576 data: 4 4 5 5 6 6
17907 017/300 Read block 1432 len 16384
17908 018/300 Flush
17909 019/300 Write block 3608 len 16384 data: 4 4 3 4
17910 020/300 Flush
17911 021/300 Write block 4101 len 8192 data: 2 2
17912 022/300 Read block 1877 len 20480
17913 023/300 Read block 1466 len 24576
17914 024/300 Read block 3919 len 32768
17915 025/300 Write block 4058 len 24576 data: 2 3 4 5 5 5
17916 026/300 Read block 1141 len 36864
17917 027/300 Write block 4203 len 24576 data: 3 2 2 2 3 3
17918 028/300 Write block 4 len 12288 data: 3 3 3
17919 029/300 Write block 2470 len 16384 data: 4 3 3 3
17920 030/300 Write block 3905 len 32768 data: 5 5 4 4 3 4 5 6
17921 031/300 Read block 1455 len 16384
17922 032/300 Flush
17923 033/300 Flush
17924 034/300 Read block 651 len 36864
17925 035/300 Write block 3220 len 20480 data: 4 4 3 3 3
17926 036/300 Write block 4473 len 24576 data: 5 5 7 7 6 7
17927 037/300 Read block 4816 len 8192
17928 038/300 Read block 3832 len 8192
17929 039/300 Read block 256 len 36864
17930 040/300 Read block 449 len 8192
17931 041/300 Read block 4475 len 32768
17932 042/300 Read block 3009 len 12288
17933 043/300 Read block 2770 len 16384
17934 044/300 Read block 1991 len 32768
17935 045/300 Write block 2198 len 28672 data: 5 5 5 5 5 6 6
17936 046/300 Read block 109 len 12288
17937 047/300 Read block 4844 len 8192
17938 048/300 Read block 197 len 16384
17939 049/300 Read block 4756 len 20480
17940 050/300 Read block 2612 len 24576
17941 051/300 Read block 3368 len 28672
17942 052/300 Read block 869 len 28672
17943 053/300 Write block 4102 len 32768 data: 3 3 3 3 2 2 3 3
17944 054/300 Write block 382 len 40960 data: 3 2 3 3 2 2 2 2 2 2
17945 055/300 Read block 3747 len 40960
17946 056/300 Write block 205 len 8192 data: 4 4
17947 057/300 Read block 983 len 36864
17948 058/300 Read block 4167 len 24576
17949 059/300 Flush
17950 060/300 Read block 4824 len 12288
17951 061/300 Write block 585 len 20480 data: 4 4 4 3 3
17952 062/300 Write block 2472 len 8192 data: 4 4
17953 063/300 Read block 914 len 4096
17954 064/300 Read block 2013 len 16384
17955 065/300 Write block 1133 len 16384 data: 3 3 3 3
17956 066/300 Read block 1063 len 40960
17957 067/300 Write block 2382 len 12288 data: 5 5 5
17958 068/300 Write block 245 len 16384 data: 3 3 3 3
17959 069/300 Flush
17960 070/300 Flush
17961 071/300 Read block 1327 len 40960
17962 072/300 Write block 3018 len 20480 data: 2 2 2 2 3
17963 073/300 Write block 4825 len 8192 data: 4 3
17964 074/300 Write block 2465 len 40960 data: 5 3 3 3 3 5 4 5 5 3
17965 075/300 Read block 3938 len 36864
17966 076/300 Read block 247 len 16384
17967 077/300 Read block 759 len 12288
17968 078/300 Flush
17969 079/300 Write block 2345 len 28672 data: 4 5 5 5 4 4 3
17970 080/300 Read block 2962 len 36864
17971 081/300 Flush
17972 082/300 Flush
17973 083/300 Write block 3856 len 20480 data: 4 4 4 4 4
17974 084/300 Write block 1606 len 28672 data: 4 3 3 4 4 4 4
17975 085/300 Read block 1345 len 8192
17976 086/300 Read block 3529 len 40960
17977 087/300 Write block 824 len 32768 data: 4 5 5 4 6 6 6 6
17978 088/300 Read block 3307 len 12288
17979 089/300 Write block 3720 len 8192 data: 7 6
17980 090/300 Write block 4738 len 20480 data: 6 6 6 6 7
17981 091/300 Write block 3556 len 4096 data: 3
17982 092/300 Write block 1512 len 20480 data: 4 4 3 3 2
17983 093/300 Read block 1543 len 28672
17984 094/300 Read block 618 len 4096
17985 095/300 Read block 454 len 12288
17986 096/300 Write block 4373 len 40960 data: 4 5 6 5 5 5 4 3 3 3
17987 097/300 Read block 4026 len 16384
17988 098/300 Write block 4555 len 16384 data: 4 4 3 3
17989 099/300 Write block 4614 len 8192 data: 5 5
17990 100/300 Read block 1815 len 4096
17991 101/300 Read block 2111 len 12288
17992 102/300 Write block 2385 len 40960 data: 3 3 3 3 3 3 2 4 3 3
17993 103/300 Read block 3681 len 24576
17994 104/300 Read block 1658 len 24576
17995 105/300 Read block 2284 len 28672
17996 106/300 Write block 4481 len 12288 data: 4 4 4
17997 107/300 Read block 2334 len 20480
17998 108/300 Read block 1808 len 12288
17999 109/300 Write block 3410 len 16384 data: 5 5 5 5
18000 110/300 Read block 957 len 24576
18001 111/300 Read block 3324 len 8192
18002 112/300 Read block 3715 len 40960
18003 113/300 Read block 2608 len 28672
18004 114/300 Read block 4543 len 16384
18005 115/300 Flush
18006 116/300 Write block 3762 len 24576 data: 4 4 3 3 3 4
18007 117/300 Read block 4664 len 16384
18008 118/300 Read block 2941 len 12288
18009 119/300 Read block 1698 len 8192
18010 120/300 Read block 2401 len 8192
18011 121/300 Flush
18012 122/300 Read block 4149 len 40960
18013 123/300 Read block 2321 len 40960
18014 124/300 Read block 3192 len 12288
18015 125/300 Read block 297 len 8192
18016 126/300 Flush
18017 127/300 Read block 842 len 12288
18018 128/300 Read block 2635 len 4096
18019 129/300 Flush
18020 130/300 Write block 408 len 4096 data: 4
18021 131/300 Write block 2834 len 36864 data: 5 3 3 2 2 3 3 3 3
18022 132/300 Flush
18023 133/300 Read block 865 len 4096
18024 134/300 Flush
18025 135/300 Write block 818 len 32768 data: 4 4 3 3 4 4 5 6
18026 136/300 Read block 1184 len 36864
18027 137/300 Read block 1705 len 20480
18028 138/300 Write block 2114 len 20480 data: 3 3 3 3 3
18029 139/300 Read block 1581 len 36864
18030 140/300 Read block 3588 len 16384
18031 141/300 Read block 274 len 28672
18032 142/300 Write block 2362 len 20480 data: 5 7 7 5 6
18033 143/300 Read block 3620 len 28672
18034 144/300 Read block 4077 len 24576
18035 145/300 Read block 230 len 32768
18036 146/300 Write block 2485 len 24576 data: 2 2 2 2 2 2
18037 147/300 Read block 4271 len 16384
18038 148/300 Write block 4730 len 24576 data: 4 4 6 7 7 7
18039 149/300 Read block 306 len 12288
18040 150/300 Write block 3118 len 16384 data: 4 4 3 3
18041 151/300 Flush
18042 152/300 Write block 4800 len 12288 data: 5 7 6
18043 153/300 Write block 3161 len 12288 data: 4 4 5
18044 154/300 Read block 536 len 20480
18045 155/300 Write block 4890 len 24576 data: 5 4 3 4 4 5
18046 156/300 Write block 4896 len 4096 data: 5
18047 157/300 Flush
18048 158/300 Write block 4647 len 8192 data: 5 5
18049 159/300 Write block 2892 len 16384 data: 2 2 3 2
18050 160/300 Read block 3714 len 28672
18051 161/300 Read block 473 len 20480
18052 162/300 Read block 3699 len 28672
18053 163/300 Write block 4716 len 28672 data: 2 2 2 2 4 4 4
18054 164/300 Write block 1899 len 40960 data: 5 5 5 5 5 5 5 7 7 6
18055 165/300 Read block 3447 len 16384
18056 166/300 Read block 3126 len 8192
18057 167/300 Write block 3091 len 40960 data: 4 3 3 3 3 3 3 2 2 4
18058 168/300 Write block 2337 len 8192 data: 3 4
18059 169/300 Read block 1199 len 32768
18060 170/300 Read block 159 len 8192
18061 171/300 Write block 3571 len 28672 data: 7 6 6 4 4 5 5
18062 172/300 Write block 4510 len 8192 data: 8 9
18063 173/300 Write block 3202 len 8192 data: 3 5
18064 174/300 Read block 4755 len 16384
18065 175/300 Write block 1154 len 12288 data: 4 4 4
18066 176/300 Write block 1407 len 12288 data: 2 2 3
18067 177/300 Write block 4023 len 16384 data: 3 5 4 4
18068 178/300 Read block 618 len 20480
18069 179/300 Read block 3142 len 12288
18070 180/300 Write block 4238 len 8192 data: 3 3
18071 181/300 Read block 4687 len 24576
18072 182/300 Read block 451 len 16384
18073 183/300 Write block 557 len 12288 data: 3 3 3
18074 184/300 Read block 4756 len 32768
18075 185/300 Read block 4323 len 16384
18076 186/300 Flush
18077 187/300 Write block 2255 len 20480 data: 2 2 3 3 4
18078 188/300 Read block 1036 len 16384
18079 189/300 Write block 600 len 16384 data: 4 4 2 3
18080 190/300 Write block 2607 len 36864 data: 5 4 3 3 2 2 2 2 2
18081 191/300 Flush
18082 192/300 Read block 3247 len 32768
18083 193/300 Read block 276 len 12288
18084 194/300 Flush
18085 195/300 Write block 3601 len 28672 data: 5 2 2 2 2 2 4
18086 196/300 Write block 960 len 32768 data: 2 3 2 2 2 3 3 3
18087 197/300 Write block 576 len 28672 data: 3 3 3 3 3 3 4
18088 198/300 Read block 1429 len 20480
18089 199/300 Read block 1686 len 4096
18090 200/300 Write block 4286 len 24576 data: 2 2 2 3 3 3
18091 201/300 Flush
18092 202/300 Read block 3766 len 32768
18093 203/300 Read block 996 len 8192
18094 204/300 Write block 3861 len 8192 data: 2 2
18095 205/300 Flush
18096 206/300 Read block 4101 len 8192
18097 207/300 Write block 3844 len 28672 data: 4 4 4 3 3 4 3
18098 208/300 Write block 2012 len 24576 data: 4 4 3 3 3 6
18099 209/300 Read block 2570 len 32768
18100 210/300 Read block 4651 len 24576
18101 211/300 Read block 729 len 24576
18102 212/300 Read block 3648 len 32768
18103 213/300 Read block 2152 len 24576
18104 214/300 Write block 4384 len 24576 data: 3 3 2 2 2 2
18105 215/300 Read block 3041 len 16384
18106 216/300 Write block 3912 len 36864 data: 7 5 5 4 5 4 5 5 4
18107 217/300 Write block 4164 len 20480 data: 3 3 4 5 5
18108 218/300 Read block 1407 len 40960
18109 219/300 Write block 4952 len 24576 data: 3 4 4 5 5 6
18110 220/300 Read block 383 len 20480
18111 221/300 Read block 1672 len 20480
18112 222/300 Write block 426 len 12288 data: 4 3 4
18113 223/300 Read block 4467 len 20480
18114 224/300 Write block 64 len 28672 data: 3 5 5 5 4 4 3
18115 225/300 Write block 1922 len 28672 data: 4 4 4 4 3 3 3
18116 226/300 Read block 4503 len 36864
18117 227/300 Read block 1644 len 28672
18118 228/300 Flush
18119 229/300 Write block 3609 len 32768 data: 5 4 5 4 4 4 5 5
18120 230/300 Read block 996 len 24576
18121 231/300 Read block 3080 len 28672
18122 232/300 Flush
18123 233/300 Read block 444 len 28672
18124 234/300 Read block 1824 len 28672
18125 235/300 Read block 2869 len 28672
18126 236/300 Read block 1895 len 4096
18127 237/300 Flush
18128 238/300 Read block 278 len 8192
18129 239/300 Read block 464 len 32768
18130 240/300 Read block 3017 len 32768
18131 241/300 Read block 3742 len 32768
18132 242/300 Flush
18133 243/300 Read block 1541 len 40960
18134 244/300 Write block 2592 len 40960 data: 4 5 5 5 5 6 5 5 5 6
18135 245/300 Read block 473 len 20480
18136 246/300 Flush
18137 247/300 Write block 3688 len 12288 data: 5 4 4
18138 248/300 Read block 2920 len 12288
18139 249/300 Flush
18140 250/300 Read block 2997 len 12288
18141 251/300 Write block 7 len 28672 data: 3 2 3 3 4 5 5
18142 252/300 Flush
18143 253/300 Write block 2279 len 4096 data: 4
18144 254/300 Write block 3044 len 4096 data: 6
18145 255/300 Write block 1973 len 4096 data: 4
18146 256/300 Read block 964 len 12288
18147 257/300 Read block 2589 len 40960
18148 258/300 Read block 1350 len 20480
18149 259/300 Write block 4171 len 40960 data: 4 3 3 3 3 4 4 4 4 3
18150 260/300 Write block 3779 len 4096 data: 8
18151 261/300 Read block 4210 len 32768
18152 262/300 Write block 2774 len 36864 data: 4 4 4 4 4 3 3 3 4
18153 263/300 Write block 4475 len 24576 data: 8 8 7 8 6 6
18154 264/300 Write block 66 len 8192 data: 6 6
18155 265/300 Write block 4829 len 36864 data: 6 7 8 9 8 8 7 9 6
18156 266/300 Write block 2314 len 40960 data: 7 5 5 5 4 5 5 3 3 4
18157 267/300 Write block 2641 len 8192 data: 3 3
18158 268/300 Read block 2736 len 12288
18159 269/300 Read block 41 len 28672
18160 270/300 Write block 3678 len 28672 data: 6 6 6 7 6 5 6
18161 271/300 Read block 1492 len 40960
18162 272/300 Read block 2552 len 20480
18163 273/300 Write block 2595 len 16384 data: 6 6 7 6
18164 274/300 Write block 2020 len 24576 data: 6 5 4 4 4 4
18165 275/300 Read block 275 len 4096
18166 276/300 Write block 2723 len 36864 data: 2 2 2 2 2 2 3 3 3
18167 277/300 Write block 4734 len 16384 data: 8 8 6 5
18168 278/300 Read block 4315 len 24576
18169 279/300 Read block 3635 len 8192
18170 280/300 Write block 2417 len 8192 data: 2 2
18171 281/300 Read block 4663 len 8192
18172 282/300 Read block 2123 len 24576
18173 283/300 Write block 2241 len 36864 data: 4 4 4 3 3 2 2 2 3
18174 284/300 Write block 1476 len 12288 data: 4 4 4
18175 285/300 Flush
18176 286/300 Read block 2001 len 28672
18177 287/300 Write block 4835 len 8192 data: 8 10
18178 288/300 Flush
18179 289/300 Read block 1006 len 16384
18180 290/300 Read block 412 len 40960
18181 291/300 Read block 3104 len 20480
18182 292/300 Write block 4099 len 28672 data: 4 3 3 4 4 4 4
18183 293/300 Flush
18184 294/300 Write block 1897 len 20480 data: 5 5 6 6 6
18185 295/300 Read block 4684 len 28672
18186 296/300 Read block 1016 len 20480
18187 297/300 Flush
18188 298/300 Read block 4355 len 20480
18189 299/300 Read block 2408 len 32768
18190 300/300 Write block 2622 len 36864 data: 4 6 6 5 4 4 4 3 3
18191 301/300 Flush
18192 Main task has work to do, go find it
18193 got dsc Start(2)
18194 start 2
18195 Feb 07 20:11:25.263 INFO request completed, latency_us: 226, response_code: 204, uri: /start/cid/2, method: POST, req_id: 947810ef-9791-49d1-a1cc-7481619cfd5f, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
18196 Replay: started 2, returned:Ok(())
18197 ----------------------------------------------------------------
18198 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18199 States: Active Active Offline
18200 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18201 5228 Acked Read 9 Done Done Sent false
18202 5229 Acked Read 4 Done Done Sent false
18203 5230 Acked Read 7 Done Done Sent false
18204 5231 Acked Write 7 Done Done Sent false
18205 5232 Acked Read 5 Done Done Sent false
18206 5233 Acked Read 3 Done Done Sent false
18207 5234 Acked Read 8 Done Done Sent false
18208 5235 Acked Write 9 Done Done Sent false
18209 5236 Acked Write 1 Done Done Sent false
18210 5237 Acked Read 2 Done Done Sent false
18211 5238 Acked Read 3 Done Done Sent false
18212 5239 Acked Read 6 Done Done Sent false
18213 5240 Acked Read 3 Done Done Sent false
18214 5241 Acked Read 10 Done Done Sent false
18215 5242 Acked Read 1 Done Done Sent false
18216 5243 Acked Write 6 Done Done Sent false
18217 5244 Acked Read 4 Done Done Sent false
18218 5245 Acked Flush 0 Done Done Sent false
18219 5246 Acked Write 4 Done Done Sent false
18220 5247 Acked Flush 0 Done Done Sent false
18221 5248 Acked Write 2 Done Done Sent false
18222 5249 Acked Read 5 Done Done Sent false
18223 5250 Acked Read 6 Done Done Sent false
18224 5251 Acked Read 8 Done Done Sent false
18225 5252 Acked Write 6 Done Done Sent false
18226 5253 Acked Read 9 Done Done Sent false
18227 5254 Acked Write 6 Done Done Sent false
18228 5255 Acked Write 3 Done Done Sent false
18229 5256 Acked Write 4 Done Done Sent false
18230 5257 Acked Write 8 Done Done Sent false
18231 5258 Acked Read 4 Done Done Sent false
18232 5259 Acked Flush 0 Done Done Sent false
18233 5260 Acked Flush 0 Done Done Sent false
18234 5261 Acked Read 9 Done Done Sent false
18235 5262 Acked Write 5 Done Done Sent false
18236 5263 Acked Write 6 Done Done Sent false
18237 5264 Acked Read 2 Done Done Sent false
18238 5265 Acked Read 2 Done Done Sent false
18239 5266 Acked Read 9 Done Done Sent false
18240 5267 Acked Read 2 Done Done Sent false
18241 5268 Acked Read 8 Done Done Sent false
18242 5269 Acked Read 3 Done Done Sent false
18243 5270 Acked Read 4 Done Done Sent false
18244 5271 Acked Read 8 Done Done Sent false
18245 5272 Acked Write 7 Done Done Sent false
18246 5273 Acked Read 3 Done Done Sent false
18247 5274 Acked Read 2 Done Done Sent false
18248 5275 Acked Read 4 Done Done Sent false
18249 5276 Acked Read 5 Done Done Sent false
18250 5277 Acked Read 6 Done Done Sent false
18251 5278 Acked Read 7 Done Done Sent false
18252 5279 Acked Read 7 Done Done Sent false
18253 5280 Acked Write 8 Done Done Sent false
18254 5281 Acked Write 10 Done Done Sent false
18255 5282 Acked Read 10 Done Done Sent false
18256 5283 Acked Write 2 Done Done Sent false
18257 5284 Acked Read 9 Done Done Sent false
18258 5285 Acked Read 6 Done Done Sent false
18259 5286 Acked Flush 0 Done Done Sent false
18260 5287 Acked Read 3 Done Done Sent false
18261 5288 Acked Write 5 Done Done Sent false
18262 5289 Acked Write 2 Done Done Sent false
18263 5290 Acked Read 1 Done Done Sent false
18264 5291 Acked Read 4 Done Done Sent false
18265 5292 Acked Write 4 Done Done Sent false
18266 5293 Acked Read 10 Done Done Sent false
18267 5294 Acked Write 3 Done Done Sent false
18268 5295 Acked Write 4 Done Done Sent false
18269 5296 Acked Flush 0 Done Done Sent false
18270 5297 Acked Flush 0 Done Done Sent false
18271 5298 Acked Read 10 Done Done Sent false
18272 5299 Acked Write 5 Done Done Sent false
18273 5300 Acked Write 2 Done Done Sent false
18274 5301 Acked Write 10 Done Done Sent false
18275 5302 Acked Read 9 Done Done Sent false
18276 5303 Acked Read 4 Done Done Sent false
18277 5304 Acked Read 3 Done Done Sent false
18278 5305 Acked Flush 0 Done Done Sent false
18279 5306 Acked Write 7 Done Done Sent false
18280 5307 Acked Read 9 Done Done Sent false
18281 5308 Acked Flush 0 Done Done Sent false
18282 5309 Acked Flush 0 Done Done Sent false
18283 5310 Acked Write 5 Done Done Sent false
18284 5311 Acked Write 7 Done Done Sent false
18285 5312 Acked Read 2 Done Done Sent false
18286 5313 Acked Read 10 Done Done Sent false
18287 5314 Acked Write 8 Done Done Sent false
18288 5315 Acked Read 3 Done Done Sent false
18289 5316 Acked Write 2 Done Done Sent false
18290 5317 Acked Write 5 Done Done Sent false
18291 5318 Acked Write 1 Done Done Sent false
18292 5319 Acked Write 5 Done Done Sent false
18293 5320 Acked Read 7 Done Done Sent false
18294 5321 Acked Read 1 Done Done Sent false
18295 5322 Acked Read 3 Done Done Sent false
18296 5323 Acked Write 10 Done Done Sent false
18297 5324 Acked Read 4 Done Done Sent false
18298 5325 Acked Write 4 Done Done Sent false
18299 5326 Acked Write 2 Done Done Sent false
18300 5327 Acked Read 1 Done Done Sent false
18301 5328 Acked Read 3 Done Done Sent false
18302 5329 Acked Write 10 Done Done Sent false
18303 5330 Acked Read 6 Done Done Sent false
18304 5331 Acked Read 6 Done Done Sent false
18305 5332 Acked Read 7 Done Done Sent false
18306 5333 Acked Write 3 Done Done Sent false
18307 5334 Acked Read 5 Done Done Sent false
18308 5335 Acked Read 3 Done Done Sent false
18309 5336 Acked Write 4 Done Done Sent false
18310 5337 Acked Read 6 Done Done Sent false
18311 5338 Acked Read 2 Done Done Sent false
18312 5339 Acked Read 10 Done Done Sent false
18313 5340 Acked Read 7 Done Done Sent false
18314 5341 Acked Read 4 Done Done Sent false
18315 5342 Acked Flush 0 Done Done Sent false
18316 5343 Acked Write 6 Done Done Sent false
18317 5344 Acked Read 4 Done Done Sent false
18318 5345 Acked Read 3 Done Done Sent false
18319 5346 Acked Read 2 Done Done Sent false
18320 5347 Acked Read 2 Done Done Sent false
18321 5348 Acked Flush 0 Done Done Sent false
18322 5349 Acked Read 10 Done Done Sent false
18323 5350 Acked Read 10 Done Done Sent false
18324 5351 Acked Read 3 Done Done Sent false
18325 5352 Acked Read 2 Done Done Sent false
18326 5353 Acked Flush 0 Done Done Sent false
18327 5354 Acked Read 3 Done Done Sent false
18328 5355 Acked Read 1 Done Done Sent false
18329 5356 Acked Flush 0 Done Done Sent false
18330 5357 Acked Write 1 Done Done Sent false
18331 5358 Acked Write 9 Done Done Sent false
18332 5359 Acked Flush 0 Done Done Sent false
18333 5360 Acked Read 1 Done Done Sent false
18334 5361 Acked Flush 0 Done Done Sent false
18335 5362 Acked Write 8 Done Done Sent false
18336 5363 Acked Read 9 Done Done Sent false
18337 5364 Acked Read 5 Done Done Sent false
18338 5365 Acked Write 5 Done Done Sent false
18339 5366 Acked Read 9 Done Done Sent false
18340 5367 Acked Read 4 Done Done Sent false
18341 5368 Acked Read 7 Done Done Sent false
18342 5369 Acked Write 5 Done Done Sent false
18343 5370 Acked Read 7 Done Done Sent false
18344 5371 Acked Read 6 Done Done Sent false
18345 5372 Acked Read 8 Done Done Sent false
18346 5373 Acked Write 6 Done Done Sent false
18347 5374 Acked Read 4 Done Done Sent false
18348 5375 Acked Write 6 Done Done Sent false
18349 5376 Acked Read 3 Done Done Sent false
18350 5377 Acked Write 4 Done Done Sent false
18351 5378 Acked Flush 0 Done Done Sent false
18352 5379 Acked Write 3 Done Done Sent false
18353 5380 Acked Write 3 Done Done Sent false
18354 5381 Acked Read 5 Done Done Sent false
18355 5382 Acked Write 6 Done Done Sent false
18356 5383 Acked Write 1 Done Done Sent false
18357 5384 Acked Flush 0 Done Done Sent false
18358 5385 Acked Write 2 Done Done Sent false
18359 5386 Acked Write 4 Done Done Sent false
18360 5387 Acked Read 7 Done Done Sent false
18361 5388 Acked Read 5 Done Done Sent false
18362 5389 Acked Read 7 Done Done Sent false
18363 5390 Acked Write 7 Done Done Sent false
18364 5391 Acked Write 10 Done Done Sent false
18365 5392 Acked Read 4 Done Done Sent false
18366 5393 Acked Read 2 Done Done Sent false
18367 5394 Acked Write 10 Done Done Sent false
18368 5395 Acked Write 2 Done Done Sent false
18369 5396 Acked Read 8 Done Done Sent false
18370 5397 Acked Read 2 Done Done Sent false
18371 5398 Acked Write 7 Done Done Sent false
18372 5399 Acked Write 2 Done Done Sent false
18373 5400 Acked Write 2 Done Done Sent false
18374 5401 Acked Read 4 Done Done Sent false
18375 5402 Acked Write 3 Done Done Sent false
18376 5403 Acked Write 3 Done Done Sent false
18377 5404 Acked Write 4 Done Done Sent false
18378 5405 Acked Read 5 Done Done Sent false
18379 5406 Acked Read 3 Done Done Sent false
18380 5407 Acked Write 2 Done Done Sent false
18381 5408 Acked Read 6 Done Done Sent false
18382 5409 Acked Read 4 Done Done Sent false
18383 5410 Acked Write 3 Done Done Sent false
18384 5411 Acked Read 8 Done Done Sent false
18385 5412 Acked Read 4 Done Done Sent false
18386 5413 Acked Flush 0 Done Done Sent false
18387 5414 Acked Write 5 Done Done Sent false
18388 5415 Acked Read 4 Done Done Sent false
18389 5416 Acked Write 4 Done Done Sent false
18390 5417 Acked Write 9 Done Done Sent false
18391 5418 Acked Flush 0 Done Done Sent false
18392 5419 Acked Read 8 Done Done Sent false
18393 5420 Acked Read 3 Done Done Sent false
18394 5421 Acked Flush 0 Done Done Sent false
18395 5422 Acked Write 7 Done Done Sent false
18396 5423 Acked Write 8 Done Done Sent false
18397 5424 Acked Write 7 Done Done Sent false
18398 5425 Acked Read 5 Done Done Sent false
18399 5426 Acked Read 1 Done Done Sent false
18400 5427 Acked Write 6 Done Done Sent false
18401 5428 Acked Flush 0 Done Done Sent false
18402 5429 Acked Read 8 Done Done Sent false
18403 5430 Acked Read 2 Done Done Sent false
18404 5431 Acked Write 2 Done Done Sent false
18405 5432 Acked Flush 0 Done Done Sent false
18406 5433 Acked Read 2 Done Done Sent false
18407 5434 Acked Write 7 Done Done Sent false
18408 5435 Acked Write 6 Done Done Sent false
18409 5436 Acked Read 8 Done Done Sent false
18410 5437 Acked Read 6 Done Done Sent false
18411 5438 Acked Read 6 Done Done Sent false
18412 5439 Acked Read 8 Done Done Sent false
18413 5440 Acked Read 6 Done Done Sent false
18414 5441 Acked Write 6 Done Done Sent false
18415 5442 Acked Read 4 Done Done Sent false
18416 5443 Acked Write 9 Done Done Sent false
18417 5444 Acked Write 5 Done Done Sent false
18418 5445 Acked Read 10 Done Done Sent false
18419 5446 Acked Write 6 Done Done Sent false
18420 5447 Acked Read 5 Done Done Sent false
18421 5448 Acked Read 5 Done Done Sent false
18422 5449 Acked Write 3 Done Done Sent false
18423 5450 Acked Read 5 Done Done Sent false
18424 5451 Acked Write 7 Done Done Sent false
18425 5452 Acked Write 7 Done Done Sent false
18426 5453 Acked Read 9 Done Done Sent false
18427 5454 Acked Read 7 Done Done Sent false
18428 5455 Acked Flush 0 Done Done Sent false
18429 5456 Acked Write 8 Done Done Sent false
18430 5457 Acked Read 6 Done Done Sent false
18431 5458 Acked Read 7 Done Done Sent false
18432 5459 Acked Flush 0 Done Done Sent false
18433 5460 Acked Read 7 Done Done Sent false
18434 5461 Acked Read 7 Done Done Sent false
18435 5462 Acked Read 7 Done Done Sent false
18436 5463 Acked Read 1 Done Done Sent false
18437 5464 Acked Flush 0 Done Done Sent false
18438 5465 Acked Read 2 Done Done Sent false
18439 5466 Acked Read 8 Done Done Sent false
18440 5467 Acked Read 8 Done Done Sent false
18441 5468 Acked Read 8 Done Done Sent false
18442 5469 Acked Flush 0 Done Done Sent false
18443 5470 Acked Read 10 Done Done Sent false
18444 5471 Acked Write 10 Done Done Sent false
18445 5472 Acked Read 5 Done Done Sent false
18446 5473 Acked Flush 0 Done Done Sent false
18447 5474 Acked Write 3 Done Done Sent false
18448 5475 Acked Read 3 Done Done Sent false
18449 5476 Acked Flush 0 Done Done Sent false
18450 5477 Acked Read 3 Done Done Sent false
18451 5478 Acked Write 7 Done Done Sent false
18452 5479 Acked Flush 0 Done Done Sent false
18453 5480 Acked Write 1 Done Done Sent false
18454 5481 Acked Write 1 Done Done Sent false
18455 5482 Acked Write 1 Done Done Sent false
18456 5483 Acked Read 3 Done Done Sent false
18457 5484 Acked Read 10 Done Done Sent false
18458 5485 Acked Read 5 Done Done Sent false
18459 5486 Acked Write 10 Done Done Sent false
18460 5487 Acked Write 1 Done Done Sent false
18461 5488 Acked Read 8 Done Done Sent false
18462 5489 Acked Write 9 Done Done Sent false
18463 5490 Acked Write 6 Done Done Sent false
18464 5491 Acked Write 2 Done Done Sent false
18465 5492 Acked Write 9 Done Done Sent false
18466 5493 Acked Write 10 Done Done Sent false
18467 5494 Acked Write 2 Done Done Sent false
18468 5495 Acked Read 3 Done Done Sent false
18469 5496 Acked Read 7 Done Done Sent false
18470 5497 Acked Write 7 Done Done Sent false
18471 5498 Acked Read 10 Done Done Sent false
18472 5499 Acked Read 5 Done Done Sent false
18473 5500 Acked Write 4 Done Done Sent false
18474 5501 Acked Write 6 Done Done Sent false
18475 5502 Acked Read 1 Done Done Sent false
18476 5503 Acked Write 9 Done Done Sent false
18477 5504 Acked Write 4 Done Done Sent false
18478 5505 Acked Read 6 Done Done Sent false
18479 5506 Acked Read 2 Done Done Sent false
18480 5507 Acked Write 2 Done Done Sent false
18481 5508 Acked Read 2 Done Done Sent false
18482 5509 Acked Read 6 Done Done Sent false
18483 5510 Acked Write 9 Done Done Sent false
18484 5511 Acked Write 3 Done Done Sent false
18485 5512 Acked Flush 0 Done Done Sent false
18486 5513 Acked Read 7 Done Done Sent false
18487 5514 Acked Write 2 Done Done Sent false
18488 5515 Acked Flush 0 Done Done Sent false
18489 5516 Acked Read 4 Done Done Sent false
18490 5517 Acked Read 10 Done Done Sent false
18491 5518 Acked Read 5 Done Done Sent false
18492 5519 Acked Write 7 Done Done Sent false
18493 5520 Acked Flush 0 Done Done Sent false
18494 5521 Acked Write 5 Done Done Sent false
18495 5522 Acked Read 7 Done Done Sent false
18496 5523 Acked Read 5 Done Done Sent false
18497 5524 Acked Flush 0 Done Done Sent false
18498 5525 Acked Read 5 Done Done Sent false
18499 5526 Acked Read 8 Done Done Sent false
18500 5527 Acked Write 9 Done Done Sent false
18501 5528 Acked Flush 0 Done Done Sent false
18502 STATES DS:0 DS:1 DS:2 TOTAL
18503 Sent 0 0 301 301
18504 Done 301 301 0 602
18505 Skipped 0 0 0 0
18506 Error 0 0 0 0
18507 Last Flush: Some(JobId(5528)) Some(JobId(5528)) Some(JobId(5227))
18508 Downstairs last five completed: 5227 5226 5225 5224 5223
18509 Upstairs last five completed: 5528 5527 5526 5525 5524
18510 CLIENT: Up:0 ds:301 act:2
185112025-02-07T20:11:25.765ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
185122025-02-07T20:11:26.342ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
185132025-02-07T20:11:27.343ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
185142025-02-07T20:11:28.345ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
18515 ----------------------------------------------------------------
18516 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18517 States: Active Active Offline
18518 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18519 5228 Acked Read 9 Done Done Sent false
18520 5229 Acked Read 4 Done Done Sent false
18521 5230 Acked Read 7 Done Done Sent false
18522 5231 Acked Write 7 Done Done Sent false
18523 5232 Acked Read 5 Done Done Sent false
18524 5233 Acked Read 3 Done Done Sent false
18525 5234 Acked Read 8 Done Done Sent false
18526 5235 Acked Write 9 Done Done Sent false
18527 5236 Acked Write 1 Done Done Sent false
18528 5237 Acked Read 2 Done Done Sent false
18529 5238 Acked Read 3 Done Done Sent false
18530 5239 Acked Read 6 Done Done Sent false
18531 5240 Acked Read 3 Done Done Sent false
18532 5241 Acked Read 10 Done Done Sent false
18533 5242 Acked Read 1 Done Done Sent false
18534 5243 Acked Write 6 Done Done Sent false
18535 5244 Acked Read 4 Done Done Sent false
18536 5245 Acked Flush 0 Done Done Sent false
18537 5246 Acked Write 4 Done Done Sent false
18538 5247 Acked Flush 0 Done Done Sent false
18539 5248 Acked Write 2 Done Done Sent false
18540 5249 Acked Read 5 Done Done Sent false
18541 5250 Acked Read 6 Done Done Sent false
18542 5251 Acked Read 8 Done Done Sent false
18543 5252 Acked Write 6 Done Done Sent false
18544 5253 Acked Read 9 Done Done Sent false
18545 5254 Acked Write 6 Done Done Sent false
18546 5255 Acked Write 3 Done Done Sent false
18547 5256 Acked Write 4 Done Done Sent false
18548 5257 Acked Write 8 Done Done Sent false
18549 5258 Acked Read 4 Done Done Sent false
18550 5259 Acked Flush 0 Done Done Sent false
18551 5260 Acked Flush 0 Done Done Sent false
18552 5261 Acked Read 9 Done Done Sent false
18553 5262 Acked Write 5 Done Done Sent false
18554 5263 Acked Write 6 Done Done Sent false
18555 5264 Acked Read 2 Done Done Sent false
18556 5265 Acked Read 2 Done Done Sent false
18557 5266 Acked Read 9 Done Done Sent false
18558 5267 Acked Read 2 Done Done Sent false
18559 5268 Acked Read 8 Done Done Sent false
18560 5269 Acked Read 3 Done Done Sent false
18561 5270 Acked Read 4 Done Done Sent false
18562 5271 Acked Read 8 Done Done Sent false
18563 5272 Acked Write 7 Done Done Sent false
18564 5273 Acked Read 3 Done Done Sent false
18565 5274 Acked Read 2 Done Done Sent false
18566 5275 Acked Read 4 Done Done Sent false
18567 5276 Acked Read 5 Done Done Sent false
18568 5277 Acked Read 6 Done Done Sent false
18569 5278 Acked Read 7 Done Done Sent false
18570 5279 Acked Read 7 Done Done Sent false
18571 5280 Acked Write 8 Done Done Sent false
18572 5281 Acked Write 10 Done Done Sent false
18573 5282 Acked Read 10 Done Done Sent false
18574 5283 Acked Write 2 Done Done Sent false
18575 5284 Acked Read 9 Done Done Sent false
18576 5285 Acked Read 6 Done Done Sent false
18577 5286 Acked Flush 0 Done Done Sent false
18578 5287 Acked Read 3 Done Done Sent false
18579 5288 Acked Write 5 Done Done Sent false
18580 5289 Acked Write 2 Done Done Sent false
18581 5290 Acked Read 1 Done Done Sent false
18582 5291 Acked Read 4 Done Done Sent false
18583 5292 Acked Write 4 Done Done Sent false
18584 5293 Acked Read 10 Done Done Sent false
18585 5294 Acked Write 3 Done Done Sent false
18586 5295 Acked Write 4 Done Done Sent false
18587 5296 Acked Flush 0 Done Done Sent false
18588 5297 Acked Flush 0 Done Done Sent false
18589 5298 Acked Read 10 Done Done Sent false
18590 5299 Acked Write 5 Done Done Sent false
18591 5300 Acked Write 2 Done Done Sent false
18592 5301 Acked Write 10 Done Done Sent false
18593 5302 Acked Read 9 Done Done Sent false
18594 5303 Acked Read 4 Done Done Sent false
18595 5304 Acked Read 3 Done Done Sent false
18596 5305 Acked Flush 0 Done Done Sent false
18597 5306 Acked Write 7 Done Done Sent false
18598 5307 Acked Read 9 Done Done Sent false
18599 5308 Acked Flush 0 Done Done Sent false
18600 5309 Acked Flush 0 Done Done Sent false
18601 5310 Acked Write 5 Done Done Sent false
18602 5311 Acked Write 7 Done Done Sent false
18603 5312 Acked Read 2 Done Done Sent false
18604 5313 Acked Read 10 Done Done Sent false
18605 5314 Acked Write 8 Done Done Sent false
18606 5315 Acked Read 3 Done Done Sent false
18607 5316 Acked Write 2 Done Done Sent false
18608 5317 Acked Write 5 Done Done Sent false
18609 5318 Acked Write 1 Done Done Sent false
18610 5319 Acked Write 5 Done Done Sent false
18611 5320 Acked Read 7 Done Done Sent false
18612 5321 Acked Read 1 Done Done Sent false
18613 5322 Acked Read 3 Done Done Sent false
18614 5323 Acked Write 10 Done Done Sent false
18615 5324 Acked Read 4 Done Done Sent false
18616 5325 Acked Write 4 Done Done Sent false
18617 5326 Acked Write 2 Done Done Sent false
18618 5327 Acked Read 1 Done Done Sent false
18619 5328 Acked Read 3 Done Done Sent false
18620 5329 Acked Write 10 Done Done Sent false
18621 5330 Acked Read 6 Done Done Sent false
18622 5331 Acked Read 6 Done Done Sent false
18623 5332 Acked Read 7 Done Done Sent false
18624 5333 Acked Write 3 Done Done Sent false
18625 5334 Acked Read 5 Done Done Sent false
18626 5335 Acked Read 3 Done Done Sent false
18627 5336 Acked Write 4 Done Done Sent false
18628 5337 Acked Read 6 Done Done Sent false
18629 5338 Acked Read 2 Done Done Sent false
18630 5339 Acked Read 10 Done Done Sent false
18631 5340 Acked Read 7 Done Done Sent false
18632 5341 Acked Read 4 Done Done Sent false
18633 5342 Acked Flush 0 Done Done Sent false
18634 5343 Acked Write 6 Done Done Sent false
18635 5344 Acked Read 4 Done Done Sent false
18636 5345 Acked Read 3 Done Done Sent false
18637 5346 Acked Read 2 Done Done Sent false
18638 5347 Acked Read 2 Done Done Sent false
18639 5348 Acked Flush 0 Done Done Sent false
18640 5349 Acked Read 10 Done Done Sent false
18641 5350 Acked Read 10 Done Done Sent false
18642 5351 Acked Read 3 Done Done Sent false
18643 5352 Acked Read 2 Done Done Sent false
18644 5353 Acked Flush 0 Done Done Sent false
18645 5354 Acked Read 3 Done Done Sent false
18646 5355 Acked Read 1 Done Done Sent false
18647 5356 Acked Flush 0 Done Done Sent false
18648 5357 Acked Write 1 Done Done Sent false
18649 5358 Acked Write 9 Done Done Sent false
18650 5359 Acked Flush 0 Done Done Sent false
18651 5360 Acked Read 1 Done Done Sent false
18652 5361 Acked Flush 0 Done Done Sent false
18653 5362 Acked Write 8 Done Done Sent false
18654 5363 Acked Read 9 Done Done Sent false
18655 5364 Acked Read 5 Done Done Sent false
18656 5365 Acked Write 5 Done Done Sent false
18657 5366 Acked Read 9 Done Done Sent false
18658 5367 Acked Read 4 Done Done Sent false
18659 5368 Acked Read 7 Done Done Sent false
18660 5369 Acked Write 5 Done Done Sent false
18661 5370 Acked Read 7 Done Done Sent false
18662 5371 Acked Read 6 Done Done Sent false
18663 5372 Acked Read 8 Done Done Sent false
18664 5373 Acked Write 6 Done Done Sent false
18665 5374 Acked Read 4 Done Done Sent false
18666 5375 Acked Write 6 Done Done Sent false
18667 5376 Acked Read 3 Done Done Sent false
18668 5377 Acked Write 4 Done Done Sent false
18669 5378 Acked Flush 0 Done Done Sent false
18670 5379 Acked Write 3 Done Done Sent false
18671 5380 Acked Write 3 Done Done Sent false
18672 5381 Acked Read 5 Done Done Sent false
18673 5382 Acked Write 6 Done Done Sent false
18674 5383 Acked Write 1 Done Done Sent false
18675 5384 Acked Flush 0 Done Done Sent false
18676 5385 Acked Write 2 Done Done Sent false
18677 5386 Acked Write 4 Done Done Sent false
18678 5387 Acked Read 7 Done Done Sent false
18679 5388 Acked Read 5 Done Done Sent false
18680 5389 Acked Read 7 Done Done Sent false
18681 5390 Acked Write 7 Done Done Sent false
18682 5391 Acked Write 10 Done Done Sent false
18683 5392 Acked Read 4 Done Done Sent false
18684 5393 Acked Read 2 Done Done Sent false
18685 5394 Acked Write 10 Done Done Sent false
18686 5395 Acked Write 2 Done Done Sent false
18687 5396 Acked Read 8 Done Done Sent false
18688 5397 Acked Read 2 Done Done Sent false
18689 5398 Acked Write 7 Done Done Sent false
18690 5399 Acked Write 2 Done Done Sent false
18691 5400 Acked Write 2 Done Done Sent false
18692 5401 Acked Read 4 Done Done Sent false
18693 5402 Acked Write 3 Done Done Sent false
18694 5403 Acked Write 3 Done Done Sent false
18695 5404 Acked Write 4 Done Done Sent false
18696 5405 Acked Read 5 Done Done Sent false
18697 5406 Acked Read 3 Done Done Sent false
18698 5407 Acked Write 2 Done Done Sent false
18699 5408 Acked Read 6 Done Done Sent false
18700 5409 Acked Read 4 Done Done Sent false
18701 5410 Acked Write 3 Done Done Sent false
18702 5411 Acked Read 8 Done Done Sent false
18703 5412 Acked Read 4 Done Done Sent false
18704 5413 Acked Flush 0 Done Done Sent false
18705 5414 Acked Write 5 Done Done Sent false
18706 5415 Acked Read 4 Done Done Sent false
18707 5416 Acked Write 4 Done Done Sent false
18708 5417 Acked Write 9 Done Done Sent false
18709 5418 Acked Flush 0 Done Done Sent false
18710 5419 Acked Read 8 Done Done Sent false
18711 5420 Acked Read 3 Done Done Sent false
18712 5421 Acked Flush 0 Done Done Sent false
18713 5422 Acked Write 7 Done Done Sent false
18714 5423 Acked Write 8 Done Done Sent false
18715 5424 Acked Write 7 Done Done Sent false
18716 5425 Acked Read 5 Done Done Sent false
18717 5426 Acked Read 1 Done Done Sent false
18718 5427 Acked Write 6 Done Done Sent false
18719 5428 Acked Flush 0 Done Done Sent false
18720 5429 Acked Read 8 Done Done Sent false
18721 5430 Acked Read 2 Done Done Sent false
18722 5431 Acked Write 2 Done Done Sent false
18723 5432 Acked Flush 0 Done Done Sent false
18724 5433 Acked Read 2 Done Done Sent false
18725 5434 Acked Write 7 Done Done Sent false
18726 5435 Acked Write 6 Done Done Sent false
18727 5436 Acked Read 8 Done Done Sent false
18728 5437 Acked Read 6 Done Done Sent false
18729 5438 Acked Read 6 Done Done Sent false
18730 5439 Acked Read 8 Done Done Sent false
18731 5440 Acked Read 6 Done Done Sent false
18732 5441 Acked Write 6 Done Done Sent false
18733 5442 Acked Read 4 Done Done Sent false
18734 5443 Acked Write 9 Done Done Sent false
18735 5444 Acked Write 5 Done Done Sent false
18736 5445 Acked Read 10 Done Done Sent false
18737 5446 Acked Write 6 Done Done Sent false
18738 5447 Acked Read 5 Done Done Sent false
18739 5448 Acked Read 5 Done Done Sent false
18740 5449 Acked Write 3 Done Done Sent false
18741 5450 Acked Read 5 Done Done Sent false
18742 5451 Acked Write 7 Done Done Sent false
18743 5452 Acked Write 7 Done Done Sent false
18744 5453 Acked Read 9 Done Done Sent false
18745 5454 Acked Read 7 Done Done Sent false
18746 5455 Acked Flush 0 Done Done Sent false
18747 5456 Acked Write 8 Done Done Sent false
18748 5457 Acked Read 6 Done Done Sent false
18749 5458 Acked Read 7 Done Done Sent false
18750 5459 Acked Flush 0 Done Done Sent false
18751 5460 Acked Read 7 Done Done Sent false
18752 5461 Acked Read 7 Done Done Sent false
18753 5462 Acked Read 7 Done Done Sent false
18754 5463 Acked Read 1 Done Done Sent false
18755 5464 Acked Flush 0 Done Done Sent false
18756 5465 Acked Read 2 Done Done Sent false
18757 5466 Acked Read 8 Done Done Sent false
18758 5467 Acked Read 8 Done Done Sent false
18759 5468 Acked Read 8 Done Done Sent false
18760 5469 Acked Flush 0 Done Done Sent false
18761 5470 Acked Read 10 Done Done Sent false
18762 5471 Acked Write 10 Done Done Sent false
18763 5472 Acked Read 5 Done Done Sent false
18764 5473 Acked Flush 0 Done Done Sent false
18765 5474 Acked Write 3 Done Done Sent false
18766 5475 Acked Read 3 Done Done Sent false
18767 5476 Acked Flush 0 Done Done Sent false
18768 5477 Acked Read 3 Done Done Sent false
18769 5478 Acked Write 7 Done Done Sent false
18770 5479 Acked Flush 0 Done Done Sent false
18771 5480 Acked Write 1 Done Done Sent false
18772 5481 Acked Write 1 Done Done Sent false
18773 5482 Acked Write 1 Done Done Sent false
18774 5483 Acked Read 3 Done Done Sent false
18775 5484 Acked Read 10 Done Done Sent false
18776 5485 Acked Read 5 Done Done Sent false
18777 5486 Acked Write 10 Done Done Sent false
18778 5487 Acked Write 1 Done Done Sent false
18779 5488 Acked Read 8 Done Done Sent false
18780 5489 Acked Write 9 Done Done Sent false
18781 5490 Acked Write 6 Done Done Sent false
18782 5491 Acked Write 2 Done Done Sent false
18783 5492 Acked Write 9 Done Done Sent false
18784 5493 Acked Write 10 Done Done Sent false
18785 5494 Acked Write 2 Done Done Sent false
18786 5495 Acked Read 3 Done Done Sent false
18787 5496 Acked Read 7 Done Done Sent false
18788 5497 Acked Write 7 Done Done Sent false
18789 5498 Acked Read 10 Done Done Sent false
18790 5499 Acked Read 5 Done Done Sent false
18791 5500 Acked Write 4 Done Done Sent false
18792 5501 Acked Write 6 Done Done Sent false
18793 5502 Acked Read 1 Done Done Sent false
18794 5503 Acked Write 9 Done Done Sent false
18795 5504 Acked Write 4 Done Done Sent false
18796 5505 Acked Read 6 Done Done Sent false
18797 5506 Acked Read 2 Done Done Sent false
18798 5507 Acked Write 2 Done Done Sent false
18799 5508 Acked Read 2 Done Done Sent false
18800 5509 Acked Read 6 Done Done Sent false
18801 5510 Acked Write 9 Done Done Sent false
18802 5511 Acked Write 3 Done Done Sent false
18803 5512 Acked Flush 0 Done Done Sent false
18804 5513 Acked Read 7 Done Done Sent false
18805 5514 Acked Write 2 Done Done Sent false
18806 5515 Acked Flush 0 Done Done Sent false
18807 5516 Acked Read 4 Done Done Sent false
18808 5517 Acked Read 10 Done Done Sent false
18809 5518 Acked Read 5 Done Done Sent false
18810 5519 Acked Write 7 Done Done Sent false
18811 5520 Acked Flush 0 Done Done Sent false
18812 5521 Acked Write 5 Done Done Sent false
18813 5522 Acked Read 7 Done Done Sent false
18814 5523 Acked Read 5 Done Done Sent false
18815 5524 Acked Flush 0 Done Done Sent false
18816 5525 Acked Read 5 Done Done Sent false
18817 5526 Acked Read 8 Done Done Sent false
18818 5527 Acked Write 9 Done Done Sent false
18819 5528 Acked Flush 0 Done Done Sent false
18820 STATES DS:0 DS:1 DS:2 TOTAL
18821 Sent 0 0 301 301
18822 Done 301 301 0 602
18823 Skipped 0 0 0 0
18824 Error 0 0 0 0
18825 Last Flush: Some(JobId(5528)) Some(JobId(5528)) Some(JobId(5227))
18826 Downstairs last five completed: 5227 5226 5225 5224 5223
18827 Upstairs last five completed: 5528 5527 5526 5525 5524
18828 CLIENT: Up:0 ds:301 act:2
188292025-02-07T20:11:29.767ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
188302025-02-07T20:11:30.349ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
188312025-02-07T20:11:31.017ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
188322025-02-07T20:11:31.017ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
188332025-02-07T20:11:31.017ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
188342025-02-07T20:11:31.017ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
188352025-02-07T20:11:31.017ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
18836 [8830] got start action
18837 [8830] I am going to restart
18838 Starting downstairs at port 8830
18839 Make output file at "/tmp/dsc/downstairs-8830.txt"
18840 [8830][2] reports Starting
18841 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1224)
18842 [8830][2] reports Running
188432025-02-07T20:11:31.518ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
188442025-02-07T20:11:32.351ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
18845 ----------------------------------------------------------------
18846 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18847 States: Active Active Offline
18848 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18849 5228 Acked Read 9 Done Done Sent false
18850 5229 Acked Read 4 Done Done Sent false
18851 5230 Acked Read 7 Done Done Sent false
18852 5231 Acked Write 7 Done Done Sent false
18853 5232 Acked Read 5 Done Done Sent false
18854 5233 Acked Read 3 Done Done Sent false
18855 5234 Acked Read 8 Done Done Sent false
18856 5235 Acked Write 9 Done Done Sent false
18857 5236 Acked Write 1 Done Done Sent false
18858 5237 Acked Read 2 Done Done Sent false
18859 5238 Acked Read 3 Done Done Sent false
18860 5239 Acked Read 6 Done Done Sent false
18861 5240 Acked Read 3 Done Done Sent false
18862 5241 Acked Read 10 Done Done Sent false
18863 5242 Acked Read 1 Done Done Sent false
18864 5243 Acked Write 6 Done Done Sent false
18865 5244 Acked Read 4 Done Done Sent false
18866 5245 Acked Flush 0 Done Done Sent false
18867 5246 Acked Write 4 Done Done Sent false
18868 5247 Acked Flush 0 Done Done Sent false
18869 5248 Acked Write 2 Done Done Sent false
18870 5249 Acked Read 5 Done Done Sent false
18871 5250 Acked Read 6 Done Done Sent false
18872 5251 Acked Read 8 Done Done Sent false
18873 5252 Acked Write 6 Done Done Sent false
18874 5253 Acked Read 9 Done Done Sent false
18875 5254 Acked Write 6 Done Done Sent false
18876 5255 Acked Write 3 Done Done Sent false
18877 5256 Acked Write 4 Done Done Sent false
18878 5257 Acked Write 8 Done Done Sent false
18879 5258 Acked Read 4 Done Done Sent false
18880 5259 Acked Flush 0 Done Done Sent false
18881 5260 Acked Flush 0 Done Done Sent false
18882 5261 Acked Read 9 Done Done Sent false
18883 5262 Acked Write 5 Done Done Sent false
18884 5263 Acked Write 6 Done Done Sent false
18885 5264 Acked Read 2 Done Done Sent false
18886 5265 Acked Read 2 Done Done Sent false
18887 5266 Acked Read 9 Done Done Sent false
18888 5267 Acked Read 2 Done Done Sent false
18889 5268 Acked Read 8 Done Done Sent false
18890 5269 Acked Read 3 Done Done Sent false
18891 5270 Acked Read 4 Done Done Sent false
18892 5271 Acked Read 8 Done Done Sent false
18893 5272 Acked Write 7 Done Done Sent false
18894 5273 Acked Read 3 Done Done Sent false
18895 5274 Acked Read 2 Done Done Sent false
18896 5275 Acked Read 4 Done Done Sent false
18897 5276 Acked Read 5 Done Done Sent false
18898 5277 Acked Read 6 Done Done Sent false
18899 5278 Acked Read 7 Done Done Sent false
18900 5279 Acked Read 7 Done Done Sent false
18901 5280 Acked Write 8 Done Done Sent false
18902 5281 Acked Write 10 Done Done Sent false
18903 5282 Acked Read 10 Done Done Sent false
18904 5283 Acked Write 2 Done Done Sent false
18905 5284 Acked Read 9 Done Done Sent false
18906 5285 Acked Read 6 Done Done Sent false
18907 5286 Acked Flush 0 Done Done Sent false
18908 5287 Acked Read 3 Done Done Sent false
18909 5288 Acked Write 5 Done Done Sent false
18910 5289 Acked Write 2 Done Done Sent false
18911 5290 Acked Read 1 Done Done Sent false
18912 5291 Acked Read 4 Done Done Sent false
18913 5292 Acked Write 4 Done Done Sent false
18914 5293 Acked Read 10 Done Done Sent false
18915 5294 Acked Write 3 Done Done Sent false
18916 5295 Acked Write 4 Done Done Sent false
18917 5296 Acked Flush 0 Done Done Sent false
18918 5297 Acked Flush 0 Done Done Sent false
18919 5298 Acked Read 10 Done Done Sent false
18920 5299 Acked Write 5 Done Done Sent false
18921 5300 Acked Write 2 Done Done Sent false
18922 5301 Acked Write 10 Done Done Sent false
18923 5302 Acked Read 9 Done Done Sent false
18924 5303 Acked Read 4 Done Done Sent false
18925 5304 Acked Read 3 Done Done Sent false
18926 5305 Acked Flush 0 Done Done Sent false
18927 5306 Acked Write 7 Done Done Sent false
18928 5307 Acked Read 9 Done Done Sent false
18929 5308 Acked Flush 0 Done Done Sent false
18930 5309 Acked Flush 0 Done Done Sent false
18931 5310 Acked Write 5 Done Done Sent false
18932 5311 Acked Write 7 Done Done Sent false
18933 5312 Acked Read 2 Done Done Sent false
18934 5313 Acked Read 10 Done Done Sent false
18935 5314 Acked Write 8 Done Done Sent false
18936 5315 Acked Read 3 Done Done Sent false
18937 5316 Acked Write 2 Done Done Sent false
18938 5317 Acked Write 5 Done Done Sent false
18939 5318 Acked Write 1 Done Done Sent false
18940 5319 Acked Write 5 Done Done Sent false
18941 5320 Acked Read 7 Done Done Sent false
18942 5321 Acked Read 1 Done Done Sent false
18943 5322 Acked Read 3 Done Done Sent false
18944 5323 Acked Write 10 Done Done Sent false
18945 5324 Acked Read 4 Done Done Sent false
18946 5325 Acked Write 4 Done Done Sent false
18947 5326 Acked Write 2 Done Done Sent false
18948 5327 Acked Read 1 Done Done Sent false
18949 5328 Acked Read 3 Done Done Sent false
18950 5329 Acked Write 10 Done Done Sent false
18951 5330 Acked Read 6 Done Done Sent false
18952 5331 Acked Read 6 Done Done Sent false
18953 5332 Acked Read 7 Done Done Sent false
18954 5333 Acked Write 3 Done Done Sent false
18955 5334 Acked Read 5 Done Done Sent false
18956 5335 Acked Read 3 Done Done Sent false
18957 5336 Acked Write 4 Done Done Sent false
18958 5337 Acked Read 6 Done Done Sent false
18959 5338 Acked Read 2 Done Done Sent false
18960 5339 Acked Read 10 Done Done Sent false
18961 5340 Acked Read 7 Done Done Sent false
18962 5341 Acked Read 4 Done Done Sent false
18963 5342 Acked Flush 0 Done Done Sent false
18964 5343 Acked Write 6 Done Done Sent false
18965 5344 Acked Read 4 Done Done Sent false
18966 5345 Acked Read 3 Done Done Sent false
18967 5346 Acked Read 2 Done Done Sent false
18968 5347 Acked Read 2 Done Done Sent false
18969 5348 Acked Flush 0 Done Done Sent false
18970 5349 Acked Read 10 Done Done Sent false
18971 5350 Acked Read 10 Done Done Sent false
18972 5351 Acked Read 3 Done Done Sent false
18973 5352 Acked Read 2 Done Done Sent false
18974 5353 Acked Flush 0 Done Done Sent false
18975 5354 Acked Read 3 Done Done Sent false
18976 5355 Acked Read 1 Done Done Sent false
18977 5356 Acked Flush 0 Done Done Sent false
18978 5357 Acked Write 1 Done Done Sent false
18979 5358 Acked Write 9 Done Done Sent false
18980 5359 Acked Flush 0 Done Done Sent false
18981 5360 Acked Read 1 Done Done Sent false
18982 5361 Acked Flush 0 Done Done Sent false
18983 5362 Acked Write 8 Done Done Sent false
18984 5363 Acked Read 9 Done Done Sent false
18985 5364 Acked Read 5 Done Done Sent false
18986 5365 Acked Write 5 Done Done Sent false
18987 5366 Acked Read 9 Done Done Sent false
18988 5367 Acked Read 4 Done Done Sent false
18989 5368 Acked Read 7 Done Done Sent false
18990 5369 Acked Write 5 Done Done Sent false
18991 5370 Acked Read 7 Done Done Sent false
18992 5371 Acked Read 6 Done Done Sent false
18993 5372 Acked Read 8 Done Done Sent false
18994 5373 Acked Write 6 Done Done Sent false
18995 5374 Acked Read 4 Done Done Sent false
18996 5375 Acked Write 6 Done Done Sent false
18997 5376 Acked Read 3 Done Done Sent false
18998 5377 Acked Write 4 Done Done Sent false
18999 5378 Acked Flush 0 Done Done Sent false
19000 5379 Acked Write 3 Done Done Sent false
19001 5380 Acked Write 3 Done Done Sent false
19002 5381 Acked Read 5 Done Done Sent false
19003 5382 Acked Write 6 Done Done Sent false
19004 5383 Acked Write 1 Done Done Sent false
19005 5384 Acked Flush 0 Done Done Sent false
19006 5385 Acked Write 2 Done Done Sent false
19007 5386 Acked Write 4 Done Done Sent false
19008 5387 Acked Read 7 Done Done Sent false
19009 5388 Acked Read 5 Done Done Sent false
19010 5389 Acked Read 7 Done Done Sent false
19011 5390 Acked Write 7 Done Done Sent false
19012 5391 Acked Write 10 Done Done Sent false
19013 5392 Acked Read 4 Done Done Sent false
19014 5393 Acked Read 2 Done Done Sent false
19015 5394 Acked Write 10 Done Done Sent false
19016 5395 Acked Write 2 Done Done Sent false
19017 5396 Acked Read 8 Done Done Sent false
19018 5397 Acked Read 2 Done Done Sent false
19019 5398 Acked Write 7 Done Done Sent false
19020 5399 Acked Write 2 Done Done Sent false
19021 5400 Acked Write 2 Done Done Sent false
19022 5401 Acked Read 4 Done Done Sent false
19023 5402 Acked Write 3 Done Done Sent false
19024 5403 Acked Write 3 Done Done Sent false
19025 5404 Acked Write 4 Done Done Sent false
19026 5405 Acked Read 5 Done Done Sent false
19027 5406 Acked Read 3 Done Done Sent false
19028 5407 Acked Write 2 Done Done Sent false
19029 5408 Acked Read 6 Done Done Sent false
19030 5409 Acked Read 4 Done Done Sent false
19031 5410 Acked Write 3 Done Done Sent false
19032 5411 Acked Read 8 Done Done Sent false
19033 5412 Acked Read 4 Done Done Sent false
19034 5413 Acked Flush 0 Done Done Sent false
19035 5414 Acked Write 5 Done Done Sent false
19036 5415 Acked Read 4 Done Done Sent false
19037 5416 Acked Write 4 Done Done Sent false
19038 5417 Acked Write 9 Done Done Sent false
19039 5418 Acked Flush 0 Done Done Sent false
19040 5419 Acked Read 8 Done Done Sent false
19041 5420 Acked Read 3 Done Done Sent false
19042 5421 Acked Flush 0 Done Done Sent false
19043 5422 Acked Write 7 Done Done Sent false
19044 5423 Acked Write 8 Done Done Sent false
19045 5424 Acked Write 7 Done Done Sent false
19046 5425 Acked Read 5 Done Done Sent false
19047 5426 Acked Read 1 Done Done Sent false
19048 5427 Acked Write 6 Done Done Sent false
19049 5428 Acked Flush 0 Done Done Sent false
19050 5429 Acked Read 8 Done Done Sent false
19051 5430 Acked Read 2 Done Done Sent false
19052 5431 Acked Write 2 Done Done Sent false
19053 5432 Acked Flush 0 Done Done Sent false
19054 5433 Acked Read 2 Done Done Sent false
19055 5434 Acked Write 7 Done Done Sent false
19056 5435 Acked Write 6 Done Done Sent false
19057 5436 Acked Read 8 Done Done Sent false
19058 5437 Acked Read 6 Done Done Sent false
19059 5438 Acked Read 6 Done Done Sent false
19060 5439 Acked Read 8 Done Done Sent false
19061 5440 Acked Read 6 Done Done Sent false
19062 5441 Acked Write 6 Done Done Sent false
19063 5442 Acked Read 4 Done Done Sent false
19064 5443 Acked Write 9 Done Done Sent false
19065 5444 Acked Write 5 Done Done Sent false
19066 5445 Acked Read 10 Done Done Sent false
19067 5446 Acked Write 6 Done Done Sent false
19068 5447 Acked Read 5 Done Done Sent false
19069 5448 Acked Read 5 Done Done Sent false
19070 5449 Acked Write 3 Done Done Sent false
19071 5450 Acked Read 5 Done Done Sent false
19072 5451 Acked Write 7 Done Done Sent false
19073 5452 Acked Write 7 Done Done Sent false
19074 5453 Acked Read 9 Done Done Sent false
19075 5454 Acked Read 7 Done Done Sent false
19076 5455 Acked Flush 0 Done Done Sent false
19077 5456 Acked Write 8 Done Done Sent false
19078 5457 Acked Read 6 Done Done Sent false
19079 5458 Acked Read 7 Done Done Sent false
19080 5459 Acked Flush 0 Done Done Sent false
19081 5460 Acked Read 7 Done Done Sent false
19082 5461 Acked Read 7 Done Done Sent false
19083 5462 Acked Read 7 Done Done Sent false
19084 5463 Acked Read 1 Done Done Sent false
19085 5464 Acked Flush 0 Done Done Sent false
19086 5465 Acked Read 2 Done Done Sent false
19087 5466 Acked Read 8 Done Done Sent false
19088 5467 Acked Read 8 Done Done Sent false
19089 5468 Acked Read 8 Done Done Sent false
19090 5469 Acked Flush 0 Done Done Sent false
19091 5470 Acked Read 10 Done Done Sent false
19092 5471 Acked Write 10 Done Done Sent false
19093 5472 Acked Read 5 Done Done Sent false
19094 5473 Acked Flush 0 Done Done Sent false
19095 5474 Acked Write 3 Done Done Sent false
19096 5475 Acked Read 3 Done Done Sent false
19097 5476 Acked Flush 0 Done Done Sent false
19098 5477 Acked Read 3 Done Done Sent false
19099 5478 Acked Write 7 Done Done Sent false
19100 5479 Acked Flush 0 Done Done Sent false
19101 5480 Acked Write 1 Done Done Sent false
19102 5481 Acked Write 1 Done Done Sent false
19103 5482 Acked Write 1 Done Done Sent false
19104 5483 Acked Read 3 Done Done Sent false
19105 5484 Acked Read 10 Done Done Sent false
19106 5485 Acked Read 5 Done Done Sent false
19107 5486 Acked Write 10 Done Done Sent false
19108 5487 Acked Write 1 Done Done Sent false
19109 5488 Acked Read 8 Done Done Sent false
19110 5489 Acked Write 9 Done Done Sent false
19111 5490 Acked Write 6 Done Done Sent false
19112 5491 Acked Write 2 Done Done Sent false
19113 5492 Acked Write 9 Done Done Sent false
19114 5493 Acked Write 10 Done Done Sent false
19115 5494 Acked Write 2 Done Done Sent false
19116 5495 Acked Read 3 Done Done Sent false
19117 5496 Acked Read 7 Done Done Sent false
19118 5497 Acked Write 7 Done Done Sent false
19119 5498 Acked Read 10 Done Done Sent false
19120 5499 Acked Read 5 Done Done Sent false
19121 5500 Acked Write 4 Done Done Sent false
19122 5501 Acked Write 6 Done Done Sent false
19123 5502 Acked Read 1 Done Done Sent false
19124 5503 Acked Write 9 Done Done Sent false
19125 5504 Acked Write 4 Done Done Sent false
19126 5505 Acked Read 6 Done Done Sent false
19127 5506 Acked Read 2 Done Done Sent false
19128 5507 Acked Write 2 Done Done Sent false
19129 5508 Acked Read 2 Done Done Sent false
19130 5509 Acked Read 6 Done Done Sent false
19131 5510 Acked Write 9 Done Done Sent false
19132 5511 Acked Write 3 Done Done Sent false
19133 5512 Acked Flush 0 Done Done Sent false
19134 5513 Acked Read 7 Done Done Sent false
19135 5514 Acked Write 2 Done Done Sent false
19136 5515 Acked Flush 0 Done Done Sent false
19137 5516 Acked Read 4 Done Done Sent false
19138 5517 Acked Read 10 Done Done Sent false
19139 5518 Acked Read 5 Done Done Sent false
19140 5519 Acked Write 7 Done Done Sent false
19141 5520 Acked Flush 0 Done Done Sent false
19142 5521 Acked Write 5 Done Done Sent false
19143 5522 Acked Read 7 Done Done Sent false
19144 5523 Acked Read 5 Done Done Sent false
19145 5524 Acked Flush 0 Done Done Sent false
19146 5525 Acked Read 5 Done Done Sent false
19147 5526 Acked Read 8 Done Done Sent false
19148 5527 Acked Write 9 Done Done Sent false
19149 5528 Acked Flush 0 Done Done Sent false
19150 STATES DS:0 DS:1 DS:2 TOTAL
19151 Sent 0 0 301 301
19152 Done 301 301 0 602
19153 Skipped 0 0 0 0
19154 Error 0 0 0 0
19155 Last Flush: Some(JobId(5528)) Some(JobId(5528)) Some(JobId(5227))
19156 Downstairs last five completed: 5227 5226 5225 5224 5223
19157 Upstairs last five completed: 5528 5527 5526 5525 5524
19158 CLIENT: Up:0 ds:301 act:2
191592025-02-07T20:11:34.355ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
191602025-02-07T20:11:35.357ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
191612025-02-07T20:11:36.359ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
19162 ----------------------------------------------------------------
19163 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19164 States: Active Active Offline
19165 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19166 5228 Acked Read 9 Done Done Sent false
19167 5229 Acked Read 4 Done Done Sent false
19168 5230 Acked Read 7 Done Done Sent false
19169 5231 Acked Write 7 Done Done Sent false
19170 5232 Acked Read 5 Done Done Sent false
19171 5233 Acked Read 3 Done Done Sent false
19172 5234 Acked Read 8 Done Done Sent false
19173 5235 Acked Write 9 Done Done Sent false
19174 5236 Acked Write 1 Done Done Sent false
19175 5237 Acked Read 2 Done Done Sent false
19176 5238 Acked Read 3 Done Done Sent false
19177 5239 Acked Read 6 Done Done Sent false
19178 5240 Acked Read 3 Done Done Sent false
19179 5241 Acked Read 10 Done Done Sent false
19180 5242 Acked Read 1 Done Done Sent false
19181 5243 Acked Write 6 Done Done Sent false
19182 5244 Acked Read 4 Done Done Sent false
19183 5245 Acked Flush 0 Done Done Sent false
19184 5246 Acked Write 4 Done Done Sent false
19185 5247 Acked Flush 0 Done Done Sent false
19186 5248 Acked Write 2 Done Done Sent false
19187 5249 Acked Read 5 Done Done Sent false
19188 5250 Acked Read 6 Done Done Sent false
19189 5251 Acked Read 8 Done Done Sent false
19190 5252 Acked Write 6 Done Done Sent false
19191 5253 Acked Read 9 Done Done Sent false
19192 5254 Acked Write 6 Done Done Sent false
19193 5255 Acked Write 3 Done Done Sent false
19194 5256 Acked Write 4 Done Done Sent false
19195 5257 Acked Write 8 Done Done Sent false
19196 5258 Acked Read 4 Done Done Sent false
19197 5259 Acked Flush 0 Done Done Sent false
19198 5260 Acked Flush 0 Done Done Sent false
19199 5261 Acked Read 9 Done Done Sent false
19200 5262 Acked Write 5 Done Done Sent false
19201 5263 Acked Write 6 Done Done Sent false
19202 5264 Acked Read 2 Done Done Sent false
19203 5265 Acked Read 2 Done Done Sent false
19204 5266 Acked Read 9 Done Done Sent false
19205 5267 Acked Read 2 Done Done Sent false
19206 5268 Acked Read 8 Done Done Sent false
19207 5269 Acked Read 3 Done Done Sent false
19208 5270 Acked Read 4 Done Done Sent false
19209 5271 Acked Read 8 Done Done Sent false
19210 5272 Acked Write 7 Done Done Sent false
19211 5273 Acked Read 3 Done Done Sent false
19212 5274 Acked Read 2 Done Done Sent false
19213 5275 Acked Read 4 Done Done Sent false
19214 5276 Acked Read 5 Done Done Sent false
19215 5277 Acked Read 6 Done Done Sent false
19216 5278 Acked Read 7 Done Done Sent false
19217 5279 Acked Read 7 Done Done Sent false
19218 5280 Acked Write 8 Done Done Sent false
19219 5281 Acked Write 10 Done Done Sent false
19220 5282 Acked Read 10 Done Done Sent false
19221 5283 Acked Write 2 Done Done Sent false
19222 5284 Acked Read 9 Done Done Sent false
19223 5285 Acked Read 6 Done Done Sent false
19224 5286 Acked Flush 0 Done Done Sent false
19225 5287 Acked Read 3 Done Done Sent false
19226 5288 Acked Write 5 Done Done Sent false
19227 5289 Acked Write 2 Done Done Sent false
19228 5290 Acked Read 1 Done Done Sent false
19229 5291 Acked Read 4 Done Done Sent false
19230 5292 Acked Write 4 Done Done Sent false
19231 5293 Acked Read 10 Done Done Sent false
19232 5294 Acked Write 3 Done Done Sent false
19233 5295 Acked Write 4 Done Done Sent false
19234 5296 Acked Flush 0 Done Done Sent false
19235 5297 Acked Flush 0 Done Done Sent false
19236 5298 Acked Read 10 Done Done Sent false
19237 5299 Acked Write 5 Done Done Sent false
19238 5300 Acked Write 2 Done Done Sent false
19239 5301 Acked Write 10 Done Done Sent false
19240 5302 Acked Read 9 Done Done Sent false
19241 5303 Acked Read 4 Done Done Sent false
19242 5304 Acked Read 3 Done Done Sent false
19243 5305 Acked Flush 0 Done Done Sent false
19244 5306 Acked Write 7 Done Done Sent false
19245 5307 Acked Read 9 Done Done Sent false
19246 5308 Acked Flush 0 Done Done Sent false
19247 5309 Acked Flush 0 Done Done Sent false
19248 5310 Acked Write 5 Done Done Sent false
19249 5311 Acked Write 7 Done Done Sent false
19250 5312 Acked Read 2 Done Done Sent false
19251 5313 Acked Read 10 Done Done Sent false
19252 5314 Acked Write 8 Done Done Sent false
19253 5315 Acked Read 3 Done Done Sent false
19254 5316 Acked Write 2 Done Done Sent false
19255 5317 Acked Write 5 Done Done Sent false
19256 5318 Acked Write 1 Done Done Sent false
19257 5319 Acked Write 5 Done Done Sent false
19258 5320 Acked Read 7 Done Done Sent false
19259 5321 Acked Read 1 Done Done Sent false
19260 5322 Acked Read 3 Done Done Sent false
19261 5323 Acked Write 10 Done Done Sent false
19262 5324 Acked Read 4 Done Done Sent false
19263 5325 Acked Write 4 Done Done Sent false
19264 5326 Acked Write 2 Done Done Sent false
19265 5327 Acked Read 1 Done Done Sent false
19266 5328 Acked Read 3 Done Done Sent false
19267 5329 Acked Write 10 Done Done Sent false
19268 5330 Acked Read 6 Done Done Sent false
19269 5331 Acked Read 6 Done Done Sent false
19270 5332 Acked Read 7 Done Done Sent false
19271 5333 Acked Write 3 Done Done Sent false
19272 5334 Acked Read 5 Done Done Sent false
19273 5335 Acked Read 3 Done Done Sent false
19274 5336 Acked Write 4 Done Done Sent false
19275 5337 Acked Read 6 Done Done Sent false
19276 5338 Acked Read 2 Done Done Sent false
19277 5339 Acked Read 10 Done Done Sent false
19278 5340 Acked Read 7 Done Done Sent false
19279 5341 Acked Read 4 Done Done Sent false
19280 5342 Acked Flush 0 Done Done Sent false
19281 5343 Acked Write 6 Done Done Sent false
19282 5344 Acked Read 4 Done Done Sent false
19283 5345 Acked Read 3 Done Done Sent false
19284 5346 Acked Read 2 Done Done Sent false
19285 5347 Acked Read 2 Done Done Sent false
19286 5348 Acked Flush 0 Done Done Sent false
19287 5349 Acked Read 10 Done Done Sent false
19288 5350 Acked Read 10 Done Done Sent false
19289 5351 Acked Read 3 Done Done Sent false
19290 5352 Acked Read 2 Done Done Sent false
19291 5353 Acked Flush 0 Done Done Sent false
19292 5354 Acked Read 3 Done Done Sent false
19293 5355 Acked Read 1 Done Done Sent false
19294 5356 Acked Flush 0 Done Done Sent false
19295 5357 Acked Write 1 Done Done Sent false
19296 5358 Acked Write 9 Done Done Sent false
19297 5359 Acked Flush 0 Done Done Sent false
19298 5360 Acked Read 1 Done Done Sent false
19299 5361 Acked Flush 0 Done Done Sent false
19300 5362 Acked Write 8 Done Done Sent false
19301 5363 Acked Read 9 Done Done Sent false
19302 5364 Acked Read 5 Done Done Sent false
19303 5365 Acked Write 5 Done Done Sent false
19304 5366 Acked Read 9 Done Done Sent false
19305 5367 Acked Read 4 Done Done Sent false
19306 5368 Acked Read 7 Done Done Sent false
19307 5369 Acked Write 5 Done Done Sent false
19308 5370 Acked Read 7 Done Done Sent false
19309 5371 Acked Read 6 Done Done Sent false
19310 5372 Acked Read 8 Done Done Sent false
19311 5373 Acked Write 6 Done Done Sent false
19312 5374 Acked Read 4 Done Done Sent false
19313 5375 Acked Write 6 Done Done Sent false
19314 5376 Acked Read 3 Done Done Sent false
19315 5377 Acked Write 4 Done Done Sent false
19316 5378 Acked Flush 0 Done Done Sent false
19317 5379 Acked Write 3 Done Done Sent false
19318 5380 Acked Write 3 Done Done Sent false
19319 5381 Acked Read 5 Done Done Sent false
19320 5382 Acked Write 6 Done Done Sent false
19321 5383 Acked Write 1 Done Done Sent false
19322 5384 Acked Flush 0 Done Done Sent false
19323 5385 Acked Write 2 Done Done Sent false
19324 5386 Acked Write 4 Done Done Sent false
19325 5387 Acked Read 7 Done Done Sent false
19326 5388 Acked Read 5 Done Done Sent false
19327 5389 Acked Read 7 Done Done Sent false
19328 5390 Acked Write 7 Done Done Sent false
19329 5391 Acked Write 10 Done Done Sent false
19330 5392 Acked Read 4 Done Done Sent false
19331 5393 Acked Read 2 Done Done Sent false
19332 5394 Acked Write 10 Done Done Sent false
19333 5395 Acked Write 2 Done Done Sent false
19334 5396 Acked Read 8 Done Done Sent false
19335 5397 Acked Read 2 Done Done Sent false
19336 5398 Acked Write 7 Done Done Sent false
19337 5399 Acked Write 2 Done Done Sent false
19338 5400 Acked Write 2 Done Done Sent false
19339 5401 Acked Read 4 Done Done Sent false
19340 5402 Acked Write 3 Done Done Sent false
19341 5403 Acked Write 3 Done Done Sent false
19342 5404 Acked Write 4 Done Done Sent false
19343 5405 Acked Read 5 Done Done Sent false
19344 5406 Acked Read 3 Done Done Sent false
19345 5407 Acked Write 2 Done Done Sent false
19346 5408 Acked Read 6 Done Done Sent false
19347 5409 Acked Read 4 Done Done Sent false
19348 5410 Acked Write 3 Done Done Sent false
19349 5411 Acked Read 8 Done Done Sent false
19350 5412 Acked Read 4 Done Done Sent false
19351 5413 Acked Flush 0 Done Done Sent false
19352 5414 Acked Write 5 Done Done Sent false
19353 5415 Acked Read 4 Done Done Sent false
19354 5416 Acked Write 4 Done Done Sent false
19355 5417 Acked Write 9 Done Done Sent false
19356 5418 Acked Flush 0 Done Done Sent false
19357 5419 Acked Read 8 Done Done Sent false
19358 5420 Acked Read 3 Done Done Sent false
19359 5421 Acked Flush 0 Done Done Sent false
19360 5422 Acked Write 7 Done Done Sent false
19361 5423 Acked Write 8 Done Done Sent false
19362 5424 Acked Write 7 Done Done Sent false
19363 5425 Acked Read 5 Done Done Sent false
19364 5426 Acked Read 1 Done Done Sent false
19365 5427 Acked Write 6 Done Done Sent false
19366 5428 Acked Flush 0 Done Done Sent false
19367 5429 Acked Read 8 Done Done Sent false
19368 5430 Acked Read 2 Done Done Sent false
19369 5431 Acked Write 2 Done Done Sent false
19370 5432 Acked Flush 0 Done Done Sent false
19371 5433 Acked Read 2 Done Done Sent false
19372 5434 Acked Write 7 Done Done Sent false
19373 5435 Acked Write 6 Done Done Sent false
19374 5436 Acked Read 8 Done Done Sent false
19375 5437 Acked Read 6 Done Done Sent false
19376 5438 Acked Read 6 Done Done Sent false
19377 5439 Acked Read 8 Done Done Sent false
19378 5440 Acked Read 6 Done Done Sent false
19379 5441 Acked Write 6 Done Done Sent false
19380 5442 Acked Read 4 Done Done Sent false
19381 5443 Acked Write 9 Done Done Sent false
19382 5444 Acked Write 5 Done Done Sent false
19383 5445 Acked Read 10 Done Done Sent false
19384 5446 Acked Write 6 Done Done Sent false
19385 5447 Acked Read 5 Done Done Sent false
19386 5448 Acked Read 5 Done Done Sent false
19387 5449 Acked Write 3 Done Done Sent false
19388 5450 Acked Read 5 Done Done Sent false
19389 5451 Acked Write 7 Done Done Sent false
19390 5452 Acked Write 7 Done Done Sent false
19391 5453 Acked Read 9 Done Done Sent false
19392 5454 Acked Read 7 Done Done Sent false
19393 5455 Acked Flush 0 Done Done Sent false
19394 5456 Acked Write 8 Done Done Sent false
19395 5457 Acked Read 6 Done Done Sent false
19396 5458 Acked Read 7 Done Done Sent false
19397 5459 Acked Flush 0 Done Done Sent false
19398 5460 Acked Read 7 Done Done Sent false
19399 5461 Acked Read 7 Done Done Sent false
19400 5462 Acked Read 7 Done Done Sent false
19401 5463 Acked Read 1 Done Done Sent false
19402 5464 Acked Flush 0 Done Done Sent false
19403 5465 Acked Read 2 Done Done Sent false
19404 5466 Acked Read 8 Done Done Sent false
19405 5467 Acked Read 8 Done Done Sent false
19406 5468 Acked Read 8 Done Done Sent false
19407 5469 Acked Flush 0 Done Done Sent false
19408 5470 Acked Read 10 Done Done Sent false
19409 5471 Acked Write 10 Done Done Sent false
19410 5472 Acked Read 5 Done Done Sent false
19411 5473 Acked Flush 0 Done Done Sent false
19412 5474 Acked Write 3 Done Done Sent false
19413 5475 Acked Read 3 Done Done Sent false
19414 5476 Acked Flush 0 Done Done Sent false
19415 5477 Acked Read 3 Done Done Sent false
19416 5478 Acked Write 7 Done Done Sent false
19417 5479 Acked Flush 0 Done Done Sent false
19418 5480 Acked Write 1 Done Done Sent false
19419 5481 Acked Write 1 Done Done Sent false
19420 5482 Acked Write 1 Done Done Sent false
19421 5483 Acked Read 3 Done Done Sent false
19422 5484 Acked Read 10 Done Done Sent false
19423 5485 Acked Read 5 Done Done Sent false
19424 5486 Acked Write 10 Done Done Sent false
19425 5487 Acked Write 1 Done Done Sent false
19426 5488 Acked Read 8 Done Done Sent false
19427 5489 Acked Write 9 Done Done Sent false
19428 5490 Acked Write 6 Done Done Sent false
19429 5491 Acked Write 2 Done Done Sent false
19430 5492 Acked Write 9 Done Done Sent false
19431 5493 Acked Write 10 Done Done Sent false
19432 5494 Acked Write 2 Done Done Sent false
19433 5495 Acked Read 3 Done Done Sent false
19434 5496 Acked Read 7 Done Done Sent false
19435 5497 Acked Write 7 Done Done Sent false
19436 5498 Acked Read 10 Done Done Sent false
19437 5499 Acked Read 5 Done Done Sent false
19438 5500 Acked Write 4 Done Done Sent false
19439 5501 Acked Write 6 Done Done Sent false
19440 5502 Acked Read 1 Done Done Sent false
19441 5503 Acked Write 9 Done Done Sent false
19442 5504 Acked Write 4 Done Done Sent false
19443 5505 Acked Read 6 Done Done Sent false
19444 5506 Acked Read 2 Done Done Sent false
19445 5507 Acked Write 2 Done Done Sent false
19446 5508 Acked Read 2 Done Done Sent false
19447 5509 Acked Read 6 Done Done Sent false
19448 5510 Acked Write 9 Done Done Sent false
19449 5511 Acked Write 3 Done Done Sent false
19450 5512 Acked Flush 0 Done Done Sent false
19451 5513 Acked Read 7 Done Done Sent false
19452 5514 Acked Write 2 Done Done Sent false
19453 5515 Acked Flush 0 Done Done Sent false
19454 5516 Acked Read 4 Done Done Sent false
19455 5517 Acked Read 10 Done Done Sent false
19456 5518 Acked Read 5 Done Done Sent false
19457 5519 Acked Write 7 Done Done Sent false
19458 5520 Acked Flush 0 Done Done Sent false
19459 5521 Acked Write 5 Done Done Sent false
19460 5522 Acked Read 7 Done Done Sent false
19461 5523 Acked Read 5 Done Done Sent false
19462 5524 Acked Flush 0 Done Done Sent false
19463 5525 Acked Read 5 Done Done Sent false
19464 5526 Acked Read 8 Done Done Sent false
19465 5527 Acked Write 9 Done Done Sent false
19466 5528 Acked Flush 0 Done Done Sent false
19467 STATES DS:0 DS:1 DS:2 TOTAL
19468 Sent 0 0 301 301
19469 Done 301 301 0 602
19470 Skipped 0 0 0 0
19471 Error 0 0 0 0
19472 Last Flush: Some(JobId(5528)) Some(JobId(5528)) Some(JobId(5227))
19473 Downstairs last five completed: 5227 5226 5225 5224 5223
19474 Upstairs last five completed: 5528 5527 5526 5525 5524
19475 CLIENT: Up:0 ds:301 act:2
194762025-02-07T20:11:37.771ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
194772025-02-07T20:11:38.362ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
194782025-02-07T20:11:39.365ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
194792025-02-07T20:11:40.366ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
194802025-02-07T20:11:41.018ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
194812025-02-07T20:11:41.018ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43554) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
194822025-02-07T20:11:41.019ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
194832025-02-07T20:11:41.020ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
194842025-02-07T20:11:41.020ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
194852025-02-07T20:11:41.020ZINFOcrucible: send last flush ID to this DS: Some(JobId(5227)) = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
194862025-02-07T20:11:41.020ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
194872025-02-07T20:11:41.020ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(5227)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
194882025-02-07T20:11:41.020ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(5227)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
19489 ----------------------------------------------------------------
19490 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19491 Downstairs last five completed: 5528 5527 5526 5525 5524
19492 Upstairs last five completed: 5528 5527 5526 5525 5524
19493 CLIENT: Up:0 ds:0 act:3
19494 Replay: All jobs finished
19495 Main task has work to do, go find it
19496 got dsc Stop(0)
19497 stop 0
19498 Feb 07 20:11:41.272 INFO request completed, latency_us: 379[8810] Got stop action so:false kr:false
19499 , response_code: 204, uri: /stop/cid/0, method: POST, req_id: 8168a193-e72e-4afa-aa33-e9b2d62d258c, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
195002025-02-07T20:11:41.273ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
195012025-02-07T20:11:41.273ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
19502 Feb 07 20:11:41.273 INFO request completed, latency_us: 175{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:11:41.273222191Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
19503 , response_code: 200, uri: /state/cid/0, method{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-07T20:11:41.273263448Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
19504 : GET, req_id: 326a8fc6-b24d-47d6-8e91-47b2a55716d0, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
19505 [8810] Exited with: ExitStatus(unix_wait_status(9))
19506 [8810][0] reports Exit
19507 Feb 07 20:11:45.275 INFO request completed, latency_us: 308, response_code: 200, uri: /state/cid/0, method: GET, req_id: c22121d6-860b-4633-8d76-3819425a9feb, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
19508 001/300 Read block 151 len 24576
19509 002/300 Write block 4579 len 36864 data: 2 2 3 3 3 3 3 3 3
19510 003/300 Write block 1247 len 40960 data: 4 4 4 3 4 3 3 3 2 2
19511 004/300 Write block 449 len 12288 data: 5 6 5
19512 005/300 Read block 4615 len 40960
19513 006/300 Read block 4211 len 36864
19514 007/300 Write block 4061 len 8192 data: 6 6
19515 008/300 Read block 3834 len 28672
19516 009/300 Write block 2974 len 20480 data: 4 4 4 5 6
19517 010/300 Write block 2523 len 12288 data: 4 4 4
19518 011/300 Read block 2958 len 12288
19519 012/300 Write block 1615 len 36864 data: 3 3 2 3 3 3 3 3 3
19520 013/300 Flush
19521 014/300 Write block 3215 len 24576 data: 4 4 3 4 4 5
19522 015/300 Write block 2189 len 4096 data: 4
19523 016/300 Read block 929 len 16384
19524 017/300 Flush
19525 018/300 Read block 3951 len 20480
19526 019/300 Read block 4167 len 28672
19527 020/300 Read block 2478 len 20480
19528 021/300 Read block 565 len 32768
19529 022/300 Read block 1724 len 8192
19530 023/300 Read block 389 len 4096
19531 024/300 Read block 2906 len 36864
19532 025/300 Flush
19533 026/300 Read block 2042 len 12288
19534 027/300 Flush
19535 028/300 Read block 2428 len 36864
19536 029/300 Flush
19537 030/300 Read block 1426 len 20480
19538 031/300 Write block 217 len 40960 data: 3 3 3 3 4 4 4 3 3 3
19539 032/300 Write block 3372 len 4096 data: 3
19540 033/300 Read block 1824 len 40960
19541 034/300 Flush
19542 035/300 Read block 2329 len 24576
19543 036/300 Read block 2188 len 36864
19544 037/300 Write block 3898 len 32768 data: 4 4 4 4 4 5 4 6
19545 038/300 Write block 2336 len 32768 data: 3 4 5 4 5 7 6 4
19546 039/300 Flush
19547 040/300 Read block 1322 len 28672
19548 041/300 Write block 1665 len 20480 data: 2 2 2 2 2
19549 042/300 Write block 4017 len 16384 data: 3 3 3 2
19550 043/300 Write block 583 len 16384 data: 5 4 5 5
19551 044/300 Read block 4612 len 28672
19552 045/300 Write block 2882 len 32768 data: 2 2 2 2 2 3 3 3
19553 046/300 Write block 2923 len 20480 data: 3 3 2 2 3
19554 047/300 Read block 4739 len 24576
19555 048/300 Write block 1951 len 28672 data: 4 4 4 4 4 4 2
19556 049/300 Read block 3000 len 40960
19557 050/300 Read block 3495 len 8192
19558 051/300 Read block 4382 len 16384
19559 052/300 Read block 189 len 12288
19560 053/300 Flush
19561 054/300 Flush
19562 055/300 Write block 514 len 8192 data: 3 2
19563 056/300 Flush
19564 057/300 Write block 2763 len 20480 data: 3 2 2 2 3
19565 058/300 Write block 329 len 16384 data: 3 3 4 5
19566 059/300 Write block 4675 len 40960 data: 4 3 3 4 4 4 4 4 4 5
19567 060/300 Read block 1579 len 12288
19568 061/300 Read block 4128 len 12288
19569 062/300 Flush
19570 063/300 Flush
19571 064/300 Read block 2937 len 32768
19572 065/300 Read block 1589 len 8192
19573 066/300 Write block 2904 len 36864 data: 3 3 4 4 5 5 5 4 4
19574 067/300 Write block 2092 len 20480 data: 4 3 3 3 3
19575 068/300 Read block 2158 len 8192
19576 069/300 Read block 3842 len 24576
19577 070/300 Read block 1639 len 20480
19578 071/300 Write block 3104 len 8192 data: 2 4
19579 072/300 Read block 3463 len 4096
19580 073/300 Write block 27 len 16384 data: 3 3 4 3
19581 074/300 Read block 4555 len 36864
19582 075/300 Write block 1615 len 36864 data: 4 4 3 4 4 4 4 4 4
19583 076/300 Write block 757 len 12288 data: 4 5 6
19584 077/300 Flush
19585 078/300 Read block 3010 len 36864
19586 079/300 Write block 190 len 20480 data: 3 3 3 2 2
19587 080/300 Read block 1420 len 12288
19588 081/300 Read block 559 len 24576
19589 082/300 Write block 830 len 16384 data: 7 7 5 4
19590 083/300 Write block 4792 len 16384 data: 4 5 5 5
19591 084/300 Read block 2014 len 12288
19592 085/300 Read block 2272 len 8192
19593 086/300 Flush
19594 087/300 Write block 2365 len 32768 data: 6 7 6 7 4 4 4 4
19595 088/300 Write block 862 len 12288 data: 3 3 2
19596 089/300 Write block 572 len 32768 data: 3 3 4 4 4 4 4 4
19597 090/300 Read block 571 len 40960
19598 091/300 Read block 1599 len 36864
19599 092/300 Flush
19600 093/300 Read block 750 len 40960
19601 094/300 Read block 2374 len 8192
19602 095/300 Read block 2890 len 16384
19603 096/300 Write block 2181 len 12288 data: 4 2 2
19604 097/300 Write block 4920 len 8192 data: 4 3
19605 098/300 Read block 2881 len 20480
19606 099/300 Read block 1587 len 20480
19607 100/300 Write block 2356 len 40960 data: 4 4 5 5 5 5 6 8 8 7
19608 101/300 Write block 1090 len 4096 data: 3
19609 102/300 Read block 3674 len 4096
19610 103/300 Write block 4597 len 32768 data: 4 4 3 3 3 4 4 3
19611 104/300 Read block 3085 len 20480
19612 105/300 Read block 2154 len 20480
19613 106/300 Read block 2619 len 16384
19614 107/300 Read block 2293 len 12288
19615 108/300 Read block 4587 len 20480
19616 109/300 Write block 2090 len 12288 data: 4 5 5
19617 110/300 Read block 1391 len 12288
19618 111/300 Write block 386 len 36864 data: 3 3 3 3 3 3 2 2 2
19619 112/300 Read block 3780 len 28672
19620 113/300 Write block 3526 len 20480 data: 2 2 3 3 3
19621 114/300 Read block 3337 len 12288
19622 115/300 Write block 4332 len 16384 data: 6 4 4 4
19623 116/300 Write block 4715 len 36864 data: 2 3 3 3 3 5 5 5 4
19624 117/300 Write block 2581 len 4096 data: 4
19625 118/300 Read block 4848 len 40960
19626 119/300 Read block 1654 len 4096
19627 120/300 Read block 4356 len 16384
19628 121/300 Read block 4421 len 12288
19629 122/300 Read block 4594 len 12288
19630 123/300 Write block 1863 len 4096 data: 5
19631 124/300 Flush
19632 125/300 Write block 593 len 12288 data: 3 3 3
19633 126/300 Read block 923 len 12288
19634 127/300 Read block 4116 len 28672
19635 128/300 Read block 3388 len 36864
19636 129/300 Write block 1397 len 28672 data: 4 4 3 4 4 3 2
19637 130/300 Write block 448 len 20480 data: 6 6 7 6 5
19638 131/300 Write block 4637 len 28672 data: 4 4 4 3 2 2 3
19639 132/300 Read block 1910 len 20480
19640 133/300 Flush
19641 134/300 Flush
19642 135/300 Write block 563 len 40960 data: 5 5 3 3 3 3 2 3 3 4
19643 136/300 Read block 3525 len 20480
19644 137/300 Write block 2562 len 12288 data: 3 3 4
19645 138/300 Write block 3682 len 32768 data: 7 6 7 5 6 6 6 5
19646 139/300 Read block 3355 len 36864
19647 140/300 Flush
19648 141/300 Write block 136 len 12288 data: 2 2 2
19649 142/300 Write block 920 len 4096 data: 3
19650 143/300 Read block 3770 len 36864
19651 144/300 Flush
19652 145/300 Read block 3072 len 40960
19653 146/300 Write block 1186 len 24576 data: 3 5 4 4 4 4
19654 147/300 Write block 2751 len 36864 data: 3 2 4 5 6 6 4 4 4
19655 148/300 Read block 4599 len 36864
19656 149/300 Read block 624 len 28672
19657 150/300 Read block 1095 len 12288
19658 151/300 Write block 1825 len 40960 data: 4 4 4 5 5 5 6 6 6 8
19659 152/300 Read block 390 len 4096
19660 153/300 Read block 3879 len 20480
19661 154/300 Write block 4158 len 12288 data: 4 4 4
19662 155/300 Flush
19663 156/300 Write block 1772 len 12288 data: 3 3 2
19664 157/300 Read block 1402 len 36864
19665 158/300 Write block 364 len 36864 data: 4 4 5 4 4 5 4 4 3
19666 159/300 Read block 1299 len 24576
19667 160/300 Read block 1971 len 4096
19668 161/300 Read block 682 len 12288
19669 162/300 Write block 4207 len 36864 data: 4 4 3 3 3 3 3 3 4
19670 163/300 Write block 2075 len 8192 data: 5 4
19671 164/300 Read block 920 len 12288
19672 165/300 Read block 422 len 28672
19673 166/300 Read block 2094 len 40960
19674 167/300 Write block 1340 len 40960 data: 3 3 3 4 5 6 5 4 3 4
19675 168/300 Read block 2873 len 16384
19676 169/300 Read block 2454 len 40960
19677 170/300 Write block 1972 len 4096 data: 4
19678 171/300 Write block 3362 len 16384 data: 5 3 3 5
19679 172/300 Read block 2397 len 4096
19680 173/300 Read block 755 len 12288
19681 174/300 Read block 2534 len 32768
19682 175/300 Read block 4529 len 16384
19683 176/300 Write block 2593 len 24576 data: 6 6 7 7 8 7
19684 177/300 Read block 548 len 16384
19685 178/300 Read block 3099 len 24576
19686 179/300 Read block 4192 len 32768
19687 180/300 Read block 4795 len 16384
19688 181/300 Read block 2741 len 20480
19689 182/300 Read block 320 len 28672
19690 183/300 Write block 2938 len 16384 data: 3 4 3 2
19691 184/300 Read block 1301 len 8192
19692 185/300 Write block 2324 len 32768 data: 3 3 3 3 3 3 3 3
19693 186/300 Write block 648 len 20480 data: 5 5 4 4 4
19694 187/300 Flush
19695 188/300 Read block 763 len 40960
19696 189/300 Read block 1230 len 28672
19697 190/300 Read block 4294 len 40960
19698 191/300 Read block 3113 len 8192
19699 192/300 Read block 1069 len 20480
19700 193/300 Write block 1133 len 32768 data: 4 4 4 4 3 3 3 3
19701 194/300 Write block 3416 len 36864 data: 5 4 3 3 3 3 3 3 5
19702 195/300 Read block 2506 len 40960
19703 196/300 Write block 2737 len 4096 data: 4
19704 197/300 Read block 2102 len 24576
19705 198/300 Read block 3552 len 16384
19706 199/300 Write block 3844 len 8192 data: 5 5
19707 200/300 Write block 1616 len 8192 data: 5 4
19708 201/300 Read block 577 len 12288
19709 202/300 Write block 1505 len 8192 data: 5 6
19710 203/300 Write block 2739 len 16384 data: 4 4 3 4
19711 204/300 Write block 290 len 12288 data: 4 3 3
19712 205/300 Read block 1851 len 8192
19713 206/300 Write block 1120 len 4096 data: 4
19714 207/300 Write block 1352 len 16384 data: 4 4 3 3
19715 208/300 Write block 2197 len 20480 data: 4 6 6 6 6
19716 209/300 Read block 1251 len 20480
19717 210/300 Write block 3231 len 32768 data: 4 4 4 4 4 2 2 3
19718 211/300 Write block 4773 len 20480 data: 4 3 3 3 3
19719 212/300 Write block 2181 len 16384 data: 5 3 3 2
19720 213/300 Write block 1673 len 24576 data: 3 4 4 4 4 5
19721 214/300 Write block 2308 len 32768 data: 5 5 5 6 6 6 8 6
19722 215/300 Flush
19723 216/300 Read block 240 len 16384
19724 217/300 Read block 1790 len 28672
19725 218/300 Write block 1911 len 16384 data: 5 5 6 6
19726 219/300 Read block 577 len 40960
19727 220/300 Write block 4489 len 16384 data: 4 5 5 5
19728 221/300 Read block 2093 len 12288
19729 222/300 Read block 2663 len 8192
19730 223/300 Write block 4972 len 20480 data: 2 3 4 4 3
19731 224/300 Write block 1738 len 24576 data: 2 3 4 3 3 3
19732 225/300 Write block 2382 len 12288 data: 6 6 6
19733 226/300 Write block 3600 len 36864 data: 5 6 3 3 3 3 3 5 5
19734 227/300 Flush
19735 228/300 Read block 381 len 36864
19736 229/300 Write block 3497 len 28672 data: 4 5 4 3 3 3 2
19737 230/300 Read block 3873 len 12288
19738 231/300 Write block 232 len 40960 data: 3 4 4 4 5 3 3 4 4 3
19739 232/300 Write block 557 len 16384 data: 4 4 4 5
19740 233/300 Write block 2120 len 8192 data: 4 5
19741 234/300 Flush
19742 235/300 Write block 669 len 28672 data: 2 3 3 3 3 3 4
19743 236/300 Read block 223 len 24576
19744 237/300 Write block 612 len 28672 data: 6 6 6 4 4 5 6
19745 238/300 Read block 1587 len 8192
19746 239/300 Read block 1418 len 32768
19747 240/300 Write block 4939 len 40960 data: 3 3 3 4 5 6 6 5 4 4
19748 241/300 Write block 4326 len 28672 data: 6 4 5 3 5 6 7
19749 242/300 Flush
19750 243/300 Write block 124 len 16384 data: 3 3 3 4
19751 244/300 Read block 4414 len 32768
19752 245/300 Flush
19753 246/300 Read block 2888 len 40960
19754 247/300 Write block 1652 len 24576 data: 7 7 4 5 5 6
19755 248/300 Read block 4860 len 24576
19756 249/300 Read block 1973 len 32768
19757 250/300 Write block 1757 len 12288 data: 4 4 4
19758 251/300 Write block 768 len 36864 data: 4 4 6 5 4 4 4 4 4
19759 252/300 Flush
19760 253/300 Write block 2008 len 4096 data: 5
19761 254/300 Read block 3098 len 24576
19762 255/300 Read block 753 len 36864
19763 256/300 Write block 4565 len 16384 data: 4 4 4 4
19764 257/300 Read block 1190 len 24576
19765 258/300 Flush
19766 259/300 Write block 679 len 24576 data: 4 4 4 3 5 5
19767 260/300 Write block 4060 len 8192 data: 5 7
19768 261/300 Write block 800 len 20480 data: 3 3 3 3 3
19769 262/300 Write block 2698 len 8192 data: 3 3
19770 263/300 Read block 4977 len 16384
19771 264/300 Write block 620 len 24576 data: 4 4 5 5 4 3
19772 265/300 Write block 2277 len 40960 data: 5 5 5 3 3 3 3 3 3 3
19773 266/300 Read block 2589 len 28672
19774 267/300 Write block 198 len 8192 data: 5 5
19775 268/300 Read block 965 len 40960
19776 269/300 Write block 4017 len 16384 data: 4 4 4 3
19777 270/300 Write block 1506 len 20480 data: 7 6 7 6 6
19778 271/300 Write block 2989 len 12288 data: 3 4 4
19779 272/300 Read block 890 len 28672
19780 273/300 Read block 419 len 28672
19781 274/300 Write block 3710 len 40960 data: 5 3 3 3 3 3 4 4 4 5
19782 275/300 Read block 4406 len 20480
19783 276/300 Write block 3837 len 20480 data: 4 4 3 4 3
19784 277/300 Write block 2529 len 8192 data: 3 3
19785 278/300 Read block 3322 len 36864
19786 279/300 Write block 1422 len 20480 data: 4 3 3 3 2
19787 280/300 Read block 4349 len 28672
19788 281/300 Read block 6 len 36864
19789 282/300 Flush
19790 283/300 Read block 3772 len 40960
19791 284/300 Write block 3995 len 16384 data: 6 6 6 6
19792 285/300 Read block 1005 len 24576
19793 286/300 Read block 4 len 12288
19794 287/300 Read block 2911 len 20480
19795 288/300 Write block 2981 len 4096 data: 6
19796 289/300 Read block 2820 len 16384
19797 290/300 Flush
19798 291/300 Flush
19799 292/300 Write block 216 len 28672 data: 4 4 4 4 4 5 5
19800 293/300 Flush
19801 294/300 Write block 623 len 32768 data: 6 5 4 3 5 5 4 5
19802 295/300 Flush
19803 296/300 Read block 2100 len 12288
19804 297/300 Write block 3433 len 20480 data: 6 5 5 5 5
19805 298/300 Read block 593 len 8192
19806 299/300 Write block 385 len 24576 data: 4 4 4 4 4 4
19807 300/300 Write block 1987 len 20480 data: 5 5 4 3 4
19808 301/300 Read block 4205 len 16384
19809 Main task has work to do, go find it
19810 got dsc Start(0)
19811 start 0
19812 Feb 07 20:11:45.515 INFO request completed, latency_us: 247, response_code: 204, uri: /start/cid/0, method: POST, req_id: 6e1a4f5b-fded-4848-a864-8964e081db73, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
19813 Replay: started 0, returned:Ok(())
19814 ----------------------------------------------------------------
19815 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19816 States: Offline Active Active
19817 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19818 5529 Acked Read 6 Sent Done Done false
19819 5530 Acked Write 9 Sent Done Done false
19820 5531 Acked Write 10 Sent Done Done false
19821 5532 Acked Write 3 Sent Done Done false
19822 5533 Acked Read 10 Sent Done Done false
19823 5534 Acked Read 9 Sent Done Done false
19824 5535 Acked Write 2 Sent Done Done false
19825 5536 Acked Read 7 Sent Done Done false
19826 5537 Acked Write 5 Sent Done Done false
19827 5538 Acked Write 3 Sent Done Done false
19828 5539 Acked Read 3 Sent Done Done false
19829 5540 Acked Write 9 Sent Done Done false
19830 5541 Acked Flush 0 Sent Done Done false
19831 5542 Acked Write 6 Sent Done Done false
19832 5543 Acked Write 1 Sent Done Done false
19833 5544 Acked Read 4 Sent Done Done false
19834 5545 Acked Flush 0 Sent Done Done false
19835 5546 Acked Read 5 Sent Done Done false
19836 5547 Acked Read 7 Sent Done Done false
19837 5548 Acked Read 5 Sent Done Done false
19838 5549 Acked Read 8 Sent Done Done false
19839 5550 Acked Read 2 Sent Done Done false
19840 5551 Acked Read 1 Sent Done Done false
19841 5552 Acked Read 9 Sent Done Done false
19842 5553 Acked Flush 0 Sent Done Done false
19843 5554 Acked Read 3 Sent Done Done false
19844 5555 Acked Flush 0 Sent Done Done false
19845 5556 Acked Read 9 Sent Done Done false
19846 5557 Acked Flush 0 Sent Done Done false
19847 5558 Acked Read 5 Sent Done Done false
19848 5559 Acked Write 10 Sent Done Done false
19849 5560 Acked Write 1 Sent Done Done false
19850 5561 Acked Read 10 Sent Done Done false
19851 5562 Acked Flush 0 Sent Done Done false
19852 5563 Acked Read 6 Sent Done Done false
19853 5564 Acked Read 9 Sent Done Done false
19854 5565 Acked Write 8 Sent Done Done false
19855 5566 Acked Write 8 Sent Done Done false
19856 5567 Acked Flush 0 Sent Done Done false
19857 5568 Acked Read 7 Sent Done Done false
19858 5569 Acked Write 5 Sent Done Done false
19859 5570 Acked Write 4 Sent Done Done false
19860 5571 Acked Write 4 Sent Done Done false
19861 5572 Acked Read 7 Sent Done Done false
19862 5573 Acked Write 8 Sent Done Done false
19863 5574 Acked Write 5 Sent Done Done false
19864 5575 Acked Read 6 Sent Done Done false
19865 5576 Acked Write 7 Sent Done Done false
19866 5577 Acked Read 10 Sent Done Done false
19867 5578 Acked Read 2 Sent Done Done false
19868 5579 Acked Read 4 Sent Done Done false
19869 5580 Acked Read 3 Sent Done Done false
19870 5581 Acked Flush 0 Sent Done Done false
19871 5582 Acked Flush 0 Sent Done Done false
19872 5583 Acked Write 2 Sent Done Done false
19873 5584 Acked Flush 0 Sent Done Done false
19874 5585 Acked Write 5 Sent Done Done false
19875 5586 Acked Write 4 Sent Done Done false
19876 5587 Acked Write 10 Sent Done Done false
19877 5588 Acked Read 3 Sent Done Done false
19878 5589 Acked Read 3 Sent Done Done false
19879 5590 Acked Flush 0 Sent Done Done false
19880 5591 Acked Flush 0 Sent Done Done false
19881 5592 Acked Read 8 Sent Done Done false
19882 5593 Acked Read 2 Sent Done Done false
19883 5594 Acked Write 9 Sent Done Done false
19884 5595 Acked Write 5 Sent Done Done false
19885 5596 Acked Read 2 Sent Done Done false
19886 5597 Acked Read 6 Sent Done Done false
19887 5598 Acked Read 5 Sent Done Done false
19888 5599 Acked Write 2 Sent Done Done false
19889 5600 Acked Read 1 Sent Done Done false
19890 5601 Acked Write 4 Sent Done Done false
19891 5602 Acked Read 9 Sent Done Done false
19892 5603 Acked Write 9 Sent Done Done false
19893 5604 Acked Write 3 Sent Done Done false
19894 5605 Acked Flush 0 Sent Done Done false
19895 5606 Acked Read 9 Sent Done Done false
19896 5607 Acked Write 5 Sent Done Done false
19897 5608 Acked Read 3 Sent Done Done false
19898 5609 Acked Read 6 Sent Done Done false
19899 5610 Acked Write 4 Sent Done Done false
19900 5611 Acked Write 4 Sent Done Done false
19901 5612 Acked Read 3 Sent Done Done false
19902 5613 Acked Read 2 Sent Done Done false
19903 5614 Acked Flush 0 Sent Done Done false
19904 5615 Acked Write 8 Sent Done Done false
19905 5616 Acked Write 3 Sent Done Done false
19906 5617 Acked Write 8 Sent Done Done false
19907 5618 Acked Read 10 Sent Done Done false
19908 5619 Acked Read 9 Sent Done Done false
19909 5620 Acked Flush 0 Sent Done Done false
19910 5621 Acked Read 10 Sent Done Done false
19911 5622 Acked Read 2 Sent Done Done false
19912 5623 Acked Read 4 Sent Done Done false
19913 5624 Acked Write 3 Sent Done Done false
19914 5625 Acked Write 2 Sent Done Done false
19915 5626 Acked Read 5 Sent Done Done false
19916 5627 Acked Read 5 Sent Done Done false
19917 5628 Acked Write 10 Sent Done Done false
19918 5629 Acked Write 1 Sent Done Done false
19919 5630 Acked Read 1 Sent Done Done false
19920 5631 Acked Write 8 Sent Done Done false
19921 5632 Acked Read 5 Sent Done Done false
19922 5633 Acked Read 5 Sent Done Done false
19923 5634 Acked Read 4 Sent Done Done false
19924 5635 Acked Read 3 Sent Done Done false
19925 5636 Acked Read 5 Sent Done Done false
19926 5637 Acked Write 3 Sent Done Done false
19927 5638 Acked Read 3 Sent Done Done false
19928 5639 Acked Write 9 Sent Done Done false
19929 5640 Acked Read 7 Sent Done Done false
19930 5641 Acked Write 5 Sent Done Done false
19931 5642 Acked Read 3 Sent Done Done false
19932 5643 Acked Write 4 Sent Done Done false
19933 5644 Acked Write 9 Sent Done Done false
19934 5645 Acked Write 1 Sent Done Done false
19935 5646 Acked Read 10 Sent Done Done false
19936 5647 Acked Read 1 Sent Done Done false
19937 5648 Acked Read 4 Sent Done Done false
19938 5649 Acked Read 3 Sent Done Done false
19939 5650 Acked Read 3 Sent Done Done false
19940 5651 Acked Write 1 Sent Done Done false
19941 5652 Acked Flush 0 Sent Done Done false
19942 5653 Acked Write 3 Sent Done Done false
19943 5654 Acked Read 3 Sent Done Done false
19944 5655 Acked Read 7 Sent Done Done false
19945 5656 Acked Read 9 Sent Done Done false
19946 5657 Acked Write 7 Sent Done Done false
19947 5658 Acked Write 5 Sent Done Done false
19948 5659 Acked Write 7 Sent Done Done false
19949 5660 Acked Read 5 Sent Done Done false
19950 5661 Acked Flush 0 Sent Done Done false
19951 5662 Acked Flush 0 Sent Done Done false
19952 5663 Acked Write 10 Sent Done Done false
19953 5664 Acked Read 5 Sent Done Done false
19954 5665 Acked Write 3 Sent Done Done false
19955 5666 Acked Write 8 Sent Done Done false
19956 5667 Acked Read 9 Sent Done Done false
19957 5668 Acked Flush 0 Sent Done Done false
19958 5669 Acked Write 3 Sent Done Done false
19959 5670 Acked Write 1 Sent Done Done false
19960 5671 Acked Read 9 Sent Done Done false
19961 5672 Acked Flush 0 Sent Done Done false
19962 5673 Acked Read 10 Sent Done Done false
19963 5674 Acked Write 6 Sent Done Done false
19964 5675 Acked Write 9 Sent Done Done false
19965 5676 Acked Read 9 Sent Done Done false
19966 5677 Acked Read 7 Sent Done Done false
19967 5678 Acked Read 3 Sent Done Done false
19968 5679 Acked Write 10 Sent Done Done false
19969 5680 Acked Read 1 Sent Done Done false
19970 5681 Acked Read 5 Sent Done Done false
19971 5682 Acked Write 3 Sent Done Done false
19972 5683 Acked Flush 0 Sent Done Done false
19973 5684 Acked Write 3 Sent Done Done false
19974 5685 Acked Read 9 Sent Done Done false
19975 5686 Acked Write 9 Sent Done Done false
19976 5687 Acked Read 6 Sent Done Done false
19977 5688 Acked Read 1 Sent Done Done false
19978 5689 Acked Read 3 Sent Done Done false
19979 5690 Acked Write 9 Sent Done Done false
19980 5691 Acked Write 2 Sent Done Done false
19981 5692 Acked Read 3 Sent Done Done false
19982 5693 Acked Read 7 Sent Done Done false
19983 5694 Acked Read 10 Sent Done Done false
19984 5695 Acked Write 10 Sent Done Done false
19985 5696 Acked Read 4 Sent Done Done false
19986 5697 Acked Read 10 Sent Done Done false
19987 5698 Acked Write 1 Sent Done Done false
19988 5699 Acked Write 4 Sent Done Done false
19989 5700 Acked Read 1 Sent Done Done false
19990 5701 Acked Read 3 Sent Done Done false
19991 5702 Acked Read 8 Sent Done Done false
19992 5703 Acked Read 4 Sent Done Done false
19993 5704 Acked Write 6 Sent Done Done false
19994 5705 Acked Read 4 Sent Done Done false
19995 5706 Acked Read 6 Sent Done Done false
19996 5707 Acked Read 8 Sent Done Done false
19997 5708 Acked Read 4 Sent Done Done false
19998 5709 Acked Read 5 Sent Done Done false
19999 5710 Acked Read 7 Sent Done Done false
20000 5711 Acked Write 4 Sent Done Done false
20001 5712 Acked Read 2 Sent Done Done false
20002 5713 Acked Write 8 Sent Done Done false
20003 5714 Acked Write 5 Sent Done Done false
20004 5715 Acked Flush 0 Sent Done Done false
20005 5716 Acked Read 10 Sent Done Done false
20006 5717 Acked Read 7 Sent Done Done false
20007 5718 Acked Read 10 Sent Done Done false
20008 5719 Acked Read 2 Sent Done Done false
20009 5720 Acked Read 5 Sent Done Done false
20010 5721 Acked Write 8 Sent Done Done false
20011 5722 Acked Write 9 Sent Done Done false
20012 5723 Acked Read 10 Sent Done Done false
20013 5724 Acked Write 1 Sent Done Done false
20014 5725 Acked Read 6 Sent Done Done false
20015 5726 Acked Read 4 Sent Done Done false
20016 5727 Acked Write 2 Sent Done Done false
20017 5728 Acked Write 2 Sent Done Done false
20018 5729 Acked Read 3 Sent Done Done false
20019 5730 Acked Write 2 Sent Done Done false
20020 5731 Acked Write 4 Sent Done Done false
20021 5732 Acked Write 3 Sent Done Done false
20022 5733 Acked Read 2 Sent Done Done false
20023 5734 Acked Write 1 Sent Done Done false
20024 5735 Acked Write 4 Sent Done Done false
20025 5736 Acked Write 5 Sent Done Done false
20026 5737 Acked Read 5 Sent Done Done false
20027 5738 Acked Write 8 Sent Done Done false
20028 5739 Acked Write 5 Sent Done Done false
20029 5740 Acked Write 4 Sent Done Done false
20030 5741 Acked Write 6 Sent Done Done false
20031 5742 Acked Write 8 Sent Done Done false
20032 5743 Acked Flush 0 Sent Done Done false
20033 5744 Acked Read 4 Sent Done Done false
20034 5745 Acked Read 7 Sent Done Done false
20035 5746 Acked Write 4 Sent Done Done false
20036 5747 Acked Read 10 Sent Done Done false
20037 5748 Acked Write 4 Sent Done Done false
20038 5749 Acked Read 3 Sent Done Done false
20039 5750 Acked Read 2 Sent Done Done false
20040 5751 Acked Write 5 Sent Done Done false
20041 5752 Acked Write 6 Sent Done Done false
20042 5753 Acked Write 3 Sent Done Done false
20043 5754 Acked Write 9 Sent Done Done false
20044 5755 Acked Flush 0 Sent Done Done false
20045 5756 Acked Read 9 Sent Done Done false
20046 5757 Acked Write 7 Sent Done Done false
20047 5758 Acked Read 3 Sent Done Done false
20048 5759 Acked Write 10 Sent Done Done false
20049 5760 Acked Write 4 Sent Done Done false
20050 5761 Acked Write 2 Sent Done Done false
20051 5762 Acked Flush 0 Sent Done Done false
20052 5763 Acked Write 7 Sent Done Done false
20053 5764 Acked Read 6 Sent Done Done false
20054 5765 Acked Write 7 Sent Done Done false
20055 5766 Acked Read 2 Sent Done Done false
20056 5767 Acked Read 8 Sent Done Done false
20057 5768 Acked Write 10 Sent Done Done false
20058 5769 Acked Write 7 Sent Done Done false
20059 5770 Acked Flush 0 Sent Done Done false
20060 5771 Acked Write 4 Sent Done Done false
20061 5772 Acked Read 8 Sent Done Done false
20062 5773 Acked Flush 0 Sent Done Done false
20063 5774 Acked Read 10 Sent Done Done false
20064 5775 Acked Write 6 Sent Done Done false
20065 5776 Acked Read 6 Sent Done Done false
20066 5777 Acked Read 8 Sent Done Done false
20067 5778 Acked Write 3 Sent Done Done false
20068 5779 Acked Write 9 Sent Done Done false
20069 5780 Acked Flush 0 Sent Done Done false
20070 5781 Acked Write 1 Sent Done Done false
20071 5782 Acked Read 6 Sent Done Done false
20072 5783 Acked Read 9 Sent Done Done false
20073 5784 Acked Write 4 Sent Done Done false
20074 5785 Acked Read 6 Sent Done Done false
20075 5786 Acked Flush 0 Sent Done Done false
20076 5787 Acked Write 6 Sent Done Done false
20077 5788 Acked Write 2 Sent Done Done false
20078 5789 Acked Write 5 Sent Done Done false
20079 5790 Acked Write 2 Sent Done Done false
20080 5791 Acked Read 4 Sent Done Done false
20081 5792 Acked Write 6 Sent Done Done false
20082 5793 Acked Write 10 Sent Done Done false
20083 5794 Acked Read 7 Sent Done Done false
20084 5795 Acked Write 2 Sent Done Done false
20085 5796 Acked Read 10 Sent Done Done false
20086 5797 Acked Write 4 Sent Done Done false
20087 5798 Acked Write 5 Sent Done Done false
20088 5799 Acked Write 3 Sent Done Done false
20089 5800 Acked Read 7 Sent Done Done false
20090 5801 Acked Read 7 Sent Done Done false
20091 5802 Acked Write 10 Sent Done Done false
20092 5803 Acked Read 5 Sent Done Done false
20093 5804 Acked Write 5 Sent Done Done false
20094 5805 Acked Write 2 Sent Done Done false
20095 5806 Acked Read 9 Sent Done Done false
20096 5807 Acked Write 5 Sent Done Done false
20097 5808 Acked Read 7 Sent Done Done false
20098 5809 Acked Read 9 Sent Done Done false
20099 5810 Acked Flush 0 Sent Done Done false
20100 5811 Acked Read 10 Sent Done Done false
20101 5812 Acked Write 4 Sent Done Done false
20102 5813 Acked Read 6 Sent Done Done false
20103 5814 Acked Read 3 Sent Done Done false
20104 5815 Acked Read 5 Sent Done Done false
20105 5816 Acked Write 1 Sent Done Done false
20106 5817 Acked Read 4 Sent Done Done false
20107 5818 Acked Flush 0 Sent Done Done false
20108 5819 Acked Flush 0 Sent Done Done false
20109 5820 Acked Write 7 Sent Done Done false
20110 5821 Acked Flush 0 Sent Done Done false
20111 5822 Acked Write 8 Sent Done Done false
20112 5823 Acked Flush 0 Sent Done Done false
20113 5824 Acked Read 3 Sent Done Done false
20114 5825 Acked Write 5 Sent Done Done false
20115 5826 Acked Read 2 Sent Done Done false
20116 5827 Acked Write 6 Sent Done Done false
20117 5828 Acked Write 5 Sent Done Done false
20118 5829 Acked Read 4 Sent Done Done false
20119 STATES DS:0 DS:1 DS:2 TOTAL
20120 Sent 301 0 0 301
20121 Done 0 301 301 602
20122 Skipped 0 0 0 0
20123 Error 0 0 0 0
20124 Last Flush: Some(JobId(5528)) Some(JobId(5823)) Some(JobId(5823))
20125 Downstairs last five completed: 5528 5527 5526 5525 5524
20126 Upstairs last five completed: 5829 5828 5827 5826 5825
20127 CLIENT: Up:0 ds:301 act:2
201282025-02-07T20:11:46.521ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
201292025-02-07T20:11:47.378ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
201302025-02-07T20:11:48.379ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
20131 [8810] got start action
20132 [8810] I am going to restart
20133 Starting downstairs at port 8810
20134 Make output file at "/tmp/dsc/downstairs-8810.txt"
20135 [8810][0] reports Starting
20136 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1225)
20137 [8810][0] reports Running
201382025-02-07T20:11:49.380ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
20139 ----------------------------------------------------------------
20140 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20141 States: Offline Active Active
20142 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20143 5529 Acked Read 6 Sent Done Done false
20144 5530 Acked Write 9 Sent Done Done false
20145 5531 Acked Write 10 Sent Done Done false
20146 5532 Acked Write 3 Sent Done Done false
20147 5533 Acked Read 10 Sent Done Done false
20148 5534 Acked Read 9 Sent Done Done false
20149 5535 Acked Write 2 Sent Done Done false
20150 5536 Acked Read 7 Sent Done Done false
20151 5537 Acked Write 5 Sent Done Done false
20152 5538 Acked Write 3 Sent Done Done false
20153 5539 Acked Read 3 Sent Done Done false
20154 5540 Acked Write 9 Sent Done Done false
20155 5541 Acked Flush 0 Sent Done Done false
20156 5542 Acked Write 6 Sent Done Done false
20157 5543 Acked Write 1 Sent Done Done false
20158 5544 Acked Read 4 Sent Done Done false
20159 5545 Acked Flush 0 Sent Done Done false
20160 5546 Acked Read 5 Sent Done Done false
20161 5547 Acked Read 7 Sent Done Done false
20162 5548 Acked Read 5 Sent Done Done false
20163 5549 Acked Read 8 Sent Done Done false
20164 5550 Acked Read 2 Sent Done Done false
20165 5551 Acked Read 1 Sent Done Done false
20166 5552 Acked Read 9 Sent Done Done false
20167 5553 Acked Flush 0 Sent Done Done false
20168 5554 Acked Read 3 Sent Done Done false
20169 5555 Acked Flush 0 Sent Done Done false
20170 5556 Acked Read 9 Sent Done Done false
20171 5557 Acked Flush 0 Sent Done Done false
20172 5558 Acked Read 5 Sent Done Done false
20173 5559 Acked Write 10 Sent Done Done false
20174 5560 Acked Write 1 Sent Done Done false
20175 5561 Acked Read 10 Sent Done Done false
20176 5562 Acked Flush 0 Sent Done Done false
20177 5563 Acked Read 6 Sent Done Done false
20178 5564 Acked Read 9 Sent Done Done false
20179 5565 Acked Write 8 Sent Done Done false
20180 5566 Acked Write 8 Sent Done Done false
20181 5567 Acked Flush 0 Sent Done Done false
20182 5568 Acked Read 7 Sent Done Done false
20183 5569 Acked Write 5 Sent Done Done false
20184 5570 Acked Write 4 Sent Done Done false
20185 5571 Acked Write 4 Sent Done Done false
20186 5572 Acked Read 7 Sent Done Done false
20187 5573 Acked Write 8 Sent Done Done false
20188 5574 Acked Write 5 Sent Done Done false
20189 5575 Acked Read 6 Sent Done Done false
20190 5576 Acked Write 7 Sent Done Done false
20191 5577 Acked Read 10 Sent Done Done false
20192 5578 Acked Read 2 Sent Done Done false
20193 5579 Acked Read 4 Sent Done Done false
20194 5580 Acked Read 3 Sent Done Done false
20195 5581 Acked Flush 0 Sent Done Done false
20196 5582 Acked Flush 0 Sent Done Done false
20197 5583 Acked Write 2 Sent Done Done false
20198 5584 Acked Flush 0 Sent Done Done false
20199 5585 Acked Write 5 Sent Done Done false
20200 5586 Acked Write 4 Sent Done Done false
20201 5587 Acked Write 10 Sent Done Done false
20202 5588 Acked Read 3 Sent Done Done false
20203 5589 Acked Read 3 Sent Done Done false
20204 5590 Acked Flush 0 Sent Done Done false
20205 5591 Acked Flush 0 Sent Done Done false
20206 5592 Acked Read 8 Sent Done Done false
20207 5593 Acked Read 2 Sent Done Done false
20208 5594 Acked Write 9 Sent Done Done false
20209 5595 Acked Write 5 Sent Done Done false
20210 5596 Acked Read 2 Sent Done Done false
20211 5597 Acked Read 6 Sent Done Done false
20212 5598 Acked Read 5 Sent Done Done false
20213 5599 Acked Write 2 Sent Done Done false
20214 5600 Acked Read 1 Sent Done Done false
20215 5601 Acked Write 4 Sent Done Done false
20216 5602 Acked Read 9 Sent Done Done false
20217 5603 Acked Write 9 Sent Done Done false
20218 5604 Acked Write 3 Sent Done Done false
20219 5605 Acked Flush 0 Sent Done Done false
20220 5606 Acked Read 9 Sent Done Done false
20221 5607 Acked Write 5 Sent Done Done false
20222 5608 Acked Read 3 Sent Done Done false
20223 5609 Acked Read 6 Sent Done Done false
20224 5610 Acked Write 4 Sent Done Done false
20225 5611 Acked Write 4 Sent Done Done false
20226 5612 Acked Read 3 Sent Done Done false
20227 5613 Acked Read 2 Sent Done Done false
20228 5614 Acked Flush 0 Sent Done Done false
20229 5615 Acked Write 8 Sent Done Done false
20230 5616 Acked Write 3 Sent Done Done false
20231 5617 Acked Write 8 Sent Done Done false
20232 5618 Acked Read 10 Sent Done Done false
20233 5619 Acked Read 9 Sent Done Done false
20234 5620 Acked Flush 0 Sent Done Done false
20235 5621 Acked Read 10 Sent Done Done false
20236 5622 Acked Read 2 Sent Done Done false
20237 5623 Acked Read 4 Sent Done Done false
20238 5624 Acked Write 3 Sent Done Done false
20239 5625 Acked Write 2 Sent Done Done false
20240 5626 Acked Read 5 Sent Done Done false
20241 5627 Acked Read 5 Sent Done Done false
20242 5628 Acked Write 10 Sent Done Done false
20243 5629 Acked Write 1 Sent Done Done false
20244 5630 Acked Read 1 Sent Done Done false
20245 5631 Acked Write 8 Sent Done Done false
20246 5632 Acked Read 5 Sent Done Done false
20247 5633 Acked Read 5 Sent Done Done false
20248 5634 Acked Read 4 Sent Done Done false
20249 5635 Acked Read 3 Sent Done Done false
20250 5636 Acked Read 5 Sent Done Done false
20251 5637 Acked Write 3 Sent Done Done false
20252 5638 Acked Read 3 Sent Done Done false
20253 5639 Acked Write 9 Sent Done Done false
20254 5640 Acked Read 7 Sent Done Done false
20255 5641 Acked Write 5 Sent Done Done false
20256 5642 Acked Read 3 Sent Done Done false
20257 5643 Acked Write 4 Sent Done Done false
20258 5644 Acked Write 9 Sent Done Done false
20259 5645 Acked Write 1 Sent Done Done false
20260 5646 Acked Read 10 Sent Done Done false
20261 5647 Acked Read 1 Sent Done Done false
20262 5648 Acked Read 4 Sent Done Done false
20263 5649 Acked Read 3 Sent Done Done false
20264 5650 Acked Read 3 Sent Done Done false
20265 5651 Acked Write 1 Sent Done Done false
20266 5652 Acked Flush 0 Sent Done Done false
20267 5653 Acked Write 3 Sent Done Done false
20268 5654 Acked Read 3 Sent Done Done false
20269 5655 Acked Read 7 Sent Done Done false
20270 5656 Acked Read 9 Sent Done Done false
20271 5657 Acked Write 7 Sent Done Done false
20272 5658 Acked Write 5 Sent Done Done false
20273 5659 Acked Write 7 Sent Done Done false
20274 5660 Acked Read 5 Sent Done Done false
20275 5661 Acked Flush 0 Sent Done Done false
20276 5662 Acked Flush 0 Sent Done Done false
20277 5663 Acked Write 10 Sent Done Done false
20278 5664 Acked Read 5 Sent Done Done false
20279 5665 Acked Write 3 Sent Done Done false
20280 5666 Acked Write 8 Sent Done Done false
20281 5667 Acked Read 9 Sent Done Done false
20282 5668 Acked Flush 0 Sent Done Done false
20283 5669 Acked Write 3 Sent Done Done false
20284 5670 Acked Write 1 Sent Done Done false
20285 5671 Acked Read 9 Sent Done Done false
20286 5672 Acked Flush 0 Sent Done Done false
20287 5673 Acked Read 10 Sent Done Done false
20288 5674 Acked Write 6 Sent Done Done false
20289 5675 Acked Write 9 Sent Done Done false
20290 5676 Acked Read 9 Sent Done Done false
20291 5677 Acked Read 7 Sent Done Done false
20292 5678 Acked Read 3 Sent Done Done false
20293 5679 Acked Write 10 Sent Done Done false
20294 5680 Acked Read 1 Sent Done Done false
20295 5681 Acked Read 5 Sent Done Done false
20296 5682 Acked Write 3 Sent Done Done false
20297 5683 Acked Flush 0 Sent Done Done false
20298 5684 Acked Write 3 Sent Done Done false
20299 5685 Acked Read 9 Sent Done Done false
20300 5686 Acked Write 9 Sent Done Done false
20301 5687 Acked Read 6 Sent Done Done false
20302 5688 Acked Read 1 Sent Done Done false
20303 5689 Acked Read 3 Sent Done Done false
20304 5690 Acked Write 9 Sent Done Done false
20305 5691 Acked Write 2 Sent Done Done false
20306 5692 Acked Read 3 Sent Done Done false
20307 5693 Acked Read 7 Sent Done Done false
20308 5694 Acked Read 10 Sent Done Done false
20309 5695 Acked Write 10 Sent Done Done false
20310 5696 Acked Read 4 Sent Done Done false
20311 5697 Acked Read 10 Sent Done Done false
20312 5698 Acked Write 1 Sent Done Done false
20313 5699 Acked Write 4 Sent Done Done false
20314 5700 Acked Read 1 Sent Done Done false
20315 5701 Acked Read 3 Sent Done Done false
20316 5702 Acked Read 8 Sent Done Done false
20317 5703 Acked Read 4 Sent Done Done false
20318 5704 Acked Write 6 Sent Done Done false
20319 5705 Acked Read 4 Sent Done Done false
20320 5706 Acked Read 6 Sent Done Done false
20321 5707 Acked Read 8 Sent Done Done false
20322 5708 Acked Read 4 Sent Done Done false
20323 5709 Acked Read 5 Sent Done Done false
20324 5710 Acked Read 7 Sent Done Done false
20325 5711 Acked Write 4 Sent Done Done false
20326 5712 Acked Read 2 Sent Done Done false
20327 5713 Acked Write 8 Sent Done Done false
20328 5714 Acked Write 5 Sent Done Done false
20329 5715 Acked Flush 0 Sent Done Done false
20330 5716 Acked Read 10 Sent Done Done false
20331 5717 Acked Read 7 Sent Done Done false
20332 5718 Acked Read 10 Sent Done Done false
20333 5719 Acked Read 2 Sent Done Done false
20334 5720 Acked Read 5 Sent Done Done false
20335 5721 Acked Write 8 Sent Done Done false
20336 5722 Acked Write 9 Sent Done Done false
20337 5723 Acked Read 10 Sent Done Done false
20338 5724 Acked Write 1 Sent Done Done false
20339 5725 Acked Read 6 Sent Done Done false
20340 5726 Acked Read 4 Sent Done Done false
20341 5727 Acked Write 2 Sent Done Done false
20342 5728 Acked Write 2 Sent Done Done false
20343 5729 Acked Read 3 Sent Done Done false
20344 5730 Acked Write 2 Sent Done Done false
20345 5731 Acked Write 4 Sent Done Done false
20346 5732 Acked Write 3 Sent Done Done false
20347 5733 Acked Read 2 Sent Done Done false
20348 5734 Acked Write 1 Sent Done Done false
20349 5735 Acked Write 4 Sent Done Done false
20350 5736 Acked Write 5 Sent Done Done false
20351 5737 Acked Read 5 Sent Done Done false
20352 5738 Acked Write 8 Sent Done Done false
20353 5739 Acked Write 5 Sent Done Done false
20354 5740 Acked Write 4 Sent Done Done false
20355 5741 Acked Write 6 Sent Done Done false
20356 5742 Acked Write 8 Sent Done Done false
20357 5743 Acked Flush 0 Sent Done Done false
20358 5744 Acked Read 4 Sent Done Done false
20359 5745 Acked Read 7 Sent Done Done false
20360 5746 Acked Write 4 Sent Done Done false
20361 5747 Acked Read 10 Sent Done Done false
20362 5748 Acked Write 4 Sent Done Done false
20363 5749 Acked Read 3 Sent Done Done false
20364 5750 Acked Read 2 Sent Done Done false
20365 5751 Acked Write 5 Sent Done Done false
20366 5752 Acked Write 6 Sent Done Done false
20367 5753 Acked Write 3 Sent Done Done false
20368 5754 Acked Write 9 Sent Done Done false
20369 5755 Acked Flush 0 Sent Done Done false
20370 5756 Acked Read 9 Sent Done Done false
20371 5757 Acked Write 7 Sent Done Done false
20372 5758 Acked Read 3 Sent Done Done false
20373 5759 Acked Write 10 Sent Done Done false
20374 5760 Acked Write 4 Sent Done Done false
20375 5761 Acked Write 2 Sent Done Done false
20376 5762 Acked Flush 0 Sent Done Done false
20377 5763 Acked Write 7 Sent Done Done false
20378 5764 Acked Read 6 Sent Done Done false
20379 5765 Acked Write 7 Sent Done Done false
20380 5766 Acked Read 2 Sent Done Done false
20381 5767 Acked Read 8 Sent Done Done false
20382 5768 Acked Write 10 Sent Done Done false
20383 5769 Acked Write 7 Sent Done Done false
20384 5770 Acked Flush 0 Sent Done Done false
20385 5771 Acked Write 4 Sent Done Done false
20386 5772 Acked Read 8 Sent Done Done false
20387 5773 Acked Flush 0 Sent Done Done false
20388 5774 Acked Read 10 Sent Done Done false
20389 5775 Acked Write 6 Sent Done Done false
20390 5776 Acked Read 6 Sent Done Done false
20391 5777 Acked Read 8 Sent Done Done false
20392 5778 Acked Write 3 Sent Done Done false
20393 5779 Acked Write 9 Sent Done Done false
20394 5780 Acked Flush 0 Sent Done Done false
20395 5781 Acked Write 1 Sent Done Done false
20396 5782 Acked Read 6 Sent Done Done false
20397 5783 Acked Read 9 Sent Done Done false
20398 5784 Acked Write 4 Sent Done Done false
20399 5785 Acked Read 6 Sent Done Done false
20400 5786 Acked Flush 0 Sent Done Done false
20401 5787 Acked Write 6 Sent Done Done false
20402 5788 Acked Write 2 Sent Done Done false
20403 5789 Acked Write 5 Sent Done Done false
20404 5790 Acked Write 2 Sent Done Done false
20405 5791 Acked Read 4 Sent Done Done false
20406 5792 Acked Write 6 Sent Done Done false
20407 5793 Acked Write 10 Sent Done Done false
20408 5794 Acked Read 7 Sent Done Done false
20409 5795 Acked Write 2 Sent Done Done false
20410 5796 Acked Read 10 Sent Done Done false
20411 5797 Acked Write 4 Sent Done Done false
20412 5798 Acked Write 5 Sent Done Done false
20413 5799 Acked Write 3 Sent Done Done false
20414 5800 Acked Read 7 Sent Done Done false
20415 5801 Acked Read 7 Sent Done Done false
20416 5802 Acked Write 10 Sent Done Done false
20417 5803 Acked Read 5 Sent Done Done false
20418 5804 Acked Write 5 Sent Done Done false
20419 5805 Acked Write 2 Sent Done Done false
20420 5806 Acked Read 9 Sent Done Done false
20421 5807 Acked Write 5 Sent Done Done false
20422 5808 Acked Read 7 Sent Done Done false
20423 5809 Acked Read 9 Sent Done Done false
20424 5810 Acked Flush 0 Sent Done Done false
20425 5811 Acked Read 10 Sent Done Done false
20426 5812 Acked Write 4 Sent Done Done false
20427 5813 Acked Read 6 Sent Done Done false
20428 5814 Acked Read 3 Sent Done Done false
20429 5815 Acked Read 5 Sent Done Done false
20430 5816 Acked Write 1 Sent Done Done false
20431 5817 Acked Read 4 Sent Done Done false
20432 5818 Acked Flush 0 Sent Done Done false
20433 5819 Acked Flush 0 Sent Done Done false
20434 5820 Acked Write 7 Sent Done Done false
20435 5821 Acked Flush 0 Sent Done Done false
20436 5822 Acked Write 8 Sent Done Done false
20437 5823 Acked Flush 0 Sent Done Done false
20438 5824 Acked Read 3 Sent Done Done false
20439 5825 Acked Write 5 Sent Done Done false
20440 5826 Acked Read 2 Sent Done Done false
20441 5827 Acked Write 6 Sent Done Done false
20442 5828 Acked Write 5 Sent Done Done false
20443 5829 Acked Read 4 Sent Done Done false
20444 STATES DS:0 DS:1 DS:2 TOTAL
20445 Sent 301 0 0 301
20446 Done 0 301 301 602
20447 Skipped 0 0 0 0
20448 Error 0 0 0 0
20449 Last Flush: Some(JobId(5528)) Some(JobId(5823)) Some(JobId(5823))
20450 Downstairs last five completed: 5528 5527 5526 5525 5524
20451 Upstairs last five completed: 5829 5828 5827 5826 5825
20452 CLIENT: Up:0 ds:301 act:2
204532025-02-07T20:11:50.382ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
204542025-02-07T20:11:51.274ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
204552025-02-07T20:11:51.274ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52196) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
204562025-02-07T20:11:51.275ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
204572025-02-07T20:11:51.276ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
204582025-02-07T20:11:51.276ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
204592025-02-07T20:11:51.276ZINFOcrucible: send last flush ID to this DS: Some(JobId(5528)) = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
204602025-02-07T20:11:51.276ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
204612025-02-07T20:11:51.276ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(5528)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
204622025-02-07T20:11:51.276ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(5528)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
20463 ----------------------------------------------------------------
20464 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20465 Downstairs last five completed: 5830 5829 5828 5827 5826
20466 Upstairs last five completed: 5830 5829 5828 5827 5826
20467 CLIENT: Up:0 ds:0 act:3
20468 Replay: All jobs finished
20469 Main task has work to do, go find it
20470 got dsc Stop(2)
20471 stop 2
20472 Feb 07 20:11:53.521 [8830] Got stop action so:false kr:false
20473 INFO request completed, latency_us: 342, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 7e7145ed-e15f-46ee-9299-f9bce4d7cccc, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
20474 Feb 07 20:11:53.522 INFO{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:11:53.522083882Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"2","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
20475 request completed, latency_us: 136{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:11:53.522147416Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"2","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
20476 , {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:11:53.522186284Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
20477 response_code: 200, uri: /state/cid/2, method: GET, req_id: a96bbe31-8519-43a1-9994-df2b474aee3f, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
204782025-02-07T20:11:53.522ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
20479 [8830] Exited with: ExitStatus(unix_wait_status(9))
20480 [8830][2] reports Exit
20481 Feb 07 20:11:57.524 INFO request completed, latency_us: 249, response_code: 200, uri: /state/cid/2, method: GET, req_id: 99e9c06d-0363-41b5-97e4-364dca03ce83, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
20482 001/300 Write block 4525 len 12288 data: 4 4 5
20483 002/300 Write block 4826 len 8192 data: 4 3
20484 003/300 Write block 4142 len 28672 data: 4 4 5 4 4 5 5
20485 004/300 Write block 3749 len 24576 data: 6 6 6 5 5 5
20486 005/300 Read block 3445 len 36864
20487 006/300 Write block 4627 len 12288 data: 4 3 2
20488 007/300 Read block 518 len 16384
20489 008/300 Read block 1213 len 40960
20490 009/300 Read block 1420 len 12288
20491 010/300 Flush
20492 011/300 Write block 1221 len 40960 data: 5 6 8 8 7 6 6 4 5 5
20493 012/300 Read block 2045 len 12288
20494 013/300 Write block 178 len 12288 data: 3 3 2
20495 014/300 Read block 4214 len 24576
20496 015/300 Flush
20497 016/300 Write block 2341 len 20480 data: 8 7 5 4 5
20498 017/300 Read block 2625 len 12288
20499 018/300 Read block 1659 len 4096
20500 019/300 Read block 1692 len 28672
20501 020/300 Read block 4883 len 8192
20502 021/300 Write block 658 len 16384 data: 3 3 4 4
20503 022/300 Read block 2211 len 40960
20504 023/300 Write block 3955 len 40960 data: 4 4 3 4 4 4 3 3 4 4
20505 024/300 Flush
20506 025/300 Write block 3876 len 8192 data: 3 4
20507 026/300 Read block 3678 len 32768
20508 027/300 Read block 1261 len 36864
20509 028/300 Read block 4671 len 4096
20510 029/300 Read block 1350 len 8192
20511 030/300 Write block 3563 len 12288 data: 2 4 4
20512 031/300 Write block 915 len 12288 data: 2 2 2
20513 032/300 Write block 4399 len 32768 data: 3 3 2 2 2 4 5 4
20514 033/300 Flush
20515 034/300 Read block 3843 len 36864
20516 035/300 Read block 793 len 36864
20517 036/300 Flush
20518 037/300 Read block 2762 len 8192
20519 038/300 Write block 1761 len 16384 data: 5 3 4 4
20520 039/300 Flush
20521 040/300 Write block 225 len 12288 data: 4 4 2
20522 041/300 Flush
20523 042/300 Write block 16 len 36864 data: 3 3 2 2 4 4 4 4 5
20524 043/300 Read block 635 len 20480
20525 044/300 Write block 3350 len 4096 data: 5
20526 045/300 Read block 4888 len 12288
20527 046/300 Write block 4228 len 8192 data: 4 4
20528 047/300 Flush
20529 048/300 Write block 1745 len 12288 data: 3 3 3
20530 049/300 Write block 2481 len 4096 data: 2
20531 050/300 Write block 472 len 40960 data: 6 6 5 5 3 4 4 4 4 2
20532 051/300 Read block 2879 len 20480
20533 052/300 Flush
20534 053/300 Read block 4613 len 24576
20535 054/300 Write block 2460 len 24576 data: 4 4 2 4 5 6
20536 055/300 Write block 3438 len 12288 data: 5 5 3
20537 056/300 Flush
20538 057/300 Flush
20539 058/300 Read block 558 len 36864
20540 059/300 Write block 418 len 40960 data: 5 6 6 5 5 4 5 4 5 4
20541 060/300 Read block 2169 len 24576
20542 061/300 Read block 1459 len 4096
20543 062/300 Read block 2535 len 12288
20544 063/300 Flush
20545 064/300 Read block 2644 len 40960
20546 065/300 Read block 430 len 8192
20547 066/300 Write block 2593 len 8192 data: 7 7
20548 067/300 Read block 582 len 32768
20549 068/300 Write block 1246 len 40960 data: 3 5 5 5 4 5 4 4 4 3
20550 069/300 Read block 4250 len 4096
20551 070/300 Write block 3175 len 4096 data: 3
20552 071/300 Flush
20553 072/300 Write block 1393 len 32768 data: 4 3 3 4 5 5 4 5
20554 073/300 Read block 3661 len 40960
20555 074/300 Read block 2376 len 36864
20556 075/300 Read block 4035 len 8192
20557 076/300 Read block 2173 len 40960
20558 077/300 Write block 3265 len 8192 data: 5 5
20559 078/300 Write block 1327 len 36864 data: 6 4 5 6 6 5 4 4 3
20560 079/300 Flush
20561 080/300 Write block 263 len 4096 data: 3
20562 081/300 Write block 1011 len 36864 data: 2 2 3 4 4 4 5 4 7
20563 082/300 Read block 272 len 4096
20564 083/300 Read block 1499 len 20480
20565 084/300 Write block 4141 len 32768 data: 4 5 5 6 5 5 6 6
20566 085/300 Read block 2552 len 36864
20567 086/300 Flush
20568 087/300 Write block 3618 len 4096 data: 5
20569 088/300 Write block 640 len 8192 data: 3 3
20570 089/300 Read block 4609 len 24576
20571 090/300 Write block 457 len 4096 data: 6
20572 091/300 Write block 337 len 16384 data: 7 6 6 6
20573 092/300 Write block 3652 len 20480 data: 4 4 6 6 5
20574 093/300 Write block 2396 len 8192 data: 5 5
20575 094/300 Flush
20576 095/300 Read block 1504 len 20480
20577 096/300 Write block 237 len 16384 data: 4 4 5 5
20578 097/300 Flush
20579 098/300 Read block 4553 len 20480
20580 099/300 Write block 1166 len 4096 data: 2
20581 100/300 Read block 2928 len 16384
20582 101/300 Write block 3704 len 20480 data: 5 5 4 4 6
20583 102/300 Read block 430 len 16384
20584 103/300 Write block 4057 len 4096 data: 2
20585 104/300 Write block 4588 len 20480 data: 4 5 4 4 4
20586 105/300 Read block 4966 len 16384
20587 106/300 Read block 4682 len 28672
20588 107/300 Write block 216 len 20480 data: 5 5 5 5 5
20589 108/300 Read block 2174 len 32768
20590 109/300 Write block 4212 len 32768 data: 4 4 4 5 4 6 6 6
20591 110/300 Read block 1867 len 12288
20592 111/300 Write block 2768 len 28672 data: 3 3 4 4 4 2 5
20593 112/300 Read block 1048 len 20480
20594 113/300 Write block 1227 len 40960 data: 7 5 6 6 5 4 4 4 6 4
20595 114/300 Read block 2465 len 4096
20596 115/300 Write block 418 len 12288 data: 6 7 7
20597 116/300 Read block 275 len 20480
20598 117/300 Read block 1215 len 8192
20599 118/300 Read block 2478 len 40960
20600 119/300 Read block 3362 len 24576
20601 120/300 Read block 4860 len 36864
20602 121/300 Write block 3309 len 12288 data: 4 3 4
20603 122/300 Write block 1440 len 20480 data: 8 5 4 3 3
20604 123/300 Write block 2364 len 28672 data: 9 8 8 7 8 5 5
20605 124/300 Flush
20606 125/300 Read block 2500 len 12288
20607 126/300 Flush
20608 127/300 Write block 1044 len 36864 data: 5 5 5 5 4 4 4 5 5
20609 128/300 Read block 3653 len 16384
20610 129/300 Write block 691 len 20480 data: 3 2 2 2 3
20611 130/300 Read block 4439 len 32768
20612 131/300 Flush
20613 132/300 Read block 3691 len 12288
20614 133/300 Write block 4204 len 36864 data: 3 3 3 5 5 4 4 4 5
20615 134/300 Write block 4641 len 12288 data: 3 3 4
20616 135/300 Write block 4234 len 24576 data: 3 2 3 3 4 4
20617 136/300 Read block 1007 len 20480
20618 137/300 Read block 505 len 40960
20619 138/300 Read block 711 len 12288
20620 139/300 Write block 3883 len 32768 data: 7 4 4 4 4 5 4 4
20621 140/300 Write block 124 len 16384 data: 4 4 4 5
20622 141/300 Write block 526 len 16384 data: 3 4 6 5
20623 142/300 Flush
20624 143/300 Write block 2837 len 16384 data: 3 3 4 4
20625 144/300 Read block 3388 len 20480
20626 145/300 Write block 2089 len 40960 data: 4 5 6 6 4 4 4 4 3 3
20627 146/300 Write block 18 len 16384 data: 3 3 5 5
20628 147/300 Write block 765 len 32768 data: 3 4 4 5 5 7 6 5
20629 148/300 Read block 4575 len 20480
20630 149/300 Read block 2851 len 28672
20631 150/300 Write block 31 len 8192 data: 4 4
20632 151/300 Flush
20633 152/300 Write block 2637 len 4096 data: 3
20634 153/300 Write block 3907 len 36864 data: 5 5 4 5 6 8 6 6 5
20635 154/300 Read block 3359 len 12288
20636 155/300 Read block 4389 len 40960
20637 156/300 Read block 78 len 8192
20638 157/300 Read block 266 len 20480
20639 158/300 Read block 4224 len 8192
20640 159/300 Read block 1142 len 28672
20641 160/300 Read block 508 len 16384
20642 161/300 Read block 2003 len 36864
20643 162/300 Write block 3871 len 36864 data: 3 2 2 2 3 4 5 2 2
20644 163/300 Write block 695 len 32768 data: 4 3 3 4 3 3 3 3
20645 164/300 Write block 3367 len 24576 data: 5 4 3 3 3 4
20646 165/300 Read block 341 len 20480
20647 166/300 Read block 3609 len 12288
20648 167/300 Read block 3049 len 16384
20649 168/300 Read block 4163 len 40960
20650 169/300 Write block 4735 len 20480 data: 9 7 6 7 7
20651 170/300 Read block 2673 len 36864
20652 171/300 Read block 3358 len 36864
20653 172/300 Write block 3880 len 4096 data: 2
20654 173/300 Write block 699 len 4096 data: 4
20655 174/300 Write block 3144 len 12288 data: 4 5 5
20656 175/300 Write block 3678 len 36864 data: 7 7 7 8 8 7 8 6 7
20657 176/300 Write block 4151 len 4096 data: 4
20658 177/300 Write block 2020 len 24576 data: 7 6 5 5 5 5
20659 178/300 Write block 3271 len 20480 data: 4 5 5 6 7
20660 179/300 Read block 1344 len 4096
20661 180/300 Flush
20662 181/300 Read block 3741 len 12288
20663 182/300 Read block 4788 len 28672
20664 183/300 Read block 2429 len 4096
20665 184/300 Write block 1191 len 28672 data: 5 4 3 2 5 4 3
20666 185/300 Read block 1722 len 8192
20667 186/300 Read block 4750 len 36864
20668 187/300 Read block 730 len 24576
20669 188/300 Write block 4119 len 4096 data: 3
20670 189/300 Write block 3261 len 20480 data: 6 6 6 5 6
20671 190/300 Write block 822 len 36864 data: 5 5 6 7 6 5 7 7 8
20672 191/300 Read block 309 len 32768
20673 192/300 Flush
20674 193/300 Read block 376 len 20480
20675 194/300 Read block 4017 len 28672
20676 195/300 Read block 833 len 24576
20677 196/300 Read block 1459 len 20480
20678 197/300 Write block 247 len 24576 data: 4 4 3 3 3 4
20679 198/300 Read block 4255 len 24576
20680 199/300 Write block 1820 len 8192 data: 2 3
20681 200/300 Write block 3257 len 20480 data: 4 4 5 6 7
20682 201/300 Flush
20683 202/300 Write block 1782 len 32768 data: 5 6 4 4 4 5 4 4
20684 203/300 Read block 3384 len 12288
20685 204/300 Write block 3984 len 8192 data: 4 3
20686 205/300 Read block 2383 len 32768
20687 206/300 Read block 746 len 4096
20688 207/300 Write block 1341 len 24576 data: 4 4 5 6 7 6
20689 208/300 Read block 4432 len 32768
20690 209/300 Flush
20691 210/300 Read block 3782 len 20480
20692 211/300 Write block 3476 len 20480 data: 6 7 7 6 6
20693 212/300 Read block 2192 len 40960
20694 213/300 Read block 2415 len 16384
20695 214/300 Write block 4060 len 8192 data: 6 8
20696 215/300 Read block 1955 len 24576
20697 216/300 Read block 987 len 40960
20698 217/300 Write block 2255 len 40960 data: 3 3 4 4 5 4 6 6 6 5
20699 218/300 Read block 3805 len 36864
20700 219/300 Read block 2470 len 4096
20701 220/300 Read block 4571 len 4096
20702 221/300 Flush
20703 222/300 Read block 1002 len 28672
20704 223/300 Write block 3082 len 12288 data: 4 4 4
20705 224/300 Write block 1930 len 28672 data: 3 4 4 4 4 4 4
20706 225/300 Read block 1609 len 4096
20707 226/300 Write block 1942 len 36864 data: 3 3 3 3 2 3 4 4 4
20708 227/300 Read block 3397 len 20480
20709 228/300 Read block 862 len 4096
20710 229/300 Read block 499 len 16384
20711 230/300 Read block 1912 len 12288
20712 231/300 Read block 2225 len 4096
20713 232/300 Read block 4615 len 24576
20714 233/300 Write block 2498 len 20480 data: 3 3 2 3 3
20715 234/300 Read block 2345 len 28672
20716 235/300 Read block 3707 len 36864
20717 236/300 Read block 4078 len 24576
20718 237/300 Write block 4394 len 8192 data: 3 3
20719 238/300 Write block 1749 len 20480 data: 3 3 3 3 3
20720 239/300 Read block 3670 len 32768
20721 240/300 Write block 4715 len 12288 data: 3 4 4
20722 241/300 Write block 209 len 36864 data: 3 4 3 3 3 4 4 6 6
20723 242/300 Write block 45 len 20480 data: 5 3 3 3 3
20724 243/300 Write block 2019 len 24576 data: 6 8 7 6 6 6
20725 244/300 Read block 4319 len 20480
20726 245/300 Read block 4238 len 8192
20727 246/300 Read block 111 len 24576
20728 247/300 Read block 586 len 8192
20729 248/300 Read block 1065 len 8192
20730 249/300 Read block 599 len 32768
20731 250/300 Flush
20732 251/300 Read block 3242 len 40960
20733 252/300 Flush
20734 253/300 Read block 3847 len 4096
20735 254/300 Write block 98 len 36864 data: 5 4 4 4 4 4 3 3 2
20736 255/300 Write block 2537 len 20480 data: 4 4 5 4 4
20737 256/300 Read block 501 len 16384
20738 257/300 Read block 235 len 36864
20739 258/300 Read block 2744 len 24576
20740 259/300 Flush
20741 260/300 Read block 655 len 28672
20742 261/300 Flush
20743 262/300 Read block 2358 len 12288
20744 263/300 Read block 1657 len 8192
20745 264/300 Write block 4262 len 8192 data: 3 3
20746 265/300 Read block 558 len 4096
20747 266/300 Flush
20748 267/300 Read block 2882 len 12288
20749 268/300 Read block 4690 len 40960
20750 269/300 Write block 792 len 24576 data: 4 4 4 3 4 4
20751 270/300 Write block 3066 len 24576 data: 3 2 2 3 3 3
20752 271/300 Read block 4094 len 8192
20753 272/300 Write block 119 len 24576 data: 4 4 6 5 4 5
20754 273/300 Read block 4213 len 16384
20755 274/300 Flush
20756 275/300 Read block 3789 len 24576
20757 276/300 Write block 2951 len 12288 data: 3 5 6
20758 277/300 Write block 3997 len 32768 data: 7 7 5 5 4 4 4 4
20759 278/300 Read block 867 len 12288
20760 279/300 Read block 1366 len 16384
20761 280/300 Write block 3635 len 40960 data: 3 3 3 3 3 3 2 2 3 3
20762 281/300 Read block 1507 len 8192
20763 282/300 Read block 2666 len 36864
20764 283/300 Write block 186 len 36864 data: 6 5 5 4 4 4 4 3 3
20765 284/300 Flush
20766 285/300 Write block 4975 len 40960 data: 5 4 3 3 5 5 4 4 3 4
20767 286/300 Write block 4709 len 28672 data: 2 2 2 2 2 2 4
20768 287/300 Flush
20769 288/300 Write block 3946 len 40960 data: 5 7 7 5 5 3 3 3 3 5
20770 289/300 Read block 3421 len 28672
20771 290/300 Write block 2205 len 16384 data: 6 6 5 5
20772 291/300 Write block 3177 len 12288 data: 3 3 2
20773 292/300 Read block 1411 len 32768
20774 293/300 Read block 1168 len 16384
20775 294/300 Read block 1918 len 40960
20776 295/300 Read block 3717 len 28672
20777 296/300 Write block 2244 len 16384 data: 4 4 3 3
20778 297/300 Write block 3820 len 24576 data: 5 3 3 3 3 3
20779 298/300 Flush
20780 299/300 Read block 1806 len 32768
20781 300/300 Read block 4821 len 12288
20782 301/300 Write block 1681 len 36864 data: 4 4 5 6 5 5 5 5 4
20783 Main task has work to do, go find it
20784 got dsc Start(2)
20785 start 2
20786 Feb 07 20:11:57.760 INFO request completed, latency_us: 234, response_code: 204, uri: /start/cid/2, method: POST, req_id: 6419d612-cc91-42e1-95ef-255f01aa5075, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
20787 Replay: started 2, returned:Ok(())
20788 ----------------------------------------------------------------
20789 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20790 States: Active Active Offline
20791 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20792 5831 Acked Write 3 Done Done Sent false
20793 5832 Acked Write 2 Done Done Sent false
20794 5833 Acked Write 7 Done Done Sent false
20795 5834 Acked Write 6 Done Done Sent false
20796 5835 Acked Read 9 Done Done Sent false
20797 5836 Acked Write 3 Done Done Sent false
20798 5837 Acked Read 4 Done Done Sent false
20799 5838 Acked Read 10 Done Done Sent false
20800 5839 Acked Read 3 Done Done Sent false
20801 5840 Acked Flush 0 Done Done Sent false
20802 5841 Acked Write 10 Done Done Sent false
20803 5842 Acked Read 3 Done Done Sent false
20804 5843 Acked Write 3 Done Done Sent false
20805 5844 Acked Read 6 Done Done Sent false
20806 5845 Acked Flush 0 Done Done Sent false
20807 5846 Acked Write 5 Done Done Sent false
20808 5847 Acked Read 3 Done Done Sent false
20809 5848 Acked Read 1 Done Done Sent false
20810 5849 Acked Read 7 Done Done Sent false
20811 5850 Acked Read 2 Done Done Sent false
20812 5851 Acked Write 4 Done Done Sent false
20813 5852 Acked Read 10 Done Done Sent false
20814 5853 Acked Write 10 Done Done Sent false
20815 5854 Acked Flush 0 Done Done Sent false
20816 5855 Acked Write 2 Done Done Sent false
20817 5856 Acked Read 8 Done Done Sent false
20818 5857 Acked Read 9 Done Done Sent false
20819 5858 Acked Read 1 Done Done Sent false
20820 5859 Acked Read 2 Done Done Sent false
20821 5860 Acked Write 3 Done Done Sent false
20822 5861 Acked Write 3 Done Done Sent false
20823 5862 Acked Write 8 Done Done Sent false
20824 5863 Acked Flush 0 Done Done Sent false
20825 5864 Acked Read 9 Done Done Sent false
20826 5865 Acked Read 9 Done Done Sent false
20827 5866 Acked Flush 0 Done Done Sent false
20828 5867 Acked Read 2 Done Done Sent false
20829 5868 Acked Write 4 Done Done Sent false
20830 5869 Acked Flush 0 Done Done Sent false
20831 5870 Acked Write 3 Done Done Sent false
20832 5871 Acked Flush 0 Done Done Sent false
20833 5872 Acked Write 9 Done Done Sent false
20834 5873 Acked Read 5 Done Done Sent false
20835 5874 Acked Write 1 Done Done Sent false
20836 5875 Acked Read 3 Done Done Sent false
20837 5876 Acked Write 2 Done Done Sent false
20838 5877 Acked Flush 0 Done Done Sent false
20839 5878 Acked Write 3 Done Done Sent false
20840 5879 Acked Write 1 Done Done Sent false
20841 5880 Acked Write 10 Done Done Sent false
20842 5881 Acked Read 5 Done Done Sent false
20843 5882 Acked Flush 0 Done Done Sent false
20844 5883 Acked Read 6 Done Done Sent false
20845 5884 Acked Write 6 Done Done Sent false
20846 5885 Acked Write 3 Done Done Sent false
20847 5886 Acked Flush 0 Done Done Sent false
20848 5887 Acked Flush 0 Done Done Sent false
20849 5888 Acked Read 9 Done Done Sent false
20850 5889 Acked Write 10 Done Done Sent false
20851 5890 Acked Read 6 Done Done Sent false
20852 5891 Acked Read 1 Done Done Sent false
20853 5892 Acked Read 3 Done Done Sent false
20854 5893 Acked Flush 0 Done Done Sent false
20855 5894 Acked Read 10 Done Done Sent false
20856 5895 Acked Read 2 Done Done Sent false
20857 5896 Acked Write 2 Done Done Sent false
20858 5897 Acked Read 8 Done Done Sent false
20859 5898 Acked Write 10 Done Done Sent false
20860 5899 Acked Read 1 Done Done Sent false
20861 5900 Acked Write 1 Done Done Sent false
20862 5901 Acked Flush 0 Done Done Sent false
20863 5902 Acked Write 8 Done Done Sent false
20864 5903 Acked Read 10 Done Done Sent false
20865 5904 Acked Read 9 Done Done Sent false
20866 5905 Acked Read 2 Done Done Sent false
20867 5906 Acked Read 10 Done Done Sent false
20868 5907 Acked Write 2 Done Done Sent false
20869 5908 Acked Write 9 Done Done Sent false
20870 5909 Acked Flush 0 Done Done Sent false
20871 5910 Acked Write 1 Done Done Sent false
20872 5911 Acked Write 9 Done Done Sent false
20873 5912 Acked Read 1 Done Done Sent false
20874 5913 Acked Read 5 Done Done Sent false
20875 5914 Acked Write 8 Done Done Sent false
20876 5915 Acked Read 9 Done Done Sent false
20877 5916 Acked Flush 0 Done Done Sent false
20878 5917 Acked Write 1 Done Done Sent false
20879 5918 Acked Write 2 Done Done Sent false
20880 5919 Acked Read 6 Done Done Sent false
20881 5920 Acked Write 1 Done Done Sent false
20882 5921 Acked Write 4 Done Done Sent false
20883 5922 Acked Write 5 Done Done Sent false
20884 5923 Acked Write 2 Done Done Sent false
20885 5924 Acked Flush 0 Done Done Sent false
20886 5925 Acked Read 5 Done Done Sent false
20887 5926 Acked Write 4 Done Done Sent false
20888 5927 Acked Flush 0 Done Done Sent false
20889 5928 Acked Read 5 Done Done Sent false
20890 5929 Acked Write 1 Done Done Sent false
20891 5930 Acked Read 4 Done Done Sent false
20892 5931 Acked Write 5 Done Done Sent false
20893 5932 Acked Read 4 Done Done Sent false
20894 5933 Acked Write 1 Done Done Sent false
20895 5934 Acked Write 5 Done Done Sent false
20896 5935 Acked Read 4 Done Done Sent false
20897 5936 Acked Read 7 Done Done Sent false
20898 5937 Acked Write 5 Done Done Sent false
20899 5938 Acked Read 8 Done Done Sent false
20900 5939 Acked Write 8 Done Done Sent false
20901 5940 Acked Read 3 Done Done Sent false
20902 5941 Acked Write 7 Done Done Sent false
20903 5942 Acked Read 5 Done Done Sent false
20904 5943 Acked Write 10 Done Done Sent false
20905 5944 Acked Read 1 Done Done Sent false
20906 5945 Acked Write 3 Done Done Sent false
20907 5946 Acked Read 5 Done Done Sent false
20908 5947 Acked Read 2 Done Done Sent false
20909 5948 Acked Read 10 Done Done Sent false
20910 5949 Acked Read 6 Done Done Sent false
20911 5950 Acked Read 9 Done Done Sent false
20912 5951 Acked Write 3 Done Done Sent false
20913 5952 Acked Write 5 Done Done Sent false
20914 5953 Acked Write 7 Done Done Sent false
20915 5954 Acked Flush 0 Done Done Sent false
20916 5955 Acked Read 3 Done Done Sent false
20917 5956 Acked Flush 0 Done Done Sent false
20918 5957 Acked Write 9 Done Done Sent false
20919 5958 Acked Read 4 Done Done Sent false
20920 5959 Acked Write 5 Done Done Sent false
20921 5960 Acked Read 8 Done Done Sent false
20922 5961 Acked Flush 0 Done Done Sent false
20923 5962 Acked Read 3 Done Done Sent false
20924 5963 Acked Write 9 Done Done Sent false
20925 5964 Acked Write 3 Done Done Sent false
20926 5965 Acked Write 6 Done Done Sent false
20927 5966 Acked Read 5 Done Done Sent false
20928 5967 Acked Read 10 Done Done Sent false
20929 5968 Acked Read 3 Done Done Sent false
20930 5969 Acked Write 8 Done Done Sent false
20931 5970 Acked Write 4 Done Done Sent false
20932 5971 Acked Write 4 Done Done Sent false
20933 5972 Acked Flush 0 Done Done Sent false
20934 5973 Acked Write 4 Done Done Sent false
20935 5974 Acked Read 5 Done Done Sent false
20936 5975 Acked Write 10 Done Done Sent false
20937 5976 Acked Write 4 Done Done Sent false
20938 5977 Acked Write 8 Done Done Sent false
20939 5978 Acked Read 5 Done Done Sent false
20940 5979 Acked Read 7 Done Done Sent false
20941 5980 Acked Write 2 Done Done Sent false
20942 5981 Acked Flush 0 Done Done Sent false
20943 5982 Acked Write 1 Done Done Sent false
20944 5983 Acked Write 9 Done Done Sent false
20945 5984 Acked Read 3 Done Done Sent false
20946 5985 Acked Read 10 Done Done Sent false
20947 5986 Acked Read 2 Done Done Sent false
20948 5987 Acked Read 5 Done Done Sent false
20949 5988 Acked Read 2 Done Done Sent false
20950 5989 Acked Read 7 Done Done Sent false
20951 5990 Acked Read 4 Done Done Sent false
20952 5991 Acked Read 9 Done Done Sent false
20953 5992 Acked Write 9 Done Done Sent false
20954 5993 Acked Write 8 Done Done Sent false
20955 5994 Acked Write 6 Done Done Sent false
20956 5995 Acked Read 5 Done Done Sent false
20957 5996 Acked Read 3 Done Done Sent false
20958 5997 Acked Read 4 Done Done Sent false
20959 5998 Acked Read 10 Done Done Sent false
20960 5999 Acked Write 5 Done Done Sent false
20961 6000 Acked Read 9 Done Done Sent false
20962 6001 Acked Read 9 Done Done Sent false
20963 6002 Acked Write 1 Done Done Sent false
20964 6003 Acked Write 1 Done Done Sent false
20965 6004 Acked Write 3 Done Done Sent false
20966 6005 Acked Write 9 Done Done Sent false
20967 6006 Acked Write 1 Done Done Sent false
20968 6007 Acked Write 6 Done Done Sent false
20969 6008 Acked Write 5 Done Done Sent false
20970 6009 Acked Read 1 Done Done Sent false
20971 6010 Acked Flush 0 Done Done Sent false
20972 6011 Acked Read 3 Done Done Sent false
20973 6012 Acked Read 7 Done Done Sent false
20974 6013 Acked Read 1 Done Done Sent false
20975 6014 Acked Write 7 Done Done Sent false
20976 6015 Acked Read 2 Done Done Sent false
20977 6016 Acked Read 9 Done Done Sent false
20978 6017 Acked Read 6 Done Done Sent false
20979 6018 Acked Write 1 Done Done Sent false
20980 6019 Acked Write 5 Done Done Sent false
20981 6020 Acked Write 9 Done Done Sent false
20982 6021 Acked Read 8 Done Done Sent false
20983 6022 Acked Flush 0 Done Done Sent false
20984 6023 Acked Read 5 Done Done Sent false
20985 6024 Acked Read 7 Done Done Sent false
20986 6025 Acked Read 6 Done Done Sent false
20987 6026 Acked Read 5 Done Done Sent false
20988 6027 Acked Write 6 Done Done Sent false
20989 6028 Acked Read 6 Done Done Sent false
20990 6029 Acked Write 2 Done Done Sent false
20991 6030 Acked Write 5 Done Done Sent false
20992 6031 Acked Flush 0 Done Done Sent false
20993 6032 Acked Write 8 Done Done Sent false
20994 6033 Acked Read 3 Done Done Sent false
20995 6034 Acked Write 2 Done Done Sent false
20996 6035 Acked Read 8 Done Done Sent false
20997 6036 Acked Read 1 Done Done Sent false
20998 6037 Acked Write 6 Done Done Sent false
20999 6038 Acked Read 8 Done Done Sent false
21000 6039 Acked Flush 0 Done Done Sent false
21001 6040 Acked Read 5 Done Done Sent false
21002 6041 Acked Write 5 Done Done Sent false
21003 6042 Acked Read 10 Done Done Sent false
21004 6043 Acked Read 4 Done Done Sent false
21005 6044 Acked Write 2 Done Done Sent false
21006 6045 Acked Read 6 Done Done Sent false
21007 6046 Acked Read 10 Done Done Sent false
21008 6047 Acked Write 10 Done Done Sent false
21009 6048 Acked Read 9 Done Done Sent false
21010 6049 Acked Read 1 Done Done Sent false
21011 6050 Acked Read 1 Done Done Sent false
21012 6051 Acked Flush 0 Done Done Sent false
21013 6052 Acked Read 7 Done Done Sent false
21014 6053 Acked Write 3 Done Done Sent false
21015 6054 Acked Write 7 Done Done Sent false
21016 6055 Acked Read 1 Done Done Sent false
21017 6056 Acked Write 9 Done Done Sent false
21018 6057 Acked Read 5 Done Done Sent false
21019 6058 Acked Read 1 Done Done Sent false
21020 6059 Acked Read 4 Done Done Sent false
21021 6060 Acked Read 3 Done Done Sent false
21022 6061 Acked Read 1 Done Done Sent false
21023 6062 Acked Read 6 Done Done Sent false
21024 6063 Acked Write 5 Done Done Sent false
21025 6064 Acked Read 7 Done Done Sent false
21026 6065 Acked Read 9 Done Done Sent false
21027 6066 Acked Read 6 Done Done Sent false
21028 6067 Acked Write 2 Done Done Sent false
21029 6068 Acked Write 5 Done Done Sent false
21030 6069 Acked Read 8 Done Done Sent false
21031 6070 Acked Write 3 Done Done Sent false
21032 6071 Acked Write 9 Done Done Sent false
21033 6072 Acked Write 5 Done Done Sent false
21034 6073 Acked Write 6 Done Done Sent false
21035 6074 Acked Read 5 Done Done Sent false
21036 6075 Acked Read 2 Done Done Sent false
21037 6076 Acked Read 6 Done Done Sent false
21038 6077 Acked Read 2 Done Done Sent false
21039 6078 Acked Read 2 Done Done Sent false
21040 6079 Acked Read 8 Done Done Sent false
21041 6080 Acked Flush 0 Done Done Sent false
21042 6081 Acked Read 10 Done Done Sent false
21043 6082 Acked Flush 0 Done Done Sent false
21044 6083 Acked Read 1 Done Done Sent false
21045 6084 Acked Write 9 Done Done Sent false
21046 6085 Acked Write 5 Done Done Sent false
21047 6086 Acked Read 4 Done Done Sent false
21048 6087 Acked Read 9 Done Done Sent false
21049 6088 Acked Read 6 Done Done Sent false
21050 6089 Acked Flush 0 Done Done Sent false
21051 6090 Acked Read 7 Done Done Sent false
21052 6091 Acked Flush 0 Done Done Sent false
21053 6092 Acked Read 3 Done Done Sent false
21054 6093 Acked Read 2 Done Done Sent false
21055 6094 Acked Write 2 Done Done Sent false
21056 6095 Acked Read 1 Done Done Sent false
21057 6096 Acked Flush 0 Done Done Sent false
21058 6097 Acked Read 3 Done Done Sent false
21059 6098 Acked Read 10 Done Done Sent false
21060 6099 Acked Write 6 Done Done Sent false
21061 6100 Acked Write 6 Done Done Sent false
21062 6101 Acked Read 2 Done Done Sent false
21063 6102 Acked Write 6 Done Done Sent false
21064 6103 Acked Read 4 Done Done Sent false
21065 6104 Acked Flush 0 Done Done Sent false
21066 6105 Acked Read 6 Done Done Sent false
21067 6106 Acked Write 3 Done Done Sent false
21068 6107 Acked Write 8 Done Done Sent false
21069 6108 Acked Read 3 Done Done Sent false
21070 6109 Acked Read 4 Done Done Sent false
21071 6110 Acked Write 10 Done Done Sent false
21072 6111 Acked Read 2 Done Done Sent false
21073 6112 Acked Read 9 Done Done Sent false
21074 6113 Acked Write 9 Done Done Sent false
21075 6114 Acked Flush 0 Done Done Sent false
21076 6115 Acked Write 10 Done Done Sent false
21077 6116 Acked Write 7 Done Done Sent false
21078 6117 Acked Flush 0 Done Done Sent false
21079 6118 Acked Write 10 Done Done Sent false
21080 6119 Acked Read 7 Done Done Sent false
21081 6120 Acked Write 4 Done Done Sent false
21082 6121 Acked Write 3 Done Done Sent false
21083 6122 Acked Read 8 Done Done Sent false
21084 6123 Acked Read 4 Done Done Sent false
21085 6124 Acked Read 10 Done Done Sent false
21086 6125 Acked Read 7 Done Done Sent false
21087 6126 Acked Write 4 Done Done Sent false
21088 6127 Acked Write 6 Done Done Sent false
21089 6128 Acked Flush 0 Done Done Sent false
21090 6129 Acked Read 8 Done Done Sent false
21091 6130 Acked Read 3 Done Done Sent false
21092 6131 Acked Write 9 Done Done Sent false
21093 STATES DS:0 DS:1 DS:2 TOTAL
21094 Sent 0 0 301 301
21095 Done 301 301 0 602
21096 Skipped 0 0 0 0
21097 Error 0 0 0 0
21098 Last Flush: Some(JobId(6128)) Some(JobId(6128)) Some(JobId(5830))
21099 Downstairs last five completed: 5830 5829 5828 5827 5826
21100 Upstairs last five completed: 6131 6130 6129 6128 6127
21101 CLIENT: Up:0 ds:301 act:2
211022025-02-07T20:11:58.396ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
211032025-02-07T20:11:59.398ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
211042025-02-07T20:12:00.400ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
211052025-02-07T20:12:00.901ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
21106 ----------------------------------------------------------------
21107 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21108 States: Active Active Offline
21109 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21110 5831 Acked Write 3 Done Done Sent false
21111 5832 Acked Write 2 Done Done Sent false
21112 5833 Acked Write 7 Done Done Sent false
21113 5834 Acked Write 6 Done Done Sent false
21114 5835 Acked Read 9 Done Done Sent false
21115 5836 Acked Write 3 Done Done Sent false
21116 5837 Acked Read 4 Done Done Sent false
21117 5838 Acked Read 10 Done Done Sent false
21118 5839 Acked Read 3 Done Done Sent false
21119 5840 Acked Flush 0 Done Done Sent false
21120 5841 Acked Write 10 Done Done Sent false
21121 5842 Acked Read 3 Done Done Sent false
21122 5843 Acked Write 3 Done Done Sent false
21123 5844 Acked Read 6 Done Done Sent false
21124 5845 Acked Flush 0 Done Done Sent false
21125 5846 Acked Write 5 Done Done Sent false
21126 5847 Acked Read 3 Done Done Sent false
21127 5848 Acked Read 1 Done Done Sent false
21128 5849 Acked Read 7 Done Done Sent false
21129 5850 Acked Read 2 Done Done Sent false
21130 5851 Acked Write 4 Done Done Sent false
21131 5852 Acked Read 10 Done Done Sent false
21132 5853 Acked Write 10 Done Done Sent false
21133 5854 Acked Flush 0 Done Done Sent false
21134 5855 Acked Write 2 Done Done Sent false
21135 5856 Acked Read 8 Done Done Sent false
21136 5857 Acked Read 9 Done Done Sent false
21137 5858 Acked Read 1 Done Done Sent false
21138 5859 Acked Read 2 Done Done Sent false
21139 5860 Acked Write 3 Done Done Sent false
21140 5861 Acked Write 3 Done Done Sent false
21141 5862 Acked Write 8 Done Done Sent false
21142 5863 Acked Flush 0 Done Done Sent false
21143 5864 Acked Read 9 Done Done Sent false
21144 5865 Acked Read 9 Done Done Sent false
21145 5866 Acked Flush 0 Done Done Sent false
21146 5867 Acked Read 2 Done Done Sent false
21147 5868 Acked Write 4 Done Done Sent false
21148 5869 Acked Flush 0 Done Done Sent false
21149 5870 Acked Write 3 Done Done Sent false
21150 5871 Acked Flush 0 Done Done Sent false
21151 5872 Acked Write 9 Done Done Sent false
21152 5873 Acked Read 5 Done Done Sent false
21153 5874 Acked Write 1 Done Done Sent false
21154 5875 Acked Read 3 Done Done Sent false
21155 5876 Acked Write 2 Done Done Sent false
21156 5877 Acked Flush 0 Done Done Sent false
21157 5878 Acked Write 3 Done Done Sent false
21158 5879 Acked Write 1 Done Done Sent false
21159 5880 Acked Write 10 Done Done Sent false
21160 5881 Acked Read 5 Done Done Sent false
21161 5882 Acked Flush 0 Done Done Sent false
21162 5883 Acked Read 6 Done Done Sent false
21163 5884 Acked Write 6 Done Done Sent false
21164 5885 Acked Write 3 Done Done Sent false
21165 5886 Acked Flush 0 Done Done Sent false
21166 5887 Acked Flush 0 Done Done Sent false
21167 5888 Acked Read 9 Done Done Sent false
21168 5889 Acked Write 10 Done Done Sent false
21169 5890 Acked Read 6 Done Done Sent false
21170 5891 Acked Read 1 Done Done Sent false
21171 5892 Acked Read 3 Done Done Sent false
21172 5893 Acked Flush 0 Done Done Sent false
21173 5894 Acked Read 10 Done Done Sent false
21174 5895 Acked Read 2 Done Done Sent false
21175 5896 Acked Write 2 Done Done Sent false
21176 5897 Acked Read 8 Done Done Sent false
21177 5898 Acked Write 10 Done Done Sent false
21178 5899 Acked Read 1 Done Done Sent false
21179 5900 Acked Write 1 Done Done Sent false
21180 5901 Acked Flush 0 Done Done Sent false
21181 5902 Acked Write 8 Done Done Sent false
21182 5903 Acked Read 10 Done Done Sent false
21183 5904 Acked Read 9 Done Done Sent false
21184 5905 Acked Read 2 Done Done Sent false
21185 5906 Acked Read 10 Done Done Sent false
21186 5907 Acked Write 2 Done Done Sent false
21187 5908 Acked Write 9 Done Done Sent false
21188 5909 Acked Flush 0 Done Done Sent false
21189 5910 Acked Write 1 Done Done Sent false
21190 5911 Acked Write 9 Done Done Sent false
21191 5912 Acked Read 1 Done Done Sent false
21192 5913 Acked Read 5 Done Done Sent false
21193 5914 Acked Write 8 Done Done Sent false
21194 5915 Acked Read 9 Done Done Sent false
21195 5916 Acked Flush 0 Done Done Sent false
21196 5917 Acked Write 1 Done Done Sent false
21197 5918 Acked Write 2 Done Done Sent false
21198 5919 Acked Read 6 Done Done Sent false
21199 5920 Acked Write 1 Done Done Sent false
21200 5921 Acked Write 4 Done Done Sent false
21201 5922 Acked Write 5 Done Done Sent false
21202 5923 Acked Write 2 Done Done Sent false
21203 5924 Acked Flush 0 Done Done Sent false
21204 5925 Acked Read 5 Done Done Sent false
21205 5926 Acked Write 4 Done Done Sent false
21206 5927 Acked Flush 0 Done Done Sent false
21207 5928 Acked Read 5 Done Done Sent false
21208 5929 Acked Write 1 Done Done Sent false
21209 5930 Acked Read 4 Done Done Sent false
21210 5931 Acked Write 5 Done Done Sent false
21211 5932 Acked Read 4 Done Done Sent false
21212 5933 Acked Write 1 Done Done Sent false
21213 5934 Acked Write 5 Done Done Sent false
21214 5935 Acked Read 4 Done Done Sent false
21215 5936 Acked Read 7 Done Done Sent false
21216 5937 Acked Write 5 Done Done Sent false
21217 5938 Acked Read 8 Done Done Sent false
21218 5939 Acked Write 8 Done Done Sent false
21219 5940 Acked Read 3 Done Done Sent false
21220 5941 Acked Write 7 Done Done Sent false
21221 5942 Acked Read 5 Done Done Sent false
21222 5943 Acked Write 10 Done Done Sent false
21223 5944 Acked Read 1 Done Done Sent false
21224 5945 Acked Write 3 Done Done Sent false
21225 5946 Acked Read 5 Done Done Sent false
21226 5947 Acked Read 2 Done Done Sent false
21227 5948 Acked Read 10 Done Done Sent false
21228 5949 Acked Read 6 Done Done Sent false
21229 5950 Acked Read 9 Done Done Sent false
21230 5951 Acked Write 3 Done Done Sent false
21231 5952 Acked Write 5 Done Done Sent false
21232 5953 Acked Write 7 Done Done Sent false
21233 5954 Acked Flush 0 Done Done Sent false
21234 5955 Acked Read 3 Done Done Sent false
21235 5956 Acked Flush 0 Done Done Sent false
21236 5957 Acked Write 9 Done Done Sent false
21237 5958 Acked Read 4 Done Done Sent false
21238 5959 Acked Write 5 Done Done Sent false
21239 5960 Acked Read 8 Done Done Sent false
21240 5961 Acked Flush 0 Done Done Sent false
21241 5962 Acked Read 3 Done Done Sent false
21242 5963 Acked Write 9 Done Done Sent false
21243 5964 Acked Write 3 Done Done Sent false
21244 5965 Acked Write 6 Done Done Sent false
21245 5966 Acked Read 5 Done Done Sent false
21246 5967 Acked Read 10 Done Done Sent false
21247 5968 Acked Read 3 Done Done Sent false
21248 5969 Acked Write 8 Done Done Sent false
21249 5970 Acked Write 4 Done Done Sent false
21250 5971 Acked Write 4 Done Done Sent false
21251 5972 Acked Flush 0 Done Done Sent false
21252 5973 Acked Write 4 Done Done Sent false
21253 5974 Acked Read 5 Done Done Sent false
21254 5975 Acked Write 10 Done Done Sent false
21255 5976 Acked Write 4 Done Done Sent false
21256 5977 Acked Write 8 Done Done Sent false
21257 5978 Acked Read 5 Done Done Sent false
21258 5979 Acked Read 7 Done Done Sent false
21259 5980 Acked Write 2 Done Done Sent false
21260 5981 Acked Flush 0 Done Done Sent false
21261 5982 Acked Write 1 Done Done Sent false
21262 5983 Acked Write 9 Done Done Sent false
21263 5984 Acked Read 3 Done Done Sent false
21264 5985 Acked Read 10 Done Done Sent false
21265 5986 Acked Read 2 Done Done Sent false
21266 5987 Acked Read 5 Done Done Sent false
21267 5988 Acked Read 2 Done Done Sent false
21268 5989 Acked Read 7 Done Done Sent false
21269 5990 Acked Read 4 Done Done Sent false
21270 5991 Acked Read 9 Done Done Sent false
21271 5992 Acked Write 9 Done Done Sent false
21272 5993 Acked Write 8 Done Done Sent false
21273 5994 Acked Write 6 Done Done Sent false
21274 5995 Acked Read 5 Done Done Sent false
21275 5996 Acked Read 3 Done Done Sent false
21276 5997 Acked Read 4 Done Done Sent false
21277 5998 Acked Read 10 Done Done Sent false
21278 5999 Acked Write 5 Done Done Sent false
21279 6000 Acked Read 9 Done Done Sent false
21280 6001 Acked Read 9 Done Done Sent false
21281 6002 Acked Write 1 Done Done Sent false
21282 6003 Acked Write 1 Done Done Sent false
21283 6004 Acked Write 3 Done Done Sent false
21284 6005 Acked Write 9 Done Done Sent false
21285 6006 Acked Write 1 Done Done Sent false
21286 6007 Acked Write 6 Done Done Sent false
21287 6008 Acked Write 5 Done Done Sent false
21288 6009 Acked Read 1 Done Done Sent false
21289 6010 Acked Flush 0 Done Done Sent false
21290 6011 Acked Read 3 Done Done Sent false
21291 6012 Acked Read 7 Done Done Sent false
21292 6013 Acked Read 1 Done Done Sent false
21293 6014 Acked Write 7 Done Done Sent false
21294 6015 Acked Read 2 Done Done Sent false
21295 6016 Acked Read 9 Done Done Sent false
21296 6017 Acked Read 6 Done Done Sent false
21297 6018 Acked Write 1 Done Done Sent false
21298 6019 Acked Write 5 Done Done Sent false
21299 6020 Acked Write 9 Done Done Sent false
21300 6021 Acked Read 8 Done Done Sent false
21301 6022 Acked Flush 0 Done Done Sent false
21302 6023 Acked Read 5 Done Done Sent false
21303 6024 Acked Read 7 Done Done Sent false
21304 6025 Acked Read 6 Done Done Sent false
21305 6026 Acked Read 5 Done Done Sent false
21306 6027 Acked Write 6 Done Done Sent false
21307 6028 Acked Read 6 Done Done Sent false
21308 6029 Acked Write 2 Done Done Sent false
21309 6030 Acked Write 5 Done Done Sent false
21310 6031 Acked Flush 0 Done Done Sent false
21311 6032 Acked Write 8 Done Done Sent false
21312 6033 Acked Read 3 Done Done Sent false
21313 6034 Acked Write 2 Done Done Sent false
21314 6035 Acked Read 8 Done Done Sent false
21315 6036 Acked Read 1 Done Done Sent false
21316 6037 Acked Write 6 Done Done Sent false
21317 6038 Acked Read 8 Done Done Sent false
21318 6039 Acked Flush 0 Done Done Sent false
21319 6040 Acked Read 5 Done Done Sent false
21320 6041 Acked Write 5 Done Done Sent false
21321 6042 Acked Read 10 Done Done Sent false
21322 6043 Acked Read 4 Done Done Sent false
21323 6044 Acked Write 2 Done Done Sent false
21324 6045 Acked Read 6 Done Done Sent false
21325 6046 Acked Read 10 Done Done Sent false
21326 6047 Acked Write 10 Done Done Sent false
21327 6048 Acked Read 9 Done Done Sent false
21328 6049 Acked Read 1 Done Done Sent false
21329 6050 Acked Read 1 Done Done Sent false
21330 6051 Acked Flush 0 Done Done Sent false
21331 6052 Acked Read 7 Done Done Sent false
21332 6053 Acked Write 3 Done Done Sent false
21333 6054 Acked Write 7 Done Done Sent false
21334 6055 Acked Read 1 Done Done Sent false
21335 6056 Acked Write 9 Done Done Sent false
21336 6057 Acked Read 5 Done Done Sent false
21337 6058 Acked Read 1 Done Done Sent false
21338 6059 Acked Read 4 Done Done Sent false
21339 6060 Acked Read 3 Done Done Sent false
21340 6061 Acked Read 1 Done Done Sent false
21341 6062 Acked Read 6 Done Done Sent false
21342 6063 Acked Write 5 Done Done Sent false
21343 6064 Acked Read 7 Done Done Sent false
21344 6065 Acked Read 9 Done Done Sent false
21345 6066 Acked Read 6 Done Done Sent false
21346 6067 Acked Write 2 Done Done Sent false
21347 6068 Acked Write 5 Done Done Sent false
21348 6069 Acked Read 8 Done Done Sent false
21349 6070 Acked Write 3 Done Done Sent false
21350 6071 Acked Write 9 Done Done Sent false
21351 6072 Acked Write 5 Done Done Sent false
21352 6073 Acked Write 6 Done Done Sent false
21353 6074 Acked Read 5 Done Done Sent false
21354 6075 Acked Read 2 Done Done Sent false
21355 6076 Acked Read 6 Done Done Sent false
21356 6077 Acked Read 2 Done Done Sent false
21357 6078 Acked Read 2 Done Done Sent false
21358 6079 Acked Read 8 Done Done Sent false
21359 6080 Acked Flush 0 Done Done Sent false
21360 6081 Acked Read 10 Done Done Sent false
21361 6082 Acked Flush 0 Done Done Sent false
21362 6083 Acked Read 1 Done Done Sent false
21363 6084 Acked Write 9 Done Done Sent false
21364 6085 Acked Write 5 Done Done Sent false
21365 6086 Acked Read 4 Done Done Sent false
21366 6087 Acked Read 9 Done Done Sent false
21367 6088 Acked Read 6 Done Done Sent false
21368 6089 Acked Flush 0 Done Done Sent false
21369 6090 Acked Read 7 Done Done Sent false
21370 6091 Acked Flush 0 Done Done Sent false
21371 6092 Acked Read 3 Done Done Sent false
21372 6093 Acked Read 2 Done Done Sent false
21373 6094 Acked Write 2 Done Done Sent false
21374 6095 Acked Read 1 Done Done Sent false
21375 6096 Acked Flush 0 Done Done Sent false
21376 6097 Acked Read 3 Done Done Sent false
21377 6098 Acked Read 10 Done Done Sent false
21378 6099 Acked Write 6 Done Done Sent false
21379 6100 Acked Write 6 Done Done Sent false
21380 6101 Acked Read 2 Done Done Sent false
21381 6102 Acked Write 6 Done Done Sent false
21382 6103 Acked Read 4 Done Done Sent false
21383 6104 Acked Flush 0 Done Done Sent false
21384 6105 Acked Read 6 Done Done Sent false
21385 6106 Acked Write 3 Done Done Sent false
21386 6107 Acked Write 8 Done Done Sent false
21387 6108 Acked Read 3 Done Done Sent false
21388 6109 Acked Read 4 Done Done Sent false
21389 6110 Acked Write 10 Done Done Sent false
21390 6111 Acked Read 2 Done Done Sent false
21391 6112 Acked Read 9 Done Done Sent false
21392 6113 Acked Write 9 Done Done Sent false
21393 6114 Acked Flush 0 Done Done Sent false
21394 6115 Acked Write 10 Done Done Sent false
21395 6116 Acked Write 7 Done Done Sent false
21396 6117 Acked Flush 0 Done Done Sent false
21397 6118 Acked Write 10 Done Done Sent false
21398 6119 Acked Read 7 Done Done Sent false
21399 6120 Acked Write 4 Done Done Sent false
21400 6121 Acked Write 3 Done Done Sent false
21401 6122 Acked Read 8 Done Done Sent false
21402 6123 Acked Read 4 Done Done Sent false
21403 6124 Acked Read 10 Done Done Sent false
21404 6125 Acked Read 7 Done Done Sent false
21405 6126 Acked Write 4 Done Done Sent false
21406 6127 Acked Write 6 Done Done Sent false
21407 6128 Acked Flush 0 Done Done Sent false
21408 6129 Acked Read 8 Done Done Sent false
21409 6130 Acked Read 3 Done Done Sent false
21410 6131 Acked Write 9 Done Done Sent false
21411 STATES DS:0 DS:1 DS:2 TOTAL
21412 Sent 0 0 301 301
21413 Done 301 301 0 602
21414 Skipped 0 0 0 0
21415 Error 0 0 0 0
21416 Last Flush: Some(JobId(6128)) Some(JobId(6128)) Some(JobId(5830))
21417 Downstairs last five completed: 5830 5829 5828 5827 5826
21418 Upstairs last five completed: 6131 6130 6129 6128 6127
21419 CLIENT: Up:0 ds:301 act:2
214202025-02-07T20:12:02.404ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
214212025-02-07T20:12:03.406ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
214222025-02-07T20:12:03.523ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
214232025-02-07T20:12:03.523ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
214242025-02-07T20:12:03.523ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
214252025-02-07T20:12:03.523ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
214262025-02-07T20:12:03.523ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
214272025-02-07T20:12:04.408ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
214282025-02-07T20:12:05.410ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
21429 [8830] got start action
21430 [8830] I am going to restart
21431 Starting downstairs at port 8830
21432 Make output file at "/tmp/dsc/downstairs-8830.txt"
21433 [8830][2] reports Starting
21434 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1226)
21435 [8830][2] reports Running
21436 ----------------------------------------------------------------
21437 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21438 States: Active Active Offline
21439 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21440 5831 Acked Write 3 Done Done Sent false
21441 5832 Acked Write 2 Done Done Sent false
21442 5833 Acked Write 7 Done Done Sent false
21443 5834 Acked Write 6 Done Done Sent false
21444 5835 Acked Read 9 Done Done Sent false
21445 5836 Acked Write 3 Done Done Sent false
21446 5837 Acked Read 4 Done Done Sent false
21447 5838 Acked Read 10 Done Done Sent false
21448 5839 Acked Read 3 Done Done Sent false
21449 5840 Acked Flush 0 Done Done Sent false
21450 5841 Acked Write 10 Done Done Sent false
21451 5842 Acked Read 3 Done Done Sent false
21452 5843 Acked Write 3 Done Done Sent false
21453 5844 Acked Read 6 Done Done Sent false
21454 5845 Acked Flush 0 Done Done Sent false
21455 5846 Acked Write 5 Done Done Sent false
21456 5847 Acked Read 3 Done Done Sent false
21457 5848 Acked Read 1 Done Done Sent false
21458 5849 Acked Read 7 Done Done Sent false
21459 5850 Acked Read 2 Done Done Sent false
21460 5851 Acked Write 4 Done Done Sent false
21461 5852 Acked Read 10 Done Done Sent false
21462 5853 Acked Write 10 Done Done Sent false
21463 5854 Acked Flush 0 Done Done Sent false
21464 5855 Acked Write 2 Done Done Sent false
21465 5856 Acked Read 8 Done Done Sent false
21466 5857 Acked Read 9 Done Done Sent false
21467 5858 Acked Read 1 Done Done Sent false
21468 5859 Acked Read 2 Done Done Sent false
21469 5860 Acked Write 3 Done Done Sent false
21470 5861 Acked Write 3 Done Done Sent false
21471 5862 Acked Write 8 Done Done Sent false
21472 5863 Acked Flush 0 Done Done Sent false
21473 5864 Acked Read 9 Done Done Sent false
21474 5865 Acked Read 9 Done Done Sent false
21475 5866 Acked Flush 0 Done Done Sent false
21476 5867 Acked Read 2 Done Done Sent false
21477 5868 Acked Write 4 Done Done Sent false
21478 5869 Acked Flush 0 Done Done Sent false
21479 5870 Acked Write 3 Done Done Sent false
21480 5871 Acked Flush 0 Done Done Sent false
21481 5872 Acked Write 9 Done Done Sent false
21482 5873 Acked Read 5 Done Done Sent false
21483 5874 Acked Write 1 Done Done Sent false
21484 5875 Acked Read 3 Done Done Sent false
21485 5876 Acked Write 2 Done Done Sent false
21486 5877 Acked Flush 0 Done Done Sent false
21487 5878 Acked Write 3 Done Done Sent false
21488 5879 Acked Write 1 Done Done Sent false
21489 5880 Acked Write 10 Done Done Sent false
21490 5881 Acked Read 5 Done Done Sent false
21491 5882 Acked Flush 0 Done Done Sent false
21492 5883 Acked Read 6 Done Done Sent false
21493 5884 Acked Write 6 Done Done Sent false
21494 5885 Acked Write 3 Done Done Sent false
21495 5886 Acked Flush 0 Done Done Sent false
21496 5887 Acked Flush 0 Done Done Sent false
21497 5888 Acked Read 9 Done Done Sent false
21498 5889 Acked Write 10 Done Done Sent false
21499 5890 Acked Read 6 Done Done Sent false
21500 5891 Acked Read 1 Done Done Sent false
21501 5892 Acked Read 3 Done Done Sent false
21502 5893 Acked Flush 0 Done Done Sent false
21503 5894 Acked Read 10 Done Done Sent false
21504 5895 Acked Read 2 Done Done Sent false
21505 5896 Acked Write 2 Done Done Sent false
21506 5897 Acked Read 8 Done Done Sent false
21507 5898 Acked Write 10 Done Done Sent false
21508 5899 Acked Read 1 Done Done Sent false
21509 5900 Acked Write 1 Done Done Sent false
21510 5901 Acked Flush 0 Done Done Sent false
21511 5902 Acked Write 8 Done Done Sent false
21512 5903 Acked Read 10 Done Done Sent false
21513 5904 Acked Read 9 Done Done Sent false
21514 5905 Acked Read 2 Done Done Sent false
21515 5906 Acked Read 10 Done Done Sent false
21516 5907 Acked Write 2 Done Done Sent false
21517 5908 Acked Write 9 Done Done Sent false
21518 5909 Acked Flush 0 Done Done Sent false
21519 5910 Acked Write 1 Done Done Sent false
21520 5911 Acked Write 9 Done Done Sent false
21521 5912 Acked Read 1 Done Done Sent false
21522 5913 Acked Read 5 Done Done Sent false
21523 5914 Acked Write 8 Done Done Sent false
21524 5915 Acked Read 9 Done Done Sent false
21525 5916 Acked Flush 0 Done Done Sent false
21526 5917 Acked Write 1 Done Done Sent false
21527 5918 Acked Write 2 Done Done Sent false
21528 5919 Acked Read 6 Done Done Sent false
21529 5920 Acked Write 1 Done Done Sent false
21530 5921 Acked Write 4 Done Done Sent false
21531 5922 Acked Write 5 Done Done Sent false
21532 5923 Acked Write 2 Done Done Sent false
21533 5924 Acked Flush 0 Done Done Sent false
21534 5925 Acked Read 5 Done Done Sent false
21535 5926 Acked Write 4 Done Done Sent false
21536 5927 Acked Flush 0 Done Done Sent false
21537 5928 Acked Read 5 Done Done Sent false
21538 5929 Acked Write 1 Done Done Sent false
21539 5930 Acked Read 4 Done Done Sent false
21540 5931 Acked Write 5 Done Done Sent false
21541 5932 Acked Read 4 Done Done Sent false
21542 5933 Acked Write 1 Done Done Sent false
21543 5934 Acked Write 5 Done Done Sent false
21544 5935 Acked Read 4 Done Done Sent false
21545 5936 Acked Read 7 Done Done Sent false
21546 5937 Acked Write 5 Done Done Sent false
21547 5938 Acked Read 8 Done Done Sent false
21548 5939 Acked Write 8 Done Done Sent false
21549 5940 Acked Read 3 Done Done Sent false
21550 5941 Acked Write 7 Done Done Sent false
21551 5942 Acked Read 5 Done Done Sent false
21552 5943 Acked Write 10 Done Done Sent false
21553 5944 Acked Read 1 Done Done Sent false
21554 5945 Acked Write 3 Done Done Sent false
21555 5946 Acked Read 5 Done Done Sent false
21556 5947 Acked Read 2 Done Done Sent false
21557 5948 Acked Read 10 Done Done Sent false
21558 5949 Acked Read 6 Done Done Sent false
21559 5950 Acked Read 9 Done Done Sent false
21560 5951 Acked Write 3 Done Done Sent false
21561 5952 Acked Write 5 Done Done Sent false
21562 5953 Acked Write 7 Done Done Sent false
21563 5954 Acked Flush 0 Done Done Sent false
21564 5955 Acked Read 3 Done Done Sent false
21565 5956 Acked Flush 0 Done Done Sent false
21566 5957 Acked Write 9 Done Done Sent false
21567 5958 Acked Read 4 Done Done Sent false
21568 5959 Acked Write 5 Done Done Sent false
21569 5960 Acked Read 8 Done Done Sent false
21570 5961 Acked Flush 0 Done Done Sent false
21571 5962 Acked Read 3 Done Done Sent false
21572 5963 Acked Write 9 Done Done Sent false
21573 5964 Acked Write 3 Done Done Sent false
21574 5965 Acked Write 6 Done Done Sent false
21575 5966 Acked Read 5 Done Done Sent false
21576 5967 Acked Read 10 Done Done Sent false
21577 5968 Acked Read 3 Done Done Sent false
21578 5969 Acked Write 8 Done Done Sent false
21579 5970 Acked Write 4 Done Done Sent false
21580 5971 Acked Write 4 Done Done Sent false
21581 5972 Acked Flush 0 Done Done Sent false
21582 5973 Acked Write 4 Done Done Sent false
21583 5974 Acked Read 5 Done Done Sent false
21584 5975 Acked Write 10 Done Done Sent false
21585 5976 Acked Write 4 Done Done Sent false
21586 5977 Acked Write 8 Done Done Sent false
21587 5978 Acked Read 5 Done Done Sent false
21588 5979 Acked Read 7 Done Done Sent false
21589 5980 Acked Write 2 Done Done Sent false
21590 5981 Acked Flush 0 Done Done Sent false
21591 5982 Acked Write 1 Done Done Sent false
21592 5983 Acked Write 9 Done Done Sent false
21593 5984 Acked Read 3 Done Done Sent false
21594 5985 Acked Read 10 Done Done Sent false
21595 5986 Acked Read 2 Done Done Sent false
21596 5987 Acked Read 5 Done Done Sent false
21597 5988 Acked Read 2 Done Done Sent false
21598 5989 Acked Read 7 Done Done Sent false
21599 5990 Acked Read 4 Done Done Sent false
21600 5991 Acked Read 9 Done Done Sent false
21601 5992 Acked Write 9 Done Done Sent false
21602 5993 Acked Write 8 Done Done Sent false
21603 5994 Acked Write 6 Done Done Sent false
21604 5995 Acked Read 5 Done Done Sent false
21605 5996 Acked Read 3 Done Done Sent false
21606 5997 Acked Read 4 Done Done Sent false
21607 5998 Acked Read 10 Done Done Sent false
21608 5999 Acked Write 5 Done Done Sent false
21609 6000 Acked Read 9 Done Done Sent false
21610 6001 Acked Read 9 Done Done Sent false
21611 6002 Acked Write 1 Done Done Sent false
21612 6003 Acked Write 1 Done Done Sent false
21613 6004 Acked Write 3 Done Done Sent false
21614 6005 Acked Write 9 Done Done Sent false
21615 6006 Acked Write 1 Done Done Sent false
21616 6007 Acked Write 6 Done Done Sent false
21617 6008 Acked Write 5 Done Done Sent false
21618 6009 Acked Read 1 Done Done Sent false
21619 6010 Acked Flush 0 Done Done Sent false
21620 6011 Acked Read 3 Done Done Sent false
21621 6012 Acked Read 7 Done Done Sent false
21622 6013 Acked Read 1 Done Done Sent false
21623 6014 Acked Write 7 Done Done Sent false
21624 6015 Acked Read 2 Done Done Sent false
21625 6016 Acked Read 9 Done Done Sent false
21626 6017 Acked Read 6 Done Done Sent false
21627 6018 Acked Write 1 Done Done Sent false
21628 6019 Acked Write 5 Done Done Sent false
21629 6020 Acked Write 9 Done Done Sent false
21630 6021 Acked Read 8 Done Done Sent false
21631 6022 Acked Flush 0 Done Done Sent false
21632 6023 Acked Read 5 Done Done Sent false
21633 6024 Acked Read 7 Done Done Sent false
21634 6025 Acked Read 6 Done Done Sent false
21635 6026 Acked Read 5 Done Done Sent false
21636 6027 Acked Write 6 Done Done Sent false
21637 6028 Acked Read 6 Done Done Sent false
21638 6029 Acked Write 2 Done Done Sent false
21639 6030 Acked Write 5 Done Done Sent false
21640 6031 Acked Flush 0 Done Done Sent false
21641 6032 Acked Write 8 Done Done Sent false
21642 6033 Acked Read 3 Done Done Sent false
21643 6034 Acked Write 2 Done Done Sent false
21644 6035 Acked Read 8 Done Done Sent false
21645 6036 Acked Read 1 Done Done Sent false
21646 6037 Acked Write 6 Done Done Sent false
21647 6038 Acked Read 8 Done Done Sent false
21648 6039 Acked Flush 0 Done Done Sent false
21649 6040 Acked Read 5 Done Done Sent false
21650 6041 Acked Write 5 Done Done Sent false
21651 6042 Acked Read 10 Done Done Sent false
21652 6043 Acked Read 4 Done Done Sent false
21653 6044 Acked Write 2 Done Done Sent false
21654 6045 Acked Read 6 Done Done Sent false
21655 6046 Acked Read 10 Done Done Sent false
21656 6047 Acked Write 10 Done Done Sent false
21657 6048 Acked Read 9 Done Done Sent false
21658 6049 Acked Read 1 Done Done Sent false
21659 6050 Acked Read 1 Done Done Sent false
21660 6051 Acked Flush 0 Done Done Sent false
21661 6052 Acked Read 7 Done Done Sent false
21662 6053 Acked Write 3 Done Done Sent false
21663 6054 Acked Write 7 Done Done Sent false
21664 6055 Acked Read 1 Done Done Sent false
21665 6056 Acked Write 9 Done Done Sent false
21666 6057 Acked Read 5 Done Done Sent false
21667 6058 Acked Read 1 Done Done Sent false
21668 6059 Acked Read 4 Done Done Sent false
21669 6060 Acked Read 3 Done Done Sent false
21670 6061 Acked Read 1 Done Done Sent false
21671 6062 Acked Read 6 Done Done Sent false
21672 6063 Acked Write 5 Done Done Sent false
21673 6064 Acked Read 7 Done Done Sent false
21674 6065 Acked Read 9 Done Done Sent false
21675 6066 Acked Read 6 Done Done Sent false
21676 6067 Acked Write 2 Done Done Sent false
21677 6068 Acked Write 5 Done Done Sent false
21678 6069 Acked Read 8 Done Done Sent false
21679 6070 Acked Write 3 Done Done Sent false
21680 6071 Acked Write 9 Done Done Sent false
21681 6072 Acked Write 5 Done Done Sent false
21682 6073 Acked Write 6 Done Done Sent false
21683 6074 Acked Read 5 Done Done Sent false
21684 6075 Acked Read 2 Done Done Sent false
21685 6076 Acked Read 6 Done Done Sent false
21686 6077 Acked Read 2 Done Done Sent false
21687 6078 Acked Read 2 Done Done Sent false
21688 6079 Acked Read 8 Done Done Sent false
21689 6080 Acked Flush 0 Done Done Sent false
21690 6081 Acked Read 10 Done Done Sent false
21691 6082 Acked Flush 0 Done Done Sent false
21692 6083 Acked Read 1 Done Done Sent false
21693 6084 Acked Write 9 Done Done Sent false
21694 6085 Acked Write 5 Done Done Sent false
21695 6086 Acked Read 4 Done Done Sent false
21696 6087 Acked Read 9 Done Done Sent false
21697 6088 Acked Read 6 Done Done Sent false
21698 6089 Acked Flush 0 Done Done Sent false
21699 6090 Acked Read 7 Done Done Sent false
21700 6091 Acked Flush 0 Done Done Sent false
21701 6092 Acked Read 3 Done Done Sent false
21702 6093 Acked Read 2 Done Done Sent false
21703 6094 Acked Write 2 Done Done Sent false
21704 6095 Acked Read 1 Done Done Sent false
21705 6096 Acked Flush 0 Done Done Sent false
21706 6097 Acked Read 3 Done Done Sent false
21707 6098 Acked Read 10 Done Done Sent false
21708 6099 Acked Write 6 Done Done Sent false
21709 6100 Acked Write 6 Done Done Sent false
21710 6101 Acked Read 2 Done Done Sent false
21711 6102 Acked Write 6 Done Done Sent false
21712 6103 Acked Read 4 Done Done Sent false
21713 6104 Acked Flush 0 Done Done Sent false
21714 6105 Acked Read 6 Done Done Sent false
21715 6106 Acked Write 3 Done Done Sent false
21716 6107 Acked Write 8 Done Done Sent false
21717 6108 Acked Read 3 Done Done Sent false
21718 6109 Acked Read 4 Done Done Sent false
21719 6110 Acked Write 10 Done Done Sent false
21720 6111 Acked Read 2 Done Done Sent false
21721 6112 Acked Read 9 Done Done Sent false
21722 6113 Acked Write 9 Done Done Sent false
21723 6114 Acked Flush 0 Done Done Sent false
21724 6115 Acked Write 10 Done Done Sent false
21725 6116 Acked Write 7 Done Done Sent false
21726 6117 Acked Flush 0 Done Done Sent false
21727 6118 Acked Write 10 Done Done Sent false
21728 6119 Acked Read 7 Done Done Sent false
21729 6120 Acked Write 4 Done Done Sent false
21730 6121 Acked Write 3 Done Done Sent false
21731 6122 Acked Read 8 Done Done Sent false
21732 6123 Acked Read 4 Done Done Sent false
21733 6124 Acked Read 10 Done Done Sent false
21734 6125 Acked Read 7 Done Done Sent false
21735 6126 Acked Write 4 Done Done Sent false
21736 6127 Acked Write 6 Done Done Sent false
21737 6128 Acked Flush 0 Done Done Sent false
21738 6129 Acked Read 8 Done Done Sent false
21739 6130 Acked Read 3 Done Done Sent false
21740 6131 Acked Write 9 Done Done Sent false
21741 STATES DS:0 DS:1 DS:2 TOTAL
21742 Sent 0 0 301 301
21743 Done 301 301 0 602
21744 Skipped 0 0 0 0
21745 Error 0 0 0 0
21746 Last Flush: Some(JobId(6128)) Some(JobId(6128)) Some(JobId(5830))
21747 Downstairs last five completed: 5830 5829 5828 5827 5826
21748 Upstairs last five completed: 6131 6130 6129 6128 6127
21749 CLIENT: Up:0 ds:301 act:2
217502025-02-07T20:12:06.779ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
217512025-02-07T20:12:07.414ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
217522025-02-07T20:12:08.417ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
217532025-02-07T20:12:09.417ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
21754 ----------------------------------------------------------------
21755 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21756 States: Active Active Offline
21757 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21758 5831 Acked Write 3 Done Done Sent false
21759 5832 Acked Write 2 Done Done Sent false
21760 5833 Acked Write 7 Done Done Sent false
21761 5834 Acked Write 6 Done Done Sent false
21762 5835 Acked Read 9 Done Done Sent false
21763 5836 Acked Write 3 Done Done Sent false
21764 5837 Acked Read 4 Done Done Sent false
21765 5838 Acked Read 10 Done Done Sent false
21766 5839 Acked Read 3 Done Done Sent false
21767 5840 Acked Flush 0 Done Done Sent false
21768 5841 Acked Write 10 Done Done Sent false
21769 5842 Acked Read 3 Done Done Sent false
21770 5843 Acked Write 3 Done Done Sent false
21771 5844 Acked Read 6 Done Done Sent false
21772 5845 Acked Flush 0 Done Done Sent false
21773 5846 Acked Write 5 Done Done Sent false
21774 5847 Acked Read 3 Done Done Sent false
21775 5848 Acked Read 1 Done Done Sent false
21776 5849 Acked Read 7 Done Done Sent false
21777 5850 Acked Read 2 Done Done Sent false
21778 5851 Acked Write 4 Done Done Sent false
21779 5852 Acked Read 10 Done Done Sent false
21780 5853 Acked Write 10 Done Done Sent false
21781 5854 Acked Flush 0 Done Done Sent false
21782 5855 Acked Write 2 Done Done Sent false
21783 5856 Acked Read 8 Done Done Sent false
21784 5857 Acked Read 9 Done Done Sent false
21785 5858 Acked Read 1 Done Done Sent false
21786 5859 Acked Read 2 Done Done Sent false
21787 5860 Acked Write 3 Done Done Sent false
21788 5861 Acked Write 3 Done Done Sent false
21789 5862 Acked Write 8 Done Done Sent false
21790 5863 Acked Flush 0 Done Done Sent false
21791 5864 Acked Read 9 Done Done Sent false
21792 5865 Acked Read 9 Done Done Sent false
21793 5866 Acked Flush 0 Done Done Sent false
21794 5867 Acked Read 2 Done Done Sent false
21795 5868 Acked Write 4 Done Done Sent false
21796 5869 Acked Flush 0 Done Done Sent false
21797 5870 Acked Write 3 Done Done Sent false
21798 5871 Acked Flush 0 Done Done Sent false
21799 5872 Acked Write 9 Done Done Sent false
21800 5873 Acked Read 5 Done Done Sent false
21801 5874 Acked Write 1 Done Done Sent false
21802 5875 Acked Read 3 Done Done Sent false
21803 5876 Acked Write 2 Done Done Sent false
21804 5877 Acked Flush 0 Done Done Sent false
21805 5878 Acked Write 3 Done Done Sent false
21806 5879 Acked Write 1 Done Done Sent false
21807 5880 Acked Write 10 Done Done Sent false
21808 5881 Acked Read 5 Done Done Sent false
21809 5882 Acked Flush 0 Done Done Sent false
21810 5883 Acked Read 6 Done Done Sent false
21811 5884 Acked Write 6 Done Done Sent false
21812 5885 Acked Write 3 Done Done Sent false
21813 5886 Acked Flush 0 Done Done Sent false
21814 5887 Acked Flush 0 Done Done Sent false
21815 5888 Acked Read 9 Done Done Sent false
21816 5889 Acked Write 10 Done Done Sent false
21817 5890 Acked Read 6 Done Done Sent false
21818 5891 Acked Read 1 Done Done Sent false
21819 5892 Acked Read 3 Done Done Sent false
21820 5893 Acked Flush 0 Done Done Sent false
21821 5894 Acked Read 10 Done Done Sent false
21822 5895 Acked Read 2 Done Done Sent false
21823 5896 Acked Write 2 Done Done Sent false
21824 5897 Acked Read 8 Done Done Sent false
21825 5898 Acked Write 10 Done Done Sent false
21826 5899 Acked Read 1 Done Done Sent false
21827 5900 Acked Write 1 Done Done Sent false
21828 5901 Acked Flush 0 Done Done Sent false
21829 5902 Acked Write 8 Done Done Sent false
21830 5903 Acked Read 10 Done Done Sent false
21831 5904 Acked Read 9 Done Done Sent false
21832 5905 Acked Read 2 Done Done Sent false
21833 5906 Acked Read 10 Done Done Sent false
21834 5907 Acked Write 2 Done Done Sent false
21835 5908 Acked Write 9 Done Done Sent false
21836 5909 Acked Flush 0 Done Done Sent false
21837 5910 Acked Write 1 Done Done Sent false
21838 5911 Acked Write 9 Done Done Sent false
21839 5912 Acked Read 1 Done Done Sent false
21840 5913 Acked Read 5 Done Done Sent false
21841 5914 Acked Write 8 Done Done Sent false
21842 5915 Acked Read 9 Done Done Sent false
21843 5916 Acked Flush 0 Done Done Sent false
21844 5917 Acked Write 1 Done Done Sent false
21845 5918 Acked Write 2 Done Done Sent false
21846 5919 Acked Read 6 Done Done Sent false
21847 5920 Acked Write 1 Done Done Sent false
21848 5921 Acked Write 4 Done Done Sent false
21849 5922 Acked Write 5 Done Done Sent false
21850 5923 Acked Write 2 Done Done Sent false
21851 5924 Acked Flush 0 Done Done Sent false
21852 5925 Acked Read 5 Done Done Sent false
21853 5926 Acked Write 4 Done Done Sent false
21854 5927 Acked Flush 0 Done Done Sent false
21855 5928 Acked Read 5 Done Done Sent false
21856 5929 Acked Write 1 Done Done Sent false
21857 5930 Acked Read 4 Done Done Sent false
21858 5931 Acked Write 5 Done Done Sent false
21859 5932 Acked Read 4 Done Done Sent false
21860 5933 Acked Write 1 Done Done Sent false
21861 5934 Acked Write 5 Done Done Sent false
21862 5935 Acked Read 4 Done Done Sent false
21863 5936 Acked Read 7 Done Done Sent false
21864 5937 Acked Write 5 Done Done Sent false
21865 5938 Acked Read 8 Done Done Sent false
21866 5939 Acked Write 8 Done Done Sent false
21867 5940 Acked Read 3 Done Done Sent false
21868 5941 Acked Write 7 Done Done Sent false
21869 5942 Acked Read 5 Done Done Sent false
21870 5943 Acked Write 10 Done Done Sent false
21871 5944 Acked Read 1 Done Done Sent false
21872 5945 Acked Write 3 Done Done Sent false
21873 5946 Acked Read 5 Done Done Sent false
21874 5947 Acked Read 2 Done Done Sent false
21875 5948 Acked Read 10 Done Done Sent false
21876 5949 Acked Read 6 Done Done Sent false
21877 5950 Acked Read 9 Done Done Sent false
21878 5951 Acked Write 3 Done Done Sent false
21879 5952 Acked Write 5 Done Done Sent false
21880 5953 Acked Write 7 Done Done Sent false
21881 5954 Acked Flush 0 Done Done Sent false
21882 5955 Acked Read 3 Done Done Sent false
21883 5956 Acked Flush 0 Done Done Sent false
21884 5957 Acked Write 9 Done Done Sent false
21885 5958 Acked Read 4 Done Done Sent false
21886 5959 Acked Write 5 Done Done Sent false
21887 5960 Acked Read 8 Done Done Sent false
21888 5961 Acked Flush 0 Done Done Sent false
21889 5962 Acked Read 3 Done Done Sent false
21890 5963 Acked Write 9 Done Done Sent false
21891 5964 Acked Write 3 Done Done Sent false
21892 5965 Acked Write 6 Done Done Sent false
21893 5966 Acked Read 5 Done Done Sent false
21894 5967 Acked Read 10 Done Done Sent false
21895 5968 Acked Read 3 Done Done Sent false
21896 5969 Acked Write 8 Done Done Sent false
21897 5970 Acked Write 4 Done Done Sent false
21898 5971 Acked Write 4 Done Done Sent false
21899 5972 Acked Flush 0 Done Done Sent false
21900 5973 Acked Write 4 Done Done Sent false
21901 5974 Acked Read 5 Done Done Sent false
21902 5975 Acked Write 10 Done Done Sent false
21903 5976 Acked Write 4 Done Done Sent false
21904 5977 Acked Write 8 Done Done Sent false
21905 5978 Acked Read 5 Done Done Sent false
21906 5979 Acked Read 7 Done Done Sent false
21907 5980 Acked Write 2 Done Done Sent false
21908 5981 Acked Flush 0 Done Done Sent false
21909 5982 Acked Write 1 Done Done Sent false
21910 5983 Acked Write 9 Done Done Sent false
21911 5984 Acked Read 3 Done Done Sent false
21912 5985 Acked Read 10 Done Done Sent false
21913 5986 Acked Read 2 Done Done Sent false
21914 5987 Acked Read 5 Done Done Sent false
21915 5988 Acked Read 2 Done Done Sent false
21916 5989 Acked Read 7 Done Done Sent false
21917 5990 Acked Read 4 Done Done Sent false
21918 5991 Acked Read 9 Done Done Sent false
21919 5992 Acked Write 9 Done Done Sent false
21920 5993 Acked Write 8 Done Done Sent false
21921 5994 Acked Write 6 Done Done Sent false
21922 5995 Acked Read 5 Done Done Sent false
21923 5996 Acked Read 3 Done Done Sent false
21924 5997 Acked Read 4 Done Done Sent false
21925 5998 Acked Read 10 Done Done Sent false
21926 5999 Acked Write 5 Done Done Sent false
21927 6000 Acked Read 9 Done Done Sent false
21928 6001 Acked Read 9 Done Done Sent false
21929 6002 Acked Write 1 Done Done Sent false
21930 6003 Acked Write 1 Done Done Sent false
21931 6004 Acked Write 3 Done Done Sent false
21932 6005 Acked Write 9 Done Done Sent false
21933 6006 Acked Write 1 Done Done Sent false
21934 6007 Acked Write 6 Done Done Sent false
21935 6008 Acked Write 5 Done Done Sent false
21936 6009 Acked Read 1 Done Done Sent false
21937 6010 Acked Flush 0 Done Done Sent false
21938 6011 Acked Read 3 Done Done Sent false
21939 6012 Acked Read 7 Done Done Sent false
21940 6013 Acked Read 1 Done Done Sent false
21941 6014 Acked Write 7 Done Done Sent false
21942 6015 Acked Read 2 Done Done Sent false
21943 6016 Acked Read 9 Done Done Sent false
21944 6017 Acked Read 6 Done Done Sent false
21945 6018 Acked Write 1 Done Done Sent false
21946 6019 Acked Write 5 Done Done Sent false
21947 6020 Acked Write 9 Done Done Sent false
21948 6021 Acked Read 8 Done Done Sent false
21949 6022 Acked Flush 0 Done Done Sent false
21950 6023 Acked Read 5 Done Done Sent false
21951 6024 Acked Read 7 Done Done Sent false
21952 6025 Acked Read 6 Done Done Sent false
21953 6026 Acked Read 5 Done Done Sent false
21954 6027 Acked Write 6 Done Done Sent false
21955 6028 Acked Read 6 Done Done Sent false
21956 6029 Acked Write 2 Done Done Sent false
21957 6030 Acked Write 5 Done Done Sent false
21958 6031 Acked Flush 0 Done Done Sent false
21959 6032 Acked Write 8 Done Done Sent false
21960 6033 Acked Read 3 Done Done Sent false
21961 6034 Acked Write 2 Done Done Sent false
21962 6035 Acked Read 8 Done Done Sent false
21963 6036 Acked Read 1 Done Done Sent false
21964 6037 Acked Write 6 Done Done Sent false
21965 6038 Acked Read 8 Done Done Sent false
21966 6039 Acked Flush 0 Done Done Sent false
21967 6040 Acked Read 5 Done Done Sent false
21968 6041 Acked Write 5 Done Done Sent false
21969 6042 Acked Read 10 Done Done Sent false
21970 6043 Acked Read 4 Done Done Sent false
21971 6044 Acked Write 2 Done Done Sent false
21972 6045 Acked Read 6 Done Done Sent false
21973 6046 Acked Read 10 Done Done Sent false
21974 6047 Acked Write 10 Done Done Sent false
21975 6048 Acked Read 9 Done Done Sent false
21976 6049 Acked Read 1 Done Done Sent false
21977 6050 Acked Read 1 Done Done Sent false
21978 6051 Acked Flush 0 Done Done Sent false
21979 6052 Acked Read 7 Done Done Sent false
21980 6053 Acked Write 3 Done Done Sent false
21981 6054 Acked Write 7 Done Done Sent false
21982 6055 Acked Read 1 Done Done Sent false
21983 6056 Acked Write 9 Done Done Sent false
21984 6057 Acked Read 5 Done Done Sent false
21985 6058 Acked Read 1 Done Done Sent false
21986 6059 Acked Read 4 Done Done Sent false
21987 6060 Acked Read 3 Done Done Sent false
21988 6061 Acked Read 1 Done Done Sent false
21989 6062 Acked Read 6 Done Done Sent false
21990 6063 Acked Write 5 Done Done Sent false
21991 6064 Acked Read 7 Done Done Sent false
21992 6065 Acked Read 9 Done Done Sent false
21993 6066 Acked Read 6 Done Done Sent false
21994 6067 Acked Write 2 Done Done Sent false
21995 6068 Acked Write 5 Done Done Sent false
21996 6069 Acked Read 8 Done Done Sent false
21997 6070 Acked Write 3 Done Done Sent false
21998 6071 Acked Write 9 Done Done Sent false
21999 6072 Acked Write 5 Done Done Sent false
22000 6073 Acked Write 6 Done Done Sent false
22001 6074 Acked Read 5 Done Done Sent false
22002 6075 Acked Read 2 Done Done Sent false
22003 6076 Acked Read 6 Done Done Sent false
22004 6077 Acked Read 2 Done Done Sent false
22005 6078 Acked Read 2 Done Done Sent false
22006 6079 Acked Read 8 Done Done Sent false
22007 6080 Acked Flush 0 Done Done Sent false
22008 6081 Acked Read 10 Done Done Sent false
22009 6082 Acked Flush 0 Done Done Sent false
22010 6083 Acked Read 1 Done Done Sent false
22011 6084 Acked Write 9 Done Done Sent false
22012 6085 Acked Write 5 Done Done Sent false
22013 6086 Acked Read 4 Done Done Sent false
22014 6087 Acked Read 9 Done Done Sent false
22015 6088 Acked Read 6 Done Done Sent false
22016 6089 Acked Flush 0 Done Done Sent false
22017 6090 Acked Read 7 Done Done Sent false
22018 6091 Acked Flush 0 Done Done Sent false
22019 6092 Acked Read 3 Done Done Sent false
22020 6093 Acked Read 2 Done Done Sent false
22021 6094 Acked Write 2 Done Done Sent false
22022 6095 Acked Read 1 Done Done Sent false
22023 6096 Acked Flush 0 Done Done Sent false
22024 6097 Acked Read 3 Done Done Sent false
22025 6098 Acked Read 10 Done Done Sent false
22026 6099 Acked Write 6 Done Done Sent false
22027 6100 Acked Write 6 Done Done Sent false
22028 6101 Acked Read 2 Done Done Sent false
22029 6102 Acked Write 6 Done Done Sent false
22030 6103 Acked Read 4 Done Done Sent false
22031 6104 Acked Flush 0 Done Done Sent false
22032 6105 Acked Read 6 Done Done Sent false
22033 6106 Acked Write 3 Done Done Sent false
22034 6107 Acked Write 8 Done Done Sent false
22035 6108 Acked Read 3 Done Done Sent false
22036 6109 Acked Read 4 Done Done Sent false
22037 6110 Acked Write 10 Done Done Sent false
22038 6111 Acked Read 2 Done Done Sent false
22039 6112 Acked Read 9 Done Done Sent false
22040 6113 Acked Write 9 Done Done Sent false
22041 6114 Acked Flush 0 Done Done Sent false
22042 6115 Acked Write 10 Done Done Sent false
22043 6116 Acked Write 7 Done Done Sent false
22044 6117 Acked Flush 0 Done Done Sent false
22045 6118 Acked Write 10 Done Done Sent false
22046 6119 Acked Read 7 Done Done Sent false
22047 6120 Acked Write 4 Done Done Sent false
22048 6121 Acked Write 3 Done Done Sent false
22049 6122 Acked Read 8 Done Done Sent false
22050 6123 Acked Read 4 Done Done Sent false
22051 6124 Acked Read 10 Done Done Sent false
22052 6125 Acked Read 7 Done Done Sent false
22053 6126 Acked Write 4 Done Done Sent false
22054 6127 Acked Write 6 Done Done Sent false
22055 6128 Acked Flush 0 Done Done Sent false
22056 6129 Acked Read 8 Done Done Sent false
22057 6130 Acked Read 3 Done Done Sent false
22058 6131 Acked Write 9 Done Done Sent false
22059 STATES DS:0 DS:1 DS:2 TOTAL
22060 Sent 0 0 301 301
22061 Done 301 301 0 602
22062 Skipped 0 0 0 0
22063 Error 0 0 0 0
22064 Last Flush: Some(JobId(6128)) Some(JobId(6128)) Some(JobId(5830))
22065 Downstairs last five completed: 5830 5829 5828 5827 5826
22066 Upstairs last five completed: 6131 6130 6129 6128 6127
22067 CLIENT: Up:0 ds:301 act:2
220682025-02-07T20:12:10.418ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220692025-02-07T20:12:11.780ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220702025-02-07T20:12:12.421ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220712025-02-07T20:12:13.423ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220722025-02-07T20:12:13.524ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220732025-02-07T20:12:13.524ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59629) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220742025-02-07T20:12:13.525ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220752025-02-07T20:12:13.526ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220762025-02-07T20:12:13.526ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220772025-02-07T20:12:13.526ZINFOcrucible: send last flush ID to this DS: Some(JobId(5830)) = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220782025-02-07T20:12:13.526ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220792025-02-07T20:12:13.526ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(5830)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
220802025-02-07T20:12:13.526ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(5830)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
22081 ----------------------------------------------------------------
22082 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:3
22083 States: Active Active Active
22084 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22085 6129 Acked Read 8 Done Done Done true
22086 6130 Acked Read 3 Done Done Done true
22087 6131 Acked Write 9 Done Done Done true
22088 STATES DS:0 DS:1 DS:2 TOTAL
22089 Sent 0 0 0 0
22090 Done 3 3 3 9
22091 Skipped 0 0 0 0
22092 Error 0 0 0 0
22093 Last Flush: Some(JobId(6128)) Some(JobId(6128)) Some(JobId(6128))
22094 Downstairs last five completed: 6128 6127 6126 6125 6124
22095 Upstairs last five completed: 6131 6130 6129 6128 6127
22096 CLIENT: Up:0 ds:3 act:3
22097 ----------------------------------------------------------------
22098 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22099 Downstairs last five completed: 6132 6131 6130 6129 6128
22100 Upstairs last five completed: 6132 6131 6130 6129 6128
22101 CLIENT: Up:0 ds:0 act:3
22102 Replay: All jobs finished
22103 Main task has work to do, go find it
22104 got dsc Stop(0)
22105 stop 0
22106 Feb 07 20:12:17.772 INFO request completed[8810] Got stop action so:false kr:false
22107 , latency_us: 367, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 41299a85-fd45-4e30-91c6-9624323880f4, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
221082025-02-07T20:12:17.772ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
221092025-02-07T20:12:17.772ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
221102025-02-07T20:12:17.772ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
22111 Feb 07 20:12:17.773 INFO request completed, latency_us: 166, response_code: 200, uri: /state/cid/0, method: GET, req_id{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-07T20:12:17.77313904Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
22112 : 7cfc70ee-0584-4f48-af6e-049e77259fb6, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
22113 [8810] Exited with: ExitStatus(unix_wait_status(9))
22114 [8810][0] reports Exit
22115 Feb 07 20:12:21.775 INFO request completed, latency_us: 330, response_code: 200, uri: /state/cid/0, method: GET, req_id: 3932ed87-ae16-49a7-86be-723c1dfe21e5, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
22116 001/300 Read block 2997 len 8192
22117 002/300 Write block 4156 len 24576 data: 3 3 5 5 5 3
22118 003/300 Read block 2074 len 24576
22119 004/300 Read block 4363 len 28672
22120 005/300 Read block 4860 len 4096
22121 006/300 Read block 732 len 36864
22122 007/300 Read block 4525 len 16384
22123 008/300 Flush
22124 009/300 Read block 1278 len 28672
22125 010/300 Read block 1347 len 20480
22126 011/300 Write block 3584 len 32768 data: 3 3 3 3 3 4 3 3
22127 012/300 Write block 3044 len 40960 data: 7 6 5 4 5 7 7 7 9 6
22128 013/300 Write block 2406 len 8192 data: 2 2
22129 014/300 Read block 3064 len 24576
22130 015/300 Flush
22131 016/300 Read block 3305 len 24576
22132 017/300 Write block 4649 len 36864 data: 5 6 6 4 4 4 4 3 3
22133 018/300 Write block 1045 len 4096 data: 6
22134 019/300 Read block 3146 len 28672
22135 020/300 Read block 1681 len 32768
22136 021/300 Write block 4189 len 24576 data: 4 4 5 5 6 4
22137 022/300 Read block 4001 len 40960
22138 023/300 Flush
22139 024/300 Write block 1827 len 32768 data: 5 6 6 6 7 7 7 9
22140 025/300 Write block 4435 len 28672 data: 2 3 4 4 4 4 3
22141 026/300 Flush
22142 027/300 Write block 20 len 16384 data: 6 6 5 5
22143 028/300 Read block 596 len 4096
22144 029/300 Write block 883 len 28672 data: 4 4 4 6 5 4 4
22145 030/300 Read block 1690 len 20480
22146 031/300 Write block 1078 len 24576 data: 6 4 4 3 3 4
22147 032/300 Write block 1832 len 40960 data: 8 8 10 8 5 5 5 3 3 2
22148 033/300 Write block 2265 len 28672 data: 3 5 6 5 5 5 6
22149 034/300 Write block 1066 len 12288 data: 2 2 3
22150 035/300 Read block 2740 len 28672
22151 036/300 Read block 88 len 8192
22152 037/300 Write block 1327 len 32768 data: 7 5 6 7 7 6 5 5
22153 038/300 Read block 2396 len 32768
22154 039/300 Read block 568 len 36864
22155 040/300 Read block 1753 len 16384
22156 041/300 Read block 4978 len 8192
22157 042/300 Read block 566 len 32768
22158 043/300 Write block 324 len 8192 data: 3 3
22159 044/300 Read block 3942 len 28672
22160 045/300 Write block 1974 len 32768 data: 4 4 4 4 4 3 5 5
22161 046/300 Write block 506 len 8192 data: 4 3
22162 047/300 Write block 3533 len 28672 data: 3 3 4 4 3 4 5
22163 048/300 Read block 3829 len 16384
22164 049/300 Write block 4705 len 28672 data: 3 3 3 3 3 3 3
22165 050/300 Read block 2524 len 32768
22166 051/300 Write block 2013 len 40960 data: 5 4 4 4 7 5 7 9 8 7
22167 052/300 Read block 1403 len 32768
22168 053/300 Write block 2931 len 36864 data: 4 4 5 5 5 5 4 4 5
22169 054/300 Read block 3563 len 4096
22170 055/300 Read block 633 len 40960
22171 056/300 Read block 904 len 24576
22172 057/300 Read block 72 len 32768
22173 058/300 Read block 1632 len 8192
22174 059/300 Write block 4157 len 28672 data: 4 6 6 6 4 3 3
22175 060/300 Flush
22176 061/300 Read block 1348 len 24576
22177 062/300 Read block 3516 len 40960
22178 063/300 Read block 3047 len 16384
22179 064/300 Flush
22180 065/300 Read block 984 len 16384
22181 066/300 Write block 1377 len 8192 data: 2 2
22182 067/300 Read block 3813 len 24576
22183 068/300 Flush
22184 069/300 Read block 3149 len 4096
22185 070/300 Write block 3201 len 36864 data: 3 4 6 6 6 5 4 5 6
22186 071/300 Read block 3641 len 36864
22187 072/300 Write block 378 len 20480 data: 2 3 3 4 4
22188 073/300 Write block 4827 len 4096 data: 4
22189 074/300 Flush
22190 075/300 Read block 501 len 12288
22191 076/300 Read block 2742 len 24576
22192 077/300 Write block 2599 len 12288 data: 6 6 7
22193 078/300 Read block 1953 len 4096
22194 079/300 Read block 1237 len 40960
22195 080/300 Read block 3443 len 8192
22196 081/300 Write block 3238 len 8192 data: 4 3
22197 082/300 Write block 3901 len 8192 data: 5 5
22198 083/300 Write block 568 len 4096 data: 4
22199 084/300 Write block 1860 len 32768 data: 6 5 4 6 6 6 7 6
22200 085/300 Write block 2840 len 8192 data: 5 4
22201 086/300 Write block 907 len 28672 data: 3 2 2 3 3 3 3
22202 087/300 Write block 985 len 16384 data: 4 4 5 5
22203 088/300 Read block 3816 len 24576
22204 089/300 Write block 1370 len 8192 data: 3 3
22205 090/300 Read block 920 len 4096
22206 091/300 Write block 3666 len 20480 data: 8 6 6 6 4
22207 092/300 Flush
22208 093/300 Write block 84 len 24576 data: 3 3 2 2 2 2
22209 094/300 Write block 733 len 16384 data: 3 4 4 4
22210 095/300 Write block 2630 len 8192 data: 4 3
22211 096/300 Write block 2670 len 8192 data: 4 4
22212 097/300 Read block 4880 len 28672
22213 098/300 Write block 3402 len 8192 data: 4 4
22214 099/300 Read block 644 len 12288
22215 100/300 Write block 902 len 36864 data: 5 5 5 6 5 4 3 3 4
22216 101/300 Read block 2096 len 20480
22217 102/300 Read block 3584 len 36864
22218 103/300 Write block 2636 len 28672 data: 3 4 3 3 3 4 4
22219 104/300 Write block 1071 len 8192 data: 5 7
22220 105/300 Write block 3861 len 8192 data: 3 3
22221 106/300 Read block 234 len 32768
22222 107/300 Write block 452 len 16384 data: 6 4 4 5
22223 108/300 Flush
22224 109/300 Read block 1292 len 8192
22225 110/300 Write block 2350 len 12288 data: 5 4 3
22226 111/300 Write block 4163 len 32768 data: 4 4 4 5 6 6 5 4
22227 112/300 Read block 648 len 24576
22228 113/300 Read block 2648 len 4096
22229 114/300 Read block 4234 len 8192
22230 115/300 Read block 2172 len 12288
22231 116/300 Read block 4281 len 28672
22232 117/300 Write block 4236 len 12288 data: 4 4 5
22233 118/300 Read block 3971 len 12288
22234 119/300 Flush
22235 120/300 Read block 642 len 32768
22236 121/300 Read block 1538 len 8192
22237 122/300 Flush
22238 123/300 Write block 2661 len 32768 data: 4 4 4 4 4 4 4 4
22239 124/300 Write block 786 len 36864 data: 2 2 2 2 2 3 5 5 5
22240 125/300 Write block 4914 len 28672 data: 6 5 5 4 3 3 5
22241 126/300 Flush
22242 127/300 Write block 3170 len 20480 data: 5 4 4 5 3
22243 128/300 Read block 3758 len 16384
22244 129/300 Read block 2221 len 28672
22245 130/300 Read block 76 len 28672
22246 131/300 Read block 1626 len 4096
22247 132/300 Flush
22248 133/300 Write block 1783 len 32768 data: 7 5 5 5 6 5 5 3
22249 134/300 Write block 3749 len 24576 data: 7 7 7 6 6 6
22250 135/300 Write block 1351 len 20480 data: 5 5 5 4 4
22251 136/300 Write block 4923 len 20480 data: 3 4 4 4 4
22252 137/300 Write block 4321 len 20480 data: 2 3 3 3 5
22253 138/300 Write block 3458 len 4096 data: 3
22254 139/300 Read block 683 len 32768
22255 140/300 Write block 303 len 20480 data: 4 4 4 5 4
22256 141/300 Read block 4582 len 20480
22257 142/300 Write block 2440 len 4096 data: 4
22258 143/300 Write block 959 len 40960 data: 2 3 4 3 3 3 4 4 4 3
22259 144/300 Read block 838 len 12288
22260 145/300 Flush
22261 146/300 Write block 2643 len 32768 data: 2 2 3 3 4 3 3 3
22262 147/300 Read block 3860 len 8192
22263 148/300 Flush
22264 149/300 Write block 1577 len 32768 data: 4 4 4 5 6 5 4 4
22265 150/300 Read block 1864 len 8192
22266 151/300 Flush
22267 152/300 Write block 4338 len 16384 data: 3 4 5 5
22268 153/300 Write block 2098 len 40960 data: 4 4 4 4 4 4 4 3 3 2
22269 154/300 Write block 689 len 36864 data: 3 3 4 3 3 3 5 4 4
22270 155/300 Write block 2947 len 8192 data: 3 3
22271 156/300 Read block 492 len 28672
22272 157/300 Read block 2345 len 8192
22273 158/300 Write block 4498 len 16384 data: 8 8 8 8
22274 159/300 Write block 2741 len 40960 data: 4 5 4 5 5 4 4 4 3 3
22275 160/300 Read block 1777 len 40960
22276 161/300 Read block 618 len 4096
22277 162/300 Write block 3541 len 16384 data: 6 7 6 4
22278 163/300 Read block 767 len 12288
22279 164/300 Write block 2282 len 12288 data: 4 4 4
22280 165/300 Flush
22281 166/300 Write block 4312 len 28672 data: 2 3 2 3 3 3 3
22282 167/300 Read block 3271 len 16384
22283 168/300 Flush
22284 169/300 Write block 371 len 20480 data: 5 4 3 3 2
22285 170/300 Write block 3448 len 8192 data: 3 3
22286 171/300 Write block 3505 len 4096 data: 2
22287 172/300 Write block 3678 len 40960 data: 8 8 8 9 9 8 9 7 8 7
22288 173/300 Read block 2392 len 8192
22289 174/300 Write block 4511 len 4096 data: 10
22290 175/300 Read block 4630 len 24576
22291 176/300 Flush
22292 177/300 Write block 1175 len 24576 data: 3 3 3 3 4 4
22293 178/300 Write block 2096 len 24576 data: 5 4 5 5 5 5
22294 179/300 Read block 1430 len 36864
22295 180/300 Flush
22296 181/300 Write block 4685 len 4096 data: 5
22297 182/300 Read block 4718 len 12288
22298 183/300 Write block 3919 len 20480 data: 6 5 3 3 4
22299 184/300 Flush
22300 185/300 Read block 840 len 12288
22301 186/300 Write block 779 len 24576 data: 3 2 2 2 2 2
22302 187/300 Read block 2178 len 8192
22303 188/300 Write block 2697 len 40960 data: 3 4 4 3 2 2 3 4 4 4
22304 189/300 Flush
22305 190/300 Write block 3949 len 36864 data: 6 6 4 4 4 4 6 5 4
22306 191/300 Read block 4925 len 20480
22307 192/300 Write block 4244 len 28672 data: 2 3 4 4 5 4 5
22308 193/300 Read block 3755 len 20480
22309 194/300 Write block 3162 len 12288 data: 5 6 6
22310 195/300 Write block 4920 len 32768 data: 6 4 3 4 5 5 5 5
22311 196/300 Read block 3626 len 16384
22312 197/300 Write block 2344 len 16384 data: 5 6 6 6
22313 198/300 Read block 2982 len 8192
22314 199/300 Flush
22315 200/300 Write block 604 len 4096 data: 2
22316 201/300 Write block 2354 len 24576 data: 3 4 5 5 6 6
22317 202/300 Write block 1780 len 40960 data: 4 5 6 8 6 6 6 7 6 6
22318 203/300 Read block 3255 len 12288
22319 204/300 Write block 1836 len 4096 data: 6
22320 205/300 Read block 1951 len 20480
22321 206/300 Read block 416 len 24576
22322 207/300 Read block 1352 len 12288
22323 208/300 Write block 1779 len 20480 data: 5 5 6 7 9
22324 209/300 Write block 4018 len 8192 data: 5 5
22325 210/300 Write block 4938 len 24576 data: 3 4 4 4 5 6
22326 211/300 Read block 3156 len 36864
22327 212/300 Read block 3630 len 8192
22328 213/300 Write block 1999 len 28672 data: 9 6 5 5 6 7 6
22329 214/300 Flush
22330 215/300 Read block 1730 len 32768
22331 216/300 Read block 2864 len 40960
22332 217/300 Read block 1811 len 8192
22333 218/300 Read block 4205 len 4096
22334 219/300 Write block 3896 len 16384 data: 3 4 5 5
22335 220/300 Write block 3489 len 12288 data: 4 4 4
22336 221/300 Read block 44 len 8192
22337 222/300 Read block 1027 len 8192
22338 223/300 Write block 2666 len 12288 data: 5 5 5
22339 224/300 Read block 557 len 8192
22340 225/300 Read block 3666 len 16384
22341 226/300 Read block 2008 len 12288
22342 227/300 Read block 494 len 12288
22343 228/300 Read block 3036 len 20480
22344 229/300 Write block 199 len 28672 data: 6 5 4 4 4 4 5
22345 230/300 Write block 430 len 20480 data: 4 3 4 4 5
22346 231/300 Read block 4880 len 20480
22347 232/300 Read block 4370 len 16384
22348 233/300 Read block 2369 len 16384
22349 234/300 Write block 165 len 40960 data: 3 3 4 4 4 4 4 5 5 4
22350 235/300 Write block 1037 len 16384 data: 6 5 4 5
22351 236/300 Read block 3049 len 8192
22352 237/300 Read block 1143 len 8192
22353 238/300 Write block 2506 len 24576 data: 4 5 5 3 3 3
22354 239/300 Read block 856 len 36864
22355 240/300 Write block 2495 len 32768 data: 2 2 3 4 4 3 4 4
22356 241/300 Write block 1584 len 16384 data: 5 5 5 4
22357 242/300 Read block 1120 len 8192
22358 243/300 Write block 846 len 8192 data: 4 3
22359 244/300 Flush
22360 245/300 Write block 1632 len 16384 data: 5 5 3 3
22361 246/300 Flush
22362 247/300 Write block 4044 len 8192 data: 5 5
22363 248/300 Write block 169 len 16384 data: 5 5 5 6
22364 249/300 Read block 771 len 8192
22365 250/300 Read block 1526 len 8192
22366 251/300 Flush
22367 252/300 Read block 2915 len 16384
22368 253/300 Read block 4307 len 40960
22369 254/300 Read block 3302 len 20480
22370 255/300 Flush
22371 256/300 Read block 2872 len 32768
22372 257/300 Write block 3025 len 12288 data: 2 2 2
22373 258/300 Write block 4036 len 36864 data: 3 4 4 4 4 4 5 5 6
22374 259/300 Read block 919 len 4096
22375 260/300 Write block 2959 len 12288 data: 4 5 5
22376 261/300 Read block 3470 len 16384
22377 262/300 Read block 883 len 12288
22378 263/300 Write block 4668 len 4096 data: 6
22379 264/300 Flush
22380 265/300 Write block 4468 len 24576 data: 7 7 7 7 8 6
22381 266/300 Write block 2700 len 28672 data: 4 3 3 4 5 5 5
22382 267/300 Read block 1608 len 20480
22383 268/300 Write block 4636 len 28672 data: 4 5 5 5 4 4 4
22384 269/300 Write block 2170 len 24576 data: 7 7 6 6 6 7
22385 270/300 Read block 904 len 20480
22386 271/300 Read block 4376 len 32768
22387 272/300 Read block 3521 len 36864
22388 273/300 Write block 2451 len 4096 data: 5
22389 274/300 Write block 3308 len 4096 data: 5
22390 275/300 Read block 4727 len 40960
22391 276/300 Write block 3869 len 24576 data: 3 3 4 3 3 3
22392 277/300 Write block 3708 len 40960 data: 7 5 6 4 4 4 4 4 5 5
22393 278/300 Flush
22394 279/300 Read block 3019 len 12288
22395 280/300 Read block 3902 len 16384
22396 281/300 Write block 481 len 32768 data: 3 2 2 3 3 3 3 3
22397 282/300 Read block 3779 len 16384
22398 283/300 Write block 1696 len 28672 data: 7 6 4 4 4 4 4
22399 284/300 Write block 1677 len 8192 data: 5 6
22400 285/300 Read block 4199 len 28672
22401 286/300 Read block 3435 len 32768
22402 287/300 Read block 2871 len 8192
22403 288/300 Write block 2441 len 40960 data: 4 4 4 3 3 4 5 5 5 5
22404 289/300 Read block 2192 len 40960
22405 290/300 Read block 2967 len 20480
22406 291/300 Flush
22407 292/300 Read block 884 len 40960
22408 293/300 Flush
22409 294/300 Write block 4844 len 32768 data: 5 4 3 3 5 6 5 5
22410 295/300 Write block 517 len 24576 data: 3 3 3 3 3 3
22411 296/300 Read block 4081 len 36864
22412 297/300 Write block 520 len 40960 data: 4 4 4 4 3 3 4 5 7 6
22413 298/300 Write block 3628 len 12288 data: 2 2 2
22414 299/300 Read block 19 len 36864
22415 300/300 Read block 2967 len 8192
22416 301/300 Read block 4778 len 28672
22417 Main task has work to do, go find it
22418 got dsc Start(0)
22419 start 0
22420 Feb 07 20:12:22.005 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/0, method: POST, req_id: 6d75945f-5524-4100-aac3-aed4f9925d7d, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
22421 Replay: started 0, returned:Ok(())
22422 ----------------------------------------------------------------
22423 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22424 States: Offline Active Active
22425 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22426 6133 Acked Read 2 Sent Done Done false
22427 6134 Acked Write 6 Sent Done Done false
22428 6135 Acked Read 6 Sent Done Done false
22429 6136 Acked Read 7 Sent Done Done false
22430 6137 Acked Read 1 Sent Done Done false
22431 6138 Acked Read 9 Sent Done Done false
22432 6139 Acked Read 4 Sent Done Done false
22433 6140 Acked Flush 0 Sent Done Done false
22434 6141 Acked Read 7 Sent Done Done false
22435 6142 Acked Read 5 Sent Done Done false
22436 6143 Acked Write 8 Sent Done Done false
22437 6144 Acked Write 10 Sent Done Done false
22438 6145 Acked Write 2 Sent Done Done false
22439 6146 Acked Read 6 Sent Done Done false
22440 6147 Acked Flush 0 Sent Done Done false
22441 6148 Acked Read 6 Sent Done Done false
22442 6149 Acked Write 9 Sent Done Done false
22443 6150 Acked Write 1 Sent Done Done false
22444 6151 Acked Read 7 Sent Done Done false
22445 6152 Acked Read 8 Sent Done Done false
22446 6153 Acked Write 6 Sent Done Done false
22447 6154 Acked Read 10 Sent Done Done false
22448 6155 Acked Flush 0 Sent Done Done false
22449 6156 Acked Write 8 Sent Done Done false
22450 6157 Acked Write 7 Sent Done Done false
22451 6158 Acked Flush 0 Sent Done Done false
22452 6159 Acked Write 4 Sent Done Done false
22453 6160 Acked Read 1 Sent Done Done false
22454 6161 Acked Write 7 Sent Done Done false
22455 6162 Acked Read 5 Sent Done Done false
22456 6163 Acked Write 6 Sent Done Done false
22457 6164 Acked Write 10 Sent Done Done false
22458 6165 Acked Write 7 Sent Done Done false
22459 6166 Acked Write 3 Sent Done Done false
22460 6167 Acked Read 7 Sent Done Done false
22461 6168 Acked Read 2 Sent Done Done false
22462 6169 Acked Write 8 Sent Done Done false
22463 6170 Acked Read 8 Sent Done Done false
22464 6171 Acked Read 9 Sent Done Done false
22465 6172 Acked Read 4 Sent Done Done false
22466 6173 Acked Read 2 Sent Done Done false
22467 6174 Acked Read 8 Sent Done Done false
22468 6175 Acked Write 2 Sent Done Done false
22469 6176 Acked Read 7 Sent Done Done false
22470 6177 Acked Write 8 Sent Done Done false
22471 6178 Acked Write 2 Sent Done Done false
22472 6179 Acked Write 7 Sent Done Done false
22473 6180 Acked Read 4 Sent Done Done false
22474 6181 Acked Write 7 Sent Done Done false
22475 6182 Acked Read 8 Sent Done Done false
22476 6183 Acked Write 10 Sent Done Done false
22477 6184 Acked Read 8 Sent Done Done false
22478 6185 Acked Write 9 Sent Done Done false
22479 6186 Acked Read 1 Sent Done Done false
22480 6187 Acked Read 10 Sent Done Done false
22481 6188 Acked Read 6 Sent Done Done false
22482 6189 Acked Read 8 Sent Done Done false
22483 6190 Acked Read 2 Sent Done Done false
22484 6191 Acked Write 7 Sent Done Done false
22485 6192 Acked Flush 0 Sent Done Done false
22486 6193 Acked Read 6 Sent Done Done false
22487 6194 Acked Read 10 Sent Done Done false
22488 6195 Acked Read 4 Sent Done Done false
22489 6196 Acked Flush 0 Sent Done Done false
22490 6197 Acked Read 4 Sent Done Done false
22491 6198 Acked Write 2 Sent Done Done false
22492 6199 Acked Read 6 Sent Done Done false
22493 6200 Acked Flush 0 Sent Done Done false
22494 6201 Acked Read 1 Sent Done Done false
22495 6202 Acked Write 9 Sent Done Done false
22496 6203 Acked Read 9 Sent Done Done false
22497 6204 Acked Write 5 Sent Done Done false
22498 6205 Acked Write 1 Sent Done Done false
22499 6206 Acked Flush 0 Sent Done Done false
22500 6207 Acked Read 3 Sent Done Done false
22501 6208 Acked Read 6 Sent Done Done false
22502 6209 Acked Write 3 Sent Done Done false
22503 6210 Acked Read 1 Sent Done Done false
22504 6211 Acked Read 10 Sent Done Done false
22505 6212 Acked Read 2 Sent Done Done false
22506 6213 Acked Write 2 Sent Done Done false
22507 6214 Acked Write 2 Sent Done Done false
22508 6215 Acked Write 1 Sent Done Done false
22509 6216 Acked Write 8 Sent Done Done false
22510 6217 Acked Write 2 Sent Done Done false
22511 6218 Acked Write 7 Sent Done Done false
22512 6219 Acked Write 4 Sent Done Done false
22513 6220 Acked Read 6 Sent Done Done false
22514 6221 Acked Write 2 Sent Done Done false
22515 6222 Acked Read 1 Sent Done Done false
22516 6223 Acked Write 5 Sent Done Done false
22517 6224 Acked Flush 0 Sent Done Done false
22518 6225 Acked Write 6 Sent Done Done false
22519 6226 Acked Write 4 Sent Done Done false
22520 6227 Acked Write 2 Sent Done Done false
22521 6228 Acked Write 2 Sent Done Done false
22522 6229 Acked Read 7 Sent Done Done false
22523 6230 Acked Write 2 Sent Done Done false
22524 6231 Acked Read 3 Sent Done Done false
22525 6232 Acked Write 9 Sent Done Done false
22526 6233 Acked Read 5 Sent Done Done false
22527 6234 Acked Read 9 Sent Done Done false
22528 6235 Acked Write 7 Sent Done Done false
22529 6236 Acked Write 2 Sent Done Done false
22530 6237 Acked Write 2 Sent Done Done false
22531 6238 Acked Read 8 Sent Done Done false
22532 6239 Acked Write 4 Sent Done Done false
22533 6240 Acked Flush 0 Sent Done Done false
22534 6241 Acked Read 2 Sent Done Done false
22535 6242 Acked Write 3 Sent Done Done false
22536 6243 Acked Write 8 Sent Done Done false
22537 6244 Acked Read 6 Sent Done Done false
22538 6245 Acked Read 1 Sent Done Done false
22539 6246 Acked Read 2 Sent Done Done false
22540 6247 Acked Read 3 Sent Done Done false
22541 6248 Acked Read 7 Sent Done Done false
22542 6249 Acked Write 3 Sent Done Done false
22543 6250 Acked Read 3 Sent Done Done false
22544 6251 Acked Flush 0 Sent Done Done false
22545 6252 Acked Read 8 Sent Done Done false
22546 6253 Acked Read 2 Sent Done Done false
22547 6254 Acked Flush 0 Sent Done Done false
22548 6255 Acked Write 8 Sent Done Done false
22549 6256 Acked Write 9 Sent Done Done false
22550 6257 Acked Write 7 Sent Done Done false
22551 6258 Acked Flush 0 Sent Done Done false
22552 6259 Acked Write 5 Sent Done Done false
22553 6260 Acked Read 4 Sent Done Done false
22554 6261 Acked Read 7 Sent Done Done false
22555 6262 Acked Read 7 Sent Done Done false
22556 6263 Acked Read 1 Sent Done Done false
22557 6264 Acked Flush 0 Sent Done Done false
22558 6265 Acked Write 8 Sent Done Done false
22559 6266 Acked Write 6 Sent Done Done false
22560 6267 Acked Write 5 Sent Done Done false
22561 6268 Acked Write 5 Sent Done Done false
22562 6269 Acked Write 5 Sent Done Done false
22563 6270 Acked Write 1 Sent Done Done false
22564 6271 Acked Read 8 Sent Done Done false
22565 6272 Acked Write 5 Sent Done Done false
22566 6273 Acked Read 5 Sent Done Done false
22567 6274 Acked Write 1 Sent Done Done false
22568 6275 Acked Write 10 Sent Done Done false
22569 6276 Acked Read 3 Sent Done Done false
22570 6277 Acked Flush 0 Sent Done Done false
22571 6278 Acked Write 8 Sent Done Done false
22572 6279 Acked Read 2 Sent Done Done false
22573 6280 Acked Flush 0 Sent Done Done false
22574 6281 Acked Write 8 Sent Done Done false
22575 6282 Acked Read 2 Sent Done Done false
22576 6283 Acked Flush 0 Sent Done Done false
22577 6284 Acked Write 4 Sent Done Done false
22578 6285 Acked Write 10 Sent Done Done false
22579 6286 Acked Write 9 Sent Done Done false
22580 6287 Acked Write 2 Sent Done Done false
22581 6288 Acked Read 7 Sent Done Done false
22582 6289 Acked Read 2 Sent Done Done false
22583 6290 Acked Write 4 Sent Done Done false
22584 6291 Acked Write 10 Sent Done Done false
22585 6292 Acked Read 10 Sent Done Done false
22586 6293 Acked Read 1 Sent Done Done false
22587 6294 Acked Write 4 Sent Done Done false
22588 6295 Acked Read 3 Sent Done Done false
22589 6296 Acked Write 3 Sent Done Done false
22590 6297 Acked Flush 0 Sent Done Done false
22591 6298 Acked Write 7 Sent Done Done false
22592 6299 Acked Read 4 Sent Done Done false
22593 6300 Acked Flush 0 Sent Done Done false
22594 6301 Acked Write 5 Sent Done Done false
22595 6302 Acked Write 2 Sent Done Done false
22596 6303 Acked Write 1 Sent Done Done false
22597 6304 Acked Write 10 Sent Done Done false
22598 6305 Acked Read 2 Sent Done Done false
22599 6306 Acked Write 1 Sent Done Done false
22600 6307 Acked Read 6 Sent Done Done false
22601 6308 Acked Flush 0 Sent Done Done false
22602 6309 Acked Write 6 Sent Done Done false
22603 6310 Acked Write 6 Sent Done Done false
22604 6311 Acked Read 9 Sent Done Done false
22605 6312 Acked Flush 0 Sent Done Done false
22606 6313 Acked Write 1 Sent Done Done false
22607 6314 Acked Read 3 Sent Done Done false
22608 6315 Acked Write 5 Sent Done Done false
22609 6316 Acked Flush 0 Sent Done Done false
22610 6317 Acked Read 3 Sent Done Done false
22611 6318 Acked Write 6 Sent Done Done false
22612 6319 Acked Read 2 Sent Done Done false
22613 6320 Acked Write 10 Sent Done Done false
22614 6321 Acked Flush 0 Sent Done Done false
22615 6322 Acked Write 9 Sent Done Done false
22616 6323 Acked Read 5 Sent Done Done false
22617 6324 Acked Write 7 Sent Done Done false
22618 6325 Acked Read 5 Sent Done Done false
22619 6326 Acked Write 3 Sent Done Done false
22620 6327 Acked Write 8 Sent Done Done false
22621 6328 Acked Read 4 Sent Done Done false
22622 6329 Acked Write 4 Sent Done Done false
22623 6330 Acked Read 2 Sent Done Done false
22624 6331 Acked Flush 0 Sent Done Done false
22625 6332 Acked Write 1 Sent Done Done false
22626 6333 Acked Write 6 Sent Done Done false
22627 6334 Acked Write 10 Sent Done Done false
22628 6335 Acked Read 3 Sent Done Done false
22629 6336 Acked Write 1 Sent Done Done false
22630 6337 Acked Read 5 Sent Done Done false
22631 6338 Acked Read 6 Sent Done Done false
22632 6339 Acked Read 3 Sent Done Done false
22633 6340 Acked Write 5 Sent Done Done false
22634 6341 Acked Write 2 Sent Done Done false
22635 6342 Acked Write 6 Sent Done Done false
22636 6343 Acked Read 9 Sent Done Done false
22637 6344 Acked Read 2 Sent Done Done false
22638 6345 Acked Write 7 Sent Done Done false
22639 6346 Acked Flush 0 Sent Done Done false
22640 6347 Acked Read 8 Sent Done Done false
22641 6348 Acked Read 10 Sent Done Done false
22642 6349 Acked Read 2 Sent Done Done false
22643 6350 Acked Read 1 Sent Done Done false
22644 6351 Acked Write 4 Sent Done Done false
22645 6352 Acked Write 3 Sent Done Done false
22646 6353 Acked Read 2 Sent Done Done false
22647 6354 Acked Read 2 Sent Done Done false
22648 6355 Acked Write 3 Sent Done Done false
22649 6356 Acked Read 2 Sent Done Done false
22650 6357 Acked Read 4 Sent Done Done false
22651 6358 Acked Read 3 Sent Done Done false
22652 6359 Acked Read 3 Sent Done Done false
22653 6360 Acked Read 5 Sent Done Done false
22654 6361 Acked Write 7 Sent Done Done false
22655 6362 Acked Write 5 Sent Done Done false
22656 6363 Acked Read 5 Sent Done Done false
22657 6364 Acked Read 4 Sent Done Done false
22658 6365 Acked Read 4 Sent Done Done false
22659 6366 Acked Write 10 Sent Done Done false
22660 6367 Acked Write 4 Sent Done Done false
22661 6368 Acked Read 2 Sent Done Done false
22662 6369 Acked Read 2 Sent Done Done false
22663 6370 Acked Write 6 Sent Done Done false
22664 6371 Acked Read 9 Sent Done Done false
22665 6372 Acked Write 8 Sent Done Done false
22666 6373 Acked Write 4 Sent Done Done false
22667 6374 Acked Read 2 Sent Done Done false
22668 6375 Acked Write 2 Sent Done Done false
22669 6376 Acked Flush 0 Sent Done Done false
22670 6377 Acked Write 4 Sent Done Done false
22671 6378 Acked Flush 0 Sent Done Done false
22672 6379 Acked Write 2 Sent Done Done false
22673 6380 Acked Write 4 Sent Done Done false
22674 6381 Acked Read 2 Sent Done Done false
22675 6382 Acked Read 2 Sent Done Done false
22676 6383 Acked Flush 0 Sent Done Done false
22677 6384 Acked Read 4 Sent Done Done false
22678 6385 Acked Read 10 Sent Done Done false
22679 6386 Acked Read 5 Sent Done Done false
22680 6387 Acked Flush 0 Sent Done Done false
22681 6388 Acked Read 8 Sent Done Done false
22682 6389 Acked Write 3 Sent Done Done false
22683 6390 Acked Write 9 Sent Done Done false
22684 6391 Acked Read 1 Sent Done Done false
22685 6392 Acked Write 3 Sent Done Done false
22686 6393 Acked Read 4 Sent Done Done false
22687 6394 Acked Read 3 Sent Done Done false
22688 6395 Acked Write 1 Sent Done Done false
22689 6396 Acked Flush 0 Sent Done Done false
22690 6397 Acked Write 6 Sent Done Done false
22691 6398 Acked Write 7 Sent Done Done false
22692 6399 Acked Read 5 Sent Done Done false
22693 6400 Acked Write 7 Sent Done Done false
22694 6401 Acked Write 6 Sent Done Done false
22695 6402 Acked Read 5 Sent Done Done false
22696 6403 Acked Read 8 Sent Done Done false
22697 6404 Acked Read 9 Sent Done Done false
22698 6405 Acked Write 1 Sent Done Done false
22699 6406 Acked Write 1 Sent Done Done false
22700 6407 Acked Read 10 Sent Done Done false
22701 6408 Acked Write 6 Sent Done Done false
22702 6409 Acked Write 10 Sent Done Done false
22703 6410 Acked Flush 0 Sent Done Done false
22704 6411 Acked Read 3 Sent Done Done false
22705 6412 Acked Read 4 Sent Done Done false
22706 6413 Acked Write 8 Sent Done Done false
22707 6414 Acked Read 4 Sent Done Done false
22708 6415 Acked Write 7 Sent Done Done false
22709 6416 Acked Write 2 Sent Done Done false
22710 6417 Acked Read 7 Sent Done Done false
22711 6418 Acked Read 8 Sent Done Done false
22712 6419 Acked Read 2 Sent Done Done false
22713 6420 Acked Write 10 Sent Done Done false
22714 6421 Acked Read 10 Sent Done Done false
22715 6422 Acked Read 5 Sent Done Done false
22716 6423 Acked Flush 0 Sent Done Done false
22717 6424 Acked Read 10 Sent Done Done false
22718 6425 Acked Flush 0 Sent Done Done false
22719 6426 Acked Write 8 Sent Done Done false
22720 6427 Acked Write 6 Sent Done Done false
22721 6428 Acked Read 9 Sent Done Done false
22722 6429 Acked Write 10 Sent Done Done false
22723 6430 Acked Write 3 Sent Done Done false
22724 6431 Acked Read 9 Sent Done Done false
22725 6432 Acked Read 2 Sent Done Done false
22726 6433 Acked Read 7 Sent Done Done false
22727 STATES DS:0 DS:1 DS:2 TOTAL
22728 Sent 301 0 0 301
22729 Done 0 301 301 602
22730 Skipped 0 0 0 0
22731 Error 0 0 0 0
22732 Last Flush: Some(JobId(6132)) Some(JobId(6425)) Some(JobId(6425))
22733 Downstairs last five completed: 6132 6131 6130 6129 6128
22734 Upstairs last five completed: 6433 6432 6431 6430 6429
22735 CLIENT: Up:0 ds:301 act:2
227362025-02-07T20:12:22.507ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
227372025-02-07T20:12:23.440ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
22738 [8810] got start action
22739 [8810] I am going to restart
22740 Starting downstairs at port 8810
22741 Make output file at "/tmp/dsc/downstairs-8810.txt"
22742 [8810][0] reports Starting
22743 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1227)
22744 [8810][0] reports Running
227452025-02-07T20:12:24.441ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
227462025-02-07T20:12:25.443ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
22747 ----------------------------------------------------------------
22748 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22749 States: Offline Active Active
22750 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22751 6133 Acked Read 2 Sent Done Done false
22752 6134 Acked Write 6 Sent Done Done false
22753 6135 Acked Read 6 Sent Done Done false
22754 6136 Acked Read 7 Sent Done Done false
22755 6137 Acked Read 1 Sent Done Done false
22756 6138 Acked Read 9 Sent Done Done false
22757 6139 Acked Read 4 Sent Done Done false
22758 6140 Acked Flush 0 Sent Done Done false
22759 6141 Acked Read 7 Sent Done Done false
22760 6142 Acked Read 5 Sent Done Done false
22761 6143 Acked Write 8 Sent Done Done false
22762 6144 Acked Write 10 Sent Done Done false
22763 6145 Acked Write 2 Sent Done Done false
22764 6146 Acked Read 6 Sent Done Done false
22765 6147 Acked Flush 0 Sent Done Done false
22766 6148 Acked Read 6 Sent Done Done false
22767 6149 Acked Write 9 Sent Done Done false
22768 6150 Acked Write 1 Sent Done Done false
22769 6151 Acked Read 7 Sent Done Done false
22770 6152 Acked Read 8 Sent Done Done false
22771 6153 Acked Write 6 Sent Done Done false
22772 6154 Acked Read 10 Sent Done Done false
22773 6155 Acked Flush 0 Sent Done Done false
22774 6156 Acked Write 8 Sent Done Done false
22775 6157 Acked Write 7 Sent Done Done false
22776 6158 Acked Flush 0 Sent Done Done false
22777 6159 Acked Write 4 Sent Done Done false
22778 6160 Acked Read 1 Sent Done Done false
22779 6161 Acked Write 7 Sent Done Done false
22780 6162 Acked Read 5 Sent Done Done false
22781 6163 Acked Write 6 Sent Done Done false
22782 6164 Acked Write 10 Sent Done Done false
22783 6165 Acked Write 7 Sent Done Done false
22784 6166 Acked Write 3 Sent Done Done false
22785 6167 Acked Read 7 Sent Done Done false
22786 6168 Acked Read 2 Sent Done Done false
22787 6169 Acked Write 8 Sent Done Done false
22788 6170 Acked Read 8 Sent Done Done false
22789 6171 Acked Read 9 Sent Done Done false
22790 6172 Acked Read 4 Sent Done Done false
22791 6173 Acked Read 2 Sent Done Done false
22792 6174 Acked Read 8 Sent Done Done false
22793 6175 Acked Write 2 Sent Done Done false
22794 6176 Acked Read 7 Sent Done Done false
22795 6177 Acked Write 8 Sent Done Done false
22796 6178 Acked Write 2 Sent Done Done false
22797 6179 Acked Write 7 Sent Done Done false
22798 6180 Acked Read 4 Sent Done Done false
22799 6181 Acked Write 7 Sent Done Done false
22800 6182 Acked Read 8 Sent Done Done false
22801 6183 Acked Write 10 Sent Done Done false
22802 6184 Acked Read 8 Sent Done Done false
22803 6185 Acked Write 9 Sent Done Done false
22804 6186 Acked Read 1 Sent Done Done false
22805 6187 Acked Read 10 Sent Done Done false
22806 6188 Acked Read 6 Sent Done Done false
22807 6189 Acked Read 8 Sent Done Done false
22808 6190 Acked Read 2 Sent Done Done false
22809 6191 Acked Write 7 Sent Done Done false
22810 6192 Acked Flush 0 Sent Done Done false
22811 6193 Acked Read 6 Sent Done Done false
22812 6194 Acked Read 10 Sent Done Done false
22813 6195 Acked Read 4 Sent Done Done false
22814 6196 Acked Flush 0 Sent Done Done false
22815 6197 Acked Read 4 Sent Done Done false
22816 6198 Acked Write 2 Sent Done Done false
22817 6199 Acked Read 6 Sent Done Done false
22818 6200 Acked Flush 0 Sent Done Done false
22819 6201 Acked Read 1 Sent Done Done false
22820 6202 Acked Write 9 Sent Done Done false
22821 6203 Acked Read 9 Sent Done Done false
22822 6204 Acked Write 5 Sent Done Done false
22823 6205 Acked Write 1 Sent Done Done false
22824 6206 Acked Flush 0 Sent Done Done false
22825 6207 Acked Read 3 Sent Done Done false
22826 6208 Acked Read 6 Sent Done Done false
22827 6209 Acked Write 3 Sent Done Done false
22828 6210 Acked Read 1 Sent Done Done false
22829 6211 Acked Read 10 Sent Done Done false
22830 6212 Acked Read 2 Sent Done Done false
22831 6213 Acked Write 2 Sent Done Done false
22832 6214 Acked Write 2 Sent Done Done false
22833 6215 Acked Write 1 Sent Done Done false
22834 6216 Acked Write 8 Sent Done Done false
22835 6217 Acked Write 2 Sent Done Done false
22836 6218 Acked Write 7 Sent Done Done false
22837 6219 Acked Write 4 Sent Done Done false
22838 6220 Acked Read 6 Sent Done Done false
22839 6221 Acked Write 2 Sent Done Done false
22840 6222 Acked Read 1 Sent Done Done false
22841 6223 Acked Write 5 Sent Done Done false
22842 6224 Acked Flush 0 Sent Done Done false
22843 6225 Acked Write 6 Sent Done Done false
22844 6226 Acked Write 4 Sent Done Done false
22845 6227 Acked Write 2 Sent Done Done false
22846 6228 Acked Write 2 Sent Done Done false
22847 6229 Acked Read 7 Sent Done Done false
22848 6230 Acked Write 2 Sent Done Done false
22849 6231 Acked Read 3 Sent Done Done false
22850 6232 Acked Write 9 Sent Done Done false
22851 6233 Acked Read 5 Sent Done Done false
22852 6234 Acked Read 9 Sent Done Done false
22853 6235 Acked Write 7 Sent Done Done false
22854 6236 Acked Write 2 Sent Done Done false
22855 6237 Acked Write 2 Sent Done Done false
22856 6238 Acked Read 8 Sent Done Done false
22857 6239 Acked Write 4 Sent Done Done false
22858 6240 Acked Flush 0 Sent Done Done false
22859 6241 Acked Read 2 Sent Done Done false
22860 6242 Acked Write 3 Sent Done Done false
22861 6243 Acked Write 8 Sent Done Done false
22862 6244 Acked Read 6 Sent Done Done false
22863 6245 Acked Read 1 Sent Done Done false
22864 6246 Acked Read 2 Sent Done Done false
22865 6247 Acked Read 3 Sent Done Done false
22866 6248 Acked Read 7 Sent Done Done false
22867 6249 Acked Write 3 Sent Done Done false
22868 6250 Acked Read 3 Sent Done Done false
22869 6251 Acked Flush 0 Sent Done Done false
22870 6252 Acked Read 8 Sent Done Done false
22871 6253 Acked Read 2 Sent Done Done false
22872 6254 Acked Flush 0 Sent Done Done false
22873 6255 Acked Write 8 Sent Done Done false
22874 6256 Acked Write 9 Sent Done Done false
22875 6257 Acked Write 7 Sent Done Done false
22876 6258 Acked Flush 0 Sent Done Done false
22877 6259 Acked Write 5 Sent Done Done false
22878 6260 Acked Read 4 Sent Done Done false
22879 6261 Acked Read 7 Sent Done Done false
22880 6262 Acked Read 7 Sent Done Done false
22881 6263 Acked Read 1 Sent Done Done false
22882 6264 Acked Flush 0 Sent Done Done false
22883 6265 Acked Write 8 Sent Done Done false
22884 6266 Acked Write 6 Sent Done Done false
22885 6267 Acked Write 5 Sent Done Done false
22886 6268 Acked Write 5 Sent Done Done false
22887 6269 Acked Write 5 Sent Done Done false
22888 6270 Acked Write 1 Sent Done Done false
22889 6271 Acked Read 8 Sent Done Done false
22890 6272 Acked Write 5 Sent Done Done false
22891 6273 Acked Read 5 Sent Done Done false
22892 6274 Acked Write 1 Sent Done Done false
22893 6275 Acked Write 10 Sent Done Done false
22894 6276 Acked Read 3 Sent Done Done false
22895 6277 Acked Flush 0 Sent Done Done false
22896 6278 Acked Write 8 Sent Done Done false
22897 6279 Acked Read 2 Sent Done Done false
22898 6280 Acked Flush 0 Sent Done Done false
22899 6281 Acked Write 8 Sent Done Done false
22900 6282 Acked Read 2 Sent Done Done false
22901 6283 Acked Flush 0 Sent Done Done false
22902 6284 Acked Write 4 Sent Done Done false
22903 6285 Acked Write 10 Sent Done Done false
22904 6286 Acked Write 9 Sent Done Done false
22905 6287 Acked Write 2 Sent Done Done false
22906 6288 Acked Read 7 Sent Done Done false
22907 6289 Acked Read 2 Sent Done Done false
22908 6290 Acked Write 4 Sent Done Done false
22909 6291 Acked Write 10 Sent Done Done false
22910 6292 Acked Read 10 Sent Done Done false
22911 6293 Acked Read 1 Sent Done Done false
22912 6294 Acked Write 4 Sent Done Done false
22913 6295 Acked Read 3 Sent Done Done false
22914 6296 Acked Write 3 Sent Done Done false
22915 6297 Acked Flush 0 Sent Done Done false
22916 6298 Acked Write 7 Sent Done Done false
22917 6299 Acked Read 4 Sent Done Done false
22918 6300 Acked Flush 0 Sent Done Done false
22919 6301 Acked Write 5 Sent Done Done false
22920 6302 Acked Write 2 Sent Done Done false
22921 6303 Acked Write 1 Sent Done Done false
22922 6304 Acked Write 10 Sent Done Done false
22923 6305 Acked Read 2 Sent Done Done false
22924 6306 Acked Write 1 Sent Done Done false
22925 6307 Acked Read 6 Sent Done Done false
22926 6308 Acked Flush 0 Sent Done Done false
22927 6309 Acked Write 6 Sent Done Done false
22928 6310 Acked Write 6 Sent Done Done false
22929 6311 Acked Read 9 Sent Done Done false
22930 6312 Acked Flush 0 Sent Done Done false
22931 6313 Acked Write 1 Sent Done Done false
22932 6314 Acked Read 3 Sent Done Done false
22933 6315 Acked Write 5 Sent Done Done false
22934 6316 Acked Flush 0 Sent Done Done false
22935 6317 Acked Read 3 Sent Done Done false
22936 6318 Acked Write 6 Sent Done Done false
22937 6319 Acked Read 2 Sent Done Done false
22938 6320 Acked Write 10 Sent Done Done false
22939 6321 Acked Flush 0 Sent Done Done false
22940 6322 Acked Write 9 Sent Done Done false
22941 6323 Acked Read 5 Sent Done Done false
22942 6324 Acked Write 7 Sent Done Done false
22943 6325 Acked Read 5 Sent Done Done false
22944 6326 Acked Write 3 Sent Done Done false
22945 6327 Acked Write 8 Sent Done Done false
22946 6328 Acked Read 4 Sent Done Done false
22947 6329 Acked Write 4 Sent Done Done false
22948 6330 Acked Read 2 Sent Done Done false
22949 6331 Acked Flush 0 Sent Done Done false
22950 6332 Acked Write 1 Sent Done Done false
22951 6333 Acked Write 6 Sent Done Done false
22952 6334 Acked Write 10 Sent Done Done false
22953 6335 Acked Read 3 Sent Done Done false
22954 6336 Acked Write 1 Sent Done Done false
22955 6337 Acked Read 5 Sent Done Done false
22956 6338 Acked Read 6 Sent Done Done false
22957 6339 Acked Read 3 Sent Done Done false
22958 6340 Acked Write 5 Sent Done Done false
22959 6341 Acked Write 2 Sent Done Done false
22960 6342 Acked Write 6 Sent Done Done false
22961 6343 Acked Read 9 Sent Done Done false
22962 6344 Acked Read 2 Sent Done Done false
22963 6345 Acked Write 7 Sent Done Done false
22964 6346 Acked Flush 0 Sent Done Done false
22965 6347 Acked Read 8 Sent Done Done false
22966 6348 Acked Read 10 Sent Done Done false
22967 6349 Acked Read 2 Sent Done Done false
22968 6350 Acked Read 1 Sent Done Done false
22969 6351 Acked Write 4 Sent Done Done false
22970 6352 Acked Write 3 Sent Done Done false
22971 6353 Acked Read 2 Sent Done Done false
22972 6354 Acked Read 2 Sent Done Done false
22973 6355 Acked Write 3 Sent Done Done false
22974 6356 Acked Read 2 Sent Done Done false
22975 6357 Acked Read 4 Sent Done Done false
22976 6358 Acked Read 3 Sent Done Done false
22977 6359 Acked Read 3 Sent Done Done false
22978 6360 Acked Read 5 Sent Done Done false
22979 6361 Acked Write 7 Sent Done Done false
22980 6362 Acked Write 5 Sent Done Done false
22981 6363 Acked Read 5 Sent Done Done false
22982 6364 Acked Read 4 Sent Done Done false
22983 6365 Acked Read 4 Sent Done Done false
22984 6366 Acked Write 10 Sent Done Done false
22985 6367 Acked Write 4 Sent Done Done false
22986 6368 Acked Read 2 Sent Done Done false
22987 6369 Acked Read 2 Sent Done Done false
22988 6370 Acked Write 6 Sent Done Done false
22989 6371 Acked Read 9 Sent Done Done false
22990 6372 Acked Write 8 Sent Done Done false
22991 6373 Acked Write 4 Sent Done Done false
22992 6374 Acked Read 2 Sent Done Done false
22993 6375 Acked Write 2 Sent Done Done false
22994 6376 Acked Flush 0 Sent Done Done false
22995 6377 Acked Write 4 Sent Done Done false
22996 6378 Acked Flush 0 Sent Done Done false
22997 6379 Acked Write 2 Sent Done Done false
22998 6380 Acked Write 4 Sent Done Done false
22999 6381 Acked Read 2 Sent Done Done false
23000 6382 Acked Read 2 Sent Done Done false
23001 6383 Acked Flush 0 Sent Done Done false
23002 6384 Acked Read 4 Sent Done Done false
23003 6385 Acked Read 10 Sent Done Done false
23004 6386 Acked Read 5 Sent Done Done false
23005 6387 Acked Flush 0 Sent Done Done false
23006 6388 Acked Read 8 Sent Done Done false
23007 6389 Acked Write 3 Sent Done Done false
23008 6390 Acked Write 9 Sent Done Done false
23009 6391 Acked Read 1 Sent Done Done false
23010 6392 Acked Write 3 Sent Done Done false
23011 6393 Acked Read 4 Sent Done Done false
23012 6394 Acked Read 3 Sent Done Done false
23013 6395 Acked Write 1 Sent Done Done false
23014 6396 Acked Flush 0 Sent Done Done false
23015 6397 Acked Write 6 Sent Done Done false
23016 6398 Acked Write 7 Sent Done Done false
23017 6399 Acked Read 5 Sent Done Done false
23018 6400 Acked Write 7 Sent Done Done false
23019 6401 Acked Write 6 Sent Done Done false
23020 6402 Acked Read 5 Sent Done Done false
23021 6403 Acked Read 8 Sent Done Done false
23022 6404 Acked Read 9 Sent Done Done false
23023 6405 Acked Write 1 Sent Done Done false
23024 6406 Acked Write 1 Sent Done Done false
23025 6407 Acked Read 10 Sent Done Done false
23026 6408 Acked Write 6 Sent Done Done false
23027 6409 Acked Write 10 Sent Done Done false
23028 6410 Acked Flush 0 Sent Done Done false
23029 6411 Acked Read 3 Sent Done Done false
23030 6412 Acked Read 4 Sent Done Done false
23031 6413 Acked Write 8 Sent Done Done false
23032 6414 Acked Read 4 Sent Done Done false
23033 6415 Acked Write 7 Sent Done Done false
23034 6416 Acked Write 2 Sent Done Done false
23035 6417 Acked Read 7 Sent Done Done false
23036 6418 Acked Read 8 Sent Done Done false
23037 6419 Acked Read 2 Sent Done Done false
23038 6420 Acked Write 10 Sent Done Done false
23039 6421 Acked Read 10 Sent Done Done false
23040 6422 Acked Read 5 Sent Done Done false
23041 6423 Acked Flush 0 Sent Done Done false
23042 6424 Acked Read 10 Sent Done Done false
23043 6425 Acked Flush 0 Sent Done Done false
23044 6426 Acked Write 8 Sent Done Done false
23045 6427 Acked Write 6 Sent Done Done false
23046 6428 Acked Read 9 Sent Done Done false
23047 6429 Acked Write 10 Sent Done Done false
23048 6430 Acked Write 3 Sent Done Done false
23049 6431 Acked Read 9 Sent Done Done false
23050 6432 Acked Read 2 Sent Done Done false
23051 6433 Acked Read 7 Sent Done Done false
23052 STATES DS:0 DS:1 DS:2 TOTAL
23053 Sent 301 0 0 301
23054 Done 0 301 301 602
23055 Skipped 0 0 0 0
23056 Error 0 0 0 0
23057 Last Flush: Some(JobId(6132)) Some(JobId(6425)) Some(JobId(6425))
23058 Downstairs last five completed: 6132 6131 6130 6129 6128
23059 Upstairs last five completed: 6433 6432 6431 6430 6429
23060 CLIENT: Up:0 ds:301 act:2
230612025-02-07T20:12:26.509ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
230622025-02-07T20:12:27.446ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
230632025-02-07T20:12:27.774ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
230642025-02-07T20:12:27.774ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41901) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
230652025-02-07T20:12:27.775ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
230662025-02-07T20:12:27.776ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
230672025-02-07T20:12:27.776ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
230682025-02-07T20:12:27.776ZINFOcrucible: send last flush ID to this DS: Some(JobId(6132)) = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
230692025-02-07T20:12:27.776ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
230702025-02-07T20:12:27.776ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(6132)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
230712025-02-07T20:12:27.776ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(6132)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
23072 ----------------------------------------------------------------
23073 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23074 Downstairs last five completed: 6434 6433 6432 6431 6430
23075 Upstairs last five completed: 6434 6433 6432 6431 6430
23076 CLIENT: Up:0 ds:0 act:3
23077 Replay: All jobs finished
23078 Main task has work to do, go find it
23079 got dsc Stop(0)
23080 stop 0
23081 [8810] Got stop action so:false kr:false
23082 Feb 07 20:12:30.010 INFO request completed, latency_us: 383, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 5c6447fb-5e8a-4c34-8412-f887f8ef8fc7, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
23083 Feb 07 20:12:30.011 INFO request completed, {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:12:30.01164788Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
23084 latency_us: 141, response_code: 200, uri: /state/cid/0, method: GET, req_id: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:12:30.011696746Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
23085 588b1e98-51de-41d0-9c43-65bd918ec81f, remote_addr: 127.0.0.{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:12:30.011715395Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
23086 1:43825, local_addr: 127.0.0.1:9998
230872025-02-07T20:12:30.011ZINFOcrucible: client task is exiting = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
23088 [8810] Exited with: ExitStatus(unix_wait_status(9))
23089 [8810][0] reports Exit
23090 Feb 07 20:12:34.014 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/0, method: GET, req_id: 63432c96-d6aa-428c-8486-3d0e63d5d50d, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
23091 001/300 Read block 1375 len 4096
23092 002/300 Read block 1324 len 28672
23093 003/300 Read block 3954 len 12288
23094 004/300 Read block 198 len 16384
23095 005/300 Read block 2585 len 24576
23096 006/300 Read block 4408 len 4096
23097 007/300 Flush
23098 008/300 Read block 682 len 28672
23099 009/300 Read block 3487 len 24576
23100 010/300 Write block 1006 len 24576 data: 5 4 3 2 2 3
23101 011/300 Write block 3064 len 8192 data: 3 4
23102 012/300 Write block 4439 len 4096 data: 5
23103 013/300 Write block 1198 len 36864 data: 3 3 4 4 4 3 3 3 3
23104 014/300 Read block 4618 len 24576
23105 015/300 Write block 1666 len 12288 data: 3 3 3
23106 016/300 Read block 4012 len 36864
23107 017/300 Read block 1185 len 20480
23108 018/300 Write block 2319 len 12288 data: 6 6 4
23109 019/300 Read block 2571 len 40960
23110 020/300 Read block 209 len 8192
23111 021/300 Flush
23112 022/300 Read block 4692 len 12288
23113 023/300 Read block 2661 len 16384
23114 024/300 Read block 4155 len 32768
23115 025/300 Read block 2047 len 12288
23116 026/300 Write block 104 len 32768 data: 4 4 3 2 3 3 4 4
23117 027/300 Write block 3155 len 8192 data: 5 6
23118 028/300 Read block 629 len 32768
23119 029/300 Write block 4039 len 40960 data: 5 5 5 6 6 7 6 3 3 3
23120 030/300 Read block 2782 len 28672
23121 031/300 Read block 3197 len 28672
23122 032/300 Write block 4244 len 28672 data: 3 4 5 5 6 5 6
23123 033/300 Write block 2551 len 36864 data: 2 2 2 2 3 3 3 2 3
23124 034/300 Read block 27 len 4096
23125 035/300 Write block 2763 len 28672 data: 4 3 3 3 4 4 4
23126 036/300 Read block 3481 len 40960
23127 037/300 Read block 211 len 28672
23128 038/300 Write block 4122 len 4096 data: 4
23129 039/300 Read block 4725 len 24576
23130 040/300 Write block 2001 len 40960 data: 6 6 7 8 7 6 5 6 5 4
23131 041/300 Read block 1396 len 16384
23132 042/300 Read block 4444 len 12288
23133 043/300 Read block 1059 len 8192
23134 044/300 Write block 2405 len 40960 data: 2 3 3 2 2 2 2 2 2 2
23135 045/300 Flush
23136 046/300 Write block 4154 len 8192 data: 3 3
23137 047/300 Write block 1629 len 8192 data: 2 3
23138 048/300 Read block 258 len 20480
23139 049/300 Read block 941 len 8192
23140 050/300 Read block 2218 len 4096
23141 051/300 Read block 42 len 8192
23142 052/300 Write block 1333 len 36864 data: 6 6 4 4 4 4 3 4 5
23143 053/300 Write block 2883 len 4096 data: 3
23144 054/300 Read block 3103 len 28672
23145 055/300 Read block 4409 len 36864
23146 056/300 Read block 3255 len 16384
23147 057/300 Write block 2708 len 12288 data: 5 3 4
23148 058/300 Read block 2749 len 24576
23149 059/300 Write block 2002 len 32768 data: 7 8 9 8 7 6 7 6
23150 060/300 Write block 619 len 8192 data: 4 5
23151 061/300 Write block 1995 len 4096 data: 5
23152 062/300 Read block 4480 len 4096
23153 063/300 Write block 4131 len 28672 data: 4 3 3 3 3 3 3
23154 064/300 Write block 4788 len 20480 data: 6 6 6 5 5
23155 065/300 Read block 1890 len 40960
23156 066/300 Read block 2 len 4096
23157 067/300 Flush
23158 068/300 Write block 4972 len 32768 data: 3 4 5 6 5 4 4 6
23159 069/300 Read block 210 len 24576
23160 070/300 Flush
23161 071/300 Read block 3010 len 36864
23162 072/300 Read block 4716 len 20480
23163 073/300 Read block 2074 len 32768
23164 074/300 Write block 1297 len 16384 data: 6 6 6 6
23165 075/300 Write block 1828 len 8192 data: 7 7
23166 076/300 Write block 1691 len 32768 data: 5 4 6 7 8 8 7 5
23167 077/300 Read block 2915 len 20480
23168 078/300 Write block 550 len 16384 data: 3 3 2 2
23169 079/300 Flush
23170 080/300 Read block 2174 len 12288
23171 081/300 Write block 2705 len 12288 data: 6 6 5
23172 082/300 Read block 2306 len 12288
23173 083/300 Read block 4424 len 36864
23174 084/300 Read block 3971 len 24576
23175 085/300 Read block 1596 len 32768
23176 086/300 Write block 4636 len 24576 data: 5 6 6 6 5 5
23177 087/300 Read block 1181 len 40960
23178 088/300 Write block 189 len 12288 data: 5 5 5
23179 089/300 Write block 237 len 24576 data: 5 5 6 6 4 3
23180 090/300 Read block 2126 len 16384
23181 091/300 Read block 1823 len 40960
23182 092/300 Read block 3528 len 8192
23183 093/300 Read block 1751 len 40960
23184 094/300 Read block 1354 len 20480
23185 095/300 Write block 221 len 8192 data: 6 6
23186 096/300 Write block 3115 len 12288 data: 3 4 3
23187 097/300 Read block 4915 len 36864
23188 098/300 Write block 555 len 32768 data: 2 3 5 5 5 6 6 6
23189 099/300 Read block 4186 len 24576
23190 100/300 Read block 976 len 16384
23191 101/300 Read block 3606 len 20480
23192 102/300 Read block 4460 len 36864
23193 103/300 Read block 3979 len 36864
23194 104/300 Flush
23195 105/300 Read block 4524 len 32768
23196 106/300 Write block 3994 len 8192 data: 5 7
23197 107/300 Read block 2419 len 28672
23198 108/300 Write block 3247 len 4096 data: 4
23199 109/300 Read block 106 len 4096
23200 110/300 Read block 444 len 40960
23201 111/300 Read block 66 len 4096
23202 112/300 Write block 1048 len 28672 data: 5 5 5 6 6 5 5
23203 113/300 Write block 4580 len 40960 data: 3 4 4 4 4 4 4 4 5 6
23204 114/300 Read block 427 len 4096
23205 115/300 Write block 645 len 12288 data: 7 6 5
23206 116/300 Write block 3069 len 40960 data: 4 4 4 5 5 4 5 4 4 4
23207 117/300 Write block 3500 len 4096 data: 4
23208 118/300 Write block 2930 len 20480 data: 3 5 5 6 6
23209 119/300 Write block 4544 len 12288 data: 2 2 2
23210 120/300 Write block 1440 len 24576 data: 9 6 5 4 4 3
23211 121/300 Read block 3912 len 12288
23212 122/300 Read block 3144 len 28672
23213 123/300 Read block 1436 len 4096
23214 124/300 Read block 145 len 8192
23215 125/300 Read block 231 len 16384
23216 126/300 Read block 3209 len 36864
23217 127/300 Read block 3640 len 4096
23218 128/300 Write block 3005 len 20480 data: 6 5 5 3 4
23219 129/300 Read block 2938 len 32768
23220 130/300 Write block 886 len 40960 data: 7 6 5 5 3 4 4 3 3 3
23221 131/300 Read block 4437 len 12288
23222 132/300 Read block 2800 len 8192
23223 133/300 Read block 3947 len 4096
23224 134/300 Flush
23225 135/300 Read block 3372 len 8192
23226 136/300 Read block 3433 len 24576
23227 137/300 Write block 3823 len 24576 data: 4 4 4 3 3 3
23228 138/300 Read block 3400 len 36864
23229 139/300 Write block 2926 len 12288 data: 3 4 3
23230 140/300 Flush
23231 141/300 Write block 4988 len 20480 data: 5 5 5 4 4
23232 142/300 Read block 1989 len 16384
23233 143/300 Write block 2762 len 16384 data: 3 5 4 4
23234 144/300 Read block 580 len 20480
23235 145/300 Write block 2031 len 40960 data: 3 3 3 4 5 4 4 4 3 3
23236 146/300 Write block 3183 len 36864 data: 3 3 3 3 3 3 3 3 3
23237 147/300 Read block 3506 len 24576
23238 148/300 Read block 4280 len 8192
23239 149/300 Read block 2889 len 12288
23240 150/300 Read block 1859 len 8192
23241 151/300 Read block 4449 len 12288
23242 152/300 Read block 2610 len 20480
23243 153/300 Read block 821 len 20480
23244 154/300 Read block 4095 len 32768
23245 155/300 Write block 2329 len 4096 data: 4
23246 156/300 Read block 179 len 4096
23247 157/300 Write block 943 len 16384 data: 4 3 3 4
23248 158/300 Read block 3901 len 36864
23249 159/300 Read block 1952 len 4096
23250 160/300 Write block 1720 len 4096 data: 3
23251 161/300 Read block 2249 len 24576
23252 162/300 Write block 280 len 8192 data: 5 4
23253 163/300 Write block 1314 len 12288 data: 4 3 5
23254 164/300 Read block 1277 len 40960
23255 165/300 Read block 2276 len 4096
23256 166/300 Read block 3835 len 12288
23257 167/300 Read block 3031 len 28672
23258 168/300 Write block 2297 len 16384 data: 4 4 4 3
23259 169/300 Write block 4545 len 16384 data: 3 3 2 2
23260 170/300 Write block 3079 len 40960 data: 5 5 5 5 5 5 3 3 3 4
23261 171/300 Write block 1604 len 24576 data: 3 3 5 4 4 5
23262 172/300 Read block 4372 len 36864
23263 173/300 Flush
23264 174/300 Flush
23265 175/300 Flush
23266 176/300 Write block 4519 len 4096 data: 5
23267 177/300 Read block 546 len 40960
23268 178/300 Read block 3340 len 12288
23269 179/300 Write block 839 len 20480 data: 4 4 4 5 4
23270 180/300 Read block 1784 len 4096
23271 181/300 Read block 133 len 12288
23272 182/300 Read block 1045 len 32768
23273 183/300 Read block 3313 len 12288
23274 184/300 Write block 1992 len 36864 data: 4 5 5 6 5 6 7 10 7
23275 185/300 Read block 4119 len 12288
23276 186/300 Flush
23277 187/300 Write block 2093 len 32768 data: 5 5 5 6 5 6 6 6
23278 188/300 Write block 3916 len 20480 data: 6 5 6 7 6
23279 189/300 Read block 3074 len 36864
23280 190/300 Read block 85 len 16384
23281 191/300 Read block 3889 len 24576
23282 192/300 Write block 87 len 32768 data: 3 3 3 2 2 3 4 3
23283 193/300 Read block 1338 len 32768
23284 194/300 Write block 4472 len 40960 data: 9 7 6 9 9 8 9 7 7 5
23285 195/300 Read block 1084 len 28672
23286 196/300 Write block 4059 len 40960 data: 4 7 9 7 6 5 5 5 5 5
23287 197/300 Read block 2023 len 32768
23288 198/300 Read block 3871 len 12288
23289 199/300 Write block 1396 len 24576 data: 5 6 6 5 6 5
23290 200/300 Read block 1729 len 8192
23291 201/300 Read block 2120 len 12288
23292 202/300 Read block 4321 len 24576
23293 203/300 Read block 924 len 32768
23294 204/300 Flush
23295 205/300 Flush
23296 206/300 Write block 4189 len 24576 data: 5 5 6 6 7 5
23297 207/300 Read block 2289 len 4096
23298 208/300 Read block 3651 len 4096
23299 209/300 Write block 3673 len 12288 data: 5 5 5
23300 210/300 Write block 2046 len 20480 data: 3 3 2 2 2
23301 211/300 Write block 3275 len 4096 data: 8
23302 212/300 Flush
23303 213/300 Write block 3765 len 12288 data: 4 4 5
23304 214/300 Write block 343 len 4096 data: 3
23305 215/300 Read block 2409 len 40960
23306 216/300 Flush
23307 217/300 Read block 3212 len 40960
23308 218/300 Read block 4605 len 12288
23309 219/300 Write block 4538 len 20480 data: 4 5 5 5 3
23310 220/300 Read block 2635 len 28672
23311 221/300 Flush
23312 222/300 Read block 895 len 32768
23313 223/300 Read block 936 len 4096
23314 224/300 Read block 3948 len 20480
23315 225/300 Flush
23316 226/300 Flush
23317 227/300 Write block 3574 len 12288 data: 5 5 6
23318 228/300 Read block 667 len 4096
23319 229/300 Write block 2685 len 8192 data: 6 6
23320 230/300 Read block 249 len 8192
23321 231/300 Write block 3528 len 32768 data: 4 4 4 3 3 4 4 5
23322 232/300 Read block 1897 len 24576
23323 233/300 Write block 52 len 8192 data: 3 3
23324 234/300 Read block 3568 len 8192
23325 235/300 Read block 2559 len 8192
23326 236/300 Write block 389 len 12288 data: 5 5 4
23327 237/300 Read block 301 len 4096
23328 238/300 Write block 1718 len 36864 data: 5 4 4 3 3 3 3 2 2
23329 239/300 Read block 2708 len 32768
23330 240/300 Read block 4915 len 20480
23331 241/300 Read block 1885 len 4096
23332 242/300 Read block 589 len 12288
23333 243/300 Read block 917 len 28672
23334 244/300 Write block 3940 len 8192 data: 2 2
23335 245/300 Read block 3604 len 36864
23336 246/300 Read block 4596 len 40960
23337 247/300 Flush
23338 248/300 Flush
23339 249/300 Read block 3085 len 8192
23340 250/300 Flush
23341 251/300 Write block 699 len 32768 data: 5 4 4 4 3 4 4 5
23342 252/300 Read block 1966 len 36864
23343 253/300 Write block 1186 len 16384 data: 4 6 5 5
23344 254/300 Read block 1931 len 40960
23345 255/300 Write block 4940 len 4096 data: 5
23346 256/300 Read block 1306 len 16384
23347 257/300 Read block 550 len 24576
23348 258/300 Read block 4003 len 16384
23349 259/300 Write block 4133 len 4096 data: 4
23350 260/300 Write block 4184 len 24576 data: 3 3 3 3 3 6
23351 261/300 Write block 2860 len 12288 data: 6 7 7
23352 262/300 Read block 667 len 16384
23353 263/300 Write block 1843 len 4096 data: 3
23354 264/300 Read block 3806 len 28672
23355 265/300 Write block 194 len 12288 data: 4 2 2
23356 266/300 Read block 2810 len 32768
23357 267/300 Write block 1045 len 20480 data: 7 6 6 6 6
23358 268/300 Read block 803 len 12288
23359 269/300 Read block 1054 len 32768
23360 270/300 Write block 2837 len 20480 data: 4 4 5 6 5
23361 271/300 Write block 966 len 4096 data: 5
23362 272/300 Read block 121 len 32768
23363 273/300 Read block 3109 len 12288
23364 274/300 Write block 1701 len 40960 data: 5 5 4 4 3 4 3 3 4 4
23365 275/300 Read block 642 len 4096
23366 276/300 Write block 1465 len 40960 data: 2 3 3 3 3 3 3 4 4 5
23367 277/300 Read block 3115 len 16384
23368 278/300 Read block 172 len 8192
23369 279/300 Read block 1984 len 12288
23370 280/300 Read block 4518 len 20480
23371 281/300 Write block 3760 len 20480 data: 5 5 5 5 4
23372 282/300 Write block 4498 len 36864 data: 9 9 9 9 10 7 8 6 6
23373 283/300 Read block 4906 len 28672
23374 284/300 Flush
23375 285/300 Flush
23376 286/300 Read block 2755 len 8192
23377 287/300 Write block 4500 len 16384 data: 10 10 11 8
23378 288/300 Write block 3742 len 32768 data: 3 3 4 3 5 5 5 8
23379 289/300 Read block 2118 len 32768
23380 290/300 Write block 877 len 32768 data: 2 2 2 2 2 4 5 5
23381 291/300 Read block 4217 len 32768
23382 292/300 Read block 3959 len 12288
23383 293/300 Read block 4494 len 36864
23384 294/300 Read block 378 len 12288
23385 295/300 Read block 3212 len 16384
23386 296/300 Read block 3647 len 28672
23387 297/300 Read block 1996 len 24576
23388 298/300 Write block 3628 len 12288 data: 3 3 3
23389 299/300 Write block 4573 len 20480 data: 2 2 2 2 2
23390 300/300 Flush
23391 301/300 Read block 4930 len 16384
23392 Main task has work to do, go find it
23393 got dsc Start(0)
23394 start 0
23395 Feb 07 20:12:34.258 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/0, method: POST, req_id: 15f00583-6cf7-434e-917f-58e2edb5dee1, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
23396 Replay: started 0, returned:Ok(())
23397 ----------------------------------------------------------------
23398 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23399 States: Offline Active Active
23400 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23401 6435 Acked Read 1 Sent Done Done false
23402 6436 Acked Read 7 Sent Done Done false
23403 6437 Acked Read 3 Sent Done Done false
23404 6438 Acked Read 4 Sent Done Done false
23405 6439 Acked Read 6 Sent Done Done false
23406 6440 Acked Read 1 Sent Done Done false
23407 6441 Acked Flush 0 Sent Done Done false
23408 6442 Acked Read 7 Sent Done Done false
23409 6443 Acked Read 6 Sent Done Done false
23410 6444 Acked Write 6 Sent Done Done false
23411 6445 Acked Write 2 Sent Done Done false
23412 6446 Acked Write 1 Sent Done Done false
23413 6447 Acked Write 9 Sent Done Done false
23414 6448 Acked Read 6 Sent Done Done false
23415 6449 Acked Write 3 Sent Done Done false
23416 6450 Acked Read 9 Sent Done Done false
23417 6451 Acked Read 5 Sent Done Done false
23418 6452 Acked Write 3 Sent Done Done false
23419 6453 Acked Read 10 Sent Done Done false
23420 6454 Acked Read 2 Sent Done Done false
23421 6455 Acked Flush 0 Sent Done Done false
23422 6456 Acked Read 3 Sent Done Done false
23423 6457 Acked Read 4 Sent Done Done false
23424 6458 Acked Read 8 Sent Done Done false
23425 6459 Acked Read 3 Sent Done Done false
23426 6460 Acked Write 8 Sent Done Done false
23427 6461 Acked Write 2 Sent Done Done false
23428 6462 Acked Read 8 Sent Done Done false
23429 6463 Acked Write 10 Sent Done Done false
23430 6464 Acked Read 7 Sent Done Done false
23431 6465 Acked Read 7 Sent Done Done false
23432 6466 Acked Write 7 Sent Done Done false
23433 6467 Acked Write 9 Sent Done Done false
23434 6468 Acked Read 1 Sent Done Done false
23435 6469 Acked Write 7 Sent Done Done false
23436 6470 Acked Read 10 Sent Done Done false
23437 6471 Acked Read 7 Sent Done Done false
23438 6472 Acked Write 1 Sent Done Done false
23439 6473 Acked Read 6 Sent Done Done false
23440 6474 Acked Write 10 Sent Done Done false
23441 6475 Acked Read 4 Sent Done Done false
23442 6476 Acked Read 3 Sent Done Done false
23443 6477 Acked Read 2 Sent Done Done false
23444 6478 Acked Write 10 Sent Done Done false
23445 6479 Acked Flush 0 Sent Done Done false
23446 6480 Acked Write 2 Sent Done Done false
23447 6481 Acked Write 2 Sent Done Done false
23448 6482 Acked Read 5 Sent Done Done false
23449 6483 Acked Read 2 Sent Done Done false
23450 6484 Acked Read 1 Sent Done Done false
23451 6485 Acked Read 2 Sent Done Done false
23452 6486 Acked Write 9 Sent Done Done false
23453 6487 Acked Write 1 Sent Done Done false
23454 6488 Acked Read 7 Sent Done Done false
23455 6489 Acked Read 9 Sent Done Done false
23456 6490 Acked Read 4 Sent Done Done false
23457 6491 Acked Write 3 Sent Done Done false
23458 6492 Acked Read 6 Sent Done Done false
23459 6493 Acked Write 8 Sent Done Done false
23460 6494 Acked Write 2 Sent Done Done false
23461 6495 Acked Write 1 Sent Done Done false
23462 6496 Acked Read 1 Sent Done Done false
23463 6497 Acked Write 7 Sent Done Done false
23464 6498 Acked Write 5 Sent Done Done false
23465 6499 Acked Read 10 Sent Done Done false
23466 6500 Acked Read 1 Sent Done Done false
23467 6501 Acked Flush 0 Sent Done Done false
23468 6502 Acked Write 8 Sent Done Done false
23469 6503 Acked Read 6 Sent Done Done false
23470 6504 Acked Flush 0 Sent Done Done false
23471 6505 Acked Read 9 Sent Done Done false
23472 6506 Acked Read 5 Sent Done Done false
23473 6507 Acked Read 8 Sent Done Done false
23474 6508 Acked Write 4 Sent Done Done false
23475 6509 Acked Write 2 Sent Done Done false
23476 6510 Acked Write 8 Sent Done Done false
23477 6511 Acked Read 5 Sent Done Done false
23478 6512 Acked Write 4 Sent Done Done false
23479 6513 Acked Flush 0 Sent Done Done false
23480 6514 Acked Read 3 Sent Done Done false
23481 6515 Acked Write 3 Sent Done Done false
23482 6516 Acked Read 3 Sent Done Done false
23483 6517 Acked Read 9 Sent Done Done false
23484 6518 Acked Read 6 Sent Done Done false
23485 6519 Acked Read 8 Sent Done Done false
23486 6520 Acked Write 6 Sent Done Done false
23487 6521 Acked Read 10 Sent Done Done false
23488 6522 Acked Write 3 Sent Done Done false
23489 6523 Acked Write 6 Sent Done Done false
23490 6524 Acked Read 4 Sent Done Done false
23491 6525 Acked Read 10 Sent Done Done false
23492 6526 Acked Read 2 Sent Done Done false
23493 6527 Acked Read 10 Sent Done Done false
23494 6528 Acked Read 5 Sent Done Done false
23495 6529 Acked Write 2 Sent Done Done false
23496 6530 Acked Write 3 Sent Done Done false
23497 6531 Acked Read 9 Sent Done Done false
23498 6532 Acked Write 8 Sent Done Done false
23499 6533 Acked Read 6 Sent Done Done false
23500 6534 Acked Read 4 Sent Done Done false
23501 6535 Acked Read 5 Sent Done Done false
23502 6536 Acked Read 9 Sent Done Done false
23503 6537 Acked Read 9 Sent Done Done false
23504 6538 Acked Flush 0 Sent Done Done false
23505 6539 Acked Read 8 Sent Done Done false
23506 6540 Acked Write 2 Sent Done Done false
23507 6541 Acked Read 7 Sent Done Done false
23508 6542 Acked Write 1 Sent Done Done false
23509 6543 Acked Read 1 Sent Done Done false
23510 6544 Acked Read 10 Sent Done Done false
23511 6545 Acked Read 1 Sent Done Done false
23512 6546 Acked Write 7 Sent Done Done false
23513 6547 Acked Write 10 Sent Done Done false
23514 6548 Acked Read 1 Sent Done Done false
23515 6549 Acked Write 3 Sent Done Done false
23516 6550 Acked Write 10 Sent Done Done false
23517 6551 Acked Write 1 Sent Done Done false
23518 6552 Acked Write 5 Sent Done Done false
23519 6553 Acked Write 3 Sent Done Done false
23520 6554 Acked Write 6 Sent Done Done false
23521 6555 Acked Read 3 Sent Done Done false
23522 6556 Acked Read 7 Sent Done Done false
23523 6557 Acked Read 1 Sent Done Done false
23524 6558 Acked Read 2 Sent Done Done false
23525 6559 Acked Read 4 Sent Done Done false
23526 6560 Acked Read 9 Sent Done Done false
23527 6561 Acked Read 1 Sent Done Done false
23528 6562 Acked Write 5 Sent Done Done false
23529 6563 Acked Read 8 Sent Done Done false
23530 6564 Acked Write 10 Sent Done Done false
23531 6565 Acked Read 3 Sent Done Done false
23532 6566 Acked Read 2 Sent Done Done false
23533 6567 Acked Read 1 Sent Done Done false
23534 6568 Acked Flush 0 Sent Done Done false
23535 6569 Acked Read 2 Sent Done Done false
23536 6570 Acked Read 6 Sent Done Done false
23537 6571 Acked Write 6 Sent Done Done false
23538 6572 Acked Read 9 Sent Done Done false
23539 6573 Acked Write 3 Sent Done Done false
23540 6574 Acked Flush 0 Sent Done Done false
23541 6575 Acked Write 5 Sent Done Done false
23542 6576 Acked Read 4 Sent Done Done false
23543 6577 Acked Write 4 Sent Done Done false
23544 6578 Acked Read 5 Sent Done Done false
23545 6579 Acked Write 10 Sent Done Done false
23546 6580 Acked Write 9 Sent Done Done false
23547 6581 Acked Read 6 Sent Done Done false
23548 6582 Acked Read 2 Sent Done Done false
23549 6583 Acked Read 3 Sent Done Done false
23550 6584 Acked Read 2 Sent Done Done false
23551 6585 Acked Read 3 Sent Done Done false
23552 6586 Acked Read 5 Sent Done Done false
23553 6587 Acked Read 5 Sent Done Done false
23554 6588 Acked Read 8 Sent Done Done false
23555 6589 Acked Write 1 Sent Done Done false
23556 6590 Acked Read 1 Sent Done Done false
23557 6591 Acked Write 4 Sent Done Done false
23558 6592 Acked Read 9 Sent Done Done false
23559 6593 Acked Read 1 Sent Done Done false
23560 6594 Acked Write 1 Sent Done Done false
23561 6595 Acked Read 6 Sent Done Done false
23562 6596 Acked Write 2 Sent Done Done false
23563 6597 Acked Write 3 Sent Done Done false
23564 6598 Acked Read 10 Sent Done Done false
23565 6599 Acked Read 1 Sent Done Done false
23566 6600 Acked Read 3 Sent Done Done false
23567 6601 Acked Read 7 Sent Done Done false
23568 6602 Acked Write 4 Sent Done Done false
23569 6603 Acked Write 4 Sent Done Done false
23570 6604 Acked Write 10 Sent Done Done false
23571 6605 Acked Write 6 Sent Done Done false
23572 6606 Acked Read 9 Sent Done Done false
23573 6607 Acked Flush 0 Sent Done Done false
23574 6608 Acked Flush 0 Sent Done Done false
23575 6609 Acked Flush 0 Sent Done Done false
23576 6610 Acked Write 1 Sent Done Done false
23577 6611 Acked Read 10 Sent Done Done false
23578 6612 Acked Read 3 Sent Done Done false
23579 6613 Acked Write 5 Sent Done Done false
23580 6614 Acked Read 1 Sent Done Done false
23581 6615 Acked Read 3 Sent Done Done false
23582 6616 Acked Read 8 Sent Done Done false
23583 6617 Acked Read 3 Sent Done Done false
23584 6618 Acked Write 9 Sent Done Done false
23585 6619 Acked Read 3 Sent Done Done false
23586 6620 Acked Flush 0 Sent Done Done false
23587 6621 Acked Write 8 Sent Done Done false
23588 6622 Acked Write 5 Sent Done Done false
23589 6623 Acked Read 9 Sent Done Done false
23590 6624 Acked Read 4 Sent Done Done false
23591 6625 Acked Read 6 Sent Done Done false
23592 6626 Acked Write 8 Sent Done Done false
23593 6627 Acked Read 8 Sent Done Done false
23594 6628 Acked Write 10 Sent Done Done false
23595 6629 Acked Read 7 Sent Done Done false
23596 6630 Acked Write 10 Sent Done Done false
23597 6631 Acked Read 8 Sent Done Done false
23598 6632 Acked Read 3 Sent Done Done false
23599 6633 Acked Write 6 Sent Done Done false
23600 6634 Acked Read 2 Sent Done Done false
23601 6635 Acked Read 3 Sent Done Done false
23602 6636 Acked Read 6 Sent Done Done false
23603 6637 Acked Read 8 Sent Done Done false
23604 6638 Acked Flush 0 Sent Done Done false
23605 6639 Acked Flush 0 Sent Done Done false
23606 6640 Acked Write 6 Sent Done Done false
23607 6641 Acked Read 1 Sent Done Done false
23608 6642 Acked Read 1 Sent Done Done false
23609 6643 Acked Write 3 Sent Done Done false
23610 6644 Acked Write 5 Sent Done Done false
23611 6645 Acked Write 1 Sent Done Done false
23612 6646 Acked Flush 0 Sent Done Done false
23613 6647 Acked Write 3 Sent Done Done false
23614 6648 Acked Write 1 Sent Done Done false
23615 6649 Acked Read 10 Sent Done Done false
23616 6650 Acked Flush 0 Sent Done Done false
23617 6651 Acked Read 10 Sent Done Done false
23618 6652 Acked Read 3 Sent Done Done false
23619 6653 Acked Write 5 Sent Done Done false
23620 6654 Acked Read 7 Sent Done Done false
23621 6655 Acked Flush 0 Sent Done Done false
23622 6656 Acked Read 8 Sent Done Done false
23623 6657 Acked Read 1 Sent Done Done false
23624 6658 Acked Read 5 Sent Done Done false
23625 6659 Acked Flush 0 Sent Done Done false
23626 6660 Acked Flush 0 Sent Done Done false
23627 6661 Acked Write 3 Sent Done Done false
23628 6662 Acked Read 1 Sent Done Done false
23629 6663 Acked Write 2 Sent Done Done false
23630 6664 Acked Read 2 Sent Done Done false
23631 6665 Acked Write 8 Sent Done Done false
23632 6666 Acked Read 6 Sent Done Done false
23633 6667 Acked Write 2 Sent Done Done false
23634 6668 Acked Read 2 Sent Done Done false
23635 6669 Acked Read 2 Sent Done Done false
23636 6670 Acked Write 3 Sent Done Done false
23637 6671 Acked Read 1 Sent Done Done false
23638 6672 Acked Write 9 Sent Done Done false
23639 6673 Acked Read 8 Sent Done Done false
23640 6674 Acked Read 5 Sent Done Done false
23641 6675 Acked Read 1 Sent Done Done false
23642 6676 Acked Read 3 Sent Done Done false
23643 6677 Acked Read 7 Sent Done Done false
23644 6678 Acked Write 2 Sent Done Done false
23645 6679 Acked Read 9 Sent Done Done false
23646 6680 Acked Read 10 Sent Done Done false
23647 6681 Acked Flush 0 Sent Done Done false
23648 6682 Acked Flush 0 Sent Done Done false
23649 6683 Acked Read 2 Sent Done Done false
23650 6684 Acked Flush 0 Sent Done Done false
23651 6685 Acked Write 8 Sent Done Done false
23652 6686 Acked Read 9 Sent Done Done false
23653 6687 Acked Write 4 Sent Done Done false
23654 6688 Acked Read 10 Sent Done Done false
23655 6689 Acked Write 1 Sent Done Done false
23656 6690 Acked Read 4 Sent Done Done false
23657 6691 Acked Read 6 Sent Done Done false
23658 6692 Acked Read 4 Sent Done Done false
23659 6693 Acked Write 1 Sent Done Done false
23660 6694 Acked Write 6 Sent Done Done false
23661 6695 Acked Write 3 Sent Done Done false
23662 6696 Acked Read 4 Sent Done Done false
23663 6697 Acked Write 1 Sent Done Done false
23664 6698 Acked Read 7 Sent Done Done false
23665 6699 Acked Write 3 Sent Done Done false
23666 6700 Acked Read 8 Sent Done Done false
23667 6701 Acked Write 5 Sent Done Done false
23668 6702 Acked Read 3 Sent Done Done false
23669 6703 Acked Read 8 Sent Done Done false
23670 6704 Acked Write 5 Sent Done Done false
23671 6705 Acked Write 1 Sent Done Done false
23672 6706 Acked Read 8 Sent Done Done false
23673 6707 Acked Read 3 Sent Done Done false
23674 6708 Acked Write 10 Sent Done Done false
23675 6709 Acked Read 1 Sent Done Done false
23676 6710 Acked Write 10 Sent Done Done false
23677 6711 Acked Read 4 Sent Done Done false
23678 6712 Acked Read 2 Sent Done Done false
23679 6713 Acked Read 3 Sent Done Done false
23680 6714 Acked Read 5 Sent Done Done false
23681 6715 Acked Write 5 Sent Done Done false
23682 6716 Acked Write 9 Sent Done Done false
23683 6717 Acked Read 7 Sent Done Done false
23684 6718 Acked Flush 0 Sent Done Done false
23685 6719 Acked Flush 0 Sent Done Done false
23686 6720 Acked Read 2 Sent Done Done false
23687 6721 Acked Write 4 Sent Done Done false
23688 6722 Acked Write 8 Sent Done Done false
23689 6723 Acked Read 8 Sent Done Done false
23690 6724 Acked Write 8 Sent Done Done false
23691 6725 Acked Read 8 Sent Done Done false
23692 6726 Acked Read 3 Sent Done Done false
23693 6727 Acked Read 9 Sent Done Done false
23694 6728 Acked Read 3 Sent Done Done false
23695 6729 Acked Read 4 Sent Done Done false
23696 6730 Acked Read 7 Sent Done Done false
23697 6731 Acked Read 6 Sent Done Done false
23698 6732 Acked Write 3 Sent Done Done false
23699 6733 Acked Write 5 Sent Done Done false
23700 6734 Acked Flush 0 Sent Done Done false
23701 6735 Acked Read 4 Sent Done Done false
23702 STATES DS:0 DS:1 DS:2 TOTAL
23703 Sent 301 0 0 301
23704 Done 0 301 301 602
23705 Skipped 0 0 0 0
23706 Error 0 0 0 0
23707 Last Flush: Some(JobId(6434)) Some(JobId(6734)) Some(JobId(6734))
23708 Downstairs last five completed: 6434 6433 6432 6431 6430
23709 Upstairs last five completed: 6735 6734 6733 6732 6731
23710 CLIENT: Up:0 ds:301 act:2
237112025-02-07T20:12:34.760ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
237122025-02-07T20:12:35.460ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
23713 [8810] got start action
23714 [8810] I am going to restart
23715 Starting downstairs at port 8810
23716 Make output file at "/tmp/dsc/downstairs-8810.txt"
23717 [8810][0] reports Starting
23718 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1228)
23719 [8810][0] reports Running
237202025-02-07T20:12:36.462ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
237212025-02-07T20:12:37.464ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
23722 ----------------------------------------------------------------
23723 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23724 States: Offline Active Active
23725 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23726 6435 Acked Read 1 Sent Done Done false
23727 6436 Acked Read 7 Sent Done Done false
23728 6437 Acked Read 3 Sent Done Done false
23729 6438 Acked Read 4 Sent Done Done false
23730 6439 Acked Read 6 Sent Done Done false
23731 6440 Acked Read 1 Sent Done Done false
23732 6441 Acked Flush 0 Sent Done Done false
23733 6442 Acked Read 7 Sent Done Done false
23734 6443 Acked Read 6 Sent Done Done false
23735 6444 Acked Write 6 Sent Done Done false
23736 6445 Acked Write 2 Sent Done Done false
23737 6446 Acked Write 1 Sent Done Done false
23738 6447 Acked Write 9 Sent Done Done false
23739 6448 Acked Read 6 Sent Done Done false
23740 6449 Acked Write 3 Sent Done Done false
23741 6450 Acked Read 9 Sent Done Done false
23742 6451 Acked Read 5 Sent Done Done false
23743 6452 Acked Write 3 Sent Done Done false
23744 6453 Acked Read 10 Sent Done Done false
23745 6454 Acked Read 2 Sent Done Done false
23746 6455 Acked Flush 0 Sent Done Done false
23747 6456 Acked Read 3 Sent Done Done false
23748 6457 Acked Read 4 Sent Done Done false
23749 6458 Acked Read 8 Sent Done Done false
23750 6459 Acked Read 3 Sent Done Done false
23751 6460 Acked Write 8 Sent Done Done false
23752 6461 Acked Write 2 Sent Done Done false
23753 6462 Acked Read 8 Sent Done Done false
23754 6463 Acked Write 10 Sent Done Done false
23755 6464 Acked Read 7 Sent Done Done false
23756 6465 Acked Read 7 Sent Done Done false
23757 6466 Acked Write 7 Sent Done Done false
23758 6467 Acked Write 9 Sent Done Done false
23759 6468 Acked Read 1 Sent Done Done false
23760 6469 Acked Write 7 Sent Done Done false
23761 6470 Acked Read 10 Sent Done Done false
23762 6471 Acked Read 7 Sent Done Done false
23763 6472 Acked Write 1 Sent Done Done false
23764 6473 Acked Read 6 Sent Done Done false
23765 6474 Acked Write 10 Sent Done Done false
23766 6475 Acked Read 4 Sent Done Done false
23767 6476 Acked Read 3 Sent Done Done false
23768 6477 Acked Read 2 Sent Done Done false
23769 6478 Acked Write 10 Sent Done Done false
23770 6479 Acked Flush 0 Sent Done Done false
23771 6480 Acked Write 2 Sent Done Done false
23772 6481 Acked Write 2 Sent Done Done false
23773 6482 Acked Read 5 Sent Done Done false
23774 6483 Acked Read 2 Sent Done Done false
23775 6484 Acked Read 1 Sent Done Done false
23776 6485 Acked Read 2 Sent Done Done false
23777 6486 Acked Write 9 Sent Done Done false
23778 6487 Acked Write 1 Sent Done Done false
23779 6488 Acked Read 7 Sent Done Done false
23780 6489 Acked Read 9 Sent Done Done false
23781 6490 Acked Read 4 Sent Done Done false
23782 6491 Acked Write 3 Sent Done Done false
23783 6492 Acked Read 6 Sent Done Done false
23784 6493 Acked Write 8 Sent Done Done false
23785 6494 Acked Write 2 Sent Done Done false
23786 6495 Acked Write 1 Sent Done Done false
23787 6496 Acked Read 1 Sent Done Done false
23788 6497 Acked Write 7 Sent Done Done false
23789 6498 Acked Write 5 Sent Done Done false
23790 6499 Acked Read 10 Sent Done Done false
23791 6500 Acked Read 1 Sent Done Done false
23792 6501 Acked Flush 0 Sent Done Done false
23793 6502 Acked Write 8 Sent Done Done false
23794 6503 Acked Read 6 Sent Done Done false
23795 6504 Acked Flush 0 Sent Done Done false
23796 6505 Acked Read 9 Sent Done Done false
23797 6506 Acked Read 5 Sent Done Done false
23798 6507 Acked Read 8 Sent Done Done false
23799 6508 Acked Write 4 Sent Done Done false
23800 6509 Acked Write 2 Sent Done Done false
23801 6510 Acked Write 8 Sent Done Done false
23802 6511 Acked Read 5 Sent Done Done false
23803 6512 Acked Write 4 Sent Done Done false
23804 6513 Acked Flush 0 Sent Done Done false
23805 6514 Acked Read 3 Sent Done Done false
23806 6515 Acked Write 3 Sent Done Done false
23807 6516 Acked Read 3 Sent Done Done false
23808 6517 Acked Read 9 Sent Done Done false
23809 6518 Acked Read 6 Sent Done Done false
23810 6519 Acked Read 8 Sent Done Done false
23811 6520 Acked Write 6 Sent Done Done false
23812 6521 Acked Read 10 Sent Done Done false
23813 6522 Acked Write 3 Sent Done Done false
23814 6523 Acked Write 6 Sent Done Done false
23815 6524 Acked Read 4 Sent Done Done false
23816 6525 Acked Read 10 Sent Done Done false
23817 6526 Acked Read 2 Sent Done Done false
23818 6527 Acked Read 10 Sent Done Done false
23819 6528 Acked Read 5 Sent Done Done false
23820 6529 Acked Write 2 Sent Done Done false
23821 6530 Acked Write 3 Sent Done Done false
23822 6531 Acked Read 9 Sent Done Done false
23823 6532 Acked Write 8 Sent Done Done false
23824 6533 Acked Read 6 Sent Done Done false
23825 6534 Acked Read 4 Sent Done Done false
23826 6535 Acked Read 5 Sent Done Done false
23827 6536 Acked Read 9 Sent Done Done false
23828 6537 Acked Read 9 Sent Done Done false
23829 6538 Acked Flush 0 Sent Done Done false
23830 6539 Acked Read 8 Sent Done Done false
23831 6540 Acked Write 2 Sent Done Done false
23832 6541 Acked Read 7 Sent Done Done false
23833 6542 Acked Write 1 Sent Done Done false
23834 6543 Acked Read 1 Sent Done Done false
23835 6544 Acked Read 10 Sent Done Done false
23836 6545 Acked Read 1 Sent Done Done false
23837 6546 Acked Write 7 Sent Done Done false
23838 6547 Acked Write 10 Sent Done Done false
23839 6548 Acked Read 1 Sent Done Done false
23840 6549 Acked Write 3 Sent Done Done false
23841 6550 Acked Write 10 Sent Done Done false
23842 6551 Acked Write 1 Sent Done Done false
23843 6552 Acked Write 5 Sent Done Done false
23844 6553 Acked Write 3 Sent Done Done false
23845 6554 Acked Write 6 Sent Done Done false
23846 6555 Acked Read 3 Sent Done Done false
23847 6556 Acked Read 7 Sent Done Done false
23848 6557 Acked Read 1 Sent Done Done false
23849 6558 Acked Read 2 Sent Done Done false
23850 6559 Acked Read 4 Sent Done Done false
23851 6560 Acked Read 9 Sent Done Done false
23852 6561 Acked Read 1 Sent Done Done false
23853 6562 Acked Write 5 Sent Done Done false
23854 6563 Acked Read 8 Sent Done Done false
23855 6564 Acked Write 10 Sent Done Done false
23856 6565 Acked Read 3 Sent Done Done false
23857 6566 Acked Read 2 Sent Done Done false
23858 6567 Acked Read 1 Sent Done Done false
23859 6568 Acked Flush 0 Sent Done Done false
23860 6569 Acked Read 2 Sent Done Done false
23861 6570 Acked Read 6 Sent Done Done false
23862 6571 Acked Write 6 Sent Done Done false
23863 6572 Acked Read 9 Sent Done Done false
23864 6573 Acked Write 3 Sent Done Done false
23865 6574 Acked Flush 0 Sent Done Done false
23866 6575 Acked Write 5 Sent Done Done false
23867 6576 Acked Read 4 Sent Done Done false
23868 6577 Acked Write 4 Sent Done Done false
23869 6578 Acked Read 5 Sent Done Done false
23870 6579 Acked Write 10 Sent Done Done false
23871 6580 Acked Write 9 Sent Done Done false
23872 6581 Acked Read 6 Sent Done Done false
23873 6582 Acked Read 2 Sent Done Done false
23874 6583 Acked Read 3 Sent Done Done false
23875 6584 Acked Read 2 Sent Done Done false
23876 6585 Acked Read 3 Sent Done Done false
23877 6586 Acked Read 5 Sent Done Done false
23878 6587 Acked Read 5 Sent Done Done false
23879 6588 Acked Read 8 Sent Done Done false
23880 6589 Acked Write 1 Sent Done Done false
23881 6590 Acked Read 1 Sent Done Done false
23882 6591 Acked Write 4 Sent Done Done false
23883 6592 Acked Read 9 Sent Done Done false
23884 6593 Acked Read 1 Sent Done Done false
23885 6594 Acked Write 1 Sent Done Done false
23886 6595 Acked Read 6 Sent Done Done false
23887 6596 Acked Write 2 Sent Done Done false
23888 6597 Acked Write 3 Sent Done Done false
23889 6598 Acked Read 10 Sent Done Done false
23890 6599 Acked Read 1 Sent Done Done false
23891 6600 Acked Read 3 Sent Done Done false
23892 6601 Acked Read 7 Sent Done Done false
23893 6602 Acked Write 4 Sent Done Done false
23894 6603 Acked Write 4 Sent Done Done false
23895 6604 Acked Write 10 Sent Done Done false
23896 6605 Acked Write 6 Sent Done Done false
23897 6606 Acked Read 9 Sent Done Done false
23898 6607 Acked Flush 0 Sent Done Done false
23899 6608 Acked Flush 0 Sent Done Done false
23900 6609 Acked Flush 0 Sent Done Done false
23901 6610 Acked Write 1 Sent Done Done false
23902 6611 Acked Read 10 Sent Done Done false
23903 6612 Acked Read 3 Sent Done Done false
23904 6613 Acked Write 5 Sent Done Done false
23905 6614 Acked Read 1 Sent Done Done false
23906 6615 Acked Read 3 Sent Done Done false
23907 6616 Acked Read 8 Sent Done Done false
23908 6617 Acked Read 3 Sent Done Done false
23909 6618 Acked Write 9 Sent Done Done false
23910 6619 Acked Read 3 Sent Done Done false
23911 6620 Acked Flush 0 Sent Done Done false
23912 6621 Acked Write 8 Sent Done Done false
23913 6622 Acked Write 5 Sent Done Done false
23914 6623 Acked Read 9 Sent Done Done false
23915 6624 Acked Read 4 Sent Done Done false
23916 6625 Acked Read 6 Sent Done Done false
23917 6626 Acked Write 8 Sent Done Done false
23918 6627 Acked Read 8 Sent Done Done false
23919 6628 Acked Write 10 Sent Done Done false
23920 6629 Acked Read 7 Sent Done Done false
23921 6630 Acked Write 10 Sent Done Done false
23922 6631 Acked Read 8 Sent Done Done false
23923 6632 Acked Read 3 Sent Done Done false
23924 6633 Acked Write 6 Sent Done Done false
23925 6634 Acked Read 2 Sent Done Done false
23926 6635 Acked Read 3 Sent Done Done false
23927 6636 Acked Read 6 Sent Done Done false
23928 6637 Acked Read 8 Sent Done Done false
23929 6638 Acked Flush 0 Sent Done Done false
23930 6639 Acked Flush 0 Sent Done Done false
23931 6640 Acked Write 6 Sent Done Done false
23932 6641 Acked Read 1 Sent Done Done false
23933 6642 Acked Read 1 Sent Done Done false
23934 6643 Acked Write 3 Sent Done Done false
23935 6644 Acked Write 5 Sent Done Done false
23936 6645 Acked Write 1 Sent Done Done false
23937 6646 Acked Flush 0 Sent Done Done false
23938 6647 Acked Write 3 Sent Done Done false
23939 6648 Acked Write 1 Sent Done Done false
23940 6649 Acked Read 10 Sent Done Done false
23941 6650 Acked Flush 0 Sent Done Done false
23942 6651 Acked Read 10 Sent Done Done false
23943 6652 Acked Read 3 Sent Done Done false
23944 6653 Acked Write 5 Sent Done Done false
23945 6654 Acked Read 7 Sent Done Done false
23946 6655 Acked Flush 0 Sent Done Done false
23947 6656 Acked Read 8 Sent Done Done false
23948 6657 Acked Read 1 Sent Done Done false
23949 6658 Acked Read 5 Sent Done Done false
23950 6659 Acked Flush 0 Sent Done Done false
23951 6660 Acked Flush 0 Sent Done Done false
23952 6661 Acked Write 3 Sent Done Done false
23953 6662 Acked Read 1 Sent Done Done false
23954 6663 Acked Write 2 Sent Done Done false
23955 6664 Acked Read 2 Sent Done Done false
23956 6665 Acked Write 8 Sent Done Done false
23957 6666 Acked Read 6 Sent Done Done false
23958 6667 Acked Write 2 Sent Done Done false
23959 6668 Acked Read 2 Sent Done Done false
23960 6669 Acked Read 2 Sent Done Done false
23961 6670 Acked Write 3 Sent Done Done false
23962 6671 Acked Read 1 Sent Done Done false
23963 6672 Acked Write 9 Sent Done Done false
23964 6673 Acked Read 8 Sent Done Done false
23965 6674 Acked Read 5 Sent Done Done false
23966 6675 Acked Read 1 Sent Done Done false
23967 6676 Acked Read 3 Sent Done Done false
23968 6677 Acked Read 7 Sent Done Done false
23969 6678 Acked Write 2 Sent Done Done false
23970 6679 Acked Read 9 Sent Done Done false
23971 6680 Acked Read 10 Sent Done Done false
23972 6681 Acked Flush 0 Sent Done Done false
23973 6682 Acked Flush 0 Sent Done Done false
23974 6683 Acked Read 2 Sent Done Done false
23975 6684 Acked Flush 0 Sent Done Done false
23976 6685 Acked Write 8 Sent Done Done false
23977 6686 Acked Read 9 Sent Done Done false
23978 6687 Acked Write 4 Sent Done Done false
23979 6688 Acked Read 10 Sent Done Done false
23980 6689 Acked Write 1 Sent Done Done false
23981 6690 Acked Read 4 Sent Done Done false
23982 6691 Acked Read 6 Sent Done Done false
23983 6692 Acked Read 4 Sent Done Done false
23984 6693 Acked Write 1 Sent Done Done false
23985 6694 Acked Write 6 Sent Done Done false
23986 6695 Acked Write 3 Sent Done Done false
23987 6696 Acked Read 4 Sent Done Done false
23988 6697 Acked Write 1 Sent Done Done false
23989 6698 Acked Read 7 Sent Done Done false
23990 6699 Acked Write 3 Sent Done Done false
23991 6700 Acked Read 8 Sent Done Done false
23992 6701 Acked Write 5 Sent Done Done false
23993 6702 Acked Read 3 Sent Done Done false
23994 6703 Acked Read 8 Sent Done Done false
23995 6704 Acked Write 5 Sent Done Done false
23996 6705 Acked Write 1 Sent Done Done false
23997 6706 Acked Read 8 Sent Done Done false
23998 6707 Acked Read 3 Sent Done Done false
23999 6708 Acked Write 10 Sent Done Done false
24000 6709 Acked Read 1 Sent Done Done false
24001 6710 Acked Write 10 Sent Done Done false
24002 6711 Acked Read 4 Sent Done Done false
24003 6712 Acked Read 2 Sent Done Done false
24004 6713 Acked Read 3 Sent Done Done false
24005 6714 Acked Read 5 Sent Done Done false
24006 6715 Acked Write 5 Sent Done Done false
24007 6716 Acked Write 9 Sent Done Done false
24008 6717 Acked Read 7 Sent Done Done false
24009 6718 Acked Flush 0 Sent Done Done false
24010 6719 Acked Flush 0 Sent Done Done false
24011 6720 Acked Read 2 Sent Done Done false
24012 6721 Acked Write 4 Sent Done Done false
24013 6722 Acked Write 8 Sent Done Done false
24014 6723 Acked Read 8 Sent Done Done false
24015 6724 Acked Write 8 Sent Done Done false
24016 6725 Acked Read 8 Sent Done Done false
24017 6726 Acked Read 3 Sent Done Done false
24018 6727 Acked Read 9 Sent Done Done false
24019 6728 Acked Read 3 Sent Done Done false
24020 6729 Acked Read 4 Sent Done Done false
24021 6730 Acked Read 7 Sent Done Done false
24022 6731 Acked Read 6 Sent Done Done false
24023 6732 Acked Write 3 Sent Done Done false
24024 6733 Acked Write 5 Sent Done Done false
24025 6734 Acked Flush 0 Sent Done Done false
24026 6735 Acked Read 4 Sent Done Done false
24027 STATES DS:0 DS:1 DS:2 TOTAL
24028 Sent 301 0 0 301
24029 Done 0 301 301 602
24030 Skipped 0 0 0 0
24031 Error 0 0 0 0
24032 Last Flush: Some(JobId(6434)) Some(JobId(6734)) Some(JobId(6734))
24033 Downstairs last five completed: 6434 6433 6432 6431 6430
24034 Upstairs last five completed: 6735 6734 6733 6732 6731
24035 CLIENT: Up:0 ds:301 act:2
240362025-02-07T20:12:39.468ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
240372025-02-07T20:12:40.012ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
240382025-02-07T20:12:40.012ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59693) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
240392025-02-07T20:12:40.013ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
240402025-02-07T20:12:40.013ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
240412025-02-07T20:12:40.014ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
240422025-02-07T20:12:40.014ZINFOcrucible: send last flush ID to this DS: Some(JobId(6434)) = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
240432025-02-07T20:12:40.014ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
240442025-02-07T20:12:40.014ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(6434)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
240452025-02-07T20:12:40.014ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(6434)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
24046 ----------------------------------------------------------------
24047 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24048 Downstairs last five completed: 6736 6735 6734 6733 6732
24049 Upstairs last five completed: 6736 6735 6734 6733 6732
24050 CLIENT: Up:0 ds:0 act:3
24051 Replay: All jobs finished
24052 Main task has work to do, go find it
24053 got dsc Stop(1)
24054 stop 1
24055 [8820] Got stop action so:false kr:false
24056 Feb 07 20:12:42.264 INFO request completed, latency_us: 368, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 1c4cc72e-9526-488a-bf08-39e99ec73dab, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
24057 Feb 07 20:12:42.265 INFO request completed, latency_us: 160, response_code: 200, uri: /state/cid/1, method: GET, req_id: edaa23e7-e416-4a6d-9642-af8699c795bd, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:12:42.265365697Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"1","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
24058 :9998
240592025-02-07T20:12:42.265ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
240602025-02-07T20:12:42.265ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
240612025-02-07T20:12:42.265ZINFOcrucible: client task is exiting = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
24062 [8820] Exited with: ExitStatus(unix_wait_status(9))
24063 [8820][1] reports Exit
24064 Feb 07 20:12:46.267 INFO request completed, latency_us: 384, response_code: 200, uri: /state/cid/1, method: GET, req_id: f1df6e72-74bf-44c7-8c37-8f47aa54821d, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
24065 001/300 Write block 2756 len 4096 data: 7
24066 002/300 Read block 2034 len 32768
24067 003/300 Read block 3405 len 36864
24068 004/300 Read block 545 len 24576
24069 005/300 Read block 4390 len 20480
24070 006/300 Read block 2630 len 40960
24071 007/300 Write block 4515 len 24576 data: 6 5 5 5 6 6
24072 008/300 Flush
24073 009/300 Flush
24074 010/300 Read block 1711 len 12288
24075 011/300 Read block 2763 len 4096
24076 012/300 Flush
24077 013/300 Read block 1746 len 40960
24078 014/300 Read block 3357 len 32768
24079 015/300 Read block 3372 len 4096
24080 016/300 Flush
24081 017/300 Write block 1880 len 28672 data: 3 3 3 3 3 3 3
24082 018/300 Write block 4872 len 16384 data: 3 3 3 3
24083 019/300 Write block 4863 len 12288 data: 5 5 4
24084 020/300 Read block 3677 len 8192
24085 021/300 Read block 3395 len 24576
24086 022/300 Write block 2578 len 36864 data: 2 3 4 5 3 3 3 3 4
24087 023/300 Write block 4820 len 4096 data: 5
24088 024/300 Write block 1332 len 24576 data: 7 7 7 5 5 5
24089 025/300 Read block 3417 len 28672
24090 026/300 Read block 2637 len 32768
24091 027/300 Read block 4335 len 12288
24092 028/300 Write block 3758 len 32768 data: 5 5 6 6 6 6 5 5
24093 029/300 Read block 4950 len 8192
24094 030/300 Read block 1322 len 4096
24095 031/300 Write block 1512 len 20480 data: 5 5 4 4 3
24096 032/300 Write block 4700 len 40960 data: 3 3 3 3 3 4 4 4 4 4
24097 033/300 Read block 3330 len 20480
24098 034/300 Flush
24099 035/300 Read block 308 len 8192
24100 036/300 Read block 4690 len 28672
24101 037/300 Read block 2654 len 36864
24102 038/300 Read block 1325 len 4096
24103 039/300 Read block 4379 len 20480
24104 040/300 Write block 1407 len 16384 data: 3 3 4 3
24105 041/300 Write block 1414 len 32768 data: 3 3 2 3 3 4 4 5
24106 042/300 Write block 2580 len 36864 data: 5 6 4 4 4 4 5 2 4
24107 043/300 Write block 1717 len 28672 data: 4 6 5 5 4 4 4
24108 044/300 Read block 1922 len 12288
24109 045/300 Write block 4830 len 20480 data: 8 9 10 9 9
24110 046/300 Read block 4033 len 4096
24111 047/300 Flush
24112 048/300 Write block 3224 len 28672 data: 4 4 4 3 4 4 4
24113 049/300 Flush
24114 050/300 Write block 1321 len 4096 data: 7
24115 051/300 Read block 1923 len 20480
24116 052/300 Flush
24117 053/300 Write block 1937 len 40960 data: 3 4 3 3 3 4 4 4 4 3
24118 054/300 Write block 756 len 4096 data: 5
24119 055/300 Write block 3597 len 8192 data: 3 2
24120 056/300 Read block 3001 len 4096
24121 057/300 Write block 2697 len 28672 data: 4 5 5 5 4 4 5
24122 058/300 Write block 2982 len 4096 data: 6
24123 059/300 Flush
24124 060/300 Read block 3105 len 4096
24125 061/300 Write block 4659 len 24576 data: 2 2 2 4 4 3
24126 062/300 Read block 3841 len 28672
24127 063/300 Write block 206 len 32768 data: 5 3 2 4 5 4 4 4
24128 064/300 Read block 462 len 36864
24129 065/300 Write block 4500 len 12288 data: 11 11 12
24130 066/300 Write block 4281 len 24576 data: 3 2 2 2 2 3
24131 067/300 Read block 679 len 4096
24132 068/300 Read block 3680 len 16384
24133 069/300 Read block 807 len 4096
24134 070/300 Read block 4441 len 8192
24135 071/300 Write block 3626 len 8192 data: 2 2
24136 072/300 Read block 2647 len 40960
24137 073/300 Read block 4122 len 16384
24138 074/300 Read block 2957 len 36864
24139 075/300 Flush
24140 076/300 Flush
24141 077/300 Write block 2896 len 24576 data: 3 3 3 3 4 4
24142 078/300 Write block 1101 len 24576 data: 5 5 5 5 5 5
24143 079/300 Write block 4943 len 28672 data: 7 7 7 6 5 5 5
24144 080/300 Write block 1301 len 20480 data: 7 7 6 5 4
24145 081/300 Read block 4494 len 20480
24146 082/300 Write block 1458 len 8192 data: 6 5
24147 083/300 Write block 3119 len 28672 data: 5 4 4 3 3 3 5
24148 084/300 Read block 97 len 20480
24149 085/300 Read block 1962 len 20480
24150 086/300 Write block 4311 len 32768 data: 2 3 4 3 4 4 4 4
24151 087/300 Write block 1215 len 40960 data: 4 4 4 5 5 6 6 7 9 9
24152 088/300 Write block 423 len 12288 data: 5 6 5
24153 089/300 Write block 4748 len 24576 data: 6 5 5 6 5 4
24154 090/300 Read block 2013 len 4096
24155 091/300 Write block 1833 len 36864 data: 9 11 9 7 6 6 4 4 3
24156 092/300 Write block 734 len 8192 data: 5 5
24157 093/300 Write block 2472 len 8192 data: 6 6
24158 094/300 Read block 3137 len 8192
24159 095/300 Write block 2288 len 24576 data: 6 6 6 5 5 4
24160 096/300 Read block 3036 len 4096
24161 097/300 Write block 650 len 4096 data: 5
24162 098/300 Write block 4005 len 24576 data: 3 2 4 4 4 4
24163 099/300 Write block 1388 len 28672 data: 3 4 5 5 5 5 4
24164 100/300 Write block 966 len 16384 data: 6 5 4 2
24165 101/300 Read block 3644 len 16384
24166 102/300 Write block 3799 len 24576 data: 5 5 5 4 4 4
24167 103/300 Read block 804 len 12288
24168 104/300 Read block 1797 len 4096
24169 105/300 Read block 4818 len 24576
24170 106/300 Write block 92 len 24576 data: 4 5 4 4 5 5
24171 107/300 Read block 4964 len 28672
24172 108/300 Read block 2179 len 12288
24173 109/300 Flush
24174 110/300 Write block 3311 len 12288 data: 5 4 4
24175 111/300 Read block 1428 len 28672
24176 112/300 Write block 3091 len 20480 data: 5 4 4 4 4
24177 113/300 Flush
24178 114/300 Write block 2473 len 20480 data: 7 4 3 3 2
24179 115/300 Read block 3429 len 40960
24180 116/300 Read block 3195 len 28672
24181 117/300 Flush
24182 118/300 Flush
24183 119/300 Write block 1316 len 20480 data: 6 5 5 4 6
24184 120/300 Write block 336 len 20480 data: 7 8 7 7 7
24185 121/300 Read block 3556 len 24576
24186 122/300 Read block 2732 len 4096
24187 123/300 Write block 2218 len 36864 data: 3 2 2 2 3 3 3 3 3
24188 124/300 Flush
24189 125/300 Read block 1564 len 8192
24190 126/300 Flush
24191 127/300 Flush
24192 128/300 Write block 4893 len 16384 data: 5 5 6 6
24193 129/300 Flush
24194 130/300 Write block 2386 len 24576 data: 4 4 4 4 4 3
24195 131/300 Read block 1754 len 32768
24196 132/300 Write block 4182 len 4096 data: 3
24197 133/300 Write block 2603 len 12288 data: 6 6 6
24198 134/300 Write block 3583 len 32768 data: 3 4 4 4 4 4 5 4
24199 135/300 Read block 4808 len 40960
24200 136/300 Read block 3648 len 4096
24201 137/300 Read block 3872 len 8192
24202 138/300 Read block 3999 len 16384
24203 139/300 Write block 4731 len 4096 data: 5
24204 140/300 Read block 4765 len 32768
24205 141/300 Read block 587 len 28672
24206 142/300 Read block 3018 len 8192
24207 143/300 Write block 2523 len 28672 data: 5 5 5 4 4 4 4
24208 144/300 Read block 3601 len 8192
24209 145/300 Write block 1360 len 4096 data: 3
24210 146/300 Read block 1983 len 24576
24211 147/300 Read block 1014 len 8192
24212 148/300 Read block 273 len 4096
24213 149/300 Read block 2322 len 20480
24214 150/300 Write block 373 len 12288 data: 4 4 3
24215 151/300 Write block 4923 len 16384 data: 5 6 6 6
24216 152/300 Write block 1341 len 32768 data: 6 5 6 7 8 7 5 4
24217 153/300 Read block 2981 len 28672
24218 154/300 Read block 1572 len 12288
24219 155/300 Read block 2846 len 32768
24220 156/300 Read block 4637 len 40960
24221 157/300 Flush
24222 158/300 Write block 3154 len 16384 data: 4 6 7 6
24223 159/300 Read block 4556 len 4096
24224 160/300 Write block 669 len 16384 data: 3 4 4 4
24225 161/300 Read block 3561 len 20480
24226 162/300 Write block 1918 len 28672 data: 5 6 3 4 5 5 5
24227 163/300 Read block 1255 len 24576
24228 164/300 Flush
24229 165/300 Write block 553 len 16384 data: 3 2 3 4
24230 166/300 Write block 1777 len 36864 data: 5 5 6 6 7 8 10 7 7
24231 167/300 Write block 1646 len 24576 data: 5 5 5 5 5 7
24232 168/300 Write block 2333 len 16384 data: 3 3 4 4
24233 169/300 Read block 4253 len 36864
24234 170/300 Read block 3883 len 24576
24235 171/300 Read block 3584 len 28672
24236 172/300 Write block 3799 len 36864 data: 6 6 6 5 5 5 4 3 3
24237 173/300 Read block 3799 len 36864
24238 174/300 Read block 1542 len 20480
24239 175/300 Write block 1062 len 36864 data: 4 4 2 2 3 3 4 3 4
24240 176/300 Write block 3835 len 32768 data: 5 5 5 5 4 5 4 4
24241 177/300 Read block 21 len 16384
24242 178/300 Flush
24243 179/300 Read block 4931 len 24576
24244 180/300 Read block 2554 len 12288
24245 181/300 Read block 729 len 4096
24246 182/300 Write block 3805 len 36864 data: 5 4 4 3 4 4 5 6 6
24247 183/300 Read block 17 len 36864
24248 184/300 Write block 1067 len 4096 data: 4
24249 185/300 Read block 1934 len 12288
24250 186/300 Read block 1942 len 28672
24251 187/300 Read block 418 len 32768
24252 188/300 Write block 3787 len 40960 data: 4 4 2 2 2 3 3 3 3 3
24253 189/300 Read block 3978 len 12288
24254 190/300 Read block 1585 len 40960
24255 191/300 Read block 31 len 40960
24256 192/300 Read block 373 len 12288
24257 193/300 Read block 4437 len 16384
24258 194/300 Flush
24259 195/300 Write block 2805 len 40960 data: 8 8 8 8 7 6 5 4 5 5
24260 196/300 Read block 4801 len 20480
24261 197/300 Write block 1952 len 8192 data: 5 5
24262 198/300 Write block 1254 len 32768 data: 5 4 3 3 3 3 2 3
24263 199/300 Read block 1813 len 16384
24264 200/300 Write block 2538 len 24576 data: 5 6 5 5 4 4
24265 201/300 Write block 2269 len 16384 data: 6 6 7 6
24266 202/300 Write block 4485 len 20480 data: 3 2 3 4 5
24267 203/300 Read block 1036 len 8192
24268 204/300 Write block 3804 len 16384 data: 6 6 5 5
24269 205/300 Flush
24270 206/300 Write block 3701 len 40960 data: 5 6 6 6 6 5 5 8 6 7
24271 207/300 Write block 3365 len 8192 data: 6 6
24272 208/300 Write block 4454 len 24576 data: 4 4 4 4 5 4
24273 209/300 Flush
24274 210/300 Write block 4192 len 24576 data: 7 8 6 4 5 5
24275 211/300 Write block 3332 len 40960 data: 5 7 7 5 5 5 4 4 4 5
24276 212/300 Write block 4306 len 28672 data: 3 4 3 3 4 3 4
24277 213/300 Read block 3552 len 32768
24278 214/300 Read block 1042 len 36864
24279 215/300 Write block 4199 len 28672 data: 5 5 4 3 4 4 4
24280 216/300 Write block 4168 len 36864 data: 7 6 5 5 4 4 4 4 5
24281 217/300 Read block 2263 len 24576
24282 218/300 Write block 927 len 12288 data: 2 3 4
24283 219/300 Read block 3355 len 16384
24284 220/300 Read block 4405 len 36864
24285 221/300 Flush
24286 222/300 Flush
24287 223/300 Write block 2735 len 4096 data: 4
24288 224/300 Read block 255 len 28672
24289 225/300 Read block 2847 len 28672
24290 226/300 Read block 661 len 24576
24291 227/300 Read block 922 len 12288
24292 228/300 Read block 4244 len 4096
24293 229/300 Read block 4474 len 16384
24294 230/300 Read block 2414 len 8192
24295 231/300 Write block 1357 len 20480 data: 3 3 4 4 3
24296 232/300 Read block 1411 len 28672
24297 233/300 Flush
24298 234/300 Write block 966 len 40960 data: 7 6 5 3 2 2 2 3 4 4
24299 235/300 Read block 3738 len 20480
24300 236/300 Write block 1143 len 4096 data: 2
24301 237/300 Read block 2694 len 40960
24302 238/300 Read block 2229 len 4096
24303 239/300 Write block 842 len 16384 data: 6 5 4 4
24304 240/300 Read block 1946 len 20480
24305 241/300 Write block 1979 len 8192 data: 4 6
24306 242/300 Write block 4979 len 8192 data: 7 6
24307 243/300 Read block 544 len 28672
24308 244/300 Write block 2222 len 28672 data: 4 4 4 4 4 3 3
24309 245/300 Flush
24310 246/300 Read block 3805 len 8192
24311 247/300 Read block 4828 len 32768
24312 248/300 Read block 3720 len 12288
24313 249/300 Read block 2336 len 36864
24314 250/300 Write block 109 len 4096 data: 4
24315 251/300 Flush
24316 252/300 Read block 1388 len 40960
24317 253/300 Write block 4343 len 20480 data: 5 4 4 4 4
24318 254/300 Write block 3141 len 36864 data: 7 6 5 5 6 6 4 5 4
24319 255/300 Read block 3137 len 32768
24320 256/300 Read block 311 len 16384
24321 257/300 Flush
24322 258/300 Read block 590 len 20480
24323 259/300 Write block 570 len 32768 data: 4 4 5 4 5 5 5 5
24324 260/300 Write block 4974 len 28672 data: 6 7 6 5 5 8 7
24325 261/300 Read block 4331 len 20480
24326 262/300 Read block 2273 len 40960
24327 263/300 Write block 1746 len 16384 data: 4 4 3 4
24328 264/300 Read block 2196 len 20480
24329 265/300 Write block 1112 len 8192 data: 6 7
24330 266/300 Read block 4162 len 36864
24331 267/300 Write block 4807 len 40960 data: 4 4 3 3 3 4 4 4 5 5
24332 268/300 Read block 574 len 20480
24333 269/300 Read block 1199 len 36864
24334 270/300 Write block 4703 len 32768 data: 4 4 5 5 5 5 5 4
24335 271/300 Write block 603 len 12288 data: 4 3 2
24336 272/300 Read block 1297 len 40960
24337 273/300 Read block 3333 len 16384
24338 274/300 Read block 1797 len 12288
24339 275/300 Flush
24340 276/300 Write block 4677 len 8192 data: 4 5
24341 277/300 Read block 92 len 20480
24342 278/300 Read block 3051 len 4096
24343 279/300 Read block 2392 len 20480
24344 280/300 Write block 3581 len 8192 data: 4 3
24345 281/300 Write block 2803 len 12288 data: 9 8 9
24346 282/300 Write block 2732 len 36864 data: 3 3 2 5 4 5 4 5 5
24347 283/300 Write block 1394 len 24576 data: 5 4 6 7 7 6
24348 284/300 Read block 4689 len 32768
24349 285/300 Read block 2156 len 20480
24350 286/300 Write block 4224 len 4096 data: 4
24351 287/300 Read block 2219 len 40960
24352 288/300 Read block 1246 len 20480
24353 289/300 Read block 2042 len 20480
24354 290/300 Write block 34 len 24576 data: 4 4 4 3 4 6
24355 291/300 Write block 4314 len 28672 data: 4 5 5 5 5 3 3
24356 292/300 Read block 4881 len 12288
24357 293/300 Read block 3670 len 32768
24358 294/300 Read block 3803 len 40960
24359 295/300 Read block 3971 len 32768
24360 296/300 Flush
24361 297/300 Read block 3593 len 24576
24362 298/300 Write block 2046 len 32768 data: 4 4 3 3 3 3 3 5
24363 299/300 Read block 1654 len 8192
24364 300/300 Write block 3397 len 8192 data: 6 7
24365 301/300 Read block 3460 len 28672
24366 Main task has work to do, go find it
24367 got dsc Start(1)
24368 start 1
24369 Feb 07 20:12:46.504 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/1, method: POST, req_id: 8344a02b-96ed-4b9c-9623-22441c37f0c0, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
24370 Replay: started 1, returned:Ok(())
24371 ----------------------------------------------------------------
24372 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24373 States: Active Offline Active
24374 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24375 6737 Acked Write 1 Done Sent Done false
24376 6738 Acked Read 8 Done Sent Done false
24377 6739 Acked Read 9 Done Sent Done false
24378 6740 Acked Read 6 Done Sent Done false
24379 6741 Acked Read 5 Done Sent Done false
24380 6742 Acked Read 10 Done Sent Done false
24381 6743 Acked Write 6 Done Sent Done false
24382 6744 Acked Flush 0 Done Sent Done false
24383 6745 Acked Flush 0 Done Sent Done false
24384 6746 Acked Read 3 Done Sent Done false
24385 6747 Acked Read 1 Done Sent Done false
24386 6748 Acked Flush 0 Done Sent Done false
24387 6749 Acked Read 10 Done Sent Done false
24388 6750 Acked Read 8 Done Sent Done false
24389 6751 Acked Read 1 Done Sent Done false
24390 6752 Acked Flush 0 Done Sent Done false
24391 6753 Acked Write 7 Done Sent Done false
24392 6754 Acked Write 4 Done Sent Done false
24393 6755 Acked Write 3 Done Sent Done false
24394 6756 Acked Read 2 Done Sent Done false
24395 6757 Acked Read 6 Done Sent Done false
24396 6758 Acked Write 9 Done Sent Done false
24397 6759 Acked Write 1 Done Sent Done false
24398 6760 Acked Write 6 Done Sent Done false
24399 6761 Acked Read 7 Done Sent Done false
24400 6762 Acked Read 8 Done Sent Done false
24401 6763 Acked Read 3 Done Sent Done false
24402 6764 Acked Write 8 Done Sent Done false
24403 6765 Acked Read 2 Done Sent Done false
24404 6766 Acked Read 1 Done Sent Done false
24405 6767 Acked Write 5 Done Sent Done false
24406 6768 Acked Write 10 Done Sent Done false
24407 6769 Acked Read 5 Done Sent Done false
24408 6770 Acked Flush 0 Done Sent Done false
24409 6771 Acked Read 2 Done Sent Done false
24410 6772 Acked Read 7 Done Sent Done false
24411 6773 Acked Read 9 Done Sent Done false
24412 6774 Acked Read 1 Done Sent Done false
24413 6775 Acked Read 5 Done Sent Done false
24414 6776 Acked Write 4 Done Sent Done false
24415 6777 Acked Write 8 Done Sent Done false
24416 6778 Acked Write 9 Done Sent Done false
24417 6779 Acked Write 7 Done Sent Done false
24418 6780 Acked Read 3 Done Sent Done false
24419 6781 Acked Write 5 Done Sent Done false
24420 6782 Acked Read 1 Done Sent Done false
24421 6783 Acked Flush 0 Done Sent Done false
24422 6784 Acked Write 7 Done Sent Done false
24423 6785 Acked Flush 0 Done Sent Done false
24424 6786 Acked Write 1 Done Sent Done false
24425 6787 Acked Read 5 Done Sent Done false
24426 6788 Acked Flush 0 Done Sent Done false
24427 6789 Acked Write 10 Done Sent Done false
24428 6790 Acked Write 1 Done Sent Done false
24429 6791 Acked Write 2 Done Sent Done false
24430 6792 Acked Read 1 Done Sent Done false
24431 6793 Acked Write 7 Done Sent Done false
24432 6794 Acked Write 1 Done Sent Done false
24433 6795 Acked Flush 0 Done Sent Done false
24434 6796 Acked Read 1 Done Sent Done false
24435 6797 Acked Write 6 Done Sent Done false
24436 6798 Acked Read 7 Done Sent Done false
24437 6799 Acked Write 8 Done Sent Done false
24438 6800 Acked Read 9 Done Sent Done false
24439 6801 Acked Write 3 Done Sent Done false
24440 6802 Acked Write 6 Done Sent Done false
24441 6803 Acked Read 1 Done Sent Done false
24442 6804 Acked Read 4 Done Sent Done false
24443 6805 Acked Read 1 Done Sent Done false
24444 6806 Acked Read 2 Done Sent Done false
24445 6807 Acked Write 2 Done Sent Done false
24446 6808 Acked Read 10 Done Sent Done false
24447 6809 Acked Read 4 Done Sent Done false
24448 6810 Acked Read 9 Done Sent Done false
24449 6811 Acked Flush 0 Done Sent Done false
24450 6812 Acked Flush 0 Done Sent Done false
24451 6813 Acked Write 6 Done Sent Done false
24452 6814 Acked Write 6 Done Sent Done false
24453 6815 Acked Write 7 Done Sent Done false
24454 6816 Acked Write 5 Done Sent Done false
24455 6817 Acked Read 5 Done Sent Done false
24456 6818 Acked Write 2 Done Sent Done false
24457 6819 Acked Write 7 Done Sent Done false
24458 6820 Acked Read 5 Done Sent Done false
24459 6821 Acked Read 5 Done Sent Done false
24460 6822 Acked Write 8 Done Sent Done false
24461 6823 Acked Write 10 Done Sent Done false
24462 6824 Acked Write 3 Done Sent Done false
24463 6825 Acked Write 6 Done Sent Done false
24464 6826 Acked Read 1 Done Sent Done false
24465 6827 Acked Write 9 Done Sent Done false
24466 6828 Acked Write 2 Done Sent Done false
24467 6829 Acked Write 2 Done Sent Done false
24468 6830 Acked Read 2 Done Sent Done false
24469 6831 Acked Write 6 Done Sent Done false
24470 6832 Acked Read 1 Done Sent Done false
24471 6833 Acked Write 1 Done Sent Done false
24472 6834 Acked Write 6 Done Sent Done false
24473 6835 Acked Write 7 Done Sent Done false
24474 6836 Acked Write 4 Done Sent Done false
24475 6837 Acked Read 4 Done Sent Done false
24476 6838 Acked Write 6 Done Sent Done false
24477 6839 Acked Read 3 Done Sent Done false
24478 6840 Acked Read 1 Done Sent Done false
24479 6841 Acked Read 6 Done Sent Done false
24480 6842 Acked Write 6 Done Sent Done false
24481 6843 Acked Read 7 Done Sent Done false
24482 6844 Acked Read 3 Done Sent Done false
24483 6845 Acked Flush 0 Done Sent Done false
24484 6846 Acked Write 3 Done Sent Done false
24485 6847 Acked Read 7 Done Sent Done false
24486 6848 Acked Write 5 Done Sent Done false
24487 6849 Acked Flush 0 Done Sent Done false
24488 6850 Acked Write 5 Done Sent Done false
24489 6851 Acked Read 10 Done Sent Done false
24490 6852 Acked Read 7 Done Sent Done false
24491 6853 Acked Flush 0 Done Sent Done false
24492 6854 Acked Flush 0 Done Sent Done false
24493 6855 Acked Write 5 Done Sent Done false
24494 6856 Acked Write 5 Done Sent Done false
24495 6857 Acked Read 6 Done Sent Done false
24496 6858 Acked Read 1 Done Sent Done false
24497 6859 Acked Write 9 Done Sent Done false
24498 6860 Acked Flush 0 Done Sent Done false
24499 6861 Acked Read 2 Done Sent Done false
24500 6862 Acked Flush 0 Done Sent Done false
24501 6863 Acked Flush 0 Done Sent Done false
24502 6864 Acked Write 4 Done Sent Done false
24503 6865 Acked Flush 0 Done Sent Done false
24504 6866 Acked Write 6 Done Sent Done false
24505 6867 Acked Read 8 Done Sent Done false
24506 6868 Acked Write 1 Done Sent Done false
24507 6869 Acked Write 3 Done Sent Done false
24508 6870 Acked Write 8 Done Sent Done false
24509 6871 Acked Read 10 Done Sent Done false
24510 6872 Acked Read 1 Done Sent Done false
24511 6873 Acked Read 2 Done Sent Done false
24512 6874 Acked Read 4 Done Sent Done false
24513 6875 Acked Write 1 Done Sent Done false
24514 6876 Acked Read 8 Done Sent Done false
24515 6877 Acked Read 7 Done Sent Done false
24516 6878 Acked Read 2 Done Sent Done false
24517 6879 Acked Write 7 Done Sent Done false
24518 6880 Acked Read 2 Done Sent Done false
24519 6881 Acked Write 1 Done Sent Done false
24520 6882 Acked Read 6 Done Sent Done false
24521 6883 Acked Read 2 Done Sent Done false
24522 6884 Acked Read 1 Done Sent Done false
24523 6885 Acked Read 5 Done Sent Done false
24524 6886 Acked Write 3 Done Sent Done false
24525 6887 Acked Write 4 Done Sent Done false
24526 6888 Acked Write 8 Done Sent Done false
24527 6889 Acked Read 7 Done Sent Done false
24528 6890 Acked Read 3 Done Sent Done false
24529 6891 Acked Read 8 Done Sent Done false
24530 6892 Acked Read 10 Done Sent Done false
24531 6893 Acked Flush 0 Done Sent Done false
24532 6894 Acked Write 4 Done Sent Done false
24533 6895 Acked Read 1 Done Sent Done false
24534 6896 Acked Write 4 Done Sent Done false
24535 6897 Acked Read 5 Done Sent Done false
24536 6898 Acked Write 7 Done Sent Done false
24537 6899 Acked Read 6 Done Sent Done false
24538 6900 Acked Flush 0 Done Sent Done false
24539 6901 Acked Write 4 Done Sent Done false
24540 6902 Acked Write 9 Done Sent Done false
24541 6903 Acked Write 6 Done Sent Done false
24542 6904 Acked Write 4 Done Sent Done false
24543 6905 Acked Read 9 Done Sent Done false
24544 6906 Acked Read 6 Done Sent Done false
24545 6907 Acked Read 7 Done Sent Done false
24546 6908 Acked Write 9 Done Sent Done false
24547 6909 Acked Read 9 Done Sent Done false
24548 6910 Acked Read 5 Done Sent Done false
24549 6911 Acked Write 9 Done Sent Done false
24550 6912 Acked Write 8 Done Sent Done false
24551 6913 Acked Read 4 Done Sent Done false
24552 6914 Acked Flush 0 Done Sent Done false
24553 6915 Acked Read 6 Done Sent Done false
24554 6916 Acked Read 3 Done Sent Done false
24555 6917 Acked Read 1 Done Sent Done false
24556 6918 Acked Write 9 Done Sent Done false
24557 6919 Acked Read 9 Done Sent Done false
24558 6920 Acked Write 1 Done Sent Done false
24559 6921 Acked Read 3 Done Sent Done false
24560 6922 Acked Read 7 Done Sent Done false
24561 6923 Acked Read 8 Done Sent Done false
24562 6924 Acked Write 10 Done Sent Done false
24563 6925 Acked Read 3 Done Sent Done false
24564 6926 Acked Read 10 Done Sent Done false
24565 6927 Acked Read 10 Done Sent Done false
24566 6928 Acked Read 3 Done Sent Done false
24567 6929 Acked Read 4 Done Sent Done false
24568 6930 Acked Flush 0 Done Sent Done false
24569 6931 Acked Write 10 Done Sent Done false
24570 6932 Acked Read 5 Done Sent Done false
24571 6933 Acked Write 2 Done Sent Done false
24572 6934 Acked Write 8 Done Sent Done false
24573 6935 Acked Read 4 Done Sent Done false
24574 6936 Acked Write 6 Done Sent Done false
24575 6937 Acked Write 4 Done Sent Done false
24576 6938 Acked Write 5 Done Sent Done false
24577 6939 Acked Read 2 Done Sent Done false
24578 6940 Acked Write 4 Done Sent Done false
24579 6941 Acked Flush 0 Done Sent Done false
24580 6942 Acked Write 10 Done Sent Done false
24581 6943 Acked Write 2 Done Sent Done false
24582 6944 Acked Write 6 Done Sent Done false
24583 6945 Acked Flush 0 Done Sent Done false
24584 6946 Acked Write 6 Done Sent Done false
24585 6947 Acked Write 10 Done Sent Done false
24586 6948 Acked Write 7 Done Sent Done false
24587 6949 Acked Read 8 Done Sent Done false
24588 6950 Acked Read 9 Done Sent Done false
24589 6951 Acked Write 7 Done Sent Done false
24590 6952 Acked Write 9 Done Sent Done false
24591 6953 Acked Read 6 Done Sent Done false
24592 6954 Acked Write 3 Done Sent Done false
24593 6955 Acked Read 4 Done Sent Done false
24594 6956 Acked Read 9 Done Sent Done false
24595 6957 Acked Flush 0 Done Sent Done false
24596 6958 Acked Flush 0 Done Sent Done false
24597 6959 Acked Write 1 Done Sent Done false
24598 6960 Acked Read 7 Done Sent Done false
24599 6961 Acked Read 7 Done Sent Done false
24600 6962 Acked Read 6 Done Sent Done false
24601 6963 Acked Read 3 Done Sent Done false
24602 6964 Acked Read 1 Done Sent Done false
24603 6965 Acked Read 4 Done Sent Done false
24604 6966 Acked Read 2 Done Sent Done false
24605 6967 Acked Write 5 Done Sent Done false
24606 6968 Acked Read 7 Done Sent Done false
24607 6969 Acked Flush 0 Done Sent Done false
24608 6970 Acked Write 10 Done Sent Done false
24609 6971 Acked Read 5 Done Sent Done false
24610 6972 Acked Write 1 Done Sent Done false
24611 6973 Acked Read 10 Done Sent Done false
24612 6974 Acked Read 1 Done Sent Done false
24613 6975 Acked Write 4 Done Sent Done false
24614 6976 Acked Read 5 Done Sent Done false
24615 6977 Acked Write 2 Done Sent Done false
24616 6978 Acked Write 2 Done Sent Done false
24617 6979 Acked Read 7 Done Sent Done false
24618 6980 Acked Write 7 Done Sent Done false
24619 6981 Acked Flush 0 Done Sent Done false
24620 6982 Acked Read 2 Done Sent Done false
24621 6983 Acked Read 8 Done Sent Done false
24622 6984 Acked Read 3 Done Sent Done false
24623 6985 Acked Read 9 Done Sent Done false
24624 6986 Acked Write 1 Done Sent Done false
24625 6987 Acked Flush 0 Done Sent Done false
24626 6988 Acked Read 10 Done Sent Done false
24627 6989 Acked Write 5 Done Sent Done false
24628 6990 Acked Write 9 Done Sent Done false
24629 6991 Acked Read 8 Done Sent Done false
24630 6992 Acked Read 4 Done Sent Done false
24631 6993 Acked Flush 0 Done Sent Done false
24632 6994 Acked Read 5 Done Sent Done false
24633 6995 Acked Write 8 Done Sent Done false
24634 6996 Acked Write 7 Done Sent Done false
24635 6997 Acked Read 5 Done Sent Done false
24636 6998 Acked Read 10 Done Sent Done false
24637 6999 Acked Write 4 Done Sent Done false
24638 7000 Acked Read 5 Done Sent Done false
24639 7001 Acked Write 2 Done Sent Done false
24640 7002 Acked Read 9 Done Sent Done false
24641 7003 Acked Write 10 Done Sent Done false
24642 7004 Acked Read 5 Done Sent Done false
24643 7005 Acked Read 9 Done Sent Done false
24644 7006 Acked Write 8 Done Sent Done false
24645 7007 Acked Write 3 Done Sent Done false
24646 7008 Acked Read 10 Done Sent Done false
24647 7009 Acked Read 4 Done Sent Done false
24648 7010 Acked Read 3 Done Sent Done false
24649 7011 Acked Flush 0 Done Sent Done false
24650 7012 Acked Write 2 Done Sent Done false
24651 7013 Acked Read 5 Done Sent Done false
24652 7014 Acked Read 1 Done Sent Done false
24653 7015 Acked Read 5 Done Sent Done false
24654 7016 Acked Write 2 Done Sent Done false
24655 7017 Acked Write 3 Done Sent Done false
24656 7018 Acked Write 9 Done Sent Done false
24657 7019 Acked Write 6 Done Sent Done false
24658 7020 Acked Read 8 Done Sent Done false
24659 7021 Acked Read 5 Done Sent Done false
24660 7022 Acked Write 1 Done Sent Done false
24661 7023 Acked Read 10 Done Sent Done false
24662 7024 Acked Read 5 Done Sent Done false
24663 7025 Acked Read 5 Done Sent Done false
24664 7026 Acked Write 6 Done Sent Done false
24665 7027 Acked Write 7 Done Sent Done false
24666 7028 Acked Read 3 Done Sent Done false
24667 7029 Acked Read 8 Done Sent Done false
24668 7030 Acked Read 10 Done Sent Done false
24669 7031 Acked Read 8 Done Sent Done false
24670 7032 Acked Flush 0 Done Sent Done false
24671 7033 Acked Read 6 Done Sent Done false
24672 7034 Acked Write 8 Done Sent Done false
24673 7035 Acked Read 2 Done Sent Done false
24674 7036 Acked Write 2 Done Sent Done false
24675 7037 Acked Read 7 Done Sent Done false
24676 STATES DS:0 DS:1 DS:2 TOTAL
24677 Sent 0 301 0 301
24678 Done 301 0 301 602
24679 Skipped 0 0 0 0
24680 Error 0 0 0 0
24681 Last Flush: Some(JobId(7032)) Some(JobId(6736)) Some(JobId(7032))
24682 Downstairs last five completed: 6736 6735 6734 6733 6732
24683 Upstairs last five completed: 7037 7036 7035 7034 7033
24684 CLIENT: Up:0 ds:301 act:2
246852025-02-07T20:12:47.481ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
246862025-02-07T20:12:48.483ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
246872025-02-07T20:12:49.485ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
24688 ----------------------------------------------------------------
24689 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24690 States: Active Offline Active
24691 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24692 6737 Acked Write 1 Done Sent Done false
24693 6738 Acked Read 8 Done Sent Done false
24694 6739 Acked Read 9 Done Sent Done false
24695 6740 Acked Read 6 Done Sent Done false
24696 6741 Acked Read 5 Done Sent Done false
24697 6742 Acked Read 10 Done Sent Done false
24698 6743 Acked Write 6 Done Sent Done false
24699 6744 Acked Flush 0 Done Sent Done false
24700 6745 Acked Flush 0 Done Sent Done false
24701 6746 Acked Read 3 Done Sent Done false
24702 6747 Acked Read 1 Done Sent Done false
24703 6748 Acked Flush 0 Done Sent Done false
24704 6749 Acked Read 10 Done Sent Done false
24705 6750 Acked Read 8 Done Sent Done false
24706 6751 Acked Read 1 Done Sent Done false
24707 6752 Acked Flush 0 Done Sent Done false
24708 6753 Acked Write 7 Done Sent Done false
24709 6754 Acked Write 4 Done Sent Done false
24710 6755 Acked Write 3 Done Sent Done false
24711 6756 Acked Read 2 Done Sent Done false
24712 6757 Acked Read 6 Done Sent Done false
24713 6758 Acked Write 9 Done Sent Done false
24714 6759 Acked Write 1 Done Sent Done false
24715 6760 Acked Write 6 Done Sent Done false
24716 6761 Acked Read 7 Done Sent Done false
24717 6762 Acked Read 8 Done Sent Done false
24718 6763 Acked Read 3 Done Sent Done false
24719 6764 Acked Write 8 Done Sent Done false
24720 6765 Acked Read 2 Done Sent Done false
24721 6766 Acked Read 1 Done Sent Done false
24722 6767 Acked Write 5 Done Sent Done false
24723 6768 Acked Write 10 Done Sent Done false
24724 6769 Acked Read 5 Done Sent Done false
24725 6770 Acked Flush 0 Done Sent Done false
24726 6771 Acked Read 2 Done Sent Done false
24727 6772 Acked Read 7 Done Sent Done false
24728 6773 Acked Read 9 Done Sent Done false
24729 6774 Acked Read 1 Done Sent Done false
24730 6775 Acked Read 5 Done Sent Done false
24731 6776 Acked Write 4 Done Sent Done false
24732 6777 Acked Write 8 Done Sent Done false
24733 6778 Acked Write 9 Done Sent Done false
24734 6779 Acked Write 7 Done Sent Done false
24735 6780 Acked Read 3 Done Sent Done false
24736 6781 Acked Write 5 Done Sent Done false
24737 6782 Acked Read 1 Done Sent Done false
24738 6783 Acked Flush 0 Done Sent Done false
24739 6784 Acked Write 7 Done Sent Done false
24740 6785 Acked Flush 0 Done Sent Done false
24741 6786 Acked Write 1 Done Sent Done false
24742 6787 Acked Read 5 Done Sent Done false
24743 6788 Acked Flush 0 Done Sent Done false
24744 6789 Acked Write 10 Done Sent Done false
24745 6790 Acked Write 1 Done Sent Done false
24746 6791 Acked Write 2 Done Sent Done false
24747 6792 Acked Read 1 Done Sent Done false
24748 6793 Acked Write 7 Done Sent Done false
24749 6794 Acked Write 1 Done Sent Done false
24750 6795 Acked Flush 0 Done Sent Done false
24751 6796 Acked Read 1 Done Sent Done false
24752 6797 Acked Write 6 Done Sent Done false
24753 6798 Acked Read 7 Done Sent Done false
24754 6799 Acked Write 8 Done Sent Done false
24755 6800 Acked Read 9 Done Sent Done false
24756 6801 Acked Write 3 Done Sent Done false
24757 6802 Acked Write 6 Done Sent Done false
24758 6803 Acked Read 1 Done Sent Done false
24759 6804 Acked Read 4 Done Sent Done false
24760 6805 Acked Read 1 Done Sent Done false
24761 6806 Acked Read 2 Done Sent Done false
24762 6807 Acked Write 2 Done Sent Done false
24763 6808 Acked Read 10 Done Sent Done false
24764 6809 Acked Read 4 Done Sent Done false
24765 6810 Acked Read 9 Done Sent Done false
24766 6811 Acked Flush 0 Done Sent Done false
24767 6812 Acked Flush 0 Done Sent Done false
24768 6813 Acked Write 6 Done Sent Done false
24769 6814 Acked Write 6 Done Sent Done false
24770 6815 Acked Write 7 Done Sent Done false
24771 6816 Acked Write 5 Done Sent Done false
24772 6817 Acked Read 5 Done Sent Done false
24773 6818 Acked Write 2 Done Sent Done false
24774 6819 Acked Write 7 Done Sent Done false
24775 6820 Acked Read 5 Done Sent Done false
24776 6821 Acked Read 5 Done Sent Done false
24777 6822 Acked Write 8 Done Sent Done false
24778 6823 Acked Write 10 Done Sent Done false
24779 6824 Acked Write 3 Done Sent Done false
24780 6825 Acked Write 6 Done Sent Done false
24781 6826 Acked Read 1 Done Sent Done false
24782 6827 Acked Write 9 Done Sent Done false
24783 6828 Acked Write 2 Done Sent Done false
24784 6829 Acked Write 2 Done Sent Done false
24785 6830 Acked Read 2 Done Sent Done false
24786 6831 Acked Write 6 Done Sent Done false
24787 6832 Acked Read 1 Done Sent Done false
24788 6833 Acked Write 1 Done Sent Done false
24789 6834 Acked Write 6 Done Sent Done false
24790 6835 Acked Write 7 Done Sent Done false
24791 6836 Acked Write 4 Done Sent Done false
24792 6837 Acked Read 4 Done Sent Done false
24793 6838 Acked Write 6 Done Sent Done false
24794 6839 Acked Read 3 Done Sent Done false
24795 6840 Acked Read 1 Done Sent Done false
24796 6841 Acked Read 6 Done Sent Done false
24797 6842 Acked Write 6 Done Sent Done false
24798 6843 Acked Read 7 Done Sent Done false
24799 6844 Acked Read 3 Done Sent Done false
24800 6845 Acked Flush 0 Done Sent Done false
24801 6846 Acked Write 3 Done Sent Done false
24802 6847 Acked Read 7 Done Sent Done false
24803 6848 Acked Write 5 Done Sent Done false
24804 6849 Acked Flush 0 Done Sent Done false
24805 6850 Acked Write 5 Done Sent Done false
24806 6851 Acked Read 10 Done Sent Done false
24807 6852 Acked Read 7 Done Sent Done false
24808 6853 Acked Flush 0 Done Sent Done false
24809 6854 Acked Flush 0 Done Sent Done false
24810 6855 Acked Write 5 Done Sent Done false
24811 6856 Acked Write 5 Done Sent Done false
24812 6857 Acked Read 6 Done Sent Done false
24813 6858 Acked Read 1 Done Sent Done false
24814 6859 Acked Write 9 Done Sent Done false
24815 6860 Acked Flush 0 Done Sent Done false
24816 6861 Acked Read 2 Done Sent Done false
24817 6862 Acked Flush 0 Done Sent Done false
24818 6863 Acked Flush 0 Done Sent Done false
24819 6864 Acked Write 4 Done Sent Done false
24820 6865 Acked Flush 0 Done Sent Done false
24821 6866 Acked Write 6 Done Sent Done false
24822 6867 Acked Read 8 Done Sent Done false
24823 6868 Acked Write 1 Done Sent Done false
24824 6869 Acked Write 3 Done Sent Done false
24825 6870 Acked Write 8 Done Sent Done false
24826 6871 Acked Read 10 Done Sent Done false
24827 6872 Acked Read 1 Done Sent Done false
24828 6873 Acked Read 2 Done Sent Done false
24829 6874 Acked Read 4 Done Sent Done false
24830 6875 Acked Write 1 Done Sent Done false
24831 6876 Acked Read 8 Done Sent Done false
24832 6877 Acked Read 7 Done Sent Done false
24833 6878 Acked Read 2 Done Sent Done false
24834 6879 Acked Write 7 Done Sent Done false
24835 6880 Acked Read 2 Done Sent Done false
24836 6881 Acked Write 1 Done Sent Done false
24837 6882 Acked Read 6 Done Sent Done false
24838 6883 Acked Read 2 Done Sent Done false
24839 6884 Acked Read 1 Done Sent Done false
24840 6885 Acked Read 5 Done Sent Done false
24841 6886 Acked Write 3 Done Sent Done false
24842 6887 Acked Write 4 Done Sent Done false
24843 6888 Acked Write 8 Done Sent Done false
24844 6889 Acked Read 7 Done Sent Done false
24845 6890 Acked Read 3 Done Sent Done false
24846 6891 Acked Read 8 Done Sent Done false
24847 6892 Acked Read 10 Done Sent Done false
24848 6893 Acked Flush 0 Done Sent Done false
24849 6894 Acked Write 4 Done Sent Done false
24850 6895 Acked Read 1 Done Sent Done false
24851 6896 Acked Write 4 Done Sent Done false
24852 6897 Acked Read 5 Done Sent Done false
24853 6898 Acked Write 7 Done Sent Done false
24854 6899 Acked Read 6 Done Sent Done false
24855 6900 Acked Flush 0 Done Sent Done false
24856 6901 Acked Write 4 Done Sent Done false
24857 6902 Acked Write 9 Done Sent Done false
24858 6903 Acked Write 6 Done Sent Done false
24859 6904 Acked Write 4 Done Sent Done false
24860 6905 Acked Read 9 Done Sent Done false
24861 6906 Acked Read 6 Done Sent Done false
24862 6907 Acked Read 7 Done Sent Done false
24863 6908 Acked Write 9 Done Sent Done false
24864 6909 Acked Read 9 Done Sent Done false
24865 6910 Acked Read 5 Done Sent Done false
24866 6911 Acked Write 9 Done Sent Done false
24867 6912 Acked Write 8 Done Sent Done false
24868 6913 Acked Read 4 Done Sent Done false
24869 6914 Acked Flush 0 Done Sent Done false
24870 6915 Acked Read 6 Done Sent Done false
24871 6916 Acked Read 3 Done Sent Done false
24872 6917 Acked Read 1 Done Sent Done false
24873 6918 Acked Write 9 Done Sent Done false
24874 6919 Acked Read 9 Done Sent Done false
24875 6920 Acked Write 1 Done Sent Done false
24876 6921 Acked Read 3 Done Sent Done false
24877 6922 Acked Read 7 Done Sent Done false
24878 6923 Acked Read 8 Done Sent Done false
24879 6924 Acked Write 10 Done Sent Done false
24880 6925 Acked Read 3 Done Sent Done false
24881 6926 Acked Read 10 Done Sent Done false
24882 6927 Acked Read 10 Done Sent Done false
24883 6928 Acked Read 3 Done Sent Done false
24884 6929 Acked Read 4 Done Sent Done false
24885 6930 Acked Flush 0 Done Sent Done false
24886 6931 Acked Write 10 Done Sent Done false
24887 6932 Acked Read 5 Done Sent Done false
24888 6933 Acked Write 2 Done Sent Done false
24889 6934 Acked Write 8 Done Sent Done false
24890 6935 Acked Read 4 Done Sent Done false
24891 6936 Acked Write 6 Done Sent Done false
24892 6937 Acked Write 4 Done Sent Done false
24893 6938 Acked Write 5 Done Sent Done false
24894 6939 Acked Read 2 Done Sent Done false
24895 6940 Acked Write 4 Done Sent Done false
24896 6941 Acked Flush 0 Done Sent Done false
24897 6942 Acked Write 10 Done Sent Done false
24898 6943 Acked Write 2 Done Sent Done false
24899 6944 Acked Write 6 Done Sent Done false
24900 6945 Acked Flush 0 Done Sent Done false
24901 6946 Acked Write 6 Done Sent Done false
24902 6947 Acked Write 10 Done Sent Done false
24903 6948 Acked Write 7 Done Sent Done false
24904 6949 Acked Read 8 Done Sent Done false
24905 6950 Acked Read 9 Done Sent Done false
24906 6951 Acked Write 7 Done Sent Done false
24907 6952 Acked Write 9 Done Sent Done false
24908 6953 Acked Read 6 Done Sent Done false
24909 6954 Acked Write 3 Done Sent Done false
24910 6955 Acked Read 4 Done Sent Done false
24911 6956 Acked Read 9 Done Sent Done false
24912 6957 Acked Flush 0 Done Sent Done false
24913 6958 Acked Flush 0 Done Sent Done false
24914 6959 Acked Write 1 Done Sent Done false
24915 6960 Acked Read 7 Done Sent Done false
24916 6961 Acked Read 7 Done Sent Done false
24917 6962 Acked Read 6 Done Sent Done false
24918 6963 Acked Read 3 Done Sent Done false
24919 6964 Acked Read 1 Done Sent Done false
24920 6965 Acked Read 4 Done Sent Done false
24921 6966 Acked Read 2 Done Sent Done false
24922 6967 Acked Write 5 Done Sent Done false
24923 6968 Acked Read 7 Done Sent Done false
24924 6969 Acked Flush 0 Done Sent Done false
24925 6970 Acked Write 10 Done Sent Done false
24926 6971 Acked Read 5 Done Sent Done false
24927 6972 Acked Write 1 Done Sent Done false
24928 6973 Acked Read 10 Done Sent Done false
24929 6974 Acked Read 1 Done Sent Done false
24930 6975 Acked Write 4 Done Sent Done false
24931 6976 Acked Read 5 Done Sent Done false
24932 6977 Acked Write 2 Done Sent Done false
24933 6978 Acked Write 2 Done Sent Done false
24934 6979 Acked Read 7 Done Sent Done false
24935 6980 Acked Write 7 Done Sent Done false
24936 6981 Acked Flush 0 Done Sent Done false
24937 6982 Acked Read 2 Done Sent Done false
24938 6983 Acked Read 8 Done Sent Done false
24939 6984 Acked Read 3 Done Sent Done false
24940 6985 Acked Read 9 Done Sent Done false
24941 6986 Acked Write 1 Done Sent Done false
24942 6987 Acked Flush 0 Done Sent Done false
24943 6988 Acked Read 10 Done Sent Done false
24944 6989 Acked Write 5 Done Sent Done false
24945 6990 Acked Write 9 Done Sent Done false
24946 6991 Acked Read 8 Done Sent Done false
24947 6992 Acked Read 4 Done Sent Done false
24948 6993 Acked Flush 0 Done Sent Done false
24949 6994 Acked Read 5 Done Sent Done false
24950 6995 Acked Write 8 Done Sent Done false
24951 6996 Acked Write 7 Done Sent Done false
24952 6997 Acked Read 5 Done Sent Done false
24953 6998 Acked Read 10 Done Sent Done false
24954 6999 Acked Write 4 Done Sent Done false
24955 7000 Acked Read 5 Done Sent Done false
24956 7001 Acked Write 2 Done Sent Done false
24957 7002 Acked Read 9 Done Sent Done false
24958 7003 Acked Write 10 Done Sent Done false
24959 7004 Acked Read 5 Done Sent Done false
24960 7005 Acked Read 9 Done Sent Done false
24961 7006 Acked Write 8 Done Sent Done false
24962 7007 Acked Write 3 Done Sent Done false
24963 7008 Acked Read 10 Done Sent Done false
24964 7009 Acked Read 4 Done Sent Done false
24965 7010 Acked Read 3 Done Sent Done false
24966 7011 Acked Flush 0 Done Sent Done false
24967 7012 Acked Write 2 Done Sent Done false
24968 7013 Acked Read 5 Done Sent Done false
24969 7014 Acked Read 1 Done Sent Done false
24970 7015 Acked Read 5 Done Sent Done false
24971 7016 Acked Write 2 Done Sent Done false
24972 7017 Acked Write 3 Done Sent Done false
24973 7018 Acked Write 9 Done Sent Done false
24974 7019 Acked Write 6 Done Sent Done false
24975 7020 Acked Read 8 Done Sent Done false
24976 7021 Acked Read 5 Done Sent Done false
24977 7022 Acked Write 1 Done Sent Done false
24978 7023 Acked Read 10 Done Sent Done false
24979 7024 Acked Read 5 Done Sent Done false
24980 7025 Acked Read 5 Done Sent Done false
24981 7026 Acked Write 6 Done Sent Done false
24982 7027 Acked Write 7 Done Sent Done false
24983 7028 Acked Read 3 Done Sent Done false
24984 7029 Acked Read 8 Done Sent Done false
24985 7030 Acked Read 10 Done Sent Done false
24986 7031 Acked Read 8 Done Sent Done false
24987 7032 Acked Flush 0 Done Sent Done false
24988 7033 Acked Read 6 Done Sent Done false
24989 7034 Acked Write 8 Done Sent Done false
24990 7035 Acked Read 2 Done Sent Done false
24991 7036 Acked Write 2 Done Sent Done false
24992 7037 Acked Read 7 Done Sent Done false
24993 STATES DS:0 DS:1 DS:2 TOTAL
24994 Sent 0 301 0 301
24995 Done 301 0 301 602
24996 Skipped 0 0 0 0
24997 Error 0 0 0 0
24998 Last Flush: Some(JobId(7032)) Some(JobId(6736)) Some(JobId(7032))
24999 Downstairs last five completed: 6736 6735 6734 6733 6732
25000 Upstairs last five completed: 7037 7036 7035 7034 7033
25001 CLIENT: Up:0 ds:301 act:2
250022025-02-07T20:12:51.489ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
250032025-02-07T20:12:52.266ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
250042025-02-07T20:12:52.266ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
250052025-02-07T20:12:52.266ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
250062025-02-07T20:12:52.266ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
250072025-02-07T20:12:52.266ZINFOcrucible: client task is exiting = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
250082025-02-07T20:12:52.767ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
250092025-02-07T20:12:53.492ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
25010 [8820] got start action
25011 [8820] I am going to restart
25012 Starting downstairs at port 8820
25013 Make output file at "/tmp/dsc/downstairs-8820.txt"
25014 [8820][1] reports Starting
25015 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1229)
25016 [8820][1] reports Running
250172025-02-07T20:12:54.493ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
25018 ----------------------------------------------------------------
25019 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25020 States: Active Offline Active
25021 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25022 6737 Acked Write 1 Done Sent Done false
25023 6738 Acked Read 8 Done Sent Done false
25024 6739 Acked Read 9 Done Sent Done false
25025 6740 Acked Read 6 Done Sent Done false
25026 6741 Acked Read 5 Done Sent Done false
25027 6742 Acked Read 10 Done Sent Done false
25028 6743 Acked Write 6 Done Sent Done false
25029 6744 Acked Flush 0 Done Sent Done false
25030 6745 Acked Flush 0 Done Sent Done false
25031 6746 Acked Read 3 Done Sent Done false
25032 6747 Acked Read 1 Done Sent Done false
25033 6748 Acked Flush 0 Done Sent Done false
25034 6749 Acked Read 10 Done Sent Done false
25035 6750 Acked Read 8 Done Sent Done false
25036 6751 Acked Read 1 Done Sent Done false
25037 6752 Acked Flush 0 Done Sent Done false
25038 6753 Acked Write 7 Done Sent Done false
25039 6754 Acked Write 4 Done Sent Done false
25040 6755 Acked Write 3 Done Sent Done false
25041 6756 Acked Read 2 Done Sent Done false
25042 6757 Acked Read 6 Done Sent Done false
25043 6758 Acked Write 9 Done Sent Done false
25044 6759 Acked Write 1 Done Sent Done false
25045 6760 Acked Write 6 Done Sent Done false
25046 6761 Acked Read 7 Done Sent Done false
25047 6762 Acked Read 8 Done Sent Done false
25048 6763 Acked Read 3 Done Sent Done false
25049 6764 Acked Write 8 Done Sent Done false
25050 6765 Acked Read 2 Done Sent Done false
25051 6766 Acked Read 1 Done Sent Done false
25052 6767 Acked Write 5 Done Sent Done false
25053 6768 Acked Write 10 Done Sent Done false
25054 6769 Acked Read 5 Done Sent Done false
25055 6770 Acked Flush 0 Done Sent Done false
25056 6771 Acked Read 2 Done Sent Done false
25057 6772 Acked Read 7 Done Sent Done false
25058 6773 Acked Read 9 Done Sent Done false
25059 6774 Acked Read 1 Done Sent Done false
25060 6775 Acked Read 5 Done Sent Done false
25061 6776 Acked Write 4 Done Sent Done false
25062 6777 Acked Write 8 Done Sent Done false
25063 6778 Acked Write 9 Done Sent Done false
25064 6779 Acked Write 7 Done Sent Done false
25065 6780 Acked Read 3 Done Sent Done false
25066 6781 Acked Write 5 Done Sent Done false
25067 6782 Acked Read 1 Done Sent Done false
25068 6783 Acked Flush 0 Done Sent Done false
25069 6784 Acked Write 7 Done Sent Done false
25070 6785 Acked Flush 0 Done Sent Done false
25071 6786 Acked Write 1 Done Sent Done false
25072 6787 Acked Read 5 Done Sent Done false
25073 6788 Acked Flush 0 Done Sent Done false
25074 6789 Acked Write 10 Done Sent Done false
25075 6790 Acked Write 1 Done Sent Done false
25076 6791 Acked Write 2 Done Sent Done false
25077 6792 Acked Read 1 Done Sent Done false
25078 6793 Acked Write 7 Done Sent Done false
25079 6794 Acked Write 1 Done Sent Done false
25080 6795 Acked Flush 0 Done Sent Done false
25081 6796 Acked Read 1 Done Sent Done false
25082 6797 Acked Write 6 Done Sent Done false
25083 6798 Acked Read 7 Done Sent Done false
25084 6799 Acked Write 8 Done Sent Done false
25085 6800 Acked Read 9 Done Sent Done false
25086 6801 Acked Write 3 Done Sent Done false
25087 6802 Acked Write 6 Done Sent Done false
25088 6803 Acked Read 1 Done Sent Done false
25089 6804 Acked Read 4 Done Sent Done false
25090 6805 Acked Read 1 Done Sent Done false
25091 6806 Acked Read 2 Done Sent Done false
25092 6807 Acked Write 2 Done Sent Done false
25093 6808 Acked Read 10 Done Sent Done false
25094 6809 Acked Read 4 Done Sent Done false
25095 6810 Acked Read 9 Done Sent Done false
25096 6811 Acked Flush 0 Done Sent Done false
25097 6812 Acked Flush 0 Done Sent Done false
25098 6813 Acked Write 6 Done Sent Done false
25099 6814 Acked Write 6 Done Sent Done false
25100 6815 Acked Write 7 Done Sent Done false
25101 6816 Acked Write 5 Done Sent Done false
25102 6817 Acked Read 5 Done Sent Done false
25103 6818 Acked Write 2 Done Sent Done false
25104 6819 Acked Write 7 Done Sent Done false
25105 6820 Acked Read 5 Done Sent Done false
25106 6821 Acked Read 5 Done Sent Done false
25107 6822 Acked Write 8 Done Sent Done false
25108 6823 Acked Write 10 Done Sent Done false
25109 6824 Acked Write 3 Done Sent Done false
25110 6825 Acked Write 6 Done Sent Done false
25111 6826 Acked Read 1 Done Sent Done false
25112 6827 Acked Write 9 Done Sent Done false
25113 6828 Acked Write 2 Done Sent Done false
25114 6829 Acked Write 2 Done Sent Done false
25115 6830 Acked Read 2 Done Sent Done false
25116 6831 Acked Write 6 Done Sent Done false
25117 6832 Acked Read 1 Done Sent Done false
25118 6833 Acked Write 1 Done Sent Done false
25119 6834 Acked Write 6 Done Sent Done false
25120 6835 Acked Write 7 Done Sent Done false
25121 6836 Acked Write 4 Done Sent Done false
25122 6837 Acked Read 4 Done Sent Done false
25123 6838 Acked Write 6 Done Sent Done false
25124 6839 Acked Read 3 Done Sent Done false
25125 6840 Acked Read 1 Done Sent Done false
25126 6841 Acked Read 6 Done Sent Done false
25127 6842 Acked Write 6 Done Sent Done false
25128 6843 Acked Read 7 Done Sent Done false
25129 6844 Acked Read 3 Done Sent Done false
25130 6845 Acked Flush 0 Done Sent Done false
25131 6846 Acked Write 3 Done Sent Done false
25132 6847 Acked Read 7 Done Sent Done false
25133 6848 Acked Write 5 Done Sent Done false
25134 6849 Acked Flush 0 Done Sent Done false
25135 6850 Acked Write 5 Done Sent Done false
25136 6851 Acked Read 10 Done Sent Done false
25137 6852 Acked Read 7 Done Sent Done false
25138 6853 Acked Flush 0 Done Sent Done false
25139 6854 Acked Flush 0 Done Sent Done false
25140 6855 Acked Write 5 Done Sent Done false
25141 6856 Acked Write 5 Done Sent Done false
25142 6857 Acked Read 6 Done Sent Done false
25143 6858 Acked Read 1 Done Sent Done false
25144 6859 Acked Write 9 Done Sent Done false
25145 6860 Acked Flush 0 Done Sent Done false
25146 6861 Acked Read 2 Done Sent Done false
25147 6862 Acked Flush 0 Done Sent Done false
25148 6863 Acked Flush 0 Done Sent Done false
25149 6864 Acked Write 4 Done Sent Done false
25150 6865 Acked Flush 0 Done Sent Done false
25151 6866 Acked Write 6 Done Sent Done false
25152 6867 Acked Read 8 Done Sent Done false
25153 6868 Acked Write 1 Done Sent Done false
25154 6869 Acked Write 3 Done Sent Done false
25155 6870 Acked Write 8 Done Sent Done false
25156 6871 Acked Read 10 Done Sent Done false
25157 6872 Acked Read 1 Done Sent Done false
25158 6873 Acked Read 2 Done Sent Done false
25159 6874 Acked Read 4 Done Sent Done false
25160 6875 Acked Write 1 Done Sent Done false
25161 6876 Acked Read 8 Done Sent Done false
25162 6877 Acked Read 7 Done Sent Done false
25163 6878 Acked Read 2 Done Sent Done false
25164 6879 Acked Write 7 Done Sent Done false
25165 6880 Acked Read 2 Done Sent Done false
25166 6881 Acked Write 1 Done Sent Done false
25167 6882 Acked Read 6 Done Sent Done false
25168 6883 Acked Read 2 Done Sent Done false
25169 6884 Acked Read 1 Done Sent Done false
25170 6885 Acked Read 5 Done Sent Done false
25171 6886 Acked Write 3 Done Sent Done false
25172 6887 Acked Write 4 Done Sent Done false
25173 6888 Acked Write 8 Done Sent Done false
25174 6889 Acked Read 7 Done Sent Done false
25175 6890 Acked Read 3 Done Sent Done false
25176 6891 Acked Read 8 Done Sent Done false
25177 6892 Acked Read 10 Done Sent Done false
25178 6893 Acked Flush 0 Done Sent Done false
25179 6894 Acked Write 4 Done Sent Done false
25180 6895 Acked Read 1 Done Sent Done false
25181 6896 Acked Write 4 Done Sent Done false
25182 6897 Acked Read 5 Done Sent Done false
25183 6898 Acked Write 7 Done Sent Done false
25184 6899 Acked Read 6 Done Sent Done false
25185 6900 Acked Flush 0 Done Sent Done false
25186 6901 Acked Write 4 Done Sent Done false
25187 6902 Acked Write 9 Done Sent Done false
25188 6903 Acked Write 6 Done Sent Done false
25189 6904 Acked Write 4 Done Sent Done false
25190 6905 Acked Read 9 Done Sent Done false
25191 6906 Acked Read 6 Done Sent Done false
25192 6907 Acked Read 7 Done Sent Done false
25193 6908 Acked Write 9 Done Sent Done false
25194 6909 Acked Read 9 Done Sent Done false
25195 6910 Acked Read 5 Done Sent Done false
25196 6911 Acked Write 9 Done Sent Done false
25197 6912 Acked Write 8 Done Sent Done false
25198 6913 Acked Read 4 Done Sent Done false
25199 6914 Acked Flush 0 Done Sent Done false
25200 6915 Acked Read 6 Done Sent Done false
25201 6916 Acked Read 3 Done Sent Done false
25202 6917 Acked Read 1 Done Sent Done false
25203 6918 Acked Write 9 Done Sent Done false
25204 6919 Acked Read 9 Done Sent Done false
25205 6920 Acked Write 1 Done Sent Done false
25206 6921 Acked Read 3 Done Sent Done false
25207 6922 Acked Read 7 Done Sent Done false
25208 6923 Acked Read 8 Done Sent Done false
25209 6924 Acked Write 10 Done Sent Done false
25210 6925 Acked Read 3 Done Sent Done false
25211 6926 Acked Read 10 Done Sent Done false
25212 6927 Acked Read 10 Done Sent Done false
25213 6928 Acked Read 3 Done Sent Done false
25214 6929 Acked Read 4 Done Sent Done false
25215 6930 Acked Flush 0 Done Sent Done false
25216 6931 Acked Write 10 Done Sent Done false
25217 6932 Acked Read 5 Done Sent Done false
25218 6933 Acked Write 2 Done Sent Done false
25219 6934 Acked Write 8 Done Sent Done false
25220 6935 Acked Read 4 Done Sent Done false
25221 6936 Acked Write 6 Done Sent Done false
25222 6937 Acked Write 4 Done Sent Done false
25223 6938 Acked Write 5 Done Sent Done false
25224 6939 Acked Read 2 Done Sent Done false
25225 6940 Acked Write 4 Done Sent Done false
25226 6941 Acked Flush 0 Done Sent Done false
25227 6942 Acked Write 10 Done Sent Done false
25228 6943 Acked Write 2 Done Sent Done false
25229 6944 Acked Write 6 Done Sent Done false
25230 6945 Acked Flush 0 Done Sent Done false
25231 6946 Acked Write 6 Done Sent Done false
25232 6947 Acked Write 10 Done Sent Done false
25233 6948 Acked Write 7 Done Sent Done false
25234 6949 Acked Read 8 Done Sent Done false
25235 6950 Acked Read 9 Done Sent Done false
25236 6951 Acked Write 7 Done Sent Done false
25237 6952 Acked Write 9 Done Sent Done false
25238 6953 Acked Read 6 Done Sent Done false
25239 6954 Acked Write 3 Done Sent Done false
25240 6955 Acked Read 4 Done Sent Done false
25241 6956 Acked Read 9 Done Sent Done false
25242 6957 Acked Flush 0 Done Sent Done false
25243 6958 Acked Flush 0 Done Sent Done false
25244 6959 Acked Write 1 Done Sent Done false
25245 6960 Acked Read 7 Done Sent Done false
25246 6961 Acked Read 7 Done Sent Done false
25247 6962 Acked Read 6 Done Sent Done false
25248 6963 Acked Read 3 Done Sent Done false
25249 6964 Acked Read 1 Done Sent Done false
25250 6965 Acked Read 4 Done Sent Done false
25251 6966 Acked Read 2 Done Sent Done false
25252 6967 Acked Write 5 Done Sent Done false
25253 6968 Acked Read 7 Done Sent Done false
25254 6969 Acked Flush 0 Done Sent Done false
25255 6970 Acked Write 10 Done Sent Done false
25256 6971 Acked Read 5 Done Sent Done false
25257 6972 Acked Write 1 Done Sent Done false
25258 6973 Acked Read 10 Done Sent Done false
25259 6974 Acked Read 1 Done Sent Done false
25260 6975 Acked Write 4 Done Sent Done false
25261 6976 Acked Read 5 Done Sent Done false
25262 6977 Acked Write 2 Done Sent Done false
25263 6978 Acked Write 2 Done Sent Done false
25264 6979 Acked Read 7 Done Sent Done false
25265 6980 Acked Write 7 Done Sent Done false
25266 6981 Acked Flush 0 Done Sent Done false
25267 6982 Acked Read 2 Done Sent Done false
25268 6983 Acked Read 8 Done Sent Done false
25269 6984 Acked Read 3 Done Sent Done false
25270 6985 Acked Read 9 Done Sent Done false
25271 6986 Acked Write 1 Done Sent Done false
25272 6987 Acked Flush 0 Done Sent Done false
25273 6988 Acked Read 10 Done Sent Done false
25274 6989 Acked Write 5 Done Sent Done false
25275 6990 Acked Write 9 Done Sent Done false
25276 6991 Acked Read 8 Done Sent Done false
25277 6992 Acked Read 4 Done Sent Done false
25278 6993 Acked Flush 0 Done Sent Done false
25279 6994 Acked Read 5 Done Sent Done false
25280 6995 Acked Write 8 Done Sent Done false
25281 6996 Acked Write 7 Done Sent Done false
25282 6997 Acked Read 5 Done Sent Done false
25283 6998 Acked Read 10 Done Sent Done false
25284 6999 Acked Write 4 Done Sent Done false
25285 7000 Acked Read 5 Done Sent Done false
25286 7001 Acked Write 2 Done Sent Done false
25287 7002 Acked Read 9 Done Sent Done false
25288 7003 Acked Write 10 Done Sent Done false
25289 7004 Acked Read 5 Done Sent Done false
25290 7005 Acked Read 9 Done Sent Done false
25291 7006 Acked Write 8 Done Sent Done false
25292 7007 Acked Write 3 Done Sent Done false
25293 7008 Acked Read 10 Done Sent Done false
25294 7009 Acked Read 4 Done Sent Done false
25295 7010 Acked Read 3 Done Sent Done false
25296 7011 Acked Flush 0 Done Sent Done false
25297 7012 Acked Write 2 Done Sent Done false
25298 7013 Acked Read 5 Done Sent Done false
25299 7014 Acked Read 1 Done Sent Done false
25300 7015 Acked Read 5 Done Sent Done false
25301 7016 Acked Write 2 Done Sent Done false
25302 7017 Acked Write 3 Done Sent Done false
25303 7018 Acked Write 9 Done Sent Done false
25304 7019 Acked Write 6 Done Sent Done false
25305 7020 Acked Read 8 Done Sent Done false
25306 7021 Acked Read 5 Done Sent Done false
25307 7022 Acked Write 1 Done Sent Done false
25308 7023 Acked Read 10 Done Sent Done false
25309 7024 Acked Read 5 Done Sent Done false
25310 7025 Acked Read 5 Done Sent Done false
25311 7026 Acked Write 6 Done Sent Done false
25312 7027 Acked Write 7 Done Sent Done false
25313 7028 Acked Read 3 Done Sent Done false
25314 7029 Acked Read 8 Done Sent Done false
25315 7030 Acked Read 10 Done Sent Done false
25316 7031 Acked Read 8 Done Sent Done false
25317 7032 Acked Flush 0 Done Sent Done false
25318 7033 Acked Read 6 Done Sent Done false
25319 7034 Acked Write 8 Done Sent Done false
25320 7035 Acked Read 2 Done Sent Done false
25321 7036 Acked Write 2 Done Sent Done false
25322 7037 Acked Read 7 Done Sent Done false
25323 STATES DS:0 DS:1 DS:2 TOTAL
25324 Sent 0 301 0 301
25325 Done 301 0 301 602
25326 Skipped 0 0 0 0
25327 Error 0 0 0 0
25328 Last Flush: Some(JobId(7032)) Some(JobId(6736)) Some(JobId(7032))
25329 Downstairs last five completed: 6736 6735 6734 6733 6732
25330 Upstairs last five completed: 7037 7036 7035 7034 7033
25331 CLIENT: Up:0 ds:301 act:2
253322025-02-07T20:12:55.517ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
253332025-02-07T20:12:56.497ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
253342025-02-07T20:12:57.499ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
253352025-02-07T20:12:58.501ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
25336 ----------------------------------------------------------------
25337 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25338 States: Active Offline Active
25339 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25340 6737 Acked Write 1 Done Sent Done false
25341 6738 Acked Read 8 Done Sent Done false
25342 6739 Acked Read 9 Done Sent Done false
25343 6740 Acked Read 6 Done Sent Done false
25344 6741 Acked Read 5 Done Sent Done false
25345 6742 Acked Read 10 Done Sent Done false
25346 6743 Acked Write 6 Done Sent Done false
25347 6744 Acked Flush 0 Done Sent Done false
25348 6745 Acked Flush 0 Done Sent Done false
25349 6746 Acked Read 3 Done Sent Done false
25350 6747 Acked Read 1 Done Sent Done false
25351 6748 Acked Flush 0 Done Sent Done false
25352 6749 Acked Read 10 Done Sent Done false
25353 6750 Acked Read 8 Done Sent Done false
25354 6751 Acked Read 1 Done Sent Done false
25355 6752 Acked Flush 0 Done Sent Done false
25356 6753 Acked Write 7 Done Sent Done false
25357 6754 Acked Write 4 Done Sent Done false
25358 6755 Acked Write 3 Done Sent Done false
25359 6756 Acked Read 2 Done Sent Done false
25360 6757 Acked Read 6 Done Sent Done false
25361 6758 Acked Write 9 Done Sent Done false
25362 6759 Acked Write 1 Done Sent Done false
25363 6760 Acked Write 6 Done Sent Done false
25364 6761 Acked Read 7 Done Sent Done false
25365 6762 Acked Read 8 Done Sent Done false
25366 6763 Acked Read 3 Done Sent Done false
25367 6764 Acked Write 8 Done Sent Done false
25368 6765 Acked Read 2 Done Sent Done false
25369 6766 Acked Read 1 Done Sent Done false
25370 6767 Acked Write 5 Done Sent Done false
25371 6768 Acked Write 10 Done Sent Done false
25372 6769 Acked Read 5 Done Sent Done false
25373 6770 Acked Flush 0 Done Sent Done false
25374 6771 Acked Read 2 Done Sent Done false
25375 6772 Acked Read 7 Done Sent Done false
25376 6773 Acked Read 9 Done Sent Done false
25377 6774 Acked Read 1 Done Sent Done false
25378 6775 Acked Read 5 Done Sent Done false
25379 6776 Acked Write 4 Done Sent Done false
25380 6777 Acked Write 8 Done Sent Done false
25381 6778 Acked Write 9 Done Sent Done false
25382 6779 Acked Write 7 Done Sent Done false
25383 6780 Acked Read 3 Done Sent Done false
25384 6781 Acked Write 5 Done Sent Done false
25385 6782 Acked Read 1 Done Sent Done false
25386 6783 Acked Flush 0 Done Sent Done false
25387 6784 Acked Write 7 Done Sent Done false
25388 6785 Acked Flush 0 Done Sent Done false
25389 6786 Acked Write 1 Done Sent Done false
25390 6787 Acked Read 5 Done Sent Done false
25391 6788 Acked Flush 0 Done Sent Done false
25392 6789 Acked Write 10 Done Sent Done false
25393 6790 Acked Write 1 Done Sent Done false
25394 6791 Acked Write 2 Done Sent Done false
25395 6792 Acked Read 1 Done Sent Done false
25396 6793 Acked Write 7 Done Sent Done false
25397 6794 Acked Write 1 Done Sent Done false
25398 6795 Acked Flush 0 Done Sent Done false
25399 6796 Acked Read 1 Done Sent Done false
25400 6797 Acked Write 6 Done Sent Done false
25401 6798 Acked Read 7 Done Sent Done false
25402 6799 Acked Write 8 Done Sent Done false
25403 6800 Acked Read 9 Done Sent Done false
25404 6801 Acked Write 3 Done Sent Done false
25405 6802 Acked Write 6 Done Sent Done false
25406 6803 Acked Read 1 Done Sent Done false
25407 6804 Acked Read 4 Done Sent Done false
25408 6805 Acked Read 1 Done Sent Done false
25409 6806 Acked Read 2 Done Sent Done false
25410 6807 Acked Write 2 Done Sent Done false
25411 6808 Acked Read 10 Done Sent Done false
25412 6809 Acked Read 4 Done Sent Done false
25413 6810 Acked Read 9 Done Sent Done false
25414 6811 Acked Flush 0 Done Sent Done false
25415 6812 Acked Flush 0 Done Sent Done false
25416 6813 Acked Write 6 Done Sent Done false
25417 6814 Acked Write 6 Done Sent Done false
25418 6815 Acked Write 7 Done Sent Done false
25419 6816 Acked Write 5 Done Sent Done false
25420 6817 Acked Read 5 Done Sent Done false
25421 6818 Acked Write 2 Done Sent Done false
25422 6819 Acked Write 7 Done Sent Done false
25423 6820 Acked Read 5 Done Sent Done false
25424 6821 Acked Read 5 Done Sent Done false
25425 6822 Acked Write 8 Done Sent Done false
25426 6823 Acked Write 10 Done Sent Done false
25427 6824 Acked Write 3 Done Sent Done false
25428 6825 Acked Write 6 Done Sent Done false
25429 6826 Acked Read 1 Done Sent Done false
25430 6827 Acked Write 9 Done Sent Done false
25431 6828 Acked Write 2 Done Sent Done false
25432 6829 Acked Write 2 Done Sent Done false
25433 6830 Acked Read 2 Done Sent Done false
25434 6831 Acked Write 6 Done Sent Done false
25435 6832 Acked Read 1 Done Sent Done false
25436 6833 Acked Write 1 Done Sent Done false
25437 6834 Acked Write 6 Done Sent Done false
25438 6835 Acked Write 7 Done Sent Done false
25439 6836 Acked Write 4 Done Sent Done false
25440 6837 Acked Read 4 Done Sent Done false
25441 6838 Acked Write 6 Done Sent Done false
25442 6839 Acked Read 3 Done Sent Done false
25443 6840 Acked Read 1 Done Sent Done false
25444 6841 Acked Read 6 Done Sent Done false
25445 6842 Acked Write 6 Done Sent Done false
25446 6843 Acked Read 7 Done Sent Done false
25447 6844 Acked Read 3 Done Sent Done false
25448 6845 Acked Flush 0 Done Sent Done false
25449 6846 Acked Write 3 Done Sent Done false
25450 6847 Acked Read 7 Done Sent Done false
25451 6848 Acked Write 5 Done Sent Done false
25452 6849 Acked Flush 0 Done Sent Done false
25453 6850 Acked Write 5 Done Sent Done false
25454 6851 Acked Read 10 Done Sent Done false
25455 6852 Acked Read 7 Done Sent Done false
25456 6853 Acked Flush 0 Done Sent Done false
25457 6854 Acked Flush 0 Done Sent Done false
25458 6855 Acked Write 5 Done Sent Done false
25459 6856 Acked Write 5 Done Sent Done false
25460 6857 Acked Read 6 Done Sent Done false
25461 6858 Acked Read 1 Done Sent Done false
25462 6859 Acked Write 9 Done Sent Done false
25463 6860 Acked Flush 0 Done Sent Done false
25464 6861 Acked Read 2 Done Sent Done false
25465 6862 Acked Flush 0 Done Sent Done false
25466 6863 Acked Flush 0 Done Sent Done false
25467 6864 Acked Write 4 Done Sent Done false
25468 6865 Acked Flush 0 Done Sent Done false
25469 6866 Acked Write 6 Done Sent Done false
25470 6867 Acked Read 8 Done Sent Done false
25471 6868 Acked Write 1 Done Sent Done false
25472 6869 Acked Write 3 Done Sent Done false
25473 6870 Acked Write 8 Done Sent Done false
25474 6871 Acked Read 10 Done Sent Done false
25475 6872 Acked Read 1 Done Sent Done false
25476 6873 Acked Read 2 Done Sent Done false
25477 6874 Acked Read 4 Done Sent Done false
25478 6875 Acked Write 1 Done Sent Done false
25479 6876 Acked Read 8 Done Sent Done false
25480 6877 Acked Read 7 Done Sent Done false
25481 6878 Acked Read 2 Done Sent Done false
25482 6879 Acked Write 7 Done Sent Done false
25483 6880 Acked Read 2 Done Sent Done false
25484 6881 Acked Write 1 Done Sent Done false
25485 6882 Acked Read 6 Done Sent Done false
25486 6883 Acked Read 2 Done Sent Done false
25487 6884 Acked Read 1 Done Sent Done false
25488 6885 Acked Read 5 Done Sent Done false
25489 6886 Acked Write 3 Done Sent Done false
25490 6887 Acked Write 4 Done Sent Done false
25491 6888 Acked Write 8 Done Sent Done false
25492 6889 Acked Read 7 Done Sent Done false
25493 6890 Acked Read 3 Done Sent Done false
25494 6891 Acked Read 8 Done Sent Done false
25495 6892 Acked Read 10 Done Sent Done false
25496 6893 Acked Flush 0 Done Sent Done false
25497 6894 Acked Write 4 Done Sent Done false
25498 6895 Acked Read 1 Done Sent Done false
25499 6896 Acked Write 4 Done Sent Done false
25500 6897 Acked Read 5 Done Sent Done false
25501 6898 Acked Write 7 Done Sent Done false
25502 6899 Acked Read 6 Done Sent Done false
25503 6900 Acked Flush 0 Done Sent Done false
25504 6901 Acked Write 4 Done Sent Done false
25505 6902 Acked Write 9 Done Sent Done false
25506 6903 Acked Write 6 Done Sent Done false
25507 6904 Acked Write 4 Done Sent Done false
25508 6905 Acked Read 9 Done Sent Done false
25509 6906 Acked Read 6 Done Sent Done false
25510 6907 Acked Read 7 Done Sent Done false
25511 6908 Acked Write 9 Done Sent Done false
25512 6909 Acked Read 9 Done Sent Done false
25513 6910 Acked Read 5 Done Sent Done false
25514 6911 Acked Write 9 Done Sent Done false
25515 6912 Acked Write 8 Done Sent Done false
25516 6913 Acked Read 4 Done Sent Done false
25517 6914 Acked Flush 0 Done Sent Done false
25518 6915 Acked Read 6 Done Sent Done false
25519 6916 Acked Read 3 Done Sent Done false
25520 6917 Acked Read 1 Done Sent Done false
25521 6918 Acked Write 9 Done Sent Done false
25522 6919 Acked Read 9 Done Sent Done false
25523 6920 Acked Write 1 Done Sent Done false
25524 6921 Acked Read 3 Done Sent Done false
25525 6922 Acked Read 7 Done Sent Done false
25526 6923 Acked Read 8 Done Sent Done false
25527 6924 Acked Write 10 Done Sent Done false
25528 6925 Acked Read 3 Done Sent Done false
25529 6926 Acked Read 10 Done Sent Done false
25530 6927 Acked Read 10 Done Sent Done false
25531 6928 Acked Read 3 Done Sent Done false
25532 6929 Acked Read 4 Done Sent Done false
25533 6930 Acked Flush 0 Done Sent Done false
25534 6931 Acked Write 10 Done Sent Done false
25535 6932 Acked Read 5 Done Sent Done false
25536 6933 Acked Write 2 Done Sent Done false
25537 6934 Acked Write 8 Done Sent Done false
25538 6935 Acked Read 4 Done Sent Done false
25539 6936 Acked Write 6 Done Sent Done false
25540 6937 Acked Write 4 Done Sent Done false
25541 6938 Acked Write 5 Done Sent Done false
25542 6939 Acked Read 2 Done Sent Done false
25543 6940 Acked Write 4 Done Sent Done false
25544 6941 Acked Flush 0 Done Sent Done false
25545 6942 Acked Write 10 Done Sent Done false
25546 6943 Acked Write 2 Done Sent Done false
25547 6944 Acked Write 6 Done Sent Done false
25548 6945 Acked Flush 0 Done Sent Done false
25549 6946 Acked Write 6 Done Sent Done false
25550 6947 Acked Write 10 Done Sent Done false
25551 6948 Acked Write 7 Done Sent Done false
25552 6949 Acked Read 8 Done Sent Done false
25553 6950 Acked Read 9 Done Sent Done false
25554 6951 Acked Write 7 Done Sent Done false
25555 6952 Acked Write 9 Done Sent Done false
25556 6953 Acked Read 6 Done Sent Done false
25557 6954 Acked Write 3 Done Sent Done false
25558 6955 Acked Read 4 Done Sent Done false
25559 6956 Acked Read 9 Done Sent Done false
25560 6957 Acked Flush 0 Done Sent Done false
25561 6958 Acked Flush 0 Done Sent Done false
25562 6959 Acked Write 1 Done Sent Done false
25563 6960 Acked Read 7 Done Sent Done false
25564 6961 Acked Read 7 Done Sent Done false
25565 6962 Acked Read 6 Done Sent Done false
25566 6963 Acked Read 3 Done Sent Done false
25567 6964 Acked Read 1 Done Sent Done false
25568 6965 Acked Read 4 Done Sent Done false
25569 6966 Acked Read 2 Done Sent Done false
25570 6967 Acked Write 5 Done Sent Done false
25571 6968 Acked Read 7 Done Sent Done false
25572 6969 Acked Flush 0 Done Sent Done false
25573 6970 Acked Write 10 Done Sent Done false
25574 6971 Acked Read 5 Done Sent Done false
25575 6972 Acked Write 1 Done Sent Done false
25576 6973 Acked Read 10 Done Sent Done false
25577 6974 Acked Read 1 Done Sent Done false
25578 6975 Acked Write 4 Done Sent Done false
25579 6976 Acked Read 5 Done Sent Done false
25580 6977 Acked Write 2 Done Sent Done false
25581 6978 Acked Write 2 Done Sent Done false
25582 6979 Acked Read 7 Done Sent Done false
25583 6980 Acked Write 7 Done Sent Done false
25584 6981 Acked Flush 0 Done Sent Done false
25585 6982 Acked Read 2 Done Sent Done false
25586 6983 Acked Read 8 Done Sent Done false
25587 6984 Acked Read 3 Done Sent Done false
25588 6985 Acked Read 9 Done Sent Done false
25589 6986 Acked Write 1 Done Sent Done false
25590 6987 Acked Flush 0 Done Sent Done false
25591 6988 Acked Read 10 Done Sent Done false
25592 6989 Acked Write 5 Done Sent Done false
25593 6990 Acked Write 9 Done Sent Done false
25594 6991 Acked Read 8 Done Sent Done false
25595 6992 Acked Read 4 Done Sent Done false
25596 6993 Acked Flush 0 Done Sent Done false
25597 6994 Acked Read 5 Done Sent Done false
25598 6995 Acked Write 8 Done Sent Done false
25599 6996 Acked Write 7 Done Sent Done false
25600 6997 Acked Read 5 Done Sent Done false
25601 6998 Acked Read 10 Done Sent Done false
25602 6999 Acked Write 4 Done Sent Done false
25603 7000 Acked Read 5 Done Sent Done false
25604 7001 Acked Write 2 Done Sent Done false
25605 7002 Acked Read 9 Done Sent Done false
25606 7003 Acked Write 10 Done Sent Done false
25607 7004 Acked Read 5 Done Sent Done false
25608 7005 Acked Read 9 Done Sent Done false
25609 7006 Acked Write 8 Done Sent Done false
25610 7007 Acked Write 3 Done Sent Done false
25611 7008 Acked Read 10 Done Sent Done false
25612 7009 Acked Read 4 Done Sent Done false
25613 7010 Acked Read 3 Done Sent Done false
25614 7011 Acked Flush 0 Done Sent Done false
25615 7012 Acked Write 2 Done Sent Done false
25616 7013 Acked Read 5 Done Sent Done false
25617 7014 Acked Read 1 Done Sent Done false
25618 7015 Acked Read 5 Done Sent Done false
25619 7016 Acked Write 2 Done Sent Done false
25620 7017 Acked Write 3 Done Sent Done false
25621 7018 Acked Write 9 Done Sent Done false
25622 7019 Acked Write 6 Done Sent Done false
25623 7020 Acked Read 8 Done Sent Done false
25624 7021 Acked Read 5 Done Sent Done false
25625 7022 Acked Write 1 Done Sent Done false
25626 7023 Acked Read 10 Done Sent Done false
25627 7024 Acked Read 5 Done Sent Done false
25628 7025 Acked Read 5 Done Sent Done false
25629 7026 Acked Write 6 Done Sent Done false
25630 7027 Acked Write 7 Done Sent Done false
25631 7028 Acked Read 3 Done Sent Done false
25632 7029 Acked Read 8 Done Sent Done false
25633 7030 Acked Read 10 Done Sent Done false
25634 7031 Acked Read 8 Done Sent Done false
25635 7032 Acked Flush 0 Done Sent Done false
25636 7033 Acked Read 6 Done Sent Done false
25637 7034 Acked Write 8 Done Sent Done false
25638 7035 Acked Read 2 Done Sent Done false
25639 7036 Acked Write 2 Done Sent Done false
25640 7037 Acked Read 7 Done Sent Done false
25641 STATES DS:0 DS:1 DS:2 TOTAL
25642 Sent 0 301 0 301
25643 Done 301 0 301 602
25644 Skipped 0 0 0 0
25645 Error 0 0 0 0
25646 Last Flush: Some(JobId(7032)) Some(JobId(6736)) Some(JobId(7032))
25647 Downstairs last five completed: 6736 6735 6734 6733 6732
25648 Upstairs last five completed: 7037 7036 7035 7034 7033
25649 CLIENT: Up:0 ds:301 act:2
256502025-02-07T20:12:59.502ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
256512025-02-07T20:13:00.517ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
256522025-02-07T20:13:01.506ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
256532025-02-07T20:13:02.267ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
256542025-02-07T20:13:02.267ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43942) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
256552025-02-07T20:13:02.268ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
256562025-02-07T20:13:02.269ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
256572025-02-07T20:13:02.269ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
256582025-02-07T20:13:02.269ZINFOcrucible: send last flush ID to this DS: Some(JobId(6736)) = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
256592025-02-07T20:13:02.269ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
256602025-02-07T20:13:02.269ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(6736)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
256612025-02-07T20:13:02.269ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(6736)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
25662 ----------------------------------------------------------------
25663 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:5
25664 States: Active Active Active
25665 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25666 7033 Acked Read 6 Done Done Done true
25667 7034 Acked Write 8 Done Done Done true
25668 7035 Acked Read 2 Done Done Done true
25669 7036 Acked Write 2 Done Done Done true
25670 7037 Acked Read 7 Done Done Done true
25671 STATES DS:0 DS:1 DS:2 TOTAL
25672 Sent 0 0 0 0
25673 Done 5 5 5 15
25674 Skipped 0 0 0 0
25675 Error 0 0 0 0
25676 Last Flush: Some(JobId(7032)) Some(JobId(7032)) Some(JobId(7032))
25677 Downstairs last five completed: 7032 7031 7030 7029 7028
25678 Upstairs last five completed: 7037 7036 7035 7034 7033
25679 CLIENT: Up:0 ds:5 act:3
25680 ----------------------------------------------------------------
25681 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25682 Downstairs last five completed: 7038 7037 7036 7035 7034
25683 Upstairs last five completed: 7038 7037 7036 7035 7034
25684 CLIENT: Up:0 ds:0 act:3
25685 Replay: All jobs finished
25686 Main task has work to do, go find it
25687 got dsc Stop(2)
25688 stop 2
25689 [8830] Got stop action so:false kr:false
25690 Feb 07 20:13:06.515 INFO request completed, latency_us: 345, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 1ad381db-6c52-4e48-ad9e-ce7828fd8277, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
256912025-02-07T20:13:06.516ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
25692 Feb 07 20:13:06.516 INFO request completed, latency_us: 131, response_code{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:13:06.516347849Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"2","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
25693 : 200, uri: /state/cid/2, {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:13:06.516370847Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
25694 method: GET, req_id: 3dd45b45-dfb7-44bc-80dc-d8213859f1fb, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
256952025-02-07T20:13:06.516ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
25696 [8830] Exited with: ExitStatus(unix_wait_status(9))
25697 [8830][2] reports Exit
25698 Feb 07 20:13:10.518 INFO request completed, latency_us: 282, response_code: 200, uri: /state/cid/2, method: GET, req_id: 880f5188-f8d0-4685-9795-75e0771e87b3, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
25699 001/300 Read block 3667 len 28672
25700 002/300 Write block 2451 len 40960 data: 6 6 6 5 5 4 6 6 4 5
25701 003/300 Write block 2601 len 4096 data: 8
25702 004/300 Write block 3654 len 28672 data: 7 7 6 5 5 7 8
25703 005/300 Read block 4873 len 28672
25704 006/300 Read block 2157 len 4096
25705 007/300 Read block 3876 len 28672
25706 008/300 Write block 4429 len 24576 data: 2 2 3 3 2 2
25707 009/300 Write block 101 len 4096 data: 5
25708 010/300 Read block 1751 len 20480
25709 011/300 Read block 2771 len 16384
25710 012/300 Write block 3949 len 16384 data: 7 7 5 5
25711 013/300 Read block 1353 len 20480
25712 014/300 Read block 808 len 28672
25713 015/300 Read block 4358 len 24576
25714 016/300 Write block 3841 len 20480 data: 5 5 4 6 6
25715 017/300 Write block 3322 len 32768 data: 4 4 5 5 5 5 3 4
25716 018/300 Read block 4807 len 32768
25717 019/300 Read block 4037 len 28672
25718 020/300 Write block 4649 len 36864 data: 6 7 7 5 5 5 5 4 4
25719 021/300 Read block 860 len 20480
25720 022/300 Read block 4190 len 12288
25721 023/300 Read block 2448 len 40960
25722 024/300 Write block 3283 len 32768 data: 4 2 2 2 2 2 2 2
25723 025/300 Read block 4764 len 40960
25724 026/300 Read block 4522 len 20480
25725 027/300 Write block 2881 len 28672 data: 2 3 4 3 3 3 4
25726 028/300 Read block 1242 len 28672
25727 029/300 Read block 4389 len 16384
25728 030/300 Read block 3691 len 32768
25729 031/300 Write block 900 len 28672 data: 4 4 6 6 6 7 6
25730 032/300 Read block 3752 len 16384
25731 033/300 Write block 4120 len 20480 data: 4 4 5 4 6
25732 034/300 Write block 3036 len 16384 data: 2 4 5 5
25733 035/300 Read block 1964 len 16384
25734 036/300 Read block 4830 len 8192
25735 037/300 Write block 661 len 32768 data: 5 4 3 3 3 3 3 3
25736 038/300 Read block 2521 len 16384
25737 039/300 Read block 341 len 24576
25738 040/300 Write block 2395 len 28672 data: 5 6 6 5 5 5 6
25739 041/300 Read block 1858 len 36864
25740 042/300 Write block 4780 len 40960 data: 3 2 2 3 4 4 5 6 7 7
25741 043/300 Read block 4025 len 20480
25742 044/300 Write block 3986 len 4096 data: 4
25743 045/300 Read block 2698 len 24576
25744 046/300 Write block 4351 len 4096 data: 3
25745 047/300 Read block 1865 len 4096
25746 048/300 Flush
25747 049/300 Read block 3496 len 20480
25748 050/300 Write block 4347 len 16384 data: 5 2 3 3
25749 051/300 Write block 1024 len 4096 data: 5
25750 052/300 Write block 2117 len 16384 data: 4 4 3 5
25751 053/300 Flush
25752 054/300 Read block 4763 len 32768
25753 055/300 Flush
25754 056/300 Read block 4725 len 20480
25755 057/300 Read block 3232 len 8192
25756 058/300 Write block 2981 len 28672 data: 7 7 5 3 4 4 4
25757 059/300 Read block 904 len 4096
25758 060/300 Read block 269 len 36864
25759 061/300 Write block 4949 len 12288 data: 6 5 4
25760 062/300 Write block 1590 len 20480 data: 4 4 4 4 3
25761 063/300 Read block 2302 len 24576
25762 064/300 Read block 3371 len 28672
25763 065/300 Write block 4432 len 4096 data: 4
25764 066/300 Read block 407 len 32768
25765 067/300 Read block 1766 len 8192
25766 068/300 Write block 3483 len 28672 data: 7 5 4 5 4 4 5
25767 069/300 Flush
25768 070/300 Write block 1103 len 24576 data: 6 6 6 6 4 5
25769 071/300 Read block 3004 len 12288
25770 072/300 Write block 4630 len 12288 data: 2 2 2
25771 073/300 Write block 269 len 12288 data: 5 5 4
25772 074/300 Read block 4046 len 40960
25773 075/300 Read block 4684 len 8192
25774 076/300 Read block 2938 len 4096
25775 077/300 Write block 386 len 32768 data: 5 5 5 6 6 5 3 3
25776 078/300 Write block 1815 len 32768 data: 6 6 5 4 3 3 4 3
25777 079/300 Read block 558 len 24576
25778 080/300 Read block 2972 len 28672
25779 081/300 Write block 3940 len 20480 data: 3 3 2 3 4
25780 082/300 Flush
25781 083/300 Read block 2141 len 4096
25782 084/300 Read block 1568 len 32768
25783 085/300 Write block 1999 len 40960 data: 11 8 7 8 9 10 9 8 7 8
25784 086/300 Read block 2536 len 12288
25785 087/300 Write block 577 len 16384 data: 6 5 5 4
25786 088/300 Write block 4538 len 12288 data: 5 6 6
25787 089/300 Write block 3085 len 20480 data: 4 4 4 5 4
25788 090/300 Write block 580 len 12288 data: 5 4 5
25789 091/300 Read block 3180 len 20480
25790 092/300 Read block 3896 len 28672
25791 093/300 Read block 4730 len 32768
25792 094/300 Read block 1636 len 32768
25793 095/300 Read block 3686 len 4096
25794 096/300 Read block 901 len 24576
25795 097/300 Write block 4318 len 28672 data: 6 4 4 3 4 4 4
25796 098/300 Flush
25797 099/300 Write block 4938 len 32768 data: 4 5 6 5 6 8 8 8
25798 100/300 Read block 1278 len 16384
25799 101/300 Read block 2942 len 40960
25800 102/300 Read block 4584 len 8192
25801 103/300 Write block 1450 len 4096 data: 3
25802 104/300 Write block 339 len 20480 data: 8 8 3 3 4
25803 105/300 Read block 4375 len 28672
25804 106/300 Read block 3110 len 20480
25805 107/300 Read block 4815 len 8192
25806 108/300 Write block 192 len 20480 data: 5 4 5 3 3
25807 109/300 Write block 3189 len 24576 data: 4 4 4 3 4 3
25808 110/300 Read block 4923 len 16384
25809 111/300 Read block 4744 len 4096
25810 112/300 Write block 2855 len 20480 data: 2 3 3 4 4
25811 113/300 Write block 4320 len 32768 data: 5 4 5 5 5 6 7 5
25812 114/300 Write block 1835 len 28672 data: 10 8 7 7 5 5 4
25813 115/300 Read block 4427 len 36864
25814 116/300 Read block 1710 len 20480
25815 117/300 Read block 4750 len 28672
25816 118/300 Flush
25817 119/300 Write block 4393 len 36864 data: 2 4 4 3 3 3 4 4 3
25818 120/300 Read block 4595 len 20480
25819 121/300 Read block 4509 len 36864
25820 122/300 Flush
25821 123/300 Read block 4436 len 24576
25822 124/300 Write block 1796 len 24576 data: 3 3 3 4 3 3
25823 125/300 Flush
25824 126/300 Read block 4519 len 28672
25825 127/300 Write block 296 len 40960 data: 3 2 3 4 4 4 4 5 5 5
25826 128/300 Write block 2469 len 36864 data: 4 6 5 7 8 5 4 4 3
25827 129/300 Write block 2558 len 8192 data: 3 4
25828 130/300 Read block 2915 len 32768
25829 131/300 Read block 4332 len 4096
25830 132/300 Flush
25831 133/300 Write block 2854 len 24576 data: 2 3 4 4 5 5
25832 134/300 Flush
25833 135/300 Read block 3190 len 8192
25834 136/300 Read block 719 len 12288
25835 137/300 Read block 3809 len 8192
25836 138/300 Read block 1581 len 8192
25837 139/300 Read block 4604 len 40960
25838 140/300 Read block 1968 len 36864
25839 141/300 Flush
25840 142/300 Read block 211 len 8192
25841 143/300 Write block 4130 len 32768 data: 4 5 4 5 4 4 4 4
25842 144/300 Write block 788 len 40960 data: 3 3 3 4 6 6 6 4 5 5
25843 145/300 Write block 4913 len 16384 data: 6 7 6 6
25844 146/300 Read block 1803 len 36864
25845 147/300 Write block 2901 len 28672 data: 5 4 3 4 4 5 5
25846 148/300 Write block 4464 len 16384 data: 6 8 9 9
25847 149/300 Flush
25848 150/300 Flush
25849 151/300 Write block 2329 len 8192 data: 5 4
25850 152/300 Write block 1955 len 36864 data: 5 5 3 2 3 3 2 2 2
25851 153/300 Read block 870 len 36864
25852 154/300 Read block 2486 len 12288
25853 155/300 Read block 1621 len 12288
25854 156/300 Read block 1903 len 28672
25855 157/300 Write block 1063 len 28672 data: 5 3 3 4 5 5 4
25856 158/300 Read block 2472 len 8192
25857 159/300 Read block 3933 len 28672
25858 160/300 Flush
25859 161/300 Flush
25860 162/300 Read block 21 len 16384
25861 163/300 Write block 3675 len 8192 data: 6 5
25862 164/300 Read block 1392 len 8192
25863 165/300 Write block 275 len 24576 data: 5 4 4 5 5 6
25864 166/300 Flush
25865 167/300 Read block 149 len 4096
25866 168/300 Write block 4189 len 40960 data: 7 6 7 8 9 7 5 6 6 6
25867 169/300 Read block 407 len 32768
25868 170/300 Write block 4814 len 4096 data: 5
25869 171/300 Read block 937 len 20480
25870 172/300 Read block 2159 len 28672
25871 173/300 Write block 2849 len 28672 data: 2 2 2 2 2 3 4
25872 174/300 Read block 144 len 40960
25873 175/300 Read block 552 len 28672
25874 176/300 Read block 965 len 32768
25875 177/300 Read block 2391 len 28672
25876 178/300 Read block 4086 len 32768
25877 179/300 Read block 569 len 4096
25878 180/300 Write block 3073 len 24576 data: 6 5 6 5 5 5
25879 181/300 Flush
25880 182/300 Read block 1773 len 12288
25881 183/300 Write block 3877 len 32768 data: 6 3 3 3 3 3 8 5
25882 184/300 Write block 4238 len 36864 data: 6 5 3 3 2 2 4 5 6
25883 185/300 Read block 374 len 12288
25884 186/300 Flush
25885 187/300 Flush
25886 188/300 Flush
25887 189/300 Read block 3829 len 16384
25888 190/300 Read block 2484 len 24576
25889 191/300 Write block 4048 len 16384 data: 4 4 2 3
25890 192/300 Read block 3853 len 36864
25891 193/300 Read block 4563 len 8192
25892 194/300 Write block 1862 len 40960 data: 5 7 7 7 8 7 6 5 3 4
25893 195/300 Read block 1776 len 20480
25894 196/300 Read block 803 len 32768
25895 197/300 Write block 837 len 20480 data: 3 3 5 5 5
25896 198/300 Read block 390 len 16384
25897 199/300 Read block 1739 len 20480
25898 200/300 Read block 3454 len 24576
25899 201/300 Write block 1352 len 4096 data: 6
25900 202/300 Flush
25901 203/300 Read block 4239 len 16384
25902 204/300 Read block 1257 len 4096
25903 205/300 Write block 928 len 28672 data: 4 5 4 3 3 3 3
25904 206/300 Write block 3325 len 28672 data: 6 6 6 4 5 4 5
25905 207/300 Write block 4290 len 28672 data: 4 4 4 3 3 5 5
25906 208/300 Write block 259 len 32768 data: 4 4 4 4 4 3 3 4
25907 209/300 Read block 4859 len 24576
25908 210/300 Write block 1151 len 36864 data: 2 3 3 5 5 5 4 4 4
25909 211/300 Write block 4021 len 16384 data: 2 3 4 6
25910 212/300 Write block 899 len 24576 data: 4 5 5 7 7 7
25911 213/300 Read block 4334 len 20480
25912 214/300 Write block 498 len 20480 data: 7 7 6 6 6
25913 215/300 Write block 419 len 24576 data: 8 8 6 6 6 7
25914 216/300 Write block 718 len 4096 data: 5
25915 217/300 Read block 480 len 24576
25916 218/300 Write block 2014 len 36864 data: 5 5 5 8 6 8 10 9 8
25917 219/300 Read block 4754 len 36864
25918 220/300 Write block 3148 len 36864 data: 6 5 4 5 5 5 5 7 8
25919 221/300 Flush
25920 222/300 Read block 1571 len 28672
25921 223/300 Read block 4403 len 24576
25922 224/300 Write block 4434 len 40960 data: 3 3 4 5 5 6 5 4 3 3
25923 225/300 Read block 742 len 32768
25924 226/300 Read block 4149 len 16384
25925 227/300 Read block 1221 len 24576
25926 228/300 Read block 3459 len 24576
25927 229/300 Read block 4250 len 28672
25928 230/300 Read block 4270 len 8192
25929 231/300 Read block 1169 len 12288
25930 232/300 Flush
25931 233/300 Write block 1838 len 40960 data: 8 6 6 5 2 4 4 6 5 5
25932 234/300 Read block 3996 len 32768
25933 235/300 Read block 2269 len 40960
25934 236/300 Write block 4574 len 16384 data: 3 3 3 3
25935 237/300 Write block 4024 len 36864 data: 7 5 5 5 5 6 6 7 7
25936 238/300 Read block 4993 len 4096
25937 239/300 Flush
25938 240/300 Write block 4761 len 4096 data: 3
25939 241/300 Read block 3247 len 8192
25940 242/300 Read block 2395 len 28672
25941 243/300 Flush
25942 244/300 Write block 4383 len 12288 data: 4 4 4
25943 245/300 Read block 948 len 28672
25944 246/300 Read block 466 len 12288
25945 247/300 Read block 3592 len 32768
25946 248/300 Write block 4234 len 40960 data: 4 3 5 5 7 6 4 4 3 3
25947 249/300 Read block 4751 len 8192
25948 250/300 Write block 287 len 12288 data: 5 4 5
25949 251/300 Flush
25950 252/300 Read block 3600 len 36864
25951 253/300 Read block 3333 len 36864
25952 254/300 Read block 2769 len 12288
25953 255/300 Write block 957 len 28672 data: 2 2 3 4 5 4 4
25954 256/300 Read block 4160 len 4096
25955 257/300 Read block 1518 len 24576
25956 258/300 Write block 4575 len 16384 data: 4 4 4 2
25957 259/300 Read block 2120 len 20480
25958 260/300 Read block 2525 len 40960
25959 261/300 Write block 2476 len 8192 data: 5 4
25960 262/300 Write block 157 len 36864 data: 3 3 4 4 4 5 5 3 4
25961 263/300 Read block 1121 len 24576
25962 264/300 Read block 4928 len 32768
25963 265/300 Write block 304 len 8192 data: 6 6
25964 266/300 Write block 3295 len 8192 data: 2 3
25965 267/300 Read block 3535 len 4096
25966 268/300 Read block 3398 len 20480
25967 269/300 Read block 4733 len 24576
25968 270/300 Write block 2226 len 16384 data: 5 4 4 3
25969 271/300 Read block 1181 len 20480
25970 272/300 Write block 4773 len 36864 data: 5 4 4 4 4 3 3 4 3
25971 273/300 Read block 2403 len 16384
25972 274/300 Read block 2790 len 36864
25973 275/300 Read block 1592 len 40960
25974 276/300 Flush
25975 277/300 Read block 832 len 16384
25976 278/300 Read block 2860 len 36864
25977 279/300 Flush
25978 280/300 Read block 2939 len 24576
25979 281/300 Read block 2734 len 4096
25980 282/300 Write block 4144 len 12288 data: 7 6 6
25981 283/300 Read block 3798 len 40960
25982 284/300 Read block 2875 len 36864
25983 285/300 Read block 3261 len 8192
25984 286/300 Write block 2234 len 8192 data: 3 4
25985 287/300 Write block 2151 len 12288 data: 3 4 3
25986 288/300 Write block 4705 len 36864 data: 6 6 6 6 6 5 4 3 3
25987 289/300 Write block 2958 len 28672 data: 4 5 6 6 5 4 4
25988 290/300 Read block 3427 len 8192
25989 291/300 Write block 3849 len 24576 data: 5 4 4 3 4 5
25990 292/300 Read block 4348 len 24576
25991 293/300 Read block 2295 len 16384
25992 294/300 Write block 309 len 28672 data: 4 5 7 7 6 6 6
25993 295/300 Write block 4421 len 12288 data: 4 5 5
25994 296/300 Flush
25995 297/300 Read block 1100 len 28672
25996 298/300 Write block 2470 len 32768 data: 7 6 8 9 6 5 6 5
25997 299/300 Read block 345 len 4096
25998 300/300 Read block 2700 len 24576
25999 301/300 Read block 4123 len 40960
26000 Main task has work to do, go find it
26001 got dsc Start(2)
26002 start 2
26003 Feb 07 20:13:10.773 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/2, method: POST, req_id: 533b45af-07d7-42f6-9fe5-72dcb727ce41, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
26004 Replay: started 2, returned:Ok(())
26005 ----------------------------------------------------------------
26006 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26007 States: Active Active Offline
26008 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26009 7039 Acked Read 7 Done Done Sent false
26010 7040 Acked Write 10 Done Done Sent false
26011 7041 Acked Write 1 Done Done Sent false
26012 7042 Acked Write 7 Done Done Sent false
26013 7043 Acked Read 7 Done Done Sent false
26014 7044 Acked Read 1 Done Done Sent false
26015 7045 Acked Read 7 Done Done Sent false
26016 7046 Acked Write 6 Done Done Sent false
26017 7047 Acked Write 1 Done Done Sent false
26018 7048 Acked Read 5 Done Done Sent false
26019 7049 Acked Read 4 Done Done Sent false
26020 7050 Acked Write 4 Done Done Sent false
26021 7051 Acked Read 5 Done Done Sent false
26022 7052 Acked Read 7 Done Done Sent false
26023 7053 Acked Read 6 Done Done Sent false
26024 7054 Acked Write 5 Done Done Sent false
26025 7055 Acked Write 8 Done Done Sent false
26026 7056 Acked Read 8 Done Done Sent false
26027 7057 Acked Read 7 Done Done Sent false
26028 7058 Acked Write 9 Done Done Sent false
26029 7059 Acked Read 5 Done Done Sent false
26030 7060 Acked Read 3 Done Done Sent false
26031 7061 Acked Read 10 Done Done Sent false
26032 7062 Acked Write 8 Done Done Sent false
26033 7063 Acked Read 10 Done Done Sent false
26034 7064 Acked Read 5 Done Done Sent false
26035 7065 Acked Write 7 Done Done Sent false
26036 7066 Acked Read 7 Done Done Sent false
26037 7067 Acked Read 4 Done Done Sent false
26038 7068 Acked Read 8 Done Done Sent false
26039 7069 Acked Write 7 Done Done Sent false
26040 7070 Acked Read 4 Done Done Sent false
26041 7071 Acked Write 5 Done Done Sent false
26042 7072 Acked Write 4 Done Done Sent false
26043 7073 Acked Read 4 Done Done Sent false
26044 7074 Acked Read 2 Done Done Sent false
26045 7075 Acked Write 8 Done Done Sent false
26046 7076 Acked Read 4 Done Done Sent false
26047 7077 Acked Read 6 Done Done Sent false
26048 7078 Acked Write 7 Done Done Sent false
26049 7079 Acked Read 9 Done Done Sent false
26050 7080 Acked Write 10 Done Done Sent false
26051 7081 Acked Read 5 Done Done Sent false
26052 7082 Acked Write 1 Done Done Sent false
26053 7083 Acked Read 6 Done Done Sent false
26054 7084 Acked Write 1 Done Done Sent false
26055 7085 Acked Read 1 Done Done Sent false
26056 7086 Acked Flush 0 Done Done Sent false
26057 7087 Acked Read 5 Done Done Sent false
26058 7088 Acked Write 4 Done Done Sent false
26059 7089 Acked Write 1 Done Done Sent false
26060 7090 Acked Write 4 Done Done Sent false
26061 7091 Acked Flush 0 Done Done Sent false
26062 7092 Acked Read 8 Done Done Sent false
26063 7093 Acked Flush 0 Done Done Sent false
26064 7094 Acked Read 5 Done Done Sent false
26065 7095 Acked Read 2 Done Done Sent false
26066 7096 Acked Write 7 Done Done Sent false
26067 7097 Acked Read 1 Done Done Sent false
26068 7098 Acked Read 9 Done Done Sent false
26069 7099 Acked Write 3 Done Done Sent false
26070 7100 Acked Write 5 Done Done Sent false
26071 7101 Acked Read 6 Done Done Sent false
26072 7102 Acked Read 7 Done Done Sent false
26073 7103 Acked Write 1 Done Done Sent false
26074 7104 Acked Read 8 Done Done Sent false
26075 7105 Acked Read 2 Done Done Sent false
26076 7106 Acked Write 7 Done Done Sent false
26077 7107 Acked Flush 0 Done Done Sent false
26078 7108 Acked Write 6 Done Done Sent false
26079 7109 Acked Read 3 Done Done Sent false
26080 7110 Acked Write 3 Done Done Sent false
26081 7111 Acked Write 3 Done Done Sent false
26082 7112 Acked Read 10 Done Done Sent false
26083 7113 Acked Read 2 Done Done Sent false
26084 7114 Acked Read 1 Done Done Sent false
26085 7115 Acked Write 8 Done Done Sent false
26086 7116 Acked Write 8 Done Done Sent false
26087 7117 Acked Read 6 Done Done Sent false
26088 7118 Acked Read 7 Done Done Sent false
26089 7119 Acked Write 5 Done Done Sent false
26090 7120 Acked Flush 0 Done Done Sent false
26091 7121 Acked Read 1 Done Done Sent false
26092 7122 Acked Read 8 Done Done Sent false
26093 7123 Acked Write 10 Done Done Sent false
26094 7124 Acked Read 3 Done Done Sent false
26095 7125 Acked Write 4 Done Done Sent false
26096 7126 Acked Write 3 Done Done Sent false
26097 7127 Acked Write 5 Done Done Sent false
26098 7128 Acked Write 3 Done Done Sent false
26099 7129 Acked Read 5 Done Done Sent false
26100 7130 Acked Read 7 Done Done Sent false
26101 7131 Acked Read 8 Done Done Sent false
26102 7132 Acked Read 8 Done Done Sent false
26103 7133 Acked Read 1 Done Done Sent false
26104 7134 Acked Read 6 Done Done Sent false
26105 7135 Acked Write 7 Done Done Sent false
26106 7136 Acked Flush 0 Done Done Sent false
26107 7137 Acked Write 8 Done Done Sent false
26108 7138 Acked Read 4 Done Done Sent false
26109 7139 Acked Read 10 Done Done Sent false
26110 7140 Acked Read 2 Done Done Sent false
26111 7141 Acked Write 1 Done Done Sent false
26112 7142 Acked Write 5 Done Done Sent false
26113 7143 Acked Read 7 Done Done Sent false
26114 7144 Acked Read 5 Done Done Sent false
26115 7145 Acked Read 2 Done Done Sent false
26116 7146 Acked Write 5 Done Done Sent false
26117 7147 Acked Write 6 Done Done Sent false
26118 7148 Acked Read 4 Done Done Sent false
26119 7149 Acked Read 1 Done Done Sent false
26120 7150 Acked Write 5 Done Done Sent false
26121 7151 Acked Write 8 Done Done Sent false
26122 7152 Acked Write 7 Done Done Sent false
26123 7153 Acked Read 9 Done Done Sent false
26124 7154 Acked Read 5 Done Done Sent false
26125 7155 Acked Read 7 Done Done Sent false
26126 7156 Acked Flush 0 Done Done Sent false
26127 7157 Acked Write 9 Done Done Sent false
26128 7158 Acked Read 5 Done Done Sent false
26129 7159 Acked Read 9 Done Done Sent false
26130 7160 Acked Flush 0 Done Done Sent false
26131 7161 Acked Read 6 Done Done Sent false
26132 7162 Acked Write 6 Done Done Sent false
26133 7163 Acked Flush 0 Done Done Sent false
26134 7164 Acked Read 7 Done Done Sent false
26135 7165 Acked Write 10 Done Done Sent false
26136 7166 Acked Write 9 Done Done Sent false
26137 7167 Acked Write 2 Done Done Sent false
26138 7168 Acked Read 8 Done Done Sent false
26139 7169 Acked Read 1 Done Done Sent false
26140 7170 Acked Flush 0 Done Done Sent false
26141 7171 Acked Write 6 Done Done Sent false
26142 7172 Acked Flush 0 Done Done Sent false
26143 7173 Acked Read 2 Done Done Sent false
26144 7174 Acked Read 3 Done Done Sent false
26145 7175 Acked Read 2 Done Done Sent false
26146 7176 Acked Read 2 Done Done Sent false
26147 7177 Acked Read 10 Done Done Sent false
26148 7178 Acked Read 9 Done Done Sent false
26149 7179 Acked Flush 0 Done Done Sent false
26150 7180 Acked Read 2 Done Done Sent false
26151 7181 Acked Write 8 Done Done Sent false
26152 7182 Acked Write 10 Done Done Sent false
26153 7183 Acked Write 4 Done Done Sent false
26154 7184 Acked Read 9 Done Done Sent false
26155 7185 Acked Write 7 Done Done Sent false
26156 7186 Acked Write 4 Done Done Sent false
26157 7187 Acked Flush 0 Done Done Sent false
26158 7188 Acked Flush 0 Done Done Sent false
26159 7189 Acked Write 2 Done Done Sent false
26160 7190 Acked Write 9 Done Done Sent false
26161 7191 Acked Read 9 Done Done Sent false
26162 7192 Acked Read 3 Done Done Sent false
26163 7193 Acked Read 3 Done Done Sent false
26164 7194 Acked Read 7 Done Done Sent false
26165 7195 Acked Write 7 Done Done Sent false
26166 7196 Acked Read 2 Done Done Sent false
26167 7197 Acked Read 7 Done Done Sent false
26168 7198 Acked Flush 0 Done Done Sent false
26169 7199 Acked Flush 0 Done Done Sent false
26170 7200 Acked Read 4 Done Done Sent false
26171 7201 Acked Write 2 Done Done Sent false
26172 7202 Acked Read 2 Done Done Sent false
26173 7203 Acked Write 6 Done Done Sent false
26174 7204 Acked Flush 0 Done Done Sent false
26175 7205 Acked Read 1 Done Done Sent false
26176 7206 Acked Write 10 Done Done Sent false
26177 7207 Acked Read 8 Done Done Sent false
26178 7208 Acked Write 1 Done Done Sent false
26179 7209 Acked Read 5 Done Done Sent false
26180 7210 Acked Read 7 Done Done Sent false
26181 7211 Acked Write 7 Done Done Sent false
26182 7212 Acked Read 10 Done Done Sent false
26183 7213 Acked Read 7 Done Done Sent false
26184 7214 Acked Read 8 Done Done Sent false
26185 7215 Acked Read 7 Done Done Sent false
26186 7216 Acked Read 8 Done Done Sent false
26187 7217 Acked Read 1 Done Done Sent false
26188 7218 Acked Write 6 Done Done Sent false
26189 7219 Acked Flush 0 Done Done Sent false
26190 7220 Acked Read 3 Done Done Sent false
26191 7221 Acked Write 8 Done Done Sent false
26192 7222 Acked Write 9 Done Done Sent false
26193 7223 Acked Read 3 Done Done Sent false
26194 7224 Acked Flush 0 Done Done Sent false
26195 7225 Acked Flush 0 Done Done Sent false
26196 7226 Acked Flush 0 Done Done Sent false
26197 7227 Acked Read 4 Done Done Sent false
26198 7228 Acked Read 6 Done Done Sent false
26199 7229 Acked Write 4 Done Done Sent false
26200 7230 Acked Read 9 Done Done Sent false
26201 7231 Acked Read 2 Done Done Sent false
26202 7232 Acked Write 10 Done Done Sent false
26203 7233 Acked Read 5 Done Done Sent false
26204 7234 Acked Read 8 Done Done Sent false
26205 7235 Acked Write 5 Done Done Sent false
26206 7236 Acked Read 4 Done Done Sent false
26207 7237 Acked Read 5 Done Done Sent false
26208 7238 Acked Read 6 Done Done Sent false
26209 7239 Acked Write 1 Done Done Sent false
26210 7240 Acked Flush 0 Done Done Sent false
26211 7241 Acked Read 4 Done Done Sent false
26212 7242 Acked Read 1 Done Done Sent false
26213 7243 Acked Write 7 Done Done Sent false
26214 7244 Acked Write 7 Done Done Sent false
26215 7245 Acked Write 7 Done Done Sent false
26216 7246 Acked Write 8 Done Done Sent false
26217 7247 Acked Read 6 Done Done Sent false
26218 7248 Acked Write 9 Done Done Sent false
26219 7249 Acked Write 4 Done Done Sent false
26220 7250 Acked Write 6 Done Done Sent false
26221 7251 Acked Read 5 Done Done Sent false
26222 7252 Acked Write 5 Done Done Sent false
26223 7253 Acked Write 6 Done Done Sent false
26224 7254 Acked Write 1 Done Done Sent false
26225 7255 Acked Read 6 Done Done Sent false
26226 7256 Acked Write 9 Done Done Sent false
26227 7257 Acked Read 9 Done Done Sent false
26228 7258 Acked Write 9 Done Done Sent false
26229 7259 Acked Flush 0 Done Done Sent false
26230 7260 Acked Read 7 Done Done Sent false
26231 7261 Acked Read 6 Done Done Sent false
26232 7262 Acked Write 10 Done Done Sent false
26233 7263 Acked Read 8 Done Done Sent false
26234 7264 Acked Read 4 Done Done Sent false
26235 7265 Acked Read 6 Done Done Sent false
26236 7266 Acked Read 6 Done Done Sent false
26237 7267 Acked Read 7 Done Done Sent false
26238 7268 Acked Read 2 Done Done Sent false
26239 7269 Acked Read 3 Done Done Sent false
26240 7270 Acked Flush 0 Done Done Sent false
26241 7271 Acked Write 10 Done Done Sent false
26242 7272 Acked Read 8 Done Done Sent false
26243 7273 Acked Read 10 Done Done Sent false
26244 7274 Acked Write 4 Done Done Sent false
26245 7275 Acked Write 9 Done Done Sent false
26246 7276 Acked Read 1 Done Done Sent false
26247 7277 Acked Flush 0 Done Done Sent false
26248 7278 Acked Write 1 Done Done Sent false
26249 7279 Acked Read 2 Done Done Sent false
26250 7280 Acked Read 7 Done Done Sent false
26251 7281 Acked Flush 0 Done Done Sent false
26252 7282 Acked Write 3 Done Done Sent false
26253 7283 Acked Read 7 Done Done Sent false
26254 7284 Acked Read 3 Done Done Sent false
26255 7285 Acked Read 8 Done Done Sent false
26256 7286 Acked Write 10 Done Done Sent false
26257 7287 Acked Read 2 Done Done Sent false
26258 7288 Acked Write 3 Done Done Sent false
26259 7289 Acked Flush 0 Done Done Sent false
26260 7290 Acked Read 9 Done Done Sent false
26261 7291 Acked Read 9 Done Done Sent false
26262 7292 Acked Read 3 Done Done Sent false
26263 7293 Acked Write 7 Done Done Sent false
26264 7294 Acked Read 1 Done Done Sent false
26265 7295 Acked Read 6 Done Done Sent false
26266 7296 Acked Write 4 Done Done Sent false
26267 7297 Acked Read 5 Done Done Sent false
26268 7298 Acked Read 10 Done Done Sent false
26269 7299 Acked Write 2 Done Done Sent false
26270 7300 Acked Write 9 Done Done Sent false
26271 7301 Acked Read 6 Done Done Sent false
26272 7302 Acked Read 8 Done Done Sent false
26273 7303 Acked Write 2 Done Done Sent false
26274 7304 Acked Write 2 Done Done Sent false
26275 7305 Acked Read 1 Done Done Sent false
26276 7306 Acked Read 5 Done Done Sent false
26277 7307 Acked Read 6 Done Done Sent false
26278 7308 Acked Write 4 Done Done Sent false
26279 7309 Acked Read 5 Done Done Sent false
26280 7310 Acked Write 9 Done Done Sent false
26281 7311 Acked Read 4 Done Done Sent false
26282 7312 Acked Read 9 Done Done Sent false
26283 7313 Acked Read 10 Done Done Sent false
26284 7314 Acked Flush 0 Done Done Sent false
26285 7315 Acked Read 4 Done Done Sent false
26286 7316 Acked Read 9 Done Done Sent false
26287 7317 Acked Flush 0 Done Done Sent false
26288 7318 Acked Read 6 Done Done Sent false
26289 7319 Acked Read 1 Done Done Sent false
26290 7320 Acked Write 3 Done Done Sent false
26291 7321 Acked Read 10 Done Done Sent false
26292 7322 Acked Read 9 Done Done Sent false
26293 7323 Acked Read 2 Done Done Sent false
26294 7324 Acked Write 2 Done Done Sent false
26295 7325 Acked Write 3 Done Done Sent false
26296 7326 Acked Write 9 Done Done Sent false
26297 7327 Acked Write 7 Done Done Sent false
26298 7328 Acked Read 2 Done Done Sent false
26299 7329 Acked Write 6 Done Done Sent false
26300 7330 Acked Read 6 Done Done Sent false
26301 7331 Acked Read 4 Done Done Sent false
26302 7332 Acked Write 7 Done Done Sent false
26303 7333 Acked Write 3 Done Done Sent false
26304 7334 Acked Flush 0 Done Done Sent false
26305 7335 Acked Read 7 Done Done Sent false
26306 7336 Acked Write 8 Done Done Sent false
26307 7337 Acked Read 1 Done Done Sent false
26308 7338 Acked Read 6 Done Done Sent false
26309 7339 Acked Read 10 Done Done Sent false
26310 STATES DS:0 DS:1 DS:2 TOTAL
26311 Sent 0 0 301 301
26312 Done 301 301 0 602
26313 Skipped 0 0 0 0
26314 Error 0 0 0 0
26315 Last Flush: Some(JobId(7334)) Some(JobId(7334)) Some(JobId(7038))
26316 Downstairs last five completed: 7038 7037 7036 7035 7034
26317 Upstairs last five completed: 7339 7338 7337 7336 7335
26318 CLIENT: Up:0 ds:301 act:2
263192025-02-07T20:13:11.520ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
26320 [8830] got start action
26321 [8830] I am going to restart
26322 Starting downstairs at port 8830
26323 Make output file at "/tmp/dsc/downstairs-8830.txt"
26324 [8830][2] reports Starting
26325 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1230)
26326 [8830][2] reports Running
263272025-02-07T20:13:12.770ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
263282025-02-07T20:13:13.522ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
263292025-02-07T20:13:14.523ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
26330 ----------------------------------------------------------------
26331 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26332 States: Active Active Offline
26333 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26334 7039 Acked Read 7 Done Done Sent false
26335 7040 Acked Write 10 Done Done Sent false
26336 7041 Acked Write 1 Done Done Sent false
26337 7042 Acked Write 7 Done Done Sent false
26338 7043 Acked Read 7 Done Done Sent false
26339 7044 Acked Read 1 Done Done Sent false
26340 7045 Acked Read 7 Done Done Sent false
26341 7046 Acked Write 6 Done Done Sent false
26342 7047 Acked Write 1 Done Done Sent false
26343 7048 Acked Read 5 Done Done Sent false
26344 7049 Acked Read 4 Done Done Sent false
26345 7050 Acked Write 4 Done Done Sent false
26346 7051 Acked Read 5 Done Done Sent false
26347 7052 Acked Read 7 Done Done Sent false
26348 7053 Acked Read 6 Done Done Sent false
26349 7054 Acked Write 5 Done Done Sent false
26350 7055 Acked Write 8 Done Done Sent false
26351 7056 Acked Read 8 Done Done Sent false
26352 7057 Acked Read 7 Done Done Sent false
26353 7058 Acked Write 9 Done Done Sent false
26354 7059 Acked Read 5 Done Done Sent false
26355 7060 Acked Read 3 Done Done Sent false
26356 7061 Acked Read 10 Done Done Sent false
26357 7062 Acked Write 8 Done Done Sent false
26358 7063 Acked Read 10 Done Done Sent false
26359 7064 Acked Read 5 Done Done Sent false
26360 7065 Acked Write 7 Done Done Sent false
26361 7066 Acked Read 7 Done Done Sent false
26362 7067 Acked Read 4 Done Done Sent false
26363 7068 Acked Read 8 Done Done Sent false
26364 7069 Acked Write 7 Done Done Sent false
26365 7070 Acked Read 4 Done Done Sent false
26366 7071 Acked Write 5 Done Done Sent false
26367 7072 Acked Write 4 Done Done Sent false
26368 7073 Acked Read 4 Done Done Sent false
26369 7074 Acked Read 2 Done Done Sent false
26370 7075 Acked Write 8 Done Done Sent false
26371 7076 Acked Read 4 Done Done Sent false
26372 7077 Acked Read 6 Done Done Sent false
26373 7078 Acked Write 7 Done Done Sent false
26374 7079 Acked Read 9 Done Done Sent false
26375 7080 Acked Write 10 Done Done Sent false
26376 7081 Acked Read 5 Done Done Sent false
26377 7082 Acked Write 1 Done Done Sent false
26378 7083 Acked Read 6 Done Done Sent false
26379 7084 Acked Write 1 Done Done Sent false
26380 7085 Acked Read 1 Done Done Sent false
26381 7086 Acked Flush 0 Done Done Sent false
26382 7087 Acked Read 5 Done Done Sent false
26383 7088 Acked Write 4 Done Done Sent false
26384 7089 Acked Write 1 Done Done Sent false
26385 7090 Acked Write 4 Done Done Sent false
26386 7091 Acked Flush 0 Done Done Sent false
26387 7092 Acked Read 8 Done Done Sent false
26388 7093 Acked Flush 0 Done Done Sent false
26389 7094 Acked Read 5 Done Done Sent false
26390 7095 Acked Read 2 Done Done Sent false
26391 7096 Acked Write 7 Done Done Sent false
26392 7097 Acked Read 1 Done Done Sent false
26393 7098 Acked Read 9 Done Done Sent false
26394 7099 Acked Write 3 Done Done Sent false
26395 7100 Acked Write 5 Done Done Sent false
26396 7101 Acked Read 6 Done Done Sent false
26397 7102 Acked Read 7 Done Done Sent false
26398 7103 Acked Write 1 Done Done Sent false
26399 7104 Acked Read 8 Done Done Sent false
26400 7105 Acked Read 2 Done Done Sent false
26401 7106 Acked Write 7 Done Done Sent false
26402 7107 Acked Flush 0 Done Done Sent false
26403 7108 Acked Write 6 Done Done Sent false
26404 7109 Acked Read 3 Done Done Sent false
26405 7110 Acked Write 3 Done Done Sent false
26406 7111 Acked Write 3 Done Done Sent false
26407 7112 Acked Read 10 Done Done Sent false
26408 7113 Acked Read 2 Done Done Sent false
26409 7114 Acked Read 1 Done Done Sent false
26410 7115 Acked Write 8 Done Done Sent false
26411 7116 Acked Write 8 Done Done Sent false
26412 7117 Acked Read 6 Done Done Sent false
26413 7118 Acked Read 7 Done Done Sent false
26414 7119 Acked Write 5 Done Done Sent false
26415 7120 Acked Flush 0 Done Done Sent false
26416 7121 Acked Read 1 Done Done Sent false
26417 7122 Acked Read 8 Done Done Sent false
26418 7123 Acked Write 10 Done Done Sent false
26419 7124 Acked Read 3 Done Done Sent false
26420 7125 Acked Write 4 Done Done Sent false
26421 7126 Acked Write 3 Done Done Sent false
26422 7127 Acked Write 5 Done Done Sent false
26423 7128 Acked Write 3 Done Done Sent false
26424 7129 Acked Read 5 Done Done Sent false
26425 7130 Acked Read 7 Done Done Sent false
26426 7131 Acked Read 8 Done Done Sent false
26427 7132 Acked Read 8 Done Done Sent false
26428 7133 Acked Read 1 Done Done Sent false
26429 7134 Acked Read 6 Done Done Sent false
26430 7135 Acked Write 7 Done Done Sent false
26431 7136 Acked Flush 0 Done Done Sent false
26432 7137 Acked Write 8 Done Done Sent false
26433 7138 Acked Read 4 Done Done Sent false
26434 7139 Acked Read 10 Done Done Sent false
26435 7140 Acked Read 2 Done Done Sent false
26436 7141 Acked Write 1 Done Done Sent false
26437 7142 Acked Write 5 Done Done Sent false
26438 7143 Acked Read 7 Done Done Sent false
26439 7144 Acked Read 5 Done Done Sent false
26440 7145 Acked Read 2 Done Done Sent false
26441 7146 Acked Write 5 Done Done Sent false
26442 7147 Acked Write 6 Done Done Sent false
26443 7148 Acked Read 4 Done Done Sent false
26444 7149 Acked Read 1 Done Done Sent false
26445 7150 Acked Write 5 Done Done Sent false
26446 7151 Acked Write 8 Done Done Sent false
26447 7152 Acked Write 7 Done Done Sent false
26448 7153 Acked Read 9 Done Done Sent false
26449 7154 Acked Read 5 Done Done Sent false
26450 7155 Acked Read 7 Done Done Sent false
26451 7156 Acked Flush 0 Done Done Sent false
26452 7157 Acked Write 9 Done Done Sent false
26453 7158 Acked Read 5 Done Done Sent false
26454 7159 Acked Read 9 Done Done Sent false
26455 7160 Acked Flush 0 Done Done Sent false
26456 7161 Acked Read 6 Done Done Sent false
26457 7162 Acked Write 6 Done Done Sent false
26458 7163 Acked Flush 0 Done Done Sent false
26459 7164 Acked Read 7 Done Done Sent false
26460 7165 Acked Write 10 Done Done Sent false
26461 7166 Acked Write 9 Done Done Sent false
26462 7167 Acked Write 2 Done Done Sent false
26463 7168 Acked Read 8 Done Done Sent false
26464 7169 Acked Read 1 Done Done Sent false
26465 7170 Acked Flush 0 Done Done Sent false
26466 7171 Acked Write 6 Done Done Sent false
26467 7172 Acked Flush 0 Done Done Sent false
26468 7173 Acked Read 2 Done Done Sent false
26469 7174 Acked Read 3 Done Done Sent false
26470 7175 Acked Read 2 Done Done Sent false
26471 7176 Acked Read 2 Done Done Sent false
26472 7177 Acked Read 10 Done Done Sent false
26473 7178 Acked Read 9 Done Done Sent false
26474 7179 Acked Flush 0 Done Done Sent false
26475 7180 Acked Read 2 Done Done Sent false
26476 7181 Acked Write 8 Done Done Sent false
26477 7182 Acked Write 10 Done Done Sent false
26478 7183 Acked Write 4 Done Done Sent false
26479 7184 Acked Read 9 Done Done Sent false
26480 7185 Acked Write 7 Done Done Sent false
26481 7186 Acked Write 4 Done Done Sent false
26482 7187 Acked Flush 0 Done Done Sent false
26483 7188 Acked Flush 0 Done Done Sent false
26484 7189 Acked Write 2 Done Done Sent false
26485 7190 Acked Write 9 Done Done Sent false
26486 7191 Acked Read 9 Done Done Sent false
26487 7192 Acked Read 3 Done Done Sent false
26488 7193 Acked Read 3 Done Done Sent false
26489 7194 Acked Read 7 Done Done Sent false
26490 7195 Acked Write 7 Done Done Sent false
26491 7196 Acked Read 2 Done Done Sent false
26492 7197 Acked Read 7 Done Done Sent false
26493 7198 Acked Flush 0 Done Done Sent false
26494 7199 Acked Flush 0 Done Done Sent false
26495 7200 Acked Read 4 Done Done Sent false
26496 7201 Acked Write 2 Done Done Sent false
26497 7202 Acked Read 2 Done Done Sent false
26498 7203 Acked Write 6 Done Done Sent false
26499 7204 Acked Flush 0 Done Done Sent false
26500 7205 Acked Read 1 Done Done Sent false
26501 7206 Acked Write 10 Done Done Sent false
26502 7207 Acked Read 8 Done Done Sent false
26503 7208 Acked Write 1 Done Done Sent false
26504 7209 Acked Read 5 Done Done Sent false
26505 7210 Acked Read 7 Done Done Sent false
26506 7211 Acked Write 7 Done Done Sent false
26507 7212 Acked Read 10 Done Done Sent false
26508 7213 Acked Read 7 Done Done Sent false
26509 7214 Acked Read 8 Done Done Sent false
26510 7215 Acked Read 7 Done Done Sent false
26511 7216 Acked Read 8 Done Done Sent false
26512 7217 Acked Read 1 Done Done Sent false
26513 7218 Acked Write 6 Done Done Sent false
26514 7219 Acked Flush 0 Done Done Sent false
26515 7220 Acked Read 3 Done Done Sent false
26516 7221 Acked Write 8 Done Done Sent false
26517 7222 Acked Write 9 Done Done Sent false
26518 7223 Acked Read 3 Done Done Sent false
26519 7224 Acked Flush 0 Done Done Sent false
26520 7225 Acked Flush 0 Done Done Sent false
26521 7226 Acked Flush 0 Done Done Sent false
26522 7227 Acked Read 4 Done Done Sent false
26523 7228 Acked Read 6 Done Done Sent false
26524 7229 Acked Write 4 Done Done Sent false
26525 7230 Acked Read 9 Done Done Sent false
26526 7231 Acked Read 2 Done Done Sent false
26527 7232 Acked Write 10 Done Done Sent false
26528 7233 Acked Read 5 Done Done Sent false
26529 7234 Acked Read 8 Done Done Sent false
26530 7235 Acked Write 5 Done Done Sent false
26531 7236 Acked Read 4 Done Done Sent false
26532 7237 Acked Read 5 Done Done Sent false
26533 7238 Acked Read 6 Done Done Sent false
26534 7239 Acked Write 1 Done Done Sent false
26535 7240 Acked Flush 0 Done Done Sent false
26536 7241 Acked Read 4 Done Done Sent false
26537 7242 Acked Read 1 Done Done Sent false
26538 7243 Acked Write 7 Done Done Sent false
26539 7244 Acked Write 7 Done Done Sent false
26540 7245 Acked Write 7 Done Done Sent false
26541 7246 Acked Write 8 Done Done Sent false
26542 7247 Acked Read 6 Done Done Sent false
26543 7248 Acked Write 9 Done Done Sent false
26544 7249 Acked Write 4 Done Done Sent false
26545 7250 Acked Write 6 Done Done Sent false
26546 7251 Acked Read 5 Done Done Sent false
26547 7252 Acked Write 5 Done Done Sent false
26548 7253 Acked Write 6 Done Done Sent false
26549 7254 Acked Write 1 Done Done Sent false
26550 7255 Acked Read 6 Done Done Sent false
26551 7256 Acked Write 9 Done Done Sent false
26552 7257 Acked Read 9 Done Done Sent false
26553 7258 Acked Write 9 Done Done Sent false
26554 7259 Acked Flush 0 Done Done Sent false
26555 7260 Acked Read 7 Done Done Sent false
26556 7261 Acked Read 6 Done Done Sent false
26557 7262 Acked Write 10 Done Done Sent false
26558 7263 Acked Read 8 Done Done Sent false
26559 7264 Acked Read 4 Done Done Sent false
26560 7265 Acked Read 6 Done Done Sent false
26561 7266 Acked Read 6 Done Done Sent false
26562 7267 Acked Read 7 Done Done Sent false
26563 7268 Acked Read 2 Done Done Sent false
26564 7269 Acked Read 3 Done Done Sent false
26565 7270 Acked Flush 0 Done Done Sent false
26566 7271 Acked Write 10 Done Done Sent false
26567 7272 Acked Read 8 Done Done Sent false
26568 7273 Acked Read 10 Done Done Sent false
26569 7274 Acked Write 4 Done Done Sent false
26570 7275 Acked Write 9 Done Done Sent false
26571 7276 Acked Read 1 Done Done Sent false
26572 7277 Acked Flush 0 Done Done Sent false
26573 7278 Acked Write 1 Done Done Sent false
26574 7279 Acked Read 2 Done Done Sent false
26575 7280 Acked Read 7 Done Done Sent false
26576 7281 Acked Flush 0 Done Done Sent false
26577 7282 Acked Write 3 Done Done Sent false
26578 7283 Acked Read 7 Done Done Sent false
26579 7284 Acked Read 3 Done Done Sent false
26580 7285 Acked Read 8 Done Done Sent false
26581 7286 Acked Write 10 Done Done Sent false
26582 7287 Acked Read 2 Done Done Sent false
26583 7288 Acked Write 3 Done Done Sent false
26584 7289 Acked Flush 0 Done Done Sent false
26585 7290 Acked Read 9 Done Done Sent false
26586 7291 Acked Read 9 Done Done Sent false
26587 7292 Acked Read 3 Done Done Sent false
26588 7293 Acked Write 7 Done Done Sent false
26589 7294 Acked Read 1 Done Done Sent false
26590 7295 Acked Read 6 Done Done Sent false
26591 7296 Acked Write 4 Done Done Sent false
26592 7297 Acked Read 5 Done Done Sent false
26593 7298 Acked Read 10 Done Done Sent false
26594 7299 Acked Write 2 Done Done Sent false
26595 7300 Acked Write 9 Done Done Sent false
26596 7301 Acked Read 6 Done Done Sent false
26597 7302 Acked Read 8 Done Done Sent false
26598 7303 Acked Write 2 Done Done Sent false
26599 7304 Acked Write 2 Done Done Sent false
26600 7305 Acked Read 1 Done Done Sent false
26601 7306 Acked Read 5 Done Done Sent false
26602 7307 Acked Read 6 Done Done Sent false
26603 7308 Acked Write 4 Done Done Sent false
26604 7309 Acked Read 5 Done Done Sent false
26605 7310 Acked Write 9 Done Done Sent false
26606 7311 Acked Read 4 Done Done Sent false
26607 7312 Acked Read 9 Done Done Sent false
26608 7313 Acked Read 10 Done Done Sent false
26609 7314 Acked Flush 0 Done Done Sent false
26610 7315 Acked Read 4 Done Done Sent false
26611 7316 Acked Read 9 Done Done Sent false
26612 7317 Acked Flush 0 Done Done Sent false
26613 7318 Acked Read 6 Done Done Sent false
26614 7319 Acked Read 1 Done Done Sent false
26615 7320 Acked Write 3 Done Done Sent false
26616 7321 Acked Read 10 Done Done Sent false
26617 7322 Acked Read 9 Done Done Sent false
26618 7323 Acked Read 2 Done Done Sent false
26619 7324 Acked Write 2 Done Done Sent false
26620 7325 Acked Write 3 Done Done Sent false
26621 7326 Acked Write 9 Done Done Sent false
26622 7327 Acked Write 7 Done Done Sent false
26623 7328 Acked Read 2 Done Done Sent false
26624 7329 Acked Write 6 Done Done Sent false
26625 7330 Acked Read 6 Done Done Sent false
26626 7331 Acked Read 4 Done Done Sent false
26627 7332 Acked Write 7 Done Done Sent false
26628 7333 Acked Write 3 Done Done Sent false
26629 7334 Acked Flush 0 Done Done Sent false
26630 7335 Acked Read 7 Done Done Sent false
26631 7336 Acked Write 8 Done Done Sent false
26632 7337 Acked Read 1 Done Done Sent false
26633 7338 Acked Read 6 Done Done Sent false
26634 7339 Acked Read 10 Done Done Sent false
26635 STATES DS:0 DS:1 DS:2 TOTAL
26636 Sent 0 0 301 301
26637 Done 301 301 0 602
26638 Skipped 0 0 0 0
26639 Error 0 0 0 0
26640 Last Flush: Some(JobId(7334)) Some(JobId(7334)) Some(JobId(7038))
26641 Downstairs last five completed: 7038 7037 7036 7035 7034
26642 Upstairs last five completed: 7339 7338 7337 7336 7335
26643 CLIENT: Up:0 ds:301 act:2
266442025-02-07T20:13:15.525ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
266452025-02-07T20:13:16.517ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
266462025-02-07T20:13:16.517ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47212) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
266472025-02-07T20:13:16.518ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
266482025-02-07T20:13:16.519ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
266492025-02-07T20:13:16.519ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
266502025-02-07T20:13:16.519ZINFOcrucible: send last flush ID to this DS: Some(JobId(7038)) = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
266512025-02-07T20:13:16.519ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
266522025-02-07T20:13:16.519ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7038)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
266532025-02-07T20:13:16.519ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7038)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
26654 ----------------------------------------------------------------
26655 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26656 Downstairs last five completed: 7340 7339 7338 7337 7336
26657 Upstairs last five completed: 7340 7339 7338 7337 7336
26658 CLIENT: Up:0 ds:0 act:3
26659 Replay: All jobs finished
26660 Main task has work to do, go find it
26661 got dsc Stop(1)
26662 stop 1
26663 [8820] Got stop action so:false kr:false
26664 Feb 07 20:13:18.779 INFO request completed, latency_us: 423, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 9011e0e5-3f6d-4a6f-8733-b5a6c0b061c2, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
26665 Feb 07 20:13:18.779 INFO request completed, latency_us: 103, response_code: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:13:18.779872549Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"1","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
26666 200, uri: /state/cid/1, method: GET, req_id: 3554ebd1-98f1-4ba8-b677-340af7a33870, remote_addr: 127.0.0{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:13:18.779949403Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"1","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
26667 .1:43825, local_addr: 127.0.0.1:9998
266682025-02-07T20:13:18.779ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
266692025-02-07T20:13:18.779ZINFOcrucible: client task is exiting = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
26670 [8820] Exited with: ExitStatus(unix_wait_status(9))
26671 [8820][1] reports Exit
26672 Feb 07 20:13:22.782 INFO request completed, latency_us: 194, response_code: 200, uri: /state/cid/1, method: GET, req_id: a0df5a47-39e6-402f-b18d-db98c47959d4, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
26673 001/300 Read block 766 len 40960
26674 002/300 Read block 4107 len 32768
26675 003/300 Read block 2636 len 32768
26676 004/300 Flush
26677 005/300 Write block 2640 len 36864 data: 4 5 5 3 3 4 4 5 4
26678 006/300 Read block 293 len 40960
26679 007/300 Write block 4489 len 24576 data: 6 6 6 6 7 7
26680 008/300 Read block 1217 len 24576
26681 009/300 Read block 4926 len 40960
26682 010/300 Read block 4954 len 36864
26683 011/300 Write block 23 len 8192 data: 6 6
26684 012/300 Read block 2156 len 40960
26685 013/300 Flush
26686 014/300 Read block 228 len 28672
26687 015/300 Write block 2070 len 32768 data: 3 3 4 4 4 6 5 4
26688 016/300 Read block 2488 len 16384
26689 017/300 Flush
26690 018/300 Read block 4903 len 8192
26691 019/300 Write block 2024 len 12288 data: 7 6 5
26692 020/300 Read block 681 len 36864
26693 021/300 Read block 4622 len 12288
26694 022/300 Read block 2461 len 40960
26695 023/300 Read block 1527 len 8192
26696 024/300 Read block 3676 len 28672
26697 025/300 Read block 4403 len 36864
26698 026/300 Read block 454 len 40960
26699 027/300 Read block 723 len 12288
26700 028/300 Read block 4968 len 24576
26701 029/300 Flush
26702 030/300 Write block 4688 len 28672 data: 4 4 4 5 4 4 4
26703 031/300 Read block 4464 len 12288
26704 032/300 Write block 1865 len 8192 data: 8 9
26705 033/300 Read block 1412 len 28672
26706 034/300 Read block 2582 len 16384
26707 035/300 Write block 2247 len 8192 data: 4 3
26708 036/300 Read block 1060 len 28672
26709 037/300 Flush
26710 038/300 Write block 515 len 8192 data: 3 3
26711 039/300 Write block 2822 len 36864 data: 3 4 4 3 3 3 3 3 4
26712 040/300 Read block 1669 len 16384
26713 041/300 Write block 1894 len 36864 data: 6 5 5 6 6 7 7 7 6
26714 042/300 Flush
26715 043/300 Read block 268 len 32768
26716 044/300 Write block 4246 len 28672 data: 7 6 7 6 7 5 6
26717 045/300 Write block 1344 len 20480 data: 8 9 8 6 5
26718 046/300 Read block 2961 len 12288
26719 047/300 Read block 1843 len 28672
26720 048/300 Read block 2914 len 36864
26721 049/300 Read block 621 len 12288
26722 050/300 Write block 2568 len 40960 data: 4 4 4 4 4 4 2 2 2 2
26723 051/300 Read block 4410 len 36864
26724 052/300 Read block 920 len 4096
26725 053/300 Write block 4638 len 40960 data: 7 7 6 6 5 5 3 3 4 6
26726 054/300 Read block 1526 len 40960
26727 055/300 Read block 240 len 40960
26728 056/300 Read block 588 len 32768
26729 057/300 Read block 4391 len 28672
26730 058/300 Read block 4787 len 32768
26731 059/300 Read block 4094 len 32768
26732 060/300 Write block 226 len 8192 data: 5 3
26733 061/300 Read block 870 len 28672
26734 062/300 Write block 1550 len 40960 data: 3 3 3 3 3 3 3 4 4 4
26735 063/300 Flush
26736 064/300 Write block 4073 len 8192 data: 3 3
26737 065/300 Write block 1739 len 40960 data: 4 5 4 4 4 3 4 5 5 4
26738 066/300 Read block 995 len 32768
26739 067/300 Write block 1814 len 4096 data: 7
26740 068/300 Write block 4345 len 20480 data: 5 5 6 3 4
26741 069/300 Read block 2198 len 40960
26742 070/300 Read block 872 len 20480
26743 071/300 Read block 938 len 32768
26744 072/300 Write block 4406 len 8192 data: 5 5
26745 073/300 Flush
26746 074/300 Flush
26747 075/300 Write block 4265 len 24576 data: 4 4 3 3 3 3
26748 076/300 Write block 3487 len 12288 data: 5 5 6
26749 077/300 Write block 303 len 4096 data: 6
26750 078/300 Flush
26751 079/300 Write block 3151 len 28672 data: 6 6 6 6 8 9 7
26752 080/300 Write block 3490 len 12288 data: 5 5 4
26753 081/300 Read block 93 len 36864
26754 082/300 Write block 3260 len 40960 data: 7 8 7 7 6 7 6 5 3 4
26755 083/300 Flush
26756 084/300 Write block 3607 len 16384 data: 6 6 6 5
26757 085/300 Flush
26758 086/300 Write block 636 len 20480 data: 3 3 3 3 4
26759 087/300 Read block 1295 len 40960
26760 088/300 Read block 4974 len 20480
26761 089/300 Write block 923 len 32768 data: 2 2 2 2 3 5 6 5
26762 090/300 Flush
26763 091/300 Write block 4572 len 32768 data: 3 3 4 5 5 5 3 3
26764 092/300 Write block 1634 len 16384 data: 4 4 3 3
26765 093/300 Write block 3789 len 40960 data: 3 3 3 4 4 4 4 4 5 4
26766 094/300 Read block 145 len 40960
26767 095/300 Write block 3582 len 12288 data: 4 4 5
26768 096/300 Read block 3461 len 8192
26769 097/300 Flush
26770 098/300 Write block 3630 len 4096 data: 4
26771 099/300 Write block 205 len 20480 data: 6 6 4 3 5
26772 100/300 Write block 4137 len 12288 data: 5 3 3
26773 101/300 Read block 3760 len 36864
26774 102/300 Read block 2252 len 4096
26775 103/300 Write block 686 len 8192 data: 5 4
26776 104/300 Read block 3651 len 28672
26777 105/300 Read block 4430 len 28672
26778 106/300 Read block 862 len 20480
26779 107/300 Write block 2132 len 16384 data: 4 3 3 3
26780 108/300 Write block 2381 len 20480 data: 4 7 7 7 4
26781 109/300 Flush
26782 110/300 Read block 1114 len 40960
26783 111/300 Read block 891 len 4096
26784 112/300 Write block 2850 len 28672 data: 3 3 3 3 4 5 5
26785 113/300 Read block 788 len 40960
26786 114/300 Write block 829 len 20480 data: 8 9 8 6 5
26787 115/300 Read block 4457 len 40960
26788 116/300 Write block 2645 len 16384 data: 5 5 6 5
26789 117/300 Read block 4386 len 40960
26790 118/300 Write block 1055 len 32768 data: 4 5 6 6 5 5 5 5
26791 119/300 Read block 778 len 8192
26792 120/300 Read block 2016 len 32768
26793 121/300 Write block 607 len 20480 data: 3 3 3 3 3
26794 122/300 Flush
26795 123/300 Write block 4497 len 32768 data: 8 10 10 12 12 13 9 9
26796 124/300 Write block 3432 len 28672 data: 5 7 6 6 6 6 6
26797 125/300 Read block 3168 len 8192
26798 126/300 Flush
26799 127/300 Write block 4408 len 12288 data: 6 5 5
26800 128/300 Read block 4915 len 12288
26801 129/300 Write block 3205 len 4096 data: 7
26802 130/300 Read block 2496 len 24576
26803 131/300 Write block 4975 len 36864 data: 8 7 6 6 9 8 5 5 4
26804 132/300 Read block 131 len 24576
26805 133/300 Write block 2691 len 4096 data: 5
26806 134/300 Write block 3377 len 32768 data: 3 4 3 5 5 5 5 5
26807 135/300 Write block 2538 len 16384 data: 6 7 6 6
26808 136/300 Read block 3651 len 8192
26809 137/300 Read block 2212 len 4096
26810 138/300 Write block 2114 len 16384 data: 4 4 4 5
26811 139/300 Read block 442 len 20480
26812 140/300 Read block 878 len 12288
26813 141/300 Flush
26814 142/300 Flush
26815 143/300 Flush
26816 144/300 Write block 1661 len 16384 data: 4 4 3 2
26817 145/300 Write block 4865 len 4096 data: 5
26818 146/300 Write block 3526 len 20480 data: 3 3 5 5 5
26819 147/300 Read block 3232 len 16384
26820 148/300 Read block 2455 len 24576
26821 149/300 Read block 4557 len 20480
26822 150/300 Flush
26823 151/300 Read block 4578 len 8192
26824 152/300 Write block 3219 len 8192 data: 5 6
26825 153/300 Read block 1675 len 20480
26826 154/300 Read block 3939 len 8192
26827 155/300 Read block 2201 len 40960
26828 156/300 Write block 2827 len 28672 data: 4 4 4 5 4 6 5
26829 157/300 Write block 1716 len 20480 data: 4 5 7 6 6
26830 158/300 Read block 1689 len 4096
26831 159/300 Read block 1851 len 40960
26832 160/300 Read block 2691 len 4096
26833 161/300 Read block 1531 len 12288
26834 162/300 Read block 1564 len 40960
26835 163/300 Read block 3463 len 20480
26836 164/300 Read block 290 len 4096
26837 165/300 Flush
26838 166/300 Write block 3392 len 32768 data: 5 5 7 6 6 7 8 7
26839 167/300 Read block 4358 len 40960
26840 168/300 Read block 4506 len 4096
26841 169/300 Read block 1586 len 24576
26842 170/300 Write block 1708 len 36864 data: 4 5 5 4 4 5 4 4 5
26843 171/300 Read block 4749 len 24576
26844 172/300 Read block 3887 len 16384
26845 173/300 Read block 3736 len 32768
26846 174/300 Flush
26847 175/300 Read block 1622 len 12288
26848 176/300 Write block 4617 len 36864 data: 4 5 5 7 5 6 6 6 5
26849 177/300 Read block 2061 len 36864
26850 178/300 Read block 4445 len 4096
26851 179/300 Write block 3888 len 8192 data: 6 5
26852 180/300 Read block 1935 len 40960
26853 181/300 Write block 4756 len 40960 data: 5 4 2 2 2 4 3 3 3 2
26854 182/300 Read block 834 len 36864
26855 183/300 Read block 3875 len 20480
26856 184/300 Read block 4196 len 32768
26857 185/300 Write block 4620 len 20480 data: 8 6 7 7 7
26858 186/300 Read block 2838 len 16384
26859 187/300 Write block 1187 len 8192 data: 7 6
26860 188/300 Write block 1022 len 16384 data: 8 6 6 5
26861 189/300 Read block 2702 len 4096
26862 190/300 Read block 591 len 40960
26863 191/300 Flush
26864 192/300 Read block 4962 len 12288
26865 193/300 Write block 2320 len 8192 data: 7 5
26866 194/300 Flush
26867 195/300 Read block 1224 len 16384
26868 196/300 Read block 4928 len 32768
26869 197/300 Write block 883 len 12288 data: 6 6 5
26870 198/300 Write block 2311 len 8192 data: 7 7
26871 199/300 Flush
26872 200/300 Read block 4160 len 4096
26873 201/300 Read block 4599 len 16384
26874 202/300 Read block 4939 len 24576
26875 203/300 Read block 4432 len 24576
26876 204/300 Write block 630 len 16384 data: 6 5 4 4
26877 205/300 Flush
26878 206/300 Write block 39 len 40960 data: 7 5 5 5 5 5 6 4 4 4
26879 207/300 Read block 4388 len 16384
26880 208/300 Read block 647 len 40960
26881 209/300 Read block 3920 len 8192
26882 210/300 Read block 2977 len 32768
26883 211/300 Write block 4211 len 12288 data: 5 6 5
26884 212/300 Read block 742 len 32768
26885 213/300 Write block 66 len 20480 data: 7 7 5 5 4
26886 214/300 Read block 135 len 8192
26887 215/300 Read block 1518 len 28672
26888 216/300 Read block 4087 len 20480
26889 217/300 Read block 2915 len 20480
26890 218/300 Write block 2931 len 40960 data: 6 6 7 7 6 6 5 5 6 4
26891 219/300 Read block 4169 len 40960
26892 220/300 Read block 3139 len 20480
26893 221/300 Read block 3241 len 36864
26894 222/300 Write block 1600 len 16384 data: 3 4 4 4
26895 223/300 Read block 8 len 8192
26896 224/300 Read block 2962 len 16384
26897 225/300 Read block 2589 len 32768
26898 226/300 Flush
26899 227/300 Read block 1178 len 32768
26900 228/300 Write block 482 len 12288 data: 3 3 4
26901 229/300 Write block 2338 len 40960 data: 6 5 6 9 8 6 6 7 7 7
26902 230/300 Flush
26903 231/300 Write block 4816 len 24576 data: 6 8 8 6 6 4
26904 232/300 Write block 4569 len 40960 data: 4 4 4 4 4 5 6 6 6 4
26905 233/300 Write block 1707 len 28672 data: 4 5 6 6 5 5 6
26906 234/300 Flush
26907 235/300 Flush
26908 236/300 Read block 1876 len 28672
26909 237/300 Read block 4751 len 8192
26910 238/300 Read block 4747 len 12288
26911 239/300 Read block 4534 len 24576
26912 240/300 Read block 3361 len 32768
26913 241/300 Write block 2926 len 12288 data: 4 5 4
26914 242/300 Read block 4628 len 4096
26915 243/300 Write block 4391 len 24576 data: 3 2 3 5 5 4
26916 244/300 Read block 3663 len 40960
26917 245/300 Flush
26918 246/300 Write block 278 len 8192 data: 6 6
26919 247/300 Write block 4400 len 16384 data: 5 4 3 3
26920 248/300 Write block 770 len 20480 data: 8 7 6 5 5
26921 249/300 Read block 3387 len 4096
26922 250/300 Read block 257 len 20480
26923 251/300 Read block 2143 len 8192
26924 252/300 Read block 4708 len 24576
26925 253/300 Write block 3198 len 4096 data: 4
26926 254/300 Write block 1398 len 28672 data: 8 7 7 6 4 3 2
26927 255/300 Read block 3130 len 32768
26928 256/300 Read block 879 len 8192
26929 257/300 Read block 1028 len 16384
26930 258/300 Read block 4346 len 40960
26931 259/300 Read block 3194 len 20480
26932 260/300 Write block 808 len 32768 data: 2 2 5 5 6 8 6 6
26933 261/300 Write block 680 len 20480 data: 5 5 4 6 6
26934 262/300 Read block 1413 len 16384
26935 263/300 Flush
26936 264/300 Read block 2639 len 24576
26937 265/300 Write block 718 len 36864 data: 6 5 4 5 5 4 5 6 4
26938 266/300 Write block 4249 len 16384 data: 7 8 6 7
26939 267/300 Read block 4678 len 12288
26940 268/300 Write block 831 len 4096 data: 9
26941 269/300 Read block 3834 len 12288
26942 270/300 Write block 2163 len 40960 data: 5 7 6 6 5 3 6 8 8 7
26943 271/300 Write block 3217 len 4096 data: 4
26944 272/300 Write block 18 len 4096 data: 4
26945 273/300 Read block 1186 len 16384
26946 274/300 Flush
26947 275/300 Read block 331 len 36864
26948 276/300 Read block 4228 len 8192
26949 277/300 Write block 3090 len 36864 data: 4 6 5 5 5 5 4 4 3
26950 278/300 Write block 1792 len 8192 data: 4 4
26951 279/300 Read block 4393 len 20480
26952 280/300 Read block 2938 len 36864
26953 281/300 Flush
26954 282/300 Read block 2036 len 4096
26955 283/300 Write block 1261 len 32768 data: 4 3 4 4 4 4 3 2
26956 284/300 Read block 246 len 8192
26957 285/300 Read block 4224 len 24576
26958 286/300 Read block 2736 len 32768
26959 287/300 Read block 4803 len 20480
26960 288/300 Read block 4114 len 12288
26961 289/300 Write block 746 len 20480 data: 3 3 3 3 3
26962 290/300 Read block 1388 len 40960
26963 291/300 Write block 4630 len 36864 data: 3 3 3 2 2 2 6 7 8
26964 292/300 Write block 4421 len 20480 data: 5 6 6 4 3
26965 293/300 Read block 2524 len 28672
26966 294/300 Read block 7 len 24576
26967 295/300 Write block 6 len 32768 data: 4 4 3 4 4 5 6 6
26968 296/300 Flush
26969 297/300 Flush
26970 298/300 Read block 2457 len 16384
26971 299/300 Read block 240 len 16384
26972 300/300 Read block 1941 len 20480
26973 301/300 Write block 4218 len 40960 data: 7 7 6 5 6 6 5 4 4 4
26974 Main task has work to do, go find it
26975 got dsc Start(1)
26976 start 1
26977 Feb 07 20:13:23.033 INFO request completed, latency_us: 203, response_code: 204, uri: /start/cid/1, method: POST, req_id: 9e841736-8728-4c45-aca1-0ef92e2cabb8, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
26978 Replay: started 1, returned:Ok(())
26979 ----------------------------------------------------------------
26980 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26981 States: Active Offline Active
26982 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26983 7341 Acked Read 10 Done Sent Done false
26984 7342 Acked Read 8 Done Sent Done false
26985 7343 Acked Read 8 Done Sent Done false
26986 7344 Acked Flush 0 Done Sent Done false
26987 7345 Acked Write 9 Done Sent Done false
26988 7346 Acked Read 10 Done Sent Done false
26989 7347 Acked Write 6 Done Sent Done false
26990 7348 Acked Read 6 Done Sent Done false
26991 7349 Acked Read 10 Done Sent Done false
26992 7350 Acked Read 9 Done Sent Done false
26993 7351 Acked Write 2 Done Sent Done false
26994 7352 Acked Read 10 Done Sent Done false
26995 7353 Acked Flush 0 Done Sent Done false
26996 7354 Acked Read 7 Done Sent Done false
26997 7355 Acked Write 8 Done Sent Done false
26998 7356 Acked Read 4 Done Sent Done false
26999 7357 Acked Flush 0 Done Sent Done false
27000 7358 Acked Read 2 Done Sent Done false
27001 7359 Acked Write 3 Done Sent Done false
27002 7360 Acked Read 9 Done Sent Done false
27003 7361 Acked Read 3 Done Sent Done false
27004 7362 Acked Read 10 Done Sent Done false
27005 7363 Acked Read 2 Done Sent Done false
27006 7364 Acked Read 7 Done Sent Done false
27007 7365 Acked Read 9 Done Sent Done false
27008 7366 Acked Read 10 Done Sent Done false
27009 7367 Acked Read 3 Done Sent Done false
27010 7368 Acked Read 6 Done Sent Done false
27011 7369 Acked Flush 0 Done Sent Done false
27012 7370 Acked Write 7 Done Sent Done false
27013 7371 Acked Read 3 Done Sent Done false
27014 7372 Acked Write 2 Done Sent Done false
27015 7373 Acked Read 7 Done Sent Done false
27016 7374 Acked Read 4 Done Sent Done false
27017 7375 Acked Write 2 Done Sent Done false
27018 7376 Acked Read 7 Done Sent Done false
27019 7377 Acked Flush 0 Done Sent Done false
27020 7378 Acked Write 2 Done Sent Done false
27021 7379 Acked Write 9 Done Sent Done false
27022 7380 Acked Read 4 Done Sent Done false
27023 7381 Acked Write 9 Done Sent Done false
27024 7382 Acked Flush 0 Done Sent Done false
27025 7383 Acked Read 8 Done Sent Done false
27026 7384 Acked Write 7 Done Sent Done false
27027 7385 Acked Write 5 Done Sent Done false
27028 7386 Acked Read 3 Done Sent Done false
27029 7387 Acked Read 7 Done Sent Done false
27030 7388 Acked Read 9 Done Sent Done false
27031 7389 Acked Read 3 Done Sent Done false
27032 7390 Acked Write 10 Done Sent Done false
27033 7391 Acked Read 9 Done Sent Done false
27034 7392 Acked Read 1 Done Sent Done false
27035 7393 Acked Write 10 Done Sent Done false
27036 7394 Acked Read 10 Done Sent Done false
27037 7395 Acked Read 10 Done Sent Done false
27038 7396 Acked Read 8 Done Sent Done false
27039 7397 Acked Read 7 Done Sent Done false
27040 7398 Acked Read 8 Done Sent Done false
27041 7399 Acked Read 8 Done Sent Done false
27042 7400 Acked Write 2 Done Sent Done false
27043 7401 Acked Read 7 Done Sent Done false
27044 7402 Acked Write 10 Done Sent Done false
27045 7403 Acked Flush 0 Done Sent Done false
27046 7404 Acked Write 2 Done Sent Done false
27047 7405 Acked Write 10 Done Sent Done false
27048 7406 Acked Read 8 Done Sent Done false
27049 7407 Acked Write 1 Done Sent Done false
27050 7408 Acked Write 5 Done Sent Done false
27051 7409 Acked Read 10 Done Sent Done false
27052 7410 Acked Read 5 Done Sent Done false
27053 7411 Acked Read 8 Done Sent Done false
27054 7412 Acked Write 2 Done Sent Done false
27055 7413 Acked Flush 0 Done Sent Done false
27056 7414 Acked Flush 0 Done Sent Done false
27057 7415 Acked Write 6 Done Sent Done false
27058 7416 Acked Write 3 Done Sent Done false
27059 7417 Acked Write 1 Done Sent Done false
27060 7418 Acked Flush 0 Done Sent Done false
27061 7419 Acked Write 7 Done Sent Done false
27062 7420 Acked Write 3 Done Sent Done false
27063 7421 Acked Read 9 Done Sent Done false
27064 7422 Acked Write 10 Done Sent Done false
27065 7423 Acked Flush 0 Done Sent Done false
27066 7424 Acked Write 4 Done Sent Done false
27067 7425 Acked Flush 0 Done Sent Done false
27068 7426 Acked Write 5 Done Sent Done false
27069 7427 Acked Read 10 Done Sent Done false
27070 7428 Acked Read 5 Done Sent Done false
27071 7429 Acked Write 8 Done Sent Done false
27072 7430 Acked Flush 0 Done Sent Done false
27073 7431 Acked Write 8 Done Sent Done false
27074 7432 Acked Write 4 Done Sent Done false
27075 7433 Acked Write 10 Done Sent Done false
27076 7434 Acked Read 10 Done Sent Done false
27077 7435 Acked Write 3 Done Sent Done false
27078 7436 Acked Read 2 Done Sent Done false
27079 7437 Acked Flush 0 Done Sent Done false
27080 7438 Acked Write 1 Done Sent Done false
27081 7439 Acked Write 5 Done Sent Done false
27082 7440 Acked Write 3 Done Sent Done false
27083 7441 Acked Read 9 Done Sent Done false
27084 7442 Acked Read 1 Done Sent Done false
27085 7443 Acked Write 2 Done Sent Done false
27086 7444 Acked Read 7 Done Sent Done false
27087 7445 Acked Read 7 Done Sent Done false
27088 7446 Acked Read 5 Done Sent Done false
27089 7447 Acked Write 4 Done Sent Done false
27090 7448 Acked Write 5 Done Sent Done false
27091 7449 Acked Flush 0 Done Sent Done false
27092 7450 Acked Read 10 Done Sent Done false
27093 7451 Acked Read 1 Done Sent Done false
27094 7452 Acked Write 7 Done Sent Done false
27095 7453 Acked Read 10 Done Sent Done false
27096 7454 Acked Write 5 Done Sent Done false
27097 7455 Acked Read 10 Done Sent Done false
27098 7456 Acked Write 4 Done Sent Done false
27099 7457 Acked Read 10 Done Sent Done false
27100 7458 Acked Write 8 Done Sent Done false
27101 7459 Acked Read 2 Done Sent Done false
27102 7460 Acked Read 8 Done Sent Done false
27103 7461 Acked Write 5 Done Sent Done false
27104 7462 Acked Flush 0 Done Sent Done false
27105 7463 Acked Write 8 Done Sent Done false
27106 7464 Acked Write 7 Done Sent Done false
27107 7465 Acked Read 2 Done Sent Done false
27108 7466 Acked Flush 0 Done Sent Done false
27109 7467 Acked Write 3 Done Sent Done false
27110 7468 Acked Read 3 Done Sent Done false
27111 7469 Acked Write 1 Done Sent Done false
27112 7470 Acked Read 6 Done Sent Done false
27113 7471 Acked Write 9 Done Sent Done false
27114 7472 Acked Read 6 Done Sent Done false
27115 7473 Acked Write 1 Done Sent Done false
27116 7474 Acked Write 8 Done Sent Done false
27117 7475 Acked Write 4 Done Sent Done false
27118 7476 Acked Read 2 Done Sent Done false
27119 7477 Acked Read 1 Done Sent Done false
27120 7478 Acked Write 4 Done Sent Done false
27121 7479 Acked Read 5 Done Sent Done false
27122 7480 Acked Read 3 Done Sent Done false
27123 7481 Acked Flush 0 Done Sent Done false
27124 7482 Acked Flush 0 Done Sent Done false
27125 7483 Acked Flush 0 Done Sent Done false
27126 7484 Acked Write 4 Done Sent Done false
27127 7485 Acked Write 1 Done Sent Done false
27128 7486 Acked Write 5 Done Sent Done false
27129 7487 Acked Read 4 Done Sent Done false
27130 7488 Acked Read 6 Done Sent Done false
27131 7489 Acked Read 5 Done Sent Done false
27132 7490 Acked Flush 0 Done Sent Done false
27133 7491 Acked Read 2 Done Sent Done false
27134 7492 Acked Write 2 Done Sent Done false
27135 7493 Acked Read 5 Done Sent Done false
27136 7494 Acked Read 2 Done Sent Done false
27137 7495 Acked Read 10 Done Sent Done false
27138 7496 Acked Write 7 Done Sent Done false
27139 7497 Acked Write 5 Done Sent Done false
27140 7498 Acked Read 1 Done Sent Done false
27141 7499 Acked Read 10 Done Sent Done false
27142 7500 Acked Read 1 Done Sent Done false
27143 7501 Acked Read 3 Done Sent Done false
27144 7502 Acked Read 10 Done Sent Done false
27145 7503 Acked Read 5 Done Sent Done false
27146 7504 Acked Read 1 Done Sent Done false
27147 7505 Acked Flush 0 Done Sent Done false
27148 7506 Acked Write 8 Done Sent Done false
27149 7507 Acked Read 10 Done Sent Done false
27150 7508 Acked Read 1 Done Sent Done false
27151 7509 Acked Read 6 Done Sent Done false
27152 7510 Acked Write 9 Done Sent Done false
27153 7511 Acked Read 6 Done Sent Done false
27154 7512 Acked Read 4 Done Sent Done false
27155 7513 Acked Read 8 Done Sent Done false
27156 7514 Acked Flush 0 Done Sent Done false
27157 7515 Acked Read 3 Done Sent Done false
27158 7516 Acked Write 9 Done Sent Done false
27159 7517 Acked Read 9 Done Sent Done false
27160 7518 Acked Read 1 Done Sent Done false
27161 7519 Acked Write 2 Done Sent Done false
27162 7520 Acked Read 10 Done Sent Done false
27163 7521 Acked Write 10 Done Sent Done false
27164 7522 Acked Read 9 Done Sent Done false
27165 7523 Acked Read 5 Done Sent Done false
27166 7524 Acked Read 8 Done Sent Done false
27167 7525 Acked Write 5 Done Sent Done false
27168 7526 Acked Read 4 Done Sent Done false
27169 7527 Acked Write 2 Done Sent Done false
27170 7528 Acked Write 4 Done Sent Done false
27171 7529 Acked Read 1 Done Sent Done false
27172 7530 Acked Read 10 Done Sent Done false
27173 7531 Acked Flush 0 Done Sent Done false
27174 7532 Acked Read 3 Done Sent Done false
27175 7533 Acked Write 2 Done Sent Done false
27176 7534 Acked Flush 0 Done Sent Done false
27177 7535 Acked Read 4 Done Sent Done false
27178 7536 Acked Read 8 Done Sent Done false
27179 7537 Acked Write 3 Done Sent Done false
27180 7538 Acked Write 2 Done Sent Done false
27181 7539 Acked Flush 0 Done Sent Done false
27182 7540 Acked Read 1 Done Sent Done false
27183 7541 Acked Read 4 Done Sent Done false
27184 7542 Acked Read 6 Done Sent Done false
27185 7543 Acked Read 6 Done Sent Done false
27186 7544 Acked Write 4 Done Sent Done false
27187 7545 Acked Flush 0 Done Sent Done false
27188 7546 Acked Write 10 Done Sent Done false
27189 7547 Acked Read 4 Done Sent Done false
27190 7548 Acked Read 10 Done Sent Done false
27191 7549 Acked Read 2 Done Sent Done false
27192 7550 Acked Read 8 Done Sent Done false
27193 7551 Acked Write 3 Done Sent Done false
27194 7552 Acked Read 8 Done Sent Done false
27195 7553 Acked Write 5 Done Sent Done false
27196 7554 Acked Read 2 Done Sent Done false
27197 7555 Acked Read 7 Done Sent Done false
27198 7556 Acked Read 5 Done Sent Done false
27199 7557 Acked Read 5 Done Sent Done false
27200 7558 Acked Write 10 Done Sent Done false
27201 7559 Acked Read 10 Done Sent Done false
27202 7560 Acked Read 5 Done Sent Done false
27203 7561 Acked Read 9 Done Sent Done false
27204 7562 Acked Write 4 Done Sent Done false
27205 7563 Acked Read 2 Done Sent Done false
27206 7564 Acked Read 4 Done Sent Done false
27207 7565 Acked Read 8 Done Sent Done false
27208 7566 Acked Flush 0 Done Sent Done false
27209 7567 Acked Read 8 Done Sent Done false
27210 7568 Acked Write 3 Done Sent Done false
27211 7569 Acked Write 10 Done Sent Done false
27212 7570 Acked Flush 0 Done Sent Done false
27213 7571 Acked Write 6 Done Sent Done false
27214 7572 Acked Write 10 Done Sent Done false
27215 7573 Acked Write 7 Done Sent Done false
27216 7574 Acked Flush 0 Done Sent Done false
27217 7575 Acked Flush 0 Done Sent Done false
27218 7576 Acked Read 7 Done Sent Done false
27219 7577 Acked Read 2 Done Sent Done false
27220 7578 Acked Read 3 Done Sent Done false
27221 7579 Acked Read 6 Done Sent Done false
27222 7580 Acked Read 8 Done Sent Done false
27223 7581 Acked Write 3 Done Sent Done false
27224 7582 Acked Read 1 Done Sent Done false
27225 7583 Acked Write 6 Done Sent Done false
27226 7584 Acked Read 10 Done Sent Done false
27227 7585 Acked Flush 0 Done Sent Done false
27228 7586 Acked Write 2 Done Sent Done false
27229 7587 Acked Write 4 Done Sent Done false
27230 7588 Acked Write 5 Done Sent Done false
27231 7589 Acked Read 1 Done Sent Done false
27232 7590 Acked Read 5 Done Sent Done false
27233 7591 Acked Read 2 Done Sent Done false
27234 7592 Acked Read 6 Done Sent Done false
27235 7593 Acked Write 1 Done Sent Done false
27236 7594 Acked Write 7 Done Sent Done false
27237 7595 Acked Read 8 Done Sent Done false
27238 7596 Acked Read 2 Done Sent Done false
27239 7597 Acked Read 4 Done Sent Done false
27240 7598 Acked Read 10 Done Sent Done false
27241 7599 Acked Read 5 Done Sent Done false
27242 7600 Acked Write 8 Done Sent Done false
27243 7601 Acked Write 5 Done Sent Done false
27244 7602 Acked Read 4 Done Sent Done false
27245 7603 Acked Flush 0 Done Sent Done false
27246 7604 Acked Read 6 Done Sent Done false
27247 7605 Acked Write 9 Done Sent Done false
27248 7606 Acked Write 4 Done Sent Done false
27249 7607 Acked Read 3 Done Sent Done false
27250 7608 Acked Write 1 Done Sent Done false
27251 7609 Acked Read 3 Done Sent Done false
27252 7610 Acked Write 10 Done Sent Done false
27253 7611 Acked Write 1 Done Sent Done false
27254 7612 Acked Write 1 Done Sent Done false
27255 7613 Acked Read 4 Done Sent Done false
27256 7614 Acked Flush 0 Done Sent Done false
27257 7615 Acked Read 9 Done Sent Done false
27258 7616 Acked Read 2 Done Sent Done false
27259 7617 Acked Write 9 Done Sent Done false
27260 7618 Acked Write 2 Done Sent Done false
27261 7619 Acked Read 5 Done Sent Done false
27262 7620 Acked Read 9 Done Sent Done false
27263 7621 Acked Flush 0 Done Sent Done false
27264 7622 Acked Read 1 Done Sent Done false
27265 7623 Acked Write 8 Done Sent Done false
27266 7624 Acked Read 2 Done Sent Done false
27267 7625 Acked Read 6 Done Sent Done false
27268 7626 Acked Read 8 Done Sent Done false
27269 7627 Acked Read 5 Done Sent Done false
27270 7628 Acked Read 3 Done Sent Done false
27271 7629 Acked Write 5 Done Sent Done false
27272 7630 Acked Read 10 Done Sent Done false
27273 7631 Acked Write 9 Done Sent Done false
27274 7632 Acked Write 5 Done Sent Done false
27275 7633 Acked Read 7 Done Sent Done false
27276 7634 Acked Read 6 Done Sent Done false
27277 7635 Acked Write 8 Done Sent Done false
27278 7636 Acked Flush 0 Done Sent Done false
27279 7637 Acked Flush 0 Done Sent Done false
27280 7638 Acked Read 4 Done Sent Done false
27281 7639 Acked Read 4 Done Sent Done false
27282 7640 Acked Read 5 Done Sent Done false
27283 7641 Acked Write 10 Done Sent Done false
27284 STATES DS:0 DS:1 DS:2 TOTAL
27285 Sent 0 301 0 301
27286 Done 301 0 301 602
27287 Skipped 0 0 0 0
27288 Error 0 0 0 0
27289 Last Flush: Some(JobId(7637)) Some(JobId(7340)) Some(JobId(7637))
27290 Downstairs last five completed: 7340 7339 7338 7337 7336
27291 Upstairs last five completed: 7641 7640 7639 7638 7637
27292 CLIENT: Up:0 ds:301 act:2
272932025-02-07T20:13:23.539ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
272942025-02-07T20:13:24.540ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
272952025-02-07T20:13:25.542ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27296 [8820] got start action
27297 [8820] I am going to restart
27298 Starting downstairs at port 8820
27299 Make output file at "/tmp/dsc/downstairs-8820.txt"
27300 [8820][1] reports Starting
27301 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1231)
27302 [8820][1] reports Running
273032025-02-07T20:13:27.022ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27304 ----------------------------------------------------------------
27305 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27306 States: Active Offline Active
27307 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27308 7341 Acked Read 10 Done Sent Done false
27309 7342 Acked Read 8 Done Sent Done false
27310 7343 Acked Read 8 Done Sent Done false
27311 7344 Acked Flush 0 Done Sent Done false
27312 7345 Acked Write 9 Done Sent Done false
27313 7346 Acked Read 10 Done Sent Done false
27314 7347 Acked Write 6 Done Sent Done false
27315 7348 Acked Read 6 Done Sent Done false
27316 7349 Acked Read 10 Done Sent Done false
27317 7350 Acked Read 9 Done Sent Done false
27318 7351 Acked Write 2 Done Sent Done false
27319 7352 Acked Read 10 Done Sent Done false
27320 7353 Acked Flush 0 Done Sent Done false
27321 7354 Acked Read 7 Done Sent Done false
27322 7355 Acked Write 8 Done Sent Done false
27323 7356 Acked Read 4 Done Sent Done false
27324 7357 Acked Flush 0 Done Sent Done false
27325 7358 Acked Read 2 Done Sent Done false
27326 7359 Acked Write 3 Done Sent Done false
27327 7360 Acked Read 9 Done Sent Done false
27328 7361 Acked Read 3 Done Sent Done false
27329 7362 Acked Read 10 Done Sent Done false
27330 7363 Acked Read 2 Done Sent Done false
27331 7364 Acked Read 7 Done Sent Done false
27332 7365 Acked Read 9 Done Sent Done false
27333 7366 Acked Read 10 Done Sent Done false
27334 7367 Acked Read 3 Done Sent Done false
27335 7368 Acked Read 6 Done Sent Done false
27336 7369 Acked Flush 0 Done Sent Done false
27337 7370 Acked Write 7 Done Sent Done false
27338 7371 Acked Read 3 Done Sent Done false
27339 7372 Acked Write 2 Done Sent Done false
27340 7373 Acked Read 7 Done Sent Done false
27341 7374 Acked Read 4 Done Sent Done false
27342 7375 Acked Write 2 Done Sent Done false
27343 7376 Acked Read 7 Done Sent Done false
27344 7377 Acked Flush 0 Done Sent Done false
27345 7378 Acked Write 2 Done Sent Done false
27346 7379 Acked Write 9 Done Sent Done false
27347 7380 Acked Read 4 Done Sent Done false
27348 7381 Acked Write 9 Done Sent Done false
27349 7382 Acked Flush 0 Done Sent Done false
27350 7383 Acked Read 8 Done Sent Done false
27351 7384 Acked Write 7 Done Sent Done false
27352 7385 Acked Write 5 Done Sent Done false
27353 7386 Acked Read 3 Done Sent Done false
27354 7387 Acked Read 7 Done Sent Done false
27355 7388 Acked Read 9 Done Sent Done false
27356 7389 Acked Read 3 Done Sent Done false
27357 7390 Acked Write 10 Done Sent Done false
27358 7391 Acked Read 9 Done Sent Done false
27359 7392 Acked Read 1 Done Sent Done false
27360 7393 Acked Write 10 Done Sent Done false
27361 7394 Acked Read 10 Done Sent Done false
27362 7395 Acked Read 10 Done Sent Done false
27363 7396 Acked Read 8 Done Sent Done false
27364 7397 Acked Read 7 Done Sent Done false
27365 7398 Acked Read 8 Done Sent Done false
27366 7399 Acked Read 8 Done Sent Done false
27367 7400 Acked Write 2 Done Sent Done false
27368 7401 Acked Read 7 Done Sent Done false
27369 7402 Acked Write 10 Done Sent Done false
27370 7403 Acked Flush 0 Done Sent Done false
27371 7404 Acked Write 2 Done Sent Done false
27372 7405 Acked Write 10 Done Sent Done false
27373 7406 Acked Read 8 Done Sent Done false
27374 7407 Acked Write 1 Done Sent Done false
27375 7408 Acked Write 5 Done Sent Done false
27376 7409 Acked Read 10 Done Sent Done false
27377 7410 Acked Read 5 Done Sent Done false
27378 7411 Acked Read 8 Done Sent Done false
27379 7412 Acked Write 2 Done Sent Done false
27380 7413 Acked Flush 0 Done Sent Done false
27381 7414 Acked Flush 0 Done Sent Done false
27382 7415 Acked Write 6 Done Sent Done false
27383 7416 Acked Write 3 Done Sent Done false
27384 7417 Acked Write 1 Done Sent Done false
27385 7418 Acked Flush 0 Done Sent Done false
27386 7419 Acked Write 7 Done Sent Done false
27387 7420 Acked Write 3 Done Sent Done false
27388 7421 Acked Read 9 Done Sent Done false
27389 7422 Acked Write 10 Done Sent Done false
27390 7423 Acked Flush 0 Done Sent Done false
27391 7424 Acked Write 4 Done Sent Done false
27392 7425 Acked Flush 0 Done Sent Done false
27393 7426 Acked Write 5 Done Sent Done false
27394 7427 Acked Read 10 Done Sent Done false
27395 7428 Acked Read 5 Done Sent Done false
27396 7429 Acked Write 8 Done Sent Done false
27397 7430 Acked Flush 0 Done Sent Done false
27398 7431 Acked Write 8 Done Sent Done false
27399 7432 Acked Write 4 Done Sent Done false
27400 7433 Acked Write 10 Done Sent Done false
27401 7434 Acked Read 10 Done Sent Done false
27402 7435 Acked Write 3 Done Sent Done false
27403 7436 Acked Read 2 Done Sent Done false
27404 7437 Acked Flush 0 Done Sent Done false
27405 7438 Acked Write 1 Done Sent Done false
27406 7439 Acked Write 5 Done Sent Done false
27407 7440 Acked Write 3 Done Sent Done false
27408 7441 Acked Read 9 Done Sent Done false
27409 7442 Acked Read 1 Done Sent Done false
27410 7443 Acked Write 2 Done Sent Done false
27411 7444 Acked Read 7 Done Sent Done false
27412 7445 Acked Read 7 Done Sent Done false
27413 7446 Acked Read 5 Done Sent Done false
27414 7447 Acked Write 4 Done Sent Done false
27415 7448 Acked Write 5 Done Sent Done false
27416 7449 Acked Flush 0 Done Sent Done false
27417 7450 Acked Read 10 Done Sent Done false
27418 7451 Acked Read 1 Done Sent Done false
27419 7452 Acked Write 7 Done Sent Done false
27420 7453 Acked Read 10 Done Sent Done false
27421 7454 Acked Write 5 Done Sent Done false
27422 7455 Acked Read 10 Done Sent Done false
27423 7456 Acked Write 4 Done Sent Done false
27424 7457 Acked Read 10 Done Sent Done false
27425 7458 Acked Write 8 Done Sent Done false
27426 7459 Acked Read 2 Done Sent Done false
27427 7460 Acked Read 8 Done Sent Done false
27428 7461 Acked Write 5 Done Sent Done false
27429 7462 Acked Flush 0 Done Sent Done false
27430 7463 Acked Write 8 Done Sent Done false
27431 7464 Acked Write 7 Done Sent Done false
27432 7465 Acked Read 2 Done Sent Done false
27433 7466 Acked Flush 0 Done Sent Done false
27434 7467 Acked Write 3 Done Sent Done false
27435 7468 Acked Read 3 Done Sent Done false
27436 7469 Acked Write 1 Done Sent Done false
27437 7470 Acked Read 6 Done Sent Done false
27438 7471 Acked Write 9 Done Sent Done false
27439 7472 Acked Read 6 Done Sent Done false
27440 7473 Acked Write 1 Done Sent Done false
27441 7474 Acked Write 8 Done Sent Done false
27442 7475 Acked Write 4 Done Sent Done false
27443 7476 Acked Read 2 Done Sent Done false
27444 7477 Acked Read 1 Done Sent Done false
27445 7478 Acked Write 4 Done Sent Done false
27446 7479 Acked Read 5 Done Sent Done false
27447 7480 Acked Read 3 Done Sent Done false
27448 7481 Acked Flush 0 Done Sent Done false
27449 7482 Acked Flush 0 Done Sent Done false
27450 7483 Acked Flush 0 Done Sent Done false
27451 7484 Acked Write 4 Done Sent Done false
27452 7485 Acked Write 1 Done Sent Done false
27453 7486 Acked Write 5 Done Sent Done false
27454 7487 Acked Read 4 Done Sent Done false
27455 7488 Acked Read 6 Done Sent Done false
27456 7489 Acked Read 5 Done Sent Done false
27457 7490 Acked Flush 0 Done Sent Done false
27458 7491 Acked Read 2 Done Sent Done false
27459 7492 Acked Write 2 Done Sent Done false
27460 7493 Acked Read 5 Done Sent Done false
27461 7494 Acked Read 2 Done Sent Done false
27462 7495 Acked Read 10 Done Sent Done false
27463 7496 Acked Write 7 Done Sent Done false
27464 7497 Acked Write 5 Done Sent Done false
27465 7498 Acked Read 1 Done Sent Done false
27466 7499 Acked Read 10 Done Sent Done false
27467 7500 Acked Read 1 Done Sent Done false
27468 7501 Acked Read 3 Done Sent Done false
27469 7502 Acked Read 10 Done Sent Done false
27470 7503 Acked Read 5 Done Sent Done false
27471 7504 Acked Read 1 Done Sent Done false
27472 7505 Acked Flush 0 Done Sent Done false
27473 7506 Acked Write 8 Done Sent Done false
27474 7507 Acked Read 10 Done Sent Done false
27475 7508 Acked Read 1 Done Sent Done false
27476 7509 Acked Read 6 Done Sent Done false
27477 7510 Acked Write 9 Done Sent Done false
27478 7511 Acked Read 6 Done Sent Done false
27479 7512 Acked Read 4 Done Sent Done false
27480 7513 Acked Read 8 Done Sent Done false
27481 7514 Acked Flush 0 Done Sent Done false
27482 7515 Acked Read 3 Done Sent Done false
27483 7516 Acked Write 9 Done Sent Done false
27484 7517 Acked Read 9 Done Sent Done false
27485 7518 Acked Read 1 Done Sent Done false
27486 7519 Acked Write 2 Done Sent Done false
27487 7520 Acked Read 10 Done Sent Done false
27488 7521 Acked Write 10 Done Sent Done false
27489 7522 Acked Read 9 Done Sent Done false
27490 7523 Acked Read 5 Done Sent Done false
27491 7524 Acked Read 8 Done Sent Done false
27492 7525 Acked Write 5 Done Sent Done false
27493 7526 Acked Read 4 Done Sent Done false
27494 7527 Acked Write 2 Done Sent Done false
27495 7528 Acked Write 4 Done Sent Done false
27496 7529 Acked Read 1 Done Sent Done false
27497 7530 Acked Read 10 Done Sent Done false
27498 7531 Acked Flush 0 Done Sent Done false
27499 7532 Acked Read 3 Done Sent Done false
27500 7533 Acked Write 2 Done Sent Done false
27501 7534 Acked Flush 0 Done Sent Done false
27502 7535 Acked Read 4 Done Sent Done false
27503 7536 Acked Read 8 Done Sent Done false
27504 7537 Acked Write 3 Done Sent Done false
27505 7538 Acked Write 2 Done Sent Done false
27506 7539 Acked Flush 0 Done Sent Done false
27507 7540 Acked Read 1 Done Sent Done false
27508 7541 Acked Read 4 Done Sent Done false
27509 7542 Acked Read 6 Done Sent Done false
27510 7543 Acked Read 6 Done Sent Done false
27511 7544 Acked Write 4 Done Sent Done false
27512 7545 Acked Flush 0 Done Sent Done false
27513 7546 Acked Write 10 Done Sent Done false
27514 7547 Acked Read 4 Done Sent Done false
27515 7548 Acked Read 10 Done Sent Done false
27516 7549 Acked Read 2 Done Sent Done false
27517 7550 Acked Read 8 Done Sent Done false
27518 7551 Acked Write 3 Done Sent Done false
27519 7552 Acked Read 8 Done Sent Done false
27520 7553 Acked Write 5 Done Sent Done false
27521 7554 Acked Read 2 Done Sent Done false
27522 7555 Acked Read 7 Done Sent Done false
27523 7556 Acked Read 5 Done Sent Done false
27524 7557 Acked Read 5 Done Sent Done false
27525 7558 Acked Write 10 Done Sent Done false
27526 7559 Acked Read 10 Done Sent Done false
27527 7560 Acked Read 5 Done Sent Done false
27528 7561 Acked Read 9 Done Sent Done false
27529 7562 Acked Write 4 Done Sent Done false
27530 7563 Acked Read 2 Done Sent Done false
27531 7564 Acked Read 4 Done Sent Done false
27532 7565 Acked Read 8 Done Sent Done false
27533 7566 Acked Flush 0 Done Sent Done false
27534 7567 Acked Read 8 Done Sent Done false
27535 7568 Acked Write 3 Done Sent Done false
27536 7569 Acked Write 10 Done Sent Done false
27537 7570 Acked Flush 0 Done Sent Done false
27538 7571 Acked Write 6 Done Sent Done false
27539 7572 Acked Write 10 Done Sent Done false
27540 7573 Acked Write 7 Done Sent Done false
27541 7574 Acked Flush 0 Done Sent Done false
27542 7575 Acked Flush 0 Done Sent Done false
27543 7576 Acked Read 7 Done Sent Done false
27544 7577 Acked Read 2 Done Sent Done false
27545 7578 Acked Read 3 Done Sent Done false
27546 7579 Acked Read 6 Done Sent Done false
27547 7580 Acked Read 8 Done Sent Done false
27548 7581 Acked Write 3 Done Sent Done false
27549 7582 Acked Read 1 Done Sent Done false
27550 7583 Acked Write 6 Done Sent Done false
27551 7584 Acked Read 10 Done Sent Done false
27552 7585 Acked Flush 0 Done Sent Done false
27553 7586 Acked Write 2 Done Sent Done false
27554 7587 Acked Write 4 Done Sent Done false
27555 7588 Acked Write 5 Done Sent Done false
27556 7589 Acked Read 1 Done Sent Done false
27557 7590 Acked Read 5 Done Sent Done false
27558 7591 Acked Read 2 Done Sent Done false
27559 7592 Acked Read 6 Done Sent Done false
27560 7593 Acked Write 1 Done Sent Done false
27561 7594 Acked Write 7 Done Sent Done false
27562 7595 Acked Read 8 Done Sent Done false
27563 7596 Acked Read 2 Done Sent Done false
27564 7597 Acked Read 4 Done Sent Done false
27565 7598 Acked Read 10 Done Sent Done false
27566 7599 Acked Read 5 Done Sent Done false
27567 7600 Acked Write 8 Done Sent Done false
27568 7601 Acked Write 5 Done Sent Done false
27569 7602 Acked Read 4 Done Sent Done false
27570 7603 Acked Flush 0 Done Sent Done false
27571 7604 Acked Read 6 Done Sent Done false
27572 7605 Acked Write 9 Done Sent Done false
27573 7606 Acked Write 4 Done Sent Done false
27574 7607 Acked Read 3 Done Sent Done false
27575 7608 Acked Write 1 Done Sent Done false
27576 7609 Acked Read 3 Done Sent Done false
27577 7610 Acked Write 10 Done Sent Done false
27578 7611 Acked Write 1 Done Sent Done false
27579 7612 Acked Write 1 Done Sent Done false
27580 7613 Acked Read 4 Done Sent Done false
27581 7614 Acked Flush 0 Done Sent Done false
27582 7615 Acked Read 9 Done Sent Done false
27583 7616 Acked Read 2 Done Sent Done false
27584 7617 Acked Write 9 Done Sent Done false
27585 7618 Acked Write 2 Done Sent Done false
27586 7619 Acked Read 5 Done Sent Done false
27587 7620 Acked Read 9 Done Sent Done false
27588 7621 Acked Flush 0 Done Sent Done false
27589 7622 Acked Read 1 Done Sent Done false
27590 7623 Acked Write 8 Done Sent Done false
27591 7624 Acked Read 2 Done Sent Done false
27592 7625 Acked Read 6 Done Sent Done false
27593 7626 Acked Read 8 Done Sent Done false
27594 7627 Acked Read 5 Done Sent Done false
27595 7628 Acked Read 3 Done Sent Done false
27596 7629 Acked Write 5 Done Sent Done false
27597 7630 Acked Read 10 Done Sent Done false
27598 7631 Acked Write 9 Done Sent Done false
27599 7632 Acked Write 5 Done Sent Done false
27600 7633 Acked Read 7 Done Sent Done false
27601 7634 Acked Read 6 Done Sent Done false
27602 7635 Acked Write 8 Done Sent Done false
27603 7636 Acked Flush 0 Done Sent Done false
27604 7637 Acked Flush 0 Done Sent Done false
27605 7638 Acked Read 4 Done Sent Done false
27606 7639 Acked Read 4 Done Sent Done false
27607 7640 Acked Read 5 Done Sent Done false
27608 7641 Acked Write 10 Done Sent Done false
27609 STATES DS:0 DS:1 DS:2 TOTAL
27610 Sent 0 301 0 301
27611 Done 301 0 301 602
27612 Skipped 0 0 0 0
27613 Error 0 0 0 0
27614 Last Flush: Some(JobId(7637)) Some(JobId(7340)) Some(JobId(7637))
27615 Downstairs last five completed: 7340 7339 7338 7337 7336
27616 Upstairs last five completed: 7641 7640 7639 7638 7637
27617 CLIENT: Up:0 ds:301 act:2
276182025-02-07T20:13:27.545ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
276192025-02-07T20:13:28.546ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
276202025-02-07T20:13:28.781ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
276212025-02-07T20:13:28.781ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41605) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
276222025-02-07T20:13:28.782ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
276232025-02-07T20:13:28.782ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
276242025-02-07T20:13:28.782ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
276252025-02-07T20:13:28.782ZINFOcrucible: send last flush ID to this DS: Some(JobId(7340)) = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
276262025-02-07T20:13:28.782ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
276272025-02-07T20:13:28.782ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7340)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
276282025-02-07T20:13:28.783ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7340)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27629 ----------------------------------------------------------------
27630 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27631 Downstairs last five completed: 7642 7641 7640 7639 7638
27632 Upstairs last five completed: 7642 7641 7640 7639 7638
27633 CLIENT: Up:0 ds:0 act:3
27634 Replay: All jobs finished
27635 Main task has work to do, go find it
27636 got dsc Stop(0)
27637 stop 0
27638 [8810] Got stop action so:false kr:false
27639 Feb 07 20:13:31.041 INFO request completed, latency_us: 654, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 2e2d235f-baf7-48a0-9725-5f5ceb88d972, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
276402025-02-07T20:13:31.041ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
276412025-02-07T20:13:31.041ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
276422025-02-07T20:13:31.041ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
27643 Feb 07 20:13:31.041 INFO request completed, latency_us: 125, response_code: 200, uri: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-07T20:13:31.041992062Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
27644 /state/cid/0, method: GET, req_id: db9dff52-d131-4a85-bd41-6d95fa7dd84b, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
27645 [8810] Exited with: ExitStatus(unix_wait_status(9))
27646 [8810][0] reports Exit
27647 Feb 07 20:13:35.044 INFO request completed, latency_us: 224, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1ea9fd90-1a21-4064-8807-889c2606e996, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
27648 001/300 Write block 3789 len 24576 data: 4 4 4 5 5 5
27649 002/300 Flush
27650 003/300 Write block 4004 len 40960 data: 5 4 3 5 5 5 5 4 3 3
27651 004/300 Flush
27652 005/300 Read block 4211 len 28672
27653 006/300 Read block 3716 len 28672
27654 007/300 Read block 3630 len 12288
27655 008/300 Write block 3078 len 8192 data: 6 6
27656 009/300 Write block 3904 len 8192 data: 5 7
27657 010/300 Read block 443 len 24576
27658 011/300 Write block 3293 len 8192 data: 2 2
27659 012/300 Read block 1156 len 4096
27660 013/300 Write block 3481 len 32768 data: 6 6 8 6 5 6 6 6
27661 014/300 Read block 3583 len 8192
27662 015/300 Flush
27663 016/300 Read block 4197 len 20480
27664 017/300 Write block 1105 len 4096 data: 7
27665 018/300 Flush
27666 019/300 Read block 4633 len 12288
27667 020/300 Write block 4561 len 40960 data: 2 3 3 3 5 5 5 5 5 5
27668 021/300 Read block 4499 len 20480
27669 022/300 Read block 392 len 20480
27670 023/300 Write block 1467 len 28672 data: 4 4 4 4 4 5 5
27671 024/300 Write block 859 len 36864 data: 4 4 3 4 4 3 2 3 2
27672 025/300 Write block 1285 len 12288 data: 4 4 3
27673 026/300 Write block 1995 len 16384 data: 7 6 7 8
27674 027/300 Read block 4952 len 36864
27675 028/300 Write block 4687 len 20480 data: 3 5 5 5 6
27676 029/300 Read block 3784 len 36864
27677 030/300 Write block 2007 len 20480 data: 8 9 7 5 3
27678 031/300 Read block 1104 len 36864
27679 032/300 Flush
27680 033/300 Read block 2501 len 12288
27681 034/300 Write block 129 len 4096 data: 3
27682 035/300 Write block 55 len 8192 data: 3 2
27683 036/300 Read block 1754 len 12288
27684 037/300 Read block 3101 len 4096
27685 038/300 Read block 3824 len 40960
27686 039/300 Read block 3866 len 8192
27687 040/300 Write block 2613 len 32768 data: 3 3 3 3 5 4 4 4
27688 041/300 Write block 1730 len 32768 data: 2 2 2 2 2 2 2 2
27689 042/300 Read block 1584 len 12288
27690 043/300 Read block 3732 len 20480
27691 044/300 Write block 3944 len 16384 data: 5 4 6 8
27692 045/300 Read block 2582 len 16384
27693 046/300 Read block 1107 len 12288
27694 047/300 Write block 1526 len 8192 data: 6 4
27695 048/300 Write block 2100 len 36864 data: 7 6 5 5 5 4 4 3 3
27696 049/300 Write block 132 len 4096 data: 4
27697 050/300 Read block 2124 len 16384
27698 051/300 Read block 368 len 32768
27699 052/300 Write block 3325 len 40960 data: 7 7 7 5 6 5 6 6 8 8
27700 053/300 Write block 1819 len 32768 data: 4 4 5 4 4 4 5 5
27701 054/300 Write block 1597 len 28672 data: 2 3 3 4 5 5 5
27702 055/300 Read block 3755 len 12288
27703 056/300 Read block 4448 len 12288
27704 057/300 Write block 4741 len 32768 data: 7 8 8 7 7 5 7 7
27705 058/300 Read block 4254 len 4096
27706 059/300 Write block 48 len 20480 data: 5 4 3 4 4
27707 060/300 Read block 4574 len 40960
27708 061/300 Flush
27709 062/300 Write block 865 len 12288 data: 3 4 3
27710 063/300 Read block 3253 len 36864
27711 064/300 Write block 2282 len 32768 data: 5 5 5 4 4 4 7 7
27712 065/300 Read block 2106 len 40960
27713 066/300 Flush
27714 067/300 Write block 561 len 12288 data: 7 7 6
27715 068/300 Flush
27716 069/300 Read block 485 len 16384
27717 070/300 Write block 1562 len 16384 data: 3 2 2 2
27718 071/300 Read block 4802 len 32768
27719 072/300 Read block 2226 len 36864
27720 073/300 Write block 4678 len 32768 data: 6 5 5 5 5 5 6 6
27721 074/300 Read block 3536 len 24576
27722 075/300 Read block 4204 len 20480
27723 076/300 Write block 2426 len 20480 data: 4 3 3 4 4
27724 077/300 Read block 1419 len 40960
27725 078/300 Write block 717 len 36864 data: 5 7 6 5 6 6 5 6 7
27726 079/300 Read block 4243 len 12288
27727 080/300 Read block 3241 len 12288
27728 081/300 Flush
27729 082/300 Write block 3187 len 8192 data: 4 4
27730 083/300 Read block 2215 len 32768
27731 084/300 Write block 2744 len 20480 data: 6 6 5 5 5
27732 085/300 Read block 2151 len 24576
27733 086/300 Read block 4539 len 40960
27734 087/300 Read block 3768 len 40960
27735 088/300 Read block 1127 len 32768
27736 089/300 Read block 3464 len 8192
27737 090/300 Read block 4916 len 4096
27738 091/300 Write block 543 len 20480 data: 4 3 2 3 4
27739 092/300 Read block 4201 len 20480
27740 093/300 Flush
27741 094/300 Read block 2106 len 40960
27742 095/300 Read block 3787 len 8192
27743 096/300 Read block 3435 len 36864
27744 097/300 Read block 2642 len 20480
27745 098/300 Write block 1828 len 12288 data: 8 8 7
27746 099/300 Write block 3957 len 32768 data: 5 5 5 5 4 4 5 5
27747 100/300 Read block 1014 len 24576
27748 101/300 Flush
27749 102/300 Flush
27750 103/300 Flush
27751 104/300 Read block 2466 len 8192
27752 105/300 Read block 953 len 20480
27753 106/300 Read block 862 len 28672
27754 107/300 Read block 3746 len 28672
27755 108/300 Read block 2086 len 4096
27756 109/300 Read block 4810 len 8192
27757 110/300 Flush
27758 111/300 Write block 3966 len 20480 data: 4 4 2 3 3
27759 112/300 Write block 165 len 12288 data: 5 4 5
27760 113/300 Read block 4906 len 12288
27761 114/300 Read block 2888 len 20480
27762 115/300 Read block 3890 len 28672
27763 116/300 Write block 3026 len 12288 data: 3 3 2
27764 117/300 Flush
27765 118/300 Write block 4075 len 40960 data: 3 3 3 3 3 2 2 4 5 5
27766 119/300 Flush
27767 120/300 Write block 1019 len 12288 data: 8 7 8
27768 121/300 Flush
27769 122/300 Read block 3071 len 12288
27770 123/300 Read block 2206 len 20480
27771 124/300 Read block 4389 len 8192
27772 125/300 Flush
27773 126/300 Read block 1533 len 20480
27774 127/300 Write block 3631 len 32768 data: 2 2 2 3 4 4 4 4
27775 128/300 Read block 1949 len 24576
27776 129/300 Read block 728 len 32768
27777 130/300 Write block 3191 len 4096 data: 5
27778 131/300 Write block 1365 len 20480 data: 4 3 3 4 3
27779 132/300 Write block 1936 len 24576 data: 5 4 5 4 4 4
27780 133/300 Write block 1984 len 16384 data: 5 4 4 6
27781 134/300 Write block 1205 len 12288 data: 4 4 4
27782 135/300 Read block 4542 len 40960
27783 136/300 Read block 324 len 8192
27784 137/300 Write block 3609 len 4096 data: 7
27785 138/300 Flush
27786 139/300 Read block 823 len 28672
27787 140/300 Write block 1071 len 28672 data: 6 8 6 7 6 5 6
27788 141/300 Write block 3098 len 4096 data: 4
27789 142/300 Read block 4144 len 16384
27790 143/300 Read block 3268 len 40960
27791 144/300 Flush
27792 145/300 Read block 152 len 36864
27793 146/300 Write block 3912 len 12288 data: 9 7 7
27794 147/300 Flush
27795 148/300 Read block 1215 len 40960
27796 149/300 Read block 3493 len 4096
27797 150/300 Read block 966 len 28672
27798 151/300 Read block 2254 len 16384
27799 152/300 Write block 1847 len 24576 data: 6 7 7 6 6 4
27800 153/300 Flush
27801 154/300 Read block 1207 len 12288
27802 155/300 Read block 597 len 28672
27803 156/300 Read block 1180 len 36864
27804 157/300 Read block 3421 len 32768
27805 158/300 Write block 459 len 32768 data: 5 6 5 6 6 6 4 6
27806 159/300 Read block 3354 len 28672
27807 160/300 Flush
27808 161/300 Write block 2811 len 24576 data: 6 5 6 6 5 5
27809 162/300 Write block 3784 len 40960 data: 3 3 4 5 5 5 5 5 6 6
27810 163/300 Write block 1699 len 24576 data: 5 5 6 6 5 5
27811 164/300 Write block 3552 len 36864 data: 2 2 2 2 4 2 2 2 2
27812 165/300 Write block 2326 len 12288 data: 4 4 4
27813 166/300 Read block 1638 len 24576
27814 167/300 Write block 514 len 8192 data: 4 4
27815 168/300 Flush
27816 169/300 Read block 4624 len 32768
27817 170/300 Read block 4965 len 40960
27818 171/300 Write block 2886 len 32768 data: 4 5 4 4 2 2 3 3
27819 172/300 Read block 1987 len 12288
27820 173/300 Read block 169 len 40960
27821 174/300 Write block 4982 len 20480 data: 6 5 5 3 4
27822 175/300 Read block 1882 len 40960
27823 176/300 Write block 4253 len 16384 data: 5 4 4 4
27824 177/300 Flush
27825 178/300 Write block 4054 len 20480 data: 5 4 3 3 3
27826 179/300 Write block 2950 len 32768 data: 3 4 6 7 5 5 6 5
27827 180/300 Write block 1473 len 20480 data: 6 6 5 5 5
27828 181/300 Read block 2897 len 16384
27829 182/300 Write block 99 len 8192 data: 5 5
27830 183/300 Read block 2737 len 20480
27831 184/300 Write block 2301 len 4096 data: 3
27832 185/300 Read block 3243 len 16384
27833 186/300 Read block 1346 len 16384
27834 187/300 Read block 1587 len 16384
27835 188/300 Read block 3031 len 16384
27836 189/300 Read block 1099 len 8192
27837 190/300 Write block 3418 len 12288 data: 4 4 4
27838 191/300 Read block 2932 len 36864
27839 192/300 Write block 1010 len 20480 data: 3 4 3 4 5
27840 193/300 Write block 301 len 4096 data: 5
27841 194/300 Write block 2774 len 40960 data: 6 5 5 5 5 4 4 4 5 3
27842 195/300 Read block 2773 len 16384
27843 196/300 Write block 2118 len 28672 data: 5 4 6 6 5 6 6
27844 197/300 Read block 4795 len 8192
27845 198/300 Read block 4464 len 40960
27846 199/300 Read block 4210 len 16384
27847 200/300 Write block 2838 len 20480 data: 5 6 7 6 4
27848 201/300 Write block 2195 len 40960 data: 5 5 5 7 7 7 7 6 7 7
27849 202/300 Read block 4775 len 36864
27850 203/300 Read block 970 len 8192
27851 204/300 Read block 4527 len 28672
27852 205/300 Write block 934 len 40960 data: 4 3 3 3 2 3 3 3 4 5
27853 206/300 Write block 3394 len 32768 data: 8 7 7 8 9 8 5 5
27854 207/300 Write block 4543 len 28672 data: 3 3 4 4 3 3 3
27855 208/300 Read block 4135 len 20480
27856 209/300 Read block 4524 len 12288
27857 210/300 Read block 1242 len 12288
27858 211/300 Read block 1079 len 16384
27859 212/300 Read block 583 len 24576
27860 213/300 Flush
27861 214/300 Write block 1566 len 40960 data: 3 2 2 3 3 3 3 3 4 4
27862 215/300 Read block 3258 len 32768
27863 216/300 Read block 1976 len 4096
27864 217/300 Read block 1287 len 8192
27865 218/300 Write block 2658 len 8192 data: 2 2
27866 219/300 Flush
27867 220/300 Flush
27868 221/300 Write block 916 len 36864 data: 3 3 2 2 4 3 3 3 3
27869 222/300 Write block 3950 len 12288 data: 8 6 6
27870 223/300 Read block 9 len 24576
27871 224/300 Read block 2166 len 12288
27872 225/300 Flush
27873 226/300 Write block 4841 len 36864 data: 6 5 5 6 5 4 4 6 7
27874 227/300 Write block 4796 len 20480 data: 4 4 4 5 6
27875 228/300 Read block 1212 len 36864
27876 229/300 Read block 3877 len 36864
27877 230/300 Flush
27878 231/300 Flush
27879 232/300 Write block 4061 len 32768 data: 10 8 7 6 6 6 6 6
27880 233/300 Flush
27881 234/300 Write block 1364 len 12288 data: 4 5 4
27882 235/300 Write block 3824 len 28672 data: 5 5 4 4 4 3 3
27883 236/300 Write block 22 len 4096 data: 6
27884 237/300 Write block 4127 len 20480 data: 5 4 4 5 6
27885 238/300 Read block 4944 len 4096
27886 239/300 Write block 211 len 28672 data: 5 5 5 5 5 7 7
27887 240/300 Read block 4905 len 16384
27888 241/300 Flush
27889 242/300 Write block 1026 len 16384 data: 5 6 6 6
27890 243/300 Write block 3962 len 12288 data: 5 6 6
27891 244/300 Write block 2742 len 24576 data: 6 5 7 7 6 6
27892 245/300 Write block 2236 len 12288 data: 4 5 5
27893 246/300 Write block 4071 len 32768 data: 5 3 4 4 4 4 4 4
27894 247/300 Read block 420 len 28672
27895 248/300 Write block 1223 len 36864 data: 10 10 8 7 8 6 7 7 6
27896 249/300 Read block 4128 len 16384
27897 250/300 Read block 722 len 8192
27898 251/300 Read block 4624 len 40960
27899 252/300 Read block 2044 len 24576
27900 253/300 Read block 1310 len 24576
27901 254/300 Write block 1426 len 36864 data: 3 2 2 2 2 2 2 4 6
27902 255/300 Read block 1761 len 12288
27903 256/300 Write block 4323 len 24576 data: 6 6 7 8 6 6
27904 257/300 Write block 259 len 4096 data: 5
27905 258/300 Flush
27906 259/300 Write block 773 len 12288 data: 6 6 5
27907 260/300 Write block 2294 len 20480 data: 4 4 4 5 5
27908 261/300 Write block 960 len 28672 data: 5 6 5 5 4 5 8
27909 262/300 Write block 546 len 28672 data: 4 5 4 3 4 4 3
27910 263/300 Read block 4504 len 32768
27911 264/300 Read block 1862 len 20480
27912 265/300 Write block 4617 len 24576 data: 5 6 6 9 7 8
27913 266/300 Read block 2461 len 28672
27914 267/300 Write block 3807 len 24576 data: 6 4 5 5 6 7
27915 268/300 Read block 332 len 12288
27916 269/300 Read block 2336 len 32768
27917 270/300 Write block 3237 len 8192 data: 3 5
27918 271/300 Flush
27919 272/300 Write block 1007 len 12288 data: 5 4 3
27920 273/300 Write block 2600 len 36864 data: 7 9 6 7 7 7 5 6 5
27921 274/300 Write block 500 len 4096 data: 7
27922 275/300 Read block 3114 len 32768
27923 276/300 Read block 3486 len 24576
27924 277/300 Write block 4641 len 20480 data: 7 6 6 4 4
27925 278/300 Read block 892 len 36864
27926 279/300 Flush
27927 280/300 Read block 3077 len 24576
27928 281/300 Write block 813 len 32768 data: 9 7 7 6 4 5 5 4
27929 282/300 Write block 1500 len 32768 data: 3 4 4 5 5 6 8 7
27930 283/300 Write block 948 len 16384 data: 3 3 3 3
27931 284/300 Read block 1501 len 16384
27932 285/300 Flush
27933 286/300 Write block 2967 len 28672 data: 4 5 5 5 5 4 4
27934 287/300 Flush
27935 288/300 Flush
27936 289/300 Read block 2039 len 16384
27937 290/300 Write block 4772 len 32768 data: 3 6 5 5 5 5 4 4
27938 291/300 Read block 1417 len 24576
27939 292/300 Write block 3705 len 16384 data: 7 6 6 9
27940 293/300 Read block 2927 len 20480
27941 294/300 Flush
27942 295/300 Read block 1014 len 4096
27943 296/300 Read block 3814 len 4096
27944 297/300 Write block 738 len 28672 data: 4 4 3 3 3 3 4
27945 298/300 Read block 999 len 28672
27946 299/300 Read block 1896 len 36864
27947 300/300 Write block 2348 len 4096 data: 6
27948 301/300 Read block 3674 len 40960
27949 Main task has work to do, go find it
27950 got dsc Start(0)
27951 start 0
27952 Feb 07 20:13:35.285 INFO request completed, latency_us: 239, response_code: 204, uri: /start/cid/0, method: POST, req_id: 5b211d4d-7088-4562-8471-41b6830e3518, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
27953 Replay: started 0, returned:Ok(())
27954 ----------------------------------------------------------------
27955 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27956 States: Offline Active Active
27957 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27958 7643 Acked Write 6 Sent Done Done false
27959 7644 Acked Flush 0 Sent Done Done false
27960 7645 Acked Write 10 Sent Done Done false
27961 7646 Acked Flush 0 Sent Done Done false
27962 7647 Acked Read 7 Sent Done Done false
27963 7648 Acked Read 7 Sent Done Done false
27964 7649 Acked Read 3 Sent Done Done false
27965 7650 Acked Write 2 Sent Done Done false
27966 7651 Acked Write 2 Sent Done Done false
27967 7652 Acked Read 6 Sent Done Done false
27968 7653 Acked Write 2 Sent Done Done false
27969 7654 Acked Read 1 Sent Done Done false
27970 7655 Acked Write 8 Sent Done Done false
27971 7656 Acked Read 2 Sent Done Done false
27972 7657 Acked Flush 0 Sent Done Done false
27973 7658 Acked Read 5 Sent Done Done false
27974 7659 Acked Write 1 Sent Done Done false
27975 7660 Acked Flush 0 Sent Done Done false
27976 7661 Acked Read 3 Sent Done Done false
27977 7662 Acked Write 10 Sent Done Done false
27978 7663 Acked Read 5 Sent Done Done false
27979 7664 Acked Read 5 Sent Done Done false
27980 7665 Acked Write 7 Sent Done Done false
27981 7666 Acked Write 9 Sent Done Done false
27982 7667 Acked Write 3 Sent Done Done false
27983 7668 Acked Write 4 Sent Done Done false
27984 7669 Acked Read 9 Sent Done Done false
27985 7670 Acked Write 5 Sent Done Done false
27986 7671 Acked Read 9 Sent Done Done false
27987 7672 Acked Write 5 Sent Done Done false
27988 7673 Acked Read 9 Sent Done Done false
27989 7674 Acked Flush 0 Sent Done Done false
27990 7675 Acked Read 3 Sent Done Done false
27991 7676 Acked Write 1 Sent Done Done false
27992 7677 Acked Write 2 Sent Done Done false
27993 7678 Acked Read 3 Sent Done Done false
27994 7679 Acked Read 1 Sent Done Done false
27995 7680 Acked Read 10 Sent Done Done false
27996 7681 Acked Read 2 Sent Done Done false
27997 7682 Acked Write 8 Sent Done Done false
27998 7683 Acked Write 8 Sent Done Done false
27999 7684 Acked Read 3 Sent Done Done false
28000 7685 Acked Read 5 Sent Done Done false
28001 7686 Acked Write 4 Sent Done Done false
28002 7687 Acked Read 4 Sent Done Done false
28003 7688 Acked Read 3 Sent Done Done false
28004 7689 Acked Write 2 Sent Done Done false
28005 7690 Acked Write 9 Sent Done Done false
28006 7691 Acked Write 1 Sent Done Done false
28007 7692 Acked Read 4 Sent Done Done false
28008 7693 Acked Read 8 Sent Done Done false
28009 7694 Acked Write 10 Sent Done Done false
28010 7695 Acked Write 8 Sent Done Done false
28011 7696 Acked Write 7 Sent Done Done false
28012 7697 Acked Read 3 Sent Done Done false
28013 7698 Acked Read 3 Sent Done Done false
28014 7699 Acked Write 8 Sent Done Done false
28015 7700 Acked Read 1 Sent Done Done false
28016 7701 Acked Write 5 Sent Done Done false
28017 7702 Acked Read 10 Sent Done Done false
28018 7703 Acked Flush 0 Sent Done Done false
28019 7704 Acked Write 3 Sent Done Done false
28020 7705 Acked Read 9 Sent Done Done false
28021 7706 Acked Write 8 Sent Done Done false
28022 7707 Acked Read 10 Sent Done Done false
28023 7708 Acked Flush 0 Sent Done Done false
28024 7709 Acked Write 3 Sent Done Done false
28025 7710 Acked Flush 0 Sent Done Done false
28026 7711 Acked Read 4 Sent Done Done false
28027 7712 Acked Write 4 Sent Done Done false
28028 7713 Acked Read 8 Sent Done Done false
28029 7714 Acked Read 9 Sent Done Done false
28030 7715 Acked Write 8 Sent Done Done false
28031 7716 Acked Read 6 Sent Done Done false
28032 7717 Acked Read 5 Sent Done Done false
28033 7718 Acked Write 5 Sent Done Done false
28034 7719 Acked Read 10 Sent Done Done false
28035 7720 Acked Write 9 Sent Done Done false
28036 7721 Acked Read 3 Sent Done Done false
28037 7722 Acked Read 3 Sent Done Done false
28038 7723 Acked Flush 0 Sent Done Done false
28039 7724 Acked Write 2 Sent Done Done false
28040 7725 Acked Read 8 Sent Done Done false
28041 7726 Acked Write 5 Sent Done Done false
28042 7727 Acked Read 6 Sent Done Done false
28043 7728 Acked Read 10 Sent Done Done false
28044 7729 Acked Read 10 Sent Done Done false
28045 7730 Acked Read 8 Sent Done Done false
28046 7731 Acked Read 2 Sent Done Done false
28047 7732 Acked Read 1 Sent Done Done false
28048 7733 Acked Write 5 Sent Done Done false
28049 7734 Acked Read 5 Sent Done Done false
28050 7735 Acked Flush 0 Sent Done Done false
28051 7736 Acked Read 10 Sent Done Done false
28052 7737 Acked Read 2 Sent Done Done false
28053 7738 Acked Read 9 Sent Done Done false
28054 7739 Acked Read 5 Sent Done Done false
28055 7740 Acked Write 3 Sent Done Done false
28056 7741 Acked Write 8 Sent Done Done false
28057 7742 Acked Read 6 Sent Done Done false
28058 7743 Acked Flush 0 Sent Done Done false
28059 7744 Acked Flush 0 Sent Done Done false
28060 7745 Acked Flush 0 Sent Done Done false
28061 7746 Acked Read 2 Sent Done Done false
28062 7747 Acked Read 5 Sent Done Done false
28063 7748 Acked Read 7 Sent Done Done false
28064 7749 Acked Read 7 Sent Done Done false
28065 7750 Acked Read 1 Sent Done Done false
28066 7751 Acked Read 2 Sent Done Done false
28067 7752 Acked Flush 0 Sent Done Done false
28068 7753 Acked Write 5 Sent Done Done false
28069 7754 Acked Write 3 Sent Done Done false
28070 7755 Acked Read 3 Sent Done Done false
28071 7756 Acked Read 5 Sent Done Done false
28072 7757 Acked Read 7 Sent Done Done false
28073 7758 Acked Write 3 Sent Done Done false
28074 7759 Acked Flush 0 Sent Done Done false
28075 7760 Acked Write 10 Sent Done Done false
28076 7761 Acked Flush 0 Sent Done Done false
28077 7762 Acked Write 3 Sent Done Done false
28078 7763 Acked Flush 0 Sent Done Done false
28079 7764 Acked Read 3 Sent Done Done false
28080 7765 Acked Read 5 Sent Done Done false
28081 7766 Acked Read 2 Sent Done Done false
28082 7767 Acked Flush 0 Sent Done Done false
28083 7768 Acked Read 5 Sent Done Done false
28084 7769 Acked Write 8 Sent Done Done false
28085 7770 Acked Read 6 Sent Done Done false
28086 7771 Acked Read 8 Sent Done Done false
28087 7772 Acked Write 1 Sent Done Done false
28088 7773 Acked Write 5 Sent Done Done false
28089 7774 Acked Write 6 Sent Done Done false
28090 7775 Acked Write 4 Sent Done Done false
28091 7776 Acked Write 3 Sent Done Done false
28092 7777 Acked Read 10 Sent Done Done false
28093 7778 Acked Read 2 Sent Done Done false
28094 7779 Acked Write 1 Sent Done Done false
28095 7780 Acked Flush 0 Sent Done Done false
28096 7781 Acked Read 7 Sent Done Done false
28097 7782 Acked Write 7 Sent Done Done false
28098 7783 Acked Write 1 Sent Done Done false
28099 7784 Acked Read 4 Sent Done Done false
28100 7785 Acked Read 10 Sent Done Done false
28101 7786 Acked Flush 0 Sent Done Done false
28102 7787 Acked Read 9 Sent Done Done false
28103 7788 Acked Write 3 Sent Done Done false
28104 7789 Acked Flush 0 Sent Done Done false
28105 7790 Acked Read 10 Sent Done Done false
28106 7791 Acked Read 1 Sent Done Done false
28107 7792 Acked Read 7 Sent Done Done false
28108 7793 Acked Read 4 Sent Done Done false
28109 7794 Acked Write 6 Sent Done Done false
28110 7795 Acked Flush 0 Sent Done Done false
28111 7796 Acked Read 3 Sent Done Done false
28112 7797 Acked Read 7 Sent Done Done false
28113 7798 Acked Read 9 Sent Done Done false
28114 7799 Acked Read 8 Sent Done Done false
28115 7800 Acked Write 8 Sent Done Done false
28116 7801 Acked Read 7 Sent Done Done false
28117 7802 Acked Flush 0 Sent Done Done false
28118 7803 Acked Write 6 Sent Done Done false
28119 7804 Acked Write 10 Sent Done Done false
28120 7805 Acked Write 6 Sent Done Done false
28121 7806 Acked Write 9 Sent Done Done false
28122 7807 Acked Write 3 Sent Done Done false
28123 7808 Acked Read 6 Sent Done Done false
28124 7809 Acked Write 2 Sent Done Done false
28125 7810 Acked Flush 0 Sent Done Done false
28126 7811 Acked Read 8 Sent Done Done false
28127 7812 Acked Read 10 Sent Done Done false
28128 7813 Acked Write 8 Sent Done Done false
28129 7814 Acked Read 3 Sent Done Done false
28130 7815 Acked Read 10 Sent Done Done false
28131 7816 Acked Write 5 Sent Done Done false
28132 7817 Acked Read 10 Sent Done Done false
28133 7818 Acked Write 4 Sent Done Done false
28134 7819 Acked Flush 0 Sent Done Done false
28135 7820 Acked Write 5 Sent Done Done false
28136 7821 Acked Write 8 Sent Done Done false
28137 7822 Acked Write 5 Sent Done Done false
28138 7823 Acked Read 4 Sent Done Done false
28139 7824 Acked Write 2 Sent Done Done false
28140 7825 Acked Read 5 Sent Done Done false
28141 7826 Acked Write 1 Sent Done Done false
28142 7827 Acked Read 4 Sent Done Done false
28143 7828 Acked Read 4 Sent Done Done false
28144 7829 Acked Read 4 Sent Done Done false
28145 7830 Acked Read 4 Sent Done Done false
28146 7831 Acked Read 2 Sent Done Done false
28147 7832 Acked Write 3 Sent Done Done false
28148 7833 Acked Read 9 Sent Done Done false
28149 7834 Acked Write 5 Sent Done Done false
28150 7835 Acked Write 1 Sent Done Done false
28151 7836 Acked Write 10 Sent Done Done false
28152 7837 Acked Read 4 Sent Done Done false
28153 7838 Acked Write 7 Sent Done Done false
28154 7839 Acked Read 2 Sent Done Done false
28155 7840 Acked Read 10 Sent Done Done false
28156 7841 Acked Read 4 Sent Done Done false
28157 7842 Acked Write 5 Sent Done Done false
28158 7843 Acked Write 10 Sent Done Done false
28159 7844 Acked Read 9 Sent Done Done false
28160 7845 Acked Read 2 Sent Done Done false
28161 7846 Acked Read 7 Sent Done Done false
28162 7847 Acked Write 10 Sent Done Done false
28163 7848 Acked Write 8 Sent Done Done false
28164 7849 Acked Write 7 Sent Done Done false
28165 7850 Acked Read 5 Sent Done Done false
28166 7851 Acked Read 3 Sent Done Done false
28167 7852 Acked Read 3 Sent Done Done false
28168 7853 Acked Read 4 Sent Done Done false
28169 7854 Acked Read 6 Sent Done Done false
28170 7855 Acked Flush 0 Sent Done Done false
28171 7856 Acked Write 10 Sent Done Done false
28172 7857 Acked Read 8 Sent Done Done false
28173 7858 Acked Read 1 Sent Done Done false
28174 7859 Acked Read 2 Sent Done Done false
28175 7860 Acked Write 2 Sent Done Done false
28176 7861 Acked Flush 0 Sent Done Done false
28177 7862 Acked Flush 0 Sent Done Done false
28178 7863 Acked Write 9 Sent Done Done false
28179 7864 Acked Write 3 Sent Done Done false
28180 7865 Acked Read 6 Sent Done Done false
28181 7866 Acked Read 3 Sent Done Done false
28182 7867 Acked Flush 0 Sent Done Done false
28183 7868 Acked Write 9 Sent Done Done false
28184 7869 Acked Write 5 Sent Done Done false
28185 7870 Acked Read 9 Sent Done Done false
28186 7871 Acked Read 9 Sent Done Done false
28187 7872 Acked Flush 0 Sent Done Done false
28188 7873 Acked Flush 0 Sent Done Done false
28189 7874 Acked Write 8 Sent Done Done false
28190 7875 Acked Flush 0 Sent Done Done false
28191 7876 Acked Write 3 Sent Done Done false
28192 7877 Acked Write 7 Sent Done Done false
28193 7878 Acked Write 1 Sent Done Done false
28194 7879 Acked Write 5 Sent Done Done false
28195 7880 Acked Read 1 Sent Done Done false
28196 7881 Acked Write 7 Sent Done Done false
28197 7882 Acked Read 4 Sent Done Done false
28198 7883 Acked Flush 0 Sent Done Done false
28199 7884 Acked Write 4 Sent Done Done false
28200 7885 Acked Write 3 Sent Done Done false
28201 7886 Acked Write 6 Sent Done Done false
28202 7887 Acked Write 3 Sent Done Done false
28203 7888 Acked Write 8 Sent Done Done false
28204 7889 Acked Read 7 Sent Done Done false
28205 7890 Acked Write 9 Sent Done Done false
28206 7891 Acked Read 4 Sent Done Done false
28207 7892 Acked Read 2 Sent Done Done false
28208 7893 Acked Read 10 Sent Done Done false
28209 7894 Acked Read 6 Sent Done Done false
28210 7895 Acked Read 6 Sent Done Done false
28211 7896 Acked Write 9 Sent Done Done false
28212 7897 Acked Read 3 Sent Done Done false
28213 7898 Acked Write 6 Sent Done Done false
28214 7899 Acked Write 1 Sent Done Done false
28215 7900 Acked Flush 0 Sent Done Done false
28216 7901 Acked Write 3 Sent Done Done false
28217 7902 Acked Write 5 Sent Done Done false
28218 7903 Acked Write 7 Sent Done Done false
28219 7904 Acked Write 7 Sent Done Done false
28220 7905 Acked Read 8 Sent Done Done false
28221 7906 Acked Read 5 Sent Done Done false
28222 7907 Acked Write 6 Sent Done Done false
28223 7908 Acked Read 7 Sent Done Done false
28224 7909 Acked Write 6 Sent Done Done false
28225 7910 Acked Read 3 Sent Done Done false
28226 7911 Acked Read 8 Sent Done Done false
28227 7912 Acked Write 2 Sent Done Done false
28228 7913 Acked Flush 0 Sent Done Done false
28229 7914 Acked Write 3 Sent Done Done false
28230 7915 Acked Write 9 Sent Done Done false
28231 7916 Acked Write 1 Sent Done Done false
28232 7917 Acked Read 8 Sent Done Done false
28233 7918 Acked Read 6 Sent Done Done false
28234 7919 Acked Write 5 Sent Done Done false
28235 7920 Acked Read 9 Sent Done Done false
28236 7921 Acked Flush 0 Sent Done Done false
28237 7922 Acked Read 6 Sent Done Done false
28238 7923 Acked Write 8 Sent Done Done false
28239 7924 Acked Write 8 Sent Done Done false
28240 7925 Acked Write 4 Sent Done Done false
28241 7926 Acked Read 4 Sent Done Done false
28242 7927 Acked Flush 0 Sent Done Done false
28243 7928 Acked Write 7 Sent Done Done false
28244 7929 Acked Flush 0 Sent Done Done false
28245 7930 Acked Flush 0 Sent Done Done false
28246 7931 Acked Read 4 Sent Done Done false
28247 7932 Acked Write 8 Sent Done Done false
28248 7933 Acked Read 6 Sent Done Done false
28249 7934 Acked Write 4 Sent Done Done false
28250 7935 Acked Read 5 Sent Done Done false
28251 7936 Acked Flush 0 Sent Done Done false
28252 7937 Acked Read 1 Sent Done Done false
28253 7938 Acked Read 1 Sent Done Done false
28254 7939 Acked Write 7 Sent Done Done false
28255 7940 Acked Read 7 Sent Done Done false
28256 7941 Acked Read 9 Sent Done Done false
28257 7942 Acked Write 1 Sent Done Done false
28258 7943 Acked Read 10 Sent Done Done false
28259 STATES DS:0 DS:1 DS:2 TOTAL
28260 Sent 301 0 0 301
28261 Done 0 301 301 602
28262 Skipped 0 0 0 0
28263 Error 0 0 0 0
28264 Last Flush: Some(JobId(7642)) Some(JobId(7936)) Some(JobId(7936))
28265 Downstairs last five completed: 7642 7641 7640 7639 7638
28266 Upstairs last five completed: 7943 7942 7941 7940 7939
28267 CLIENT: Up:0 ds:301 act:2
282682025-02-07T20:13:35.787ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
282692025-02-07T20:13:37.024ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
28270 [8810] got start action
28271 [8810] I am going to restart
28272 Starting downstairs at port 8810
28273 Make output file at "/tmp/dsc/downstairs-8810.txt"
28274 [8810][0] reports Starting
28275 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1232)
28276 [8810][0] reports Running
282772025-02-07T20:13:37.561ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
282782025-02-07T20:13:38.561ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
28279 ----------------------------------------------------------------
28280 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28281 States: Offline Active Active
28282 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28283 7643 Acked Write 6 Sent Done Done false
28284 7644 Acked Flush 0 Sent Done Done false
28285 7645 Acked Write 10 Sent Done Done false
28286 7646 Acked Flush 0 Sent Done Done false
28287 7647 Acked Read 7 Sent Done Done false
28288 7648 Acked Read 7 Sent Done Done false
28289 7649 Acked Read 3 Sent Done Done false
28290 7650 Acked Write 2 Sent Done Done false
28291 7651 Acked Write 2 Sent Done Done false
28292 7652 Acked Read 6 Sent Done Done false
28293 7653 Acked Write 2 Sent Done Done false
28294 7654 Acked Read 1 Sent Done Done false
28295 7655 Acked Write 8 Sent Done Done false
28296 7656 Acked Read 2 Sent Done Done false
28297 7657 Acked Flush 0 Sent Done Done false
28298 7658 Acked Read 5 Sent Done Done false
28299 7659 Acked Write 1 Sent Done Done false
28300 7660 Acked Flush 0 Sent Done Done false
28301 7661 Acked Read 3 Sent Done Done false
28302 7662 Acked Write 10 Sent Done Done false
28303 7663 Acked Read 5 Sent Done Done false
28304 7664 Acked Read 5 Sent Done Done false
28305 7665 Acked Write 7 Sent Done Done false
28306 7666 Acked Write 9 Sent Done Done false
28307 7667 Acked Write 3 Sent Done Done false
28308 7668 Acked Write 4 Sent Done Done false
28309 7669 Acked Read 9 Sent Done Done false
28310 7670 Acked Write 5 Sent Done Done false
28311 7671 Acked Read 9 Sent Done Done false
28312 7672 Acked Write 5 Sent Done Done false
28313 7673 Acked Read 9 Sent Done Done false
28314 7674 Acked Flush 0 Sent Done Done false
28315 7675 Acked Read 3 Sent Done Done false
28316 7676 Acked Write 1 Sent Done Done false
28317 7677 Acked Write 2 Sent Done Done false
28318 7678 Acked Read 3 Sent Done Done false
28319 7679 Acked Read 1 Sent Done Done false
28320 7680 Acked Read 10 Sent Done Done false
28321 7681 Acked Read 2 Sent Done Done false
28322 7682 Acked Write 8 Sent Done Done false
28323 7683 Acked Write 8 Sent Done Done false
28324 7684 Acked Read 3 Sent Done Done false
28325 7685 Acked Read 5 Sent Done Done false
28326 7686 Acked Write 4 Sent Done Done false
28327 7687 Acked Read 4 Sent Done Done false
28328 7688 Acked Read 3 Sent Done Done false
28329 7689 Acked Write 2 Sent Done Done false
28330 7690 Acked Write 9 Sent Done Done false
28331 7691 Acked Write 1 Sent Done Done false
28332 7692 Acked Read 4 Sent Done Done false
28333 7693 Acked Read 8 Sent Done Done false
28334 7694 Acked Write 10 Sent Done Done false
28335 7695 Acked Write 8 Sent Done Done false
28336 7696 Acked Write 7 Sent Done Done false
28337 7697 Acked Read 3 Sent Done Done false
28338 7698 Acked Read 3 Sent Done Done false
28339 7699 Acked Write 8 Sent Done Done false
28340 7700 Acked Read 1 Sent Done Done false
28341 7701 Acked Write 5 Sent Done Done false
28342 7702 Acked Read 10 Sent Done Done false
28343 7703 Acked Flush 0 Sent Done Done false
28344 7704 Acked Write 3 Sent Done Done false
28345 7705 Acked Read 9 Sent Done Done false
28346 7706 Acked Write 8 Sent Done Done false
28347 7707 Acked Read 10 Sent Done Done false
28348 7708 Acked Flush 0 Sent Done Done false
28349 7709 Acked Write 3 Sent Done Done false
28350 7710 Acked Flush 0 Sent Done Done false
28351 7711 Acked Read 4 Sent Done Done false
28352 7712 Acked Write 4 Sent Done Done false
28353 7713 Acked Read 8 Sent Done Done false
28354 7714 Acked Read 9 Sent Done Done false
28355 7715 Acked Write 8 Sent Done Done false
28356 7716 Acked Read 6 Sent Done Done false
28357 7717 Acked Read 5 Sent Done Done false
28358 7718 Acked Write 5 Sent Done Done false
28359 7719 Acked Read 10 Sent Done Done false
28360 7720 Acked Write 9 Sent Done Done false
28361 7721 Acked Read 3 Sent Done Done false
28362 7722 Acked Read 3 Sent Done Done false
28363 7723 Acked Flush 0 Sent Done Done false
28364 7724 Acked Write 2 Sent Done Done false
28365 7725 Acked Read 8 Sent Done Done false
28366 7726 Acked Write 5 Sent Done Done false
28367 7727 Acked Read 6 Sent Done Done false
28368 7728 Acked Read 10 Sent Done Done false
28369 7729 Acked Read 10 Sent Done Done false
28370 7730 Acked Read 8 Sent Done Done false
28371 7731 Acked Read 2 Sent Done Done false
28372 7732 Acked Read 1 Sent Done Done false
28373 7733 Acked Write 5 Sent Done Done false
28374 7734 Acked Read 5 Sent Done Done false
28375 7735 Acked Flush 0 Sent Done Done false
28376 7736 Acked Read 10 Sent Done Done false
28377 7737 Acked Read 2 Sent Done Done false
28378 7738 Acked Read 9 Sent Done Done false
28379 7739 Acked Read 5 Sent Done Done false
28380 7740 Acked Write 3 Sent Done Done false
28381 7741 Acked Write 8 Sent Done Done false
28382 7742 Acked Read 6 Sent Done Done false
28383 7743 Acked Flush 0 Sent Done Done false
28384 7744 Acked Flush 0 Sent Done Done false
28385 7745 Acked Flush 0 Sent Done Done false
28386 7746 Acked Read 2 Sent Done Done false
28387 7747 Acked Read 5 Sent Done Done false
28388 7748 Acked Read 7 Sent Done Done false
28389 7749 Acked Read 7 Sent Done Done false
28390 7750 Acked Read 1 Sent Done Done false
28391 7751 Acked Read 2 Sent Done Done false
28392 7752 Acked Flush 0 Sent Done Done false
28393 7753 Acked Write 5 Sent Done Done false
28394 7754 Acked Write 3 Sent Done Done false
28395 7755 Acked Read 3 Sent Done Done false
28396 7756 Acked Read 5 Sent Done Done false
28397 7757 Acked Read 7 Sent Done Done false
28398 7758 Acked Write 3 Sent Done Done false
28399 7759 Acked Flush 0 Sent Done Done false
28400 7760 Acked Write 10 Sent Done Done false
28401 7761 Acked Flush 0 Sent Done Done false
28402 7762 Acked Write 3 Sent Done Done false
28403 7763 Acked Flush 0 Sent Done Done false
28404 7764 Acked Read 3 Sent Done Done false
28405 7765 Acked Read 5 Sent Done Done false
28406 7766 Acked Read 2 Sent Done Done false
28407 7767 Acked Flush 0 Sent Done Done false
28408 7768 Acked Read 5 Sent Done Done false
28409 7769 Acked Write 8 Sent Done Done false
28410 7770 Acked Read 6 Sent Done Done false
28411 7771 Acked Read 8 Sent Done Done false
28412 7772 Acked Write 1 Sent Done Done false
28413 7773 Acked Write 5 Sent Done Done false
28414 7774 Acked Write 6 Sent Done Done false
28415 7775 Acked Write 4 Sent Done Done false
28416 7776 Acked Write 3 Sent Done Done false
28417 7777 Acked Read 10 Sent Done Done false
28418 7778 Acked Read 2 Sent Done Done false
28419 7779 Acked Write 1 Sent Done Done false
28420 7780 Acked Flush 0 Sent Done Done false
28421 7781 Acked Read 7 Sent Done Done false
28422 7782 Acked Write 7 Sent Done Done false
28423 7783 Acked Write 1 Sent Done Done false
28424 7784 Acked Read 4 Sent Done Done false
28425 7785 Acked Read 10 Sent Done Done false
28426 7786 Acked Flush 0 Sent Done Done false
28427 7787 Acked Read 9 Sent Done Done false
28428 7788 Acked Write 3 Sent Done Done false
28429 7789 Acked Flush 0 Sent Done Done false
28430 7790 Acked Read 10 Sent Done Done false
28431 7791 Acked Read 1 Sent Done Done false
28432 7792 Acked Read 7 Sent Done Done false
28433 7793 Acked Read 4 Sent Done Done false
28434 7794 Acked Write 6 Sent Done Done false
28435 7795 Acked Flush 0 Sent Done Done false
28436 7796 Acked Read 3 Sent Done Done false
28437 7797 Acked Read 7 Sent Done Done false
28438 7798 Acked Read 9 Sent Done Done false
28439 7799 Acked Read 8 Sent Done Done false
28440 7800 Acked Write 8 Sent Done Done false
28441 7801 Acked Read 7 Sent Done Done false
28442 7802 Acked Flush 0 Sent Done Done false
28443 7803 Acked Write 6 Sent Done Done false
28444 7804 Acked Write 10 Sent Done Done false
28445 7805 Acked Write 6 Sent Done Done false
28446 7806 Acked Write 9 Sent Done Done false
28447 7807 Acked Write 3 Sent Done Done false
28448 7808 Acked Read 6 Sent Done Done false
28449 7809 Acked Write 2 Sent Done Done false
28450 7810 Acked Flush 0 Sent Done Done false
28451 7811 Acked Read 8 Sent Done Done false
28452 7812 Acked Read 10 Sent Done Done false
28453 7813 Acked Write 8 Sent Done Done false
28454 7814 Acked Read 3 Sent Done Done false
28455 7815 Acked Read 10 Sent Done Done false
28456 7816 Acked Write 5 Sent Done Done false
28457 7817 Acked Read 10 Sent Done Done false
28458 7818 Acked Write 4 Sent Done Done false
28459 7819 Acked Flush 0 Sent Done Done false
28460 7820 Acked Write 5 Sent Done Done false
28461 7821 Acked Write 8 Sent Done Done false
28462 7822 Acked Write 5 Sent Done Done false
28463 7823 Acked Read 4 Sent Done Done false
28464 7824 Acked Write 2 Sent Done Done false
28465 7825 Acked Read 5 Sent Done Done false
28466 7826 Acked Write 1 Sent Done Done false
28467 7827 Acked Read 4 Sent Done Done false
28468 7828 Acked Read 4 Sent Done Done false
28469 7829 Acked Read 4 Sent Done Done false
28470 7830 Acked Read 4 Sent Done Done false
28471 7831 Acked Read 2 Sent Done Done false
28472 7832 Acked Write 3 Sent Done Done false
28473 7833 Acked Read 9 Sent Done Done false
28474 7834 Acked Write 5 Sent Done Done false
28475 7835 Acked Write 1 Sent Done Done false
28476 7836 Acked Write 10 Sent Done Done false
28477 7837 Acked Read 4 Sent Done Done false
28478 7838 Acked Write 7 Sent Done Done false
28479 7839 Acked Read 2 Sent Done Done false
28480 7840 Acked Read 10 Sent Done Done false
28481 7841 Acked Read 4 Sent Done Done false
28482 7842 Acked Write 5 Sent Done Done false
28483 7843 Acked Write 10 Sent Done Done false
28484 7844 Acked Read 9 Sent Done Done false
28485 7845 Acked Read 2 Sent Done Done false
28486 7846 Acked Read 7 Sent Done Done false
28487 7847 Acked Write 10 Sent Done Done false
28488 7848 Acked Write 8 Sent Done Done false
28489 7849 Acked Write 7 Sent Done Done false
28490 7850 Acked Read 5 Sent Done Done false
28491 7851 Acked Read 3 Sent Done Done false
28492 7852 Acked Read 3 Sent Done Done false
28493 7853 Acked Read 4 Sent Done Done false
28494 7854 Acked Read 6 Sent Done Done false
28495 7855 Acked Flush 0 Sent Done Done false
28496 7856 Acked Write 10 Sent Done Done false
28497 7857 Acked Read 8 Sent Done Done false
28498 7858 Acked Read 1 Sent Done Done false
28499 7859 Acked Read 2 Sent Done Done false
28500 7860 Acked Write 2 Sent Done Done false
28501 7861 Acked Flush 0 Sent Done Done false
28502 7862 Acked Flush 0 Sent Done Done false
28503 7863 Acked Write 9 Sent Done Done false
28504 7864 Acked Write 3 Sent Done Done false
28505 7865 Acked Read 6 Sent Done Done false
28506 7866 Acked Read 3 Sent Done Done false
28507 7867 Acked Flush 0 Sent Done Done false
28508 7868 Acked Write 9 Sent Done Done false
28509 7869 Acked Write 5 Sent Done Done false
28510 7870 Acked Read 9 Sent Done Done false
28511 7871 Acked Read 9 Sent Done Done false
28512 7872 Acked Flush 0 Sent Done Done false
28513 7873 Acked Flush 0 Sent Done Done false
28514 7874 Acked Write 8 Sent Done Done false
28515 7875 Acked Flush 0 Sent Done Done false
28516 7876 Acked Write 3 Sent Done Done false
28517 7877 Acked Write 7 Sent Done Done false
28518 7878 Acked Write 1 Sent Done Done false
28519 7879 Acked Write 5 Sent Done Done false
28520 7880 Acked Read 1 Sent Done Done false
28521 7881 Acked Write 7 Sent Done Done false
28522 7882 Acked Read 4 Sent Done Done false
28523 7883 Acked Flush 0 Sent Done Done false
28524 7884 Acked Write 4 Sent Done Done false
28525 7885 Acked Write 3 Sent Done Done false
28526 7886 Acked Write 6 Sent Done Done false
28527 7887 Acked Write 3 Sent Done Done false
28528 7888 Acked Write 8 Sent Done Done false
28529 7889 Acked Read 7 Sent Done Done false
28530 7890 Acked Write 9 Sent Done Done false
28531 7891 Acked Read 4 Sent Done Done false
28532 7892 Acked Read 2 Sent Done Done false
28533 7893 Acked Read 10 Sent Done Done false
28534 7894 Acked Read 6 Sent Done Done false
28535 7895 Acked Read 6 Sent Done Done false
28536 7896 Acked Write 9 Sent Done Done false
28537 7897 Acked Read 3 Sent Done Done false
28538 7898 Acked Write 6 Sent Done Done false
28539 7899 Acked Write 1 Sent Done Done false
28540 7900 Acked Flush 0 Sent Done Done false
28541 7901 Acked Write 3 Sent Done Done false
28542 7902 Acked Write 5 Sent Done Done false
28543 7903 Acked Write 7 Sent Done Done false
28544 7904 Acked Write 7 Sent Done Done false
28545 7905 Acked Read 8 Sent Done Done false
28546 7906 Acked Read 5 Sent Done Done false
28547 7907 Acked Write 6 Sent Done Done false
28548 7908 Acked Read 7 Sent Done Done false
28549 7909 Acked Write 6 Sent Done Done false
28550 7910 Acked Read 3 Sent Done Done false
28551 7911 Acked Read 8 Sent Done Done false
28552 7912 Acked Write 2 Sent Done Done false
28553 7913 Acked Flush 0 Sent Done Done false
28554 7914 Acked Write 3 Sent Done Done false
28555 7915 Acked Write 9 Sent Done Done false
28556 7916 Acked Write 1 Sent Done Done false
28557 7917 Acked Read 8 Sent Done Done false
28558 7918 Acked Read 6 Sent Done Done false
28559 7919 Acked Write 5 Sent Done Done false
28560 7920 Acked Read 9 Sent Done Done false
28561 7921 Acked Flush 0 Sent Done Done false
28562 7922 Acked Read 6 Sent Done Done false
28563 7923 Acked Write 8 Sent Done Done false
28564 7924 Acked Write 8 Sent Done Done false
28565 7925 Acked Write 4 Sent Done Done false
28566 7926 Acked Read 4 Sent Done Done false
28567 7927 Acked Flush 0 Sent Done Done false
28568 7928 Acked Write 7 Sent Done Done false
28569 7929 Acked Flush 0 Sent Done Done false
28570 7930 Acked Flush 0 Sent Done Done false
28571 7931 Acked Read 4 Sent Done Done false
28572 7932 Acked Write 8 Sent Done Done false
28573 7933 Acked Read 6 Sent Done Done false
28574 7934 Acked Write 4 Sent Done Done false
28575 7935 Acked Read 5 Sent Done Done false
28576 7936 Acked Flush 0 Sent Done Done false
28577 7937 Acked Read 1 Sent Done Done false
28578 7938 Acked Read 1 Sent Done Done false
28579 7939 Acked Write 7 Sent Done Done false
28580 7940 Acked Read 7 Sent Done Done false
28581 7941 Acked Read 9 Sent Done Done false
28582 7942 Acked Write 1 Sent Done Done false
28583 7943 Acked Read 10 Sent Done Done false
28584 STATES DS:0 DS:1 DS:2 TOTAL
28585 Sent 301 0 0 301
28586 Done 0 301 301 602
28587 Skipped 0 0 0 0
28588 Error 0 0 0 0
28589 Last Flush: Some(JobId(7642)) Some(JobId(7936)) Some(JobId(7936))
28590 Downstairs last five completed: 7642 7641 7640 7639 7638
28591 Upstairs last five completed: 7943 7942 7941 7940 7939
28592 CLIENT: Up:0 ds:301 act:2
285932025-02-07T20:13:39.789ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
285942025-02-07T20:13:40.565ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
285952025-02-07T20:13:41.044ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
285962025-02-07T20:13:41.044ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37514) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
285972025-02-07T20:13:41.045ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
285982025-02-07T20:13:41.045ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
285992025-02-07T20:13:41.045ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
286002025-02-07T20:13:41.045ZINFOcrucible: send last flush ID to this DS: Some(JobId(7642)) = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
286012025-02-07T20:13:41.045ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
286022025-02-07T20:13:41.045ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(7642)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
286032025-02-07T20:13:41.046ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(7642)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
28604 ----------------------------------------------------------------
28605 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28606 Downstairs last five completed: 7944 7943 7942 7941 7940
28607 Upstairs last five completed: 7944 7943 7942 7941 7940
28608 CLIENT: Up:0 ds:0 act:3
28609 Replay: All jobs finished
28610 Main task has work to do, go find it
28611 got dsc Stop(2)
28612 stop 2
28613 Feb 07 20:13:43.290 [8830] Got stop action so:false kr:false
28614 INFO request completed, latency_us: 381, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 949e0451-366f-4670-ae0a-26cd06152e2c, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
286152025-02-07T20:13:43.290ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
286162025-02-07T20:13:43.291ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
286172025-02-07T20:13:43.291ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
286182025-02-07T20:13:43.291ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
28619 Feb 07 20:13:43.291 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/2, method: GET, req_id: 64407c2c-5277-4042-adc1-e259943a5366, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
28620 [8830] Exited with: ExitStatus(unix_wait_status(9))
28621 [8830][2] reports Exit
28622 Feb 07 20:13:47.293 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/2, method: GET, req_id: 016e9f78-0543-440f-a206-6412e140181c, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
28623 001/300 Write block 4761 len 32768 data: 5 4 4 4 3 2 2 2
28624 002/300 Write block 1763 len 12288 data: 5 5 4
28625 003/300 Read block 1434 len 36864
28626 004/300 Read block 2919 len 16384
28627 005/300 Write block 1510 len 8192 data: 7 6
28628 006/300 Read block 676 len 16384
28629 007/300 Write block 939 len 24576 data: 4 4 4 5 6 4
28630 008/300 Read block 1492 len 16384
28631 009/300 Read block 3970 len 28672
28632 010/300 Read block 3631 len 4096
28633 011/300 Write block 4642 len 20480 data: 7 7 5 5 5
28634 012/300 Read block 1365 len 8192
28635 013/300 Write block 1362 len 40960 data: 3 3 5 6 5 4 5 4 4 4
28636 014/300 Write block 3002 len 16384 data: 6 6 7 7
28637 015/300 Read block 3884 len 8192
28638 016/300 Read block 237 len 4096
28639 017/300 Read block 266 len 16384
28640 018/300 Write block 1742 len 32768 data: 5 5 4 5 6 6 5 5
28641 019/300 Write block 4750 len 32768 data: 6 7 6 5 5 5 6 5
28642 020/300 Write block 21 len 36864 data: 7 7 7 7 4 3 4 4 5
28643 021/300 Read block 1059 len 28672
28644 022/300 Read block 3232 len 4096
28645 023/300 Read block 4689 len 36864
28646 024/300 Read block 3587 len 40960
28647 025/300 Read block 3962 len 16384
28648 026/300 Read block 734 len 32768
28649 027/300 Write block 3892 len 8192 data: 3 3
28650 028/300 Flush
28651 029/300 Write block 467 len 24576 data: 6 6 5 5 5 7
28652 030/300 Write block 4569 len 4096 data: 6
28653 031/300 Write block 2735 len 32768 data: 6 5 6 5 6 6 5 7
28654 032/300 Write block 811 len 8192 data: 6 7
28655 033/300 Write block 3838 len 8192 data: 6 5
28656 034/300 Write block 1287 len 32768 data: 4 4 3 3 3 4 4 4
28657 035/300 Read block 2657 len 24576
28658 036/300 Read block 3081 len 32768
28659 037/300 Flush
28660 038/300 Read block 4675 len 4096
28661 039/300 Write block 3502 len 4096 data: 4
28662 040/300 Read block 100 len 12288
28663 041/300 Flush
28664 042/300 Read block 2139 len 12288
28665 043/300 Write block 3143 len 4096 data: 6
28666 044/300 Read block 4053 len 32768
28667 045/300 Read block 4687 len 20480
28668 046/300 Write block 2499 len 36864 data: 5 4 5 5 3 3 4 5 6
28669 047/300 Write block 2776 len 40960 data: 6 6 6 5 5 5 6 4 5 3
28670 048/300 Read block 3464 len 24576
28671 049/300 Read block 4086 len 24576
28672 050/300 Write block 4653 len 32768 data: 6 6 6 5 5 3 3 3
28673 051/300 Read block 647 len 32768
28674 052/300 Read block 1451 len 4096
28675 053/300 Write block 4335 len 28672 data: 5 4 2 4 5 6 6
28676 054/300 Flush
28677 055/300 Read block 3532 len 28672
28678 056/300 Write block 2713 len 20480 data: 6 5 4 3 4
28679 057/300 Read block 4367 len 40960
28680 058/300 Read block 2135 len 32768
28681 059/300 Read block 2032 len 12288
28682 060/300 Read block 3784 len 4096
28683 061/300 Read block 707 len 8192
28684 062/300 Read block 629 len 20480
28685 063/300 Write block 3976 len 32768 data: 2 2 2 2 2 2 3 3
28686 064/300 Read block 3065 len 4096
28687 065/300 Write block 262 len 20480 data: 5 5 4 4 5
28688 066/300 Read block 1910 len 12288
28689 067/300 Write block 3368 len 36864 data: 5 4 4 4 5 3 4 3 3
28690 068/300 Read block 2287 len 8192
28691 069/300 Read block 2313 len 24576
28692 070/300 Read block 127 len 36864
28693 071/300 Read block 3990 len 4096
28694 072/300 Write block 3101 len 40960 data: 2 3 2 3 5 5 5 5 5 5
28695 073/300 Read block 474 len 4096
28696 074/300 Read block 2286 len 12288
28697 075/300 Write block 224 len 20480 data: 4 5 6 4 2
28698 076/300 Read block 1342 len 16384
28699 077/300 Read block 4433 len 16384
28700 078/300 Read block 2136 len 36864
28701 079/300 Read block 2230 len 24576
28702 080/300 Write block 2285 len 32768 data: 5 5 5 8 8 7 6 6
28703 081/300 Read block 1607 len 4096
28704 082/300 Read block 687 len 16384
28705 083/300 Write block 3483 len 8192 data: 9 7
28706 084/300 Flush
28707 085/300 Write block 300 len 36864 data: 5 6 5 7 7 7 6 5 4
28708 086/300 Write block 775 len 20480 data: 6 5 5 5 4
28709 087/300 Read block 2412 len 36864
28710 088/300 Read block 1245 len 8192
28711 089/300 Read block 2067 len 8192
28712 090/300 Read block 2848 len 12288
28713 091/300 Write block 338 len 36864 data: 8 9 9 4 4 5 2 2 2
28714 092/300 Read block 1540 len 12288
28715 093/300 Read block 2031 len 28672
28716 094/300 Read block 3359 len 36864
28717 095/300 Read block 2032 len 40960
28718 096/300 Write block 2814 len 32768 data: 7 6 6 4 4 3 4 4
28719 097/300 Write block 2248 len 8192 data: 4 4
28720 098/300 Write block 490 len 28672 data: 3 3 3 4 4 3 4
28721 099/300 Write block 2093 len 28672 data: 6 6 6 7 6 7 7
28722 100/300 Write block 4705 len 32768 data: 7 7 7 7 7 6 5 4
28723 101/300 Write block 3813 len 16384 data: 7 5 7 8
28724 102/300 Flush
28725 103/300 Write block 398 len 4096 data: 2
28726 104/300 Read block 1173 len 36864
28727 105/300 Read block 1368 len 40960
28728 106/300 Write block 1629 len 8192 data: 3 4
28729 107/300 Read block 4361 len 16384
28730 108/300 Read block 1831 len 8192
28731 109/300 Read block 755 len 24576
28732 110/300 Write block 4450 len 8192 data: 4 4
28733 111/300 Read block 1127 len 20480
28734 112/300 Read block 4055 len 16384
28735 113/300 Read block 4598 len 24576
28736 114/300 Read block 3556 len 36864
28737 115/300 Read block 602 len 16384
28738 116/300 Write block 1192 len 32768 data: 5 4 3 6 5 4 4 4
28739 117/300 Flush
28740 118/300 Write block 3658 len 36864 data: 6 8 9 7 8 7 7 7 9
28741 119/300 Read block 740 len 32768
28742 120/300 Write block 3513 len 20480 data: 3 3 3 3 2
28743 121/300 Write block 1889 len 16384 data: 4 4 4 5
28744 122/300 Read block 4128 len 36864
28745 123/300 Write block 280 len 20480 data: 7 5 4 3 3
28746 124/300 Read block 4835 len 36864
28747 125/300 Read block 2637 len 16384
28748 126/300 Write block 1324 len 28672 data: 7 7 6 8 6 7 8
28749 127/300 Read block 2263 len 16384
28750 128/300 Read block 4456 len 16384
28751 129/300 Read block 840 len 40960
28752 130/300 Read block 1227 len 36864
28753 131/300 Read block 1550 len 28672
28754 132/300 Write block 4371 len 20480 data: 3 4 5 6 7
28755 133/300 Flush
28756 134/300 Read block 506 len 4096
28757 135/300 Read block 680 len 40960
28758 136/300 Read block 4713 len 32768
28759 137/300 Write block 1316 len 32768 data: 7 6 6 5 7 8 8 7
28760 138/300 Read block 763 len 24576
28761 139/300 Read block 684 len 16384
28762 140/300 Read block 4293 len 36864
28763 141/300 Read block 2757 len 16384
28764 142/300 Write block 2660 len 8192 data: 3 5
28765 143/300 Write block 3557 len 12288 data: 3 3 3
28766 144/300 Read block 500 len 36864
28767 145/300 Read block 3845 len 16384
28768 146/300 Flush
28769 147/300 Read block 3449 len 20480
28770 148/300 Write block 2276 len 4096 data: 5
28771 149/300 Read block 4037 len 28672
28772 150/300 Write block 4172 len 28672 data: 5 5 5 5 6 5 5
28773 151/300 Read block 2976 len 20480
28774 152/300 Read block 947 len 16384
28775 153/300 Read block 711 len 24576
28776 154/300 Read block 2083 len 4096
28777 155/300 Read block 3404 len 16384
28778 156/300 Read block 1094 len 32768
28779 157/300 Flush
28780 158/300 Read block 3460 len 40960
28781 159/300 Flush
28782 160/300 Read block 1743 len 24576
28783 161/300 Write block 4059 len 36864 data: 5 8 11 9 8 7 7 7 7
28784 162/300 Read block 1458 len 40960
28785 163/300 Read block 2916 len 36864
28786 164/300 Read block 570 len 20480
28787 165/300 Read block 3815 len 4096
28788 166/300 Write block 1337 len 12288 data: 6 5 4
28789 167/300 Read block 4312 len 12288
28790 168/300 Read block 2263 len 40960
28791 169/300 Flush
28792 170/300 Write block 4187 len 24576 data: 4 4 8 7 8 9
28793 171/300 Read block 4296 len 24576
28794 172/300 Write block 2308 len 24576 data: 6 6 6 8 8 7
28795 173/300 Write block 4754 len 28672 data: 6 6 7 6 3 3 3
28796 174/300 Read block 465 len 16384
28797 175/300 Read block 4134 len 12288
28798 176/300 Write block 2543 len 40960 data: 5 3 3 3 2 2 2 2 3 3
28799 177/300 Read block 1964 len 8192
28800 178/300 Flush
28801 179/300 Write block 3280 len 28672 data: 4 4 3 5 3 3 3
28802 180/300 Read block 4016 len 36864
28803 181/300 Write block 1055 len 8192 data: 5 6
28804 182/300 Write block 810 len 8192 data: 6 7
28805 183/300 Read block 2550 len 32768
28806 184/300 Read block 1917 len 12288
28807 185/300 Read block 3264 len 28672
28808 186/300 Write block 2087 len 16384 data: 6 4 5 6
28809 187/300 Write block 3740 len 8192 data: 4 4
28810 188/300 Write block 587 len 20480 data: 5 4 4 2 3
28811 189/300 Read block 4954 len 32768
28812 190/300 Write block 2126 len 4096 data: 6
28813 191/300 Read block 2204 len 8192
28814 192/300 Read block 2751 len 16384
28815 193/300 Read block 4259 len 20480
28816 194/300 Read block 713 len 36864
28817 195/300 Write block 147 len 12288 data: 2 3 3
28818 196/300 Read block 4969 len 24576
28819 197/300 Write block 49 len 36864 data: 5 4 5 5 4 3 4 3 4
28820 198/300 Write block 1634 len 40960 data: 5 5 4 4 3 5 4 5 4 4
28821 199/300 Read block 337 len 4096
28822 200/300 Read block 1812 len 36864
28823 201/300 Write block 2307 len 4096 data: 3
28824 202/300 Read block 2362 len 12288
28825 203/300 Write block 277 len 36864 data: 5 7 7 8 6 5 4 4 4
28826 204/300 Read block 1341 len 40960
28827 205/300 Read block 1974 len 8192
28828 206/300 Write block 2048 len 32768 data: 4 4 4 4 4 6 5 5
28829 207/300 Write block 4580 len 20480 data: 4 5 5 5 5
28830 208/300 Read block 3783 len 28672
28831 209/300 Write block 3309 len 28672 data: 5 4 6 5 5 5 5
28832 210/300 Read block 1759 len 8192
28833 211/300 Write block 1438 len 20480 data: 7 8 10 7 6
28834 212/300 Write block 2459 len 40960 data: 5 6 5 3 5 6 7 4 4 4
28835 213/300 Write block 3683 len 32768 data: 9 10 8 9 8 7 6 5
28836 214/300 Write block 636 len 12288 data: 4 4 4
28837 215/300 Read block 2152 len 4096
28838 216/300 Read block 3108 len 16384
28839 217/300 Read block 1334 len 20480
28840 218/300 Read block 4231 len 8192
28841 219/300 Write block 1000 len 28672 data: 8 10 9 6 6 5 6
28842 220/300 Read block 4606 len 20480
28843 221/300 Write block 2111 len 24576 data: 3 3 3 5 5 5
28844 222/300 Write block 1694 len 12288 data: 8 9 9
28845 223/300 Write block 4588 len 16384 data: 6 7 5 5
28846 224/300 Write block 2396 len 4096 data: 7
28847 225/300 Write block 210 len 36864 data: 6 6 6 6 6 6 8 8 6
28848 226/300 Read block 1995 len 24576
28849 227/300 Read block 4664 len 32768
28850 228/300 Read block 4719 len 4096
28851 229/300 Read block 2815 len 8192
28852 230/300 Read block 2446 len 28672
28853 231/300 Flush
28854 232/300 Flush
28855 233/300 Read block 1746 len 36864
28856 234/300 Read block 4923 len 36864
28857 235/300 Write block 248 len 8192 data: 5 4
28858 236/300 Write block 820 len 40960 data: 5 4 6 6 7 8 7 6 8 9
28859 237/300 Write block 3460 len 20480 data: 4 2 2 3 3
28860 238/300 Read block 3560 len 32768
28861 239/300 Read block 426 len 28672
28862 240/300 Read block 3887 len 28672
28863 241/300 Read block 644 len 4096
28864 242/300 Write block 1658 len 20480 data: 8 6 5 5 5
28865 243/300 Write block 714 len 16384 data: 5 5 5 6
28866 244/300 Read block 2511 len 40960
28867 245/300 Write block 728 len 4096 data: 3
28868 246/300 Write block 972 len 4096 data: 3
28869 247/300 Write block 812 len 20480 data: 8 10 8 8 7
28870 248/300 Flush
28871 249/300 Read block 1428 len 24576
28872 250/300 Read block 39 len 40960
28873 251/300 Write block 1936 len 24576 data: 6 5 6 5 5 5
28874 252/300 Write block 1481 len 40960 data: 3 2 2 3 3 4 6 6 6 6
28875 253/300 Read block 2704 len 20480
28876 254/300 Flush
28877 255/300 Read block 3454 len 12288
28878 256/300 Write block 2882 len 20480 data: 4 5 4 4 5
28879 257/300 Flush
28880 258/300 Read block 1262 len 12288
28881 259/300 Read block 635 len 36864
28882 260/300 Read block 55 len 20480
28883 261/300 Write block 2734 len 28672 data: 3 7 6 7 6 7 7
28884 262/300 Write block 1288 len 28672 data: 5 4 4 4 5 5 5
28885 263/300 Write block 1620 len 16384 data: 5 5 5 5
28886 264/300 Read block 4187 len 28672
28887 265/300 Write block 2580 len 8192 data: 6 7
28888 266/300 Write block 266 len 28672 data: 6 4 4 6 6 5 4
28889 267/300 Read block 782 len 4096
28890 268/300 Read block 2799 len 20480
28891 269/300 Read block 2091 len 8192
28892 270/300 Write block 2948 len 8192 data: 4 4
28893 271/300 Read block 2348 len 12288
28894 272/300 Read block 2757 len 12288
28895 273/300 Read block 4282 len 40960
28896 274/300 Flush
28897 275/300 Read block 1973 len 20480
28898 276/300 Write block 1077 len 28672 data: 7 7 5 5 4 4 5
28899 277/300 Read block 3805 len 8192
28900 278/300 Write block 2342 len 16384 data: 9 7 7 8
28901 279/300 Write block 2267 len 40960 data: 7 6 7 7 8 7 7 7 7 6
28902 280/300 Read block 4605 len 36864
28903 281/300 Write block 3938 len 24576 data: 3 3 4 4 3 4
28904 282/300 Write block 4433 len 28672 data: 3 4 4 5 6 6 7
28905 283/300 Read block 2684 len 28672
28906 284/300 Write block 3626 len 36864 data: 3 3 4 4 5 3 3 3 4
28907 285/300 Read block 2729 len 12288
28908 286/300 Read block 4895 len 8192
28909 287/300 Read block 360 len 16384
28910 288/300 Read block 813 len 16384
28911 289/300 Read block 4539 len 16384
28912 290/300 Write block 3734 len 4096 data: 6
28913 291/300 Write block 4830 len 8192 data: 9 10
28914 292/300 Read block 589 len 36864
28915 293/300 Write block 3570 len 20480 data: 7 8 7 7 6
28916 294/300 Read block 4394 len 28672
28917 295/300 Read block 3608 len 8192
28918 296/300 Read block 4584 len 12288
28919 297/300 Write block 2024 len 4096 data: 8
28920 298/300 Read block 1512 len 16384
28921 299/300 Read block 1274 len 8192
28922 300/300 Write block 2119 len 20480 data: 5 7 7 6 7
28923 301/300 Write block 1303 len 28672 data: 7 6 5 3 3 3 3
28924 Main task has work to do, go find it
28925 got dsc Start(2)
28926 start 2
28927 Feb 07 20:13:47.540 INFO request completed, latency_us: 232, response_code: 204, uri: /start/cid/2, method: POST, req_id: 74151627-95c4-49a7-b9e1-da1819ce494a, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
28928 Replay: started 2, returned:Ok(())
28929 ----------------------------------------------------------------
28930 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28931 States: Active Active Offline
28932 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28933 7945 Acked Write 8 Done Done Sent false
28934 7946 Acked Write 3 Done Done Sent false
28935 7947 Acked Read 9 Done Done Sent false
28936 7948 Acked Read 4 Done Done Sent false
28937 7949 Acked Write 2 Done Done Sent false
28938 7950 Acked Read 4 Done Done Sent false
28939 7951 Acked Write 6 Done Done Sent false
28940 7952 Acked Read 4 Done Done Sent false
28941 7953 Acked Read 7 Done Done Sent false
28942 7954 Acked Read 1 Done Done Sent false
28943 7955 Acked Write 5 Done Done Sent false
28944 7956 Acked Read 2 Done Done Sent false
28945 7957 Acked Write 10 Done Done Sent false
28946 7958 Acked Write 4 Done Done Sent false
28947 7959 Acked Read 2 Done Done Sent false
28948 7960 Acked Read 1 Done Done Sent false
28949 7961 Acked Read 4 Done Done Sent false
28950 7962 Acked Write 8 Done Done Sent false
28951 7963 Acked Write 8 Done Done Sent false
28952 7964 Acked Write 9 Done Done Sent false
28953 7965 Acked Read 7 Done Done Sent false
28954 7966 Acked Read 1 Done Done Sent false
28955 7967 Acked Read 9 Done Done Sent false
28956 7968 Acked Read 10 Done Done Sent false
28957 7969 Acked Read 4 Done Done Sent false
28958 7970 Acked Read 8 Done Done Sent false
28959 7971 Acked Write 2 Done Done Sent false
28960 7972 Acked Flush 0 Done Done Sent false
28961 7973 Acked Write 6 Done Done Sent false
28962 7974 Acked Write 1 Done Done Sent false
28963 7975 Acked Write 8 Done Done Sent false
28964 7976 Acked Write 2 Done Done Sent false
28965 7977 Acked Write 2 Done Done Sent false
28966 7978 Acked Write 8 Done Done Sent false
28967 7979 Acked Read 6 Done Done Sent false
28968 7980 Acked Read 8 Done Done Sent false
28969 7981 Acked Flush 0 Done Done Sent false
28970 7982 Acked Read 1 Done Done Sent false
28971 7983 Acked Write 1 Done Done Sent false
28972 7984 Acked Read 3 Done Done Sent false
28973 7985 Acked Flush 0 Done Done Sent false
28974 7986 Acked Read 3 Done Done Sent false
28975 7987 Acked Write 1 Done Done Sent false
28976 7988 Acked Read 8 Done Done Sent false
28977 7989 Acked Read 5 Done Done Sent false
28978 7990 Acked Write 9 Done Done Sent false
28979 7991 Acked Write 10 Done Done Sent false
28980 7992 Acked Read 6 Done Done Sent false
28981 7993 Acked Read 6 Done Done Sent false
28982 7994 Acked Write 8 Done Done Sent false
28983 7995 Acked Read 8 Done Done Sent false
28984 7996 Acked Read 1 Done Done Sent false
28985 7997 Acked Write 7 Done Done Sent false
28986 7998 Acked Flush 0 Done Done Sent false
28987 7999 Acked Read 7 Done Done Sent false
28988 8000 Acked Write 5 Done Done Sent false
28989 8001 Acked Read 10 Done Done Sent false
28990 8002 Acked Read 8 Done Done Sent false
28991 8003 Acked Read 3 Done Done Sent false
28992 8004 Acked Read 1 Done Done Sent false
28993 8005 Acked Read 2 Done Done Sent false
28994 8006 Acked Read 5 Done Done Sent false
28995 8007 Acked Write 8 Done Done Sent false
28996 8008 Acked Read 1 Done Done Sent false
28997 8009 Acked Write 5 Done Done Sent false
28998 8010 Acked Read 3 Done Done Sent false
28999 8011 Acked Write 9 Done Done Sent false
29000 8012 Acked Read 2 Done Done Sent false
29001 8013 Acked Read 6 Done Done Sent false
29002 8014 Acked Read 9 Done Done Sent false
29003 8015 Acked Read 1 Done Done Sent false
29004 8016 Acked Write 10 Done Done Sent false
29005 8017 Acked Read 1 Done Done Sent false
29006 8018 Acked Read 3 Done Done Sent false
29007 8019 Acked Write 5 Done Done Sent false
29008 8020 Acked Read 4 Done Done Sent false
29009 8021 Acked Read 4 Done Done Sent false
29010 8022 Acked Read 9 Done Done Sent false
29011 8023 Acked Read 6 Done Done Sent false
29012 8024 Acked Write 8 Done Done Sent false
29013 8025 Acked Read 1 Done Done Sent false
29014 8026 Acked Read 4 Done Done Sent false
29015 8027 Acked Write 2 Done Done Sent false
29016 8028 Acked Flush 0 Done Done Sent false
29017 8029 Acked Write 9 Done Done Sent false
29018 8030 Acked Write 5 Done Done Sent false
29019 8031 Acked Read 9 Done Done Sent false
29020 8032 Acked Read 2 Done Done Sent false
29021 8033 Acked Read 2 Done Done Sent false
29022 8034 Acked Read 3 Done Done Sent false
29023 8035 Acked Write 9 Done Done Sent false
29024 8036 Acked Read 3 Done Done Sent false
29025 8037 Acked Read 7 Done Done Sent false
29026 8038 Acked Read 9 Done Done Sent false
29027 8039 Acked Read 10 Done Done Sent false
29028 8040 Acked Write 8 Done Done Sent false
29029 8041 Acked Write 2 Done Done Sent false
29030 8042 Acked Write 7 Done Done Sent false
29031 8043 Acked Write 7 Done Done Sent false
29032 8044 Acked Write 8 Done Done Sent false
29033 8045 Acked Write 4 Done Done Sent false
29034 8046 Acked Flush 0 Done Done Sent false
29035 8047 Acked Write 1 Done Done Sent false
29036 8048 Acked Read 9 Done Done Sent false
29037 8049 Acked Read 10 Done Done Sent false
29038 8050 Acked Write 2 Done Done Sent false
29039 8051 Acked Read 4 Done Done Sent false
29040 8052 Acked Read 2 Done Done Sent false
29041 8053 Acked Read 6 Done Done Sent false
29042 8054 Acked Write 2 Done Done Sent false
29043 8055 Acked Read 5 Done Done Sent false
29044 8056 Acked Read 4 Done Done Sent false
29045 8057 Acked Read 6 Done Done Sent false
29046 8058 Acked Read 9 Done Done Sent false
29047 8059 Acked Read 4 Done Done Sent false
29048 8060 Acked Write 8 Done Done Sent false
29049 8061 Acked Flush 0 Done Done Sent false
29050 8062 Acked Write 9 Done Done Sent false
29051 8063 Acked Read 8 Done Done Sent false
29052 8064 Acked Write 5 Done Done Sent false
29053 8065 Acked Write 4 Done Done Sent false
29054 8066 Acked Read 9 Done Done Sent false
29055 8067 Acked Write 5 Done Done Sent false
29056 8068 Acked Read 9 Done Done Sent false
29057 8069 Acked Read 4 Done Done Sent false
29058 8070 Acked Write 7 Done Done Sent false
29059 8071 Acked Read 4 Done Done Sent false
29060 8072 Acked Read 4 Done Done Sent false
29061 8073 Acked Read 10 Done Done Sent false
29062 8074 Acked Read 9 Done Done Sent false
29063 8075 Acked Read 7 Done Done Sent false
29064 8076 Acked Write 5 Done Done Sent false
29065 8077 Acked Flush 0 Done Done Sent false
29066 8078 Acked Read 1 Done Done Sent false
29067 8079 Acked Read 10 Done Done Sent false
29068 8080 Acked Read 8 Done Done Sent false
29069 8081 Acked Write 8 Done Done Sent false
29070 8082 Acked Read 6 Done Done Sent false
29071 8083 Acked Read 4 Done Done Sent false
29072 8084 Acked Read 9 Done Done Sent false
29073 8085 Acked Read 4 Done Done Sent false
29074 8086 Acked Write 2 Done Done Sent false
29075 8087 Acked Write 3 Done Done Sent false
29076 8088 Acked Read 9 Done Done Sent false
29077 8089 Acked Read 4 Done Done Sent false
29078 8090 Acked Flush 0 Done Done Sent false
29079 8091 Acked Read 5 Done Done Sent false
29080 8092 Acked Write 1 Done Done Sent false
29081 8093 Acked Read 7 Done Done Sent false
29082 8094 Acked Write 7 Done Done Sent false
29083 8095 Acked Read 5 Done Done Sent false
29084 8096 Acked Read 4 Done Done Sent false
29085 8097 Acked Read 6 Done Done Sent false
29086 8098 Acked Read 1 Done Done Sent false
29087 8099 Acked Read 4 Done Done Sent false
29088 8100 Acked Read 8 Done Done Sent false
29089 8101 Acked Flush 0 Done Done Sent false
29090 8102 Acked Read 10 Done Done Sent false
29091 8103 Acked Flush 0 Done Done Sent false
29092 8104 Acked Read 6 Done Done Sent false
29093 8105 Acked Write 9 Done Done Sent false
29094 8106 Acked Read 10 Done Done Sent false
29095 8107 Acked Read 9 Done Done Sent false
29096 8108 Acked Read 5 Done Done Sent false
29097 8109 Acked Read 1 Done Done Sent false
29098 8110 Acked Write 3 Done Done Sent false
29099 8111 Acked Read 3 Done Done Sent false
29100 8112 Acked Read 10 Done Done Sent false
29101 8113 Acked Flush 0 Done Done Sent false
29102 8114 Acked Write 6 Done Done Sent false
29103 8115 Acked Read 6 Done Done Sent false
29104 8116 Acked Write 6 Done Done Sent false
29105 8117 Acked Write 7 Done Done Sent false
29106 8118 Acked Read 4 Done Done Sent false
29107 8119 Acked Read 3 Done Done Sent false
29108 8120 Acked Write 10 Done Done Sent false
29109 8121 Acked Read 2 Done Done Sent false
29110 8122 Acked Flush 0 Done Done Sent false
29111 8123 Acked Write 7 Done Done Sent false
29112 8124 Acked Read 9 Done Done Sent false
29113 8125 Acked Write 2 Done Done Sent false
29114 8126 Acked Write 2 Done Done Sent false
29115 8127 Acked Read 8 Done Done Sent false
29116 8128 Acked Read 3 Done Done Sent false
29117 8129 Acked Read 7 Done Done Sent false
29118 8130 Acked Write 4 Done Done Sent false
29119 8131 Acked Write 2 Done Done Sent false
29120 8132 Acked Write 5 Done Done Sent false
29121 8133 Acked Read 8 Done Done Sent false
29122 8134 Acked Write 1 Done Done Sent false
29123 8135 Acked Read 2 Done Done Sent false
29124 8136 Acked Read 4 Done Done Sent false
29125 8137 Acked Read 5 Done Done Sent false
29126 8138 Acked Read 9 Done Done Sent false
29127 8139 Acked Write 3 Done Done Sent false
29128 8140 Acked Read 6 Done Done Sent false
29129 8141 Acked Write 9 Done Done Sent false
29130 8142 Acked Write 10 Done Done Sent false
29131 8143 Acked Read 1 Done Done Sent false
29132 8144 Acked Read 9 Done Done Sent false
29133 8145 Acked Write 1 Done Done Sent false
29134 8146 Acked Read 3 Done Done Sent false
29135 8147 Acked Write 9 Done Done Sent false
29136 8148 Acked Read 10 Done Done Sent false
29137 8149 Acked Read 2 Done Done Sent false
29138 8150 Acked Write 8 Done Done Sent false
29139 8151 Acked Write 5 Done Done Sent false
29140 8152 Acked Read 7 Done Done Sent false
29141 8153 Acked Write 7 Done Done Sent false
29142 8154 Acked Read 2 Done Done Sent false
29143 8155 Acked Write 5 Done Done Sent false
29144 8156 Acked Write 10 Done Done Sent false
29145 8157 Acked Write 8 Done Done Sent false
29146 8158 Acked Write 3 Done Done Sent false
29147 8159 Acked Read 1 Done Done Sent false
29148 8160 Acked Read 4 Done Done Sent false
29149 8161 Acked Read 5 Done Done Sent false
29150 8162 Acked Read 2 Done Done Sent false
29151 8163 Acked Write 7 Done Done Sent false
29152 8164 Acked Read 5 Done Done Sent false
29153 8165 Acked Write 6 Done Done Sent false
29154 8166 Acked Write 3 Done Done Sent false
29155 8167 Acked Write 4 Done Done Sent false
29156 8168 Acked Write 1 Done Done Sent false
29157 8169 Acked Write 9 Done Done Sent false
29158 8170 Acked Read 6 Done Done Sent false
29159 8171 Acked Read 8 Done Done Sent false
29160 8172 Acked Read 1 Done Done Sent false
29161 8173 Acked Read 2 Done Done Sent false
29162 8174 Acked Read 7 Done Done Sent false
29163 8175 Acked Flush 0 Done Done Sent false
29164 8176 Acked Flush 0 Done Done Sent false
29165 8177 Acked Read 9 Done Done Sent false
29166 8178 Acked Read 9 Done Done Sent false
29167 8179 Acked Write 2 Done Done Sent false
29168 8180 Acked Write 10 Done Done Sent false
29169 8181 Acked Write 5 Done Done Sent false
29170 8182 Acked Read 8 Done Done Sent false
29171 8183 Acked Read 7 Done Done Sent false
29172 8184 Acked Read 7 Done Done Sent false
29173 8185 Acked Read 1 Done Done Sent false
29174 8186 Acked Write 5 Done Done Sent false
29175 8187 Acked Write 4 Done Done Sent false
29176 8188 Acked Read 10 Done Done Sent false
29177 8189 Acked Write 1 Done Done Sent false
29178 8190 Acked Write 1 Done Done Sent false
29179 8191 Acked Write 5 Done Done Sent false
29180 8192 Acked Flush 0 Done Done Sent false
29181 8193 Acked Read 6 Done Done Sent false
29182 8194 Acked Read 10 Done Done Sent false
29183 8195 Acked Write 6 Done Done Sent false
29184 8196 Acked Write 10 Done Done Sent false
29185 8197 Acked Read 5 Done Done Sent false
29186 8198 Acked Flush 0 Done Done Sent false
29187 8199 Acked Read 3 Done Done Sent false
29188 8200 Acked Write 5 Done Done Sent false
29189 8201 Acked Flush 0 Done Done Sent false
29190 8202 Acked Read 3 Done Done Sent false
29191 8203 Acked Read 9 Done Done Sent false
29192 8204 Acked Read 5 Done Done Sent false
29193 8205 Acked Write 7 Done Done Sent false
29194 8206 Acked Write 7 Done Done Sent false
29195 8207 Acked Write 4 Done Done Sent false
29196 8208 Acked Read 7 Done Done Sent false
29197 8209 Acked Write 2 Done Done Sent false
29198 8210 Acked Write 7 Done Done Sent false
29199 8211 Acked Read 1 Done Done Sent false
29200 8212 Acked Read 5 Done Done Sent false
29201 8213 Acked Read 2 Done Done Sent false
29202 8214 Acked Write 2 Done Done Sent false
29203 8215 Acked Read 3 Done Done Sent false
29204 8216 Acked Read 3 Done Done Sent false
29205 8217 Acked Read 10 Done Done Sent false
29206 8218 Acked Flush 0 Done Done Sent false
29207 8219 Acked Read 5 Done Done Sent false
29208 8220 Acked Write 7 Done Done Sent false
29209 8221 Acked Read 2 Done Done Sent false
29210 8222 Acked Write 4 Done Done Sent false
29211 8223 Acked Write 10 Done Done Sent false
29212 8224 Acked Read 9 Done Done Sent false
29213 8225 Acked Write 6 Done Done Sent false
29214 8226 Acked Write 7 Done Done Sent false
29215 8227 Acked Read 7 Done Done Sent false
29216 8228 Acked Write 9 Done Done Sent false
29217 8229 Acked Read 3 Done Done Sent false
29218 8230 Acked Read 2 Done Done Sent false
29219 8231 Acked Read 4 Done Done Sent false
29220 8232 Acked Read 4 Done Done Sent false
29221 8233 Acked Read 4 Done Done Sent false
29222 8234 Acked Write 1 Done Done Sent false
29223 8235 Acked Write 2 Done Done Sent false
29224 8236 Acked Read 9 Done Done Sent false
29225 8237 Acked Write 5 Done Done Sent false
29226 8238 Acked Read 7 Done Done Sent false
29227 8239 Acked Read 2 Done Done Sent false
29228 8240 Acked Read 3 Done Done Sent false
29229 8241 Acked Write 1 Done Done Sent false
29230 8242 Acked Read 4 Done Done Sent false
29231 8243 Acked Read 2 Done Done Sent false
29232 8244 Acked Write 5 Done Done Sent false
29233 8245 Acked Write 7 Done Done Sent false
29234 STATES DS:0 DS:1 DS:2 TOTAL
29235 Sent 0 0 301 301
29236 Done 301 301 0 602
29237 Skipped 0 0 0 0
29238 Error 0 0 0 0
29239 Last Flush: Some(JobId(8218)) Some(JobId(8218)) Some(JobId(7944))
29240 Downstairs last five completed: 7944 7943 7942 7941 7940
29241 Upstairs last five completed: 8245 8244 8243 8242 8241
29242 CLIENT: Up:0 ds:301 act:2
292432025-02-07T20:13:48.042ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
292442025-02-07T20:13:48.578ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
29245 [8830] got start action
29246 [8830] I am going to restart
29247 Starting downstairs at port 8830
29248 Make output file at "/tmp/dsc/downstairs-8830.txt"
29249 [8830][2] reports Starting
29250 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1233)
29251 [8830][2] reports Running
292522025-02-07T20:13:49.578ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
292532025-02-07T20:13:50.579ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
29254 ----------------------------------------------------------------
29255 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29256 States: Active Active Offline
29257 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29258 7945 Acked Write 8 Done Done Sent false
29259 7946 Acked Write 3 Done Done Sent false
29260 7947 Acked Read 9 Done Done Sent false
29261 7948 Acked Read 4 Done Done Sent false
29262 7949 Acked Write 2 Done Done Sent false
29263 7950 Acked Read 4 Done Done Sent false
29264 7951 Acked Write 6 Done Done Sent false
29265 7952 Acked Read 4 Done Done Sent false
29266 7953 Acked Read 7 Done Done Sent false
29267 7954 Acked Read 1 Done Done Sent false
29268 7955 Acked Write 5 Done Done Sent false
29269 7956 Acked Read 2 Done Done Sent false
29270 7957 Acked Write 10 Done Done Sent false
29271 7958 Acked Write 4 Done Done Sent false
29272 7959 Acked Read 2 Done Done Sent false
29273 7960 Acked Read 1 Done Done Sent false
29274 7961 Acked Read 4 Done Done Sent false
29275 7962 Acked Write 8 Done Done Sent false
29276 7963 Acked Write 8 Done Done Sent false
29277 7964 Acked Write 9 Done Done Sent false
29278 7965 Acked Read 7 Done Done Sent false
29279 7966 Acked Read 1 Done Done Sent false
29280 7967 Acked Read 9 Done Done Sent false
29281 7968 Acked Read 10 Done Done Sent false
29282 7969 Acked Read 4 Done Done Sent false
29283 7970 Acked Read 8 Done Done Sent false
29284 7971 Acked Write 2 Done Done Sent false
29285 7972 Acked Flush 0 Done Done Sent false
29286 7973 Acked Write 6 Done Done Sent false
29287 7974 Acked Write 1 Done Done Sent false
29288 7975 Acked Write 8 Done Done Sent false
29289 7976 Acked Write 2 Done Done Sent false
29290 7977 Acked Write 2 Done Done Sent false
29291 7978 Acked Write 8 Done Done Sent false
29292 7979 Acked Read 6 Done Done Sent false
29293 7980 Acked Read 8 Done Done Sent false
29294 7981 Acked Flush 0 Done Done Sent false
29295 7982 Acked Read 1 Done Done Sent false
29296 7983 Acked Write 1 Done Done Sent false
29297 7984 Acked Read 3 Done Done Sent false
29298 7985 Acked Flush 0 Done Done Sent false
29299 7986 Acked Read 3 Done Done Sent false
29300 7987 Acked Write 1 Done Done Sent false
29301 7988 Acked Read 8 Done Done Sent false
29302 7989 Acked Read 5 Done Done Sent false
29303 7990 Acked Write 9 Done Done Sent false
29304 7991 Acked Write 10 Done Done Sent false
29305 7992 Acked Read 6 Done Done Sent false
29306 7993 Acked Read 6 Done Done Sent false
29307 7994 Acked Write 8 Done Done Sent false
29308 7995 Acked Read 8 Done Done Sent false
29309 7996 Acked Read 1 Done Done Sent false
29310 7997 Acked Write 7 Done Done Sent false
29311 7998 Acked Flush 0 Done Done Sent false
29312 7999 Acked Read 7 Done Done Sent false
29313 8000 Acked Write 5 Done Done Sent false
29314 8001 Acked Read 10 Done Done Sent false
29315 8002 Acked Read 8 Done Done Sent false
29316 8003 Acked Read 3 Done Done Sent false
29317 8004 Acked Read 1 Done Done Sent false
29318 8005 Acked Read 2 Done Done Sent false
29319 8006 Acked Read 5 Done Done Sent false
29320 8007 Acked Write 8 Done Done Sent false
29321 8008 Acked Read 1 Done Done Sent false
29322 8009 Acked Write 5 Done Done Sent false
29323 8010 Acked Read 3 Done Done Sent false
29324 8011 Acked Write 9 Done Done Sent false
29325 8012 Acked Read 2 Done Done Sent false
29326 8013 Acked Read 6 Done Done Sent false
29327 8014 Acked Read 9 Done Done Sent false
29328 8015 Acked Read 1 Done Done Sent false
29329 8016 Acked Write 10 Done Done Sent false
29330 8017 Acked Read 1 Done Done Sent false
29331 8018 Acked Read 3 Done Done Sent false
29332 8019 Acked Write 5 Done Done Sent false
29333 8020 Acked Read 4 Done Done Sent false
29334 8021 Acked Read 4 Done Done Sent false
29335 8022 Acked Read 9 Done Done Sent false
29336 8023 Acked Read 6 Done Done Sent false
29337 8024 Acked Write 8 Done Done Sent false
29338 8025 Acked Read 1 Done Done Sent false
29339 8026 Acked Read 4 Done Done Sent false
29340 8027 Acked Write 2 Done Done Sent false
29341 8028 Acked Flush 0 Done Done Sent false
29342 8029 Acked Write 9 Done Done Sent false
29343 8030 Acked Write 5 Done Done Sent false
29344 8031 Acked Read 9 Done Done Sent false
29345 8032 Acked Read 2 Done Done Sent false
29346 8033 Acked Read 2 Done Done Sent false
29347 8034 Acked Read 3 Done Done Sent false
29348 8035 Acked Write 9 Done Done Sent false
29349 8036 Acked Read 3 Done Done Sent false
29350 8037 Acked Read 7 Done Done Sent false
29351 8038 Acked Read 9 Done Done Sent false
29352 8039 Acked Read 10 Done Done Sent false
29353 8040 Acked Write 8 Done Done Sent false
29354 8041 Acked Write 2 Done Done Sent false
29355 8042 Acked Write 7 Done Done Sent false
29356 8043 Acked Write 7 Done Done Sent false
29357 8044 Acked Write 8 Done Done Sent false
29358 8045 Acked Write 4 Done Done Sent false
29359 8046 Acked Flush 0 Done Done Sent false
29360 8047 Acked Write 1 Done Done Sent false
29361 8048 Acked Read 9 Done Done Sent false
29362 8049 Acked Read 10 Done Done Sent false
29363 8050 Acked Write 2 Done Done Sent false
29364 8051 Acked Read 4 Done Done Sent false
29365 8052 Acked Read 2 Done Done Sent false
29366 8053 Acked Read 6 Done Done Sent false
29367 8054 Acked Write 2 Done Done Sent false
29368 8055 Acked Read 5 Done Done Sent false
29369 8056 Acked Read 4 Done Done Sent false
29370 8057 Acked Read 6 Done Done Sent false
29371 8058 Acked Read 9 Done Done Sent false
29372 8059 Acked Read 4 Done Done Sent false
29373 8060 Acked Write 8 Done Done Sent false
29374 8061 Acked Flush 0 Done Done Sent false
29375 8062 Acked Write 9 Done Done Sent false
29376 8063 Acked Read 8 Done Done Sent false
29377 8064 Acked Write 5 Done Done Sent false
29378 8065 Acked Write 4 Done Done Sent false
29379 8066 Acked Read 9 Done Done Sent false
29380 8067 Acked Write 5 Done Done Sent false
29381 8068 Acked Read 9 Done Done Sent false
29382 8069 Acked Read 4 Done Done Sent false
29383 8070 Acked Write 7 Done Done Sent false
29384 8071 Acked Read 4 Done Done Sent false
29385 8072 Acked Read 4 Done Done Sent false
29386 8073 Acked Read 10 Done Done Sent false
29387 8074 Acked Read 9 Done Done Sent false
29388 8075 Acked Read 7 Done Done Sent false
29389 8076 Acked Write 5 Done Done Sent false
29390 8077 Acked Flush 0 Done Done Sent false
29391 8078 Acked Read 1 Done Done Sent false
29392 8079 Acked Read 10 Done Done Sent false
29393 8080 Acked Read 8 Done Done Sent false
29394 8081 Acked Write 8 Done Done Sent false
29395 8082 Acked Read 6 Done Done Sent false
29396 8083 Acked Read 4 Done Done Sent false
29397 8084 Acked Read 9 Done Done Sent false
29398 8085 Acked Read 4 Done Done Sent false
29399 8086 Acked Write 2 Done Done Sent false
29400 8087 Acked Write 3 Done Done Sent false
29401 8088 Acked Read 9 Done Done Sent false
29402 8089 Acked Read 4 Done Done Sent false
29403 8090 Acked Flush 0 Done Done Sent false
29404 8091 Acked Read 5 Done Done Sent false
29405 8092 Acked Write 1 Done Done Sent false
29406 8093 Acked Read 7 Done Done Sent false
29407 8094 Acked Write 7 Done Done Sent false
29408 8095 Acked Read 5 Done Done Sent false
29409 8096 Acked Read 4 Done Done Sent false
29410 8097 Acked Read 6 Done Done Sent false
29411 8098 Acked Read 1 Done Done Sent false
29412 8099 Acked Read 4 Done Done Sent false
29413 8100 Acked Read 8 Done Done Sent false
29414 8101 Acked Flush 0 Done Done Sent false
29415 8102 Acked Read 10 Done Done Sent false
29416 8103 Acked Flush 0 Done Done Sent false
29417 8104 Acked Read 6 Done Done Sent false
29418 8105 Acked Write 9 Done Done Sent false
29419 8106 Acked Read 10 Done Done Sent false
29420 8107 Acked Read 9 Done Done Sent false
29421 8108 Acked Read 5 Done Done Sent false
29422 8109 Acked Read 1 Done Done Sent false
29423 8110 Acked Write 3 Done Done Sent false
29424 8111 Acked Read 3 Done Done Sent false
29425 8112 Acked Read 10 Done Done Sent false
29426 8113 Acked Flush 0 Done Done Sent false
29427 8114 Acked Write 6 Done Done Sent false
29428 8115 Acked Read 6 Done Done Sent false
29429 8116 Acked Write 6 Done Done Sent false
29430 8117 Acked Write 7 Done Done Sent false
29431 8118 Acked Read 4 Done Done Sent false
29432 8119 Acked Read 3 Done Done Sent false
29433 8120 Acked Write 10 Done Done Sent false
29434 8121 Acked Read 2 Done Done Sent false
29435 8122 Acked Flush 0 Done Done Sent false
29436 8123 Acked Write 7 Done Done Sent false
29437 8124 Acked Read 9 Done Done Sent false
29438 8125 Acked Write 2 Done Done Sent false
29439 8126 Acked Write 2 Done Done Sent false
29440 8127 Acked Read 8 Done Done Sent false
29441 8128 Acked Read 3 Done Done Sent false
29442 8129 Acked Read 7 Done Done Sent false
29443 8130 Acked Write 4 Done Done Sent false
29444 8131 Acked Write 2 Done Done Sent false
29445 8132 Acked Write 5 Done Done Sent false
29446 8133 Acked Read 8 Done Done Sent false
29447 8134 Acked Write 1 Done Done Sent false
29448 8135 Acked Read 2 Done Done Sent false
29449 8136 Acked Read 4 Done Done Sent false
29450 8137 Acked Read 5 Done Done Sent false
29451 8138 Acked Read 9 Done Done Sent false
29452 8139 Acked Write 3 Done Done Sent false
29453 8140 Acked Read 6 Done Done Sent false
29454 8141 Acked Write 9 Done Done Sent false
29455 8142 Acked Write 10 Done Done Sent false
29456 8143 Acked Read 1 Done Done Sent false
29457 8144 Acked Read 9 Done Done Sent false
29458 8145 Acked Write 1 Done Done Sent false
29459 8146 Acked Read 3 Done Done Sent false
29460 8147 Acked Write 9 Done Done Sent false
29461 8148 Acked Read 10 Done Done Sent false
29462 8149 Acked Read 2 Done Done Sent false
29463 8150 Acked Write 8 Done Done Sent false
29464 8151 Acked Write 5 Done Done Sent false
29465 8152 Acked Read 7 Done Done Sent false
29466 8153 Acked Write 7 Done Done Sent false
29467 8154 Acked Read 2 Done Done Sent false
29468 8155 Acked Write 5 Done Done Sent false
29469 8156 Acked Write 10 Done Done Sent false
29470 8157 Acked Write 8 Done Done Sent false
29471 8158 Acked Write 3 Done Done Sent false
29472 8159 Acked Read 1 Done Done Sent false
29473 8160 Acked Read 4 Done Done Sent false
29474 8161 Acked Read 5 Done Done Sent false
29475 8162 Acked Read 2 Done Done Sent false
29476 8163 Acked Write 7 Done Done Sent false
29477 8164 Acked Read 5 Done Done Sent false
29478 8165 Acked Write 6 Done Done Sent false
29479 8166 Acked Write 3 Done Done Sent false
29480 8167 Acked Write 4 Done Done Sent false
29481 8168 Acked Write 1 Done Done Sent false
29482 8169 Acked Write 9 Done Done Sent false
29483 8170 Acked Read 6 Done Done Sent false
29484 8171 Acked Read 8 Done Done Sent false
29485 8172 Acked Read 1 Done Done Sent false
29486 8173 Acked Read 2 Done Done Sent false
29487 8174 Acked Read 7 Done Done Sent false
29488 8175 Acked Flush 0 Done Done Sent false
29489 8176 Acked Flush 0 Done Done Sent false
29490 8177 Acked Read 9 Done Done Sent false
29491 8178 Acked Read 9 Done Done Sent false
29492 8179 Acked Write 2 Done Done Sent false
29493 8180 Acked Write 10 Done Done Sent false
29494 8181 Acked Write 5 Done Done Sent false
29495 8182 Acked Read 8 Done Done Sent false
29496 8183 Acked Read 7 Done Done Sent false
29497 8184 Acked Read 7 Done Done Sent false
29498 8185 Acked Read 1 Done Done Sent false
29499 8186 Acked Write 5 Done Done Sent false
29500 8187 Acked Write 4 Done Done Sent false
29501 8188 Acked Read 10 Done Done Sent false
29502 8189 Acked Write 1 Done Done Sent false
29503 8190 Acked Write 1 Done Done Sent false
29504 8191 Acked Write 5 Done Done Sent false
29505 8192 Acked Flush 0 Done Done Sent false
29506 8193 Acked Read 6 Done Done Sent false
29507 8194 Acked Read 10 Done Done Sent false
29508 8195 Acked Write 6 Done Done Sent false
29509 8196 Acked Write 10 Done Done Sent false
29510 8197 Acked Read 5 Done Done Sent false
29511 8198 Acked Flush 0 Done Done Sent false
29512 8199 Acked Read 3 Done Done Sent false
29513 8200 Acked Write 5 Done Done Sent false
29514 8201 Acked Flush 0 Done Done Sent false
29515 8202 Acked Read 3 Done Done Sent false
29516 8203 Acked Read 9 Done Done Sent false
29517 8204 Acked Read 5 Done Done Sent false
29518 8205 Acked Write 7 Done Done Sent false
29519 8206 Acked Write 7 Done Done Sent false
29520 8207 Acked Write 4 Done Done Sent false
29521 8208 Acked Read 7 Done Done Sent false
29522 8209 Acked Write 2 Done Done Sent false
29523 8210 Acked Write 7 Done Done Sent false
29524 8211 Acked Read 1 Done Done Sent false
29525 8212 Acked Read 5 Done Done Sent false
29526 8213 Acked Read 2 Done Done Sent false
29527 8214 Acked Write 2 Done Done Sent false
29528 8215 Acked Read 3 Done Done Sent false
29529 8216 Acked Read 3 Done Done Sent false
29530 8217 Acked Read 10 Done Done Sent false
29531 8218 Acked Flush 0 Done Done Sent false
29532 8219 Acked Read 5 Done Done Sent false
29533 8220 Acked Write 7 Done Done Sent false
29534 8221 Acked Read 2 Done Done Sent false
29535 8222 Acked Write 4 Done Done Sent false
29536 8223 Acked Write 10 Done Done Sent false
29537 8224 Acked Read 9 Done Done Sent false
29538 8225 Acked Write 6 Done Done Sent false
29539 8226 Acked Write 7 Done Done Sent false
29540 8227 Acked Read 7 Done Done Sent false
29541 8228 Acked Write 9 Done Done Sent false
29542 8229 Acked Read 3 Done Done Sent false
29543 8230 Acked Read 2 Done Done Sent false
29544 8231 Acked Read 4 Done Done Sent false
29545 8232 Acked Read 4 Done Done Sent false
29546 8233 Acked Read 4 Done Done Sent false
29547 8234 Acked Write 1 Done Done Sent false
29548 8235 Acked Write 2 Done Done Sent false
29549 8236 Acked Read 9 Done Done Sent false
29550 8237 Acked Write 5 Done Done Sent false
29551 8238 Acked Read 7 Done Done Sent false
29552 8239 Acked Read 2 Done Done Sent false
29553 8240 Acked Read 3 Done Done Sent false
29554 8241 Acked Write 1 Done Done Sent false
29555 8242 Acked Read 4 Done Done Sent false
29556 8243 Acked Read 2 Done Done Sent false
29557 8244 Acked Write 5 Done Done Sent false
29558 8245 Acked Write 7 Done Done Sent false
29559 STATES DS:0 DS:1 DS:2 TOTAL
29560 Sent 0 0 301 301
29561 Done 301 301 0 602
29562 Skipped 0 0 0 0
29563 Error 0 0 0 0
29564 Last Flush: Some(JobId(8218)) Some(JobId(8218)) Some(JobId(7944))
29565 Downstairs last five completed: 7944 7943 7942 7941 7940
29566 Upstairs last five completed: 8245 8244 8243 8242 8241
29567 CLIENT: Up:0 ds:301 act:2
295682025-02-07T20:13:52.045ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
295692025-02-07T20:13:52.582ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
295702025-02-07T20:13:53.292ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
295712025-02-07T20:13:53.292ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:57404) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
295722025-02-07T20:13:53.293ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
295732025-02-07T20:13:53.294ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
295742025-02-07T20:13:53.294ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
295752025-02-07T20:13:53.294ZINFOcrucible: send last flush ID to this DS: Some(JobId(7944)) = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
295762025-02-07T20:13:53.294ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
295772025-02-07T20:13:53.294ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7944)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
295782025-02-07T20:13:53.294ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7944)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
29579 ----------------------------------------------------------------
29580 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29581 Downstairs last five completed: 8246 8245 8244 8243 8242
29582 Upstairs last five completed: 8246 8245 8244 8243 8242
29583 CLIENT: Up:0 ds:0 act:3
29584 Replay: All jobs finished
29585 Main task has work to do, go find it
29586 got dsc Stop(0)
29587 stop 0
29588 Feb 07 20:13:55.546 INFO request completed[8810] Got stop action so:false kr:false
29589 , latency_us: 381, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 9820089f-effc-4603-a5a3-6d1b2cbccc75, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
295902025-02-07T20:13:55.547ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
29591 Feb 07 20:13:55.547 INFO request completed, latency_us: 116, response_code: 200, uri: /state/cid/0, method: GET, req_id: 581685ea-6425-4187-a3f2-df73a01a2790, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
295922025-02-07T20:13:55.547ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
295932025-02-07T20:13:55.547ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
295942025-02-07T20:13:55.547ZINFOcrucible: client task is exiting = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
29595 [8810] Exited with: ExitStatus(unix_wait_status(9))
29596 [8810][0] reports Exit
29597 Feb 07 20:13:59.550 INFO request completed, latency_us: 257, response_code: 200, uri: /state/cid/0, method: GET, req_id: 7c1db8e1-13e7-46dc-a8f2-0f2adf9fe1b3, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
29598 001/300 Write block 3553 len 12288 data: 3 3 3
29599 002/300 Write block 1720 len 4096 data: 7
29600 003/300 Write block 1046 len 28672 data: 7 7 7 7 6 7 7
29601 004/300 Write block 407 len 16384 data: 3 5 4 4
29602 005/300 Write block 3354 len 4096 data: 4
29603 006/300 Write block 2707 len 36864 data: 6 6 4 5 4 5 7 6 5
29604 007/300 Write block 3576 len 20480 data: 7 6 5 4 4
29605 008/300 Write block 2592 len 28672 data: 5 8 8 8 8 9 8
29606 009/300 Write block 2610 len 36864 data: 4 3 3 4 4 4 4 6 5
29607 010/300 Read block 359 len 32768
29608 011/300 Read block 2520 len 4096
29609 012/300 Write block 2767 len 8192 data: 5 5
29610 013/300 Read block 3155 len 4096
29611 014/300 Write block 1313 len 36864 data: 3 5 4 8 7 7 6 8 9
29612 015/300 Write block 1605 len 20480 data: 4 6 5 5 6
29613 016/300 Write block 1570 len 16384 data: 4 4 4 4
29614 017/300 Write block 202 len 36864 data: 5 5 5 7 7 5 4 6 7
29615 018/300 Read block 4058 len 40960
29616 019/300 Read block 3528 len 4096
29617 020/300 Flush
29618 021/300 Write block 2771 len 40960 data: 5 5 3 7 6 7 7 7 6 6
29619 022/300 Flush
29620 023/300 Flush
29621 024/300 Write block 310 len 40960 data: 6 8 8 7 7 7 5 5 5 6
29622 025/300 Read block 3326 len 8192
29623 026/300 Flush
29624 027/300 Write block 4797 len 24576 data: 5 5 6 7 8 7
29625 028/300 Read block 4810 len 20480
29626 029/300 Flush
29627 030/300 Flush
29628 031/300 Write block 1566 len 12288 data: 4 3 3
29629 032/300 Flush
29630 033/300 Read block 2363 len 40960
29631 034/300 Write block 4209 len 8192 data: 5 5
29632 035/300 Write block 3457 len 4096 data: 3
29633 036/300 Write block 1870 len 36864 data: 4 5 4 3 2 2 2 2 3
29634 037/300 Read block 2186 len 40960
29635 038/300 Write block 2274 len 32768 data: 8 8 7 6 6 6 4 4
29636 039/300 Read block 3608 len 40960
29637 040/300 Read block 2129 len 32768
29638 041/300 Write block 3816 len 8192 data: 9 8
29639 042/300 Write block 2358 len 16384 data: 7 7 6 6
29640 043/300 Read block 1392 len 24576
29641 044/300 Read block 2004 len 20480
29642 045/300 Read block 236 len 20480
29643 046/300 Write block 3766 len 8192 data: 5 6
29644 047/300 Write block 3820 len 16384 data: 6 4 4 5
29645 048/300 Read block 3383 len 40960
29646 049/300 Write block 4844 len 4096 data: 7
29647 050/300 Read block 4350 len 12288
29648 051/300 Read block 1491 len 36864
29649 052/300 Read block 707 len 12288
29650 053/300 Read block 302 len 36864
29651 054/300 Write block 3676 len 16384 data: 6 7 9 9
29652 055/300 Read block 2579 len 24576
29653 056/300 Flush
29654 057/300 Write block 1894 len 28672 data: 7 6 6 7 7 8 8
29655 058/300 Read block 3648 len 32768
29656 059/300 Write block 4224 len 28672 data: 6 5 5 5 5 5 3
29657 060/300 Write block 4267 len 40960 data: 4 4 4 4 3 3 4 4 4 5
29658 061/300 Write block 3097 len 8192 data: 5 5
29659 062/300 Write block 4601 len 36864 data: 4 5 5 4 3 4 4 4 4
29660 063/300 Write block 2462 len 24576 data: 4 6 7 8 5 5
29661 064/300 Flush
29662 065/300 Write block 4892 len 20480 data: 4 6 6 7 7
29663 066/300 Read block 4788 len 12288
29664 067/300 Read block 572 len 28672
29665 068/300 Write block 1008 len 32768 data: 5 4 4 5 4 5 6 5
29666 069/300 Write block 2339 len 24576 data: 6 7 10 10 8 8
29667 070/300 Write block 4278 len 36864 data: 4 4 3 4 3 3 3 3 4
29668 071/300 Read block 21 len 32768
29669 072/300 Read block 3536 len 20480
29670 073/300 Write block 4506 len 16384 data: 7 7 7 8
29671 074/300 Read block 1008 len 4096
29672 075/300 Write block 2882 len 16384 data: 5 6 5 5
29673 076/300 Read block 2692 len 24576
29674 077/300 Write block 3548 len 8192 data: 3 2
29675 078/300 Read block 944 len 12288
29676 079/300 Read block 1428 len 36864
29677 080/300 Write block 4548 len 16384 data: 4 4 3 3
29678 081/300 Write block 251 len 28672 data: 4 5 4 3 4 5 5
29679 082/300 Read block 562 len 8192
29680 083/300 Read block 783 len 32768
29681 084/300 Flush
29682 085/300 Flush
29683 086/300 Read block 3287 len 40960
29684 087/300 Write block 2515 len 8192 data: 5 5
29685 088/300 Write block 2898 len 12288 data: 4 4 5
29686 089/300 Read block 1673 len 20480
29687 090/300 Read block 3383 len 24576
29688 091/300 Read block 2304 len 40960
29689 092/300 Read block 4694 len 20480
29690 093/300 Read block 3798 len 24576
29691 094/300 Write block 4646 len 8192 data: 6 7
29692 095/300 Flush
29693 096/300 Write block 1076 len 36864 data: 6 8 8 6 6 5 5 6 5
29694 097/300 Read block 3057 len 32768
29695 098/300 Flush
29696 099/300 Read block 735 len 8192
29697 100/300 Read block 2532 len 8192
29698 101/300 Write block 4500 len 24576 data: 13 13 14 10 10 7
29699 102/300 Read block 4348 len 32768
29700 103/300 Read block 1705 len 40960
29701 104/300 Read block 2031 len 20480
29702 105/300 Flush
29703 106/300 Read block 4494 len 8192
29704 107/300 Read block 4925 len 4096
29705 108/300 Write block 2504 len 32768 data: 4 5 6 7 6 4 4 4
29706 109/300 Write block 1998 len 16384 data: 9 12 9 8
29707 110/300 Write block 1461 len 24576 data: 4 4 2 2 3 4
29708 111/300 Read block 560 len 12288
29709 112/300 Read block 2418 len 24576
29710 113/300 Write block 492 len 8192 data: 4 5
29711 114/300 Flush
29712 115/300 Read block 1975 len 40960
29713 116/300 Read block 4843 len 28672
29714 117/300 Read block 1064 len 40960
29715 118/300 Read block 2579 len 40960
29716 119/300 Read block 1408 len 20480
29717 120/300 Flush
29718 121/300 Write block 325 len 40960 data: 4 3 3 3 4 4 5 6 5 8
29719 122/300 Write block 4957 len 36864 data: 7 6 6 7 8 8 7 6 6
29720 123/300 Read block 3429 len 40960
29721 124/300 Write block 4259 len 20480 data: 4 3 3 4 4
29722 125/300 Read block 3191 len 36864
29723 126/300 Write block 4511 len 36864 data: 11 7 7 7 7 6 6 6 7
29724 127/300 Read block 3470 len 36864
29725 128/300 Read block 4864 len 8192
29726 129/300 Write block 1578 len 24576 data: 5 5 6 7 6 5
29727 130/300 Write block 1010 len 24576 data: 5 6 5 6 7 6
29728 131/300 Write block 2784 len 24576 data: 6 4 3 3 3 3
29729 132/300 Read block 34 len 20480
29730 133/300 Read block 218 len 24576
29731 134/300 Write block 443 len 28672 data: 2 3 4 4 5 7 7
29732 135/300 Read block 218 len 28672
29733 136/300 Write block 4373 len 4096 data: 6
29734 137/300 Read block 155 len 4096
29735 138/300 Flush
29736 139/300 Write block 3054 len 16384 data: 6 6 4 3
29737 140/300 Write block 4289 len 12288 data: 4 5 5
29738 141/300 Read block 1457 len 16384
29739 142/300 Read block 1913 len 16384
29740 143/300 Read block 1955 len 8192
29741 144/300 Read block 2940 len 36864
29742 145/300 Read block 778 len 12288
29743 146/300 Read block 1001 len 36864
29744 147/300 Write block 3474 len 8192 data: 4 5
29745 148/300 Flush
29746 149/300 Write block 216 len 24576 data: 9 9 7 6 6 7
29747 150/300 Write block 148 len 24576 data: 4 4 4 4 4 4
29748 151/300 Flush
29749 152/300 Write block 4129 len 16384 data: 5 6 7 5
29750 153/300 Read block 3031 len 28672
29751 154/300 Write block 3990 len 16384 data: 3 4 4 5
29752 155/300 Flush
29753 156/300 Write block 4752 len 40960 data: 7 6 7 7 8 7 4 4 4 6
29754 157/300 Read block 93 len 20480
29755 158/300 Read block 3109 len 16384
29756 159/300 Flush
29757 160/300 Write block 2847 len 4096 data: 3
29758 161/300 Write block 4129 len 24576 data: 6 7 8 6 6 5
29759 162/300 Write block 36 len 28672 data: 5 4 5 8 6 6 6
29760 163/300 Read block 2735 len 16384
29761 164/300 Write block 109 len 8192 data: 5 5
29762 165/300 Read block 4931 len 28672
29763 166/300 Read block 990 len 4096
29764 167/300 Read block 1433 len 12288
29765 168/300 Write block 2282 len 8192 data: 6 6
29766 169/300 Write block 987 len 36864 data: 6 6 5 5 4 5 5 5 5
29767 170/300 Read block 4523 len 16384
29768 171/300 Read block 470 len 16384
29769 172/300 Write block 4542 len 20480 data: 4 4 4 5 5
29770 173/300 Write block 1908 len 24576 data: 7 6 5 6 6 7
29771 174/300 Write block 61 len 8192 data: 3 2
29772 175/300 Read block 4028 len 12288
29773 176/300 Read block 2175 len 16384
29774 177/300 Write block 2409 len 28672 data: 3 3 3 3 3 3 2
29775 178/300 Write block 429 len 32768 data: 4 5 4 5 5 6 4 4
29776 179/300 Read block 1686 len 4096
29777 180/300 Read block 1396 len 24576
29778 181/300 Read block 3242 len 28672
29779 182/300 Flush
29780 183/300 Write block 4208 len 20480 data: 6 6 6 6 7
29781 184/300 Read block 4013 len 32768
29782 185/300 Read block 1586 len 28672
29783 186/300 Read block 3976 len 8192
29784 187/300 Read block 3137 len 36864
29785 188/300 Read block 2146 len 36864
29786 189/300 Read block 3591 len 24576
29787 190/300 Write block 1722 len 12288 data: 5 5 4
29788 191/300 Read block 1053 len 24576
29789 192/300 Write block 1737 len 24576 data: 3 3 5 6 5 6
29790 193/300 Read block 4833 len 12288
29791 194/300 Read block 3768 len 32768
29792 195/300 Flush
29793 196/300 Write block 1419 len 12288 data: 5 5 6
29794 197/300 Read block 3873 len 16384
29795 198/300 Read block 2063 len 12288
29796 199/300 Read block 847 len 24576
29797 200/300 Flush
29798 201/300 Flush
29799 202/300 Flush
29800 203/300 Flush
29801 204/300 Read block 4425 len 20480
29802 205/300 Read block 3367 len 40960
29803 206/300 Flush
29804 207/300 Write block 3305 len 36864 data: 7 6 6 6 6 5 7 6 6
29805 208/300 Write block 882 len 36864 data: 5 7 7 6 8 7 6 6 4
29806 209/300 Read block 3458 len 40960
29807 210/300 Write block 431 len 40960 data: 5 6 6 7 5 5 4 3 3 3
29808 211/300 Read block 1210 len 36864
29809 212/300 Flush
29810 213/300 Read block 994 len 36864
29811 214/300 Read block 753 len 36864
29812 215/300 Flush
29813 216/300 Write block 701 len 36864 data: 5 5 4 5 5 6 5 4 4
29814 217/300 Flush
29815 218/300 Read block 1142 len 40960
29816 219/300 Read block 3363 len 12288
29817 220/300 Read block 2886 len 32768
29818 221/300 Read block 913 len 12288
29819 222/300 Write block 1407 len 12288 data: 4 4 5
29820 223/300 Write block 2562 len 4096 data: 4
29821 224/300 Write block 80 len 36864 data: 4 4 3 3 4 4 3 4 4
29822 225/300 Read block 812 len 16384
29823 226/300 Read block 1180 len 28672
29824 227/300 Read block 3269 len 12288
29825 228/300 Read block 1287 len 4096
29826 229/300 Read block 4382 len 4096
29827 230/300 Read block 3703 len 8192
29828 231/300 Read block 3302 len 28672
29829 232/300 Write block 143 len 12288 data: 2 3 3
29830 233/300 Write block 2298 len 40960 data: 6 5 4 4 3 4 3 3 3 4
29831 234/300 Read block 2430 len 8192
29832 235/300 Read block 2113 len 20480
29833 236/300 Write block 100 len 12288 data: 6 6 5
29834 237/300 Write block 4568 len 8192 data: 6 7
29835 238/300 Write block 4939 len 40960 data: 6 7 6 7 9 9 9 7 6 6
29836 239/300 Write block 2591 len 8192 data: 4 6
29837 240/300 Read block 2581 len 36864
29838 241/300 Flush
29839 242/300 Flush
29840 243/300 Read block 2543 len 16384
29841 244/300 Read block 2943 len 24576
29842 245/300 Read block 126 len 16384
29843 246/300 Write block 4834 len 8192 data: 10 9
29844 247/300 Write block 1540 len 36864 data: 5 5 5 5 4 4 4 4 2
29845 248/300 Read block 4126 len 40960
29846 249/300 Write block 3004 len 24576 data: 8 8 6 6 4 5
29847 250/300 Write block 4670 len 16384 data: 6 5 5 6
29848 251/300 Write block 1947 len 32768 data: 4 5 5 5 5 6 6 5
29849 252/300 Read block 4359 len 36864
29850 253/300 Read block 3581 len 32768
29851 254/300 Read block 3696 len 20480
29852 255/300 Read block 4427 len 36864
29853 256/300 Read block 3536 len 40960
29854 257/300 Read block 2926 len 8192
29855 258/300 Read block 454 len 32768
29856 259/300 Read block 1897 len 24576
29857 260/300 Read block 3207 len 24576
29858 261/300 Write block 426 len 40960 data: 6 5 5 5 6 6 7 7 8 6
29859 262/300 Write block 2353 len 32768 data: 3 4 5 6 6 8 8 7
29860 263/300 Write block 4104 len 20480 data: 5 5 3 3 4
29861 264/300 Read block 784 len 8192
29862 265/300 Read block 3690 len 40960
29863 266/300 Write block 1984 len 36864 data: 6 5 5 7 6 5 4 5 5
29864 267/300 Write block 2527 len 12288 data: 5 5 5
29865 268/300 Read block 1370 len 40960
29866 269/300 Read block 1662 len 20480
29867 270/300 Write block 2980 len 8192 data: 6 8
29868 271/300 Write block 2247 len 4096 data: 5
29869 272/300 Write block 2154 len 36864 data: 3 5 6 4 4 4 4 4 5
29870 273/300 Read block 1517 len 12288
29871 274/300 Read block 562 len 24576
29872 275/300 Read block 85 len 24576
29873 276/300 Read block 4771 len 36864
29874 277/300 Flush
29875 278/300 Write block 1846 len 8192 data: 6 7
29876 279/300 Write block 2906 len 20480 data: 6 6 6 6 6
29877 280/300 Read block 570 len 4096
29878 281/300 Read block 2391 len 8192
29879 282/300 Flush
29880 283/300 Read block 3791 len 28672
29881 284/300 Read block 1267 len 20480
29882 285/300 Read block 4799 len 28672
29883 286/300 Read block 1098 len 12288
29884 287/300 Read block 2200 len 28672
29885 288/300 Write block 1782 len 24576 data: 9 11 8 8 7 8
29886 289/300 Read block 1359 len 4096
29887 290/300 Write block 1612 len 32768 data: 5 4 3 5 6 5 5 5
29888 291/300 Read block 3698 len 24576
29889 292/300 Write block 743 len 16384 data: 4 5 4 4
29890 293/300 Write block 567 len 24576 data: 4 5 3 5 5 6
29891 294/300 Read block 2283 len 40960
29892 295/300 Write block 2375 len 40960 data: 5 5 7 7 5 4 5 8 8 8
29893 296/300 Read block 1095 len 8192
29894 297/300 Read block 4706 len 4096
29895 298/300 Write block 2911 len 16384 data: 5 5 4 4
29896 299/300 Read block 4327 len 28672
29897 300/300 Write block 2264 len 4096 data: 6
29898 301/300 Read block 3505 len 8192
29899 Main task has work to do, go find it
29900 got dsc Start(0)
29901 start 0
29902 Feb 07 20:13:59.794 INFO request completed, latency_us: 197, response_code: 204, uri: /start/cid/0, method: POST, req_id: f8c81b35-0a83-4db1-9906-18e6939b04ec, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
29903 Replay: started 0, returned:Ok(())
29904 ----------------------------------------------------------------
29905 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29906 States: Offline Active Active
29907 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29908 8247 Acked Write 3 Sent Done Done false
29909 8248 Acked Write 1 Sent Done Done false
29910 8249 Acked Write 7 Sent Done Done false
29911 8250 Acked Write 4 Sent Done Done false
29912 8251 Acked Write 1 Sent Done Done false
29913 8252 Acked Write 9 Sent Done Done false
29914 8253 Acked Write 5 Sent Done Done false
29915 8254 Acked Write 7 Sent Done Done false
29916 8255 Acked Write 9 Sent Done Done false
29917 8256 Acked Read 8 Sent Done Done false
29918 8257 Acked Read 1 Sent Done Done false
29919 8258 Acked Write 2 Sent Done Done false
29920 8259 Acked Read 1 Sent Done Done false
29921 8260 Acked Write 9 Sent Done Done false
29922 8261 Acked Write 5 Sent Done Done false
29923 8262 Acked Write 4 Sent Done Done false
29924 8263 Acked Write 9 Sent Done Done false
29925 8264 Acked Read 10 Sent Done Done false
29926 8265 Acked Read 1 Sent Done Done false
29927 8266 Acked Flush 0 Sent Done Done false
29928 8267 Acked Write 10 Sent Done Done false
29929 8268 Acked Flush 0 Sent Done Done false
29930 8269 Acked Flush 0 Sent Done Done false
29931 8270 Acked Write 10 Sent Done Done false
29932 8271 Acked Read 2 Sent Done Done false
29933 8272 Acked Flush 0 Sent Done Done false
29934 8273 Acked Write 6 Sent Done Done false
29935 8274 Acked Read 5 Sent Done Done false
29936 8275 Acked Flush 0 Sent Done Done false
29937 8276 Acked Flush 0 Sent Done Done false
29938 8277 Acked Write 3 Sent Done Done false
29939 8278 Acked Flush 0 Sent Done Done false
29940 8279 Acked Read 10 Sent Done Done false
29941 8280 Acked Write 2 Sent Done Done false
29942 8281 Acked Write 1 Sent Done Done false
29943 8282 Acked Write 9 Sent Done Done false
29944 8283 Acked Read 10 Sent Done Done false
29945 8284 Acked Write 8 Sent Done Done false
29946 8285 Acked Read 10 Sent Done Done false
29947 8286 Acked Read 8 Sent Done Done false
29948 8287 Acked Write 2 Sent Done Done false
29949 8288 Acked Write 4 Sent Done Done false
29950 8289 Acked Read 6 Sent Done Done false
29951 8290 Acked Read 5 Sent Done Done false
29952 8291 Acked Read 5 Sent Done Done false
29953 8292 Acked Write 2 Sent Done Done false
29954 8293 Acked Write 4 Sent Done Done false
29955 8294 Acked Read 10 Sent Done Done false
29956 8295 Acked Write 1 Sent Done Done false
29957 8296 Acked Read 3 Sent Done Done false
29958 8297 Acked Read 9 Sent Done Done false
29959 8298 Acked Read 3 Sent Done Done false
29960 8299 Acked Read 9 Sent Done Done false
29961 8300 Acked Write 4 Sent Done Done false
29962 8301 Acked Read 6 Sent Done Done false
29963 8302 Acked Flush 0 Sent Done Done false
29964 8303 Acked Write 7 Sent Done Done false
29965 8304 Acked Read 8 Sent Done Done false
29966 8305 Acked Write 7 Sent Done Done false
29967 8306 Acked Write 10 Sent Done Done false
29968 8307 Acked Write 2 Sent Done Done false
29969 8308 Acked Write 9 Sent Done Done false
29970 8309 Acked Write 6 Sent Done Done false
29971 8310 Acked Flush 0 Sent Done Done false
29972 8311 Acked Write 5 Sent Done Done false
29973 8312 Acked Read 3 Sent Done Done false
29974 8313 Acked Read 7 Sent Done Done false
29975 8314 Acked Write 8 Sent Done Done false
29976 8315 Acked Write 6 Sent Done Done false
29977 8316 Acked Write 9 Sent Done Done false
29978 8317 Acked Read 8 Sent Done Done false
29979 8318 Acked Read 5 Sent Done Done false
29980 8319 Acked Write 4 Sent Done Done false
29981 8320 Acked Read 1 Sent Done Done false
29982 8321 Acked Write 4 Sent Done Done false
29983 8322 Acked Read 6 Sent Done Done false
29984 8323 Acked Write 2 Sent Done Done false
29985 8324 Acked Read 3 Sent Done Done false
29986 8325 Acked Read 9 Sent Done Done false
29987 8326 Acked Write 4 Sent Done Done false
29988 8327 Acked Write 7 Sent Done Done false
29989 8328 Acked Read 2 Sent Done Done false
29990 8329 Acked Read 8 Sent Done Done false
29991 8330 Acked Flush 0 Sent Done Done false
29992 8331 Acked Flush 0 Sent Done Done false
29993 8332 Acked Read 10 Sent Done Done false
29994 8333 Acked Write 2 Sent Done Done false
29995 8334 Acked Write 3 Sent Done Done false
29996 8335 Acked Read 5 Sent Done Done false
29997 8336 Acked Read 6 Sent Done Done false
29998 8337 Acked Read 10 Sent Done Done false
29999 8338 Acked Read 5 Sent Done Done false
30000 8339 Acked Read 6 Sent Done Done false
30001 8340 Acked Write 2 Sent Done Done false
30002 8341 Acked Flush 0 Sent Done Done false
30003 8342 Acked Write 9 Sent Done Done false
30004 8343 Acked Read 8 Sent Done Done false
30005 8344 Acked Flush 0 Sent Done Done false
30006 8345 Acked Read 2 Sent Done Done false
30007 8346 Acked Read 2 Sent Done Done false
30008 8347 Acked Write 6 Sent Done Done false
30009 8348 Acked Read 8 Sent Done Done false
30010 8349 Acked Read 10 Sent Done Done false
30011 8350 Acked Read 5 Sent Done Done false
30012 8351 Acked Flush 0 Sent Done Done false
30013 8352 Acked Read 2 Sent Done Done false
30014 8353 Acked Read 1 Sent Done Done false
30015 8354 Acked Write 8 Sent Done Done false
30016 8355 Acked Write 4 Sent Done Done false
30017 8356 Acked Write 6 Sent Done Done false
30018 8357 Acked Read 3 Sent Done Done false
30019 8358 Acked Read 6 Sent Done Done false
30020 8359 Acked Write 2 Sent Done Done false
30021 8360 Acked Flush 0 Sent Done Done false
30022 8361 Acked Read 10 Sent Done Done false
30023 8362 Acked Read 7 Sent Done Done false
30024 8363 Acked Read 10 Sent Done Done false
30025 8364 Acked Read 10 Sent Done Done false
30026 8365 Acked Read 5 Sent Done Done false
30027 8366 Acked Flush 0 Sent Done Done false
30028 8367 Acked Write 10 Sent Done Done false
30029 8368 Acked Write 9 Sent Done Done false
30030 8369 Acked Read 10 Sent Done Done false
30031 8370 Acked Write 5 Sent Done Done false
30032 8371 Acked Read 9 Sent Done Done false
30033 8372 Acked Write 9 Sent Done Done false
30034 8373 Acked Read 9 Sent Done Done false
30035 8374 Acked Read 2 Sent Done Done false
30036 8375 Acked Write 6 Sent Done Done false
30037 8376 Acked Write 6 Sent Done Done false
30038 8377 Acked Write 6 Sent Done Done false
30039 8378 Acked Read 5 Sent Done Done false
30040 8379 Acked Read 6 Sent Done Done false
30041 8380 Acked Write 7 Sent Done Done false
30042 8381 Acked Read 7 Sent Done Done false
30043 8382 Acked Write 1 Sent Done Done false
30044 8383 Acked Read 1 Sent Done Done false
30045 8384 Acked Flush 0 Sent Done Done false
30046 8385 Acked Write 4 Sent Done Done false
30047 8386 Acked Write 3 Sent Done Done false
30048 8387 Acked Read 4 Sent Done Done false
30049 8388 Acked Read 4 Sent Done Done false
30050 8389 Acked Read 2 Sent Done Done false
30051 8390 Acked Read 9 Sent Done Done false
30052 8391 Acked Read 3 Sent Done Done false
30053 8392 Acked Read 9 Sent Done Done false
30054 8393 Acked Write 2 Sent Done Done false
30055 8394 Acked Flush 0 Sent Done Done false
30056 8395 Acked Write 6 Sent Done Done false
30057 8396 Acked Write 6 Sent Done Done false
30058 8397 Acked Flush 0 Sent Done Done false
30059 8398 Acked Write 4 Sent Done Done false
30060 8399 Acked Read 7 Sent Done Done false
30061 8400 Acked Write 4 Sent Done Done false
30062 8401 Acked Flush 0 Sent Done Done false
30063 8402 Acked Write 10 Sent Done Done false
30064 8403 Acked Read 5 Sent Done Done false
30065 8404 Acked Read 4 Sent Done Done false
30066 8405 Acked Flush 0 Sent Done Done false
30067 8406 Acked Write 1 Sent Done Done false
30068 8407 Acked Write 6 Sent Done Done false
30069 8408 Acked Write 7 Sent Done Done false
30070 8409 Acked Read 4 Sent Done Done false
30071 8410 Acked Write 2 Sent Done Done false
30072 8411 Acked Read 7 Sent Done Done false
30073 8412 Acked Read 1 Sent Done Done false
30074 8413 Acked Read 3 Sent Done Done false
30075 8414 Acked Write 2 Sent Done Done false
30076 8415 Acked Write 9 Sent Done Done false
30077 8416 Acked Read 4 Sent Done Done false
30078 8417 Acked Read 4 Sent Done Done false
30079 8418 Acked Write 5 Sent Done Done false
30080 8419 Acked Write 6 Sent Done Done false
30081 8420 Acked Write 2 Sent Done Done false
30082 8421 Acked Read 3 Sent Done Done false
30083 8422 Acked Read 4 Sent Done Done false
30084 8423 Acked Write 7 Sent Done Done false
30085 8424 Acked Write 8 Sent Done Done false
30086 8425 Acked Read 1 Sent Done Done false
30087 8426 Acked Read 6 Sent Done Done false
30088 8427 Acked Read 7 Sent Done Done false
30089 8428 Acked Flush 0 Sent Done Done false
30090 8429 Acked Write 5 Sent Done Done false
30091 8430 Acked Read 8 Sent Done Done false
30092 8431 Acked Read 7 Sent Done Done false
30093 8432 Acked Read 2 Sent Done Done false
30094 8433 Acked Read 9 Sent Done Done false
30095 8434 Acked Read 9 Sent Done Done false
30096 8435 Acked Read 6 Sent Done Done false
30097 8436 Acked Write 3 Sent Done Done false
30098 8437 Acked Read 6 Sent Done Done false
30099 8438 Acked Write 6 Sent Done Done false
30100 8439 Acked Read 3 Sent Done Done false
30101 8440 Acked Read 8 Sent Done Done false
30102 8441 Acked Flush 0 Sent Done Done false
30103 8442 Acked Write 3 Sent Done Done false
30104 8443 Acked Read 4 Sent Done Done false
30105 8444 Acked Read 3 Sent Done Done false
30106 8445 Acked Read 6 Sent Done Done false
30107 8446 Acked Flush 0 Sent Done Done false
30108 8447 Acked Flush 0 Sent Done Done false
30109 8448 Acked Flush 0 Sent Done Done false
30110 8449 Acked Flush 0 Sent Done Done false
30111 8450 Acked Read 5 Sent Done Done false
30112 8451 Acked Read 10 Sent Done Done false
30113 8452 Acked Flush 0 Sent Done Done false
30114 8453 Acked Write 9 Sent Done Done false
30115 8454 Acked Write 9 Sent Done Done false
30116 8455 Acked Read 10 Sent Done Done false
30117 8456 Acked Write 10 Sent Done Done false
30118 8457 Acked Read 9 Sent Done Done false
30119 8458 Acked Flush 0 Sent Done Done false
30120 8459 Acked Read 9 Sent Done Done false
30121 8460 Acked Read 9 Sent Done Done false
30122 8461 Acked Flush 0 Sent Done Done false
30123 8462 Acked Write 9 Sent Done Done false
30124 8463 Acked Flush 0 Sent Done Done false
30125 8464 Acked Read 10 Sent Done Done false
30126 8465 Acked Read 3 Sent Done Done false
30127 8466 Acked Read 8 Sent Done Done false
30128 8467 Acked Read 3 Sent Done Done false
30129 8468 Acked Write 3 Sent Done Done false
30130 8469 Acked Write 1 Sent Done Done false
30131 8470 Acked Write 9 Sent Done Done false
30132 8471 Acked Read 4 Sent Done Done false
30133 8472 Acked Read 7 Sent Done Done false
30134 8473 Acked Read 3 Sent Done Done false
30135 8474 Acked Read 1 Sent Done Done false
30136 8475 Acked Read 1 Sent Done Done false
30137 8476 Acked Read 2 Sent Done Done false
30138 8477 Acked Read 7 Sent Done Done false
30139 8478 Acked Write 3 Sent Done Done false
30140 8479 Acked Write 10 Sent Done Done false
30141 8480 Acked Read 2 Sent Done Done false
30142 8481 Acked Read 5 Sent Done Done false
30143 8482 Acked Write 3 Sent Done Done false
30144 8483 Acked Write 2 Sent Done Done false
30145 8484 Acked Write 10 Sent Done Done false
30146 8485 Acked Write 2 Sent Done Done false
30147 8486 Acked Read 9 Sent Done Done false
30148 8487 Acked Flush 0 Sent Done Done false
30149 8488 Acked Flush 0 Sent Done Done false
30150 8489 Acked Read 4 Sent Done Done false
30151 8490 Acked Read 6 Sent Done Done false
30152 8491 Acked Read 4 Sent Done Done false
30153 8492 Acked Write 2 Sent Done Done false
30154 8493 Acked Write 9 Sent Done Done false
30155 8494 Acked Read 10 Sent Done Done false
30156 8495 Acked Write 6 Sent Done Done false
30157 8496 Acked Write 4 Sent Done Done false
30158 8497 Acked Write 8 Sent Done Done false
30159 8498 Acked Read 9 Sent Done Done false
30160 8499 Acked Read 8 Sent Done Done false
30161 8500 Acked Read 5 Sent Done Done false
30162 8501 Acked Read 9 Sent Done Done false
30163 8502 Acked Read 10 Sent Done Done false
30164 8503 Acked Read 2 Sent Done Done false
30165 8504 Acked Read 8 Sent Done Done false
30166 8505 Acked Read 6 Sent Done Done false
30167 8506 Acked Read 6 Sent Done Done false
30168 8507 Acked Write 10 Sent Done Done false
30169 8508 Acked Write 8 Sent Done Done false
30170 8509 Acked Write 5 Sent Done Done false
30171 8510 Acked Read 2 Sent Done Done false
30172 8511 Acked Read 10 Sent Done Done false
30173 8512 Acked Write 9 Sent Done Done false
30174 8513 Acked Write 3 Sent Done Done false
30175 8514 Acked Read 10 Sent Done Done false
30176 8515 Acked Read 5 Sent Done Done false
30177 8516 Acked Write 2 Sent Done Done false
30178 8517 Acked Write 1 Sent Done Done false
30179 8518 Acked Write 9 Sent Done Done false
30180 8519 Acked Read 3 Sent Done Done false
30181 8520 Acked Read 6 Sent Done Done false
30182 8521 Acked Read 6 Sent Done Done false
30183 8522 Acked Read 9 Sent Done Done false
30184 8523 Acked Flush 0 Sent Done Done false
30185 8524 Acked Write 2 Sent Done Done false
30186 8525 Acked Write 5 Sent Done Done false
30187 8526 Acked Read 1 Sent Done Done false
30188 8527 Acked Read 2 Sent Done Done false
30189 8528 Acked Flush 0 Sent Done Done false
30190 8529 Acked Read 7 Sent Done Done false
30191 8530 Acked Read 5 Sent Done Done false
30192 8531 Acked Read 7 Sent Done Done false
30193 8532 Acked Read 3 Sent Done Done false
30194 8533 Acked Read 7 Sent Done Done false
30195 8534 Acked Write 6 Sent Done Done false
30196 8535 Acked Read 1 Sent Done Done false
30197 8536 Acked Write 8 Sent Done Done false
30198 8537 Acked Read 6 Sent Done Done false
30199 8538 Acked Write 4 Sent Done Done false
30200 8539 Acked Write 6 Sent Done Done false
30201 8540 Acked Read 10 Sent Done Done false
30202 8541 Acked Write 10 Sent Done Done false
30203 8542 Acked Read 2 Sent Done Done false
30204 8543 Acked Read 1 Sent Done Done false
30205 8544 Acked Write 4 Sent Done Done false
30206 8545 Acked Read 7 Sent Done Done false
30207 8546 Acked Write 1 Sent Done Done false
30208 8547 Acked Read 2 Sent Done Done false
30209 STATES DS:0 DS:1 DS:2 TOTAL
30210 Sent 301 0 0 301
30211 Done 0 301 301 602
30212 Skipped 0 0 0 0
30213 Error 0 0 0 0
30214 Last Flush: Some(JobId(8246)) Some(JobId(8528)) Some(JobId(8528))
30215 Downstairs last five completed: 8246 8245 8244 8243 8242
30216 Upstairs last five completed: 8547 8546 8545 8544 8543
30217 CLIENT: Up:0 ds:301 act:2
302182025-02-07T20:14:00.596ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
302192025-02-07T20:14:01.598ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
302202025-02-07T20:14:02.099ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
302212025-02-07T20:14:02.600ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
30222 [8810] got start action
30223 [8810] I am going to restart
30224 Starting downstairs at port 8810
30225 Make output file at "/tmp/dsc/downstairs-8810.txt"
30226 [8810][0] reports Starting
30227 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1234)
30228 [8810][0] reports Running
30229 ----------------------------------------------------------------
30230 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30231 States: Offline Active Active
30232 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30233 8247 Acked Write 3 Sent Done Done false
30234 8248 Acked Write 1 Sent Done Done false
30235 8249 Acked Write 7 Sent Done Done false
30236 8250 Acked Write 4 Sent Done Done false
30237 8251 Acked Write 1 Sent Done Done false
30238 8252 Acked Write 9 Sent Done Done false
30239 8253 Acked Write 5 Sent Done Done false
30240 8254 Acked Write 7 Sent Done Done false
30241 8255 Acked Write 9 Sent Done Done false
30242 8256 Acked Read 8 Sent Done Done false
30243 8257 Acked Read 1 Sent Done Done false
30244 8258 Acked Write 2 Sent Done Done false
30245 8259 Acked Read 1 Sent Done Done false
30246 8260 Acked Write 9 Sent Done Done false
30247 8261 Acked Write 5 Sent Done Done false
30248 8262 Acked Write 4 Sent Done Done false
30249 8263 Acked Write 9 Sent Done Done false
30250 8264 Acked Read 10 Sent Done Done false
30251 8265 Acked Read 1 Sent Done Done false
30252 8266 Acked Flush 0 Sent Done Done false
30253 8267 Acked Write 10 Sent Done Done false
30254 8268 Acked Flush 0 Sent Done Done false
30255 8269 Acked Flush 0 Sent Done Done false
30256 8270 Acked Write 10 Sent Done Done false
30257 8271 Acked Read 2 Sent Done Done false
30258 8272 Acked Flush 0 Sent Done Done false
30259 8273 Acked Write 6 Sent Done Done false
30260 8274 Acked Read 5 Sent Done Done false
30261 8275 Acked Flush 0 Sent Done Done false
30262 8276 Acked Flush 0 Sent Done Done false
30263 8277 Acked Write 3 Sent Done Done false
30264 8278 Acked Flush 0 Sent Done Done false
30265 8279 Acked Read 10 Sent Done Done false
30266 8280 Acked Write 2 Sent Done Done false
30267 8281 Acked Write 1 Sent Done Done false
30268 8282 Acked Write 9 Sent Done Done false
30269 8283 Acked Read 10 Sent Done Done false
30270 8284 Acked Write 8 Sent Done Done false
30271 8285 Acked Read 10 Sent Done Done false
30272 8286 Acked Read 8 Sent Done Done false
30273 8287 Acked Write 2 Sent Done Done false
30274 8288 Acked Write 4 Sent Done Done false
30275 8289 Acked Read 6 Sent Done Done false
30276 8290 Acked Read 5 Sent Done Done false
30277 8291 Acked Read 5 Sent Done Done false
30278 8292 Acked Write 2 Sent Done Done false
30279 8293 Acked Write 4 Sent Done Done false
30280 8294 Acked Read 10 Sent Done Done false
30281 8295 Acked Write 1 Sent Done Done false
30282 8296 Acked Read 3 Sent Done Done false
30283 8297 Acked Read 9 Sent Done Done false
30284 8298 Acked Read 3 Sent Done Done false
30285 8299 Acked Read 9 Sent Done Done false
30286 8300 Acked Write 4 Sent Done Done false
30287 8301 Acked Read 6 Sent Done Done false
30288 8302 Acked Flush 0 Sent Done Done false
30289 8303 Acked Write 7 Sent Done Done false
30290 8304 Acked Read 8 Sent Done Done false
30291 8305 Acked Write 7 Sent Done Done false
30292 8306 Acked Write 10 Sent Done Done false
30293 8307 Acked Write 2 Sent Done Done false
30294 8308 Acked Write 9 Sent Done Done false
30295 8309 Acked Write 6 Sent Done Done false
30296 8310 Acked Flush 0 Sent Done Done false
30297 8311 Acked Write 5 Sent Done Done false
30298 8312 Acked Read 3 Sent Done Done false
30299 8313 Acked Read 7 Sent Done Done false
30300 8314 Acked Write 8 Sent Done Done false
30301 8315 Acked Write 6 Sent Done Done false
30302 8316 Acked Write 9 Sent Done Done false
30303 8317 Acked Read 8 Sent Done Done false
30304 8318 Acked Read 5 Sent Done Done false
30305 8319 Acked Write 4 Sent Done Done false
30306 8320 Acked Read 1 Sent Done Done false
30307 8321 Acked Write 4 Sent Done Done false
30308 8322 Acked Read 6 Sent Done Done false
30309 8323 Acked Write 2 Sent Done Done false
30310 8324 Acked Read 3 Sent Done Done false
30311 8325 Acked Read 9 Sent Done Done false
30312 8326 Acked Write 4 Sent Done Done false
30313 8327 Acked Write 7 Sent Done Done false
30314 8328 Acked Read 2 Sent Done Done false
30315 8329 Acked Read 8 Sent Done Done false
30316 8330 Acked Flush 0 Sent Done Done false
30317 8331 Acked Flush 0 Sent Done Done false
30318 8332 Acked Read 10 Sent Done Done false
30319 8333 Acked Write 2 Sent Done Done false
30320 8334 Acked Write 3 Sent Done Done false
30321 8335 Acked Read 5 Sent Done Done false
30322 8336 Acked Read 6 Sent Done Done false
30323 8337 Acked Read 10 Sent Done Done false
30324 8338 Acked Read 5 Sent Done Done false
30325 8339 Acked Read 6 Sent Done Done false
30326 8340 Acked Write 2 Sent Done Done false
30327 8341 Acked Flush 0 Sent Done Done false
30328 8342 Acked Write 9 Sent Done Done false
30329 8343 Acked Read 8 Sent Done Done false
30330 8344 Acked Flush 0 Sent Done Done false
30331 8345 Acked Read 2 Sent Done Done false
30332 8346 Acked Read 2 Sent Done Done false
30333 8347 Acked Write 6 Sent Done Done false
30334 8348 Acked Read 8 Sent Done Done false
30335 8349 Acked Read 10 Sent Done Done false
30336 8350 Acked Read 5 Sent Done Done false
30337 8351 Acked Flush 0 Sent Done Done false
30338 8352 Acked Read 2 Sent Done Done false
30339 8353 Acked Read 1 Sent Done Done false
30340 8354 Acked Write 8 Sent Done Done false
30341 8355 Acked Write 4 Sent Done Done false
30342 8356 Acked Write 6 Sent Done Done false
30343 8357 Acked Read 3 Sent Done Done false
30344 8358 Acked Read 6 Sent Done Done false
30345 8359 Acked Write 2 Sent Done Done false
30346 8360 Acked Flush 0 Sent Done Done false
30347 8361 Acked Read 10 Sent Done Done false
30348 8362 Acked Read 7 Sent Done Done false
30349 8363 Acked Read 10 Sent Done Done false
30350 8364 Acked Read 10 Sent Done Done false
30351 8365 Acked Read 5 Sent Done Done false
30352 8366 Acked Flush 0 Sent Done Done false
30353 8367 Acked Write 10 Sent Done Done false
30354 8368 Acked Write 9 Sent Done Done false
30355 8369 Acked Read 10 Sent Done Done false
30356 8370 Acked Write 5 Sent Done Done false
30357 8371 Acked Read 9 Sent Done Done false
30358 8372 Acked Write 9 Sent Done Done false
30359 8373 Acked Read 9 Sent Done Done false
30360 8374 Acked Read 2 Sent Done Done false
30361 8375 Acked Write 6 Sent Done Done false
30362 8376 Acked Write 6 Sent Done Done false
30363 8377 Acked Write 6 Sent Done Done false
30364 8378 Acked Read 5 Sent Done Done false
30365 8379 Acked Read 6 Sent Done Done false
30366 8380 Acked Write 7 Sent Done Done false
30367 8381 Acked Read 7 Sent Done Done false
30368 8382 Acked Write 1 Sent Done Done false
30369 8383 Acked Read 1 Sent Done Done false
30370 8384 Acked Flush 0 Sent Done Done false
30371 8385 Acked Write 4 Sent Done Done false
30372 8386 Acked Write 3 Sent Done Done false
30373 8387 Acked Read 4 Sent Done Done false
30374 8388 Acked Read 4 Sent Done Done false
30375 8389 Acked Read 2 Sent Done Done false
30376 8390 Acked Read 9 Sent Done Done false
30377 8391 Acked Read 3 Sent Done Done false
30378 8392 Acked Read 9 Sent Done Done false
30379 8393 Acked Write 2 Sent Done Done false
30380 8394 Acked Flush 0 Sent Done Done false
30381 8395 Acked Write 6 Sent Done Done false
30382 8396 Acked Write 6 Sent Done Done false
30383 8397 Acked Flush 0 Sent Done Done false
30384 8398 Acked Write 4 Sent Done Done false
30385 8399 Acked Read 7 Sent Done Done false
30386 8400 Acked Write 4 Sent Done Done false
30387 8401 Acked Flush 0 Sent Done Done false
30388 8402 Acked Write 10 Sent Done Done false
30389 8403 Acked Read 5 Sent Done Done false
30390 8404 Acked Read 4 Sent Done Done false
30391 8405 Acked Flush 0 Sent Done Done false
30392 8406 Acked Write 1 Sent Done Done false
30393 8407 Acked Write 6 Sent Done Done false
30394 8408 Acked Write 7 Sent Done Done false
30395 8409 Acked Read 4 Sent Done Done false
30396 8410 Acked Write 2 Sent Done Done false
30397 8411 Acked Read 7 Sent Done Done false
30398 8412 Acked Read 1 Sent Done Done false
30399 8413 Acked Read 3 Sent Done Done false
30400 8414 Acked Write 2 Sent Done Done false
30401 8415 Acked Write 9 Sent Done Done false
30402 8416 Acked Read 4 Sent Done Done false
30403 8417 Acked Read 4 Sent Done Done false
30404 8418 Acked Write 5 Sent Done Done false
30405 8419 Acked Write 6 Sent Done Done false
30406 8420 Acked Write 2 Sent Done Done false
30407 8421 Acked Read 3 Sent Done Done false
30408 8422 Acked Read 4 Sent Done Done false
30409 8423 Acked Write 7 Sent Done Done false
30410 8424 Acked Write 8 Sent Done Done false
30411 8425 Acked Read 1 Sent Done Done false
30412 8426 Acked Read 6 Sent Done Done false
30413 8427 Acked Read 7 Sent Done Done false
30414 8428 Acked Flush 0 Sent Done Done false
30415 8429 Acked Write 5 Sent Done Done false
30416 8430 Acked Read 8 Sent Done Done false
30417 8431 Acked Read 7 Sent Done Done false
30418 8432 Acked Read 2 Sent Done Done false
30419 8433 Acked Read 9 Sent Done Done false
30420 8434 Acked Read 9 Sent Done Done false
30421 8435 Acked Read 6 Sent Done Done false
30422 8436 Acked Write 3 Sent Done Done false
30423 8437 Acked Read 6 Sent Done Done false
30424 8438 Acked Write 6 Sent Done Done false
30425 8439 Acked Read 3 Sent Done Done false
30426 8440 Acked Read 8 Sent Done Done false
30427 8441 Acked Flush 0 Sent Done Done false
30428 8442 Acked Write 3 Sent Done Done false
30429 8443 Acked Read 4 Sent Done Done false
30430 8444 Acked Read 3 Sent Done Done false
30431 8445 Acked Read 6 Sent Done Done false
30432 8446 Acked Flush 0 Sent Done Done false
30433 8447 Acked Flush 0 Sent Done Done false
30434 8448 Acked Flush 0 Sent Done Done false
30435 8449 Acked Flush 0 Sent Done Done false
30436 8450 Acked Read 5 Sent Done Done false
30437 8451 Acked Read 10 Sent Done Done false
30438 8452 Acked Flush 0 Sent Done Done false
30439 8453 Acked Write 9 Sent Done Done false
30440 8454 Acked Write 9 Sent Done Done false
30441 8455 Acked Read 10 Sent Done Done false
30442 8456 Acked Write 10 Sent Done Done false
30443 8457 Acked Read 9 Sent Done Done false
30444 8458 Acked Flush 0 Sent Done Done false
30445 8459 Acked Read 9 Sent Done Done false
30446 8460 Acked Read 9 Sent Done Done false
30447 8461 Acked Flush 0 Sent Done Done false
30448 8462 Acked Write 9 Sent Done Done false
30449 8463 Acked Flush 0 Sent Done Done false
30450 8464 Acked Read 10 Sent Done Done false
30451 8465 Acked Read 3 Sent Done Done false
30452 8466 Acked Read 8 Sent Done Done false
30453 8467 Acked Read 3 Sent Done Done false
30454 8468 Acked Write 3 Sent Done Done false
30455 8469 Acked Write 1 Sent Done Done false
30456 8470 Acked Write 9 Sent Done Done false
30457 8471 Acked Read 4 Sent Done Done false
30458 8472 Acked Read 7 Sent Done Done false
30459 8473 Acked Read 3 Sent Done Done false
30460 8474 Acked Read 1 Sent Done Done false
30461 8475 Acked Read 1 Sent Done Done false
30462 8476 Acked Read 2 Sent Done Done false
30463 8477 Acked Read 7 Sent Done Done false
30464 8478 Acked Write 3 Sent Done Done false
30465 8479 Acked Write 10 Sent Done Done false
30466 8480 Acked Read 2 Sent Done Done false
30467 8481 Acked Read 5 Sent Done Done false
30468 8482 Acked Write 3 Sent Done Done false
30469 8483 Acked Write 2 Sent Done Done false
30470 8484 Acked Write 10 Sent Done Done false
30471 8485 Acked Write 2 Sent Done Done false
30472 8486 Acked Read 9 Sent Done Done false
30473 8487 Acked Flush 0 Sent Done Done false
30474 8488 Acked Flush 0 Sent Done Done false
30475 8489 Acked Read 4 Sent Done Done false
30476 8490 Acked Read 6 Sent Done Done false
30477 8491 Acked Read 4 Sent Done Done false
30478 8492 Acked Write 2 Sent Done Done false
30479 8493 Acked Write 9 Sent Done Done false
30480 8494 Acked Read 10 Sent Done Done false
30481 8495 Acked Write 6 Sent Done Done false
30482 8496 Acked Write 4 Sent Done Done false
30483 8497 Acked Write 8 Sent Done Done false
30484 8498 Acked Read 9 Sent Done Done false
30485 8499 Acked Read 8 Sent Done Done false
30486 8500 Acked Read 5 Sent Done Done false
30487 8501 Acked Read 9 Sent Done Done false
30488 8502 Acked Read 10 Sent Done Done false
30489 8503 Acked Read 2 Sent Done Done false
30490 8504 Acked Read 8 Sent Done Done false
30491 8505 Acked Read 6 Sent Done Done false
30492 8506 Acked Read 6 Sent Done Done false
30493 8507 Acked Write 10 Sent Done Done false
30494 8508 Acked Write 8 Sent Done Done false
30495 8509 Acked Write 5 Sent Done Done false
30496 8510 Acked Read 2 Sent Done Done false
30497 8511 Acked Read 10 Sent Done Done false
30498 8512 Acked Write 9 Sent Done Done false
30499 8513 Acked Write 3 Sent Done Done false
30500 8514 Acked Read 10 Sent Done Done false
30501 8515 Acked Read 5 Sent Done Done false
30502 8516 Acked Write 2 Sent Done Done false
30503 8517 Acked Write 1 Sent Done Done false
30504 8518 Acked Write 9 Sent Done Done false
30505 8519 Acked Read 3 Sent Done Done false
30506 8520 Acked Read 6 Sent Done Done false
30507 8521 Acked Read 6 Sent Done Done false
30508 8522 Acked Read 9 Sent Done Done false
30509 8523 Acked Flush 0 Sent Done Done false
30510 8524 Acked Write 2 Sent Done Done false
30511 8525 Acked Write 5 Sent Done Done false
30512 8526 Acked Read 1 Sent Done Done false
30513 8527 Acked Read 2 Sent Done Done false
30514 8528 Acked Flush 0 Sent Done Done false
30515 8529 Acked Read 7 Sent Done Done false
30516 8530 Acked Read 5 Sent Done Done false
30517 8531 Acked Read 7 Sent Done Done false
30518 8532 Acked Read 3 Sent Done Done false
30519 8533 Acked Read 7 Sent Done Done false
30520 8534 Acked Write 6 Sent Done Done false
30521 8535 Acked Read 1 Sent Done Done false
30522 8536 Acked Write 8 Sent Done Done false
30523 8537 Acked Read 6 Sent Done Done false
30524 8538 Acked Write 4 Sent Done Done false
30525 8539 Acked Write 6 Sent Done Done false
30526 8540 Acked Read 10 Sent Done Done false
30527 8541 Acked Write 10 Sent Done Done false
30528 8542 Acked Read 2 Sent Done Done false
30529 8543 Acked Read 1 Sent Done Done false
30530 8544 Acked Write 4 Sent Done Done false
30531 8545 Acked Read 7 Sent Done Done false
30532 8546 Acked Write 1 Sent Done Done false
30533 8547 Acked Read 2 Sent Done Done false
30534 STATES DS:0 DS:1 DS:2 TOTAL
30535 Sent 301 0 0 301
30536 Done 0 301 301 602
30537 Skipped 0 0 0 0
30538 Error 0 0 0 0
30539 Last Flush: Some(JobId(8246)) Some(JobId(8528)) Some(JobId(8528))
30540 Downstairs last five completed: 8246 8245 8244 8243 8242
30541 Upstairs last five completed: 8547 8546 8545 8544 8543
30542 CLIENT: Up:0 ds:301 act:2
305432025-02-07T20:14:04.603ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
305442025-02-07T20:14:05.548ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
305452025-02-07T20:14:05.548ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56161) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
305462025-02-07T20:14:05.549ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
305472025-02-07T20:14:05.549ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
305482025-02-07T20:14:05.549ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
305492025-02-07T20:14:05.549ZINFOcrucible: send last flush ID to this DS: Some(JobId(8246)) = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
305502025-02-07T20:14:05.549ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
305512025-02-07T20:14:05.549ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(8246)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
305522025-02-07T20:14:05.550ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(8246)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
30553 ----------------------------------------------------------------
30554 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30555 Downstairs last five completed: 8548 8547 8546 8545 8544
30556 Upstairs last five completed: 8548 8547 8546 8545 8544
30557 CLIENT: Up:0 ds:0 act:3
30558 Replay: All jobs finished
30559 Main task has work to do, go find it
30560 got dsc Stop(1)
30561 stop 1
30562 Feb 07 20:14:07.800[8820] Got stop action so:false kr:false
30563 INFO request completed, latency_us: 361, response_code: 204, uri: /stop/cid/1, method: POST, req_id: f4bcdfe0-41d2-4739-81bc-a1b853edef20, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
30564 Feb 07 20:14:07.801 INFO request completed, latency_us: 113, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4b5f253b-fdc3-4b85-aa3c-589eba2db573, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
305652025-02-07T20:14:07.801ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
305662025-02-07T20:14:07.801ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
305672025-02-07T20:14:07.801ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
305682025-02-07T20:14:07.801ZINFOcrucible: client task is exiting = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
30569 [8820] Exited with: ExitStatus(unix_wait_status(9))
30570 [8820][1] reports Exit
30571 Feb 07 20:14:11.803 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1990c6d5-54ef-44de-967d-c16e5203c28f, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
30572 001/300 Read block 4663 len 32768
30573 002/300 Read block 331 len 8192
30574 003/300 Read block 2163 len 40960
30575 004/300 Read block 2564 len 8192
30576 005/300 Flush
30577 006/300 Read block 1091 len 28672
30578 007/300 Read block 2498 len 20480
30579 008/300 Read block 1851 len 4096
30580 009/300 Read block 3150 len 40960
30581 010/300 Flush
30582 011/300 Read block 2644 len 24576
30583 012/300 Write block 3551 len 16384 data: 2 3 4 4
30584 013/300 Read block 5 len 32768
30585 014/300 Flush
30586 015/300 Write block 3697 len 24576 data: 6 5 5 4 6 7
30587 016/300 Read block 3208 len 36864
30588 017/300 Write block 2038 len 24576 data: 5 4 4 3 3 2
30589 018/300 Read block 1081 len 4096
30590 019/300 Read block 300 len 12288
30591 020/300 Read block 1737 len 20480
30592 021/300 Write block 3033 len 8192 data: 2 2
30593 022/300 Write block 4011 len 36864 data: 5 4 4 3 3 3 5 6 6
30594 023/300 Write block 2412 len 4096 data: 4
30595 024/300 Write block 1607 len 16384 data: 6 6 7 5
30596 025/300 Write block 1616 len 36864 data: 7 6 6 6 6 6 6 6 4
30597 026/300 Write block 3105 len 20480 data: 6 6 6 6 6
30598 027/300 Read block 1224 len 24576
30599 028/300 Read block 1655 len 36864
30600 029/300 Write block 4212 len 36864 data: 8 6 5 6 5 7 8 8 7
30601 030/300 Read block 2632 len 8192
30602 031/300 Flush
30603 032/300 Read block 1055 len 28672
30604 033/300 Write block 4366 len 28672 data: 3 3 3 2 3 4 5
30605 034/300 Read block 686 len 20480
30606 035/300 Write block 4035 len 36864 data: 4 4 5 5 6 6 6 7 7
30607 036/300 Read block 1359 len 16384
30608 037/300 Write block 3816 len 16384 data: 10 9 8 7
30609 038/300 Write block 561 len 12288 data: 8 8 7
30610 039/300 Read block 4450 len 16384
30611 040/300 Read block 487 len 32768
30612 041/300 Write block 1156 len 36864 data: 6 5 5 5 5 4 3 3 3
30613 042/300 Read block 4639 len 28672
30614 043/300 Write block 764 len 12288 data: 3 4 5
30615 044/300 Write block 427 len 12288 data: 6 6 6
30616 045/300 Write block 2952 len 32768 data: 7 8 6 6 7 6 5 6
30617 046/300 Read block 4372 len 12288
30618 047/300 Read block 1502 len 4096
30619 048/300 Flush
30620 049/300 Read block 183 len 12288
30621 050/300 Flush
30622 051/300 Read block 1320 len 8192
30623 052/300 Read block 3796 len 36864
30624 053/300 Read block 2959 len 24576
30625 054/300 Write block 1956 len 36864 data: 6 4 3 4 4 3 3 3 2
30626 055/300 Read block 1483 len 28672
30627 056/300 Read block 4249 len 4096
30628 057/300 Write block 1512 len 36864 data: 6 6 5 5 4 3 3 3 4
30629 058/300 Read block 1520 len 8192
30630 059/300 Write block 3946 len 32768 data: 7 9 8 8 9 7 7 5
30631 060/300 Write block 4900 len 32768 data: 4 5 8 7 7 7 7 6
30632 061/300 Write block 3346 len 20480 data: 3 4 4 5 6
30633 062/300 Flush
30634 063/300 Write block 366 len 40960 data: 6 5 5 6 5 6 5 5 5 4
30635 064/300 Write block 583 len 20480 data: 6 5 6 6 6
30636 065/300 Read block 624 len 24576
30637 066/300 Write block 3489 len 4096 data: 7
30638 067/300 Write block 263 len 24576 data: 6 5 5 7 5 5
30639 068/300 Write block 3677 len 40960 data: 8 10 10 9 10 10 10 11 9 10
30640 069/300 Read block 4308 len 12288
30641 070/300 Read block 2197 len 20480
30642 071/300 Write block 2712 len 12288 data: 6 8 7
30643 072/300 Read block 1809 len 40960
30644 073/300 Write block 4059 len 40960 data: 6 9 12 10 9 8 8 8 8 7
30645 074/300 Read block 3150 len 20480
30646 075/300 Read block 2689 len 16384
30647 076/300 Write block 2854 len 20480 data: 5 6 6 5 6
30648 077/300 Read block 1457 len 40960
30649 078/300 Read block 340 len 36864
30650 079/300 Write block 1466 len 8192 data: 5 5
30651 080/300 Read block 110 len 32768
30652 081/300 Read block 2264 len 12288
30653 082/300 Read block 2857 len 32768
30654 083/300 Read block 1621 len 32768
30655 084/300 Read block 1670 len 36864
30656 085/300 Read block 4150 len 12288
30657 086/300 Write block 403 len 8192 data: 2 2
30658 087/300 Write block 3410 len 4096 data: 6
30659 088/300 Write block 1203 len 16384 data: 4 4 5 5
30660 089/300 Read block 3697 len 24576
30661 090/300 Read block 63 len 12288
30662 091/300 Write block 1155 len 32768 data: 6 7 6 6 6 6 5 4
30663 092/300 Write block 387 len 36864 data: 6 6 7 7 6 4 4 3 2
30664 093/300 Read block 2816 len 20480
30665 094/300 Read block 2237 len 40960
30666 095/300 Read block 4587 len 4096
30667 096/300 Write block 1904 len 8192 data: 6 6
30668 097/300 Read block 2559 len 40960
30669 098/300 Write block 1525 len 8192 data: 6 7
30670 099/300 Write block 2558 len 12288 data: 4 5 3
30671 100/300 Write block 3443 len 8192 data: 4 6
30672 101/300 Flush
30673 102/300 Write block 4588 len 36864 data: 7 8 6 6 5 5 6 4 4
30674 103/300 Read block 2535 len 12288
30675 104/300 Read block 2813 len 8192
30676 105/300 Write block 4462 len 12288 data: 6 6 7
30677 106/300 Write block 87 len 16384 data: 5 5 4 3
30678 107/300 Read block 2809 len 8192
30679 108/300 Write block 1003 len 16384 data: 7 7 6 7
30680 109/300 Write block 1123 len 8192 data: 4 3
30681 110/300 Read block 3398 len 36864
30682 111/300 Write block 1842 len 16384 data: 3 5 5 7
30683 112/300 Read block 2335 len 16384
30684 113/300 Read block 3215 len 32768
30685 114/300 Read block 1619 len 12288
30686 115/300 Flush
30687 116/300 Write block 1652 len 36864 data: 8 8 5 6 6 7 9 7 6
30688 117/300 Read block 4120 len 32768
30689 118/300 Read block 2942 len 8192
30690 119/300 Read block 1305 len 4096
30691 120/300 Read block 3914 len 8192
30692 121/300 Read block 4624 len 20480
30693 122/300 Read block 110 len 32768
30694 123/300 Write block 1753 len 8192 data: 4 3
30695 124/300 Read block 2296 len 16384
30696 125/300 Read block 276 len 12288
30697 126/300 Read block 2724 len 32768
30698 127/300 Read block 4201 len 28672
30699 128/300 Read block 1870 len 24576
30700 129/300 Write block 4997 len 8192 data: 3 3
30701 130/300 Read block 2547 len 20480
30702 131/300 Read block 1779 len 12288
30703 132/300 Read block 425 len 12288
30704 133/300 Write block 2293 len 40960 data: 5 5 5 5 6 7 6 5 5 4
30705 134/300 Read block 1876 len 24576
30706 135/300 Write block 4461 len 28672 data: 6 7 7 8 9 10 10
30707 136/300 Read block 854 len 16384
30708 137/300 Read block 3470 len 28672
30709 138/300 Read block 3260 len 4096
30710 139/300 Read block 3944 len 24576
30711 140/300 Write block 585 len 28672 data: 7 7 7 5 5 3 4
30712 141/300 Read block 3862 len 36864
30713 142/300 Read block 1047 len 40960
30714 143/300 Write block 62 len 32768 data: 3 2 4 6 8 8 6 6
30715 144/300 Read block 807 len 20480
30716 145/300 Write block 1940 len 36864 data: 6 6 5 5 5 5 4 5 6
30717 146/300 Write block 1709 len 4096 data: 7
30718 147/300 Read block 1177 len 8192
30719 148/300 Write block 3643 len 40960 data: 4 4 3 3 3 4 5 5 5 5
30720 149/300 Read block 3409 len 32768
30721 150/300 Read block 1410 len 28672
30722 151/300 Read block 3217 len 20480
30723 152/300 Write block 2393 len 32768 data: 4 4 6 8 7 6 6 6
30724 153/300 Read block 3888 len 36864
30725 154/300 Read block 1440 len 32768
30726 155/300 Read block 265 len 16384
30727 156/300 Write block 4460 len 4096 data: 5
30728 157/300 Write block 2044 len 40960 data: 2 3 5 5 5 5 5 5 5 7
30729 158/300 Write block 1317 len 16384 data: 8 8 7 9
30730 159/300 Read block 3121 len 40960
30731 160/300 Read block 223 len 32768
30732 161/300 Read block 3188 len 16384
30733 162/300 Read block 4651 len 24576
30734 163/300 Read block 2840 len 12288
30735 164/300 Flush
30736 165/300 Flush
30737 166/300 Flush
30738 167/300 Write block 2400 len 28672 data: 7 7 3 3 2 3 4
30739 168/300 Read block 2968 len 40960
30740 169/300 Write block 3667 len 28672 data: 7 7 7 5 5 4 6
30741 170/300 Flush
30742 171/300 Read block 4462 len 12288
30743 172/300 Read block 707 len 28672
30744 173/300 Read block 1860 len 20480
30745 174/300 Write block 1181 len 16384 data: 4 4 4 3
30746 175/300 Read block 4820 len 32768
30747 176/300 Read block 2212 len 4096
30748 177/300 Read block 75 len 20480
30749 178/300 Flush
30750 179/300 Read block 1931 len 36864
30751 180/300 Read block 3460 len 20480
30752 181/300 Write block 2746 len 12288 data: 7 7 6
30753 182/300 Read block 2617 len 36864
30754 183/300 Write block 2603 len 40960 data: 8 8 8 6 7 6 4 5 4 4
30755 184/300 Write block 138 len 32768 data: 3 2 2 2 2 3 4 4
30756 185/300 Write block 4796 len 32768 data: 5 6 6 7 8 9 8 6
30757 186/300 Flush
30758 187/300 Read block 1615 len 24576
30759 188/300 Flush
30760 189/300 Read block 1372 len 4096
30761 190/300 Flush
30762 191/300 Read block 1943 len 36864
30763 192/300 Write block 2433 len 28672 data: 4 4 4 4 4 4 3
30764 193/300 Read block 3910 len 32768
30765 194/300 Read block 1726 len 8192
30766 195/300 Write block 3745 len 40960 data: 4 6 6 6 9 8 8 7 7 7
30767 196/300 Write block 186 len 20480 data: 7 6 6 6 6
30768 197/300 Read block 3583 len 12288
30769 198/300 Read block 521 len 28672
30770 199/300 Read block 1931 len 40960
30771 200/300 Write block 1350 len 8192 data: 5 6
30772 201/300 Write block 1254 len 12288 data: 6 5 4
30773 202/300 Read block 3035 len 16384
30774 203/300 Write block 1482 len 40960 data: 3 3 4 4 5 7 7 7 7 6
30775 204/300 Read block 2267 len 32768
30776 205/300 Write block 1468 len 16384 data: 5 5 5 5
30777 206/300 Read block 2740 len 40960
30778 207/300 Flush
30779 208/300 Write block 468 len 12288 data: 7 6 6
30780 209/300 Read block 2792 len 40960
30781 210/300 Read block 3847 len 4096
30782 211/300 Read block 2440 len 36864
30783 212/300 Read block 4753 len 36864
30784 213/300 Write block 2624 len 28672 data: 7 6 5 5 5 4 5
30785 214/300 Write block 4038 len 8192 data: 6 7
30786 215/300 Write block 3929 len 28672 data: 4 3 4 4 4 4 4
30787 216/300 Flush
30788 217/300 Write block 1598 len 8192 data: 4 4
30789 218/300 Read block 3192 len 32768
30790 219/300 Read block 2713 len 40960
30791 220/300 Write block 3199 len 40960 data: 4 4 4 5 7 7 8 6 5 6
30792 221/300 Read block 2969 len 28672
30793 222/300 Read block 3325 len 16384
30794 223/300 Write block 3858 len 4096 data: 5
30795 224/300 Read block 3403 len 8192
30796 225/300 Read block 3509 len 32768
30797 226/300 Write block 76 len 24576 data: 4 4 5 5 5 5
30798 227/300 Read block 3268 len 36864
30799 228/300 Read block 4454 len 16384
30800 229/300 Read block 1399 len 8192
30801 230/300 Read block 2262 len 16384
30802 231/300 Write block 3609 len 8192 data: 8 6
30803 232/300 Write block 1648 len 20480 data: 6 6 6 8 9
30804 233/300 Read block 4657 len 8192
30805 234/300 Write block 2959 len 16384 data: 7 7 7 6
30806 235/300 Read block 4959 len 40960
30807 236/300 Read block 4802 len 24576
30808 237/300 Read block 3566 len 8192
30809 238/300 Write block 2769 len 40960 data: 5 5 6 6 4 8 7 8 8 8
30810 239/300 Read block 2882 len 20480
30811 240/300 Read block 2115 len 40960
30812 241/300 Read block 3856 len 24576
30813 242/300 Write block 2868 len 16384 data: 5 4 4 3
30814 243/300 Write block 2482 len 4096 data: 2
30815 244/300 Flush
30816 245/300 Write block 2967 len 12288 data: 5 6 6
30817 246/300 Read block 1005 len 36864
30818 247/300 Read block 1520 len 24576
30819 248/300 Read block 4363 len 20480
30820 249/300 Write block 3898 len 36864 data: 6 6 5 6 6 6 6 8 6
30821 250/300 Read block 389 len 8192
30822 251/300 Write block 546 len 16384 data: 5 6 5 4
30823 252/300 Read block 4103 len 20480
30824 253/300 Read block 3861 len 16384
30825 254/300 Read block 3917 len 32768
30826 255/300 Write block 4087 len 16384 data: 4 5 5 5
30827 256/300 Write block 3703 len 4096 data: 7
30828 257/300 Write block 328 len 4096 data: 4
30829 258/300 Flush
30830 259/300 Write block 3732 len 24576 data: 5 6 7 6 4 3
30831 260/300 Write block 4554 len 28672 data: 4 5 5 4 4 3 2
30832 261/300 Flush
30833 262/300 Read block 591 len 40960
30834 263/300 Read block 3507 len 28672
30835 264/300 Flush
30836 265/300 Read block 1608 len 8192
30837 266/300 Write block 4159 len 20480 data: 7 7 5 4 5
30838 267/300 Read block 3542 len 16384
30839 268/300 Write block 1383 len 4096 data: 2
30840 269/300 Read block 1476 len 20480
30841 270/300 Write block 3446 len 32768 data: 6 5 4 4 3 3 3 4
30842 271/300 Read block 3093 len 36864
30843 272/300 Read block 2495 len 32768
30844 273/300 Flush
30845 274/300 Read block 3428 len 20480
30846 275/300 Flush
30847 276/300 Read block 2956 len 4096
30848 277/300 Read block 3408 len 32768
30849 278/300 Flush
30850 279/300 Write block 1030 len 28672 data: 4 3 2 2 4 4 4
30851 280/300 Read block 4881 len 20480
30852 281/300 Flush
30853 282/300 Read block 4456 len 36864
30854 283/300 Read block 1634 len 20480
30855 284/300 Write block 3391 len 24576 data: 3 6 6 9 8 8
30856 285/300 Read block 4064 len 36864
30857 286/300 Write block 4267 len 32768 data: 5 5 5 5 4 4 5 5
30858 287/300 Write block 1612 len 8192 data: 6 5
30859 288/300 Write block 2163 len 20480 data: 6 8 7 7 6
30860 289/300 Read block 443 len 32768
30861 290/300 Read block 3818 len 32768
30862 291/300 Flush
30863 292/300 Write block 2229 len 20480 data: 4 3 2 2 3
30864 293/300 Read block 4405 len 40960
30865 294/300 Write block 1149 len 12288 data: 3 2 3
30866 295/300 Read block 2151 len 4096
30867 296/300 Read block 4243 len 4096
30868 297/300 Write block 1315 len 24576 data: 5 9 9 9 8 10
30869 298/300 Write block 436 len 40960 data: 6 5 4 4 4 2 2 3 4 5
30870 299/300 Write block 4618 len 20480 data: 7 7 10 8 9
30871 300/300 Read block 4748 len 8192
30872 301/300 Read block 1513 len 36864
30873 Main task has work to do, go find it
30874 got dsc Start(1)
30875 start 1
30876 Feb 07 20:14:12.061 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/1, method: POST, req_id: 6d3f3180-345f-4bd3-b3be-5c7e1469f43e, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
30877 Replay: started 1, returned:Ok(())
30878 ----------------------------------------------------------------
30879 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30880 States: Active Offline Active
30881 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30882 8549 Acked Read 8 Done Sent Done false
30883 8550 Acked Read 2 Done Sent Done false
30884 8551 Acked Read 10 Done Sent Done false
30885 8552 Acked Read 2 Done Sent Done false
30886 8553 Acked Flush 0 Done Sent Done false
30887 8554 Acked Read 7 Done Sent Done false
30888 8555 Acked Read 5 Done Sent Done false
30889 8556 Acked Read 1 Done Sent Done false
30890 8557 Acked Read 10 Done Sent Done false
30891 8558 Acked Flush 0 Done Sent Done false
30892 8559 Acked Read 6 Done Sent Done false
30893 8560 Acked Write 4 Done Sent Done false
30894 8561 Acked Read 8 Done Sent Done false
30895 8562 Acked Flush 0 Done Sent Done false
30896 8563 Acked Write 6 Done Sent Done false
30897 8564 Acked Read 9 Done Sent Done false
30898 8565 Acked Write 6 Done Sent Done false
30899 8566 Acked Read 1 Done Sent Done false
30900 8567 Acked Read 3 Done Sent Done false
30901 8568 Acked Read 5 Done Sent Done false
30902 8569 Acked Write 2 Done Sent Done false
30903 8570 Acked Write 9 Done Sent Done false
30904 8571 Acked Write 1 Done Sent Done false
30905 8572 Acked Write 4 Done Sent Done false
30906 8573 Acked Write 9 Done Sent Done false
30907 8574 Acked Write 5 Done Sent Done false
30908 8575 Acked Read 6 Done Sent Done false
30909 8576 Acked Read 9 Done Sent Done false
30910 8577 Acked Write 9 Done Sent Done false
30911 8578 Acked Read 2 Done Sent Done false
30912 8579 Acked Flush 0 Done Sent Done false
30913 8580 Acked Read 7 Done Sent Done false
30914 8581 Acked Write 7 Done Sent Done false
30915 8582 Acked Read 5 Done Sent Done false
30916 8583 Acked Write 9 Done Sent Done false
30917 8584 Acked Read 4 Done Sent Done false
30918 8585 Acked Write 4 Done Sent Done false
30919 8586 Acked Write 3 Done Sent Done false
30920 8587 Acked Read 4 Done Sent Done false
30921 8588 Acked Read 8 Done Sent Done false
30922 8589 Acked Write 9 Done Sent Done false
30923 8590 Acked Read 7 Done Sent Done false
30924 8591 Acked Write 3 Done Sent Done false
30925 8592 Acked Write 3 Done Sent Done false
30926 8593 Acked Write 8 Done Sent Done false
30927 8594 Acked Read 3 Done Sent Done false
30928 8595 Acked Read 1 Done Sent Done false
30929 8596 Acked Flush 0 Done Sent Done false
30930 8597 Acked Read 3 Done Sent Done false
30931 8598 Acked Flush 0 Done Sent Done false
30932 8599 Acked Read 2 Done Sent Done false
30933 8600 Acked Read 9 Done Sent Done false
30934 8601 Acked Read 6 Done Sent Done false
30935 8602 Acked Write 9 Done Sent Done false
30936 8603 Acked Read 7 Done Sent Done false
30937 8604 Acked Read 1 Done Sent Done false
30938 8605 Acked Write 9 Done Sent Done false
30939 8606 Acked Read 2 Done Sent Done false
30940 8607 Acked Write 8 Done Sent Done false
30941 8608 Acked Write 8 Done Sent Done false
30942 8609 Acked Write 5 Done Sent Done false
30943 8610 Acked Flush 0 Done Sent Done false
30944 8611 Acked Write 10 Done Sent Done false
30945 8612 Acked Write 5 Done Sent Done false
30946 8613 Acked Read 6 Done Sent Done false
30947 8614 Acked Write 1 Done Sent Done false
30948 8615 Acked Write 6 Done Sent Done false
30949 8616 Acked Write 10 Done Sent Done false
30950 8617 Acked Read 3 Done Sent Done false
30951 8618 Acked Read 5 Done Sent Done false
30952 8619 Acked Write 3 Done Sent Done false
30953 8620 Acked Read 10 Done Sent Done false
30954 8621 Acked Write 10 Done Sent Done false
30955 8622 Acked Read 5 Done Sent Done false
30956 8623 Acked Read 4 Done Sent Done false
30957 8624 Acked Write 5 Done Sent Done false
30958 8625 Acked Read 10 Done Sent Done false
30959 8626 Acked Read 9 Done Sent Done false
30960 8627 Acked Write 2 Done Sent Done false
30961 8628 Acked Read 8 Done Sent Done false
30962 8629 Acked Read 3 Done Sent Done false
30963 8630 Acked Read 8 Done Sent Done false
30964 8631 Acked Read 8 Done Sent Done false
30965 8632 Acked Read 9 Done Sent Done false
30966 8633 Acked Read 3 Done Sent Done false
30967 8634 Acked Write 2 Done Sent Done false
30968 8635 Acked Write 1 Done Sent Done false
30969 8636 Acked Write 4 Done Sent Done false
30970 8637 Acked Read 6 Done Sent Done false
30971 8638 Acked Read 3 Done Sent Done false
30972 8639 Acked Write 8 Done Sent Done false
30973 8640 Acked Write 9 Done Sent Done false
30974 8641 Acked Read 5 Done Sent Done false
30975 8642 Acked Read 10 Done Sent Done false
30976 8643 Acked Read 1 Done Sent Done false
30977 8644 Acked Write 2 Done Sent Done false
30978 8645 Acked Read 10 Done Sent Done false
30979 8646 Acked Write 2 Done Sent Done false
30980 8647 Acked Write 3 Done Sent Done false
30981 8648 Acked Write 2 Done Sent Done false
30982 8649 Acked Flush 0 Done Sent Done false
30983 8650 Acked Write 9 Done Sent Done false
30984 8651 Acked Read 3 Done Sent Done false
30985 8652 Acked Read 2 Done Sent Done false
30986 8653 Acked Write 3 Done Sent Done false
30987 8654 Acked Write 4 Done Sent Done false
30988 8655 Acked Read 2 Done Sent Done false
30989 8656 Acked Write 4 Done Sent Done false
30990 8657 Acked Write 2 Done Sent Done false
30991 8658 Acked Read 9 Done Sent Done false
30992 8659 Acked Write 4 Done Sent Done false
30993 8660 Acked Read 4 Done Sent Done false
30994 8661 Acked Read 8 Done Sent Done false
30995 8662 Acked Read 3 Done Sent Done false
30996 8663 Acked Flush 0 Done Sent Done false
30997 8664 Acked Write 9 Done Sent Done false
30998 8665 Acked Read 8 Done Sent Done false
30999 8666 Acked Read 2 Done Sent Done false
31000 8667 Acked Read 1 Done Sent Done false
31001 8668 Acked Read 2 Done Sent Done false
31002 8669 Acked Read 5 Done Sent Done false
31003 8670 Acked Read 8 Done Sent Done false
31004 8671 Acked Write 2 Done Sent Done false
31005 8672 Acked Read 4 Done Sent Done false
31006 8673 Acked Read 3 Done Sent Done false
31007 8674 Acked Read 8 Done Sent Done false
31008 8675 Acked Read 7 Done Sent Done false
31009 8676 Acked Read 6 Done Sent Done false
31010 8677 Acked Write 2 Done Sent Done false
31011 8678 Acked Read 5 Done Sent Done false
31012 8679 Acked Read 3 Done Sent Done false
31013 8680 Acked Read 3 Done Sent Done false
31014 8681 Acked Write 10 Done Sent Done false
31015 8682 Acked Read 6 Done Sent Done false
31016 8683 Acked Write 7 Done Sent Done false
31017 8684 Acked Read 4 Done Sent Done false
31018 8685 Acked Read 7 Done Sent Done false
31019 8686 Acked Read 1 Done Sent Done false
31020 8687 Acked Read 6 Done Sent Done false
31021 8688 Acked Write 7 Done Sent Done false
31022 8689 Acked Read 9 Done Sent Done false
31023 8690 Acked Read 10 Done Sent Done false
31024 8691 Acked Write 8 Done Sent Done false
31025 8692 Acked Read 5 Done Sent Done false
31026 8693 Acked Write 9 Done Sent Done false
31027 8694 Acked Write 1 Done Sent Done false
31028 8695 Acked Read 2 Done Sent Done false
31029 8696 Acked Write 10 Done Sent Done false
31030 8697 Acked Read 8 Done Sent Done false
31031 8698 Acked Read 7 Done Sent Done false
31032 8699 Acked Read 5 Done Sent Done false
31033 8700 Acked Write 8 Done Sent Done false
31034 8701 Acked Read 9 Done Sent Done false
31035 8702 Acked Read 8 Done Sent Done false
31036 8703 Acked Read 4 Done Sent Done false
31037 8704 Acked Write 1 Done Sent Done false
31038 8705 Acked Write 10 Done Sent Done false
31039 8706 Acked Write 4 Done Sent Done false
31040 8707 Acked Read 10 Done Sent Done false
31041 8708 Acked Read 8 Done Sent Done false
31042 8709 Acked Read 4 Done Sent Done false
31043 8710 Acked Read 6 Done Sent Done false
31044 8711 Acked Read 3 Done Sent Done false
31045 8712 Acked Flush 0 Done Sent Done false
31046 8713 Acked Flush 0 Done Sent Done false
31047 8714 Acked Flush 0 Done Sent Done false
31048 8715 Acked Write 7 Done Sent Done false
31049 8716 Acked Read 10 Done Sent Done false
31050 8717 Acked Write 7 Done Sent Done false
31051 8718 Acked Flush 0 Done Sent Done false
31052 8719 Acked Read 3 Done Sent Done false
31053 8720 Acked Read 7 Done Sent Done false
31054 8721 Acked Read 5 Done Sent Done false
31055 8722 Acked Write 4 Done Sent Done false
31056 8723 Acked Read 8 Done Sent Done false
31057 8724 Acked Read 1 Done Sent Done false
31058 8725 Acked Read 5 Done Sent Done false
31059 8726 Acked Flush 0 Done Sent Done false
31060 8727 Acked Read 9 Done Sent Done false
31061 8728 Acked Read 5 Done Sent Done false
31062 8729 Acked Write 3 Done Sent Done false
31063 8730 Acked Read 9 Done Sent Done false
31064 8731 Acked Write 10 Done Sent Done false
31065 8732 Acked Write 8 Done Sent Done false
31066 8733 Acked Write 8 Done Sent Done false
31067 8734 Acked Flush 0 Done Sent Done false
31068 8735 Acked Read 6 Done Sent Done false
31069 8736 Acked Flush 0 Done Sent Done false
31070 8737 Acked Read 1 Done Sent Done false
31071 8738 Acked Flush 0 Done Sent Done false
31072 8739 Acked Read 9 Done Sent Done false
31073 8740 Acked Write 7 Done Sent Done false
31074 8741 Acked Read 8 Done Sent Done false
31075 8742 Acked Read 2 Done Sent Done false
31076 8743 Acked Write 10 Done Sent Done false
31077 8744 Acked Write 5 Done Sent Done false
31078 8745 Acked Read 3 Done Sent Done false
31079 8746 Acked Read 7 Done Sent Done false
31080 8747 Acked Read 10 Done Sent Done false
31081 8748 Acked Write 2 Done Sent Done false
31082 8749 Acked Write 3 Done Sent Done false
31083 8750 Acked Read 4 Done Sent Done false
31084 8751 Acked Write 10 Done Sent Done false
31085 8752 Acked Read 8 Done Sent Done false
31086 8753 Acked Write 4 Done Sent Done false
31087 8754 Acked Read 10 Done Sent Done false
31088 8755 Acked Flush 0 Done Sent Done false
31089 8756 Acked Write 3 Done Sent Done false
31090 8757 Acked Read 10 Done Sent Done false
31091 8758 Acked Read 1 Done Sent Done false
31092 8759 Acked Read 9 Done Sent Done false
31093 8760 Acked Read 9 Done Sent Done false
31094 8761 Acked Write 7 Done Sent Done false
31095 8762 Acked Write 2 Done Sent Done false
31096 8763 Acked Write 7 Done Sent Done false
31097 8764 Acked Flush 0 Done Sent Done false
31098 8765 Acked Write 2 Done Sent Done false
31099 8766 Acked Read 8 Done Sent Done false
31100 8767 Acked Read 10 Done Sent Done false
31101 8768 Acked Write 10 Done Sent Done false
31102 8769 Acked Read 7 Done Sent Done false
31103 8770 Acked Read 4 Done Sent Done false
31104 8771 Acked Write 1 Done Sent Done false
31105 8772 Acked Read 2 Done Sent Done false
31106 8773 Acked Read 8 Done Sent Done false
31107 8774 Acked Write 6 Done Sent Done false
31108 8775 Acked Read 9 Done Sent Done false
31109 8776 Acked Read 4 Done Sent Done false
31110 8777 Acked Read 2 Done Sent Done false
31111 8778 Acked Read 4 Done Sent Done false
31112 8779 Acked Write 2 Done Sent Done false
31113 8780 Acked Write 5 Done Sent Done false
31114 8781 Acked Read 2 Done Sent Done false
31115 8782 Acked Write 4 Done Sent Done false
31116 8783 Acked Read 10 Done Sent Done false
31117 8784 Acked Read 6 Done Sent Done false
31118 8785 Acked Read 2 Done Sent Done false
31119 8786 Acked Write 10 Done Sent Done false
31120 8787 Acked Read 5 Done Sent Done false
31121 8788 Acked Read 10 Done Sent Done false
31122 8789 Acked Read 6 Done Sent Done false
31123 8790 Acked Write 4 Done Sent Done false
31124 8791 Acked Write 1 Done Sent Done false
31125 8792 Acked Flush 0 Done Sent Done false
31126 8793 Acked Write 3 Done Sent Done false
31127 8794 Acked Read 9 Done Sent Done false
31128 8795 Acked Read 6 Done Sent Done false
31129 8796 Acked Read 5 Done Sent Done false
31130 8797 Acked Write 9 Done Sent Done false
31131 8798 Acked Read 2 Done Sent Done false
31132 8799 Acked Write 4 Done Sent Done false
31133 8800 Acked Read 5 Done Sent Done false
31134 8801 Acked Read 4 Done Sent Done false
31135 8802 Acked Read 8 Done Sent Done false
31136 8803 Acked Write 4 Done Sent Done false
31137 8804 Acked Write 1 Done Sent Done false
31138 8805 Acked Write 1 Done Sent Done false
31139 8806 Acked Flush 0 Done Sent Done false
31140 8807 Acked Write 6 Done Sent Done false
31141 8808 Acked Write 7 Done Sent Done false
31142 8809 Acked Flush 0 Done Sent Done false
31143 8810 Acked Read 10 Done Sent Done false
31144 8811 Acked Read 7 Done Sent Done false
31145 8812 Acked Flush 0 Done Sent Done false
31146 8813 Acked Read 2 Done Sent Done false
31147 8814 Acked Write 5 Done Sent Done false
31148 8815 Acked Read 4 Done Sent Done false
31149 8816 Acked Write 1 Done Sent Done false
31150 8817 Acked Read 5 Done Sent Done false
31151 8818 Acked Write 8 Done Sent Done false
31152 8819 Acked Read 9 Done Sent Done false
31153 8820 Acked Read 8 Done Sent Done false
31154 8821 Acked Flush 0 Done Sent Done false
31155 8822 Acked Read 5 Done Sent Done false
31156 8823 Acked Flush 0 Done Sent Done false
31157 8824 Acked Read 1 Done Sent Done false
31158 8825 Acked Read 8 Done Sent Done false
31159 8826 Acked Flush 0 Done Sent Done false
31160 8827 Acked Write 7 Done Sent Done false
31161 8828 Acked Read 5 Done Sent Done false
31162 8829 Acked Flush 0 Done Sent Done false
31163 8830 Acked Read 9 Done Sent Done false
31164 8831 Acked Read 5 Done Sent Done false
31165 8832 Acked Write 6 Done Sent Done false
31166 8833 Acked Read 9 Done Sent Done false
31167 8834 Acked Write 8 Done Sent Done false
31168 8835 Acked Write 2 Done Sent Done false
31169 8836 Acked Write 5 Done Sent Done false
31170 8837 Acked Read 8 Done Sent Done false
31171 8838 Acked Read 8 Done Sent Done false
31172 8839 Acked Flush 0 Done Sent Done false
31173 8840 Acked Write 5 Done Sent Done false
31174 8841 Acked Read 10 Done Sent Done false
31175 8842 Acked Write 3 Done Sent Done false
31176 8843 Acked Read 1 Done Sent Done false
31177 8844 Acked Read 1 Done Sent Done false
31178 8845 Acked Write 6 Done Sent Done false
31179 8846 Acked Write 10 Done Sent Done false
31180 8847 Acked Write 5 Done Sent Done false
31181 8848 Acked Read 2 Done Sent Done false
31182 8849 Acked Read 9 Done Sent Done false
31183 STATES DS:0 DS:1 DS:2 TOTAL
31184 Sent 0 301 0 301
31185 Done 301 0 301 602
31186 Skipped 0 0 0 0
31187 Error 0 0 0 0
31188 Last Flush: Some(JobId(8839)) Some(JobId(8548)) Some(JobId(8839))
31189 Downstairs last five completed: 8548 8547 8546 8545 8544
31190 Upstairs last five completed: 8849 8848 8847 8846 8845
31191 CLIENT: Up:0 ds:301 act:2
311922025-02-07T20:14:12.613ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
311932025-02-07T20:14:13.799ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
311942025-02-07T20:14:14.617ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
31195 [8820] got start action
31196 [8820] I am going to restart
31197 Starting downstairs at port 8820
31198 Make output file at "/tmp/dsc/downstairs-8820.txt"
31199 [8820][1] reports Starting
31200 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1235)
31201 [8820][1] reports Running
312022025-02-07T20:14:16.053ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
31203 ----------------------------------------------------------------
31204 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31205 States: Active Offline Active
31206 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31207 8549 Acked Read 8 Done Sent Done false
31208 8550 Acked Read 2 Done Sent Done false
31209 8551 Acked Read 10 Done Sent Done false
31210 8552 Acked Read 2 Done Sent Done false
31211 8553 Acked Flush 0 Done Sent Done false
31212 8554 Acked Read 7 Done Sent Done false
31213 8555 Acked Read 5 Done Sent Done false
31214 8556 Acked Read 1 Done Sent Done false
31215 8557 Acked Read 10 Done Sent Done false
31216 8558 Acked Flush 0 Done Sent Done false
31217 8559 Acked Read 6 Done Sent Done false
31218 8560 Acked Write 4 Done Sent Done false
31219 8561 Acked Read 8 Done Sent Done false
31220 8562 Acked Flush 0 Done Sent Done false
31221 8563 Acked Write 6 Done Sent Done false
31222 8564 Acked Read 9 Done Sent Done false
31223 8565 Acked Write 6 Done Sent Done false
31224 8566 Acked Read 1 Done Sent Done false
31225 8567 Acked Read 3 Done Sent Done false
31226 8568 Acked Read 5 Done Sent Done false
31227 8569 Acked Write 2 Done Sent Done false
31228 8570 Acked Write 9 Done Sent Done false
31229 8571 Acked Write 1 Done Sent Done false
31230 8572 Acked Write 4 Done Sent Done false
31231 8573 Acked Write 9 Done Sent Done false
31232 8574 Acked Write 5 Done Sent Done false
31233 8575 Acked Read 6 Done Sent Done false
31234 8576 Acked Read 9 Done Sent Done false
31235 8577 Acked Write 9 Done Sent Done false
31236 8578 Acked Read 2 Done Sent Done false
31237 8579 Acked Flush 0 Done Sent Done false
31238 8580 Acked Read 7 Done Sent Done false
31239 8581 Acked Write 7 Done Sent Done false
31240 8582 Acked Read 5 Done Sent Done false
31241 8583 Acked Write 9 Done Sent Done false
31242 8584 Acked Read 4 Done Sent Done false
31243 8585 Acked Write 4 Done Sent Done false
31244 8586 Acked Write 3 Done Sent Done false
31245 8587 Acked Read 4 Done Sent Done false
31246 8588 Acked Read 8 Done Sent Done false
31247 8589 Acked Write 9 Done Sent Done false
31248 8590 Acked Read 7 Done Sent Done false
31249 8591 Acked Write 3 Done Sent Done false
31250 8592 Acked Write 3 Done Sent Done false
31251 8593 Acked Write 8 Done Sent Done false
31252 8594 Acked Read 3 Done Sent Done false
31253 8595 Acked Read 1 Done Sent Done false
31254 8596 Acked Flush 0 Done Sent Done false
31255 8597 Acked Read 3 Done Sent Done false
31256 8598 Acked Flush 0 Done Sent Done false
31257 8599 Acked Read 2 Done Sent Done false
31258 8600 Acked Read 9 Done Sent Done false
31259 8601 Acked Read 6 Done Sent Done false
31260 8602 Acked Write 9 Done Sent Done false
31261 8603 Acked Read 7 Done Sent Done false
31262 8604 Acked Read 1 Done Sent Done false
31263 8605 Acked Write 9 Done Sent Done false
31264 8606 Acked Read 2 Done Sent Done false
31265 8607 Acked Write 8 Done Sent Done false
31266 8608 Acked Write 8 Done Sent Done false
31267 8609 Acked Write 5 Done Sent Done false
31268 8610 Acked Flush 0 Done Sent Done false
31269 8611 Acked Write 10 Done Sent Done false
31270 8612 Acked Write 5 Done Sent Done false
31271 8613 Acked Read 6 Done Sent Done false
31272 8614 Acked Write 1 Done Sent Done false
31273 8615 Acked Write 6 Done Sent Done false
31274 8616 Acked Write 10 Done Sent Done false
31275 8617 Acked Read 3 Done Sent Done false
31276 8618 Acked Read 5 Done Sent Done false
31277 8619 Acked Write 3 Done Sent Done false
31278 8620 Acked Read 10 Done Sent Done false
31279 8621 Acked Write 10 Done Sent Done false
31280 8622 Acked Read 5 Done Sent Done false
31281 8623 Acked Read 4 Done Sent Done false
31282 8624 Acked Write 5 Done Sent Done false
31283 8625 Acked Read 10 Done Sent Done false
31284 8626 Acked Read 9 Done Sent Done false
31285 8627 Acked Write 2 Done Sent Done false
31286 8628 Acked Read 8 Done Sent Done false
31287 8629 Acked Read 3 Done Sent Done false
31288 8630 Acked Read 8 Done Sent Done false
31289 8631 Acked Read 8 Done Sent Done false
31290 8632 Acked Read 9 Done Sent Done false
31291 8633 Acked Read 3 Done Sent Done false
31292 8634 Acked Write 2 Done Sent Done false
31293 8635 Acked Write 1 Done Sent Done false
31294 8636 Acked Write 4 Done Sent Done false
31295 8637 Acked Read 6 Done Sent Done false
31296 8638 Acked Read 3 Done Sent Done false
31297 8639 Acked Write 8 Done Sent Done false
31298 8640 Acked Write 9 Done Sent Done false
31299 8641 Acked Read 5 Done Sent Done false
31300 8642 Acked Read 10 Done Sent Done false
31301 8643 Acked Read 1 Done Sent Done false
31302 8644 Acked Write 2 Done Sent Done false
31303 8645 Acked Read 10 Done Sent Done false
31304 8646 Acked Write 2 Done Sent Done false
31305 8647 Acked Write 3 Done Sent Done false
31306 8648 Acked Write 2 Done Sent Done false
31307 8649 Acked Flush 0 Done Sent Done false
31308 8650 Acked Write 9 Done Sent Done false
31309 8651 Acked Read 3 Done Sent Done false
31310 8652 Acked Read 2 Done Sent Done false
31311 8653 Acked Write 3 Done Sent Done false
31312 8654 Acked Write 4 Done Sent Done false
31313 8655 Acked Read 2 Done Sent Done false
31314 8656 Acked Write 4 Done Sent Done false
31315 8657 Acked Write 2 Done Sent Done false
31316 8658 Acked Read 9 Done Sent Done false
31317 8659 Acked Write 4 Done Sent Done false
31318 8660 Acked Read 4 Done Sent Done false
31319 8661 Acked Read 8 Done Sent Done false
31320 8662 Acked Read 3 Done Sent Done false
31321 8663 Acked Flush 0 Done Sent Done false
31322 8664 Acked Write 9 Done Sent Done false
31323 8665 Acked Read 8 Done Sent Done false
31324 8666 Acked Read 2 Done Sent Done false
31325 8667 Acked Read 1 Done Sent Done false
31326 8668 Acked Read 2 Done Sent Done false
31327 8669 Acked Read 5 Done Sent Done false
31328 8670 Acked Read 8 Done Sent Done false
31329 8671 Acked Write 2 Done Sent Done false
31330 8672 Acked Read 4 Done Sent Done false
31331 8673 Acked Read 3 Done Sent Done false
31332 8674 Acked Read 8 Done Sent Done false
31333 8675 Acked Read 7 Done Sent Done false
31334 8676 Acked Read 6 Done Sent Done false
31335 8677 Acked Write 2 Done Sent Done false
31336 8678 Acked Read 5 Done Sent Done false
31337 8679 Acked Read 3 Done Sent Done false
31338 8680 Acked Read 3 Done Sent Done false
31339 8681 Acked Write 10 Done Sent Done false
31340 8682 Acked Read 6 Done Sent Done false
31341 8683 Acked Write 7 Done Sent Done false
31342 8684 Acked Read 4 Done Sent Done false
31343 8685 Acked Read 7 Done Sent Done false
31344 8686 Acked Read 1 Done Sent Done false
31345 8687 Acked Read 6 Done Sent Done false
31346 8688 Acked Write 7 Done Sent Done false
31347 8689 Acked Read 9 Done Sent Done false
31348 8690 Acked Read 10 Done Sent Done false
31349 8691 Acked Write 8 Done Sent Done false
31350 8692 Acked Read 5 Done Sent Done false
31351 8693 Acked Write 9 Done Sent Done false
31352 8694 Acked Write 1 Done Sent Done false
31353 8695 Acked Read 2 Done Sent Done false
31354 8696 Acked Write 10 Done Sent Done false
31355 8697 Acked Read 8 Done Sent Done false
31356 8698 Acked Read 7 Done Sent Done false
31357 8699 Acked Read 5 Done Sent Done false
31358 8700 Acked Write 8 Done Sent Done false
31359 8701 Acked Read 9 Done Sent Done false
31360 8702 Acked Read 8 Done Sent Done false
31361 8703 Acked Read 4 Done Sent Done false
31362 8704 Acked Write 1 Done Sent Done false
31363 8705 Acked Write 10 Done Sent Done false
31364 8706 Acked Write 4 Done Sent Done false
31365 8707 Acked Read 10 Done Sent Done false
31366 8708 Acked Read 8 Done Sent Done false
31367 8709 Acked Read 4 Done Sent Done false
31368 8710 Acked Read 6 Done Sent Done false
31369 8711 Acked Read 3 Done Sent Done false
31370 8712 Acked Flush 0 Done Sent Done false
31371 8713 Acked Flush 0 Done Sent Done false
31372 8714 Acked Flush 0 Done Sent Done false
31373 8715 Acked Write 7 Done Sent Done false
31374 8716 Acked Read 10 Done Sent Done false
31375 8717 Acked Write 7 Done Sent Done false
31376 8718 Acked Flush 0 Done Sent Done false
31377 8719 Acked Read 3 Done Sent Done false
31378 8720 Acked Read 7 Done Sent Done false
31379 8721 Acked Read 5 Done Sent Done false
31380 8722 Acked Write 4 Done Sent Done false
31381 8723 Acked Read 8 Done Sent Done false
31382 8724 Acked Read 1 Done Sent Done false
31383 8725 Acked Read 5 Done Sent Done false
31384 8726 Acked Flush 0 Done Sent Done false
31385 8727 Acked Read 9 Done Sent Done false
31386 8728 Acked Read 5 Done Sent Done false
31387 8729 Acked Write 3 Done Sent Done false
31388 8730 Acked Read 9 Done Sent Done false
31389 8731 Acked Write 10 Done Sent Done false
31390 8732 Acked Write 8 Done Sent Done false
31391 8733 Acked Write 8 Done Sent Done false
31392 8734 Acked Flush 0 Done Sent Done false
31393 8735 Acked Read 6 Done Sent Done false
31394 8736 Acked Flush 0 Done Sent Done false
31395 8737 Acked Read 1 Done Sent Done false
31396 8738 Acked Flush 0 Done Sent Done false
31397 8739 Acked Read 9 Done Sent Done false
31398 8740 Acked Write 7 Done Sent Done false
31399 8741 Acked Read 8 Done Sent Done false
31400 8742 Acked Read 2 Done Sent Done false
31401 8743 Acked Write 10 Done Sent Done false
31402 8744 Acked Write 5 Done Sent Done false
31403 8745 Acked Read 3 Done Sent Done false
31404 8746 Acked Read 7 Done Sent Done false
31405 8747 Acked Read 10 Done Sent Done false
31406 8748 Acked Write 2 Done Sent Done false
31407 8749 Acked Write 3 Done Sent Done false
31408 8750 Acked Read 4 Done Sent Done false
31409 8751 Acked Write 10 Done Sent Done false
31410 8752 Acked Read 8 Done Sent Done false
31411 8753 Acked Write 4 Done Sent Done false
31412 8754 Acked Read 10 Done Sent Done false
31413 8755 Acked Flush 0 Done Sent Done false
31414 8756 Acked Write 3 Done Sent Done false
31415 8757 Acked Read 10 Done Sent Done false
31416 8758 Acked Read 1 Done Sent Done false
31417 8759 Acked Read 9 Done Sent Done false
31418 8760 Acked Read 9 Done Sent Done false
31419 8761 Acked Write 7 Done Sent Done false
31420 8762 Acked Write 2 Done Sent Done false
31421 8763 Acked Write 7 Done Sent Done false
31422 8764 Acked Flush 0 Done Sent Done false
31423 8765 Acked Write 2 Done Sent Done false
31424 8766 Acked Read 8 Done Sent Done false
31425 8767 Acked Read 10 Done Sent Done false
31426 8768 Acked Write 10 Done Sent Done false
31427 8769 Acked Read 7 Done Sent Done false
31428 8770 Acked Read 4 Done Sent Done false
31429 8771 Acked Write 1 Done Sent Done false
31430 8772 Acked Read 2 Done Sent Done false
31431 8773 Acked Read 8 Done Sent Done false
31432 8774 Acked Write 6 Done Sent Done false
31433 8775 Acked Read 9 Done Sent Done false
31434 8776 Acked Read 4 Done Sent Done false
31435 8777 Acked Read 2 Done Sent Done false
31436 8778 Acked Read 4 Done Sent Done false
31437 8779 Acked Write 2 Done Sent Done false
31438 8780 Acked Write 5 Done Sent Done false
31439 8781 Acked Read 2 Done Sent Done false
31440 8782 Acked Write 4 Done Sent Done false
31441 8783 Acked Read 10 Done Sent Done false
31442 8784 Acked Read 6 Done Sent Done false
31443 8785 Acked Read 2 Done Sent Done false
31444 8786 Acked Write 10 Done Sent Done false
31445 8787 Acked Read 5 Done Sent Done false
31446 8788 Acked Read 10 Done Sent Done false
31447 8789 Acked Read 6 Done Sent Done false
31448 8790 Acked Write 4 Done Sent Done false
31449 8791 Acked Write 1 Done Sent Done false
31450 8792 Acked Flush 0 Done Sent Done false
31451 8793 Acked Write 3 Done Sent Done false
31452 8794 Acked Read 9 Done Sent Done false
31453 8795 Acked Read 6 Done Sent Done false
31454 8796 Acked Read 5 Done Sent Done false
31455 8797 Acked Write 9 Done Sent Done false
31456 8798 Acked Read 2 Done Sent Done false
31457 8799 Acked Write 4 Done Sent Done false
31458 8800 Acked Read 5 Done Sent Done false
31459 8801 Acked Read 4 Done Sent Done false
31460 8802 Acked Read 8 Done Sent Done false
31461 8803 Acked Write 4 Done Sent Done false
31462 8804 Acked Write 1 Done Sent Done false
31463 8805 Acked Write 1 Done Sent Done false
31464 8806 Acked Flush 0 Done Sent Done false
31465 8807 Acked Write 6 Done Sent Done false
31466 8808 Acked Write 7 Done Sent Done false
31467 8809 Acked Flush 0 Done Sent Done false
31468 8810 Acked Read 10 Done Sent Done false
31469 8811 Acked Read 7 Done Sent Done false
31470 8812 Acked Flush 0 Done Sent Done false
31471 8813 Acked Read 2 Done Sent Done false
31472 8814 Acked Write 5 Done Sent Done false
31473 8815 Acked Read 4 Done Sent Done false
31474 8816 Acked Write 1 Done Sent Done false
31475 8817 Acked Read 5 Done Sent Done false
31476 8818 Acked Write 8 Done Sent Done false
31477 8819 Acked Read 9 Done Sent Done false
31478 8820 Acked Read 8 Done Sent Done false
31479 8821 Acked Flush 0 Done Sent Done false
31480 8822 Acked Read 5 Done Sent Done false
31481 8823 Acked Flush 0 Done Sent Done false
31482 8824 Acked Read 1 Done Sent Done false
31483 8825 Acked Read 8 Done Sent Done false
31484 8826 Acked Flush 0 Done Sent Done false
31485 8827 Acked Write 7 Done Sent Done false
31486 8828 Acked Read 5 Done Sent Done false
31487 8829 Acked Flush 0 Done Sent Done false
31488 8830 Acked Read 9 Done Sent Done false
31489 8831 Acked Read 5 Done Sent Done false
31490 8832 Acked Write 6 Done Sent Done false
31491 8833 Acked Read 9 Done Sent Done false
31492 8834 Acked Write 8 Done Sent Done false
31493 8835 Acked Write 2 Done Sent Done false
31494 8836 Acked Write 5 Done Sent Done false
31495 8837 Acked Read 8 Done Sent Done false
31496 8838 Acked Read 8 Done Sent Done false
31497 8839 Acked Flush 0 Done Sent Done false
31498 8840 Acked Write 5 Done Sent Done false
31499 8841 Acked Read 10 Done Sent Done false
31500 8842 Acked Write 3 Done Sent Done false
31501 8843 Acked Read 1 Done Sent Done false
31502 8844 Acked Read 1 Done Sent Done false
31503 8845 Acked Write 6 Done Sent Done false
31504 8846 Acked Write 10 Done Sent Done false
31505 8847 Acked Write 5 Done Sent Done false
31506 8848 Acked Read 2 Done Sent Done false
31507 8849 Acked Read 9 Done Sent Done false
31508 STATES DS:0 DS:1 DS:2 TOTAL
31509 Sent 0 301 0 301
31510 Done 301 0 301 602
31511 Skipped 0 0 0 0
31512 Error 0 0 0 0
31513 Last Flush: Some(JobId(8839)) Some(JobId(8548)) Some(JobId(8839))
31514 Downstairs last five completed: 8548 8547 8546 8545 8544
31515 Upstairs last five completed: 8849 8848 8847 8846 8845
31516 CLIENT: Up:0 ds:301 act:2
315172025-02-07T20:14:16.621ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
315182025-02-07T20:14:17.623ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
315192025-02-07T20:14:17.802ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
315202025-02-07T20:14:17.802ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38463) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
315212025-02-07T20:14:17.803ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
315222025-02-07T20:14:17.804ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
315232025-02-07T20:14:17.804ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
315242025-02-07T20:14:17.804ZINFOcrucible: send last flush ID to this DS: Some(JobId(8548)) = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
315252025-02-07T20:14:17.804ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
315262025-02-07T20:14:17.804ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(8548)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
315272025-02-07T20:14:17.804ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(8548)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
31528 ----------------------------------------------------------------
31529 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31530 Downstairs last five completed: 8850 8849 8848 8847 8846
31531 Upstairs last five completed: 8850 8849 8848 8847 8846
31532 CLIENT: Up:0 ds:0 act:3
31533 Replay: All jobs finished
31534 Main task has work to do, go find it
31535 got dsc Stop(1)
31536 stop 1
31537 Feb 07 20:14:20.067[8820] Got stop action so:false kr:false
31538 INFO request completed, latency_us: 473, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 281ffc2d-28af-4471-b97b-b3f28267b834, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
31539 Feb 07 20:14:20.067 INFO request completed, latency_us: 127, response_code: 200, {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:14:20.067822326Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"1","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
31540 uri: /state/cid/1, method: GET, req_id: 04b16492-a18e-4ca8-9cad-315d8cd7ca19, remote_addr: 127.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:14:20.067874372Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"1","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
31541 0{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-07T20:14:20.06790105Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
31542 .0.1:43825, local_addr: 127.0.0.1:9998
315432025-02-07T20:14:20.067ZINFOcrucible: client task is exiting = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
31544 [8820] Exited with: ExitStatus(unix_wait_status(9))
31545 [8820][1] reports Exit
31546 Feb 07 20:14:24.070 INFO request completed, latency_us: 363, response_code: 200, uri: /state/cid/1, method: GET, req_id: 8825f23f-0a88-406b-8287-7539fbcd0af8, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
31547 001/300 Read block 3181 len 24576
31548 002/300 Flush
31549 003/300 Read block 4145 len 40960
31550 004/300 Read block 4447 len 32768
31551 005/300 Read block 2018 len 40960
31552 006/300 Write block 4276 len 20480 data: 6 5 5 5 4
31553 007/300 Write block 4592 len 8192 data: 6 6
31554 008/300 Read block 225 len 40960
31555 009/300 Write block 4858 len 8192 data: 4 5
31556 010/300 Flush
31557 011/300 Write block 219 len 12288 data: 7 7 8
31558 012/300 Write block 3798 len 36864 data: 5 7 7 7 6 6 7 7 6
31559 013/300 Flush
31560 014/300 Read block 3091 len 20480
31561 015/300 Write block 4226 len 36864 data: 6 6 6 6 4 3 3 3 5
31562 016/300 Read block 1346 len 24576
31563 017/300 Write block 3523 len 20480 data: 5 4 4 4 4
31564 018/300 Read block 4993 len 8192
31565 019/300 Read block 3338 len 20480
31566 020/300 Read block 1055 len 24576
31567 021/300 Write block 2219 len 8192 data: 3 3
31568 022/300 Write block 1818 len 36864 data: 5 5 5 6 5 5 5 6 6
31569 023/300 Read block 1097 len 24576
31570 024/300 Read block 2977 len 32768
31571 025/300 Write block 1164 len 32768 data: 4 3 3 3 3 4 3 3
31572 026/300 Flush
31573 027/300 Read block 1448 len 4096
31574 028/300 Read block 3572 len 24576
31575 029/300 Read block 798 len 8192
31576 030/300 Write block 3519 len 12288 data: 2 4 4
31577 031/300 Write block 2975 len 40960 data: 5 5 6 7 6 7 9 8 6 4
31578 032/300 Read block 77 len 20480
31579 033/300 Write block 2549 len 32768 data: 3 3 4 4 3 3 4 4
31580 034/300 Read block 2167 len 16384
31581 035/300 Write block 4150 len 20480 data: 4 5 4 4 4
31582 036/300 Write block 4291 len 12288 data: 6 5 4
31583 037/300 Flush
31584 038/300 Flush
31585 039/300 Read block 114 len 8192
31586 040/300 Write block 2510 len 20480 data: 5 5 3 3 4
31587 041/300 Read block 1403 len 32768
31588 042/300 Write block 1726 len 32768 data: 3 2 2 2 3 3 3 3
31589 043/300 Write block 2406 len 24576 data: 5 4 3 4 4 4
31590 044/300 Write block 1425 len 36864 data: 4 4 3 3 3 3 3 3 5
31591 045/300 Write block 2137 len 16384 data: 3 3 3 4
31592 046/300 Write block 874 len 24576 data: 2 2 2 3 3 3
31593 047/300 Write block 592 len 32768 data: 4 4 4 4 3 4 4 4
31594 048/300 Read block 3113 len 8192
31595 049/300 Write block 610 len 28672 data: 4 4 7 7 7 5 5
31596 050/300 Write block 4139 len 12288 data: 4 4 5
31597 051/300 Flush
31598 052/300 Write block 3501 len 40960 data: 4 5 3 2 3 3 2 2 2 3
31599 053/300 Write block 4560 len 28672 data: 3 3 4 4 4 6 6
31600 054/300 Read block 4985 len 40960
31601 055/300 Flush
31602 056/300 Write block 350 len 16384 data: 2 2 2 2
31603 057/300 Flush
31604 058/300 Read block 2543 len 8192
31605 059/300 Read block 4031 len 12288
31606 060/300 Read block 2066 len 32768
31607 061/300 Read block 3051 len 24576
31608 062/300 Read block 3248 len 36864
31609 063/300 Write block 3511 len 8192 data: 3 4
31610 064/300 Read block 2475 len 20480
31611 065/300 Read block 2872 len 12288
31612 066/300 Read block 2663 len 4096
31613 067/300 Read block 1651 len 24576
31614 068/300 Read block 2106 len 32768
31615 069/300 Read block 3416 len 24576
31616 070/300 Read block 1494 len 24576
31617 071/300 Read block 598 len 12288
31618 072/300 Write block 3753 len 4096 data: 8
31619 073/300 Read block 545 len 4096
31620 074/300 Write block 4649 len 32768 data: 7 8 8 6 7 7 7 6
31621 075/300 Read block 4780 len 4096
31622 076/300 Read block 2342 len 20480
31623 077/300 Read block 845 len 36864
31624 078/300 Write block 1967 len 36864 data: 3 3 3 3 4 5 5 5 5
31625 079/300 Read block 328 len 32768
31626 080/300 Read block 1995 len 28672
31627 081/300 Write block 1282 len 20480 data: 4 5 5 5 5
31628 082/300 Write block 3465 len 32768 data: 3 2 2 2 2 3 4 3
31629 083/300 Read block 2255 len 40960
31630 084/300 Write block 2182 len 16384 data: 4 4 3 2
31631 085/300 Write block 4754 len 24576 data: 8 8 9 8 5 5
31632 086/300 Flush
31633 087/300 Read block 133 len 20480
31634 088/300 Read block 3214 len 32768
31635 089/300 Read block 404 len 32768
31636 090/300 Write block 3141 len 28672 data: 8 7 7 6 7 7 5
31637 091/300 Write block 3715 len 28672 data: 5 6 6 5 6 8 7
31638 092/300 Read block 26 len 16384
31639 093/300 Read block 1020 len 24576
31640 094/300 Write block 959 len 12288 data: 4 6 7
31641 095/300 Flush
31642 096/300 Write block 732 len 40960 data: 3 4 6 6 5 4 5 5 4 4
31643 097/300 Read block 105 len 28672
31644 098/300 Write block 2919 len 4096 data: 4
31645 099/300 Read block 3961 len 28672
31646 100/300 Read block 167 len 20480
31647 101/300 Read block 1717 len 24576
31648 102/300 Read block 3575 len 40960
31649 103/300 Write block 1281 len 16384 data: 3 5 6 6
31650 104/300 Write block 3970 len 32768 data: 4 3 3 3 2 2 3 3
31651 105/300 Read block 4861 len 8192
31652 106/300 Read block 1066 len 16384
31653 107/300 Write block 404 len 40960 data: 3 2 2 4 6 5 5 4 5 4
31654 108/300 Read block 3266 len 16384
31655 109/300 Read block 1906 len 4096
31656 110/300 Read block 1435 len 4096
31657 111/300 Read block 2631 len 4096
31658 112/300 Write block 1213 len 32768 data: 6 5 5 5 5 6 6 7
31659 113/300 Read block 1345 len 36864
31660 114/300 Read block 2600 len 32768
31661 115/300 Read block 4500 len 8192
31662 116/300 Write block 2147 len 20480 data: 3 3 2 3 4
31663 117/300 Read block 2500 len 40960
31664 118/300 Read block 481 len 40960
31665 119/300 Write block 4021 len 32768 data: 3 4 5 8 6 6 6 6
31666 120/300 Read block 3055 len 40960
31667 121/300 Read block 1440 len 32768
31668 122/300 Write block 3621 len 16384 data: 3 4 4 2
31669 123/300 Read block 1978 len 12288
31670 124/300 Read block 802 len 24576
31671 125/300 Read block 1495 len 8192
31672 126/300 Flush
31673 127/300 Write block 26 len 28672 data: 4 5 5 6 4 5 5
31674 128/300 Read block 2312 len 36864
31675 129/300 Read block 1949 len 20480
31676 130/300 Read block 1866 len 32768
31677 131/300 Write block 2710 len 32768 data: 6 5 7 9 8 6 4 5
31678 132/300 Read block 4187 len 4096
31679 133/300 Read block 4210 len 28672
31680 134/300 Flush
31681 135/300 Write block 676 len 12288 data: 4 5 6
31682 136/300 Write block 471 len 8192 data: 6 8
31683 137/300 Read block 929 len 40960
31684 138/300 Write block 297 len 20480 data: 3 4 5 6 7
31685 139/300 Write block 958 len 24576 data: 3 5 7 8 6 6
31686 140/300 Write block 2723 len 8192 data: 3 3
31687 141/300 Read block 91 len 24576
31688 142/300 Read block 294 len 28672
31689 143/300 Read block 631 len 32768
31690 144/300 Write block 3093 len 36864 data: 6 6 6 5 6 6 3 5 3
31691 145/300 Read block 727 len 36864
31692 146/300 Read block 2432 len 12288
31693 147/300 Write block 3435 len 16384 data: 7 7 7 7
31694 148/300 Read block 4661 len 36864
31695 149/300 Read block 2261 len 4096
31696 150/300 Read block 911 len 28672
31697 151/300 Write block 4105 len 4096 data: 6
31698 152/300 Write block 4314 len 40960 data: 5 6 6 6 7 5 6 5 6 7
31699 153/300 Read block 4212 len 28672
31700 154/300 Read block 3645 len 4096
31701 155/300 Read block 870 len 16384
31702 156/300 Write block 3989 len 28672 data: 4 4 5 5 6 6 8
31703 157/300 Read block 1609 len 36864
31704 158/300 Read block 983 len 4096
31705 159/300 Write block 4810 len 40960 data: 4 4 5 5 6 6 7 9 9 7
31706 160/300 Write block 1573 len 32768 data: 5 5 5 4 5 6 6 7
31707 161/300 Write block 4213 len 20480 data: 7 6 7 6 8
31708 162/300 Write block 1374 len 24576 data: 2 2 2 3 3 3
31709 163/300 Read block 288 len 4096
31710 164/300 Read block 635 len 32768
31711 165/300 Write block 4154 len 28672 data: 5 4 4 5 7 8 8
31712 166/300 Write block 54 len 16384 data: 4 5 4 5
31713 167/300 Read block 3896 len 20480
31714 168/300 Read block 4793 len 28672
31715 169/300 Read block 4000 len 20480
31716 170/300 Read block 754 len 36864
31717 171/300 Read block 4814 len 36864
31718 172/300 Read block 3963 len 12288
31719 173/300 Write block 4289 len 4096 data: 5
31720 174/300 Read block 3512 len 16384
31721 175/300 Write block 4102 len 8192 data: 5 5
31722 176/300 Write block 273 len 40960 data: 4 5 6 5 6 8 8 9 7 6
31723 177/300 Read block 3110 len 36864
31724 178/300 Read block 328 len 16384
31725 179/300 Read block 1722 len 24576
31726 180/300 Read block 4715 len 24576
31727 181/300 Read block 1944 len 28672
31728 182/300 Write block 1961 len 28672 data: 4 4 4 3 2 2 4
31729 183/300 Flush
31730 184/300 Write block 2354 len 4096 data: 5
31731 185/300 Flush
31732 186/300 Read block 1428 len 20480
31733 187/300 Flush
31734 188/300 Write block 2281 len 28672 data: 5 7 7 6 6 6 6
31735 189/300 Read block 1830 len 36864
31736 190/300 Write block 2078 len 12288 data: 6 5 5
31737 191/300 Read block 4625 len 40960
31738 192/300 Read block 1994 len 4096
31739 193/300 Read block 2139 len 32768
31740 194/300 Flush
31741 195/300 Read block 4791 len 32768
31742 196/300 Write block 3459 len 12288 data: 4 5 3
31743 197/300 Write block 4636 len 40960 data: 7 8 9 8 7 8 8 8 6 6
31744 198/300 Flush
31745 199/300 Read block 3230 len 36864
31746 200/300 Write block 838 len 36864 data: 4 6 6 6 7 6 5 5 5
31747 201/300 Read block 1672 len 16384
31748 202/300 Write block 1656 len 20480 data: 7 8 10 8 7
31749 203/300 Read block 4868 len 12288
31750 204/300 Write block 235 len 32768 data: 5 6 6 6 7 7 5 4
31751 205/300 Read block 797 len 12288
31752 206/300 Read block 1692 len 4096
31753 207/300 Read block 3838 len 16384
31754 208/300 Read block 4402 len 32768
31755 209/300 Write block 1422 len 20480 data: 5 4 4 5 5
31756 210/300 Write block 2945 len 12288 data: 3 3 4
31757 211/300 Read block 4077 len 4096
31758 212/300 Write block 4843 len 20480 data: 6 8 6 5 5
31759 213/300 Read block 4456 len 20480
31760 214/300 Read block 738 len 32768
31761 215/300 Write block 121 len 32768 data: 7 6 5 6 5 5 6 3
31762 216/300 Read block 4372 len 8192
31763 217/300 Write block 2788 len 8192 data: 4 4
31764 218/300 Write block 3344 len 20480 data: 3 2 4 5 5
31765 219/300 Write block 3138 len 20480 data: 5 6 7 9 8
31766 220/300 Write block 846 len 4096 data: 6
31767 221/300 Read block 604 len 4096
31768 222/300 Read block 4801 len 40960
31769 223/300 Write block 1094 len 36864 data: 3 3 4 3 3 3 3 6 6
31770 224/300 Flush
31771 225/300 Read block 2541 len 32768
31772 226/300 Read block 3978 len 12288
31773 227/300 Read block 2440 len 20480
31774 228/300 Write block 2036 len 16384 data: 5 5 6 5
31775 229/300 Write block 2640 len 32768 data: 5 6 6 4 4 6 6 7
31776 230/300 Write block 1332 len 12288 data: 8 8 8
31777 231/300 Read block 3105 len 8192
31778 232/300 Read block 3166 len 24576
31779 233/300 Write block 4157 len 4096 data: 6
31780 234/300 Write block 3149 len 40960 data: 6 5 7 7 7 7 9 10 8 6
31781 235/300 Write block 4129 len 24576 data: 7 8 9 7 7 6
31782 236/300 Read block 2397 len 40960
31783 237/300 Read block 3676 len 24576
31784 238/300 Write block 1693 len 36864 data: 7 9 10 10 8 6 6 6 7
31785 239/300 Read block 3800 len 16384
31786 240/300 Write block 1719 len 28672 data: 7 8 5 6 6 5 3
31787 241/300 Write block 3710 len 28672 data: 8 5 5 5 5 6 7
31788 242/300 Write block 3481 len 40960 data: 7 7 10 8 6 7 7 7 8 6
31789 243/300 Read block 2727 len 20480
31790 244/300 Read block 2280 len 16384
31791 245/300 Read block 4142 len 36864
31792 246/300 Read block 2664 len 20480
31793 247/300 Read block 4484 len 12288
31794 248/300 Read block 693 len 40960
31795 249/300 Write block 3814 len 24576 data: 6 8 11 10 9 8
31796 250/300 Flush
31797 251/300 Write block 4870 len 32768 data: 4 3 4 4 4 4 2 2
31798 252/300 Read block 3558 len 16384
31799 253/300 Flush
31800 254/300 Write block 4290 len 40960 data: 6 7 6 5 4 6 6 4 4 4
31801 255/300 Flush
31802 256/300 Write block 1877 len 4096 data: 3
31803 257/300 Write block 4849 len 32768 data: 8 6 6 5 5 4 5 5
31804 258/300 Write block 313 len 32768 data: 8 8 8 6 6 6 7 5
31805 259/300 Write block 3448 len 28672 data: 5 5 4 4 4 5 4
31806 260/300 Flush
31807 261/300 Read block 4877 len 8192
31808 262/300 Read block 3351 len 28672
31809 263/300 Read block 2769 len 24576
31810 264/300 Write block 3563 len 36864 data: 3 5 5 4 4 5 6 8 9
31811 265/300 Read block 2496 len 16384
31812 266/300 Flush
31813 267/300 Read block 3472 len 36864
31814 268/300 Write block 3181 len 8192 data: 2 3
31815 269/300 Read block 3650 len 32768
31816 270/300 Read block 3500 len 20480
31817 271/300 Read block 3553 len 28672
31818 272/300 Write block 1624 len 40960 data: 5 3 3 3 3 4 5 5 6 6
31819 273/300 Read block 1957 len 8192
31820 274/300 Write block 3507 len 20480 data: 3 3 3 4 4
31821 275/300 Write block 135 len 12288 data: 2 3 3
31822 276/300 Write block 576 len 4096 data: 6
31823 277/300 Write block 4948 len 8192 data: 7 7
31824 278/300 Flush
31825 279/300 Read block 3180 len 4096
31826 280/300 Read block 3792 len 8192
31827 281/300 Write block 912 len 20480 data: 4 4 3 3 4
31828 282/300 Write block 359 len 24576 data: 2 2 5 5 5 5
31829 283/300 Read block 4278 len 4096
31830 284/300 Read block 3615 len 16384
31831 285/300 Read block 394 len 32768
31832 286/300 Read block 1888 len 8192
31833 287/300 Write block 432 len 16384 data: 8 8 9 7
31834 288/300 Read block 2066 len 12288
31835 289/300 Write block 1826 len 40960 data: 7 6 9 9 8 8 9 10 12 11
31836 290/300 Write block 4015 len 32768 data: 4 4 6 7 7 4 4 5
31837 291/300 Read block 2638 len 16384
31838 292/300 Write block 2819 len 8192 data: 4 5
31839 293/300 Read block 198 len 28672
31840 294/300 Read block 4666 len 20480
31841 295/300 Read block 4364 len 20480
31842 296/300 Write block 762 len 40960 data: 7 5 4 5 6 5 6 6 9 8
31843 297/300 Read block 2985 len 24576
31844 298/300 Write block 2109 len 36864 data: 3 3 4 4 4 6 6 6 6
31845 299/300 Read block 3788 len 4096
31846 300/300 Write block 2011 len 8192 data: 4 5
31847 301/300 Write block 88 len 28672 data: 6 5 4 3 5 6 5
31848 Main task has work to do, go find it
31849 got dsc Start(1)
31850 start 1
31851 Feb 07 20:14:24.320 INFO request completed, latency_us: 225, response_code: 204, uri: /start/cid/1, method: POST, req_id: 4d17a1c8-3d52-41ab-bb3d-e49f1812ad52, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
31852 Replay: started 1, returned:Ok(())
31853 ----------------------------------------------------------------
31854 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31855 States: Active Offline Active
31856 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31857 8851 Acked Read 6 Done Sent Done false
31858 8852 Acked Flush 0 Done Sent Done false
31859 8853 Acked Read 10 Done Sent Done false
31860 8854 Acked Read 8 Done Sent Done false
31861 8855 Acked Read 10 Done Sent Done false
31862 8856 Acked Write 5 Done Sent Done false
31863 8857 Acked Write 2 Done Sent Done false
31864 8858 Acked Read 10 Done Sent Done false
31865 8859 Acked Write 2 Done Sent Done false
31866 8860 Acked Flush 0 Done Sent Done false
31867 8861 Acked Write 3 Done Sent Done false
31868 8862 Acked Write 9 Done Sent Done false
31869 8863 Acked Flush 0 Done Sent Done false
31870 8864 Acked Read 5 Done Sent Done false
31871 8865 Acked Write 9 Done Sent Done false
31872 8866 Acked Read 6 Done Sent Done false
31873 8867 Acked Write 5 Done Sent Done false
31874 8868 Acked Read 2 Done Sent Done false
31875 8869 Acked Read 5 Done Sent Done false
31876 8870 Acked Read 6 Done Sent Done false
31877 8871 Acked Write 2 Done Sent Done false
31878 8872 Acked Write 9 Done Sent Done false
31879 8873 Acked Read 6 Done Sent Done false
31880 8874 Acked Read 8 Done Sent Done false
31881 8875 Acked Write 8 Done Sent Done false
31882 8876 Acked Flush 0 Done Sent Done false
31883 8877 Acked Read 1 Done Sent Done false
31884 8878 Acked Read 6 Done Sent Done false
31885 8879 Acked Read 2 Done Sent Done false
31886 8880 Acked Write 3 Done Sent Done false
31887 8881 Acked Write 10 Done Sent Done false
31888 8882 Acked Read 5 Done Sent Done false
31889 8883 Acked Write 8 Done Sent Done false
31890 8884 Acked Read 4 Done Sent Done false
31891 8885 Acked Write 5 Done Sent Done false
31892 8886 Acked Write 3 Done Sent Done false
31893 8887 Acked Flush 0 Done Sent Done false
31894 8888 Acked Flush 0 Done Sent Done false
31895 8889 Acked Read 2 Done Sent Done false
31896 8890 Acked Write 5 Done Sent Done false
31897 8891 Acked Read 8 Done Sent Done false
31898 8892 Acked Write 8 Done Sent Done false
31899 8893 Acked Write 6 Done Sent Done false
31900 8894 Acked Write 9 Done Sent Done false
31901 8895 Acked Write 4 Done Sent Done false
31902 8896 Acked Write 6 Done Sent Done false
31903 8897 Acked Write 8 Done Sent Done false
31904 8898 Acked Read 2 Done Sent Done false
31905 8899 Acked Write 7 Done Sent Done false
31906 8900 Acked Write 3 Done Sent Done false
31907 8901 Acked Flush 0 Done Sent Done false
31908 8902 Acked Write 10 Done Sent Done false
31909 8903 Acked Write 7 Done Sent Done false
31910 8904 Acked Read 10 Done Sent Done false
31911 8905 Acked Flush 0 Done Sent Done false
31912 8906 Acked Write 4 Done Sent Done false
31913 8907 Acked Flush 0 Done Sent Done false
31914 8908 Acked Read 2 Done Sent Done false
31915 8909 Acked Read 3 Done Sent Done false
31916 8910 Acked Read 8 Done Sent Done false
31917 8911 Acked Read 6 Done Sent Done false
31918 8912 Acked Read 9 Done Sent Done false
31919 8913 Acked Write 2 Done Sent Done false
31920 8914 Acked Read 5 Done Sent Done false
31921 8915 Acked Read 3 Done Sent Done false
31922 8916 Acked Read 1 Done Sent Done false
31923 8917 Acked Read 6 Done Sent Done false
31924 8918 Acked Read 8 Done Sent Done false
31925 8919 Acked Read 6 Done Sent Done false
31926 8920 Acked Read 6 Done Sent Done false
31927 8921 Acked Read 3 Done Sent Done false
31928 8922 Acked Write 1 Done Sent Done false
31929 8923 Acked Read 1 Done Sent Done false
31930 8924 Acked Write 8 Done Sent Done false
31931 8925 Acked Read 1 Done Sent Done false
31932 8926 Acked Read 5 Done Sent Done false
31933 8927 Acked Read 9 Done Sent Done false
31934 8928 Acked Write 9 Done Sent Done false
31935 8929 Acked Read 8 Done Sent Done false
31936 8930 Acked Read 7 Done Sent Done false
31937 8931 Acked Write 5 Done Sent Done false
31938 8932 Acked Write 8 Done Sent Done false
31939 8933 Acked Read 10 Done Sent Done false
31940 8934 Acked Write 4 Done Sent Done false
31941 8935 Acked Write 6 Done Sent Done false
31942 8936 Acked Flush 0 Done Sent Done false
31943 8937 Acked Read 5 Done Sent Done false
31944 8938 Acked Read 8 Done Sent Done false
31945 8939 Acked Read 8 Done Sent Done false
31946 8940 Acked Write 7 Done Sent Done false
31947 8941 Acked Write 7 Done Sent Done false
31948 8942 Acked Read 4 Done Sent Done false
31949 8943 Acked Read 6 Done Sent Done false
31950 8944 Acked Write 3 Done Sent Done false
31951 8945 Acked Flush 0 Done Sent Done false
31952 8946 Acked Write 10 Done Sent Done false
31953 8947 Acked Read 7 Done Sent Done false
31954 8948 Acked Write 1 Done Sent Done false
31955 8949 Acked Read 7 Done Sent Done false
31956 8950 Acked Read 5 Done Sent Done false
31957 8951 Acked Read 6 Done Sent Done false
31958 8952 Acked Read 10 Done Sent Done false
31959 8953 Acked Write 4 Done Sent Done false
31960 8954 Acked Write 8 Done Sent Done false
31961 8955 Acked Read 2 Done Sent Done false
31962 8956 Acked Read 4 Done Sent Done false
31963 8957 Acked Write 10 Done Sent Done false
31964 8958 Acked Read 4 Done Sent Done false
31965 8959 Acked Read 1 Done Sent Done false
31966 8960 Acked Read 1 Done Sent Done false
31967 8961 Acked Read 1 Done Sent Done false
31968 8962 Acked Write 8 Done Sent Done false
31969 8963 Acked Read 9 Done Sent Done false
31970 8964 Acked Read 8 Done Sent Done false
31971 8965 Acked Read 2 Done Sent Done false
31972 8966 Acked Write 5 Done Sent Done false
31973 8967 Acked Read 10 Done Sent Done false
31974 8968 Acked Read 10 Done Sent Done false
31975 8969 Acked Write 8 Done Sent Done false
31976 8970 Acked Read 10 Done Sent Done false
31977 8971 Acked Read 8 Done Sent Done false
31978 8972 Acked Write 4 Done Sent Done false
31979 8973 Acked Read 3 Done Sent Done false
31980 8974 Acked Read 6 Done Sent Done false
31981 8975 Acked Read 2 Done Sent Done false
31982 8976 Acked Flush 0 Done Sent Done false
31983 8977 Acked Write 7 Done Sent Done false
31984 8978 Acked Read 9 Done Sent Done false
31985 8979 Acked Read 5 Done Sent Done false
31986 8980 Acked Read 8 Done Sent Done false
31987 8981 Acked Write 8 Done Sent Done false
31988 8982 Acked Read 1 Done Sent Done false
31989 8983 Acked Read 7 Done Sent Done false
31990 8984 Acked Flush 0 Done Sent Done false
31991 8985 Acked Write 3 Done Sent Done false
31992 8986 Acked Write 2 Done Sent Done false
31993 8987 Acked Read 10 Done Sent Done false
31994 8988 Acked Write 5 Done Sent Done false
31995 8989 Acked Write 6 Done Sent Done false
31996 8990 Acked Write 2 Done Sent Done false
31997 8991 Acked Read 6 Done Sent Done false
31998 8992 Acked Read 7 Done Sent Done false
31999 8993 Acked Read 8 Done Sent Done false
32000 8994 Acked Write 9 Done Sent Done false
32001 8995 Acked Read 9 Done Sent Done false
32002 8996 Acked Read 3 Done Sent Done false
32003 8997 Acked Write 4 Done Sent Done false
32004 8998 Acked Read 9 Done Sent Done false
32005 8999 Acked Read 1 Done Sent Done false
32006 9000 Acked Read 7 Done Sent Done false
32007 9001 Acked Write 1 Done Sent Done false
32008 9002 Acked Write 10 Done Sent Done false
32009 9003 Acked Read 7 Done Sent Done false
32010 9004 Acked Read 1 Done Sent Done false
32011 9005 Acked Read 4 Done Sent Done false
32012 9006 Acked Write 7 Done Sent Done false
32013 9007 Acked Read 9 Done Sent Done false
32014 9008 Acked Read 1 Done Sent Done false
32015 9009 Acked Write 10 Done Sent Done false
32016 9010 Acked Write 8 Done Sent Done false
32017 9011 Acked Write 5 Done Sent Done false
32018 9012 Acked Write 6 Done Sent Done false
32019 9013 Acked Read 1 Done Sent Done false
32020 9014 Acked Read 8 Done Sent Done false
32021 9015 Acked Write 7 Done Sent Done false
32022 9016 Acked Write 4 Done Sent Done false
32023 9017 Acked Read 5 Done Sent Done false
32024 9018 Acked Read 7 Done Sent Done false
32025 9019 Acked Read 5 Done Sent Done false
32026 9020 Acked Read 9 Done Sent Done false
32027 9021 Acked Read 9 Done Sent Done false
32028 9022 Acked Read 3 Done Sent Done false
32029 9023 Acked Write 1 Done Sent Done false
32030 9024 Acked Read 4 Done Sent Done false
32031 9025 Acked Write 2 Done Sent Done false
32032 9026 Acked Write 10 Done Sent Done false
32033 9027 Acked Read 9 Done Sent Done false
32034 9028 Acked Read 4 Done Sent Done false
32035 9029 Acked Read 6 Done Sent Done false
32036 9030 Acked Read 6 Done Sent Done false
32037 9031 Acked Read 7 Done Sent Done false
32038 9032 Acked Write 7 Done Sent Done false
32039 9033 Acked Flush 0 Done Sent Done false
32040 9034 Acked Write 1 Done Sent Done false
32041 9035 Acked Flush 0 Done Sent Done false
32042 9036 Acked Read 5 Done Sent Done false
32043 9037 Acked Flush 0 Done Sent Done false
32044 9038 Acked Write 7 Done Sent Done false
32045 9039 Acked Read 9 Done Sent Done false
32046 9040 Acked Write 3 Done Sent Done false
32047 9041 Acked Read 10 Done Sent Done false
32048 9042 Acked Read 1 Done Sent Done false
32049 9043 Acked Read 8 Done Sent Done false
32050 9044 Acked Flush 0 Done Sent Done false
32051 9045 Acked Read 8 Done Sent Done false
32052 9046 Acked Write 3 Done Sent Done false
32053 9047 Acked Write 10 Done Sent Done false
32054 9048 Acked Flush 0 Done Sent Done false
32055 9049 Acked Read 9 Done Sent Done false
32056 9050 Acked Write 9 Done Sent Done false
32057 9051 Acked Read 4 Done Sent Done false
32058 9052 Acked Write 5 Done Sent Done false
32059 9053 Acked Read 3 Done Sent Done false
32060 9054 Acked Write 8 Done Sent Done false
32061 9055 Acked Read 3 Done Sent Done false
32062 9056 Acked Read 1 Done Sent Done false
32063 9057 Acked Read 4 Done Sent Done false
32064 9058 Acked Read 8 Done Sent Done false
32065 9059 Acked Write 5 Done Sent Done false
32066 9060 Acked Write 3 Done Sent Done false
32067 9061 Acked Read 1 Done Sent Done false
32068 9062 Acked Write 5 Done Sent Done false
32069 9063 Acked Read 5 Done Sent Done false
32070 9064 Acked Read 8 Done Sent Done false
32071 9065 Acked Write 8 Done Sent Done false
32072 9066 Acked Read 2 Done Sent Done false
32073 9067 Acked Write 2 Done Sent Done false
32074 9068 Acked Write 5 Done Sent Done false
32075 9069 Acked Write 5 Done Sent Done false
32076 9070 Acked Write 1 Done Sent Done false
32077 9071 Acked Read 1 Done Sent Done false
32078 9072 Acked Read 10 Done Sent Done false
32079 9073 Acked Write 9 Done Sent Done false
32080 9074 Acked Flush 0 Done Sent Done false
32081 9075 Acked Read 8 Done Sent Done false
32082 9076 Acked Read 3 Done Sent Done false
32083 9077 Acked Read 5 Done Sent Done false
32084 9078 Acked Write 4 Done Sent Done false
32085 9079 Acked Write 8 Done Sent Done false
32086 9080 Acked Write 3 Done Sent Done false
32087 9081 Acked Read 2 Done Sent Done false
32088 9082 Acked Read 6 Done Sent Done false
32089 9083 Acked Write 1 Done Sent Done false
32090 9084 Acked Write 10 Done Sent Done false
32091 9085 Acked Write 6 Done Sent Done false
32092 9086 Acked Read 10 Done Sent Done false
32093 9087 Acked Read 6 Done Sent Done false
32094 9088 Acked Write 9 Done Sent Done false
32095 9089 Acked Read 4 Done Sent Done false
32096 9090 Acked Write 7 Done Sent Done false
32097 9091 Acked Write 7 Done Sent Done false
32098 9092 Acked Write 10 Done Sent Done false
32099 9093 Acked Read 5 Done Sent Done false
32100 9094 Acked Read 4 Done Sent Done false
32101 9095 Acked Read 9 Done Sent Done false
32102 9096 Acked Read 5 Done Sent Done false
32103 9097 Acked Read 3 Done Sent Done false
32104 9098 Acked Read 10 Done Sent Done false
32105 9099 Acked Write 6 Done Sent Done false
32106 9100 Acked Flush 0 Done Sent Done false
32107 9101 Acked Write 8 Done Sent Done false
32108 9102 Acked Read 4 Done Sent Done false
32109 9103 Acked Flush 0 Done Sent Done false
32110 9104 Acked Write 10 Done Sent Done false
32111 9105 Acked Flush 0 Done Sent Done false
32112 9106 Acked Write 1 Done Sent Done false
32113 9107 Acked Write 8 Done Sent Done false
32114 9108 Acked Write 8 Done Sent Done false
32115 9109 Acked Write 7 Done Sent Done false
32116 9110 Acked Flush 0 Done Sent Done false
32117 9111 Acked Read 2 Done Sent Done false
32118 9112 Acked Read 7 Done Sent Done false
32119 9113 Acked Read 6 Done Sent Done false
32120 9114 Acked Write 9 Done Sent Done false
32121 9115 Acked Read 4 Done Sent Done false
32122 9116 Acked Flush 0 Done Sent Done false
32123 9117 Acked Read 9 Done Sent Done false
32124 9118 Acked Write 2 Done Sent Done false
32125 9119 Acked Read 8 Done Sent Done false
32126 9120 Acked Read 5 Done Sent Done false
32127 9121 Acked Read 7 Done Sent Done false
32128 9122 Acked Write 10 Done Sent Done false
32129 9123 Acked Read 2 Done Sent Done false
32130 9124 Acked Write 5 Done Sent Done false
32131 9125 Acked Write 3 Done Sent Done false
32132 9126 Acked Write 1 Done Sent Done false
32133 9127 Acked Write 2 Done Sent Done false
32134 9128 Acked Flush 0 Done Sent Done false
32135 9129 Acked Read 1 Done Sent Done false
32136 9130 Acked Read 2 Done Sent Done false
32137 9131 Acked Write 5 Done Sent Done false
32138 9132 Acked Write 6 Done Sent Done false
32139 9133 Acked Read 1 Done Sent Done false
32140 9134 Acked Read 4 Done Sent Done false
32141 9135 Acked Read 8 Done Sent Done false
32142 9136 Acked Read 2 Done Sent Done false
32143 9137 Acked Write 4 Done Sent Done false
32144 9138 Acked Read 3 Done Sent Done false
32145 9139 Acked Write 10 Done Sent Done false
32146 9140 Acked Write 8 Done Sent Done false
32147 9141 Acked Read 4 Done Sent Done false
32148 9142 Acked Write 2 Done Sent Done false
32149 9143 Acked Read 7 Done Sent Done false
32150 9144 Acked Read 5 Done Sent Done false
32151 9145 Acked Read 5 Done Sent Done false
32152 9146 Acked Write 10 Done Sent Done false
32153 9147 Acked Read 6 Done Sent Done false
32154 9148 Acked Write 9 Done Sent Done false
32155 9149 Acked Read 1 Done Sent Done false
32156 9150 Acked Write 2 Done Sent Done false
32157 9151 Acked Write 7 Done Sent Done false
32158 STATES DS:0 DS:1 DS:2 TOTAL
32159 Sent 0 301 0 301
32160 Done 301 0 301 602
32161 Skipped 0 0 0 0
32162 Error 0 0 0 0
32163 Last Flush: Some(JobId(9128)) Some(JobId(8850)) Some(JobId(9128))
32164 Downstairs last five completed: 8850 8849 8848 8847 8846
32165 Upstairs last five completed: 9151 9150 9149 9148 9147
32166 CLIENT: Up:0 ds:301 act:2
321672025-02-07T20:14:24.822ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
321682025-02-07T20:14:26.054ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
321692025-02-07T20:14:26.636ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
321702025-02-07T20:14:27.638ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
32171 [8820] got start action
32172 [8820] I am going to restart
32173 Starting downstairs at port 8820
32174 Make output file at "/tmp/dsc/downstairs-8820.txt"
32175 [8820][1] reports Starting
32176 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1236)
32177 [8820][1] reports Running
32178 ----------------------------------------------------------------
32179 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32180 States: Active Offline Active
32181 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32182 8851 Acked Read 6 Done Sent Done false
32183 8852 Acked Flush 0 Done Sent Done false
32184 8853 Acked Read 10 Done Sent Done false
32185 8854 Acked Read 8 Done Sent Done false
32186 8855 Acked Read 10 Done Sent Done false
32187 8856 Acked Write 5 Done Sent Done false
32188 8857 Acked Write 2 Done Sent Done false
32189 8858 Acked Read 10 Done Sent Done false
32190 8859 Acked Write 2 Done Sent Done false
32191 8860 Acked Flush 0 Done Sent Done false
32192 8861 Acked Write 3 Done Sent Done false
32193 8862 Acked Write 9 Done Sent Done false
32194 8863 Acked Flush 0 Done Sent Done false
32195 8864 Acked Read 5 Done Sent Done false
32196 8865 Acked Write 9 Done Sent Done false
32197 8866 Acked Read 6 Done Sent Done false
32198 8867 Acked Write 5 Done Sent Done false
32199 8868 Acked Read 2 Done Sent Done false
32200 8869 Acked Read 5 Done Sent Done false
32201 8870 Acked Read 6 Done Sent Done false
32202 8871 Acked Write 2 Done Sent Done false
32203 8872 Acked Write 9 Done Sent Done false
32204 8873 Acked Read 6 Done Sent Done false
32205 8874 Acked Read 8 Done Sent Done false
32206 8875 Acked Write 8 Done Sent Done false
32207 8876 Acked Flush 0 Done Sent Done false
32208 8877 Acked Read 1 Done Sent Done false
32209 8878 Acked Read 6 Done Sent Done false
32210 8879 Acked Read 2 Done Sent Done false
32211 8880 Acked Write 3 Done Sent Done false
32212 8881 Acked Write 10 Done Sent Done false
32213 8882 Acked Read 5 Done Sent Done false
32214 8883 Acked Write 8 Done Sent Done false
32215 8884 Acked Read 4 Done Sent Done false
32216 8885 Acked Write 5 Done Sent Done false
32217 8886 Acked Write 3 Done Sent Done false
32218 8887 Acked Flush 0 Done Sent Done false
32219 8888 Acked Flush 0 Done Sent Done false
32220 8889 Acked Read 2 Done Sent Done false
32221 8890 Acked Write 5 Done Sent Done false
32222 8891 Acked Read 8 Done Sent Done false
32223 8892 Acked Write 8 Done Sent Done false
32224 8893 Acked Write 6 Done Sent Done false
32225 8894 Acked Write 9 Done Sent Done false
32226 8895 Acked Write 4 Done Sent Done false
32227 8896 Acked Write 6 Done Sent Done false
32228 8897 Acked Write 8 Done Sent Done false
32229 8898 Acked Read 2 Done Sent Done false
32230 8899 Acked Write 7 Done Sent Done false
32231 8900 Acked Write 3 Done Sent Done false
32232 8901 Acked Flush 0 Done Sent Done false
32233 8902 Acked Write 10 Done Sent Done false
32234 8903 Acked Write 7 Done Sent Done false
32235 8904 Acked Read 10 Done Sent Done false
32236 8905 Acked Flush 0 Done Sent Done false
32237 8906 Acked Write 4 Done Sent Done false
32238 8907 Acked Flush 0 Done Sent Done false
32239 8908 Acked Read 2 Done Sent Done false
32240 8909 Acked Read 3 Done Sent Done false
32241 8910 Acked Read 8 Done Sent Done false
32242 8911 Acked Read 6 Done Sent Done false
32243 8912 Acked Read 9 Done Sent Done false
32244 8913 Acked Write 2 Done Sent Done false
32245 8914 Acked Read 5 Done Sent Done false
32246 8915 Acked Read 3 Done Sent Done false
32247 8916 Acked Read 1 Done Sent Done false
32248 8917 Acked Read 6 Done Sent Done false
32249 8918 Acked Read 8 Done Sent Done false
32250 8919 Acked Read 6 Done Sent Done false
32251 8920 Acked Read 6 Done Sent Done false
32252 8921 Acked Read 3 Done Sent Done false
32253 8922 Acked Write 1 Done Sent Done false
32254 8923 Acked Read 1 Done Sent Done false
32255 8924 Acked Write 8 Done Sent Done false
32256 8925 Acked Read 1 Done Sent Done false
32257 8926 Acked Read 5 Done Sent Done false
32258 8927 Acked Read 9 Done Sent Done false
32259 8928 Acked Write 9 Done Sent Done false
32260 8929 Acked Read 8 Done Sent Done false
32261 8930 Acked Read 7 Done Sent Done false
32262 8931 Acked Write 5 Done Sent Done false
32263 8932 Acked Write 8 Done Sent Done false
32264 8933 Acked Read 10 Done Sent Done false
32265 8934 Acked Write 4 Done Sent Done false
32266 8935 Acked Write 6 Done Sent Done false
32267 8936 Acked Flush 0 Done Sent Done false
32268 8937 Acked Read 5 Done Sent Done false
32269 8938 Acked Read 8 Done Sent Done false
32270 8939 Acked Read 8 Done Sent Done false
32271 8940 Acked Write 7 Done Sent Done false
32272 8941 Acked Write 7 Done Sent Done false
32273 8942 Acked Read 4 Done Sent Done false
32274 8943 Acked Read 6 Done Sent Done false
32275 8944 Acked Write 3 Done Sent Done false
32276 8945 Acked Flush 0 Done Sent Done false
32277 8946 Acked Write 10 Done Sent Done false
32278 8947 Acked Read 7 Done Sent Done false
32279 8948 Acked Write 1 Done Sent Done false
32280 8949 Acked Read 7 Done Sent Done false
32281 8950 Acked Read 5 Done Sent Done false
32282 8951 Acked Read 6 Done Sent Done false
32283 8952 Acked Read 10 Done Sent Done false
32284 8953 Acked Write 4 Done Sent Done false
32285 8954 Acked Write 8 Done Sent Done false
32286 8955 Acked Read 2 Done Sent Done false
32287 8956 Acked Read 4 Done Sent Done false
32288 8957 Acked Write 10 Done Sent Done false
32289 8958 Acked Read 4 Done Sent Done false
32290 8959 Acked Read 1 Done Sent Done false
32291 8960 Acked Read 1 Done Sent Done false
32292 8961 Acked Read 1 Done Sent Done false
32293 8962 Acked Write 8 Done Sent Done false
32294 8963 Acked Read 9 Done Sent Done false
32295 8964 Acked Read 8 Done Sent Done false
32296 8965 Acked Read 2 Done Sent Done false
32297 8966 Acked Write 5 Done Sent Done false
32298 8967 Acked Read 10 Done Sent Done false
32299 8968 Acked Read 10 Done Sent Done false
32300 8969 Acked Write 8 Done Sent Done false
32301 8970 Acked Read 10 Done Sent Done false
32302 8971 Acked Read 8 Done Sent Done false
32303 8972 Acked Write 4 Done Sent Done false
32304 8973 Acked Read 3 Done Sent Done false
32305 8974 Acked Read 6 Done Sent Done false
32306 8975 Acked Read 2 Done Sent Done false
32307 8976 Acked Flush 0 Done Sent Done false
32308 8977 Acked Write 7 Done Sent Done false
32309 8978 Acked Read 9 Done Sent Done false
32310 8979 Acked Read 5 Done Sent Done false
32311 8980 Acked Read 8 Done Sent Done false
32312 8981 Acked Write 8 Done Sent Done false
32313 8982 Acked Read 1 Done Sent Done false
32314 8983 Acked Read 7 Done Sent Done false
32315 8984 Acked Flush 0 Done Sent Done false
32316 8985 Acked Write 3 Done Sent Done false
32317 8986 Acked Write 2 Done Sent Done false
32318 8987 Acked Read 10 Done Sent Done false
32319 8988 Acked Write 5 Done Sent Done false
32320 8989 Acked Write 6 Done Sent Done false
32321 8990 Acked Write 2 Done Sent Done false
32322 8991 Acked Read 6 Done Sent Done false
32323 8992 Acked Read 7 Done Sent Done false
32324 8993 Acked Read 8 Done Sent Done false
32325 8994 Acked Write 9 Done Sent Done false
32326 8995 Acked Read 9 Done Sent Done false
32327 8996 Acked Read 3 Done Sent Done false
32328 8997 Acked Write 4 Done Sent Done false
32329 8998 Acked Read 9 Done Sent Done false
32330 8999 Acked Read 1 Done Sent Done false
32331 9000 Acked Read 7 Done Sent Done false
32332 9001 Acked Write 1 Done Sent Done false
32333 9002 Acked Write 10 Done Sent Done false
32334 9003 Acked Read 7 Done Sent Done false
32335 9004 Acked Read 1 Done Sent Done false
32336 9005 Acked Read 4 Done Sent Done false
32337 9006 Acked Write 7 Done Sent Done false
32338 9007 Acked Read 9 Done Sent Done false
32339 9008 Acked Read 1 Done Sent Done false
32340 9009 Acked Write 10 Done Sent Done false
32341 9010 Acked Write 8 Done Sent Done false
32342 9011 Acked Write 5 Done Sent Done false
32343 9012 Acked Write 6 Done Sent Done false
32344 9013 Acked Read 1 Done Sent Done false
32345 9014 Acked Read 8 Done Sent Done false
32346 9015 Acked Write 7 Done Sent Done false
32347 9016 Acked Write 4 Done Sent Done false
32348 9017 Acked Read 5 Done Sent Done false
32349 9018 Acked Read 7 Done Sent Done false
32350 9019 Acked Read 5 Done Sent Done false
32351 9020 Acked Read 9 Done Sent Done false
32352 9021 Acked Read 9 Done Sent Done false
32353 9022 Acked Read 3 Done Sent Done false
32354 9023 Acked Write 1 Done Sent Done false
32355 9024 Acked Read 4 Done Sent Done false
32356 9025 Acked Write 2 Done Sent Done false
32357 9026 Acked Write 10 Done Sent Done false
32358 9027 Acked Read 9 Done Sent Done false
32359 9028 Acked Read 4 Done Sent Done false
32360 9029 Acked Read 6 Done Sent Done false
32361 9030 Acked Read 6 Done Sent Done false
32362 9031 Acked Read 7 Done Sent Done false
32363 9032 Acked Write 7 Done Sent Done false
32364 9033 Acked Flush 0 Done Sent Done false
32365 9034 Acked Write 1 Done Sent Done false
32366 9035 Acked Flush 0 Done Sent Done false
32367 9036 Acked Read 5 Done Sent Done false
32368 9037 Acked Flush 0 Done Sent Done false
32369 9038 Acked Write 7 Done Sent Done false
32370 9039 Acked Read 9 Done Sent Done false
32371 9040 Acked Write 3 Done Sent Done false
32372 9041 Acked Read 10 Done Sent Done false
32373 9042 Acked Read 1 Done Sent Done false
32374 9043 Acked Read 8 Done Sent Done false
32375 9044 Acked Flush 0 Done Sent Done false
32376 9045 Acked Read 8 Done Sent Done false
32377 9046 Acked Write 3 Done Sent Done false
32378 9047 Acked Write 10 Done Sent Done false
32379 9048 Acked Flush 0 Done Sent Done false
32380 9049 Acked Read 9 Done Sent Done false
32381 9050 Acked Write 9 Done Sent Done false
32382 9051 Acked Read 4 Done Sent Done false
32383 9052 Acked Write 5 Done Sent Done false
32384 9053 Acked Read 3 Done Sent Done false
32385 9054 Acked Write 8 Done Sent Done false
32386 9055 Acked Read 3 Done Sent Done false
32387 9056 Acked Read 1 Done Sent Done false
32388 9057 Acked Read 4 Done Sent Done false
32389 9058 Acked Read 8 Done Sent Done false
32390 9059 Acked Write 5 Done Sent Done false
32391 9060 Acked Write 3 Done Sent Done false
32392 9061 Acked Read 1 Done Sent Done false
32393 9062 Acked Write 5 Done Sent Done false
32394 9063 Acked Read 5 Done Sent Done false
32395 9064 Acked Read 8 Done Sent Done false
32396 9065 Acked Write 8 Done Sent Done false
32397 9066 Acked Read 2 Done Sent Done false
32398 9067 Acked Write 2 Done Sent Done false
32399 9068 Acked Write 5 Done Sent Done false
32400 9069 Acked Write 5 Done Sent Done false
32401 9070 Acked Write 1 Done Sent Done false
32402 9071 Acked Read 1 Done Sent Done false
32403 9072 Acked Read 10 Done Sent Done false
32404 9073 Acked Write 9 Done Sent Done false
32405 9074 Acked Flush 0 Done Sent Done false
32406 9075 Acked Read 8 Done Sent Done false
32407 9076 Acked Read 3 Done Sent Done false
32408 9077 Acked Read 5 Done Sent Done false
32409 9078 Acked Write 4 Done Sent Done false
32410 9079 Acked Write 8 Done Sent Done false
32411 9080 Acked Write 3 Done Sent Done false
32412 9081 Acked Read 2 Done Sent Done false
32413 9082 Acked Read 6 Done Sent Done false
32414 9083 Acked Write 1 Done Sent Done false
32415 9084 Acked Write 10 Done Sent Done false
32416 9085 Acked Write 6 Done Sent Done false
32417 9086 Acked Read 10 Done Sent Done false
32418 9087 Acked Read 6 Done Sent Done false
32419 9088 Acked Write 9 Done Sent Done false
32420 9089 Acked Read 4 Done Sent Done false
32421 9090 Acked Write 7 Done Sent Done false
32422 9091 Acked Write 7 Done Sent Done false
32423 9092 Acked Write 10 Done Sent Done false
32424 9093 Acked Read 5 Done Sent Done false
32425 9094 Acked Read 4 Done Sent Done false
32426 9095 Acked Read 9 Done Sent Done false
32427 9096 Acked Read 5 Done Sent Done false
32428 9097 Acked Read 3 Done Sent Done false
32429 9098 Acked Read 10 Done Sent Done false
32430 9099 Acked Write 6 Done Sent Done false
32431 9100 Acked Flush 0 Done Sent Done false
32432 9101 Acked Write 8 Done Sent Done false
32433 9102 Acked Read 4 Done Sent Done false
32434 9103 Acked Flush 0 Done Sent Done false
32435 9104 Acked Write 10 Done Sent Done false
32436 9105 Acked Flush 0 Done Sent Done false
32437 9106 Acked Write 1 Done Sent Done false
32438 9107 Acked Write 8 Done Sent Done false
32439 9108 Acked Write 8 Done Sent Done false
32440 9109 Acked Write 7 Done Sent Done false
32441 9110 Acked Flush 0 Done Sent Done false
32442 9111 Acked Read 2 Done Sent Done false
32443 9112 Acked Read 7 Done Sent Done false
32444 9113 Acked Read 6 Done Sent Done false
32445 9114 Acked Write 9 Done Sent Done false
32446 9115 Acked Read 4 Done Sent Done false
32447 9116 Acked Flush 0 Done Sent Done false
32448 9117 Acked Read 9 Done Sent Done false
32449 9118 Acked Write 2 Done Sent Done false
32450 9119 Acked Read 8 Done Sent Done false
32451 9120 Acked Read 5 Done Sent Done false
32452 9121 Acked Read 7 Done Sent Done false
32453 9122 Acked Write 10 Done Sent Done false
32454 9123 Acked Read 2 Done Sent Done false
32455 9124 Acked Write 5 Done Sent Done false
32456 9125 Acked Write 3 Done Sent Done false
32457 9126 Acked Write 1 Done Sent Done false
32458 9127 Acked Write 2 Done Sent Done false
32459 9128 Acked Flush 0 Done Sent Done false
32460 9129 Acked Read 1 Done Sent Done false
32461 9130 Acked Read 2 Done Sent Done false
32462 9131 Acked Write 5 Done Sent Done false
32463 9132 Acked Write 6 Done Sent Done false
32464 9133 Acked Read 1 Done Sent Done false
32465 9134 Acked Read 4 Done Sent Done false
32466 9135 Acked Read 8 Done Sent Done false
32467 9136 Acked Read 2 Done Sent Done false
32468 9137 Acked Write 4 Done Sent Done false
32469 9138 Acked Read 3 Done Sent Done false
32470 9139 Acked Write 10 Done Sent Done false
32471 9140 Acked Write 8 Done Sent Done false
32472 9141 Acked Read 4 Done Sent Done false
32473 9142 Acked Write 2 Done Sent Done false
32474 9143 Acked Read 7 Done Sent Done false
32475 9144 Acked Read 5 Done Sent Done false
32476 9145 Acked Read 5 Done Sent Done false
32477 9146 Acked Write 10 Done Sent Done false
32478 9147 Acked Read 6 Done Sent Done false
32479 9148 Acked Write 9 Done Sent Done false
32480 9149 Acked Read 1 Done Sent Done false
32481 9150 Acked Write 2 Done Sent Done false
32482 9151 Acked Write 7 Done Sent Done false
32483 STATES DS:0 DS:1 DS:2 TOTAL
32484 Sent 0 301 0 301
32485 Done 301 0 301 602
32486 Skipped 0 0 0 0
32487 Error 0 0 0 0
32488 Last Flush: Some(JobId(9128)) Some(JobId(8850)) Some(JobId(9128))
32489 Downstairs last five completed: 8850 8849 8848 8847 8846
32490 Upstairs last five completed: 9151 9150 9149 9148 9147
32491 CLIENT: Up:0 ds:301 act:2
324922025-02-07T20:14:28.824ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
324932025-02-07T20:14:29.641ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
324942025-02-07T20:14:30.069ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
324952025-02-07T20:14:30.069ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48215) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
324962025-02-07T20:14:30.070ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
324972025-02-07T20:14:30.071ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
324982025-02-07T20:14:30.071ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
324992025-02-07T20:14:30.071ZINFOcrucible: send last flush ID to this DS: Some(JobId(8850)) = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
325002025-02-07T20:14:30.071ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
325012025-02-07T20:14:30.071ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(8850)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
325022025-02-07T20:14:30.071ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(8850)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
32503 ----------------------------------------------------------------
32504 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
32505 Downstairs last five completed: 9152 9151 9150 9149 9148
32506 Upstairs last five completed: 9152 9151 9150 9149 9148
32507 CLIENT: Up:0 ds:0 act:3
32508 Replay: All jobs finished
32509 Main task has work to do, go find it
32510 got dsc Stop(0)
32511 stop 0
32512 [8810] Got stop action so:false kr:false
32513 Feb 07 20:14:32.326 INFO request completed, latency_us: 398, response_code: 204, uri: /stop/cid/0, method: POST, req_id: e672d75a-0a9a-4842-a1d5-b27230d8885e, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
325142025-02-07T20:14:32.327ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
325152025-02-07T20:14:32.327ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
325162025-02-07T20:14:32.327ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
32517 Feb 07 20:14:32.327 {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-07T20:14:32.327336606Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
32518 INFO request completed, latency_us: 166, response_code: 200, uri: /state/cid/0, method: GET, req_id: e7293c86-6c98-442c-a76c-d4c0ab77553a, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
32519 [8810] Exited with: ExitStatus(unix_wait_status(9))
32520 [8810][0] reports Exit
32521 Feb 07 20:14:36.329 INFO request completed, latency_us: 319, response_code: 200, uri: /state/cid/0, method: GET, req_id: c8cbb747-877d-4fc3-bc03-1eb99223d7a9, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
32522 001/300 Read block 340 len 28672
32523 002/300 Read block 2926 len 16384
32524 003/300 Read block 4860 len 20480
32525 004/300 Read block 2754 len 4096
32526 005/300 Read block 4310 len 36864
32527 006/300 Read block 415 len 32768
32528 007/300 Write block 3073 len 20480 data: 7 6 7 6 6
32529 008/300 Write block 2597 len 4096 data: 10
32530 009/300 Write block 2940 len 40960 data: 5 3 3 3 3 4 4 5 5 5
32531 010/300 Write block 580 len 40960 data: 6 5 6 7 6 8 8 8 6 6
32532 011/300 Write block 4346 len 20480 data: 6 7 4 5 4
32533 012/300 Write block 4509 len 28672 data: 9 9 12 8 8 8 8
32534 013/300 Write block 4803 len 20480 data: 7 5 5 6 5
32535 014/300 Flush
32536 015/300 Write block 1212 len 4096 data: 3
32537 016/300 Read block 2274 len 20480
32538 017/300 Read block 4099 len 32768
32539 018/300 Read block 2674 len 32768
32540 019/300 Read block 2115 len 24576
32541 020/300 Flush
32542 021/300 Read block 1915 len 28672
32543 022/300 Read block 4532 len 20480
32544 023/300 Read block 1857 len 28672
32545 024/300 Read block 3078 len 28672
32546 025/300 Flush
32547 026/300 Read block 498 len 36864
32548 027/300 Write block 3607 len 24576 data: 7 7 9 7 6 5
32549 028/300 Flush
32550 029/300 Write block 712 len 36864 data: 5 6 6 6 6 7 8 7 6
32551 030/300 Write block 426 len 16384 data: 7 7 7 7
32552 031/300 Write block 1868 len 20480 data: 7 6 5 6 5
32553 032/300 Read block 4888 len 32768
32554 033/300 Read block 2415 len 24576
32555 034/300 Write block 4944 len 20480 data: 10 10 8 7 8
32556 035/300 Write block 4158 len 4096 data: 8
32557 036/300 Read block 2405 len 24576
32558 037/300 Write block 2509 len 20480 data: 5 6 6 4 4
32559 038/300 Read block 764 len 8192
32560 039/300 Read block 4876 len 4096
32561 040/300 Write block 1057 len 28672 data: 7 7 6 6 6 6 6
32562 041/300 Write block 2965 len 28672 data: 4 3 6 7 7 6 6
32563 042/300 Read block 615 len 4096
32564 043/300 Read block 4252 len 40960
32565 044/300 Write block 3568 len 36864 data: 6 7 9 10 8 8 7 6 8
32566 045/300 Read block 4665 len 20480
32567 046/300 Flush
32568 047/300 Flush
32569 048/300 Write block 710 len 36864 data: 4 4 6 7 7 7 7 8 9
32570 049/300 Write block 2004 len 4096 data: 11
32571 050/300 Read block 3191 len 8192
32572 051/300 Flush
32573 052/300 Write block 3838 len 32768 data: 7 6 6 6 6 5 7 7
32574 053/300 Write block 320 len 32768 data: 6 4 4 4 4 5 4 4
32575 054/300 Write block 2787 len 12288 data: 4 5 5
32576 055/300 Write block 717 len 12288 data: 9 10 8
32577 056/300 Read block 3683 len 16384
32578 057/300 Read block 2160 len 16384
32579 058/300 Read block 2614 len 32768
32580 059/300 Read block 2519 len 8192
32581 060/300 Flush
32582 061/300 Write block 435 len 28672 data: 8 7 6 5 5 5 3
32583 062/300 Read block 3974 len 16384
32584 063/300 Write block 4116 len 20480 data: 4 4 3 4 5
32585 064/300 Write block 2800 len 40960 data: 7 9 10 10 9 10 9 9 9 8
32586 065/300 Write block 1315 len 16384 data: 6 10 10 10
32587 066/300 Read block 3319 len 8192
32588 067/300 Flush
32589 068/300 Write block 1356 len 12288 data: 3 4 4
32590 069/300 Read block 1299 len 28672
32591 070/300 Read block 2842 len 12288
32592 071/300 Flush
32593 072/300 Write block 4920 len 36864 data: 7 5 4 6 7 7 7 6 2
32594 073/300 Read block 2041 len 32768
32595 074/300 Write block 4964 len 8192 data: 7 7
32596 075/300 Read block 233 len 20480
32597 076/300 Write block 1628 len 16384 data: 4 5 6 6
32598 077/300 Read block 3726 len 28672
32599 078/300 Read block 4727 len 24576
32600 079/300 Read block 1864 len 24576
32601 080/300 Read block 2008 len 32768
32602 081/300 Read block 20 len 28672
32603 082/300 Write block 1952 len 32768 data: 7 7 6 6 7 5 4 5
32604 083/300 Read block 4453 len 20480
32605 084/300 Read block 2196 len 4096
32606 085/300 Flush
32607 086/300 Read block 3723 len 12288
32608 087/300 Write block 1361 len 32768 data: 4 4 4 6 7 6 5 6
32609 088/300 Read block 422 len 20480
32610 089/300 Read block 3939 len 40960
32611 090/300 Flush
32612 091/300 Read block 3365 len 4096
32613 092/300 Write block 380 len 4096 data: 4
32614 093/300 Read block 720 len 40960
32615 094/300 Flush
32616 095/300 Read block 242 len 8192
32617 096/300 Flush
32618 097/300 Write block 4430 len 12288 data: 3 4 5
32619 098/300 Write block 4145 len 28672 data: 7 7 7 7 4 5 6
32620 099/300 Read block 47 len 4096
32621 100/300 Read block 1549 len 4096
32622 101/300 Write block 1733 len 24576 data: 4 3 3 3 4 4
32623 102/300 Write block 794 len 36864 data: 7 5 6 6 4 4 4 4 4
32624 103/300 Read block 2216 len 36864
32625 104/300 Read block 1390 len 28672
32626 105/300 Read block 2219 len 12288
32627 106/300 Read block 1743 len 24576
32628 107/300 Write block 1891 len 40960 data: 5 6 5 8 7 7 8 8 9 9
32629 108/300 Write block 3944 len 12288 data: 6 5 8
32630 109/300 Write block 1010 len 8192 data: 6 7
32631 110/300 Flush
32632 111/300 Write block 2006 len 36864 data: 9 9 10 8 6 5 6 6 6
32633 112/300 Read block 3051 len 20480
32634 113/300 Read block 4676 len 12288
32635 114/300 Read block 184 len 32768
32636 115/300 Flush
32637 116/300 Read block 1683 len 8192
32638 117/300 Write block 2673 len 28672 data: 5 5 5 5 4 4 4
32639 118/300 Write block 2239 len 40960 data: 5 5 5 5 5 5 5 4 6 5
32640 119/300 Read block 3026 len 28672
32641 120/300 Read block 4793 len 40960
32642 121/300 Read block 3171 len 32768
32643 122/300 Write block 2859 len 40960 data: 6 7 8 8 7 6 5 4 4 6
32644 123/300 Read block 2978 len 40960
32645 124/300 Write block 1102 len 4096 data: 7
32646 125/300 Write block 4124 len 24576 data: 7 6 5 6 5 8
32647 126/300 Read block 3889 len 4096
32648 127/300 Write block 3729 len 16384 data: 3 3 3 6
32649 128/300 Write block 449 len 32768 data: 8 8 7 7 5 5 6 4
32650 129/300 Write block 4237 len 24576 data: 6 8 7 5 5 4
32651 130/300 Read block 4406 len 12288
32652 131/300 Read block 2561 len 24576
32653 132/300 Write block 3118 len 16384 data: 5 6 5 5
32654 133/300 Read block 333 len 4096
32655 134/300 Read block 4797 len 16384
32656 135/300 Read block 2725 len 8192
32657 136/300 Read block 4936 len 24576
32658 137/300 Read block 171 len 12288
32659 138/300 Read block 4165 len 36864
32660 139/300 Write block 918 len 16384 data: 3 3 5 4
32661 140/300 Read block 1958 len 24576
32662 141/300 Read block 4030 len 32768
32663 142/300 Read block 2234 len 20480
32664 143/300 Write block 1767 len 24576 data: 5 5 4 4 4 4
32665 144/300 Write block 2035 len 12288 data: 6 6 6
32666 145/300 Flush
32667 146/300 Flush
32668 147/300 Read block 1200 len 4096
32669 148/300 Write block 2300 len 32768 data: 6 6 5 5 4 4 4 5
32670 149/300 Write block 2743 len 8192 data: 6 8
32671 150/300 Read block 353 len 16384
32672 151/300 Write block 4338 len 32768 data: 5 6 7 7 5 6 5 6
32673 152/300 Write block 4895 len 28672 data: 8 8 6 7 7 5 6
32674 153/300 Write block 2185 len 24576 data: 3 3 3 4 5 4
32675 154/300 Write block 2036 len 32768 data: 7 7 7 6 5 4 4 3
32676 155/300 Write block 2445 len 40960 data: 4 5 6 6 6 6 7 7 7 6
32677 156/300 Read block 4646 len 16384
32678 157/300 Write block 4708 len 32768 data: 8 8 7 6 5 4 3 5
32679 158/300 Read block 1167 len 24576
32680 159/300 Read block 14 len 16384
32681 160/300 Read block 4000 len 24576
32682 161/300 Write block 2344 len 40960 data: 9 9 8 8 7 5 6 5 4 4
32683 162/300 Read block 3231 len 20480
32684 163/300 Read block 4416 len 24576
32685 164/300 Read block 1950 len 4096
32686 165/300 Read block 638 len 36864
32687 166/300 Read block 3035 len 32768
32688 167/300 Read block 519 len 20480
32689 168/300 Write block 2745 len 12288 data: 8 8 8
32690 169/300 Write block 4902 len 28672 data: 9 8 8 8 8 7 7
32691 170/300 Read block 1807 len 32768
32692 171/300 Read block 321 len 16384
32693 172/300 Read block 512 len 28672
32694 173/300 Write block 461 len 40960 data: 6 7 7 7 5 7 7 8 7 7
32695 174/300 Read block 2721 len 12288
32696 175/300 Write block 4105 len 36864 data: 7 4 4 5 4 3 3 3 3
32697 176/300 Read block 2841 len 24576
32698 177/300 Write block 2069 len 24576 data: 4 4 4 5 5 5
32699 178/300 Write block 1715 len 40960 data: 5 6 6 8 8 9 6 7 7 6
32700 179/300 Read block 2223 len 36864
32701 180/300 Flush
32702 181/300 Read block 3442 len 12288
32703 182/300 Write block 4512 len 32768 data: 9 9 9 9 7 7 7 8
32704 183/300 Flush
32705 184/300 Read block 3252 len 20480
32706 185/300 Read block 2817 len 36864
32707 186/300 Write block 2383 len 28672 data: 9 9 5 5 5 5 5
32708 187/300 Write block 1611 len 24576 data: 5 7 6 4 6 8
32709 188/300 Write block 3972 len 8192 data: 4 4
32710 189/300 Read block 2407 len 16384
32711 190/300 Read block 4656 len 20480
32712 191/300 Read block 1050 len 24576
32713 192/300 Read block 2094 len 24576
32714 193/300 Write block 688 len 36864 data: 3 4 4 5 4 4 4 6 5
32715 194/300 Write block 2129 len 36864 data: 5 4 3 5 4 4 4 3 4
32716 195/300 Flush
32717 196/300 Write block 3846 len 12288 data: 5 4 4
32718 197/300 Read block 3363 len 40960
32719 198/300 Write block 4001 len 28672 data: 5 5 5 6 5 4 6
32720 199/300 Flush
32721 200/300 Read block 3802 len 36864
32722 201/300 Read block 2509 len 40960
32723 202/300 Read block 426 len 40960
32724 203/300 Flush
32725 204/300 Read block 147 len 28672
32726 205/300 Write block 3952 len 40960 data: 8 6 5 7 6 6 6 6 6 5
32727 206/300 Read block 1767 len 16384
32728 207/300 Write block 3805 len 28672 data: 8 7 7 5 6 6 7
32729 208/300 Write block 2871 len 32768 data: 4 2 2 2 2 2 2 2
32730 209/300 Read block 84 len 36864
32731 210/300 Read block 4355 len 16384
32732 211/300 Read block 4038 len 20480
32733 212/300 Read block 910 len 8192
32734 213/300 Write block 4297 len 4096 data: 5
32735 214/300 Write block 369 len 16384 data: 7 6 7 6
32736 215/300 Read block 4704 len 24576
32737 216/300 Write block 4273 len 32768 data: 6 6 5 7 6 6 6 5
32738 217/300 Flush
32739 218/300 Write block 2230 len 36864 data: 4 3 3 4 4 5 5 6 6
32740 219/300 Flush
32741 220/300 Write block 2353 len 24576 data: 5 6 6 7 7 9
32742 221/300 Read block 3942 len 16384
32743 222/300 Read block 1378 len 8192
32744 223/300 Read block 4742 len 28672
32745 224/300 Write block 4876 len 20480 data: 3 3 3 3 4
32746 225/300 Read block 2556 len 24576
32747 226/300 Read block 1990 len 32768
32748 227/300 Read block 4137 len 12288
32749 228/300 Read block 3518 len 36864
32750 229/300 Read block 913 len 28672
32751 230/300 Write block 4260 len 36864 data: 4 4 5 5 3 5 5 6 6
32752 231/300 Write block 1946 len 40960 data: 5 6 7 6 6 6 8 8 7 7
32753 232/300 Read block 3230 len 36864
32754 233/300 Write block 1644 len 40960 data: 4 6 6 6 7 7 7 9 10 9
32755 234/300 Write block 4283 len 28672 data: 4 4 4 5 3 3 6
32756 235/300 Write block 812 len 4096 data: 9
32757 236/300 Write block 3868 len 16384 data: 4 4 4 5
32758 237/300 Write block 3390 len 12288 data: 3 4 7
32759 238/300 Read block 2944 len 36864
32760 239/300 Read block 2874 len 40960
32761 240/300 Flush
32762 241/300 Write block 2225 len 12288 data: 5 6 5
32763 242/300 Write block 3084 len 16384 data: 6 5 5 5
32764 243/300 Read block 4748 len 36864
32765 244/300 Flush
32766 245/300 Read block 1134 len 28672
32767 246/300 Flush
32768 247/300 Read block 498 len 32768
32769 248/300 Write block 2524 len 20480 data: 6 6 5 6 6
32770 249/300 Read block 3015 len 28672
32771 250/300 Write block 451 len 16384 data: 8 8 6 6
32772 251/300 Read block 722 len 8192
32773 252/300 Write block 3609 len 16384 data: 10 8 7 6
32774 253/300 Read block 2713 len 24576
32775 254/300 Write block 604 len 20480 data: 4 3 2 4 4
32776 255/300 Read block 2240 len 12288
32777 256/300 Write block 4989 len 32768 data: 6 6 5 5 4 4 4 3
32778 257/300 Read block 3883 len 12288
32779 258/300 Read block 3755 len 40960
32780 259/300 Flush
32781 260/300 Read block 2493 len 12288
32782 261/300 Read block 1453 len 16384
32783 262/300 Write block 2232 len 36864 data: 4 5 5 6 6 7 7 6 6
32784 263/300 Read block 1262 len 12288
32785 264/300 Read block 4017 len 32768
32786 265/300 Write block 3374 len 24576 data: 5 4 4 4 5 4
32787 266/300 Read block 1196 len 12288
32788 267/300 Write block 4138 len 4096 data: 4
32789 268/300 Read block 4184 len 28672
32790 269/300 Read block 2872 len 24576
32791 270/300 Write block 3531 len 12288 data: 4 4 5
32792 271/300 Write block 3227 len 16384 data: 4 5 5 5
32793 272/300 Write block 4762 len 12288 data: 5 5 5
32794 273/300 Write block 4923 len 28672 data: 7 8 8 8 7 3 2
32795 274/300 Write block 4738 len 12288 data: 8 8 7
32796 275/300 Write block 4959 len 20480 data: 7 8 9 9 8
32797 276/300 Write block 2312 len 12288 data: 9 8 9
32798 277/300 Write block 3571 len 20480 data: 11 9 9 8 7
32799 278/300 Read block 3481 len 12288
32800 279/300 Write block 1719 len 36864 data: 9 10 7 8 8 7 4 4 3
32801 280/300 Read block 3252 len 40960
32802 281/300 Read block 3450 len 8192
32803 282/300 Read block 1756 len 16384
32804 283/300 Write block 3022 len 40960 data: 4 2 2 3 4 4 3 2 2 2
32805 284/300 Flush
32806 285/300 Read block 3588 len 12288
32807 286/300 Write block 786 len 40960 data: 3 3 4 4 4 5 7 7 8 6
32808 287/300 Read block 624 len 24576
32809 288/300 Flush
32810 289/300 Write block 4904 len 36864 data: 9 9 9 8 8 5 4 5 5
32811 290/300 Write block 4068 len 24576 data: 8 4 4 6 4 5
32812 291/300 Flush
32813 292/300 Read block 1920 len 40960
32814 293/300 Write block 3563 len 4096 data: 4
32815 294/300 Write block 1825 len 20480 data: 7 8 7 10 10
32816 295/300 Read block 178 len 24576
32817 296/300 Write block 796 len 12288 data: 7 7 5
32818 297/300 Read block 412 len 36864
32819 298/300 Flush
32820 299/300 Write block 2085 len 36864 data: 5 6 7 5 6 7 7 7 7
32821 300/300 Flush
32822 301/300 Read block 930 len 16384
32823 Main task has work to do, go find it
32824 got dsc Start(0)
32825 start 0
32826 Feb 07 20:14:36.579 INFO request completed, latency_us: 201, response_code: 204, uri: /start/cid/0, method: POST, req_id: 8689dc9b-2855-4a26-a255-0d70dcb128fc, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
32827 Replay: started 0, returned:Ok(())
32828 ----------------------------------------------------------------
32829 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32830 States: Offline Active Active
32831 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32832 9153 Acked Read 7 Sent Done Done false
32833 9154 Acked Read 4 Sent Done Done false
32834 9155 Acked Read 5 Sent Done Done false
32835 9156 Acked Read 1 Sent Done Done false
32836 9157 Acked Read 9 Sent Done Done false
32837 9158 Acked Read 8 Sent Done Done false
32838 9159 Acked Write 5 Sent Done Done false
32839 9160 Acked Write 1 Sent Done Done false
32840 9161 Acked Write 10 Sent Done Done false
32841 9162 Acked Write 10 Sent Done Done false
32842 9163 Acked Write 5 Sent Done Done false
32843 9164 Acked Write 7 Sent Done Done false
32844 9165 Acked Write 5 Sent Done Done false
32845 9166 Acked Flush 0 Sent Done Done false
32846 9167 Acked Write 1 Sent Done Done false
32847 9168 Acked Read 5 Sent Done Done false
32848 9169 Acked Read 8 Sent Done Done false
32849 9170 Acked Read 8 Sent Done Done false
32850 9171 Acked Read 6 Sent Done Done false
32851 9172 Acked Flush 0 Sent Done Done false
32852 9173 Acked Read 7 Sent Done Done false
32853 9174 Acked Read 5 Sent Done Done false
32854 9175 Acked Read 7 Sent Done Done false
32855 9176 Acked Read 7 Sent Done Done false
32856 9177 Acked Flush 0 Sent Done Done false
32857 9178 Acked Read 9 Sent Done Done false
32858 9179 Acked Write 6 Sent Done Done false
32859 9180 Acked Flush 0 Sent Done Done false
32860 9181 Acked Write 9 Sent Done Done false
32861 9182 Acked Write 4 Sent Done Done false
32862 9183 Acked Write 5 Sent Done Done false
32863 9184 Acked Read 8 Sent Done Done false
32864 9185 Acked Read 6 Sent Done Done false
32865 9186 Acked Write 5 Sent Done Done false
32866 9187 Acked Write 1 Sent Done Done false
32867 9188 Acked Read 6 Sent Done Done false
32868 9189 Acked Write 5 Sent Done Done false
32869 9190 Acked Read 2 Sent Done Done false
32870 9191 Acked Read 1 Sent Done Done false
32871 9192 Acked Write 7 Sent Done Done false
32872 9193 Acked Write 7 Sent Done Done false
32873 9194 Acked Read 1 Sent Done Done false
32874 9195 Acked Read 10 Sent Done Done false
32875 9196 Acked Write 9 Sent Done Done false
32876 9197 Acked Read 5 Sent Done Done false
32877 9198 Acked Flush 0 Sent Done Done false
32878 9199 Acked Flush 0 Sent Done Done false
32879 9200 Acked Write 9 Sent Done Done false
32880 9201 Acked Write 1 Sent Done Done false
32881 9202 Acked Read 2 Sent Done Done false
32882 9203 Acked Flush 0 Sent Done Done false
32883 9204 Acked Write 8 Sent Done Done false
32884 9205 Acked Write 8 Sent Done Done false
32885 9206 Acked Write 3 Sent Done Done false
32886 9207 Acked Write 3 Sent Done Done false
32887 9208 Acked Read 4 Sent Done Done false
32888 9209 Acked Read 4 Sent Done Done false
32889 9210 Acked Read 8 Sent Done Done false
32890 9211 Acked Read 2 Sent Done Done false
32891 9212 Acked Flush 0 Sent Done Done false
32892 9213 Acked Write 7 Sent Done Done false
32893 9214 Acked Read 4 Sent Done Done false
32894 9215 Acked Write 5 Sent Done Done false
32895 9216 Acked Write 10 Sent Done Done false
32896 9217 Acked Write 4 Sent Done Done false
32897 9218 Acked Read 2 Sent Done Done false
32898 9219 Acked Flush 0 Sent Done Done false
32899 9220 Acked Write 3 Sent Done Done false
32900 9221 Acked Read 7 Sent Done Done false
32901 9222 Acked Read 3 Sent Done Done false
32902 9223 Acked Flush 0 Sent Done Done false
32903 9224 Acked Write 9 Sent Done Done false
32904 9225 Acked Read 8 Sent Done Done false
32905 9226 Acked Write 2 Sent Done Done false
32906 9227 Acked Read 5 Sent Done Done false
32907 9228 Acked Write 4 Sent Done Done false
32908 9229 Acked Read 7 Sent Done Done false
32909 9230 Acked Read 6 Sent Done Done false
32910 9231 Acked Read 6 Sent Done Done false
32911 9232 Acked Read 8 Sent Done Done false
32912 9233 Acked Read 7 Sent Done Done false
32913 9234 Acked Write 8 Sent Done Done false
32914 9235 Acked Read 5 Sent Done Done false
32915 9236 Acked Read 1 Sent Done Done false
32916 9237 Acked Flush 0 Sent Done Done false
32917 9238 Acked Read 3 Sent Done Done false
32918 9239 Acked Write 8 Sent Done Done false
32919 9240 Acked Read 5 Sent Done Done false
32920 9241 Acked Read 10 Sent Done Done false
32921 9242 Acked Flush 0 Sent Done Done false
32922 9243 Acked Read 1 Sent Done Done false
32923 9244 Acked Write 1 Sent Done Done false
32924 9245 Acked Read 10 Sent Done Done false
32925 9246 Acked Flush 0 Sent Done Done false
32926 9247 Acked Read 2 Sent Done Done false
32927 9248 Acked Flush 0 Sent Done Done false
32928 9249 Acked Write 3 Sent Done Done false
32929 9250 Acked Write 7 Sent Done Done false
32930 9251 Acked Read 1 Sent Done Done false
32931 9252 Acked Read 1 Sent Done Done false
32932 9253 Acked Write 6 Sent Done Done false
32933 9254 Acked Write 9 Sent Done Done false
32934 9255 Acked Read 9 Sent Done Done false
32935 9256 Acked Read 7 Sent Done Done false
32936 9257 Acked Read 3 Sent Done Done false
32937 9258 Acked Read 6 Sent Done Done false
32938 9259 Acked Write 10 Sent Done Done false
32939 9260 Acked Write 3 Sent Done Done false
32940 9261 Acked Write 2 Sent Done Done false
32941 9262 Acked Flush 0 Sent Done Done false
32942 9263 Acked Write 9 Sent Done Done false
32943 9264 Acked Read 5 Sent Done Done false
32944 9265 Acked Read 3 Sent Done Done false
32945 9266 Acked Read 8 Sent Done Done false
32946 9267 Acked Flush 0 Sent Done Done false
32947 9268 Acked Read 2 Sent Done Done false
32948 9269 Acked Write 7 Sent Done Done false
32949 9270 Acked Write 10 Sent Done Done false
32950 9271 Acked Read 7 Sent Done Done false
32951 9272 Acked Read 10 Sent Done Done false
32952 9273 Acked Read 8 Sent Done Done false
32953 9274 Acked Write 10 Sent Done Done false
32954 9275 Acked Read 10 Sent Done Done false
32955 9276 Acked Write 1 Sent Done Done false
32956 9277 Acked Write 6 Sent Done Done false
32957 9278 Acked Read 1 Sent Done Done false
32958 9279 Acked Write 4 Sent Done Done false
32959 9280 Acked Write 8 Sent Done Done false
32960 9281 Acked Write 6 Sent Done Done false
32961 9282 Acked Read 3 Sent Done Done false
32962 9283 Acked Read 6 Sent Done Done false
32963 9284 Acked Write 4 Sent Done Done false
32964 9285 Acked Read 1 Sent Done Done false
32965 9286 Acked Read 4 Sent Done Done false
32966 9287 Acked Read 2 Sent Done Done false
32967 9288 Acked Read 6 Sent Done Done false
32968 9289 Acked Read 3 Sent Done Done false
32969 9290 Acked Read 9 Sent Done Done false
32970 9291 Acked Write 4 Sent Done Done false
32971 9292 Acked Read 6 Sent Done Done false
32972 9293 Acked Read 8 Sent Done Done false
32973 9294 Acked Read 5 Sent Done Done false
32974 9295 Acked Write 6 Sent Done Done false
32975 9296 Acked Write 3 Sent Done Done false
32976 9297 Acked Flush 0 Sent Done Done false
32977 9298 Acked Flush 0 Sent Done Done false
32978 9299 Acked Read 1 Sent Done Done false
32979 9300 Acked Write 8 Sent Done Done false
32980 9301 Acked Write 2 Sent Done Done false
32981 9302 Acked Read 4 Sent Done Done false
32982 9303 Acked Write 8 Sent Done Done false
32983 9304 Acked Write 7 Sent Done Done false
32984 9305 Acked Write 6 Sent Done Done false
32985 9306 Acked Write 8 Sent Done Done false
32986 9307 Acked Write 10 Sent Done Done false
32987 9308 Acked Read 4 Sent Done Done false
32988 9309 Acked Write 8 Sent Done Done false
32989 9310 Acked Read 6 Sent Done Done false
32990 9311 Acked Read 4 Sent Done Done false
32991 9312 Acked Read 6 Sent Done Done false
32992 9313 Acked Write 10 Sent Done Done false
32993 9314 Acked Read 5 Sent Done Done false
32994 9315 Acked Read 6 Sent Done Done false
32995 9316 Acked Read 1 Sent Done Done false
32996 9317 Acked Read 9 Sent Done Done false
32997 9318 Acked Read 8 Sent Done Done false
32998 9319 Acked Read 5 Sent Done Done false
32999 9320 Acked Write 3 Sent Done Done false
33000 9321 Acked Write 7 Sent Done Done false
33001 9322 Acked Read 8 Sent Done Done false
33002 9323 Acked Read 4 Sent Done Done false
33003 9324 Acked Read 7 Sent Done Done false
33004 9325 Acked Write 10 Sent Done Done false
33005 9326 Acked Read 3 Sent Done Done false
33006 9327 Acked Write 9 Sent Done Done false
33007 9328 Acked Read 6 Sent Done Done false
33008 9329 Acked Write 6 Sent Done Done false
33009 9330 Acked Write 10 Sent Done Done false
33010 9331 Acked Read 9 Sent Done Done false
33011 9332 Acked Flush 0 Sent Done Done false
33012 9333 Acked Read 3 Sent Done Done false
33013 9334 Acked Write 8 Sent Done Done false
33014 9335 Acked Flush 0 Sent Done Done false
33015 9336 Acked Read 5 Sent Done Done false
33016 9337 Acked Read 9 Sent Done Done false
33017 9338 Acked Write 7 Sent Done Done false
33018 9339 Acked Write 6 Sent Done Done false
33019 9340 Acked Write 2 Sent Done Done false
33020 9341 Acked Read 4 Sent Done Done false
33021 9342 Acked Read 5 Sent Done Done false
33022 9343 Acked Read 6 Sent Done Done false
33023 9344 Acked Read 6 Sent Done Done false
33024 9345 Acked Write 9 Sent Done Done false
33025 9346 Acked Write 9 Sent Done Done false
33026 9347 Acked Flush 0 Sent Done Done false
33027 9348 Acked Write 3 Sent Done Done false
33028 9349 Acked Read 10 Sent Done Done false
33029 9350 Acked Write 7 Sent Done Done false
33030 9351 Acked Flush 0 Sent Done Done false
33031 9352 Acked Read 9 Sent Done Done false
33032 9353 Acked Read 10 Sent Done Done false
33033 9354 Acked Read 10 Sent Done Done false
33034 9355 Acked Flush 0 Sent Done Done false
33035 9356 Acked Read 7 Sent Done Done false
33036 9357 Acked Write 10 Sent Done Done false
33037 9358 Acked Read 4 Sent Done Done false
33038 9359 Acked Write 7 Sent Done Done false
33039 9360 Acked Write 8 Sent Done Done false
33040 9361 Acked Read 9 Sent Done Done false
33041 9362 Acked Read 4 Sent Done Done false
33042 9363 Acked Read 5 Sent Done Done false
33043 9364 Acked Read 2 Sent Done Done false
33044 9365 Acked Write 1 Sent Done Done false
33045 9366 Acked Write 4 Sent Done Done false
33046 9367 Acked Read 6 Sent Done Done false
33047 9368 Acked Write 8 Sent Done Done false
33048 9369 Acked Flush 0 Sent Done Done false
33049 9370 Acked Write 9 Sent Done Done false
33050 9371 Acked Flush 0 Sent Done Done false
33051 9372 Acked Write 6 Sent Done Done false
33052 9373 Acked Read 4 Sent Done Done false
33053 9374 Acked Read 2 Sent Done Done false
33054 9375 Acked Read 7 Sent Done Done false
33055 9376 Acked Write 5 Sent Done Done false
33056 9377 Acked Read 6 Sent Done Done false
33057 9378 Acked Read 8 Sent Done Done false
33058 9379 Acked Read 3 Sent Done Done false
33059 9380 Acked Read 9 Sent Done Done false
33060 9381 Acked Read 7 Sent Done Done false
33061 9382 Acked Write 9 Sent Done Done false
33062 9383 Acked Write 10 Sent Done Done false
33063 9384 Acked Read 9 Sent Done Done false
33064 9385 Acked Write 10 Sent Done Done false
33065 9386 Acked Write 7 Sent Done Done false
33066 9387 Acked Write 1 Sent Done Done false
33067 9388 Acked Write 4 Sent Done Done false
33068 9389 Acked Write 3 Sent Done Done false
33069 9390 Acked Read 9 Sent Done Done false
33070 9391 Acked Read 10 Sent Done Done false
33071 9392 Acked Flush 0 Sent Done Done false
33072 9393 Acked Write 3 Sent Done Done false
33073 9394 Acked Write 4 Sent Done Done false
33074 9395 Acked Read 9 Sent Done Done false
33075 9396 Acked Flush 0 Sent Done Done false
33076 9397 Acked Read 7 Sent Done Done false
33077 9398 Acked Flush 0 Sent Done Done false
33078 9399 Acked Read 8 Sent Done Done false
33079 9400 Acked Write 5 Sent Done Done false
33080 9401 Acked Read 7 Sent Done Done false
33081 9402 Acked Write 4 Sent Done Done false
33082 9403 Acked Read 2 Sent Done Done false
33083 9404 Acked Write 4 Sent Done Done false
33084 9405 Acked Read 6 Sent Done Done false
33085 9406 Acked Write 5 Sent Done Done false
33086 9407 Acked Read 3 Sent Done Done false
33087 9408 Acked Write 8 Sent Done Done false
33088 9409 Acked Read 3 Sent Done Done false
33089 9410 Acked Read 10 Sent Done Done false
33090 9411 Acked Flush 0 Sent Done Done false
33091 9412 Acked Read 3 Sent Done Done false
33092 9413 Acked Read 4 Sent Done Done false
33093 9414 Acked Write 9 Sent Done Done false
33094 9415 Acked Read 3 Sent Done Done false
33095 9416 Acked Read 8 Sent Done Done false
33096 9417 Acked Write 6 Sent Done Done false
33097 9418 Acked Read 3 Sent Done Done false
33098 9419 Acked Write 1 Sent Done Done false
33099 9420 Acked Read 7 Sent Done Done false
33100 9421 Acked Read 6 Sent Done Done false
33101 9422 Acked Write 3 Sent Done Done false
33102 9423 Acked Write 4 Sent Done Done false
33103 9424 Acked Write 3 Sent Done Done false
33104 9425 Acked Write 7 Sent Done Done false
33105 9426 Acked Write 3 Sent Done Done false
33106 9427 Acked Write 5 Sent Done Done false
33107 9428 Acked Write 3 Sent Done Done false
33108 9429 Acked Write 5 Sent Done Done false
33109 9430 Acked Read 3 Sent Done Done false
33110 9431 Acked Write 9 Sent Done Done false
33111 9432 Acked Read 10 Sent Done Done false
33112 9433 Acked Read 2 Sent Done Done false
33113 9434 Acked Read 4 Sent Done Done false
33114 9435 Acked Write 10 Sent Done Done false
33115 9436 Acked Flush 0 Sent Done Done false
33116 9437 Acked Read 3 Sent Done Done false
33117 9438 Acked Write 10 Sent Done Done false
33118 9439 Acked Read 6 Sent Done Done false
33119 9440 Acked Flush 0 Sent Done Done false
33120 9441 Acked Write 9 Sent Done Done false
33121 9442 Acked Write 6 Sent Done Done false
33122 9443 Acked Flush 0 Sent Done Done false
33123 9444 Acked Read 10 Sent Done Done false
33124 9445 Acked Write 1 Sent Done Done false
33125 9446 Acked Write 5 Sent Done Done false
33126 9447 Acked Read 6 Sent Done Done false
33127 9448 Acked Write 3 Sent Done Done false
33128 9449 Acked Read 9 Sent Done Done false
33129 9450 Acked Flush 0 Sent Done Done false
33130 9451 Acked Write 9 Sent Done Done false
33131 9452 Acked Flush 0 Sent Done Done false
33132 9453 Acked Read 4 Sent Done Done false
33133 STATES DS:0 DS:1 DS:2 TOTAL
33134 Sent 301 0 0 301
33135 Done 0 301 301 602
33136 Skipped 0 0 0 0
33137 Error 0 0 0 0
33138 Last Flush: Some(JobId(9152)) Some(JobId(9452)) Some(JobId(9452))
33139 Downstairs last five completed: 9152 9151 9150 9149 9148
33140 Upstairs last five completed: 9453 9452 9451 9450 9449
33141 CLIENT: Up:0 ds:301 act:2
331422025-02-07T20:14:37.081ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
331432025-02-07T20:14:37.655ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
331442025-02-07T20:14:38.804ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
331452025-02-07T20:14:39.659ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
33146 [8810] got start action
33147 [8810] I am going to restart
33148 Starting downstairs at port 8810
33149 Make output file at "/tmp/dsc/downstairs-8810.txt"
33150 [8810][0] reports Starting
33151 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1237)
33152 [8810][0] reports Running
33153 ----------------------------------------------------------------
33154 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33155 States: Offline Active Active
33156 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33157 9153 Acked Read 7 Sent Done Done false
33158 9154 Acked Read 4 Sent Done Done false
33159 9155 Acked Read 5 Sent Done Done false
33160 9156 Acked Read 1 Sent Done Done false
33161 9157 Acked Read 9 Sent Done Done false
33162 9158 Acked Read 8 Sent Done Done false
33163 9159 Acked Write 5 Sent Done Done false
33164 9160 Acked Write 1 Sent Done Done false
33165 9161 Acked Write 10 Sent Done Done false
33166 9162 Acked Write 10 Sent Done Done false
33167 9163 Acked Write 5 Sent Done Done false
33168 9164 Acked Write 7 Sent Done Done false
33169 9165 Acked Write 5 Sent Done Done false
33170 9166 Acked Flush 0 Sent Done Done false
33171 9167 Acked Write 1 Sent Done Done false
33172 9168 Acked Read 5 Sent Done Done false
33173 9169 Acked Read 8 Sent Done Done false
33174 9170 Acked Read 8 Sent Done Done false
33175 9171 Acked Read 6 Sent Done Done false
33176 9172 Acked Flush 0 Sent Done Done false
33177 9173 Acked Read 7 Sent Done Done false
33178 9174 Acked Read 5 Sent Done Done false
33179 9175 Acked Read 7 Sent Done Done false
33180 9176 Acked Read 7 Sent Done Done false
33181 9177 Acked Flush 0 Sent Done Done false
33182 9178 Acked Read 9 Sent Done Done false
33183 9179 Acked Write 6 Sent Done Done false
33184 9180 Acked Flush 0 Sent Done Done false
33185 9181 Acked Write 9 Sent Done Done false
33186 9182 Acked Write 4 Sent Done Done false
33187 9183 Acked Write 5 Sent Done Done false
33188 9184 Acked Read 8 Sent Done Done false
33189 9185 Acked Read 6 Sent Done Done false
33190 9186 Acked Write 5 Sent Done Done false
33191 9187 Acked Write 1 Sent Done Done false
33192 9188 Acked Read 6 Sent Done Done false
33193 9189 Acked Write 5 Sent Done Done false
33194 9190 Acked Read 2 Sent Done Done false
33195 9191 Acked Read 1 Sent Done Done false
33196 9192 Acked Write 7 Sent Done Done false
33197 9193 Acked Write 7 Sent Done Done false
33198 9194 Acked Read 1 Sent Done Done false
33199 9195 Acked Read 10 Sent Done Done false
33200 9196 Acked Write 9 Sent Done Done false
33201 9197 Acked Read 5 Sent Done Done false
33202 9198 Acked Flush 0 Sent Done Done false
33203 9199 Acked Flush 0 Sent Done Done false
33204 9200 Acked Write 9 Sent Done Done false
33205 9201 Acked Write 1 Sent Done Done false
33206 9202 Acked Read 2 Sent Done Done false
33207 9203 Acked Flush 0 Sent Done Done false
33208 9204 Acked Write 8 Sent Done Done false
33209 9205 Acked Write 8 Sent Done Done false
33210 9206 Acked Write 3 Sent Done Done false
33211 9207 Acked Write 3 Sent Done Done false
33212 9208 Acked Read 4 Sent Done Done false
33213 9209 Acked Read 4 Sent Done Done false
33214 9210 Acked Read 8 Sent Done Done false
33215 9211 Acked Read 2 Sent Done Done false
33216 9212 Acked Flush 0 Sent Done Done false
33217 9213 Acked Write 7 Sent Done Done false
33218 9214 Acked Read 4 Sent Done Done false
33219 9215 Acked Write 5 Sent Done Done false
33220 9216 Acked Write 10 Sent Done Done false
33221 9217 Acked Write 4 Sent Done Done false
33222 9218 Acked Read 2 Sent Done Done false
33223 9219 Acked Flush 0 Sent Done Done false
33224 9220 Acked Write 3 Sent Done Done false
33225 9221 Acked Read 7 Sent Done Done false
33226 9222 Acked Read 3 Sent Done Done false
33227 9223 Acked Flush 0 Sent Done Done false
33228 9224 Acked Write 9 Sent Done Done false
33229 9225 Acked Read 8 Sent Done Done false
33230 9226 Acked Write 2 Sent Done Done false
33231 9227 Acked Read 5 Sent Done Done false
33232 9228 Acked Write 4 Sent Done Done false
33233 9229 Acked Read 7 Sent Done Done false
33234 9230 Acked Read 6 Sent Done Done false
33235 9231 Acked Read 6 Sent Done Done false
33236 9232 Acked Read 8 Sent Done Done false
33237 9233 Acked Read 7 Sent Done Done false
33238 9234 Acked Write 8 Sent Done Done false
33239 9235 Acked Read 5 Sent Done Done false
33240 9236 Acked Read 1 Sent Done Done false
33241 9237 Acked Flush 0 Sent Done Done false
33242 9238 Acked Read 3 Sent Done Done false
33243 9239 Acked Write 8 Sent Done Done false
33244 9240 Acked Read 5 Sent Done Done false
33245 9241 Acked Read 10 Sent Done Done false
33246 9242 Acked Flush 0 Sent Done Done false
33247 9243 Acked Read 1 Sent Done Done false
33248 9244 Acked Write 1 Sent Done Done false
33249 9245 Acked Read 10 Sent Done Done false
33250 9246 Acked Flush 0 Sent Done Done false
33251 9247 Acked Read 2 Sent Done Done false
33252 9248 Acked Flush 0 Sent Done Done false
33253 9249 Acked Write 3 Sent Done Done false
33254 9250 Acked Write 7 Sent Done Done false
33255 9251 Acked Read 1 Sent Done Done false
33256 9252 Acked Read 1 Sent Done Done false
33257 9253 Acked Write 6 Sent Done Done false
33258 9254 Acked Write 9 Sent Done Done false
33259 9255 Acked Read 9 Sent Done Done false
33260 9256 Acked Read 7 Sent Done Done false
33261 9257 Acked Read 3 Sent Done Done false
33262 9258 Acked Read 6 Sent Done Done false
33263 9259 Acked Write 10 Sent Done Done false
33264 9260 Acked Write 3 Sent Done Done false
33265 9261 Acked Write 2 Sent Done Done false
33266 9262 Acked Flush 0 Sent Done Done false
33267 9263 Acked Write 9 Sent Done Done false
33268 9264 Acked Read 5 Sent Done Done false
33269 9265 Acked Read 3 Sent Done Done false
33270 9266 Acked Read 8 Sent Done Done false
33271 9267 Acked Flush 0 Sent Done Done false
33272 9268 Acked Read 2 Sent Done Done false
33273 9269 Acked Write 7 Sent Done Done false
33274 9270 Acked Write 10 Sent Done Done false
33275 9271 Acked Read 7 Sent Done Done false
33276 9272 Acked Read 10 Sent Done Done false
33277 9273 Acked Read 8 Sent Done Done false
33278 9274 Acked Write 10 Sent Done Done false
33279 9275 Acked Read 10 Sent Done Done false
33280 9276 Acked Write 1 Sent Done Done false
33281 9277 Acked Write 6 Sent Done Done false
33282 9278 Acked Read 1 Sent Done Done false
33283 9279 Acked Write 4 Sent Done Done false
33284 9280 Acked Write 8 Sent Done Done false
33285 9281 Acked Write 6 Sent Done Done false
33286 9282 Acked Read 3 Sent Done Done false
33287 9283 Acked Read 6 Sent Done Done false
33288 9284 Acked Write 4 Sent Done Done false
33289 9285 Acked Read 1 Sent Done Done false
33290 9286 Acked Read 4 Sent Done Done false
33291 9287 Acked Read 2 Sent Done Done false
33292 9288 Acked Read 6 Sent Done Done false
33293 9289 Acked Read 3 Sent Done Done false
33294 9290 Acked Read 9 Sent Done Done false
33295 9291 Acked Write 4 Sent Done Done false
33296 9292 Acked Read 6 Sent Done Done false
33297 9293 Acked Read 8 Sent Done Done false
33298 9294 Acked Read 5 Sent Done Done false
33299 9295 Acked Write 6 Sent Done Done false
33300 9296 Acked Write 3 Sent Done Done false
33301 9297 Acked Flush 0 Sent Done Done false
33302 9298 Acked Flush 0 Sent Done Done false
33303 9299 Acked Read 1 Sent Done Done false
33304 9300 Acked Write 8 Sent Done Done false
33305 9301 Acked Write 2 Sent Done Done false
33306 9302 Acked Read 4 Sent Done Done false
33307 9303 Acked Write 8 Sent Done Done false
33308 9304 Acked Write 7 Sent Done Done false
33309 9305 Acked Write 6 Sent Done Done false
33310 9306 Acked Write 8 Sent Done Done false
33311 9307 Acked Write 10 Sent Done Done false
33312 9308 Acked Read 4 Sent Done Done false
33313 9309 Acked Write 8 Sent Done Done false
33314 9310 Acked Read 6 Sent Done Done false
33315 9311 Acked Read 4 Sent Done Done false
33316 9312 Acked Read 6 Sent Done Done false
33317 9313 Acked Write 10 Sent Done Done false
33318 9314 Acked Read 5 Sent Done Done false
33319 9315 Acked Read 6 Sent Done Done false
33320 9316 Acked Read 1 Sent Done Done false
33321 9317 Acked Read 9 Sent Done Done false
33322 9318 Acked Read 8 Sent Done Done false
33323 9319 Acked Read 5 Sent Done Done false
33324 9320 Acked Write 3 Sent Done Done false
33325 9321 Acked Write 7 Sent Done Done false
33326 9322 Acked Read 8 Sent Done Done false
33327 9323 Acked Read 4 Sent Done Done false
33328 9324 Acked Read 7 Sent Done Done false
33329 9325 Acked Write 10 Sent Done Done false
33330 9326 Acked Read 3 Sent Done Done false
33331 9327 Acked Write 9 Sent Done Done false
33332 9328 Acked Read 6 Sent Done Done false
33333 9329 Acked Write 6 Sent Done Done false
33334 9330 Acked Write 10 Sent Done Done false
33335 9331 Acked Read 9 Sent Done Done false
33336 9332 Acked Flush 0 Sent Done Done false
33337 9333 Acked Read 3 Sent Done Done false
33338 9334 Acked Write 8 Sent Done Done false
33339 9335 Acked Flush 0 Sent Done Done false
33340 9336 Acked Read 5 Sent Done Done false
33341 9337 Acked Read 9 Sent Done Done false
33342 9338 Acked Write 7 Sent Done Done false
33343 9339 Acked Write 6 Sent Done Done false
33344 9340 Acked Write 2 Sent Done Done false
33345 9341 Acked Read 4 Sent Done Done false
33346 9342 Acked Read 5 Sent Done Done false
33347 9343 Acked Read 6 Sent Done Done false
33348 9344 Acked Read 6 Sent Done Done false
33349 9345 Acked Write 9 Sent Done Done false
33350 9346 Acked Write 9 Sent Done Done false
33351 9347 Acked Flush 0 Sent Done Done false
33352 9348 Acked Write 3 Sent Done Done false
33353 9349 Acked Read 10 Sent Done Done false
33354 9350 Acked Write 7 Sent Done Done false
33355 9351 Acked Flush 0 Sent Done Done false
33356 9352 Acked Read 9 Sent Done Done false
33357 9353 Acked Read 10 Sent Done Done false
33358 9354 Acked Read 10 Sent Done Done false
33359 9355 Acked Flush 0 Sent Done Done false
33360 9356 Acked Read 7 Sent Done Done false
33361 9357 Acked Write 10 Sent Done Done false
33362 9358 Acked Read 4 Sent Done Done false
33363 9359 Acked Write 7 Sent Done Done false
33364 9360 Acked Write 8 Sent Done Done false
33365 9361 Acked Read 9 Sent Done Done false
33366 9362 Acked Read 4 Sent Done Done false
33367 9363 Acked Read 5 Sent Done Done false
33368 9364 Acked Read 2 Sent Done Done false
33369 9365 Acked Write 1 Sent Done Done false
33370 9366 Acked Write 4 Sent Done Done false
33371 9367 Acked Read 6 Sent Done Done false
33372 9368 Acked Write 8 Sent Done Done false
33373 9369 Acked Flush 0 Sent Done Done false
33374 9370 Acked Write 9 Sent Done Done false
33375 9371 Acked Flush 0 Sent Done Done false
33376 9372 Acked Write 6 Sent Done Done false
33377 9373 Acked Read 4 Sent Done Done false
33378 9374 Acked Read 2 Sent Done Done false
33379 9375 Acked Read 7 Sent Done Done false
33380 9376 Acked Write 5 Sent Done Done false
33381 9377 Acked Read 6 Sent Done Done false
33382 9378 Acked Read 8 Sent Done Done false
33383 9379 Acked Read 3 Sent Done Done false
33384 9380 Acked Read 9 Sent Done Done false
33385 9381 Acked Read 7 Sent Done Done false
33386 9382 Acked Write 9 Sent Done Done false
33387 9383 Acked Write 10 Sent Done Done false
33388 9384 Acked Read 9 Sent Done Done false
33389 9385 Acked Write 10 Sent Done Done false
33390 9386 Acked Write 7 Sent Done Done false
33391 9387 Acked Write 1 Sent Done Done false
33392 9388 Acked Write 4 Sent Done Done false
33393 9389 Acked Write 3 Sent Done Done false
33394 9390 Acked Read 9 Sent Done Done false
33395 9391 Acked Read 10 Sent Done Done false
33396 9392 Acked Flush 0 Sent Done Done false
33397 9393 Acked Write 3 Sent Done Done false
33398 9394 Acked Write 4 Sent Done Done false
33399 9395 Acked Read 9 Sent Done Done false
33400 9396 Acked Flush 0 Sent Done Done false
33401 9397 Acked Read 7 Sent Done Done false
33402 9398 Acked Flush 0 Sent Done Done false
33403 9399 Acked Read 8 Sent Done Done false
33404 9400 Acked Write 5 Sent Done Done false
33405 9401 Acked Read 7 Sent Done Done false
33406 9402 Acked Write 4 Sent Done Done false
33407 9403 Acked Read 2 Sent Done Done false
33408 9404 Acked Write 4 Sent Done Done false
33409 9405 Acked Read 6 Sent Done Done false
33410 9406 Acked Write 5 Sent Done Done false
33411 9407 Acked Read 3 Sent Done Done false
33412 9408 Acked Write 8 Sent Done Done false
33413 9409 Acked Read 3 Sent Done Done false
33414 9410 Acked Read 10 Sent Done Done false
33415 9411 Acked Flush 0 Sent Done Done false
33416 9412 Acked Read 3 Sent Done Done false
33417 9413 Acked Read 4 Sent Done Done false
33418 9414 Acked Write 9 Sent Done Done false
33419 9415 Acked Read 3 Sent Done Done false
33420 9416 Acked Read 8 Sent Done Done false
33421 9417 Acked Write 6 Sent Done Done false
33422 9418 Acked Read 3 Sent Done Done false
33423 9419 Acked Write 1 Sent Done Done false
33424 9420 Acked Read 7 Sent Done Done false
33425 9421 Acked Read 6 Sent Done Done false
33426 9422 Acked Write 3 Sent Done Done false
33427 9423 Acked Write 4 Sent Done Done false
33428 9424 Acked Write 3 Sent Done Done false
33429 9425 Acked Write 7 Sent Done Done false
33430 9426 Acked Write 3 Sent Done Done false
33431 9427 Acked Write 5 Sent Done Done false
33432 9428 Acked Write 3 Sent Done Done false
33433 9429 Acked Write 5 Sent Done Done false
33434 9430 Acked Read 3 Sent Done Done false
33435 9431 Acked Write 9 Sent Done Done false
33436 9432 Acked Read 10 Sent Done Done false
33437 9433 Acked Read 2 Sent Done Done false
33438 9434 Acked Read 4 Sent Done Done false
33439 9435 Acked Write 10 Sent Done Done false
33440 9436 Acked Flush 0 Sent Done Done false
33441 9437 Acked Read 3 Sent Done Done false
33442 9438 Acked Write 10 Sent Done Done false
33443 9439 Acked Read 6 Sent Done Done false
33444 9440 Acked Flush 0 Sent Done Done false
33445 9441 Acked Write 9 Sent Done Done false
33446 9442 Acked Write 6 Sent Done Done false
33447 9443 Acked Flush 0 Sent Done Done false
33448 9444 Acked Read 10 Sent Done Done false
33449 9445 Acked Write 1 Sent Done Done false
33450 9446 Acked Write 5 Sent Done Done false
33451 9447 Acked Read 6 Sent Done Done false
33452 9448 Acked Write 3 Sent Done Done false
33453 9449 Acked Read 9 Sent Done Done false
33454 9450 Acked Flush 0 Sent Done Done false
33455 9451 Acked Write 9 Sent Done Done false
33456 9452 Acked Flush 0 Sent Done Done false
33457 9453 Acked Read 4 Sent Done Done false
33458 STATES DS:0 DS:1 DS:2 TOTAL
33459 Sent 301 0 0 301
33460 Done 0 301 301 602
33461 Skipped 0 0 0 0
33462 Error 0 0 0 0
33463 Last Flush: Some(JobId(9152)) Some(JobId(9452)) Some(JobId(9452))
33464 Downstairs last five completed: 9152 9151 9150 9149 9148
33465 Upstairs last five completed: 9453 9452 9451 9450 9449
33466 CLIENT: Up:0 ds:301 act:2
334672025-02-07T20:14:41.084ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
334682025-02-07T20:14:41.663ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
334692025-02-07T20:14:42.328ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
334702025-02-07T20:14:42.328ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61073) = io task client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
334712025-02-07T20:14:42.329ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
334722025-02-07T20:14:42.330ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
334732025-02-07T20:14:42.330ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
334742025-02-07T20:14:42.330ZINFOcrucible: send last flush ID to this DS: Some(JobId(9152)) = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
334752025-02-07T20:14:42.330ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
334762025-02-07T20:14:42.330ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(9152)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
334772025-02-07T20:14:42.330ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(9152)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
33478 ----------------------------------------------------------------
33479 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33480 Downstairs last five completed: 9454 9453 9452 9451 9450
33481 Upstairs last five completed: 9454 9453 9452 9451 9450
33482 CLIENT: Up:0 ds:0 act:3
33483 Replay: All jobs finished
33484 Main task has work to do, go find it
33485 got dsc Stop(1)
33486 stop 1
33487 [8820] Got stop action so:false kr:false
33488 Feb 07 20:14:44.584 INFO request completed, latency_us: 375, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 31ec1652-9b27-4555-9b55-d05733749309, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
334892025-02-07T20:14:44.584ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
334902025-02-07T20:14:44.584ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
334912025-02-07T20:14:44.584ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
33492 Feb 07 20:14:44.585 INFO request completed, latency_us: 123, response_code: 200, uri: /state/cid/1, method: GET, req_id: 0c11207b-2449-4630-9924-5e20cb0dc826, remote_addr:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-07T20:14:44.585086985Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2","pid":1209,"":"io task","client":"1","":"downstairs","session_id":"8f7ba378-6b82-4db6-a812-9286715f4b09"}
33493 127.0.0.1:43825, local_addr: 127.0.0.1:9998
33494 [8820] Exited with: ExitStatus(unix_wait_status(9))
33495 [8820][1] reports Exit
33496 Feb 07 20:14:48.587 INFO request completed, latency_us: 222, response_code: 200, uri: /state/cid/1, method: GET, req_id: 97944f72-2608-4f96-bcc2-086d27cff471, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
33497 001/300 Write block 775 len 20480 data: 7 6 6 6 5
33498 002/300 Flush
33499 003/300 Read block 4371 len 16384
33500 004/300 Read block 1111 len 12288
33501 005/300 Write block 1418 len 28672 data: 4 6 6 7 6 5 5
33502 006/300 Read block 1733 len 12288
33503 007/300 Flush
33504 008/300 Read block 4796 len 20480
33505 009/300 Read block 4178 len 40960
33506 010/300 Flush
33507 011/300 Write block 4943 len 32768 data: 10 11 11 9 8 9 8 6
33508 012/300 Read block 1579 len 20480
33509 013/300 Write block 3669 len 20480 data: 8 6 6 5 7
33510 014/300 Write block 2435 len 8192 data: 5 5
33511 015/300 Write block 441 len 8192 data: 4 3
33512 016/300 Write block 356 len 8192 data: 2 3
33513 017/300 Write block 4918 len 20480 data: 4 4 8 6 5
33514 018/300 Read block 1364 len 36864
33515 019/300 Read block 1964 len 12288
33516 020/300 Read block 3035 len 4096
33517 021/300 Read block 4139 len 4096
33518 022/300 Flush
33519 023/300 Write block 272 len 40960 data: 5 5 6 7 6 7 9 9 10 8
33520 024/300 Read block 3920 len 8192
33521 025/300 Read block 1636 len 36864
33522 026/300 Read block 1614 len 16384
33523 027/300 Write block 1089 len 24576 data: 3 4 3 3 3 4
33524 028/300 Read block 209 len 20480
33525 029/300 Write block 2226 len 12288 data: 7 6 5
33526 030/300 Write block 1794 len 24576 data: 4 4 4 4 4 5
33527 031/300 Read block 1564 len 8192
33528 032/300 Read block 3485 len 36864
33529 033/300 Read block 873 len 8192
33530 034/300 Write block 1770 len 4096 data: 5
33531 035/300 Read block 4429 len 16384
33532 036/300 Flush
33533 037/300 Write block 322 len 4096 data: 5
33534 038/300 Write block 912 len 36864 data: 5 5 4 4 5 4 4 4 6
33535 039/300 Write block 4115 len 20480 data: 3 5 5 4 5
33536 040/300 Flush
33537 041/300 Read block 3339 len 4096
33538 042/300 Flush
33539 043/300 Read block 3938 len 40960
33540 044/300 Read block 882 len 28672
33541 045/300 Read block 2068 len 40960
33542 046/300 Flush
33543 047/300 Read block 2773 len 40960
33544 048/300 Read block 1874 len 12288
33545 049/300 Read block 4792 len 16384
33546 050/300 Read block 1554 len 16384
33547 051/300 Write block 3869 len 32768 data: 5 5 6 4 4 4 4 5
33548 052/300 Read block 4263 len 40960
33549 053/300 Read block 3631 len 20480
33550 054/300 Read block 189 len 4096
33551 055/300 Read block 4142 len 36864
33552 056/300 Write block 3872 len 32768 data: 5 5 5 5 6 7 4 4
33553 057/300 Read block 627 len 12288
33554 058/300 Read block 4518 len 36864
33555 059/300 Flush
33556 060/300 Write block 1884 len 32768 data: 4 4 4 4 4 5 5 6
33557 061/300 Read block 2439 len 12288
33558 062/300 Write block 42 len 20480 data: 7 6 6 7 5
33559 063/300 Read block 11 len 24576
33560 064/300 Read block 3059 len 8192
33561 065/300 Read block 693 len 36864
33562 066/300 Read block 1603 len 4096
33563 067/300 Read block 2242 len 32768
33564 068/300 Write block 3472 len 24576 data: 4 3 5 6 7 8
33565 069/300 Read block 933 len 36864
33566 070/300 Flush
33567 071/300 Write block 33 len 40960 data: 4 5 5 6 5 6 9 7 7 8
33568 072/300 Read block 77 len 24576
33569 073/300 Write block 3532 len 36864 data: 5 6 5 6 5 4 5 6 6
33570 074/300 Write block 4810 len 12288 data: 5 5 6
33571 075/300 Write block 3046 len 12288 data: 6 5 6
33572 076/300 Read block 456 len 36864
33573 077/300 Read block 1352 len 24576
33574 078/300 Write block 322 len 8192 data: 6 5
33575 079/300 Write block 1666 len 20480 data: 4 4 4 3 2
33576 080/300 Flush
33577 081/300 Read block 4265 len 28672
33578 082/300 Write block 4939 len 4096 data: 7
33579 083/300 Read block 71 len 28672
33580 084/300 Write block 4041 len 16384 data: 7 8 8 8
33581 085/300 Read block 3727 len 32768
33582 086/300 Write block 128 len 16384 data: 4 4 3 4
33583 087/300 Read block 3129 len 12288
33584 088/300 Read block 4852 len 32768
33585 089/300 Read block 2780 len 40960
33586 090/300 Write block 1293 len 12288 data: 6 6 4
33587 091/300 Read block 2877 len 28672
33588 092/300 Write block 1383 len 28672 data: 3 2 3 3 3 4 5
33589 093/300 Write block 371 len 20480 data: 8 7 6 6 5
33590 094/300 Flush
33591 095/300 Write block 1788 len 20480 data: 7 7 4 3 5
33592 096/300 Write block 369 len 40960 data: 8 7 9 8 7 7 6 2 2 3
33593 097/300 Write block 3688 len 8192 data: 8 7
33594 098/300 Read block 1400 len 12288
33595 099/300 Read block 2498 len 16384
33596 100/300 Write block 264 len 4096 data: 6
33597 101/300 Read block 2814 len 40960
33598 102/300 Read block 3245 len 32768
33599 103/300 Read block 3579 len 12288
33600 104/300 Flush
33601 105/300 Read block 1913 len 12288
33602 106/300 Write block 92 len 20480 data: 6 7 6 5 6
33603 107/300 Write block 1733 len 28672 data: 5 4 4 4 5 5 6
33604 108/300 Read block 4741 len 20480
33605 109/300 Read block 2515 len 4096
33606 110/300 Write block 1752 len 12288 data: 4 5 4
33607 111/300 Read block 2507 len 24576
33608 112/300 Write block 798 len 36864 data: 6 5 5 5 5 4 4 3 3
33609 113/300 Write block 4102 len 20480 data: 6 6 6 8 5
33610 114/300 Write block 286 len 36864 data: 4 6 5 6 5 4 4 3 3
33611 115/300 Read block 979 len 8192
33612 116/300 Write block 1006 len 32768 data: 8 6 6 5 7 8 6 7
33613 117/300 Write block 2274 len 24576 data: 9 9 8 7 7 7
33614 118/300 Read block 4757 len 40960
33615 119/300 Read block 986 len 12288
33616 120/300 Read block 1692 len 4096
33617 121/300 Read block 3365 len 12288
33618 122/300 Read block 482 len 12288
33619 123/300 Read block 2949 len 24576
33620 124/300 Write block 3888 len 24576 data: 7 6 5 3 4 4
33621 125/300 Write block 2820 len 8192 data: 6 5
33622 126/300 Read block 1408 len 40960
33623 127/300 Read block 4880 len 32768
33624 128/300 Write block 4372 len 4096 data: 6
33625 129/300 Read block 686 len 24576
33626 130/300 Write block 2038 len 24576 data: 8 7 6 5 5 4
33627 131/300 Read block 3363 len 16384
33628 132/300 Read block 636 len 16384
33629 133/300 Read block 2173 len 12288
33630 134/300 Write block 4591 len 24576 data: 7 7 7 7 5 5
33631 135/300 Read block 1977 len 40960
33632 136/300 Read block 4552 len 12288
33633 137/300 Write block 2745 len 24576 data: 9 9 9 7 4 4
33634 138/300 Write block 2302 len 28672 data: 6 6 5 5 5 6 7
33635 139/300 Write block 4443 len 32768 data: 4 3 2 2 2 2 3 5
33636 140/300 Read block 2260 len 40960
33637 141/300 Write block 4602 len 4096 data: 6
33638 142/300 Flush
33639 143/300 Read block 4005 len 16384
33640 144/300 Read block 2655 len 12288
33641 145/300 Flush
33642 146/300 Write block 319 len 24576 data: 8 7 5 7 6 5
33643 147/300 Flush
33644 148/300 Read block 3593 len 8192
33645 149/300 Read block 2103 len 8192
33646 150/300 Write block 1360 len 4096 data: 5
33647 151/300 Read block 542 len 4096
33648 152/300 Read block 3404 len 24576
33649 153/300 Write block 3874 len 12288 data: 6 6 7
33650 154/300 Write block 4640 len 40960 data: 8 9 9 9 7 7 7 8 6 8
33651 155/300 Flush
33652 156/300 Read block 110 len 16384
33653 157/300 Read block 564 len 40960
33654 158/300 Read block 4607 len 32768
33655 159/300 Read block 3639 len 28672
33656 160/300 Read block 4108 len 24576
33657 161/300 Read block 984 len 32768
33658 162/300 Read block 3164 len 16384
33659 163/300 Read block 1528 len 12288
33660 164/300 Read block 1883 len 40960
33661 165/300 Read block 2870 len 16384
33662 166/300 Flush
33663 167/300 Read block 2291 len 36864
33664 168/300 Flush
33665 169/300 Read block 2984 len 4096
33666 170/300 Read block 3426 len 8192
33667 171/300 Write block 4003 len 36864 data: 6 7 6 5 7 6 6 6 6
33668 172/300 Read block 3910 len 40960
33669 173/300 Write block 468 len 40960 data: 9 8 8 7 9 7 6 6 4 5
33670 174/300 Read block 1227 len 40960
33671 175/300 Read block 1052 len 12288
33672 176/300 Write block 3163 len 36864 data: 7 7 6 5 6 6 6 6 5
33673 177/300 Read block 2344 len 28672
33674 178/300 Read block 3835 len 8192
33675 179/300 Read block 95 len 36864
33676 180/300 Read block 2417 len 40960
33677 181/300 Read block 1537 len 40960
33678 182/300 Flush
33679 183/300 Write block 745 len 4096 data: 5
33680 184/300 Write block 2727 len 40960 data: 3 3 4 4 4 4 4 4 8 7
33681 185/300 Read block 3677 len 36864
33682 186/300 Write block 4404 len 16384 data: 5 6 6 6
33683 187/300 Write block 1226 len 36864 data: 8 9 7 8 8 7 5 5 5
33684 188/300 Write block 3130 len 8192 data: 6 6
33685 189/300 Read block 4506 len 28672
33686 190/300 Write block 1898 len 40960 data: 9 10 10 8 7 6 7 7 8 8
33687 191/300 Read block 521 len 40960
33688 192/300 Write block 3134 len 20480 data: 5 4 5 5 6
33689 193/300 Read block 4703 len 12288
33690 194/300 Read block 2023 len 4096
33691 195/300 Write block 2962 len 4096 data: 7
33692 196/300 Write block 1243 len 40960 data: 3 2 2 4 6 6 6 5 6 5
33693 197/300 Read block 3496 len 16384
33694 198/300 Write block 1219 len 36864 data: 7 8 7 8 11 11 9 9 10
33695 199/300 Flush
33696 200/300 Write block 3367 len 4096 data: 6
33697 201/300 Read block 842 len 12288
33698 202/300 Read block 4550 len 36864
33699 203/300 Read block 694 len 4096
33700 204/300 Write block 1338 len 28672 data: 6 5 5 7 6 7 9
33701 205/300 Read block 1487 len 24576
33702 206/300 Flush
33703 207/300 Write block 3546 len 4096 data: 3
33704 208/300 Flush
33705 209/300 Write block 1842 len 28672 data: 4 6 6 8 7 8 8
33706 210/300 Read block 2266 len 4096
33707 211/300 Write block 1839 len 4096 data: 7
33708 212/300 Write block 4403 len 32768 data: 4 6 7 7 7 7 6 6
33709 213/300 Read block 937 len 36864
33710 214/300 Write block 1746 len 32768 data: 7 7 6 6 4 4 5 6
33711 215/300 Read block 371 len 16384
33712 216/300 Read block 3308 len 20480
33713 217/300 Read block 1967 len 24576
33714 218/300 Write block 3641 len 20480 data: 3 3 5 5 4
33715 219/300 Read block 3548 len 16384
33716 220/300 Write block 2283 len 8192 data: 8 7
33717 221/300 Read block 1004 len 16384
33718 222/300 Read block 91 len 20480
33719 223/300 Write block 539 len 24576 data: 3 3 4 4 5 4
33720 224/300 Read block 1925 len 16384
33721 225/300 Flush
33722 226/300 Write block 4882 len 20480 data: 5 5 4 4 5
33723 227/300 Write block 4250 len 8192 data: 9 7
33724 228/300 Read block 2435 len 32768
33725 229/300 Read block 1066 len 40960
33726 230/300 Write block 4454 len 4096 data: 5
33727 231/300 Read block 4176 len 40960
33728 232/300 Flush
33729 233/300 Flush
33730 234/300 Write block 74 len 8192 data: 4 5
33731 235/300 Read block 4281 len 20480
33732 236/300 Read block 4778 len 32768
33733 237/300 Read block 3533 len 8192
33734 238/300 Flush
33735 239/300 Write block 2237 len 16384 data: 8 8 7 7
33736 240/300 Flush
33737 241/300 Write block 727 len 20480 data: 3 4 4 4 4
33738 242/300 Read block 1351 len 36864
33739 243/300 Read block 386 len 4096
33740 244/300 Write block 1529 len 20480 data: 4 3 3 2 3
33741 245/300 Write block 1580 len 36864 data: 8 8 7 6 6 6 6 5 4
33742 246/300 Read block 994 len 12288
33743 247/300 Read block 3384 len 32768
33744 248/300 Flush
33745 249/300 Write block 3041 len 12288 data: 5 5 5
33746 250/300 Read block 247 len 40960
33747 251/300 Read block 3315 len 16384
33748 252/300 Read block 1927 len 40960
33749 253/300 Read block 907 len 12288
33750 254/300 Read block 2559 len 32768
33751 255/300 Write block 1448 len 24576 data: 4 4 4 3 3 3
33752 256/300 Flush
33753 257/300 Write block 1138 len 36864 data: 4 4 4 2 2 3 2 3 3
33754 258/300 Read block 4070 len 12288
33755 259/300 Read block 3020 len 40960
33756 260/300 Read block 2832 len 36864
33757 261/300 Read block 2443 len 36864
33758 262/300 Read block 1141 len 4096
33759 263/300 Write block 410 len 28672 data: 6 5 6 5 3 3 4
33760 264/300 Read block 1136 len 32768
33761 265/300 Read block 1427 len 28672
33762 266/300 Flush
33763 267/300 Read block 496 len 24576
33764 268/300 Read block 1854 len 32768
33765 269/300 Read block 240 len 4096
33766 270/300 Read block 3404 len 12288
33767 271/300 Write block 3135 len 32768 data: 5 6 6 7 7 8 10 9
33768 272/300 Write block 4572 len 32768 data: 5 5 6 7 7 7 5 4
33769 273/300 Read block 1202 len 12288
33770 274/300 Read block 2205 len 28672
33771 275/300 Read block 2359 len 8192
33772 276/300 Read block 2805 len 4096
33773 277/300 Write block 4546 len 40960 data: 6 4 5 5 4 4 4 4 5 6
33774 278/300 Read block 3538 len 12288
33775 279/300 Write block 3579 len 24576 data: 5 5 5 5 5 6
33776 280/300 Read block 2801 len 4096
33777 281/300 Flush
33778 282/300 Read block 3611 len 24576
33779 283/300 Read block 984 len 24576
33780 284/300 Read block 1500 len 16384
33781 285/300 Read block 4131 len 28672
33782 286/300 Read block 2160 len 12288
33783 287/300 Read block 2648 len 4096
33784 288/300 Write block 4703 len 36864 data: 5 5 8 8 8 9 9 8 7
33785 289/300 Write block 909 len 20480 data: 4 5 4 6 6
33786 290/300 Write block 4055 len 32768 data: 5 4 4 4 7 10 13 11
33787 291/300 Read block 1550 len 20480
33788 292/300 Read block 377 len 8192
33789 293/300 Write block 2179 len 24576 data: 4 4 6 5 5 4
33790 294/300 Write block 1146 len 16384 data: 4 3 3 4
33791 295/300 Write block 1033 len 24576 data: 3 5 5 5 7 6
33792 296/300 Read block 971 len 4096
33793 297/300 Write block 2031 len 24576 data: 4 4 4 5 7 8
33794 298/300 Read block 1303 len 8192
33795 299/300 Flush
33796 300/300 Read block 743 len 40960
33797 301/300 Read block 936 len 8192
33798 Main task has work to do, go find it
33799 got dsc Start(1)
33800 start 1
33801 Feb 07 20:14:48.838 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/1, method: POST, req_id: 788ecef1-99ed-4c61-a399-57517663e7d6, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
33802 Replay: started 1, returned:Ok(())
33803 ----------------------------------------------------------------
33804 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33805 States: Active Offline Active
33806 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33807 9455 Acked Write 5 Done Sent Done false
33808 9456 Acked Flush 0 Done Sent Done false
33809 9457 Acked Read 4 Done Sent Done false
33810 9458 Acked Read 3 Done Sent Done false
33811 9459 Acked Write 7 Done Sent Done false
33812 9460 Acked Read 3 Done Sent Done false
33813 9461 Acked Flush 0 Done Sent Done false
33814 9462 Acked Read 5 Done Sent Done false
33815 9463 Acked Read 10 Done Sent Done false
33816 9464 Acked Flush 0 Done Sent Done false
33817 9465 Acked Write 8 Done Sent Done false
33818 9466 Acked Read 5 Done Sent Done false
33819 9467 Acked Write 5 Done Sent Done false
33820 9468 Acked Write 2 Done Sent Done false
33821 9469 Acked Write 2 Done Sent Done false
33822 9470 Acked Write 2 Done Sent Done false
33823 9471 Acked Write 5 Done Sent Done false
33824 9472 Acked Read 9 Done Sent Done false
33825 9473 Acked Read 3 Done Sent Done false
33826 9474 Acked Read 1 Done Sent Done false
33827 9475 Acked Read 1 Done Sent Done false
33828 9476 Acked Flush 0 Done Sent Done false
33829 9477 Acked Write 10 Done Sent Done false
33830 9478 Acked Read 2 Done Sent Done false
33831 9479 Acked Read 9 Done Sent Done false
33832 9480 Acked Read 4 Done Sent Done false
33833 9481 Acked Write 6 Done Sent Done false
33834 9482 Acked Read 5 Done Sent Done false
33835 9483 Acked Write 3 Done Sent Done false
33836 9484 Acked Write 6 Done Sent Done false
33837 9485 Acked Read 2 Done Sent Done false
33838 9486 Acked Read 9 Done Sent Done false
33839 9487 Acked Read 2 Done Sent Done false
33840 9488 Acked Write 1 Done Sent Done false
33841 9489 Acked Read 4 Done Sent Done false
33842 9490 Acked Flush 0 Done Sent Done false
33843 9491 Acked Write 1 Done Sent Done false
33844 9492 Acked Write 9 Done Sent Done false
33845 9493 Acked Write 5 Done Sent Done false
33846 9494 Acked Flush 0 Done Sent Done false
33847 9495 Acked Read 1 Done Sent Done false
33848 9496 Acked Flush 0 Done Sent Done false
33849 9497 Acked Read 10 Done Sent Done false
33850 9498 Acked Read 7 Done Sent Done false
33851 9499 Acked Read 10 Done Sent Done false
33852 9500 Acked Flush 0 Done Sent Done false
33853 9501 Acked Read 10 Done Sent Done false
33854 9502 Acked Read 3 Done Sent Done false
33855 9503 Acked Read 4 Done Sent Done false
33856 9504 Acked Read 4 Done Sent Done false
33857 9505 Acked Write 8 Done Sent Done false
33858 9506 Acked Read 10 Done Sent Done false
33859 9507 Acked Read 5 Done Sent Done false
33860 9508 Acked Read 1 Done Sent Done false
33861 9509 Acked Read 9 Done Sent Done false
33862 9510 Acked Write 8 Done Sent Done false
33863 9511 Acked Read 3 Done Sent Done false
33864 9512 Acked Read 9 Done Sent Done false
33865 9513 Acked Flush 0 Done Sent Done false
33866 9514 Acked Write 8 Done Sent Done false
33867 9515 Acked Read 3 Done Sent Done false
33868 9516 Acked Write 5 Done Sent Done false
33869 9517 Acked Read 6 Done Sent Done false
33870 9518 Acked Read 2 Done Sent Done false
33871 9519 Acked Read 9 Done Sent Done false
33872 9520 Acked Read 1 Done Sent Done false
33873 9521 Acked Read 8 Done Sent Done false
33874 9522 Acked Write 6 Done Sent Done false
33875 9523 Acked Read 9 Done Sent Done false
33876 9524 Acked Flush 0 Done Sent Done false
33877 9525 Acked Write 10 Done Sent Done false
33878 9526 Acked Read 6 Done Sent Done false
33879 9527 Acked Write 9 Done Sent Done false
33880 9528 Acked Write 3 Done Sent Done false
33881 9529 Acked Write 3 Done Sent Done false
33882 9530 Acked Read 9 Done Sent Done false
33883 9531 Acked Read 6 Done Sent Done false
33884 9532 Acked Write 2 Done Sent Done false
33885 9533 Acked Write 5 Done Sent Done false
33886 9534 Acked Flush 0 Done Sent Done false
33887 9535 Acked Read 7 Done Sent Done false
33888 9536 Acked Write 1 Done Sent Done false
33889 9537 Acked Read 7 Done Sent Done false
33890 9538 Acked Write 4 Done Sent Done false
33891 9539 Acked Read 8 Done Sent Done false
33892 9540 Acked Write 4 Done Sent Done false
33893 9541 Acked Read 3 Done Sent Done false
33894 9542 Acked Read 8 Done Sent Done false
33895 9543 Acked Read 10 Done Sent Done false
33896 9544 Acked Write 3 Done Sent Done false
33897 9545 Acked Read 7 Done Sent Done false
33898 9546 Acked Write 7 Done Sent Done false
33899 9547 Acked Write 5 Done Sent Done false
33900 9548 Acked Flush 0 Done Sent Done false
33901 9549 Acked Write 5 Done Sent Done false
33902 9550 Acked Write 10 Done Sent Done false
33903 9551 Acked Write 2 Done Sent Done false
33904 9552 Acked Read 3 Done Sent Done false
33905 9553 Acked Read 4 Done Sent Done false
33906 9554 Acked Write 1 Done Sent Done false
33907 9555 Acked Read 10 Done Sent Done false
33908 9556 Acked Read 8 Done Sent Done false
33909 9557 Acked Read 3 Done Sent Done false
33910 9558 Acked Flush 0 Done Sent Done false
33911 9559 Acked Read 3 Done Sent Done false
33912 9560 Acked Write 5 Done Sent Done false
33913 9561 Acked Write 7 Done Sent Done false
33914 9562 Acked Read 5 Done Sent Done false
33915 9563 Acked Read 1 Done Sent Done false
33916 9564 Acked Write 3 Done Sent Done false
33917 9565 Acked Read 6 Done Sent Done false
33918 9566 Acked Write 9 Done Sent Done false
33919 9567 Acked Write 5 Done Sent Done false
33920 9568 Acked Write 9 Done Sent Done false
33921 9569 Acked Read 2 Done Sent Done false
33922 9570 Acked Write 8 Done Sent Done false
33923 9571 Acked Write 6 Done Sent Done false
33924 9572 Acked Read 10 Done Sent Done false
33925 9573 Acked Read 3 Done Sent Done false
33926 9574 Acked Read 1 Done Sent Done false
33927 9575 Acked Read 3 Done Sent Done false
33928 9576 Acked Read 3 Done Sent Done false
33929 9577 Acked Read 6 Done Sent Done false
33930 9578 Acked Write 6 Done Sent Done false
33931 9579 Acked Write 2 Done Sent Done false
33932 9580 Acked Read 10 Done Sent Done false
33933 9581 Acked Read 8 Done Sent Done false
33934 9582 Acked Write 1 Done Sent Done false
33935 9583 Acked Read 6 Done Sent Done false
33936 9584 Acked Write 6 Done Sent Done false
33937 9585 Acked Read 4 Done Sent Done false
33938 9586 Acked Read 4 Done Sent Done false
33939 9587 Acked Read 3 Done Sent Done false
33940 9588 Acked Write 6 Done Sent Done false
33941 9589 Acked Read 10 Done Sent Done false
33942 9590 Acked Read 3 Done Sent Done false
33943 9591 Acked Write 6 Done Sent Done false
33944 9592 Acked Write 7 Done Sent Done false
33945 9593 Acked Write 8 Done Sent Done false
33946 9594 Acked Read 10 Done Sent Done false
33947 9595 Acked Write 1 Done Sent Done false
33948 9596 Acked Flush 0 Done Sent Done false
33949 9597 Acked Read 4 Done Sent Done false
33950 9598 Acked Read 3 Done Sent Done false
33951 9599 Acked Flush 0 Done Sent Done false
33952 9600 Acked Write 6 Done Sent Done false
33953 9601 Acked Flush 0 Done Sent Done false
33954 9602 Acked Read 2 Done Sent Done false
33955 9603 Acked Read 2 Done Sent Done false
33956 9604 Acked Write 1 Done Sent Done false
33957 9605 Acked Read 1 Done Sent Done false
33958 9606 Acked Read 6 Done Sent Done false
33959 9607 Acked Write 3 Done Sent Done false
33960 9608 Acked Write 10 Done Sent Done false
33961 9609 Acked Flush 0 Done Sent Done false
33962 9610 Acked Read 4 Done Sent Done false
33963 9611 Acked Read 10 Done Sent Done false
33964 9612 Acked Read 8 Done Sent Done false
33965 9613 Acked Read 7 Done Sent Done false
33966 9614 Acked Read 6 Done Sent Done false
33967 9615 Acked Read 8 Done Sent Done false
33968 9616 Acked Read 4 Done Sent Done false
33969 9617 Acked Read 3 Done Sent Done false
33970 9618 Acked Read 10 Done Sent Done false
33971 9619 Acked Read 4 Done Sent Done false
33972 9620 Acked Flush 0 Done Sent Done false
33973 9621 Acked Read 9 Done Sent Done false
33974 9622 Acked Flush 0 Done Sent Done false
33975 9623 Acked Read 1 Done Sent Done false
33976 9624 Acked Read 2 Done Sent Done false
33977 9625 Acked Write 9 Done Sent Done false
33978 9626 Acked Read 10 Done Sent Done false
33979 9627 Acked Write 10 Done Sent Done false
33980 9628 Acked Read 10 Done Sent Done false
33981 9629 Acked Read 3 Done Sent Done false
33982 9630 Acked Write 9 Done Sent Done false
33983 9631 Acked Read 7 Done Sent Done false
33984 9632 Acked Read 2 Done Sent Done false
33985 9633 Acked Read 9 Done Sent Done false
33986 9634 Acked Read 10 Done Sent Done false
33987 9635 Acked Read 10 Done Sent Done false
33988 9636 Acked Flush 0 Done Sent Done false
33989 9637 Acked Write 1 Done Sent Done false
33990 9638 Acked Write 10 Done Sent Done false
33991 9639 Acked Read 9 Done Sent Done false
33992 9640 Acked Write 4 Done Sent Done false
33993 9641 Acked Write 9 Done Sent Done false
33994 9642 Acked Write 2 Done Sent Done false
33995 9643 Acked Read 7 Done Sent Done false
33996 9644 Acked Write 10 Done Sent Done false
33997 9645 Acked Read 10 Done Sent Done false
33998 9646 Acked Write 5 Done Sent Done false
33999 9647 Acked Read 3 Done Sent Done false
34000 9648 Acked Read 1 Done Sent Done false
34001 9649 Acked Write 1 Done Sent Done false
34002 9650 Acked Write 10 Done Sent Done false
34003 9651 Acked Read 4 Done Sent Done false
34004 9652 Acked Write 9 Done Sent Done false
34005 9653 Acked Flush 0 Done Sent Done false
34006 9654 Acked Write 1 Done Sent Done false
34007 9655 Acked Read 3 Done Sent Done false
34008 9656 Acked Read 9 Done Sent Done false
34009 9657 Acked Read 1 Done Sent Done false
34010 9658 Acked Write 7 Done Sent Done false
34011 9659 Acked Read 6 Done Sent Done false
34012 9660 Acked Flush 0 Done Sent Done false
34013 9661 Acked Write 1 Done Sent Done false
34014 9662 Acked Flush 0 Done Sent Done false
34015 9663 Acked Write 7 Done Sent Done false
34016 9664 Acked Read 1 Done Sent Done false
34017 9665 Acked Write 1 Done Sent Done false
34018 9666 Acked Write 8 Done Sent Done false
34019 9667 Acked Read 9 Done Sent Done false
34020 9668 Acked Write 8 Done Sent Done false
34021 9669 Acked Read 4 Done Sent Done false
34022 9670 Acked Read 5 Done Sent Done false
34023 9671 Acked Read 6 Done Sent Done false
34024 9672 Acked Write 5 Done Sent Done false
34025 9673 Acked Read 4 Done Sent Done false
34026 9674 Acked Write 2 Done Sent Done false
34027 9675 Acked Read 4 Done Sent Done false
34028 9676 Acked Read 5 Done Sent Done false
34029 9677 Acked Write 6 Done Sent Done false
34030 9678 Acked Read 4 Done Sent Done false
34031 9679 Acked Flush 0 Done Sent Done false
34032 9680 Acked Write 5 Done Sent Done false
34033 9681 Acked Write 2 Done Sent Done false
34034 9682 Acked Read 8 Done Sent Done false
34035 9683 Acked Read 10 Done Sent Done false
34036 9684 Acked Write 1 Done Sent Done false
34037 9685 Acked Read 10 Done Sent Done false
34038 9686 Acked Flush 0 Done Sent Done false
34039 9687 Acked Flush 0 Done Sent Done false
34040 9688 Acked Write 2 Done Sent Done false
34041 9689 Acked Read 5 Done Sent Done false
34042 9690 Acked Read 8 Done Sent Done false
34043 9691 Acked Read 2 Done Sent Done false
34044 9692 Acked Flush 0 Done Sent Done false
34045 9693 Acked Write 4 Done Sent Done false
34046 9694 Acked Flush 0 Done Sent Done false
34047 9695 Acked Write 5 Done Sent Done false
34048 9696 Acked Read 9 Done Sent Done false
34049 9697 Acked Read 1 Done Sent Done false
34050 9698 Acked Write 5 Done Sent Done false
34051 9699 Acked Write 9 Done Sent Done false
34052 9700 Acked Read 3 Done Sent Done false
34053 9701 Acked Read 8 Done Sent Done false
34054 9702 Acked Flush 0 Done Sent Done false
34055 9703 Acked Write 3 Done Sent Done false
34056 9704 Acked Read 10 Done Sent Done false
34057 9705 Acked Read 4 Done Sent Done false
34058 9706 Acked Read 10 Done Sent Done false
34059 9707 Acked Read 3 Done Sent Done false
34060 9708 Acked Read 8 Done Sent Done false
34061 9709 Acked Write 6 Done Sent Done false
34062 9710 Acked Flush 0 Done Sent Done false
34063 9711 Acked Write 9 Done Sent Done false
34064 9712 Acked Read 3 Done Sent Done false
34065 9713 Acked Read 10 Done Sent Done false
34066 9714 Acked Read 9 Done Sent Done false
34067 9715 Acked Read 9 Done Sent Done false
34068 9716 Acked Read 1 Done Sent Done false
34069 9717 Acked Write 7 Done Sent Done false
34070 9718 Acked Read 8 Done Sent Done false
34071 9719 Acked Read 7 Done Sent Done false
34072 9720 Acked Flush 0 Done Sent Done false
34073 9721 Acked Read 6 Done Sent Done false
34074 9722 Acked Read 8 Done Sent Done false
34075 9723 Acked Read 1 Done Sent Done false
34076 9724 Acked Read 3 Done Sent Done false
34077 9725 Acked Write 8 Done Sent Done false
34078 9726 Acked Write 8 Done Sent Done false
34079 9727 Acked Read 3 Done Sent Done false
34080 9728 Acked Read 7 Done Sent Done false
34081 9729 Acked Read 2 Done Sent Done false
34082 9730 Acked Read 1 Done Sent Done false
34083 9731 Acked Write 10 Done Sent Done false
34084 9732 Acked Read 3 Done Sent Done false
34085 9733 Acked Write 6 Done Sent Done false
34086 9734 Acked Read 1 Done Sent Done false
34087 9735 Acked Flush 0 Done Sent Done false
34088 9736 Acked Read 6 Done Sent Done false
34089 9737 Acked Read 6 Done Sent Done false
34090 9738 Acked Read 4 Done Sent Done false
34091 9739 Acked Read 7 Done Sent Done false
34092 9740 Acked Read 3 Done Sent Done false
34093 9741 Acked Read 1 Done Sent Done false
34094 9742 Acked Write 9 Done Sent Done false
34095 9743 Acked Write 5 Done Sent Done false
34096 9744 Acked Write 8 Done Sent Done false
34097 9745 Acked Read 5 Done Sent Done false
34098 9746 Acked Read 2 Done Sent Done false
34099 9747 Acked Write 6 Done Sent Done false
34100 9748 Acked Write 4 Done Sent Done false
34101 9749 Acked Write 6 Done Sent Done false
34102 9750 Acked Read 1 Done Sent Done false
34103 9751 Acked Write 6 Done Sent Done false
34104 9752 Acked Read 2 Done Sent Done false
34105 9753 Acked Flush 0 Done Sent Done false
34106 9754 Acked Read 10 Done Sent Done false
34107 9755 Acked Read 2 Done Sent Done false
34108 STATES DS:0 DS:1 DS:2 TOTAL
34109 Sent 0 301 0 301
34110 Done 301 0 301 602
34111 Skipped 0 0 0 0
34112 Error 0 0 0 0
34113 Last Flush: Some(JobId(9753)) Some(JobId(9454)) Some(JobId(9753))
34114 Downstairs last five completed: 9454 9453 9452 9451 9450
34115 Upstairs last five completed: 9755 9754 9753 9752 9751
34116 CLIENT: Up:0 ds:301 act:2
341172025-02-07T20:14:49.674ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
34118 [8820] got start action
34119 [8820] I am going to restart
34120 Starting downstairs at port 8820
34121 Make output file at "/tmp/dsc/downstairs-8820.txt"
34122 [8820][1] reports Starting
34123 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1238)
34124 [8820][1] reports Running
341252025-02-07T20:14:50.676ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
341262025-02-07T20:14:51.676ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
341272025-02-07T20:14:52.832ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
34128 ----------------------------------------------------------------
34129 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34130 States: Active Offline Active
34131 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34132 9455 Acked Write 5 Done Sent Done false
34133 9456 Acked Flush 0 Done Sent Done false
34134 9457 Acked Read 4 Done Sent Done false
34135 9458 Acked Read 3 Done Sent Done false
34136 9459 Acked Write 7 Done Sent Done false
34137 9460 Acked Read 3 Done Sent Done false
34138 9461 Acked Flush 0 Done Sent Done false
34139 9462 Acked Read 5 Done Sent Done false
34140 9463 Acked Read 10 Done Sent Done false
34141 9464 Acked Flush 0 Done Sent Done false
34142 9465 Acked Write 8 Done Sent Done false
34143 9466 Acked Read 5 Done Sent Done false
34144 9467 Acked Write 5 Done Sent Done false
34145 9468 Acked Write 2 Done Sent Done false
34146 9469 Acked Write 2 Done Sent Done false
34147 9470 Acked Write 2 Done Sent Done false
34148 9471 Acked Write 5 Done Sent Done false
34149 9472 Acked Read 9 Done Sent Done false
34150 9473 Acked Read 3 Done Sent Done false
34151 9474 Acked Read 1 Done Sent Done false
34152 9475 Acked Read 1 Done Sent Done false
34153 9476 Acked Flush 0 Done Sent Done false
34154 9477 Acked Write 10 Done Sent Done false
34155 9478 Acked Read 2 Done Sent Done false
34156 9479 Acked Read 9 Done Sent Done false
34157 9480 Acked Read 4 Done Sent Done false
34158 9481 Acked Write 6 Done Sent Done false
34159 9482 Acked Read 5 Done Sent Done false
34160 9483 Acked Write 3 Done Sent Done false
34161 9484 Acked Write 6 Done Sent Done false
34162 9485 Acked Read 2 Done Sent Done false
34163 9486 Acked Read 9 Done Sent Done false
34164 9487 Acked Read 2 Done Sent Done false
34165 9488 Acked Write 1 Done Sent Done false
34166 9489 Acked Read 4 Done Sent Done false
34167 9490 Acked Flush 0 Done Sent Done false
34168 9491 Acked Write 1 Done Sent Done false
34169 9492 Acked Write 9 Done Sent Done false
34170 9493 Acked Write 5 Done Sent Done false
34171 9494 Acked Flush 0 Done Sent Done false
34172 9495 Acked Read 1 Done Sent Done false
34173 9496 Acked Flush 0 Done Sent Done false
34174 9497 Acked Read 10 Done Sent Done false
34175 9498 Acked Read 7 Done Sent Done false
34176 9499 Acked Read 10 Done Sent Done false
34177 9500 Acked Flush 0 Done Sent Done false
34178 9501 Acked Read 10 Done Sent Done false
34179 9502 Acked Read 3 Done Sent Done false
34180 9503 Acked Read 4 Done Sent Done false
34181 9504 Acked Read 4 Done Sent Done false
34182 9505 Acked Write 8 Done Sent Done false
34183 9506 Acked Read 10 Done Sent Done false
34184 9507 Acked Read 5 Done Sent Done false
34185 9508 Acked Read 1 Done Sent Done false
34186 9509 Acked Read 9 Done Sent Done false
34187 9510 Acked Write 8 Done Sent Done false
34188 9511 Acked Read 3 Done Sent Done false
34189 9512 Acked Read 9 Done Sent Done false
34190 9513 Acked Flush 0 Done Sent Done false
34191 9514 Acked Write 8 Done Sent Done false
34192 9515 Acked Read 3 Done Sent Done false
34193 9516 Acked Write 5 Done Sent Done false
34194 9517 Acked Read 6 Done Sent Done false
34195 9518 Acked Read 2 Done Sent Done false
34196 9519 Acked Read 9 Done Sent Done false
34197 9520 Acked Read 1 Done Sent Done false
34198 9521 Acked Read 8 Done Sent Done false
34199 9522 Acked Write 6 Done Sent Done false
34200 9523 Acked Read 9 Done Sent Done false
34201 9524 Acked Flush 0 Done Sent Done false
34202 9525 Acked Write 10 Done Sent Done false
34203 9526 Acked Read 6 Done Sent Done false
34204 9527 Acked Write 9 Done Sent Done false
34205 9528 Acked Write 3 Done Sent Done false
34206 9529 Acked Write 3 Done Sent Done false
34207 9530 Acked Read 9 Done Sent Done false
34208 9531 Acked Read 6 Done Sent Done false
34209 9532 Acked Write 2 Done Sent Done false
34210 9533 Acked Write 5 Done Sent Done false
34211 9534 Acked Flush 0 Done Sent Done false
34212 9535 Acked Read 7 Done Sent Done false
34213 9536 Acked Write 1 Done Sent Done false
34214 9537 Acked Read 7 Done Sent Done false
34215 9538 Acked Write 4 Done Sent Done false
34216 9539 Acked Read 8 Done Sent Done false
34217 9540 Acked Write 4 Done Sent Done false
34218 9541 Acked Read 3 Done Sent Done false
34219 9542 Acked Read 8 Done Sent Done false
34220 9543 Acked Read 10 Done Sent Done false
34221 9544 Acked Write 3 Done Sent Done false
34222 9545 Acked Read 7 Done Sent Done false
34223 9546 Acked Write 7 Done Sent Done false
34224 9547 Acked Write 5 Done Sent Done false
34225 9548 Acked Flush 0 Done Sent Done false
34226 9549 Acked Write 5 Done Sent Done false
34227 9550 Acked Write 10 Done Sent Done false
34228 9551 Acked Write 2 Done Sent Done false
34229 9552 Acked Read 3 Done Sent Done false
34230 9553 Acked Read 4 Done Sent Done false
34231 9554 Acked Write 1 Done Sent Done false
34232 9555 Acked Read 10 Done Sent Done false
34233 9556 Acked Read 8 Done Sent Done false
34234 9557 Acked Read 3 Done Sent Done false
34235 9558 Acked Flush 0 Done Sent Done false
34236 9559 Acked Read 3 Done Sent Done false
34237 9560 Acked Write 5 Done Sent Done false
34238 9561 Acked Write 7 Done Sent Done false
34239 9562 Acked Read 5 Done Sent Done false
34240 9563 Acked Read 1 Done Sent Done false
34241 9564 Acked Write 3 Done Sent Done false
34242 9565 Acked Read 6 Done Sent Done false
34243 9566 Acked Write 9 Done Sent Done false
34244 9567 Acked Write 5 Done Sent Done false
34245 9568 Acked Write 9 Done Sent Done false
34246 9569 Acked Read 2 Done Sent Done false
34247 9570 Acked Write 8 Done Sent Done false
34248 9571 Acked Write 6 Done Sent Done false
34249 9572 Acked Read 10 Done Sent Done false
34250 9573 Acked Read 3 Done Sent Done false
34251 9574 Acked Read 1 Done Sent Done false
34252 9575 Acked Read 3 Done Sent Done false
34253 9576 Acked Read 3 Done Sent Done false
34254 9577 Acked Read 6 Done Sent Done false
34255 9578 Acked Write 6 Done Sent Done false
34256 9579 Acked Write 2 Done Sent Done false
34257 9580 Acked Read 10 Done Sent Done false
34258 9581 Acked Read 8 Done Sent Done false
34259 9582 Acked Write 1 Done Sent Done false
34260 9583 Acked Read 6 Done Sent Done false
34261 9584 Acked Write 6 Done Sent Done false
34262 9585 Acked Read 4 Done Sent Done false
34263 9586 Acked Read 4 Done Sent Done false
34264 9587 Acked Read 3 Done Sent Done false
34265 9588 Acked Write 6 Done Sent Done false
34266 9589 Acked Read 10 Done Sent Done false
34267 9590 Acked Read 3 Done Sent Done false
34268 9591 Acked Write 6 Done Sent Done false
34269 9592 Acked Write 7 Done Sent Done false
34270 9593 Acked Write 8 Done Sent Done false
34271 9594 Acked Read 10 Done Sent Done false
34272 9595 Acked Write 1 Done Sent Done false
34273 9596 Acked Flush 0 Done Sent Done false
34274 9597 Acked Read 4 Done Sent Done false
34275 9598 Acked Read 3 Done Sent Done false
34276 9599 Acked Flush 0 Done Sent Done false
34277 9600 Acked Write 6 Done Sent Done false
34278 9601 Acked Flush 0 Done Sent Done false
34279 9602 Acked Read 2 Done Sent Done false
34280 9603 Acked Read 2 Done Sent Done false
34281 9604 Acked Write 1 Done Sent Done false
34282 9605 Acked Read 1 Done Sent Done false
34283 9606 Acked Read 6 Done Sent Done false
34284 9607 Acked Write 3 Done Sent Done false
34285 9608 Acked Write 10 Done Sent Done false
34286 9609 Acked Flush 0 Done Sent Done false
34287 9610 Acked Read 4 Done Sent Done false
34288 9611 Acked Read 10 Done Sent Done false
34289 9612 Acked Read 8 Done Sent Done false
34290 9613 Acked Read 7 Done Sent Done false
34291 9614 Acked Read 6 Done Sent Done false
34292 9615 Acked Read 8 Done Sent Done false
34293 9616 Acked Read 4 Done Sent Done false
34294 9617 Acked Read 3 Done Sent Done false
34295 9618 Acked Read 10 Done Sent Done false
34296 9619 Acked Read 4 Done Sent Done false
34297 9620 Acked Flush 0 Done Sent Done false
34298 9621 Acked Read 9 Done Sent Done false
34299 9622 Acked Flush 0 Done Sent Done false
34300 9623 Acked Read 1 Done Sent Done false
34301 9624 Acked Read 2 Done Sent Done false
34302 9625 Acked Write 9 Done Sent Done false
34303 9626 Acked Read 10 Done Sent Done false
34304 9627 Acked Write 10 Done Sent Done false
34305 9628 Acked Read 10 Done Sent Done false
34306 9629 Acked Read 3 Done Sent Done false
34307 9630 Acked Write 9 Done Sent Done false
34308 9631 Acked Read 7 Done Sent Done false
34309 9632 Acked Read 2 Done Sent Done false
34310 9633 Acked Read 9 Done Sent Done false
34311 9634 Acked Read 10 Done Sent Done false
34312 9635 Acked Read 10 Done Sent Done false
34313 9636 Acked Flush 0 Done Sent Done false
34314 9637 Acked Write 1 Done Sent Done false
34315 9638 Acked Write 10 Done Sent Done false
34316 9639 Acked Read 9 Done Sent Done false
34317 9640 Acked Write 4 Done Sent Done false
34318 9641 Acked Write 9 Done Sent Done false
34319 9642 Acked Write 2 Done Sent Done false
34320 9643 Acked Read 7 Done Sent Done false
34321 9644 Acked Write 10 Done Sent Done false
34322 9645 Acked Read 10 Done Sent Done false
34323 9646 Acked Write 5 Done Sent Done false
34324 9647 Acked Read 3 Done Sent Done false
34325 9648 Acked Read 1 Done Sent Done false
34326 9649 Acked Write 1 Done Sent Done false
34327 9650 Acked Write 10 Done Sent Done false
34328 9651 Acked Read 4 Done Sent Done false
34329 9652 Acked Write 9 Done Sent Done false
34330 9653 Acked Flush 0 Done Sent Done false
34331 9654 Acked Write 1 Done Sent Done false
34332 9655 Acked Read 3 Done Sent Done false
34333 9656 Acked Read 9 Done Sent Done false
34334 9657 Acked Read 1 Done Sent Done false
34335 9658 Acked Write 7 Done Sent Done false
34336 9659 Acked Read 6 Done Sent Done false
34337 9660 Acked Flush 0 Done Sent Done false
34338 9661 Acked Write 1 Done Sent Done false
34339 9662 Acked Flush 0 Done Sent Done false
34340 9663 Acked Write 7 Done Sent Done false
34341 9664 Acked Read 1 Done Sent Done false
34342 9665 Acked Write 1 Done Sent Done false
34343 9666 Acked Write 8 Done Sent Done false
34344 9667 Acked Read 9 Done Sent Done false
34345 9668 Acked Write 8 Done Sent Done false
34346 9669 Acked Read 4 Done Sent Done false
34347 9670 Acked Read 5 Done Sent Done false
34348 9671 Acked Read 6 Done Sent Done false
34349 9672 Acked Write 5 Done Sent Done false
34350 9673 Acked Read 4 Done Sent Done false
34351 9674 Acked Write 2 Done Sent Done false
34352 9675 Acked Read 4 Done Sent Done false
34353 9676 Acked Read 5 Done Sent Done false
34354 9677 Acked Write 6 Done Sent Done false
34355 9678 Acked Read 4 Done Sent Done false
34356 9679 Acked Flush 0 Done Sent Done false
34357 9680 Acked Write 5 Done Sent Done false
34358 9681 Acked Write 2 Done Sent Done false
34359 9682 Acked Read 8 Done Sent Done false
34360 9683 Acked Read 10 Done Sent Done false
34361 9684 Acked Write 1 Done Sent Done false
34362 9685 Acked Read 10 Done Sent Done false
34363 9686 Acked Flush 0 Done Sent Done false
34364 9687 Acked Flush 0 Done Sent Done false
34365 9688 Acked Write 2 Done Sent Done false
34366 9689 Acked Read 5 Done Sent Done false
34367 9690 Acked Read 8 Done Sent Done false
34368 9691 Acked Read 2 Done Sent Done false
34369 9692 Acked Flush 0 Done Sent Done false
34370 9693 Acked Write 4 Done Sent Done false
34371 9694 Acked Flush 0 Done Sent Done false
34372 9695 Acked Write 5 Done Sent Done false
34373 9696 Acked Read 9 Done Sent Done false
34374 9697 Acked Read 1 Done Sent Done false
34375 9698 Acked Write 5 Done Sent Done false
34376 9699 Acked Write 9 Done Sent Done false
34377 9700 Acked Read 3 Done Sent Done false
34378 9701 Acked Read 8 Done Sent Done false
34379 9702 Acked Flush 0 Done Sent Done false
34380 9703 Acked Write 3 Done Sent Done false
34381 9704 Acked Read 10 Done Sent Done false
34382 9705 Acked Read 4 Done Sent Done false
34383 9706 Acked Read 10 Done Sent Done false
34384 9707 Acked Read 3 Done Sent Done false
34385 9708 Acked Read 8 Done Sent Done false
34386 9709 Acked Write 6 Done Sent Done false
34387 9710 Acked Flush 0 Done Sent Done false
34388 9711 Acked Write 9 Done Sent Done false
34389 9712 Acked Read 3 Done Sent Done false
34390 9713 Acked Read 10 Done Sent Done false
34391 9714 Acked Read 9 Done Sent Done false
34392 9715 Acked Read 9 Done Sent Done false
34393 9716 Acked Read 1 Done Sent Done false
34394 9717 Acked Write 7 Done Sent Done false
34395 9718 Acked Read 8 Done Sent Done false
34396 9719 Acked Read 7 Done Sent Done false
34397 9720 Acked Flush 0 Done Sent Done false
34398 9721 Acked Read 6 Done Sent Done false
34399 9722 Acked Read 8 Done Sent Done false
34400 9723 Acked Read 1 Done Sent Done false
34401 9724 Acked Read 3 Done Sent Done false
34402 9725 Acked Write 8 Done Sent Done false
34403 9726 Acked Write 8 Done Sent Done false
34404 9727 Acked Read 3 Done Sent Done false
34405 9728 Acked Read 7 Done Sent Done false
34406 9729 Acked Read 2 Done Sent Done false
34407 9730 Acked Read 1 Done Sent Done false
34408 9731 Acked Write 10 Done Sent Done false
34409 9732 Acked Read 3 Done Sent Done false
34410 9733 Acked Write 6 Done Sent Done false
34411 9734 Acked Read 1 Done Sent Done false
34412 9735 Acked Flush 0 Done Sent Done false
34413 9736 Acked Read 6 Done Sent Done false
34414 9737 Acked Read 6 Done Sent Done false
34415 9738 Acked Read 4 Done Sent Done false
34416 9739 Acked Read 7 Done Sent Done false
34417 9740 Acked Read 3 Done Sent Done false
34418 9741 Acked Read 1 Done Sent Done false
34419 9742 Acked Write 9 Done Sent Done false
34420 9743 Acked Write 5 Done Sent Done false
34421 9744 Acked Write 8 Done Sent Done false
34422 9745 Acked Read 5 Done Sent Done false
34423 9746 Acked Read 2 Done Sent Done false
34424 9747 Acked Write 6 Done Sent Done false
34425 9748 Acked Write 4 Done Sent Done false
34426 9749 Acked Write 6 Done Sent Done false
34427 9750 Acked Read 1 Done Sent Done false
34428 9751 Acked Write 6 Done Sent Done false
34429 9752 Acked Read 2 Done Sent Done false
34430 9753 Acked Flush 0 Done Sent Done false
34431 9754 Acked Read 10 Done Sent Done false
34432 9755 Acked Read 2 Done Sent Done false
34433 STATES DS:0 DS:1 DS:2 TOTAL
34434 Sent 0 301 0 301
34435 Done 301 0 301 602
34436 Skipped 0 0 0 0
34437 Error 0 0 0 0
34438 Last Flush: Some(JobId(9753)) Some(JobId(9454)) Some(JobId(9753))
34439 Downstairs last five completed: 9454 9453 9452 9451 9450
34440 Upstairs last five completed: 9755 9754 9753 9752 9751
34441 CLIENT: Up:0 ds:301 act:2
344422025-02-07T20:14:53.807ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
344432025-02-07T20:14:54.586ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
344442025-02-07T20:14:54.586ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64177) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
344452025-02-07T20:14:54.587ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
344462025-02-07T20:14:54.588ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
344472025-02-07T20:14:54.588ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
344482025-02-07T20:14:54.588ZINFOcrucible: send last flush ID to this DS: Some(JobId(9454)) = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
344492025-02-07T20:14:54.588ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
344502025-02-07T20:14:54.588ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(9454)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
344512025-02-07T20:14:54.588ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(9454)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
34452 ----------------------------------------------------------------
34453 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
34454 Downstairs last five completed: 9756 9755 9754 9753 9752
34455 Upstairs last five completed: 9756 9755 9754 9753 9752
34456 CLIENT: Up:0 ds:0 act:3
34457 Replay: All jobs finished
34458 Main task has work to do, go find it
34459 got dsc Stop(1)
34460 stop 1
34461 Feb 07 20:14:56.843[8820] Got stop action so:false kr:false
34462 INFO request completed, latency_us: 357, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 70e41f03-ea0c-4485-acaf-7ef36a07b7a9, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
34463 Feb 07 20:14:56.844 INFO request completed, latency_us: 108, response_code: 200, uri: /state/cid/1, method: GET, req_id: 89837a0f-ce82-48d8-b11d-7c4311b8c15e, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
344642025-02-07T20:14:56.844ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
344652025-02-07T20:14:56.844ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
344662025-02-07T20:14:56.844ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
344672025-02-07T20:14:56.844ZINFOcrucible: client task is exiting = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
34468 [8820] Exited with: ExitStatus(unix_wait_status(9))
34469 [8820][1] reports Exit
34470 Feb 07 20:15:00.845 INFO request completed, latency_us: 228, response_code: 200, uri: /state/cid/1, method: GET, req_id: 33697a06-2ad6-49f1-8d80-43763ec86e7c, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
34471 001/300 Flush
34472 002/300 Read block 4693 len 4096
34473 003/300 Write block 2486 len 40960 data: 3 3 3 3 3 2 2 2 2 3
34474 004/300 Write block 47 len 36864 data: 5 6 6 5 6 6 5 5 6
34475 005/300 Write block 4643 len 4096 data: 10
34476 006/300 Read block 1425 len 4096
34477 007/300 Write block 418 len 16384 data: 7 9 9 7
34478 008/300 Write block 1568 len 28672 data: 4 4 5 5 5 6 6
34479 009/300 Flush
34480 010/300 Write block 1183 len 16384 data: 5 4 3 5
34481 011/300 Read block 34 len 24576
34482 012/300 Write block 22 len 12288 data: 8 8 8
34483 013/300 Read block 552 len 40960
34484 014/300 Read block 1898 len 8192
34485 015/300 Flush
34486 016/300 Read block 1478 len 24576
34487 017/300 Read block 4864 len 36864
34488 018/300 Read block 1162 len 4096
34489 019/300 Read block 4619 len 32768
34490 020/300 Read block 3094 len 16384
34491 021/300 Read block 2896 len 28672
34492 022/300 Flush
34493 023/300 Write block 354 len 8192 data: 2 2
34494 024/300 Read block 1180 len 24576
34495 025/300 Write block 2543 len 28672 data: 6 4 4 4 3 3 4
34496 026/300 Flush
34497 027/300 Read block 4549 len 12288
34498 028/300 Write block 160 len 16384 data: 5 5 6 6
34499 029/300 Read block 21 len 16384
34500 030/300 Read block 2802 len 36864
34501 031/300 Write block 1878 len 40960 data: 4 3 4 4 4 4 5 5 5 5
34502 032/300 Read block 2532 len 24576
34503 033/300 Read block 4186 len 16384
34504 034/300 Write block 690 len 36864 data: 5 6 5 5 5 7 6 5 5
34505 035/300 Read block 4851 len 16384
34506 036/300 Flush
34507 037/300 Read block 3113 len 12288
34508 038/300 Read block 143 len 24576
34509 039/300 Flush
34510 040/300 Read block 2082 len 36864
34511 041/300 Read block 3212 len 20480
34512 042/300 Write block 3943 len 8192 data: 5 7
34513 043/300 Write block 2909 len 12288 data: 7 7 6
34514 044/300 Write block 2251 len 20480 data: 4 4 4 3 4
34515 045/300 Write block 544 len 8192 data: 5 3
34516 046/300 Write block 3341 len 40960 data: 6 4 3 4 3 5 6 6 6 7
34517 047/300 Write block 151 len 28672 data: 5 5 5 4 3 3 4
34518 048/300 Read block 2103 len 12288
34519 049/300 Read block 1422 len 4096
34520 050/300 Read block 4959 len 40960
34521 051/300 Flush
34522 052/300 Read block 4328 len 28672
34523 053/300 Write block 4752 len 20480 data: 8 7 9 9 10
34524 054/300 Read block 3275 len 40960
34525 055/300 Flush
34526 056/300 Read block 3918 len 32768
34527 057/300 Read block 1553 len 4096
34528 058/300 Read block 1372 len 16384
34529 059/300 Flush
34530 060/300 Write block 4195 len 24576 data: 6 7 7 7 6 6
34531 061/300 Write block 2141 len 8192 data: 3 3
34532 062/300 Read block 185 len 4096
34533 063/300 Flush
34534 064/300 Write block 3671 len 12288 data: 7 6 8
34535 065/300 Write block 4701 len 24576 data: 4 4 6 6 9 9
34536 066/300 Flush
34537 067/300 Read block 4840 len 8192
34538 068/300 Write block 3462 len 8192 data: 3 4
34539 069/300 Write block 1339 len 28672 data: 6 6 8 7 8 10 10
34540 070/300 Read block 1457 len 40960
34541 071/300 Read block 3109 len 12288
34542 072/300 Flush
34543 073/300 Flush
34544 074/300 Read block 2920 len 8192
34545 075/300 Read block 485 len 12288
34546 076/300 Read block 4629 len 4096
34547 077/300 Write block 3361 len 4096 data: 5
34548 078/300 Read block 4521 len 32768
34549 079/300 Read block 3733 len 36864
34550 080/300 Read block 1187 len 24576
34551 081/300 Read block 378 len 32768
34552 082/300 Write block 4968 len 32768 data: 5 5 4 3 4 5 7 9
34553 083/300 Write block 3435 len 8192 data: 8 8
34554 084/300 Write block 1007 len 40960 data: 7 7 6 8 9 7 8 8 7 5
34555 085/300 Write block 2758 len 24576 data: 5 5 5 4 4 6
34556 086/300 Read block 4567 len 16384
34557 087/300 Flush
34558 088/300 Read block 3511 len 12288
34559 089/300 Read block 3120 len 4096
34560 090/300 Write block 354 len 28672 data: 3 3 3 4 2 3 3
34561 091/300 Write block 2873 len 16384 data: 3 3 3 3
34562 092/300 Read block 2364 len 12288
34563 093/300 Write block 4416 len 40960 data: 5 5 4 4 4 6 7 7 5 4
34564 094/300 Write block 2512 len 12288 data: 5 5 5
34565 095/300 Read block 3760 len 24576
34566 096/300 Read block 2864 len 24576
34567 097/300 Read block 1336 len 8192
34568 098/300 Flush
34569 099/300 Read block 4437 len 12288
34570 100/300 Read block 2406 len 8192
34571 101/300 Write block 4972 len 16384 data: 5 6 8 10
34572 102/300 Read block 2258 len 4096
34573 103/300 Read block 2482 len 20480
34574 104/300 Read block 2040 len 36864
34575 105/300 Read block 384 len 16384
34576 106/300 Read block 4053 len 36864
34577 107/300 Flush
34578 108/300 Read block 1247 len 32768
34579 109/300 Read block 1041 len 24576
34580 110/300 Write block 4548 len 36864 data: 6 6 5 5 5 5 6 7 6
34581 111/300 Read block 4407 len 32768
34582 112/300 Read block 4739 len 24576
34583 113/300 Read block 2595 len 12288
34584 114/300 Read block 2107 len 40960
34585 115/300 Write block 142 len 40960 data: 3 4 5 5 3 3 5 5 5 6
34586 116/300 Write block 3983 len 28672 data: 4 5 4 5 4 4 5
34587 117/300 Write block 2794 len 20480 data: 4 5 5 5 5
34588 118/300 Flush
34589 119/300 Write block 2987 len 28672 data: 5 3 4 5 5 4 3
34590 120/300 Write block 3611 len 4096 data: 8
34591 121/300 Read block 936 len 12288
34592 122/300 Read block 2749 len 40960
34593 123/300 Write block 4853 len 4096 data: 6
34594 124/300 Write block 4473 len 36864 data: 8 7 10 10 9 10 8 8 6
34595 125/300 Read block 679 len 12288
34596 126/300 Write block 3067 len 20480 data: 3 3 5 5 5
34597 127/300 Write block 2869 len 8192 data: 5 5
34598 128/300 Read block 2015 len 28672
34599 129/300 Read block 4818 len 8192
34600 130/300 Read block 1648 len 28672
34601 131/300 Write block 1692 len 28672 data: 5 8 10 11 11 9 7
34602 132/300 Read block 4871 len 28672
34603 133/300 Write block 862 len 8192 data: 5 5
34604 134/300 Write block 1219 len 32768 data: 8 9 8 9 12 12 10 10
34605 135/300 Read block 4304 len 36864
34606 136/300 Write block 4902 len 8192 data: 10 9
34607 137/300 Read block 2209 len 4096
34608 138/300 Write block 4961 len 16384 data: 10 10 9 8
34609 139/300 Read block 751 len 32768
34610 140/300 Write block 2319 len 8192 data: 7 8
34611 141/300 Read block 663 len 16384
34612 142/300 Write block 1162 len 32768 data: 5 4 5 4 4 4 4 5
34613 143/300 Write block 920 len 4096 data: 7
34614 144/300 Flush
34615 145/300 Read block 4899 len 24576
34616 146/300 Read block 1283 len 32768
34617 147/300 Read block 2291 len 36864
34618 148/300 Read block 4472 len 36864
34619 149/300 Write block 736 len 36864 data: 6 5 6 6 5 5 4 5 6
34620 150/300 Write block 1337 len 20480 data: 7 7 7 7 9
34621 151/300 Read block 328 len 24576
34622 152/300 Flush
34623 153/300 Write block 1577 len 32768 data: 6 7 7 9 9 8 7 7
34624 154/300 Write block 4534 len 4096 data: 3
34625 155/300 Write block 1895 len 28672 data: 8 8 9 10 11 11 9
34626 156/300 Write block 4164 len 16384 data: 5 5 6 7
34627 157/300 Write block 1515 len 16384 data: 6 5 4 4
34628 158/300 Read block 256 len 4096
34629 159/300 Write block 75 len 36864 data: 6 5 5 6 6 6 6 4 4
34630 160/300 Write block 4611 len 28672 data: 7 5 5 6 6 5 6
34631 161/300 Read block 4194 len 36864
34632 162/300 Read block 4128 len 36864
34633 163/300 Read block 687 len 32768
34634 164/300 Read block 4935 len 8192
34635 165/300 Write block 467 len 28672 data: 8 10 9 9 8 10 8
34636 166/300 Read block 285 len 4096
34637 167/300 Read block 1305 len 32768
34638 168/300 Flush
34639 169/300 Read block 2303 len 32768
34640 170/300 Write block 4895 len 36864 data: 9 9 7 8 8 6 7 11 10
34641 171/300 Read block 4275 len 4096
34642 172/300 Write block 2695 len 4096 data: 4
34643 173/300 Write block 2866 len 4096 data: 5
34644 174/300 Read block 2287 len 12288
34645 175/300 Write block 487 len 12288 data: 4 4 3
34646 176/300 Read block 3370 len 24576
34647 177/300 Read block 683 len 24576
34648 178/300 Write block 4326 len 32768 data: 9 7 7 4 6 7 8 5
34649 179/300 Write block 1252 len 4096 data: 6
34650 180/300 Write block 2904 len 4096 data: 5
34651 181/300 Read block 3808 len 12288
34652 182/300 Read block 2998 len 12288
34653 183/300 Read block 4286 len 12288
34654 184/300 Read block 1126 len 40960
34655 185/300 Read block 27 len 20480
34656 186/300 Read block 2831 len 28672
34657 187/300 Read block 4534 len 8192
34658 188/300 Read block 1969 len 4096
34659 189/300 Write block 3489 len 8192 data: 9 7
34660 190/300 Read block 2979 len 32768
34661 191/300 Read block 4381 len 16384
34662 192/300 Flush
34663 193/300 Write block 552 len 36864 data: 4 4 3 4 5 6 6 6 7
34664 194/300 Read block 4963 len 8192
34665 195/300 Read block 1984 len 16384
34666 196/300 Read block 4403 len 20480
34667 197/300 Write block 3037 len 4096 data: 5
34668 198/300 Read block 1457 len 16384
34669 199/300 Read block 1080 len 24576
34670 200/300 Read block 1656 len 24576
34671 201/300 Write block 2963 len 12288 data: 5 5 5
34672 202/300 Write block 4309 len 4096 data: 4
34673 203/300 Write block 2137 len 8192 data: 5 4
34674 204/300 Read block 4017 len 12288
34675 205/300 Write block 1736 len 8192 data: 5 6
34676 206/300 Flush
34677 207/300 Read block 67 len 24576
34678 208/300 Read block 2691 len 40960
34679 209/300 Read block 2696 len 8192
34680 210/300 Write block 4234 len 16384 data: 6 4 6 7
34681 211/300 Write block 2334 len 24576 data: 4 5 5 5 7 7
34682 212/300 Read block 1918 len 8192
34683 213/300 Read block 2229 len 28672
34684 214/300 Write block 1647 len 16384 data: 7 8 8 8
34685 215/300 Write block 322 len 24576 data: 8 7 6 6 5 5
34686 216/300 Read block 1847 len 8192
34687 217/300 Flush
34688 218/300 Write block 4121 len 12288 data: 5 6 5
34689 219/300 Read block 3921 len 12288
34690 220/300 Flush
34691 221/300 Write block 4851 len 20480 data: 7 6 7 5 6
34692 222/300 Write block 4364 len 4096 data: 2
34693 223/300 Write block 2523 len 16384 data: 6 7 7 6
34694 224/300 Read block 3373 len 20480
34695 225/300 Read block 3114 len 4096
34696 226/300 Flush
34697 227/300 Read block 424 len 24576
34698 228/300 Read block 1854 len 40960
34699 229/300 Flush
34700 230/300 Read block 4829 len 36864
34701 231/300 Read block 4119 len 8192
34702 232/300 Read block 1648 len 40960
34703 233/300 Read block 4739 len 36864
34704 234/300 Write block 4171 len 36864 data: 6 6 6 6 6 7 6 6 5
34705 235/300 Write block 3930 len 4096 data: 4
34706 236/300 Read block 348 len 4096
34707 237/300 Flush
34708 238/300 Write block 4217 len 36864 data: 9 9 9 8 6 7 7 7 6
34709 239/300 Write block 4448 len 4096 data: 3
34710 240/300 Write block 914 len 16384 data: 5 5 6 5
34711 241/300 Write block 3804 len 16384 data: 8 9 8 8
34712 242/300 Write block 555 len 16384 data: 5 6 7 7
34713 243/300 Write block 4397 len 20480 data: 4 4 5 6 5
34714 244/300 Read block 3335 len 8192
34715 245/300 Flush
34716 246/300 Write block 1433 len 4096 data: 6
34717 247/300 Write block 3772 len 28672 data: 4 5 7 8 7 9 9
34718 248/300 Flush
34719 249/300 Read block 289 len 24576
34720 250/300 Read block 601 len 20480
34721 251/300 Read block 1468 len 8192
34722 252/300 Read block 2308 len 24576
34723 253/300 Write block 638 len 4096 data: 5
34724 254/300 Read block 52 len 20480
34725 255/300 Read block 4189 len 28672
34726 256/300 Write block 4607 len 8192 data: 5 5
34727 257/300 Read block 4989 len 12288
34728 258/300 Read block 250 len 24576
34729 259/300 Read block 1289 len 24576
34730 260/300 Read block 181 len 20480
34731 261/300 Write block 3606 len 40960 data: 4 8 8 11 9 9 7 5 5 6
34732 262/300 Write block 226 len 36864 data: 7 5 3 3 3 3 4 5 5
34733 263/300 Write block 1081 len 4096 data: 6
34734 264/300 Read block 1391 len 28672
34735 265/300 Flush
34736 266/300 Read block 2743 len 16384
34737 267/300 Write block 4997 len 8192 data: 4 4
34738 268/300 Write block 1312 len 32768 data: 3 4 6 7 11 11 11 9
34739 269/300 Write block 3041 len 8192 data: 6 6
34740 270/300 Write block 2350 len 16384 data: 7 6 5 6
34741 271/300 Flush
34742 272/300 Read block 1796 len 28672
34743 273/300 Write block 372 len 24576 data: 9 8 8 7 3 3
34744 274/300 Write block 445 len 28672 data: 6 5 6 8 9 9 9
34745 275/300 Read block 1780 len 36864
34746 276/300 Flush
34747 277/300 Read block 2142 len 24576
34748 278/300 Read block 1649 len 40960
34749 279/300 Read block 2786 len 8192
34750 280/300 Write block 3254 len 12288 data: 7 6 5
34751 281/300 Read block 1226 len 20480
34752 282/300 Write block 552 len 20480 data: 5 5 4 6 7
34753 283/300 Write block 3712 len 32768 data: 6 6 6 7 8 7 6 7
34754 284/300 Read block 3046 len 24576
34755 285/300 Write block 616 len 36864 data: 6 6 7 5 6 5 6 7 6
34756 286/300 Read block 3555 len 16384
34757 287/300 Flush
34758 288/300 Write block 2753 len 20480 data: 5 6 7 8 5
34759 289/300 Write block 2763 len 20480 data: 7 5 5 4 6
34760 290/300 Write block 3834 len 20480 data: 5 6 6 6 8
34761 291/300 Flush
34762 292/300 Read block 4018 len 16384
34763 293/300 Read block 2758 len 4096
34764 294/300 Write block 4299 len 20480 data: 5 3 4 3 3
34765 295/300 Write block 1026 len 4096 data: 6
34766 296/300 Read block 36 len 12288
34767 297/300 Read block 2673 len 16384
34768 298/300 Write block 1232 len 40960 data: 6 6 6 7 5 4 3 4 4 4
34769 299/300 Read block 1227 len 24576
34770 300/300 Read block 1411 len 20480
34771 301/300 Flush
34772 Main task has work to do, go find it
34773 got dsc Start(1)
34774 start 1
34775 Feb 07 20:15:01.077 INFO request completed, latency_us: 201, response_code: 204, uri: /start/cid/1, method: POST, req_id: cb21773f-fe7d-4046-820c-8eb7651d12bf, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
34776 Replay: started 1, returned:Ok(())
34777 ----------------------------------------------------------------
34778 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34779 States: Active Offline Active
34780 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34781 9757 Acked Flush 0 Done Sent Done false
34782 9758 Acked Read 1 Done Sent Done false
34783 9759 Acked Write 10 Done Sent Done false
34784 9760 Acked Write 9 Done Sent Done false
34785 9761 Acked Write 1 Done Sent Done false
34786 9762 Acked Read 1 Done Sent Done false
34787 9763 Acked Write 4 Done Sent Done false
34788 9764 Acked Write 7 Done Sent Done false
34789 9765 Acked Flush 0 Done Sent Done false
34790 9766 Acked Write 4 Done Sent Done false
34791 9767 Acked Read 6 Done Sent Done false
34792 9768 Acked Write 3 Done Sent Done false
34793 9769 Acked Read 10 Done Sent Done false
34794 9770 Acked Read 2 Done Sent Done false
34795 9771 Acked Flush 0 Done Sent Done false
34796 9772 Acked Read 6 Done Sent Done false
34797 9773 Acked Read 9 Done Sent Done false
34798 9774 Acked Read 1 Done Sent Done false
34799 9775 Acked Read 8 Done Sent Done false
34800 9776 Acked Read 4 Done Sent Done false
34801 9777 Acked Read 7 Done Sent Done false
34802 9778 Acked Flush 0 Done Sent Done false
34803 9779 Acked Write 2 Done Sent Done false
34804 9780 Acked Read 6 Done Sent Done false
34805 9781 Acked Write 7 Done Sent Done false
34806 9782 Acked Flush 0 Done Sent Done false
34807 9783 Acked Read 3 Done Sent Done false
34808 9784 Acked Write 4 Done Sent Done false
34809 9785 Acked Read 4 Done Sent Done false
34810 9786 Acked Read 9 Done Sent Done false
34811 9787 Acked Write 10 Done Sent Done false
34812 9788 Acked Read 6 Done Sent Done false
34813 9789 Acked Read 4 Done Sent Done false
34814 9790 Acked Write 9 Done Sent Done false
34815 9791 Acked Read 4 Done Sent Done false
34816 9792 Acked Flush 0 Done Sent Done false
34817 9793 Acked Read 3 Done Sent Done false
34818 9794 Acked Read 6 Done Sent Done false
34819 9795 Acked Flush 0 Done Sent Done false
34820 9796 Acked Read 9 Done Sent Done false
34821 9797 Acked Read 5 Done Sent Done false
34822 9798 Acked Write 2 Done Sent Done false
34823 9799 Acked Write 3 Done Sent Done false
34824 9800 Acked Write 5 Done Sent Done false
34825 9801 Acked Write 2 Done Sent Done false
34826 9802 Acked Write 10 Done Sent Done false
34827 9803 Acked Write 7 Done Sent Done false
34828 9804 Acked Read 3 Done Sent Done false
34829 9805 Acked Read 1 Done Sent Done false
34830 9806 Acked Read 10 Done Sent Done false
34831 9807 Acked Flush 0 Done Sent Done false
34832 9808 Acked Read 7 Done Sent Done false
34833 9809 Acked Write 5 Done Sent Done false
34834 9810 Acked Read 10 Done Sent Done false
34835 9811 Acked Flush 0 Done Sent Done false
34836 9812 Acked Read 8 Done Sent Done false
34837 9813 Acked Read 1 Done Sent Done false
34838 9814 Acked Read 4 Done Sent Done false
34839 9815 Acked Flush 0 Done Sent Done false
34840 9816 Acked Write 6 Done Sent Done false
34841 9817 Acked Write 2 Done Sent Done false
34842 9818 Acked Read 1 Done Sent Done false
34843 9819 Acked Flush 0 Done Sent Done false
34844 9820 Acked Write 3 Done Sent Done false
34845 9821 Acked Write 6 Done Sent Done false
34846 9822 Acked Flush 0 Done Sent Done false
34847 9823 Acked Read 2 Done Sent Done false
34848 9824 Acked Write 2 Done Sent Done false
34849 9825 Acked Write 7 Done Sent Done false
34850 9826 Acked Read 10 Done Sent Done false
34851 9827 Acked Read 3 Done Sent Done false
34852 9828 Acked Flush 0 Done Sent Done false
34853 9829 Acked Flush 0 Done Sent Done false
34854 9830 Acked Read 2 Done Sent Done false
34855 9831 Acked Read 3 Done Sent Done false
34856 9832 Acked Read 1 Done Sent Done false
34857 9833 Acked Write 1 Done Sent Done false
34858 9834 Acked Read 8 Done Sent Done false
34859 9835 Acked Read 9 Done Sent Done false
34860 9836 Acked Read 6 Done Sent Done false
34861 9837 Acked Read 8 Done Sent Done false
34862 9838 Acked Write 8 Done Sent Done false
34863 9839 Acked Write 2 Done Sent Done false
34864 9840 Acked Write 10 Done Sent Done false
34865 9841 Acked Write 6 Done Sent Done false
34866 9842 Acked Read 4 Done Sent Done false
34867 9843 Acked Flush 0 Done Sent Done false
34868 9844 Acked Read 3 Done Sent Done false
34869 9845 Acked Read 1 Done Sent Done false
34870 9846 Acked Write 7 Done Sent Done false
34871 9847 Acked Write 4 Done Sent Done false
34872 9848 Acked Read 3 Done Sent Done false
34873 9849 Acked Write 10 Done Sent Done false
34874 9850 Acked Write 3 Done Sent Done false
34875 9851 Acked Read 6 Done Sent Done false
34876 9852 Acked Read 6 Done Sent Done false
34877 9853 Acked Read 2 Done Sent Done false
34878 9854 Acked Flush 0 Done Sent Done false
34879 9855 Acked Read 3 Done Sent Done false
34880 9856 Acked Read 2 Done Sent Done false
34881 9857 Acked Write 4 Done Sent Done false
34882 9858 Acked Read 1 Done Sent Done false
34883 9859 Acked Read 5 Done Sent Done false
34884 9860 Acked Read 9 Done Sent Done false
34885 9861 Acked Read 4 Done Sent Done false
34886 9862 Acked Read 9 Done Sent Done false
34887 9863 Acked Flush 0 Done Sent Done false
34888 9864 Acked Read 8 Done Sent Done false
34889 9865 Acked Read 6 Done Sent Done false
34890 9866 Acked Write 9 Done Sent Done false
34891 9867 Acked Read 8 Done Sent Done false
34892 9868 Acked Read 6 Done Sent Done false
34893 9869 Acked Read 3 Done Sent Done false
34894 9870 Acked Read 10 Done Sent Done false
34895 9871 Acked Write 10 Done Sent Done false
34896 9872 Acked Write 7 Done Sent Done false
34897 9873 Acked Write 5 Done Sent Done false
34898 9874 Acked Flush 0 Done Sent Done false
34899 9875 Acked Write 7 Done Sent Done false
34900 9876 Acked Write 1 Done Sent Done false
34901 9877 Acked Read 3 Done Sent Done false
34902 9878 Acked Read 10 Done Sent Done false
34903 9879 Acked Write 1 Done Sent Done false
34904 9880 Acked Write 9 Done Sent Done false
34905 9881 Acked Read 3 Done Sent Done false
34906 9882 Acked Write 5 Done Sent Done false
34907 9883 Acked Write 2 Done Sent Done false
34908 9884 Acked Read 7 Done Sent Done false
34909 9885 Acked Read 2 Done Sent Done false
34910 9886 Acked Read 7 Done Sent Done false
34911 9887 Acked Write 7 Done Sent Done false
34912 9888 Acked Read 7 Done Sent Done false
34913 9889 Acked Write 2 Done Sent Done false
34914 9890 Acked Write 8 Done Sent Done false
34915 9891 Acked Read 9 Done Sent Done false
34916 9892 Acked Write 2 Done Sent Done false
34917 9893 Acked Read 1 Done Sent Done false
34918 9894 Acked Write 4 Done Sent Done false
34919 9895 Acked Read 8 Done Sent Done false
34920 9896 Acked Write 2 Done Sent Done false
34921 9897 Acked Read 4 Done Sent Done false
34922 9898 Acked Write 8 Done Sent Done false
34923 9899 Acked Write 1 Done Sent Done false
34924 9900 Acked Flush 0 Done Sent Done false
34925 9901 Acked Read 6 Done Sent Done false
34926 9902 Acked Read 8 Done Sent Done false
34927 9903 Acked Read 9 Done Sent Done false
34928 9904 Acked Read 9 Done Sent Done false
34929 9905 Acked Write 9 Done Sent Done false
34930 9906 Acked Write 5 Done Sent Done false
34931 9907 Acked Read 6 Done Sent Done false
34932 9908 Acked Flush 0 Done Sent Done false
34933 9909 Acked Write 8 Done Sent Done false
34934 9910 Acked Write 1 Done Sent Done false
34935 9911 Acked Write 7 Done Sent Done false
34936 9912 Acked Write 4 Done Sent Done false
34937 9913 Acked Write 4 Done Sent Done false
34938 9914 Acked Read 1 Done Sent Done false
34939 9915 Acked Write 9 Done Sent Done false
34940 9916 Acked Write 7 Done Sent Done false
34941 9917 Acked Read 9 Done Sent Done false
34942 9918 Acked Read 9 Done Sent Done false
34943 9919 Acked Read 8 Done Sent Done false
34944 9920 Acked Read 2 Done Sent Done false
34945 9921 Acked Write 7 Done Sent Done false
34946 9922 Acked Read 1 Done Sent Done false
34947 9923 Acked Read 8 Done Sent Done false
34948 9924 Acked Flush 0 Done Sent Done false
34949 9925 Acked Read 8 Done Sent Done false
34950 9926 Acked Write 9 Done Sent Done false
34951 9927 Acked Read 1 Done Sent Done false
34952 9928 Acked Write 1 Done Sent Done false
34953 9929 Acked Write 1 Done Sent Done false
34954 9930 Acked Read 3 Done Sent Done false
34955 9931 Acked Write 3 Done Sent Done false
34956 9932 Acked Read 6 Done Sent Done false
34957 9933 Acked Read 6 Done Sent Done false
34958 9934 Acked Write 8 Done Sent Done false
34959 9935 Acked Write 1 Done Sent Done false
34960 9936 Acked Write 1 Done Sent Done false
34961 9937 Acked Read 3 Done Sent Done false
34962 9938 Acked Read 3 Done Sent Done false
34963 9939 Acked Read 3 Done Sent Done false
34964 9940 Acked Read 10 Done Sent Done false
34965 9941 Acked Read 5 Done Sent Done false
34966 9942 Acked Read 7 Done Sent Done false
34967 9943 Acked Read 2 Done Sent Done false
34968 9944 Acked Read 1 Done Sent Done false
34969 9945 Acked Write 2 Done Sent Done false
34970 9946 Acked Read 8 Done Sent Done false
34971 9947 Acked Read 4 Done Sent Done false
34972 9948 Acked Flush 0 Done Sent Done false
34973 9949 Acked Write 9 Done Sent Done false
34974 9950 Acked Read 2 Done Sent Done false
34975 9951 Acked Read 4 Done Sent Done false
34976 9952 Acked Read 5 Done Sent Done false
34977 9953 Acked Write 1 Done Sent Done false
34978 9954 Acked Read 4 Done Sent Done false
34979 9955 Acked Read 6 Done Sent Done false
34980 9956 Acked Read 6 Done Sent Done false
34981 9957 Acked Write 3 Done Sent Done false
34982 9958 Acked Write 1 Done Sent Done false
34983 9959 Acked Write 2 Done Sent Done false
34984 9960 Acked Read 3 Done Sent Done false
34985 9961 Acked Write 2 Done Sent Done false
34986 9962 Acked Flush 0 Done Sent Done false
34987 9963 Acked Read 6 Done Sent Done false
34988 9964 Acked Read 10 Done Sent Done false
34989 9965 Acked Read 2 Done Sent Done false
34990 9966 Acked Write 4 Done Sent Done false
34991 9967 Acked Write 6 Done Sent Done false
34992 9968 Acked Read 2 Done Sent Done false
34993 9969 Acked Read 7 Done Sent Done false
34994 9970 Acked Write 4 Done Sent Done false
34995 9971 Acked Write 6 Done Sent Done false
34996 9972 Acked Read 2 Done Sent Done false
34997 9973 Acked Flush 0 Done Sent Done false
34998 9974 Acked Write 3 Done Sent Done false
34999 9975 Acked Read 3 Done Sent Done false
35000 9976 Acked Flush 0 Done Sent Done false
35001 9977 Acked Write 5 Done Sent Done false
35002 9978 Acked Write 1 Done Sent Done false
35003 9979 Acked Write 4 Done Sent Done false
35004 9980 Acked Read 5 Done Sent Done false
35005 9981 Acked Read 1 Done Sent Done false
35006 9982 Acked Flush 0 Done Sent Done false
35007 9983 Acked Read 6 Done Sent Done false
35008 9984 Acked Read 10 Done Sent Done false
35009 9985 Acked Flush 0 Done Sent Done false
35010 9986 Acked Read 9 Done Sent Done false
35011 9987 Acked Read 2 Done Sent Done false
35012 9988 Acked Read 10 Done Sent Done false
35013 9989 Acked Read 9 Done Sent Done false
35014 9990 Acked Write 9 Done Sent Done false
35015 9991 Acked Write 1 Done Sent Done false
35016 9992 Acked Read 1 Done Sent Done false
35017 9993 Acked Flush 0 Done Sent Done false
35018 9994 Acked Write 9 Done Sent Done false
35019 9995 Acked Write 1 Done Sent Done false
35020 9996 Acked Write 4 Done Sent Done false
35021 9997 Acked Write 4 Done Sent Done false
35022 9998 Acked Write 4 Done Sent Done false
35023 9999 Acked Write 5 Done Sent Done false
35024 10000 Acked Read 2 Done Sent Done false
35025 10001 Acked Flush 0 Done Sent Done false
35026 10002 Acked Write 1 Done Sent Done false
35027 10003 Acked Write 7 Done Sent Done false
35028 10004 Acked Flush 0 Done Sent Done false
35029 10005 Acked Read 6 Done Sent Done false
35030 10006 Acked Read 5 Done Sent Done false
35031 10007 Acked Read 2 Done Sent Done false
35032 10008 Acked Read 6 Done Sent Done false
35033 10009 Acked Write 1 Done Sent Done false
35034 10010 Acked Read 5 Done Sent Done false
35035 10011 Acked Read 7 Done Sent Done false
35036 10012 Acked Write 2 Done Sent Done false
35037 10013 Acked Read 3 Done Sent Done false
35038 10014 Acked Read 6 Done Sent Done false
35039 10015 Acked Read 6 Done Sent Done false
35040 10016 Acked Read 5 Done Sent Done false
35041 10017 Acked Write 10 Done Sent Done false
35042 10018 Acked Write 9 Done Sent Done false
35043 10019 Acked Write 1 Done Sent Done false
35044 10020 Acked Read 7 Done Sent Done false
35045 10021 Acked Flush 0 Done Sent Done false
35046 10022 Acked Read 4 Done Sent Done false
35047 10023 Acked Write 2 Done Sent Done false
35048 10024 Acked Write 8 Done Sent Done false
35049 10025 Acked Write 2 Done Sent Done false
35050 10026 Acked Write 4 Done Sent Done false
35051 10027 Acked Flush 0 Done Sent Done false
35052 10028 Acked Read 7 Done Sent Done false
35053 10029 Acked Write 6 Done Sent Done false
35054 10030 Acked Write 7 Done Sent Done false
35055 10031 Acked Read 9 Done Sent Done false
35056 10032 Acked Flush 0 Done Sent Done false
35057 10033 Acked Read 6 Done Sent Done false
35058 10034 Acked Read 10 Done Sent Done false
35059 10035 Acked Read 2 Done Sent Done false
35060 10036 Acked Write 3 Done Sent Done false
35061 10037 Acked Read 5 Done Sent Done false
35062 10038 Acked Write 5 Done Sent Done false
35063 10039 Acked Write 8 Done Sent Done false
35064 10040 Acked Read 6 Done Sent Done false
35065 10041 Acked Write 9 Done Sent Done false
35066 10042 Acked Read 4 Done Sent Done false
35067 10043 Acked Flush 0 Done Sent Done false
35068 10044 Acked Write 5 Done Sent Done false
35069 10045 Acked Write 5 Done Sent Done false
35070 10046 Acked Write 5 Done Sent Done false
35071 10047 Acked Flush 0 Done Sent Done false
35072 10048 Acked Read 4 Done Sent Done false
35073 10049 Acked Read 1 Done Sent Done false
35074 10050 Acked Write 5 Done Sent Done false
35075 10051 Acked Write 1 Done Sent Done false
35076 10052 Acked Read 3 Done Sent Done false
35077 10053 Acked Read 4 Done Sent Done false
35078 10054 Acked Write 10 Done Sent Done false
35079 10055 Acked Read 6 Done Sent Done false
35080 10056 Acked Read 5 Done Sent Done false
35081 10057 Acked Flush 0 Done Sent Done false
35082 STATES DS:0 DS:1 DS:2 TOTAL
35083 Sent 0 301 0 301
35084 Done 301 0 301 602
35085 Skipped 0 0 0 0
35086 Error 0 0 0 0
35087 Last Flush: Some(JobId(10057)) Some(JobId(9756)) Some(JobId(10057))
35088 Downstairs last five completed: 9756 9755 9754 9753 9752
35089 Upstairs last five completed: 10057 10056 10055 10054 10053
35090 CLIENT: Up:0 ds:301 act:2
350912025-02-07T20:15:01.692ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
350922025-02-07T20:15:02.834ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
35093 [8820] got start action
35094 [8820] I am going to restart
35095 Starting downstairs at port 8820
35096 Make output file at "/tmp/dsc/downstairs-8820.txt"
35097 [8820][1] reports Starting
35098 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1239)
35099 [8820][1] reports Running
351002025-02-07T20:15:03.809ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
351012025-02-07T20:15:04.696ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
35102 ----------------------------------------------------------------
35103 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35104 States: Active Offline Active
35105 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35106 9757 Acked Flush 0 Done Sent Done false
35107 9758 Acked Read 1 Done Sent Done false
35108 9759 Acked Write 10 Done Sent Done false
35109 9760 Acked Write 9 Done Sent Done false
35110 9761 Acked Write 1 Done Sent Done false
35111 9762 Acked Read 1 Done Sent Done false
35112 9763 Acked Write 4 Done Sent Done false
35113 9764 Acked Write 7 Done Sent Done false
35114 9765 Acked Flush 0 Done Sent Done false
35115 9766 Acked Write 4 Done Sent Done false
35116 9767 Acked Read 6 Done Sent Done false
35117 9768 Acked Write 3 Done Sent Done false
35118 9769 Acked Read 10 Done Sent Done false
35119 9770 Acked Read 2 Done Sent Done false
35120 9771 Acked Flush 0 Done Sent Done false
35121 9772 Acked Read 6 Done Sent Done false
35122 9773 Acked Read 9 Done Sent Done false
35123 9774 Acked Read 1 Done Sent Done false
35124 9775 Acked Read 8 Done Sent Done false
35125 9776 Acked Read 4 Done Sent Done false
35126 9777 Acked Read 7 Done Sent Done false
35127 9778 Acked Flush 0 Done Sent Done false
35128 9779 Acked Write 2 Done Sent Done false
35129 9780 Acked Read 6 Done Sent Done false
35130 9781 Acked Write 7 Done Sent Done false
35131 9782 Acked Flush 0 Done Sent Done false
35132 9783 Acked Read 3 Done Sent Done false
35133 9784 Acked Write 4 Done Sent Done false
35134 9785 Acked Read 4 Done Sent Done false
35135 9786 Acked Read 9 Done Sent Done false
35136 9787 Acked Write 10 Done Sent Done false
35137 9788 Acked Read 6 Done Sent Done false
35138 9789 Acked Read 4 Done Sent Done false
35139 9790 Acked Write 9 Done Sent Done false
35140 9791 Acked Read 4 Done Sent Done false
35141 9792 Acked Flush 0 Done Sent Done false
35142 9793 Acked Read 3 Done Sent Done false
35143 9794 Acked Read 6 Done Sent Done false
35144 9795 Acked Flush 0 Done Sent Done false
35145 9796 Acked Read 9 Done Sent Done false
35146 9797 Acked Read 5 Done Sent Done false
35147 9798 Acked Write 2 Done Sent Done false
35148 9799 Acked Write 3 Done Sent Done false
35149 9800 Acked Write 5 Done Sent Done false
35150 9801 Acked Write 2 Done Sent Done false
35151 9802 Acked Write 10 Done Sent Done false
35152 9803 Acked Write 7 Done Sent Done false
35153 9804 Acked Read 3 Done Sent Done false
35154 9805 Acked Read 1 Done Sent Done false
35155 9806 Acked Read 10 Done Sent Done false
35156 9807 Acked Flush 0 Done Sent Done false
35157 9808 Acked Read 7 Done Sent Done false
35158 9809 Acked Write 5 Done Sent Done false
35159 9810 Acked Read 10 Done Sent Done false
35160 9811 Acked Flush 0 Done Sent Done false
35161 9812 Acked Read 8 Done Sent Done false
35162 9813 Acked Read 1 Done Sent Done false
35163 9814 Acked Read 4 Done Sent Done false
35164 9815 Acked Flush 0 Done Sent Done false
35165 9816 Acked Write 6 Done Sent Done false
35166 9817 Acked Write 2 Done Sent Done false
35167 9818 Acked Read 1 Done Sent Done false
35168 9819 Acked Flush 0 Done Sent Done false
35169 9820 Acked Write 3 Done Sent Done false
35170 9821 Acked Write 6 Done Sent Done false
35171 9822 Acked Flush 0 Done Sent Done false
35172 9823 Acked Read 2 Done Sent Done false
35173 9824 Acked Write 2 Done Sent Done false
35174 9825 Acked Write 7 Done Sent Done false
35175 9826 Acked Read 10 Done Sent Done false
35176 9827 Acked Read 3 Done Sent Done false
35177 9828 Acked Flush 0 Done Sent Done false
35178 9829 Acked Flush 0 Done Sent Done false
35179 9830 Acked Read 2 Done Sent Done false
35180 9831 Acked Read 3 Done Sent Done false
35181 9832 Acked Read 1 Done Sent Done false
35182 9833 Acked Write 1 Done Sent Done false
35183 9834 Acked Read 8 Done Sent Done false
35184 9835 Acked Read 9 Done Sent Done false
35185 9836 Acked Read 6 Done Sent Done false
35186 9837 Acked Read 8 Done Sent Done false
35187 9838 Acked Write 8 Done Sent Done false
35188 9839 Acked Write 2 Done Sent Done false
35189 9840 Acked Write 10 Done Sent Done false
35190 9841 Acked Write 6 Done Sent Done false
35191 9842 Acked Read 4 Done Sent Done false
35192 9843 Acked Flush 0 Done Sent Done false
35193 9844 Acked Read 3 Done Sent Done false
35194 9845 Acked Read 1 Done Sent Done false
35195 9846 Acked Write 7 Done Sent Done false
35196 9847 Acked Write 4 Done Sent Done false
35197 9848 Acked Read 3 Done Sent Done false
35198 9849 Acked Write 10 Done Sent Done false
35199 9850 Acked Write 3 Done Sent Done false
35200 9851 Acked Read 6 Done Sent Done false
35201 9852 Acked Read 6 Done Sent Done false
35202 9853 Acked Read 2 Done Sent Done false
35203 9854 Acked Flush 0 Done Sent Done false
35204 9855 Acked Read 3 Done Sent Done false
35205 9856 Acked Read 2 Done Sent Done false
35206 9857 Acked Write 4 Done Sent Done false
35207 9858 Acked Read 1 Done Sent Done false
35208 9859 Acked Read 5 Done Sent Done false
35209 9860 Acked Read 9 Done Sent Done false
35210 9861 Acked Read 4 Done Sent Done false
35211 9862 Acked Read 9 Done Sent Done false
35212 9863 Acked Flush 0 Done Sent Done false
35213 9864 Acked Read 8 Done Sent Done false
35214 9865 Acked Read 6 Done Sent Done false
35215 9866 Acked Write 9 Done Sent Done false
35216 9867 Acked Read 8 Done Sent Done false
35217 9868 Acked Read 6 Done Sent Done false
35218 9869 Acked Read 3 Done Sent Done false
35219 9870 Acked Read 10 Done Sent Done false
35220 9871 Acked Write 10 Done Sent Done false
35221 9872 Acked Write 7 Done Sent Done false
35222 9873 Acked Write 5 Done Sent Done false
35223 9874 Acked Flush 0 Done Sent Done false
35224 9875 Acked Write 7 Done Sent Done false
35225 9876 Acked Write 1 Done Sent Done false
35226 9877 Acked Read 3 Done Sent Done false
35227 9878 Acked Read 10 Done Sent Done false
35228 9879 Acked Write 1 Done Sent Done false
35229 9880 Acked Write 9 Done Sent Done false
35230 9881 Acked Read 3 Done Sent Done false
35231 9882 Acked Write 5 Done Sent Done false
35232 9883 Acked Write 2 Done Sent Done false
35233 9884 Acked Read 7 Done Sent Done false
35234 9885 Acked Read 2 Done Sent Done false
35235 9886 Acked Read 7 Done Sent Done false
35236 9887 Acked Write 7 Done Sent Done false
35237 9888 Acked Read 7 Done Sent Done false
35238 9889 Acked Write 2 Done Sent Done false
35239 9890 Acked Write 8 Done Sent Done false
35240 9891 Acked Read 9 Done Sent Done false
35241 9892 Acked Write 2 Done Sent Done false
35242 9893 Acked Read 1 Done Sent Done false
35243 9894 Acked Write 4 Done Sent Done false
35244 9895 Acked Read 8 Done Sent Done false
35245 9896 Acked Write 2 Done Sent Done false
35246 9897 Acked Read 4 Done Sent Done false
35247 9898 Acked Write 8 Done Sent Done false
35248 9899 Acked Write 1 Done Sent Done false
35249 9900 Acked Flush 0 Done Sent Done false
35250 9901 Acked Read 6 Done Sent Done false
35251 9902 Acked Read 8 Done Sent Done false
35252 9903 Acked Read 9 Done Sent Done false
35253 9904 Acked Read 9 Done Sent Done false
35254 9905 Acked Write 9 Done Sent Done false
35255 9906 Acked Write 5 Done Sent Done false
35256 9907 Acked Read 6 Done Sent Done false
35257 9908 Acked Flush 0 Done Sent Done false
35258 9909 Acked Write 8 Done Sent Done false
35259 9910 Acked Write 1 Done Sent Done false
35260 9911 Acked Write 7 Done Sent Done false
35261 9912 Acked Write 4 Done Sent Done false
35262 9913 Acked Write 4 Done Sent Done false
35263 9914 Acked Read 1 Done Sent Done false
35264 9915 Acked Write 9 Done Sent Done false
35265 9916 Acked Write 7 Done Sent Done false
35266 9917 Acked Read 9 Done Sent Done false
35267 9918 Acked Read 9 Done Sent Done false
35268 9919 Acked Read 8 Done Sent Done false
35269 9920 Acked Read 2 Done Sent Done false
35270 9921 Acked Write 7 Done Sent Done false
35271 9922 Acked Read 1 Done Sent Done false
35272 9923 Acked Read 8 Done Sent Done false
35273 9924 Acked Flush 0 Done Sent Done false
35274 9925 Acked Read 8 Done Sent Done false
35275 9926 Acked Write 9 Done Sent Done false
35276 9927 Acked Read 1 Done Sent Done false
35277 9928 Acked Write 1 Done Sent Done false
35278 9929 Acked Write 1 Done Sent Done false
35279 9930 Acked Read 3 Done Sent Done false
35280 9931 Acked Write 3 Done Sent Done false
35281 9932 Acked Read 6 Done Sent Done false
35282 9933 Acked Read 6 Done Sent Done false
35283 9934 Acked Write 8 Done Sent Done false
35284 9935 Acked Write 1 Done Sent Done false
35285 9936 Acked Write 1 Done Sent Done false
35286 9937 Acked Read 3 Done Sent Done false
35287 9938 Acked Read 3 Done Sent Done false
35288 9939 Acked Read 3 Done Sent Done false
35289 9940 Acked Read 10 Done Sent Done false
35290 9941 Acked Read 5 Done Sent Done false
35291 9942 Acked Read 7 Done Sent Done false
35292 9943 Acked Read 2 Done Sent Done false
35293 9944 Acked Read 1 Done Sent Done false
35294 9945 Acked Write 2 Done Sent Done false
35295 9946 Acked Read 8 Done Sent Done false
35296 9947 Acked Read 4 Done Sent Done false
35297 9948 Acked Flush 0 Done Sent Done false
35298 9949 Acked Write 9 Done Sent Done false
35299 9950 Acked Read 2 Done Sent Done false
35300 9951 Acked Read 4 Done Sent Done false
35301 9952 Acked Read 5 Done Sent Done false
35302 9953 Acked Write 1 Done Sent Done false
35303 9954 Acked Read 4 Done Sent Done false
35304 9955 Acked Read 6 Done Sent Done false
35305 9956 Acked Read 6 Done Sent Done false
35306 9957 Acked Write 3 Done Sent Done false
35307 9958 Acked Write 1 Done Sent Done false
35308 9959 Acked Write 2 Done Sent Done false
35309 9960 Acked Read 3 Done Sent Done false
35310 9961 Acked Write 2 Done Sent Done false
35311 9962 Acked Flush 0 Done Sent Done false
35312 9963 Acked Read 6 Done Sent Done false
35313 9964 Acked Read 10 Done Sent Done false
35314 9965 Acked Read 2 Done Sent Done false
35315 9966 Acked Write 4 Done Sent Done false
35316 9967 Acked Write 6 Done Sent Done false
35317 9968 Acked Read 2 Done Sent Done false
35318 9969 Acked Read 7 Done Sent Done false
35319 9970 Acked Write 4 Done Sent Done false
35320 9971 Acked Write 6 Done Sent Done false
35321 9972 Acked Read 2 Done Sent Done false
35322 9973 Acked Flush 0 Done Sent Done false
35323 9974 Acked Write 3 Done Sent Done false
35324 9975 Acked Read 3 Done Sent Done false
35325 9976 Acked Flush 0 Done Sent Done false
35326 9977 Acked Write 5 Done Sent Done false
35327 9978 Acked Write 1 Done Sent Done false
35328 9979 Acked Write 4 Done Sent Done false
35329 9980 Acked Read 5 Done Sent Done false
35330 9981 Acked Read 1 Done Sent Done false
35331 9982 Acked Flush 0 Done Sent Done false
35332 9983 Acked Read 6 Done Sent Done false
35333 9984 Acked Read 10 Done Sent Done false
35334 9985 Acked Flush 0 Done Sent Done false
35335 9986 Acked Read 9 Done Sent Done false
35336 9987 Acked Read 2 Done Sent Done false
35337 9988 Acked Read 10 Done Sent Done false
35338 9989 Acked Read 9 Done Sent Done false
35339 9990 Acked Write 9 Done Sent Done false
35340 9991 Acked Write 1 Done Sent Done false
35341 9992 Acked Read 1 Done Sent Done false
35342 9993 Acked Flush 0 Done Sent Done false
35343 9994 Acked Write 9 Done Sent Done false
35344 9995 Acked Write 1 Done Sent Done false
35345 9996 Acked Write 4 Done Sent Done false
35346 9997 Acked Write 4 Done Sent Done false
35347 9998 Acked Write 4 Done Sent Done false
35348 9999 Acked Write 5 Done Sent Done false
35349 10000 Acked Read 2 Done Sent Done false
35350 10001 Acked Flush 0 Done Sent Done false
35351 10002 Acked Write 1 Done Sent Done false
35352 10003 Acked Write 7 Done Sent Done false
35353 10004 Acked Flush 0 Done Sent Done false
35354 10005 Acked Read 6 Done Sent Done false
35355 10006 Acked Read 5 Done Sent Done false
35356 10007 Acked Read 2 Done Sent Done false
35357 10008 Acked Read 6 Done Sent Done false
35358 10009 Acked Write 1 Done Sent Done false
35359 10010 Acked Read 5 Done Sent Done false
35360 10011 Acked Read 7 Done Sent Done false
35361 10012 Acked Write 2 Done Sent Done false
35362 10013 Acked Read 3 Done Sent Done false
35363 10014 Acked Read 6 Done Sent Done false
35364 10015 Acked Read 6 Done Sent Done false
35365 10016 Acked Read 5 Done Sent Done false
35366 10017 Acked Write 10 Done Sent Done false
35367 10018 Acked Write 9 Done Sent Done false
35368 10019 Acked Write 1 Done Sent Done false
35369 10020 Acked Read 7 Done Sent Done false
35370 10021 Acked Flush 0 Done Sent Done false
35371 10022 Acked Read 4 Done Sent Done false
35372 10023 Acked Write 2 Done Sent Done false
35373 10024 Acked Write 8 Done Sent Done false
35374 10025 Acked Write 2 Done Sent Done false
35375 10026 Acked Write 4 Done Sent Done false
35376 10027 Acked Flush 0 Done Sent Done false
35377 10028 Acked Read 7 Done Sent Done false
35378 10029 Acked Write 6 Done Sent Done false
35379 10030 Acked Write 7 Done Sent Done false
35380 10031 Acked Read 9 Done Sent Done false
35381 10032 Acked Flush 0 Done Sent Done false
35382 10033 Acked Read 6 Done Sent Done false
35383 10034 Acked Read 10 Done Sent Done false
35384 10035 Acked Read 2 Done Sent Done false
35385 10036 Acked Write 3 Done Sent Done false
35386 10037 Acked Read 5 Done Sent Done false
35387 10038 Acked Write 5 Done Sent Done false
35388 10039 Acked Write 8 Done Sent Done false
35389 10040 Acked Read 6 Done Sent Done false
35390 10041 Acked Write 9 Done Sent Done false
35391 10042 Acked Read 4 Done Sent Done false
35392 10043 Acked Flush 0 Done Sent Done false
35393 10044 Acked Write 5 Done Sent Done false
35394 10045 Acked Write 5 Done Sent Done false
35395 10046 Acked Write 5 Done Sent Done false
35396 10047 Acked Flush 0 Done Sent Done false
35397 10048 Acked Read 4 Done Sent Done false
35398 10049 Acked Read 1 Done Sent Done false
35399 10050 Acked Write 5 Done Sent Done false
35400 10051 Acked Write 1 Done Sent Done false
35401 10052 Acked Read 3 Done Sent Done false
35402 10053 Acked Read 4 Done Sent Done false
35403 10054 Acked Write 10 Done Sent Done false
35404 10055 Acked Read 6 Done Sent Done false
35405 10056 Acked Read 5 Done Sent Done false
35406 10057 Acked Flush 0 Done Sent Done false
35407 STATES DS:0 DS:1 DS:2 TOTAL
35408 Sent 0 301 0 301
35409 Done 301 0 301 602
35410 Skipped 0 0 0 0
35411 Error 0 0 0 0
35412 Last Flush: Some(JobId(10057)) Some(JobId(9756)) Some(JobId(10057))
35413 Downstairs last five completed: 9756 9755 9754 9753 9752
35414 Upstairs last five completed: 10057 10056 10055 10054 10053
35415 CLIENT: Up:0 ds:301 act:2
354162025-02-07T20:15:05.698ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354172025-02-07T20:15:06.699ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354182025-02-07T20:15:06.845ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354192025-02-07T20:15:06.845ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56453) = io task client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354202025-02-07T20:15:06.847ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354212025-02-07T20:15:06.847ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354222025-02-07T20:15:06.847ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354232025-02-07T20:15:06.847ZINFOcrucible: send last flush ID to this DS: Some(JobId(9756)) = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354242025-02-07T20:15:06.847ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354252025-02-07T20:15:06.847ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(9756)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354262025-02-07T20:15:06.847ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(9756)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
35427 ----------------------------------------------------------------
35428 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35429 Downstairs last five completed: 10057 10056 10055 10054 10053
35430 Upstairs last five completed: 10057 10056 10055 10054 10053
35431 CLIENT: Up:0 ds:0 act:3
35432 Replay: All jobs finished
35433 Main task has work to do, go find it
35434 got dsc Stop(2)
35435 stop 2
35436 Feb 07 20:15:09.083 INFO request completed, [8830] Got stop action so:false kr:false
35437 latency_us: 392, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 7f077608-532c-4528-9019-28923ff8ac88, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
354382025-02-07T20:15:09.083ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354392025-02-07T20:15:09.083ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354402025-02-07T20:15:09.083ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
354412025-02-07T20:15:09.084ZINFOcrucible: client task is exiting = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
35442 Feb 07 20:15:09.084 INFO request completed, latency_us: 149, response_code: 200, uri: /state/cid/2, method: GET, req_id: b939207f-4577-49e7-93df-94c6dbe1680b, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
35443 [8830] Exited with: ExitStatus(unix_wait_status(9))
35444 [8830][2] reports Exit
35445 Feb 07 20:15:13.086 INFO request completed, latency_us: 188, response_code: 200, uri: /state/cid/2, method: GET, req_id: f78b6fdb-ffd3-4c18-8199-798ab1dff846, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
35446 001/300 Read block 3411 len 36864
35447 002/300 Read block 3266 len 24576
35448 003/300 Read block 1722 len 4096
35449 004/300 Write block 923 len 32768 data: 4 4 3 3 4 6 7 6
35450 005/300 Write block 2053 len 36864 data: 8 6 6 6 3 2 2 2 2
35451 006/300 Read block 2419 len 32768
35452 007/300 Write block 4611 len 32768 data: 8 6 6 7 7 6 7 8
35453 008/300 Write block 398 len 40960 data: 3 2 2 3 2 3 4 3 3 5
35454 009/300 Write block 4577 len 20480 data: 8 6 5 5 6
35455 010/300 Read block 3686 len 8192
35456 011/300 Read block 3316 len 24576
35457 012/300 Write block 2430 len 40960 data: 5 5 5 5 5 6 6 5 5 4
35458 013/300 Write block 1052 len 20480 data: 8 6 6 6 7
35459 014/300 Read block 190 len 24576
35460 015/300 Flush
35461 016/300 Read block 2475 len 40960
35462 017/300 Read block 4717 len 20480
35463 018/300 Write block 315 len 12288 data: 9 7 7
35464 019/300 Write block 1841 len 4096 data: 6
35465 020/300 Write block 3428 len 28672 data: 5 5 5 5 6 8 7
35466 021/300 Write block 3399 len 24576 data: 9 6 6 5 5 4
35467 022/300 Write block 271 len 12288 data: 6 6 6
35468 023/300 Read block 2443 len 12288
35469 024/300 Write block 3111 len 16384 data: 6 5 5 5
35470 025/300 Write block 1232 len 36864 data: 7 7 7 8 6 5 4 5 5
35471 026/300 Write block 1734 len 16384 data: 5 5 6 7
35472 027/300 Write block 4636 len 8192 data: 8 9
35473 028/300 Write block 520 len 24576 data: 5 5 5 5 4 4
35474 029/300 Read block 1038 len 40960
35475 030/300 Write block 1536 len 40960 data: 3 3 4 5 6 6 6 6 5 5
35476 031/300 Flush
35477 032/300 Write block 2508 len 28672 data: 7 6 7 7 6 6 6
35478 033/300 Write block 582 len 20480 data: 7 8 7 9 9
35479 034/300 Read block 4985 len 16384
35480 035/300 Write block 2115 len 32768 data: 7 7 7 6 6 8 8 7
35481 036/300 Read block 837 len 32768
35482 037/300 Read block 3989 len 28672
35483 038/300 Flush
35484 039/300 Write block 1371 len 36864 data: 5 2 2 3 3 3 4 4 4
35485 040/300 Write block 2294 len 20480 data: 6 6 6 7 8
35486 041/300 Read block 780 len 36864
35487 042/300 Read block 4825 len 32768
35488 043/300 Write block 2091 len 4096 data: 8
35489 044/300 Read block 4915 len 28672
35490 045/300 Write block 263 len 8192 data: 7 7
35491 046/300 Read block 1389 len 32768
35492 047/300 Read block 494 len 32768
35493 048/300 Write block 2779 len 12288 data: 7 7 6
35494 049/300 Read block 3704 len 8192
35495 050/300 Read block 4785 len 24576
35496 051/300 Flush
35497 052/300 Read block 3631 len 8192
35498 053/300 Write block 3535 len 32768 data: 7 6 5 6 7 7 7 8
35499 054/300 Write block 1121 len 12288 data: 6 5 5
35500 055/300 Read block 1339 len 16384
35501 056/300 Write block 1265 len 28672 data: 5 5 4 3 2 3 3
35502 057/300 Read block 2942 len 4096
35503 058/300 Flush
35504 059/300 Write block 4720 len 40960 data: 6 6 6 5 4 4 4 4 3 4
35505 060/300 Read block 468 len 24576
35506 061/300 Write block 3162 len 20480 data: 6 8 8 7 6
35507 062/300 Read block 1172 len 40960
35508 063/300 Write block 4894 len 4096 data: 7
35509 064/300 Read block 50 len 32768
35510 065/300 Write block 2726 len 36864 data: 3 4 4 5 5 5 5 5 5
35511 066/300 Read block 2159 len 12288
35512 067/300 Write block 1456 len 36864 data: 6 6 7 6 4 5 5 3 3
35513 068/300 Read block 1075 len 28672
35514 069/300 Write block 3314 len 8192 data: 6 6
35515 070/300 Read block 2278 len 28672
35516 071/300 Read block 4732 len 36864
35517 072/300 Read block 2471 len 16384
35518 073/300 Read block 1283 len 20480
35519 074/300 Write block 3597 len 36864 data: 4 3 3 6 7 4 4 4 4
35520 075/300 Write block 3734 len 16384 data: 8 7 5 4
35521 076/300 Flush
35522 077/300 Write block 3743 len 12288 data: 4 5 5
35523 078/300 Write block 50 len 36864 data: 6 7 7 6 6 7 5 6 4
35524 079/300 Read block 4488 len 20480
35525 080/300 Read block 1689 len 12288
35526 081/300 Write block 2951 len 36864 data: 5 8 9 7 7 8 7 6 8
35527 082/300 Write block 2456 len 40960 data: 5 7 7 6 7 6 5 7 8 9
35528 083/300 Write block 1589 len 24576 data: 5 5 5 5 5 4
35529 084/300 Read block 916 len 4096
35530 085/300 Write block 2525 len 36864 data: 8 7 7 7 6 4 2 2 2
35531 086/300 Write block 4978 len 36864 data: 7 10 9 6 7 6 6 4 5
35532 087/300 Read block 2753 len 32768
35533 088/300 Write block 3667 len 32768 data: 8 8 9 7 8 7 9 6
35534 089/300 Read block 1700 len 40960
35535 090/300 Write block 3913 len 32768 data: 8 8 6 7 6 7 8 7
35536 091/300 Write block 2203 len 16384 data: 8 8 7 7
35537 092/300 Read block 4489 len 20480
35538 093/300 Read block 1606 len 20480
35539 094/300 Write block 4611 len 16384 data: 9 7 7 8
35540 095/300 Read block 2414 len 32768
35541 096/300 Read block 3932 len 36864
35542 097/300 Read block 2451 len 20480
35543 098/300 Read block 1504 len 12288
35544 099/300 Write block 2410 len 24576 data: 5 5 5 4 4 3
35545 100/300 Read block 3624 len 16384
35546 101/300 Read block 3267 len 40960
35547 102/300 Flush
35548 103/300 Flush
35549 104/300 Read block 4324 len 36864
35550 105/300 Write block 1611 len 4096 data: 6
35551 106/300 Read block 3242 len 4096
35552 107/300 Flush
35553 108/300 Read block 1271 len 4096
35554 109/300 Write block 3457 len 28672 data: 4 4 5 6 4 4 5
35555 110/300 Read block 3419 len 40960
35556 111/300 Read block 3905 len 12288
35557 112/300 Read block 2849 len 24576
35558 113/300 Read block 4324 len 4096
35559 114/300 Read block 2074 len 32768
35560 115/300 Read block 2994 len 8192
35561 116/300 Read block 4548 len 40960
35562 117/300 Flush
35563 118/300 Read block 2856 len 36864
35564 119/300 Read block 3224 len 20480
35565 120/300 Read block 2265 len 40960
35566 121/300 Write block 3732 len 8192 data: 7 7
35567 122/300 Write block 4459 len 40960 data: 5 6 7 8 8 9 10 11 11 8
35568 123/300 Read block 2788 len 36864
35569 124/300 Read block 2034 len 36864
35570 125/300 Flush
35571 126/300 Write block 3521 len 8192 data: 5 5
35572 127/300 Write block 4755 len 16384 data: 10 11 9 6
35573 128/300 Read block 3312 len 12288
35574 129/300 Read block 576 len 24576
35575 130/300 Read block 224 len 24576
35576 131/300 Write block 2462 len 16384 data: 6 8 9 10
35577 132/300 Flush
35578 133/300 Write block 2280 len 4096 data: 5
35579 134/300 Write block 2395 len 20480 data: 7 9 8 7 7
35580 135/300 Flush
35581 136/300 Read block 2943 len 20480
35582 137/300 Write block 2866 len 40960 data: 6 5 7 6 6 5 3 4 4 4
35583 138/300 Read block 1213 len 40960
35584 139/300 Read block 4609 len 8192
35585 140/300 Read block 779 len 36864
35586 141/300 Read block 2816 len 16384
35587 142/300 Read block 3713 len 16384
35588 143/300 Write block 136 len 12288 data: 4 4 4
35589 144/300 Write block 3745 len 36864 data: 6 7 7 7 10 9 9 8 9
35590 145/300 Read block 776 len 28672
35591 146/300 Write block 1172 len 28672 data: 3 3 3 4 4 4 4
35592 147/300 Write block 196 len 8192 data: 4 4
35593 148/300 Read block 3442 len 32768
35594 149/300 Write block 695 len 8192 data: 8 7
35595 150/300 Read block 676 len 20480
35596 151/300 Write block 4636 len 28672 data: 9 10 10 9 9 10 10
35597 152/300 Write block 226 len 8192 data: 8 6
35598 153/300 Read block 4439 len 24576
35599 154/300 Read block 4218 len 28672
35600 155/300 Read block 1870 len 36864
35601 156/300 Read block 906 len 24576
35602 157/300 Read block 3812 len 8192
35603 158/300 Write block 1774 len 16384 data: 3 3 4 6
35604 159/300 Flush
35605 160/300 Read block 1139 len 16384
35606 161/300 Flush
35607 162/300 Write block 3212 len 20480 data: 7 6 5 5 5
35608 163/300 Read block 462 len 20480
35609 164/300 Read block 1503 len 32768
35610 165/300 Read block 3281 len 20480
35611 166/300 Write block 3913 len 8192 data: 9 9
35612 167/300 Read block 1736 len 8192
35613 168/300 Read block 708 len 12288
35614 169/300 Read block 764 len 4096
35615 170/300 Read block 3264 len 8192
35616 171/300 Read block 1142 len 4096
35617 172/300 Write block 2956 len 32768 data: 9 8 7 9 8 8 8 6
35618 173/300 Read block 2832 len 20480
35619 174/300 Read block 1359 len 36864
35620 175/300 Write block 1376 len 4096 data: 4
35621 176/300 Flush
35622 177/300 Write block 4270 len 8192 data: 6 5
35623 178/300 Write block 4751 len 40960 data: 8 9 8 10 11 12 10 7 6 5
35624 179/300 Write block 4490 len 24576 data: 7 7 7 8 8 8
35625 180/300 Read block 4303 len 16384
35626 181/300 Read block 629 len 4096
35627 182/300 Write block 2940 len 28672 data: 6 4 4 4 4 5 5
35628 183/300 Read block 2287 len 36864
35629 184/300 Read block 3400 len 36864
35630 185/300 Read block 3826 len 24576
35631 186/300 Flush
35632 187/300 Read block 841 len 12288
35633 188/300 Write block 297 len 8192 data: 4 5
35634 189/300 Write block 4619 len 8192 data: 8 11
35635 190/300 Read block 2801 len 40960
35636 191/300 Read block 1668 len 28672
35637 192/300 Read block 2810 len 36864
35638 193/300 Write block 1324 len 16384 data: 8 8 7 9
35639 194/300 Write block 4886 len 16384 data: 6 5 5 5
35640 195/300 Read block 2387 len 12288
35641 196/300 Write block 2732 len 12288 data: 6 6 6
35642 197/300 Read block 4968 len 32768
35643 198/300 Write block 3616 len 20480 data: 6 5 6 5 3
35644 199/300 Read block 3705 len 36864
35645 200/300 Read block 4469 len 16384
35646 201/300 Write block 2735 len 36864 data: 9 8 8 7 8 8 6 8 7
35647 202/300 Read block 1562 len 36864
35648 203/300 Read block 760 len 28672
35649 204/300 Read block 3438 len 8192
35650 205/300 Write block 3419 len 28672 data: 5 5 4 4 4 6 5
35651 206/300 Write block 3131 len 36864 data: 7 6 7 6 6 7 7 8 8
35652 207/300 Write block 3799 len 16384 data: 8 8 8 7
35653 208/300 Write block 4719 len 40960 data: 4 7 7 7 6 5 5 5 5 4
35654 209/300 Read block 1444 len 36864
35655 210/300 Read block 2656 len 8192
35656 211/300 Write block 2130 len 12288 data: 5 4 6
35657 212/300 Read block 697 len 20480
35658 213/300 Write block 927 len 40960 data: 5 7 8 7 4 4 4 5 4 4
35659 214/300 Write block 3798 len 24576 data: 6 9 9 9 8 7
35660 215/300 Write block 125 len 24576 data: 6 6 7 5 5 4
35661 216/300 Read block 3612 len 16384
35662 217/300 Flush
35663 218/300 Write block 3437 len 16384 data: 8 8 6 4
35664 219/300 Read block 1409 len 16384
35665 220/300 Write block 3222 len 24576 data: 4 4 5 5 5 5
35666 221/300 Read block 630 len 32768
35667 222/300 Write block 3817 len 20480 data: 11 10 9 7 5
35668 223/300 Read block 3463 len 24576
35669 224/300 Read block 868 len 32768
35670 225/300 Write block 3265 len 40960 data: 8 7 6 4 5 4 5 6 6 7
35671 226/300 Read block 610 len 32768
35672 227/300 Write block 2595 len 20480 data: 9 9 11 9 7
35673 228/300 Read block 2237 len 28672
35674 229/300 Read block 2274 len 32768
35675 230/300 Read block 416 len 8192
35676 231/300 Write block 1143 len 20480 data: 4 3 4 5 4
35677 232/300 Read block 1544 len 40960
35678 233/300 Read block 3646 len 20480
35679 234/300 Write block 3497 len 4096 data: 5
35680 235/300 Write block 3909 len 20480 data: 5 6 7 10 10
35681 236/300 Write block 3260 len 20480 data: 8 9 8 8 7
35682 237/300 Read block 4713 len 24576
35683 238/300 Read block 296 len 36864
35684 239/300 Write block 4116 len 40960 data: 6 6 5 6 6 6 7 6 8 7
35685 240/300 Write block 2595 len 28672 data: 10 10 12 10 8 8 10
35686 241/300 Write block 735 len 24576 data: 7 7 6 7 7 6
35687 242/300 Write block 589 len 16384 data: 7 4 5 5
35688 243/300 Write block 671 len 16384 data: 5 5 4 4
35689 244/300 Read block 3383 len 20480
35690 245/300 Read block 4781 len 8192
35691 246/300 Flush
35692 247/300 Read block 594 len 40960
35693 248/300 Write block 1147 len 20480 data: 5 4 5 3 4
35694 249/300 Read block 4835 len 32768
35695 250/300 Read block 3993 len 8192
35696 251/300 Read block 2561 len 28672
35697 252/300 Write block 91 len 20480 data: 4 7 8 7 6
35698 253/300 Read block 4077 len 12288
35699 254/300 Read block 2020 len 32768
35700 255/300 Write block 1172 len 36864 data: 4 4 4 5 5 5 5 5 5
35701 256/300 Read block 1917 len 16384
35702 257/300 Write block 41 len 12288 data: 8 9 7
35703 258/300 Write block 4751 len 32768 data: 9 10 9 11 12 13 11 8
35704 259/300 Write block 3537 len 36864 data: 6 7 8 8 8 9 7 5 4
35705 260/300 Write block 2959 len 36864 data: 10 9 9 9 7 6 6 4 7
35706 261/300 Write block 3688 len 4096 data: 9
35707 262/300 Read block 1524 len 20480
35708 263/300 Flush
35709 264/300 Write block 4372 len 40960 data: 7 7 7 8 6 6 6 5 4 4
35710 265/300 Read block 2222 len 12288
35711 266/300 Read block 3725 len 28672
35712 267/300 Read block 1356 len 40960
35713 268/300 Write block 1170 len 16384 data: 4 4 5 5
35714 269/300 Read block 2963 len 28672
35715 270/300 Read block 771 len 28672
35716 271/300 Write block 387 len 32768 data: 7 7 8 8 7 5 5 4
35717 272/300 Read block 3682 len 20480
35718 273/300 Write block 2898 len 36864 data: 5 5 6 6 5 4 6 5 7
35719 274/300 Write block 3103 len 20480 data: 3 4 7 7 7
35720 275/300 Write block 1179 len 12288 data: 6 6 5
35721 276/300 Read block 410 len 20480
35722 277/300 Write block 2530 len 36864 data: 5 3 3 3 2 2 3 5 7
35723 278/300 Read block 4365 len 8192
35724 279/300 Read block 2148 len 4096
35725 280/300 Read block 1836 len 28672
35726 281/300 Read block 106 len 20480
35727 282/300 Flush
35728 283/300 Flush
35729 284/300 Read block 3277 len 12288
35730 285/300 Write block 485 len 4096 data: 4
35731 286/300 Read block 1696 len 20480
35732 287/300 Write block 2862 len 16384 data: 9 8 7 6
35733 288/300 Write block 4232 len 32768 data: 4 4 7 5 7 8 9 8
35734 289/300 Read block 1334 len 4096
35735 290/300 Read block 788 len 28672
35736 291/300 Read block 4720 len 20480
35737 292/300 Write block 1104 len 12288 data: 7 8 7
35738 293/300 Read block 2315 len 4096
35739 294/300 Read block 4782 len 32768
35740 295/300 Write block 3219 len 36864 data: 6 7 5 5 5 6 6 6 6
35741 296/300 Read block 2433 len 36864
35742 297/300 Read block 4393 len 32768
35743 298/300 Read block 21 len 24576
35744 299/300 Read block 4093 len 24576
35745 300/300 Write block 326 len 12288 data: 6 6 5
35746 301/300 Flush
35747 Main task has work to do, go find it
35748 got dsc Start(2)
35749 start 2
35750 Feb 07 20:15:13.338 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5ca3fa35-9c81-4de1-9b61-413425aa47d0, remote_addr: 127.0.0.1:43825, local_addr: 127.0.0.1:9998
35751 Replay: started 2, returned:Ok(())
35752 ----------------------------------------------------------------
35753 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35754 States: Active Active Offline
35755 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35756 10058 Acked Read 9 Done Done Sent false
35757 10059 Acked Read 6 Done Done Sent false
35758 10060 Acked Read 1 Done Done Sent false
35759 10061 Acked Write 8 Done Done Sent false
35760 10062 Acked Write 9 Done Done Sent false
35761 10063 Acked Read 8 Done Done Sent false
35762 10064 Acked Write 8 Done Done Sent false
35763 10065 Acked Write 10 Done Done Sent false
35764 10066 Acked Write 5 Done Done Sent false
35765 10067 Acked Read 2 Done Done Sent false
35766 10068 Acked Read 6 Done Done Sent false
35767 10069 Acked Write 10 Done Done Sent false
35768 10070 Acked Write 5 Done Done Sent false
35769 10071 Acked Read 6 Done Done Sent false
35770 10072 Acked Flush 0 Done Done Sent false
35771 10073 Acked Read 10 Done Done Sent false
35772 10074 Acked Read 5 Done Done Sent false
35773 10075 Acked Write 3 Done Done Sent false
35774 10076 Acked Write 1 Done Done Sent false
35775 10077 Acked Write 7 Done Done Sent false
35776 10078 Acked Write 6 Done Done Sent false
35777 10079 Acked Write 3 Done Done Sent false
35778 10080 Acked Read 3 Done Done Sent false
35779 10081 Acked Write 4 Done Done Sent false
35780 10082 Acked Write 9 Done Done Sent false
35781 10083 Acked Write 4 Done Done Sent false
35782 10084 Acked Write 2 Done Done Sent false
35783 10085 Acked Write 6 Done Done Sent false
35784 10086 Acked Read 10 Done Done Sent false
35785 10087 Acked Write 10 Done Done Sent false
35786 10088 Acked Flush 0 Done Done Sent false
35787 10089 Acked Write 7 Done Done Sent false
35788 10090 Acked Write 5 Done Done Sent false
35789 10091 Acked Read 4 Done Done Sent false
35790 10092 Acked Write 8 Done Done Sent false
35791 10093 Acked Read 8 Done Done Sent false
35792 10094 Acked Read 7 Done Done Sent false
35793 10095 Acked Flush 0 Done Done Sent false
35794 10096 Acked Write 9 Done Done Sent false
35795 10097 Acked Write 5 Done Done Sent false
35796 10098 Acked Read 9 Done Done Sent false
35797 10099 Acked Read 8 Done Done Sent false
35798 10100 Acked Write 1 Done Done Sent false
35799 10101 Acked Read 7 Done Done Sent false
35800 10102 Acked Write 2 Done Done Sent false
35801 10103 Acked Read 8 Done Done Sent false
35802 10104 Acked Read 8 Done Done Sent false
35803 10105 Acked Write 3 Done Done Sent false
35804 10106 Acked Read 2 Done Done Sent false
35805 10107 Acked Read 6 Done Done Sent false
35806 10108 Acked Flush 0 Done Done Sent false
35807 10109 Acked Read 2 Done Done Sent false
35808 10110 Acked Write 8 Done Done Sent false
35809 10111 Acked Write 3 Done Done Sent false
35810 10112 Acked Read 4 Done Done Sent false
35811 10113 Acked Write 7 Done Done Sent false
35812 10114 Acked Read 1 Done Done Sent false
35813 10115 Acked Flush 0 Done Done Sent false
35814 10116 Acked Write 10 Done Done Sent false
35815 10117 Acked Read 6 Done Done Sent false
35816 10118 Acked Write 5 Done Done Sent false
35817 10119 Acked Read 10 Done Done Sent false
35818 10120 Acked Write 1 Done Done Sent false
35819 10121 Acked Read 8 Done Done Sent false
35820 10122 Acked Write 9 Done Done Sent false
35821 10123 Acked Read 3 Done Done Sent false
35822 10124 Acked Write 9 Done Done Sent false
35823 10125 Acked Read 7 Done Done Sent false
35824 10126 Acked Write 2 Done Done Sent false
35825 10127 Acked Read 7 Done Done Sent false
35826 10128 Acked Read 9 Done Done Sent false
35827 10129 Acked Read 4 Done Done Sent false
35828 10130 Acked Read 5 Done Done Sent false
35829 10131 Acked Write 9 Done Done Sent false
35830 10132 Acked Write 4 Done Done Sent false
35831 10133 Acked Flush 0 Done Done Sent false
35832 10134 Acked Write 3 Done Done Sent false
35833 10135 Acked Write 9 Done Done Sent false
35834 10136 Acked Read 5 Done Done Sent false
35835 10137 Acked Read 3 Done Done Sent false
35836 10138 Acked Write 9 Done Done Sent false
35837 10139 Acked Write 10 Done Done Sent false
35838 10140 Acked Write 6 Done Done Sent false
35839 10141 Acked Read 1 Done Done Sent false
35840 10142 Acked Write 9 Done Done Sent false
35841 10143 Acked Write 9 Done Done Sent false
35842 10144 Acked Read 8 Done Done Sent false
35843 10145 Acked Write 8 Done Done Sent false
35844 10146 Acked Read 10 Done Done Sent false
35845 10147 Acked Write 8 Done Done Sent false
35846 10148 Acked Write 4 Done Done Sent false
35847 10149 Acked Read 5 Done Done Sent false
35848 10150 Acked Read 5 Done Done Sent false
35849 10151 Acked Write 4 Done Done Sent false
35850 10152 Acked Read 8 Done Done Sent false
35851 10153 Acked Read 9 Done Done Sent false
35852 10154 Acked Read 5 Done Done Sent false
35853 10155 Acked Read 3 Done Done Sent false
35854 10156 Acked Write 6 Done Done Sent false
35855 10157 Acked Read 4 Done Done Sent false
35856 10158 Acked Read 10 Done Done Sent false
35857 10159 Acked Flush 0 Done Done Sent false
35858 10160 Acked Flush 0 Done Done Sent false
35859 10161 Acked Read 9 Done Done Sent false
35860 10162 Acked Write 1 Done Done Sent false
35861 10163 Acked Read 1 Done Done Sent false
35862 10164 Acked Flush 0 Done Done Sent false
35863 10165 Acked Read 1 Done Done Sent false
35864 10166 Acked Write 7 Done Done Sent false
35865 10167 Acked Read 10 Done Done Sent false
35866 10168 Acked Read 3 Done Done Sent false
35867 10169 Acked Read 6 Done Done Sent false
35868 10170 Acked Read 1 Done Done Sent false
35869 10171 Acked Read 8 Done Done Sent false
35870 10172 Acked Read 2 Done Done Sent false
35871 10173 Acked Read 10 Done Done Sent false
35872 10174 Acked Flush 0 Done Done Sent false
35873 10175 Acked Read 9 Done Done Sent false
35874 10176 Acked Read 5 Done Done Sent false
35875 10177 Acked Read 10 Done Done Sent false
35876 10178 Acked Write 2 Done Done Sent false
35877 10179 Acked Write 10 Done Done Sent false
35878 10180 Acked Read 9 Done Done Sent false
35879 10181 Acked Read 9 Done Done Sent false
35880 10182 Acked Flush 0 Done Done Sent false
35881 10183 Acked Write 2 Done Done Sent false
35882 10184 Acked Write 4 Done Done Sent false
35883 10185 Acked Read 3 Done Done Sent false
35884 10186 Acked Read 6 Done Done Sent false
35885 10187 Acked Read 6 Done Done Sent false
35886 10188 Acked Write 4 Done Done Sent false
35887 10189 Acked Flush 0 Done Done Sent false
35888 10190 Acked Write 1 Done Done Sent false
35889 10191 Acked Write 5 Done Done Sent false
35890 10192 Acked Flush 0 Done Done Sent false
35891 10193 Acked Read 5 Done Done Sent false
35892 10194 Acked Write 10 Done Done Sent false
35893 10195 Acked Read 10 Done Done Sent false
35894 10196 Acked Read 2 Done Done Sent false
35895 10197 Acked Read 9 Done Done Sent false
35896 10198 Acked Read 4 Done Done Sent false
35897 10199 Acked Read 4 Done Done Sent false
35898 10200 Acked Write 3 Done Done Sent false
35899 10201 Acked Write 9 Done Done Sent false
35900 10202 Acked Read 7 Done Done Sent false
35901 10203 Acked Write 7 Done Done Sent false
35902 10204 Acked Write 2 Done Done Sent false
35903 10205 Acked Read 8 Done Done Sent false
35904 10206 Acked Write 2 Done Done Sent false
35905 10207 Acked Read 5 Done Done Sent false
35906 10208 Acked Write 7 Done Done Sent false
35907 10209 Acked Write 2 Done Done Sent false
35908 10210 Acked Read 6 Done Done Sent false
35909 10211 Acked Read 7 Done Done Sent false
35910 10212 Acked Read 9 Done Done Sent false
35911 10213 Acked Read 6 Done Done Sent false
35912 10214 Acked Read 2 Done Done Sent false
35913 10215 Acked Write 4 Done Done Sent false
35914 10216 Acked Flush 0 Done Done Sent false
35915 10217 Acked Read 4 Done Done Sent false
35916 10218 Acked Flush 0 Done Done Sent false
35917 10219 Acked Write 5 Done Done Sent false
35918 10220 Acked Read 5 Done Done Sent false
35919 10221 Acked Read 8 Done Done Sent false
35920 10222 Acked Read 5 Done Done Sent false
35921 10223 Acked Write 2 Done Done Sent false
35922 10224 Acked Read 2 Done Done Sent false
35923 10225 Acked Read 3 Done Done Sent false
35924 10226 Acked Read 1 Done Done Sent false
35925 10227 Acked Read 2 Done Done Sent false
35926 10228 Acked Read 1 Done Done Sent false
35927 10229 Acked Write 8 Done Done Sent false
35928 10230 Acked Read 5 Done Done Sent false
35929 10231 Acked Read 9 Done Done Sent false
35930 10232 Acked Write 1 Done Done Sent false
35931 10233 Acked Flush 0 Done Done Sent false
35932 10234 Acked Write 2 Done Done Sent false
35933 10235 Acked Write 10 Done Done Sent false
35934 10236 Acked Write 6 Done Done Sent false
35935 10237 Acked Read 4 Done Done Sent false
35936 10238 Acked Read 1 Done Done Sent false
35937 10239 Acked Write 7 Done Done Sent false
35938 10240 Acked Read 9 Done Done Sent false
35939 10241 Acked Read 9 Done Done Sent false
35940 10242 Acked Read 6 Done Done Sent false
35941 10243 Acked Flush 0 Done Done Sent false
35942 10244 Acked Read 3 Done Done Sent false
35943 10245 Acked Write 2 Done Done Sent false
35944 10246 Acked Write 2 Done Done Sent false
35945 10247 Acked Read 10 Done Done Sent false
35946 10248 Acked Read 7 Done Done Sent false
35947 10249 Acked Read 9 Done Done Sent false
35948 10250 Acked Write 4 Done Done Sent false
35949 10251 Acked Write 4 Done Done Sent false
35950 10252 Acked Read 3 Done Done Sent false
35951 10253 Acked Write 3 Done Done Sent false
35952 10254 Acked Read 8 Done Done Sent false
35953 10255 Acked Write 5 Done Done Sent false
35954 10256 Acked Read 9 Done Done Sent false
35955 10257 Acked Read 4 Done Done Sent false
35956 10258 Acked Write 9 Done Done Sent false
35957 10259 Acked Read 9 Done Done Sent false
35958 10260 Acked Read 7 Done Done Sent false
35959 10261 Acked Read 2 Done Done Sent false
35960 10262 Acked Write 7 Done Done Sent false
35961 10263 Acked Write 9 Done Done Sent false
35962 10264 Acked Write 4 Done Done Sent false
35963 10265 Acked Write 10 Done Done Sent false
35964 10266 Acked Read 9 Done Done Sent false
35965 10267 Acked Read 2 Done Done Sent false
35966 10268 Acked Write 3 Done Done Sent false
35967 10269 Acked Read 5 Done Done Sent false
35968 10270 Acked Write 10 Done Done Sent false
35969 10271 Acked Write 6 Done Done Sent false
35970 10272 Acked Write 6 Done Done Sent false
35971 10273 Acked Read 4 Done Done Sent false
35972 10274 Acked Flush 0 Done Done Sent false
35973 10275 Acked Write 4 Done Done Sent false
35974 10276 Acked Read 4 Done Done Sent false
35975 10277 Acked Write 6 Done Done Sent false
35976 10278 Acked Read 8 Done Done Sent false
35977 10279 Acked Write 5 Done Done Sent false
35978 10280 Acked Read 6 Done Done Sent false
35979 10281 Acked Read 8 Done Done Sent false
35980 10282 Acked Write 10 Done Done Sent false
35981 10283 Acked Read 8 Done Done Sent false
35982 10284 Acked Write 5 Done Done Sent false
35983 10285 Acked Read 7 Done Done Sent false
35984 10286 Acked Read 8 Done Done Sent false
35985 10287 Acked Read 2 Done Done Sent false
35986 10288 Acked Write 5 Done Done Sent false
35987 10289 Acked Read 10 Done Done Sent false
35988 10290 Acked Read 5 Done Done Sent false
35989 10291 Acked Write 1 Done Done Sent false
35990 10292 Acked Write 5 Done Done Sent false
35991 10293 Acked Write 5 Done Done Sent false
35992 10294 Acked Read 6 Done Done Sent false
35993 10295 Acked Read 9 Done Done Sent false
35994 10296 Acked Write 10 Done Done Sent false
35995 10297 Acked Write 7 Done Done Sent false
35996 10298 Acked Write 6 Done Done Sent false
35997 10299 Acked Write 4 Done Done Sent false
35998 10300 Acked Write 4 Done Done Sent false
35999 10301 Acked Read 5 Done Done Sent false
36000 10302 Acked Read 2 Done Done Sent false
36001 10303 Acked Flush 0 Done Done Sent false
36002 10304 Acked Read 10 Done Done Sent false
36003 10305 Acked Write 5 Done Done Sent false
36004 10306 Acked Read 8 Done Done Sent false
36005 10307 Acked Read 2 Done Done Sent false
36006 10308 Acked Read 7 Done Done Sent false
36007 10309 Acked Write 5 Done Done Sent false
36008 10310 Acked Read 3 Done Done Sent false
36009 10311 Acked Read 8 Done Done Sent false
36010 10312 Acked Write 9 Done Done Sent false
36011 10313 Acked Read 4 Done Done Sent false
36012 10314 Acked Write 3 Done Done Sent false
36013 10315 Acked Write 8 Done Done Sent false
36014 10316 Acked Write 9 Done Done Sent false
36015 10317 Acked Write 9 Done Done Sent false
36016 10318 Acked Write 1 Done Done Sent false
36017 10319 Acked Read 5 Done Done Sent false
36018 10320 Acked Flush 0 Done Done Sent false
36019 10321 Acked Write 10 Done Done Sent false
36020 10322 Acked Read 3 Done Done Sent false
36021 10323 Acked Read 7 Done Done Sent false
36022 10324 Acked Read 10 Done Done Sent false
36023 10325 Acked Write 4 Done Done Sent false
36024 10326 Acked Read 7 Done Done Sent false
36025 10327 Acked Read 7 Done Done Sent false
36026 10328 Acked Write 8 Done Done Sent false
36027 10329 Acked Read 5 Done Done Sent false
36028 10330 Acked Write 9 Done Done Sent false
36029 10331 Acked Write 5 Done Done Sent false
36030 10332 Acked Write 3 Done Done Sent false
36031 10333 Acked Read 5 Done Done Sent false
36032 10334 Acked Write 9 Done Done Sent false
36033 10335 Acked Read 2 Done Done Sent false
36034 10336 Acked Read 1 Done Done Sent false
36035 10337 Acked Read 7 Done Done Sent false
36036 10338 Acked Read 5 Done Done Sent false
36037 10339 Acked Flush 0 Done Done Sent false
36038 10340 Acked Flush 0 Done Done Sent false
36039 10341 Acked Read 3 Done Done Sent false
36040 10342 Acked Write 1 Done Done Sent false
36041 10343 Acked Read 5 Done Done Sent false
36042 10344 Acked Write 4 Done Done Sent false
36043 10345 Acked Write 8 Done Done Sent false
36044 10346 Acked Read 1 Done Done Sent false
36045 10347 Acked Read 7 Done Done Sent false
36046 10348 Acked Read 5 Done Done Sent false
36047 10349 Acked Write 3 Done Done Sent false
36048 10350 Acked Read 1 Done Done Sent false
36049 10351 Acked Read 8 Done Done Sent false
36050 10352 Acked Write 9 Done Done Sent false
36051 10353 Acked Read 9 Done Done Sent false
36052 10354 Acked Read 8 Done Done Sent false
36053 10355 Acked Read 6 Done Done Sent false
36054 10356 Acked Read 6 Done Done Sent false
36055 10357 Acked Write 3 Done Done Sent false
36056 10358 Acked Flush 0 Done Done Sent false
36057 STATES DS:0 DS:1 DS:2 TOTAL
36058 Sent 0 0 301 301
36059 Done 301 301 0 602
36060 Skipped 0 0 0 0
36061 Error 0 0 0 0
36062 Last Flush: Some(JobId(10358)) Some(JobId(10358)) Some(JobId(10057))
36063 Downstairs last five completed: 10057 10056 10055 10054 10053
36064 Upstairs last five completed: 10358 10357 10356 10355 10354
36065 CLIENT: Up:0 ds:301 act:2
360662025-02-07T20:15:13.840ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
360672025-02-07T20:15:14.712ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
36068 [8830] got start action
36069 [8830] I am going to restart
36070 Starting downstairs at port 8830
36071 Make output file at "/tmp/dsc/downstairs-8830.txt"
36072 [8830][2] reports Starting
36073 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1240)
36074 [8830][2] reports Running
360752025-02-07T20:15:15.713ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
360762025-02-07T20:15:16.714ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
36077 ----------------------------------------------------------------
36078 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
36079 States: Active Active Offline
36080 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36081 10058 Acked Read 9 Done Done Sent false
36082 10059 Acked Read 6 Done Done Sent false
36083 10060 Acked Read 1 Done Done Sent false
36084 10061 Acked Write 8 Done Done Sent false
36085 10062 Acked Write 9 Done Done Sent false
36086 10063 Acked Read 8 Done Done Sent false
36087 10064 Acked Write 8 Done Done Sent false
36088 10065 Acked Write 10 Done Done Sent false
36089 10066 Acked Write 5 Done Done Sent false
36090 10067 Acked Read 2 Done Done Sent false
36091 10068 Acked Read 6 Done Done Sent false
36092 10069 Acked Write 10 Done Done Sent false
36093 10070 Acked Write 5 Done Done Sent false
36094 10071 Acked Read 6 Done Done Sent false
36095 10072 Acked Flush 0 Done Done Sent false
36096 10073 Acked Read 10 Done Done Sent false
36097 10074 Acked Read 5 Done Done Sent false
36098 10075 Acked Write 3 Done Done Sent false
36099 10076 Acked Write 1 Done Done Sent false
36100 10077 Acked Write 7 Done Done Sent false
36101 10078 Acked Write 6 Done Done Sent false
36102 10079 Acked Write 3 Done Done Sent false
36103 10080 Acked Read 3 Done Done Sent false
36104 10081 Acked Write 4 Done Done Sent false
36105 10082 Acked Write 9 Done Done Sent false
36106 10083 Acked Write 4 Done Done Sent false
36107 10084 Acked Write 2 Done Done Sent false
36108 10085 Acked Write 6 Done Done Sent false
36109 10086 Acked Read 10 Done Done Sent false
36110 10087 Acked Write 10 Done Done Sent false
36111 10088 Acked Flush 0 Done Done Sent false
36112 10089 Acked Write 7 Done Done Sent false
36113 10090 Acked Write 5 Done Done Sent false
36114 10091 Acked Read 4 Done Done Sent false
36115 10092 Acked Write 8 Done Done Sent false
36116 10093 Acked Read 8 Done Done Sent false
36117 10094 Acked Read 7 Done Done Sent false
36118 10095 Acked Flush 0 Done Done Sent false
36119 10096 Acked Write 9 Done Done Sent false
36120 10097 Acked Write 5 Done Done Sent false
36121 10098 Acked Read 9 Done Done Sent false
36122 10099 Acked Read 8 Done Done Sent false
36123 10100 Acked Write 1 Done Done Sent false
36124 10101 Acked Read 7 Done Done Sent false
36125 10102 Acked Write 2 Done Done Sent false
36126 10103 Acked Read 8 Done Done Sent false
36127 10104 Acked Read 8 Done Done Sent false
36128 10105 Acked Write 3 Done Done Sent false
36129 10106 Acked Read 2 Done Done Sent false
36130 10107 Acked Read 6 Done Done Sent false
36131 10108 Acked Flush 0 Done Done Sent false
36132 10109 Acked Read 2 Done Done Sent false
36133 10110 Acked Write 8 Done Done Sent false
36134 10111 Acked Write 3 Done Done Sent false
36135 10112 Acked Read 4 Done Done Sent false
36136 10113 Acked Write 7 Done Done Sent false
36137 10114 Acked Read 1 Done Done Sent false
36138 10115 Acked Flush 0 Done Done Sent false
36139 10116 Acked Write 10 Done Done Sent false
36140 10117 Acked Read 6 Done Done Sent false
36141 10118 Acked Write 5 Done Done Sent false
36142 10119 Acked Read 10 Done Done Sent false
36143 10120 Acked Write 1 Done Done Sent false
36144 10121 Acked Read 8 Done Done Sent false
36145 10122 Acked Write 9 Done Done Sent false
36146 10123 Acked Read 3 Done Done Sent false
36147 10124 Acked Write 9 Done Done Sent false
36148 10125 Acked Read 7 Done Done Sent false
36149 10126 Acked Write 2 Done Done Sent false
36150 10127 Acked Read 7 Done Done Sent false
36151 10128 Acked Read 9 Done Done Sent false
36152 10129 Acked Read 4 Done Done Sent false
36153 10130 Acked Read 5 Done Done Sent false
36154 10131 Acked Write 9 Done Done Sent false
36155 10132 Acked Write 4 Done Done Sent false
36156 10133 Acked Flush 0 Done Done Sent false
36157 10134 Acked Write 3 Done Done Sent false
36158 10135 Acked Write 9 Done Done Sent false
36159 10136 Acked Read 5 Done Done Sent false
36160 10137 Acked Read 3 Done Done Sent false
36161 10138 Acked Write 9 Done Done Sent false
36162 10139 Acked Write 10 Done Done Sent false
36163 10140 Acked Write 6 Done Done Sent false
36164 10141 Acked Read 1 Done Done Sent false
36165 10142 Acked Write 9 Done Done Sent false
36166 10143 Acked Write 9 Done Done Sent false
36167 10144 Acked Read 8 Done Done Sent false
36168 10145 Acked Write 8 Done Done Sent false
36169 10146 Acked Read 10 Done Done Sent false
36170 10147 Acked Write 8 Done Done Sent false
36171 10148 Acked Write 4 Done Done Sent false
36172 10149 Acked Read 5 Done Done Sent false
36173 10150 Acked Read 5 Done Done Sent false
36174 10151 Acked Write 4 Done Done Sent false
36175 10152 Acked Read 8 Done Done Sent false
36176 10153 Acked Read 9 Done Done Sent false
36177 10154 Acked Read 5 Done Done Sent false
36178 10155 Acked Read 3 Done Done Sent false
36179 10156 Acked Write 6 Done Done Sent false
36180 10157 Acked Read 4 Done Done Sent false
36181 10158 Acked Read 10 Done Done Sent false
36182 10159 Acked Flush 0 Done Done Sent false
36183 10160 Acked Flush 0 Done Done Sent false
36184 10161 Acked Read 9 Done Done Sent false
36185 10162 Acked Write 1 Done Done Sent false
36186 10163 Acked Read 1 Done Done Sent false
36187 10164 Acked Flush 0 Done Done Sent false
36188 10165 Acked Read 1 Done Done Sent false
36189 10166 Acked Write 7 Done Done Sent false
36190 10167 Acked Read 10 Done Done Sent false
36191 10168 Acked Read 3 Done Done Sent false
36192 10169 Acked Read 6 Done Done Sent false
36193 10170 Acked Read 1 Done Done Sent false
36194 10171 Acked Read 8 Done Done Sent false
36195 10172 Acked Read 2 Done Done Sent false
36196 10173 Acked Read 10 Done Done Sent false
36197 10174 Acked Flush 0 Done Done Sent false
36198 10175 Acked Read 9 Done Done Sent false
36199 10176 Acked Read 5 Done Done Sent false
36200 10177 Acked Read 10 Done Done Sent false
36201 10178 Acked Write 2 Done Done Sent false
36202 10179 Acked Write 10 Done Done Sent false
36203 10180 Acked Read 9 Done Done Sent false
36204 10181 Acked Read 9 Done Done Sent false
36205 10182 Acked Flush 0 Done Done Sent false
36206 10183 Acked Write 2 Done Done Sent false
36207 10184 Acked Write 4 Done Done Sent false
36208 10185 Acked Read 3 Done Done Sent false
36209 10186 Acked Read 6 Done Done Sent false
36210 10187 Acked Read 6 Done Done Sent false
36211 10188 Acked Write 4 Done Done Sent false
36212 10189 Acked Flush 0 Done Done Sent false
36213 10190 Acked Write 1 Done Done Sent false
36214 10191 Acked Write 5 Done Done Sent false
36215 10192 Acked Flush 0 Done Done Sent false
36216 10193 Acked Read 5 Done Done Sent false
36217 10194 Acked Write 10 Done Done Sent false
36218 10195 Acked Read 10 Done Done Sent false
36219 10196 Acked Read 2 Done Done Sent false
36220 10197 Acked Read 9 Done Done Sent false
36221 10198 Acked Read 4 Done Done Sent false
36222 10199 Acked Read 4 Done Done Sent false
36223 10200 Acked Write 3 Done Done Sent false
36224 10201 Acked Write 9 Done Done Sent false
36225 10202 Acked Read 7 Done Done Sent false
36226 10203 Acked Write 7 Done Done Sent false
36227 10204 Acked Write 2 Done Done Sent false
36228 10205 Acked Read 8 Done Done Sent false
36229 10206 Acked Write 2 Done Done Sent false
36230 10207 Acked Read 5 Done Done Sent false
36231 10208 Acked Write 7 Done Done Sent false
36232 10209 Acked Write 2 Done Done Sent false
36233 10210 Acked Read 6 Done Done Sent false
36234 10211 Acked Read 7 Done Done Sent false
36235 10212 Acked Read 9 Done Done Sent false
36236 10213 Acked Read 6 Done Done Sent false
36237 10214 Acked Read 2 Done Done Sent false
36238 10215 Acked Write 4 Done Done Sent false
36239 10216 Acked Flush 0 Done Done Sent false
36240 10217 Acked Read 4 Done Done Sent false
36241 10218 Acked Flush 0 Done Done Sent false
36242 10219 Acked Write 5 Done Done Sent false
36243 10220 Acked Read 5 Done Done Sent false
36244 10221 Acked Read 8 Done Done Sent false
36245 10222 Acked Read 5 Done Done Sent false
36246 10223 Acked Write 2 Done Done Sent false
36247 10224 Acked Read 2 Done Done Sent false
36248 10225 Acked Read 3 Done Done Sent false
36249 10226 Acked Read 1 Done Done Sent false
36250 10227 Acked Read 2 Done Done Sent false
36251 10228 Acked Read 1 Done Done Sent false
36252 10229 Acked Write 8 Done Done Sent false
36253 10230 Acked Read 5 Done Done Sent false
36254 10231 Acked Read 9 Done Done Sent false
36255 10232 Acked Write 1 Done Done Sent false
36256 10233 Acked Flush 0 Done Done Sent false
36257 10234 Acked Write 2 Done Done Sent false
36258 10235 Acked Write 10 Done Done Sent false
36259 10236 Acked Write 6 Done Done Sent false
36260 10237 Acked Read 4 Done Done Sent false
36261 10238 Acked Read 1 Done Done Sent false
36262 10239 Acked Write 7 Done Done Sent false
36263 10240 Acked Read 9 Done Done Sent false
36264 10241 Acked Read 9 Done Done Sent false
36265 10242 Acked Read 6 Done Done Sent false
36266 10243 Acked Flush 0 Done Done Sent false
36267 10244 Acked Read 3 Done Done Sent false
36268 10245 Acked Write 2 Done Done Sent false
36269 10246 Acked Write 2 Done Done Sent false
36270 10247 Acked Read 10 Done Done Sent false
36271 10248 Acked Read 7 Done Done Sent false
36272 10249 Acked Read 9 Done Done Sent false
36273 10250 Acked Write 4 Done Done Sent false
36274 10251 Acked Write 4 Done Done Sent false
36275 10252 Acked Read 3 Done Done Sent false
36276 10253 Acked Write 3 Done Done Sent false
36277 10254 Acked Read 8 Done Done Sent false
36278 10255 Acked Write 5 Done Done Sent false
36279 10256 Acked Read 9 Done Done Sent false
36280 10257 Acked Read 4 Done Done Sent false
36281 10258 Acked Write 9 Done Done Sent false
36282 10259 Acked Read 9 Done Done Sent false
36283 10260 Acked Read 7 Done Done Sent false
36284 10261 Acked Read 2 Done Done Sent false
36285 10262 Acked Write 7 Done Done Sent false
36286 10263 Acked Write 9 Done Done Sent false
36287 10264 Acked Write 4 Done Done Sent false
36288 10265 Acked Write 10 Done Done Sent false
36289 10266 Acked Read 9 Done Done Sent false
36290 10267 Acked Read 2 Done Done Sent false
36291 10268 Acked Write 3 Done Done Sent false
36292 10269 Acked Read 5 Done Done Sent false
36293 10270 Acked Write 10 Done Done Sent false
36294 10271 Acked Write 6 Done Done Sent false
36295 10272 Acked Write 6 Done Done Sent false
36296 10273 Acked Read 4 Done Done Sent false
36297 10274 Acked Flush 0 Done Done Sent false
36298 10275 Acked Write 4 Done Done Sent false
36299 10276 Acked Read 4 Done Done Sent false
36300 10277 Acked Write 6 Done Done Sent false
36301 10278 Acked Read 8 Done Done Sent false
36302 10279 Acked Write 5 Done Done Sent false
36303 10280 Acked Read 6 Done Done Sent false
36304 10281 Acked Read 8 Done Done Sent false
36305 10282 Acked Write 10 Done Done Sent false
36306 10283 Acked Read 8 Done Done Sent false
36307 10284 Acked Write 5 Done Done Sent false
36308 10285 Acked Read 7 Done Done Sent false
36309 10286 Acked Read 8 Done Done Sent false
36310 10287 Acked Read 2 Done Done Sent false
36311 10288 Acked Write 5 Done Done Sent false
36312 10289 Acked Read 10 Done Done Sent false
36313 10290 Acked Read 5 Done Done Sent false
36314 10291 Acked Write 1 Done Done Sent false
36315 10292 Acked Write 5 Done Done Sent false
36316 10293 Acked Write 5 Done Done Sent false
36317 10294 Acked Read 6 Done Done Sent false
36318 10295 Acked Read 9 Done Done Sent false
36319 10296 Acked Write 10 Done Done Sent false
36320 10297 Acked Write 7 Done Done Sent false
36321 10298 Acked Write 6 Done Done Sent false
36322 10299 Acked Write 4 Done Done Sent false
36323 10300 Acked Write 4 Done Done Sent false
36324 10301 Acked Read 5 Done Done Sent false
36325 10302 Acked Read 2 Done Done Sent false
36326 10303 Acked Flush 0 Done Done Sent false
36327 10304 Acked Read 10 Done Done Sent false
36328 10305 Acked Write 5 Done Done Sent false
36329 10306 Acked Read 8 Done Done Sent false
36330 10307 Acked Read 2 Done Done Sent false
36331 10308 Acked Read 7 Done Done Sent false
36332 10309 Acked Write 5 Done Done Sent false
36333 10310 Acked Read 3 Done Done Sent false
36334 10311 Acked Read 8 Done Done Sent false
36335 10312 Acked Write 9 Done Done Sent false
36336 10313 Acked Read 4 Done Done Sent false
36337 10314 Acked Write 3 Done Done Sent false
36338 10315 Acked Write 8 Done Done Sent false
36339 10316 Acked Write 9 Done Done Sent false
36340 10317 Acked Write 9 Done Done Sent false
36341 10318 Acked Write 1 Done Done Sent false
36342 10319 Acked Read 5 Done Done Sent false
36343 10320 Acked Flush 0 Done Done Sent false
36344 10321 Acked Write 10 Done Done Sent false
36345 10322 Acked Read 3 Done Done Sent false
36346 10323 Acked Read 7 Done Done Sent false
36347 10324 Acked Read 10 Done Done Sent false
36348 10325 Acked Write 4 Done Done Sent false
36349 10326 Acked Read 7 Done Done Sent false
36350 10327 Acked Read 7 Done Done Sent false
36351 10328 Acked Write 8 Done Done Sent false
36352 10329 Acked Read 5 Done Done Sent false
36353 10330 Acked Write 9 Done Done Sent false
36354 10331 Acked Write 5 Done Done Sent false
36355 10332 Acked Write 3 Done Done Sent false
36356 10333 Acked Read 5 Done Done Sent false
36357 10334 Acked Write 9 Done Done Sent false
36358 10335 Acked Read 2 Done Done Sent false
36359 10336 Acked Read 1 Done Done Sent false
36360 10337 Acked Read 7 Done Done Sent false
36361 10338 Acked Read 5 Done Done Sent false
36362 10339 Acked Flush 0 Done Done Sent false
36363 10340 Acked Flush 0 Done Done Sent false
36364 10341 Acked Read 3 Done Done Sent false
36365 10342 Acked Write 1 Done Done Sent false
36366 10343 Acked Read 5 Done Done Sent false
36367 10344 Acked Write 4 Done Done Sent false
36368 10345 Acked Write 8 Done Done Sent false
36369 10346 Acked Read 1 Done Done Sent false
36370 10347 Acked Read 7 Done Done Sent false
36371 10348 Acked Read 5 Done Done Sent false
36372 10349 Acked Write 3 Done Done Sent false
36373 10350 Acked Read 1 Done Done Sent false
36374 10351 Acked Read 8 Done Done Sent false
36375 10352 Acked Write 9 Done Done Sent false
36376 10353 Acked Read 9 Done Done Sent false
36377 10354 Acked Read 8 Done Done Sent false
36378 10355 Acked Read 6 Done Done Sent false
36379 10356 Acked Read 6 Done Done Sent false
36380 10357 Acked Write 3 Done Done Sent false
36381 10358 Acked Flush 0 Done Done Sent false
36382 STATES DS:0 DS:1 DS:2 TOTAL
36383 Sent 0 0 301 301
36384 Done 301 301 0 602
36385 Skipped 0 0 0 0
36386 Error 0 0 0 0
36387 Last Flush: Some(JobId(10358)) Some(JobId(10358)) Some(JobId(10057))
36388 Downstairs last five completed: 10057 10056 10055 10054 10053
36389 Upstairs last five completed: 10358 10357 10356 10355 10354
36390 CLIENT: Up:0 ds:301 act:2
363912025-02-07T20:15:17.843ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
363922025-02-07T20:15:18.717ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
363932025-02-07T20:15:19.085ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
363942025-02-07T20:15:19.085ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59844) = io task client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
363952025-02-07T20:15:19.086ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
363962025-02-07T20:15:19.087ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
363972025-02-07T20:15:19.087ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
363982025-02-07T20:15:19.087ZINFOcrucible: send last flush ID to this DS: Some(JobId(10057)) = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
363992025-02-07T20:15:19.087ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
364002025-02-07T20:15:19.087ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(10057)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
364012025-02-07T20:15:19.087ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(10057)) = downstairs session_id = 8f7ba378-6b82-4db6-a812-9286715f4b09
36402 ----------------------------------------------------------------
36403 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36404 Downstairs last five completed: 10358 10357 10356 10355 10354
36405 Upstairs last five completed: 10358 10357 10356 10355 10354
36406 CLIENT: Up:0 ds:0 act:3
36407 Replay: All jobs finished
36408 Test replay has completed
36409 Wrote out file "/tmp/test_replay_verify.log"
36410 CLIENT: Tests done. All submitted work has been ACK'd
36411 ----------------------------------------------------------------
36412 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36413 Downstairs last five completed: 10358 10357 10356 10355 10354
36414 Upstairs last five completed: 10358 10357 10356 10355 10354
36415 CLIENT: Up:0 ds:0 act:3
36416 CLIENT: All jobs finished, all DS active.
36417 Replays:30 time: 8:04
36418 Do final verify
36419 Feb 07 20:15:21.418 INFO accepted connection, remote_addr: 127.0.0.1:45072, local_addr: 127.0.0.1:9998
36420 Feb 07 20:15:21.419 INFO request completed, latency_us: 200, response_code: 200, uri: /regioninfo, method: GET, req_id: 960eb6fa-e163-4033-ae56-b67a7bb8cb0a, remote_addr: 127.0.0.1:45072, local_addr: 127.0.0.1:9998
364212025-02-07T20:15:21.420ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
36422 Feb 07 20:15:21.420 INFO request completed, latency_us: 98, response_code: 200, uri: /regioncount, method: GET, req_id: d20bf79d-7f64-4de4-b5b9-c90de6169b94, remote_addr: 127.0.0.1:45072, local_addr: 127.0.0.1:9998
364232025-02-07T20:15:21.420ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
36424 Feb 07 20:15:21.421 INFO request completed, latency_us: 111, response_code: 200, uri: /port/cid/0, method: GET, req_id: eb885789-d119-4225-af09-7260d079fbce, remote_addr: 127.0.0.1:45072, local_addr: 127.0.0.1:9998
36425 Feb 07 20:15:21.421 INFO request completed, latency_us: 93, response_code: 200, uri: /port/cid/1, method: GET, req_id: 650c41cc-bbbe-42a3-914f-bef782805250, remote_addr: 127.0.0.1:45072, local_addr: 127.0.0.1:9998
36426 Feb 07 20:15:21.422 INFO request completed, latency_us: 100, response_code: 200, uri: /port/cid/2, method: GET, req_id: 14c3df75-edb6-44e1-bf12-54f30e0fbd00, remote_addr: 127.0.0.1:45072, local_addr: 127.0.0.1:9998
364272025-02-07T20:15:21.422ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
364282025-02-07T20:15:21.422ZINFOcrucible: Upstairs starts
364292025-02-07T20:15:21.422ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "764da36f6a4255a5834591b69c6e5591e6ce3a04", git_commit_timestamp: "2025-02-07T19:51:46.000000000Z", git_branch: "alan/agent-needs-workers", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
364302025-02-07T20:15:21.422ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
364312025-02-07T20:15:21.422ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
364322025-02-07T20:15:21.423ZINFOcrucible: Crucible d76276b0-aa7c-407b-a9c9-5192057036e9 has session id: fa4752e1-7377-4a9d-b00c-8af1dab9c8b0 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364332025-02-07T20:15:21.423ZINFOcrucible: Upstairs opts: Upstairs UUID: d76276b0-aa7c-407b-a9c9-5192057036e9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364342025-02-07T20:15:21.423ZINFOcrucible: Crucible stats registered with UUID: d76276b0-aa7c-407b-a9c9-5192057036e9 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364352025-02-07T20:15:21.423ZWARNcrucible: could not find Downstairs address for Nexus session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364362025-02-07T20:15:21.424ZINFOcrucible: Downstairs in volume = 3 task = crutest
364372025-02-07T20:15:21.424ZINFOcrucible: The guest has requested activation with gen:3
364382025-02-07T20:15:21.424ZINFOcrucible: d76276b0-aa7c-407b-a9c9-5192057036e9 active request set session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364392025-02-07T20:15:21.424ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364402025-02-07T20:15:21.424ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364412025-02-07T20:15:21.424ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364422025-02-07T20:15:21.424ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364432025-02-07T20:15:21.424ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364442025-02-07T20:15:21.424ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364452025-02-07T20:15:21.424ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364462025-02-07T20:15:21.424ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364472025-02-07T20:15:21.424ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364482025-02-07T20:15:21.424ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59731) = io task client = 2 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364492025-02-07T20:15:21.424ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65535) = io task client = 1 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364502025-02-07T20:15:21.424ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52251) = io task client = 0 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364512025-02-07T20:15:21.425ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364522025-02-07T20:15:21.425ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364532025-02-07T20:15:21.425ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364542025-02-07T20:15:21.425ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364552025-02-07T20:15:21.425ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364562025-02-07T20:15:21.426ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364572025-02-07T20:15:21.426ZINFOcrucible: Waiting for more clients to be ready session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364582025-02-07T20:15:21.426ZINFOcrucible: Waiting for more clients to be ready session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364592025-02-07T20:15:21.426ZINFOcrucible: [0]R flush_numbers[0..12]: [990, 988, 988, 993, 993, 989, 989, 989, 953, 988, 971, 993] = downstairs session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364602025-02-07T20:15:21.426ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364612025-02-07T20:15:21.426ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364622025-02-07T20:15:21.427ZINFOcrucible: [1]R flush_numbers[0..12]: [990, 988, 988, 993, 993, 989, 989, 989, 953, 988, 971, 993] = downstairs session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364632025-02-07T20:15:21.427ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364642025-02-07T20:15:21.427ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364652025-02-07T20:15:21.427ZINFOcrucible: [2]R flush_numbers[0..12]: [990, 988, 988, 993, 993, 989, 989, 989, 953, 988, 971, 993] = downstairs session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364662025-02-07T20:15:21.427ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
36467 {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-07T20:15:21.427088442Z","hostname":"w-01JKH09F5XYF0CC4T98BFKX0P2"Wait for a query_work_queue command to finish before sending IO
36468 ,"pid":1245,"":"downstairs","":"downstairs","session_id":"fa4752e1-7377-4a9d-b00c-8af1dab9c8b0"}
364692025-02-07T20:15:21.427ZINFOcrucible: Max found gen is 3 = downstairs session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364702025-02-07T20:15:21.427ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364712025-02-07T20:15:21.427ZINFOcrucible: Next flush: 994 = downstairs session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364722025-02-07T20:15:21.427ZINFOcrucible: All extents match = downstairs session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364732025-02-07T20:15:21.427ZINFOcrucible: No downstairs reconciliation required session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364742025-02-07T20:15:21.427ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364752025-02-07T20:15:21.427ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364762025-02-07T20:15:21.427ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364772025-02-07T20:15:21.427ZINFOcrucible: All required reconciliation work is completed session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364782025-02-07T20:15:21.427ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364792025-02-07T20:15:21.427ZINFOcrucible: d76276b0-aa7c-407b-a9c9-5192057036e9 is now active with session: fa4752e1-7377-4a9d-b00c-8af1dab9c8b0 session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364802025-02-07T20:15:21.427ZINFOcrucible: Set Active after no reconciliation session_id = fa4752e1-7377-4a9d-b00c-8af1dab9c8b0
364812025-02-07T20:15:21.427ZINFOcrucible: The guest has finished waiting for activation with:3
36482 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
36483 Loading write count information from file "/tmp/test_replay_verify.log"
36484 Read and Verify all blocks (0..5000 range:false)
36485 Wrote out file "/tmp/test_replay_verify.log"
36486 Verify test completed
36487 Wrote out file "/tmp/test_replay_verify.log"
36488 CLIENT: Tests done. All submitted work has been ACK'd
36489 ----------------------------------------------------------------
36490 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
36491 States: Active Active Active
36492 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36493 1000 Acked Read 256 Done Done Done false
36494 1001 Acked Read 256 Done Done Done false
36495 1002 Acked Read 256 Done Done Done false
36496 1003 Acked Read 256 Done Done Done false
36497 1004 Acked Read 256 Done Done Done false
36498 1005 Acked Read 256 Done Done Done false
36499 1006 Acked Read 256 Done Done Done false
36500 1007 Acked Read 256 Done Done Done false
36501 1008 Acked Read 256 Done Done Done false
36502 1009 Acked Read 256 Done Done Done false
36503 1010 Acked Read 256 Done Done Done false
36504 1011 Acked Read 256 Done Done Done false
36505 1012 Acked Read 256 Done Done Done false
36506 1013 Acked Read 256 Done Done Done false
36507 1014 Acked Read 256 Done Done Done false
36508 1015 Acked Read 256 Done Done Done false
36509 1016 Acked Read 256 Done Done Done false
36510 1017 Acked Read 136 Done Done Done false
36511 1018 Acked Read 256 Done Done Done false
36512 1019 Acked Read 256 Done Done Done false
36513 STATES DS:0 DS:1 DS:2 TOTAL
36514 Sent 0 0 0 0
36515 Done 20 20 20 60
36516 Skipped 0 0 0 0
36517 Error 0 0 0 0
36518 Last Flush: None None None
36519 Downstairs last five completed:
36520 Upstairs last five completed: 1019 1018 1017 1016 1015
36521 CLIENT: Up:0 ds:20 act:3
36522 ----------------------------------------------------------------
36523 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
36524 Downstairs last five completed: 1020 1019 1018 1017 1016
36525 Upstairs last five completed: 1020 1019 1018 1017 1016
36526 CLIENT: Up:0 ds:0 act:3
36527 CLIENT: All crucible jobs finished, exiting program
36528 Feb 07 20:15:25.891 INFO accepted connection, remote_addr: 127.0.0.1:43564, local_addr: 127.0.0.1:9998
36529 Main task has work to do, go find it
36530 got dsc Shutdown
36531 Shutdown
36532 Shut it down
36533 Feb 07 20:15:25.892 INFO request completed, latency_us: 174, response_code: 204, [8810] Disable keep_running
36534 uri[8810] Got stop action so:false kr:false
36535 : /shutdown, method: POST, req_id: bc41d0b1-c488-4cbc-bafd-92f10a6ad77e, remote_addr: 127.0.0.1:43564, local_addr: 127.0.0.1:9998
36536 [8820] Disable keep_running
36537 [8820] Got stop action so:false kr:false
36538 [8830] Disable keep_running
36539 [8830] Got stop action so:false kr:false
36540 [8820] Exited with: ExitStatus(unix_wait_status(9))
36541 [8830] Exited with: ExitStatus(unix_wait_status(9))
36542 [8820][1] reports Exit
36543 [8830][2] reports Exit
36544 [8810] Exited with: ExitStatus(unix_wait_status(9))
36545 [8810][0] reports Exit
36546 February 7, 2025 at 08:15:30 PM UTC Test ends with 0