Max level shown:
1 starting February 28, 2025 at 06:00:53 AM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.34060928
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.021993108
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.020925635
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start ds: 8820
14 start ds: 8830
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8810
17 Starting downstairs at port 8820
18 Make output file at "/tmp/dsc/downstairs-8820.txt"
19 Make output file at "/tmp/dsc/downstairs-8810.txt"
20 Starting downstairs at port 8830
21 Make output file at "/tmp/dsc/downstairs-8830.txt"
22 [8810][0] initial start wait reports Starting
23 [8820][1] initial start wait reports Starting
24 [8830][2] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1201)
26 [8810][0] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1202)
28 [8820][1] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1203)
30 [8830][2] initial start wait reports Running
31 All downstairs are running
32 Feb 28 06:00:54.078 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Feb 28 06:00:59.160 INFO accepted connection, remote_addr: 127.0.0.1:53442, local_addr: 127.0.0.1:9998
36 Feb 28 06:00:59.162 INFO request completed, latency_us: 394, response_code: 200, uri: /regioninfo, method: GET, req_id: a60912e9-076f-486f-b7b3-7e4b826f914e, remote_addr: 127.0.0.1:53442, local_addr: 127.0.0.1:9998
37 Feb 28 06:00:59.164 INFO request completed, latency_us: 114, response_code: 200, uri: /regioncount, method: GET, req_id: 44e823ac-c66f-4956-8967-28bc7ef7cdf8, remote_addr: 127.0.0.1:53442, local_addr: 127.0.0.1:9998
382025-02-28T06:00:59.164ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-02-28T06:00:59.164ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Feb 28 06:00:59.165 INFO request completed, latency_us: 175, response_code: 200, uri: /port/cid/0, method: GET, req_id: a266326f-09b3-44da-a78e-1042881ce6a6, remote_addr: 127.0.0.1:53442, local_addr: 127.0.0.1:9998
41 Feb 28 06:00:59.165 INFO request completed, latency_us: 88, response_code: 200, uri: /port/cid/1, method: GET, req_id: a244afb5-bfbc-479c-aeae-06d19a06f7de, remote_addr: 127.0.0.1:53442, local_addr: 127.0.0.1:9998
42 Feb 28 06:00:59.166 INFO request completed, latency_us: 87, response_code: 200, uri: /port/cid/2, method: GET, req_id: 4a775af0-69e9-41a3-bf87-499abb342f8c, remote_addr: 127.0.0.1:53442, local_addr: 127.0.0.1:9998
432025-02-28T06:00:59.166ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-02-28T06:00:59.166ZINFOcrucible: Upstairs starts
452025-02-28T06:00:59.166ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "0c103f9ae2997264530dbd443fadbdf83878dac4", git_commit_timestamp: "2025-02-28T05:45:10.000000000Z", git_branch: "renovate/uuid-1.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-02-28T06:00:59.166ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-02-28T06:00:59.166ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-02-28T06:00:59.167ZINFOcrucible: Crucible 46604218-9055-4498-b8c8-dba792c5ec55 has session id: 03892bbc-2ae7-40f4-9218-bbfee5a2de00 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
492025-02-28T06:00:59.167ZINFOcrucible: Upstairs opts: Upstairs UUID: 46604218-9055-4498-b8c8-dba792c5ec55, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
502025-02-28T06:00:59.167ZINFOcrucible: Crucible stats registered with UUID: 46604218-9055-4498-b8c8-dba792c5ec55 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
512025-02-28T06:00:59.167ZWARNcrucible: could not find Downstairs address for Nexus session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
522025-02-28T06:00:59.169ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-02-28T06:00:59.169ZINFOcrucible: The guest has requested activation with gen:1
542025-02-28T06:00:59.169ZINFOcrucible: 46604218-9055-4498-b8c8-dba792c5ec55 active request set session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
552025-02-28T06:00:59.169ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
562025-02-28T06:00:59.169ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
572025-02-28T06:00:59.169ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
582025-02-28T06:00:59.169ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
592025-02-28T06:00:59.169ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
602025-02-28T06:00:59.169ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
612025-02-28T06:00:59.169ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
622025-02-28T06:00:59.169ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
632025-02-28T06:00:59.169ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
642025-02-28T06:00:59.169ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51786) = io task client = 2 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
652025-02-28T06:00:59.169ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36704) = io task client = 0 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
662025-02-28T06:00:59.169ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43639) = io task client = 1 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
672025-02-28T06:00:59.171ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
682025-02-28T06:00:59.171ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
692025-02-28T06:00:59.171ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
702025-02-28T06:00:59.172ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
712025-02-28T06:00:59.172ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
722025-02-28T06:00:59.172ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
732025-02-28T06:00:59.173ZINFOcrucible: Waiting for more clients to be ready session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
742025-02-28T06:00:59.173ZINFOcrucible: Waiting for more clients to be ready session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
752025-02-28T06:00:59.173ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
762025-02-28T06:00:59.173ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
772025-02-28T06:00:59.173ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
782025-02-28T06:00:59.173ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
792025-02-28T06:00:59.173ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
802025-02-28T06:00:59.173ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
812025-02-28T06:00:59.173ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
822025-02-28T06:00:59.173ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
832025-02-28T06:00:59.173ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
842025-02-28T06:00:59.173ZINFOcrucible: Max found gen is 1 = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
852025-02-28T06:00:59.173ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
862025-02-28T06:00:59.173ZINFOcrucible: Next flush: 1 = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
872025-02-28T06:00:59.173ZINFOcrucible: All extents match = downstairs session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
882025-02-28T06:00:59.173ZINFOcrucible: No downstairs reconciliation required session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
892025-02-28T06:00:59.173ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
902025-02-28T06:00:59.173ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
912025-02-28T06:00:59.173ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
922025-02-28T06:00:59.173ZINFOcrucible: All required reconciliation work is completed session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
932025-02-28T06:00:59.173ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
94 {"msg":"46604218-9055-4498-b8c8-dba792c5ec55 is now active with session: 03892bbc-2ae7-40f4-9218-bbfee5a2de00","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:00:59.173728428Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","Activate successful
95 pid":Wait for a query_work_queue command to finish before sending IO
96 1207,"session_id":"03892bbc-2ae7-40f4-9218-bbfee5a2de00"}
972025-02-28T06:00:59.173ZINFOcrucible: Set Active after no reconciliation session_id = 03892bbc-2ae7-40f4-9218-bbfee5a2de00
982025-02-28T06:00:59.173ZINFOcrucible: The guest has finished waiting for activation with:1
99 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
100 Fill test
101 Read and Verify all blocks (0..5000 range:false)
102 Wrote out file "/tmp/test_replay_verify.log"
103 CLIENT: Tests done. All submitted work has been ACK'd
104 ----------------------------------------------------------------
105 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
106 States: Active Active Active
107 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
108 1021 Acked Read 256 Done Done Done false
109 1022 Acked Read 256 Done Done Done false
110 1023 Acked Read 256 Done Done Done false
111 1024 Acked Read 256 Done Done Done false
112 1025 Acked Read 256 Done Done Done false
113 1026 Acked Read 256 Done Done Done false
114 1027 Acked Read 256 Done Done Done false
115 1028 Acked Read 256 Done Done Done false
116 1029 Acked Read 256 Done Done Done false
117 1030 Acked Read 256 Done Done Done false
118 1031 Acked Read 256 Done Done Done false
119 1032 Acked Read 256 Done Done Done false
120 1033 Acked Read 256 Done Done Done false
121 1034 Acked Read 256 Done Done Done false
122 1035 Acked Read 256 Done Done Done false
123 1036 Acked Read 256 Done Done Done false
124 1037 Acked Read 256 Done Done Done false
125 1038 Acked Read 136 Done Done Done false
126 1039 Acked Read 256 Done Done Done false
127 1040 Acked Read 256 Done Done Done false
128 STATES DS:0 DS:1 DS:2 TOTAL
129 Sent 0 0 0 0
130 Done 20 20 20 60
131 Skipped 0 0 0 0
132 Error 0 0 0 0
133 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
134 Downstairs last five completed: 1020 1019 1018 1017 1016
135 Upstairs last five completed: 1040 1039 1038 1037 1036
136 CLIENT: Up:0 ds:20 act:3
137 ----------------------------------------------------------------
138 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
139 Downstairs last five completed: 1041 1040 1039 1038 1037
140 Upstairs last five completed: 1041 1040 1039 1038 1037
141 CLIENT: Up:0 ds:0 act:3
142 CLIENT: All crucible jobs finished, exiting program
1432025-02-28T06:01:03.764ZWARNcrucible: Guest handle has been dropped
144 Replay loop starts now February 28, 2025 at 06:01:03 AM UTC
145 Feb 28 06:01:03.835 INFO accepted connection, remote_addr: 127.0.0.1:50173, local_addr: 127.0.0.1:9998
146 Feb 28 06:01:03.835 INFO request completed, latency_us: 171, response_code: 200, uri: /regioninfo, method: GET, req_id: f2655093-b468-485f-930b-437639575d5a, remote_addr: 127.0.0.1:50173, local_addr: 127.0.0.1:9998
1472025-02-28T06:01:03.836ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
148 Feb 28 06:01:03.836 INFO request completed, latency_us: 84, response_code: 200, uri: /regioncount, method: GET, req_id: 911ef9b1-3eb1-48dc-8c06-0436bf3b8e33, remote_addr: 127.0.0.1:50173, local_addr: 127.0.0.1:9998
1492025-02-28T06:01:03.837ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
150 Feb 28 06:01:03.837 INFO request completed, latency_us: 128, response_code: 200, uri: /port/cid/0, method: GET, req_id: 8dc0fcdc-de28-44cd-9e56-0f2049c4a1a3, remote_addr: 127.0.0.1:50173, local_addr: 127.0.0.1:9998
151 Feb 28 06:01:03.838 INFO request completed, latency_us: 102, response_code: 200, uri: /port/cid/1, method: GET, req_id: 350d8791-8487-4934-953b-e7537df3f574, remote_addr: 127.0.0.1:50173, local_addr: 127.0.0.1:9998
152 Feb 28 06:01:03.838 INFO request completed, latency_us: 84, response_code: 200, uri: /port/cid/2, method: GET, req_id: f0fe630b-bf01-4186-a72d-9ff45252fb31, remote_addr: 127.0.0.1:50173, local_addr: 127.0.0.1:9998
1532025-02-28T06:01:03.838ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1542025-02-28T06:01:03.838ZINFOcrucible: Upstairs starts
1552025-02-28T06:01:03.838ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "0c103f9ae2997264530dbd443fadbdf83878dac4", git_commit_timestamp: "2025-02-28T05:45:10.000000000Z", git_branch: "renovate/uuid-1.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1562025-02-28T06:01:03.838ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1572025-02-28T06:01:03.838ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1582025-02-28T06:01:03.839ZINFOcrucible: Crucible 1ce0952d-43d2-414d-849d-64dd1aa825bb has session id: 969fa3b2-3050-4644-bbdf-c5076af6b813 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1592025-02-28T06:01:03.839ZINFOcrucible: Upstairs opts: Upstairs UUID: 1ce0952d-43d2-414d-849d-64dd1aa825bb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1602025-02-28T06:01:03.839ZINFOcrucible: Crucible stats registered with UUID: 1ce0952d-43d2-414d-849d-64dd1aa825bb session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1612025-02-28T06:01:03.839ZWARNcrucible: could not find Downstairs address for Nexus session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1622025-02-28T06:01:03.840ZINFOcrucible: Downstairs in volume = 3 task = crutest
1632025-02-28T06:01:03.840ZINFOcrucible: The guest has requested activation with gen:2
1642025-02-28T06:01:03.840ZINFOcrucible: 1ce0952d-43d2-414d-849d-64dd1aa825bb active request set session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1652025-02-28T06:01:03.840ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1662025-02-28T06:01:03.840ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1672025-02-28T06:01:03.840ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1682025-02-28T06:01:03.840ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1692025-02-28T06:01:03.840ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1702025-02-28T06:01:03.840ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1712025-02-28T06:01:03.840ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1722025-02-28T06:01:03.840ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1732025-02-28T06:01:03.840ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1742025-02-28T06:01:03.841ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53702) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1752025-02-28T06:01:03.841ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60632) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1762025-02-28T06:01:03.841ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:57554) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1772025-02-28T06:01:03.841ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1782025-02-28T06:01:03.841ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1792025-02-28T06:01:03.841ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1802025-02-28T06:01:03.842ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1812025-02-28T06:01:03.842ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1822025-02-28T06:01:03.842ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1832025-02-28T06:01:03.842ZINFOcrucible: Waiting for more clients to be ready session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1842025-02-28T06:01:03.842ZINFOcrucible: Waiting for more clients to be ready session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1852025-02-28T06:01:03.842ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1862025-02-28T06:01:03.843ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1872025-02-28T06:01:03.843ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1882025-02-28T06:01:03.843ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1892025-02-28T06:01:03.843ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1902025-02-28T06:01:03.843ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1912025-02-28T06:01:03.843ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1922025-02-28T06:01:03.843ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
193 {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Activate successful
194 ,Wait for a query_work_queue command to finish before sending IO
195 "time":"2025-02-28T06:01:03.843115408Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"downstairs","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
1962025-02-28T06:01:03.843ZINFOcrucible: Max found gen is 2 = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1972025-02-28T06:01:03.843ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1982025-02-28T06:01:03.843ZINFOcrucible: Next flush: 2 = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1992025-02-28T06:01:03.843ZINFOcrucible: All extents match = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2002025-02-28T06:01:03.843ZINFOcrucible: No downstairs reconciliation required session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2012025-02-28T06:01:03.843ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2022025-02-28T06:01:03.843ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2032025-02-28T06:01:03.843ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2042025-02-28T06:01:03.843ZINFOcrucible: All required reconciliation work is completed session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2052025-02-28T06:01:03.843ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2062025-02-28T06:01:03.843ZINFOcrucible: 1ce0952d-43d2-414d-849d-64dd1aa825bb is now active with session: 969fa3b2-3050-4644-bbdf-c5076af6b813 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2072025-02-28T06:01:03.843ZINFOcrucible: Set Active after no reconciliation session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2082025-02-28T06:01:03.843ZINFOcrucible: The guest has finished waiting for activation with:2
209 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
210 Loading write count information from file "/tmp/test_replay_verify.log"
211 Feb 28 06:01:03.876 INFO accepted connection, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
212 Main task has work to do, go find it
213 got dsc Stop(1)
214 stop 1
215 Feb 28 06:01:03.877 INFO request completed, latency_us: 195, response_code: 204, uri: /stop/cid/1, method: POST, req_id: fb1fceb5-7be1-43c4-8e1c-8c89ea75b5a8, remote_addr: 127.0.0.1:63908, local_addr[8820] Got stop action so:false kr:false
216 : 127.0.0.1:9998
2172025-02-28T06:01:03.877ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2182025-02-28T06:01:03.877ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2192025-02-28T06:01:03.877ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2202025-02-28T06:01:03.877ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
221 Feb 28 06:01:03.878 INFO request completed, latency_us: 302, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6b654463-e477-4552-ab05-0be23952096d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
222 [8820] Exited with: ExitStatus(unix_wait_status(9))
223 [8820][1] reports Exit
224 Feb 28 06:01:07.879 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6a168064-2039-465a-8b09-3c2748560d7e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
225 001/300 Write block 3859 len 12288 data: 2 2 2
226 002/300 Write block 971 len 4096 data: 2
227 003/300 Write block 378 len 36864 data: 2 2 2 2 2 2 2 2 2
228 004/300 Write block 1950 len 40960 data: 2 2 2 2 2 2 2 2 2 2
229 005/300 Flush
230 006/300 Read block 1359 len 40960
231 007/300 Read block 3799 len 8192
232 008/300 Write block 3 len 40960 data: 2 2 2 2 2 2 2 2 2 2
233 009/300 Flush
234 010/300 Read block 3654 len 32768
235 011/300 Flush
236 012/300 Read block 3941 len 36864
237 013/300 Read block 985 len 4096
238 014/300 Read block 1597 len 8192
239 015/300 Read block 1891 len 32768
240 016/300 Read block 2009 len 36864
241 017/300 Read block 4051 len 12288
242 018/300 Read block 445 len 36864
243 019/300 Write block 2597 len 20480 data: 2 2 2 2 2
244 020/300 Read block 1277 len 4096
245 021/300 Write block 4020 len 24576 data: 2 2 2 2 2 2
246 022/300 Write block 413 len 24576 data: 2 2 2 2 2 2
247 023/300 Read block 4891 len 8192
248 024/300 Write block 3781 len 32768 data: 2 2 2 2 2 2 2 2
249 025/300 Read block 3259 len 24576
250 026/300 Read block 2979 len 36864
251 027/300 Write block 4717 len 16384 data: 2 2 2 2
252 028/300 Write block 1851 len 8192 data: 2 2
253 029/300 Read block 1660 len 12288
254 030/300 Write block 800 len 8192 data: 2 2
255 031/300 Write block 4352 len 32768 data: 2 2 2 2 2 2 2 2
256 032/300 Read block 4630 len 12288
257 033/300 Flush
258 034/300 Write block 2375 len 16384 data: 2 2 2 2
259 035/300 Write block 1347 len 36864 data: 2 2 2 2 2 2 2 2 2
260 036/300 Write block 1688 len 40960 data: 2 2 2 2 2 2 2 2 2 2
261 037/300 Read block 3420 len 36864
262 038/300 Read block 749 len 16384
263 039/300 Write block 3522 len 32768 data: 2 2 2 2 2 2 2 2
264 040/300 Write block 690 len 12288 data: 2 2 2
265 041/300 Read block 3326 len 16384
266 042/300 Read block 3156 len 32768
267 043/300 Write block 1521 len 40960 data: 2 2 2 2 2 2 2 2 2 2
268 044/300 Read block 2115 len 8192
269 045/300 Read block 941 len 8192
270 046/300 Read block 2173 len 24576
271 047/300 Write block 4039 len 4096 data: 2
272 048/300 Write block 4888 len 36864 data: 2 2 2 2 2 2 2 2 2
273 049/300 Write block 4542 len 40960 data: 2 2 2 2 2 2 2 2 2 2
274 050/300 Flush
275 051/300 Flush
276 052/300 Read block 1071 len 20480
277 053/300 Read block 1498 len 36864
278 054/300 Write block 41 len 8192 data: 2 2
279 055/300 Write block 1517 len 24576 data: 2 2 2 2 3 3
280 056/300 Read block 227 len 32768
281 057/300 Write block 4267 len 24576 data: 2 2 2 2 2 2
282 058/300 Write block 4407 len 28672 data: 2 2 2 2 2 2 2
283 059/300 Flush
284 060/300 Write block 1291 len 16384 data: 2 2 2 2
285 061/300 Write block 1668 len 36864 data: 2 2 2 2 2 2 2 2 2
286 062/300 Write block 2938 len 24576 data: 2 2 2 2 2 2
287 063/300 Read block 728 len 40960
288 064/300 Read block 4960 len 4096
289 065/300 Write block 2961 len 40960 data: 2 2 2 2 2 2 2 2 2 2
290 066/300 Read block 3906 len 32768
291 067/300 Write block 2273 len 8192 data: 2 2
292 068/300 Flush
293 069/300 Flush
294 070/300 Write block 4888 len 16384 data: 3 3 3 3
295 071/300 Read block 3141 len 4096
296 072/300 Write block 221 len 28672 data: 2 2 2 2 2 2 2
297 073/300 Read block 4057 len 32768
298 074/300 Write block 75 len 20480 data: 2 2 2 2 2
299 075/300 Read block 1837 len 32768
300 076/300 Write block 3416 len 8192 data: 2 2
301 077/300 Read block 4701 len 12288
302 078/300 Write block 1613 len 32768 data: 2 2 2 2 2 2 2 2
303 079/300 Read block 79 len 8192
304 080/300 Read block 2569 len 16384
305 081/300 Write block 4209 len 20480 data: 2 2 2 2 2
306 082/300 Write block 4190 len 40960 data: 2 2 2 2 2 2 2 2 2 2
307 083/300 Write block 749 len 40960 data: 2 2 2 2 2 2 2 2 2 2
308 084/300 Read block 3852 len 16384
309 085/300 Write block 3589 len 12288 data: 2 2 2
310 086/300 Read block 568 len 40960
311 087/300 Read block 3786 len 32768
312 088/300 Write block 337 len 8192 data: 2 2
313 089/300 Write block 857 len 32768 data: 2 2 2 2 2 2 2 2
314 090/300 Write block 841 len 24576 data: 2 2 2 2 2 2
315 091/300 Flush
316 092/300 Write block 2783 len 40960 data: 2 2 2 2 2 2 2 2 2 2
317 093/300 Read block 4955 len 20480
318 094/300 Flush
319 095/300 Write block 4405 len 32768 data: 2 2 3 3 3 3 3 3
320 096/300 Read block 947 len 16384
321 097/300 Read block 4255 len 8192
322 098/300 Read block 3446 len 32768
323 099/300 Read block 2158 len 8192
324 100/300 Write block 4375 len 28672 data: 2 2 2 2 2 2 2
325 101/300 Read block 1539 len 20480
326 102/300 Read block 3711 len 32768
327 103/300 Read block 1577 len 24576
328 104/300 Flush
329 105/300 Write block 4764 len 36864 data: 2 2 2 2 2 2 2 2 2
330 106/300 Read block 636 len 24576
331 107/300 Write block 829 len 28672 data: 2 2 2 2 2 2 2
332 108/300 Read block 3901 len 4096
333 109/300 Write block 1142 len 20480 data: 2 2 2 2 2
334 110/300 Read block 4611 len 12288
335 111/300 Read block 2879 len 28672
336 112/300 Write block 2234 len 28672 data: 2 2 2 2 2 2 2
337 113/300 Read block 953 len 24576
338 114/300 Flush
339 115/300 Read block 1584 len 16384
340 116/300 Write block 3180 len 8192 data: 2 2
341 117/300 Write block 144 len 28672 data: 2 2 2 2 2 2 2
342 118/300 Flush
343 119/300 Write block 2392 len 8192 data: 2 2
344 120/300 Read block 4345 len 4096
345 121/300 Flush
346 122/300 Read block 4300 len 4096
347 123/300 Write block 3417 len 8192 data: 3 2
348 124/300 Read block 4402 len 28672
349 125/300 Read block 4524 len 40960
350 126/300 Flush
351 127/300 Read block 2144 len 20480
352 128/300 Read block 4599 len 8192
353 129/300 Write block 107 len 24576 data: 2 2 2 2 2 2
354 130/300 Read block 2414 len 36864
355 131/300 Flush
356 132/300 Read block 3088 len 8192
357 133/300 Read block 642 len 8192
358 134/300 Write block 866 len 20480 data: 2 2 2 2 2
359 135/300 Read block 4069 len 28672
360 136/300 Read block 3238 len 28672
361 137/300 Read block 1694 len 4096
362 138/300 Read block 76 len 16384
363 139/300 Read block 4023 len 4096
364 140/300 Read block 194 len 36864
365 141/300 Flush
366 142/300 Read block 4901 len 8192
367 143/300 Read block 1505 len 20480
368 144/300 Read block 4790 len 4096
369 145/300 Flush
370 146/300 Write block 938 len 40960 data: 2 2 2 2 2 2 2 2 2 2
371 147/300 Read block 3806 len 8192
372 148/300 Write block 4271 len 36864 data: 3 3 2 2 2 2 2 2 2
373 149/300 Read block 4887 len 36864
374 150/300 Write block 2353 len 4096 data: 2
375 151/300 Flush
376 152/300 Write block 1501 len 4096 data: 2
377 153/300 Write block 178 len 4096 data: 2
378 154/300 Read block 3084 len 36864
379 155/300 Write block 1674 len 28672 data: 3 3 3 2 2 2 2
380 156/300 Read block 4190 len 4096
381 157/300 Write block 1676 len 20480 data: 4 3 3 3 3
382 158/300 Flush
383 159/300 Write block 3688 len 4096 data: 2
384 160/300 Write block 678 len 16384 data: 2 2 2 2
385 161/300 Read block 1421 len 32768
386 162/300 Write block 2730 len 4096 data: 2
387 163/300 Read block 1274 len 28672
388 164/300 Read block 3719 len 8192
389 165/300 Read block 563 len 12288
390 166/300 Read block 1900 len 8192
391 167/300 Write block 1179 len 32768 data: 2 2 2 2 2 2 2 2
392 168/300 Read block 689 len 36864
393 169/300 Read block 3086 len 8192
394 170/300 Write block 2363 len 16384 data: 2 2 2 2
395 171/300 Write block 4984 len 16384 data: 2 2 2 2
396 172/300 Read block 3894 len 40960
397 173/300 Read block 2882 len 32768
398 174/300 Read block 1216 len 16384
399 175/300 Write block 4963 len 20480 data: 2 2 2 2 2
400 176/300 Write block 4238 len 24576 data: 2 2 2 2 2 2
401 177/300 Read block 226 len 4096
402 178/300 Read block 896 len 32768
403 179/300 Write block 4005 len 28672 data: 2 2 2 2 2 2 2
404 180/300 Read block 1586 len 4096
405 181/300 Write block 4842 len 4096 data: 2
406 182/300 Read block 4078 len 16384
407 183/300 Write block 428 len 24576 data: 2 2 2 2 2 2
408 184/300 Write block 3194 len 24576 data: 2 2 2 2 2 2
409 185/300 Write block 2146 len 28672 data: 2 2 2 2 2 2 2
410 186/300 Read block 2769 len 36864
411 187/300 Write block 625 len 24576 data: 2 2 2 2 2 2
412 188/300 Flush
413 189/300 Read block 1185 len 12288
414 190/300 Write block 4951 len 16384 data: 2 2 2 2
415 191/300 Read block 1825 len 32768
416 192/300 Write block 1568 len 40960 data: 2 2 2 2 2 2 2 2 2 2
417 193/300 Write block 4831 len 8192 data: 2 2
418 194/300 Read block 2709 len 8192
419 195/300 Write block 3364 len 12288 data: 2 2 2
420 196/300 Read block 3531 len 40960
421 197/300 Write block 1459 len 24576 data: 2 2 2 2 2 2
422 198/300 Write block 736 len 4096 data: 2
423 199/300 Flush
424 200/300 Read block 3456 len 24576
425 201/300 Write block 4828 len 16384 data: 2 2 2 3
426 202/300 Write block 3053 len 16384 data: 2 2 2 2
427 203/300 Read block 3927 len 4096
428 204/300 Read block 438 len 36864
429 205/300 Read block 3626 len 4096
430 206/300 Read block 3036 len 16384
431 207/300 Read block 1246 len 12288
432 208/300 Write block 887 len 8192 data: 2 2
433 209/300 Read block 1801 len 16384
434 210/300 Write block 1409 len 24576 data: 2 2 2 2 2 2
435 211/300 Write block 234 len 20480 data: 2 2 2 2 2
436 212/300 Read block 1261 len 4096
437 213/300 Read block 330 len 4096
438 214/300 Write block 3090 len 20480 data: 2 2 2 2 2
439 215/300 Write block 3687 len 8192 data: 2 3
440 216/300 Flush
441 217/300 Read block 1497 len 36864
442 218/300 Flush
443 219/300 Write block 3114 len 32768 data: 2 2 2 2 2 2 2 2
444 220/300 Read block 2951 len 28672
445 221/300 Write block 214 len 32768 data: 2 2 2 2 2 2 2 3
446 222/300 Read block 1877 len 20480
447 223/300 Flush
448 224/300 Read block 4978 len 32768
449 225/300 Write block 1527 len 40960 data: 3 3 3 3 2 2 2 2 2 2
450 226/300 Read block 459 len 24576
451 227/300 Write block 2259 len 8192 data: 2 2
452 228/300 Write block 1359 len 12288 data: 2 2 2
453 229/300 Read block 4895 len 16384
454 230/300 Write block 1037 len 40960 data: 2 2 2 2 2 2 2 2 2 2
455 231/300 Read block 1145 len 4096
456 232/300 Write block 4190 len 16384 data: 3 3 3 3
457 233/300 Read block 590 len 12288
458 234/300 Read block 1415 len 24576
459 235/300 Read block 2871 len 4096
460 236/300 Read block 4530 len 36864
461 237/300 Read block 718 len 36864
462 238/300 Flush
463 239/300 Read block 617 len 28672
464 240/300 Read block 4387 len 36864
465 241/300 Write block 2168 len 4096 data: 2
466 242/300 Read block 966 len 24576
467 243/300 Read block 1486 len 12288
468 244/300 Read block 1295 len 28672
469 245/300 Write block 2863 len 4096 data: 2
470 246/300 Flush
471 247/300 Write block 1148 len 40960 data: 2 2 2 2 2 2 2 2 2 2
472 248/300 Read block 1199 len 40960
473 249/300 Flush
474 250/300 Read block 909 len 28672
475 251/300 Flush
476 252/300 Write block 144 len 24576 data: 3 3 3 3 3 3
477 253/300 Write block 2420 len 32768 data: 2 2 2 2 2 2 2 2
478 254/300 Flush
479 255/300 Read block 3472 len 20480
480 256/300 Write block 521 len 16384 data: 2 2 2 2
481 257/300 Write block 2544 len 32768 data: 2 2 2 2 2 2 2 2
482 258/300 Write block 1489 len 16384 data: 2 2 2 2
483 259/300 Read block 1064 len 4096
484 260/300 Read block 1082 len 12288
485 261/300 Read block 1817 len 16384
486 262/300 Write block 419 len 16384 data: 2 2 2 2
487 263/300 Read block 1733 len 4096
488 264/300 Write block 3110 len 40960 data: 2 2 2 2 3 3 3 3 3 3
489 265/300 Read block 4947 len 28672
490 266/300 Read block 1569 len 12288
491 267/300 Read block 296 len 32768
492 268/300 Read block 3837 len 36864
493 269/300 Write block 3795 len 28672 data: 2 2 2 2 2 2 2
494 270/300 Read block 790 len 24576
495 271/300 Write block 1078 len 24576 data: 2 2 2 2 2 2
496 272/300 Flush
497 273/300 Write block 812 len 12288 data: 2 2 2
498 274/300 Read block 2483 len 4096
499 275/300 Read block 2144 len 40960
500 276/300 Write block 2195 len 12288 data: 2 2 2
501 277/300 Read block 1859 len 36864
502 278/300 Read block 4964 len 36864
503 279/300 Read block 1441 len 16384
504 280/300 Read block 3921 len 4096
505 281/300 Read block 334 len 8192
506 282/300 Read block 4391 len 40960
507 283/300 Write block 2021 len 4096 data: 2
508 284/300 Read block 1420 len 28672
509 285/300 Read block 4465 len 12288
510 286/300 Flush
511 287/300 Flush
512 288/300 Read block 4698 len 16384
513 289/300 Read block 3340 len 24576
514 290/300 Read block 4972 len 4096
515 291/300 Read block 2234 len 8192
516 292/300 Read block 3759 len 40960
517 293/300 Read block 2757 len 20480
518 294/300 Read block 305 len 32768
519 295/300 Read block 3827 len 20480
520 296/300 Read block 1541 len 36864
521 297/300 Read block 4476 len 36864
522 298/300 Read block 1822 len 4096
523 299/300 Read block 4638 len 24576
524 300/300 Read block 4917 len 8192
525 301/300 Flush
526 Main task has work to do, go find it
527 got dsc Start(1)
528 start 1
529 Feb 28 06:01:08.121 INFO request completed, latency_us: 248, response_code: 204, uri: /start/cid/1, method: POST, req_id: 5c66957c-1a76-4416-b394-effd9308bc25, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
530 Replay: started 1, returned:Ok(())
531 ----------------------------------------------------------------
532 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
533 States: Active Offline Active
534 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
535 1000 Acked Write 3 Done Sent Done false
536 1001 Acked Write 1 Done Sent Done false
537 1002 Acked Write 9 Done Sent Done false
538 1003 Acked Write 10 Done Sent Done false
539 1004 Acked Flush 0 Done Sent Done false
540 1005 Acked Read 10 Done Sent Done false
541 1006 Acked Read 2 Done Sent Done false
542 1007 Acked Write 10 Done Sent Done false
543 1008 Acked Flush 0 Done Sent Done false
544 1009 Acked Read 8 Done Sent Done false
545 1010 Acked Flush 0 Done Sent Done false
546 1011 Acked Read 9 Done Sent Done false
547 1012 Acked Read 1 Done Sent Done false
548 1013 Acked Read 2 Done Sent Done false
549 1014 Acked Read 8 Done Sent Done false
550 1015 Acked Read 9 Done Sent Done false
551 1016 Acked Read 3 Done Sent Done false
552 1017 Acked Read 9 Done Sent Done false
553 1018 Acked Write 5 Done Sent Done false
554 1019 Acked Read 1 Done Sent Done false
555 1020 Acked Write 6 Done Sent Done false
556 1021 Acked Write 6 Done Sent Done false
557 1022 Acked Read 2 Done Sent Done false
558 1023 Acked Write 8 Done Sent Done false
559 1024 Acked Read 6 Done Sent Done false
560 1025 Acked Read 9 Done Sent Done false
561 1026 Acked Write 4 Done Sent Done false
562 1027 Acked Write 2 Done Sent Done false
563 1028 Acked Read 3 Done Sent Done false
564 1029 Acked Write 2 Done Sent Done false
565 1030 Acked Write 8 Done Sent Done false
566 1031 Acked Read 3 Done Sent Done false
567 1032 Acked Flush 0 Done Sent Done false
568 1033 Acked Write 4 Done Sent Done false
569 1034 Acked Write 9 Done Sent Done false
570 1035 Acked Write 10 Done Sent Done false
571 1036 Acked Read 9 Done Sent Done false
572 1037 Acked Read 4 Done Sent Done false
573 1038 Acked Write 8 Done Sent Done false
574 1039 Acked Write 3 Done Sent Done false
575 1040 Acked Read 4 Done Sent Done false
576 1041 Acked Read 8 Done Sent Done false
577 1042 Acked Write 10 Done Sent Done false
578 1043 Acked Read 2 Done Sent Done false
579 1044 Acked Read 2 Done Sent Done false
580 1045 Acked Read 6 Done Sent Done false
581 1046 Acked Write 1 Done Sent Done false
582 1047 Acked Write 9 Done Sent Done false
583 1048 Acked Write 10 Done Sent Done false
584 1049 Acked Flush 0 Done Sent Done false
585 1050 Acked Flush 0 Done Sent Done false
586 1051 Acked Read 5 Done Sent Done false
587 1052 Acked Read 9 Done Sent Done false
588 1053 Acked Write 2 Done Sent Done false
589 1054 Acked Write 6 Done Sent Done false
590 1055 Acked Read 8 Done Sent Done false
591 1056 Acked Write 6 Done Sent Done false
592 1057 Acked Write 7 Done Sent Done false
593 1058 Acked Flush 0 Done Sent Done false
594 1059 Acked Write 4 Done Sent Done false
595 1060 Acked Write 9 Done Sent Done false
596 1061 Acked Write 6 Done Sent Done false
597 1062 Acked Read 10 Done Sent Done false
598 1063 Acked Read 1 Done Sent Done false
599 1064 Acked Write 10 Done Sent Done false
600 1065 Acked Read 8 Done Sent Done false
601 1066 Acked Write 2 Done Sent Done false
602 1067 Acked Flush 0 Done Sent Done false
603 1068 Acked Flush 0 Done Sent Done false
604 1069 Acked Write 4 Done Sent Done false
605 1070 Acked Read 1 Done Sent Done false
606 1071 Acked Write 7 Done Sent Done false
607 1072 Acked Read 8 Done Sent Done false
608 1073 Acked Write 5 Done Sent Done false
609 1074 Acked Read 8 Done Sent Done false
610 1075 Acked Write 2 Done Sent Done false
611 1076 Acked Read 3 Done Sent Done false
612 1077 Acked Write 8 Done Sent Done false
613 1078 Acked Read 2 Done Sent Done false
614 1079 Acked Read 4 Done Sent Done false
615 1080 Acked Write 5 Done Sent Done false
616 1081 Acked Write 10 Done Sent Done false
617 1082 Acked Write 10 Done Sent Done false
618 1083 Acked Read 4 Done Sent Done false
619 1084 Acked Write 3 Done Sent Done false
620 1085 Acked Read 10 Done Sent Done false
621 1086 Acked Read 8 Done Sent Done false
622 1087 Acked Write 2 Done Sent Done false
623 1088 Acked Write 8 Done Sent Done false
624 1089 Acked Write 6 Done Sent Done false
625 1090 Acked Flush 0 Done Sent Done false
626 1091 Acked Write 10 Done Sent Done false
627 1092 Acked Read 5 Done Sent Done false
628 1093 Acked Flush 0 Done Sent Done false
629 1094 Acked Write 8 Done Sent Done false
630 1095 Acked Read 4 Done Sent Done false
631 1096 Acked Read 2 Done Sent Done false
632 1097 Acked Read 8 Done Sent Done false
633 1098 Acked Read 2 Done Sent Done false
634 1099 Acked Write 7 Done Sent Done false
635 1100 Acked Read 5 Done Sent Done false
636 1101 Acked Read 8 Done Sent Done false
637 1102 Acked Read 6 Done Sent Done false
638 1103 Acked Flush 0 Done Sent Done false
639 1104 Acked Write 9 Done Sent Done false
640 1105 Acked Read 6 Done Sent Done false
641 1106 Acked Write 7 Done Sent Done false
642 1107 Acked Read 1 Done Sent Done false
643 1108 Acked Write 5 Done Sent Done false
644 1109 Acked Read 3 Done Sent Done false
645 1110 Acked Read 7 Done Sent Done false
646 1111 Acked Write 7 Done Sent Done false
647 1112 Acked Read 6 Done Sent Done false
648 1113 Acked Flush 0 Done Sent Done false
649 1114 Acked Read 4 Done Sent Done false
650 1115 Acked Write 2 Done Sent Done false
651 1116 Acked Write 7 Done Sent Done false
652 1117 Acked Flush 0 Done Sent Done false
653 1118 Acked Write 2 Done Sent Done false
654 1119 Acked Read 1 Done Sent Done false
655 1120 Acked Flush 0 Done Sent Done false
656 1121 Acked Read 1 Done Sent Done false
657 1122 Acked Write 2 Done Sent Done false
658 1123 Acked Read 7 Done Sent Done false
659 1124 Acked Read 10 Done Sent Done false
660 1125 Acked Flush 0 Done Sent Done false
661 1126 Acked Read 5 Done Sent Done false
662 1127 Acked Read 2 Done Sent Done false
663 1128 Acked Write 6 Done Sent Done false
664 1129 Acked Read 9 Done Sent Done false
665 1130 Acked Flush 0 Done Sent Done false
666 1131 Acked Read 2 Done Sent Done false
667 1132 Acked Read 2 Done Sent Done false
668 1133 Acked Write 5 Done Sent Done false
669 1134 Acked Read 7 Done Sent Done false
670 1135 Acked Read 7 Done Sent Done false
671 1136 Acked Read 1 Done Sent Done false
672 1137 Acked Read 4 Done Sent Done false
673 1138 Acked Read 1 Done Sent Done false
674 1139 Acked Read 9 Done Sent Done false
675 1140 Acked Flush 0 Done Sent Done false
676 1141 Acked Read 2 Done Sent Done false
677 1142 Acked Read 5 Done Sent Done false
678 1143 Acked Read 1 Done Sent Done false
679 1144 Acked Flush 0 Done Sent Done false
680 1145 Acked Write 10 Done Sent Done false
681 1146 Acked Read 2 Done Sent Done false
682 1147 Acked Write 9 Done Sent Done false
683 1148 Acked Read 9 Done Sent Done false
684 1149 Acked Write 1 Done Sent Done false
685 1150 Acked Flush 0 Done Sent Done false
686 1151 Acked Write 1 Done Sent Done false
687 1152 Acked Write 1 Done Sent Done false
688 1153 Acked Read 9 Done Sent Done false
689 1154 Acked Write 7 Done Sent Done false
690 1155 Acked Read 1 Done Sent Done false
691 1156 Acked Write 5 Done Sent Done false
692 1157 Acked Flush 0 Done Sent Done false
693 1158 Acked Write 1 Done Sent Done false
694 1159 Acked Write 4 Done Sent Done false
695 1160 Acked Read 8 Done Sent Done false
696 1161 Acked Write 1 Done Sent Done false
697 1162 Acked Read 7 Done Sent Done false
698 1163 Acked Read 2 Done Sent Done false
699 1164 Acked Read 3 Done Sent Done false
700 1165 Acked Read 2 Done Sent Done false
701 1166 Acked Write 8 Done Sent Done false
702 1167 Acked Read 9 Done Sent Done false
703 1168 Acked Read 2 Done Sent Done false
704 1169 Acked Write 4 Done Sent Done false
705 1170 Acked Write 4 Done Sent Done false
706 1171 Acked Read 10 Done Sent Done false
707 1172 Acked Read 8 Done Sent Done false
708 1173 Acked Read 4 Done Sent Done false
709 1174 Acked Write 5 Done Sent Done false
710 1175 Acked Write 6 Done Sent Done false
711 1176 Acked Read 1 Done Sent Done false
712 1177 Acked Read 8 Done Sent Done false
713 1178 Acked Write 7 Done Sent Done false
714 1179 Acked Read 1 Done Sent Done false
715 1180 Acked Write 1 Done Sent Done false
716 1181 Acked Read 4 Done Sent Done false
717 1182 Acked Write 6 Done Sent Done false
718 1183 Acked Write 6 Done Sent Done false
719 1184 Acked Write 7 Done Sent Done false
720 1185 Acked Read 9 Done Sent Done false
721 1186 Acked Write 6 Done Sent Done false
722 1187 Acked Flush 0 Done Sent Done false
723 1188 Acked Read 3 Done Sent Done false
724 1189 Acked Write 4 Done Sent Done false
725 1190 Acked Read 8 Done Sent Done false
726 1191 Acked Write 10 Done Sent Done false
727 1192 Acked Write 2 Done Sent Done false
728 1193 Acked Read 2 Done Sent Done false
729 1194 Acked Write 3 Done Sent Done false
730 1195 Acked Read 10 Done Sent Done false
731 1196 Acked Write 6 Done Sent Done false
732 1197 Acked Write 1 Done Sent Done false
733 1198 Acked Flush 0 Done Sent Done false
734 1199 Acked Read 6 Done Sent Done false
735 1200 Acked Write 4 Done Sent Done false
736 1201 Acked Write 4 Done Sent Done false
737 1202 Acked Read 1 Done Sent Done false
738 1203 Acked Read 9 Done Sent Done false
739 1204 Acked Read 1 Done Sent Done false
740 1205 Acked Read 4 Done Sent Done false
741 1206 Acked Read 3 Done Sent Done false
742 1207 Acked Write 2 Done Sent Done false
743 1208 Acked Read 4 Done Sent Done false
744 1209 Acked Write 6 Done Sent Done false
745 1210 Acked Write 5 Done Sent Done false
746 1211 Acked Read 1 Done Sent Done false
747 1212 Acked Read 1 Done Sent Done false
748 1213 Acked Write 5 Done Sent Done false
749 1214 Acked Write 2 Done Sent Done false
750 1215 Acked Flush 0 Done Sent Done false
751 1216 Acked Read 9 Done Sent Done false
752 1217 Acked Flush 0 Done Sent Done false
753 1218 Acked Write 8 Done Sent Done false
754 1219 Acked Read 7 Done Sent Done false
755 1220 Acked Write 8 Done Sent Done false
756 1221 Acked Read 5 Done Sent Done false
757 1222 Acked Flush 0 Done Sent Done false
758 1223 Acked Read 8 Done Sent Done false
759 1224 Acked Write 10 Done Sent Done false
760 1225 Acked Read 6 Done Sent Done false
761 1226 Acked Write 2 Done Sent Done false
762 1227 Acked Write 3 Done Sent Done false
763 1228 Acked Read 4 Done Sent Done false
764 1229 Acked Write 10 Done Sent Done false
765 1230 Acked Read 1 Done Sent Done false
766 1231 Acked Write 4 Done Sent Done false
767 1232 Acked Read 3 Done Sent Done false
768 1233 Acked Read 6 Done Sent Done false
769 1234 Acked Read 1 Done Sent Done false
770 1235 Acked Read 9 Done Sent Done false
771 1236 Acked Read 9 Done Sent Done false
772 1237 Acked Flush 0 Done Sent Done false
773 1238 Acked Read 7 Done Sent Done false
774 1239 Acked Read 9 Done Sent Done false
775 1240 Acked Write 1 Done Sent Done false
776 1241 Acked Read 6 Done Sent Done false
777 1242 Acked Read 3 Done Sent Done false
778 1243 Acked Read 7 Done Sent Done false
779 1244 Acked Write 1 Done Sent Done false
780 1245 Acked Flush 0 Done Sent Done false
781 1246 Acked Write 10 Done Sent Done false
782 1247 Acked Read 10 Done Sent Done false
783 1248 Acked Flush 0 Done Sent Done false
784 1249 Acked Read 7 Done Sent Done false
785 1250 Acked Flush 0 Done Sent Done false
786 1251 Acked Write 6 Done Sent Done false
787 1252 Acked Write 8 Done Sent Done false
788 1253 Acked Flush 0 Done Sent Done false
789 1254 Acked Read 5 Done Sent Done false
790 1255 Acked Write 4 Done Sent Done false
791 1256 Acked Write 8 Done Sent Done false
792 1257 Acked Write 4 Done Sent Done false
793 1258 Acked Read 1 Done Sent Done false
794 1259 Acked Read 3 Done Sent Done false
795 1260 Acked Read 4 Done Sent Done false
796 1261 Acked Write 4 Done Sent Done false
797 1262 Acked Read 1 Done Sent Done false
798 1263 Acked Write 10 Done Sent Done false
799 1264 Acked Read 7 Done Sent Done false
800 1265 Acked Read 3 Done Sent Done false
801 1266 Acked Read 8 Done Sent Done false
802 1267 Acked Read 9 Done Sent Done false
803 1268 Acked Write 7 Done Sent Done false
804 1269 Acked Read 6 Done Sent Done false
805 1270 Acked Write 6 Done Sent Done false
806 1271 Acked Flush 0 Done Sent Done false
807 1272 Acked Write 3 Done Sent Done false
808 1273 Acked Read 1 Done Sent Done false
809 1274 Acked Read 10 Done Sent Done false
810 1275 Acked Write 3 Done Sent Done false
811 1276 Acked Read 9 Done Sent Done false
812 1277 Acked Read 9 Done Sent Done false
813 1278 Acked Read 4 Done Sent Done false
814 1279 Acked Read 1 Done Sent Done false
815 1280 Acked Read 2 Done Sent Done false
816 1281 Acked Read 10 Done Sent Done false
817 1282 Acked Write 1 Done Sent Done false
818 1283 Acked Read 7 Done Sent Done false
819 1284 Acked Read 3 Done Sent Done false
820 1285 Acked Flush 0 Done Sent Done false
821 1286 Acked Flush 0 Done Sent Done false
822 1287 Acked Read 4 Done Sent Done false
823 1288 Acked Read 6 Done Sent Done false
824 1289 Acked Read 1 Done Sent Done false
825 1290 Acked Read 2 Done Sent Done false
826 1291 Acked Read 10 Done Sent Done false
827 1292 Acked Read 5 Done Sent Done false
828 1293 Acked Read 8 Done Sent Done false
829 1294 Acked Read 5 Done Sent Done false
830 1295 Acked Read 9 Done Sent Done false
831 1296 Acked Read 9 Done Sent Done false
832 1297 Acked Read 1 Done Sent Done false
833 1298 Acked Read 6 Done Sent Done false
834 1299 Acked Read 2 Done Sent Done false
835 1300 Acked Flush 0 Done Sent Done false
836 STATES DS:0 DS:1 DS:2 TOTAL
837 Sent 0 301 0 301
838 Done 301 0 301 602
839 Skipped 0 0 0 0
840 Error 0 0 0 0
841 Last Flush: Some(JobId(1300)) None Some(JobId(1300))
842 Downstairs last five completed:
843 Upstairs last five completed: 1300 1299 1298 1297 1296
844 CLIENT: Up:0 ds:301 act:2
8452025-02-28T06:01:08.623ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
8462025-02-28T06:01:09.350ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
8472025-02-28T06:01:10.352ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
8482025-02-28T06:01:11.354ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
849 ----------------------------------------------------------------
850 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
851 States: Active Offline Active
852 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
853 1000 Acked Write 3 Done Sent Done false
854 1001 Acked Write 1 Done Sent Done false
855 1002 Acked Write 9 Done Sent Done false
856 1003 Acked Write 10 Done Sent Done false
857 1004 Acked Flush 0 Done Sent Done false
858 1005 Acked Read 10 Done Sent Done false
859 1006 Acked Read 2 Done Sent Done false
860 1007 Acked Write 10 Done Sent Done false
861 1008 Acked Flush 0 Done Sent Done false
862 1009 Acked Read 8 Done Sent Done false
863 1010 Acked Flush 0 Done Sent Done false
864 1011 Acked Read 9 Done Sent Done false
865 1012 Acked Read 1 Done Sent Done false
866 1013 Acked Read 2 Done Sent Done false
867 1014 Acked Read 8 Done Sent Done false
868 1015 Acked Read 9 Done Sent Done false
869 1016 Acked Read 3 Done Sent Done false
870 1017 Acked Read 9 Done Sent Done false
871 1018 Acked Write 5 Done Sent Done false
872 1019 Acked Read 1 Done Sent Done false
873 1020 Acked Write 6 Done Sent Done false
874 1021 Acked Write 6 Done Sent Done false
875 1022 Acked Read 2 Done Sent Done false
876 1023 Acked Write 8 Done Sent Done false
877 1024 Acked Read 6 Done Sent Done false
878 1025 Acked Read 9 Done Sent Done false
879 1026 Acked Write 4 Done Sent Done false
880 1027 Acked Write 2 Done Sent Done false
881 1028 Acked Read 3 Done Sent Done false
882 1029 Acked Write 2 Done Sent Done false
883 1030 Acked Write 8 Done Sent Done false
884 1031 Acked Read 3 Done Sent Done false
885 1032 Acked Flush 0 Done Sent Done false
886 1033 Acked Write 4 Done Sent Done false
887 1034 Acked Write 9 Done Sent Done false
888 1035 Acked Write 10 Done Sent Done false
889 1036 Acked Read 9 Done Sent Done false
890 1037 Acked Read 4 Done Sent Done false
891 1038 Acked Write 8 Done Sent Done false
892 1039 Acked Write 3 Done Sent Done false
893 1040 Acked Read 4 Done Sent Done false
894 1041 Acked Read 8 Done Sent Done false
895 1042 Acked Write 10 Done Sent Done false
896 1043 Acked Read 2 Done Sent Done false
897 1044 Acked Read 2 Done Sent Done false
898 1045 Acked Read 6 Done Sent Done false
899 1046 Acked Write 1 Done Sent Done false
900 1047 Acked Write 9 Done Sent Done false
901 1048 Acked Write 10 Done Sent Done false
902 1049 Acked Flush 0 Done Sent Done false
903 1050 Acked Flush 0 Done Sent Done false
904 1051 Acked Read 5 Done Sent Done false
905 1052 Acked Read 9 Done Sent Done false
906 1053 Acked Write 2 Done Sent Done false
907 1054 Acked Write 6 Done Sent Done false
908 1055 Acked Read 8 Done Sent Done false
909 1056 Acked Write 6 Done Sent Done false
910 1057 Acked Write 7 Done Sent Done false
911 1058 Acked Flush 0 Done Sent Done false
912 1059 Acked Write 4 Done Sent Done false
913 1060 Acked Write 9 Done Sent Done false
914 1061 Acked Write 6 Done Sent Done false
915 1062 Acked Read 10 Done Sent Done false
916 1063 Acked Read 1 Done Sent Done false
917 1064 Acked Write 10 Done Sent Done false
918 1065 Acked Read 8 Done Sent Done false
919 1066 Acked Write 2 Done Sent Done false
920 1067 Acked Flush 0 Done Sent Done false
921 1068 Acked Flush 0 Done Sent Done false
922 1069 Acked Write 4 Done Sent Done false
923 1070 Acked Read 1 Done Sent Done false
924 1071 Acked Write 7 Done Sent Done false
925 1072 Acked Read 8 Done Sent Done false
926 1073 Acked Write 5 Done Sent Done false
927 1074 Acked Read 8 Done Sent Done false
928 1075 Acked Write 2 Done Sent Done false
929 1076 Acked Read 3 Done Sent Done false
930 1077 Acked Write 8 Done Sent Done false
931 1078 Acked Read 2 Done Sent Done false
932 1079 Acked Read 4 Done Sent Done false
933 1080 Acked Write 5 Done Sent Done false
934 1081 Acked Write 10 Done Sent Done false
935 1082 Acked Write 10 Done Sent Done false
936 1083 Acked Read 4 Done Sent Done false
937 1084 Acked Write 3 Done Sent Done false
938 1085 Acked Read 10 Done Sent Done false
939 1086 Acked Read 8 Done Sent Done false
940 1087 Acked Write 2 Done Sent Done false
941 1088 Acked Write 8 Done Sent Done false
942 1089 Acked Write 6 Done Sent Done false
943 1090 Acked Flush 0 Done Sent Done false
944 1091 Acked Write 10 Done Sent Done false
945 1092 Acked Read 5 Done Sent Done false
946 1093 Acked Flush 0 Done Sent Done false
947 1094 Acked Write 8 Done Sent Done false
948 1095 Acked Read 4 Done Sent Done false
949 1096 Acked Read 2 Done Sent Done false
950 1097 Acked Read 8 Done Sent Done false
951 1098 Acked Read 2 Done Sent Done false
952 1099 Acked Write 7 Done Sent Done false
953 1100 Acked Read 5 Done Sent Done false
954 1101 Acked Read 8 Done Sent Done false
955 1102 Acked Read 6 Done Sent Done false
956 1103 Acked Flush 0 Done Sent Done false
957 1104 Acked Write 9 Done Sent Done false
958 1105 Acked Read 6 Done Sent Done false
959 1106 Acked Write 7 Done Sent Done false
960 1107 Acked Read 1 Done Sent Done false
961 1108 Acked Write 5 Done Sent Done false
962 1109 Acked Read 3 Done Sent Done false
963 1110 Acked Read 7 Done Sent Done false
964 1111 Acked Write 7 Done Sent Done false
965 1112 Acked Read 6 Done Sent Done false
966 1113 Acked Flush 0 Done Sent Done false
967 1114 Acked Read 4 Done Sent Done false
968 1115 Acked Write 2 Done Sent Done false
969 1116 Acked Write 7 Done Sent Done false
970 1117 Acked Flush 0 Done Sent Done false
971 1118 Acked Write 2 Done Sent Done false
972 1119 Acked Read 1 Done Sent Done false
973 1120 Acked Flush 0 Done Sent Done false
974 1121 Acked Read 1 Done Sent Done false
975 1122 Acked Write 2 Done Sent Done false
976 1123 Acked Read 7 Done Sent Done false
977 1124 Acked Read 10 Done Sent Done false
978 1125 Acked Flush 0 Done Sent Done false
979 1126 Acked Read 5 Done Sent Done false
980 1127 Acked Read 2 Done Sent Done false
981 1128 Acked Write 6 Done Sent Done false
982 1129 Acked Read 9 Done Sent Done false
983 1130 Acked Flush 0 Done Sent Done false
984 1131 Acked Read 2 Done Sent Done false
985 1132 Acked Read 2 Done Sent Done false
986 1133 Acked Write 5 Done Sent Done false
987 1134 Acked Read 7 Done Sent Done false
988 1135 Acked Read 7 Done Sent Done false
989 1136 Acked Read 1 Done Sent Done false
990 1137 Acked Read 4 Done Sent Done false
991 1138 Acked Read 1 Done Sent Done false
992 1139 Acked Read 9 Done Sent Done false
993 1140 Acked Flush 0 Done Sent Done false
994 1141 Acked Read 2 Done Sent Done false
995 1142 Acked Read 5 Done Sent Done false
996 1143 Acked Read 1 Done Sent Done false
997 1144 Acked Flush 0 Done Sent Done false
998 1145 Acked Write 10 Done Sent Done false
999 1146 Acked Read 2 Done Sent Done false
1000 1147 Acked Write 9 Done Sent Done false
1001 1148 Acked Read 9 Done Sent Done false
1002 1149 Acked Write 1 Done Sent Done false
1003 1150 Acked Flush 0 Done Sent Done false
1004 1151 Acked Write 1 Done Sent Done false
1005 1152 Acked Write 1 Done Sent Done false
1006 1153 Acked Read 9 Done Sent Done false
1007 1154 Acked Write 7 Done Sent Done false
1008 1155 Acked Read 1 Done Sent Done false
1009 1156 Acked Write 5 Done Sent Done false
1010 1157 Acked Flush 0 Done Sent Done false
1011 1158 Acked Write 1 Done Sent Done false
1012 1159 Acked Write 4 Done Sent Done false
1013 1160 Acked Read 8 Done Sent Done false
1014 1161 Acked Write 1 Done Sent Done false
1015 1162 Acked Read 7 Done Sent Done false
1016 1163 Acked Read 2 Done Sent Done false
1017 1164 Acked Read 3 Done Sent Done false
1018 1165 Acked Read 2 Done Sent Done false
1019 1166 Acked Write 8 Done Sent Done false
1020 1167 Acked Read 9 Done Sent Done false
1021 1168 Acked Read 2 Done Sent Done false
1022 1169 Acked Write 4 Done Sent Done false
1023 1170 Acked Write 4 Done Sent Done false
1024 1171 Acked Read 10 Done Sent Done false
1025 1172 Acked Read 8 Done Sent Done false
1026 1173 Acked Read 4 Done Sent Done false
1027 1174 Acked Write 5 Done Sent Done false
1028 1175 Acked Write 6 Done Sent Done false
1029 1176 Acked Read 1 Done Sent Done false
1030 1177 Acked Read 8 Done Sent Done false
1031 1178 Acked Write 7 Done Sent Done false
1032 1179 Acked Read 1 Done Sent Done false
1033 1180 Acked Write 1 Done Sent Done false
1034 1181 Acked Read 4 Done Sent Done false
1035 1182 Acked Write 6 Done Sent Done false
1036 1183 Acked Write 6 Done Sent Done false
1037 1184 Acked Write 7 Done Sent Done false
1038 1185 Acked Read 9 Done Sent Done false
1039 1186 Acked Write 6 Done Sent Done false
1040 1187 Acked Flush 0 Done Sent Done false
1041 1188 Acked Read 3 Done Sent Done false
1042 1189 Acked Write 4 Done Sent Done false
1043 1190 Acked Read 8 Done Sent Done false
1044 1191 Acked Write 10 Done Sent Done false
1045 1192 Acked Write 2 Done Sent Done false
1046 1193 Acked Read 2 Done Sent Done false
1047 1194 Acked Write 3 Done Sent Done false
1048 1195 Acked Read 10 Done Sent Done false
1049 1196 Acked Write 6 Done Sent Done false
1050 1197 Acked Write 1 Done Sent Done false
1051 1198 Acked Flush 0 Done Sent Done false
1052 1199 Acked Read 6 Done Sent Done false
1053 1200 Acked Write 4 Done Sent Done false
1054 1201 Acked Write 4 Done Sent Done false
1055 1202 Acked Read 1 Done Sent Done false
1056 1203 Acked Read 9 Done Sent Done false
1057 1204 Acked Read 1 Done Sent Done false
1058 1205 Acked Read 4 Done Sent Done false
1059 1206 Acked Read 3 Done Sent Done false
1060 1207 Acked Write 2 Done Sent Done false
1061 1208 Acked Read 4 Done Sent Done false
1062 1209 Acked Write 6 Done Sent Done false
1063 1210 Acked Write 5 Done Sent Done false
1064 1211 Acked Read 1 Done Sent Done false
1065 1212 Acked Read 1 Done Sent Done false
1066 1213 Acked Write 5 Done Sent Done false
1067 1214 Acked Write 2 Done Sent Done false
1068 1215 Acked Flush 0 Done Sent Done false
1069 1216 Acked Read 9 Done Sent Done false
1070 1217 Acked Flush 0 Done Sent Done false
1071 1218 Acked Write 8 Done Sent Done false
1072 1219 Acked Read 7 Done Sent Done false
1073 1220 Acked Write 8 Done Sent Done false
1074 1221 Acked Read 5 Done Sent Done false
1075 1222 Acked Flush 0 Done Sent Done false
1076 1223 Acked Read 8 Done Sent Done false
1077 1224 Acked Write 10 Done Sent Done false
1078 1225 Acked Read 6 Done Sent Done false
1079 1226 Acked Write 2 Done Sent Done false
1080 1227 Acked Write 3 Done Sent Done false
1081 1228 Acked Read 4 Done Sent Done false
1082 1229 Acked Write 10 Done Sent Done false
1083 1230 Acked Read 1 Done Sent Done false
1084 1231 Acked Write 4 Done Sent Done false
1085 1232 Acked Read 3 Done Sent Done false
1086 1233 Acked Read 6 Done Sent Done false
1087 1234 Acked Read 1 Done Sent Done false
1088 1235 Acked Read 9 Done Sent Done false
1089 1236 Acked Read 9 Done Sent Done false
1090 1237 Acked Flush 0 Done Sent Done false
1091 1238 Acked Read 7 Done Sent Done false
1092 1239 Acked Read 9 Done Sent Done false
1093 1240 Acked Write 1 Done Sent Done false
1094 1241 Acked Read 6 Done Sent Done false
1095 1242 Acked Read 3 Done Sent Done false
1096 1243 Acked Read 7 Done Sent Done false
1097 1244 Acked Write 1 Done Sent Done false
1098 1245 Acked Flush 0 Done Sent Done false
1099 1246 Acked Write 10 Done Sent Done false
1100 1247 Acked Read 10 Done Sent Done false
1101 1248 Acked Flush 0 Done Sent Done false
1102 1249 Acked Read 7 Done Sent Done false
1103 1250 Acked Flush 0 Done Sent Done false
1104 1251 Acked Write 6 Done Sent Done false
1105 1252 Acked Write 8 Done Sent Done false
1106 1253 Acked Flush 0 Done Sent Done false
1107 1254 Acked Read 5 Done Sent Done false
1108 1255 Acked Write 4 Done Sent Done false
1109 1256 Acked Write 8 Done Sent Done false
1110 1257 Acked Write 4 Done Sent Done false
1111 1258 Acked Read 1 Done Sent Done false
1112 1259 Acked Read 3 Done Sent Done false
1113 1260 Acked Read 4 Done Sent Done false
1114 1261 Acked Write 4 Done Sent Done false
1115 1262 Acked Read 1 Done Sent Done false
1116 1263 Acked Write 10 Done Sent Done false
1117 1264 Acked Read 7 Done Sent Done false
1118 1265 Acked Read 3 Done Sent Done false
1119 1266 Acked Read 8 Done Sent Done false
1120 1267 Acked Read 9 Done Sent Done false
1121 1268 Acked Write 7 Done Sent Done false
1122 1269 Acked Read 6 Done Sent Done false
1123 1270 Acked Write 6 Done Sent Done false
1124 1271 Acked Flush 0 Done Sent Done false
1125 1272 Acked Write 3 Done Sent Done false
1126 1273 Acked Read 1 Done Sent Done false
1127 1274 Acked Read 10 Done Sent Done false
1128 1275 Acked Write 3 Done Sent Done false
1129 1276 Acked Read 9 Done Sent Done false
1130 1277 Acked Read 9 Done Sent Done false
1131 1278 Acked Read 4 Done Sent Done false
1132 1279 Acked Read 1 Done Sent Done false
1133 1280 Acked Read 2 Done Sent Done false
1134 1281 Acked Read 10 Done Sent Done false
1135 1282 Acked Write 1 Done Sent Done false
1136 1283 Acked Read 7 Done Sent Done false
1137 1284 Acked Read 3 Done Sent Done false
1138 1285 Acked Flush 0 Done Sent Done false
1139 1286 Acked Flush 0 Done Sent Done false
1140 1287 Acked Read 4 Done Sent Done false
1141 1288 Acked Read 6 Done Sent Done false
1142 1289 Acked Read 1 Done Sent Done false
1143 1290 Acked Read 2 Done Sent Done false
1144 1291 Acked Read 10 Done Sent Done false
1145 1292 Acked Read 5 Done Sent Done false
1146 1293 Acked Read 8 Done Sent Done false
1147 1294 Acked Read 5 Done Sent Done false
1148 1295 Acked Read 9 Done Sent Done false
1149 1296 Acked Read 9 Done Sent Done false
1150 1297 Acked Read 1 Done Sent Done false
1151 1298 Acked Read 6 Done Sent Done false
1152 1299 Acked Read 2 Done Sent Done false
1153 1300 Acked Flush 0 Done Sent Done false
1154 STATES DS:0 DS:1 DS:2 TOTAL
1155 Sent 0 301 0 301
1156 Done 301 0 301 602
1157 Skipped 0 0 0 0
1158 Error 0 0 0 0
1159 Last Flush: Some(JobId(1300)) None Some(JobId(1300))
1160 Downstairs last five completed:
1161 Upstairs last five completed: 1300 1299 1298 1297 1296
1162 CLIENT: Up:0 ds:301 act:2
11632025-02-28T06:01:12.625ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
11642025-02-28T06:01:13.358ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
11652025-02-28T06:01:13.879ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
11662025-02-28T06:01:13.879ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
11672025-02-28T06:01:13.879ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
11682025-02-28T06:01:13.879ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
11692025-02-28T06:01:13.879ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
11702025-02-28T06:01:14.380ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
11712025-02-28T06:01:15.361ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1172 [8820] got start action
1173 [8820] I am going to restart
1174 Starting downstairs at port 8820
1175 Make output file at "/tmp/dsc/downstairs-8820.txt"
1176 [8820][1] reports Starting
1177 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1212)
1178 [8820][1] reports Running
1179 ----------------------------------------------------------------
1180 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1181 States: Active Offline Active
1182 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1183 1000 Acked Write 3 Done Sent Done false
1184 1001 Acked Write 1 Done Sent Done false
1185 1002 Acked Write 9 Done Sent Done false
1186 1003 Acked Write 10 Done Sent Done false
1187 1004 Acked Flush 0 Done Sent Done false
1188 1005 Acked Read 10 Done Sent Done false
1189 1006 Acked Read 2 Done Sent Done false
1190 1007 Acked Write 10 Done Sent Done false
1191 1008 Acked Flush 0 Done Sent Done false
1192 1009 Acked Read 8 Done Sent Done false
1193 1010 Acked Flush 0 Done Sent Done false
1194 1011 Acked Read 9 Done Sent Done false
1195 1012 Acked Read 1 Done Sent Done false
1196 1013 Acked Read 2 Done Sent Done false
1197 1014 Acked Read 8 Done Sent Done false
1198 1015 Acked Read 9 Done Sent Done false
1199 1016 Acked Read 3 Done Sent Done false
1200 1017 Acked Read 9 Done Sent Done false
1201 1018 Acked Write 5 Done Sent Done false
1202 1019 Acked Read 1 Done Sent Done false
1203 1020 Acked Write 6 Done Sent Done false
1204 1021 Acked Write 6 Done Sent Done false
1205 1022 Acked Read 2 Done Sent Done false
1206 1023 Acked Write 8 Done Sent Done false
1207 1024 Acked Read 6 Done Sent Done false
1208 1025 Acked Read 9 Done Sent Done false
1209 1026 Acked Write 4 Done Sent Done false
1210 1027 Acked Write 2 Done Sent Done false
1211 1028 Acked Read 3 Done Sent Done false
1212 1029 Acked Write 2 Done Sent Done false
1213 1030 Acked Write 8 Done Sent Done false
1214 1031 Acked Read 3 Done Sent Done false
1215 1032 Acked Flush 0 Done Sent Done false
1216 1033 Acked Write 4 Done Sent Done false
1217 1034 Acked Write 9 Done Sent Done false
1218 1035 Acked Write 10 Done Sent Done false
1219 1036 Acked Read 9 Done Sent Done false
1220 1037 Acked Read 4 Done Sent Done false
1221 1038 Acked Write 8 Done Sent Done false
1222 1039 Acked Write 3 Done Sent Done false
1223 1040 Acked Read 4 Done Sent Done false
1224 1041 Acked Read 8 Done Sent Done false
1225 1042 Acked Write 10 Done Sent Done false
1226 1043 Acked Read 2 Done Sent Done false
1227 1044 Acked Read 2 Done Sent Done false
1228 1045 Acked Read 6 Done Sent Done false
1229 1046 Acked Write 1 Done Sent Done false
1230 1047 Acked Write 9 Done Sent Done false
1231 1048 Acked Write 10 Done Sent Done false
1232 1049 Acked Flush 0 Done Sent Done false
1233 1050 Acked Flush 0 Done Sent Done false
1234 1051 Acked Read 5 Done Sent Done false
1235 1052 Acked Read 9 Done Sent Done false
1236 1053 Acked Write 2 Done Sent Done false
1237 1054 Acked Write 6 Done Sent Done false
1238 1055 Acked Read 8 Done Sent Done false
1239 1056 Acked Write 6 Done Sent Done false
1240 1057 Acked Write 7 Done Sent Done false
1241 1058 Acked Flush 0 Done Sent Done false
1242 1059 Acked Write 4 Done Sent Done false
1243 1060 Acked Write 9 Done Sent Done false
1244 1061 Acked Write 6 Done Sent Done false
1245 1062 Acked Read 10 Done Sent Done false
1246 1063 Acked Read 1 Done Sent Done false
1247 1064 Acked Write 10 Done Sent Done false
1248 1065 Acked Read 8 Done Sent Done false
1249 1066 Acked Write 2 Done Sent Done false
1250 1067 Acked Flush 0 Done Sent Done false
1251 1068 Acked Flush 0 Done Sent Done false
1252 1069 Acked Write 4 Done Sent Done false
1253 1070 Acked Read 1 Done Sent Done false
1254 1071 Acked Write 7 Done Sent Done false
1255 1072 Acked Read 8 Done Sent Done false
1256 1073 Acked Write 5 Done Sent Done false
1257 1074 Acked Read 8 Done Sent Done false
1258 1075 Acked Write 2 Done Sent Done false
1259 1076 Acked Read 3 Done Sent Done false
1260 1077 Acked Write 8 Done Sent Done false
1261 1078 Acked Read 2 Done Sent Done false
1262 1079 Acked Read 4 Done Sent Done false
1263 1080 Acked Write 5 Done Sent Done false
1264 1081 Acked Write 10 Done Sent Done false
1265 1082 Acked Write 10 Done Sent Done false
1266 1083 Acked Read 4 Done Sent Done false
1267 1084 Acked Write 3 Done Sent Done false
1268 1085 Acked Read 10 Done Sent Done false
1269 1086 Acked Read 8 Done Sent Done false
1270 1087 Acked Write 2 Done Sent Done false
1271 1088 Acked Write 8 Done Sent Done false
1272 1089 Acked Write 6 Done Sent Done false
1273 1090 Acked Flush 0 Done Sent Done false
1274 1091 Acked Write 10 Done Sent Done false
1275 1092 Acked Read 5 Done Sent Done false
1276 1093 Acked Flush 0 Done Sent Done false
1277 1094 Acked Write 8 Done Sent Done false
1278 1095 Acked Read 4 Done Sent Done false
1279 1096 Acked Read 2 Done Sent Done false
1280 1097 Acked Read 8 Done Sent Done false
1281 1098 Acked Read 2 Done Sent Done false
1282 1099 Acked Write 7 Done Sent Done false
1283 1100 Acked Read 5 Done Sent Done false
1284 1101 Acked Read 8 Done Sent Done false
1285 1102 Acked Read 6 Done Sent Done false
1286 1103 Acked Flush 0 Done Sent Done false
1287 1104 Acked Write 9 Done Sent Done false
1288 1105 Acked Read 6 Done Sent Done false
1289 1106 Acked Write 7 Done Sent Done false
1290 1107 Acked Read 1 Done Sent Done false
1291 1108 Acked Write 5 Done Sent Done false
1292 1109 Acked Read 3 Done Sent Done false
1293 1110 Acked Read 7 Done Sent Done false
1294 1111 Acked Write 7 Done Sent Done false
1295 1112 Acked Read 6 Done Sent Done false
1296 1113 Acked Flush 0 Done Sent Done false
1297 1114 Acked Read 4 Done Sent Done false
1298 1115 Acked Write 2 Done Sent Done false
1299 1116 Acked Write 7 Done Sent Done false
1300 1117 Acked Flush 0 Done Sent Done false
1301 1118 Acked Write 2 Done Sent Done false
1302 1119 Acked Read 1 Done Sent Done false
1303 1120 Acked Flush 0 Done Sent Done false
1304 1121 Acked Read 1 Done Sent Done false
1305 1122 Acked Write 2 Done Sent Done false
1306 1123 Acked Read 7 Done Sent Done false
1307 1124 Acked Read 10 Done Sent Done false
1308 1125 Acked Flush 0 Done Sent Done false
1309 1126 Acked Read 5 Done Sent Done false
1310 1127 Acked Read 2 Done Sent Done false
1311 1128 Acked Write 6 Done Sent Done false
1312 1129 Acked Read 9 Done Sent Done false
1313 1130 Acked Flush 0 Done Sent Done false
1314 1131 Acked Read 2 Done Sent Done false
1315 1132 Acked Read 2 Done Sent Done false
1316 1133 Acked Write 5 Done Sent Done false
1317 1134 Acked Read 7 Done Sent Done false
1318 1135 Acked Read 7 Done Sent Done false
1319 1136 Acked Read 1 Done Sent Done false
1320 1137 Acked Read 4 Done Sent Done false
1321 1138 Acked Read 1 Done Sent Done false
1322 1139 Acked Read 9 Done Sent Done false
1323 1140 Acked Flush 0 Done Sent Done false
1324 1141 Acked Read 2 Done Sent Done false
1325 1142 Acked Read 5 Done Sent Done false
1326 1143 Acked Read 1 Done Sent Done false
1327 1144 Acked Flush 0 Done Sent Done false
1328 1145 Acked Write 10 Done Sent Done false
1329 1146 Acked Read 2 Done Sent Done false
1330 1147 Acked Write 9 Done Sent Done false
1331 1148 Acked Read 9 Done Sent Done false
1332 1149 Acked Write 1 Done Sent Done false
1333 1150 Acked Flush 0 Done Sent Done false
1334 1151 Acked Write 1 Done Sent Done false
1335 1152 Acked Write 1 Done Sent Done false
1336 1153 Acked Read 9 Done Sent Done false
1337 1154 Acked Write 7 Done Sent Done false
1338 1155 Acked Read 1 Done Sent Done false
1339 1156 Acked Write 5 Done Sent Done false
1340 1157 Acked Flush 0 Done Sent Done false
1341 1158 Acked Write 1 Done Sent Done false
1342 1159 Acked Write 4 Done Sent Done false
1343 1160 Acked Read 8 Done Sent Done false
1344 1161 Acked Write 1 Done Sent Done false
1345 1162 Acked Read 7 Done Sent Done false
1346 1163 Acked Read 2 Done Sent Done false
1347 1164 Acked Read 3 Done Sent Done false
1348 1165 Acked Read 2 Done Sent Done false
1349 1166 Acked Write 8 Done Sent Done false
1350 1167 Acked Read 9 Done Sent Done false
1351 1168 Acked Read 2 Done Sent Done false
1352 1169 Acked Write 4 Done Sent Done false
1353 1170 Acked Write 4 Done Sent Done false
1354 1171 Acked Read 10 Done Sent Done false
1355 1172 Acked Read 8 Done Sent Done false
1356 1173 Acked Read 4 Done Sent Done false
1357 1174 Acked Write 5 Done Sent Done false
1358 1175 Acked Write 6 Done Sent Done false
1359 1176 Acked Read 1 Done Sent Done false
1360 1177 Acked Read 8 Done Sent Done false
1361 1178 Acked Write 7 Done Sent Done false
1362 1179 Acked Read 1 Done Sent Done false
1363 1180 Acked Write 1 Done Sent Done false
1364 1181 Acked Read 4 Done Sent Done false
1365 1182 Acked Write 6 Done Sent Done false
1366 1183 Acked Write 6 Done Sent Done false
1367 1184 Acked Write 7 Done Sent Done false
1368 1185 Acked Read 9 Done Sent Done false
1369 1186 Acked Write 6 Done Sent Done false
1370 1187 Acked Flush 0 Done Sent Done false
1371 1188 Acked Read 3 Done Sent Done false
1372 1189 Acked Write 4 Done Sent Done false
1373 1190 Acked Read 8 Done Sent Done false
1374 1191 Acked Write 10 Done Sent Done false
1375 1192 Acked Write 2 Done Sent Done false
1376 1193 Acked Read 2 Done Sent Done false
1377 1194 Acked Write 3 Done Sent Done false
1378 1195 Acked Read 10 Done Sent Done false
1379 1196 Acked Write 6 Done Sent Done false
1380 1197 Acked Write 1 Done Sent Done false
1381 1198 Acked Flush 0 Done Sent Done false
1382 1199 Acked Read 6 Done Sent Done false
1383 1200 Acked Write 4 Done Sent Done false
1384 1201 Acked Write 4 Done Sent Done false
1385 1202 Acked Read 1 Done Sent Done false
1386 1203 Acked Read 9 Done Sent Done false
1387 1204 Acked Read 1 Done Sent Done false
1388 1205 Acked Read 4 Done Sent Done false
1389 1206 Acked Read 3 Done Sent Done false
1390 1207 Acked Write 2 Done Sent Done false
1391 1208 Acked Read 4 Done Sent Done false
1392 1209 Acked Write 6 Done Sent Done false
1393 1210 Acked Write 5 Done Sent Done false
1394 1211 Acked Read 1 Done Sent Done false
1395 1212 Acked Read 1 Done Sent Done false
1396 1213 Acked Write 5 Done Sent Done false
1397 1214 Acked Write 2 Done Sent Done false
1398 1215 Acked Flush 0 Done Sent Done false
1399 1216 Acked Read 9 Done Sent Done false
1400 1217 Acked Flush 0 Done Sent Done false
1401 1218 Acked Write 8 Done Sent Done false
1402 1219 Acked Read 7 Done Sent Done false
1403 1220 Acked Write 8 Done Sent Done false
1404 1221 Acked Read 5 Done Sent Done false
1405 1222 Acked Flush 0 Done Sent Done false
1406 1223 Acked Read 8 Done Sent Done false
1407 1224 Acked Write 10 Done Sent Done false
1408 1225 Acked Read 6 Done Sent Done false
1409 1226 Acked Write 2 Done Sent Done false
1410 1227 Acked Write 3 Done Sent Done false
1411 1228 Acked Read 4 Done Sent Done false
1412 1229 Acked Write 10 Done Sent Done false
1413 1230 Acked Read 1 Done Sent Done false
1414 1231 Acked Write 4 Done Sent Done false
1415 1232 Acked Read 3 Done Sent Done false
1416 1233 Acked Read 6 Done Sent Done false
1417 1234 Acked Read 1 Done Sent Done false
1418 1235 Acked Read 9 Done Sent Done false
1419 1236 Acked Read 9 Done Sent Done false
1420 1237 Acked Flush 0 Done Sent Done false
1421 1238 Acked Read 7 Done Sent Done false
1422 1239 Acked Read 9 Done Sent Done false
1423 1240 Acked Write 1 Done Sent Done false
1424 1241 Acked Read 6 Done Sent Done false
1425 1242 Acked Read 3 Done Sent Done false
1426 1243 Acked Read 7 Done Sent Done false
1427 1244 Acked Write 1 Done Sent Done false
1428 1245 Acked Flush 0 Done Sent Done false
1429 1246 Acked Write 10 Done Sent Done false
1430 1247 Acked Read 10 Done Sent Done false
1431 1248 Acked Flush 0 Done Sent Done false
1432 1249 Acked Read 7 Done Sent Done false
1433 1250 Acked Flush 0 Done Sent Done false
1434 1251 Acked Write 6 Done Sent Done false
1435 1252 Acked Write 8 Done Sent Done false
1436 1253 Acked Flush 0 Done Sent Done false
1437 1254 Acked Read 5 Done Sent Done false
1438 1255 Acked Write 4 Done Sent Done false
1439 1256 Acked Write 8 Done Sent Done false
1440 1257 Acked Write 4 Done Sent Done false
1441 1258 Acked Read 1 Done Sent Done false
1442 1259 Acked Read 3 Done Sent Done false
1443 1260 Acked Read 4 Done Sent Done false
1444 1261 Acked Write 4 Done Sent Done false
1445 1262 Acked Read 1 Done Sent Done false
1446 1263 Acked Write 10 Done Sent Done false
1447 1264 Acked Read 7 Done Sent Done false
1448 1265 Acked Read 3 Done Sent Done false
1449 1266 Acked Read 8 Done Sent Done false
1450 1267 Acked Read 9 Done Sent Done false
1451 1268 Acked Write 7 Done Sent Done false
1452 1269 Acked Read 6 Done Sent Done false
1453 1270 Acked Write 6 Done Sent Done false
1454 1271 Acked Flush 0 Done Sent Done false
1455 1272 Acked Write 3 Done Sent Done false
1456 1273 Acked Read 1 Done Sent Done false
1457 1274 Acked Read 10 Done Sent Done false
1458 1275 Acked Write 3 Done Sent Done false
1459 1276 Acked Read 9 Done Sent Done false
1460 1277 Acked Read 9 Done Sent Done false
1461 1278 Acked Read 4 Done Sent Done false
1462 1279 Acked Read 1 Done Sent Done false
1463 1280 Acked Read 2 Done Sent Done false
1464 1281 Acked Read 10 Done Sent Done false
1465 1282 Acked Write 1 Done Sent Done false
1466 1283 Acked Read 7 Done Sent Done false
1467 1284 Acked Read 3 Done Sent Done false
1468 1285 Acked Flush 0 Done Sent Done false
1469 1286 Acked Flush 0 Done Sent Done false
1470 1287 Acked Read 4 Done Sent Done false
1471 1288 Acked Read 6 Done Sent Done false
1472 1289 Acked Read 1 Done Sent Done false
1473 1290 Acked Read 2 Done Sent Done false
1474 1291 Acked Read 10 Done Sent Done false
1475 1292 Acked Read 5 Done Sent Done false
1476 1293 Acked Read 8 Done Sent Done false
1477 1294 Acked Read 5 Done Sent Done false
1478 1295 Acked Read 9 Done Sent Done false
1479 1296 Acked Read 9 Done Sent Done false
1480 1297 Acked Read 1 Done Sent Done false
1481 1298 Acked Read 6 Done Sent Done false
1482 1299 Acked Read 2 Done Sent Done false
1483 1300 Acked Flush 0 Done Sent Done false
1484 STATES DS:0 DS:1 DS:2 TOTAL
1485 Sent 0 301 0 301
1486 Done 301 0 301 602
1487 Skipped 0 0 0 0
1488 Error 0 0 0 0
1489 Last Flush: Some(JobId(1300)) None Some(JobId(1300))
1490 Downstairs last five completed:
1491 Upstairs last five completed: 1300 1299 1298 1297 1296
1492 CLIENT: Up:0 ds:301 act:2
14932025-02-28T06:01:16.627ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
14942025-02-28T06:01:17.364ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
14952025-02-28T06:01:18.365ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
14962025-02-28T06:01:19.367ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1497 ----------------------------------------------------------------
1498 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1499 States: Active Offline Active
1500 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1501 1000 Acked Write 3 Done Sent Done false
1502 1001 Acked Write 1 Done Sent Done false
1503 1002 Acked Write 9 Done Sent Done false
1504 1003 Acked Write 10 Done Sent Done false
1505 1004 Acked Flush 0 Done Sent Done false
1506 1005 Acked Read 10 Done Sent Done false
1507 1006 Acked Read 2 Done Sent Done false
1508 1007 Acked Write 10 Done Sent Done false
1509 1008 Acked Flush 0 Done Sent Done false
1510 1009 Acked Read 8 Done Sent Done false
1511 1010 Acked Flush 0 Done Sent Done false
1512 1011 Acked Read 9 Done Sent Done false
1513 1012 Acked Read 1 Done Sent Done false
1514 1013 Acked Read 2 Done Sent Done false
1515 1014 Acked Read 8 Done Sent Done false
1516 1015 Acked Read 9 Done Sent Done false
1517 1016 Acked Read 3 Done Sent Done false
1518 1017 Acked Read 9 Done Sent Done false
1519 1018 Acked Write 5 Done Sent Done false
1520 1019 Acked Read 1 Done Sent Done false
1521 1020 Acked Write 6 Done Sent Done false
1522 1021 Acked Write 6 Done Sent Done false
1523 1022 Acked Read 2 Done Sent Done false
1524 1023 Acked Write 8 Done Sent Done false
1525 1024 Acked Read 6 Done Sent Done false
1526 1025 Acked Read 9 Done Sent Done false
1527 1026 Acked Write 4 Done Sent Done false
1528 1027 Acked Write 2 Done Sent Done false
1529 1028 Acked Read 3 Done Sent Done false
1530 1029 Acked Write 2 Done Sent Done false
1531 1030 Acked Write 8 Done Sent Done false
1532 1031 Acked Read 3 Done Sent Done false
1533 1032 Acked Flush 0 Done Sent Done false
1534 1033 Acked Write 4 Done Sent Done false
1535 1034 Acked Write 9 Done Sent Done false
1536 1035 Acked Write 10 Done Sent Done false
1537 1036 Acked Read 9 Done Sent Done false
1538 1037 Acked Read 4 Done Sent Done false
1539 1038 Acked Write 8 Done Sent Done false
1540 1039 Acked Write 3 Done Sent Done false
1541 1040 Acked Read 4 Done Sent Done false
1542 1041 Acked Read 8 Done Sent Done false
1543 1042 Acked Write 10 Done Sent Done false
1544 1043 Acked Read 2 Done Sent Done false
1545 1044 Acked Read 2 Done Sent Done false
1546 1045 Acked Read 6 Done Sent Done false
1547 1046 Acked Write 1 Done Sent Done false
1548 1047 Acked Write 9 Done Sent Done false
1549 1048 Acked Write 10 Done Sent Done false
1550 1049 Acked Flush 0 Done Sent Done false
1551 1050 Acked Flush 0 Done Sent Done false
1552 1051 Acked Read 5 Done Sent Done false
1553 1052 Acked Read 9 Done Sent Done false
1554 1053 Acked Write 2 Done Sent Done false
1555 1054 Acked Write 6 Done Sent Done false
1556 1055 Acked Read 8 Done Sent Done false
1557 1056 Acked Write 6 Done Sent Done false
1558 1057 Acked Write 7 Done Sent Done false
1559 1058 Acked Flush 0 Done Sent Done false
1560 1059 Acked Write 4 Done Sent Done false
1561 1060 Acked Write 9 Done Sent Done false
1562 1061 Acked Write 6 Done Sent Done false
1563 1062 Acked Read 10 Done Sent Done false
1564 1063 Acked Read 1 Done Sent Done false
1565 1064 Acked Write 10 Done Sent Done false
1566 1065 Acked Read 8 Done Sent Done false
1567 1066 Acked Write 2 Done Sent Done false
1568 1067 Acked Flush 0 Done Sent Done false
1569 1068 Acked Flush 0 Done Sent Done false
1570 1069 Acked Write 4 Done Sent Done false
1571 1070 Acked Read 1 Done Sent Done false
1572 1071 Acked Write 7 Done Sent Done false
1573 1072 Acked Read 8 Done Sent Done false
1574 1073 Acked Write 5 Done Sent Done false
1575 1074 Acked Read 8 Done Sent Done false
1576 1075 Acked Write 2 Done Sent Done false
1577 1076 Acked Read 3 Done Sent Done false
1578 1077 Acked Write 8 Done Sent Done false
1579 1078 Acked Read 2 Done Sent Done false
1580 1079 Acked Read 4 Done Sent Done false
1581 1080 Acked Write 5 Done Sent Done false
1582 1081 Acked Write 10 Done Sent Done false
1583 1082 Acked Write 10 Done Sent Done false
1584 1083 Acked Read 4 Done Sent Done false
1585 1084 Acked Write 3 Done Sent Done false
1586 1085 Acked Read 10 Done Sent Done false
1587 1086 Acked Read 8 Done Sent Done false
1588 1087 Acked Write 2 Done Sent Done false
1589 1088 Acked Write 8 Done Sent Done false
1590 1089 Acked Write 6 Done Sent Done false
1591 1090 Acked Flush 0 Done Sent Done false
1592 1091 Acked Write 10 Done Sent Done false
1593 1092 Acked Read 5 Done Sent Done false
1594 1093 Acked Flush 0 Done Sent Done false
1595 1094 Acked Write 8 Done Sent Done false
1596 1095 Acked Read 4 Done Sent Done false
1597 1096 Acked Read 2 Done Sent Done false
1598 1097 Acked Read 8 Done Sent Done false
1599 1098 Acked Read 2 Done Sent Done false
1600 1099 Acked Write 7 Done Sent Done false
1601 1100 Acked Read 5 Done Sent Done false
1602 1101 Acked Read 8 Done Sent Done false
1603 1102 Acked Read 6 Done Sent Done false
1604 1103 Acked Flush 0 Done Sent Done false
1605 1104 Acked Write 9 Done Sent Done false
1606 1105 Acked Read 6 Done Sent Done false
1607 1106 Acked Write 7 Done Sent Done false
1608 1107 Acked Read 1 Done Sent Done false
1609 1108 Acked Write 5 Done Sent Done false
1610 1109 Acked Read 3 Done Sent Done false
1611 1110 Acked Read 7 Done Sent Done false
1612 1111 Acked Write 7 Done Sent Done false
1613 1112 Acked Read 6 Done Sent Done false
1614 1113 Acked Flush 0 Done Sent Done false
1615 1114 Acked Read 4 Done Sent Done false
1616 1115 Acked Write 2 Done Sent Done false
1617 1116 Acked Write 7 Done Sent Done false
1618 1117 Acked Flush 0 Done Sent Done false
1619 1118 Acked Write 2 Done Sent Done false
1620 1119 Acked Read 1 Done Sent Done false
1621 1120 Acked Flush 0 Done Sent Done false
1622 1121 Acked Read 1 Done Sent Done false
1623 1122 Acked Write 2 Done Sent Done false
1624 1123 Acked Read 7 Done Sent Done false
1625 1124 Acked Read 10 Done Sent Done false
1626 1125 Acked Flush 0 Done Sent Done false
1627 1126 Acked Read 5 Done Sent Done false
1628 1127 Acked Read 2 Done Sent Done false
1629 1128 Acked Write 6 Done Sent Done false
1630 1129 Acked Read 9 Done Sent Done false
1631 1130 Acked Flush 0 Done Sent Done false
1632 1131 Acked Read 2 Done Sent Done false
1633 1132 Acked Read 2 Done Sent Done false
1634 1133 Acked Write 5 Done Sent Done false
1635 1134 Acked Read 7 Done Sent Done false
1636 1135 Acked Read 7 Done Sent Done false
1637 1136 Acked Read 1 Done Sent Done false
1638 1137 Acked Read 4 Done Sent Done false
1639 1138 Acked Read 1 Done Sent Done false
1640 1139 Acked Read 9 Done Sent Done false
1641 1140 Acked Flush 0 Done Sent Done false
1642 1141 Acked Read 2 Done Sent Done false
1643 1142 Acked Read 5 Done Sent Done false
1644 1143 Acked Read 1 Done Sent Done false
1645 1144 Acked Flush 0 Done Sent Done false
1646 1145 Acked Write 10 Done Sent Done false
1647 1146 Acked Read 2 Done Sent Done false
1648 1147 Acked Write 9 Done Sent Done false
1649 1148 Acked Read 9 Done Sent Done false
1650 1149 Acked Write 1 Done Sent Done false
1651 1150 Acked Flush 0 Done Sent Done false
1652 1151 Acked Write 1 Done Sent Done false
1653 1152 Acked Write 1 Done Sent Done false
1654 1153 Acked Read 9 Done Sent Done false
1655 1154 Acked Write 7 Done Sent Done false
1656 1155 Acked Read 1 Done Sent Done false
1657 1156 Acked Write 5 Done Sent Done false
1658 1157 Acked Flush 0 Done Sent Done false
1659 1158 Acked Write 1 Done Sent Done false
1660 1159 Acked Write 4 Done Sent Done false
1661 1160 Acked Read 8 Done Sent Done false
1662 1161 Acked Write 1 Done Sent Done false
1663 1162 Acked Read 7 Done Sent Done false
1664 1163 Acked Read 2 Done Sent Done false
1665 1164 Acked Read 3 Done Sent Done false
1666 1165 Acked Read 2 Done Sent Done false
1667 1166 Acked Write 8 Done Sent Done false
1668 1167 Acked Read 9 Done Sent Done false
1669 1168 Acked Read 2 Done Sent Done false
1670 1169 Acked Write 4 Done Sent Done false
1671 1170 Acked Write 4 Done Sent Done false
1672 1171 Acked Read 10 Done Sent Done false
1673 1172 Acked Read 8 Done Sent Done false
1674 1173 Acked Read 4 Done Sent Done false
1675 1174 Acked Write 5 Done Sent Done false
1676 1175 Acked Write 6 Done Sent Done false
1677 1176 Acked Read 1 Done Sent Done false
1678 1177 Acked Read 8 Done Sent Done false
1679 1178 Acked Write 7 Done Sent Done false
1680 1179 Acked Read 1 Done Sent Done false
1681 1180 Acked Write 1 Done Sent Done false
1682 1181 Acked Read 4 Done Sent Done false
1683 1182 Acked Write 6 Done Sent Done false
1684 1183 Acked Write 6 Done Sent Done false
1685 1184 Acked Write 7 Done Sent Done false
1686 1185 Acked Read 9 Done Sent Done false
1687 1186 Acked Write 6 Done Sent Done false
1688 1187 Acked Flush 0 Done Sent Done false
1689 1188 Acked Read 3 Done Sent Done false
1690 1189 Acked Write 4 Done Sent Done false
1691 1190 Acked Read 8 Done Sent Done false
1692 1191 Acked Write 10 Done Sent Done false
1693 1192 Acked Write 2 Done Sent Done false
1694 1193 Acked Read 2 Done Sent Done false
1695 1194 Acked Write 3 Done Sent Done false
1696 1195 Acked Read 10 Done Sent Done false
1697 1196 Acked Write 6 Done Sent Done false
1698 1197 Acked Write 1 Done Sent Done false
1699 1198 Acked Flush 0 Done Sent Done false
1700 1199 Acked Read 6 Done Sent Done false
1701 1200 Acked Write 4 Done Sent Done false
1702 1201 Acked Write 4 Done Sent Done false
1703 1202 Acked Read 1 Done Sent Done false
1704 1203 Acked Read 9 Done Sent Done false
1705 1204 Acked Read 1 Done Sent Done false
1706 1205 Acked Read 4 Done Sent Done false
1707 1206 Acked Read 3 Done Sent Done false
1708 1207 Acked Write 2 Done Sent Done false
1709 1208 Acked Read 4 Done Sent Done false
1710 1209 Acked Write 6 Done Sent Done false
1711 1210 Acked Write 5 Done Sent Done false
1712 1211 Acked Read 1 Done Sent Done false
1713 1212 Acked Read 1 Done Sent Done false
1714 1213 Acked Write 5 Done Sent Done false
1715 1214 Acked Write 2 Done Sent Done false
1716 1215 Acked Flush 0 Done Sent Done false
1717 1216 Acked Read 9 Done Sent Done false
1718 1217 Acked Flush 0 Done Sent Done false
1719 1218 Acked Write 8 Done Sent Done false
1720 1219 Acked Read 7 Done Sent Done false
1721 1220 Acked Write 8 Done Sent Done false
1722 1221 Acked Read 5 Done Sent Done false
1723 1222 Acked Flush 0 Done Sent Done false
1724 1223 Acked Read 8 Done Sent Done false
1725 1224 Acked Write 10 Done Sent Done false
1726 1225 Acked Read 6 Done Sent Done false
1727 1226 Acked Write 2 Done Sent Done false
1728 1227 Acked Write 3 Done Sent Done false
1729 1228 Acked Read 4 Done Sent Done false
1730 1229 Acked Write 10 Done Sent Done false
1731 1230 Acked Read 1 Done Sent Done false
1732 1231 Acked Write 4 Done Sent Done false
1733 1232 Acked Read 3 Done Sent Done false
1734 1233 Acked Read 6 Done Sent Done false
1735 1234 Acked Read 1 Done Sent Done false
1736 1235 Acked Read 9 Done Sent Done false
1737 1236 Acked Read 9 Done Sent Done false
1738 1237 Acked Flush 0 Done Sent Done false
1739 1238 Acked Read 7 Done Sent Done false
1740 1239 Acked Read 9 Done Sent Done false
1741 1240 Acked Write 1 Done Sent Done false
1742 1241 Acked Read 6 Done Sent Done false
1743 1242 Acked Read 3 Done Sent Done false
1744 1243 Acked Read 7 Done Sent Done false
1745 1244 Acked Write 1 Done Sent Done false
1746 1245 Acked Flush 0 Done Sent Done false
1747 1246 Acked Write 10 Done Sent Done false
1748 1247 Acked Read 10 Done Sent Done false
1749 1248 Acked Flush 0 Done Sent Done false
1750 1249 Acked Read 7 Done Sent Done false
1751 1250 Acked Flush 0 Done Sent Done false
1752 1251 Acked Write 6 Done Sent Done false
1753 1252 Acked Write 8 Done Sent Done false
1754 1253 Acked Flush 0 Done Sent Done false
1755 1254 Acked Read 5 Done Sent Done false
1756 1255 Acked Write 4 Done Sent Done false
1757 1256 Acked Write 8 Done Sent Done false
1758 1257 Acked Write 4 Done Sent Done false
1759 1258 Acked Read 1 Done Sent Done false
1760 1259 Acked Read 3 Done Sent Done false
1761 1260 Acked Read 4 Done Sent Done false
1762 1261 Acked Write 4 Done Sent Done false
1763 1262 Acked Read 1 Done Sent Done false
1764 1263 Acked Write 10 Done Sent Done false
1765 1264 Acked Read 7 Done Sent Done false
1766 1265 Acked Read 3 Done Sent Done false
1767 1266 Acked Read 8 Done Sent Done false
1768 1267 Acked Read 9 Done Sent Done false
1769 1268 Acked Write 7 Done Sent Done false
1770 1269 Acked Read 6 Done Sent Done false
1771 1270 Acked Write 6 Done Sent Done false
1772 1271 Acked Flush 0 Done Sent Done false
1773 1272 Acked Write 3 Done Sent Done false
1774 1273 Acked Read 1 Done Sent Done false
1775 1274 Acked Read 10 Done Sent Done false
1776 1275 Acked Write 3 Done Sent Done false
1777 1276 Acked Read 9 Done Sent Done false
1778 1277 Acked Read 9 Done Sent Done false
1779 1278 Acked Read 4 Done Sent Done false
1780 1279 Acked Read 1 Done Sent Done false
1781 1280 Acked Read 2 Done Sent Done false
1782 1281 Acked Read 10 Done Sent Done false
1783 1282 Acked Write 1 Done Sent Done false
1784 1283 Acked Read 7 Done Sent Done false
1785 1284 Acked Read 3 Done Sent Done false
1786 1285 Acked Flush 0 Done Sent Done false
1787 1286 Acked Flush 0 Done Sent Done false
1788 1287 Acked Read 4 Done Sent Done false
1789 1288 Acked Read 6 Done Sent Done false
1790 1289 Acked Read 1 Done Sent Done false
1791 1290 Acked Read 2 Done Sent Done false
1792 1291 Acked Read 10 Done Sent Done false
1793 1292 Acked Read 5 Done Sent Done false
1794 1293 Acked Read 8 Done Sent Done false
1795 1294 Acked Read 5 Done Sent Done false
1796 1295 Acked Read 9 Done Sent Done false
1797 1296 Acked Read 9 Done Sent Done false
1798 1297 Acked Read 1 Done Sent Done false
1799 1298 Acked Read 6 Done Sent Done false
1800 1299 Acked Read 2 Done Sent Done false
1801 1300 Acked Flush 0 Done Sent Done false
1802 STATES DS:0 DS:1 DS:2 TOTAL
1803 Sent 0 301 0 301
1804 Done 301 0 301 602
1805 Skipped 0 0 0 0
1806 Error 0 0 0 0
1807 Last Flush: Some(JobId(1300)) None Some(JobId(1300))
1808 Downstairs last five completed:
1809 Upstairs last five completed: 1300 1299 1298 1297 1296
1810 CLIENT: Up:0 ds:301 act:2
18112025-02-28T06:01:20.630ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18122025-02-28T06:01:21.369ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18132025-02-28T06:01:22.372ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18142025-02-28T06:01:23.374ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18152025-02-28T06:01:23.879ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18162025-02-28T06:01:23.880ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56891) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18172025-02-28T06:01:23.881ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18182025-02-28T06:01:23.881ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18192025-02-28T06:01:23.881ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18202025-02-28T06:01:23.881ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18212025-02-28T06:01:23.881ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18222025-02-28T06:01:23.881ZINFOcrucible: [1] client re-new 301 jobs since flush None = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18232025-02-28T06:01:23.882ZINFOcrucible: [1] Replayed 301 jobs since flush: None = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1824 ----------------------------------------------------------------
1825 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1826 Downstairs last five completed: 1300 1299 1298 1297 1296
1827 Upstairs last five completed: 1300 1299 1298 1297 1296
1828 CLIENT: Up:0 ds:0 act:3
1829 Replay: All jobs finished
1830 Main task has work to do, go find it
1831 got dsc Stop(1)
1832 stop 1
1833 Feb 28 06:01:24.131 INFO request completed, latency_us: 341, response_code: 204, uri: /stop/cid/1, method: POST, req_id: aeca7b05-817c-4d05-a394-dc815cffbc58, [8820] Got stop action so:false kr:false
1834 remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
18352025-02-28T06:01:24.132ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18362025-02-28T06:01:24.132ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18372025-02-28T06:01:24.132ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
1838 Feb 28 06:01:24.132 INFO request completed, latency_us: 148, response_code: 200, uri: /state/cid/1, method:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:24.13250304Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
1839 GET, req_id: 04e5f861-8a7e-458d-bacc-a34495637078, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
1840 [8820] Exited with: ExitStatus(unix_wait_status(9))
1841 [8820][1] reports Exit
1842 Feb 28 06:01:28.135 INFO request completed, latency_us: 253, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1fc4a83e-2897-4f27-94a6-bf0cce55d4f4, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
1843 001/300 Read block 1436 len 8192
1844 002/300 Read block 143 len 24576
1845 003/300 Write block 1280 len 20480 data: 2 2 2 2 2
1846 004/300 Read block 1579 len 24576
1847 005/300 Read block 2440 len 16384
1848 006/300 Read block 4182 len 16384
1849 007/300 Flush
1850 008/300 Write block 3297 len 12288 data: 2 2 2
1851 009/300 Write block 1546 len 24576 data: 2 2 2 2 2 2
1852 010/300 Flush
1853 011/300 Read block 204 len 16384
1854 012/300 Read block 331 len 32768
1855 013/300 Read block 1734 len 12288
1856 014/300 Read block 1638 len 24576
1857 015/300 Write block 3071 len 16384 data: 2 2 2 2
1858 016/300 Write block 4528 len 16384 data: 2 2 2 2
1859 017/300 Read block 3602 len 32768
1860 018/300 Flush
1861 019/300 Write block 4678 len 16384 data: 2 2 2 2
1862 020/300 Write block 1581 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1863 021/300 Read block 145 len 4096
1864 022/300 Read block 1661 len 20480
1865 023/300 Write block 580 len 28672 data: 2 2 2 2 2 2 2
1866 024/300 Write block 2764 len 4096 data: 2
1867 025/300 Read block 4476 len 36864
1868 026/300 Read block 227 len 40960
1869 027/300 Write block 1465 len 20480 data: 2 2 2 2 2
1870 028/300 Write block 3244 len 28672 data: 2 2 2 2 2 2 2
1871 029/300 Read block 4324 len 8192
1872 030/300 Flush
1873 031/300 Read block 422 len 8192
1874 032/300 Read block 180 len 40960
1875 033/300 Read block 271 len 40960
1876 034/300 Write block 438 len 12288 data: 2 2 2
1877 035/300 Flush
1878 036/300 Read block 2334 len 12288
1879 037/300 Read block 3872 len 16384
1880 038/300 Flush
1881 039/300 Write block 4887 len 20480 data: 2 4 4 4 4
1882 040/300 Write block 1932 len 24576 data: 2 2 2 2 2 2
1883 041/300 Flush
1884 042/300 Read block 1422 len 20480
1885 043/300 Write block 1537 len 8192 data: 2 2
1886 044/300 Read block 537 len 28672
1887 045/300 Read block 3927 len 16384
1888 046/300 Read block 102 len 28672
1889 047/300 Write block 3158 len 12288 data: 2 2 2
1890 048/300 Flush
1891 049/300 Read block 3050 len 36864
1892 050/300 Write block 905 len 16384 data: 2 2 2 2
1893 051/300 Read block 4672 len 8192
1894 052/300 Read block 2450 len 16384
1895 053/300 Write block 1176 len 16384 data: 2 2 2 3
1896 054/300 Write block 2583 len 36864 data: 2 2 2 2 2 2 2 2 2
1897 055/300 Write block 500 len 32768 data: 2 2 2 2 2 2 2 2
1898 056/300 Flush
1899 057/300 Read block 1495 len 20480
1900 058/300 Read block 3094 len 28672
1901 059/300 Write block 3780 len 16384 data: 2 3 3 3
1902 060/300 Write block 2190 len 36864 data: 2 2 2 2 2 3 3 3 2
1903 061/300 Read block 4066 len 28672
1904 062/300 Flush
1905 063/300 Read block 123 len 36864
1906 064/300 Flush
1907 065/300 Write block 3996 len 28672 data: 2 2 2 2 2 2 2
1908 066/300 Read block 742 len 12288
1909 067/300 Read block 4198 len 4096
1910 068/300 Write block 4280 len 32768 data: 2 2 2 2 2 2 2 2
1911 069/300 Read block 4207 len 40960
1912 070/300 Write block 2937 len 24576 data: 2 3 3 3 3 3
1913 071/300 Read block 2377 len 12288
1914 072/300 Read block 776 len 24576
1915 073/300 Flush
1916 074/300 Read block 3435 len 12288
1917 075/300 Read block 4274 len 16384
1918 076/300 Read block 3268 len 32768
1919 077/300 Read block 2720 len 16384
1920 078/300 Read block 848 len 28672
1921 079/300 Read block 2409 len 24576
1922 080/300 Write block 1402 len 40960 data: 2 2 2 2 2 2 2 3 3 3
1923 081/300 Write block 2599 len 32768 data: 3 3 3 2 2 2 2 2
1924 082/300 Read block 313 len 8192
1925 083/300 Read block 3820 len 40960
1926 084/300 Read block 1899 len 12288
1927 085/300 Read block 2199 len 16384
1928 086/300 Read block 3386 len 16384
1929 087/300 Read block 4553 len 16384
1930 088/300 Write block 3794 len 4096 data: 2
1931 089/300 Read block 78 len 32768
1932 090/300 Read block 4397 len 20480
1933 091/300 Flush
1934 092/300 Read block 4927 len 8192
1935 093/300 Read block 2374 len 24576
1936 094/300 Read block 3477 len 36864
1937 095/300 Write block 237 len 8192 data: 3 3
1938 096/300 Read block 286 len 12288
1939 097/300 Write block 3306 len 16384 data: 2 2 2 2
1940 098/300 Read block 2634 len 40960
1941 099/300 Read block 2545 len 36864
1942 100/300 Write block 2521 len 36864 data: 2 2 2 2 2 2 2 2 2
1943 101/300 Read block 3325 len 8192
1944 102/300 Write block 3925 len 8192 data: 2 2
1945 103/300 Read block 585 len 20480
1946 104/300 Write block 2199 len 24576 data: 2 2 2 2 2 2
1947 105/300 Read block 1257 len 16384
1948 106/300 Read block 3974 len 12288
1949 107/300 Flush
1950 108/300 Read block 573 len 40960
1951 109/300 Read block 1897 len 32768
1952 110/300 Write block 3628 len 8192 data: 2 2
1953 111/300 Write block 1761 len 24576 data: 2 2 2 2 2 2
1954 112/300 Write block 2713 len 24576 data: 2 2 2 2 2 2
1955 113/300 Read block 612 len 28672
1956 114/300 Read block 3073 len 4096
1957 115/300 Read block 3358 len 20480
1958 116/300 Write block 2670 len 28672 data: 2 2 2 2 2 2 2
1959 117/300 Write block 1334 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1960 118/300 Read block 3137 len 40960
1961 119/300 Write block 4897 len 16384 data: 2 2 2 2
1962 120/300 Read block 4241 len 36864
1963 121/300 Read block 249 len 20480
1964 122/300 Write block 3336 len 20480 data: 2 2 2 2 2
1965 123/300 Read block 3823 len 16384
1966 124/300 Write block 2227 len 32768 data: 2 2 2 2 2 2 2 3
1967 125/300 Read block 1498 len 24576
1968 126/300 Write block 2511 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1969 127/300 Read block 2187 len 28672
1970 128/300 Write block 3282 len 4096 data: 2
1971 129/300 Read block 3868 len 16384
1972 130/300 Flush
1973 131/300 Read block 1935 len 4096
1974 132/300 Flush
1975 133/300 Read block 4970 len 36864
1976 134/300 Read block 4406 len 24576
1977 135/300 Write block 4776 len 24576 data: 2 2 2 2 2 2
1978 136/300 Write block 4178 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1979 137/300 Write block 32 len 24576 data: 2 2 2 2 2 2
1980 138/300 Read block 3874 len 40960
1981 139/300 Read block 1272 len 16384
1982 140/300 Read block 3031 len 28672
1983 141/300 Write block 2624 len 4096 data: 2
1984 142/300 Flush
1985 143/300 Read block 4482 len 4096
1986 144/300 Read block 763 len 20480
1987 145/300 Write block 4729 len 32768 data: 2 2 2 2 2 2 2 2
1988 146/300 Read block 1530 len 28672
1989 147/300 Read block 3034 len 32768
1990 148/300 Write block 1891 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1991 149/300 Flush
1992 150/300 Read block 774 len 8192
1993 151/300 Read block 2727 len 8192
1994 152/300 Write block 75 len 20480 data: 3 3 3 3 3
1995 153/300 Read block 2777 len 8192
1996 154/300 Read block 270 len 4096
1997 155/300 Read block 2140 len 32768
1998 156/300 Write block 2117 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1999 157/300 Read block 1344 len 28672
2000 158/300 Flush
2001 159/300 Read block 1828 len 36864
2002 160/300 Read block 1224 len 28672
2003 161/300 Flush
2004 162/300 Read block 2577 len 28672
2005 163/300 Read block 644 len 4096
2006 164/300 Write block 1578 len 40960 data: 2 2 2 3 3 3 3 3 3 3
2007 165/300 Read block 4744 len 16384
2008 166/300 Write block 4518 len 16384 data: 2 2 2 2
2009 167/300 Read block 791 len 8192
2010 168/300 Write block 961 len 4096 data: 2
2011 169/300 Read block 197 len 36864
2012 170/300 Read block 3877 len 36864
2013 171/300 Write block 4780 len 40960 data: 3 3 2 2 2 2 2 2 2 2
2014 172/300 Write block 4004 len 20480 data: 2 3 3 3 3
2015 173/300 Read block 818 len 12288
2016 174/300 Read block 3643 len 8192
2017 175/300 Read block 555 len 8192
2018 176/300 Write block 1758 len 32768 data: 2 2 2 3 3 3 3 3
2019 177/300 Write block 1682 len 24576 data: 2 2 2 2 2 2
2020 178/300 Write block 2253 len 12288 data: 2 2 2
2021 179/300 Read block 4050 len 20480
2022 180/300 Read block 3693 len 12288
2023 181/300 Write block 3100 len 24576 data: 2 2 2 2 2 2
2024 182/300 Read block 431 len 32768
2025 183/300 Write block 281 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2026 184/300 Read block 2466 len 32768
2027 185/300 Write block 3335 len 28672 data: 2 3 3 3 3 3 2
2028 186/300 Write block 271 len 32768 data: 2 2 2 2 2 2 2 2
2029 187/300 Read block 2616 len 8192
2030 188/300 Read block 2690 len 28672
2031 189/300 Write block 2231 len 16384 data: 3 3 3 4
2032 190/300 Write block 3939 len 32768 data: 2 2 2 2 2 2 2 2
2033 191/300 Read block 4443 len 16384
2034 192/300 Read block 3643 len 4096
2035 193/300 Read block 242 len 40960
2036 194/300 Flush
2037 195/300 Read block 3074 len 12288
2038 196/300 Write block 551 len 12288 data: 2 2 2
2039 197/300 Read block 872 len 8192
2040 198/300 Write block 2439 len 8192 data: 2 2
2041 199/300 Write block 1271 len 16384 data: 2 2 2 2
2042 200/300 Flush
2043 201/300 Read block 112 len 24576
2044 202/300 Write block 552 len 8192 data: 3 3
2045 203/300 Write block 2799 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2046 204/300 Read block 72 len 40960
2047 205/300 Write block 3156 len 28672 data: 2 2 3 3 3 2 2
2048 206/300 Write block 2750 len 8192 data: 2 2
2049 207/300 Write block 1000 len 36864 data: 2 2 2 2 2 2 2 2 2
2050 208/300 Read block 224 len 20480
2051 209/300 Flush
2052 210/300 Read block 1132 len 4096
2053 211/300 Write block 4535 len 24576 data: 2 2 2 2 2 2
2054 212/300 Write block 1049 len 24576 data: 2 2 2 2 2 2
2055 213/300 Read block 4445 len 28672
2056 214/300 Read block 4535 len 12288
2057 215/300 Read block 4962 len 40960
2058 216/300 Read block 4129 len 12288
2059 217/300 Write block 2317 len 12288 data: 2 2 2
2060 218/300 Write block 3302 len 24576 data: 2 2 2 2 3 3
2061 219/300 Read block 1036 len 16384
2062 220/300 Read block 2972 len 32768
2063 221/300 Write block 2990 len 12288 data: 2 2 2
2064 222/300 Write block 3024 len 4096 data: 2
2065 223/300 Read block 797 len 20480
2066 224/300 Read block 1677 len 12288
2067 225/300 Write block 2336 len 36864 data: 2 2 2 2 2 2 2 2 2
2068 226/300 Write block 4840 len 24576 data: 2 2 3 2 2 2
2069 227/300 Read block 2009 len 24576
2070 228/300 Read block 2551 len 20480
2071 229/300 Write block 779 len 28672 data: 2 2 2 2 2 2 2
2072 230/300 Read block 1561 len 40960
2073 231/300 Read block 3825 len 8192
2074 232/300 Read block 2984 len 4096
2075 233/300 Write block 910 len 24576 data: 2 2 2 2 2 2
2076 234/300 Read block 3459 len 28672
2077 235/300 Read block 1409 len 12288
2078 236/300 Read block 3321 len 36864
2079 237/300 Read block 4946 len 32768
2080 238/300 Read block 470 len 36864
2081 239/300 Read block 1357 len 24576
2082 240/300 Write block 2603 len 8192 data: 3 3
2083 241/300 Read block 3473 len 8192
2084 242/300 Flush
2085 243/300 Read block 1658 len 32768
2086 244/300 Read block 1641 len 28672
2087 245/300 Read block 2690 len 24576
2088 246/300 Flush
2089 247/300 Write block 2805 len 12288 data: 3 3 3
2090 248/300 Read block 3509 len 24576
2091 249/300 Read block 1548 len 4096
2092 250/300 Read block 4038 len 16384
2093 251/300 Write block 3873 len 16384 data: 2 2 2 2
2094 252/300 Write block 948 len 16384 data: 2 2 2 2
2095 253/300 Flush
2096 254/300 Read block 367 len 12288
2097 255/300 Write block 929 len 16384 data: 2 2 2 2
2098 256/300 Read block 1331 len 40960
2099 257/300 Read block 1689 len 8192
2100 258/300 Read block 4174 len 4096
2101 259/300 Read block 4155 len 20480
2102 260/300 Read block 1276 len 32768
2103 261/300 Write block 1604 len 12288 data: 2 2 2
2104 262/300 Write block 3341 len 28672 data: 3 2 2 2 2 2 2
2105 263/300 Read block 1929 len 8192
2106 264/300 Read block 3950 len 16384
2107 265/300 Read block 4891 len 12288
2108 266/300 Read block 25 len 4096
2109 267/300 Write block 1350 len 16384 data: 3 3 3 3
2110 268/300 Read block 2230 len 32768
2111 269/300 Write block 62 len 12288 data: 2 2 2
2112 270/300 Read block 2953 len 36864
2113 271/300 Read block 4344 len 32768
2114 272/300 Read block 4121 len 20480
2115 273/300 Write block 122 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2116 274/300 Read block 1463 len 4096
2117 275/300 Read block 1361 len 8192
2118 276/300 Flush
2119 277/300 Read block 2745 len 4096
2120 278/300 Write block 2124 len 16384 data: 3 3 3 2
2121 279/300 Write block 3883 len 16384 data: 2 2 2 2
2122 280/300 Read block 3494 len 8192
2123 281/300 Read block 440 len 28672
2124 282/300 Read block 2047 len 12288
2125 283/300 Write block 4408 len 20480 data: 4 4 4 4 4
2126 284/300 Read block 4108 len 20480
2127 285/300 Read block 3188 len 4096
2128 286/300 Read block 2558 len 12288
2129 287/300 Write block 569 len 20480 data: 2 2 2 2 2
2130 288/300 Write block 1448 len 28672 data: 2 2 2 2 2 2 2
2131 289/300 Read block 1535 len 4096
2132 290/300 Read block 4468 len 40960
2133 291/300 Read block 3677 len 12288
2134 292/300 Read block 2618 len 4096
2135 293/300 Read block 4226 len 40960
2136 294/300 Read block 4266 len 36864
2137 295/300 Read block 3018 len 28672
2138 296/300 Flush
2139 297/300 Read block 4630 len 4096
2140 298/300 Write block 373 len 32768 data: 2 2 2 2 2 3 3 3
2141 299/300 Write block 1691 len 8192 data: 3 3
2142 300/300 Read block 3953 len 24576
2143 301/300 Write block 3681 len 16384 data: 2 2 2 2
2144 Main task has work to do, go find it
2145 got dsc Start(1)
2146 start 1
2147 Feb 28 06:01:28.383 INFO request completed, latency_us: 185, response_code: 204, uri: /start/cid/1, method: POST, req_id: 91251bca-0438-4afd-9141-a832189e6d2f, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
2148 Replay: started 1, returned:Ok(())
2149 ----------------------------------------------------------------
2150 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2151 States: Active Offline Active
2152 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2153 1301 Acked Read 2 Done Sent Done false
2154 1302 Acked Read 6 Done Sent Done false
2155 1303 Acked Write 5 Done Sent Done false
2156 1304 Acked Read 6 Done Sent Done false
2157 1305 Acked Read 4 Done Sent Done false
2158 1306 Acked Read 4 Done Sent Done false
2159 1307 Acked Flush 0 Done Sent Done false
2160 1308 Acked Write 3 Done Sent Done false
2161 1309 Acked Write 6 Done Sent Done false
2162 1310 Acked Flush 0 Done Sent Done false
2163 1311 Acked Read 4 Done Sent Done false
2164 1312 Acked Read 8 Done Sent Done false
2165 1313 Acked Read 3 Done Sent Done false
2166 1314 Acked Read 6 Done Sent Done false
2167 1315 Acked Write 4 Done Sent Done false
2168 1316 Acked Write 4 Done Sent Done false
2169 1317 Acked Read 8 Done Sent Done false
2170 1318 Acked Flush 0 Done Sent Done false
2171 1319 Acked Write 4 Done Sent Done false
2172 1320 Acked Write 10 Done Sent Done false
2173 1321 Acked Read 1 Done Sent Done false
2174 1322 Acked Read 5 Done Sent Done false
2175 1323 Acked Write 7 Done Sent Done false
2176 1324 Acked Write 1 Done Sent Done false
2177 1325 Acked Read 9 Done Sent Done false
2178 1326 Acked Read 10 Done Sent Done false
2179 1327 Acked Write 5 Done Sent Done false
2180 1328 Acked Write 7 Done Sent Done false
2181 1329 Acked Read 2 Done Sent Done false
2182 1330 Acked Flush 0 Done Sent Done false
2183 1331 Acked Read 2 Done Sent Done false
2184 1332 Acked Read 10 Done Sent Done false
2185 1333 Acked Read 10 Done Sent Done false
2186 1334 Acked Write 3 Done Sent Done false
2187 1335 Acked Flush 0 Done Sent Done false
2188 1336 Acked Read 3 Done Sent Done false
2189 1337 Acked Read 4 Done Sent Done false
2190 1338 Acked Flush 0 Done Sent Done false
2191 1339 Acked Write 5 Done Sent Done false
2192 1340 Acked Write 6 Done Sent Done false
2193 1341 Acked Flush 0 Done Sent Done false
2194 1342 Acked Read 5 Done Sent Done false
2195 1343 Acked Write 2 Done Sent Done false
2196 1344 Acked Read 7 Done Sent Done false
2197 1345 Acked Read 4 Done Sent Done false
2198 1346 Acked Read 7 Done Sent Done false
2199 1347 Acked Write 3 Done Sent Done false
2200 1348 Acked Flush 0 Done Sent Done false
2201 1349 Acked Read 9 Done Sent Done false
2202 1350 Acked Write 4 Done Sent Done false
2203 1351 Acked Read 2 Done Sent Done false
2204 1352 Acked Read 4 Done Sent Done false
2205 1353 Acked Write 4 Done Sent Done false
2206 1354 Acked Write 9 Done Sent Done false
2207 1355 Acked Write 8 Done Sent Done false
2208 1356 Acked Flush 0 Done Sent Done false
2209 1357 Acked Read 5 Done Sent Done false
2210 1358 Acked Read 7 Done Sent Done false
2211 1359 Acked Write 4 Done Sent Done false
2212 1360 Acked Write 9 Done Sent Done false
2213 1361 Acked Read 7 Done Sent Done false
2214 1362 Acked Flush 0 Done Sent Done false
2215 1363 Acked Read 9 Done Sent Done false
2216 1364 Acked Flush 0 Done Sent Done false
2217 1365 Acked Write 7 Done Sent Done false
2218 1366 Acked Read 3 Done Sent Done false
2219 1367 Acked Read 1 Done Sent Done false
2220 1368 Acked Write 8 Done Sent Done false
2221 1369 Acked Read 10 Done Sent Done false
2222 1370 Acked Write 6 Done Sent Done false
2223 1371 Acked Read 3 Done Sent Done false
2224 1372 Acked Read 6 Done Sent Done false
2225 1373 Acked Flush 0 Done Sent Done false
2226 1374 Acked Read 3 Done Sent Done false
2227 1375 Acked Read 4 Done Sent Done false
2228 1376 Acked Read 8 Done Sent Done false
2229 1377 Acked Read 4 Done Sent Done false
2230 1378 Acked Read 7 Done Sent Done false
2231 1379 Acked Read 6 Done Sent Done false
2232 1380 Acked Write 10 Done Sent Done false
2233 1381 Acked Write 8 Done Sent Done false
2234 1382 Acked Read 2 Done Sent Done false
2235 1383 Acked Read 10 Done Sent Done false
2236 1384 Acked Read 3 Done Sent Done false
2237 1385 Acked Read 4 Done Sent Done false
2238 1386 Acked Read 4 Done Sent Done false
2239 1387 Acked Read 4 Done Sent Done false
2240 1388 Acked Write 1 Done Sent Done false
2241 1389 Acked Read 8 Done Sent Done false
2242 1390 Acked Read 5 Done Sent Done false
2243 1391 Acked Flush 0 Done Sent Done false
2244 1392 Acked Read 2 Done Sent Done false
2245 1393 Acked Read 6 Done Sent Done false
2246 1394 Acked Read 9 Done Sent Done false
2247 1395 Acked Write 2 Done Sent Done false
2248 1396 Acked Read 3 Done Sent Done false
2249 1397 Acked Write 4 Done Sent Done false
2250 1398 Acked Read 10 Done Sent Done false
2251 1399 Acked Read 9 Done Sent Done false
2252 1400 Acked Write 9 Done Sent Done false
2253 1401 Acked Read 2 Done Sent Done false
2254 1402 Acked Write 2 Done Sent Done false
2255 1403 Acked Read 5 Done Sent Done false
2256 1404 Acked Write 6 Done Sent Done false
2257 1405 Acked Read 4 Done Sent Done false
2258 1406 Acked Read 3 Done Sent Done false
2259 1407 Acked Flush 0 Done Sent Done false
2260 1408 Acked Read 10 Done Sent Done false
2261 1409 Acked Read 8 Done Sent Done false
2262 1410 Acked Write 2 Done Sent Done false
2263 1411 Acked Write 6 Done Sent Done false
2264 1412 Acked Write 6 Done Sent Done false
2265 1413 Acked Read 7 Done Sent Done false
2266 1414 Acked Read 1 Done Sent Done false
2267 1415 Acked Read 5 Done Sent Done false
2268 1416 Acked Write 7 Done Sent Done false
2269 1417 Acked Write 10 Done Sent Done false
2270 1418 Acked Read 10 Done Sent Done false
2271 1419 Acked Write 4 Done Sent Done false
2272 1420 Acked Read 9 Done Sent Done false
2273 1421 Acked Read 5 Done Sent Done false
2274 1422 Acked Write 5 Done Sent Done false
2275 1423 Acked Read 4 Done Sent Done false
2276 1424 Acked Write 8 Done Sent Done false
2277 1425 Acked Read 6 Done Sent Done false
2278 1426 Acked Write 10 Done Sent Done false
2279 1427 Acked Read 7 Done Sent Done false
2280 1428 Acked Write 1 Done Sent Done false
2281 1429 Acked Read 4 Done Sent Done false
2282 1430 Acked Flush 0 Done Sent Done false
2283 1431 Acked Read 1 Done Sent Done false
2284 1432 Acked Flush 0 Done Sent Done false
2285 1433 Acked Read 9 Done Sent Done false
2286 1434 Acked Read 6 Done Sent Done false
2287 1435 Acked Write 6 Done Sent Done false
2288 1436 Acked Write 10 Done Sent Done false
2289 1437 Acked Write 6 Done Sent Done false
2290 1438 Acked Read 10 Done Sent Done false
2291 1439 Acked Read 4 Done Sent Done false
2292 1440 Acked Read 7 Done Sent Done false
2293 1441 Acked Write 1 Done Sent Done false
2294 1442 Acked Flush 0 Done Sent Done false
2295 1443 Acked Read 1 Done Sent Done false
2296 1444 Acked Read 5 Done Sent Done false
2297 1445 Acked Write 8 Done Sent Done false
2298 1446 Acked Read 7 Done Sent Done false
2299 1447 Acked Read 8 Done Sent Done false
2300 1448 Acked Write 10 Done Sent Done false
2301 1449 Acked Flush 0 Done Sent Done false
2302 1450 Acked Read 2 Done Sent Done false
2303 1451 Acked Read 2 Done Sent Done false
2304 1452 Acked Write 5 Done Sent Done false
2305 1453 Acked Read 2 Done Sent Done false
2306 1454 Acked Read 1 Done Sent Done false
2307 1455 Acked Read 8 Done Sent Done false
2308 1456 Acked Write 10 Done Sent Done false
2309 1457 Acked Read 7 Done Sent Done false
2310 1458 Acked Flush 0 Done Sent Done false
2311 1459 Acked Read 9 Done Sent Done false
2312 1460 Acked Read 7 Done Sent Done false
2313 1461 Acked Flush 0 Done Sent Done false
2314 1462 Acked Read 7 Done Sent Done false
2315 1463 Acked Read 1 Done Sent Done false
2316 1464 Acked Write 10 Done Sent Done false
2317 1465 Acked Read 4 Done Sent Done false
2318 1466 Acked Write 4 Done Sent Done false
2319 1467 Acked Read 2 Done Sent Done false
2320 1468 Acked Write 1 Done Sent Done false
2321 1469 Acked Read 9 Done Sent Done false
2322 1470 Acked Read 9 Done Sent Done false
2323 1471 Acked Write 10 Done Sent Done false
2324 1472 Acked Write 5 Done Sent Done false
2325 1473 Acked Read 3 Done Sent Done false
2326 1474 Acked Read 2 Done Sent Done false
2327 1475 Acked Read 2 Done Sent Done false
2328 1476 Acked Write 8 Done Sent Done false
2329 1477 Acked Write 6 Done Sent Done false
2330 1478 Acked Write 3 Done Sent Done false
2331 1479 Acked Read 5 Done Sent Done false
2332 1480 Acked Read 3 Done Sent Done false
2333 1481 Acked Write 6 Done Sent Done false
2334 1482 Acked Read 8 Done Sent Done false
2335 1483 Acked Write 10 Done Sent Done false
2336 1484 Acked Read 8 Done Sent Done false
2337 1485 Acked Write 7 Done Sent Done false
2338 1486 Acked Write 8 Done Sent Done false
2339 1487 Acked Read 2 Done Sent Done false
2340 1488 Acked Read 7 Done Sent Done false
2341 1489 Acked Write 4 Done Sent Done false
2342 1490 Acked Write 8 Done Sent Done false
2343 1491 Acked Read 4 Done Sent Done false
2344 1492 Acked Read 1 Done Sent Done false
2345 1493 Acked Read 10 Done Sent Done false
2346 1494 Acked Flush 0 Done Sent Done false
2347 1495 Acked Read 3 Done Sent Done false
2348 1496 Acked Write 3 Done Sent Done false
2349 1497 Acked Read 2 Done Sent Done false
2350 1498 Acked Write 2 Done Sent Done false
2351 1499 Acked Write 4 Done Sent Done false
2352 1500 Acked Flush 0 Done Sent Done false
2353 1501 Acked Read 6 Done Sent Done false
2354 1502 Acked Write 2 Done Sent Done false
2355 1503 Acked Write 10 Done Sent Done false
2356 1504 Acked Read 10 Done Sent Done false
2357 1505 Acked Write 7 Done Sent Done false
2358 1506 Acked Write 2 Done Sent Done false
2359 1507 Acked Write 9 Done Sent Done false
2360 1508 Acked Read 5 Done Sent Done false
2361 1509 Acked Flush 0 Done Sent Done false
2362 1510 Acked Read 1 Done Sent Done false
2363 1511 Acked Write 6 Done Sent Done false
2364 1512 Acked Write 6 Done Sent Done false
2365 1513 Acked Read 7 Done Sent Done false
2366 1514 Acked Read 3 Done Sent Done false
2367 1515 Acked Read 10 Done Sent Done false
2368 1516 Acked Read 3 Done Sent Done false
2369 1517 Acked Write 3 Done Sent Done false
2370 1518 Acked Write 6 Done Sent Done false
2371 1519 Acked Read 4 Done Sent Done false
2372 1520 Acked Read 8 Done Sent Done false
2373 1521 Acked Write 3 Done Sent Done false
2374 1522 Acked Write 1 Done Sent Done false
2375 1523 Acked Read 5 Done Sent Done false
2376 1524 Acked Read 3 Done Sent Done false
2377 1525 Acked Write 9 Done Sent Done false
2378 1526 Acked Write 6 Done Sent Done false
2379 1527 Acked Read 6 Done Sent Done false
2380 1528 Acked Read 5 Done Sent Done false
2381 1529 Acked Write 7 Done Sent Done false
2382 1530 Acked Read 10 Done Sent Done false
2383 1531 Acked Read 2 Done Sent Done false
2384 1532 Acked Read 1 Done Sent Done false
2385 1533 Acked Write 6 Done Sent Done false
2386 1534 Acked Read 7 Done Sent Done false
2387 1535 Acked Read 3 Done Sent Done false
2388 1536 Acked Read 9 Done Sent Done false
2389 1537 Acked Read 8 Done Sent Done false
2390 1538 Acked Read 9 Done Sent Done false
2391 1539 Acked Read 6 Done Sent Done false
2392 1540 Acked Write 2 Done Sent Done false
2393 1541 Acked Read 2 Done Sent Done false
2394 1542 Acked Flush 0 Done Sent Done false
2395 1543 Acked Read 8 Done Sent Done false
2396 1544 Acked Read 7 Done Sent Done false
2397 1545 Acked Read 6 Done Sent Done false
2398 1546 Acked Flush 0 Done Sent Done false
2399 1547 Acked Write 3 Done Sent Done false
2400 1548 Acked Read 6 Done Sent Done false
2401 1549 Acked Read 1 Done Sent Done false
2402 1550 Acked Read 4 Done Sent Done false
2403 1551 Acked Write 4 Done Sent Done false
2404 1552 Acked Write 4 Done Sent Done false
2405 1553 Acked Flush 0 Done Sent Done false
2406 1554 Acked Read 3 Done Sent Done false
2407 1555 Acked Write 4 Done Sent Done false
2408 1556 Acked Read 10 Done Sent Done false
2409 1557 Acked Read 2 Done Sent Done false
2410 1558 Acked Read 1 Done Sent Done false
2411 1559 Acked Read 5 Done Sent Done false
2412 1560 Acked Read 8 Done Sent Done false
2413 1561 Acked Write 3 Done Sent Done false
2414 1562 Acked Write 7 Done Sent Done false
2415 1563 Acked Read 2 Done Sent Done false
2416 1564 Acked Read 4 Done Sent Done false
2417 1565 Acked Read 3 Done Sent Done false
2418 1566 Acked Read 1 Done Sent Done false
2419 1567 Acked Write 4 Done Sent Done false
2420 1568 Acked Read 8 Done Sent Done false
2421 1569 Acked Write 3 Done Sent Done false
2422 1570 Acked Read 9 Done Sent Done false
2423 1571 Acked Read 8 Done Sent Done false
2424 1572 Acked Read 5 Done Sent Done false
2425 1573 Acked Write 10 Done Sent Done false
2426 1574 Acked Read 1 Done Sent Done false
2427 1575 Acked Read 2 Done Sent Done false
2428 1576 Acked Flush 0 Done Sent Done false
2429 1577 Acked Read 1 Done Sent Done false
2430 1578 Acked Write 4 Done Sent Done false
2431 1579 Acked Write 4 Done Sent Done false
2432 1580 Acked Read 2 Done Sent Done false
2433 1581 Acked Read 7 Done Sent Done false
2434 1582 Acked Read 3 Done Sent Done false
2435 1583 Acked Write 5 Done Sent Done false
2436 1584 Acked Read 5 Done Sent Done false
2437 1585 Acked Read 1 Done Sent Done false
2438 1586 Acked Read 3 Done Sent Done false
2439 1587 Acked Write 5 Done Sent Done false
2440 1588 Acked Write 7 Done Sent Done false
2441 1589 Acked Read 1 Done Sent Done false
2442 1590 Acked Read 10 Done Sent Done false
2443 1591 Acked Read 3 Done Sent Done false
2444 1592 Acked Read 1 Done Sent Done false
2445 1593 Acked Read 10 Done Sent Done false
2446 1594 Acked Read 9 Done Sent Done false
2447 1595 Acked Read 7 Done Sent Done false
2448 1596 Acked Flush 0 Done Sent Done false
2449 1597 Acked Read 1 Done Sent Done false
2450 1598 Acked Write 8 Done Sent Done false
2451 1599 Acked Write 2 Done Sent Done false
2452 1600 Acked Read 6 Done Sent Done false
2453 1601 Acked Write 4 Done Sent Done false
2454 STATES DS:0 DS:1 DS:2 TOTAL
2455 Sent 0 301 0 301
2456 Done 301 0 301 602
2457 Skipped 0 0 0 0
2458 Error 0 0 0 0
2459 Last Flush: Some(JobId(1596)) Some(JobId(1300)) Some(JobId(1596))
2460 Downstairs last five completed: 1300 1299 1298 1297 1296
2461 Upstairs last five completed: 1601 1600 1599 1598 1597
2462 CLIENT: Up:0 ds:301 act:2
24632025-02-28T06:01:29.382ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
24642025-02-28T06:01:30.383ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
24652025-02-28T06:01:31.385ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2466 ----------------------------------------------------------------
2467 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2468 States: Active Offline Active
2469 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2470 1301 Acked Read 2 Done Sent Done false
2471 1302 Acked Read 6 Done Sent Done false
2472 1303 Acked Write 5 Done Sent Done false
2473 1304 Acked Read 6 Done Sent Done false
2474 1305 Acked Read 4 Done Sent Done false
2475 1306 Acked Read 4 Done Sent Done false
2476 1307 Acked Flush 0 Done Sent Done false
2477 1308 Acked Write 3 Done Sent Done false
2478 1309 Acked Write 6 Done Sent Done false
2479 1310 Acked Flush 0 Done Sent Done false
2480 1311 Acked Read 4 Done Sent Done false
2481 1312 Acked Read 8 Done Sent Done false
2482 1313 Acked Read 3 Done Sent Done false
2483 1314 Acked Read 6 Done Sent Done false
2484 1315 Acked Write 4 Done Sent Done false
2485 1316 Acked Write 4 Done Sent Done false
2486 1317 Acked Read 8 Done Sent Done false
2487 1318 Acked Flush 0 Done Sent Done false
2488 1319 Acked Write 4 Done Sent Done false
2489 1320 Acked Write 10 Done Sent Done false
2490 1321 Acked Read 1 Done Sent Done false
2491 1322 Acked Read 5 Done Sent Done false
2492 1323 Acked Write 7 Done Sent Done false
2493 1324 Acked Write 1 Done Sent Done false
2494 1325 Acked Read 9 Done Sent Done false
2495 1326 Acked Read 10 Done Sent Done false
2496 1327 Acked Write 5 Done Sent Done false
2497 1328 Acked Write 7 Done Sent Done false
2498 1329 Acked Read 2 Done Sent Done false
2499 1330 Acked Flush 0 Done Sent Done false
2500 1331 Acked Read 2 Done Sent Done false
2501 1332 Acked Read 10 Done Sent Done false
2502 1333 Acked Read 10 Done Sent Done false
2503 1334 Acked Write 3 Done Sent Done false
2504 1335 Acked Flush 0 Done Sent Done false
2505 1336 Acked Read 3 Done Sent Done false
2506 1337 Acked Read 4 Done Sent Done false
2507 1338 Acked Flush 0 Done Sent Done false
2508 1339 Acked Write 5 Done Sent Done false
2509 1340 Acked Write 6 Done Sent Done false
2510 1341 Acked Flush 0 Done Sent Done false
2511 1342 Acked Read 5 Done Sent Done false
2512 1343 Acked Write 2 Done Sent Done false
2513 1344 Acked Read 7 Done Sent Done false
2514 1345 Acked Read 4 Done Sent Done false
2515 1346 Acked Read 7 Done Sent Done false
2516 1347 Acked Write 3 Done Sent Done false
2517 1348 Acked Flush 0 Done Sent Done false
2518 1349 Acked Read 9 Done Sent Done false
2519 1350 Acked Write 4 Done Sent Done false
2520 1351 Acked Read 2 Done Sent Done false
2521 1352 Acked Read 4 Done Sent Done false
2522 1353 Acked Write 4 Done Sent Done false
2523 1354 Acked Write 9 Done Sent Done false
2524 1355 Acked Write 8 Done Sent Done false
2525 1356 Acked Flush 0 Done Sent Done false
2526 1357 Acked Read 5 Done Sent Done false
2527 1358 Acked Read 7 Done Sent Done false
2528 1359 Acked Write 4 Done Sent Done false
2529 1360 Acked Write 9 Done Sent Done false
2530 1361 Acked Read 7 Done Sent Done false
2531 1362 Acked Flush 0 Done Sent Done false
2532 1363 Acked Read 9 Done Sent Done false
2533 1364 Acked Flush 0 Done Sent Done false
2534 1365 Acked Write 7 Done Sent Done false
2535 1366 Acked Read 3 Done Sent Done false
2536 1367 Acked Read 1 Done Sent Done false
2537 1368 Acked Write 8 Done Sent Done false
2538 1369 Acked Read 10 Done Sent Done false
2539 1370 Acked Write 6 Done Sent Done false
2540 1371 Acked Read 3 Done Sent Done false
2541 1372 Acked Read 6 Done Sent Done false
2542 1373 Acked Flush 0 Done Sent Done false
2543 1374 Acked Read 3 Done Sent Done false
2544 1375 Acked Read 4 Done Sent Done false
2545 1376 Acked Read 8 Done Sent Done false
2546 1377 Acked Read 4 Done Sent Done false
2547 1378 Acked Read 7 Done Sent Done false
2548 1379 Acked Read 6 Done Sent Done false
2549 1380 Acked Write 10 Done Sent Done false
2550 1381 Acked Write 8 Done Sent Done false
2551 1382 Acked Read 2 Done Sent Done false
2552 1383 Acked Read 10 Done Sent Done false
2553 1384 Acked Read 3 Done Sent Done false
2554 1385 Acked Read 4 Done Sent Done false
2555 1386 Acked Read 4 Done Sent Done false
2556 1387 Acked Read 4 Done Sent Done false
2557 1388 Acked Write 1 Done Sent Done false
2558 1389 Acked Read 8 Done Sent Done false
2559 1390 Acked Read 5 Done Sent Done false
2560 1391 Acked Flush 0 Done Sent Done false
2561 1392 Acked Read 2 Done Sent Done false
2562 1393 Acked Read 6 Done Sent Done false
2563 1394 Acked Read 9 Done Sent Done false
2564 1395 Acked Write 2 Done Sent Done false
2565 1396 Acked Read 3 Done Sent Done false
2566 1397 Acked Write 4 Done Sent Done false
2567 1398 Acked Read 10 Done Sent Done false
2568 1399 Acked Read 9 Done Sent Done false
2569 1400 Acked Write 9 Done Sent Done false
2570 1401 Acked Read 2 Done Sent Done false
2571 1402 Acked Write 2 Done Sent Done false
2572 1403 Acked Read 5 Done Sent Done false
2573 1404 Acked Write 6 Done Sent Done false
2574 1405 Acked Read 4 Done Sent Done false
2575 1406 Acked Read 3 Done Sent Done false
2576 1407 Acked Flush 0 Done Sent Done false
2577 1408 Acked Read 10 Done Sent Done false
2578 1409 Acked Read 8 Done Sent Done false
2579 1410 Acked Write 2 Done Sent Done false
2580 1411 Acked Write 6 Done Sent Done false
2581 1412 Acked Write 6 Done Sent Done false
2582 1413 Acked Read 7 Done Sent Done false
2583 1414 Acked Read 1 Done Sent Done false
2584 1415 Acked Read 5 Done Sent Done false
2585 1416 Acked Write 7 Done Sent Done false
2586 1417 Acked Write 10 Done Sent Done false
2587 1418 Acked Read 10 Done Sent Done false
2588 1419 Acked Write 4 Done Sent Done false
2589 1420 Acked Read 9 Done Sent Done false
2590 1421 Acked Read 5 Done Sent Done false
2591 1422 Acked Write 5 Done Sent Done false
2592 1423 Acked Read 4 Done Sent Done false
2593 1424 Acked Write 8 Done Sent Done false
2594 1425 Acked Read 6 Done Sent Done false
2595 1426 Acked Write 10 Done Sent Done false
2596 1427 Acked Read 7 Done Sent Done false
2597 1428 Acked Write 1 Done Sent Done false
2598 1429 Acked Read 4 Done Sent Done false
2599 1430 Acked Flush 0 Done Sent Done false
2600 1431 Acked Read 1 Done Sent Done false
2601 1432 Acked Flush 0 Done Sent Done false
2602 1433 Acked Read 9 Done Sent Done false
2603 1434 Acked Read 6 Done Sent Done false
2604 1435 Acked Write 6 Done Sent Done false
2605 1436 Acked Write 10 Done Sent Done false
2606 1437 Acked Write 6 Done Sent Done false
2607 1438 Acked Read 10 Done Sent Done false
2608 1439 Acked Read 4 Done Sent Done false
2609 1440 Acked Read 7 Done Sent Done false
2610 1441 Acked Write 1 Done Sent Done false
2611 1442 Acked Flush 0 Done Sent Done false
2612 1443 Acked Read 1 Done Sent Done false
2613 1444 Acked Read 5 Done Sent Done false
2614 1445 Acked Write 8 Done Sent Done false
2615 1446 Acked Read 7 Done Sent Done false
2616 1447 Acked Read 8 Done Sent Done false
2617 1448 Acked Write 10 Done Sent Done false
2618 1449 Acked Flush 0 Done Sent Done false
2619 1450 Acked Read 2 Done Sent Done false
2620 1451 Acked Read 2 Done Sent Done false
2621 1452 Acked Write 5 Done Sent Done false
2622 1453 Acked Read 2 Done Sent Done false
2623 1454 Acked Read 1 Done Sent Done false
2624 1455 Acked Read 8 Done Sent Done false
2625 1456 Acked Write 10 Done Sent Done false
2626 1457 Acked Read 7 Done Sent Done false
2627 1458 Acked Flush 0 Done Sent Done false
2628 1459 Acked Read 9 Done Sent Done false
2629 1460 Acked Read 7 Done Sent Done false
2630 1461 Acked Flush 0 Done Sent Done false
2631 1462 Acked Read 7 Done Sent Done false
2632 1463 Acked Read 1 Done Sent Done false
2633 1464 Acked Write 10 Done Sent Done false
2634 1465 Acked Read 4 Done Sent Done false
2635 1466 Acked Write 4 Done Sent Done false
2636 1467 Acked Read 2 Done Sent Done false
2637 1468 Acked Write 1 Done Sent Done false
2638 1469 Acked Read 9 Done Sent Done false
2639 1470 Acked Read 9 Done Sent Done false
2640 1471 Acked Write 10 Done Sent Done false
2641 1472 Acked Write 5 Done Sent Done false
2642 1473 Acked Read 3 Done Sent Done false
2643 1474 Acked Read 2 Done Sent Done false
2644 1475 Acked Read 2 Done Sent Done false
2645 1476 Acked Write 8 Done Sent Done false
2646 1477 Acked Write 6 Done Sent Done false
2647 1478 Acked Write 3 Done Sent Done false
2648 1479 Acked Read 5 Done Sent Done false
2649 1480 Acked Read 3 Done Sent Done false
2650 1481 Acked Write 6 Done Sent Done false
2651 1482 Acked Read 8 Done Sent Done false
2652 1483 Acked Write 10 Done Sent Done false
2653 1484 Acked Read 8 Done Sent Done false
2654 1485 Acked Write 7 Done Sent Done false
2655 1486 Acked Write 8 Done Sent Done false
2656 1487 Acked Read 2 Done Sent Done false
2657 1488 Acked Read 7 Done Sent Done false
2658 1489 Acked Write 4 Done Sent Done false
2659 1490 Acked Write 8 Done Sent Done false
2660 1491 Acked Read 4 Done Sent Done false
2661 1492 Acked Read 1 Done Sent Done false
2662 1493 Acked Read 10 Done Sent Done false
2663 1494 Acked Flush 0 Done Sent Done false
2664 1495 Acked Read 3 Done Sent Done false
2665 1496 Acked Write 3 Done Sent Done false
2666 1497 Acked Read 2 Done Sent Done false
2667 1498 Acked Write 2 Done Sent Done false
2668 1499 Acked Write 4 Done Sent Done false
2669 1500 Acked Flush 0 Done Sent Done false
2670 1501 Acked Read 6 Done Sent Done false
2671 1502 Acked Write 2 Done Sent Done false
2672 1503 Acked Write 10 Done Sent Done false
2673 1504 Acked Read 10 Done Sent Done false
2674 1505 Acked Write 7 Done Sent Done false
2675 1506 Acked Write 2 Done Sent Done false
2676 1507 Acked Write 9 Done Sent Done false
2677 1508 Acked Read 5 Done Sent Done false
2678 1509 Acked Flush 0 Done Sent Done false
2679 1510 Acked Read 1 Done Sent Done false
2680 1511 Acked Write 6 Done Sent Done false
2681 1512 Acked Write 6 Done Sent Done false
2682 1513 Acked Read 7 Done Sent Done false
2683 1514 Acked Read 3 Done Sent Done false
2684 1515 Acked Read 10 Done Sent Done false
2685 1516 Acked Read 3 Done Sent Done false
2686 1517 Acked Write 3 Done Sent Done false
2687 1518 Acked Write 6 Done Sent Done false
2688 1519 Acked Read 4 Done Sent Done false
2689 1520 Acked Read 8 Done Sent Done false
2690 1521 Acked Write 3 Done Sent Done false
2691 1522 Acked Write 1 Done Sent Done false
2692 1523 Acked Read 5 Done Sent Done false
2693 1524 Acked Read 3 Done Sent Done false
2694 1525 Acked Write 9 Done Sent Done false
2695 1526 Acked Write 6 Done Sent Done false
2696 1527 Acked Read 6 Done Sent Done false
2697 1528 Acked Read 5 Done Sent Done false
2698 1529 Acked Write 7 Done Sent Done false
2699 1530 Acked Read 10 Done Sent Done false
2700 1531 Acked Read 2 Done Sent Done false
2701 1532 Acked Read 1 Done Sent Done false
2702 1533 Acked Write 6 Done Sent Done false
2703 1534 Acked Read 7 Done Sent Done false
2704 1535 Acked Read 3 Done Sent Done false
2705 1536 Acked Read 9 Done Sent Done false
2706 1537 Acked Read 8 Done Sent Done false
2707 1538 Acked Read 9 Done Sent Done false
2708 1539 Acked Read 6 Done Sent Done false
2709 1540 Acked Write 2 Done Sent Done false
2710 1541 Acked Read 2 Done Sent Done false
2711 1542 Acked Flush 0 Done Sent Done false
2712 1543 Acked Read 8 Done Sent Done false
2713 1544 Acked Read 7 Done Sent Done false
2714 1545 Acked Read 6 Done Sent Done false
2715 1546 Acked Flush 0 Done Sent Done false
2716 1547 Acked Write 3 Done Sent Done false
2717 1548 Acked Read 6 Done Sent Done false
2718 1549 Acked Read 1 Done Sent Done false
2719 1550 Acked Read 4 Done Sent Done false
2720 1551 Acked Write 4 Done Sent Done false
2721 1552 Acked Write 4 Done Sent Done false
2722 1553 Acked Flush 0 Done Sent Done false
2723 1554 Acked Read 3 Done Sent Done false
2724 1555 Acked Write 4 Done Sent Done false
2725 1556 Acked Read 10 Done Sent Done false
2726 1557 Acked Read 2 Done Sent Done false
2727 1558 Acked Read 1 Done Sent Done false
2728 1559 Acked Read 5 Done Sent Done false
2729 1560 Acked Read 8 Done Sent Done false
2730 1561 Acked Write 3 Done Sent Done false
2731 1562 Acked Write 7 Done Sent Done false
2732 1563 Acked Read 2 Done Sent Done false
2733 1564 Acked Read 4 Done Sent Done false
2734 1565 Acked Read 3 Done Sent Done false
2735 1566 Acked Read 1 Done Sent Done false
2736 1567 Acked Write 4 Done Sent Done false
2737 1568 Acked Read 8 Done Sent Done false
2738 1569 Acked Write 3 Done Sent Done false
2739 1570 Acked Read 9 Done Sent Done false
2740 1571 Acked Read 8 Done Sent Done false
2741 1572 Acked Read 5 Done Sent Done false
2742 1573 Acked Write 10 Done Sent Done false
2743 1574 Acked Read 1 Done Sent Done false
2744 1575 Acked Read 2 Done Sent Done false
2745 1576 Acked Flush 0 Done Sent Done false
2746 1577 Acked Read 1 Done Sent Done false
2747 1578 Acked Write 4 Done Sent Done false
2748 1579 Acked Write 4 Done Sent Done false
2749 1580 Acked Read 2 Done Sent Done false
2750 1581 Acked Read 7 Done Sent Done false
2751 1582 Acked Read 3 Done Sent Done false
2752 1583 Acked Write 5 Done Sent Done false
2753 1584 Acked Read 5 Done Sent Done false
2754 1585 Acked Read 1 Done Sent Done false
2755 1586 Acked Read 3 Done Sent Done false
2756 1587 Acked Write 5 Done Sent Done false
2757 1588 Acked Write 7 Done Sent Done false
2758 1589 Acked Read 1 Done Sent Done false
2759 1590 Acked Read 10 Done Sent Done false
2760 1591 Acked Read 3 Done Sent Done false
2761 1592 Acked Read 1 Done Sent Done false
2762 1593 Acked Read 10 Done Sent Done false
2763 1594 Acked Read 9 Done Sent Done false
2764 1595 Acked Read 7 Done Sent Done false
2765 1596 Acked Flush 0 Done Sent Done false
2766 1597 Acked Read 1 Done Sent Done false
2767 1598 Acked Write 8 Done Sent Done false
2768 1599 Acked Write 2 Done Sent Done false
2769 1600 Acked Read 6 Done Sent Done false
2770 1601 Acked Write 4 Done Sent Done false
2771 STATES DS:0 DS:1 DS:2 TOTAL
2772 Sent 0 301 0 301
2773 Done 301 0 301 602
2774 Skipped 0 0 0 0
2775 Error 0 0 0 0
2776 Last Flush: Some(JobId(1596)) Some(JobId(1300)) Some(JobId(1596))
2777 Downstairs last five completed: 1300 1299 1298 1297 1296
2778 Upstairs last five completed: 1601 1600 1599 1598 1597
2779 CLIENT: Up:0 ds:301 act:2
27802025-02-28T06:01:32.887ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
27812025-02-28T06:01:33.389ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
27822025-02-28T06:01:34.133ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
27832025-02-28T06:01:34.133ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
27842025-02-28T06:01:34.133ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
27852025-02-28T06:01:34.133ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
27862025-02-28T06:01:34.133ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2787 [8820] got start action
2788 [8820] I am going to restart
2789 Starting downstairs at port 8820
2790 Make output file at "/tmp/dsc/downstairs-8820.txt"
2791 [8820][1] reports Starting
2792 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1213)
2793 [8820][1] reports Running
27942025-02-28T06:01:34.635ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
27952025-02-28T06:01:35.393ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
2796 ----------------------------------------------------------------
2797 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2798 States: Active Offline Active
2799 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2800 1301 Acked Read 2 Done Sent Done false
2801 1302 Acked Read 6 Done Sent Done false
2802 1303 Acked Write 5 Done Sent Done false
2803 1304 Acked Read 6 Done Sent Done false
2804 1305 Acked Read 4 Done Sent Done false
2805 1306 Acked Read 4 Done Sent Done false
2806 1307 Acked Flush 0 Done Sent Done false
2807 1308 Acked Write 3 Done Sent Done false
2808 1309 Acked Write 6 Done Sent Done false
2809 1310 Acked Flush 0 Done Sent Done false
2810 1311 Acked Read 4 Done Sent Done false
2811 1312 Acked Read 8 Done Sent Done false
2812 1313 Acked Read 3 Done Sent Done false
2813 1314 Acked Read 6 Done Sent Done false
2814 1315 Acked Write 4 Done Sent Done false
2815 1316 Acked Write 4 Done Sent Done false
2816 1317 Acked Read 8 Done Sent Done false
2817 1318 Acked Flush 0 Done Sent Done false
2818 1319 Acked Write 4 Done Sent Done false
2819 1320 Acked Write 10 Done Sent Done false
2820 1321 Acked Read 1 Done Sent Done false
2821 1322 Acked Read 5 Done Sent Done false
2822 1323 Acked Write 7 Done Sent Done false
2823 1324 Acked Write 1 Done Sent Done false
2824 1325 Acked Read 9 Done Sent Done false
2825 1326 Acked Read 10 Done Sent Done false
2826 1327 Acked Write 5 Done Sent Done false
2827 1328 Acked Write 7 Done Sent Done false
2828 1329 Acked Read 2 Done Sent Done false
2829 1330 Acked Flush 0 Done Sent Done false
2830 1331 Acked Read 2 Done Sent Done false
2831 1332 Acked Read 10 Done Sent Done false
2832 1333 Acked Read 10 Done Sent Done false
2833 1334 Acked Write 3 Done Sent Done false
2834 1335 Acked Flush 0 Done Sent Done false
2835 1336 Acked Read 3 Done Sent Done false
2836 1337 Acked Read 4 Done Sent Done false
2837 1338 Acked Flush 0 Done Sent Done false
2838 1339 Acked Write 5 Done Sent Done false
2839 1340 Acked Write 6 Done Sent Done false
2840 1341 Acked Flush 0 Done Sent Done false
2841 1342 Acked Read 5 Done Sent Done false
2842 1343 Acked Write 2 Done Sent Done false
2843 1344 Acked Read 7 Done Sent Done false
2844 1345 Acked Read 4 Done Sent Done false
2845 1346 Acked Read 7 Done Sent Done false
2846 1347 Acked Write 3 Done Sent Done false
2847 1348 Acked Flush 0 Done Sent Done false
2848 1349 Acked Read 9 Done Sent Done false
2849 1350 Acked Write 4 Done Sent Done false
2850 1351 Acked Read 2 Done Sent Done false
2851 1352 Acked Read 4 Done Sent Done false
2852 1353 Acked Write 4 Done Sent Done false
2853 1354 Acked Write 9 Done Sent Done false
2854 1355 Acked Write 8 Done Sent Done false
2855 1356 Acked Flush 0 Done Sent Done false
2856 1357 Acked Read 5 Done Sent Done false
2857 1358 Acked Read 7 Done Sent Done false
2858 1359 Acked Write 4 Done Sent Done false
2859 1360 Acked Write 9 Done Sent Done false
2860 1361 Acked Read 7 Done Sent Done false
2861 1362 Acked Flush 0 Done Sent Done false
2862 1363 Acked Read 9 Done Sent Done false
2863 1364 Acked Flush 0 Done Sent Done false
2864 1365 Acked Write 7 Done Sent Done false
2865 1366 Acked Read 3 Done Sent Done false
2866 1367 Acked Read 1 Done Sent Done false
2867 1368 Acked Write 8 Done Sent Done false
2868 1369 Acked Read 10 Done Sent Done false
2869 1370 Acked Write 6 Done Sent Done false
2870 1371 Acked Read 3 Done Sent Done false
2871 1372 Acked Read 6 Done Sent Done false
2872 1373 Acked Flush 0 Done Sent Done false
2873 1374 Acked Read 3 Done Sent Done false
2874 1375 Acked Read 4 Done Sent Done false
2875 1376 Acked Read 8 Done Sent Done false
2876 1377 Acked Read 4 Done Sent Done false
2877 1378 Acked Read 7 Done Sent Done false
2878 1379 Acked Read 6 Done Sent Done false
2879 1380 Acked Write 10 Done Sent Done false
2880 1381 Acked Write 8 Done Sent Done false
2881 1382 Acked Read 2 Done Sent Done false
2882 1383 Acked Read 10 Done Sent Done false
2883 1384 Acked Read 3 Done Sent Done false
2884 1385 Acked Read 4 Done Sent Done false
2885 1386 Acked Read 4 Done Sent Done false
2886 1387 Acked Read 4 Done Sent Done false
2887 1388 Acked Write 1 Done Sent Done false
2888 1389 Acked Read 8 Done Sent Done false
2889 1390 Acked Read 5 Done Sent Done false
2890 1391 Acked Flush 0 Done Sent Done false
2891 1392 Acked Read 2 Done Sent Done false
2892 1393 Acked Read 6 Done Sent Done false
2893 1394 Acked Read 9 Done Sent Done false
2894 1395 Acked Write 2 Done Sent Done false
2895 1396 Acked Read 3 Done Sent Done false
2896 1397 Acked Write 4 Done Sent Done false
2897 1398 Acked Read 10 Done Sent Done false
2898 1399 Acked Read 9 Done Sent Done false
2899 1400 Acked Write 9 Done Sent Done false
2900 1401 Acked Read 2 Done Sent Done false
2901 1402 Acked Write 2 Done Sent Done false
2902 1403 Acked Read 5 Done Sent Done false
2903 1404 Acked Write 6 Done Sent Done false
2904 1405 Acked Read 4 Done Sent Done false
2905 1406 Acked Read 3 Done Sent Done false
2906 1407 Acked Flush 0 Done Sent Done false
2907 1408 Acked Read 10 Done Sent Done false
2908 1409 Acked Read 8 Done Sent Done false
2909 1410 Acked Write 2 Done Sent Done false
2910 1411 Acked Write 6 Done Sent Done false
2911 1412 Acked Write 6 Done Sent Done false
2912 1413 Acked Read 7 Done Sent Done false
2913 1414 Acked Read 1 Done Sent Done false
2914 1415 Acked Read 5 Done Sent Done false
2915 1416 Acked Write 7 Done Sent Done false
2916 1417 Acked Write 10 Done Sent Done false
2917 1418 Acked Read 10 Done Sent Done false
2918 1419 Acked Write 4 Done Sent Done false
2919 1420 Acked Read 9 Done Sent Done false
2920 1421 Acked Read 5 Done Sent Done false
2921 1422 Acked Write 5 Done Sent Done false
2922 1423 Acked Read 4 Done Sent Done false
2923 1424 Acked Write 8 Done Sent Done false
2924 1425 Acked Read 6 Done Sent Done false
2925 1426 Acked Write 10 Done Sent Done false
2926 1427 Acked Read 7 Done Sent Done false
2927 1428 Acked Write 1 Done Sent Done false
2928 1429 Acked Read 4 Done Sent Done false
2929 1430 Acked Flush 0 Done Sent Done false
2930 1431 Acked Read 1 Done Sent Done false
2931 1432 Acked Flush 0 Done Sent Done false
2932 1433 Acked Read 9 Done Sent Done false
2933 1434 Acked Read 6 Done Sent Done false
2934 1435 Acked Write 6 Done Sent Done false
2935 1436 Acked Write 10 Done Sent Done false
2936 1437 Acked Write 6 Done Sent Done false
2937 1438 Acked Read 10 Done Sent Done false
2938 1439 Acked Read 4 Done Sent Done false
2939 1440 Acked Read 7 Done Sent Done false
2940 1441 Acked Write 1 Done Sent Done false
2941 1442 Acked Flush 0 Done Sent Done false
2942 1443 Acked Read 1 Done Sent Done false
2943 1444 Acked Read 5 Done Sent Done false
2944 1445 Acked Write 8 Done Sent Done false
2945 1446 Acked Read 7 Done Sent Done false
2946 1447 Acked Read 8 Done Sent Done false
2947 1448 Acked Write 10 Done Sent Done false
2948 1449 Acked Flush 0 Done Sent Done false
2949 1450 Acked Read 2 Done Sent Done false
2950 1451 Acked Read 2 Done Sent Done false
2951 1452 Acked Write 5 Done Sent Done false
2952 1453 Acked Read 2 Done Sent Done false
2953 1454 Acked Read 1 Done Sent Done false
2954 1455 Acked Read 8 Done Sent Done false
2955 1456 Acked Write 10 Done Sent Done false
2956 1457 Acked Read 7 Done Sent Done false
2957 1458 Acked Flush 0 Done Sent Done false
2958 1459 Acked Read 9 Done Sent Done false
2959 1460 Acked Read 7 Done Sent Done false
2960 1461 Acked Flush 0 Done Sent Done false
2961 1462 Acked Read 7 Done Sent Done false
2962 1463 Acked Read 1 Done Sent Done false
2963 1464 Acked Write 10 Done Sent Done false
2964 1465 Acked Read 4 Done Sent Done false
2965 1466 Acked Write 4 Done Sent Done false
2966 1467 Acked Read 2 Done Sent Done false
2967 1468 Acked Write 1 Done Sent Done false
2968 1469 Acked Read 9 Done Sent Done false
2969 1470 Acked Read 9 Done Sent Done false
2970 1471 Acked Write 10 Done Sent Done false
2971 1472 Acked Write 5 Done Sent Done false
2972 1473 Acked Read 3 Done Sent Done false
2973 1474 Acked Read 2 Done Sent Done false
2974 1475 Acked Read 2 Done Sent Done false
2975 1476 Acked Write 8 Done Sent Done false
2976 1477 Acked Write 6 Done Sent Done false
2977 1478 Acked Write 3 Done Sent Done false
2978 1479 Acked Read 5 Done Sent Done false
2979 1480 Acked Read 3 Done Sent Done false
2980 1481 Acked Write 6 Done Sent Done false
2981 1482 Acked Read 8 Done Sent Done false
2982 1483 Acked Write 10 Done Sent Done false
2983 1484 Acked Read 8 Done Sent Done false
2984 1485 Acked Write 7 Done Sent Done false
2985 1486 Acked Write 8 Done Sent Done false
2986 1487 Acked Read 2 Done Sent Done false
2987 1488 Acked Read 7 Done Sent Done false
2988 1489 Acked Write 4 Done Sent Done false
2989 1490 Acked Write 8 Done Sent Done false
2990 1491 Acked Read 4 Done Sent Done false
2991 1492 Acked Read 1 Done Sent Done false
2992 1493 Acked Read 10 Done Sent Done false
2993 1494 Acked Flush 0 Done Sent Done false
2994 1495 Acked Read 3 Done Sent Done false
2995 1496 Acked Write 3 Done Sent Done false
2996 1497 Acked Read 2 Done Sent Done false
2997 1498 Acked Write 2 Done Sent Done false
2998 1499 Acked Write 4 Done Sent Done false
2999 1500 Acked Flush 0 Done Sent Done false
3000 1501 Acked Read 6 Done Sent Done false
3001 1502 Acked Write 2 Done Sent Done false
3002 1503 Acked Write 10 Done Sent Done false
3003 1504 Acked Read 10 Done Sent Done false
3004 1505 Acked Write 7 Done Sent Done false
3005 1506 Acked Write 2 Done Sent Done false
3006 1507 Acked Write 9 Done Sent Done false
3007 1508 Acked Read 5 Done Sent Done false
3008 1509 Acked Flush 0 Done Sent Done false
3009 1510 Acked Read 1 Done Sent Done false
3010 1511 Acked Write 6 Done Sent Done false
3011 1512 Acked Write 6 Done Sent Done false
3012 1513 Acked Read 7 Done Sent Done false
3013 1514 Acked Read 3 Done Sent Done false
3014 1515 Acked Read 10 Done Sent Done false
3015 1516 Acked Read 3 Done Sent Done false
3016 1517 Acked Write 3 Done Sent Done false
3017 1518 Acked Write 6 Done Sent Done false
3018 1519 Acked Read 4 Done Sent Done false
3019 1520 Acked Read 8 Done Sent Done false
3020 1521 Acked Write 3 Done Sent Done false
3021 1522 Acked Write 1 Done Sent Done false
3022 1523 Acked Read 5 Done Sent Done false
3023 1524 Acked Read 3 Done Sent Done false
3024 1525 Acked Write 9 Done Sent Done false
3025 1526 Acked Write 6 Done Sent Done false
3026 1527 Acked Read 6 Done Sent Done false
3027 1528 Acked Read 5 Done Sent Done false
3028 1529 Acked Write 7 Done Sent Done false
3029 1530 Acked Read 10 Done Sent Done false
3030 1531 Acked Read 2 Done Sent Done false
3031 1532 Acked Read 1 Done Sent Done false
3032 1533 Acked Write 6 Done Sent Done false
3033 1534 Acked Read 7 Done Sent Done false
3034 1535 Acked Read 3 Done Sent Done false
3035 1536 Acked Read 9 Done Sent Done false
3036 1537 Acked Read 8 Done Sent Done false
3037 1538 Acked Read 9 Done Sent Done false
3038 1539 Acked Read 6 Done Sent Done false
3039 1540 Acked Write 2 Done Sent Done false
3040 1541 Acked Read 2 Done Sent Done false
3041 1542 Acked Flush 0 Done Sent Done false
3042 1543 Acked Read 8 Done Sent Done false
3043 1544 Acked Read 7 Done Sent Done false
3044 1545 Acked Read 6 Done Sent Done false
3045 1546 Acked Flush 0 Done Sent Done false
3046 1547 Acked Write 3 Done Sent Done false
3047 1548 Acked Read 6 Done Sent Done false
3048 1549 Acked Read 1 Done Sent Done false
3049 1550 Acked Read 4 Done Sent Done false
3050 1551 Acked Write 4 Done Sent Done false
3051 1552 Acked Write 4 Done Sent Done false
3052 1553 Acked Flush 0 Done Sent Done false
3053 1554 Acked Read 3 Done Sent Done false
3054 1555 Acked Write 4 Done Sent Done false
3055 1556 Acked Read 10 Done Sent Done false
3056 1557 Acked Read 2 Done Sent Done false
3057 1558 Acked Read 1 Done Sent Done false
3058 1559 Acked Read 5 Done Sent Done false
3059 1560 Acked Read 8 Done Sent Done false
3060 1561 Acked Write 3 Done Sent Done false
3061 1562 Acked Write 7 Done Sent Done false
3062 1563 Acked Read 2 Done Sent Done false
3063 1564 Acked Read 4 Done Sent Done false
3064 1565 Acked Read 3 Done Sent Done false
3065 1566 Acked Read 1 Done Sent Done false
3066 1567 Acked Write 4 Done Sent Done false
3067 1568 Acked Read 8 Done Sent Done false
3068 1569 Acked Write 3 Done Sent Done false
3069 1570 Acked Read 9 Done Sent Done false
3070 1571 Acked Read 8 Done Sent Done false
3071 1572 Acked Read 5 Done Sent Done false
3072 1573 Acked Write 10 Done Sent Done false
3073 1574 Acked Read 1 Done Sent Done false
3074 1575 Acked Read 2 Done Sent Done false
3075 1576 Acked Flush 0 Done Sent Done false
3076 1577 Acked Read 1 Done Sent Done false
3077 1578 Acked Write 4 Done Sent Done false
3078 1579 Acked Write 4 Done Sent Done false
3079 1580 Acked Read 2 Done Sent Done false
3080 1581 Acked Read 7 Done Sent Done false
3081 1582 Acked Read 3 Done Sent Done false
3082 1583 Acked Write 5 Done Sent Done false
3083 1584 Acked Read 5 Done Sent Done false
3084 1585 Acked Read 1 Done Sent Done false
3085 1586 Acked Read 3 Done Sent Done false
3086 1587 Acked Write 5 Done Sent Done false
3087 1588 Acked Write 7 Done Sent Done false
3088 1589 Acked Read 1 Done Sent Done false
3089 1590 Acked Read 10 Done Sent Done false
3090 1591 Acked Read 3 Done Sent Done false
3091 1592 Acked Read 1 Done Sent Done false
3092 1593 Acked Read 10 Done Sent Done false
3093 1594 Acked Read 9 Done Sent Done false
3094 1595 Acked Read 7 Done Sent Done false
3095 1596 Acked Flush 0 Done Sent Done false
3096 1597 Acked Read 1 Done Sent Done false
3097 1598 Acked Write 8 Done Sent Done false
3098 1599 Acked Write 2 Done Sent Done false
3099 1600 Acked Read 6 Done Sent Done false
3100 1601 Acked Write 4 Done Sent Done false
3101 STATES DS:0 DS:1 DS:2 TOTAL
3102 Sent 0 301 0 301
3103 Done 301 0 301 602
3104 Skipped 0 0 0 0
3105 Error 0 0 0 0
3106 Last Flush: Some(JobId(1596)) Some(JobId(1300)) Some(JobId(1596))
3107 Downstairs last five completed: 1300 1299 1298 1297 1296
3108 Upstairs last five completed: 1601 1600 1599 1598 1597
3109 CLIENT: Up:0 ds:301 act:2
31102025-02-28T06:01:36.889ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
31112025-02-28T06:01:37.396ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
31122025-02-28T06:01:38.397ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
31132025-02-28T06:01:39.399ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
3114 ----------------------------------------------------------------
3115 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3116 States: Active Offline Active
3117 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3118 1301 Acked Read 2 Done Sent Done false
3119 1302 Acked Read 6 Done Sent Done false
3120 1303 Acked Write 5 Done Sent Done false
3121 1304 Acked Read 6 Done Sent Done false
3122 1305 Acked Read 4 Done Sent Done false
3123 1306 Acked Read 4 Done Sent Done false
3124 1307 Acked Flush 0 Done Sent Done false
3125 1308 Acked Write 3 Done Sent Done false
3126 1309 Acked Write 6 Done Sent Done false
3127 1310 Acked Flush 0 Done Sent Done false
3128 1311 Acked Read 4 Done Sent Done false
3129 1312 Acked Read 8 Done Sent Done false
3130 1313 Acked Read 3 Done Sent Done false
3131 1314 Acked Read 6 Done Sent Done false
3132 1315 Acked Write 4 Done Sent Done false
3133 1316 Acked Write 4 Done Sent Done false
3134 1317 Acked Read 8 Done Sent Done false
3135 1318 Acked Flush 0 Done Sent Done false
3136 1319 Acked Write 4 Done Sent Done false
3137 1320 Acked Write 10 Done Sent Done false
3138 1321 Acked Read 1 Done Sent Done false
3139 1322 Acked Read 5 Done Sent Done false
3140 1323 Acked Write 7 Done Sent Done false
3141 1324 Acked Write 1 Done Sent Done false
3142 1325 Acked Read 9 Done Sent Done false
3143 1326 Acked Read 10 Done Sent Done false
3144 1327 Acked Write 5 Done Sent Done false
3145 1328 Acked Write 7 Done Sent Done false
3146 1329 Acked Read 2 Done Sent Done false
3147 1330 Acked Flush 0 Done Sent Done false
3148 1331 Acked Read 2 Done Sent Done false
3149 1332 Acked Read 10 Done Sent Done false
3150 1333 Acked Read 10 Done Sent Done false
3151 1334 Acked Write 3 Done Sent Done false
3152 1335 Acked Flush 0 Done Sent Done false
3153 1336 Acked Read 3 Done Sent Done false
3154 1337 Acked Read 4 Done Sent Done false
3155 1338 Acked Flush 0 Done Sent Done false
3156 1339 Acked Write 5 Done Sent Done false
3157 1340 Acked Write 6 Done Sent Done false
3158 1341 Acked Flush 0 Done Sent Done false
3159 1342 Acked Read 5 Done Sent Done false
3160 1343 Acked Write 2 Done Sent Done false
3161 1344 Acked Read 7 Done Sent Done false
3162 1345 Acked Read 4 Done Sent Done false
3163 1346 Acked Read 7 Done Sent Done false
3164 1347 Acked Write 3 Done Sent Done false
3165 1348 Acked Flush 0 Done Sent Done false
3166 1349 Acked Read 9 Done Sent Done false
3167 1350 Acked Write 4 Done Sent Done false
3168 1351 Acked Read 2 Done Sent Done false
3169 1352 Acked Read 4 Done Sent Done false
3170 1353 Acked Write 4 Done Sent Done false
3171 1354 Acked Write 9 Done Sent Done false
3172 1355 Acked Write 8 Done Sent Done false
3173 1356 Acked Flush 0 Done Sent Done false
3174 1357 Acked Read 5 Done Sent Done false
3175 1358 Acked Read 7 Done Sent Done false
3176 1359 Acked Write 4 Done Sent Done false
3177 1360 Acked Write 9 Done Sent Done false
3178 1361 Acked Read 7 Done Sent Done false
3179 1362 Acked Flush 0 Done Sent Done false
3180 1363 Acked Read 9 Done Sent Done false
3181 1364 Acked Flush 0 Done Sent Done false
3182 1365 Acked Write 7 Done Sent Done false
3183 1366 Acked Read 3 Done Sent Done false
3184 1367 Acked Read 1 Done Sent Done false
3185 1368 Acked Write 8 Done Sent Done false
3186 1369 Acked Read 10 Done Sent Done false
3187 1370 Acked Write 6 Done Sent Done false
3188 1371 Acked Read 3 Done Sent Done false
3189 1372 Acked Read 6 Done Sent Done false
3190 1373 Acked Flush 0 Done Sent Done false
3191 1374 Acked Read 3 Done Sent Done false
3192 1375 Acked Read 4 Done Sent Done false
3193 1376 Acked Read 8 Done Sent Done false
3194 1377 Acked Read 4 Done Sent Done false
3195 1378 Acked Read 7 Done Sent Done false
3196 1379 Acked Read 6 Done Sent Done false
3197 1380 Acked Write 10 Done Sent Done false
3198 1381 Acked Write 8 Done Sent Done false
3199 1382 Acked Read 2 Done Sent Done false
3200 1383 Acked Read 10 Done Sent Done false
3201 1384 Acked Read 3 Done Sent Done false
3202 1385 Acked Read 4 Done Sent Done false
3203 1386 Acked Read 4 Done Sent Done false
3204 1387 Acked Read 4 Done Sent Done false
3205 1388 Acked Write 1 Done Sent Done false
3206 1389 Acked Read 8 Done Sent Done false
3207 1390 Acked Read 5 Done Sent Done false
3208 1391 Acked Flush 0 Done Sent Done false
3209 1392 Acked Read 2 Done Sent Done false
3210 1393 Acked Read 6 Done Sent Done false
3211 1394 Acked Read 9 Done Sent Done false
3212 1395 Acked Write 2 Done Sent Done false
3213 1396 Acked Read 3 Done Sent Done false
3214 1397 Acked Write 4 Done Sent Done false
3215 1398 Acked Read 10 Done Sent Done false
3216 1399 Acked Read 9 Done Sent Done false
3217 1400 Acked Write 9 Done Sent Done false
3218 1401 Acked Read 2 Done Sent Done false
3219 1402 Acked Write 2 Done Sent Done false
3220 1403 Acked Read 5 Done Sent Done false
3221 1404 Acked Write 6 Done Sent Done false
3222 1405 Acked Read 4 Done Sent Done false
3223 1406 Acked Read 3 Done Sent Done false
3224 1407 Acked Flush 0 Done Sent Done false
3225 1408 Acked Read 10 Done Sent Done false
3226 1409 Acked Read 8 Done Sent Done false
3227 1410 Acked Write 2 Done Sent Done false
3228 1411 Acked Write 6 Done Sent Done false
3229 1412 Acked Write 6 Done Sent Done false
3230 1413 Acked Read 7 Done Sent Done false
3231 1414 Acked Read 1 Done Sent Done false
3232 1415 Acked Read 5 Done Sent Done false
3233 1416 Acked Write 7 Done Sent Done false
3234 1417 Acked Write 10 Done Sent Done false
3235 1418 Acked Read 10 Done Sent Done false
3236 1419 Acked Write 4 Done Sent Done false
3237 1420 Acked Read 9 Done Sent Done false
3238 1421 Acked Read 5 Done Sent Done false
3239 1422 Acked Write 5 Done Sent Done false
3240 1423 Acked Read 4 Done Sent Done false
3241 1424 Acked Write 8 Done Sent Done false
3242 1425 Acked Read 6 Done Sent Done false
3243 1426 Acked Write 10 Done Sent Done false
3244 1427 Acked Read 7 Done Sent Done false
3245 1428 Acked Write 1 Done Sent Done false
3246 1429 Acked Read 4 Done Sent Done false
3247 1430 Acked Flush 0 Done Sent Done false
3248 1431 Acked Read 1 Done Sent Done false
3249 1432 Acked Flush 0 Done Sent Done false
3250 1433 Acked Read 9 Done Sent Done false
3251 1434 Acked Read 6 Done Sent Done false
3252 1435 Acked Write 6 Done Sent Done false
3253 1436 Acked Write 10 Done Sent Done false
3254 1437 Acked Write 6 Done Sent Done false
3255 1438 Acked Read 10 Done Sent Done false
3256 1439 Acked Read 4 Done Sent Done false
3257 1440 Acked Read 7 Done Sent Done false
3258 1441 Acked Write 1 Done Sent Done false
3259 1442 Acked Flush 0 Done Sent Done false
3260 1443 Acked Read 1 Done Sent Done false
3261 1444 Acked Read 5 Done Sent Done false
3262 1445 Acked Write 8 Done Sent Done false
3263 1446 Acked Read 7 Done Sent Done false
3264 1447 Acked Read 8 Done Sent Done false
3265 1448 Acked Write 10 Done Sent Done false
3266 1449 Acked Flush 0 Done Sent Done false
3267 1450 Acked Read 2 Done Sent Done false
3268 1451 Acked Read 2 Done Sent Done false
3269 1452 Acked Write 5 Done Sent Done false
3270 1453 Acked Read 2 Done Sent Done false
3271 1454 Acked Read 1 Done Sent Done false
3272 1455 Acked Read 8 Done Sent Done false
3273 1456 Acked Write 10 Done Sent Done false
3274 1457 Acked Read 7 Done Sent Done false
3275 1458 Acked Flush 0 Done Sent Done false
3276 1459 Acked Read 9 Done Sent Done false
3277 1460 Acked Read 7 Done Sent Done false
3278 1461 Acked Flush 0 Done Sent Done false
3279 1462 Acked Read 7 Done Sent Done false
3280 1463 Acked Read 1 Done Sent Done false
3281 1464 Acked Write 10 Done Sent Done false
3282 1465 Acked Read 4 Done Sent Done false
3283 1466 Acked Write 4 Done Sent Done false
3284 1467 Acked Read 2 Done Sent Done false
3285 1468 Acked Write 1 Done Sent Done false
3286 1469 Acked Read 9 Done Sent Done false
3287 1470 Acked Read 9 Done Sent Done false
3288 1471 Acked Write 10 Done Sent Done false
3289 1472 Acked Write 5 Done Sent Done false
3290 1473 Acked Read 3 Done Sent Done false
3291 1474 Acked Read 2 Done Sent Done false
3292 1475 Acked Read 2 Done Sent Done false
3293 1476 Acked Write 8 Done Sent Done false
3294 1477 Acked Write 6 Done Sent Done false
3295 1478 Acked Write 3 Done Sent Done false
3296 1479 Acked Read 5 Done Sent Done false
3297 1480 Acked Read 3 Done Sent Done false
3298 1481 Acked Write 6 Done Sent Done false
3299 1482 Acked Read 8 Done Sent Done false
3300 1483 Acked Write 10 Done Sent Done false
3301 1484 Acked Read 8 Done Sent Done false
3302 1485 Acked Write 7 Done Sent Done false
3303 1486 Acked Write 8 Done Sent Done false
3304 1487 Acked Read 2 Done Sent Done false
3305 1488 Acked Read 7 Done Sent Done false
3306 1489 Acked Write 4 Done Sent Done false
3307 1490 Acked Write 8 Done Sent Done false
3308 1491 Acked Read 4 Done Sent Done false
3309 1492 Acked Read 1 Done Sent Done false
3310 1493 Acked Read 10 Done Sent Done false
3311 1494 Acked Flush 0 Done Sent Done false
3312 1495 Acked Read 3 Done Sent Done false
3313 1496 Acked Write 3 Done Sent Done false
3314 1497 Acked Read 2 Done Sent Done false
3315 1498 Acked Write 2 Done Sent Done false
3316 1499 Acked Write 4 Done Sent Done false
3317 1500 Acked Flush 0 Done Sent Done false
3318 1501 Acked Read 6 Done Sent Done false
3319 1502 Acked Write 2 Done Sent Done false
3320 1503 Acked Write 10 Done Sent Done false
3321 1504 Acked Read 10 Done Sent Done false
3322 1505 Acked Write 7 Done Sent Done false
3323 1506 Acked Write 2 Done Sent Done false
3324 1507 Acked Write 9 Done Sent Done false
3325 1508 Acked Read 5 Done Sent Done false
3326 1509 Acked Flush 0 Done Sent Done false
3327 1510 Acked Read 1 Done Sent Done false
3328 1511 Acked Write 6 Done Sent Done false
3329 1512 Acked Write 6 Done Sent Done false
3330 1513 Acked Read 7 Done Sent Done false
3331 1514 Acked Read 3 Done Sent Done false
3332 1515 Acked Read 10 Done Sent Done false
3333 1516 Acked Read 3 Done Sent Done false
3334 1517 Acked Write 3 Done Sent Done false
3335 1518 Acked Write 6 Done Sent Done false
3336 1519 Acked Read 4 Done Sent Done false
3337 1520 Acked Read 8 Done Sent Done false
3338 1521 Acked Write 3 Done Sent Done false
3339 1522 Acked Write 1 Done Sent Done false
3340 1523 Acked Read 5 Done Sent Done false
3341 1524 Acked Read 3 Done Sent Done false
3342 1525 Acked Write 9 Done Sent Done false
3343 1526 Acked Write 6 Done Sent Done false
3344 1527 Acked Read 6 Done Sent Done false
3345 1528 Acked Read 5 Done Sent Done false
3346 1529 Acked Write 7 Done Sent Done false
3347 1530 Acked Read 10 Done Sent Done false
3348 1531 Acked Read 2 Done Sent Done false
3349 1532 Acked Read 1 Done Sent Done false
3350 1533 Acked Write 6 Done Sent Done false
3351 1534 Acked Read 7 Done Sent Done false
3352 1535 Acked Read 3 Done Sent Done false
3353 1536 Acked Read 9 Done Sent Done false
3354 1537 Acked Read 8 Done Sent Done false
3355 1538 Acked Read 9 Done Sent Done false
3356 1539 Acked Read 6 Done Sent Done false
3357 1540 Acked Write 2 Done Sent Done false
3358 1541 Acked Read 2 Done Sent Done false
3359 1542 Acked Flush 0 Done Sent Done false
3360 1543 Acked Read 8 Done Sent Done false
3361 1544 Acked Read 7 Done Sent Done false
3362 1545 Acked Read 6 Done Sent Done false
3363 1546 Acked Flush 0 Done Sent Done false
3364 1547 Acked Write 3 Done Sent Done false
3365 1548 Acked Read 6 Done Sent Done false
3366 1549 Acked Read 1 Done Sent Done false
3367 1550 Acked Read 4 Done Sent Done false
3368 1551 Acked Write 4 Done Sent Done false
3369 1552 Acked Write 4 Done Sent Done false
3370 1553 Acked Flush 0 Done Sent Done false
3371 1554 Acked Read 3 Done Sent Done false
3372 1555 Acked Write 4 Done Sent Done false
3373 1556 Acked Read 10 Done Sent Done false
3374 1557 Acked Read 2 Done Sent Done false
3375 1558 Acked Read 1 Done Sent Done false
3376 1559 Acked Read 5 Done Sent Done false
3377 1560 Acked Read 8 Done Sent Done false
3378 1561 Acked Write 3 Done Sent Done false
3379 1562 Acked Write 7 Done Sent Done false
3380 1563 Acked Read 2 Done Sent Done false
3381 1564 Acked Read 4 Done Sent Done false
3382 1565 Acked Read 3 Done Sent Done false
3383 1566 Acked Read 1 Done Sent Done false
3384 1567 Acked Write 4 Done Sent Done false
3385 1568 Acked Read 8 Done Sent Done false
3386 1569 Acked Write 3 Done Sent Done false
3387 1570 Acked Read 9 Done Sent Done false
3388 1571 Acked Read 8 Done Sent Done false
3389 1572 Acked Read 5 Done Sent Done false
3390 1573 Acked Write 10 Done Sent Done false
3391 1574 Acked Read 1 Done Sent Done false
3392 1575 Acked Read 2 Done Sent Done false
3393 1576 Acked Flush 0 Done Sent Done false
3394 1577 Acked Read 1 Done Sent Done false
3395 1578 Acked Write 4 Done Sent Done false
3396 1579 Acked Write 4 Done Sent Done false
3397 1580 Acked Read 2 Done Sent Done false
3398 1581 Acked Read 7 Done Sent Done false
3399 1582 Acked Read 3 Done Sent Done false
3400 1583 Acked Write 5 Done Sent Done false
3401 1584 Acked Read 5 Done Sent Done false
3402 1585 Acked Read 1 Done Sent Done false
3403 1586 Acked Read 3 Done Sent Done false
3404 1587 Acked Write 5 Done Sent Done false
3405 1588 Acked Write 7 Done Sent Done false
3406 1589 Acked Read 1 Done Sent Done false
3407 1590 Acked Read 10 Done Sent Done false
3408 1591 Acked Read 3 Done Sent Done false
3409 1592 Acked Read 1 Done Sent Done false
3410 1593 Acked Read 10 Done Sent Done false
3411 1594 Acked Read 9 Done Sent Done false
3412 1595 Acked Read 7 Done Sent Done false
3413 1596 Acked Flush 0 Done Sent Done false
3414 1597 Acked Read 1 Done Sent Done false
3415 1598 Acked Write 8 Done Sent Done false
3416 1599 Acked Write 2 Done Sent Done false
3417 1600 Acked Read 6 Done Sent Done false
3418 1601 Acked Write 4 Done Sent Done false
3419 STATES DS:0 DS:1 DS:2 TOTAL
3420 Sent 0 301 0 301
3421 Done 301 0 301 602
3422 Skipped 0 0 0 0
3423 Error 0 0 0 0
3424 Last Flush: Some(JobId(1596)) Some(JobId(1300)) Some(JobId(1596))
3425 Downstairs last five completed: 1300 1299 1298 1297 1296
3426 Upstairs last five completed: 1601 1600 1599 1598 1597
3427 CLIENT: Up:0 ds:301 act:2
34282025-02-28T06:01:40.891ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34292025-02-28T06:01:41.402ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34302025-02-28T06:01:42.403ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34312025-02-28T06:01:43.405ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34322025-02-28T06:01:44.134ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34332025-02-28T06:01:44.135ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47964) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34342025-02-28T06:01:44.136ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34352025-02-28T06:01:44.136ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34362025-02-28T06:01:44.136ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34372025-02-28T06:01:44.136ZINFOcrucible: send last flush ID to this DS: Some(JobId(1300)) = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34382025-02-28T06:01:44.136ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34392025-02-28T06:01:44.136ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(1300)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34402025-02-28T06:01:44.136ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(1300)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
3441 ----------------------------------------------------------------
3442 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:5
3443 States: Active Active Active
3444 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3445 1597 Acked Read 1 Done Done Done true
3446 1598 Acked Write 8 Done Done Done true
3447 1599 Acked Write 2 Done Done Done true
3448 1600 Acked Read 6 Done Done Done true
3449 1601 Acked Write 4 Done Done Done true
3450 STATES DS:0 DS:1 DS:2 TOTAL
3451 Sent 0 0 0 0
3452 Done 5 5 5 15
3453 Skipped 0 0 0 0
3454 Error 0 0 0 0
3455 Last Flush: Some(JobId(1596)) Some(JobId(1596)) Some(JobId(1596))
3456 Downstairs last five completed: 1596 1595 1594 1593 1592
3457 Upstairs last five completed: 1601 1600 1599 1598 1597
3458 CLIENT: Up:0 ds:5 act:3
3459 ----------------------------------------------------------------
3460 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3461 Downstairs last five completed: 1602 1601 1600 1599 1598
3462 Upstairs last five completed: 1602 1601 1600 1599 1598
3463 CLIENT: Up:0 ds:0 act:3
3464 Replay: All jobs finished
3465 Main task has work to do, go find it
3466 got dsc Stop(1)
3467 stop 1
3468 Feb 28 06:01:48.393 INFO request completed, latency_us: 363, [8820] Got stop action so:false kr:false
3469 response_code: 204, uri: /stop/cid/1, method: POST, req_id: 2cbcbefd-ad05-4664-8466-bf54009c63d9, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
34702025-02-28T06:01:48.393ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34712025-02-28T06:01:48.393ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34722025-02-28T06:01:48.393ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
3473 Feb 28 06:01:48.393 INFO request completed, latency_us: 133, response_code: 200, uri: /state/cid/1, method: GET, req_id: 7f6b3f5f-ae30-40ae-b541-595af776e11a, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:01:48.393888306Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
3474 9998
3475 [8820] Exited with: ExitStatus(unix_wait_status(9))
3476 [8820][1] reports Exit
3477 Feb 28 06:01:52.396 INFO request completed, latency_us: 353, response_code: 200, uri: /state/cid/1, method: GET, req_id: 21708c3b-f2ef-44e9-a1e6-8770bbd43775, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
3478 001/300 Write block 4415 len 36864 data: 2 2 2 2 2 2 2 2 2
3479 002/300 Write block 3108 len 40960 data: 2 2 3 3 3 3 4 4 4 4
3480 003/300 Write block 4647 len 4096 data: 2
3481 004/300 Write block 978 len 24576 data: 2 2 2 2 2 2
3482 005/300 Read block 2590 len 32768
3483 006/300 Read block 1008 len 24576
3484 007/300 Read block 4306 len 4096
3485 008/300 Write block 2257 len 24576 data: 2 2 3 3 2 2
3486 009/300 Write block 3665 len 24576 data: 2 2 2 2 2 2
3487 010/300 Read block 87 len 36864
3488 011/300 Write block 1554 len 32768 data: 2 2 2 2 2 2 2 2
3489 012/300 Write block 3623 len 20480 data: 2 2 2 2 2
3490 013/300 Write block 3907 len 20480 data: 2 2 2 2 2
3491 014/300 Write block 3027 len 32768 data: 2 2 2 2 2 2 2 2
3492 015/300 Read block 767 len 8192
3493 016/300 Flush
3494 017/300 Flush
3495 018/300 Write block 4475 len 20480 data: 2 2 2 2 2
3496 019/300 Read block 2841 len 40960
3497 020/300 Read block 3723 len 40960
3498 021/300 Read block 659 len 12288
3499 022/300 Read block 3468 len 36864
3500 023/300 Read block 1850 len 12288
3501 024/300 Read block 2820 len 40960
3502 025/300 Write block 2364 len 4096 data: 3
3503 026/300 Read block 3191 len 4096
3504 027/300 Write block 2128 len 28672 data: 2 2 2 2 2 2 2
3505 028/300 Read block 3329 len 12288
3506 029/300 Write block 4405 len 40960 data: 3 3 4 5 5 5 5 5 3 2
3507 030/300 Write block 2209 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3508 031/300 Read block 2200 len 36864
3509 032/300 Read block 3284 len 32768
3510 033/300 Read block 2740 len 8192
3511 034/300 Flush
3512 035/300 Write block 3611 len 28672 data: 2 2 2 2 2 2 2
3513 036/300 Flush
3514 037/300 Read block 2444 len 8192
3515 038/300 Read block 1060 len 12288
3516 039/300 Flush
3517 040/300 Write block 422 len 12288 data: 3 2 2
3518 041/300 Read block 231 len 12288
3519 042/300 Write block 2630 len 12288 data: 2 2 2
3520 043/300 Write block 2108 len 28672 data: 2 2 2 2 2 2 2
3521 044/300 Read block 1554 len 24576
3522 045/300 Flush
3523 046/300 Read block 2554 len 12288
3524 047/300 Write block 151 len 20480 data: 2 2 2 2 2
3525 048/300 Write block 4295 len 12288 data: 2 2 2
3526 049/300 Write block 3751 len 12288 data: 2 2 2
3527 050/300 Read block 3313 len 12288
3528 051/300 Flush
3529 052/300 Flush
3530 053/300 Read block 2928 len 24576
3531 054/300 Write block 1324 len 32768 data: 2 2 2 2 2 2 2 2
3532 055/300 Read block 1057 len 24576
3533 056/300 Write block 4808 len 16384 data: 2 2 2 2
3534 057/300 Flush
3535 058/300 Flush
3536 059/300 Write block 172 len 24576 data: 2 2 2 2 2 2
3537 060/300 Read block 4040 len 32768
3538 061/300 Read block 1105 len 28672
3539 062/300 Write block 2275 len 36864 data: 2 2 2 2 2 2 2 2 2
3540 063/300 Flush
3541 064/300 Read block 252 len 32768
3542 065/300 Write block 2622 len 40960 data: 2 2 3 2 2 2 2 2 3 3
3543 066/300 Write block 1125 len 12288 data: 2 2 2
3544 067/300 Write block 4493 len 36864 data: 2 2 2 2 2 2 2 2 2
3545 068/300 Read block 1429 len 32768
3546 069/300 Write block 1706 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3547 070/300 Read block 685 len 36864
3548 071/300 Write block 3430 len 4096 data: 2
3549 072/300 Read block 4669 len 40960
3550 073/300 Write block 785 len 12288 data: 3 2 2
3551 074/300 Write block 3648 len 24576 data: 2 2 2 2 2 2
3552 075/300 Read block 1615 len 16384
3553 076/300 Read block 4655 len 36864
3554 077/300 Read block 2011 len 8192
3555 078/300 Read block 1066 len 24576
3556 079/300 Flush
3557 080/300 Read block 1165 len 24576
3558 081/300 Read block 806 len 12288
3559 082/300 Write block 4112 len 24576 data: 2 2 2 2 2 2
3560 083/300 Read block 4307 len 24576
3561 084/300 Write block 3269 len 16384 data: 2 2 2 2
3562 085/300 Write block 3852 len 24576 data: 2 2 2 2 2 2
3563 086/300 Write block 1308 len 8192 data: 2 2
3564 087/300 Read block 4429 len 28672
3565 088/300 Read block 2590 len 24576
3566 089/300 Read block 3866 len 32768
3567 090/300 Write block 2794 len 32768 data: 2 2 2 2 2 3 3 3
3568 091/300 Write block 4122 len 20480 data: 2 2 2 2 2
3569 092/300 Write block 1620 len 8192 data: 3 2
3570 093/300 Write block 402 len 36864 data: 2 2 2 2 2 2 2 2 2
3571 094/300 Write block 1561 len 8192 data: 3 2
3572 095/300 Write block 2740 len 36864 data: 2 2 2 2 2 2 2 2 2
3573 096/300 Read block 4567 len 8192
3574 097/300 Write block 902 len 4096 data: 2
3575 098/300 Write block 2107 len 8192 data: 2 3
3576 099/300 Write block 81 len 12288 data: 2 2 2
3577 100/300 Read block 2928 len 8192
3578 101/300 Read block 3396 len 12288
3579 102/300 Write block 2064 len 36864 data: 2 2 2 2 2 2 2 2 2
3580 103/300 Write block 2362 len 40960 data: 2 3 4 3 3 2 2 2 2 2
3581 104/300 Read block 685 len 40960
3582 105/300 Read block 4860 len 16384
3583 106/300 Write block 4613 len 12288 data: 2 2 2
3584 107/300 Read block 1420 len 16384
3585 108/300 Read block 4968 len 40960
3586 109/300 Write block 816 len 16384 data: 2 2 2 2
3587 110/300 Write block 4441 len 20480 data: 2 2 2 2 2
3588 111/300 Write block 3029 len 20480 data: 3 3 3 3 3
3589 112/300 Write block 2981 len 16384 data: 2 2 2 2
3590 113/300 Write block 455 len 8192 data: 2 2
3591 114/300 Flush
3592 115/300 Flush
3593 116/300 Write block 3869 len 40960 data: 2 2 2 2 3 3 3 3 2 2
3594 117/300 Write block 3480 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3595 118/300 Flush
3596 119/300 Write block 2376 len 36864 data: 3 3 3 2 2 2 2 2 2
3597 120/300 Read block 658 len 28672
3598 121/300 Read block 4249 len 24576
3599 122/300 Write block 4079 len 4096 data: 2
3600 123/300 Write block 2563 len 8192 data: 2 2
3601 124/300 Write block 3800 len 28672 data: 3 3 2 2 2 2 2
3602 125/300 Read block 1044 len 4096
3603 126/300 Read block 2928 len 36864
3604 127/300 Read block 4261 len 24576
3605 128/300 Flush
3606 129/300 Write block 4048 len 12288 data: 2 2 2
3607 130/300 Read block 1799 len 36864
3608 131/300 Write block 1624 len 32768 data: 2 2 2 2 2 2 2 2
3609 132/300 Flush
3610 133/300 Write block 3741 len 20480 data: 2 2 2 2 2
3611 134/300 Read block 1481 len 40960
3612 135/300 Write block 3605 len 24576 data: 2 2 2 2 2 2
3613 136/300 Read block 4106 len 4096
3614 137/300 Read block 2367 len 24576
3615 138/300 Read block 814 len 24576
3616 139/300 Read block 974 len 40960
3617 140/300 Write block 1624 len 28672 data: 3 3 3 3 3 3 3
3618 141/300 Write block 442 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3619 142/300 Write block 470 len 36864 data: 2 2 2 2 2 2 2 2 2
3620 143/300 Flush
3621 144/300 Read block 2397 len 24576
3622 145/300 Flush
3623 146/300 Write block 1319 len 20480 data: 2 2 2 2 2
3624 147/300 Read block 2524 len 28672
3625 148/300 Read block 3043 len 40960
3626 149/300 Read block 3185 len 28672
3627 150/300 Write block 2230 len 8192 data: 3 4
3628 151/300 Write block 4364 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3629 152/300 Write block 2687 len 20480 data: 2 2 2 2 2
3630 153/300 Read block 3149 len 12288
3631 154/300 Read block 3239 len 20480
3632 155/300 Flush
3633 156/300 Write block 2962 len 32768 data: 3 3 3 3 3 3 3 3
3634 157/300 Flush
3635 158/300 Read block 3406 len 32768
3636 159/300 Read block 4930 len 24576
3637 160/300 Write block 4631 len 16384 data: 2 2 2 2
3638 161/300 Read block 1751 len 4096
3639 162/300 Read block 1777 len 12288
3640 163/300 Write block 469 len 8192 data: 2 3
3641 164/300 Write block 2689 len 40960 data: 3 3 3 2 2 2 2 2 2 2
3642 165/300 Write block 436 len 40960 data: 2 2 3 3 3 2 3 3 3 3
3643 166/300 Read block 1904 len 28672
3644 167/300 Read block 2988 len 8192
3645 168/300 Write block 3341 len 24576 data: 4 3 3 3 3 3
3646 169/300 Flush
3647 170/300 Read block 1653 len 32768
3648 171/300 Read block 4797 len 40960
3649 172/300 Write block 2871 len 16384 data: 2 2 2 2
3650 173/300 Write block 2662 len 24576 data: 2 2 2 2 2 2
3651 174/300 Flush
3652 175/300 Flush
3653 176/300 Read block 1746 len 32768
3654 177/300 Write block 2516 len 40960 data: 3 3 3 3 3 3 3 3 3 3
3655 178/300 Read block 3668 len 20480
3656 179/300 Write block 1040 len 24576 data: 3 3 3 3 3 3
3657 180/300 Read block 454 len 4096
3658 181/300 Write block 1016 len 28672 data: 2 2 2 2 2 2 2
3659 182/300 Read block 345 len 40960
3660 183/300 Read block 3468 len 20480
3661 184/300 Read block 4046 len 24576
3662 185/300 Read block 4199 len 8192
3663 186/300 Read block 3564 len 28672
3664 187/300 Read block 4606 len 4096
3665 188/300 Read block 4911 len 20480
3666 189/300 Read block 3705 len 12288
3667 190/300 Write block 4145 len 28672 data: 2 2 2 2 2 2 2
3668 191/300 Read block 4865 len 8192
3669 192/300 Write block 2356 len 36864 data: 2 2 2 2 2 2 3 4 5
3670 193/300 Read block 134 len 20480
3671 194/300 Read block 4959 len 28672
3672 195/300 Flush
3673 196/300 Write block 667 len 36864 data: 2 2 2 2 2 2 2 2 2
3674 197/300 Write block 4095 len 4096 data: 2
3675 198/300 Write block 4062 len 32768 data: 2 2 2 2 2 2 2 2
3676 199/300 Flush
3677 200/300 Read block 2004 len 28672
3678 201/300 Flush
3679 202/300 Read block 133 len 28672
3680 203/300 Write block 1399 len 28672 data: 2 2 2 3 3 3 3
3681 204/300 Write block 2510 len 20480 data: 2 3 3 3 3
3682 205/300 Read block 2930 len 12288
3683 206/300 Write block 532 len 12288 data: 2 2 2
3684 207/300 Flush
3685 208/300 Write block 4068 len 20480 data: 3 3 2 2 2
3686 209/300 Read block 1027 len 24576
3687 210/300 Read block 3736 len 36864
3688 211/300 Write block 547 len 40960 data: 2 2 2 2 3 4 4 2 2 2
3689 212/300 Flush
3690 213/300 Write block 1261 len 20480 data: 2 2 2 2 2
3691 214/300 Read block 1382 len 28672
3692 215/300 Write block 1894 len 8192 data: 3 3
3693 216/300 Write block 667 len 32768 data: 3 3 3 3 3 3 3 3
3694 217/300 Read block 549 len 20480
3695 218/300 Read block 4521 len 16384
3696 219/300 Read block 1288 len 20480
3697 220/300 Write block 796 len 12288 data: 2 2 2
3698 221/300 Read block 2508 len 24576
3699 222/300 Write block 1578 len 32768 data: 3 3 3 4 4 4 4 4
3700 223/300 Read block 1666 len 4096
3701 224/300 Read block 2010 len 32768
3702 225/300 Read block 3964 len 16384
3703 226/300 Read block 4755 len 40960
3704 227/300 Read block 2324 len 28672
3705 228/300 Write block 1988 len 20480 data: 2 2 2 2 2
3706 229/300 Write block 1654 len 4096 data: 2
3707 230/300 Read block 1057 len 36864
3708 231/300 Write block 885 len 32768 data: 2 2 3 3 2 2 2 2
3709 232/300 Write block 156 len 4096 data: 2
3710 233/300 Read block 2137 len 4096
3711 234/300 Write block 4930 len 24576 data: 2 2 2 2 2 2
3712 235/300 Read block 3196 len 8192
3713 236/300 Read block 659 len 40960
3714 237/300 Read block 1626 len 4096
3715 238/300 Read block 3715 len 40960
3716 239/300 Read block 1225 len 24576
3717 240/300 Write block 4625 len 20480 data: 2 2 2 2 2
3718 241/300 Write block 4127 len 20480 data: 2 2 2 2 2
3719 242/300 Read block 3771 len 8192
3720 243/300 Write block 1063 len 4096 data: 2
3721 244/300 Read block 4975 len 32768
3722 245/300 Read block 4167 len 8192
3723 246/300 Write block 2179 len 20480 data: 2 2 2 2 2
3724 247/300 Write block 4769 len 12288 data: 3 3 3
3725 248/300 Read block 1202 len 16384
3726 249/300 Write block 3633 len 28672 data: 2 2 2 2 2 2 2
3727 250/300 Write block 377 len 28672 data: 3 4 4 4 3 3 3
3728 251/300 Read block 3663 len 12288
3729 252/300 Write block 341 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3730 253/300 Write block 430 len 28672 data: 3 3 3 3 2 2 3
3731 254/300 Read block 1946 len 28672
3732 255/300 Read block 4362 len 8192
3733 256/300 Write block 924 len 20480 data: 2 2 2 2 2
3734 257/300 Write block 4623 len 8192 data: 2 2
3735 258/300 Read block 4555 len 12288
3736 259/300 Read block 2782 len 8192
3737 260/300 Read block 4397 len 16384
3738 261/300 Write block 805 len 36864 data: 2 2 2 2 2 2 2 3 3
3739 262/300 Flush
3740 263/300 Write block 2316 len 32768 data: 2 3 3 3 2 2 2 2
3741 264/300 Flush
3742 265/300 Read block 1304 len 32768
3743 266/300 Flush
3744 267/300 Read block 977 len 24576
3745 268/300 Read block 2392 len 28672
3746 269/300 Read block 4580 len 28672
3747 270/300 Write block 2646 len 24576 data: 2 2 2 2 2 2
3748 271/300 Read block 2216 len 40960
3749 272/300 Write block 2019 len 24576 data: 2 2 3 2 2 2
3750 273/300 Flush
3751 274/300 Write block 2885 len 32768 data: 2 2 2 2 2 2 2 2
3752 275/300 Write block 3452 len 32768 data: 2 2 2 2 2 2 2 2
3753 276/300 Write block 930 len 24576 data: 3 3 3 2 2 2
3754 277/300 Write block 983 len 36864 data: 3 2 2 2 2 2 2 2 2
3755 278/300 Read block 1507 len 12288
3756 279/300 Read block 2443 len 16384
3757 280/300 Read block 2667 len 16384
3758 281/300 Read block 2691 len 28672
3759 282/300 Read block 4994 len 16384
3760 283/300 Read block 4918 len 36864
3761 284/300 Write block 1694 len 32768 data: 3 3 3 3 2 2 2 2
3762 285/300 Write block 853 len 12288 data: 2 2 2
3763 286/300 Read block 4291 len 4096
3764 287/300 Read block 1536 len 8192
3765 288/300 Write block 468 len 12288 data: 2 3 4
3766 289/300 Read block 2397 len 4096
3767 290/300 Write block 4495 len 28672 data: 3 3 3 3 3 3 3
3768 291/300 Read block 3284 len 40960
3769 292/300 Flush
3770 293/300 Flush
3771 294/300 Read block 3778 len 28672
3772 295/300 Write block 274 len 28672 data: 3 3 3 3 3 2 2
3773 296/300 Flush
3774 297/300 Write block 4255 len 36864 data: 2 2 2 2 2 2 2 2 2
3775 298/300 Read block 276 len 24576
3776 299/300 Write block 1176 len 24576 data: 3 3 3 4 3 3
3777 300/300 Write block 3902 len 24576 data: 2 2 2 2 2 3
3778 301/300 Flush
3779 Main task has work to do, go find it
3780 got dsc Start(1)
3781 start 1
3782 Feb 28 06:01:52.633 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/1, method: POST, req_id: 5230b673-1158-4b66-8fc1-6277212ce6b3, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
3783 Replay: started 1, returned:Ok(())
3784 ----------------------------------------------------------------
3785 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3786 States: Active Offline Active
3787 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3788 1603 Acked Write 9 Done Sent Done false
3789 1604 Acked Write 10 Done Sent Done false
3790 1605 Acked Write 1 Done Sent Done false
3791 1606 Acked Write 6 Done Sent Done false
3792 1607 Acked Read 8 Done Sent Done false
3793 1608 Acked Read 6 Done Sent Done false
3794 1609 Acked Read 1 Done Sent Done false
3795 1610 Acked Write 6 Done Sent Done false
3796 1611 Acked Write 6 Done Sent Done false
3797 1612 Acked Read 9 Done Sent Done false
3798 1613 Acked Write 8 Done Sent Done false
3799 1614 Acked Write 5 Done Sent Done false
3800 1615 Acked Write 5 Done Sent Done false
3801 1616 Acked Write 8 Done Sent Done false
3802 1617 Acked Read 2 Done Sent Done false
3803 1618 Acked Flush 0 Done Sent Done false
3804 1619 Acked Flush 0 Done Sent Done false
3805 1620 Acked Write 5 Done Sent Done false
3806 1621 Acked Read 10 Done Sent Done false
3807 1622 Acked Read 10 Done Sent Done false
3808 1623 Acked Read 3 Done Sent Done false
3809 1624 Acked Read 9 Done Sent Done false
3810 1625 Acked Read 3 Done Sent Done false
3811 1626 Acked Read 10 Done Sent Done false
3812 1627 Acked Write 1 Done Sent Done false
3813 1628 Acked Read 1 Done Sent Done false
3814 1629 Acked Write 7 Done Sent Done false
3815 1630 Acked Read 3 Done Sent Done false
3816 1631 Acked Write 10 Done Sent Done false
3817 1632 Acked Write 10 Done Sent Done false
3818 1633 Acked Read 9 Done Sent Done false
3819 1634 Acked Read 8 Done Sent Done false
3820 1635 Acked Read 2 Done Sent Done false
3821 1636 Acked Flush 0 Done Sent Done false
3822 1637 Acked Write 7 Done Sent Done false
3823 1638 Acked Flush 0 Done Sent Done false
3824 1639 Acked Read 2 Done Sent Done false
3825 1640 Acked Read 3 Done Sent Done false
3826 1641 Acked Flush 0 Done Sent Done false
3827 1642 Acked Write 3 Done Sent Done false
3828 1643 Acked Read 3 Done Sent Done false
3829 1644 Acked Write 3 Done Sent Done false
3830 1645 Acked Write 7 Done Sent Done false
3831 1646 Acked Read 6 Done Sent Done false
3832 1647 Acked Flush 0 Done Sent Done false
3833 1648 Acked Read 3 Done Sent Done false
3834 1649 Acked Write 5 Done Sent Done false
3835 1650 Acked Write 3 Done Sent Done false
3836 1651 Acked Write 3 Done Sent Done false
3837 1652 Acked Read 3 Done Sent Done false
3838 1653 Acked Flush 0 Done Sent Done false
3839 1654 Acked Flush 0 Done Sent Done false
3840 1655 Acked Read 6 Done Sent Done false
3841 1656 Acked Write 8 Done Sent Done false
3842 1657 Acked Read 6 Done Sent Done false
3843 1658 Acked Write 4 Done Sent Done false
3844 1659 Acked Flush 0 Done Sent Done false
3845 1660 Acked Flush 0 Done Sent Done false
3846 1661 Acked Write 6 Done Sent Done false
3847 1662 Acked Read 8 Done Sent Done false
3848 1663 Acked Read 7 Done Sent Done false
3849 1664 Acked Write 9 Done Sent Done false
3850 1665 Acked Flush 0 Done Sent Done false
3851 1666 Acked Read 8 Done Sent Done false
3852 1667 Acked Write 10 Done Sent Done false
3853 1668 Acked Write 3 Done Sent Done false
3854 1669 Acked Write 9 Done Sent Done false
3855 1670 Acked Read 8 Done Sent Done false
3856 1671 Acked Write 10 Done Sent Done false
3857 1672 Acked Read 9 Done Sent Done false
3858 1673 Acked Write 1 Done Sent Done false
3859 1674 Acked Read 10 Done Sent Done false
3860 1675 Acked Write 3 Done Sent Done false
3861 1676 Acked Write 6 Done Sent Done false
3862 1677 Acked Read 4 Done Sent Done false
3863 1678 Acked Read 9 Done Sent Done false
3864 1679 Acked Read 2 Done Sent Done false
3865 1680 Acked Read 6 Done Sent Done false
3866 1681 Acked Flush 0 Done Sent Done false
3867 1682 Acked Read 6 Done Sent Done false
3868 1683 Acked Read 3 Done Sent Done false
3869 1684 Acked Write 6 Done Sent Done false
3870 1685 Acked Read 6 Done Sent Done false
3871 1686 Acked Write 4 Done Sent Done false
3872 1687 Acked Write 6 Done Sent Done false
3873 1688 Acked Write 2 Done Sent Done false
3874 1689 Acked Read 7 Done Sent Done false
3875 1690 Acked Read 6 Done Sent Done false
3876 1691 Acked Read 8 Done Sent Done false
3877 1692 Acked Write 8 Done Sent Done false
3878 1693 Acked Write 5 Done Sent Done false
3879 1694 Acked Write 2 Done Sent Done false
3880 1695 Acked Write 9 Done Sent Done false
3881 1696 Acked Write 2 Done Sent Done false
3882 1697 Acked Write 9 Done Sent Done false
3883 1698 Acked Read 2 Done Sent Done false
3884 1699 Acked Write 1 Done Sent Done false
3885 1700 Acked Write 2 Done Sent Done false
3886 1701 Acked Write 3 Done Sent Done false
3887 1702 Acked Read 2 Done Sent Done false
3888 1703 Acked Read 3 Done Sent Done false
3889 1704 Acked Write 9 Done Sent Done false
3890 1705 Acked Write 10 Done Sent Done false
3891 1706 Acked Read 10 Done Sent Done false
3892 1707 Acked Read 4 Done Sent Done false
3893 1708 Acked Write 3 Done Sent Done false
3894 1709 Acked Read 4 Done Sent Done false
3895 1710 Acked Read 10 Done Sent Done false
3896 1711 Acked Write 4 Done Sent Done false
3897 1712 Acked Write 5 Done Sent Done false
3898 1713 Acked Write 5 Done Sent Done false
3899 1714 Acked Write 4 Done Sent Done false
3900 1715 Acked Write 2 Done Sent Done false
3901 1716 Acked Flush 0 Done Sent Done false
3902 1717 Acked Flush 0 Done Sent Done false
3903 1718 Acked Write 10 Done Sent Done false
3904 1719 Acked Write 10 Done Sent Done false
3905 1720 Acked Flush 0 Done Sent Done false
3906 1721 Acked Write 9 Done Sent Done false
3907 1722 Acked Read 7 Done Sent Done false
3908 1723 Acked Read 6 Done Sent Done false
3909 1724 Acked Write 1 Done Sent Done false
3910 1725 Acked Write 2 Done Sent Done false
3911 1726 Acked Write 7 Done Sent Done false
3912 1727 Acked Read 1 Done Sent Done false
3913 1728 Acked Read 9 Done Sent Done false
3914 1729 Acked Read 6 Done Sent Done false
3915 1730 Acked Flush 0 Done Sent Done false
3916 1731 Acked Write 3 Done Sent Done false
3917 1732 Acked Read 9 Done Sent Done false
3918 1733 Acked Write 8 Done Sent Done false
3919 1734 Acked Flush 0 Done Sent Done false
3920 1735 Acked Write 5 Done Sent Done false
3921 1736 Acked Read 10 Done Sent Done false
3922 1737 Acked Write 6 Done Sent Done false
3923 1738 Acked Read 1 Done Sent Done false
3924 1739 Acked Read 6 Done Sent Done false
3925 1740 Acked Read 6 Done Sent Done false
3926 1741 Acked Read 10 Done Sent Done false
3927 1742 Acked Write 7 Done Sent Done false
3928 1743 Acked Write 10 Done Sent Done false
3929 1744 Acked Write 9 Done Sent Done false
3930 1745 Acked Flush 0 Done Sent Done false
3931 1746 Acked Read 6 Done Sent Done false
3932 1747 Acked Flush 0 Done Sent Done false
3933 1748 Acked Write 5 Done Sent Done false
3934 1749 Acked Read 7 Done Sent Done false
3935 1750 Acked Read 10 Done Sent Done false
3936 1751 Acked Read 7 Done Sent Done false
3937 1752 Acked Write 2 Done Sent Done false
3938 1753 Acked Write 10 Done Sent Done false
3939 1754 Acked Write 5 Done Sent Done false
3940 1755 Acked Read 3 Done Sent Done false
3941 1756 Acked Read 5 Done Sent Done false
3942 1757 Acked Flush 0 Done Sent Done false
3943 1758 Acked Write 8 Done Sent Done false
3944 1759 Acked Flush 0 Done Sent Done false
3945 1760 Acked Read 8 Done Sent Done false
3946 1761 Acked Read 6 Done Sent Done false
3947 1762 Acked Write 4 Done Sent Done false
3948 1763 Acked Read 1 Done Sent Done false
3949 1764 Acked Read 3 Done Sent Done false
3950 1765 Acked Write 2 Done Sent Done false
3951 1766 Acked Write 10 Done Sent Done false
3952 1767 Acked Write 10 Done Sent Done false
3953 1768 Acked Read 7 Done Sent Done false
3954 1769 Acked Read 2 Done Sent Done false
3955 1770 Acked Write 6 Done Sent Done false
3956 1771 Acked Flush 0 Done Sent Done false
3957 1772 Acked Read 8 Done Sent Done false
3958 1773 Acked Read 10 Done Sent Done false
3959 1774 Acked Write 4 Done Sent Done false
3960 1775 Acked Write 6 Done Sent Done false
3961 1776 Acked Flush 0 Done Sent Done false
3962 1777 Acked Flush 0 Done Sent Done false
3963 1778 Acked Read 8 Done Sent Done false
3964 1779 Acked Write 10 Done Sent Done false
3965 1780 Acked Read 5 Done Sent Done false
3966 1781 Acked Write 6 Done Sent Done false
3967 1782 Acked Read 1 Done Sent Done false
3968 1783 Acked Write 7 Done Sent Done false
3969 1784 Acked Read 10 Done Sent Done false
3970 1785 Acked Read 5 Done Sent Done false
3971 1786 Acked Read 6 Done Sent Done false
3972 1787 Acked Read 2 Done Sent Done false
3973 1788 Acked Read 7 Done Sent Done false
3974 1789 Acked Read 1 Done Sent Done false
3975 1790 Acked Read 5 Done Sent Done false
3976 1791 Acked Read 3 Done Sent Done false
3977 1792 Acked Write 7 Done Sent Done false
3978 1793 Acked Read 2 Done Sent Done false
3979 1794 Acked Write 9 Done Sent Done false
3980 1795 Acked Read 5 Done Sent Done false
3981 1796 Acked Read 7 Done Sent Done false
3982 1797 Acked Flush 0 Done Sent Done false
3983 1798 Acked Write 9 Done Sent Done false
3984 1799 Acked Write 1 Done Sent Done false
3985 1800 Acked Write 8 Done Sent Done false
3986 1801 Acked Flush 0 Done Sent Done false
3987 1802 Acked Read 7 Done Sent Done false
3988 1803 Acked Flush 0 Done Sent Done false
3989 1804 Acked Read 7 Done Sent Done false
3990 1805 Acked Write 7 Done Sent Done false
3991 1806 Acked Write 5 Done Sent Done false
3992 1807 Acked Read 3 Done Sent Done false
3993 1808 Acked Write 3 Done Sent Done false
3994 1809 Acked Flush 0 Done Sent Done false
3995 1810 Acked Write 5 Done Sent Done false
3996 1811 Acked Read 6 Done Sent Done false
3997 1812 Acked Read 9 Done Sent Done false
3998 1813 Acked Write 10 Done Sent Done false
3999 1814 Acked Flush 0 Done Sent Done false
4000 1815 Acked Write 5 Done Sent Done false
4001 1816 Acked Read 7 Done Sent Done false
4002 1817 Acked Write 2 Done Sent Done false
4003 1818 Acked Write 8 Done Sent Done false
4004 1819 Acked Read 5 Done Sent Done false
4005 1820 Acked Read 4 Done Sent Done false
4006 1821 Acked Read 5 Done Sent Done false
4007 1822 Acked Write 3 Done Sent Done false
4008 1823 Acked Read 6 Done Sent Done false
4009 1824 Acked Write 8 Done Sent Done false
4010 1825 Acked Read 1 Done Sent Done false
4011 1826 Acked Read 8 Done Sent Done false
4012 1827 Acked Read 4 Done Sent Done false
4013 1828 Acked Read 10 Done Sent Done false
4014 1829 Acked Read 7 Done Sent Done false
4015 1830 Acked Write 5 Done Sent Done false
4016 1831 Acked Write 1 Done Sent Done false
4017 1832 Acked Read 9 Done Sent Done false
4018 1833 Acked Write 8 Done Sent Done false
4019 1834 Acked Write 1 Done Sent Done false
4020 1835 Acked Read 1 Done Sent Done false
4021 1836 Acked Write 6 Done Sent Done false
4022 1837 Acked Read 2 Done Sent Done false
4023 1838 Acked Read 10 Done Sent Done false
4024 1839 Acked Read 1 Done Sent Done false
4025 1840 Acked Read 10 Done Sent Done false
4026 1841 Acked Read 6 Done Sent Done false
4027 1842 Acked Write 5 Done Sent Done false
4028 1843 Acked Write 5 Done Sent Done false
4029 1844 Acked Read 2 Done Sent Done false
4030 1845 Acked Write 1 Done Sent Done false
4031 1846 Acked Read 8 Done Sent Done false
4032 1847 Acked Read 2 Done Sent Done false
4033 1848 Acked Write 5 Done Sent Done false
4034 1849 Acked Write 3 Done Sent Done false
4035 1850 Acked Read 4 Done Sent Done false
4036 1851 Acked Write 7 Done Sent Done false
4037 1852 Acked Write 7 Done Sent Done false
4038 1853 Acked Read 3 Done Sent Done false
4039 1854 Acked Write 10 Done Sent Done false
4040 1855 Acked Write 7 Done Sent Done false
4041 1856 Acked Read 7 Done Sent Done false
4042 1857 Acked Read 2 Done Sent Done false
4043 1858 Acked Write 5 Done Sent Done false
4044 1859 Acked Write 2 Done Sent Done false
4045 1860 Acked Read 3 Done Sent Done false
4046 1861 Acked Read 2 Done Sent Done false
4047 1862 Acked Read 4 Done Sent Done false
4048 1863 Acked Write 9 Done Sent Done false
4049 1864 Acked Flush 0 Done Sent Done false
4050 1865 Acked Write 8 Done Sent Done false
4051 1866 Acked Flush 0 Done Sent Done false
4052 1867 Acked Read 8 Done Sent Done false
4053 1868 Acked Flush 0 Done Sent Done false
4054 1869 Acked Read 6 Done Sent Done false
4055 1870 Acked Read 7 Done Sent Done false
4056 1871 Acked Read 7 Done Sent Done false
4057 1872 Acked Write 6 Done Sent Done false
4058 1873 Acked Read 10 Done Sent Done false
4059 1874 Acked Write 6 Done Sent Done false
4060 1875 Acked Flush 0 Done Sent Done false
4061 1876 Acked Write 8 Done Sent Done false
4062 1877 Acked Write 8 Done Sent Done false
4063 1878 Acked Write 6 Done Sent Done false
4064 1879 Acked Write 9 Done Sent Done false
4065 1880 Acked Read 3 Done Sent Done false
4066 1881 Acked Read 4 Done Sent Done false
4067 1882 Acked Read 4 Done Sent Done false
4068 1883 Acked Read 7 Done Sent Done false
4069 1884 Acked Read 4 Done Sent Done false
4070 1885 Acked Read 9 Done Sent Done false
4071 1886 Acked Write 8 Done Sent Done false
4072 1887 Acked Write 3 Done Sent Done false
4073 1888 Acked Read 1 Done Sent Done false
4074 1889 Acked Read 2 Done Sent Done false
4075 1890 Acked Write 3 Done Sent Done false
4076 1891 Acked Read 1 Done Sent Done false
4077 1892 Acked Write 7 Done Sent Done false
4078 1893 Acked Read 10 Done Sent Done false
4079 1894 Acked Flush 0 Done Sent Done false
4080 1895 Acked Flush 0 Done Sent Done false
4081 1896 Acked Read 7 Done Sent Done false
4082 1897 Acked Write 7 Done Sent Done false
4083 1898 Acked Flush 0 Done Sent Done false
4084 1899 Acked Write 9 Done Sent Done false
4085 1900 Acked Read 6 Done Sent Done false
4086 1901 Acked Write 6 Done Sent Done false
4087 1902 Acked Write 6 Done Sent Done false
4088 1903 Acked Flush 0 Done Sent Done false
4089 STATES DS:0 DS:1 DS:2 TOTAL
4090 Sent 0 301 0 301
4091 Done 301 0 301 602
4092 Skipped 0 0 0 0
4093 Error 0 0 0 0
4094 Last Flush: Some(JobId(1903)) Some(JobId(1602)) Some(JobId(1903))
4095 Downstairs last five completed: 1602 1601 1600 1599 1598
4096 Upstairs last five completed: 1903 1902 1901 1900 1899
4097 CLIENT: Up:0 ds:301 act:2
40982025-02-28T06:01:53.422ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
4099 [8820] got start action
4100 [8820] I am going to restart
4101 Starting downstairs at port 8820
4102 Make output file at "/tmp/dsc/downstairs-8820.txt"
4103 [8820][1] reports Starting
4104 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1214)
4105 [8820][1] reports Running
41062025-02-28T06:01:54.424ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
41072025-02-28T06:01:55.426ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
41082025-02-28T06:01:56.428ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
4109 ----------------------------------------------------------------
4110 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4111 States: Active Offline Active
4112 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4113 1603 Acked Write 9 Done Sent Done false
4114 1604 Acked Write 10 Done Sent Done false
4115 1605 Acked Write 1 Done Sent Done false
4116 1606 Acked Write 6 Done Sent Done false
4117 1607 Acked Read 8 Done Sent Done false
4118 1608 Acked Read 6 Done Sent Done false
4119 1609 Acked Read 1 Done Sent Done false
4120 1610 Acked Write 6 Done Sent Done false
4121 1611 Acked Write 6 Done Sent Done false
4122 1612 Acked Read 9 Done Sent Done false
4123 1613 Acked Write 8 Done Sent Done false
4124 1614 Acked Write 5 Done Sent Done false
4125 1615 Acked Write 5 Done Sent Done false
4126 1616 Acked Write 8 Done Sent Done false
4127 1617 Acked Read 2 Done Sent Done false
4128 1618 Acked Flush 0 Done Sent Done false
4129 1619 Acked Flush 0 Done Sent Done false
4130 1620 Acked Write 5 Done Sent Done false
4131 1621 Acked Read 10 Done Sent Done false
4132 1622 Acked Read 10 Done Sent Done false
4133 1623 Acked Read 3 Done Sent Done false
4134 1624 Acked Read 9 Done Sent Done false
4135 1625 Acked Read 3 Done Sent Done false
4136 1626 Acked Read 10 Done Sent Done false
4137 1627 Acked Write 1 Done Sent Done false
4138 1628 Acked Read 1 Done Sent Done false
4139 1629 Acked Write 7 Done Sent Done false
4140 1630 Acked Read 3 Done Sent Done false
4141 1631 Acked Write 10 Done Sent Done false
4142 1632 Acked Write 10 Done Sent Done false
4143 1633 Acked Read 9 Done Sent Done false
4144 1634 Acked Read 8 Done Sent Done false
4145 1635 Acked Read 2 Done Sent Done false
4146 1636 Acked Flush 0 Done Sent Done false
4147 1637 Acked Write 7 Done Sent Done false
4148 1638 Acked Flush 0 Done Sent Done false
4149 1639 Acked Read 2 Done Sent Done false
4150 1640 Acked Read 3 Done Sent Done false
4151 1641 Acked Flush 0 Done Sent Done false
4152 1642 Acked Write 3 Done Sent Done false
4153 1643 Acked Read 3 Done Sent Done false
4154 1644 Acked Write 3 Done Sent Done false
4155 1645 Acked Write 7 Done Sent Done false
4156 1646 Acked Read 6 Done Sent Done false
4157 1647 Acked Flush 0 Done Sent Done false
4158 1648 Acked Read 3 Done Sent Done false
4159 1649 Acked Write 5 Done Sent Done false
4160 1650 Acked Write 3 Done Sent Done false
4161 1651 Acked Write 3 Done Sent Done false
4162 1652 Acked Read 3 Done Sent Done false
4163 1653 Acked Flush 0 Done Sent Done false
4164 1654 Acked Flush 0 Done Sent Done false
4165 1655 Acked Read 6 Done Sent Done false
4166 1656 Acked Write 8 Done Sent Done false
4167 1657 Acked Read 6 Done Sent Done false
4168 1658 Acked Write 4 Done Sent Done false
4169 1659 Acked Flush 0 Done Sent Done false
4170 1660 Acked Flush 0 Done Sent Done false
4171 1661 Acked Write 6 Done Sent Done false
4172 1662 Acked Read 8 Done Sent Done false
4173 1663 Acked Read 7 Done Sent Done false
4174 1664 Acked Write 9 Done Sent Done false
4175 1665 Acked Flush 0 Done Sent Done false
4176 1666 Acked Read 8 Done Sent Done false
4177 1667 Acked Write 10 Done Sent Done false
4178 1668 Acked Write 3 Done Sent Done false
4179 1669 Acked Write 9 Done Sent Done false
4180 1670 Acked Read 8 Done Sent Done false
4181 1671 Acked Write 10 Done Sent Done false
4182 1672 Acked Read 9 Done Sent Done false
4183 1673 Acked Write 1 Done Sent Done false
4184 1674 Acked Read 10 Done Sent Done false
4185 1675 Acked Write 3 Done Sent Done false
4186 1676 Acked Write 6 Done Sent Done false
4187 1677 Acked Read 4 Done Sent Done false
4188 1678 Acked Read 9 Done Sent Done false
4189 1679 Acked Read 2 Done Sent Done false
4190 1680 Acked Read 6 Done Sent Done false
4191 1681 Acked Flush 0 Done Sent Done false
4192 1682 Acked Read 6 Done Sent Done false
4193 1683 Acked Read 3 Done Sent Done false
4194 1684 Acked Write 6 Done Sent Done false
4195 1685 Acked Read 6 Done Sent Done false
4196 1686 Acked Write 4 Done Sent Done false
4197 1687 Acked Write 6 Done Sent Done false
4198 1688 Acked Write 2 Done Sent Done false
4199 1689 Acked Read 7 Done Sent Done false
4200 1690 Acked Read 6 Done Sent Done false
4201 1691 Acked Read 8 Done Sent Done false
4202 1692 Acked Write 8 Done Sent Done false
4203 1693 Acked Write 5 Done Sent Done false
4204 1694 Acked Write 2 Done Sent Done false
4205 1695 Acked Write 9 Done Sent Done false
4206 1696 Acked Write 2 Done Sent Done false
4207 1697 Acked Write 9 Done Sent Done false
4208 1698 Acked Read 2 Done Sent Done false
4209 1699 Acked Write 1 Done Sent Done false
4210 1700 Acked Write 2 Done Sent Done false
4211 1701 Acked Write 3 Done Sent Done false
4212 1702 Acked Read 2 Done Sent Done false
4213 1703 Acked Read 3 Done Sent Done false
4214 1704 Acked Write 9 Done Sent Done false
4215 1705 Acked Write 10 Done Sent Done false
4216 1706 Acked Read 10 Done Sent Done false
4217 1707 Acked Read 4 Done Sent Done false
4218 1708 Acked Write 3 Done Sent Done false
4219 1709 Acked Read 4 Done Sent Done false
4220 1710 Acked Read 10 Done Sent Done false
4221 1711 Acked Write 4 Done Sent Done false
4222 1712 Acked Write 5 Done Sent Done false
4223 1713 Acked Write 5 Done Sent Done false
4224 1714 Acked Write 4 Done Sent Done false
4225 1715 Acked Write 2 Done Sent Done false
4226 1716 Acked Flush 0 Done Sent Done false
4227 1717 Acked Flush 0 Done Sent Done false
4228 1718 Acked Write 10 Done Sent Done false
4229 1719 Acked Write 10 Done Sent Done false
4230 1720 Acked Flush 0 Done Sent Done false
4231 1721 Acked Write 9 Done Sent Done false
4232 1722 Acked Read 7 Done Sent Done false
4233 1723 Acked Read 6 Done Sent Done false
4234 1724 Acked Write 1 Done Sent Done false
4235 1725 Acked Write 2 Done Sent Done false
4236 1726 Acked Write 7 Done Sent Done false
4237 1727 Acked Read 1 Done Sent Done false
4238 1728 Acked Read 9 Done Sent Done false
4239 1729 Acked Read 6 Done Sent Done false
4240 1730 Acked Flush 0 Done Sent Done false
4241 1731 Acked Write 3 Done Sent Done false
4242 1732 Acked Read 9 Done Sent Done false
4243 1733 Acked Write 8 Done Sent Done false
4244 1734 Acked Flush 0 Done Sent Done false
4245 1735 Acked Write 5 Done Sent Done false
4246 1736 Acked Read 10 Done Sent Done false
4247 1737 Acked Write 6 Done Sent Done false
4248 1738 Acked Read 1 Done Sent Done false
4249 1739 Acked Read 6 Done Sent Done false
4250 1740 Acked Read 6 Done Sent Done false
4251 1741 Acked Read 10 Done Sent Done false
4252 1742 Acked Write 7 Done Sent Done false
4253 1743 Acked Write 10 Done Sent Done false
4254 1744 Acked Write 9 Done Sent Done false
4255 1745 Acked Flush 0 Done Sent Done false
4256 1746 Acked Read 6 Done Sent Done false
4257 1747 Acked Flush 0 Done Sent Done false
4258 1748 Acked Write 5 Done Sent Done false
4259 1749 Acked Read 7 Done Sent Done false
4260 1750 Acked Read 10 Done Sent Done false
4261 1751 Acked Read 7 Done Sent Done false
4262 1752 Acked Write 2 Done Sent Done false
4263 1753 Acked Write 10 Done Sent Done false
4264 1754 Acked Write 5 Done Sent Done false
4265 1755 Acked Read 3 Done Sent Done false
4266 1756 Acked Read 5 Done Sent Done false
4267 1757 Acked Flush 0 Done Sent Done false
4268 1758 Acked Write 8 Done Sent Done false
4269 1759 Acked Flush 0 Done Sent Done false
4270 1760 Acked Read 8 Done Sent Done false
4271 1761 Acked Read 6 Done Sent Done false
4272 1762 Acked Write 4 Done Sent Done false
4273 1763 Acked Read 1 Done Sent Done false
4274 1764 Acked Read 3 Done Sent Done false
4275 1765 Acked Write 2 Done Sent Done false
4276 1766 Acked Write 10 Done Sent Done false
4277 1767 Acked Write 10 Done Sent Done false
4278 1768 Acked Read 7 Done Sent Done false
4279 1769 Acked Read 2 Done Sent Done false
4280 1770 Acked Write 6 Done Sent Done false
4281 1771 Acked Flush 0 Done Sent Done false
4282 1772 Acked Read 8 Done Sent Done false
4283 1773 Acked Read 10 Done Sent Done false
4284 1774 Acked Write 4 Done Sent Done false
4285 1775 Acked Write 6 Done Sent Done false
4286 1776 Acked Flush 0 Done Sent Done false
4287 1777 Acked Flush 0 Done Sent Done false
4288 1778 Acked Read 8 Done Sent Done false
4289 1779 Acked Write 10 Done Sent Done false
4290 1780 Acked Read 5 Done Sent Done false
4291 1781 Acked Write 6 Done Sent Done false
4292 1782 Acked Read 1 Done Sent Done false
4293 1783 Acked Write 7 Done Sent Done false
4294 1784 Acked Read 10 Done Sent Done false
4295 1785 Acked Read 5 Done Sent Done false
4296 1786 Acked Read 6 Done Sent Done false
4297 1787 Acked Read 2 Done Sent Done false
4298 1788 Acked Read 7 Done Sent Done false
4299 1789 Acked Read 1 Done Sent Done false
4300 1790 Acked Read 5 Done Sent Done false
4301 1791 Acked Read 3 Done Sent Done false
4302 1792 Acked Write 7 Done Sent Done false
4303 1793 Acked Read 2 Done Sent Done false
4304 1794 Acked Write 9 Done Sent Done false
4305 1795 Acked Read 5 Done Sent Done false
4306 1796 Acked Read 7 Done Sent Done false
4307 1797 Acked Flush 0 Done Sent Done false
4308 1798 Acked Write 9 Done Sent Done false
4309 1799 Acked Write 1 Done Sent Done false
4310 1800 Acked Write 8 Done Sent Done false
4311 1801 Acked Flush 0 Done Sent Done false
4312 1802 Acked Read 7 Done Sent Done false
4313 1803 Acked Flush 0 Done Sent Done false
4314 1804 Acked Read 7 Done Sent Done false
4315 1805 Acked Write 7 Done Sent Done false
4316 1806 Acked Write 5 Done Sent Done false
4317 1807 Acked Read 3 Done Sent Done false
4318 1808 Acked Write 3 Done Sent Done false
4319 1809 Acked Flush 0 Done Sent Done false
4320 1810 Acked Write 5 Done Sent Done false
4321 1811 Acked Read 6 Done Sent Done false
4322 1812 Acked Read 9 Done Sent Done false
4323 1813 Acked Write 10 Done Sent Done false
4324 1814 Acked Flush 0 Done Sent Done false
4325 1815 Acked Write 5 Done Sent Done false
4326 1816 Acked Read 7 Done Sent Done false
4327 1817 Acked Write 2 Done Sent Done false
4328 1818 Acked Write 8 Done Sent Done false
4329 1819 Acked Read 5 Done Sent Done false
4330 1820 Acked Read 4 Done Sent Done false
4331 1821 Acked Read 5 Done Sent Done false
4332 1822 Acked Write 3 Done Sent Done false
4333 1823 Acked Read 6 Done Sent Done false
4334 1824 Acked Write 8 Done Sent Done false
4335 1825 Acked Read 1 Done Sent Done false
4336 1826 Acked Read 8 Done Sent Done false
4337 1827 Acked Read 4 Done Sent Done false
4338 1828 Acked Read 10 Done Sent Done false
4339 1829 Acked Read 7 Done Sent Done false
4340 1830 Acked Write 5 Done Sent Done false
4341 1831 Acked Write 1 Done Sent Done false
4342 1832 Acked Read 9 Done Sent Done false
4343 1833 Acked Write 8 Done Sent Done false
4344 1834 Acked Write 1 Done Sent Done false
4345 1835 Acked Read 1 Done Sent Done false
4346 1836 Acked Write 6 Done Sent Done false
4347 1837 Acked Read 2 Done Sent Done false
4348 1838 Acked Read 10 Done Sent Done false
4349 1839 Acked Read 1 Done Sent Done false
4350 1840 Acked Read 10 Done Sent Done false
4351 1841 Acked Read 6 Done Sent Done false
4352 1842 Acked Write 5 Done Sent Done false
4353 1843 Acked Write 5 Done Sent Done false
4354 1844 Acked Read 2 Done Sent Done false
4355 1845 Acked Write 1 Done Sent Done false
4356 1846 Acked Read 8 Done Sent Done false
4357 1847 Acked Read 2 Done Sent Done false
4358 1848 Acked Write 5 Done Sent Done false
4359 1849 Acked Write 3 Done Sent Done false
4360 1850 Acked Read 4 Done Sent Done false
4361 1851 Acked Write 7 Done Sent Done false
4362 1852 Acked Write 7 Done Sent Done false
4363 1853 Acked Read 3 Done Sent Done false
4364 1854 Acked Write 10 Done Sent Done false
4365 1855 Acked Write 7 Done Sent Done false
4366 1856 Acked Read 7 Done Sent Done false
4367 1857 Acked Read 2 Done Sent Done false
4368 1858 Acked Write 5 Done Sent Done false
4369 1859 Acked Write 2 Done Sent Done false
4370 1860 Acked Read 3 Done Sent Done false
4371 1861 Acked Read 2 Done Sent Done false
4372 1862 Acked Read 4 Done Sent Done false
4373 1863 Acked Write 9 Done Sent Done false
4374 1864 Acked Flush 0 Done Sent Done false
4375 1865 Acked Write 8 Done Sent Done false
4376 1866 Acked Flush 0 Done Sent Done false
4377 1867 Acked Read 8 Done Sent Done false
4378 1868 Acked Flush 0 Done Sent Done false
4379 1869 Acked Read 6 Done Sent Done false
4380 1870 Acked Read 7 Done Sent Done false
4381 1871 Acked Read 7 Done Sent Done false
4382 1872 Acked Write 6 Done Sent Done false
4383 1873 Acked Read 10 Done Sent Done false
4384 1874 Acked Write 6 Done Sent Done false
4385 1875 Acked Flush 0 Done Sent Done false
4386 1876 Acked Write 8 Done Sent Done false
4387 1877 Acked Write 8 Done Sent Done false
4388 1878 Acked Write 6 Done Sent Done false
4389 1879 Acked Write 9 Done Sent Done false
4390 1880 Acked Read 3 Done Sent Done false
4391 1881 Acked Read 4 Done Sent Done false
4392 1882 Acked Read 4 Done Sent Done false
4393 1883 Acked Read 7 Done Sent Done false
4394 1884 Acked Read 4 Done Sent Done false
4395 1885 Acked Read 9 Done Sent Done false
4396 1886 Acked Write 8 Done Sent Done false
4397 1887 Acked Write 3 Done Sent Done false
4398 1888 Acked Read 1 Done Sent Done false
4399 1889 Acked Read 2 Done Sent Done false
4400 1890 Acked Write 3 Done Sent Done false
4401 1891 Acked Read 1 Done Sent Done false
4402 1892 Acked Write 7 Done Sent Done false
4403 1893 Acked Read 10 Done Sent Done false
4404 1894 Acked Flush 0 Done Sent Done false
4405 1895 Acked Flush 0 Done Sent Done false
4406 1896 Acked Read 7 Done Sent Done false
4407 1897 Acked Write 7 Done Sent Done false
4408 1898 Acked Flush 0 Done Sent Done false
4409 1899 Acked Write 9 Done Sent Done false
4410 1900 Acked Read 6 Done Sent Done false
4411 1901 Acked Write 6 Done Sent Done false
4412 1902 Acked Write 6 Done Sent Done false
4413 1903 Acked Flush 0 Done Sent Done false
4414 STATES DS:0 DS:1 DS:2 TOTAL
4415 Sent 0 301 0 301
4416 Done 301 0 301 602
4417 Skipped 0 0 0 0
4418 Error 0 0 0 0
4419 Last Flush: Some(JobId(1903)) Some(JobId(1602)) Some(JobId(1903))
4420 Downstairs last five completed: 1602 1601 1600 1599 1598
4421 Upstairs last five completed: 1903 1902 1901 1900 1899
4422 CLIENT: Up:0 ds:301 act:2
44232025-02-28T06:01:57.430ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
44242025-02-28T06:01:58.394ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
44252025-02-28T06:01:58.395ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42781) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
44262025-02-28T06:01:58.396ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
44272025-02-28T06:01:58.396ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
44282025-02-28T06:01:58.396ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
44292025-02-28T06:01:58.396ZINFOcrucible: send last flush ID to this DS: Some(JobId(1602)) = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
44302025-02-28T06:01:58.396ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
44312025-02-28T06:01:58.396ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(1602)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
44322025-02-28T06:01:58.397ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(1602)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
4433 ----------------------------------------------------------------
4434 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4435 Downstairs last five completed: 1903 1902 1901 1900 1899
4436 Upstairs last five completed: 1903 1902 1901 1900 1899
4437 CLIENT: Up:0 ds:0 act:3
4438 Replay: All jobs finished
4439 Main task has work to do, go find it
4440 got dsc Stop(1)
4441 stop 1
4442 Feb 28 06:02:00.638 INFO request completed, latency_us: 364[8820] Got stop action so:false kr:false
4443 , response_code: 204, uri: /stop/cid/1, method: POST, req_id: 4cd98aba-6d45-460f-9782-52a99cc40e48, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
4444 Feb 28 06:02:00.638 INFO request completed, latency_us: 126, response_code: 200, uri: /state/cid/1, method: GET, req_id: c7e8827e-ef2d-43fa-b753-54738213e2d2, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
44452025-02-28T06:02:00.638ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
44462025-02-28T06:02:00.638ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
44472025-02-28T06:02:00.638ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
44482025-02-28T06:02:00.639ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
4449 [8820] Exited with: ExitStatus(unix_wait_status(9))
4450 [8820][1] reports Exit
4451 Feb 28 06:02:04.641 INFO request completed, latency_us: 338, response_code: 200, uri: /state/cid/1, method: GET, req_id: 291837d2-ee4c-4dc8-b126-d9922c4a647e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
4452 001/300 Write block 447 len 40960 data: 3 3 3 3 3 2 2 2 3 3
4453 002/300 Read block 1603 len 24576
4454 003/300 Write block 1383 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4455 004/300 Read block 1082 len 12288
4456 005/300 Read block 1936 len 12288
4457 006/300 Read block 1784 len 28672
4458 007/300 Read block 4869 len 28672
4459 008/300 Write block 991 len 16384 data: 3 2 2 2
4460 009/300 Flush
4461 010/300 Read block 758 len 8192
4462 011/300 Write block 1869 len 4096 data: 2
4463 012/300 Read block 4065 len 32768
4464 013/300 Read block 1447 len 24576
4465 014/300 Read block 3430 len 16384
4466 015/300 Read block 1564 len 28672
4467 016/300 Read block 4976 len 16384
4468 017/300 Read block 2985 len 32768
4469 018/300 Write block 291 len 36864 data: 2 2 2 2 2 2 2 2 2
4470 019/300 Read block 1434 len 40960
4471 020/300 Read block 4365 len 16384
4472 021/300 Flush
4473 022/300 Write block 191 len 16384 data: 2 2 2 2
4474 023/300 Write block 1512 len 12288 data: 2 2 2
4475 024/300 Read block 2101 len 16384
4476 025/300 Write block 2283 len 16384 data: 3 2 2 2
4477 026/300 Read block 3808 len 36864
4478 027/300 Read block 681 len 8192
4479 028/300 Write block 2234 len 12288 data: 5 3 3
4480 029/300 Read block 391 len 8192
4481 030/300 Read block 4387 len 40960
4482 031/300 Read block 3311 len 12288
4483 032/300 Write block 2186 len 28672 data: 2 2 2 2 3 3 3
4484 033/300 Read block 4975 len 4096
4485 034/300 Read block 4529 len 20480
4486 035/300 Read block 616 len 12288
4487 036/300 Read block 4876 len 28672
4488 037/300 Flush
4489 038/300 Write block 2521 len 20480 data: 4 4 4 4 4
4490 039/300 Read block 4406 len 12288
4491 040/300 Write block 4694 len 16384 data: 2 2 2 2
4492 041/300 Read block 1984 len 24576
4493 042/300 Write block 2319 len 16384 data: 4 3 3 3
4494 043/300 Write block 1398 len 20480 data: 2 3 3 3 4
4495 044/300 Read block 2593 len 24576
4496 045/300 Read block 2129 len 20480
4497 046/300 Write block 4897 len 24576 data: 3 3 3 3 2 2
4498 047/300 Write block 135 len 24576 data: 2 2 2 2 2 2
4499 048/300 Write block 675 len 40960 data: 3 2 2 3 3 3 3 2 2 2
4500 049/300 Write block 2445 len 12288 data: 2 2 2
4501 050/300 Flush
4502 051/300 Write block 3731 len 28672 data: 2 2 2 2 2 2 2
4503 052/300 Write block 4492 len 24576 data: 2 3 3 4 4 4
4504 053/300 Write block 3648 len 40960 data: 3 3 3 3 3 3 2 2 2 2
4505 054/300 Write block 897 len 28672 data: 2 2 2 2 2 3 2
4506 055/300 Write block 4898 len 32768 data: 4 4 4 3 3 2 2 2
4507 056/300 Read block 830 len 8192
4508 057/300 Write block 3122 len 16384 data: 2 2 2 2
4509 058/300 Read block 2651 len 24576
4510 059/300 Read block 2561 len 16384
4511 060/300 Read block 1019 len 28672
4512 061/300 Write block 844 len 8192 data: 3 3
4513 062/300 Read block 3508 len 16384
4514 063/300 Write block 679 len 28672 data: 4 4 4 3 3 3 2
4515 064/300 Read block 523 len 40960
4516 065/300 Write block 2083 len 28672 data: 2 2 2 2 2 2 2
4517 066/300 Write block 3441 len 32768 data: 2 2 2 2 2 2 2 2
4518 067/300 Write block 537 len 16384 data: 2 2 2 2
4519 068/300 Read block 4415 len 12288
4520 069/300 Read block 4718 len 32768
4521 070/300 Read block 2219 len 28672
4522 071/300 Write block 3475 len 4096 data: 2
4523 072/300 Write block 679 len 4096 data: 5
4524 073/300 Write block 2824 len 12288 data: 2 2 2
4525 074/300 Read block 2 len 24576
4526 075/300 Write block 2212 len 32768 data: 3 3 3 3 3 3 3 2
4527 076/300 Write block 446 len 36864 data: 3 4 4 4 4 4 3 3 3
4528 077/300 Write block 4336 len 16384 data: 2 2 2 2
4529 078/300 Read block 3148 len 40960
4530 079/300 Write block 4333 len 32768 data: 2 2 2 3 3 3 3 2
4531 080/300 Write block 319 len 20480 data: 2 2 2 2 2
4532 081/300 Flush
4533 082/300 Flush
4534 083/300 Read block 1761 len 40960
4535 084/300 Read block 3469 len 24576
4536 085/300 Flush
4537 086/300 Read block 978 len 36864
4538 087/300 Read block 15 len 40960
4539 088/300 Read block 234 len 40960
4540 089/300 Read block 1175 len 20480
4541 090/300 Write block 38 len 20480 data: 2 2 2 3 3
4542 091/300 Flush
4543 092/300 Read block 4382 len 20480
4544 093/300 Read block 2485 len 12288
4545 094/300 Read block 1454 len 36864
4546 095/300 Write block 2724 len 40960 data: 2 2 2 2 2 2 3 2 2 2
4547 096/300 Flush
4548 097/300 Read block 2751 len 40960
4549 098/300 Flush
4550 099/300 Flush
4551 100/300 Write block 577 len 28672 data: 2 2 2 3 3 3 3
4552 101/300 Read block 116 len 16384
4553 102/300 Read block 1186 len 24576
4554 103/300 Flush
4555 104/300 Read block 3322 len 32768
4556 105/300 Write block 133 len 20480 data: 2 2 3 3 3
4557 106/300 Read block 4358 len 36864
4558 107/300 Read block 2042 len 28672
4559 108/300 Write block 4141 len 40960 data: 2 2 2 2 3 3 3 3 3 3
4560 109/300 Write block 1864 len 40960 data: 2 2 2 2 2 3 2 2 2 2
4561 110/300 Write block 4499 len 36864 data: 4 4 4 2 2 2 2 2 2
4562 111/300 Write block 2701 len 32768 data: 2 2 2 2 2 2 2 2
4563 112/300 Write block 4498 len 36864 data: 4 5 5 5 3 3 3 3 3
4564 113/300 Read block 4105 len 12288
4565 114/300 Write block 2653 len 28672 data: 2 2 2 2 2 2 2
4566 115/300 Read block 2783 len 4096
4567 116/300 Read block 3112 len 40960
4568 117/300 Read block 2095 len 28672
4569 118/300 Read block 3226 len 28672
4570 119/300 Read block 2882 len 16384
4571 120/300 Write block 3296 len 28672 data: 2 3 3 3 2 2 3
4572 121/300 Read block 235 len 40960
4573 122/300 Write block 4290 len 12288 data: 2 2 2
4574 123/300 Write block 1160 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4575 124/300 Flush
4576 125/300 Read block 2059 len 8192
4577 126/300 Read block 266 len 24576
4578 127/300 Write block 283 len 8192 data: 3 3
4579 128/300 Flush
4580 129/300 Write block 1632 len 24576 data: 2 2 2 2 2 2
4581 130/300 Write block 2273 len 24576 data: 3 3 3 3 3 3
4582 131/300 Read block 1590 len 16384
4583 132/300 Read block 3795 len 32768
4584 133/300 Flush
4585 134/300 Read block 1776 len 32768
4586 135/300 Read block 4247 len 4096
4587 136/300 Write block 2209 len 40960 data: 3 3 3 4 4 4 4 4 4 4
4588 137/300 Read block 1906 len 28672
4589 138/300 Write block 2594 len 4096 data: 2
4590 139/300 Flush
4591 140/300 Write block 1891 len 24576 data: 3 3 3 4 4 3
4592 141/300 Read block 1156 len 24576
4593 142/300 Write block 2656 len 20480 data: 3 3 3 3 2
4594 143/300 Read block 2671 len 32768
4595 144/300 Write block 4054 len 28672 data: 2 2 2 2 2 2 2
4596 145/300 Read block 3396 len 8192
4597 146/300 Flush
4598 147/300 Write block 3101 len 36864 data: 3 3 3 3 3 2 2 3 3
4599 148/300 Read block 4071 len 40960
4600 149/300 Write block 3882 len 28672 data: 2 3 3 3 3 2 2
4601 150/300 Write block 4184 len 16384 data: 3 3 3 3
4602 151/300 Flush
4603 152/300 Flush
4604 153/300 Read block 4970 len 24576
4605 154/300 Flush
4606 155/300 Read block 209 len 8192
4607 156/300 Read block 3482 len 28672
4608 157/300 Read block 619 len 36864
4609 158/300 Flush
4610 159/300 Read block 3 len 20480
4611 160/300 Read block 1667 len 8192
4612 161/300 Write block 1838 len 36864 data: 2 2 2 2 2 2 2 2 2
4613 162/300 Read block 2191 len 8192
4614 163/300 Write block 2137 len 4096 data: 2
4615 164/300 Read block 1068 len 20480
4616 165/300 Write block 3897 len 20480 data: 2 2 2 2 2
4617 166/300 Read block 913 len 20480
4618 167/300 Write block 2985 len 32768 data: 2 2 2 2 2 3 3 3
4619 168/300 Read block 765 len 28672
4620 169/300 Read block 1825 len 8192
4621 170/300 Write block 4830 len 8192 data: 3 4
4622 171/300 Flush
4623 172/300 Write block 1656 len 20480 data: 2 2 2 2 2
4624 173/300 Read block 1912 len 20480
4625 174/300 Read block 325 len 8192
4626 175/300 Read block 2686 len 8192
4627 176/300 Write block 3685 len 8192 data: 2 2
4628 177/300 Flush
4629 178/300 Read block 4569 len 32768
4630 179/300 Read block 1226 len 4096
4631 180/300 Read block 988 len 36864
4632 181/300 Write block 2236 len 12288 data: 4 3 3
4633 182/300 Read block 4677 len 36864
4634 183/300 Write block 2705 len 36864 data: 3 3 3 3 2 2 2 2 3
4635 184/300 Write block 3376 len 4096 data: 2
4636 185/300 Write block 149 len 24576 data: 4 3 3 3 3 3
4637 186/300 Write block 21 len 20480 data: 2 2 2 2 2
4638 187/300 Read block 404 len 24576
4639 188/300 Read block 2412 len 12288
4640 189/300 Flush
4641 190/300 Write block 133 len 28672 data: 3 3 4 4 4 3 3
4642 191/300 Flush
4643 192/300 Read block 1980 len 4096
4644 193/300 Write block 1901 len 4096 data: 2
4645 194/300 Read block 4723 len 12288
4646 195/300 Read block 2175 len 28672
4647 196/300 Write block 3434 len 24576 data: 2 2 2 2 2 2
4648 197/300 Flush
4649 198/300 Write block 2223 len 28672 data: 2 2 2 2 3 3 3
4650 199/300 Read block 1174 len 24576
4651 200/300 Write block 2709 len 36864 data: 3 3 3 3 4 3 3 3 3
4652 201/300 Flush
4653 202/300 Flush
4654 203/300 Read block 3275 len 20480
4655 204/300 Write block 3435 len 16384 data: 3 3 3 3
4656 205/300 Read block 1149 len 8192
4657 206/300 Flush
4658 207/300 Flush
4659 208/300 Read block 1580 len 24576
4660 209/300 Write block 3938 len 16384 data: 2 3 3 3
4661 210/300 Write block 752 len 12288 data: 3 3 3
4662 211/300 Write block 2374 len 12288 data: 2 3 4
4663 212/300 Read block 1143 len 4096
4664 213/300 Write block 4878 len 40960 data: 2 2 2 2 2 2 2 2 2 3
4665 214/300 Write block 730 len 36864 data: 2 2 2 2 2 2 3 2 2
4666 215/300 Write block 823 len 4096 data: 2
4667 216/300 Read block 4663 len 24576
4668 217/300 Read block 3646 len 4096
4669 218/300 Write block 2789 len 4096 data: 3
4670 219/300 Read block 4795 len 20480
4671 220/300 Read block 2870 len 16384
4672 221/300 Read block 3023 len 12288
4673 222/300 Write block 1739 len 12288 data: 2 2 2
4674 223/300 Write block 3479 len 8192 data: 2 3
4675 224/300 Write block 3497 len 28672 data: 2 2 2 2 2 2 2
4676 225/300 Read block 4236 len 4096
4677 226/300 Write block 2779 len 36864 data: 2 2 2 2 3 3 3 3 3
4678 227/300 Read block 1366 len 8192
4679 228/300 Write block 4978 len 8192 data: 2 2
4680 229/300 Flush
4681 230/300 Write block 2512 len 36864 data: 4 4 4 3 4 4 4 4 4
4682 231/300 Read block 4288 len 20480
4683 232/300 Write block 3018 len 24576 data: 2 2 2 2 2 2
4684 233/300 Read block 3107 len 28672
4685 234/300 Read block 3822 len 16384
4686 235/300 Read block 2185 len 16384
4687 236/300 Flush
4688 237/300 Flush
4689 238/300 Flush
4690 239/300 Read block 4699 len 4096
4691 240/300 Write block 3527 len 8192 data: 3 3
4692 241/300 Write block 2678 len 32768 data: 2 2 2 2 2 2 2 2
4693 242/300 Write block 437 len 36864 data: 3 4 4 4 3 4 4 4 4
4694 243/300 Write block 4070 len 28672 data: 3 3 3 2 2 2 2
4695 244/300 Read block 70 len 40960
4696 245/300 Read block 2355 len 20480
4697 246/300 Write block 443 len 40960 data: 5 5 5 4 5 5 5 5 5 4
4698 247/300 Write block 911 len 12288 data: 3 3 3
4699 248/300 Read block 844 len 16384
4700 249/300 Read block 2231 len 4096
4701 250/300 Read block 1921 len 32768
4702 251/300 Write block 3738 len 32768 data: 2 2 2 3 3 3 3 3
4703 252/300 Flush
4704 253/300 Read block 1580 len 36864
4705 254/300 Read block 2153 len 32768
4706 255/300 Write block 2619 len 4096 data: 2
4707 256/300 Flush
4708 257/300 Read block 1679 len 8192
4709 258/300 Write block 336 len 32768 data: 2 3 3 2 2 3 3 3
4710 259/300 Read block 798 len 28672
4711 260/300 Write block 1080 len 8192 data: 3 3
4712 261/300 Read block 162 len 24576
4713 262/300 Write block 4925 len 8192 data: 2 2
4714 263/300 Read block 3931 len 20480
4715 264/300 Read block 4491 len 32768
4716 265/300 Read block 3211 len 16384
4717 266/300 Read block 4647 len 28672
4718 267/300 Read block 2932 len 28672
4719 268/300 Write block 3073 len 28672 data: 3 3 2 2 2 2 2
4720 269/300 Write block 4436 len 12288 data: 2 2 2
4721 270/300 Write block 1696 len 32768 data: 4 4 3 3 3 3 2 2
4722 271/300 Read block 335 len 24576
4723 272/300 Read block 1844 len 36864
4724 273/300 Read block 3055 len 12288
4725 274/300 Read block 973 len 12288
4726 275/300 Flush
4727 276/300 Read block 31 len 8192
4728 277/300 Flush
4729 278/300 Write block 3400 len 12288 data: 2 2 2
4730 279/300 Write block 1332 len 24576 data: 2 2 3 3 3 3
4731 280/300 Read block 375 len 36864
4732 281/300 Write block 1538 len 24576 data: 3 2 2 2 2 2
4733 282/300 Read block 3192 len 28672
4734 283/300 Read block 2084 len 4096
4735 284/300 Flush
4736 285/300 Write block 452 len 24576 data: 5 4 4 4 4 2
4737 286/300 Write block 2947 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4738 287/300 Read block 3466 len 4096
4739 288/300 Write block 237 len 12288 data: 4 4 2
4740 289/300 Write block 4165 len 16384 data: 2 2 2 2
4741 290/300 Read block 243 len 16384
4742 291/300 Read block 739 len 8192
4743 292/300 Read block 309 len 16384
4744 293/300 Write block 3934 len 36864 data: 2 2 2 2 3 4 4 4 3
4745 294/300 Read block 1068 len 20480
4746 295/300 Write block 2307 len 32768 data: 2 2 2 2 2 2 2 2
4747 296/300 Read block 3529 len 12288
4748 297/300 Write block 3067 len 4096 data: 2
4749 298/300 Read block 1163 len 24576
4750 299/300 Flush
4751 300/300 Read block 3138 len 40960
4752 301/300 Read block 4645 len 8192
4753 Main task has work to do, go find it
4754 got dsc Start(1)
4755 start 1
4756 Feb 28 06:02:04.876 INFO request completed, latency_us: 182, response_code: 204, uri: /start/cid/1, method: POST, req_id: 94672950-8000-408d-b70c-2c271212ce6d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
4757 Replay: started 1, returned:Ok(())
4758 ----------------------------------------------------------------
4759 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4760 States: Active Offline Active
4761 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4762 1904 Acked Write 10 Done Sent Done false
4763 1905 Acked Read 6 Done Sent Done false
4764 1906 Acked Write 10 Done Sent Done false
4765 1907 Acked Read 3 Done Sent Done false
4766 1908 Acked Read 3 Done Sent Done false
4767 1909 Acked Read 7 Done Sent Done false
4768 1910 Acked Read 7 Done Sent Done false
4769 1911 Acked Write 4 Done Sent Done false
4770 1912 Acked Flush 0 Done Sent Done false
4771 1913 Acked Read 2 Done Sent Done false
4772 1914 Acked Write 1 Done Sent Done false
4773 1915 Acked Read 8 Done Sent Done false
4774 1916 Acked Read 6 Done Sent Done false
4775 1917 Acked Read 4 Done Sent Done false
4776 1918 Acked Read 7 Done Sent Done false
4777 1919 Acked Read 4 Done Sent Done false
4778 1920 Acked Read 8 Done Sent Done false
4779 1921 Acked Write 9 Done Sent Done false
4780 1922 Acked Read 10 Done Sent Done false
4781 1923 Acked Read 4 Done Sent Done false
4782 1924 Acked Flush 0 Done Sent Done false
4783 1925 Acked Write 4 Done Sent Done false
4784 1926 Acked Write 3 Done Sent Done false
4785 1927 Acked Read 4 Done Sent Done false
4786 1928 Acked Write 4 Done Sent Done false
4787 1929 Acked Read 9 Done Sent Done false
4788 1930 Acked Read 2 Done Sent Done false
4789 1931 Acked Write 3 Done Sent Done false
4790 1932 Acked Read 2 Done Sent Done false
4791 1933 Acked Read 10 Done Sent Done false
4792 1934 Acked Read 3 Done Sent Done false
4793 1935 Acked Write 7 Done Sent Done false
4794 1936 Acked Read 1 Done Sent Done false
4795 1937 Acked Read 5 Done Sent Done false
4796 1938 Acked Read 3 Done Sent Done false
4797 1939 Acked Read 7 Done Sent Done false
4798 1940 Acked Flush 0 Done Sent Done false
4799 1941 Acked Write 5 Done Sent Done false
4800 1942 Acked Read 3 Done Sent Done false
4801 1943 Acked Write 4 Done Sent Done false
4802 1944 Acked Read 6 Done Sent Done false
4803 1945 Acked Write 4 Done Sent Done false
4804 1946 Acked Write 5 Done Sent Done false
4805 1947 Acked Read 6 Done Sent Done false
4806 1948 Acked Read 5 Done Sent Done false
4807 1949 Acked Write 6 Done Sent Done false
4808 1950 Acked Write 6 Done Sent Done false
4809 1951 Acked Write 10 Done Sent Done false
4810 1952 Acked Write 3 Done Sent Done false
4811 1953 Acked Flush 0 Done Sent Done false
4812 1954 Acked Write 7 Done Sent Done false
4813 1955 Acked Write 6 Done Sent Done false
4814 1956 Acked Write 10 Done Sent Done false
4815 1957 Acked Write 7 Done Sent Done false
4816 1958 Acked Write 8 Done Sent Done false
4817 1959 Acked Read 2 Done Sent Done false
4818 1960 Acked Write 4 Done Sent Done false
4819 1961 Acked Read 6 Done Sent Done false
4820 1962 Acked Read 4 Done Sent Done false
4821 1963 Acked Read 7 Done Sent Done false
4822 1964 Acked Write 2 Done Sent Done false
4823 1965 Acked Read 4 Done Sent Done false
4824 1966 Acked Write 7 Done Sent Done false
4825 1967 Acked Read 10 Done Sent Done false
4826 1968 Acked Write 7 Done Sent Done false
4827 1969 Acked Write 8 Done Sent Done false
4828 1970 Acked Write 4 Done Sent Done false
4829 1971 Acked Read 3 Done Sent Done false
4830 1972 Acked Read 8 Done Sent Done false
4831 1973 Acked Read 7 Done Sent Done false
4832 1974 Acked Write 1 Done Sent Done false
4833 1975 Acked Write 1 Done Sent Done false
4834 1976 Acked Write 3 Done Sent Done false
4835 1977 Acked Read 6 Done Sent Done false
4836 1978 Acked Write 8 Done Sent Done false
4837 1979 Acked Write 9 Done Sent Done false
4838 1980 Acked Write 4 Done Sent Done false
4839 1981 Acked Read 10 Done Sent Done false
4840 1982 Acked Write 8 Done Sent Done false
4841 1983 Acked Write 5 Done Sent Done false
4842 1984 Acked Flush 0 Done Sent Done false
4843 1985 Acked Flush 0 Done Sent Done false
4844 1986 Acked Read 10 Done Sent Done false
4845 1987 Acked Read 6 Done Sent Done false
4846 1988 Acked Flush 0 Done Sent Done false
4847 1989 Acked Read 9 Done Sent Done false
4848 1990 Acked Read 10 Done Sent Done false
4849 1991 Acked Read 10 Done Sent Done false
4850 1992 Acked Read 5 Done Sent Done false
4851 1993 Acked Write 5 Done Sent Done false
4852 1994 Acked Flush 0 Done Sent Done false
4853 1995 Acked Read 5 Done Sent Done false
4854 1996 Acked Read 3 Done Sent Done false
4855 1997 Acked Read 9 Done Sent Done false
4856 1998 Acked Write 10 Done Sent Done false
4857 1999 Acked Flush 0 Done Sent Done false
4858 2000 Acked Read 10 Done Sent Done false
4859 2001 Acked Flush 0 Done Sent Done false
4860 2002 Acked Flush 0 Done Sent Done false
4861 2003 Acked Write 7 Done Sent Done false
4862 2004 Acked Read 4 Done Sent Done false
4863 2005 Acked Read 6 Done Sent Done false
4864 2006 Acked Flush 0 Done Sent Done false
4865 2007 Acked Read 8 Done Sent Done false
4866 2008 Acked Write 5 Done Sent Done false
4867 2009 Acked Read 9 Done Sent Done false
4868 2010 Acked Read 7 Done Sent Done false
4869 2011 Acked Write 10 Done Sent Done false
4870 2012 Acked Write 10 Done Sent Done false
4871 2013 Acked Write 9 Done Sent Done false
4872 2014 Acked Write 8 Done Sent Done false
4873 2015 Acked Write 9 Done Sent Done false
4874 2016 Acked Read 3 Done Sent Done false
4875 2017 Acked Write 7 Done Sent Done false
4876 2018 Acked Read 1 Done Sent Done false
4877 2019 Acked Read 10 Done Sent Done false
4878 2020 Acked Read 7 Done Sent Done false
4879 2021 Acked Read 7 Done Sent Done false
4880 2022 Acked Read 4 Done Sent Done false
4881 2023 Acked Write 7 Done Sent Done false
4882 2024 Acked Read 10 Done Sent Done false
4883 2025 Acked Write 3 Done Sent Done false
4884 2026 Acked Write 10 Done Sent Done false
4885 2027 Acked Flush 0 Done Sent Done false
4886 2028 Acked Read 2 Done Sent Done false
4887 2029 Acked Read 6 Done Sent Done false
4888 2030 Acked Write 2 Done Sent Done false
4889 2031 Acked Flush 0 Done Sent Done false
4890 2032 Acked Write 6 Done Sent Done false
4891 2033 Acked Write 6 Done Sent Done false
4892 2034 Acked Read 4 Done Sent Done false
4893 2035 Acked Read 8 Done Sent Done false
4894 2036 Acked Flush 0 Done Sent Done false
4895 2037 Acked Read 8 Done Sent Done false
4896 2038 Acked Read 1 Done Sent Done false
4897 2039 Acked Write 10 Done Sent Done false
4898 2040 Acked Read 7 Done Sent Done false
4899 2041 Acked Write 1 Done Sent Done false
4900 2042 Acked Flush 0 Done Sent Done false
4901 2043 Acked Write 6 Done Sent Done false
4902 2044 Acked Read 6 Done Sent Done false
4903 2045 Acked Write 5 Done Sent Done false
4904 2046 Acked Read 8 Done Sent Done false
4905 2047 Acked Write 7 Done Sent Done false
4906 2048 Acked Read 2 Done Sent Done false
4907 2049 Acked Flush 0 Done Sent Done false
4908 2050 Acked Write 9 Done Sent Done false
4909 2051 Acked Read 10 Done Sent Done false
4910 2052 Acked Write 7 Done Sent Done false
4911 2053 Acked Write 4 Done Sent Done false
4912 2054 Acked Flush 0 Done Sent Done false
4913 2055 Acked Flush 0 Done Sent Done false
4914 2056 Acked Read 6 Done Sent Done false
4915 2057 Acked Flush 0 Done Sent Done false
4916 2058 Acked Read 2 Done Sent Done false
4917 2059 Acked Read 7 Done Sent Done false
4918 2060 Acked Read 9 Done Sent Done false
4919 2061 Acked Flush 0 Done Sent Done false
4920 2062 Acked Read 5 Done Sent Done false
4921 2063 Acked Read 2 Done Sent Done false
4922 2064 Acked Write 9 Done Sent Done false
4923 2065 Acked Read 2 Done Sent Done false
4924 2066 Acked Write 1 Done Sent Done false
4925 2067 Acked Read 5 Done Sent Done false
4926 2068 Acked Write 5 Done Sent Done false
4927 2069 Acked Read 5 Done Sent Done false
4928 2070 Acked Write 8 Done Sent Done false
4929 2071 Acked Read 7 Done Sent Done false
4930 2072 Acked Read 2 Done Sent Done false
4931 2073 Acked Write 2 Done Sent Done false
4932 2074 Acked Flush 0 Done Sent Done false
4933 2075 Acked Write 5 Done Sent Done false
4934 2076 Acked Read 5 Done Sent Done false
4935 2077 Acked Read 2 Done Sent Done false
4936 2078 Acked Read 2 Done Sent Done false
4937 2079 Acked Write 2 Done Sent Done false
4938 2080 Acked Flush 0 Done Sent Done false
4939 2081 Acked Read 8 Done Sent Done false
4940 2082 Acked Read 1 Done Sent Done false
4941 2083 Acked Read 9 Done Sent Done false
4942 2084 Acked Write 3 Done Sent Done false
4943 2085 Acked Read 9 Done Sent Done false
4944 2086 Acked Write 9 Done Sent Done false
4945 2087 Acked Write 1 Done Sent Done false
4946 2088 Acked Write 6 Done Sent Done false
4947 2089 Acked Write 5 Done Sent Done false
4948 2090 Acked Read 6 Done Sent Done false
4949 2091 Acked Read 3 Done Sent Done false
4950 2092 Acked Flush 0 Done Sent Done false
4951 2093 Acked Write 7 Done Sent Done false
4952 2094 Acked Flush 0 Done Sent Done false
4953 2095 Acked Read 1 Done Sent Done false
4954 2096 Acked Write 1 Done Sent Done false
4955 2097 Acked Read 3 Done Sent Done false
4956 2098 Acked Read 7 Done Sent Done false
4957 2099 Acked Write 6 Done Sent Done false
4958 2100 Acked Flush 0 Done Sent Done false
4959 2101 Acked Write 7 Done Sent Done false
4960 2102 Acked Read 6 Done Sent Done false
4961 2103 Acked Write 9 Done Sent Done false
4962 2104 Acked Flush 0 Done Sent Done false
4963 2105 Acked Flush 0 Done Sent Done false
4964 2106 Acked Read 5 Done Sent Done false
4965 2107 Acked Write 4 Done Sent Done false
4966 2108 Acked Read 2 Done Sent Done false
4967 2109 Acked Flush 0 Done Sent Done false
4968 2110 Acked Flush 0 Done Sent Done false
4969 2111 Acked Read 6 Done Sent Done false
4970 2112 Acked Write 4 Done Sent Done false
4971 2113 Acked Write 3 Done Sent Done false
4972 2114 Acked Write 3 Done Sent Done false
4973 2115 Acked Read 1 Done Sent Done false
4974 2116 Acked Write 10 Done Sent Done false
4975 2117 Acked Write 9 Done Sent Done false
4976 2118 Acked Write 1 Done Sent Done false
4977 2119 Acked Read 6 Done Sent Done false
4978 2120 Acked Read 1 Done Sent Done false
4979 2121 Acked Write 1 Done Sent Done false
4980 2122 Acked Read 5 Done Sent Done false
4981 2123 Acked Read 4 Done Sent Done false
4982 2124 Acked Read 3 Done Sent Done false
4983 2125 Acked Write 3 Done Sent Done false
4984 2126 Acked Write 2 Done Sent Done false
4985 2127 Acked Write 7 Done Sent Done false
4986 2128 Acked Read 1 Done Sent Done false
4987 2129 Acked Write 9 Done Sent Done false
4988 2130 Acked Read 2 Done Sent Done false
4989 2131 Acked Write 2 Done Sent Done false
4990 2132 Acked Flush 0 Done Sent Done false
4991 2133 Acked Write 9 Done Sent Done false
4992 2134 Acked Read 5 Done Sent Done false
4993 2135 Acked Write 6 Done Sent Done false
4994 2136 Acked Read 7 Done Sent Done false
4995 2137 Acked Read 4 Done Sent Done false
4996 2138 Acked Read 4 Done Sent Done false
4997 2139 Acked Flush 0 Done Sent Done false
4998 2140 Acked Flush 0 Done Sent Done false
4999 2141 Acked Flush 0 Done Sent Done false
5000 2142 Acked Read 1 Done Sent Done false
5001 2143 Acked Write 2 Done Sent Done false
5002 2144 Acked Write 8 Done Sent Done false
5003 2145 Acked Write 9 Done Sent Done false
5004 2146 Acked Write 7 Done Sent Done false
5005 2147 Acked Read 10 Done Sent Done false
5006 2148 Acked Read 5 Done Sent Done false
5007 2149 Acked Write 10 Done Sent Done false
5008 2150 Acked Write 3 Done Sent Done false
5009 2151 Acked Read 4 Done Sent Done false
5010 2152 Acked Read 1 Done Sent Done false
5011 2153 Acked Read 8 Done Sent Done false
5012 2154 Acked Write 8 Done Sent Done false
5013 2155 Acked Flush 0 Done Sent Done false
5014 2156 Acked Read 9 Done Sent Done false
5015 2157 Acked Read 8 Done Sent Done false
5016 2158 Acked Write 1 Done Sent Done false
5017 2159 Acked Flush 0 Done Sent Done false
5018 2160 Acked Read 2 Done Sent Done false
5019 2161 Acked Write 8 Done Sent Done false
5020 2162 Acked Read 7 Done Sent Done false
5021 2163 Acked Write 2 Done Sent Done false
5022 2164 Acked Read 6 Done Sent Done false
5023 2165 Acked Write 2 Done Sent Done false
5024 2166 Acked Read 5 Done Sent Done false
5025 2167 Acked Read 8 Done Sent Done false
5026 2168 Acked Read 4 Done Sent Done false
5027 2169 Acked Read 7 Done Sent Done false
5028 2170 Acked Read 7 Done Sent Done false
5029 2171 Acked Write 7 Done Sent Done false
5030 2172 Acked Write 3 Done Sent Done false
5031 2173 Acked Write 8 Done Sent Done false
5032 2174 Acked Read 6 Done Sent Done false
5033 2175 Acked Read 9 Done Sent Done false
5034 2176 Acked Read 3 Done Sent Done false
5035 2177 Acked Read 3 Done Sent Done false
5036 2178 Acked Flush 0 Done Sent Done false
5037 2179 Acked Read 2 Done Sent Done false
5038 2180 Acked Flush 0 Done Sent Done false
5039 2181 Acked Write 3 Done Sent Done false
5040 2182 Acked Write 6 Done Sent Done false
5041 2183 Acked Read 9 Done Sent Done false
5042 2184 Acked Write 6 Done Sent Done false
5043 2185 Acked Read 7 Done Sent Done false
5044 2186 Acked Read 1 Done Sent Done false
5045 2187 Acked Flush 0 Done Sent Done false
5046 2188 Acked Write 6 Done Sent Done false
5047 2189 Acked Write 10 Done Sent Done false
5048 2190 Acked Read 1 Done Sent Done false
5049 2191 Acked Write 3 Done Sent Done false
5050 2192 Acked Write 4 Done Sent Done false
5051 2193 Acked Read 4 Done Sent Done false
5052 2194 Acked Read 2 Done Sent Done false
5053 2195 Acked Read 4 Done Sent Done false
5054 2196 Acked Write 9 Done Sent Done false
5055 2197 Acked Read 5 Done Sent Done false
5056 2198 Acked Write 8 Done Sent Done false
5057 2199 Acked Read 3 Done Sent Done false
5058 2200 Acked Write 1 Done Sent Done false
5059 2201 Acked Read 6 Done Sent Done false
5060 2202 Acked Flush 0 Done Sent Done false
5061 2203 Acked Read 10 Done Sent Done false
5062 2204 Acked Read 2 Done Sent Done false
5063 STATES DS:0 DS:1 DS:2 TOTAL
5064 Sent 0 301 0 301
5065 Done 301 0 301 602
5066 Skipped 0 0 0 0
5067 Error 0 0 0 0
5068 Last Flush: Some(JobId(2202)) Some(JobId(1903)) Some(JobId(2202))
5069 Downstairs last five completed: 1903 1902 1901 1900 1899
5070 Upstairs last five completed: 2204 2203 2202 2201 2200
5071 CLIENT: Up:0 ds:301 act:2
50722025-02-28T06:02:05.443ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
50732025-02-28T06:02:06.443ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
5074 [8820] got start action
5075 [8820] I am going to restart
5076 Starting downstairs at port 8820
5077 Make output file at "/tmp/dsc/downstairs-8820.txt"
5078 [8820][1] reports Starting
5079 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1215)
5080 [8820][1] reports Running
50812025-02-28T06:02:07.445ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
50822025-02-28T06:02:08.445ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
5083 ----------------------------------------------------------------
5084 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5085 States: Active Offline Active
5086 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5087 1904 Acked Write 10 Done Sent Done false
5088 1905 Acked Read 6 Done Sent Done false
5089 1906 Acked Write 10 Done Sent Done false
5090 1907 Acked Read 3 Done Sent Done false
5091 1908 Acked Read 3 Done Sent Done false
5092 1909 Acked Read 7 Done Sent Done false
5093 1910 Acked Read 7 Done Sent Done false
5094 1911 Acked Write 4 Done Sent Done false
5095 1912 Acked Flush 0 Done Sent Done false
5096 1913 Acked Read 2 Done Sent Done false
5097 1914 Acked Write 1 Done Sent Done false
5098 1915 Acked Read 8 Done Sent Done false
5099 1916 Acked Read 6 Done Sent Done false
5100 1917 Acked Read 4 Done Sent Done false
5101 1918 Acked Read 7 Done Sent Done false
5102 1919 Acked Read 4 Done Sent Done false
5103 1920 Acked Read 8 Done Sent Done false
5104 1921 Acked Write 9 Done Sent Done false
5105 1922 Acked Read 10 Done Sent Done false
5106 1923 Acked Read 4 Done Sent Done false
5107 1924 Acked Flush 0 Done Sent Done false
5108 1925 Acked Write 4 Done Sent Done false
5109 1926 Acked Write 3 Done Sent Done false
5110 1927 Acked Read 4 Done Sent Done false
5111 1928 Acked Write 4 Done Sent Done false
5112 1929 Acked Read 9 Done Sent Done false
5113 1930 Acked Read 2 Done Sent Done false
5114 1931 Acked Write 3 Done Sent Done false
5115 1932 Acked Read 2 Done Sent Done false
5116 1933 Acked Read 10 Done Sent Done false
5117 1934 Acked Read 3 Done Sent Done false
5118 1935 Acked Write 7 Done Sent Done false
5119 1936 Acked Read 1 Done Sent Done false
5120 1937 Acked Read 5 Done Sent Done false
5121 1938 Acked Read 3 Done Sent Done false
5122 1939 Acked Read 7 Done Sent Done false
5123 1940 Acked Flush 0 Done Sent Done false
5124 1941 Acked Write 5 Done Sent Done false
5125 1942 Acked Read 3 Done Sent Done false
5126 1943 Acked Write 4 Done Sent Done false
5127 1944 Acked Read 6 Done Sent Done false
5128 1945 Acked Write 4 Done Sent Done false
5129 1946 Acked Write 5 Done Sent Done false
5130 1947 Acked Read 6 Done Sent Done false
5131 1948 Acked Read 5 Done Sent Done false
5132 1949 Acked Write 6 Done Sent Done false
5133 1950 Acked Write 6 Done Sent Done false
5134 1951 Acked Write 10 Done Sent Done false
5135 1952 Acked Write 3 Done Sent Done false
5136 1953 Acked Flush 0 Done Sent Done false
5137 1954 Acked Write 7 Done Sent Done false
5138 1955 Acked Write 6 Done Sent Done false
5139 1956 Acked Write 10 Done Sent Done false
5140 1957 Acked Write 7 Done Sent Done false
5141 1958 Acked Write 8 Done Sent Done false
5142 1959 Acked Read 2 Done Sent Done false
5143 1960 Acked Write 4 Done Sent Done false
5144 1961 Acked Read 6 Done Sent Done false
5145 1962 Acked Read 4 Done Sent Done false
5146 1963 Acked Read 7 Done Sent Done false
5147 1964 Acked Write 2 Done Sent Done false
5148 1965 Acked Read 4 Done Sent Done false
5149 1966 Acked Write 7 Done Sent Done false
5150 1967 Acked Read 10 Done Sent Done false
5151 1968 Acked Write 7 Done Sent Done false
5152 1969 Acked Write 8 Done Sent Done false
5153 1970 Acked Write 4 Done Sent Done false
5154 1971 Acked Read 3 Done Sent Done false
5155 1972 Acked Read 8 Done Sent Done false
5156 1973 Acked Read 7 Done Sent Done false
5157 1974 Acked Write 1 Done Sent Done false
5158 1975 Acked Write 1 Done Sent Done false
5159 1976 Acked Write 3 Done Sent Done false
5160 1977 Acked Read 6 Done Sent Done false
5161 1978 Acked Write 8 Done Sent Done false
5162 1979 Acked Write 9 Done Sent Done false
5163 1980 Acked Write 4 Done Sent Done false
5164 1981 Acked Read 10 Done Sent Done false
5165 1982 Acked Write 8 Done Sent Done false
5166 1983 Acked Write 5 Done Sent Done false
5167 1984 Acked Flush 0 Done Sent Done false
5168 1985 Acked Flush 0 Done Sent Done false
5169 1986 Acked Read 10 Done Sent Done false
5170 1987 Acked Read 6 Done Sent Done false
5171 1988 Acked Flush 0 Done Sent Done false
5172 1989 Acked Read 9 Done Sent Done false
5173 1990 Acked Read 10 Done Sent Done false
5174 1991 Acked Read 10 Done Sent Done false
5175 1992 Acked Read 5 Done Sent Done false
5176 1993 Acked Write 5 Done Sent Done false
5177 1994 Acked Flush 0 Done Sent Done false
5178 1995 Acked Read 5 Done Sent Done false
5179 1996 Acked Read 3 Done Sent Done false
5180 1997 Acked Read 9 Done Sent Done false
5181 1998 Acked Write 10 Done Sent Done false
5182 1999 Acked Flush 0 Done Sent Done false
5183 2000 Acked Read 10 Done Sent Done false
5184 2001 Acked Flush 0 Done Sent Done false
5185 2002 Acked Flush 0 Done Sent Done false
5186 2003 Acked Write 7 Done Sent Done false
5187 2004 Acked Read 4 Done Sent Done false
5188 2005 Acked Read 6 Done Sent Done false
5189 2006 Acked Flush 0 Done Sent Done false
5190 2007 Acked Read 8 Done Sent Done false
5191 2008 Acked Write 5 Done Sent Done false
5192 2009 Acked Read 9 Done Sent Done false
5193 2010 Acked Read 7 Done Sent Done false
5194 2011 Acked Write 10 Done Sent Done false
5195 2012 Acked Write 10 Done Sent Done false
5196 2013 Acked Write 9 Done Sent Done false
5197 2014 Acked Write 8 Done Sent Done false
5198 2015 Acked Write 9 Done Sent Done false
5199 2016 Acked Read 3 Done Sent Done false
5200 2017 Acked Write 7 Done Sent Done false
5201 2018 Acked Read 1 Done Sent Done false
5202 2019 Acked Read 10 Done Sent Done false
5203 2020 Acked Read 7 Done Sent Done false
5204 2021 Acked Read 7 Done Sent Done false
5205 2022 Acked Read 4 Done Sent Done false
5206 2023 Acked Write 7 Done Sent Done false
5207 2024 Acked Read 10 Done Sent Done false
5208 2025 Acked Write 3 Done Sent Done false
5209 2026 Acked Write 10 Done Sent Done false
5210 2027 Acked Flush 0 Done Sent Done false
5211 2028 Acked Read 2 Done Sent Done false
5212 2029 Acked Read 6 Done Sent Done false
5213 2030 Acked Write 2 Done Sent Done false
5214 2031 Acked Flush 0 Done Sent Done false
5215 2032 Acked Write 6 Done Sent Done false
5216 2033 Acked Write 6 Done Sent Done false
5217 2034 Acked Read 4 Done Sent Done false
5218 2035 Acked Read 8 Done Sent Done false
5219 2036 Acked Flush 0 Done Sent Done false
5220 2037 Acked Read 8 Done Sent Done false
5221 2038 Acked Read 1 Done Sent Done false
5222 2039 Acked Write 10 Done Sent Done false
5223 2040 Acked Read 7 Done Sent Done false
5224 2041 Acked Write 1 Done Sent Done false
5225 2042 Acked Flush 0 Done Sent Done false
5226 2043 Acked Write 6 Done Sent Done false
5227 2044 Acked Read 6 Done Sent Done false
5228 2045 Acked Write 5 Done Sent Done false
5229 2046 Acked Read 8 Done Sent Done false
5230 2047 Acked Write 7 Done Sent Done false
5231 2048 Acked Read 2 Done Sent Done false
5232 2049 Acked Flush 0 Done Sent Done false
5233 2050 Acked Write 9 Done Sent Done false
5234 2051 Acked Read 10 Done Sent Done false
5235 2052 Acked Write 7 Done Sent Done false
5236 2053 Acked Write 4 Done Sent Done false
5237 2054 Acked Flush 0 Done Sent Done false
5238 2055 Acked Flush 0 Done Sent Done false
5239 2056 Acked Read 6 Done Sent Done false
5240 2057 Acked Flush 0 Done Sent Done false
5241 2058 Acked Read 2 Done Sent Done false
5242 2059 Acked Read 7 Done Sent Done false
5243 2060 Acked Read 9 Done Sent Done false
5244 2061 Acked Flush 0 Done Sent Done false
5245 2062 Acked Read 5 Done Sent Done false
5246 2063 Acked Read 2 Done Sent Done false
5247 2064 Acked Write 9 Done Sent Done false
5248 2065 Acked Read 2 Done Sent Done false
5249 2066 Acked Write 1 Done Sent Done false
5250 2067 Acked Read 5 Done Sent Done false
5251 2068 Acked Write 5 Done Sent Done false
5252 2069 Acked Read 5 Done Sent Done false
5253 2070 Acked Write 8 Done Sent Done false
5254 2071 Acked Read 7 Done Sent Done false
5255 2072 Acked Read 2 Done Sent Done false
5256 2073 Acked Write 2 Done Sent Done false
5257 2074 Acked Flush 0 Done Sent Done false
5258 2075 Acked Write 5 Done Sent Done false
5259 2076 Acked Read 5 Done Sent Done false
5260 2077 Acked Read 2 Done Sent Done false
5261 2078 Acked Read 2 Done Sent Done false
5262 2079 Acked Write 2 Done Sent Done false
5263 2080 Acked Flush 0 Done Sent Done false
5264 2081 Acked Read 8 Done Sent Done false
5265 2082 Acked Read 1 Done Sent Done false
5266 2083 Acked Read 9 Done Sent Done false
5267 2084 Acked Write 3 Done Sent Done false
5268 2085 Acked Read 9 Done Sent Done false
5269 2086 Acked Write 9 Done Sent Done false
5270 2087 Acked Write 1 Done Sent Done false
5271 2088 Acked Write 6 Done Sent Done false
5272 2089 Acked Write 5 Done Sent Done false
5273 2090 Acked Read 6 Done Sent Done false
5274 2091 Acked Read 3 Done Sent Done false
5275 2092 Acked Flush 0 Done Sent Done false
5276 2093 Acked Write 7 Done Sent Done false
5277 2094 Acked Flush 0 Done Sent Done false
5278 2095 Acked Read 1 Done Sent Done false
5279 2096 Acked Write 1 Done Sent Done false
5280 2097 Acked Read 3 Done Sent Done false
5281 2098 Acked Read 7 Done Sent Done false
5282 2099 Acked Write 6 Done Sent Done false
5283 2100 Acked Flush 0 Done Sent Done false
5284 2101 Acked Write 7 Done Sent Done false
5285 2102 Acked Read 6 Done Sent Done false
5286 2103 Acked Write 9 Done Sent Done false
5287 2104 Acked Flush 0 Done Sent Done false
5288 2105 Acked Flush 0 Done Sent Done false
5289 2106 Acked Read 5 Done Sent Done false
5290 2107 Acked Write 4 Done Sent Done false
5291 2108 Acked Read 2 Done Sent Done false
5292 2109 Acked Flush 0 Done Sent Done false
5293 2110 Acked Flush 0 Done Sent Done false
5294 2111 Acked Read 6 Done Sent Done false
5295 2112 Acked Write 4 Done Sent Done false
5296 2113 Acked Write 3 Done Sent Done false
5297 2114 Acked Write 3 Done Sent Done false
5298 2115 Acked Read 1 Done Sent Done false
5299 2116 Acked Write 10 Done Sent Done false
5300 2117 Acked Write 9 Done Sent Done false
5301 2118 Acked Write 1 Done Sent Done false
5302 2119 Acked Read 6 Done Sent Done false
5303 2120 Acked Read 1 Done Sent Done false
5304 2121 Acked Write 1 Done Sent Done false
5305 2122 Acked Read 5 Done Sent Done false
5306 2123 Acked Read 4 Done Sent Done false
5307 2124 Acked Read 3 Done Sent Done false
5308 2125 Acked Write 3 Done Sent Done false
5309 2126 Acked Write 2 Done Sent Done false
5310 2127 Acked Write 7 Done Sent Done false
5311 2128 Acked Read 1 Done Sent Done false
5312 2129 Acked Write 9 Done Sent Done false
5313 2130 Acked Read 2 Done Sent Done false
5314 2131 Acked Write 2 Done Sent Done false
5315 2132 Acked Flush 0 Done Sent Done false
5316 2133 Acked Write 9 Done Sent Done false
5317 2134 Acked Read 5 Done Sent Done false
5318 2135 Acked Write 6 Done Sent Done false
5319 2136 Acked Read 7 Done Sent Done false
5320 2137 Acked Read 4 Done Sent Done false
5321 2138 Acked Read 4 Done Sent Done false
5322 2139 Acked Flush 0 Done Sent Done false
5323 2140 Acked Flush 0 Done Sent Done false
5324 2141 Acked Flush 0 Done Sent Done false
5325 2142 Acked Read 1 Done Sent Done false
5326 2143 Acked Write 2 Done Sent Done false
5327 2144 Acked Write 8 Done Sent Done false
5328 2145 Acked Write 9 Done Sent Done false
5329 2146 Acked Write 7 Done Sent Done false
5330 2147 Acked Read 10 Done Sent Done false
5331 2148 Acked Read 5 Done Sent Done false
5332 2149 Acked Write 10 Done Sent Done false
5333 2150 Acked Write 3 Done Sent Done false
5334 2151 Acked Read 4 Done Sent Done false
5335 2152 Acked Read 1 Done Sent Done false
5336 2153 Acked Read 8 Done Sent Done false
5337 2154 Acked Write 8 Done Sent Done false
5338 2155 Acked Flush 0 Done Sent Done false
5339 2156 Acked Read 9 Done Sent Done false
5340 2157 Acked Read 8 Done Sent Done false
5341 2158 Acked Write 1 Done Sent Done false
5342 2159 Acked Flush 0 Done Sent Done false
5343 2160 Acked Read 2 Done Sent Done false
5344 2161 Acked Write 8 Done Sent Done false
5345 2162 Acked Read 7 Done Sent Done false
5346 2163 Acked Write 2 Done Sent Done false
5347 2164 Acked Read 6 Done Sent Done false
5348 2165 Acked Write 2 Done Sent Done false
5349 2166 Acked Read 5 Done Sent Done false
5350 2167 Acked Read 8 Done Sent Done false
5351 2168 Acked Read 4 Done Sent Done false
5352 2169 Acked Read 7 Done Sent Done false
5353 2170 Acked Read 7 Done Sent Done false
5354 2171 Acked Write 7 Done Sent Done false
5355 2172 Acked Write 3 Done Sent Done false
5356 2173 Acked Write 8 Done Sent Done false
5357 2174 Acked Read 6 Done Sent Done false
5358 2175 Acked Read 9 Done Sent Done false
5359 2176 Acked Read 3 Done Sent Done false
5360 2177 Acked Read 3 Done Sent Done false
5361 2178 Acked Flush 0 Done Sent Done false
5362 2179 Acked Read 2 Done Sent Done false
5363 2180 Acked Flush 0 Done Sent Done false
5364 2181 Acked Write 3 Done Sent Done false
5365 2182 Acked Write 6 Done Sent Done false
5366 2183 Acked Read 9 Done Sent Done false
5367 2184 Acked Write 6 Done Sent Done false
5368 2185 Acked Read 7 Done Sent Done false
5369 2186 Acked Read 1 Done Sent Done false
5370 2187 Acked Flush 0 Done Sent Done false
5371 2188 Acked Write 6 Done Sent Done false
5372 2189 Acked Write 10 Done Sent Done false
5373 2190 Acked Read 1 Done Sent Done false
5374 2191 Acked Write 3 Done Sent Done false
5375 2192 Acked Write 4 Done Sent Done false
5376 2193 Acked Read 4 Done Sent Done false
5377 2194 Acked Read 2 Done Sent Done false
5378 2195 Acked Read 4 Done Sent Done false
5379 2196 Acked Write 9 Done Sent Done false
5380 2197 Acked Read 5 Done Sent Done false
5381 2198 Acked Write 8 Done Sent Done false
5382 2199 Acked Read 3 Done Sent Done false
5383 2200 Acked Write 1 Done Sent Done false
5384 2201 Acked Read 6 Done Sent Done false
5385 2202 Acked Flush 0 Done Sent Done false
5386 2203 Acked Read 10 Done Sent Done false
5387 2204 Acked Read 2 Done Sent Done false
5388 STATES DS:0 DS:1 DS:2 TOTAL
5389 Sent 0 301 0 301
5390 Done 301 0 301 602
5391 Skipped 0 0 0 0
5392 Error 0 0 0 0
5393 Last Flush: Some(JobId(2202)) Some(JobId(1903)) Some(JobId(2202))
5394 Downstairs last five completed: 1903 1902 1901 1900 1899
5395 Upstairs last five completed: 2204 2203 2202 2201 2200
5396 CLIENT: Up:0 ds:301 act:2
53972025-02-28T06:02:09.447ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
53982025-02-28T06:02:10.447ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
53992025-02-28T06:02:10.640ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
54002025-02-28T06:02:10.640ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37128) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
54012025-02-28T06:02:10.641ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
54022025-02-28T06:02:10.642ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
54032025-02-28T06:02:10.642ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
54042025-02-28T06:02:10.642ZINFOcrucible: send last flush ID to this DS: Some(JobId(1903)) = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
54052025-02-28T06:02:10.642ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
54062025-02-28T06:02:10.642ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(1903)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
54072025-02-28T06:02:10.642ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(1903)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
5408 ----------------------------------------------------------------
5409 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5410 Downstairs last five completed: 2205 2204 2203 2202 2201
5411 Upstairs last five completed: 2205 2204 2203 2202 2201
5412 CLIENT: Up:0 ds:0 act:3
5413 Replay: All jobs finished
5414 Main task has work to do, go find it
5415 got dsc Stop(1)
5416 stop 1
5417 Feb 28 06:02:12.882 INFO[8820] Got stop action so:false kr:false
5418 request completed, latency_us: 342, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 3086fd52-3f58-48fe-b182-c7de404c4107, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
5419 Feb 28 06:02:12.882 INFO request completed, latency_us: 121, response_code: 200, uri: /state/cid/1, method: GET, req_id: b06365b1-cc7d-4a4e-a62d-7f2b3b7f6801, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
54202025-02-28T06:02:12.882ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
54212025-02-28T06:02:12.882ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
54222025-02-28T06:02:12.882ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
54232025-02-28T06:02:12.882ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
5424 [8820] Exited with: ExitStatus(unix_wait_status(9))
5425 [8820][1] reports Exit
5426 Feb 28 06:02:16.885 INFO request completed, latency_us: 399, response_code: 200, uri: /state/cid/1, method: GET, req_id: 275ec45e-eee3-40e0-b3bb-c67ecf225d73, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
5427 001/300 Write block 4967 len 16384 data: 3 2 2 2
5428 002/300 Read block 3779 len 32768
5429 003/300 Read block 209 len 40960
5430 004/300 Read block 4935 len 32768
5431 005/300 Write block 3351 len 24576 data: 2 2 2 2 2 2
5432 006/300 Flush
5433 007/300 Write block 1198 len 4096 data: 2
5434 008/300 Read block 3251 len 36864
5435 009/300 Write block 3974 len 36864 data: 2 2 2 2 2 2 2 2 2
5436 010/300 Read block 3960 len 24576
5437 011/300 Read block 4832 len 28672
5438 012/300 Write block 303 len 24576 data: 2 2 2 2 2 2
5439 013/300 Read block 4476 len 36864
5440 014/300 Read block 4976 len 20480
5441 015/300 Write block 80 len 28672 data: 2 3 3 3 2 2 2
5442 016/300 Read block 4976 len 24576
5443 017/300 Write block 1084 len 24576 data: 2 2 2 2 2 2
5444 018/300 Read block 2357 len 16384
5445 019/300 Write block 493 len 28672 data: 2 2 2 2 2 2 2
5446 020/300 Read block 1783 len 16384
5447 021/300 Read block 98 len 32768
5448 022/300 Write block 51 len 32768 data: 2 2 2 2 2 2 2 2
5449 023/300 Read block 4041 len 4096
5450 024/300 Read block 2425 len 8192
5451 025/300 Write block 174 len 20480 data: 3 3 3 3 3
5452 026/300 Read block 4124 len 36864
5453 027/300 Read block 1878 len 24576
5454 028/300 Write block 4524 len 28672 data: 2 2 2 2 3 3 3
5455 029/300 Read block 1383 len 28672
5456 030/300 Write block 3873 len 32768 data: 4 4 4 4 3 3 2 2
5457 031/300 Flush
5458 032/300 Read block 4882 len 8192
5459 033/300 Flush
5460 034/300 Read block 2739 len 4096
5461 035/300 Write block 2213 len 28672 data: 5 5 5 5 5 5 3
5462 036/300 Flush
5463 037/300 Read block 2218 len 28672
5464 038/300 Flush
5465 039/300 Write block 4043 len 16384 data: 2 2 2 2
5466 040/300 Write block 4630 len 16384 data: 2 3 3 3
5467 041/300 Write block 2306 len 16384 data: 2 3 3 3
5468 042/300 Write block 1473 len 8192 data: 2 2
5469 043/300 Read block 1873 len 16384
5470 044/300 Write block 2000 len 16384 data: 2 2 2 2
5471 045/300 Read block 1179 len 12288
5472 046/300 Write block 3743 len 36864 data: 4 4 4 2 2 2 2 2 3
5473 047/300 Write block 4796 len 12288 data: 2 2 2
5474 048/300 Write block 1161 len 4096 data: 3
5475 049/300 Write block 449 len 24576 data: 6 6 6 6 5 5
5476 050/300 Read block 2163 len 12288
5477 051/300 Write block 4777 len 12288 data: 3 3 3
5478 052/300 Write block 3791 len 8192 data: 2 2
5479 053/300 Read block 1529 len 28672
5480 054/300 Flush
5481 055/300 Read block 3042 len 40960
5482 056/300 Read block 2927 len 4096
5483 057/300 Read block 4734 len 4096
5484 058/300 Write block 160 len 12288 data: 2 2 2
5485 059/300 Write block 1052 len 20480 data: 3 3 3 2 2
5486 060/300 Write block 251 len 12288 data: 2 2 2
5487 061/300 Flush
5488 062/300 Read block 4402 len 40960
5489 063/300 Read block 839 len 20480
5490 064/300 Flush
5491 065/300 Write block 1983 len 28672 data: 2 2 2 2 2 3 3
5492 066/300 Read block 1562 len 28672
5493 067/300 Read block 1914 len 40960
5494 068/300 Write block 1084 len 32768 data: 3 3 3 3 3 3 2 2
5495 069/300 Read block 4398 len 16384
5496 070/300 Write block 358 len 36864 data: 2 2 2 2 2 2 2 2 2
5497 071/300 Write block 4825 len 32768 data: 2 2 2 3 3 4 5 3
5498 072/300 Write block 188 len 36864 data: 2 2 2 3 3 3 3 2 2
5499 073/300 Read block 3622 len 8192
5500 074/300 Write block 3410 len 8192 data: 2 2
5501 075/300 Read block 3750 len 24576
5502 076/300 Read block 3873 len 36864
5503 077/300 Read block 2555 len 40960
5504 078/300 Read block 3781 len 36864
5505 079/300 Read block 664 len 8192
5506 080/300 Read block 3678 len 36864
5507 081/300 Read block 1804 len 16384
5508 082/300 Write block 1440 len 36864 data: 2 2 2 2 2 2 2 2 3
5509 083/300 Read block 1817 len 12288
5510 084/300 Read block 264 len 40960
5511 085/300 Write block 2256 len 8192 data: 2 3
5512 086/300 Write block 2811 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5513 087/300 Flush
5514 088/300 Read block 4321 len 12288
5515 089/300 Write block 2957 len 12288 data: 2 2 2
5516 090/300 Read block 1109 len 16384
5517 091/300 Read block 4648 len 8192
5518 092/300 Write block 4280 len 12288 data: 3 3 3
5519 093/300 Read block 3114 len 20480
5520 094/300 Read block 844 len 32768
5521 095/300 Read block 181 len 12288
5522 096/300 Read block 1397 len 4096
5523 097/300 Read block 4678 len 24576
5524 098/300 Write block 1636 len 8192 data: 3 3
5525 099/300 Write block 4699 len 20480 data: 2 2 2 2 2
5526 100/300 Write block 2471 len 36864 data: 2 2 2 2 2 2 2 2 2
5527 101/300 Read block 431 len 12288
5528 102/300 Write block 1613 len 20480 data: 3 3 3 3 3
5529 103/300 Read block 4113 len 32768
5530 104/300 Write block 1247 len 16384 data: 2 2 2 2
5531 105/300 Read block 64 len 8192
5532 106/300 Read block 2624 len 4096
5533 107/300 Read block 419 len 24576
5534 108/300 Read block 2259 len 32768
5535 109/300 Read block 201 len 28672
5536 110/300 Read block 1034 len 20480
5537 111/300 Write block 4652 len 28672 data: 2 2 2 2 2 2 2
5538 112/300 Write block 2189 len 8192 data: 3 4
5539 113/300 Write block 472 len 28672 data: 3 3 3 3 3 3 3
5540 114/300 Write block 3931 len 40960 data: 2 2 2 3 3 3 3 4 5 5
5541 115/300 Read block 2130 len 40960
5542 116/300 Write block 2318 len 8192 data: 4 5
5543 117/300 Write block 4289 len 40960 data: 2 3 3 3 2 2 3 3 3 2
5544 118/300 Write block 509 len 4096 data: 2
5545 119/300 Read block 4708 len 16384
5546 120/300 Read block 2836 len 20480
5547 121/300 Write block 667 len 24576 data: 4 4 4 4 4 4
5548 122/300 Read block 1374 len 40960
5549 123/300 Write block 1660 len 40960 data: 3 2 2 2 2 2 2 2 3 3
5550 124/300 Read block 869 len 24576
5551 125/300 Read block 4338 len 24576
5552 126/300 Write block 700 len 20480 data: 2 2 2 2 2
5553 127/300 Write block 2178 len 20480 data: 2 3 3 3 3
5554 128/300 Write block 2929 len 24576 data: 2 2 2 2 2 2
5555 129/300 Read block 1298 len 12288
5556 130/300 Flush
5557 131/300 Read block 1912 len 16384
5558 132/300 Flush
5559 133/300 Write block 1621 len 4096 data: 3
5560 134/300 Flush
5561 135/300 Read block 4631 len 4096
5562 136/300 Write block 345 len 36864 data: 3 3 3 3 3 3 2 2 2
5563 137/300 Write block 1441 len 36864 data: 3 3 3 3 3 3 3 4 3
5564 138/300 Read block 669 len 28672
5565 139/300 Write block 4226 len 24576 data: 2 2 2 2 2 2
5566 140/300 Write block 526 len 40960 data: 2 2 2 2 2 2 3 3 3 2
5567 141/300 Read block 934 len 12288
5568 142/300 Write block 4647 len 8192 data: 3 2
5569 143/300 Read block 4465 len 12288
5570 144/300 Read block 4403 len 20480
5571 145/300 Write block 2179 len 20480 data: 4 4 4 4 3
5572 146/300 Write block 3323 len 4096 data: 2
5573 147/300 Write block 1077 len 16384 data: 2 3 3 4
5574 148/300 Read block 1877 len 40960
5575 149/300 Read block 3604 len 28672
5576 150/300 Write block 4791 len 20480 data: 2 2 2 2 2
5577 151/300 Read block 4975 len 8192
5578 152/300 Write block 2757 len 28672 data: 2 2 2 2 2 2 2
5579 153/300 Write block 295 len 28672 data: 3 3 3 3 3 2 2
5580 154/300 Write block 4407 len 36864 data: 5 6 6 6 6 6 4 3 3
5581 155/300 Read block 1220 len 20480
5582 156/300 Write block 2595 len 8192 data: 2 2
5583 157/300 Flush
5584 158/300 Read block 3232 len 40960
5585 159/300 Read block 4770 len 12288
5586 160/300 Read block 1569 len 40960
5587 161/300 Read block 1971 len 4096
5588 162/300 Write block 4461 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5589 163/300 Read block 34 len 28672
5590 164/300 Read block 4467 len 20480
5591 165/300 Write block 4186 len 24576 data: 4 4 2 2 4 4
5592 166/300 Read block 4162 len 40960
5593 167/300 Write block 4383 len 16384 data: 2 2 2 2
5594 168/300 Read block 4668 len 24576
5595 169/300 Write block 414 len 40960 data: 3 3 3 3 3 3 3 3 4 3
5596 170/300 Read block 4381 len 12288
5597 171/300 Read block 2981 len 40960
5598 172/300 Read block 3796 len 16384
5599 173/300 Flush
5600 174/300 Read block 3707 len 24576
5601 175/300 Read block 3490 len 8192
5602 176/300 Flush
5603 177/300 Read block 2212 len 8192
5604 178/300 Write block 3662 len 32768 data: 2 2 2 3 3 3 3 3
5605 179/300 Flush
5606 180/300 Write block 127 len 28672 data: 3 3 3 3 3 2 4
5607 181/300 Write block 2323 len 32768 data: 3 2 2 2 2 2 2 2
5608 182/300 Write block 2487 len 12288 data: 2 2 2
5609 183/300 Write block 1329 len 16384 data: 3 3 3 3
5610 184/300 Write block 3970 len 40960 data: 2 2 2 2 3 3 3 3 3 3
5611 185/300 Write block 2547 len 24576 data: 3 3 3 3 3 2
5612 186/300 Read block 4469 len 20480
5613 187/300 Flush
5614 188/300 Read block 1715 len 12288
5615 189/300 Write block 4138 len 24576 data: 2 2 2 3 3 3
5616 190/300 Write block 4557 len 36864 data: 2 2 2 2 2 2 2 2 2
5617 191/300 Write block 2983 len 12288 data: 3 3 3
5618 192/300 Write block 4906 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5619 193/300 Read block 3143 len 32768
5620 194/300 Write block 4052 len 20480 data: 2 2 3 3 3
5621 195/300 Write block 2045 len 36864 data: 2 2 2 2 2 2 2 2 2
5622 196/300 Write block 1254 len 20480 data: 2 2 2 2 2
5623 197/300 Write block 99 len 36864 data: 2 2 2 2 2 2 2 2 3
5624 198/300 Write block 572 len 36864 data: 3 3 2 2 2 3 3 3 4
5625 199/300 Read block 3109 len 4096
5626 200/300 Write block 1220 len 20480 data: 2 2 2 2 2
5627 201/300 Read block 1440 len 28672
5628 202/300 Read block 4342 len 20480
5629 203/300 Read block 3940 len 8192
5630 204/300 Write block 1219 len 4096 data: 2
5631 205/300 Read block 1550 len 16384
5632 206/300 Read block 1074 len 40960
5633 207/300 Write block 2747 len 4096 data: 3
5634 208/300 Flush
5635 209/300 Read block 1474 len 40960
5636 210/300 Flush
5637 211/300 Read block 4488 len 16384
5638 212/300 Write block 236 len 28672 data: 3 5 5 3 2 2 2
5639 213/300 Read block 3598 len 36864
5640 214/300 Write block 2579 len 12288 data: 2 2 2
5641 215/300 Write block 4788 len 40960 data: 3 3 2 3 3 3 3 3 3 3
5642 216/300 Write block 1172 len 16384 data: 2 2 2 2
5643 217/300 Write block 2278 len 16384 data: 4 3 3 3
5644 218/300 Read block 4788 len 20480
5645 219/300 Read block 2846 len 32768
5646 220/300 Write block 2842 len 36864 data: 2 2 2 2 2 2 2 2 2
5647 221/300 Flush
5648 222/300 Flush
5649 223/300 Read block 831 len 8192
5650 224/300 Write block 4271 len 16384 data: 4 4 3 3
5651 225/300 Read block 2268 len 20480
5652 226/300 Write block 3848 len 28672 data: 2 2 2 2 3 3 3
5653 227/300 Read block 171 len 4096
5654 228/300 Read block 2657 len 28672
5655 229/300 Read block 1905 len 4096
5656 230/300 Write block 458 len 16384 data: 2 2 2 2
5657 231/300 Write block 3718 len 28672 data: 2 2 2 2 2 2 2
5658 232/300 Write block 4319 len 20480 data: 2 2 2 2 2
5659 233/300 Read block 498 len 20480
5660 234/300 Read block 4644 len 28672
5661 235/300 Read block 1022 len 28672
5662 236/300 Read block 1656 len 36864
5663 237/300 Read block 4943 len 8192
5664 238/300 Read block 2122 len 16384
5665 239/300 Write block 1203 len 16384 data: 2 2 2 2
5666 240/300 Read block 14 len 16384
5667 241/300 Read block 3655 len 28672
5668 242/300 Write block 3742 len 24576 data: 4 5 5 5 3 3
5669 243/300 Write block 3411 len 16384 data: 3 2 2 2
5670 244/300 Flush
5671 245/300 Read block 4872 len 24576
5672 246/300 Write block 870 len 32768 data: 3 2 2 2 2 2 2 2
5673 247/300 Read block 3432 len 20480
5674 248/300 Read block 2931 len 16384
5675 249/300 Write block 3293 len 24576 data: 2 2 2 3 4 4
5676 250/300 Read block 1332 len 32768
5677 251/300 Read block 1930 len 20480
5678 252/300 Write block 111 len 8192 data: 3 3
5679 253/300 Write block 4971 len 16384 data: 2 2 2 2
5680 254/300 Write block 2968 len 20480 data: 4 4 3 2 2
5681 255/300 Write block 4737 len 36864 data: 2 2 2 2 2 2 2 2 2
5682 256/300 Write block 598 len 24576 data: 2 2 2 2 2 2
5683 257/300 Read block 3482 len 8192
5684 258/300 Read block 2418 len 24576
5685 259/300 Read block 3927 len 24576
5686 260/300 Write block 1839 len 8192 data: 3 3
5687 261/300 Flush
5688 262/300 Write block 3520 len 40960 data: 2 2 3 3 3 3 3 4 4 3
5689 263/300 Read block 718 len 36864
5690 264/300 Write block 4593 len 12288 data: 2 2 2
5691 265/300 Read block 3249 len 36864
5692 266/300 Read block 2201 len 20480
5693 267/300 Flush
5694 268/300 Write block 1747 len 16384 data: 2 2 2 2
5695 269/300 Write block 1534 len 36864 data: 3 3 3 3 4 3 3 3 3
5696 270/300 Write block 2369 len 4096 data: 3
5697 271/300 Read block 1995 len 28672
5698 272/300 Write block 1451 len 28672 data: 3 3 3 3 2 2 2
5699 273/300 Read block 1670 len 32768
5700 274/300 Write block 791 len 20480 data: 2 2 2 2 2
5701 275/300 Read block 285 len 28672
5702 276/300 Read block 1260 len 20480
5703 277/300 Write block 1186 len 40960 data: 3 2 2 2 2 2 2 2 2 2
5704 278/300 Write block 4647 len 36864 data: 4 3 2 2 2 3 3 3 3
5705 279/300 Write block 3533 len 8192 data: 2 2
5706 280/300 Write block 2861 len 40960 data: 2 2 3 2 2 2 2 2 2 2
5707 281/300 Read block 1285 len 16384
5708 282/300 Write block 25 len 16384 data: 3 2 2 2
5709 283/300 Read block 2482 len 4096
5710 284/300 Read block 4752 len 8192
5711 285/300 Read block 4265 len 20480
5712 286/300 Read block 3101 len 32768
5713 287/300 Read block 3075 len 8192
5714 288/300 Read block 4797 len 20480
5715 289/300 Read block 3618 len 40960
5716 290/300 Write block 380 len 28672 data: 5 4 4 4 3 3 3
5717 291/300 Write block 2749 len 32768 data: 2 3 3 2 2 2 2 2
5718 292/300 Write block 477 len 8192 data: 4 4
5719 293/300 Read block 2302 len 12288
5720 294/300 Read block 4698 len 20480
5721 295/300 Write block 2279 len 20480 data: 4 4 4 3 4
5722 296/300 Write block 1889 len 20480 data: 2 2 4 4 4
5723 297/300 Read block 4783 len 28672
5724 298/300 Write block 1514 len 20480 data: 3 2 2 3 3
5725 299/300 Write block 264 len 20480 data: 2 2 2 2 2
5726 300/300 Write block 1936 len 36864 data: 3 3 2 2 2 2 2 2 2
5727 301/300 Flush
5728 Main task has work to do, go find it
5729 got dsc Start(1)
5730 start 1
5731 Feb 28 06:02:17.123 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/1, method: POST, req_id: 551c31d2-9831-41ff-bcfd-1f485fd0f484, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
5732 Replay: started 1, returned:Ok(())
5733 ----------------------------------------------------------------
5734 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5735 States: Active Offline Active
5736 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5737 2206 Acked Write 4 Done Sent Done false
5738 2207 Acked Read 8 Done Sent Done false
5739 2208 Acked Read 10 Done Sent Done false
5740 2209 Acked Read 8 Done Sent Done false
5741 2210 Acked Write 6 Done Sent Done false
5742 2211 Acked Flush 0 Done Sent Done false
5743 2212 Acked Write 1 Done Sent Done false
5744 2213 Acked Read 9 Done Sent Done false
5745 2214 Acked Write 9 Done Sent Done false
5746 2215 Acked Read 6 Done Sent Done false
5747 2216 Acked Read 7 Done Sent Done false
5748 2217 Acked Write 6 Done Sent Done false
5749 2218 Acked Read 9 Done Sent Done false
5750 2219 Acked Read 5 Done Sent Done false
5751 2220 Acked Write 7 Done Sent Done false
5752 2221 Acked Read 6 Done Sent Done false
5753 2222 Acked Write 6 Done Sent Done false
5754 2223 Acked Read 4 Done Sent Done false
5755 2224 Acked Write 7 Done Sent Done false
5756 2225 Acked Read 4 Done Sent Done false
5757 2226 Acked Read 8 Done Sent Done false
5758 2227 Acked Write 8 Done Sent Done false
5759 2228 Acked Read 1 Done Sent Done false
5760 2229 Acked Read 2 Done Sent Done false
5761 2230 Acked Write 5 Done Sent Done false
5762 2231 Acked Read 9 Done Sent Done false
5763 2232 Acked Read 6 Done Sent Done false
5764 2233 Acked Write 7 Done Sent Done false
5765 2234 Acked Read 7 Done Sent Done false
5766 2235 Acked Write 8 Done Sent Done false
5767 2236 Acked Flush 0 Done Sent Done false
5768 2237 Acked Read 2 Done Sent Done false
5769 2238 Acked Flush 0 Done Sent Done false
5770 2239 Acked Read 1 Done Sent Done false
5771 2240 Acked Write 7 Done Sent Done false
5772 2241 Acked Flush 0 Done Sent Done false
5773 2242 Acked Read 7 Done Sent Done false
5774 2243 Acked Flush 0 Done Sent Done false
5775 2244 Acked Write 4 Done Sent Done false
5776 2245 Acked Write 4 Done Sent Done false
5777 2246 Acked Write 4 Done Sent Done false
5778 2247 Acked Write 2 Done Sent Done false
5779 2248 Acked Read 4 Done Sent Done false
5780 2249 Acked Write 4 Done Sent Done false
5781 2250 Acked Read 3 Done Sent Done false
5782 2251 Acked Write 9 Done Sent Done false
5783 2252 Acked Write 3 Done Sent Done false
5784 2253 Acked Write 1 Done Sent Done false
5785 2254 Acked Write 6 Done Sent Done false
5786 2255 Acked Read 3 Done Sent Done false
5787 2256 Acked Write 3 Done Sent Done false
5788 2257 Acked Write 2 Done Sent Done false
5789 2258 Acked Read 7 Done Sent Done false
5790 2259 Acked Flush 0 Done Sent Done false
5791 2260 Acked Read 10 Done Sent Done false
5792 2261 Acked Read 1 Done Sent Done false
5793 2262 Acked Read 1 Done Sent Done false
5794 2263 Acked Write 3 Done Sent Done false
5795 2264 Acked Write 5 Done Sent Done false
5796 2265 Acked Write 3 Done Sent Done false
5797 2266 Acked Flush 0 Done Sent Done false
5798 2267 Acked Read 10 Done Sent Done false
5799 2268 Acked Read 5 Done Sent Done false
5800 2269 Acked Flush 0 Done Sent Done false
5801 2270 Acked Write 7 Done Sent Done false
5802 2271 Acked Read 7 Done Sent Done false
5803 2272 Acked Read 10 Done Sent Done false
5804 2273 Acked Write 8 Done Sent Done false
5805 2274 Acked Read 4 Done Sent Done false
5806 2275 Acked Write 9 Done Sent Done false
5807 2276 Acked Write 8 Done Sent Done false
5808 2277 Acked Write 9 Done Sent Done false
5809 2278 Acked Read 2 Done Sent Done false
5810 2279 Acked Write 2 Done Sent Done false
5811 2280 Acked Read 6 Done Sent Done false
5812 2281 Acked Read 9 Done Sent Done false
5813 2282 Acked Read 10 Done Sent Done false
5814 2283 Acked Read 9 Done Sent Done false
5815 2284 Acked Read 2 Done Sent Done false
5816 2285 Acked Read 9 Done Sent Done false
5817 2286 Acked Read 4 Done Sent Done false
5818 2287 Acked Write 9 Done Sent Done false
5819 2288 Acked Read 3 Done Sent Done false
5820 2289 Acked Read 10 Done Sent Done false
5821 2290 Acked Write 2 Done Sent Done false
5822 2291 Acked Write 10 Done Sent Done false
5823 2292 Acked Flush 0 Done Sent Done false
5824 2293 Acked Read 3 Done Sent Done false
5825 2294 Acked Write 3 Done Sent Done false
5826 2295 Acked Read 4 Done Sent Done false
5827 2296 Acked Read 2 Done Sent Done false
5828 2297 Acked Write 3 Done Sent Done false
5829 2298 Acked Read 5 Done Sent Done false
5830 2299 Acked Read 8 Done Sent Done false
5831 2300 Acked Read 3 Done Sent Done false
5832 2301 Acked Read 1 Done Sent Done false
5833 2302 Acked Read 6 Done Sent Done false
5834 2303 Acked Write 2 Done Sent Done false
5835 2304 Acked Write 5 Done Sent Done false
5836 2305 Acked Write 9 Done Sent Done false
5837 2306 Acked Read 3 Done Sent Done false
5838 2307 Acked Write 5 Done Sent Done false
5839 2308 Acked Read 8 Done Sent Done false
5840 2309 Acked Write 4 Done Sent Done false
5841 2310 Acked Read 2 Done Sent Done false
5842 2311 Acked Read 1 Done Sent Done false
5843 2312 Acked Read 6 Done Sent Done false
5844 2313 Acked Read 8 Done Sent Done false
5845 2314 Acked Read 7 Done Sent Done false
5846 2315 Acked Read 5 Done Sent Done false
5847 2316 Acked Write 7 Done Sent Done false
5848 2317 Acked Write 2 Done Sent Done false
5849 2318 Acked Write 7 Done Sent Done false
5850 2319 Acked Write 10 Done Sent Done false
5851 2320 Acked Read 10 Done Sent Done false
5852 2321 Acked Write 2 Done Sent Done false
5853 2322 Acked Write 10 Done Sent Done false
5854 2323 Acked Write 1 Done Sent Done false
5855 2324 Acked Read 4 Done Sent Done false
5856 2325 Acked Read 5 Done Sent Done false
5857 2326 Acked Write 6 Done Sent Done false
5858 2327 Acked Read 10 Done Sent Done false
5859 2328 Acked Write 10 Done Sent Done false
5860 2329 Acked Read 6 Done Sent Done false
5861 2330 Acked Read 6 Done Sent Done false
5862 2331 Acked Write 5 Done Sent Done false
5863 2332 Acked Write 5 Done Sent Done false
5864 2333 Acked Write 6 Done Sent Done false
5865 2334 Acked Read 3 Done Sent Done false
5866 2335 Acked Flush 0 Done Sent Done false
5867 2336 Acked Read 4 Done Sent Done false
5868 2337 Acked Flush 0 Done Sent Done false
5869 2338 Acked Write 1 Done Sent Done false
5870 2339 Acked Flush 0 Done Sent Done false
5871 2340 Acked Read 1 Done Sent Done false
5872 2341 Acked Write 9 Done Sent Done false
5873 2342 Acked Write 9 Done Sent Done false
5874 2343 Acked Read 7 Done Sent Done false
5875 2344 Acked Write 6 Done Sent Done false
5876 2345 Acked Write 10 Done Sent Done false
5877 2346 Acked Read 3 Done Sent Done false
5878 2347 Acked Write 2 Done Sent Done false
5879 2348 Acked Read 3 Done Sent Done false
5880 2349 Acked Read 5 Done Sent Done false
5881 2350 Acked Write 5 Done Sent Done false
5882 2351 Acked Write 1 Done Sent Done false
5883 2352 Acked Write 4 Done Sent Done false
5884 2353 Acked Read 10 Done Sent Done false
5885 2354 Acked Read 7 Done Sent Done false
5886 2355 Acked Write 5 Done Sent Done false
5887 2356 Acked Read 2 Done Sent Done false
5888 2357 Acked Write 7 Done Sent Done false
5889 2358 Acked Write 7 Done Sent Done false
5890 2359 Acked Write 9 Done Sent Done false
5891 2360 Acked Read 5 Done Sent Done false
5892 2361 Acked Write 2 Done Sent Done false
5893 2362 Acked Flush 0 Done Sent Done false
5894 2363 Acked Read 10 Done Sent Done false
5895 2364 Acked Read 3 Done Sent Done false
5896 2365 Acked Read 10 Done Sent Done false
5897 2366 Acked Read 1 Done Sent Done false
5898 2367 Acked Write 10 Done Sent Done false
5899 2368 Acked Read 7 Done Sent Done false
5900 2369 Acked Read 5 Done Sent Done false
5901 2370 Acked Write 6 Done Sent Done false
5902 2371 Acked Read 10 Done Sent Done false
5903 2372 Acked Write 4 Done Sent Done false
5904 2373 Acked Read 6 Done Sent Done false
5905 2374 Acked Write 10 Done Sent Done false
5906 2375 Acked Read 3 Done Sent Done false
5907 2376 Acked Read 10 Done Sent Done false
5908 2377 Acked Read 4 Done Sent Done false
5909 2378 Acked Flush 0 Done Sent Done false
5910 2379 Acked Read 6 Done Sent Done false
5911 2380 Acked Read 2 Done Sent Done false
5912 2381 Acked Flush 0 Done Sent Done false
5913 2382 Acked Read 2 Done Sent Done false
5914 2383 Acked Write 8 Done Sent Done false
5915 2384 Acked Flush 0 Done Sent Done false
5916 2385 Acked Write 7 Done Sent Done false
5917 2386 Acked Write 8 Done Sent Done false
5918 2387 Acked Write 3 Done Sent Done false
5919 2388 Acked Write 4 Done Sent Done false
5920 2389 Acked Write 10 Done Sent Done false
5921 2390 Acked Write 6 Done Sent Done false
5922 2391 Acked Read 5 Done Sent Done false
5923 2392 Acked Flush 0 Done Sent Done false
5924 2393 Acked Read 3 Done Sent Done false
5925 2394 Acked Write 6 Done Sent Done false
5926 2395 Acked Write 9 Done Sent Done false
5927 2396 Acked Write 3 Done Sent Done false
5928 2397 Acked Write 10 Done Sent Done false
5929 2398 Acked Read 8 Done Sent Done false
5930 2399 Acked Write 5 Done Sent Done false
5931 2400 Acked Write 9 Done Sent Done false
5932 2401 Acked Write 5 Done Sent Done false
5933 2402 Acked Write 9 Done Sent Done false
5934 2403 Acked Write 9 Done Sent Done false
5935 2404 Acked Read 1 Done Sent Done false
5936 2405 Acked Write 5 Done Sent Done false
5937 2406 Acked Read 7 Done Sent Done false
5938 2407 Acked Read 5 Done Sent Done false
5939 2408 Acked Read 2 Done Sent Done false
5940 2409 Acked Write 1 Done Sent Done false
5941 2410 Acked Read 4 Done Sent Done false
5942 2411 Acked Read 10 Done Sent Done false
5943 2412 Acked Write 1 Done Sent Done false
5944 2413 Acked Flush 0 Done Sent Done false
5945 2414 Acked Read 10 Done Sent Done false
5946 2415 Acked Flush 0 Done Sent Done false
5947 2416 Acked Read 4 Done Sent Done false
5948 2417 Acked Write 7 Done Sent Done false
5949 2418 Acked Read 9 Done Sent Done false
5950 2419 Acked Write 3 Done Sent Done false
5951 2420 Acked Write 10 Done Sent Done false
5952 2421 Acked Write 4 Done Sent Done false
5953 2422 Acked Write 4 Done Sent Done false
5954 2423 Acked Read 5 Done Sent Done false
5955 2424 Acked Read 8 Done Sent Done false
5956 2425 Acked Write 9 Done Sent Done false
5957 2426 Acked Flush 0 Done Sent Done false
5958 2427 Acked Flush 0 Done Sent Done false
5959 2428 Acked Read 2 Done Sent Done false
5960 2429 Acked Write 4 Done Sent Done false
5961 2430 Acked Read 5 Done Sent Done false
5962 2431 Acked Write 7 Done Sent Done false
5963 2432 Acked Read 1 Done Sent Done false
5964 2433 Acked Read 7 Done Sent Done false
5965 2434 Acked Read 1 Done Sent Done false
5966 2435 Acked Write 4 Done Sent Done false
5967 2436 Acked Write 7 Done Sent Done false
5968 2437 Acked Write 5 Done Sent Done false
5969 2438 Acked Read 5 Done Sent Done false
5970 2439 Acked Read 7 Done Sent Done false
5971 2440 Acked Read 7 Done Sent Done false
5972 2441 Acked Read 9 Done Sent Done false
5973 2442 Acked Read 2 Done Sent Done false
5974 2443 Acked Read 4 Done Sent Done false
5975 2444 Acked Write 4 Done Sent Done false
5976 2445 Acked Read 4 Done Sent Done false
5977 2446 Acked Read 7 Done Sent Done false
5978 2447 Acked Write 6 Done Sent Done false
5979 2448 Acked Write 4 Done Sent Done false
5980 2449 Acked Flush 0 Done Sent Done false
5981 2450 Acked Read 6 Done Sent Done false
5982 2451 Acked Write 8 Done Sent Done false
5983 2452 Acked Read 5 Done Sent Done false
5984 2453 Acked Read 4 Done Sent Done false
5985 2454 Acked Write 6 Done Sent Done false
5986 2455 Acked Read 8 Done Sent Done false
5987 2456 Acked Read 5 Done Sent Done false
5988 2457 Acked Write 2 Done Sent Done false
5989 2458 Acked Write 4 Done Sent Done false
5990 2459 Acked Write 5 Done Sent Done false
5991 2460 Acked Write 9 Done Sent Done false
5992 2461 Acked Write 6 Done Sent Done false
5993 2462 Acked Read 2 Done Sent Done false
5994 2463 Acked Read 6 Done Sent Done false
5995 2464 Acked Read 6 Done Sent Done false
5996 2465 Acked Write 2 Done Sent Done false
5997 2466 Acked Flush 0 Done Sent Done false
5998 2467 Acked Write 10 Done Sent Done false
5999 2468 Acked Read 9 Done Sent Done false
6000 2469 Acked Write 3 Done Sent Done false
6001 2470 Acked Read 9 Done Sent Done false
6002 2471 Acked Read 5 Done Sent Done false
6003 2472 Acked Flush 0 Done Sent Done false
6004 2473 Acked Write 4 Done Sent Done false
6005 2474 Acked Write 9 Done Sent Done false
6006 2475 Acked Write 1 Done Sent Done false
6007 2476 Acked Read 7 Done Sent Done false
6008 2477 Acked Write 7 Done Sent Done false
6009 2478 Acked Read 8 Done Sent Done false
6010 2479 Acked Write 5 Done Sent Done false
6011 2480 Acked Read 7 Done Sent Done false
6012 2481 Acked Read 5 Done Sent Done false
6013 2482 Acked Write 10 Done Sent Done false
6014 2483 Acked Write 9 Done Sent Done false
6015 2484 Acked Write 2 Done Sent Done false
6016 2485 Acked Write 10 Done Sent Done false
6017 2486 Acked Read 4 Done Sent Done false
6018 2487 Acked Write 4 Done Sent Done false
6019 2488 Acked Read 1 Done Sent Done false
6020 2489 Acked Read 2 Done Sent Done false
6021 2490 Acked Read 5 Done Sent Done false
6022 2491 Acked Read 8 Done Sent Done false
6023 2492 Acked Read 2 Done Sent Done false
6024 2493 Acked Read 5 Done Sent Done false
6025 2494 Acked Read 10 Done Sent Done false
6026 2495 Acked Write 7 Done Sent Done false
6027 2496 Acked Write 8 Done Sent Done false
6028 2497 Acked Write 2 Done Sent Done false
6029 2498 Acked Read 3 Done Sent Done false
6030 2499 Acked Read 5 Done Sent Done false
6031 2500 Acked Write 5 Done Sent Done false
6032 2501 Acked Write 5 Done Sent Done false
6033 2502 Acked Read 7 Done Sent Done false
6034 2503 Acked Write 5 Done Sent Done false
6035 2504 Acked Write 5 Done Sent Done false
6036 2505 Acked Write 9 Done Sent Done false
6037 2506 Acked Flush 0 Done Sent Done false
6038 STATES DS:0 DS:1 DS:2 TOTAL
6039 Sent 0 301 0 301
6040 Done 301 0 301 602
6041 Skipped 0 0 0 0
6042 Error 0 0 0 0
6043 Last Flush: Some(JobId(2506)) Some(JobId(2205)) Some(JobId(2506))
6044 Downstairs last five completed: 2205 2204 2203 2202 2201
6045 Upstairs last five completed: 2506 2505 2504 2503 2502
6046 CLIENT: Up:0 ds:301 act:2
60472025-02-28T06:02:17.625ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
60482025-02-28T06:02:18.462ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
60492025-02-28T06:02:19.463ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
60502025-02-28T06:02:20.465ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
6051 ----------------------------------------------------------------
6052 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6053 States: Active Offline Active
6054 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6055 2206 Acked Write 4 Done Sent Done false
6056 2207 Acked Read 8 Done Sent Done false
6057 2208 Acked Read 10 Done Sent Done false
6058 2209 Acked Read 8 Done Sent Done false
6059 2210 Acked Write 6 Done Sent Done false
6060 2211 Acked Flush 0 Done Sent Done false
6061 2212 Acked Write 1 Done Sent Done false
6062 2213 Acked Read 9 Done Sent Done false
6063 2214 Acked Write 9 Done Sent Done false
6064 2215 Acked Read 6 Done Sent Done false
6065 2216 Acked Read 7 Done Sent Done false
6066 2217 Acked Write 6 Done Sent Done false
6067 2218 Acked Read 9 Done Sent Done false
6068 2219 Acked Read 5 Done Sent Done false
6069 2220 Acked Write 7 Done Sent Done false
6070 2221 Acked Read 6 Done Sent Done false
6071 2222 Acked Write 6 Done Sent Done false
6072 2223 Acked Read 4 Done Sent Done false
6073 2224 Acked Write 7 Done Sent Done false
6074 2225 Acked Read 4 Done Sent Done false
6075 2226 Acked Read 8 Done Sent Done false
6076 2227 Acked Write 8 Done Sent Done false
6077 2228 Acked Read 1 Done Sent Done false
6078 2229 Acked Read 2 Done Sent Done false
6079 2230 Acked Write 5 Done Sent Done false
6080 2231 Acked Read 9 Done Sent Done false
6081 2232 Acked Read 6 Done Sent Done false
6082 2233 Acked Write 7 Done Sent Done false
6083 2234 Acked Read 7 Done Sent Done false
6084 2235 Acked Write 8 Done Sent Done false
6085 2236 Acked Flush 0 Done Sent Done false
6086 2237 Acked Read 2 Done Sent Done false
6087 2238 Acked Flush 0 Done Sent Done false
6088 2239 Acked Read 1 Done Sent Done false
6089 2240 Acked Write 7 Done Sent Done false
6090 2241 Acked Flush 0 Done Sent Done false
6091 2242 Acked Read 7 Done Sent Done false
6092 2243 Acked Flush 0 Done Sent Done false
6093 2244 Acked Write 4 Done Sent Done false
6094 2245 Acked Write 4 Done Sent Done false
6095 2246 Acked Write 4 Done Sent Done false
6096 2247 Acked Write 2 Done Sent Done false
6097 2248 Acked Read 4 Done Sent Done false
6098 2249 Acked Write 4 Done Sent Done false
6099 2250 Acked Read 3 Done Sent Done false
6100 2251 Acked Write 9 Done Sent Done false
6101 2252 Acked Write 3 Done Sent Done false
6102 2253 Acked Write 1 Done Sent Done false
6103 2254 Acked Write 6 Done Sent Done false
6104 2255 Acked Read 3 Done Sent Done false
6105 2256 Acked Write 3 Done Sent Done false
6106 2257 Acked Write 2 Done Sent Done false
6107 2258 Acked Read 7 Done Sent Done false
6108 2259 Acked Flush 0 Done Sent Done false
6109 2260 Acked Read 10 Done Sent Done false
6110 2261 Acked Read 1 Done Sent Done false
6111 2262 Acked Read 1 Done Sent Done false
6112 2263 Acked Write 3 Done Sent Done false
6113 2264 Acked Write 5 Done Sent Done false
6114 2265 Acked Write 3 Done Sent Done false
6115 2266 Acked Flush 0 Done Sent Done false
6116 2267 Acked Read 10 Done Sent Done false
6117 2268 Acked Read 5 Done Sent Done false
6118 2269 Acked Flush 0 Done Sent Done false
6119 2270 Acked Write 7 Done Sent Done false
6120 2271 Acked Read 7 Done Sent Done false
6121 2272 Acked Read 10 Done Sent Done false
6122 2273 Acked Write 8 Done Sent Done false
6123 2274 Acked Read 4 Done Sent Done false
6124 2275 Acked Write 9 Done Sent Done false
6125 2276 Acked Write 8 Done Sent Done false
6126 2277 Acked Write 9 Done Sent Done false
6127 2278 Acked Read 2 Done Sent Done false
6128 2279 Acked Write 2 Done Sent Done false
6129 2280 Acked Read 6 Done Sent Done false
6130 2281 Acked Read 9 Done Sent Done false
6131 2282 Acked Read 10 Done Sent Done false
6132 2283 Acked Read 9 Done Sent Done false
6133 2284 Acked Read 2 Done Sent Done false
6134 2285 Acked Read 9 Done Sent Done false
6135 2286 Acked Read 4 Done Sent Done false
6136 2287 Acked Write 9 Done Sent Done false
6137 2288 Acked Read 3 Done Sent Done false
6138 2289 Acked Read 10 Done Sent Done false
6139 2290 Acked Write 2 Done Sent Done false
6140 2291 Acked Write 10 Done Sent Done false
6141 2292 Acked Flush 0 Done Sent Done false
6142 2293 Acked Read 3 Done Sent Done false
6143 2294 Acked Write 3 Done Sent Done false
6144 2295 Acked Read 4 Done Sent Done false
6145 2296 Acked Read 2 Done Sent Done false
6146 2297 Acked Write 3 Done Sent Done false
6147 2298 Acked Read 5 Done Sent Done false
6148 2299 Acked Read 8 Done Sent Done false
6149 2300 Acked Read 3 Done Sent Done false
6150 2301 Acked Read 1 Done Sent Done false
6151 2302 Acked Read 6 Done Sent Done false
6152 2303 Acked Write 2 Done Sent Done false
6153 2304 Acked Write 5 Done Sent Done false
6154 2305 Acked Write 9 Done Sent Done false
6155 2306 Acked Read 3 Done Sent Done false
6156 2307 Acked Write 5 Done Sent Done false
6157 2308 Acked Read 8 Done Sent Done false
6158 2309 Acked Write 4 Done Sent Done false
6159 2310 Acked Read 2 Done Sent Done false
6160 2311 Acked Read 1 Done Sent Done false
6161 2312 Acked Read 6 Done Sent Done false
6162 2313 Acked Read 8 Done Sent Done false
6163 2314 Acked Read 7 Done Sent Done false
6164 2315 Acked Read 5 Done Sent Done false
6165 2316 Acked Write 7 Done Sent Done false
6166 2317 Acked Write 2 Done Sent Done false
6167 2318 Acked Write 7 Done Sent Done false
6168 2319 Acked Write 10 Done Sent Done false
6169 2320 Acked Read 10 Done Sent Done false
6170 2321 Acked Write 2 Done Sent Done false
6171 2322 Acked Write 10 Done Sent Done false
6172 2323 Acked Write 1 Done Sent Done false
6173 2324 Acked Read 4 Done Sent Done false
6174 2325 Acked Read 5 Done Sent Done false
6175 2326 Acked Write 6 Done Sent Done false
6176 2327 Acked Read 10 Done Sent Done false
6177 2328 Acked Write 10 Done Sent Done false
6178 2329 Acked Read 6 Done Sent Done false
6179 2330 Acked Read 6 Done Sent Done false
6180 2331 Acked Write 5 Done Sent Done false
6181 2332 Acked Write 5 Done Sent Done false
6182 2333 Acked Write 6 Done Sent Done false
6183 2334 Acked Read 3 Done Sent Done false
6184 2335 Acked Flush 0 Done Sent Done false
6185 2336 Acked Read 4 Done Sent Done false
6186 2337 Acked Flush 0 Done Sent Done false
6187 2338 Acked Write 1 Done Sent Done false
6188 2339 Acked Flush 0 Done Sent Done false
6189 2340 Acked Read 1 Done Sent Done false
6190 2341 Acked Write 9 Done Sent Done false
6191 2342 Acked Write 9 Done Sent Done false
6192 2343 Acked Read 7 Done Sent Done false
6193 2344 Acked Write 6 Done Sent Done false
6194 2345 Acked Write 10 Done Sent Done false
6195 2346 Acked Read 3 Done Sent Done false
6196 2347 Acked Write 2 Done Sent Done false
6197 2348 Acked Read 3 Done Sent Done false
6198 2349 Acked Read 5 Done Sent Done false
6199 2350 Acked Write 5 Done Sent Done false
6200 2351 Acked Write 1 Done Sent Done false
6201 2352 Acked Write 4 Done Sent Done false
6202 2353 Acked Read 10 Done Sent Done false
6203 2354 Acked Read 7 Done Sent Done false
6204 2355 Acked Write 5 Done Sent Done false
6205 2356 Acked Read 2 Done Sent Done false
6206 2357 Acked Write 7 Done Sent Done false
6207 2358 Acked Write 7 Done Sent Done false
6208 2359 Acked Write 9 Done Sent Done false
6209 2360 Acked Read 5 Done Sent Done false
6210 2361 Acked Write 2 Done Sent Done false
6211 2362 Acked Flush 0 Done Sent Done false
6212 2363 Acked Read 10 Done Sent Done false
6213 2364 Acked Read 3 Done Sent Done false
6214 2365 Acked Read 10 Done Sent Done false
6215 2366 Acked Read 1 Done Sent Done false
6216 2367 Acked Write 10 Done Sent Done false
6217 2368 Acked Read 7 Done Sent Done false
6218 2369 Acked Read 5 Done Sent Done false
6219 2370 Acked Write 6 Done Sent Done false
6220 2371 Acked Read 10 Done Sent Done false
6221 2372 Acked Write 4 Done Sent Done false
6222 2373 Acked Read 6 Done Sent Done false
6223 2374 Acked Write 10 Done Sent Done false
6224 2375 Acked Read 3 Done Sent Done false
6225 2376 Acked Read 10 Done Sent Done false
6226 2377 Acked Read 4 Done Sent Done false
6227 2378 Acked Flush 0 Done Sent Done false
6228 2379 Acked Read 6 Done Sent Done false
6229 2380 Acked Read 2 Done Sent Done false
6230 2381 Acked Flush 0 Done Sent Done false
6231 2382 Acked Read 2 Done Sent Done false
6232 2383 Acked Write 8 Done Sent Done false
6233 2384 Acked Flush 0 Done Sent Done false
6234 2385 Acked Write 7 Done Sent Done false
6235 2386 Acked Write 8 Done Sent Done false
6236 2387 Acked Write 3 Done Sent Done false
6237 2388 Acked Write 4 Done Sent Done false
6238 2389 Acked Write 10 Done Sent Done false
6239 2390 Acked Write 6 Done Sent Done false
6240 2391 Acked Read 5 Done Sent Done false
6241 2392 Acked Flush 0 Done Sent Done false
6242 2393 Acked Read 3 Done Sent Done false
6243 2394 Acked Write 6 Done Sent Done false
6244 2395 Acked Write 9 Done Sent Done false
6245 2396 Acked Write 3 Done Sent Done false
6246 2397 Acked Write 10 Done Sent Done false
6247 2398 Acked Read 8 Done Sent Done false
6248 2399 Acked Write 5 Done Sent Done false
6249 2400 Acked Write 9 Done Sent Done false
6250 2401 Acked Write 5 Done Sent Done false
6251 2402 Acked Write 9 Done Sent Done false
6252 2403 Acked Write 9 Done Sent Done false
6253 2404 Acked Read 1 Done Sent Done false
6254 2405 Acked Write 5 Done Sent Done false
6255 2406 Acked Read 7 Done Sent Done false
6256 2407 Acked Read 5 Done Sent Done false
6257 2408 Acked Read 2 Done Sent Done false
6258 2409 Acked Write 1 Done Sent Done false
6259 2410 Acked Read 4 Done Sent Done false
6260 2411 Acked Read 10 Done Sent Done false
6261 2412 Acked Write 1 Done Sent Done false
6262 2413 Acked Flush 0 Done Sent Done false
6263 2414 Acked Read 10 Done Sent Done false
6264 2415 Acked Flush 0 Done Sent Done false
6265 2416 Acked Read 4 Done Sent Done false
6266 2417 Acked Write 7 Done Sent Done false
6267 2418 Acked Read 9 Done Sent Done false
6268 2419 Acked Write 3 Done Sent Done false
6269 2420 Acked Write 10 Done Sent Done false
6270 2421 Acked Write 4 Done Sent Done false
6271 2422 Acked Write 4 Done Sent Done false
6272 2423 Acked Read 5 Done Sent Done false
6273 2424 Acked Read 8 Done Sent Done false
6274 2425 Acked Write 9 Done Sent Done false
6275 2426 Acked Flush 0 Done Sent Done false
6276 2427 Acked Flush 0 Done Sent Done false
6277 2428 Acked Read 2 Done Sent Done false
6278 2429 Acked Write 4 Done Sent Done false
6279 2430 Acked Read 5 Done Sent Done false
6280 2431 Acked Write 7 Done Sent Done false
6281 2432 Acked Read 1 Done Sent Done false
6282 2433 Acked Read 7 Done Sent Done false
6283 2434 Acked Read 1 Done Sent Done false
6284 2435 Acked Write 4 Done Sent Done false
6285 2436 Acked Write 7 Done Sent Done false
6286 2437 Acked Write 5 Done Sent Done false
6287 2438 Acked Read 5 Done Sent Done false
6288 2439 Acked Read 7 Done Sent Done false
6289 2440 Acked Read 7 Done Sent Done false
6290 2441 Acked Read 9 Done Sent Done false
6291 2442 Acked Read 2 Done Sent Done false
6292 2443 Acked Read 4 Done Sent Done false
6293 2444 Acked Write 4 Done Sent Done false
6294 2445 Acked Read 4 Done Sent Done false
6295 2446 Acked Read 7 Done Sent Done false
6296 2447 Acked Write 6 Done Sent Done false
6297 2448 Acked Write 4 Done Sent Done false
6298 2449 Acked Flush 0 Done Sent Done false
6299 2450 Acked Read 6 Done Sent Done false
6300 2451 Acked Write 8 Done Sent Done false
6301 2452 Acked Read 5 Done Sent Done false
6302 2453 Acked Read 4 Done Sent Done false
6303 2454 Acked Write 6 Done Sent Done false
6304 2455 Acked Read 8 Done Sent Done false
6305 2456 Acked Read 5 Done Sent Done false
6306 2457 Acked Write 2 Done Sent Done false
6307 2458 Acked Write 4 Done Sent Done false
6308 2459 Acked Write 5 Done Sent Done false
6309 2460 Acked Write 9 Done Sent Done false
6310 2461 Acked Write 6 Done Sent Done false
6311 2462 Acked Read 2 Done Sent Done false
6312 2463 Acked Read 6 Done Sent Done false
6313 2464 Acked Read 6 Done Sent Done false
6314 2465 Acked Write 2 Done Sent Done false
6315 2466 Acked Flush 0 Done Sent Done false
6316 2467 Acked Write 10 Done Sent Done false
6317 2468 Acked Read 9 Done Sent Done false
6318 2469 Acked Write 3 Done Sent Done false
6319 2470 Acked Read 9 Done Sent Done false
6320 2471 Acked Read 5 Done Sent Done false
6321 2472 Acked Flush 0 Done Sent Done false
6322 2473 Acked Write 4 Done Sent Done false
6323 2474 Acked Write 9 Done Sent Done false
6324 2475 Acked Write 1 Done Sent Done false
6325 2476 Acked Read 7 Done Sent Done false
6326 2477 Acked Write 7 Done Sent Done false
6327 2478 Acked Read 8 Done Sent Done false
6328 2479 Acked Write 5 Done Sent Done false
6329 2480 Acked Read 7 Done Sent Done false
6330 2481 Acked Read 5 Done Sent Done false
6331 2482 Acked Write 10 Done Sent Done false
6332 2483 Acked Write 9 Done Sent Done false
6333 2484 Acked Write 2 Done Sent Done false
6334 2485 Acked Write 10 Done Sent Done false
6335 2486 Acked Read 4 Done Sent Done false
6336 2487 Acked Write 4 Done Sent Done false
6337 2488 Acked Read 1 Done Sent Done false
6338 2489 Acked Read 2 Done Sent Done false
6339 2490 Acked Read 5 Done Sent Done false
6340 2491 Acked Read 8 Done Sent Done false
6341 2492 Acked Read 2 Done Sent Done false
6342 2493 Acked Read 5 Done Sent Done false
6343 2494 Acked Read 10 Done Sent Done false
6344 2495 Acked Write 7 Done Sent Done false
6345 2496 Acked Write 8 Done Sent Done false
6346 2497 Acked Write 2 Done Sent Done false
6347 2498 Acked Read 3 Done Sent Done false
6348 2499 Acked Read 5 Done Sent Done false
6349 2500 Acked Write 5 Done Sent Done false
6350 2501 Acked Write 5 Done Sent Done false
6351 2502 Acked Read 7 Done Sent Done false
6352 2503 Acked Write 5 Done Sent Done false
6353 2504 Acked Write 5 Done Sent Done false
6354 2505 Acked Write 9 Done Sent Done false
6355 2506 Acked Flush 0 Done Sent Done false
6356 STATES DS:0 DS:1 DS:2 TOTAL
6357 Sent 0 301 0 301
6358 Done 301 0 301 602
6359 Skipped 0 0 0 0
6360 Error 0 0 0 0
6361 Last Flush: Some(JobId(2506)) Some(JobId(2205)) Some(JobId(2506))
6362 Downstairs last five completed: 2205 2204 2203 2202 2201
6363 Upstairs last five completed: 2506 2505 2504 2503 2502
6364 CLIENT: Up:0 ds:301 act:2
63652025-02-28T06:02:21.627ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
63662025-02-28T06:02:22.468ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
63672025-02-28T06:02:22.884ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
63682025-02-28T06:02:22.884ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
63692025-02-28T06:02:22.884ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
63702025-02-28T06:02:22.884ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
63712025-02-28T06:02:22.884ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
6372 [8820] got start action
6373 [8820] I am going to restart
6374 Starting downstairs at port 8820
6375 Make output file at "/tmp/dsc/downstairs-8820.txt"
6376 [8820][1] reports Starting
6377 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1216)
6378 [8820][1] reports Running
63792025-02-28T06:02:23.469ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
63802025-02-28T06:02:24.471ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
6381 ----------------------------------------------------------------
6382 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6383 States: Active Offline Active
6384 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6385 2206 Acked Write 4 Done Sent Done false
6386 2207 Acked Read 8 Done Sent Done false
6387 2208 Acked Read 10 Done Sent Done false
6388 2209 Acked Read 8 Done Sent Done false
6389 2210 Acked Write 6 Done Sent Done false
6390 2211 Acked Flush 0 Done Sent Done false
6391 2212 Acked Write 1 Done Sent Done false
6392 2213 Acked Read 9 Done Sent Done false
6393 2214 Acked Write 9 Done Sent Done false
6394 2215 Acked Read 6 Done Sent Done false
6395 2216 Acked Read 7 Done Sent Done false
6396 2217 Acked Write 6 Done Sent Done false
6397 2218 Acked Read 9 Done Sent Done false
6398 2219 Acked Read 5 Done Sent Done false
6399 2220 Acked Write 7 Done Sent Done false
6400 2221 Acked Read 6 Done Sent Done false
6401 2222 Acked Write 6 Done Sent Done false
6402 2223 Acked Read 4 Done Sent Done false
6403 2224 Acked Write 7 Done Sent Done false
6404 2225 Acked Read 4 Done Sent Done false
6405 2226 Acked Read 8 Done Sent Done false
6406 2227 Acked Write 8 Done Sent Done false
6407 2228 Acked Read 1 Done Sent Done false
6408 2229 Acked Read 2 Done Sent Done false
6409 2230 Acked Write 5 Done Sent Done false
6410 2231 Acked Read 9 Done Sent Done false
6411 2232 Acked Read 6 Done Sent Done false
6412 2233 Acked Write 7 Done Sent Done false
6413 2234 Acked Read 7 Done Sent Done false
6414 2235 Acked Write 8 Done Sent Done false
6415 2236 Acked Flush 0 Done Sent Done false
6416 2237 Acked Read 2 Done Sent Done false
6417 2238 Acked Flush 0 Done Sent Done false
6418 2239 Acked Read 1 Done Sent Done false
6419 2240 Acked Write 7 Done Sent Done false
6420 2241 Acked Flush 0 Done Sent Done false
6421 2242 Acked Read 7 Done Sent Done false
6422 2243 Acked Flush 0 Done Sent Done false
6423 2244 Acked Write 4 Done Sent Done false
6424 2245 Acked Write 4 Done Sent Done false
6425 2246 Acked Write 4 Done Sent Done false
6426 2247 Acked Write 2 Done Sent Done false
6427 2248 Acked Read 4 Done Sent Done false
6428 2249 Acked Write 4 Done Sent Done false
6429 2250 Acked Read 3 Done Sent Done false
6430 2251 Acked Write 9 Done Sent Done false
6431 2252 Acked Write 3 Done Sent Done false
6432 2253 Acked Write 1 Done Sent Done false
6433 2254 Acked Write 6 Done Sent Done false
6434 2255 Acked Read 3 Done Sent Done false
6435 2256 Acked Write 3 Done Sent Done false
6436 2257 Acked Write 2 Done Sent Done false
6437 2258 Acked Read 7 Done Sent Done false
6438 2259 Acked Flush 0 Done Sent Done false
6439 2260 Acked Read 10 Done Sent Done false
6440 2261 Acked Read 1 Done Sent Done false
6441 2262 Acked Read 1 Done Sent Done false
6442 2263 Acked Write 3 Done Sent Done false
6443 2264 Acked Write 5 Done Sent Done false
6444 2265 Acked Write 3 Done Sent Done false
6445 2266 Acked Flush 0 Done Sent Done false
6446 2267 Acked Read 10 Done Sent Done false
6447 2268 Acked Read 5 Done Sent Done false
6448 2269 Acked Flush 0 Done Sent Done false
6449 2270 Acked Write 7 Done Sent Done false
6450 2271 Acked Read 7 Done Sent Done false
6451 2272 Acked Read 10 Done Sent Done false
6452 2273 Acked Write 8 Done Sent Done false
6453 2274 Acked Read 4 Done Sent Done false
6454 2275 Acked Write 9 Done Sent Done false
6455 2276 Acked Write 8 Done Sent Done false
6456 2277 Acked Write 9 Done Sent Done false
6457 2278 Acked Read 2 Done Sent Done false
6458 2279 Acked Write 2 Done Sent Done false
6459 2280 Acked Read 6 Done Sent Done false
6460 2281 Acked Read 9 Done Sent Done false
6461 2282 Acked Read 10 Done Sent Done false
6462 2283 Acked Read 9 Done Sent Done false
6463 2284 Acked Read 2 Done Sent Done false
6464 2285 Acked Read 9 Done Sent Done false
6465 2286 Acked Read 4 Done Sent Done false
6466 2287 Acked Write 9 Done Sent Done false
6467 2288 Acked Read 3 Done Sent Done false
6468 2289 Acked Read 10 Done Sent Done false
6469 2290 Acked Write 2 Done Sent Done false
6470 2291 Acked Write 10 Done Sent Done false
6471 2292 Acked Flush 0 Done Sent Done false
6472 2293 Acked Read 3 Done Sent Done false
6473 2294 Acked Write 3 Done Sent Done false
6474 2295 Acked Read 4 Done Sent Done false
6475 2296 Acked Read 2 Done Sent Done false
6476 2297 Acked Write 3 Done Sent Done false
6477 2298 Acked Read 5 Done Sent Done false
6478 2299 Acked Read 8 Done Sent Done false
6479 2300 Acked Read 3 Done Sent Done false
6480 2301 Acked Read 1 Done Sent Done false
6481 2302 Acked Read 6 Done Sent Done false
6482 2303 Acked Write 2 Done Sent Done false
6483 2304 Acked Write 5 Done Sent Done false
6484 2305 Acked Write 9 Done Sent Done false
6485 2306 Acked Read 3 Done Sent Done false
6486 2307 Acked Write 5 Done Sent Done false
6487 2308 Acked Read 8 Done Sent Done false
6488 2309 Acked Write 4 Done Sent Done false
6489 2310 Acked Read 2 Done Sent Done false
6490 2311 Acked Read 1 Done Sent Done false
6491 2312 Acked Read 6 Done Sent Done false
6492 2313 Acked Read 8 Done Sent Done false
6493 2314 Acked Read 7 Done Sent Done false
6494 2315 Acked Read 5 Done Sent Done false
6495 2316 Acked Write 7 Done Sent Done false
6496 2317 Acked Write 2 Done Sent Done false
6497 2318 Acked Write 7 Done Sent Done false
6498 2319 Acked Write 10 Done Sent Done false
6499 2320 Acked Read 10 Done Sent Done false
6500 2321 Acked Write 2 Done Sent Done false
6501 2322 Acked Write 10 Done Sent Done false
6502 2323 Acked Write 1 Done Sent Done false
6503 2324 Acked Read 4 Done Sent Done false
6504 2325 Acked Read 5 Done Sent Done false
6505 2326 Acked Write 6 Done Sent Done false
6506 2327 Acked Read 10 Done Sent Done false
6507 2328 Acked Write 10 Done Sent Done false
6508 2329 Acked Read 6 Done Sent Done false
6509 2330 Acked Read 6 Done Sent Done false
6510 2331 Acked Write 5 Done Sent Done false
6511 2332 Acked Write 5 Done Sent Done false
6512 2333 Acked Write 6 Done Sent Done false
6513 2334 Acked Read 3 Done Sent Done false
6514 2335 Acked Flush 0 Done Sent Done false
6515 2336 Acked Read 4 Done Sent Done false
6516 2337 Acked Flush 0 Done Sent Done false
6517 2338 Acked Write 1 Done Sent Done false
6518 2339 Acked Flush 0 Done Sent Done false
6519 2340 Acked Read 1 Done Sent Done false
6520 2341 Acked Write 9 Done Sent Done false
6521 2342 Acked Write 9 Done Sent Done false
6522 2343 Acked Read 7 Done Sent Done false
6523 2344 Acked Write 6 Done Sent Done false
6524 2345 Acked Write 10 Done Sent Done false
6525 2346 Acked Read 3 Done Sent Done false
6526 2347 Acked Write 2 Done Sent Done false
6527 2348 Acked Read 3 Done Sent Done false
6528 2349 Acked Read 5 Done Sent Done false
6529 2350 Acked Write 5 Done Sent Done false
6530 2351 Acked Write 1 Done Sent Done false
6531 2352 Acked Write 4 Done Sent Done false
6532 2353 Acked Read 10 Done Sent Done false
6533 2354 Acked Read 7 Done Sent Done false
6534 2355 Acked Write 5 Done Sent Done false
6535 2356 Acked Read 2 Done Sent Done false
6536 2357 Acked Write 7 Done Sent Done false
6537 2358 Acked Write 7 Done Sent Done false
6538 2359 Acked Write 9 Done Sent Done false
6539 2360 Acked Read 5 Done Sent Done false
6540 2361 Acked Write 2 Done Sent Done false
6541 2362 Acked Flush 0 Done Sent Done false
6542 2363 Acked Read 10 Done Sent Done false
6543 2364 Acked Read 3 Done Sent Done false
6544 2365 Acked Read 10 Done Sent Done false
6545 2366 Acked Read 1 Done Sent Done false
6546 2367 Acked Write 10 Done Sent Done false
6547 2368 Acked Read 7 Done Sent Done false
6548 2369 Acked Read 5 Done Sent Done false
6549 2370 Acked Write 6 Done Sent Done false
6550 2371 Acked Read 10 Done Sent Done false
6551 2372 Acked Write 4 Done Sent Done false
6552 2373 Acked Read 6 Done Sent Done false
6553 2374 Acked Write 10 Done Sent Done false
6554 2375 Acked Read 3 Done Sent Done false
6555 2376 Acked Read 10 Done Sent Done false
6556 2377 Acked Read 4 Done Sent Done false
6557 2378 Acked Flush 0 Done Sent Done false
6558 2379 Acked Read 6 Done Sent Done false
6559 2380 Acked Read 2 Done Sent Done false
6560 2381 Acked Flush 0 Done Sent Done false
6561 2382 Acked Read 2 Done Sent Done false
6562 2383 Acked Write 8 Done Sent Done false
6563 2384 Acked Flush 0 Done Sent Done false
6564 2385 Acked Write 7 Done Sent Done false
6565 2386 Acked Write 8 Done Sent Done false
6566 2387 Acked Write 3 Done Sent Done false
6567 2388 Acked Write 4 Done Sent Done false
6568 2389 Acked Write 10 Done Sent Done false
6569 2390 Acked Write 6 Done Sent Done false
6570 2391 Acked Read 5 Done Sent Done false
6571 2392 Acked Flush 0 Done Sent Done false
6572 2393 Acked Read 3 Done Sent Done false
6573 2394 Acked Write 6 Done Sent Done false
6574 2395 Acked Write 9 Done Sent Done false
6575 2396 Acked Write 3 Done Sent Done false
6576 2397 Acked Write 10 Done Sent Done false
6577 2398 Acked Read 8 Done Sent Done false
6578 2399 Acked Write 5 Done Sent Done false
6579 2400 Acked Write 9 Done Sent Done false
6580 2401 Acked Write 5 Done Sent Done false
6581 2402 Acked Write 9 Done Sent Done false
6582 2403 Acked Write 9 Done Sent Done false
6583 2404 Acked Read 1 Done Sent Done false
6584 2405 Acked Write 5 Done Sent Done false
6585 2406 Acked Read 7 Done Sent Done false
6586 2407 Acked Read 5 Done Sent Done false
6587 2408 Acked Read 2 Done Sent Done false
6588 2409 Acked Write 1 Done Sent Done false
6589 2410 Acked Read 4 Done Sent Done false
6590 2411 Acked Read 10 Done Sent Done false
6591 2412 Acked Write 1 Done Sent Done false
6592 2413 Acked Flush 0 Done Sent Done false
6593 2414 Acked Read 10 Done Sent Done false
6594 2415 Acked Flush 0 Done Sent Done false
6595 2416 Acked Read 4 Done Sent Done false
6596 2417 Acked Write 7 Done Sent Done false
6597 2418 Acked Read 9 Done Sent Done false
6598 2419 Acked Write 3 Done Sent Done false
6599 2420 Acked Write 10 Done Sent Done false
6600 2421 Acked Write 4 Done Sent Done false
6601 2422 Acked Write 4 Done Sent Done false
6602 2423 Acked Read 5 Done Sent Done false
6603 2424 Acked Read 8 Done Sent Done false
6604 2425 Acked Write 9 Done Sent Done false
6605 2426 Acked Flush 0 Done Sent Done false
6606 2427 Acked Flush 0 Done Sent Done false
6607 2428 Acked Read 2 Done Sent Done false
6608 2429 Acked Write 4 Done Sent Done false
6609 2430 Acked Read 5 Done Sent Done false
6610 2431 Acked Write 7 Done Sent Done false
6611 2432 Acked Read 1 Done Sent Done false
6612 2433 Acked Read 7 Done Sent Done false
6613 2434 Acked Read 1 Done Sent Done false
6614 2435 Acked Write 4 Done Sent Done false
6615 2436 Acked Write 7 Done Sent Done false
6616 2437 Acked Write 5 Done Sent Done false
6617 2438 Acked Read 5 Done Sent Done false
6618 2439 Acked Read 7 Done Sent Done false
6619 2440 Acked Read 7 Done Sent Done false
6620 2441 Acked Read 9 Done Sent Done false
6621 2442 Acked Read 2 Done Sent Done false
6622 2443 Acked Read 4 Done Sent Done false
6623 2444 Acked Write 4 Done Sent Done false
6624 2445 Acked Read 4 Done Sent Done false
6625 2446 Acked Read 7 Done Sent Done false
6626 2447 Acked Write 6 Done Sent Done false
6627 2448 Acked Write 4 Done Sent Done false
6628 2449 Acked Flush 0 Done Sent Done false
6629 2450 Acked Read 6 Done Sent Done false
6630 2451 Acked Write 8 Done Sent Done false
6631 2452 Acked Read 5 Done Sent Done false
6632 2453 Acked Read 4 Done Sent Done false
6633 2454 Acked Write 6 Done Sent Done false
6634 2455 Acked Read 8 Done Sent Done false
6635 2456 Acked Read 5 Done Sent Done false
6636 2457 Acked Write 2 Done Sent Done false
6637 2458 Acked Write 4 Done Sent Done false
6638 2459 Acked Write 5 Done Sent Done false
6639 2460 Acked Write 9 Done Sent Done false
6640 2461 Acked Write 6 Done Sent Done false
6641 2462 Acked Read 2 Done Sent Done false
6642 2463 Acked Read 6 Done Sent Done false
6643 2464 Acked Read 6 Done Sent Done false
6644 2465 Acked Write 2 Done Sent Done false
6645 2466 Acked Flush 0 Done Sent Done false
6646 2467 Acked Write 10 Done Sent Done false
6647 2468 Acked Read 9 Done Sent Done false
6648 2469 Acked Write 3 Done Sent Done false
6649 2470 Acked Read 9 Done Sent Done false
6650 2471 Acked Read 5 Done Sent Done false
6651 2472 Acked Flush 0 Done Sent Done false
6652 2473 Acked Write 4 Done Sent Done false
6653 2474 Acked Write 9 Done Sent Done false
6654 2475 Acked Write 1 Done Sent Done false
6655 2476 Acked Read 7 Done Sent Done false
6656 2477 Acked Write 7 Done Sent Done false
6657 2478 Acked Read 8 Done Sent Done false
6658 2479 Acked Write 5 Done Sent Done false
6659 2480 Acked Read 7 Done Sent Done false
6660 2481 Acked Read 5 Done Sent Done false
6661 2482 Acked Write 10 Done Sent Done false
6662 2483 Acked Write 9 Done Sent Done false
6663 2484 Acked Write 2 Done Sent Done false
6664 2485 Acked Write 10 Done Sent Done false
6665 2486 Acked Read 4 Done Sent Done false
6666 2487 Acked Write 4 Done Sent Done false
6667 2488 Acked Read 1 Done Sent Done false
6668 2489 Acked Read 2 Done Sent Done false
6669 2490 Acked Read 5 Done Sent Done false
6670 2491 Acked Read 8 Done Sent Done false
6671 2492 Acked Read 2 Done Sent Done false
6672 2493 Acked Read 5 Done Sent Done false
6673 2494 Acked Read 10 Done Sent Done false
6674 2495 Acked Write 7 Done Sent Done false
6675 2496 Acked Write 8 Done Sent Done false
6676 2497 Acked Write 2 Done Sent Done false
6677 2498 Acked Read 3 Done Sent Done false
6678 2499 Acked Read 5 Done Sent Done false
6679 2500 Acked Write 5 Done Sent Done false
6680 2501 Acked Write 5 Done Sent Done false
6681 2502 Acked Read 7 Done Sent Done false
6682 2503 Acked Write 5 Done Sent Done false
6683 2504 Acked Write 5 Done Sent Done false
6684 2505 Acked Write 9 Done Sent Done false
6685 2506 Acked Flush 0 Done Sent Done false
6686 STATES DS:0 DS:1 DS:2 TOTAL
6687 Sent 0 301 0 301
6688 Done 301 0 301 602
6689 Skipped 0 0 0 0
6690 Error 0 0 0 0
6691 Last Flush: Some(JobId(2506)) Some(JobId(2205)) Some(JobId(2506))
6692 Downstairs last five completed: 2205 2204 2203 2202 2201
6693 Upstairs last five completed: 2506 2505 2504 2503 2502
6694 CLIENT: Up:0 ds:301 act:2
66952025-02-28T06:02:25.630ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
66962025-02-28T06:02:26.475ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
66972025-02-28T06:02:26.975ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
66982025-02-28T06:02:27.477ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
66992025-02-28T06:02:28.479ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
6700 ----------------------------------------------------------------
6701 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6702 States: Active Offline Active
6703 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6704 2206 Acked Write 4 Done Sent Done false
6705 2207 Acked Read 8 Done Sent Done false
6706 2208 Acked Read 10 Done Sent Done false
6707 2209 Acked Read 8 Done Sent Done false
6708 2210 Acked Write 6 Done Sent Done false
6709 2211 Acked Flush 0 Done Sent Done false
6710 2212 Acked Write 1 Done Sent Done false
6711 2213 Acked Read 9 Done Sent Done false
6712 2214 Acked Write 9 Done Sent Done false
6713 2215 Acked Read 6 Done Sent Done false
6714 2216 Acked Read 7 Done Sent Done false
6715 2217 Acked Write 6 Done Sent Done false
6716 2218 Acked Read 9 Done Sent Done false
6717 2219 Acked Read 5 Done Sent Done false
6718 2220 Acked Write 7 Done Sent Done false
6719 2221 Acked Read 6 Done Sent Done false
6720 2222 Acked Write 6 Done Sent Done false
6721 2223 Acked Read 4 Done Sent Done false
6722 2224 Acked Write 7 Done Sent Done false
6723 2225 Acked Read 4 Done Sent Done false
6724 2226 Acked Read 8 Done Sent Done false
6725 2227 Acked Write 8 Done Sent Done false
6726 2228 Acked Read 1 Done Sent Done false
6727 2229 Acked Read 2 Done Sent Done false
6728 2230 Acked Write 5 Done Sent Done false
6729 2231 Acked Read 9 Done Sent Done false
6730 2232 Acked Read 6 Done Sent Done false
6731 2233 Acked Write 7 Done Sent Done false
6732 2234 Acked Read 7 Done Sent Done false
6733 2235 Acked Write 8 Done Sent Done false
6734 2236 Acked Flush 0 Done Sent Done false
6735 2237 Acked Read 2 Done Sent Done false
6736 2238 Acked Flush 0 Done Sent Done false
6737 2239 Acked Read 1 Done Sent Done false
6738 2240 Acked Write 7 Done Sent Done false
6739 2241 Acked Flush 0 Done Sent Done false
6740 2242 Acked Read 7 Done Sent Done false
6741 2243 Acked Flush 0 Done Sent Done false
6742 2244 Acked Write 4 Done Sent Done false
6743 2245 Acked Write 4 Done Sent Done false
6744 2246 Acked Write 4 Done Sent Done false
6745 2247 Acked Write 2 Done Sent Done false
6746 2248 Acked Read 4 Done Sent Done false
6747 2249 Acked Write 4 Done Sent Done false
6748 2250 Acked Read 3 Done Sent Done false
6749 2251 Acked Write 9 Done Sent Done false
6750 2252 Acked Write 3 Done Sent Done false
6751 2253 Acked Write 1 Done Sent Done false
6752 2254 Acked Write 6 Done Sent Done false
6753 2255 Acked Read 3 Done Sent Done false
6754 2256 Acked Write 3 Done Sent Done false
6755 2257 Acked Write 2 Done Sent Done false
6756 2258 Acked Read 7 Done Sent Done false
6757 2259 Acked Flush 0 Done Sent Done false
6758 2260 Acked Read 10 Done Sent Done false
6759 2261 Acked Read 1 Done Sent Done false
6760 2262 Acked Read 1 Done Sent Done false
6761 2263 Acked Write 3 Done Sent Done false
6762 2264 Acked Write 5 Done Sent Done false
6763 2265 Acked Write 3 Done Sent Done false
6764 2266 Acked Flush 0 Done Sent Done false
6765 2267 Acked Read 10 Done Sent Done false
6766 2268 Acked Read 5 Done Sent Done false
6767 2269 Acked Flush 0 Done Sent Done false
6768 2270 Acked Write 7 Done Sent Done false
6769 2271 Acked Read 7 Done Sent Done false
6770 2272 Acked Read 10 Done Sent Done false
6771 2273 Acked Write 8 Done Sent Done false
6772 2274 Acked Read 4 Done Sent Done false
6773 2275 Acked Write 9 Done Sent Done false
6774 2276 Acked Write 8 Done Sent Done false
6775 2277 Acked Write 9 Done Sent Done false
6776 2278 Acked Read 2 Done Sent Done false
6777 2279 Acked Write 2 Done Sent Done false
6778 2280 Acked Read 6 Done Sent Done false
6779 2281 Acked Read 9 Done Sent Done false
6780 2282 Acked Read 10 Done Sent Done false
6781 2283 Acked Read 9 Done Sent Done false
6782 2284 Acked Read 2 Done Sent Done false
6783 2285 Acked Read 9 Done Sent Done false
6784 2286 Acked Read 4 Done Sent Done false
6785 2287 Acked Write 9 Done Sent Done false
6786 2288 Acked Read 3 Done Sent Done false
6787 2289 Acked Read 10 Done Sent Done false
6788 2290 Acked Write 2 Done Sent Done false
6789 2291 Acked Write 10 Done Sent Done false
6790 2292 Acked Flush 0 Done Sent Done false
6791 2293 Acked Read 3 Done Sent Done false
6792 2294 Acked Write 3 Done Sent Done false
6793 2295 Acked Read 4 Done Sent Done false
6794 2296 Acked Read 2 Done Sent Done false
6795 2297 Acked Write 3 Done Sent Done false
6796 2298 Acked Read 5 Done Sent Done false
6797 2299 Acked Read 8 Done Sent Done false
6798 2300 Acked Read 3 Done Sent Done false
6799 2301 Acked Read 1 Done Sent Done false
6800 2302 Acked Read 6 Done Sent Done false
6801 2303 Acked Write 2 Done Sent Done false
6802 2304 Acked Write 5 Done Sent Done false
6803 2305 Acked Write 9 Done Sent Done false
6804 2306 Acked Read 3 Done Sent Done false
6805 2307 Acked Write 5 Done Sent Done false
6806 2308 Acked Read 8 Done Sent Done false
6807 2309 Acked Write 4 Done Sent Done false
6808 2310 Acked Read 2 Done Sent Done false
6809 2311 Acked Read 1 Done Sent Done false
6810 2312 Acked Read 6 Done Sent Done false
6811 2313 Acked Read 8 Done Sent Done false
6812 2314 Acked Read 7 Done Sent Done false
6813 2315 Acked Read 5 Done Sent Done false
6814 2316 Acked Write 7 Done Sent Done false
6815 2317 Acked Write 2 Done Sent Done false
6816 2318 Acked Write 7 Done Sent Done false
6817 2319 Acked Write 10 Done Sent Done false
6818 2320 Acked Read 10 Done Sent Done false
6819 2321 Acked Write 2 Done Sent Done false
6820 2322 Acked Write 10 Done Sent Done false
6821 2323 Acked Write 1 Done Sent Done false
6822 2324 Acked Read 4 Done Sent Done false
6823 2325 Acked Read 5 Done Sent Done false
6824 2326 Acked Write 6 Done Sent Done false
6825 2327 Acked Read 10 Done Sent Done false
6826 2328 Acked Write 10 Done Sent Done false
6827 2329 Acked Read 6 Done Sent Done false
6828 2330 Acked Read 6 Done Sent Done false
6829 2331 Acked Write 5 Done Sent Done false
6830 2332 Acked Write 5 Done Sent Done false
6831 2333 Acked Write 6 Done Sent Done false
6832 2334 Acked Read 3 Done Sent Done false
6833 2335 Acked Flush 0 Done Sent Done false
6834 2336 Acked Read 4 Done Sent Done false
6835 2337 Acked Flush 0 Done Sent Done false
6836 2338 Acked Write 1 Done Sent Done false
6837 2339 Acked Flush 0 Done Sent Done false
6838 2340 Acked Read 1 Done Sent Done false
6839 2341 Acked Write 9 Done Sent Done false
6840 2342 Acked Write 9 Done Sent Done false
6841 2343 Acked Read 7 Done Sent Done false
6842 2344 Acked Write 6 Done Sent Done false
6843 2345 Acked Write 10 Done Sent Done false
6844 2346 Acked Read 3 Done Sent Done false
6845 2347 Acked Write 2 Done Sent Done false
6846 2348 Acked Read 3 Done Sent Done false
6847 2349 Acked Read 5 Done Sent Done false
6848 2350 Acked Write 5 Done Sent Done false
6849 2351 Acked Write 1 Done Sent Done false
6850 2352 Acked Write 4 Done Sent Done false
6851 2353 Acked Read 10 Done Sent Done false
6852 2354 Acked Read 7 Done Sent Done false
6853 2355 Acked Write 5 Done Sent Done false
6854 2356 Acked Read 2 Done Sent Done false
6855 2357 Acked Write 7 Done Sent Done false
6856 2358 Acked Write 7 Done Sent Done false
6857 2359 Acked Write 9 Done Sent Done false
6858 2360 Acked Read 5 Done Sent Done false
6859 2361 Acked Write 2 Done Sent Done false
6860 2362 Acked Flush 0 Done Sent Done false
6861 2363 Acked Read 10 Done Sent Done false
6862 2364 Acked Read 3 Done Sent Done false
6863 2365 Acked Read 10 Done Sent Done false
6864 2366 Acked Read 1 Done Sent Done false
6865 2367 Acked Write 10 Done Sent Done false
6866 2368 Acked Read 7 Done Sent Done false
6867 2369 Acked Read 5 Done Sent Done false
6868 2370 Acked Write 6 Done Sent Done false
6869 2371 Acked Read 10 Done Sent Done false
6870 2372 Acked Write 4 Done Sent Done false
6871 2373 Acked Read 6 Done Sent Done false
6872 2374 Acked Write 10 Done Sent Done false
6873 2375 Acked Read 3 Done Sent Done false
6874 2376 Acked Read 10 Done Sent Done false
6875 2377 Acked Read 4 Done Sent Done false
6876 2378 Acked Flush 0 Done Sent Done false
6877 2379 Acked Read 6 Done Sent Done false
6878 2380 Acked Read 2 Done Sent Done false
6879 2381 Acked Flush 0 Done Sent Done false
6880 2382 Acked Read 2 Done Sent Done false
6881 2383 Acked Write 8 Done Sent Done false
6882 2384 Acked Flush 0 Done Sent Done false
6883 2385 Acked Write 7 Done Sent Done false
6884 2386 Acked Write 8 Done Sent Done false
6885 2387 Acked Write 3 Done Sent Done false
6886 2388 Acked Write 4 Done Sent Done false
6887 2389 Acked Write 10 Done Sent Done false
6888 2390 Acked Write 6 Done Sent Done false
6889 2391 Acked Read 5 Done Sent Done false
6890 2392 Acked Flush 0 Done Sent Done false
6891 2393 Acked Read 3 Done Sent Done false
6892 2394 Acked Write 6 Done Sent Done false
6893 2395 Acked Write 9 Done Sent Done false
6894 2396 Acked Write 3 Done Sent Done false
6895 2397 Acked Write 10 Done Sent Done false
6896 2398 Acked Read 8 Done Sent Done false
6897 2399 Acked Write 5 Done Sent Done false
6898 2400 Acked Write 9 Done Sent Done false
6899 2401 Acked Write 5 Done Sent Done false
6900 2402 Acked Write 9 Done Sent Done false
6901 2403 Acked Write 9 Done Sent Done false
6902 2404 Acked Read 1 Done Sent Done false
6903 2405 Acked Write 5 Done Sent Done false
6904 2406 Acked Read 7 Done Sent Done false
6905 2407 Acked Read 5 Done Sent Done false
6906 2408 Acked Read 2 Done Sent Done false
6907 2409 Acked Write 1 Done Sent Done false
6908 2410 Acked Read 4 Done Sent Done false
6909 2411 Acked Read 10 Done Sent Done false
6910 2412 Acked Write 1 Done Sent Done false
6911 2413 Acked Flush 0 Done Sent Done false
6912 2414 Acked Read 10 Done Sent Done false
6913 2415 Acked Flush 0 Done Sent Done false
6914 2416 Acked Read 4 Done Sent Done false
6915 2417 Acked Write 7 Done Sent Done false
6916 2418 Acked Read 9 Done Sent Done false
6917 2419 Acked Write 3 Done Sent Done false
6918 2420 Acked Write 10 Done Sent Done false
6919 2421 Acked Write 4 Done Sent Done false
6920 2422 Acked Write 4 Done Sent Done false
6921 2423 Acked Read 5 Done Sent Done false
6922 2424 Acked Read 8 Done Sent Done false
6923 2425 Acked Write 9 Done Sent Done false
6924 2426 Acked Flush 0 Done Sent Done false
6925 2427 Acked Flush 0 Done Sent Done false
6926 2428 Acked Read 2 Done Sent Done false
6927 2429 Acked Write 4 Done Sent Done false
6928 2430 Acked Read 5 Done Sent Done false
6929 2431 Acked Write 7 Done Sent Done false
6930 2432 Acked Read 1 Done Sent Done false
6931 2433 Acked Read 7 Done Sent Done false
6932 2434 Acked Read 1 Done Sent Done false
6933 2435 Acked Write 4 Done Sent Done false
6934 2436 Acked Write 7 Done Sent Done false
6935 2437 Acked Write 5 Done Sent Done false
6936 2438 Acked Read 5 Done Sent Done false
6937 2439 Acked Read 7 Done Sent Done false
6938 2440 Acked Read 7 Done Sent Done false
6939 2441 Acked Read 9 Done Sent Done false
6940 2442 Acked Read 2 Done Sent Done false
6941 2443 Acked Read 4 Done Sent Done false
6942 2444 Acked Write 4 Done Sent Done false
6943 2445 Acked Read 4 Done Sent Done false
6944 2446 Acked Read 7 Done Sent Done false
6945 2447 Acked Write 6 Done Sent Done false
6946 2448 Acked Write 4 Done Sent Done false
6947 2449 Acked Flush 0 Done Sent Done false
6948 2450 Acked Read 6 Done Sent Done false
6949 2451 Acked Write 8 Done Sent Done false
6950 2452 Acked Read 5 Done Sent Done false
6951 2453 Acked Read 4 Done Sent Done false
6952 2454 Acked Write 6 Done Sent Done false
6953 2455 Acked Read 8 Done Sent Done false
6954 2456 Acked Read 5 Done Sent Done false
6955 2457 Acked Write 2 Done Sent Done false
6956 2458 Acked Write 4 Done Sent Done false
6957 2459 Acked Write 5 Done Sent Done false
6958 2460 Acked Write 9 Done Sent Done false
6959 2461 Acked Write 6 Done Sent Done false
6960 2462 Acked Read 2 Done Sent Done false
6961 2463 Acked Read 6 Done Sent Done false
6962 2464 Acked Read 6 Done Sent Done false
6963 2465 Acked Write 2 Done Sent Done false
6964 2466 Acked Flush 0 Done Sent Done false
6965 2467 Acked Write 10 Done Sent Done false
6966 2468 Acked Read 9 Done Sent Done false
6967 2469 Acked Write 3 Done Sent Done false
6968 2470 Acked Read 9 Done Sent Done false
6969 2471 Acked Read 5 Done Sent Done false
6970 2472 Acked Flush 0 Done Sent Done false
6971 2473 Acked Write 4 Done Sent Done false
6972 2474 Acked Write 9 Done Sent Done false
6973 2475 Acked Write 1 Done Sent Done false
6974 2476 Acked Read 7 Done Sent Done false
6975 2477 Acked Write 7 Done Sent Done false
6976 2478 Acked Read 8 Done Sent Done false
6977 2479 Acked Write 5 Done Sent Done false
6978 2480 Acked Read 7 Done Sent Done false
6979 2481 Acked Read 5 Done Sent Done false
6980 2482 Acked Write 10 Done Sent Done false
6981 2483 Acked Write 9 Done Sent Done false
6982 2484 Acked Write 2 Done Sent Done false
6983 2485 Acked Write 10 Done Sent Done false
6984 2486 Acked Read 4 Done Sent Done false
6985 2487 Acked Write 4 Done Sent Done false
6986 2488 Acked Read 1 Done Sent Done false
6987 2489 Acked Read 2 Done Sent Done false
6988 2490 Acked Read 5 Done Sent Done false
6989 2491 Acked Read 8 Done Sent Done false
6990 2492 Acked Read 2 Done Sent Done false
6991 2493 Acked Read 5 Done Sent Done false
6992 2494 Acked Read 10 Done Sent Done false
6993 2495 Acked Write 7 Done Sent Done false
6994 2496 Acked Write 8 Done Sent Done false
6995 2497 Acked Write 2 Done Sent Done false
6996 2498 Acked Read 3 Done Sent Done false
6997 2499 Acked Read 5 Done Sent Done false
6998 2500 Acked Write 5 Done Sent Done false
6999 2501 Acked Write 5 Done Sent Done false
7000 2502 Acked Read 7 Done Sent Done false
7001 2503 Acked Write 5 Done Sent Done false
7002 2504 Acked Write 5 Done Sent Done false
7003 2505 Acked Write 9 Done Sent Done false
7004 2506 Acked Flush 0 Done Sent Done false
7005 STATES DS:0 DS:1 DS:2 TOTAL
7006 Sent 0 301 0 301
7007 Done 301 0 301 602
7008 Skipped 0 0 0 0
7009 Error 0 0 0 0
7010 Last Flush: Some(JobId(2506)) Some(JobId(2205)) Some(JobId(2506))
7011 Downstairs last five completed: 2205 2204 2203 2202 2201
7012 Upstairs last five completed: 2506 2505 2504 2503 2502
7013 CLIENT: Up:0 ds:301 act:2
70142025-02-28T06:02:29.632ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70152025-02-28T06:02:30.483ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70162025-02-28T06:02:31.483ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70172025-02-28T06:02:32.486ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70182025-02-28T06:02:32.885ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70192025-02-28T06:02:32.885ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53284) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70202025-02-28T06:02:32.887ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70212025-02-28T06:02:32.887ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70222025-02-28T06:02:32.887ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70232025-02-28T06:02:32.887ZINFOcrucible: send last flush ID to this DS: Some(JobId(2205)) = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70242025-02-28T06:02:32.887ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70252025-02-28T06:02:32.887ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2205)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70262025-02-28T06:02:32.888ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2205)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
7027 ----------------------------------------------------------------
7028 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7029 Downstairs last five completed: 2506 2505 2504 2503 2502
7030 Upstairs last five completed: 2506 2505 2504 2503 2502
7031 CLIENT: Up:0 ds:0 act:3
7032 Replay: All jobs finished
7033 Main task has work to do, go find it
7034 got dsc Stop(0)
7035 stop 0
7036 Feb 28 06:02:33.133 [8810] Got stop action so:false kr:false
7037 INFO request completed, latency_us: 321, response_code: 204, uri: /stop/cid/0, method: POST, req_id: da63dc11-7cab-418a-b755-4aae559de780, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
70382025-02-28T06:02:33.134ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70392025-02-28T06:02:33.134ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70402025-02-28T06:02:33.134ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
70412025-02-28T06:02:33.134ZINFOcrucible: client task is exiting = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
7042 Feb 28 06:02:33.134 INFO request completed, latency_us: 148, response_code: 200, uri: /state/cid/0, method: GET, req_id: 25bf103d-fe57-4109-b016-3d6772ac5eed, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
7043 [8810] Exited with: ExitStatus(unix_wait_status(9))
7044 [8810][0] reports Exit
7045 Feb 28 06:02:37.135 INFO request completed, latency_us: 238, response_code: 200, uri: /state/cid/0, method: GET, req_id: 0859d4b5-1748-4d78-832e-7b0ed9bc9b76, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
7046 001/300 Read block 4363 len 12288
7047 002/300 Read block 3888 len 16384
7048 003/300 Write block 795 len 24576 data: 3 3 3 3 2 3
7049 004/300 Read block 2678 len 28672
7050 005/300 Flush
7051 006/300 Write block 1633 len 24576 data: 3 3 3 4 4 2
7052 007/300 Read block 1099 len 20480
7053 008/300 Write block 4390 len 32768 data: 2 2 2 2 2 2 2 2
7054 009/300 Write block 1874 len 36864 data: 2 2 2 2 2 2 2 2 2
7055 010/300 Write block 3717 len 32768 data: 2 3 3 3 3 3 3 3
7056 011/300 Flush
7057 012/300 Write block 3312 len 8192 data: 2 2
7058 013/300 Write block 3098 len 32768 data: 2 2 3 4 4 4 4 4
7059 014/300 Write block 3360 len 40960 data: 2 2 2 2 3 3 3 2 2 2
7060 015/300 Read block 3772 len 20480
7061 016/300 Write block 4018 len 8192 data: 2 2
7062 017/300 Write block 4632 len 36864 data: 4 4 3 2 2 2 2 2 2
7063 018/300 Read block 4085 len 16384
7064 019/300 Write block 962 len 28672 data: 2 2 2 2 2 2 2
7065 020/300 Flush
7066 021/300 Write block 1456 len 12288 data: 3 3 2
7067 022/300 Read block 3197 len 36864
7068 023/300 Read block 2195 len 24576
7069 024/300 Write block 4937 len 12288 data: 2 2 2
7070 025/300 Flush
7071 026/300 Write block 4621 len 32768 data: 2 2 3 3 3 3 3 3
7072 027/300 Write block 1730 len 20480 data: 2 2 2 2 2
7073 028/300 Read block 4399 len 40960
7074 029/300 Write block 1969 len 28672 data: 2 2 2 2 2 2 2
7075 030/300 Write block 3007 len 36864 data: 2 2 2 2 2 2 2 2 2
7076 031/300 Read block 4295 len 16384
7077 032/300 Write block 2115 len 40960 data: 2 2 3 3 3 3 3 3 3 4
7078 033/300 Flush
7079 034/300 Write block 844 len 12288 data: 4 4 3
7080 035/300 Write block 3222 len 20480 data: 2 2 2 2 2
7081 036/300 Read block 4047 len 40960
7082 037/300 Read block 2508 len 28672
7083 038/300 Write block 192 len 40960 data: 4 4 4 3 3 2 2 2 2 2
7084 039/300 Read block 4385 len 8192
7085 040/300 Read block 793 len 28672
7086 041/300 Write block 115 len 4096 data: 2
7087 042/300 Write block 4544 len 16384 data: 3 3 3 3
7088 043/300 Write block 2611 len 12288 data: 2 2 2
7089 044/300 Read block 1807 len 24576
7090 045/300 Read block 1399 len 32768
7091 046/300 Write block 4779 len 20480 data: 4 4 4 3 3
7092 047/300 Write block 1795 len 28672 data: 2 2 2 2 2 2 2
7093 048/300 Read block 502 len 12288
7094 049/300 Read block 2410 len 12288
7095 050/300 Write block 2455 len 16384 data: 2 2 2 2
7096 051/300 Write block 153 len 16384 data: 4 4 3 3
7097 052/300 Read block 3114 len 8192
7098 053/300 Write block 1391 len 8192 data: 3 3
7099 054/300 Write block 392 len 4096 data: 2
7100 055/300 Write block 709 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7101 056/300 Write block 2783 len 4096 data: 4
7102 057/300 Read block 553 len 16384
7103 058/300 Write block 4386 len 24576 data: 3 2 2 2 3 3
7104 059/300 Read block 4735 len 12288
7105 060/300 Write block 4881 len 36864 data: 3 3 3 3 3 3 4 5 5
7106 061/300 Write block 2470 len 12288 data: 2 3 3
7107 062/300 Write block 2090 len 36864 data: 2 2 2 2 2 2 2 2 2
7108 063/300 Read block 3737 len 16384
7109 064/300 Write block 1727 len 24576 data: 2 2 2 3 3 3
7110 065/300 Read block 2396 len 12288
7111 066/300 Write block 1174 len 28672 data: 3 3 4 4 4 5 4
7112 067/300 Read block 4102 len 24576
7113 068/300 Write block 1700 len 20480 data: 4 4 3 3 2
7114 069/300 Read block 4084 len 4096
7115 070/300 Write block 1001 len 12288 data: 3 3 3
7116 071/300 Write block 4427 len 20480 data: 2 2 2 2 2
7117 072/300 Read block 1056 len 8192
7118 073/300 Write block 827 len 20480 data: 2 2 3 3 3
7119 074/300 Write block 4101 len 16384 data: 2 2 2 2
7120 075/300 Write block 4630 len 8192 data: 3 4
7121 076/300 Read block 1491 len 4096
7122 077/300 Read block 3161 len 28672
7123 078/300 Write block 3330 len 12288 data: 2 2 2
7124 079/300 Read block 4159 len 36864
7125 080/300 Flush
7126 081/300 Flush
7127 082/300 Write block 706 len 32768 data: 2 2 2 3 3 3 3 3
7128 083/300 Read block 3446 len 28672
7129 084/300 Flush
7130 085/300 Write block 230 len 8192 data: 2 2
7131 086/300 Write block 1373 len 28672 data: 2 2 2 2 2 2 2
7132 087/300 Write block 3954 len 36864 data: 2 2 2 2 2 2 2 2 2
7133 088/300 Read block 3672 len 40960
7134 089/300 Write block 4159 len 40960 data: 2 2 2 2 2 2 3 3 3 3
7135 090/300 Read block 3205 len 8192
7136 091/300 Write block 4322 len 36864 data: 3 3 2 2 2 2 2 2 2
7137 092/300 Read block 2384 len 28672
7138 093/300 Read block 26 len 4096
7139 094/300 Write block 1852 len 12288 data: 3 2 2
7140 095/300 Read block 4888 len 40960
7141 096/300 Read block 2522 len 36864
7142 097/300 Read block 729 len 4096
7143 098/300 Write block 4529 len 8192 data: 4 4
7144 099/300 Read block 4753 len 12288
7145 100/300 Read block 691 len 20480
7146 101/300 Read block 1352 len 28672
7147 102/300 Read block 1923 len 36864
7148 103/300 Write block 4851 len 24576 data: 2 2 2 2 2 2
7149 104/300 Write block 957 len 36864 data: 2 2 2 2 3 3 3 3 3
7150 105/300 Write block 4076 len 24576 data: 3 2 2 3 2 2
7151 106/300 Read block 4427 len 28672
7152 107/300 Flush
7153 108/300 Write block 4835 len 20480 data: 2 2 2 2 2
7154 109/300 Write block 2626 len 4096 data: 3
7155 110/300 Read block 3261 len 24576
7156 111/300 Read block 1881 len 4096
7157 112/300 Read block 1731 len 24576
7158 113/300 Flush
7159 114/300 Read block 2086 len 16384
7160 115/300 Read block 3480 len 28672
7161 116/300 Read block 1 len 20480
7162 117/300 Read block 3559 len 40960
7163 118/300 Write block 515 len 32768 data: 2 2 2 2 2 2 3 3
7164 119/300 Read block 1729 len 12288
7165 120/300 Read block 1127 len 20480
7166 121/300 Read block 1087 len 12288
7167 122/300 Read block 1000 len 20480
7168 123/300 Write block 2202 len 4096 data: 3
7169 124/300 Write block 3025 len 16384 data: 2 2 3 3
7170 125/300 Read block 4594 len 20480
7171 126/300 Read block 4011 len 32768
7172 127/300 Write block 3308 len 40960 data: 3 3 2 2 3 3 2 2 2 2
7173 128/300 Write block 3739 len 4096 data: 3
7174 129/300 Read block 1681 len 4096
7175 130/300 Write block 2776 len 36864 data: 2 2 2 3 3 3 3 5 4
7176 131/300 Read block 77 len 32768
7177 132/300 Write block 632 len 12288 data: 2 2 2
7178 133/300 Write block 3510 len 28672 data: 2 2 2 2 2 2 2
7179 134/300 Read block 727 len 8192
7180 135/300 Read block 1001 len 4096
7181 136/300 Write block 3724 len 8192 data: 4 2
7182 137/300 Read block 1372 len 8192
7183 138/300 Write block 4982 len 24576 data: 2 2 3 3 3 3
7184 139/300 Read block 21 len 8192
7185 140/300 Read block 2702 len 40960
7186 141/300 Read block 33 len 36864
7187 142/300 Write block 4313 len 36864 data: 2 2 2 2 2 2 3 3 3
7188 143/300 Read block 3309 len 12288
7189 144/300 Read block 1864 len 16384
7190 145/300 Write block 1260 len 20480 data: 2 3 3 3 3
7191 146/300 Write block 641 len 32768 data: 2 2 2 2 2 2 2 2
7192 147/300 Flush
7193 148/300 Write block 4110 len 4096 data: 2
7194 149/300 Read block 4305 len 20480
7195 150/300 Write block 620 len 32768 data: 2 2 2 2 2 3 3 3
7196 151/300 Read block 3384 len 40960
7197 152/300 Write block 1064 len 12288 data: 2 2 2
7198 153/300 Read block 2518 len 16384
7199 154/300 Read block 1961 len 28672
7200 155/300 Write block 2852 len 28672 data: 2 2 2 2 2 2 2
7201 156/300 Write block 1946 len 12288 data: 2 2 2
7202 157/300 Read block 821 len 28672
7203 158/300 Write block 3592 len 32768 data: 2 2 2 2 2 2 2 2
7204 159/300 Write block 2690 len 36864 data: 4 4 3 3 3 3 3 3 3
7205 160/300 Write block 427 len 40960 data: 2 3 3 4 4 4 4 3 3 4
7206 161/300 Read block 3351 len 32768
7207 162/300 Write block 3165 len 24576 data: 2 2 2 2 2 2
7208 163/300 Flush
7209 164/300 Read block 1170 len 40960
7210 165/300 Write block 1066 len 20480 data: 3 2 2 2 2
7211 166/300 Write block 3121 len 8192 data: 3 3
7212 167/300 Read block 3632 len 20480
7213 168/300 Write block 3027 len 12288 data: 4 4 4
7214 169/300 Read block 69 len 20480
7215 170/300 Read block 4374 len 8192
7216 171/300 Read block 406 len 40960
7217 172/300 Write block 926 len 4096 data: 3
7218 173/300 Read block 4034 len 32768
7219 174/300 Flush
7220 175/300 Write block 2923 len 16384 data: 2 2 2 2
7221 176/300 Write block 1358 len 32768 data: 2 3 3 3 2 2 2 2
7222 177/300 Flush
7223 178/300 Write block 4204 len 28672 data: 2 2 2 2 2 3 3
7224 179/300 Read block 1182 len 24576
7225 180/300 Read block 890 len 32768
7226 181/300 Write block 4616 len 4096 data: 2
7227 182/300 Read block 2275 len 28672
7228 183/300 Read block 4755 len 4096
7229 184/300 Read block 1547 len 36864
7230 185/300 Read block 4208 len 24576
7231 186/300 Read block 446 len 28672
7232 187/300 Read block 3417 len 16384
7233 188/300 Read block 4425 len 8192
7234 189/300 Read block 1252 len 4096
7235 190/300 Write block 3082 len 12288 data: 2 2 2
7236 191/300 Read block 3154 len 36864
7237 192/300 Flush
7238 193/300 Write block 101 len 24576 data: 3 3 3 3 3 3
7239 194/300 Write block 2866 len 28672 data: 3 3 3 3 3 3 3
7240 195/300 Write block 4775 len 4096 data: 2
7241 196/300 Read block 2190 len 16384
7242 197/300 Flush
7243 198/300 Read block 2070 len 12288
7244 199/300 Read block 834 len 32768
7245 200/300 Write block 3205 len 12288 data: 2 2 2
7246 201/300 Write block 2917 len 8192 data: 2 2
7247 202/300 Read block 862 len 16384
7248 203/300 Write block 218 len 8192 data: 3 3
7249 204/300 Read block 1957 len 16384
7250 205/300 Write block 3461 len 4096 data: 2
7251 206/300 Flush
7252 207/300 Read block 3650 len 12288
7253 208/300 Write block 4926 len 8192 data: 3 2
7254 209/300 Write block 4756 len 36864 data: 2 2 2 2 2 2 2 2 3
7255 210/300 Read block 3848 len 12288
7256 211/300 Read block 3318 len 24576
7257 212/300 Write block 161 len 24576 data: 3 3 2 2 2 2
7258 213/300 Read block 3864 len 12288
7259 214/300 Read block 4964 len 4096
7260 215/300 Write block 1789 len 20480 data: 2 2 2 2 2
7261 216/300 Read block 1509 len 16384
7262 217/300 Write block 277 len 36864 data: 4 4 3 3 3 3 4 4 3
7263 218/300 Read block 693 len 16384
7264 219/300 Flush
7265 220/300 Read block 3922 len 20480
7266 221/300 Write block 4837 len 36864 data: 3 3 3 3 3 4 3 3 3
7267 222/300 Write block 4980 len 32768 data: 2 2 3 3 4 4 4 4
7268 223/300 Flush
7269 224/300 Write block 893 len 8192 data: 2 2
7270 225/300 Write block 2569 len 20480 data: 2 2 2 2 2
7271 226/300 Read block 2429 len 20480
7272 227/300 Write block 4853 len 16384 data: 3 3 3 3
7273 228/300 Read block 1392 len 36864
7274 229/300 Write block 1356 len 24576 data: 2 2 3 4 4 4
7275 230/300 Write block 1278 len 24576 data: 2 2 3 3 3 3
7276 231/300 Read block 332 len 12288
7277 232/300 Write block 2454 len 8192 data: 2 3
7278 233/300 Read block 3949 len 8192
7279 234/300 Read block 2269 len 4096
7280 235/300 Flush
7281 236/300 Write block 3760 len 12288 data: 2 2 2
7282 237/300 Read block 1757 len 20480
7283 238/300 Flush
7284 239/300 Write block 4903 len 16384 data: 3 3 3 3
7285 240/300 Write block 4678 len 24576 data: 3 3 3 3 2 2
7286 241/300 Write block 3912 len 20480 data: 2 2 2 2 2
7287 242/300 Write block 2664 len 28672 data: 3 3 3 3 2 2 3
7288 243/300 Read block 2081 len 12288
7289 244/300 Write block 4426 len 16384 data: 2 3 3 3
7290 245/300 Read block 1882 len 12288
7291 246/300 Write block 1848 len 40960 data: 2 2 2 3 4 3 3 2 2 2
7292 247/300 Write block 2028 len 24576 data: 2 2 2 2 2 2
7293 248/300 Read block 4931 len 28672
7294 249/300 Write block 3864 len 28672 data: 2 2 2 2 2 3 3
7295 250/300 Write block 862 len 24576 data: 3 3 3 2 3 3
7296 251/300 Write block 1178 len 16384 data: 5 6 5 4
7297 252/300 Read block 2696 len 24576
7298 253/300 Read block 4829 len 24576
7299 254/300 Write block 416 len 24576 data: 4 4 4 4 4 4
7300 255/300 Write block 1025 len 12288 data: 2 2 2
7301 256/300 Read block 2899 len 12288
7302 257/300 Read block 716 len 36864
7303 258/300 Read block 2358 len 36864
7304 259/300 Write block 4590 len 36864 data: 2 2 2 3 3 3 2 2 2
7305 260/300 Read block 1953 len 4096
7306 261/300 Write block 1821 len 36864 data: 2 2 2 2 2 2 2 2 2
7307 262/300 Write block 3471 len 36864 data: 2 2 2 2 3 2 2 2 3
7308 263/300 Flush
7309 264/300 Write block 1995 len 24576 data: 2 2 2 2 2 3
7310 265/300 Read block 2490 len 28672
7311 266/300 Flush
7312 267/300 Write block 2898 len 4096 data: 2
7313 268/300 Write block 3297 len 4096 data: 5
7314 269/300 Write block 2720 len 8192 data: 2 2
7315 270/300 Write block 3212 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7316 271/300 Read block 2339 len 24576
7317 272/300 Write block 53 len 32768 data: 3 3 3 3 3 3 2 2
7318 273/300 Read block 101 len 32768
7319 274/300 Read block 3167 len 28672
7320 275/300 Read block 235 len 36864
7321 276/300 Write block 3524 len 36864 data: 4 4 4 5 5 4 2 2 2
7322 277/300 Read block 619 len 32768
7323 278/300 Read block 1223 len 32768
7324 279/300 Write block 647 len 36864 data: 3 3 2 2 2 2 2 2 2
7325 280/300 Read block 615 len 4096
7326 281/300 Read block 2701 len 16384
7327 282/300 Read block 1176 len 12288
7328 283/300 Read block 4450 len 28672
7329 284/300 Write block 1261 len 40960 data: 4 4 4 4 3 2 2 2 2 2
7330 285/300 Write block 2484 len 12288 data: 2 2 2
7331 286/300 Write block 245 len 8192 data: 2 2
7332 287/300 Write block 2957 len 24576 data: 3 3 3 2 3 4
7333 288/300 Read block 500 len 24576
7334 289/300 Read block 1210 len 40960
7335 290/300 Read block 1981 len 28672
7336 291/300 Read block 4863 len 36864
7337 292/300 Read block 2617 len 32768
7338 293/300 Write block 1340 len 24576 data: 3 3 3 3 2 2
7339 294/300 Write block 2568 len 12288 data: 2 3 3
7340 295/300 Read block 1861 len 32768
7341 296/300 Write block 1723 len 32768 data: 2 2 2 2 3 3 3 4
7342 297/300 Read block 1394 len 16384
7343 298/300 Write block 1354 len 36864 data: 3 3 3 3 4 5 5 5 3
7344 299/300 Write block 4249 len 12288 data: 2 2 2
7345 300/300 Read block 1760 len 20480
7346 301/300 Read block 1776 len 8192
7347 Main task has work to do, go find it
7348 got dsc Start(0)
7349 start 0
7350 Feb 28 06:02:37.367 INFO request completed, latency_us: 242, response_code: 204, uri: /start/cid/0, method: POST, req_id: 243f4aa8-3818-4d51-a346-6757f84dfbb3, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
7351 Replay: started 0, returned:Ok(())
7352 ----------------------------------------------------------------
7353 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7354 States: Offline Active Active
7355 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7356 2507 Acked Read 3 Sent Done Done false
7357 2508 Acked Read 4 Sent Done Done false
7358 2509 Acked Write 6 Sent Done Done false
7359 2510 Acked Read 7 Sent Done Done false
7360 2511 Acked Flush 0 Sent Done Done false
7361 2512 Acked Write 6 Sent Done Done false
7362 2513 Acked Read 5 Sent Done Done false
7363 2514 Acked Write 8 Sent Done Done false
7364 2515 Acked Write 9 Sent Done Done false
7365 2516 Acked Write 8 Sent Done Done false
7366 2517 Acked Flush 0 Sent Done Done false
7367 2518 Acked Write 2 Sent Done Done false
7368 2519 Acked Write 8 Sent Done Done false
7369 2520 Acked Write 10 Sent Done Done false
7370 2521 Acked Read 5 Sent Done Done false
7371 2522 Acked Write 2 Sent Done Done false
7372 2523 Acked Write 9 Sent Done Done false
7373 2524 Acked Read 4 Sent Done Done false
7374 2525 Acked Write 7 Sent Done Done false
7375 2526 Acked Flush 0 Sent Done Done false
7376 2527 Acked Write 3 Sent Done Done false
7377 2528 Acked Read 9 Sent Done Done false
7378 2529 Acked Read 6 Sent Done Done false
7379 2530 Acked Write 3 Sent Done Done false
7380 2531 Acked Flush 0 Sent Done Done false
7381 2532 Acked Write 8 Sent Done Done false
7382 2533 Acked Write 5 Sent Done Done false
7383 2534 Acked Read 10 Sent Done Done false
7384 2535 Acked Write 7 Sent Done Done false
7385 2536 Acked Write 9 Sent Done Done false
7386 2537 Acked Read 4 Sent Done Done false
7387 2538 Acked Write 10 Sent Done Done false
7388 2539 Acked Flush 0 Sent Done Done false
7389 2540 Acked Write 3 Sent Done Done false
7390 2541 Acked Write 5 Sent Done Done false
7391 2542 Acked Read 10 Sent Done Done false
7392 2543 Acked Read 7 Sent Done Done false
7393 2544 Acked Write 10 Sent Done Done false
7394 2545 Acked Read 2 Sent Done Done false
7395 2546 Acked Read 7 Sent Done Done false
7396 2547 Acked Write 1 Sent Done Done false
7397 2548 Acked Write 4 Sent Done Done false
7398 2549 Acked Write 3 Sent Done Done false
7399 2550 Acked Read 6 Sent Done Done false
7400 2551 Acked Read 8 Sent Done Done false
7401 2552 Acked Write 5 Sent Done Done false
7402 2553 Acked Write 7 Sent Done Done false
7403 2554 Acked Read 3 Sent Done Done false
7404 2555 Acked Read 3 Sent Done Done false
7405 2556 Acked Write 4 Sent Done Done false
7406 2557 Acked Write 4 Sent Done Done false
7407 2558 Acked Read 2 Sent Done Done false
7408 2559 Acked Write 2 Sent Done Done false
7409 2560 Acked Write 1 Sent Done Done false
7410 2561 Acked Write 10 Sent Done Done false
7411 2562 Acked Write 1 Sent Done Done false
7412 2563 Acked Read 4 Sent Done Done false
7413 2564 Acked Write 6 Sent Done Done false
7414 2565 Acked Read 3 Sent Done Done false
7415 2566 Acked Write 9 Sent Done Done false
7416 2567 Acked Write 3 Sent Done Done false
7417 2568 Acked Write 9 Sent Done Done false
7418 2569 Acked Read 4 Sent Done Done false
7419 2570 Acked Write 6 Sent Done Done false
7420 2571 Acked Read 3 Sent Done Done false
7421 2572 Acked Write 7 Sent Done Done false
7422 2573 Acked Read 6 Sent Done Done false
7423 2574 Acked Write 5 Sent Done Done false
7424 2575 Acked Read 1 Sent Done Done false
7425 2576 Acked Write 3 Sent Done Done false
7426 2577 Acked Write 5 Sent Done Done false
7427 2578 Acked Read 2 Sent Done Done false
7428 2579 Acked Write 5 Sent Done Done false
7429 2580 Acked Write 4 Sent Done Done false
7430 2581 Acked Write 2 Sent Done Done false
7431 2582 Acked Read 1 Sent Done Done false
7432 2583 Acked Read 7 Sent Done Done false
7433 2584 Acked Write 3 Sent Done Done false
7434 2585 Acked Read 9 Sent Done Done false
7435 2586 Acked Flush 0 Sent Done Done false
7436 2587 Acked Flush 0 Sent Done Done false
7437 2588 Acked Write 8 Sent Done Done false
7438 2589 Acked Read 7 Sent Done Done false
7439 2590 Acked Flush 0 Sent Done Done false
7440 2591 Acked Write 2 Sent Done Done false
7441 2592 Acked Write 7 Sent Done Done false
7442 2593 Acked Write 9 Sent Done Done false
7443 2594 Acked Read 10 Sent Done Done false
7444 2595 Acked Write 10 Sent Done Done false
7445 2596 Acked Read 2 Sent Done Done false
7446 2597 Acked Write 9 Sent Done Done false
7447 2598 Acked Read 7 Sent Done Done false
7448 2599 Acked Read 1 Sent Done Done false
7449 2600 Acked Write 3 Sent Done Done false
7450 2601 Acked Read 10 Sent Done Done false
7451 2602 Acked Read 9 Sent Done Done false
7452 2603 Acked Read 1 Sent Done Done false
7453 2604 Acked Write 2 Sent Done Done false
7454 2605 Acked Read 3 Sent Done Done false
7455 2606 Acked Read 5 Sent Done Done false
7456 2607 Acked Read 7 Sent Done Done false
7457 2608 Acked Read 9 Sent Done Done false
7458 2609 Acked Write 6 Sent Done Done false
7459 2610 Acked Write 9 Sent Done Done false
7460 2611 Acked Write 6 Sent Done Done false
7461 2612 Acked Read 7 Sent Done Done false
7462 2613 Acked Flush 0 Sent Done Done false
7463 2614 Acked Write 5 Sent Done Done false
7464 2615 Acked Write 1 Sent Done Done false
7465 2616 Acked Read 6 Sent Done Done false
7466 2617 Acked Read 1 Sent Done Done false
7467 2618 Acked Read 6 Sent Done Done false
7468 2619 Acked Flush 0 Sent Done Done false
7469 2620 Acked Read 4 Sent Done Done false
7470 2621 Acked Read 7 Sent Done Done false
7471 2622 Acked Read 5 Sent Done Done false
7472 2623 Acked Read 10 Sent Done Done false
7473 2624 Acked Write 8 Sent Done Done false
7474 2625 Acked Read 3 Sent Done Done false
7475 2626 Acked Read 5 Sent Done Done false
7476 2627 Acked Read 3 Sent Done Done false
7477 2628 Acked Read 5 Sent Done Done false
7478 2629 Acked Write 1 Sent Done Done false
7479 2630 Acked Write 4 Sent Done Done false
7480 2631 Acked Read 5 Sent Done Done false
7481 2632 Acked Read 8 Sent Done Done false
7482 2633 Acked Write 10 Sent Done Done false
7483 2634 Acked Write 1 Sent Done Done false
7484 2635 Acked Read 1 Sent Done Done false
7485 2636 Acked Write 9 Sent Done Done false
7486 2637 Acked Read 8 Sent Done Done false
7487 2638 Acked Write 3 Sent Done Done false
7488 2639 Acked Write 7 Sent Done Done false
7489 2640 Acked Read 2 Sent Done Done false
7490 2641 Acked Read 1 Sent Done Done false
7491 2642 Acked Write 2 Sent Done Done false
7492 2643 Acked Read 2 Sent Done Done false
7493 2644 Acked Write 6 Sent Done Done false
7494 2645 Acked Read 2 Sent Done Done false
7495 2646 Acked Read 10 Sent Done Done false
7496 2647 Acked Read 9 Sent Done Done false
7497 2648 Acked Write 9 Sent Done Done false
7498 2649 Acked Read 3 Sent Done Done false
7499 2650 Acked Read 4 Sent Done Done false
7500 2651 Acked Write 5 Sent Done Done false
7501 2652 Acked Write 8 Sent Done Done false
7502 2653 Acked Flush 0 Sent Done Done false
7503 2654 Acked Write 1 Sent Done Done false
7504 2655 Acked Read 5 Sent Done Done false
7505 2656 Acked Write 8 Sent Done Done false
7506 2657 Acked Read 10 Sent Done Done false
7507 2658 Acked Write 3 Sent Done Done false
7508 2659 Acked Read 4 Sent Done Done false
7509 2660 Acked Read 7 Sent Done Done false
7510 2661 Acked Write 7 Sent Done Done false
7511 2662 Acked Write 3 Sent Done Done false
7512 2663 Acked Read 7 Sent Done Done false
7513 2664 Acked Write 8 Sent Done Done false
7514 2665 Acked Write 9 Sent Done Done false
7515 2666 Acked Write 10 Sent Done Done false
7516 2667 Acked Read 8 Sent Done Done false
7517 2668 Acked Write 6 Sent Done Done false
7518 2669 Acked Flush 0 Sent Done Done false
7519 2670 Acked Read 10 Sent Done Done false
7520 2671 Acked Write 5 Sent Done Done false
7521 2672 Acked Write 2 Sent Done Done false
7522 2673 Acked Read 5 Sent Done Done false
7523 2674 Acked Write 3 Sent Done Done false
7524 2675 Acked Read 5 Sent Done Done false
7525 2676 Acked Read 2 Sent Done Done false
7526 2677 Acked Read 10 Sent Done Done false
7527 2678 Acked Write 1 Sent Done Done false
7528 2679 Acked Read 8 Sent Done Done false
7529 2680 Acked Flush 0 Sent Done Done false
7530 2681 Acked Write 4 Sent Done Done false
7531 2682 Acked Write 8 Sent Done Done false
7532 2683 Acked Flush 0 Sent Done Done false
7533 2684 Acked Write 7 Sent Done Done false
7534 2685 Acked Read 6 Sent Done Done false
7535 2686 Acked Read 8 Sent Done Done false
7536 2687 Acked Write 1 Sent Done Done false
7537 2688 Acked Read 7 Sent Done Done false
7538 2689 Acked Read 1 Sent Done Done false
7539 2690 Acked Read 9 Sent Done Done false
7540 2691 Acked Read 6 Sent Done Done false
7541 2692 Acked Read 7 Sent Done Done false
7542 2693 Acked Read 4 Sent Done Done false
7543 2694 Acked Read 2 Sent Done Done false
7544 2695 Acked Read 1 Sent Done Done false
7545 2696 Acked Write 3 Sent Done Done false
7546 2697 Acked Read 9 Sent Done Done false
7547 2698 Acked Flush 0 Sent Done Done false
7548 2699 Acked Write 6 Sent Done Done false
7549 2700 Acked Write 7 Sent Done Done false
7550 2701 Acked Write 1 Sent Done Done false
7551 2702 Acked Read 4 Sent Done Done false
7552 2703 Acked Flush 0 Sent Done Done false
7553 2704 Acked Read 3 Sent Done Done false
7554 2705 Acked Read 8 Sent Done Done false
7555 2706 Acked Write 3 Sent Done Done false
7556 2707 Acked Write 2 Sent Done Done false
7557 2708 Acked Read 4 Sent Done Done false
7558 2709 Acked Write 2 Sent Done Done false
7559 2710 Acked Read 4 Sent Done Done false
7560 2711 Acked Write 1 Sent Done Done false
7561 2712 Acked Flush 0 Sent Done Done false
7562 2713 Acked Read 3 Sent Done Done false
7563 2714 Acked Write 2 Sent Done Done false
7564 2715 Acked Write 9 Sent Done Done false
7565 2716 Acked Read 3 Sent Done Done false
7566 2717 Acked Read 6 Sent Done Done false
7567 2718 Acked Write 6 Sent Done Done false
7568 2719 Acked Read 3 Sent Done Done false
7569 2720 Acked Read 1 Sent Done Done false
7570 2721 Acked Write 5 Sent Done Done false
7571 2722 Acked Read 4 Sent Done Done false
7572 2723 Acked Write 9 Sent Done Done false
7573 2724 Acked Read 4 Sent Done Done false
7574 2725 Acked Flush 0 Sent Done Done false
7575 2726 Acked Read 5 Sent Done Done false
7576 2727 Acked Write 9 Sent Done Done false
7577 2728 Acked Write 8 Sent Done Done false
7578 2729 Acked Flush 0 Sent Done Done false
7579 2730 Acked Write 2 Sent Done Done false
7580 2731 Acked Write 5 Sent Done Done false
7581 2732 Acked Read 5 Sent Done Done false
7582 2733 Acked Write 4 Sent Done Done false
7583 2734 Acked Read 9 Sent Done Done false
7584 2735 Acked Write 6 Sent Done Done false
7585 2736 Acked Write 6 Sent Done Done false
7586 2737 Acked Read 3 Sent Done Done false
7587 2738 Acked Write 2 Sent Done Done false
7588 2739 Acked Read 2 Sent Done Done false
7589 2740 Acked Read 1 Sent Done Done false
7590 2741 Acked Flush 0 Sent Done Done false
7591 2742 Acked Write 3 Sent Done Done false
7592 2743 Acked Read 5 Sent Done Done false
7593 2744 Acked Flush 0 Sent Done Done false
7594 2745 Acked Write 4 Sent Done Done false
7595 2746 Acked Write 6 Sent Done Done false
7596 2747 Acked Write 5 Sent Done Done false
7597 2748 Acked Write 7 Sent Done Done false
7598 2749 Acked Read 3 Sent Done Done false
7599 2750 Acked Write 4 Sent Done Done false
7600 2751 Acked Read 3 Sent Done Done false
7601 2752 Acked Write 10 Sent Done Done false
7602 2753 Acked Write 6 Sent Done Done false
7603 2754 Acked Read 7 Sent Done Done false
7604 2755 Acked Write 7 Sent Done Done false
7605 2756 Acked Write 6 Sent Done Done false
7606 2757 Acked Write 4 Sent Done Done false
7607 2758 Acked Read 6 Sent Done Done false
7608 2759 Acked Read 6 Sent Done Done false
7609 2760 Acked Write 6 Sent Done Done false
7610 2761 Acked Write 3 Sent Done Done false
7611 2762 Acked Read 3 Sent Done Done false
7612 2763 Acked Read 9 Sent Done Done false
7613 2764 Acked Read 9 Sent Done Done false
7614 2765 Acked Write 9 Sent Done Done false
7615 2766 Acked Read 1 Sent Done Done false
7616 2767 Acked Write 9 Sent Done Done false
7617 2768 Acked Write 9 Sent Done Done false
7618 2769 Acked Flush 0 Sent Done Done false
7619 2770 Acked Write 6 Sent Done Done false
7620 2771 Acked Read 7 Sent Done Done false
7621 2772 Acked Flush 0 Sent Done Done false
7622 2773 Acked Write 1 Sent Done Done false
7623 2774 Acked Write 1 Sent Done Done false
7624 2775 Acked Write 2 Sent Done Done false
7625 2776 Acked Write 10 Sent Done Done false
7626 2777 Acked Read 6 Sent Done Done false
7627 2778 Acked Write 8 Sent Done Done false
7628 2779 Acked Read 8 Sent Done Done false
7629 2780 Acked Read 7 Sent Done Done false
7630 2781 Acked Read 9 Sent Done Done false
7631 2782 Acked Write 9 Sent Done Done false
7632 2783 Acked Read 8 Sent Done Done false
7633 2784 Acked Read 8 Sent Done Done false
7634 2785 Acked Write 9 Sent Done Done false
7635 2786 Acked Read 1 Sent Done Done false
7636 2787 Acked Read 4 Sent Done Done false
7637 2788 Acked Read 3 Sent Done Done false
7638 2789 Acked Read 7 Sent Done Done false
7639 2790 Acked Write 10 Sent Done Done false
7640 2791 Acked Write 3 Sent Done Done false
7641 2792 Acked Write 2 Sent Done Done false
7642 2793 Acked Write 6 Sent Done Done false
7643 2794 Acked Read 6 Sent Done Done false
7644 2795 Acked Read 10 Sent Done Done false
7645 2796 Acked Read 7 Sent Done Done false
7646 2797 Acked Read 9 Sent Done Done false
7647 2798 Acked Read 8 Sent Done Done false
7648 2799 Acked Write 6 Sent Done Done false
7649 2800 Acked Write 3 Sent Done Done false
7650 2801 Acked Read 8 Sent Done Done false
7651 2802 Acked Write 8 Sent Done Done false
7652 2803 Acked Read 4 Sent Done Done false
7653 2804 Acked Write 9 Sent Done Done false
7654 2805 Acked Write 3 Sent Done Done false
7655 2806 Acked Read 5 Sent Done Done false
7656 2807 Acked Read 2 Sent Done Done false
7657 STATES DS:0 DS:1 DS:2 TOTAL
7658 Sent 301 0 0 301
7659 Done 0 301 301 602
7660 Skipped 0 0 0 0
7661 Error 0 0 0 0
7662 Last Flush: Some(JobId(2506)) Some(JobId(2772)) Some(JobId(2772))
7663 Downstairs last five completed: 2506 2505 2504 2503 2502
7664 Upstairs last five completed: 2807 2806 2805 2804 2803
7665 CLIENT: Up:0 ds:301 act:2
76662025-02-28T06:02:37.869ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
76672025-02-28T06:02:38.495ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
76682025-02-28T06:02:39.497ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
76692025-02-28T06:02:40.499ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
7670 ----------------------------------------------------------------
7671 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7672 States: Offline Active Active
7673 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7674 2507 Acked Read 3 Sent Done Done false
7675 2508 Acked Read 4 Sent Done Done false
7676 2509 Acked Write 6 Sent Done Done false
7677 2510 Acked Read 7 Sent Done Done false
7678 2511 Acked Flush 0 Sent Done Done false
7679 2512 Acked Write 6 Sent Done Done false
7680 2513 Acked Read 5 Sent Done Done false
7681 2514 Acked Write 8 Sent Done Done false
7682 2515 Acked Write 9 Sent Done Done false
7683 2516 Acked Write 8 Sent Done Done false
7684 2517 Acked Flush 0 Sent Done Done false
7685 2518 Acked Write 2 Sent Done Done false
7686 2519 Acked Write 8 Sent Done Done false
7687 2520 Acked Write 10 Sent Done Done false
7688 2521 Acked Read 5 Sent Done Done false
7689 2522 Acked Write 2 Sent Done Done false
7690 2523 Acked Write 9 Sent Done Done false
7691 2524 Acked Read 4 Sent Done Done false
7692 2525 Acked Write 7 Sent Done Done false
7693 2526 Acked Flush 0 Sent Done Done false
7694 2527 Acked Write 3 Sent Done Done false
7695 2528 Acked Read 9 Sent Done Done false
7696 2529 Acked Read 6 Sent Done Done false
7697 2530 Acked Write 3 Sent Done Done false
7698 2531 Acked Flush 0 Sent Done Done false
7699 2532 Acked Write 8 Sent Done Done false
7700 2533 Acked Write 5 Sent Done Done false
7701 2534 Acked Read 10 Sent Done Done false
7702 2535 Acked Write 7 Sent Done Done false
7703 2536 Acked Write 9 Sent Done Done false
7704 2537 Acked Read 4 Sent Done Done false
7705 2538 Acked Write 10 Sent Done Done false
7706 2539 Acked Flush 0 Sent Done Done false
7707 2540 Acked Write 3 Sent Done Done false
7708 2541 Acked Write 5 Sent Done Done false
7709 2542 Acked Read 10 Sent Done Done false
7710 2543 Acked Read 7 Sent Done Done false
7711 2544 Acked Write 10 Sent Done Done false
7712 2545 Acked Read 2 Sent Done Done false
7713 2546 Acked Read 7 Sent Done Done false
7714 2547 Acked Write 1 Sent Done Done false
7715 2548 Acked Write 4 Sent Done Done false
7716 2549 Acked Write 3 Sent Done Done false
7717 2550 Acked Read 6 Sent Done Done false
7718 2551 Acked Read 8 Sent Done Done false
7719 2552 Acked Write 5 Sent Done Done false
7720 2553 Acked Write 7 Sent Done Done false
7721 2554 Acked Read 3 Sent Done Done false
7722 2555 Acked Read 3 Sent Done Done false
7723 2556 Acked Write 4 Sent Done Done false
7724 2557 Acked Write 4 Sent Done Done false
7725 2558 Acked Read 2 Sent Done Done false
7726 2559 Acked Write 2 Sent Done Done false
7727 2560 Acked Write 1 Sent Done Done false
7728 2561 Acked Write 10 Sent Done Done false
7729 2562 Acked Write 1 Sent Done Done false
7730 2563 Acked Read 4 Sent Done Done false
7731 2564 Acked Write 6 Sent Done Done false
7732 2565 Acked Read 3 Sent Done Done false
7733 2566 Acked Write 9 Sent Done Done false
7734 2567 Acked Write 3 Sent Done Done false
7735 2568 Acked Write 9 Sent Done Done false
7736 2569 Acked Read 4 Sent Done Done false
7737 2570 Acked Write 6 Sent Done Done false
7738 2571 Acked Read 3 Sent Done Done false
7739 2572 Acked Write 7 Sent Done Done false
7740 2573 Acked Read 6 Sent Done Done false
7741 2574 Acked Write 5 Sent Done Done false
7742 2575 Acked Read 1 Sent Done Done false
7743 2576 Acked Write 3 Sent Done Done false
7744 2577 Acked Write 5 Sent Done Done false
7745 2578 Acked Read 2 Sent Done Done false
7746 2579 Acked Write 5 Sent Done Done false
7747 2580 Acked Write 4 Sent Done Done false
7748 2581 Acked Write 2 Sent Done Done false
7749 2582 Acked Read 1 Sent Done Done false
7750 2583 Acked Read 7 Sent Done Done false
7751 2584 Acked Write 3 Sent Done Done false
7752 2585 Acked Read 9 Sent Done Done false
7753 2586 Acked Flush 0 Sent Done Done false
7754 2587 Acked Flush 0 Sent Done Done false
7755 2588 Acked Write 8 Sent Done Done false
7756 2589 Acked Read 7 Sent Done Done false
7757 2590 Acked Flush 0 Sent Done Done false
7758 2591 Acked Write 2 Sent Done Done false
7759 2592 Acked Write 7 Sent Done Done false
7760 2593 Acked Write 9 Sent Done Done false
7761 2594 Acked Read 10 Sent Done Done false
7762 2595 Acked Write 10 Sent Done Done false
7763 2596 Acked Read 2 Sent Done Done false
7764 2597 Acked Write 9 Sent Done Done false
7765 2598 Acked Read 7 Sent Done Done false
7766 2599 Acked Read 1 Sent Done Done false
7767 2600 Acked Write 3 Sent Done Done false
7768 2601 Acked Read 10 Sent Done Done false
7769 2602 Acked Read 9 Sent Done Done false
7770 2603 Acked Read 1 Sent Done Done false
7771 2604 Acked Write 2 Sent Done Done false
7772 2605 Acked Read 3 Sent Done Done false
7773 2606 Acked Read 5 Sent Done Done false
7774 2607 Acked Read 7 Sent Done Done false
7775 2608 Acked Read 9 Sent Done Done false
7776 2609 Acked Write 6 Sent Done Done false
7777 2610 Acked Write 9 Sent Done Done false
7778 2611 Acked Write 6 Sent Done Done false
7779 2612 Acked Read 7 Sent Done Done false
7780 2613 Acked Flush 0 Sent Done Done false
7781 2614 Acked Write 5 Sent Done Done false
7782 2615 Acked Write 1 Sent Done Done false
7783 2616 Acked Read 6 Sent Done Done false
7784 2617 Acked Read 1 Sent Done Done false
7785 2618 Acked Read 6 Sent Done Done false
7786 2619 Acked Flush 0 Sent Done Done false
7787 2620 Acked Read 4 Sent Done Done false
7788 2621 Acked Read 7 Sent Done Done false
7789 2622 Acked Read 5 Sent Done Done false
7790 2623 Acked Read 10 Sent Done Done false
7791 2624 Acked Write 8 Sent Done Done false
7792 2625 Acked Read 3 Sent Done Done false
7793 2626 Acked Read 5 Sent Done Done false
7794 2627 Acked Read 3 Sent Done Done false
7795 2628 Acked Read 5 Sent Done Done false
7796 2629 Acked Write 1 Sent Done Done false
7797 2630 Acked Write 4 Sent Done Done false
7798 2631 Acked Read 5 Sent Done Done false
7799 2632 Acked Read 8 Sent Done Done false
7800 2633 Acked Write 10 Sent Done Done false
7801 2634 Acked Write 1 Sent Done Done false
7802 2635 Acked Read 1 Sent Done Done false
7803 2636 Acked Write 9 Sent Done Done false
7804 2637 Acked Read 8 Sent Done Done false
7805 2638 Acked Write 3 Sent Done Done false
7806 2639 Acked Write 7 Sent Done Done false
7807 2640 Acked Read 2 Sent Done Done false
7808 2641 Acked Read 1 Sent Done Done false
7809 2642 Acked Write 2 Sent Done Done false
7810 2643 Acked Read 2 Sent Done Done false
7811 2644 Acked Write 6 Sent Done Done false
7812 2645 Acked Read 2 Sent Done Done false
7813 2646 Acked Read 10 Sent Done Done false
7814 2647 Acked Read 9 Sent Done Done false
7815 2648 Acked Write 9 Sent Done Done false
7816 2649 Acked Read 3 Sent Done Done false
7817 2650 Acked Read 4 Sent Done Done false
7818 2651 Acked Write 5 Sent Done Done false
7819 2652 Acked Write 8 Sent Done Done false
7820 2653 Acked Flush 0 Sent Done Done false
7821 2654 Acked Write 1 Sent Done Done false
7822 2655 Acked Read 5 Sent Done Done false
7823 2656 Acked Write 8 Sent Done Done false
7824 2657 Acked Read 10 Sent Done Done false
7825 2658 Acked Write 3 Sent Done Done false
7826 2659 Acked Read 4 Sent Done Done false
7827 2660 Acked Read 7 Sent Done Done false
7828 2661 Acked Write 7 Sent Done Done false
7829 2662 Acked Write 3 Sent Done Done false
7830 2663 Acked Read 7 Sent Done Done false
7831 2664 Acked Write 8 Sent Done Done false
7832 2665 Acked Write 9 Sent Done Done false
7833 2666 Acked Write 10 Sent Done Done false
7834 2667 Acked Read 8 Sent Done Done false
7835 2668 Acked Write 6 Sent Done Done false
7836 2669 Acked Flush 0 Sent Done Done false
7837 2670 Acked Read 10 Sent Done Done false
7838 2671 Acked Write 5 Sent Done Done false
7839 2672 Acked Write 2 Sent Done Done false
7840 2673 Acked Read 5 Sent Done Done false
7841 2674 Acked Write 3 Sent Done Done false
7842 2675 Acked Read 5 Sent Done Done false
7843 2676 Acked Read 2 Sent Done Done false
7844 2677 Acked Read 10 Sent Done Done false
7845 2678 Acked Write 1 Sent Done Done false
7846 2679 Acked Read 8 Sent Done Done false
7847 2680 Acked Flush 0 Sent Done Done false
7848 2681 Acked Write 4 Sent Done Done false
7849 2682 Acked Write 8 Sent Done Done false
7850 2683 Acked Flush 0 Sent Done Done false
7851 2684 Acked Write 7 Sent Done Done false
7852 2685 Acked Read 6 Sent Done Done false
7853 2686 Acked Read 8 Sent Done Done false
7854 2687 Acked Write 1 Sent Done Done false
7855 2688 Acked Read 7 Sent Done Done false
7856 2689 Acked Read 1 Sent Done Done false
7857 2690 Acked Read 9 Sent Done Done false
7858 2691 Acked Read 6 Sent Done Done false
7859 2692 Acked Read 7 Sent Done Done false
7860 2693 Acked Read 4 Sent Done Done false
7861 2694 Acked Read 2 Sent Done Done false
7862 2695 Acked Read 1 Sent Done Done false
7863 2696 Acked Write 3 Sent Done Done false
7864 2697 Acked Read 9 Sent Done Done false
7865 2698 Acked Flush 0 Sent Done Done false
7866 2699 Acked Write 6 Sent Done Done false
7867 2700 Acked Write 7 Sent Done Done false
7868 2701 Acked Write 1 Sent Done Done false
7869 2702 Acked Read 4 Sent Done Done false
7870 2703 Acked Flush 0 Sent Done Done false
7871 2704 Acked Read 3 Sent Done Done false
7872 2705 Acked Read 8 Sent Done Done false
7873 2706 Acked Write 3 Sent Done Done false
7874 2707 Acked Write 2 Sent Done Done false
7875 2708 Acked Read 4 Sent Done Done false
7876 2709 Acked Write 2 Sent Done Done false
7877 2710 Acked Read 4 Sent Done Done false
7878 2711 Acked Write 1 Sent Done Done false
7879 2712 Acked Flush 0 Sent Done Done false
7880 2713 Acked Read 3 Sent Done Done false
7881 2714 Acked Write 2 Sent Done Done false
7882 2715 Acked Write 9 Sent Done Done false
7883 2716 Acked Read 3 Sent Done Done false
7884 2717 Acked Read 6 Sent Done Done false
7885 2718 Acked Write 6 Sent Done Done false
7886 2719 Acked Read 3 Sent Done Done false
7887 2720 Acked Read 1 Sent Done Done false
7888 2721 Acked Write 5 Sent Done Done false
7889 2722 Acked Read 4 Sent Done Done false
7890 2723 Acked Write 9 Sent Done Done false
7891 2724 Acked Read 4 Sent Done Done false
7892 2725 Acked Flush 0 Sent Done Done false
7893 2726 Acked Read 5 Sent Done Done false
7894 2727 Acked Write 9 Sent Done Done false
7895 2728 Acked Write 8 Sent Done Done false
7896 2729 Acked Flush 0 Sent Done Done false
7897 2730 Acked Write 2 Sent Done Done false
7898 2731 Acked Write 5 Sent Done Done false
7899 2732 Acked Read 5 Sent Done Done false
7900 2733 Acked Write 4 Sent Done Done false
7901 2734 Acked Read 9 Sent Done Done false
7902 2735 Acked Write 6 Sent Done Done false
7903 2736 Acked Write 6 Sent Done Done false
7904 2737 Acked Read 3 Sent Done Done false
7905 2738 Acked Write 2 Sent Done Done false
7906 2739 Acked Read 2 Sent Done Done false
7907 2740 Acked Read 1 Sent Done Done false
7908 2741 Acked Flush 0 Sent Done Done false
7909 2742 Acked Write 3 Sent Done Done false
7910 2743 Acked Read 5 Sent Done Done false
7911 2744 Acked Flush 0 Sent Done Done false
7912 2745 Acked Write 4 Sent Done Done false
7913 2746 Acked Write 6 Sent Done Done false
7914 2747 Acked Write 5 Sent Done Done false
7915 2748 Acked Write 7 Sent Done Done false
7916 2749 Acked Read 3 Sent Done Done false
7917 2750 Acked Write 4 Sent Done Done false
7918 2751 Acked Read 3 Sent Done Done false
7919 2752 Acked Write 10 Sent Done Done false
7920 2753 Acked Write 6 Sent Done Done false
7921 2754 Acked Read 7 Sent Done Done false
7922 2755 Acked Write 7 Sent Done Done false
7923 2756 Acked Write 6 Sent Done Done false
7924 2757 Acked Write 4 Sent Done Done false
7925 2758 Acked Read 6 Sent Done Done false
7926 2759 Acked Read 6 Sent Done Done false
7927 2760 Acked Write 6 Sent Done Done false
7928 2761 Acked Write 3 Sent Done Done false
7929 2762 Acked Read 3 Sent Done Done false
7930 2763 Acked Read 9 Sent Done Done false
7931 2764 Acked Read 9 Sent Done Done false
7932 2765 Acked Write 9 Sent Done Done false
7933 2766 Acked Read 1 Sent Done Done false
7934 2767 Acked Write 9 Sent Done Done false
7935 2768 Acked Write 9 Sent Done Done false
7936 2769 Acked Flush 0 Sent Done Done false
7937 2770 Acked Write 6 Sent Done Done false
7938 2771 Acked Read 7 Sent Done Done false
7939 2772 Acked Flush 0 Sent Done Done false
7940 2773 Acked Write 1 Sent Done Done false
7941 2774 Acked Write 1 Sent Done Done false
7942 2775 Acked Write 2 Sent Done Done false
7943 2776 Acked Write 10 Sent Done Done false
7944 2777 Acked Read 6 Sent Done Done false
7945 2778 Acked Write 8 Sent Done Done false
7946 2779 Acked Read 8 Sent Done Done false
7947 2780 Acked Read 7 Sent Done Done false
7948 2781 Acked Read 9 Sent Done Done false
7949 2782 Acked Write 9 Sent Done Done false
7950 2783 Acked Read 8 Sent Done Done false
7951 2784 Acked Read 8 Sent Done Done false
7952 2785 Acked Write 9 Sent Done Done false
7953 2786 Acked Read 1 Sent Done Done false
7954 2787 Acked Read 4 Sent Done Done false
7955 2788 Acked Read 3 Sent Done Done false
7956 2789 Acked Read 7 Sent Done Done false
7957 2790 Acked Write 10 Sent Done Done false
7958 2791 Acked Write 3 Sent Done Done false
7959 2792 Acked Write 2 Sent Done Done false
7960 2793 Acked Write 6 Sent Done Done false
7961 2794 Acked Read 6 Sent Done Done false
7962 2795 Acked Read 10 Sent Done Done false
7963 2796 Acked Read 7 Sent Done Done false
7964 2797 Acked Read 9 Sent Done Done false
7965 2798 Acked Read 8 Sent Done Done false
7966 2799 Acked Write 6 Sent Done Done false
7967 2800 Acked Write 3 Sent Done Done false
7968 2801 Acked Read 8 Sent Done Done false
7969 2802 Acked Write 8 Sent Done Done false
7970 2803 Acked Read 4 Sent Done Done false
7971 2804 Acked Write 9 Sent Done Done false
7972 2805 Acked Write 3 Sent Done Done false
7973 2806 Acked Read 5 Sent Done Done false
7974 2807 Acked Read 2 Sent Done Done false
7975 STATES DS:0 DS:1 DS:2 TOTAL
7976 Sent 301 0 0 301
7977 Done 0 301 301 602
7978 Skipped 0 0 0 0
7979 Error 0 0 0 0
7980 Last Flush: Some(JobId(2506)) Some(JobId(2772)) Some(JobId(2772))
7981 Downstairs last five completed: 2506 2505 2504 2503 2502
7982 Upstairs last five completed: 2807 2806 2805 2804 2803
7983 CLIENT: Up:0 ds:301 act:2
79842025-02-28T06:02:41.871ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
79852025-02-28T06:02:42.501ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
79862025-02-28T06:02:43.135ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
79872025-02-28T06:02:43.135ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
79882025-02-28T06:02:43.135ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
79892025-02-28T06:02:43.135ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
79902025-02-28T06:02:43.135ZINFOcrucible: client task is exiting = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
79912025-02-28T06:02:43.636ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
79922025-02-28T06:02:44.505ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
7993 [8810] got start action
7994 [8810] I am going to restart
7995 Starting downstairs at port 8810
7996 Make output file at "/tmp/dsc/downstairs-8810.txt"
7997 [8810][0] reports Starting
7998 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1217)
7999 [8810][0] reports Running
8000 ----------------------------------------------------------------
8001 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8002 States: Offline Active Active
8003 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8004 2507 Acked Read 3 Sent Done Done false
8005 2508 Acked Read 4 Sent Done Done false
8006 2509 Acked Write 6 Sent Done Done false
8007 2510 Acked Read 7 Sent Done Done false
8008 2511 Acked Flush 0 Sent Done Done false
8009 2512 Acked Write 6 Sent Done Done false
8010 2513 Acked Read 5 Sent Done Done false
8011 2514 Acked Write 8 Sent Done Done false
8012 2515 Acked Write 9 Sent Done Done false
8013 2516 Acked Write 8 Sent Done Done false
8014 2517 Acked Flush 0 Sent Done Done false
8015 2518 Acked Write 2 Sent Done Done false
8016 2519 Acked Write 8 Sent Done Done false
8017 2520 Acked Write 10 Sent Done Done false
8018 2521 Acked Read 5 Sent Done Done false
8019 2522 Acked Write 2 Sent Done Done false
8020 2523 Acked Write 9 Sent Done Done false
8021 2524 Acked Read 4 Sent Done Done false
8022 2525 Acked Write 7 Sent Done Done false
8023 2526 Acked Flush 0 Sent Done Done false
8024 2527 Acked Write 3 Sent Done Done false
8025 2528 Acked Read 9 Sent Done Done false
8026 2529 Acked Read 6 Sent Done Done false
8027 2530 Acked Write 3 Sent Done Done false
8028 2531 Acked Flush 0 Sent Done Done false
8029 2532 Acked Write 8 Sent Done Done false
8030 2533 Acked Write 5 Sent Done Done false
8031 2534 Acked Read 10 Sent Done Done false
8032 2535 Acked Write 7 Sent Done Done false
8033 2536 Acked Write 9 Sent Done Done false
8034 2537 Acked Read 4 Sent Done Done false
8035 2538 Acked Write 10 Sent Done Done false
8036 2539 Acked Flush 0 Sent Done Done false
8037 2540 Acked Write 3 Sent Done Done false
8038 2541 Acked Write 5 Sent Done Done false
8039 2542 Acked Read 10 Sent Done Done false
8040 2543 Acked Read 7 Sent Done Done false
8041 2544 Acked Write 10 Sent Done Done false
8042 2545 Acked Read 2 Sent Done Done false
8043 2546 Acked Read 7 Sent Done Done false
8044 2547 Acked Write 1 Sent Done Done false
8045 2548 Acked Write 4 Sent Done Done false
8046 2549 Acked Write 3 Sent Done Done false
8047 2550 Acked Read 6 Sent Done Done false
8048 2551 Acked Read 8 Sent Done Done false
8049 2552 Acked Write 5 Sent Done Done false
8050 2553 Acked Write 7 Sent Done Done false
8051 2554 Acked Read 3 Sent Done Done false
8052 2555 Acked Read 3 Sent Done Done false
8053 2556 Acked Write 4 Sent Done Done false
8054 2557 Acked Write 4 Sent Done Done false
8055 2558 Acked Read 2 Sent Done Done false
8056 2559 Acked Write 2 Sent Done Done false
8057 2560 Acked Write 1 Sent Done Done false
8058 2561 Acked Write 10 Sent Done Done false
8059 2562 Acked Write 1 Sent Done Done false
8060 2563 Acked Read 4 Sent Done Done false
8061 2564 Acked Write 6 Sent Done Done false
8062 2565 Acked Read 3 Sent Done Done false
8063 2566 Acked Write 9 Sent Done Done false
8064 2567 Acked Write 3 Sent Done Done false
8065 2568 Acked Write 9 Sent Done Done false
8066 2569 Acked Read 4 Sent Done Done false
8067 2570 Acked Write 6 Sent Done Done false
8068 2571 Acked Read 3 Sent Done Done false
8069 2572 Acked Write 7 Sent Done Done false
8070 2573 Acked Read 6 Sent Done Done false
8071 2574 Acked Write 5 Sent Done Done false
8072 2575 Acked Read 1 Sent Done Done false
8073 2576 Acked Write 3 Sent Done Done false
8074 2577 Acked Write 5 Sent Done Done false
8075 2578 Acked Read 2 Sent Done Done false
8076 2579 Acked Write 5 Sent Done Done false
8077 2580 Acked Write 4 Sent Done Done false
8078 2581 Acked Write 2 Sent Done Done false
8079 2582 Acked Read 1 Sent Done Done false
8080 2583 Acked Read 7 Sent Done Done false
8081 2584 Acked Write 3 Sent Done Done false
8082 2585 Acked Read 9 Sent Done Done false
8083 2586 Acked Flush 0 Sent Done Done false
8084 2587 Acked Flush 0 Sent Done Done false
8085 2588 Acked Write 8 Sent Done Done false
8086 2589 Acked Read 7 Sent Done Done false
8087 2590 Acked Flush 0 Sent Done Done false
8088 2591 Acked Write 2 Sent Done Done false
8089 2592 Acked Write 7 Sent Done Done false
8090 2593 Acked Write 9 Sent Done Done false
8091 2594 Acked Read 10 Sent Done Done false
8092 2595 Acked Write 10 Sent Done Done false
8093 2596 Acked Read 2 Sent Done Done false
8094 2597 Acked Write 9 Sent Done Done false
8095 2598 Acked Read 7 Sent Done Done false
8096 2599 Acked Read 1 Sent Done Done false
8097 2600 Acked Write 3 Sent Done Done false
8098 2601 Acked Read 10 Sent Done Done false
8099 2602 Acked Read 9 Sent Done Done false
8100 2603 Acked Read 1 Sent Done Done false
8101 2604 Acked Write 2 Sent Done Done false
8102 2605 Acked Read 3 Sent Done Done false
8103 2606 Acked Read 5 Sent Done Done false
8104 2607 Acked Read 7 Sent Done Done false
8105 2608 Acked Read 9 Sent Done Done false
8106 2609 Acked Write 6 Sent Done Done false
8107 2610 Acked Write 9 Sent Done Done false
8108 2611 Acked Write 6 Sent Done Done false
8109 2612 Acked Read 7 Sent Done Done false
8110 2613 Acked Flush 0 Sent Done Done false
8111 2614 Acked Write 5 Sent Done Done false
8112 2615 Acked Write 1 Sent Done Done false
8113 2616 Acked Read 6 Sent Done Done false
8114 2617 Acked Read 1 Sent Done Done false
8115 2618 Acked Read 6 Sent Done Done false
8116 2619 Acked Flush 0 Sent Done Done false
8117 2620 Acked Read 4 Sent Done Done false
8118 2621 Acked Read 7 Sent Done Done false
8119 2622 Acked Read 5 Sent Done Done false
8120 2623 Acked Read 10 Sent Done Done false
8121 2624 Acked Write 8 Sent Done Done false
8122 2625 Acked Read 3 Sent Done Done false
8123 2626 Acked Read 5 Sent Done Done false
8124 2627 Acked Read 3 Sent Done Done false
8125 2628 Acked Read 5 Sent Done Done false
8126 2629 Acked Write 1 Sent Done Done false
8127 2630 Acked Write 4 Sent Done Done false
8128 2631 Acked Read 5 Sent Done Done false
8129 2632 Acked Read 8 Sent Done Done false
8130 2633 Acked Write 10 Sent Done Done false
8131 2634 Acked Write 1 Sent Done Done false
8132 2635 Acked Read 1 Sent Done Done false
8133 2636 Acked Write 9 Sent Done Done false
8134 2637 Acked Read 8 Sent Done Done false
8135 2638 Acked Write 3 Sent Done Done false
8136 2639 Acked Write 7 Sent Done Done false
8137 2640 Acked Read 2 Sent Done Done false
8138 2641 Acked Read 1 Sent Done Done false
8139 2642 Acked Write 2 Sent Done Done false
8140 2643 Acked Read 2 Sent Done Done false
8141 2644 Acked Write 6 Sent Done Done false
8142 2645 Acked Read 2 Sent Done Done false
8143 2646 Acked Read 10 Sent Done Done false
8144 2647 Acked Read 9 Sent Done Done false
8145 2648 Acked Write 9 Sent Done Done false
8146 2649 Acked Read 3 Sent Done Done false
8147 2650 Acked Read 4 Sent Done Done false
8148 2651 Acked Write 5 Sent Done Done false
8149 2652 Acked Write 8 Sent Done Done false
8150 2653 Acked Flush 0 Sent Done Done false
8151 2654 Acked Write 1 Sent Done Done false
8152 2655 Acked Read 5 Sent Done Done false
8153 2656 Acked Write 8 Sent Done Done false
8154 2657 Acked Read 10 Sent Done Done false
8155 2658 Acked Write 3 Sent Done Done false
8156 2659 Acked Read 4 Sent Done Done false
8157 2660 Acked Read 7 Sent Done Done false
8158 2661 Acked Write 7 Sent Done Done false
8159 2662 Acked Write 3 Sent Done Done false
8160 2663 Acked Read 7 Sent Done Done false
8161 2664 Acked Write 8 Sent Done Done false
8162 2665 Acked Write 9 Sent Done Done false
8163 2666 Acked Write 10 Sent Done Done false
8164 2667 Acked Read 8 Sent Done Done false
8165 2668 Acked Write 6 Sent Done Done false
8166 2669 Acked Flush 0 Sent Done Done false
8167 2670 Acked Read 10 Sent Done Done false
8168 2671 Acked Write 5 Sent Done Done false
8169 2672 Acked Write 2 Sent Done Done false
8170 2673 Acked Read 5 Sent Done Done false
8171 2674 Acked Write 3 Sent Done Done false
8172 2675 Acked Read 5 Sent Done Done false
8173 2676 Acked Read 2 Sent Done Done false
8174 2677 Acked Read 10 Sent Done Done false
8175 2678 Acked Write 1 Sent Done Done false
8176 2679 Acked Read 8 Sent Done Done false
8177 2680 Acked Flush 0 Sent Done Done false
8178 2681 Acked Write 4 Sent Done Done false
8179 2682 Acked Write 8 Sent Done Done false
8180 2683 Acked Flush 0 Sent Done Done false
8181 2684 Acked Write 7 Sent Done Done false
8182 2685 Acked Read 6 Sent Done Done false
8183 2686 Acked Read 8 Sent Done Done false
8184 2687 Acked Write 1 Sent Done Done false
8185 2688 Acked Read 7 Sent Done Done false
8186 2689 Acked Read 1 Sent Done Done false
8187 2690 Acked Read 9 Sent Done Done false
8188 2691 Acked Read 6 Sent Done Done false
8189 2692 Acked Read 7 Sent Done Done false
8190 2693 Acked Read 4 Sent Done Done false
8191 2694 Acked Read 2 Sent Done Done false
8192 2695 Acked Read 1 Sent Done Done false
8193 2696 Acked Write 3 Sent Done Done false
8194 2697 Acked Read 9 Sent Done Done false
8195 2698 Acked Flush 0 Sent Done Done false
8196 2699 Acked Write 6 Sent Done Done false
8197 2700 Acked Write 7 Sent Done Done false
8198 2701 Acked Write 1 Sent Done Done false
8199 2702 Acked Read 4 Sent Done Done false
8200 2703 Acked Flush 0 Sent Done Done false
8201 2704 Acked Read 3 Sent Done Done false
8202 2705 Acked Read 8 Sent Done Done false
8203 2706 Acked Write 3 Sent Done Done false
8204 2707 Acked Write 2 Sent Done Done false
8205 2708 Acked Read 4 Sent Done Done false
8206 2709 Acked Write 2 Sent Done Done false
8207 2710 Acked Read 4 Sent Done Done false
8208 2711 Acked Write 1 Sent Done Done false
8209 2712 Acked Flush 0 Sent Done Done false
8210 2713 Acked Read 3 Sent Done Done false
8211 2714 Acked Write 2 Sent Done Done false
8212 2715 Acked Write 9 Sent Done Done false
8213 2716 Acked Read 3 Sent Done Done false
8214 2717 Acked Read 6 Sent Done Done false
8215 2718 Acked Write 6 Sent Done Done false
8216 2719 Acked Read 3 Sent Done Done false
8217 2720 Acked Read 1 Sent Done Done false
8218 2721 Acked Write 5 Sent Done Done false
8219 2722 Acked Read 4 Sent Done Done false
8220 2723 Acked Write 9 Sent Done Done false
8221 2724 Acked Read 4 Sent Done Done false
8222 2725 Acked Flush 0 Sent Done Done false
8223 2726 Acked Read 5 Sent Done Done false
8224 2727 Acked Write 9 Sent Done Done false
8225 2728 Acked Write 8 Sent Done Done false
8226 2729 Acked Flush 0 Sent Done Done false
8227 2730 Acked Write 2 Sent Done Done false
8228 2731 Acked Write 5 Sent Done Done false
8229 2732 Acked Read 5 Sent Done Done false
8230 2733 Acked Write 4 Sent Done Done false
8231 2734 Acked Read 9 Sent Done Done false
8232 2735 Acked Write 6 Sent Done Done false
8233 2736 Acked Write 6 Sent Done Done false
8234 2737 Acked Read 3 Sent Done Done false
8235 2738 Acked Write 2 Sent Done Done false
8236 2739 Acked Read 2 Sent Done Done false
8237 2740 Acked Read 1 Sent Done Done false
8238 2741 Acked Flush 0 Sent Done Done false
8239 2742 Acked Write 3 Sent Done Done false
8240 2743 Acked Read 5 Sent Done Done false
8241 2744 Acked Flush 0 Sent Done Done false
8242 2745 Acked Write 4 Sent Done Done false
8243 2746 Acked Write 6 Sent Done Done false
8244 2747 Acked Write 5 Sent Done Done false
8245 2748 Acked Write 7 Sent Done Done false
8246 2749 Acked Read 3 Sent Done Done false
8247 2750 Acked Write 4 Sent Done Done false
8248 2751 Acked Read 3 Sent Done Done false
8249 2752 Acked Write 10 Sent Done Done false
8250 2753 Acked Write 6 Sent Done Done false
8251 2754 Acked Read 7 Sent Done Done false
8252 2755 Acked Write 7 Sent Done Done false
8253 2756 Acked Write 6 Sent Done Done false
8254 2757 Acked Write 4 Sent Done Done false
8255 2758 Acked Read 6 Sent Done Done false
8256 2759 Acked Read 6 Sent Done Done false
8257 2760 Acked Write 6 Sent Done Done false
8258 2761 Acked Write 3 Sent Done Done false
8259 2762 Acked Read 3 Sent Done Done false
8260 2763 Acked Read 9 Sent Done Done false
8261 2764 Acked Read 9 Sent Done Done false
8262 2765 Acked Write 9 Sent Done Done false
8263 2766 Acked Read 1 Sent Done Done false
8264 2767 Acked Write 9 Sent Done Done false
8265 2768 Acked Write 9 Sent Done Done false
8266 2769 Acked Flush 0 Sent Done Done false
8267 2770 Acked Write 6 Sent Done Done false
8268 2771 Acked Read 7 Sent Done Done false
8269 2772 Acked Flush 0 Sent Done Done false
8270 2773 Acked Write 1 Sent Done Done false
8271 2774 Acked Write 1 Sent Done Done false
8272 2775 Acked Write 2 Sent Done Done false
8273 2776 Acked Write 10 Sent Done Done false
8274 2777 Acked Read 6 Sent Done Done false
8275 2778 Acked Write 8 Sent Done Done false
8276 2779 Acked Read 8 Sent Done Done false
8277 2780 Acked Read 7 Sent Done Done false
8278 2781 Acked Read 9 Sent Done Done false
8279 2782 Acked Write 9 Sent Done Done false
8280 2783 Acked Read 8 Sent Done Done false
8281 2784 Acked Read 8 Sent Done Done false
8282 2785 Acked Write 9 Sent Done Done false
8283 2786 Acked Read 1 Sent Done Done false
8284 2787 Acked Read 4 Sent Done Done false
8285 2788 Acked Read 3 Sent Done Done false
8286 2789 Acked Read 7 Sent Done Done false
8287 2790 Acked Write 10 Sent Done Done false
8288 2791 Acked Write 3 Sent Done Done false
8289 2792 Acked Write 2 Sent Done Done false
8290 2793 Acked Write 6 Sent Done Done false
8291 2794 Acked Read 6 Sent Done Done false
8292 2795 Acked Read 10 Sent Done Done false
8293 2796 Acked Read 7 Sent Done Done false
8294 2797 Acked Read 9 Sent Done Done false
8295 2798 Acked Read 8 Sent Done Done false
8296 2799 Acked Write 6 Sent Done Done false
8297 2800 Acked Write 3 Sent Done Done false
8298 2801 Acked Read 8 Sent Done Done false
8299 2802 Acked Write 8 Sent Done Done false
8300 2803 Acked Read 4 Sent Done Done false
8301 2804 Acked Write 9 Sent Done Done false
8302 2805 Acked Write 3 Sent Done Done false
8303 2806 Acked Read 5 Sent Done Done false
8304 2807 Acked Read 2 Sent Done Done false
8305 STATES DS:0 DS:1 DS:2 TOTAL
8306 Sent 301 0 0 301
8307 Done 0 301 301 602
8308 Skipped 0 0 0 0
8309 Error 0 0 0 0
8310 Last Flush: Some(JobId(2506)) Some(JobId(2772)) Some(JobId(2772))
8311 Downstairs last five completed: 2506 2505 2504 2503 2502
8312 Upstairs last five completed: 2807 2806 2805 2804 2803
8313 CLIENT: Up:0 ds:301 act:2
83142025-02-28T06:02:45.873ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
83152025-02-28T06:02:46.508ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
83162025-02-28T06:02:47.509ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
83172025-02-28T06:02:48.511ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
8318 ----------------------------------------------------------------
8319 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8320 States: Offline Active Active
8321 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8322 2507 Acked Read 3 Sent Done Done false
8323 2508 Acked Read 4 Sent Done Done false
8324 2509 Acked Write 6 Sent Done Done false
8325 2510 Acked Read 7 Sent Done Done false
8326 2511 Acked Flush 0 Sent Done Done false
8327 2512 Acked Write 6 Sent Done Done false
8328 2513 Acked Read 5 Sent Done Done false
8329 2514 Acked Write 8 Sent Done Done false
8330 2515 Acked Write 9 Sent Done Done false
8331 2516 Acked Write 8 Sent Done Done false
8332 2517 Acked Flush 0 Sent Done Done false
8333 2518 Acked Write 2 Sent Done Done false
8334 2519 Acked Write 8 Sent Done Done false
8335 2520 Acked Write 10 Sent Done Done false
8336 2521 Acked Read 5 Sent Done Done false
8337 2522 Acked Write 2 Sent Done Done false
8338 2523 Acked Write 9 Sent Done Done false
8339 2524 Acked Read 4 Sent Done Done false
8340 2525 Acked Write 7 Sent Done Done false
8341 2526 Acked Flush 0 Sent Done Done false
8342 2527 Acked Write 3 Sent Done Done false
8343 2528 Acked Read 9 Sent Done Done false
8344 2529 Acked Read 6 Sent Done Done false
8345 2530 Acked Write 3 Sent Done Done false
8346 2531 Acked Flush 0 Sent Done Done false
8347 2532 Acked Write 8 Sent Done Done false
8348 2533 Acked Write 5 Sent Done Done false
8349 2534 Acked Read 10 Sent Done Done false
8350 2535 Acked Write 7 Sent Done Done false
8351 2536 Acked Write 9 Sent Done Done false
8352 2537 Acked Read 4 Sent Done Done false
8353 2538 Acked Write 10 Sent Done Done false
8354 2539 Acked Flush 0 Sent Done Done false
8355 2540 Acked Write 3 Sent Done Done false
8356 2541 Acked Write 5 Sent Done Done false
8357 2542 Acked Read 10 Sent Done Done false
8358 2543 Acked Read 7 Sent Done Done false
8359 2544 Acked Write 10 Sent Done Done false
8360 2545 Acked Read 2 Sent Done Done false
8361 2546 Acked Read 7 Sent Done Done false
8362 2547 Acked Write 1 Sent Done Done false
8363 2548 Acked Write 4 Sent Done Done false
8364 2549 Acked Write 3 Sent Done Done false
8365 2550 Acked Read 6 Sent Done Done false
8366 2551 Acked Read 8 Sent Done Done false
8367 2552 Acked Write 5 Sent Done Done false
8368 2553 Acked Write 7 Sent Done Done false
8369 2554 Acked Read 3 Sent Done Done false
8370 2555 Acked Read 3 Sent Done Done false
8371 2556 Acked Write 4 Sent Done Done false
8372 2557 Acked Write 4 Sent Done Done false
8373 2558 Acked Read 2 Sent Done Done false
8374 2559 Acked Write 2 Sent Done Done false
8375 2560 Acked Write 1 Sent Done Done false
8376 2561 Acked Write 10 Sent Done Done false
8377 2562 Acked Write 1 Sent Done Done false
8378 2563 Acked Read 4 Sent Done Done false
8379 2564 Acked Write 6 Sent Done Done false
8380 2565 Acked Read 3 Sent Done Done false
8381 2566 Acked Write 9 Sent Done Done false
8382 2567 Acked Write 3 Sent Done Done false
8383 2568 Acked Write 9 Sent Done Done false
8384 2569 Acked Read 4 Sent Done Done false
8385 2570 Acked Write 6 Sent Done Done false
8386 2571 Acked Read 3 Sent Done Done false
8387 2572 Acked Write 7 Sent Done Done false
8388 2573 Acked Read 6 Sent Done Done false
8389 2574 Acked Write 5 Sent Done Done false
8390 2575 Acked Read 1 Sent Done Done false
8391 2576 Acked Write 3 Sent Done Done false
8392 2577 Acked Write 5 Sent Done Done false
8393 2578 Acked Read 2 Sent Done Done false
8394 2579 Acked Write 5 Sent Done Done false
8395 2580 Acked Write 4 Sent Done Done false
8396 2581 Acked Write 2 Sent Done Done false
8397 2582 Acked Read 1 Sent Done Done false
8398 2583 Acked Read 7 Sent Done Done false
8399 2584 Acked Write 3 Sent Done Done false
8400 2585 Acked Read 9 Sent Done Done false
8401 2586 Acked Flush 0 Sent Done Done false
8402 2587 Acked Flush 0 Sent Done Done false
8403 2588 Acked Write 8 Sent Done Done false
8404 2589 Acked Read 7 Sent Done Done false
8405 2590 Acked Flush 0 Sent Done Done false
8406 2591 Acked Write 2 Sent Done Done false
8407 2592 Acked Write 7 Sent Done Done false
8408 2593 Acked Write 9 Sent Done Done false
8409 2594 Acked Read 10 Sent Done Done false
8410 2595 Acked Write 10 Sent Done Done false
8411 2596 Acked Read 2 Sent Done Done false
8412 2597 Acked Write 9 Sent Done Done false
8413 2598 Acked Read 7 Sent Done Done false
8414 2599 Acked Read 1 Sent Done Done false
8415 2600 Acked Write 3 Sent Done Done false
8416 2601 Acked Read 10 Sent Done Done false
8417 2602 Acked Read 9 Sent Done Done false
8418 2603 Acked Read 1 Sent Done Done false
8419 2604 Acked Write 2 Sent Done Done false
8420 2605 Acked Read 3 Sent Done Done false
8421 2606 Acked Read 5 Sent Done Done false
8422 2607 Acked Read 7 Sent Done Done false
8423 2608 Acked Read 9 Sent Done Done false
8424 2609 Acked Write 6 Sent Done Done false
8425 2610 Acked Write 9 Sent Done Done false
8426 2611 Acked Write 6 Sent Done Done false
8427 2612 Acked Read 7 Sent Done Done false
8428 2613 Acked Flush 0 Sent Done Done false
8429 2614 Acked Write 5 Sent Done Done false
8430 2615 Acked Write 1 Sent Done Done false
8431 2616 Acked Read 6 Sent Done Done false
8432 2617 Acked Read 1 Sent Done Done false
8433 2618 Acked Read 6 Sent Done Done false
8434 2619 Acked Flush 0 Sent Done Done false
8435 2620 Acked Read 4 Sent Done Done false
8436 2621 Acked Read 7 Sent Done Done false
8437 2622 Acked Read 5 Sent Done Done false
8438 2623 Acked Read 10 Sent Done Done false
8439 2624 Acked Write 8 Sent Done Done false
8440 2625 Acked Read 3 Sent Done Done false
8441 2626 Acked Read 5 Sent Done Done false
8442 2627 Acked Read 3 Sent Done Done false
8443 2628 Acked Read 5 Sent Done Done false
8444 2629 Acked Write 1 Sent Done Done false
8445 2630 Acked Write 4 Sent Done Done false
8446 2631 Acked Read 5 Sent Done Done false
8447 2632 Acked Read 8 Sent Done Done false
8448 2633 Acked Write 10 Sent Done Done false
8449 2634 Acked Write 1 Sent Done Done false
8450 2635 Acked Read 1 Sent Done Done false
8451 2636 Acked Write 9 Sent Done Done false
8452 2637 Acked Read 8 Sent Done Done false
8453 2638 Acked Write 3 Sent Done Done false
8454 2639 Acked Write 7 Sent Done Done false
8455 2640 Acked Read 2 Sent Done Done false
8456 2641 Acked Read 1 Sent Done Done false
8457 2642 Acked Write 2 Sent Done Done false
8458 2643 Acked Read 2 Sent Done Done false
8459 2644 Acked Write 6 Sent Done Done false
8460 2645 Acked Read 2 Sent Done Done false
8461 2646 Acked Read 10 Sent Done Done false
8462 2647 Acked Read 9 Sent Done Done false
8463 2648 Acked Write 9 Sent Done Done false
8464 2649 Acked Read 3 Sent Done Done false
8465 2650 Acked Read 4 Sent Done Done false
8466 2651 Acked Write 5 Sent Done Done false
8467 2652 Acked Write 8 Sent Done Done false
8468 2653 Acked Flush 0 Sent Done Done false
8469 2654 Acked Write 1 Sent Done Done false
8470 2655 Acked Read 5 Sent Done Done false
8471 2656 Acked Write 8 Sent Done Done false
8472 2657 Acked Read 10 Sent Done Done false
8473 2658 Acked Write 3 Sent Done Done false
8474 2659 Acked Read 4 Sent Done Done false
8475 2660 Acked Read 7 Sent Done Done false
8476 2661 Acked Write 7 Sent Done Done false
8477 2662 Acked Write 3 Sent Done Done false
8478 2663 Acked Read 7 Sent Done Done false
8479 2664 Acked Write 8 Sent Done Done false
8480 2665 Acked Write 9 Sent Done Done false
8481 2666 Acked Write 10 Sent Done Done false
8482 2667 Acked Read 8 Sent Done Done false
8483 2668 Acked Write 6 Sent Done Done false
8484 2669 Acked Flush 0 Sent Done Done false
8485 2670 Acked Read 10 Sent Done Done false
8486 2671 Acked Write 5 Sent Done Done false
8487 2672 Acked Write 2 Sent Done Done false
8488 2673 Acked Read 5 Sent Done Done false
8489 2674 Acked Write 3 Sent Done Done false
8490 2675 Acked Read 5 Sent Done Done false
8491 2676 Acked Read 2 Sent Done Done false
8492 2677 Acked Read 10 Sent Done Done false
8493 2678 Acked Write 1 Sent Done Done false
8494 2679 Acked Read 8 Sent Done Done false
8495 2680 Acked Flush 0 Sent Done Done false
8496 2681 Acked Write 4 Sent Done Done false
8497 2682 Acked Write 8 Sent Done Done false
8498 2683 Acked Flush 0 Sent Done Done false
8499 2684 Acked Write 7 Sent Done Done false
8500 2685 Acked Read 6 Sent Done Done false
8501 2686 Acked Read 8 Sent Done Done false
8502 2687 Acked Write 1 Sent Done Done false
8503 2688 Acked Read 7 Sent Done Done false
8504 2689 Acked Read 1 Sent Done Done false
8505 2690 Acked Read 9 Sent Done Done false
8506 2691 Acked Read 6 Sent Done Done false
8507 2692 Acked Read 7 Sent Done Done false
8508 2693 Acked Read 4 Sent Done Done false
8509 2694 Acked Read 2 Sent Done Done false
8510 2695 Acked Read 1 Sent Done Done false
8511 2696 Acked Write 3 Sent Done Done false
8512 2697 Acked Read 9 Sent Done Done false
8513 2698 Acked Flush 0 Sent Done Done false
8514 2699 Acked Write 6 Sent Done Done false
8515 2700 Acked Write 7 Sent Done Done false
8516 2701 Acked Write 1 Sent Done Done false
8517 2702 Acked Read 4 Sent Done Done false
8518 2703 Acked Flush 0 Sent Done Done false
8519 2704 Acked Read 3 Sent Done Done false
8520 2705 Acked Read 8 Sent Done Done false
8521 2706 Acked Write 3 Sent Done Done false
8522 2707 Acked Write 2 Sent Done Done false
8523 2708 Acked Read 4 Sent Done Done false
8524 2709 Acked Write 2 Sent Done Done false
8525 2710 Acked Read 4 Sent Done Done false
8526 2711 Acked Write 1 Sent Done Done false
8527 2712 Acked Flush 0 Sent Done Done false
8528 2713 Acked Read 3 Sent Done Done false
8529 2714 Acked Write 2 Sent Done Done false
8530 2715 Acked Write 9 Sent Done Done false
8531 2716 Acked Read 3 Sent Done Done false
8532 2717 Acked Read 6 Sent Done Done false
8533 2718 Acked Write 6 Sent Done Done false
8534 2719 Acked Read 3 Sent Done Done false
8535 2720 Acked Read 1 Sent Done Done false
8536 2721 Acked Write 5 Sent Done Done false
8537 2722 Acked Read 4 Sent Done Done false
8538 2723 Acked Write 9 Sent Done Done false
8539 2724 Acked Read 4 Sent Done Done false
8540 2725 Acked Flush 0 Sent Done Done false
8541 2726 Acked Read 5 Sent Done Done false
8542 2727 Acked Write 9 Sent Done Done false
8543 2728 Acked Write 8 Sent Done Done false
8544 2729 Acked Flush 0 Sent Done Done false
8545 2730 Acked Write 2 Sent Done Done false
8546 2731 Acked Write 5 Sent Done Done false
8547 2732 Acked Read 5 Sent Done Done false
8548 2733 Acked Write 4 Sent Done Done false
8549 2734 Acked Read 9 Sent Done Done false
8550 2735 Acked Write 6 Sent Done Done false
8551 2736 Acked Write 6 Sent Done Done false
8552 2737 Acked Read 3 Sent Done Done false
8553 2738 Acked Write 2 Sent Done Done false
8554 2739 Acked Read 2 Sent Done Done false
8555 2740 Acked Read 1 Sent Done Done false
8556 2741 Acked Flush 0 Sent Done Done false
8557 2742 Acked Write 3 Sent Done Done false
8558 2743 Acked Read 5 Sent Done Done false
8559 2744 Acked Flush 0 Sent Done Done false
8560 2745 Acked Write 4 Sent Done Done false
8561 2746 Acked Write 6 Sent Done Done false
8562 2747 Acked Write 5 Sent Done Done false
8563 2748 Acked Write 7 Sent Done Done false
8564 2749 Acked Read 3 Sent Done Done false
8565 2750 Acked Write 4 Sent Done Done false
8566 2751 Acked Read 3 Sent Done Done false
8567 2752 Acked Write 10 Sent Done Done false
8568 2753 Acked Write 6 Sent Done Done false
8569 2754 Acked Read 7 Sent Done Done false
8570 2755 Acked Write 7 Sent Done Done false
8571 2756 Acked Write 6 Sent Done Done false
8572 2757 Acked Write 4 Sent Done Done false
8573 2758 Acked Read 6 Sent Done Done false
8574 2759 Acked Read 6 Sent Done Done false
8575 2760 Acked Write 6 Sent Done Done false
8576 2761 Acked Write 3 Sent Done Done false
8577 2762 Acked Read 3 Sent Done Done false
8578 2763 Acked Read 9 Sent Done Done false
8579 2764 Acked Read 9 Sent Done Done false
8580 2765 Acked Write 9 Sent Done Done false
8581 2766 Acked Read 1 Sent Done Done false
8582 2767 Acked Write 9 Sent Done Done false
8583 2768 Acked Write 9 Sent Done Done false
8584 2769 Acked Flush 0 Sent Done Done false
8585 2770 Acked Write 6 Sent Done Done false
8586 2771 Acked Read 7 Sent Done Done false
8587 2772 Acked Flush 0 Sent Done Done false
8588 2773 Acked Write 1 Sent Done Done false
8589 2774 Acked Write 1 Sent Done Done false
8590 2775 Acked Write 2 Sent Done Done false
8591 2776 Acked Write 10 Sent Done Done false
8592 2777 Acked Read 6 Sent Done Done false
8593 2778 Acked Write 8 Sent Done Done false
8594 2779 Acked Read 8 Sent Done Done false
8595 2780 Acked Read 7 Sent Done Done false
8596 2781 Acked Read 9 Sent Done Done false
8597 2782 Acked Write 9 Sent Done Done false
8598 2783 Acked Read 8 Sent Done Done false
8599 2784 Acked Read 8 Sent Done Done false
8600 2785 Acked Write 9 Sent Done Done false
8601 2786 Acked Read 1 Sent Done Done false
8602 2787 Acked Read 4 Sent Done Done false
8603 2788 Acked Read 3 Sent Done Done false
8604 2789 Acked Read 7 Sent Done Done false
8605 2790 Acked Write 10 Sent Done Done false
8606 2791 Acked Write 3 Sent Done Done false
8607 2792 Acked Write 2 Sent Done Done false
8608 2793 Acked Write 6 Sent Done Done false
8609 2794 Acked Read 6 Sent Done Done false
8610 2795 Acked Read 10 Sent Done Done false
8611 2796 Acked Read 7 Sent Done Done false
8612 2797 Acked Read 9 Sent Done Done false
8613 2798 Acked Read 8 Sent Done Done false
8614 2799 Acked Write 6 Sent Done Done false
8615 2800 Acked Write 3 Sent Done Done false
8616 2801 Acked Read 8 Sent Done Done false
8617 2802 Acked Write 8 Sent Done Done false
8618 2803 Acked Read 4 Sent Done Done false
8619 2804 Acked Write 9 Sent Done Done false
8620 2805 Acked Write 3 Sent Done Done false
8621 2806 Acked Read 5 Sent Done Done false
8622 2807 Acked Read 2 Sent Done Done false
8623 STATES DS:0 DS:1 DS:2 TOTAL
8624 Sent 301 0 0 301
8625 Done 0 301 301 602
8626 Skipped 0 0 0 0
8627 Error 0 0 0 0
8628 Last Flush: Some(JobId(2506)) Some(JobId(2772)) Some(JobId(2772))
8629 Downstairs last five completed: 2506 2505 2504 2503 2502
8630 Upstairs last five completed: 2807 2806 2805 2804 2803
8631 CLIENT: Up:0 ds:301 act:2
86322025-02-28T06:02:49.875ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
86332025-02-28T06:02:50.514ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
86342025-02-28T06:02:51.516ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
86352025-02-28T06:02:52.518ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
86362025-02-28T06:02:53.136ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
86372025-02-28T06:02:53.136ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46410) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
86382025-02-28T06:02:53.137ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
86392025-02-28T06:02:53.137ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
86402025-02-28T06:02:53.137ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
86412025-02-28T06:02:53.137ZINFOcrucible: send last flush ID to this DS: Some(JobId(2506)) = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
86422025-02-28T06:02:53.137ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
86432025-02-28T06:02:53.137ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2506)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
86442025-02-28T06:02:53.138ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2506)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
8645 ----------------------------------------------------------------
8646 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:35
8647 States: Active Active Active
8648 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8649 2773 Acked Write 1 Done Done Done true
8650 2774 Acked Write 1 Done Done Done true
8651 2775 Acked Write 2 Done Done Done true
8652 2776 Acked Write 10 Done Done Done true
8653 2777 Acked Read 6 Done Done Done true
8654 2778 Acked Write 8 Done Done Done true
8655 2779 Acked Read 8 Done Done Done true
8656 2780 Acked Read 7 Done Done Done true
8657 2781 Acked Read 9 Done Done Done true
8658 2782 Acked Write 9 Done Done Done true
8659 2783 Acked Read 8 Done Done Done true
8660 2784 Acked Read 8 Done Done Done true
8661 2785 Acked Write 9 Done Done Done true
8662 2786 Acked Read 1 Done Done Done true
8663 2787 Acked Read 4 Done Done Done true
8664 2788 Acked Read 3 Done Done Done true
8665 2789 Acked Read 7 Done Done Done true
8666 2790 Acked Write 10 Done Done Done true
8667 2791 Acked Write 3 Done Done Done true
8668 2792 Acked Write 2 Done Done Done true
8669 2793 Acked Write 6 Done Done Done true
8670 2794 Acked Read 6 Done Done Done true
8671 2795 Acked Read 10 Done Done Done true
8672 2796 Acked Read 7 Done Done Done true
8673 2797 Acked Read 9 Done Done Done true
8674 2798 Acked Read 8 Done Done Done true
8675 2799 Acked Write 6 Done Done Done true
8676 2800 Acked Write 3 Done Done Done true
8677 2801 Acked Read 8 Done Done Done true
8678 2802 Acked Write 8 Done Done Done true
8679 2803 Acked Read 4 Done Done Done true
8680 2804 Acked Write 9 Done Done Done true
8681 2805 Acked Write 3 Done Done Done true
8682 2806 Acked Read 5 Done Done Done true
8683 2807 Acked Read 2 Done Done Done true
8684 STATES DS:0 DS:1 DS:2 TOTAL
8685 Sent 0 0 0 0
8686 Done 35 35 35 105
8687 Skipped 0 0 0 0
8688 Error 0 0 0 0
8689 Last Flush: Some(JobId(2772)) Some(JobId(2772)) Some(JobId(2772))
8690 Downstairs last five completed: 2772 2771 2770 2769 2768
8691 Upstairs last five completed: 2807 2806 2805 2804 2803
8692 CLIENT: Up:0 ds:35 act:3
8693 ----------------------------------------------------------------
8694 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8695 Downstairs last five completed: 2808 2807 2806 2805 2804
8696 Upstairs last five completed: 2808 2807 2806 2805 2804
8697 CLIENT: Up:0 ds:0 act:3
8698 Replay: All jobs finished
8699 Main task has work to do, go find it
8700 got dsc Stop(0)
8701 stop 0
8702 Feb 28 06:02:57.378[8810] Got stop action so:false kr:false
8703 INFO request completed, latency_us: 362, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 80e0812b-96f7-41dc-bae2-bdf88a89df44, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
87042025-02-28T06:02:57.379ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
87052025-02-28T06:02:57.379ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
87062025-02-28T06:02:57.379ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
8707 Feb 28 06:02:57.379 INFO request completed, latency_us: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:02:57.379518811Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
8708 118, response_code: 200, uri: /state/cid/0, method: GET, req_id: 61bab362-d83e-436e-893a-7e8e80ddabf8, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
8709 [8810] Exited with: ExitStatus(unix_wait_status(9))
8710 [8810][0] reports Exit
8711 Feb 28 06:03:01.382 INFO request completed, latency_us: 345, response_code: 200, uri: /state/cid/0, method: GET, req_id: 083cdd87-dfe7-46a1-97e1-1139f43fcf38, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
8712 001/300 Write block 4211 len 28672 data: 3 3 3 2 2 2 2
8713 002/300 Write block 1709 len 24576 data: 3 3 3 3 3 3
8714 003/300 Write block 2430 len 4096 data: 2
8715 004/300 Read block 4934 len 20480
8716 005/300 Write block 2411 len 8192 data: 2 2
8717 006/300 Read block 1664 len 4096
8718 007/300 Read block 3183 len 24576
8719 008/300 Read block 1345 len 8192
8720 009/300 Read block 3309 len 32768
8721 010/300 Read block 4698 len 24576
8722 011/300 Read block 2474 len 16384
8723 012/300 Write block 1774 len 8192 data: 2 2
8724 013/300 Read block 4613 len 12288
8725 014/300 Read block 3185 len 32768
8726 015/300 Read block 4482 len 24576
8727 016/300 Write block 4733 len 8192 data: 3 3
8728 017/300 Flush
8729 018/300 Read block 3353 len 20480
8730 019/300 Read block 1412 len 12288
8731 020/300 Read block 2598 len 12288
8732 021/300 Read block 3211 len 16384
8733 022/300 Read block 1451 len 24576
8734 023/300 Read block 3719 len 4096
8735 024/300 Read block 963 len 36864
8736 025/300 Write block 333 len 36864 data: 2 2 2 3 4 4 3 3 4
8737 026/300 Read block 2830 len 8192
8738 027/300 Read block 194 len 8192
8739 028/300 Write block 91 len 32768 data: 2 2 2 2 2 2 2 2
8740 029/300 Read block 2664 len 12288
8741 030/300 Read block 1584 len 28672
8742 031/300 Write block 398 len 16384 data: 2 2 2 2
8743 032/300 Write block 3903 len 28672 data: 3 3 3 3 4 3 3
8744 033/300 Read block 4582 len 40960
8745 034/300 Read block 2791 len 16384
8746 035/300 Write block 508 len 16384 data: 2 3 2 2
8747 036/300 Write block 2114 len 20480 data: 3 3 3 4 4
8748 037/300 Read block 985 len 36864
8749 038/300 Read block 2450 len 36864
8750 039/300 Write block 92 len 16384 data: 3 3 3 3
8751 040/300 Read block 4218 len 36864
8752 041/300 Read block 2306 len 28672
8753 042/300 Read block 549 len 20480
8754 043/300 Write block 2442 len 32768 data: 2 2 2 3 3 3 2 2
8755 044/300 Flush
8756 045/300 Write block 3107 len 24576 data: 3 4 4 4 4 4
8757 046/300 Read block 4732 len 12288
8758 047/300 Write block 772 len 4096 data: 2
8759 048/300 Read block 2356 len 20480
8760 049/300 Write block 2061 len 20480 data: 2 2 2 3 3
8761 050/300 Read block 125 len 40960
8762 051/300 Write block 3436 len 12288 data: 4 4 4
8763 052/300 Flush
8764 053/300 Read block 4098 len 20480
8765 054/300 Read block 2351 len 24576
8766 055/300 Read block 4985 len 32768
8767 056/300 Write block 4203 len 8192 data: 2 3
8768 057/300 Read block 3175 len 8192
8769 058/300 Flush
8770 059/300 Flush
8771 060/300 Write block 1440 len 24576 data: 3 4 4 4 4 4
8772 061/300 Read block 794 len 24576
8773 062/300 Read block 609 len 20480
8774 063/300 Read block 1652 len 36864
8775 064/300 Write block 796 len 16384 data: 4 4 4 3
8776 065/300 Read block 4094 len 24576
8777 066/300 Write block 3249 len 16384 data: 3 3 2 2
8778 067/300 Flush
8779 068/300 Read block 4830 len 36864
8780 069/300 Write block 4903 len 8192 data: 4 4
8781 070/300 Read block 4848 len 12288
8782 071/300 Read block 1190 len 36864
8783 072/300 Read block 1083 len 28672
8784 073/300 Read block 2856 len 36864
8785 074/300 Read block 880 len 40960
8786 075/300 Read block 3022 len 4096
8787 076/300 Write block 4095 len 40960 data: 3 2 2 2 2 2 3 3 3 3
8788 077/300 Read block 3870 len 28672
8789 078/300 Read block 864 len 12288
8790 079/300 Write block 2452 len 36864 data: 2 2 3 4 3 3 3 2 2
8791 080/300 Write block 1485 len 36864 data: 2 2 2 2 3 3 3 3 2
8792 081/300 Write block 2740 len 28672 data: 3 3 3 3 3 3 3
8793 082/300 Write block 1491 len 8192 data: 4 4
8794 083/300 Read block 1318 len 8192
8795 084/300 Flush
8796 085/300 Read block 4069 len 4096
8797 086/300 Write block 4357 len 12288 data: 3 3 3
8798 087/300 Write block 195 len 24576 data: 4 4 3 3 3 3
8799 088/300 Write block 25 len 8192 data: 4 3
8800 089/300 Write block 3096 len 20480 data: 2 2 3 3 4
8801 090/300 Read block 4956 len 32768
8802 091/300 Read block 4235 len 36864
8803 092/300 Read block 4767 len 4096
8804 093/300 Read block 3576 len 36864
8805 094/300 Write block 3986 len 8192 data: 2 2
8806 095/300 Write block 2468 len 28672 data: 2 2 3 4 4 3 3
8807 096/300 Write block 2304 len 12288 data: 2 2 3
8808 097/300 Write block 4961 len 8192 data: 2 2
8809 098/300 Read block 2372 len 12288
8810 099/300 Write block 4832 len 28672 data: 4 2 2 3 3 4 4
8811 100/300 Write block 897 len 8192 data: 3 3
8812 101/300 Read block 4843 len 24576
8813 102/300 Write block 4307 len 4096 data: 2
8814 103/300 Read block 3665 len 32768
8815 104/300 Read block 1236 len 40960
8816 105/300 Write block 94 len 8192 data: 4 4
8817 106/300 Write block 3875 len 36864 data: 5 5 4 4 3 3 2 3 4
8818 107/300 Read block 3233 len 24576
8819 108/300 Read block 3471 len 40960
8820 109/300 Write block 3293 len 40960 data: 3 3 3 4 6 5 4 3 3 4
8821 110/300 Write block 1032 len 36864 data: 2 2 2 2 2 3 3 3 4
8822 111/300 Write block 4510 len 8192 data: 2 2
8823 112/300 Read block 693 len 32768
8824 113/300 Flush
8825 114/300 Write block 2668 len 20480 data: 3 3 4 3 3
8826 115/300 Read block 766 len 28672
8827 116/300 Write block 2128 len 4096 data: 3
8828 117/300 Write block 4430 len 16384 data: 3 3 2 2
8829 118/300 Flush
8830 119/300 Write block 827 len 4096 data: 3
8831 120/300 Read block 152 len 24576
8832 121/300 Read block 1256 len 32768
8833 122/300 Read block 3173 len 28672
8834 123/300 Read block 2696 len 12288
8835 124/300 Write block 1164 len 28672 data: 3 3 3 3 3 3 2
8836 125/300 Read block 1815 len 36864
8837 126/300 Read block 2943 len 12288
8838 127/300 Write block 870 len 8192 data: 4 3
8839 128/300 Read block 4787 len 12288
8840 129/300 Write block 2263 len 32768 data: 2 2 2 2 2 2 2 2
8841 130/300 Write block 2509 len 20480 data: 2 3 4 5 5
8842 131/300 Write block 697 len 36864 data: 2 2 2 3 3 3 3 3 2
8843 132/300 Write block 235 len 40960 data: 3 4 6 6 4 3 3 3 2 2
8844 133/300 Read block 1413 len 36864
8845 134/300 Read block 2296 len 20480
8846 135/300 Write block 1624 len 32768 data: 4 4 4 4 4 4 4 3
8847 136/300 Write block 453 len 32768 data: 6 6 5 5 3 3 3 3
8848 137/300 Read block 1225 len 16384
8849 138/300 Read block 1859 len 20480
8850 139/300 Read block 3809 len 16384
8851 140/300 Read block 186 len 16384
8852 141/300 Read block 2814 len 24576
8853 142/300 Read block 1967 len 36864
8854 143/300 Write block 2683 len 28672 data: 3 3 3 2 3 3 4
8855 144/300 Read block 636 len 16384
8856 145/300 Read block 527 len 32768
8857 146/300 Write block 2524 len 28672 data: 5 5 3 3 3 3 2
8858 147/300 Read block 2194 len 16384
8859 148/300 Write block 3776 len 36864 data: 2 2 2 2 3 4 4 4 3
8860 149/300 Read block 4108 len 28672
8861 150/300 Write block 2291 len 12288 data: 2 2 2
8862 151/300 Read block 2959 len 12288
8863 152/300 Read block 1828 len 20480
8864 153/300 Read block 2091 len 20480
8865 154/300 Write block 4798 len 20480 data: 3 2 2 2 2
8866 155/300 Read block 4892 len 32768
8867 156/300 Write block 2965 len 12288 data: 4 4 4
8868 157/300 Read block 3737 len 16384
8869 158/300 Read block 4295 len 20480
8870 159/300 Write block 4989 len 8192 data: 2 2
8871 160/300 Read block 2424 len 16384
8872 161/300 Write block 2783 len 8192 data: 6 5
8873 162/300 Read block 2131 len 24576
8874 163/300 Write block 3734 len 20480 data: 3 3 3 3 3
8875 164/300 Write block 486 len 4096 data: 2
8876 165/300 Read block 1983 len 12288
8877 166/300 Read block 4259 len 12288
8878 167/300 Read block 4857 len 32768
8879 168/300 Write block 316 len 40960 data: 2 2 2 3 3 3 3 3 2 2
8880 169/300 Read block 2815 len 8192
8881 170/300 Read block 3397 len 12288
8882 171/300 Read block 4127 len 20480
8883 172/300 Write block 4569 len 32768 data: 2 2 2 2 2 2 2 2
8884 173/300 Read block 597 len 28672
8885 174/300 Write block 4384 len 32768 data: 3 3 4 3 3 3 4 4
8886 175/300 Write block 1363 len 24576 data: 3 3 3 2 2 2
8887 176/300 Write block 495 len 28672 data: 3 3 3 3 3 3 3
8888 177/300 Write block 3585 len 36864 data: 2 2 2 2 3 3 3 3 3
8889 178/300 Read block 1762 len 12288
8890 179/300 Write block 1488 len 36864 data: 3 4 4 5 5 3 2 2 2
8891 180/300 Write block 478 len 20480 data: 5 2 2 2 2
8892 181/300 Read block 862 len 32768
8893 182/300 Read block 2163 len 40960
8894 183/300 Read block 3799 len 36864
8895 184/300 Read block 3090 len 36864
8896 185/300 Read block 2741 len 16384
8897 186/300 Read block 2482 len 20480
8898 187/300 Read block 2374 len 8192
8899 188/300 Flush
8900 189/300 Read block 2842 len 8192
8901 190/300 Write block 4879 len 12288 data: 3 3 4
8902 191/300 Flush
8903 192/300 Write block 937 len 32768 data: 2 3 3 3 3 3 3 3
8904 193/300 Write block 4696 len 16384 data: 3 3 2 3
8905 194/300 Read block 4441 len 32768
8906 195/300 Read block 3387 len 28672
8907 196/300 Read block 3178 len 20480
8908 197/300 Write block 1665 len 32768 data: 3 3 3 4 4 3 3 3
8909 198/300 Flush
8910 199/300 Read block 4135 len 24576
8911 200/300 Write block 2093 len 36864 data: 3 3 3 3 3 3 2 2 2
8912 201/300 Write block 611 len 28672 data: 2 2 2 2 2 2 2
8913 202/300 Write block 2473 len 16384 data: 4 4 3 3
8914 203/300 Flush
8915 204/300 Write block 3718 len 24576 data: 4 4 4 4 4 4
8916 205/300 Read block 3611 len 24576
8917 206/300 Read block 2673 len 4096
8918 207/300 Read block 4540 len 20480
8919 208/300 Write block 2152 len 36864 data: 3 2 2 2 2 2 2 2 2
8920 209/300 Read block 842 len 16384
8921 210/300 Write block 1715 len 4096 data: 3
8922 211/300 Read block 2358 len 16384
8923 212/300 Read block 2883 len 40960
8924 213/300 Read block 454 len 36864
8925 214/300 Write block 2684 len 16384 data: 4 4 3 4
8926 215/300 Flush
8927 216/300 Write block 4566 len 8192 data: 2 2
8928 217/300 Read block 931 len 40960
8929 218/300 Write block 2107 len 8192 data: 3 4
8930 219/300 Read block 262 len 4096
8931 220/300 Write block 927 len 28672 data: 3 3 3 4 4 4 3
8932 221/300 Read block 1439 len 4096
8933 222/300 Read block 1473 len 28672
8934 223/300 Read block 1878 len 8192
8935 224/300 Read block 4374 len 36864
8936 225/300 Flush
8937 226/300 Read block 4480 len 4096
8938 227/300 Read block 2062 len 40960
8939 228/300 Write block 2759 len 20480 data: 3 3 3 3 3
8940 229/300 Write block 739 len 8192 data: 2 2
8941 230/300 Write block 1733 len 4096 data: 3
8942 231/300 Write block 434 len 8192 data: 4 4
8943 232/300 Write block 4158 len 16384 data: 2 3 3 3
8944 233/300 Write block 4126 len 32768 data: 3 3 3 3 3 3 2 2
8945 234/300 Read block 4187 len 4096
8946 235/300 Write block 3583 len 4096 data: 2
8947 236/300 Write block 4953 len 40960 data: 3 3 2 2 2 2 2 2 3 3
8948 237/300 Read block 3667 len 28672
8949 238/300 Read block 3786 len 28672
8950 239/300 Read block 3902 len 28672
8951 240/300 Write block 1318 len 4096 data: 2
8952 241/300 Read block 3465 len 28672
8953 242/300 Write block 1177 len 24576 data: 5 6 7 6 5 3
8954 243/300 Write block 1525 len 20480 data: 3 3 4 4 4
8955 244/300 Write block 2174 len 8192 data: 2 2
8956 245/300 Read block 370 len 20480
8957 246/300 Read block 691 len 40960
8958 247/300 Read block 4134 len 24576
8959 248/300 Write block 1271 len 24576 data: 3 3 3 3 2 2
8960 249/300 Read block 581 len 32768
8961 250/300 Write block 1703 len 24576 data: 4 3 2 3 3 3
8962 251/300 Flush
8963 252/300 Read block 813 len 20480
8964 253/300 Write block 1305 len 4096 data: 2
8965 254/300 Read block 3316 len 28672
8966 255/300 Flush
8967 256/300 Write block 3322 len 4096 data: 2
8968 257/300 Read block 2238 len 8192
8969 258/300 Read block 4248 len 12288
8970 259/300 Write block 2946 len 40960 data: 2 3 3 3 3 3 3 3 3 3
8971 260/300 Read block 2461 len 8192
8972 261/300 Read block 805 len 40960
8973 262/300 Read block 2395 len 8192
8974 263/300 Read block 4170 len 28672
8975 264/300 Read block 157 len 20480
8976 265/300 Read block 3362 len 16384
8977 266/300 Write block 3126 len 36864 data: 2 2 2 2 2 2 2 2 2
8978 267/300 Read block 116 len 40960
8979 268/300 Read block 4513 len 20480
8980 269/300 Write block 2813 len 28672 data: 3 3 3 3 3 3 3
8981 270/300 Flush
8982 271/300 Flush
8983 272/300 Write block 2488 len 16384 data: 3 3 2 2
8984 273/300 Read block 2690 len 12288
8985 274/300 Write block 1124 len 32768 data: 2 3 3 3 2 2 2 2
8986 275/300 Write block 769 len 8192 data: 2 2
8987 276/300 Flush
8988 277/300 Read block 1980 len 12288
8989 278/300 Read block 3020 len 28672
8990 279/300 Write block 1466 len 36864 data: 3 3 3 3 2 2 2 3 3
8991 280/300 Write block 2416 len 16384 data: 2 2 2 2
8992 281/300 Write block 4830 len 4096 data: 5
8993 282/300 Read block 1546 len 28672
8994 283/300 Flush
8995 284/300 Write block 442 len 40960 data: 5 6 6 6 5 6 6 7 7 7
8996 285/300 Flush
8997 286/300 Read block 2538 len 40960
8998 287/300 Read block 2139 len 40960
8999 288/300 Read block 4518 len 28672
9000 289/300 Read block 4653 len 16384
9001 290/300 Flush
9002 291/300 Write block 1885 len 16384 data: 2 2 2 2
9003 292/300 Write block 1207 len 20480 data: 2 2 2 2 2
9004 293/300 Read block 2022 len 36864
9005 294/300 Flush
9006 295/300 Write block 4978 len 32768 data: 3 3 3 3 4 4 5 5
9007 296/300 Read block 3687 len 20480
9008 297/300 Write block 2483 len 36864 data: 2 3 3 3 3 4 4 3 3
9009 298/300 Flush
9010 299/300 Write block 3961 len 4096 data: 3
9011 300/300 Write block 3465 len 12288 data: 2 2 2
9012 301/300 Write block 4479 len 36864 data: 3 2 2 2 2 2 2 2 2
9013 Main task has work to do, go find it
9014 got dsc Start(0)
9015 start 0
9016 Feb 28 06:03:01.626 INFO request completed, latency_us: 235, response_code: 204, uri: /start/cid/0, method: POST, req_id: bcaa173f-edde-488f-9501-adc3d0502b87, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
9017 Replay: started 0, returned:Ok(())
9018 ----------------------------------------------------------------
9019 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9020 States: Offline Active Active
9021 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9022 2809 Acked Write 7 Sent Done Done false
9023 2810 Acked Write 6 Sent Done Done false
9024 2811 Acked Write 1 Sent Done Done false
9025 2812 Acked Read 5 Sent Done Done false
9026 2813 Acked Write 2 Sent Done Done false
9027 2814 Acked Read 1 Sent Done Done false
9028 2815 Acked Read 6 Sent Done Done false
9029 2816 Acked Read 2 Sent Done Done false
9030 2817 Acked Read 8 Sent Done Done false
9031 2818 Acked Read 6 Sent Done Done false
9032 2819 Acked Read 4 Sent Done Done false
9033 2820 Acked Write 2 Sent Done Done false
9034 2821 Acked Read 3 Sent Done Done false
9035 2822 Acked Read 8 Sent Done Done false
9036 2823 Acked Read 6 Sent Done Done false
9037 2824 Acked Write 2 Sent Done Done false
9038 2825 Acked Flush 0 Sent Done Done false
9039 2826 Acked Read 5 Sent Done Done false
9040 2827 Acked Read 3 Sent Done Done false
9041 2828 Acked Read 3 Sent Done Done false
9042 2829 Acked Read 4 Sent Done Done false
9043 2830 Acked Read 6 Sent Done Done false
9044 2831 Acked Read 1 Sent Done Done false
9045 2832 Acked Read 9 Sent Done Done false
9046 2833 Acked Write 9 Sent Done Done false
9047 2834 Acked Read 2 Sent Done Done false
9048 2835 Acked Read 2 Sent Done Done false
9049 2836 Acked Write 8 Sent Done Done false
9050 2837 Acked Read 3 Sent Done Done false
9051 2838 Acked Read 7 Sent Done Done false
9052 2839 Acked Write 4 Sent Done Done false
9053 2840 Acked Write 7 Sent Done Done false
9054 2841 Acked Read 10 Sent Done Done false
9055 2842 Acked Read 4 Sent Done Done false
9056 2843 Acked Write 4 Sent Done Done false
9057 2844 Acked Write 5 Sent Done Done false
9058 2845 Acked Read 9 Sent Done Done false
9059 2846 Acked Read 9 Sent Done Done false
9060 2847 Acked Write 4 Sent Done Done false
9061 2848 Acked Read 9 Sent Done Done false
9062 2849 Acked Read 7 Sent Done Done false
9063 2850 Acked Read 5 Sent Done Done false
9064 2851 Acked Write 8 Sent Done Done false
9065 2852 Acked Flush 0 Sent Done Done false
9066 2853 Acked Write 6 Sent Done Done false
9067 2854 Acked Read 3 Sent Done Done false
9068 2855 Acked Write 1 Sent Done Done false
9069 2856 Acked Read 5 Sent Done Done false
9070 2857 Acked Write 5 Sent Done Done false
9071 2858 Acked Read 10 Sent Done Done false
9072 2859 Acked Write 3 Sent Done Done false
9073 2860 Acked Flush 0 Sent Done Done false
9074 2861 Acked Read 5 Sent Done Done false
9075 2862 Acked Read 6 Sent Done Done false
9076 2863 Acked Read 8 Sent Done Done false
9077 2864 Acked Write 2 Sent Done Done false
9078 2865 Acked Read 2 Sent Done Done false
9079 2866 Acked Flush 0 Sent Done Done false
9080 2867 Acked Flush 0 Sent Done Done false
9081 2868 Acked Write 6 Sent Done Done false
9082 2869 Acked Read 6 Sent Done Done false
9083 2870 Acked Read 5 Sent Done Done false
9084 2871 Acked Read 9 Sent Done Done false
9085 2872 Acked Write 4 Sent Done Done false
9086 2873 Acked Read 6 Sent Done Done false
9087 2874 Acked Write 4 Sent Done Done false
9088 2875 Acked Flush 0 Sent Done Done false
9089 2876 Acked Read 9 Sent Done Done false
9090 2877 Acked Write 2 Sent Done Done false
9091 2878 Acked Read 3 Sent Done Done false
9092 2879 Acked Read 9 Sent Done Done false
9093 2880 Acked Read 7 Sent Done Done false
9094 2881 Acked Read 9 Sent Done Done false
9095 2882 Acked Read 10 Sent Done Done false
9096 2883 Acked Read 1 Sent Done Done false
9097 2884 Acked Write 10 Sent Done Done false
9098 2885 Acked Read 7 Sent Done Done false
9099 2886 Acked Read 3 Sent Done Done false
9100 2887 Acked Write 9 Sent Done Done false
9101 2888 Acked Write 9 Sent Done Done false
9102 2889 Acked Write 7 Sent Done Done false
9103 2890 Acked Write 2 Sent Done Done false
9104 2891 Acked Read 2 Sent Done Done false
9105 2892 Acked Flush 0 Sent Done Done false
9106 2893 Acked Read 1 Sent Done Done false
9107 2894 Acked Write 3 Sent Done Done false
9108 2895 Acked Write 6 Sent Done Done false
9109 2896 Acked Write 2 Sent Done Done false
9110 2897 Acked Write 5 Sent Done Done false
9111 2898 Acked Read 8 Sent Done Done false
9112 2899 Acked Read 9 Sent Done Done false
9113 2900 Acked Read 1 Sent Done Done false
9114 2901 Acked Read 9 Sent Done Done false
9115 2902 Acked Write 2 Sent Done Done false
9116 2903 Acked Write 7 Sent Done Done false
9117 2904 Acked Write 3 Sent Done Done false
9118 2905 Acked Write 2 Sent Done Done false
9119 2906 Acked Read 3 Sent Done Done false
9120 2907 Acked Write 7 Sent Done Done false
9121 2908 Acked Write 2 Sent Done Done false
9122 2909 Acked Read 6 Sent Done Done false
9123 2910 Acked Write 1 Sent Done Done false
9124 2911 Acked Read 8 Sent Done Done false
9125 2912 Acked Read 10 Sent Done Done false
9126 2913 Acked Write 2 Sent Done Done false
9127 2914 Acked Write 9 Sent Done Done false
9128 2915 Acked Read 6 Sent Done Done false
9129 2916 Acked Read 10 Sent Done Done false
9130 2917 Acked Write 10 Sent Done Done false
9131 2918 Acked Write 9 Sent Done Done false
9132 2919 Acked Write 2 Sent Done Done false
9133 2920 Acked Read 8 Sent Done Done false
9134 2921 Acked Flush 0 Sent Done Done false
9135 2922 Acked Write 5 Sent Done Done false
9136 2923 Acked Read 7 Sent Done Done false
9137 2924 Acked Write 1 Sent Done Done false
9138 2925 Acked Write 4 Sent Done Done false
9139 2926 Acked Flush 0 Sent Done Done false
9140 2927 Acked Write 1 Sent Done Done false
9141 2928 Acked Read 6 Sent Done Done false
9142 2929 Acked Read 8 Sent Done Done false
9143 2930 Acked Read 7 Sent Done Done false
9144 2931 Acked Read 3 Sent Done Done false
9145 2932 Acked Write 7 Sent Done Done false
9146 2933 Acked Read 9 Sent Done Done false
9147 2934 Acked Read 3 Sent Done Done false
9148 2935 Acked Write 2 Sent Done Done false
9149 2936 Acked Read 3 Sent Done Done false
9150 2937 Acked Write 8 Sent Done Done false
9151 2938 Acked Write 5 Sent Done Done false
9152 2939 Acked Write 9 Sent Done Done false
9153 2940 Acked Write 10 Sent Done Done false
9154 2941 Acked Read 9 Sent Done Done false
9155 2942 Acked Read 5 Sent Done Done false
9156 2943 Acked Write 8 Sent Done Done false
9157 2944 Acked Write 8 Sent Done Done false
9158 2945 Acked Read 4 Sent Done Done false
9159 2946 Acked Read 5 Sent Done Done false
9160 2947 Acked Read 4 Sent Done Done false
9161 2948 Acked Read 4 Sent Done Done false
9162 2949 Acked Read 6 Sent Done Done false
9163 2950 Acked Read 9 Sent Done Done false
9164 2951 Acked Write 7 Sent Done Done false
9165 2952 Acked Read 4 Sent Done Done false
9166 2953 Acked Read 8 Sent Done Done false
9167 2954 Acked Write 7 Sent Done Done false
9168 2955 Acked Read 4 Sent Done Done false
9169 2956 Acked Write 9 Sent Done Done false
9170 2957 Acked Read 7 Sent Done Done false
9171 2958 Acked Write 3 Sent Done Done false
9172 2959 Acked Read 3 Sent Done Done false
9173 2960 Acked Read 5 Sent Done Done false
9174 2961 Acked Read 5 Sent Done Done false
9175 2962 Acked Write 5 Sent Done Done false
9176 2963 Acked Read 8 Sent Done Done false
9177 2964 Acked Write 3 Sent Done Done false
9178 2965 Acked Read 4 Sent Done Done false
9179 2966 Acked Read 5 Sent Done Done false
9180 2967 Acked Write 2 Sent Done Done false
9181 2968 Acked Read 4 Sent Done Done false
9182 2969 Acked Write 2 Sent Done Done false
9183 2970 Acked Read 6 Sent Done Done false
9184 2971 Acked Write 5 Sent Done Done false
9185 2972 Acked Write 1 Sent Done Done false
9186 2973 Acked Read 3 Sent Done Done false
9187 2974 Acked Read 3 Sent Done Done false
9188 2975 Acked Read 8 Sent Done Done false
9189 2976 Acked Write 10 Sent Done Done false
9190 2977 Acked Read 2 Sent Done Done false
9191 2978 Acked Read 3 Sent Done Done false
9192 2979 Acked Read 5 Sent Done Done false
9193 2980 Acked Write 8 Sent Done Done false
9194 2981 Acked Read 7 Sent Done Done false
9195 2982 Acked Write 8 Sent Done Done false
9196 2983 Acked Write 6 Sent Done Done false
9197 2984 Acked Write 7 Sent Done Done false
9198 2985 Acked Write 9 Sent Done Done false
9199 2986 Acked Read 3 Sent Done Done false
9200 2987 Acked Write 9 Sent Done Done false
9201 2988 Acked Write 5 Sent Done Done false
9202 2989 Acked Read 8 Sent Done Done false
9203 2990 Acked Read 10 Sent Done Done false
9204 2991 Acked Read 9 Sent Done Done false
9205 2992 Acked Read 9 Sent Done Done false
9206 2993 Acked Read 4 Sent Done Done false
9207 2994 Acked Read 5 Sent Done Done false
9208 2995 Acked Read 2 Sent Done Done false
9209 2996 Acked Flush 0 Sent Done Done false
9210 2997 Acked Read 2 Sent Done Done false
9211 2998 Acked Write 3 Sent Done Done false
9212 2999 Acked Flush 0 Sent Done Done false
9213 3000 Acked Write 8 Sent Done Done false
9214 3001 Acked Write 4 Sent Done Done false
9215 3002 Acked Read 8 Sent Done Done false
9216 3003 Acked Read 7 Sent Done Done false
9217 3004 Acked Read 5 Sent Done Done false
9218 3005 Acked Write 8 Sent Done Done false
9219 3006 Acked Flush 0 Sent Done Done false
9220 3007 Acked Read 6 Sent Done Done false
9221 3008 Acked Write 9 Sent Done Done false
9222 3009 Acked Write 7 Sent Done Done false
9223 3010 Acked Write 4 Sent Done Done false
9224 3011 Acked Flush 0 Sent Done Done false
9225 3012 Acked Write 6 Sent Done Done false
9226 3013 Acked Read 6 Sent Done Done false
9227 3014 Acked Read 1 Sent Done Done false
9228 3015 Acked Read 5 Sent Done Done false
9229 3016 Acked Write 9 Sent Done Done false
9230 3017 Acked Read 4 Sent Done Done false
9231 3018 Acked Write 1 Sent Done Done false
9232 3019 Acked Read 4 Sent Done Done false
9233 3020 Acked Read 10 Sent Done Done false
9234 3021 Acked Read 9 Sent Done Done false
9235 3022 Acked Write 4 Sent Done Done false
9236 3023 Acked Flush 0 Sent Done Done false
9237 3024 Acked Write 2 Sent Done Done false
9238 3025 Acked Read 10 Sent Done Done false
9239 3026 Acked Write 2 Sent Done Done false
9240 3027 Acked Read 1 Sent Done Done false
9241 3028 Acked Write 7 Sent Done Done false
9242 3029 Acked Read 1 Sent Done Done false
9243 3030 Acked Read 7 Sent Done Done false
9244 3031 Acked Read 2 Sent Done Done false
9245 3032 Acked Read 9 Sent Done Done false
9246 3033 Acked Flush 0 Sent Done Done false
9247 3034 Acked Read 1 Sent Done Done false
9248 3035 Acked Read 10 Sent Done Done false
9249 3036 Acked Write 5 Sent Done Done false
9250 3037 Acked Write 2 Sent Done Done false
9251 3038 Acked Write 1 Sent Done Done false
9252 3039 Acked Write 2 Sent Done Done false
9253 3040 Acked Write 4 Sent Done Done false
9254 3041 Acked Write 8 Sent Done Done false
9255 3042 Acked Read 1 Sent Done Done false
9256 3043 Acked Write 1 Sent Done Done false
9257 3044 Acked Write 10 Sent Done Done false
9258 3045 Acked Read 7 Sent Done Done false
9259 3046 Acked Read 7 Sent Done Done false
9260 3047 Acked Read 7 Sent Done Done false
9261 3048 Acked Write 1 Sent Done Done false
9262 3049 Acked Read 7 Sent Done Done false
9263 3050 Acked Write 6 Sent Done Done false
9264 3051 Acked Write 5 Sent Done Done false
9265 3052 Acked Write 2 Sent Done Done false
9266 3053 Acked Read 5 Sent Done Done false
9267 3054 Acked Read 10 Sent Done Done false
9268 3055 Acked Read 6 Sent Done Done false
9269 3056 Acked Write 6 Sent Done Done false
9270 3057 Acked Read 8 Sent Done Done false
9271 3058 Acked Write 6 Sent Done Done false
9272 3059 Acked Flush 0 Sent Done Done false
9273 3060 Acked Read 5 Sent Done Done false
9274 3061 Acked Write 1 Sent Done Done false
9275 3062 Acked Read 7 Sent Done Done false
9276 3063 Acked Flush 0 Sent Done Done false
9277 3064 Acked Write 1 Sent Done Done false
9278 3065 Acked Read 2 Sent Done Done false
9279 3066 Acked Read 3 Sent Done Done false
9280 3067 Acked Write 10 Sent Done Done false
9281 3068 Acked Read 2 Sent Done Done false
9282 3069 Acked Read 10 Sent Done Done false
9283 3070 Acked Read 2 Sent Done Done false
9284 3071 Acked Read 7 Sent Done Done false
9285 3072 Acked Read 5 Sent Done Done false
9286 3073 Acked Read 4 Sent Done Done false
9287 3074 Acked Write 9 Sent Done Done false
9288 3075 Acked Read 10 Sent Done Done false
9289 3076 Acked Read 5 Sent Done Done false
9290 3077 Acked Write 7 Sent Done Done false
9291 3078 Acked Flush 0 Sent Done Done false
9292 3079 Acked Flush 0 Sent Done Done false
9293 3080 Acked Write 4 Sent Done Done false
9294 3081 Acked Read 3 Sent Done Done false
9295 3082 Acked Write 8 Sent Done Done false
9296 3083 Acked Write 2 Sent Done Done false
9297 3084 Acked Flush 0 Sent Done Done false
9298 3085 Acked Read 3 Sent Done Done false
9299 3086 Acked Read 7 Sent Done Done false
9300 3087 Acked Write 9 Sent Done Done false
9301 3088 Acked Write 4 Sent Done Done false
9302 3089 Acked Write 1 Sent Done Done false
9303 3090 Acked Read 7 Sent Done Done false
9304 3091 Acked Flush 0 Sent Done Done false
9305 3092 Acked Write 10 Sent Done Done false
9306 3093 Acked Flush 0 Sent Done Done false
9307 3094 Acked Read 10 Sent Done Done false
9308 3095 Acked Read 10 Sent Done Done false
9309 3096 Acked Read 7 Sent Done Done false
9310 3097 Acked Read 4 Sent Done Done false
9311 3098 Acked Flush 0 Sent Done Done false
9312 3099 Acked Write 4 Sent Done Done false
9313 3100 Acked Write 5 Sent Done Done false
9314 3101 Acked Read 9 Sent Done Done false
9315 3102 Acked Flush 0 Sent Done Done false
9316 3103 Acked Write 8 Sent Done Done false
9317 3104 Acked Read 5 Sent Done Done false
9318 3105 Acked Write 9 Sent Done Done false
9319 3106 Acked Flush 0 Sent Done Done false
9320 3107 Acked Write 1 Sent Done Done false
9321 3108 Acked Write 3 Sent Done Done false
9322 3109 Acked Write 9 Sent Done Sent false
9323 STATES DS:0 DS:1 DS:2 TOTAL
9324 Sent 301 0 1 302
9325 Done 0 301 300 601
9326 Skipped 0 0 0 0
9327 Error 0 0 0 0
9328 Last Flush: Some(JobId(2808)) Some(JobId(3106)) Some(JobId(3106))
9329 Downstairs last five completed: 2808 2807 2806 2805 2804
9330 Upstairs last five completed: 3109 3108 3107 3106 3105
9331 CLIENT: Up:0 ds:301 act:2
93322025-02-28T06:03:02.532ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
9333 [8810] got start action
9334 [8810] I am going to restart
9335 Starting downstairs at port 8810
9336 Make output file at "/tmp/dsc/downstairs-8810.txt"
9337 [8810][0] reports Starting
9338 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1218)
9339 [8810][0] reports Running
93402025-02-28T06:03:03.534ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
93412025-02-28T06:03:04.535ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
93422025-02-28T06:03:05.538ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
9343 ----------------------------------------------------------------
9344 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9345 States: Offline Active Active
9346 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9347 2809 Acked Write 7 Sent Done Done false
9348 2810 Acked Write 6 Sent Done Done false
9349 2811 Acked Write 1 Sent Done Done false
9350 2812 Acked Read 5 Sent Done Done false
9351 2813 Acked Write 2 Sent Done Done false
9352 2814 Acked Read 1 Sent Done Done false
9353 2815 Acked Read 6 Sent Done Done false
9354 2816 Acked Read 2 Sent Done Done false
9355 2817 Acked Read 8 Sent Done Done false
9356 2818 Acked Read 6 Sent Done Done false
9357 2819 Acked Read 4 Sent Done Done false
9358 2820 Acked Write 2 Sent Done Done false
9359 2821 Acked Read 3 Sent Done Done false
9360 2822 Acked Read 8 Sent Done Done false
9361 2823 Acked Read 6 Sent Done Done false
9362 2824 Acked Write 2 Sent Done Done false
9363 2825 Acked Flush 0 Sent Done Done false
9364 2826 Acked Read 5 Sent Done Done false
9365 2827 Acked Read 3 Sent Done Done false
9366 2828 Acked Read 3 Sent Done Done false
9367 2829 Acked Read 4 Sent Done Done false
9368 2830 Acked Read 6 Sent Done Done false
9369 2831 Acked Read 1 Sent Done Done false
9370 2832 Acked Read 9 Sent Done Done false
9371 2833 Acked Write 9 Sent Done Done false
9372 2834 Acked Read 2 Sent Done Done false
9373 2835 Acked Read 2 Sent Done Done false
9374 2836 Acked Write 8 Sent Done Done false
9375 2837 Acked Read 3 Sent Done Done false
9376 2838 Acked Read 7 Sent Done Done false
9377 2839 Acked Write 4 Sent Done Done false
9378 2840 Acked Write 7 Sent Done Done false
9379 2841 Acked Read 10 Sent Done Done false
9380 2842 Acked Read 4 Sent Done Done false
9381 2843 Acked Write 4 Sent Done Done false
9382 2844 Acked Write 5 Sent Done Done false
9383 2845 Acked Read 9 Sent Done Done false
9384 2846 Acked Read 9 Sent Done Done false
9385 2847 Acked Write 4 Sent Done Done false
9386 2848 Acked Read 9 Sent Done Done false
9387 2849 Acked Read 7 Sent Done Done false
9388 2850 Acked Read 5 Sent Done Done false
9389 2851 Acked Write 8 Sent Done Done false
9390 2852 Acked Flush 0 Sent Done Done false
9391 2853 Acked Write 6 Sent Done Done false
9392 2854 Acked Read 3 Sent Done Done false
9393 2855 Acked Write 1 Sent Done Done false
9394 2856 Acked Read 5 Sent Done Done false
9395 2857 Acked Write 5 Sent Done Done false
9396 2858 Acked Read 10 Sent Done Done false
9397 2859 Acked Write 3 Sent Done Done false
9398 2860 Acked Flush 0 Sent Done Done false
9399 2861 Acked Read 5 Sent Done Done false
9400 2862 Acked Read 6 Sent Done Done false
9401 2863 Acked Read 8 Sent Done Done false
9402 2864 Acked Write 2 Sent Done Done false
9403 2865 Acked Read 2 Sent Done Done false
9404 2866 Acked Flush 0 Sent Done Done false
9405 2867 Acked Flush 0 Sent Done Done false
9406 2868 Acked Write 6 Sent Done Done false
9407 2869 Acked Read 6 Sent Done Done false
9408 2870 Acked Read 5 Sent Done Done false
9409 2871 Acked Read 9 Sent Done Done false
9410 2872 Acked Write 4 Sent Done Done false
9411 2873 Acked Read 6 Sent Done Done false
9412 2874 Acked Write 4 Sent Done Done false
9413 2875 Acked Flush 0 Sent Done Done false
9414 2876 Acked Read 9 Sent Done Done false
9415 2877 Acked Write 2 Sent Done Done false
9416 2878 Acked Read 3 Sent Done Done false
9417 2879 Acked Read 9 Sent Done Done false
9418 2880 Acked Read 7 Sent Done Done false
9419 2881 Acked Read 9 Sent Done Done false
9420 2882 Acked Read 10 Sent Done Done false
9421 2883 Acked Read 1 Sent Done Done false
9422 2884 Acked Write 10 Sent Done Done false
9423 2885 Acked Read 7 Sent Done Done false
9424 2886 Acked Read 3 Sent Done Done false
9425 2887 Acked Write 9 Sent Done Done false
9426 2888 Acked Write 9 Sent Done Done false
9427 2889 Acked Write 7 Sent Done Done false
9428 2890 Acked Write 2 Sent Done Done false
9429 2891 Acked Read 2 Sent Done Done false
9430 2892 Acked Flush 0 Sent Done Done false
9431 2893 Acked Read 1 Sent Done Done false
9432 2894 Acked Write 3 Sent Done Done false
9433 2895 Acked Write 6 Sent Done Done false
9434 2896 Acked Write 2 Sent Done Done false
9435 2897 Acked Write 5 Sent Done Done false
9436 2898 Acked Read 8 Sent Done Done false
9437 2899 Acked Read 9 Sent Done Done false
9438 2900 Acked Read 1 Sent Done Done false
9439 2901 Acked Read 9 Sent Done Done false
9440 2902 Acked Write 2 Sent Done Done false
9441 2903 Acked Write 7 Sent Done Done false
9442 2904 Acked Write 3 Sent Done Done false
9443 2905 Acked Write 2 Sent Done Done false
9444 2906 Acked Read 3 Sent Done Done false
9445 2907 Acked Write 7 Sent Done Done false
9446 2908 Acked Write 2 Sent Done Done false
9447 2909 Acked Read 6 Sent Done Done false
9448 2910 Acked Write 1 Sent Done Done false
9449 2911 Acked Read 8 Sent Done Done false
9450 2912 Acked Read 10 Sent Done Done false
9451 2913 Acked Write 2 Sent Done Done false
9452 2914 Acked Write 9 Sent Done Done false
9453 2915 Acked Read 6 Sent Done Done false
9454 2916 Acked Read 10 Sent Done Done false
9455 2917 Acked Write 10 Sent Done Done false
9456 2918 Acked Write 9 Sent Done Done false
9457 2919 Acked Write 2 Sent Done Done false
9458 2920 Acked Read 8 Sent Done Done false
9459 2921 Acked Flush 0 Sent Done Done false
9460 2922 Acked Write 5 Sent Done Done false
9461 2923 Acked Read 7 Sent Done Done false
9462 2924 Acked Write 1 Sent Done Done false
9463 2925 Acked Write 4 Sent Done Done false
9464 2926 Acked Flush 0 Sent Done Done false
9465 2927 Acked Write 1 Sent Done Done false
9466 2928 Acked Read 6 Sent Done Done false
9467 2929 Acked Read 8 Sent Done Done false
9468 2930 Acked Read 7 Sent Done Done false
9469 2931 Acked Read 3 Sent Done Done false
9470 2932 Acked Write 7 Sent Done Done false
9471 2933 Acked Read 9 Sent Done Done false
9472 2934 Acked Read 3 Sent Done Done false
9473 2935 Acked Write 2 Sent Done Done false
9474 2936 Acked Read 3 Sent Done Done false
9475 2937 Acked Write 8 Sent Done Done false
9476 2938 Acked Write 5 Sent Done Done false
9477 2939 Acked Write 9 Sent Done Done false
9478 2940 Acked Write 10 Sent Done Done false
9479 2941 Acked Read 9 Sent Done Done false
9480 2942 Acked Read 5 Sent Done Done false
9481 2943 Acked Write 8 Sent Done Done false
9482 2944 Acked Write 8 Sent Done Done false
9483 2945 Acked Read 4 Sent Done Done false
9484 2946 Acked Read 5 Sent Done Done false
9485 2947 Acked Read 4 Sent Done Done false
9486 2948 Acked Read 4 Sent Done Done false
9487 2949 Acked Read 6 Sent Done Done false
9488 2950 Acked Read 9 Sent Done Done false
9489 2951 Acked Write 7 Sent Done Done false
9490 2952 Acked Read 4 Sent Done Done false
9491 2953 Acked Read 8 Sent Done Done false
9492 2954 Acked Write 7 Sent Done Done false
9493 2955 Acked Read 4 Sent Done Done false
9494 2956 Acked Write 9 Sent Done Done false
9495 2957 Acked Read 7 Sent Done Done false
9496 2958 Acked Write 3 Sent Done Done false
9497 2959 Acked Read 3 Sent Done Done false
9498 2960 Acked Read 5 Sent Done Done false
9499 2961 Acked Read 5 Sent Done Done false
9500 2962 Acked Write 5 Sent Done Done false
9501 2963 Acked Read 8 Sent Done Done false
9502 2964 Acked Write 3 Sent Done Done false
9503 2965 Acked Read 4 Sent Done Done false
9504 2966 Acked Read 5 Sent Done Done false
9505 2967 Acked Write 2 Sent Done Done false
9506 2968 Acked Read 4 Sent Done Done false
9507 2969 Acked Write 2 Sent Done Done false
9508 2970 Acked Read 6 Sent Done Done false
9509 2971 Acked Write 5 Sent Done Done false
9510 2972 Acked Write 1 Sent Done Done false
9511 2973 Acked Read 3 Sent Done Done false
9512 2974 Acked Read 3 Sent Done Done false
9513 2975 Acked Read 8 Sent Done Done false
9514 2976 Acked Write 10 Sent Done Done false
9515 2977 Acked Read 2 Sent Done Done false
9516 2978 Acked Read 3 Sent Done Done false
9517 2979 Acked Read 5 Sent Done Done false
9518 2980 Acked Write 8 Sent Done Done false
9519 2981 Acked Read 7 Sent Done Done false
9520 2982 Acked Write 8 Sent Done Done false
9521 2983 Acked Write 6 Sent Done Done false
9522 2984 Acked Write 7 Sent Done Done false
9523 2985 Acked Write 9 Sent Done Done false
9524 2986 Acked Read 3 Sent Done Done false
9525 2987 Acked Write 9 Sent Done Done false
9526 2988 Acked Write 5 Sent Done Done false
9527 2989 Acked Read 8 Sent Done Done false
9528 2990 Acked Read 10 Sent Done Done false
9529 2991 Acked Read 9 Sent Done Done false
9530 2992 Acked Read 9 Sent Done Done false
9531 2993 Acked Read 4 Sent Done Done false
9532 2994 Acked Read 5 Sent Done Done false
9533 2995 Acked Read 2 Sent Done Done false
9534 2996 Acked Flush 0 Sent Done Done false
9535 2997 Acked Read 2 Sent Done Done false
9536 2998 Acked Write 3 Sent Done Done false
9537 2999 Acked Flush 0 Sent Done Done false
9538 3000 Acked Write 8 Sent Done Done false
9539 3001 Acked Write 4 Sent Done Done false
9540 3002 Acked Read 8 Sent Done Done false
9541 3003 Acked Read 7 Sent Done Done false
9542 3004 Acked Read 5 Sent Done Done false
9543 3005 Acked Write 8 Sent Done Done false
9544 3006 Acked Flush 0 Sent Done Done false
9545 3007 Acked Read 6 Sent Done Done false
9546 3008 Acked Write 9 Sent Done Done false
9547 3009 Acked Write 7 Sent Done Done false
9548 3010 Acked Write 4 Sent Done Done false
9549 3011 Acked Flush 0 Sent Done Done false
9550 3012 Acked Write 6 Sent Done Done false
9551 3013 Acked Read 6 Sent Done Done false
9552 3014 Acked Read 1 Sent Done Done false
9553 3015 Acked Read 5 Sent Done Done false
9554 3016 Acked Write 9 Sent Done Done false
9555 3017 Acked Read 4 Sent Done Done false
9556 3018 Acked Write 1 Sent Done Done false
9557 3019 Acked Read 4 Sent Done Done false
9558 3020 Acked Read 10 Sent Done Done false
9559 3021 Acked Read 9 Sent Done Done false
9560 3022 Acked Write 4 Sent Done Done false
9561 3023 Acked Flush 0 Sent Done Done false
9562 3024 Acked Write 2 Sent Done Done false
9563 3025 Acked Read 10 Sent Done Done false
9564 3026 Acked Write 2 Sent Done Done false
9565 3027 Acked Read 1 Sent Done Done false
9566 3028 Acked Write 7 Sent Done Done false
9567 3029 Acked Read 1 Sent Done Done false
9568 3030 Acked Read 7 Sent Done Done false
9569 3031 Acked Read 2 Sent Done Done false
9570 3032 Acked Read 9 Sent Done Done false
9571 3033 Acked Flush 0 Sent Done Done false
9572 3034 Acked Read 1 Sent Done Done false
9573 3035 Acked Read 10 Sent Done Done false
9574 3036 Acked Write 5 Sent Done Done false
9575 3037 Acked Write 2 Sent Done Done false
9576 3038 Acked Write 1 Sent Done Done false
9577 3039 Acked Write 2 Sent Done Done false
9578 3040 Acked Write 4 Sent Done Done false
9579 3041 Acked Write 8 Sent Done Done false
9580 3042 Acked Read 1 Sent Done Done false
9581 3043 Acked Write 1 Sent Done Done false
9582 3044 Acked Write 10 Sent Done Done false
9583 3045 Acked Read 7 Sent Done Done false
9584 3046 Acked Read 7 Sent Done Done false
9585 3047 Acked Read 7 Sent Done Done false
9586 3048 Acked Write 1 Sent Done Done false
9587 3049 Acked Read 7 Sent Done Done false
9588 3050 Acked Write 6 Sent Done Done false
9589 3051 Acked Write 5 Sent Done Done false
9590 3052 Acked Write 2 Sent Done Done false
9591 3053 Acked Read 5 Sent Done Done false
9592 3054 Acked Read 10 Sent Done Done false
9593 3055 Acked Read 6 Sent Done Done false
9594 3056 Acked Write 6 Sent Done Done false
9595 3057 Acked Read 8 Sent Done Done false
9596 3058 Acked Write 6 Sent Done Done false
9597 3059 Acked Flush 0 Sent Done Done false
9598 3060 Acked Read 5 Sent Done Done false
9599 3061 Acked Write 1 Sent Done Done false
9600 3062 Acked Read 7 Sent Done Done false
9601 3063 Acked Flush 0 Sent Done Done false
9602 3064 Acked Write 1 Sent Done Done false
9603 3065 Acked Read 2 Sent Done Done false
9604 3066 Acked Read 3 Sent Done Done false
9605 3067 Acked Write 10 Sent Done Done false
9606 3068 Acked Read 2 Sent Done Done false
9607 3069 Acked Read 10 Sent Done Done false
9608 3070 Acked Read 2 Sent Done Done false
9609 3071 Acked Read 7 Sent Done Done false
9610 3072 Acked Read 5 Sent Done Done false
9611 3073 Acked Read 4 Sent Done Done false
9612 3074 Acked Write 9 Sent Done Done false
9613 3075 Acked Read 10 Sent Done Done false
9614 3076 Acked Read 5 Sent Done Done false
9615 3077 Acked Write 7 Sent Done Done false
9616 3078 Acked Flush 0 Sent Done Done false
9617 3079 Acked Flush 0 Sent Done Done false
9618 3080 Acked Write 4 Sent Done Done false
9619 3081 Acked Read 3 Sent Done Done false
9620 3082 Acked Write 8 Sent Done Done false
9621 3083 Acked Write 2 Sent Done Done false
9622 3084 Acked Flush 0 Sent Done Done false
9623 3085 Acked Read 3 Sent Done Done false
9624 3086 Acked Read 7 Sent Done Done false
9625 3087 Acked Write 9 Sent Done Done false
9626 3088 Acked Write 4 Sent Done Done false
9627 3089 Acked Write 1 Sent Done Done false
9628 3090 Acked Read 7 Sent Done Done false
9629 3091 Acked Flush 0 Sent Done Done false
9630 3092 Acked Write 10 Sent Done Done false
9631 3093 Acked Flush 0 Sent Done Done false
9632 3094 Acked Read 10 Sent Done Done false
9633 3095 Acked Read 10 Sent Done Done false
9634 3096 Acked Read 7 Sent Done Done false
9635 3097 Acked Read 4 Sent Done Done false
9636 3098 Acked Flush 0 Sent Done Done false
9637 3099 Acked Write 4 Sent Done Done false
9638 3100 Acked Write 5 Sent Done Done false
9639 3101 Acked Read 9 Sent Done Done false
9640 3102 Acked Flush 0 Sent Done Done false
9641 3103 Acked Write 8 Sent Done Done false
9642 3104 Acked Read 5 Sent Done Done false
9643 3105 Acked Write 9 Sent Done Done false
9644 3106 Acked Flush 0 Sent Done Done false
9645 3107 Acked Write 1 Sent Done Done false
9646 3108 Acked Write 3 Sent Done Done false
9647 3109 Acked Write 9 Sent Done Done false
9648 STATES DS:0 DS:1 DS:2 TOTAL
9649 Sent 301 0 0 301
9650 Done 0 301 301 602
9651 Skipped 0 0 0 0
9652 Error 0 0 0 0
9653 Last Flush: Some(JobId(2808)) Some(JobId(3106)) Some(JobId(3106))
9654 Downstairs last five completed: 2808 2807 2806 2805 2804
9655 Upstairs last five completed: 3109 3108 3107 3106 3105
9656 CLIENT: Up:0 ds:301 act:2
96572025-02-28T06:03:06.540ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
96582025-02-28T06:03:07.380ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
96592025-02-28T06:03:07.380ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61029) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
96602025-02-28T06:03:07.381ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
96612025-02-28T06:03:07.382ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
96622025-02-28T06:03:07.382ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
96632025-02-28T06:03:07.382ZINFOcrucible: send last flush ID to this DS: Some(JobId(2808)) = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
96642025-02-28T06:03:07.382ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
96652025-02-28T06:03:07.382ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2808)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
96662025-02-28T06:03:07.382ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2808)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
9667 ----------------------------------------------------------------
9668 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9669 Downstairs last five completed: 3110 3109 3108 3107 3106
9670 Upstairs last five completed: 3110 3109 3108 3107 3106
9671 CLIENT: Up:0 ds:0 act:3
9672 Replay: All jobs finished
9673 Main task has work to do, go find it
9674 got dsc Stop(1)
9675 stop 1
9676 Feb 28 06:03:09.632 INFO request completed, latency_us: 364, response_code: 204, uri:[8820] Got stop action so:false kr:false
9677 /stop/cid/1, method: POST, req_id: bd6f3783-e092-4cbe-b044-c18a3eb082b7, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
9678 Feb 28 06:03:09.632 INFO request completed, latency_us: 110, response_code: 200, uri: /state/cid/1, method: GET, req_id: f8d365fd-74e8-4553-bc4a-de55e447c6b9, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
96792025-02-28T06:03:09.632ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
96802025-02-28T06:03:09.632ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
96812025-02-28T06:03:09.632ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
96822025-02-28T06:03:09.633ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
9683 [8820] Exited with: ExitStatus(unix_wait_status(9))
9684 [8820][1] reports Exit
9685 Feb 28 06:03:13.635 INFO request completed, latency_us: 286, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2156f9fb-29d2-4206-bd74-c7a9e6df4e4f, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
9686 001/300 Flush
9687 002/300 Read block 4063 len 28672
9688 003/300 Write block 3453 len 28672 data: 3 3 3 3 3 3 3
9689 004/300 Read block 1330 len 20480
9690 005/300 Read block 3262 len 16384
9691 006/300 Write block 466 len 12288 data: 2 2 3
9692 007/300 Write block 4875 len 20480 data: 2 2 2 3 4
9693 008/300 Read block 599 len 24576
9694 009/300 Write block 744 len 32768 data: 2 2 2 2 2 3 3 3
9695 010/300 Read block 4107 len 12288
9696 011/300 Write block 2075 len 20480 data: 2 2 2 2 2
9697 012/300 Read block 3622 len 20480
9698 013/300 Read block 149 len 24576
9699 014/300 Write block 2028 len 8192 data: 3 3
9700 015/300 Read block 1931 len 36864
9701 016/300 Read block 2427 len 24576
9702 017/300 Read block 1134 len 40960
9703 018/300 Read block 238 len 16384
9704 019/300 Read block 651 len 8192
9705 020/300 Read block 1725 len 40960
9706 021/300 Write block 2326 len 4096 data: 3
9707 022/300 Read block 1255 len 36864
9708 023/300 Write block 1043 len 32768 data: 4 4 4 3 2 2 3 3
9709 024/300 Read block 3450 len 20480
9710 025/300 Write block 2389 len 24576 data: 2 2 2 3 3 2
9711 026/300 Read block 4805 len 16384
9712 027/300 Read block 4283 len 4096
9713 028/300 Read block 3764 len 32768
9714 029/300 Read block 4548 len 36864
9715 030/300 Read block 3108 len 40960
9716 031/300 Write block 1683 len 16384 data: 3 3 3 3
9717 032/300 Write block 4262 len 20480 data: 3 3 2 2 2
9718 033/300 Write block 4621 len 16384 data: 3 3 4 4
9719 034/300 Flush
9720 035/300 Write block 4659 len 40960 data: 2 2 2 2 2 2 2 2 2 2
9721 036/300 Write block 3112 len 12288 data: 5 4 5
9722 037/300 Write block 3532 len 20480 data: 3 3 3 2 2
9723 038/300 Write block 1941 len 28672 data: 3 3 3 3 2 3 3
9724 039/300 Read block 3931 len 20480
9725 040/300 Write block 4567 len 8192 data: 3 2
9726 041/300 Write block 1400 len 4096 data: 4
9727 042/300 Write block 894 len 36864 data: 3 2 2 4 4 3 3 3 4
9728 043/300 Write block 4339 len 40960 data: 4 3 2 2 2 2 2 2 2 2
9729 044/300 Write block 4162 len 28672 data: 3 3 3 4 4 4 4
9730 045/300 Read block 1826 len 40960
9731 046/300 Write block 1371 len 16384 data: 2 2 3 3
9732 047/300 Write block 2480 len 32768 data: 2 2 2 3 4 4 4 4
9733 048/300 Read block 94 len 28672
9734 049/300 Read block 4905 len 16384
9735 050/300 Read block 1329 len 24576
9736 051/300 Write block 1721 len 28672 data: 2 2 3 3 3 3 4
9737 052/300 Write block 1310 len 16384 data: 2 2 2 2
9738 053/300 Read block 4090 len 12288
9739 054/300 Read block 2308 len 20480
9740 055/300 Write block 2674 len 28672 data: 3 3 3 2 3 3 3
9741 056/300 Write block 4390 len 28672 data: 5 5 3 3 3 3 3
9742 057/300 Read block 1386 len 12288
9743 058/300 Write block 1138 len 32768 data: 2 2 2 2 3 3 3 3
9744 059/300 Read block 3984 len 20480
9745 060/300 Flush
9746 061/300 Read block 2180 len 8192
9747 062/300 Write block 4225 len 28672 data: 2 3 3 3 3 3 3
9748 063/300 Flush
9749 064/300 Read block 356 len 16384
9750 065/300 Read block 2923 len 40960
9751 066/300 Read block 3558 len 28672
9752 067/300 Read block 3356 len 12288
9753 068/300 Write block 11 len 40960 data: 3 3 2 2 2 2 2 2 2 2
9754 069/300 Read block 1812 len 8192
9755 070/300 Write block 3707 len 24576 data: 2 2 2 2 2 2
9756 071/300 Read block 4896 len 20480
9757 072/300 Flush
9758 073/300 Write block 4374 len 32768 data: 2 3 3 3 3 3 3 3
9759 074/300 Read block 1046 len 40960
9760 075/300 Write block 42 len 32768 data: 4 2 2 2 2 2 2 2
9761 076/300 Write block 523 len 16384 data: 3 3 2 3
9762 077/300 Write block 3810 len 4096 data: 2
9763 078/300 Read block 2037 len 20480
9764 079/300 Write block 3850 len 28672 data: 3 3 4 4 4 3 3
9765 080/300 Read block 1663 len 16384
9766 081/300 Read block 4242 len 12288
9767 082/300 Read block 4269 len 36864
9768 083/300 Read block 2482 len 8192
9769 084/300 Read block 3182 len 16384
9770 085/300 Read block 3455 len 16384
9771 086/300 Flush
9772 087/300 Write block 4229 len 12288 data: 4 4 4
9773 088/300 Flush
9774 089/300 Read block 3291 len 28672
9775 090/300 Read block 527 len 40960
9776 091/300 Read block 2804 len 28672
9777 092/300 Read block 1003 len 24576
9778 093/300 Write block 1633 len 4096 data: 4
9779 094/300 Read block 3655 len 12288
9780 095/300 Read block 3348 len 24576
9781 096/300 Read block 1707 len 28672
9782 097/300 Write block 1019 len 12288 data: 3 3 3
9783 098/300 Read block 406 len 36864
9784 099/300 Read block 4819 len 16384
9785 100/300 Read block 2864 len 24576
9786 101/300 Write block 4944 len 4096 data: 2
9787 102/300 Read block 112 len 8192
9788 103/300 Write block 4589 len 16384 data: 2 3 3 3
9789 104/300 Write block 3489 len 12288 data: 3 2 2
9790 105/300 Read block 2536 len 16384
9791 106/300 Read block 3647 len 40960
9792 107/300 Read block 1723 len 36864
9793 108/300 Read block 2135 len 32768
9794 109/300 Read block 4350 len 40960
9795 110/300 Write block 1483 len 24576 data: 2 2 3 3 3 4
9796 111/300 Read block 178 len 16384
9797 112/300 Write block 2096 len 28672 data: 4 4 4 3 3 3 2
9798 113/300 Read block 1451 len 32768
9799 114/300 Read block 4709 len 12288
9800 115/300 Read block 726 len 4096
9801 116/300 Flush
9802 117/300 Write block 1820 len 12288 data: 2 3 3
9803 118/300 Write block 4402 len 16384 data: 2 2 2 4
9804 119/300 Write block 4116 len 20480 data: 3 3 2 2 2
9805 120/300 Read block 2755 len 12288
9806 121/300 Read block 4630 len 20480
9807 122/300 Write block 2986 len 28672 data: 3 3 3 3 4 4 4
9808 123/300 Write block 2559 len 28672 data: 2 2 2 2 3 3 2
9809 124/300 Read block 1579 len 24576
9810 125/300 Read block 2108 len 32768
9811 126/300 Read block 1134 len 12288
9812 127/300 Read block 1464 len 12288
9813 128/300 Write block 716 len 20480 data: 3 3 3 2 2
9814 129/300 Write block 278 len 12288 data: 5 4 4
9815 130/300 Write block 3417 len 12288 data: 4 3 2
9816 131/300 Write block 4517 len 32768 data: 2 3 3 3 3 2 2 3
9817 132/300 Read block 2360 len 4096
9818 133/300 Read block 943 len 32768
9819 134/300 Read block 3695 len 8192
9820 135/300 Write block 3006 len 24576 data: 2 3 3 3 3 3
9821 136/300 Read block 1982 len 20480
9822 137/300 Read block 917 len 20480
9823 138/300 Read block 1865 len 16384
9824 139/300 Read block 4265 len 36864
9825 140/300 Write block 444 len 32768 data: 7 7 6 7 7 8 8 8
9826 141/300 Read block 3725 len 28672
9827 142/300 Write block 4941 len 32768 data: 2 2 2 3 2 2 2 2
9828 143/300 Read block 3296 len 24576
9829 144/300 Flush
9830 145/300 Read block 412 len 40960
9831 146/300 Read block 4339 len 8192
9832 147/300 Read block 823 len 16384
9833 148/300 Read block 2286 len 28672
9834 149/300 Write block 2662 len 36864 data: 3 3 4 4 4 4 4 4 5
9835 150/300 Write block 1270 len 20480 data: 3 4 4 4 4
9836 151/300 Read block 3340 len 40960
9837 152/300 Write block 4643 len 16384 data: 2 2 2 2
9838 153/300 Read block 3806 len 16384
9839 154/300 Read block 849 len 16384
9840 155/300 Read block 1210 len 8192
9841 156/300 Read block 4980 len 28672
9842 157/300 Flush
9843 158/300 Read block 1961 len 24576
9844 159/300 Write block 4585 len 16384 data: 2 2 2 2
9845 160/300 Read block 3248 len 16384
9846 161/300 Write block 1227 len 16384 data: 2 2 2 2
9847 162/300 Read block 2041 len 8192
9848 163/300 Read block 3227 len 24576
9849 164/300 Read block 1305 len 4096
9850 165/300 Read block 4433 len 32768
9851 166/300 Read block 2243 len 24576
9852 167/300 Write block 126 len 20480 data: 3 4 4 4 4
9853 168/300 Read block 943 len 32768
9854 169/300 Read block 3090 len 24576
9855 170/300 Read block 2821 len 40960
9856 171/300 Read block 4480 len 32768
9857 172/300 Flush
9858 173/300 Read block 3193 len 32768
9859 174/300 Write block 2700 len 40960 data: 2 3 3 3 3 4 4 4 4 4
9860 175/300 Read block 2353 len 28672
9861 176/300 Read block 3823 len 24576
9862 177/300 Write block 1205 len 4096 data: 3
9863 178/300 Read block 4278 len 36864
9864 179/300 Write block 1229 len 12288 data: 3 3 2
9865 180/300 Read block 2277 len 16384
9866 181/300 Write block 4595 len 16384 data: 4 3 3 3
9867 182/300 Write block 4521 len 32768 data: 4 3 3 4 3 3 3 4
9868 183/300 Write block 3809 len 4096 data: 2
9869 184/300 Write block 3685 len 32768 data: 3 3 3 4 2 2 2 2
9870 185/300 Write block 1308 len 24576 data: 3 3 3 3 3 3
9871 186/300 Read block 3049 len 28672
9872 187/300 Read block 2394 len 8192
9873 188/300 Write block 2730 len 32768 data: 4 3 3 3 2 2 2 2
9874 189/300 Write block 2611 len 4096 data: 3
9875 190/300 Read block 2631 len 4096
9876 191/300 Write block 4652 len 16384 data: 4 4 4 4
9877 192/300 Write block 3679 len 20480 data: 2 2 3 3 3
9878 193/300 Write block 4228 len 40960 data: 4 5 5 5 2 2 2 2 2 2
9879 194/300 Flush
9880 195/300 Write block 2400 len 4096 data: 2
9881 196/300 Read block 3305 len 24576
9882 197/300 Read block 3782 len 12288
9883 198/300 Read block 1325 len 32768
9884 199/300 Read block 1356 len 20480
9885 200/300 Write block 1593 len 20480 data: 2 2 2 2 2
9886 201/300 Read block 4178 len 12288
9887 202/300 Read block 1045 len 28672
9888 203/300 Read block 1490 len 36864
9889 204/300 Read block 2627 len 16384
9890 205/300 Write block 4332 len 24576 data: 2 3 3 3 4 4
9891 206/300 Flush
9892 207/300 Write block 701 len 40960 data: 4 4 4 4 3 3 3 3 4 4
9893 208/300 Read block 3811 len 16384
9894 209/300 Write block 4617 len 16384 data: 2 2 2 2
9895 210/300 Write block 3782 len 8192 data: 5 5
9896 211/300 Write block 469 len 16384 data: 4 5 3 4
9897 212/300 Write block 3925 len 4096 data: 3
9898 213/300 Flush
9899 214/300 Read block 3743 len 8192
9900 215/300 Flush
9901 216/300 Flush
9902 217/300 Write block 2713 len 32768 data: 5 4 4 4 4 3 2 3
9903 218/300 Write block 3405 len 24576 data: 2 2 2 2 2 3
9904 219/300 Write block 359 len 4096 data: 3
9905 220/300 Read block 1108 len 32768
9906 221/300 Read block 3409 len 8192
9907 222/300 Read block 3256 len 32768
9908 223/300 Read block 4696 len 16384
9909 224/300 Flush
9910 225/300 Flush
9911 226/300 Flush
9912 227/300 Write block 520 len 24576 data: 3 4 4 4 4 3
9913 228/300 Read block 4764 len 36864
9914 229/300 Read block 1940 len 20480
9915 230/300 Write block 2051 len 12288 data: 3 3 3
9916 231/300 Write block 3940 len 20480 data: 6 5 4 3 3
9917 232/300 Flush
9918 233/300 Read block 1278 len 40960
9919 234/300 Flush
9920 235/300 Write block 292 len 4096 data: 3
9921 236/300 Write block 2164 len 28672 data: 2 2 2 2 3 2 2
9922 237/300 Read block 4607 len 36864
9923 238/300 Read block 948 len 16384
9924 239/300 Write block 2786 len 20480 data: 4 4 3 4 3
9925 240/300 Read block 4387 len 8192
9926 241/300 Read block 49 len 36864
9927 242/300 Read block 1277 len 32768
9928 243/300 Read block 4853 len 20480
9929 244/300 Read block 3347 len 36864
9930 245/300 Read block 3717 len 4096
9931 246/300 Read block 322 len 32768
9932 247/300 Write block 1877 len 20480 data: 3 3 3 3 3
9933 248/300 Write block 3276 len 16384 data: 2 2 2 2
9934 249/300 Write block 1027 len 20480 data: 3 2 2 2 2
9935 250/300 Flush
9936 251/300 Write block 3303 len 24576 data: 3 3 3 4 4 4
9937 252/300 Read block 897 len 8192
9938 253/300 Write block 1289 len 28672 data: 2 2 3 3 3 3 2
9939 254/300 Read block 4910 len 36864
9940 255/300 Read block 2978 len 36864
9941 256/300 Write block 1735 len 36864 data: 2 2 2 2 3 3 3 2 2
9942 257/300 Read block 3813 len 40960
9943 258/300 Read block 4038 len 12288
9944 259/300 Write block 529 len 4096 data: 3
9945 260/300 Read block 4706 len 28672
9946 261/300 Read block 1859 len 24576
9947 262/300 Write block 3305 len 36864 data: 4 5 5 5 4 3 3 4 4
9948 263/300 Flush
9949 264/300 Write block 1381 len 28672 data: 2 2 3 3 3 3 3
9950 265/300 Read block 861 len 28672
9951 266/300 Write block 1566 len 4096 data: 2
9952 267/300 Flush
9953 268/300 Write block 2702 len 20480 data: 4 4 4 5 5
9954 269/300 Write block 1080 len 4096 data: 5
9955 270/300 Read block 409 len 36864
9956 271/300 Flush
9957 272/300 Flush
9958 273/300 Flush
9959 274/300 Read block 1189 len 20480
9960 275/300 Read block 3086 len 4096
9961 276/300 Write block 3320 len 20480 data: 2 2 3 3 2
9962 277/300 Write block 2882 len 32768 data: 2 2 2 3 3 3 3 3
9963 278/300 Flush
9964 279/300 Read block 1292 len 12288
9965 280/300 Flush
9966 281/300 Read block 3212 len 4096
9967 282/300 Read block 2036 len 16384
9968 283/300 Write block 3034 len 36864 data: 3 2 2 2 2 2 2 2 2
9969 284/300 Read block 758 len 16384
9970 285/300 Write block 1707 len 32768 data: 4 4 4 4 4 4 4 4
9971 286/300 Write block 162 len 12288 data: 4 3 3
9972 287/300 Write block 997 len 24576 data: 2 2 2 3 4 4
9973 288/300 Read block 4746 len 4096
9974 289/300 Write block 4514 len 16384 data: 2 2 2 3
9975 290/300 Flush
9976 291/300 Write block 3201 len 36864 data: 2 2 2 2 3 3 3 2 2
9977 292/300 Write block 3274 len 8192 data: 2 2
9978 293/300 Read block 2993 len 24576
9979 294/300 Read block 3021 len 32768
9980 295/300 Write block 419 len 4096 data: 5
9981 296/300 Read block 1659 len 36864
9982 297/300 Write block 768 len 40960 data: 2 3 3 2 3 2 2 2 2 2
9983 298/300 Read block 1148 len 12288
9984 299/300 Flush
9985 300/300 Flush
9986 301/300 Read block 3405 len 16384
9987 Main task has work to do, go find it
9988 got dsc Start(1)
9989 start 1
9990 Feb 28 06:03:13.877 INFO request completed, latency_us: 181, response_code: 204, uri: /start/cid/1, method: POST, req_id: 6fa29bd4-578f-4fbc-a27e-a87dd001b953, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
9991 Replay: started 1, returned:Ok(())
9992 ----------------------------------------------------------------
9993 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9994 States: Active Offline Active
9995 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9996 3111 Acked Flush 0 Done Sent Done false
9997 3112 Acked Read 7 Done Sent Done false
9998 3113 Acked Write 7 Done Sent Done false
9999 3114 Acked Read 5 Done Sent Done false
10000 3115 Acked Read 4 Done Sent Done false
10001 3116 Acked Write 3 Done Sent Done false
10002 3117 Acked Write 5 Done Sent Done false
10003 3118 Acked Read 6 Done Sent Done false
10004 3119 Acked Write 8 Done Sent Done false
10005 3120 Acked Read 3 Done Sent Done false
10006 3121 Acked Write 5 Done Sent Done false
10007 3122 Acked Read 5 Done Sent Done false
10008 3123 Acked Read 6 Done Sent Done false
10009 3124 Acked Write 2 Done Sent Done false
10010 3125 Acked Read 9 Done Sent Done false
10011 3126 Acked Read 6 Done Sent Done false
10012 3127 Acked Read 10 Done Sent Done false
10013 3128 Acked Read 4 Done Sent Done false
10014 3129 Acked Read 2 Done Sent Done false
10015 3130 Acked Read 10 Done Sent Done false
10016 3131 Acked Write 1 Done Sent Done false
10017 3132 Acked Read 9 Done Sent Done false
10018 3133 Acked Write 8 Done Sent Done false
10019 3134 Acked Read 5 Done Sent Done false
10020 3135 Acked Write 6 Done Sent Done false
10021 3136 Acked Read 4 Done Sent Done false
10022 3137 Acked Read 1 Done Sent Done false
10023 3138 Acked Read 8 Done Sent Done false
10024 3139 Acked Read 9 Done Sent Done false
10025 3140 Acked Read 10 Done Sent Done false
10026 3141 Acked Write 4 Done Sent Done false
10027 3142 Acked Write 5 Done Sent Done false
10028 3143 Acked Write 4 Done Sent Done false
10029 3144 Acked Flush 0 Done Sent Done false
10030 3145 Acked Write 10 Done Sent Done false
10031 3146 Acked Write 3 Done Sent Done false
10032 3147 Acked Write 5 Done Sent Done false
10033 3148 Acked Write 7 Done Sent Done false
10034 3149 Acked Read 5 Done Sent Done false
10035 3150 Acked Write 2 Done Sent Done false
10036 3151 Acked Write 1 Done Sent Done false
10037 3152 Acked Write 9 Done Sent Done false
10038 3153 Acked Write 10 Done Sent Done false
10039 3154 Acked Write 7 Done Sent Done false
10040 3155 Acked Read 10 Done Sent Done false
10041 3156 Acked Write 4 Done Sent Done false
10042 3157 Acked Write 8 Done Sent Done false
10043 3158 Acked Read 7 Done Sent Done false
10044 3159 Acked Read 4 Done Sent Done false
10045 3160 Acked Read 6 Done Sent Done false
10046 3161 Acked Write 7 Done Sent Done false
10047 3162 Acked Write 4 Done Sent Done false
10048 3163 Acked Read 3 Done Sent Done false
10049 3164 Acked Read 5 Done Sent Done false
10050 3165 Acked Write 7 Done Sent Done false
10051 3166 Acked Write 7 Done Sent Done false
10052 3167 Acked Read 3 Done Sent Done false
10053 3168 Acked Write 8 Done Sent Done false
10054 3169 Acked Read 5 Done Sent Done false
10055 3170 Acked Flush 0 Done Sent Done false
10056 3171 Acked Read 2 Done Sent Done false
10057 3172 Acked Write 7 Done Sent Done false
10058 3173 Acked Flush 0 Done Sent Done false
10059 3174 Acked Read 4 Done Sent Done false
10060 3175 Acked Read 10 Done Sent Done false
10061 3176 Acked Read 7 Done Sent Done false
10062 3177 Acked Read 3 Done Sent Done false
10063 3178 Acked Write 10 Done Sent Done false
10064 3179 Acked Read 2 Done Sent Done false
10065 3180 Acked Write 6 Done Sent Done false
10066 3181 Acked Read 5 Done Sent Done false
10067 3182 Acked Flush 0 Done Sent Done false
10068 3183 Acked Write 8 Done Sent Done false
10069 3184 Acked Read 10 Done Sent Done false
10070 3185 Acked Write 8 Done Sent Done false
10071 3186 Acked Write 4 Done Sent Done false
10072 3187 Acked Write 1 Done Sent Done false
10073 3188 Acked Read 5 Done Sent Done false
10074 3189 Acked Write 7 Done Sent Done false
10075 3190 Acked Read 4 Done Sent Done false
10076 3191 Acked Read 3 Done Sent Done false
10077 3192 Acked Read 9 Done Sent Done false
10078 3193 Acked Read 2 Done Sent Done false
10079 3194 Acked Read 4 Done Sent Done false
10080 3195 Acked Read 4 Done Sent Done false
10081 3196 Acked Flush 0 Done Sent Done false
10082 3197 Acked Write 3 Done Sent Done false
10083 3198 Acked Flush 0 Done Sent Done false
10084 3199 Acked Read 7 Done Sent Done false
10085 3200 Acked Read 10 Done Sent Done false
10086 3201 Acked Read 7 Done Sent Done false
10087 3202 Acked Read 6 Done Sent Done false
10088 3203 Acked Write 1 Done Sent Done false
10089 3204 Acked Read 3 Done Sent Done false
10090 3205 Acked Read 6 Done Sent Done false
10091 3206 Acked Read 7 Done Sent Done false
10092 3207 Acked Write 3 Done Sent Done false
10093 3208 Acked Read 9 Done Sent Done false
10094 3209 Acked Read 4 Done Sent Done false
10095 3210 Acked Read 6 Done Sent Done false
10096 3211 Acked Write 1 Done Sent Done false
10097 3212 Acked Read 2 Done Sent Done false
10098 3213 Acked Write 4 Done Sent Done false
10099 3214 Acked Write 3 Done Sent Done false
10100 3215 Acked Read 4 Done Sent Done false
10101 3216 Acked Read 10 Done Sent Done false
10102 3217 Acked Read 9 Done Sent Done false
10103 3218 Acked Read 8 Done Sent Done false
10104 3219 Acked Read 10 Done Sent Done false
10105 3220 Acked Write 6 Done Sent Done false
10106 3221 Acked Read 4 Done Sent Done false
10107 3222 Acked Write 7 Done Sent Done false
10108 3223 Acked Read 8 Done Sent Done false
10109 3224 Acked Read 3 Done Sent Done false
10110 3225 Acked Read 1 Done Sent Done false
10111 3226 Acked Flush 0 Done Sent Done false
10112 3227 Acked Write 3 Done Sent Done false
10113 3228 Acked Write 4 Done Sent Done false
10114 3229 Acked Write 5 Done Sent Done false
10115 3230 Acked Read 3 Done Sent Done false
10116 3231 Acked Read 5 Done Sent Done false
10117 3232 Acked Write 7 Done Sent Done false
10118 3233 Acked Write 7 Done Sent Done false
10119 3234 Acked Read 6 Done Sent Done false
10120 3235 Acked Read 8 Done Sent Done false
10121 3236 Acked Read 3 Done Sent Done false
10122 3237 Acked Read 3 Done Sent Done false
10123 3238 Acked Write 5 Done Sent Done false
10124 3239 Acked Write 3 Done Sent Done false
10125 3240 Acked Write 3 Done Sent Done false
10126 3241 Acked Write 8 Done Sent Done false
10127 3242 Acked Read 1 Done Sent Done false
10128 3243 Acked Read 8 Done Sent Done false
10129 3244 Acked Read 2 Done Sent Done false
10130 3245 Acked Write 6 Done Sent Done false
10131 3246 Acked Read 5 Done Sent Done false
10132 3247 Acked Read 5 Done Sent Done false
10133 3248 Acked Read 4 Done Sent Done false
10134 3249 Acked Read 9 Done Sent Done false
10135 3250 Acked Write 8 Done Sent Done false
10136 3251 Acked Read 7 Done Sent Done false
10137 3252 Acked Write 8 Done Sent Done false
10138 3253 Acked Read 6 Done Sent Done false
10139 3254 Acked Flush 0 Done Sent Done false
10140 3255 Acked Read 10 Done Sent Done false
10141 3256 Acked Read 2 Done Sent Done false
10142 3257 Acked Read 4 Done Sent Done false
10143 3258 Acked Read 7 Done Sent Done false
10144 3259 Acked Write 9 Done Sent Done false
10145 3260 Acked Write 5 Done Sent Done false
10146 3261 Acked Read 10 Done Sent Done false
10147 3262 Acked Write 4 Done Sent Done false
10148 3263 Acked Read 4 Done Sent Done false
10149 3264 Acked Read 4 Done Sent Done false
10150 3265 Acked Read 2 Done Sent Done false
10151 3266 Acked Read 7 Done Sent Done false
10152 3267 Acked Flush 0 Done Sent Done false
10153 3268 Acked Read 6 Done Sent Done false
10154 3269 Acked Write 4 Done Sent Done false
10155 3270 Acked Read 4 Done Sent Done false
10156 3271 Acked Write 4 Done Sent Done false
10157 3272 Acked Read 2 Done Sent Done false
10158 3273 Acked Read 6 Done Sent Done false
10159 3274 Acked Read 1 Done Sent Done false
10160 3275 Acked Read 8 Done Sent Done false
10161 3276 Acked Read 6 Done Sent Done false
10162 3277 Acked Write 5 Done Sent Done false
10163 3278 Acked Read 8 Done Sent Done false
10164 3279 Acked Read 6 Done Sent Done false
10165 3280 Acked Read 10 Done Sent Done false
10166 3281 Acked Read 8 Done Sent Done false
10167 3282 Acked Flush 0 Done Sent Done false
10168 3283 Acked Read 8 Done Sent Done false
10169 3284 Acked Write 10 Done Sent Done false
10170 3285 Acked Read 7 Done Sent Done false
10171 3286 Acked Read 6 Done Sent Done false
10172 3287 Acked Write 1 Done Sent Done false
10173 3288 Acked Read 9 Done Sent Done false
10174 3289 Acked Write 3 Done Sent Done false
10175 3290 Acked Read 4 Done Sent Done false
10176 3291 Acked Write 4 Done Sent Done false
10177 3292 Acked Write 8 Done Sent Done false
10178 3293 Acked Write 1 Done Sent Done false
10179 3294 Acked Write 8 Done Sent Done false
10180 3295 Acked Write 6 Done Sent Done false
10181 3296 Acked Read 7 Done Sent Done false
10182 3297 Acked Read 2 Done Sent Done false
10183 3298 Acked Write 8 Done Sent Done false
10184 3299 Acked Write 1 Done Sent Done false
10185 3300 Acked Read 1 Done Sent Done false
10186 3301 Acked Write 4 Done Sent Done false
10187 3302 Acked Write 5 Done Sent Done false
10188 3303 Acked Write 10 Done Sent Done false
10189 3304 Acked Flush 0 Done Sent Done false
10190 3305 Acked Write 1 Done Sent Done false
10191 3306 Acked Read 6 Done Sent Done false
10192 3307 Acked Read 3 Done Sent Done false
10193 3308 Acked Read 8 Done Sent Done false
10194 3309 Acked Read 5 Done Sent Done false
10195 3310 Acked Write 5 Done Sent Done false
10196 3311 Acked Read 3 Done Sent Done false
10197 3312 Acked Read 7 Done Sent Done false
10198 3313 Acked Read 9 Done Sent Done false
10199 3314 Acked Read 4 Done Sent Done false
10200 3315 Acked Write 6 Done Sent Done false
10201 3316 Acked Flush 0 Done Sent Done false
10202 3317 Acked Write 10 Done Sent Done false
10203 3318 Acked Read 4 Done Sent Done false
10204 3319 Acked Write 4 Done Sent Done false
10205 3320 Acked Write 2 Done Sent Done false
10206 3321 Acked Write 4 Done Sent Done false
10207 3322 Acked Write 1 Done Sent Done false
10208 3323 Acked Flush 0 Done Sent Done false
10209 3324 Acked Read 2 Done Sent Done false
10210 3325 Acked Flush 0 Done Sent Done false
10211 3326 Acked Flush 0 Done Sent Done false
10212 3327 Acked Write 8 Done Sent Done false
10213 3328 Acked Write 6 Done Sent Done false
10214 3329 Acked Write 1 Done Sent Done false
10215 3330 Acked Read 8 Done Sent Done false
10216 3331 Acked Read 2 Done Sent Done false
10217 3332 Acked Read 8 Done Sent Done false
10218 3333 Acked Read 4 Done Sent Done false
10219 3334 Acked Flush 0 Done Sent Done false
10220 3335 Acked Flush 0 Done Sent Done false
10221 3336 Acked Flush 0 Done Sent Done false
10222 3337 Acked Write 6 Done Sent Done false
10223 3338 Acked Read 9 Done Sent Done false
10224 3339 Acked Read 5 Done Sent Done false
10225 3340 Acked Write 3 Done Sent Done false
10226 3341 Acked Write 5 Done Sent Done false
10227 3342 Acked Flush 0 Done Sent Done false
10228 3343 Acked Read 10 Done Sent Done false
10229 3344 Acked Flush 0 Done Sent Done false
10230 3345 Acked Write 1 Done Sent Done false
10231 3346 Acked Write 7 Done Sent Done false
10232 3347 Acked Read 9 Done Sent Done false
10233 3348 Acked Read 4 Done Sent Done false
10234 3349 Acked Write 5 Done Sent Done false
10235 3350 Acked Read 2 Done Sent Done false
10236 3351 Acked Read 9 Done Sent Done false
10237 3352 Acked Read 8 Done Sent Done false
10238 3353 Acked Read 5 Done Sent Done false
10239 3354 Acked Read 9 Done Sent Done false
10240 3355 Acked Read 1 Done Sent Done false
10241 3356 Acked Read 8 Done Sent Done false
10242 3357 Acked Write 5 Done Sent Done false
10243 3358 Acked Write 4 Done Sent Done false
10244 3359 Acked Write 5 Done Sent Done false
10245 3360 Acked Flush 0 Done Sent Done false
10246 3361 Acked Write 6 Done Sent Done false
10247 3362 Acked Read 2 Done Sent Done false
10248 3363 Acked Write 7 Done Sent Done false
10249 3364 Acked Read 9 Done Sent Done false
10250 3365 Acked Read 9 Done Sent Done false
10251 3366 Acked Write 9 Done Sent Done false
10252 3367 Acked Read 10 Done Sent Done false
10253 3368 Acked Read 3 Done Sent Done false
10254 3369 Acked Write 1 Done Sent Done false
10255 3370 Acked Read 7 Done Sent Done false
10256 3371 Acked Read 6 Done Sent Done false
10257 3372 Acked Write 9 Done Sent Done false
10258 3373 Acked Flush 0 Done Sent Done false
10259 3374 Acked Write 7 Done Sent Done false
10260 3375 Acked Read 7 Done Sent Done false
10261 3376 Acked Write 1 Done Sent Done false
10262 3377 Acked Flush 0 Done Sent Done false
10263 3378 Acked Write 5 Done Sent Done false
10264 3379 Acked Write 1 Done Sent Done false
10265 3380 Acked Read 9 Done Sent Done false
10266 3381 Acked Flush 0 Done Sent Done false
10267 3382 Acked Flush 0 Done Sent Done false
10268 3383 Acked Flush 0 Done Sent Done false
10269 3384 Acked Read 5 Done Sent Done false
10270 3385 Acked Read 1 Done Sent Done false
10271 3386 Acked Write 5 Done Sent Done false
10272 3387 Acked Write 8 Done Sent Done false
10273 3388 Acked Flush 0 Done Sent Done false
10274 3389 Acked Read 3 Done Sent Done false
10275 3390 Acked Flush 0 Done Sent Done false
10276 3391 Acked Read 1 Done Sent Done false
10277 3392 Acked Read 4 Done Sent Done false
10278 3393 Acked Write 9 Done Sent Done false
10279 3394 Acked Read 4 Done Sent Done false
10280 3395 Acked Write 8 Done Sent Done false
10281 3396 Acked Write 3 Done Sent Done false
10282 3397 Acked Write 6 Done Sent Done false
10283 3398 Acked Read 1 Done Sent Done false
10284 3399 Acked Write 4 Done Sent Done false
10285 3400 Acked Flush 0 Done Sent Done false
10286 3401 Acked Write 9 Done Sent Done false
10287 3402 Acked Write 2 Done Sent Done false
10288 3403 Acked Read 6 Done Sent Done false
10289 3404 Acked Read 8 Done Sent Done false
10290 3405 Acked Write 1 Done Sent Done false
10291 3406 Acked Read 9 Done Sent Done false
10292 3407 Acked Write 10 Done Sent Done false
10293 3408 Acked Read 3 Done Sent Done false
10294 3409 Acked Flush 0 Done Sent Done false
10295 3410 Acked Flush 0 Done Sent Done false
10296 3411 Acked Read 4 Done Sent Done false
10297 STATES DS:0 DS:1 DS:2 TOTAL
10298 Sent 0 301 0 301
10299 Done 301 0 301 602
10300 Skipped 0 0 0 0
10301 Error 0 0 0 0
10302 Last Flush: Some(JobId(3410)) Some(JobId(3110)) Some(JobId(3410))
10303 Downstairs last five completed: 3110 3109 3108 3107 3106
10304 Upstairs last five completed: 3411 3410 3409 3408 3407
10305 CLIENT: Up:0 ds:301 act:2
103062025-02-28T06:03:14.556ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
103072025-02-28T06:03:15.557ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
10308 [8820] got start action
10309 [8820] I am going to restart
10310 Starting downstairs at port 8820
10311 Make output file at "/tmp/dsc/downstairs-8820.txt"
10312 [8820][1] reports Starting
10313 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1219)
10314 [8820][1] reports Running
103152025-02-28T06:03:16.559ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
10316 ----------------------------------------------------------------
10317 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10318 States: Active Offline Active
10319 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10320 3111 Acked Flush 0 Done Sent Done false
10321 3112 Acked Read 7 Done Sent Done false
10322 3113 Acked Write 7 Done Sent Done false
10323 3114 Acked Read 5 Done Sent Done false
10324 3115 Acked Read 4 Done Sent Done false
10325 3116 Acked Write 3 Done Sent Done false
10326 3117 Acked Write 5 Done Sent Done false
10327 3118 Acked Read 6 Done Sent Done false
10328 3119 Acked Write 8 Done Sent Done false
10329 3120 Acked Read 3 Done Sent Done false
10330 3121 Acked Write 5 Done Sent Done false
10331 3122 Acked Read 5 Done Sent Done false
10332 3123 Acked Read 6 Done Sent Done false
10333 3124 Acked Write 2 Done Sent Done false
10334 3125 Acked Read 9 Done Sent Done false
10335 3126 Acked Read 6 Done Sent Done false
10336 3127 Acked Read 10 Done Sent Done false
10337 3128 Acked Read 4 Done Sent Done false
10338 3129 Acked Read 2 Done Sent Done false
10339 3130 Acked Read 10 Done Sent Done false
10340 3131 Acked Write 1 Done Sent Done false
10341 3132 Acked Read 9 Done Sent Done false
10342 3133 Acked Write 8 Done Sent Done false
10343 3134 Acked Read 5 Done Sent Done false
10344 3135 Acked Write 6 Done Sent Done false
10345 3136 Acked Read 4 Done Sent Done false
10346 3137 Acked Read 1 Done Sent Done false
10347 3138 Acked Read 8 Done Sent Done false
10348 3139 Acked Read 9 Done Sent Done false
10349 3140 Acked Read 10 Done Sent Done false
10350 3141 Acked Write 4 Done Sent Done false
10351 3142 Acked Write 5 Done Sent Done false
10352 3143 Acked Write 4 Done Sent Done false
10353 3144 Acked Flush 0 Done Sent Done false
10354 3145 Acked Write 10 Done Sent Done false
10355 3146 Acked Write 3 Done Sent Done false
10356 3147 Acked Write 5 Done Sent Done false
10357 3148 Acked Write 7 Done Sent Done false
10358 3149 Acked Read 5 Done Sent Done false
10359 3150 Acked Write 2 Done Sent Done false
10360 3151 Acked Write 1 Done Sent Done false
10361 3152 Acked Write 9 Done Sent Done false
10362 3153 Acked Write 10 Done Sent Done false
10363 3154 Acked Write 7 Done Sent Done false
10364 3155 Acked Read 10 Done Sent Done false
10365 3156 Acked Write 4 Done Sent Done false
10366 3157 Acked Write 8 Done Sent Done false
10367 3158 Acked Read 7 Done Sent Done false
10368 3159 Acked Read 4 Done Sent Done false
10369 3160 Acked Read 6 Done Sent Done false
10370 3161 Acked Write 7 Done Sent Done false
10371 3162 Acked Write 4 Done Sent Done false
10372 3163 Acked Read 3 Done Sent Done false
10373 3164 Acked Read 5 Done Sent Done false
10374 3165 Acked Write 7 Done Sent Done false
10375 3166 Acked Write 7 Done Sent Done false
10376 3167 Acked Read 3 Done Sent Done false
10377 3168 Acked Write 8 Done Sent Done false
10378 3169 Acked Read 5 Done Sent Done false
10379 3170 Acked Flush 0 Done Sent Done false
10380 3171 Acked Read 2 Done Sent Done false
10381 3172 Acked Write 7 Done Sent Done false
10382 3173 Acked Flush 0 Done Sent Done false
10383 3174 Acked Read 4 Done Sent Done false
10384 3175 Acked Read 10 Done Sent Done false
10385 3176 Acked Read 7 Done Sent Done false
10386 3177 Acked Read 3 Done Sent Done false
10387 3178 Acked Write 10 Done Sent Done false
10388 3179 Acked Read 2 Done Sent Done false
10389 3180 Acked Write 6 Done Sent Done false
10390 3181 Acked Read 5 Done Sent Done false
10391 3182 Acked Flush 0 Done Sent Done false
10392 3183 Acked Write 8 Done Sent Done false
10393 3184 Acked Read 10 Done Sent Done false
10394 3185 Acked Write 8 Done Sent Done false
10395 3186 Acked Write 4 Done Sent Done false
10396 3187 Acked Write 1 Done Sent Done false
10397 3188 Acked Read 5 Done Sent Done false
10398 3189 Acked Write 7 Done Sent Done false
10399 3190 Acked Read 4 Done Sent Done false
10400 3191 Acked Read 3 Done Sent Done false
10401 3192 Acked Read 9 Done Sent Done false
10402 3193 Acked Read 2 Done Sent Done false
10403 3194 Acked Read 4 Done Sent Done false
10404 3195 Acked Read 4 Done Sent Done false
10405 3196 Acked Flush 0 Done Sent Done false
10406 3197 Acked Write 3 Done Sent Done false
10407 3198 Acked Flush 0 Done Sent Done false
10408 3199 Acked Read 7 Done Sent Done false
10409 3200 Acked Read 10 Done Sent Done false
10410 3201 Acked Read 7 Done Sent Done false
10411 3202 Acked Read 6 Done Sent Done false
10412 3203 Acked Write 1 Done Sent Done false
10413 3204 Acked Read 3 Done Sent Done false
10414 3205 Acked Read 6 Done Sent Done false
10415 3206 Acked Read 7 Done Sent Done false
10416 3207 Acked Write 3 Done Sent Done false
10417 3208 Acked Read 9 Done Sent Done false
10418 3209 Acked Read 4 Done Sent Done false
10419 3210 Acked Read 6 Done Sent Done false
10420 3211 Acked Write 1 Done Sent Done false
10421 3212 Acked Read 2 Done Sent Done false
10422 3213 Acked Write 4 Done Sent Done false
10423 3214 Acked Write 3 Done Sent Done false
10424 3215 Acked Read 4 Done Sent Done false
10425 3216 Acked Read 10 Done Sent Done false
10426 3217 Acked Read 9 Done Sent Done false
10427 3218 Acked Read 8 Done Sent Done false
10428 3219 Acked Read 10 Done Sent Done false
10429 3220 Acked Write 6 Done Sent Done false
10430 3221 Acked Read 4 Done Sent Done false
10431 3222 Acked Write 7 Done Sent Done false
10432 3223 Acked Read 8 Done Sent Done false
10433 3224 Acked Read 3 Done Sent Done false
10434 3225 Acked Read 1 Done Sent Done false
10435 3226 Acked Flush 0 Done Sent Done false
10436 3227 Acked Write 3 Done Sent Done false
10437 3228 Acked Write 4 Done Sent Done false
10438 3229 Acked Write 5 Done Sent Done false
10439 3230 Acked Read 3 Done Sent Done false
10440 3231 Acked Read 5 Done Sent Done false
10441 3232 Acked Write 7 Done Sent Done false
10442 3233 Acked Write 7 Done Sent Done false
10443 3234 Acked Read 6 Done Sent Done false
10444 3235 Acked Read 8 Done Sent Done false
10445 3236 Acked Read 3 Done Sent Done false
10446 3237 Acked Read 3 Done Sent Done false
10447 3238 Acked Write 5 Done Sent Done false
10448 3239 Acked Write 3 Done Sent Done false
10449 3240 Acked Write 3 Done Sent Done false
10450 3241 Acked Write 8 Done Sent Done false
10451 3242 Acked Read 1 Done Sent Done false
10452 3243 Acked Read 8 Done Sent Done false
10453 3244 Acked Read 2 Done Sent Done false
10454 3245 Acked Write 6 Done Sent Done false
10455 3246 Acked Read 5 Done Sent Done false
10456 3247 Acked Read 5 Done Sent Done false
10457 3248 Acked Read 4 Done Sent Done false
10458 3249 Acked Read 9 Done Sent Done false
10459 3250 Acked Write 8 Done Sent Done false
10460 3251 Acked Read 7 Done Sent Done false
10461 3252 Acked Write 8 Done Sent Done false
10462 3253 Acked Read 6 Done Sent Done false
10463 3254 Acked Flush 0 Done Sent Done false
10464 3255 Acked Read 10 Done Sent Done false
10465 3256 Acked Read 2 Done Sent Done false
10466 3257 Acked Read 4 Done Sent Done false
10467 3258 Acked Read 7 Done Sent Done false
10468 3259 Acked Write 9 Done Sent Done false
10469 3260 Acked Write 5 Done Sent Done false
10470 3261 Acked Read 10 Done Sent Done false
10471 3262 Acked Write 4 Done Sent Done false
10472 3263 Acked Read 4 Done Sent Done false
10473 3264 Acked Read 4 Done Sent Done false
10474 3265 Acked Read 2 Done Sent Done false
10475 3266 Acked Read 7 Done Sent Done false
10476 3267 Acked Flush 0 Done Sent Done false
10477 3268 Acked Read 6 Done Sent Done false
10478 3269 Acked Write 4 Done Sent Done false
10479 3270 Acked Read 4 Done Sent Done false
10480 3271 Acked Write 4 Done Sent Done false
10481 3272 Acked Read 2 Done Sent Done false
10482 3273 Acked Read 6 Done Sent Done false
10483 3274 Acked Read 1 Done Sent Done false
10484 3275 Acked Read 8 Done Sent Done false
10485 3276 Acked Read 6 Done Sent Done false
10486 3277 Acked Write 5 Done Sent Done false
10487 3278 Acked Read 8 Done Sent Done false
10488 3279 Acked Read 6 Done Sent Done false
10489 3280 Acked Read 10 Done Sent Done false
10490 3281 Acked Read 8 Done Sent Done false
10491 3282 Acked Flush 0 Done Sent Done false
10492 3283 Acked Read 8 Done Sent Done false
10493 3284 Acked Write 10 Done Sent Done false
10494 3285 Acked Read 7 Done Sent Done false
10495 3286 Acked Read 6 Done Sent Done false
10496 3287 Acked Write 1 Done Sent Done false
10497 3288 Acked Read 9 Done Sent Done false
10498 3289 Acked Write 3 Done Sent Done false
10499 3290 Acked Read 4 Done Sent Done false
10500 3291 Acked Write 4 Done Sent Done false
10501 3292 Acked Write 8 Done Sent Done false
10502 3293 Acked Write 1 Done Sent Done false
10503 3294 Acked Write 8 Done Sent Done false
10504 3295 Acked Write 6 Done Sent Done false
10505 3296 Acked Read 7 Done Sent Done false
10506 3297 Acked Read 2 Done Sent Done false
10507 3298 Acked Write 8 Done Sent Done false
10508 3299 Acked Write 1 Done Sent Done false
10509 3300 Acked Read 1 Done Sent Done false
10510 3301 Acked Write 4 Done Sent Done false
10511 3302 Acked Write 5 Done Sent Done false
10512 3303 Acked Write 10 Done Sent Done false
10513 3304 Acked Flush 0 Done Sent Done false
10514 3305 Acked Write 1 Done Sent Done false
10515 3306 Acked Read 6 Done Sent Done false
10516 3307 Acked Read 3 Done Sent Done false
10517 3308 Acked Read 8 Done Sent Done false
10518 3309 Acked Read 5 Done Sent Done false
10519 3310 Acked Write 5 Done Sent Done false
10520 3311 Acked Read 3 Done Sent Done false
10521 3312 Acked Read 7 Done Sent Done false
10522 3313 Acked Read 9 Done Sent Done false
10523 3314 Acked Read 4 Done Sent Done false
10524 3315 Acked Write 6 Done Sent Done false
10525 3316 Acked Flush 0 Done Sent Done false
10526 3317 Acked Write 10 Done Sent Done false
10527 3318 Acked Read 4 Done Sent Done false
10528 3319 Acked Write 4 Done Sent Done false
10529 3320 Acked Write 2 Done Sent Done false
10530 3321 Acked Write 4 Done Sent Done false
10531 3322 Acked Write 1 Done Sent Done false
10532 3323 Acked Flush 0 Done Sent Done false
10533 3324 Acked Read 2 Done Sent Done false
10534 3325 Acked Flush 0 Done Sent Done false
10535 3326 Acked Flush 0 Done Sent Done false
10536 3327 Acked Write 8 Done Sent Done false
10537 3328 Acked Write 6 Done Sent Done false
10538 3329 Acked Write 1 Done Sent Done false
10539 3330 Acked Read 8 Done Sent Done false
10540 3331 Acked Read 2 Done Sent Done false
10541 3332 Acked Read 8 Done Sent Done false
10542 3333 Acked Read 4 Done Sent Done false
10543 3334 Acked Flush 0 Done Sent Done false
10544 3335 Acked Flush 0 Done Sent Done false
10545 3336 Acked Flush 0 Done Sent Done false
10546 3337 Acked Write 6 Done Sent Done false
10547 3338 Acked Read 9 Done Sent Done false
10548 3339 Acked Read 5 Done Sent Done false
10549 3340 Acked Write 3 Done Sent Done false
10550 3341 Acked Write 5 Done Sent Done false
10551 3342 Acked Flush 0 Done Sent Done false
10552 3343 Acked Read 10 Done Sent Done false
10553 3344 Acked Flush 0 Done Sent Done false
10554 3345 Acked Write 1 Done Sent Done false
10555 3346 Acked Write 7 Done Sent Done false
10556 3347 Acked Read 9 Done Sent Done false
10557 3348 Acked Read 4 Done Sent Done false
10558 3349 Acked Write 5 Done Sent Done false
10559 3350 Acked Read 2 Done Sent Done false
10560 3351 Acked Read 9 Done Sent Done false
10561 3352 Acked Read 8 Done Sent Done false
10562 3353 Acked Read 5 Done Sent Done false
10563 3354 Acked Read 9 Done Sent Done false
10564 3355 Acked Read 1 Done Sent Done false
10565 3356 Acked Read 8 Done Sent Done false
10566 3357 Acked Write 5 Done Sent Done false
10567 3358 Acked Write 4 Done Sent Done false
10568 3359 Acked Write 5 Done Sent Done false
10569 3360 Acked Flush 0 Done Sent Done false
10570 3361 Acked Write 6 Done Sent Done false
10571 3362 Acked Read 2 Done Sent Done false
10572 3363 Acked Write 7 Done Sent Done false
10573 3364 Acked Read 9 Done Sent Done false
10574 3365 Acked Read 9 Done Sent Done false
10575 3366 Acked Write 9 Done Sent Done false
10576 3367 Acked Read 10 Done Sent Done false
10577 3368 Acked Read 3 Done Sent Done false
10578 3369 Acked Write 1 Done Sent Done false
10579 3370 Acked Read 7 Done Sent Done false
10580 3371 Acked Read 6 Done Sent Done false
10581 3372 Acked Write 9 Done Sent Done false
10582 3373 Acked Flush 0 Done Sent Done false
10583 3374 Acked Write 7 Done Sent Done false
10584 3375 Acked Read 7 Done Sent Done false
10585 3376 Acked Write 1 Done Sent Done false
10586 3377 Acked Flush 0 Done Sent Done false
10587 3378 Acked Write 5 Done Sent Done false
10588 3379 Acked Write 1 Done Sent Done false
10589 3380 Acked Read 9 Done Sent Done false
10590 3381 Acked Flush 0 Done Sent Done false
10591 3382 Acked Flush 0 Done Sent Done false
10592 3383 Acked Flush 0 Done Sent Done false
10593 3384 Acked Read 5 Done Sent Done false
10594 3385 Acked Read 1 Done Sent Done false
10595 3386 Acked Write 5 Done Sent Done false
10596 3387 Acked Write 8 Done Sent Done false
10597 3388 Acked Flush 0 Done Sent Done false
10598 3389 Acked Read 3 Done Sent Done false
10599 3390 Acked Flush 0 Done Sent Done false
10600 3391 Acked Read 1 Done Sent Done false
10601 3392 Acked Read 4 Done Sent Done false
10602 3393 Acked Write 9 Done Sent Done false
10603 3394 Acked Read 4 Done Sent Done false
10604 3395 Acked Write 8 Done Sent Done false
10605 3396 Acked Write 3 Done Sent Done false
10606 3397 Acked Write 6 Done Sent Done false
10607 3398 Acked Read 1 Done Sent Done false
10608 3399 Acked Write 4 Done Sent Done false
10609 3400 Acked Flush 0 Done Sent Done false
10610 3401 Acked Write 9 Done Sent Done false
10611 3402 Acked Write 2 Done Sent Done false
10612 3403 Acked Read 6 Done Sent Done false
10613 3404 Acked Read 8 Done Sent Done false
10614 3405 Acked Write 1 Done Sent Done false
10615 3406 Acked Read 9 Done Sent Done false
10616 3407 Acked Write 10 Done Sent Done false
10617 3408 Acked Read 3 Done Sent Done false
10618 3409 Acked Flush 0 Done Sent Done false
10619 3410 Acked Flush 0 Done Sent Done false
10620 3411 Acked Read 4 Done Sent Done false
10621 STATES DS:0 DS:1 DS:2 TOTAL
10622 Sent 0 301 0 301
10623 Done 301 0 301 602
10624 Skipped 0 0 0 0
10625 Error 0 0 0 0
10626 Last Flush: Some(JobId(3410)) Some(JobId(3110)) Some(JobId(3410))
10627 Downstairs last five completed: 3110 3109 3108 3107 3106
10628 Upstairs last five completed: 3411 3410 3409 3408 3407
10629 CLIENT: Up:0 ds:301 act:2
106302025-02-28T06:03:18.563ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106312025-02-28T06:03:19.565ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106322025-02-28T06:03:19.634ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106332025-02-28T06:03:19.634ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52870) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106342025-02-28T06:03:19.635ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106352025-02-28T06:03:19.636ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106362025-02-28T06:03:19.636ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106372025-02-28T06:03:19.636ZINFOcrucible: send last flush ID to this DS: Some(JobId(3110)) = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106382025-02-28T06:03:19.636ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106392025-02-28T06:03:19.636ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(3110)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106402025-02-28T06:03:19.636ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(3110)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
10641 ----------------------------------------------------------------
10642 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10643 Downstairs last five completed: 3412 3411 3410 3409 3408
10644 Upstairs last five completed: 3412 3411 3410 3409 3408
10645 CLIENT: Up:0 ds:0 act:3
10646 Replay: All jobs finished
10647 Main task has work to do, go find it
10648 got dsc Stop(2)
10649 stop 2
10650 [8830] Got stop action so:false kr:false
10651 Feb 28 06:03:21.882 INFO request completed, latency_us: 327, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 7ccec0b5-38d9-4767-bd99-c6be8b5f76f1, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
10652 Feb 28 06:03:21.883 INFO request completed, latency_us: 121, response_code: 200, uri: /state/cid/2, method: GET, req_id: 32b6a838-f234-4d60-9762-afb6912cd3be, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
106532025-02-28T06:03:21.883ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106542025-02-28T06:03:21.883ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106552025-02-28T06:03:21.883ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
106562025-02-28T06:03:21.883ZINFOcrucible: client task is exiting = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
10657 [8830] Exited with: ExitStatus(unix_wait_status(9))
10658 [8830][2] reports Exit
10659 Feb 28 06:03:25.884 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/2, method: GET, req_id: d1360034-b88b-4543-96b2-04a589fdf7c6, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
10660 001/300 Write block 2672 len 16384 data: 4 3 4 4
10661 002/300 Write block 475 len 8192 data: 4 4
10662 003/300 Flush
10663 004/300 Write block 833 len 20480 data: 3 3 3 2 2
10664 005/300 Read block 4623 len 8192
10665 006/300 Read block 3077 len 16384
10666 007/300 Read block 2965 len 32768
10667 008/300 Write block 2866 len 36864 data: 4 4 4 4 4 4 4 3 3
10668 009/300 Read block 1420 len 36864
10669 010/300 Read block 4666 len 24576
10670 011/300 Flush
10671 012/300 Write block 249 len 24576 data: 2 2 3 3 3 2
10672 013/300 Read block 3272 len 36864
10673 014/300 Read block 261 len 12288
10674 015/300 Write block 1831 len 40960 data: 2 2 2 2 2 2 2 3 4 4
10675 016/300 Read block 457 len 24576
10676 017/300 Read block 684 len 20480
10677 018/300 Write block 3597 len 4096 data: 3
10678 019/300 Write block 3220 len 40960 data: 3 3 3 3 3 3 3 2 2 2
10679 020/300 Read block 1110 len 8192
10680 021/300 Read block 3814 len 4096
10681 022/300 Read block 900 len 16384
10682 023/300 Read block 4655 len 20480
10683 024/300 Flush
10684 025/300 Write block 4815 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10685 026/300 Write block 3218 len 16384 data: 3 3 4 4
10686 027/300 Write block 3061 len 40960 data: 2 2 2 2 2 2 3 2 2 2
10687 028/300 Read block 4837 len 16384
10688 029/300 Read block 1060 len 36864
10689 030/300 Write block 219 len 24576 data: 4 3 4 3 3 3
10690 031/300 Read block 815 len 36864
10691 032/300 Write block 4692 len 8192 data: 2 2
10692 033/300 Read block 72 len 24576
10693 034/300 Read block 118 len 8192
10694 035/300 Read block 717 len 28672
10695 036/300 Write block 1901 len 12288 data: 3 2 2
10696 037/300 Write block 3720 len 36864 data: 5 5 5 5 5 3 2 2 2
10697 038/300 Write block 1440 len 28672 data: 4 5 5 5 5 5 4
10698 039/300 Write block 1322 len 32768 data: 3 3 3 3 3 3 3 4
10699 040/300 Read block 560 len 16384
10700 041/300 Read block 795 len 8192
10701 042/300 Read block 3225 len 8192
10702 043/300 Write block 4887 len 40960 data: 5 6 6 5 5 3 3 3 3 3
10703 044/300 Read block 3822 len 24576
10704 045/300 Write block 2214 len 36864 data: 6 6 6 6 6 4 2 2 2
10705 046/300 Read block 3548 len 28672
10706 047/300 Read block 3651 len 40960
10707 048/300 Read block 3848 len 4096
10708 049/300 Write block 3463 len 20480 data: 2 2 3 3 3
10709 050/300 Read block 1862 len 20480
10710 051/300 Write block 1485 len 24576 data: 4 4 4 5 5 5
10711 052/300 Write block 4212 len 24576 data: 4 4 3 3 3 3
10712 053/300 Read block 1050 len 8192
10713 054/300 Read block 1334 len 32768
10714 055/300 Read block 1769 len 8192
10715 056/300 Flush
10716 057/300 Write block 1420 len 32768 data: 2 2 2 2 2 2 2 2
10717 058/300 Write block 2272 len 36864 data: 2 4 4 4 4 4 5 5 5
10718 059/300 Flush
10719 060/300 Flush
10720 061/300 Write block 3488 len 24576 data: 3 4 3 3 2 2
10721 062/300 Write block 1616 len 36864 data: 4 4 3 3 4 4 2 2 5
10722 063/300 Read block 3637 len 36864
10723 064/300 Read block 1457 len 4096
10724 065/300 Read block 2804 len 16384
10725 066/300 Read block 494 len 12288
10726 067/300 Read block 3467 len 40960
10727 068/300 Read block 4244 len 24576
10728 069/300 Read block 4458 len 20480
10729 070/300 Write block 2950 len 36864 data: 4 4 4 4 4 4 3 4 4
10730 071/300 Read block 1827 len 24576
10731 072/300 Read block 4521 len 32768
10732 073/300 Read block 4175 len 8192
10733 074/300 Write block 4456 len 40960 data: 2 2 2 2 2 3 3 3 3 3
10734 075/300 Read block 2190 len 24576
10735 076/300 Flush
10736 077/300 Read block 275 len 4096
10737 078/300 Write block 3704 len 24576 data: 2 2 2 3 3 3
10738 079/300 Write block 3862 len 12288 data: 2 2 3
10739 080/300 Flush
10740 081/300 Flush
10741 082/300 Read block 4395 len 16384
10742 083/300 Write block 1810 len 24576 data: 2 2 2 2 2 2
10743 084/300 Read block 1174 len 28672
10744 085/300 Read block 2416 len 20480
10745 086/300 Write block 2570 len 16384 data: 4 3 3 3
10746 087/300 Write block 372 len 40960 data: 2 3 3 3 3 4 5 5 6 5
10747 088/300 Read block 116 len 28672
10748 089/300 Write block 4442 len 28672 data: 3 3 3 3 2 2 2
10749 090/300 Write block 2990 len 40960 data: 5 5 5 2 2 2 2 2 2 2
10750 091/300 Read block 3751 len 16384
10751 092/300 Read block 4248 len 12288
10752 093/300 Write block 3049 len 8192 data: 2 2
10753 094/300 Read block 3863 len 4096
10754 095/300 Read block 2327 len 36864
10755 096/300 Read block 1632 len 12288
10756 097/300 Write block 4337 len 40960 data: 5 4 5 4 3 3 3 3 3 3
10757 098/300 Write block 1264 len 4096 data: 5
10758 099/300 Read block 1565 len 4096
10759 100/300 Flush
10760 101/300 Read block 149 len 4096
10761 102/300 Write block 2440 len 36864 data: 3 2 3 3 3 4 4 4 3
10762 103/300 Read block 4442 len 32768
10763 104/300 Read block 2678 len 12288
10764 105/300 Read block 3665 len 20480
10765 106/300 Read block 1065 len 4096
10766 107/300 Flush
10767 108/300 Read block 2575 len 24576
10768 109/300 Read block 2838 len 12288
10769 110/300 Write block 3252 len 28672 data: 3 2 2 2 2 2 2
10770 111/300 Read block 3239 len 36864
10771 112/300 Read block 1303 len 24576
10772 113/300 Read block 4581 len 40960
10773 114/300 Read block 3268 len 20480
10774 115/300 Write block 4600 len 32768 data: 2 2 2 2 2 2 2 2
10775 116/300 Read block 2734 len 24576
10776 117/300 Read block 1974 len 8192
10777 118/300 Flush
10778 119/300 Write block 1169 len 40960 data: 4 3 2 3 3 4 4 5 6 7
10779 120/300 Read block 2198 len 32768
10780 121/300 Read block 3813 len 40960
10781 122/300 Write block 3056 len 4096 data: 3
10782 123/300 Write block 2348 len 8192 data: 2 2
10783 124/300 Read block 2457 len 28672
10784 125/300 Read block 574 len 12288
10785 126/300 Write block 4807 len 8192 data: 2 3
10786 127/300 Write block 242 len 40960 data: 4 3 3 3 3 2 2 3 3 4
10787 128/300 Read block 980 len 12288
10788 129/300 Flush
10789 130/300 Write block 1899 len 20480 data: 3 3 4 3 3
10790 131/300 Read block 4401 len 40960
10791 132/300 Write block 315 len 36864 data: 2 3 3 3 4 4 4 4 4
10792 133/300 Write block 1017 len 40960 data: 3 3 4 4 4 3 2 2 3 3
10793 134/300 Write block 4043 len 20480 data: 3 3 3 3 2
10794 135/300 Read block 1667 len 32768
10795 136/300 Write block 3881 len 12288 data: 3 4 5
10796 137/300 Write block 2790 len 24576 data: 4 3 3 2 3 3
10797 138/300 Read block 2728 len 12288
10798 139/300 Read block 1435 len 12288
10799 140/300 Read block 4309 len 40960
10800 141/300 Write block 4085 len 28672 data: 2 2 2 2 2 2 2
10801 142/300 Read block 4588 len 36864
10802 143/300 Flush
10803 144/300 Write block 3435 len 40960 data: 4 5 5 5 3 2 3 3 3 3
10804 145/300 Read block 4978 len 8192
10805 146/300 Write block 4644 len 8192 data: 3 3
10806 147/300 Write block 4346 len 40960 data: 4 3 3 2 2 2 3 3 3 3
10807 148/300 Write block 4081 len 4096 data: 3
10808 149/300 Write block 2918 len 40960 data: 3 2 2 2 2 3 3 3 3 2
10809 150/300 Read block 302 len 36864
10810 151/300 Write block 4724 len 36864 data: 2 2 2 2 2 3 3 3 3
10811 152/300 Flush
10812 153/300 Read block 2821 len 40960
10813 154/300 Write block 1239 len 16384 data: 2 2 2 2
10814 155/300 Read block 2038 len 24576
10815 156/300 Read block 1102 len 28672
10816 157/300 Write block 3272 len 20480 data: 3 2 3 3 3
10817 158/300 Write block 1297 len 8192 data: 2 2
10818 159/300 Write block 1445 len 4096 data: 6
10819 160/300 Write block 2056 len 32768 data: 2 2 2 2 2 3 3 3
10820 161/300 Write block 364 len 24576 data: 3 3 3 2 2 2
10821 162/300 Write block 3826 len 28672 data: 2 2 2 2 2 2 2
10822 163/300 Read block 3934 len 24576
10823 164/300 Read block 3879 len 24576
10824 165/300 Write block 1437 len 24576 data: 2 2 2 5 6 6
10825 166/300 Read block 1002 len 24576
10826 167/300 Write block 4699 len 4096 data: 4
10827 168/300 Read block 4350 len 20480
10828 169/300 Write block 2200 len 12288 data: 3 3 4
10829 170/300 Read block 1867 len 16384
10830 171/300 Read block 3359 len 8192
10831 172/300 Read block 557 len 40960
10832 173/300 Read block 3203 len 12288
10833 174/300 Write block 1255 len 12288 data: 3 3 3
10834 175/300 Write block 908 len 32768 data: 3 2 3 4 4 4 3 3
10835 176/300 Read block 123 len 24576
10836 177/300 Read block 2183 len 12288
10837 178/300 Write block 4154 len 28672 data: 2 2 2 2 3 4 4
10838 179/300 Write block 75 len 28672 data: 4 4 4 4 4 3 4
10839 180/300 Flush
10840 181/300 Read block 4112 len 16384
10841 182/300 Write block 2298 len 4096 data: 2
10842 183/300 Write block 689 len 16384 data: 2 3 3 3
10843 184/300 Read block 1385 len 40960
10844 185/300 Write block 481 len 28672 data: 3 3 2 2 2 3 2
10845 186/300 Write block 3522 len 36864 data: 4 4 5 5 5 6 6 5 3
10846 187/300 Flush
10847 188/300 Write block 435 len 12288 data: 5 5 4
10848 189/300 Read block 424 len 12288
10849 190/300 Write block 3334 len 24576 data: 2 3 4 4 4 4
10850 191/300 Read block 2578 len 40960
10851 192/300 Write block 1018 len 40960 data: 4 5 5 5 4 3 3 4 4 4
10852 193/300 Read block 3109 len 32768
10853 194/300 Write block 2591 len 12288 data: 3 2 2
10854 195/300 Write block 3407 len 8192 data: 3 3
10855 196/300 Read block 3570 len 32768
10856 197/300 Read block 3859 len 12288
10857 198/300 Flush
10858 199/300 Write block 2131 len 20480 data: 3 3 3 3 2
10859 200/300 Read block 1467 len 4096
10860 201/300 Write block 1969 len 36864 data: 3 3 3 3 3 3 3 2 2
10861 202/300 Write block 350 len 28672 data: 4 3 3 3 2 2 2
10862 203/300 Read block 3647 len 24576
10863 204/300 Read block 4963 len 4096
10864 205/300 Read block 4838 len 8192
10865 206/300 Write block 4642 len 32768 data: 2 3 4 4 3 5 4 3
10866 207/300 Read block 3430 len 40960
10867 208/300 Write block 702 len 24576 data: 5 5 5 4 4 4
10868 209/300 Read block 2529 len 4096
10869 210/300 Flush
10870 211/300 Write block 2841 len 4096 data: 2
10871 212/300 Read block 4824 len 8192
10872 213/300 Read block 90 len 4096
10873 214/300 Write block 4520 len 32768 data: 4 5 4 4 5 4 4 4
10874 215/300 Read block 525 len 20480
10875 216/300 Write block 2176 len 16384 data: 2 2 3 5
10876 217/300 Write block 4524 len 32768 data: 6 5 5 5 5 5 5 3
10877 218/300 Read block 2244 len 4096
10878 219/300 Read block 4498 len 36864
10879 220/300 Flush
10880 221/300 Read block 255 len 20480
10881 222/300 Read block 3575 len 28672
10882 223/300 Read block 4043 len 4096
10883 224/300 Write block 3903 len 12288 data: 4 4 4
10884 225/300 Write block 2570 len 20480 data: 5 4 4 4 2
10885 226/300 Write block 1326 len 24576 data: 4 4 4 5 4 4
10886 227/300 Read block 4236 len 28672
10887 228/300 Read block 2722 len 32768
10888 229/300 Write block 404 len 20480 data: 3 3 3 3 3
10889 230/300 Write block 3756 len 32768 data: 2 2 2 2 3 3 3 2
10890 231/300 Read block 2700 len 24576
10891 232/300 Read block 2986 len 28672
10892 233/300 Read block 1521 len 40960
10893 234/300 Write block 4111 len 36864 data: 2 3 3 3 3 4 4 3 3
10894 235/300 Write block 557 len 36864 data: 2 2 2 2 2 2 2 2 2
10895 236/300 Read block 3050 len 16384
10896 237/300 Write block 3370 len 8192 data: 2 2
10897 238/300 Read block 4198 len 12288
10898 239/300 Read block 4894 len 36864
10899 240/300 Read block 2180 len 12288
10900 241/300 Read block 4691 len 4096
10901 242/300 Write block 1296 len 40960 data: 2 3 3 2 2 2 2 2 2 3
10902 243/300 Read block 162 len 8192
10903 244/300 Write block 1608 len 28672 data: 2 2 2 2 2 4 4
10904 245/300 Read block 384 len 12288
10905 246/300 Read block 3933 len 4096
10906 247/300 Write block 792 len 20480 data: 3 3 3 4 5
10907 248/300 Write block 3887 len 20480 data: 3 3 2 2 2
10908 249/300 Write block 4506 len 40960 data: 4 3 2 2 3 3 2 2 3 3
10909 250/300 Read block 1101 len 12288
10910 251/300 Flush
10911 252/300 Read block 4940 len 4096
10912 253/300 Read block 2239 len 32768
10913 254/300 Write block 3114 len 24576 data: 6 5 5 5 4 4
10914 255/300 Read block 4155 len 8192
10915 256/300 Write block 3733 len 12288 data: 3 4 4
10916 257/300 Write block 2638 len 24576 data: 2 2 2 2 2 2
10917 258/300 Write block 3286 len 20480 data: 2 2 2 2 2
10918 259/300 Read block 2405 len 8192
10919 260/300 Write block 685 len 8192 data: 3 2
10920 261/300 Write block 3556 len 16384 data: 2 2 2 2
10921 262/300 Read block 547 len 32768
10922 263/300 Write block 3816 len 16384 data: 2 2 2 2
10923 264/300 Flush
10924 265/300 Read block 3021 len 24576
10925 266/300 Write block 2809 len 36864 data: 2 2 3 3 4 4 4 4 4
10926 267/300 Write block 410 len 40960 data: 3 2 2 3 4 4 5 5 5 6
10927 268/300 Write block 4736 len 28672 data: 3 3 3 3 3 3 3
10928 269/300 Write block 574 len 40960 data: 3 3 3 4 4 4 5 4 4 4
10929 270/300 Flush
10930 271/300 Read block 1666 len 20480
10931 272/300 Write block 1170 len 8192 data: 4 3
10932 273/300 Write block 3645 len 8192 data: 2 2
10933 274/300 Write block 3293 len 16384 data: 4 4 4 5
10934 275/300 Read block 1161 len 12288
10935 276/300 Read block 2916 len 24576
10936 277/300 Write block 10 len 4096 data: 3
10937 278/300 Flush
10938 279/300 Read block 1255 len 16384
10939 280/300 Write block 1098 len 8192 data: 2 2
10940 281/300 Read block 179 len 4096
10941 282/300 Flush
10942 283/300 Read block 4368 len 24576
10943 284/300 Flush
10944 285/300 Read block 2495 len 36864
10945 286/300 Write block 4667 len 20480 data: 3 3 2 2 2
10946 287/300 Write block 2146 len 32768 data: 3 3 3 3 3 3 4 3
10947 288/300 Flush
10948 289/300 Read block 2888 len 16384
10949 290/300 Read block 3214 len 24576
10950 291/300 Write block 3861 len 36864 data: 3 3 3 4 3 3 3 3 4
10951 292/300 Write block 4867 len 28672 data: 2 2 2 2 2 2 2
10952 293/300 Write block 674 len 12288 data: 4 4 3
10953 294/300 Read block 4522 len 4096
10954 295/300 Write block 2432 len 4096 data: 2
10955 296/300 Read block 3728 len 20480
10956 297/300 Read block 3012 len 20480
10957 298/300 Read block 532 len 4096
10958 299/300 Read block 4339 len 36864
10959 300/300 Read block 237 len 12288
10960 301/300 Read block 4238 len 32768
10961 Main task has work to do, go find it
10962 got dsc Start(2)
10963 start 2
10964 Feb 28 06:03:26.124 INFO request completed, latency_us: 176, response_code: 204, uri: /start/cid/2, method: POST, req_id: e488d2cf-7d79-41ef-9ae3-6f4eefadae31, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
10965 Replay: started 2, returned:Ok(())
10966 ----------------------------------------------------------------
10967 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10968 States: Active Active Offline
10969 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10970 3413 Acked Write 4 Done Done Sent false
10971 3414 Acked Write 2 Done Done Sent false
10972 3415 Acked Flush 0 Done Done Sent false
10973 3416 Acked Write 5 Done Done Sent false
10974 3417 Acked Read 2 Done Done Sent false
10975 3418 Acked Read 4 Done Done Sent false
10976 3419 Acked Read 8 Done Done Sent false
10977 3420 Acked Write 9 Done Done Sent false
10978 3421 Acked Read 9 Done Done Sent false
10979 3422 Acked Read 6 Done Done Sent false
10980 3423 Acked Flush 0 Done Done Sent false
10981 3424 Acked Write 6 Done Done Sent false
10982 3425 Acked Read 9 Done Done Sent false
10983 3426 Acked Read 3 Done Done Sent false
10984 3427 Acked Write 10 Done Done Sent false
10985 3428 Acked Read 6 Done Done Sent false
10986 3429 Acked Read 5 Done Done Sent false
10987 3430 Acked Write 1 Done Done Sent false
10988 3431 Acked Write 10 Done Done Sent false
10989 3432 Acked Read 2 Done Done Sent false
10990 3433 Acked Read 1 Done Done Sent false
10991 3434 Acked Read 4 Done Done Sent false
10992 3435 Acked Read 5 Done Done Sent false
10993 3436 Acked Flush 0 Done Done Sent false
10994 3437 Acked Write 10 Done Done Sent false
10995 3438 Acked Write 4 Done Done Sent false
10996 3439 Acked Write 10 Done Done Sent false
10997 3440 Acked Read 4 Done Done Sent false
10998 3441 Acked Read 9 Done Done Sent false
10999 3442 Acked Write 6 Done Done Sent false
11000 3443 Acked Read 9 Done Done Sent false
11001 3444 Acked Write 2 Done Done Sent false
11002 3445 Acked Read 6 Done Done Sent false
11003 3446 Acked Read 2 Done Done Sent false
11004 3447 Acked Read 7 Done Done Sent false
11005 3448 Acked Write 3 Done Done Sent false
11006 3449 Acked Write 9 Done Done Sent false
11007 3450 Acked Write 7 Done Done Sent false
11008 3451 Acked Write 8 Done Done Sent false
11009 3452 Acked Read 4 Done Done Sent false
11010 3453 Acked Read 2 Done Done Sent false
11011 3454 Acked Read 2 Done Done Sent false
11012 3455 Acked Write 10 Done Done Sent false
11013 3456 Acked Read 6 Done Done Sent false
11014 3457 Acked Write 9 Done Done Sent false
11015 3458 Acked Read 7 Done Done Sent false
11016 3459 Acked Read 10 Done Done Sent false
11017 3460 Acked Read 1 Done Done Sent false
11018 3461 Acked Write 5 Done Done Sent false
11019 3462 Acked Read 5 Done Done Sent false
11020 3463 Acked Write 6 Done Done Sent false
11021 3464 Acked Write 6 Done Done Sent false
11022 3465 Acked Read 2 Done Done Sent false
11023 3466 Acked Read 8 Done Done Sent false
11024 3467 Acked Read 2 Done Done Sent false
11025 3468 Acked Flush 0 Done Done Sent false
11026 3469 Acked Write 8 Done Done Sent false
11027 3470 Acked Write 9 Done Done Sent false
11028 3471 Acked Flush 0 Done Done Sent false
11029 3472 Acked Flush 0 Done Done Sent false
11030 3473 Acked Write 6 Done Done Sent false
11031 3474 Acked Write 9 Done Done Sent false
11032 3475 Acked Read 9 Done Done Sent false
11033 3476 Acked Read 1 Done Done Sent false
11034 3477 Acked Read 4 Done Done Sent false
11035 3478 Acked Read 3 Done Done Sent false
11036 3479 Acked Read 10 Done Done Sent false
11037 3480 Acked Read 6 Done Done Sent false
11038 3481 Acked Read 5 Done Done Sent false
11039 3482 Acked Write 9 Done Done Sent false
11040 3483 Acked Read 6 Done Done Sent false
11041 3484 Acked Read 8 Done Done Sent false
11042 3485 Acked Read 2 Done Done Sent false
11043 3486 Acked Write 10 Done Done Sent false
11044 3487 Acked Read 6 Done Done Sent false
11045 3488 Acked Flush 0 Done Done Sent false
11046 3489 Acked Read 1 Done Done Sent false
11047 3490 Acked Write 6 Done Done Sent false
11048 3491 Acked Write 3 Done Done Sent false
11049 3492 Acked Flush 0 Done Done Sent false
11050 3493 Acked Flush 0 Done Done Sent false
11051 3494 Acked Read 4 Done Done Sent false
11052 3495 Acked Write 6 Done Done Sent false
11053 3496 Acked Read 7 Done Done Sent false
11054 3497 Acked Read 5 Done Done Sent false
11055 3498 Acked Write 4 Done Done Sent false
11056 3499 Acked Write 10 Done Done Sent false
11057 3500 Acked Read 7 Done Done Sent false
11058 3501 Acked Write 7 Done Done Sent false
11059 3502 Acked Write 10 Done Done Sent false
11060 3503 Acked Read 4 Done Done Sent false
11061 3504 Acked Read 3 Done Done Sent false
11062 3505 Acked Write 2 Done Done Sent false
11063 3506 Acked Read 1 Done Done Sent false
11064 3507 Acked Read 9 Done Done Sent false
11065 3508 Acked Read 3 Done Done Sent false
11066 3509 Acked Write 10 Done Done Sent false
11067 3510 Acked Write 1 Done Done Sent false
11068 3511 Acked Read 1 Done Done Sent false
11069 3512 Acked Flush 0 Done Done Sent false
11070 3513 Acked Read 1 Done Done Sent false
11071 3514 Acked Write 9 Done Done Sent false
11072 3515 Acked Read 8 Done Done Sent false
11073 3516 Acked Read 3 Done Done Sent false
11074 3517 Acked Read 5 Done Done Sent false
11075 3518 Acked Read 1 Done Done Sent false
11076 3519 Acked Flush 0 Done Done Sent false
11077 3520 Acked Read 6 Done Done Sent false
11078 3521 Acked Read 3 Done Done Sent false
11079 3522 Acked Write 7 Done Done Sent false
11080 3523 Acked Read 9 Done Done Sent false
11081 3524 Acked Read 6 Done Done Sent false
11082 3525 Acked Read 10 Done Done Sent false
11083 3526 Acked Read 5 Done Done Sent false
11084 3527 Acked Write 8 Done Done Sent false
11085 3528 Acked Read 6 Done Done Sent false
11086 3529 Acked Read 2 Done Done Sent false
11087 3530 Acked Flush 0 Done Done Sent false
11088 3531 Acked Write 10 Done Done Sent false
11089 3532 Acked Read 8 Done Done Sent false
11090 3533 Acked Read 10 Done Done Sent false
11091 3534 Acked Write 1 Done Done Sent false
11092 3535 Acked Write 2 Done Done Sent false
11093 3536 Acked Read 7 Done Done Sent false
11094 3537 Acked Read 3 Done Done Sent false
11095 3538 Acked Write 2 Done Done Sent false
11096 3539 Acked Write 10 Done Done Sent false
11097 3540 Acked Read 3 Done Done Sent false
11098 3541 Acked Flush 0 Done Done Sent false
11099 3542 Acked Write 5 Done Done Sent false
11100 3543 Acked Read 10 Done Done Sent false
11101 3544 Acked Write 9 Done Done Sent false
11102 3545 Acked Write 10 Done Done Sent false
11103 3546 Acked Write 5 Done Done Sent false
11104 3547 Acked Read 8 Done Done Sent false
11105 3548 Acked Write 3 Done Done Sent false
11106 3549 Acked Write 6 Done Done Sent false
11107 3550 Acked Read 3 Done Done Sent false
11108 3551 Acked Read 3 Done Done Sent false
11109 3552 Acked Read 10 Done Done Sent false
11110 3553 Acked Write 7 Done Done Sent false
11111 3554 Acked Read 9 Done Done Sent false
11112 3555 Acked Flush 0 Done Done Sent false
11113 3556 Acked Write 10 Done Done Sent false
11114 3557 Acked Read 2 Done Done Sent false
11115 3558 Acked Write 2 Done Done Sent false
11116 3559 Acked Write 10 Done Done Sent false
11117 3560 Acked Write 1 Done Done Sent false
11118 3561 Acked Write 10 Done Done Sent false
11119 3562 Acked Read 9 Done Done Sent false
11120 3563 Acked Write 9 Done Done Sent false
11121 3564 Acked Flush 0 Done Done Sent false
11122 3565 Acked Read 10 Done Done Sent false
11123 3566 Acked Write 4 Done Done Sent false
11124 3567 Acked Read 6 Done Done Sent false
11125 3568 Acked Read 7 Done Done Sent false
11126 3569 Acked Write 5 Done Done Sent false
11127 3570 Acked Write 2 Done Done Sent false
11128 3571 Acked Write 1 Done Done Sent false
11129 3572 Acked Write 8 Done Done Sent false
11130 3573 Acked Write 6 Done Done Sent false
11131 3574 Acked Write 7 Done Done Sent false
11132 3575 Acked Read 6 Done Done Sent false
11133 3576 Acked Read 6 Done Done Sent false
11134 3577 Acked Write 6 Done Done Sent false
11135 3578 Acked Read 6 Done Done Sent false
11136 3579 Acked Write 1 Done Done Sent false
11137 3580 Acked Read 5 Done Done Sent false
11138 3581 Acked Write 3 Done Done Sent false
11139 3582 Acked Read 4 Done Done Sent false
11140 3583 Acked Read 2 Done Done Sent false
11141 3584 Acked Read 10 Done Done Sent false
11142 3585 Acked Read 3 Done Done Sent false
11143 3586 Acked Write 3 Done Done Sent false
11144 3587 Acked Write 8 Done Done Sent false
11145 3588 Acked Read 6 Done Done Sent false
11146 3589 Acked Read 3 Done Done Sent false
11147 3590 Acked Write 7 Done Done Sent false
11148 3591 Acked Write 7 Done Done Sent false
11149 3592 Acked Flush 0 Done Done Sent false
11150 3593 Acked Read 4 Done Done Sent false
11151 3594 Acked Write 1 Done Done Sent false
11152 3595 Acked Write 4 Done Done Sent false
11153 3596 Acked Read 10 Done Done Sent false
11154 3597 Acked Write 7 Done Done Sent false
11155 3598 Acked Write 9 Done Done Sent false
11156 3599 Acked Flush 0 Done Done Sent false
11157 3600 Acked Write 3 Done Done Sent false
11158 3601 Acked Read 3 Done Done Sent false
11159 3602 Acked Write 6 Done Done Sent false
11160 3603 Acked Read 10 Done Done Sent false
11161 3604 Acked Write 10 Done Done Sent false
11162 3605 Acked Read 8 Done Done Sent false
11163 3606 Acked Write 3 Done Done Sent false
11164 3607 Acked Write 2 Done Done Sent false
11165 3608 Acked Read 8 Done Done Sent false
11166 3609 Acked Read 3 Done Done Sent false
11167 3610 Acked Flush 0 Done Done Sent false
11168 3611 Acked Write 5 Done Done Sent false
11169 3612 Acked Read 1 Done Done Sent false
11170 3613 Acked Write 9 Done Done Sent false
11171 3614 Acked Write 7 Done Done Sent false
11172 3615 Acked Read 6 Done Done Sent false
11173 3616 Acked Read 1 Done Done Sent false
11174 3617 Acked Read 2 Done Done Sent false
11175 3618 Acked Write 8 Done Done Sent false
11176 3619 Acked Read 10 Done Done Sent false
11177 3620 Acked Write 6 Done Done Sent false
11178 3621 Acked Read 1 Done Done Sent false
11179 3622 Acked Flush 0 Done Done Sent false
11180 3623 Acked Write 1 Done Done Sent false
11181 3624 Acked Read 2 Done Done Sent false
11182 3625 Acked Read 1 Done Done Sent false
11183 3626 Acked Write 8 Done Done Sent false
11184 3627 Acked Read 5 Done Done Sent false
11185 3628 Acked Write 4 Done Done Sent false
11186 3629 Acked Write 8 Done Done Sent false
11187 3630 Acked Read 1 Done Done Sent false
11188 3631 Acked Read 9 Done Done Sent false
11189 3632 Acked Flush 0 Done Done Sent false
11190 3633 Acked Read 5 Done Done Sent false
11191 3634 Acked Read 7 Done Done Sent false
11192 3635 Acked Read 1 Done Done Sent false
11193 3636 Acked Write 3 Done Done Sent false
11194 3637 Acked Write 5 Done Done Sent false
11195 3638 Acked Write 6 Done Done Sent false
11196 3639 Acked Read 7 Done Done Sent false
11197 3640 Acked Read 8 Done Done Sent false
11198 3641 Acked Write 5 Done Done Sent false
11199 3642 Acked Write 8 Done Done Sent false
11200 3643 Acked Read 6 Done Done Sent false
11201 3644 Acked Read 7 Done Done Sent false
11202 3645 Acked Read 10 Done Done Sent false
11203 3646 Acked Write 9 Done Done Sent false
11204 3647 Acked Write 9 Done Done Sent false
11205 3648 Acked Read 4 Done Done Sent false
11206 3649 Acked Write 2 Done Done Sent false
11207 3650 Acked Read 3 Done Done Sent false
11208 3651 Acked Read 9 Done Done Sent false
11209 3652 Acked Read 3 Done Done Sent false
11210 3653 Acked Read 1 Done Done Sent false
11211 3654 Acked Write 10 Done Done Sent false
11212 3655 Acked Read 2 Done Done Sent false
11213 3656 Acked Write 7 Done Done Sent false
11214 3657 Acked Read 3 Done Done Sent false
11215 3658 Acked Read 1 Done Done Sent false
11216 3659 Acked Write 5 Done Done Sent false
11217 3660 Acked Write 5 Done Done Sent false
11218 3661 Acked Write 10 Done Done Sent false
11219 3662 Acked Read 3 Done Done Sent false
11220 3663 Acked Flush 0 Done Done Sent false
11221 3664 Acked Read 1 Done Done Sent false
11222 3665 Acked Read 8 Done Done Sent false
11223 3666 Acked Write 6 Done Done Sent false
11224 3667 Acked Read 2 Done Done Sent false
11225 3668 Acked Write 3 Done Done Sent false
11226 3669 Acked Write 6 Done Done Sent false
11227 3670 Acked Write 5 Done Done Sent false
11228 3671 Acked Read 2 Done Done Sent false
11229 3672 Acked Write 2 Done Done Sent false
11230 3673 Acked Write 4 Done Done Sent false
11231 3674 Acked Read 8 Done Done Sent false
11232 3675 Acked Write 4 Done Done Sent false
11233 3676 Acked Flush 0 Done Done Sent false
11234 3677 Acked Read 6 Done Done Sent false
11235 3678 Acked Write 9 Done Done Sent false
11236 3679 Acked Write 10 Done Done Sent false
11237 3680 Acked Write 7 Done Done Sent false
11238 3681 Acked Write 10 Done Done Sent false
11239 3682 Acked Flush 0 Done Done Sent false
11240 3683 Acked Read 5 Done Done Sent false
11241 3684 Acked Write 2 Done Done Sent false
11242 3685 Acked Write 2 Done Done Sent false
11243 3686 Acked Write 4 Done Done Sent false
11244 3687 Acked Read 3 Done Done Sent false
11245 3688 Acked Read 6 Done Done Sent false
11246 3689 Acked Write 1 Done Done Sent false
11247 3690 Acked Flush 0 Done Done Sent false
11248 3691 Acked Read 4 Done Done Sent false
11249 3692 Acked Write 2 Done Done Sent false
11250 3693 Acked Read 1 Done Done Sent false
11251 3694 Acked Flush 0 Done Done Sent false
11252 3695 Acked Read 6 Done Done Sent false
11253 3696 Acked Flush 0 Done Done Sent false
11254 3697 Acked Read 9 Done Done Sent false
11255 3698 Acked Write 5 Done Done Sent false
11256 3699 Acked Write 8 Done Done Sent false
11257 3700 Acked Flush 0 Done Done Sent false
11258 3701 Acked Read 4 Done Done Sent false
11259 3702 Acked Read 6 Done Done Sent false
11260 3703 Acked Write 9 Done Done Sent false
11261 3704 Acked Write 7 Done Done Sent false
11262 3705 Acked Write 3 Done Done Sent false
11263 3706 Acked Read 1 Done Done Sent false
11264 3707 Acked Write 1 Done Done Sent false
11265 3708 Acked Read 5 Done Done Sent false
11266 3709 Acked Read 5 Done Done Sent false
11267 3710 Acked Read 1 Done Done Sent false
11268 3711 Acked Read 9 Done Done Sent false
11269 3712 Acked Read 3 Done Done Sent false
11270 3713 Acked Read 8 Done Done Sent false
11271 STATES DS:0 DS:1 DS:2 TOTAL
11272 Sent 0 0 301 301
11273 Done 301 301 0 602
11274 Skipped 0 0 0 0
11275 Error 0 0 0 0
11276 Last Flush: Some(JobId(3700)) Some(JobId(3700)) Some(JobId(3412))
11277 Downstairs last five completed: 3412 3411 3410 3409 3408
11278 Upstairs last five completed: 3713 3712 3711 3710 3709
11279 CLIENT: Up:0 ds:301 act:2
112802025-02-28T06:03:26.626ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
112812025-02-28T06:03:27.887ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
112822025-02-28T06:03:28.581ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
112832025-02-28T06:03:29.583ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
11284 [8830] got start action
11285 [8830] I am going to restart
11286 Starting downstairs at port 8830
11287 Make output file at "/tmp/dsc/downstairs-8830.txt"
11288 [8830][2] reports Starting
11289 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1220)
11290 [8830][2] reports Running
11291 ----------------------------------------------------------------
11292 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11293 States: Active Active Offline
11294 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11295 3413 Acked Write 4 Done Done Sent false
11296 3414 Acked Write 2 Done Done Sent false
11297 3415 Acked Flush 0 Done Done Sent false
11298 3416 Acked Write 5 Done Done Sent false
11299 3417 Acked Read 2 Done Done Sent false
11300 3418 Acked Read 4 Done Done Sent false
11301 3419 Acked Read 8 Done Done Sent false
11302 3420 Acked Write 9 Done Done Sent false
11303 3421 Acked Read 9 Done Done Sent false
11304 3422 Acked Read 6 Done Done Sent false
11305 3423 Acked Flush 0 Done Done Sent false
11306 3424 Acked Write 6 Done Done Sent false
11307 3425 Acked Read 9 Done Done Sent false
11308 3426 Acked Read 3 Done Done Sent false
11309 3427 Acked Write 10 Done Done Sent false
11310 3428 Acked Read 6 Done Done Sent false
11311 3429 Acked Read 5 Done Done Sent false
11312 3430 Acked Write 1 Done Done Sent false
11313 3431 Acked Write 10 Done Done Sent false
11314 3432 Acked Read 2 Done Done Sent false
11315 3433 Acked Read 1 Done Done Sent false
11316 3434 Acked Read 4 Done Done Sent false
11317 3435 Acked Read 5 Done Done Sent false
11318 3436 Acked Flush 0 Done Done Sent false
11319 3437 Acked Write 10 Done Done Sent false
11320 3438 Acked Write 4 Done Done Sent false
11321 3439 Acked Write 10 Done Done Sent false
11322 3440 Acked Read 4 Done Done Sent false
11323 3441 Acked Read 9 Done Done Sent false
11324 3442 Acked Write 6 Done Done Sent false
11325 3443 Acked Read 9 Done Done Sent false
11326 3444 Acked Write 2 Done Done Sent false
11327 3445 Acked Read 6 Done Done Sent false
11328 3446 Acked Read 2 Done Done Sent false
11329 3447 Acked Read 7 Done Done Sent false
11330 3448 Acked Write 3 Done Done Sent false
11331 3449 Acked Write 9 Done Done Sent false
11332 3450 Acked Write 7 Done Done Sent false
11333 3451 Acked Write 8 Done Done Sent false
11334 3452 Acked Read 4 Done Done Sent false
11335 3453 Acked Read 2 Done Done Sent false
11336 3454 Acked Read 2 Done Done Sent false
11337 3455 Acked Write 10 Done Done Sent false
11338 3456 Acked Read 6 Done Done Sent false
11339 3457 Acked Write 9 Done Done Sent false
11340 3458 Acked Read 7 Done Done Sent false
11341 3459 Acked Read 10 Done Done Sent false
11342 3460 Acked Read 1 Done Done Sent false
11343 3461 Acked Write 5 Done Done Sent false
11344 3462 Acked Read 5 Done Done Sent false
11345 3463 Acked Write 6 Done Done Sent false
11346 3464 Acked Write 6 Done Done Sent false
11347 3465 Acked Read 2 Done Done Sent false
11348 3466 Acked Read 8 Done Done Sent false
11349 3467 Acked Read 2 Done Done Sent false
11350 3468 Acked Flush 0 Done Done Sent false
11351 3469 Acked Write 8 Done Done Sent false
11352 3470 Acked Write 9 Done Done Sent false
11353 3471 Acked Flush 0 Done Done Sent false
11354 3472 Acked Flush 0 Done Done Sent false
11355 3473 Acked Write 6 Done Done Sent false
11356 3474 Acked Write 9 Done Done Sent false
11357 3475 Acked Read 9 Done Done Sent false
11358 3476 Acked Read 1 Done Done Sent false
11359 3477 Acked Read 4 Done Done Sent false
11360 3478 Acked Read 3 Done Done Sent false
11361 3479 Acked Read 10 Done Done Sent false
11362 3480 Acked Read 6 Done Done Sent false
11363 3481 Acked Read 5 Done Done Sent false
11364 3482 Acked Write 9 Done Done Sent false
11365 3483 Acked Read 6 Done Done Sent false
11366 3484 Acked Read 8 Done Done Sent false
11367 3485 Acked Read 2 Done Done Sent false
11368 3486 Acked Write 10 Done Done Sent false
11369 3487 Acked Read 6 Done Done Sent false
11370 3488 Acked Flush 0 Done Done Sent false
11371 3489 Acked Read 1 Done Done Sent false
11372 3490 Acked Write 6 Done Done Sent false
11373 3491 Acked Write 3 Done Done Sent false
11374 3492 Acked Flush 0 Done Done Sent false
11375 3493 Acked Flush 0 Done Done Sent false
11376 3494 Acked Read 4 Done Done Sent false
11377 3495 Acked Write 6 Done Done Sent false
11378 3496 Acked Read 7 Done Done Sent false
11379 3497 Acked Read 5 Done Done Sent false
11380 3498 Acked Write 4 Done Done Sent false
11381 3499 Acked Write 10 Done Done Sent false
11382 3500 Acked Read 7 Done Done Sent false
11383 3501 Acked Write 7 Done Done Sent false
11384 3502 Acked Write 10 Done Done Sent false
11385 3503 Acked Read 4 Done Done Sent false
11386 3504 Acked Read 3 Done Done Sent false
11387 3505 Acked Write 2 Done Done Sent false
11388 3506 Acked Read 1 Done Done Sent false
11389 3507 Acked Read 9 Done Done Sent false
11390 3508 Acked Read 3 Done Done Sent false
11391 3509 Acked Write 10 Done Done Sent false
11392 3510 Acked Write 1 Done Done Sent false
11393 3511 Acked Read 1 Done Done Sent false
11394 3512 Acked Flush 0 Done Done Sent false
11395 3513 Acked Read 1 Done Done Sent false
11396 3514 Acked Write 9 Done Done Sent false
11397 3515 Acked Read 8 Done Done Sent false
11398 3516 Acked Read 3 Done Done Sent false
11399 3517 Acked Read 5 Done Done Sent false
11400 3518 Acked Read 1 Done Done Sent false
11401 3519 Acked Flush 0 Done Done Sent false
11402 3520 Acked Read 6 Done Done Sent false
11403 3521 Acked Read 3 Done Done Sent false
11404 3522 Acked Write 7 Done Done Sent false
11405 3523 Acked Read 9 Done Done Sent false
11406 3524 Acked Read 6 Done Done Sent false
11407 3525 Acked Read 10 Done Done Sent false
11408 3526 Acked Read 5 Done Done Sent false
11409 3527 Acked Write 8 Done Done Sent false
11410 3528 Acked Read 6 Done Done Sent false
11411 3529 Acked Read 2 Done Done Sent false
11412 3530 Acked Flush 0 Done Done Sent false
11413 3531 Acked Write 10 Done Done Sent false
11414 3532 Acked Read 8 Done Done Sent false
11415 3533 Acked Read 10 Done Done Sent false
11416 3534 Acked Write 1 Done Done Sent false
11417 3535 Acked Write 2 Done Done Sent false
11418 3536 Acked Read 7 Done Done Sent false
11419 3537 Acked Read 3 Done Done Sent false
11420 3538 Acked Write 2 Done Done Sent false
11421 3539 Acked Write 10 Done Done Sent false
11422 3540 Acked Read 3 Done Done Sent false
11423 3541 Acked Flush 0 Done Done Sent false
11424 3542 Acked Write 5 Done Done Sent false
11425 3543 Acked Read 10 Done Done Sent false
11426 3544 Acked Write 9 Done Done Sent false
11427 3545 Acked Write 10 Done Done Sent false
11428 3546 Acked Write 5 Done Done Sent false
11429 3547 Acked Read 8 Done Done Sent false
11430 3548 Acked Write 3 Done Done Sent false
11431 3549 Acked Write 6 Done Done Sent false
11432 3550 Acked Read 3 Done Done Sent false
11433 3551 Acked Read 3 Done Done Sent false
11434 3552 Acked Read 10 Done Done Sent false
11435 3553 Acked Write 7 Done Done Sent false
11436 3554 Acked Read 9 Done Done Sent false
11437 3555 Acked Flush 0 Done Done Sent false
11438 3556 Acked Write 10 Done Done Sent false
11439 3557 Acked Read 2 Done Done Sent false
11440 3558 Acked Write 2 Done Done Sent false
11441 3559 Acked Write 10 Done Done Sent false
11442 3560 Acked Write 1 Done Done Sent false
11443 3561 Acked Write 10 Done Done Sent false
11444 3562 Acked Read 9 Done Done Sent false
11445 3563 Acked Write 9 Done Done Sent false
11446 3564 Acked Flush 0 Done Done Sent false
11447 3565 Acked Read 10 Done Done Sent false
11448 3566 Acked Write 4 Done Done Sent false
11449 3567 Acked Read 6 Done Done Sent false
11450 3568 Acked Read 7 Done Done Sent false
11451 3569 Acked Write 5 Done Done Sent false
11452 3570 Acked Write 2 Done Done Sent false
11453 3571 Acked Write 1 Done Done Sent false
11454 3572 Acked Write 8 Done Done Sent false
11455 3573 Acked Write 6 Done Done Sent false
11456 3574 Acked Write 7 Done Done Sent false
11457 3575 Acked Read 6 Done Done Sent false
11458 3576 Acked Read 6 Done Done Sent false
11459 3577 Acked Write 6 Done Done Sent false
11460 3578 Acked Read 6 Done Done Sent false
11461 3579 Acked Write 1 Done Done Sent false
11462 3580 Acked Read 5 Done Done Sent false
11463 3581 Acked Write 3 Done Done Sent false
11464 3582 Acked Read 4 Done Done Sent false
11465 3583 Acked Read 2 Done Done Sent false
11466 3584 Acked Read 10 Done Done Sent false
11467 3585 Acked Read 3 Done Done Sent false
11468 3586 Acked Write 3 Done Done Sent false
11469 3587 Acked Write 8 Done Done Sent false
11470 3588 Acked Read 6 Done Done Sent false
11471 3589 Acked Read 3 Done Done Sent false
11472 3590 Acked Write 7 Done Done Sent false
11473 3591 Acked Write 7 Done Done Sent false
11474 3592 Acked Flush 0 Done Done Sent false
11475 3593 Acked Read 4 Done Done Sent false
11476 3594 Acked Write 1 Done Done Sent false
11477 3595 Acked Write 4 Done Done Sent false
11478 3596 Acked Read 10 Done Done Sent false
11479 3597 Acked Write 7 Done Done Sent false
11480 3598 Acked Write 9 Done Done Sent false
11481 3599 Acked Flush 0 Done Done Sent false
11482 3600 Acked Write 3 Done Done Sent false
11483 3601 Acked Read 3 Done Done Sent false
11484 3602 Acked Write 6 Done Done Sent false
11485 3603 Acked Read 10 Done Done Sent false
11486 3604 Acked Write 10 Done Done Sent false
11487 3605 Acked Read 8 Done Done Sent false
11488 3606 Acked Write 3 Done Done Sent false
11489 3607 Acked Write 2 Done Done Sent false
11490 3608 Acked Read 8 Done Done Sent false
11491 3609 Acked Read 3 Done Done Sent false
11492 3610 Acked Flush 0 Done Done Sent false
11493 3611 Acked Write 5 Done Done Sent false
11494 3612 Acked Read 1 Done Done Sent false
11495 3613 Acked Write 9 Done Done Sent false
11496 3614 Acked Write 7 Done Done Sent false
11497 3615 Acked Read 6 Done Done Sent false
11498 3616 Acked Read 1 Done Done Sent false
11499 3617 Acked Read 2 Done Done Sent false
11500 3618 Acked Write 8 Done Done Sent false
11501 3619 Acked Read 10 Done Done Sent false
11502 3620 Acked Write 6 Done Done Sent false
11503 3621 Acked Read 1 Done Done Sent false
11504 3622 Acked Flush 0 Done Done Sent false
11505 3623 Acked Write 1 Done Done Sent false
11506 3624 Acked Read 2 Done Done Sent false
11507 3625 Acked Read 1 Done Done Sent false
11508 3626 Acked Write 8 Done Done Sent false
11509 3627 Acked Read 5 Done Done Sent false
11510 3628 Acked Write 4 Done Done Sent false
11511 3629 Acked Write 8 Done Done Sent false
11512 3630 Acked Read 1 Done Done Sent false
11513 3631 Acked Read 9 Done Done Sent false
11514 3632 Acked Flush 0 Done Done Sent false
11515 3633 Acked Read 5 Done Done Sent false
11516 3634 Acked Read 7 Done Done Sent false
11517 3635 Acked Read 1 Done Done Sent false
11518 3636 Acked Write 3 Done Done Sent false
11519 3637 Acked Write 5 Done Done Sent false
11520 3638 Acked Write 6 Done Done Sent false
11521 3639 Acked Read 7 Done Done Sent false
11522 3640 Acked Read 8 Done Done Sent false
11523 3641 Acked Write 5 Done Done Sent false
11524 3642 Acked Write 8 Done Done Sent false
11525 3643 Acked Read 6 Done Done Sent false
11526 3644 Acked Read 7 Done Done Sent false
11527 3645 Acked Read 10 Done Done Sent false
11528 3646 Acked Write 9 Done Done Sent false
11529 3647 Acked Write 9 Done Done Sent false
11530 3648 Acked Read 4 Done Done Sent false
11531 3649 Acked Write 2 Done Done Sent false
11532 3650 Acked Read 3 Done Done Sent false
11533 3651 Acked Read 9 Done Done Sent false
11534 3652 Acked Read 3 Done Done Sent false
11535 3653 Acked Read 1 Done Done Sent false
11536 3654 Acked Write 10 Done Done Sent false
11537 3655 Acked Read 2 Done Done Sent false
11538 3656 Acked Write 7 Done Done Sent false
11539 3657 Acked Read 3 Done Done Sent false
11540 3658 Acked Read 1 Done Done Sent false
11541 3659 Acked Write 5 Done Done Sent false
11542 3660 Acked Write 5 Done Done Sent false
11543 3661 Acked Write 10 Done Done Sent false
11544 3662 Acked Read 3 Done Done Sent false
11545 3663 Acked Flush 0 Done Done Sent false
11546 3664 Acked Read 1 Done Done Sent false
11547 3665 Acked Read 8 Done Done Sent false
11548 3666 Acked Write 6 Done Done Sent false
11549 3667 Acked Read 2 Done Done Sent false
11550 3668 Acked Write 3 Done Done Sent false
11551 3669 Acked Write 6 Done Done Sent false
11552 3670 Acked Write 5 Done Done Sent false
11553 3671 Acked Read 2 Done Done Sent false
11554 3672 Acked Write 2 Done Done Sent false
11555 3673 Acked Write 4 Done Done Sent false
11556 3674 Acked Read 8 Done Done Sent false
11557 3675 Acked Write 4 Done Done Sent false
11558 3676 Acked Flush 0 Done Done Sent false
11559 3677 Acked Read 6 Done Done Sent false
11560 3678 Acked Write 9 Done Done Sent false
11561 3679 Acked Write 10 Done Done Sent false
11562 3680 Acked Write 7 Done Done Sent false
11563 3681 Acked Write 10 Done Done Sent false
11564 3682 Acked Flush 0 Done Done Sent false
11565 3683 Acked Read 5 Done Done Sent false
11566 3684 Acked Write 2 Done Done Sent false
11567 3685 Acked Write 2 Done Done Sent false
11568 3686 Acked Write 4 Done Done Sent false
11569 3687 Acked Read 3 Done Done Sent false
11570 3688 Acked Read 6 Done Done Sent false
11571 3689 Acked Write 1 Done Done Sent false
11572 3690 Acked Flush 0 Done Done Sent false
11573 3691 Acked Read 4 Done Done Sent false
11574 3692 Acked Write 2 Done Done Sent false
11575 3693 Acked Read 1 Done Done Sent false
11576 3694 Acked Flush 0 Done Done Sent false
11577 3695 Acked Read 6 Done Done Sent false
11578 3696 Acked Flush 0 Done Done Sent false
11579 3697 Acked Read 9 Done Done Sent false
11580 3698 Acked Write 5 Done Done Sent false
11581 3699 Acked Write 8 Done Done Sent false
11582 3700 Acked Flush 0 Done Done Sent false
11583 3701 Acked Read 4 Done Done Sent false
11584 3702 Acked Read 6 Done Done Sent false
11585 3703 Acked Write 9 Done Done Sent false
11586 3704 Acked Write 7 Done Done Sent false
11587 3705 Acked Write 3 Done Done Sent false
11588 3706 Acked Read 1 Done Done Sent false
11589 3707 Acked Write 1 Done Done Sent false
11590 3708 Acked Read 5 Done Done Sent false
11591 3709 Acked Read 5 Done Done Sent false
11592 3710 Acked Read 1 Done Done Sent false
11593 3711 Acked Read 9 Done Done Sent false
11594 3712 Acked Read 3 Done Done Sent false
11595 3713 Acked Read 8 Done Done Sent false
11596 STATES DS:0 DS:1 DS:2 TOTAL
11597 Sent 0 0 301 301
11598 Done 301 301 0 602
11599 Skipped 0 0 0 0
11600 Error 0 0 0 0
11601 Last Flush: Some(JobId(3700)) Some(JobId(3700)) Some(JobId(3412))
11602 Downstairs last five completed: 3412 3411 3410 3409 3408
11603 Upstairs last five completed: 3713 3712 3711 3710 3709
11604 CLIENT: Up:0 ds:301 act:2
116052025-02-28T06:03:30.628ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
116062025-02-28T06:03:31.586ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
116072025-02-28T06:03:31.884ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
116082025-02-28T06:03:31.884ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60177) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
116092025-02-28T06:03:31.886ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
116102025-02-28T06:03:31.886ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
116112025-02-28T06:03:31.886ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
116122025-02-28T06:03:31.886ZINFOcrucible: send last flush ID to this DS: Some(JobId(3412)) = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
116132025-02-28T06:03:31.886ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
116142025-02-28T06:03:31.886ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3412)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
116152025-02-28T06:03:31.886ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3412)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
11616 ----------------------------------------------------------------
11617 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11618 Downstairs last five completed: 3714 3713 3712 3711 3710
11619 Upstairs last five completed: 3714 3713 3712 3711 3710
11620 CLIENT: Up:0 ds:0 act:3
11621 Replay: All jobs finished
11622 Main task has work to do, go find it
11623 got dsc Stop(0)
11624 stop 0
11625 Feb 28 06:03:34.129 INFO request completed[8810] Got stop action so:false kr:false
11626 , latency_us: 362, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 3c412538-928e-47a0-ba49-f00443a84843, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
116272025-02-28T06:03:34.130ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
116282025-02-28T06:03:34.130ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
116292025-02-28T06:03:34.130ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
11630 Feb 28 06:03:34.130 INFO request completed, latency_us: 133, response_code: 200, uri: /state/cid/0, method: GET, req_id: ea5085f7-6aa0-4a18-87e8-de36a10033c8, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
116312025-02-28T06:03:34.130ZINFOcrucible: client task is exiting = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
11632 [8810] Exited with: ExitStatus(unix_wait_status(9))
11633 [8810][0] reports Exit
11634 Feb 28 06:03:38.132 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/0, method: GET, req_id: c71b93af-d91a-48dc-ac5b-5474876b3182, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
11635 001/300 Read block 106 len 28672
11636 002/300 Read block 2646 len 8192
11637 003/300 Write block 3163 len 32768 data: 2 2 3 3 3 3 3 3
11638 004/300 Write block 3249 len 16384 data: 4 4 3 4
11639 005/300 Read block 22 len 32768
11640 006/300 Flush
11641 007/300 Read block 2074 len 36864
11642 008/300 Read block 3140 len 16384
11643 009/300 Read block 1048 len 24576
11644 010/300 Read block 174 len 20480
11645 011/300 Read block 1211 len 36864
11646 012/300 Write block 3157 len 20480 data: 3 4 4 4 3
11647 013/300 Read block 560 len 40960
11648 014/300 Read block 3800 len 28672
11649 015/300 Read block 969 len 40960
11650 016/300 Write block 238 len 16384 data: 7 5 4 4
11651 017/300 Write block 2846 len 32768 data: 3 3 3 3 3 2 3 3
11652 018/300 Read block 3334 len 8192
11653 019/300 Write block 2139 len 8192 data: 2 2
11654 020/300 Write block 3403 len 4096 data: 2
11655 021/300 Flush
11656 022/300 Read block 2933 len 40960
11657 023/300 Read block 3543 len 4096
11658 024/300 Read block 4359 len 24576
11659 025/300 Read block 982 len 24576
11660 026/300 Write block 1978 len 28672 data: 2 2 2 2 2 3 3
11661 027/300 Read block 4375 len 40960
11662 028/300 Write block 4341 len 36864 data: 4 4 4 4 4 5 4 4 3
11663 029/300 Read block 3357 len 32768
11664 030/300 Write block 975 len 24576 data: 2 2 2 3 3 3
11665 031/300 Write block 214 len 24576 data: 3 3 3 3 4 5
11666 032/300 Flush
11667 033/300 Read block 1092 len 24576
11668 034/300 Read block 4490 len 24576
11669 035/300 Read block 3613 len 32768
11670 036/300 Read block 518 len 4096
11671 037/300 Write block 1766 len 8192 data: 3 2
11672 038/300 Flush
11673 039/300 Read block 4199 len 4096
11674 040/300 Read block 3230 len 16384
11675 041/300 Read block 3737 len 40960
11676 042/300 Write block 4271 len 36864 data: 5 5 4 4 3 3 3 3 3
11677 043/300 Write block 3478 len 40960 data: 3 4 4 3 3 3 3 3 3 3
11678 044/300 Read block 510 len 28672
11679 045/300 Write block 4225 len 40960 data: 3 4 4 5 6 6 6 3 3 3
11680 046/300 Write block 748 len 12288 data: 3 4 4
11681 047/300 Read block 2556 len 12288
11682 048/300 Read block 294 len 4096
11683 049/300 Read block 3438 len 36864
11684 050/300 Read block 4249 len 16384
11685 051/300 Write block 1171 len 16384 data: 4 4 4 5
11686 052/300 Write block 4318 len 16384 data: 3 4 4 4
11687 053/300 Read block 958 len 8192
11688 054/300 Write block 657 len 40960 data: 2 2 2 2 2 2 2 2 2 2
11689 055/300 Read block 3855 len 20480
11690 056/300 Read block 1808 len 32768
11691 057/300 Read block 4444 len 28672
11692 058/300 Write block 4762 len 28672 data: 3 3 4 3 3 3 3
11693 059/300 Flush
11694 060/300 Write block 1872 len 28672 data: 3 3 3 3 3 4 4
11695 061/300 Read block 139 len 32768
11696 062/300 Read block 1124 len 12288
11697 063/300 Write block 4383 len 40960 data: 3 4 4 5 4 4 4 6 6 4
11698 064/300 Read block 4915 len 36864
11699 065/300 Write block 3134 len 20480 data: 3 2 2 2 2
11700 066/300 Read block 4641 len 40960
11701 067/300 Read block 3938 len 12288
11702 068/300 Read block 3715 len 20480
11703 069/300 Write block 2331 len 20480 data: 2 2 2 2 2
11704 070/300 Write block 4204 len 12288 data: 4 3 3
11705 071/300 Read block 1878 len 4096
11706 072/300 Read block 4831 len 40960
11707 073/300 Write block 3888 len 24576 data: 4 3 3 3 2 2
11708 074/300 Read block 4907 len 16384
11709 075/300 Read block 3865 len 40960
11710 076/300 Write block 2851 len 36864 data: 3 4 4 3 3 3 3 3 2
11711 077/300 Write block 4632 len 32768 data: 5 5 4 3 3 3 3 3
11712 078/300 Flush
11713 079/300 Read block 2768 len 12288
11714 080/300 Write block 436 len 28672 data: 6 5 5 5 5 4 6
11715 081/300 Read block 2993 len 12288
11716 082/300 Write block 1544 len 32768 data: 2 2 3 3 3 3 3 3
11717 083/300 Flush
11718 084/300 Write block 355 len 8192 data: 3 3
11719 085/300 Write block 2475 len 24576 data: 4 4 3 3 3 3
11720 086/300 Write block 3914 len 28672 data: 3 3 3 2 2 2 2
11721 087/300 Read block 4546 len 12288
11722 088/300 Read block 3044 len 4096
11723 089/300 Read block 839 len 36864
11724 090/300 Write block 658 len 32768 data: 3 3 3 3 3 3 3 3
11725 091/300 Write block 4212 len 24576 data: 5 5 4 4 4 4
11726 092/300 Read block 3970 len 20480
11727 093/300 Write block 3829 len 4096 data: 3
11728 094/300 Read block 1275 len 32768
11729 095/300 Read block 1408 len 16384
11730 096/300 Read block 2383 len 32768
11731 097/300 Write block 2431 len 24576 data: 2 3 2 2 2 2
11732 098/300 Write block 1707 len 16384 data: 5 5 5 5
11733 099/300 Flush
11734 100/300 Write block 4862 len 28672 data: 2 2 2 2 2 3 3
11735 101/300 Write block 3807 len 16384 data: 2 2 3 3
11736 102/300 Write block 3525 len 16384 data: 6 6 7 7
11737 103/300 Write block 1641 len 8192 data: 2 2
11738 104/300 Read block 825 len 20480
11739 105/300 Read block 2181 len 4096
11740 106/300 Write block 316 len 12288 data: 4 4 4
11741 107/300 Write block 2071 len 16384 data: 3 3 2 2
11742 108/300 Write block 414 len 4096 data: 5
11743 109/300 Write block 3716 len 12288 data: 2 3 5
11744 110/300 Write block 4508 len 32768 data: 3 3 4 4 3 3 4 4
11745 111/300 Read block 3223 len 24576
11746 112/300 Flush
11747 113/300 Flush
11748 114/300 Read block 4733 len 36864
11749 115/300 Read block 4418 len 36864
11750 116/300 Write block 1954 len 24576 data: 3 3 3 3 3 3
11751 117/300 Read block 2058 len 32768
11752 118/300 Write block 3536 len 28672 data: 3 2 2 2 2 2 2
11753 119/300 Write block 2228 len 16384 data: 4 4 4 5
11754 120/300 Read block 1940 len 24576
11755 121/300 Read block 4270 len 12288
11756 122/300 Write block 4754 len 40960 data: 2 2 3 3 3 3 3 3 4 4
11757 123/300 Write block 3123 len 20480 data: 3 3 3 3 3
11758 124/300 Flush
11759 125/300 Write block 2523 len 8192 data: 5 6
11760 126/300 Read block 691 len 20480
11761 127/300 Read block 4174 len 4096
11762 128/300 Read block 2103 len 12288
11763 129/300 Read block 3839 len 8192
11764 130/300 Write block 382 len 20480 data: 5 5 4 4 4
11765 131/300 Write block 3354 len 16384 data: 3 3 3 2
11766 132/300 Read block 1476 len 16384
11767 133/300 Read block 3625 len 24576
11768 134/300 Write block 1136 len 20480 data: 2 2 3 3 3
11769 135/300 Write block 2320 len 4096 data: 4
11770 136/300 Read block 3681 len 20480
11771 137/300 Write block 1681 len 28672 data: 2 3 4 4 4 4 3
11772 138/300 Read block 710 len 24576
11773 139/300 Write block 2712 len 28672 data: 4 6 5 5 5 5 4
11774 140/300 Write block 1741 len 40960 data: 4 3 3 2 2 2 3 3 3 3
11775 141/300 Flush
11776 142/300 Write block 4780 len 12288 data: 5 5 4
11777 143/300 Read block 471 len 24576
11778 144/300 Read block 919 len 40960
11779 145/300 Flush
11780 146/300 Write block 1408 len 8192 data: 3 4
11781 147/300 Write block 2164 len 36864 data: 3 3 3 3 4 3 3 2 2
11782 148/300 Write block 4496 len 4096 data: 5
11783 149/300 Read block 3647 len 12288
11784 150/300 Write block 1423 len 4096 data: 3
11785 151/300 Write block 1031 len 4096 data: 3
11786 152/300 Read block 4939 len 32768
11787 153/300 Read block 4431 len 16384
11788 154/300 Read block 2300 len 32768
11789 155/300 Read block 4293 len 8192
11790 156/300 Read block 4847 len 36864
11791 157/300 Read block 721 len 8192
11792 158/300 Write block 4823 len 40960 data: 3 3 3 3 3 4 4 6 6 5
11793 159/300 Read block 2535 len 20480
11794 160/300 Read block 437 len 4096
11795 161/300 Read block 4663 len 4096
11796 162/300 Read block 1644 len 8192
11797 163/300 Read block 1993 len 24576
11798 164/300 Write block 1692 len 40960 data: 4 3 4 4 5 5 4 4 5 5
11799 165/300 Read block 4133 len 4096
11800 166/300 Read block 1134 len 32768
11801 167/300 Write block 4871 len 12288 data: 3 3 3
11802 168/300 Write block 408 len 4096 data: 4
11803 169/300 Read block 3085 len 28672
11804 170/300 Read block 1422 len 4096
11805 171/300 Read block 453 len 12288
11806 172/300 Write block 3058 len 28672 data: 2 2 2 3 3 3 3
11807 173/300 Read block 4714 len 20480
11808 174/300 Read block 1784 len 4096
11809 175/300 Read block 4718 len 20480
11810 176/300 Write block 1747 len 36864 data: 4 4 4 4 2 2 2 2 2
11811 177/300 Flush
11812 178/300 Flush
11813 179/300 Read block 148 len 24576
11814 180/300 Write block 1578 len 32768 data: 4 4 4 5 5 5 5 5
11815 181/300 Write block 672 len 28672 data: 5 4 5 5 4 3 4
11816 182/300 Write block 4223 len 28672 data: 2 2 4 5 5 6 7
11817 183/300 Write block 1906 len 12288 data: 2 2 2
11818 184/300 Write block 4531 len 24576 data: 4 2 2 2 3 3
11819 185/300 Write block 2234 len 24576 data: 6 4 5 4 4 3
11820 186/300 Write block 1273 len 36864 data: 5 5 3 3 2 3 3 4 4
11821 187/300 Read block 4785 len 32768
11822 188/300 Read block 4832 len 36864
11823 189/300 Read block 1511 len 32768
11824 190/300 Write block 1966 len 8192 data: 2 2
11825 191/300 Read block 4531 len 16384
11826 192/300 Read block 4708 len 32768
11827 193/300 Read block 2422 len 20480
11828 194/300 Read block 433 len 12288
11829 195/300 Write block 789 len 36864 data: 2 2 3 4 4 4 5 6 5
11830 196/300 Read block 3619 len 24576
11831 197/300 Write block 926 len 36864 data: 4 4 4 4 5 5 5 4 3
11832 198/300 Read block 3981 len 28672
11833 199/300 Read block 4454 len 12288
11834 200/300 Read block 3603 len 4096
11835 201/300 Write block 1054 len 32768 data: 4 3 3 2 2 2 2 2
11836 202/300 Read block 2912 len 12288
11837 203/300 Read block 528 len 4096
11838 204/300 Read block 688 len 32768
11839 205/300 Write block 4342 len 4096 data: 5
11840 206/300 Flush
11841 207/300 Read block 314 len 12288
11842 208/300 Read block 852 len 24576
11843 209/300 Write block 3061 len 40960 data: 4 4 4 4 3 3 4 3 3 3
11844 210/300 Write block 4365 len 4096 data: 3
11845 211/300 Read block 2539 len 28672
11846 212/300 Write block 4108 len 40960 data: 2 2 3 3 4 4 4 4 5 5
11847 213/300 Flush
11848 214/300 Read block 4122 len 40960
11849 215/300 Read block 2500 len 4096
11850 216/300 Read block 3593 len 16384
11851 217/300 Write block 4954 len 20480 data: 4 3 3 3 3
11852 218/300 Read block 771 len 36864
11853 219/300 Flush
11854 220/300 Read block 4000 len 20480
11855 221/300 Read block 2556 len 36864
11856 222/300 Write block 657 len 36864 data: 3 4 4 4 4 4 4 4 4
11857 223/300 Read block 4084 len 8192
11858 224/300 Write block 4889 len 40960 data: 7 6 6 4 4 4 4 4 4 5
11859 225/300 Write block 3699 len 16384 data: 2 2 2 2
11860 226/300 Read block 1976 len 12288
11861 227/300 Write block 3301 len 20480 data: 4 5 4 4 5
11862 228/300 Read block 3260 len 12288
11863 229/300 Write block 4503 len 20480 data: 4 4 4 5 4
11864 230/300 Write block 3536 len 28672 data: 4 3 3 3 3 3 3
11865 231/300 Read block 4597 len 12288
11866 232/300 Write block 2335 len 36864 data: 3 3 3 3 3 3 3 3 3
11867 233/300 Read block 4379 len 4096
11868 234/300 Flush
11869 235/300 Read block 438 len 32768
11870 236/300 Write block 2231 len 8192 data: 6 4
11871 237/300 Flush
11872 238/300 Read block 3158 len 20480
11873 239/300 Flush
11874 240/300 Write block 209 len 24576 data: 2 2 2 2 2 4
11875 241/300 Read block 769 len 4096
11876 242/300 Write block 4020 len 36864 data: 3 3 3 3 3 3 2 2 2
11877 243/300 Write block 2641 len 12288 data: 3 3 3
11878 244/300 Write block 4405 len 12288 data: 5 4 6
11879 245/300 Read block 2097 len 16384
11880 246/300 Read block 2572 len 12288
11881 247/300 Write block 4029 len 8192 data: 2 2
11882 248/300 Read block 3464 len 4096
11883 249/300 Read block 963 len 12288
11884 250/300 Read block 2448 len 40960
11885 251/300 Write block 3844 len 40960 data: 2 2 2 2 3 3 4 4 5 5
11886 252/300 Write block 2933 len 8192 data: 3 3
11887 253/300 Read block 1752 len 36864
11888 254/300 Write block 2685 len 20480 data: 5 4 5 4 5
11889 255/300 Write block 1575 len 24576 data: 3 3 3 5 5 5
11890 256/300 Write block 2371 len 28672 data: 3 2 2 3 4 5 4
11891 257/300 Read block 2744 len 28672
11892 258/300 Read block 1621 len 28672
11893 259/300 Read block 1214 len 8192
11894 260/300 Read block 744 len 36864
11895 261/300 Read block 2015 len 4096
11896 262/300 Read block 703 len 12288
11897 263/300 Read block 4336 len 8192
11898 264/300 Flush
11899 265/300 Flush
11900 266/300 Read block 3672 len 40960
11901 267/300 Write block 4876 len 16384 data: 3 3 4 5
11902 268/300 Read block 1286 len 24576
11903 269/300 Read block 3495 len 20480
11904 270/300 Read block 1714 len 16384
11905 271/300 Write block 3778 len 12288 data: 3 3 4
11906 272/300 Read block 3332 len 24576
11907 273/300 Read block 746 len 36864
11908 274/300 Flush
11909 275/300 Flush
11910 276/300 Read block 3146 len 20480
11911 277/300 Write block 1131 len 8192 data: 3 2
11912 278/300 Write block 2225 len 12288 data: 3 3 4
11913 279/300 Read block 4975 len 8192
11914 280/300 Write block 602 len 12288 data: 3 3 2
11915 281/300 Read block 1682 len 16384
11916 282/300 Read block 2056 len 32768
11917 283/300 Write block 3838 len 20480 data: 2 2 2 2 2
11918 284/300 Write block 4447 len 4096 data: 3
11919 285/300 Write block 2861 len 16384 data: 3 3 4 3
11920 286/300 Read block 160 len 36864
11921 287/300 Read block 3798 len 24576
11922 288/300 Read block 4908 len 32768
11923 289/300 Write block 838 len 36864 data: 2 2 2 3 3 3 5 5 4
11924 290/300 Flush
11925 291/300 Write block 1828 len 12288 data: 3 3 2
11926 292/300 Write block 2637 len 36864 data: 2 3 3 3 4 4 4 2 2
11927 293/300 Read block 1948 len 4096
11928 294/300 Write block 1253 len 40960 data: 2 3 4 4 4 3 2 3 5 5
11929 295/300 Read block 3921 len 4096
11930 296/300 Read block 2281 len 8192
11931 297/300 Read block 97 len 40960
11932 298/300 Read block 180 len 16384
11933 299/300 Write block 670 len 16384 data: 5 5 6 5
11934 300/300 Read block 2756 len 4096
11935 301/300 Write block 1741 len 28672 data: 5 4 4 3 3 3 5
11936 Main task has work to do, go find it
11937 got dsc Start(0)
11938 start 0
11939 Feb 28 06:03:38.370 INFO request completed, latency_us: 188, response_code: 204, uri: /start/cid/0, method: POST, req_id: a56164b5-3466-4eb6-b457-ee314e15cc0a, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
11940 Replay: started 0, returned:Ok(())
11941 ----------------------------------------------------------------
11942 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11943 States: Offline Active Active
11944 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11945 3715 Acked Read 7 Sent Done Done false
11946 3716 Acked Read 2 Sent Done Done false
11947 3717 Acked Write 8 Sent Done Done false
11948 3718 Acked Write 4 Sent Done Done false
11949 3719 Acked Read 8 Sent Done Done false
11950 3720 Acked Flush 0 Sent Done Done false
11951 3721 Acked Read 9 Sent Done Done false
11952 3722 Acked Read 4 Sent Done Done false
11953 3723 Acked Read 6 Sent Done Done false
11954 3724 Acked Read 5 Sent Done Done false
11955 3725 Acked Read 9 Sent Done Done false
11956 3726 Acked Write 5 Sent Done Done false
11957 3727 Acked Read 10 Sent Done Done false
11958 3728 Acked Read 7 Sent Done Done false
11959 3729 Acked Read 10 Sent Done Done false
11960 3730 Acked Write 4 Sent Done Done false
11961 3731 Acked Write 8 Sent Done Done false
11962 3732 Acked Read 2 Sent Done Done false
11963 3733 Acked Write 2 Sent Done Done false
11964 3734 Acked Write 1 Sent Done Done false
11965 3735 Acked Flush 0 Sent Done Done false
11966 3736 Acked Read 10 Sent Done Done false
11967 3737 Acked Read 1 Sent Done Done false
11968 3738 Acked Read 6 Sent Done Done false
11969 3739 Acked Read 6 Sent Done Done false
11970 3740 Acked Write 7 Sent Done Done false
11971 3741 Acked Read 10 Sent Done Done false
11972 3742 Acked Write 9 Sent Done Done false
11973 3743 Acked Read 8 Sent Done Done false
11974 3744 Acked Write 6 Sent Done Done false
11975 3745 Acked Write 6 Sent Done Done false
11976 3746 Acked Flush 0 Sent Done Done false
11977 3747 Acked Read 6 Sent Done Done false
11978 3748 Acked Read 6 Sent Done Done false
11979 3749 Acked Read 8 Sent Done Done false
11980 3750 Acked Read 1 Sent Done Done false
11981 3751 Acked Write 2 Sent Done Done false
11982 3752 Acked Flush 0 Sent Done Done false
11983 3753 Acked Read 1 Sent Done Done false
11984 3754 Acked Read 4 Sent Done Done false
11985 3755 Acked Read 10 Sent Done Done false
11986 3756 Acked Write 9 Sent Done Done false
11987 3757 Acked Write 10 Sent Done Done false
11988 3758 Acked Read 7 Sent Done Done false
11989 3759 Acked Write 10 Sent Done Done false
11990 3760 Acked Write 3 Sent Done Done false
11991 3761 Acked Read 3 Sent Done Done false
11992 3762 Acked Read 1 Sent Done Done false
11993 3763 Acked Read 9 Sent Done Done false
11994 3764 Acked Read 4 Sent Done Done false
11995 3765 Acked Write 4 Sent Done Done false
11996 3766 Acked Write 4 Sent Done Done false
11997 3767 Acked Read 2 Sent Done Done false
11998 3768 Acked Write 10 Sent Done Done false
11999 3769 Acked Read 5 Sent Done Done false
12000 3770 Acked Read 8 Sent Done Done false
12001 3771 Acked Read 7 Sent Done Done false
12002 3772 Acked Write 7 Sent Done Done false
12003 3773 Acked Flush 0 Sent Done Done false
12004 3774 Acked Write 7 Sent Done Done false
12005 3775 Acked Read 8 Sent Done Done false
12006 3776 Acked Read 3 Sent Done Done false
12007 3777 Acked Write 10 Sent Done Done false
12008 3778 Acked Read 9 Sent Done Done false
12009 3779 Acked Write 5 Sent Done Done false
12010 3780 Acked Read 10 Sent Done Done false
12011 3781 Acked Read 3 Sent Done Done false
12012 3782 Acked Read 5 Sent Done Done false
12013 3783 Acked Write 5 Sent Done Done false
12014 3784 Acked Write 3 Sent Done Done false
12015 3785 Acked Read 1 Sent Done Done false
12016 3786 Acked Read 10 Sent Done Done false
12017 3787 Acked Write 6 Sent Done Done false
12018 3788 Acked Read 4 Sent Done Done false
12019 3789 Acked Read 10 Sent Done Done false
12020 3790 Acked Write 9 Sent Done Done false
12021 3791 Acked Write 8 Sent Done Done false
12022 3792 Acked Flush 0 Sent Done Done false
12023 3793 Acked Read 3 Sent Done Done false
12024 3794 Acked Write 7 Sent Done Done false
12025 3795 Acked Read 3 Sent Done Done false
12026 3796 Acked Write 8 Sent Done Done false
12027 3797 Acked Flush 0 Sent Done Done false
12028 3798 Acked Write 2 Sent Done Done false
12029 3799 Acked Write 6 Sent Done Done false
12030 3800 Acked Write 7 Sent Done Done false
12031 3801 Acked Read 3 Sent Done Done false
12032 3802 Acked Read 1 Sent Done Done false
12033 3803 Acked Read 9 Sent Done Done false
12034 3804 Acked Write 8 Sent Done Done false
12035 3805 Acked Write 6 Sent Done Done false
12036 3806 Acked Read 5 Sent Done Done false
12037 3807 Acked Write 1 Sent Done Done false
12038 3808 Acked Read 8 Sent Done Done false
12039 3809 Acked Read 4 Sent Done Done false
12040 3810 Acked Read 8 Sent Done Done false
12041 3811 Acked Write 6 Sent Done Done false
12042 3812 Acked Write 4 Sent Done Done false
12043 3813 Acked Flush 0 Sent Done Done false
12044 3814 Acked Write 7 Sent Done Done false
12045 3815 Acked Write 4 Sent Done Done false
12046 3816 Acked Write 4 Sent Done Done false
12047 3817 Acked Write 2 Sent Done Done false
12048 3818 Acked Read 5 Sent Done Done false
12049 3819 Acked Read 1 Sent Done Done false
12050 3820 Acked Write 3 Sent Done Done false
12051 3821 Acked Write 4 Sent Done Done false
12052 3822 Acked Write 1 Sent Done Done false
12053 3823 Acked Write 3 Sent Done Done false
12054 3824 Acked Write 8 Sent Done Done false
12055 3825 Acked Read 6 Sent Done Done false
12056 3826 Acked Flush 0 Sent Done Done false
12057 3827 Acked Flush 0 Sent Done Done false
12058 3828 Acked Read 9 Sent Done Done false
12059 3829 Acked Read 9 Sent Done Done false
12060 3830 Acked Write 6 Sent Done Done false
12061 3831 Acked Read 8 Sent Done Done false
12062 3832 Acked Write 7 Sent Done Done false
12063 3833 Acked Write 4 Sent Done Done false
12064 3834 Acked Read 6 Sent Done Done false
12065 3835 Acked Read 3 Sent Done Done false
12066 3836 Acked Write 10 Sent Done Done false
12067 3837 Acked Write 5 Sent Done Done false
12068 3838 Acked Flush 0 Sent Done Done false
12069 3839 Acked Write 2 Sent Done Done false
12070 3840 Acked Read 5 Sent Done Done false
12071 3841 Acked Read 1 Sent Done Done false
12072 3842 Acked Read 3 Sent Done Done false
12073 3843 Acked Read 2 Sent Done Done false
12074 3844 Acked Write 5 Sent Done Done false
12075 3845 Acked Write 4 Sent Done Done false
12076 3846 Acked Read 4 Sent Done Done false
12077 3847 Acked Read 6 Sent Done Done false
12078 3848 Acked Write 5 Sent Done Done false
12079 3849 Acked Write 1 Sent Done Done false
12080 3850 Acked Read 5 Sent Done Done false
12081 3851 Acked Write 7 Sent Done Done false
12082 3852 Acked Read 6 Sent Done Done false
12083 3853 Acked Write 7 Sent Done Done false
12084 3854 Acked Write 10 Sent Done Done false
12085 3855 Acked Flush 0 Sent Done Done false
12086 3856 Acked Write 3 Sent Done Done false
12087 3857 Acked Read 6 Sent Done Done false
12088 3858 Acked Read 10 Sent Done Done false
12089 3859 Acked Flush 0 Sent Done Done false
12090 3860 Acked Write 2 Sent Done Done false
12091 3861 Acked Write 9 Sent Done Done false
12092 3862 Acked Write 1 Sent Done Done false
12093 3863 Acked Read 3 Sent Done Done false
12094 3864 Acked Write 1 Sent Done Done false
12095 3865 Acked Write 1 Sent Done Done false
12096 3866 Acked Read 8 Sent Done Done false
12097 3867 Acked Read 4 Sent Done Done false
12098 3868 Acked Read 8 Sent Done Done false
12099 3869 Acked Read 2 Sent Done Done false
12100 3870 Acked Read 9 Sent Done Done false
12101 3871 Acked Read 2 Sent Done Done false
12102 3872 Acked Write 10 Sent Done Done false
12103 3873 Acked Read 5 Sent Done Done false
12104 3874 Acked Read 1 Sent Done Done false
12105 3875 Acked Read 1 Sent Done Done false
12106 3876 Acked Read 2 Sent Done Done false
12107 3877 Acked Read 6 Sent Done Done false
12108 3878 Acked Write 10 Sent Done Done false
12109 3879 Acked Read 1 Sent Done Done false
12110 3880 Acked Read 8 Sent Done Done false
12111 3881 Acked Write 3 Sent Done Done false
12112 3882 Acked Write 1 Sent Done Done false
12113 3883 Acked Read 7 Sent Done Done false
12114 3884 Acked Read 1 Sent Done Done false
12115 3885 Acked Read 3 Sent Done Done false
12116 3886 Acked Write 7 Sent Done Done false
12117 3887 Acked Read 5 Sent Done Done false
12118 3888 Acked Read 1 Sent Done Done false
12119 3889 Acked Read 5 Sent Done Done false
12120 3890 Acked Write 9 Sent Done Done false
12121 3891 Acked Flush 0 Sent Done Done false
12122 3892 Acked Flush 0 Sent Done Done false
12123 3893 Acked Read 6 Sent Done Done false
12124 3894 Acked Write 8 Sent Done Done false
12125 3895 Acked Write 7 Sent Done Done false
12126 3896 Acked Write 7 Sent Done Done false
12127 3897 Acked Write 3 Sent Done Done false
12128 3898 Acked Write 6 Sent Done Done false
12129 3899 Acked Write 6 Sent Done Done false
12130 3900 Acked Write 9 Sent Done Done false
12131 3901 Acked Read 8 Sent Done Done false
12132 3902 Acked Read 9 Sent Done Done false
12133 3903 Acked Read 8 Sent Done Done false
12134 3904 Acked Write 2 Sent Done Done false
12135 3905 Acked Read 4 Sent Done Done false
12136 3906 Acked Read 8 Sent Done Done false
12137 3907 Acked Read 5 Sent Done Done false
12138 3908 Acked Read 3 Sent Done Done false
12139 3909 Acked Write 9 Sent Done Done false
12140 3910 Acked Read 6 Sent Done Done false
12141 3911 Acked Write 9 Sent Done Done false
12142 3912 Acked Read 7 Sent Done Done false
12143 3913 Acked Read 3 Sent Done Done false
12144 3914 Acked Read 1 Sent Done Done false
12145 3915 Acked Write 8 Sent Done Done false
12146 3916 Acked Read 3 Sent Done Done false
12147 3917 Acked Read 1 Sent Done Done false
12148 3918 Acked Read 8 Sent Done Done false
12149 3919 Acked Write 1 Sent Done Done false
12150 3920 Acked Flush 0 Sent Done Done false
12151 3921 Acked Read 3 Sent Done Done false
12152 3922 Acked Read 6 Sent Done Done false
12153 3923 Acked Write 10 Sent Done Done false
12154 3924 Acked Write 1 Sent Done Done false
12155 3925 Acked Read 7 Sent Done Done false
12156 3926 Acked Write 10 Sent Done Done false
12157 3927 Acked Flush 0 Sent Done Done false
12158 3928 Acked Read 10 Sent Done Done false
12159 3929 Acked Read 1 Sent Done Done false
12160 3930 Acked Read 4 Sent Done Done false
12161 3931 Acked Write 5 Sent Done Done false
12162 3932 Acked Read 9 Sent Done Done false
12163 3933 Acked Flush 0 Sent Done Done false
12164 3934 Acked Read 5 Sent Done Done false
12165 3935 Acked Read 9 Sent Done Done false
12166 3936 Acked Write 9 Sent Done Done false
12167 3937 Acked Read 2 Sent Done Done false
12168 3938 Acked Write 10 Sent Done Done false
12169 3939 Acked Write 4 Sent Done Done false
12170 3940 Acked Read 3 Sent Done Done false
12171 3941 Acked Write 5 Sent Done Done false
12172 3942 Acked Read 3 Sent Done Done false
12173 3943 Acked Write 5 Sent Done Done false
12174 3944 Acked Write 7 Sent Done Done false
12175 3945 Acked Read 3 Sent Done Done false
12176 3946 Acked Write 9 Sent Done Done false
12177 3947 Acked Read 1 Sent Done Done false
12178 3948 Acked Flush 0 Sent Done Done false
12179 3949 Acked Read 8 Sent Done Done false
12180 3950 Acked Write 2 Sent Done Done false
12181 3951 Acked Flush 0 Sent Done Done false
12182 3952 Acked Read 5 Sent Done Done false
12183 3953 Acked Flush 0 Sent Done Done false
12184 3954 Acked Write 6 Sent Done Done false
12185 3955 Acked Read 1 Sent Done Done false
12186 3956 Acked Write 9 Sent Done Done false
12187 3957 Acked Write 3 Sent Done Done false
12188 3958 Acked Write 3 Sent Done Done false
12189 3959 Acked Read 4 Sent Done Done false
12190 3960 Acked Read 3 Sent Done Done false
12191 3961 Acked Write 2 Sent Done Done false
12192 3962 Acked Read 1 Sent Done Done false
12193 3963 Acked Read 3 Sent Done Done false
12194 3964 Acked Read 10 Sent Done Done false
12195 3965 Acked Write 10 Sent Done Done false
12196 3966 Acked Write 2 Sent Done Done false
12197 3967 Acked Read 9 Sent Done Done false
12198 3968 Acked Write 5 Sent Done Done false
12199 3969 Acked Write 6 Sent Done Done false
12200 3970 Acked Write 7 Sent Done Done false
12201 3971 Acked Read 7 Sent Done Done false
12202 3972 Acked Read 7 Sent Done Done false
12203 3973 Acked Read 2 Sent Done Done false
12204 3974 Acked Read 9 Sent Done Done false
12205 3975 Acked Read 1 Sent Done Done false
12206 3976 Acked Read 3 Sent Done Done false
12207 3977 Acked Read 2 Sent Done Done false
12208 3978 Acked Flush 0 Sent Done Done false
12209 3979 Acked Flush 0 Sent Done Done false
12210 3980 Acked Read 10 Sent Done Done false
12211 3981 Acked Write 4 Sent Done Done false
12212 3982 Acked Read 6 Sent Done Done false
12213 3983 Acked Read 5 Sent Done Done false
12214 3984 Acked Read 4 Sent Done Done false
12215 3985 Acked Write 3 Sent Done Done false
12216 3986 Acked Read 6 Sent Done Done false
12217 3987 Acked Read 9 Sent Done Done false
12218 3988 Acked Flush 0 Sent Done Done false
12219 3989 Acked Flush 0 Sent Done Done false
12220 3990 Acked Read 5 Sent Done Done false
12221 3991 Acked Write 2 Sent Done Done false
12222 3992 Acked Write 3 Sent Done Done false
12223 3993 Acked Read 2 Sent Done Done false
12224 3994 Acked Write 3 Sent Done Done false
12225 3995 Acked Read 4 Sent Done Done false
12226 3996 Acked Read 8 Sent Done Done false
12227 3997 Acked Write 5 Sent Done Done false
12228 3998 Acked Write 1 Sent Done Done false
12229 3999 Acked Write 4 Sent Done Done false
12230 4000 Acked Read 9 Sent Done Done false
12231 4001 Acked Read 6 Sent Done Done false
12232 4002 Acked Read 8 Sent Done Done false
12233 4003 Acked Write 9 Sent Done Done false
12234 4004 Acked Flush 0 Sent Done Done false
12235 4005 Acked Write 3 Sent Done Done false
12236 4006 Acked Write 9 Sent Done Done false
12237 4007 Acked Read 1 Sent Done Done false
12238 4008 Acked Write 10 Sent Done Done false
12239 4009 Acked Read 1 Sent Done Done false
12240 4010 Acked Read 2 Sent Done Done false
12241 4011 Acked Read 10 Sent Done Done false
12242 4012 Acked Read 4 Sent Done Done false
12243 4013 Acked Write 4 Sent Done Done false
12244 4014 Acked Read 1 Sent Done Done false
12245 4015 Acked Write 7 Sent Done Done false
12246 STATES DS:0 DS:1 DS:2 TOTAL
12247 Sent 301 0 0 301
12248 Done 0 301 301 602
12249 Skipped 0 0 0 0
12250 Error 0 0 0 0
12251 Last Flush: Some(JobId(3714)) Some(JobId(4004)) Some(JobId(4004))
12252 Downstairs last five completed: 3714 3713 3712 3711 3710
12253 Upstairs last five completed: 4015 4014 4013 4012 4011
12254 CLIENT: Up:0 ds:301 act:2
122552025-02-28T06:03:38.873ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
122562025-02-28T06:03:39.599ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
122572025-02-28T06:03:40.600ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
122582025-02-28T06:03:41.601ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
12259 [8810] got start action
12260 [8810] I am going to restart
12261 Starting downstairs at port 8810
12262 Make output file at "/tmp/dsc/downstairs-8810.txt"
12263 [8810][0] reports Starting
12264 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1221)
12265 [8810][0] reports Running
12266 ----------------------------------------------------------------
12267 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12268 States: Offline Active Active
12269 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12270 3715 Acked Read 7 Sent Done Done false
12271 3716 Acked Read 2 Sent Done Done false
12272 3717 Acked Write 8 Sent Done Done false
12273 3718 Acked Write 4 Sent Done Done false
12274 3719 Acked Read 8 Sent Done Done false
12275 3720 Acked Flush 0 Sent Done Done false
12276 3721 Acked Read 9 Sent Done Done false
12277 3722 Acked Read 4 Sent Done Done false
12278 3723 Acked Read 6 Sent Done Done false
12279 3724 Acked Read 5 Sent Done Done false
12280 3725 Acked Read 9 Sent Done Done false
12281 3726 Acked Write 5 Sent Done Done false
12282 3727 Acked Read 10 Sent Done Done false
12283 3728 Acked Read 7 Sent Done Done false
12284 3729 Acked Read 10 Sent Done Done false
12285 3730 Acked Write 4 Sent Done Done false
12286 3731 Acked Write 8 Sent Done Done false
12287 3732 Acked Read 2 Sent Done Done false
12288 3733 Acked Write 2 Sent Done Done false
12289 3734 Acked Write 1 Sent Done Done false
12290 3735 Acked Flush 0 Sent Done Done false
12291 3736 Acked Read 10 Sent Done Done false
12292 3737 Acked Read 1 Sent Done Done false
12293 3738 Acked Read 6 Sent Done Done false
12294 3739 Acked Read 6 Sent Done Done false
12295 3740 Acked Write 7 Sent Done Done false
12296 3741 Acked Read 10 Sent Done Done false
12297 3742 Acked Write 9 Sent Done Done false
12298 3743 Acked Read 8 Sent Done Done false
12299 3744 Acked Write 6 Sent Done Done false
12300 3745 Acked Write 6 Sent Done Done false
12301 3746 Acked Flush 0 Sent Done Done false
12302 3747 Acked Read 6 Sent Done Done false
12303 3748 Acked Read 6 Sent Done Done false
12304 3749 Acked Read 8 Sent Done Done false
12305 3750 Acked Read 1 Sent Done Done false
12306 3751 Acked Write 2 Sent Done Done false
12307 3752 Acked Flush 0 Sent Done Done false
12308 3753 Acked Read 1 Sent Done Done false
12309 3754 Acked Read 4 Sent Done Done false
12310 3755 Acked Read 10 Sent Done Done false
12311 3756 Acked Write 9 Sent Done Done false
12312 3757 Acked Write 10 Sent Done Done false
12313 3758 Acked Read 7 Sent Done Done false
12314 3759 Acked Write 10 Sent Done Done false
12315 3760 Acked Write 3 Sent Done Done false
12316 3761 Acked Read 3 Sent Done Done false
12317 3762 Acked Read 1 Sent Done Done false
12318 3763 Acked Read 9 Sent Done Done false
12319 3764 Acked Read 4 Sent Done Done false
12320 3765 Acked Write 4 Sent Done Done false
12321 3766 Acked Write 4 Sent Done Done false
12322 3767 Acked Read 2 Sent Done Done false
12323 3768 Acked Write 10 Sent Done Done false
12324 3769 Acked Read 5 Sent Done Done false
12325 3770 Acked Read 8 Sent Done Done false
12326 3771 Acked Read 7 Sent Done Done false
12327 3772 Acked Write 7 Sent Done Done false
12328 3773 Acked Flush 0 Sent Done Done false
12329 3774 Acked Write 7 Sent Done Done false
12330 3775 Acked Read 8 Sent Done Done false
12331 3776 Acked Read 3 Sent Done Done false
12332 3777 Acked Write 10 Sent Done Done false
12333 3778 Acked Read 9 Sent Done Done false
12334 3779 Acked Write 5 Sent Done Done false
12335 3780 Acked Read 10 Sent Done Done false
12336 3781 Acked Read 3 Sent Done Done false
12337 3782 Acked Read 5 Sent Done Done false
12338 3783 Acked Write 5 Sent Done Done false
12339 3784 Acked Write 3 Sent Done Done false
12340 3785 Acked Read 1 Sent Done Done false
12341 3786 Acked Read 10 Sent Done Done false
12342 3787 Acked Write 6 Sent Done Done false
12343 3788 Acked Read 4 Sent Done Done false
12344 3789 Acked Read 10 Sent Done Done false
12345 3790 Acked Write 9 Sent Done Done false
12346 3791 Acked Write 8 Sent Done Done false
12347 3792 Acked Flush 0 Sent Done Done false
12348 3793 Acked Read 3 Sent Done Done false
12349 3794 Acked Write 7 Sent Done Done false
12350 3795 Acked Read 3 Sent Done Done false
12351 3796 Acked Write 8 Sent Done Done false
12352 3797 Acked Flush 0 Sent Done Done false
12353 3798 Acked Write 2 Sent Done Done false
12354 3799 Acked Write 6 Sent Done Done false
12355 3800 Acked Write 7 Sent Done Done false
12356 3801 Acked Read 3 Sent Done Done false
12357 3802 Acked Read 1 Sent Done Done false
12358 3803 Acked Read 9 Sent Done Done false
12359 3804 Acked Write 8 Sent Done Done false
12360 3805 Acked Write 6 Sent Done Done false
12361 3806 Acked Read 5 Sent Done Done false
12362 3807 Acked Write 1 Sent Done Done false
12363 3808 Acked Read 8 Sent Done Done false
12364 3809 Acked Read 4 Sent Done Done false
12365 3810 Acked Read 8 Sent Done Done false
12366 3811 Acked Write 6 Sent Done Done false
12367 3812 Acked Write 4 Sent Done Done false
12368 3813 Acked Flush 0 Sent Done Done false
12369 3814 Acked Write 7 Sent Done Done false
12370 3815 Acked Write 4 Sent Done Done false
12371 3816 Acked Write 4 Sent Done Done false
12372 3817 Acked Write 2 Sent Done Done false
12373 3818 Acked Read 5 Sent Done Done false
12374 3819 Acked Read 1 Sent Done Done false
12375 3820 Acked Write 3 Sent Done Done false
12376 3821 Acked Write 4 Sent Done Done false
12377 3822 Acked Write 1 Sent Done Done false
12378 3823 Acked Write 3 Sent Done Done false
12379 3824 Acked Write 8 Sent Done Done false
12380 3825 Acked Read 6 Sent Done Done false
12381 3826 Acked Flush 0 Sent Done Done false
12382 3827 Acked Flush 0 Sent Done Done false
12383 3828 Acked Read 9 Sent Done Done false
12384 3829 Acked Read 9 Sent Done Done false
12385 3830 Acked Write 6 Sent Done Done false
12386 3831 Acked Read 8 Sent Done Done false
12387 3832 Acked Write 7 Sent Done Done false
12388 3833 Acked Write 4 Sent Done Done false
12389 3834 Acked Read 6 Sent Done Done false
12390 3835 Acked Read 3 Sent Done Done false
12391 3836 Acked Write 10 Sent Done Done false
12392 3837 Acked Write 5 Sent Done Done false
12393 3838 Acked Flush 0 Sent Done Done false
12394 3839 Acked Write 2 Sent Done Done false
12395 3840 Acked Read 5 Sent Done Done false
12396 3841 Acked Read 1 Sent Done Done false
12397 3842 Acked Read 3 Sent Done Done false
12398 3843 Acked Read 2 Sent Done Done false
12399 3844 Acked Write 5 Sent Done Done false
12400 3845 Acked Write 4 Sent Done Done false
12401 3846 Acked Read 4 Sent Done Done false
12402 3847 Acked Read 6 Sent Done Done false
12403 3848 Acked Write 5 Sent Done Done false
12404 3849 Acked Write 1 Sent Done Done false
12405 3850 Acked Read 5 Sent Done Done false
12406 3851 Acked Write 7 Sent Done Done false
12407 3852 Acked Read 6 Sent Done Done false
12408 3853 Acked Write 7 Sent Done Done false
12409 3854 Acked Write 10 Sent Done Done false
12410 3855 Acked Flush 0 Sent Done Done false
12411 3856 Acked Write 3 Sent Done Done false
12412 3857 Acked Read 6 Sent Done Done false
12413 3858 Acked Read 10 Sent Done Done false
12414 3859 Acked Flush 0 Sent Done Done false
12415 3860 Acked Write 2 Sent Done Done false
12416 3861 Acked Write 9 Sent Done Done false
12417 3862 Acked Write 1 Sent Done Done false
12418 3863 Acked Read 3 Sent Done Done false
12419 3864 Acked Write 1 Sent Done Done false
12420 3865 Acked Write 1 Sent Done Done false
12421 3866 Acked Read 8 Sent Done Done false
12422 3867 Acked Read 4 Sent Done Done false
12423 3868 Acked Read 8 Sent Done Done false
12424 3869 Acked Read 2 Sent Done Done false
12425 3870 Acked Read 9 Sent Done Done false
12426 3871 Acked Read 2 Sent Done Done false
12427 3872 Acked Write 10 Sent Done Done false
12428 3873 Acked Read 5 Sent Done Done false
12429 3874 Acked Read 1 Sent Done Done false
12430 3875 Acked Read 1 Sent Done Done false
12431 3876 Acked Read 2 Sent Done Done false
12432 3877 Acked Read 6 Sent Done Done false
12433 3878 Acked Write 10 Sent Done Done false
12434 3879 Acked Read 1 Sent Done Done false
12435 3880 Acked Read 8 Sent Done Done false
12436 3881 Acked Write 3 Sent Done Done false
12437 3882 Acked Write 1 Sent Done Done false
12438 3883 Acked Read 7 Sent Done Done false
12439 3884 Acked Read 1 Sent Done Done false
12440 3885 Acked Read 3 Sent Done Done false
12441 3886 Acked Write 7 Sent Done Done false
12442 3887 Acked Read 5 Sent Done Done false
12443 3888 Acked Read 1 Sent Done Done false
12444 3889 Acked Read 5 Sent Done Done false
12445 3890 Acked Write 9 Sent Done Done false
12446 3891 Acked Flush 0 Sent Done Done false
12447 3892 Acked Flush 0 Sent Done Done false
12448 3893 Acked Read 6 Sent Done Done false
12449 3894 Acked Write 8 Sent Done Done false
12450 3895 Acked Write 7 Sent Done Done false
12451 3896 Acked Write 7 Sent Done Done false
12452 3897 Acked Write 3 Sent Done Done false
12453 3898 Acked Write 6 Sent Done Done false
12454 3899 Acked Write 6 Sent Done Done false
12455 3900 Acked Write 9 Sent Done Done false
12456 3901 Acked Read 8 Sent Done Done false
12457 3902 Acked Read 9 Sent Done Done false
12458 3903 Acked Read 8 Sent Done Done false
12459 3904 Acked Write 2 Sent Done Done false
12460 3905 Acked Read 4 Sent Done Done false
12461 3906 Acked Read 8 Sent Done Done false
12462 3907 Acked Read 5 Sent Done Done false
12463 3908 Acked Read 3 Sent Done Done false
12464 3909 Acked Write 9 Sent Done Done false
12465 3910 Acked Read 6 Sent Done Done false
12466 3911 Acked Write 9 Sent Done Done false
12467 3912 Acked Read 7 Sent Done Done false
12468 3913 Acked Read 3 Sent Done Done false
12469 3914 Acked Read 1 Sent Done Done false
12470 3915 Acked Write 8 Sent Done Done false
12471 3916 Acked Read 3 Sent Done Done false
12472 3917 Acked Read 1 Sent Done Done false
12473 3918 Acked Read 8 Sent Done Done false
12474 3919 Acked Write 1 Sent Done Done false
12475 3920 Acked Flush 0 Sent Done Done false
12476 3921 Acked Read 3 Sent Done Done false
12477 3922 Acked Read 6 Sent Done Done false
12478 3923 Acked Write 10 Sent Done Done false
12479 3924 Acked Write 1 Sent Done Done false
12480 3925 Acked Read 7 Sent Done Done false
12481 3926 Acked Write 10 Sent Done Done false
12482 3927 Acked Flush 0 Sent Done Done false
12483 3928 Acked Read 10 Sent Done Done false
12484 3929 Acked Read 1 Sent Done Done false
12485 3930 Acked Read 4 Sent Done Done false
12486 3931 Acked Write 5 Sent Done Done false
12487 3932 Acked Read 9 Sent Done Done false
12488 3933 Acked Flush 0 Sent Done Done false
12489 3934 Acked Read 5 Sent Done Done false
12490 3935 Acked Read 9 Sent Done Done false
12491 3936 Acked Write 9 Sent Done Done false
12492 3937 Acked Read 2 Sent Done Done false
12493 3938 Acked Write 10 Sent Done Done false
12494 3939 Acked Write 4 Sent Done Done false
12495 3940 Acked Read 3 Sent Done Done false
12496 3941 Acked Write 5 Sent Done Done false
12497 3942 Acked Read 3 Sent Done Done false
12498 3943 Acked Write 5 Sent Done Done false
12499 3944 Acked Write 7 Sent Done Done false
12500 3945 Acked Read 3 Sent Done Done false
12501 3946 Acked Write 9 Sent Done Done false
12502 3947 Acked Read 1 Sent Done Done false
12503 3948 Acked Flush 0 Sent Done Done false
12504 3949 Acked Read 8 Sent Done Done false
12505 3950 Acked Write 2 Sent Done Done false
12506 3951 Acked Flush 0 Sent Done Done false
12507 3952 Acked Read 5 Sent Done Done false
12508 3953 Acked Flush 0 Sent Done Done false
12509 3954 Acked Write 6 Sent Done Done false
12510 3955 Acked Read 1 Sent Done Done false
12511 3956 Acked Write 9 Sent Done Done false
12512 3957 Acked Write 3 Sent Done Done false
12513 3958 Acked Write 3 Sent Done Done false
12514 3959 Acked Read 4 Sent Done Done false
12515 3960 Acked Read 3 Sent Done Done false
12516 3961 Acked Write 2 Sent Done Done false
12517 3962 Acked Read 1 Sent Done Done false
12518 3963 Acked Read 3 Sent Done Done false
12519 3964 Acked Read 10 Sent Done Done false
12520 3965 Acked Write 10 Sent Done Done false
12521 3966 Acked Write 2 Sent Done Done false
12522 3967 Acked Read 9 Sent Done Done false
12523 3968 Acked Write 5 Sent Done Done false
12524 3969 Acked Write 6 Sent Done Done false
12525 3970 Acked Write 7 Sent Done Done false
12526 3971 Acked Read 7 Sent Done Done false
12527 3972 Acked Read 7 Sent Done Done false
12528 3973 Acked Read 2 Sent Done Done false
12529 3974 Acked Read 9 Sent Done Done false
12530 3975 Acked Read 1 Sent Done Done false
12531 3976 Acked Read 3 Sent Done Done false
12532 3977 Acked Read 2 Sent Done Done false
12533 3978 Acked Flush 0 Sent Done Done false
12534 3979 Acked Flush 0 Sent Done Done false
12535 3980 Acked Read 10 Sent Done Done false
12536 3981 Acked Write 4 Sent Done Done false
12537 3982 Acked Read 6 Sent Done Done false
12538 3983 Acked Read 5 Sent Done Done false
12539 3984 Acked Read 4 Sent Done Done false
12540 3985 Acked Write 3 Sent Done Done false
12541 3986 Acked Read 6 Sent Done Done false
12542 3987 Acked Read 9 Sent Done Done false
12543 3988 Acked Flush 0 Sent Done Done false
12544 3989 Acked Flush 0 Sent Done Done false
12545 3990 Acked Read 5 Sent Done Done false
12546 3991 Acked Write 2 Sent Done Done false
12547 3992 Acked Write 3 Sent Done Done false
12548 3993 Acked Read 2 Sent Done Done false
12549 3994 Acked Write 3 Sent Done Done false
12550 3995 Acked Read 4 Sent Done Done false
12551 3996 Acked Read 8 Sent Done Done false
12552 3997 Acked Write 5 Sent Done Done false
12553 3998 Acked Write 1 Sent Done Done false
12554 3999 Acked Write 4 Sent Done Done false
12555 4000 Acked Read 9 Sent Done Done false
12556 4001 Acked Read 6 Sent Done Done false
12557 4002 Acked Read 8 Sent Done Done false
12558 4003 Acked Write 9 Sent Done Done false
12559 4004 Acked Flush 0 Sent Done Done false
12560 4005 Acked Write 3 Sent Done Done false
12561 4006 Acked Write 9 Sent Done Done false
12562 4007 Acked Read 1 Sent Done Done false
12563 4008 Acked Write 10 Sent Done Done false
12564 4009 Acked Read 1 Sent Done Done false
12565 4010 Acked Read 2 Sent Done Done false
12566 4011 Acked Read 10 Sent Done Done false
12567 4012 Acked Read 4 Sent Done Done false
12568 4013 Acked Write 4 Sent Done Done false
12569 4014 Acked Read 1 Sent Done Done false
12570 4015 Acked Write 7 Sent Done Done false
12571 STATES DS:0 DS:1 DS:2 TOTAL
12572 Sent 301 0 0 301
12573 Done 0 301 301 602
12574 Skipped 0 0 0 0
12575 Error 0 0 0 0
12576 Last Flush: Some(JobId(3714)) Some(JobId(4004)) Some(JobId(4004))
12577 Downstairs last five completed: 3714 3713 3712 3711 3710
12578 Upstairs last five completed: 4015 4014 4013 4012 4011
12579 CLIENT: Up:0 ds:301 act:2
125802025-02-28T06:03:42.876ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
125812025-02-28T06:03:43.605ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
125822025-02-28T06:03:44.131ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
125832025-02-28T06:03:44.131ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:40982) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
125842025-02-28T06:03:44.132ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
125852025-02-28T06:03:44.133ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
125862025-02-28T06:03:44.133ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
125872025-02-28T06:03:44.133ZINFOcrucible: send last flush ID to this DS: Some(JobId(3714)) = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
125882025-02-28T06:03:44.133ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
125892025-02-28T06:03:44.133ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(3714)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
125902025-02-28T06:03:44.133ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(3714)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
12591 ----------------------------------------------------------------
12592 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12593 Downstairs last five completed: 4016 4015 4014 4013 4012
12594 Upstairs last five completed: 4016 4015 4014 4013 4012
12595 CLIENT: Up:0 ds:0 act:3
12596 Replay: All jobs finished
12597 Main task has work to do, go find it
12598 got dsc Stop(1)
12599 stop 1
12600 Feb 28 06:03:46.378 INFO request completed[8820] Got stop action so:false kr:false
12601 , latency_us: 658, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 57ed5b77-071d-43bf-9849-8223cee10e6f, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
126022025-02-28T06:03:46.379ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
126032025-02-28T06:03:46.379ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
12604 Feb 28 06:03:46.379 INFO request completed, latency_us: 137, response_code{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:03:46.379584088Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
12605 : 200, uri: /state/cid/1, method: GET, req_id: 1c250499-10a9-46c9-85fb-45f1177886cf, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
126062025-02-28T06:03:46.379ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
12607 [8820] Exited with: ExitStatus(unix_wait_status(9))
12608 [8820][1] reports Exit
12609 Feb 28 06:03:50.382 INFO request completed, latency_us: 196, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6681ed0f-09ae-4776-9463-ca1faa5e9eb5, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
12610 001/300 Flush
12611 002/300 Read block 3543 len 40960
12612 003/300 Read block 3062 len 40960
12613 004/300 Read block 1554 len 20480
12614 005/300 Write block 3546 len 24576 data: 2 2 2 2 2 2
12615 006/300 Write block 2511 len 8192 data: 5 6
12616 007/300 Read block 2634 len 36864
12617 008/300 Write block 502 len 32768 data: 3 3 3 3 3 3 3 4
12618 009/300 Write block 4574 len 16384 data: 3 3 3 2
12619 010/300 Flush
12620 011/300 Write block 3822 len 4096 data: 2
12621 012/300 Write block 4994 len 24576 data: 2 2 2 2 2 2
12622 013/300 Flush
12623 014/300 Read block 2308 len 32768
12624 015/300 Read block 3618 len 32768
12625 016/300 Read block 103 len 12288
12626 017/300 Write block 75 len 4096 data: 5
12627 018/300 Read block 3761 len 8192
12628 019/300 Write block 4206 len 4096 data: 4
12629 020/300 Write block 3833 len 12288 data: 2 2 2
12630 021/300 Read block 707 len 16384
12631 022/300 Read block 3218 len 20480
12632 023/300 Write block 1493 len 16384 data: 4 3 3 3
12633 024/300 Read block 3321 len 40960
12634 025/300 Write block 4684 len 16384 data: 2 2 2 2
12635 026/300 Write block 2140 len 24576 data: 3 2 2 2 2 2
12636 027/300 Write block 444 len 8192 data: 8 8
12637 028/300 Flush
12638 029/300 Write block 1681 len 16384 data: 3 4 5 5
12639 030/300 Read block 1177 len 28672
12640 031/300 Read block 1919 len 32768
12641 032/300 Write block 3748 len 32768 data: 3 3 3 4 3 3 2 2
12642 033/300 Write block 641 len 36864 data: 3 3 3 3 3 3 4 4 3
12643 034/300 Write block 2200 len 40960 data: 4 4 5 3 3 2 2 2 2 4
12644 035/300 Read block 4721 len 32768
12645 036/300 Read block 3932 len 24576
12646 037/300 Write block 3450 len 40960 data: 2 2 3 4 4 4 4 4 4 4
12647 038/300 Write block 150 len 8192 data: 4 4
12648 039/300 Write block 2058 len 16384 data: 3 3 3 4
12649 040/300 Read block 1926 len 36864
12650 041/300 Flush
12651 042/300 Read block 2628 len 36864
12652 043/300 Write block 4923 len 24576 data: 2 2 3 4 3 2
12653 044/300 Read block 4419 len 16384
12654 045/300 Read block 2021 len 8192
12655 046/300 Read block 2728 len 16384
12656 047/300 Write block 4965 len 4096 data: 3
12657 048/300 Write block 1184 len 16384 data: 3 3 4 3
12658 049/300 Read block 3802 len 20480
12659 050/300 Flush
12660 051/300 Read block 1430 len 20480
12661 052/300 Read block 1115 len 12288
12662 053/300 Read block 681 len 20480
12663 054/300 Write block 4197 len 12288 data: 3 3 3
12664 055/300 Write block 1380 len 32768 data: 2 3 3 4 4 4 4 4
12665 056/300 Write block 928 len 40960 data: 5 5 6 6 6 5 4 3 2 3
12666 057/300 Write block 4473 len 36864 data: 2 2 3 3 3 3 4 3 3
12667 058/300 Write block 3860 len 36864 data: 3 4 4 4 5 4 4 4 4
12668 059/300 Read block 334 len 36864
12669 060/300 Read block 3814 len 16384
12670 061/300 Write block 1173 len 12288 data: 5 6 5
12671 062/300 Write block 3970 len 24576 data: 3 3 3 3 4 4
12672 063/300 Read block 1547 len 32768
12673 064/300 Flush
12674 065/300 Read block 2863 len 40960
12675 066/300 Flush
12676 067/300 Read block 2702 len 40960
12677 068/300 Read block 3603 len 4096
12678 069/300 Write block 1791 len 8192 data: 3 3
12679 070/300 Read block 715 len 32768
12680 071/300 Read block 4647 len 4096
12681 072/300 Write block 18 len 32768 data: 3 3 3 3 3 3 3 5
12682 073/300 Flush
12683 074/300 Write block 2357 len 32768 data: 3 3 3 3 3 4 5 6
12684 075/300 Write block 4354 len 28672 data: 4 4 3 4 4 4 2
12685 076/300 Read block 702 len 8192
12686 077/300 Write block 1312 len 8192 data: 4 4
12687 078/300 Write block 2264 len 20480 data: 3 3 3 3 3
12688 079/300 Write block 1155 len 24576 data: 3 3 3 2 2 3
12689 080/300 Flush
12690 081/300 Write block 1077 len 24576 data: 3 4 4 6 4 3
12691 082/300 Flush
12692 083/300 Write block 3371 len 16384 data: 3 2 2 2
12693 084/300 Read block 2518 len 24576
12694 085/300 Read block 1176 len 36864
12695 086/300 Write block 2087 len 20480 data: 3 3 3 3 3
12696 087/300 Write block 4627 len 20480 data: 4 4 3 4 5
12697 088/300 Write block 4971 len 36864 data: 3 3 3 3 2 2 2 4 4
12698 089/300 Read block 2126 len 32768
12699 090/300 Flush
12700 091/300 Write block 2232 len 24576 data: 5 4 7 5 6 5
12701 092/300 Read block 2250 len 16384
12702 093/300 Read block 1335 len 28672
12703 094/300 Read block 1258 len 40960
12704 095/300 Read block 3321 len 28672
12705 096/300 Flush
12706 097/300 Read block 4852 len 36864
12707 098/300 Read block 4175 len 8192
12708 099/300 Write block 3047 len 20480 data: 2 2 3 3 2
12709 100/300 Read block 52 len 28672
12710 101/300 Write block 3603 len 20480 data: 2 2 3 3 3
12711 102/300 Flush
12712 103/300 Read block 4032 len 28672
12713 104/300 Flush
12714 105/300 Write block 3607 len 24576 data: 4 3 3 3 3 3
12715 106/300 Read block 2934 len 8192
12716 107/300 Read block 722 len 36864
12717 108/300 Read block 4388 len 16384
12718 109/300 Write block 3621 len 8192 data: 2 2
12719 110/300 Read block 1371 len 40960
12720 111/300 Read block 513 len 20480
12721 112/300 Write block 3692 len 8192 data: 3 2
12722 113/300 Read block 1656 len 24576
12723 114/300 Write block 4724 len 16384 data: 3 3 3 3
12724 115/300 Write block 3794 len 12288 data: 3 3 3
12725 116/300 Read block 441 len 16384
12726 117/300 Write block 4143 len 36864 data: 4 3 4 4 4 4 4 4 3
12727 118/300 Write block 2814 len 32768 data: 5 5 5 5 4 4 3 2
12728 119/300 Write block 2424 len 32768 data: 3 3 3 3 2 2 3 3
12729 120/300 Read block 2992 len 28672
12730 121/300 Read block 4587 len 8192
12731 122/300 Read block 3603 len 40960
12732 123/300 Read block 1 len 12288
12733 124/300 Read block 4155 len 12288
12734 125/300 Write block 2975 len 36864 data: 2 2 2 2 2 2 3 3 4
12735 126/300 Write block 3840 len 24576 data: 3 3 3 2 3 3
12736 127/300 Read block 4119 len 36864
12737 128/300 Read block 129 len 40960
12738 129/300 Write block 4572 len 16384 data: 3 3 4 4
12739 130/300 Read block 3038 len 24576
12740 131/300 Read block 847 len 16384
12741 132/300 Read block 210 len 16384
12742 133/300 Write block 2936 len 20480 data: 2 3 4 4 4
12743 134/300 Write block 4572 len 16384 data: 4 4 5 5
12744 135/300 Write block 872 len 12288 data: 3 3 3
12745 136/300 Write block 1983 len 36864 data: 4 4 3 3 3 4 4 3 3
12746 137/300 Flush
12747 138/300 Read block 3417 len 12288
12748 139/300 Read block 4481 len 16384
12749 140/300 Read block 128 len 16384
12750 141/300 Read block 1959 len 24576
12751 142/300 Read block 2716 len 36864
12752 143/300 Flush
12753 144/300 Read block 3398 len 20480
12754 145/300 Read block 3808 len 24576
12755 146/300 Write block 743 len 32768 data: 2 3 3 3 3 4 5 5
12756 147/300 Read block 483 len 8192
12757 148/300 Read block 28 len 16384
12758 149/300 Write block 607 len 8192 data: 2 2
12759 150/300 Write block 2214 len 28672 data: 7 7 7 7 7 5 3
12760 151/300 Write block 395 len 28672 data: 2 2 2 3 3 3 3
12761 152/300 Write block 2477 len 36864 data: 4 4 4 4 3 3 4 5 5
12762 153/300 Flush
12763 154/300 Read block 383 len 32768
12764 155/300 Read block 3488 len 24576
12765 156/300 Read block 1603 len 36864
12766 157/300 Write block 3301 len 36864 data: 5 6 5 5 6 6 6 6 5
12767 158/300 Read block 4578 len 40960
12768 159/300 Read block 953 len 32768
12769 160/300 Write block 1663 len 16384 data: 3 3 4 4
12770 161/300 Write block 2416 len 36864 data: 3 3 3 3 3 3 3 3 4
12771 162/300 Read block 1613 len 4096
12772 163/300 Write block 4744 len 20480 data: 3 3 2 2 2
12773 164/300 Read block 639 len 28672
12774 165/300 Write block 4481 len 40960 data: 4 3 3 3 3 3 3 2 2 2
12775 166/300 Write block 2006 len 40960 data: 2 2 2 2 2 2 2 2 2 2
12776 167/300 Write block 3343 len 24576 data: 4 4 4 4 3 2
12777 168/300 Write block 4895 len 36864 data: 5 5 5 6 5 5 4 4 5
12778 169/300 Write block 2976 len 28672 data: 3 3 3 3 3 4 4
12779 170/300 Write block 3066 len 32768 data: 4 5 4 4 4 3 3 4
12780 171/300 Read block 2045 len 36864
12781 172/300 Read block 1886 len 36864
12782 173/300 Write block 2176 len 16384 data: 3 3 4 6
12783 174/300 Read block 4243 len 4096
12784 175/300 Write block 1046 len 40960 data: 4 3 3 4 4 3 4 4 5 4
12785 176/300 Write block 1025 len 16384 data: 5 5 5 3
12786 177/300 Write block 2233 len 4096 data: 5
12787 178/300 Read block 2655 len 12288
12788 179/300 Write block 1145 len 36864 data: 4 3 2 3 3 3 3 3 3
12789 180/300 Write block 3030 len 40960 data: 4 4 4 4 4 3 3 3 3 3
12790 181/300 Write block 4029 len 16384 data: 3 3 2 2
12791 182/300 Write block 4627 len 12288 data: 5 5 4
12792 183/300 Read block 2602 len 12288
12793 184/300 Flush
12794 185/300 Write block 4400 len 24576 data: 2 2 3 3 3 6
12795 186/300 Write block 2712 len 12288 data: 5 7 6
12796 187/300 Write block 2759 len 8192 data: 4 4
12797 188/300 Flush
12798 189/300 Write block 546 len 32768 data: 2 3 3 3 3 4 5 5
12799 190/300 Write block 4235 len 12288 data: 3 3 3
12800 191/300 Read block 1269 len 8192
12801 192/300 Read block 2712 len 32768
12802 193/300 Write block 2086 len 4096 data: 3
12803 194/300 Read block 3413 len 36864
12804 195/300 Read block 2717 len 40960
12805 196/300 Write block 2841 len 28672 data: 3 3 3 3 3 4 4
12806 197/300 Write block 2025 len 40960 data: 2 2 2 4 4 3 3 3 3 2
12807 198/300 Read block 1901 len 12288
12808 199/300 Read block 289 len 36864
12809 200/300 Read block 1163 len 12288
12810 201/300 Read block 2397 len 20480
12811 202/300 Write block 2194 len 28672 data: 3 4 4 4 3 3 5
12812 203/300 Read block 2448 len 36864
12813 204/300 Flush
12814 205/300 Write block 1955 len 4096 data: 4
12815 206/300 Read block 71 len 20480
12816 207/300 Write block 1153 len 28672 data: 4 3 4 4 4 3 3
12817 208/300 Write block 1825 len 20480 data: 3 3 3 4 4
12818 209/300 Read block 4096 len 8192
12819 210/300 Read block 4768 len 20480
12820 211/300 Read block 3887 len 24576
12821 212/300 Read block 1806 len 16384
12822 213/300 Write block 2437 len 24576 data: 2 2 3 4 3 4
12823 214/300 Write block 2245 len 12288 data: 2 2 2
12824 215/300 Read block 1023 len 28672
12825 216/300 Read block 1254 len 24576
12826 217/300 Read block 3610 len 32768
12827 218/300 Read block 1199 len 12288
12828 219/300 Write block 3182 len 32768 data: 2 2 2 2 2 2 2 2
12829 220/300 Write block 2672 len 40960 data: 5 4 5 5 4 3 4 4 4 3
12830 221/300 Write block 3368 len 4096 data: 3
12831 222/300 Read block 82 len 28672
12832 223/300 Write block 2835 len 40960 data: 2 2 2 2 2 2 4 4 4 4
12833 224/300 Read block 4031 len 24576
12834 225/300 Write block 4647 len 32768 data: 6 5 4 3 3 5 5 5
12835 226/300 Read block 3139 len 40960
12836 227/300 Write block 1249 len 40960 data: 3 3 2 2 3 4 5 5 5 4
12837 228/300 Write block 2275 len 28672 data: 5 5 5 6 6 6 5
12838 229/300 Write block 3338 len 24576 data: 5 5 4 5 4 5
12839 230/300 Flush
12840 231/300 Read block 4927 len 4096
12841 232/300 Flush
12842 233/300 Write block 4428 len 16384 data: 4 4 4 4
12843 234/300 Write block 4526 len 8192 data: 6 6
12844 235/300 Read block 2516 len 32768
12845 236/300 Flush
12846 237/300 Write block 2773 len 36864 data: 2 2 2 3 3 3 4 4 4
12847 238/300 Write block 4169 len 24576 data: 2 2 2 2 2 2
12848 239/300 Read block 3602 len 36864
12849 240/300 Read block 2095 len 28672
12850 241/300 Write block 495 len 8192 data: 4 4
12851 242/300 Write block 1487 len 40960 data: 5 6 6 6 6 6 5 4 4 4
12852 243/300 Read block 1281 len 28672
12853 244/300 Flush
12854 245/300 Write block 119 len 24576 data: 2 2 2 3 3 3
12855 246/300 Read block 945 len 32768
12856 247/300 Read block 4500 len 16384
12857 248/300 Read block 501 len 12288
12858 249/300 Write block 4031 len 32768 data: 3 3 2 2 2 2 2 2
12859 250/300 Read block 312 len 12288
12860 251/300 Write block 505 len 8192 data: 4 4
12861 252/300 Read block 1781 len 40960
12862 253/300 Write block 3502 len 16384 data: 3 3 2 2
12863 254/300 Write block 2663 len 32768 data: 4 5 5 5 5 5 5 6
12864 255/300 Read block 4528 len 16384
12865 256/300 Flush
12866 257/300 Write block 676 len 4096 data: 5
12867 258/300 Read block 1819 len 16384
12868 259/300 Read block 1252 len 8192
12869 260/300 Read block 3154 len 8192
12870 261/300 Read block 2040 len 4096
12871 262/300 Read block 803 len 20480
12872 263/300 Write block 2188 len 4096 data: 3
12873 264/300 Flush
12874 265/300 Read block 3537 len 4096
12875 266/300 Write block 1009 len 36864 data: 2 2 2 2 2 2 2 3 4
12876 267/300 Flush
12877 268/300 Write block 4174 len 12288 data: 3 2 2
12878 269/300 Write block 4637 len 24576 data: 4 4 4 3 2 3
12879 270/300 Read block 4661 len 32768
12880 271/300 Write block 3344 len 16384 data: 5 5 5 4
12881 272/300 Write block 3956 len 32768 data: 3 3 3 3 3 4 3 2
12882 273/300 Read block 2350 len 12288
12883 274/300 Write block 2883 len 28672 data: 3 3 4 4 4 4 4
12884 275/300 Read block 3973 len 8192
12885 276/300 Write block 3535 len 16384 data: 3 5 4 4
12886 277/300 Read block 3746 len 16384
12887 278/300 Read block 444 len 4096
12888 279/300 Write block 3154 len 28672 data: 2 2 3 4 5 5 5
12889 280/300 Read block 2245 len 32768
12890 281/300 Read block 1770 len 36864
12891 282/300 Write block 3320 len 24576 data: 3 3 4 4 3 2
12892 283/300 Read block 4940 len 20480
12893 284/300 Write block 3692 len 4096 data: 4
12894 285/300 Write block 936 len 40960 data: 3 4 4 4 4 4 4 4 4 3
12895 286/300 Read block 1233 len 8192
12896 287/300 Write block 47 len 28672 data: 3 3 3 2 3 3 4
12897 288/300 Read block 957 len 28672
12898 289/300 Read block 963 len 4096
12899 290/300 Read block 851 len 12288
12900 291/300 Read block 306 len 12288
12901 292/300 Read block 888 len 36864
12902 293/300 Write block 2137 len 20480 data: 3 2 3 4 3
12903 294/300 Read block 1710 len 8192
12904 295/300 Flush
12905 296/300 Write block 4925 len 16384 data: 4 5 4 3
12906 297/300 Write block 256 len 4096 data: 2
12907 298/300 Read block 875 len 36864
12908 299/300 Write block 1874 len 36864 data: 4 4 4 5 5 4 4 4 3
12909 300/300 Read block 1387 len 24576
12910 301/300 Write block 1493 len 12288 data: 6 5 5
12911 Main task has work to do, go find it
12912 got dsc Start(1)
12913 start 1
12914 Feb 28 06:03:50.619 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/1, method: POST, req_id: 396e01ad-cb0a-4acc-9ddd-bd4e43d494fb, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
12915 Replay: started 1, returned:Ok(())
12916 ----------------------------------------------------------------
12917 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12918 States: Active Offline Active
12919 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12920 4017 Acked Flush 0 Done Sent Done false
12921 4018 Acked Read 10 Done Sent Done false
12922 4019 Acked Read 10 Done Sent Done false
12923 4020 Acked Read 5 Done Sent Done false
12924 4021 Acked Write 6 Done Sent Done false
12925 4022 Acked Write 2 Done Sent Done false
12926 4023 Acked Read 9 Done Sent Done false
12927 4024 Acked Write 8 Done Sent Done false
12928 4025 Acked Write 4 Done Sent Done false
12929 4026 Acked Flush 0 Done Sent Done false
12930 4027 Acked Write 1 Done Sent Done false
12931 4028 Acked Write 6 Done Sent Done false
12932 4029 Acked Flush 0 Done Sent Done false
12933 4030 Acked Read 8 Done Sent Done false
12934 4031 Acked Read 8 Done Sent Done false
12935 4032 Acked Read 3 Done Sent Done false
12936 4033 Acked Write 1 Done Sent Done false
12937 4034 Acked Read 2 Done Sent Done false
12938 4035 Acked Write 1 Done Sent Done false
12939 4036 Acked Write 3 Done Sent Done false
12940 4037 Acked Read 4 Done Sent Done false
12941 4038 Acked Read 5 Done Sent Done false
12942 4039 Acked Write 4 Done Sent Done false
12943 4040 Acked Read 10 Done Sent Done false
12944 4041 Acked Write 4 Done Sent Done false
12945 4042 Acked Write 6 Done Sent Done false
12946 4043 Acked Write 2 Done Sent Done false
12947 4044 Acked Flush 0 Done Sent Done false
12948 4045 Acked Write 4 Done Sent Done false
12949 4046 Acked Read 7 Done Sent Done false
12950 4047 Acked Read 8 Done Sent Done false
12951 4048 Acked Write 8 Done Sent Done false
12952 4049 Acked Write 9 Done Sent Done false
12953 4050 Acked Write 10 Done Sent Done false
12954 4051 Acked Read 8 Done Sent Done false
12955 4052 Acked Read 6 Done Sent Done false
12956 4053 Acked Write 10 Done Sent Done false
12957 4054 Acked Write 2 Done Sent Done false
12958 4055 Acked Write 4 Done Sent Done false
12959 4056 Acked Read 9 Done Sent Done false
12960 4057 Acked Flush 0 Done Sent Done false
12961 4058 Acked Read 9 Done Sent Done false
12962 4059 Acked Write 6 Done Sent Done false
12963 4060 Acked Read 4 Done Sent Done false
12964 4061 Acked Read 2 Done Sent Done false
12965 4062 Acked Read 4 Done Sent Done false
12966 4063 Acked Write 1 Done Sent Done false
12967 4064 Acked Write 4 Done Sent Done false
12968 4065 Acked Read 5 Done Sent Done false
12969 4066 Acked Flush 0 Done Sent Done false
12970 4067 Acked Read 5 Done Sent Done false
12971 4068 Acked Read 3 Done Sent Done false
12972 4069 Acked Read 5 Done Sent Done false
12973 4070 Acked Write 3 Done Sent Done false
12974 4071 Acked Write 8 Done Sent Done false
12975 4072 Acked Write 10 Done Sent Done false
12976 4073 Acked Write 9 Done Sent Done false
12977 4074 Acked Write 9 Done Sent Done false
12978 4075 Acked Read 9 Done Sent Done false
12979 4076 Acked Read 4 Done Sent Done false
12980 4077 Acked Write 3 Done Sent Done false
12981 4078 Acked Write 6 Done Sent Done false
12982 4079 Acked Read 8 Done Sent Done false
12983 4080 Acked Flush 0 Done Sent Done false
12984 4081 Acked Read 10 Done Sent Done false
12985 4082 Acked Flush 0 Done Sent Done false
12986 4083 Acked Read 10 Done Sent Done false
12987 4084 Acked Read 1 Done Sent Done false
12988 4085 Acked Write 2 Done Sent Done false
12989 4086 Acked Read 8 Done Sent Done false
12990 4087 Acked Read 1 Done Sent Done false
12991 4088 Acked Write 8 Done Sent Done false
12992 4089 Acked Flush 0 Done Sent Done false
12993 4090 Acked Write 8 Done Sent Done false
12994 4091 Acked Write 7 Done Sent Done false
12995 4092 Acked Read 2 Done Sent Done false
12996 4093 Acked Write 2 Done Sent Done false
12997 4094 Acked Write 5 Done Sent Done false
12998 4095 Acked Write 6 Done Sent Done false
12999 4096 Acked Flush 0 Done Sent Done false
13000 4097 Acked Write 6 Done Sent Done false
13001 4098 Acked Flush 0 Done Sent Done false
13002 4099 Acked Write 4 Done Sent Done false
13003 4100 Acked Read 6 Done Sent Done false
13004 4101 Acked Read 9 Done Sent Done false
13005 4102 Acked Write 5 Done Sent Done false
13006 4103 Acked Write 5 Done Sent Done false
13007 4104 Acked Write 9 Done Sent Done false
13008 4105 Acked Read 8 Done Sent Done false
13009 4106 Acked Flush 0 Done Sent Done false
13010 4107 Acked Write 6 Done Sent Done false
13011 4108 Acked Read 4 Done Sent Done false
13012 4109 Acked Read 7 Done Sent Done false
13013 4110 Acked Read 10 Done Sent Done false
13014 4111 Acked Read 7 Done Sent Done false
13015 4112 Acked Flush 0 Done Sent Done false
13016 4113 Acked Read 9 Done Sent Done false
13017 4114 Acked Read 2 Done Sent Done false
13018 4115 Acked Write 5 Done Sent Done false
13019 4116 Acked Read 7 Done Sent Done false
13020 4117 Acked Write 5 Done Sent Done false
13021 4118 Acked Flush 0 Done Sent Done false
13022 4119 Acked Read 7 Done Sent Done false
13023 4120 Acked Flush 0 Done Sent Done false
13024 4121 Acked Write 6 Done Sent Done false
13025 4122 Acked Read 2 Done Sent Done false
13026 4123 Acked Read 9 Done Sent Done false
13027 4124 Acked Read 4 Done Sent Done false
13028 4125 Acked Write 2 Done Sent Done false
13029 4126 Acked Read 10 Done Sent Done false
13030 4127 Acked Read 5 Done Sent Done false
13031 4128 Acked Write 2 Done Sent Done false
13032 4129 Acked Read 6 Done Sent Done false
13033 4130 Acked Write 4 Done Sent Done false
13034 4131 Acked Write 3 Done Sent Done false
13035 4132 Acked Read 4 Done Sent Done false
13036 4133 Acked Write 9 Done Sent Done false
13037 4134 Acked Write 8 Done Sent Done false
13038 4135 Acked Write 8 Done Sent Done false
13039 4136 Acked Read 7 Done Sent Done false
13040 4137 Acked Read 2 Done Sent Done false
13041 4138 Acked Read 10 Done Sent Done false
13042 4139 Acked Read 3 Done Sent Done false
13043 4140 Acked Read 3 Done Sent Done false
13044 4141 Acked Write 9 Done Sent Done false
13045 4142 Acked Write 6 Done Sent Done false
13046 4143 Acked Read 9 Done Sent Done false
13047 4144 Acked Read 10 Done Sent Done false
13048 4145 Acked Write 4 Done Sent Done false
13049 4146 Acked Read 6 Done Sent Done false
13050 4147 Acked Read 4 Done Sent Done false
13051 4148 Acked Read 4 Done Sent Done false
13052 4149 Acked Write 5 Done Sent Done false
13053 4150 Acked Write 4 Done Sent Done false
13054 4151 Acked Write 3 Done Sent Done false
13055 4152 Acked Write 9 Done Sent Done false
13056 4153 Acked Flush 0 Done Sent Done false
13057 4154 Acked Read 3 Done Sent Done false
13058 4155 Acked Read 4 Done Sent Done false
13059 4156 Acked Read 4 Done Sent Done false
13060 4157 Acked Read 6 Done Sent Done false
13061 4158 Acked Read 9 Done Sent Done false
13062 4159 Acked Flush 0 Done Sent Done false
13063 4160 Acked Read 5 Done Sent Done false
13064 4161 Acked Read 6 Done Sent Done false
13065 4162 Acked Write 8 Done Sent Done false
13066 4163 Acked Read 2 Done Sent Done false
13067 4164 Acked Read 4 Done Sent Done false
13068 4165 Acked Write 2 Done Sent Done false
13069 4166 Acked Write 7 Done Sent Done false
13070 4167 Acked Write 7 Done Sent Done false
13071 4168 Acked Write 9 Done Sent Done false
13072 4169 Acked Flush 0 Done Sent Done false
13073 4170 Acked Read 8 Done Sent Done false
13074 4171 Acked Read 6 Done Sent Done false
13075 4172 Acked Read 9 Done Sent Done false
13076 4173 Acked Write 9 Done Sent Done false
13077 4174 Acked Read 10 Done Sent Done false
13078 4175 Acked Read 8 Done Sent Done false
13079 4176 Acked Write 4 Done Sent Done false
13080 4177 Acked Write 9 Done Sent Done false
13081 4178 Acked Read 1 Done Sent Done false
13082 4179 Acked Write 5 Done Sent Done false
13083 4180 Acked Read 7 Done Sent Done false
13084 4181 Acked Write 10 Done Sent Done false
13085 4182 Acked Write 10 Done Sent Done false
13086 4183 Acked Write 6 Done Sent Done false
13087 4184 Acked Write 9 Done Sent Done false
13088 4185 Acked Write 7 Done Sent Done false
13089 4186 Acked Write 8 Done Sent Done false
13090 4187 Acked Read 9 Done Sent Done false
13091 4188 Acked Read 9 Done Sent Done false
13092 4189 Acked Write 4 Done Sent Done false
13093 4190 Acked Read 1 Done Sent Done false
13094 4191 Acked Write 10 Done Sent Done false
13095 4192 Acked Write 4 Done Sent Done false
13096 4193 Acked Write 1 Done Sent Done false
13097 4194 Acked Read 3 Done Sent Done false
13098 4195 Acked Write 9 Done Sent Done false
13099 4196 Acked Write 10 Done Sent Done false
13100 4197 Acked Write 4 Done Sent Done false
13101 4198 Acked Write 3 Done Sent Done false
13102 4199 Acked Read 3 Done Sent Done false
13103 4200 Acked Flush 0 Done Sent Done false
13104 4201 Acked Write 6 Done Sent Done false
13105 4202 Acked Write 3 Done Sent Done false
13106 4203 Acked Write 2 Done Sent Done false
13107 4204 Acked Flush 0 Done Sent Done false
13108 4205 Acked Write 8 Done Sent Done false
13109 4206 Acked Write 3 Done Sent Done false
13110 4207 Acked Read 2 Done Sent Done false
13111 4208 Acked Read 8 Done Sent Done false
13112 4209 Acked Write 1 Done Sent Done false
13113 4210 Acked Read 9 Done Sent Done false
13114 4211 Acked Read 10 Done Sent Done false
13115 4212 Acked Write 7 Done Sent Done false
13116 4213 Acked Write 10 Done Sent Done false
13117 4214 Acked Read 3 Done Sent Done false
13118 4215 Acked Read 9 Done Sent Done false
13119 4216 Acked Read 3 Done Sent Done false
13120 4217 Acked Read 5 Done Sent Done false
13121 4218 Acked Write 7 Done Sent Done false
13122 4219 Acked Read 9 Done Sent Done false
13123 4220 Acked Flush 0 Done Sent Done false
13124 4221 Acked Write 1 Done Sent Done false
13125 4222 Acked Read 5 Done Sent Done false
13126 4223 Acked Write 7 Done Sent Done false
13127 4224 Acked Write 5 Done Sent Done false
13128 4225 Acked Read 2 Done Sent Done false
13129 4226 Acked Read 5 Done Sent Done false
13130 4227 Acked Read 6 Done Sent Done false
13131 4228 Acked Read 4 Done Sent Done false
13132 4229 Acked Write 6 Done Sent Done false
13133 4230 Acked Write 3 Done Sent Done false
13134 4231 Acked Read 7 Done Sent Done false
13135 4232 Acked Read 6 Done Sent Done false
13136 4233 Acked Read 8 Done Sent Done false
13137 4234 Acked Read 3 Done Sent Done false
13138 4235 Acked Write 8 Done Sent Done false
13139 4236 Acked Write 10 Done Sent Done false
13140 4237 Acked Write 1 Done Sent Done false
13141 4238 Acked Read 7 Done Sent Done false
13142 4239 Acked Write 10 Done Sent Done false
13143 4240 Acked Read 6 Done Sent Done false
13144 4241 Acked Write 8 Done Sent Done false
13145 4242 Acked Read 10 Done Sent Done false
13146 4243 Acked Write 10 Done Sent Done false
13147 4244 Acked Write 7 Done Sent Done false
13148 4245 Acked Write 6 Done Sent Done false
13149 4246 Acked Flush 0 Done Sent Done false
13150 4247 Acked Read 1 Done Sent Done false
13151 4248 Acked Flush 0 Done Sent Done false
13152 4249 Acked Write 4 Done Sent Done false
13153 4250 Acked Write 2 Done Sent Done false
13154 4251 Acked Read 8 Done Sent Done false
13155 4252 Acked Flush 0 Done Sent Done false
13156 4253 Acked Write 9 Done Sent Done false
13157 4254 Acked Write 6 Done Sent Done false
13158 4255 Acked Read 9 Done Sent Done false
13159 4256 Acked Read 7 Done Sent Done false
13160 4257 Acked Write 2 Done Sent Done false
13161 4258 Acked Write 10 Done Sent Done false
13162 4259 Acked Read 7 Done Sent Done false
13163 4260 Acked Flush 0 Done Sent Done false
13164 4261 Acked Write 6 Done Sent Done false
13165 4262 Acked Read 8 Done Sent Done false
13166 4263 Acked Read 4 Done Sent Done false
13167 4264 Acked Read 3 Done Sent Done false
13168 4265 Acked Write 8 Done Sent Done false
13169 4266 Acked Read 3 Done Sent Done false
13170 4267 Acked Write 2 Done Sent Done false
13171 4268 Acked Read 10 Done Sent Done false
13172 4269 Acked Write 4 Done Sent Done false
13173 4270 Acked Write 8 Done Sent Done false
13174 4271 Acked Read 4 Done Sent Done false
13175 4272 Acked Flush 0 Done Sent Done false
13176 4273 Acked Write 1 Done Sent Done false
13177 4274 Acked Read 4 Done Sent Done false
13178 4275 Acked Read 2 Done Sent Done false
13179 4276 Acked Read 2 Done Sent Done false
13180 4277 Acked Read 1 Done Sent Done false
13181 4278 Acked Read 5 Done Sent Done false
13182 4279 Acked Write 1 Done Sent Done false
13183 4280 Acked Flush 0 Done Sent Done false
13184 4281 Acked Read 1 Done Sent Done false
13185 4282 Acked Write 9 Done Sent Done false
13186 4283 Acked Flush 0 Done Sent Done false
13187 4284 Acked Write 3 Done Sent Done false
13188 4285 Acked Write 6 Done Sent Done false
13189 4286 Acked Read 8 Done Sent Done false
13190 4287 Acked Write 4 Done Sent Done false
13191 4288 Acked Write 8 Done Sent Done false
13192 4289 Acked Read 3 Done Sent Done false
13193 4290 Acked Write 7 Done Sent Done false
13194 4291 Acked Read 2 Done Sent Done false
13195 4292 Acked Write 4 Done Sent Done false
13196 4293 Acked Read 4 Done Sent Done false
13197 4294 Acked Read 1 Done Sent Done false
13198 4295 Acked Write 7 Done Sent Done false
13199 4296 Acked Read 8 Done Sent Done false
13200 4297 Acked Read 9 Done Sent Done false
13201 4298 Acked Write 6 Done Sent Done false
13202 4299 Acked Read 5 Done Sent Done false
13203 4300 Acked Write 1 Done Sent Done false
13204 4301 Acked Write 10 Done Sent Done false
13205 4302 Acked Read 2 Done Sent Done false
13206 4303 Acked Write 7 Done Sent Done false
13207 4304 Acked Read 7 Done Sent Done false
13208 4305 Acked Read 1 Done Sent Done false
13209 4306 Acked Read 3 Done Sent Done false
13210 4307 Acked Read 3 Done Sent Done false
13211 4308 Acked Read 9 Done Sent Done false
13212 4309 Acked Write 5 Done Sent Done false
13213 4310 Acked Read 2 Done Sent Done false
13214 4311 Acked Flush 0 Done Sent Done false
13215 4312 Acked Write 4 Done Sent Done false
13216 4313 Acked Write 1 Done Sent Done false
13217 4314 Acked Read 9 Done Sent Done false
13218 4315 Acked Write 9 Done Sent Done false
13219 4316 Acked Read 6 Done Sent Done false
13220 4317 Acked Write 3 Done Sent Done false
13221 STATES DS:0 DS:1 DS:2 TOTAL
13222 Sent 0 301 0 301
13223 Done 301 0 301 602
13224 Skipped 0 0 0 0
13225 Error 0 0 0 0
13226 Last Flush: Some(JobId(4311)) Some(JobId(4016)) Some(JobId(4311))
13227 Downstairs last five completed: 4016 4015 4014 4013 4012
13228 Upstairs last five completed: 4317 4316 4315 4314 4313
13229 CLIENT: Up:0 ds:301 act:2
132302025-02-28T06:03:51.618ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
132312025-02-28T06:03:52.620ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
132322025-02-28T06:03:53.622ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
13233 ----------------------------------------------------------------
13234 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13235 States: Active Offline Active
13236 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13237 4017 Acked Flush 0 Done Sent Done false
13238 4018 Acked Read 10 Done Sent Done false
13239 4019 Acked Read 10 Done Sent Done false
13240 4020 Acked Read 5 Done Sent Done false
13241 4021 Acked Write 6 Done Sent Done false
13242 4022 Acked Write 2 Done Sent Done false
13243 4023 Acked Read 9 Done Sent Done false
13244 4024 Acked Write 8 Done Sent Done false
13245 4025 Acked Write 4 Done Sent Done false
13246 4026 Acked Flush 0 Done Sent Done false
13247 4027 Acked Write 1 Done Sent Done false
13248 4028 Acked Write 6 Done Sent Done false
13249 4029 Acked Flush 0 Done Sent Done false
13250 4030 Acked Read 8 Done Sent Done false
13251 4031 Acked Read 8 Done Sent Done false
13252 4032 Acked Read 3 Done Sent Done false
13253 4033 Acked Write 1 Done Sent Done false
13254 4034 Acked Read 2 Done Sent Done false
13255 4035 Acked Write 1 Done Sent Done false
13256 4036 Acked Write 3 Done Sent Done false
13257 4037 Acked Read 4 Done Sent Done false
13258 4038 Acked Read 5 Done Sent Done false
13259 4039 Acked Write 4 Done Sent Done false
13260 4040 Acked Read 10 Done Sent Done false
13261 4041 Acked Write 4 Done Sent Done false
13262 4042 Acked Write 6 Done Sent Done false
13263 4043 Acked Write 2 Done Sent Done false
13264 4044 Acked Flush 0 Done Sent Done false
13265 4045 Acked Write 4 Done Sent Done false
13266 4046 Acked Read 7 Done Sent Done false
13267 4047 Acked Read 8 Done Sent Done false
13268 4048 Acked Write 8 Done Sent Done false
13269 4049 Acked Write 9 Done Sent Done false
13270 4050 Acked Write 10 Done Sent Done false
13271 4051 Acked Read 8 Done Sent Done false
13272 4052 Acked Read 6 Done Sent Done false
13273 4053 Acked Write 10 Done Sent Done false
13274 4054 Acked Write 2 Done Sent Done false
13275 4055 Acked Write 4 Done Sent Done false
13276 4056 Acked Read 9 Done Sent Done false
13277 4057 Acked Flush 0 Done Sent Done false
13278 4058 Acked Read 9 Done Sent Done false
13279 4059 Acked Write 6 Done Sent Done false
13280 4060 Acked Read 4 Done Sent Done false
13281 4061 Acked Read 2 Done Sent Done false
13282 4062 Acked Read 4 Done Sent Done false
13283 4063 Acked Write 1 Done Sent Done false
13284 4064 Acked Write 4 Done Sent Done false
13285 4065 Acked Read 5 Done Sent Done false
13286 4066 Acked Flush 0 Done Sent Done false
13287 4067 Acked Read 5 Done Sent Done false
13288 4068 Acked Read 3 Done Sent Done false
13289 4069 Acked Read 5 Done Sent Done false
13290 4070 Acked Write 3 Done Sent Done false
13291 4071 Acked Write 8 Done Sent Done false
13292 4072 Acked Write 10 Done Sent Done false
13293 4073 Acked Write 9 Done Sent Done false
13294 4074 Acked Write 9 Done Sent Done false
13295 4075 Acked Read 9 Done Sent Done false
13296 4076 Acked Read 4 Done Sent Done false
13297 4077 Acked Write 3 Done Sent Done false
13298 4078 Acked Write 6 Done Sent Done false
13299 4079 Acked Read 8 Done Sent Done false
13300 4080 Acked Flush 0 Done Sent Done false
13301 4081 Acked Read 10 Done Sent Done false
13302 4082 Acked Flush 0 Done Sent Done false
13303 4083 Acked Read 10 Done Sent Done false
13304 4084 Acked Read 1 Done Sent Done false
13305 4085 Acked Write 2 Done Sent Done false
13306 4086 Acked Read 8 Done Sent Done false
13307 4087 Acked Read 1 Done Sent Done false
13308 4088 Acked Write 8 Done Sent Done false
13309 4089 Acked Flush 0 Done Sent Done false
13310 4090 Acked Write 8 Done Sent Done false
13311 4091 Acked Write 7 Done Sent Done false
13312 4092 Acked Read 2 Done Sent Done false
13313 4093 Acked Write 2 Done Sent Done false
13314 4094 Acked Write 5 Done Sent Done false
13315 4095 Acked Write 6 Done Sent Done false
13316 4096 Acked Flush 0 Done Sent Done false
13317 4097 Acked Write 6 Done Sent Done false
13318 4098 Acked Flush 0 Done Sent Done false
13319 4099 Acked Write 4 Done Sent Done false
13320 4100 Acked Read 6 Done Sent Done false
13321 4101 Acked Read 9 Done Sent Done false
13322 4102 Acked Write 5 Done Sent Done false
13323 4103 Acked Write 5 Done Sent Done false
13324 4104 Acked Write 9 Done Sent Done false
13325 4105 Acked Read 8 Done Sent Done false
13326 4106 Acked Flush 0 Done Sent Done false
13327 4107 Acked Write 6 Done Sent Done false
13328 4108 Acked Read 4 Done Sent Done false
13329 4109 Acked Read 7 Done Sent Done false
13330 4110 Acked Read 10 Done Sent Done false
13331 4111 Acked Read 7 Done Sent Done false
13332 4112 Acked Flush 0 Done Sent Done false
13333 4113 Acked Read 9 Done Sent Done false
13334 4114 Acked Read 2 Done Sent Done false
13335 4115 Acked Write 5 Done Sent Done false
13336 4116 Acked Read 7 Done Sent Done false
13337 4117 Acked Write 5 Done Sent Done false
13338 4118 Acked Flush 0 Done Sent Done false
13339 4119 Acked Read 7 Done Sent Done false
13340 4120 Acked Flush 0 Done Sent Done false
13341 4121 Acked Write 6 Done Sent Done false
13342 4122 Acked Read 2 Done Sent Done false
13343 4123 Acked Read 9 Done Sent Done false
13344 4124 Acked Read 4 Done Sent Done false
13345 4125 Acked Write 2 Done Sent Done false
13346 4126 Acked Read 10 Done Sent Done false
13347 4127 Acked Read 5 Done Sent Done false
13348 4128 Acked Write 2 Done Sent Done false
13349 4129 Acked Read 6 Done Sent Done false
13350 4130 Acked Write 4 Done Sent Done false
13351 4131 Acked Write 3 Done Sent Done false
13352 4132 Acked Read 4 Done Sent Done false
13353 4133 Acked Write 9 Done Sent Done false
13354 4134 Acked Write 8 Done Sent Done false
13355 4135 Acked Write 8 Done Sent Done false
13356 4136 Acked Read 7 Done Sent Done false
13357 4137 Acked Read 2 Done Sent Done false
13358 4138 Acked Read 10 Done Sent Done false
13359 4139 Acked Read 3 Done Sent Done false
13360 4140 Acked Read 3 Done Sent Done false
13361 4141 Acked Write 9 Done Sent Done false
13362 4142 Acked Write 6 Done Sent Done false
13363 4143 Acked Read 9 Done Sent Done false
13364 4144 Acked Read 10 Done Sent Done false
13365 4145 Acked Write 4 Done Sent Done false
13366 4146 Acked Read 6 Done Sent Done false
13367 4147 Acked Read 4 Done Sent Done false
13368 4148 Acked Read 4 Done Sent Done false
13369 4149 Acked Write 5 Done Sent Done false
13370 4150 Acked Write 4 Done Sent Done false
13371 4151 Acked Write 3 Done Sent Done false
13372 4152 Acked Write 9 Done Sent Done false
13373 4153 Acked Flush 0 Done Sent Done false
13374 4154 Acked Read 3 Done Sent Done false
13375 4155 Acked Read 4 Done Sent Done false
13376 4156 Acked Read 4 Done Sent Done false
13377 4157 Acked Read 6 Done Sent Done false
13378 4158 Acked Read 9 Done Sent Done false
13379 4159 Acked Flush 0 Done Sent Done false
13380 4160 Acked Read 5 Done Sent Done false
13381 4161 Acked Read 6 Done Sent Done false
13382 4162 Acked Write 8 Done Sent Done false
13383 4163 Acked Read 2 Done Sent Done false
13384 4164 Acked Read 4 Done Sent Done false
13385 4165 Acked Write 2 Done Sent Done false
13386 4166 Acked Write 7 Done Sent Done false
13387 4167 Acked Write 7 Done Sent Done false
13388 4168 Acked Write 9 Done Sent Done false
13389 4169 Acked Flush 0 Done Sent Done false
13390 4170 Acked Read 8 Done Sent Done false
13391 4171 Acked Read 6 Done Sent Done false
13392 4172 Acked Read 9 Done Sent Done false
13393 4173 Acked Write 9 Done Sent Done false
13394 4174 Acked Read 10 Done Sent Done false
13395 4175 Acked Read 8 Done Sent Done false
13396 4176 Acked Write 4 Done Sent Done false
13397 4177 Acked Write 9 Done Sent Done false
13398 4178 Acked Read 1 Done Sent Done false
13399 4179 Acked Write 5 Done Sent Done false
13400 4180 Acked Read 7 Done Sent Done false
13401 4181 Acked Write 10 Done Sent Done false
13402 4182 Acked Write 10 Done Sent Done false
13403 4183 Acked Write 6 Done Sent Done false
13404 4184 Acked Write 9 Done Sent Done false
13405 4185 Acked Write 7 Done Sent Done false
13406 4186 Acked Write 8 Done Sent Done false
13407 4187 Acked Read 9 Done Sent Done false
13408 4188 Acked Read 9 Done Sent Done false
13409 4189 Acked Write 4 Done Sent Done false
13410 4190 Acked Read 1 Done Sent Done false
13411 4191 Acked Write 10 Done Sent Done false
13412 4192 Acked Write 4 Done Sent Done false
13413 4193 Acked Write 1 Done Sent Done false
13414 4194 Acked Read 3 Done Sent Done false
13415 4195 Acked Write 9 Done Sent Done false
13416 4196 Acked Write 10 Done Sent Done false
13417 4197 Acked Write 4 Done Sent Done false
13418 4198 Acked Write 3 Done Sent Done false
13419 4199 Acked Read 3 Done Sent Done false
13420 4200 Acked Flush 0 Done Sent Done false
13421 4201 Acked Write 6 Done Sent Done false
13422 4202 Acked Write 3 Done Sent Done false
13423 4203 Acked Write 2 Done Sent Done false
13424 4204 Acked Flush 0 Done Sent Done false
13425 4205 Acked Write 8 Done Sent Done false
13426 4206 Acked Write 3 Done Sent Done false
13427 4207 Acked Read 2 Done Sent Done false
13428 4208 Acked Read 8 Done Sent Done false
13429 4209 Acked Write 1 Done Sent Done false
13430 4210 Acked Read 9 Done Sent Done false
13431 4211 Acked Read 10 Done Sent Done false
13432 4212 Acked Write 7 Done Sent Done false
13433 4213 Acked Write 10 Done Sent Done false
13434 4214 Acked Read 3 Done Sent Done false
13435 4215 Acked Read 9 Done Sent Done false
13436 4216 Acked Read 3 Done Sent Done false
13437 4217 Acked Read 5 Done Sent Done false
13438 4218 Acked Write 7 Done Sent Done false
13439 4219 Acked Read 9 Done Sent Done false
13440 4220 Acked Flush 0 Done Sent Done false
13441 4221 Acked Write 1 Done Sent Done false
13442 4222 Acked Read 5 Done Sent Done false
13443 4223 Acked Write 7 Done Sent Done false
13444 4224 Acked Write 5 Done Sent Done false
13445 4225 Acked Read 2 Done Sent Done false
13446 4226 Acked Read 5 Done Sent Done false
13447 4227 Acked Read 6 Done Sent Done false
13448 4228 Acked Read 4 Done Sent Done false
13449 4229 Acked Write 6 Done Sent Done false
13450 4230 Acked Write 3 Done Sent Done false
13451 4231 Acked Read 7 Done Sent Done false
13452 4232 Acked Read 6 Done Sent Done false
13453 4233 Acked Read 8 Done Sent Done false
13454 4234 Acked Read 3 Done Sent Done false
13455 4235 Acked Write 8 Done Sent Done false
13456 4236 Acked Write 10 Done Sent Done false
13457 4237 Acked Write 1 Done Sent Done false
13458 4238 Acked Read 7 Done Sent Done false
13459 4239 Acked Write 10 Done Sent Done false
13460 4240 Acked Read 6 Done Sent Done false
13461 4241 Acked Write 8 Done Sent Done false
13462 4242 Acked Read 10 Done Sent Done false
13463 4243 Acked Write 10 Done Sent Done false
13464 4244 Acked Write 7 Done Sent Done false
13465 4245 Acked Write 6 Done Sent Done false
13466 4246 Acked Flush 0 Done Sent Done false
13467 4247 Acked Read 1 Done Sent Done false
13468 4248 Acked Flush 0 Done Sent Done false
13469 4249 Acked Write 4 Done Sent Done false
13470 4250 Acked Write 2 Done Sent Done false
13471 4251 Acked Read 8 Done Sent Done false
13472 4252 Acked Flush 0 Done Sent Done false
13473 4253 Acked Write 9 Done Sent Done false
13474 4254 Acked Write 6 Done Sent Done false
13475 4255 Acked Read 9 Done Sent Done false
13476 4256 Acked Read 7 Done Sent Done false
13477 4257 Acked Write 2 Done Sent Done false
13478 4258 Acked Write 10 Done Sent Done false
13479 4259 Acked Read 7 Done Sent Done false
13480 4260 Acked Flush 0 Done Sent Done false
13481 4261 Acked Write 6 Done Sent Done false
13482 4262 Acked Read 8 Done Sent Done false
13483 4263 Acked Read 4 Done Sent Done false
13484 4264 Acked Read 3 Done Sent Done false
13485 4265 Acked Write 8 Done Sent Done false
13486 4266 Acked Read 3 Done Sent Done false
13487 4267 Acked Write 2 Done Sent Done false
13488 4268 Acked Read 10 Done Sent Done false
13489 4269 Acked Write 4 Done Sent Done false
13490 4270 Acked Write 8 Done Sent Done false
13491 4271 Acked Read 4 Done Sent Done false
13492 4272 Acked Flush 0 Done Sent Done false
13493 4273 Acked Write 1 Done Sent Done false
13494 4274 Acked Read 4 Done Sent Done false
13495 4275 Acked Read 2 Done Sent Done false
13496 4276 Acked Read 2 Done Sent Done false
13497 4277 Acked Read 1 Done Sent Done false
13498 4278 Acked Read 5 Done Sent Done false
13499 4279 Acked Write 1 Done Sent Done false
13500 4280 Acked Flush 0 Done Sent Done false
13501 4281 Acked Read 1 Done Sent Done false
13502 4282 Acked Write 9 Done Sent Done false
13503 4283 Acked Flush 0 Done Sent Done false
13504 4284 Acked Write 3 Done Sent Done false
13505 4285 Acked Write 6 Done Sent Done false
13506 4286 Acked Read 8 Done Sent Done false
13507 4287 Acked Write 4 Done Sent Done false
13508 4288 Acked Write 8 Done Sent Done false
13509 4289 Acked Read 3 Done Sent Done false
13510 4290 Acked Write 7 Done Sent Done false
13511 4291 Acked Read 2 Done Sent Done false
13512 4292 Acked Write 4 Done Sent Done false
13513 4293 Acked Read 4 Done Sent Done false
13514 4294 Acked Read 1 Done Sent Done false
13515 4295 Acked Write 7 Done Sent Done false
13516 4296 Acked Read 8 Done Sent Done false
13517 4297 Acked Read 9 Done Sent Done false
13518 4298 Acked Write 6 Done Sent Done false
13519 4299 Acked Read 5 Done Sent Done false
13520 4300 Acked Write 1 Done Sent Done false
13521 4301 Acked Write 10 Done Sent Done false
13522 4302 Acked Read 2 Done Sent Done false
13523 4303 Acked Write 7 Done Sent Done false
13524 4304 Acked Read 7 Done Sent Done false
13525 4305 Acked Read 1 Done Sent Done false
13526 4306 Acked Read 3 Done Sent Done false
13527 4307 Acked Read 3 Done Sent Done false
13528 4308 Acked Read 9 Done Sent Done false
13529 4309 Acked Write 5 Done Sent Done false
13530 4310 Acked Read 2 Done Sent Done false
13531 4311 Acked Flush 0 Done Sent Done false
13532 4312 Acked Write 4 Done Sent Done false
13533 4313 Acked Write 1 Done Sent Done false
13534 4314 Acked Read 9 Done Sent Done false
13535 4315 Acked Write 9 Done Sent Done false
13536 4316 Acked Read 6 Done Sent Done false
13537 4317 Acked Write 3 Done Sent Done false
13538 STATES DS:0 DS:1 DS:2 TOTAL
13539 Sent 0 301 0 301
13540 Done 301 0 301 602
13541 Skipped 0 0 0 0
13542 Error 0 0 0 0
13543 Last Flush: Some(JobId(4311)) Some(JobId(4016)) Some(JobId(4311))
13544 Downstairs last five completed: 4016 4015 4014 4013 4012
13545 Upstairs last five completed: 4317 4316 4315 4314 4313
13546 CLIENT: Up:0 ds:301 act:2
135472025-02-28T06:03:55.123ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
135482025-02-28T06:03:55.625ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
135492025-02-28T06:03:56.381ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
135502025-02-28T06:03:56.381ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
135512025-02-28T06:03:56.381ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
135522025-02-28T06:03:56.381ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
135532025-02-28T06:03:56.382ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
135542025-02-28T06:03:56.883ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
135552025-02-28T06:03:57.627ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
13556 ----------------------------------------------------------------
13557 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13558 States: Active Offline Active
13559 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13560 4017 Acked Flush 0 Done Sent Done false
13561 4018 Acked Read 10 Done Sent Done false
13562 4019 Acked Read 10 Done Sent Done false
13563 4020 Acked Read 5 Done Sent Done false
13564 4021 Acked Write 6 Done Sent Done false
13565 4022 Acked Write 2 Done Sent Done false
13566 4023 Acked Read 9 Done Sent Done false
13567 4024 Acked Write 8 Done Sent Done false
13568 4025 Acked Write 4 Done Sent Done false
13569 4026 Acked Flush 0 Done Sent Done false
13570 4027 Acked Write 1 Done Sent Done false
13571 4028 Acked Write 6 Done Sent Done false
13572 4029 Acked Flush 0 Done Sent Done false
13573 4030 Acked Read 8 Done Sent Done false
13574 4031 Acked Read 8 Done Sent Done false
13575 4032 Acked Read 3 Done Sent Done false
13576 4033 Acked Write 1 Done Sent Done false
13577 4034 Acked Read 2 Done Sent Done false
13578 4035 Acked Write 1 Done Sent Done false
13579 4036 Acked Write 3 Done Sent Done false
13580 4037 Acked Read 4 Done Sent Done false
13581 4038 Acked Read 5 Done Sent Done false
13582 4039 Acked Write 4 Done Sent Done false
13583 4040 Acked Read 10 Done Sent Done false
13584 4041 Acked Write 4 Done Sent Done false
13585 4042 Acked Write 6 Done Sent Done false
13586 4043 Acked Write 2 Done Sent Done false
13587 4044 Acked Flush 0 Done Sent Done false
13588 4045 Acked Write 4 Done Sent Done false
13589 4046 Acked Read 7 Done Sent Done false
13590 4047 Acked Read 8 Done Sent Done false
13591 4048 Acked Write 8 Done Sent Done false
13592 4049 Acked Write 9 Done Sent Done false
13593 4050 Acked Write 10 Done Sent Done false
13594 4051 Acked Read 8 Done Sent Done false
13595 4052 Acked Read 6 Done Sent Done false
13596 4053 Acked Write 10 Done Sent Done false
13597 4054 Acked Write 2 Done Sent Done false
13598 4055 Acked Write 4 Done Sent Done false
13599 4056 Acked Read 9 Done Sent Done false
13600 4057 Acked Flush 0 Done Sent Done false
13601 4058 Acked Read 9 Done Sent Done false
13602 4059 Acked Write 6 Done Sent Done false
13603 4060 Acked Read 4 Done Sent Done false
13604 4061 Acked Read 2 Done Sent Done false
13605 4062 Acked Read 4 Done Sent Done false
13606 4063 Acked Write 1 Done Sent Done false
13607 4064 Acked Write 4 Done Sent Done false
13608 4065 Acked Read 5 Done Sent Done false
13609 4066 Acked Flush 0 Done Sent Done false
13610 4067 Acked Read 5 Done Sent Done false
13611 4068 Acked Read 3 Done Sent Done false
13612 4069 Acked Read 5 Done Sent Done false
13613 4070 Acked Write 3 Done Sent Done false
13614 4071 Acked Write 8 Done Sent Done false
13615 4072 Acked Write 10 Done Sent Done false
13616 4073 Acked Write 9 Done Sent Done false
13617 4074 Acked Write 9 Done Sent Done false
13618 4075 Acked Read 9 Done Sent Done false
13619 4076 Acked Read 4 Done Sent Done false
13620 4077 Acked Write 3 Done Sent Done false
13621 4078 Acked Write 6 Done Sent Done false
13622 4079 Acked Read 8 Done Sent Done false
13623 4080 Acked Flush 0 Done Sent Done false
13624 4081 Acked Read 10 Done Sent Done false
13625 4082 Acked Flush 0 Done Sent Done false
13626 4083 Acked Read 10 Done Sent Done false
13627 4084 Acked Read 1 Done Sent Done false
13628 4085 Acked Write 2 Done Sent Done false
13629 4086 Acked Read 8 Done Sent Done false
13630 4087 Acked Read 1 Done Sent Done false
13631 4088 Acked Write 8 Done Sent Done false
13632 4089 Acked Flush 0 Done Sent Done false
13633 4090 Acked Write 8 Done Sent Done false
13634 4091 Acked Write 7 Done Sent Done false
13635 4092 Acked Read 2 Done Sent Done false
13636 4093 Acked Write 2 Done Sent Done false
13637 4094 Acked Write 5 Done Sent Done false
13638 4095 Acked Write 6 Done Sent Done false
13639 4096 Acked Flush 0 Done Sent Done false
13640 4097 Acked Write 6 Done Sent Done false
13641 4098 Acked Flush 0 Done Sent Done false
13642 4099 Acked Write 4 Done Sent Done false
13643 4100 Acked Read 6 Done Sent Done false
13644 4101 Acked Read 9 Done Sent Done false
13645 4102 Acked Write 5 Done Sent Done false
13646 4103 Acked Write 5 Done Sent Done false
13647 4104 Acked Write 9 Done Sent Done false
13648 4105 Acked Read 8 Done Sent Done false
13649 4106 Acked Flush 0 Done Sent Done false
13650 4107 Acked Write 6 Done Sent Done false
13651 4108 Acked Read 4 Done Sent Done false
13652 4109 Acked Read 7 Done Sent Done false
13653 4110 Acked Read 10 Done Sent Done false
13654 4111 Acked Read 7 Done Sent Done false
13655 4112 Acked Flush 0 Done Sent Done false
13656 4113 Acked Read 9 Done Sent Done false
13657 4114 Acked Read 2 Done Sent Done false
13658 4115 Acked Write 5 Done Sent Done false
13659 4116 Acked Read 7 Done Sent Done false
13660 4117 Acked Write 5 Done Sent Done false
13661 4118 Acked Flush 0 Done Sent Done false
13662 4119 Acked Read 7 Done Sent Done false
13663 4120 Acked Flush 0 Done Sent Done false
13664 4121 Acked Write 6 Done Sent Done false
13665 4122 Acked Read 2 Done Sent Done false
13666 4123 Acked Read 9 Done Sent Done false
13667 4124 Acked Read 4 Done Sent Done false
13668 4125 Acked Write 2 Done Sent Done false
13669 4126 Acked Read 10 Done Sent Done false
13670 4127 Acked Read 5 Done Sent Done false
13671 4128 Acked Write 2 Done Sent Done false
13672 4129 Acked Read 6 Done Sent Done false
13673 4130 Acked Write 4 Done Sent Done false
13674 4131 Acked Write 3 Done Sent Done false
13675 4132 Acked Read 4 Done Sent Done false
13676 4133 Acked Write 9 Done Sent Done false
13677 4134 Acked Write 8 Done Sent Done false
13678 4135 Acked Write 8 Done Sent Done false
13679 4136 Acked Read 7 Done Sent Done false
13680 4137 Acked Read 2 Done Sent Done false
13681 4138 Acked Read 10 Done Sent Done false
13682 4139 Acked Read 3 Done Sent Done false
13683 4140 Acked Read 3 Done Sent Done false
13684 4141 Acked Write 9 Done Sent Done false
13685 4142 Acked Write 6 Done Sent Done false
13686 4143 Acked Read 9 Done Sent Done false
13687 4144 Acked Read 10 Done Sent Done false
13688 4145 Acked Write 4 Done Sent Done false
13689 4146 Acked Read 6 Done Sent Done false
13690 4147 Acked Read 4 Done Sent Done false
13691 4148 Acked Read 4 Done Sent Done false
13692 4149 Acked Write 5 Done Sent Done false
13693 4150 Acked Write 4 Done Sent Done false
13694 4151 Acked Write 3 Done Sent Done false
13695 4152 Acked Write 9 Done Sent Done false
13696 4153 Acked Flush 0 Done Sent Done false
13697 4154 Acked Read 3 Done Sent Done false
13698 4155 Acked Read 4 Done Sent Done false
13699 4156 Acked Read 4 Done Sent Done false
13700 4157 Acked Read 6 Done Sent Done false
13701 4158 Acked Read 9 Done Sent Done false
13702 4159 Acked Flush 0 Done Sent Done false
13703 4160 Acked Read 5 Done Sent Done false
13704 4161 Acked Read 6 Done Sent Done false
13705 4162 Acked Write 8 Done Sent Done false
13706 4163 Acked Read 2 Done Sent Done false
13707 4164 Acked Read 4 Done Sent Done false
13708 4165 Acked Write 2 Done Sent Done false
13709 4166 Acked Write 7 Done Sent Done false
13710 4167 Acked Write 7 Done Sent Done false
13711 4168 Acked Write 9 Done Sent Done false
13712 4169 Acked Flush 0 Done Sent Done false
13713 4170 Acked Read 8 Done Sent Done false
13714 4171 Acked Read 6 Done Sent Done false
13715 4172 Acked Read 9 Done Sent Done false
13716 4173 Acked Write 9 Done Sent Done false
13717 4174 Acked Read 10 Done Sent Done false
13718 4175 Acked Read 8 Done Sent Done false
13719 4176 Acked Write 4 Done Sent Done false
13720 4177 Acked Write 9 Done Sent Done false
13721 4178 Acked Read 1 Done Sent Done false
13722 4179 Acked Write 5 Done Sent Done false
13723 4180 Acked Read 7 Done Sent Done false
13724 4181 Acked Write 10 Done Sent Done false
13725 4182 Acked Write 10 Done Sent Done false
13726 4183 Acked Write 6 Done Sent Done false
13727 4184 Acked Write 9 Done Sent Done false
13728 4185 Acked Write 7 Done Sent Done false
13729 4186 Acked Write 8 Done Sent Done false
13730 4187 Acked Read 9 Done Sent Done false
13731 4188 Acked Read 9 Done Sent Done false
13732 4189 Acked Write 4 Done Sent Done false
13733 4190 Acked Read 1 Done Sent Done false
13734 4191 Acked Write 10 Done Sent Done false
13735 4192 Acked Write 4 Done Sent Done false
13736 4193 Acked Write 1 Done Sent Done false
13737 4194 Acked Read 3 Done Sent Done false
13738 4195 Acked Write 9 Done Sent Done false
13739 4196 Acked Write 10 Done Sent Done false
13740 4197 Acked Write 4 Done Sent Done false
13741 4198 Acked Write 3 Done Sent Done false
13742 4199 Acked Read 3 Done Sent Done false
13743 4200 Acked Flush 0 Done Sent Done false
13744 4201 Acked Write 6 Done Sent Done false
13745 4202 Acked Write 3 Done Sent Done false
13746 4203 Acked Write 2 Done Sent Done false
13747 4204 Acked Flush 0 Done Sent Done false
13748 4205 Acked Write 8 Done Sent Done false
13749 4206 Acked Write 3 Done Sent Done false
13750 4207 Acked Read 2 Done Sent Done false
13751 4208 Acked Read 8 Done Sent Done false
13752 4209 Acked Write 1 Done Sent Done false
13753 4210 Acked Read 9 Done Sent Done false
13754 4211 Acked Read 10 Done Sent Done false
13755 4212 Acked Write 7 Done Sent Done false
13756 4213 Acked Write 10 Done Sent Done false
13757 4214 Acked Read 3 Done Sent Done false
13758 4215 Acked Read 9 Done Sent Done false
13759 4216 Acked Read 3 Done Sent Done false
13760 4217 Acked Read 5 Done Sent Done false
13761 4218 Acked Write 7 Done Sent Done false
13762 4219 Acked Read 9 Done Sent Done false
13763 4220 Acked Flush 0 Done Sent Done false
13764 4221 Acked Write 1 Done Sent Done false
13765 4222 Acked Read 5 Done Sent Done false
13766 4223 Acked Write 7 Done Sent Done false
13767 4224 Acked Write 5 Done Sent Done false
13768 4225 Acked Read 2 Done Sent Done false
13769 4226 Acked Read 5 Done Sent Done false
13770 4227 Acked Read 6 Done Sent Done false
13771 4228 Acked Read 4 Done Sent Done false
13772 4229 Acked Write 6 Done Sent Done false
13773 4230 Acked Write 3 Done Sent Done false
13774 4231 Acked Read 7 Done Sent Done false
13775 4232 Acked Read 6 Done Sent Done false
13776 4233 Acked Read 8 Done Sent Done false
13777 4234 Acked Read 3 Done Sent Done false
13778 4235 Acked Write 8 Done Sent Done false
13779 4236 Acked Write 10 Done Sent Done false
13780 4237 Acked Write 1 Done Sent Done false
13781 4238 Acked Read 7 Done Sent Done false
13782 4239 Acked Write 10 Done Sent Done false
13783 4240 Acked Read 6 Done Sent Done false
13784 4241 Acked Write 8 Done Sent Done false
13785 4242 Acked Read 10 Done Sent Done false
13786 4243 Acked Write 10 Done Sent Done false
13787 4244 Acked Write 7 Done Sent Done false
13788 4245 Acked Write 6 Done Sent Done false
13789 4246 Acked Flush 0 Done Sent Done false
13790 4247 Acked Read 1 Done Sent Done false
13791 4248 Acked Flush 0 Done Sent Done false
13792 4249 Acked Write 4 Done Sent Done false
13793 4250 Acked Write 2 Done Sent Done false
13794 4251 Acked Read 8 Done Sent Done false
13795 4252 Acked Flush 0 Done Sent Done false
13796 4253 Acked Write 9 Done Sent Done false
13797 4254 Acked Write 6 Done Sent Done false
13798 4255 Acked Read 9 Done Sent Done false
13799 4256 Acked Read 7 Done Sent Done false
13800 4257 Acked Write 2 Done Sent Done false
13801 4258 Acked Write 10 Done Sent Done false
13802 4259 Acked Read 7 Done Sent Done false
13803 4260 Acked Flush 0 Done Sent Done false
13804 4261 Acked Write 6 Done Sent Done false
13805 4262 Acked Read 8 Done Sent Done false
13806 4263 Acked Read 4 Done Sent Done false
13807 4264 Acked Read 3 Done Sent Done false
13808 4265 Acked Write 8 Done Sent Done false
13809 4266 Acked Read 3 Done Sent Done false
13810 4267 Acked Write 2 Done Sent Done false
13811 4268 Acked Read 10 Done Sent Done false
13812 4269 Acked Write 4 Done Sent Done false
13813 4270 Acked Write 8 Done Sent Done false
13814 4271 Acked Read 4 Done Sent Done false
13815 4272 Acked Flush 0 Done Sent Done false
13816 4273 Acked Write 1 Done Sent Done false
13817 4274 Acked Read 4 Done Sent Done false
13818 4275 Acked Read 2 Done Sent Done false
13819 4276 Acked Read 2 Done Sent Done false
13820 4277 Acked Read 1 Done Sent Done false
13821 4278 Acked Read 5 Done Sent Done false
13822 4279 Acked Write 1 Done Sent Done false
13823 4280 Acked Flush 0 Done Sent Done false
13824 4281 Acked Read 1 Done Sent Done false
13825 4282 Acked Write 9 Done Sent Done false
13826 4283 Acked Flush 0 Done Sent Done false
13827 4284 Acked Write 3 Done Sent Done false
13828 4285 Acked Write 6 Done Sent Done false
13829 4286 Acked Read 8 Done Sent Done false
13830 4287 Acked Write 4 Done Sent Done false
13831 4288 Acked Write 8 Done Sent Done false
13832 4289 Acked Read 3 Done Sent Done false
13833 4290 Acked Write 7 Done Sent Done false
13834 4291 Acked Read 2 Done Sent Done false
13835 4292 Acked Write 4 Done Sent Done false
13836 4293 Acked Read 4 Done Sent Done false
13837 4294 Acked Read 1 Done Sent Done false
13838 4295 Acked Write 7 Done Sent Done false
13839 4296 Acked Read 8 Done Sent Done false
13840 4297 Acked Read 9 Done Sent Done false
13841 4298 Acked Write 6 Done Sent Done false
13842 4299 Acked Read 5 Done Sent Done false
13843 4300 Acked Write 1 Done Sent Done false
13844 4301 Acked Write 10 Done Sent Done false
13845 4302 Acked Read 2 Done Sent Done false
13846 4303 Acked Write 7 Done Sent Done false
13847 4304 Acked Read 7 Done Sent Done false
13848 4305 Acked Read 1 Done Sent Done false
13849 4306 Acked Read 3 Done Sent Done false
13850 4307 Acked Read 3 Done Sent Done false
13851 4308 Acked Read 9 Done Sent Done false
13852 4309 Acked Write 5 Done Sent Done false
13853 4310 Acked Read 2 Done Sent Done false
13854 4311 Acked Flush 0 Done Sent Done false
13855 4312 Acked Write 4 Done Sent Done false
13856 4313 Acked Write 1 Done Sent Done false
13857 4314 Acked Read 9 Done Sent Done false
13858 4315 Acked Write 9 Done Sent Done false
13859 4316 Acked Read 6 Done Sent Done false
13860 4317 Acked Write 3 Done Sent Done false
13861 STATES DS:0 DS:1 DS:2 TOTAL
13862 Sent 0 301 0 301
13863 Done 301 0 301 602
13864 Skipped 0 0 0 0
13865 Error 0 0 0 0
13866 Last Flush: Some(JobId(4311)) Some(JobId(4016)) Some(JobId(4311))
13867 Downstairs last five completed: 4016 4015 4014 4013 4012
13868 Upstairs last five completed: 4317 4316 4315 4314 4313
13869 CLIENT: Up:0 ds:301 act:2
138702025-02-28T06:03:59.125ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
138712025-02-28T06:03:59.637ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
138722025-02-28T06:04:00.633ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
138732025-02-28T06:04:01.633ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
13874 ----------------------------------------------------------------
13875 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13876 States: Active Offline Active
13877 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13878 4017 Acked Flush 0 Done Sent Done false
13879 4018 Acked Read 10 Done Sent Done false
13880 4019 Acked Read 10 Done Sent Done false
13881 4020 Acked Read 5 Done Sent Done false
13882 4021 Acked Write 6 Done Sent Done false
13883 4022 Acked Write 2 Done Sent Done false
13884 4023 Acked Read 9 Done Sent Done false
13885 4024 Acked Write 8 Done Sent Done false
13886 4025 Acked Write 4 Done Sent Done false
13887 4026 Acked Flush 0 Done Sent Done false
13888 4027 Acked Write 1 Done Sent Done false
13889 4028 Acked Write 6 Done Sent Done false
13890 4029 Acked Flush 0 Done Sent Done false
13891 4030 Acked Read 8 Done Sent Done false
13892 4031 Acked Read 8 Done Sent Done false
13893 4032 Acked Read 3 Done Sent Done false
13894 4033 Acked Write 1 Done Sent Done false
13895 4034 Acked Read 2 Done Sent Done false
13896 4035 Acked Write 1 Done Sent Done false
13897 4036 Acked Write 3 Done Sent Done false
13898 4037 Acked Read 4 Done Sent Done false
13899 4038 Acked Read 5 Done Sent Done false
13900 4039 Acked Write 4 Done Sent Done false
13901 4040 Acked Read 10 Done Sent Done false
13902 4041 Acked Write 4 Done Sent Done false
13903 4042 Acked Write 6 Done Sent Done false
13904 4043 Acked Write 2 Done Sent Done false
13905 4044 Acked Flush 0 Done Sent Done false
13906 4045 Acked Write 4 Done Sent Done false
13907 4046 Acked Read 7 Done Sent Done false
13908 4047 Acked Read 8 Done Sent Done false
13909 4048 Acked Write 8 Done Sent Done false
13910 4049 Acked Write 9 Done Sent Done false
13911 4050 Acked Write 10 Done Sent Done false
13912 4051 Acked Read 8 Done Sent Done false
13913 4052 Acked Read 6 Done Sent Done false
13914 4053 Acked Write 10 Done Sent Done false
13915 4054 Acked Write 2 Done Sent Done false
13916 4055 Acked Write 4 Done Sent Done false
13917 4056 Acked Read 9 Done Sent Done false
13918 4057 Acked Flush 0 Done Sent Done false
13919 4058 Acked Read 9 Done Sent Done false
13920 4059 Acked Write 6 Done Sent Done false
13921 4060 Acked Read 4 Done Sent Done false
13922 4061 Acked Read 2 Done Sent Done false
13923 4062 Acked Read 4 Done Sent Done false
13924 4063 Acked Write 1 Done Sent Done false
13925 4064 Acked Write 4 Done Sent Done false
13926 4065 Acked Read 5 Done Sent Done false
13927 4066 Acked Flush 0 Done Sent Done false
13928 4067 Acked Read 5 Done Sent Done false
13929 4068 Acked Read 3 Done Sent Done false
13930 4069 Acked Read 5 Done Sent Done false
13931 4070 Acked Write 3 Done Sent Done false
13932 4071 Acked Write 8 Done Sent Done false
13933 4072 Acked Write 10 Done Sent Done false
13934 4073 Acked Write 9 Done Sent Done false
13935 4074 Acked Write 9 Done Sent Done false
13936 4075 Acked Read 9 Done Sent Done false
13937 4076 Acked Read 4 Done Sent Done false
13938 4077 Acked Write 3 Done Sent Done false
13939 4078 Acked Write 6 Done Sent Done false
13940 4079 Acked Read 8 Done Sent Done false
13941 4080 Acked Flush 0 Done Sent Done false
13942 4081 Acked Read 10 Done Sent Done false
13943 4082 Acked Flush 0 Done Sent Done false
13944 4083 Acked Read 10 Done Sent Done false
13945 4084 Acked Read 1 Done Sent Done false
13946 4085 Acked Write 2 Done Sent Done false
13947 4086 Acked Read 8 Done Sent Done false
13948 4087 Acked Read 1 Done Sent Done false
13949 4088 Acked Write 8 Done Sent Done false
13950 4089 Acked Flush 0 Done Sent Done false
13951 4090 Acked Write 8 Done Sent Done false
13952 4091 Acked Write 7 Done Sent Done false
13953 4092 Acked Read 2 Done Sent Done false
13954 4093 Acked Write 2 Done Sent Done false
13955 4094 Acked Write 5 Done Sent Done false
13956 4095 Acked Write 6 Done Sent Done false
13957 4096 Acked Flush 0 Done Sent Done false
13958 4097 Acked Write 6 Done Sent Done false
13959 4098 Acked Flush 0 Done Sent Done false
13960 4099 Acked Write 4 Done Sent Done false
13961 4100 Acked Read 6 Done Sent Done false
13962 4101 Acked Read 9 Done Sent Done false
13963 4102 Acked Write 5 Done Sent Done false
13964 4103 Acked Write 5 Done Sent Done false
13965 4104 Acked Write 9 Done Sent Done false
13966 4105 Acked Read 8 Done Sent Done false
13967 4106 Acked Flush 0 Done Sent Done false
13968 4107 Acked Write 6 Done Sent Done false
13969 4108 Acked Read 4 Done Sent Done false
13970 4109 Acked Read 7 Done Sent Done false
13971 4110 Acked Read 10 Done Sent Done false
13972 4111 Acked Read 7 Done Sent Done false
13973 4112 Acked Flush 0 Done Sent Done false
13974 4113 Acked Read 9 Done Sent Done false
13975 4114 Acked Read 2 Done Sent Done false
13976 4115 Acked Write 5 Done Sent Done false
13977 4116 Acked Read 7 Done Sent Done false
13978 4117 Acked Write 5 Done Sent Done false
13979 4118 Acked Flush 0 Done Sent Done false
13980 4119 Acked Read 7 Done Sent Done false
13981 4120 Acked Flush 0 Done Sent Done false
13982 4121 Acked Write 6 Done Sent Done false
13983 4122 Acked Read 2 Done Sent Done false
13984 4123 Acked Read 9 Done Sent Done false
13985 4124 Acked Read 4 Done Sent Done false
13986 4125 Acked Write 2 Done Sent Done false
13987 4126 Acked Read 10 Done Sent Done false
13988 4127 Acked Read 5 Done Sent Done false
13989 4128 Acked Write 2 Done Sent Done false
13990 4129 Acked Read 6 Done Sent Done false
13991 4130 Acked Write 4 Done Sent Done false
13992 4131 Acked Write 3 Done Sent Done false
13993 4132 Acked Read 4 Done Sent Done false
13994 4133 Acked Write 9 Done Sent Done false
13995 4134 Acked Write 8 Done Sent Done false
13996 4135 Acked Write 8 Done Sent Done false
13997 4136 Acked Read 7 Done Sent Done false
13998 4137 Acked Read 2 Done Sent Done false
13999 4138 Acked Read 10 Done Sent Done false
14000 4139 Acked Read 3 Done Sent Done false
14001 4140 Acked Read 3 Done Sent Done false
14002 4141 Acked Write 9 Done Sent Done false
14003 4142 Acked Write 6 Done Sent Done false
14004 4143 Acked Read 9 Done Sent Done false
14005 4144 Acked Read 10 Done Sent Done false
14006 4145 Acked Write 4 Done Sent Done false
14007 4146 Acked Read 6 Done Sent Done false
14008 4147 Acked Read 4 Done Sent Done false
14009 4148 Acked Read 4 Done Sent Done false
14010 4149 Acked Write 5 Done Sent Done false
14011 4150 Acked Write 4 Done Sent Done false
14012 4151 Acked Write 3 Done Sent Done false
14013 4152 Acked Write 9 Done Sent Done false
14014 4153 Acked Flush 0 Done Sent Done false
14015 4154 Acked Read 3 Done Sent Done false
14016 4155 Acked Read 4 Done Sent Done false
14017 4156 Acked Read 4 Done Sent Done false
14018 4157 Acked Read 6 Done Sent Done false
14019 4158 Acked Read 9 Done Sent Done false
14020 4159 Acked Flush 0 Done Sent Done false
14021 4160 Acked Read 5 Done Sent Done false
14022 4161 Acked Read 6 Done Sent Done false
14023 4162 Acked Write 8 Done Sent Done false
14024 4163 Acked Read 2 Done Sent Done false
14025 4164 Acked Read 4 Done Sent Done false
14026 4165 Acked Write 2 Done Sent Done false
14027 4166 Acked Write 7 Done Sent Done false
14028 4167 Acked Write 7 Done Sent Done false
14029 4168 Acked Write 9 Done Sent Done false
14030 4169 Acked Flush 0 Done Sent Done false
14031 4170 Acked Read 8 Done Sent Done false
14032 4171 Acked Read 6 Done Sent Done false
14033 4172 Acked Read 9 Done Sent Done false
14034 4173 Acked Write 9 Done Sent Done false
14035 4174 Acked Read 10 Done Sent Done false
14036 4175 Acked Read 8 Done Sent Done false
14037 4176 Acked Write 4 Done Sent Done false
14038 4177 Acked Write 9 Done Sent Done false
14039 4178 Acked Read 1 Done Sent Done false
14040 4179 Acked Write 5 Done Sent Done false
14041 4180 Acked Read 7 Done Sent Done false
14042 4181 Acked Write 10 Done Sent Done false
14043 4182 Acked Write 10 Done Sent Done false
14044 4183 Acked Write 6 Done Sent Done false
14045 4184 Acked Write 9 Done Sent Done false
14046 4185 Acked Write 7 Done Sent Done false
14047 4186 Acked Write 8 Done Sent Done false
14048 4187 Acked Read 9 Done Sent Done false
14049 4188 Acked Read 9 Done Sent Done false
14050 4189 Acked Write 4 Done Sent Done false
14051 4190 Acked Read 1 Done Sent Done false
14052 4191 Acked Write 10 Done Sent Done false
14053 4192 Acked Write 4 Done Sent Done false
14054 4193 Acked Write 1 Done Sent Done false
14055 4194 Acked Read 3 Done Sent Done false
14056 4195 Acked Write 9 Done Sent Done false
14057 4196 Acked Write 10 Done Sent Done false
14058 4197 Acked Write 4 Done Sent Done false
14059 4198 Acked Write 3 Done Sent Done false
14060 4199 Acked Read 3 Done Sent Done false
14061 4200 Acked Flush 0 Done Sent Done false
14062 4201 Acked Write 6 Done Sent Done false
14063 4202 Acked Write 3 Done Sent Done false
14064 4203 Acked Write 2 Done Sent Done false
14065 4204 Acked Flush 0 Done Sent Done false
14066 4205 Acked Write 8 Done Sent Done false
14067 4206 Acked Write 3 Done Sent Done false
14068 4207 Acked Read 2 Done Sent Done false
14069 4208 Acked Read 8 Done Sent Done false
14070 4209 Acked Write 1 Done Sent Done false
14071 4210 Acked Read 9 Done Sent Done false
14072 4211 Acked Read 10 Done Sent Done false
14073 4212 Acked Write 7 Done Sent Done false
14074 4213 Acked Write 10 Done Sent Done false
14075 4214 Acked Read 3 Done Sent Done false
14076 4215 Acked Read 9 Done Sent Done false
14077 4216 Acked Read 3 Done Sent Done false
14078 4217 Acked Read 5 Done Sent Done false
14079 4218 Acked Write 7 Done Sent Done false
14080 4219 Acked Read 9 Done Sent Done false
14081 4220 Acked Flush 0 Done Sent Done false
14082 4221 Acked Write 1 Done Sent Done false
14083 4222 Acked Read 5 Done Sent Done false
14084 4223 Acked Write 7 Done Sent Done false
14085 4224 Acked Write 5 Done Sent Done false
14086 4225 Acked Read 2 Done Sent Done false
14087 4226 Acked Read 5 Done Sent Done false
14088 4227 Acked Read 6 Done Sent Done false
14089 4228 Acked Read 4 Done Sent Done false
14090 4229 Acked Write 6 Done Sent Done false
14091 4230 Acked Write 3 Done Sent Done false
14092 4231 Acked Read 7 Done Sent Done false
14093 4232 Acked Read 6 Done Sent Done false
14094 4233 Acked Read 8 Done Sent Done false
14095 4234 Acked Read 3 Done Sent Done false
14096 4235 Acked Write 8 Done Sent Done false
14097 4236 Acked Write 10 Done Sent Done false
14098 4237 Acked Write 1 Done Sent Done false
14099 4238 Acked Read 7 Done Sent Done false
14100 4239 Acked Write 10 Done Sent Done false
14101 4240 Acked Read 6 Done Sent Done false
14102 4241 Acked Write 8 Done Sent Done false
14103 4242 Acked Read 10 Done Sent Done false
14104 4243 Acked Write 10 Done Sent Done false
14105 4244 Acked Write 7 Done Sent Done false
14106 4245 Acked Write 6 Done Sent Done false
14107 4246 Acked Flush 0 Done Sent Done false
14108 4247 Acked Read 1 Done Sent Done false
14109 4248 Acked Flush 0 Done Sent Done false
14110 4249 Acked Write 4 Done Sent Done false
14111 4250 Acked Write 2 Done Sent Done false
14112 4251 Acked Read 8 Done Sent Done false
14113 4252 Acked Flush 0 Done Sent Done false
14114 4253 Acked Write 9 Done Sent Done false
14115 4254 Acked Write 6 Done Sent Done false
14116 4255 Acked Read 9 Done Sent Done false
14117 4256 Acked Read 7 Done Sent Done false
14118 4257 Acked Write 2 Done Sent Done false
14119 4258 Acked Write 10 Done Sent Done false
14120 4259 Acked Read 7 Done Sent Done false
14121 4260 Acked Flush 0 Done Sent Done false
14122 4261 Acked Write 6 Done Sent Done false
14123 4262 Acked Read 8 Done Sent Done false
14124 4263 Acked Read 4 Done Sent Done false
14125 4264 Acked Read 3 Done Sent Done false
14126 4265 Acked Write 8 Done Sent Done false
14127 4266 Acked Read 3 Done Sent Done false
14128 4267 Acked Write 2 Done Sent Done false
14129 4268 Acked Read 10 Done Sent Done false
14130 4269 Acked Write 4 Done Sent Done false
14131 4270 Acked Write 8 Done Sent Done false
14132 4271 Acked Read 4 Done Sent Done false
14133 4272 Acked Flush 0 Done Sent Done false
14134 4273 Acked Write 1 Done Sent Done false
14135 4274 Acked Read 4 Done Sent Done false
14136 4275 Acked Read 2 Done Sent Done false
14137 4276 Acked Read 2 Done Sent Done false
14138 4277 Acked Read 1 Done Sent Done false
14139 4278 Acked Read 5 Done Sent Done false
14140 4279 Acked Write 1 Done Sent Done false
14141 4280 Acked Flush 0 Done Sent Done false
14142 4281 Acked Read 1 Done Sent Done false
14143 4282 Acked Write 9 Done Sent Done false
14144 4283 Acked Flush 0 Done Sent Done false
14145 4284 Acked Write 3 Done Sent Done false
14146 4285 Acked Write 6 Done Sent Done false
14147 4286 Acked Read 8 Done Sent Done false
14148 4287 Acked Write 4 Done Sent Done false
14149 4288 Acked Write 8 Done Sent Done false
14150 4289 Acked Read 3 Done Sent Done false
14151 4290 Acked Write 7 Done Sent Done false
14152 4291 Acked Read 2 Done Sent Done false
14153 4292 Acked Write 4 Done Sent Done false
14154 4293 Acked Read 4 Done Sent Done false
14155 4294 Acked Read 1 Done Sent Done false
14156 4295 Acked Write 7 Done Sent Done false
14157 4296 Acked Read 8 Done Sent Done false
14158 4297 Acked Read 9 Done Sent Done false
14159 4298 Acked Write 6 Done Sent Done false
14160 4299 Acked Read 5 Done Sent Done false
14161 4300 Acked Write 1 Done Sent Done false
14162 4301 Acked Write 10 Done Sent Done false
14163 4302 Acked Read 2 Done Sent Done false
14164 4303 Acked Write 7 Done Sent Done false
14165 4304 Acked Read 7 Done Sent Done false
14166 4305 Acked Read 1 Done Sent Done false
14167 4306 Acked Read 3 Done Sent Done false
14168 4307 Acked Read 3 Done Sent Done false
14169 4308 Acked Read 9 Done Sent Done false
14170 4309 Acked Write 5 Done Sent Done false
14171 4310 Acked Read 2 Done Sent Done false
14172 4311 Acked Flush 0 Done Sent Done false
14173 4312 Acked Write 4 Done Sent Done false
14174 4313 Acked Write 1 Done Sent Done false
14175 4314 Acked Read 9 Done Sent Done false
14176 4315 Acked Write 9 Done Sent Done false
14177 4316 Acked Read 6 Done Sent Done false
14178 4317 Acked Write 3 Done Sent Done false
14179 STATES DS:0 DS:1 DS:2 TOTAL
14180 Sent 0 301 0 301
14181 Done 301 0 301 602
14182 Skipped 0 0 0 0
14183 Error 0 0 0 0
14184 Last Flush: Some(JobId(4311)) Some(JobId(4016)) Some(JobId(4311))
14185 Downstairs last five completed: 4016 4015 4014 4013 4012
14186 Upstairs last five completed: 4317 4316 4315 4314 4313
14187 CLIENT: Up:0 ds:301 act:2
141882025-02-28T06:04:03.126ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
141892025-02-28T06:04:03.638ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
14190 [8820] got start action
14191 [8820] I am going to restart
14192 Starting downstairs at port 8820
14193 Make output file at "/tmp/dsc/downstairs-8820.txt"
14194 [8820][1] reports Starting
14195 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1222)
14196 [8820][1] reports Running
141972025-02-28T06:04:04.639ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
141982025-02-28T06:04:05.640ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
141992025-02-28T06:04:06.383ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
142002025-02-28T06:04:06.383ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56219) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
142012025-02-28T06:04:06.384ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
142022025-02-28T06:04:06.384ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
142032025-02-28T06:04:06.384ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
142042025-02-28T06:04:06.384ZINFOcrucible: send last flush ID to this DS: Some(JobId(4016)) = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
142052025-02-28T06:04:06.384ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
142062025-02-28T06:04:06.384ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4016)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
142072025-02-28T06:04:06.385ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4016)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
14208 ----------------------------------------------------------------
14209 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:6
14210 States: Active Active Active
14211 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14212 4312 Acked Write 4 Done Done Done true
14213 4313 Acked Write 1 Done Done Done true
14214 4314 Acked Read 9 Done Done Done true
14215 4315 Acked Write 9 Done Done Done true
14216 4316 Acked Read 6 Done Done Done true
14217 4317 Acked Write 3 Done Done Done true
14218 STATES DS:0 DS:1 DS:2 TOTAL
14219 Sent 0 0 0 0
14220 Done 6 6 6 18
14221 Skipped 0 0 0 0
14222 Error 0 0 0 0
14223 Last Flush: Some(JobId(4311)) Some(JobId(4311)) Some(JobId(4311))
14224 Downstairs last five completed: 4311 4310 4309 4308 4307
14225 Upstairs last five completed: 4317 4316 4315 4314 4313
14226 CLIENT: Up:0 ds:6 act:3
14227 ----------------------------------------------------------------
14228 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14229 Downstairs last five completed: 4318 4317 4316 4315 4314
14230 Upstairs last five completed: 4318 4317 4316 4315 4314
14231 CLIENT: Up:0 ds:0 act:3
14232 Replay: All jobs finished
14233 Main task has work to do, go find it
14234 got dsc Stop(2)
14235 stop 2
14236 [8830] Got stop action so:false kr:false
14237 Feb 28 06:04:10.628 INFO request completed, latency_us: 319, response_code: 204, uri: /stop/cid/2, method: POST, req_id: bf66f5b6-de9a-45f4-8a5d-1c1ae13f5b98, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
142382025-02-28T06:04:10.629ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
14239 Feb 28 06:04:10.629{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:10.629526365Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
14240 INFO request completed, latency_us: 110, response_code: 200, uri: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:10.629553348Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
14241 /state/cid/2, method: GET, req_id: 4a253fc8-e9f5-4d8d-b4dd-e9a2e3cb517e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:04:10.629618513Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
14242 
14243 [8830] Exited with: ExitStatus(unix_wait_status(9))
14244 [8830][2] reports Exit
14245 Feb 28 06:04:14.631 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/2, method: GET, req_id: 58297db9-d4e8-4498-81fb-1d89f50af94a, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
14246 001/300 Write block 3548 len 28672 data: 3 3 3 3 2 2 2
14247 002/300 Write block 113 len 12288 data: 2 2 3
14248 003/300 Read block 3582 len 8192
14249 004/300 Flush
14250 005/300 Read block 2009 len 8192
14251 006/300 Write block 3850 len 12288 data: 5 5 6
14252 007/300 Write block 382 len 8192 data: 6 6
14253 008/300 Read block 2819 len 40960
14254 009/300 Read block 3979 len 32768
14255 010/300 Read block 4224 len 8192
14256 011/300 Write block 704 len 32768 data: 6 5 5 5 4 5 5 4
14257 012/300 Flush
14258 013/300 Read block 1091 len 12288
14259 014/300 Read block 2365 len 20480
14260 015/300 Read block 652 len 16384
14261 016/300 Read block 2068 len 20480
14262 017/300 Read block 3314 len 24576
14263 018/300 Write block 2769 len 16384 data: 2 2 2 2
14264 019/300 Write block 2002 len 24576 data: 3 3 2 2 3 3
14265 020/300 Read block 4924 len 16384
14266 021/300 Write block 2311 len 32768 data: 3 3 3 3 2 3 4 5
14267 022/300 Read block 1728 len 16384
14268 023/300 Read block 2924 len 4096
14269 024/300 Read block 3983 len 16384
14270 025/300 Write block 2298 len 16384 data: 3 2 2 2
14271 026/300 Flush
14272 027/300 Write block 854 len 16384 data: 3 3 2 3
14273 028/300 Write block 287 len 36864 data: 3 3 3 3 3 4 3 3 4
14274 029/300 Read block 207 len 4096
14275 030/300 Read block 1732 len 28672
14276 031/300 Read block 624 len 12288
14277 032/300 Flush
14278 033/300 Read block 1858 len 4096
14279 034/300 Read block 9 len 40960
14280 035/300 Read block 391 len 36864
14281 036/300 Flush
14282 037/300 Read block 4511 len 12288
14283 038/300 Read block 2259 len 8192
14284 039/300 Read block 1485 len 40960
14285 040/300 Flush
14286 041/300 Read block 2826 len 32768
14287 042/300 Write block 3015 len 12288 data: 3 2 2
14288 043/300 Write block 873 len 4096 data: 4
14289 044/300 Write block 2214 len 28672 data: 8 8 8 8 8 6 4
14290 045/300 Write block 54 len 32768 data: 4 4 4 4 4 3 3 2
14291 046/300 Write block 949 len 32768 data: 3 3 3 2 2 2 2 2
14292 047/300 Write block 70 len 20480 data: 2 2 2 2 2
14293 048/300 Write block 1988 len 8192 data: 5 5
14294 049/300 Write block 498 len 28672 data: 4 4 4 4 4 4 4
14295 050/300 Read block 4559 len 16384
14296 051/300 Read block 58 len 24576
14297 052/300 Read block 116 len 28672
14298 053/300 Read block 3314 len 36864
14299 054/300 Read block 1581 len 4096
14300 055/300 Write block 816 len 12288 data: 3 3 3
14301 056/300 Write block 2086 len 24576 data: 4 4 4 4 4 4
14302 057/300 Write block 95 len 4096 data: 5
14303 058/300 Read block 148 len 20480
14304 059/300 Write block 4371 len 4096 data: 3
14305 060/300 Read block 1462 len 4096
14306 061/300 Read block 3874 len 40960
14307 062/300 Flush
14308 063/300 Read block 2807 len 32768
14309 064/300 Read block 4673 len 8192
14310 065/300 Read block 1352 len 24576
14311 066/300 Write block 325 len 8192 data: 3 2
14312 067/300 Read block 2550 len 24576
14313 068/300 Read block 2704 len 32768
14314 069/300 Read block 3345 len 16384
14315 070/300 Read block 108 len 12288
14316 071/300 Write block 3141 len 16384 data: 2 2 2 2
14317 072/300 Flush
14318 073/300 Flush
14319 074/300 Write block 4267 len 16384 data: 3 3 3 3
14320 075/300 Read block 1500 len 40960
14321 076/300 Read block 2390 len 36864
14322 077/300 Write block 1992 len 12288 data: 3 2 2
14323 078/300 Write block 4062 len 4096 data: 3
14324 079/300 Write block 61 len 24576 data: 3 3 3 3 2 2
14325 080/300 Write block 3106 len 36864 data: 3 4 5 5 5 5 6 5 7
14326 081/300 Flush
14327 082/300 Read block 3390 len 20480
14328 083/300 Read block 4413 len 4096
14329 084/300 Write block 3378 len 8192 data: 2 2
14330 085/300 Read block 40 len 8192
14331 086/300 Read block 1707 len 12288
14332 087/300 Read block 3974 len 4096
14333 088/300 Write block 984 len 32768 data: 3 3 3 3 3 3 3 4
14334 089/300 Flush
14335 090/300 Write block 4298 len 4096 data: 3
14336 091/300 Write block 159 len 12288 data: 2 3 4
14337 092/300 Write block 3376 len 16384 data: 3 2 3 3
14338 093/300 Read block 3224 len 36864
14339 094/300 Flush
14340 095/300 Read block 4813 len 16384
14341 096/300 Read block 617 len 36864
14342 097/300 Write block 3307 len 40960 data: 7 7 6 4 4 5 5 3 3 3
14343 098/300 Read block 2794 len 32768
14344 099/300 Write block 3156 len 8192 data: 4 5
14345 100/300 Read block 3985 len 32768
14346 101/300 Flush
14347 102/300 Write block 2491 len 40960 data: 4 2 2 2 2 2 2 2 2 2
14348 103/300 Read block 3393 len 24576
14349 104/300 Write block 2302 len 16384 data: 2 2 3 3
14350 105/300 Write block 4755 len 24576 data: 3 4 4 4 4 4
14351 106/300 Read block 281 len 24576
14352 107/300 Write block 2165 len 16384 data: 4 4 4 5
14353 108/300 Write block 1844 len 32768 data: 3 3 3 2 3 3 3 4
14354 109/300 Write block 553 len 4096 data: 6
14355 110/300 Write block 2097 len 24576 data: 5 5 4 4 4 3
14356 111/300 Read block 33 len 40960
14357 112/300 Read block 1629 len 12288
14358 113/300 Read block 1610 len 20480
14359 114/300 Read block 2603 len 36864
14360 115/300 Read block 1720 len 32768
14361 116/300 Read block 4736 len 12288
14362 117/300 Read block 1499 len 32768
14363 118/300 Read block 848 len 20480
14364 119/300 Flush
14365 120/300 Write block 870 len 8192 data: 5 4
14366 121/300 Write block 2290 len 16384 data: 2 3 3 3
14367 122/300 Read block 3574 len 32768
14368 123/300 Write block 3481 len 12288 data: 4 4 4
14369 124/300 Read block 3291 len 12288
14370 125/300 Write block 4066 len 8192 data: 3 3
14371 126/300 Flush
14372 127/300 Flush
14373 128/300 Flush
14374 129/300 Read block 1569 len 8192
14375 130/300 Read block 1325 len 24576
14376 131/300 Write block 3043 len 24576 data: 2 2 2 2 3 3
14377 132/300 Write block 1820 len 8192 data: 3 4
14378 133/300 Write block 4375 len 24576 data: 4 4 4 4 4 4
14379 134/300 Write block 2913 len 20480 data: 2 2 2 2 3
14380 135/300 Read block 1431 len 40960
14381 136/300 Read block 4250 len 40960
14382 137/300 Read block 2576 len 36864
14383 138/300 Read block 1043 len 16384
14384 139/300 Write block 1252 len 24576 data: 3 4 5 6 6 6
14385 140/300 Flush
14386 141/300 Write block 515 len 40960 data: 3 3 3 3 3 4 5 5 5 5
14387 142/300 Flush
14388 143/300 Write block 4436 len 28672 data: 3 3 3 2 2 3 4
14389 144/300 Read block 1239 len 40960
14390 145/300 Write block 2184 len 12288 data: 2 2 3
14391 146/300 Write block 4306 len 28672 data: 2 3 2 2 2 2 2
14392 147/300 Write block 3706 len 8192 data: 3 4
14393 148/300 Write block 2553 len 24576 data: 2 2 2 2 2 2
14394 149/300 Flush
14395 150/300 Write block 1201 len 36864 data: 2 2 3 3 4 3 3 3 3
14396 151/300 Read block 4016 len 40960
14397 152/300 Write block 2373 len 4096 data: 3
14398 153/300 Write block 1286 len 32768 data: 2 2 2 3 3 4 4 4
14399 154/300 Read block 2378 len 4096
14400 155/300 Read block 2116 len 36864
14401 156/300 Write block 1752 len 36864 data: 3 3 3 3 2 2 3 3 3
14402 157/300 Read block 1665 len 4096
14403 158/300 Write block 1419 len 28672 data: 2 3 3 3 4 3 3
14404 159/300 Read block 4042 len 32768
14405 160/300 Write block 3557 len 20480 data: 3 3 3 2 2
14406 161/300 Write block 1503 len 32768 data: 2 2 2 2 2 2 2 2
14407 162/300 Write block 4828 len 20480 data: 5 5 7 7 6
14408 163/300 Read block 3379 len 24576
14409 164/300 Flush
14410 165/300 Read block 3009 len 24576
14411 166/300 Write block 1210 len 32768 data: 3 3 2 2 2 2 2 2
14412 167/300 Read block 1171 len 36864
14413 168/300 Read block 3358 len 28672
14414 169/300 Write block 1802 len 4096 data: 2
14415 170/300 Flush
14416 171/300 Write block 4783 len 32768 data: 4 3 3 3 3 4 4 3
14417 172/300 Write block 3568 len 16384 data: 2 2 2 2
14418 173/300 Write block 2897 len 16384 data: 2 3 2 2
14419 174/300 Flush
14420 175/300 Read block 200 len 16384
14421 176/300 Write block 207 len 24576 data: 2 2 3 3 3 3
14422 177/300 Read block 2606 len 36864
14423 178/300 Write block 3457 len 24576 data: 5 5 5 2 3 2
14424 179/300 Read block 889 len 24576
14425 180/300 Read block 4611 len 28672
14426 181/300 Write block 654 len 36864 data: 3 3 2 4 5 5 5 5 5
14427 182/300 Write block 183 len 36864 data: 2 2 2 2 2 3 3 3 4
14428 183/300 Write block 2735 len 28672 data: 3 3 3 2 2 4 4
14429 184/300 Write block 2032 len 32768 data: 4 4 3 2 2 2 2 2
14430 185/300 Write block 2490 len 4096 data: 4
14431 186/300 Read block 119 len 36864
14432 187/300 Read block 3344 len 12288
14433 188/300 Read block 4573 len 24576
14434 189/300 Read block 3487 len 4096
14435 190/300 Write block 2647 len 20480 data: 3 3 3 3 3
14436 191/300 Read block 3456 len 36864
14437 192/300 Write block 4996 len 12288 data: 3 3 3
14438 193/300 Write block 2917 len 20480 data: 4 4 3 3 3
14439 194/300 Write block 2221 len 32768 data: 3 3 3 3 4 4 5 5
14440 195/300 Write block 2262 len 40960 data: 3 3 4 4 4 4 4 3 3 2
14441 196/300 Read block 2346 len 32768
14442 197/300 Read block 624 len 20480
14443 198/300 Flush
14444 199/300 Read block 4588 len 32768
14445 200/300 Read block 3359 len 16384
14446 201/300 Read block 1359 len 40960
14447 202/300 Read block 3279 len 20480
14448 203/300 Read block 3436 len 20480
14449 204/300 Read block 3887 len 28672
14450 205/300 Read block 2336 len 28672
14451 206/300 Read block 3663 len 24576
14452 207/300 Read block 393 len 36864
14453 208/300 Write block 739 len 16384 data: 3 3 2 2
14454 209/300 Read block 145 len 40960
14455 210/300 Write block 4833 len 24576 data: 3 3 4 4 5 5
14456 211/300 Flush
14457 212/300 Flush
14458 213/300 Read block 4635 len 20480
14459 214/300 Read block 2688 len 32768
14460 215/300 Write block 3143 len 12288 data: 3 3 2
14461 216/300 Read block 262 len 24576
14462 217/300 Write block 226 len 36864 data: 3 3 2 2 3 3 2 2 3
14463 218/300 Flush
14464 219/300 Read block 4725 len 36864
14465 220/300 Read block 3036 len 24576
14466 221/300 Read block 323 len 40960
14467 222/300 Read block 1051 len 4096
14468 223/300 Read block 336 len 8192
14469 224/300 Write block 3850 len 28672 data: 6 6 7 6 5 4 4
14470 225/300 Read block 4809 len 28672
14471 226/300 Write block 811 len 32768 data: 3 4 4 3 2 4 4 4
14472 227/300 Read block 4031 len 8192
14473 228/300 Write block 474 len 12288 data: 4 5 5
14474 229/300 Read block 1564 len 24576
14475 230/300 Write block 1433 len 20480 data: 2 2 2 2 3
14476 231/300 Write block 4096 len 32768 data: 3 3 3 3 3 4 4 4
14477 232/300 Read block 4059 len 40960
14478 233/300 Write block 2729 len 20480 data: 3 5 4 4 4
14479 234/300 Write block 188 len 40960 data: 4 4 4 5 5 5 5 5 5 4
14480 235/300 Flush
14481 236/300 Write block 4224 len 32768 data: 3 5 6 6 7 8 7 7
14482 237/300 Read block 75 len 20480
14483 238/300 Flush
14484 239/300 Read block 189 len 12288
14485 240/300 Read block 1342 len 36864
14486 241/300 Flush
14487 242/300 Write block 1394 len 12288 data: 2 2 2
14488 243/300 Read block 2493 len 28672
14489 244/300 Flush
14490 245/300 Read block 1684 len 24576
14491 246/300 Write block 4127 len 28672 data: 4 4 4 4 4 3 3
14492 247/300 Write block 63 len 8192 data: 4 4
14493 248/300 Read block 4559 len 12288
14494 249/300 Read block 4892 len 8192
14495 250/300 Read block 2359 len 8192
14496 251/300 Read block 4862 len 8192
14497 252/300 Read block 4441 len 32768
14498 253/300 Read block 3345 len 12288
14499 254/300 Write block 1477 len 40960 data: 2 2 2 2 2 2 3 3 5 5
14500 255/300 Read block 971 len 28672
14501 256/300 Read block 1584 len 20480
14502 257/300 Flush
14503 258/300 Write block 2276 len 40960 data: 6 6 7 7 7 6 4 5 3 3
14504 259/300 Read block 1862 len 40960
14505 260/300 Read block 920 len 4096
14506 261/300 Read block 4131 len 24576
14507 262/300 Write block 2020 len 4096 data: 3
14508 263/300 Write block 4950 len 40960 data: 2 3 3 4 5 4 4 4 4 3
14509 264/300 Read block 3517 len 24576
14510 265/300 Read block 863 len 36864
14511 266/300 Read block 78 len 36864
14512 267/300 Flush
14513 268/300 Write block 2806 len 24576 data: 4 4 3 3 3 4
14514 269/300 Write block 2121 len 8192 data: 4 4
14515 270/300 Read block 2080 len 12288
14516 271/300 Write block 311 len 4096 data: 2
14517 272/300 Read block 3034 len 40960
14518 273/300 Read block 3423 len 28672
14519 274/300 Read block 3241 len 28672
14520 275/300 Read block 1865 len 20480
14521 276/300 Read block 1601 len 40960
14522 277/300 Write block 3347 len 20480 data: 5 3 2 2 3
14523 278/300 Flush
14524 279/300 Read block 1665 len 20480
14525 280/300 Read block 4788 len 40960
14526 281/300 Read block 3440 len 40960
14527 282/300 Write block 1008 len 40960 data: 3 3 3 3 3 3 3 3 4 5
14528 283/300 Write block 958 len 12288 data: 3 3 3
14529 284/300 Write block 1283 len 4096 data: 4
14530 285/300 Write block 2026 len 32768 data: 3 3 5 5 4 4 5 5
14531 286/300 Read block 3571 len 8192
14532 287/300 Read block 1274 len 20480
14533 288/300 Flush
14534 289/300 Read block 3874 len 12288
14535 290/300 Write block 3159 len 8192 data: 6 6
14536 291/300 Read block 560 len 4096
14537 292/300 Write block 4285 len 40960 data: 3 3 3 2 3 4 4 4 3 3
14538 293/300 Write block 3101 len 12288 data: 5 5 5
14539 294/300 Read block 453 len 12288
14540 295/300 Read block 281 len 40960
14541 296/300 Write block 3792 len 32768 data: 3 2 4 4 4 3 3 3
14542 297/300 Write block 4592 len 32768 data: 4 4 4 5 4 4 4 2
14543 298/300 Read block 4816 len 4096
14544 299/300 Flush
14545 300/300 Read block 4916 len 36864
14546 301/300 Write block 975 len 16384 data: 3 3 3 4
14547 Main task has work to do, go find it
14548 got dsc Start(2)
14549 start 2
14550 Feb 28 06:04:14.874 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1f7506d9-18fb-47f3-9ec1-e5758272bb42, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
14551 Replay: started 2, returned:Ok(())
14552 ----------------------------------------------------------------
14553 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14554 States: Active Active Offline
14555 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14556 4319 Acked Write 7 Done Done Sent false
14557 4320 Acked Write 3 Done Done Sent false
14558 4321 Acked Read 2 Done Done Sent false
14559 4322 Acked Flush 0 Done Done Sent false
14560 4323 Acked Read 2 Done Done Sent false
14561 4324 Acked Write 3 Done Done Sent false
14562 4325 Acked Write 2 Done Done Sent false
14563 4326 Acked Read 10 Done Done Sent false
14564 4327 Acked Read 8 Done Done Sent false
14565 4328 Acked Read 2 Done Done Sent false
14566 4329 Acked Write 8 Done Done Sent false
14567 4330 Acked Flush 0 Done Done Sent false
14568 4331 Acked Read 3 Done Done Sent false
14569 4332 Acked Read 5 Done Done Sent false
14570 4333 Acked Read 4 Done Done Sent false
14571 4334 Acked Read 5 Done Done Sent false
14572 4335 Acked Read 6 Done Done Sent false
14573 4336 Acked Write 4 Done Done Sent false
14574 4337 Acked Write 6 Done Done Sent false
14575 4338 Acked Read 4 Done Done Sent false
14576 4339 Acked Write 8 Done Done Sent false
14577 4340 Acked Read 4 Done Done Sent false
14578 4341 Acked Read 1 Done Done Sent false
14579 4342 Acked Read 4 Done Done Sent false
14580 4343 Acked Write 4 Done Done Sent false
14581 4344 Acked Flush 0 Done Done Sent false
14582 4345 Acked Write 4 Done Done Sent false
14583 4346 Acked Write 9 Done Done Sent false
14584 4347 Acked Read 1 Done Done Sent false
14585 4348 Acked Read 7 Done Done Sent false
14586 4349 Acked Read 3 Done Done Sent false
14587 4350 Acked Flush 0 Done Done Sent false
14588 4351 Acked Read 1 Done Done Sent false
14589 4352 Acked Read 10 Done Done Sent false
14590 4353 Acked Read 9 Done Done Sent false
14591 4354 Acked Flush 0 Done Done Sent false
14592 4355 Acked Read 3 Done Done Sent false
14593 4356 Acked Read 2 Done Done Sent false
14594 4357 Acked Read 10 Done Done Sent false
14595 4358 Acked Flush 0 Done Done Sent false
14596 4359 Acked Read 8 Done Done Sent false
14597 4360 Acked Write 3 Done Done Sent false
14598 4361 Acked Write 1 Done Done Sent false
14599 4362 Acked Write 7 Done Done Sent false
14600 4363 Acked Write 8 Done Done Sent false
14601 4364 Acked Write 8 Done Done Sent false
14602 4365 Acked Write 5 Done Done Sent false
14603 4366 Acked Write 2 Done Done Sent false
14604 4367 Acked Write 7 Done Done Sent false
14605 4368 Acked Read 4 Done Done Sent false
14606 4369 Acked Read 6 Done Done Sent false
14607 4370 Acked Read 7 Done Done Sent false
14608 4371 Acked Read 9 Done Done Sent false
14609 4372 Acked Read 1 Done Done Sent false
14610 4373 Acked Write 3 Done Done Sent false
14611 4374 Acked Write 6 Done Done Sent false
14612 4375 Acked Write 1 Done Done Sent false
14613 4376 Acked Read 5 Done Done Sent false
14614 4377 Acked Write 1 Done Done Sent false
14615 4378 Acked Read 1 Done Done Sent false
14616 4379 Acked Read 10 Done Done Sent false
14617 4380 Acked Flush 0 Done Done Sent false
14618 4381 Acked Read 8 Done Done Sent false
14619 4382 Acked Read 2 Done Done Sent false
14620 4383 Acked Read 6 Done Done Sent false
14621 4384 Acked Write 2 Done Done Sent false
14622 4385 Acked Read 6 Done Done Sent false
14623 4386 Acked Read 8 Done Done Sent false
14624 4387 Acked Read 4 Done Done Sent false
14625 4388 Acked Read 3 Done Done Sent false
14626 4389 Acked Write 4 Done Done Sent false
14627 4390 Acked Flush 0 Done Done Sent false
14628 4391 Acked Flush 0 Done Done Sent false
14629 4392 Acked Write 4 Done Done Sent false
14630 4393 Acked Read 10 Done Done Sent false
14631 4394 Acked Read 9 Done Done Sent false
14632 4395 Acked Write 3 Done Done Sent false
14633 4396 Acked Write 1 Done Done Sent false
14634 4397 Acked Write 6 Done Done Sent false
14635 4398 Acked Write 9 Done Done Sent false
14636 4399 Acked Flush 0 Done Done Sent false
14637 4400 Acked Read 5 Done Done Sent false
14638 4401 Acked Read 1 Done Done Sent false
14639 4402 Acked Write 2 Done Done Sent false
14640 4403 Acked Read 2 Done Done Sent false
14641 4404 Acked Read 3 Done Done Sent false
14642 4405 Acked Read 1 Done Done Sent false
14643 4406 Acked Write 8 Done Done Sent false
14644 4407 Acked Flush 0 Done Done Sent false
14645 4408 Acked Write 1 Done Done Sent false
14646 4409 Acked Write 3 Done Done Sent false
14647 4410 Acked Write 4 Done Done Sent false
14648 4411 Acked Read 9 Done Done Sent false
14649 4412 Acked Flush 0 Done Done Sent false
14650 4413 Acked Read 4 Done Done Sent false
14651 4414 Acked Read 9 Done Done Sent false
14652 4415 Acked Write 10 Done Done Sent false
14653 4416 Acked Read 8 Done Done Sent false
14654 4417 Acked Write 2 Done Done Sent false
14655 4418 Acked Read 8 Done Done Sent false
14656 4419 Acked Flush 0 Done Done Sent false
14657 4420 Acked Write 10 Done Done Sent false
14658 4421 Acked Read 6 Done Done Sent false
14659 4422 Acked Write 4 Done Done Sent false
14660 4423 Acked Write 6 Done Done Sent false
14661 4424 Acked Read 6 Done Done Sent false
14662 4425 Acked Write 4 Done Done Sent false
14663 4426 Acked Write 8 Done Done Sent false
14664 4427 Acked Write 1 Done Done Sent false
14665 4428 Acked Write 6 Done Done Sent false
14666 4429 Acked Read 10 Done Done Sent false
14667 4430 Acked Read 3 Done Done Sent false
14668 4431 Acked Read 5 Done Done Sent false
14669 4432 Acked Read 9 Done Done Sent false
14670 4433 Acked Read 8 Done Done Sent false
14671 4434 Acked Read 3 Done Done Sent false
14672 4435 Acked Read 8 Done Done Sent false
14673 4436 Acked Read 5 Done Done Sent false
14674 4437 Acked Flush 0 Done Done Sent false
14675 4438 Acked Write 2 Done Done Sent false
14676 4439 Acked Write 4 Done Done Sent false
14677 4440 Acked Read 8 Done Done Sent false
14678 4441 Acked Write 3 Done Done Sent false
14679 4442 Acked Read 3 Done Done Sent false
14680 4443 Acked Write 2 Done Done Sent false
14681 4444 Acked Flush 0 Done Done Sent false
14682 4445 Acked Flush 0 Done Done Sent false
14683 4446 Acked Flush 0 Done Done Sent false
14684 4447 Acked Read 2 Done Done Sent false
14685 4448 Acked Read 6 Done Done Sent false
14686 4449 Acked Write 6 Done Done Sent false
14687 4450 Acked Write 2 Done Done Sent false
14688 4451 Acked Write 6 Done Done Sent false
14689 4452 Acked Write 5 Done Done Sent false
14690 4453 Acked Read 10 Done Done Sent false
14691 4454 Acked Read 10 Done Done Sent false
14692 4455 Acked Read 9 Done Done Sent false
14693 4456 Acked Read 4 Done Done Sent false
14694 4457 Acked Write 6 Done Done Sent false
14695 4458 Acked Flush 0 Done Done Sent false
14696 4459 Acked Write 10 Done Done Sent false
14697 4460 Acked Flush 0 Done Done Sent false
14698 4461 Acked Write 7 Done Done Sent false
14699 4462 Acked Read 10 Done Done Sent false
14700 4463 Acked Write 3 Done Done Sent false
14701 4464 Acked Write 7 Done Done Sent false
14702 4465 Acked Write 2 Done Done Sent false
14703 4466 Acked Write 6 Done Done Sent false
14704 4467 Acked Flush 0 Done Done Sent false
14705 4468 Acked Write 9 Done Done Sent false
14706 4469 Acked Read 10 Done Done Sent false
14707 4470 Acked Write 1 Done Done Sent false
14708 4471 Acked Write 8 Done Done Sent false
14709 4472 Acked Read 1 Done Done Sent false
14710 4473 Acked Read 9 Done Done Sent false
14711 4474 Acked Write 9 Done Done Sent false
14712 4475 Acked Read 1 Done Done Sent false
14713 4476 Acked Write 7 Done Done Sent false
14714 4477 Acked Read 8 Done Done Sent false
14715 4478 Acked Write 5 Done Done Sent false
14716 4479 Acked Write 8 Done Done Sent false
14717 4480 Acked Write 5 Done Done Sent false
14718 4481 Acked Read 6 Done Done Sent false
14719 4482 Acked Flush 0 Done Done Sent false
14720 4483 Acked Read 6 Done Done Sent false
14721 4484 Acked Write 8 Done Done Sent false
14722 4485 Acked Read 9 Done Done Sent false
14723 4486 Acked Read 7 Done Done Sent false
14724 4487 Acked Write 1 Done Done Sent false
14725 4488 Acked Flush 0 Done Done Sent false
14726 4489 Acked Write 8 Done Done Sent false
14727 4490 Acked Write 4 Done Done Sent false
14728 4491 Acked Write 4 Done Done Sent false
14729 4492 Acked Flush 0 Done Done Sent false
14730 4493 Acked Read 4 Done Done Sent false
14731 4494 Acked Write 6 Done Done Sent false
14732 4495 Acked Read 9 Done Done Sent false
14733 4496 Acked Write 6 Done Done Sent false
14734 4497 Acked Read 6 Done Done Sent false
14735 4498 Acked Read 7 Done Done Sent false
14736 4499 Acked Write 9 Done Done Sent false
14737 4500 Acked Write 9 Done Done Sent false
14738 4501 Acked Write 7 Done Done Sent false
14739 4502 Acked Write 8 Done Done Sent false
14740 4503 Acked Write 1 Done Done Sent false
14741 4504 Acked Read 9 Done Done Sent false
14742 4505 Acked Read 3 Done Done Sent false
14743 4506 Acked Read 6 Done Done Sent false
14744 4507 Acked Read 1 Done Done Sent false
14745 4508 Acked Write 5 Done Done Sent false
14746 4509 Acked Read 9 Done Done Sent false
14747 4510 Acked Write 3 Done Done Sent false
14748 4511 Acked Write 5 Done Done Sent false
14749 4512 Acked Write 8 Done Done Sent false
14750 4513 Acked Write 10 Done Done Sent false
14751 4514 Acked Read 8 Done Done Sent false
14752 4515 Acked Read 5 Done Done Sent false
14753 4516 Acked Flush 0 Done Done Sent false
14754 4517 Acked Read 8 Done Done Sent false
14755 4518 Acked Read 4 Done Done Sent false
14756 4519 Acked Read 10 Done Done Sent false
14757 4520 Acked Read 5 Done Done Sent false
14758 4521 Acked Read 5 Done Done Sent false
14759 4522 Acked Read 7 Done Done Sent false
14760 4523 Acked Read 7 Done Done Sent false
14761 4524 Acked Read 6 Done Done Sent false
14762 4525 Acked Read 9 Done Done Sent false
14763 4526 Acked Write 4 Done Done Sent false
14764 4527 Acked Read 10 Done Done Sent false
14765 4528 Acked Write 6 Done Done Sent false
14766 4529 Acked Flush 0 Done Done Sent false
14767 4530 Acked Flush 0 Done Done Sent false
14768 4531 Acked Read 5 Done Done Sent false
14769 4532 Acked Read 8 Done Done Sent false
14770 4533 Acked Write 3 Done Done Sent false
14771 4534 Acked Read 6 Done Done Sent false
14772 4535 Acked Write 9 Done Done Sent false
14773 4536 Acked Flush 0 Done Done Sent false
14774 4537 Acked Read 9 Done Done Sent false
14775 4538 Acked Read 6 Done Done Sent false
14776 4539 Acked Read 10 Done Done Sent false
14777 4540 Acked Read 1 Done Done Sent false
14778 4541 Acked Read 2 Done Done Sent false
14779 4542 Acked Write 7 Done Done Sent false
14780 4543 Acked Read 7 Done Done Sent false
14781 4544 Acked Write 8 Done Done Sent false
14782 4545 Acked Read 2 Done Done Sent false
14783 4546 Acked Write 3 Done Done Sent false
14784 4547 Acked Read 6 Done Done Sent false
14785 4548 Acked Write 5 Done Done Sent false
14786 4549 Acked Write 8 Done Done Sent false
14787 4550 Acked Read 10 Done Done Sent false
14788 4551 Acked Write 5 Done Done Sent false
14789 4552 Acked Write 10 Done Done Sent false
14790 4553 Acked Flush 0 Done Done Sent false
14791 4554 Acked Write 8 Done Done Sent false
14792 4555 Acked Read 5 Done Done Sent false
14793 4556 Acked Flush 0 Done Done Sent false
14794 4557 Acked Read 3 Done Done Sent false
14795 4558 Acked Read 9 Done Done Sent false
14796 4559 Acked Flush 0 Done Done Sent false
14797 4560 Acked Write 3 Done Done Sent false
14798 4561 Acked Read 7 Done Done Sent false
14799 4562 Acked Flush 0 Done Done Sent false
14800 4563 Acked Read 6 Done Done Sent false
14801 4564 Acked Write 7 Done Done Sent false
14802 4565 Acked Write 2 Done Done Sent false
14803 4566 Acked Read 3 Done Done Sent false
14804 4567 Acked Read 2 Done Done Sent false
14805 4568 Acked Read 2 Done Done Sent false
14806 4569 Acked Read 2 Done Done Sent false
14807 4570 Acked Read 8 Done Done Sent false
14808 4571 Acked Read 3 Done Done Sent false
14809 4572 Acked Write 10 Done Done Sent false
14810 4573 Acked Read 7 Done Done Sent false
14811 4574 Acked Read 5 Done Done Sent false
14812 4575 Acked Flush 0 Done Done Sent false
14813 4576 Acked Write 10 Done Done Sent false
14814 4577 Acked Read 10 Done Done Sent false
14815 4578 Acked Read 1 Done Done Sent false
14816 4579 Acked Read 6 Done Done Sent false
14817 4580 Acked Write 1 Done Done Sent false
14818 4581 Acked Write 10 Done Done Sent false
14819 4582 Acked Read 6 Done Done Sent false
14820 4583 Acked Read 9 Done Done Sent false
14821 4584 Acked Read 9 Done Done Sent false
14822 4585 Acked Flush 0 Done Done Sent false
14823 4586 Acked Write 6 Done Done Sent false
14824 4587 Acked Write 2 Done Done Sent false
14825 4588 Acked Read 3 Done Done Sent false
14826 4589 Acked Write 1 Done Done Sent false
14827 4590 Acked Read 10 Done Done Sent false
14828 4591 Acked Read 7 Done Done Sent false
14829 4592 Acked Read 7 Done Done Sent false
14830 4593 Acked Read 5 Done Done Sent false
14831 4594 Acked Read 10 Done Done Sent false
14832 4595 Acked Write 5 Done Done Sent false
14833 4596 Acked Flush 0 Done Done Sent false
14834 4597 Acked Read 5 Done Done Sent false
14835 4598 Acked Read 10 Done Done Sent false
14836 4599 Acked Read 10 Done Done Sent false
14837 4600 Acked Write 10 Done Done Sent false
14838 4601 Acked Write 3 Done Done Sent false
14839 4602 Acked Write 1 Done Done Sent false
14840 4603 Acked Write 8 Done Done Sent false
14841 4604 Acked Read 2 Done Done Sent false
14842 4605 Acked Read 5 Done Done Sent false
14843 4606 Acked Flush 0 Done Done Sent false
14844 4607 Acked Read 3 Done Done Sent false
14845 4608 Acked Write 2 Done Done Sent false
14846 4609 Acked Read 1 Done Done Sent false
14847 4610 Acked Write 10 Done Done Sent false
14848 4611 Acked Write 3 Done Done Sent false
14849 4612 Acked Read 3 Done Done Sent false
14850 4613 Acked Read 10 Done Done Sent false
14851 4614 Acked Write 8 Done Done Sent false
14852 4615 Acked Write 8 Done Done Sent false
14853 4616 Acked Read 1 Done Done Sent false
14854 4617 Acked Flush 0 Done Done Sent false
14855 4618 Acked Read 9 Done Done Sent false
14856 4619 Acked Write 4 Done Done Sent false
14857 STATES DS:0 DS:1 DS:2 TOTAL
14858 Sent 0 0 301 301
14859 Done 301 301 0 602
14860 Skipped 0 0 0 0
14861 Error 0 0 0 0
14862 Last Flush: Some(JobId(4617)) Some(JobId(4617)) Some(JobId(4318))
14863 Downstairs last five completed: 4318 4317 4316 4315 4314
14864 Upstairs last five completed: 4619 4618 4617 4616 4615
14865 CLIENT: Up:0 ds:301 act:2
148662025-02-28T06:04:15.658ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
14867 [8830] got start action
14868 [8830] I am going to restart
14869 Starting downstairs at port 8830
14870 Make output file at "/tmp/dsc/downstairs-8830.txt"
14871 [8830][2] reports Starting
14872 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1223)
14873 [8830][2] reports Running
148742025-02-28T06:04:16.885ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
148752025-02-28T06:04:17.662ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
148762025-02-28T06:04:18.664ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
14877 ----------------------------------------------------------------
14878 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14879 States: Active Active Offline
14880 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14881 4319 Acked Write 7 Done Done Sent false
14882 4320 Acked Write 3 Done Done Sent false
14883 4321 Acked Read 2 Done Done Sent false
14884 4322 Acked Flush 0 Done Done Sent false
14885 4323 Acked Read 2 Done Done Sent false
14886 4324 Acked Write 3 Done Done Sent false
14887 4325 Acked Write 2 Done Done Sent false
14888 4326 Acked Read 10 Done Done Sent false
14889 4327 Acked Read 8 Done Done Sent false
14890 4328 Acked Read 2 Done Done Sent false
14891 4329 Acked Write 8 Done Done Sent false
14892 4330 Acked Flush 0 Done Done Sent false
14893 4331 Acked Read 3 Done Done Sent false
14894 4332 Acked Read 5 Done Done Sent false
14895 4333 Acked Read 4 Done Done Sent false
14896 4334 Acked Read 5 Done Done Sent false
14897 4335 Acked Read 6 Done Done Sent false
14898 4336 Acked Write 4 Done Done Sent false
14899 4337 Acked Write 6 Done Done Sent false
14900 4338 Acked Read 4 Done Done Sent false
14901 4339 Acked Write 8 Done Done Sent false
14902 4340 Acked Read 4 Done Done Sent false
14903 4341 Acked Read 1 Done Done Sent false
14904 4342 Acked Read 4 Done Done Sent false
14905 4343 Acked Write 4 Done Done Sent false
14906 4344 Acked Flush 0 Done Done Sent false
14907 4345 Acked Write 4 Done Done Sent false
14908 4346 Acked Write 9 Done Done Sent false
14909 4347 Acked Read 1 Done Done Sent false
14910 4348 Acked Read 7 Done Done Sent false
14911 4349 Acked Read 3 Done Done Sent false
14912 4350 Acked Flush 0 Done Done Sent false
14913 4351 Acked Read 1 Done Done Sent false
14914 4352 Acked Read 10 Done Done Sent false
14915 4353 Acked Read 9 Done Done Sent false
14916 4354 Acked Flush 0 Done Done Sent false
14917 4355 Acked Read 3 Done Done Sent false
14918 4356 Acked Read 2 Done Done Sent false
14919 4357 Acked Read 10 Done Done Sent false
14920 4358 Acked Flush 0 Done Done Sent false
14921 4359 Acked Read 8 Done Done Sent false
14922 4360 Acked Write 3 Done Done Sent false
14923 4361 Acked Write 1 Done Done Sent false
14924 4362 Acked Write 7 Done Done Sent false
14925 4363 Acked Write 8 Done Done Sent false
14926 4364 Acked Write 8 Done Done Sent false
14927 4365 Acked Write 5 Done Done Sent false
14928 4366 Acked Write 2 Done Done Sent false
14929 4367 Acked Write 7 Done Done Sent false
14930 4368 Acked Read 4 Done Done Sent false
14931 4369 Acked Read 6 Done Done Sent false
14932 4370 Acked Read 7 Done Done Sent false
14933 4371 Acked Read 9 Done Done Sent false
14934 4372 Acked Read 1 Done Done Sent false
14935 4373 Acked Write 3 Done Done Sent false
14936 4374 Acked Write 6 Done Done Sent false
14937 4375 Acked Write 1 Done Done Sent false
14938 4376 Acked Read 5 Done Done Sent false
14939 4377 Acked Write 1 Done Done Sent false
14940 4378 Acked Read 1 Done Done Sent false
14941 4379 Acked Read 10 Done Done Sent false
14942 4380 Acked Flush 0 Done Done Sent false
14943 4381 Acked Read 8 Done Done Sent false
14944 4382 Acked Read 2 Done Done Sent false
14945 4383 Acked Read 6 Done Done Sent false
14946 4384 Acked Write 2 Done Done Sent false
14947 4385 Acked Read 6 Done Done Sent false
14948 4386 Acked Read 8 Done Done Sent false
14949 4387 Acked Read 4 Done Done Sent false
14950 4388 Acked Read 3 Done Done Sent false
14951 4389 Acked Write 4 Done Done Sent false
14952 4390 Acked Flush 0 Done Done Sent false
14953 4391 Acked Flush 0 Done Done Sent false
14954 4392 Acked Write 4 Done Done Sent false
14955 4393 Acked Read 10 Done Done Sent false
14956 4394 Acked Read 9 Done Done Sent false
14957 4395 Acked Write 3 Done Done Sent false
14958 4396 Acked Write 1 Done Done Sent false
14959 4397 Acked Write 6 Done Done Sent false
14960 4398 Acked Write 9 Done Done Sent false
14961 4399 Acked Flush 0 Done Done Sent false
14962 4400 Acked Read 5 Done Done Sent false
14963 4401 Acked Read 1 Done Done Sent false
14964 4402 Acked Write 2 Done Done Sent false
14965 4403 Acked Read 2 Done Done Sent false
14966 4404 Acked Read 3 Done Done Sent false
14967 4405 Acked Read 1 Done Done Sent false
14968 4406 Acked Write 8 Done Done Sent false
14969 4407 Acked Flush 0 Done Done Sent false
14970 4408 Acked Write 1 Done Done Sent false
14971 4409 Acked Write 3 Done Done Sent false
14972 4410 Acked Write 4 Done Done Sent false
14973 4411 Acked Read 9 Done Done Sent false
14974 4412 Acked Flush 0 Done Done Sent false
14975 4413 Acked Read 4 Done Done Sent false
14976 4414 Acked Read 9 Done Done Sent false
14977 4415 Acked Write 10 Done Done Sent false
14978 4416 Acked Read 8 Done Done Sent false
14979 4417 Acked Write 2 Done Done Sent false
14980 4418 Acked Read 8 Done Done Sent false
14981 4419 Acked Flush 0 Done Done Sent false
14982 4420 Acked Write 10 Done Done Sent false
14983 4421 Acked Read 6 Done Done Sent false
14984 4422 Acked Write 4 Done Done Sent false
14985 4423 Acked Write 6 Done Done Sent false
14986 4424 Acked Read 6 Done Done Sent false
14987 4425 Acked Write 4 Done Done Sent false
14988 4426 Acked Write 8 Done Done Sent false
14989 4427 Acked Write 1 Done Done Sent false
14990 4428 Acked Write 6 Done Done Sent false
14991 4429 Acked Read 10 Done Done Sent false
14992 4430 Acked Read 3 Done Done Sent false
14993 4431 Acked Read 5 Done Done Sent false
14994 4432 Acked Read 9 Done Done Sent false
14995 4433 Acked Read 8 Done Done Sent false
14996 4434 Acked Read 3 Done Done Sent false
14997 4435 Acked Read 8 Done Done Sent false
14998 4436 Acked Read 5 Done Done Sent false
14999 4437 Acked Flush 0 Done Done Sent false
15000 4438 Acked Write 2 Done Done Sent false
15001 4439 Acked Write 4 Done Done Sent false
15002 4440 Acked Read 8 Done Done Sent false
15003 4441 Acked Write 3 Done Done Sent false
15004 4442 Acked Read 3 Done Done Sent false
15005 4443 Acked Write 2 Done Done Sent false
15006 4444 Acked Flush 0 Done Done Sent false
15007 4445 Acked Flush 0 Done Done Sent false
15008 4446 Acked Flush 0 Done Done Sent false
15009 4447 Acked Read 2 Done Done Sent false
15010 4448 Acked Read 6 Done Done Sent false
15011 4449 Acked Write 6 Done Done Sent false
15012 4450 Acked Write 2 Done Done Sent false
15013 4451 Acked Write 6 Done Done Sent false
15014 4452 Acked Write 5 Done Done Sent false
15015 4453 Acked Read 10 Done Done Sent false
15016 4454 Acked Read 10 Done Done Sent false
15017 4455 Acked Read 9 Done Done Sent false
15018 4456 Acked Read 4 Done Done Sent false
15019 4457 Acked Write 6 Done Done Sent false
15020 4458 Acked Flush 0 Done Done Sent false
15021 4459 Acked Write 10 Done Done Sent false
15022 4460 Acked Flush 0 Done Done Sent false
15023 4461 Acked Write 7 Done Done Sent false
15024 4462 Acked Read 10 Done Done Sent false
15025 4463 Acked Write 3 Done Done Sent false
15026 4464 Acked Write 7 Done Done Sent false
15027 4465 Acked Write 2 Done Done Sent false
15028 4466 Acked Write 6 Done Done Sent false
15029 4467 Acked Flush 0 Done Done Sent false
15030 4468 Acked Write 9 Done Done Sent false
15031 4469 Acked Read 10 Done Done Sent false
15032 4470 Acked Write 1 Done Done Sent false
15033 4471 Acked Write 8 Done Done Sent false
15034 4472 Acked Read 1 Done Done Sent false
15035 4473 Acked Read 9 Done Done Sent false
15036 4474 Acked Write 9 Done Done Sent false
15037 4475 Acked Read 1 Done Done Sent false
15038 4476 Acked Write 7 Done Done Sent false
15039 4477 Acked Read 8 Done Done Sent false
15040 4478 Acked Write 5 Done Done Sent false
15041 4479 Acked Write 8 Done Done Sent false
15042 4480 Acked Write 5 Done Done Sent false
15043 4481 Acked Read 6 Done Done Sent false
15044 4482 Acked Flush 0 Done Done Sent false
15045 4483 Acked Read 6 Done Done Sent false
15046 4484 Acked Write 8 Done Done Sent false
15047 4485 Acked Read 9 Done Done Sent false
15048 4486 Acked Read 7 Done Done Sent false
15049 4487 Acked Write 1 Done Done Sent false
15050 4488 Acked Flush 0 Done Done Sent false
15051 4489 Acked Write 8 Done Done Sent false
15052 4490 Acked Write 4 Done Done Sent false
15053 4491 Acked Write 4 Done Done Sent false
15054 4492 Acked Flush 0 Done Done Sent false
15055 4493 Acked Read 4 Done Done Sent false
15056 4494 Acked Write 6 Done Done Sent false
15057 4495 Acked Read 9 Done Done Sent false
15058 4496 Acked Write 6 Done Done Sent false
15059 4497 Acked Read 6 Done Done Sent false
15060 4498 Acked Read 7 Done Done Sent false
15061 4499 Acked Write 9 Done Done Sent false
15062 4500 Acked Write 9 Done Done Sent false
15063 4501 Acked Write 7 Done Done Sent false
15064 4502 Acked Write 8 Done Done Sent false
15065 4503 Acked Write 1 Done Done Sent false
15066 4504 Acked Read 9 Done Done Sent false
15067 4505 Acked Read 3 Done Done Sent false
15068 4506 Acked Read 6 Done Done Sent false
15069 4507 Acked Read 1 Done Done Sent false
15070 4508 Acked Write 5 Done Done Sent false
15071 4509 Acked Read 9 Done Done Sent false
15072 4510 Acked Write 3 Done Done Sent false
15073 4511 Acked Write 5 Done Done Sent false
15074 4512 Acked Write 8 Done Done Sent false
15075 4513 Acked Write 10 Done Done Sent false
15076 4514 Acked Read 8 Done Done Sent false
15077 4515 Acked Read 5 Done Done Sent false
15078 4516 Acked Flush 0 Done Done Sent false
15079 4517 Acked Read 8 Done Done Sent false
15080 4518 Acked Read 4 Done Done Sent false
15081 4519 Acked Read 10 Done Done Sent false
15082 4520 Acked Read 5 Done Done Sent false
15083 4521 Acked Read 5 Done Done Sent false
15084 4522 Acked Read 7 Done Done Sent false
15085 4523 Acked Read 7 Done Done Sent false
15086 4524 Acked Read 6 Done Done Sent false
15087 4525 Acked Read 9 Done Done Sent false
15088 4526 Acked Write 4 Done Done Sent false
15089 4527 Acked Read 10 Done Done Sent false
15090 4528 Acked Write 6 Done Done Sent false
15091 4529 Acked Flush 0 Done Done Sent false
15092 4530 Acked Flush 0 Done Done Sent false
15093 4531 Acked Read 5 Done Done Sent false
15094 4532 Acked Read 8 Done Done Sent false
15095 4533 Acked Write 3 Done Done Sent false
15096 4534 Acked Read 6 Done Done Sent false
15097 4535 Acked Write 9 Done Done Sent false
15098 4536 Acked Flush 0 Done Done Sent false
15099 4537 Acked Read 9 Done Done Sent false
15100 4538 Acked Read 6 Done Done Sent false
15101 4539 Acked Read 10 Done Done Sent false
15102 4540 Acked Read 1 Done Done Sent false
15103 4541 Acked Read 2 Done Done Sent false
15104 4542 Acked Write 7 Done Done Sent false
15105 4543 Acked Read 7 Done Done Sent false
15106 4544 Acked Write 8 Done Done Sent false
15107 4545 Acked Read 2 Done Done Sent false
15108 4546 Acked Write 3 Done Done Sent false
15109 4547 Acked Read 6 Done Done Sent false
15110 4548 Acked Write 5 Done Done Sent false
15111 4549 Acked Write 8 Done Done Sent false
15112 4550 Acked Read 10 Done Done Sent false
15113 4551 Acked Write 5 Done Done Sent false
15114 4552 Acked Write 10 Done Done Sent false
15115 4553 Acked Flush 0 Done Done Sent false
15116 4554 Acked Write 8 Done Done Sent false
15117 4555 Acked Read 5 Done Done Sent false
15118 4556 Acked Flush 0 Done Done Sent false
15119 4557 Acked Read 3 Done Done Sent false
15120 4558 Acked Read 9 Done Done Sent false
15121 4559 Acked Flush 0 Done Done Sent false
15122 4560 Acked Write 3 Done Done Sent false
15123 4561 Acked Read 7 Done Done Sent false
15124 4562 Acked Flush 0 Done Done Sent false
15125 4563 Acked Read 6 Done Done Sent false
15126 4564 Acked Write 7 Done Done Sent false
15127 4565 Acked Write 2 Done Done Sent false
15128 4566 Acked Read 3 Done Done Sent false
15129 4567 Acked Read 2 Done Done Sent false
15130 4568 Acked Read 2 Done Done Sent false
15131 4569 Acked Read 2 Done Done Sent false
15132 4570 Acked Read 8 Done Done Sent false
15133 4571 Acked Read 3 Done Done Sent false
15134 4572 Acked Write 10 Done Done Sent false
15135 4573 Acked Read 7 Done Done Sent false
15136 4574 Acked Read 5 Done Done Sent false
15137 4575 Acked Flush 0 Done Done Sent false
15138 4576 Acked Write 10 Done Done Sent false
15139 4577 Acked Read 10 Done Done Sent false
15140 4578 Acked Read 1 Done Done Sent false
15141 4579 Acked Read 6 Done Done Sent false
15142 4580 Acked Write 1 Done Done Sent false
15143 4581 Acked Write 10 Done Done Sent false
15144 4582 Acked Read 6 Done Done Sent false
15145 4583 Acked Read 9 Done Done Sent false
15146 4584 Acked Read 9 Done Done Sent false
15147 4585 Acked Flush 0 Done Done Sent false
15148 4586 Acked Write 6 Done Done Sent false
15149 4587 Acked Write 2 Done Done Sent false
15150 4588 Acked Read 3 Done Done Sent false
15151 4589 Acked Write 1 Done Done Sent false
15152 4590 Acked Read 10 Done Done Sent false
15153 4591 Acked Read 7 Done Done Sent false
15154 4592 Acked Read 7 Done Done Sent false
15155 4593 Acked Read 5 Done Done Sent false
15156 4594 Acked Read 10 Done Done Sent false
15157 4595 Acked Write 5 Done Done Sent false
15158 4596 Acked Flush 0 Done Done Sent false
15159 4597 Acked Read 5 Done Done Sent false
15160 4598 Acked Read 10 Done Done Sent false
15161 4599 Acked Read 10 Done Done Sent false
15162 4600 Acked Write 10 Done Done Sent false
15163 4601 Acked Write 3 Done Done Sent false
15164 4602 Acked Write 1 Done Done Sent false
15165 4603 Acked Write 8 Done Done Sent false
15166 4604 Acked Read 2 Done Done Sent false
15167 4605 Acked Read 5 Done Done Sent false
15168 4606 Acked Flush 0 Done Done Sent false
15169 4607 Acked Read 3 Done Done Sent false
15170 4608 Acked Write 2 Done Done Sent false
15171 4609 Acked Read 1 Done Done Sent false
15172 4610 Acked Write 10 Done Done Sent false
15173 4611 Acked Write 3 Done Done Sent false
15174 4612 Acked Read 3 Done Done Sent false
15175 4613 Acked Read 10 Done Done Sent false
15176 4614 Acked Write 8 Done Done Sent false
15177 4615 Acked Write 8 Done Done Sent false
15178 4616 Acked Read 1 Done Done Sent false
15179 4617 Acked Flush 0 Done Done Sent false
15180 4618 Acked Read 9 Done Done Sent false
15181 4619 Acked Write 4 Done Done Sent false
15182 STATES DS:0 DS:1 DS:2 TOTAL
15183 Sent 0 0 301 301
15184 Done 301 301 0 602
15185 Skipped 0 0 0 0
15186 Error 0 0 0 0
15187 Last Flush: Some(JobId(4617)) Some(JobId(4617)) Some(JobId(4318))
15188 Downstairs last five completed: 4318 4317 4316 4315 4314
15189 Upstairs last five completed: 4619 4618 4617 4616 4615
15190 CLIENT: Up:0 ds:301 act:2
151912025-02-28T06:04:19.666ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
151922025-02-28T06:04:20.630ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
151932025-02-28T06:04:20.630ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41280) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
151942025-02-28T06:04:20.631ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
151952025-02-28T06:04:20.632ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
151962025-02-28T06:04:20.632ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
151972025-02-28T06:04:20.632ZINFOcrucible: send last flush ID to this DS: Some(JobId(4318)) = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
151982025-02-28T06:04:20.632ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
151992025-02-28T06:04:20.632ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4318)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
152002025-02-28T06:04:20.632ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4318)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
15201 ----------------------------------------------------------------
15202 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15203 Downstairs last five completed: 4620 4619 4618 4617 4616
15204 Upstairs last five completed: 4620 4619 4618 4617 4616
15205 CLIENT: Up:0 ds:0 act:3
15206 Replay: All jobs finished
15207 Main task has work to do, go find it
15208 got dsc Stop(0)
15209 stop 0
15210 Feb 28 06:04:22.880 INFO request completed, latency_us: 409, [8810] Got stop action so:false kr:false
15211 response_code: 204, uri: /stop/cid/0, method: POST, req_id: 848f1a7b-d1dd-4d0a-85bd-9f8dd8651c02, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
152122025-02-28T06:04:22.880ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
152132025-02-28T06:04:22.880ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
152142025-02-28T06:04:22.880ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
15215 Feb 28 06:04:22.880 INFO request completed, latency_us: 121, response_code: 200, uri: /state/cid/0, method: GET, req_id: dd9c85eb-9905-4a27-aa4d-dd1b3e7fa0db, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
152162025-02-28T06:04:22.881ZINFOcrucible: client task is exiting = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
15217 [8810] Exited with: ExitStatus(unix_wait_status(9))
15218 [8810][0] reports Exit
15219 Feb 28 06:04:26.882 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/0, method: GET, req_id: 93353865-ce57-4863-81ee-7f723bcf5000, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
15220 001/300 Write block 2615 len 16384 data: 2 2 2 2
15221 002/300 Write block 4874 len 40960 data: 2 3 4 4 5 6 4 5 4 4
15222 003/300 Write block 3752 len 4096 data: 4
15223 004/300 Flush
15224 005/300 Flush
15225 006/300 Read block 2985 len 24576
15226 007/300 Write block 242 len 16384 data: 5 4 4 4
15227 008/300 Write block 1564 len 24576 data: 2 2 3 2 3 3
15228 009/300 Write block 2960 len 24576 data: 3 4 5 4 4 5
15229 010/300 Write block 3266 len 24576 data: 2 2 2 3 3 3
15230 011/300 Write block 4596 len 8192 data: 5 5
15231 012/300 Read block 2327 len 40960
15232 013/300 Read block 1264 len 36864
15233 014/300 Read block 3970 len 20480
15234 015/300 Read block 3741 len 20480
15235 016/300 Read block 3986 len 20480
15236 017/300 Read block 1323 len 4096
15237 018/300 Read block 1125 len 16384
15238 019/300 Read block 2907 len 24576
15239 020/300 Read block 4384 len 16384
15240 021/300 Read block 3840 len 24576
15241 022/300 Write block 4957 len 40960 data: 5 5 4 3 4 4 3 3 4 3
15242 023/300 Read block 3772 len 4096
15243 024/300 Write block 1712 len 28672 data: 5 5 5 4 2 2 2
15244 025/300 Read block 1236 len 8192
15245 026/300 Read block 2956 len 24576
15246 027/300 Write block 303 len 4096 data: 3
15247 028/300 Read block 3361 len 40960
15248 029/300 Read block 2009 len 4096
15249 030/300 Read block 3634 len 28672
15250 031/300 Write block 792 len 12288 data: 5 5 5
15251 032/300 Write block 1575 len 4096 data: 4
15252 033/300 Read block 562 len 40960
15253 034/300 Write block 3035 len 12288 data: 4 4 4
15254 035/300 Flush
15255 036/300 Flush
15256 037/300 Read block 853 len 40960
15257 038/300 Write block 1660 len 4096 data: 4
15258 039/300 Write block 2124 len 20480 data: 5 4 4 3 4
15259 040/300 Read block 297 len 24576
15260 041/300 Read block 4940 len 8192
15261 042/300 Read block 3165 len 12288
15262 043/300 Write block 4968 len 20480 data: 3 3 3 4 4
15263 044/300 Read block 1300 len 32768
15264 045/300 Read block 565 len 16384
15265 046/300 Write block 1843 len 20480 data: 3 4 4 4 3
15266 047/300 Read block 4682 len 12288
15267 048/300 Write block 3563 len 40960 data: 2 2 2 2 2 3 3 3 3 2
15268 049/300 Write block 1368 len 4096 data: 3
15269 050/300 Flush
15270 051/300 Write block 4502 len 24576 data: 4 5 5 5 6 5
15271 052/300 Write block 260 len 28672 data: 2 2 2 2 3 3 3
15272 053/300 Flush
15273 054/300 Read block 169 len 32768
15274 055/300 Flush
15275 056/300 Flush
15276 057/300 Read block 3856 len 40960
15277 058/300 Write block 757 len 8192 data: 3 3
15278 059/300 Write block 2537 len 28672 data: 2 2 2 2 2 2 2
15279 060/300 Write block 1116 len 20480 data: 2 2 2 2 2
15280 061/300 Write block 3242 len 24576 data: 2 2 3 3 3 3
15281 062/300 Read block 2246 len 16384
15282 063/300 Read block 3288 len 16384
15283 064/300 Read block 3528 len 16384
15284 065/300 Read block 1468 len 32768
15285 066/300 Write block 352 len 24576 data: 4 4 3 4 4 2
15286 067/300 Write block 2824 len 20480 data: 3 3 3 2 2
15287 068/300 Read block 2537 len 16384
15288 069/300 Read block 473 len 8192
15289 070/300 Read block 512 len 36864
15290 071/300 Write block 2996 len 24576 data: 3 3 3 3 2 2
15291 072/300 Write block 496 len 8192 data: 5 4
15292 073/300 Read block 869 len 20480
15293 074/300 Write block 1971 len 20480 data: 4 4 4 4 4
15294 075/300 Flush
15295 076/300 Read block 2161 len 16384
15296 077/300 Write block 4625 len 40960 data: 4 4 6 6 5 5 6 6 6 5
15297 078/300 Read block 4160 len 16384
15298 079/300 Read block 2603 len 24576
15299 080/300 Read block 3172 len 28672
15300 081/300 Flush
15301 082/300 Write block 4036 len 24576 data: 3 3 3 3 2 2
15302 083/300 Write block 3197 len 40960 data: 3 3 3 2 3 3 3 3 4 4
15303 084/300 Write block 90 len 28672 data: 2 3 4 4 5 6 3
15304 085/300 Flush
15305 086/300 Write block 1058 len 12288 data: 3 3 3
15306 087/300 Write block 1069 len 40960 data: 3 3 2 2 2 2 2 2 4 5
15307 088/300 Read block 1529 len 8192
15308 089/300 Read block 2791 len 32768
15309 090/300 Flush
15310 091/300 Read block 4419 len 4096
15311 092/300 Write block 1889 len 40960 data: 3 3 5 5 5 5 5 4 3 3
15312 093/300 Read block 316 len 8192
15313 094/300 Read block 2150 len 8192
15314 095/300 Flush
15315 096/300 Write block 2164 len 40960 data: 4 5 5 5 6 4 4 3 3 2
15316 097/300 Read block 2696 len 4096
15317 098/300 Write block 4153 len 32768 data: 2 3 3 3 3 4 5 5
15318 099/300 Read block 497 len 40960
15319 100/300 Flush
15320 101/300 Write block 4482 len 40960 data: 4 4 4 4 4 4 3 3 3 2
15321 102/300 Read block 404 len 12288
15322 103/300 Flush
15323 104/300 Read block 2735 len 36864
15324 105/300 Read block 4022 len 4096
15325 106/300 Read block 1437 len 40960
15326 107/300 Write block 3473 len 20480 data: 3 3 4 3 3
15327 108/300 Write block 1647 len 40960 data: 2 2 2 2 2 2 2 3 2 3
15328 109/300 Write block 3728 len 16384 data: 3 2 2 3
15329 110/300 Read block 84 len 8192
15330 111/300 Write block 3869 len 24576 data: 5 4 3 3 5 5
15331 112/300 Write block 1247 len 4096 data: 3
15332 113/300 Write block 3348 len 12288 data: 4 3 3
15333 114/300 Write block 327 len 4096 data: 2
15334 115/300 Read block 3276 len 4096
15335 116/300 Read block 3936 len 36864
15336 117/300 Flush
15337 118/300 Read block 3667 len 32768
15338 119/300 Read block 4789 len 32768
15339 120/300 Write block 3545 len 28672 data: 2 3 3 4 4 4 4
15340 121/300 Write block 219 len 4096 data: 6
15341 122/300 Write block 4427 len 24576 data: 4 5 5 5 5 3
15342 123/300 Read block 9 len 16384
15343 124/300 Read block 352 len 8192
15344 125/300 Read block 787 len 28672
15345 126/300 Write block 483 len 32768 data: 3 3 3 4 3 2 2 2
15346 127/300 Read block 3714 len 32768
15347 128/300 Write block 2803 len 32768 data: 3 3 4 5 5 4 4 4
15348 129/300 Read block 4868 len 40960
15349 130/300 Write block 3424 len 36864 data: 2 2 2 2 2 2 3 2 2
15350 131/300 Write block 4728 len 36864 data: 3 4 4 4 4 4 4 3 4
15351 132/300 Read block 224 len 28672
15352 133/300 Write block 2000 len 24576 data: 4 3 4 4 3 3
15353 134/300 Read block 1005 len 20480
15354 135/300 Read block 447 len 36864
15355 136/300 Write block 3480 len 40960 data: 5 5 5 5 4 4 4 4 4 5
15356 137/300 Read block 3895 len 24576
15357 138/300 Flush
15358 139/300 Read block 3782 len 12288
15359 140/300 Write block 798 len 40960 data: 5 4 4 3 2 2 2 3 3 3
15360 141/300 Read block 2502 len 28672
15361 142/300 Write block 4677 len 32768 data: 2 4 4 4 4 3 3 3
15362 143/300 Read block 1591 len 28672
15363 144/300 Read block 4015 len 4096
15364 145/300 Write block 3625 len 4096 data: 3
15365 146/300 Read block 1614 len 40960
15366 147/300 Read block 1166 len 16384
15367 148/300 Flush
15368 149/300 Read block 3409 len 32768
15369 150/300 Write block 3628 len 32768 data: 3 3 2 2 2 3 3 3
15370 151/300 Read block 4889 len 28672
15371 152/300 Read block 648 len 28672
15372 153/300 Read block 2662 len 24576
15373 154/300 Read block 1510 len 28672
15374 155/300 Read block 489 len 28672
15375 156/300 Write block 659 len 32768 data: 6 6 6 6 5 5 5 3
15376 157/300 Write block 4077 len 20480 data: 3 3 4 3 4
15377 158/300 Write block 2064 len 20480 data: 4 4 3 3 3
15378 159/300 Read block 1616 len 32768
15379 160/300 Read block 80 len 24576
15380 161/300 Read block 3739 len 28672
15381 162/300 Write block 3199 len 8192 data: 4 3
15382 163/300 Read block 3208 len 16384
15383 164/300 Write block 4377 len 36864 data: 5 5 5 5 4 2 4 5 5
15384 165/300 Write block 1720 len 16384 data: 2 3 3 4
15385 166/300 Read block 4533 len 16384
15386 167/300 Write block 2283 len 16384 data: 6 4 4 3
15387 168/300 Write block 3804 len 24576 data: 3 3 3 3 3 4
15388 169/300 Read block 619 len 40960
15389 170/300 Read block 2706 len 40960
15390 171/300 Read block 4421 len 16384
15391 172/300 Read block 4801 len 32768
15392 173/300 Read block 2555 len 36864
15393 174/300 Read block 2902 len 12288
15394 175/300 Read block 1299 len 8192
15395 176/300 Read block 687 len 36864
15396 177/300 Write block 1843 len 4096 data: 4
15397 178/300 Read block 4886 len 28672
15398 179/300 Write block 1039 len 16384 data: 4 5 4 4
15399 180/300 Read block 4518 len 24576
15400 181/300 Read block 3889 len 12288
15401 182/300 Write block 4711 len 32768 data: 2 2 2 2 2 2 3 3
15402 183/300 Write block 4038 len 32768 data: 4 4 3 3 2 4 4 4
15403 184/300 Write block 4838 len 12288 data: 6 4 4
15404 185/300 Read block 1227 len 24576
15405 186/300 Read block 2045 len 12288
15406 187/300 Read block 2185 len 28672
15407 188/300 Read block 1049 len 4096
15408 189/300 Read block 4907 len 4096
15409 190/300 Write block 3479 len 32768 data: 5 6 6 6 6 5 5 5
15410 191/300 Read block 750 len 12288
15411 192/300 Read block 1902 len 20480
15412 193/300 Read block 2268 len 8192
15413 194/300 Read block 2938 len 4096
15414 195/300 Flush
15415 196/300 Write block 2230 len 20480 data: 5 7 6 6 8
15416 197/300 Read block 3960 len 8192
15417 198/300 Write block 3717 len 40960 data: 4 6 5 6 6 6 6 6 4 3
15418 199/300 Write block 3559 len 8192 data: 4 3
15419 200/300 Write block 4349 len 40960 data: 4 3 3 4 4 5 5 4 5 5
15420 201/300 Write block 4369 len 12288 data: 3 3 4
15421 202/300 Write block 3538 len 4096 data: 5
15422 203/300 Write block 4619 len 4096 data: 3
15423 204/300 Write block 2091 len 40960 data: 5 3 4 4 4 5 6 6 5 5
15424 205/300 Write block 2694 len 8192 data: 4 4
15425 206/300 Read block 2589 len 20480
15426 207/300 Write block 4609 len 24576 data: 2 2 2 2 3 3
15427 208/300 Write block 2442 len 20480 data: 5 4 4 5 5
15428 209/300 Read block 3601 len 28672
15429 210/300 Write block 1511 len 24576 data: 2 3 3 4 3 3
15430 211/300 Read block 462 len 16384
15431 212/300 Write block 2923 len 20480 data: 4 4 4 4 3
15432 213/300 Read block 1724 len 12288
15433 214/300 Read block 492 len 24576
15434 215/300 Read block 1535 len 40960
15435 216/300 Read block 2547 len 16384
15436 217/300 Flush
15437 218/300 Write block 2819 len 32768 data: 5 4 3 2 2 4 4 4
15438 219/300 Write block 3207 len 20480 data: 4 3 3 2 2
15439 220/300 Flush
15440 221/300 Read block 4016 len 8192
15441 222/300 Read block 4733 len 32768
15442 223/300 Write block 4882 len 36864 data: 5 5 4 4 4 6 7 8 7
15443 224/300 Write block 2174 len 16384 data: 3 3 4 4
15444 225/300 Read block 3517 len 20480
15445 226/300 Flush
15446 227/300 Write block 2472 len 24576 data: 5 5 5 5 5 5
15447 228/300 Write block 120 len 12288 data: 3 3 4
15448 229/300 Write block 3143 len 16384 data: 4 4 3 2
15449 230/300 Write block 3335 len 4096 data: 4
15450 231/300 Write block 2657 len 16384 data: 4 4 4 3
15451 232/300 Write block 3093 len 28672 data: 3 3 2 3 3 4 4
15452 233/300 Write block 664 len 40960 data: 6 6 4 5 5 5 6 6 7 6
15453 234/300 Write block 1019 len 32768 data: 6 6 6 5 4 4 6 6
15454 235/300 Flush
15455 236/300 Write block 3842 len 28672 data: 4 3 4 4 3 3 4
15456 237/300 Write block 2455 len 28672 data: 5 4 4 4 3 3 2
15457 238/300 Read block 2506 len 24576
15458 239/300 Write block 215 len 4096 data: 4
15459 240/300 Read block 1016 len 32768
15460 241/300 Write block 4345 len 36864 data: 5 6 5 5 5 4 4 5 5
15461 242/300 Read block 3175 len 12288
15462 243/300 Read block 2439 len 8192
15463 244/300 Write block 3936 len 4096 data: 4
15464 245/300 Flush
15465 246/300 Read block 586 len 4096
15466 247/300 Write block 2011 len 16384 data: 3 3 3 3
15467 248/300 Read block 2803 len 24576
15468 249/300 Write block 3201 len 28672 data: 4 4 4 4 5 5 5
15469 250/300 Read block 431 len 32768
15470 251/300 Read block 4085 len 4096
15471 252/300 Write block 2937 len 32768 data: 4 5 5 5 4 4 3 2
15472 253/300 Write block 1049 len 36864 data: 5 5 4 5 5 6 5 4 3
15473 254/300 Flush
15474 255/300 Write block 1253 len 12288 data: 5 6 7
15475 256/300 Write block 2393 len 12288 data: 4 3 2
15476 257/300 Write block 4342 len 40960 data: 6 5 5 6 7 6 6 6 5 5
15477 258/300 Read block 4046 len 8192
15478 259/300 Read block 1227 len 20480
15479 260/300 Write block 1256 len 4096 data: 7
15480 261/300 Write block 4135 len 40960 data: 2 2 2 3 3 3 4 4 5 4
15481 262/300 Flush
15482 263/300 Write block 319 len 4096 data: 5
15483 264/300 Write block 3468 len 4096 data: 2
15484 265/300 Write block 2318 len 40960 data: 6 6 5 4 4 4 3 3 4 3
15485 266/300 Read block 2980 len 8192
15486 267/300 Read block 2460 len 36864
15487 268/300 Flush
15488 269/300 Write block 2256 len 20480 data: 3 4 3 4 4
15489 270/300 Read block 2740 len 24576
15490 271/300 Read block 2460 len 16384
15491 272/300 Write block 2739 len 12288 data: 3 5 5
15492 273/300 Flush
15493 274/300 Read block 1428 len 12288
15494 275/300 Read block 4913 len 28672
15495 276/300 Read block 1159 len 32768
15496 277/300 Write block 1018 len 36864 data: 5 7 7 7 6 5 5 7 7
15497 278/300 Write block 3507 len 12288 data: 2 2 2
15498 279/300 Read block 1972 len 40960
15499 280/300 Read block 2980 len 28672
15500 281/300 Write block 2385 len 24576 data: 2 2 2 2 3 3
15501 282/300 Write block 876 len 16384 data: 3 3 2 2
15502 283/300 Read block 3256 len 32768
15503 284/300 Write block 775 len 32768 data: 3 3 3 2 3 3 3 3
15504 285/300 Read block 4573 len 20480
15505 286/300 Write block 3480 len 8192 data: 7 7
15506 287/300 Write block 735 len 40960 data: 3 4 3 3 4 4 3 3 3 4
15507 288/300 Flush
15508 289/300 Read block 447 len 40960
15509 290/300 Read block 4570 len 32768
15510 291/300 Write block 4171 len 32768 data: 3 3 3 4 3 3 2 3
15511 292/300 Read block 151 len 24576
15512 293/300 Write block 4032 len 40960 data: 4 3 3 3 4 4 5 5 4 4
15513 294/300 Flush
15514 295/300 Write block 4086 len 40960 data: 3 3 3 3 3 3 2 2 2 4
15515 296/300 Write block 1836 len 24576 data: 3 3 4 5 5 3
15516 297/300 Write block 2986 len 32768 data: 4 4 4 4 6 6 6 3
15517 298/300 Write block 2992 len 40960 data: 7 4 3 3 4 4 4 4 3 3
15518 299/300 Write block 4849 len 12288 data: 2 2 3
15519 300/300 Write block 1722 len 16384 data: 4 5 4 4
15520 301/300 Write block 4076 len 20480 data: 4 4 4 5 4
15521 Main task has work to do, go find it
15522 got dsc Start(0)
15523 start 0
15524 Feb 28 06:04:27.121 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/0, method: POST, req_id: 5e59e4b7-331f-488b-8747-b0f6f13e196c, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
15525 Replay: started 0, returned:Ok(())
15526 ----------------------------------------------------------------
15527 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15528 States: Offline Active Active
15529 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15530 4621 Acked Write 4 Sent Done Done false
15531 4622 Acked Write 10 Sent Done Done false
15532 4623 Acked Write 1 Sent Done Done false
15533 4624 Acked Flush 0 Sent Done Done false
15534 4625 Acked Flush 0 Sent Done Done false
15535 4626 Acked Read 6 Sent Done Done false
15536 4627 Acked Write 4 Sent Done Done false
15537 4628 Acked Write 6 Sent Done Done false
15538 4629 Acked Write 6 Sent Done Done false
15539 4630 Acked Write 6 Sent Done Done false
15540 4631 Acked Write 2 Sent Done Done false
15541 4632 Acked Read 10 Sent Done Done false
15542 4633 Acked Read 9 Sent Done Done false
15543 4634 Acked Read 5 Sent Done Done false
15544 4635 Acked Read 5 Sent Done Done false
15545 4636 Acked Read 5 Sent Done Done false
15546 4637 Acked Read 1 Sent Done Done false
15547 4638 Acked Read 4 Sent Done Done false
15548 4639 Acked Read 6 Sent Done Done false
15549 4640 Acked Read 4 Sent Done Done false
15550 4641 Acked Read 6 Sent Done Done false
15551 4642 Acked Write 10 Sent Done Done false
15552 4643 Acked Read 1 Sent Done Done false
15553 4644 Acked Write 7 Sent Done Done false
15554 4645 Acked Read 2 Sent Done Done false
15555 4646 Acked Read 6 Sent Done Done false
15556 4647 Acked Write 1 Sent Done Done false
15557 4648 Acked Read 10 Sent Done Done false
15558 4649 Acked Read 1 Sent Done Done false
15559 4650 Acked Read 7 Sent Done Done false
15560 4651 Acked Write 3 Sent Done Done false
15561 4652 Acked Write 1 Sent Done Done false
15562 4653 Acked Read 10 Sent Done Done false
15563 4654 Acked Write 3 Sent Done Done false
15564 4655 Acked Flush 0 Sent Done Done false
15565 4656 Acked Flush 0 Sent Done Done false
15566 4657 Acked Read 10 Sent Done Done false
15567 4658 Acked Write 1 Sent Done Done false
15568 4659 Acked Write 5 Sent Done Done false
15569 4660 Acked Read 6 Sent Done Done false
15570 4661 Acked Read 2 Sent Done Done false
15571 4662 Acked Read 3 Sent Done Done false
15572 4663 Acked Write 5 Sent Done Done false
15573 4664 Acked Read 8 Sent Done Done false
15574 4665 Acked Read 4 Sent Done Done false
15575 4666 Acked Write 5 Sent Done Done false
15576 4667 Acked Read 3 Sent Done Done false
15577 4668 Acked Write 10 Sent Done Done false
15578 4669 Acked Write 1 Sent Done Done false
15579 4670 Acked Flush 0 Sent Done Done false
15580 4671 Acked Write 6 Sent Done Done false
15581 4672 Acked Write 7 Sent Done Done false
15582 4673 Acked Flush 0 Sent Done Done false
15583 4674 Acked Read 8 Sent Done Done false
15584 4675 Acked Flush 0 Sent Done Done false
15585 4676 Acked Flush 0 Sent Done Done false
15586 4677 Acked Read 10 Sent Done Done false
15587 4678 Acked Write 2 Sent Done Done false
15588 4679 Acked Write 7 Sent Done Done false
15589 4680 Acked Write 5 Sent Done Done false
15590 4681 Acked Write 6 Sent Done Done false
15591 4682 Acked Read 4 Sent Done Done false
15592 4683 Acked Read 4 Sent Done Done false
15593 4684 Acked Read 4 Sent Done Done false
15594 4685 Acked Read 8 Sent Done Done false
15595 4686 Acked Write 6 Sent Done Done false
15596 4687 Acked Write 5 Sent Done Done false
15597 4688 Acked Read 4 Sent Done Done false
15598 4689 Acked Read 2 Sent Done Done false
15599 4690 Acked Read 9 Sent Done Done false
15600 4691 Acked Write 6 Sent Done Done false
15601 4692 Acked Write 2 Sent Done Done false
15602 4693 Acked Read 5 Sent Done Done false
15603 4694 Acked Write 5 Sent Done Done false
15604 4695 Acked Flush 0 Sent Done Done false
15605 4696 Acked Read 4 Sent Done Done false
15606 4697 Acked Write 10 Sent Done Done false
15607 4698 Acked Read 4 Sent Done Done false
15608 4699 Acked Read 6 Sent Done Done false
15609 4700 Acked Read 7 Sent Done Done false
15610 4701 Acked Flush 0 Sent Done Done false
15611 4702 Acked Write 6 Sent Done Done false
15612 4703 Acked Write 10 Sent Done Done false
15613 4704 Acked Write 7 Sent Done Done false
15614 4705 Acked Flush 0 Sent Done Done false
15615 4706 Acked Write 3 Sent Done Done false
15616 4707 Acked Write 10 Sent Done Done false
15617 4708 Acked Read 2 Sent Done Done false
15618 4709 Acked Read 8 Sent Done Done false
15619 4710 Acked Flush 0 Sent Done Done false
15620 4711 Acked Read 1 Sent Done Done false
15621 4712 Acked Write 10 Sent Done Done false
15622 4713 Acked Read 2 Sent Done Done false
15623 4714 Acked Read 2 Sent Done Done false
15624 4715 Acked Flush 0 Sent Done Done false
15625 4716 Acked Write 10 Sent Done Done false
15626 4717 Acked Read 1 Sent Done Done false
15627 4718 Acked Write 8 Sent Done Done false
15628 4719 Acked Read 10 Sent Done Done false
15629 4720 Acked Flush 0 Sent Done Done false
15630 4721 Acked Write 10 Sent Done Done false
15631 4722 Acked Read 3 Sent Done Done false
15632 4723 Acked Flush 0 Sent Done Done false
15633 4724 Acked Read 9 Sent Done Done false
15634 4725 Acked Read 1 Sent Done Done false
15635 4726 Acked Read 10 Sent Done Done false
15636 4727 Acked Write 5 Sent Done Done false
15637 4728 Acked Write 10 Sent Done Done false
15638 4729 Acked Write 4 Sent Done Done false
15639 4730 Acked Read 2 Sent Done Done false
15640 4731 Acked Write 6 Sent Done Done false
15641 4732 Acked Write 1 Sent Done Done false
15642 4733 Acked Write 3 Sent Done Done false
15643 4734 Acked Write 1 Sent Done Done false
15644 4735 Acked Read 1 Sent Done Done false
15645 4736 Acked Read 9 Sent Done Done false
15646 4737 Acked Flush 0 Sent Done Done false
15647 4738 Acked Read 8 Sent Done Done false
15648 4739 Acked Read 8 Sent Done Done false
15649 4740 Acked Write 7 Sent Done Done false
15650 4741 Acked Write 1 Sent Done Done false
15651 4742 Acked Write 6 Sent Done Done false
15652 4743 Acked Read 4 Sent Done Done false
15653 4744 Acked Read 2 Sent Done Done false
15654 4745 Acked Read 7 Sent Done Done false
15655 4746 Acked Write 8 Sent Done Done false
15656 4747 Acked Read 8 Sent Done Done false
15657 4748 Acked Write 8 Sent Done Done false
15658 4749 Acked Read 10 Sent Done Done false
15659 4750 Acked Write 9 Sent Done Done false
15660 4751 Acked Write 9 Sent Done Done false
15661 4752 Acked Read 7 Sent Done Done false
15662 4753 Acked Write 6 Sent Done Done false
15663 4754 Acked Read 5 Sent Done Done false
15664 4755 Acked Read 9 Sent Done Done false
15665 4756 Acked Write 10 Sent Done Done false
15666 4757 Acked Read 6 Sent Done Done false
15667 4758 Acked Flush 0 Sent Done Done false
15668 4759 Acked Read 3 Sent Done Done false
15669 4760 Acked Write 10 Sent Done Done false
15670 4761 Acked Read 7 Sent Done Done false
15671 4762 Acked Write 8 Sent Done Done false
15672 4763 Acked Read 7 Sent Done Done false
15673 4764 Acked Read 1 Sent Done Done false
15674 4765 Acked Write 1 Sent Done Done false
15675 4766 Acked Read 10 Sent Done Done false
15676 4767 Acked Read 4 Sent Done Done false
15677 4768 Acked Flush 0 Sent Done Done false
15678 4769 Acked Read 8 Sent Done Done false
15679 4770 Acked Write 8 Sent Done Done false
15680 4771 Acked Read 7 Sent Done Done false
15681 4772 Acked Read 7 Sent Done Done false
15682 4773 Acked Read 6 Sent Done Done false
15683 4774 Acked Read 7 Sent Done Done false
15684 4775 Acked Read 7 Sent Done Done false
15685 4776 Acked Write 8 Sent Done Done false
15686 4777 Acked Write 5 Sent Done Done false
15687 4778 Acked Write 5 Sent Done Done false
15688 4779 Acked Read 8 Sent Done Done false
15689 4780 Acked Read 6 Sent Done Done false
15690 4781 Acked Read 7 Sent Done Done false
15691 4782 Acked Write 2 Sent Done Done false
15692 4783 Acked Read 4 Sent Done Done false
15693 4784 Acked Write 9 Sent Done Done false
15694 4785 Acked Write 4 Sent Done Done false
15695 4786 Acked Read 4 Sent Done Done false
15696 4787 Acked Write 4 Sent Done Done false
15697 4788 Acked Write 6 Sent Done Done false
15698 4789 Acked Read 10 Sent Done Done false
15699 4790 Acked Read 10 Sent Done Done false
15700 4791 Acked Read 4 Sent Done Done false
15701 4792 Acked Read 8 Sent Done Done false
15702 4793 Acked Read 9 Sent Done Done false
15703 4794 Acked Read 3 Sent Done Done false
15704 4795 Acked Read 2 Sent Done Done false
15705 4796 Acked Read 9 Sent Done Done false
15706 4797 Acked Write 1 Sent Done Done false
15707 4798 Acked Read 7 Sent Done Done false
15708 4799 Acked Write 4 Sent Done Done false
15709 4800 Acked Read 6 Sent Done Done false
15710 4801 Acked Read 3 Sent Done Done false
15711 4802 Acked Write 8 Sent Done Done false
15712 4803 Acked Write 8 Sent Done Done false
15713 4804 Acked Write 3 Sent Done Done false
15714 4805 Acked Read 6 Sent Done Done false
15715 4806 Acked Read 3 Sent Done Done false
15716 4807 Acked Read 7 Sent Done Done false
15717 4808 Acked Read 1 Sent Done Done false
15718 4809 Acked Read 1 Sent Done Done false
15719 4810 Acked Write 8 Sent Done Done false
15720 4811 Acked Read 3 Sent Done Done false
15721 4812 Acked Read 5 Sent Done Done false
15722 4813 Acked Read 2 Sent Done Done false
15723 4814 Acked Read 1 Sent Done Done false
15724 4815 Acked Flush 0 Sent Done Done false
15725 4816 Acked Write 5 Sent Done Done false
15726 4817 Acked Read 2 Sent Done Done false
15727 4818 Acked Write 10 Sent Done Done false
15728 4819 Acked Write 2 Sent Done Done false
15729 4820 Acked Write 10 Sent Done Done false
15730 4821 Acked Write 3 Sent Done Done false
15731 4822 Acked Write 1 Sent Done Done false
15732 4823 Acked Write 1 Sent Done Done false
15733 4824 Acked Write 10 Sent Done Done false
15734 4825 Acked Write 2 Sent Done Done false
15735 4826 Acked Read 5 Sent Done Done false
15736 4827 Acked Write 6 Sent Done Done false
15737 4828 Acked Write 5 Sent Done Done false
15738 4829 Acked Read 7 Sent Done Done false
15739 4830 Acked Write 6 Sent Done Done false
15740 4831 Acked Read 4 Sent Done Done false
15741 4832 Acked Write 5 Sent Done Done false
15742 4833 Acked Read 3 Sent Done Done false
15743 4834 Acked Read 6 Sent Done Done false
15744 4835 Acked Read 10 Sent Done Done false
15745 4836 Acked Read 4 Sent Done Done false
15746 4837 Acked Flush 0 Sent Done Done false
15747 4838 Acked Write 8 Sent Done Done false
15748 4839 Acked Write 5 Sent Done Done false
15749 4840 Acked Flush 0 Sent Done Done false
15750 4841 Acked Read 2 Sent Done Done false
15751 4842 Acked Read 8 Sent Done Done false
15752 4843 Acked Write 9 Sent Done Done false
15753 4844 Acked Write 4 Sent Done Done false
15754 4845 Acked Read 5 Sent Done Done false
15755 4846 Acked Flush 0 Sent Done Done false
15756 4847 Acked Write 6 Sent Done Done false
15757 4848 Acked Write 3 Sent Done Done false
15758 4849 Acked Write 4 Sent Done Done false
15759 4850 Acked Write 1 Sent Done Done false
15760 4851 Acked Write 4 Sent Done Done false
15761 4852 Acked Write 7 Sent Done Done false
15762 4853 Acked Write 10 Sent Done Done false
15763 4854 Acked Write 8 Sent Done Done false
15764 4855 Acked Flush 0 Sent Done Done false
15765 4856 Acked Write 7 Sent Done Done false
15766 4857 Acked Write 7 Sent Done Done false
15767 4858 Acked Read 6 Sent Done Done false
15768 4859 Acked Write 1 Sent Done Done false
15769 4860 Acked Read 8 Sent Done Done false
15770 4861 Acked Write 9 Sent Done Done false
15771 4862 Acked Read 3 Sent Done Done false
15772 4863 Acked Read 2 Sent Done Done false
15773 4864 Acked Write 1 Sent Done Done false
15774 4865 Acked Flush 0 Sent Done Done false
15775 4866 Acked Read 1 Sent Done Done false
15776 4867 Acked Write 4 Sent Done Done false
15777 4868 Acked Read 6 Sent Done Done false
15778 4869 Acked Write 7 Sent Done Done false
15779 4870 Acked Read 8 Sent Done Done false
15780 4871 Acked Read 1 Sent Done Done false
15781 4872 Acked Write 8 Sent Done Done false
15782 4873 Acked Write 9 Sent Done Done false
15783 4874 Acked Flush 0 Sent Done Done false
15784 4875 Acked Write 3 Sent Done Done false
15785 4876 Acked Write 3 Sent Done Done false
15786 4877 Acked Write 10 Sent Done Done false
15787 4878 Acked Read 2 Sent Done Done false
15788 4879 Acked Read 5 Sent Done Done false
15789 4880 Acked Write 1 Sent Done Done false
15790 4881 Acked Write 10 Sent Done Done false
15791 4882 Acked Flush 0 Sent Done Done false
15792 4883 Acked Write 1 Sent Done Done false
15793 4884 Acked Write 1 Sent Done Done false
15794 4885 Acked Write 10 Sent Done Done false
15795 4886 Acked Read 2 Sent Done Done false
15796 4887 Acked Read 9 Sent Done Done false
15797 4888 Acked Flush 0 Sent Done Done false
15798 4889 Acked Write 5 Sent Done Done false
15799 4890 Acked Read 6 Sent Done Done false
15800 4891 Acked Read 4 Sent Done Done false
15801 4892 Acked Write 3 Sent Done Done false
15802 4893 Acked Flush 0 Sent Done Done false
15803 4894 Acked Read 3 Sent Done Done false
15804 4895 Acked Read 7 Sent Done Done false
15805 4896 Acked Read 8 Sent Done Done false
15806 4897 Acked Write 9 Sent Done Done false
15807 4898 Acked Write 3 Sent Done Done false
15808 4899 Acked Read 10 Sent Done Done false
15809 4900 Acked Read 7 Sent Done Done false
15810 4901 Acked Write 6 Sent Done Done false
15811 4902 Acked Write 4 Sent Done Done false
15812 4903 Acked Read 8 Sent Done Done false
15813 4904 Acked Write 8 Sent Done Done false
15814 4905 Acked Read 5 Sent Done Done false
15815 4906 Acked Write 2 Sent Done Done false
15816 4907 Acked Write 10 Sent Done Done false
15817 4908 Acked Flush 0 Sent Done Done false
15818 4909 Acked Read 10 Sent Done Done false
15819 4910 Acked Read 8 Sent Done Done false
15820 4911 Acked Write 8 Sent Done Done false
15821 4912 Acked Read 6 Sent Done Done false
15822 4913 Acked Write 10 Sent Done Done false
15823 4914 Acked Flush 0 Sent Done Done false
15824 4915 Acked Write 10 Sent Done Done false
15825 4916 Acked Write 6 Sent Done Done false
15826 4917 Acked Write 8 Sent Done Done false
15827 4918 Acked Write 10 Sent Done Done false
15828 4919 Acked Write 3 Sent Done Done false
15829 4920 Acked Write 4 Sent Done Done false
15830 4921 Acked Write 5 Sent Done Done false
15831 STATES DS:0 DS:1 DS:2 TOTAL
15832 Sent 301 0 0 301
15833 Done 0 301 301 602
15834 Skipped 0 0 0 0
15835 Error 0 0 0 0
15836 Last Flush: Some(JobId(4620)) Some(JobId(4914)) Some(JobId(4914))
15837 Downstairs last five completed: 4620 4619 4618 4617 4616
15838 Upstairs last five completed: 4921 4920 4919 4918 4917
15839 CLIENT: Up:0 ds:301 act:2
158402025-02-28T06:04:27.681ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
158412025-02-28T06:04:28.681ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
158422025-02-28T06:04:29.684ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
15843 [8810] got start action
15844 [8810] I am going to restart
15845 Starting downstairs at port 8810
15846 Make output file at "/tmp/dsc/downstairs-8810.txt"
15847 [8810][0] reports Starting
15848 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1224)
15849 [8810][0] reports Running
15850 ----------------------------------------------------------------
15851 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15852 States: Offline Active Active
15853 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15854 4621 Acked Write 4 Sent Done Done false
15855 4622 Acked Write 10 Sent Done Done false
15856 4623 Acked Write 1 Sent Done Done false
15857 4624 Acked Flush 0 Sent Done Done false
15858 4625 Acked Flush 0 Sent Done Done false
15859 4626 Acked Read 6 Sent Done Done false
15860 4627 Acked Write 4 Sent Done Done false
15861 4628 Acked Write 6 Sent Done Done false
15862 4629 Acked Write 6 Sent Done Done false
15863 4630 Acked Write 6 Sent Done Done false
15864 4631 Acked Write 2 Sent Done Done false
15865 4632 Acked Read 10 Sent Done Done false
15866 4633 Acked Read 9 Sent Done Done false
15867 4634 Acked Read 5 Sent Done Done false
15868 4635 Acked Read 5 Sent Done Done false
15869 4636 Acked Read 5 Sent Done Done false
15870 4637 Acked Read 1 Sent Done Done false
15871 4638 Acked Read 4 Sent Done Done false
15872 4639 Acked Read 6 Sent Done Done false
15873 4640 Acked Read 4 Sent Done Done false
15874 4641 Acked Read 6 Sent Done Done false
15875 4642 Acked Write 10 Sent Done Done false
15876 4643 Acked Read 1 Sent Done Done false
15877 4644 Acked Write 7 Sent Done Done false
15878 4645 Acked Read 2 Sent Done Done false
15879 4646 Acked Read 6 Sent Done Done false
15880 4647 Acked Write 1 Sent Done Done false
15881 4648 Acked Read 10 Sent Done Done false
15882 4649 Acked Read 1 Sent Done Done false
15883 4650 Acked Read 7 Sent Done Done false
15884 4651 Acked Write 3 Sent Done Done false
15885 4652 Acked Write 1 Sent Done Done false
15886 4653 Acked Read 10 Sent Done Done false
15887 4654 Acked Write 3 Sent Done Done false
15888 4655 Acked Flush 0 Sent Done Done false
15889 4656 Acked Flush 0 Sent Done Done false
15890 4657 Acked Read 10 Sent Done Done false
15891 4658 Acked Write 1 Sent Done Done false
15892 4659 Acked Write 5 Sent Done Done false
15893 4660 Acked Read 6 Sent Done Done false
15894 4661 Acked Read 2 Sent Done Done false
15895 4662 Acked Read 3 Sent Done Done false
15896 4663 Acked Write 5 Sent Done Done false
15897 4664 Acked Read 8 Sent Done Done false
15898 4665 Acked Read 4 Sent Done Done false
15899 4666 Acked Write 5 Sent Done Done false
15900 4667 Acked Read 3 Sent Done Done false
15901 4668 Acked Write 10 Sent Done Done false
15902 4669 Acked Write 1 Sent Done Done false
15903 4670 Acked Flush 0 Sent Done Done false
15904 4671 Acked Write 6 Sent Done Done false
15905 4672 Acked Write 7 Sent Done Done false
15906 4673 Acked Flush 0 Sent Done Done false
15907 4674 Acked Read 8 Sent Done Done false
15908 4675 Acked Flush 0 Sent Done Done false
15909 4676 Acked Flush 0 Sent Done Done false
15910 4677 Acked Read 10 Sent Done Done false
15911 4678 Acked Write 2 Sent Done Done false
15912 4679 Acked Write 7 Sent Done Done false
15913 4680 Acked Write 5 Sent Done Done false
15914 4681 Acked Write 6 Sent Done Done false
15915 4682 Acked Read 4 Sent Done Done false
15916 4683 Acked Read 4 Sent Done Done false
15917 4684 Acked Read 4 Sent Done Done false
15918 4685 Acked Read 8 Sent Done Done false
15919 4686 Acked Write 6 Sent Done Done false
15920 4687 Acked Write 5 Sent Done Done false
15921 4688 Acked Read 4 Sent Done Done false
15922 4689 Acked Read 2 Sent Done Done false
15923 4690 Acked Read 9 Sent Done Done false
15924 4691 Acked Write 6 Sent Done Done false
15925 4692 Acked Write 2 Sent Done Done false
15926 4693 Acked Read 5 Sent Done Done false
15927 4694 Acked Write 5 Sent Done Done false
15928 4695 Acked Flush 0 Sent Done Done false
15929 4696 Acked Read 4 Sent Done Done false
15930 4697 Acked Write 10 Sent Done Done false
15931 4698 Acked Read 4 Sent Done Done false
15932 4699 Acked Read 6 Sent Done Done false
15933 4700 Acked Read 7 Sent Done Done false
15934 4701 Acked Flush 0 Sent Done Done false
15935 4702 Acked Write 6 Sent Done Done false
15936 4703 Acked Write 10 Sent Done Done false
15937 4704 Acked Write 7 Sent Done Done false
15938 4705 Acked Flush 0 Sent Done Done false
15939 4706 Acked Write 3 Sent Done Done false
15940 4707 Acked Write 10 Sent Done Done false
15941 4708 Acked Read 2 Sent Done Done false
15942 4709 Acked Read 8 Sent Done Done false
15943 4710 Acked Flush 0 Sent Done Done false
15944 4711 Acked Read 1 Sent Done Done false
15945 4712 Acked Write 10 Sent Done Done false
15946 4713 Acked Read 2 Sent Done Done false
15947 4714 Acked Read 2 Sent Done Done false
15948 4715 Acked Flush 0 Sent Done Done false
15949 4716 Acked Write 10 Sent Done Done false
15950 4717 Acked Read 1 Sent Done Done false
15951 4718 Acked Write 8 Sent Done Done false
15952 4719 Acked Read 10 Sent Done Done false
15953 4720 Acked Flush 0 Sent Done Done false
15954 4721 Acked Write 10 Sent Done Done false
15955 4722 Acked Read 3 Sent Done Done false
15956 4723 Acked Flush 0 Sent Done Done false
15957 4724 Acked Read 9 Sent Done Done false
15958 4725 Acked Read 1 Sent Done Done false
15959 4726 Acked Read 10 Sent Done Done false
15960 4727 Acked Write 5 Sent Done Done false
15961 4728 Acked Write 10 Sent Done Done false
15962 4729 Acked Write 4 Sent Done Done false
15963 4730 Acked Read 2 Sent Done Done false
15964 4731 Acked Write 6 Sent Done Done false
15965 4732 Acked Write 1 Sent Done Done false
15966 4733 Acked Write 3 Sent Done Done false
15967 4734 Acked Write 1 Sent Done Done false
15968 4735 Acked Read 1 Sent Done Done false
15969 4736 Acked Read 9 Sent Done Done false
15970 4737 Acked Flush 0 Sent Done Done false
15971 4738 Acked Read 8 Sent Done Done false
15972 4739 Acked Read 8 Sent Done Done false
15973 4740 Acked Write 7 Sent Done Done false
15974 4741 Acked Write 1 Sent Done Done false
15975 4742 Acked Write 6 Sent Done Done false
15976 4743 Acked Read 4 Sent Done Done false
15977 4744 Acked Read 2 Sent Done Done false
15978 4745 Acked Read 7 Sent Done Done false
15979 4746 Acked Write 8 Sent Done Done false
15980 4747 Acked Read 8 Sent Done Done false
15981 4748 Acked Write 8 Sent Done Done false
15982 4749 Acked Read 10 Sent Done Done false
15983 4750 Acked Write 9 Sent Done Done false
15984 4751 Acked Write 9 Sent Done Done false
15985 4752 Acked Read 7 Sent Done Done false
15986 4753 Acked Write 6 Sent Done Done false
15987 4754 Acked Read 5 Sent Done Done false
15988 4755 Acked Read 9 Sent Done Done false
15989 4756 Acked Write 10 Sent Done Done false
15990 4757 Acked Read 6 Sent Done Done false
15991 4758 Acked Flush 0 Sent Done Done false
15992 4759 Acked Read 3 Sent Done Done false
15993 4760 Acked Write 10 Sent Done Done false
15994 4761 Acked Read 7 Sent Done Done false
15995 4762 Acked Write 8 Sent Done Done false
15996 4763 Acked Read 7 Sent Done Done false
15997 4764 Acked Read 1 Sent Done Done false
15998 4765 Acked Write 1 Sent Done Done false
15999 4766 Acked Read 10 Sent Done Done false
16000 4767 Acked Read 4 Sent Done Done false
16001 4768 Acked Flush 0 Sent Done Done false
16002 4769 Acked Read 8 Sent Done Done false
16003 4770 Acked Write 8 Sent Done Done false
16004 4771 Acked Read 7 Sent Done Done false
16005 4772 Acked Read 7 Sent Done Done false
16006 4773 Acked Read 6 Sent Done Done false
16007 4774 Acked Read 7 Sent Done Done false
16008 4775 Acked Read 7 Sent Done Done false
16009 4776 Acked Write 8 Sent Done Done false
16010 4777 Acked Write 5 Sent Done Done false
16011 4778 Acked Write 5 Sent Done Done false
16012 4779 Acked Read 8 Sent Done Done false
16013 4780 Acked Read 6 Sent Done Done false
16014 4781 Acked Read 7 Sent Done Done false
16015 4782 Acked Write 2 Sent Done Done false
16016 4783 Acked Read 4 Sent Done Done false
16017 4784 Acked Write 9 Sent Done Done false
16018 4785 Acked Write 4 Sent Done Done false
16019 4786 Acked Read 4 Sent Done Done false
16020 4787 Acked Write 4 Sent Done Done false
16021 4788 Acked Write 6 Sent Done Done false
16022 4789 Acked Read 10 Sent Done Done false
16023 4790 Acked Read 10 Sent Done Done false
16024 4791 Acked Read 4 Sent Done Done false
16025 4792 Acked Read 8 Sent Done Done false
16026 4793 Acked Read 9 Sent Done Done false
16027 4794 Acked Read 3 Sent Done Done false
16028 4795 Acked Read 2 Sent Done Done false
16029 4796 Acked Read 9 Sent Done Done false
16030 4797 Acked Write 1 Sent Done Done false
16031 4798 Acked Read 7 Sent Done Done false
16032 4799 Acked Write 4 Sent Done Done false
16033 4800 Acked Read 6 Sent Done Done false
16034 4801 Acked Read 3 Sent Done Done false
16035 4802 Acked Write 8 Sent Done Done false
16036 4803 Acked Write 8 Sent Done Done false
16037 4804 Acked Write 3 Sent Done Done false
16038 4805 Acked Read 6 Sent Done Done false
16039 4806 Acked Read 3 Sent Done Done false
16040 4807 Acked Read 7 Sent Done Done false
16041 4808 Acked Read 1 Sent Done Done false
16042 4809 Acked Read 1 Sent Done Done false
16043 4810 Acked Write 8 Sent Done Done false
16044 4811 Acked Read 3 Sent Done Done false
16045 4812 Acked Read 5 Sent Done Done false
16046 4813 Acked Read 2 Sent Done Done false
16047 4814 Acked Read 1 Sent Done Done false
16048 4815 Acked Flush 0 Sent Done Done false
16049 4816 Acked Write 5 Sent Done Done false
16050 4817 Acked Read 2 Sent Done Done false
16051 4818 Acked Write 10 Sent Done Done false
16052 4819 Acked Write 2 Sent Done Done false
16053 4820 Acked Write 10 Sent Done Done false
16054 4821 Acked Write 3 Sent Done Done false
16055 4822 Acked Write 1 Sent Done Done false
16056 4823 Acked Write 1 Sent Done Done false
16057 4824 Acked Write 10 Sent Done Done false
16058 4825 Acked Write 2 Sent Done Done false
16059 4826 Acked Read 5 Sent Done Done false
16060 4827 Acked Write 6 Sent Done Done false
16061 4828 Acked Write 5 Sent Done Done false
16062 4829 Acked Read 7 Sent Done Done false
16063 4830 Acked Write 6 Sent Done Done false
16064 4831 Acked Read 4 Sent Done Done false
16065 4832 Acked Write 5 Sent Done Done false
16066 4833 Acked Read 3 Sent Done Done false
16067 4834 Acked Read 6 Sent Done Done false
16068 4835 Acked Read 10 Sent Done Done false
16069 4836 Acked Read 4 Sent Done Done false
16070 4837 Acked Flush 0 Sent Done Done false
16071 4838 Acked Write 8 Sent Done Done false
16072 4839 Acked Write 5 Sent Done Done false
16073 4840 Acked Flush 0 Sent Done Done false
16074 4841 Acked Read 2 Sent Done Done false
16075 4842 Acked Read 8 Sent Done Done false
16076 4843 Acked Write 9 Sent Done Done false
16077 4844 Acked Write 4 Sent Done Done false
16078 4845 Acked Read 5 Sent Done Done false
16079 4846 Acked Flush 0 Sent Done Done false
16080 4847 Acked Write 6 Sent Done Done false
16081 4848 Acked Write 3 Sent Done Done false
16082 4849 Acked Write 4 Sent Done Done false
16083 4850 Acked Write 1 Sent Done Done false
16084 4851 Acked Write 4 Sent Done Done false
16085 4852 Acked Write 7 Sent Done Done false
16086 4853 Acked Write 10 Sent Done Done false
16087 4854 Acked Write 8 Sent Done Done false
16088 4855 Acked Flush 0 Sent Done Done false
16089 4856 Acked Write 7 Sent Done Done false
16090 4857 Acked Write 7 Sent Done Done false
16091 4858 Acked Read 6 Sent Done Done false
16092 4859 Acked Write 1 Sent Done Done false
16093 4860 Acked Read 8 Sent Done Done false
16094 4861 Acked Write 9 Sent Done Done false
16095 4862 Acked Read 3 Sent Done Done false
16096 4863 Acked Read 2 Sent Done Done false
16097 4864 Acked Write 1 Sent Done Done false
16098 4865 Acked Flush 0 Sent Done Done false
16099 4866 Acked Read 1 Sent Done Done false
16100 4867 Acked Write 4 Sent Done Done false
16101 4868 Acked Read 6 Sent Done Done false
16102 4869 Acked Write 7 Sent Done Done false
16103 4870 Acked Read 8 Sent Done Done false
16104 4871 Acked Read 1 Sent Done Done false
16105 4872 Acked Write 8 Sent Done Done false
16106 4873 Acked Write 9 Sent Done Done false
16107 4874 Acked Flush 0 Sent Done Done false
16108 4875 Acked Write 3 Sent Done Done false
16109 4876 Acked Write 3 Sent Done Done false
16110 4877 Acked Write 10 Sent Done Done false
16111 4878 Acked Read 2 Sent Done Done false
16112 4879 Acked Read 5 Sent Done Done false
16113 4880 Acked Write 1 Sent Done Done false
16114 4881 Acked Write 10 Sent Done Done false
16115 4882 Acked Flush 0 Sent Done Done false
16116 4883 Acked Write 1 Sent Done Done false
16117 4884 Acked Write 1 Sent Done Done false
16118 4885 Acked Write 10 Sent Done Done false
16119 4886 Acked Read 2 Sent Done Done false
16120 4887 Acked Read 9 Sent Done Done false
16121 4888 Acked Flush 0 Sent Done Done false
16122 4889 Acked Write 5 Sent Done Done false
16123 4890 Acked Read 6 Sent Done Done false
16124 4891 Acked Read 4 Sent Done Done false
16125 4892 Acked Write 3 Sent Done Done false
16126 4893 Acked Flush 0 Sent Done Done false
16127 4894 Acked Read 3 Sent Done Done false
16128 4895 Acked Read 7 Sent Done Done false
16129 4896 Acked Read 8 Sent Done Done false
16130 4897 Acked Write 9 Sent Done Done false
16131 4898 Acked Write 3 Sent Done Done false
16132 4899 Acked Read 10 Sent Done Done false
16133 4900 Acked Read 7 Sent Done Done false
16134 4901 Acked Write 6 Sent Done Done false
16135 4902 Acked Write 4 Sent Done Done false
16136 4903 Acked Read 8 Sent Done Done false
16137 4904 Acked Write 8 Sent Done Done false
16138 4905 Acked Read 5 Sent Done Done false
16139 4906 Acked Write 2 Sent Done Done false
16140 4907 Acked Write 10 Sent Done Done false
16141 4908 Acked Flush 0 Sent Done Done false
16142 4909 Acked Read 10 Sent Done Done false
16143 4910 Acked Read 8 Sent Done Done false
16144 4911 Acked Write 8 Sent Done Done false
16145 4912 Acked Read 6 Sent Done Done false
16146 4913 Acked Write 10 Sent Done Done false
16147 4914 Acked Flush 0 Sent Done Done false
16148 4915 Acked Write 10 Sent Done Done false
16149 4916 Acked Write 6 Sent Done Done false
16150 4917 Acked Write 8 Sent Done Done false
16151 4918 Acked Write 10 Sent Done Done false
16152 4919 Acked Write 3 Sent Done Done false
16153 4920 Acked Write 4 Sent Done Done false
16154 4921 Acked Write 5 Sent Done Done false
16155 STATES DS:0 DS:1 DS:2 TOTAL
16156 Sent 301 0 0 301
16157 Done 0 301 301 602
16158 Skipped 0 0 0 0
16159 Error 0 0 0 0
16160 Last Flush: Some(JobId(4620)) Some(JobId(4914)) Some(JobId(4914))
16161 Downstairs last five completed: 4620 4619 4618 4617 4616
16162 Upstairs last five completed: 4921 4920 4919 4918 4917
16163 CLIENT: Up:0 ds:301 act:2
161642025-02-28T06:04:31.890ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
161652025-02-28T06:04:32.689ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
161662025-02-28T06:04:32.881ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
161672025-02-28T06:04:32.882ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50445) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
161682025-02-28T06:04:32.883ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
161692025-02-28T06:04:32.883ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
161702025-02-28T06:04:32.883ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
161712025-02-28T06:04:32.883ZINFOcrucible: send last flush ID to this DS: Some(JobId(4620)) = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
161722025-02-28T06:04:32.883ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
161732025-02-28T06:04:32.883ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4620)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
161742025-02-28T06:04:32.883ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4620)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
16175 ----------------------------------------------------------------
16176 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16177 Downstairs last five completed: 4922 4921 4920 4919 4918
16178 Upstairs last five completed: 4922 4921 4920 4919 4918
16179 CLIENT: Up:0 ds:0 act:3
16180 Replay: All jobs finished
16181 Main task has work to do, go find it
16182 got dsc Stop(2)
16183 stop 2
16184 [8830] Got stop action so:false kr:false
16185 Feb 28 06:04:35.127 INFO request completed, latency_us: 356, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 78ace500-74f4-4c8e-9e1c-c591b6972b07, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
161862025-02-28T06:04:35.127ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
161872025-02-28T06:04:35.128ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
16188 Feb 28 06:04:35.128 INFO request completed, latency_us: 130{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:04:35.128086618Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
16189 , response_code: 200, uri: /state/cid/2, method: GET, req_id: 27226116-49bb-4742-9c7b-947b5c117b32, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
161902025-02-28T06:04:35.128ZINFOcrucible: client task is exiting = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
16191 [8830] Exited with: ExitStatus(unix_wait_status(9))
16192 [8830][2] reports Exit
16193 Feb 28 06:04:39.130 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6e150c67-ae22-4f3b-a755-facf37086e18, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
16194 001/300 Flush
16195 002/300 Read block 4451 len 16384
16196 003/300 Read block 4164 len 12288
16197 004/300 Write block 2561 len 36864 data: 3 3 4 4 3 2 2 3 4
16198 005/300 Read block 1304 len 36864
16199 006/300 Read block 4109 len 28672
16200 007/300 Write block 1432 len 28672 data: 2 3 3 3 3 4 3
16201 008/300 Read block 2570 len 4096
16202 009/300 Write block 1489 len 24576 data: 7 7 7 7 7 6
16203 010/300 Read block 681 len 20480
16204 011/300 Write block 4260 len 4096 data: 3
16205 012/300 Read block 1256 len 24576
16206 013/300 Read block 2788 len 4096
16207 014/300 Read block 3205 len 20480
16208 015/300 Read block 1342 len 12288
16209 016/300 Flush
16210 017/300 Read block 2403 len 20480
16211 018/300 Read block 3766 len 36864
16212 019/300 Flush
16213 020/300 Read block 4364 len 8192
16214 021/300 Read block 1422 len 24576
16215 022/300 Flush
16216 023/300 Write block 1421 len 28672 data: 4 4 5 4 4 3 3
16217 024/300 Write block 1982 len 4096 data: 3
16218 025/300 Write block 4228 len 8192 data: 8 9
16219 026/300 Write block 570 len 24576 data: 3 3 4 4 4 4
16220 027/300 Read block 3676 len 32768
16221 028/300 Write block 422 len 20480 data: 5 4 3 2 2
16222 029/300 Write block 2109 len 8192 data: 3 3
16223 030/300 Write block 4227 len 28672 data: 7 9 10 8 8 4 4
16224 031/300 Read block 2560 len 32768
16225 032/300 Read block 133 len 36864
16226 033/300 Flush
16227 034/300 Write block 702 len 12288 data: 6 6 7
16228 035/300 Write block 1372 len 36864 data: 3 4 4 3 3 3 3 3 3
16229 036/300 Read block 1486 len 4096
16230 037/300 Read block 2660 len 32768
16231 038/300 Read block 4301 len 40960
16232 039/300 Read block 693 len 16384
16233 040/300 Write block 93 len 16384 data: 5 6 7 4
16234 041/300 Write block 2450 len 40960 data: 2 2 3 3 4 6 5 5 5 4
16235 042/300 Read block 4877 len 40960
16236 043/300 Write block 2529 len 8192 data: 4 3
16237 044/300 Read block 3023 len 24576
16238 045/300 Flush
16239 046/300 Read block 3470 len 24576
16240 047/300 Write block 4595 len 16384 data: 6 6 6 5
16241 048/300 Read block 2388 len 8192
16242 049/300 Write block 1219 len 28672 data: 3 3 3 3 3 3 2
16243 050/300 Read block 4645 len 4096
16244 051/300 Write block 915 len 36864 data: 4 2 2 2 2 2 2 2 2
16245 052/300 Read block 371 len 32768
16246 053/300 Read block 3478 len 16384
16247 054/300 Write block 2166 len 8192 data: 6 6
16248 055/300 Write block 2339 len 24576 data: 4 4 4 4 4 3
16249 056/300 Write block 618 len 32768 data: 2 2 3 3 3 3 3 4
16250 057/300 Write block 1769 len 28672 data: 2 2 2 2 2 3 3
16251 058/300 Write block 4378 len 24576 data: 6 6 6 5 3 5
16252 059/300 Flush
16253 060/300 Read block 2815 len 40960
16254 061/300 Write block 2388 len 4096 data: 3
16255 062/300 Read block 2068 len 20480
16256 063/300 Read block 4900 len 8192
16257 064/300 Read block 1730 len 28672
16258 065/300 Read block 3013 len 24576
16259 066/300 Read block 2248 len 8192
16260 067/300 Write block 3373 len 4096 data: 3
16261 068/300 Write block 3742 len 32768 data: 5 6 6 6 4 4 4 4
16262 069/300 Read block 575 len 32768
16263 070/300 Write block 4913 len 28672 data: 3 3 3 2 2 2 2
16264 071/300 Write block 297 len 12288 data: 4 4 4
16265 072/300 Write block 2208 len 20480 data: 3 5 4 4 5
16266 073/300 Read block 1657 len 8192
16267 074/300 Flush
16268 075/300 Write block 4443 len 12288 data: 4 4 4
16269 076/300 Read block 3646 len 40960
16270 077/300 Write block 2733 len 16384 data: 5 3 4 4
16271 078/300 Write block 4839 len 24576 data: 5 5 4 5 4 4
16272 079/300 Write block 1724 len 40960 data: 5 5 4 5 4 4 5 4 4 4
16273 080/300 Write block 3843 len 28672 data: 4 5 5 4 4 5 4
16274 081/300 Write block 1810 len 8192 data: 3 3
16275 082/300 Write block 4133 len 24576 data: 4 2 3 3 3 4
16276 083/300 Flush
16277 084/300 Write block 4756 len 16384 data: 5 5 5 5
16278 085/300 Write block 2833 len 4096 data: 2
16279 086/300 Write block 2183 len 32768 data: 4 3 3 4 3 4 4 5
16280 087/300 Write block 4828 len 24576 data: 6 6 8 8 7 4
16281 088/300 Read block 2107 len 28672
16282 089/300 Write block 599 len 8192 data: 3 3
16283 090/300 Read block 1421 len 28672
16284 091/300 Read block 1251 len 32768
16285 092/300 Read block 4875 len 16384
16286 093/300 Write block 1312 len 24576 data: 5 5 2 2 2 2
16287 094/300 Read block 3674 len 16384
16288 095/300 Write block 491 len 16384 data: 2 2 3 3
16289 096/300 Write block 4693 len 36864 data: 3 3 3 4 4 3 5 3 3
16290 097/300 Write block 3918 len 16384 data: 3 3 3 2
16291 098/300 Read block 4397 len 20480
16292 099/300 Write block 3236 len 16384 data: 2 2 2 2
16293 100/300 Read block 4949 len 32768
16294 101/300 Read block 2893 len 20480
16295 102/300 Write block 2931 len 4096 data: 3
16296 103/300 Read block 3179 len 12288
16297 104/300 Read block 75 len 16384
16298 105/300 Flush
16299 106/300 Read block 808 len 4096
16300 107/300 Write block 4861 len 20480 data: 2 3 3 3 3
16301 108/300 Flush
16302 109/300 Read block 1310 len 40960
16303 110/300 Read block 456 len 16384
16304 111/300 Read block 962 len 4096
16305 112/300 Read block 552 len 8192
16306 113/300 Read block 4754 len 8192
16307 114/300 Write block 144 len 24576 data: 4 4 4 4 4 5
16308 115/300 Write block 273 len 24576 data: 3 4 4 4 5 6
16309 116/300 Write block 1151 len 20480 data: 4 4 5 4 5
16310 117/300 Write block 4227 len 8192 data: 8 10
16311 118/300 Write block 2384 len 8192 data: 3 3
16312 119/300 Read block 1770 len 20480
16313 120/300 Write block 2672 len 20480 data: 6 5 6 6 5
16314 121/300 Read block 976 len 4096
16315 122/300 Read block 3303 len 8192
16316 123/300 Write block 1400 len 4096 data: 5
16317 124/300 Read block 317 len 24576
16318 125/300 Write block 3985 len 12288 data: 2 3 3
16319 126/300 Read block 4204 len 20480
16320 127/300 Write block 4093 len 36864 data: 3 3 5 4 4 4 4 4 5
16321 128/300 Read block 3413 len 32768
16322 129/300 Write block 567 len 8192 data: 2 2
16323 130/300 Read block 2940 len 16384
16324 131/300 Read block 340 len 4096
16325 132/300 Write block 783 len 24576 data: 3 3 4 3 3 2
16326 133/300 Read block 4931 len 20480
16327 134/300 Write block 2003 len 4096 data: 5
16328 135/300 Write block 2713 len 20480 data: 8 7 6 6 6
16329 136/300 Write block 720 len 36864 data: 3 2 2 2 2 2 2 2 2
16330 137/300 Write block 1586 len 12288 data: 4 4 3
16331 138/300 Read block 4379 len 8192
16332 139/300 Flush
16333 140/300 Write block 3773 len 8192 data: 2 2
16334 141/300 Write block 1816 len 16384 data: 2 2 2 2
16335 142/300 Flush
16336 143/300 Read block 3298 len 16384
16337 144/300 Read block 1803 len 40960
16338 145/300 Read block 1696 len 32768
16339 146/300 Write block 4461 len 8192 data: 4 4
16340 147/300 Flush
16341 148/300 Write block 194 len 28672 data: 6 6 6 5 4 4 4
16342 149/300 Read block 3474 len 36864
16343 150/300 Read block 4712 len 40960
16344 151/300 Read block 2137 len 28672
16345 152/300 Write block 35 len 28672 data: 3 3 3 3 3 3 4
16346 153/300 Read block 975 len 28672
16347 154/300 Read block 472 len 32768
16348 155/300 Write block 1250 len 4096 data: 4
16349 156/300 Write block 1793 len 40960 data: 3 2 3 3 3 3 3 3 3 3
16350 157/300 Read block 4841 len 8192
16351 158/300 Write block 681 len 16384 data: 5 4 4 4
16352 159/300 Write block 3807 len 8192 data: 4 4
16353 160/300 Write block 2594 len 40960 data: 3 3 3 3 3 4 4 4 3 4
16354 161/300 Read block 2576 len 40960
16355 162/300 Read block 2042 len 40960
16356 163/300 Write block 1927 len 28672 data: 2 2 2 2 2 3 3
16357 164/300 Write block 1179 len 28672 data: 8 7 6 4 3 4 4
16358 165/300 Read block 4471 len 4096
16359 166/300 Flush
16360 167/300 Read block 1007 len 40960
16361 168/300 Flush
16362 169/300 Read block 567 len 40960
16363 170/300 Flush
16364 171/300 Write block 1197 len 12288 data: 2 3 2
16365 172/300 Write block 680 len 16384 data: 5 6 5 5
16366 173/300 Write block 965 len 24576 data: 4 3 3 3 2 2
16367 174/300 Write block 1837 len 36864 data: 4 5 6 6 4 3 5 5 5
16368 175/300 Read block 2742 len 24576
16369 176/300 Read block 616 len 24576
16370 177/300 Flush
16371 178/300 Write block 1934 len 4096 data: 3
16372 179/300 Write block 1835 len 8192 data: 3 4
16373 180/300 Read block 4852 len 12288
16374 181/300 Read block 2695 len 8192
16375 182/300 Write block 2447 len 20480 data: 5 4 3 3 3
16376 183/300 Read block 1653 len 36864
16377 184/300 Read block 1157 len 24576
16378 185/300 Read block 1891 len 24576
16379 186/300 Read block 4238 len 4096
16380 187/300 Write block 268 len 28672 data: 3 2 2 3 3 4 5
16381 188/300 Read block 2510 len 8192
16382 189/300 Read block 2762 len 36864
16383 190/300 Read block 4919 len 24576
16384 191/300 Read block 4695 len 4096
16385 192/300 Flush
16386 193/300 Flush
16387 194/300 Read block 4154 len 28672
16388 195/300 Write block 2110 len 20480 data: 4 3 3 3 4
16389 196/300 Read block 3013 len 32768
16390 197/300 Read block 982 len 8192
16391 198/300 Read block 4771 len 36864
16392 199/300 Read block 340 len 4096
16393 200/300 Read block 4762 len 8192
16394 201/300 Read block 4582 len 32768
16395 202/300 Read block 4747 len 16384
16396 203/300 Read block 4744 len 16384
16397 204/300 Read block 2780 len 8192
16398 205/300 Read block 1917 len 32768
16399 206/300 Read block 4795 len 24576
16400 207/300 Write block 1335 len 32768 data: 4 4 4 3 3 4 4 4
16401 208/300 Write block 4805 len 12288 data: 2 2 3
16402 209/300 Write block 3705 len 12288 data: 3 4 5
16403 210/300 Read block 4669 len 16384
16404 211/300 Flush
16405 212/300 Read block 1129 len 36864
16406 213/300 Read block 2195 len 16384
16407 214/300 Write block 687 len 8192 data: 2 2
16408 215/300 Write block 1709 len 36864 data: 6 6 5 6 6 6 5 3 3
16409 216/300 Write block 3986 len 40960 data: 4 4 2 2 2 2 2 2 2 2
16410 217/300 Read block 2960 len 4096
16411 218/300 Write block 4482 len 28672 data: 5 5 5 5 5 5 4
16412 219/300 Read block 855 len 32768
16413 220/300 Read block 1830 len 28672
16414 221/300 Write block 716 len 24576 data: 4 4 4 3 4 3
16415 222/300 Read block 778 len 20480
16416 223/300 Write block 4770 len 32768 data: 4 4 3 2 2 3 3 4
16417 224/300 Read block 1445 len 16384
16418 225/300 Read block 3584 len 16384
16419 226/300 Read block 1441 len 12288
16420 227/300 Read block 391 len 36864
16421 228/300 Read block 1440 len 24576
16422 229/300 Flush
16423 230/300 Write block 3762 len 32768 data: 4 3 2 2 2 2 2 2
16424 231/300 Flush
16425 232/300 Write block 1653 len 40960 data: 3 4 3 4 3 3 3 5 3 3
16426 233/300 Write block 3398 len 36864 data: 2 2 3 3 3 3 2 3 3
16427 234/300 Write block 956 len 8192 data: 3 3
16428 235/300 Read block 3907 len 28672
16429 236/300 Write block 2519 len 36864 data: 5 5 5 5 6 7 6 4 4
16430 237/300 Read block 3590 len 28672
16431 238/300 Flush
16432 239/300 Write block 3545 len 16384 data: 3 4 4 5
16433 240/300 Write block 2742 len 8192 data: 4 4
16434 241/300 Read block 2703 len 24576
16435 242/300 Read block 2891 len 4096
16436 243/300 Read block 2867 len 4096
16437 244/300 Flush
16438 245/300 Read block 2920 len 32768
16439 246/300 Flush
16440 247/300 Flush
16441 248/300 Read block 900 len 16384
16442 249/300 Write block 3249 len 28672 data: 5 5 4 5 3 3 3
16443 250/300 Flush
16444 251/300 Flush
16445 252/300 Read block 751 len 40960
16446 253/300 Write block 3608 len 4096 data: 4
16447 254/300 Flush
16448 255/300 Write block 3823 len 4096 data: 2
16449 256/300 Write block 3395 len 24576 data: 2 2 2 3 3 4
16450 257/300 Write block 2240 len 4096 data: 3
16451 258/300 Read block 3321 len 16384
16452 259/300 Read block 2645 len 4096
16453 260/300 Write block 1081 len 12288 data: 5 4 3
16454 261/300 Read block 369 len 36864
16455 262/300 Read block 1161 len 16384
16456 263/300 Write block 2549 len 40960 data: 4 4 4 3 3 3 3 3 3 3
16457 264/300 Write block 3355 len 12288 data: 4 4 3
16458 265/300 Read block 1177 len 28672
16459 266/300 Write block 808 len 12288 data: 3 3 3
16460 267/300 Read block 2713 len 40960
16461 268/300 Write block 678 len 8192 data: 5 6
16462 269/300 Read block 3694 len 12288
16463 270/300 Read block 1356 len 28672
16464 271/300 Read block 2402 len 24576
16465 272/300 Flush
16466 273/300 Read block 1653 len 24576
16467 274/300 Read block 1163 len 32768
16468 275/300 Read block 560 len 8192
16469 276/300 Write block 373 len 8192 data: 4 4
16470 277/300 Write block 2446 len 28672 data: 6 6 5 4 4 4 4
16471 278/300 Flush
16472 279/300 Read block 2858 len 28672
16473 280/300 Read block 556 len 12288
16474 281/300 Write block 3285 len 36864 data: 2 3 3 3 3 3 2 2 5
16475 282/300 Write block 1565 len 40960 data: 3 4 3 4 4 3 3 3 3 3
16476 283/300 Write block 3497 len 12288 data: 3 3 3
16477 284/300 Read block 2555 len 28672
16478 285/300 Write block 772 len 36864 data: 4 3 3 4 4 4 3 4 4
16479 286/300 Read block 3877 len 24576
16480 287/300 Flush
16481 288/300 Flush
16482 289/300 Read block 2516 len 8192
16483 290/300 Read block 4778 len 20480
16484 291/300 Read block 2606 len 28672
16485 292/300 Write block 2876 len 16384 data: 2 2 2 2
16486 293/300 Read block 169 len 8192
16487 294/300 Read block 1248 len 12288
16488 295/300 Read block 4848 len 32768
16489 296/300 Write block 3154 len 24576 data: 3 3 5 6 6 7
16490 297/300 Read block 1322 len 28672
16491 298/300 Write block 1337 len 4096 data: 5
16492 299/300 Read block 3588 len 20480
16493 300/300 Read block 1786 len 28672
16494 301/300 Write block 2895 len 24576 data: 2 2 3 4 3 3
16495 Main task has work to do, go find it
16496 got dsc Start(2)
16497 start 2
16498 Feb 28 06:04:39.363 INFO request completed, latency_us: 178, response_code: 204, uri: /start/cid/2, method: POST, req_id: eaccfbff-40a6-44c8-8200-62dd7390c4b5, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
16499 Replay: started 2, returned:Ok(())
16500 ----------------------------------------------------------------
16501 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16502 States: Active Active Offline
16503 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16504 4923 Acked Flush 0 Done Done Sent false
16505 4924 Acked Read 4 Done Done Sent false
16506 4925 Acked Read 3 Done Done Sent false
16507 4926 Acked Write 9 Done Done Sent false
16508 4927 Acked Read 9 Done Done Sent false
16509 4928 Acked Read 7 Done Done Sent false
16510 4929 Acked Write 7 Done Done Sent false
16511 4930 Acked Read 1 Done Done Sent false
16512 4931 Acked Write 6 Done Done Sent false
16513 4932 Acked Read 5 Done Done Sent false
16514 4933 Acked Write 1 Done Done Sent false
16515 4934 Acked Read 6 Done Done Sent false
16516 4935 Acked Read 1 Done Done Sent false
16517 4936 Acked Read 5 Done Done Sent false
16518 4937 Acked Read 3 Done Done Sent false
16519 4938 Acked Flush 0 Done Done Sent false
16520 4939 Acked Read 5 Done Done Sent false
16521 4940 Acked Read 9 Done Done Sent false
16522 4941 Acked Flush 0 Done Done Sent false
16523 4942 Acked Read 2 Done Done Sent false
16524 4943 Acked Read 6 Done Done Sent false
16525 4944 Acked Flush 0 Done Done Sent false
16526 4945 Acked Write 7 Done Done Sent false
16527 4946 Acked Write 1 Done Done Sent false
16528 4947 Acked Write 2 Done Done Sent false
16529 4948 Acked Write 6 Done Done Sent false
16530 4949 Acked Read 8 Done Done Sent false
16531 4950 Acked Write 5 Done Done Sent false
16532 4951 Acked Write 2 Done Done Sent false
16533 4952 Acked Write 7 Done Done Sent false
16534 4953 Acked Read 8 Done Done Sent false
16535 4954 Acked Read 9 Done Done Sent false
16536 4955 Acked Flush 0 Done Done Sent false
16537 4956 Acked Write 3 Done Done Sent false
16538 4957 Acked Write 9 Done Done Sent false
16539 4958 Acked Read 1 Done Done Sent false
16540 4959 Acked Read 8 Done Done Sent false
16541 4960 Acked Read 10 Done Done Sent false
16542 4961 Acked Read 4 Done Done Sent false
16543 4962 Acked Write 4 Done Done Sent false
16544 4963 Acked Write 10 Done Done Sent false
16545 4964 Acked Read 10 Done Done Sent false
16546 4965 Acked Write 2 Done Done Sent false
16547 4966 Acked Read 6 Done Done Sent false
16548 4967 Acked Flush 0 Done Done Sent false
16549 4968 Acked Read 6 Done Done Sent false
16550 4969 Acked Write 4 Done Done Sent false
16551 4970 Acked Read 2 Done Done Sent false
16552 4971 Acked Write 7 Done Done Sent false
16553 4972 Acked Read 1 Done Done Sent false
16554 4973 Acked Write 9 Done Done Sent false
16555 4974 Acked Read 8 Done Done Sent false
16556 4975 Acked Read 4 Done Done Sent false
16557 4976 Acked Write 2 Done Done Sent false
16558 4977 Acked Write 6 Done Done Sent false
16559 4978 Acked Write 8 Done Done Sent false
16560 4979 Acked Write 7 Done Done Sent false
16561 4980 Acked Write 6 Done Done Sent false
16562 4981 Acked Flush 0 Done Done Sent false
16563 4982 Acked Read 10 Done Done Sent false
16564 4983 Acked Write 1 Done Done Sent false
16565 4984 Acked Read 5 Done Done Sent false
16566 4985 Acked Read 2 Done Done Sent false
16567 4986 Acked Read 7 Done Done Sent false
16568 4987 Acked Read 6 Done Done Sent false
16569 4988 Acked Read 2 Done Done Sent false
16570 4989 Acked Write 1 Done Done Sent false
16571 4990 Acked Write 8 Done Done Sent false
16572 4991 Acked Read 8 Done Done Sent false
16573 4992 Acked Write 7 Done Done Sent false
16574 4993 Acked Write 3 Done Done Sent false
16575 4994 Acked Write 5 Done Done Sent false
16576 4995 Acked Read 2 Done Done Sent false
16577 4996 Acked Flush 0 Done Done Sent false
16578 4997 Acked Write 3 Done Done Sent false
16579 4998 Acked Read 10 Done Done Sent false
16580 4999 Acked Write 4 Done Done Sent false
16581 5000 Acked Write 6 Done Done Sent false
16582 5001 Acked Write 10 Done Done Sent false
16583 5002 Acked Write 7 Done Done Sent false
16584 5003 Acked Write 2 Done Done Sent false
16585 5004 Acked Write 6 Done Done Sent false
16586 5005 Acked Flush 0 Done Done Sent false
16587 5006 Acked Write 4 Done Done Sent false
16588 5007 Acked Write 1 Done Done Sent false
16589 5008 Acked Write 8 Done Done Sent false
16590 5009 Acked Write 6 Done Done Sent false
16591 5010 Acked Read 7 Done Done Sent false
16592 5011 Acked Write 2 Done Done Sent false
16593 5012 Acked Read 7 Done Done Sent false
16594 5013 Acked Read 8 Done Done Sent false
16595 5014 Acked Read 4 Done Done Sent false
16596 5015 Acked Write 6 Done Done Sent false
16597 5016 Acked Read 4 Done Done Sent false
16598 5017 Acked Write 4 Done Done Sent false
16599 5018 Acked Write 9 Done Done Sent false
16600 5019 Acked Write 4 Done Done Sent false
16601 5020 Acked Read 5 Done Done Sent false
16602 5021 Acked Write 4 Done Done Sent false
16603 5022 Acked Read 8 Done Done Sent false
16604 5023 Acked Read 5 Done Done Sent false
16605 5024 Acked Write 1 Done Done Sent false
16606 5025 Acked Read 3 Done Done Sent false
16607 5026 Acked Read 4 Done Done Sent false
16608 5027 Acked Flush 0 Done Done Sent false
16609 5028 Acked Read 1 Done Done Sent false
16610 5029 Acked Write 5 Done Done Sent false
16611 5030 Acked Flush 0 Done Done Sent false
16612 5031 Acked Read 10 Done Done Sent false
16613 5032 Acked Read 4 Done Done Sent false
16614 5033 Acked Read 1 Done Done Sent false
16615 5034 Acked Read 2 Done Done Sent false
16616 5035 Acked Read 2 Done Done Sent false
16617 5036 Acked Write 6 Done Done Sent false
16618 5037 Acked Write 6 Done Done Sent false
16619 5038 Acked Write 5 Done Done Sent false
16620 5039 Acked Write 2 Done Done Sent false
16621 5040 Acked Write 2 Done Done Sent false
16622 5041 Acked Read 5 Done Done Sent false
16623 5042 Acked Write 5 Done Done Sent false
16624 5043 Acked Read 1 Done Done Sent false
16625 5044 Acked Read 2 Done Done Sent false
16626 5045 Acked Write 1 Done Done Sent false
16627 5046 Acked Read 6 Done Done Sent false
16628 5047 Acked Write 3 Done Done Sent false
16629 5048 Acked Read 5 Done Done Sent false
16630 5049 Acked Write 9 Done Done Sent false
16631 5050 Acked Read 8 Done Done Sent false
16632 5051 Acked Write 2 Done Done Sent false
16633 5052 Acked Read 4 Done Done Sent false
16634 5053 Acked Read 1 Done Done Sent false
16635 5054 Acked Write 6 Done Done Sent false
16636 5055 Acked Read 5 Done Done Sent false
16637 5056 Acked Write 1 Done Done Sent false
16638 5057 Acked Write 5 Done Done Sent false
16639 5058 Acked Write 9 Done Done Sent false
16640 5059 Acked Write 3 Done Done Sent false
16641 5060 Acked Read 2 Done Done Sent false
16642 5061 Acked Flush 0 Done Done Sent false
16643 5062 Acked Write 2 Done Done Sent false
16644 5063 Acked Write 4 Done Done Sent false
16645 5064 Acked Flush 0 Done Done Sent false
16646 5065 Acked Read 4 Done Done Sent false
16647 5066 Acked Read 10 Done Done Sent false
16648 5067 Acked Read 8 Done Done Sent false
16649 5068 Acked Write 2 Done Done Sent false
16650 5069 Acked Flush 0 Done Done Sent false
16651 5070 Acked Write 7 Done Done Sent false
16652 5071 Acked Read 9 Done Done Sent false
16653 5072 Acked Read 10 Done Done Sent false
16654 5073 Acked Read 7 Done Done Sent false
16655 5074 Acked Write 7 Done Done Sent false
16656 5075 Acked Read 7 Done Done Sent false
16657 5076 Acked Read 8 Done Done Sent false
16658 5077 Acked Write 1 Done Done Sent false
16659 5078 Acked Write 10 Done Done Sent false
16660 5079 Acked Read 2 Done Done Sent false
16661 5080 Acked Write 4 Done Done Sent false
16662 5081 Acked Write 2 Done Done Sent false
16663 5082 Acked Write 10 Done Done Sent false
16664 5083 Acked Read 10 Done Done Sent false
16665 5084 Acked Read 10 Done Done Sent false
16666 5085 Acked Write 7 Done Done Sent false
16667 5086 Acked Write 7 Done Done Sent false
16668 5087 Acked Read 1 Done Done Sent false
16669 5088 Acked Flush 0 Done Done Sent false
16670 5089 Acked Read 10 Done Done Sent false
16671 5090 Acked Flush 0 Done Done Sent false
16672 5091 Acked Read 10 Done Done Sent false
16673 5092 Acked Flush 0 Done Done Sent false
16674 5093 Acked Write 3 Done Done Sent false
16675 5094 Acked Write 4 Done Done Sent false
16676 5095 Acked Write 6 Done Done Sent false
16677 5096 Acked Write 9 Done Done Sent false
16678 5097 Acked Read 6 Done Done Sent false
16679 5098 Acked Read 6 Done Done Sent false
16680 5099 Acked Flush 0 Done Done Sent false
16681 5100 Acked Write 1 Done Done Sent false
16682 5101 Acked Write 2 Done Done Sent false
16683 5102 Acked Read 3 Done Done Sent false
16684 5103 Acked Read 2 Done Done Sent false
16685 5104 Acked Write 5 Done Done Sent false
16686 5105 Acked Read 9 Done Done Sent false
16687 5106 Acked Read 6 Done Done Sent false
16688 5107 Acked Read 6 Done Done Sent false
16689 5108 Acked Read 1 Done Done Sent false
16690 5109 Acked Write 7 Done Done Sent false
16691 5110 Acked Read 2 Done Done Sent false
16692 5111 Acked Read 9 Done Done Sent false
16693 5112 Acked Read 6 Done Done Sent false
16694 5113 Acked Read 1 Done Done Sent false
16695 5114 Acked Flush 0 Done Done Sent false
16696 5115 Acked Flush 0 Done Done Sent false
16697 5116 Acked Read 7 Done Done Sent false
16698 5117 Acked Write 5 Done Done Sent false
16699 5118 Acked Read 8 Done Done Sent false
16700 5119 Acked Read 2 Done Done Sent false
16701 5120 Acked Read 9 Done Done Sent false
16702 5121 Acked Read 1 Done Done Sent false
16703 5122 Acked Read 2 Done Done Sent false
16704 5123 Acked Read 8 Done Done Sent false
16705 5124 Acked Read 4 Done Done Sent false
16706 5125 Acked Read 4 Done Done Sent false
16707 5126 Acked Read 2 Done Done Sent false
16708 5127 Acked Read 8 Done Done Sent false
16709 5128 Acked Read 6 Done Done Sent false
16710 5129 Acked Write 8 Done Done Sent false
16711 5130 Acked Write 3 Done Done Sent false
16712 5131 Acked Write 3 Done Done Sent false
16713 5132 Acked Read 4 Done Done Sent false
16714 5133 Acked Flush 0 Done Done Sent false
16715 5134 Acked Read 9 Done Done Sent false
16716 5135 Acked Read 4 Done Done Sent false
16717 5136 Acked Write 2 Done Done Sent false
16718 5137 Acked Write 9 Done Done Sent false
16719 5138 Acked Write 10 Done Done Sent false
16720 5139 Acked Read 1 Done Done Sent false
16721 5140 Acked Write 7 Done Done Sent false
16722 5141 Acked Read 8 Done Done Sent false
16723 5142 Acked Read 7 Done Done Sent false
16724 5143 Acked Write 6 Done Done Sent false
16725 5144 Acked Read 5 Done Done Sent false
16726 5145 Acked Write 8 Done Done Sent false
16727 5146 Acked Read 4 Done Done Sent false
16728 5147 Acked Read 4 Done Done Sent false
16729 5148 Acked Read 3 Done Done Sent false
16730 5149 Acked Read 9 Done Done Sent false
16731 5150 Acked Read 6 Done Done Sent false
16732 5151 Acked Flush 0 Done Done Sent false
16733 5152 Acked Write 8 Done Done Sent false
16734 5153 Acked Flush 0 Done Done Sent false
16735 5154 Acked Write 10 Done Done Sent false
16736 5155 Acked Write 9 Done Done Sent false
16737 5156 Acked Write 2 Done Done Sent false
16738 5157 Acked Read 7 Done Done Sent false
16739 5158 Acked Write 9 Done Done Sent false
16740 5159 Acked Read 7 Done Done Sent false
16741 5160 Acked Flush 0 Done Done Sent false
16742 5161 Acked Write 4 Done Done Sent false
16743 5162 Acked Write 2 Done Done Sent false
16744 5163 Acked Read 6 Done Done Sent false
16745 5164 Acked Read 1 Done Done Sent false
16746 5165 Acked Read 1 Done Done Sent false
16747 5166 Acked Flush 0 Done Done Sent false
16748 5167 Acked Read 8 Done Done Sent false
16749 5168 Acked Flush 0 Done Done Sent false
16750 5169 Acked Flush 0 Done Done Sent false
16751 5170 Acked Read 4 Done Done Sent false
16752 5171 Acked Write 7 Done Done Sent false
16753 5172 Acked Flush 0 Done Done Sent false
16754 5173 Acked Flush 0 Done Done Sent false
16755 5174 Acked Read 10 Done Done Sent false
16756 5175 Acked Write 1 Done Done Sent false
16757 5176 Acked Flush 0 Done Done Sent false
16758 5177 Acked Write 1 Done Done Sent false
16759 5178 Acked Write 6 Done Done Sent false
16760 5179 Acked Write 1 Done Done Sent false
16761 5180 Acked Read 4 Done Done Sent false
16762 5181 Acked Read 1 Done Done Sent false
16763 5182 Acked Write 3 Done Done Sent false
16764 5183 Acked Read 9 Done Done Sent false
16765 5184 Acked Read 4 Done Done Sent false
16766 5185 Acked Write 10 Done Done Sent false
16767 5186 Acked Write 3 Done Done Sent false
16768 5187 Acked Read 7 Done Done Sent false
16769 5188 Acked Write 3 Done Done Sent false
16770 5189 Acked Read 10 Done Done Sent false
16771 5190 Acked Write 2 Done Done Sent false
16772 5191 Acked Read 3 Done Done Sent false
16773 5192 Acked Read 7 Done Done Sent false
16774 5193 Acked Read 6 Done Done Sent false
16775 5194 Acked Flush 0 Done Done Sent false
16776 5195 Acked Read 6 Done Done Sent false
16777 5196 Acked Read 8 Done Done Sent false
16778 5197 Acked Read 2 Done Done Sent false
16779 5198 Acked Write 2 Done Done Sent false
16780 5199 Acked Write 7 Done Done Sent false
16781 5200 Acked Flush 0 Done Done Sent false
16782 5201 Acked Read 7 Done Done Sent false
16783 5202 Acked Read 3 Done Done Sent false
16784 5203 Acked Write 9 Done Done Sent false
16785 5204 Acked Write 10 Done Done Sent false
16786 5205 Acked Write 3 Done Done Sent false
16787 5206 Acked Read 7 Done Done Sent false
16788 5207 Acked Write 9 Done Done Sent false
16789 5208 Acked Read 6 Done Done Sent false
16790 5209 Acked Flush 0 Done Done Sent false
16791 5210 Acked Flush 0 Done Done Sent false
16792 5211 Acked Read 2 Done Done Sent false
16793 5212 Acked Read 5 Done Done Sent false
16794 5213 Acked Read 7 Done Done Sent false
16795 5214 Acked Write 4 Done Done Sent false
16796 5215 Acked Read 2 Done Done Sent false
16797 5216 Acked Read 3 Done Done Sent false
16798 5217 Acked Read 8 Done Done Sent false
16799 5218 Acked Write 6 Done Done Sent false
16800 5219 Acked Read 7 Done Done Sent false
16801 5220 Acked Write 1 Done Done Sent false
16802 5221 Acked Read 5 Done Done Sent false
16803 5222 Acked Read 7 Done Done Sent false
16804 5223 Acked Write 6 Done Done Sent false
16805 STATES DS:0 DS:1 DS:2 TOTAL
16806 Sent 0 0 301 301
16807 Done 301 301 0 602
16808 Skipped 0 0 0 0
16809 Error 0 0 0 0
16810 Last Flush: Some(JobId(5210)) Some(JobId(5210)) Some(JobId(4922))
16811 Downstairs last five completed: 4922 4921 4920 4919 4918
16812 Upstairs last five completed: 5223 5222 5221 5220 5219
16813 CLIENT: Up:0 ds:301 act:2
168142025-02-28T06:04:39.864ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
168152025-02-28T06:04:40.703ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
168162025-02-28T06:04:41.892ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
168172025-02-28T06:04:42.705ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
16818 ----------------------------------------------------------------
16819 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16820 States: Active Active Offline
16821 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16822 4923 Acked Flush 0 Done Done Sent false
16823 4924 Acked Read 4 Done Done Sent false
16824 4925 Acked Read 3 Done Done Sent false
16825 4926 Acked Write 9 Done Done Sent false
16826 4927 Acked Read 9 Done Done Sent false
16827 4928 Acked Read 7 Done Done Sent false
16828 4929 Acked Write 7 Done Done Sent false
16829 4930 Acked Read 1 Done Done Sent false
16830 4931 Acked Write 6 Done Done Sent false
16831 4932 Acked Read 5 Done Done Sent false
16832 4933 Acked Write 1 Done Done Sent false
16833 4934 Acked Read 6 Done Done Sent false
16834 4935 Acked Read 1 Done Done Sent false
16835 4936 Acked Read 5 Done Done Sent false
16836 4937 Acked Read 3 Done Done Sent false
16837 4938 Acked Flush 0 Done Done Sent false
16838 4939 Acked Read 5 Done Done Sent false
16839 4940 Acked Read 9 Done Done Sent false
16840 4941 Acked Flush 0 Done Done Sent false
16841 4942 Acked Read 2 Done Done Sent false
16842 4943 Acked Read 6 Done Done Sent false
16843 4944 Acked Flush 0 Done Done Sent false
16844 4945 Acked Write 7 Done Done Sent false
16845 4946 Acked Write 1 Done Done Sent false
16846 4947 Acked Write 2 Done Done Sent false
16847 4948 Acked Write 6 Done Done Sent false
16848 4949 Acked Read 8 Done Done Sent false
16849 4950 Acked Write 5 Done Done Sent false
16850 4951 Acked Write 2 Done Done Sent false
16851 4952 Acked Write 7 Done Done Sent false
16852 4953 Acked Read 8 Done Done Sent false
16853 4954 Acked Read 9 Done Done Sent false
16854 4955 Acked Flush 0 Done Done Sent false
16855 4956 Acked Write 3 Done Done Sent false
16856 4957 Acked Write 9 Done Done Sent false
16857 4958 Acked Read 1 Done Done Sent false
16858 4959 Acked Read 8 Done Done Sent false
16859 4960 Acked Read 10 Done Done Sent false
16860 4961 Acked Read 4 Done Done Sent false
16861 4962 Acked Write 4 Done Done Sent false
16862 4963 Acked Write 10 Done Done Sent false
16863 4964 Acked Read 10 Done Done Sent false
16864 4965 Acked Write 2 Done Done Sent false
16865 4966 Acked Read 6 Done Done Sent false
16866 4967 Acked Flush 0 Done Done Sent false
16867 4968 Acked Read 6 Done Done Sent false
16868 4969 Acked Write 4 Done Done Sent false
16869 4970 Acked Read 2 Done Done Sent false
16870 4971 Acked Write 7 Done Done Sent false
16871 4972 Acked Read 1 Done Done Sent false
16872 4973 Acked Write 9 Done Done Sent false
16873 4974 Acked Read 8 Done Done Sent false
16874 4975 Acked Read 4 Done Done Sent false
16875 4976 Acked Write 2 Done Done Sent false
16876 4977 Acked Write 6 Done Done Sent false
16877 4978 Acked Write 8 Done Done Sent false
16878 4979 Acked Write 7 Done Done Sent false
16879 4980 Acked Write 6 Done Done Sent false
16880 4981 Acked Flush 0 Done Done Sent false
16881 4982 Acked Read 10 Done Done Sent false
16882 4983 Acked Write 1 Done Done Sent false
16883 4984 Acked Read 5 Done Done Sent false
16884 4985 Acked Read 2 Done Done Sent false
16885 4986 Acked Read 7 Done Done Sent false
16886 4987 Acked Read 6 Done Done Sent false
16887 4988 Acked Read 2 Done Done Sent false
16888 4989 Acked Write 1 Done Done Sent false
16889 4990 Acked Write 8 Done Done Sent false
16890 4991 Acked Read 8 Done Done Sent false
16891 4992 Acked Write 7 Done Done Sent false
16892 4993 Acked Write 3 Done Done Sent false
16893 4994 Acked Write 5 Done Done Sent false
16894 4995 Acked Read 2 Done Done Sent false
16895 4996 Acked Flush 0 Done Done Sent false
16896 4997 Acked Write 3 Done Done Sent false
16897 4998 Acked Read 10 Done Done Sent false
16898 4999 Acked Write 4 Done Done Sent false
16899 5000 Acked Write 6 Done Done Sent false
16900 5001 Acked Write 10 Done Done Sent false
16901 5002 Acked Write 7 Done Done Sent false
16902 5003 Acked Write 2 Done Done Sent false
16903 5004 Acked Write 6 Done Done Sent false
16904 5005 Acked Flush 0 Done Done Sent false
16905 5006 Acked Write 4 Done Done Sent false
16906 5007 Acked Write 1 Done Done Sent false
16907 5008 Acked Write 8 Done Done Sent false
16908 5009 Acked Write 6 Done Done Sent false
16909 5010 Acked Read 7 Done Done Sent false
16910 5011 Acked Write 2 Done Done Sent false
16911 5012 Acked Read 7 Done Done Sent false
16912 5013 Acked Read 8 Done Done Sent false
16913 5014 Acked Read 4 Done Done Sent false
16914 5015 Acked Write 6 Done Done Sent false
16915 5016 Acked Read 4 Done Done Sent false
16916 5017 Acked Write 4 Done Done Sent false
16917 5018 Acked Write 9 Done Done Sent false
16918 5019 Acked Write 4 Done Done Sent false
16919 5020 Acked Read 5 Done Done Sent false
16920 5021 Acked Write 4 Done Done Sent false
16921 5022 Acked Read 8 Done Done Sent false
16922 5023 Acked Read 5 Done Done Sent false
16923 5024 Acked Write 1 Done Done Sent false
16924 5025 Acked Read 3 Done Done Sent false
16925 5026 Acked Read 4 Done Done Sent false
16926 5027 Acked Flush 0 Done Done Sent false
16927 5028 Acked Read 1 Done Done Sent false
16928 5029 Acked Write 5 Done Done Sent false
16929 5030 Acked Flush 0 Done Done Sent false
16930 5031 Acked Read 10 Done Done Sent false
16931 5032 Acked Read 4 Done Done Sent false
16932 5033 Acked Read 1 Done Done Sent false
16933 5034 Acked Read 2 Done Done Sent false
16934 5035 Acked Read 2 Done Done Sent false
16935 5036 Acked Write 6 Done Done Sent false
16936 5037 Acked Write 6 Done Done Sent false
16937 5038 Acked Write 5 Done Done Sent false
16938 5039 Acked Write 2 Done Done Sent false
16939 5040 Acked Write 2 Done Done Sent false
16940 5041 Acked Read 5 Done Done Sent false
16941 5042 Acked Write 5 Done Done Sent false
16942 5043 Acked Read 1 Done Done Sent false
16943 5044 Acked Read 2 Done Done Sent false
16944 5045 Acked Write 1 Done Done Sent false
16945 5046 Acked Read 6 Done Done Sent false
16946 5047 Acked Write 3 Done Done Sent false
16947 5048 Acked Read 5 Done Done Sent false
16948 5049 Acked Write 9 Done Done Sent false
16949 5050 Acked Read 8 Done Done Sent false
16950 5051 Acked Write 2 Done Done Sent false
16951 5052 Acked Read 4 Done Done Sent false
16952 5053 Acked Read 1 Done Done Sent false
16953 5054 Acked Write 6 Done Done Sent false
16954 5055 Acked Read 5 Done Done Sent false
16955 5056 Acked Write 1 Done Done Sent false
16956 5057 Acked Write 5 Done Done Sent false
16957 5058 Acked Write 9 Done Done Sent false
16958 5059 Acked Write 3 Done Done Sent false
16959 5060 Acked Read 2 Done Done Sent false
16960 5061 Acked Flush 0 Done Done Sent false
16961 5062 Acked Write 2 Done Done Sent false
16962 5063 Acked Write 4 Done Done Sent false
16963 5064 Acked Flush 0 Done Done Sent false
16964 5065 Acked Read 4 Done Done Sent false
16965 5066 Acked Read 10 Done Done Sent false
16966 5067 Acked Read 8 Done Done Sent false
16967 5068 Acked Write 2 Done Done Sent false
16968 5069 Acked Flush 0 Done Done Sent false
16969 5070 Acked Write 7 Done Done Sent false
16970 5071 Acked Read 9 Done Done Sent false
16971 5072 Acked Read 10 Done Done Sent false
16972 5073 Acked Read 7 Done Done Sent false
16973 5074 Acked Write 7 Done Done Sent false
16974 5075 Acked Read 7 Done Done Sent false
16975 5076 Acked Read 8 Done Done Sent false
16976 5077 Acked Write 1 Done Done Sent false
16977 5078 Acked Write 10 Done Done Sent false
16978 5079 Acked Read 2 Done Done Sent false
16979 5080 Acked Write 4 Done Done Sent false
16980 5081 Acked Write 2 Done Done Sent false
16981 5082 Acked Write 10 Done Done Sent false
16982 5083 Acked Read 10 Done Done Sent false
16983 5084 Acked Read 10 Done Done Sent false
16984 5085 Acked Write 7 Done Done Sent false
16985 5086 Acked Write 7 Done Done Sent false
16986 5087 Acked Read 1 Done Done Sent false
16987 5088 Acked Flush 0 Done Done Sent false
16988 5089 Acked Read 10 Done Done Sent false
16989 5090 Acked Flush 0 Done Done Sent false
16990 5091 Acked Read 10 Done Done Sent false
16991 5092 Acked Flush 0 Done Done Sent false
16992 5093 Acked Write 3 Done Done Sent false
16993 5094 Acked Write 4 Done Done Sent false
16994 5095 Acked Write 6 Done Done Sent false
16995 5096 Acked Write 9 Done Done Sent false
16996 5097 Acked Read 6 Done Done Sent false
16997 5098 Acked Read 6 Done Done Sent false
16998 5099 Acked Flush 0 Done Done Sent false
16999 5100 Acked Write 1 Done Done Sent false
17000 5101 Acked Write 2 Done Done Sent false
17001 5102 Acked Read 3 Done Done Sent false
17002 5103 Acked Read 2 Done Done Sent false
17003 5104 Acked Write 5 Done Done Sent false
17004 5105 Acked Read 9 Done Done Sent false
17005 5106 Acked Read 6 Done Done Sent false
17006 5107 Acked Read 6 Done Done Sent false
17007 5108 Acked Read 1 Done Done Sent false
17008 5109 Acked Write 7 Done Done Sent false
17009 5110 Acked Read 2 Done Done Sent false
17010 5111 Acked Read 9 Done Done Sent false
17011 5112 Acked Read 6 Done Done Sent false
17012 5113 Acked Read 1 Done Done Sent false
17013 5114 Acked Flush 0 Done Done Sent false
17014 5115 Acked Flush 0 Done Done Sent false
17015 5116 Acked Read 7 Done Done Sent false
17016 5117 Acked Write 5 Done Done Sent false
17017 5118 Acked Read 8 Done Done Sent false
17018 5119 Acked Read 2 Done Done Sent false
17019 5120 Acked Read 9 Done Done Sent false
17020 5121 Acked Read 1 Done Done Sent false
17021 5122 Acked Read 2 Done Done Sent false
17022 5123 Acked Read 8 Done Done Sent false
17023 5124 Acked Read 4 Done Done Sent false
17024 5125 Acked Read 4 Done Done Sent false
17025 5126 Acked Read 2 Done Done Sent false
17026 5127 Acked Read 8 Done Done Sent false
17027 5128 Acked Read 6 Done Done Sent false
17028 5129 Acked Write 8 Done Done Sent false
17029 5130 Acked Write 3 Done Done Sent false
17030 5131 Acked Write 3 Done Done Sent false
17031 5132 Acked Read 4 Done Done Sent false
17032 5133 Acked Flush 0 Done Done Sent false
17033 5134 Acked Read 9 Done Done Sent false
17034 5135 Acked Read 4 Done Done Sent false
17035 5136 Acked Write 2 Done Done Sent false
17036 5137 Acked Write 9 Done Done Sent false
17037 5138 Acked Write 10 Done Done Sent false
17038 5139 Acked Read 1 Done Done Sent false
17039 5140 Acked Write 7 Done Done Sent false
17040 5141 Acked Read 8 Done Done Sent false
17041 5142 Acked Read 7 Done Done Sent false
17042 5143 Acked Write 6 Done Done Sent false
17043 5144 Acked Read 5 Done Done Sent false
17044 5145 Acked Write 8 Done Done Sent false
17045 5146 Acked Read 4 Done Done Sent false
17046 5147 Acked Read 4 Done Done Sent false
17047 5148 Acked Read 3 Done Done Sent false
17048 5149 Acked Read 9 Done Done Sent false
17049 5150 Acked Read 6 Done Done Sent false
17050 5151 Acked Flush 0 Done Done Sent false
17051 5152 Acked Write 8 Done Done Sent false
17052 5153 Acked Flush 0 Done Done Sent false
17053 5154 Acked Write 10 Done Done Sent false
17054 5155 Acked Write 9 Done Done Sent false
17055 5156 Acked Write 2 Done Done Sent false
17056 5157 Acked Read 7 Done Done Sent false
17057 5158 Acked Write 9 Done Done Sent false
17058 5159 Acked Read 7 Done Done Sent false
17059 5160 Acked Flush 0 Done Done Sent false
17060 5161 Acked Write 4 Done Done Sent false
17061 5162 Acked Write 2 Done Done Sent false
17062 5163 Acked Read 6 Done Done Sent false
17063 5164 Acked Read 1 Done Done Sent false
17064 5165 Acked Read 1 Done Done Sent false
17065 5166 Acked Flush 0 Done Done Sent false
17066 5167 Acked Read 8 Done Done Sent false
17067 5168 Acked Flush 0 Done Done Sent false
17068 5169 Acked Flush 0 Done Done Sent false
17069 5170 Acked Read 4 Done Done Sent false
17070 5171 Acked Write 7 Done Done Sent false
17071 5172 Acked Flush 0 Done Done Sent false
17072 5173 Acked Flush 0 Done Done Sent false
17073 5174 Acked Read 10 Done Done Sent false
17074 5175 Acked Write 1 Done Done Sent false
17075 5176 Acked Flush 0 Done Done Sent false
17076 5177 Acked Write 1 Done Done Sent false
17077 5178 Acked Write 6 Done Done Sent false
17078 5179 Acked Write 1 Done Done Sent false
17079 5180 Acked Read 4 Done Done Sent false
17080 5181 Acked Read 1 Done Done Sent false
17081 5182 Acked Write 3 Done Done Sent false
17082 5183 Acked Read 9 Done Done Sent false
17083 5184 Acked Read 4 Done Done Sent false
17084 5185 Acked Write 10 Done Done Sent false
17085 5186 Acked Write 3 Done Done Sent false
17086 5187 Acked Read 7 Done Done Sent false
17087 5188 Acked Write 3 Done Done Sent false
17088 5189 Acked Read 10 Done Done Sent false
17089 5190 Acked Write 2 Done Done Sent false
17090 5191 Acked Read 3 Done Done Sent false
17091 5192 Acked Read 7 Done Done Sent false
17092 5193 Acked Read 6 Done Done Sent false
17093 5194 Acked Flush 0 Done Done Sent false
17094 5195 Acked Read 6 Done Done Sent false
17095 5196 Acked Read 8 Done Done Sent false
17096 5197 Acked Read 2 Done Done Sent false
17097 5198 Acked Write 2 Done Done Sent false
17098 5199 Acked Write 7 Done Done Sent false
17099 5200 Acked Flush 0 Done Done Sent false
17100 5201 Acked Read 7 Done Done Sent false
17101 5202 Acked Read 3 Done Done Sent false
17102 5203 Acked Write 9 Done Done Sent false
17103 5204 Acked Write 10 Done Done Sent false
17104 5205 Acked Write 3 Done Done Sent false
17105 5206 Acked Read 7 Done Done Sent false
17106 5207 Acked Write 9 Done Done Sent false
17107 5208 Acked Read 6 Done Done Sent false
17108 5209 Acked Flush 0 Done Done Sent false
17109 5210 Acked Flush 0 Done Done Sent false
17110 5211 Acked Read 2 Done Done Sent false
17111 5212 Acked Read 5 Done Done Sent false
17112 5213 Acked Read 7 Done Done Sent false
17113 5214 Acked Write 4 Done Done Sent false
17114 5215 Acked Read 2 Done Done Sent false
17115 5216 Acked Read 3 Done Done Sent false
17116 5217 Acked Read 8 Done Done Sent false
17117 5218 Acked Write 6 Done Done Sent false
17118 5219 Acked Read 7 Done Done Sent false
17119 5220 Acked Write 1 Done Done Sent false
17120 5221 Acked Read 5 Done Done Sent false
17121 5222 Acked Read 7 Done Done Sent false
17122 5223 Acked Write 6 Done Done Sent false
17123 STATES DS:0 DS:1 DS:2 TOTAL
17124 Sent 0 0 301 301
17125 Done 301 301 0 602
17126 Skipped 0 0 0 0
17127 Error 0 0 0 0
17128 Last Flush: Some(JobId(5210)) Some(JobId(5210)) Some(JobId(4922))
17129 Downstairs last five completed: 4922 4921 4920 4919 4918
17130 Upstairs last five completed: 5223 5222 5221 5220 5219
17131 CLIENT: Up:0 ds:301 act:2
171322025-02-28T06:04:43.868ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
171332025-02-28T06:04:44.708ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
171342025-02-28T06:04:45.129ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
171352025-02-28T06:04:45.129ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
171362025-02-28T06:04:45.129ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
171372025-02-28T06:04:45.129ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
171382025-02-28T06:04:45.129ZINFOcrucible: client task is exiting = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
171392025-02-28T06:04:45.709ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
171402025-02-28T06:04:46.209ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
171412025-02-28T06:04:46.894ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
17142 ----------------------------------------------------------------
17143 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17144 States: Active Active Offline
17145 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17146 4923 Acked Flush 0 Done Done Sent false
17147 4924 Acked Read 4 Done Done Sent false
17148 4925 Acked Read 3 Done Done Sent false
17149 4926 Acked Write 9 Done Done Sent false
17150 4927 Acked Read 9 Done Done Sent false
17151 4928 Acked Read 7 Done Done Sent false
17152 4929 Acked Write 7 Done Done Sent false
17153 4930 Acked Read 1 Done Done Sent false
17154 4931 Acked Write 6 Done Done Sent false
17155 4932 Acked Read 5 Done Done Sent false
17156 4933 Acked Write 1 Done Done Sent false
17157 4934 Acked Read 6 Done Done Sent false
17158 4935 Acked Read 1 Done Done Sent false
17159 4936 Acked Read 5 Done Done Sent false
17160 4937 Acked Read 3 Done Done Sent false
17161 4938 Acked Flush 0 Done Done Sent false
17162 4939 Acked Read 5 Done Done Sent false
17163 4940 Acked Read 9 Done Done Sent false
17164 4941 Acked Flush 0 Done Done Sent false
17165 4942 Acked Read 2 Done Done Sent false
17166 4943 Acked Read 6 Done Done Sent false
17167 4944 Acked Flush 0 Done Done Sent false
17168 4945 Acked Write 7 Done Done Sent false
17169 4946 Acked Write 1 Done Done Sent false
17170 4947 Acked Write 2 Done Done Sent false
17171 4948 Acked Write 6 Done Done Sent false
17172 4949 Acked Read 8 Done Done Sent false
17173 4950 Acked Write 5 Done Done Sent false
17174 4951 Acked Write 2 Done Done Sent false
17175 4952 Acked Write 7 Done Done Sent false
17176 4953 Acked Read 8 Done Done Sent false
17177 4954 Acked Read 9 Done Done Sent false
17178 4955 Acked Flush 0 Done Done Sent false
17179 4956 Acked Write 3 Done Done Sent false
17180 4957 Acked Write 9 Done Done Sent false
17181 4958 Acked Read 1 Done Done Sent false
17182 4959 Acked Read 8 Done Done Sent false
17183 4960 Acked Read 10 Done Done Sent false
17184 4961 Acked Read 4 Done Done Sent false
17185 4962 Acked Write 4 Done Done Sent false
17186 4963 Acked Write 10 Done Done Sent false
17187 4964 Acked Read 10 Done Done Sent false
17188 4965 Acked Write 2 Done Done Sent false
17189 4966 Acked Read 6 Done Done Sent false
17190 4967 Acked Flush 0 Done Done Sent false
17191 4968 Acked Read 6 Done Done Sent false
17192 4969 Acked Write 4 Done Done Sent false
17193 4970 Acked Read 2 Done Done Sent false
17194 4971 Acked Write 7 Done Done Sent false
17195 4972 Acked Read 1 Done Done Sent false
17196 4973 Acked Write 9 Done Done Sent false
17197 4974 Acked Read 8 Done Done Sent false
17198 4975 Acked Read 4 Done Done Sent false
17199 4976 Acked Write 2 Done Done Sent false
17200 4977 Acked Write 6 Done Done Sent false
17201 4978 Acked Write 8 Done Done Sent false
17202 4979 Acked Write 7 Done Done Sent false
17203 4980 Acked Write 6 Done Done Sent false
17204 4981 Acked Flush 0 Done Done Sent false
17205 4982 Acked Read 10 Done Done Sent false
17206 4983 Acked Write 1 Done Done Sent false
17207 4984 Acked Read 5 Done Done Sent false
17208 4985 Acked Read 2 Done Done Sent false
17209 4986 Acked Read 7 Done Done Sent false
17210 4987 Acked Read 6 Done Done Sent false
17211 4988 Acked Read 2 Done Done Sent false
17212 4989 Acked Write 1 Done Done Sent false
17213 4990 Acked Write 8 Done Done Sent false
17214 4991 Acked Read 8 Done Done Sent false
17215 4992 Acked Write 7 Done Done Sent false
17216 4993 Acked Write 3 Done Done Sent false
17217 4994 Acked Write 5 Done Done Sent false
17218 4995 Acked Read 2 Done Done Sent false
17219 4996 Acked Flush 0 Done Done Sent false
17220 4997 Acked Write 3 Done Done Sent false
17221 4998 Acked Read 10 Done Done Sent false
17222 4999 Acked Write 4 Done Done Sent false
17223 5000 Acked Write 6 Done Done Sent false
17224 5001 Acked Write 10 Done Done Sent false
17225 5002 Acked Write 7 Done Done Sent false
17226 5003 Acked Write 2 Done Done Sent false
17227 5004 Acked Write 6 Done Done Sent false
17228 5005 Acked Flush 0 Done Done Sent false
17229 5006 Acked Write 4 Done Done Sent false
17230 5007 Acked Write 1 Done Done Sent false
17231 5008 Acked Write 8 Done Done Sent false
17232 5009 Acked Write 6 Done Done Sent false
17233 5010 Acked Read 7 Done Done Sent false
17234 5011 Acked Write 2 Done Done Sent false
17235 5012 Acked Read 7 Done Done Sent false
17236 5013 Acked Read 8 Done Done Sent false
17237 5014 Acked Read 4 Done Done Sent false
17238 5015 Acked Write 6 Done Done Sent false
17239 5016 Acked Read 4 Done Done Sent false
17240 5017 Acked Write 4 Done Done Sent false
17241 5018 Acked Write 9 Done Done Sent false
17242 5019 Acked Write 4 Done Done Sent false
17243 5020 Acked Read 5 Done Done Sent false
17244 5021 Acked Write 4 Done Done Sent false
17245 5022 Acked Read 8 Done Done Sent false
17246 5023 Acked Read 5 Done Done Sent false
17247 5024 Acked Write 1 Done Done Sent false
17248 5025 Acked Read 3 Done Done Sent false
17249 5026 Acked Read 4 Done Done Sent false
17250 5027 Acked Flush 0 Done Done Sent false
17251 5028 Acked Read 1 Done Done Sent false
17252 5029 Acked Write 5 Done Done Sent false
17253 5030 Acked Flush 0 Done Done Sent false
17254 5031 Acked Read 10 Done Done Sent false
17255 5032 Acked Read 4 Done Done Sent false
17256 5033 Acked Read 1 Done Done Sent false
17257 5034 Acked Read 2 Done Done Sent false
17258 5035 Acked Read 2 Done Done Sent false
17259 5036 Acked Write 6 Done Done Sent false
17260 5037 Acked Write 6 Done Done Sent false
17261 5038 Acked Write 5 Done Done Sent false
17262 5039 Acked Write 2 Done Done Sent false
17263 5040 Acked Write 2 Done Done Sent false
17264 5041 Acked Read 5 Done Done Sent false
17265 5042 Acked Write 5 Done Done Sent false
17266 5043 Acked Read 1 Done Done Sent false
17267 5044 Acked Read 2 Done Done Sent false
17268 5045 Acked Write 1 Done Done Sent false
17269 5046 Acked Read 6 Done Done Sent false
17270 5047 Acked Write 3 Done Done Sent false
17271 5048 Acked Read 5 Done Done Sent false
17272 5049 Acked Write 9 Done Done Sent false
17273 5050 Acked Read 8 Done Done Sent false
17274 5051 Acked Write 2 Done Done Sent false
17275 5052 Acked Read 4 Done Done Sent false
17276 5053 Acked Read 1 Done Done Sent false
17277 5054 Acked Write 6 Done Done Sent false
17278 5055 Acked Read 5 Done Done Sent false
17279 5056 Acked Write 1 Done Done Sent false
17280 5057 Acked Write 5 Done Done Sent false
17281 5058 Acked Write 9 Done Done Sent false
17282 5059 Acked Write 3 Done Done Sent false
17283 5060 Acked Read 2 Done Done Sent false
17284 5061 Acked Flush 0 Done Done Sent false
17285 5062 Acked Write 2 Done Done Sent false
17286 5063 Acked Write 4 Done Done Sent false
17287 5064 Acked Flush 0 Done Done Sent false
17288 5065 Acked Read 4 Done Done Sent false
17289 5066 Acked Read 10 Done Done Sent false
17290 5067 Acked Read 8 Done Done Sent false
17291 5068 Acked Write 2 Done Done Sent false
17292 5069 Acked Flush 0 Done Done Sent false
17293 5070 Acked Write 7 Done Done Sent false
17294 5071 Acked Read 9 Done Done Sent false
17295 5072 Acked Read 10 Done Done Sent false
17296 5073 Acked Read 7 Done Done Sent false
17297 5074 Acked Write 7 Done Done Sent false
17298 5075 Acked Read 7 Done Done Sent false
17299 5076 Acked Read 8 Done Done Sent false
17300 5077 Acked Write 1 Done Done Sent false
17301 5078 Acked Write 10 Done Done Sent false
17302 5079 Acked Read 2 Done Done Sent false
17303 5080 Acked Write 4 Done Done Sent false
17304 5081 Acked Write 2 Done Done Sent false
17305 5082 Acked Write 10 Done Done Sent false
17306 5083 Acked Read 10 Done Done Sent false
17307 5084 Acked Read 10 Done Done Sent false
17308 5085 Acked Write 7 Done Done Sent false
17309 5086 Acked Write 7 Done Done Sent false
17310 5087 Acked Read 1 Done Done Sent false
17311 5088 Acked Flush 0 Done Done Sent false
17312 5089 Acked Read 10 Done Done Sent false
17313 5090 Acked Flush 0 Done Done Sent false
17314 5091 Acked Read 10 Done Done Sent false
17315 5092 Acked Flush 0 Done Done Sent false
17316 5093 Acked Write 3 Done Done Sent false
17317 5094 Acked Write 4 Done Done Sent false
17318 5095 Acked Write 6 Done Done Sent false
17319 5096 Acked Write 9 Done Done Sent false
17320 5097 Acked Read 6 Done Done Sent false
17321 5098 Acked Read 6 Done Done Sent false
17322 5099 Acked Flush 0 Done Done Sent false
17323 5100 Acked Write 1 Done Done Sent false
17324 5101 Acked Write 2 Done Done Sent false
17325 5102 Acked Read 3 Done Done Sent false
17326 5103 Acked Read 2 Done Done Sent false
17327 5104 Acked Write 5 Done Done Sent false
17328 5105 Acked Read 9 Done Done Sent false
17329 5106 Acked Read 6 Done Done Sent false
17330 5107 Acked Read 6 Done Done Sent false
17331 5108 Acked Read 1 Done Done Sent false
17332 5109 Acked Write 7 Done Done Sent false
17333 5110 Acked Read 2 Done Done Sent false
17334 5111 Acked Read 9 Done Done Sent false
17335 5112 Acked Read 6 Done Done Sent false
17336 5113 Acked Read 1 Done Done Sent false
17337 5114 Acked Flush 0 Done Done Sent false
17338 5115 Acked Flush 0 Done Done Sent false
17339 5116 Acked Read 7 Done Done Sent false
17340 5117 Acked Write 5 Done Done Sent false
17341 5118 Acked Read 8 Done Done Sent false
17342 5119 Acked Read 2 Done Done Sent false
17343 5120 Acked Read 9 Done Done Sent false
17344 5121 Acked Read 1 Done Done Sent false
17345 5122 Acked Read 2 Done Done Sent false
17346 5123 Acked Read 8 Done Done Sent false
17347 5124 Acked Read 4 Done Done Sent false
17348 5125 Acked Read 4 Done Done Sent false
17349 5126 Acked Read 2 Done Done Sent false
17350 5127 Acked Read 8 Done Done Sent false
17351 5128 Acked Read 6 Done Done Sent false
17352 5129 Acked Write 8 Done Done Sent false
17353 5130 Acked Write 3 Done Done Sent false
17354 5131 Acked Write 3 Done Done Sent false
17355 5132 Acked Read 4 Done Done Sent false
17356 5133 Acked Flush 0 Done Done Sent false
17357 5134 Acked Read 9 Done Done Sent false
17358 5135 Acked Read 4 Done Done Sent false
17359 5136 Acked Write 2 Done Done Sent false
17360 5137 Acked Write 9 Done Done Sent false
17361 5138 Acked Write 10 Done Done Sent false
17362 5139 Acked Read 1 Done Done Sent false
17363 5140 Acked Write 7 Done Done Sent false
17364 5141 Acked Read 8 Done Done Sent false
17365 5142 Acked Read 7 Done Done Sent false
17366 5143 Acked Write 6 Done Done Sent false
17367 5144 Acked Read 5 Done Done Sent false
17368 5145 Acked Write 8 Done Done Sent false
17369 5146 Acked Read 4 Done Done Sent false
17370 5147 Acked Read 4 Done Done Sent false
17371 5148 Acked Read 3 Done Done Sent false
17372 5149 Acked Read 9 Done Done Sent false
17373 5150 Acked Read 6 Done Done Sent false
17374 5151 Acked Flush 0 Done Done Sent false
17375 5152 Acked Write 8 Done Done Sent false
17376 5153 Acked Flush 0 Done Done Sent false
17377 5154 Acked Write 10 Done Done Sent false
17378 5155 Acked Write 9 Done Done Sent false
17379 5156 Acked Write 2 Done Done Sent false
17380 5157 Acked Read 7 Done Done Sent false
17381 5158 Acked Write 9 Done Done Sent false
17382 5159 Acked Read 7 Done Done Sent false
17383 5160 Acked Flush 0 Done Done Sent false
17384 5161 Acked Write 4 Done Done Sent false
17385 5162 Acked Write 2 Done Done Sent false
17386 5163 Acked Read 6 Done Done Sent false
17387 5164 Acked Read 1 Done Done Sent false
17388 5165 Acked Read 1 Done Done Sent false
17389 5166 Acked Flush 0 Done Done Sent false
17390 5167 Acked Read 8 Done Done Sent false
17391 5168 Acked Flush 0 Done Done Sent false
17392 5169 Acked Flush 0 Done Done Sent false
17393 5170 Acked Read 4 Done Done Sent false
17394 5171 Acked Write 7 Done Done Sent false
17395 5172 Acked Flush 0 Done Done Sent false
17396 5173 Acked Flush 0 Done Done Sent false
17397 5174 Acked Read 10 Done Done Sent false
17398 5175 Acked Write 1 Done Done Sent false
17399 5176 Acked Flush 0 Done Done Sent false
17400 5177 Acked Write 1 Done Done Sent false
17401 5178 Acked Write 6 Done Done Sent false
17402 5179 Acked Write 1 Done Done Sent false
17403 5180 Acked Read 4 Done Done Sent false
17404 5181 Acked Read 1 Done Done Sent false
17405 5182 Acked Write 3 Done Done Sent false
17406 5183 Acked Read 9 Done Done Sent false
17407 5184 Acked Read 4 Done Done Sent false
17408 5185 Acked Write 10 Done Done Sent false
17409 5186 Acked Write 3 Done Done Sent false
17410 5187 Acked Read 7 Done Done Sent false
17411 5188 Acked Write 3 Done Done Sent false
17412 5189 Acked Read 10 Done Done Sent false
17413 5190 Acked Write 2 Done Done Sent false
17414 5191 Acked Read 3 Done Done Sent false
17415 5192 Acked Read 7 Done Done Sent false
17416 5193 Acked Read 6 Done Done Sent false
17417 5194 Acked Flush 0 Done Done Sent false
17418 5195 Acked Read 6 Done Done Sent false
17419 5196 Acked Read 8 Done Done Sent false
17420 5197 Acked Read 2 Done Done Sent false
17421 5198 Acked Write 2 Done Done Sent false
17422 5199 Acked Write 7 Done Done Sent false
17423 5200 Acked Flush 0 Done Done Sent false
17424 5201 Acked Read 7 Done Done Sent false
17425 5202 Acked Read 3 Done Done Sent false
17426 5203 Acked Write 9 Done Done Sent false
17427 5204 Acked Write 10 Done Done Sent false
17428 5205 Acked Write 3 Done Done Sent false
17429 5206 Acked Read 7 Done Done Sent false
17430 5207 Acked Write 9 Done Done Sent false
17431 5208 Acked Read 6 Done Done Sent false
17432 5209 Acked Flush 0 Done Done Sent false
17433 5210 Acked Flush 0 Done Done Sent false
17434 5211 Acked Read 2 Done Done Sent false
17435 5212 Acked Read 5 Done Done Sent false
17436 5213 Acked Read 7 Done Done Sent false
17437 5214 Acked Write 4 Done Done Sent false
17438 5215 Acked Read 2 Done Done Sent false
17439 5216 Acked Read 3 Done Done Sent false
17440 5217 Acked Read 8 Done Done Sent false
17441 5218 Acked Write 6 Done Done Sent false
17442 5219 Acked Read 7 Done Done Sent false
17443 5220 Acked Write 1 Done Done Sent false
17444 5221 Acked Read 5 Done Done Sent false
17445 5222 Acked Read 7 Done Done Sent false
17446 5223 Acked Write 6 Done Done Sent false
17447 STATES DS:0 DS:1 DS:2 TOTAL
17448 Sent 0 0 301 301
17449 Done 301 301 0 602
17450 Skipped 0 0 0 0
17451 Error 0 0 0 0
17452 Last Flush: Some(JobId(5210)) Some(JobId(5210)) Some(JobId(4922))
17453 Downstairs last five completed: 4922 4921 4920 4919 4918
17454 Upstairs last five completed: 5223 5222 5221 5220 5219
17455 CLIENT: Up:0 ds:301 act:2
174562025-02-28T06:04:47.869ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
174572025-02-28T06:04:48.714ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
17458 [8830] got start action
17459 [8830] I am going to restart
17460 Starting downstairs at port 8830
17461 Make output file at "/tmp/dsc/downstairs-8830.txt"
17462 [8830][2] reports Starting
17463 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1225)
17464 [8830][2] reports Running
174652025-02-28T06:04:49.715ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
174662025-02-28T06:04:50.717ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
17467 ----------------------------------------------------------------
17468 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17469 States: Active Active Offline
17470 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17471 4923 Acked Flush 0 Done Done Sent false
17472 4924 Acked Read 4 Done Done Sent false
17473 4925 Acked Read 3 Done Done Sent false
17474 4926 Acked Write 9 Done Done Sent false
17475 4927 Acked Read 9 Done Done Sent false
17476 4928 Acked Read 7 Done Done Sent false
17477 4929 Acked Write 7 Done Done Sent false
17478 4930 Acked Read 1 Done Done Sent false
17479 4931 Acked Write 6 Done Done Sent false
17480 4932 Acked Read 5 Done Done Sent false
17481 4933 Acked Write 1 Done Done Sent false
17482 4934 Acked Read 6 Done Done Sent false
17483 4935 Acked Read 1 Done Done Sent false
17484 4936 Acked Read 5 Done Done Sent false
17485 4937 Acked Read 3 Done Done Sent false
17486 4938 Acked Flush 0 Done Done Sent false
17487 4939 Acked Read 5 Done Done Sent false
17488 4940 Acked Read 9 Done Done Sent false
17489 4941 Acked Flush 0 Done Done Sent false
17490 4942 Acked Read 2 Done Done Sent false
17491 4943 Acked Read 6 Done Done Sent false
17492 4944 Acked Flush 0 Done Done Sent false
17493 4945 Acked Write 7 Done Done Sent false
17494 4946 Acked Write 1 Done Done Sent false
17495 4947 Acked Write 2 Done Done Sent false
17496 4948 Acked Write 6 Done Done Sent false
17497 4949 Acked Read 8 Done Done Sent false
17498 4950 Acked Write 5 Done Done Sent false
17499 4951 Acked Write 2 Done Done Sent false
17500 4952 Acked Write 7 Done Done Sent false
17501 4953 Acked Read 8 Done Done Sent false
17502 4954 Acked Read 9 Done Done Sent false
17503 4955 Acked Flush 0 Done Done Sent false
17504 4956 Acked Write 3 Done Done Sent false
17505 4957 Acked Write 9 Done Done Sent false
17506 4958 Acked Read 1 Done Done Sent false
17507 4959 Acked Read 8 Done Done Sent false
17508 4960 Acked Read 10 Done Done Sent false
17509 4961 Acked Read 4 Done Done Sent false
17510 4962 Acked Write 4 Done Done Sent false
17511 4963 Acked Write 10 Done Done Sent false
17512 4964 Acked Read 10 Done Done Sent false
17513 4965 Acked Write 2 Done Done Sent false
17514 4966 Acked Read 6 Done Done Sent false
17515 4967 Acked Flush 0 Done Done Sent false
17516 4968 Acked Read 6 Done Done Sent false
17517 4969 Acked Write 4 Done Done Sent false
17518 4970 Acked Read 2 Done Done Sent false
17519 4971 Acked Write 7 Done Done Sent false
17520 4972 Acked Read 1 Done Done Sent false
17521 4973 Acked Write 9 Done Done Sent false
17522 4974 Acked Read 8 Done Done Sent false
17523 4975 Acked Read 4 Done Done Sent false
17524 4976 Acked Write 2 Done Done Sent false
17525 4977 Acked Write 6 Done Done Sent false
17526 4978 Acked Write 8 Done Done Sent false
17527 4979 Acked Write 7 Done Done Sent false
17528 4980 Acked Write 6 Done Done Sent false
17529 4981 Acked Flush 0 Done Done Sent false
17530 4982 Acked Read 10 Done Done Sent false
17531 4983 Acked Write 1 Done Done Sent false
17532 4984 Acked Read 5 Done Done Sent false
17533 4985 Acked Read 2 Done Done Sent false
17534 4986 Acked Read 7 Done Done Sent false
17535 4987 Acked Read 6 Done Done Sent false
17536 4988 Acked Read 2 Done Done Sent false
17537 4989 Acked Write 1 Done Done Sent false
17538 4990 Acked Write 8 Done Done Sent false
17539 4991 Acked Read 8 Done Done Sent false
17540 4992 Acked Write 7 Done Done Sent false
17541 4993 Acked Write 3 Done Done Sent false
17542 4994 Acked Write 5 Done Done Sent false
17543 4995 Acked Read 2 Done Done Sent false
17544 4996 Acked Flush 0 Done Done Sent false
17545 4997 Acked Write 3 Done Done Sent false
17546 4998 Acked Read 10 Done Done Sent false
17547 4999 Acked Write 4 Done Done Sent false
17548 5000 Acked Write 6 Done Done Sent false
17549 5001 Acked Write 10 Done Done Sent false
17550 5002 Acked Write 7 Done Done Sent false
17551 5003 Acked Write 2 Done Done Sent false
17552 5004 Acked Write 6 Done Done Sent false
17553 5005 Acked Flush 0 Done Done Sent false
17554 5006 Acked Write 4 Done Done Sent false
17555 5007 Acked Write 1 Done Done Sent false
17556 5008 Acked Write 8 Done Done Sent false
17557 5009 Acked Write 6 Done Done Sent false
17558 5010 Acked Read 7 Done Done Sent false
17559 5011 Acked Write 2 Done Done Sent false
17560 5012 Acked Read 7 Done Done Sent false
17561 5013 Acked Read 8 Done Done Sent false
17562 5014 Acked Read 4 Done Done Sent false
17563 5015 Acked Write 6 Done Done Sent false
17564 5016 Acked Read 4 Done Done Sent false
17565 5017 Acked Write 4 Done Done Sent false
17566 5018 Acked Write 9 Done Done Sent false
17567 5019 Acked Write 4 Done Done Sent false
17568 5020 Acked Read 5 Done Done Sent false
17569 5021 Acked Write 4 Done Done Sent false
17570 5022 Acked Read 8 Done Done Sent false
17571 5023 Acked Read 5 Done Done Sent false
17572 5024 Acked Write 1 Done Done Sent false
17573 5025 Acked Read 3 Done Done Sent false
17574 5026 Acked Read 4 Done Done Sent false
17575 5027 Acked Flush 0 Done Done Sent false
17576 5028 Acked Read 1 Done Done Sent false
17577 5029 Acked Write 5 Done Done Sent false
17578 5030 Acked Flush 0 Done Done Sent false
17579 5031 Acked Read 10 Done Done Sent false
17580 5032 Acked Read 4 Done Done Sent false
17581 5033 Acked Read 1 Done Done Sent false
17582 5034 Acked Read 2 Done Done Sent false
17583 5035 Acked Read 2 Done Done Sent false
17584 5036 Acked Write 6 Done Done Sent false
17585 5037 Acked Write 6 Done Done Sent false
17586 5038 Acked Write 5 Done Done Sent false
17587 5039 Acked Write 2 Done Done Sent false
17588 5040 Acked Write 2 Done Done Sent false
17589 5041 Acked Read 5 Done Done Sent false
17590 5042 Acked Write 5 Done Done Sent false
17591 5043 Acked Read 1 Done Done Sent false
17592 5044 Acked Read 2 Done Done Sent false
17593 5045 Acked Write 1 Done Done Sent false
17594 5046 Acked Read 6 Done Done Sent false
17595 5047 Acked Write 3 Done Done Sent false
17596 5048 Acked Read 5 Done Done Sent false
17597 5049 Acked Write 9 Done Done Sent false
17598 5050 Acked Read 8 Done Done Sent false
17599 5051 Acked Write 2 Done Done Sent false
17600 5052 Acked Read 4 Done Done Sent false
17601 5053 Acked Read 1 Done Done Sent false
17602 5054 Acked Write 6 Done Done Sent false
17603 5055 Acked Read 5 Done Done Sent false
17604 5056 Acked Write 1 Done Done Sent false
17605 5057 Acked Write 5 Done Done Sent false
17606 5058 Acked Write 9 Done Done Sent false
17607 5059 Acked Write 3 Done Done Sent false
17608 5060 Acked Read 2 Done Done Sent false
17609 5061 Acked Flush 0 Done Done Sent false
17610 5062 Acked Write 2 Done Done Sent false
17611 5063 Acked Write 4 Done Done Sent false
17612 5064 Acked Flush 0 Done Done Sent false
17613 5065 Acked Read 4 Done Done Sent false
17614 5066 Acked Read 10 Done Done Sent false
17615 5067 Acked Read 8 Done Done Sent false
17616 5068 Acked Write 2 Done Done Sent false
17617 5069 Acked Flush 0 Done Done Sent false
17618 5070 Acked Write 7 Done Done Sent false
17619 5071 Acked Read 9 Done Done Sent false
17620 5072 Acked Read 10 Done Done Sent false
17621 5073 Acked Read 7 Done Done Sent false
17622 5074 Acked Write 7 Done Done Sent false
17623 5075 Acked Read 7 Done Done Sent false
17624 5076 Acked Read 8 Done Done Sent false
17625 5077 Acked Write 1 Done Done Sent false
17626 5078 Acked Write 10 Done Done Sent false
17627 5079 Acked Read 2 Done Done Sent false
17628 5080 Acked Write 4 Done Done Sent false
17629 5081 Acked Write 2 Done Done Sent false
17630 5082 Acked Write 10 Done Done Sent false
17631 5083 Acked Read 10 Done Done Sent false
17632 5084 Acked Read 10 Done Done Sent false
17633 5085 Acked Write 7 Done Done Sent false
17634 5086 Acked Write 7 Done Done Sent false
17635 5087 Acked Read 1 Done Done Sent false
17636 5088 Acked Flush 0 Done Done Sent false
17637 5089 Acked Read 10 Done Done Sent false
17638 5090 Acked Flush 0 Done Done Sent false
17639 5091 Acked Read 10 Done Done Sent false
17640 5092 Acked Flush 0 Done Done Sent false
17641 5093 Acked Write 3 Done Done Sent false
17642 5094 Acked Write 4 Done Done Sent false
17643 5095 Acked Write 6 Done Done Sent false
17644 5096 Acked Write 9 Done Done Sent false
17645 5097 Acked Read 6 Done Done Sent false
17646 5098 Acked Read 6 Done Done Sent false
17647 5099 Acked Flush 0 Done Done Sent false
17648 5100 Acked Write 1 Done Done Sent false
17649 5101 Acked Write 2 Done Done Sent false
17650 5102 Acked Read 3 Done Done Sent false
17651 5103 Acked Read 2 Done Done Sent false
17652 5104 Acked Write 5 Done Done Sent false
17653 5105 Acked Read 9 Done Done Sent false
17654 5106 Acked Read 6 Done Done Sent false
17655 5107 Acked Read 6 Done Done Sent false
17656 5108 Acked Read 1 Done Done Sent false
17657 5109 Acked Write 7 Done Done Sent false
17658 5110 Acked Read 2 Done Done Sent false
17659 5111 Acked Read 9 Done Done Sent false
17660 5112 Acked Read 6 Done Done Sent false
17661 5113 Acked Read 1 Done Done Sent false
17662 5114 Acked Flush 0 Done Done Sent false
17663 5115 Acked Flush 0 Done Done Sent false
17664 5116 Acked Read 7 Done Done Sent false
17665 5117 Acked Write 5 Done Done Sent false
17666 5118 Acked Read 8 Done Done Sent false
17667 5119 Acked Read 2 Done Done Sent false
17668 5120 Acked Read 9 Done Done Sent false
17669 5121 Acked Read 1 Done Done Sent false
17670 5122 Acked Read 2 Done Done Sent false
17671 5123 Acked Read 8 Done Done Sent false
17672 5124 Acked Read 4 Done Done Sent false
17673 5125 Acked Read 4 Done Done Sent false
17674 5126 Acked Read 2 Done Done Sent false
17675 5127 Acked Read 8 Done Done Sent false
17676 5128 Acked Read 6 Done Done Sent false
17677 5129 Acked Write 8 Done Done Sent false
17678 5130 Acked Write 3 Done Done Sent false
17679 5131 Acked Write 3 Done Done Sent false
17680 5132 Acked Read 4 Done Done Sent false
17681 5133 Acked Flush 0 Done Done Sent false
17682 5134 Acked Read 9 Done Done Sent false
17683 5135 Acked Read 4 Done Done Sent false
17684 5136 Acked Write 2 Done Done Sent false
17685 5137 Acked Write 9 Done Done Sent false
17686 5138 Acked Write 10 Done Done Sent false
17687 5139 Acked Read 1 Done Done Sent false
17688 5140 Acked Write 7 Done Done Sent false
17689 5141 Acked Read 8 Done Done Sent false
17690 5142 Acked Read 7 Done Done Sent false
17691 5143 Acked Write 6 Done Done Sent false
17692 5144 Acked Read 5 Done Done Sent false
17693 5145 Acked Write 8 Done Done Sent false
17694 5146 Acked Read 4 Done Done Sent false
17695 5147 Acked Read 4 Done Done Sent false
17696 5148 Acked Read 3 Done Done Sent false
17697 5149 Acked Read 9 Done Done Sent false
17698 5150 Acked Read 6 Done Done Sent false
17699 5151 Acked Flush 0 Done Done Sent false
17700 5152 Acked Write 8 Done Done Sent false
17701 5153 Acked Flush 0 Done Done Sent false
17702 5154 Acked Write 10 Done Done Sent false
17703 5155 Acked Write 9 Done Done Sent false
17704 5156 Acked Write 2 Done Done Sent false
17705 5157 Acked Read 7 Done Done Sent false
17706 5158 Acked Write 9 Done Done Sent false
17707 5159 Acked Read 7 Done Done Sent false
17708 5160 Acked Flush 0 Done Done Sent false
17709 5161 Acked Write 4 Done Done Sent false
17710 5162 Acked Write 2 Done Done Sent false
17711 5163 Acked Read 6 Done Done Sent false
17712 5164 Acked Read 1 Done Done Sent false
17713 5165 Acked Read 1 Done Done Sent false
17714 5166 Acked Flush 0 Done Done Sent false
17715 5167 Acked Read 8 Done Done Sent false
17716 5168 Acked Flush 0 Done Done Sent false
17717 5169 Acked Flush 0 Done Done Sent false
17718 5170 Acked Read 4 Done Done Sent false
17719 5171 Acked Write 7 Done Done Sent false
17720 5172 Acked Flush 0 Done Done Sent false
17721 5173 Acked Flush 0 Done Done Sent false
17722 5174 Acked Read 10 Done Done Sent false
17723 5175 Acked Write 1 Done Done Sent false
17724 5176 Acked Flush 0 Done Done Sent false
17725 5177 Acked Write 1 Done Done Sent false
17726 5178 Acked Write 6 Done Done Sent false
17727 5179 Acked Write 1 Done Done Sent false
17728 5180 Acked Read 4 Done Done Sent false
17729 5181 Acked Read 1 Done Done Sent false
17730 5182 Acked Write 3 Done Done Sent false
17731 5183 Acked Read 9 Done Done Sent false
17732 5184 Acked Read 4 Done Done Sent false
17733 5185 Acked Write 10 Done Done Sent false
17734 5186 Acked Write 3 Done Done Sent false
17735 5187 Acked Read 7 Done Done Sent false
17736 5188 Acked Write 3 Done Done Sent false
17737 5189 Acked Read 10 Done Done Sent false
17738 5190 Acked Write 2 Done Done Sent false
17739 5191 Acked Read 3 Done Done Sent false
17740 5192 Acked Read 7 Done Done Sent false
17741 5193 Acked Read 6 Done Done Sent false
17742 5194 Acked Flush 0 Done Done Sent false
17743 5195 Acked Read 6 Done Done Sent false
17744 5196 Acked Read 8 Done Done Sent false
17745 5197 Acked Read 2 Done Done Sent false
17746 5198 Acked Write 2 Done Done Sent false
17747 5199 Acked Write 7 Done Done Sent false
17748 5200 Acked Flush 0 Done Done Sent false
17749 5201 Acked Read 7 Done Done Sent false
17750 5202 Acked Read 3 Done Done Sent false
17751 5203 Acked Write 9 Done Done Sent false
17752 5204 Acked Write 10 Done Done Sent false
17753 5205 Acked Write 3 Done Done Sent false
17754 5206 Acked Read 7 Done Done Sent false
17755 5207 Acked Write 9 Done Done Sent false
17756 5208 Acked Read 6 Done Done Sent false
17757 5209 Acked Flush 0 Done Done Sent false
17758 5210 Acked Flush 0 Done Done Sent false
17759 5211 Acked Read 2 Done Done Sent false
17760 5212 Acked Read 5 Done Done Sent false
17761 5213 Acked Read 7 Done Done Sent false
17762 5214 Acked Write 4 Done Done Sent false
17763 5215 Acked Read 2 Done Done Sent false
17764 5216 Acked Read 3 Done Done Sent false
17765 5217 Acked Read 8 Done Done Sent false
17766 5218 Acked Write 6 Done Done Sent false
17767 5219 Acked Read 7 Done Done Sent false
17768 5220 Acked Write 1 Done Done Sent false
17769 5221 Acked Read 5 Done Done Sent false
17770 5222 Acked Read 7 Done Done Sent false
17771 5223 Acked Write 6 Done Done Sent false
17772 STATES DS:0 DS:1 DS:2 TOTAL
17773 Sent 0 0 301 301
17774 Done 301 301 0 602
17775 Skipped 0 0 0 0
17776 Error 0 0 0 0
17777 Last Flush: Some(JobId(5210)) Some(JobId(5210)) Some(JobId(4922))
17778 Downstairs last five completed: 4922 4921 4920 4919 4918
17779 Upstairs last five completed: 5223 5222 5221 5220 5219
17780 CLIENT: Up:0 ds:301 act:2
177812025-02-28T06:04:51.895ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
177822025-02-28T06:04:52.721ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
177832025-02-28T06:04:53.721ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
177842025-02-28T06:04:54.722ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
177852025-02-28T06:04:55.131ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
177862025-02-28T06:04:55.131ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36327) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
177872025-02-28T06:04:55.132ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
177882025-02-28T06:04:55.132ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
177892025-02-28T06:04:55.132ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
177902025-02-28T06:04:55.132ZINFOcrucible: send last flush ID to this DS: Some(JobId(4922)) = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
177912025-02-28T06:04:55.132ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
177922025-02-28T06:04:55.132ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4922)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
177932025-02-28T06:04:55.133ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4922)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
17794 ----------------------------------------------------------------
17795 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:13
17796 States: Active Active Active
17797 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17798 5211 Acked Read 2 Done Done Done true
17799 5212 Acked Read 5 Done Done Done true
17800 5213 Acked Read 7 Done Done Done true
17801 5214 Acked Write 4 Done Done Done true
17802 5215 Acked Read 2 Done Done Done true
17803 5216 Acked Read 3 Done Done Done true
17804 5217 Acked Read 8 Done Done Done true
17805 5218 Acked Write 6 Done Done Done true
17806 5219 Acked Read 7 Done Done Done true
17807 5220 Acked Write 1 Done Done Done true
17808 5221 Acked Read 5 Done Done Done true
17809 5222 Acked Read 7 Done Done Done true
17810 5223 Acked Write 6 Done Done Done true
17811 STATES DS:0 DS:1 DS:2 TOTAL
17812 Sent 0 0 0 0
17813 Done 13 13 13 39
17814 Skipped 0 0 0 0
17815 Error 0 0 0 0
17816 Last Flush: Some(JobId(5210)) Some(JobId(5210)) Some(JobId(5210))
17817 Downstairs last five completed: 5210 5209 5208 5207 5206
17818 Upstairs last five completed: 5223 5222 5221 5220 5219
17819 CLIENT: Up:0 ds:13 act:3
17820 ----------------------------------------------------------------
17821 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17822 Downstairs last five completed: 5224 5223 5222 5221 5220
17823 Upstairs last five completed: 5224 5223 5222 5221 5220
17824 CLIENT: Up:0 ds:0 act:3
17825 Replay: All jobs finished
17826 Main task has work to do, go find it
17827 got dsc Stop(2)
17828 stop 2
17829 Feb 28 06:04:59.374 [8830] Got stop action so:false kr:false
17830 INFO request completed, latency_us: 334, response_code: 204, uri: /stop/cid/2, method: POST, req_id: f249af23-ed04-4480-b732-3e37c5d68655, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
178312025-02-28T06:04:59.375ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
178322025-02-28T06:04:59.375ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
178332025-02-28T06:04:59.375ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
17834 Feb 28 06:04:59.375 INFO request completed, latency_us: 110, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5a2bac78-f902-4b6a-add9-09ed6bfb084e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
178352025-02-28T06:04:59.375ZINFOcrucible: client task is exiting = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
17836 [8830] Exited with: ExitStatus(unix_wait_status(9))
17837 [8830][2] reports Exit
17838 Feb 28 06:05:03.377 INFO request completed, latency_us: 314, response_code: 200, uri: /state/cid/2, method: GET, req_id: f0f67ab0-86d2-4253-b700-9d3711c09ec3, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
17839 001/300 Read block 2627 len 32768
17840 002/300 Write block 1290 len 8192 data: 4 5
17841 003/300 Write block 418 len 28672 data: 6 7 5 5 6 5 4
17842 004/300 Read block 3520 len 12288
17843 005/300 Write block 4513 len 8192 data: 4 5
17844 006/300 Write block 1894 len 8192 data: 6 6
17845 007/300 Read block 488 len 12288
17846 008/300 Write block 1355 len 4096 data: 4
17847 009/300 Write block 208 len 32768 data: 3 4 4 4 4 3 5 5
17848 010/300 Read block 2048 len 8192
17849 011/300 Write block 1656 len 24576 data: 5 4 4 4 6 4
17850 012/300 Read block 925 len 20480
17851 013/300 Read block 702 len 16384
17852 014/300 Read block 3215 len 20480
17853 015/300 Read block 4259 len 20480
17854 016/300 Write block 2542 len 20480 data: 3 3 3 3 3
17855 017/300 Read block 3065 len 20480
17856 018/300 Write block 997 len 36864 data: 3 3 3 4 5 5 4 3 3
17857 019/300 Read block 752 len 4096
17858 020/300 Write block 1794 len 40960 data: 3 4 4 4 4 4 4 4 4 2
17859 021/300 Write block 256 len 24576 data: 3 2 2 2 3 3
17860 022/300 Read block 953 len 20480
17861 023/300 Write block 3380 len 24576 data: 2 2 2 2 2 2
17862 024/300 Write block 4006 len 32768 data: 4 4 4 3 3 3 2 2
17863 025/300 Read block 4162 len 28672
17864 026/300 Read block 147 len 16384
17865 027/300 Read block 3865 len 24576
17866 028/300 Read block 3471 len 8192
17867 029/300 Read block 2930 len 36864
17868 030/300 Write block 4695 len 36864 data: 4 5 5 4 6 4 4 3 3
17869 031/300 Read block 271 len 24576
17870 032/300 Flush
17871 033/300 Write block 4649 len 32768 data: 5 4 4 6 6 6 5 3
17872 034/300 Read block 2826 len 40960
17873 035/300 Write block 2467 len 8192 data: 2 3
17874 036/300 Flush
17875 037/300 Read block 1197 len 20480
17876 038/300 Read block 460 len 12288
17877 039/300 Read block 447 len 20480
17878 040/300 Flush
17879 041/300 Write block 2132 len 28672 data: 4 4 4 3 2 4 3
17880 042/300 Flush
17881 043/300 Write block 141 len 4096 data: 2
17882 044/300 Read block 1001 len 20480
17883 045/300 Write block 1904 len 12288 data: 2 2 3
17884 046/300 Read block 262 len 20480
17885 047/300 Write block 3465 len 24576 data: 4 4 4 3 2 2
17886 048/300 Read block 3333 len 36864
17887 049/300 Write block 4810 len 8192 data: 3 3
17888 050/300 Read block 1338 len 16384
17889 051/300 Read block 2448 len 4096
17890 052/300 Read block 565 len 16384
17891 053/300 Read block 3217 len 16384
17892 054/300 Write block 438 len 24576 data: 6 6 6 5 7 7
17893 055/300 Write block 4883 len 20480 data: 6 5 5 5 7
17894 056/300 Write block 3096 len 24576 data: 4 4 5 5 5 6
17895 057/300 Flush
17896 058/300 Write block 4323 len 12288 data: 4 3 3
17897 059/300 Write block 3740 len 32768 data: 3 4 6 7 7 7 5 5
17898 060/300 Write block 4268 len 32768 data: 4 4 4 6 6 5 5 4
17899 061/300 Write block 812 len 16384 data: 5 5 4 3
17900 062/300 Flush
17901 063/300 Read block 3749 len 32768
17902 064/300 Write block 1819 len 4096 data: 3
17903 065/300 Read block 3255 len 20480
17904 066/300 Write block 4388 len 4096 data: 5
17905 067/300 Read block 1013 len 40960
17906 068/300 Read block 1028 len 16384
17907 069/300 Read block 1555 len 28672
17908 070/300 Read block 1308 len 4096
17909 071/300 Read block 1550 len 32768
17910 072/300 Write block 3142 len 20480 data: 3 5 5 4 3
17911 073/300 Read block 4786 len 40960
17912 074/300 Read block 2329 len 20480
17913 075/300 Flush
17914 076/300 Flush
17915 077/300 Read block 4192 len 20480
17916 078/300 Read block 947 len 8192
17917 079/300 Write block 2884 len 4096 data: 4
17918 080/300 Read block 3564 len 4096
17919 081/300 Flush
17920 082/300 Read block 1231 len 8192
17921 083/300 Read block 2412 len 32768
17922 084/300 Read block 2531 len 32768
17923 085/300 Write block 3184 len 20480 data: 3 3 3 3 3
17924 086/300 Write block 3522 len 8192 data: 5 5
17925 087/300 Read block 1361 len 24576
17926 088/300 Read block 4308 len 24576
17927 089/300 Read block 1804 len 36864
17928 090/300 Write block 1414 len 32768 data: 3 2 2 2 2 3 4 5
17929 091/300 Write block 1947 len 8192 data: 4 3
17930 092/300 Flush
17931 093/300 Read block 774 len 36864
17932 094/300 Read block 1682 len 8192
17933 095/300 Read block 3556 len 28672
17934 096/300 Read block 3711 len 36864
17935 097/300 Write block 3088 len 24576 data: 2 2 3 3 3 4
17936 098/300 Read block 633 len 8192
17937 099/300 Write block 1809 len 32768 data: 2 4 4 3 3 3 3 3
17938 100/300 Read block 4536 len 32768
17939 101/300 Read block 2752 len 36864
17940 102/300 Write block 709 len 40960 data: 6 6 5 4 4 3 3 5 5 5
17941 103/300 Read block 2992 len 24576
17942 104/300 Read block 2344 len 40960
17943 105/300 Read block 1249 len 32768
17944 106/300 Read block 4908 len 28672
17945 107/300 Read block 545 len 12288
17946 108/300 Read block 4513 len 28672
17947 109/300 Read block 3543 len 32768
17948 110/300 Write block 1365 len 28672 data: 4 3 3 4 2 2 3
17949 111/300 Read block 1555 len 4096
17950 112/300 Write block 3811 len 12288 data: 2 2 2
17951 113/300 Read block 900 len 12288
17952 114/300 Write block 3348 len 28672 data: 5 4 4 4 3 3 4
17953 115/300 Read block 3273 len 8192
17954 116/300 Write block 2675 len 28672 data: 7 6 4 5 5 5 4
17955 117/300 Read block 4243 len 28672
17956 118/300 Write block 467 len 8192 data: 3 4
17957 119/300 Read block 813 len 16384
17958 120/300 Write block 133 len 40960 data: 5 4 5 5 5 4 4 3 3 2
17959 121/300 Read block 1277 len 12288
17960 122/300 Write block 3843 len 4096 data: 5
17961 123/300 Write block 3056 len 24576 data: 4 2 3 3 3 5
17962 124/300 Write block 4756 len 4096 data: 6
17963 125/300 Read block 1155 len 4096
17964 126/300 Read block 4988 len 32768
17965 127/300 Flush
17966 128/300 Read block 3099 len 20480
17967 129/300 Flush
17968 130/300 Write block 3456 len 4096 data: 5
17969 131/300 Write block 4563 len 12288 data: 3 3 3
17970 132/300 Write block 3731 len 20480 data: 4 3 4 5 5
17971 133/300 Flush
17972 134/300 Read block 4059 len 4096
17973 135/300 Read block 4157 len 36864
17974 136/300 Write block 3610 len 8192 data: 4 4
17975 137/300 Read block 840 len 24576
17976 138/300 Read block 1733 len 24576
17977 139/300 Write block 2684 len 40960 data: 5 6 5 6 5 6 5 5 4 4
17978 140/300 Read block 1536 len 36864
17979 141/300 Flush
17980 142/300 Read block 1921 len 36864
17981 143/300 Write block 3611 len 20480 data: 5 4 3 3 3
17982 144/300 Read block 879 len 28672
17983 145/300 Read block 893 len 16384
17984 146/300 Write block 4591 len 24576 data: 4 5 5 5 7 7
17985 147/300 Read block 1994 len 8192
17986 148/300 Read block 3376 len 20480
17987 149/300 Flush
17988 150/300 Write block 2625 len 32768 data: 3 4 3 3 3 4 4 3
17989 151/300 Write block 2457 len 32768 data: 6 6 5 4 3 2 2 2
17990 152/300 Read block 4388 len 36864
17991 153/300 Read block 3948 len 20480
17992 154/300 Flush
17993 155/300 Read block 293 len 36864
17994 156/300 Read block 4860 len 28672
17995 157/300 Read block 3175 len 12288
17996 158/300 Flush
17997 159/300 Write block 1122 len 36864 data: 2 2 3 4 4 4 3 3 3
17998 160/300 Write block 2083 len 36864 data: 3 3 3 5 5 5 5 5 6
17999 161/300 Read block 228 len 36864
18000 162/300 Write block 424 len 32768 data: 5 3 3 3 4 4 5 5
18001 163/300 Write block 4757 len 12288 data: 6 6 6
18002 164/300 Flush
18003 165/300 Read block 4718 len 24576
18004 166/300 Write block 4913 len 4096 data: 4
18005 167/300 Write block 4709 len 32768 data: 2 2 3 3 3 3 3 3
18006 168/300 Write block 1071 len 20480 data: 3 3 3 3 3
18007 169/300 Read block 4238 len 24576
18008 170/300 Flush
18009 171/300 Read block 3694 len 40960
18010 172/300 Read block 642 len 24576
18011 173/300 Read block 3368 len 20480
18012 174/300 Write block 4834 len 40960 data: 4 5 5 6 7 6 6 5 6 5
18013 175/300 Write block 4295 len 28672 data: 4 4 4 4 2 2 2
18014 176/300 Write block 3630 len 16384 data: 3 3 3 4
18015 177/300 Read block 3960 len 4096
18016 178/300 Write block 921 len 20480 data: 3 3 3 3 3
18017 179/300 Flush
18018 180/300 Write block 1459 len 4096 data: 3
18019 181/300 Write block 1658 len 36864 data: 5 5 7 5 4 4 4 5 5
18020 182/300 Flush
18021 183/300 Flush
18022 184/300 Read block 2064 len 40960
18023 185/300 Write block 2291 len 4096 data: 4
18024 186/300 Read block 2944 len 20480
18025 187/300 Read block 2594 len 28672
18026 188/300 Read block 3513 len 16384
18027 189/300 Read block 1828 len 24576
18028 190/300 Write block 4003 len 40960 data: 2 3 4 5 5 5 4 4 4 3
18029 191/300 Read block 1457 len 36864
18030 192/300 Write block 4428 len 36864 data: 6 6 6 6 4 3 2 2 4
18031 193/300 Read block 155 len 16384
18032 194/300 Flush
18033 195/300 Write block 1222 len 24576 data: 4 4 4 3 2 3
18034 196/300 Write block 3647 len 12288 data: 2 4 4
18035 197/300 Read block 3041 len 4096
18036 198/300 Read block 1415 len 8192
18037 199/300 Write block 10 len 4096 data: 4
18038 200/300 Write block 4089 len 16384 data: 4 4 4 3
18039 201/300 Read block 2161 len 36864
18040 202/300 Write block 2811 len 24576 data: 5 4 5 6 6 6
18041 203/300 Write block 2522 len 20480 data: 6 7 8 7 5
18042 204/300 Read block 2078 len 12288
18043 205/300 Write block 2307 len 32768 data: 4 4 4 3 4 4 4 4
18044 206/300 Read block 2156 len 40960
18045 207/300 Read block 3485 len 20480
18046 208/300 Write block 2068 len 28672 data: 4 3 3 4 4 3 3
18047 209/300 Read block 551 len 28672
18048 210/300 Write block 1297 len 32768 data: 4 4 3 3 3 3 3 3
18049 211/300 Write block 715 len 4096 data: 4
18050 212/300 Write block 3938 len 12288 data: 5 6 7
18051 213/300 Write block 2684 len 20480 data: 6 7 6 7 6
18052 214/300 Read block 3745 len 8192
18053 215/300 Read block 2885 len 20480
18054 216/300 Write block 1862 len 12288 data: 2 2 3
18055 217/300 Write block 140 len 20480 data: 4 4 3 2 5
18056 218/300 Write block 4428 len 32768 data: 7 7 7 7 5 4 3 3
18057 219/300 Read block 1571 len 28672
18058 220/300 Write block 4355 len 4096 data: 6
18059 221/300 Read block 4447 len 28672
18060 222/300 Read block 2238 len 36864
18061 223/300 Read block 2934 len 8192
18062 224/300 Read block 1230 len 28672
18063 225/300 Flush
18064 226/300 Write block 949 len 28672 data: 4 4 4 3 3 3 3
18065 227/300 Read block 2000 len 12288
18066 228/300 Flush
18067 229/300 Read block 3009 len 24576
18068 230/300 Read block 1547 len 8192
18069 231/300 Write block 1377 len 28672 data: 4 4 4 4 4 4 5
18070 232/300 Write block 1577 len 28672 data: 4 6 6 6 6 6 6
18071 233/300 Read block 1472 len 24576
18072 234/300 Read block 1354 len 32768
18073 235/300 Write block 4252 len 16384 data: 2 2 2 3
18074 236/300 Write block 3179 len 28672 data: 2 3 3 3 3 4 4
18075 237/300 Write block 286 len 4096 data: 3
18076 238/300 Write block 653 len 4096 data: 3
18077 239/300 Write block 2913 len 12288 data: 3 3 3
18078 240/300 Write block 1425 len 32768 data: 5 4 4 2 2 2 2 3
18079 241/300 Flush
18080 242/300 Read block 4944 len 24576
18081 243/300 Write block 1912 len 40960 data: 2 2 2 2 2 2 2 2 2 2
18082 244/300 Read block 748 len 36864
18083 245/300 Flush
18084 246/300 Flush
18085 247/300 Read block 719 len 24576
18086 248/300 Read block 2560 len 20480
18087 249/300 Flush
18088 250/300 Write block 747 len 20480 data: 4 5 6 6 4
18089 251/300 Write block 4563 len 20480 data: 4 4 4 3 4
18090 252/300 Read block 3059 len 12288
18091 253/300 Read block 3907 len 24576
18092 254/300 Write block 676 len 16384 data: 6 4 6 7
18093 255/300 Flush
18094 256/300 Read block 1527 len 24576
18095 257/300 Read block 1431 len 36864
18096 258/300 Read block 3376 len 20480
18097 259/300 Read block 3868 len 16384
18098 260/300 Write block 597 len 20480 data: 2 3 4 4 3
18099 261/300 Flush
18100 262/300 Write block 2571 len 16384 data: 5 5 5 3
18101 263/300 Read block 1332 len 32768
18102 264/300 Read block 3808 len 16384
18103 265/300 Read block 10 len 12288
18104 266/300 Flush
18105 267/300 Read block 4052 len 36864
18106 268/300 Flush
18107 269/300 Read block 4204 len 40960
18108 270/300 Read block 2035 len 24576
18109 271/300 Read block 1853 len 32768
18110 272/300 Flush
18111 273/300 Read block 3836 len 8192
18112 274/300 Write block 3293 len 4096 data: 6
18113 275/300 Flush
18114 276/300 Read block 3562 len 24576
18115 277/300 Read block 1284 len 32768
18116 278/300 Read block 3024 len 24576
18117 279/300 Read block 3766 len 4096
18118 280/300 Write block 2748 len 12288 data: 3 3 4
18119 281/300 Read block 143 len 32768
18120 282/300 Write block 4605 len 32768 data: 3 3 3 2 3 3 3 3
18121 283/300 Write block 4823 len 32768 data: 4 4 4 4 4 7 7 9
18122 284/300 Read block 4443 len 24576
18123 285/300 Read block 567 len 20480
18124 286/300 Read block 1226 len 16384
18125 287/300 Read block 4191 len 20480
18126 288/300 Read block 2563 len 40960
18127 289/300 Read block 476 len 16384
18128 290/300 Flush
18129 291/300 Write block 4186 len 32768 data: 5 5 3 3 5 5 4 4
18130 292/300 Write block 1608 len 8192 data: 3 3
18131 293/300 Write block 3869 len 8192 data: 6 5
18132 294/300 Write block 4043 len 16384 data: 5 5 5 4
18133 295/300 Write block 2503 len 4096 data: 2
18134 296/300 Flush
18135 297/300 Flush
18136 298/300 Read block 744 len 32768
18137 299/300 Read block 3585 len 40960
18138 300/300 Write block 941 len 8192 data: 5 5
18139 301/300 Read block 2994 len 24576
18140 Main task has work to do, go find it
18141 got dsc Start(2)
18142 start 2
18143 Feb 28 06:05:03.619 INFO request completed, latency_us: 180, response_code: 204, uri: /start/cid/2, method: POST, req_id: dad4bdbe-1952-4efd-b269-7b029b015be1, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
18144 Replay: started 2, returned:Ok(())
18145 ----------------------------------------------------------------
18146 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18147 States: Active Active Offline
18148 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18149 5225 Acked Read 8 Done Done Sent false
18150 5226 Acked Write 2 Done Done Sent false
18151 5227 Acked Write 7 Done Done Sent false
18152 5228 Acked Read 3 Done Done Sent false
18153 5229 Acked Write 2 Done Done Sent false
18154 5230 Acked Write 2 Done Done Sent false
18155 5231 Acked Read 3 Done Done Sent false
18156 5232 Acked Write 1 Done Done Sent false
18157 5233 Acked Write 8 Done Done Sent false
18158 5234 Acked Read 2 Done Done Sent false
18159 5235 Acked Write 6 Done Done Sent false
18160 5236 Acked Read 5 Done Done Sent false
18161 5237 Acked Read 4 Done Done Sent false
18162 5238 Acked Read 5 Done Done Sent false
18163 5239 Acked Read 5 Done Done Sent false
18164 5240 Acked Write 5 Done Done Sent false
18165 5241 Acked Read 5 Done Done Sent false
18166 5242 Acked Write 9 Done Done Sent false
18167 5243 Acked Read 1 Done Done Sent false
18168 5244 Acked Write 10 Done Done Sent false
18169 5245 Acked Write 6 Done Done Sent false
18170 5246 Acked Read 5 Done Done Sent false
18171 5247 Acked Write 6 Done Done Sent false
18172 5248 Acked Write 8 Done Done Sent false
18173 5249 Acked Read 7 Done Done Sent false
18174 5250 Acked Read 4 Done Done Sent false
18175 5251 Acked Read 6 Done Done Sent false
18176 5252 Acked Read 2 Done Done Sent false
18177 5253 Acked Read 9 Done Done Sent false
18178 5254 Acked Write 9 Done Done Sent false
18179 5255 Acked Read 6 Done Done Sent false
18180 5256 Acked Flush 0 Done Done Sent false
18181 5257 Acked Write 8 Done Done Sent false
18182 5258 Acked Read 10 Done Done Sent false
18183 5259 Acked Write 2 Done Done Sent false
18184 5260 Acked Flush 0 Done Done Sent false
18185 5261 Acked Read 5 Done Done Sent false
18186 5262 Acked Read 3 Done Done Sent false
18187 5263 Acked Read 5 Done Done Sent false
18188 5264 Acked Flush 0 Done Done Sent false
18189 5265 Acked Write 7 Done Done Sent false
18190 5266 Acked Flush 0 Done Done Sent false
18191 5267 Acked Write 1 Done Done Sent false
18192 5268 Acked Read 5 Done Done Sent false
18193 5269 Acked Write 3 Done Done Sent false
18194 5270 Acked Read 5 Done Done Sent false
18195 5271 Acked Write 6 Done Done Sent false
18196 5272 Acked Read 9 Done Done Sent false
18197 5273 Acked Write 2 Done Done Sent false
18198 5274 Acked Read 4 Done Done Sent false
18199 5275 Acked Read 1 Done Done Sent false
18200 5276 Acked Read 4 Done Done Sent false
18201 5277 Acked Read 4 Done Done Sent false
18202 5278 Acked Write 6 Done Done Sent false
18203 5279 Acked Write 5 Done Done Sent false
18204 5280 Acked Write 6 Done Done Sent false
18205 5281 Acked Flush 0 Done Done Sent false
18206 5282 Acked Write 3 Done Done Sent false
18207 5283 Acked Write 8 Done Done Sent false
18208 5284 Acked Write 8 Done Done Sent false
18209 5285 Acked Write 4 Done Done Sent false
18210 5286 Acked Flush 0 Done Done Sent false
18211 5287 Acked Read 8 Done Done Sent false
18212 5288 Acked Write 1 Done Done Sent false
18213 5289 Acked Read 5 Done Done Sent false
18214 5290 Acked Write 1 Done Done Sent false
18215 5291 Acked Read 10 Done Done Sent false
18216 5292 Acked Read 4 Done Done Sent false
18217 5293 Acked Read 7 Done Done Sent false
18218 5294 Acked Read 1 Done Done Sent false
18219 5295 Acked Read 8 Done Done Sent false
18220 5296 Acked Write 5 Done Done Sent false
18221 5297 Acked Read 10 Done Done Sent false
18222 5298 Acked Read 5 Done Done Sent false
18223 5299 Acked Flush 0 Done Done Sent false
18224 5300 Acked Flush 0 Done Done Sent false
18225 5301 Acked Read 5 Done Done Sent false
18226 5302 Acked Read 2 Done Done Sent false
18227 5303 Acked Write 1 Done Done Sent false
18228 5304 Acked Read 1 Done Done Sent false
18229 5305 Acked Flush 0 Done Done Sent false
18230 5306 Acked Read 2 Done Done Sent false
18231 5307 Acked Read 8 Done Done Sent false
18232 5308 Acked Read 8 Done Done Sent false
18233 5309 Acked Write 5 Done Done Sent false
18234 5310 Acked Write 2 Done Done Sent false
18235 5311 Acked Read 6 Done Done Sent false
18236 5312 Acked Read 6 Done Done Sent false
18237 5313 Acked Read 9 Done Done Sent false
18238 5314 Acked Write 8 Done Done Sent false
18239 5315 Acked Write 2 Done Done Sent false
18240 5316 Acked Flush 0 Done Done Sent false
18241 5317 Acked Read 9 Done Done Sent false
18242 5318 Acked Read 2 Done Done Sent false
18243 5319 Acked Read 7 Done Done Sent false
18244 5320 Acked Read 9 Done Done Sent false
18245 5321 Acked Write 6 Done Done Sent false
18246 5322 Acked Read 2 Done Done Sent false
18247 5323 Acked Write 8 Done Done Sent false
18248 5324 Acked Read 8 Done Done Sent false
18249 5325 Acked Read 9 Done Done Sent false
18250 5326 Acked Write 10 Done Done Sent false
18251 5327 Acked Read 6 Done Done Sent false
18252 5328 Acked Read 10 Done Done Sent false
18253 5329 Acked Read 8 Done Done Sent false
18254 5330 Acked Read 7 Done Done Sent false
18255 5331 Acked Read 3 Done Done Sent false
18256 5332 Acked Read 7 Done Done Sent false
18257 5333 Acked Read 8 Done Done Sent false
18258 5334 Acked Write 7 Done Done Sent false
18259 5335 Acked Read 1 Done Done Sent false
18260 5336 Acked Write 3 Done Done Sent false
18261 5337 Acked Read 3 Done Done Sent false
18262 5338 Acked Write 7 Done Done Sent false
18263 5339 Acked Read 2 Done Done Sent false
18264 5340 Acked Write 7 Done Done Sent false
18265 5341 Acked Read 7 Done Done Sent false
18266 5342 Acked Write 2 Done Done Sent false
18267 5343 Acked Read 4 Done Done Sent false
18268 5344 Acked Write 10 Done Done Sent false
18269 5345 Acked Read 3 Done Done Sent false
18270 5346 Acked Write 1 Done Done Sent false
18271 5347 Acked Write 6 Done Done Sent false
18272 5348 Acked Write 1 Done Done Sent false
18273 5349 Acked Read 1 Done Done Sent false
18274 5350 Acked Read 8 Done Done Sent false
18275 5351 Acked Flush 0 Done Done Sent false
18276 5352 Acked Read 5 Done Done Sent false
18277 5353 Acked Flush 0 Done Done Sent false
18278 5354 Acked Write 1 Done Done Sent false
18279 5355 Acked Write 3 Done Done Sent false
18280 5356 Acked Write 5 Done Done Sent false
18281 5357 Acked Flush 0 Done Done Sent false
18282 5358 Acked Read 1 Done Done Sent false
18283 5359 Acked Read 9 Done Done Sent false
18284 5360 Acked Write 2 Done Done Sent false
18285 5361 Acked Read 6 Done Done Sent false
18286 5362 Acked Read 6 Done Done Sent false
18287 5363 Acked Write 10 Done Done Sent false
18288 5364 Acked Read 9 Done Done Sent false
18289 5365 Acked Flush 0 Done Done Sent false
18290 5366 Acked Read 9 Done Done Sent false
18291 5367 Acked Write 5 Done Done Sent false
18292 5368 Acked Read 7 Done Done Sent false
18293 5369 Acked Read 4 Done Done Sent false
18294 5370 Acked Write 6 Done Done Sent false
18295 5371 Acked Read 2 Done Done Sent false
18296 5372 Acked Read 5 Done Done Sent false
18297 5373 Acked Flush 0 Done Done Sent false
18298 5374 Acked Write 8 Done Done Sent false
18299 5375 Acked Write 8 Done Done Sent false
18300 5376 Acked Read 9 Done Done Sent false
18301 5377 Acked Read 5 Done Done Sent false
18302 5378 Acked Flush 0 Done Done Sent false
18303 5379 Acked Read 9 Done Done Sent false
18304 5380 Acked Read 7 Done Done Sent false
18305 5381 Acked Read 3 Done Done Sent false
18306 5382 Acked Flush 0 Done Done Sent false
18307 5383 Acked Write 9 Done Done Sent false
18308 5384 Acked Write 9 Done Done Sent false
18309 5385 Acked Read 9 Done Done Sent false
18310 5386 Acked Write 8 Done Done Sent false
18311 5387 Acked Write 3 Done Done Sent false
18312 5388 Acked Flush 0 Done Done Sent false
18313 5389 Acked Read 6 Done Done Sent false
18314 5390 Acked Write 1 Done Done Sent false
18315 5391 Acked Write 8 Done Done Sent false
18316 5392 Acked Write 5 Done Done Sent false
18317 5393 Acked Read 6 Done Done Sent false
18318 5394 Acked Flush 0 Done Done Sent false
18319 5395 Acked Read 10 Done Done Sent false
18320 5396 Acked Read 6 Done Done Sent false
18321 5397 Acked Read 5 Done Done Sent false
18322 5398 Acked Write 10 Done Done Sent false
18323 5399 Acked Write 7 Done Done Sent false
18324 5400 Acked Write 4 Done Done Sent false
18325 5401 Acked Read 1 Done Done Sent false
18326 5402 Acked Write 5 Done Done Sent false
18327 5403 Acked Flush 0 Done Done Sent false
18328 5404 Acked Write 1 Done Done Sent false
18329 5405 Acked Write 9 Done Done Sent false
18330 5406 Acked Flush 0 Done Done Sent false
18331 5407 Acked Flush 0 Done Done Sent false
18332 5408 Acked Read 10 Done Done Sent false
18333 5409 Acked Write 1 Done Done Sent false
18334 5410 Acked Read 5 Done Done Sent false
18335 5411 Acked Read 7 Done Done Sent false
18336 5412 Acked Read 4 Done Done Sent false
18337 5413 Acked Read 6 Done Done Sent false
18338 5414 Acked Write 10 Done Done Sent false
18339 5415 Acked Read 9 Done Done Sent false
18340 5416 Acked Write 9 Done Done Sent false
18341 5417 Acked Read 4 Done Done Sent false
18342 5418 Acked Flush 0 Done Done Sent false
18343 5419 Acked Write 6 Done Done Sent false
18344 5420 Acked Write 3 Done Done Sent false
18345 5421 Acked Read 1 Done Done Sent false
18346 5422 Acked Read 2 Done Done Sent false
18347 5423 Acked Write 1 Done Done Sent false
18348 5424 Acked Write 4 Done Done Sent false
18349 5425 Acked Read 9 Done Done Sent false
18350 5426 Acked Write 6 Done Done Sent false
18351 5427 Acked Write 5 Done Done Sent false
18352 5428 Acked Read 3 Done Done Sent false
18353 5429 Acked Write 8 Done Done Sent false
18354 5430 Acked Read 10 Done Done Sent false
18355 5431 Acked Read 5 Done Done Sent false
18356 5432 Acked Write 7 Done Done Sent false
18357 5433 Acked Read 7 Done Done Sent false
18358 5434 Acked Write 8 Done Done Sent false
18359 5435 Acked Write 1 Done Done Sent false
18360 5436 Acked Write 3 Done Done Sent false
18361 5437 Acked Write 5 Done Done Sent false
18362 5438 Acked Read 2 Done Done Sent false
18363 5439 Acked Read 5 Done Done Sent false
18364 5440 Acked Write 3 Done Done Sent false
18365 5441 Acked Write 5 Done Done Sent false
18366 5442 Acked Write 8 Done Done Sent false
18367 5443 Acked Read 7 Done Done Sent false
18368 5444 Acked Write 1 Done Done Sent false
18369 5445 Acked Read 7 Done Done Sent false
18370 5446 Acked Read 9 Done Done Sent false
18371 5447 Acked Read 2 Done Done Sent false
18372 5448 Acked Read 7 Done Done Sent false
18373 5449 Acked Flush 0 Done Done Sent false
18374 5450 Acked Write 7 Done Done Sent false
18375 5451 Acked Read 3 Done Done Sent false
18376 5452 Acked Flush 0 Done Done Sent false
18377 5453 Acked Read 6 Done Done Sent false
18378 5454 Acked Read 2 Done Done Sent false
18379 5455 Acked Write 7 Done Done Sent false
18380 5456 Acked Write 7 Done Done Sent false
18381 5457 Acked Read 6 Done Done Sent false
18382 5458 Acked Read 8 Done Done Sent false
18383 5459 Acked Write 4 Done Done Sent false
18384 5460 Acked Write 7 Done Done Sent false
18385 5461 Acked Write 1 Done Done Sent false
18386 5462 Acked Write 1 Done Done Sent false
18387 5463 Acked Write 3 Done Done Sent false
18388 5464 Acked Write 8 Done Done Sent false
18389 5465 Acked Flush 0 Done Done Sent false
18390 5466 Acked Read 6 Done Done Sent false
18391 5467 Acked Write 10 Done Done Sent false
18392 5468 Acked Read 9 Done Done Sent false
18393 5469 Acked Flush 0 Done Done Sent false
18394 5470 Acked Flush 0 Done Done Sent false
18395 5471 Acked Read 6 Done Done Sent false
18396 5472 Acked Read 5 Done Done Sent false
18397 5473 Acked Flush 0 Done Done Sent false
18398 5474 Acked Write 5 Done Done Sent false
18399 5475 Acked Write 5 Done Done Sent false
18400 5476 Acked Read 3 Done Done Sent false
18401 5477 Acked Read 6 Done Done Sent false
18402 5478 Acked Write 4 Done Done Sent false
18403 5479 Acked Flush 0 Done Done Sent false
18404 5480 Acked Read 6 Done Done Sent false
18405 5481 Acked Read 9 Done Done Sent false
18406 5482 Acked Read 5 Done Done Sent false
18407 5483 Acked Read 4 Done Done Sent false
18408 5484 Acked Write 5 Done Done Sent false
18409 5485 Acked Flush 0 Done Done Sent false
18410 5486 Acked Write 4 Done Done Sent false
18411 5487 Acked Read 8 Done Done Sent false
18412 5488 Acked Read 4 Done Done Sent false
18413 5489 Acked Read 3 Done Done Sent false
18414 5490 Acked Flush 0 Done Done Sent false
18415 5491 Acked Read 9 Done Done Sent false
18416 5492 Acked Flush 0 Done Done Sent false
18417 5493 Acked Read 10 Done Done Sent false
18418 5494 Acked Read 6 Done Done Sent false
18419 5495 Acked Read 8 Done Done Sent false
18420 5496 Acked Flush 0 Done Done Sent false
18421 5497 Acked Read 2 Done Done Sent false
18422 5498 Acked Write 1 Done Done Sent false
18423 5499 Acked Flush 0 Done Done Sent false
18424 5500 Acked Read 6 Done Done Sent false
18425 5501 Acked Read 8 Done Done Sent false
18426 5502 Acked Read 6 Done Done Sent false
18427 5503 Acked Read 1 Done Done Sent false
18428 5504 Acked Write 3 Done Done Sent false
18429 5505 Acked Read 8 Done Done Sent false
18430 5506 Acked Write 8 Done Done Sent false
18431 5507 Acked Write 8 Done Done Sent false
18432 5508 Acked Read 6 Done Done Sent false
18433 5509 Acked Read 5 Done Done Sent false
18434 5510 Acked Read 4 Done Done Sent false
18435 5511 Acked Read 5 Done Done Sent false
18436 5512 Acked Read 10 Done Done Sent false
18437 5513 Acked Read 4 Done Done Sent false
18438 5514 Acked Flush 0 Done Done Sent false
18439 5515 Acked Write 8 Done Done Sent false
18440 5516 Acked Write 2 Done Done Sent false
18441 5517 Acked Write 2 Done Done Sent false
18442 5518 Acked Write 4 Done Done Sent false
18443 5519 Acked Write 1 Done Done Sent false
18444 5520 Acked Flush 0 Done Done Sent false
18445 5521 Acked Flush 0 Done Done Sent false
18446 5522 Acked Read 8 Done Done Sent false
18447 5523 Acked Read 10 Done Done Sent false
18448 5524 Acked Write 2 Done Done Sent false
18449 5525 Acked Read 6 Done Done Sent false
18450 STATES DS:0 DS:1 DS:2 TOTAL
18451 Sent 0 0 301 301
18452 Done 301 301 0 602
18453 Skipped 0 0 0 0
18454 Error 0 0 0 0
18455 Last Flush: Some(JobId(5521)) Some(JobId(5521)) Some(JobId(5224))
18456 Downstairs last five completed: 5224 5223 5222 5221 5220
18457 Upstairs last five completed: 5525 5524 5523 5522 5521
18458 CLIENT: Up:0 ds:301 act:2
184592025-02-28T06:05:04.121ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
184602025-02-28T06:05:04.742ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18461 [8830] got start action
18462 [8830] I am going to restart
18463 Starting downstairs at port 8830
18464 Make output file at "/tmp/dsc/downstairs-8830.txt"
18465 [8830][2] reports Starting
18466 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1226)
18467 [8830][2] reports Running
184682025-02-28T06:05:05.743ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
184692025-02-28T06:05:06.896ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18470 ----------------------------------------------------------------
18471 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18472 States: Active Active Offline
18473 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18474 5225 Acked Read 8 Done Done Sent false
18475 5226 Acked Write 2 Done Done Sent false
18476 5227 Acked Write 7 Done Done Sent false
18477 5228 Acked Read 3 Done Done Sent false
18478 5229 Acked Write 2 Done Done Sent false
18479 5230 Acked Write 2 Done Done Sent false
18480 5231 Acked Read 3 Done Done Sent false
18481 5232 Acked Write 1 Done Done Sent false
18482 5233 Acked Write 8 Done Done Sent false
18483 5234 Acked Read 2 Done Done Sent false
18484 5235 Acked Write 6 Done Done Sent false
18485 5236 Acked Read 5 Done Done Sent false
18486 5237 Acked Read 4 Done Done Sent false
18487 5238 Acked Read 5 Done Done Sent false
18488 5239 Acked Read 5 Done Done Sent false
18489 5240 Acked Write 5 Done Done Sent false
18490 5241 Acked Read 5 Done Done Sent false
18491 5242 Acked Write 9 Done Done Sent false
18492 5243 Acked Read 1 Done Done Sent false
18493 5244 Acked Write 10 Done Done Sent false
18494 5245 Acked Write 6 Done Done Sent false
18495 5246 Acked Read 5 Done Done Sent false
18496 5247 Acked Write 6 Done Done Sent false
18497 5248 Acked Write 8 Done Done Sent false
18498 5249 Acked Read 7 Done Done Sent false
18499 5250 Acked Read 4 Done Done Sent false
18500 5251 Acked Read 6 Done Done Sent false
18501 5252 Acked Read 2 Done Done Sent false
18502 5253 Acked Read 9 Done Done Sent false
18503 5254 Acked Write 9 Done Done Sent false
18504 5255 Acked Read 6 Done Done Sent false
18505 5256 Acked Flush 0 Done Done Sent false
18506 5257 Acked Write 8 Done Done Sent false
18507 5258 Acked Read 10 Done Done Sent false
18508 5259 Acked Write 2 Done Done Sent false
18509 5260 Acked Flush 0 Done Done Sent false
18510 5261 Acked Read 5 Done Done Sent false
18511 5262 Acked Read 3 Done Done Sent false
18512 5263 Acked Read 5 Done Done Sent false
18513 5264 Acked Flush 0 Done Done Sent false
18514 5265 Acked Write 7 Done Done Sent false
18515 5266 Acked Flush 0 Done Done Sent false
18516 5267 Acked Write 1 Done Done Sent false
18517 5268 Acked Read 5 Done Done Sent false
18518 5269 Acked Write 3 Done Done Sent false
18519 5270 Acked Read 5 Done Done Sent false
18520 5271 Acked Write 6 Done Done Sent false
18521 5272 Acked Read 9 Done Done Sent false
18522 5273 Acked Write 2 Done Done Sent false
18523 5274 Acked Read 4 Done Done Sent false
18524 5275 Acked Read 1 Done Done Sent false
18525 5276 Acked Read 4 Done Done Sent false
18526 5277 Acked Read 4 Done Done Sent false
18527 5278 Acked Write 6 Done Done Sent false
18528 5279 Acked Write 5 Done Done Sent false
18529 5280 Acked Write 6 Done Done Sent false
18530 5281 Acked Flush 0 Done Done Sent false
18531 5282 Acked Write 3 Done Done Sent false
18532 5283 Acked Write 8 Done Done Sent false
18533 5284 Acked Write 8 Done Done Sent false
18534 5285 Acked Write 4 Done Done Sent false
18535 5286 Acked Flush 0 Done Done Sent false
18536 5287 Acked Read 8 Done Done Sent false
18537 5288 Acked Write 1 Done Done Sent false
18538 5289 Acked Read 5 Done Done Sent false
18539 5290 Acked Write 1 Done Done Sent false
18540 5291 Acked Read 10 Done Done Sent false
18541 5292 Acked Read 4 Done Done Sent false
18542 5293 Acked Read 7 Done Done Sent false
18543 5294 Acked Read 1 Done Done Sent false
18544 5295 Acked Read 8 Done Done Sent false
18545 5296 Acked Write 5 Done Done Sent false
18546 5297 Acked Read 10 Done Done Sent false
18547 5298 Acked Read 5 Done Done Sent false
18548 5299 Acked Flush 0 Done Done Sent false
18549 5300 Acked Flush 0 Done Done Sent false
18550 5301 Acked Read 5 Done Done Sent false
18551 5302 Acked Read 2 Done Done Sent false
18552 5303 Acked Write 1 Done Done Sent false
18553 5304 Acked Read 1 Done Done Sent false
18554 5305 Acked Flush 0 Done Done Sent false
18555 5306 Acked Read 2 Done Done Sent false
18556 5307 Acked Read 8 Done Done Sent false
18557 5308 Acked Read 8 Done Done Sent false
18558 5309 Acked Write 5 Done Done Sent false
18559 5310 Acked Write 2 Done Done Sent false
18560 5311 Acked Read 6 Done Done Sent false
18561 5312 Acked Read 6 Done Done Sent false
18562 5313 Acked Read 9 Done Done Sent false
18563 5314 Acked Write 8 Done Done Sent false
18564 5315 Acked Write 2 Done Done Sent false
18565 5316 Acked Flush 0 Done Done Sent false
18566 5317 Acked Read 9 Done Done Sent false
18567 5318 Acked Read 2 Done Done Sent false
18568 5319 Acked Read 7 Done Done Sent false
18569 5320 Acked Read 9 Done Done Sent false
18570 5321 Acked Write 6 Done Done Sent false
18571 5322 Acked Read 2 Done Done Sent false
18572 5323 Acked Write 8 Done Done Sent false
18573 5324 Acked Read 8 Done Done Sent false
18574 5325 Acked Read 9 Done Done Sent false
18575 5326 Acked Write 10 Done Done Sent false
18576 5327 Acked Read 6 Done Done Sent false
18577 5328 Acked Read 10 Done Done Sent false
18578 5329 Acked Read 8 Done Done Sent false
18579 5330 Acked Read 7 Done Done Sent false
18580 5331 Acked Read 3 Done Done Sent false
18581 5332 Acked Read 7 Done Done Sent false
18582 5333 Acked Read 8 Done Done Sent false
18583 5334 Acked Write 7 Done Done Sent false
18584 5335 Acked Read 1 Done Done Sent false
18585 5336 Acked Write 3 Done Done Sent false
18586 5337 Acked Read 3 Done Done Sent false
18587 5338 Acked Write 7 Done Done Sent false
18588 5339 Acked Read 2 Done Done Sent false
18589 5340 Acked Write 7 Done Done Sent false
18590 5341 Acked Read 7 Done Done Sent false
18591 5342 Acked Write 2 Done Done Sent false
18592 5343 Acked Read 4 Done Done Sent false
18593 5344 Acked Write 10 Done Done Sent false
18594 5345 Acked Read 3 Done Done Sent false
18595 5346 Acked Write 1 Done Done Sent false
18596 5347 Acked Write 6 Done Done Sent false
18597 5348 Acked Write 1 Done Done Sent false
18598 5349 Acked Read 1 Done Done Sent false
18599 5350 Acked Read 8 Done Done Sent false
18600 5351 Acked Flush 0 Done Done Sent false
18601 5352 Acked Read 5 Done Done Sent false
18602 5353 Acked Flush 0 Done Done Sent false
18603 5354 Acked Write 1 Done Done Sent false
18604 5355 Acked Write 3 Done Done Sent false
18605 5356 Acked Write 5 Done Done Sent false
18606 5357 Acked Flush 0 Done Done Sent false
18607 5358 Acked Read 1 Done Done Sent false
18608 5359 Acked Read 9 Done Done Sent false
18609 5360 Acked Write 2 Done Done Sent false
18610 5361 Acked Read 6 Done Done Sent false
18611 5362 Acked Read 6 Done Done Sent false
18612 5363 Acked Write 10 Done Done Sent false
18613 5364 Acked Read 9 Done Done Sent false
18614 5365 Acked Flush 0 Done Done Sent false
18615 5366 Acked Read 9 Done Done Sent false
18616 5367 Acked Write 5 Done Done Sent false
18617 5368 Acked Read 7 Done Done Sent false
18618 5369 Acked Read 4 Done Done Sent false
18619 5370 Acked Write 6 Done Done Sent false
18620 5371 Acked Read 2 Done Done Sent false
18621 5372 Acked Read 5 Done Done Sent false
18622 5373 Acked Flush 0 Done Done Sent false
18623 5374 Acked Write 8 Done Done Sent false
18624 5375 Acked Write 8 Done Done Sent false
18625 5376 Acked Read 9 Done Done Sent false
18626 5377 Acked Read 5 Done Done Sent false
18627 5378 Acked Flush 0 Done Done Sent false
18628 5379 Acked Read 9 Done Done Sent false
18629 5380 Acked Read 7 Done Done Sent false
18630 5381 Acked Read 3 Done Done Sent false
18631 5382 Acked Flush 0 Done Done Sent false
18632 5383 Acked Write 9 Done Done Sent false
18633 5384 Acked Write 9 Done Done Sent false
18634 5385 Acked Read 9 Done Done Sent false
18635 5386 Acked Write 8 Done Done Sent false
18636 5387 Acked Write 3 Done Done Sent false
18637 5388 Acked Flush 0 Done Done Sent false
18638 5389 Acked Read 6 Done Done Sent false
18639 5390 Acked Write 1 Done Done Sent false
18640 5391 Acked Write 8 Done Done Sent false
18641 5392 Acked Write 5 Done Done Sent false
18642 5393 Acked Read 6 Done Done Sent false
18643 5394 Acked Flush 0 Done Done Sent false
18644 5395 Acked Read 10 Done Done Sent false
18645 5396 Acked Read 6 Done Done Sent false
18646 5397 Acked Read 5 Done Done Sent false
18647 5398 Acked Write 10 Done Done Sent false
18648 5399 Acked Write 7 Done Done Sent false
18649 5400 Acked Write 4 Done Done Sent false
18650 5401 Acked Read 1 Done Done Sent false
18651 5402 Acked Write 5 Done Done Sent false
18652 5403 Acked Flush 0 Done Done Sent false
18653 5404 Acked Write 1 Done Done Sent false
18654 5405 Acked Write 9 Done Done Sent false
18655 5406 Acked Flush 0 Done Done Sent false
18656 5407 Acked Flush 0 Done Done Sent false
18657 5408 Acked Read 10 Done Done Sent false
18658 5409 Acked Write 1 Done Done Sent false
18659 5410 Acked Read 5 Done Done Sent false
18660 5411 Acked Read 7 Done Done Sent false
18661 5412 Acked Read 4 Done Done Sent false
18662 5413 Acked Read 6 Done Done Sent false
18663 5414 Acked Write 10 Done Done Sent false
18664 5415 Acked Read 9 Done Done Sent false
18665 5416 Acked Write 9 Done Done Sent false
18666 5417 Acked Read 4 Done Done Sent false
18667 5418 Acked Flush 0 Done Done Sent false
18668 5419 Acked Write 6 Done Done Sent false
18669 5420 Acked Write 3 Done Done Sent false
18670 5421 Acked Read 1 Done Done Sent false
18671 5422 Acked Read 2 Done Done Sent false
18672 5423 Acked Write 1 Done Done Sent false
18673 5424 Acked Write 4 Done Done Sent false
18674 5425 Acked Read 9 Done Done Sent false
18675 5426 Acked Write 6 Done Done Sent false
18676 5427 Acked Write 5 Done Done Sent false
18677 5428 Acked Read 3 Done Done Sent false
18678 5429 Acked Write 8 Done Done Sent false
18679 5430 Acked Read 10 Done Done Sent false
18680 5431 Acked Read 5 Done Done Sent false
18681 5432 Acked Write 7 Done Done Sent false
18682 5433 Acked Read 7 Done Done Sent false
18683 5434 Acked Write 8 Done Done Sent false
18684 5435 Acked Write 1 Done Done Sent false
18685 5436 Acked Write 3 Done Done Sent false
18686 5437 Acked Write 5 Done Done Sent false
18687 5438 Acked Read 2 Done Done Sent false
18688 5439 Acked Read 5 Done Done Sent false
18689 5440 Acked Write 3 Done Done Sent false
18690 5441 Acked Write 5 Done Done Sent false
18691 5442 Acked Write 8 Done Done Sent false
18692 5443 Acked Read 7 Done Done Sent false
18693 5444 Acked Write 1 Done Done Sent false
18694 5445 Acked Read 7 Done Done Sent false
18695 5446 Acked Read 9 Done Done Sent false
18696 5447 Acked Read 2 Done Done Sent false
18697 5448 Acked Read 7 Done Done Sent false
18698 5449 Acked Flush 0 Done Done Sent false
18699 5450 Acked Write 7 Done Done Sent false
18700 5451 Acked Read 3 Done Done Sent false
18701 5452 Acked Flush 0 Done Done Sent false
18702 5453 Acked Read 6 Done Done Sent false
18703 5454 Acked Read 2 Done Done Sent false
18704 5455 Acked Write 7 Done Done Sent false
18705 5456 Acked Write 7 Done Done Sent false
18706 5457 Acked Read 6 Done Done Sent false
18707 5458 Acked Read 8 Done Done Sent false
18708 5459 Acked Write 4 Done Done Sent false
18709 5460 Acked Write 7 Done Done Sent false
18710 5461 Acked Write 1 Done Done Sent false
18711 5462 Acked Write 1 Done Done Sent false
18712 5463 Acked Write 3 Done Done Sent false
18713 5464 Acked Write 8 Done Done Sent false
18714 5465 Acked Flush 0 Done Done Sent false
18715 5466 Acked Read 6 Done Done Sent false
18716 5467 Acked Write 10 Done Done Sent false
18717 5468 Acked Read 9 Done Done Sent false
18718 5469 Acked Flush 0 Done Done Sent false
18719 5470 Acked Flush 0 Done Done Sent false
18720 5471 Acked Read 6 Done Done Sent false
18721 5472 Acked Read 5 Done Done Sent false
18722 5473 Acked Flush 0 Done Done Sent false
18723 5474 Acked Write 5 Done Done Sent false
18724 5475 Acked Write 5 Done Done Sent false
18725 5476 Acked Read 3 Done Done Sent false
18726 5477 Acked Read 6 Done Done Sent false
18727 5478 Acked Write 4 Done Done Sent false
18728 5479 Acked Flush 0 Done Done Sent false
18729 5480 Acked Read 6 Done Done Sent false
18730 5481 Acked Read 9 Done Done Sent false
18731 5482 Acked Read 5 Done Done Sent false
18732 5483 Acked Read 4 Done Done Sent false
18733 5484 Acked Write 5 Done Done Sent false
18734 5485 Acked Flush 0 Done Done Sent false
18735 5486 Acked Write 4 Done Done Sent false
18736 5487 Acked Read 8 Done Done Sent false
18737 5488 Acked Read 4 Done Done Sent false
18738 5489 Acked Read 3 Done Done Sent false
18739 5490 Acked Flush 0 Done Done Sent false
18740 5491 Acked Read 9 Done Done Sent false
18741 5492 Acked Flush 0 Done Done Sent false
18742 5493 Acked Read 10 Done Done Sent false
18743 5494 Acked Read 6 Done Done Sent false
18744 5495 Acked Read 8 Done Done Sent false
18745 5496 Acked Flush 0 Done Done Sent false
18746 5497 Acked Read 2 Done Done Sent false
18747 5498 Acked Write 1 Done Done Sent false
18748 5499 Acked Flush 0 Done Done Sent false
18749 5500 Acked Read 6 Done Done Sent false
18750 5501 Acked Read 8 Done Done Sent false
18751 5502 Acked Read 6 Done Done Sent false
18752 5503 Acked Read 1 Done Done Sent false
18753 5504 Acked Write 3 Done Done Sent false
18754 5505 Acked Read 8 Done Done Sent false
18755 5506 Acked Write 8 Done Done Sent false
18756 5507 Acked Write 8 Done Done Sent false
18757 5508 Acked Read 6 Done Done Sent false
18758 5509 Acked Read 5 Done Done Sent false
18759 5510 Acked Read 4 Done Done Sent false
18760 5511 Acked Read 5 Done Done Sent false
18761 5512 Acked Read 10 Done Done Sent false
18762 5513 Acked Read 4 Done Done Sent false
18763 5514 Acked Flush 0 Done Done Sent false
18764 5515 Acked Write 8 Done Done Sent false
18765 5516 Acked Write 2 Done Done Sent false
18766 5517 Acked Write 2 Done Done Sent false
18767 5518 Acked Write 4 Done Done Sent false
18768 5519 Acked Write 1 Done Done Sent false
18769 5520 Acked Flush 0 Done Done Sent false
18770 5521 Acked Flush 0 Done Done Sent false
18771 5522 Acked Read 8 Done Done Sent false
18772 5523 Acked Read 10 Done Done Sent false
18773 5524 Acked Write 2 Done Done Sent false
18774 5525 Acked Read 6 Done Done Sent false
18775 STATES DS:0 DS:1 DS:2 TOTAL
18776 Sent 0 0 301 301
18777 Done 301 301 0 602
18778 Skipped 0 0 0 0
18779 Error 0 0 0 0
18780 Last Flush: Some(JobId(5521)) Some(JobId(5521)) Some(JobId(5224))
18781 Downstairs last five completed: 5224 5223 5222 5221 5220
18782 Upstairs last five completed: 5525 5524 5523 5522 5521
18783 CLIENT: Up:0 ds:301 act:2
187842025-02-28T06:05:08.748ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
187852025-02-28T06:05:09.376ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
187862025-02-28T06:05:09.376ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48058) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
187872025-02-28T06:05:09.377ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
187882025-02-28T06:05:09.377ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
187892025-02-28T06:05:09.378ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
187902025-02-28T06:05:09.378ZINFOcrucible: send last flush ID to this DS: Some(JobId(5224)) = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
187912025-02-28T06:05:09.378ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
187922025-02-28T06:05:09.378ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(5224)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
187932025-02-28T06:05:09.378ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(5224)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18794 ----------------------------------------------------------------
18795 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18796 Downstairs last five completed: 5526 5525 5524 5523 5522
18797 Upstairs last five completed: 5526 5525 5524 5523 5522
18798 CLIENT: Up:0 ds:0 act:3
18799 Replay: All jobs finished
18800 Main task has work to do, go find it
18801 got dsc Stop(1)
18802 stop 1
18803 Feb 28 06:05:11.624 INFO request completed, latency_us: 376, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 1bc2441d-e1b1-416c-9d10-f7a0acbf9204[8820] Got stop action so:false kr:false
18804 , remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
188052025-02-28T06:05:11.625ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
188062025-02-28T06:05:11.625ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
188072025-02-28T06:05:11.625ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
188082025-02-28T06:05:11.625ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
18809 Feb 28 06:05:11.625 INFO request completed, latency_us: 167, response_code: 200, uri: /state/cid/1, method: GET, req_id: df4f2adf-9855-45d6-91cf-77908657d784, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
18810 [8820] Exited with: ExitStatus(unix_wait_status(9))
18811 [8820][1] reports Exit
18812 Feb 28 06:05:15.627 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/1, method: GET, req_id: 459c192d-0563-446d-a0f4-645b3f45b2fd, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
18813 001/300 Read block 1408 len 40960
18814 002/300 Read block 2016 len 36864
18815 003/300 Read block 3268 len 36864
18816 004/300 Write block 2843 len 40960 data: 5 5 4 5 5 4 4 4 4 5
18817 005/300 Read block 3798 len 24576
18818 006/300 Read block 1515 len 16384
18819 007/300 Read block 4250 len 36864
18820 008/300 Read block 3195 len 4096
18821 009/300 Read block 4669 len 40960
18822 010/300 Write block 899 len 40960 data: 4 4 4 5 3 2 3 3 3 4
18823 011/300 Read block 638 len 8192
18824 012/300 Read block 4820 len 24576
18825 013/300 Write block 3492 len 36864 data: 3 3 2 2 2 4 4 4 3
18826 014/300 Read block 1249 len 28672
18827 015/300 Read block 4124 len 40960
18828 016/300 Read block 883 len 40960
18829 017/300 Read block 3212 len 8192
18830 018/300 Read block 1276 len 40960
18831 019/300 Read block 4830 len 20480
18832 020/300 Flush
18833 021/300 Read block 2764 len 24576
18834 022/300 Flush
18835 023/300 Flush
18836 024/300 Read block 2568 len 32768
18837 025/300 Read block 4900 len 40960
18838 026/300 Read block 4391 len 12288
18839 027/300 Write block 4770 len 4096 data: 5
18840 028/300 Write block 4246 len 40960 data: 2 2 2 3 3 3 3 3 3 4
18841 029/300 Read block 1068 len 36864
18842 030/300 Read block 2665 len 16384
18843 031/300 Read block 2953 len 20480
18844 032/300 Read block 1277 len 16384
18845 033/300 Write block 1065 len 32768 data: 3 4 3 3 4 4 4 4
18846 034/300 Write block 4440 len 20480 data: 3 4 5 5 5
18847 035/300 Read block 3705 len 12288
18848 036/300 Write block 3017 len 32768 data: 3 3 3 3 3 3 3 3
18849 037/300 Write block 558 len 32768 data: 3 3 3 3 3 3 3 3
18850 038/300 Read block 4905 len 16384
18851 039/300 Read block 937 len 28672
18852 040/300 Read block 1510 len 36864
18853 041/300 Write block 3881 len 40960 data: 4 5 6 4 4 4 4 5 4 4
18854 042/300 Write block 370 len 4096 data: 2
18855 043/300 Write block 2444 len 4096 data: 5
18856 044/300 Write block 2854 len 20480 data: 4 4 4 4 4
18857 045/300 Write block 1783 len 4096 data: 2
18858 046/300 Write block 1836 len 12288 data: 5 5 6
18859 047/300 Write block 1153 len 36864 data: 6 5 6 5 5 4 4 4 4
18860 048/300 Read block 2086 len 24576
18861 049/300 Write block 4612 len 32768 data: 4 4 4 3 3 3 3 4
18862 050/300 Read block 33 len 16384
18863 051/300 Read block 3772 len 40960
18864 052/300 Write block 4342 len 24576 data: 7 6 6 7 8 7
18865 053/300 Read block 4206 len 28672
18866 054/300 Write block 4077 len 24576 data: 5 5 6 5 5 2
18867 055/300 Write block 409 len 28672 data: 3 4 3 3 4 6 5
18868 056/300 Read block 469 len 20480
18869 057/300 Read block 4000 len 32768
18870 058/300 Read block 2473 len 40960
18871 059/300 Read block 346 len 20480
18872 060/300 Flush
18873 061/300 Read block 1490 len 12288
18874 062/300 Write block 2900 len 20480 data: 4 2 2 2 2
18875 063/300 Read block 234 len 24576
18876 064/300 Read block 1158 len 4096
18877 065/300 Write block 2472 len 8192 data: 6 6
18878 066/300 Read block 1186 len 36864
18879 067/300 Read block 1845 len 32768
18880 068/300 Write block 256 len 8192 data: 4 3
18881 069/300 Read block 3705 len 40960
18882 070/300 Write block 4194 len 24576 data: 3 3 3 4 4 4
18883 071/300 Read block 4171 len 8192
18884 072/300 Read block 3034 len 4096
18885 073/300 Write block 91 len 28672 data: 4 5 6 7 8 5 3
18886 074/300 Write block 2014 len 20480 data: 4 3 2 2 2
18887 075/300 Write block 3933 len 36864 data: 3 4 4 5 4 6 7 8 6
18888 076/300 Write block 550 len 24576 data: 4 5 6 7 3 3
18889 077/300 Flush
18890 078/300 Write block 3619 len 32768 data: 2 2 3 3 3 3 4 3
18891 079/300 Read block 516 len 36864
18892 080/300 Write block 265 len 28672 data: 4 4 3 4 3 3 4
18893 081/300 Read block 3754 len 4096
18894 082/300 Write block 2922 len 16384 data: 3 5 5 5
18895 083/300 Read block 3997 len 8192
18896 084/300 Write block 918 len 20480 data: 3 3 3 4 4
18897 085/300 Read block 1675 len 12288
18898 086/300 Read block 3656 len 16384
18899 087/300 Flush
18900 088/300 Write block 2086 len 20480 data: 6 6 6 6 6
18901 089/300 Flush
18902 090/300 Read block 2858 len 36864
18903 091/300 Read block 2006 len 28672
18904 092/300 Write block 1152 len 4096 data: 5
18905 093/300 Write block 4455 len 40960 data: 2 3 3 3 3 3 5 5 4 4
18906 094/300 Read block 3522 len 40960
18907 095/300 Read block 4260 len 8192
18908 096/300 Read block 3342 len 28672
18909 097/300 Read block 4194 len 8192
18910 098/300 Write block 4725 len 32768 data: 4 4 4 4 5 5 5 5
18911 099/300 Read block 4338 len 16384
18912 100/300 Write block 4843 len 20480 data: 6 5 4 2 2
18913 101/300 Flush
18914 102/300 Write block 1784 len 8192 data: 2 2
18915 103/300 Read block 4000 len 40960
18916 104/300 Read block 4921 len 28672
18917 105/300 Read block 450 len 20480
18918 106/300 Read block 2204 len 12288
18919 107/300 Read block 1791 len 24576
18920 108/300 Read block 1368 len 40960
18921 109/300 Read block 3897 len 4096
18922 110/300 Write block 4756 len 32768 data: 7 7 7 7 5 4 5 5
18923 111/300 Write block 4055 len 40960 data: 4 4 3 3 3 3 2 4 3 3
18924 112/300 Write block 3997 len 32768 data: 3 3 3 3 3 3 3 4
18925 113/300 Write block 4529 len 36864 data: 6 6 5 3 3 3 4 4 3
18926 114/300 Read block 2346 len 4096
18927 115/300 Write block 4669 len 36864 data: 3 3 3 2 2 2 2 2 3
18928 116/300 Write block 1631 len 12288 data: 4 3 5
18929 117/300 Read block 1878 len 16384
18930 118/300 Write block 459 len 16384 data: 4 4 3 2
18931 119/300 Read block 3862 len 4096
18932 120/300 Write block 1792 len 4096 data: 4
18933 121/300 Read block 1043 len 20480
18934 122/300 Write block 476 len 36864 data: 6 5 6 3 3 4 4 4 4
18935 123/300 Read block 2477 len 12288
18936 124/300 Read block 3103 len 8192
18937 125/300 Read block 2277 len 16384
18938 126/300 Write block 4528 len 12288 data: 6 7 7
18939 127/300 Read block 1207 len 12288
18940 128/300 Read block 648 len 20480
18941 129/300 Write block 1534 len 32768 data: 4 4 4 4 5 4 4 4
18942 130/300 Read block 452 len 4096
18943 131/300 Read block 3938 len 24576
18944 132/300 Read block 2219 len 4096
18945 133/300 Flush
18946 134/300 Read block 4491 len 40960
18947 135/300 Write block 1175 len 16384 data: 6 6 7 8
18948 136/300 Flush
18949 137/300 Read block 1462 len 24576
18950 138/300 Read block 1200 len 4096
18951 139/300 Read block 4421 len 36864
18952 140/300 Read block 40 len 40960
18953 141/300 Write block 1642 len 40960 data: 3 2 2 2 2 3 3 3 3 3
18954 142/300 Read block 2920 len 8192
18955 143/300 Read block 3866 len 32768
18956 144/300 Write block 2926 len 4096 data: 5
18957 145/300 Write block 4921 len 28672 data: 2 2 3 3 5 6 5
18958 146/300 Flush
18959 147/300 Write block 1162 len 32768 data: 3 3 4 4 4 4 4 5
18960 148/300 Read block 1242 len 12288
18961 149/300 Write block 4266 len 28672 data: 3 4 5 5 5 7 7
18962 150/300 Write block 2142 len 16384 data: 3 3 3 3
18963 151/300 Read block 356 len 16384
18964 152/300 Write block 4066 len 4096 data: 4
18965 153/300 Write block 2669 len 32768 data: 6 7 4 7 6 7 8 7
18966 154/300 Write block 203 len 36864 data: 2 2 2 2 3 4 5 5 5
18967 155/300 Read block 1222 len 28672
18968 156/300 Read block 732 len 4096
18969 157/300 Flush
18970 158/300 Read block 668 len 8192
18971 159/300 Flush
18972 160/300 Write block 2135 len 12288 data: 4 3 5
18973 161/300 Read block 701 len 24576
18974 162/300 Read block 2991 len 16384
18975 163/300 Write block 3083 len 24576 data: 3 3 2 2 2 3
18976 164/300 Read block 1526 len 8192
18977 165/300 Read block 4704 len 24576
18978 166/300 Flush
18979 167/300 Read block 166 len 20480
18980 168/300 Write block 2342 len 16384 data: 5 5 4 2
18981 169/300 Write block 413 len 40960 data: 5 7 6 6 6 7 8 6 6 7
18982 170/300 Write block 829 len 12288 data: 4 4 4
18983 171/300 Read block 1581 len 24576
18984 172/300 Write block 2483 len 20480 data: 5 6 6 5 5
18985 173/300 Read block 3018 len 12288
18986 174/300 Flush
18987 175/300 Write block 277 len 16384 data: 6 7 5 5
18988 176/300 Write block 3715 len 4096 data: 2
18989 177/300 Write block 4139 len 24576 data: 4 4 5 5 6 5
18990 178/300 Read block 709 len 24576
18991 179/300 Read block 2449 len 32768
18992 180/300 Flush
18993 181/300 Read block 1902 len 20480
18994 182/300 Write block 2795 len 12288 data: 4 3 3
18995 183/300 Read block 596 len 24576
18996 184/300 Write block 4769 len 36864 data: 4 6 5 4 3 3 4 4 5
18997 185/300 Read block 3320 len 20480
18998 186/300 Read block 322 len 16384
18999 187/300 Flush
19000 188/300 Write block 1131 len 28672 data: 4 3 2 2 2 3 3
19001 189/300 Write block 4520 len 32768 data: 5 6 5 5 7 6 7 7
19002 190/300 Write block 980 len 20480 data: 4 3 3 4 4
19003 191/300 Read block 2598 len 24576
19004 192/300 Read block 1954 len 28672
19005 193/300 Write block 2636 len 20480 data: 2 3 4 4 4
19006 194/300 Flush
19007 195/300 Write block 403 len 4096 data: 3
19008 196/300 Write block 3107 len 24576 data: 5 6 6 6 6 7
19009 197/300 Read block 429 len 8192
19010 198/300 Read block 4677 len 8192
19011 199/300 Write block 2794 len 32768 data: 4 5 4 4 3 4 4 4
19012 200/300 Write block 800 len 28672 data: 5 4 3 3 3 4 4
19013 201/300 Read block 508 len 8192
19014 202/300 Read block 2364 len 24576
19015 203/300 Flush
19016 204/300 Read block 4784 len 4096
19017 205/300 Write block 3715 len 20480 data: 3 3 5 7 6
19018 206/300 Read block 4072 len 28672
19019 207/300 Write block 4629 len 28672 data: 6 6 7 7 7 6 4
19020 208/300 Read block 1799 len 16384
19021 209/300 Read block 1215 len 8192
19022 210/300 Read block 1627 len 4096
19023 211/300 Read block 2932 len 20480
19024 212/300 Read block 1029 len 28672
19025 213/300 Read block 4597 len 24576
19026 214/300 Read block 2747 len 32768
19027 215/300 Read block 2213 len 4096
19028 216/300 Read block 4769 len 36864
19029 217/300 Read block 4580 len 32768
19030 218/300 Write block 3788 len 4096 data: 3
19031 219/300 Read block 3612 len 28672
19032 220/300 Read block 1542 len 28672
19033 221/300 Write block 759 len 4096 data: 2
19034 222/300 Write block 2267 len 32768 data: 5 5 4 4 3 3 5 5
19035 223/300 Read block 1554 len 12288
19036 224/300 Read block 731 len 40960
19037 225/300 Read block 716 len 12288
19038 226/300 Read block 4471 len 12288
19039 227/300 Read block 2838 len 16384
19040 228/300 Write block 1364 len 16384 data: 4 5 4 4
19041 229/300 Write block 4092 len 32768 data: 4 4 4 6 5 5 5 5
19042 230/300 Write block 2344 len 28672 data: 5 3 2 2 3 3 2
19043 231/300 Write block 4626 len 40960 data: 5 7 7 7 7 8 8 8 7 5
19044 232/300 Read block 2028 len 8192
19045 233/300 Write block 4051 len 8192 data: 2 3
19046 234/300 Read block 1613 len 28672
19047 235/300 Write block 1981 len 28672 data: 3 4 5 5 4 4 4
19048 236/300 Flush
19049 237/300 Read block 3636 len 16384
19050 238/300 Write block 1686 len 32768 data: 5 4 3 3 3 4 5 4
19051 239/300 Read block 539 len 32768
19052 240/300 Read block 4284 len 24576
19053 241/300 Read block 464 len 36864
19054 242/300 Write block 4804 len 20480 data: 2 3 3 4 4
19055 243/300 Write block 3610 len 40960 data: 5 6 5 4 4 4 3 3 2 3
19056 244/300 Write block 3431 len 36864 data: 3 3 2 3 5 6 6 6 4
19057 245/300 Write block 4212 len 8192 data: 6 6
19058 246/300 Read block 843 len 36864
19059 247/300 Read block 3326 len 20480
19060 248/300 Read block 4919 len 28672
19061 249/300 Write block 4038 len 40960 data: 6 6 5 5 3 6 6 6 5 3
19062 250/300 Read block 3141 len 24576
19063 251/300 Write block 2184 len 36864 data: 4 4 5 4 5 5 6 4 4
19064 252/300 Write block 3636 len 8192 data: 3 3
19065 253/300 Write block 3804 len 32768 data: 4 4 4 5 5 5 4 3
19066 254/300 Write block 2015 len 24576 data: 4 3 3 3 3 4
19067 255/300 Write block 3492 len 32768 data: 4 4 3 3 3 5 5 5
19068 256/300 Write block 2527 len 40960 data: 5 4 5 4 2 2 2 2 2 2
19069 257/300 Read block 4955 len 24576
19070 258/300 Read block 2711 len 24576
19071 259/300 Write block 2181 len 24576 data: 5 5 5 5 5 6
19072 260/300 Read block 2711 len 20480
19073 261/300 Read block 1705 len 28672
19074 262/300 Read block 416 len 16384
19075 263/300 Write block 165 len 12288 data: 3 3 2
19076 264/300 Read block 3124 len 40960
19077 265/300 Write block 3561 len 24576 data: 3 2 3 3 3 3
19078 266/300 Read block 2564 len 20480
19079 267/300 Read block 1794 len 28672
19080 268/300 Read block 3942 len 32768
19081 269/300 Write block 2148 len 36864 data: 4 4 4 4 5 4 3 3 3
19082 270/300 Read block 1567 len 4096
19083 271/300 Flush
19084 272/300 Write block 2259 len 24576 data: 5 5 3 4 4 5
19085 273/300 Write block 1399 len 32768 data: 4 6 4 5 4 4 4 3
19086 274/300 Write block 3396 len 4096 data: 3
19087 275/300 Flush
19088 276/300 Read block 607 len 32768
19089 277/300 Read block 1255 len 8192
19090 278/300 Write block 2120 len 16384 data: 4 5 5 4
19091 279/300 Read block 4964 len 36864
19092 280/300 Read block 3765 len 36864
19093 281/300 Write block 4018 len 8192 data: 3 3
19094 282/300 Write block 4874 len 28672 data: 3 4 5 5 6 7 5
19095 283/300 Write block 678 len 12288 data: 7 8 6
19096 284/300 Flush
19097 285/300 Write block 4659 len 12288 data: 3 3 3
19098 286/300 Write block 3129 len 20480 data: 3 3 3 3 3
19099 287/300 Read block 2357 len 36864
19100 288/300 Write block 1699 len 20480 data: 5 6 6 4 5
19101 289/300 Read block 1402 len 32768
19102 290/300 Read block 3642 len 40960
19103 291/300 Read block 4859 len 8192
19104 292/300 Flush
19105 293/300 Write block 4353 len 4096 data: 6
19106 294/300 Read block 1781 len 40960
19107 295/300 Write block 2560 len 28672 data: 3 4 4 5 5 4 3
19108 296/300 Read block 3553 len 12288
19109 297/300 Read block 4033 len 24576
19110 298/300 Write block 1693 len 8192 data: 5 5
19111 299/300 Write block 2884 len 4096 data: 5
19112 300/300 Flush
19113 301/300 Read block 2649 len 28672
19114 Main task has work to do, go find it
19115 got dsc Start(1)
19116 start 1
19117 Feb 28 06:05:15.875 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/1, method: POST, req_id: 3bcf45c2-782f-4504-a362-19c27f935d2d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
19118 Replay: started 1, returned:Ok(())
19119 ----------------------------------------------------------------
19120 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19121 States: Active Offline Active
19122 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19123 5527 Acked Read 10 Done Sent Done false
19124 5528 Acked Read 9 Done Sent Done false
19125 5529 Acked Read 9 Done Sent Done false
19126 5530 Acked Write 10 Done Sent Done false
19127 5531 Acked Read 6 Done Sent Done false
19128 5532 Acked Read 4 Done Sent Done false
19129 5533 Acked Read 9 Done Sent Done false
19130 5534 Acked Read 1 Done Sent Done false
19131 5535 Acked Read 10 Done Sent Done false
19132 5536 Acked Write 10 Done Sent Done false
19133 5537 Acked Read 2 Done Sent Done false
19134 5538 Acked Read 6 Done Sent Done false
19135 5539 Acked Write 9 Done Sent Done false
19136 5540 Acked Read 7 Done Sent Done false
19137 5541 Acked Read 10 Done Sent Done false
19138 5542 Acked Read 10 Done Sent Done false
19139 5543 Acked Read 2 Done Sent Done false
19140 5544 Acked Read 10 Done Sent Done false
19141 5545 Acked Read 5 Done Sent Done false
19142 5546 Acked Flush 0 Done Sent Done false
19143 5547 Acked Read 6 Done Sent Done false
19144 5548 Acked Flush 0 Done Sent Done false
19145 5549 Acked Flush 0 Done Sent Done false
19146 5550 Acked Read 8 Done Sent Done false
19147 5551 Acked Read 10 Done Sent Done false
19148 5552 Acked Read 3 Done Sent Done false
19149 5553 Acked Write 1 Done Sent Done false
19150 5554 Acked Write 10 Done Sent Done false
19151 5555 Acked Read 9 Done Sent Done false
19152 5556 Acked Read 4 Done Sent Done false
19153 5557 Acked Read 5 Done Sent Done false
19154 5558 Acked Read 4 Done Sent Done false
19155 5559 Acked Write 8 Done Sent Done false
19156 5560 Acked Write 5 Done Sent Done false
19157 5561 Acked Read 3 Done Sent Done false
19158 5562 Acked Write 8 Done Sent Done false
19159 5563 Acked Write 8 Done Sent Done false
19160 5564 Acked Read 4 Done Sent Done false
19161 5565 Acked Read 7 Done Sent Done false
19162 5566 Acked Read 9 Done Sent Done false
19163 5567 Acked Write 10 Done Sent Done false
19164 5568 Acked Write 1 Done Sent Done false
19165 5569 Acked Write 1 Done Sent Done false
19166 5570 Acked Write 5 Done Sent Done false
19167 5571 Acked Write 1 Done Sent Done false
19168 5572 Acked Write 3 Done Sent Done false
19169 5573 Acked Write 9 Done Sent Done false
19170 5574 Acked Read 6 Done Sent Done false
19171 5575 Acked Write 8 Done Sent Done false
19172 5576 Acked Read 4 Done Sent Done false
19173 5577 Acked Read 10 Done Sent Done false
19174 5578 Acked Write 6 Done Sent Done false
19175 5579 Acked Read 7 Done Sent Done false
19176 5580 Acked Write 6 Done Sent Done false
19177 5581 Acked Write 7 Done Sent Done false
19178 5582 Acked Read 5 Done Sent Done false
19179 5583 Acked Read 8 Done Sent Done false
19180 5584 Acked Read 10 Done Sent Done false
19181 5585 Acked Read 5 Done Sent Done false
19182 5586 Acked Flush 0 Done Sent Done false
19183 5587 Acked Read 3 Done Sent Done false
19184 5588 Acked Write 5 Done Sent Done false
19185 5589 Acked Read 6 Done Sent Done false
19186 5590 Acked Read 1 Done Sent Done false
19187 5591 Acked Write 2 Done Sent Done false
19188 5592 Acked Read 9 Done Sent Done false
19189 5593 Acked Read 8 Done Sent Done false
19190 5594 Acked Write 2 Done Sent Done false
19191 5595 Acked Read 10 Done Sent Done false
19192 5596 Acked Write 6 Done Sent Done false
19193 5597 Acked Read 2 Done Sent Done false
19194 5598 Acked Read 1 Done Sent Done false
19195 5599 Acked Write 7 Done Sent Done false
19196 5600 Acked Write 5 Done Sent Done false
19197 5601 Acked Write 9 Done Sent Done false
19198 5602 Acked Write 6 Done Sent Done false
19199 5603 Acked Flush 0 Done Sent Done false
19200 5604 Acked Write 8 Done Sent Done false
19201 5605 Acked Read 9 Done Sent Done false
19202 5606 Acked Write 7 Done Sent Done false
19203 5607 Acked Read 1 Done Sent Done false
19204 5608 Acked Write 4 Done Sent Done false
19205 5609 Acked Read 2 Done Sent Done false
19206 5610 Acked Write 5 Done Sent Done false
19207 5611 Acked Read 3 Done Sent Done false
19208 5612 Acked Read 4 Done Sent Done false
19209 5613 Acked Flush 0 Done Sent Done false
19210 5614 Acked Write 5 Done Sent Done false
19211 5615 Acked Flush 0 Done Sent Done false
19212 5616 Acked Read 9 Done Sent Done false
19213 5617 Acked Read 7 Done Sent Done false
19214 5618 Acked Write 1 Done Sent Done false
19215 5619 Acked Write 10 Done Sent Done false
19216 5620 Acked Read 10 Done Sent Done false
19217 5621 Acked Read 2 Done Sent Done false
19218 5622 Acked Read 7 Done Sent Done false
19219 5623 Acked Read 2 Done Sent Done false
19220 5624 Acked Write 8 Done Sent Done false
19221 5625 Acked Read 4 Done Sent Done false
19222 5626 Acked Write 5 Done Sent Done false
19223 5627 Acked Flush 0 Done Sent Done false
19224 5628 Acked Write 2 Done Sent Done false
19225 5629 Acked Read 10 Done Sent Done false
19226 5630 Acked Read 7 Done Sent Done false
19227 5631 Acked Read 5 Done Sent Done false
19228 5632 Acked Read 3 Done Sent Done false
19229 5633 Acked Read 6 Done Sent Done false
19230 5634 Acked Read 10 Done Sent Done false
19231 5635 Acked Read 1 Done Sent Done false
19232 5636 Acked Write 8 Done Sent Done false
19233 5637 Acked Write 10 Done Sent Done false
19234 5638 Acked Write 8 Done Sent Done false
19235 5639 Acked Write 9 Done Sent Done false
19236 5640 Acked Read 1 Done Sent Done false
19237 5641 Acked Write 9 Done Sent Done false
19238 5642 Acked Write 3 Done Sent Done false
19239 5643 Acked Read 4 Done Sent Done false
19240 5644 Acked Write 4 Done Sent Done false
19241 5645 Acked Read 1 Done Sent Done false
19242 5646 Acked Write 1 Done Sent Done false
19243 5647 Acked Read 5 Done Sent Done false
19244 5648 Acked Write 9 Done Sent Done false
19245 5649 Acked Read 3 Done Sent Done false
19246 5650 Acked Read 2 Done Sent Done false
19247 5651 Acked Read 4 Done Sent Done false
19248 5652 Acked Write 3 Done Sent Done false
19249 5653 Acked Read 3 Done Sent Done false
19250 5654 Acked Read 5 Done Sent Done false
19251 5655 Acked Write 8 Done Sent Done false
19252 5656 Acked Read 1 Done Sent Done false
19253 5657 Acked Read 6 Done Sent Done false
19254 5658 Acked Read 1 Done Sent Done false
19255 5659 Acked Flush 0 Done Sent Done false
19256 5660 Acked Read 10 Done Sent Done false
19257 5661 Acked Write 4 Done Sent Done false
19258 5662 Acked Flush 0 Done Sent Done false
19259 5663 Acked Read 6 Done Sent Done false
19260 5664 Acked Read 1 Done Sent Done false
19261 5665 Acked Read 9 Done Sent Done false
19262 5666 Acked Read 10 Done Sent Done false
19263 5667 Acked Write 10 Done Sent Done false
19264 5668 Acked Read 2 Done Sent Done false
19265 5669 Acked Read 8 Done Sent Done false
19266 5670 Acked Write 1 Done Sent Done false
19267 5671 Acked Write 7 Done Sent Done false
19268 5672 Acked Flush 0 Done Sent Done false
19269 5673 Acked Write 8 Done Sent Done false
19270 5674 Acked Read 3 Done Sent Done false
19271 5675 Acked Write 7 Done Sent Done false
19272 5676 Acked Write 4 Done Sent Done false
19273 5677 Acked Read 4 Done Sent Done false
19274 5678 Acked Write 1 Done Sent Done false
19275 5679 Acked Write 8 Done Sent Done false
19276 5680 Acked Write 9 Done Sent Done false
19277 5681 Acked Read 7 Done Sent Done false
19278 5682 Acked Read 1 Done Sent Done false
19279 5683 Acked Flush 0 Done Sent Done false
19280 5684 Acked Read 2 Done Sent Done false
19281 5685 Acked Flush 0 Done Sent Done false
19282 5686 Acked Write 3 Done Sent Done false
19283 5687 Acked Read 6 Done Sent Done false
19284 5688 Acked Read 4 Done Sent Done false
19285 5689 Acked Write 6 Done Sent Done false
19286 5690 Acked Read 2 Done Sent Done false
19287 5691 Acked Read 6 Done Sent Done false
19288 5692 Acked Flush 0 Done Sent Done false
19289 5693 Acked Read 5 Done Sent Done false
19290 5694 Acked Write 4 Done Sent Done false
19291 5695 Acked Write 10 Done Sent Done false
19292 5696 Acked Write 3 Done Sent Done false
19293 5697 Acked Read 6 Done Sent Done false
19294 5698 Acked Write 5 Done Sent Done false
19295 5699 Acked Read 3 Done Sent Done false
19296 5700 Acked Flush 0 Done Sent Done false
19297 5701 Acked Write 4 Done Sent Done false
19298 5702 Acked Write 1 Done Sent Done false
19299 5703 Acked Write 6 Done Sent Done false
19300 5704 Acked Read 6 Done Sent Done false
19301 5705 Acked Read 8 Done Sent Done false
19302 5706 Acked Flush 0 Done Sent Done false
19303 5707 Acked Read 5 Done Sent Done false
19304 5708 Acked Write 3 Done Sent Done false
19305 5709 Acked Read 6 Done Sent Done false
19306 5710 Acked Write 9 Done Sent Done false
19307 5711 Acked Read 5 Done Sent Done false
19308 5712 Acked Read 4 Done Sent Done false
19309 5713 Acked Flush 0 Done Sent Done false
19310 5714 Acked Write 7 Done Sent Done false
19311 5715 Acked Write 8 Done Sent Done false
19312 5716 Acked Write 5 Done Sent Done false
19313 5717 Acked Read 6 Done Sent Done false
19314 5718 Acked Read 7 Done Sent Done false
19315 5719 Acked Write 5 Done Sent Done false
19316 5720 Acked Flush 0 Done Sent Done false
19317 5721 Acked Write 1 Done Sent Done false
19318 5722 Acked Write 6 Done Sent Done false
19319 5723 Acked Read 2 Done Sent Done false
19320 5724 Acked Read 2 Done Sent Done false
19321 5725 Acked Write 8 Done Sent Done false
19322 5726 Acked Write 7 Done Sent Done false
19323 5727 Acked Read 2 Done Sent Done false
19324 5728 Acked Read 6 Done Sent Done false
19325 5729 Acked Flush 0 Done Sent Done false
19326 5730 Acked Read 1 Done Sent Done false
19327 5731 Acked Write 5 Done Sent Done false
19328 5732 Acked Read 7 Done Sent Done false
19329 5733 Acked Write 7 Done Sent Done false
19330 5734 Acked Read 4 Done Sent Done false
19331 5735 Acked Read 2 Done Sent Done false
19332 5736 Acked Read 1 Done Sent Done false
19333 5737 Acked Read 5 Done Sent Done false
19334 5738 Acked Read 7 Done Sent Done false
19335 5739 Acked Read 6 Done Sent Done false
19336 5740 Acked Read 8 Done Sent Done false
19337 5741 Acked Read 1 Done Sent Done false
19338 5742 Acked Read 9 Done Sent Done false
19339 5743 Acked Read 8 Done Sent Done false
19340 5744 Acked Write 1 Done Sent Done false
19341 5745 Acked Read 7 Done Sent Done false
19342 5746 Acked Read 7 Done Sent Done false
19343 5747 Acked Write 1 Done Sent Done false
19344 5748 Acked Write 8 Done Sent Done false
19345 5749 Acked Read 3 Done Sent Done false
19346 5750 Acked Read 10 Done Sent Done false
19347 5751 Acked Read 3 Done Sent Done false
19348 5752 Acked Read 3 Done Sent Done false
19349 5753 Acked Read 4 Done Sent Done false
19350 5754 Acked Write 4 Done Sent Done false
19351 5755 Acked Write 8 Done Sent Done false
19352 5756 Acked Write 7 Done Sent Done false
19353 5757 Acked Write 10 Done Sent Done false
19354 5758 Acked Read 2 Done Sent Done false
19355 5759 Acked Write 2 Done Sent Done false
19356 5760 Acked Read 7 Done Sent Done false
19357 5761 Acked Write 7 Done Sent Done false
19358 5762 Acked Flush 0 Done Sent Done false
19359 5763 Acked Read 4 Done Sent Done false
19360 5764 Acked Write 8 Done Sent Done false
19361 5765 Acked Read 8 Done Sent Done false
19362 5766 Acked Read 6 Done Sent Done false
19363 5767 Acked Read 9 Done Sent Done false
19364 5768 Acked Write 5 Done Sent Done false
19365 5769 Acked Write 10 Done Sent Done false
19366 5770 Acked Write 9 Done Sent Done false
19367 5771 Acked Write 2 Done Sent Done false
19368 5772 Acked Read 9 Done Sent Done false
19369 5773 Acked Read 5 Done Sent Done false
19370 5774 Acked Read 7 Done Sent Done false
19371 5775 Acked Write 10 Done Sent Done false
19372 5776 Acked Read 6 Done Sent Done false
19373 5777 Acked Write 9 Done Sent Done false
19374 5778 Acked Write 2 Done Sent Done false
19375 5779 Acked Write 8 Done Sent Done false
19376 5780 Acked Write 6 Done Sent Done false
19377 5781 Acked Write 8 Done Sent Done false
19378 5782 Acked Write 10 Done Sent Done false
19379 5783 Acked Read 6 Done Sent Done false
19380 5784 Acked Read 6 Done Sent Done false
19381 5785 Acked Write 6 Done Sent Done false
19382 5786 Acked Read 5 Done Sent Done false
19383 5787 Acked Read 7 Done Sent Done false
19384 5788 Acked Read 4 Done Sent Done false
19385 5789 Acked Write 3 Done Sent Done false
19386 5790 Acked Read 10 Done Sent Done false
19387 5791 Acked Write 6 Done Sent Done false
19388 5792 Acked Read 5 Done Sent Done false
19389 5793 Acked Read 7 Done Sent Done false
19390 5794 Acked Read 8 Done Sent Done false
19391 5795 Acked Write 9 Done Sent Done false
19392 5796 Acked Read 1 Done Sent Done false
19393 5797 Acked Flush 0 Done Sent Done false
19394 5798 Acked Write 6 Done Sent Done false
19395 5799 Acked Write 8 Done Sent Done false
19396 5800 Acked Write 1 Done Sent Done false
19397 5801 Acked Flush 0 Done Sent Done false
19398 5802 Acked Read 8 Done Sent Done false
19399 5803 Acked Read 2 Done Sent Done false
19400 5804 Acked Write 4 Done Sent Done false
19401 5805 Acked Read 9 Done Sent Done false
19402 5806 Acked Read 9 Done Sent Done false
19403 5807 Acked Write 2 Done Sent Done false
19404 5808 Acked Write 7 Done Sent Done false
19405 5809 Acked Write 3 Done Sent Done false
19406 5810 Acked Flush 0 Done Sent Done false
19407 5811 Acked Write 3 Done Sent Done false
19408 5812 Acked Write 5 Done Sent Done false
19409 5813 Acked Read 9 Done Sent Done false
19410 5814 Acked Write 5 Done Sent Done false
19411 5815 Acked Read 8 Done Sent Done false
19412 5816 Acked Read 10 Done Sent Done false
19413 5817 Acked Read 2 Done Sent Done false
19414 5818 Acked Flush 0 Done Sent Done false
19415 5819 Acked Write 1 Done Sent Done false
19416 5820 Acked Read 10 Done Sent Done false
19417 5821 Acked Write 7 Done Sent Done false
19418 5822 Acked Read 3 Done Sent Done false
19419 5823 Acked Read 6 Done Sent Done false
19420 5824 Acked Write 2 Done Sent Done false
19421 5825 Acked Write 1 Done Sent Done false
19422 5826 Acked Flush 0 Done Sent Done false
19423 5827 Acked Read 7 Done Sent Done false
19424 STATES DS:0 DS:1 DS:2 TOTAL
19425 Sent 0 301 0 301
19426 Done 301 0 301 602
19427 Skipped 0 0 0 0
19428 Error 0 0 0 0
19429 Last Flush: Some(JobId(5826)) Some(JobId(5526)) Some(JobId(5826))
19430 Downstairs last five completed: 5526 5525 5524 5523 5522
19431 Upstairs last five completed: 5827 5826 5825 5824 5823
19432 CLIENT: Up:0 ds:301 act:2
194332025-02-28T06:05:16.761ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
19434 [8820] got start action
19435 [8820] I am going to restart
19436 Starting downstairs at port 8820
19437 Make output file at "/tmp/dsc/downstairs-8820.txt"
19438 [8820][1] reports Starting
19439 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1227)
19440 [8820][1] reports Running
194412025-02-28T06:05:17.763ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
194422025-02-28T06:05:18.765ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
19443 ----------------------------------------------------------------
19444 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19445 States: Active Offline Active
19446 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19447 5527 Acked Read 10 Done Sent Done false
19448 5528 Acked Read 9 Done Sent Done false
19449 5529 Acked Read 9 Done Sent Done false
19450 5530 Acked Write 10 Done Sent Done false
19451 5531 Acked Read 6 Done Sent Done false
19452 5532 Acked Read 4 Done Sent Done false
19453 5533 Acked Read 9 Done Sent Done false
19454 5534 Acked Read 1 Done Sent Done false
19455 5535 Acked Read 10 Done Sent Done false
19456 5536 Acked Write 10 Done Sent Done false
19457 5537 Acked Read 2 Done Sent Done false
19458 5538 Acked Read 6 Done Sent Done false
19459 5539 Acked Write 9 Done Sent Done false
19460 5540 Acked Read 7 Done Sent Done false
19461 5541 Acked Read 10 Done Sent Done false
19462 5542 Acked Read 10 Done Sent Done false
19463 5543 Acked Read 2 Done Sent Done false
19464 5544 Acked Read 10 Done Sent Done false
19465 5545 Acked Read 5 Done Sent Done false
19466 5546 Acked Flush 0 Done Sent Done false
19467 5547 Acked Read 6 Done Sent Done false
19468 5548 Acked Flush 0 Done Sent Done false
19469 5549 Acked Flush 0 Done Sent Done false
19470 5550 Acked Read 8 Done Sent Done false
19471 5551 Acked Read 10 Done Sent Done false
19472 5552 Acked Read 3 Done Sent Done false
19473 5553 Acked Write 1 Done Sent Done false
19474 5554 Acked Write 10 Done Sent Done false
19475 5555 Acked Read 9 Done Sent Done false
19476 5556 Acked Read 4 Done Sent Done false
19477 5557 Acked Read 5 Done Sent Done false
19478 5558 Acked Read 4 Done Sent Done false
19479 5559 Acked Write 8 Done Sent Done false
19480 5560 Acked Write 5 Done Sent Done false
19481 5561 Acked Read 3 Done Sent Done false
19482 5562 Acked Write 8 Done Sent Done false
19483 5563 Acked Write 8 Done Sent Done false
19484 5564 Acked Read 4 Done Sent Done false
19485 5565 Acked Read 7 Done Sent Done false
19486 5566 Acked Read 9 Done Sent Done false
19487 5567 Acked Write 10 Done Sent Done false
19488 5568 Acked Write 1 Done Sent Done false
19489 5569 Acked Write 1 Done Sent Done false
19490 5570 Acked Write 5 Done Sent Done false
19491 5571 Acked Write 1 Done Sent Done false
19492 5572 Acked Write 3 Done Sent Done false
19493 5573 Acked Write 9 Done Sent Done false
19494 5574 Acked Read 6 Done Sent Done false
19495 5575 Acked Write 8 Done Sent Done false
19496 5576 Acked Read 4 Done Sent Done false
19497 5577 Acked Read 10 Done Sent Done false
19498 5578 Acked Write 6 Done Sent Done false
19499 5579 Acked Read 7 Done Sent Done false
19500 5580 Acked Write 6 Done Sent Done false
19501 5581 Acked Write 7 Done Sent Done false
19502 5582 Acked Read 5 Done Sent Done false
19503 5583 Acked Read 8 Done Sent Done false
19504 5584 Acked Read 10 Done Sent Done false
19505 5585 Acked Read 5 Done Sent Done false
19506 5586 Acked Flush 0 Done Sent Done false
19507 5587 Acked Read 3 Done Sent Done false
19508 5588 Acked Write 5 Done Sent Done false
19509 5589 Acked Read 6 Done Sent Done false
19510 5590 Acked Read 1 Done Sent Done false
19511 5591 Acked Write 2 Done Sent Done false
19512 5592 Acked Read 9 Done Sent Done false
19513 5593 Acked Read 8 Done Sent Done false
19514 5594 Acked Write 2 Done Sent Done false
19515 5595 Acked Read 10 Done Sent Done false
19516 5596 Acked Write 6 Done Sent Done false
19517 5597 Acked Read 2 Done Sent Done false
19518 5598 Acked Read 1 Done Sent Done false
19519 5599 Acked Write 7 Done Sent Done false
19520 5600 Acked Write 5 Done Sent Done false
19521 5601 Acked Write 9 Done Sent Done false
19522 5602 Acked Write 6 Done Sent Done false
19523 5603 Acked Flush 0 Done Sent Done false
19524 5604 Acked Write 8 Done Sent Done false
19525 5605 Acked Read 9 Done Sent Done false
19526 5606 Acked Write 7 Done Sent Done false
19527 5607 Acked Read 1 Done Sent Done false
19528 5608 Acked Write 4 Done Sent Done false
19529 5609 Acked Read 2 Done Sent Done false
19530 5610 Acked Write 5 Done Sent Done false
19531 5611 Acked Read 3 Done Sent Done false
19532 5612 Acked Read 4 Done Sent Done false
19533 5613 Acked Flush 0 Done Sent Done false
19534 5614 Acked Write 5 Done Sent Done false
19535 5615 Acked Flush 0 Done Sent Done false
19536 5616 Acked Read 9 Done Sent Done false
19537 5617 Acked Read 7 Done Sent Done false
19538 5618 Acked Write 1 Done Sent Done false
19539 5619 Acked Write 10 Done Sent Done false
19540 5620 Acked Read 10 Done Sent Done false
19541 5621 Acked Read 2 Done Sent Done false
19542 5622 Acked Read 7 Done Sent Done false
19543 5623 Acked Read 2 Done Sent Done false
19544 5624 Acked Write 8 Done Sent Done false
19545 5625 Acked Read 4 Done Sent Done false
19546 5626 Acked Write 5 Done Sent Done false
19547 5627 Acked Flush 0 Done Sent Done false
19548 5628 Acked Write 2 Done Sent Done false
19549 5629 Acked Read 10 Done Sent Done false
19550 5630 Acked Read 7 Done Sent Done false
19551 5631 Acked Read 5 Done Sent Done false
19552 5632 Acked Read 3 Done Sent Done false
19553 5633 Acked Read 6 Done Sent Done false
19554 5634 Acked Read 10 Done Sent Done false
19555 5635 Acked Read 1 Done Sent Done false
19556 5636 Acked Write 8 Done Sent Done false
19557 5637 Acked Write 10 Done Sent Done false
19558 5638 Acked Write 8 Done Sent Done false
19559 5639 Acked Write 9 Done Sent Done false
19560 5640 Acked Read 1 Done Sent Done false
19561 5641 Acked Write 9 Done Sent Done false
19562 5642 Acked Write 3 Done Sent Done false
19563 5643 Acked Read 4 Done Sent Done false
19564 5644 Acked Write 4 Done Sent Done false
19565 5645 Acked Read 1 Done Sent Done false
19566 5646 Acked Write 1 Done Sent Done false
19567 5647 Acked Read 5 Done Sent Done false
19568 5648 Acked Write 9 Done Sent Done false
19569 5649 Acked Read 3 Done Sent Done false
19570 5650 Acked Read 2 Done Sent Done false
19571 5651 Acked Read 4 Done Sent Done false
19572 5652 Acked Write 3 Done Sent Done false
19573 5653 Acked Read 3 Done Sent Done false
19574 5654 Acked Read 5 Done Sent Done false
19575 5655 Acked Write 8 Done Sent Done false
19576 5656 Acked Read 1 Done Sent Done false
19577 5657 Acked Read 6 Done Sent Done false
19578 5658 Acked Read 1 Done Sent Done false
19579 5659 Acked Flush 0 Done Sent Done false
19580 5660 Acked Read 10 Done Sent Done false
19581 5661 Acked Write 4 Done Sent Done false
19582 5662 Acked Flush 0 Done Sent Done false
19583 5663 Acked Read 6 Done Sent Done false
19584 5664 Acked Read 1 Done Sent Done false
19585 5665 Acked Read 9 Done Sent Done false
19586 5666 Acked Read 10 Done Sent Done false
19587 5667 Acked Write 10 Done Sent Done false
19588 5668 Acked Read 2 Done Sent Done false
19589 5669 Acked Read 8 Done Sent Done false
19590 5670 Acked Write 1 Done Sent Done false
19591 5671 Acked Write 7 Done Sent Done false
19592 5672 Acked Flush 0 Done Sent Done false
19593 5673 Acked Write 8 Done Sent Done false
19594 5674 Acked Read 3 Done Sent Done false
19595 5675 Acked Write 7 Done Sent Done false
19596 5676 Acked Write 4 Done Sent Done false
19597 5677 Acked Read 4 Done Sent Done false
19598 5678 Acked Write 1 Done Sent Done false
19599 5679 Acked Write 8 Done Sent Done false
19600 5680 Acked Write 9 Done Sent Done false
19601 5681 Acked Read 7 Done Sent Done false
19602 5682 Acked Read 1 Done Sent Done false
19603 5683 Acked Flush 0 Done Sent Done false
19604 5684 Acked Read 2 Done Sent Done false
19605 5685 Acked Flush 0 Done Sent Done false
19606 5686 Acked Write 3 Done Sent Done false
19607 5687 Acked Read 6 Done Sent Done false
19608 5688 Acked Read 4 Done Sent Done false
19609 5689 Acked Write 6 Done Sent Done false
19610 5690 Acked Read 2 Done Sent Done false
19611 5691 Acked Read 6 Done Sent Done false
19612 5692 Acked Flush 0 Done Sent Done false
19613 5693 Acked Read 5 Done Sent Done false
19614 5694 Acked Write 4 Done Sent Done false
19615 5695 Acked Write 10 Done Sent Done false
19616 5696 Acked Write 3 Done Sent Done false
19617 5697 Acked Read 6 Done Sent Done false
19618 5698 Acked Write 5 Done Sent Done false
19619 5699 Acked Read 3 Done Sent Done false
19620 5700 Acked Flush 0 Done Sent Done false
19621 5701 Acked Write 4 Done Sent Done false
19622 5702 Acked Write 1 Done Sent Done false
19623 5703 Acked Write 6 Done Sent Done false
19624 5704 Acked Read 6 Done Sent Done false
19625 5705 Acked Read 8 Done Sent Done false
19626 5706 Acked Flush 0 Done Sent Done false
19627 5707 Acked Read 5 Done Sent Done false
19628 5708 Acked Write 3 Done Sent Done false
19629 5709 Acked Read 6 Done Sent Done false
19630 5710 Acked Write 9 Done Sent Done false
19631 5711 Acked Read 5 Done Sent Done false
19632 5712 Acked Read 4 Done Sent Done false
19633 5713 Acked Flush 0 Done Sent Done false
19634 5714 Acked Write 7 Done Sent Done false
19635 5715 Acked Write 8 Done Sent Done false
19636 5716 Acked Write 5 Done Sent Done false
19637 5717 Acked Read 6 Done Sent Done false
19638 5718 Acked Read 7 Done Sent Done false
19639 5719 Acked Write 5 Done Sent Done false
19640 5720 Acked Flush 0 Done Sent Done false
19641 5721 Acked Write 1 Done Sent Done false
19642 5722 Acked Write 6 Done Sent Done false
19643 5723 Acked Read 2 Done Sent Done false
19644 5724 Acked Read 2 Done Sent Done false
19645 5725 Acked Write 8 Done Sent Done false
19646 5726 Acked Write 7 Done Sent Done false
19647 5727 Acked Read 2 Done Sent Done false
19648 5728 Acked Read 6 Done Sent Done false
19649 5729 Acked Flush 0 Done Sent Done false
19650 5730 Acked Read 1 Done Sent Done false
19651 5731 Acked Write 5 Done Sent Done false
19652 5732 Acked Read 7 Done Sent Done false
19653 5733 Acked Write 7 Done Sent Done false
19654 5734 Acked Read 4 Done Sent Done false
19655 5735 Acked Read 2 Done Sent Done false
19656 5736 Acked Read 1 Done Sent Done false
19657 5737 Acked Read 5 Done Sent Done false
19658 5738 Acked Read 7 Done Sent Done false
19659 5739 Acked Read 6 Done Sent Done false
19660 5740 Acked Read 8 Done Sent Done false
19661 5741 Acked Read 1 Done Sent Done false
19662 5742 Acked Read 9 Done Sent Done false
19663 5743 Acked Read 8 Done Sent Done false
19664 5744 Acked Write 1 Done Sent Done false
19665 5745 Acked Read 7 Done Sent Done false
19666 5746 Acked Read 7 Done Sent Done false
19667 5747 Acked Write 1 Done Sent Done false
19668 5748 Acked Write 8 Done Sent Done false
19669 5749 Acked Read 3 Done Sent Done false
19670 5750 Acked Read 10 Done Sent Done false
19671 5751 Acked Read 3 Done Sent Done false
19672 5752 Acked Read 3 Done Sent Done false
19673 5753 Acked Read 4 Done Sent Done false
19674 5754 Acked Write 4 Done Sent Done false
19675 5755 Acked Write 8 Done Sent Done false
19676 5756 Acked Write 7 Done Sent Done false
19677 5757 Acked Write 10 Done Sent Done false
19678 5758 Acked Read 2 Done Sent Done false
19679 5759 Acked Write 2 Done Sent Done false
19680 5760 Acked Read 7 Done Sent Done false
19681 5761 Acked Write 7 Done Sent Done false
19682 5762 Acked Flush 0 Done Sent Done false
19683 5763 Acked Read 4 Done Sent Done false
19684 5764 Acked Write 8 Done Sent Done false
19685 5765 Acked Read 8 Done Sent Done false
19686 5766 Acked Read 6 Done Sent Done false
19687 5767 Acked Read 9 Done Sent Done false
19688 5768 Acked Write 5 Done Sent Done false
19689 5769 Acked Write 10 Done Sent Done false
19690 5770 Acked Write 9 Done Sent Done false
19691 5771 Acked Write 2 Done Sent Done false
19692 5772 Acked Read 9 Done Sent Done false
19693 5773 Acked Read 5 Done Sent Done false
19694 5774 Acked Read 7 Done Sent Done false
19695 5775 Acked Write 10 Done Sent Done false
19696 5776 Acked Read 6 Done Sent Done false
19697 5777 Acked Write 9 Done Sent Done false
19698 5778 Acked Write 2 Done Sent Done false
19699 5779 Acked Write 8 Done Sent Done false
19700 5780 Acked Write 6 Done Sent Done false
19701 5781 Acked Write 8 Done Sent Done false
19702 5782 Acked Write 10 Done Sent Done false
19703 5783 Acked Read 6 Done Sent Done false
19704 5784 Acked Read 6 Done Sent Done false
19705 5785 Acked Write 6 Done Sent Done false
19706 5786 Acked Read 5 Done Sent Done false
19707 5787 Acked Read 7 Done Sent Done false
19708 5788 Acked Read 4 Done Sent Done false
19709 5789 Acked Write 3 Done Sent Done false
19710 5790 Acked Read 10 Done Sent Done false
19711 5791 Acked Write 6 Done Sent Done false
19712 5792 Acked Read 5 Done Sent Done false
19713 5793 Acked Read 7 Done Sent Done false
19714 5794 Acked Read 8 Done Sent Done false
19715 5795 Acked Write 9 Done Sent Done false
19716 5796 Acked Read 1 Done Sent Done false
19717 5797 Acked Flush 0 Done Sent Done false
19718 5798 Acked Write 6 Done Sent Done false
19719 5799 Acked Write 8 Done Sent Done false
19720 5800 Acked Write 1 Done Sent Done false
19721 5801 Acked Flush 0 Done Sent Done false
19722 5802 Acked Read 8 Done Sent Done false
19723 5803 Acked Read 2 Done Sent Done false
19724 5804 Acked Write 4 Done Sent Done false
19725 5805 Acked Read 9 Done Sent Done false
19726 5806 Acked Read 9 Done Sent Done false
19727 5807 Acked Write 2 Done Sent Done false
19728 5808 Acked Write 7 Done Sent Done false
19729 5809 Acked Write 3 Done Sent Done false
19730 5810 Acked Flush 0 Done Sent Done false
19731 5811 Acked Write 3 Done Sent Done false
19732 5812 Acked Write 5 Done Sent Done false
19733 5813 Acked Read 9 Done Sent Done false
19734 5814 Acked Write 5 Done Sent Done false
19735 5815 Acked Read 8 Done Sent Done false
19736 5816 Acked Read 10 Done Sent Done false
19737 5817 Acked Read 2 Done Sent Done false
19738 5818 Acked Flush 0 Done Sent Done false
19739 5819 Acked Write 1 Done Sent Done false
19740 5820 Acked Read 10 Done Sent Done false
19741 5821 Acked Write 7 Done Sent Done false
19742 5822 Acked Read 3 Done Sent Done false
19743 5823 Acked Read 6 Done Sent Done false
19744 5824 Acked Write 2 Done Sent Done false
19745 5825 Acked Write 1 Done Sent Done false
19746 5826 Acked Flush 0 Done Sent Done false
19747 5827 Acked Read 7 Done Sent Done false
19748 STATES DS:0 DS:1 DS:2 TOTAL
19749 Sent 0 301 0 301
19750 Done 301 0 301 602
19751 Skipped 0 0 0 0
19752 Error 0 0 0 0
19753 Last Flush: Some(JobId(5826)) Some(JobId(5526)) Some(JobId(5826))
19754 Downstairs last five completed: 5526 5525 5524 5523 5522
19755 Upstairs last five completed: 5827 5826 5825 5824 5823
19756 CLIENT: Up:0 ds:301 act:2
197572025-02-28T06:05:20.768ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
197582025-02-28T06:05:21.626ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
197592025-02-28T06:05:21.626ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33576) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
197602025-02-28T06:05:21.627ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
197612025-02-28T06:05:21.628ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
197622025-02-28T06:05:21.628ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
197632025-02-28T06:05:21.628ZINFOcrucible: send last flush ID to this DS: Some(JobId(5526)) = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
197642025-02-28T06:05:21.628ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
197652025-02-28T06:05:21.628ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5526)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
197662025-02-28T06:05:21.628ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5526)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
19767 ----------------------------------------------------------------
19768 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19769 Downstairs last five completed: 5828 5827 5826 5825 5824
19770 Upstairs last five completed: 5828 5827 5826 5825 5824
19771 CLIENT: Up:0 ds:0 act:3
19772 Replay: All jobs finished
19773 Main task has work to do, go find it
19774 got dsc Stop(0)
19775 stop 0
19776 Feb 28 06:05:23.881 INFO request completed, [8810] Got stop action so:false kr:false
19777 latency_us: 358, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 41f01ae6-ef21-4a44-b4b0-1a036aa56905, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
197782025-02-28T06:05:23.882ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
197792025-02-28T06:05:23.882ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
197802025-02-28T06:05:23.882ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
197812025-02-28T06:05:23.882ZINFOcrucible: client task is exiting = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
19782 Feb 28 06:05:23.882 INFO request completed, latency_us: 136, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1380ef48-91a6-4c8d-bebb-1f2f892c9341, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
19783 [8810] Exited with: ExitStatus(unix_wait_status(9))
19784 [8810][0] reports Exit
19785 Feb 28 06:05:27.885 INFO request completed, latency_us: 381, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6056b0a6-aa70-4a55-b2da-f227162966f1, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
19786 001/300 Read block 3439 len 4096
19787 002/300 Write block 215 len 36864 data: 6 4 4 5 7 4 5 4 4
19788 003/300 Write block 3215 len 8192 data: 3 3
19789 004/300 Read block 2278 len 12288
19790 005/300 Read block 767 len 12288
19791 006/300 Read block 3140 len 24576
19792 007/300 Flush
19793 008/300 Read block 4535 len 28672
19794 009/300 Flush
19795 010/300 Write block 3271 len 40960 data: 4 4 3 4 4 4 3 3 3 2
19796 011/300 Read block 86 len 20480
19797 012/300 Read block 2710 len 12288
19798 013/300 Write block 483 len 20480 data: 5 5 4 5 4
19799 014/300 Read block 3424 len 8192
19800 015/300 Write block 3690 len 20480 data: 3 3 5 3 2
19801 016/300 Read block 1464 len 8192
19802 017/300 Read block 1095 len 16384
19803 018/300 Read block 1229 len 16384
19804 019/300 Write block 421 len 12288 data: 7 8 6
19805 020/300 Write block 4387 len 40960 data: 5 6 5 7 7 5 4 4 4 4
19806 021/300 Read block 4344 len 8192
19807 022/300 Write block 3903 len 8192 data: 5 5
19808 023/300 Write block 3324 len 16384 data: 4 3 2 2
19809 024/300 Read block 3918 len 8192
19810 025/300 Write block 3208 len 40960 data: 4 4 3 3 3 3 3 4 4 3
19811 026/300 Write block 1072 len 20480 data: 5 4 4 4 3
19812 027/300 Read block 4057 len 8192
19813 028/300 Read block 3777 len 20480
19814 029/300 Read block 1429 len 20480
19815 030/300 Read block 4382 len 40960
19816 031/300 Read block 1237 len 12288
19817 032/300 Write block 942 len 40960 data: 6 5 5 4 3 3 3 5 5 5
19818 033/300 Write block 1734 len 32768 data: 3 3 3 3 3 4 4 6
19819 034/300 Read block 1567 len 32768
19820 035/300 Write block 3944 len 40960 data: 4 3 3 2 2 2 2 2 2 2
19821 036/300 Read block 3542 len 4096
19822 037/300 Write block 812 len 16384 data: 6 6 5 4
19823 038/300 Write block 3179 len 36864 data: 3 4 4 4 4 5 5 4 4
19824 039/300 Read block 3193 len 36864
19825 040/300 Read block 333 len 24576
19826 041/300 Flush
19827 042/300 Flush
19828 043/300 Flush
19829 044/300 Read block 2798 len 24576
19830 045/300 Read block 1903 len 20480
19831 046/300 Write block 4467 len 40960 data: 3 3 3 3 2 2 3 3 4 4
19832 047/300 Read block 24 len 36864
19833 048/300 Read block 1687 len 28672
19834 049/300 Write block 3993 len 28672 data: 3 3 3 3 4 4 4
19835 050/300 Write block 4902 len 32768 data: 5 6 5 4 4 3 3 3
19836 051/300 Write block 3867 len 4096 data: 5
19837 052/300 Write block 1196 len 32768 data: 2 3 4 3 2 3 3 4
19838 053/300 Write block 4687 len 4096 data: 3
19839 054/300 Write block 3580 len 12288 data: 2 2 2
19840 055/300 Flush
19841 056/300 Read block 4293 len 20480
19842 057/300 Read block 1182 len 8192
19843 058/300 Read block 3100 len 8192
19844 059/300 Read block 715 len 40960
19845 060/300 Flush
19846 061/300 Read block 2781 len 32768
19847 062/300 Read block 1845 len 12288
19848 063/300 Write block 2661 len 36864 data: 2 4 5 6 6 6 6 6 7
19849 064/300 Read block 1054 len 20480
19850 065/300 Read block 886 len 24576
19851 066/300 Write block 3947 len 8192 data: 3 3
19852 067/300 Read block 915 len 28672
19853 068/300 Write block 3612 len 16384 data: 6 5 5 5
19854 069/300 Flush
19855 070/300 Read block 1013 len 36864
19856 071/300 Flush
19857 072/300 Read block 4680 len 4096
19858 073/300 Read block 1228 len 32768
19859 074/300 Flush
19860 075/300 Read block 2673 len 16384
19861 076/300 Write block 3862 len 36864 data: 5 5 6 5 5 6 5 7 6
19862 077/300 Read block 54 len 12288
19863 078/300 Read block 2095 len 36864
19864 079/300 Write block 3164 len 36864 data: 3 4 4 4 4 4 4 2 2
19865 080/300 Flush
19866 081/300 Write block 2647 len 20480 data: 4 4 4 4 4
19867 082/300 Write block 4375 len 32768 data: 5 5 6 7 7 7 6 4
19868 083/300 Read block 4882 len 36864
19869 084/300 Write block 483 len 24576 data: 6 6 5 6 5 3
19870 085/300 Write block 3623 len 20480 data: 4 4 5 4 3
19871 086/300 Read block 2425 len 20480
19872 087/300 Read block 2492 len 28672
19873 088/300 Write block 2089 len 12288 data: 7 7 7
19874 089/300 Write block 4951 len 4096 data: 4
19875 090/300 Flush
19876 091/300 Read block 820 len 12288
19877 092/300 Write block 2748 len 16384 data: 4 4 5 4
19878 093/300 Flush
19879 094/300 Read block 967 len 36864
19880 095/300 Read block 4852 len 32768
19881 096/300 Write block 1261 len 24576 data: 6 6 5 6 4 3
19882 097/300 Flush
19883 098/300 Flush
19884 099/300 Write block 3350 len 4096 data: 5
19885 100/300 Write block 1331 len 8192 data: 5 4
19886 101/300 Flush
19887 102/300 Write block 601 len 12288 data: 4 4 4
19888 103/300 Read block 209 len 28672
19889 104/300 Write block 4419 len 12288 data: 3 3 3
19890 105/300 Write block 2012 len 24576 data: 4 4 5 5 4 4
19891 106/300 Read block 679 len 40960
19892 107/300 Write block 820 len 20480 data: 2 2 2 3 2
19893 108/300 Write block 2857 len 32768 data: 5 5 3 2 4 4 5 4
19894 109/300 Write block 4662 len 12288 data: 3 3 3
19895 110/300 Read block 2088 len 32768
19896 111/300 Read block 64 len 28672
19897 112/300 Write block 2076 len 8192 data: 3 3
19898 113/300 Write block 441 len 20480 data: 6 8 8 9 9
19899 114/300 Write block 4818 len 24576 data: 3 3 3 3 3 5
19900 115/300 Read block 4988 len 32768
19901 116/300 Read block 1000 len 4096
19902 117/300 Read block 1923 len 16384
19903 118/300 Flush
19904 119/300 Write block 4355 len 40960 data: 7 5 6 6 5 3 2 2 2 3
19905 120/300 Write block 1097 len 36864 data: 2 3 3 2 2 2 2 2 2
19906 121/300 Write block 480 len 4096 data: 4
19907 122/300 Write block 2051 len 32768 data: 4 4 4 2 2 3 3 4
19908 123/300 Read block 3928 len 28672
19909 124/300 Read block 244 len 40960
19910 125/300 Read block 819 len 20480
19911 126/300 Write block 486 len 32768 data: 7 6 4 3 3 3 3 4
19912 127/300 Flush
19913 128/300 Read block 73 len 36864
19914 129/300 Read block 4359 len 24576
19915 130/300 Flush
19916 131/300 Write block 1806 len 16384 data: 2 2 2 3
19917 132/300 Read block 4103 len 36864
19918 133/300 Write block 860 len 24576 data: 3 3 4 4 4 3
19919 134/300 Read block 301 len 24576
19920 135/300 Write block 4159 len 4096 data: 6
19921 136/300 Read block 3188 len 16384
19922 137/300 Read block 2137 len 20480
19923 138/300 Write block 946 len 24576 data: 4 4 4 6 6 6
19924 139/300 Read block 3354 len 4096
19925 140/300 Read block 3794 len 28672
19926 141/300 Write block 1737 len 20480 data: 4 4 5 5 7
19927 142/300 Write block 201 len 20480 data: 3 2 3 3 3
19928 143/300 Read block 3687 len 16384
19929 144/300 Read block 2536 len 28672
19930 145/300 Read block 3551 len 12288
19931 146/300 Write block 4963 len 36864 data: 4 4 5 4 4 4 4 4 5
19932 147/300 Write block 693 len 12288 data: 2 2 2
19933 148/300 Write block 1891 len 4096 data: 6
19934 149/300 Read block 4091 len 40960
19935 150/300 Read block 4579 len 32768
19936 151/300 Read block 3769 len 16384
19937 152/300 Write block 3200 len 24576 data: 4 5 5 5 5 6
19938 153/300 Write block 1817 len 4096 data: 3
19939 154/300 Read block 4370 len 20480
19940 155/300 Write block 1349 len 20480 data: 3 4 4 4 4
19941 156/300 Write block 417 len 24576 data: 7 8 9 7 8 9
19942 157/300 Read block 3619 len 28672
19943 158/300 Read block 2210 len 12288
19944 159/300 Write block 3151 len 36864 data: 2 2 2 4 4 6 7 7 8
19945 160/300 Read block 389 len 24576
19946 161/300 Read block 576 len 28672
19947 162/300 Read block 1278 len 28672
19948 163/300 Write block 1827 len 24576 data: 4 5 5 3 3 3
19949 164/300 Read block 3071 len 32768
19950 165/300 Read block 2154 len 12288
19951 166/300 Read block 512 len 32768
19952 167/300 Read block 1752 len 28672
19953 168/300 Write block 3933 len 28672 data: 4 5 5 6 5 7 8
19954 169/300 Write block 4041 len 16384 data: 6 4 7 7
19955 170/300 Write block 4155 len 40960 data: 4 4 4 5 7 6 4 4 4 4
19956 171/300 Read block 2839 len 24576
19957 172/300 Read block 1676 len 36864
19958 173/300 Write block 1788 len 12288 data: 2 3 3
19959 174/300 Read block 2574 len 8192
19960 175/300 Write block 4100 len 40960 data: 5 6 5 5 4 2 2 2 3 3
19961 176/300 Write block 3114 len 28672 data: 8 6 6 6 5 5 3
19962 177/300 Write block 1964 len 20480 data: 2 2 3 3 2
19963 178/300 Write block 90 len 24576 data: 3 5 6 7 8 9
19964 179/300 Read block 3158 len 28672
19965 180/300 Write block 4321 len 32768 data: 5 4 5 4 4 3 3 3
19966 181/300 Read block 2037 len 16384
19967 182/300 Read block 2620 len 20480
19968 183/300 Write block 721 len 24576 data: 4 3 3 3 3 3
19969 184/300 Read block 3184 len 8192
19970 185/300 Read block 3410 len 8192
19971 186/300 Write block 2917 len 12288 data: 5 5 4
19972 187/300 Read block 994 len 40960
19973 188/300 Write block 3919 len 16384 data: 4 4 3 2
19974 189/300 Write block 1023 len 8192 data: 6 6
19975 190/300 Read block 302 len 12288
19976 191/300 Flush
19977 192/300 Write block 4770 len 36864 data: 7 6 5 4 4 5 5 6 4
19978 193/300 Read block 65 len 20480
19979 194/300 Read block 4079 len 24576
19980 195/300 Write block 2456 len 32768 data: 6 7 7 6 5 4 3 3
19981 196/300 Write block 938 len 4096 data: 5
19982 197/300 Read block 375 len 12288
19983 198/300 Read block 3958 len 4096
19984 199/300 Read block 774 len 24576
19985 200/300 Write block 2579 len 36864 data: 3 3 3 2 3 3 3 3 3
19986 201/300 Read block 3092 len 24576
19987 202/300 Read block 4153 len 4096
19988 203/300 Read block 3070 len 40960
19989 204/300 Read block 1580 len 40960
19990 205/300 Read block 578 len 24576
19991 206/300 Write block 1540 len 28672 data: 5 5 4 3 3 3 4
19992 207/300 Read block 2822 len 28672
19993 208/300 Read block 1874 len 4096
19994 209/300 Write block 2552 len 40960 data: 4 4 4 4 4 4 4 3 4 5
19995 210/300 Read block 2860 len 12288
19996 211/300 Read block 170 len 4096
19997 212/300 Write block 1125 len 16384 data: 5 5 5 4
19998 213/300 Read block 4293 len 16384
19999 214/300 Write block 1320 len 16384 data: 3 3 4 4
20000 215/300 Write block 1128 len 36864 data: 5 4 4 5 4 3 3 3 4
20001 216/300 Write block 4599 len 40960 data: 3 3 3 3 3 3 4 4 4 3
20002 217/300 Read block 1094 len 12288
20003 218/300 Read block 744 len 20480
20004 219/300 Read block 4020 len 12288
20005 220/300 Write block 2759 len 4096 data: 5
20006 221/300 Read block 4977 len 28672
20007 222/300 Read block 976 len 20480
20008 223/300 Write block 4510 len 24576 data: 5 5 4 5 6 5
20009 224/300 Write block 1065 len 32768 data: 4 5 4 4 5 5 5 6
20010 225/300 Read block 3001 len 16384
20011 226/300 Write block 1225 len 24576 data: 4 3 4 3 4 4
20012 227/300 Read block 2452 len 24576
20013 228/300 Write block 1399 len 4096 data: 5
20014 229/300 Write block 1173 len 8192 data: 6 7
20015 230/300 Write block 3180 len 4096 data: 5
20016 231/300 Read block 2485 len 12288
20017 232/300 Read block 3367 len 20480
20018 233/300 Read block 785 len 28672
20019 234/300 Read block 958 len 40960
20020 235/300 Read block 4013 len 32768
20021 236/300 Write block 2012 len 40960 data: 5 5 6 6 5 5 4 4 5 4
20022 237/300 Read block 711 len 16384
20023 238/300 Read block 443 len 32768
20024 239/300 Write block 3101 len 40960 data: 7 6 6 5 5 4 6 7 7 7
20025 240/300 Write block 3531 len 4096 data: 3
20026 241/300 Write block 1682 len 28672 data: 5 6 6 5 6 5 4
20027 242/300 Write block 3707 len 20480 data: 6 4 4 3 3
20028 243/300 Write block 2495 len 36864 data: 3 3 3 3 3 3 2 2 3
20029 244/300 Write block 577 len 40960 data: 5 5 5 6 5 5 5 3 3 3
20030 245/300 Read block 2382 len 8192
20031 246/300 Write block 230 len 20480 data: 4 4 3 3 4
20032 247/300 Read block 349 len 28672
20033 248/300 Write block 2431 len 24576 data: 4 4 3 3 3 3
20034 249/300 Read block 4176 len 8192
20035 250/300 Write block 257 len 16384 data: 4 3 3 4
20036 251/300 Write block 1557 len 20480 data: 3 3 3 3 4
20037 252/300 Read block 1668 len 20480
20038 253/300 Write block 4809 len 20480 data: 3 4 4 2 2
20039 254/300 Write block 4155 len 32768 data: 5 5 5 6 8 7 5 5
20040 255/300 Write block 769 len 12288 data: 4 4 3
20041 256/300 Read block 747 len 4096
20042 257/300 Read block 1760 len 16384
20043 258/300 Read block 4756 len 36864
20044 259/300 Read block 2792 len 8192
20045 260/300 Flush
20046 261/300 Read block 1001 len 16384
20047 262/300 Write block 3470 len 32768 data: 3 3 3 4 4 5 4 4
20048 263/300 Read block 980 len 36864
20049 264/300 Read block 2087 len 20480
20050 265/300 Write block 3814 len 8192 data: 2 2
20051 266/300 Read block 4959 len 24576
20052 267/300 Read block 1982 len 20480
20053 268/300 Read block 2355 len 24576
20054 269/300 Read block 3047 len 20480
20055 270/300 Flush
20056 271/300 Read block 1535 len 12288
20057 272/300 Read block 63 len 28672
20058 273/300 Write block 2758 len 28672 data: 3 6 5 4 4 4 3
20059 274/300 Flush
20060 275/300 Read block 2099 len 40960
20061 276/300 Read block 2659 len 20480
20062 277/300 Flush
20063 278/300 Read block 1722 len 24576
20064 279/300 Write block 1703 len 40960 data: 6 4 3 4 6 6 7 7 6 7
20065 280/300 Write block 4187 len 20480 data: 6 4 4 6 6
20066 281/300 Flush
20067 282/300 Write block 202 len 12288 data: 3 4 4
20068 283/300 Read block 512 len 40960
20069 284/300 Flush
20070 285/300 Read block 1271 len 20480
20071 286/300 Flush
20072 287/300 Write block 2634 len 32768 data: 2 2 3 4 5 5 5 5
20073 288/300 Read block 1692 len 32768
20074 289/300 Write block 356 len 4096 data: 5
20075 290/300 Read block 3158 len 20480
20076 291/300 Write block 970 len 8192 data: 3 3
20077 292/300 Read block 4992 len 12288
20078 293/300 Read block 4111 len 8192
20079 294/300 Flush
20080 295/300 Read block 4844 len 36864
20081 296/300 Write block 2150 len 8192 data: 5 5
20082 297/300 Flush
20083 298/300 Write block 83 len 32768 data: 4 3 3 3 2 2 2 4
20084 299/300 Write block 413 len 8192 data: 6 8
20085 300/300 Read block 4110 len 20480
20086 301/300 Read block 835 len 20480
20087 Main task has work to do, go find it
20088 got dsc Start(0)
20089 start 0
20090 Feb 28 06:05:28.126 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/0, method: POST, req_id: 3f28db01-95c5-4034-823c-fd3b930a7a3e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
20091 Replay: started 0, returned:Ok(())
20092 ----------------------------------------------------------------
20093 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20094 States: Offline Active Active
20095 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20096 5829 Acked Read 1 Sent Done Done false
20097 5830 Acked Write 9 Sent Done Done false
20098 5831 Acked Write 2 Sent Done Done false
20099 5832 Acked Read 3 Sent Done Done false
20100 5833 Acked Read 3 Sent Done Done false
20101 5834 Acked Read 6 Sent Done Done false
20102 5835 Acked Flush 0 Sent Done Done false
20103 5836 Acked Read 7 Sent Done Done false
20104 5837 Acked Flush 0 Sent Done Done false
20105 5838 Acked Write 10 Sent Done Done false
20106 5839 Acked Read 5 Sent Done Done false
20107 5840 Acked Read 3 Sent Done Done false
20108 5841 Acked Write 5 Sent Done Done false
20109 5842 Acked Read 2 Sent Done Done false
20110 5843 Acked Write 5 Sent Done Done false
20111 5844 Acked Read 2 Sent Done Done false
20112 5845 Acked Read 4 Sent Done Done false
20113 5846 Acked Read 4 Sent Done Done false
20114 5847 Acked Write 3 Sent Done Done false
20115 5848 Acked Write 10 Sent Done Done false
20116 5849 Acked Read 2 Sent Done Done false
20117 5850 Acked Write 2 Sent Done Done false
20118 5851 Acked Write 4 Sent Done Done false
20119 5852 Acked Read 2 Sent Done Done false
20120 5853 Acked Write 10 Sent Done Done false
20121 5854 Acked Write 5 Sent Done Done false
20122 5855 Acked Read 2 Sent Done Done false
20123 5856 Acked Read 5 Sent Done Done false
20124 5857 Acked Read 5 Sent Done Done false
20125 5858 Acked Read 10 Sent Done Done false
20126 5859 Acked Read 3 Sent Done Done false
20127 5860 Acked Write 10 Sent Done Done false
20128 5861 Acked Write 8 Sent Done Done false
20129 5862 Acked Read 8 Sent Done Done false
20130 5863 Acked Write 10 Sent Done Done false
20131 5864 Acked Read 1 Sent Done Done false
20132 5865 Acked Write 4 Sent Done Done false
20133 5866 Acked Write 9 Sent Done Done false
20134 5867 Acked Read 9 Sent Done Done false
20135 5868 Acked Read 6 Sent Done Done false
20136 5869 Acked Flush 0 Sent Done Done false
20137 5870 Acked Flush 0 Sent Done Done false
20138 5871 Acked Flush 0 Sent Done Done false
20139 5872 Acked Read 6 Sent Done Done false
20140 5873 Acked Read 5 Sent Done Done false
20141 5874 Acked Write 10 Sent Done Done false
20142 5875 Acked Read 9 Sent Done Done false
20143 5876 Acked Read 7 Sent Done Done false
20144 5877 Acked Write 7 Sent Done Done false
20145 5878 Acked Write 8 Sent Done Done false
20146 5879 Acked Write 1 Sent Done Done false
20147 5880 Acked Write 8 Sent Done Done false
20148 5881 Acked Write 1 Sent Done Done false
20149 5882 Acked Write 3 Sent Done Done false
20150 5883 Acked Flush 0 Sent Done Done false
20151 5884 Acked Read 5 Sent Done Done false
20152 5885 Acked Read 2 Sent Done Done false
20153 5886 Acked Read 2 Sent Done Done false
20154 5887 Acked Read 10 Sent Done Done false
20155 5888 Acked Flush 0 Sent Done Done false
20156 5889 Acked Read 8 Sent Done Done false
20157 5890 Acked Read 3 Sent Done Done false
20158 5891 Acked Write 9 Sent Done Done false
20159 5892 Acked Read 5 Sent Done Done false
20160 5893 Acked Read 6 Sent Done Done false
20161 5894 Acked Write 2 Sent Done Done false
20162 5895 Acked Read 7 Sent Done Done false
20163 5896 Acked Write 4 Sent Done Done false
20164 5897 Acked Flush 0 Sent Done Done false
20165 5898 Acked Read 9 Sent Done Done false
20166 5899 Acked Flush 0 Sent Done Done false
20167 5900 Acked Read 1 Sent Done Done false
20168 5901 Acked Read 8 Sent Done Done false
20169 5902 Acked Flush 0 Sent Done Done false
20170 5903 Acked Read 4 Sent Done Done false
20171 5904 Acked Write 9 Sent Done Done false
20172 5905 Acked Read 3 Sent Done Done false
20173 5906 Acked Read 9 Sent Done Done false
20174 5907 Acked Write 9 Sent Done Done false
20175 5908 Acked Flush 0 Sent Done Done false
20176 5909 Acked Write 5 Sent Done Done false
20177 5910 Acked Write 8 Sent Done Done false
20178 5911 Acked Read 9 Sent Done Done false
20179 5912 Acked Write 6 Sent Done Done false
20180 5913 Acked Write 5 Sent Done Done false
20181 5914 Acked Read 5 Sent Done Done false
20182 5915 Acked Read 7 Sent Done Done false
20183 5916 Acked Write 3 Sent Done Done false
20184 5917 Acked Write 1 Sent Done Done false
20185 5918 Acked Flush 0 Sent Done Done false
20186 5919 Acked Read 3 Sent Done Done false
20187 5920 Acked Write 4 Sent Done Done false
20188 5921 Acked Flush 0 Sent Done Done false
20189 5922 Acked Read 9 Sent Done Done false
20190 5923 Acked Read 8 Sent Done Done false
20191 5924 Acked Write 6 Sent Done Done false
20192 5925 Acked Flush 0 Sent Done Done false
20193 5926 Acked Flush 0 Sent Done Done false
20194 5927 Acked Write 1 Sent Done Done false
20195 5928 Acked Write 2 Sent Done Done false
20196 5929 Acked Flush 0 Sent Done Done false
20197 5930 Acked Write 3 Sent Done Done false
20198 5931 Acked Read 7 Sent Done Done false
20199 5932 Acked Write 3 Sent Done Done false
20200 5933 Acked Write 6 Sent Done Done false
20201 5934 Acked Read 10 Sent Done Done false
20202 5935 Acked Write 5 Sent Done Done false
20203 5936 Acked Write 8 Sent Done Done false
20204 5937 Acked Write 3 Sent Done Done false
20205 5938 Acked Read 8 Sent Done Done false
20206 5939 Acked Read 7 Sent Done Done false
20207 5940 Acked Write 2 Sent Done Done false
20208 5941 Acked Write 5 Sent Done Done false
20209 5942 Acked Write 6 Sent Done Done false
20210 5943 Acked Read 8 Sent Done Done false
20211 5944 Acked Read 1 Sent Done Done false
20212 5945 Acked Read 4 Sent Done Done false
20213 5946 Acked Flush 0 Sent Done Done false
20214 5947 Acked Write 10 Sent Done Done false
20215 5948 Acked Write 9 Sent Done Done false
20216 5949 Acked Write 1 Sent Done Done false
20217 5950 Acked Write 8 Sent Done Done false
20218 5951 Acked Read 7 Sent Done Done false
20219 5952 Acked Read 10 Sent Done Done false
20220 5953 Acked Read 5 Sent Done Done false
20221 5954 Acked Write 8 Sent Done Done false
20222 5955 Acked Flush 0 Sent Done Done false
20223 5956 Acked Read 9 Sent Done Done false
20224 5957 Acked Read 6 Sent Done Done false
20225 5958 Acked Flush 0 Sent Done Done false
20226 5959 Acked Write 4 Sent Done Done false
20227 5960 Acked Read 9 Sent Done Done false
20228 5961 Acked Write 6 Sent Done Done false
20229 5962 Acked Read 6 Sent Done Done false
20230 5963 Acked Write 1 Sent Done Done false
20231 5964 Acked Read 4 Sent Done Done false
20232 5965 Acked Read 5 Sent Done Done false
20233 5966 Acked Write 6 Sent Done Done false
20234 5967 Acked Read 1 Sent Done Done false
20235 5968 Acked Read 7 Sent Done Done false
20236 5969 Acked Write 5 Sent Done Done false
20237 5970 Acked Write 5 Sent Done Done false
20238 5971 Acked Read 4 Sent Done Done false
20239 5972 Acked Read 7 Sent Done Done false
20240 5973 Acked Read 3 Sent Done Done false
20241 5974 Acked Write 9 Sent Done Done false
20242 5975 Acked Write 3 Sent Done Done false
20243 5976 Acked Write 1 Sent Done Done false
20244 5977 Acked Read 10 Sent Done Done false
20245 5978 Acked Read 8 Sent Done Done false
20246 5979 Acked Read 4 Sent Done Done false
20247 5980 Acked Write 6 Sent Done Done false
20248 5981 Acked Write 1 Sent Done Done false
20249 5982 Acked Read 5 Sent Done Done false
20250 5983 Acked Write 5 Sent Done Done false
20251 5984 Acked Write 6 Sent Done Done false
20252 5985 Acked Read 7 Sent Done Done false
20253 5986 Acked Read 3 Sent Done Done false
20254 5987 Acked Write 9 Sent Done Done false
20255 5988 Acked Read 6 Sent Done Done false
20256 5989 Acked Read 7 Sent Done Done false
20257 5990 Acked Read 7 Sent Done Done false
20258 5991 Acked Write 6 Sent Done Done false
20259 5992 Acked Read 8 Sent Done Done false
20260 5993 Acked Read 3 Sent Done Done false
20261 5994 Acked Read 8 Sent Done Done false
20262 5995 Acked Read 7 Sent Done Done false
20263 5996 Acked Write 7 Sent Done Done false
20264 5997 Acked Write 4 Sent Done Done false
20265 5998 Acked Write 10 Sent Done Done false
20266 5999 Acked Read 6 Sent Done Done false
20267 6000 Acked Read 9 Sent Done Done false
20268 6001 Acked Write 3 Sent Done Done false
20269 6002 Acked Read 2 Sent Done Done false
20270 6003 Acked Write 10 Sent Done Done false
20271 6004 Acked Write 7 Sent Done Done false
20272 6005 Acked Write 5 Sent Done Done false
20273 6006 Acked Write 6 Sent Done Done false
20274 6007 Acked Read 7 Sent Done Done false
20275 6008 Acked Write 8 Sent Done Done false
20276 6009 Acked Read 4 Sent Done Done false
20277 6010 Acked Read 5 Sent Done Done false
20278 6011 Acked Write 6 Sent Done Done false
20279 6012 Acked Read 2 Sent Done Done false
20280 6013 Acked Read 2 Sent Done Done false
20281 6014 Acked Write 3 Sent Done Done false
20282 6015 Acked Read 10 Sent Done Done false
20283 6016 Acked Write 4 Sent Done Done false
20284 6017 Acked Write 2 Sent Done Done false
20285 6018 Acked Read 3 Sent Done Done false
20286 6019 Acked Flush 0 Sent Done Done false
20287 6020 Acked Write 9 Sent Done Done false
20288 6021 Acked Read 5 Sent Done Done false
20289 6022 Acked Read 6 Sent Done Done false
20290 6023 Acked Write 8 Sent Done Done false
20291 6024 Acked Write 1 Sent Done Done false
20292 6025 Acked Read 3 Sent Done Done false
20293 6026 Acked Read 1 Sent Done Done false
20294 6027 Acked Read 6 Sent Done Done false
20295 6028 Acked Write 9 Sent Done Done false
20296 6029 Acked Read 6 Sent Done Done false
20297 6030 Acked Read 1 Sent Done Done false
20298 6031 Acked Read 10 Sent Done Done false
20299 6032 Acked Read 10 Sent Done Done false
20300 6033 Acked Read 6 Sent Done Done false
20301 6034 Acked Write 7 Sent Done Done false
20302 6035 Acked Read 7 Sent Done Done false
20303 6036 Acked Read 1 Sent Done Done false
20304 6037 Acked Write 10 Sent Done Done false
20305 6038 Acked Read 3 Sent Done Done false
20306 6039 Acked Read 1 Sent Done Done false
20307 6040 Acked Write 4 Sent Done Done false
20308 6041 Acked Read 4 Sent Done Done false
20309 6042 Acked Write 4 Sent Done Done false
20310 6043 Acked Write 9 Sent Done Done false
20311 6044 Acked Write 10 Sent Done Done false
20312 6045 Acked Read 3 Sent Done Done false
20313 6046 Acked Read 5 Sent Done Done false
20314 6047 Acked Read 3 Sent Done Done false
20315 6048 Acked Write 1 Sent Done Done false
20316 6049 Acked Read 7 Sent Done Done false
20317 6050 Acked Read 5 Sent Done Done false
20318 6051 Acked Write 6 Sent Done Done false
20319 6052 Acked Write 8 Sent Done Done false
20320 6053 Acked Read 4 Sent Done Done false
20321 6054 Acked Write 6 Sent Done Done false
20322 6055 Acked Read 6 Sent Done Done false
20323 6056 Acked Write 1 Sent Done Done false
20324 6057 Acked Write 2 Sent Done Done false
20325 6058 Acked Write 1 Sent Done Done false
20326 6059 Acked Read 3 Sent Done Done false
20327 6060 Acked Read 5 Sent Done Done false
20328 6061 Acked Read 7 Sent Done Done false
20329 6062 Acked Read 10 Sent Done Done false
20330 6063 Acked Read 8 Sent Done Done false
20331 6064 Acked Write 10 Sent Done Done false
20332 6065 Acked Read 4 Sent Done Done false
20333 6066 Acked Read 8 Sent Done Done false
20334 6067 Acked Write 10 Sent Done Done false
20335 6068 Acked Write 1 Sent Done Done false
20336 6069 Acked Write 7 Sent Done Done false
20337 6070 Acked Write 5 Sent Done Done false
20338 6071 Acked Write 9 Sent Done Done false
20339 6072 Acked Write 10 Sent Done Done false
20340 6073 Acked Read 2 Sent Done Done false
20341 6074 Acked Write 5 Sent Done Done false
20342 6075 Acked Read 7 Sent Done Done false
20343 6076 Acked Write 6 Sent Done Done false
20344 6077 Acked Read 2 Sent Done Done false
20345 6078 Acked Write 4 Sent Done Done false
20346 6079 Acked Write 5 Sent Done Done false
20347 6080 Acked Read 5 Sent Done Done false
20348 6081 Acked Write 5 Sent Done Done false
20349 6082 Acked Write 8 Sent Done Done false
20350 6083 Acked Write 3 Sent Done Done false
20351 6084 Acked Read 1 Sent Done Done false
20352 6085 Acked Read 4 Sent Done Done false
20353 6086 Acked Read 9 Sent Done Done false
20354 6087 Acked Read 2 Sent Done Done false
20355 6088 Acked Flush 0 Sent Done Done false
20356 6089 Acked Read 4 Sent Done Done false
20357 6090 Acked Write 8 Sent Done Done false
20358 6091 Acked Read 9 Sent Done Done false
20359 6092 Acked Read 5 Sent Done Done false
20360 6093 Acked Write 2 Sent Done Done false
20361 6094 Acked Read 6 Sent Done Done false
20362 6095 Acked Read 5 Sent Done Done false
20363 6096 Acked Read 6 Sent Done Done false
20364 6097 Acked Read 5 Sent Done Done false
20365 6098 Acked Flush 0 Sent Done Done false
20366 6099 Acked Read 3 Sent Done Done false
20367 6100 Acked Read 7 Sent Done Done false
20368 6101 Acked Write 7 Sent Done Done false
20369 6102 Acked Flush 0 Sent Done Done false
20370 6103 Acked Read 10 Sent Done Done false
20371 6104 Acked Read 5 Sent Done Done false
20372 6105 Acked Flush 0 Sent Done Done false
20373 6106 Acked Read 6 Sent Done Done false
20374 6107 Acked Write 10 Sent Done Done false
20375 6108 Acked Write 5 Sent Done Done false
20376 6109 Acked Flush 0 Sent Done Done false
20377 6110 Acked Write 3 Sent Done Done false
20378 6111 Acked Read 10 Sent Done Done false
20379 6112 Acked Flush 0 Sent Done Done false
20380 6113 Acked Read 5 Sent Done Done false
20381 6114 Acked Flush 0 Sent Done Done false
20382 6115 Acked Write 8 Sent Done Done false
20383 6116 Acked Read 8 Sent Done Done false
20384 6117 Acked Write 1 Sent Done Done false
20385 6118 Acked Read 5 Sent Done Done false
20386 6119 Acked Write 2 Sent Done Done false
20387 6120 Acked Read 3 Sent Done Done false
20388 6121 Acked Read 2 Sent Done Done false
20389 6122 Acked Flush 0 Sent Done Done false
20390 6123 Acked Read 9 Sent Done Done false
20391 6124 Acked Write 2 Sent Done Done false
20392 6125 Acked Flush 0 Sent Done Done false
20393 6126 Acked Write 8 Sent Done Done false
20394 6127 Acked Write 2 Sent Done Done false
20395 6128 Acked Read 5 Sent Done Done false
20396 6129 Acked Read 5 Sent Done Done false
20397 STATES DS:0 DS:1 DS:2 TOTAL
20398 Sent 301 0 0 301
20399 Done 0 301 301 602
20400 Skipped 0 0 0 0
20401 Error 0 0 0 0
20402 Last Flush: Some(JobId(5828)) Some(JobId(6125)) Some(JobId(6125))
20403 Downstairs last five completed: 5828 5827 5826 5825 5824
20404 Upstairs last five completed: 6129 6128 6127 6126 6125
20405 CLIENT: Up:0 ds:301 act:2
204062025-02-28T06:05:28.784ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
204072025-02-28T06:05:29.882ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
204082025-02-28T06:05:30.788ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
20409 ----------------------------------------------------------------
20410 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20411 States: Offline Active Active
20412 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20413 5829 Acked Read 1 Sent Done Done false
20414 5830 Acked Write 9 Sent Done Done false
20415 5831 Acked Write 2 Sent Done Done false
20416 5832 Acked Read 3 Sent Done Done false
20417 5833 Acked Read 3 Sent Done Done false
20418 5834 Acked Read 6 Sent Done Done false
20419 5835 Acked Flush 0 Sent Done Done false
20420 5836 Acked Read 7 Sent Done Done false
20421 5837 Acked Flush 0 Sent Done Done false
20422 5838 Acked Write 10 Sent Done Done false
20423 5839 Acked Read 5 Sent Done Done false
20424 5840 Acked Read 3 Sent Done Done false
20425 5841 Acked Write 5 Sent Done Done false
20426 5842 Acked Read 2 Sent Done Done false
20427 5843 Acked Write 5 Sent Done Done false
20428 5844 Acked Read 2 Sent Done Done false
20429 5845 Acked Read 4 Sent Done Done false
20430 5846 Acked Read 4 Sent Done Done false
20431 5847 Acked Write 3 Sent Done Done false
20432 5848 Acked Write 10 Sent Done Done false
20433 5849 Acked Read 2 Sent Done Done false
20434 5850 Acked Write 2 Sent Done Done false
20435 5851 Acked Write 4 Sent Done Done false
20436 5852 Acked Read 2 Sent Done Done false
20437 5853 Acked Write 10 Sent Done Done false
20438 5854 Acked Write 5 Sent Done Done false
20439 5855 Acked Read 2 Sent Done Done false
20440 5856 Acked Read 5 Sent Done Done false
20441 5857 Acked Read 5 Sent Done Done false
20442 5858 Acked Read 10 Sent Done Done false
20443 5859 Acked Read 3 Sent Done Done false
20444 5860 Acked Write 10 Sent Done Done false
20445 5861 Acked Write 8 Sent Done Done false
20446 5862 Acked Read 8 Sent Done Done false
20447 5863 Acked Write 10 Sent Done Done false
20448 5864 Acked Read 1 Sent Done Done false
20449 5865 Acked Write 4 Sent Done Done false
20450 5866 Acked Write 9 Sent Done Done false
20451 5867 Acked Read 9 Sent Done Done false
20452 5868 Acked Read 6 Sent Done Done false
20453 5869 Acked Flush 0 Sent Done Done false
20454 5870 Acked Flush 0 Sent Done Done false
20455 5871 Acked Flush 0 Sent Done Done false
20456 5872 Acked Read 6 Sent Done Done false
20457 5873 Acked Read 5 Sent Done Done false
20458 5874 Acked Write 10 Sent Done Done false
20459 5875 Acked Read 9 Sent Done Done false
20460 5876 Acked Read 7 Sent Done Done false
20461 5877 Acked Write 7 Sent Done Done false
20462 5878 Acked Write 8 Sent Done Done false
20463 5879 Acked Write 1 Sent Done Done false
20464 5880 Acked Write 8 Sent Done Done false
20465 5881 Acked Write 1 Sent Done Done false
20466 5882 Acked Write 3 Sent Done Done false
20467 5883 Acked Flush 0 Sent Done Done false
20468 5884 Acked Read 5 Sent Done Done false
20469 5885 Acked Read 2 Sent Done Done false
20470 5886 Acked Read 2 Sent Done Done false
20471 5887 Acked Read 10 Sent Done Done false
20472 5888 Acked Flush 0 Sent Done Done false
20473 5889 Acked Read 8 Sent Done Done false
20474 5890 Acked Read 3 Sent Done Done false
20475 5891 Acked Write 9 Sent Done Done false
20476 5892 Acked Read 5 Sent Done Done false
20477 5893 Acked Read 6 Sent Done Done false
20478 5894 Acked Write 2 Sent Done Done false
20479 5895 Acked Read 7 Sent Done Done false
20480 5896 Acked Write 4 Sent Done Done false
20481 5897 Acked Flush 0 Sent Done Done false
20482 5898 Acked Read 9 Sent Done Done false
20483 5899 Acked Flush 0 Sent Done Done false
20484 5900 Acked Read 1 Sent Done Done false
20485 5901 Acked Read 8 Sent Done Done false
20486 5902 Acked Flush 0 Sent Done Done false
20487 5903 Acked Read 4 Sent Done Done false
20488 5904 Acked Write 9 Sent Done Done false
20489 5905 Acked Read 3 Sent Done Done false
20490 5906 Acked Read 9 Sent Done Done false
20491 5907 Acked Write 9 Sent Done Done false
20492 5908 Acked Flush 0 Sent Done Done false
20493 5909 Acked Write 5 Sent Done Done false
20494 5910 Acked Write 8 Sent Done Done false
20495 5911 Acked Read 9 Sent Done Done false
20496 5912 Acked Write 6 Sent Done Done false
20497 5913 Acked Write 5 Sent Done Done false
20498 5914 Acked Read 5 Sent Done Done false
20499 5915 Acked Read 7 Sent Done Done false
20500 5916 Acked Write 3 Sent Done Done false
20501 5917 Acked Write 1 Sent Done Done false
20502 5918 Acked Flush 0 Sent Done Done false
20503 5919 Acked Read 3 Sent Done Done false
20504 5920 Acked Write 4 Sent Done Done false
20505 5921 Acked Flush 0 Sent Done Done false
20506 5922 Acked Read 9 Sent Done Done false
20507 5923 Acked Read 8 Sent Done Done false
20508 5924 Acked Write 6 Sent Done Done false
20509 5925 Acked Flush 0 Sent Done Done false
20510 5926 Acked Flush 0 Sent Done Done false
20511 5927 Acked Write 1 Sent Done Done false
20512 5928 Acked Write 2 Sent Done Done false
20513 5929 Acked Flush 0 Sent Done Done false
20514 5930 Acked Write 3 Sent Done Done false
20515 5931 Acked Read 7 Sent Done Done false
20516 5932 Acked Write 3 Sent Done Done false
20517 5933 Acked Write 6 Sent Done Done false
20518 5934 Acked Read 10 Sent Done Done false
20519 5935 Acked Write 5 Sent Done Done false
20520 5936 Acked Write 8 Sent Done Done false
20521 5937 Acked Write 3 Sent Done Done false
20522 5938 Acked Read 8 Sent Done Done false
20523 5939 Acked Read 7 Sent Done Done false
20524 5940 Acked Write 2 Sent Done Done false
20525 5941 Acked Write 5 Sent Done Done false
20526 5942 Acked Write 6 Sent Done Done false
20527 5943 Acked Read 8 Sent Done Done false
20528 5944 Acked Read 1 Sent Done Done false
20529 5945 Acked Read 4 Sent Done Done false
20530 5946 Acked Flush 0 Sent Done Done false
20531 5947 Acked Write 10 Sent Done Done false
20532 5948 Acked Write 9 Sent Done Done false
20533 5949 Acked Write 1 Sent Done Done false
20534 5950 Acked Write 8 Sent Done Done false
20535 5951 Acked Read 7 Sent Done Done false
20536 5952 Acked Read 10 Sent Done Done false
20537 5953 Acked Read 5 Sent Done Done false
20538 5954 Acked Write 8 Sent Done Done false
20539 5955 Acked Flush 0 Sent Done Done false
20540 5956 Acked Read 9 Sent Done Done false
20541 5957 Acked Read 6 Sent Done Done false
20542 5958 Acked Flush 0 Sent Done Done false
20543 5959 Acked Write 4 Sent Done Done false
20544 5960 Acked Read 9 Sent Done Done false
20545 5961 Acked Write 6 Sent Done Done false
20546 5962 Acked Read 6 Sent Done Done false
20547 5963 Acked Write 1 Sent Done Done false
20548 5964 Acked Read 4 Sent Done Done false
20549 5965 Acked Read 5 Sent Done Done false
20550 5966 Acked Write 6 Sent Done Done false
20551 5967 Acked Read 1 Sent Done Done false
20552 5968 Acked Read 7 Sent Done Done false
20553 5969 Acked Write 5 Sent Done Done false
20554 5970 Acked Write 5 Sent Done Done false
20555 5971 Acked Read 4 Sent Done Done false
20556 5972 Acked Read 7 Sent Done Done false
20557 5973 Acked Read 3 Sent Done Done false
20558 5974 Acked Write 9 Sent Done Done false
20559 5975 Acked Write 3 Sent Done Done false
20560 5976 Acked Write 1 Sent Done Done false
20561 5977 Acked Read 10 Sent Done Done false
20562 5978 Acked Read 8 Sent Done Done false
20563 5979 Acked Read 4 Sent Done Done false
20564 5980 Acked Write 6 Sent Done Done false
20565 5981 Acked Write 1 Sent Done Done false
20566 5982 Acked Read 5 Sent Done Done false
20567 5983 Acked Write 5 Sent Done Done false
20568 5984 Acked Write 6 Sent Done Done false
20569 5985 Acked Read 7 Sent Done Done false
20570 5986 Acked Read 3 Sent Done Done false
20571 5987 Acked Write 9 Sent Done Done false
20572 5988 Acked Read 6 Sent Done Done false
20573 5989 Acked Read 7 Sent Done Done false
20574 5990 Acked Read 7 Sent Done Done false
20575 5991 Acked Write 6 Sent Done Done false
20576 5992 Acked Read 8 Sent Done Done false
20577 5993 Acked Read 3 Sent Done Done false
20578 5994 Acked Read 8 Sent Done Done false
20579 5995 Acked Read 7 Sent Done Done false
20580 5996 Acked Write 7 Sent Done Done false
20581 5997 Acked Write 4 Sent Done Done false
20582 5998 Acked Write 10 Sent Done Done false
20583 5999 Acked Read 6 Sent Done Done false
20584 6000 Acked Read 9 Sent Done Done false
20585 6001 Acked Write 3 Sent Done Done false
20586 6002 Acked Read 2 Sent Done Done false
20587 6003 Acked Write 10 Sent Done Done false
20588 6004 Acked Write 7 Sent Done Done false
20589 6005 Acked Write 5 Sent Done Done false
20590 6006 Acked Write 6 Sent Done Done false
20591 6007 Acked Read 7 Sent Done Done false
20592 6008 Acked Write 8 Sent Done Done false
20593 6009 Acked Read 4 Sent Done Done false
20594 6010 Acked Read 5 Sent Done Done false
20595 6011 Acked Write 6 Sent Done Done false
20596 6012 Acked Read 2 Sent Done Done false
20597 6013 Acked Read 2 Sent Done Done false
20598 6014 Acked Write 3 Sent Done Done false
20599 6015 Acked Read 10 Sent Done Done false
20600 6016 Acked Write 4 Sent Done Done false
20601 6017 Acked Write 2 Sent Done Done false
20602 6018 Acked Read 3 Sent Done Done false
20603 6019 Acked Flush 0 Sent Done Done false
20604 6020 Acked Write 9 Sent Done Done false
20605 6021 Acked Read 5 Sent Done Done false
20606 6022 Acked Read 6 Sent Done Done false
20607 6023 Acked Write 8 Sent Done Done false
20608 6024 Acked Write 1 Sent Done Done false
20609 6025 Acked Read 3 Sent Done Done false
20610 6026 Acked Read 1 Sent Done Done false
20611 6027 Acked Read 6 Sent Done Done false
20612 6028 Acked Write 9 Sent Done Done false
20613 6029 Acked Read 6 Sent Done Done false
20614 6030 Acked Read 1 Sent Done Done false
20615 6031 Acked Read 10 Sent Done Done false
20616 6032 Acked Read 10 Sent Done Done false
20617 6033 Acked Read 6 Sent Done Done false
20618 6034 Acked Write 7 Sent Done Done false
20619 6035 Acked Read 7 Sent Done Done false
20620 6036 Acked Read 1 Sent Done Done false
20621 6037 Acked Write 10 Sent Done Done false
20622 6038 Acked Read 3 Sent Done Done false
20623 6039 Acked Read 1 Sent Done Done false
20624 6040 Acked Write 4 Sent Done Done false
20625 6041 Acked Read 4 Sent Done Done false
20626 6042 Acked Write 4 Sent Done Done false
20627 6043 Acked Write 9 Sent Done Done false
20628 6044 Acked Write 10 Sent Done Done false
20629 6045 Acked Read 3 Sent Done Done false
20630 6046 Acked Read 5 Sent Done Done false
20631 6047 Acked Read 3 Sent Done Done false
20632 6048 Acked Write 1 Sent Done Done false
20633 6049 Acked Read 7 Sent Done Done false
20634 6050 Acked Read 5 Sent Done Done false
20635 6051 Acked Write 6 Sent Done Done false
20636 6052 Acked Write 8 Sent Done Done false
20637 6053 Acked Read 4 Sent Done Done false
20638 6054 Acked Write 6 Sent Done Done false
20639 6055 Acked Read 6 Sent Done Done false
20640 6056 Acked Write 1 Sent Done Done false
20641 6057 Acked Write 2 Sent Done Done false
20642 6058 Acked Write 1 Sent Done Done false
20643 6059 Acked Read 3 Sent Done Done false
20644 6060 Acked Read 5 Sent Done Done false
20645 6061 Acked Read 7 Sent Done Done false
20646 6062 Acked Read 10 Sent Done Done false
20647 6063 Acked Read 8 Sent Done Done false
20648 6064 Acked Write 10 Sent Done Done false
20649 6065 Acked Read 4 Sent Done Done false
20650 6066 Acked Read 8 Sent Done Done false
20651 6067 Acked Write 10 Sent Done Done false
20652 6068 Acked Write 1 Sent Done Done false
20653 6069 Acked Write 7 Sent Done Done false
20654 6070 Acked Write 5 Sent Done Done false
20655 6071 Acked Write 9 Sent Done Done false
20656 6072 Acked Write 10 Sent Done Done false
20657 6073 Acked Read 2 Sent Done Done false
20658 6074 Acked Write 5 Sent Done Done false
20659 6075 Acked Read 7 Sent Done Done false
20660 6076 Acked Write 6 Sent Done Done false
20661 6077 Acked Read 2 Sent Done Done false
20662 6078 Acked Write 4 Sent Done Done false
20663 6079 Acked Write 5 Sent Done Done false
20664 6080 Acked Read 5 Sent Done Done false
20665 6081 Acked Write 5 Sent Done Done false
20666 6082 Acked Write 8 Sent Done Done false
20667 6083 Acked Write 3 Sent Done Done false
20668 6084 Acked Read 1 Sent Done Done false
20669 6085 Acked Read 4 Sent Done Done false
20670 6086 Acked Read 9 Sent Done Done false
20671 6087 Acked Read 2 Sent Done Done false
20672 6088 Acked Flush 0 Sent Done Done false
20673 6089 Acked Read 4 Sent Done Done false
20674 6090 Acked Write 8 Sent Done Done false
20675 6091 Acked Read 9 Sent Done Done false
20676 6092 Acked Read 5 Sent Done Done false
20677 6093 Acked Write 2 Sent Done Done false
20678 6094 Acked Read 6 Sent Done Done false
20679 6095 Acked Read 5 Sent Done Done false
20680 6096 Acked Read 6 Sent Done Done false
20681 6097 Acked Read 5 Sent Done Done false
20682 6098 Acked Flush 0 Sent Done Done false
20683 6099 Acked Read 3 Sent Done Done false
20684 6100 Acked Read 7 Sent Done Done false
20685 6101 Acked Write 7 Sent Done Done false
20686 6102 Acked Flush 0 Sent Done Done false
20687 6103 Acked Read 10 Sent Done Done false
20688 6104 Acked Read 5 Sent Done Done false
20689 6105 Acked Flush 0 Sent Done Done false
20690 6106 Acked Read 6 Sent Done Done false
20691 6107 Acked Write 10 Sent Done Done false
20692 6108 Acked Write 5 Sent Done Done false
20693 6109 Acked Flush 0 Sent Done Done false
20694 6110 Acked Write 3 Sent Done Done false
20695 6111 Acked Read 10 Sent Done Done false
20696 6112 Acked Flush 0 Sent Done Done false
20697 6113 Acked Read 5 Sent Done Done false
20698 6114 Acked Flush 0 Sent Done Done false
20699 6115 Acked Write 8 Sent Done Done false
20700 6116 Acked Read 8 Sent Done Done false
20701 6117 Acked Write 1 Sent Done Done false
20702 6118 Acked Read 5 Sent Done Done false
20703 6119 Acked Write 2 Sent Done Done false
20704 6120 Acked Read 3 Sent Done Done false
20705 6121 Acked Read 2 Sent Done Done false
20706 6122 Acked Flush 0 Sent Done Done false
20707 6123 Acked Read 9 Sent Done Done false
20708 6124 Acked Write 2 Sent Done Done false
20709 6125 Acked Flush 0 Sent Done Done false
20710 6126 Acked Write 8 Sent Done Done false
20711 6127 Acked Write 2 Sent Done Done false
20712 6128 Acked Read 5 Sent Done Done false
20713 6129 Acked Read 5 Sent Done Done false
20714 STATES DS:0 DS:1 DS:2 TOTAL
20715 Sent 301 0 0 301
20716 Done 0 301 301 602
20717 Skipped 0 0 0 0
20718 Error 0 0 0 0
20719 Last Flush: Some(JobId(5828)) Some(JobId(6125)) Some(JobId(6125))
20720 Downstairs last five completed: 5828 5827 5826 5825 5824
20721 Upstairs last five completed: 6129 6128 6127 6126 6125
20722 CLIENT: Up:0 ds:301 act:2
207232025-02-28T06:05:32.792ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
207242025-02-28T06:05:33.793ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
207252025-02-28T06:05:33.883ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
207262025-02-28T06:05:33.883ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
207272025-02-28T06:05:33.883ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
207282025-02-28T06:05:33.883ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
207292025-02-28T06:05:33.883ZINFOcrucible: client task is exiting = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
207302025-02-28T06:05:34.884ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
207312025-02-28T06:05:35.797ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
20732 [8810] got start action
20733 [8810] I am going to restart
20734 Starting downstairs at port 8810
20735 Make output file at "/tmp/dsc/downstairs-8810.txt"
20736 [8810][0] reports Starting
20737 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1228)
20738 [8810][0] reports Running
20739 ----------------------------------------------------------------
20740 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20741 States: Offline Active Active
20742 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20743 5829 Acked Read 1 Sent Done Done false
20744 5830 Acked Write 9 Sent Done Done false
20745 5831 Acked Write 2 Sent Done Done false
20746 5832 Acked Read 3 Sent Done Done false
20747 5833 Acked Read 3 Sent Done Done false
20748 5834 Acked Read 6 Sent Done Done false
20749 5835 Acked Flush 0 Sent Done Done false
20750 5836 Acked Read 7 Sent Done Done false
20751 5837 Acked Flush 0 Sent Done Done false
20752 5838 Acked Write 10 Sent Done Done false
20753 5839 Acked Read 5 Sent Done Done false
20754 5840 Acked Read 3 Sent Done Done false
20755 5841 Acked Write 5 Sent Done Done false
20756 5842 Acked Read 2 Sent Done Done false
20757 5843 Acked Write 5 Sent Done Done false
20758 5844 Acked Read 2 Sent Done Done false
20759 5845 Acked Read 4 Sent Done Done false
20760 5846 Acked Read 4 Sent Done Done false
20761 5847 Acked Write 3 Sent Done Done false
20762 5848 Acked Write 10 Sent Done Done false
20763 5849 Acked Read 2 Sent Done Done false
20764 5850 Acked Write 2 Sent Done Done false
20765 5851 Acked Write 4 Sent Done Done false
20766 5852 Acked Read 2 Sent Done Done false
20767 5853 Acked Write 10 Sent Done Done false
20768 5854 Acked Write 5 Sent Done Done false
20769 5855 Acked Read 2 Sent Done Done false
20770 5856 Acked Read 5 Sent Done Done false
20771 5857 Acked Read 5 Sent Done Done false
20772 5858 Acked Read 10 Sent Done Done false
20773 5859 Acked Read 3 Sent Done Done false
20774 5860 Acked Write 10 Sent Done Done false
20775 5861 Acked Write 8 Sent Done Done false
20776 5862 Acked Read 8 Sent Done Done false
20777 5863 Acked Write 10 Sent Done Done false
20778 5864 Acked Read 1 Sent Done Done false
20779 5865 Acked Write 4 Sent Done Done false
20780 5866 Acked Write 9 Sent Done Done false
20781 5867 Acked Read 9 Sent Done Done false
20782 5868 Acked Read 6 Sent Done Done false
20783 5869 Acked Flush 0 Sent Done Done false
20784 5870 Acked Flush 0 Sent Done Done false
20785 5871 Acked Flush 0 Sent Done Done false
20786 5872 Acked Read 6 Sent Done Done false
20787 5873 Acked Read 5 Sent Done Done false
20788 5874 Acked Write 10 Sent Done Done false
20789 5875 Acked Read 9 Sent Done Done false
20790 5876 Acked Read 7 Sent Done Done false
20791 5877 Acked Write 7 Sent Done Done false
20792 5878 Acked Write 8 Sent Done Done false
20793 5879 Acked Write 1 Sent Done Done false
20794 5880 Acked Write 8 Sent Done Done false
20795 5881 Acked Write 1 Sent Done Done false
20796 5882 Acked Write 3 Sent Done Done false
20797 5883 Acked Flush 0 Sent Done Done false
20798 5884 Acked Read 5 Sent Done Done false
20799 5885 Acked Read 2 Sent Done Done false
20800 5886 Acked Read 2 Sent Done Done false
20801 5887 Acked Read 10 Sent Done Done false
20802 5888 Acked Flush 0 Sent Done Done false
20803 5889 Acked Read 8 Sent Done Done false
20804 5890 Acked Read 3 Sent Done Done false
20805 5891 Acked Write 9 Sent Done Done false
20806 5892 Acked Read 5 Sent Done Done false
20807 5893 Acked Read 6 Sent Done Done false
20808 5894 Acked Write 2 Sent Done Done false
20809 5895 Acked Read 7 Sent Done Done false
20810 5896 Acked Write 4 Sent Done Done false
20811 5897 Acked Flush 0 Sent Done Done false
20812 5898 Acked Read 9 Sent Done Done false
20813 5899 Acked Flush 0 Sent Done Done false
20814 5900 Acked Read 1 Sent Done Done false
20815 5901 Acked Read 8 Sent Done Done false
20816 5902 Acked Flush 0 Sent Done Done false
20817 5903 Acked Read 4 Sent Done Done false
20818 5904 Acked Write 9 Sent Done Done false
20819 5905 Acked Read 3 Sent Done Done false
20820 5906 Acked Read 9 Sent Done Done false
20821 5907 Acked Write 9 Sent Done Done false
20822 5908 Acked Flush 0 Sent Done Done false
20823 5909 Acked Write 5 Sent Done Done false
20824 5910 Acked Write 8 Sent Done Done false
20825 5911 Acked Read 9 Sent Done Done false
20826 5912 Acked Write 6 Sent Done Done false
20827 5913 Acked Write 5 Sent Done Done false
20828 5914 Acked Read 5 Sent Done Done false
20829 5915 Acked Read 7 Sent Done Done false
20830 5916 Acked Write 3 Sent Done Done false
20831 5917 Acked Write 1 Sent Done Done false
20832 5918 Acked Flush 0 Sent Done Done false
20833 5919 Acked Read 3 Sent Done Done false
20834 5920 Acked Write 4 Sent Done Done false
20835 5921 Acked Flush 0 Sent Done Done false
20836 5922 Acked Read 9 Sent Done Done false
20837 5923 Acked Read 8 Sent Done Done false
20838 5924 Acked Write 6 Sent Done Done false
20839 5925 Acked Flush 0 Sent Done Done false
20840 5926 Acked Flush 0 Sent Done Done false
20841 5927 Acked Write 1 Sent Done Done false
20842 5928 Acked Write 2 Sent Done Done false
20843 5929 Acked Flush 0 Sent Done Done false
20844 5930 Acked Write 3 Sent Done Done false
20845 5931 Acked Read 7 Sent Done Done false
20846 5932 Acked Write 3 Sent Done Done false
20847 5933 Acked Write 6 Sent Done Done false
20848 5934 Acked Read 10 Sent Done Done false
20849 5935 Acked Write 5 Sent Done Done false
20850 5936 Acked Write 8 Sent Done Done false
20851 5937 Acked Write 3 Sent Done Done false
20852 5938 Acked Read 8 Sent Done Done false
20853 5939 Acked Read 7 Sent Done Done false
20854 5940 Acked Write 2 Sent Done Done false
20855 5941 Acked Write 5 Sent Done Done false
20856 5942 Acked Write 6 Sent Done Done false
20857 5943 Acked Read 8 Sent Done Done false
20858 5944 Acked Read 1 Sent Done Done false
20859 5945 Acked Read 4 Sent Done Done false
20860 5946 Acked Flush 0 Sent Done Done false
20861 5947 Acked Write 10 Sent Done Done false
20862 5948 Acked Write 9 Sent Done Done false
20863 5949 Acked Write 1 Sent Done Done false
20864 5950 Acked Write 8 Sent Done Done false
20865 5951 Acked Read 7 Sent Done Done false
20866 5952 Acked Read 10 Sent Done Done false
20867 5953 Acked Read 5 Sent Done Done false
20868 5954 Acked Write 8 Sent Done Done false
20869 5955 Acked Flush 0 Sent Done Done false
20870 5956 Acked Read 9 Sent Done Done false
20871 5957 Acked Read 6 Sent Done Done false
20872 5958 Acked Flush 0 Sent Done Done false
20873 5959 Acked Write 4 Sent Done Done false
20874 5960 Acked Read 9 Sent Done Done false
20875 5961 Acked Write 6 Sent Done Done false
20876 5962 Acked Read 6 Sent Done Done false
20877 5963 Acked Write 1 Sent Done Done false
20878 5964 Acked Read 4 Sent Done Done false
20879 5965 Acked Read 5 Sent Done Done false
20880 5966 Acked Write 6 Sent Done Done false
20881 5967 Acked Read 1 Sent Done Done false
20882 5968 Acked Read 7 Sent Done Done false
20883 5969 Acked Write 5 Sent Done Done false
20884 5970 Acked Write 5 Sent Done Done false
20885 5971 Acked Read 4 Sent Done Done false
20886 5972 Acked Read 7 Sent Done Done false
20887 5973 Acked Read 3 Sent Done Done false
20888 5974 Acked Write 9 Sent Done Done false
20889 5975 Acked Write 3 Sent Done Done false
20890 5976 Acked Write 1 Sent Done Done false
20891 5977 Acked Read 10 Sent Done Done false
20892 5978 Acked Read 8 Sent Done Done false
20893 5979 Acked Read 4 Sent Done Done false
20894 5980 Acked Write 6 Sent Done Done false
20895 5981 Acked Write 1 Sent Done Done false
20896 5982 Acked Read 5 Sent Done Done false
20897 5983 Acked Write 5 Sent Done Done false
20898 5984 Acked Write 6 Sent Done Done false
20899 5985 Acked Read 7 Sent Done Done false
20900 5986 Acked Read 3 Sent Done Done false
20901 5987 Acked Write 9 Sent Done Done false
20902 5988 Acked Read 6 Sent Done Done false
20903 5989 Acked Read 7 Sent Done Done false
20904 5990 Acked Read 7 Sent Done Done false
20905 5991 Acked Write 6 Sent Done Done false
20906 5992 Acked Read 8 Sent Done Done false
20907 5993 Acked Read 3 Sent Done Done false
20908 5994 Acked Read 8 Sent Done Done false
20909 5995 Acked Read 7 Sent Done Done false
20910 5996 Acked Write 7 Sent Done Done false
20911 5997 Acked Write 4 Sent Done Done false
20912 5998 Acked Write 10 Sent Done Done false
20913 5999 Acked Read 6 Sent Done Done false
20914 6000 Acked Read 9 Sent Done Done false
20915 6001 Acked Write 3 Sent Done Done false
20916 6002 Acked Read 2 Sent Done Done false
20917 6003 Acked Write 10 Sent Done Done false
20918 6004 Acked Write 7 Sent Done Done false
20919 6005 Acked Write 5 Sent Done Done false
20920 6006 Acked Write 6 Sent Done Done false
20921 6007 Acked Read 7 Sent Done Done false
20922 6008 Acked Write 8 Sent Done Done false
20923 6009 Acked Read 4 Sent Done Done false
20924 6010 Acked Read 5 Sent Done Done false
20925 6011 Acked Write 6 Sent Done Done false
20926 6012 Acked Read 2 Sent Done Done false
20927 6013 Acked Read 2 Sent Done Done false
20928 6014 Acked Write 3 Sent Done Done false
20929 6015 Acked Read 10 Sent Done Done false
20930 6016 Acked Write 4 Sent Done Done false
20931 6017 Acked Write 2 Sent Done Done false
20932 6018 Acked Read 3 Sent Done Done false
20933 6019 Acked Flush 0 Sent Done Done false
20934 6020 Acked Write 9 Sent Done Done false
20935 6021 Acked Read 5 Sent Done Done false
20936 6022 Acked Read 6 Sent Done Done false
20937 6023 Acked Write 8 Sent Done Done false
20938 6024 Acked Write 1 Sent Done Done false
20939 6025 Acked Read 3 Sent Done Done false
20940 6026 Acked Read 1 Sent Done Done false
20941 6027 Acked Read 6 Sent Done Done false
20942 6028 Acked Write 9 Sent Done Done false
20943 6029 Acked Read 6 Sent Done Done false
20944 6030 Acked Read 1 Sent Done Done false
20945 6031 Acked Read 10 Sent Done Done false
20946 6032 Acked Read 10 Sent Done Done false
20947 6033 Acked Read 6 Sent Done Done false
20948 6034 Acked Write 7 Sent Done Done false
20949 6035 Acked Read 7 Sent Done Done false
20950 6036 Acked Read 1 Sent Done Done false
20951 6037 Acked Write 10 Sent Done Done false
20952 6038 Acked Read 3 Sent Done Done false
20953 6039 Acked Read 1 Sent Done Done false
20954 6040 Acked Write 4 Sent Done Done false
20955 6041 Acked Read 4 Sent Done Done false
20956 6042 Acked Write 4 Sent Done Done false
20957 6043 Acked Write 9 Sent Done Done false
20958 6044 Acked Write 10 Sent Done Done false
20959 6045 Acked Read 3 Sent Done Done false
20960 6046 Acked Read 5 Sent Done Done false
20961 6047 Acked Read 3 Sent Done Done false
20962 6048 Acked Write 1 Sent Done Done false
20963 6049 Acked Read 7 Sent Done Done false
20964 6050 Acked Read 5 Sent Done Done false
20965 6051 Acked Write 6 Sent Done Done false
20966 6052 Acked Write 8 Sent Done Done false
20967 6053 Acked Read 4 Sent Done Done false
20968 6054 Acked Write 6 Sent Done Done false
20969 6055 Acked Read 6 Sent Done Done false
20970 6056 Acked Write 1 Sent Done Done false
20971 6057 Acked Write 2 Sent Done Done false
20972 6058 Acked Write 1 Sent Done Done false
20973 6059 Acked Read 3 Sent Done Done false
20974 6060 Acked Read 5 Sent Done Done false
20975 6061 Acked Read 7 Sent Done Done false
20976 6062 Acked Read 10 Sent Done Done false
20977 6063 Acked Read 8 Sent Done Done false
20978 6064 Acked Write 10 Sent Done Done false
20979 6065 Acked Read 4 Sent Done Done false
20980 6066 Acked Read 8 Sent Done Done false
20981 6067 Acked Write 10 Sent Done Done false
20982 6068 Acked Write 1 Sent Done Done false
20983 6069 Acked Write 7 Sent Done Done false
20984 6070 Acked Write 5 Sent Done Done false
20985 6071 Acked Write 9 Sent Done Done false
20986 6072 Acked Write 10 Sent Done Done false
20987 6073 Acked Read 2 Sent Done Done false
20988 6074 Acked Write 5 Sent Done Done false
20989 6075 Acked Read 7 Sent Done Done false
20990 6076 Acked Write 6 Sent Done Done false
20991 6077 Acked Read 2 Sent Done Done false
20992 6078 Acked Write 4 Sent Done Done false
20993 6079 Acked Write 5 Sent Done Done false
20994 6080 Acked Read 5 Sent Done Done false
20995 6081 Acked Write 5 Sent Done Done false
20996 6082 Acked Write 8 Sent Done Done false
20997 6083 Acked Write 3 Sent Done Done false
20998 6084 Acked Read 1 Sent Done Done false
20999 6085 Acked Read 4 Sent Done Done false
21000 6086 Acked Read 9 Sent Done Done false
21001 6087 Acked Read 2 Sent Done Done false
21002 6088 Acked Flush 0 Sent Done Done false
21003 6089 Acked Read 4 Sent Done Done false
21004 6090 Acked Write 8 Sent Done Done false
21005 6091 Acked Read 9 Sent Done Done false
21006 6092 Acked Read 5 Sent Done Done false
21007 6093 Acked Write 2 Sent Done Done false
21008 6094 Acked Read 6 Sent Done Done false
21009 6095 Acked Read 5 Sent Done Done false
21010 6096 Acked Read 6 Sent Done Done false
21011 6097 Acked Read 5 Sent Done Done false
21012 6098 Acked Flush 0 Sent Done Done false
21013 6099 Acked Read 3 Sent Done Done false
21014 6100 Acked Read 7 Sent Done Done false
21015 6101 Acked Write 7 Sent Done Done false
21016 6102 Acked Flush 0 Sent Done Done false
21017 6103 Acked Read 10 Sent Done Done false
21018 6104 Acked Read 5 Sent Done Done false
21019 6105 Acked Flush 0 Sent Done Done false
21020 6106 Acked Read 6 Sent Done Done false
21021 6107 Acked Write 10 Sent Done Done false
21022 6108 Acked Write 5 Sent Done Done false
21023 6109 Acked Flush 0 Sent Done Done false
21024 6110 Acked Write 3 Sent Done Done false
21025 6111 Acked Read 10 Sent Done Done false
21026 6112 Acked Flush 0 Sent Done Done false
21027 6113 Acked Read 5 Sent Done Done false
21028 6114 Acked Flush 0 Sent Done Done false
21029 6115 Acked Write 8 Sent Done Done false
21030 6116 Acked Read 8 Sent Done Done false
21031 6117 Acked Write 1 Sent Done Done false
21032 6118 Acked Read 5 Sent Done Done false
21033 6119 Acked Write 2 Sent Done Done false
21034 6120 Acked Read 3 Sent Done Done false
21035 6121 Acked Read 2 Sent Done Done false
21036 6122 Acked Flush 0 Sent Done Done false
21037 6123 Acked Read 9 Sent Done Done false
21038 6124 Acked Write 2 Sent Done Done false
21039 6125 Acked Flush 0 Sent Done Done false
21040 6126 Acked Write 8 Sent Done Done false
21041 6127 Acked Write 2 Sent Done Done false
21042 6128 Acked Read 5 Sent Done Done false
21043 6129 Acked Read 5 Sent Done Done false
21044 STATES DS:0 DS:1 DS:2 TOTAL
21045 Sent 301 0 0 301
21046 Done 0 301 301 602
21047 Skipped 0 0 0 0
21048 Error 0 0 0 0
21049 Last Flush: Some(JobId(5828)) Some(JobId(6125)) Some(JobId(6125))
21050 Downstairs last five completed: 5828 5827 5826 5825 5824
21051 Upstairs last five completed: 6129 6128 6127 6126 6125
21052 CLIENT: Up:0 ds:301 act:2
210532025-02-28T06:05:37.130ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
210542025-02-28T06:05:37.801ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
210552025-02-28T06:05:38.803ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
210562025-02-28T06:05:39.885ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
21057 ----------------------------------------------------------------
21058 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21059 States: Offline Active Active
21060 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21061 5829 Acked Read 1 Sent Done Done false
21062 5830 Acked Write 9 Sent Done Done false
21063 5831 Acked Write 2 Sent Done Done false
21064 5832 Acked Read 3 Sent Done Done false
21065 5833 Acked Read 3 Sent Done Done false
21066 5834 Acked Read 6 Sent Done Done false
21067 5835 Acked Flush 0 Sent Done Done false
21068 5836 Acked Read 7 Sent Done Done false
21069 5837 Acked Flush 0 Sent Done Done false
21070 5838 Acked Write 10 Sent Done Done false
21071 5839 Acked Read 5 Sent Done Done false
21072 5840 Acked Read 3 Sent Done Done false
21073 5841 Acked Write 5 Sent Done Done false
21074 5842 Acked Read 2 Sent Done Done false
21075 5843 Acked Write 5 Sent Done Done false
21076 5844 Acked Read 2 Sent Done Done false
21077 5845 Acked Read 4 Sent Done Done false
21078 5846 Acked Read 4 Sent Done Done false
21079 5847 Acked Write 3 Sent Done Done false
21080 5848 Acked Write 10 Sent Done Done false
21081 5849 Acked Read 2 Sent Done Done false
21082 5850 Acked Write 2 Sent Done Done false
21083 5851 Acked Write 4 Sent Done Done false
21084 5852 Acked Read 2 Sent Done Done false
21085 5853 Acked Write 10 Sent Done Done false
21086 5854 Acked Write 5 Sent Done Done false
21087 5855 Acked Read 2 Sent Done Done false
21088 5856 Acked Read 5 Sent Done Done false
21089 5857 Acked Read 5 Sent Done Done false
21090 5858 Acked Read 10 Sent Done Done false
21091 5859 Acked Read 3 Sent Done Done false
21092 5860 Acked Write 10 Sent Done Done false
21093 5861 Acked Write 8 Sent Done Done false
21094 5862 Acked Read 8 Sent Done Done false
21095 5863 Acked Write 10 Sent Done Done false
21096 5864 Acked Read 1 Sent Done Done false
21097 5865 Acked Write 4 Sent Done Done false
21098 5866 Acked Write 9 Sent Done Done false
21099 5867 Acked Read 9 Sent Done Done false
21100 5868 Acked Read 6 Sent Done Done false
21101 5869 Acked Flush 0 Sent Done Done false
21102 5870 Acked Flush 0 Sent Done Done false
21103 5871 Acked Flush 0 Sent Done Done false
21104 5872 Acked Read 6 Sent Done Done false
21105 5873 Acked Read 5 Sent Done Done false
21106 5874 Acked Write 10 Sent Done Done false
21107 5875 Acked Read 9 Sent Done Done false
21108 5876 Acked Read 7 Sent Done Done false
21109 5877 Acked Write 7 Sent Done Done false
21110 5878 Acked Write 8 Sent Done Done false
21111 5879 Acked Write 1 Sent Done Done false
21112 5880 Acked Write 8 Sent Done Done false
21113 5881 Acked Write 1 Sent Done Done false
21114 5882 Acked Write 3 Sent Done Done false
21115 5883 Acked Flush 0 Sent Done Done false
21116 5884 Acked Read 5 Sent Done Done false
21117 5885 Acked Read 2 Sent Done Done false
21118 5886 Acked Read 2 Sent Done Done false
21119 5887 Acked Read 10 Sent Done Done false
21120 5888 Acked Flush 0 Sent Done Done false
21121 5889 Acked Read 8 Sent Done Done false
21122 5890 Acked Read 3 Sent Done Done false
21123 5891 Acked Write 9 Sent Done Done false
21124 5892 Acked Read 5 Sent Done Done false
21125 5893 Acked Read 6 Sent Done Done false
21126 5894 Acked Write 2 Sent Done Done false
21127 5895 Acked Read 7 Sent Done Done false
21128 5896 Acked Write 4 Sent Done Done false
21129 5897 Acked Flush 0 Sent Done Done false
21130 5898 Acked Read 9 Sent Done Done false
21131 5899 Acked Flush 0 Sent Done Done false
21132 5900 Acked Read 1 Sent Done Done false
21133 5901 Acked Read 8 Sent Done Done false
21134 5902 Acked Flush 0 Sent Done Done false
21135 5903 Acked Read 4 Sent Done Done false
21136 5904 Acked Write 9 Sent Done Done false
21137 5905 Acked Read 3 Sent Done Done false
21138 5906 Acked Read 9 Sent Done Done false
21139 5907 Acked Write 9 Sent Done Done false
21140 5908 Acked Flush 0 Sent Done Done false
21141 5909 Acked Write 5 Sent Done Done false
21142 5910 Acked Write 8 Sent Done Done false
21143 5911 Acked Read 9 Sent Done Done false
21144 5912 Acked Write 6 Sent Done Done false
21145 5913 Acked Write 5 Sent Done Done false
21146 5914 Acked Read 5 Sent Done Done false
21147 5915 Acked Read 7 Sent Done Done false
21148 5916 Acked Write 3 Sent Done Done false
21149 5917 Acked Write 1 Sent Done Done false
21150 5918 Acked Flush 0 Sent Done Done false
21151 5919 Acked Read 3 Sent Done Done false
21152 5920 Acked Write 4 Sent Done Done false
21153 5921 Acked Flush 0 Sent Done Done false
21154 5922 Acked Read 9 Sent Done Done false
21155 5923 Acked Read 8 Sent Done Done false
21156 5924 Acked Write 6 Sent Done Done false
21157 5925 Acked Flush 0 Sent Done Done false
21158 5926 Acked Flush 0 Sent Done Done false
21159 5927 Acked Write 1 Sent Done Done false
21160 5928 Acked Write 2 Sent Done Done false
21161 5929 Acked Flush 0 Sent Done Done false
21162 5930 Acked Write 3 Sent Done Done false
21163 5931 Acked Read 7 Sent Done Done false
21164 5932 Acked Write 3 Sent Done Done false
21165 5933 Acked Write 6 Sent Done Done false
21166 5934 Acked Read 10 Sent Done Done false
21167 5935 Acked Write 5 Sent Done Done false
21168 5936 Acked Write 8 Sent Done Done false
21169 5937 Acked Write 3 Sent Done Done false
21170 5938 Acked Read 8 Sent Done Done false
21171 5939 Acked Read 7 Sent Done Done false
21172 5940 Acked Write 2 Sent Done Done false
21173 5941 Acked Write 5 Sent Done Done false
21174 5942 Acked Write 6 Sent Done Done false
21175 5943 Acked Read 8 Sent Done Done false
21176 5944 Acked Read 1 Sent Done Done false
21177 5945 Acked Read 4 Sent Done Done false
21178 5946 Acked Flush 0 Sent Done Done false
21179 5947 Acked Write 10 Sent Done Done false
21180 5948 Acked Write 9 Sent Done Done false
21181 5949 Acked Write 1 Sent Done Done false
21182 5950 Acked Write 8 Sent Done Done false
21183 5951 Acked Read 7 Sent Done Done false
21184 5952 Acked Read 10 Sent Done Done false
21185 5953 Acked Read 5 Sent Done Done false
21186 5954 Acked Write 8 Sent Done Done false
21187 5955 Acked Flush 0 Sent Done Done false
21188 5956 Acked Read 9 Sent Done Done false
21189 5957 Acked Read 6 Sent Done Done false
21190 5958 Acked Flush 0 Sent Done Done false
21191 5959 Acked Write 4 Sent Done Done false
21192 5960 Acked Read 9 Sent Done Done false
21193 5961 Acked Write 6 Sent Done Done false
21194 5962 Acked Read 6 Sent Done Done false
21195 5963 Acked Write 1 Sent Done Done false
21196 5964 Acked Read 4 Sent Done Done false
21197 5965 Acked Read 5 Sent Done Done false
21198 5966 Acked Write 6 Sent Done Done false
21199 5967 Acked Read 1 Sent Done Done false
21200 5968 Acked Read 7 Sent Done Done false
21201 5969 Acked Write 5 Sent Done Done false
21202 5970 Acked Write 5 Sent Done Done false
21203 5971 Acked Read 4 Sent Done Done false
21204 5972 Acked Read 7 Sent Done Done false
21205 5973 Acked Read 3 Sent Done Done false
21206 5974 Acked Write 9 Sent Done Done false
21207 5975 Acked Write 3 Sent Done Done false
21208 5976 Acked Write 1 Sent Done Done false
21209 5977 Acked Read 10 Sent Done Done false
21210 5978 Acked Read 8 Sent Done Done false
21211 5979 Acked Read 4 Sent Done Done false
21212 5980 Acked Write 6 Sent Done Done false
21213 5981 Acked Write 1 Sent Done Done false
21214 5982 Acked Read 5 Sent Done Done false
21215 5983 Acked Write 5 Sent Done Done false
21216 5984 Acked Write 6 Sent Done Done false
21217 5985 Acked Read 7 Sent Done Done false
21218 5986 Acked Read 3 Sent Done Done false
21219 5987 Acked Write 9 Sent Done Done false
21220 5988 Acked Read 6 Sent Done Done false
21221 5989 Acked Read 7 Sent Done Done false
21222 5990 Acked Read 7 Sent Done Done false
21223 5991 Acked Write 6 Sent Done Done false
21224 5992 Acked Read 8 Sent Done Done false
21225 5993 Acked Read 3 Sent Done Done false
21226 5994 Acked Read 8 Sent Done Done false
21227 5995 Acked Read 7 Sent Done Done false
21228 5996 Acked Write 7 Sent Done Done false
21229 5997 Acked Write 4 Sent Done Done false
21230 5998 Acked Write 10 Sent Done Done false
21231 5999 Acked Read 6 Sent Done Done false
21232 6000 Acked Read 9 Sent Done Done false
21233 6001 Acked Write 3 Sent Done Done false
21234 6002 Acked Read 2 Sent Done Done false
21235 6003 Acked Write 10 Sent Done Done false
21236 6004 Acked Write 7 Sent Done Done false
21237 6005 Acked Write 5 Sent Done Done false
21238 6006 Acked Write 6 Sent Done Done false
21239 6007 Acked Read 7 Sent Done Done false
21240 6008 Acked Write 8 Sent Done Done false
21241 6009 Acked Read 4 Sent Done Done false
21242 6010 Acked Read 5 Sent Done Done false
21243 6011 Acked Write 6 Sent Done Done false
21244 6012 Acked Read 2 Sent Done Done false
21245 6013 Acked Read 2 Sent Done Done false
21246 6014 Acked Write 3 Sent Done Done false
21247 6015 Acked Read 10 Sent Done Done false
21248 6016 Acked Write 4 Sent Done Done false
21249 6017 Acked Write 2 Sent Done Done false
21250 6018 Acked Read 3 Sent Done Done false
21251 6019 Acked Flush 0 Sent Done Done false
21252 6020 Acked Write 9 Sent Done Done false
21253 6021 Acked Read 5 Sent Done Done false
21254 6022 Acked Read 6 Sent Done Done false
21255 6023 Acked Write 8 Sent Done Done false
21256 6024 Acked Write 1 Sent Done Done false
21257 6025 Acked Read 3 Sent Done Done false
21258 6026 Acked Read 1 Sent Done Done false
21259 6027 Acked Read 6 Sent Done Done false
21260 6028 Acked Write 9 Sent Done Done false
21261 6029 Acked Read 6 Sent Done Done false
21262 6030 Acked Read 1 Sent Done Done false
21263 6031 Acked Read 10 Sent Done Done false
21264 6032 Acked Read 10 Sent Done Done false
21265 6033 Acked Read 6 Sent Done Done false
21266 6034 Acked Write 7 Sent Done Done false
21267 6035 Acked Read 7 Sent Done Done false
21268 6036 Acked Read 1 Sent Done Done false
21269 6037 Acked Write 10 Sent Done Done false
21270 6038 Acked Read 3 Sent Done Done false
21271 6039 Acked Read 1 Sent Done Done false
21272 6040 Acked Write 4 Sent Done Done false
21273 6041 Acked Read 4 Sent Done Done false
21274 6042 Acked Write 4 Sent Done Done false
21275 6043 Acked Write 9 Sent Done Done false
21276 6044 Acked Write 10 Sent Done Done false
21277 6045 Acked Read 3 Sent Done Done false
21278 6046 Acked Read 5 Sent Done Done false
21279 6047 Acked Read 3 Sent Done Done false
21280 6048 Acked Write 1 Sent Done Done false
21281 6049 Acked Read 7 Sent Done Done false
21282 6050 Acked Read 5 Sent Done Done false
21283 6051 Acked Write 6 Sent Done Done false
21284 6052 Acked Write 8 Sent Done Done false
21285 6053 Acked Read 4 Sent Done Done false
21286 6054 Acked Write 6 Sent Done Done false
21287 6055 Acked Read 6 Sent Done Done false
21288 6056 Acked Write 1 Sent Done Done false
21289 6057 Acked Write 2 Sent Done Done false
21290 6058 Acked Write 1 Sent Done Done false
21291 6059 Acked Read 3 Sent Done Done false
21292 6060 Acked Read 5 Sent Done Done false
21293 6061 Acked Read 7 Sent Done Done false
21294 6062 Acked Read 10 Sent Done Done false
21295 6063 Acked Read 8 Sent Done Done false
21296 6064 Acked Write 10 Sent Done Done false
21297 6065 Acked Read 4 Sent Done Done false
21298 6066 Acked Read 8 Sent Done Done false
21299 6067 Acked Write 10 Sent Done Done false
21300 6068 Acked Write 1 Sent Done Done false
21301 6069 Acked Write 7 Sent Done Done false
21302 6070 Acked Write 5 Sent Done Done false
21303 6071 Acked Write 9 Sent Done Done false
21304 6072 Acked Write 10 Sent Done Done false
21305 6073 Acked Read 2 Sent Done Done false
21306 6074 Acked Write 5 Sent Done Done false
21307 6075 Acked Read 7 Sent Done Done false
21308 6076 Acked Write 6 Sent Done Done false
21309 6077 Acked Read 2 Sent Done Done false
21310 6078 Acked Write 4 Sent Done Done false
21311 6079 Acked Write 5 Sent Done Done false
21312 6080 Acked Read 5 Sent Done Done false
21313 6081 Acked Write 5 Sent Done Done false
21314 6082 Acked Write 8 Sent Done Done false
21315 6083 Acked Write 3 Sent Done Done false
21316 6084 Acked Read 1 Sent Done Done false
21317 6085 Acked Read 4 Sent Done Done false
21318 6086 Acked Read 9 Sent Done Done false
21319 6087 Acked Read 2 Sent Done Done false
21320 6088 Acked Flush 0 Sent Done Done false
21321 6089 Acked Read 4 Sent Done Done false
21322 6090 Acked Write 8 Sent Done Done false
21323 6091 Acked Read 9 Sent Done Done false
21324 6092 Acked Read 5 Sent Done Done false
21325 6093 Acked Write 2 Sent Done Done false
21326 6094 Acked Read 6 Sent Done Done false
21327 6095 Acked Read 5 Sent Done Done false
21328 6096 Acked Read 6 Sent Done Done false
21329 6097 Acked Read 5 Sent Done Done false
21330 6098 Acked Flush 0 Sent Done Done false
21331 6099 Acked Read 3 Sent Done Done false
21332 6100 Acked Read 7 Sent Done Done false
21333 6101 Acked Write 7 Sent Done Done false
21334 6102 Acked Flush 0 Sent Done Done false
21335 6103 Acked Read 10 Sent Done Done false
21336 6104 Acked Read 5 Sent Done Done false
21337 6105 Acked Flush 0 Sent Done Done false
21338 6106 Acked Read 6 Sent Done Done false
21339 6107 Acked Write 10 Sent Done Done false
21340 6108 Acked Write 5 Sent Done Done false
21341 6109 Acked Flush 0 Sent Done Done false
21342 6110 Acked Write 3 Sent Done Done false
21343 6111 Acked Read 10 Sent Done Done false
21344 6112 Acked Flush 0 Sent Done Done false
21345 6113 Acked Read 5 Sent Done Done false
21346 6114 Acked Flush 0 Sent Done Done false
21347 6115 Acked Write 8 Sent Done Done false
21348 6116 Acked Read 8 Sent Done Done false
21349 6117 Acked Write 1 Sent Done Done false
21350 6118 Acked Read 5 Sent Done Done false
21351 6119 Acked Write 2 Sent Done Done false
21352 6120 Acked Read 3 Sent Done Done false
21353 6121 Acked Read 2 Sent Done Done false
21354 6122 Acked Flush 0 Sent Done Done false
21355 6123 Acked Read 9 Sent Done Done false
21356 6124 Acked Write 2 Sent Done Done false
21357 6125 Acked Flush 0 Sent Done Done false
21358 6126 Acked Write 8 Sent Done Done false
21359 6127 Acked Write 2 Sent Done Done false
21360 6128 Acked Read 5 Sent Done Done false
21361 6129 Acked Read 5 Sent Done Done false
21362 STATES DS:0 DS:1 DS:2 TOTAL
21363 Sent 301 0 0 301
21364 Done 0 301 301 602
21365 Skipped 0 0 0 0
21366 Error 0 0 0 0
21367 Last Flush: Some(JobId(5828)) Some(JobId(6125)) Some(JobId(6125))
21368 Downstairs last five completed: 5828 5827 5826 5825 5824
21369 Upstairs last five completed: 6129 6128 6127 6126 6125
21370 CLIENT: Up:0 ds:301 act:2
213712025-02-28T06:05:40.807ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
213722025-02-28T06:05:42.132ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
213732025-02-28T06:05:42.811ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
213742025-02-28T06:05:43.813ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
213752025-02-28T06:05:43.884ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
213762025-02-28T06:05:43.885ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63838) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
213772025-02-28T06:05:43.885ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
213782025-02-28T06:05:43.886ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
213792025-02-28T06:05:43.886ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
213802025-02-28T06:05:43.886ZINFOcrucible: send last flush ID to this DS: Some(JobId(5828)) = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
213812025-02-28T06:05:43.886ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
213822025-02-28T06:05:43.886ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(5828)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
213832025-02-28T06:05:43.886ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(5828)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
21384 ----------------------------------------------------------------
21385 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:4
21386 States: Active Active Active
21387 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21388 6126 Acked Write 8 Done Done Done true
21389 6127 Acked Write 2 Done Done Done true
21390 6128 Acked Read 5 Done Done Done true
21391 6129 Acked Read 5 Done Done Done true
21392 STATES DS:0 DS:1 DS:2 TOTAL
21393 Sent 0 0 0 0
21394 Done 4 4 4 12
21395 Skipped 0 0 0 0
21396 Error 0 0 0 0
21397 Last Flush: Some(JobId(6125)) Some(JobId(6125)) Some(JobId(6125))
21398 Downstairs last five completed: 6125 6124 6123 6122 6121
21399 Upstairs last five completed: 6129 6128 6127 6126 6125
21400 CLIENT: Up:0 ds:4 act:3
21401 ----------------------------------------------------------------
21402 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21403 Downstairs last five completed: 6130 6129 6128 6127 6126
21404 Upstairs last five completed: 6130 6129 6128 6127 6126
21405 CLIENT: Up:0 ds:0 act:3
21406 Replay: All jobs finished
21407 Main task has work to do, go find it
21408 got dsc Stop(0)
21409 stop 0
21410 Feb 28 06:05:48.138 INFO request completed, latency_us: 362, response_code: 204, uri: /stop/cid/0[8810] Got stop action so:false kr:false
21411 , method: POST, req_id: deb6970b-f511-4973-8371-0585464063f2, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
214122025-02-28T06:05:48.139ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
214132025-02-28T06:05:48.139ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
214142025-02-28T06:05:48.139ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
21415 Feb 28 06:05:48.139 INFO request completed, latency_us: 143, response_code:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:05:48.139384317Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
21416 200, uri: /state/cid/0, method: GET, req_id: 2ebecbeb-dc79-4a8c-b558-38da0b3c2e77, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
21417 [8810] Exited with: ExitStatus(unix_wait_status(9))
21418 [8810][0] reports Exit
21419 Feb 28 06:05:52.141 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/0, method: GET, req_id: f77cd126-d788-46f0-b5bb-3eacd146d0f5, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
21420 001/300 Read block 3985 len 8192
21421 002/300 Read block 4255 len 16384
21422 003/300 Flush
21423 004/300 Read block 1637 len 8192
21424 005/300 Write block 3919 len 16384 data: 5 5 4 3
21425 006/300 Read block 1575 len 36864
21426 007/300 Flush
21427 008/300 Write block 1120 len 36864 data: 3 2 3 3 4 6 6 6 6
21428 009/300 Write block 3428 len 36864 data: 3 3 4 4 4 3 4 6 7
21429 010/300 Flush
21430 011/300 Read block 2349 len 4096
21431 012/300 Flush
21432 013/300 Read block 2131 len 28672
21433 014/300 Write block 3325 len 20480 data: 4 3 3 2 2
21434 015/300 Write block 1554 len 16384 data: 3 3 3 4
21435 016/300 Write block 295 len 4096 data: 5
21436 017/300 Write block 1967 len 40960 data: 4 3 4 4 5 5 5 5 5 3
21437 018/300 Write block 3585 len 20480 data: 3 3 3 3 4
21438 019/300 Flush
21439 020/300 Write block 4303 len 28672 data: 2 2 2 3 4 3 3
21440 021/300 Read block 4595 len 16384
21441 022/300 Write block 4776 len 20480 data: 6 7 5 5 6
21442 023/300 Read block 1340 len 32768
21443 024/300 Read block 909 len 24576
21444 025/300 Read block 2552 len 32768
21445 026/300 Write block 4347 len 28672 data: 8 7 7 6 6 6 7
21446 027/300 Read block 1814 len 4096
21447 028/300 Read block 214 len 20480
21448 029/300 Read block 1603 len 36864
21449 030/300 Write block 4882 len 24576 data: 6 7 6 6 6 8
21450 031/300 Write block 4813 len 24576 data: 3 2 3 3 3 4
21451 032/300 Read block 4754 len 36864
21452 033/300 Read block 4318 len 20480
21453 034/300 Read block 1617 len 8192
21454 035/300 Read block 1253 len 8192
21455 036/300 Write block 4075 len 12288 data: 3 5 6
21456 037/300 Write block 3974 len 40960 data: 5 5 4 4 4 4 3 3 3 2
21457 038/300 Read block 1723 len 20480
21458 039/300 Flush
21459 040/300 Flush
21460 041/300 Read block 74 len 12288
21461 042/300 Write block 4730 len 24576 data: 6 6 6 5 5 4
21462 043/300 Write block 719 len 12288 data: 4 5 5
21463 044/300 Read block 801 len 32768
21464 045/300 Write block 4783 len 16384 data: 5 4 4 4
21465 046/300 Read block 4768 len 36864
21466 047/300 Write block 813 len 8192 data: 7 6
21467 048/300 Read block 3509 len 4096
21468 049/300 Write block 4203 len 24576 data: 3 5 4 5 3 3
21469 050/300 Read block 4260 len 36864
21470 051/300 Read block 2483 len 32768
21471 052/300 Read block 225 len 36864
21472 053/300 Write block 20 len 28672 data: 4 4 4 4 4 6 4
21473 054/300 Read block 3297 len 16384
21474 055/300 Write block 234 len 16384 data: 5 4 5 7
21475 056/300 Read block 3588 len 36864
21476 057/300 Flush
21477 058/300 Write block 1982 len 32768 data: 5 6 6 5 5 5 6 6
21478 059/300 Read block 4581 len 32768
21479 060/300 Read block 153 len 20480
21480 061/300 Read block 1719 len 20480
21481 062/300 Read block 3780 len 36864
21482 063/300 Read block 4231 len 32768
21483 064/300 Flush
21484 065/300 Write block 3823 len 28672 data: 3 2 2 3 3 3 4
21485 066/300 Flush
21486 067/300 Flush
21487 068/300 Flush
21488 069/300 Read block 1399 len 36864
21489 070/300 Write block 3986 len 36864 data: 5 5 3 3 3 3 3 4 4
21490 071/300 Write block 3847 len 24576 data: 5 6 5 7 7 8
21491 072/300 Read block 3819 len 12288
21492 073/300 Write block 3662 len 12288 data: 3 3 3
21493 074/300 Write block 2716 len 16384 data: 7 7 5 3
21494 075/300 Write block 1716 len 36864 data: 4 4 3 2 3 4 5 6 6
21495 076/300 Write block 3729 len 36864 data: 3 3 5 4 5 6 6 4 4
21496 077/300 Write block 465 len 28672 data: 2 3 4 5 5 6 4
21497 078/300 Write block 2172 len 36864 data: 4 3 4 4 5 5 5 7 5
21498 079/300 Flush
21499 080/300 Read block 2247 len 36864
21500 081/300 Read block 4702 len 28672
21501 082/300 Read block 1810 len 20480
21502 083/300 Read block 421 len 4096
21503 084/300 Write block 3340 len 36864 data: 5 6 5 6 6 6 6 6 6
21504 085/300 Write block 4701 len 32768 data: 5 4 4 2 2 2 2 2
21505 086/300 Flush
21506 087/300 Write block 4066 len 24576 data: 5 4 4 4 4 4
21507 088/300 Write block 423 len 4096 data: 7
21508 089/300 Read block 4091 len 4096
21509 090/300 Write block 1319 len 40960 data: 3 4 4 5 5 4 4 5 5 5
21510 091/300 Read block 972 len 28672
21511 092/300 Write block 4768 len 24576 data: 4 5 8 7 6 5
21512 093/300 Read block 811 len 4096
21513 094/300 Read block 3568 len 16384
21514 095/300 Read block 2401 len 4096
21515 096/300 Read block 4365 len 24576
21516 097/300 Read block 330 len 4096
21517 098/300 Read block 2056 len 8192
21518 099/300 Flush
21519 100/300 Read block 1558 len 32768
21520 101/300 Flush
21521 102/300 Read block 3245 len 28672
21522 103/300 Read block 1322 len 32768
21523 104/300 Read block 565 len 20480
21524 105/300 Read block 894 len 4096
21525 106/300 Read block 4158 len 28672
21526 107/300 Write block 4967 len 8192 data: 5 5
21527 108/300 Read block 2127 len 32768
21528 109/300 Read block 426 len 4096
21529 110/300 Write block 2685 len 12288 data: 8 7 8
21530 111/300 Write block 3564 len 4096 data: 4
21531 112/300 Read block 3645 len 24576
21532 113/300 Read block 1225 len 8192
21533 114/300 Write block 4114 len 28672 data: 5 5 6 6 4 4 3
21534 115/300 Write block 848 len 8192 data: 2 2
21535 116/300 Read block 778 len 32768
21536 117/300 Flush
21537 118/300 Write block 1877 len 8192 data: 6 6
21538 119/300 Read block 1582 len 4096
21539 120/300 Write block 4712 len 8192 data: 4 4
21540 121/300 Read block 3247 len 8192
21541 122/300 Write block 2818 len 20480 data: 5 6 5 4 3
21542 123/300 Write block 3848 len 12288 data: 7 6 8
21543 124/300 Write block 3529 len 32768 data: 6 4 4 4 4 4 4 6
21544 125/300 Read block 1445 len 24576
21545 126/300 Read block 4133 len 32768
21546 127/300 Write block 1788 len 32768 data: 3 4 4 4 5 4 4 5
21547 128/300 Write block 2045 len 12288 data: 3 3 3
21548 129/300 Read block 3741 len 8192
21549 130/300 Flush
21550 131/300 Write block 4968 len 36864 data: 6 5 5 6 5 4 4 3 3
21551 132/300 Read block 1495 len 28672
21552 133/300 Read block 1494 len 28672
21553 134/300 Read block 502 len 12288
21554 135/300 Write block 3838 len 16384 data: 3 3 4 4
21555 136/300 Read block 1675 len 24576
21556 137/300 Read block 174 len 24576
21557 138/300 Read block 2981 len 16384
21558 139/300 Read block 3565 len 40960
21559 140/300 Write block 4615 len 4096 data: 4
21560 141/300 Write block 1436 len 8192 data: 4 5
21561 142/300 Write block 217 len 32768 data: 5 6 8 5 6 5 5 4
21562 143/300 Write block 549 len 16384 data: 4 5 6 7
21563 144/300 Read block 2594 len 4096
21564 145/300 Write block 3831 len 40960 data: 3 3 3 3 3 2 2 4 4 5
21565 146/300 Write block 1874 len 36864 data: 5 5 5 7 7 5 5 5 4
21566 147/300 Write block 292 len 8192 data: 5 4
21567 148/300 Read block 2638 len 16384
21568 149/300 Flush
21569 150/300 Read block 3120 len 16384
21570 151/300 Read block 1430 len 12288
21571 152/300 Read block 2015 len 16384
21572 153/300 Read block 1579 len 40960
21573 154/300 Write block 3627 len 12288 data: 4 4 4
21574 155/300 Write block 1676 len 36864 data: 5 4 4 4 4 4 6 7 7
21575 156/300 Read block 4132 len 8192
21576 157/300 Read block 2277 len 4096
21577 158/300 Read block 773 len 8192
21578 159/300 Write block 3328 len 36864 data: 3 3 3 3 3 2 3 5 5
21579 160/300 Read block 989 len 12288
21580 161/300 Write block 692 len 40960 data: 4 3 3 3 2 3 3 3 4 5
21581 162/300 Read block 3211 len 28672
21582 163/300 Write block 1151 len 24576 data: 5 6 7 6 7 6
21583 164/300 Flush
21584 165/300 Write block 2254 len 4096 data: 3
21585 166/300 Read block 4104 len 40960
21586 167/300 Read block 763 len 20480
21587 168/300 Read block 1888 len 24576
21588 169/300 Read block 2823 len 20480
21589 170/300 Write block 1164 len 16384 data: 5 5 5 5
21590 171/300 Read block 4715 len 16384
21591 172/300 Flush
21592 173/300 Read block 1708 len 28672
21593 174/300 Read block 2791 len 4096
21594 175/300 Write block 3676 len 12288 data: 2 2 2
21595 176/300 Write block 4709 len 40960 data: 3 3 4 5 5 4 4 4 4 4
21596 177/300 Read block 3956 len 8192
21597 178/300 Read block 4542 len 8192
21598 179/300 Read block 2532 len 12288
21599 180/300 Write block 590 len 40960 data: 2 2 2 2 2 2 2 3 4 5
21600 181/300 Write block 1157 len 12288 data: 6 5 5
21601 182/300 Write block 3942 len 12288 data: 5 4 5
21602 183/300 Read block 723 len 36864
21603 184/300 Read block 2083 len 8192
21604 185/300 Write block 1246 len 16384 data: 2 4 3 4
21605 186/300 Read block 2752 len 24576
21606 187/300 Read block 3637 len 36864
21607 188/300 Read block 4143 len 32768
21608 189/300 Read block 1438 len 16384
21609 190/300 Read block 2819 len 20480
21610 191/300 Read block 3368 len 24576
21611 192/300 Read block 899 len 28672
21612 193/300 Flush
21613 194/300 Read block 1301 len 12288
21614 195/300 Flush
21615 196/300 Read block 4785 len 28672
21616 197/300 Read block 4082 len 36864
21617 198/300 Write block 2102 len 4096 data: 4
21618 199/300 Flush
21619 200/300 Flush
21620 201/300 Read block 280 len 4096
21621 202/300 Flush
21622 203/300 Read block 4139 len 16384
21623 204/300 Read block 861 len 12288
21624 205/300 Write block 3001 len 24576 data: 4 2 2 2 2 3
21625 206/300 Write block 1471 len 36864 data: 3 3 4 4 2 2 3 3 3
21626 207/300 Write block 557 len 40960 data: 3 4 4 4 4 4 4 4 4 2
21627 208/300 Read block 4410 len 4096
21628 209/300 Flush
21629 210/300 Read block 3109 len 36864
21630 211/300 Read block 3996 len 32768
21631 212/300 Write block 3363 len 8192 data: 3 4
21632 213/300 Read block 1142 len 8192
21633 214/300 Flush
21634 215/300 Write block 2687 len 4096 data: 9
21635 216/300 Write block 4320 len 20480 data: 5 6 5 6 5
21636 217/300 Read block 4871 len 28672
21637 218/300 Flush
21638 219/300 Read block 4437 len 16384
21639 220/300 Read block 3007 len 4096
21640 221/300 Flush
21641 222/300 Read block 1665 len 12288
21642 223/300 Write block 2692 len 16384 data: 5 5 5 5
21643 224/300 Read block 4092 len 36864
21644 225/300 Flush
21645 226/300 Write block 2986 len 36864 data: 5 5 5 5 7 7 8 5 4
21646 227/300 Flush
21647 228/300 Read block 2288 len 28672
21648 229/300 Flush
21649 230/300 Write block 2370 len 4096 data: 3
21650 231/300 Read block 1237 len 36864
21651 232/300 Write block 4716 len 12288 data: 5 5 5
21652 233/300 Read block 1076 len 12288
21653 234/300 Read block 3562 len 4096
21654 235/300 Write block 1240 len 20480 data: 3 3 3 2 2
21655 236/300 Read block 4159 len 24576
21656 237/300 Read block 2711 len 32768
21657 238/300 Flush
21658 239/300 Write block 4864 len 28672 data: 4 4 3 4 4 3 3
21659 240/300 Read block 2942 len 28672
21660 241/300 Read block 922 len 16384
21661 242/300 Write block 2600 len 12288 data: 5 5 4
21662 243/300 Read block 2737 len 24576
21663 244/300 Read block 2100 len 12288
21664 245/300 Flush
21665 246/300 Flush
21666 247/300 Write block 1754 len 32768 data: 4 4 3 3 4 4 4 4
21667 248/300 Flush
21668 249/300 Write block 3156 len 24576 data: 7 8 8 9 7 4
21669 250/300 Read block 584 len 32768
21670 251/300 Read block 2530 len 8192
21671 252/300 Read block 3458 len 28672
21672 253/300 Write block 2764 len 24576 data: 4 2 2 2 2 3
21673 254/300 Read block 4766 len 40960
21674 255/300 Read block 2469 len 40960
21675 256/300 Read block 4300 len 20480
21676 257/300 Write block 3979 len 4096 data: 5
21677 258/300 Write block 4550 len 24576 data: 3 3 2 2 2 2
21678 259/300 Write block 4470 len 4096 data: 4
21679 260/300 Read block 1576 len 12288
21680 261/300 Read block 3079 len 40960
21681 262/300 Read block 3515 len 16384
21682 263/300 Read block 172 len 20480
21683 264/300 Read block 2901 len 24576
21684 265/300 Write block 474 len 40960 data: 5 6 7 6 7 4 5 5 5 7
21685 266/300 Read block 1428 len 28672
21686 267/300 Read block 4432 len 40960
21687 268/300 Read block 284 len 8192
21688 269/300 Write block 3593 len 32768 data: 4 3 3 3 4 3 3 2
21689 270/300 Write block 206 len 12288 data: 3 4 5
21690 271/300 Read block 419 len 20480
21691 272/300 Read block 4718 len 28672
21692 273/300 Write block 498 len 12288 data: 5 5 5
21693 274/300 Write block 2501 len 36864 data: 3 3 4 2 2 2 2 2 3
21694 275/300 Write block 1195 len 24576 data: 3 3 4 5 4 3
21695 276/300 Read block 4025 len 28672
21696 277/300 Write block 1187 len 32768 data: 4 3 3 3 3 3 3 3
21697 278/300 Read block 3687 len 24576
21698 279/300 Write block 4887 len 32768 data: 9 8 9 8 7 5 5 5
21699 280/300 Write block 2288 len 40960 data: 2 2 3 5 4 4 2 2 2 2
21700 281/300 Write block 4394 len 24576 data: 5 5 5 3 2 2
21701 282/300 Flush
21702 283/300 Write block 1273 len 16384 data: 6 6 4 4
21703 284/300 Write block 1055 len 32768 data: 6 5 4 4 4 4 3 2
21704 285/300 Read block 1292 len 20480
21705 286/300 Write block 2921 len 40960 data: 4 4 6 6 6 6 4 2 3 3
21706 287/300 Read block 2109 len 8192
21707 288/300 Read block 1706 len 8192
21708 289/300 Read block 1051 len 8192
21709 290/300 Write block 2669 len 12288 data: 8 8 5
21710 291/300 Write block 2184 len 28672 data: 6 6 7 5 6 6 7
21711 292/300 Flush
21712 293/300 Write block 4958 len 16384 data: 6 5 4 5
21713 294/300 Write block 2021 len 40960 data: 5 3 3 3 3 4 4 6 6 5
21714 295/300 Read block 164 len 12288
21715 296/300 Write block 1467 len 40960 data: 4 4 4 3 4 4 5 5 3 3
21716 297/300 Read block 4230 len 4096
21717 298/300 Write block 2085 len 4096 data: 4
21718 299/300 Write block 334 len 20480 data: 3 3 4 5 5
21719 300/300 Read block 1112 len 24576
21720 301/300 Write block 3343 len 32768 data: 7 7 7 7 7 7 5 6
21721 Main task has work to do, go find it
21722 got dsc Start(0)
21723 start 0
21724 Feb 28 06:05:52.376 INFO request completed, latency_us: 174, response_code: 204, uri: /start/cid/0, method: POST, req_id: 610ad222-6002-4f5c-aeaf-90e3d7435f4a, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
21725 Replay: started 0, returned:Ok(())
21726 ----------------------------------------------------------------
21727 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21728 States: Offline Active Active
21729 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21730 6131 Acked Read 2 Sent Done Done false
21731 6132 Acked Read 4 Sent Done Done false
21732 6133 Acked Flush 0 Sent Done Done false
21733 6134 Acked Read 2 Sent Done Done false
21734 6135 Acked Write 4 Sent Done Done false
21735 6136 Acked Read 9 Sent Done Done false
21736 6137 Acked Flush 0 Sent Done Done false
21737 6138 Acked Write 9 Sent Done Done false
21738 6139 Acked Write 9 Sent Done Done false
21739 6140 Acked Flush 0 Sent Done Done false
21740 6141 Acked Read 1 Sent Done Done false
21741 6142 Acked Flush 0 Sent Done Done false
21742 6143 Acked Read 7 Sent Done Done false
21743 6144 Acked Write 5 Sent Done Done false
21744 6145 Acked Write 4 Sent Done Done false
21745 6146 Acked Write 1 Sent Done Done false
21746 6147 Acked Write 10 Sent Done Done false
21747 6148 Acked Write 5 Sent Done Done false
21748 6149 Acked Flush 0 Sent Done Done false
21749 6150 Acked Write 7 Sent Done Done false
21750 6151 Acked Read 4 Sent Done Done false
21751 6152 Acked Write 5 Sent Done Done false
21752 6153 Acked Read 8 Sent Done Done false
21753 6154 Acked Read 6 Sent Done Done false
21754 6155 Acked Read 8 Sent Done Done false
21755 6156 Acked Write 7 Sent Done Done false
21756 6157 Acked Read 1 Sent Done Done false
21757 6158 Acked Read 5 Sent Done Done false
21758 6159 Acked Read 9 Sent Done Done false
21759 6160 Acked Write 6 Sent Done Done false
21760 6161 Acked Write 6 Sent Done Done false
21761 6162 Acked Read 9 Sent Done Done false
21762 6163 Acked Read 5 Sent Done Done false
21763 6164 Acked Read 2 Sent Done Done false
21764 6165 Acked Read 2 Sent Done Done false
21765 6166 Acked Write 3 Sent Done Done false
21766 6167 Acked Write 10 Sent Done Done false
21767 6168 Acked Read 5 Sent Done Done false
21768 6169 Acked Flush 0 Sent Done Done false
21769 6170 Acked Flush 0 Sent Done Done false
21770 6171 Acked Read 3 Sent Done Done false
21771 6172 Acked Write 6 Sent Done Done false
21772 6173 Acked Write 3 Sent Done Done false
21773 6174 Acked Read 8 Sent Done Done false
21774 6175 Acked Write 4 Sent Done Done false
21775 6176 Acked Read 9 Sent Done Done false
21776 6177 Acked Write 2 Sent Done Done false
21777 6178 Acked Read 1 Sent Done Done false
21778 6179 Acked Write 6 Sent Done Done false
21779 6180 Acked Read 9 Sent Done Done false
21780 6181 Acked Read 8 Sent Done Done false
21781 6182 Acked Read 9 Sent Done Done false
21782 6183 Acked Write 7 Sent Done Done false
21783 6184 Acked Read 4 Sent Done Done false
21784 6185 Acked Write 4 Sent Done Done false
21785 6186 Acked Read 9 Sent Done Done false
21786 6187 Acked Flush 0 Sent Done Done false
21787 6188 Acked Write 8 Sent Done Done false
21788 6189 Acked Read 8 Sent Done Done false
21789 6190 Acked Read 5 Sent Done Done false
21790 6191 Acked Read 5 Sent Done Done false
21791 6192 Acked Read 9 Sent Done Done false
21792 6193 Acked Read 8 Sent Done Done false
21793 6194 Acked Flush 0 Sent Done Done false
21794 6195 Acked Write 7 Sent Done Done false
21795 6196 Acked Flush 0 Sent Done Done false
21796 6197 Acked Flush 0 Sent Done Done false
21797 6198 Acked Flush 0 Sent Done Done false
21798 6199 Acked Read 9 Sent Done Done false
21799 6200 Acked Write 9 Sent Done Done false
21800 6201 Acked Write 6 Sent Done Done false
21801 6202 Acked Read 3 Sent Done Done false
21802 6203 Acked Write 3 Sent Done Done false
21803 6204 Acked Write 4 Sent Done Done false
21804 6205 Acked Write 9 Sent Done Done false
21805 6206 Acked Write 9 Sent Done Done false
21806 6207 Acked Write 7 Sent Done Done false
21807 6208 Acked Write 9 Sent Done Done false
21808 6209 Acked Flush 0 Sent Done Done false
21809 6210 Acked Read 9 Sent Done Done false
21810 6211 Acked Read 7 Sent Done Done false
21811 6212 Acked Read 5 Sent Done Done false
21812 6213 Acked Read 1 Sent Done Done false
21813 6214 Acked Write 9 Sent Done Done false
21814 6215 Acked Write 8 Sent Done Done false
21815 6216 Acked Flush 0 Sent Done Done false
21816 6217 Acked Write 6 Sent Done Done false
21817 6218 Acked Write 1 Sent Done Done false
21818 6219 Acked Read 1 Sent Done Done false
21819 6220 Acked Write 10 Sent Done Done false
21820 6221 Acked Read 7 Sent Done Done false
21821 6222 Acked Write 6 Sent Done Done false
21822 6223 Acked Read 1 Sent Done Done false
21823 6224 Acked Read 4 Sent Done Done false
21824 6225 Acked Read 1 Sent Done Done false
21825 6226 Acked Read 6 Sent Done Done false
21826 6227 Acked Read 1 Sent Done Done false
21827 6228 Acked Read 2 Sent Done Done false
21828 6229 Acked Flush 0 Sent Done Done false
21829 6230 Acked Read 8 Sent Done Done false
21830 6231 Acked Flush 0 Sent Done Done false
21831 6232 Acked Read 7 Sent Done Done false
21832 6233 Acked Read 8 Sent Done Done false
21833 6234 Acked Read 5 Sent Done Done false
21834 6235 Acked Read 1 Sent Done Done false
21835 6236 Acked Read 7 Sent Done Done false
21836 6237 Acked Write 2 Sent Done Done false
21837 6238 Acked Read 8 Sent Done Done false
21838 6239 Acked Read 1 Sent Done Done false
21839 6240 Acked Write 3 Sent Done Done false
21840 6241 Acked Write 1 Sent Done Done false
21841 6242 Acked Read 6 Sent Done Done false
21842 6243 Acked Read 2 Sent Done Done false
21843 6244 Acked Write 7 Sent Done Done false
21844 6245 Acked Write 2 Sent Done Done false
21845 6246 Acked Read 8 Sent Done Done false
21846 6247 Acked Flush 0 Sent Done Done false
21847 6248 Acked Write 2 Sent Done Done false
21848 6249 Acked Read 1 Sent Done Done false
21849 6250 Acked Write 2 Sent Done Done false
21850 6251 Acked Read 2 Sent Done Done false
21851 6252 Acked Write 5 Sent Done Done false
21852 6253 Acked Write 3 Sent Done Done false
21853 6254 Acked Write 8 Sent Done Done false
21854 6255 Acked Read 6 Sent Done Done false
21855 6256 Acked Read 8 Sent Done Done false
21856 6257 Acked Write 8 Sent Done Done false
21857 6258 Acked Write 3 Sent Done Done false
21858 6259 Acked Read 2 Sent Done Done false
21859 6260 Acked Flush 0 Sent Done Done false
21860 6261 Acked Write 9 Sent Done Done false
21861 6262 Acked Read 7 Sent Done Done false
21862 6263 Acked Read 7 Sent Done Done false
21863 6264 Acked Read 3 Sent Done Done false
21864 6265 Acked Write 4 Sent Done Done false
21865 6266 Acked Read 6 Sent Done Done false
21866 6267 Acked Read 6 Sent Done Done false
21867 6268 Acked Read 4 Sent Done Done false
21868 6269 Acked Read 10 Sent Done Done false
21869 6270 Acked Write 1 Sent Done Done false
21870 6271 Acked Write 2 Sent Done Done false
21871 6272 Acked Write 8 Sent Done Done false
21872 6273 Acked Write 4 Sent Done Done false
21873 6274 Acked Read 1 Sent Done Done false
21874 6275 Acked Write 10 Sent Done Done false
21875 6276 Acked Write 9 Sent Done Done false
21876 6277 Acked Write 2 Sent Done Done false
21877 6278 Acked Read 4 Sent Done Done false
21878 6279 Acked Flush 0 Sent Done Done false
21879 6280 Acked Read 4 Sent Done Done false
21880 6281 Acked Read 3 Sent Done Done false
21881 6282 Acked Read 4 Sent Done Done false
21882 6283 Acked Read 10 Sent Done Done false
21883 6284 Acked Write 3 Sent Done Done false
21884 6285 Acked Write 9 Sent Done Done false
21885 6286 Acked Read 2 Sent Done Done false
21886 6287 Acked Read 1 Sent Done Done false
21887 6288 Acked Read 2 Sent Done Done false
21888 6289 Acked Write 9 Sent Done Done false
21889 6290 Acked Read 3 Sent Done Done false
21890 6291 Acked Write 10 Sent Done Done false
21891 6292 Acked Read 7 Sent Done Done false
21892 6293 Acked Write 6 Sent Done Done false
21893 6294 Acked Flush 0 Sent Done Done false
21894 6295 Acked Write 1 Sent Done Done false
21895 6296 Acked Read 10 Sent Done Done false
21896 6297 Acked Read 5 Sent Done Done false
21897 6298 Acked Read 6 Sent Done Done false
21898 6299 Acked Read 5 Sent Done Done false
21899 6300 Acked Write 4 Sent Done Done false
21900 6301 Acked Read 4 Sent Done Done false
21901 6302 Acked Flush 0 Sent Done Done false
21902 6303 Acked Read 7 Sent Done Done false
21903 6304 Acked Read 1 Sent Done Done false
21904 6305 Acked Write 3 Sent Done Done false
21905 6306 Acked Write 10 Sent Done Done false
21906 6307 Acked Read 2 Sent Done Done false
21907 6308 Acked Read 2 Sent Done Done false
21908 6309 Acked Read 3 Sent Done Done false
21909 6310 Acked Write 10 Sent Done Done false
21910 6311 Acked Write 3 Sent Done Done false
21911 6312 Acked Write 3 Sent Done Done false
21912 6313 Acked Read 9 Sent Done Done false
21913 6314 Acked Read 2 Sent Done Done false
21914 6315 Acked Write 4 Sent Done Done false
21915 6316 Acked Read 6 Sent Done Done false
21916 6317 Acked Read 9 Sent Done Done false
21917 6318 Acked Read 8 Sent Done Done false
21918 6319 Acked Read 4 Sent Done Done false
21919 6320 Acked Read 5 Sent Done Done false
21920 6321 Acked Read 6 Sent Done Done false
21921 6322 Acked Read 7 Sent Done Done false
21922 6323 Acked Flush 0 Sent Done Done false
21923 6324 Acked Read 3 Sent Done Done false
21924 6325 Acked Flush 0 Sent Done Done false
21925 6326 Acked Read 7 Sent Done Done false
21926 6327 Acked Read 9 Sent Done Done false
21927 6328 Acked Write 1 Sent Done Done false
21928 6329 Acked Flush 0 Sent Done Done false
21929 6330 Acked Flush 0 Sent Done Done false
21930 6331 Acked Read 1 Sent Done Done false
21931 6332 Acked Flush 0 Sent Done Done false
21932 6333 Acked Read 4 Sent Done Done false
21933 6334 Acked Read 3 Sent Done Done false
21934 6335 Acked Write 6 Sent Done Done false
21935 6336 Acked Write 9 Sent Done Done false
21936 6337 Acked Write 10 Sent Done Done false
21937 6338 Acked Read 1 Sent Done Done false
21938 6339 Acked Flush 0 Sent Done Done false
21939 6340 Acked Read 9 Sent Done Done false
21940 6341 Acked Read 8 Sent Done Done false
21941 6342 Acked Write 2 Sent Done Done false
21942 6343 Acked Read 2 Sent Done Done false
21943 6344 Acked Flush 0 Sent Done Done false
21944 6345 Acked Write 1 Sent Done Done false
21945 6346 Acked Write 5 Sent Done Done false
21946 6347 Acked Read 7 Sent Done Done false
21947 6348 Acked Flush 0 Sent Done Done false
21948 6349 Acked Read 4 Sent Done Done false
21949 6350 Acked Read 1 Sent Done Done false
21950 6351 Acked Flush 0 Sent Done Done false
21951 6352 Acked Read 3 Sent Done Done false
21952 6353 Acked Write 4 Sent Done Done false
21953 6354 Acked Read 9 Sent Done Done false
21954 6355 Acked Flush 0 Sent Done Done false
21955 6356 Acked Write 9 Sent Done Done false
21956 6357 Acked Flush 0 Sent Done Done false
21957 6358 Acked Read 7 Sent Done Done false
21958 6359 Acked Flush 0 Sent Done Done false
21959 6360 Acked Write 1 Sent Done Done false
21960 6361 Acked Read 9 Sent Done Done false
21961 6362 Acked Write 3 Sent Done Done false
21962 6363 Acked Read 3 Sent Done Done false
21963 6364 Acked Read 1 Sent Done Done false
21964 6365 Acked Write 5 Sent Done Done false
21965 6366 Acked Read 6 Sent Done Done false
21966 6367 Acked Read 8 Sent Done Done false
21967 6368 Acked Flush 0 Sent Done Done false
21968 6369 Acked Write 7 Sent Done Done false
21969 6370 Acked Read 7 Sent Done Done false
21970 6371 Acked Read 4 Sent Done Done false
21971 6372 Acked Write 3 Sent Done Done false
21972 6373 Acked Read 6 Sent Done Done false
21973 6374 Acked Read 3 Sent Done Done false
21974 6375 Acked Flush 0 Sent Done Done false
21975 6376 Acked Flush 0 Sent Done Done false
21976 6377 Acked Write 8 Sent Done Done false
21977 6378 Acked Flush 0 Sent Done Done false
21978 6379 Acked Write 6 Sent Done Done false
21979 6380 Acked Read 8 Sent Done Done false
21980 6381 Acked Read 2 Sent Done Done false
21981 6382 Acked Read 7 Sent Done Done false
21982 6383 Acked Write 6 Sent Done Done false
21983 6384 Acked Read 10 Sent Done Done false
21984 6385 Acked Read 10 Sent Done Done false
21985 6386 Acked Read 5 Sent Done Done false
21986 6387 Acked Write 1 Sent Done Done false
21987 6388 Acked Write 6 Sent Done Done false
21988 6389 Acked Write 1 Sent Done Done false
21989 6390 Acked Read 3 Sent Done Done false
21990 6391 Acked Read 10 Sent Done Done false
21991 6392 Acked Read 4 Sent Done Done false
21992 6393 Acked Read 5 Sent Done Done false
21993 6394 Acked Read 6 Sent Done Done false
21994 6395 Acked Write 10 Sent Done Done false
21995 6396 Acked Read 7 Sent Done Done false
21996 6397 Acked Read 10 Sent Done Done false
21997 6398 Acked Read 2 Sent Done Done false
21998 6399 Acked Write 8 Sent Done Done false
21999 6400 Acked Write 3 Sent Done Done false
22000 6401 Acked Read 5 Sent Done Done false
22001 6402 Acked Read 7 Sent Done Done false
22002 6403 Acked Write 3 Sent Done Done false
22003 6404 Acked Write 9 Sent Done Done false
22004 6405 Acked Write 6 Sent Done Done false
22005 6406 Acked Read 7 Sent Done Done false
22006 6407 Acked Write 8 Sent Done Done false
22007 6408 Acked Read 6 Sent Done Done false
22008 6409 Acked Write 8 Sent Done Done false
22009 6410 Acked Write 10 Sent Done Done false
22010 6411 Acked Write 6 Sent Done Done false
22011 6412 Acked Flush 0 Sent Done Done false
22012 6413 Acked Write 4 Sent Done Done false
22013 6414 Acked Write 8 Sent Done Done false
22014 6415 Acked Read 5 Sent Done Done false
22015 6416 Acked Write 10 Sent Done Done false
22016 6417 Acked Read 2 Sent Done Done false
22017 6418 Acked Read 2 Sent Done Done false
22018 6419 Acked Read 2 Sent Done Done false
22019 6420 Acked Write 3 Sent Done Done false
22020 6421 Acked Write 7 Sent Done Done false
22021 6422 Acked Flush 0 Sent Done Done false
22022 6423 Acked Write 4 Sent Done Done false
22023 6424 Acked Write 10 Sent Done Done false
22024 6425 Acked Read 3 Sent Done Done false
22025 6426 Acked Write 10 Sent Done Done false
22026 6427 Acked Read 1 Sent Done Done false
22027 6428 Acked Write 1 Sent Done Done false
22028 6429 Acked Write 5 Sent Done Done false
22029 6430 Acked Read 6 Sent Done Done false
22030 6431 Acked Write 8 Sent Done Done false
22031 STATES DS:0 DS:1 DS:2 TOTAL
22032 Sent 301 0 0 301
22033 Done 0 301 301 602
22034 Skipped 0 0 0 0
22035 Error 0 0 0 0
22036 Last Flush: Some(JobId(6130)) Some(JobId(6422)) Some(JobId(6422))
22037 Downstairs last five completed: 6130 6129 6128 6127 6126
22038 Upstairs last five completed: 6431 6430 6429 6428 6427
22039 CLIENT: Up:0 ds:301 act:2
220402025-02-28T06:05:52.878ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
220412025-02-28T06:05:53.828ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
220422025-02-28T06:05:54.887ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
220432025-02-28T06:05:55.833ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
22044 [8810] got start action
22045 [8810] I am going to restart
22046 Starting downstairs at port 8810
22047 Make output file at "/tmp/dsc/downstairs-8810.txt"
22048 [8810][0] reports Starting
22049 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1229)
22050 [8810][0] reports Running
22051 ----------------------------------------------------------------
22052 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22053 States: Offline Active Active
22054 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22055 6131 Acked Read 2 Sent Done Done false
22056 6132 Acked Read 4 Sent Done Done false
22057 6133 Acked Flush 0 Sent Done Done false
22058 6134 Acked Read 2 Sent Done Done false
22059 6135 Acked Write 4 Sent Done Done false
22060 6136 Acked Read 9 Sent Done Done false
22061 6137 Acked Flush 0 Sent Done Done false
22062 6138 Acked Write 9 Sent Done Done false
22063 6139 Acked Write 9 Sent Done Done false
22064 6140 Acked Flush 0 Sent Done Done false
22065 6141 Acked Read 1 Sent Done Done false
22066 6142 Acked Flush 0 Sent Done Done false
22067 6143 Acked Read 7 Sent Done Done false
22068 6144 Acked Write 5 Sent Done Done false
22069 6145 Acked Write 4 Sent Done Done false
22070 6146 Acked Write 1 Sent Done Done false
22071 6147 Acked Write 10 Sent Done Done false
22072 6148 Acked Write 5 Sent Done Done false
22073 6149 Acked Flush 0 Sent Done Done false
22074 6150 Acked Write 7 Sent Done Done false
22075 6151 Acked Read 4 Sent Done Done false
22076 6152 Acked Write 5 Sent Done Done false
22077 6153 Acked Read 8 Sent Done Done false
22078 6154 Acked Read 6 Sent Done Done false
22079 6155 Acked Read 8 Sent Done Done false
22080 6156 Acked Write 7 Sent Done Done false
22081 6157 Acked Read 1 Sent Done Done false
22082 6158 Acked Read 5 Sent Done Done false
22083 6159 Acked Read 9 Sent Done Done false
22084 6160 Acked Write 6 Sent Done Done false
22085 6161 Acked Write 6 Sent Done Done false
22086 6162 Acked Read 9 Sent Done Done false
22087 6163 Acked Read 5 Sent Done Done false
22088 6164 Acked Read 2 Sent Done Done false
22089 6165 Acked Read 2 Sent Done Done false
22090 6166 Acked Write 3 Sent Done Done false
22091 6167 Acked Write 10 Sent Done Done false
22092 6168 Acked Read 5 Sent Done Done false
22093 6169 Acked Flush 0 Sent Done Done false
22094 6170 Acked Flush 0 Sent Done Done false
22095 6171 Acked Read 3 Sent Done Done false
22096 6172 Acked Write 6 Sent Done Done false
22097 6173 Acked Write 3 Sent Done Done false
22098 6174 Acked Read 8 Sent Done Done false
22099 6175 Acked Write 4 Sent Done Done false
22100 6176 Acked Read 9 Sent Done Done false
22101 6177 Acked Write 2 Sent Done Done false
22102 6178 Acked Read 1 Sent Done Done false
22103 6179 Acked Write 6 Sent Done Done false
22104 6180 Acked Read 9 Sent Done Done false
22105 6181 Acked Read 8 Sent Done Done false
22106 6182 Acked Read 9 Sent Done Done false
22107 6183 Acked Write 7 Sent Done Done false
22108 6184 Acked Read 4 Sent Done Done false
22109 6185 Acked Write 4 Sent Done Done false
22110 6186 Acked Read 9 Sent Done Done false
22111 6187 Acked Flush 0 Sent Done Done false
22112 6188 Acked Write 8 Sent Done Done false
22113 6189 Acked Read 8 Sent Done Done false
22114 6190 Acked Read 5 Sent Done Done false
22115 6191 Acked Read 5 Sent Done Done false
22116 6192 Acked Read 9 Sent Done Done false
22117 6193 Acked Read 8 Sent Done Done false
22118 6194 Acked Flush 0 Sent Done Done false
22119 6195 Acked Write 7 Sent Done Done false
22120 6196 Acked Flush 0 Sent Done Done false
22121 6197 Acked Flush 0 Sent Done Done false
22122 6198 Acked Flush 0 Sent Done Done false
22123 6199 Acked Read 9 Sent Done Done false
22124 6200 Acked Write 9 Sent Done Done false
22125 6201 Acked Write 6 Sent Done Done false
22126 6202 Acked Read 3 Sent Done Done false
22127 6203 Acked Write 3 Sent Done Done false
22128 6204 Acked Write 4 Sent Done Done false
22129 6205 Acked Write 9 Sent Done Done false
22130 6206 Acked Write 9 Sent Done Done false
22131 6207 Acked Write 7 Sent Done Done false
22132 6208 Acked Write 9 Sent Done Done false
22133 6209 Acked Flush 0 Sent Done Done false
22134 6210 Acked Read 9 Sent Done Done false
22135 6211 Acked Read 7 Sent Done Done false
22136 6212 Acked Read 5 Sent Done Done false
22137 6213 Acked Read 1 Sent Done Done false
22138 6214 Acked Write 9 Sent Done Done false
22139 6215 Acked Write 8 Sent Done Done false
22140 6216 Acked Flush 0 Sent Done Done false
22141 6217 Acked Write 6 Sent Done Done false
22142 6218 Acked Write 1 Sent Done Done false
22143 6219 Acked Read 1 Sent Done Done false
22144 6220 Acked Write 10 Sent Done Done false
22145 6221 Acked Read 7 Sent Done Done false
22146 6222 Acked Write 6 Sent Done Done false
22147 6223 Acked Read 1 Sent Done Done false
22148 6224 Acked Read 4 Sent Done Done false
22149 6225 Acked Read 1 Sent Done Done false
22150 6226 Acked Read 6 Sent Done Done false
22151 6227 Acked Read 1 Sent Done Done false
22152 6228 Acked Read 2 Sent Done Done false
22153 6229 Acked Flush 0 Sent Done Done false
22154 6230 Acked Read 8 Sent Done Done false
22155 6231 Acked Flush 0 Sent Done Done false
22156 6232 Acked Read 7 Sent Done Done false
22157 6233 Acked Read 8 Sent Done Done false
22158 6234 Acked Read 5 Sent Done Done false
22159 6235 Acked Read 1 Sent Done Done false
22160 6236 Acked Read 7 Sent Done Done false
22161 6237 Acked Write 2 Sent Done Done false
22162 6238 Acked Read 8 Sent Done Done false
22163 6239 Acked Read 1 Sent Done Done false
22164 6240 Acked Write 3 Sent Done Done false
22165 6241 Acked Write 1 Sent Done Done false
22166 6242 Acked Read 6 Sent Done Done false
22167 6243 Acked Read 2 Sent Done Done false
22168 6244 Acked Write 7 Sent Done Done false
22169 6245 Acked Write 2 Sent Done Done false
22170 6246 Acked Read 8 Sent Done Done false
22171 6247 Acked Flush 0 Sent Done Done false
22172 6248 Acked Write 2 Sent Done Done false
22173 6249 Acked Read 1 Sent Done Done false
22174 6250 Acked Write 2 Sent Done Done false
22175 6251 Acked Read 2 Sent Done Done false
22176 6252 Acked Write 5 Sent Done Done false
22177 6253 Acked Write 3 Sent Done Done false
22178 6254 Acked Write 8 Sent Done Done false
22179 6255 Acked Read 6 Sent Done Done false
22180 6256 Acked Read 8 Sent Done Done false
22181 6257 Acked Write 8 Sent Done Done false
22182 6258 Acked Write 3 Sent Done Done false
22183 6259 Acked Read 2 Sent Done Done false
22184 6260 Acked Flush 0 Sent Done Done false
22185 6261 Acked Write 9 Sent Done Done false
22186 6262 Acked Read 7 Sent Done Done false
22187 6263 Acked Read 7 Sent Done Done false
22188 6264 Acked Read 3 Sent Done Done false
22189 6265 Acked Write 4 Sent Done Done false
22190 6266 Acked Read 6 Sent Done Done false
22191 6267 Acked Read 6 Sent Done Done false
22192 6268 Acked Read 4 Sent Done Done false
22193 6269 Acked Read 10 Sent Done Done false
22194 6270 Acked Write 1 Sent Done Done false
22195 6271 Acked Write 2 Sent Done Done false
22196 6272 Acked Write 8 Sent Done Done false
22197 6273 Acked Write 4 Sent Done Done false
22198 6274 Acked Read 1 Sent Done Done false
22199 6275 Acked Write 10 Sent Done Done false
22200 6276 Acked Write 9 Sent Done Done false
22201 6277 Acked Write 2 Sent Done Done false
22202 6278 Acked Read 4 Sent Done Done false
22203 6279 Acked Flush 0 Sent Done Done false
22204 6280 Acked Read 4 Sent Done Done false
22205 6281 Acked Read 3 Sent Done Done false
22206 6282 Acked Read 4 Sent Done Done false
22207 6283 Acked Read 10 Sent Done Done false
22208 6284 Acked Write 3 Sent Done Done false
22209 6285 Acked Write 9 Sent Done Done false
22210 6286 Acked Read 2 Sent Done Done false
22211 6287 Acked Read 1 Sent Done Done false
22212 6288 Acked Read 2 Sent Done Done false
22213 6289 Acked Write 9 Sent Done Done false
22214 6290 Acked Read 3 Sent Done Done false
22215 6291 Acked Write 10 Sent Done Done false
22216 6292 Acked Read 7 Sent Done Done false
22217 6293 Acked Write 6 Sent Done Done false
22218 6294 Acked Flush 0 Sent Done Done false
22219 6295 Acked Write 1 Sent Done Done false
22220 6296 Acked Read 10 Sent Done Done false
22221 6297 Acked Read 5 Sent Done Done false
22222 6298 Acked Read 6 Sent Done Done false
22223 6299 Acked Read 5 Sent Done Done false
22224 6300 Acked Write 4 Sent Done Done false
22225 6301 Acked Read 4 Sent Done Done false
22226 6302 Acked Flush 0 Sent Done Done false
22227 6303 Acked Read 7 Sent Done Done false
22228 6304 Acked Read 1 Sent Done Done false
22229 6305 Acked Write 3 Sent Done Done false
22230 6306 Acked Write 10 Sent Done Done false
22231 6307 Acked Read 2 Sent Done Done false
22232 6308 Acked Read 2 Sent Done Done false
22233 6309 Acked Read 3 Sent Done Done false
22234 6310 Acked Write 10 Sent Done Done false
22235 6311 Acked Write 3 Sent Done Done false
22236 6312 Acked Write 3 Sent Done Done false
22237 6313 Acked Read 9 Sent Done Done false
22238 6314 Acked Read 2 Sent Done Done false
22239 6315 Acked Write 4 Sent Done Done false
22240 6316 Acked Read 6 Sent Done Done false
22241 6317 Acked Read 9 Sent Done Done false
22242 6318 Acked Read 8 Sent Done Done false
22243 6319 Acked Read 4 Sent Done Done false
22244 6320 Acked Read 5 Sent Done Done false
22245 6321 Acked Read 6 Sent Done Done false
22246 6322 Acked Read 7 Sent Done Done false
22247 6323 Acked Flush 0 Sent Done Done false
22248 6324 Acked Read 3 Sent Done Done false
22249 6325 Acked Flush 0 Sent Done Done false
22250 6326 Acked Read 7 Sent Done Done false
22251 6327 Acked Read 9 Sent Done Done false
22252 6328 Acked Write 1 Sent Done Done false
22253 6329 Acked Flush 0 Sent Done Done false
22254 6330 Acked Flush 0 Sent Done Done false
22255 6331 Acked Read 1 Sent Done Done false
22256 6332 Acked Flush 0 Sent Done Done false
22257 6333 Acked Read 4 Sent Done Done false
22258 6334 Acked Read 3 Sent Done Done false
22259 6335 Acked Write 6 Sent Done Done false
22260 6336 Acked Write 9 Sent Done Done false
22261 6337 Acked Write 10 Sent Done Done false
22262 6338 Acked Read 1 Sent Done Done false
22263 6339 Acked Flush 0 Sent Done Done false
22264 6340 Acked Read 9 Sent Done Done false
22265 6341 Acked Read 8 Sent Done Done false
22266 6342 Acked Write 2 Sent Done Done false
22267 6343 Acked Read 2 Sent Done Done false
22268 6344 Acked Flush 0 Sent Done Done false
22269 6345 Acked Write 1 Sent Done Done false
22270 6346 Acked Write 5 Sent Done Done false
22271 6347 Acked Read 7 Sent Done Done false
22272 6348 Acked Flush 0 Sent Done Done false
22273 6349 Acked Read 4 Sent Done Done false
22274 6350 Acked Read 1 Sent Done Done false
22275 6351 Acked Flush 0 Sent Done Done false
22276 6352 Acked Read 3 Sent Done Done false
22277 6353 Acked Write 4 Sent Done Done false
22278 6354 Acked Read 9 Sent Done Done false
22279 6355 Acked Flush 0 Sent Done Done false
22280 6356 Acked Write 9 Sent Done Done false
22281 6357 Acked Flush 0 Sent Done Done false
22282 6358 Acked Read 7 Sent Done Done false
22283 6359 Acked Flush 0 Sent Done Done false
22284 6360 Acked Write 1 Sent Done Done false
22285 6361 Acked Read 9 Sent Done Done false
22286 6362 Acked Write 3 Sent Done Done false
22287 6363 Acked Read 3 Sent Done Done false
22288 6364 Acked Read 1 Sent Done Done false
22289 6365 Acked Write 5 Sent Done Done false
22290 6366 Acked Read 6 Sent Done Done false
22291 6367 Acked Read 8 Sent Done Done false
22292 6368 Acked Flush 0 Sent Done Done false
22293 6369 Acked Write 7 Sent Done Done false
22294 6370 Acked Read 7 Sent Done Done false
22295 6371 Acked Read 4 Sent Done Done false
22296 6372 Acked Write 3 Sent Done Done false
22297 6373 Acked Read 6 Sent Done Done false
22298 6374 Acked Read 3 Sent Done Done false
22299 6375 Acked Flush 0 Sent Done Done false
22300 6376 Acked Flush 0 Sent Done Done false
22301 6377 Acked Write 8 Sent Done Done false
22302 6378 Acked Flush 0 Sent Done Done false
22303 6379 Acked Write 6 Sent Done Done false
22304 6380 Acked Read 8 Sent Done Done false
22305 6381 Acked Read 2 Sent Done Done false
22306 6382 Acked Read 7 Sent Done Done false
22307 6383 Acked Write 6 Sent Done Done false
22308 6384 Acked Read 10 Sent Done Done false
22309 6385 Acked Read 10 Sent Done Done false
22310 6386 Acked Read 5 Sent Done Done false
22311 6387 Acked Write 1 Sent Done Done false
22312 6388 Acked Write 6 Sent Done Done false
22313 6389 Acked Write 1 Sent Done Done false
22314 6390 Acked Read 3 Sent Done Done false
22315 6391 Acked Read 10 Sent Done Done false
22316 6392 Acked Read 4 Sent Done Done false
22317 6393 Acked Read 5 Sent Done Done false
22318 6394 Acked Read 6 Sent Done Done false
22319 6395 Acked Write 10 Sent Done Done false
22320 6396 Acked Read 7 Sent Done Done false
22321 6397 Acked Read 10 Sent Done Done false
22322 6398 Acked Read 2 Sent Done Done false
22323 6399 Acked Write 8 Sent Done Done false
22324 6400 Acked Write 3 Sent Done Done false
22325 6401 Acked Read 5 Sent Done Done false
22326 6402 Acked Read 7 Sent Done Done false
22327 6403 Acked Write 3 Sent Done Done false
22328 6404 Acked Write 9 Sent Done Done false
22329 6405 Acked Write 6 Sent Done Done false
22330 6406 Acked Read 7 Sent Done Done false
22331 6407 Acked Write 8 Sent Done Done false
22332 6408 Acked Read 6 Sent Done Done false
22333 6409 Acked Write 8 Sent Done Done false
22334 6410 Acked Write 10 Sent Done Done false
22335 6411 Acked Write 6 Sent Done Done false
22336 6412 Acked Flush 0 Sent Done Done false
22337 6413 Acked Write 4 Sent Done Done false
22338 6414 Acked Write 8 Sent Done Done false
22339 6415 Acked Read 5 Sent Done Done false
22340 6416 Acked Write 10 Sent Done Done false
22341 6417 Acked Read 2 Sent Done Done false
22342 6418 Acked Read 2 Sent Done Done false
22343 6419 Acked Read 2 Sent Done Done false
22344 6420 Acked Write 3 Sent Done Done false
22345 6421 Acked Write 7 Sent Done Done false
22346 6422 Acked Flush 0 Sent Done Done false
22347 6423 Acked Write 4 Sent Done Done false
22348 6424 Acked Write 10 Sent Done Done false
22349 6425 Acked Read 3 Sent Done Done false
22350 6426 Acked Write 10 Sent Done Done false
22351 6427 Acked Read 1 Sent Done Done false
22352 6428 Acked Write 1 Sent Done Done false
22353 6429 Acked Write 5 Sent Done Done false
22354 6430 Acked Read 6 Sent Done Done false
22355 6431 Acked Write 8 Sent Done Done false
22356 STATES DS:0 DS:1 DS:2 TOTAL
22357 Sent 301 0 0 301
22358 Done 0 301 301 602
22359 Skipped 0 0 0 0
22360 Error 0 0 0 0
22361 Last Flush: Some(JobId(6130)) Some(JobId(6422)) Some(JobId(6422))
22362 Downstairs last five completed: 6130 6129 6128 6127 6126
22363 Upstairs last five completed: 6431 6430 6429 6428 6427
22364 CLIENT: Up:0 ds:301 act:2
223652025-02-28T06:05:57.135ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223662025-02-28T06:05:57.837ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223672025-02-28T06:05:58.139ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223682025-02-28T06:05:58.140ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51046) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223692025-02-28T06:05:58.140ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223702025-02-28T06:05:58.141ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223712025-02-28T06:05:58.141ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223722025-02-28T06:05:58.141ZINFOcrucible: send last flush ID to this DS: Some(JobId(6130)) = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223732025-02-28T06:05:58.141ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223742025-02-28T06:05:58.141ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(6130)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223752025-02-28T06:05:58.141ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(6130)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
22376 ----------------------------------------------------------------
22377 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22378 Downstairs last five completed: 6432 6431 6430 6429 6428
22379 Upstairs last five completed: 6432 6431 6430 6429 6428
22380 CLIENT: Up:0 ds:0 act:3
22381 Replay: All jobs finished
22382 Main task has work to do, go find it
22383 got dsc Stop(1)
22384 stop 1
22385 Feb 28 06:06:00.381 [8820] Got stop action so:false kr:false
22386 INFO request completed, latency_us: 334, response_code: 204, uri: /stop/cid/1, method: POST, req_id: ef22cb81-e4ca-4eb4-ac79-bb55dcf4e66e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
223872025-02-28T06:06:00.382ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223882025-02-28T06:06:00.382ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223892025-02-28T06:06:00.382ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
223902025-02-28T06:06:00.382ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
22391 Feb 28 06:06:00.382 INFO request completed, latency_us: 225, response_code: 200, uri: /state/cid/1, method: GET, req_id: 331cfa79-0187-40f3-8a86-720e79b66881, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
22392 [8820] Exited with: ExitStatus(unix_wait_status(9))
22393 [8820][1] reports Exit
22394 Feb 28 06:06:04.384 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/1, method: GET, req_id: 00ad2030-d195-45ad-a3ec-d8ca45201fd3, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
22395 001/300 Write block 1641 len 12288 data: 3 4 3
22396 002/300 Write block 2423 len 24576 data: 4 5 4 4 4 3
22397 003/300 Write block 3682 len 12288 data: 4 4 3
22398 004/300 Flush
22399 005/300 Read block 3304 len 4096
22400 006/300 Read block 3139 len 8192
22401 007/300 Write block 334 len 4096 data: 4
22402 008/300 Write block 2494 len 16384 data: 3 4 4 4
22403 009/300 Read block 1128 len 36864
22404 010/300 Read block 2784 len 24576
22405 011/300 Write block 1886 len 40960 data: 3 3 3 4 4 7 6 6 7 7
22406 012/300 Read block 4819 len 28672
22407 013/300 Write block 4893 len 40960 data: 6 6 6 6 6 7 6 6 5 6
22408 014/300 Read block 424 len 8192
22409 015/300 Read block 3599 len 8192
22410 016/300 Write block 369 len 40960 data: 3 3 2 3 5 5 4 4 5 6
22411 017/300 Read block 161 len 32768
22412 018/300 Read block 142 len 40960
22413 019/300 Write block 3389 len 12288 data: 2 2 2
22414 020/300 Read block 3962 len 4096
22415 021/300 Flush
22416 022/300 Read block 1785 len 16384
22417 023/300 Read block 4540 len 40960
22418 024/300 Write block 650 len 12288 data: 3 3 3
22419 025/300 Flush
22420 026/300 Read block 1512 len 4096
22421 027/300 Write block 3983 len 16384 data: 3 2 3 6
22422 028/300 Write block 3144 len 20480 data: 6 5 4 2 2
22423 029/300 Read block 4345 len 8192
22424 030/300 Write block 2730 len 36864 data: 6 5 5 6 4 5 5 4 3
22425 031/300 Write block 3089 len 24576 data: 3 4 4 4 5 4
22426 032/300 Write block 2522 len 24576 data: 7 8 9 8 6 6
22427 033/300 Read block 4183 len 16384
22428 034/300 Read block 2850 len 4096
22429 035/300 Write block 821 len 12288 data: 3 3 4
22430 036/300 Write block 1999 len 32768 data: 3 5 4 5 6 4 4 4
22431 037/300 Read block 105 len 36864
22432 038/300 Read block 1161 len 36864
22433 039/300 Write block 1865 len 36864 data: 3 3 3 3 4 3 3 4 4
22434 040/300 Read block 1813 len 16384
22435 041/300 Read block 1368 len 28672
22436 042/300 Read block 1709 len 40960
22437 043/300 Write block 2003 len 8192 data: 7 5
22438 044/300 Write block 987 len 40960 data: 4 4 4 4 5 3 3 3 2 2
22439 045/300 Flush
22440 046/300 Read block 2131 len 32768
22441 047/300 Write block 839 len 16384 data: 3 3 4 4
22442 048/300 Flush
22443 049/300 Read block 549 len 20480
22444 050/300 Write block 1663 len 36864 data: 5 5 6 6 4 5 5 4 4
22445 051/300 Read block 4421 len 36864
22446 052/300 Read block 2651 len 28672
22447 053/300 Read block 4196 len 16384
22448 054/300 Write block 3493 len 12288 data: 5 4 4
22449 055/300 Write block 3009 len 28672 data: 4 4 4 3 3 3 4
22450 056/300 Read block 488 len 8192
22451 057/300 Read block 1374 len 24576
22452 058/300 Read block 1205 len 12288
22453 059/300 Read block 114 len 12288
22454 060/300 Write block 2974 len 28672 data: 2 3 4 4 4 4 4
22455 061/300 Read block 1008 len 28672
22456 062/300 Read block 3403 len 4096
22457 063/300 Write block 4426 len 20480 data: 3 5 8 8 8
22458 064/300 Read block 927 len 28672
22459 065/300 Write block 3645 len 12288 data: 3 3 3
22460 066/300 Write block 4464 len 36864 data: 5 4 3 4 4 4 5 3 3
22461 067/300 Read block 1633 len 16384
22462 068/300 Write block 4649 len 40960 data: 6 5 5 7 7 7 6 4 3 3
22463 069/300 Write block 1600 len 40960 data: 2 2 2 2 3 3 3 2 4 4
22464 070/300 Read block 1388 len 4096
22465 071/300 Write block 152 len 24576 data: 4 5 5 4 4 2
22466 072/300 Write block 4886 len 40960 data: 7 10 9 10 9 8 6 7 7 7
22467 073/300 Read block 1308 len 40960
22468 074/300 Read block 4856 len 12288
22469 075/300 Flush
22470 076/300 Read block 2815 len 36864
22471 077/300 Read block 2873 len 28672
22472 078/300 Write block 1960 len 24576 data: 2 2 2 2 3 3
22473 079/300 Read block 3379 len 4096
22474 080/300 Read block 2386 len 36864
22475 081/300 Read block 4988 len 20480
22476 082/300 Flush
22477 083/300 Write block 4950 len 36864 data: 3 5 4 5 6 5 5 6 7
22478 084/300 Read block 4875 len 12288
22479 085/300 Read block 3826 len 40960
22480 086/300 Read block 3837 len 24576
22481 087/300 Write block 1623 len 36864 data: 3 6 5 5 5 5 5 5 5
22482 088/300 Write block 1821 len 28672 data: 5 4 3 3 4 4 5
22483 089/300 Write block 445 len 20480 data: 10 7 8 8 9
22484 090/300 Flush
22485 091/300 Flush
22486 092/300 Write block 2788 len 28672 data: 4 5 5 4 4 3 5
22487 093/300 Read block 2597 len 28672
22488 094/300 Read block 1150 len 24576
22489 095/300 Read block 957 len 4096
22490 096/300 Flush
22491 097/300 Write block 4312 len 12288 data: 3 3 3
22492 098/300 Write block 124 len 36864 data: 4 3 4 5 5 5 5 4 3
22493 099/300 Read block 4840 len 36864
22494 100/300 Write block 1134 len 20480 data: 4 4 5 4 4
22495 101/300 Read block 4568 len 28672
22496 102/300 Read block 3620 len 16384
22497 103/300 Write block 4266 len 8192 data: 4 5
22498 104/300 Flush
22499 105/300 Write block 321 len 12288 data: 5 5 5
22500 106/300 Read block 4511 len 28672
22501 107/300 Write block 436 len 36864 data: 7 6 7 7 7 7 9 9 10
22502 108/300 Read block 4243 len 12288
22503 109/300 Write block 73 len 40960 data: 3 3 6 5 5 5 5 4 5 4
22504 110/300 Read block 2190 len 24576
22505 111/300 Write block 767 len 4096 data: 2
22506 112/300 Read block 2744 len 12288
22507 113/300 Read block 4652 len 28672
22508 114/300 Write block 3934 len 28672 data: 6 6 7 6 8 9 9
22509 115/300 Write block 718 len 4096 data: 6
22510 116/300 Read block 2847 len 20480
22511 117/300 Write block 1873 len 20480 data: 5 6 6 6 8
22512 118/300 Read block 736 len 32768
22513 119/300 Read block 2127 len 4096
22514 120/300 Read block 2342 len 28672
22515 121/300 Read block 629 len 40960
22516 122/300 Read block 4172 len 12288
22517 123/300 Flush
22518 124/300 Flush
22519 125/300 Write block 4036 len 20480 data: 5 5 7 7 6
22520 126/300 Read block 3978 len 28672
22521 127/300 Read block 4846 len 4096
22522 128/300 Flush
22523 129/300 Read block 2799 len 28672
22524 130/300 Write block 2755 len 16384 data: 3 3 3 4
22525 131/300 Read block 2143 len 4096
22526 132/300 Read block 141 len 24576
22527 133/300 Read block 3906 len 16384
22528 134/300 Read block 72 len 32768
22529 135/300 Write block 2173 len 8192 data: 4 5
22530 136/300 Read block 900 len 40960
22531 137/300 Write block 4158 len 20480 data: 7 9 8 6 6
22532 138/300 Write block 4683 len 40960 data: 4 4 3 3 4 2 2 2 2 3
22533 139/300 Read block 1594 len 28672
22534 140/300 Read block 1719 len 24576
22535 141/300 Write block 1368 len 20480 data: 5 3 3 4 4
22536 142/300 Read block 643 len 20480
22537 143/300 Read block 1010 len 40960
22538 144/300 Read block 4716 len 4096
22539 145/300 Write block 3340 len 16384 data: 6 7 6 8
22540 146/300 Read block 4742 len 32768
22541 147/300 Write block 3731 len 4096 data: 6
22542 148/300 Write block 3343 len 4096 data: 9
22543 149/300 Flush
22544 150/300 Write block 2715 len 32768 data: 7 8 8 6 4 4 3 2
22545 151/300 Write block 3690 len 8192 data: 4 4
22546 152/300 Read block 3324 len 32768
22547 153/300 Read block 1354 len 36864
22548 154/300 Write block 2803 len 28672 data: 4 4 5 6 6 5 5
22549 155/300 Write block 4111 len 4096 data: 4
22550 156/300 Read block 353 len 28672
22551 157/300 Read block 4091 len 40960
22552 158/300 Write block 804 len 40960 data: 4 5 5 4 4 4 4 4 7 8
22553 159/300 Read block 1889 len 8192
22554 160/300 Write block 378 len 40960 data: 7 6 7 6 7 7 5 5 5 2
22555 161/300 Read block 2188 len 28672
22556 162/300 Read block 1772 len 8192
22557 163/300 Write block 1499 len 20480 data: 2 2 3 2 3
22558 164/300 Write block 1783 len 32768 data: 3 3 3 2 2 4 5 5
22559 165/300 Write block 4037 len 8192 data: 6 8
22560 166/300 Read block 4174 len 8192
22561 167/300 Write block 4819 len 40960 data: 4 4 4 4 6 5 5 5 5 8
22562 168/300 Flush
22563 169/300 Read block 134 len 24576
22564 170/300 Read block 4974 len 8192
22565 171/300 Read block 2680 len 16384
22566 172/300 Read block 1544 len 16384
22567 173/300 Write block 3256 len 20480 data: 3 3 3 2 2
22568 174/300 Read block 1825 len 4096
22569 175/300 Read block 3658 len 8192
22570 176/300 Write block 2667 len 28672 data: 7 7 9 9 6 8 7
22571 177/300 Read block 4669 len 36864
22572 178/300 Write block 1351 len 40960 data: 5 5 5 4 5 4 4 5 6 6
22573 179/300 Flush
22574 180/300 Read block 4129 len 8192
22575 181/300 Read block 4307 len 4096
22576 182/300 Read block 1272 len 4096
22577 183/300 Read block 1434 len 20480
22578 184/300 Read block 2807 len 8192
22579 185/300 Flush
22580 186/300 Write block 829 len 24576 data: 5 5 5 3 4 4
22581 187/300 Read block 2539 len 8192
22582 188/300 Write block 1106 len 16384 data: 2 2 2 2
22583 189/300 Write block 3366 len 40960 data: 4 3 4 3 3 4 3 4 3 2
22584 190/300 Read block 3340 len 24576
22585 191/300 Read block 1912 len 28672
22586 192/300 Write block 3242 len 4096 data: 3
22587 193/300 Write block 2931 len 20480 data: 4 3 4 4 2
22588 194/300 Write block 1926 len 16384 data: 2 3 3 3
22589 195/300 Read block 3395 len 28672
22590 196/300 Read block 3799 len 40960
22591 197/300 Read block 1776 len 36864
22592 198/300 Read block 1563 len 4096
22593 199/300 Write block 3771 len 16384 data: 2 2 3 3
22594 200/300 Write block 859 len 28672 data: 3 4 4 5 5 5 4
22595 201/300 Write block 2373 len 16384 data: 4 4 5 6
22596 202/300 Read block 3460 len 8192
22597 203/300 Flush
22598 204/300 Write block 2138 len 24576 data: 4 4 5 4 4 4
22599 205/300 Read block 468 len 32768
22600 206/300 Write block 4539 len 12288 data: 3 3 2
22601 207/300 Write block 1417 len 8192 data: 3 3
22602 208/300 Flush
22603 209/300 Write block 394 len 4096 data: 2
22604 210/300 Write block 1410 len 36864 data: 4 4 3 3 4 3 3 4 4
22605 211/300 Read block 4302 len 32768
22606 212/300 Read block 110 len 8192
22607 213/300 Flush
22608 214/300 Read block 2215 len 32768
22609 215/300 Write block 697 len 20480 data: 4 4 4 5 6
22610 216/300 Read block 1147 len 28672
22611 217/300 Read block 3245 len 4096
22612 218/300 Write block 579 len 40960 data: 6 7 6 6 6 4 4 4 2 2
22613 219/300 Write block 2116 len 16384 data: 4 5 5 4
22614 220/300 Flush
22615 221/300 Read block 402 len 16384
22616 222/300 Read block 379 len 40960
22617 223/300 Read block 840 len 36864
22618 224/300 Write block 3373 len 8192 data: 5 4
22619 225/300 Read block 3520 len 12288
22620 226/300 Read block 2321 len 4096
22621 227/300 Read block 4202 len 8192
22622 228/300 Write block 935 len 40960 data: 4 4 5 6 5 5 6 7 6 6
22623 229/300 Write block 823 len 4096 data: 5
22624 230/300 Flush
22625 231/300 Write block 3300 len 8192 data: 4 6
22626 232/300 Flush
22627 233/300 Write block 4909 len 28672 data: 4 3 3 3 5 4 4
22628 234/300 Read block 18 len 4096
22629 235/300 Write block 2693 len 32768 data: 6 6 6 4 4 4 2 3
22630 236/300 Write block 941 len 16384 data: 7 8 7 7
22631 237/300 Read block 1947 len 24576
22632 238/300 Read block 1920 len 24576
22633 239/300 Write block 2604 len 8192 data: 4 3
22634 240/300 Read block 517 len 4096
22635 241/300 Read block 908 len 8192
22636 242/300 Read block 939 len 20480
22637 243/300 Read block 2255 len 12288
22638 244/300 Read block 793 len 24576
22639 245/300 Write block 2582 len 16384 data: 3 4 4 4
22640 246/300 Write block 1338 len 24576 data: 4 4 5 5 5 4
22641 247/300 Read block 1483 len 16384
22642 248/300 Flush
22643 249/300 Read block 2305 len 4096
22644 250/300 Read block 2654 len 20480
22645 251/300 Read block 132 len 4096
22646 252/300 Flush
22647 253/300 Read block 88 len 28672
22648 254/300 Read block 4452 len 4096
22649 255/300 Write block 3332 len 12288 data: 4 3 4
22650 256/300 Read block 2299 len 24576
22651 257/300 Read block 2477 len 36864
22652 258/300 Flush
22653 259/300 Read block 4594 len 12288
22654 260/300 Read block 4872 len 12288
22655 261/300 Flush
22656 262/300 Write block 4868 len 4096 data: 5
22657 263/300 Write block 1913 len 20480 data: 3 3 3 3 3
22658 264/300 Write block 93 len 28672 data: 8 9 10 6 4 3 3
22659 265/300 Read block 434 len 40960
22660 266/300 Write block 2185 len 28672 data: 7 8 6 7 7 8 5
22661 267/300 Write block 4347 len 4096 data: 9
22662 268/300 Read block 1945 len 40960
22663 269/300 Read block 1095 len 28672
22664 270/300 Write block 1408 len 32768 data: 4 5 5 5 4 4 5 4
22665 271/300 Read block 2212 len 32768
22666 272/300 Read block 202 len 32768
22667 273/300 Read block 3122 len 36864
22668 274/300 Write block 1015 len 4096 data: 4
22669 275/300 Write block 1376 len 4096 data: 4
22670 276/300 Read block 2341 len 24576
22671 277/300 Flush
22672 278/300 Read block 4088 len 8192
22673 279/300 Write block 4750 len 16384 data: 2 2 2 2
22674 280/300 Read block 2052 len 8192
22675 281/300 Read block 4201 len 24576
22676 282/300 Write block 1306 len 24576 data: 2 2 4 4 4 4
22677 283/300 Read block 2569 len 40960
22678 284/300 Read block 4584 len 8192
22679 285/300 Read block 4729 len 24576
22680 286/300 Read block 1361 len 24576
22681 287/300 Write block 1762 len 20480 data: 4 4 4 4 4
22682 288/300 Write block 1062 len 20480 data: 3 3 3 5 6
22683 289/300 Read block 1613 len 28672
22684 290/300 Write block 3105 len 36864 data: 6 5 7 8 8 8 7 8 6
22685 291/300 Read block 4836 len 32768
22686 292/300 Write block 1509 len 16384 data: 3 3 3 4
22687 293/300 Read block 591 len 16384
22688 294/300 Write block 4336 len 4096 data: 5
22689 295/300 Write block 1802 len 16384 data: 5 3 2 2
22690 296/300 Read block 3639 len 12288
22691 297/300 Read block 4333 len 28672
22692 298/300 Write block 1477 len 32768 data: 4 4 4 3 3 3 4 4
22693 299/300 Write block 3602 len 4096 data: 2
22694 300/300 Read block 1741 len 24576
22695 301/300 Write block 418 len 24576 data: 9 10 8 9 10 8
22696 Main task has work to do, go find it
22697 got dsc Start(1)
22698 start 1
22699 Feb 28 06:06:04.625 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/1, method: POST, req_id: da75fd40-c177-4014-b28f-1742a2f968d4, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
22700 Replay: started 1, returned:Ok(())
22701 ----------------------------------------------------------------
22702 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22703 States: Active Offline Active
22704 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22705 6433 Acked Write 3 Done Sent Done false
22706 6434 Acked Write 6 Done Sent Done false
22707 6435 Acked Write 3 Done Sent Done false
22708 6436 Acked Flush 0 Done Sent Done false
22709 6437 Acked Read 1 Done Sent Done false
22710 6438 Acked Read 2 Done Sent Done false
22711 6439 Acked Write 1 Done Sent Done false
22712 6440 Acked Write 4 Done Sent Done false
22713 6441 Acked Read 9 Done Sent Done false
22714 6442 Acked Read 6 Done Sent Done false
22715 6443 Acked Write 10 Done Sent Done false
22716 6444 Acked Read 7 Done Sent Done false
22717 6445 Acked Write 10 Done Sent Done false
22718 6446 Acked Read 2 Done Sent Done false
22719 6447 Acked Read 2 Done Sent Done false
22720 6448 Acked Write 10 Done Sent Done false
22721 6449 Acked Read 8 Done Sent Done false
22722 6450 Acked Read 10 Done Sent Done false
22723 6451 Acked Write 3 Done Sent Done false
22724 6452 Acked Read 1 Done Sent Done false
22725 6453 Acked Flush 0 Done Sent Done false
22726 6454 Acked Read 4 Done Sent Done false
22727 6455 Acked Read 10 Done Sent Done false
22728 6456 Acked Write 3 Done Sent Done false
22729 6457 Acked Flush 0 Done Sent Done false
22730 6458 Acked Read 1 Done Sent Done false
22731 6459 Acked Write 4 Done Sent Done false
22732 6460 Acked Write 5 Done Sent Done false
22733 6461 Acked Read 2 Done Sent Done false
22734 6462 Acked Write 9 Done Sent Done false
22735 6463 Acked Write 6 Done Sent Done false
22736 6464 Acked Write 6 Done Sent Done false
22737 6465 Acked Read 4 Done Sent Done false
22738 6466 Acked Read 1 Done Sent Done false
22739 6467 Acked Write 3 Done Sent Done false
22740 6468 Acked Write 8 Done Sent Done false
22741 6469 Acked Read 9 Done Sent Done false
22742 6470 Acked Read 9 Done Sent Done false
22743 6471 Acked Write 9 Done Sent Done false
22744 6472 Acked Read 4 Done Sent Done false
22745 6473 Acked Read 7 Done Sent Done false
22746 6474 Acked Read 10 Done Sent Done false
22747 6475 Acked Write 2 Done Sent Done false
22748 6476 Acked Write 10 Done Sent Done false
22749 6477 Acked Flush 0 Done Sent Done false
22750 6478 Acked Read 8 Done Sent Done false
22751 6479 Acked Write 4 Done Sent Done false
22752 6480 Acked Flush 0 Done Sent Done false
22753 6481 Acked Read 5 Done Sent Done false
22754 6482 Acked Write 9 Done Sent Done false
22755 6483 Acked Read 9 Done Sent Done false
22756 6484 Acked Read 7 Done Sent Done false
22757 6485 Acked Read 4 Done Sent Done false
22758 6486 Acked Write 3 Done Sent Done false
22759 6487 Acked Write 7 Done Sent Done false
22760 6488 Acked Read 2 Done Sent Done false
22761 6489 Acked Read 6 Done Sent Done false
22762 6490 Acked Read 3 Done Sent Done false
22763 6491 Acked Read 3 Done Sent Done false
22764 6492 Acked Write 7 Done Sent Done false
22765 6493 Acked Read 7 Done Sent Done false
22766 6494 Acked Read 1 Done Sent Done false
22767 6495 Acked Write 5 Done Sent Done false
22768 6496 Acked Read 7 Done Sent Done false
22769 6497 Acked Write 3 Done Sent Done false
22770 6498 Acked Write 9 Done Sent Done false
22771 6499 Acked Read 4 Done Sent Done false
22772 6500 Acked Write 10 Done Sent Done false
22773 6501 Acked Write 10 Done Sent Done false
22774 6502 Acked Read 1 Done Sent Done false
22775 6503 Acked Write 6 Done Sent Done false
22776 6504 Acked Write 10 Done Sent Done false
22777 6505 Acked Read 10 Done Sent Done false
22778 6506 Acked Read 3 Done Sent Done false
22779 6507 Acked Flush 0 Done Sent Done false
22780 6508 Acked Read 9 Done Sent Done false
22781 6509 Acked Read 7 Done Sent Done false
22782 6510 Acked Write 6 Done Sent Done false
22783 6511 Acked Read 1 Done Sent Done false
22784 6512 Acked Read 9 Done Sent Done false
22785 6513 Acked Read 5 Done Sent Done false
22786 6514 Acked Flush 0 Done Sent Done false
22787 6515 Acked Write 9 Done Sent Done false
22788 6516 Acked Read 3 Done Sent Done false
22789 6517 Acked Read 10 Done Sent Done false
22790 6518 Acked Read 6 Done Sent Done false
22791 6519 Acked Write 9 Done Sent Done false
22792 6520 Acked Write 7 Done Sent Done false
22793 6521 Acked Write 5 Done Sent Done false
22794 6522 Acked Flush 0 Done Sent Done false
22795 6523 Acked Flush 0 Done Sent Done false
22796 6524 Acked Write 7 Done Sent Done false
22797 6525 Acked Read 7 Done Sent Done false
22798 6526 Acked Read 6 Done Sent Done false
22799 6527 Acked Read 1 Done Sent Done false
22800 6528 Acked Flush 0 Done Sent Done false
22801 6529 Acked Write 3 Done Sent Done false
22802 6530 Acked Write 9 Done Sent Done false
22803 6531 Acked Read 9 Done Sent Done false
22804 6532 Acked Write 5 Done Sent Done false
22805 6533 Acked Read 7 Done Sent Done false
22806 6534 Acked Read 4 Done Sent Done false
22807 6535 Acked Write 2 Done Sent Done false
22808 6536 Acked Flush 0 Done Sent Done false
22809 6537 Acked Write 3 Done Sent Done false
22810 6538 Acked Read 7 Done Sent Done false
22811 6539 Acked Write 9 Done Sent Done false
22812 6540 Acked Read 3 Done Sent Done false
22813 6541 Acked Write 10 Done Sent Done false
22814 6542 Acked Read 6 Done Sent Done false
22815 6543 Acked Write 1 Done Sent Done false
22816 6544 Acked Read 3 Done Sent Done false
22817 6545 Acked Read 7 Done Sent Done false
22818 6546 Acked Write 7 Done Sent Done false
22819 6547 Acked Write 1 Done Sent Done false
22820 6548 Acked Read 5 Done Sent Done false
22821 6549 Acked Write 5 Done Sent Done false
22822 6550 Acked Read 8 Done Sent Done false
22823 6551 Acked Read 1 Done Sent Done false
22824 6552 Acked Read 7 Done Sent Done false
22825 6553 Acked Read 10 Done Sent Done false
22826 6554 Acked Read 3 Done Sent Done false
22827 6555 Acked Flush 0 Done Sent Done false
22828 6556 Acked Flush 0 Done Sent Done false
22829 6557 Acked Write 5 Done Sent Done false
22830 6558 Acked Read 7 Done Sent Done false
22831 6559 Acked Read 1 Done Sent Done false
22832 6560 Acked Flush 0 Done Sent Done false
22833 6561 Acked Read 7 Done Sent Done false
22834 6562 Acked Write 4 Done Sent Done false
22835 6563 Acked Read 1 Done Sent Done false
22836 6564 Acked Read 6 Done Sent Done false
22837 6565 Acked Read 4 Done Sent Done false
22838 6566 Acked Read 8 Done Sent Done false
22839 6567 Acked Write 2 Done Sent Done false
22840 6568 Acked Read 10 Done Sent Done false
22841 6569 Acked Write 5 Done Sent Done false
22842 6570 Acked Write 10 Done Sent Done false
22843 6571 Acked Read 7 Done Sent Done false
22844 6572 Acked Read 6 Done Sent Done false
22845 6573 Acked Write 5 Done Sent Done false
22846 6574 Acked Read 5 Done Sent Done false
22847 6575 Acked Read 10 Done Sent Done false
22848 6576 Acked Read 1 Done Sent Done false
22849 6577 Acked Write 4 Done Sent Done false
22850 6578 Acked Read 8 Done Sent Done false
22851 6579 Acked Write 1 Done Sent Done false
22852 6580 Acked Write 1 Done Sent Done false
22853 6581 Acked Flush 0 Done Sent Done false
22854 6582 Acked Write 8 Done Sent Done false
22855 6583 Acked Write 2 Done Sent Done false
22856 6584 Acked Read 8 Done Sent Done false
22857 6585 Acked Read 9 Done Sent Done false
22858 6586 Acked Write 7 Done Sent Done false
22859 6587 Acked Write 1 Done Sent Done false
22860 6588 Acked Read 7 Done Sent Done false
22861 6589 Acked Read 10 Done Sent Done false
22862 6590 Acked Write 10 Done Sent Done false
22863 6591 Acked Read 2 Done Sent Done false
22864 6592 Acked Write 10 Done Sent Done false
22865 6593 Acked Read 7 Done Sent Done false
22866 6594 Acked Read 2 Done Sent Done false
22867 6595 Acked Write 5 Done Sent Done false
22868 6596 Acked Write 8 Done Sent Done false
22869 6597 Acked Write 2 Done Sent Done false
22870 6598 Acked Read 2 Done Sent Done false
22871 6599 Acked Write 10 Done Sent Done false
22872 6600 Acked Flush 0 Done Sent Done false
22873 6601 Acked Read 6 Done Sent Done false
22874 6602 Acked Read 2 Done Sent Done false
22875 6603 Acked Read 4 Done Sent Done false
22876 6604 Acked Read 4 Done Sent Done false
22877 6605 Acked Write 5 Done Sent Done false
22878 6606 Acked Read 1 Done Sent Done false
22879 6607 Acked Read 2 Done Sent Done false
22880 6608 Acked Write 7 Done Sent Done false
22881 6609 Acked Read 9 Done Sent Done false
22882 6610 Acked Write 10 Done Sent Done false
22883 6611 Acked Flush 0 Done Sent Done false
22884 6612 Acked Read 2 Done Sent Done false
22885 6613 Acked Read 1 Done Sent Done false
22886 6614 Acked Read 1 Done Sent Done false
22887 6615 Acked Read 5 Done Sent Done false
22888 6616 Acked Read 2 Done Sent Done false
22889 6617 Acked Flush 0 Done Sent Done false
22890 6618 Acked Write 6 Done Sent Done false
22891 6619 Acked Read 2 Done Sent Done false
22892 6620 Acked Write 4 Done Sent Done false
22893 6621 Acked Write 10 Done Sent Done false
22894 6622 Acked Read 6 Done Sent Done false
22895 6623 Acked Read 7 Done Sent Done false
22896 6624 Acked Write 1 Done Sent Done false
22897 6625 Acked Write 5 Done Sent Done false
22898 6626 Acked Write 4 Done Sent Done false
22899 6627 Acked Read 7 Done Sent Done false
22900 6628 Acked Read 10 Done Sent Done false
22901 6629 Acked Read 9 Done Sent Done false
22902 6630 Acked Read 1 Done Sent Done false
22903 6631 Acked Write 4 Done Sent Done false
22904 6632 Acked Write 7 Done Sent Done false
22905 6633 Acked Write 4 Done Sent Done false
22906 6634 Acked Read 2 Done Sent Done false
22907 6635 Acked Flush 0 Done Sent Done false
22908 6636 Acked Write 6 Done Sent Done false
22909 6637 Acked Read 8 Done Sent Done false
22910 6638 Acked Write 3 Done Sent Done false
22911 6639 Acked Write 2 Done Sent Done false
22912 6640 Acked Flush 0 Done Sent Done false
22913 6641 Acked Write 1 Done Sent Done false
22914 6642 Acked Write 9 Done Sent Done false
22915 6643 Acked Read 8 Done Sent Done false
22916 6644 Acked Read 2 Done Sent Done false
22917 6645 Acked Flush 0 Done Sent Done false
22918 6646 Acked Read 8 Done Sent Done false
22919 6647 Acked Write 5 Done Sent Done false
22920 6648 Acked Read 7 Done Sent Done false
22921 6649 Acked Read 1 Done Sent Done false
22922 6650 Acked Write 10 Done Sent Done false
22923 6651 Acked Write 4 Done Sent Done false
22924 6652 Acked Flush 0 Done Sent Done false
22925 6653 Acked Read 4 Done Sent Done false
22926 6654 Acked Read 10 Done Sent Done false
22927 6655 Acked Read 9 Done Sent Done false
22928 6656 Acked Write 2 Done Sent Done false
22929 6657 Acked Read 3 Done Sent Done false
22930 6658 Acked Read 1 Done Sent Done false
22931 6659 Acked Read 2 Done Sent Done false
22932 6660 Acked Write 10 Done Sent Done false
22933 6661 Acked Write 1 Done Sent Done false
22934 6662 Acked Flush 0 Done Sent Done false
22935 6663 Acked Write 2 Done Sent Done false
22936 6664 Acked Flush 0 Done Sent Done false
22937 6665 Acked Write 7 Done Sent Done false
22938 6666 Acked Read 1 Done Sent Done false
22939 6667 Acked Write 8 Done Sent Done false
22940 6668 Acked Write 4 Done Sent Done false
22941 6669 Acked Read 6 Done Sent Done false
22942 6670 Acked Read 6 Done Sent Done false
22943 6671 Acked Write 2 Done Sent Done false
22944 6672 Acked Read 1 Done Sent Done false
22945 6673 Acked Read 2 Done Sent Done false
22946 6674 Acked Read 5 Done Sent Done false
22947 6675 Acked Read 3 Done Sent Done false
22948 6676 Acked Read 6 Done Sent Done false
22949 6677 Acked Write 4 Done Sent Done false
22950 6678 Acked Write 6 Done Sent Done false
22951 6679 Acked Read 4 Done Sent Done false
22952 6680 Acked Flush 0 Done Sent Done false
22953 6681 Acked Read 1 Done Sent Done false
22954 6682 Acked Read 5 Done Sent Done false
22955 6683 Acked Read 1 Done Sent Done false
22956 6684 Acked Flush 0 Done Sent Done false
22957 6685 Acked Read 7 Done Sent Done false
22958 6686 Acked Read 1 Done Sent Done false
22959 6687 Acked Write 3 Done Sent Done false
22960 6688 Acked Read 6 Done Sent Done false
22961 6689 Acked Read 9 Done Sent Done false
22962 6690 Acked Flush 0 Done Sent Done false
22963 6691 Acked Read 3 Done Sent Done false
22964 6692 Acked Read 3 Done Sent Done false
22965 6693 Acked Flush 0 Done Sent Done false
22966 6694 Acked Write 1 Done Sent Done false
22967 6695 Acked Write 5 Done Sent Done false
22968 6696 Acked Write 7 Done Sent Done false
22969 6697 Acked Read 10 Done Sent Done false
22970 6698 Acked Write 7 Done Sent Done false
22971 6699 Acked Write 1 Done Sent Done false
22972 6700 Acked Read 10 Done Sent Done false
22973 6701 Acked Read 7 Done Sent Done false
22974 6702 Acked Write 8 Done Sent Done false
22975 6703 Acked Read 8 Done Sent Done false
22976 6704 Acked Read 8 Done Sent Done false
22977 6705 Acked Read 9 Done Sent Done false
22978 6706 Acked Write 1 Done Sent Done false
22979 6707 Acked Write 1 Done Sent Done false
22980 6708 Acked Read 6 Done Sent Done false
22981 6709 Acked Flush 0 Done Sent Done false
22982 6710 Acked Read 2 Done Sent Done false
22983 6711 Acked Write 4 Done Sent Done false
22984 6712 Acked Read 2 Done Sent Done false
22985 6713 Acked Read 6 Done Sent Done false
22986 6714 Acked Write 6 Done Sent Done false
22987 6715 Acked Read 10 Done Sent Done false
22988 6716 Acked Read 2 Done Sent Done false
22989 6717 Acked Read 6 Done Sent Done false
22990 6718 Acked Read 6 Done Sent Done false
22991 6719 Acked Write 5 Done Sent Done false
22992 6720 Acked Write 5 Done Sent Done false
22993 6721 Acked Read 7 Done Sent Done false
22994 6722 Acked Write 9 Done Sent Done false
22995 6723 Acked Read 8 Done Sent Done false
22996 6724 Acked Write 4 Done Sent Done false
22997 6725 Acked Read 4 Done Sent Done false
22998 6726 Acked Write 1 Done Sent Done false
22999 6727 Acked Write 4 Done Sent Done false
23000 6728 Acked Read 3 Done Sent Done false
23001 6729 Acked Read 7 Done Sent Done false
23002 6730 Acked Write 8 Done Sent Done false
23003 6731 Acked Write 1 Done Sent Done false
23004 6732 Acked Read 6 Done Sent Done false
23005 6733 Acked Write 6 Done Sent Done false
23006 STATES DS:0 DS:1 DS:2 TOTAL
23007 Sent 0 301 0 301
23008 Done 301 0 301 602
23009 Skipped 0 0 0 0
23010 Error 0 0 0 0
23011 Last Flush: Some(JobId(6709)) Some(JobId(6432)) Some(JobId(6709))
23012 Downstairs last five completed: 6432 6431 6430 6429 6428
23013 Upstairs last five completed: 6733 6732 6731 6730 6729
23014 CLIENT: Up:0 ds:301 act:2
230152025-02-28T06:06:05.127ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
230162025-02-28T06:06:05.848ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
23017 [8820] got start action
23018 [8820] I am going to restart
23019 Starting downstairs at port 8820
23020 Make output file at "/tmp/dsc/downstairs-8820.txt"
23021 [8820][1] reports Starting
23022 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1230)
23023 [8820][1] reports Running
230242025-02-28T06:06:06.850ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
230252025-02-28T06:06:07.852ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
23026 ----------------------------------------------------------------
23027 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23028 States: Active Offline Active
23029 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23030 6433 Acked Write 3 Done Sent Done false
23031 6434 Acked Write 6 Done Sent Done false
23032 6435 Acked Write 3 Done Sent Done false
23033 6436 Acked Flush 0 Done Sent Done false
23034 6437 Acked Read 1 Done Sent Done false
23035 6438 Acked Read 2 Done Sent Done false
23036 6439 Acked Write 1 Done Sent Done false
23037 6440 Acked Write 4 Done Sent Done false
23038 6441 Acked Read 9 Done Sent Done false
23039 6442 Acked Read 6 Done Sent Done false
23040 6443 Acked Write 10 Done Sent Done false
23041 6444 Acked Read 7 Done Sent Done false
23042 6445 Acked Write 10 Done Sent Done false
23043 6446 Acked Read 2 Done Sent Done false
23044 6447 Acked Read 2 Done Sent Done false
23045 6448 Acked Write 10 Done Sent Done false
23046 6449 Acked Read 8 Done Sent Done false
23047 6450 Acked Read 10 Done Sent Done false
23048 6451 Acked Write 3 Done Sent Done false
23049 6452 Acked Read 1 Done Sent Done false
23050 6453 Acked Flush 0 Done Sent Done false
23051 6454 Acked Read 4 Done Sent Done false
23052 6455 Acked Read 10 Done Sent Done false
23053 6456 Acked Write 3 Done Sent Done false
23054 6457 Acked Flush 0 Done Sent Done false
23055 6458 Acked Read 1 Done Sent Done false
23056 6459 Acked Write 4 Done Sent Done false
23057 6460 Acked Write 5 Done Sent Done false
23058 6461 Acked Read 2 Done Sent Done false
23059 6462 Acked Write 9 Done Sent Done false
23060 6463 Acked Write 6 Done Sent Done false
23061 6464 Acked Write 6 Done Sent Done false
23062 6465 Acked Read 4 Done Sent Done false
23063 6466 Acked Read 1 Done Sent Done false
23064 6467 Acked Write 3 Done Sent Done false
23065 6468 Acked Write 8 Done Sent Done false
23066 6469 Acked Read 9 Done Sent Done false
23067 6470 Acked Read 9 Done Sent Done false
23068 6471 Acked Write 9 Done Sent Done false
23069 6472 Acked Read 4 Done Sent Done false
23070 6473 Acked Read 7 Done Sent Done false
23071 6474 Acked Read 10 Done Sent Done false
23072 6475 Acked Write 2 Done Sent Done false
23073 6476 Acked Write 10 Done Sent Done false
23074 6477 Acked Flush 0 Done Sent Done false
23075 6478 Acked Read 8 Done Sent Done false
23076 6479 Acked Write 4 Done Sent Done false
23077 6480 Acked Flush 0 Done Sent Done false
23078 6481 Acked Read 5 Done Sent Done false
23079 6482 Acked Write 9 Done Sent Done false
23080 6483 Acked Read 9 Done Sent Done false
23081 6484 Acked Read 7 Done Sent Done false
23082 6485 Acked Read 4 Done Sent Done false
23083 6486 Acked Write 3 Done Sent Done false
23084 6487 Acked Write 7 Done Sent Done false
23085 6488 Acked Read 2 Done Sent Done false
23086 6489 Acked Read 6 Done Sent Done false
23087 6490 Acked Read 3 Done Sent Done false
23088 6491 Acked Read 3 Done Sent Done false
23089 6492 Acked Write 7 Done Sent Done false
23090 6493 Acked Read 7 Done Sent Done false
23091 6494 Acked Read 1 Done Sent Done false
23092 6495 Acked Write 5 Done Sent Done false
23093 6496 Acked Read 7 Done Sent Done false
23094 6497 Acked Write 3 Done Sent Done false
23095 6498 Acked Write 9 Done Sent Done false
23096 6499 Acked Read 4 Done Sent Done false
23097 6500 Acked Write 10 Done Sent Done false
23098 6501 Acked Write 10 Done Sent Done false
23099 6502 Acked Read 1 Done Sent Done false
23100 6503 Acked Write 6 Done Sent Done false
23101 6504 Acked Write 10 Done Sent Done false
23102 6505 Acked Read 10 Done Sent Done false
23103 6506 Acked Read 3 Done Sent Done false
23104 6507 Acked Flush 0 Done Sent Done false
23105 6508 Acked Read 9 Done Sent Done false
23106 6509 Acked Read 7 Done Sent Done false
23107 6510 Acked Write 6 Done Sent Done false
23108 6511 Acked Read 1 Done Sent Done false
23109 6512 Acked Read 9 Done Sent Done false
23110 6513 Acked Read 5 Done Sent Done false
23111 6514 Acked Flush 0 Done Sent Done false
23112 6515 Acked Write 9 Done Sent Done false
23113 6516 Acked Read 3 Done Sent Done false
23114 6517 Acked Read 10 Done Sent Done false
23115 6518 Acked Read 6 Done Sent Done false
23116 6519 Acked Write 9 Done Sent Done false
23117 6520 Acked Write 7 Done Sent Done false
23118 6521 Acked Write 5 Done Sent Done false
23119 6522 Acked Flush 0 Done Sent Done false
23120 6523 Acked Flush 0 Done Sent Done false
23121 6524 Acked Write 7 Done Sent Done false
23122 6525 Acked Read 7 Done Sent Done false
23123 6526 Acked Read 6 Done Sent Done false
23124 6527 Acked Read 1 Done Sent Done false
23125 6528 Acked Flush 0 Done Sent Done false
23126 6529 Acked Write 3 Done Sent Done false
23127 6530 Acked Write 9 Done Sent Done false
23128 6531 Acked Read 9 Done Sent Done false
23129 6532 Acked Write 5 Done Sent Done false
23130 6533 Acked Read 7 Done Sent Done false
23131 6534 Acked Read 4 Done Sent Done false
23132 6535 Acked Write 2 Done Sent Done false
23133 6536 Acked Flush 0 Done Sent Done false
23134 6537 Acked Write 3 Done Sent Done false
23135 6538 Acked Read 7 Done Sent Done false
23136 6539 Acked Write 9 Done Sent Done false
23137 6540 Acked Read 3 Done Sent Done false
23138 6541 Acked Write 10 Done Sent Done false
23139 6542 Acked Read 6 Done Sent Done false
23140 6543 Acked Write 1 Done Sent Done false
23141 6544 Acked Read 3 Done Sent Done false
23142 6545 Acked Read 7 Done Sent Done false
23143 6546 Acked Write 7 Done Sent Done false
23144 6547 Acked Write 1 Done Sent Done false
23145 6548 Acked Read 5 Done Sent Done false
23146 6549 Acked Write 5 Done Sent Done false
23147 6550 Acked Read 8 Done Sent Done false
23148 6551 Acked Read 1 Done Sent Done false
23149 6552 Acked Read 7 Done Sent Done false
23150 6553 Acked Read 10 Done Sent Done false
23151 6554 Acked Read 3 Done Sent Done false
23152 6555 Acked Flush 0 Done Sent Done false
23153 6556 Acked Flush 0 Done Sent Done false
23154 6557 Acked Write 5 Done Sent Done false
23155 6558 Acked Read 7 Done Sent Done false
23156 6559 Acked Read 1 Done Sent Done false
23157 6560 Acked Flush 0 Done Sent Done false
23158 6561 Acked Read 7 Done Sent Done false
23159 6562 Acked Write 4 Done Sent Done false
23160 6563 Acked Read 1 Done Sent Done false
23161 6564 Acked Read 6 Done Sent Done false
23162 6565 Acked Read 4 Done Sent Done false
23163 6566 Acked Read 8 Done Sent Done false
23164 6567 Acked Write 2 Done Sent Done false
23165 6568 Acked Read 10 Done Sent Done false
23166 6569 Acked Write 5 Done Sent Done false
23167 6570 Acked Write 10 Done Sent Done false
23168 6571 Acked Read 7 Done Sent Done false
23169 6572 Acked Read 6 Done Sent Done false
23170 6573 Acked Write 5 Done Sent Done false
23171 6574 Acked Read 5 Done Sent Done false
23172 6575 Acked Read 10 Done Sent Done false
23173 6576 Acked Read 1 Done Sent Done false
23174 6577 Acked Write 4 Done Sent Done false
23175 6578 Acked Read 8 Done Sent Done false
23176 6579 Acked Write 1 Done Sent Done false
23177 6580 Acked Write 1 Done Sent Done false
23178 6581 Acked Flush 0 Done Sent Done false
23179 6582 Acked Write 8 Done Sent Done false
23180 6583 Acked Write 2 Done Sent Done false
23181 6584 Acked Read 8 Done Sent Done false
23182 6585 Acked Read 9 Done Sent Done false
23183 6586 Acked Write 7 Done Sent Done false
23184 6587 Acked Write 1 Done Sent Done false
23185 6588 Acked Read 7 Done Sent Done false
23186 6589 Acked Read 10 Done Sent Done false
23187 6590 Acked Write 10 Done Sent Done false
23188 6591 Acked Read 2 Done Sent Done false
23189 6592 Acked Write 10 Done Sent Done false
23190 6593 Acked Read 7 Done Sent Done false
23191 6594 Acked Read 2 Done Sent Done false
23192 6595 Acked Write 5 Done Sent Done false
23193 6596 Acked Write 8 Done Sent Done false
23194 6597 Acked Write 2 Done Sent Done false
23195 6598 Acked Read 2 Done Sent Done false
23196 6599 Acked Write 10 Done Sent Done false
23197 6600 Acked Flush 0 Done Sent Done false
23198 6601 Acked Read 6 Done Sent Done false
23199 6602 Acked Read 2 Done Sent Done false
23200 6603 Acked Read 4 Done Sent Done false
23201 6604 Acked Read 4 Done Sent Done false
23202 6605 Acked Write 5 Done Sent Done false
23203 6606 Acked Read 1 Done Sent Done false
23204 6607 Acked Read 2 Done Sent Done false
23205 6608 Acked Write 7 Done Sent Done false
23206 6609 Acked Read 9 Done Sent Done false
23207 6610 Acked Write 10 Done Sent Done false
23208 6611 Acked Flush 0 Done Sent Done false
23209 6612 Acked Read 2 Done Sent Done false
23210 6613 Acked Read 1 Done Sent Done false
23211 6614 Acked Read 1 Done Sent Done false
23212 6615 Acked Read 5 Done Sent Done false
23213 6616 Acked Read 2 Done Sent Done false
23214 6617 Acked Flush 0 Done Sent Done false
23215 6618 Acked Write 6 Done Sent Done false
23216 6619 Acked Read 2 Done Sent Done false
23217 6620 Acked Write 4 Done Sent Done false
23218 6621 Acked Write 10 Done Sent Done false
23219 6622 Acked Read 6 Done Sent Done false
23220 6623 Acked Read 7 Done Sent Done false
23221 6624 Acked Write 1 Done Sent Done false
23222 6625 Acked Write 5 Done Sent Done false
23223 6626 Acked Write 4 Done Sent Done false
23224 6627 Acked Read 7 Done Sent Done false
23225 6628 Acked Read 10 Done Sent Done false
23226 6629 Acked Read 9 Done Sent Done false
23227 6630 Acked Read 1 Done Sent Done false
23228 6631 Acked Write 4 Done Sent Done false
23229 6632 Acked Write 7 Done Sent Done false
23230 6633 Acked Write 4 Done Sent Done false
23231 6634 Acked Read 2 Done Sent Done false
23232 6635 Acked Flush 0 Done Sent Done false
23233 6636 Acked Write 6 Done Sent Done false
23234 6637 Acked Read 8 Done Sent Done false
23235 6638 Acked Write 3 Done Sent Done false
23236 6639 Acked Write 2 Done Sent Done false
23237 6640 Acked Flush 0 Done Sent Done false
23238 6641 Acked Write 1 Done Sent Done false
23239 6642 Acked Write 9 Done Sent Done false
23240 6643 Acked Read 8 Done Sent Done false
23241 6644 Acked Read 2 Done Sent Done false
23242 6645 Acked Flush 0 Done Sent Done false
23243 6646 Acked Read 8 Done Sent Done false
23244 6647 Acked Write 5 Done Sent Done false
23245 6648 Acked Read 7 Done Sent Done false
23246 6649 Acked Read 1 Done Sent Done false
23247 6650 Acked Write 10 Done Sent Done false
23248 6651 Acked Write 4 Done Sent Done false
23249 6652 Acked Flush 0 Done Sent Done false
23250 6653 Acked Read 4 Done Sent Done false
23251 6654 Acked Read 10 Done Sent Done false
23252 6655 Acked Read 9 Done Sent Done false
23253 6656 Acked Write 2 Done Sent Done false
23254 6657 Acked Read 3 Done Sent Done false
23255 6658 Acked Read 1 Done Sent Done false
23256 6659 Acked Read 2 Done Sent Done false
23257 6660 Acked Write 10 Done Sent Done false
23258 6661 Acked Write 1 Done Sent Done false
23259 6662 Acked Flush 0 Done Sent Done false
23260 6663 Acked Write 2 Done Sent Done false
23261 6664 Acked Flush 0 Done Sent Done false
23262 6665 Acked Write 7 Done Sent Done false
23263 6666 Acked Read 1 Done Sent Done false
23264 6667 Acked Write 8 Done Sent Done false
23265 6668 Acked Write 4 Done Sent Done false
23266 6669 Acked Read 6 Done Sent Done false
23267 6670 Acked Read 6 Done Sent Done false
23268 6671 Acked Write 2 Done Sent Done false
23269 6672 Acked Read 1 Done Sent Done false
23270 6673 Acked Read 2 Done Sent Done false
23271 6674 Acked Read 5 Done Sent Done false
23272 6675 Acked Read 3 Done Sent Done false
23273 6676 Acked Read 6 Done Sent Done false
23274 6677 Acked Write 4 Done Sent Done false
23275 6678 Acked Write 6 Done Sent Done false
23276 6679 Acked Read 4 Done Sent Done false
23277 6680 Acked Flush 0 Done Sent Done false
23278 6681 Acked Read 1 Done Sent Done false
23279 6682 Acked Read 5 Done Sent Done false
23280 6683 Acked Read 1 Done Sent Done false
23281 6684 Acked Flush 0 Done Sent Done false
23282 6685 Acked Read 7 Done Sent Done false
23283 6686 Acked Read 1 Done Sent Done false
23284 6687 Acked Write 3 Done Sent Done false
23285 6688 Acked Read 6 Done Sent Done false
23286 6689 Acked Read 9 Done Sent Done false
23287 6690 Acked Flush 0 Done Sent Done false
23288 6691 Acked Read 3 Done Sent Done false
23289 6692 Acked Read 3 Done Sent Done false
23290 6693 Acked Flush 0 Done Sent Done false
23291 6694 Acked Write 1 Done Sent Done false
23292 6695 Acked Write 5 Done Sent Done false
23293 6696 Acked Write 7 Done Sent Done false
23294 6697 Acked Read 10 Done Sent Done false
23295 6698 Acked Write 7 Done Sent Done false
23296 6699 Acked Write 1 Done Sent Done false
23297 6700 Acked Read 10 Done Sent Done false
23298 6701 Acked Read 7 Done Sent Done false
23299 6702 Acked Write 8 Done Sent Done false
23300 6703 Acked Read 8 Done Sent Done false
23301 6704 Acked Read 8 Done Sent Done false
23302 6705 Acked Read 9 Done Sent Done false
23303 6706 Acked Write 1 Done Sent Done false
23304 6707 Acked Write 1 Done Sent Done false
23305 6708 Acked Read 6 Done Sent Done false
23306 6709 Acked Flush 0 Done Sent Done false
23307 6710 Acked Read 2 Done Sent Done false
23308 6711 Acked Write 4 Done Sent Done false
23309 6712 Acked Read 2 Done Sent Done false
23310 6713 Acked Read 6 Done Sent Done false
23311 6714 Acked Write 6 Done Sent Done false
23312 6715 Acked Read 10 Done Sent Done false
23313 6716 Acked Read 2 Done Sent Done false
23314 6717 Acked Read 6 Done Sent Done false
23315 6718 Acked Read 6 Done Sent Done false
23316 6719 Acked Write 5 Done Sent Done false
23317 6720 Acked Write 5 Done Sent Done false
23318 6721 Acked Read 7 Done Sent Done false
23319 6722 Acked Write 9 Done Sent Done false
23320 6723 Acked Read 8 Done Sent Done false
23321 6724 Acked Write 4 Done Sent Done false
23322 6725 Acked Read 4 Done Sent Done false
23323 6726 Acked Write 1 Done Sent Done false
23324 6727 Acked Write 4 Done Sent Done false
23325 6728 Acked Read 3 Done Sent Done false
23326 6729 Acked Read 7 Done Sent Done false
23327 6730 Acked Write 8 Done Sent Done false
23328 6731 Acked Write 1 Done Sent Done false
23329 6732 Acked Read 6 Done Sent Done false
23330 6733 Acked Write 6 Done Sent Done false
23331 STATES DS:0 DS:1 DS:2 TOTAL
23332 Sent 0 301 0 301
23333 Done 301 0 301 602
23334 Skipped 0 0 0 0
23335 Error 0 0 0 0
23336 Last Flush: Some(JobId(6709)) Some(JobId(6432)) Some(JobId(6709))
23337 Downstairs last five completed: 6432 6431 6430 6429 6428
23338 Upstairs last five completed: 6733 6732 6731 6730 6729
23339 CLIENT: Up:0 ds:301 act:2
233402025-02-28T06:06:09.129ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
233412025-02-28T06:06:09.891ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
233422025-02-28T06:06:10.382ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
233432025-02-28T06:06:10.383ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59902) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
233442025-02-28T06:06:10.384ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
233452025-02-28T06:06:10.384ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
233462025-02-28T06:06:10.384ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
233472025-02-28T06:06:10.384ZINFOcrucible: send last flush ID to this DS: Some(JobId(6432)) = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
233482025-02-28T06:06:10.384ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
233492025-02-28T06:06:10.384ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(6432)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
233502025-02-28T06:06:10.385ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(6432)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
23351 ----------------------------------------------------------------
23352 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23353 Downstairs last five completed: 6734 6733 6732 6731 6730
23354 Upstairs last five completed: 6734 6733 6732 6731 6730
23355 CLIENT: Up:0 ds:0 act:3
23356 Replay: All jobs finished
23357 Main task has work to do, go find it
23358 got dsc Stop(2)
23359 stop 2
23360 Feb 28 06:06:12.629 INFO request completed, latency_us: 370, response_code: 204, uri:[8830] Got stop action so:false kr:false
23361 /stop/cid/2, method: POST, req_id: 2bf9ade2-578d-4171-a4ca-9b9eaf9a317e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
23362 Feb 28 06:06:12.630 INFO request completed, latency_us: 131, response_code: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:12.630711964Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
23363 200, uri: /state/cid/2, method: GET, req_id: 9c69820d-91df-4f71-b616-2e7fa7a6bf34, remote_addr: 127.0{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:12.630795374Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
23364 .0.1:63908, local_addr:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:12.630835304Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
23365 127.0.0.1:9998
233662025-02-28T06:06:12.630ZINFOcrucible: client task is exiting = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
23367 [8830] Exited with: ExitStatus(unix_wait_status(9))
23368 [8830][2] reports Exit
23369 Feb 28 06:06:16.633 INFO request completed, latency_us: 253, response_code: 200, uri: /state/cid/2, method: GET, req_id: 39f06c08-ea0a-45f7-b7de-6d207abf38a6, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
23370 001/300 Write block 1072 len 20480 data: 7 5 5 5 4
23371 002/300 Write block 1939 len 16384 data: 3 3 4 4
23372 003/300 Read block 4069 len 40960
23373 004/300 Read block 4749 len 40960
23374 005/300 Read block 3436 len 24576
23375 006/300 Write block 3803 len 8192 data: 3 5
23376 007/300 Write block 4501 len 8192 data: 6 5
23377 008/300 Write block 2833 len 32768 data: 3 2 3 3 3 3 3 3
23378 009/300 Write block 1579 len 20480 data: 7 7 7 7 7
23379 010/300 Write block 1326 len 28672 data: 6 6 6 6 5 6 5
23380 011/300 Write block 849 len 28672 data: 3 2 2 2 3 4 4
23381 012/300 Write block 4397 len 40960 data: 4 3 3 3 3 4 4 4 7 5
23382 013/300 Read block 566 len 36864
23383 014/300 Write block 3015 len 12288 data: 5 3 4
23384 015/300 Read block 3837 len 12288
23385 016/300 Flush
23386 017/300 Read block 996 len 20480
23387 018/300 Read block 1205 len 40960
23388 019/300 Read block 3000 len 8192
23389 020/300 Write block 4480 len 16384 data: 4 5 6 6
23390 021/300 Write block 890 len 28672 data: 3 3 3 3 4 3 3
23391 022/300 Read block 2768 len 12288
23392 023/300 Read block 432 len 8192
23393 024/300 Read block 2672 len 12288
23394 025/300 Read block 454 len 20480
23395 026/300 Flush
23396 027/300 Read block 333 len 4096
23397 028/300 Write block 3714 len 24576 data: 2 4 4 6 8 7
23398 029/300 Read block 764 len 4096
23399 030/300 Read block 766 len 24576
23400 031/300 Write block 543 len 36864 data: 2 2 2 3 4 4 5 6 7
23401 032/300 Write block 341 len 32768 data: 5 4 4 3 4 4 4 4
23402 033/300 Write block 1059 len 16384 data: 5 5 4 4
23403 034/300 Read block 2294 len 4096
23404 035/300 Flush
23405 036/300 Write block 3545 len 8192 data: 4 5
23406 037/300 Read block 2307 len 4096
23407 038/300 Write block 2604 len 24576 data: 5 4 3 2 2 2
23408 039/300 Write block 2110 len 8192 data: 5 4
23409 040/300 Read block 2958 len 4096
23410 041/300 Read block 1017 len 4096
23411 042/300 Write block 2528 len 8192 data: 5 6
23412 043/300 Write block 1281 len 36864 data: 5 4 5 3 2 3 3 3 4
23413 044/300 Read block 1615 len 8192
23414 045/300 Write block 2500 len 4096 data: 4
23415 046/300 Read block 4739 len 8192
23416 047/300 Write block 3000 len 20480 data: 4 5 3 3 3
23417 048/300 Read block 4186 len 36864
23418 049/300 Write block 2348 len 24576 data: 4 4 3 2 2 3
23419 050/300 Read block 2012 len 32768
23420 051/300 Read block 4118 len 20480
23421 052/300 Write block 553 len 28672 data: 8 4 4 3 4 5 5
23422 053/300 Read block 2286 len 8192
23423 054/300 Read block 2007 len 24576
23424 055/300 Read block 1985 len 40960
23425 056/300 Flush
23426 057/300 Read block 1383 len 28672
23427 058/300 Read block 2960 len 12288
23428 059/300 Write block 2961 len 16384 data: 5 6 5 5
23429 060/300 Read block 2099 len 16384
23430 061/300 Read block 4427 len 20480
23431 062/300 Write block 1226 len 12288 data: 4 5 4
23432 063/300 Read block 3416 len 20480
23433 064/300 Read block 2386 len 4096
23434 065/300 Flush
23435 066/300 Read block 1902 len 8192
23436 067/300 Write block 271 len 4096 data: 5
23437 068/300 Flush
23438 069/300 Write block 1625 len 40960 data: 6 6 6 6 6 6 6 4 6 4
23439 070/300 Flush
23440 071/300 Read block 2297 len 32768
23441 072/300 Read block 3850 len 8192
23442 073/300 Read block 3500 len 12288
23443 074/300 Read block 1852 len 4096
23444 075/300 Read block 3608 len 28672
23445 076/300 Read block 1601 len 24576
23446 077/300 Read block 2458 len 8192
23447 078/300 Write block 2085 len 32768 data: 5 7 7 7 8 8 8 4
23448 079/300 Read block 1975 len 20480
23449 080/300 Read block 2465 len 16384
23450 081/300 Write block 3219 len 24576 data: 4 5 5 4 4 4
23451 082/300 Write block 2908 len 28672 data: 2 2 2 2 2 4 4
23452 083/300 Read block 2951 len 28672
23453 084/300 Write block 1848 len 20480 data: 4 4 4 5 5
23454 085/300 Write block 4847 len 32768 data: 3 2 3 3 4 3 4 4
23455 086/300 Read block 3777 len 28672
23456 087/300 Read block 2142 len 40960
23457 088/300 Read block 3410 len 32768
23458 089/300 Read block 747 len 36864
23459 090/300 Read block 2189 len 28672
23460 091/300 Read block 4792 len 32768
23461 092/300 Write block 3360 len 8192 data: 3 3
23462 093/300 Read block 1791 len 32768
23463 094/300 Read block 3062 len 24576
23464 095/300 Read block 171 len 4096
23465 096/300 Write block 1465 len 32768 data: 3 4 5 5 5 4 5 5
23466 097/300 Read block 2768 len 16384
23467 098/300 Read block 2605 len 24576
23468 099/300 Write block 1316 len 4096 data: 3
23469 100/300 Read block 142 len 40960
23470 101/300 Flush
23471 102/300 Read block 4510 len 24576
23472 103/300 Flush
23473 104/300 Read block 851 len 32768
23474 105/300 Read block 4301 len 32768
23475 106/300 Flush
23476 107/300 Flush
23477 108/300 Write block 1011 len 36864 data: 4 4 4 4 5 5 6 6 8
23478 109/300 Write block 3634 len 4096 data: 4
23479 110/300 Write block 526 len 4096 data: 4
23480 111/300 Write block 1722 len 16384 data: 6 7 7 6
23481 112/300 Write block 601 len 32768 data: 5 5 5 3 2 2 3 3
23482 113/300 Write block 4836 len 28672 data: 6 7 8 7 7 6 7
23483 114/300 Read block 180 len 16384
23484 115/300 Flush
23485 116/300 Read block 3875 len 16384
23486 117/300 Flush
23487 118/300 Read block 3962 len 28672
23488 119/300 Write block 2577 len 24576 data: 2 2 4 4 4 4
23489 120/300 Read block 4724 len 32768
23490 121/300 Read block 927 len 20480
23491 122/300 Write block 4982 len 4096 data: 5
23492 123/300 Write block 1173 len 28672 data: 7 8 7 7 8 9 9
23493 124/300 Write block 1507 len 4096 data: 3
23494 125/300 Read block 1400 len 8192
23495 126/300 Read block 246 len 4096
23496 127/300 Write block 4134 len 4096 data: 3
23497 128/300 Read block 2244 len 32768
23498 129/300 Flush
23499 130/300 Read block 473 len 12288
23500 131/300 Read block 1482 len 36864
23501 132/300 Flush
23502 133/300 Write block 2093 len 16384 data: 5 5 5 6
23503 134/300 Write block 3076 len 20480 data: 3 3 3 3 2
23504 135/300 Read block 913 len 4096
23505 136/300 Write block 3670 len 16384 data: 3 2 2 2
23506 137/300 Read block 2135 len 12288
23507 138/300 Read block 4781 len 28672
23508 139/300 Read block 3850 len 24576
23509 140/300 Read block 4746 len 8192
23510 141/300 Read block 506 len 12288
23511 142/300 Read block 4198 len 40960
23512 143/300 Write block 1212 len 28672 data: 3 3 3 3 3 3 2
23513 144/300 Read block 4077 len 32768
23514 145/300 Read block 1189 len 12288
23515 146/300 Read block 1888 len 32768
23516 147/300 Read block 323 len 16384
23517 148/300 Read block 903 len 16384
23518 149/300 Write block 2352 len 36864 data: 3 4 2 2 3 4 4 4 4
23519 150/300 Read block 4580 len 12288
23520 151/300 Write block 1929 len 16384 data: 4 3 3 4
23521 152/300 Read block 4369 len 28672
23522 153/300 Write block 1169 len 24576 data: 6 5 5 5 8 9
23523 154/300 Read block 2669 len 20480
23524 155/300 Read block 3000 len 32768
23525 156/300 Read block 4553 len 24576
23526 157/300 Write block 163 len 8192 data: 4 4
23527 158/300 Write block 4485 len 24576 data: 6 6 6 5 4 4
23528 159/300 Read block 4897 len 16384
23529 160/300 Flush
23530 161/300 Read block 1451 len 16384
23531 162/300 Read block 3781 len 32768
23532 163/300 Write block 3199 len 40960 data: 5 5 6 6 6 6 7 6 6 5
23533 164/300 Read block 3987 len 20480
23534 165/300 Read block 1256 len 24576
23535 166/300 Read block 546 len 8192
23536 167/300 Flush
23537 168/300 Write block 4494 len 8192 data: 4 5
23538 169/300 Read block 642 len 12288
23539 170/300 Write block 494 len 20480 data: 4 5 6 5 6
23540 171/300 Read block 708 len 16384
23541 172/300 Flush
23542 173/300 Read block 2941 len 4096
23543 174/300 Read block 818 len 4096
23544 175/300 Read block 226 len 24576
23545 176/300 Flush
23546 177/300 Read block 1814 len 20480
23547 178/300 Read block 735 len 36864
23548 179/300 Read block 2491 len 8192
23549 180/300 Write block 3667 len 40960 data: 4 4 4 4 3 3 3 2 2 3
23550 181/300 Read block 3785 len 24576
23551 182/300 Write block 2894 len 20480 data: 2 3 3 4 5
23552 183/300 Read block 3425 len 28672
23553 184/300 Read block 3638 len 40960
23554 185/300 Write block 2871 len 40960 data: 5 5 4 4 2 3 3 3 3 2
23555 186/300 Read block 4605 len 24576
23556 187/300 Write block 1586 len 32768 data: 5 5 4 3 3 2 2 3
23557 188/300 Write block 4106 len 16384 data: 3 3 4 4
23558 189/300 Read block 2959 len 12288
23559 190/300 Read block 3828 len 20480
23560 191/300 Read block 3654 len 16384
23561 192/300 Flush
23562 193/300 Write block 4159 len 40960 data: 10 9 7 7 5 5 5 5 5 5
23563 194/300 Write block 1591 len 28672 data: 3 3 4 3 3 3 3
23564 195/300 Read block 3952 len 36864
23565 196/300 Write block 4725 len 40960 data: 5 5 5 5 6 7 7 7 6 6
23566 197/300 Write block 3327 len 4096 data: 4
23567 198/300 Read block 148 len 28672
23568 199/300 Read block 3153 len 12288
23569 200/300 Flush
23570 201/300 Read block 10 len 16384
23571 202/300 Write block 4277 len 20480 data: 4 4 4 4 4
23572 203/300 Read block 1604 len 20480
23573 204/300 Write block 4303 len 36864 data: 3 3 3 4 5 4 4 3 3
23574 205/300 Read block 3548 len 4096
23575 206/300 Read block 1252 len 32768
23576 207/300 Read block 1609 len 36864
23577 208/300 Read block 4314 len 20480
23578 209/300 Write block 3845 len 20480 data: 6 5 6 8 7
23579 210/300 Read block 2070 len 16384
23580 211/300 Write block 3614 len 24576 data: 6 6 4 4 3 4
23581 212/300 Read block 4892 len 8192
23582 213/300 Write block 1516 len 24576 data: 4 4 4 3 3 4
23583 214/300 Write block 3089 len 28672 data: 4 5 5 5 6 5 3
23584 215/300 Read block 2401 len 40960
23585 216/300 Write block 871 len 4096 data: 5
23586 217/300 Write block 4 len 12288 data: 3 3 3
23587 218/300 Read block 2583 len 36864
23588 219/300 Write block 1311 len 36864 data: 5 6 6 3 3 4 3 3 4
23589 220/300 Write block 290 len 8192 data: 4 4
23590 221/300 Write block 4538 len 28672 data: 3 4 4 3 3 3 4
23591 222/300 Flush
23592 223/300 Write block 974 len 40960 data: 2 4 4 4 5 4 5 4 4 5
23593 224/300 Read block 2548 len 4096
23594 225/300 Write block 1665 len 32768 data: 7 7 5 6 6 5 5 4
23595 226/300 Flush
23596 227/300 Flush
23597 228/300 Write block 1709 len 16384 data: 8 8 7 8
23598 229/300 Read block 4047 len 16384
23599 230/300 Read block 215 len 4096
23600 231/300 Flush
23601 232/300 Read block 2859 len 4096
23602 233/300 Write block 4320 len 36864 data: 6 7 6 7 6 5 4 4 4
23603 234/300 Write block 3762 len 36864 data: 5 4 3 3 3 3 3 3 2
23604 235/300 Read block 2639 len 40960
23605 236/300 Write block 813 len 16384 data: 9 7 5 5
23606 237/300 Read block 2536 len 8192
23607 238/300 Read block 4610 len 24576
23608 239/300 Write block 648 len 36864 data: 5 4 4 4 4 4 4 4 3
23609 240/300 Write block 2226 len 28672 data: 5 6 6 5 6 8 7
23610 241/300 Read block 4690 len 12288
23611 242/300 Write block 3074 len 16384 data: 4 3 4 4
23612 243/300 Read block 864 len 24576
23613 244/300 Read block 4815 len 8192
23614 245/300 Write block 17 len 24576 data: 3 4 4 5 5 5
23615 246/300 Write block 2748 len 20480 data: 5 5 6 5 3
23616 247/300 Write block 1551 len 28672 data: 4 2 2 4 4 4 5
23617 248/300 Write block 2818 len 40960 data: 6 7 6 5 4 3 5 5 5 3
23618 249/300 Write block 1450 len 16384 data: 3 4 4 4
23619 250/300 Read block 2131 len 20480
23620 251/300 Write block 2642 len 40960 data: 5 5 3 3 3 5 5 5 5 5
23621 252/300 Flush
23622 253/300 Write block 2418 len 24576 data: 4 4 4 4 4 5
23623 254/300 Read block 1516 len 4096
23624 255/300 Write block 155 len 24576 data: 5 5 3 2 3 4
23625 256/300 Read block 1717 len 16384
23626 257/300 Write block 64 len 24576 data: 5 3 3 2 2 2
23627 258/300 Flush
23628 259/300 Write block 4755 len 32768 data: 4 8 8 8 8 6 5 6
23629 260/300 Read block 4418 len 12288
23630 261/300 Read block 1511 len 32768
23631 262/300 Write block 4688 len 20480 data: 3 3 3 3 4
23632 263/300 Read block 4843 len 4096
23633 264/300 Write block 4547 len 28672 data: 4 3 3 4 4 3 3
23634 265/300 Read block 226 len 40960
23635 266/300 Read block 3064 len 36864
23636 267/300 Flush
23637 268/300 Read block 2437 len 12288
23638 269/300 Read block 2517 len 16384
23639 270/300 Write block 662 len 8192 data: 7 6
23640 271/300 Write block 78 len 16384 data: 6 6 5 6
23641 272/300 Flush
23642 273/300 Read block 1327 len 32768
23643 274/300 Write block 3080 len 24576 data: 3 2 3 4 4 3
23644 275/300 Write block 4063 len 8192 data: 4 4
23645 276/300 Write block 1566 len 28672 data: 5 4 5 5 4 4 4
23646 277/300 Read block 1128 len 12288
23647 278/300 Read block 1828 len 24576
23648 279/300 Read block 2130 len 40960
23649 280/300 Write block 1971 len 24576 data: 6 6 6 6 6 4
23650 281/300 Read block 4762 len 28672
23651 282/300 Write block 1809 len 24576 data: 4 5 5 4 4 4
23652 283/300 Read block 2817 len 28672
23653 284/300 Write block 367 len 12288 data: 3 3 4
23654 285/300 Write block 479 len 20480 data: 5 6 6 6 8
23655 286/300 Flush
23656 287/300 Write block 3246 len 36864 data: 4 4 3 6 6 5 6 4 4
23657 288/300 Write block 1379 len 16384 data: 5 5 5 5
23658 289/300 Flush
23659 290/300 Read block 3690 len 24576
23660 291/300 Flush
23661 292/300 Write block 1586 len 24576 data: 6 6 5 4 4 4
23662 293/300 Read block 919 len 28672
23663 294/300 Write block 1816 len 8192 data: 4 4
23664 295/300 Write block 588 len 36864 data: 3 2 3 3 3 3 3 3 3
23665 296/300 Read block 2109 len 8192
23666 297/300 Write block 4832 len 20480 data: 8 5 5 6 7
23667 298/300 Read block 4553 len 36864
23668 299/300 Write block 522 len 20480 data: 6 6 6 4 5
23669 300/300 Read block 1422 len 8192
23670 301/300 Read block 3543 len 24576
23671 Main task has work to do, go find it
23672 got dsc Start(2)
23673 start 2
23674 Feb 28 06:06:16.872 INFO request completed, latency_us: 178, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1ef05e60-d8bd-44aa-9041-1773891c3067, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
23675 Replay: started 2, returned:Ok(())
23676 ----------------------------------------------------------------
23677 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23678 States: Active Active Offline
23679 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23680 6735 Acked Write 5 Done Done Sent false
23681 6736 Acked Write 4 Done Done Sent false
23682 6737 Acked Read 10 Done Done Sent false
23683 6738 Acked Read 10 Done Done Sent false
23684 6739 Acked Read 6 Done Done Sent false
23685 6740 Acked Write 2 Done Done Sent false
23686 6741 Acked Write 2 Done Done Sent false
23687 6742 Acked Write 8 Done Done Sent false
23688 6743 Acked Write 5 Done Done Sent false
23689 6744 Acked Write 7 Done Done Sent false
23690 6745 Acked Write 7 Done Done Sent false
23691 6746 Acked Write 10 Done Done Sent false
23692 6747 Acked Read 9 Done Done Sent false
23693 6748 Acked Write 3 Done Done Sent false
23694 6749 Acked Read 3 Done Done Sent false
23695 6750 Acked Flush 0 Done Done Sent false
23696 6751 Acked Read 5 Done Done Sent false
23697 6752 Acked Read 10 Done Done Sent false
23698 6753 Acked Read 2 Done Done Sent false
23699 6754 Acked Write 4 Done Done Sent false
23700 6755 Acked Write 7 Done Done Sent false
23701 6756 Acked Read 3 Done Done Sent false
23702 6757 Acked Read 2 Done Done Sent false
23703 6758 Acked Read 3 Done Done Sent false
23704 6759 Acked Read 5 Done Done Sent false
23705 6760 Acked Flush 0 Done Done Sent false
23706 6761 Acked Read 1 Done Done Sent false
23707 6762 Acked Write 6 Done Done Sent false
23708 6763 Acked Read 1 Done Done Sent false
23709 6764 Acked Read 6 Done Done Sent false
23710 6765 Acked Write 9 Done Done Sent false
23711 6766 Acked Write 8 Done Done Sent false
23712 6767 Acked Write 4 Done Done Sent false
23713 6768 Acked Read 1 Done Done Sent false
23714 6769 Acked Flush 0 Done Done Sent false
23715 6770 Acked Write 2 Done Done Sent false
23716 6771 Acked Read 1 Done Done Sent false
23717 6772 Acked Write 6 Done Done Sent false
23718 6773 Acked Write 2 Done Done Sent false
23719 6774 Acked Read 1 Done Done Sent false
23720 6775 Acked Read 1 Done Done Sent false
23721 6776 Acked Write 2 Done Done Sent false
23722 6777 Acked Write 9 Done Done Sent false
23723 6778 Acked Read 2 Done Done Sent false
23724 6779 Acked Write 1 Done Done Sent false
23725 6780 Acked Read 2 Done Done Sent false
23726 6781 Acked Write 5 Done Done Sent false
23727 6782 Acked Read 9 Done Done Sent false
23728 6783 Acked Write 6 Done Done Sent false
23729 6784 Acked Read 8 Done Done Sent false
23730 6785 Acked Read 5 Done Done Sent false
23731 6786 Acked Write 7 Done Done Sent false
23732 6787 Acked Read 2 Done Done Sent false
23733 6788 Acked Read 6 Done Done Sent false
23734 6789 Acked Read 10 Done Done Sent false
23735 6790 Acked Flush 0 Done Done Sent false
23736 6791 Acked Read 7 Done Done Sent false
23737 6792 Acked Read 3 Done Done Sent false
23738 6793 Acked Write 4 Done Done Sent false
23739 6794 Acked Read 4 Done Done Sent false
23740 6795 Acked Read 5 Done Done Sent false
23741 6796 Acked Write 3 Done Done Sent false
23742 6797 Acked Read 5 Done Done Sent false
23743 6798 Acked Read 1 Done Done Sent false
23744 6799 Acked Flush 0 Done Done Sent false
23745 6800 Acked Read 2 Done Done Sent false
23746 6801 Acked Write 1 Done Done Sent false
23747 6802 Acked Flush 0 Done Done Sent false
23748 6803 Acked Write 10 Done Done Sent false
23749 6804 Acked Flush 0 Done Done Sent false
23750 6805 Acked Read 8 Done Done Sent false
23751 6806 Acked Read 2 Done Done Sent false
23752 6807 Acked Read 3 Done Done Sent false
23753 6808 Acked Read 1 Done Done Sent false
23754 6809 Acked Read 7 Done Done Sent false
23755 6810 Acked Read 6 Done Done Sent false
23756 6811 Acked Read 2 Done Done Sent false
23757 6812 Acked Write 8 Done Done Sent false
23758 6813 Acked Read 5 Done Done Sent false
23759 6814 Acked Read 4 Done Done Sent false
23760 6815 Acked Write 6 Done Done Sent false
23761 6816 Acked Write 7 Done Done Sent false
23762 6817 Acked Read 7 Done Done Sent false
23763 6818 Acked Write 5 Done Done Sent false
23764 6819 Acked Write 8 Done Done Sent false
23765 6820 Acked Read 7 Done Done Sent false
23766 6821 Acked Read 10 Done Done Sent false
23767 6822 Acked Read 8 Done Done Sent false
23768 6823 Acked Read 9 Done Done Sent false
23769 6824 Acked Read 7 Done Done Sent false
23770 6825 Acked Read 8 Done Done Sent false
23771 6826 Acked Write 2 Done Done Sent false
23772 6827 Acked Read 8 Done Done Sent false
23773 6828 Acked Read 6 Done Done Sent false
23774 6829 Acked Read 1 Done Done Sent false
23775 6830 Acked Write 8 Done Done Sent false
23776 6831 Acked Read 4 Done Done Sent false
23777 6832 Acked Read 6 Done Done Sent false
23778 6833 Acked Write 1 Done Done Sent false
23779 6834 Acked Read 10 Done Done Sent false
23780 6835 Acked Flush 0 Done Done Sent false
23781 6836 Acked Read 6 Done Done Sent false
23782 6837 Acked Flush 0 Done Done Sent false
23783 6838 Acked Read 8 Done Done Sent false
23784 6839 Acked Read 8 Done Done Sent false
23785 6840 Acked Flush 0 Done Done Sent false
23786 6841 Acked Flush 0 Done Done Sent false
23787 6842 Acked Write 9 Done Done Sent false
23788 6843 Acked Write 1 Done Done Sent false
23789 6844 Acked Write 1 Done Done Sent false
23790 6845 Acked Write 4 Done Done Sent false
23791 6846 Acked Write 8 Done Done Sent false
23792 6847 Acked Write 7 Done Done Sent false
23793 6848 Acked Read 4 Done Done Sent false
23794 6849 Acked Flush 0 Done Done Sent false
23795 6850 Acked Read 4 Done Done Sent false
23796 6851 Acked Flush 0 Done Done Sent false
23797 6852 Acked Read 7 Done Done Sent false
23798 6853 Acked Write 6 Done Done Sent false
23799 6854 Acked Read 8 Done Done Sent false
23800 6855 Acked Read 5 Done Done Sent false
23801 6856 Acked Write 1 Done Done Sent false
23802 6857 Acked Write 7 Done Done Sent false
23803 6858 Acked Write 1 Done Done Sent false
23804 6859 Acked Read 2 Done Done Sent false
23805 6860 Acked Read 1 Done Done Sent false
23806 6861 Acked Write 1 Done Done Sent false
23807 6862 Acked Read 8 Done Done Sent false
23808 6863 Acked Flush 0 Done Done Sent false
23809 6864 Acked Read 3 Done Done Sent false
23810 6865 Acked Read 9 Done Done Sent false
23811 6866 Acked Flush 0 Done Done Sent false
23812 6867 Acked Write 4 Done Done Sent false
23813 6868 Acked Write 5 Done Done Sent false
23814 6869 Acked Read 1 Done Done Sent false
23815 6870 Acked Write 4 Done Done Sent false
23816 6871 Acked Read 3 Done Done Sent false
23817 6872 Acked Read 7 Done Done Sent false
23818 6873 Acked Read 6 Done Done Sent false
23819 6874 Acked Read 2 Done Done Sent false
23820 6875 Acked Read 3 Done Done Sent false
23821 6876 Acked Read 10 Done Done Sent false
23822 6877 Acked Write 7 Done Done Sent false
23823 6878 Acked Read 8 Done Done Sent false
23824 6879 Acked Read 3 Done Done Sent false
23825 6880 Acked Read 8 Done Done Sent false
23826 6881 Acked Read 4 Done Done Sent false
23827 6882 Acked Read 4 Done Done Sent false
23828 6883 Acked Write 9 Done Done Sent false
23829 6884 Acked Read 3 Done Done Sent false
23830 6885 Acked Write 4 Done Done Sent false
23831 6886 Acked Read 7 Done Done Sent false
23832 6887 Acked Write 6 Done Done Sent false
23833 6888 Acked Read 5 Done Done Sent false
23834 6889 Acked Read 8 Done Done Sent false
23835 6890 Acked Read 6 Done Done Sent false
23836 6891 Acked Write 2 Done Done Sent false
23837 6892 Acked Write 6 Done Done Sent false
23838 6893 Acked Read 4 Done Done Sent false
23839 6894 Acked Flush 0 Done Done Sent false
23840 6895 Acked Read 4 Done Done Sent false
23841 6896 Acked Read 8 Done Done Sent false
23842 6897 Acked Write 10 Done Done Sent false
23843 6898 Acked Read 5 Done Done Sent false
23844 6899 Acked Read 6 Done Done Sent false
23845 6900 Acked Read 2 Done Done Sent false
23846 6901 Acked Flush 0 Done Done Sent false
23847 6902 Acked Write 2 Done Done Sent false
23848 6903 Acked Read 3 Done Done Sent false
23849 6904 Acked Write 5 Done Done Sent false
23850 6905 Acked Read 4 Done Done Sent false
23851 6906 Acked Flush 0 Done Done Sent false
23852 6907 Acked Read 1 Done Done Sent false
23853 6908 Acked Read 1 Done Done Sent false
23854 6909 Acked Read 6 Done Done Sent false
23855 6910 Acked Flush 0 Done Done Sent false
23856 6911 Acked Read 5 Done Done Sent false
23857 6912 Acked Read 9 Done Done Sent false
23858 6913 Acked Read 2 Done Done Sent false
23859 6914 Acked Write 10 Done Done Sent false
23860 6915 Acked Read 6 Done Done Sent false
23861 6916 Acked Write 5 Done Done Sent false
23862 6917 Acked Read 7 Done Done Sent false
23863 6918 Acked Read 10 Done Done Sent false
23864 6919 Acked Write 10 Done Done Sent false
23865 6920 Acked Read 6 Done Done Sent false
23866 6921 Acked Write 8 Done Done Sent false
23867 6922 Acked Write 4 Done Done Sent false
23868 6923 Acked Read 3 Done Done Sent false
23869 6924 Acked Read 5 Done Done Sent false
23870 6925 Acked Read 4 Done Done Sent false
23871 6926 Acked Flush 0 Done Done Sent false
23872 6927 Acked Write 10 Done Done Sent false
23873 6928 Acked Write 7 Done Done Sent false
23874 6929 Acked Read 9 Done Done Sent false
23875 6930 Acked Write 10 Done Done Sent false
23876 6931 Acked Write 1 Done Done Sent false
23877 6932 Acked Read 7 Done Done Sent false
23878 6933 Acked Read 3 Done Done Sent false
23879 6934 Acked Flush 0 Done Done Sent false
23880 6935 Acked Read 4 Done Done Sent false
23881 6936 Acked Write 5 Done Done Sent false
23882 6937 Acked Read 5 Done Done Sent false
23883 6938 Acked Write 9 Done Done Sent false
23884 6939 Acked Read 1 Done Done Sent false
23885 6940 Acked Read 8 Done Done Sent false
23886 6941 Acked Read 9 Done Done Sent false
23887 6942 Acked Read 5 Done Done Sent false
23888 6943 Acked Write 5 Done Done Sent false
23889 6944 Acked Read 4 Done Done Sent false
23890 6945 Acked Write 6 Done Done Sent false
23891 6946 Acked Read 2 Done Done Sent false
23892 6947 Acked Write 6 Done Done Sent false
23893 6948 Acked Write 7 Done Done Sent false
23894 6949 Acked Read 10 Done Done Sent false
23895 6950 Acked Write 1 Done Done Sent false
23896 6951 Acked Write 3 Done Done Sent false
23897 6952 Acked Read 9 Done Done Sent false
23898 6953 Acked Write 9 Done Done Sent false
23899 6954 Acked Write 2 Done Done Sent false
23900 6955 Acked Write 7 Done Done Sent false
23901 6956 Acked Flush 0 Done Done Sent false
23902 6957 Acked Write 10 Done Done Sent false
23903 6958 Acked Read 1 Done Done Sent false
23904 6959 Acked Write 8 Done Done Sent false
23905 6960 Acked Flush 0 Done Done Sent false
23906 6961 Acked Flush 0 Done Done Sent false
23907 6962 Acked Write 4 Done Done Sent false
23908 6963 Acked Read 4 Done Done Sent false
23909 6964 Acked Read 1 Done Done Sent false
23910 6965 Acked Flush 0 Done Done Sent false
23911 6966 Acked Read 1 Done Done Sent false
23912 6967 Acked Write 9 Done Done Sent false
23913 6968 Acked Write 9 Done Done Sent false
23914 6969 Acked Read 10 Done Done Sent false
23915 6970 Acked Write 4 Done Done Sent false
23916 6971 Acked Read 2 Done Done Sent false
23917 6972 Acked Read 6 Done Done Sent false
23918 6973 Acked Write 9 Done Done Sent false
23919 6974 Acked Write 7 Done Done Sent false
23920 6975 Acked Read 3 Done Done Sent false
23921 6976 Acked Write 4 Done Done Sent false
23922 6977 Acked Read 6 Done Done Sent false
23923 6978 Acked Read 2 Done Done Sent false
23924 6979 Acked Write 6 Done Done Sent false
23925 6980 Acked Write 5 Done Done Sent false
23926 6981 Acked Write 7 Done Done Sent false
23927 6982 Acked Write 10 Done Done Sent false
23928 6983 Acked Write 4 Done Done Sent false
23929 6984 Acked Read 5 Done Done Sent false
23930 6985 Acked Write 10 Done Done Sent false
23931 6986 Acked Flush 0 Done Done Sent false
23932 6987 Acked Write 6 Done Done Sent false
23933 6988 Acked Read 1 Done Done Sent false
23934 6989 Acked Write 6 Done Done Sent false
23935 6990 Acked Read 4 Done Done Sent false
23936 6991 Acked Write 6 Done Done Sent false
23937 6992 Acked Flush 0 Done Done Sent false
23938 6993 Acked Write 8 Done Done Sent false
23939 6994 Acked Read 3 Done Done Sent false
23940 6995 Acked Read 8 Done Done Sent false
23941 6996 Acked Write 5 Done Done Sent false
23942 6997 Acked Read 1 Done Done Sent false
23943 6998 Acked Write 7 Done Done Sent false
23944 6999 Acked Read 10 Done Done Sent false
23945 7000 Acked Read 9 Done Done Sent false
23946 7001 Acked Flush 0 Done Done Sent false
23947 7002 Acked Read 3 Done Done Sent false
23948 7003 Acked Read 4 Done Done Sent false
23949 7004 Acked Write 2 Done Done Sent false
23950 7005 Acked Write 4 Done Done Sent false
23951 7006 Acked Flush 0 Done Done Sent false
23952 7007 Acked Read 8 Done Done Sent false
23953 7008 Acked Write 6 Done Done Sent false
23954 7009 Acked Write 2 Done Done Sent false
23955 7010 Acked Write 7 Done Done Sent false
23956 7011 Acked Read 3 Done Done Sent false
23957 7012 Acked Read 6 Done Done Sent false
23958 7013 Acked Read 10 Done Done Sent false
23959 7014 Acked Write 6 Done Done Sent false
23960 7015 Acked Read 7 Done Done Sent false
23961 7016 Acked Write 6 Done Done Sent false
23962 7017 Acked Read 7 Done Done Sent false
23963 7018 Acked Write 3 Done Done Sent false
23964 7019 Acked Write 5 Done Done Sent false
23965 7020 Acked Flush 0 Done Done Sent false
23966 7021 Acked Write 9 Done Done Sent false
23967 7022 Acked Write 4 Done Done Sent false
23968 7023 Acked Flush 0 Done Done Sent false
23969 7024 Acked Read 6 Done Done Sent false
23970 7025 Acked Flush 0 Done Done Sent false
23971 7026 Acked Write 6 Done Done Sent false
23972 7027 Acked Read 7 Done Done Sent false
23973 7028 Acked Write 2 Done Done Sent false
23974 7029 Acked Write 9 Done Done Sent false
23975 7030 Acked Read 2 Done Done Sent false
23976 7031 Acked Write 5 Done Done Sent false
23977 7032 Acked Read 9 Done Done Sent false
23978 7033 Acked Write 5 Done Done Sent false
23979 7034 Acked Read 2 Done Done Sent false
23980 7035 Acked Read 6 Done Done Sent false
23981 STATES DS:0 DS:1 DS:2 TOTAL
23982 Sent 0 0 301 301
23983 Done 301 301 0 602
23984 Skipped 0 0 0 0
23985 Error 0 0 0 0
23986 Last Flush: Some(JobId(7025)) Some(JobId(7025)) Some(JobId(6734))
23987 Downstairs last five completed: 6734 6733 6732 6731 6730
23988 Upstairs last five completed: 7035 7034 7033 7032 7031
23989 CLIENT: Up:0 ds:301 act:2
239902025-02-28T06:06:17.867ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
23991 [8830] got start action
23992 [8830] I am going to restart
23993 Starting downstairs at port 8830
23994 Make output file at "/tmp/dsc/downstairs-8830.txt"
23995 [8830][2] reports Starting
23996 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1231)
23997 [8830][2] reports Running
239982025-02-28T06:06:18.869ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
239992025-02-28T06:06:19.871ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
24000 ----------------------------------------------------------------
24001 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24002 States: Active Active Offline
24003 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24004 6735 Acked Write 5 Done Done Sent false
24005 6736 Acked Write 4 Done Done Sent false
24006 6737 Acked Read 10 Done Done Sent false
24007 6738 Acked Read 10 Done Done Sent false
24008 6739 Acked Read 6 Done Done Sent false
24009 6740 Acked Write 2 Done Done Sent false
24010 6741 Acked Write 2 Done Done Sent false
24011 6742 Acked Write 8 Done Done Sent false
24012 6743 Acked Write 5 Done Done Sent false
24013 6744 Acked Write 7 Done Done Sent false
24014 6745 Acked Write 7 Done Done Sent false
24015 6746 Acked Write 10 Done Done Sent false
24016 6747 Acked Read 9 Done Done Sent false
24017 6748 Acked Write 3 Done Done Sent false
24018 6749 Acked Read 3 Done Done Sent false
24019 6750 Acked Flush 0 Done Done Sent false
24020 6751 Acked Read 5 Done Done Sent false
24021 6752 Acked Read 10 Done Done Sent false
24022 6753 Acked Read 2 Done Done Sent false
24023 6754 Acked Write 4 Done Done Sent false
24024 6755 Acked Write 7 Done Done Sent false
24025 6756 Acked Read 3 Done Done Sent false
24026 6757 Acked Read 2 Done Done Sent false
24027 6758 Acked Read 3 Done Done Sent false
24028 6759 Acked Read 5 Done Done Sent false
24029 6760 Acked Flush 0 Done Done Sent false
24030 6761 Acked Read 1 Done Done Sent false
24031 6762 Acked Write 6 Done Done Sent false
24032 6763 Acked Read 1 Done Done Sent false
24033 6764 Acked Read 6 Done Done Sent false
24034 6765 Acked Write 9 Done Done Sent false
24035 6766 Acked Write 8 Done Done Sent false
24036 6767 Acked Write 4 Done Done Sent false
24037 6768 Acked Read 1 Done Done Sent false
24038 6769 Acked Flush 0 Done Done Sent false
24039 6770 Acked Write 2 Done Done Sent false
24040 6771 Acked Read 1 Done Done Sent false
24041 6772 Acked Write 6 Done Done Sent false
24042 6773 Acked Write 2 Done Done Sent false
24043 6774 Acked Read 1 Done Done Sent false
24044 6775 Acked Read 1 Done Done Sent false
24045 6776 Acked Write 2 Done Done Sent false
24046 6777 Acked Write 9 Done Done Sent false
24047 6778 Acked Read 2 Done Done Sent false
24048 6779 Acked Write 1 Done Done Sent false
24049 6780 Acked Read 2 Done Done Sent false
24050 6781 Acked Write 5 Done Done Sent false
24051 6782 Acked Read 9 Done Done Sent false
24052 6783 Acked Write 6 Done Done Sent false
24053 6784 Acked Read 8 Done Done Sent false
24054 6785 Acked Read 5 Done Done Sent false
24055 6786 Acked Write 7 Done Done Sent false
24056 6787 Acked Read 2 Done Done Sent false
24057 6788 Acked Read 6 Done Done Sent false
24058 6789 Acked Read 10 Done Done Sent false
24059 6790 Acked Flush 0 Done Done Sent false
24060 6791 Acked Read 7 Done Done Sent false
24061 6792 Acked Read 3 Done Done Sent false
24062 6793 Acked Write 4 Done Done Sent false
24063 6794 Acked Read 4 Done Done Sent false
24064 6795 Acked Read 5 Done Done Sent false
24065 6796 Acked Write 3 Done Done Sent false
24066 6797 Acked Read 5 Done Done Sent false
24067 6798 Acked Read 1 Done Done Sent false
24068 6799 Acked Flush 0 Done Done Sent false
24069 6800 Acked Read 2 Done Done Sent false
24070 6801 Acked Write 1 Done Done Sent false
24071 6802 Acked Flush 0 Done Done Sent false
24072 6803 Acked Write 10 Done Done Sent false
24073 6804 Acked Flush 0 Done Done Sent false
24074 6805 Acked Read 8 Done Done Sent false
24075 6806 Acked Read 2 Done Done Sent false
24076 6807 Acked Read 3 Done Done Sent false
24077 6808 Acked Read 1 Done Done Sent false
24078 6809 Acked Read 7 Done Done Sent false
24079 6810 Acked Read 6 Done Done Sent false
24080 6811 Acked Read 2 Done Done Sent false
24081 6812 Acked Write 8 Done Done Sent false
24082 6813 Acked Read 5 Done Done Sent false
24083 6814 Acked Read 4 Done Done Sent false
24084 6815 Acked Write 6 Done Done Sent false
24085 6816 Acked Write 7 Done Done Sent false
24086 6817 Acked Read 7 Done Done Sent false
24087 6818 Acked Write 5 Done Done Sent false
24088 6819 Acked Write 8 Done Done Sent false
24089 6820 Acked Read 7 Done Done Sent false
24090 6821 Acked Read 10 Done Done Sent false
24091 6822 Acked Read 8 Done Done Sent false
24092 6823 Acked Read 9 Done Done Sent false
24093 6824 Acked Read 7 Done Done Sent false
24094 6825 Acked Read 8 Done Done Sent false
24095 6826 Acked Write 2 Done Done Sent false
24096 6827 Acked Read 8 Done Done Sent false
24097 6828 Acked Read 6 Done Done Sent false
24098 6829 Acked Read 1 Done Done Sent false
24099 6830 Acked Write 8 Done Done Sent false
24100 6831 Acked Read 4 Done Done Sent false
24101 6832 Acked Read 6 Done Done Sent false
24102 6833 Acked Write 1 Done Done Sent false
24103 6834 Acked Read 10 Done Done Sent false
24104 6835 Acked Flush 0 Done Done Sent false
24105 6836 Acked Read 6 Done Done Sent false
24106 6837 Acked Flush 0 Done Done Sent false
24107 6838 Acked Read 8 Done Done Sent false
24108 6839 Acked Read 8 Done Done Sent false
24109 6840 Acked Flush 0 Done Done Sent false
24110 6841 Acked Flush 0 Done Done Sent false
24111 6842 Acked Write 9 Done Done Sent false
24112 6843 Acked Write 1 Done Done Sent false
24113 6844 Acked Write 1 Done Done Sent false
24114 6845 Acked Write 4 Done Done Sent false
24115 6846 Acked Write 8 Done Done Sent false
24116 6847 Acked Write 7 Done Done Sent false
24117 6848 Acked Read 4 Done Done Sent false
24118 6849 Acked Flush 0 Done Done Sent false
24119 6850 Acked Read 4 Done Done Sent false
24120 6851 Acked Flush 0 Done Done Sent false
24121 6852 Acked Read 7 Done Done Sent false
24122 6853 Acked Write 6 Done Done Sent false
24123 6854 Acked Read 8 Done Done Sent false
24124 6855 Acked Read 5 Done Done Sent false
24125 6856 Acked Write 1 Done Done Sent false
24126 6857 Acked Write 7 Done Done Sent false
24127 6858 Acked Write 1 Done Done Sent false
24128 6859 Acked Read 2 Done Done Sent false
24129 6860 Acked Read 1 Done Done Sent false
24130 6861 Acked Write 1 Done Done Sent false
24131 6862 Acked Read 8 Done Done Sent false
24132 6863 Acked Flush 0 Done Done Sent false
24133 6864 Acked Read 3 Done Done Sent false
24134 6865 Acked Read 9 Done Done Sent false
24135 6866 Acked Flush 0 Done Done Sent false
24136 6867 Acked Write 4 Done Done Sent false
24137 6868 Acked Write 5 Done Done Sent false
24138 6869 Acked Read 1 Done Done Sent false
24139 6870 Acked Write 4 Done Done Sent false
24140 6871 Acked Read 3 Done Done Sent false
24141 6872 Acked Read 7 Done Done Sent false
24142 6873 Acked Read 6 Done Done Sent false
24143 6874 Acked Read 2 Done Done Sent false
24144 6875 Acked Read 3 Done Done Sent false
24145 6876 Acked Read 10 Done Done Sent false
24146 6877 Acked Write 7 Done Done Sent false
24147 6878 Acked Read 8 Done Done Sent false
24148 6879 Acked Read 3 Done Done Sent false
24149 6880 Acked Read 8 Done Done Sent false
24150 6881 Acked Read 4 Done Done Sent false
24151 6882 Acked Read 4 Done Done Sent false
24152 6883 Acked Write 9 Done Done Sent false
24153 6884 Acked Read 3 Done Done Sent false
24154 6885 Acked Write 4 Done Done Sent false
24155 6886 Acked Read 7 Done Done Sent false
24156 6887 Acked Write 6 Done Done Sent false
24157 6888 Acked Read 5 Done Done Sent false
24158 6889 Acked Read 8 Done Done Sent false
24159 6890 Acked Read 6 Done Done Sent false
24160 6891 Acked Write 2 Done Done Sent false
24161 6892 Acked Write 6 Done Done Sent false
24162 6893 Acked Read 4 Done Done Sent false
24163 6894 Acked Flush 0 Done Done Sent false
24164 6895 Acked Read 4 Done Done Sent false
24165 6896 Acked Read 8 Done Done Sent false
24166 6897 Acked Write 10 Done Done Sent false
24167 6898 Acked Read 5 Done Done Sent false
24168 6899 Acked Read 6 Done Done Sent false
24169 6900 Acked Read 2 Done Done Sent false
24170 6901 Acked Flush 0 Done Done Sent false
24171 6902 Acked Write 2 Done Done Sent false
24172 6903 Acked Read 3 Done Done Sent false
24173 6904 Acked Write 5 Done Done Sent false
24174 6905 Acked Read 4 Done Done Sent false
24175 6906 Acked Flush 0 Done Done Sent false
24176 6907 Acked Read 1 Done Done Sent false
24177 6908 Acked Read 1 Done Done Sent false
24178 6909 Acked Read 6 Done Done Sent false
24179 6910 Acked Flush 0 Done Done Sent false
24180 6911 Acked Read 5 Done Done Sent false
24181 6912 Acked Read 9 Done Done Sent false
24182 6913 Acked Read 2 Done Done Sent false
24183 6914 Acked Write 10 Done Done Sent false
24184 6915 Acked Read 6 Done Done Sent false
24185 6916 Acked Write 5 Done Done Sent false
24186 6917 Acked Read 7 Done Done Sent false
24187 6918 Acked Read 10 Done Done Sent false
24188 6919 Acked Write 10 Done Done Sent false
24189 6920 Acked Read 6 Done Done Sent false
24190 6921 Acked Write 8 Done Done Sent false
24191 6922 Acked Write 4 Done Done Sent false
24192 6923 Acked Read 3 Done Done Sent false
24193 6924 Acked Read 5 Done Done Sent false
24194 6925 Acked Read 4 Done Done Sent false
24195 6926 Acked Flush 0 Done Done Sent false
24196 6927 Acked Write 10 Done Done Sent false
24197 6928 Acked Write 7 Done Done Sent false
24198 6929 Acked Read 9 Done Done Sent false
24199 6930 Acked Write 10 Done Done Sent false
24200 6931 Acked Write 1 Done Done Sent false
24201 6932 Acked Read 7 Done Done Sent false
24202 6933 Acked Read 3 Done Done Sent false
24203 6934 Acked Flush 0 Done Done Sent false
24204 6935 Acked Read 4 Done Done Sent false
24205 6936 Acked Write 5 Done Done Sent false
24206 6937 Acked Read 5 Done Done Sent false
24207 6938 Acked Write 9 Done Done Sent false
24208 6939 Acked Read 1 Done Done Sent false
24209 6940 Acked Read 8 Done Done Sent false
24210 6941 Acked Read 9 Done Done Sent false
24211 6942 Acked Read 5 Done Done Sent false
24212 6943 Acked Write 5 Done Done Sent false
24213 6944 Acked Read 4 Done Done Sent false
24214 6945 Acked Write 6 Done Done Sent false
24215 6946 Acked Read 2 Done Done Sent false
24216 6947 Acked Write 6 Done Done Sent false
24217 6948 Acked Write 7 Done Done Sent false
24218 6949 Acked Read 10 Done Done Sent false
24219 6950 Acked Write 1 Done Done Sent false
24220 6951 Acked Write 3 Done Done Sent false
24221 6952 Acked Read 9 Done Done Sent false
24222 6953 Acked Write 9 Done Done Sent false
24223 6954 Acked Write 2 Done Done Sent false
24224 6955 Acked Write 7 Done Done Sent false
24225 6956 Acked Flush 0 Done Done Sent false
24226 6957 Acked Write 10 Done Done Sent false
24227 6958 Acked Read 1 Done Done Sent false
24228 6959 Acked Write 8 Done Done Sent false
24229 6960 Acked Flush 0 Done Done Sent false
24230 6961 Acked Flush 0 Done Done Sent false
24231 6962 Acked Write 4 Done Done Sent false
24232 6963 Acked Read 4 Done Done Sent false
24233 6964 Acked Read 1 Done Done Sent false
24234 6965 Acked Flush 0 Done Done Sent false
24235 6966 Acked Read 1 Done Done Sent false
24236 6967 Acked Write 9 Done Done Sent false
24237 6968 Acked Write 9 Done Done Sent false
24238 6969 Acked Read 10 Done Done Sent false
24239 6970 Acked Write 4 Done Done Sent false
24240 6971 Acked Read 2 Done Done Sent false
24241 6972 Acked Read 6 Done Done Sent false
24242 6973 Acked Write 9 Done Done Sent false
24243 6974 Acked Write 7 Done Done Sent false
24244 6975 Acked Read 3 Done Done Sent false
24245 6976 Acked Write 4 Done Done Sent false
24246 6977 Acked Read 6 Done Done Sent false
24247 6978 Acked Read 2 Done Done Sent false
24248 6979 Acked Write 6 Done Done Sent false
24249 6980 Acked Write 5 Done Done Sent false
24250 6981 Acked Write 7 Done Done Sent false
24251 6982 Acked Write 10 Done Done Sent false
24252 6983 Acked Write 4 Done Done Sent false
24253 6984 Acked Read 5 Done Done Sent false
24254 6985 Acked Write 10 Done Done Sent false
24255 6986 Acked Flush 0 Done Done Sent false
24256 6987 Acked Write 6 Done Done Sent false
24257 6988 Acked Read 1 Done Done Sent false
24258 6989 Acked Write 6 Done Done Sent false
24259 6990 Acked Read 4 Done Done Sent false
24260 6991 Acked Write 6 Done Done Sent false
24261 6992 Acked Flush 0 Done Done Sent false
24262 6993 Acked Write 8 Done Done Sent false
24263 6994 Acked Read 3 Done Done Sent false
24264 6995 Acked Read 8 Done Done Sent false
24265 6996 Acked Write 5 Done Done Sent false
24266 6997 Acked Read 1 Done Done Sent false
24267 6998 Acked Write 7 Done Done Sent false
24268 6999 Acked Read 10 Done Done Sent false
24269 7000 Acked Read 9 Done Done Sent false
24270 7001 Acked Flush 0 Done Done Sent false
24271 7002 Acked Read 3 Done Done Sent false
24272 7003 Acked Read 4 Done Done Sent false
24273 7004 Acked Write 2 Done Done Sent false
24274 7005 Acked Write 4 Done Done Sent false
24275 7006 Acked Flush 0 Done Done Sent false
24276 7007 Acked Read 8 Done Done Sent false
24277 7008 Acked Write 6 Done Done Sent false
24278 7009 Acked Write 2 Done Done Sent false
24279 7010 Acked Write 7 Done Done Sent false
24280 7011 Acked Read 3 Done Done Sent false
24281 7012 Acked Read 6 Done Done Sent false
24282 7013 Acked Read 10 Done Done Sent false
24283 7014 Acked Write 6 Done Done Sent false
24284 7015 Acked Read 7 Done Done Sent false
24285 7016 Acked Write 6 Done Done Sent false
24286 7017 Acked Read 7 Done Done Sent false
24287 7018 Acked Write 3 Done Done Sent false
24288 7019 Acked Write 5 Done Done Sent false
24289 7020 Acked Flush 0 Done Done Sent false
24290 7021 Acked Write 9 Done Done Sent false
24291 7022 Acked Write 4 Done Done Sent false
24292 7023 Acked Flush 0 Done Done Sent false
24293 7024 Acked Read 6 Done Done Sent false
24294 7025 Acked Flush 0 Done Done Sent false
24295 7026 Acked Write 6 Done Done Sent false
24296 7027 Acked Read 7 Done Done Sent false
24297 7028 Acked Write 2 Done Done Sent false
24298 7029 Acked Write 9 Done Done Sent false
24299 7030 Acked Read 2 Done Done Sent false
24300 7031 Acked Write 5 Done Done Sent false
24301 7032 Acked Read 9 Done Done Sent false
24302 7033 Acked Write 5 Done Done Sent false
24303 7034 Acked Read 2 Done Done Sent false
24304 7035 Acked Read 6 Done Done Sent false
24305 STATES DS:0 DS:1 DS:2 TOTAL
24306 Sent 0 0 301 301
24307 Done 301 301 0 602
24308 Skipped 0 0 0 0
24309 Error 0 0 0 0
24310 Last Flush: Some(JobId(7025)) Some(JobId(7025)) Some(JobId(6734))
24311 Downstairs last five completed: 6734 6733 6732 6731 6730
24312 Upstairs last five completed: 7035 7034 7033 7032 7031
24313 CLIENT: Up:0 ds:301 act:2
243142025-02-28T06:06:21.874ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
243152025-02-28T06:06:22.632ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
243162025-02-28T06:06:22.632ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:35913) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
243172025-02-28T06:06:22.633ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
243182025-02-28T06:06:22.634ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
243192025-02-28T06:06:22.634ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
243202025-02-28T06:06:22.634ZINFOcrucible: send last flush ID to this DS: Some(JobId(6734)) = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
243212025-02-28T06:06:22.634ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
243222025-02-28T06:06:22.634ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6734)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
243232025-02-28T06:06:22.634ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6734)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
24324 ----------------------------------------------------------------
24325 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24326 Downstairs last five completed: 7036 7035 7034 7033 7032
24327 Upstairs last five completed: 7036 7035 7034 7033 7032
24328 CLIENT: Up:0 ds:0 act:3
24329 Replay: All jobs finished
24330 Main task has work to do, go find it
24331 got dsc Stop(1)
24332 stop 1
24333 Feb 28 06:06:24.877[8820] Got stop action so:false kr:false
24334 INFO request completed, latency_us: 328, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 8db6f951-308f-4113-9945-44dc3d20bebc, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
24335 Feb 28 06:06:24.878 INFO request completed, latency_us: 121, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2a87e503-4aba-49ec-8fab-a6ef89ec9838, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:24.878679225Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
24336 .1:9998
243372025-02-28T06:06:24.878ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
243382025-02-28T06:06:24.878ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
243392025-02-28T06:06:24.878ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
24340 [8820] Exited with: ExitStatus(unix_wait_status(9))
24341 [8820][1] reports Exit
24342 Feb 28 06:06:28.881 INFO request completed, latency_us: 312, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1c464c3a-1784-411b-beb8-00051d28d847, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
24343 001/300 Read block 4028 len 40960
24344 002/300 Flush
24345 003/300 Read block 1692 len 8192
24346 004/300 Write block 3319 len 24576 data: 2 4 4 5 5 5
24347 005/300 Read block 2857 len 28672
24348 006/300 Write block 1040 len 24576 data: 6 5 5 5 5 5
24349 007/300 Read block 373 len 16384
24350 008/300 Write block 3361 len 32768 data: 4 3 4 5 4 5 4 5
24351 009/300 Read block 3657 len 16384
24352 010/300 Read block 424 len 36864
24353 011/300 Read block 3822 len 8192
24354 012/300 Write block 2356 len 28672 data: 4 5 5 5 5 4 5
24355 013/300 Read block 3689 len 32768
24356 014/300 Read block 3145 len 16384
24357 015/300 Read block 4274 len 20480
24358 016/300 Flush
24359 017/300 Write block 4354 len 24576 data: 6 8 6 7 7 6
24360 018/300 Flush
24361 019/300 Read block 2860 len 28672
24362 020/300 Write block 1298 len 40960 data: 5 4 4 4 4 4 4 4 3 3
24363 021/300 Read block 1065 len 28672
24364 022/300 Write block 1120 len 32768 data: 4 3 4 4 5 7 7 7
24365 023/300 Write block 2435 len 36864 data: 4 4 3 3 4 5 4 6 5
24366 024/300 Write block 190 len 28672 data: 5 6 6 6 7 7 7
24367 025/300 Read block 1622 len 40960
24368 026/300 Read block 2146 len 20480
24369 027/300 Read block 745 len 24576
24370 028/300 Read block 368 len 12288
24371 029/300 Flush
24372 030/300 Read block 3193 len 8192
24373 031/300 Read block 3558 len 12288
24374 032/300 Read block 3083 len 8192
24375 033/300 Write block 1049 len 4096 data: 6
24376 034/300 Read block 2121 len 8192
24377 035/300 Read block 359 len 36864
24378 036/300 Write block 1942 len 32768 data: 5 4 4 3 4 5 4 2
24379 037/300 Read block 4673 len 28672
24380 038/300 Read block 744 len 40960
24381 039/300 Write block 2626 len 40960 data: 5 4 4 4 5 5 4 2 3 3
24382 040/300 Read block 2408 len 20480
24383 041/300 Write block 2461 len 4096 data: 5
24384 042/300 Write block 4496 len 24576 data: 6 5 5 6 6 7
24385 043/300 Flush
24386 044/300 Read block 248 len 40960
24387 045/300 Read block 4105 len 24576
24388 046/300 Read block 146 len 24576
24389 047/300 Read block 3966 len 36864
24390 048/300 Write block 4358 len 12288 data: 8 7 4
24391 049/300 Read block 4752 len 28672
24392 050/300 Read block 4691 len 40960
24393 051/300 Read block 842 len 40960
24394 052/300 Write block 2534 len 4096 data: 3
24395 053/300 Read block 4328 len 20480
24396 054/300 Read block 346 len 36864
24397 055/300 Read block 2725 len 40960
24398 056/300 Write block 1335 len 24576 data: 5 5 6 5 5 6
24399 057/300 Write block 3209 len 32768 data: 5 4 4 4 4 4 5 5
24400 058/300 Write block 3108 len 20480 data: 9 9 9 8 9
24401 059/300 Write block 525 len 40960 data: 5 6 3 3 4 3 3 4 4 4
24402 060/300 Write block 3633 len 16384 data: 5 5 4 4
24403 061/300 Read block 3853 len 16384
24404 062/300 Write block 4228 len 12288 data: 11 11 9
24405 063/300 Read block 559 len 12288
24406 064/300 Flush
24407 065/300 Write block 1540 len 4096 data: 6
24408 066/300 Read block 1040 len 24576
24409 067/300 Read block 3126 len 36864
24410 068/300 Read block 221 len 4096
24411 069/300 Write block 1454 len 28672 data: 4 3 4 4 3 4 3
24412 070/300 Read block 3406 len 32768
24413 071/300 Write block 641 len 20480 data: 4 4 4 4 4
24414 072/300 Write block 466 len 20480 data: 4 5 6 6 7
24415 073/300 Read block 2433 len 8192
24416 074/300 Write block 4306 len 28672 data: 5 6 5 5 4 4 4
24417 075/300 Write block 2536 len 8192 data: 3 3
24418 076/300 Read block 2622 len 40960
24419 077/300 Read block 4205 len 36864
24420 078/300 Read block 2433 len 28672
24421 079/300 Write block 4204 len 32768 data: 6 5 6 4 4 4 4 4
24422 080/300 Read block 1713 len 32768
24423 081/300 Write block 4865 len 4096 data: 5
24424 082/300 Write block 4596 len 28672 data: 8 7 6 4 4 4 4
24425 083/300 Read block 3512 len 4096
24426 084/300 Write block 2130 len 20480 data: 3 4 5 5 5
24427 085/300 Write block 2721 len 40960 data: 4 3 2 3 3 3 3 3 4 7
24428 086/300 Write block 3207 len 8192 data: 7 6
24429 087/300 Read block 877 len 16384
24430 088/300 Write block 561 len 28672 data: 5 5 5 5 5 3 3
24431 089/300 Write block 2620 len 40960 data: 2 2 3 3 4 4 6 5 5 5
24432 090/300 Write block 338 len 36864 data: 6 4 4 6 5 5 4 5 5
24433 091/300 Write block 4973 len 24576 data: 5 5 4 4 3 5
24434 092/300 Read block 98 len 24576
24435 093/300 Read block 455 len 28672
24436 094/300 Write block 2451 len 32768 data: 5 5 4 5 7 7 8 8
24437 095/300 Read block 1614 len 32768
24438 096/300 Read block 2979 len 16384
24439 097/300 Write block 1546 len 40960 data: 5 4 4 4 4 5 3 3 5 5
24440 098/300 Read block 826 len 12288
24441 099/300 Read block 2055 len 12288
24442 100/300 Read block 3772 len 12288
24443 101/300 Write block 3148 len 24576 data: 3 2 2 3 3 3
24444 102/300 Flush
24445 103/300 Read block 3453 len 8192
24446 104/300 Write block 345 len 4096 data: 6
24447 105/300 Write block 3964 len 24576 data: 2 2 2 2 2 2
24448 106/300 Read block 3696 len 32768
24449 107/300 Flush
24450 108/300 Flush
24451 109/300 Write block 1118 len 16384 data: 3 3 5 4
24452 110/300 Read block 3358 len 28672
24453 111/300 Write block 3006 len 4096 data: 4
24454 112/300 Read block 4745 len 8192
24455 113/300 Read block 104 len 40960
24456 114/300 Read block 499 len 8192
24457 115/300 Write block 3703 len 24576 data: 2 3 4 5 7 5
24458 116/300 Flush
24459 117/300 Read block 1 len 16384
24460 118/300 Read block 4952 len 12288
24461 119/300 Write block 1190 len 4096 data: 4
24462 120/300 Write block 4404 len 40960 data: 5 8 6 7 7 7 7 7 7 5
24463 121/300 Write block 146 len 24576 data: 5 5 5 6 5 5
24464 122/300 Write block 2905 len 24576 data: 2 2 2 3 3 3
24465 123/300 Read block 28 len 16384
24466 124/300 Read block 811 len 36864
24467 125/300 Write block 994 len 24576 data: 4 3 3 4 4 4
24468 126/300 Write block 994 len 4096 data: 5
24469 127/300 Write block 4835 len 8192 data: 7 8
24470 128/300 Read block 4787 len 12288
24471 129/300 Write block 1716 len 28672 data: 5 5 4 3 4 5 7
24472 130/300 Flush
24473 131/300 Write block 4968 len 24576 data: 7 6 6 7 6 6
24474 132/300 Write block 505 len 40960 data: 5 5 4 4 5 3 3 2 2 2
24475 133/300 Write block 1747 len 20480 data: 6 5 5 5 3
24476 134/300 Read block 3005 len 28672
24477 135/300 Read block 2024 len 16384
24478 136/300 Read block 1749 len 8192
24479 137/300 Flush
24480 138/300 Read block 3997 len 4096
24481 139/300 Read block 635 len 28672
24482 140/300 Write block 3616 len 36864 data: 5 5 4 5 3 4 4 5 5
24483 141/300 Write block 1393 len 20480 data: 2 3 3 3 2
24484 142/300 Read block 2894 len 20480
24485 143/300 Write block 2751 len 4096 data: 6
24486 144/300 Read block 3709 len 4096
24487 145/300 Write block 4486 len 24576 data: 7 7 6 5 5 3
24488 146/300 Write block 3538 len 20480 data: 6 4 4 4 4
24489 147/300 Write block 1142 len 32768 data: 4 4 4 5 4 3 4 4
24490 148/300 Read block 1716 len 16384
24491 149/300 Write block 4747 len 20480 data: 3 3 2 3 3
24492 150/300 Write block 742 len 20480 data: 4 4 5 4 4
24493 151/300 Read block 4890 len 32768
24494 152/300 Read block 1003 len 36864
24495 153/300 Read block 4328 len 32768
24496 154/300 Write block 1993 len 32768 data: 3 3 3 3 3 3 4 6
24497 155/300 Read block 4361 len 16384
24498 156/300 Flush
24499 157/300 Write block 3354 len 28672 data: 5 5 5 4 2 2 4
24500 158/300 Read block 3330 len 12288
24501 159/300 Write block 256 len 24576 data: 5 5 4 4 5 4
24502 160/300 Write block 1251 len 20480 data: 3 4 6 7 8
24503 161/300 Read block 4779 len 12288
24504 162/300 Read block 1672 len 40960
24505 163/300 Write block 461 len 8192 data: 4 3
24506 164/300 Read block 4090 len 24576
24507 165/300 Read block 3268 len 12288
24508 166/300 Write block 3266 len 16384 data: 3 3 3 4
24509 167/300 Read block 4540 len 16384
24510 168/300 Write block 2199 len 32768 data: 4 6 5 6 4 4 3 3
24511 169/300 Read block 4049 len 20480
24512 170/300 Read block 4551 len 40960
24513 171/300 Read block 1478 len 8192
24514 172/300 Flush
24515 173/300 Read block 2419 len 16384
24516 174/300 Read block 2429 len 8192
24517 175/300 Read block 4814 len 20480
24518 176/300 Write block 1433 len 20480 data: 4 4 4 5 6
24519 177/300 Read block 269 len 16384
24520 178/300 Flush
24521 179/300 Read block 4743 len 12288
24522 180/300 Write block 1082 len 8192 data: 5 4
24523 181/300 Write block 1608 len 28672 data: 5 5 3 3 3 5 5
24524 182/300 Write block 774 len 16384 data: 4 5 5 5
24525 183/300 Write block 2911 len 24576 data: 3 3 5 5 4 3
24526 184/300 Flush
24527 185/300 Read block 619 len 32768
24528 186/300 Read block 731 len 20480
24529 187/300 Read block 1642 len 8192
24530 188/300 Read block 4960 len 24576
24531 189/300 Write block 2795 len 32768 data: 6 5 5 4 5 5 5 3
24532 190/300 Read block 1018 len 24576
24533 191/300 Flush
24534 192/300 Read block 1080 len 8192
24535 193/300 Read block 1151 len 40960
24536 194/300 Read block 1736 len 20480
24537 195/300 Write block 723 len 8192 data: 4 4
24538 196/300 Write block 2603 len 4096 data: 5
24539 197/300 Read block 475 len 12288
24540 198/300 Flush
24541 199/300 Flush
24542 200/300 Read block 2908 len 4096
24543 201/300 Write block 1420 len 12288 data: 5 6 5
24544 202/300 Read block 2344 len 32768
24545 203/300 Read block 2094 len 32768
24546 204/300 Read block 2431 len 12288
24547 205/300 Read block 2556 len 32768
24548 206/300 Read block 17 len 40960
24549 207/300 Write block 2931 len 8192 data: 5 4
24550 208/300 Write block 1156 len 36864 data: 7 7 6 6 5 5 4 4 6
24551 209/300 Write block 47 len 16384 data: 4 4 4 3
24552 210/300 Read block 1896 len 20480
24553 211/300 Read block 1249 len 20480
24554 212/300 Write block 2905 len 24576 data: 3 3 3 4 4 4
24555 213/300 Write block 515 len 40960 data: 4 4 4 4 4 5 6 7 7 7
24556 214/300 Flush
24557 215/300 Read block 4775 len 8192
24558 216/300 Write block 2651 len 40960 data: 6 2 3 3 3 4 5 5 5 4
24559 217/300 Read block 2912 len 20480
24560 218/300 Write block 805 len 36864 data: 6 6 5 5 5 5 5 8 10
24561 219/300 Write block 928 len 20480 data: 6 6 7 7 7
24562 220/300 Write block 188 len 16384 data: 5 5 6 7
24563 221/300 Read block 3013 len 36864
24564 222/300 Read block 2050 len 28672
24565 223/300 Read block 3817 len 8192
24566 224/300 Read block 836 len 28672
24567 225/300 Write block 3684 len 40960 data: 4 4 4 4 5 3 5 5 6 4
24568 226/300 Flush
24569 227/300 Write block 3675 len 24576 data: 3 4 3 3 3 3
24570 228/300 Write block 3418 len 28672 data: 4 3 2 2 2 2 3
24571 229/300 Read block 505 len 24576
24572 230/300 Read block 3514 len 40960
24573 231/300 Write block 2673 len 28672 data: 8 8 9 8 5 6 6
24574 232/300 Write block 79 len 20480 data: 7 6 7 5 5
24575 233/300 Write block 3588 len 16384 data: 4 5 4 4
24576 234/300 Write block 2785 len 32768 data: 4 5 5 5 6 6 5 5
24577 235/300 Read block 3018 len 16384
24578 236/300 Read block 594 len 32768
24579 237/300 Read block 4629 len 24576
24580 238/300 Write block 4096 len 36864 data: 6 6 6 6 6 7 6 6 5
24581 239/300 Read block 4293 len 8192
24582 240/300 Flush
24583 241/300 Read block 684 len 40960
24584 242/300 Write block 3768 len 32768 data: 4 4 3 3 3 4 4 2
24585 243/300 Write block 1261 len 32768 data: 7 7 6 7 5 4 3 3
24586 244/300 Flush
24587 245/300 Write block 432 len 40960 data: 5 5 5 6 8 7 8 8 8 8
24588 246/300 Write block 1145 len 32768 data: 6 5 4 5 5 4 6 7
24589 247/300 Read block 1246 len 8192
24590 248/300 Write block 3325 len 40960 data: 5 4 5 4 4 4 4 5 4 5
24591 249/300 Read block 195 len 16384
24592 250/300 Read block 3272 len 8192
24593 251/300 Flush
24594 252/300 Write block 4045 len 16384 data: 7 6 4 3
24595 253/300 Read block 3187 len 36864
24596 254/300 Read block 4480 len 32768
24597 255/300 Read block 139 len 16384
24598 256/300 Read block 4577 len 20480
24599 257/300 Read block 454 len 20480
24600 258/300 Read block 942 len 40960
24601 259/300 Write block 2 len 32768 data: 2 3 4 4 4 3 3 3
24602 260/300 Read block 3529 len 4096
24603 261/300 Read block 4940 len 16384
24604 262/300 Read block 2082 len 16384
24605 263/300 Write block 4887 len 8192 data: 11 10
24606 264/300 Write block 1053 len 8192 data: 6 7
24607 265/300 Read block 231 len 4096
24608 266/300 Read block 1462 len 12288
24609 267/300 Read block 1760 len 36864
24610 268/300 Read block 1429 len 8192
24611 269/300 Write block 1869 len 4096 data: 5
24612 270/300 Write block 2878 len 28672 data: 4 4 3 2 3 4 6
24613 271/300 Read block 2136 len 24576
24614 272/300 Write block 1499 len 36864 data: 3 3 4 3 4 3 3 3 4
24615 273/300 Read block 1747 len 36864
24616 274/300 Write block 3474 len 20480 data: 5 6 5 5 4
24617 275/300 Write block 2467 len 8192 data: 3 4
24618 276/300 Write block 4888 len 40960 data: 11 11 10 9 7 8 8 8 7 7
24619 277/300 Write block 623 len 12288 data: 4 4 5
24620 278/300 Read block 4450 len 20480
24621 279/300 Read block 1610 len 24576
24622 280/300 Flush
24623 281/300 Read block 755 len 16384
24624 282/300 Write block 901 len 16384 data: 5 6 4 3
24625 283/300 Write block 1309 len 12288 data: 5 5 6
24626 284/300 Write block 2419 len 12288 data: 5 5 5
24627 285/300 Read block 4834 len 36864
24628 286/300 Read block 1862 len 16384
24629 287/300 Write block 3742 len 24576 data: 7 8 8 8 6 6
24630 288/300 Read block 3523 len 16384
24631 289/300 Read block 1613 len 16384
24632 290/300 Read block 999 len 8192
24633 291/300 Flush
24634 292/300 Read block 4139 len 24576
24635 293/300 Flush
24636 294/300 Read block 3208 len 4096
24637 295/300 Flush
24638 296/300 Write block 3515 len 32768 data: 3 3 2 2 2 3 3 6
24639 297/300 Write block 2039 len 32768 data: 3 2 2 2 2 2 4 4
24640 298/300 Write block 1037 len 16384 data: 4 4 5 7
24641 299/300 Flush
24642 300/300 Read block 3506 len 32768
24643 301/300 Read block 4405 len 12288
24644 Main task has work to do, go find it
24645 got dsc Start(1)
24646 start 1
24647 Feb 28 06:06:29.125 INFO request completed, latency_us: 179, response_code: 204, uri: /start/cid/1, method: POST, req_id: bdd32c47-d06e-4b96-b21d-eb448149e30b, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
24648 Replay: started 1, returned:Ok(())
24649 ----------------------------------------------------------------
24650 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24651 States: Active Offline Active
24652 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24653 7037 Acked Read 10 Done Sent Done false
24654 7038 Acked Flush 0 Done Sent Done false
24655 7039 Acked Read 2 Done Sent Done false
24656 7040 Acked Write 6 Done Sent Done false
24657 7041 Acked Read 7 Done Sent Done false
24658 7042 Acked Write 6 Done Sent Done false
24659 7043 Acked Read 4 Done Sent Done false
24660 7044 Acked Write 8 Done Sent Done false
24661 7045 Acked Read 4 Done Sent Done false
24662 7046 Acked Read 9 Done Sent Done false
24663 7047 Acked Read 2 Done Sent Done false
24664 7048 Acked Write 7 Done Sent Done false
24665 7049 Acked Read 8 Done Sent Done false
24666 7050 Acked Read 4 Done Sent Done false
24667 7051 Acked Read 5 Done Sent Done false
24668 7052 Acked Flush 0 Done Sent Done false
24669 7053 Acked Write 6 Done Sent Done false
24670 7054 Acked Flush 0 Done Sent Done false
24671 7055 Acked Read 7 Done Sent Done false
24672 7056 Acked Write 10 Done Sent Done false
24673 7057 Acked Read 7 Done Sent Done false
24674 7058 Acked Write 8 Done Sent Done false
24675 7059 Acked Write 9 Done Sent Done false
24676 7060 Acked Write 7 Done Sent Done false
24677 7061 Acked Read 10 Done Sent Done false
24678 7062 Acked Read 5 Done Sent Done false
24679 7063 Acked Read 6 Done Sent Done false
24680 7064 Acked Read 3 Done Sent Done false
24681 7065 Acked Flush 0 Done Sent Done false
24682 7066 Acked Read 2 Done Sent Done false
24683 7067 Acked Read 3 Done Sent Done false
24684 7068 Acked Read 2 Done Sent Done false
24685 7069 Acked Write 1 Done Sent Done false
24686 7070 Acked Read 2 Done Sent Done false
24687 7071 Acked Read 9 Done Sent Done false
24688 7072 Acked Write 8 Done Sent Done false
24689 7073 Acked Read 7 Done Sent Done false
24690 7074 Acked Read 10 Done Sent Done false
24691 7075 Acked Write 10 Done Sent Done false
24692 7076 Acked Read 5 Done Sent Done false
24693 7077 Acked Write 1 Done Sent Done false
24694 7078 Acked Write 6 Done Sent Done false
24695 7079 Acked Flush 0 Done Sent Done false
24696 7080 Acked Read 10 Done Sent Done false
24697 7081 Acked Read 6 Done Sent Done false
24698 7082 Acked Read 6 Done Sent Done false
24699 7083 Acked Read 9 Done Sent Done false
24700 7084 Acked Write 3 Done Sent Done false
24701 7085 Acked Read 7 Done Sent Done false
24702 7086 Acked Read 10 Done Sent Done false
24703 7087 Acked Read 10 Done Sent Done false
24704 7088 Acked Write 1 Done Sent Done false
24705 7089 Acked Read 5 Done Sent Done false
24706 7090 Acked Read 9 Done Sent Done false
24707 7091 Acked Read 10 Done Sent Done false
24708 7092 Acked Write 6 Done Sent Done false
24709 7093 Acked Write 8 Done Sent Done false
24710 7094 Acked Write 5 Done Sent Done false
24711 7095 Acked Write 10 Done Sent Done false
24712 7096 Acked Write 4 Done Sent Done false
24713 7097 Acked Read 4 Done Sent Done false
24714 7098 Acked Write 3 Done Sent Done false
24715 7099 Acked Read 3 Done Sent Done false
24716 7100 Acked Flush 0 Done Sent Done false
24717 7101 Acked Write 1 Done Sent Done false
24718 7102 Acked Read 6 Done Sent Done false
24719 7103 Acked Read 9 Done Sent Done false
24720 7104 Acked Read 1 Done Sent Done false
24721 7105 Acked Write 7 Done Sent Done false
24722 7106 Acked Read 8 Done Sent Done false
24723 7107 Acked Write 5 Done Sent Done false
24724 7108 Acked Write 5 Done Sent Done false
24725 7109 Acked Read 2 Done Sent Done false
24726 7110 Acked Write 7 Done Sent Done false
24727 7111 Acked Write 2 Done Sent Done false
24728 7112 Acked Read 10 Done Sent Done false
24729 7113 Acked Read 9 Done Sent Done false
24730 7114 Acked Read 7 Done Sent Done false
24731 7115 Acked Write 8 Done Sent Done false
24732 7116 Acked Read 8 Done Sent Done false
24733 7117 Acked Write 1 Done Sent Done false
24734 7118 Acked Write 7 Done Sent Done false
24735 7119 Acked Read 1 Done Sent Done false
24736 7120 Acked Write 5 Done Sent Done false
24737 7121 Acked Write 10 Done Sent Done false
24738 7122 Acked Write 2 Done Sent Done false
24739 7123 Acked Read 4 Done Sent Done false
24740 7124 Acked Write 7 Done Sent Done false
24741 7125 Acked Write 10 Done Sent Done false
24742 7126 Acked Write 9 Done Sent Done false
24743 7127 Acked Write 6 Done Sent Done false
24744 7128 Acked Read 6 Done Sent Done false
24745 7129 Acked Read 7 Done Sent Done false
24746 7130 Acked Write 8 Done Sent Done false
24747 7131 Acked Read 8 Done Sent Done false
24748 7132 Acked Read 4 Done Sent Done false
24749 7133 Acked Write 10 Done Sent Done false
24750 7134 Acked Read 3 Done Sent Done false
24751 7135 Acked Read 3 Done Sent Done false
24752 7136 Acked Read 3 Done Sent Done false
24753 7137 Acked Write 6 Done Sent Done false
24754 7138 Acked Flush 0 Done Sent Done false
24755 7139 Acked Read 2 Done Sent Done false
24756 7140 Acked Write 1 Done Sent Done false
24757 7141 Acked Write 6 Done Sent Done false
24758 7142 Acked Read 8 Done Sent Done false
24759 7143 Acked Flush 0 Done Sent Done false
24760 7144 Acked Flush 0 Done Sent Done false
24761 7145 Acked Write 4 Done Sent Done false
24762 7146 Acked Read 7 Done Sent Done false
24763 7147 Acked Write 1 Done Sent Done false
24764 7148 Acked Read 2 Done Sent Done false
24765 7149 Acked Read 10 Done Sent Done false
24766 7150 Acked Read 2 Done Sent Done false
24767 7151 Acked Write 6 Done Sent Done false
24768 7152 Acked Flush 0 Done Sent Done false
24769 7153 Acked Read 4 Done Sent Done false
24770 7154 Acked Read 3 Done Sent Done false
24771 7155 Acked Write 1 Done Sent Done false
24772 7156 Acked Write 10 Done Sent Done false
24773 7157 Acked Write 6 Done Sent Done false
24774 7158 Acked Write 6 Done Sent Done false
24775 7159 Acked Read 4 Done Sent Done false
24776 7160 Acked Read 9 Done Sent Done false
24777 7161 Acked Write 6 Done Sent Done false
24778 7162 Acked Write 1 Done Sent Done false
24779 7163 Acked Write 2 Done Sent Done false
24780 7164 Acked Read 3 Done Sent Done false
24781 7165 Acked Write 7 Done Sent Done false
24782 7166 Acked Flush 0 Done Sent Done false
24783 7167 Acked Write 6 Done Sent Done false
24784 7168 Acked Write 10 Done Sent Done false
24785 7169 Acked Write 5 Done Sent Done false
24786 7170 Acked Read 7 Done Sent Done false
24787 7171 Acked Read 4 Done Sent Done false
24788 7172 Acked Read 2 Done Sent Done false
24789 7173 Acked Flush 0 Done Sent Done false
24790 7174 Acked Read 1 Done Sent Done false
24791 7175 Acked Read 7 Done Sent Done false
24792 7176 Acked Write 9 Done Sent Done false
24793 7177 Acked Write 5 Done Sent Done false
24794 7178 Acked Read 5 Done Sent Done false
24795 7179 Acked Write 1 Done Sent Done false
24796 7180 Acked Read 1 Done Sent Done false
24797 7181 Acked Write 6 Done Sent Done false
24798 7182 Acked Write 5 Done Sent Done false
24799 7183 Acked Write 8 Done Sent Done false
24800 7184 Acked Read 4 Done Sent Done false
24801 7185 Acked Write 5 Done Sent Done false
24802 7186 Acked Write 5 Done Sent Done false
24803 7187 Acked Read 8 Done Sent Done false
24804 7188 Acked Read 9 Done Sent Done false
24805 7189 Acked Read 8 Done Sent Done false
24806 7190 Acked Write 8 Done Sent Done false
24807 7191 Acked Read 4 Done Sent Done false
24808 7192 Acked Flush 0 Done Sent Done false
24809 7193 Acked Write 7 Done Sent Done false
24810 7194 Acked Read 3 Done Sent Done false
24811 7195 Acked Write 6 Done Sent Done false
24812 7196 Acked Write 5 Done Sent Done false
24813 7197 Acked Read 3 Done Sent Done false
24814 7198 Acked Read 10 Done Sent Done false
24815 7199 Acked Write 2 Done Sent Done false
24816 7200 Acked Read 6 Done Sent Done false
24817 7201 Acked Read 3 Done Sent Done false
24818 7202 Acked Write 4 Done Sent Done false
24819 7203 Acked Read 4 Done Sent Done false
24820 7204 Acked Write 8 Done Sent Done false
24821 7205 Acked Read 5 Done Sent Done false
24822 7206 Acked Read 10 Done Sent Done false
24823 7207 Acked Read 2 Done Sent Done false
24824 7208 Acked Flush 0 Done Sent Done false
24825 7209 Acked Read 4 Done Sent Done false
24826 7210 Acked Read 2 Done Sent Done false
24827 7211 Acked Read 5 Done Sent Done false
24828 7212 Acked Write 5 Done Sent Done false
24829 7213 Acked Read 4 Done Sent Done false
24830 7214 Acked Flush 0 Done Sent Done false
24831 7215 Acked Read 3 Done Sent Done false
24832 7216 Acked Write 2 Done Sent Done false
24833 7217 Acked Write 7 Done Sent Done false
24834 7218 Acked Write 4 Done Sent Done false
24835 7219 Acked Write 6 Done Sent Done false
24836 7220 Acked Flush 0 Done Sent Done false
24837 7221 Acked Read 8 Done Sent Done false
24838 7222 Acked Read 5 Done Sent Done false
24839 7223 Acked Read 2 Done Sent Done false
24840 7224 Acked Read 6 Done Sent Done false
24841 7225 Acked Write 8 Done Sent Done false
24842 7226 Acked Read 6 Done Sent Done false
24843 7227 Acked Flush 0 Done Sent Done false
24844 7228 Acked Read 2 Done Sent Done false
24845 7229 Acked Read 10 Done Sent Done false
24846 7230 Acked Read 5 Done Sent Done false
24847 7231 Acked Write 2 Done Sent Done false
24848 7232 Acked Write 1 Done Sent Done false
24849 7233 Acked Read 3 Done Sent Done false
24850 7234 Acked Flush 0 Done Sent Done false
24851 7235 Acked Flush 0 Done Sent Done false
24852 7236 Acked Read 1 Done Sent Done false
24853 7237 Acked Write 3 Done Sent Done false
24854 7238 Acked Read 8 Done Sent Done false
24855 7239 Acked Read 8 Done Sent Done false
24856 7240 Acked Read 3 Done Sent Done false
24857 7241 Acked Read 8 Done Sent Done false
24858 7242 Acked Read 10 Done Sent Done false
24859 7243 Acked Write 2 Done Sent Done false
24860 7244 Acked Write 9 Done Sent Done false
24861 7245 Acked Write 4 Done Sent Done false
24862 7246 Acked Read 5 Done Sent Done false
24863 7247 Acked Read 5 Done Sent Done false
24864 7248 Acked Write 6 Done Sent Done false
24865 7249 Acked Write 10 Done Sent Done false
24866 7250 Acked Flush 0 Done Sent Done false
24867 7251 Acked Read 2 Done Sent Done false
24868 7252 Acked Write 10 Done Sent Done false
24869 7253 Acked Read 5 Done Sent Done false
24870 7254 Acked Write 9 Done Sent Done false
24871 7255 Acked Write 5 Done Sent Done false
24872 7256 Acked Write 4 Done Sent Done false
24873 7257 Acked Read 9 Done Sent Done false
24874 7258 Acked Read 7 Done Sent Done false
24875 7259 Acked Read 2 Done Sent Done false
24876 7260 Acked Read 7 Done Sent Done false
24877 7261 Acked Write 10 Done Sent Done false
24878 7262 Acked Flush 0 Done Sent Done false
24879 7263 Acked Write 6 Done Sent Done false
24880 7264 Acked Write 7 Done Sent Done false
24881 7265 Acked Read 6 Done Sent Done false
24882 7266 Acked Read 10 Done Sent Done false
24883 7267 Acked Write 7 Done Sent Done false
24884 7268 Acked Write 5 Done Sent Done false
24885 7269 Acked Write 4 Done Sent Done false
24886 7270 Acked Write 8 Done Sent Done false
24887 7271 Acked Read 4 Done Sent Done false
24888 7272 Acked Read 8 Done Sent Done false
24889 7273 Acked Read 6 Done Sent Done false
24890 7274 Acked Write 9 Done Sent Done false
24891 7275 Acked Read 2 Done Sent Done false
24892 7276 Acked Flush 0 Done Sent Done false
24893 7277 Acked Read 10 Done Sent Done false
24894 7278 Acked Write 8 Done Sent Done false
24895 7279 Acked Write 8 Done Sent Done false
24896 7280 Acked Flush 0 Done Sent Done false
24897 7281 Acked Write 10 Done Sent Done false
24898 7282 Acked Write 8 Done Sent Done false
24899 7283 Acked Read 2 Done Sent Done false
24900 7284 Acked Write 10 Done Sent Done false
24901 7285 Acked Read 4 Done Sent Done false
24902 7286 Acked Read 2 Done Sent Done false
24903 7287 Acked Flush 0 Done Sent Done false
24904 7288 Acked Write 4 Done Sent Done false
24905 7289 Acked Read 9 Done Sent Done false
24906 7290 Acked Read 8 Done Sent Done false
24907 7291 Acked Read 4 Done Sent Done false
24908 7292 Acked Read 5 Done Sent Done false
24909 7293 Acked Read 5 Done Sent Done false
24910 7294 Acked Read 10 Done Sent Done false
24911 7295 Acked Write 8 Done Sent Done false
24912 7296 Acked Read 1 Done Sent Done false
24913 7297 Acked Read 4 Done Sent Done false
24914 7298 Acked Read 4 Done Sent Done false
24915 7299 Acked Write 2 Done Sent Done false
24916 7300 Acked Write 2 Done Sent Done false
24917 7301 Acked Read 1 Done Sent Done false
24918 7302 Acked Read 3 Done Sent Done false
24919 7303 Acked Read 9 Done Sent Done false
24920 7304 Acked Read 2 Done Sent Done false
24921 7305 Acked Write 1 Done Sent Done false
24922 7306 Acked Write 7 Done Sent Done false
24923 7307 Acked Read 6 Done Sent Done false
24924 7308 Acked Write 9 Done Sent Done false
24925 7309 Acked Read 9 Done Sent Done false
24926 7310 Acked Write 5 Done Sent Done false
24927 7311 Acked Write 2 Done Sent Done false
24928 7312 Acked Write 10 Done Sent Done false
24929 7313 Acked Write 3 Done Sent Done false
24930 7314 Acked Read 5 Done Sent Done false
24931 7315 Acked Read 6 Done Sent Done false
24932 7316 Acked Flush 0 Done Sent Done false
24933 7317 Acked Read 4 Done Sent Done false
24934 7318 Acked Write 4 Done Sent Done false
24935 7319 Acked Write 3 Done Sent Done false
24936 7320 Acked Write 3 Done Sent Done false
24937 7321 Acked Read 9 Done Sent Done false
24938 7322 Acked Read 4 Done Sent Done false
24939 7323 Acked Write 6 Done Sent Done false
24940 7324 Acked Read 4 Done Sent Done false
24941 7325 Acked Read 4 Done Sent Done false
24942 7326 Acked Read 2 Done Sent Done false
24943 7327 Acked Flush 0 Done Sent Done false
24944 7328 Acked Read 6 Done Sent Done false
24945 7329 Acked Flush 0 Done Sent Done false
24946 7330 Acked Read 1 Done Sent Done false
24947 7331 Acked Flush 0 Done Sent Done false
24948 7332 Acked Write 8 Done Sent Done false
24949 7333 Acked Write 8 Done Sent Done false
24950 7334 Acked Write 4 Done Sent Done false
24951 7335 Acked Flush 0 Done Sent Done false
24952 7336 Acked Read 8 Done Sent Done false
24953 7337 Acked Read 3 Done Sent Done false
24954 STATES DS:0 DS:1 DS:2 TOTAL
24955 Sent 0 301 0 301
24956 Done 301 0 301 602
24957 Skipped 0 0 0 0
24958 Error 0 0 0 0
24959 Last Flush: Some(JobId(7335)) Some(JobId(7036)) Some(JobId(7335))
24960 Downstairs last five completed: 7036 7035 7034 7033 7032
24961 Upstairs last five completed: 7337 7336 7335 7334 7333
24962 CLIENT: Up:0 ds:301 act:2
249632025-02-28T06:06:29.888ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
249642025-02-28T06:06:30.889ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
249652025-02-28T06:06:31.891ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
24966 ----------------------------------------------------------------
24967 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24968 States: Active Offline Active
24969 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24970 7037 Acked Read 10 Done Sent Done false
24971 7038 Acked Flush 0 Done Sent Done false
24972 7039 Acked Read 2 Done Sent Done false
24973 7040 Acked Write 6 Done Sent Done false
24974 7041 Acked Read 7 Done Sent Done false
24975 7042 Acked Write 6 Done Sent Done false
24976 7043 Acked Read 4 Done Sent Done false
24977 7044 Acked Write 8 Done Sent Done false
24978 7045 Acked Read 4 Done Sent Done false
24979 7046 Acked Read 9 Done Sent Done false
24980 7047 Acked Read 2 Done Sent Done false
24981 7048 Acked Write 7 Done Sent Done false
24982 7049 Acked Read 8 Done Sent Done false
24983 7050 Acked Read 4 Done Sent Done false
24984 7051 Acked Read 5 Done Sent Done false
24985 7052 Acked Flush 0 Done Sent Done false
24986 7053 Acked Write 6 Done Sent Done false
24987 7054 Acked Flush 0 Done Sent Done false
24988 7055 Acked Read 7 Done Sent Done false
24989 7056 Acked Write 10 Done Sent Done false
24990 7057 Acked Read 7 Done Sent Done false
24991 7058 Acked Write 8 Done Sent Done false
24992 7059 Acked Write 9 Done Sent Done false
24993 7060 Acked Write 7 Done Sent Done false
24994 7061 Acked Read 10 Done Sent Done false
24995 7062 Acked Read 5 Done Sent Done false
24996 7063 Acked Read 6 Done Sent Done false
24997 7064 Acked Read 3 Done Sent Done false
24998 7065 Acked Flush 0 Done Sent Done false
24999 7066 Acked Read 2 Done Sent Done false
25000 7067 Acked Read 3 Done Sent Done false
25001 7068 Acked Read 2 Done Sent Done false
25002 7069 Acked Write 1 Done Sent Done false
25003 7070 Acked Read 2 Done Sent Done false
25004 7071 Acked Read 9 Done Sent Done false
25005 7072 Acked Write 8 Done Sent Done false
25006 7073 Acked Read 7 Done Sent Done false
25007 7074 Acked Read 10 Done Sent Done false
25008 7075 Acked Write 10 Done Sent Done false
25009 7076 Acked Read 5 Done Sent Done false
25010 7077 Acked Write 1 Done Sent Done false
25011 7078 Acked Write 6 Done Sent Done false
25012 7079 Acked Flush 0 Done Sent Done false
25013 7080 Acked Read 10 Done Sent Done false
25014 7081 Acked Read 6 Done Sent Done false
25015 7082 Acked Read 6 Done Sent Done false
25016 7083 Acked Read 9 Done Sent Done false
25017 7084 Acked Write 3 Done Sent Done false
25018 7085 Acked Read 7 Done Sent Done false
25019 7086 Acked Read 10 Done Sent Done false
25020 7087 Acked Read 10 Done Sent Done false
25021 7088 Acked Write 1 Done Sent Done false
25022 7089 Acked Read 5 Done Sent Done false
25023 7090 Acked Read 9 Done Sent Done false
25024 7091 Acked Read 10 Done Sent Done false
25025 7092 Acked Write 6 Done Sent Done false
25026 7093 Acked Write 8 Done Sent Done false
25027 7094 Acked Write 5 Done Sent Done false
25028 7095 Acked Write 10 Done Sent Done false
25029 7096 Acked Write 4 Done Sent Done false
25030 7097 Acked Read 4 Done Sent Done false
25031 7098 Acked Write 3 Done Sent Done false
25032 7099 Acked Read 3 Done Sent Done false
25033 7100 Acked Flush 0 Done Sent Done false
25034 7101 Acked Write 1 Done Sent Done false
25035 7102 Acked Read 6 Done Sent Done false
25036 7103 Acked Read 9 Done Sent Done false
25037 7104 Acked Read 1 Done Sent Done false
25038 7105 Acked Write 7 Done Sent Done false
25039 7106 Acked Read 8 Done Sent Done false
25040 7107 Acked Write 5 Done Sent Done false
25041 7108 Acked Write 5 Done Sent Done false
25042 7109 Acked Read 2 Done Sent Done false
25043 7110 Acked Write 7 Done Sent Done false
25044 7111 Acked Write 2 Done Sent Done false
25045 7112 Acked Read 10 Done Sent Done false
25046 7113 Acked Read 9 Done Sent Done false
25047 7114 Acked Read 7 Done Sent Done false
25048 7115 Acked Write 8 Done Sent Done false
25049 7116 Acked Read 8 Done Sent Done false
25050 7117 Acked Write 1 Done Sent Done false
25051 7118 Acked Write 7 Done Sent Done false
25052 7119 Acked Read 1 Done Sent Done false
25053 7120 Acked Write 5 Done Sent Done false
25054 7121 Acked Write 10 Done Sent Done false
25055 7122 Acked Write 2 Done Sent Done false
25056 7123 Acked Read 4 Done Sent Done false
25057 7124 Acked Write 7 Done Sent Done false
25058 7125 Acked Write 10 Done Sent Done false
25059 7126 Acked Write 9 Done Sent Done false
25060 7127 Acked Write 6 Done Sent Done false
25061 7128 Acked Read 6 Done Sent Done false
25062 7129 Acked Read 7 Done Sent Done false
25063 7130 Acked Write 8 Done Sent Done false
25064 7131 Acked Read 8 Done Sent Done false
25065 7132 Acked Read 4 Done Sent Done false
25066 7133 Acked Write 10 Done Sent Done false
25067 7134 Acked Read 3 Done Sent Done false
25068 7135 Acked Read 3 Done Sent Done false
25069 7136 Acked Read 3 Done Sent Done false
25070 7137 Acked Write 6 Done Sent Done false
25071 7138 Acked Flush 0 Done Sent Done false
25072 7139 Acked Read 2 Done Sent Done false
25073 7140 Acked Write 1 Done Sent Done false
25074 7141 Acked Write 6 Done Sent Done false
25075 7142 Acked Read 8 Done Sent Done false
25076 7143 Acked Flush 0 Done Sent Done false
25077 7144 Acked Flush 0 Done Sent Done false
25078 7145 Acked Write 4 Done Sent Done false
25079 7146 Acked Read 7 Done Sent Done false
25080 7147 Acked Write 1 Done Sent Done false
25081 7148 Acked Read 2 Done Sent Done false
25082 7149 Acked Read 10 Done Sent Done false
25083 7150 Acked Read 2 Done Sent Done false
25084 7151 Acked Write 6 Done Sent Done false
25085 7152 Acked Flush 0 Done Sent Done false
25086 7153 Acked Read 4 Done Sent Done false
25087 7154 Acked Read 3 Done Sent Done false
25088 7155 Acked Write 1 Done Sent Done false
25089 7156 Acked Write 10 Done Sent Done false
25090 7157 Acked Write 6 Done Sent Done false
25091 7158 Acked Write 6 Done Sent Done false
25092 7159 Acked Read 4 Done Sent Done false
25093 7160 Acked Read 9 Done Sent Done false
25094 7161 Acked Write 6 Done Sent Done false
25095 7162 Acked Write 1 Done Sent Done false
25096 7163 Acked Write 2 Done Sent Done false
25097 7164 Acked Read 3 Done Sent Done false
25098 7165 Acked Write 7 Done Sent Done false
25099 7166 Acked Flush 0 Done Sent Done false
25100 7167 Acked Write 6 Done Sent Done false
25101 7168 Acked Write 10 Done Sent Done false
25102 7169 Acked Write 5 Done Sent Done false
25103 7170 Acked Read 7 Done Sent Done false
25104 7171 Acked Read 4 Done Sent Done false
25105 7172 Acked Read 2 Done Sent Done false
25106 7173 Acked Flush 0 Done Sent Done false
25107 7174 Acked Read 1 Done Sent Done false
25108 7175 Acked Read 7 Done Sent Done false
25109 7176 Acked Write 9 Done Sent Done false
25110 7177 Acked Write 5 Done Sent Done false
25111 7178 Acked Read 5 Done Sent Done false
25112 7179 Acked Write 1 Done Sent Done false
25113 7180 Acked Read 1 Done Sent Done false
25114 7181 Acked Write 6 Done Sent Done false
25115 7182 Acked Write 5 Done Sent Done false
25116 7183 Acked Write 8 Done Sent Done false
25117 7184 Acked Read 4 Done Sent Done false
25118 7185 Acked Write 5 Done Sent Done false
25119 7186 Acked Write 5 Done Sent Done false
25120 7187 Acked Read 8 Done Sent Done false
25121 7188 Acked Read 9 Done Sent Done false
25122 7189 Acked Read 8 Done Sent Done false
25123 7190 Acked Write 8 Done Sent Done false
25124 7191 Acked Read 4 Done Sent Done false
25125 7192 Acked Flush 0 Done Sent Done false
25126 7193 Acked Write 7 Done Sent Done false
25127 7194 Acked Read 3 Done Sent Done false
25128 7195 Acked Write 6 Done Sent Done false
25129 7196 Acked Write 5 Done Sent Done false
25130 7197 Acked Read 3 Done Sent Done false
25131 7198 Acked Read 10 Done Sent Done false
25132 7199 Acked Write 2 Done Sent Done false
25133 7200 Acked Read 6 Done Sent Done false
25134 7201 Acked Read 3 Done Sent Done false
25135 7202 Acked Write 4 Done Sent Done false
25136 7203 Acked Read 4 Done Sent Done false
25137 7204 Acked Write 8 Done Sent Done false
25138 7205 Acked Read 5 Done Sent Done false
25139 7206 Acked Read 10 Done Sent Done false
25140 7207 Acked Read 2 Done Sent Done false
25141 7208 Acked Flush 0 Done Sent Done false
25142 7209 Acked Read 4 Done Sent Done false
25143 7210 Acked Read 2 Done Sent Done false
25144 7211 Acked Read 5 Done Sent Done false
25145 7212 Acked Write 5 Done Sent Done false
25146 7213 Acked Read 4 Done Sent Done false
25147 7214 Acked Flush 0 Done Sent Done false
25148 7215 Acked Read 3 Done Sent Done false
25149 7216 Acked Write 2 Done Sent Done false
25150 7217 Acked Write 7 Done Sent Done false
25151 7218 Acked Write 4 Done Sent Done false
25152 7219 Acked Write 6 Done Sent Done false
25153 7220 Acked Flush 0 Done Sent Done false
25154 7221 Acked Read 8 Done Sent Done false
25155 7222 Acked Read 5 Done Sent Done false
25156 7223 Acked Read 2 Done Sent Done false
25157 7224 Acked Read 6 Done Sent Done false
25158 7225 Acked Write 8 Done Sent Done false
25159 7226 Acked Read 6 Done Sent Done false
25160 7227 Acked Flush 0 Done Sent Done false
25161 7228 Acked Read 2 Done Sent Done false
25162 7229 Acked Read 10 Done Sent Done false
25163 7230 Acked Read 5 Done Sent Done false
25164 7231 Acked Write 2 Done Sent Done false
25165 7232 Acked Write 1 Done Sent Done false
25166 7233 Acked Read 3 Done Sent Done false
25167 7234 Acked Flush 0 Done Sent Done false
25168 7235 Acked Flush 0 Done Sent Done false
25169 7236 Acked Read 1 Done Sent Done false
25170 7237 Acked Write 3 Done Sent Done false
25171 7238 Acked Read 8 Done Sent Done false
25172 7239 Acked Read 8 Done Sent Done false
25173 7240 Acked Read 3 Done Sent Done false
25174 7241 Acked Read 8 Done Sent Done false
25175 7242 Acked Read 10 Done Sent Done false
25176 7243 Acked Write 2 Done Sent Done false
25177 7244 Acked Write 9 Done Sent Done false
25178 7245 Acked Write 4 Done Sent Done false
25179 7246 Acked Read 5 Done Sent Done false
25180 7247 Acked Read 5 Done Sent Done false
25181 7248 Acked Write 6 Done Sent Done false
25182 7249 Acked Write 10 Done Sent Done false
25183 7250 Acked Flush 0 Done Sent Done false
25184 7251 Acked Read 2 Done Sent Done false
25185 7252 Acked Write 10 Done Sent Done false
25186 7253 Acked Read 5 Done Sent Done false
25187 7254 Acked Write 9 Done Sent Done false
25188 7255 Acked Write 5 Done Sent Done false
25189 7256 Acked Write 4 Done Sent Done false
25190 7257 Acked Read 9 Done Sent Done false
25191 7258 Acked Read 7 Done Sent Done false
25192 7259 Acked Read 2 Done Sent Done false
25193 7260 Acked Read 7 Done Sent Done false
25194 7261 Acked Write 10 Done Sent Done false
25195 7262 Acked Flush 0 Done Sent Done false
25196 7263 Acked Write 6 Done Sent Done false
25197 7264 Acked Write 7 Done Sent Done false
25198 7265 Acked Read 6 Done Sent Done false
25199 7266 Acked Read 10 Done Sent Done false
25200 7267 Acked Write 7 Done Sent Done false
25201 7268 Acked Write 5 Done Sent Done false
25202 7269 Acked Write 4 Done Sent Done false
25203 7270 Acked Write 8 Done Sent Done false
25204 7271 Acked Read 4 Done Sent Done false
25205 7272 Acked Read 8 Done Sent Done false
25206 7273 Acked Read 6 Done Sent Done false
25207 7274 Acked Write 9 Done Sent Done false
25208 7275 Acked Read 2 Done Sent Done false
25209 7276 Acked Flush 0 Done Sent Done false
25210 7277 Acked Read 10 Done Sent Done false
25211 7278 Acked Write 8 Done Sent Done false
25212 7279 Acked Write 8 Done Sent Done false
25213 7280 Acked Flush 0 Done Sent Done false
25214 7281 Acked Write 10 Done Sent Done false
25215 7282 Acked Write 8 Done Sent Done false
25216 7283 Acked Read 2 Done Sent Done false
25217 7284 Acked Write 10 Done Sent Done false
25218 7285 Acked Read 4 Done Sent Done false
25219 7286 Acked Read 2 Done Sent Done false
25220 7287 Acked Flush 0 Done Sent Done false
25221 7288 Acked Write 4 Done Sent Done false
25222 7289 Acked Read 9 Done Sent Done false
25223 7290 Acked Read 8 Done Sent Done false
25224 7291 Acked Read 4 Done Sent Done false
25225 7292 Acked Read 5 Done Sent Done false
25226 7293 Acked Read 5 Done Sent Done false
25227 7294 Acked Read 10 Done Sent Done false
25228 7295 Acked Write 8 Done Sent Done false
25229 7296 Acked Read 1 Done Sent Done false
25230 7297 Acked Read 4 Done Sent Done false
25231 7298 Acked Read 4 Done Sent Done false
25232 7299 Acked Write 2 Done Sent Done false
25233 7300 Acked Write 2 Done Sent Done false
25234 7301 Acked Read 1 Done Sent Done false
25235 7302 Acked Read 3 Done Sent Done false
25236 7303 Acked Read 9 Done Sent Done false
25237 7304 Acked Read 2 Done Sent Done false
25238 7305 Acked Write 1 Done Sent Done false
25239 7306 Acked Write 7 Done Sent Done false
25240 7307 Acked Read 6 Done Sent Done false
25241 7308 Acked Write 9 Done Sent Done false
25242 7309 Acked Read 9 Done Sent Done false
25243 7310 Acked Write 5 Done Sent Done false
25244 7311 Acked Write 2 Done Sent Done false
25245 7312 Acked Write 10 Done Sent Done false
25246 7313 Acked Write 3 Done Sent Done false
25247 7314 Acked Read 5 Done Sent Done false
25248 7315 Acked Read 6 Done Sent Done false
25249 7316 Acked Flush 0 Done Sent Done false
25250 7317 Acked Read 4 Done Sent Done false
25251 7318 Acked Write 4 Done Sent Done false
25252 7319 Acked Write 3 Done Sent Done false
25253 7320 Acked Write 3 Done Sent Done false
25254 7321 Acked Read 9 Done Sent Done false
25255 7322 Acked Read 4 Done Sent Done false
25256 7323 Acked Write 6 Done Sent Done false
25257 7324 Acked Read 4 Done Sent Done false
25258 7325 Acked Read 4 Done Sent Done false
25259 7326 Acked Read 2 Done Sent Done false
25260 7327 Acked Flush 0 Done Sent Done false
25261 7328 Acked Read 6 Done Sent Done false
25262 7329 Acked Flush 0 Done Sent Done false
25263 7330 Acked Read 1 Done Sent Done false
25264 7331 Acked Flush 0 Done Sent Done false
25265 7332 Acked Write 8 Done Sent Done false
25266 7333 Acked Write 8 Done Sent Done false
25267 7334 Acked Write 4 Done Sent Done false
25268 7335 Acked Flush 0 Done Sent Done false
25269 7336 Acked Read 8 Done Sent Done false
25270 7337 Acked Read 3 Done Sent Done false
25271 STATES DS:0 DS:1 DS:2 TOTAL
25272 Sent 0 301 0 301
25273 Done 301 0 301 602
25274 Skipped 0 0 0 0
25275 Error 0 0 0 0
25276 Last Flush: Some(JobId(7335)) Some(JobId(7036)) Some(JobId(7335))
25277 Downstairs last five completed: 7036 7035 7034 7033 7032
25278 Upstairs last five completed: 7337 7336 7335 7334 7333
25279 CLIENT: Up:0 ds:301 act:2
252802025-02-28T06:06:33.894ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
252812025-02-28T06:06:34.879ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
252822025-02-28T06:06:34.880ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
252832025-02-28T06:06:34.880ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
252842025-02-28T06:06:34.880ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
252852025-02-28T06:06:34.880ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
25286 [8820] got start action
25287 [8820] I am going to restart
25288 Starting downstairs at port 8820
25289 Make output file at "/tmp/dsc/downstairs-8820.txt"
25290 [8820][1] reports Starting
25291 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1232)
25292 [8820][1] reports Running
252932025-02-28T06:06:35.381ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
252942025-02-28T06:06:35.897ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
252952025-02-28T06:06:36.397ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
252962025-02-28T06:06:36.898ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
25297 ----------------------------------------------------------------
25298 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25299 States: Active Offline Active
25300 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25301 7037 Acked Read 10 Done Sent Done false
25302 7038 Acked Flush 0 Done Sent Done false
25303 7039 Acked Read 2 Done Sent Done false
25304 7040 Acked Write 6 Done Sent Done false
25305 7041 Acked Read 7 Done Sent Done false
25306 7042 Acked Write 6 Done Sent Done false
25307 7043 Acked Read 4 Done Sent Done false
25308 7044 Acked Write 8 Done Sent Done false
25309 7045 Acked Read 4 Done Sent Done false
25310 7046 Acked Read 9 Done Sent Done false
25311 7047 Acked Read 2 Done Sent Done false
25312 7048 Acked Write 7 Done Sent Done false
25313 7049 Acked Read 8 Done Sent Done false
25314 7050 Acked Read 4 Done Sent Done false
25315 7051 Acked Read 5 Done Sent Done false
25316 7052 Acked Flush 0 Done Sent Done false
25317 7053 Acked Write 6 Done Sent Done false
25318 7054 Acked Flush 0 Done Sent Done false
25319 7055 Acked Read 7 Done Sent Done false
25320 7056 Acked Write 10 Done Sent Done false
25321 7057 Acked Read 7 Done Sent Done false
25322 7058 Acked Write 8 Done Sent Done false
25323 7059 Acked Write 9 Done Sent Done false
25324 7060 Acked Write 7 Done Sent Done false
25325 7061 Acked Read 10 Done Sent Done false
25326 7062 Acked Read 5 Done Sent Done false
25327 7063 Acked Read 6 Done Sent Done false
25328 7064 Acked Read 3 Done Sent Done false
25329 7065 Acked Flush 0 Done Sent Done false
25330 7066 Acked Read 2 Done Sent Done false
25331 7067 Acked Read 3 Done Sent Done false
25332 7068 Acked Read 2 Done Sent Done false
25333 7069 Acked Write 1 Done Sent Done false
25334 7070 Acked Read 2 Done Sent Done false
25335 7071 Acked Read 9 Done Sent Done false
25336 7072 Acked Write 8 Done Sent Done false
25337 7073 Acked Read 7 Done Sent Done false
25338 7074 Acked Read 10 Done Sent Done false
25339 7075 Acked Write 10 Done Sent Done false
25340 7076 Acked Read 5 Done Sent Done false
25341 7077 Acked Write 1 Done Sent Done false
25342 7078 Acked Write 6 Done Sent Done false
25343 7079 Acked Flush 0 Done Sent Done false
25344 7080 Acked Read 10 Done Sent Done false
25345 7081 Acked Read 6 Done Sent Done false
25346 7082 Acked Read 6 Done Sent Done false
25347 7083 Acked Read 9 Done Sent Done false
25348 7084 Acked Write 3 Done Sent Done false
25349 7085 Acked Read 7 Done Sent Done false
25350 7086 Acked Read 10 Done Sent Done false
25351 7087 Acked Read 10 Done Sent Done false
25352 7088 Acked Write 1 Done Sent Done false
25353 7089 Acked Read 5 Done Sent Done false
25354 7090 Acked Read 9 Done Sent Done false
25355 7091 Acked Read 10 Done Sent Done false
25356 7092 Acked Write 6 Done Sent Done false
25357 7093 Acked Write 8 Done Sent Done false
25358 7094 Acked Write 5 Done Sent Done false
25359 7095 Acked Write 10 Done Sent Done false
25360 7096 Acked Write 4 Done Sent Done false
25361 7097 Acked Read 4 Done Sent Done false
25362 7098 Acked Write 3 Done Sent Done false
25363 7099 Acked Read 3 Done Sent Done false
25364 7100 Acked Flush 0 Done Sent Done false
25365 7101 Acked Write 1 Done Sent Done false
25366 7102 Acked Read 6 Done Sent Done false
25367 7103 Acked Read 9 Done Sent Done false
25368 7104 Acked Read 1 Done Sent Done false
25369 7105 Acked Write 7 Done Sent Done false
25370 7106 Acked Read 8 Done Sent Done false
25371 7107 Acked Write 5 Done Sent Done false
25372 7108 Acked Write 5 Done Sent Done false
25373 7109 Acked Read 2 Done Sent Done false
25374 7110 Acked Write 7 Done Sent Done false
25375 7111 Acked Write 2 Done Sent Done false
25376 7112 Acked Read 10 Done Sent Done false
25377 7113 Acked Read 9 Done Sent Done false
25378 7114 Acked Read 7 Done Sent Done false
25379 7115 Acked Write 8 Done Sent Done false
25380 7116 Acked Read 8 Done Sent Done false
25381 7117 Acked Write 1 Done Sent Done false
25382 7118 Acked Write 7 Done Sent Done false
25383 7119 Acked Read 1 Done Sent Done false
25384 7120 Acked Write 5 Done Sent Done false
25385 7121 Acked Write 10 Done Sent Done false
25386 7122 Acked Write 2 Done Sent Done false
25387 7123 Acked Read 4 Done Sent Done false
25388 7124 Acked Write 7 Done Sent Done false
25389 7125 Acked Write 10 Done Sent Done false
25390 7126 Acked Write 9 Done Sent Done false
25391 7127 Acked Write 6 Done Sent Done false
25392 7128 Acked Read 6 Done Sent Done false
25393 7129 Acked Read 7 Done Sent Done false
25394 7130 Acked Write 8 Done Sent Done false
25395 7131 Acked Read 8 Done Sent Done false
25396 7132 Acked Read 4 Done Sent Done false
25397 7133 Acked Write 10 Done Sent Done false
25398 7134 Acked Read 3 Done Sent Done false
25399 7135 Acked Read 3 Done Sent Done false
25400 7136 Acked Read 3 Done Sent Done false
25401 7137 Acked Write 6 Done Sent Done false
25402 7138 Acked Flush 0 Done Sent Done false
25403 7139 Acked Read 2 Done Sent Done false
25404 7140 Acked Write 1 Done Sent Done false
25405 7141 Acked Write 6 Done Sent Done false
25406 7142 Acked Read 8 Done Sent Done false
25407 7143 Acked Flush 0 Done Sent Done false
25408 7144 Acked Flush 0 Done Sent Done false
25409 7145 Acked Write 4 Done Sent Done false
25410 7146 Acked Read 7 Done Sent Done false
25411 7147 Acked Write 1 Done Sent Done false
25412 7148 Acked Read 2 Done Sent Done false
25413 7149 Acked Read 10 Done Sent Done false
25414 7150 Acked Read 2 Done Sent Done false
25415 7151 Acked Write 6 Done Sent Done false
25416 7152 Acked Flush 0 Done Sent Done false
25417 7153 Acked Read 4 Done Sent Done false
25418 7154 Acked Read 3 Done Sent Done false
25419 7155 Acked Write 1 Done Sent Done false
25420 7156 Acked Write 10 Done Sent Done false
25421 7157 Acked Write 6 Done Sent Done false
25422 7158 Acked Write 6 Done Sent Done false
25423 7159 Acked Read 4 Done Sent Done false
25424 7160 Acked Read 9 Done Sent Done false
25425 7161 Acked Write 6 Done Sent Done false
25426 7162 Acked Write 1 Done Sent Done false
25427 7163 Acked Write 2 Done Sent Done false
25428 7164 Acked Read 3 Done Sent Done false
25429 7165 Acked Write 7 Done Sent Done false
25430 7166 Acked Flush 0 Done Sent Done false
25431 7167 Acked Write 6 Done Sent Done false
25432 7168 Acked Write 10 Done Sent Done false
25433 7169 Acked Write 5 Done Sent Done false
25434 7170 Acked Read 7 Done Sent Done false
25435 7171 Acked Read 4 Done Sent Done false
25436 7172 Acked Read 2 Done Sent Done false
25437 7173 Acked Flush 0 Done Sent Done false
25438 7174 Acked Read 1 Done Sent Done false
25439 7175 Acked Read 7 Done Sent Done false
25440 7176 Acked Write 9 Done Sent Done false
25441 7177 Acked Write 5 Done Sent Done false
25442 7178 Acked Read 5 Done Sent Done false
25443 7179 Acked Write 1 Done Sent Done false
25444 7180 Acked Read 1 Done Sent Done false
25445 7181 Acked Write 6 Done Sent Done false
25446 7182 Acked Write 5 Done Sent Done false
25447 7183 Acked Write 8 Done Sent Done false
25448 7184 Acked Read 4 Done Sent Done false
25449 7185 Acked Write 5 Done Sent Done false
25450 7186 Acked Write 5 Done Sent Done false
25451 7187 Acked Read 8 Done Sent Done false
25452 7188 Acked Read 9 Done Sent Done false
25453 7189 Acked Read 8 Done Sent Done false
25454 7190 Acked Write 8 Done Sent Done false
25455 7191 Acked Read 4 Done Sent Done false
25456 7192 Acked Flush 0 Done Sent Done false
25457 7193 Acked Write 7 Done Sent Done false
25458 7194 Acked Read 3 Done Sent Done false
25459 7195 Acked Write 6 Done Sent Done false
25460 7196 Acked Write 5 Done Sent Done false
25461 7197 Acked Read 3 Done Sent Done false
25462 7198 Acked Read 10 Done Sent Done false
25463 7199 Acked Write 2 Done Sent Done false
25464 7200 Acked Read 6 Done Sent Done false
25465 7201 Acked Read 3 Done Sent Done false
25466 7202 Acked Write 4 Done Sent Done false
25467 7203 Acked Read 4 Done Sent Done false
25468 7204 Acked Write 8 Done Sent Done false
25469 7205 Acked Read 5 Done Sent Done false
25470 7206 Acked Read 10 Done Sent Done false
25471 7207 Acked Read 2 Done Sent Done false
25472 7208 Acked Flush 0 Done Sent Done false
25473 7209 Acked Read 4 Done Sent Done false
25474 7210 Acked Read 2 Done Sent Done false
25475 7211 Acked Read 5 Done Sent Done false
25476 7212 Acked Write 5 Done Sent Done false
25477 7213 Acked Read 4 Done Sent Done false
25478 7214 Acked Flush 0 Done Sent Done false
25479 7215 Acked Read 3 Done Sent Done false
25480 7216 Acked Write 2 Done Sent Done false
25481 7217 Acked Write 7 Done Sent Done false
25482 7218 Acked Write 4 Done Sent Done false
25483 7219 Acked Write 6 Done Sent Done false
25484 7220 Acked Flush 0 Done Sent Done false
25485 7221 Acked Read 8 Done Sent Done false
25486 7222 Acked Read 5 Done Sent Done false
25487 7223 Acked Read 2 Done Sent Done false
25488 7224 Acked Read 6 Done Sent Done false
25489 7225 Acked Write 8 Done Sent Done false
25490 7226 Acked Read 6 Done Sent Done false
25491 7227 Acked Flush 0 Done Sent Done false
25492 7228 Acked Read 2 Done Sent Done false
25493 7229 Acked Read 10 Done Sent Done false
25494 7230 Acked Read 5 Done Sent Done false
25495 7231 Acked Write 2 Done Sent Done false
25496 7232 Acked Write 1 Done Sent Done false
25497 7233 Acked Read 3 Done Sent Done false
25498 7234 Acked Flush 0 Done Sent Done false
25499 7235 Acked Flush 0 Done Sent Done false
25500 7236 Acked Read 1 Done Sent Done false
25501 7237 Acked Write 3 Done Sent Done false
25502 7238 Acked Read 8 Done Sent Done false
25503 7239 Acked Read 8 Done Sent Done false
25504 7240 Acked Read 3 Done Sent Done false
25505 7241 Acked Read 8 Done Sent Done false
25506 7242 Acked Read 10 Done Sent Done false
25507 7243 Acked Write 2 Done Sent Done false
25508 7244 Acked Write 9 Done Sent Done false
25509 7245 Acked Write 4 Done Sent Done false
25510 7246 Acked Read 5 Done Sent Done false
25511 7247 Acked Read 5 Done Sent Done false
25512 7248 Acked Write 6 Done Sent Done false
25513 7249 Acked Write 10 Done Sent Done false
25514 7250 Acked Flush 0 Done Sent Done false
25515 7251 Acked Read 2 Done Sent Done false
25516 7252 Acked Write 10 Done Sent Done false
25517 7253 Acked Read 5 Done Sent Done false
25518 7254 Acked Write 9 Done Sent Done false
25519 7255 Acked Write 5 Done Sent Done false
25520 7256 Acked Write 4 Done Sent Done false
25521 7257 Acked Read 9 Done Sent Done false
25522 7258 Acked Read 7 Done Sent Done false
25523 7259 Acked Read 2 Done Sent Done false
25524 7260 Acked Read 7 Done Sent Done false
25525 7261 Acked Write 10 Done Sent Done false
25526 7262 Acked Flush 0 Done Sent Done false
25527 7263 Acked Write 6 Done Sent Done false
25528 7264 Acked Write 7 Done Sent Done false
25529 7265 Acked Read 6 Done Sent Done false
25530 7266 Acked Read 10 Done Sent Done false
25531 7267 Acked Write 7 Done Sent Done false
25532 7268 Acked Write 5 Done Sent Done false
25533 7269 Acked Write 4 Done Sent Done false
25534 7270 Acked Write 8 Done Sent Done false
25535 7271 Acked Read 4 Done Sent Done false
25536 7272 Acked Read 8 Done Sent Done false
25537 7273 Acked Read 6 Done Sent Done false
25538 7274 Acked Write 9 Done Sent Done false
25539 7275 Acked Read 2 Done Sent Done false
25540 7276 Acked Flush 0 Done Sent Done false
25541 7277 Acked Read 10 Done Sent Done false
25542 7278 Acked Write 8 Done Sent Done false
25543 7279 Acked Write 8 Done Sent Done false
25544 7280 Acked Flush 0 Done Sent Done false
25545 7281 Acked Write 10 Done Sent Done false
25546 7282 Acked Write 8 Done Sent Done false
25547 7283 Acked Read 2 Done Sent Done false
25548 7284 Acked Write 10 Done Sent Done false
25549 7285 Acked Read 4 Done Sent Done false
25550 7286 Acked Read 2 Done Sent Done false
25551 7287 Acked Flush 0 Done Sent Done false
25552 7288 Acked Write 4 Done Sent Done false
25553 7289 Acked Read 9 Done Sent Done false
25554 7290 Acked Read 8 Done Sent Done false
25555 7291 Acked Read 4 Done Sent Done false
25556 7292 Acked Read 5 Done Sent Done false
25557 7293 Acked Read 5 Done Sent Done false
25558 7294 Acked Read 10 Done Sent Done false
25559 7295 Acked Write 8 Done Sent Done false
25560 7296 Acked Read 1 Done Sent Done false
25561 7297 Acked Read 4 Done Sent Done false
25562 7298 Acked Read 4 Done Sent Done false
25563 7299 Acked Write 2 Done Sent Done false
25564 7300 Acked Write 2 Done Sent Done false
25565 7301 Acked Read 1 Done Sent Done false
25566 7302 Acked Read 3 Done Sent Done false
25567 7303 Acked Read 9 Done Sent Done false
25568 7304 Acked Read 2 Done Sent Done false
25569 7305 Acked Write 1 Done Sent Done false
25570 7306 Acked Write 7 Done Sent Done false
25571 7307 Acked Read 6 Done Sent Done false
25572 7308 Acked Write 9 Done Sent Done false
25573 7309 Acked Read 9 Done Sent Done false
25574 7310 Acked Write 5 Done Sent Done false
25575 7311 Acked Write 2 Done Sent Done false
25576 7312 Acked Write 10 Done Sent Done false
25577 7313 Acked Write 3 Done Sent Done false
25578 7314 Acked Read 5 Done Sent Done false
25579 7315 Acked Read 6 Done Sent Done false
25580 7316 Acked Flush 0 Done Sent Done false
25581 7317 Acked Read 4 Done Sent Done false
25582 7318 Acked Write 4 Done Sent Done false
25583 7319 Acked Write 3 Done Sent Done false
25584 7320 Acked Write 3 Done Sent Done false
25585 7321 Acked Read 9 Done Sent Done false
25586 7322 Acked Read 4 Done Sent Done false
25587 7323 Acked Write 6 Done Sent Done false
25588 7324 Acked Read 4 Done Sent Done false
25589 7325 Acked Read 4 Done Sent Done false
25590 7326 Acked Read 2 Done Sent Done false
25591 7327 Acked Flush 0 Done Sent Done false
25592 7328 Acked Read 6 Done Sent Done false
25593 7329 Acked Flush 0 Done Sent Done false
25594 7330 Acked Read 1 Done Sent Done false
25595 7331 Acked Flush 0 Done Sent Done false
25596 7332 Acked Write 8 Done Sent Done false
25597 7333 Acked Write 8 Done Sent Done false
25598 7334 Acked Write 4 Done Sent Done false
25599 7335 Acked Flush 0 Done Sent Done false
25600 7336 Acked Read 8 Done Sent Done false
25601 7337 Acked Read 3 Done Sent Done false
25602 STATES DS:0 DS:1 DS:2 TOTAL
25603 Sent 0 301 0 301
25604 Done 301 0 301 602
25605 Skipped 0 0 0 0
25606 Error 0 0 0 0
25607 Last Flush: Some(JobId(7335)) Some(JobId(7036)) Some(JobId(7335))
25608 Downstairs last five completed: 7036 7035 7034 7033 7032
25609 Upstairs last five completed: 7337 7336 7335 7334 7333
25610 CLIENT: Up:0 ds:301 act:2
256112025-02-28T06:06:38.137ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
256122025-02-28T06:06:38.902ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
256132025-02-28T06:06:39.904ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
256142025-02-28T06:06:40.906ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
25615 ----------------------------------------------------------------
25616 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25617 States: Active Offline Active
25618 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25619 7037 Acked Read 10 Done Sent Done false
25620 7038 Acked Flush 0 Done Sent Done false
25621 7039 Acked Read 2 Done Sent Done false
25622 7040 Acked Write 6 Done Sent Done false
25623 7041 Acked Read 7 Done Sent Done false
25624 7042 Acked Write 6 Done Sent Done false
25625 7043 Acked Read 4 Done Sent Done false
25626 7044 Acked Write 8 Done Sent Done false
25627 7045 Acked Read 4 Done Sent Done false
25628 7046 Acked Read 9 Done Sent Done false
25629 7047 Acked Read 2 Done Sent Done false
25630 7048 Acked Write 7 Done Sent Done false
25631 7049 Acked Read 8 Done Sent Done false
25632 7050 Acked Read 4 Done Sent Done false
25633 7051 Acked Read 5 Done Sent Done false
25634 7052 Acked Flush 0 Done Sent Done false
25635 7053 Acked Write 6 Done Sent Done false
25636 7054 Acked Flush 0 Done Sent Done false
25637 7055 Acked Read 7 Done Sent Done false
25638 7056 Acked Write 10 Done Sent Done false
25639 7057 Acked Read 7 Done Sent Done false
25640 7058 Acked Write 8 Done Sent Done false
25641 7059 Acked Write 9 Done Sent Done false
25642 7060 Acked Write 7 Done Sent Done false
25643 7061 Acked Read 10 Done Sent Done false
25644 7062 Acked Read 5 Done Sent Done false
25645 7063 Acked Read 6 Done Sent Done false
25646 7064 Acked Read 3 Done Sent Done false
25647 7065 Acked Flush 0 Done Sent Done false
25648 7066 Acked Read 2 Done Sent Done false
25649 7067 Acked Read 3 Done Sent Done false
25650 7068 Acked Read 2 Done Sent Done false
25651 7069 Acked Write 1 Done Sent Done false
25652 7070 Acked Read 2 Done Sent Done false
25653 7071 Acked Read 9 Done Sent Done false
25654 7072 Acked Write 8 Done Sent Done false
25655 7073 Acked Read 7 Done Sent Done false
25656 7074 Acked Read 10 Done Sent Done false
25657 7075 Acked Write 10 Done Sent Done false
25658 7076 Acked Read 5 Done Sent Done false
25659 7077 Acked Write 1 Done Sent Done false
25660 7078 Acked Write 6 Done Sent Done false
25661 7079 Acked Flush 0 Done Sent Done false
25662 7080 Acked Read 10 Done Sent Done false
25663 7081 Acked Read 6 Done Sent Done false
25664 7082 Acked Read 6 Done Sent Done false
25665 7083 Acked Read 9 Done Sent Done false
25666 7084 Acked Write 3 Done Sent Done false
25667 7085 Acked Read 7 Done Sent Done false
25668 7086 Acked Read 10 Done Sent Done false
25669 7087 Acked Read 10 Done Sent Done false
25670 7088 Acked Write 1 Done Sent Done false
25671 7089 Acked Read 5 Done Sent Done false
25672 7090 Acked Read 9 Done Sent Done false
25673 7091 Acked Read 10 Done Sent Done false
25674 7092 Acked Write 6 Done Sent Done false
25675 7093 Acked Write 8 Done Sent Done false
25676 7094 Acked Write 5 Done Sent Done false
25677 7095 Acked Write 10 Done Sent Done false
25678 7096 Acked Write 4 Done Sent Done false
25679 7097 Acked Read 4 Done Sent Done false
25680 7098 Acked Write 3 Done Sent Done false
25681 7099 Acked Read 3 Done Sent Done false
25682 7100 Acked Flush 0 Done Sent Done false
25683 7101 Acked Write 1 Done Sent Done false
25684 7102 Acked Read 6 Done Sent Done false
25685 7103 Acked Read 9 Done Sent Done false
25686 7104 Acked Read 1 Done Sent Done false
25687 7105 Acked Write 7 Done Sent Done false
25688 7106 Acked Read 8 Done Sent Done false
25689 7107 Acked Write 5 Done Sent Done false
25690 7108 Acked Write 5 Done Sent Done false
25691 7109 Acked Read 2 Done Sent Done false
25692 7110 Acked Write 7 Done Sent Done false
25693 7111 Acked Write 2 Done Sent Done false
25694 7112 Acked Read 10 Done Sent Done false
25695 7113 Acked Read 9 Done Sent Done false
25696 7114 Acked Read 7 Done Sent Done false
25697 7115 Acked Write 8 Done Sent Done false
25698 7116 Acked Read 8 Done Sent Done false
25699 7117 Acked Write 1 Done Sent Done false
25700 7118 Acked Write 7 Done Sent Done false
25701 7119 Acked Read 1 Done Sent Done false
25702 7120 Acked Write 5 Done Sent Done false
25703 7121 Acked Write 10 Done Sent Done false
25704 7122 Acked Write 2 Done Sent Done false
25705 7123 Acked Read 4 Done Sent Done false
25706 7124 Acked Write 7 Done Sent Done false
25707 7125 Acked Write 10 Done Sent Done false
25708 7126 Acked Write 9 Done Sent Done false
25709 7127 Acked Write 6 Done Sent Done false
25710 7128 Acked Read 6 Done Sent Done false
25711 7129 Acked Read 7 Done Sent Done false
25712 7130 Acked Write 8 Done Sent Done false
25713 7131 Acked Read 8 Done Sent Done false
25714 7132 Acked Read 4 Done Sent Done false
25715 7133 Acked Write 10 Done Sent Done false
25716 7134 Acked Read 3 Done Sent Done false
25717 7135 Acked Read 3 Done Sent Done false
25718 7136 Acked Read 3 Done Sent Done false
25719 7137 Acked Write 6 Done Sent Done false
25720 7138 Acked Flush 0 Done Sent Done false
25721 7139 Acked Read 2 Done Sent Done false
25722 7140 Acked Write 1 Done Sent Done false
25723 7141 Acked Write 6 Done Sent Done false
25724 7142 Acked Read 8 Done Sent Done false
25725 7143 Acked Flush 0 Done Sent Done false
25726 7144 Acked Flush 0 Done Sent Done false
25727 7145 Acked Write 4 Done Sent Done false
25728 7146 Acked Read 7 Done Sent Done false
25729 7147 Acked Write 1 Done Sent Done false
25730 7148 Acked Read 2 Done Sent Done false
25731 7149 Acked Read 10 Done Sent Done false
25732 7150 Acked Read 2 Done Sent Done false
25733 7151 Acked Write 6 Done Sent Done false
25734 7152 Acked Flush 0 Done Sent Done false
25735 7153 Acked Read 4 Done Sent Done false
25736 7154 Acked Read 3 Done Sent Done false
25737 7155 Acked Write 1 Done Sent Done false
25738 7156 Acked Write 10 Done Sent Done false
25739 7157 Acked Write 6 Done Sent Done false
25740 7158 Acked Write 6 Done Sent Done false
25741 7159 Acked Read 4 Done Sent Done false
25742 7160 Acked Read 9 Done Sent Done false
25743 7161 Acked Write 6 Done Sent Done false
25744 7162 Acked Write 1 Done Sent Done false
25745 7163 Acked Write 2 Done Sent Done false
25746 7164 Acked Read 3 Done Sent Done false
25747 7165 Acked Write 7 Done Sent Done false
25748 7166 Acked Flush 0 Done Sent Done false
25749 7167 Acked Write 6 Done Sent Done false
25750 7168 Acked Write 10 Done Sent Done false
25751 7169 Acked Write 5 Done Sent Done false
25752 7170 Acked Read 7 Done Sent Done false
25753 7171 Acked Read 4 Done Sent Done false
25754 7172 Acked Read 2 Done Sent Done false
25755 7173 Acked Flush 0 Done Sent Done false
25756 7174 Acked Read 1 Done Sent Done false
25757 7175 Acked Read 7 Done Sent Done false
25758 7176 Acked Write 9 Done Sent Done false
25759 7177 Acked Write 5 Done Sent Done false
25760 7178 Acked Read 5 Done Sent Done false
25761 7179 Acked Write 1 Done Sent Done false
25762 7180 Acked Read 1 Done Sent Done false
25763 7181 Acked Write 6 Done Sent Done false
25764 7182 Acked Write 5 Done Sent Done false
25765 7183 Acked Write 8 Done Sent Done false
25766 7184 Acked Read 4 Done Sent Done false
25767 7185 Acked Write 5 Done Sent Done false
25768 7186 Acked Write 5 Done Sent Done false
25769 7187 Acked Read 8 Done Sent Done false
25770 7188 Acked Read 9 Done Sent Done false
25771 7189 Acked Read 8 Done Sent Done false
25772 7190 Acked Write 8 Done Sent Done false
25773 7191 Acked Read 4 Done Sent Done false
25774 7192 Acked Flush 0 Done Sent Done false
25775 7193 Acked Write 7 Done Sent Done false
25776 7194 Acked Read 3 Done Sent Done false
25777 7195 Acked Write 6 Done Sent Done false
25778 7196 Acked Write 5 Done Sent Done false
25779 7197 Acked Read 3 Done Sent Done false
25780 7198 Acked Read 10 Done Sent Done false
25781 7199 Acked Write 2 Done Sent Done false
25782 7200 Acked Read 6 Done Sent Done false
25783 7201 Acked Read 3 Done Sent Done false
25784 7202 Acked Write 4 Done Sent Done false
25785 7203 Acked Read 4 Done Sent Done false
25786 7204 Acked Write 8 Done Sent Done false
25787 7205 Acked Read 5 Done Sent Done false
25788 7206 Acked Read 10 Done Sent Done false
25789 7207 Acked Read 2 Done Sent Done false
25790 7208 Acked Flush 0 Done Sent Done false
25791 7209 Acked Read 4 Done Sent Done false
25792 7210 Acked Read 2 Done Sent Done false
25793 7211 Acked Read 5 Done Sent Done false
25794 7212 Acked Write 5 Done Sent Done false
25795 7213 Acked Read 4 Done Sent Done false
25796 7214 Acked Flush 0 Done Sent Done false
25797 7215 Acked Read 3 Done Sent Done false
25798 7216 Acked Write 2 Done Sent Done false
25799 7217 Acked Write 7 Done Sent Done false
25800 7218 Acked Write 4 Done Sent Done false
25801 7219 Acked Write 6 Done Sent Done false
25802 7220 Acked Flush 0 Done Sent Done false
25803 7221 Acked Read 8 Done Sent Done false
25804 7222 Acked Read 5 Done Sent Done false
25805 7223 Acked Read 2 Done Sent Done false
25806 7224 Acked Read 6 Done Sent Done false
25807 7225 Acked Write 8 Done Sent Done false
25808 7226 Acked Read 6 Done Sent Done false
25809 7227 Acked Flush 0 Done Sent Done false
25810 7228 Acked Read 2 Done Sent Done false
25811 7229 Acked Read 10 Done Sent Done false
25812 7230 Acked Read 5 Done Sent Done false
25813 7231 Acked Write 2 Done Sent Done false
25814 7232 Acked Write 1 Done Sent Done false
25815 7233 Acked Read 3 Done Sent Done false
25816 7234 Acked Flush 0 Done Sent Done false
25817 7235 Acked Flush 0 Done Sent Done false
25818 7236 Acked Read 1 Done Sent Done false
25819 7237 Acked Write 3 Done Sent Done false
25820 7238 Acked Read 8 Done Sent Done false
25821 7239 Acked Read 8 Done Sent Done false
25822 7240 Acked Read 3 Done Sent Done false
25823 7241 Acked Read 8 Done Sent Done false
25824 7242 Acked Read 10 Done Sent Done false
25825 7243 Acked Write 2 Done Sent Done false
25826 7244 Acked Write 9 Done Sent Done false
25827 7245 Acked Write 4 Done Sent Done false
25828 7246 Acked Read 5 Done Sent Done false
25829 7247 Acked Read 5 Done Sent Done false
25830 7248 Acked Write 6 Done Sent Done false
25831 7249 Acked Write 10 Done Sent Done false
25832 7250 Acked Flush 0 Done Sent Done false
25833 7251 Acked Read 2 Done Sent Done false
25834 7252 Acked Write 10 Done Sent Done false
25835 7253 Acked Read 5 Done Sent Done false
25836 7254 Acked Write 9 Done Sent Done false
25837 7255 Acked Write 5 Done Sent Done false
25838 7256 Acked Write 4 Done Sent Done false
25839 7257 Acked Read 9 Done Sent Done false
25840 7258 Acked Read 7 Done Sent Done false
25841 7259 Acked Read 2 Done Sent Done false
25842 7260 Acked Read 7 Done Sent Done false
25843 7261 Acked Write 10 Done Sent Done false
25844 7262 Acked Flush 0 Done Sent Done false
25845 7263 Acked Write 6 Done Sent Done false
25846 7264 Acked Write 7 Done Sent Done false
25847 7265 Acked Read 6 Done Sent Done false
25848 7266 Acked Read 10 Done Sent Done false
25849 7267 Acked Write 7 Done Sent Done false
25850 7268 Acked Write 5 Done Sent Done false
25851 7269 Acked Write 4 Done Sent Done false
25852 7270 Acked Write 8 Done Sent Done false
25853 7271 Acked Read 4 Done Sent Done false
25854 7272 Acked Read 8 Done Sent Done false
25855 7273 Acked Read 6 Done Sent Done false
25856 7274 Acked Write 9 Done Sent Done false
25857 7275 Acked Read 2 Done Sent Done false
25858 7276 Acked Flush 0 Done Sent Done false
25859 7277 Acked Read 10 Done Sent Done false
25860 7278 Acked Write 8 Done Sent Done false
25861 7279 Acked Write 8 Done Sent Done false
25862 7280 Acked Flush 0 Done Sent Done false
25863 7281 Acked Write 10 Done Sent Done false
25864 7282 Acked Write 8 Done Sent Done false
25865 7283 Acked Read 2 Done Sent Done false
25866 7284 Acked Write 10 Done Sent Done false
25867 7285 Acked Read 4 Done Sent Done false
25868 7286 Acked Read 2 Done Sent Done false
25869 7287 Acked Flush 0 Done Sent Done false
25870 7288 Acked Write 4 Done Sent Done false
25871 7289 Acked Read 9 Done Sent Done false
25872 7290 Acked Read 8 Done Sent Done false
25873 7291 Acked Read 4 Done Sent Done false
25874 7292 Acked Read 5 Done Sent Done false
25875 7293 Acked Read 5 Done Sent Done false
25876 7294 Acked Read 10 Done Sent Done false
25877 7295 Acked Write 8 Done Sent Done false
25878 7296 Acked Read 1 Done Sent Done false
25879 7297 Acked Read 4 Done Sent Done false
25880 7298 Acked Read 4 Done Sent Done false
25881 7299 Acked Write 2 Done Sent Done false
25882 7300 Acked Write 2 Done Sent Done false
25883 7301 Acked Read 1 Done Sent Done false
25884 7302 Acked Read 3 Done Sent Done false
25885 7303 Acked Read 9 Done Sent Done false
25886 7304 Acked Read 2 Done Sent Done false
25887 7305 Acked Write 1 Done Sent Done false
25888 7306 Acked Write 7 Done Sent Done false
25889 7307 Acked Read 6 Done Sent Done false
25890 7308 Acked Write 9 Done Sent Done false
25891 7309 Acked Read 9 Done Sent Done false
25892 7310 Acked Write 5 Done Sent Done false
25893 7311 Acked Write 2 Done Sent Done false
25894 7312 Acked Write 10 Done Sent Done false
25895 7313 Acked Write 3 Done Sent Done false
25896 7314 Acked Read 5 Done Sent Done false
25897 7315 Acked Read 6 Done Sent Done false
25898 7316 Acked Flush 0 Done Sent Done false
25899 7317 Acked Read 4 Done Sent Done false
25900 7318 Acked Write 4 Done Sent Done false
25901 7319 Acked Write 3 Done Sent Done false
25902 7320 Acked Write 3 Done Sent Done false
25903 7321 Acked Read 9 Done Sent Done false
25904 7322 Acked Read 4 Done Sent Done false
25905 7323 Acked Write 6 Done Sent Done false
25906 7324 Acked Read 4 Done Sent Done false
25907 7325 Acked Read 4 Done Sent Done false
25908 7326 Acked Read 2 Done Sent Done false
25909 7327 Acked Flush 0 Done Sent Done false
25910 7328 Acked Read 6 Done Sent Done false
25911 7329 Acked Flush 0 Done Sent Done false
25912 7330 Acked Read 1 Done Sent Done false
25913 7331 Acked Flush 0 Done Sent Done false
25914 7332 Acked Write 8 Done Sent Done false
25915 7333 Acked Write 8 Done Sent Done false
25916 7334 Acked Write 4 Done Sent Done false
25917 7335 Acked Flush 0 Done Sent Done false
25918 7336 Acked Read 8 Done Sent Done false
25919 7337 Acked Read 3 Done Sent Done false
25920 STATES DS:0 DS:1 DS:2 TOTAL
25921 Sent 0 301 0 301
25922 Done 301 0 301 602
25923 Skipped 0 0 0 0
25924 Error 0 0 0 0
25925 Last Flush: Some(JobId(7335)) Some(JobId(7036)) Some(JobId(7335))
25926 Downstairs last five completed: 7036 7035 7034 7033 7032
25927 Upstairs last five completed: 7337 7336 7335 7334 7333
25928 CLIENT: Up:0 ds:301 act:2
259292025-02-28T06:06:41.908ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
259302025-02-28T06:06:43.139ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
259312025-02-28T06:06:43.912ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
259322025-02-28T06:06:44.881ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
259332025-02-28T06:06:44.881ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54743) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
259342025-02-28T06:06:44.882ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
259352025-02-28T06:06:44.883ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
259362025-02-28T06:06:44.883ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
259372025-02-28T06:06:44.883ZINFOcrucible: send last flush ID to this DS: Some(JobId(7036)) = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
259382025-02-28T06:06:44.883ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
259392025-02-28T06:06:44.883ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7036)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
259402025-02-28T06:06:44.883ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7036)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
25941 ----------------------------------------------------------------
25942 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:2
25943 States: Active Active Active
25944 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25945 7336 Acked Read 8 Done Done Done true
25946 7337 Acked Read 3 Done Done Done true
25947 STATES DS:0 DS:1 DS:2 TOTAL
25948 Sent 0 0 0 0
25949 Done 2 2 2 6
25950 Skipped 0 0 0 0
25951 Error 0 0 0 0
25952 Last Flush: Some(JobId(7335)) Some(JobId(7335)) Some(JobId(7335))
25953 Downstairs last five completed: 7335 7334 7333 7332 7331
25954 Upstairs last five completed: 7337 7336 7335 7334 7333
25955 CLIENT: Up:0 ds:2 act:3
25956 ----------------------------------------------------------------
25957 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25958 Downstairs last five completed: 7338 7337 7336 7335 7334
25959 Upstairs last five completed: 7338 7337 7336 7335 7334
25960 CLIENT: Up:0 ds:0 act:3
25961 Replay: All jobs finished
25962 Main task has work to do, go find it
25963 got dsc Stop(1)
25964 stop 1
25965 Feb 28 06:06:49.136 INFO request completed, latency_us: 495[8820] Got stop action so:false kr:false
25966 , response_code: 204, uri: /stop/cid/1, method: POST, req_id: 6dd990bf-b99c-4948-848e-753885f0867e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
25967 Feb 28 06:06:49.137 INFO request completed, latency_us: 158, response_code: 200, uri: /state/cid/1, method: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:49.137181493Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
25968 GET, req_id: 58f73262-0c46-4b79-b299-f2802a12dc64, remote_addr: 127.0.0.1:63908, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:49.137264533Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
25969 local_addr: 127.0.0.1:9998{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:06:49.137302584Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
25970 
259712025-02-28T06:06:49.137ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
25972 [8820] Exited with: ExitStatus(unix_wait_status(9))
25973 [8820][1] reports Exit
25974 Feb 28 06:06:53.139 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/1, method: GET, req_id: 38235a08-4405-4a0e-8913-e9310ffacbf1, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
25975 001/300 Flush
25976 002/300 Flush
25977 003/300 Flush
25978 004/300 Write block 2607 len 20480 data: 3 3 3 2 4
25979 005/300 Read block 4059 len 4096
25980 006/300 Read block 2440 len 20480
25981 007/300 Write block 2751 len 24576 data: 7 4 3 3 4 4
25982 008/300 Read block 7 len 36864
25983 009/300 Write block 2274 len 32768 data: 6 6 7 7 8 8 8 7
25984 010/300 Read block 2058 len 4096
25985 011/300 Read block 1949 len 20480
25986 012/300 Write block 4329 len 20480 data: 3 3 2 3 4
25987 013/300 Read block 140 len 28672
25988 014/300 Read block 2326 len 16384
25989 015/300 Read block 1811 len 40960
25990 016/300 Write block 1137 len 40960 data: 5 5 4 4 3 5 5 5 7 6
25991 017/300 Write block 4598 len 16384 data: 7 5 5 5
25992 018/300 Write block 1930 len 20480 data: 4 4 5 4 4
25993 019/300 Read block 3548 len 36864
25994 020/300 Read block 4689 len 24576
25995 021/300 Flush
25996 022/300 Write block 2065 len 16384 data: 5 4 4 5
25997 023/300 Write block 3424 len 16384 data: 4 3 3 3
25998 024/300 Read block 4125 len 4096
25999 025/300 Read block 285 len 32768
26000 026/300 Read block 4132 len 8192
26001 027/300 Write block 3323 len 20480 data: 6 6 6 5 6
26002 028/300 Write block 1218 len 28672 data: 3 4 4 4 5 5 5
26003 029/300 Read block 3828 len 16384
26004 030/300 Read block 2139 len 24576
26005 031/300 Read block 2697 len 40960
26006 032/300 Write block 2892 len 32768 data: 3 2 3 4 4 5 6 4
26007 033/300 Write block 4646 len 12288 data: 4 7 6
26008 034/300 Read block 1790 len 32768
26009 035/300 Read block 3597 len 36864
26010 036/300 Write block 4501 len 8192 data: 8 6
26011 037/300 Read block 2045 len 36864
26012 038/300 Read block 2321 len 20480
26013 039/300 Flush
26014 040/300 Read block 2178 len 4096
26015 041/300 Read block 4439 len 28672
26016 042/300 Read block 556 len 32768
26017 043/300 Flush
26018 044/300 Read block 1739 len 28672
26019 045/300 Read block 4026 len 28672
26020 046/300 Read block 4736 len 28672
26021 047/300 Write block 740 len 28672 data: 5 4 5 5 6 5 5
26022 048/300 Read block 1238 len 24576
26023 049/300 Flush
26024 050/300 Read block 2710 len 4096
26025 051/300 Write block 3213 len 36864 data: 5 5 6 6 4 4 5 6 6
26026 052/300 Read block 2021 len 24576
26027 053/300 Write block 4729 len 8192 data: 7 8
26028 054/300 Write block 2508 len 16384 data: 3 4 4 6
26029 055/300 Write block 2994 len 40960 data: 5 4 5 5 5 5 5 6 4 4
26030 056/300 Read block 1924 len 4096
26031 057/300 Write block 1497 len 8192 data: 2 2
26032 058/300 Read block 4616 len 24576
26033 059/300 Read block 2399 len 24576
26034 060/300 Read block 931 len 16384
26035 061/300 Read block 1478 len 40960
26036 062/300 Read block 2949 len 24576
26037 063/300 Flush
26038 064/300 Read block 4383 len 36864
26039 065/300 Read block 51 len 32768
26040 066/300 Write block 3462 len 16384 data: 3 3 3 5
26041 067/300 Write block 3943 len 12288 data: 5 6 4
26042 068/300 Read block 157 len 16384
26043 069/300 Read block 42 len 12288
26044 070/300 Read block 4256 len 8192
26045 071/300 Flush
26046 072/300 Read block 3636 len 16384
26047 073/300 Write block 4476 len 4096 data: 5
26048 074/300 Read block 3619 len 24576
26049 075/300 Write block 440 len 20480 data: 9 9 10 10 11
26050 076/300 Write block 3865 len 16384 data: 6 6 7 6
26051 077/300 Read block 4732 len 40960
26052 078/300 Write block 53 len 24576 data: 5 5 5 5 5 5
26053 079/300 Write block 668 len 12288 data: 6 6 7
26054 080/300 Flush
26055 081/300 Read block 3189 len 40960
26056 082/300 Write block 4419 len 4096 data: 4
26057 083/300 Write block 2111 len 36864 data: 5 4 4 5 4 5 6 6 5
26058 084/300 Read block 3404 len 4096
26059 085/300 Read block 1051 len 8192
26060 086/300 Write block 4192 len 28672 data: 5 5 4 4 4 5 5
26061 087/300 Read block 2696 len 28672
26062 088/300 Read block 4307 len 4096
26063 089/300 Flush
26064 090/300 Flush
26065 091/300 Read block 2700 len 12288
26066 092/300 Write block 2948 len 28672 data: 4 4 5 5 5 5 5
26067 093/300 Read block 3534 len 36864
26068 094/300 Write block 2120 len 28672 data: 5 6 6 5 6 5 5
26069 095/300 Read block 3900 len 24576
26070 096/300 Read block 1880 len 8192
26071 097/300 Write block 1424 len 20480 data: 5 6 5 5 3
26072 098/300 Read block 4859 len 24576
26073 099/300 Read block 3996 len 20480
26074 100/300 Read block 2924 len 24576
26075 101/300 Flush
26076 102/300 Flush
26077 103/300 Read block 3507 len 36864
26078 104/300 Write block 1934 len 12288 data: 5 3 4
26079 105/300 Read block 3445 len 16384
26080 106/300 Read block 4961 len 32768
26081 107/300 Read block 3353 len 40960
26082 108/300 Read block 4552 len 8192
26083 109/300 Write block 4086 len 8192 data: 4 4
26084 110/300 Read block 3230 len 32768
26085 111/300 Read block 4287 len 40960
26086 112/300 Write block 743 len 32768 data: 6 7 6 6 5 6 7 7
26087 113/300 Read block 3982 len 20480
26088 114/300 Flush
26089 115/300 Read block 541 len 16384
26090 116/300 Write block 4693 len 32768 data: 4 4 5 6 6 5 7 5
26091 117/300 Read block 3059 len 36864
26092 118/300 Flush
26093 119/300 Read block 4574 len 12288
26094 120/300 Read block 3381 len 32768
26095 121/300 Read block 92 len 32768
26096 122/300 Read block 1165 len 32768
26097 123/300 Read block 1519 len 8192
26098 124/300 Read block 4081 len 36864
26099 125/300 Write block 1055 len 8192 data: 7 6
26100 126/300 Write block 4949 len 40960 data: 2 4 6 5 6 7 6 6 7 8
26101 127/300 Read block 683 len 20480
26102 128/300 Read block 3941 len 4096
26103 129/300 Flush
26104 130/300 Write block 2452 len 16384 data: 6 5 6 8
26105 131/300 Write block 1162 len 28672 data: 5 5 7 6 6 6 5
26106 132/300 Write block 3383 len 28672 data: 3 3 3 2 2 2 3
26107 133/300 Read block 2934 len 20480
26108 134/300 Read block 481 len 16384
26109 135/300 Write block 2080 len 28672 data: 2 2 2 4 4 6 8
26110 136/300 Read block 737 len 40960
26111 137/300 Write block 3690 len 40960 data: 6 6 7 5 3 2 2 2 2 3
26112 138/300 Write block 2584 len 20480 data: 5 5 4 4 3
26113 139/300 Flush
26114 140/300 Write block 1758 len 24576 data: 5 5 5 5 5 5
26115 141/300 Write block 768 len 24576 data: 3 5 5 4 5 4
26116 142/300 Read block 1154 len 8192
26117 143/300 Read block 4519 len 28672
26118 144/300 Read block 441 len 36864
26119 145/300 Write block 4593 len 24576 data: 6 6 8 9 8 8
26120 146/300 Read block 2942 len 40960
26121 147/300 Read block 4833 len 28672
26122 148/300 Read block 3216 len 12288
26123 149/300 Read block 2480 len 36864
26124 150/300 Read block 3381 len 20480
26125 151/300 Read block 2569 len 24576
26126 152/300 Read block 1992 len 40960
26127 153/300 Read block 3607 len 16384
26128 154/300 Read block 4919 len 20480
26129 155/300 Read block 428 len 28672
26130 156/300 Read block 392 len 12288
26131 157/300 Read block 4157 len 12288
26132 158/300 Read block 1090 len 40960
26133 159/300 Read block 3039 len 8192
26134 160/300 Flush
26135 161/300 Write block 273 len 24576 data: 5 6 5 5 7 8
26136 162/300 Read block 3172 len 28672
26137 163/300 Read block 3826 len 20480
26138 164/300 Read block 4182 len 24576
26139 165/300 Read block 981 len 28672
26140 166/300 Write block 586 len 20480 data: 5 3 4 3 4
26141 167/300 Read block 466 len 28672
26142 168/300 Read block 4899 len 28672
26143 169/300 Read block 4683 len 4096
26144 170/300 Read block 3692 len 20480
26145 171/300 Read block 1860 len 24576
26146 172/300 Write block 424 len 4096 data: 6
26147 173/300 Read block 4521 len 36864
26148 174/300 Read block 872 len 20480
26149 175/300 Write block 4534 len 12288 data: 4 5 5
26150 176/300 Write block 4794 len 20480 data: 4 4 4 4 4
26151 177/300 Write block 4977 len 32768 data: 4 6 5 4 4 6 5 6
26152 178/300 Flush
26153 179/300 Read block 1549 len 32768
26154 180/300 Read block 955 len 16384
26155 181/300 Flush
26156 182/300 Write block 3931 len 12288 data: 3 3 5
26157 183/300 Read block 4411 len 8192
26158 184/300 Read block 2393 len 8192
26159 185/300 Read block 331 len 16384
26160 186/300 Read block 890 len 20480
26161 187/300 Read block 381 len 40960
26162 188/300 Read block 981 len 4096
26163 189/300 Read block 2771 len 4096
26164 190/300 Read block 4 len 36864
26165 191/300 Read block 2644 len 16384
26166 192/300 Flush
26167 193/300 Read block 2201 len 24576
26168 194/300 Write block 3766 len 40960 data: 4 4 5 5 4 4 4 5 5 3
26169 195/300 Write block 1170 len 40960 data: 6 6 6 9 10 8 8 9 10 10
26170 196/300 Write block 170 len 36864 data: 2 2 3 3 4 4 4 4 4
26171 197/300 Write block 4231 len 16384 data: 9 5 5 4
26172 198/300 Write block 1470 len 8192 data: 5 6
26173 199/300 Read block 1048 len 20480
26174 200/300 Read block 2800 len 8192
26175 201/300 Write block 1869 len 8192 data: 6 4
26176 202/300 Write block 1287 len 40960 data: 4 4 5 5 6 5 5 4 3 3
26177 203/300 Read block 2146 len 12288
26178 204/300 Read block 2801 len 40960
26179 205/300 Read block 4442 len 16384
26180 206/300 Write block 1490 len 12288 data: 8 8 8
26181 207/300 Read block 546 len 16384
26182 208/300 Write block 4379 len 12288 data: 8 8 7
26183 209/300 Flush
26184 210/300 Read block 602 len 40960
26185 211/300 Read block 1109 len 16384
26186 212/300 Read block 1932 len 36864
26187 213/300 Read block 3952 len 20480
26188 214/300 Write block 4925 len 36864 data: 6 7 6 4 2 3 3 3 3
26189 215/300 Write block 2371 len 24576 data: 4 3 5 5 6 7
26190 216/300 Flush
26191 217/300 Write block 2441 len 20480 data: 5 7 6 6 6
26192 218/300 Read block 2256 len 36864
26193 219/300 Read block 2188 len 40960
26194 220/300 Read block 862 len 28672
26195 221/300 Write block 412 len 4096 data: 4
26196 222/300 Flush
26197 223/300 Write block 3249 len 12288 data: 7 7 6
26198 224/300 Write block 3778 len 36864 data: 4 4 5 5 6 6 4 3 3
26199 225/300 Write block 812 len 16384 data: 9 11 8 6
26200 226/300 Write block 2702 len 40960 data: 5 5 5 6 6 5 5 5 4 4
26201 227/300 Write block 4512 len 36864 data: 5 6 7 6 3 4 4 4 6
26202 228/300 Write block 2128 len 40960 data: 5 3 4 5 6 6 6 5 4 6
26203 229/300 Read block 4221 len 32768
26204 230/300 Write block 4215 len 8192 data: 5 5
26205 231/300 Write block 1644 len 32768 data: 3 3 3 4 4 4 4 4
26206 232/300 Read block 1288 len 32768
26207 233/300 Write block 4449 len 12288 data: 2 2 2
26208 234/300 Read block 4677 len 28672
26209 235/300 Write block 1529 len 24576 data: 5 4 3 3 3 5
26210 236/300 Read block 2092 len 16384
26211 237/300 Read block 199 len 8192
26212 238/300 Read block 4570 len 32768
26213 239/300 Write block 2601 len 32768 data: 6 5 6 6 5 4 4 4
26214 240/300 Read block 4149 len 32768
26215 241/300 Write block 213 len 32768 data: 4 6 7 5 6 7 9 6
26216 242/300 Write block 4312 len 28672 data: 5 4 4 3 3 3 4
26217 243/300 Read block 4943 len 28672
26218 244/300 Write block 4579 len 20480 data: 2 2 2 2 2
26219 245/300 Write block 1752 len 24576 data: 4 4 5 5 4 4
26220 246/300 Read block 3865 len 36864
26221 247/300 Write block 524 len 8192 data: 8 6
26222 248/300 Write block 2613 len 32768 data: 3 2 3 3 3 3 3 3
26223 249/300 Read block 1328 len 20480
26224 250/300 Read block 84 len 40960
26225 251/300 Write block 3419 len 20480 data: 4 3 3 3 3
26226 252/300 Write block 1868 len 24576 data: 4 7 5 4 5 6
26227 253/300 Write block 73 len 20480 data: 4 4 7 6 6
26228 254/300 Write block 3528 len 24576 data: 8 7 5 5 5 5
26229 255/300 Write block 912 len 36864 data: 5 5 4 5 3 3 4 4 4
26230 256/300 Read block 3526 len 20480
26231 257/300 Write block 387 len 36864 data: 3 2 2 2 2 3 2 3 3
26232 258/300 Write block 1308 len 24576 data: 5 6 6 7 7 7
26233 259/300 Read block 4679 len 28672
26234 260/300 Write block 4863 len 8192 data: 4 5
26235 261/300 Read block 841 len 8192
26236 262/300 Read block 1985 len 32768
26237 263/300 Write block 4101 len 24576 data: 8 7 7 6 3 4
26238 264/300 Read block 1769 len 28672
26239 265/300 Write block 582 len 28672 data: 7 7 5 5 6 4 5
26240 266/300 Write block 2607 len 24576 data: 5 5 4 3 5 3
26241 267/300 Read block 4916 len 20480
26242 268/300 Read block 4338 len 12288
26243 269/300 Read block 4287 len 32768
26244 270/300 Read block 3547 len 40960
26245 271/300 Read block 54 len 40960
26246 272/300 Flush
26247 273/300 Write block 406 len 16384 data: 4 4 5 4
26248 274/300 Write block 4336 len 36864 data: 6 6 5 6 5 5 8 7 7
26249 275/300 Flush
26250 276/300 Write block 1139 len 20480 data: 5 5 4 6 6
26251 277/300 Write block 2772 len 12288 data: 3 3 3
26252 278/300 Read block 4609 len 8192
26253 279/300 Write block 1662 len 36864 data: 5 6 6 8 8 6 7 7 6
26254 280/300 Read block 1342 len 8192
26255 281/300 Read block 3349 len 20480
26256 282/300 Write block 4684 len 16384 data: 5 4 4 5
26257 283/300 Write block 3743 len 12288 data: 9 9 9
26258 284/300 Write block 4510 len 4096 data: 6
26259 285/300 Read block 3098 len 28672
26260 286/300 Write block 4784 len 4096 data: 5
26261 287/300 Write block 2252 len 20480 data: 2 3 4 3 4
26262 288/300 Read block 2913 len 24576
26263 289/300 Flush
26264 290/300 Write block 4646 len 36864 data: 5 8 7 7 6 6 8 8 8
26265 291/300 Flush
26266 292/300 Read block 1122 len 36864
26267 293/300 Write block 4554 len 40960 data: 3 3 2 3 3 3 3 3 3 5
26268 294/300 Read block 4175 len 8192
26269 295/300 Write block 3816 len 36864 data: 3 3 3 3 2 2 3 4 3
26270 296/300 Read block 1254 len 12288
26271 297/300 Flush
26272 298/300 Write block 4752 len 12288 data: 3 3 3
26273 299/300 Write block 3287 len 20480 data: 4 4 4 4 3
26274 300/300 Read block 878 len 20480
26275 301/300 Read block 2821 len 36864
26276 Main task has work to do, go find it
26277 got dsc Start(1)
26278 start 1
26279 Feb 28 06:06:53.390 INFO request completed, latency_us: 191, response_code: 204, uri: /start/cid/1, method: POST, req_id: 377e2997-78d2-4fff-a5f6-7fe8f7c0f798, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
26280 Replay: started 1, returned:Ok(())
26281 ----------------------------------------------------------------
26282 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26283 States: Active Offline Active
26284 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26285 7339 Acked Flush 0 Done Sent Done false
26286 7340 Acked Flush 0 Done Sent Done false
26287 7341 Acked Flush 0 Done Sent Done false
26288 7342 Acked Write 5 Done Sent Done false
26289 7343 Acked Read 1 Done Sent Done false
26290 7344 Acked Read 5 Done Sent Done false
26291 7345 Acked Write 6 Done Sent Done false
26292 7346 Acked Read 9 Done Sent Done false
26293 7347 Acked Write 8 Done Sent Done false
26294 7348 Acked Read 1 Done Sent Done false
26295 7349 Acked Read 5 Done Sent Done false
26296 7350 Acked Write 5 Done Sent Done false
26297 7351 Acked Read 7 Done Sent Done false
26298 7352 Acked Read 4 Done Sent Done false
26299 7353 Acked Read 10 Done Sent Done false
26300 7354 Acked Write 10 Done Sent Done false
26301 7355 Acked Write 4 Done Sent Done false
26302 7356 Acked Write 5 Done Sent Done false
26303 7357 Acked Read 9 Done Sent Done false
26304 7358 Acked Read 6 Done Sent Done false
26305 7359 Acked Flush 0 Done Sent Done false
26306 7360 Acked Write 4 Done Sent Done false
26307 7361 Acked Write 4 Done Sent Done false
26308 7362 Acked Read 1 Done Sent Done false
26309 7363 Acked Read 8 Done Sent Done false
26310 7364 Acked Read 2 Done Sent Done false
26311 7365 Acked Write 5 Done Sent Done false
26312 7366 Acked Write 7 Done Sent Done false
26313 7367 Acked Read 4 Done Sent Done false
26314 7368 Acked Read 6 Done Sent Done false
26315 7369 Acked Read 10 Done Sent Done false
26316 7370 Acked Write 8 Done Sent Done false
26317 7371 Acked Write 3 Done Sent Done false
26318 7372 Acked Read 8 Done Sent Done false
26319 7373 Acked Read 9 Done Sent Done false
26320 7374 Acked Write 2 Done Sent Done false
26321 7375 Acked Read 9 Done Sent Done false
26322 7376 Acked Read 5 Done Sent Done false
26323 7377 Acked Flush 0 Done Sent Done false
26324 7378 Acked Read 1 Done Sent Done false
26325 7379 Acked Read 7 Done Sent Done false
26326 7380 Acked Read 8 Done Sent Done false
26327 7381 Acked Flush 0 Done Sent Done false
26328 7382 Acked Read 7 Done Sent Done false
26329 7383 Acked Read 7 Done Sent Done false
26330 7384 Acked Read 7 Done Sent Done false
26331 7385 Acked Write 7 Done Sent Done false
26332 7386 Acked Read 6 Done Sent Done false
26333 7387 Acked Flush 0 Done Sent Done false
26334 7388 Acked Read 1 Done Sent Done false
26335 7389 Acked Write 9 Done Sent Done false
26336 7390 Acked Read 6 Done Sent Done false
26337 7391 Acked Write 2 Done Sent Done false
26338 7392 Acked Write 4 Done Sent Done false
26339 7393 Acked Write 10 Done Sent Done false
26340 7394 Acked Read 1 Done Sent Done false
26341 7395 Acked Write 2 Done Sent Done false
26342 7396 Acked Read 6 Done Sent Done false
26343 7397 Acked Read 6 Done Sent Done false
26344 7398 Acked Read 4 Done Sent Done false
26345 7399 Acked Read 10 Done Sent Done false
26346 7400 Acked Read 6 Done Sent Done false
26347 7401 Acked Flush 0 Done Sent Done false
26348 7402 Acked Read 9 Done Sent Done false
26349 7403 Acked Read 8 Done Sent Done false
26350 7404 Acked Write 4 Done Sent Done false
26351 7405 Acked Write 3 Done Sent Done false
26352 7406 Acked Read 4 Done Sent Done false
26353 7407 Acked Read 3 Done Sent Done false
26354 7408 Acked Read 2 Done Sent Done false
26355 7409 Acked Flush 0 Done Sent Done false
26356 7410 Acked Read 4 Done Sent Done false
26357 7411 Acked Write 1 Done Sent Done false
26358 7412 Acked Read 6 Done Sent Done false
26359 7413 Acked Write 5 Done Sent Done false
26360 7414 Acked Write 4 Done Sent Done false
26361 7415 Acked Read 10 Done Sent Done false
26362 7416 Acked Write 6 Done Sent Done false
26363 7417 Acked Write 3 Done Sent Done false
26364 7418 Acked Flush 0 Done Sent Done false
26365 7419 Acked Read 10 Done Sent Done false
26366 7420 Acked Write 1 Done Sent Done false
26367 7421 Acked Write 9 Done Sent Done false
26368 7422 Acked Read 1 Done Sent Done false
26369 7423 Acked Read 2 Done Sent Done false
26370 7424 Acked Write 7 Done Sent Done false
26371 7425 Acked Read 7 Done Sent Done false
26372 7426 Acked Read 1 Done Sent Done false
26373 7427 Acked Flush 0 Done Sent Done false
26374 7428 Acked Flush 0 Done Sent Done false
26375 7429 Acked Read 3 Done Sent Done false
26376 7430 Acked Write 7 Done Sent Done false
26377 7431 Acked Read 9 Done Sent Done false
26378 7432 Acked Write 7 Done Sent Done false
26379 7433 Acked Read 6 Done Sent Done false
26380 7434 Acked Read 2 Done Sent Done false
26381 7435 Acked Write 5 Done Sent Done false
26382 7436 Acked Read 6 Done Sent Done false
26383 7437 Acked Read 5 Done Sent Done false
26384 7438 Acked Read 6 Done Sent Done false
26385 7439 Acked Flush 0 Done Sent Done false
26386 7440 Acked Flush 0 Done Sent Done false
26387 7441 Acked Read 9 Done Sent Done false
26388 7442 Acked Write 3 Done Sent Done false
26389 7443 Acked Read 4 Done Sent Done false
26390 7444 Acked Read 8 Done Sent Done false
26391 7445 Acked Read 10 Done Sent Done false
26392 7446 Acked Read 2 Done Sent Done false
26393 7447 Acked Write 2 Done Sent Done false
26394 7448 Acked Read 8 Done Sent Done false
26395 7449 Acked Read 10 Done Sent Done false
26396 7450 Acked Write 8 Done Sent Done false
26397 7451 Acked Read 5 Done Sent Done false
26398 7452 Acked Flush 0 Done Sent Done false
26399 7453 Acked Read 4 Done Sent Done false
26400 7454 Acked Write 8 Done Sent Done false
26401 7455 Acked Read 9 Done Sent Done false
26402 7456 Acked Flush 0 Done Sent Done false
26403 7457 Acked Read 3 Done Sent Done false
26404 7458 Acked Read 8 Done Sent Done false
26405 7459 Acked Read 8 Done Sent Done false
26406 7460 Acked Read 8 Done Sent Done false
26407 7461 Acked Read 2 Done Sent Done false
26408 7462 Acked Read 9 Done Sent Done false
26409 7463 Acked Write 2 Done Sent Done false
26410 7464 Acked Write 10 Done Sent Done false
26411 7465 Acked Read 5 Done Sent Done false
26412 7466 Acked Read 1 Done Sent Done false
26413 7467 Acked Flush 0 Done Sent Done false
26414 7468 Acked Write 4 Done Sent Done false
26415 7469 Acked Write 7 Done Sent Done false
26416 7470 Acked Write 7 Done Sent Done false
26417 7471 Acked Read 5 Done Sent Done false
26418 7472 Acked Read 4 Done Sent Done false
26419 7473 Acked Write 7 Done Sent Done false
26420 7474 Acked Read 10 Done Sent Done false
26421 7475 Acked Write 10 Done Sent Done false
26422 7476 Acked Write 5 Done Sent Done false
26423 7477 Acked Flush 0 Done Sent Done false
26424 7478 Acked Write 6 Done Sent Done false
26425 7479 Acked Write 6 Done Sent Done false
26426 7480 Acked Read 2 Done Sent Done false
26427 7481 Acked Read 7 Done Sent Done false
26428 7482 Acked Read 9 Done Sent Done false
26429 7483 Acked Write 6 Done Sent Done false
26430 7484 Acked Read 10 Done Sent Done false
26431 7485 Acked Read 7 Done Sent Done false
26432 7486 Acked Read 3 Done Sent Done false
26433 7487 Acked Read 9 Done Sent Done false
26434 7488 Acked Read 5 Done Sent Done false
26435 7489 Acked Read 6 Done Sent Done false
26436 7490 Acked Read 10 Done Sent Done false
26437 7491 Acked Read 4 Done Sent Done false
26438 7492 Acked Read 5 Done Sent Done false
26439 7493 Acked Read 7 Done Sent Done false
26440 7494 Acked Read 3 Done Sent Done false
26441 7495 Acked Read 3 Done Sent Done false
26442 7496 Acked Read 10 Done Sent Done false
26443 7497 Acked Read 2 Done Sent Done false
26444 7498 Acked Flush 0 Done Sent Done false
26445 7499 Acked Write 6 Done Sent Done false
26446 7500 Acked Read 7 Done Sent Done false
26447 7501 Acked Read 5 Done Sent Done false
26448 7502 Acked Read 6 Done Sent Done false
26449 7503 Acked Read 7 Done Sent Done false
26450 7504 Acked Write 5 Done Sent Done false
26451 7505 Acked Read 7 Done Sent Done false
26452 7506 Acked Read 7 Done Sent Done false
26453 7507 Acked Read 1 Done Sent Done false
26454 7508 Acked Read 5 Done Sent Done false
26455 7509 Acked Read 6 Done Sent Done false
26456 7510 Acked Write 1 Done Sent Done false
26457 7511 Acked Read 9 Done Sent Done false
26458 7512 Acked Read 5 Done Sent Done false
26459 7513 Acked Write 3 Done Sent Done false
26460 7514 Acked Write 5 Done Sent Done false
26461 7515 Acked Write 8 Done Sent Done false
26462 7516 Acked Flush 0 Done Sent Done false
26463 7517 Acked Read 8 Done Sent Done false
26464 7518 Acked Read 4 Done Sent Done false
26465 7519 Acked Flush 0 Done Sent Done false
26466 7520 Acked Write 3 Done Sent Done false
26467 7521 Acked Read 2 Done Sent Done false
26468 7522 Acked Read 2 Done Sent Done false
26469 7523 Acked Read 4 Done Sent Done false
26470 7524 Acked Read 5 Done Sent Done false
26471 7525 Acked Read 10 Done Sent Done false
26472 7526 Acked Read 1 Done Sent Done false
26473 7527 Acked Read 1 Done Sent Done false
26474 7528 Acked Read 9 Done Sent Done false
26475 7529 Acked Read 4 Done Sent Done false
26476 7530 Acked Flush 0 Done Sent Done false
26477 7531 Acked Read 6 Done Sent Done false
26478 7532 Acked Write 10 Done Sent Done false
26479 7533 Acked Write 10 Done Sent Done false
26480 7534 Acked Write 9 Done Sent Done false
26481 7535 Acked Write 4 Done Sent Done false
26482 7536 Acked Write 2 Done Sent Done false
26483 7537 Acked Read 5 Done Sent Done false
26484 7538 Acked Read 2 Done Sent Done false
26485 7539 Acked Write 2 Done Sent Done false
26486 7540 Acked Write 10 Done Sent Done false
26487 7541 Acked Read 3 Done Sent Done false
26488 7542 Acked Read 10 Done Sent Done false
26489 7543 Acked Read 4 Done Sent Done false
26490 7544 Acked Write 3 Done Sent Done false
26491 7545 Acked Read 4 Done Sent Done false
26492 7546 Acked Write 3 Done Sent Done false
26493 7547 Acked Flush 0 Done Sent Done false
26494 7548 Acked Read 10 Done Sent Done false
26495 7549 Acked Read 4 Done Sent Done false
26496 7550 Acked Read 9 Done Sent Done false
26497 7551 Acked Read 5 Done Sent Done false
26498 7552 Acked Write 9 Done Sent Done false
26499 7553 Acked Write 6 Done Sent Done false
26500 7554 Acked Flush 0 Done Sent Done false
26501 7555 Acked Write 5 Done Sent Done false
26502 7556 Acked Read 9 Done Sent Done false
26503 7557 Acked Read 10 Done Sent Done false
26504 7558 Acked Read 7 Done Sent Done false
26505 7559 Acked Write 1 Done Sent Done false
26506 7560 Acked Flush 0 Done Sent Done false
26507 7561 Acked Write 3 Done Sent Done false
26508 7562 Acked Write 9 Done Sent Done false
26509 7563 Acked Write 4 Done Sent Done false
26510 7564 Acked Write 10 Done Sent Done false
26511 7565 Acked Write 9 Done Sent Done false
26512 7566 Acked Write 10 Done Sent Done false
26513 7567 Acked Read 8 Done Sent Done false
26514 7568 Acked Write 2 Done Sent Done false
26515 7569 Acked Write 8 Done Sent Done false
26516 7570 Acked Read 8 Done Sent Done false
26517 7571 Acked Write 3 Done Sent Done false
26518 7572 Acked Read 7 Done Sent Done false
26519 7573 Acked Write 6 Done Sent Done false
26520 7574 Acked Read 4 Done Sent Done false
26521 7575 Acked Read 2 Done Sent Done false
26522 7576 Acked Read 8 Done Sent Done false
26523 7577 Acked Write 8 Done Sent Done false
26524 7578 Acked Read 8 Done Sent Done false
26525 7579 Acked Write 8 Done Sent Done false
26526 7580 Acked Write 7 Done Sent Done false
26527 7581 Acked Read 7 Done Sent Done false
26528 7582 Acked Write 5 Done Sent Done false
26529 7583 Acked Write 6 Done Sent Done false
26530 7584 Acked Read 9 Done Sent Done false
26531 7585 Acked Write 2 Done Sent Done false
26532 7586 Acked Write 8 Done Sent Done false
26533 7587 Acked Read 5 Done Sent Done false
26534 7588 Acked Read 10 Done Sent Done false
26535 7589 Acked Write 5 Done Sent Done false
26536 7590 Acked Write 6 Done Sent Done false
26537 7591 Acked Write 5 Done Sent Done false
26538 7592 Acked Write 6 Done Sent Done false
26539 7593 Acked Write 9 Done Sent Done false
26540 7594 Acked Read 5 Done Sent Done false
26541 7595 Acked Write 9 Done Sent Done false
26542 7596 Acked Write 6 Done Sent Done false
26543 7597 Acked Read 7 Done Sent Done false
26544 7598 Acked Write 2 Done Sent Done false
26545 7599 Acked Read 2 Done Sent Done false
26546 7600 Acked Read 8 Done Sent Done false
26547 7601 Acked Write 6 Done Sent Done false
26548 7602 Acked Read 7 Done Sent Done false
26549 7603 Acked Write 7 Done Sent Done false
26550 7604 Acked Write 6 Done Sent Done false
26551 7605 Acked Read 5 Done Sent Done false
26552 7606 Acked Read 3 Done Sent Done false
26553 7607 Acked Read 8 Done Sent Done false
26554 7608 Acked Read 10 Done Sent Done false
26555 7609 Acked Read 10 Done Sent Done false
26556 7610 Acked Flush 0 Done Sent Done false
26557 7611 Acked Write 4 Done Sent Done false
26558 7612 Acked Write 9 Done Sent Done false
26559 7613 Acked Flush 0 Done Sent Done false
26560 7614 Acked Write 5 Done Sent Done false
26561 7615 Acked Write 3 Done Sent Done false
26562 7616 Acked Read 2 Done Sent Done false
26563 7617 Acked Write 9 Done Sent Done false
26564 7618 Acked Read 2 Done Sent Done false
26565 7619 Acked Read 5 Done Sent Done false
26566 7620 Acked Write 4 Done Sent Done false
26567 7621 Acked Write 3 Done Sent Done false
26568 7622 Acked Write 1 Done Sent Done false
26569 7623 Acked Read 7 Done Sent Done false
26570 7624 Acked Write 1 Done Sent Done false
26571 7625 Acked Write 5 Done Sent Done false
26572 7626 Acked Read 6 Done Sent Done false
26573 7627 Acked Flush 0 Done Sent Done false
26574 7628 Acked Write 9 Done Sent Done false
26575 7629 Acked Flush 0 Done Sent Done false
26576 7630 Acked Read 9 Done Sent Done false
26577 7631 Acked Write 10 Done Sent Done false
26578 7632 Acked Read 2 Done Sent Done false
26579 7633 Acked Write 9 Done Sent Done false
26580 7634 Acked Read 3 Done Sent Done false
26581 7635 Acked Flush 0 Done Sent Done false
26582 7636 Acked Write 3 Done Sent Done false
26583 7637 Acked Write 5 Done Sent Done false
26584 7638 Acked Read 5 Done Sent Done false
26585 7639 Acked Read 9 Done Sent Done false
26586 STATES DS:0 DS:1 DS:2 TOTAL
26587 Sent 0 301 0 301
26588 Done 301 0 301 602
26589 Skipped 0 0 0 0
26590 Error 0 0 0 0
26591 Last Flush: Some(JobId(7635)) Some(JobId(7338)) Some(JobId(7635))
26592 Downstairs last five completed: 7338 7337 7336 7335 7334
26593 Upstairs last five completed: 7639 7638 7637 7636 7635
26594 CLIENT: Up:0 ds:301 act:2
265952025-02-28T06:06:53.928ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
265962025-02-28T06:06:54.930ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
26597 [8820] got start action
26598 [8820] I am going to restart
26599 Starting downstairs at port 8820
26600 Make output file at "/tmp/dsc/downstairs-8820.txt"
26601 [8820][1] reports Starting
26602 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1233)
26603 [8820][1] reports Running
266042025-02-28T06:06:55.932ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
266052025-02-28T06:06:56.932ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
26606 ----------------------------------------------------------------
26607 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26608 States: Active Offline Active
26609 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26610 7339 Acked Flush 0 Done Sent Done false
26611 7340 Acked Flush 0 Done Sent Done false
26612 7341 Acked Flush 0 Done Sent Done false
26613 7342 Acked Write 5 Done Sent Done false
26614 7343 Acked Read 1 Done Sent Done false
26615 7344 Acked Read 5 Done Sent Done false
26616 7345 Acked Write 6 Done Sent Done false
26617 7346 Acked Read 9 Done Sent Done false
26618 7347 Acked Write 8 Done Sent Done false
26619 7348 Acked Read 1 Done Sent Done false
26620 7349 Acked Read 5 Done Sent Done false
26621 7350 Acked Write 5 Done Sent Done false
26622 7351 Acked Read 7 Done Sent Done false
26623 7352 Acked Read 4 Done Sent Done false
26624 7353 Acked Read 10 Done Sent Done false
26625 7354 Acked Write 10 Done Sent Done false
26626 7355 Acked Write 4 Done Sent Done false
26627 7356 Acked Write 5 Done Sent Done false
26628 7357 Acked Read 9 Done Sent Done false
26629 7358 Acked Read 6 Done Sent Done false
26630 7359 Acked Flush 0 Done Sent Done false
26631 7360 Acked Write 4 Done Sent Done false
26632 7361 Acked Write 4 Done Sent Done false
26633 7362 Acked Read 1 Done Sent Done false
26634 7363 Acked Read 8 Done Sent Done false
26635 7364 Acked Read 2 Done Sent Done false
26636 7365 Acked Write 5 Done Sent Done false
26637 7366 Acked Write 7 Done Sent Done false
26638 7367 Acked Read 4 Done Sent Done false
26639 7368 Acked Read 6 Done Sent Done false
26640 7369 Acked Read 10 Done Sent Done false
26641 7370 Acked Write 8 Done Sent Done false
26642 7371 Acked Write 3 Done Sent Done false
26643 7372 Acked Read 8 Done Sent Done false
26644 7373 Acked Read 9 Done Sent Done false
26645 7374 Acked Write 2 Done Sent Done false
26646 7375 Acked Read 9 Done Sent Done false
26647 7376 Acked Read 5 Done Sent Done false
26648 7377 Acked Flush 0 Done Sent Done false
26649 7378 Acked Read 1 Done Sent Done false
26650 7379 Acked Read 7 Done Sent Done false
26651 7380 Acked Read 8 Done Sent Done false
26652 7381 Acked Flush 0 Done Sent Done false
26653 7382 Acked Read 7 Done Sent Done false
26654 7383 Acked Read 7 Done Sent Done false
26655 7384 Acked Read 7 Done Sent Done false
26656 7385 Acked Write 7 Done Sent Done false
26657 7386 Acked Read 6 Done Sent Done false
26658 7387 Acked Flush 0 Done Sent Done false
26659 7388 Acked Read 1 Done Sent Done false
26660 7389 Acked Write 9 Done Sent Done false
26661 7390 Acked Read 6 Done Sent Done false
26662 7391 Acked Write 2 Done Sent Done false
26663 7392 Acked Write 4 Done Sent Done false
26664 7393 Acked Write 10 Done Sent Done false
26665 7394 Acked Read 1 Done Sent Done false
26666 7395 Acked Write 2 Done Sent Done false
26667 7396 Acked Read 6 Done Sent Done false
26668 7397 Acked Read 6 Done Sent Done false
26669 7398 Acked Read 4 Done Sent Done false
26670 7399 Acked Read 10 Done Sent Done false
26671 7400 Acked Read 6 Done Sent Done false
26672 7401 Acked Flush 0 Done Sent Done false
26673 7402 Acked Read 9 Done Sent Done false
26674 7403 Acked Read 8 Done Sent Done false
26675 7404 Acked Write 4 Done Sent Done false
26676 7405 Acked Write 3 Done Sent Done false
26677 7406 Acked Read 4 Done Sent Done false
26678 7407 Acked Read 3 Done Sent Done false
26679 7408 Acked Read 2 Done Sent Done false
26680 7409 Acked Flush 0 Done Sent Done false
26681 7410 Acked Read 4 Done Sent Done false
26682 7411 Acked Write 1 Done Sent Done false
26683 7412 Acked Read 6 Done Sent Done false
26684 7413 Acked Write 5 Done Sent Done false
26685 7414 Acked Write 4 Done Sent Done false
26686 7415 Acked Read 10 Done Sent Done false
26687 7416 Acked Write 6 Done Sent Done false
26688 7417 Acked Write 3 Done Sent Done false
26689 7418 Acked Flush 0 Done Sent Done false
26690 7419 Acked Read 10 Done Sent Done false
26691 7420 Acked Write 1 Done Sent Done false
26692 7421 Acked Write 9 Done Sent Done false
26693 7422 Acked Read 1 Done Sent Done false
26694 7423 Acked Read 2 Done Sent Done false
26695 7424 Acked Write 7 Done Sent Done false
26696 7425 Acked Read 7 Done Sent Done false
26697 7426 Acked Read 1 Done Sent Done false
26698 7427 Acked Flush 0 Done Sent Done false
26699 7428 Acked Flush 0 Done Sent Done false
26700 7429 Acked Read 3 Done Sent Done false
26701 7430 Acked Write 7 Done Sent Done false
26702 7431 Acked Read 9 Done Sent Done false
26703 7432 Acked Write 7 Done Sent Done false
26704 7433 Acked Read 6 Done Sent Done false
26705 7434 Acked Read 2 Done Sent Done false
26706 7435 Acked Write 5 Done Sent Done false
26707 7436 Acked Read 6 Done Sent Done false
26708 7437 Acked Read 5 Done Sent Done false
26709 7438 Acked Read 6 Done Sent Done false
26710 7439 Acked Flush 0 Done Sent Done false
26711 7440 Acked Flush 0 Done Sent Done false
26712 7441 Acked Read 9 Done Sent Done false
26713 7442 Acked Write 3 Done Sent Done false
26714 7443 Acked Read 4 Done Sent Done false
26715 7444 Acked Read 8 Done Sent Done false
26716 7445 Acked Read 10 Done Sent Done false
26717 7446 Acked Read 2 Done Sent Done false
26718 7447 Acked Write 2 Done Sent Done false
26719 7448 Acked Read 8 Done Sent Done false
26720 7449 Acked Read 10 Done Sent Done false
26721 7450 Acked Write 8 Done Sent Done false
26722 7451 Acked Read 5 Done Sent Done false
26723 7452 Acked Flush 0 Done Sent Done false
26724 7453 Acked Read 4 Done Sent Done false
26725 7454 Acked Write 8 Done Sent Done false
26726 7455 Acked Read 9 Done Sent Done false
26727 7456 Acked Flush 0 Done Sent Done false
26728 7457 Acked Read 3 Done Sent Done false
26729 7458 Acked Read 8 Done Sent Done false
26730 7459 Acked Read 8 Done Sent Done false
26731 7460 Acked Read 8 Done Sent Done false
26732 7461 Acked Read 2 Done Sent Done false
26733 7462 Acked Read 9 Done Sent Done false
26734 7463 Acked Write 2 Done Sent Done false
26735 7464 Acked Write 10 Done Sent Done false
26736 7465 Acked Read 5 Done Sent Done false
26737 7466 Acked Read 1 Done Sent Done false
26738 7467 Acked Flush 0 Done Sent Done false
26739 7468 Acked Write 4 Done Sent Done false
26740 7469 Acked Write 7 Done Sent Done false
26741 7470 Acked Write 7 Done Sent Done false
26742 7471 Acked Read 5 Done Sent Done false
26743 7472 Acked Read 4 Done Sent Done false
26744 7473 Acked Write 7 Done Sent Done false
26745 7474 Acked Read 10 Done Sent Done false
26746 7475 Acked Write 10 Done Sent Done false
26747 7476 Acked Write 5 Done Sent Done false
26748 7477 Acked Flush 0 Done Sent Done false
26749 7478 Acked Write 6 Done Sent Done false
26750 7479 Acked Write 6 Done Sent Done false
26751 7480 Acked Read 2 Done Sent Done false
26752 7481 Acked Read 7 Done Sent Done false
26753 7482 Acked Read 9 Done Sent Done false
26754 7483 Acked Write 6 Done Sent Done false
26755 7484 Acked Read 10 Done Sent Done false
26756 7485 Acked Read 7 Done Sent Done false
26757 7486 Acked Read 3 Done Sent Done false
26758 7487 Acked Read 9 Done Sent Done false
26759 7488 Acked Read 5 Done Sent Done false
26760 7489 Acked Read 6 Done Sent Done false
26761 7490 Acked Read 10 Done Sent Done false
26762 7491 Acked Read 4 Done Sent Done false
26763 7492 Acked Read 5 Done Sent Done false
26764 7493 Acked Read 7 Done Sent Done false
26765 7494 Acked Read 3 Done Sent Done false
26766 7495 Acked Read 3 Done Sent Done false
26767 7496 Acked Read 10 Done Sent Done false
26768 7497 Acked Read 2 Done Sent Done false
26769 7498 Acked Flush 0 Done Sent Done false
26770 7499 Acked Write 6 Done Sent Done false
26771 7500 Acked Read 7 Done Sent Done false
26772 7501 Acked Read 5 Done Sent Done false
26773 7502 Acked Read 6 Done Sent Done false
26774 7503 Acked Read 7 Done Sent Done false
26775 7504 Acked Write 5 Done Sent Done false
26776 7505 Acked Read 7 Done Sent Done false
26777 7506 Acked Read 7 Done Sent Done false
26778 7507 Acked Read 1 Done Sent Done false
26779 7508 Acked Read 5 Done Sent Done false
26780 7509 Acked Read 6 Done Sent Done false
26781 7510 Acked Write 1 Done Sent Done false
26782 7511 Acked Read 9 Done Sent Done false
26783 7512 Acked Read 5 Done Sent Done false
26784 7513 Acked Write 3 Done Sent Done false
26785 7514 Acked Write 5 Done Sent Done false
26786 7515 Acked Write 8 Done Sent Done false
26787 7516 Acked Flush 0 Done Sent Done false
26788 7517 Acked Read 8 Done Sent Done false
26789 7518 Acked Read 4 Done Sent Done false
26790 7519 Acked Flush 0 Done Sent Done false
26791 7520 Acked Write 3 Done Sent Done false
26792 7521 Acked Read 2 Done Sent Done false
26793 7522 Acked Read 2 Done Sent Done false
26794 7523 Acked Read 4 Done Sent Done false
26795 7524 Acked Read 5 Done Sent Done false
26796 7525 Acked Read 10 Done Sent Done false
26797 7526 Acked Read 1 Done Sent Done false
26798 7527 Acked Read 1 Done Sent Done false
26799 7528 Acked Read 9 Done Sent Done false
26800 7529 Acked Read 4 Done Sent Done false
26801 7530 Acked Flush 0 Done Sent Done false
26802 7531 Acked Read 6 Done Sent Done false
26803 7532 Acked Write 10 Done Sent Done false
26804 7533 Acked Write 10 Done Sent Done false
26805 7534 Acked Write 9 Done Sent Done false
26806 7535 Acked Write 4 Done Sent Done false
26807 7536 Acked Write 2 Done Sent Done false
26808 7537 Acked Read 5 Done Sent Done false
26809 7538 Acked Read 2 Done Sent Done false
26810 7539 Acked Write 2 Done Sent Done false
26811 7540 Acked Write 10 Done Sent Done false
26812 7541 Acked Read 3 Done Sent Done false
26813 7542 Acked Read 10 Done Sent Done false
26814 7543 Acked Read 4 Done Sent Done false
26815 7544 Acked Write 3 Done Sent Done false
26816 7545 Acked Read 4 Done Sent Done false
26817 7546 Acked Write 3 Done Sent Done false
26818 7547 Acked Flush 0 Done Sent Done false
26819 7548 Acked Read 10 Done Sent Done false
26820 7549 Acked Read 4 Done Sent Done false
26821 7550 Acked Read 9 Done Sent Done false
26822 7551 Acked Read 5 Done Sent Done false
26823 7552 Acked Write 9 Done Sent Done false
26824 7553 Acked Write 6 Done Sent Done false
26825 7554 Acked Flush 0 Done Sent Done false
26826 7555 Acked Write 5 Done Sent Done false
26827 7556 Acked Read 9 Done Sent Done false
26828 7557 Acked Read 10 Done Sent Done false
26829 7558 Acked Read 7 Done Sent Done false
26830 7559 Acked Write 1 Done Sent Done false
26831 7560 Acked Flush 0 Done Sent Done false
26832 7561 Acked Write 3 Done Sent Done false
26833 7562 Acked Write 9 Done Sent Done false
26834 7563 Acked Write 4 Done Sent Done false
26835 7564 Acked Write 10 Done Sent Done false
26836 7565 Acked Write 9 Done Sent Done false
26837 7566 Acked Write 10 Done Sent Done false
26838 7567 Acked Read 8 Done Sent Done false
26839 7568 Acked Write 2 Done Sent Done false
26840 7569 Acked Write 8 Done Sent Done false
26841 7570 Acked Read 8 Done Sent Done false
26842 7571 Acked Write 3 Done Sent Done false
26843 7572 Acked Read 7 Done Sent Done false
26844 7573 Acked Write 6 Done Sent Done false
26845 7574 Acked Read 4 Done Sent Done false
26846 7575 Acked Read 2 Done Sent Done false
26847 7576 Acked Read 8 Done Sent Done false
26848 7577 Acked Write 8 Done Sent Done false
26849 7578 Acked Read 8 Done Sent Done false
26850 7579 Acked Write 8 Done Sent Done false
26851 7580 Acked Write 7 Done Sent Done false
26852 7581 Acked Read 7 Done Sent Done false
26853 7582 Acked Write 5 Done Sent Done false
26854 7583 Acked Write 6 Done Sent Done false
26855 7584 Acked Read 9 Done Sent Done false
26856 7585 Acked Write 2 Done Sent Done false
26857 7586 Acked Write 8 Done Sent Done false
26858 7587 Acked Read 5 Done Sent Done false
26859 7588 Acked Read 10 Done Sent Done false
26860 7589 Acked Write 5 Done Sent Done false
26861 7590 Acked Write 6 Done Sent Done false
26862 7591 Acked Write 5 Done Sent Done false
26863 7592 Acked Write 6 Done Sent Done false
26864 7593 Acked Write 9 Done Sent Done false
26865 7594 Acked Read 5 Done Sent Done false
26866 7595 Acked Write 9 Done Sent Done false
26867 7596 Acked Write 6 Done Sent Done false
26868 7597 Acked Read 7 Done Sent Done false
26869 7598 Acked Write 2 Done Sent Done false
26870 7599 Acked Read 2 Done Sent Done false
26871 7600 Acked Read 8 Done Sent Done false
26872 7601 Acked Write 6 Done Sent Done false
26873 7602 Acked Read 7 Done Sent Done false
26874 7603 Acked Write 7 Done Sent Done false
26875 7604 Acked Write 6 Done Sent Done false
26876 7605 Acked Read 5 Done Sent Done false
26877 7606 Acked Read 3 Done Sent Done false
26878 7607 Acked Read 8 Done Sent Done false
26879 7608 Acked Read 10 Done Sent Done false
26880 7609 Acked Read 10 Done Sent Done false
26881 7610 Acked Flush 0 Done Sent Done false
26882 7611 Acked Write 4 Done Sent Done false
26883 7612 Acked Write 9 Done Sent Done false
26884 7613 Acked Flush 0 Done Sent Done false
26885 7614 Acked Write 5 Done Sent Done false
26886 7615 Acked Write 3 Done Sent Done false
26887 7616 Acked Read 2 Done Sent Done false
26888 7617 Acked Write 9 Done Sent Done false
26889 7618 Acked Read 2 Done Sent Done false
26890 7619 Acked Read 5 Done Sent Done false
26891 7620 Acked Write 4 Done Sent Done false
26892 7621 Acked Write 3 Done Sent Done false
26893 7622 Acked Write 1 Done Sent Done false
26894 7623 Acked Read 7 Done Sent Done false
26895 7624 Acked Write 1 Done Sent Done false
26896 7625 Acked Write 5 Done Sent Done false
26897 7626 Acked Read 6 Done Sent Done false
26898 7627 Acked Flush 0 Done Sent Done false
26899 7628 Acked Write 9 Done Sent Done false
26900 7629 Acked Flush 0 Done Sent Done false
26901 7630 Acked Read 9 Done Sent Done false
26902 7631 Acked Write 10 Done Sent Done false
26903 7632 Acked Read 2 Done Sent Done false
26904 7633 Acked Write 9 Done Sent Done false
26905 7634 Acked Read 3 Done Sent Done false
26906 7635 Acked Flush 0 Done Sent Done false
26907 7636 Acked Write 3 Done Sent Done false
26908 7637 Acked Write 5 Done Sent Done false
26909 7638 Acked Read 5 Done Sent Done false
26910 7639 Acked Read 9 Done Sent Done false
26911 STATES DS:0 DS:1 DS:2 TOTAL
26912 Sent 0 301 0 301
26913 Done 301 0 301 602
26914 Skipped 0 0 0 0
26915 Error 0 0 0 0
26916 Last Flush: Some(JobId(7635)) Some(JobId(7338)) Some(JobId(7635))
26917 Downstairs last five completed: 7338 7337 7336 7335 7334
26918 Upstairs last five completed: 7639 7638 7637 7636 7635
26919 CLIENT: Up:0 ds:301 act:2
269202025-02-28T06:06:58.142ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
269212025-02-28T06:06:58.937ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
269222025-02-28T06:06:59.138ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
269232025-02-28T06:06:59.138ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44062) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
269242025-02-28T06:06:59.139ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
269252025-02-28T06:06:59.140ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
269262025-02-28T06:06:59.140ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
269272025-02-28T06:06:59.140ZINFOcrucible: send last flush ID to this DS: Some(JobId(7338)) = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
269282025-02-28T06:06:59.140ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
269292025-02-28T06:06:59.140ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7338)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
269302025-02-28T06:06:59.140ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7338)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
26931 ----------------------------------------------------------------
26932 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26933 Downstairs last five completed: 7640 7639 7638 7637 7636
26934 Upstairs last five completed: 7640 7639 7638 7637 7636
26935 CLIENT: Up:0 ds:0 act:3
26936 Replay: All jobs finished
26937 Main task has work to do, go find it
26938 got dsc Stop(0)
26939 stop 0
26940 [8810] Got stop action so:false kr:false
26941 Feb 28 06:07:01.395 INFO request completed, latency_us: 358, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 5bfaf8e0-4f20-4368-8c7c-d18f2c185a5e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
269422025-02-28T06:07:01.395ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
269432025-02-28T06:07:01.395ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
269442025-02-28T06:07:01.395ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
26945 Feb 28 06:07:01.396 INFO request completed, latency_us: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:01.396007914Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
26946 154, response_code: 200, uri: /state/cid/0, method: GET, req_id: 7d1082fa-4b70-46a3-b6e2-7472c6f38a54, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
26947 [8810] Exited with: ExitStatus(unix_wait_status(9))
26948 [8810][0] reports Exit
26949 Feb 28 06:07:05.397 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/0, method: GET, req_id: ac63af56-12cb-4e13-baa9-5424b23f22a5, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
26950 001/300 Write block 2575 len 40960 data: 2 2 3 3 5 5 5 5 5 6
26951 002/300 Read block 609 len 36864
26952 003/300 Read block 2856 len 32768
26953 004/300 Write block 1785 len 12288 data: 4 3 3
26954 005/300 Read block 3235 len 16384
26955 006/300 Flush
26956 007/300 Write block 547 len 16384 data: 5 5 6 7
26957 008/300 Write block 4184 len 40960 data: 4 4 6 7 5 5 7 7 6 6
26958 009/300 Read block 4417 len 4096
26959 010/300 Write block 3219 len 16384 data: 6 7 7 5
26960 011/300 Read block 2952 len 40960
26961 012/300 Read block 3475 len 40960
26962 013/300 Read block 2098 len 36864
26963 014/300 Read block 1329 len 12288
26964 015/300 Write block 3739 len 24576 data: 4 4 5 8 10 10
26965 016/300 Read block 1633 len 8192
26966 017/300 Read block 2779 len 28672
26967 018/300 Read block 663 len 36864
26968 019/300 Write block 3169 len 32768 data: 5 5 3 3 2 2 2 2
26969 020/300 Read block 4765 len 32768
26970 021/300 Flush
26971 022/300 Flush
26972 023/300 Write block 3996 len 4096 data: 4
26973 024/300 Flush
26974 025/300 Write block 4865 len 4096 data: 6
26975 026/300 Flush
26976 027/300 Write block 4264 len 20480 data: 3 3 5 6 6
26977 028/300 Read block 1709 len 32768
26978 029/300 Read block 811 len 36864
26979 030/300 Read block 2581 len 4096
26980 031/300 Read block 2662 len 12288
26981 032/300 Write block 561 len 20480 data: 6 6 6 6 6
26982 033/300 Write block 2837 len 40960 data: 4 4 4 4 5 5 6 6 5 6
26983 034/300 Read block 2475 len 8192
26984 035/300 Flush
26985 036/300 Read block 3931 len 8192
26986 037/300 Write block 1051 len 8192 data: 5 6
26987 038/300 Read block 4595 len 40960
26988 039/300 Write block 4442 len 4096 data: 6
26989 040/300 Read block 3608 len 16384
26990 041/300 Read block 1903 len 20480
26991 042/300 Write block 3414 len 40960 data: 3 2 3 5 5 5 4 4 4 4
26992 043/300 Write block 4472 len 32768 data: 4 4 4 5 6 4 4 5
26993 044/300 Write block 4481 len 4096 data: 6
26994 045/300 Write block 203 len 16384 data: 5 5 4 4
26995 046/300 Write block 1068 len 8192 data: 5 6
26996 047/300 Write block 2081 len 12288 data: 3 3 5
26997 048/300 Read block 155 len 4096
26998 049/300 Read block 2750 len 20480
26999 050/300 Read block 1695 len 40960
27000 051/300 Write block 4879 len 4096 data: 8
27001 052/300 Flush
27002 053/300 Read block 2223 len 20480
27003 054/300 Read block 2853 len 32768
27004 055/300 Read block 3416 len 36864
27005 056/300 Read block 2307 len 40960
27006 057/300 Write block 4543 len 16384 data: 4 5 4 4
27007 058/300 Read block 494 len 24576
27008 059/300 Read block 2441 len 16384
27009 060/300 Read block 4166 len 28672
27010 061/300 Write block 4594 len 20480 data: 7 9 10 9 9
27011 062/300 Write block 1728 len 4096 data: 5
27012 063/300 Flush
27013 064/300 Read block 741 len 28672
27014 065/300 Write block 2657 len 4096 data: 6
27015 066/300 Read block 4168 len 4096
27016 067/300 Write block 2672 len 36864 data: 9 9 9 10 9 6 7 7 6
27017 068/300 Flush
27018 069/300 Read block 2701 len 4096
27019 070/300 Write block 4964 len 24576 data: 5 6 5 6 8 7
27020 071/300 Read block 4058 len 12288
27021 072/300 Read block 2129 len 16384
27022 073/300 Write block 3461 len 28672 data: 4 4 4 4 6 5 5
27023 074/300 Read block 33 len 12288
27024 075/300 Write block 2657 len 8192 data: 7 6
27025 076/300 Write block 4405 len 16384 data: 9 7 8 8
27026 077/300 Write block 4133 len 28672 data: 5 4 4 4 4 5 5
27027 078/300 Read block 3487 len 36864
27028 079/300 Read block 1386 len 8192
27029 080/300 Read block 1359 len 16384
27030 081/300 Write block 4596 len 20480 data: 11 10 10 6 6
27031 082/300 Read block 2960 len 24576
27032 083/300 Write block 1702 len 16384 data: 5 7 5 4
27033 084/300 Write block 713 len 24576 data: 5 4 5 6 6 7
27034 085/300 Write block 1441 len 8192 data: 7 7
27035 086/300 Write block 3734 len 20480 data: 7 7 5 5 4
27036 087/300 Read block 4116 len 28672
27037 088/300 Read block 1847 len 8192
27038 089/300 Write block 2468 len 8192 data: 5 3
27039 090/300 Read block 2933 len 36864
27040 091/300 Read block 4820 len 20480
27041 092/300 Read block 1858 len 16384
27042 093/300 Write block 2216 len 20480 data: 9 9 9 7 5
27043 094/300 Read block 3178 len 32768
27044 095/300 Write block 4267 len 20480 data: 7 7 6 6 8
27045 096/300 Read block 2434 len 20480
27046 097/300 Read block 2576 len 16384
27047 098/300 Read block 686 len 4096
27048 099/300 Read block 1217 len 24576
27049 100/300 Read block 4066 len 28672
27050 101/300 Read block 2082 len 4096
27051 102/300 Read block 2092 len 12288
27052 103/300 Read block 2146 len 28672
27053 104/300 Read block 618 len 36864
27054 105/300 Write block 1665 len 28672 data: 9 9 7 8 8 7 6
27055 106/300 Read block 3567 len 40960
27056 107/300 Write block 4599 len 12288 data: 7 7 6
27057 108/300 Write block 3407 len 32768 data: 4 4 3 4 4 3 3 4
27058 109/300 Read block 3903 len 32768
27059 110/300 Read block 604 len 20480
27060 111/300 Write block 3771 len 12288 data: 5 5 6
27061 112/300 Read block 3591 len 8192
27062 113/300 Flush
27063 114/300 Read block 3997 len 20480
27064 115/300 Read block 1132 len 4096
27065 116/300 Write block 1385 len 36864 data: 5 5 5 3 3 3 4 4 3
27066 117/300 Write block 4886 len 28672 data: 8 12 12 12 11 10 8
27067 118/300 Write block 3410 len 16384 data: 5 5 4 4
27068 119/300 Flush
27069 120/300 Read block 2807 len 40960
27070 121/300 Write block 2564 len 28672 data: 6 5 4 3 4 5 6
27071 122/300 Write block 1130 len 32768 data: 5 6 5 4 5 5 6 6
27072 123/300 Flush
27073 124/300 Read block 859 len 32768
27074 125/300 Write block 644 len 28672 data: 5 5 4 5 6 5 5
27075 126/300 Read block 4161 len 20480
27076 127/300 Read block 2738 len 32768
27077 128/300 Write block 4829 len 8192 data: 8 10
27078 129/300 Read block 2403 len 8192
27079 130/300 Read block 2805 len 12288
27080 131/300 Write block 2812 len 20480 data: 5 6 7 7 7
27081 132/300 Read block 1877 len 36864
27082 133/300 Read block 71 len 8192
27083 134/300 Read block 2524 len 4096
27084 135/300 Read block 1859 len 40960
27085 136/300 Read block 2288 len 32768
27086 137/300 Write block 2477 len 20480 data: 6 5 5 5 4
27087 138/300 Write block 1092 len 24576 data: 2 2 2 2 2 3
27088 139/300 Read block 443 len 12288
27089 140/300 Read block 2916 len 12288
27090 141/300 Read block 2948 len 4096
27091 142/300 Write block 4262 len 20480 data: 4 4 4 4 6
27092 143/300 Write block 1134 len 32768 data: 6 6 7 7 6 6 6 5
27093 144/300 Read block 2620 len 4096
27094 145/300 Write block 103 len 24576 data: 4 4 4 4 4 3
27095 146/300 Write block 3767 len 8192 data: 5 6
27096 147/300 Write block 1531 len 32768 data: 4 4 4 6 5 5 5 6
27097 148/300 Write block 1357 len 20480 data: 5 6 7 7 6
27098 149/300 Read block 262 len 16384
27099 150/300 Read block 2586 len 36864
27100 151/300 Read block 3294 len 16384
27101 152/300 Read block 1890 len 20480
27102 153/300 Read block 2752 len 28672
27103 154/300 Flush
27104 155/300 Write block 501 len 36864 data: 5 5 5 5 6 6 5 5 6
27105 156/300 Read block 2127 len 40960
27106 157/300 Read block 2001 len 12288
27107 158/300 Read block 2528 len 4096
27108 159/300 Read block 897 len 40960
27109 160/300 Write block 1640 len 20480 data: 2 4 5 4 4
27110 161/300 Read block 2441 len 4096
27111 162/300 Flush
27112 163/300 Read block 1855 len 36864
27113 164/300 Write block 2889 len 24576 data: 5 3 3 4 3 4
27114 165/300 Read block 4185 len 4096
27115 166/300 Read block 1029 len 12288
27116 167/300 Write block 3989 len 24576 data: 4 4 4 4 5 5
27117 168/300 Write block 4673 len 32768 data: 3 3 3 3 4 5 5 5
27118 169/300 Read block 343 len 28672
27119 170/300 Write block 3641 len 4096 data: 2
27120 171/300 Read block 1733 len 28672
27121 172/300 Read block 213 len 40960
27122 173/300 Read block 1025 len 28672
27123 174/300 Read block 3849 len 4096
27124 175/300 Write block 2637 len 28672 data: 5 6 6 6 6 6 6
27125 176/300 Write block 1318 len 24576 data: 4 5 5 5 6 6
27126 177/300 Write block 3984 len 32768 data: 3 4 7 6 4 5 5 5
27127 178/300 Read block 4056 len 12288
27128 179/300 Write block 1999 len 40960 data: 5 7 5 6 8 6 5 5 4 3
27129 180/300 Write block 1355 len 40960 data: 6 5 6 7 8 8 7 4 4 5
27130 181/300 Read block 2292 len 24576
27131 182/300 Write block 1044 len 40960 data: 6 6 5 4 4 7 6 6 7 7
27132 183/300 Write block 2189 len 4096 data: 8
27133 184/300 Flush
27134 185/300 Read block 755 len 28672
27135 186/300 Read block 2081 len 28672
27136 187/300 Write block 4801 len 40960 data: 3 3 2 3 4 4 5 5 4 5
27137 188/300 Flush
27138 189/300 Read block 1277 len 36864
27139 190/300 Write block 1456 len 36864 data: 5 5 4 5 4 3 3 3 3
27140 191/300 Write block 2901 len 36864 data: 3 3 3 3 4 4 4 5 5
27141 192/300 Write block 535 len 40960 data: 3 2 3 3 3 3 2 2 3 3
27142 193/300 Write block 534 len 16384 data: 5 4 3 4
27143 194/300 Read block 4416 len 16384
27144 195/300 Read block 4615 len 16384
27145 196/300 Write block 94 len 12288 data: 10 11 7
27146 197/300 Read block 679 len 40960
27147 198/300 Write block 390 len 16384 data: 3 3 4 3
27148 199/300 Read block 1568 len 4096
27149 200/300 Write block 3702 len 4096 data: 3
27150 201/300 Write block 4507 len 16384 data: 6 4 4 7
27151 202/300 Write block 3390 len 24576 data: 3 3 2 2 2 3
27152 203/300 Write block 4733 len 16384 data: 7 7 5 5
27153 204/300 Write block 2087 len 32768 data: 8 8 9 9 9 5 6 6
27154 205/300 Write block 2648 len 40960 data: 6 6 6 7 3 4 4 4 5 8
27155 206/300 Read block 599 len 36864
27156 207/300 Write block 1869 len 24576 data: 8 6 5 6 7 7
27157 208/300 Read block 1218 len 28672
27158 209/300 Read block 3612 len 24576
27159 210/300 Flush
27160 211/300 Read block 3805 len 28672
27161 212/300 Write block 4327 len 8192 data: 5 5
27162 213/300 Read block 1066 len 8192
27163 214/300 Write block 620 len 32768 data: 4 4 4 5 5 6 4 4
27164 215/300 Write block 3018 len 36864 data: 4 4 4 4 4 4 4 3 3
27165 216/300 Write block 3995 len 8192 data: 4 5
27166 217/300 Read block 1776 len 36864
27167 218/300 Write block 2825 len 28672 data: 6 6 4 3 2 2 2
27168 219/300 Write block 4832 len 20480 data: 9 6 6 8 9
27169 220/300 Flush
27170 221/300 Read block 1007 len 12288
27171 222/300 Flush
27172 223/300 Write block 3604 len 28672 data: 3 4 4 5 5 4 6
27173 224/300 Write block 1864 len 36864 data: 4 4 4 4 5 9 7 6 7
27174 225/300 Write block 671 len 12288 data: 7 8 7
27175 226/300 Flush
27176 227/300 Read block 779 len 20480
27177 228/300 Write block 1034 len 40960 data: 3 3 3 5 5 6 8 6 6 6
27178 229/300 Write block 229 len 12288 data: 3 5 5
27179 230/300 Read block 2800 len 4096
27180 231/300 Read block 43 len 36864
27181 232/300 Read block 2720 len 40960
27182 233/300 Read block 3190 len 4096
27183 234/300 Write block 1169 len 4096 data: 7
27184 235/300 Write block 3869 len 28672 data: 8 7 4 4 6 6 6
27185 236/300 Read block 3722 len 32768
27186 237/300 Write block 4507 len 36864 data: 7 5 5 8 6 6 7 8 7
27187 238/300 Read block 582 len 20480
27188 239/300 Write block 1972 len 16384 data: 7 7 7 7
27189 240/300 Write block 2129 len 16384 data: 4 5 6 7
27190 241/300 Write block 4675 len 4096 data: 4
27191 242/300 Read block 1290 len 36864
27192 243/300 Flush
27193 244/300 Write block 488 len 8192 data: 5 4
27194 245/300 Write block 2427 len 40960 data: 5 4 3 4 5 5 4 4 5 5
27195 246/300 Read block 4548 len 28672
27196 247/300 Write block 748 len 16384 data: 7 8 8 5
27197 248/300 Flush
27198 249/300 Read block 2619 len 40960
27199 250/300 Read block 583 len 40960
27200 251/300 Write block 706 len 28672 data: 6 6 5 7 7 6 5
27201 252/300 Write block 1365 len 40960 data: 6 5 5 6 4 4 5 5 5 5
27202 253/300 Write block 1264 len 24576 data: 8 6 5 4 4 3
27203 254/300 Write block 313 len 12288 data: 2 2 3
27204 255/300 Write block 1396 len 4096 data: 4
27205 256/300 Read block 1043 len 20480
27206 257/300 Read block 383 len 28672
27207 258/300 Flush
27208 259/300 Read block 3730 len 40960
27209 260/300 Write block 1700 len 4096 data: 7
27210 261/300 Write block 2890 len 32768 data: 4 4 5 4 5 5 5 6
27211 262/300 Read block 2059 len 12288
27212 263/300 Write block 4407 len 16384 data: 9 9 8 8
27213 264/300 Read block 1356 len 32768
27214 265/300 Write block 3094 len 16384 data: 6 4 5 5
27215 266/300 Read block 883 len 28672
27216 267/300 Read block 503 len 20480
27217 268/300 Write block 3749 len 4096 data: 5
27218 269/300 Write block 2419 len 4096 data: 6
27219 270/300 Write block 413 len 40960 data: 7 9 7 7 8 10 11 9 10 11
27220 271/300 Write block 351 len 8192 data: 4 5
27221 272/300 Write block 1767 len 4096 data: 3
27222 273/300 Write block 3148 len 36864 data: 4 3 3 4 4 4 5 5 8
27223 274/300 Read block 4510 len 24576
27224 275/300 Read block 1134 len 20480
27225 276/300 Write block 3526 len 36864 data: 7 8 9 8 6 6 6 6 5
27226 277/300 Write block 1758 len 8192 data: 6 6
27227 278/300 Read block 2856 len 8192
27228 279/300 Write block 1925 len 12288 data: 2 3 4
27229 280/300 Write block 3103 len 20480 data: 7 6 7 6 8
27230 281/300 Write block 3864 len 40960 data: 7 7 7 8 7 9 8 5 5 7
27231 282/300 Read block 1498 len 16384
27232 283/300 Write block 3515 len 12288 data: 4 4 3
27233 284/300 Read block 1253 len 20480
27234 285/300 Read block 649 len 4096
27235 286/300 Write block 4871 len 16384 data: 4 4 4 4
27236 287/300 Write block 2267 len 32768 data: 6 6 5 5 4 4 6 7
27237 288/300 Write block 3443 len 20480 data: 4 4 3 3 3
27238 289/300 Read block 504 len 16384
27239 290/300 Read block 4405 len 24576
27240 291/300 Write block 2958 len 24576 data: 5 4 4 6 7 6
27241 292/300 Read block 150 len 20480
27242 293/300 Write block 4298 len 8192 data: 5 3
27243 294/300 Write block 4235 len 12288 data: 4 4 4
27244 295/300 Read block 4931 len 32768
27245 296/300 Write block 310 len 24576 data: 2 3 2 3 3 4
27246 297/300 Flush
27247 298/300 Write block 3032 len 40960 data: 5 5 5 5 5 5 4 4 3 3
27248 299/300 Read block 497 len 28672
27249 300/300 Read block 1829 len 36864
27250 301/300 Read block 166 len 12288
27251 Main task has work to do, go find it
27252 got dsc Start(0)
27253 start 0
27254 Feb 28 06:07:05.635 INFO request completed, latency_us: 216, response_code: 204, uri: /start/cid/0, method: POST, req_id: ab8db44f-a524-4572-af33-6cdbe37f5f46, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
27255 Replay: started 0, returned:Ok(())
27256 ----------------------------------------------------------------
27257 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27258 States: Offline Active Active
27259 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27260 7641 Acked Write 10 Sent Done Done false
27261 7642 Acked Read 9 Sent Done Done false
27262 7643 Acked Read 8 Sent Done Done false
27263 7644 Acked Write 3 Sent Done Done false
27264 7645 Acked Read 4 Sent Done Done false
27265 7646 Acked Flush 0 Sent Done Done false
27266 7647 Acked Write 4 Sent Done Done false
27267 7648 Acked Write 10 Sent Done Done false
27268 7649 Acked Read 1 Sent Done Done false
27269 7650 Acked Write 4 Sent Done Done false
27270 7651 Acked Read 10 Sent Done Done false
27271 7652 Acked Read 10 Sent Done Done false
27272 7653 Acked Read 9 Sent Done Done false
27273 7654 Acked Read 3 Sent Done Done false
27274 7655 Acked Write 6 Sent Done Done false
27275 7656 Acked Read 2 Sent Done Done false
27276 7657 Acked Read 7 Sent Done Done false
27277 7658 Acked Read 9 Sent Done Done false
27278 7659 Acked Write 8 Sent Done Done false
27279 7660 Acked Read 8 Sent Done Done false
27280 7661 Acked Flush 0 Sent Done Done false
27281 7662 Acked Flush 0 Sent Done Done false
27282 7663 Acked Write 1 Sent Done Done false
27283 7664 Acked Flush 0 Sent Done Done false
27284 7665 Acked Write 1 Sent Done Done false
27285 7666 Acked Flush 0 Sent Done Done false
27286 7667 Acked Write 5 Sent Done Done false
27287 7668 Acked Read 8 Sent Done Done false
27288 7669 Acked Read 9 Sent Done Done false
27289 7670 Acked Read 1 Sent Done Done false
27290 7671 Acked Read 3 Sent Done Done false
27291 7672 Acked Write 5 Sent Done Done false
27292 7673 Acked Write 10 Sent Done Done false
27293 7674 Acked Read 2 Sent Done Done false
27294 7675 Acked Flush 0 Sent Done Done false
27295 7676 Acked Read 2 Sent Done Done false
27296 7677 Acked Write 2 Sent Done Done false
27297 7678 Acked Read 10 Sent Done Done false
27298 7679 Acked Write 1 Sent Done Done false
27299 7680 Acked Read 4 Sent Done Done false
27300 7681 Acked Read 5 Sent Done Done false
27301 7682 Acked Write 10 Sent Done Done false
27302 7683 Acked Write 8 Sent Done Done false
27303 7684 Acked Write 1 Sent Done Done false
27304 7685 Acked Write 4 Sent Done Done false
27305 7686 Acked Write 2 Sent Done Done false
27306 7687 Acked Write 3 Sent Done Done false
27307 7688 Acked Read 1 Sent Done Done false
27308 7689 Acked Read 5 Sent Done Done false
27309 7690 Acked Read 10 Sent Done Done false
27310 7691 Acked Write 1 Sent Done Done false
27311 7692 Acked Flush 0 Sent Done Done false
27312 7693 Acked Read 5 Sent Done Done false
27313 7694 Acked Read 8 Sent Done Done false
27314 7695 Acked Read 9 Sent Done Done false
27315 7696 Acked Read 10 Sent Done Done false
27316 7697 Acked Write 4 Sent Done Done false
27317 7698 Acked Read 6 Sent Done Done false
27318 7699 Acked Read 4 Sent Done Done false
27319 7700 Acked Read 7 Sent Done Done false
27320 7701 Acked Write 5 Sent Done Done false
27321 7702 Acked Write 1 Sent Done Done false
27322 7703 Acked Flush 0 Sent Done Done false
27323 7704 Acked Read 7 Sent Done Done false
27324 7705 Acked Write 1 Sent Done Done false
27325 7706 Acked Read 1 Sent Done Done false
27326 7707 Acked Write 9 Sent Done Done false
27327 7708 Acked Flush 0 Sent Done Done false
27328 7709 Acked Read 1 Sent Done Done false
27329 7710 Acked Write 6 Sent Done Done false
27330 7711 Acked Read 3 Sent Done Done false
27331 7712 Acked Read 4 Sent Done Done false
27332 7713 Acked Write 7 Sent Done Done false
27333 7714 Acked Read 3 Sent Done Done false
27334 7715 Acked Write 2 Sent Done Done false
27335 7716 Acked Write 4 Sent Done Done false
27336 7717 Acked Write 7 Sent Done Done false
27337 7718 Acked Read 9 Sent Done Done false
27338 7719 Acked Read 2 Sent Done Done false
27339 7720 Acked Read 4 Sent Done Done false
27340 7721 Acked Write 5 Sent Done Done false
27341 7722 Acked Read 6 Sent Done Done false
27342 7723 Acked Write 4 Sent Done Done false
27343 7724 Acked Write 6 Sent Done Done false
27344 7725 Acked Write 2 Sent Done Done false
27345 7726 Acked Write 5 Sent Done Done false
27346 7727 Acked Read 7 Sent Done Done false
27347 7728 Acked Read 2 Sent Done Done false
27348 7729 Acked Write 2 Sent Done Done false
27349 7730 Acked Read 9 Sent Done Done false
27350 7731 Acked Read 5 Sent Done Done false
27351 7732 Acked Read 4 Sent Done Done false
27352 7733 Acked Write 5 Sent Done Done false
27353 7734 Acked Read 8 Sent Done Done false
27354 7735 Acked Write 5 Sent Done Done false
27355 7736 Acked Read 5 Sent Done Done false
27356 7737 Acked Read 4 Sent Done Done false
27357 7738 Acked Read 1 Sent Done Done false
27358 7739 Acked Read 6 Sent Done Done false
27359 7740 Acked Read 7 Sent Done Done false
27360 7741 Acked Read 1 Sent Done Done false
27361 7742 Acked Read 3 Sent Done Done false
27362 7743 Acked Read 7 Sent Done Done false
27363 7744 Acked Read 9 Sent Done Done false
27364 7745 Acked Write 7 Sent Done Done false
27365 7746 Acked Read 10 Sent Done Done false
27366 7747 Acked Write 3 Sent Done Done false
27367 7748 Acked Write 8 Sent Done Done false
27368 7749 Acked Read 8 Sent Done Done false
27369 7750 Acked Read 5 Sent Done Done false
27370 7751 Acked Write 3 Sent Done Done false
27371 7752 Acked Read 2 Sent Done Done false
27372 7753 Acked Flush 0 Sent Done Done false
27373 7754 Acked Read 5 Sent Done Done false
27374 7755 Acked Read 1 Sent Done Done false
27375 7756 Acked Write 9 Sent Done Done false
27376 7757 Acked Write 7 Sent Done Done false
27377 7758 Acked Write 4 Sent Done Done false
27378 7759 Acked Flush 0 Sent Done Done false
27379 7760 Acked Read 10 Sent Done Done false
27380 7761 Acked Write 7 Sent Done Done false
27381 7762 Acked Write 8 Sent Done Done false
27382 7763 Acked Flush 0 Sent Done Done false
27383 7764 Acked Read 8 Sent Done Done false
27384 7765 Acked Write 7 Sent Done Done false
27385 7766 Acked Read 5 Sent Done Done false
27386 7767 Acked Read 8 Sent Done Done false
27387 7768 Acked Write 2 Sent Done Done false
27388 7769 Acked Read 2 Sent Done Done false
27389 7770 Acked Read 3 Sent Done Done false
27390 7771 Acked Write 5 Sent Done Done false
27391 7772 Acked Read 9 Sent Done Done false
27392 7773 Acked Read 2 Sent Done Done false
27393 7774 Acked Read 1 Sent Done Done false
27394 7775 Acked Read 10 Sent Done Done false
27395 7776 Acked Read 8 Sent Done Done false
27396 7777 Acked Write 5 Sent Done Done false
27397 7778 Acked Write 6 Sent Done Done false
27398 7779 Acked Read 3 Sent Done Done false
27399 7780 Acked Read 3 Sent Done Done false
27400 7781 Acked Read 1 Sent Done Done false
27401 7782 Acked Write 5 Sent Done Done false
27402 7783 Acked Write 8 Sent Done Done false
27403 7784 Acked Read 1 Sent Done Done false
27404 7785 Acked Write 6 Sent Done Done false
27405 7786 Acked Write 2 Sent Done Done false
27406 7787 Acked Write 8 Sent Done Done false
27407 7788 Acked Write 5 Sent Done Done false
27408 7789 Acked Read 4 Sent Done Done false
27409 7790 Acked Read 9 Sent Done Done false
27410 7791 Acked Read 4 Sent Done Done false
27411 7792 Acked Read 5 Sent Done Done false
27412 7793 Acked Read 7 Sent Done Done false
27413 7794 Acked Flush 0 Sent Done Done false
27414 7795 Acked Write 9 Sent Done Done false
27415 7796 Acked Read 10 Sent Done Done false
27416 7797 Acked Read 3 Sent Done Done false
27417 7798 Acked Read 1 Sent Done Done false
27418 7799 Acked Read 10 Sent Done Done false
27419 7800 Acked Write 5 Sent Done Done false
27420 7801 Acked Read 1 Sent Done Done false
27421 7802 Acked Flush 0 Sent Done Done false
27422 7803 Acked Read 9 Sent Done Done false
27423 7804 Acked Write 6 Sent Done Done false
27424 7805 Acked Read 1 Sent Done Done false
27425 7806 Acked Read 3 Sent Done Done false
27426 7807 Acked Write 6 Sent Done Done false
27427 7808 Acked Write 8 Sent Done Done false
27428 7809 Acked Read 7 Sent Done Done false
27429 7810 Acked Write 1 Sent Done Done false
27430 7811 Acked Read 7 Sent Done Done false
27431 7812 Acked Read 10 Sent Done Done false
27432 7813 Acked Read 7 Sent Done Done false
27433 7814 Acked Read 1 Sent Done Done false
27434 7815 Acked Write 7 Sent Done Done false
27435 7816 Acked Write 6 Sent Done Done false
27436 7817 Acked Write 8 Sent Done Done false
27437 7818 Acked Read 3 Sent Done Done false
27438 7819 Acked Write 10 Sent Done Done false
27439 7820 Acked Write 10 Sent Done Done false
27440 7821 Acked Read 6 Sent Done Done false
27441 7822 Acked Write 10 Sent Done Done false
27442 7823 Acked Write 1 Sent Done Done false
27443 7824 Acked Flush 0 Sent Done Done false
27444 7825 Acked Read 7 Sent Done Done false
27445 7826 Acked Read 7 Sent Done Done false
27446 7827 Acked Write 10 Sent Done Done false
27447 7828 Acked Flush 0 Sent Done Done false
27448 7829 Acked Read 9 Sent Done Done false
27449 7830 Acked Write 9 Sent Done Done false
27450 7831 Acked Write 9 Sent Done Done false
27451 7832 Acked Write 10 Sent Done Done false
27452 7833 Acked Write 4 Sent Done Done false
27453 7834 Acked Read 4 Sent Done Done false
27454 7835 Acked Read 4 Sent Done Done false
27455 7836 Acked Write 3 Sent Done Done false
27456 7837 Acked Read 10 Sent Done Done false
27457 7838 Acked Write 4 Sent Done Done false
27458 7839 Acked Read 1 Sent Done Done false
27459 7840 Acked Write 1 Sent Done Done false
27460 7841 Acked Write 4 Sent Done Done false
27461 7842 Acked Write 6 Sent Done Done false
27462 7843 Acked Write 4 Sent Done Done false
27463 7844 Acked Write 8 Sent Done Done false
27464 7845 Acked Write 10 Sent Done Done false
27465 7846 Acked Read 9 Sent Done Done false
27466 7847 Acked Write 6 Sent Done Done false
27467 7848 Acked Read 7 Sent Done Done false
27468 7849 Acked Read 6 Sent Done Done false
27469 7850 Acked Flush 0 Sent Done Done false
27470 7851 Acked Read 7 Sent Done Done false
27471 7852 Acked Write 2 Sent Done Done false
27472 7853 Acked Read 2 Sent Done Done false
27473 7854 Acked Write 8 Sent Done Done false
27474 7855 Acked Write 9 Sent Done Done false
27475 7856 Acked Write 2 Sent Done Done false
27476 7857 Acked Read 9 Sent Done Done false
27477 7858 Acked Write 7 Sent Done Done false
27478 7859 Acked Write 5 Sent Done Done false
27479 7860 Acked Flush 0 Sent Done Done false
27480 7861 Acked Read 3 Sent Done Done false
27481 7862 Acked Flush 0 Sent Done Done false
27482 7863 Acked Write 7 Sent Done Done false
27483 7864 Acked Write 9 Sent Done Done false
27484 7865 Acked Write 3 Sent Done Done false
27485 7866 Acked Flush 0 Sent Done Done false
27486 7867 Acked Read 5 Sent Done Done false
27487 7868 Acked Write 10 Sent Done Done false
27488 7869 Acked Write 3 Sent Done Done false
27489 7870 Acked Read 1 Sent Done Done false
27490 7871 Acked Read 9 Sent Done Done false
27491 7872 Acked Read 10 Sent Done Done false
27492 7873 Acked Read 1 Sent Done Done false
27493 7874 Acked Write 1 Sent Done Done false
27494 7875 Acked Write 7 Sent Done Done false
27495 7876 Acked Read 8 Sent Done Done false
27496 7877 Acked Write 9 Sent Done Done false
27497 7878 Acked Read 5 Sent Done Done false
27498 7879 Acked Write 4 Sent Done Done false
27499 7880 Acked Write 4 Sent Done Done false
27500 7881 Acked Write 1 Sent Done Done false
27501 7882 Acked Read 9 Sent Done Done false
27502 7883 Acked Flush 0 Sent Done Done false
27503 7884 Acked Write 2 Sent Done Done false
27504 7885 Acked Write 10 Sent Done Done false
27505 7886 Acked Read 7 Sent Done Done false
27506 7887 Acked Write 4 Sent Done Done false
27507 7888 Acked Flush 0 Sent Done Done false
27508 7889 Acked Read 10 Sent Done Done false
27509 7890 Acked Read 10 Sent Done Done false
27510 7891 Acked Write 7 Sent Done Done false
27511 7892 Acked Write 10 Sent Done Done false
27512 7893 Acked Write 6 Sent Done Done false
27513 7894 Acked Write 3 Sent Done Done false
27514 7895 Acked Write 1 Sent Done Done false
27515 7896 Acked Read 5 Sent Done Done false
27516 7897 Acked Read 7 Sent Done Done false
27517 7898 Acked Flush 0 Sent Done Done false
27518 7899 Acked Read 10 Sent Done Done false
27519 7900 Acked Write 1 Sent Done Done false
27520 7901 Acked Write 8 Sent Done Done false
27521 7902 Acked Read 3 Sent Done Done false
27522 7903 Acked Write 4 Sent Done Done false
27523 7904 Acked Read 8 Sent Done Done false
27524 7905 Acked Write 4 Sent Done Done false
27525 7906 Acked Read 7 Sent Done Done false
27526 7907 Acked Read 5 Sent Done Done false
27527 7908 Acked Write 1 Sent Done Done false
27528 7909 Acked Write 1 Sent Done Done false
27529 7910 Acked Write 10 Sent Done Done false
27530 7911 Acked Write 2 Sent Done Done false
27531 7912 Acked Write 1 Sent Done Done false
27532 7913 Acked Write 9 Sent Done Done false
27533 7914 Acked Read 6 Sent Done Done false
27534 7915 Acked Read 5 Sent Done Done false
27535 7916 Acked Write 9 Sent Done Done false
27536 7917 Acked Write 2 Sent Done Done false
27537 7918 Acked Read 2 Sent Done Done false
27538 7919 Acked Write 3 Sent Done Done false
27539 7920 Acked Write 5 Sent Done Done false
27540 7921 Acked Write 10 Sent Done Done false
27541 7922 Acked Read 4 Sent Done Done false
27542 7923 Acked Write 3 Sent Done Done false
27543 7924 Acked Read 5 Sent Done Done false
27544 7925 Acked Read 1 Sent Done Done false
27545 7926 Acked Write 4 Sent Done Done false
27546 7927 Acked Write 8 Sent Done Done false
27547 7928 Acked Write 5 Sent Done Done false
27548 7929 Acked Read 4 Sent Done Done false
27549 7930 Acked Read 6 Sent Done Done false
27550 7931 Acked Write 6 Sent Done Done false
27551 7932 Acked Read 5 Sent Done Done false
27552 7933 Acked Write 2 Sent Done Done false
27553 7934 Acked Write 3 Sent Done Done false
27554 7935 Acked Read 8 Sent Done Done false
27555 7936 Acked Write 6 Sent Done Done false
27556 7937 Acked Flush 0 Sent Done Done false
27557 7938 Acked Write 10 Sent Done Done false
27558 7939 Acked Read 7 Sent Done Done false
27559 7940 Acked Read 9 Sent Done Done false
27560 7941 Acked Read 3 Sent Done Done false
27561 STATES DS:0 DS:1 DS:2 TOTAL
27562 Sent 301 0 0 301
27563 Done 0 301 301 602
27564 Skipped 0 0 0 0
27565 Error 0 0 0 0
27566 Last Flush: Some(JobId(7640)) Some(JobId(7937)) Some(JobId(7937))
27567 Downstairs last five completed: 7640 7639 7638 7637 7636
27568 Upstairs last five completed: 7941 7940 7939 7938 7937
27569 CLIENT: Up:0 ds:301 act:2
275702025-02-28T06:07:06.138ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
275712025-02-28T06:07:06.951ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
275722025-02-28T06:07:08.143ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
275732025-02-28T06:07:08.955ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
27574 ----------------------------------------------------------------
27575 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27576 States: Offline Active Active
27577 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27578 7641 Acked Write 10 Sent Done Done false
27579 7642 Acked Read 9 Sent Done Done false
27580 7643 Acked Read 8 Sent Done Done false
27581 7644 Acked Write 3 Sent Done Done false
27582 7645 Acked Read 4 Sent Done Done false
27583 7646 Acked Flush 0 Sent Done Done false
27584 7647 Acked Write 4 Sent Done Done false
27585 7648 Acked Write 10 Sent Done Done false
27586 7649 Acked Read 1 Sent Done Done false
27587 7650 Acked Write 4 Sent Done Done false
27588 7651 Acked Read 10 Sent Done Done false
27589 7652 Acked Read 10 Sent Done Done false
27590 7653 Acked Read 9 Sent Done Done false
27591 7654 Acked Read 3 Sent Done Done false
27592 7655 Acked Write 6 Sent Done Done false
27593 7656 Acked Read 2 Sent Done Done false
27594 7657 Acked Read 7 Sent Done Done false
27595 7658 Acked Read 9 Sent Done Done false
27596 7659 Acked Write 8 Sent Done Done false
27597 7660 Acked Read 8 Sent Done Done false
27598 7661 Acked Flush 0 Sent Done Done false
27599 7662 Acked Flush 0 Sent Done Done false
27600 7663 Acked Write 1 Sent Done Done false
27601 7664 Acked Flush 0 Sent Done Done false
27602 7665 Acked Write 1 Sent Done Done false
27603 7666 Acked Flush 0 Sent Done Done false
27604 7667 Acked Write 5 Sent Done Done false
27605 7668 Acked Read 8 Sent Done Done false
27606 7669 Acked Read 9 Sent Done Done false
27607 7670 Acked Read 1 Sent Done Done false
27608 7671 Acked Read 3 Sent Done Done false
27609 7672 Acked Write 5 Sent Done Done false
27610 7673 Acked Write 10 Sent Done Done false
27611 7674 Acked Read 2 Sent Done Done false
27612 7675 Acked Flush 0 Sent Done Done false
27613 7676 Acked Read 2 Sent Done Done false
27614 7677 Acked Write 2 Sent Done Done false
27615 7678 Acked Read 10 Sent Done Done false
27616 7679 Acked Write 1 Sent Done Done false
27617 7680 Acked Read 4 Sent Done Done false
27618 7681 Acked Read 5 Sent Done Done false
27619 7682 Acked Write 10 Sent Done Done false
27620 7683 Acked Write 8 Sent Done Done false
27621 7684 Acked Write 1 Sent Done Done false
27622 7685 Acked Write 4 Sent Done Done false
27623 7686 Acked Write 2 Sent Done Done false
27624 7687 Acked Write 3 Sent Done Done false
27625 7688 Acked Read 1 Sent Done Done false
27626 7689 Acked Read 5 Sent Done Done false
27627 7690 Acked Read 10 Sent Done Done false
27628 7691 Acked Write 1 Sent Done Done false
27629 7692 Acked Flush 0 Sent Done Done false
27630 7693 Acked Read 5 Sent Done Done false
27631 7694 Acked Read 8 Sent Done Done false
27632 7695 Acked Read 9 Sent Done Done false
27633 7696 Acked Read 10 Sent Done Done false
27634 7697 Acked Write 4 Sent Done Done false
27635 7698 Acked Read 6 Sent Done Done false
27636 7699 Acked Read 4 Sent Done Done false
27637 7700 Acked Read 7 Sent Done Done false
27638 7701 Acked Write 5 Sent Done Done false
27639 7702 Acked Write 1 Sent Done Done false
27640 7703 Acked Flush 0 Sent Done Done false
27641 7704 Acked Read 7 Sent Done Done false
27642 7705 Acked Write 1 Sent Done Done false
27643 7706 Acked Read 1 Sent Done Done false
27644 7707 Acked Write 9 Sent Done Done false
27645 7708 Acked Flush 0 Sent Done Done false
27646 7709 Acked Read 1 Sent Done Done false
27647 7710 Acked Write 6 Sent Done Done false
27648 7711 Acked Read 3 Sent Done Done false
27649 7712 Acked Read 4 Sent Done Done false
27650 7713 Acked Write 7 Sent Done Done false
27651 7714 Acked Read 3 Sent Done Done false
27652 7715 Acked Write 2 Sent Done Done false
27653 7716 Acked Write 4 Sent Done Done false
27654 7717 Acked Write 7 Sent Done Done false
27655 7718 Acked Read 9 Sent Done Done false
27656 7719 Acked Read 2 Sent Done Done false
27657 7720 Acked Read 4 Sent Done Done false
27658 7721 Acked Write 5 Sent Done Done false
27659 7722 Acked Read 6 Sent Done Done false
27660 7723 Acked Write 4 Sent Done Done false
27661 7724 Acked Write 6 Sent Done Done false
27662 7725 Acked Write 2 Sent Done Done false
27663 7726 Acked Write 5 Sent Done Done false
27664 7727 Acked Read 7 Sent Done Done false
27665 7728 Acked Read 2 Sent Done Done false
27666 7729 Acked Write 2 Sent Done Done false
27667 7730 Acked Read 9 Sent Done Done false
27668 7731 Acked Read 5 Sent Done Done false
27669 7732 Acked Read 4 Sent Done Done false
27670 7733 Acked Write 5 Sent Done Done false
27671 7734 Acked Read 8 Sent Done Done false
27672 7735 Acked Write 5 Sent Done Done false
27673 7736 Acked Read 5 Sent Done Done false
27674 7737 Acked Read 4 Sent Done Done false
27675 7738 Acked Read 1 Sent Done Done false
27676 7739 Acked Read 6 Sent Done Done false
27677 7740 Acked Read 7 Sent Done Done false
27678 7741 Acked Read 1 Sent Done Done false
27679 7742 Acked Read 3 Sent Done Done false
27680 7743 Acked Read 7 Sent Done Done false
27681 7744 Acked Read 9 Sent Done Done false
27682 7745 Acked Write 7 Sent Done Done false
27683 7746 Acked Read 10 Sent Done Done false
27684 7747 Acked Write 3 Sent Done Done false
27685 7748 Acked Write 8 Sent Done Done false
27686 7749 Acked Read 8 Sent Done Done false
27687 7750 Acked Read 5 Sent Done Done false
27688 7751 Acked Write 3 Sent Done Done false
27689 7752 Acked Read 2 Sent Done Done false
27690 7753 Acked Flush 0 Sent Done Done false
27691 7754 Acked Read 5 Sent Done Done false
27692 7755 Acked Read 1 Sent Done Done false
27693 7756 Acked Write 9 Sent Done Done false
27694 7757 Acked Write 7 Sent Done Done false
27695 7758 Acked Write 4 Sent Done Done false
27696 7759 Acked Flush 0 Sent Done Done false
27697 7760 Acked Read 10 Sent Done Done false
27698 7761 Acked Write 7 Sent Done Done false
27699 7762 Acked Write 8 Sent Done Done false
27700 7763 Acked Flush 0 Sent Done Done false
27701 7764 Acked Read 8 Sent Done Done false
27702 7765 Acked Write 7 Sent Done Done false
27703 7766 Acked Read 5 Sent Done Done false
27704 7767 Acked Read 8 Sent Done Done false
27705 7768 Acked Write 2 Sent Done Done false
27706 7769 Acked Read 2 Sent Done Done false
27707 7770 Acked Read 3 Sent Done Done false
27708 7771 Acked Write 5 Sent Done Done false
27709 7772 Acked Read 9 Sent Done Done false
27710 7773 Acked Read 2 Sent Done Done false
27711 7774 Acked Read 1 Sent Done Done false
27712 7775 Acked Read 10 Sent Done Done false
27713 7776 Acked Read 8 Sent Done Done false
27714 7777 Acked Write 5 Sent Done Done false
27715 7778 Acked Write 6 Sent Done Done false
27716 7779 Acked Read 3 Sent Done Done false
27717 7780 Acked Read 3 Sent Done Done false
27718 7781 Acked Read 1 Sent Done Done false
27719 7782 Acked Write 5 Sent Done Done false
27720 7783 Acked Write 8 Sent Done Done false
27721 7784 Acked Read 1 Sent Done Done false
27722 7785 Acked Write 6 Sent Done Done false
27723 7786 Acked Write 2 Sent Done Done false
27724 7787 Acked Write 8 Sent Done Done false
27725 7788 Acked Write 5 Sent Done Done false
27726 7789 Acked Read 4 Sent Done Done false
27727 7790 Acked Read 9 Sent Done Done false
27728 7791 Acked Read 4 Sent Done Done false
27729 7792 Acked Read 5 Sent Done Done false
27730 7793 Acked Read 7 Sent Done Done false
27731 7794 Acked Flush 0 Sent Done Done false
27732 7795 Acked Write 9 Sent Done Done false
27733 7796 Acked Read 10 Sent Done Done false
27734 7797 Acked Read 3 Sent Done Done false
27735 7798 Acked Read 1 Sent Done Done false
27736 7799 Acked Read 10 Sent Done Done false
27737 7800 Acked Write 5 Sent Done Done false
27738 7801 Acked Read 1 Sent Done Done false
27739 7802 Acked Flush 0 Sent Done Done false
27740 7803 Acked Read 9 Sent Done Done false
27741 7804 Acked Write 6 Sent Done Done false
27742 7805 Acked Read 1 Sent Done Done false
27743 7806 Acked Read 3 Sent Done Done false
27744 7807 Acked Write 6 Sent Done Done false
27745 7808 Acked Write 8 Sent Done Done false
27746 7809 Acked Read 7 Sent Done Done false
27747 7810 Acked Write 1 Sent Done Done false
27748 7811 Acked Read 7 Sent Done Done false
27749 7812 Acked Read 10 Sent Done Done false
27750 7813 Acked Read 7 Sent Done Done false
27751 7814 Acked Read 1 Sent Done Done false
27752 7815 Acked Write 7 Sent Done Done false
27753 7816 Acked Write 6 Sent Done Done false
27754 7817 Acked Write 8 Sent Done Done false
27755 7818 Acked Read 3 Sent Done Done false
27756 7819 Acked Write 10 Sent Done Done false
27757 7820 Acked Write 10 Sent Done Done false
27758 7821 Acked Read 6 Sent Done Done false
27759 7822 Acked Write 10 Sent Done Done false
27760 7823 Acked Write 1 Sent Done Done false
27761 7824 Acked Flush 0 Sent Done Done false
27762 7825 Acked Read 7 Sent Done Done false
27763 7826 Acked Read 7 Sent Done Done false
27764 7827 Acked Write 10 Sent Done Done false
27765 7828 Acked Flush 0 Sent Done Done false
27766 7829 Acked Read 9 Sent Done Done false
27767 7830 Acked Write 9 Sent Done Done false
27768 7831 Acked Write 9 Sent Done Done false
27769 7832 Acked Write 10 Sent Done Done false
27770 7833 Acked Write 4 Sent Done Done false
27771 7834 Acked Read 4 Sent Done Done false
27772 7835 Acked Read 4 Sent Done Done false
27773 7836 Acked Write 3 Sent Done Done false
27774 7837 Acked Read 10 Sent Done Done false
27775 7838 Acked Write 4 Sent Done Done false
27776 7839 Acked Read 1 Sent Done Done false
27777 7840 Acked Write 1 Sent Done Done false
27778 7841 Acked Write 4 Sent Done Done false
27779 7842 Acked Write 6 Sent Done Done false
27780 7843 Acked Write 4 Sent Done Done false
27781 7844 Acked Write 8 Sent Done Done false
27782 7845 Acked Write 10 Sent Done Done false
27783 7846 Acked Read 9 Sent Done Done false
27784 7847 Acked Write 6 Sent Done Done false
27785 7848 Acked Read 7 Sent Done Done false
27786 7849 Acked Read 6 Sent Done Done false
27787 7850 Acked Flush 0 Sent Done Done false
27788 7851 Acked Read 7 Sent Done Done false
27789 7852 Acked Write 2 Sent Done Done false
27790 7853 Acked Read 2 Sent Done Done false
27791 7854 Acked Write 8 Sent Done Done false
27792 7855 Acked Write 9 Sent Done Done false
27793 7856 Acked Write 2 Sent Done Done false
27794 7857 Acked Read 9 Sent Done Done false
27795 7858 Acked Write 7 Sent Done Done false
27796 7859 Acked Write 5 Sent Done Done false
27797 7860 Acked Flush 0 Sent Done Done false
27798 7861 Acked Read 3 Sent Done Done false
27799 7862 Acked Flush 0 Sent Done Done false
27800 7863 Acked Write 7 Sent Done Done false
27801 7864 Acked Write 9 Sent Done Done false
27802 7865 Acked Write 3 Sent Done Done false
27803 7866 Acked Flush 0 Sent Done Done false
27804 7867 Acked Read 5 Sent Done Done false
27805 7868 Acked Write 10 Sent Done Done false
27806 7869 Acked Write 3 Sent Done Done false
27807 7870 Acked Read 1 Sent Done Done false
27808 7871 Acked Read 9 Sent Done Done false
27809 7872 Acked Read 10 Sent Done Done false
27810 7873 Acked Read 1 Sent Done Done false
27811 7874 Acked Write 1 Sent Done Done false
27812 7875 Acked Write 7 Sent Done Done false
27813 7876 Acked Read 8 Sent Done Done false
27814 7877 Acked Write 9 Sent Done Done false
27815 7878 Acked Read 5 Sent Done Done false
27816 7879 Acked Write 4 Sent Done Done false
27817 7880 Acked Write 4 Sent Done Done false
27818 7881 Acked Write 1 Sent Done Done false
27819 7882 Acked Read 9 Sent Done Done false
27820 7883 Acked Flush 0 Sent Done Done false
27821 7884 Acked Write 2 Sent Done Done false
27822 7885 Acked Write 10 Sent Done Done false
27823 7886 Acked Read 7 Sent Done Done false
27824 7887 Acked Write 4 Sent Done Done false
27825 7888 Acked Flush 0 Sent Done Done false
27826 7889 Acked Read 10 Sent Done Done false
27827 7890 Acked Read 10 Sent Done Done false
27828 7891 Acked Write 7 Sent Done Done false
27829 7892 Acked Write 10 Sent Done Done false
27830 7893 Acked Write 6 Sent Done Done false
27831 7894 Acked Write 3 Sent Done Done false
27832 7895 Acked Write 1 Sent Done Done false
27833 7896 Acked Read 5 Sent Done Done false
27834 7897 Acked Read 7 Sent Done Done false
27835 7898 Acked Flush 0 Sent Done Done false
27836 7899 Acked Read 10 Sent Done Done false
27837 7900 Acked Write 1 Sent Done Done false
27838 7901 Acked Write 8 Sent Done Done false
27839 7902 Acked Read 3 Sent Done Done false
27840 7903 Acked Write 4 Sent Done Done false
27841 7904 Acked Read 8 Sent Done Done false
27842 7905 Acked Write 4 Sent Done Done false
27843 7906 Acked Read 7 Sent Done Done false
27844 7907 Acked Read 5 Sent Done Done false
27845 7908 Acked Write 1 Sent Done Done false
27846 7909 Acked Write 1 Sent Done Done false
27847 7910 Acked Write 10 Sent Done Done false
27848 7911 Acked Write 2 Sent Done Done false
27849 7912 Acked Write 1 Sent Done Done false
27850 7913 Acked Write 9 Sent Done Done false
27851 7914 Acked Read 6 Sent Done Done false
27852 7915 Acked Read 5 Sent Done Done false
27853 7916 Acked Write 9 Sent Done Done false
27854 7917 Acked Write 2 Sent Done Done false
27855 7918 Acked Read 2 Sent Done Done false
27856 7919 Acked Write 3 Sent Done Done false
27857 7920 Acked Write 5 Sent Done Done false
27858 7921 Acked Write 10 Sent Done Done false
27859 7922 Acked Read 4 Sent Done Done false
27860 7923 Acked Write 3 Sent Done Done false
27861 7924 Acked Read 5 Sent Done Done false
27862 7925 Acked Read 1 Sent Done Done false
27863 7926 Acked Write 4 Sent Done Done false
27864 7927 Acked Write 8 Sent Done Done false
27865 7928 Acked Write 5 Sent Done Done false
27866 7929 Acked Read 4 Sent Done Done false
27867 7930 Acked Read 6 Sent Done Done false
27868 7931 Acked Write 6 Sent Done Done false
27869 7932 Acked Read 5 Sent Done Done false
27870 7933 Acked Write 2 Sent Done Done false
27871 7934 Acked Write 3 Sent Done Done false
27872 7935 Acked Read 8 Sent Done Done false
27873 7936 Acked Write 6 Sent Done Done false
27874 7937 Acked Flush 0 Sent Done Done false
27875 7938 Acked Write 10 Sent Done Done false
27876 7939 Acked Read 7 Sent Done Done false
27877 7940 Acked Read 9 Sent Done Done false
27878 7941 Acked Read 3 Sent Done Done false
27879 STATES DS:0 DS:1 DS:2 TOTAL
27880 Sent 301 0 0 301
27881 Done 0 301 301 602
27882 Skipped 0 0 0 0
27883 Error 0 0 0 0
27884 Last Flush: Some(JobId(7640)) Some(JobId(7937)) Some(JobId(7937))
27885 Downstairs last five completed: 7640 7639 7638 7637 7636
27886 Upstairs last five completed: 7941 7940 7939 7938 7937
27887 CLIENT: Up:0 ds:301 act:2
278882025-02-28T06:07:10.139ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
278892025-02-28T06:07:10.958ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
278902025-02-28T06:07:11.397ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
278912025-02-28T06:07:11.397ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
278922025-02-28T06:07:11.397ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
278932025-02-28T06:07:11.397ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
278942025-02-28T06:07:11.397ZINFOcrucible: client task is exiting = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
278952025-02-28T06:07:11.960ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
278962025-02-28T06:07:13.144ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
27897 [8810] got start action
27898 [8810] I am going to restart
27899 Starting downstairs at port 8810
27900 Make output file at "/tmp/dsc/downstairs-8810.txt"
27901 [8810][0] reports Starting
27902 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1234)
27903 [8810][0] reports Running
27904 ----------------------------------------------------------------
27905 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27906 States: Offline Active Active
27907 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27908 7641 Acked Write 10 Sent Done Done false
27909 7642 Acked Read 9 Sent Done Done false
27910 7643 Acked Read 8 Sent Done Done false
27911 7644 Acked Write 3 Sent Done Done false
27912 7645 Acked Read 4 Sent Done Done false
27913 7646 Acked Flush 0 Sent Done Done false
27914 7647 Acked Write 4 Sent Done Done false
27915 7648 Acked Write 10 Sent Done Done false
27916 7649 Acked Read 1 Sent Done Done false
27917 7650 Acked Write 4 Sent Done Done false
27918 7651 Acked Read 10 Sent Done Done false
27919 7652 Acked Read 10 Sent Done Done false
27920 7653 Acked Read 9 Sent Done Done false
27921 7654 Acked Read 3 Sent Done Done false
27922 7655 Acked Write 6 Sent Done Done false
27923 7656 Acked Read 2 Sent Done Done false
27924 7657 Acked Read 7 Sent Done Done false
27925 7658 Acked Read 9 Sent Done Done false
27926 7659 Acked Write 8 Sent Done Done false
27927 7660 Acked Read 8 Sent Done Done false
27928 7661 Acked Flush 0 Sent Done Done false
27929 7662 Acked Flush 0 Sent Done Done false
27930 7663 Acked Write 1 Sent Done Done false
27931 7664 Acked Flush 0 Sent Done Done false
27932 7665 Acked Write 1 Sent Done Done false
27933 7666 Acked Flush 0 Sent Done Done false
27934 7667 Acked Write 5 Sent Done Done false
27935 7668 Acked Read 8 Sent Done Done false
27936 7669 Acked Read 9 Sent Done Done false
27937 7670 Acked Read 1 Sent Done Done false
27938 7671 Acked Read 3 Sent Done Done false
27939 7672 Acked Write 5 Sent Done Done false
27940 7673 Acked Write 10 Sent Done Done false
27941 7674 Acked Read 2 Sent Done Done false
27942 7675 Acked Flush 0 Sent Done Done false
27943 7676 Acked Read 2 Sent Done Done false
27944 7677 Acked Write 2 Sent Done Done false
27945 7678 Acked Read 10 Sent Done Done false
27946 7679 Acked Write 1 Sent Done Done false
27947 7680 Acked Read 4 Sent Done Done false
27948 7681 Acked Read 5 Sent Done Done false
27949 7682 Acked Write 10 Sent Done Done false
27950 7683 Acked Write 8 Sent Done Done false
27951 7684 Acked Write 1 Sent Done Done false
27952 7685 Acked Write 4 Sent Done Done false
27953 7686 Acked Write 2 Sent Done Done false
27954 7687 Acked Write 3 Sent Done Done false
27955 7688 Acked Read 1 Sent Done Done false
27956 7689 Acked Read 5 Sent Done Done false
27957 7690 Acked Read 10 Sent Done Done false
27958 7691 Acked Write 1 Sent Done Done false
27959 7692 Acked Flush 0 Sent Done Done false
27960 7693 Acked Read 5 Sent Done Done false
27961 7694 Acked Read 8 Sent Done Done false
27962 7695 Acked Read 9 Sent Done Done false
27963 7696 Acked Read 10 Sent Done Done false
27964 7697 Acked Write 4 Sent Done Done false
27965 7698 Acked Read 6 Sent Done Done false
27966 7699 Acked Read 4 Sent Done Done false
27967 7700 Acked Read 7 Sent Done Done false
27968 7701 Acked Write 5 Sent Done Done false
27969 7702 Acked Write 1 Sent Done Done false
27970 7703 Acked Flush 0 Sent Done Done false
27971 7704 Acked Read 7 Sent Done Done false
27972 7705 Acked Write 1 Sent Done Done false
27973 7706 Acked Read 1 Sent Done Done false
27974 7707 Acked Write 9 Sent Done Done false
27975 7708 Acked Flush 0 Sent Done Done false
27976 7709 Acked Read 1 Sent Done Done false
27977 7710 Acked Write 6 Sent Done Done false
27978 7711 Acked Read 3 Sent Done Done false
27979 7712 Acked Read 4 Sent Done Done false
27980 7713 Acked Write 7 Sent Done Done false
27981 7714 Acked Read 3 Sent Done Done false
27982 7715 Acked Write 2 Sent Done Done false
27983 7716 Acked Write 4 Sent Done Done false
27984 7717 Acked Write 7 Sent Done Done false
27985 7718 Acked Read 9 Sent Done Done false
27986 7719 Acked Read 2 Sent Done Done false
27987 7720 Acked Read 4 Sent Done Done false
27988 7721 Acked Write 5 Sent Done Done false
27989 7722 Acked Read 6 Sent Done Done false
27990 7723 Acked Write 4 Sent Done Done false
27991 7724 Acked Write 6 Sent Done Done false
27992 7725 Acked Write 2 Sent Done Done false
27993 7726 Acked Write 5 Sent Done Done false
27994 7727 Acked Read 7 Sent Done Done false
27995 7728 Acked Read 2 Sent Done Done false
27996 7729 Acked Write 2 Sent Done Done false
27997 7730 Acked Read 9 Sent Done Done false
27998 7731 Acked Read 5 Sent Done Done false
27999 7732 Acked Read 4 Sent Done Done false
28000 7733 Acked Write 5 Sent Done Done false
28001 7734 Acked Read 8 Sent Done Done false
28002 7735 Acked Write 5 Sent Done Done false
28003 7736 Acked Read 5 Sent Done Done false
28004 7737 Acked Read 4 Sent Done Done false
28005 7738 Acked Read 1 Sent Done Done false
28006 7739 Acked Read 6 Sent Done Done false
28007 7740 Acked Read 7 Sent Done Done false
28008 7741 Acked Read 1 Sent Done Done false
28009 7742 Acked Read 3 Sent Done Done false
28010 7743 Acked Read 7 Sent Done Done false
28011 7744 Acked Read 9 Sent Done Done false
28012 7745 Acked Write 7 Sent Done Done false
28013 7746 Acked Read 10 Sent Done Done false
28014 7747 Acked Write 3 Sent Done Done false
28015 7748 Acked Write 8 Sent Done Done false
28016 7749 Acked Read 8 Sent Done Done false
28017 7750 Acked Read 5 Sent Done Done false
28018 7751 Acked Write 3 Sent Done Done false
28019 7752 Acked Read 2 Sent Done Done false
28020 7753 Acked Flush 0 Sent Done Done false
28021 7754 Acked Read 5 Sent Done Done false
28022 7755 Acked Read 1 Sent Done Done false
28023 7756 Acked Write 9 Sent Done Done false
28024 7757 Acked Write 7 Sent Done Done false
28025 7758 Acked Write 4 Sent Done Done false
28026 7759 Acked Flush 0 Sent Done Done false
28027 7760 Acked Read 10 Sent Done Done false
28028 7761 Acked Write 7 Sent Done Done false
28029 7762 Acked Write 8 Sent Done Done false
28030 7763 Acked Flush 0 Sent Done Done false
28031 7764 Acked Read 8 Sent Done Done false
28032 7765 Acked Write 7 Sent Done Done false
28033 7766 Acked Read 5 Sent Done Done false
28034 7767 Acked Read 8 Sent Done Done false
28035 7768 Acked Write 2 Sent Done Done false
28036 7769 Acked Read 2 Sent Done Done false
28037 7770 Acked Read 3 Sent Done Done false
28038 7771 Acked Write 5 Sent Done Done false
28039 7772 Acked Read 9 Sent Done Done false
28040 7773 Acked Read 2 Sent Done Done false
28041 7774 Acked Read 1 Sent Done Done false
28042 7775 Acked Read 10 Sent Done Done false
28043 7776 Acked Read 8 Sent Done Done false
28044 7777 Acked Write 5 Sent Done Done false
28045 7778 Acked Write 6 Sent Done Done false
28046 7779 Acked Read 3 Sent Done Done false
28047 7780 Acked Read 3 Sent Done Done false
28048 7781 Acked Read 1 Sent Done Done false
28049 7782 Acked Write 5 Sent Done Done false
28050 7783 Acked Write 8 Sent Done Done false
28051 7784 Acked Read 1 Sent Done Done false
28052 7785 Acked Write 6 Sent Done Done false
28053 7786 Acked Write 2 Sent Done Done false
28054 7787 Acked Write 8 Sent Done Done false
28055 7788 Acked Write 5 Sent Done Done false
28056 7789 Acked Read 4 Sent Done Done false
28057 7790 Acked Read 9 Sent Done Done false
28058 7791 Acked Read 4 Sent Done Done false
28059 7792 Acked Read 5 Sent Done Done false
28060 7793 Acked Read 7 Sent Done Done false
28061 7794 Acked Flush 0 Sent Done Done false
28062 7795 Acked Write 9 Sent Done Done false
28063 7796 Acked Read 10 Sent Done Done false
28064 7797 Acked Read 3 Sent Done Done false
28065 7798 Acked Read 1 Sent Done Done false
28066 7799 Acked Read 10 Sent Done Done false
28067 7800 Acked Write 5 Sent Done Done false
28068 7801 Acked Read 1 Sent Done Done false
28069 7802 Acked Flush 0 Sent Done Done false
28070 7803 Acked Read 9 Sent Done Done false
28071 7804 Acked Write 6 Sent Done Done false
28072 7805 Acked Read 1 Sent Done Done false
28073 7806 Acked Read 3 Sent Done Done false
28074 7807 Acked Write 6 Sent Done Done false
28075 7808 Acked Write 8 Sent Done Done false
28076 7809 Acked Read 7 Sent Done Done false
28077 7810 Acked Write 1 Sent Done Done false
28078 7811 Acked Read 7 Sent Done Done false
28079 7812 Acked Read 10 Sent Done Done false
28080 7813 Acked Read 7 Sent Done Done false
28081 7814 Acked Read 1 Sent Done Done false
28082 7815 Acked Write 7 Sent Done Done false
28083 7816 Acked Write 6 Sent Done Done false
28084 7817 Acked Write 8 Sent Done Done false
28085 7818 Acked Read 3 Sent Done Done false
28086 7819 Acked Write 10 Sent Done Done false
28087 7820 Acked Write 10 Sent Done Done false
28088 7821 Acked Read 6 Sent Done Done false
28089 7822 Acked Write 10 Sent Done Done false
28090 7823 Acked Write 1 Sent Done Done false
28091 7824 Acked Flush 0 Sent Done Done false
28092 7825 Acked Read 7 Sent Done Done false
28093 7826 Acked Read 7 Sent Done Done false
28094 7827 Acked Write 10 Sent Done Done false
28095 7828 Acked Flush 0 Sent Done Done false
28096 7829 Acked Read 9 Sent Done Done false
28097 7830 Acked Write 9 Sent Done Done false
28098 7831 Acked Write 9 Sent Done Done false
28099 7832 Acked Write 10 Sent Done Done false
28100 7833 Acked Write 4 Sent Done Done false
28101 7834 Acked Read 4 Sent Done Done false
28102 7835 Acked Read 4 Sent Done Done false
28103 7836 Acked Write 3 Sent Done Done false
28104 7837 Acked Read 10 Sent Done Done false
28105 7838 Acked Write 4 Sent Done Done false
28106 7839 Acked Read 1 Sent Done Done false
28107 7840 Acked Write 1 Sent Done Done false
28108 7841 Acked Write 4 Sent Done Done false
28109 7842 Acked Write 6 Sent Done Done false
28110 7843 Acked Write 4 Sent Done Done false
28111 7844 Acked Write 8 Sent Done Done false
28112 7845 Acked Write 10 Sent Done Done false
28113 7846 Acked Read 9 Sent Done Done false
28114 7847 Acked Write 6 Sent Done Done false
28115 7848 Acked Read 7 Sent Done Done false
28116 7849 Acked Read 6 Sent Done Done false
28117 7850 Acked Flush 0 Sent Done Done false
28118 7851 Acked Read 7 Sent Done Done false
28119 7852 Acked Write 2 Sent Done Done false
28120 7853 Acked Read 2 Sent Done Done false
28121 7854 Acked Write 8 Sent Done Done false
28122 7855 Acked Write 9 Sent Done Done false
28123 7856 Acked Write 2 Sent Done Done false
28124 7857 Acked Read 9 Sent Done Done false
28125 7858 Acked Write 7 Sent Done Done false
28126 7859 Acked Write 5 Sent Done Done false
28127 7860 Acked Flush 0 Sent Done Done false
28128 7861 Acked Read 3 Sent Done Done false
28129 7862 Acked Flush 0 Sent Done Done false
28130 7863 Acked Write 7 Sent Done Done false
28131 7864 Acked Write 9 Sent Done Done false
28132 7865 Acked Write 3 Sent Done Done false
28133 7866 Acked Flush 0 Sent Done Done false
28134 7867 Acked Read 5 Sent Done Done false
28135 7868 Acked Write 10 Sent Done Done false
28136 7869 Acked Write 3 Sent Done Done false
28137 7870 Acked Read 1 Sent Done Done false
28138 7871 Acked Read 9 Sent Done Done false
28139 7872 Acked Read 10 Sent Done Done false
28140 7873 Acked Read 1 Sent Done Done false
28141 7874 Acked Write 1 Sent Done Done false
28142 7875 Acked Write 7 Sent Done Done false
28143 7876 Acked Read 8 Sent Done Done false
28144 7877 Acked Write 9 Sent Done Done false
28145 7878 Acked Read 5 Sent Done Done false
28146 7879 Acked Write 4 Sent Done Done false
28147 7880 Acked Write 4 Sent Done Done false
28148 7881 Acked Write 1 Sent Done Done false
28149 7882 Acked Read 9 Sent Done Done false
28150 7883 Acked Flush 0 Sent Done Done false
28151 7884 Acked Write 2 Sent Done Done false
28152 7885 Acked Write 10 Sent Done Done false
28153 7886 Acked Read 7 Sent Done Done false
28154 7887 Acked Write 4 Sent Done Done false
28155 7888 Acked Flush 0 Sent Done Done false
28156 7889 Acked Read 10 Sent Done Done false
28157 7890 Acked Read 10 Sent Done Done false
28158 7891 Acked Write 7 Sent Done Done false
28159 7892 Acked Write 10 Sent Done Done false
28160 7893 Acked Write 6 Sent Done Done false
28161 7894 Acked Write 3 Sent Done Done false
28162 7895 Acked Write 1 Sent Done Done false
28163 7896 Acked Read 5 Sent Done Done false
28164 7897 Acked Read 7 Sent Done Done false
28165 7898 Acked Flush 0 Sent Done Done false
28166 7899 Acked Read 10 Sent Done Done false
28167 7900 Acked Write 1 Sent Done Done false
28168 7901 Acked Write 8 Sent Done Done false
28169 7902 Acked Read 3 Sent Done Done false
28170 7903 Acked Write 4 Sent Done Done false
28171 7904 Acked Read 8 Sent Done Done false
28172 7905 Acked Write 4 Sent Done Done false
28173 7906 Acked Read 7 Sent Done Done false
28174 7907 Acked Read 5 Sent Done Done false
28175 7908 Acked Write 1 Sent Done Done false
28176 7909 Acked Write 1 Sent Done Done false
28177 7910 Acked Write 10 Sent Done Done false
28178 7911 Acked Write 2 Sent Done Done false
28179 7912 Acked Write 1 Sent Done Done false
28180 7913 Acked Write 9 Sent Done Done false
28181 7914 Acked Read 6 Sent Done Done false
28182 7915 Acked Read 5 Sent Done Done false
28183 7916 Acked Write 9 Sent Done Done false
28184 7917 Acked Write 2 Sent Done Done false
28185 7918 Acked Read 2 Sent Done Done false
28186 7919 Acked Write 3 Sent Done Done false
28187 7920 Acked Write 5 Sent Done Done false
28188 7921 Acked Write 10 Sent Done Done false
28189 7922 Acked Read 4 Sent Done Done false
28190 7923 Acked Write 3 Sent Done Done false
28191 7924 Acked Read 5 Sent Done Done false
28192 7925 Acked Read 1 Sent Done Done false
28193 7926 Acked Write 4 Sent Done Done false
28194 7927 Acked Write 8 Sent Done Done false
28195 7928 Acked Write 5 Sent Done Done false
28196 7929 Acked Read 4 Sent Done Done false
28197 7930 Acked Read 6 Sent Done Done false
28198 7931 Acked Write 6 Sent Done Done false
28199 7932 Acked Read 5 Sent Done Done false
28200 7933 Acked Write 2 Sent Done Done false
28201 7934 Acked Write 3 Sent Done Done false
28202 7935 Acked Read 8 Sent Done Done false
28203 7936 Acked Write 6 Sent Done Done false
28204 7937 Acked Flush 0 Sent Done Done false
28205 7938 Acked Write 10 Sent Done Done false
28206 7939 Acked Read 7 Sent Done Done false
28207 7940 Acked Read 9 Sent Done Done false
28208 7941 Acked Read 3 Sent Done Done false
28209 STATES DS:0 DS:1 DS:2 TOTAL
28210 Sent 301 0 0 301
28211 Done 0 301 301 602
28212 Skipped 0 0 0 0
28213 Error 0 0 0 0
28214 Last Flush: Some(JobId(7640)) Some(JobId(7937)) Some(JobId(7937))
28215 Downstairs last five completed: 7640 7639 7638 7637 7636
28216 Upstairs last five completed: 7941 7940 7939 7938 7937
28217 CLIENT: Up:0 ds:301 act:2
282182025-02-28T06:07:14.142ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
282192025-02-28T06:07:14.965ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
282202025-02-28T06:07:15.965ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
282212025-02-28T06:07:16.968ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
28222 ----------------------------------------------------------------
28223 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28224 States: Offline Active Active
28225 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28226 7641 Acked Write 10 Sent Done Done false
28227 7642 Acked Read 9 Sent Done Done false
28228 7643 Acked Read 8 Sent Done Done false
28229 7644 Acked Write 3 Sent Done Done false
28230 7645 Acked Read 4 Sent Done Done false
28231 7646 Acked Flush 0 Sent Done Done false
28232 7647 Acked Write 4 Sent Done Done false
28233 7648 Acked Write 10 Sent Done Done false
28234 7649 Acked Read 1 Sent Done Done false
28235 7650 Acked Write 4 Sent Done Done false
28236 7651 Acked Read 10 Sent Done Done false
28237 7652 Acked Read 10 Sent Done Done false
28238 7653 Acked Read 9 Sent Done Done false
28239 7654 Acked Read 3 Sent Done Done false
28240 7655 Acked Write 6 Sent Done Done false
28241 7656 Acked Read 2 Sent Done Done false
28242 7657 Acked Read 7 Sent Done Done false
28243 7658 Acked Read 9 Sent Done Done false
28244 7659 Acked Write 8 Sent Done Done false
28245 7660 Acked Read 8 Sent Done Done false
28246 7661 Acked Flush 0 Sent Done Done false
28247 7662 Acked Flush 0 Sent Done Done false
28248 7663 Acked Write 1 Sent Done Done false
28249 7664 Acked Flush 0 Sent Done Done false
28250 7665 Acked Write 1 Sent Done Done false
28251 7666 Acked Flush 0 Sent Done Done false
28252 7667 Acked Write 5 Sent Done Done false
28253 7668 Acked Read 8 Sent Done Done false
28254 7669 Acked Read 9 Sent Done Done false
28255 7670 Acked Read 1 Sent Done Done false
28256 7671 Acked Read 3 Sent Done Done false
28257 7672 Acked Write 5 Sent Done Done false
28258 7673 Acked Write 10 Sent Done Done false
28259 7674 Acked Read 2 Sent Done Done false
28260 7675 Acked Flush 0 Sent Done Done false
28261 7676 Acked Read 2 Sent Done Done false
28262 7677 Acked Write 2 Sent Done Done false
28263 7678 Acked Read 10 Sent Done Done false
28264 7679 Acked Write 1 Sent Done Done false
28265 7680 Acked Read 4 Sent Done Done false
28266 7681 Acked Read 5 Sent Done Done false
28267 7682 Acked Write 10 Sent Done Done false
28268 7683 Acked Write 8 Sent Done Done false
28269 7684 Acked Write 1 Sent Done Done false
28270 7685 Acked Write 4 Sent Done Done false
28271 7686 Acked Write 2 Sent Done Done false
28272 7687 Acked Write 3 Sent Done Done false
28273 7688 Acked Read 1 Sent Done Done false
28274 7689 Acked Read 5 Sent Done Done false
28275 7690 Acked Read 10 Sent Done Done false
28276 7691 Acked Write 1 Sent Done Done false
28277 7692 Acked Flush 0 Sent Done Done false
28278 7693 Acked Read 5 Sent Done Done false
28279 7694 Acked Read 8 Sent Done Done false
28280 7695 Acked Read 9 Sent Done Done false
28281 7696 Acked Read 10 Sent Done Done false
28282 7697 Acked Write 4 Sent Done Done false
28283 7698 Acked Read 6 Sent Done Done false
28284 7699 Acked Read 4 Sent Done Done false
28285 7700 Acked Read 7 Sent Done Done false
28286 7701 Acked Write 5 Sent Done Done false
28287 7702 Acked Write 1 Sent Done Done false
28288 7703 Acked Flush 0 Sent Done Done false
28289 7704 Acked Read 7 Sent Done Done false
28290 7705 Acked Write 1 Sent Done Done false
28291 7706 Acked Read 1 Sent Done Done false
28292 7707 Acked Write 9 Sent Done Done false
28293 7708 Acked Flush 0 Sent Done Done false
28294 7709 Acked Read 1 Sent Done Done false
28295 7710 Acked Write 6 Sent Done Done false
28296 7711 Acked Read 3 Sent Done Done false
28297 7712 Acked Read 4 Sent Done Done false
28298 7713 Acked Write 7 Sent Done Done false
28299 7714 Acked Read 3 Sent Done Done false
28300 7715 Acked Write 2 Sent Done Done false
28301 7716 Acked Write 4 Sent Done Done false
28302 7717 Acked Write 7 Sent Done Done false
28303 7718 Acked Read 9 Sent Done Done false
28304 7719 Acked Read 2 Sent Done Done false
28305 7720 Acked Read 4 Sent Done Done false
28306 7721 Acked Write 5 Sent Done Done false
28307 7722 Acked Read 6 Sent Done Done false
28308 7723 Acked Write 4 Sent Done Done false
28309 7724 Acked Write 6 Sent Done Done false
28310 7725 Acked Write 2 Sent Done Done false
28311 7726 Acked Write 5 Sent Done Done false
28312 7727 Acked Read 7 Sent Done Done false
28313 7728 Acked Read 2 Sent Done Done false
28314 7729 Acked Write 2 Sent Done Done false
28315 7730 Acked Read 9 Sent Done Done false
28316 7731 Acked Read 5 Sent Done Done false
28317 7732 Acked Read 4 Sent Done Done false
28318 7733 Acked Write 5 Sent Done Done false
28319 7734 Acked Read 8 Sent Done Done false
28320 7735 Acked Write 5 Sent Done Done false
28321 7736 Acked Read 5 Sent Done Done false
28322 7737 Acked Read 4 Sent Done Done false
28323 7738 Acked Read 1 Sent Done Done false
28324 7739 Acked Read 6 Sent Done Done false
28325 7740 Acked Read 7 Sent Done Done false
28326 7741 Acked Read 1 Sent Done Done false
28327 7742 Acked Read 3 Sent Done Done false
28328 7743 Acked Read 7 Sent Done Done false
28329 7744 Acked Read 9 Sent Done Done false
28330 7745 Acked Write 7 Sent Done Done false
28331 7746 Acked Read 10 Sent Done Done false
28332 7747 Acked Write 3 Sent Done Done false
28333 7748 Acked Write 8 Sent Done Done false
28334 7749 Acked Read 8 Sent Done Done false
28335 7750 Acked Read 5 Sent Done Done false
28336 7751 Acked Write 3 Sent Done Done false
28337 7752 Acked Read 2 Sent Done Done false
28338 7753 Acked Flush 0 Sent Done Done false
28339 7754 Acked Read 5 Sent Done Done false
28340 7755 Acked Read 1 Sent Done Done false
28341 7756 Acked Write 9 Sent Done Done false
28342 7757 Acked Write 7 Sent Done Done false
28343 7758 Acked Write 4 Sent Done Done false
28344 7759 Acked Flush 0 Sent Done Done false
28345 7760 Acked Read 10 Sent Done Done false
28346 7761 Acked Write 7 Sent Done Done false
28347 7762 Acked Write 8 Sent Done Done false
28348 7763 Acked Flush 0 Sent Done Done false
28349 7764 Acked Read 8 Sent Done Done false
28350 7765 Acked Write 7 Sent Done Done false
28351 7766 Acked Read 5 Sent Done Done false
28352 7767 Acked Read 8 Sent Done Done false
28353 7768 Acked Write 2 Sent Done Done false
28354 7769 Acked Read 2 Sent Done Done false
28355 7770 Acked Read 3 Sent Done Done false
28356 7771 Acked Write 5 Sent Done Done false
28357 7772 Acked Read 9 Sent Done Done false
28358 7773 Acked Read 2 Sent Done Done false
28359 7774 Acked Read 1 Sent Done Done false
28360 7775 Acked Read 10 Sent Done Done false
28361 7776 Acked Read 8 Sent Done Done false
28362 7777 Acked Write 5 Sent Done Done false
28363 7778 Acked Write 6 Sent Done Done false
28364 7779 Acked Read 3 Sent Done Done false
28365 7780 Acked Read 3 Sent Done Done false
28366 7781 Acked Read 1 Sent Done Done false
28367 7782 Acked Write 5 Sent Done Done false
28368 7783 Acked Write 8 Sent Done Done false
28369 7784 Acked Read 1 Sent Done Done false
28370 7785 Acked Write 6 Sent Done Done false
28371 7786 Acked Write 2 Sent Done Done false
28372 7787 Acked Write 8 Sent Done Done false
28373 7788 Acked Write 5 Sent Done Done false
28374 7789 Acked Read 4 Sent Done Done false
28375 7790 Acked Read 9 Sent Done Done false
28376 7791 Acked Read 4 Sent Done Done false
28377 7792 Acked Read 5 Sent Done Done false
28378 7793 Acked Read 7 Sent Done Done false
28379 7794 Acked Flush 0 Sent Done Done false
28380 7795 Acked Write 9 Sent Done Done false
28381 7796 Acked Read 10 Sent Done Done false
28382 7797 Acked Read 3 Sent Done Done false
28383 7798 Acked Read 1 Sent Done Done false
28384 7799 Acked Read 10 Sent Done Done false
28385 7800 Acked Write 5 Sent Done Done false
28386 7801 Acked Read 1 Sent Done Done false
28387 7802 Acked Flush 0 Sent Done Done false
28388 7803 Acked Read 9 Sent Done Done false
28389 7804 Acked Write 6 Sent Done Done false
28390 7805 Acked Read 1 Sent Done Done false
28391 7806 Acked Read 3 Sent Done Done false
28392 7807 Acked Write 6 Sent Done Done false
28393 7808 Acked Write 8 Sent Done Done false
28394 7809 Acked Read 7 Sent Done Done false
28395 7810 Acked Write 1 Sent Done Done false
28396 7811 Acked Read 7 Sent Done Done false
28397 7812 Acked Read 10 Sent Done Done false
28398 7813 Acked Read 7 Sent Done Done false
28399 7814 Acked Read 1 Sent Done Done false
28400 7815 Acked Write 7 Sent Done Done false
28401 7816 Acked Write 6 Sent Done Done false
28402 7817 Acked Write 8 Sent Done Done false
28403 7818 Acked Read 3 Sent Done Done false
28404 7819 Acked Write 10 Sent Done Done false
28405 7820 Acked Write 10 Sent Done Done false
28406 7821 Acked Read 6 Sent Done Done false
28407 7822 Acked Write 10 Sent Done Done false
28408 7823 Acked Write 1 Sent Done Done false
28409 7824 Acked Flush 0 Sent Done Done false
28410 7825 Acked Read 7 Sent Done Done false
28411 7826 Acked Read 7 Sent Done Done false
28412 7827 Acked Write 10 Sent Done Done false
28413 7828 Acked Flush 0 Sent Done Done false
28414 7829 Acked Read 9 Sent Done Done false
28415 7830 Acked Write 9 Sent Done Done false
28416 7831 Acked Write 9 Sent Done Done false
28417 7832 Acked Write 10 Sent Done Done false
28418 7833 Acked Write 4 Sent Done Done false
28419 7834 Acked Read 4 Sent Done Done false
28420 7835 Acked Read 4 Sent Done Done false
28421 7836 Acked Write 3 Sent Done Done false
28422 7837 Acked Read 10 Sent Done Done false
28423 7838 Acked Write 4 Sent Done Done false
28424 7839 Acked Read 1 Sent Done Done false
28425 7840 Acked Write 1 Sent Done Done false
28426 7841 Acked Write 4 Sent Done Done false
28427 7842 Acked Write 6 Sent Done Done false
28428 7843 Acked Write 4 Sent Done Done false
28429 7844 Acked Write 8 Sent Done Done false
28430 7845 Acked Write 10 Sent Done Done false
28431 7846 Acked Read 9 Sent Done Done false
28432 7847 Acked Write 6 Sent Done Done false
28433 7848 Acked Read 7 Sent Done Done false
28434 7849 Acked Read 6 Sent Done Done false
28435 7850 Acked Flush 0 Sent Done Done false
28436 7851 Acked Read 7 Sent Done Done false
28437 7852 Acked Write 2 Sent Done Done false
28438 7853 Acked Read 2 Sent Done Done false
28439 7854 Acked Write 8 Sent Done Done false
28440 7855 Acked Write 9 Sent Done Done false
28441 7856 Acked Write 2 Sent Done Done false
28442 7857 Acked Read 9 Sent Done Done false
28443 7858 Acked Write 7 Sent Done Done false
28444 7859 Acked Write 5 Sent Done Done false
28445 7860 Acked Flush 0 Sent Done Done false
28446 7861 Acked Read 3 Sent Done Done false
28447 7862 Acked Flush 0 Sent Done Done false
28448 7863 Acked Write 7 Sent Done Done false
28449 7864 Acked Write 9 Sent Done Done false
28450 7865 Acked Write 3 Sent Done Done false
28451 7866 Acked Flush 0 Sent Done Done false
28452 7867 Acked Read 5 Sent Done Done false
28453 7868 Acked Write 10 Sent Done Done false
28454 7869 Acked Write 3 Sent Done Done false
28455 7870 Acked Read 1 Sent Done Done false
28456 7871 Acked Read 9 Sent Done Done false
28457 7872 Acked Read 10 Sent Done Done false
28458 7873 Acked Read 1 Sent Done Done false
28459 7874 Acked Write 1 Sent Done Done false
28460 7875 Acked Write 7 Sent Done Done false
28461 7876 Acked Read 8 Sent Done Done false
28462 7877 Acked Write 9 Sent Done Done false
28463 7878 Acked Read 5 Sent Done Done false
28464 7879 Acked Write 4 Sent Done Done false
28465 7880 Acked Write 4 Sent Done Done false
28466 7881 Acked Write 1 Sent Done Done false
28467 7882 Acked Read 9 Sent Done Done false
28468 7883 Acked Flush 0 Sent Done Done false
28469 7884 Acked Write 2 Sent Done Done false
28470 7885 Acked Write 10 Sent Done Done false
28471 7886 Acked Read 7 Sent Done Done false
28472 7887 Acked Write 4 Sent Done Done false
28473 7888 Acked Flush 0 Sent Done Done false
28474 7889 Acked Read 10 Sent Done Done false
28475 7890 Acked Read 10 Sent Done Done false
28476 7891 Acked Write 7 Sent Done Done false
28477 7892 Acked Write 10 Sent Done Done false
28478 7893 Acked Write 6 Sent Done Done false
28479 7894 Acked Write 3 Sent Done Done false
28480 7895 Acked Write 1 Sent Done Done false
28481 7896 Acked Read 5 Sent Done Done false
28482 7897 Acked Read 7 Sent Done Done false
28483 7898 Acked Flush 0 Sent Done Done false
28484 7899 Acked Read 10 Sent Done Done false
28485 7900 Acked Write 1 Sent Done Done false
28486 7901 Acked Write 8 Sent Done Done false
28487 7902 Acked Read 3 Sent Done Done false
28488 7903 Acked Write 4 Sent Done Done false
28489 7904 Acked Read 8 Sent Done Done false
28490 7905 Acked Write 4 Sent Done Done false
28491 7906 Acked Read 7 Sent Done Done false
28492 7907 Acked Read 5 Sent Done Done false
28493 7908 Acked Write 1 Sent Done Done false
28494 7909 Acked Write 1 Sent Done Done false
28495 7910 Acked Write 10 Sent Done Done false
28496 7911 Acked Write 2 Sent Done Done false
28497 7912 Acked Write 1 Sent Done Done false
28498 7913 Acked Write 9 Sent Done Done false
28499 7914 Acked Read 6 Sent Done Done false
28500 7915 Acked Read 5 Sent Done Done false
28501 7916 Acked Write 9 Sent Done Done false
28502 7917 Acked Write 2 Sent Done Done false
28503 7918 Acked Read 2 Sent Done Done false
28504 7919 Acked Write 3 Sent Done Done false
28505 7920 Acked Write 5 Sent Done Done false
28506 7921 Acked Write 10 Sent Done Done false
28507 7922 Acked Read 4 Sent Done Done false
28508 7923 Acked Write 3 Sent Done Done false
28509 7924 Acked Read 5 Sent Done Done false
28510 7925 Acked Read 1 Sent Done Done false
28511 7926 Acked Write 4 Sent Done Done false
28512 7927 Acked Write 8 Sent Done Done false
28513 7928 Acked Write 5 Sent Done Done false
28514 7929 Acked Read 4 Sent Done Done false
28515 7930 Acked Read 6 Sent Done Done false
28516 7931 Acked Write 6 Sent Done Done false
28517 7932 Acked Read 5 Sent Done Done false
28518 7933 Acked Write 2 Sent Done Done false
28519 7934 Acked Write 3 Sent Done Done false
28520 7935 Acked Read 8 Sent Done Done false
28521 7936 Acked Write 6 Sent Done Done false
28522 7937 Acked Flush 0 Sent Done Done false
28523 7938 Acked Write 10 Sent Done Done false
28524 7939 Acked Read 7 Sent Done Done false
28525 7940 Acked Read 9 Sent Done Done false
28526 7941 Acked Read 3 Sent Done Done false
28527 STATES DS:0 DS:1 DS:2 TOTAL
28528 Sent 301 0 0 301
28529 Done 0 301 301 602
28530 Skipped 0 0 0 0
28531 Error 0 0 0 0
28532 Last Flush: Some(JobId(7640)) Some(JobId(7937)) Some(JobId(7937))
28533 Downstairs last five completed: 7640 7639 7638 7637 7636
28534 Upstairs last five completed: 7941 7940 7939 7938 7937
28535 CLIENT: Up:0 ds:301 act:2
285362025-02-28T06:07:18.146ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
285372025-02-28T06:07:18.972ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
285382025-02-28T06:07:19.974ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
285392025-02-28T06:07:20.976ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
285402025-02-28T06:07:21.399ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
285412025-02-28T06:07:21.399ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:35367) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
285422025-02-28T06:07:21.401ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
285432025-02-28T06:07:21.401ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
285442025-02-28T06:07:21.401ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
285452025-02-28T06:07:21.401ZINFOcrucible: send last flush ID to this DS: Some(JobId(7640)) = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
285462025-02-28T06:07:21.401ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
285472025-02-28T06:07:21.401ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(7640)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
285482025-02-28T06:07:21.402ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(7640)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
28549 ----------------------------------------------------------------
28550 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:4
28551 States: Active Active Active
28552 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28553 7938 Acked Write 10 Done Done Done true
28554 7939 Acked Read 7 Done Done Done true
28555 7940 Acked Read 9 Done Done Done true
28556 7941 Acked Read 3 Done Done Done true
28557 STATES DS:0 DS:1 DS:2 TOTAL
28558 Sent 0 0 0 0
28559 Done 4 4 4 12
28560 Skipped 0 0 0 0
28561 Error 0 0 0 0
28562 Last Flush: Some(JobId(7937)) Some(JobId(7937)) Some(JobId(7937))
28563 Downstairs last five completed: 7937 7936 7935 7934 7933
28564 Upstairs last five completed: 7941 7940 7939 7938 7937
28565 CLIENT: Up:0 ds:4 act:3
28566 ----------------------------------------------------------------
28567 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28568 Downstairs last five completed: 7942 7941 7940 7939 7938
28569 Upstairs last five completed: 7942 7941 7940 7939 7938
28570 CLIENT: Up:0 ds:0 act:3
28571 Replay: All jobs finished
28572 Main task has work to do, go find it
28573 got dsc Stop(0)
28574 stop 0
28575 Feb 28 06:07:25.646 [8810] Got stop action so:false kr:false
28576 INFO request completed, latency_us: 365, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 9847fb62-9fa5-4b89-a07c-68c92e504a40, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
285772025-02-28T06:07:25.647ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
28578 Feb 28 06:07:25.647 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/0, method: GET, req_id: 7b57a94b-041b-4b65-9a79-85d454478e06, remote_addr: 127.0.0.1:63908, local_addr: 127.0.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:25.647165041Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
28579 0{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:07:25.647202792Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
28580 .1:9998
285812025-02-28T06:07:25.647ZINFOcrucible: client task is exiting = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
28582 [8810] Exited with: ExitStatus(unix_wait_status(9))
28583 [8810][0] reports Exit
28584 Feb 28 06:07:29.648 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/0, method: GET, req_id: acc55c25-cd21-453a-84f8-4d3fa8f64a8d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
28585 001/300 Flush
28586 002/300 Write block 3375 len 20480 data: 3 4 3 4 4
28587 003/300 Write block 2352 len 24576 data: 4 5 3 3 5 6
28588 004/300 Read block 793 len 24576
28589 005/300 Write block 4511 len 16384 data: 7 7 8 9
28590 006/300 Read block 1506 len 20480
28591 007/300 Write block 584 len 28672 data: 6 6 7 5 6 4 5
28592 008/300 Write block 1340 len 28672 data: 7 6 6 5 3 3 2
28593 009/300 Read block 2989 len 20480
28594 010/300 Read block 4512 len 16384
28595 011/300 Read block 1434 len 36864
28596 012/300 Read block 1747 len 20480
28597 013/300 Read block 2478 len 4096
28598 014/300 Read block 79 len 24576
28599 015/300 Read block 1796 len 40960
28600 016/300 Read block 1206 len 24576
28601 017/300 Write block 2401 len 16384 data: 2 2 2 2
28602 018/300 Flush
28603 019/300 Write block 3769 len 36864 data: 6 5 6 6 7 6 4 3 3
28604 020/300 Read block 2836 len 28672
28605 021/300 Read block 2578 len 4096
28606 022/300 Flush
28607 023/300 Write block 1894 len 4096 data: 8
28608 024/300 Read block 422 len 32768
28609 025/300 Write block 3154 len 28672 data: 6 6 9 9 9 10 8
28610 026/300 Read block 3444 len 20480
28611 027/300 Read block 2482 len 24576
28612 028/300 Read block 4018 len 32768
28613 029/300 Flush
28614 030/300 Write block 4328 len 12288 data: 6 4 4
28615 031/300 Read block 997 len 28672
28616 032/300 Read block 3983 len 16384
28617 033/300 Read block 1546 len 40960
28618 034/300 Read block 4333 len 20480
28619 035/300 Write block 208 len 16384 data: 6 6 6 6
28620 036/300 Write block 2939 len 12288 data: 6 6 5
28621 037/300 Read block 1373 len 16384
28622 038/300 Flush
28623 039/300 Write block 2601 len 12288 data: 7 6 7
28624 040/300 Write block 465 len 24576 data: 3 5 6 7 7 8
28625 041/300 Write block 3034 len 24576 data: 6 6 6 6 5 5
28626 042/300 Read block 3461 len 12288
28627 043/300 Read block 16 len 40960
28628 044/300 Read block 225 len 28672
28629 045/300 Write block 4171 len 4096 data: 4
28630 046/300 Flush
28631 047/300 Read block 3160 len 4096
28632 048/300 Read block 3221 len 4096
28633 049/300 Read block 3960 len 28672
28634 050/300 Write block 1415 len 8192 data: 5 4
28635 051/300 Read block 4693 len 12288
28636 052/300 Read block 2384 len 36864
28637 053/300 Flush
28638 054/300 Flush
28639 055/300 Read block 120 len 36864
28640 056/300 Read block 3023 len 12288
28641 057/300 Read block 772 len 40960
28642 058/300 Write block 4087 len 8192 data: 5 4
28643 059/300 Flush
28644 060/300 Read block 2236 len 12288
28645 061/300 Read block 4727 len 4096
28646 062/300 Read block 1130 len 4096
28647 063/300 Read block 3644 len 28672
28648 064/300 Flush
28649 065/300 Read block 4532 len 32768
28650 066/300 Read block 936 len 32768
28651 067/300 Read block 1499 len 32768
28652 068/300 Write block 1319 len 28672 data: 6 6 6 7 7 5 5
28653 069/300 Write block 2063 len 28672 data: 4 5 6 5 5 6 4
28654 070/300 Read block 2174 len 4096
28655 071/300 Read block 177 len 36864
28656 072/300 Read block 1802 len 4096
28657 073/300 Read block 3119 len 36864
28658 074/300 Read block 1074 len 40960
28659 075/300 Write block 3662 len 4096 data: 4
28660 076/300 Read block 2997 len 20480
28661 077/300 Read block 4383 len 36864
28662 078/300 Write block 4471 len 16384 data: 4 5 5 5
28663 079/300 Read block 3094 len 40960
28664 080/300 Read block 4150 len 28672
28665 081/300 Write block 2854 len 16384 data: 5 5 5 6
28666 082/300 Write block 4340 len 20480 data: 6 6 9 8 8
28667 083/300 Write block 1520 len 20480 data: 4 5 4 3 3
28668 084/300 Read block 1407 len 28672
28669 085/300 Read block 329 len 20480
28670 086/300 Read block 1480 len 12288
28671 087/300 Write block 4336 len 20480 data: 7 7 6 7 7
28672 088/300 Write block 3445 len 28672 data: 4 4 4 3 2 3 3
28673 089/300 Read block 123 len 40960
28674 090/300 Read block 4160 len 28672
28675 091/300 Write block 1522 len 32768 data: 5 4 4 4 4 5 5 6
28676 092/300 Read block 3118 len 12288
28677 093/300 Read block 18 len 20480
28678 094/300 Write block 58 len 24576 data: 6 4 4 4 4 5
28679 095/300 Read block 2426 len 28672
28680 096/300 Read block 72 len 28672
28681 097/300 Write block 3280 len 20480 data: 3 2 3 2 2
28682 098/300 Read block 3470 len 24576
28683 099/300 Read block 2833 len 8192
28684 100/300 Read block 2228 len 36864
28685 101/300 Flush
28686 102/300 Read block 845 len 8192
28687 103/300 Write block 4734 len 32768 data: 8 6 6 4 4 4 4 4
28688 104/300 Write block 3480 len 4096 data: 8
28689 105/300 Write block 3267 len 36864 data: 4 4 5 4 5 5 4 5 5
28690 106/300 Read block 2474 len 8192
28691 107/300 Write block 471 len 28672 data: 5 5 4 6 7 8 7
28692 108/300 Read block 698 len 24576
28693 109/300 Read block 1015 len 24576
28694 110/300 Flush
28695 111/300 Write block 1151 len 20480 data: 7 8 8 7 8
28696 112/300 Write block 2795 len 40960 data: 7 6 6 5 6 6 6 4 5 5
28697 113/300 Flush
28698 114/300 Read block 4252 len 40960
28699 115/300 Write block 1005 len 24576 data: 4 3 3 4 4 4
28700 116/300 Write block 2297 len 40960 data: 3 4 3 3 3 3 3 4 4 4
28701 117/300 Read block 3559 len 8192
28702 118/300 Write block 860 len 24576 data: 5 5 6 6 6 5
28703 119/300 Read block 3968 len 36864
28704 120/300 Read block 1509 len 28672
28705 121/300 Write block 4704 len 32768 data: 3 3 3 3 3 4 4 5
28706 122/300 Write block 2230 len 16384 data: 7 9 8 7
28707 123/300 Write block 4272 len 36864 data: 8 6 6 5 4 5 5 5 5
28708 124/300 Write block 2302 len 4096 data: 4
28709 125/300 Read block 884 len 40960
28710 126/300 Read block 2182 len 20480
28711 127/300 Write block 918 len 40960 data: 5 5 5 5 5 4 4 4 5 5
28712 128/300 Read block 3115 len 20480
28713 129/300 Write block 3592 len 24576 data: 4 5 4 4 4 5
28714 130/300 Flush
28715 131/300 Write block 2946 len 12288 data: 3 4 5
28716 132/300 Read block 4059 len 4096
28717 133/300 Flush
28718 134/300 Read block 454 len 36864
28719 135/300 Write block 999 len 24576 data: 5 5 6 6 5 4
28720 136/300 Write block 2042 len 8192 data: 3 3
28721 137/300 Flush
28722 138/300 Read block 1220 len 20480
28723 139/300 Read block 4963 len 32768
28724 140/300 Read block 2765 len 24576
28725 141/300 Flush
28726 142/300 Read block 784 len 8192
28727 143/300 Write block 3866 len 20480 data: 8 9 8 10 9
28728 144/300 Read block 2780 len 8192
28729 145/300 Read block 4923 len 20480
28730 146/300 Read block 167 len 28672
28731 147/300 Write block 1628 len 4096 data: 7
28732 148/300 Read block 2297 len 16384
28733 149/300 Read block 1645 len 12288
28734 150/300 Read block 4764 len 36864
28735 151/300 Write block 4513 len 8192 data: 9 10
28736 152/300 Read block 3288 len 12288
28737 153/300 Read block 2709 len 16384
28738 154/300 Write block 71 len 28672 data: 3 3 5 5 8 7 7
28739 155/300 Read block 1584 len 32768
28740 156/300 Read block 1522 len 16384
28741 157/300 Read block 1047 len 4096
28742 158/300 Read block 2109 len 40960
28743 159/300 Read block 3029 len 12288
28744 160/300 Flush
28745 161/300 Read block 4248 len 28672
28746 162/300 Read block 689 len 8192
28747 163/300 Write block 1964 len 4096 data: 4
28748 164/300 Read block 398 len 8192
28749 165/300 Read block 3740 len 4096
28750 166/300 Write block 3417 len 24576 data: 6 6 6 5 5 5
28751 167/300 Read block 3895 len 16384
28752 168/300 Read block 2574 len 40960
28753 169/300 Read block 1364 len 32768
28754 170/300 Flush
28755 171/300 Write block 1963 len 20480 data: 3 5 4 4 5
28756 172/300 Read block 1536 len 20480
28757 173/300 Write block 2765 len 16384 data: 3 3 3 3
28758 174/300 Write block 4088 len 4096 data: 5
28759 175/300 Read block 3710 len 24576
28760 176/300 Write block 2758 len 16384 data: 5 7 6 5
28761 177/300 Read block 1148 len 40960
28762 178/300 Read block 2160 len 36864
28763 179/300 Write block 1 len 20480 data: 2 3 4 5 5
28764 180/300 Read block 584 len 20480
28765 181/300 Read block 4217 len 8192
28766 182/300 Flush
28767 183/300 Write block 2714 len 12288 data: 8 8 9
28768 184/300 Flush
28769 185/300 Read block 4890 len 16384
28770 186/300 Write block 1293 len 36864 data: 6 5 4 4 5 6 5 5 5
28771 187/300 Write block 3072 len 28672 data: 4 5 5 4 5 5 4
28772 188/300 Read block 404 len 36864
28773 189/300 Write block 3135 len 24576 data: 3 3 3 3 2 2
28774 190/300 Read block 2546 len 20480
28775 191/300 Write block 4188 len 28672 data: 6 6 8 8 7 7 5
28776 192/300 Flush
28777 193/300 Write block 4363 len 8192 data: 3 4
28778 194/300 Read block 2296 len 28672
28779 195/300 Write block 4006 len 16384 data: 6 6 6 5
28780 196/300 Write block 4561 len 12288 data: 4 4 6
28781 197/300 Flush
28782 198/300 Read block 4171 len 8192
28783 199/300 Read block 1540 len 16384
28784 200/300 Write block 2259 len 24576 data: 6 6 4 5 5 6
28785 201/300 Read block 686 len 24576
28786 202/300 Read block 4857 len 8192
28787 203/300 Read block 406 len 12288
28788 204/300 Write block 4175 len 8192 data: 4 4
28789 205/300 Flush
28790 206/300 Read block 3854 len 12288
28791 207/300 Write block 244 len 36864 data: 5 5 4 3 3 4 4 5 4
28792 208/300 Write block 949 len 16384 data: 7 7 7 4
28793 209/300 Flush
28794 210/300 Write block 4733 len 28672 data: 8 9 7 7 5 5 5
28795 211/300 Read block 3963 len 24576
28796 212/300 Read block 3415 len 32768
28797 213/300 Write block 3559 len 4096 data: 5
28798 214/300 Read block 2586 len 4096
28799 215/300 Read block 4175 len 28672
28800 216/300 Write block 4050 len 8192 data: 3 3
28801 217/300 Read block 1049 len 8192
28802 218/300 Write block 1379 len 4096 data: 6
28803 219/300 Read block 3775 len 40960
28804 220/300 Read block 3830 len 20480
28805 221/300 Read block 3685 len 16384
28806 222/300 Write block 1743 len 32768 data: 5 4 4 4 7 6 6 6
28807 223/300 Write block 322 len 8192 data: 6 6
28808 224/300 Read block 176 len 36864
28809 225/300 Write block 567 len 40960 data: 4 3 3 4 4 5 5 5 5 4
28810 226/300 Write block 4979 len 20480 data: 6 5 5 7 6
28811 227/300 Write block 1469 len 28672 data: 6 6 7 6 6 6 4
28812 228/300 Write block 3898 len 4096 data: 3
28813 229/300 Write block 3640 len 32768 data: 2 3 2 2 2 4 4 4
28814 230/300 Read block 2286 len 8192
28815 231/300 Write block 4213 len 24576 data: 7 5 6 6 5 2
28816 232/300 Flush
28817 233/300 Read block 4609 len 8192
28818 234/300 Read block 781 len 12288
28819 235/300 Write block 2670 len 8192 data: 10 7
28820 236/300 Flush
28821 237/300 Flush
28822 238/300 Read block 4386 len 20480
28823 239/300 Read block 1497 len 36864
28824 240/300 Read block 2589 len 16384
28825 241/300 Write block 453 len 16384 data: 7 7 6 6
28826 242/300 Flush
28827 243/300 Read block 3273 len 40960
28828 244/300 Write block 3909 len 40960 data: 4 3 3 3 3 4 4 4 3 4
28829 245/300 Read block 2161 len 24576
28830 246/300 Read block 2373 len 20480
28831 247/300 Read block 1927 len 4096
28832 248/300 Write block 2858 len 32768 data: 6 4 3 5 5 6 5 3
28833 249/300 Read block 3743 len 4096
28834 250/300 Read block 802 len 36864
28835 251/300 Read block 3420 len 16384
28836 252/300 Write block 3081 len 20480 data: 3 4 5 5 4
28837 253/300 Write block 742 len 36864 data: 6 7 8 7 7 6 8 9 9
28838 254/300 Write block 4546 len 32768 data: 5 5 4 4 5 5 4 4
28839 255/300 Read block 3888 len 8192
28840 256/300 Write block 1230 len 4096 data: 5
28841 257/300 Read block 2839 len 40960
28842 258/300 Write block 871 len 40960 data: 6 4 5 4 3 4 4 3 3 2
28843 259/300 Write block 1217 len 4096 data: 4
28844 260/300 Flush
28845 261/300 Read block 3726 len 4096
28846 262/300 Write block 3526 len 36864 data: 8 9 10 9 7 7 7 7 6
28847 263/300 Read block 2960 len 12288
28848 264/300 Read block 788 len 12288
28849 265/300 Write block 3290 len 32768 data: 5 4 3 7 5 5 6 7
28850 266/300 Read block 1598 len 12288
28851 267/300 Flush
28852 268/300 Write block 1467 len 4096 data: 6
28853 269/300 Write block 2512 len 8192 data: 7 6
28854 270/300 Read block 4983 len 36864
28855 271/300 Read block 3450 len 40960
28856 272/300 Write block 2536 len 32768 data: 4 4 3 3 3 3 4 4
28857 273/300 Write block 4685 len 32768 data: 5 5 6 4 4 4 4 5
28858 274/300 Write block 4937 len 28672 data: 3 3 3 2 3 3 3
28859 275/300 Write block 1459 len 8192 data: 6 5
28860 276/300 Write block 585 len 20480 data: 7 8 6 7 5
28861 277/300 Write block 2274 len 8192 data: 8 7
28862 278/300 Write block 3071 len 40960 data: 4 5 6 6 5 6 6 5 4 4
28863 279/300 Write block 1676 len 4096 data: 6
28864 280/300 Read block 4925 len 4096
28865 281/300 Read block 497 len 4096
28866 282/300 Read block 262 len 24576
28867 283/300 Read block 4462 len 40960
28868 284/300 Read block 1632 len 24576
28869 285/300 Write block 4757 len 4096 data: 9
28870 286/300 Read block 2978 len 40960
28871 287/300 Flush
28872 288/300 Read block 4138 len 8192
28873 289/300 Read block 3104 len 28672
28874 290/300 Flush
28875 291/300 Flush
28876 292/300 Read block 1312 len 40960
28877 293/300 Read block 4947 len 16384
28878 294/300 Read block 126 len 36864
28879 295/300 Write block 724 len 4096 data: 5
28880 296/300 Flush
28881 297/300 Flush
28882 298/300 Read block 1198 len 8192
28883 299/300 Flush
28884 300/300 Read block 4158 len 16384
28885 301/300 Write block 905 len 16384 data: 4 4 4 5
28886 Main task has work to do, go find it
28887 got dsc Start(0)
28888 start 0
28889 Feb 28 06:07:29.889 INFO request completed, latency_us: 242, response_code: 204, uri: /start/cid/0, method: POST, req_id: 9818b1f0-5206-4c4b-8102-63f4d6293b16, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
28890 Replay: started 0, returned:Ok(())
28891 ----------------------------------------------------------------
28892 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28893 States: Offline Active Active
28894 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28895 7943 Acked Flush 0 Sent Done Done false
28896 7944 Acked Write 5 Sent Done Done false
28897 7945 Acked Write 6 Sent Done Done false
28898 7946 Acked Read 6 Sent Done Done false
28899 7947 Acked Write 4 Sent Done Done false
28900 7948 Acked Read 5 Sent Done Done false
28901 7949 Acked Write 7 Sent Done Done false
28902 7950 Acked Write 7 Sent Done Done false
28903 7951 Acked Read 5 Sent Done Done false
28904 7952 Acked Read 4 Sent Done Done false
28905 7953 Acked Read 9 Sent Done Done false
28906 7954 Acked Read 5 Sent Done Done false
28907 7955 Acked Read 1 Sent Done Done false
28908 7956 Acked Read 6 Sent Done Done false
28909 7957 Acked Read 10 Sent Done Done false
28910 7958 Acked Read 6 Sent Done Done false
28911 7959 Acked Write 4 Sent Done Done false
28912 7960 Acked Flush 0 Sent Done Done false
28913 7961 Acked Write 9 Sent Done Done false
28914 7962 Acked Read 7 Sent Done Done false
28915 7963 Acked Read 1 Sent Done Done false
28916 7964 Acked Flush 0 Sent Done Done false
28917 7965 Acked Write 1 Sent Done Done false
28918 7966 Acked Read 8 Sent Done Done false
28919 7967 Acked Write 7 Sent Done Done false
28920 7968 Acked Read 5 Sent Done Done false
28921 7969 Acked Read 6 Sent Done Done false
28922 7970 Acked Read 8 Sent Done Done false
28923 7971 Acked Flush 0 Sent Done Done false
28924 7972 Acked Write 3 Sent Done Done false
28925 7973 Acked Read 7 Sent Done Done false
28926 7974 Acked Read 4 Sent Done Done false
28927 7975 Acked Read 10 Sent Done Done false
28928 7976 Acked Read 5 Sent Done Done false
28929 7977 Acked Write 4 Sent Done Done false
28930 7978 Acked Write 3 Sent Done Done false
28931 7979 Acked Read 4 Sent Done Done false
28932 7980 Acked Flush 0 Sent Done Done false
28933 7981 Acked Write 3 Sent Done Done false
28934 7982 Acked Write 6 Sent Done Done false
28935 7983 Acked Write 6 Sent Done Done false
28936 7984 Acked Read 3 Sent Done Done false
28937 7985 Acked Read 10 Sent Done Done false
28938 7986 Acked Read 7 Sent Done Done false
28939 7987 Acked Write 1 Sent Done Done false
28940 7988 Acked Flush 0 Sent Done Done false
28941 7989 Acked Read 1 Sent Done Done false
28942 7990 Acked Read 1 Sent Done Done false
28943 7991 Acked Read 7 Sent Done Done false
28944 7992 Acked Write 2 Sent Done Done false
28945 7993 Acked Read 3 Sent Done Done false
28946 7994 Acked Read 9 Sent Done Done false
28947 7995 Acked Flush 0 Sent Done Done false
28948 7996 Acked Flush 0 Sent Done Done false
28949 7997 Acked Read 9 Sent Done Done false
28950 7998 Acked Read 3 Sent Done Done false
28951 7999 Acked Read 10 Sent Done Done false
28952 8000 Acked Write 2 Sent Done Done false
28953 8001 Acked Flush 0 Sent Done Done false
28954 8002 Acked Read 3 Sent Done Done false
28955 8003 Acked Read 1 Sent Done Done false
28956 8004 Acked Read 1 Sent Done Done false
28957 8005 Acked Read 7 Sent Done Done false
28958 8006 Acked Flush 0 Sent Done Done false
28959 8007 Acked Read 8 Sent Done Done false
28960 8008 Acked Read 8 Sent Done Done false
28961 8009 Acked Read 8 Sent Done Done false
28962 8010 Acked Write 7 Sent Done Done false
28963 8011 Acked Write 7 Sent Done Done false
28964 8012 Acked Read 1 Sent Done Done false
28965 8013 Acked Read 9 Sent Done Done false
28966 8014 Acked Read 1 Sent Done Done false
28967 8015 Acked Read 9 Sent Done Done false
28968 8016 Acked Read 10 Sent Done Done false
28969 8017 Acked Write 1 Sent Done Done false
28970 8018 Acked Read 5 Sent Done Done false
28971 8019 Acked Read 9 Sent Done Done false
28972 8020 Acked Write 4 Sent Done Done false
28973 8021 Acked Read 10 Sent Done Done false
28974 8022 Acked Read 7 Sent Done Done false
28975 8023 Acked Write 4 Sent Done Done false
28976 8024 Acked Write 5 Sent Done Done false
28977 8025 Acked Write 5 Sent Done Done false
28978 8026 Acked Read 7 Sent Done Done false
28979 8027 Acked Read 5 Sent Done Done false
28980 8028 Acked Read 3 Sent Done Done false
28981 8029 Acked Write 5 Sent Done Done false
28982 8030 Acked Write 7 Sent Done Done false
28983 8031 Acked Read 10 Sent Done Done false
28984 8032 Acked Read 7 Sent Done Done false
28985 8033 Acked Write 8 Sent Done Done false
28986 8034 Acked Read 3 Sent Done Done false
28987 8035 Acked Read 5 Sent Done Done false
28988 8036 Acked Write 6 Sent Done Done false
28989 8037 Acked Read 7 Sent Done Done false
28990 8038 Acked Read 7 Sent Done Done false
28991 8039 Acked Write 5 Sent Done Done false
28992 8040 Acked Read 6 Sent Done Done false
28993 8041 Acked Read 2 Sent Done Done false
28994 8042 Acked Read 9 Sent Done Done false
28995 8043 Acked Flush 0 Sent Done Done false
28996 8044 Acked Read 2 Sent Done Done false
28997 8045 Acked Write 8 Sent Done Done false
28998 8046 Acked Write 1 Sent Done Done false
28999 8047 Acked Write 9 Sent Done Done false
29000 8048 Acked Read 2 Sent Done Done false
29001 8049 Acked Write 7 Sent Done Done false
29002 8050 Acked Read 6 Sent Done Done false
29003 8051 Acked Read 6 Sent Done Done false
29004 8052 Acked Flush 0 Sent Done Done false
29005 8053 Acked Write 5 Sent Done Done false
29006 8054 Acked Write 10 Sent Done Done false
29007 8055 Acked Flush 0 Sent Done Done false
29008 8056 Acked Read 10 Sent Done Done false
29009 8057 Acked Write 6 Sent Done Done false
29010 8058 Acked Write 10 Sent Done Done false
29011 8059 Acked Read 2 Sent Done Done false
29012 8060 Acked Write 6 Sent Done Done false
29013 8061 Acked Read 9 Sent Done Done false
29014 8062 Acked Read 7 Sent Done Done false
29015 8063 Acked Write 8 Sent Done Done false
29016 8064 Acked Write 4 Sent Done Done false
29017 8065 Acked Write 9 Sent Done Done false
29018 8066 Acked Write 1 Sent Done Done false
29019 8067 Acked Read 10 Sent Done Done false
29020 8068 Acked Read 5 Sent Done Done false
29021 8069 Acked Write 10 Sent Done Done false
29022 8070 Acked Read 5 Sent Done Done false
29023 8071 Acked Write 6 Sent Done Done false
29024 8072 Acked Flush 0 Sent Done Done false
29025 8073 Acked Write 3 Sent Done Done false
29026 8074 Acked Read 1 Sent Done Done false
29027 8075 Acked Flush 0 Sent Done Done false
29028 8076 Acked Read 9 Sent Done Done false
29029 8077 Acked Write 6 Sent Done Done false
29030 8078 Acked Write 2 Sent Done Done false
29031 8079 Acked Flush 0 Sent Done Done false
29032 8080 Acked Read 5 Sent Done Done false
29033 8081 Acked Read 8 Sent Done Done false
29034 8082 Acked Read 6 Sent Done Done false
29035 8083 Acked Flush 0 Sent Done Done false
29036 8084 Acked Read 2 Sent Done Done false
29037 8085 Acked Write 5 Sent Done Done false
29038 8086 Acked Read 2 Sent Done Done false
29039 8087 Acked Read 5 Sent Done Done false
29040 8088 Acked Read 7 Sent Done Done false
29041 8089 Acked Write 1 Sent Done Done false
29042 8090 Acked Read 4 Sent Done Done false
29043 8091 Acked Read 3 Sent Done Done false
29044 8092 Acked Read 9 Sent Done Done false
29045 8093 Acked Write 2 Sent Done Done false
29046 8094 Acked Read 3 Sent Done Done false
29047 8095 Acked Read 4 Sent Done Done false
29048 8096 Acked Write 7 Sent Done Done false
29049 8097 Acked Read 8 Sent Done Done false
29050 8098 Acked Read 4 Sent Done Done false
29051 8099 Acked Read 1 Sent Done Done false
29052 8100 Acked Read 10 Sent Done Done false
29053 8101 Acked Read 3 Sent Done Done false
29054 8102 Acked Flush 0 Sent Done Done false
29055 8103 Acked Read 7 Sent Done Done false
29056 8104 Acked Read 2 Sent Done Done false
29057 8105 Acked Write 1 Sent Done Done false
29058 8106 Acked Read 2 Sent Done Done false
29059 8107 Acked Read 1 Sent Done Done false
29060 8108 Acked Write 6 Sent Done Done false
29061 8109 Acked Read 4 Sent Done Done false
29062 8110 Acked Read 10 Sent Done Done false
29063 8111 Acked Read 8 Sent Done Done false
29064 8112 Acked Flush 0 Sent Done Done false
29065 8113 Acked Write 5 Sent Done Done false
29066 8114 Acked Read 5 Sent Done Done false
29067 8115 Acked Write 4 Sent Done Done false
29068 8116 Acked Write 1 Sent Done Done false
29069 8117 Acked Read 6 Sent Done Done false
29070 8118 Acked Write 4 Sent Done Done false
29071 8119 Acked Read 10 Sent Done Done false
29072 8120 Acked Read 9 Sent Done Done false
29073 8121 Acked Write 5 Sent Done Done false
29074 8122 Acked Read 5 Sent Done Done false
29075 8123 Acked Read 2 Sent Done Done false
29076 8124 Acked Flush 0 Sent Done Done false
29077 8125 Acked Write 3 Sent Done Done false
29078 8126 Acked Flush 0 Sent Done Done false
29079 8127 Acked Read 4 Sent Done Done false
29080 8128 Acked Write 9 Sent Done Done false
29081 8129 Acked Write 7 Sent Done Done false
29082 8130 Acked Read 9 Sent Done Done false
29083 8131 Acked Write 6 Sent Done Done false
29084 8132 Acked Read 5 Sent Done Done false
29085 8133 Acked Write 7 Sent Done Done false
29086 8134 Acked Flush 0 Sent Done Done false
29087 8135 Acked Write 2 Sent Done Done false
29088 8136 Acked Read 7 Sent Done Done false
29089 8137 Acked Write 4 Sent Done Done false
29090 8138 Acked Write 3 Sent Done Done false
29091 8139 Acked Flush 0 Sent Done Done false
29092 8140 Acked Read 2 Sent Done Done false
29093 8141 Acked Read 4 Sent Done Done false
29094 8142 Acked Write 6 Sent Done Done false
29095 8143 Acked Read 6 Sent Done Done false
29096 8144 Acked Read 2 Sent Done Done false
29097 8145 Acked Read 3 Sent Done Done false
29098 8146 Acked Write 2 Sent Done Done false
29099 8147 Acked Flush 0 Sent Done Done false
29100 8148 Acked Read 3 Sent Done Done false
29101 8149 Acked Write 9 Sent Done Done false
29102 8150 Acked Write 4 Sent Done Done false
29103 8151 Acked Flush 0 Sent Done Done false
29104 8152 Acked Write 7 Sent Done Done false
29105 8153 Acked Read 6 Sent Done Done false
29106 8154 Acked Read 8 Sent Done Done false
29107 8155 Acked Write 1 Sent Done Done false
29108 8156 Acked Read 1 Sent Done Done false
29109 8157 Acked Read 7 Sent Done Done false
29110 8158 Acked Write 2 Sent Done Done false
29111 8159 Acked Read 2 Sent Done Done false
29112 8160 Acked Write 1 Sent Done Done false
29113 8161 Acked Read 10 Sent Done Done false
29114 8162 Acked Read 5 Sent Done Done false
29115 8163 Acked Read 4 Sent Done Done false
29116 8164 Acked Write 8 Sent Done Done false
29117 8165 Acked Write 2 Sent Done Done false
29118 8166 Acked Read 9 Sent Done Done false
29119 8167 Acked Write 10 Sent Done Done false
29120 8168 Acked Write 5 Sent Done Done false
29121 8169 Acked Write 7 Sent Done Done false
29122 8170 Acked Write 1 Sent Done Done false
29123 8171 Acked Write 8 Sent Done Done false
29124 8172 Acked Read 2 Sent Done Done false
29125 8173 Acked Write 6 Sent Done Done false
29126 8174 Acked Flush 0 Sent Done Done false
29127 8175 Acked Read 2 Sent Done Done false
29128 8176 Acked Read 3 Sent Done Done false
29129 8177 Acked Write 2 Sent Done Done false
29130 8178 Acked Flush 0 Sent Done Done false
29131 8179 Acked Flush 0 Sent Done Done false
29132 8180 Acked Read 5 Sent Done Done false
29133 8181 Acked Read 9 Sent Done Done false
29134 8182 Acked Read 4 Sent Done Done false
29135 8183 Acked Write 4 Sent Done Done false
29136 8184 Acked Flush 0 Sent Done Done false
29137 8185 Acked Read 10 Sent Done Done false
29138 8186 Acked Write 10 Sent Done Done false
29139 8187 Acked Read 6 Sent Done Done false
29140 8188 Acked Read 5 Sent Done Done false
29141 8189 Acked Read 1 Sent Done Done false
29142 8190 Acked Write 8 Sent Done Done false
29143 8191 Acked Read 1 Sent Done Done false
29144 8192 Acked Read 9 Sent Done Done false
29145 8193 Acked Read 4 Sent Done Done false
29146 8194 Acked Write 5 Sent Done Done false
29147 8195 Acked Write 9 Sent Done Done false
29148 8196 Acked Write 8 Sent Done Done false
29149 8197 Acked Read 2 Sent Done Done false
29150 8198 Acked Write 1 Sent Done Done false
29151 8199 Acked Read 10 Sent Done Done false
29152 8200 Acked Write 10 Sent Done Done false
29153 8201 Acked Write 1 Sent Done Done false
29154 8202 Acked Flush 0 Sent Done Done false
29155 8203 Acked Read 1 Sent Done Done false
29156 8204 Acked Write 9 Sent Done Done false
29157 8205 Acked Read 3 Sent Done Done false
29158 8206 Acked Read 3 Sent Done Done false
29159 8207 Acked Write 8 Sent Done Done false
29160 8208 Acked Read 3 Sent Done Done false
29161 8209 Acked Flush 0 Sent Done Done false
29162 8210 Acked Write 1 Sent Done Done false
29163 8211 Acked Write 2 Sent Done Done false
29164 8212 Acked Read 9 Sent Done Done false
29165 8213 Acked Read 10 Sent Done Done false
29166 8214 Acked Write 8 Sent Done Done false
29167 8215 Acked Write 8 Sent Done Done false
29168 8216 Acked Write 7 Sent Done Done false
29169 8217 Acked Write 2 Sent Done Done false
29170 8218 Acked Write 5 Sent Done Done false
29171 8219 Acked Write 2 Sent Done Done false
29172 8220 Acked Write 10 Sent Done Done false
29173 8221 Acked Write 1 Sent Done Done false
29174 8222 Acked Read 1 Sent Done Done false
29175 8223 Acked Read 1 Sent Done Done false
29176 8224 Acked Read 6 Sent Done Done false
29177 8225 Acked Read 10 Sent Done Done false
29178 8226 Acked Read 6 Sent Done Done false
29179 8227 Acked Write 1 Sent Done Done false
29180 8228 Acked Read 10 Sent Done Done false
29181 8229 Acked Flush 0 Sent Done Done false
29182 8230 Acked Read 2 Sent Done Done false
29183 8231 Acked Read 7 Sent Done Done false
29184 8232 Acked Flush 0 Sent Done Done false
29185 8233 Acked Flush 0 Sent Done Done false
29186 8234 Acked Read 10 Sent Done Done false
29187 8235 Acked Read 4 Sent Done Done false
29188 8236 Acked Read 9 Sent Done Done false
29189 8237 Acked Write 1 Sent Done Done false
29190 8238 Acked Flush 0 Sent Done Done false
29191 8239 Acked Flush 0 Sent Done Done false
29192 8240 Acked Read 2 Sent Done Done false
29193 8241 Acked Flush 0 Sent Done Done false
29194 8242 Acked Read 4 Sent Done Done false
29195 8243 Acked Write 4 Sent Done Done false
29196 STATES DS:0 DS:1 DS:2 TOTAL
29197 Sent 301 0 0 301
29198 Done 0 301 301 602
29199 Skipped 0 0 0 0
29200 Error 0 0 0 0
29201 Last Flush: Some(JobId(7942)) Some(JobId(8241)) Some(JobId(8241))
29202 Downstairs last five completed: 7942 7941 7940 7939 7938
29203 Upstairs last five completed: 8243 8242 8241 8240 8239
29204 CLIENT: Up:0 ds:301 act:2
292052025-02-28T06:07:30.391ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
292062025-02-28T06:07:30.992ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
29207 [8810] got start action
29208 [8810] I am going to restart
29209 Starting downstairs at port 8810
29210 Make output file at "/tmp/dsc/downstairs-8810.txt"
29211 [8810][0] reports Starting
29212 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1235)
29213 [8810][0] reports Running
292142025-02-28T06:07:31.994ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
292152025-02-28T06:07:33.147ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
29216 ----------------------------------------------------------------
29217 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29218 States: Offline Active Active
29219 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29220 7943 Acked Flush 0 Sent Done Done false
29221 7944 Acked Write 5 Sent Done Done false
29222 7945 Acked Write 6 Sent Done Done false
29223 7946 Acked Read 6 Sent Done Done false
29224 7947 Acked Write 4 Sent Done Done false
29225 7948 Acked Read 5 Sent Done Done false
29226 7949 Acked Write 7 Sent Done Done false
29227 7950 Acked Write 7 Sent Done Done false
29228 7951 Acked Read 5 Sent Done Done false
29229 7952 Acked Read 4 Sent Done Done false
29230 7953 Acked Read 9 Sent Done Done false
29231 7954 Acked Read 5 Sent Done Done false
29232 7955 Acked Read 1 Sent Done Done false
29233 7956 Acked Read 6 Sent Done Done false
29234 7957 Acked Read 10 Sent Done Done false
29235 7958 Acked Read 6 Sent Done Done false
29236 7959 Acked Write 4 Sent Done Done false
29237 7960 Acked Flush 0 Sent Done Done false
29238 7961 Acked Write 9 Sent Done Done false
29239 7962 Acked Read 7 Sent Done Done false
29240 7963 Acked Read 1 Sent Done Done false
29241 7964 Acked Flush 0 Sent Done Done false
29242 7965 Acked Write 1 Sent Done Done false
29243 7966 Acked Read 8 Sent Done Done false
29244 7967 Acked Write 7 Sent Done Done false
29245 7968 Acked Read 5 Sent Done Done false
29246 7969 Acked Read 6 Sent Done Done false
29247 7970 Acked Read 8 Sent Done Done false
29248 7971 Acked Flush 0 Sent Done Done false
29249 7972 Acked Write 3 Sent Done Done false
29250 7973 Acked Read 7 Sent Done Done false
29251 7974 Acked Read 4 Sent Done Done false
29252 7975 Acked Read 10 Sent Done Done false
29253 7976 Acked Read 5 Sent Done Done false
29254 7977 Acked Write 4 Sent Done Done false
29255 7978 Acked Write 3 Sent Done Done false
29256 7979 Acked Read 4 Sent Done Done false
29257 7980 Acked Flush 0 Sent Done Done false
29258 7981 Acked Write 3 Sent Done Done false
29259 7982 Acked Write 6 Sent Done Done false
29260 7983 Acked Write 6 Sent Done Done false
29261 7984 Acked Read 3 Sent Done Done false
29262 7985 Acked Read 10 Sent Done Done false
29263 7986 Acked Read 7 Sent Done Done false
29264 7987 Acked Write 1 Sent Done Done false
29265 7988 Acked Flush 0 Sent Done Done false
29266 7989 Acked Read 1 Sent Done Done false
29267 7990 Acked Read 1 Sent Done Done false
29268 7991 Acked Read 7 Sent Done Done false
29269 7992 Acked Write 2 Sent Done Done false
29270 7993 Acked Read 3 Sent Done Done false
29271 7994 Acked Read 9 Sent Done Done false
29272 7995 Acked Flush 0 Sent Done Done false
29273 7996 Acked Flush 0 Sent Done Done false
29274 7997 Acked Read 9 Sent Done Done false
29275 7998 Acked Read 3 Sent Done Done false
29276 7999 Acked Read 10 Sent Done Done false
29277 8000 Acked Write 2 Sent Done Done false
29278 8001 Acked Flush 0 Sent Done Done false
29279 8002 Acked Read 3 Sent Done Done false
29280 8003 Acked Read 1 Sent Done Done false
29281 8004 Acked Read 1 Sent Done Done false
29282 8005 Acked Read 7 Sent Done Done false
29283 8006 Acked Flush 0 Sent Done Done false
29284 8007 Acked Read 8 Sent Done Done false
29285 8008 Acked Read 8 Sent Done Done false
29286 8009 Acked Read 8 Sent Done Done false
29287 8010 Acked Write 7 Sent Done Done false
29288 8011 Acked Write 7 Sent Done Done false
29289 8012 Acked Read 1 Sent Done Done false
29290 8013 Acked Read 9 Sent Done Done false
29291 8014 Acked Read 1 Sent Done Done false
29292 8015 Acked Read 9 Sent Done Done false
29293 8016 Acked Read 10 Sent Done Done false
29294 8017 Acked Write 1 Sent Done Done false
29295 8018 Acked Read 5 Sent Done Done false
29296 8019 Acked Read 9 Sent Done Done false
29297 8020 Acked Write 4 Sent Done Done false
29298 8021 Acked Read 10 Sent Done Done false
29299 8022 Acked Read 7 Sent Done Done false
29300 8023 Acked Write 4 Sent Done Done false
29301 8024 Acked Write 5 Sent Done Done false
29302 8025 Acked Write 5 Sent Done Done false
29303 8026 Acked Read 7 Sent Done Done false
29304 8027 Acked Read 5 Sent Done Done false
29305 8028 Acked Read 3 Sent Done Done false
29306 8029 Acked Write 5 Sent Done Done false
29307 8030 Acked Write 7 Sent Done Done false
29308 8031 Acked Read 10 Sent Done Done false
29309 8032 Acked Read 7 Sent Done Done false
29310 8033 Acked Write 8 Sent Done Done false
29311 8034 Acked Read 3 Sent Done Done false
29312 8035 Acked Read 5 Sent Done Done false
29313 8036 Acked Write 6 Sent Done Done false
29314 8037 Acked Read 7 Sent Done Done false
29315 8038 Acked Read 7 Sent Done Done false
29316 8039 Acked Write 5 Sent Done Done false
29317 8040 Acked Read 6 Sent Done Done false
29318 8041 Acked Read 2 Sent Done Done false
29319 8042 Acked Read 9 Sent Done Done false
29320 8043 Acked Flush 0 Sent Done Done false
29321 8044 Acked Read 2 Sent Done Done false
29322 8045 Acked Write 8 Sent Done Done false
29323 8046 Acked Write 1 Sent Done Done false
29324 8047 Acked Write 9 Sent Done Done false
29325 8048 Acked Read 2 Sent Done Done false
29326 8049 Acked Write 7 Sent Done Done false
29327 8050 Acked Read 6 Sent Done Done false
29328 8051 Acked Read 6 Sent Done Done false
29329 8052 Acked Flush 0 Sent Done Done false
29330 8053 Acked Write 5 Sent Done Done false
29331 8054 Acked Write 10 Sent Done Done false
29332 8055 Acked Flush 0 Sent Done Done false
29333 8056 Acked Read 10 Sent Done Done false
29334 8057 Acked Write 6 Sent Done Done false
29335 8058 Acked Write 10 Sent Done Done false
29336 8059 Acked Read 2 Sent Done Done false
29337 8060 Acked Write 6 Sent Done Done false
29338 8061 Acked Read 9 Sent Done Done false
29339 8062 Acked Read 7 Sent Done Done false
29340 8063 Acked Write 8 Sent Done Done false
29341 8064 Acked Write 4 Sent Done Done false
29342 8065 Acked Write 9 Sent Done Done false
29343 8066 Acked Write 1 Sent Done Done false
29344 8067 Acked Read 10 Sent Done Done false
29345 8068 Acked Read 5 Sent Done Done false
29346 8069 Acked Write 10 Sent Done Done false
29347 8070 Acked Read 5 Sent Done Done false
29348 8071 Acked Write 6 Sent Done Done false
29349 8072 Acked Flush 0 Sent Done Done false
29350 8073 Acked Write 3 Sent Done Done false
29351 8074 Acked Read 1 Sent Done Done false
29352 8075 Acked Flush 0 Sent Done Done false
29353 8076 Acked Read 9 Sent Done Done false
29354 8077 Acked Write 6 Sent Done Done false
29355 8078 Acked Write 2 Sent Done Done false
29356 8079 Acked Flush 0 Sent Done Done false
29357 8080 Acked Read 5 Sent Done Done false
29358 8081 Acked Read 8 Sent Done Done false
29359 8082 Acked Read 6 Sent Done Done false
29360 8083 Acked Flush 0 Sent Done Done false
29361 8084 Acked Read 2 Sent Done Done false
29362 8085 Acked Write 5 Sent Done Done false
29363 8086 Acked Read 2 Sent Done Done false
29364 8087 Acked Read 5 Sent Done Done false
29365 8088 Acked Read 7 Sent Done Done false
29366 8089 Acked Write 1 Sent Done Done false
29367 8090 Acked Read 4 Sent Done Done false
29368 8091 Acked Read 3 Sent Done Done false
29369 8092 Acked Read 9 Sent Done Done false
29370 8093 Acked Write 2 Sent Done Done false
29371 8094 Acked Read 3 Sent Done Done false
29372 8095 Acked Read 4 Sent Done Done false
29373 8096 Acked Write 7 Sent Done Done false
29374 8097 Acked Read 8 Sent Done Done false
29375 8098 Acked Read 4 Sent Done Done false
29376 8099 Acked Read 1 Sent Done Done false
29377 8100 Acked Read 10 Sent Done Done false
29378 8101 Acked Read 3 Sent Done Done false
29379 8102 Acked Flush 0 Sent Done Done false
29380 8103 Acked Read 7 Sent Done Done false
29381 8104 Acked Read 2 Sent Done Done false
29382 8105 Acked Write 1 Sent Done Done false
29383 8106 Acked Read 2 Sent Done Done false
29384 8107 Acked Read 1 Sent Done Done false
29385 8108 Acked Write 6 Sent Done Done false
29386 8109 Acked Read 4 Sent Done Done false
29387 8110 Acked Read 10 Sent Done Done false
29388 8111 Acked Read 8 Sent Done Done false
29389 8112 Acked Flush 0 Sent Done Done false
29390 8113 Acked Write 5 Sent Done Done false
29391 8114 Acked Read 5 Sent Done Done false
29392 8115 Acked Write 4 Sent Done Done false
29393 8116 Acked Write 1 Sent Done Done false
29394 8117 Acked Read 6 Sent Done Done false
29395 8118 Acked Write 4 Sent Done Done false
29396 8119 Acked Read 10 Sent Done Done false
29397 8120 Acked Read 9 Sent Done Done false
29398 8121 Acked Write 5 Sent Done Done false
29399 8122 Acked Read 5 Sent Done Done false
29400 8123 Acked Read 2 Sent Done Done false
29401 8124 Acked Flush 0 Sent Done Done false
29402 8125 Acked Write 3 Sent Done Done false
29403 8126 Acked Flush 0 Sent Done Done false
29404 8127 Acked Read 4 Sent Done Done false
29405 8128 Acked Write 9 Sent Done Done false
29406 8129 Acked Write 7 Sent Done Done false
29407 8130 Acked Read 9 Sent Done Done false
29408 8131 Acked Write 6 Sent Done Done false
29409 8132 Acked Read 5 Sent Done Done false
29410 8133 Acked Write 7 Sent Done Done false
29411 8134 Acked Flush 0 Sent Done Done false
29412 8135 Acked Write 2 Sent Done Done false
29413 8136 Acked Read 7 Sent Done Done false
29414 8137 Acked Write 4 Sent Done Done false
29415 8138 Acked Write 3 Sent Done Done false
29416 8139 Acked Flush 0 Sent Done Done false
29417 8140 Acked Read 2 Sent Done Done false
29418 8141 Acked Read 4 Sent Done Done false
29419 8142 Acked Write 6 Sent Done Done false
29420 8143 Acked Read 6 Sent Done Done false
29421 8144 Acked Read 2 Sent Done Done false
29422 8145 Acked Read 3 Sent Done Done false
29423 8146 Acked Write 2 Sent Done Done false
29424 8147 Acked Flush 0 Sent Done Done false
29425 8148 Acked Read 3 Sent Done Done false
29426 8149 Acked Write 9 Sent Done Done false
29427 8150 Acked Write 4 Sent Done Done false
29428 8151 Acked Flush 0 Sent Done Done false
29429 8152 Acked Write 7 Sent Done Done false
29430 8153 Acked Read 6 Sent Done Done false
29431 8154 Acked Read 8 Sent Done Done false
29432 8155 Acked Write 1 Sent Done Done false
29433 8156 Acked Read 1 Sent Done Done false
29434 8157 Acked Read 7 Sent Done Done false
29435 8158 Acked Write 2 Sent Done Done false
29436 8159 Acked Read 2 Sent Done Done false
29437 8160 Acked Write 1 Sent Done Done false
29438 8161 Acked Read 10 Sent Done Done false
29439 8162 Acked Read 5 Sent Done Done false
29440 8163 Acked Read 4 Sent Done Done false
29441 8164 Acked Write 8 Sent Done Done false
29442 8165 Acked Write 2 Sent Done Done false
29443 8166 Acked Read 9 Sent Done Done false
29444 8167 Acked Write 10 Sent Done Done false
29445 8168 Acked Write 5 Sent Done Done false
29446 8169 Acked Write 7 Sent Done Done false
29447 8170 Acked Write 1 Sent Done Done false
29448 8171 Acked Write 8 Sent Done Done false
29449 8172 Acked Read 2 Sent Done Done false
29450 8173 Acked Write 6 Sent Done Done false
29451 8174 Acked Flush 0 Sent Done Done false
29452 8175 Acked Read 2 Sent Done Done false
29453 8176 Acked Read 3 Sent Done Done false
29454 8177 Acked Write 2 Sent Done Done false
29455 8178 Acked Flush 0 Sent Done Done false
29456 8179 Acked Flush 0 Sent Done Done false
29457 8180 Acked Read 5 Sent Done Done false
29458 8181 Acked Read 9 Sent Done Done false
29459 8182 Acked Read 4 Sent Done Done false
29460 8183 Acked Write 4 Sent Done Done false
29461 8184 Acked Flush 0 Sent Done Done false
29462 8185 Acked Read 10 Sent Done Done false
29463 8186 Acked Write 10 Sent Done Done false
29464 8187 Acked Read 6 Sent Done Done false
29465 8188 Acked Read 5 Sent Done Done false
29466 8189 Acked Read 1 Sent Done Done false
29467 8190 Acked Write 8 Sent Done Done false
29468 8191 Acked Read 1 Sent Done Done false
29469 8192 Acked Read 9 Sent Done Done false
29470 8193 Acked Read 4 Sent Done Done false
29471 8194 Acked Write 5 Sent Done Done false
29472 8195 Acked Write 9 Sent Done Done false
29473 8196 Acked Write 8 Sent Done Done false
29474 8197 Acked Read 2 Sent Done Done false
29475 8198 Acked Write 1 Sent Done Done false
29476 8199 Acked Read 10 Sent Done Done false
29477 8200 Acked Write 10 Sent Done Done false
29478 8201 Acked Write 1 Sent Done Done false
29479 8202 Acked Flush 0 Sent Done Done false
29480 8203 Acked Read 1 Sent Done Done false
29481 8204 Acked Write 9 Sent Done Done false
29482 8205 Acked Read 3 Sent Done Done false
29483 8206 Acked Read 3 Sent Done Done false
29484 8207 Acked Write 8 Sent Done Done false
29485 8208 Acked Read 3 Sent Done Done false
29486 8209 Acked Flush 0 Sent Done Done false
29487 8210 Acked Write 1 Sent Done Done false
29488 8211 Acked Write 2 Sent Done Done false
29489 8212 Acked Read 9 Sent Done Done false
29490 8213 Acked Read 10 Sent Done Done false
29491 8214 Acked Write 8 Sent Done Done false
29492 8215 Acked Write 8 Sent Done Done false
29493 8216 Acked Write 7 Sent Done Done false
29494 8217 Acked Write 2 Sent Done Done false
29495 8218 Acked Write 5 Sent Done Done false
29496 8219 Acked Write 2 Sent Done Done false
29497 8220 Acked Write 10 Sent Done Done false
29498 8221 Acked Write 1 Sent Done Done false
29499 8222 Acked Read 1 Sent Done Done false
29500 8223 Acked Read 1 Sent Done Done false
29501 8224 Acked Read 6 Sent Done Done false
29502 8225 Acked Read 10 Sent Done Done false
29503 8226 Acked Read 6 Sent Done Done false
29504 8227 Acked Write 1 Sent Done Done false
29505 8228 Acked Read 10 Sent Done Done false
29506 8229 Acked Flush 0 Sent Done Done false
29507 8230 Acked Read 2 Sent Done Done false
29508 8231 Acked Read 7 Sent Done Done false
29509 8232 Acked Flush 0 Sent Done Done false
29510 8233 Acked Flush 0 Sent Done Done false
29511 8234 Acked Read 10 Sent Done Done false
29512 8235 Acked Read 4 Sent Done Done false
29513 8236 Acked Read 9 Sent Done Done false
29514 8237 Acked Write 1 Sent Done Done false
29515 8238 Acked Flush 0 Sent Done Done false
29516 8239 Acked Flush 0 Sent Done Done false
29517 8240 Acked Read 2 Sent Done Done false
29518 8241 Acked Flush 0 Sent Done Done false
29519 8242 Acked Read 4 Sent Done Done false
29520 8243 Acked Write 4 Sent Done Done false
29521 STATES DS:0 DS:1 DS:2 TOTAL
29522 Sent 301 0 0 301
29523 Done 0 301 301 602
29524 Skipped 0 0 0 0
29525 Error 0 0 0 0
29526 Last Flush: Some(JobId(7942)) Some(JobId(8241)) Some(JobId(8241))
29527 Downstairs last five completed: 7942 7941 7940 7939 7938
29528 Upstairs last five completed: 8243 8242 8241 8240 8239
29529 CLIENT: Up:0 ds:301 act:2
295302025-02-28T06:07:34.999ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
295312025-02-28T06:07:35.648ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
295322025-02-28T06:07:35.648ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46837) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
295332025-02-28T06:07:35.649ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
295342025-02-28T06:07:35.649ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
295352025-02-28T06:07:35.649ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
295362025-02-28T06:07:35.649ZINFOcrucible: send last flush ID to this DS: Some(JobId(7942)) = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
295372025-02-28T06:07:35.649ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
295382025-02-28T06:07:35.649ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(7942)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
295392025-02-28T06:07:35.650ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(7942)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
29540 ----------------------------------------------------------------
29541 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29542 Downstairs last five completed: 8244 8243 8242 8241 8240
29543 Upstairs last five completed: 8244 8243 8242 8241 8240
29544 CLIENT: Up:0 ds:0 act:3
29545 Replay: All jobs finished
29546 Main task has work to do, go find it
29547 got dsc Stop(2)
29548 stop 2
29549 Feb 28 06:07:37.894[8830] Got stop action so:false kr:false
29550 INFO request completed, latency_us: 331, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 84abda63-d63c-452d-ae90-b9bfdb3ec74f, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
295512025-02-28T06:07:37.895ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
295522025-02-28T06:07:37.895ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
295532025-02-28T06:07:37.895ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
29554 Feb 28 06:07:37.895 INFO request completed, latency_us: 131, response_code: 200, uri: /state/cid/2, method: GET, req_id{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:07:37.895419695Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
29555 : ac171e59-e9a4-420d-92f3-682344d867a7, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
29556 [8830] Exited with: ExitStatus(unix_wait_status(9))
29557 [8830][2] reports Exit
29558 Feb 28 06:07:41.897 INFO request completed, latency_us: 315, response_code: 200, uri: /state/cid/2, method: GET, req_id: f9ca2c7e-c992-4c2a-b04c-ac39c4901a7f, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
29559 001/300 Read block 3220 len 28672
29560 002/300 Read block 3935 len 8192
29561 003/300 Write block 3657 len 36864 data: 3 2 2 2 2 5 4 4 4
29562 004/300 Flush
29563 005/300 Read block 1581 len 24576
29564 006/300 Read block 3288 len 24576
29565 007/300 Read block 321 len 32768
29566 008/300 Flush
29567 009/300 Write block 4738 len 12288 data: 6 6 5
29568 010/300 Write block 4639 len 8192 data: 5 4
29569 011/300 Flush
29570 012/300 Read block 4677 len 36864
29571 013/300 Write block 5 len 28672 data: 6 5 4 4 4 5 4
29572 014/300 Read block 2394 len 40960
29573 015/300 Write block 2537 len 12288 data: 5 4 4
29574 016/300 Write block 2259 len 12288 data: 7 7 5
29575 017/300 Read block 1334 len 28672
29576 018/300 Read block 3031 len 32768
29577 019/300 Read block 592 len 40960
29578 020/300 Read block 4350 len 12288
29579 021/300 Write block 4740 len 20480 data: 6 5 4 3 4
29580 022/300 Read block 1269 len 8192
29581 023/300 Write block 2066 len 28672 data: 6 6 7 5 4 5 5
29582 024/300 Write block 1974 len 16384 data: 8 8 5 3
29583 025/300 Read block 1070 len 20480
29584 026/300 Write block 3903 len 24576 data: 6 6 5 4 5 4
29585 027/300 Read block 2706 len 28672
29586 028/300 Write block 4958 len 4096 data: 9
29587 029/300 Write block 171 len 16384 data: 3 4 4 5
29588 030/300 Write block 2123 len 16384 data: 6 7 6 6
29589 031/300 Read block 4565 len 8192
29590 032/300 Write block 3486 len 4096 data: 6
29591 033/300 Read block 308 len 24576
29592 034/300 Write block 479 len 16384 data: 6 7 7 7
29593 035/300 Read block 3175 len 40960
29594 036/300 Write block 2303 len 4096 data: 4
29595 037/300 Read block 2781 len 20480
29596 038/300 Read block 3358 len 24576
29597 039/300 Write block 1481 len 40960 data: 4 4 5 5 6 6 6 7 8 9
29598 040/300 Read block 4372 len 20480
29599 041/300 Write block 283 len 4096 data: 5
29600 042/300 Write block 1563 len 36864 data: 2 3 4 6 5 6 6 5 5
29601 043/300 Read block 1057 len 16384
29602 044/300 Write block 4767 len 40960 data: 4 5 6 9 8 7 6 5 6 7
29603 045/300 Read block 2983 len 8192
29604 046/300 Read block 1969 len 8192
29605 047/300 Read block 3832 len 24576
29606 048/300 Flush
29607 049/300 Read block 4484 len 4096
29608 050/300 Read block 1909 len 12288
29609 051/300 Flush
29610 052/300 Write block 3593 len 12288 data: 6 5 5
29611 053/300 Write block 4637 len 24576 data: 5 5 6 5 3 4
29612 054/300 Read block 41 len 12288
29613 055/300 Read block 3571 len 40960
29614 056/300 Read block 1523 len 36864
29615 057/300 Read block 4189 len 24576
29616 058/300 Write block 2261 len 28672 data: 6 6 6 7 5 5 7
29617 059/300 Write block 4058 len 12288 data: 4 4 4
29618 060/300 Read block 1239 len 32768
29619 061/300 Write block 4542 len 24576 data: 4 5 6 5 6 6
29620 062/300 Read block 4478 len 16384
29621 063/300 Read block 115 len 36864
29622 064/300 Flush
29623 065/300 Write block 4968 len 20480 data: 9 8 7 8 7
29624 066/300 Write block 3026 len 24576 data: 4 5 5 5 5 5
29625 067/300 Write block 2285 len 16384 data: 5 4 2 3
29626 068/300 Read block 4315 len 16384
29627 069/300 Write block 627 len 32768 data: 5 3 3 3 2 3 3 3
29628 070/300 Flush
29629 071/300 Write block 2101 len 20480 data: 5 5 2 2 2
29630 072/300 Write block 2901 len 20480 data: 4 4 4 4 5
29631 073/300 Read block 3904 len 20480
29632 074/300 Write block 3233 len 4096 data: 2
29633 075/300 Read block 3396 len 40960
29634 076/300 Flush
29635 077/300 Read block 3283 len 28672
29636 078/300 Write block 2986 len 32768 data: 6 6 6 6 8 8 9 6
29637 079/300 Read block 881 len 16384
29638 080/300 Write block 1524 len 16384 data: 5 5 5 6
29639 081/300 Read block 3646 len 12288
29640 082/300 Read block 3103 len 32768
29641 083/300 Read block 2160 len 16384
29642 084/300 Read block 1152 len 12288
29643 085/300 Write block 1802 len 20480 data: 6 4 3 3 3
29644 086/300 Write block 4785 len 16384 data: 5 5 4 5
29645 087/300 Read block 814 len 32768
29646 088/300 Read block 1580 len 12288
29647 089/300 Write block 405 len 12288 data: 4 5 5
29648 090/300 Write block 1560 len 28672 data: 4 5 3 3 4 5 7
29649 091/300 Read block 4080 len 36864
29650 092/300 Read block 2326 len 36864
29651 093/300 Read block 4120 len 8192
29652 094/300 Flush
29653 095/300 Read block 4324 len 36864
29654 096/300 Read block 4326 len 36864
29655 097/300 Write block 2139 len 4096 data: 5
29656 098/300 Write block 4680 len 28672 data: 6 5 4 5 6 6 6
29657 099/300 Read block 1976 len 32768
29658 100/300 Write block 108 len 24576 data: 4 3 3 4 4 3
29659 101/300 Write block 1164 len 40960 data: 8 7 7 7 6 8 7 7 7 10
29660 102/300 Read block 2183 len 40960
29661 103/300 Write block 3678 len 36864 data: 4 4 4 4 5 5 5 5 5
29662 104/300 Write block 3532 len 12288 data: 8 8 7
29663 105/300 Read block 458 len 36864
29664 106/300 Write block 1832 len 20480 data: 4 3 3 4 6
29665 107/300 Flush
29666 108/300 Write block 2058 len 20480 data: 5 4 4 5 4
29667 109/300 Read block 1815 len 28672
29668 110/300 Read block 4569 len 32768
29669 111/300 Write block 2870 len 36864 data: 5 6 6 5 5 3 4 4 5
29670 112/300 Read block 4881 len 32768
29671 113/300 Read block 1530 len 40960
29672 114/300 Write block 918 len 40960 data: 6 6 6 6 6 5 5 5 6 6
29673 115/300 Read block 2551 len 24576
29674 116/300 Read block 54 len 8192
29675 117/300 Read block 1656 len 28672
29676 118/300 Write block 3755 len 36864 data: 3 3 3 3 3 4 4 6 5
29677 119/300 Read block 31 len 40960
29678 120/300 Read block 1545 len 4096
29679 121/300 Read block 2723 len 20480
29680 122/300 Read block 1512 len 8192
29681 123/300 Write block 3321 len 8192 data: 5 6
29682 124/300 Write block 208 len 40960 data: 7 7 7 7 5 5 7 8 6 7
29683 125/300 Write block 4064 len 32768 data: 5 3 6 5 5 5 5 5
29684 126/300 Read block 3675 len 4096
29685 127/300 Read block 1249 len 4096
29686 128/300 Write block 2536 len 12288 data: 5 6 5
29687 129/300 Read block 4973 len 24576
29688 130/300 Read block 2749 len 28672
29689 131/300 Read block 565 len 16384
29690 132/300 Flush
29691 133/300 Read block 2063 len 32768
29692 134/300 Read block 1259 len 20480
29693 135/300 Read block 1564 len 40960
29694 136/300 Write block 866 len 24576 data: 4 4 3 3 6 7
29695 137/300 Read block 2075 len 8192
29696 138/300 Read block 3315 len 12288
29697 139/300 Write block 4624 len 40960 data: 5 5 6 8 8 8 8 9 9 9
29698 140/300 Write block 3512 len 16384 data: 3 3 3 5
29699 141/300 Flush
29700 142/300 Read block 2332 len 32768
29701 143/300 Read block 3895 len 20480
29702 144/300 Write block 3994 len 28672 data: 6 5 6 5 5 5 4
29703 145/300 Read block 2116 len 36864
29704 146/300 Read block 682 len 16384
29705 147/300 Read block 1020 len 40960
29706 148/300 Flush
29707 149/300 Write block 3359 len 4096 data: 3
29708 150/300 Write block 3325 len 4096 data: 7
29709 151/300 Read block 4472 len 16384
29710 152/300 Write block 2899 len 4096 data: 5
29711 153/300 Read block 2446 len 32768
29712 154/300 Write block 3697 len 24576 data: 3 3 4 3 3 4
29713 155/300 Write block 1942 len 36864 data: 6 5 5 4 5 6 5 3 3
29714 156/300 Write block 4015 len 36864 data: 2 2 2 4 4 4 4 4 4
29715 157/300 Write block 843 len 8192 data: 4 6
29716 158/300 Read block 1024 len 40960
29717 159/300 Flush
29718 160/300 Write block 907 len 32768 data: 5 6 3 4 5 6 6 5
29719 161/300 Read block 2481 len 32768
29720 162/300 Read block 2998 len 8192
29721 163/300 Write block 4793 len 28672 data: 4 5 5 5 5 5 3
29722 164/300 Read block 2066 len 24576
29723 165/300 Read block 2991 len 36864
29724 166/300 Read block 3766 len 40960
29725 167/300 Read block 2745 len 40960
29726 168/300 Write block 3550 len 40960 data: 5 5 3 3 3 2 3 4 4 6
29727 169/300 Write block 1622 len 4096 data: 3
29728 170/300 Read block 227 len 24576
29729 171/300 Write block 3263 len 8192 data: 2 2
29730 172/300 Read block 4669 len 4096
29731 173/300 Read block 849 len 36864
29732 174/300 Read block 416 len 24576
29733 175/300 Write block 2805 len 4096 data: 6
29734 176/300 Write block 516 len 24576 data: 5 5 5 5 6 7
29735 177/300 Read block 4259 len 16384
29736 178/300 Read block 3695 len 32768
29737 179/300 Read block 3163 len 12288
29738 180/300 Write block 9 len 16384 data: 5 6 5 4
29739 181/300 Write block 4070 len 24576 data: 6 6 4 3 3 4
29740 182/300 Read block 4824 len 40960
29741 183/300 Write block 4320 len 32768 data: 7 8 7 8 7 6 5 6
29742 184/300 Read block 4185 len 8192
29743 185/300 Write block 37 len 40960 data: 4 4 4 4 5 5 3 3 3 3
29744 186/300 Write block 1286 len 32768 data: 4 5 5 6 6 7 6 7
29745 187/300 Read block 4644 len 24576
29746 188/300 Write block 1954 len 20480 data: 4 5 4 4 4
29747 189/300 Read block 3790 len 12288
29748 190/300 Read block 1380 len 12288
29749 191/300 Flush
29750 192/300 Read block 1674 len 20480
29751 193/300 Read block 2436 len 32768
29752 194/300 Flush
29753 195/300 Write block 3197 len 12288 data: 4 4 6
29754 196/300 Flush
29755 197/300 Write block 594 len 20480 data: 4 4 4 4 5
29756 198/300 Read block 2549 len 8192
29757 199/300 Read block 1227 len 24576
29758 200/300 Flush
29759 201/300 Read block 4909 len 16384
29760 202/300 Write block 1426 len 8192 data: 6 6
29761 203/300 Read block 257 len 36864
29762 204/300 Read block 597 len 28672
29763 205/300 Flush
29764 206/300 Read block 4460 len 28672
29765 207/300 Flush
29766 208/300 Write block 1130 len 36864 data: 6 7 6 5 7 7 8 8 7
29767 209/300 Write block 2580 len 28672 data: 6 6 6 6 7 6 5
29768 210/300 Read block 1913 len 8192
29769 211/300 Flush
29770 212/300 Read block 275 len 8192
29771 213/300 Read block 3723 len 28672
29772 214/300 Flush
29773 215/300 Read block 1497 len 20480
29774 216/300 Read block 4035 len 12288
29775 217/300 Write block 620 len 28672 data: 5 5 5 6 6 7 5
29776 218/300 Read block 4890 len 40960
29777 219/300 Read block 685 len 32768
29778 220/300 Read block 156 len 12288
29779 221/300 Write block 640 len 36864 data: 2 5 5 5 6 6 5 6 7
29780 222/300 Read block 1580 len 12288
29781 223/300 Read block 4858 len 32768
29782 224/300 Read block 4931 len 4096
29783 225/300 Write block 3771 len 20480 data: 7 7 8 7 5
29784 226/300 Write block 3746 len 36864 data: 7 7 5 6 4 5 5 4 3
29785 227/300 Read block 715 len 4096
29786 228/300 Read block 1148 len 12288
29787 229/300 Write block 2931 len 32768 data: 6 5 5 5 3 3 5 6
29788 230/300 Write block 3035 len 36864 data: 7 7 7 6 6 4 4 3 3
29789 231/300 Flush
29790 232/300 Read block 577 len 24576
29791 233/300 Write block 4507 len 8192 data: 8 6
29792 234/300 Read block 823 len 32768
29793 235/300 Write block 2751 len 24576 data: 8 5 4 4 5 5
29794 236/300 Read block 1894 len 12288
29795 237/300 Write block 744 len 20480 data: 9 8 8 7 9
29796 238/300 Write block 1226 len 32768 data: 5 6 5 5 6 3 2 2
29797 239/300 Flush
29798 240/300 Read block 818 len 20480
29799 241/300 Read block 4471 len 12288
29800 242/300 Read block 3375 len 36864
29801 243/300 Read block 3296 len 16384
29802 244/300 Write block 2248 len 8192 data: 2 2
29803 245/300 Read block 1535 len 20480
29804 246/300 Write block 4754 len 32768 data: 4 5 9 10 9 9 7 6
29805 247/300 Read block 661 len 28672
29806 248/300 Read block 818 len 8192
29807 249/300 Write block 3279 len 4096 data: 4
29808 250/300 Write block 4232 len 36864 data: 6 6 5 5 5 5 3 3 3
29809 251/300 Write block 1147 len 24576 data: 5 6 6 5 8 9
29810 252/300 Flush
29811 253/300 Write block 3611 len 16384 data: 7 7 6 7
29812 254/300 Read block 2200 len 12288
29813 255/300 Read block 4580 len 8192
29814 256/300 Read block 3138 len 40960
29815 257/300 Write block 4887 len 40960 data: 13 13 13 12 11 9 9 9 9 8
29816 258/300 Read block 2224 len 4096
29817 259/300 Write block 4672 len 32768 data: 3 4 4 5 4 5 6 6
29818 260/300 Write block 2663 len 36864 data: 6 7 7 7 8 8 10 11 8
29819 261/300 Read block 3291 len 20480
29820 262/300 Read block 969 len 36864
29821 263/300 Flush
29822 264/300 Read block 2187 len 4096
29823 265/300 Write block 4996 len 12288 data: 4 4 4
29824 266/300 Write block 583 len 40960 data: 8 7 8 9 7 8 6 6 4 4
29825 267/300 Read block 1782 len 20480
29826 268/300 Read block 2682 len 20480
29827 269/300 Read block 4660 len 28672
29828 270/300 Read block 2366 len 20480
29829 271/300 Write block 2306 len 12288 data: 5 5 5
29830 272/300 Read block 520 len 24576
29831 273/300 Read block 3052 len 8192
29832 274/300 Read block 2267 len 40960
29833 275/300 Write block 1155 len 12288 data: 9 8 8
29834 276/300 Read block 1235 len 28672
29835 277/300 Read block 1414 len 32768
29836 278/300 Flush
29837 279/300 Read block 4744 len 28672
29838 280/300 Read block 2123 len 8192
29839 281/300 Read block 4337 len 4096
29840 282/300 Flush
29841 283/300 Read block 1086 len 40960
29842 284/300 Read block 2978 len 24576
29843 285/300 Flush
29844 286/300 Read block 3851 len 36864
29845 287/300 Read block 2651 len 20480
29846 288/300 Write block 947 len 4096 data: 5
29847 289/300 Read block 4978 len 16384
29848 290/300 Write block 79 len 24576 data: 8 7 8 6 6 4
29849 291/300 Read block 4437 len 24576
29850 292/300 Write block 20 len 40960 data: 6 6 6 5 5 7 5 3 3 2
29851 293/300 Read block 4317 len 28672
29852 294/300 Read block 4761 len 4096
29853 295/300 Read block 2384 len 32768
29854 296/300 Write block 288 len 32768 data: 4 4 5 5 6 5 4 6
29855 297/300 Write block 3759 len 24576 data: 4 5 5 7 6 4
29856 298/300 Flush
29857 299/300 Write block 2392 len 28672 data: 4 5 4 3 2 2 2
29858 300/300 Write block 26 len 4096 data: 6
29859 301/300 Read block 2772 len 20480
29860 Main task has work to do, go find it
29861 got dsc Start(2)
29862 start 2
29863 Feb 28 06:07:42.151 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: 667a4671-cae3-41ff-bc0b-6b75440a99fb, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
29864 Replay: started 2, returned:Ok(())
29865 ----------------------------------------------------------------
29866 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29867 States: Active Active Offline
29868 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29869 8245 Acked Read 7 Done Done Sent false
29870 8246 Acked Read 2 Done Done Sent false
29871 8247 Acked Write 9 Done Done Sent false
29872 8248 Acked Flush 0 Done Done Sent false
29873 8249 Acked Read 6 Done Done Sent false
29874 8250 Acked Read 6 Done Done Sent false
29875 8251 Acked Read 8 Done Done Sent false
29876 8252 Acked Flush 0 Done Done Sent false
29877 8253 Acked Write 3 Done Done Sent false
29878 8254 Acked Write 2 Done Done Sent false
29879 8255 Acked Flush 0 Done Done Sent false
29880 8256 Acked Read 9 Done Done Sent false
29881 8257 Acked Write 7 Done Done Sent false
29882 8258 Acked Read 10 Done Done Sent false
29883 8259 Acked Write 3 Done Done Sent false
29884 8260 Acked Write 3 Done Done Sent false
29885 8261 Acked Read 7 Done Done Sent false
29886 8262 Acked Read 8 Done Done Sent false
29887 8263 Acked Read 10 Done Done Sent false
29888 8264 Acked Read 3 Done Done Sent false
29889 8265 Acked Write 5 Done Done Sent false
29890 8266 Acked Read 2 Done Done Sent false
29891 8267 Acked Write 7 Done Done Sent false
29892 8268 Acked Write 4 Done Done Sent false
29893 8269 Acked Read 5 Done Done Sent false
29894 8270 Acked Write 6 Done Done Sent false
29895 8271 Acked Read 7 Done Done Sent false
29896 8272 Acked Write 1 Done Done Sent false
29897 8273 Acked Write 4 Done Done Sent false
29898 8274 Acked Write 4 Done Done Sent false
29899 8275 Acked Read 2 Done Done Sent false
29900 8276 Acked Write 1 Done Done Sent false
29901 8277 Acked Read 6 Done Done Sent false
29902 8278 Acked Write 4 Done Done Sent false
29903 8279 Acked Read 10 Done Done Sent false
29904 8280 Acked Write 1 Done Done Sent false
29905 8281 Acked Read 5 Done Done Sent false
29906 8282 Acked Read 6 Done Done Sent false
29907 8283 Acked Write 10 Done Done Sent false
29908 8284 Acked Read 5 Done Done Sent false
29909 8285 Acked Write 1 Done Done Sent false
29910 8286 Acked Write 9 Done Done Sent false
29911 8287 Acked Read 4 Done Done Sent false
29912 8288 Acked Write 10 Done Done Sent false
29913 8289 Acked Read 2 Done Done Sent false
29914 8290 Acked Read 2 Done Done Sent false
29915 8291 Acked Read 6 Done Done Sent false
29916 8292 Acked Flush 0 Done Done Sent false
29917 8293 Acked Read 1 Done Done Sent false
29918 8294 Acked Read 3 Done Done Sent false
29919 8295 Acked Flush 0 Done Done Sent false
29920 8296 Acked Write 3 Done Done Sent false
29921 8297 Acked Write 6 Done Done Sent false
29922 8298 Acked Read 3 Done Done Sent false
29923 8299 Acked Read 10 Done Done Sent false
29924 8300 Acked Read 9 Done Done Sent false
29925 8301 Acked Read 6 Done Done Sent false
29926 8302 Acked Write 7 Done Done Sent false
29927 8303 Acked Write 3 Done Done Sent false
29928 8304 Acked Read 8 Done Done Sent false
29929 8305 Acked Write 6 Done Done Sent false
29930 8306 Acked Read 4 Done Done Sent false
29931 8307 Acked Read 9 Done Done Sent false
29932 8308 Acked Flush 0 Done Done Sent false
29933 8309 Acked Write 5 Done Done Sent false
29934 8310 Acked Write 6 Done Done Sent false
29935 8311 Acked Write 4 Done Done Sent false
29936 8312 Acked Read 4 Done Done Sent false
29937 8313 Acked Write 8 Done Done Sent false
29938 8314 Acked Flush 0 Done Done Sent false
29939 8315 Acked Write 5 Done Done Sent false
29940 8316 Acked Write 5 Done Done Sent false
29941 8317 Acked Read 5 Done Done Sent false
29942 8318 Acked Write 1 Done Done Sent false
29943 8319 Acked Read 10 Done Done Sent false
29944 8320 Acked Flush 0 Done Done Sent false
29945 8321 Acked Read 7 Done Done Sent false
29946 8322 Acked Write 8 Done Done Sent false
29947 8323 Acked Read 4 Done Done Sent false
29948 8324 Acked Write 4 Done Done Sent false
29949 8325 Acked Read 3 Done Done Sent false
29950 8326 Acked Read 8 Done Done Sent false
29951 8327 Acked Read 4 Done Done Sent false
29952 8328 Acked Read 3 Done Done Sent false
29953 8329 Acked Write 5 Done Done Sent false
29954 8330 Acked Write 4 Done Done Sent false
29955 8331 Acked Read 8 Done Done Sent false
29956 8332 Acked Read 3 Done Done Sent false
29957 8333 Acked Write 3 Done Done Sent false
29958 8334 Acked Write 7 Done Done Sent false
29959 8335 Acked Read 9 Done Done Sent false
29960 8336 Acked Read 9 Done Done Sent false
29961 8337 Acked Read 2 Done Done Sent false
29962 8338 Acked Flush 0 Done Done Sent false
29963 8339 Acked Read 9 Done Done Sent false
29964 8340 Acked Read 9 Done Done Sent false
29965 8341 Acked Write 1 Done Done Sent false
29966 8342 Acked Write 7 Done Done Sent false
29967 8343 Acked Read 8 Done Done Sent false
29968 8344 Acked Write 6 Done Done Sent false
29969 8345 Acked Write 10 Done Done Sent false
29970 8346 Acked Read 10 Done Done Sent false
29971 8347 Acked Write 9 Done Done Sent false
29972 8348 Acked Write 3 Done Done Sent false
29973 8349 Acked Read 9 Done Done Sent false
29974 8350 Acked Write 5 Done Done Sent false
29975 8351 Acked Flush 0 Done Done Sent false
29976 8352 Acked Write 5 Done Done Sent false
29977 8353 Acked Read 7 Done Done Sent false
29978 8354 Acked Read 8 Done Done Sent false
29979 8355 Acked Write 9 Done Done Sent false
29980 8356 Acked Read 8 Done Done Sent false
29981 8357 Acked Read 10 Done Done Sent false
29982 8358 Acked Write 10 Done Done Sent false
29983 8359 Acked Read 6 Done Done Sent false
29984 8360 Acked Read 2 Done Done Sent false
29985 8361 Acked Read 7 Done Done Sent false
29986 8362 Acked Write 9 Done Done Sent false
29987 8363 Acked Read 10 Done Done Sent false
29988 8364 Acked Read 1 Done Done Sent false
29989 8365 Acked Read 5 Done Done Sent false
29990 8366 Acked Read 2 Done Done Sent false
29991 8367 Acked Write 2 Done Done Sent false
29992 8368 Acked Write 10 Done Done Sent false
29993 8369 Acked Write 8 Done Done Sent false
29994 8370 Acked Read 1 Done Done Sent false
29995 8371 Acked Read 1 Done Done Sent false
29996 8372 Acked Write 3 Done Done Sent false
29997 8373 Acked Read 6 Done Done Sent false
29998 8374 Acked Read 7 Done Done Sent false
29999 8375 Acked Read 4 Done Done Sent false
30000 8376 Acked Flush 0 Done Done Sent false
30001 8377 Acked Read 8 Done Done Sent false
30002 8378 Acked Read 5 Done Done Sent false
30003 8379 Acked Read 10 Done Done Sent false
30004 8380 Acked Write 6 Done Done Sent false
30005 8381 Acked Read 2 Done Done Sent false
30006 8382 Acked Read 3 Done Done Sent false
30007 8383 Acked Write 10 Done Done Sent false
30008 8384 Acked Write 4 Done Done Sent false
30009 8385 Acked Flush 0 Done Done Sent false
30010 8386 Acked Read 8 Done Done Sent false
30011 8387 Acked Read 5 Done Done Sent false
30012 8388 Acked Write 7 Done Done Sent false
30013 8389 Acked Read 9 Done Done Sent false
30014 8390 Acked Read 4 Done Done Sent false
30015 8391 Acked Read 10 Done Done Sent false
30016 8392 Acked Flush 0 Done Done Sent false
30017 8393 Acked Write 1 Done Done Sent false
30018 8394 Acked Write 1 Done Done Sent false
30019 8395 Acked Read 4 Done Done Sent false
30020 8396 Acked Write 1 Done Done Sent false
30021 8397 Acked Read 8 Done Done Sent false
30022 8398 Acked Write 6 Done Done Sent false
30023 8399 Acked Write 9 Done Done Sent false
30024 8400 Acked Write 9 Done Done Sent false
30025 8401 Acked Write 2 Done Done Sent false
30026 8402 Acked Read 10 Done Done Sent false
30027 8403 Acked Flush 0 Done Done Sent false
30028 8404 Acked Write 8 Done Done Sent false
30029 8405 Acked Read 8 Done Done Sent false
30030 8406 Acked Read 2 Done Done Sent false
30031 8407 Acked Write 7 Done Done Sent false
30032 8408 Acked Read 6 Done Done Sent false
30033 8409 Acked Read 9 Done Done Sent false
30034 8410 Acked Read 10 Done Done Sent false
30035 8411 Acked Read 10 Done Done Sent false
30036 8412 Acked Write 10 Done Done Sent false
30037 8413 Acked Write 1 Done Done Sent false
30038 8414 Acked Read 6 Done Done Sent false
30039 8415 Acked Write 2 Done Done Sent false
30040 8416 Acked Read 1 Done Done Sent false
30041 8417 Acked Read 9 Done Done Sent false
30042 8418 Acked Read 6 Done Done Sent false
30043 8419 Acked Write 1 Done Done Sent false
30044 8420 Acked Write 6 Done Done Sent false
30045 8421 Acked Read 4 Done Done Sent false
30046 8422 Acked Read 8 Done Done Sent false
30047 8423 Acked Read 3 Done Done Sent false
30048 8424 Acked Write 4 Done Done Sent false
30049 8425 Acked Write 6 Done Done Sent false
30050 8426 Acked Read 10 Done Done Sent false
30051 8427 Acked Write 8 Done Done Sent false
30052 8428 Acked Read 2 Done Done Sent false
30053 8429 Acked Write 10 Done Done Sent false
30054 8430 Acked Write 8 Done Done Sent false
30055 8431 Acked Read 6 Done Done Sent false
30056 8432 Acked Write 5 Done Done Sent false
30057 8433 Acked Read 3 Done Done Sent false
30058 8434 Acked Read 3 Done Done Sent false
30059 8435 Acked Flush 0 Done Done Sent false
30060 8436 Acked Read 5 Done Done Sent false
30061 8437 Acked Read 8 Done Done Sent false
30062 8438 Acked Flush 0 Done Done Sent false
30063 8439 Acked Write 3 Done Done Sent false
30064 8440 Acked Flush 0 Done Done Sent false
30065 8441 Acked Write 5 Done Done Sent false
30066 8442 Acked Read 2 Done Done Sent false
30067 8443 Acked Read 6 Done Done Sent false
30068 8444 Acked Flush 0 Done Done Sent false
30069 8445 Acked Read 4 Done Done Sent false
30070 8446 Acked Write 2 Done Done Sent false
30071 8447 Acked Read 9 Done Done Sent false
30072 8448 Acked Read 7 Done Done Sent false
30073 8449 Acked Flush 0 Done Done Sent false
30074 8450 Acked Read 7 Done Done Sent false
30075 8451 Acked Flush 0 Done Done Sent false
30076 8452 Acked Write 9 Done Done Sent false
30077 8453 Acked Write 7 Done Done Sent false
30078 8454 Acked Read 2 Done Done Sent false
30079 8455 Acked Flush 0 Done Done Sent false
30080 8456 Acked Read 2 Done Done Sent false
30081 8457 Acked Read 7 Done Done Sent false
30082 8458 Acked Flush 0 Done Done Sent false
30083 8459 Acked Read 5 Done Done Sent false
30084 8460 Acked Read 3 Done Done Sent false
30085 8461 Acked Write 7 Done Done Sent false
30086 8462 Acked Read 10 Done Done Sent false
30087 8463 Acked Read 8 Done Done Sent false
30088 8464 Acked Read 3 Done Done Sent false
30089 8465 Acked Write 9 Done Done Sent false
30090 8466 Acked Read 3 Done Done Sent false
30091 8467 Acked Read 8 Done Done Sent false
30092 8468 Acked Read 1 Done Done Sent false
30093 8469 Acked Write 5 Done Done Sent false
30094 8470 Acked Write 9 Done Done Sent false
30095 8471 Acked Read 1 Done Done Sent false
30096 8472 Acked Read 3 Done Done Sent false
30097 8473 Acked Write 8 Done Done Sent false
30098 8474 Acked Write 9 Done Done Sent false
30099 8475 Acked Flush 0 Done Done Sent false
30100 8476 Acked Read 6 Done Done Sent false
30101 8477 Acked Write 2 Done Done Sent false
30102 8478 Acked Read 8 Done Done Sent false
30103 8479 Acked Write 6 Done Done Sent false
30104 8480 Acked Read 3 Done Done Sent false
30105 8481 Acked Write 5 Done Done Sent false
30106 8482 Acked Write 8 Done Done Sent false
30107 8483 Acked Flush 0 Done Done Sent false
30108 8484 Acked Read 5 Done Done Sent false
30109 8485 Acked Read 3 Done Done Sent false
30110 8486 Acked Read 9 Done Done Sent false
30111 8487 Acked Read 4 Done Done Sent false
30112 8488 Acked Write 2 Done Done Sent false
30113 8489 Acked Read 5 Done Done Sent false
30114 8490 Acked Write 8 Done Done Sent false
30115 8491 Acked Read 7 Done Done Sent false
30116 8492 Acked Read 2 Done Done Sent false
30117 8493 Acked Write 1 Done Done Sent false
30118 8494 Acked Write 9 Done Done Sent false
30119 8495 Acked Write 6 Done Done Sent false
30120 8496 Acked Flush 0 Done Done Sent false
30121 8497 Acked Write 4 Done Done Sent false
30122 8498 Acked Read 3 Done Done Sent false
30123 8499 Acked Read 2 Done Done Sent false
30124 8500 Acked Read 10 Done Done Sent false
30125 8501 Acked Write 10 Done Done Sent false
30126 8502 Acked Read 1 Done Done Sent false
30127 8503 Acked Write 8 Done Done Sent false
30128 8504 Acked Write 9 Done Done Sent false
30129 8505 Acked Read 5 Done Done Sent false
30130 8506 Acked Read 9 Done Done Sent false
30131 8507 Acked Flush 0 Done Done Sent false
30132 8508 Acked Read 1 Done Done Sent false
30133 8509 Acked Write 3 Done Done Sent false
30134 8510 Acked Write 10 Done Done Sent false
30135 8511 Acked Read 5 Done Done Sent false
30136 8512 Acked Read 5 Done Done Sent false
30137 8513 Acked Read 7 Done Done Sent false
30138 8514 Acked Read 5 Done Done Sent false
30139 8515 Acked Write 3 Done Done Sent false
30140 8516 Acked Read 6 Done Done Sent false
30141 8517 Acked Read 2 Done Done Sent false
30142 8518 Acked Read 10 Done Done Sent false
30143 8519 Acked Write 3 Done Done Sent false
30144 8520 Acked Read 7 Done Done Sent false
30145 8521 Acked Read 8 Done Done Sent false
30146 8522 Acked Flush 0 Done Done Sent false
30147 8523 Acked Read 7 Done Done Sent false
30148 8524 Acked Read 2 Done Done Sent false
30149 8525 Acked Read 1 Done Done Sent false
30150 8526 Acked Flush 0 Done Done Sent false
30151 8527 Acked Read 10 Done Done Sent false
30152 8528 Acked Read 6 Done Done Sent false
30153 8529 Acked Flush 0 Done Done Sent false
30154 8530 Acked Read 9 Done Done Sent false
30155 8531 Acked Read 5 Done Done Sent false
30156 8532 Acked Write 1 Done Done Sent false
30157 8533 Acked Read 4 Done Done Sent false
30158 8534 Acked Write 6 Done Done Sent false
30159 8535 Acked Read 6 Done Done Sent false
30160 8536 Acked Write 10 Done Done Sent false
30161 8537 Acked Read 7 Done Done Sent false
30162 8538 Acked Read 1 Done Done Sent false
30163 8539 Acked Read 8 Done Done Sent false
30164 8540 Acked Write 8 Done Done Sent false
30165 8541 Acked Write 6 Done Done Sent false
30166 8542 Acked Flush 0 Done Done Sent false
30167 8543 Acked Write 7 Done Done Sent false
30168 8544 Acked Write 1 Done Done Sent false
30169 8545 Acked Read 5 Done Done Sent false
30170 STATES DS:0 DS:1 DS:2 TOTAL
30171 Sent 0 0 301 301
30172 Done 301 301 0 602
30173 Skipped 0 0 0 0
30174 Error 0 0 0 0
30175 Last Flush: Some(JobId(8542)) Some(JobId(8542)) Some(JobId(8244))
30176 Downstairs last five completed: 8244 8243 8242 8241 8240
30177 Upstairs last five completed: 8545 8544 8543 8542 8541
30178 CLIENT: Up:0 ds:301 act:2
301792025-02-28T06:07:43.014ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
30180 [8830] got start action
30181 [8830] I am going to restart
30182 Starting downstairs at port 8830
30183 Make output file at "/tmp/dsc/downstairs-8830.txt"
30184 [8830][2] reports Starting
30185 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1236)
30186 [8830][2] reports Running
301872025-02-28T06:07:44.015ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
301882025-02-28T06:07:45.018ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
301892025-02-28T06:07:46.151ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
30190 ----------------------------------------------------------------
30191 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30192 States: Active Active Offline
30193 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30194 8245 Acked Read 7 Done Done Sent false
30195 8246 Acked Read 2 Done Done Sent false
30196 8247 Acked Write 9 Done Done Sent false
30197 8248 Acked Flush 0 Done Done Sent false
30198 8249 Acked Read 6 Done Done Sent false
30199 8250 Acked Read 6 Done Done Sent false
30200 8251 Acked Read 8 Done Done Sent false
30201 8252 Acked Flush 0 Done Done Sent false
30202 8253 Acked Write 3 Done Done Sent false
30203 8254 Acked Write 2 Done Done Sent false
30204 8255 Acked Flush 0 Done Done Sent false
30205 8256 Acked Read 9 Done Done Sent false
30206 8257 Acked Write 7 Done Done Sent false
30207 8258 Acked Read 10 Done Done Sent false
30208 8259 Acked Write 3 Done Done Sent false
30209 8260 Acked Write 3 Done Done Sent false
30210 8261 Acked Read 7 Done Done Sent false
30211 8262 Acked Read 8 Done Done Sent false
30212 8263 Acked Read 10 Done Done Sent false
30213 8264 Acked Read 3 Done Done Sent false
30214 8265 Acked Write 5 Done Done Sent false
30215 8266 Acked Read 2 Done Done Sent false
30216 8267 Acked Write 7 Done Done Sent false
30217 8268 Acked Write 4 Done Done Sent false
30218 8269 Acked Read 5 Done Done Sent false
30219 8270 Acked Write 6 Done Done Sent false
30220 8271 Acked Read 7 Done Done Sent false
30221 8272 Acked Write 1 Done Done Sent false
30222 8273 Acked Write 4 Done Done Sent false
30223 8274 Acked Write 4 Done Done Sent false
30224 8275 Acked Read 2 Done Done Sent false
30225 8276 Acked Write 1 Done Done Sent false
30226 8277 Acked Read 6 Done Done Sent false
30227 8278 Acked Write 4 Done Done Sent false
30228 8279 Acked Read 10 Done Done Sent false
30229 8280 Acked Write 1 Done Done Sent false
30230 8281 Acked Read 5 Done Done Sent false
30231 8282 Acked Read 6 Done Done Sent false
30232 8283 Acked Write 10 Done Done Sent false
30233 8284 Acked Read 5 Done Done Sent false
30234 8285 Acked Write 1 Done Done Sent false
30235 8286 Acked Write 9 Done Done Sent false
30236 8287 Acked Read 4 Done Done Sent false
30237 8288 Acked Write 10 Done Done Sent false
30238 8289 Acked Read 2 Done Done Sent false
30239 8290 Acked Read 2 Done Done Sent false
30240 8291 Acked Read 6 Done Done Sent false
30241 8292 Acked Flush 0 Done Done Sent false
30242 8293 Acked Read 1 Done Done Sent false
30243 8294 Acked Read 3 Done Done Sent false
30244 8295 Acked Flush 0 Done Done Sent false
30245 8296 Acked Write 3 Done Done Sent false
30246 8297 Acked Write 6 Done Done Sent false
30247 8298 Acked Read 3 Done Done Sent false
30248 8299 Acked Read 10 Done Done Sent false
30249 8300 Acked Read 9 Done Done Sent false
30250 8301 Acked Read 6 Done Done Sent false
30251 8302 Acked Write 7 Done Done Sent false
30252 8303 Acked Write 3 Done Done Sent false
30253 8304 Acked Read 8 Done Done Sent false
30254 8305 Acked Write 6 Done Done Sent false
30255 8306 Acked Read 4 Done Done Sent false
30256 8307 Acked Read 9 Done Done Sent false
30257 8308 Acked Flush 0 Done Done Sent false
30258 8309 Acked Write 5 Done Done Sent false
30259 8310 Acked Write 6 Done Done Sent false
30260 8311 Acked Write 4 Done Done Sent false
30261 8312 Acked Read 4 Done Done Sent false
30262 8313 Acked Write 8 Done Done Sent false
30263 8314 Acked Flush 0 Done Done Sent false
30264 8315 Acked Write 5 Done Done Sent false
30265 8316 Acked Write 5 Done Done Sent false
30266 8317 Acked Read 5 Done Done Sent false
30267 8318 Acked Write 1 Done Done Sent false
30268 8319 Acked Read 10 Done Done Sent false
30269 8320 Acked Flush 0 Done Done Sent false
30270 8321 Acked Read 7 Done Done Sent false
30271 8322 Acked Write 8 Done Done Sent false
30272 8323 Acked Read 4 Done Done Sent false
30273 8324 Acked Write 4 Done Done Sent false
30274 8325 Acked Read 3 Done Done Sent false
30275 8326 Acked Read 8 Done Done Sent false
30276 8327 Acked Read 4 Done Done Sent false
30277 8328 Acked Read 3 Done Done Sent false
30278 8329 Acked Write 5 Done Done Sent false
30279 8330 Acked Write 4 Done Done Sent false
30280 8331 Acked Read 8 Done Done Sent false
30281 8332 Acked Read 3 Done Done Sent false
30282 8333 Acked Write 3 Done Done Sent false
30283 8334 Acked Write 7 Done Done Sent false
30284 8335 Acked Read 9 Done Done Sent false
30285 8336 Acked Read 9 Done Done Sent false
30286 8337 Acked Read 2 Done Done Sent false
30287 8338 Acked Flush 0 Done Done Sent false
30288 8339 Acked Read 9 Done Done Sent false
30289 8340 Acked Read 9 Done Done Sent false
30290 8341 Acked Write 1 Done Done Sent false
30291 8342 Acked Write 7 Done Done Sent false
30292 8343 Acked Read 8 Done Done Sent false
30293 8344 Acked Write 6 Done Done Sent false
30294 8345 Acked Write 10 Done Done Sent false
30295 8346 Acked Read 10 Done Done Sent false
30296 8347 Acked Write 9 Done Done Sent false
30297 8348 Acked Write 3 Done Done Sent false
30298 8349 Acked Read 9 Done Done Sent false
30299 8350 Acked Write 5 Done Done Sent false
30300 8351 Acked Flush 0 Done Done Sent false
30301 8352 Acked Write 5 Done Done Sent false
30302 8353 Acked Read 7 Done Done Sent false
30303 8354 Acked Read 8 Done Done Sent false
30304 8355 Acked Write 9 Done Done Sent false
30305 8356 Acked Read 8 Done Done Sent false
30306 8357 Acked Read 10 Done Done Sent false
30307 8358 Acked Write 10 Done Done Sent false
30308 8359 Acked Read 6 Done Done Sent false
30309 8360 Acked Read 2 Done Done Sent false
30310 8361 Acked Read 7 Done Done Sent false
30311 8362 Acked Write 9 Done Done Sent false
30312 8363 Acked Read 10 Done Done Sent false
30313 8364 Acked Read 1 Done Done Sent false
30314 8365 Acked Read 5 Done Done Sent false
30315 8366 Acked Read 2 Done Done Sent false
30316 8367 Acked Write 2 Done Done Sent false
30317 8368 Acked Write 10 Done Done Sent false
30318 8369 Acked Write 8 Done Done Sent false
30319 8370 Acked Read 1 Done Done Sent false
30320 8371 Acked Read 1 Done Done Sent false
30321 8372 Acked Write 3 Done Done Sent false
30322 8373 Acked Read 6 Done Done Sent false
30323 8374 Acked Read 7 Done Done Sent false
30324 8375 Acked Read 4 Done Done Sent false
30325 8376 Acked Flush 0 Done Done Sent false
30326 8377 Acked Read 8 Done Done Sent false
30327 8378 Acked Read 5 Done Done Sent false
30328 8379 Acked Read 10 Done Done Sent false
30329 8380 Acked Write 6 Done Done Sent false
30330 8381 Acked Read 2 Done Done Sent false
30331 8382 Acked Read 3 Done Done Sent false
30332 8383 Acked Write 10 Done Done Sent false
30333 8384 Acked Write 4 Done Done Sent false
30334 8385 Acked Flush 0 Done Done Sent false
30335 8386 Acked Read 8 Done Done Sent false
30336 8387 Acked Read 5 Done Done Sent false
30337 8388 Acked Write 7 Done Done Sent false
30338 8389 Acked Read 9 Done Done Sent false
30339 8390 Acked Read 4 Done Done Sent false
30340 8391 Acked Read 10 Done Done Sent false
30341 8392 Acked Flush 0 Done Done Sent false
30342 8393 Acked Write 1 Done Done Sent false
30343 8394 Acked Write 1 Done Done Sent false
30344 8395 Acked Read 4 Done Done Sent false
30345 8396 Acked Write 1 Done Done Sent false
30346 8397 Acked Read 8 Done Done Sent false
30347 8398 Acked Write 6 Done Done Sent false
30348 8399 Acked Write 9 Done Done Sent false
30349 8400 Acked Write 9 Done Done Sent false
30350 8401 Acked Write 2 Done Done Sent false
30351 8402 Acked Read 10 Done Done Sent false
30352 8403 Acked Flush 0 Done Done Sent false
30353 8404 Acked Write 8 Done Done Sent false
30354 8405 Acked Read 8 Done Done Sent false
30355 8406 Acked Read 2 Done Done Sent false
30356 8407 Acked Write 7 Done Done Sent false
30357 8408 Acked Read 6 Done Done Sent false
30358 8409 Acked Read 9 Done Done Sent false
30359 8410 Acked Read 10 Done Done Sent false
30360 8411 Acked Read 10 Done Done Sent false
30361 8412 Acked Write 10 Done Done Sent false
30362 8413 Acked Write 1 Done Done Sent false
30363 8414 Acked Read 6 Done Done Sent false
30364 8415 Acked Write 2 Done Done Sent false
30365 8416 Acked Read 1 Done Done Sent false
30366 8417 Acked Read 9 Done Done Sent false
30367 8418 Acked Read 6 Done Done Sent false
30368 8419 Acked Write 1 Done Done Sent false
30369 8420 Acked Write 6 Done Done Sent false
30370 8421 Acked Read 4 Done Done Sent false
30371 8422 Acked Read 8 Done Done Sent false
30372 8423 Acked Read 3 Done Done Sent false
30373 8424 Acked Write 4 Done Done Sent false
30374 8425 Acked Write 6 Done Done Sent false
30375 8426 Acked Read 10 Done Done Sent false
30376 8427 Acked Write 8 Done Done Sent false
30377 8428 Acked Read 2 Done Done Sent false
30378 8429 Acked Write 10 Done Done Sent false
30379 8430 Acked Write 8 Done Done Sent false
30380 8431 Acked Read 6 Done Done Sent false
30381 8432 Acked Write 5 Done Done Sent false
30382 8433 Acked Read 3 Done Done Sent false
30383 8434 Acked Read 3 Done Done Sent false
30384 8435 Acked Flush 0 Done Done Sent false
30385 8436 Acked Read 5 Done Done Sent false
30386 8437 Acked Read 8 Done Done Sent false
30387 8438 Acked Flush 0 Done Done Sent false
30388 8439 Acked Write 3 Done Done Sent false
30389 8440 Acked Flush 0 Done Done Sent false
30390 8441 Acked Write 5 Done Done Sent false
30391 8442 Acked Read 2 Done Done Sent false
30392 8443 Acked Read 6 Done Done Sent false
30393 8444 Acked Flush 0 Done Done Sent false
30394 8445 Acked Read 4 Done Done Sent false
30395 8446 Acked Write 2 Done Done Sent false
30396 8447 Acked Read 9 Done Done Sent false
30397 8448 Acked Read 7 Done Done Sent false
30398 8449 Acked Flush 0 Done Done Sent false
30399 8450 Acked Read 7 Done Done Sent false
30400 8451 Acked Flush 0 Done Done Sent false
30401 8452 Acked Write 9 Done Done Sent false
30402 8453 Acked Write 7 Done Done Sent false
30403 8454 Acked Read 2 Done Done Sent false
30404 8455 Acked Flush 0 Done Done Sent false
30405 8456 Acked Read 2 Done Done Sent false
30406 8457 Acked Read 7 Done Done Sent false
30407 8458 Acked Flush 0 Done Done Sent false
30408 8459 Acked Read 5 Done Done Sent false
30409 8460 Acked Read 3 Done Done Sent false
30410 8461 Acked Write 7 Done Done Sent false
30411 8462 Acked Read 10 Done Done Sent false
30412 8463 Acked Read 8 Done Done Sent false
30413 8464 Acked Read 3 Done Done Sent false
30414 8465 Acked Write 9 Done Done Sent false
30415 8466 Acked Read 3 Done Done Sent false
30416 8467 Acked Read 8 Done Done Sent false
30417 8468 Acked Read 1 Done Done Sent false
30418 8469 Acked Write 5 Done Done Sent false
30419 8470 Acked Write 9 Done Done Sent false
30420 8471 Acked Read 1 Done Done Sent false
30421 8472 Acked Read 3 Done Done Sent false
30422 8473 Acked Write 8 Done Done Sent false
30423 8474 Acked Write 9 Done Done Sent false
30424 8475 Acked Flush 0 Done Done Sent false
30425 8476 Acked Read 6 Done Done Sent false
30426 8477 Acked Write 2 Done Done Sent false
30427 8478 Acked Read 8 Done Done Sent false
30428 8479 Acked Write 6 Done Done Sent false
30429 8480 Acked Read 3 Done Done Sent false
30430 8481 Acked Write 5 Done Done Sent false
30431 8482 Acked Write 8 Done Done Sent false
30432 8483 Acked Flush 0 Done Done Sent false
30433 8484 Acked Read 5 Done Done Sent false
30434 8485 Acked Read 3 Done Done Sent false
30435 8486 Acked Read 9 Done Done Sent false
30436 8487 Acked Read 4 Done Done Sent false
30437 8488 Acked Write 2 Done Done Sent false
30438 8489 Acked Read 5 Done Done Sent false
30439 8490 Acked Write 8 Done Done Sent false
30440 8491 Acked Read 7 Done Done Sent false
30441 8492 Acked Read 2 Done Done Sent false
30442 8493 Acked Write 1 Done Done Sent false
30443 8494 Acked Write 9 Done Done Sent false
30444 8495 Acked Write 6 Done Done Sent false
30445 8496 Acked Flush 0 Done Done Sent false
30446 8497 Acked Write 4 Done Done Sent false
30447 8498 Acked Read 3 Done Done Sent false
30448 8499 Acked Read 2 Done Done Sent false
30449 8500 Acked Read 10 Done Done Sent false
30450 8501 Acked Write 10 Done Done Sent false
30451 8502 Acked Read 1 Done Done Sent false
30452 8503 Acked Write 8 Done Done Sent false
30453 8504 Acked Write 9 Done Done Sent false
30454 8505 Acked Read 5 Done Done Sent false
30455 8506 Acked Read 9 Done Done Sent false
30456 8507 Acked Flush 0 Done Done Sent false
30457 8508 Acked Read 1 Done Done Sent false
30458 8509 Acked Write 3 Done Done Sent false
30459 8510 Acked Write 10 Done Done Sent false
30460 8511 Acked Read 5 Done Done Sent false
30461 8512 Acked Read 5 Done Done Sent false
30462 8513 Acked Read 7 Done Done Sent false
30463 8514 Acked Read 5 Done Done Sent false
30464 8515 Acked Write 3 Done Done Sent false
30465 8516 Acked Read 6 Done Done Sent false
30466 8517 Acked Read 2 Done Done Sent false
30467 8518 Acked Read 10 Done Done Sent false
30468 8519 Acked Write 3 Done Done Sent false
30469 8520 Acked Read 7 Done Done Sent false
30470 8521 Acked Read 8 Done Done Sent false
30471 8522 Acked Flush 0 Done Done Sent false
30472 8523 Acked Read 7 Done Done Sent false
30473 8524 Acked Read 2 Done Done Sent false
30474 8525 Acked Read 1 Done Done Sent false
30475 8526 Acked Flush 0 Done Done Sent false
30476 8527 Acked Read 10 Done Done Sent false
30477 8528 Acked Read 6 Done Done Sent false
30478 8529 Acked Flush 0 Done Done Sent false
30479 8530 Acked Read 9 Done Done Sent false
30480 8531 Acked Read 5 Done Done Sent false
30481 8532 Acked Write 1 Done Done Sent false
30482 8533 Acked Read 4 Done Done Sent false
30483 8534 Acked Write 6 Done Done Sent false
30484 8535 Acked Read 6 Done Done Sent false
30485 8536 Acked Write 10 Done Done Sent false
30486 8537 Acked Read 7 Done Done Sent false
30487 8538 Acked Read 1 Done Done Sent false
30488 8539 Acked Read 8 Done Done Sent false
30489 8540 Acked Write 8 Done Done Sent false
30490 8541 Acked Write 6 Done Done Sent false
30491 8542 Acked Flush 0 Done Done Sent false
30492 8543 Acked Write 7 Done Done Sent false
30493 8544 Acked Write 1 Done Done Sent false
30494 8545 Acked Read 5 Done Done Sent false
30495 STATES DS:0 DS:1 DS:2 TOTAL
30496 Sent 0 0 301 301
30497 Done 301 301 0 602
30498 Skipped 0 0 0 0
30499 Error 0 0 0 0
30500 Last Flush: Some(JobId(8542)) Some(JobId(8542)) Some(JobId(8244))
30501 Downstairs last five completed: 8244 8243 8242 8241 8240
30502 Upstairs last five completed: 8545 8544 8543 8542 8541
30503 CLIENT: Up:0 ds:301 act:2
305042025-02-28T06:07:47.020ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
305052025-02-28T06:07:47.895ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
305062025-02-28T06:07:47.895ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49653) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
305072025-02-28T06:07:47.896ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
305082025-02-28T06:07:47.897ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
305092025-02-28T06:07:47.897ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
305102025-02-28T06:07:47.897ZINFOcrucible: send last flush ID to this DS: Some(JobId(8244)) = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
305112025-02-28T06:07:47.897ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
305122025-02-28T06:07:47.897ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(8244)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
305132025-02-28T06:07:47.897ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(8244)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
30514 ----------------------------------------------------------------
30515 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30516 Downstairs last five completed: 8546 8545 8544 8543 8542
30517 Upstairs last five completed: 8546 8545 8544 8543 8542
30518 CLIENT: Up:0 ds:0 act:3
30519 Replay: All jobs finished
30520 Main task has work to do, go find it
30521 got dsc Stop(1)
30522 stop 1
30523 Feb 28 06:07:50.156[8820] Got stop action so:false kr:false
30524 INFO request completed, latency_us: 362, response_code: 204, uri: /stop/cid/1, method: POST, req_id: d63e5b64-b763-497b-8931-99fedc8932cb, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
305252025-02-28T06:07:50.157ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
305262025-02-28T06:07:50.157ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
305272025-02-28T06:07:50.157ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
30528 Feb 28 06:07:50.157 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/1, method: GET, req_id: b9b6841f-e691-4373-b38c-2bc8151cd52e, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
305292025-02-28T06:07:50.157ZINFOcrucible: client task is exiting = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
30530 [8820] Exited with: ExitStatus(unix_wait_status(9))
30531 [8820][1] reports Exit
30532 Feb 28 06:07:54.159 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2a93ffde-7239-4bea-889d-3759fe7914f1, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
30533 001/300 Write block 4010 len 24576 data: 5 5 4 3 2 3
30534 002/300 Write block 2517 len 12288 data: 5 5 6
30535 003/300 Write block 823 len 12288 data: 6 3 2
30536 004/300 Read block 777 len 32768
30537 005/300 Write block 972 len 24576 data: 2 2 3 5 5 5
30538 006/300 Flush
30539 007/300 Read block 140 len 12288
30540 008/300 Write block 356 len 8192 data: 6 3
30541 009/300 Write block 2675 len 8192 data: 11 10
30542 010/300 Write block 1510 len 36864 data: 4 4 5 4 5 4 5 5 5
30543 011/300 Write block 2323 len 20480 data: 5 4 4 5 4
30544 012/300 Write block 4099 len 12288 data: 7 7 9
30545 013/300 Flush
30546 014/300 Write block 4948 len 16384 data: 3 3 5 7
30547 015/300 Write block 1066 len 24576 data: 7 5 6 7 6 6
30548 016/300 Read block 409 len 28672
30549 017/300 Write block 2996 len 8192 data: 6 6
30550 018/300 Flush
30551 019/300 Read block 2119 len 20480
30552 020/300 Read block 3229 len 20480
30553 021/300 Write block 4187 len 8192 data: 8 7
30554 022/300 Write block 766 len 4096 data: 2
30555 023/300 Write block 1773 len 20480 data: 3 4 4 2 2
30556 024/300 Read block 3336 len 40960
30557 025/300 Flush
30558 026/300 Write block 3415 len 20480 data: 3 4 7 7 7
30559 027/300 Read block 3662 len 20480
30560 028/300 Read block 2829 len 40960
30561 029/300 Read block 3097 len 36864
30562 030/300 Write block 2140 len 36864 data: 6 5 5 5 4 4 4 4 5
30563 031/300 Read block 1693 len 24576
30564 032/300 Read block 71 len 12288
30565 033/300 Read block 455 len 32768
30566 034/300 Read block 3764 len 32768
30567 035/300 Read block 4443 len 36864
30568 036/300 Read block 1220 len 12288
30569 037/300 Read block 2869 len 40960
30570 038/300 Write block 1358 len 4096 data: 8
30571 039/300 Read block 275 len 16384
30572 040/300 Read block 2509 len 16384
30573 041/300 Flush
30574 042/300 Write block 2245 len 28672 data: 3 3 3 3 3 2 2
30575 043/300 Read block 3059 len 32768
30576 044/300 Read block 477 len 8192
30577 045/300 Flush
30578 046/300 Read block 2051 len 20480
30579 047/300 Read block 1833 len 12288
30580 048/300 Read block 588 len 24576
30581 049/300 Read block 1501 len 4096
30582 050/300 Read block 2412 len 4096
30583 051/300 Write block 2122 len 16384 data: 7 7 8 7
30584 052/300 Flush
30585 053/300 Read block 1601 len 36864
30586 054/300 Read block 139 len 32768
30587 055/300 Read block 4666 len 8192
30588 056/300 Write block 4163 len 8192 data: 6 6
30589 057/300 Write block 2825 len 16384 data: 7 7 5 4
30590 058/300 Read block 970 len 4096
30591 059/300 Write block 4545 len 20480 data: 6 7 7 5 5
30592 060/300 Write block 1725 len 12288 data: 7 5 6
30593 061/300 Write block 667 len 8192 data: 6 7
30594 062/300 Write block 172 len 20480 data: 5 5 6 5 5
30595 063/300 Flush
30596 064/300 Read block 3904 len 36864
30597 065/300 Read block 137 len 40960
30598 066/300 Read block 1640 len 4096
30599 067/300 Read block 2634 len 32768
30600 068/300 Read block 1286 len 40960
30601 069/300 Write block 109 len 40960 data: 4 4 5 5 4 3 4 2 2 2
30602 070/300 Read block 3939 len 20480
30603 071/300 Read block 2344 len 20480
30604 072/300 Read block 1315 len 12288
30605 073/300 Read block 2042 len 16384
30606 074/300 Read block 4432 len 32768
30607 075/300 Read block 3341 len 4096
30608 076/300 Read block 2459 len 32768
30609 077/300 Read block 3775 len 4096
30610 078/300 Flush
30611 079/300 Write block 3922 len 12288 data: 4 2 2
30612 080/300 Write block 3200 len 40960 data: 6 7 7 7 7 8 7 8 7 6
30613 081/300 Write block 3032 len 4096 data: 6
30614 082/300 Write block 2888 len 4096 data: 5
30615 083/300 Read block 4542 len 20480
30616 084/300 Read block 2757 len 36864
30617 085/300 Read block 3097 len 8192
30618 086/300 Read block 4830 len 12288
30619 087/300 Write block 1927 len 36864 data: 5 4 5 5 5 6 5 6 4
30620 088/300 Write block 4886 len 32768 data: 9 14 14 14 13 12 10 10
30621 089/300 Read block 3139 len 32768
30622 090/300 Write block 770 len 20480 data: 6 5 6 5 5
30623 091/300 Write block 215 len 32768 data: 9 7 8 8 10 7 7 6
30624 092/300 Read block 1872 len 12288
30625 093/300 Write block 2713 len 28672 data: 9 9 9 10 9 7 5
30626 094/300 Write block 4387 len 16384 data: 6 7 6 8
30627 095/300 Write block 106 len 4096 data: 5
30628 096/300 Write block 336 len 12288 data: 5 6 7
30629 097/300 Write block 3560 len 20480 data: 4 4 3 4 5
30630 098/300 Read block 1887 len 32768
30631 099/300 Read block 3612 len 28672
30632 100/300 Read block 3343 len 24576
30633 101/300 Write block 4274 len 4096 data: 7
30634 102/300 Write block 4488 len 36864 data: 7 6 6 4 3 4 5 6 7
30635 103/300 Write block 1148 len 16384 data: 7 7 6 9
30636 104/300 Write block 867 len 16384 data: 5 4 4 7
30637 105/300 Read block 3165 len 8192
30638 106/300 Write block 3206 len 36864 data: 8 9 8 7 5 5 5 6 6
30639 107/300 Read block 2097 len 24576
30640 108/300 Read block 2786 len 20480
30641 109/300 Write block 1357 len 16384 data: 7 9 9 9
30642 110/300 Flush
30643 111/300 Read block 2663 len 32768
30644 112/300 Write block 3578 len 4096 data: 2
30645 113/300 Write block 1913 len 36864 data: 4 4 4 4 4 3 3 3 3
30646 114/300 Write block 1146 len 36864 data: 7 6 8 8 7 10 10 9 8
30647 115/300 Read block 344 len 36864
30648 116/300 Read block 4940 len 24576
30649 117/300 Read block 3732 len 8192
30650 118/300 Read block 2045 len 32768
30651 119/300 Write block 3303 len 40960 data: 6 6 7 7 8 8 7 5 5 6
30652 120/300 Read block 210 len 24576
30653 121/300 Write block 3600 len 32768 data: 3 2 3 3 4 5 5 6
30654 122/300 Write block 2379 len 40960 data: 3 3 3 3 3 4 4 3 3 4
30655 123/300 Write block 345 len 8192 data: 7 6
30656 124/300 Write block 1887 len 8192 data: 4 4
30657 125/300 Read block 428 len 36864
30658 126/300 Write block 3131 len 20480 data: 4 4 4 4 4
30659 127/300 Read block 1637 len 20480
30660 128/300 Flush
30661 129/300 Flush
30662 130/300 Read block 2832 len 16384
30663 131/300 Write block 877 len 8192 data: 5 4
30664 132/300 Read block 4483 len 40960
30665 133/300 Read block 3733 len 28672
30666 134/300 Write block 1918 len 4096 data: 4
30667 135/300 Read block 91 len 4096
30668 136/300 Flush
30669 137/300 Read block 862 len 20480
30670 138/300 Read block 3425 len 28672
30671 139/300 Write block 656 len 40960 data: 4 5 6 7 7 7 8 7 7 7
30672 140/300 Read block 1311 len 20480
30673 141/300 Read block 3067 len 40960
30674 142/300 Flush
30675 143/300 Read block 3684 len 4096
30676 144/300 Write block 2722 len 8192 data: 4 3
30677 145/300 Read block 967 len 32768
30678 146/300 Read block 906 len 4096
30679 147/300 Read block 2268 len 4096
30680 148/300 Read block 3996 len 20480
30681 149/300 Read block 2559 len 32768
30682 150/300 Read block 457 len 12288
30683 151/300 Read block 4543 len 20480
30684 152/300 Write block 2460 len 4096 data: 6
30685 153/300 Write block 2229 len 12288 data: 6 8 10
30686 154/300 Write block 388 len 16384 data: 3 3 4 4
30687 155/300 Read block 202 len 32768
30688 156/300 Write block 451 len 16384 data: 9 7 8 8
30689 157/300 Read block 1388 len 8192
30690 158/300 Read block 3214 len 20480
30691 159/300 Write block 1720 len 32768 data: 5 6 8 8 8 8 6 7
30692 160/300 Read block 1338 len 16384
30693 161/300 Read block 1416 len 16384
30694 162/300 Read block 1688 len 20480
30695 163/300 Read block 3892 len 4096
30696 164/300 Write block 1661 len 4096 data: 6
30697 165/300 Flush
30698 166/300 Read block 4091 len 8192
30699 167/300 Read block 2550 len 16384
30700 168/300 Write block 4187 len 20480 data: 9 8 7 9 9
30701 169/300 Read block 2027 len 4096
30702 170/300 Read block 264 len 12288
30703 171/300 Write block 1042 len 24576 data: 7 7 7 7 6 5
30704 172/300 Read block 287 len 28672
30705 173/300 Read block 163 len 36864
30706 174/300 Read block 4003 len 4096
30707 175/300 Read block 4817 len 28672
30708 176/300 Read block 2402 len 20480
30709 177/300 Read block 358 len 20480
30710 178/300 Read block 866 len 36864
30711 179/300 Write block 1222 len 28672 data: 6 6 6 5 6 7 6
30712 180/300 Flush
30713 181/300 Read block 2775 len 24576
30714 182/300 Read block 2023 len 12288
30715 183/300 Write block 2729 len 28672 data: 5 8 6 6 7 5 6
30716 184/300 Read block 4203 len 24576
30717 185/300 Read block 905 len 16384
30718 186/300 Read block 4339 len 20480
30719 187/300 Write block 1462 len 32768 data: 4 4 4 4 5 7 6 7
30720 188/300 Read block 195 len 40960
30721 189/300 Read block 3993 len 8192
30722 190/300 Write block 173 len 36864 data: 6 7 6 6 5 5 2 2 2
30723 191/300 Write block 4685 len 8192 data: 7 7
30724 192/300 Write block 4500 len 12288 data: 7 9 7
30725 193/300 Read block 4551 len 20480
30726 194/300 Read block 3386 len 20480
30727 195/300 Write block 4963 len 12288 data: 5 6 7
30728 196/300 Flush
30729 197/300 Read block 4256 len 8192
30730 198/300 Write block 3103 len 16384 data: 8 7 8 7
30731 199/300 Flush
30732 200/300 Write block 1983 len 8192 data: 7 7
30733 201/300 Write block 32 len 4096 data: 3
30734 202/300 Read block 2509 len 24576
30735 203/300 Read block 2057 len 4096
30736 204/300 Read block 871 len 8192
30737 205/300 Read block 3191 len 20480
30738 206/300 Write block 1458 len 28672 data: 5 7 6 4 5 5 5
30739 207/300 Write block 3680 len 12288 data: 5 5 6
30740 208/300 Read block 3663 len 4096
30741 209/300 Read block 952 len 36864
30742 210/300 Write block 4375 len 12288 data: 6 6 7
30743 211/300 Read block 658 len 16384
30744 212/300 Write block 1865 len 16384 data: 5 5 5 6
30745 213/300 Read block 564 len 40960
30746 214/300 Write block 3854 len 32768 data: 6 5 5 3 2 3 4 5
30747 215/300 Write block 2772 len 12288 data: 4 4 4
30748 216/300 Flush
30749 217/300 Read block 3472 len 8192
30750 218/300 Read block 2610 len 28672
30751 219/300 Write block 1410 len 12288 data: 6 6 5
30752 220/300 Write block 1876 len 8192 data: 7 9
30753 221/300 Write block 4880 len 32768 data: 6 6 7 8 7 7 10 15
30754 222/300 Read block 1990 len 8192
30755 223/300 Read block 2411 len 24576
30756 224/300 Read block 2904 len 36864
30757 225/300 Flush
30758 226/300 Read block 4573 len 4096
30759 227/300 Read block 4858 len 28672
30760 228/300 Read block 498 len 16384
30761 229/300 Read block 705 len 28672
30762 230/300 Write block 2526 len 32768 data: 7 7 6 7 5 3 3 3
30763 231/300 Write block 4612 len 4096 data: 5
30764 232/300 Write block 4189 len 16384 data: 8 10 10 8
30765 233/300 Write block 1184 len 36864 data: 5 5 5 5 4 4 5 4 4
30766 234/300 Write block 901 len 20480 data: 6 7 5 4 5
30767 235/300 Read block 176 len 12288
30768 236/300 Write block 1522 len 36864 data: 6 5 6 6 6 7 6 7 5
30769 237/300 Read block 2254 len 24576
30770 238/300 Read block 538 len 40960
30771 239/300 Write block 4606 len 4096 data: 5
30772 240/300 Read block 3215 len 36864
30773 241/300 Write block 395 len 8192 data: 4 3
30774 242/300 Read block 2326 len 4096
30775 243/300 Read block 2789 len 28672
30776 244/300 Read block 3119 len 32768
30777 245/300 Read block 2510 len 8192
30778 246/300 Write block 2650 len 16384 data: 7 8 4 5
30779 247/300 Flush
30780 248/300 Read block 4447 len 28672
30781 249/300 Write block 3715 len 28672 data: 5 5 7 9 8 7 7
30782 250/300 Write block 429 len 36864 data: 5 6 6 6 6 6 7 9 8
30783 251/300 Read block 1076 len 40960
30784 252/300 Read block 1854 len 12288
30785 253/300 Write block 3747 len 8192 data: 8 6
30786 254/300 Write block 2156 len 20480 data: 4 3 3 3 3
30787 255/300 Flush
30788 256/300 Flush
30789 257/300 Write block 673 len 16384 data: 8 6 6 7
30790 258/300 Read block 4221 len 40960
30791 259/300 Read block 1959 len 36864
30792 260/300 Flush
30793 261/300 Write block 1682 len 40960 data: 7 8 8 6 7 6 5 4 4 5
30794 262/300 Write block 3909 len 36864 data: 5 4 4 4 4 5 5 5 4
30795 263/300 Read block 1831 len 24576
30796 264/300 Read block 3429 len 32768
30797 265/300 Read block 1970 len 16384
30798 266/300 Read block 4317 len 20480
30799 267/300 Read block 152 len 28672
30800 268/300 Read block 1561 len 32768
30801 269/300 Read block 1294 len 28672
30802 270/300 Write block 1978 len 36864 data: 3 3 3 4 6 8 8 6 6
30803 271/300 Read block 544 len 32768
30804 272/300 Read block 4346 len 4096
30805 273/300 Write block 3051 len 24576 data: 3 2 3 3 3 5
30806 274/300 Write block 2580 len 4096 data: 7
30807 275/300 Read block 192 len 16384
30808 276/300 Write block 4491 len 8192 data: 5 4
30809 277/300 Read block 3806 len 36864
30810 278/300 Read block 3718 len 32768
30811 279/300 Read block 2979 len 36864
30812 280/300 Read block 4460 len 8192
30813 281/300 Write block 1483 len 16384 data: 6 6 7 7
30814 282/300 Read block 398 len 40960
30815 283/300 Read block 506 len 40960
30816 284/300 Write block 4532 len 8192 data: 4 4
30817 285/300 Write block 3214 len 12288 data: 7 7 7
30818 286/300 Read block 4819 len 8192
30819 287/300 Read block 1178 len 28672
30820 288/300 Write block 1113 len 16384 data: 2 2 2 3
30821 289/300 Read block 266 len 16384
30822 290/300 Read block 373 len 40960
30823 291/300 Read block 839 len 8192
30824 292/300 Read block 3036 len 40960
30825 293/300 Read block 2282 len 8192
30826 294/300 Write block 4826 len 32768 data: 6 6 9 9 11 9 10 7
30827 295/300 Read block 3310 len 12288
30828 296/300 Read block 2143 len 20480
30829 297/300 Read block 1761 len 40960
30830 298/300 Write block 3899 len 32768 data: 3 3 3 3 7 7 6 5
30831 299/300 Write block 18 len 32768 data: 5 5 7 7 7 6 6 8
30832 300/300 Write block 4518 len 32768 data: 5 5 7 7 6 6 8 7
30833 301/300 Read block 4134 len 36864
30834 Main task has work to do, go find it
30835 got dsc Start(1)
30836 start 1
30837 Feb 28 06:07:54.405 INFO request completed, latency_us: 177, response_code: 204, uri: /start/cid/1, method: POST, req_id: a526873a-9e03-48c2-875e-2cc1b43fcf85, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
30838 Replay: started 1, returned:Ok(())
30839 ----------------------------------------------------------------
30840 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30841 States: Active Offline Active
30842 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30843 8547 Acked Write 6 Done Sent Done false
30844 8548 Acked Write 3 Done Sent Done false
30845 8549 Acked Write 3 Done Sent Done false
30846 8550 Acked Read 8 Done Sent Done false
30847 8551 Acked Write 6 Done Sent Done false
30848 8552 Acked Flush 0 Done Sent Done false
30849 8553 Acked Read 3 Done Sent Done false
30850 8554 Acked Write 2 Done Sent Done false
30851 8555 Acked Write 2 Done Sent Done false
30852 8556 Acked Write 9 Done Sent Done false
30853 8557 Acked Write 5 Done Sent Done false
30854 8558 Acked Write 3 Done Sent Done false
30855 8559 Acked Flush 0 Done Sent Done false
30856 8560 Acked Write 4 Done Sent Done false
30857 8561 Acked Write 6 Done Sent Done false
30858 8562 Acked Read 7 Done Sent Done false
30859 8563 Acked Write 2 Done Sent Done false
30860 8564 Acked Flush 0 Done Sent Done false
30861 8565 Acked Read 5 Done Sent Done false
30862 8566 Acked Read 5 Done Sent Done false
30863 8567 Acked Write 2 Done Sent Done false
30864 8568 Acked Write 1 Done Sent Done false
30865 8569 Acked Write 5 Done Sent Done false
30866 8570 Acked Read 10 Done Sent Done false
30867 8571 Acked Flush 0 Done Sent Done false
30868 8572 Acked Write 5 Done Sent Done false
30869 8573 Acked Read 5 Done Sent Done false
30870 8574 Acked Read 10 Done Sent Done false
30871 8575 Acked Read 9 Done Sent Done false
30872 8576 Acked Write 9 Done Sent Done false
30873 8577 Acked Read 6 Done Sent Done false
30874 8578 Acked Read 3 Done Sent Done false
30875 8579 Acked Read 8 Done Sent Done false
30876 8580 Acked Read 8 Done Sent Done false
30877 8581 Acked Read 9 Done Sent Done false
30878 8582 Acked Read 3 Done Sent Done false
30879 8583 Acked Read 10 Done Sent Done false
30880 8584 Acked Write 1 Done Sent Done false
30881 8585 Acked Read 4 Done Sent Done false
30882 8586 Acked Read 4 Done Sent Done false
30883 8587 Acked Flush 0 Done Sent Done false
30884 8588 Acked Write 7 Done Sent Done false
30885 8589 Acked Read 8 Done Sent Done false
30886 8590 Acked Read 2 Done Sent Done false
30887 8591 Acked Flush 0 Done Sent Done false
30888 8592 Acked Read 5 Done Sent Done false
30889 8593 Acked Read 3 Done Sent Done false
30890 8594 Acked Read 6 Done Sent Done false
30891 8595 Acked Read 1 Done Sent Done false
30892 8596 Acked Read 1 Done Sent Done false
30893 8597 Acked Write 4 Done Sent Done false
30894 8598 Acked Flush 0 Done Sent Done false
30895 8599 Acked Read 9 Done Sent Done false
30896 8600 Acked Read 8 Done Sent Done false
30897 8601 Acked Read 2 Done Sent Done false
30898 8602 Acked Write 2 Done Sent Done false
30899 8603 Acked Write 4 Done Sent Done false
30900 8604 Acked Read 1 Done Sent Done false
30901 8605 Acked Write 5 Done Sent Done false
30902 8606 Acked Write 3 Done Sent Done false
30903 8607 Acked Write 2 Done Sent Done false
30904 8608 Acked Write 5 Done Sent Done false
30905 8609 Acked Flush 0 Done Sent Done false
30906 8610 Acked Read 9 Done Sent Done false
30907 8611 Acked Read 10 Done Sent Done false
30908 8612 Acked Read 1 Done Sent Done false
30909 8613 Acked Read 8 Done Sent Done false
30910 8614 Acked Read 10 Done Sent Done false
30911 8615 Acked Write 10 Done Sent Done false
30912 8616 Acked Read 5 Done Sent Done false
30913 8617 Acked Read 5 Done Sent Done false
30914 8618 Acked Read 3 Done Sent Done false
30915 8619 Acked Read 4 Done Sent Done false
30916 8620 Acked Read 8 Done Sent Done false
30917 8621 Acked Read 1 Done Sent Done false
30918 8622 Acked Read 8 Done Sent Done false
30919 8623 Acked Read 1 Done Sent Done false
30920 8624 Acked Flush 0 Done Sent Done false
30921 8625 Acked Write 3 Done Sent Done false
30922 8626 Acked Write 10 Done Sent Done false
30923 8627 Acked Write 1 Done Sent Done false
30924 8628 Acked Write 1 Done Sent Done false
30925 8629 Acked Read 5 Done Sent Done false
30926 8630 Acked Read 9 Done Sent Done false
30927 8631 Acked Read 2 Done Sent Done false
30928 8632 Acked Read 3 Done Sent Done false
30929 8633 Acked Write 9 Done Sent Done false
30930 8634 Acked Write 8 Done Sent Done false
30931 8635 Acked Read 8 Done Sent Done false
30932 8636 Acked Write 5 Done Sent Done false
30933 8637 Acked Write 8 Done Sent Done false
30934 8638 Acked Read 3 Done Sent Done false
30935 8639 Acked Write 7 Done Sent Done false
30936 8640 Acked Write 4 Done Sent Done false
30937 8641 Acked Write 1 Done Sent Done false
30938 8642 Acked Write 3 Done Sent Done false
30939 8643 Acked Write 5 Done Sent Done false
30940 8644 Acked Read 8 Done Sent Done false
30941 8645 Acked Read 7 Done Sent Done false
30942 8646 Acked Read 6 Done Sent Done false
30943 8647 Acked Write 1 Done Sent Done false
30944 8648 Acked Write 9 Done Sent Done false
30945 8649 Acked Write 4 Done Sent Done false
30946 8650 Acked Write 4 Done Sent Done false
30947 8651 Acked Read 2 Done Sent Done false
30948 8652 Acked Write 9 Done Sent Done false
30949 8653 Acked Read 6 Done Sent Done false
30950 8654 Acked Read 5 Done Sent Done false
30951 8655 Acked Write 4 Done Sent Done false
30952 8656 Acked Flush 0 Done Sent Done false
30953 8657 Acked Read 8 Done Sent Done false
30954 8658 Acked Write 1 Done Sent Done false
30955 8659 Acked Write 9 Done Sent Done false
30956 8660 Acked Write 9 Done Sent Done false
30957 8661 Acked Read 9 Done Sent Done false
30958 8662 Acked Read 6 Done Sent Done false
30959 8663 Acked Read 2 Done Sent Done false
30960 8664 Acked Read 8 Done Sent Done false
30961 8665 Acked Write 10 Done Sent Done false
30962 8666 Acked Read 6 Done Sent Done false
30963 8667 Acked Write 8 Done Sent Done false
30964 8668 Acked Write 10 Done Sent Done false
30965 8669 Acked Write 2 Done Sent Done false
30966 8670 Acked Write 2 Done Sent Done false
30967 8671 Acked Read 9 Done Sent Done false
30968 8672 Acked Write 5 Done Sent Done false
30969 8673 Acked Read 5 Done Sent Done false
30970 8674 Acked Flush 0 Done Sent Done false
30971 8675 Acked Flush 0 Done Sent Done false
30972 8676 Acked Read 4 Done Sent Done false
30973 8677 Acked Write 2 Done Sent Done false
30974 8678 Acked Read 10 Done Sent Done false
30975 8679 Acked Read 7 Done Sent Done false
30976 8680 Acked Write 1 Done Sent Done false
30977 8681 Acked Read 1 Done Sent Done false
30978 8682 Acked Flush 0 Done Sent Done false
30979 8683 Acked Read 5 Done Sent Done false
30980 8684 Acked Read 7 Done Sent Done false
30981 8685 Acked Write 10 Done Sent Done false
30982 8686 Acked Read 5 Done Sent Done false
30983 8687 Acked Read 10 Done Sent Done false
30984 8688 Acked Flush 0 Done Sent Done false
30985 8689 Acked Read 1 Done Sent Done false
30986 8690 Acked Write 2 Done Sent Done false
30987 8691 Acked Read 8 Done Sent Done false
30988 8692 Acked Read 1 Done Sent Done false
30989 8693 Acked Read 1 Done Sent Done false
30990 8694 Acked Read 5 Done Sent Done false
30991 8695 Acked Read 8 Done Sent Done false
30992 8696 Acked Read 3 Done Sent Done false
30993 8697 Acked Read 5 Done Sent Done false
30994 8698 Acked Write 1 Done Sent Done false
30995 8699 Acked Write 3 Done Sent Done false
30996 8700 Acked Write 4 Done Sent Done false
30997 8701 Acked Read 8 Done Sent Done false
30998 8702 Acked Write 4 Done Sent Done false
30999 8703 Acked Read 2 Done Sent Done false
31000 8704 Acked Read 5 Done Sent Done false
31001 8705 Acked Write 8 Done Sent Done false
31002 8706 Acked Read 4 Done Sent Done false
31003 8707 Acked Read 4 Done Sent Done false
31004 8708 Acked Read 5 Done Sent Done false
31005 8709 Acked Read 1 Done Sent Done false
31006 8710 Acked Write 1 Done Sent Done false
31007 8711 Acked Flush 0 Done Sent Done false
31008 8712 Acked Read 2 Done Sent Done false
31009 8713 Acked Read 4 Done Sent Done false
31010 8714 Acked Write 5 Done Sent Done false
31011 8715 Acked Read 1 Done Sent Done false
31012 8716 Acked Read 3 Done Sent Done false
31013 8717 Acked Write 6 Done Sent Done false
31014 8718 Acked Read 7 Done Sent Done false
31015 8719 Acked Read 9 Done Sent Done false
31016 8720 Acked Read 1 Done Sent Done false
31017 8721 Acked Read 7 Done Sent Done false
31018 8722 Acked Read 5 Done Sent Done false
31019 8723 Acked Read 5 Done Sent Done false
31020 8724 Acked Read 9 Done Sent Done false
31021 8725 Acked Write 7 Done Sent Done false
31022 8726 Acked Flush 0 Done Sent Done false
31023 8727 Acked Read 6 Done Sent Done false
31024 8728 Acked Read 3 Done Sent Done false
31025 8729 Acked Write 7 Done Sent Done false
31026 8730 Acked Read 6 Done Sent Done false
31027 8731 Acked Read 4 Done Sent Done false
31028 8732 Acked Read 5 Done Sent Done false
31029 8733 Acked Write 8 Done Sent Done false
31030 8734 Acked Read 10 Done Sent Done false
31031 8735 Acked Read 2 Done Sent Done false
31032 8736 Acked Write 9 Done Sent Done false
31033 8737 Acked Write 2 Done Sent Done false
31034 8738 Acked Write 3 Done Sent Done false
31035 8739 Acked Read 5 Done Sent Done false
31036 8740 Acked Read 5 Done Sent Done false
31037 8741 Acked Write 3 Done Sent Done false
31038 8742 Acked Flush 0 Done Sent Done false
31039 8743 Acked Read 2 Done Sent Done false
31040 8744 Acked Write 4 Done Sent Done false
31041 8745 Acked Flush 0 Done Sent Done false
31042 8746 Acked Write 2 Done Sent Done false
31043 8747 Acked Write 1 Done Sent Done false
31044 8748 Acked Read 6 Done Sent Done false
31045 8749 Acked Read 1 Done Sent Done false
31046 8750 Acked Read 2 Done Sent Done false
31047 8751 Acked Read 5 Done Sent Done false
31048 8752 Acked Write 7 Done Sent Done false
31049 8753 Acked Write 3 Done Sent Done false
31050 8754 Acked Read 1 Done Sent Done false
31051 8755 Acked Read 9 Done Sent Done false
31052 8756 Acked Write 3 Done Sent Done false
31053 8757 Acked Read 4 Done Sent Done false
31054 8758 Acked Write 4 Done Sent Done false
31055 8759 Acked Read 10 Done Sent Done false
31056 8760 Acked Write 8 Done Sent Done false
31057 8761 Acked Write 3 Done Sent Done false
31058 8762 Acked Flush 0 Done Sent Done false
31059 8763 Acked Read 2 Done Sent Done false
31060 8764 Acked Read 7 Done Sent Done false
31061 8765 Acked Write 3 Done Sent Done false
31062 8766 Acked Write 2 Done Sent Done false
31063 8767 Acked Write 8 Done Sent Done false
31064 8768 Acked Read 2 Done Sent Done false
31065 8769 Acked Read 6 Done Sent Done false
31066 8770 Acked Read 9 Done Sent Done false
31067 8771 Acked Flush 0 Done Sent Done false
31068 8772 Acked Read 1 Done Sent Done false
31069 8773 Acked Read 7 Done Sent Done false
31070 8774 Acked Read 4 Done Sent Done false
31071 8775 Acked Read 7 Done Sent Done false
31072 8776 Acked Write 8 Done Sent Done false
31073 8777 Acked Write 1 Done Sent Done false
31074 8778 Acked Write 4 Done Sent Done false
31075 8779 Acked Write 9 Done Sent Done false
31076 8780 Acked Write 5 Done Sent Done false
31077 8781 Acked Read 3 Done Sent Done false
31078 8782 Acked Write 9 Done Sent Done false
31079 8783 Acked Read 6 Done Sent Done false
31080 8784 Acked Read 10 Done Sent Done false
31081 8785 Acked Write 1 Done Sent Done false
31082 8786 Acked Read 9 Done Sent Done false
31083 8787 Acked Write 2 Done Sent Done false
31084 8788 Acked Read 1 Done Sent Done false
31085 8789 Acked Read 7 Done Sent Done false
31086 8790 Acked Read 8 Done Sent Done false
31087 8791 Acked Read 2 Done Sent Done false
31088 8792 Acked Write 4 Done Sent Done false
31089 8793 Acked Flush 0 Done Sent Done false
31090 8794 Acked Read 7 Done Sent Done false
31091 8795 Acked Write 7 Done Sent Done false
31092 8796 Acked Write 9 Done Sent Done false
31093 8797 Acked Read 10 Done Sent Done false
31094 8798 Acked Read 3 Done Sent Done false
31095 8799 Acked Write 2 Done Sent Done false
31096 8800 Acked Write 5 Done Sent Done false
31097 8801 Acked Flush 0 Done Sent Done false
31098 8802 Acked Flush 0 Done Sent Done false
31099 8803 Acked Write 4 Done Sent Done false
31100 8804 Acked Read 10 Done Sent Done false
31101 8805 Acked Read 9 Done Sent Done false
31102 8806 Acked Flush 0 Done Sent Done false
31103 8807 Acked Write 10 Done Sent Done false
31104 8808 Acked Write 9 Done Sent Done false
31105 8809 Acked Read 6 Done Sent Done false
31106 8810 Acked Read 8 Done Sent Done false
31107 8811 Acked Read 4 Done Sent Done false
31108 8812 Acked Read 5 Done Sent Done false
31109 8813 Acked Read 7 Done Sent Done false
31110 8814 Acked Read 8 Done Sent Done false
31111 8815 Acked Read 7 Done Sent Done false
31112 8816 Acked Write 9 Done Sent Done false
31113 8817 Acked Read 8 Done Sent Done false
31114 8818 Acked Read 1 Done Sent Done false
31115 8819 Acked Write 6 Done Sent Done false
31116 8820 Acked Write 1 Done Sent Done false
31117 8821 Acked Read 4 Done Sent Done false
31118 8822 Acked Write 2 Done Sent Done false
31119 8823 Acked Read 9 Done Sent Done false
31120 8824 Acked Read 8 Done Sent Done false
31121 8825 Acked Read 9 Done Sent Done false
31122 8826 Acked Read 2 Done Sent Done false
31123 8827 Acked Write 4 Done Sent Done false
31124 8828 Acked Read 10 Done Sent Done false
31125 8829 Acked Read 10 Done Sent Done false
31126 8830 Acked Write 2 Done Sent Done false
31127 8831 Acked Write 3 Done Sent Done false
31128 8832 Acked Read 2 Done Sent Done false
31129 8833 Acked Read 7 Done Sent Done false
31130 8834 Acked Write 4 Done Sent Done false
31131 8835 Acked Read 4 Done Sent Done false
31132 8836 Acked Read 10 Done Sent Done false
31133 8837 Acked Read 2 Done Sent Done false
31134 8838 Acked Read 10 Done Sent Done false
31135 8839 Acked Read 2 Done Sent Done false
31136 8840 Acked Write 8 Done Sent Done false
31137 8841 Acked Read 3 Done Sent Done false
31138 8842 Acked Read 5 Done Sent Done false
31139 8843 Acked Read 10 Done Sent Done false
31140 8844 Acked Write 8 Done Sent Done false
31141 8845 Acked Write 8 Done Sent Done false
31142 8846 Acked Write 8 Done Sent Done false
31143 8847 Acked Read 9 Done Sent Done false
31144 STATES DS:0 DS:1 DS:2 TOTAL
31145 Sent 0 301 0 301
31146 Done 301 0 301 602
31147 Skipped 0 0 0 0
31148 Error 0 0 0 0
31149 Last Flush: Some(JobId(8806)) Some(JobId(8546)) Some(JobId(8806))
31150 Downstairs last five completed: 8546 8545 8544 8543 8542
31151 Upstairs last five completed: 8847 8846 8845 8844 8843
31152 CLIENT: Up:0 ds:301 act:2
311532025-02-28T06:07:55.035ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
311542025-02-28T06:07:56.153ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
311552025-02-28T06:07:57.037ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
31156 [8820] got start action
31157 [8820] I am going to restart
31158 Starting downstairs at port 8820
31159 Make output file at "/tmp/dsc/downstairs-8820.txt"
31160 [8820][1] reports Starting
31161 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1237)
31162 [8820][1] reports Running
311632025-02-28T06:07:58.400ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
31164 ----------------------------------------------------------------
31165 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31166 States: Active Offline Active
31167 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31168 8547 Acked Write 6 Done Sent Done false
31169 8548 Acked Write 3 Done Sent Done false
31170 8549 Acked Write 3 Done Sent Done false
31171 8550 Acked Read 8 Done Sent Done false
31172 8551 Acked Write 6 Done Sent Done false
31173 8552 Acked Flush 0 Done Sent Done false
31174 8553 Acked Read 3 Done Sent Done false
31175 8554 Acked Write 2 Done Sent Done false
31176 8555 Acked Write 2 Done Sent Done false
31177 8556 Acked Write 9 Done Sent Done false
31178 8557 Acked Write 5 Done Sent Done false
31179 8558 Acked Write 3 Done Sent Done false
31180 8559 Acked Flush 0 Done Sent Done false
31181 8560 Acked Write 4 Done Sent Done false
31182 8561 Acked Write 6 Done Sent Done false
31183 8562 Acked Read 7 Done Sent Done false
31184 8563 Acked Write 2 Done Sent Done false
31185 8564 Acked Flush 0 Done Sent Done false
31186 8565 Acked Read 5 Done Sent Done false
31187 8566 Acked Read 5 Done Sent Done false
31188 8567 Acked Write 2 Done Sent Done false
31189 8568 Acked Write 1 Done Sent Done false
31190 8569 Acked Write 5 Done Sent Done false
31191 8570 Acked Read 10 Done Sent Done false
31192 8571 Acked Flush 0 Done Sent Done false
31193 8572 Acked Write 5 Done Sent Done false
31194 8573 Acked Read 5 Done Sent Done false
31195 8574 Acked Read 10 Done Sent Done false
31196 8575 Acked Read 9 Done Sent Done false
31197 8576 Acked Write 9 Done Sent Done false
31198 8577 Acked Read 6 Done Sent Done false
31199 8578 Acked Read 3 Done Sent Done false
31200 8579 Acked Read 8 Done Sent Done false
31201 8580 Acked Read 8 Done Sent Done false
31202 8581 Acked Read 9 Done Sent Done false
31203 8582 Acked Read 3 Done Sent Done false
31204 8583 Acked Read 10 Done Sent Done false
31205 8584 Acked Write 1 Done Sent Done false
31206 8585 Acked Read 4 Done Sent Done false
31207 8586 Acked Read 4 Done Sent Done false
31208 8587 Acked Flush 0 Done Sent Done false
31209 8588 Acked Write 7 Done Sent Done false
31210 8589 Acked Read 8 Done Sent Done false
31211 8590 Acked Read 2 Done Sent Done false
31212 8591 Acked Flush 0 Done Sent Done false
31213 8592 Acked Read 5 Done Sent Done false
31214 8593 Acked Read 3 Done Sent Done false
31215 8594 Acked Read 6 Done Sent Done false
31216 8595 Acked Read 1 Done Sent Done false
31217 8596 Acked Read 1 Done Sent Done false
31218 8597 Acked Write 4 Done Sent Done false
31219 8598 Acked Flush 0 Done Sent Done false
31220 8599 Acked Read 9 Done Sent Done false
31221 8600 Acked Read 8 Done Sent Done false
31222 8601 Acked Read 2 Done Sent Done false
31223 8602 Acked Write 2 Done Sent Done false
31224 8603 Acked Write 4 Done Sent Done false
31225 8604 Acked Read 1 Done Sent Done false
31226 8605 Acked Write 5 Done Sent Done false
31227 8606 Acked Write 3 Done Sent Done false
31228 8607 Acked Write 2 Done Sent Done false
31229 8608 Acked Write 5 Done Sent Done false
31230 8609 Acked Flush 0 Done Sent Done false
31231 8610 Acked Read 9 Done Sent Done false
31232 8611 Acked Read 10 Done Sent Done false
31233 8612 Acked Read 1 Done Sent Done false
31234 8613 Acked Read 8 Done Sent Done false
31235 8614 Acked Read 10 Done Sent Done false
31236 8615 Acked Write 10 Done Sent Done false
31237 8616 Acked Read 5 Done Sent Done false
31238 8617 Acked Read 5 Done Sent Done false
31239 8618 Acked Read 3 Done Sent Done false
31240 8619 Acked Read 4 Done Sent Done false
31241 8620 Acked Read 8 Done Sent Done false
31242 8621 Acked Read 1 Done Sent Done false
31243 8622 Acked Read 8 Done Sent Done false
31244 8623 Acked Read 1 Done Sent Done false
31245 8624 Acked Flush 0 Done Sent Done false
31246 8625 Acked Write 3 Done Sent Done false
31247 8626 Acked Write 10 Done Sent Done false
31248 8627 Acked Write 1 Done Sent Done false
31249 8628 Acked Write 1 Done Sent Done false
31250 8629 Acked Read 5 Done Sent Done false
31251 8630 Acked Read 9 Done Sent Done false
31252 8631 Acked Read 2 Done Sent Done false
31253 8632 Acked Read 3 Done Sent Done false
31254 8633 Acked Write 9 Done Sent Done false
31255 8634 Acked Write 8 Done Sent Done false
31256 8635 Acked Read 8 Done Sent Done false
31257 8636 Acked Write 5 Done Sent Done false
31258 8637 Acked Write 8 Done Sent Done false
31259 8638 Acked Read 3 Done Sent Done false
31260 8639 Acked Write 7 Done Sent Done false
31261 8640 Acked Write 4 Done Sent Done false
31262 8641 Acked Write 1 Done Sent Done false
31263 8642 Acked Write 3 Done Sent Done false
31264 8643 Acked Write 5 Done Sent Done false
31265 8644 Acked Read 8 Done Sent Done false
31266 8645 Acked Read 7 Done Sent Done false
31267 8646 Acked Read 6 Done Sent Done false
31268 8647 Acked Write 1 Done Sent Done false
31269 8648 Acked Write 9 Done Sent Done false
31270 8649 Acked Write 4 Done Sent Done false
31271 8650 Acked Write 4 Done Sent Done false
31272 8651 Acked Read 2 Done Sent Done false
31273 8652 Acked Write 9 Done Sent Done false
31274 8653 Acked Read 6 Done Sent Done false
31275 8654 Acked Read 5 Done Sent Done false
31276 8655 Acked Write 4 Done Sent Done false
31277 8656 Acked Flush 0 Done Sent Done false
31278 8657 Acked Read 8 Done Sent Done false
31279 8658 Acked Write 1 Done Sent Done false
31280 8659 Acked Write 9 Done Sent Done false
31281 8660 Acked Write 9 Done Sent Done false
31282 8661 Acked Read 9 Done Sent Done false
31283 8662 Acked Read 6 Done Sent Done false
31284 8663 Acked Read 2 Done Sent Done false
31285 8664 Acked Read 8 Done Sent Done false
31286 8665 Acked Write 10 Done Sent Done false
31287 8666 Acked Read 6 Done Sent Done false
31288 8667 Acked Write 8 Done Sent Done false
31289 8668 Acked Write 10 Done Sent Done false
31290 8669 Acked Write 2 Done Sent Done false
31291 8670 Acked Write 2 Done Sent Done false
31292 8671 Acked Read 9 Done Sent Done false
31293 8672 Acked Write 5 Done Sent Done false
31294 8673 Acked Read 5 Done Sent Done false
31295 8674 Acked Flush 0 Done Sent Done false
31296 8675 Acked Flush 0 Done Sent Done false
31297 8676 Acked Read 4 Done Sent Done false
31298 8677 Acked Write 2 Done Sent Done false
31299 8678 Acked Read 10 Done Sent Done false
31300 8679 Acked Read 7 Done Sent Done false
31301 8680 Acked Write 1 Done Sent Done false
31302 8681 Acked Read 1 Done Sent Done false
31303 8682 Acked Flush 0 Done Sent Done false
31304 8683 Acked Read 5 Done Sent Done false
31305 8684 Acked Read 7 Done Sent Done false
31306 8685 Acked Write 10 Done Sent Done false
31307 8686 Acked Read 5 Done Sent Done false
31308 8687 Acked Read 10 Done Sent Done false
31309 8688 Acked Flush 0 Done Sent Done false
31310 8689 Acked Read 1 Done Sent Done false
31311 8690 Acked Write 2 Done Sent Done false
31312 8691 Acked Read 8 Done Sent Done false
31313 8692 Acked Read 1 Done Sent Done false
31314 8693 Acked Read 1 Done Sent Done false
31315 8694 Acked Read 5 Done Sent Done false
31316 8695 Acked Read 8 Done Sent Done false
31317 8696 Acked Read 3 Done Sent Done false
31318 8697 Acked Read 5 Done Sent Done false
31319 8698 Acked Write 1 Done Sent Done false
31320 8699 Acked Write 3 Done Sent Done false
31321 8700 Acked Write 4 Done Sent Done false
31322 8701 Acked Read 8 Done Sent Done false
31323 8702 Acked Write 4 Done Sent Done false
31324 8703 Acked Read 2 Done Sent Done false
31325 8704 Acked Read 5 Done Sent Done false
31326 8705 Acked Write 8 Done Sent Done false
31327 8706 Acked Read 4 Done Sent Done false
31328 8707 Acked Read 4 Done Sent Done false
31329 8708 Acked Read 5 Done Sent Done false
31330 8709 Acked Read 1 Done Sent Done false
31331 8710 Acked Write 1 Done Sent Done false
31332 8711 Acked Flush 0 Done Sent Done false
31333 8712 Acked Read 2 Done Sent Done false
31334 8713 Acked Read 4 Done Sent Done false
31335 8714 Acked Write 5 Done Sent Done false
31336 8715 Acked Read 1 Done Sent Done false
31337 8716 Acked Read 3 Done Sent Done false
31338 8717 Acked Write 6 Done Sent Done false
31339 8718 Acked Read 7 Done Sent Done false
31340 8719 Acked Read 9 Done Sent Done false
31341 8720 Acked Read 1 Done Sent Done false
31342 8721 Acked Read 7 Done Sent Done false
31343 8722 Acked Read 5 Done Sent Done false
31344 8723 Acked Read 5 Done Sent Done false
31345 8724 Acked Read 9 Done Sent Done false
31346 8725 Acked Write 7 Done Sent Done false
31347 8726 Acked Flush 0 Done Sent Done false
31348 8727 Acked Read 6 Done Sent Done false
31349 8728 Acked Read 3 Done Sent Done false
31350 8729 Acked Write 7 Done Sent Done false
31351 8730 Acked Read 6 Done Sent Done false
31352 8731 Acked Read 4 Done Sent Done false
31353 8732 Acked Read 5 Done Sent Done false
31354 8733 Acked Write 8 Done Sent Done false
31355 8734 Acked Read 10 Done Sent Done false
31356 8735 Acked Read 2 Done Sent Done false
31357 8736 Acked Write 9 Done Sent Done false
31358 8737 Acked Write 2 Done Sent Done false
31359 8738 Acked Write 3 Done Sent Done false
31360 8739 Acked Read 5 Done Sent Done false
31361 8740 Acked Read 5 Done Sent Done false
31362 8741 Acked Write 3 Done Sent Done false
31363 8742 Acked Flush 0 Done Sent Done false
31364 8743 Acked Read 2 Done Sent Done false
31365 8744 Acked Write 4 Done Sent Done false
31366 8745 Acked Flush 0 Done Sent Done false
31367 8746 Acked Write 2 Done Sent Done false
31368 8747 Acked Write 1 Done Sent Done false
31369 8748 Acked Read 6 Done Sent Done false
31370 8749 Acked Read 1 Done Sent Done false
31371 8750 Acked Read 2 Done Sent Done false
31372 8751 Acked Read 5 Done Sent Done false
31373 8752 Acked Write 7 Done Sent Done false
31374 8753 Acked Write 3 Done Sent Done false
31375 8754 Acked Read 1 Done Sent Done false
31376 8755 Acked Read 9 Done Sent Done false
31377 8756 Acked Write 3 Done Sent Done false
31378 8757 Acked Read 4 Done Sent Done false
31379 8758 Acked Write 4 Done Sent Done false
31380 8759 Acked Read 10 Done Sent Done false
31381 8760 Acked Write 8 Done Sent Done false
31382 8761 Acked Write 3 Done Sent Done false
31383 8762 Acked Flush 0 Done Sent Done false
31384 8763 Acked Read 2 Done Sent Done false
31385 8764 Acked Read 7 Done Sent Done false
31386 8765 Acked Write 3 Done Sent Done false
31387 8766 Acked Write 2 Done Sent Done false
31388 8767 Acked Write 8 Done Sent Done false
31389 8768 Acked Read 2 Done Sent Done false
31390 8769 Acked Read 6 Done Sent Done false
31391 8770 Acked Read 9 Done Sent Done false
31392 8771 Acked Flush 0 Done Sent Done false
31393 8772 Acked Read 1 Done Sent Done false
31394 8773 Acked Read 7 Done Sent Done false
31395 8774 Acked Read 4 Done Sent Done false
31396 8775 Acked Read 7 Done Sent Done false
31397 8776 Acked Write 8 Done Sent Done false
31398 8777 Acked Write 1 Done Sent Done false
31399 8778 Acked Write 4 Done Sent Done false
31400 8779 Acked Write 9 Done Sent Done false
31401 8780 Acked Write 5 Done Sent Done false
31402 8781 Acked Read 3 Done Sent Done false
31403 8782 Acked Write 9 Done Sent Done false
31404 8783 Acked Read 6 Done Sent Done false
31405 8784 Acked Read 10 Done Sent Done false
31406 8785 Acked Write 1 Done Sent Done false
31407 8786 Acked Read 9 Done Sent Done false
31408 8787 Acked Write 2 Done Sent Done false
31409 8788 Acked Read 1 Done Sent Done false
31410 8789 Acked Read 7 Done Sent Done false
31411 8790 Acked Read 8 Done Sent Done false
31412 8791 Acked Read 2 Done Sent Done false
31413 8792 Acked Write 4 Done Sent Done false
31414 8793 Acked Flush 0 Done Sent Done false
31415 8794 Acked Read 7 Done Sent Done false
31416 8795 Acked Write 7 Done Sent Done false
31417 8796 Acked Write 9 Done Sent Done false
31418 8797 Acked Read 10 Done Sent Done false
31419 8798 Acked Read 3 Done Sent Done false
31420 8799 Acked Write 2 Done Sent Done false
31421 8800 Acked Write 5 Done Sent Done false
31422 8801 Acked Flush 0 Done Sent Done false
31423 8802 Acked Flush 0 Done Sent Done false
31424 8803 Acked Write 4 Done Sent Done false
31425 8804 Acked Read 10 Done Sent Done false
31426 8805 Acked Read 9 Done Sent Done false
31427 8806 Acked Flush 0 Done Sent Done false
31428 8807 Acked Write 10 Done Sent Done false
31429 8808 Acked Write 9 Done Sent Done false
31430 8809 Acked Read 6 Done Sent Done false
31431 8810 Acked Read 8 Done Sent Done false
31432 8811 Acked Read 4 Done Sent Done false
31433 8812 Acked Read 5 Done Sent Done false
31434 8813 Acked Read 7 Done Sent Done false
31435 8814 Acked Read 8 Done Sent Done false
31436 8815 Acked Read 7 Done Sent Done false
31437 8816 Acked Write 9 Done Sent Done false
31438 8817 Acked Read 8 Done Sent Done false
31439 8818 Acked Read 1 Done Sent Done false
31440 8819 Acked Write 6 Done Sent Done false
31441 8820 Acked Write 1 Done Sent Done false
31442 8821 Acked Read 4 Done Sent Done false
31443 8822 Acked Write 2 Done Sent Done false
31444 8823 Acked Read 9 Done Sent Done false
31445 8824 Acked Read 8 Done Sent Done false
31446 8825 Acked Read 9 Done Sent Done false
31447 8826 Acked Read 2 Done Sent Done false
31448 8827 Acked Write 4 Done Sent Done false
31449 8828 Acked Read 10 Done Sent Done false
31450 8829 Acked Read 10 Done Sent Done false
31451 8830 Acked Write 2 Done Sent Done false
31452 8831 Acked Write 3 Done Sent Done false
31453 8832 Acked Read 2 Done Sent Done false
31454 8833 Acked Read 7 Done Sent Done false
31455 8834 Acked Write 4 Done Sent Done false
31456 8835 Acked Read 4 Done Sent Done false
31457 8836 Acked Read 10 Done Sent Done false
31458 8837 Acked Read 2 Done Sent Done false
31459 8838 Acked Read 10 Done Sent Done false
31460 8839 Acked Read 2 Done Sent Done false
31461 8840 Acked Write 8 Done Sent Done false
31462 8841 Acked Read 3 Done Sent Done false
31463 8842 Acked Read 5 Done Sent Done false
31464 8843 Acked Read 10 Done Sent Done false
31465 8844 Acked Write 8 Done Sent Done false
31466 8845 Acked Write 8 Done Sent Done false
31467 8846 Acked Write 8 Done Sent Done false
31468 8847 Acked Read 9 Done Sent Done false
31469 STATES DS:0 DS:1 DS:2 TOTAL
31470 Sent 0 301 0 301
31471 Done 301 0 301 602
31472 Skipped 0 0 0 0
31473 Error 0 0 0 0
31474 Last Flush: Some(JobId(8806)) Some(JobId(8546)) Some(JobId(8806))
31475 Downstairs last five completed: 8546 8545 8544 8543 8542
31476 Upstairs last five completed: 8847 8846 8845 8844 8843
31477 CLIENT: Up:0 ds:301 act:2
314782025-02-28T06:07:59.042ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
314792025-02-28T06:08:00.044ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
314802025-02-28T06:08:00.159ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
314812025-02-28T06:08:00.159ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61551) = io task client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
314822025-02-28T06:08:00.161ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
314832025-02-28T06:08:00.161ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
314842025-02-28T06:08:00.161ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
314852025-02-28T06:08:00.161ZINFOcrucible: send last flush ID to this DS: Some(JobId(8546)) = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
314862025-02-28T06:08:00.161ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
314872025-02-28T06:08:00.161ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(8546)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
314882025-02-28T06:08:00.161ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(8546)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
31489 ----------------------------------------------------------------
31490 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31491 Downstairs last five completed: 8848 8847 8846 8845 8844
31492 Upstairs last five completed: 8848 8847 8846 8845 8844
31493 CLIENT: Up:0 ds:0 act:3
31494 Replay: All jobs finished
31495 Main task has work to do, go find it
31496 got dsc Stop(0)
31497 stop 0
31498 [8810] Got stop action so:false kr:false
31499 Feb 28 06:08:02.410 INFO request completed, latency_us: 318, response_code: 204, uri: /stop/cid/0, method: POST, req_id: b9a61855-e02a-4b3b-af1e-45281f08aba9, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
315002025-02-28T06:08:02.411ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
315012025-02-28T06:08:02.411ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
315022025-02-28T06:08:02.411ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
315032025-02-28T06:08:02.411ZINFOcrucible: client task is exiting = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
31504 Feb 28 06:08:02.411 INFO request completed, latency_us: 193, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9fa696d1-e837-4492-9007-2e9a1220e172, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
31505 [8810] Exited with: ExitStatus(unix_wait_status(9))
31506 [8810][0] reports Exit
31507 Feb 28 06:08:06.414 INFO request completed, latency_us: 299, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6d733b56-a0c2-4fe5-8831-9063cc81ef93, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
31508 001/300 Read block 2068 len 24576
31509 002/300 Write block 203 len 32768 data: 6 6 5 5 5 8 8 8
31510 003/300 Read block 3436 len 16384
31511 004/300 Write block 933 len 40960 data: 6 5 5 5 6 7 6 6 8 9
31512 005/300 Read block 4939 len 28672
31513 006/300 Write block 196 len 32768 data: 8 6 5 5 5 4 4 7
31514 007/300 Read block 44 len 40960
31515 008/300 Read block 3489 len 36864
31516 009/300 Flush
31517 010/300 Read block 4828 len 8192
31518 011/300 Read block 972 len 20480
31519 012/300 Read block 2868 len 16384
31520 013/300 Write block 2591 len 20480 data: 4 3 3 4 4
31521 014/300 Write block 2104 len 40960 data: 3 3 2 4 5 4 6 6 5 5
31522 015/300 Write block 2742 len 28672 data: 5 5 4 4 4 4 6
31523 016/300 Read block 4898 len 28672
31524 017/300 Flush
31525 018/300 Read block 1976 len 40960
31526 019/300 Write block 881 len 24576 data: 2 2 2 2 3 3
31527 020/300 Write block 1793 len 24576 data: 5 5 6 5 5 5
31528 021/300 Read block 2158 len 12288
31529 022/300 Read block 4011 len 24576
31530 023/300 Write block 3622 len 32768 data: 5 6 6 6 5 5 5 5
31531 024/300 Write block 1222 len 32768 data: 7 7 7 6 7 8 7 6
31532 025/300 Read block 2283 len 36864
31533 026/300 Flush
31534 027/300 Write block 2635 len 4096 data: 4
31535 028/300 Read block 4273 len 36864
31536 029/300 Read block 4813 len 16384
31537 030/300 Read block 4403 len 28672
31538 031/300 Read block 4955 len 24576
31539 032/300 Read block 3258 len 12288
31540 033/300 Flush
31541 034/300 Flush
31542 035/300 Write block 4773 len 12288 data: 7 6 7
31543 036/300 Read block 4485 len 24576
31544 037/300 Read block 847 len 4096
31545 038/300 Flush
31546 039/300 Read block 936 len 4096
31547 040/300 Read block 1276 len 24576
31548 041/300 Read block 1014 len 12288
31549 042/300 Write block 2780 len 32768 data: 5 5 4 7 6 5 6 6
31550 043/300 Write block 4059 len 16384 data: 5 5 3 5
31551 044/300 Read block 1413 len 8192
31552 045/300 Read block 1513 len 8192
31553 046/300 Read block 1041 len 12288
31554 047/300 Read block 1062 len 4096
31555 048/300 Read block 4104 len 20480
31556 049/300 Write block 3935 len 40960 data: 7 8 7 9 10 10 7 6 6 7
31557 050/300 Read block 57 len 4096
31558 051/300 Flush
31559 052/300 Read block 3412 len 12288
31560 053/300 Write block 2321 len 36864 data: 5 5 6 5 5 6 5 3 3
31561 054/300 Write block 4889 len 16384 data: 15 14 13 11
31562 055/300 Read block 2667 len 36864
31563 056/300 Write block 1319 len 32768 data: 7 7 7 8 8 6 6 7
31564 057/300 Flush
31565 058/300 Read block 1903 len 32768
31566 059/300 Read block 2052 len 32768
31567 060/300 Write block 141 len 16384 data: 5 4 3 6
31568 061/300 Write block 81 len 28672 data: 9 7 7 5 4 4 3
31569 062/300 Flush
31570 063/300 Write block 1755 len 12288 data: 6 5 5
31571 064/300 Write block 3969 len 8192 data: 3 4
31572 065/300 Write block 4333 len 12288 data: 5 4 4
31573 066/300 Read block 3534 len 4096
31574 067/300 Flush
31575 068/300 Read block 4461 len 28672
31576 069/300 Read block 1389 len 36864
31577 070/300 Flush
31578 071/300 Read block 2112 len 20480
31579 072/300 Read block 2939 len 12288
31580 073/300 Flush
31581 074/300 Read block 2871 len 16384
31582 075/300 Read block 2773 len 24576
31583 076/300 Write block 2927 len 20480 data: 5 3 4 4 7
31584 077/300 Read block 622 len 20480
31585 078/300 Read block 4734 len 40960
31586 079/300 Read block 4053 len 12288
31587 080/300 Write block 3539 len 36864 data: 5 5 5 5 2 2 5 6 5
31588 081/300 Write block 1793 len 12288 data: 6 6 7
31589 082/300 Write block 4280 len 20480 data: 6 5 4 3 3
31590 083/300 Write block 2994 len 20480 data: 6 5 7 7 6
31591 084/300 Read block 4924 len 36864
31592 085/300 Write block 3316 len 36864 data: 4 3 2 3 5 6 7 7 7
31593 086/300 Write block 4190 len 32768 data: 11 11 9 8 6 5 5 6
31594 087/300 Write block 4513 len 24576 data: 10 11 8 4 5 6
31595 088/300 Write block 4630 len 16384 data: 9 10 10 10
31596 089/300 Flush
31597 090/300 Write block 3115 len 8192 data: 7 7
31598 091/300 Read block 4669 len 8192
31599 092/300 Read block 4082 len 32768
31600 093/300 Read block 4431 len 8192
31601 094/300 Read block 1139 len 24576
31602 095/300 Write block 2849 len 32768 data: 5 5 5 6 5 6 6 6
31603 096/300 Write block 1286 len 28672 data: 5 6 6 7 7 8 7
31604 097/300 Write block 966 len 4096 data: 4
31605 098/300 Read block 4867 len 24576
31606 099/300 Flush
31607 100/300 Read block 4107 len 32768
31608 101/300 Write block 3817 len 32768 data: 4 4 4 3 3 4 5 4
31609 102/300 Write block 4372 len 24576 data: 3 3 3 7 7 8
31610 103/300 Write block 4435 len 12288 data: 4 5 4
31611 104/300 Read block 3930 len 12288
31612 105/300 Write block 2529 len 28672 data: 8 6 4 4 4 4 3
31613 106/300 Flush
31614 107/300 Read block 2201 len 12288
31615 108/300 Write block 1283 len 8192 data: 6 4
31616 109/300 Write block 2872 len 12288 data: 7 6 6
31617 110/300 Write block 15 len 20480 data: 3 3 4 6 6
31618 111/300 Flush
31619 112/300 Write block 4388 len 32768 data: 8 7 9 8 6 5 6 6
31620 113/300 Write block 1497 len 32768 data: 3 3 4 4 5 4 5 4
31621 114/300 Read block 1828 len 12288
31622 115/300 Read block 3928 len 24576
31623 116/300 Read block 1901 len 12288
31624 117/300 Write block 3538 len 12288 data: 7 6 6
31625 118/300 Read block 4215 len 12288
31626 119/300 Read block 119 len 36864
31627 120/300 Write block 1284 len 36864 data: 5 3 6 7 7 8 8 9 8
31628 121/300 Write block 4010 len 40960 data: 6 6 5 4 3 4 3 3 5 5
31629 122/300 Write block 1910 len 36864 data: 2 2 3 5 5 5 5 5 5
31630 123/300 Read block 1751 len 36864
31631 124/300 Read block 1889 len 28672
31632 125/300 Read block 4537 len 28672
31633 126/300 Read block 1562 len 32768
31634 127/300 Write block 4373 len 12288 data: 4 4 8
31635 128/300 Read block 4345 len 12288
31636 129/300 Flush
31637 130/300 Write block 2133 len 28672 data: 7 7 6 5 7 5 6
31638 131/300 Read block 4477 len 28672
31639 132/300 Read block 2485 len 32768
31640 133/300 Write block 155 len 16384 data: 6 6 4 3
31641 134/300 Read block 2628 len 20480
31642 135/300 Read block 2737 len 32768
31643 136/300 Write block 548 len 8192 data: 6 7
31644 137/300 Write block 689 len 36864 data: 3 4 4 5 4 4 4 3 5
31645 138/300 Write block 4267 len 12288 data: 8 8 7
31646 139/300 Read block 1895 len 32768
31647 140/300 Read block 4633 len 12288
31648 141/300 Write block 662 len 40960 data: 9 8 8 8 5 7 8 7 8 8
31649 142/300 Read block 2671 len 8192
31650 143/300 Write block 3789 len 32768 data: 2 2 3 4 3 5 5 5
31651 144/300 Read block 2162 len 32768
31652 145/300 Read block 362 len 4096
31653 146/300 Read block 2931 len 8192
31654 147/300 Write block 4116 len 8192 data: 7 7
31655 148/300 Write block 2139 len 16384 data: 7 7 6 6
31656 149/300 Flush
31657 150/300 Read block 1491 len 12288
31658 151/300 Write block 4443 len 24576 data: 6 6 5 3 4 3
31659 152/300 Write block 188 len 20480 data: 6 6 7 8 7
31660 153/300 Read block 171 len 4096
31661 154/300 Read block 1009 len 32768
31662 155/300 Read block 971 len 36864
31663 156/300 Write block 2264 len 12288 data: 8 6 6
31664 157/300 Flush
31665 158/300 Write block 2704 len 24576 data: 6 7 7 6 6 6
31666 159/300 Read block 3564 len 16384
31667 160/300 Read block 3241 len 40960
31668 161/300 Read block 725 len 12288
31669 162/300 Write block 2244 len 16384 data: 2 4 4 4
31670 163/300 Write block 3007 len 32768 data: 4 4 5 5 5 4 4 4
31671 164/300 Write block 1599 len 20480 data: 2 3 3 3 3
31672 165/300 Read block 2650 len 40960
31673 166/300 Write block 3897 len 24576 data: 3 4 4 4 4 4
31674 167/300 Write block 2220 len 4096 data: 6
31675 168/300 Read block 3825 len 8192
31676 169/300 Read block 4055 len 40960
31677 170/300 Write block 1485 len 24576 data: 8 8 7 8 9 10
31678 171/300 Write block 2107 len 28672 data: 5 6 5 7 7 6 6
31679 172/300 Read block 3191 len 32768
31680 173/300 Read block 3188 len 20480
31681 174/300 Read block 3643 len 8192
31682 175/300 Flush
31683 176/300 Write block 3401 len 12288 data: 4 4 4
31684 177/300 Read block 3428 len 36864
31685 178/300 Read block 1315 len 40960
31686 179/300 Read block 4608 len 36864
31687 180/300 Write block 905 len 4096 data: 6
31688 181/300 Read block 1426 len 32768
31689 182/300 Write block 1952 len 28672 data: 3 3 5 6 5 5 5
31690 183/300 Read block 3776 len 16384
31691 184/300 Read block 4249 len 20480
31692 185/300 Write block 2080 len 16384 data: 3 4 4 6
31693 186/300 Read block 4068 len 24576
31694 187/300 Flush
31695 188/300 Read block 1527 len 28672
31696 189/300 Write block 131 len 36864 data: 5 4 6 5 6 6 6 5 5
31697 190/300 Read block 3313 len 20480
31698 191/300 Read block 4244 len 8192
31699 192/300 Write block 1881 len 12288 data: 6 5 2
31700 193/300 Read block 4989 len 36864
31701 194/300 Read block 2605 len 16384
31702 195/300 Read block 3379 len 36864
31703 196/300 Write block 1801 len 8192 data: 5 7
31704 197/300 Read block 1350 len 8192
31705 198/300 Write block 895 len 40960 data: 4 4 5 5 5 5 7 8 6 5
31706 199/300 Flush
31707 200/300 Write block 3749 len 36864 data: 7 5 6 6 5 4 4 4 4
31708 201/300 Write block 1764 len 24576 data: 5 5 5 4 2 3
31709 202/300 Write block 4761 len 24576 data: 7 7 6 5 4 4
31710 203/300 Read block 1031 len 28672
31711 204/300 Read block 4420 len 32768
31712 205/300 Read block 3112 len 24576
31713 206/300 Read block 4042 len 12288
31714 207/300 Read block 4156 len 40960
31715 208/300 Write block 1982 len 28672 data: 7 9 9 7 7 6 7
31716 209/300 Read block 1977 len 12288
31717 210/300 Read block 856 len 4096
31718 211/300 Read block 2597 len 36864
31719 212/300 Flush
31720 213/300 Read block 1515 len 4096
31721 214/300 Write block 1144 len 12288 data: 6 8 8
31722 215/300 Read block 778 len 12288
31723 216/300 Flush
31724 217/300 Read block 4123 len 16384
31725 218/300 Read block 2261 len 4096
31726 219/300 Write block 2765 len 4096 data: 4
31727 220/300 Read block 820 len 40960
31728 221/300 Read block 2449 len 28672
31729 222/300 Write block 3651 len 28672 data: 4 4 4 3 3 3 4
31730 223/300 Flush
31731 224/300 Write block 2538 len 16384 data: 6 5 4 4
31732 225/300 Write block 4373 len 40960 data: 5 5 9 8 9 8 9 9 8 5
31733 226/300 Read block 3294 len 40960
31734 227/300 Read block 4740 len 36864
31735 228/300 Read block 15 len 12288
31736 229/300 Write block 748 len 20480 data: 10 10 10 6 4
31737 230/300 Write block 2717 len 28672 data: 10 8 6 5 5 5 4
31738 231/300 Write block 4058 len 24576 data: 5 6 6 4 6 5
31739 232/300 Write block 1733 len 16384 data: 5 4 4 4
31740 233/300 Read block 407 len 24576
31741 234/300 Write block 4360 len 40960 data: 5 3 3 4 5 4 3 3 3 4
31742 235/300 Read block 1281 len 24576
31743 236/300 Write block 4728 len 20480 data: 6 8 9 8 8
31744 237/300 Read block 4190 len 8192
31745 238/300 Write block 4320 len 40960 data: 8 9 8 9 8 7 6 7 7 5
31746 239/300 Write block 3208 len 16384 data: 9 8 6 6
31747 240/300 Write block 3280 len 24576 data: 4 3 4 3 3 3
31748 241/300 Read block 423 len 20480
31749 242/300 Write block 4595 len 24576 data: 10 12 11 11 8 8
31750 243/300 Read block 1348 len 28672
31751 244/300 Write block 66 len 32768 data: 4 3 3 3 3 4 4 6
31752 245/300 Write block 3335 len 28672 data: 6 6 5 6 6 7 8
31753 246/300 Write block 4140 len 8192 data: 5 6
31754 247/300 Read block 3199 len 24576
31755 248/300 Read block 3535 len 28672
31756 249/300 Read block 1393 len 4096
31757 250/300 Read block 1732 len 28672
31758 251/300 Write block 978 len 24576 data: 6 5 6 5 5 6
31759 252/300 Flush
31760 253/300 Flush
31761 254/300 Read block 2855 len 28672
31762 255/300 Write block 1606 len 4096 data: 4
31763 256/300 Write block 59 len 12288 data: 5 5 5
31764 257/300 Write block 887 len 36864 data: 4 4 3 4 4 4 4 5 5
31765 258/300 Read block 1448 len 8192
31766 259/300 Write block 3701 len 8192 data: 4 5
31767 260/300 Read block 536 len 40960
31768 261/300 Read block 2050 len 12288
31769 262/300 Read block 2334 len 36864
31770 263/300 Write block 4000 len 8192 data: 5 4
31771 264/300 Read block 1676 len 40960
31772 265/300 Read block 256 len 16384
31773 266/300 Write block 308 len 28672 data: 3 2 3 4 3 4 4
31774 267/300 Write block 4491 len 32768 data: 6 5 5 6 7 8 6 6
31775 268/300 Read block 877 len 12288
31776 269/300 Flush
31777 270/300 Write block 3837 len 40960 data: 3 5 5 6 5 5 6 6 7 6
31778 271/300 Read block 4315 len 8192
31779 272/300 Read block 4654 len 36864
31780 273/300 Read block 1272 len 40960
31781 274/300 Read block 3406 len 12288
31782 275/300 Write block 231 len 16384 data: 6 4 4 6
31783 276/300 Flush
31784 277/300 Read block 3877 len 36864
31785 278/300 Read block 2219 len 20480
31786 279/300 Read block 3392 len 4096
31787 280/300 Read block 2645 len 40960
31788 281/300 Write block 2326 len 36864 data: 7 6 4 4 3 3 3 3 3
31789 282/300 Write block 1926 len 4096 data: 4
31790 283/300 Flush
31791 284/300 Write block 1920 len 24576 data: 4 4 2 2 2 3
31792 285/300 Read block 1748 len 40960
31793 286/300 Read block 3423 len 20480
31794 287/300 Read block 2621 len 8192
31795 288/300 Read block 3289 len 16384
31796 289/300 Write block 2206 len 20480 data: 4 3 4 6 5
31797 290/300 Write block 741 len 36864 data: 5 7 8 10 9 9 8 11 11
31798 291/300 Read block 3406 len 4096
31799 292/300 Read block 242 len 28672
31800 293/300 Read block 3727 len 40960
31801 294/300 Write block 3826 len 32768 data: 4 4 4 5 3 4 4 4
31802 295/300 Read block 602 len 16384
31803 296/300 Write block 4241 len 24576 data: 3 3 3 2 2 3
31804 297/300 Read block 1667 len 20480
31805 298/300 Read block 3876 len 8192
31806 299/300 Read block 1884 len 32768
31807 300/300 Write block 3814 len 16384 data: 3 3 4 5
31808 301/300 Flush
31809 Main task has work to do, go find it
31810 got dsc Start(0)
31811 start 0
31812 Feb 28 06:08:06.658 INFO request completed, latency_us: 180, response_code: 204, uri: /start/cid/0, method: POST, req_id: 1974e88b-c5a3-472e-ba2b-cf14c5563a82, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
31813 Replay: started 0, returned:Ok(())
31814 ----------------------------------------------------------------
31815 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31816 States: Offline Active Active
31817 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31818 8849 Acked Read 6 Sent Done Done false
31819 8850 Acked Write 8 Sent Done Done false
31820 8851 Acked Read 4 Sent Done Done false
31821 8852 Acked Write 10 Sent Done Done false
31822 8853 Acked Read 7 Sent Done Done false
31823 8854 Acked Write 8 Sent Done Done false
31824 8855 Acked Read 10 Sent Done Done false
31825 8856 Acked Read 9 Sent Done Done false
31826 8857 Acked Flush 0 Sent Done Done false
31827 8858 Acked Read 2 Sent Done Done false
31828 8859 Acked Read 5 Sent Done Done false
31829 8860 Acked Read 4 Sent Done Done false
31830 8861 Acked Write 5 Sent Done Done false
31831 8862 Acked Write 10 Sent Done Done false
31832 8863 Acked Write 7 Sent Done Done false
31833 8864 Acked Read 7 Sent Done Done false
31834 8865 Acked Flush 0 Sent Done Done false
31835 8866 Acked Read 10 Sent Done Done false
31836 8867 Acked Write 6 Sent Done Done false
31837 8868 Acked Write 6 Sent Done Done false
31838 8869 Acked Read 3 Sent Done Done false
31839 8870 Acked Read 6 Sent Done Done false
31840 8871 Acked Write 8 Sent Done Done false
31841 8872 Acked Write 8 Sent Done Done false
31842 8873 Acked Read 9 Sent Done Done false
31843 8874 Acked Flush 0 Sent Done Done false
31844 8875 Acked Write 1 Sent Done Done false
31845 8876 Acked Read 9 Sent Done Done false
31846 8877 Acked Read 4 Sent Done Done false
31847 8878 Acked Read 7 Sent Done Done false
31848 8879 Acked Read 6 Sent Done Done false
31849 8880 Acked Read 3 Sent Done Done false
31850 8881 Acked Flush 0 Sent Done Done false
31851 8882 Acked Flush 0 Sent Done Done false
31852 8883 Acked Write 3 Sent Done Done false
31853 8884 Acked Read 6 Sent Done Done false
31854 8885 Acked Read 1 Sent Done Done false
31855 8886 Acked Flush 0 Sent Done Done false
31856 8887 Acked Read 1 Sent Done Done false
31857 8888 Acked Read 6 Sent Done Done false
31858 8889 Acked Read 3 Sent Done Done false
31859 8890 Acked Write 8 Sent Done Done false
31860 8891 Acked Write 4 Sent Done Done false
31861 8892 Acked Read 2 Sent Done Done false
31862 8893 Acked Read 2 Sent Done Done false
31863 8894 Acked Read 3 Sent Done Done false
31864 8895 Acked Read 1 Sent Done Done false
31865 8896 Acked Read 5 Sent Done Done false
31866 8897 Acked Write 10 Sent Done Done false
31867 8898 Acked Read 1 Sent Done Done false
31868 8899 Acked Flush 0 Sent Done Done false
31869 8900 Acked Read 3 Sent Done Done false
31870 8901 Acked Write 9 Sent Done Done false
31871 8902 Acked Write 4 Sent Done Done false
31872 8903 Acked Read 9 Sent Done Done false
31873 8904 Acked Write 8 Sent Done Done false
31874 8905 Acked Flush 0 Sent Done Done false
31875 8906 Acked Read 8 Sent Done Done false
31876 8907 Acked Read 8 Sent Done Done false
31877 8908 Acked Write 4 Sent Done Done false
31878 8909 Acked Write 7 Sent Done Done false
31879 8910 Acked Flush 0 Sent Done Done false
31880 8911 Acked Write 3 Sent Done Done false
31881 8912 Acked Write 2 Sent Done Done false
31882 8913 Acked Write 3 Sent Done Done false
31883 8914 Acked Read 1 Sent Done Done false
31884 8915 Acked Flush 0 Sent Done Done false
31885 8916 Acked Read 7 Sent Done Done false
31886 8917 Acked Read 9 Sent Done Done false
31887 8918 Acked Flush 0 Sent Done Done false
31888 8919 Acked Read 5 Sent Done Done false
31889 8920 Acked Read 3 Sent Done Done false
31890 8921 Acked Flush 0 Sent Done Done false
31891 8922 Acked Read 4 Sent Done Done false
31892 8923 Acked Read 6 Sent Done Done false
31893 8924 Acked Write 5 Sent Done Done false
31894 8925 Acked Read 5 Sent Done Done false
31895 8926 Acked Read 10 Sent Done Done false
31896 8927 Acked Read 3 Sent Done Done false
31897 8928 Acked Write 9 Sent Done Done false
31898 8929 Acked Write 3 Sent Done Done false
31899 8930 Acked Write 5 Sent Done Done false
31900 8931 Acked Write 5 Sent Done Done false
31901 8932 Acked Read 9 Sent Done Done false
31902 8933 Acked Write 9 Sent Done Done false
31903 8934 Acked Write 8 Sent Done Done false
31904 8935 Acked Write 6 Sent Done Done false
31905 8936 Acked Write 4 Sent Done Done false
31906 8937 Acked Flush 0 Sent Done Done false
31907 8938 Acked Write 2 Sent Done Done false
31908 8939 Acked Read 2 Sent Done Done false
31909 8940 Acked Read 8 Sent Done Done false
31910 8941 Acked Read 2 Sent Done Done false
31911 8942 Acked Read 6 Sent Done Done false
31912 8943 Acked Write 8 Sent Done Done false
31913 8944 Acked Write 7 Sent Done Done false
31914 8945 Acked Write 1 Sent Done Done false
31915 8946 Acked Read 6 Sent Done Done false
31916 8947 Acked Flush 0 Sent Done Done false
31917 8948 Acked Read 8 Sent Done Done false
31918 8949 Acked Write 8 Sent Done Done false
31919 8950 Acked Write 6 Sent Done Done false
31920 8951 Acked Write 3 Sent Done Done false
31921 8952 Acked Read 3 Sent Done Done false
31922 8953 Acked Write 7 Sent Done Done false
31923 8954 Acked Flush 0 Sent Done Done false
31924 8955 Acked Read 3 Sent Done Done false
31925 8956 Acked Write 2 Sent Done Done false
31926 8957 Acked Write 3 Sent Done Done false
31927 8958 Acked Write 5 Sent Done Done false
31928 8959 Acked Flush 0 Sent Done Done false
31929 8960 Acked Write 8 Sent Done Done false
31930 8961 Acked Write 8 Sent Done Done false
31931 8962 Acked Read 3 Sent Done Done false
31932 8963 Acked Read 6 Sent Done Done false
31933 8964 Acked Read 3 Sent Done Done false
31934 8965 Acked Write 3 Sent Done Done false
31935 8966 Acked Read 3 Sent Done Done false
31936 8967 Acked Read 9 Sent Done Done false
31937 8968 Acked Write 9 Sent Done Done false
31938 8969 Acked Write 10 Sent Done Done false
31939 8970 Acked Write 9 Sent Done Done false
31940 8971 Acked Read 9 Sent Done Done false
31941 8972 Acked Read 7 Sent Done Done false
31942 8973 Acked Read 7 Sent Done Done false
31943 8974 Acked Read 8 Sent Done Done false
31944 8975 Acked Write 3 Sent Done Done false
31945 8976 Acked Read 3 Sent Done Done false
31946 8977 Acked Flush 0 Sent Done Done false
31947 8978 Acked Write 7 Sent Done Done false
31948 8979 Acked Read 7 Sent Done Done false
31949 8980 Acked Read 8 Sent Done Done false
31950 8981 Acked Write 4 Sent Done Done false
31951 8982 Acked Read 5 Sent Done Done false
31952 8983 Acked Read 8 Sent Done Done false
31953 8984 Acked Write 2 Sent Done Done false
31954 8985 Acked Write 9 Sent Done Done false
31955 8986 Acked Write 3 Sent Done Done false
31956 8987 Acked Read 8 Sent Done Done false
31957 8988 Acked Read 3 Sent Done Done false
31958 8989 Acked Write 10 Sent Done Done false
31959 8990 Acked Read 2 Sent Done Done false
31960 8991 Acked Write 8 Sent Done Done false
31961 8992 Acked Read 8 Sent Done Done false
31962 8993 Acked Read 1 Sent Done Done false
31963 8994 Acked Read 2 Sent Done Done false
31964 8995 Acked Write 2 Sent Done Done false
31965 8996 Acked Write 4 Sent Done Done false
31966 8997 Acked Flush 0 Sent Done Done false
31967 8998 Acked Read 3 Sent Done Done false
31968 8999 Acked Write 6 Sent Done Done false
31969 9000 Acked Write 5 Sent Done Done false
31970 9001 Acked Read 1 Sent Done Done false
31971 9002 Acked Read 8 Sent Done Done false
31972 9003 Acked Read 9 Sent Done Done false
31973 9004 Acked Write 3 Sent Done Done false
31974 9005 Acked Flush 0 Sent Done Done false
31975 9006 Acked Write 6 Sent Done Done false
31976 9007 Acked Read 4 Sent Done Done false
31977 9008 Acked Read 10 Sent Done Done false
31978 9009 Acked Read 3 Sent Done Done false
31979 9010 Acked Write 4 Sent Done Done false
31980 9011 Acked Write 8 Sent Done Done false
31981 9012 Acked Write 5 Sent Done Done false
31982 9013 Acked Read 10 Sent Done Done false
31983 9014 Acked Write 6 Sent Done Done false
31984 9015 Acked Write 1 Sent Done Done false
31985 9016 Acked Read 2 Sent Done Done false
31986 9017 Acked Read 10 Sent Done Done false
31987 9018 Acked Write 6 Sent Done Done false
31988 9019 Acked Write 7 Sent Done Done false
31989 9020 Acked Read 8 Sent Done Done false
31990 9021 Acked Read 5 Sent Done Done false
31991 9022 Acked Read 2 Sent Done Done false
31992 9023 Acked Flush 0 Sent Done Done false
31993 9024 Acked Write 3 Sent Done Done false
31994 9025 Acked Read 9 Sent Done Done false
31995 9026 Acked Read 10 Sent Done Done false
31996 9027 Acked Read 9 Sent Done Done false
31997 9028 Acked Write 1 Sent Done Done false
31998 9029 Acked Read 8 Sent Done Done false
31999 9030 Acked Write 7 Sent Done Done false
32000 9031 Acked Read 4 Sent Done Done false
32001 9032 Acked Read 5 Sent Done Done false
32002 9033 Acked Write 4 Sent Done Done false
32003 9034 Acked Read 6 Sent Done Done false
32004 9035 Acked Flush 0 Sent Done Done false
32005 9036 Acked Read 7 Sent Done Done false
32006 9037 Acked Write 9 Sent Done Done false
32007 9038 Acked Read 5 Sent Done Done false
32008 9039 Acked Read 2 Sent Done Done false
32009 9040 Acked Write 3 Sent Done Done false
32010 9041 Acked Read 9 Sent Done Done false
32011 9042 Acked Read 4 Sent Done Done false
32012 9043 Acked Read 9 Sent Done Done false
32013 9044 Acked Write 2 Sent Done Done false
32014 9045 Acked Read 2 Sent Done Done false
32015 9046 Acked Write 10 Sent Done Done false
32016 9047 Acked Flush 0 Sent Done Done false
32017 9048 Acked Write 9 Sent Done Done false
32018 9049 Acked Write 6 Sent Done Done false
32019 9050 Acked Write 6 Sent Done Done false
32020 9051 Acked Read 7 Sent Done Done false
32021 9052 Acked Read 8 Sent Done Done false
32022 9053 Acked Read 6 Sent Done Done false
32023 9054 Acked Read 3 Sent Done Done false
32024 9055 Acked Read 10 Sent Done Done false
32025 9056 Acked Write 7 Sent Done Done false
32026 9057 Acked Read 3 Sent Done Done false
32027 9058 Acked Read 1 Sent Done Done false
32028 9059 Acked Read 9 Sent Done Done false
32029 9060 Acked Flush 0 Sent Done Done false
32030 9061 Acked Read 1 Sent Done Done false
32031 9062 Acked Write 3 Sent Done Done false
32032 9063 Acked Read 3 Sent Done Done false
32033 9064 Acked Flush 0 Sent Done Done false
32034 9065 Acked Read 4 Sent Done Done false
32035 9066 Acked Read 1 Sent Done Done false
32036 9067 Acked Write 1 Sent Done Done false
32037 9068 Acked Read 10 Sent Done Done false
32038 9069 Acked Read 7 Sent Done Done false
32039 9070 Acked Write 7 Sent Done Done false
32040 9071 Acked Flush 0 Sent Done Done false
32041 9072 Acked Write 4 Sent Done Done false
32042 9073 Acked Write 10 Sent Done Done false
32043 9074 Acked Read 10 Sent Done Done false
32044 9075 Acked Read 9 Sent Done Done false
32045 9076 Acked Read 3 Sent Done Done false
32046 9077 Acked Write 5 Sent Done Done false
32047 9078 Acked Write 7 Sent Done Done false
32048 9079 Acked Write 6 Sent Done Done false
32049 9080 Acked Write 4 Sent Done Done false
32050 9081 Acked Read 6 Sent Done Done false
32051 9082 Acked Write 10 Sent Done Done false
32052 9083 Acked Read 6 Sent Done Done false
32053 9084 Acked Write 5 Sent Done Done false
32054 9085 Acked Read 2 Sent Done Done false
32055 9086 Acked Write 10 Sent Done Done false
32056 9087 Acked Write 4 Sent Done Done false
32057 9088 Acked Write 6 Sent Done Done false
32058 9089 Acked Read 5 Sent Done Done false
32059 9090 Acked Write 6 Sent Done Done false
32060 9091 Acked Read 7 Sent Done Done false
32061 9092 Acked Write 8 Sent Done Done false
32062 9093 Acked Write 7 Sent Done Done false
32063 9094 Acked Write 2 Sent Done Done false
32064 9095 Acked Read 6 Sent Done Done false
32065 9096 Acked Read 7 Sent Done Done false
32066 9097 Acked Read 1 Sent Done Done false
32067 9098 Acked Read 7 Sent Done Done false
32068 9099 Acked Write 6 Sent Done Done false
32069 9100 Acked Flush 0 Sent Done Done false
32070 9101 Acked Flush 0 Sent Done Done false
32071 9102 Acked Read 7 Sent Done Done false
32072 9103 Acked Write 1 Sent Done Done false
32073 9104 Acked Write 3 Sent Done Done false
32074 9105 Acked Write 9 Sent Done Done false
32075 9106 Acked Read 2 Sent Done Done false
32076 9107 Acked Write 2 Sent Done Done false
32077 9108 Acked Read 10 Sent Done Done false
32078 9109 Acked Read 3 Sent Done Done false
32079 9110 Acked Read 9 Sent Done Done false
32080 9111 Acked Write 2 Sent Done Done false
32081 9112 Acked Read 10 Sent Done Done false
32082 9113 Acked Read 4 Sent Done Done false
32083 9114 Acked Write 7 Sent Done Done false
32084 9115 Acked Write 8 Sent Done Done false
32085 9116 Acked Read 3 Sent Done Done false
32086 9117 Acked Flush 0 Sent Done Done false
32087 9118 Acked Write 10 Sent Done Done false
32088 9119 Acked Read 2 Sent Done Done false
32089 9120 Acked Read 9 Sent Done Done false
32090 9121 Acked Read 10 Sent Done Done false
32091 9122 Acked Read 3 Sent Done Done false
32092 9123 Acked Write 4 Sent Done Done false
32093 9124 Acked Flush 0 Sent Done Done false
32094 9125 Acked Read 9 Sent Done Done false
32095 9126 Acked Read 5 Sent Done Done false
32096 9127 Acked Read 1 Sent Done Done false
32097 9128 Acked Read 10 Sent Done Done false
32098 9129 Acked Write 9 Sent Done Done false
32099 9130 Acked Write 1 Sent Done Done false
32100 9131 Acked Flush 0 Sent Done Done false
32101 9132 Acked Write 6 Sent Done Done false
32102 9133 Acked Read 10 Sent Done Done false
32103 9134 Acked Read 5 Sent Done Done false
32104 9135 Acked Read 2 Sent Done Done false
32105 9136 Acked Read 4 Sent Done Done false
32106 9137 Acked Write 5 Sent Done Done false
32107 9138 Acked Write 9 Sent Done Done false
32108 9139 Acked Read 1 Sent Done Done false
32109 9140 Acked Read 7 Sent Done Done false
32110 9141 Acked Read 10 Sent Done Done false
32111 9142 Acked Write 8 Sent Done Done false
32112 9143 Acked Read 4 Sent Done Done false
32113 9144 Acked Write 6 Sent Done Done false
32114 9145 Acked Read 5 Sent Done Done false
32115 9146 Acked Read 2 Sent Done Done false
32116 9147 Acked Read 8 Sent Done Done false
32117 9148 Acked Write 4 Sent Done Done false
32118 9149 Acked Flush 0 Sent Done Done false
32119 STATES DS:0 DS:1 DS:2 TOTAL
32120 Sent 301 0 0 301
32121 Done 0 301 301 602
32122 Skipped 0 0 0 0
32123 Error 0 0 0 0
32124 Last Flush: Some(JobId(8848)) Some(JobId(9149)) Some(JobId(9149))
32125 Downstairs last five completed: 8848 8847 8846 8845 8844
32126 Upstairs last five completed: 9149 9148 9147 9146 9145
32127 CLIENT: Up:0 ds:301 act:2
321282025-02-28T06:08:07.160ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
321292025-02-28T06:08:08.401ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
32130 [8810] got start action
32131 [8810] I am going to restart
32132 Starting downstairs at port 8810
32133 Make output file at "/tmp/dsc/downstairs-8810.txt"
32134 [8810][0] reports Starting
32135 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1238)
32136 [8810][0] reports Running
321372025-02-28T06:08:09.057ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
321382025-02-28T06:08:10.059ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
32139 ----------------------------------------------------------------
32140 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32141 States: Offline Active Active
32142 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32143 8849 Acked Read 6 Sent Done Done false
32144 8850 Acked Write 8 Sent Done Done false
32145 8851 Acked Read 4 Sent Done Done false
32146 8852 Acked Write 10 Sent Done Done false
32147 8853 Acked Read 7 Sent Done Done false
32148 8854 Acked Write 8 Sent Done Done false
32149 8855 Acked Read 10 Sent Done Done false
32150 8856 Acked Read 9 Sent Done Done false
32151 8857 Acked Flush 0 Sent Done Done false
32152 8858 Acked Read 2 Sent Done Done false
32153 8859 Acked Read 5 Sent Done Done false
32154 8860 Acked Read 4 Sent Done Done false
32155 8861 Acked Write 5 Sent Done Done false
32156 8862 Acked Write 10 Sent Done Done false
32157 8863 Acked Write 7 Sent Done Done false
32158 8864 Acked Read 7 Sent Done Done false
32159 8865 Acked Flush 0 Sent Done Done false
32160 8866 Acked Read 10 Sent Done Done false
32161 8867 Acked Write 6 Sent Done Done false
32162 8868 Acked Write 6 Sent Done Done false
32163 8869 Acked Read 3 Sent Done Done false
32164 8870 Acked Read 6 Sent Done Done false
32165 8871 Acked Write 8 Sent Done Done false
32166 8872 Acked Write 8 Sent Done Done false
32167 8873 Acked Read 9 Sent Done Done false
32168 8874 Acked Flush 0 Sent Done Done false
32169 8875 Acked Write 1 Sent Done Done false
32170 8876 Acked Read 9 Sent Done Done false
32171 8877 Acked Read 4 Sent Done Done false
32172 8878 Acked Read 7 Sent Done Done false
32173 8879 Acked Read 6 Sent Done Done false
32174 8880 Acked Read 3 Sent Done Done false
32175 8881 Acked Flush 0 Sent Done Done false
32176 8882 Acked Flush 0 Sent Done Done false
32177 8883 Acked Write 3 Sent Done Done false
32178 8884 Acked Read 6 Sent Done Done false
32179 8885 Acked Read 1 Sent Done Done false
32180 8886 Acked Flush 0 Sent Done Done false
32181 8887 Acked Read 1 Sent Done Done false
32182 8888 Acked Read 6 Sent Done Done false
32183 8889 Acked Read 3 Sent Done Done false
32184 8890 Acked Write 8 Sent Done Done false
32185 8891 Acked Write 4 Sent Done Done false
32186 8892 Acked Read 2 Sent Done Done false
32187 8893 Acked Read 2 Sent Done Done false
32188 8894 Acked Read 3 Sent Done Done false
32189 8895 Acked Read 1 Sent Done Done false
32190 8896 Acked Read 5 Sent Done Done false
32191 8897 Acked Write 10 Sent Done Done false
32192 8898 Acked Read 1 Sent Done Done false
32193 8899 Acked Flush 0 Sent Done Done false
32194 8900 Acked Read 3 Sent Done Done false
32195 8901 Acked Write 9 Sent Done Done false
32196 8902 Acked Write 4 Sent Done Done false
32197 8903 Acked Read 9 Sent Done Done false
32198 8904 Acked Write 8 Sent Done Done false
32199 8905 Acked Flush 0 Sent Done Done false
32200 8906 Acked Read 8 Sent Done Done false
32201 8907 Acked Read 8 Sent Done Done false
32202 8908 Acked Write 4 Sent Done Done false
32203 8909 Acked Write 7 Sent Done Done false
32204 8910 Acked Flush 0 Sent Done Done false
32205 8911 Acked Write 3 Sent Done Done false
32206 8912 Acked Write 2 Sent Done Done false
32207 8913 Acked Write 3 Sent Done Done false
32208 8914 Acked Read 1 Sent Done Done false
32209 8915 Acked Flush 0 Sent Done Done false
32210 8916 Acked Read 7 Sent Done Done false
32211 8917 Acked Read 9 Sent Done Done false
32212 8918 Acked Flush 0 Sent Done Done false
32213 8919 Acked Read 5 Sent Done Done false
32214 8920 Acked Read 3 Sent Done Done false
32215 8921 Acked Flush 0 Sent Done Done false
32216 8922 Acked Read 4 Sent Done Done false
32217 8923 Acked Read 6 Sent Done Done false
32218 8924 Acked Write 5 Sent Done Done false
32219 8925 Acked Read 5 Sent Done Done false
32220 8926 Acked Read 10 Sent Done Done false
32221 8927 Acked Read 3 Sent Done Done false
32222 8928 Acked Write 9 Sent Done Done false
32223 8929 Acked Write 3 Sent Done Done false
32224 8930 Acked Write 5 Sent Done Done false
32225 8931 Acked Write 5 Sent Done Done false
32226 8932 Acked Read 9 Sent Done Done false
32227 8933 Acked Write 9 Sent Done Done false
32228 8934 Acked Write 8 Sent Done Done false
32229 8935 Acked Write 6 Sent Done Done false
32230 8936 Acked Write 4 Sent Done Done false
32231 8937 Acked Flush 0 Sent Done Done false
32232 8938 Acked Write 2 Sent Done Done false
32233 8939 Acked Read 2 Sent Done Done false
32234 8940 Acked Read 8 Sent Done Done false
32235 8941 Acked Read 2 Sent Done Done false
32236 8942 Acked Read 6 Sent Done Done false
32237 8943 Acked Write 8 Sent Done Done false
32238 8944 Acked Write 7 Sent Done Done false
32239 8945 Acked Write 1 Sent Done Done false
32240 8946 Acked Read 6 Sent Done Done false
32241 8947 Acked Flush 0 Sent Done Done false
32242 8948 Acked Read 8 Sent Done Done false
32243 8949 Acked Write 8 Sent Done Done false
32244 8950 Acked Write 6 Sent Done Done false
32245 8951 Acked Write 3 Sent Done Done false
32246 8952 Acked Read 3 Sent Done Done false
32247 8953 Acked Write 7 Sent Done Done false
32248 8954 Acked Flush 0 Sent Done Done false
32249 8955 Acked Read 3 Sent Done Done false
32250 8956 Acked Write 2 Sent Done Done false
32251 8957 Acked Write 3 Sent Done Done false
32252 8958 Acked Write 5 Sent Done Done false
32253 8959 Acked Flush 0 Sent Done Done false
32254 8960 Acked Write 8 Sent Done Done false
32255 8961 Acked Write 8 Sent Done Done false
32256 8962 Acked Read 3 Sent Done Done false
32257 8963 Acked Read 6 Sent Done Done false
32258 8964 Acked Read 3 Sent Done Done false
32259 8965 Acked Write 3 Sent Done Done false
32260 8966 Acked Read 3 Sent Done Done false
32261 8967 Acked Read 9 Sent Done Done false
32262 8968 Acked Write 9 Sent Done Done false
32263 8969 Acked Write 10 Sent Done Done false
32264 8970 Acked Write 9 Sent Done Done false
32265 8971 Acked Read 9 Sent Done Done false
32266 8972 Acked Read 7 Sent Done Done false
32267 8973 Acked Read 7 Sent Done Done false
32268 8974 Acked Read 8 Sent Done Done false
32269 8975 Acked Write 3 Sent Done Done false
32270 8976 Acked Read 3 Sent Done Done false
32271 8977 Acked Flush 0 Sent Done Done false
32272 8978 Acked Write 7 Sent Done Done false
32273 8979 Acked Read 7 Sent Done Done false
32274 8980 Acked Read 8 Sent Done Done false
32275 8981 Acked Write 4 Sent Done Done false
32276 8982 Acked Read 5 Sent Done Done false
32277 8983 Acked Read 8 Sent Done Done false
32278 8984 Acked Write 2 Sent Done Done false
32279 8985 Acked Write 9 Sent Done Done false
32280 8986 Acked Write 3 Sent Done Done false
32281 8987 Acked Read 8 Sent Done Done false
32282 8988 Acked Read 3 Sent Done Done false
32283 8989 Acked Write 10 Sent Done Done false
32284 8990 Acked Read 2 Sent Done Done false
32285 8991 Acked Write 8 Sent Done Done false
32286 8992 Acked Read 8 Sent Done Done false
32287 8993 Acked Read 1 Sent Done Done false
32288 8994 Acked Read 2 Sent Done Done false
32289 8995 Acked Write 2 Sent Done Done false
32290 8996 Acked Write 4 Sent Done Done false
32291 8997 Acked Flush 0 Sent Done Done false
32292 8998 Acked Read 3 Sent Done Done false
32293 8999 Acked Write 6 Sent Done Done false
32294 9000 Acked Write 5 Sent Done Done false
32295 9001 Acked Read 1 Sent Done Done false
32296 9002 Acked Read 8 Sent Done Done false
32297 9003 Acked Read 9 Sent Done Done false
32298 9004 Acked Write 3 Sent Done Done false
32299 9005 Acked Flush 0 Sent Done Done false
32300 9006 Acked Write 6 Sent Done Done false
32301 9007 Acked Read 4 Sent Done Done false
32302 9008 Acked Read 10 Sent Done Done false
32303 9009 Acked Read 3 Sent Done Done false
32304 9010 Acked Write 4 Sent Done Done false
32305 9011 Acked Write 8 Sent Done Done false
32306 9012 Acked Write 5 Sent Done Done false
32307 9013 Acked Read 10 Sent Done Done false
32308 9014 Acked Write 6 Sent Done Done false
32309 9015 Acked Write 1 Sent Done Done false
32310 9016 Acked Read 2 Sent Done Done false
32311 9017 Acked Read 10 Sent Done Done false
32312 9018 Acked Write 6 Sent Done Done false
32313 9019 Acked Write 7 Sent Done Done false
32314 9020 Acked Read 8 Sent Done Done false
32315 9021 Acked Read 5 Sent Done Done false
32316 9022 Acked Read 2 Sent Done Done false
32317 9023 Acked Flush 0 Sent Done Done false
32318 9024 Acked Write 3 Sent Done Done false
32319 9025 Acked Read 9 Sent Done Done false
32320 9026 Acked Read 10 Sent Done Done false
32321 9027 Acked Read 9 Sent Done Done false
32322 9028 Acked Write 1 Sent Done Done false
32323 9029 Acked Read 8 Sent Done Done false
32324 9030 Acked Write 7 Sent Done Done false
32325 9031 Acked Read 4 Sent Done Done false
32326 9032 Acked Read 5 Sent Done Done false
32327 9033 Acked Write 4 Sent Done Done false
32328 9034 Acked Read 6 Sent Done Done false
32329 9035 Acked Flush 0 Sent Done Done false
32330 9036 Acked Read 7 Sent Done Done false
32331 9037 Acked Write 9 Sent Done Done false
32332 9038 Acked Read 5 Sent Done Done false
32333 9039 Acked Read 2 Sent Done Done false
32334 9040 Acked Write 3 Sent Done Done false
32335 9041 Acked Read 9 Sent Done Done false
32336 9042 Acked Read 4 Sent Done Done false
32337 9043 Acked Read 9 Sent Done Done false
32338 9044 Acked Write 2 Sent Done Done false
32339 9045 Acked Read 2 Sent Done Done false
32340 9046 Acked Write 10 Sent Done Done false
32341 9047 Acked Flush 0 Sent Done Done false
32342 9048 Acked Write 9 Sent Done Done false
32343 9049 Acked Write 6 Sent Done Done false
32344 9050 Acked Write 6 Sent Done Done false
32345 9051 Acked Read 7 Sent Done Done false
32346 9052 Acked Read 8 Sent Done Done false
32347 9053 Acked Read 6 Sent Done Done false
32348 9054 Acked Read 3 Sent Done Done false
32349 9055 Acked Read 10 Sent Done Done false
32350 9056 Acked Write 7 Sent Done Done false
32351 9057 Acked Read 3 Sent Done Done false
32352 9058 Acked Read 1 Sent Done Done false
32353 9059 Acked Read 9 Sent Done Done false
32354 9060 Acked Flush 0 Sent Done Done false
32355 9061 Acked Read 1 Sent Done Done false
32356 9062 Acked Write 3 Sent Done Done false
32357 9063 Acked Read 3 Sent Done Done false
32358 9064 Acked Flush 0 Sent Done Done false
32359 9065 Acked Read 4 Sent Done Done false
32360 9066 Acked Read 1 Sent Done Done false
32361 9067 Acked Write 1 Sent Done Done false
32362 9068 Acked Read 10 Sent Done Done false
32363 9069 Acked Read 7 Sent Done Done false
32364 9070 Acked Write 7 Sent Done Done false
32365 9071 Acked Flush 0 Sent Done Done false
32366 9072 Acked Write 4 Sent Done Done false
32367 9073 Acked Write 10 Sent Done Done false
32368 9074 Acked Read 10 Sent Done Done false
32369 9075 Acked Read 9 Sent Done Done false
32370 9076 Acked Read 3 Sent Done Done false
32371 9077 Acked Write 5 Sent Done Done false
32372 9078 Acked Write 7 Sent Done Done false
32373 9079 Acked Write 6 Sent Done Done false
32374 9080 Acked Write 4 Sent Done Done false
32375 9081 Acked Read 6 Sent Done Done false
32376 9082 Acked Write 10 Sent Done Done false
32377 9083 Acked Read 6 Sent Done Done false
32378 9084 Acked Write 5 Sent Done Done false
32379 9085 Acked Read 2 Sent Done Done false
32380 9086 Acked Write 10 Sent Done Done false
32381 9087 Acked Write 4 Sent Done Done false
32382 9088 Acked Write 6 Sent Done Done false
32383 9089 Acked Read 5 Sent Done Done false
32384 9090 Acked Write 6 Sent Done Done false
32385 9091 Acked Read 7 Sent Done Done false
32386 9092 Acked Write 8 Sent Done Done false
32387 9093 Acked Write 7 Sent Done Done false
32388 9094 Acked Write 2 Sent Done Done false
32389 9095 Acked Read 6 Sent Done Done false
32390 9096 Acked Read 7 Sent Done Done false
32391 9097 Acked Read 1 Sent Done Done false
32392 9098 Acked Read 7 Sent Done Done false
32393 9099 Acked Write 6 Sent Done Done false
32394 9100 Acked Flush 0 Sent Done Done false
32395 9101 Acked Flush 0 Sent Done Done false
32396 9102 Acked Read 7 Sent Done Done false
32397 9103 Acked Write 1 Sent Done Done false
32398 9104 Acked Write 3 Sent Done Done false
32399 9105 Acked Write 9 Sent Done Done false
32400 9106 Acked Read 2 Sent Done Done false
32401 9107 Acked Write 2 Sent Done Done false
32402 9108 Acked Read 10 Sent Done Done false
32403 9109 Acked Read 3 Sent Done Done false
32404 9110 Acked Read 9 Sent Done Done false
32405 9111 Acked Write 2 Sent Done Done false
32406 9112 Acked Read 10 Sent Done Done false
32407 9113 Acked Read 4 Sent Done Done false
32408 9114 Acked Write 7 Sent Done Done false
32409 9115 Acked Write 8 Sent Done Done false
32410 9116 Acked Read 3 Sent Done Done false
32411 9117 Acked Flush 0 Sent Done Done false
32412 9118 Acked Write 10 Sent Done Done false
32413 9119 Acked Read 2 Sent Done Done false
32414 9120 Acked Read 9 Sent Done Done false
32415 9121 Acked Read 10 Sent Done Done false
32416 9122 Acked Read 3 Sent Done Done false
32417 9123 Acked Write 4 Sent Done Done false
32418 9124 Acked Flush 0 Sent Done Done false
32419 9125 Acked Read 9 Sent Done Done false
32420 9126 Acked Read 5 Sent Done Done false
32421 9127 Acked Read 1 Sent Done Done false
32422 9128 Acked Read 10 Sent Done Done false
32423 9129 Acked Write 9 Sent Done Done false
32424 9130 Acked Write 1 Sent Done Done false
32425 9131 Acked Flush 0 Sent Done Done false
32426 9132 Acked Write 6 Sent Done Done false
32427 9133 Acked Read 10 Sent Done Done false
32428 9134 Acked Read 5 Sent Done Done false
32429 9135 Acked Read 2 Sent Done Done false
32430 9136 Acked Read 4 Sent Done Done false
32431 9137 Acked Write 5 Sent Done Done false
32432 9138 Acked Write 9 Sent Done Done false
32433 9139 Acked Read 1 Sent Done Done false
32434 9140 Acked Read 7 Sent Done Done false
32435 9141 Acked Read 10 Sent Done Done false
32436 9142 Acked Write 8 Sent Done Done false
32437 9143 Acked Read 4 Sent Done Done false
32438 9144 Acked Write 6 Sent Done Done false
32439 9145 Acked Read 5 Sent Done Done false
32440 9146 Acked Read 2 Sent Done Done false
32441 9147 Acked Read 8 Sent Done Done false
32442 9148 Acked Write 4 Sent Done Done false
32443 9149 Acked Flush 0 Sent Done Done false
32444 STATES DS:0 DS:1 DS:2 TOTAL
32445 Sent 301 0 0 301
32446 Done 0 301 301 602
32447 Skipped 0 0 0 0
32448 Error 0 0 0 0
32449 Last Flush: Some(JobId(8848)) Some(JobId(9149)) Some(JobId(9149))
32450 Downstairs last five completed: 8848 8847 8846 8845 8844
32451 Upstairs last five completed: 9149 9148 9147 9146 9145
32452 CLIENT: Up:0 ds:301 act:2
324532025-02-28T06:08:11.162ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
324542025-02-28T06:08:12.062ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
324552025-02-28T06:08:12.412ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
324562025-02-28T06:08:12.412ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56235) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
324572025-02-28T06:08:12.413ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
324582025-02-28T06:08:12.414ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
324592025-02-28T06:08:12.414ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
324602025-02-28T06:08:12.414ZINFOcrucible: send last flush ID to this DS: Some(JobId(8848)) = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
324612025-02-28T06:08:12.414ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
324622025-02-28T06:08:12.414ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(8848)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
324632025-02-28T06:08:12.414ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(8848)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
32464 ----------------------------------------------------------------
32465 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
32466 Downstairs last five completed: 9149 9148 9147 9146 9145
32467 Upstairs last five completed: 9149 9148 9147 9146 9145
32468 CLIENT: Up:0 ds:0 act:3
32469 Replay: All jobs finished
32470 Main task has work to do, go find it
32471 got dsc Stop(2)
32472 stop 2
32473 [8830] Got stop action so:false kr:false
32474 Feb 28 06:08:14.664 INFO request completed, latency_us: 394, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 4f4134da-3066-4fb3-9f33-25f2f200ed69, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
324752025-02-28T06:08:14.665ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
32476 Feb 28 06:08:14.665 INFO request completed{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:14.665650154Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
32477 , latency_us: 154, response_code: 200, uri: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:14.66566652Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
32478 /state/cid/2{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:14.665746761Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
32479 , method: GET, req_id: be308082-160c-4f8a-b4f2-94a9c2808a1d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
32480 [8830] Exited with: ExitStatus(unix_wait_status(9))
32481 [8830][2] reports Exit
32482 Feb 28 06:08:18.668 INFO request completed, latency_us: 376, response_code: 200, uri: /state/cid/2, method: GET, req_id: b08a0b17-0a31-42c3-8f5b-13cc495fb6ec, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
32483 001/300 Read block 1110 len 24576
32484 002/300 Read block 1886 len 8192
32485 003/300 Read block 1641 len 40960
32486 004/300 Flush
32487 005/300 Write block 1477 len 28672 data: 5 5 5 4 5 5 7
32488 006/300 Read block 361 len 16384
32489 007/300 Read block 1669 len 36864
32490 008/300 Write block 1292 len 36864 data: 9 8 6 5 5 6 7 6 6
32491 009/300 Read block 239 len 40960
32492 010/300 Read block 4077 len 32768
32493 011/300 Read block 3263 len 4096
32494 012/300 Read block 3719 len 40960
32495 013/300 Write block 1634 len 12288 data: 5 4 5
32496 014/300 Read block 3426 len 20480
32497 015/300 Write block 1220 len 12288 data: 5 5 8
32498 016/300 Write block 164 len 12288 data: 5 4 4
32499 017/300 Read block 900 len 32768
32500 018/300 Write block 1829 len 32768 data: 6 4 4 5 4 4 5 7
32501 019/300 Write block 2370 len 12288 data: 4 5 4
32502 020/300 Write block 2064 len 8192 data: 6 7
32503 021/300 Write block 2582 len 8192 data: 7 7
32504 022/300 Write block 3389 len 28672 data: 4 4 4 3 3 3 4
32505 023/300 Write block 4579 len 12288 data: 3 3 3
32506 024/300 Write block 3238 len 8192 data: 3 3
32507 025/300 Write block 3067 len 36864 data: 6 5 5 5 5 6 7 7 6
32508 026/300 Flush
32509 027/300 Write block 2734 len 8192 data: 6 7
32510 028/300 Read block 939 len 4096
32511 029/300 Write block 4659 len 4096 data: 4
32512 030/300 Write block 459 len 32768 data: 5 5 5 4 2 2 4 6
32513 031/300 Write block 4209 len 24576 data: 5 5 5 7 8 6
32514 032/300 Write block 4736 len 28672 data: 8 6 7 7 7 6 5
32515 033/300 Write block 3558 len 4096 data: 5
32516 034/300 Write block 2408 len 20480 data: 2 2 2 3 3
32517 035/300 Read block 1542 len 16384
32518 036/300 Read block 4180 len 36864
32519 037/300 Read block 1944 len 36864
32520 038/300 Flush
32521 039/300 Read block 4214 len 16384
32522 040/300 Write block 344 len 12288 data: 5 8 7
32523 041/300 Write block 4203 len 4096 data: 4
32524 042/300 Read block 1841 len 8192
32525 043/300 Flush
32526 044/300 Read block 4147 len 8192
32527 045/300 Write block 4885 len 28672 data: 8 11 16 15 16 15 14
32528 046/300 Write block 73 len 36864 data: 7 6 9 8 8 7 9 8 10
32529 047/300 Write block 1658 len 32768 data: 6 6 8 7 6 7 7 10
32530 048/300 Flush
32531 049/300 Write block 3879 len 4096 data: 4
32532 050/300 Read block 3291 len 28672
32533 051/300 Read block 2752 len 36864
32534 052/300 Read block 2920 len 36864
32535 053/300 Write block 206 len 8192 data: 6 6
32536 054/300 Write block 190 len 16384 data: 8 9 8 7
32537 055/300 Read block 4395 len 24576
32538 056/300 Read block 937 len 28672
32539 057/300 Read block 4797 len 40960
32540 058/300 Read block 1295 len 12288
32541 059/300 Read block 4402 len 8192
32542 060/300 Read block 2848 len 4096
32543 061/300 Flush
32544 062/300 Read block 3578 len 32768
32545 063/300 Write block 2082 len 28672 data: 5 7 5 7 9 9 9
32546 064/300 Flush
32547 065/300 Write block 2457 len 12288 data: 9 9 7
32548 066/300 Write block 719 len 20480 data: 5 6 6 4 5
32549 067/300 Write block 3152 len 40960 data: 5 5 7 7 10 10 10 11 9 5
32550 068/300 Flush
32551 069/300 Write block 3027 len 28672 data: 6 6 6 6 6 7 6
32552 070/300 Read block 2210 len 20480
32553 071/300 Read block 490 len 12288
32554 072/300 Read block 3735 len 12288
32555 073/300 Write block 2219 len 32768 data: 8 7 4 4 4 4 5 6
32556 074/300 Read block 535 len 36864
32557 075/300 Read block 3739 len 40960
32558 076/300 Write block 4295 len 20480 data: 5 5 5 6 4
32559 077/300 Read block 1143 len 32768
32560 078/300 Write block 3640 len 36864 data: 3 4 3 3 3 5 5 5 5
32561 079/300 Read block 427 len 16384
32562 080/300 Write block 1681 len 32768 data: 5 8 9 9 7 8 7 6
32563 081/300 Read block 2847 len 32768
32564 082/300 Write block 1444 len 16384 data: 6 7 5 4
32565 083/300 Flush
32566 084/300 Read block 1902 len 4096
32567 085/300 Write block 2197 len 16384 data: 5 4 5 7
32568 086/300 Write block 4984 len 28672 data: 7 6 5 5 2 3 3
32569 087/300 Read block 4263 len 4096
32570 088/300 Read block 1029 len 16384
32571 089/300 Write block 2628 len 12288 data: 6 6 6
32572 090/300 Write block 1704 len 4096 data: 6
32573 091/300 Write block 4093 len 24576 data: 5 5 7 7 7 7
32574 092/300 Write block 4173 len 12288 data: 4 5 5
32575 093/300 Read block 2076 len 20480
32576 094/300 Read block 583 len 36864
32577 095/300 Write block 873 len 16384 data: 6 5 4 5
32578 096/300 Write block 1199 len 40960 data: 5 4 4 4 5 4 5 4 4 4
32579 097/300 Read block 288 len 32768
32580 098/300 Flush
32581 099/300 Write block 2611 len 36864 data: 6 4 4 3 4 4 4 4 4
32582 100/300 Flush
32583 101/300 Write block 1344 len 16384 data: 4 4 3 3
32584 102/300 Read block 2393 len 12288
32585 103/300 Read block 665 len 28672
32586 104/300 Flush
32587 105/300 Read block 2061 len 4096
32588 106/300 Read block 4532 len 36864
32589 107/300 Flush
32590 108/300 Read block 2914 len 28672
32591 109/300 Read block 790 len 20480
32592 110/300 Read block 1112 len 12288
32593 111/300 Write block 2742 len 4096 data: 6
32594 112/300 Write block 1091 len 8192 data: 3 3
32595 113/300 Read block 43 len 28672
32596 114/300 Read block 3080 len 36864
32597 115/300 Flush
32598 116/300 Read block 1304 len 4096
32599 117/300 Read block 2874 len 16384
32600 118/300 Write block 3055 len 4096 data: 4
32601 119/300 Read block 678 len 28672
32602 120/300 Read block 3980 len 16384
32603 121/300 Read block 1919 len 40960
32604 122/300 Flush
32605 123/300 Read block 3841 len 28672
32606 124/300 Write block 4310 len 36864 data: 5 5 6 5 5 4 4 4 5
32607 125/300 Read block 273 len 32768
32608 126/300 Read block 2664 len 20480
32609 127/300 Write block 1456 len 40960 data: 6 6 6 8 7 5 6 6 6 5
32610 128/300 Read block 4383 len 4096
32611 129/300 Flush
32612 130/300 Write block 4019 len 32768 data: 6 5 5 5 5 4 4 3
32613 131/300 Read block 3189 len 16384
32614 132/300 Read block 4497 len 32768
32615 133/300 Write block 1976 len 8192 data: 6 4
32616 134/300 Read block 498 len 32768
32617 135/300 Write block 2261 len 28672 data: 7 7 7 9 7 7 8
32618 136/300 Write block 1655 len 16384 data: 4 6 5 7
32619 137/300 Write block 4640 len 28672 data: 6 4 5 4 5 5 6
32620 138/300 Read block 4557 len 32768
32621 139/300 Write block 4073 len 28672 data: 4 4 5 6 7 6 7
32622 140/300 Read block 2159 len 32768
32623 141/300 Read block 3446 len 28672
32624 142/300 Flush
32625 143/300 Read block 1545 len 8192
32626 144/300 Read block 3572 len 32768
32627 145/300 Write block 2460 len 20480 data: 7 6 4 4 3
32628 146/300 Read block 390 len 8192
32629 147/300 Read block 944 len 24576
32630 148/300 Read block 4411 len 32768
32631 149/300 Flush
32632 150/300 Read block 197 len 8192
32633 151/300 Write block 2033 len 24576 data: 6 4 3 3 3 3
32634 152/300 Read block 4825 len 20480
32635 153/300 Read block 817 len 32768
32636 154/300 Read block 2736 len 16384
32637 155/300 Read block 1186 len 36864
32638 156/300 Write block 1627 len 16384 data: 7 8 7 7
32639 157/300 Read block 849 len 32768
32640 158/300 Write block 2111 len 24576 data: 8 7 7 6 5 6
32641 159/300 Read block 4818 len 28672
32642 160/300 Read block 4977 len 12288
32643 161/300 Write block 3495 len 8192 data: 5 4
32644 162/300 Read block 4207 len 12288
32645 163/300 Read block 2164 len 32768
32646 164/300 Read block 1024 len 28672
32647 165/300 Write block 3437 len 8192 data: 7 7
32648 166/300 Read block 299 len 4096
32649 167/300 Flush
32650 168/300 Write block 1852 len 36864 data: 6 4 4 3 3 3 2 2 2
32651 169/300 Read block 306 len 24576
32652 170/300 Write block 3675 len 40960 data: 4 5 4 5 5 6 6 7 6 6
32653 171/300 Read block 1455 len 32768
32654 172/300 Write block 4841 len 16384 data: 7 8 7 6
32655 173/300 Write block 4002 len 28672 data: 4 4 5 5 7 7 7
32656 174/300 Read block 2875 len 20480
32657 175/300 Write block 2610 len 24576 data: 4 7 5 5 4 5
32658 176/300 Write block 1060 len 24576 data: 6 5 5 4 4 6
32659 177/300 Read block 3229 len 24576
32660 178/300 Read block 4622 len 40960
32661 179/300 Write block 262 len 24576 data: 3 3 4 5 5 4
32662 180/300 Read block 801 len 24576
32663 181/300 Read block 3141 len 4096
32664 182/300 Read block 4943 len 36864
32665 183/300 Write block 3119 len 28672 data: 6 4 4 4 4 4 4
32666 184/300 Read block 3631 len 40960
32667 185/300 Read block 4438 len 20480
32668 186/300 Write block 3707 len 20480 data: 8 6 5 4 4
32669 187/300 Read block 4902 len 28672
32670 188/300 Write block 1540 len 4096 data: 7
32671 189/300 Read block 1763 len 40960
32672 190/300 Write block 2833 len 36864 data: 4 3 4 4 5 5 5 5 6
32673 191/300 Write block 3494 len 36864 data: 5 6 5 6 6 6 4 3 4
32674 192/300 Read block 2447 len 16384
32675 193/300 Read block 1970 len 36864
32676 194/300 Read block 508 len 8192
32677 195/300 Write block 1445 len 16384 data: 8 6 5 5
32678 196/300 Read block 2579 len 32768
32679 197/300 Write block 2865 len 32768 data: 4 5 5 5 5 6 7 8
32680 198/300 Write block 291 len 40960 data: 6 7 6 5 7 4 5 5 5 3
32681 199/300 Write block 1752 len 20480 data: 5 5 6 7 6
32682 200/300 Read block 1772 len 4096
32683 201/300 Read block 1700 len 8192
32684 202/300 Write block 721 len 32768 data: 7 5 6 6 4 4 3 3
32685 203/300 Read block 3563 len 12288
32686 204/300 Write block 3306 len 8192 data: 8 9
32687 205/300 Read block 1049 len 16384
32688 206/300 Write block 4687 len 20480 data: 7 5 5 5 5
32689 207/300 Read block 2430 len 4096
32690 208/300 Write block 539 len 4096 data: 4
32691 209/300 Read block 4169 len 40960
32692 210/300 Write block 4275 len 16384 data: 6 5 6 6
32693 211/300 Read block 2902 len 4096
32694 212/300 Read block 1545 len 20480
32695 213/300 Read block 3218 len 32768
32696 214/300 Flush
32697 215/300 Read block 1925 len 24576
32698 216/300 Read block 2555 len 12288
32699 217/300 Read block 3375 len 12288
32700 218/300 Read block 1353 len 40960
32701 219/300 Read block 1384 len 12288
32702 220/300 Read block 4048 len 8192
32703 221/300 Write block 2288 len 40960 data: 4 3 4 6 5 5 3 3 3 4
32704 222/300 Flush
32705 223/300 Read block 4109 len 16384
32706 224/300 Read block 825 len 28672
32707 225/300 Read block 2078 len 28672
32708 226/300 Write block 3861 len 28672 data: 6 6 6 8 8 9 10
32709 227/300 Read block 4896 len 12288
32710 228/300 Flush
32711 229/300 Read block 4299 len 8192
32712 230/300 Write block 1309 len 12288 data: 7 7 8
32713 231/300 Read block 4104 len 28672
32714 232/300 Write block 4352 len 28672 data: 7 8 7 9 7 8 9
32715 233/300 Read block 4000 len 28672
32716 234/300 Read block 1356 len 28672
32717 235/300 Read block 931 len 28672
32718 236/300 Write block 2416 len 12288 data: 4 4 5
32719 237/300 Read block 4047 len 40960
32720 238/300 Flush
32721 239/300 Read block 3814 len 40960
32722 240/300 Write block 605 len 8192 data: 3 3
32723 241/300 Read block 2917 len 20480
32724 242/300 Write block 1222 len 40960 data: 9 8 8 7 8 9 8 7 7 4
32725 243/300 Read block 2145 len 24576
32726 244/300 Write block 3479 len 24576 data: 6 9 8 7 7 6
32727 245/300 Write block 3142 len 4096 data: 4
32728 246/300 Read block 504 len 36864
32729 247/300 Read block 2964 len 4096
32730 248/300 Write block 962 len 32768 data: 4 4 4 5 5 4 4 3
32731 249/300 Read block 3897 len 36864
32732 250/300 Read block 1246 len 8192
32733 251/300 Read block 805 len 28672
32734 252/300 Read block 480 len 32768
32735 253/300 Read block 2163 len 8192
32736 254/300 Read block 3596 len 12288
32737 255/300 Write block 2315 len 4096 data: 3
32738 256/300 Read block 276 len 36864
32739 257/300 Read block 1156 len 4096
32740 258/300 Read block 3710 len 24576
32741 259/300 Read block 2844 len 40960
32742 260/300 Write block 1496 len 12288 data: 5 4 4
32743 261/300 Read block 2244 len 28672
32744 262/300 Write block 12 len 8192 data: 5 3
32745 263/300 Write block 3819 len 16384 data: 5 4 4 5
32746 264/300 Read block 3587 len 20480
32747 265/300 Read block 2295 len 4096
32748 266/300 Write block 3617 len 16384 data: 6 5 6 4
32749 267/300 Read block 3423 len 32768
32750 268/300 Read block 892 len 20480
32751 269/300 Read block 2900 len 36864
32752 270/300 Read block 862 len 12288
32753 271/300 Write block 4460 len 8192 data: 4 6
32754 272/300 Read block 2181 len 36864
32755 273/300 Write block 386 len 32768 data: 6 4 4 4 5 5 5 4
32756 274/300 Write block 4642 len 40960 data: 6 5 6 6 7 9 8 8 7 7
32757 275/300 Read block 2451 len 32768
32758 276/300 Flush
32759 277/300 Read block 4845 len 32768
32760 278/300 Write block 3253 len 12288 data: 5 5 4
32761 279/300 Read block 3270 len 32768
32762 280/300 Read block 2269 len 28672
32763 281/300 Write block 4213 len 32768 data: 9 7 7 7 6 3 2 2
32764 282/300 Flush
32765 283/300 Read block 970 len 24576
32766 284/300 Write block 977 len 20480 data: 6 7 6 7 6
32767 285/300 Write block 1341 len 28672 data: 7 7 6 5 5 4 4
32768 286/300 Read block 3412 len 24576
32769 287/300 Flush
32770 288/300 Write block 920 len 32768 data: 7 7 7 6 6 6 7 7
32771 289/300 Read block 4769 len 12288
32772 290/300 Read block 3196 len 8192
32773 291/300 Read block 2102 len 4096
32774 292/300 Flush
32775 293/300 Read block 1485 len 8192
32776 294/300 Read block 286 len 28672
32777 295/300 Write block 4653 len 28672 data: 9 9 7 5 4 4 5
32778 296/300 Read block 2052 len 36864
32779 297/300 Read block 853 len 12288
32780 298/300 Write block 1180 len 20480 data: 8 7 5 4 6
32781 299/300 Write block 4199 len 20480 data: 5 2 2 2 5
32782 300/300 Write block 4495 len 12288 data: 8 9 7
32783 301/300 Write block 702 len 32768 data: 7 7 8 6 7 7 6 8
32784 Main task has work to do, go find it
32785 got dsc Start(2)
32786 start 2
32787 Feb 28 06:08:18.917 INFO request completed, latency_us: 192, response_code: 204, uri: /start/cid/2, method: POST, req_id: 8ba06243-e68f-4c20-ad36-5a06a50dbed5, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
32788 Replay: started 2, returned:Ok(())
32789 ----------------------------------------------------------------
32790 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32791 States: Active Active Offline
32792 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32793 9150 Acked Read 6 Done Done Sent false
32794 9151 Acked Read 2 Done Done Sent false
32795 9152 Acked Read 10 Done Done Sent false
32796 9153 Acked Flush 0 Done Done Sent false
32797 9154 Acked Write 7 Done Done Sent false
32798 9155 Acked Read 4 Done Done Sent false
32799 9156 Acked Read 9 Done Done Sent false
32800 9157 Acked Write 9 Done Done Sent false
32801 9158 Acked Read 10 Done Done Sent false
32802 9159 Acked Read 8 Done Done Sent false
32803 9160 Acked Read 1 Done Done Sent false
32804 9161 Acked Read 10 Done Done Sent false
32805 9162 Acked Write 3 Done Done Sent false
32806 9163 Acked Read 5 Done Done Sent false
32807 9164 Acked Write 3 Done Done Sent false
32808 9165 Acked Write 3 Done Done Sent false
32809 9166 Acked Read 8 Done Done Sent false
32810 9167 Acked Write 8 Done Done Sent false
32811 9168 Acked Write 3 Done Done Sent false
32812 9169 Acked Write 2 Done Done Sent false
32813 9170 Acked Write 2 Done Done Sent false
32814 9171 Acked Write 7 Done Done Sent false
32815 9172 Acked Write 3 Done Done Sent false
32816 9173 Acked Write 2 Done Done Sent false
32817 9174 Acked Write 9 Done Done Sent false
32818 9175 Acked Flush 0 Done Done Sent false
32819 9176 Acked Write 2 Done Done Sent false
32820 9177 Acked Read 1 Done Done Sent false
32821 9178 Acked Write 1 Done Done Sent false
32822 9179 Acked Write 8 Done Done Sent false
32823 9180 Acked Write 6 Done Done Sent false
32824 9181 Acked Write 7 Done Done Sent false
32825 9182 Acked Write 1 Done Done Sent false
32826 9183 Acked Write 5 Done Done Sent false
32827 9184 Acked Read 4 Done Done Sent false
32828 9185 Acked Read 9 Done Done Sent false
32829 9186 Acked Read 9 Done Done Sent false
32830 9187 Acked Flush 0 Done Done Sent false
32831 9188 Acked Read 4 Done Done Sent false
32832 9189 Acked Write 3 Done Done Sent false
32833 9190 Acked Write 1 Done Done Sent false
32834 9191 Acked Read 2 Done Done Sent false
32835 9192 Acked Flush 0 Done Done Sent false
32836 9193 Acked Read 2 Done Done Sent false
32837 9194 Acked Write 7 Done Done Sent false
32838 9195 Acked Write 9 Done Done Sent false
32839 9196 Acked Write 8 Done Done Sent false
32840 9197 Acked Flush 0 Done Done Sent false
32841 9198 Acked Write 1 Done Done Sent false
32842 9199 Acked Read 7 Done Done Sent false
32843 9200 Acked Read 9 Done Done Sent false
32844 9201 Acked Read 9 Done Done Sent false
32845 9202 Acked Write 2 Done Done Sent false
32846 9203 Acked Write 4 Done Done Sent false
32847 9204 Acked Read 6 Done Done Sent false
32848 9205 Acked Read 7 Done Done Sent false
32849 9206 Acked Read 10 Done Done Sent false
32850 9207 Acked Read 3 Done Done Sent false
32851 9208 Acked Read 2 Done Done Sent false
32852 9209 Acked Read 1 Done Done Sent false
32853 9210 Acked Flush 0 Done Done Sent false
32854 9211 Acked Read 8 Done Done Sent false
32855 9212 Acked Write 7 Done Done Sent false
32856 9213 Acked Flush 0 Done Done Sent false
32857 9214 Acked Write 3 Done Done Sent false
32858 9215 Acked Write 5 Done Done Sent false
32859 9216 Acked Write 10 Done Done Sent false
32860 9217 Acked Flush 0 Done Done Sent false
32861 9218 Acked Write 7 Done Done Sent false
32862 9219 Acked Read 5 Done Done Sent false
32863 9220 Acked Read 3 Done Done Sent false
32864 9221 Acked Read 3 Done Done Sent false
32865 9222 Acked Write 8 Done Done Sent false
32866 9223 Acked Read 9 Done Done Sent false
32867 9224 Acked Read 10 Done Done Sent false
32868 9225 Acked Write 5 Done Done Sent false
32869 9226 Acked Read 8 Done Done Sent false
32870 9227 Acked Write 9 Done Done Sent false
32871 9228 Acked Read 4 Done Done Sent false
32872 9229 Acked Write 8 Done Done Sent false
32873 9230 Acked Read 8 Done Done Sent false
32874 9231 Acked Write 4 Done Done Sent false
32875 9232 Acked Flush 0 Done Done Sent false
32876 9233 Acked Read 1 Done Done Sent false
32877 9234 Acked Write 4 Done Done Sent false
32878 9235 Acked Write 7 Done Done Sent false
32879 9236 Acked Read 1 Done Done Sent false
32880 9237 Acked Read 4 Done Done Sent false
32881 9238 Acked Write 3 Done Done Sent false
32882 9239 Acked Write 1 Done Done Sent false
32883 9240 Acked Write 6 Done Done Sent false
32884 9241 Acked Write 3 Done Done Sent false
32885 9242 Acked Read 5 Done Done Sent false
32886 9243 Acked Read 9 Done Done Sent false
32887 9244 Acked Write 4 Done Done Sent false
32888 9245 Acked Write 10 Done Done Sent false
32889 9246 Acked Read 8 Done Done Sent false
32890 9247 Acked Flush 0 Done Done Sent false
32891 9248 Acked Write 9 Done Done Sent false
32892 9249 Acked Flush 0 Done Done Sent false
32893 9250 Acked Write 4 Done Done Sent false
32894 9251 Acked Read 3 Done Done Sent false
32895 9252 Acked Read 7 Done Done Sent false
32896 9253 Acked Flush 0 Done Done Sent false
32897 9254 Acked Read 1 Done Done Sent false
32898 9255 Acked Read 9 Done Done Sent false
32899 9256 Acked Flush 0 Done Done Sent false
32900 9257 Acked Read 7 Done Done Sent false
32901 9258 Acked Read 5 Done Done Sent false
32902 9259 Acked Read 3 Done Done Sent false
32903 9260 Acked Write 1 Done Done Sent false
32904 9261 Acked Write 2 Done Done Sent false
32905 9262 Acked Read 7 Done Done Sent false
32906 9263 Acked Read 9 Done Done Sent false
32907 9264 Acked Flush 0 Done Done Sent false
32908 9265 Acked Read 1 Done Done Sent false
32909 9266 Acked Read 4 Done Done Sent false
32910 9267 Acked Write 1 Done Done Sent false
32911 9268 Acked Read 7 Done Done Sent false
32912 9269 Acked Read 4 Done Done Sent false
32913 9270 Acked Read 10 Done Done Sent false
32914 9271 Acked Flush 0 Done Done Sent false
32915 9272 Acked Read 7 Done Done Sent false
32916 9273 Acked Write 9 Done Done Sent false
32917 9274 Acked Read 8 Done Done Sent false
32918 9275 Acked Read 5 Done Done Sent false
32919 9276 Acked Write 10 Done Done Sent false
32920 9277 Acked Read 1 Done Done Sent false
32921 9278 Acked Flush 0 Done Done Sent false
32922 9279 Acked Write 8 Done Done Sent false
32923 9280 Acked Read 4 Done Done Sent false
32924 9281 Acked Read 8 Done Done Sent false
32925 9282 Acked Write 2 Done Done Sent false
32926 9283 Acked Read 8 Done Done Sent false
32927 9284 Acked Write 7 Done Done Sent false
32928 9285 Acked Write 4 Done Done Sent false
32929 9286 Acked Write 7 Done Done Sent false
32930 9287 Acked Read 8 Done Done Sent false
32931 9288 Acked Write 7 Done Done Sent false
32932 9289 Acked Read 8 Done Done Sent false
32933 9290 Acked Read 7 Done Done Sent false
32934 9291 Acked Flush 0 Done Done Sent false
32935 9292 Acked Read 2 Done Done Sent false
32936 9293 Acked Read 8 Done Done Sent false
32937 9294 Acked Write 5 Done Done Sent false
32938 9295 Acked Read 2 Done Done Sent false
32939 9296 Acked Read 6 Done Done Sent false
32940 9297 Acked Read 8 Done Done Sent false
32941 9298 Acked Flush 0 Done Done Sent false
32942 9299 Acked Read 2 Done Done Sent false
32943 9300 Acked Write 6 Done Done Sent false
32944 9301 Acked Read 5 Done Done Sent false
32945 9302 Acked Read 8 Done Done Sent false
32946 9303 Acked Read 4 Done Done Sent false
32947 9304 Acked Read 9 Done Done Sent false
32948 9305 Acked Write 4 Done Done Sent false
32949 9306 Acked Read 8 Done Done Sent false
32950 9307 Acked Write 6 Done Done Sent false
32951 9308 Acked Read 7 Done Done Sent false
32952 9309 Acked Read 3 Done Done Sent false
32953 9310 Acked Write 2 Done Done Sent false
32954 9311 Acked Read 3 Done Done Sent false
32955 9312 Acked Read 8 Done Done Sent false
32956 9313 Acked Read 7 Done Done Sent false
32957 9314 Acked Write 2 Done Done Sent false
32958 9315 Acked Read 1 Done Done Sent false
32959 9316 Acked Flush 0 Done Done Sent false
32960 9317 Acked Write 9 Done Done Sent false
32961 9318 Acked Read 6 Done Done Sent false
32962 9319 Acked Write 10 Done Done Sent false
32963 9320 Acked Read 8 Done Done Sent false
32964 9321 Acked Write 4 Done Done Sent false
32965 9322 Acked Write 7 Done Done Sent false
32966 9323 Acked Read 5 Done Done Sent false
32967 9324 Acked Write 6 Done Done Sent false
32968 9325 Acked Write 6 Done Done Sent false
32969 9326 Acked Read 6 Done Done Sent false
32970 9327 Acked Read 10 Done Done Sent false
32971 9328 Acked Write 6 Done Done Sent false
32972 9329 Acked Read 6 Done Done Sent false
32973 9330 Acked Read 1 Done Done Sent false
32974 9331 Acked Read 9 Done Done Sent false
32975 9332 Acked Write 7 Done Done Sent false
32976 9333 Acked Read 10 Done Done Sent false
32977 9334 Acked Read 5 Done Done Sent false
32978 9335 Acked Write 5 Done Done Sent false
32979 9336 Acked Read 7 Done Done Sent false
32980 9337 Acked Write 1 Done Done Sent false
32981 9338 Acked Read 10 Done Done Sent false
32982 9339 Acked Write 9 Done Done Sent false
32983 9340 Acked Write 9 Done Done Sent false
32984 9341 Acked Read 4 Done Done Sent false
32985 9342 Acked Read 9 Done Done Sent false
32986 9343 Acked Read 2 Done Done Sent false
32987 9344 Acked Write 4 Done Done Sent false
32988 9345 Acked Read 8 Done Done Sent false
32989 9346 Acked Write 8 Done Done Sent false
32990 9347 Acked Write 10 Done Done Sent false
32991 9348 Acked Write 5 Done Done Sent false
32992 9349 Acked Read 1 Done Done Sent false
32993 9350 Acked Read 2 Done Done Sent false
32994 9351 Acked Write 8 Done Done Sent false
32995 9352 Acked Read 3 Done Done Sent false
32996 9353 Acked Write 2 Done Done Sent false
32997 9354 Acked Read 4 Done Done Sent false
32998 9355 Acked Write 5 Done Done Sent false
32999 9356 Acked Read 1 Done Done Sent false
33000 9357 Acked Write 1 Done Done Sent false
33001 9358 Acked Read 10 Done Done Sent false
33002 9359 Acked Write 4 Done Done Sent false
33003 9360 Acked Read 1 Done Done Sent false
33004 9361 Acked Read 5 Done Done Sent false
33005 9362 Acked Read 8 Done Done Sent false
33006 9363 Acked Flush 0 Done Done Sent false
33007 9364 Acked Read 6 Done Done Sent false
33008 9365 Acked Read 3 Done Done Sent false
33009 9366 Acked Read 3 Done Done Sent false
33010 9367 Acked Read 10 Done Done Sent false
33011 9368 Acked Read 3 Done Done Sent false
33012 9369 Acked Read 2 Done Done Sent false
33013 9370 Acked Write 10 Done Done Sent false
33014 9371 Acked Flush 0 Done Done Sent false
33015 9372 Acked Read 4 Done Done Sent false
33016 9373 Acked Read 7 Done Done Sent false
33017 9374 Acked Read 7 Done Done Sent false
33018 9375 Acked Write 7 Done Done Sent false
33019 9376 Acked Read 3 Done Done Sent false
33020 9377 Acked Flush 0 Done Done Sent false
33021 9378 Acked Read 2 Done Done Sent false
33022 9379 Acked Write 3 Done Done Sent false
33023 9380 Acked Read 7 Done Done Sent false
33024 9381 Acked Write 7 Done Done Sent false
33025 9382 Acked Read 7 Done Done Sent false
33026 9383 Acked Read 7 Done Done Sent false
33027 9384 Acked Read 7 Done Done Sent false
33028 9385 Acked Write 3 Done Done Sent false
33029 9386 Acked Read 10 Done Done Sent false
33030 9387 Acked Flush 0 Done Done Sent false
33031 9388 Acked Read 10 Done Done Sent false
33032 9389 Acked Write 2 Done Done Sent false
33033 9390 Acked Read 5 Done Done Sent false
33034 9391 Acked Write 10 Done Done Sent false
33035 9392 Acked Read 6 Done Done Sent false
33036 9393 Acked Write 6 Done Done Sent false
33037 9394 Acked Write 1 Done Done Sent false
33038 9395 Acked Read 9 Done Done Sent false
33039 9396 Acked Read 1 Done Done Sent false
33040 9397 Acked Write 8 Done Done Sent false
33041 9398 Acked Read 9 Done Done Sent false
33042 9399 Acked Read 2 Done Done Sent false
33043 9400 Acked Read 7 Done Done Sent false
33044 9401 Acked Read 8 Done Done Sent false
33045 9402 Acked Read 2 Done Done Sent false
33046 9403 Acked Read 3 Done Done Sent false
33047 9404 Acked Write 1 Done Done Sent false
33048 9405 Acked Read 9 Done Done Sent false
33049 9406 Acked Read 1 Done Done Sent false
33050 9407 Acked Read 6 Done Done Sent false
33051 9408 Acked Read 10 Done Done Sent false
33052 9409 Acked Write 3 Done Done Sent false
33053 9410 Acked Read 7 Done Done Sent false
33054 9411 Acked Write 2 Done Done Sent false
33055 9412 Acked Write 4 Done Done Sent false
33056 9413 Acked Read 5 Done Done Sent false
33057 9414 Acked Read 1 Done Done Sent false
33058 9415 Acked Write 4 Done Done Sent false
33059 9416 Acked Read 8 Done Done Sent false
33060 9417 Acked Read 5 Done Done Sent false
33061 9418 Acked Read 9 Done Done Sent false
33062 9419 Acked Read 3 Done Done Sent false
33063 9420 Acked Write 2 Done Done Sent false
33064 9421 Acked Read 9 Done Done Sent false
33065 9422 Acked Write 8 Done Done Sent false
33066 9423 Acked Write 10 Done Done Sent false
33067 9424 Acked Read 8 Done Done Sent false
33068 9425 Acked Flush 0 Done Done Sent false
33069 9426 Acked Read 8 Done Done Sent false
33070 9427 Acked Write 3 Done Done Sent false
33071 9428 Acked Read 8 Done Done Sent false
33072 9429 Acked Read 7 Done Done Sent false
33073 9430 Acked Write 8 Done Done Sent false
33074 9431 Acked Flush 0 Done Done Sent false
33075 9432 Acked Read 6 Done Done Sent false
33076 9433 Acked Write 5 Done Done Sent false
33077 9434 Acked Write 7 Done Done Sent false
33078 9435 Acked Read 6 Done Done Sent false
33079 9436 Acked Flush 0 Done Done Sent false
33080 9437 Acked Write 8 Done Done Sent false
33081 9438 Acked Read 3 Done Done Sent false
33082 9439 Acked Read 2 Done Done Sent false
33083 9440 Acked Read 1 Done Done Sent false
33084 9441 Acked Flush 0 Done Done Sent false
33085 9442 Acked Read 2 Done Done Sent false
33086 9443 Acked Read 7 Done Done Sent false
33087 9444 Acked Write 7 Done Done Sent false
33088 9445 Acked Read 9 Done Done Sent false
33089 9446 Acked Read 3 Done Done Sent false
33090 9447 Acked Write 5 Done Done Sent false
33091 9448 Acked Write 5 Done Done Sent false
33092 9449 Acked Write 3 Done Done Sent false
33093 9450 Acked Write 8 Done Done Sent false
33094 STATES DS:0 DS:1 DS:2 TOTAL
33095 Sent 0 0 301 301
33096 Done 301 301 0 602
33097 Skipped 0 0 0 0
33098 Error 0 0 0 0
33099 Last Flush: Some(JobId(9441)) Some(JobId(9441)) Some(JobId(9149))
33100 Downstairs last five completed: 9149 9148 9147 9146 9145
33101 Upstairs last five completed: 9450 9449 9448 9447 9446
33102 CLIENT: Up:0 ds:301 act:2
331032025-02-28T06:08:19.419ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
331042025-02-28T06:08:20.076ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
33105 [8830] got start action
33106 [8830] I am going to restart
33107 Starting downstairs at port 8830
33108 Make output file at "/tmp/dsc/downstairs-8830.txt"
33109 [8830][2] reports Starting
33110 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1239)
33111 [8830][2] reports Running
331122025-02-28T06:08:21.077ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
331132025-02-28T06:08:22.078ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
33114 ----------------------------------------------------------------
33115 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33116 States: Active Active Offline
33117 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33118 9150 Acked Read 6 Done Done Sent false
33119 9151 Acked Read 2 Done Done Sent false
33120 9152 Acked Read 10 Done Done Sent false
33121 9153 Acked Flush 0 Done Done Sent false
33122 9154 Acked Write 7 Done Done Sent false
33123 9155 Acked Read 4 Done Done Sent false
33124 9156 Acked Read 9 Done Done Sent false
33125 9157 Acked Write 9 Done Done Sent false
33126 9158 Acked Read 10 Done Done Sent false
33127 9159 Acked Read 8 Done Done Sent false
33128 9160 Acked Read 1 Done Done Sent false
33129 9161 Acked Read 10 Done Done Sent false
33130 9162 Acked Write 3 Done Done Sent false
33131 9163 Acked Read 5 Done Done Sent false
33132 9164 Acked Write 3 Done Done Sent false
33133 9165 Acked Write 3 Done Done Sent false
33134 9166 Acked Read 8 Done Done Sent false
33135 9167 Acked Write 8 Done Done Sent false
33136 9168 Acked Write 3 Done Done Sent false
33137 9169 Acked Write 2 Done Done Sent false
33138 9170 Acked Write 2 Done Done Sent false
33139 9171 Acked Write 7 Done Done Sent false
33140 9172 Acked Write 3 Done Done Sent false
33141 9173 Acked Write 2 Done Done Sent false
33142 9174 Acked Write 9 Done Done Sent false
33143 9175 Acked Flush 0 Done Done Sent false
33144 9176 Acked Write 2 Done Done Sent false
33145 9177 Acked Read 1 Done Done Sent false
33146 9178 Acked Write 1 Done Done Sent false
33147 9179 Acked Write 8 Done Done Sent false
33148 9180 Acked Write 6 Done Done Sent false
33149 9181 Acked Write 7 Done Done Sent false
33150 9182 Acked Write 1 Done Done Sent false
33151 9183 Acked Write 5 Done Done Sent false
33152 9184 Acked Read 4 Done Done Sent false
33153 9185 Acked Read 9 Done Done Sent false
33154 9186 Acked Read 9 Done Done Sent false
33155 9187 Acked Flush 0 Done Done Sent false
33156 9188 Acked Read 4 Done Done Sent false
33157 9189 Acked Write 3 Done Done Sent false
33158 9190 Acked Write 1 Done Done Sent false
33159 9191 Acked Read 2 Done Done Sent false
33160 9192 Acked Flush 0 Done Done Sent false
33161 9193 Acked Read 2 Done Done Sent false
33162 9194 Acked Write 7 Done Done Sent false
33163 9195 Acked Write 9 Done Done Sent false
33164 9196 Acked Write 8 Done Done Sent false
33165 9197 Acked Flush 0 Done Done Sent false
33166 9198 Acked Write 1 Done Done Sent false
33167 9199 Acked Read 7 Done Done Sent false
33168 9200 Acked Read 9 Done Done Sent false
33169 9201 Acked Read 9 Done Done Sent false
33170 9202 Acked Write 2 Done Done Sent false
33171 9203 Acked Write 4 Done Done Sent false
33172 9204 Acked Read 6 Done Done Sent false
33173 9205 Acked Read 7 Done Done Sent false
33174 9206 Acked Read 10 Done Done Sent false
33175 9207 Acked Read 3 Done Done Sent false
33176 9208 Acked Read 2 Done Done Sent false
33177 9209 Acked Read 1 Done Done Sent false
33178 9210 Acked Flush 0 Done Done Sent false
33179 9211 Acked Read 8 Done Done Sent false
33180 9212 Acked Write 7 Done Done Sent false
33181 9213 Acked Flush 0 Done Done Sent false
33182 9214 Acked Write 3 Done Done Sent false
33183 9215 Acked Write 5 Done Done Sent false
33184 9216 Acked Write 10 Done Done Sent false
33185 9217 Acked Flush 0 Done Done Sent false
33186 9218 Acked Write 7 Done Done Sent false
33187 9219 Acked Read 5 Done Done Sent false
33188 9220 Acked Read 3 Done Done Sent false
33189 9221 Acked Read 3 Done Done Sent false
33190 9222 Acked Write 8 Done Done Sent false
33191 9223 Acked Read 9 Done Done Sent false
33192 9224 Acked Read 10 Done Done Sent false
33193 9225 Acked Write 5 Done Done Sent false
33194 9226 Acked Read 8 Done Done Sent false
33195 9227 Acked Write 9 Done Done Sent false
33196 9228 Acked Read 4 Done Done Sent false
33197 9229 Acked Write 8 Done Done Sent false
33198 9230 Acked Read 8 Done Done Sent false
33199 9231 Acked Write 4 Done Done Sent false
33200 9232 Acked Flush 0 Done Done Sent false
33201 9233 Acked Read 1 Done Done Sent false
33202 9234 Acked Write 4 Done Done Sent false
33203 9235 Acked Write 7 Done Done Sent false
33204 9236 Acked Read 1 Done Done Sent false
33205 9237 Acked Read 4 Done Done Sent false
33206 9238 Acked Write 3 Done Done Sent false
33207 9239 Acked Write 1 Done Done Sent false
33208 9240 Acked Write 6 Done Done Sent false
33209 9241 Acked Write 3 Done Done Sent false
33210 9242 Acked Read 5 Done Done Sent false
33211 9243 Acked Read 9 Done Done Sent false
33212 9244 Acked Write 4 Done Done Sent false
33213 9245 Acked Write 10 Done Done Sent false
33214 9246 Acked Read 8 Done Done Sent false
33215 9247 Acked Flush 0 Done Done Sent false
33216 9248 Acked Write 9 Done Done Sent false
33217 9249 Acked Flush 0 Done Done Sent false
33218 9250 Acked Write 4 Done Done Sent false
33219 9251 Acked Read 3 Done Done Sent false
33220 9252 Acked Read 7 Done Done Sent false
33221 9253 Acked Flush 0 Done Done Sent false
33222 9254 Acked Read 1 Done Done Sent false
33223 9255 Acked Read 9 Done Done Sent false
33224 9256 Acked Flush 0 Done Done Sent false
33225 9257 Acked Read 7 Done Done Sent false
33226 9258 Acked Read 5 Done Done Sent false
33227 9259 Acked Read 3 Done Done Sent false
33228 9260 Acked Write 1 Done Done Sent false
33229 9261 Acked Write 2 Done Done Sent false
33230 9262 Acked Read 7 Done Done Sent false
33231 9263 Acked Read 9 Done Done Sent false
33232 9264 Acked Flush 0 Done Done Sent false
33233 9265 Acked Read 1 Done Done Sent false
33234 9266 Acked Read 4 Done Done Sent false
33235 9267 Acked Write 1 Done Done Sent false
33236 9268 Acked Read 7 Done Done Sent false
33237 9269 Acked Read 4 Done Done Sent false
33238 9270 Acked Read 10 Done Done Sent false
33239 9271 Acked Flush 0 Done Done Sent false
33240 9272 Acked Read 7 Done Done Sent false
33241 9273 Acked Write 9 Done Done Sent false
33242 9274 Acked Read 8 Done Done Sent false
33243 9275 Acked Read 5 Done Done Sent false
33244 9276 Acked Write 10 Done Done Sent false
33245 9277 Acked Read 1 Done Done Sent false
33246 9278 Acked Flush 0 Done Done Sent false
33247 9279 Acked Write 8 Done Done Sent false
33248 9280 Acked Read 4 Done Done Sent false
33249 9281 Acked Read 8 Done Done Sent false
33250 9282 Acked Write 2 Done Done Sent false
33251 9283 Acked Read 8 Done Done Sent false
33252 9284 Acked Write 7 Done Done Sent false
33253 9285 Acked Write 4 Done Done Sent false
33254 9286 Acked Write 7 Done Done Sent false
33255 9287 Acked Read 8 Done Done Sent false
33256 9288 Acked Write 7 Done Done Sent false
33257 9289 Acked Read 8 Done Done Sent false
33258 9290 Acked Read 7 Done Done Sent false
33259 9291 Acked Flush 0 Done Done Sent false
33260 9292 Acked Read 2 Done Done Sent false
33261 9293 Acked Read 8 Done Done Sent false
33262 9294 Acked Write 5 Done Done Sent false
33263 9295 Acked Read 2 Done Done Sent false
33264 9296 Acked Read 6 Done Done Sent false
33265 9297 Acked Read 8 Done Done Sent false
33266 9298 Acked Flush 0 Done Done Sent false
33267 9299 Acked Read 2 Done Done Sent false
33268 9300 Acked Write 6 Done Done Sent false
33269 9301 Acked Read 5 Done Done Sent false
33270 9302 Acked Read 8 Done Done Sent false
33271 9303 Acked Read 4 Done Done Sent false
33272 9304 Acked Read 9 Done Done Sent false
33273 9305 Acked Write 4 Done Done Sent false
33274 9306 Acked Read 8 Done Done Sent false
33275 9307 Acked Write 6 Done Done Sent false
33276 9308 Acked Read 7 Done Done Sent false
33277 9309 Acked Read 3 Done Done Sent false
33278 9310 Acked Write 2 Done Done Sent false
33279 9311 Acked Read 3 Done Done Sent false
33280 9312 Acked Read 8 Done Done Sent false
33281 9313 Acked Read 7 Done Done Sent false
33282 9314 Acked Write 2 Done Done Sent false
33283 9315 Acked Read 1 Done Done Sent false
33284 9316 Acked Flush 0 Done Done Sent false
33285 9317 Acked Write 9 Done Done Sent false
33286 9318 Acked Read 6 Done Done Sent false
33287 9319 Acked Write 10 Done Done Sent false
33288 9320 Acked Read 8 Done Done Sent false
33289 9321 Acked Write 4 Done Done Sent false
33290 9322 Acked Write 7 Done Done Sent false
33291 9323 Acked Read 5 Done Done Sent false
33292 9324 Acked Write 6 Done Done Sent false
33293 9325 Acked Write 6 Done Done Sent false
33294 9326 Acked Read 6 Done Done Sent false
33295 9327 Acked Read 10 Done Done Sent false
33296 9328 Acked Write 6 Done Done Sent false
33297 9329 Acked Read 6 Done Done Sent false
33298 9330 Acked Read 1 Done Done Sent false
33299 9331 Acked Read 9 Done Done Sent false
33300 9332 Acked Write 7 Done Done Sent false
33301 9333 Acked Read 10 Done Done Sent false
33302 9334 Acked Read 5 Done Done Sent false
33303 9335 Acked Write 5 Done Done Sent false
33304 9336 Acked Read 7 Done Done Sent false
33305 9337 Acked Write 1 Done Done Sent false
33306 9338 Acked Read 10 Done Done Sent false
33307 9339 Acked Write 9 Done Done Sent false
33308 9340 Acked Write 9 Done Done Sent false
33309 9341 Acked Read 4 Done Done Sent false
33310 9342 Acked Read 9 Done Done Sent false
33311 9343 Acked Read 2 Done Done Sent false
33312 9344 Acked Write 4 Done Done Sent false
33313 9345 Acked Read 8 Done Done Sent false
33314 9346 Acked Write 8 Done Done Sent false
33315 9347 Acked Write 10 Done Done Sent false
33316 9348 Acked Write 5 Done Done Sent false
33317 9349 Acked Read 1 Done Done Sent false
33318 9350 Acked Read 2 Done Done Sent false
33319 9351 Acked Write 8 Done Done Sent false
33320 9352 Acked Read 3 Done Done Sent false
33321 9353 Acked Write 2 Done Done Sent false
33322 9354 Acked Read 4 Done Done Sent false
33323 9355 Acked Write 5 Done Done Sent false
33324 9356 Acked Read 1 Done Done Sent false
33325 9357 Acked Write 1 Done Done Sent false
33326 9358 Acked Read 10 Done Done Sent false
33327 9359 Acked Write 4 Done Done Sent false
33328 9360 Acked Read 1 Done Done Sent false
33329 9361 Acked Read 5 Done Done Sent false
33330 9362 Acked Read 8 Done Done Sent false
33331 9363 Acked Flush 0 Done Done Sent false
33332 9364 Acked Read 6 Done Done Sent false
33333 9365 Acked Read 3 Done Done Sent false
33334 9366 Acked Read 3 Done Done Sent false
33335 9367 Acked Read 10 Done Done Sent false
33336 9368 Acked Read 3 Done Done Sent false
33337 9369 Acked Read 2 Done Done Sent false
33338 9370 Acked Write 10 Done Done Sent false
33339 9371 Acked Flush 0 Done Done Sent false
33340 9372 Acked Read 4 Done Done Sent false
33341 9373 Acked Read 7 Done Done Sent false
33342 9374 Acked Read 7 Done Done Sent false
33343 9375 Acked Write 7 Done Done Sent false
33344 9376 Acked Read 3 Done Done Sent false
33345 9377 Acked Flush 0 Done Done Sent false
33346 9378 Acked Read 2 Done Done Sent false
33347 9379 Acked Write 3 Done Done Sent false
33348 9380 Acked Read 7 Done Done Sent false
33349 9381 Acked Write 7 Done Done Sent false
33350 9382 Acked Read 7 Done Done Sent false
33351 9383 Acked Read 7 Done Done Sent false
33352 9384 Acked Read 7 Done Done Sent false
33353 9385 Acked Write 3 Done Done Sent false
33354 9386 Acked Read 10 Done Done Sent false
33355 9387 Acked Flush 0 Done Done Sent false
33356 9388 Acked Read 10 Done Done Sent false
33357 9389 Acked Write 2 Done Done Sent false
33358 9390 Acked Read 5 Done Done Sent false
33359 9391 Acked Write 10 Done Done Sent false
33360 9392 Acked Read 6 Done Done Sent false
33361 9393 Acked Write 6 Done Done Sent false
33362 9394 Acked Write 1 Done Done Sent false
33363 9395 Acked Read 9 Done Done Sent false
33364 9396 Acked Read 1 Done Done Sent false
33365 9397 Acked Write 8 Done Done Sent false
33366 9398 Acked Read 9 Done Done Sent false
33367 9399 Acked Read 2 Done Done Sent false
33368 9400 Acked Read 7 Done Done Sent false
33369 9401 Acked Read 8 Done Done Sent false
33370 9402 Acked Read 2 Done Done Sent false
33371 9403 Acked Read 3 Done Done Sent false
33372 9404 Acked Write 1 Done Done Sent false
33373 9405 Acked Read 9 Done Done Sent false
33374 9406 Acked Read 1 Done Done Sent false
33375 9407 Acked Read 6 Done Done Sent false
33376 9408 Acked Read 10 Done Done Sent false
33377 9409 Acked Write 3 Done Done Sent false
33378 9410 Acked Read 7 Done Done Sent false
33379 9411 Acked Write 2 Done Done Sent false
33380 9412 Acked Write 4 Done Done Sent false
33381 9413 Acked Read 5 Done Done Sent false
33382 9414 Acked Read 1 Done Done Sent false
33383 9415 Acked Write 4 Done Done Sent false
33384 9416 Acked Read 8 Done Done Sent false
33385 9417 Acked Read 5 Done Done Sent false
33386 9418 Acked Read 9 Done Done Sent false
33387 9419 Acked Read 3 Done Done Sent false
33388 9420 Acked Write 2 Done Done Sent false
33389 9421 Acked Read 9 Done Done Sent false
33390 9422 Acked Write 8 Done Done Sent false
33391 9423 Acked Write 10 Done Done Sent false
33392 9424 Acked Read 8 Done Done Sent false
33393 9425 Acked Flush 0 Done Done Sent false
33394 9426 Acked Read 8 Done Done Sent false
33395 9427 Acked Write 3 Done Done Sent false
33396 9428 Acked Read 8 Done Done Sent false
33397 9429 Acked Read 7 Done Done Sent false
33398 9430 Acked Write 8 Done Done Sent false
33399 9431 Acked Flush 0 Done Done Sent false
33400 9432 Acked Read 6 Done Done Sent false
33401 9433 Acked Write 5 Done Done Sent false
33402 9434 Acked Write 7 Done Done Sent false
33403 9435 Acked Read 6 Done Done Sent false
33404 9436 Acked Flush 0 Done Done Sent false
33405 9437 Acked Write 8 Done Done Sent false
33406 9438 Acked Read 3 Done Done Sent false
33407 9439 Acked Read 2 Done Done Sent false
33408 9440 Acked Read 1 Done Done Sent false
33409 9441 Acked Flush 0 Done Done Sent false
33410 9442 Acked Read 2 Done Done Sent false
33411 9443 Acked Read 7 Done Done Sent false
33412 9444 Acked Write 7 Done Done Sent false
33413 9445 Acked Read 9 Done Done Sent false
33414 9446 Acked Read 3 Done Done Sent false
33415 9447 Acked Write 5 Done Done Sent false
33416 9448 Acked Write 5 Done Done Sent false
33417 9449 Acked Write 3 Done Done Sent false
33418 9450 Acked Write 8 Done Done Sent false
33419 STATES DS:0 DS:1 DS:2 TOTAL
33420 Sent 0 0 301 301
33421 Done 301 301 0 602
33422 Skipped 0 0 0 0
33423 Error 0 0 0 0
33424 Last Flush: Some(JobId(9441)) Some(JobId(9441)) Some(JobId(9149))
33425 Downstairs last five completed: 9149 9148 9147 9146 9145
33426 Upstairs last five completed: 9450 9449 9448 9447 9446
33427 CLIENT: Up:0 ds:301 act:2
334282025-02-28T06:08:23.420ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
334292025-02-28T06:08:24.081ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
334302025-02-28T06:08:24.667ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
334312025-02-28T06:08:24.667ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36846) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
334322025-02-28T06:08:24.668ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
334332025-02-28T06:08:24.669ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
334342025-02-28T06:08:24.669ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
334352025-02-28T06:08:24.669ZINFOcrucible: send last flush ID to this DS: Some(JobId(9149)) = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
334362025-02-28T06:08:24.669ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
334372025-02-28T06:08:24.669ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9149)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
334382025-02-28T06:08:24.669ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9149)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
33439 ----------------------------------------------------------------
33440 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33441 Downstairs last five completed: 9451 9450 9449 9448 9447
33442 Upstairs last five completed: 9451 9450 9449 9448 9447
33443 CLIENT: Up:0 ds:0 act:3
33444 Replay: All jobs finished
33445 Main task has work to do, go find it
33446 got dsc Stop(2)
33447 stop 2
33448 Feb 28 06:08:26.922[8830] Got stop action so:false kr:false
33449 INFO request completed, latency_us: 422, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 29d28d84-867d-483e-b263-789c7811d6cc, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
334502025-02-28T06:08:26.923ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
334512025-02-28T06:08:26.923ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
33452 Feb 28 06:08:26.923 INFO request completed, latency_us: 126{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-28T06:08:26.92324131Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
33453 , response_code: 200, uri: /state/cid/2, method: GET, req_id: 6e35b112-8f94-4494-a048-d235735ff40d, remote_addr: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-28T06:08:26.923309244Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"969fa3b2-3050-4644-bbdf-c5076af6b813"}
33454 127.0.0.1:63908, local_addr: 127.0.0.1:9998
33455 [8830] Exited with: ExitStatus(unix_wait_status(9))
33456 [8830][2] reports Exit
33457 Feb 28 06:08:30.925 INFO request completed, latency_us: 338, response_code: 200, uri: /state/cid/2, method: GET, req_id: 355d57f7-e0ba-4492-a819-da899d5d76cf, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
33458 001/300 Write block 336 len 40960 data: 6 7 8 5 5 7 6 6 6 9
33459 002/300 Write block 2873 len 40960 data: 7 7 4 5 5 6 5 4 3 4
33460 003/300 Read block 3945 len 16384
33461 004/300 Read block 629 len 12288
33462 005/300 Read block 1236 len 28672
33463 006/300 Read block 880 len 16384
33464 007/300 Read block 3563 len 8192
33465 008/300 Read block 4438 len 8192
33466 009/300 Write block 1806 len 40960 data: 4 3 3 5 6 6 5 5 5 4
33467 010/300 Write block 2596 len 20480 data: 4 4 4 5 6
33468 011/300 Read block 3356 len 40960
33469 012/300 Read block 2383 len 40960
33470 013/300 Read block 4339 len 16384
33471 014/300 Flush
33472 015/300 Write block 1717 len 32768 data: 6 5 4 6 7 9 9 9
33473 016/300 Read block 851 len 12288
33474 017/300 Write block 3171 len 16384 data: 4 4 3 3
33475 018/300 Write block 4391 len 4096 data: 9
33476 019/300 Read block 1329 len 28672
33477 020/300 Write block 1030 len 4096 data: 3
33478 021/300 Read block 3337 len 40960
33479 022/300 Write block 1738 len 24576 data: 5 6 6 8 5 6
33480 023/300 Write block 1926 len 16384 data: 5 6 5 6
33481 024/300 Read block 3048 len 36864
33482 025/300 Read block 666 len 28672
33483 026/300 Read block 3802 len 24576
33484 027/300 Flush
33485 028/300 Read block 4645 len 24576
33486 029/300 Read block 2952 len 28672
33487 030/300 Write block 1255 len 40960 data: 9 8 7 5 3 4 8 8 7 9
33488 031/300 Read block 4199 len 20480
33489 032/300 Flush
33490 033/300 Write block 508 len 28672 data: 6 7 4 4 3 3 3
33491 034/300 Read block 4576 len 24576
33492 035/300 Read block 254 len 36864
33493 036/300 Read block 319 len 24576
33494 037/300 Flush
33495 038/300 Read block 2585 len 16384
33496 039/300 Write block 4330 len 28672 data: 5 3 4 6 5 5 8
33497 040/300 Flush
33498 041/300 Read block 607 len 16384
33499 042/300 Read block 3531 len 16384
33500 043/300 Flush
33501 044/300 Read block 3739 len 32768
33502 045/300 Write block 4321 len 16384 data: 10 9 10 9
33503 046/300 Write block 2752 len 4096 data: 6
33504 047/300 Read block 692 len 16384
33505 048/300 Write block 1310 len 32768 data: 8 9 8 8 4 4 5 4
33506 049/300 Write block 2872 len 24576 data: 9 8 8 5 6 6
33507 050/300 Write block 4814 len 24576 data: 3 4 4 4 5 5
33508 051/300 Read block 298 len 20480
33509 052/300 Flush
33510 053/300 Read block 342 len 28672
33511 054/300 Write block 1332 len 8192 data: 6 3
33512 055/300 Read block 3663 len 36864
33513 056/300 Read block 4929 len 12288
33514 057/300 Read block 3970 len 40960
33515 058/300 Read block 1096 len 32768
33516 059/300 Read block 3852 len 24576
33517 060/300 Write block 4436 len 4096 data: 6
33518 061/300 Read block 3980 len 8192
33519 062/300 Read block 476 len 40960
33520 063/300 Read block 3587 len 28672
33521 064/300 Write block 2881 len 12288 data: 4 5 5
33522 065/300 Flush
33523 066/300 Read block 211 len 28672
33524 067/300 Read block 481 len 20480
33525 068/300 Write block 3580 len 40960 data: 3 3 3 3 2 4 4 4 5 6
33526 069/300 Read block 1489 len 36864
33527 070/300 Read block 207 len 20480
33528 071/300 Read block 988 len 28672
33529 072/300 Write block 438 len 36864 data: 9 9 10 10 11 11 12 11 8
33530 073/300 Flush
33531 074/300 Flush
33532 075/300 Flush
33533 076/300 Read block 213 len 16384
33534 077/300 Write block 3182 len 20480 data: 5 5 6 6 5
33535 078/300 Read block 4271 len 36864
33536 079/300 Read block 773 len 16384
33537 080/300 Read block 2885 len 28672
33538 081/300 Write block 3165 len 16384 data: 5 5 5 5
33539 082/300 Read block 4928 len 32768
33540 083/300 Read block 2132 len 20480
33541 084/300 Write block 2550 len 24576 data: 5 5 5 5 5 5
33542 085/300 Read block 3838 len 20480
33543 086/300 Write block 110 len 28672 data: 5 6 6 5 4 5 3
33544 087/300 Write block 2977 len 8192 data: 5 5
33545 088/300 Read block 45 len 8192
33546 089/300 Write block 206 len 12288 data: 7 7 9
33547 090/300 Read block 2744 len 28672
33548 091/300 Read block 2095 len 12288
33549 092/300 Read block 1180 len 16384
33550 093/300 Write block 1098 len 32768 data: 4 4 3 3 3 3 3 3
33551 094/300 Write block 979 len 20480 data: 7 8 7 6 7
33552 095/300 Write block 1716 len 8192 data: 6 7
33553 096/300 Write block 4333 len 16384 data: 7 6 6 9
33554 097/300 Read block 415 len 8192
33555 098/300 Write block 2682 len 20480 data: 3 4 7 9 8
33556 099/300 Read block 2824 len 40960
33557 100/300 Write block 1474 len 36864 data: 7 5 4 6 6 6 5 6 6
33558 101/300 Read block 1312 len 40960
33559 102/300 Read block 1032 len 32768
33560 103/300 Read block 1575 len 24576
33561 104/300 Read block 624 len 4096
33562 105/300 Write block 487 len 32768 data: 7 6 5 4 4 4 5 5
33563 106/300 Flush
33564 107/300 Read block 1345 len 12288
33565 108/300 Read block 749 len 8192
33566 109/300 Write block 1828 len 8192 data: 6 7
33567 110/300 Write block 3631 len 40960 data: 4 4 6 6 5 5 4 3 3 4
33568 111/300 Write block 3318 len 28672 data: 3 4 6 7 8 8 8
33569 112/300 Read block 2793 len 40960
33570 113/300 Write block 3945 len 36864 data: 5 4 4 4 3 3 3 3 3
33571 114/300 Write block 2658 len 36864 data: 7 6 5 3 5 7 8 8 8
33572 115/300 Write block 2314 len 36864 data: 5 4 4 5 7 7 6 6 6
33573 116/300 Write block 1124 len 16384 data: 6 8 8 8
33574 117/300 Read block 3536 len 36864
33575 118/300 Write block 1130 len 32768 data: 7 8 7 6 8 8 9 9
33576 119/300 Read block 2944 len 24576
33577 120/300 Read block 3482 len 24576
33578 121/300 Flush
33579 122/300 Read block 961 len 20480
33580 123/300 Read block 732 len 16384
33581 124/300 Read block 3054 len 28672
33582 125/300 Read block 3355 len 16384
33583 126/300 Read block 498 len 12288
33584 127/300 Write block 3682 len 28672 data: 8 7 7 6 6 5 6
33585 128/300 Write block 2413 len 12288 data: 2 2 2
33586 129/300 Read block 3286 len 16384
33587 130/300 Read block 365 len 4096
33588 131/300 Read block 3399 len 32768
33589 132/300 Read block 1596 len 28672
33590 133/300 Read block 269 len 28672
33591 134/300 Write block 4801 len 12288 data: 4 4 3
33592 135/300 Read block 3183 len 4096
33593 136/300 Write block 4531 len 32768 data: 6 5 5 5 6 6 4 4
33594 137/300 Write block 2181 len 20480 data: 6 6 6 7 8
33595 138/300 Read block 4950 len 40960
33596 139/300 Write block 4395 len 28672 data: 7 6 5 4 4 4 4
33597 140/300 Write block 4929 len 12288 data: 3 4 4
33598 141/300 Write block 4033 len 40960 data: 4 4 4 6 7 9 8 7 7 5
33599 142/300 Flush
33600 143/300 Write block 458 len 24576 data: 4 6 6 6 5 3
33601 144/300 Read block 1294 len 24576
33602 145/300 Read block 3377 len 36864
33603 146/300 Read block 4603 len 8192
33604 147/300 Write block 1835 len 40960 data: 6 8 6 7 7 7 5 4 6 6
33605 148/300 Read block 1985 len 24576
33606 149/300 Read block 391 len 32768
33607 150/300 Flush
33608 151/300 Read block 1462 len 12288
33609 152/300 Read block 135 len 8192
33610 153/300 Read block 1697 len 4096
33611 154/300 Read block 685 len 24576
33612 155/300 Write block 4417 len 40960 data: 3 3 5 4 4 3 3 2 2 4
33613 156/300 Read block 2316 len 40960
33614 157/300 Flush
33615 158/300 Read block 4270 len 8192
33616 159/300 Read block 220 len 12288
33617 160/300 Read block 4698 len 8192
33618 161/300 Write block 4692 len 40960 data: 6 5 5 6 7 7 6 8 6 6
33619 162/300 Flush
33620 163/300 Write block 4226 len 24576 data: 7 9 12 12 10 10
33621 164/300 Write block 4971 len 32768 data: 9 8 7 6 5 5 5 7
33622 165/300 Write block 4028 len 4096 data: 3
33623 166/300 Write block 70 len 24576 data: 4 5 5 8 7 10
33624 167/300 Read block 272 len 4096
33625 168/300 Read block 4903 len 4096
33626 169/300 Read block 1073 len 28672
33627 170/300 Read block 4760 len 40960
33628 171/300 Write block 1672 len 20480 data: 5 3 4 4 7
33629 172/300 Read block 4385 len 16384
33630 173/300 Write block 4143 len 32768 data: 7 6 5 5 5 5 5 5
33631 174/300 Write block 3903 len 24576 data: 8 8 7 6 6 5
33632 175/300 Read block 3812 len 12288
33633 176/300 Read block 3149 len 32768
33634 177/300 Write block 2081 len 20480 data: 5 6 8 6 8
33635 178/300 Write block 4721 len 28672 data: 2 2 2 4 6 6 6
33636 179/300 Write block 1959 len 12288 data: 4 3 3
33637 180/300 Read block 1920 len 8192
33638 181/300 Write block 1931 len 32768 data: 6 7 6 7 5 5 4 3
33639 182/300 Read block 254 len 24576
33640 183/300 Write block 165 len 40960 data: 5 5 3 2 2 3 4 6 7 8
33641 184/300 Read block 4569 len 4096
33642 185/300 Read block 3721 len 36864
33643 186/300 Read block 3801 len 4096
33644 187/300 Read block 3012 len 32768
33645 188/300 Write block 4221 len 32768 data: 2 2 3 4 6 8 10 13
33646 189/300 Write block 4810 len 8192 data: 6 5
33647 190/300 Write block 3975 len 16384 data: 6 5 5 5
33648 191/300 Read block 2017 len 36864
33649 192/300 Write block 2481 len 24576 data: 5 4 6 7 7 6
33650 193/300 Read block 823 len 20480
33651 194/300 Write block 2578 len 16384 data: 4 6 8 7
33652 195/300 Write block 4689 len 36864 data: 6 6 6 7 6 6 7 8 8
33653 196/300 Flush
33654 197/300 Read block 3246 len 12288
33655 198/300 Read block 2078 len 4096
33656 199/300 Read block 2032 len 40960
33657 200/300 Write block 3863 len 20480 data: 7 9 9 10 11
33658 201/300 Read block 1068 len 40960
33659 202/300 Read block 1204 len 20480
33660 203/300 Read block 3580 len 8192
33661 204/300 Read block 1030 len 8192
33662 205/300 Read block 1914 len 32768
33663 206/300 Read block 2860 len 40960
33664 207/300 Flush
33665 208/300 Read block 3627 len 36864
33666 209/300 Read block 683 len 36864
33667 210/300 Read block 2519 len 40960
33668 211/300 Write block 1589 len 4096 data: 5
33669 212/300 Write block 53 len 40960 data: 6 6 6 6 6 7 6 6 6 5
33670 213/300 Read block 3507 len 12288
33671 214/300 Read block 2643 len 4096
33672 215/300 Read block 3492 len 20480
33673 216/300 Read block 1893 len 4096
33674 217/300 Flush
33675 218/300 Write block 4968 len 28672 data: 10 9 8 10 9 8 7
33676 219/300 Read block 2392 len 36864
33677 220/300 Flush
33678 221/300 Write block 2586 len 32768 data: 6 5 4 3 3 5 4 4
33679 222/300 Read block 3843 len 16384
33680 223/300 Flush
33681 224/300 Write block 1948 len 8192 data: 6 4
33682 225/300 Write block 4318 len 8192 data: 6 5
33683 226/300 Flush
33684 227/300 Write block 736 len 16384 data: 5 4 4 5
33685 228/300 Write block 3902 len 12288 data: 5 9 9
33686 229/300 Write block 2964 len 8192 data: 6 6
33687 230/300 Write block 976 len 36864 data: 6 7 8 8 9 8 7 8 5
33688 231/300 Write block 3963 len 32768 data: 3 3 3 3 3 3 4 5
33689 232/300 Write block 340 len 32768 data: 6 8 7 7 7 10 8 5
33690 233/300 Write block 4276 len 16384 data: 6 7 7 6
33691 234/300 Write block 2126 len 20480 data: 7 4 6 5 6
33692 235/300 Write block 3246 len 24576 data: 5 5 4 8 8 7
33693 236/300 Write block 2828 len 4096 data: 5
33694 237/300 Read block 200 len 32768
33695 238/300 Write block 1853 len 36864 data: 5 5 4 4 4 3 3 3 2
33696 239/300 Flush
33697 240/300 Read block 1070 len 12288
33698 241/300 Read block 1605 len 24576
33699 242/300 Write block 1488 len 24576 data: 9 10 11 9 9 8
33700 243/300 Read block 2852 len 20480
33701 244/300 Read block 513 len 4096
33702 245/300 Read block 3067 len 8192
33703 246/300 Read block 3893 len 36864
33704 247/300 Read block 1405 len 8192
33705 248/300 Write block 2721 len 40960 data: 6 6 5 4 4 4 4 4 6 9
33706 249/300 Write block 953 len 8192 data: 4 4
33707 250/300 Read block 546 len 16384
33708 251/300 Write block 2427 len 40960 data: 6 5 4 5 6 6 5 5 6 6
33709 252/300 Write block 3912 len 24576 data: 5 5 6 6 6 5
33710 253/300 Read block 1764 len 20480
33711 254/300 Read block 2979 len 40960
33712 255/300 Read block 1611 len 16384
33713 256/300 Read block 458 len 20480
33714 257/300 Write block 1688 len 16384 data: 7 5 5 6
33715 258/300 Read block 3620 len 32768
33716 259/300 Write block 2617 len 28672 data: 5 5 5 4 3 4 4
33717 260/300 Write block 3318 len 36864 data: 4 5 7 8 9 9 9 8 6
33718 261/300 Write block 3617 len 20480 data: 7 6 7 5 5
33719 262/300 Read block 1179 len 12288
33720 263/300 Write block 1227 len 40960 data: 10 9 8 8 5 3 3 2 2 2
33721 264/300 Write block 374 len 36864 data: 6 5 5 6 8 7 8 7 8
33722 265/300 Read block 3045 len 20480
33723 266/300 Flush
33724 267/300 Write block 1151 len 36864 data: 11 11 10 9 10 9 9 7 7
33725 268/300 Read block 1950 len 28672
33726 269/300 Flush
33727 270/300 Write block 3457 len 40960 data: 6 6 6 3 5 5 5 5 7 6
33728 271/300 Write block 4998 len 8192 data: 5 3
33729 272/300 Write block 674 len 16384 data: 7 7 8 5
33730 273/300 Write block 1703 len 12288 data: 8 7 5
33731 274/300 Read block 3112 len 16384
33732 275/300 Write block 1744 len 32768 data: 5 5 5 8 7 7 7 4
33733 276/300 Write block 3666 len 36864 data: 4 5 5 5 5 4 4 4 3
33734 277/300 Write block 3269 len 12288 data: 6 5 6
33735 278/300 Read block 4298 len 8192
33736 279/300 Write block 4273 len 32768 data: 7 8 7 7 8 8 7 7
33737 280/300 Read block 2541 len 24576
33738 281/300 Write block 103 len 8192 data: 5 5
33739 282/300 Write block 2252 len 16384 data: 3 4 5 4
33740 283/300 Write block 4252 len 8192 data: 4 4
33741 284/300 Write block 3030 len 16384 data: 7 7 8 7
33742 285/300 Read block 1022 len 28672
33743 286/300 Read block 2338 len 40960
33744 287/300 Read block 3145 len 36864
33745 288/300 Flush
33746 289/300 Flush
33747 290/300 Read block 3630 len 8192
33748 291/300 Read block 1668 len 24576
33749 292/300 Read block 4406 len 12288
33750 293/300 Read block 67 len 24576
33751 294/300 Read block 2086 len 16384
33752 295/300 Write block 1804 len 24576 data: 4 4 5 4 4 6
33753 296/300 Read block 4042 len 12288
33754 297/300 Read block 2531 len 20480
33755 298/300 Read block 4250 len 24576
33756 299/300 Read block 1348 len 20480
33757 300/300 Read block 257 len 8192
33758 301/300 Read block 626 len 16384
33759 Main task has work to do, go find it
33760 got dsc Start(2)
33761 start 2
33762 Feb 28 06:08:31.173 INFO request completed, latency_us: 186, response_code: 204, uri: /start/cid/2, method: POST, req_id: bb8f102d-9991-4c6d-a094-f006a7096622, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
33763 Replay: started 2, returned:Ok(())
33764 ----------------------------------------------------------------
33765 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33766 States: Active Active Offline
33767 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33768 9452 Acked Write 10 Done Done Sent false
33769 9453 Acked Write 10 Done Done Sent false
33770 9454 Acked Read 4 Done Done Sent false
33771 9455 Acked Read 3 Done Done Sent false
33772 9456 Acked Read 7 Done Done Sent false
33773 9457 Acked Read 4 Done Done Sent false
33774 9458 Acked Read 2 Done Done Sent false
33775 9459 Acked Read 2 Done Done Sent false
33776 9460 Acked Write 10 Done Done Sent false
33777 9461 Acked Write 5 Done Done Sent false
33778 9462 Acked Read 10 Done Done Sent false
33779 9463 Acked Read 10 Done Done Sent false
33780 9464 Acked Read 4 Done Done Sent false
33781 9465 Acked Flush 0 Done Done Sent false
33782 9466 Acked Write 8 Done Done Sent false
33783 9467 Acked Read 3 Done Done Sent false
33784 9468 Acked Write 4 Done Done Sent false
33785 9469 Acked Write 1 Done Done Sent false
33786 9470 Acked Read 7 Done Done Sent false
33787 9471 Acked Write 1 Done Done Sent false
33788 9472 Acked Read 10 Done Done Sent false
33789 9473 Acked Write 6 Done Done Sent false
33790 9474 Acked Write 4 Done Done Sent false
33791 9475 Acked Read 9 Done Done Sent false
33792 9476 Acked Read 7 Done Done Sent false
33793 9477 Acked Read 6 Done Done Sent false
33794 9478 Acked Flush 0 Done Done Sent false
33795 9479 Acked Read 6 Done Done Sent false
33796 9480 Acked Read 7 Done Done Sent false
33797 9481 Acked Write 10 Done Done Sent false
33798 9482 Acked Read 5 Done Done Sent false
33799 9483 Acked Flush 0 Done Done Sent false
33800 9484 Acked Write 7 Done Done Sent false
33801 9485 Acked Read 6 Done Done Sent false
33802 9486 Acked Read 9 Done Done Sent false
33803 9487 Acked Read 6 Done Done Sent false
33804 9488 Acked Flush 0 Done Done Sent false
33805 9489 Acked Read 4 Done Done Sent false
33806 9490 Acked Write 7 Done Done Sent false
33807 9491 Acked Flush 0 Done Done Sent false
33808 9492 Acked Read 4 Done Done Sent false
33809 9493 Acked Read 4 Done Done Sent false
33810 9494 Acked Flush 0 Done Done Sent false
33811 9495 Acked Read 8 Done Done Sent false
33812 9496 Acked Write 4 Done Done Sent false
33813 9497 Acked Write 1 Done Done Sent false
33814 9498 Acked Read 4 Done Done Sent false
33815 9499 Acked Write 8 Done Done Sent false
33816 9500 Acked Write 6 Done Done Sent false
33817 9501 Acked Write 6 Done Done Sent false
33818 9502 Acked Read 5 Done Done Sent false
33819 9503 Acked Flush 0 Done Done Sent false
33820 9504 Acked Read 7 Done Done Sent false
33821 9505 Acked Write 2 Done Done Sent false
33822 9506 Acked Read 9 Done Done Sent false
33823 9507 Acked Read 3 Done Done Sent false
33824 9508 Acked Read 10 Done Done Sent false
33825 9509 Acked Read 8 Done Done Sent false
33826 9510 Acked Read 6 Done Done Sent false
33827 9511 Acked Write 1 Done Done Sent false
33828 9512 Acked Read 2 Done Done Sent false
33829 9513 Acked Read 10 Done Done Sent false
33830 9514 Acked Read 7 Done Done Sent false
33831 9515 Acked Write 3 Done Done Sent false
33832 9516 Acked Flush 0 Done Done Sent false
33833 9517 Acked Read 7 Done Done Sent false
33834 9518 Acked Read 5 Done Done Sent false
33835 9519 Acked Write 10 Done Done Sent false
33836 9520 Acked Read 9 Done Done Sent false
33837 9521 Acked Read 5 Done Done Sent false
33838 9522 Acked Read 7 Done Done Sent false
33839 9523 Acked Write 9 Done Done Sent false
33840 9524 Acked Flush 0 Done Done Sent false
33841 9525 Acked Flush 0 Done Done Sent false
33842 9526 Acked Flush 0 Done Done Sent false
33843 9527 Acked Read 4 Done Done Sent false
33844 9528 Acked Write 5 Done Done Sent false
33845 9529 Acked Read 9 Done Done Sent false
33846 9530 Acked Read 4 Done Done Sent false
33847 9531 Acked Read 7 Done Done Sent false
33848 9532 Acked Write 4 Done Done Sent false
33849 9533 Acked Read 8 Done Done Sent false
33850 9534 Acked Read 5 Done Done Sent false
33851 9535 Acked Write 6 Done Done Sent false
33852 9536 Acked Read 5 Done Done Sent false
33853 9537 Acked Write 7 Done Done Sent false
33854 9538 Acked Write 2 Done Done Sent false
33855 9539 Acked Read 2 Done Done Sent false
33856 9540 Acked Write 3 Done Done Sent false
33857 9541 Acked Read 7 Done Done Sent false
33858 9542 Acked Read 3 Done Done Sent false
33859 9543 Acked Read 4 Done Done Sent false
33860 9544 Acked Write 8 Done Done Sent false
33861 9545 Acked Write 5 Done Done Sent false
33862 9546 Acked Write 2 Done Done Sent false
33863 9547 Acked Write 4 Done Done Sent false
33864 9548 Acked Read 2 Done Done Sent false
33865 9549 Acked Write 5 Done Done Sent false
33866 9550 Acked Read 10 Done Done Sent false
33867 9551 Acked Write 9 Done Done Sent false
33868 9552 Acked Read 10 Done Done Sent false
33869 9553 Acked Read 8 Done Done Sent false
33870 9554 Acked Read 6 Done Done Sent false
33871 9555 Acked Read 1 Done Done Sent false
33872 9556 Acked Write 8 Done Done Sent false
33873 9557 Acked Flush 0 Done Done Sent false
33874 9558 Acked Read 3 Done Done Sent false
33875 9559 Acked Read 2 Done Done Sent false
33876 9560 Acked Write 2 Done Done Sent false
33877 9561 Acked Write 10 Done Done Sent false
33878 9562 Acked Write 7 Done Done Sent false
33879 9563 Acked Read 10 Done Done Sent false
33880 9564 Acked Write 9 Done Done Sent false
33881 9565 Acked Write 9 Done Done Sent false
33882 9566 Acked Write 9 Done Done Sent false
33883 9567 Acked Write 4 Done Done Sent false
33884 9568 Acked Read 9 Done Done Sent false
33885 9569 Acked Write 8 Done Done Sent false
33886 9570 Acked Read 6 Done Done Sent false
33887 9571 Acked Read 6 Done Done Sent false
33888 9572 Acked Flush 0 Done Done Sent false
33889 9573 Acked Read 5 Done Done Sent false
33890 9574 Acked Read 4 Done Done Sent false
33891 9575 Acked Read 7 Done Done Sent false
33892 9576 Acked Read 4 Done Done Sent false
33893 9577 Acked Read 3 Done Done Sent false
33894 9578 Acked Write 7 Done Done Sent false
33895 9579 Acked Write 3 Done Done Sent false
33896 9580 Acked Read 4 Done Done Sent false
33897 9581 Acked Read 1 Done Done Sent false
33898 9582 Acked Read 8 Done Done Sent false
33899 9583 Acked Read 7 Done Done Sent false
33900 9584 Acked Read 7 Done Done Sent false
33901 9585 Acked Write 3 Done Done Sent false
33902 9586 Acked Read 1 Done Done Sent false
33903 9587 Acked Write 8 Done Done Sent false
33904 9588 Acked Write 5 Done Done Sent false
33905 9589 Acked Read 10 Done Done Sent false
33906 9590 Acked Write 7 Done Done Sent false
33907 9591 Acked Write 3 Done Done Sent false
33908 9592 Acked Write 10 Done Done Sent false
33909 9593 Acked Flush 0 Done Done Sent false
33910 9594 Acked Write 6 Done Done Sent false
33911 9595 Acked Read 6 Done Done Sent false
33912 9596 Acked Read 9 Done Done Sent false
33913 9597 Acked Read 2 Done Done Sent false
33914 9598 Acked Write 10 Done Done Sent false
33915 9599 Acked Read 6 Done Done Sent false
33916 9600 Acked Read 8 Done Done Sent false
33917 9601 Acked Flush 0 Done Done Sent false
33918 9602 Acked Read 3 Done Done Sent false
33919 9603 Acked Read 2 Done Done Sent false
33920 9604 Acked Read 1 Done Done Sent false
33921 9605 Acked Read 6 Done Done Sent false
33922 9606 Acked Write 10 Done Done Sent false
33923 9607 Acked Read 10 Done Done Sent false
33924 9608 Acked Flush 0 Done Done Sent false
33925 9609 Acked Read 2 Done Done Sent false
33926 9610 Acked Read 3 Done Done Sent false
33927 9611 Acked Read 2 Done Done Sent false
33928 9612 Acked Write 10 Done Done Sent false
33929 9613 Acked Flush 0 Done Done Sent false
33930 9614 Acked Write 6 Done Done Sent false
33931 9615 Acked Write 8 Done Done Sent false
33932 9616 Acked Write 1 Done Done Sent false
33933 9617 Acked Write 6 Done Done Sent false
33934 9618 Acked Read 1 Done Done Sent false
33935 9619 Acked Read 1 Done Done Sent false
33936 9620 Acked Read 7 Done Done Sent false
33937 9621 Acked Read 10 Done Done Sent false
33938 9622 Acked Write 5 Done Done Sent false
33939 9623 Acked Read 4 Done Done Sent false
33940 9624 Acked Write 8 Done Done Sent false
33941 9625 Acked Write 6 Done Done Sent false
33942 9626 Acked Read 3 Done Done Sent false
33943 9627 Acked Read 8 Done Done Sent false
33944 9628 Acked Write 5 Done Done Sent false
33945 9629 Acked Write 7 Done Done Sent false
33946 9630 Acked Write 3 Done Done Sent false
33947 9631 Acked Read 2 Done Done Sent false
33948 9632 Acked Write 8 Done Done Sent false
33949 9633 Acked Read 6 Done Done Sent false
33950 9634 Acked Write 10 Done Done Sent false
33951 9635 Acked Read 1 Done Done Sent false
33952 9636 Acked Read 9 Done Done Sent false
33953 9637 Acked Read 1 Done Done Sent false
33954 9638 Acked Read 8 Done Done Sent false
33955 9639 Acked Write 8 Done Done Sent false
33956 9640 Acked Write 2 Done Done Sent false
33957 9641 Acked Write 4 Done Done Sent false
33958 9642 Acked Read 9 Done Done Sent false
33959 9643 Acked Write 6 Done Done Sent false
33960 9644 Acked Read 5 Done Done Sent false
33961 9645 Acked Write 4 Done Done Sent false
33962 9646 Acked Write 9 Done Done Sent false
33963 9647 Acked Flush 0 Done Done Sent false
33964 9648 Acked Read 3 Done Done Sent false
33965 9649 Acked Read 1 Done Done Sent false
33966 9650 Acked Read 10 Done Done Sent false
33967 9651 Acked Write 5 Done Done Sent false
33968 9652 Acked Read 10 Done Done Sent false
33969 9653 Acked Read 5 Done Done Sent false
33970 9654 Acked Read 2 Done Done Sent false
33971 9655 Acked Read 2 Done Done Sent false
33972 9656 Acked Read 8 Done Done Sent false
33973 9657 Acked Read 10 Done Done Sent false
33974 9658 Acked Flush 0 Done Done Sent false
33975 9659 Acked Read 9 Done Done Sent false
33976 9660 Acked Read 9 Done Done Sent false
33977 9661 Acked Read 10 Done Done Sent false
33978 9662 Acked Write 1 Done Done Sent false
33979 9663 Acked Write 10 Done Done Sent false
33980 9664 Acked Read 3 Done Done Sent false
33981 9665 Acked Read 1 Done Done Sent false
33982 9666 Acked Read 5 Done Done Sent false
33983 9667 Acked Read 1 Done Done Sent false
33984 9668 Acked Flush 0 Done Done Sent false
33985 9669 Acked Write 7 Done Done Sent false
33986 9670 Acked Read 9 Done Done Sent false
33987 9671 Acked Flush 0 Done Done Sent false
33988 9672 Acked Write 8 Done Done Sent false
33989 9673 Acked Read 4 Done Done Sent false
33990 9674 Acked Flush 0 Done Done Sent false
33991 9675 Acked Write 2 Done Done Sent false
33992 9676 Acked Write 2 Done Done Sent false
33993 9677 Acked Flush 0 Done Done Sent false
33994 9678 Acked Write 4 Done Done Sent false
33995 9679 Acked Write 3 Done Done Sent false
33996 9680 Acked Write 2 Done Done Sent false
33997 9681 Acked Write 9 Done Done Sent false
33998 9682 Acked Write 8 Done Done Sent false
33999 9683 Acked Write 8 Done Done Sent false
34000 9684 Acked Write 4 Done Done Sent false
34001 9685 Acked Write 5 Done Done Sent false
34002 9686 Acked Write 6 Done Done Sent false
34003 9687 Acked Write 1 Done Done Sent false
34004 9688 Acked Read 8 Done Done Sent false
34005 9689 Acked Write 9 Done Done Sent false
34006 9690 Acked Flush 0 Done Done Sent false
34007 9691 Acked Read 3 Done Done Sent false
34008 9692 Acked Read 6 Done Done Sent false
34009 9693 Acked Write 6 Done Done Sent false
34010 9694 Acked Read 5 Done Done Sent false
34011 9695 Acked Read 1 Done Done Sent false
34012 9696 Acked Read 2 Done Done Sent false
34013 9697 Acked Read 9 Done Done Sent false
34014 9698 Acked Read 2 Done Done Sent false
34015 9699 Acked Write 10 Done Done Sent false
34016 9700 Acked Write 2 Done Done Sent false
34017 9701 Acked Read 4 Done Done Sent false
34018 9702 Acked Write 10 Done Done Sent false
34019 9703 Acked Write 6 Done Done Sent false
34020 9704 Acked Read 5 Done Done Sent false
34021 9705 Acked Read 10 Done Done Sent false
34022 9706 Acked Read 4 Done Done Sent false
34023 9707 Acked Read 5 Done Done Sent false
34024 9708 Acked Write 4 Done Done Sent false
34025 9709 Acked Read 8 Done Done Sent false
34026 9710 Acked Write 7 Done Done Sent false
34027 9711 Acked Write 9 Done Done Sent false
34028 9712 Acked Write 5 Done Done Sent false
34029 9713 Acked Read 3 Done Done Sent false
34030 9714 Acked Write 10 Done Done Sent false
34031 9715 Acked Write 9 Done Done Sent false
34032 9716 Acked Read 5 Done Done Sent false
34033 9717 Acked Flush 0 Done Done Sent false
34034 9718 Acked Write 9 Done Done Sent false
34035 9719 Acked Read 7 Done Done Sent false
34036 9720 Acked Flush 0 Done Done Sent false
34037 9721 Acked Write 10 Done Done Sent false
34038 9722 Acked Write 2 Done Done Sent false
34039 9723 Acked Write 4 Done Done Sent false
34040 9724 Acked Write 3 Done Done Sent false
34041 9725 Acked Read 4 Done Done Sent false
34042 9726 Acked Write 8 Done Done Sent false
34043 9727 Acked Write 9 Done Done Sent false
34044 9728 Acked Write 3 Done Done Sent false
34045 9729 Acked Read 2 Done Done Sent false
34046 9730 Acked Write 8 Done Done Sent false
34047 9731 Acked Read 6 Done Done Sent false
34048 9732 Acked Write 2 Done Done Sent false
34049 9733 Acked Write 4 Done Done Sent false
34050 9734 Acked Write 2 Done Done Sent false
34051 9735 Acked Write 4 Done Done Sent false
34052 9736 Acked Read 7 Done Done Sent false
34053 9737 Acked Read 10 Done Done Sent false
34054 9738 Acked Read 9 Done Done Sent false
34055 9739 Acked Flush 0 Done Done Sent false
34056 9740 Acked Flush 0 Done Done Sent false
34057 9741 Acked Read 2 Done Done Sent false
34058 9742 Acked Read 6 Done Done Sent false
34059 9743 Acked Read 3 Done Done Sent false
34060 9744 Acked Read 6 Done Done Sent false
34061 9745 Acked Read 4 Done Done Sent false
34062 9746 Acked Write 6 Done Done Sent false
34063 9747 Acked Read 3 Done Done Sent false
34064 9748 Acked Read 5 Done Done Sent false
34065 9749 Acked Read 6 Done Done Sent false
34066 9750 Acked Read 5 Done Done Sent false
34067 9751 Acked Read 2 Done Done Sent false
34068 9752 Acked Read 4 Done Done Sent false
34069 STATES DS:0 DS:1 DS:2 TOTAL
34070 Sent 0 0 301 301
34071 Done 301 301 0 602
34072 Skipped 0 0 0 0
34073 Error 0 0 0 0
34074 Last Flush: Some(JobId(9740)) Some(JobId(9740)) Some(JobId(9451))
34075 Downstairs last five completed: 9451 9450 9449 9448 9447
34076 Upstairs last five completed: 9752 9751 9750 9749 9748
34077 CLIENT: Up:0 ds:301 act:2
340782025-02-28T06:08:32.093ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34079 [8830] got start action
34080 [8830] I am going to restart
34081 Starting downstairs at port 8830
34082 Make output file at "/tmp/dsc/downstairs-8830.txt"
34083 [8830][2] reports Starting
34084 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1240)
34085 [8830][2] reports Running
340862025-02-28T06:08:33.095ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
340872025-02-28T06:08:34.097ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
340882025-02-28T06:08:35.097ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34089 ----------------------------------------------------------------
34090 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34091 States: Active Active Offline
34092 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34093 9452 Acked Write 10 Done Done Sent false
34094 9453 Acked Write 10 Done Done Sent false
34095 9454 Acked Read 4 Done Done Sent false
34096 9455 Acked Read 3 Done Done Sent false
34097 9456 Acked Read 7 Done Done Sent false
34098 9457 Acked Read 4 Done Done Sent false
34099 9458 Acked Read 2 Done Done Sent false
34100 9459 Acked Read 2 Done Done Sent false
34101 9460 Acked Write 10 Done Done Sent false
34102 9461 Acked Write 5 Done Done Sent false
34103 9462 Acked Read 10 Done Done Sent false
34104 9463 Acked Read 10 Done Done Sent false
34105 9464 Acked Read 4 Done Done Sent false
34106 9465 Acked Flush 0 Done Done Sent false
34107 9466 Acked Write 8 Done Done Sent false
34108 9467 Acked Read 3 Done Done Sent false
34109 9468 Acked Write 4 Done Done Sent false
34110 9469 Acked Write 1 Done Done Sent false
34111 9470 Acked Read 7 Done Done Sent false
34112 9471 Acked Write 1 Done Done Sent false
34113 9472 Acked Read 10 Done Done Sent false
34114 9473 Acked Write 6 Done Done Sent false
34115 9474 Acked Write 4 Done Done Sent false
34116 9475 Acked Read 9 Done Done Sent false
34117 9476 Acked Read 7 Done Done Sent false
34118 9477 Acked Read 6 Done Done Sent false
34119 9478 Acked Flush 0 Done Done Sent false
34120 9479 Acked Read 6 Done Done Sent false
34121 9480 Acked Read 7 Done Done Sent false
34122 9481 Acked Write 10 Done Done Sent false
34123 9482 Acked Read 5 Done Done Sent false
34124 9483 Acked Flush 0 Done Done Sent false
34125 9484 Acked Write 7 Done Done Sent false
34126 9485 Acked Read 6 Done Done Sent false
34127 9486 Acked Read 9 Done Done Sent false
34128 9487 Acked Read 6 Done Done Sent false
34129 9488 Acked Flush 0 Done Done Sent false
34130 9489 Acked Read 4 Done Done Sent false
34131 9490 Acked Write 7 Done Done Sent false
34132 9491 Acked Flush 0 Done Done Sent false
34133 9492 Acked Read 4 Done Done Sent false
34134 9493 Acked Read 4 Done Done Sent false
34135 9494 Acked Flush 0 Done Done Sent false
34136 9495 Acked Read 8 Done Done Sent false
34137 9496 Acked Write 4 Done Done Sent false
34138 9497 Acked Write 1 Done Done Sent false
34139 9498 Acked Read 4 Done Done Sent false
34140 9499 Acked Write 8 Done Done Sent false
34141 9500 Acked Write 6 Done Done Sent false
34142 9501 Acked Write 6 Done Done Sent false
34143 9502 Acked Read 5 Done Done Sent false
34144 9503 Acked Flush 0 Done Done Sent false
34145 9504 Acked Read 7 Done Done Sent false
34146 9505 Acked Write 2 Done Done Sent false
34147 9506 Acked Read 9 Done Done Sent false
34148 9507 Acked Read 3 Done Done Sent false
34149 9508 Acked Read 10 Done Done Sent false
34150 9509 Acked Read 8 Done Done Sent false
34151 9510 Acked Read 6 Done Done Sent false
34152 9511 Acked Write 1 Done Done Sent false
34153 9512 Acked Read 2 Done Done Sent false
34154 9513 Acked Read 10 Done Done Sent false
34155 9514 Acked Read 7 Done Done Sent false
34156 9515 Acked Write 3 Done Done Sent false
34157 9516 Acked Flush 0 Done Done Sent false
34158 9517 Acked Read 7 Done Done Sent false
34159 9518 Acked Read 5 Done Done Sent false
34160 9519 Acked Write 10 Done Done Sent false
34161 9520 Acked Read 9 Done Done Sent false
34162 9521 Acked Read 5 Done Done Sent false
34163 9522 Acked Read 7 Done Done Sent false
34164 9523 Acked Write 9 Done Done Sent false
34165 9524 Acked Flush 0 Done Done Sent false
34166 9525 Acked Flush 0 Done Done Sent false
34167 9526 Acked Flush 0 Done Done Sent false
34168 9527 Acked Read 4 Done Done Sent false
34169 9528 Acked Write 5 Done Done Sent false
34170 9529 Acked Read 9 Done Done Sent false
34171 9530 Acked Read 4 Done Done Sent false
34172 9531 Acked Read 7 Done Done Sent false
34173 9532 Acked Write 4 Done Done Sent false
34174 9533 Acked Read 8 Done Done Sent false
34175 9534 Acked Read 5 Done Done Sent false
34176 9535 Acked Write 6 Done Done Sent false
34177 9536 Acked Read 5 Done Done Sent false
34178 9537 Acked Write 7 Done Done Sent false
34179 9538 Acked Write 2 Done Done Sent false
34180 9539 Acked Read 2 Done Done Sent false
34181 9540 Acked Write 3 Done Done Sent false
34182 9541 Acked Read 7 Done Done Sent false
34183 9542 Acked Read 3 Done Done Sent false
34184 9543 Acked Read 4 Done Done Sent false
34185 9544 Acked Write 8 Done Done Sent false
34186 9545 Acked Write 5 Done Done Sent false
34187 9546 Acked Write 2 Done Done Sent false
34188 9547 Acked Write 4 Done Done Sent false
34189 9548 Acked Read 2 Done Done Sent false
34190 9549 Acked Write 5 Done Done Sent false
34191 9550 Acked Read 10 Done Done Sent false
34192 9551 Acked Write 9 Done Done Sent false
34193 9552 Acked Read 10 Done Done Sent false
34194 9553 Acked Read 8 Done Done Sent false
34195 9554 Acked Read 6 Done Done Sent false
34196 9555 Acked Read 1 Done Done Sent false
34197 9556 Acked Write 8 Done Done Sent false
34198 9557 Acked Flush 0 Done Done Sent false
34199 9558 Acked Read 3 Done Done Sent false
34200 9559 Acked Read 2 Done Done Sent false
34201 9560 Acked Write 2 Done Done Sent false
34202 9561 Acked Write 10 Done Done Sent false
34203 9562 Acked Write 7 Done Done Sent false
34204 9563 Acked Read 10 Done Done Sent false
34205 9564 Acked Write 9 Done Done Sent false
34206 9565 Acked Write 9 Done Done Sent false
34207 9566 Acked Write 9 Done Done Sent false
34208 9567 Acked Write 4 Done Done Sent false
34209 9568 Acked Read 9 Done Done Sent false
34210 9569 Acked Write 8 Done Done Sent false
34211 9570 Acked Read 6 Done Done Sent false
34212 9571 Acked Read 6 Done Done Sent false
34213 9572 Acked Flush 0 Done Done Sent false
34214 9573 Acked Read 5 Done Done Sent false
34215 9574 Acked Read 4 Done Done Sent false
34216 9575 Acked Read 7 Done Done Sent false
34217 9576 Acked Read 4 Done Done Sent false
34218 9577 Acked Read 3 Done Done Sent false
34219 9578 Acked Write 7 Done Done Sent false
34220 9579 Acked Write 3 Done Done Sent false
34221 9580 Acked Read 4 Done Done Sent false
34222 9581 Acked Read 1 Done Done Sent false
34223 9582 Acked Read 8 Done Done Sent false
34224 9583 Acked Read 7 Done Done Sent false
34225 9584 Acked Read 7 Done Done Sent false
34226 9585 Acked Write 3 Done Done Sent false
34227 9586 Acked Read 1 Done Done Sent false
34228 9587 Acked Write 8 Done Done Sent false
34229 9588 Acked Write 5 Done Done Sent false
34230 9589 Acked Read 10 Done Done Sent false
34231 9590 Acked Write 7 Done Done Sent false
34232 9591 Acked Write 3 Done Done Sent false
34233 9592 Acked Write 10 Done Done Sent false
34234 9593 Acked Flush 0 Done Done Sent false
34235 9594 Acked Write 6 Done Done Sent false
34236 9595 Acked Read 6 Done Done Sent false
34237 9596 Acked Read 9 Done Done Sent false
34238 9597 Acked Read 2 Done Done Sent false
34239 9598 Acked Write 10 Done Done Sent false
34240 9599 Acked Read 6 Done Done Sent false
34241 9600 Acked Read 8 Done Done Sent false
34242 9601 Acked Flush 0 Done Done Sent false
34243 9602 Acked Read 3 Done Done Sent false
34244 9603 Acked Read 2 Done Done Sent false
34245 9604 Acked Read 1 Done Done Sent false
34246 9605 Acked Read 6 Done Done Sent false
34247 9606 Acked Write 10 Done Done Sent false
34248 9607 Acked Read 10 Done Done Sent false
34249 9608 Acked Flush 0 Done Done Sent false
34250 9609 Acked Read 2 Done Done Sent false
34251 9610 Acked Read 3 Done Done Sent false
34252 9611 Acked Read 2 Done Done Sent false
34253 9612 Acked Write 10 Done Done Sent false
34254 9613 Acked Flush 0 Done Done Sent false
34255 9614 Acked Write 6 Done Done Sent false
34256 9615 Acked Write 8 Done Done Sent false
34257 9616 Acked Write 1 Done Done Sent false
34258 9617 Acked Write 6 Done Done Sent false
34259 9618 Acked Read 1 Done Done Sent false
34260 9619 Acked Read 1 Done Done Sent false
34261 9620 Acked Read 7 Done Done Sent false
34262 9621 Acked Read 10 Done Done Sent false
34263 9622 Acked Write 5 Done Done Sent false
34264 9623 Acked Read 4 Done Done Sent false
34265 9624 Acked Write 8 Done Done Sent false
34266 9625 Acked Write 6 Done Done Sent false
34267 9626 Acked Read 3 Done Done Sent false
34268 9627 Acked Read 8 Done Done Sent false
34269 9628 Acked Write 5 Done Done Sent false
34270 9629 Acked Write 7 Done Done Sent false
34271 9630 Acked Write 3 Done Done Sent false
34272 9631 Acked Read 2 Done Done Sent false
34273 9632 Acked Write 8 Done Done Sent false
34274 9633 Acked Read 6 Done Done Sent false
34275 9634 Acked Write 10 Done Done Sent false
34276 9635 Acked Read 1 Done Done Sent false
34277 9636 Acked Read 9 Done Done Sent false
34278 9637 Acked Read 1 Done Done Sent false
34279 9638 Acked Read 8 Done Done Sent false
34280 9639 Acked Write 8 Done Done Sent false
34281 9640 Acked Write 2 Done Done Sent false
34282 9641 Acked Write 4 Done Done Sent false
34283 9642 Acked Read 9 Done Done Sent false
34284 9643 Acked Write 6 Done Done Sent false
34285 9644 Acked Read 5 Done Done Sent false
34286 9645 Acked Write 4 Done Done Sent false
34287 9646 Acked Write 9 Done Done Sent false
34288 9647 Acked Flush 0 Done Done Sent false
34289 9648 Acked Read 3 Done Done Sent false
34290 9649 Acked Read 1 Done Done Sent false
34291 9650 Acked Read 10 Done Done Sent false
34292 9651 Acked Write 5 Done Done Sent false
34293 9652 Acked Read 10 Done Done Sent false
34294 9653 Acked Read 5 Done Done Sent false
34295 9654 Acked Read 2 Done Done Sent false
34296 9655 Acked Read 2 Done Done Sent false
34297 9656 Acked Read 8 Done Done Sent false
34298 9657 Acked Read 10 Done Done Sent false
34299 9658 Acked Flush 0 Done Done Sent false
34300 9659 Acked Read 9 Done Done Sent false
34301 9660 Acked Read 9 Done Done Sent false
34302 9661 Acked Read 10 Done Done Sent false
34303 9662 Acked Write 1 Done Done Sent false
34304 9663 Acked Write 10 Done Done Sent false
34305 9664 Acked Read 3 Done Done Sent false
34306 9665 Acked Read 1 Done Done Sent false
34307 9666 Acked Read 5 Done Done Sent false
34308 9667 Acked Read 1 Done Done Sent false
34309 9668 Acked Flush 0 Done Done Sent false
34310 9669 Acked Write 7 Done Done Sent false
34311 9670 Acked Read 9 Done Done Sent false
34312 9671 Acked Flush 0 Done Done Sent false
34313 9672 Acked Write 8 Done Done Sent false
34314 9673 Acked Read 4 Done Done Sent false
34315 9674 Acked Flush 0 Done Done Sent false
34316 9675 Acked Write 2 Done Done Sent false
34317 9676 Acked Write 2 Done Done Sent false
34318 9677 Acked Flush 0 Done Done Sent false
34319 9678 Acked Write 4 Done Done Sent false
34320 9679 Acked Write 3 Done Done Sent false
34321 9680 Acked Write 2 Done Done Sent false
34322 9681 Acked Write 9 Done Done Sent false
34323 9682 Acked Write 8 Done Done Sent false
34324 9683 Acked Write 8 Done Done Sent false
34325 9684 Acked Write 4 Done Done Sent false
34326 9685 Acked Write 5 Done Done Sent false
34327 9686 Acked Write 6 Done Done Sent false
34328 9687 Acked Write 1 Done Done Sent false
34329 9688 Acked Read 8 Done Done Sent false
34330 9689 Acked Write 9 Done Done Sent false
34331 9690 Acked Flush 0 Done Done Sent false
34332 9691 Acked Read 3 Done Done Sent false
34333 9692 Acked Read 6 Done Done Sent false
34334 9693 Acked Write 6 Done Done Sent false
34335 9694 Acked Read 5 Done Done Sent false
34336 9695 Acked Read 1 Done Done Sent false
34337 9696 Acked Read 2 Done Done Sent false
34338 9697 Acked Read 9 Done Done Sent false
34339 9698 Acked Read 2 Done Done Sent false
34340 9699 Acked Write 10 Done Done Sent false
34341 9700 Acked Write 2 Done Done Sent false
34342 9701 Acked Read 4 Done Done Sent false
34343 9702 Acked Write 10 Done Done Sent false
34344 9703 Acked Write 6 Done Done Sent false
34345 9704 Acked Read 5 Done Done Sent false
34346 9705 Acked Read 10 Done Done Sent false
34347 9706 Acked Read 4 Done Done Sent false
34348 9707 Acked Read 5 Done Done Sent false
34349 9708 Acked Write 4 Done Done Sent false
34350 9709 Acked Read 8 Done Done Sent false
34351 9710 Acked Write 7 Done Done Sent false
34352 9711 Acked Write 9 Done Done Sent false
34353 9712 Acked Write 5 Done Done Sent false
34354 9713 Acked Read 3 Done Done Sent false
34355 9714 Acked Write 10 Done Done Sent false
34356 9715 Acked Write 9 Done Done Sent false
34357 9716 Acked Read 5 Done Done Sent false
34358 9717 Acked Flush 0 Done Done Sent false
34359 9718 Acked Write 9 Done Done Sent false
34360 9719 Acked Read 7 Done Done Sent false
34361 9720 Acked Flush 0 Done Done Sent false
34362 9721 Acked Write 10 Done Done Sent false
34363 9722 Acked Write 2 Done Done Sent false
34364 9723 Acked Write 4 Done Done Sent false
34365 9724 Acked Write 3 Done Done Sent false
34366 9725 Acked Read 4 Done Done Sent false
34367 9726 Acked Write 8 Done Done Sent false
34368 9727 Acked Write 9 Done Done Sent false
34369 9728 Acked Write 3 Done Done Sent false
34370 9729 Acked Read 2 Done Done Sent false
34371 9730 Acked Write 8 Done Done Sent false
34372 9731 Acked Read 6 Done Done Sent false
34373 9732 Acked Write 2 Done Done Sent false
34374 9733 Acked Write 4 Done Done Sent false
34375 9734 Acked Write 2 Done Done Sent false
34376 9735 Acked Write 4 Done Done Sent false
34377 9736 Acked Read 7 Done Done Sent false
34378 9737 Acked Read 10 Done Done Sent false
34379 9738 Acked Read 9 Done Done Sent false
34380 9739 Acked Flush 0 Done Done Sent false
34381 9740 Acked Flush 0 Done Done Sent false
34382 9741 Acked Read 2 Done Done Sent false
34383 9742 Acked Read 6 Done Done Sent false
34384 9743 Acked Read 3 Done Done Sent false
34385 9744 Acked Read 6 Done Done Sent false
34386 9745 Acked Read 4 Done Done Sent false
34387 9746 Acked Write 6 Done Done Sent false
34388 9747 Acked Read 3 Done Done Sent false
34389 9748 Acked Read 5 Done Done Sent false
34390 9749 Acked Read 6 Done Done Sent false
34391 9750 Acked Read 5 Done Done Sent false
34392 9751 Acked Read 2 Done Done Sent false
34393 9752 Acked Read 4 Done Done Sent false
34394 STATES DS:0 DS:1 DS:2 TOTAL
34395 Sent 0 0 301 301
34396 Done 301 301 0 602
34397 Skipped 0 0 0 0
34398 Error 0 0 0 0
34399 Last Flush: Some(JobId(9740)) Some(JobId(9740)) Some(JobId(9451))
34400 Downstairs last five completed: 9451 9450 9449 9448 9447
34401 Upstairs last five completed: 9752 9751 9750 9749 9748
34402 CLIENT: Up:0 ds:301 act:2
344032025-02-28T06:08:36.100ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
344042025-02-28T06:08:36.923ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
344052025-02-28T06:08:36.924ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:45461) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
344062025-02-28T06:08:36.925ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
344072025-02-28T06:08:36.925ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
344082025-02-28T06:08:36.925ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
344092025-02-28T06:08:36.925ZINFOcrucible: send last flush ID to this DS: Some(JobId(9451)) = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
344102025-02-28T06:08:36.925ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
344112025-02-28T06:08:36.925ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9451)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
344122025-02-28T06:08:36.926ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9451)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34413 ----------------------------------------------------------------
34414 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
34415 Downstairs last five completed: 9753 9752 9751 9750 9749
34416 Upstairs last five completed: 9753 9752 9751 9750 9749
34417 CLIENT: Up:0 ds:0 act:3
34418 Replay: All jobs finished
34419 Main task has work to do, go find it
34420 got dsc Stop(0)
34421 stop 0
34422 Feb 28 06:08:39.179 INFO request completed, latency_us: 397, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 4b5e5c4c-eed5-4ff3-8e4a-f942ac4f245c, remote_addr: 127.0.0.1:63908, local_addr[8810] Got stop action so:false kr:false
34423 : 127.0.0.1:9998
34424 Feb 28 06:08:39.180 INFO request completed, latency_us: 118, response_code: 200, uri: /state/cid/0, method: GET, req_id: f080f39d-75e2-44b4-a93b-6bae9b47a348, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
344252025-02-28T06:08:39.180ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
344262025-02-28T06:08:39.180ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
344272025-02-28T06:08:39.180ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
344282025-02-28T06:08:39.180ZINFOcrucible: client task is exiting = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
34429 [8810] Exited with: ExitStatus(unix_wait_status(9))
34430 [8810][0] reports Exit
34431 Feb 28 06:08:43.182 INFO request completed, latency_us: 234, response_code: 200, uri: /state/cid/0, method: GET, req_id: 508914b3-5f4d-4e4e-98a5-85ab5d551d1a, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
34432 001/300 Flush
34433 002/300 Flush
34434 003/300 Read block 3690 len 36864
34435 004/300 Write block 2679 len 36864 data: 8 7 5 4 5 8 10 9 10
34436 005/300 Read block 816 len 32768
34437 006/300 Write block 2219 len 28672 data: 9 8 5 5 5 5 6
34438 007/300 Write block 3232 len 40960 data: 2 3 2 2 3 3 4 4 2 2
34439 008/300 Write block 1505 len 4096 data: 4
34440 009/300 Write block 323 len 32768 data: 7 3 4 3 3 2 2 2
34441 010/300 Flush
34442 011/300 Write block 4740 len 28672 data: 8 7 6 4 5 4 3
34443 012/300 Read block 2559 len 24576
34444 013/300 Write block 3267 len 8192 data: 5 5
34445 014/300 Read block 626 len 4096
34446 015/300 Write block 2466 len 12288 data: 2 4 6
34447 016/300 Read block 460 len 24576
34448 017/300 Write block 3453 len 32768 data: 5 5 5 6 7 7 7 4
34449 018/300 Read block 4801 len 28672
34450 019/300 Write block 3453 len 16384 data: 6 6 6 7
34451 020/300 Write block 2572 len 40960 data: 6 6 4 3 3 4 5 7 9 8
34452 021/300 Read block 2523 len 16384
34453 022/300 Write block 1266 len 28672 data: 6 5 5 4 4 5 5
34454 023/300 Write block 1829 len 4096 data: 8
34455 024/300 Write block 4256 len 40960 data: 3 3 3 3 4 3 5 5 5 5
34456 025/300 Read block 4919 len 36864
34457 026/300 Read block 4495 len 20480
34458 027/300 Write block 3026 len 8192 data: 5 7
34459 028/300 Read block 486 len 8192
34460 029/300 Flush
34461 030/300 Read block 586 len 16384
34462 031/300 Read block 2161 len 28672
34463 032/300 Read block 43 len 24576
34464 033/300 Write block 4756 len 28672 data: 10 11 10 10 8 8 8
34465 034/300 Read block 868 len 4096
34466 035/300 Write block 918 len 24576 data: 7 7 8 8 8 7
34467 036/300 Flush
34468 037/300 Read block 1775 len 40960
34469 038/300 Write block 1061 len 16384 data: 6 6 5 5
34470 039/300 Read block 317 len 36864
34471 040/300 Write block 2263 len 28672 data: 8 10 8 8 9 7 6
34472 041/300 Read block 1114 len 28672
34473 042/300 Write block 197 len 24576 data: 7 6 6 6 5 5
34474 043/300 Read block 2628 len 8192
34475 044/300 Read block 4808 len 20480
34476 045/300 Write block 3214 len 8192 data: 8 8
34477 046/300 Read block 568 len 36864
34478 047/300 Read block 1912 len 28672
34479 048/300 Read block 2929 len 12288
34480 049/300 Write block 615 len 36864 data: 3 3 3 3 3 6 6 6 7
34481 050/300 Write block 2467 len 32768 data: 5 7 4 4 5 7 7 6
34482 051/300 Read block 2670 len 40960
34483 052/300 Write block 3699 len 32768 data: 5 4 5 6 3 4 5 6
34484 053/300 Write block 799 len 12288 data: 5 6 5
34485 054/300 Write block 4215 len 28672 data: 8 8 7 4 3 3 3
34486 055/300 Write block 4506 len 32768 data: 7 9 7 6 9 8 8 11
34487 056/300 Write block 1586 len 4096 data: 7
34488 057/300 Write block 1151 len 32768 data: 12 12 11 10 11 10 10 8
34489 058/300 Write block 420 len 8192 data: 10 11
34490 059/300 Read block 421 len 4096
34491 060/300 Read block 4808 len 8192
34492 061/300 Flush
34493 062/300 Write block 3346 len 4096 data: 8
34494 063/300 Read block 3033 len 24576
34495 064/300 Read block 4915 len 28672
34496 065/300 Read block 3485 len 8192
34497 066/300 Write block 1347 len 20480 data: 5 3 4 5 6
34498 067/300 Read block 202 len 16384
34499 068/300 Write block 3241 len 36864 data: 3 4 3 4 4 6 6 5 9
34500 069/300 Read block 4885 len 28672
34501 070/300 Write block 1962 len 28672 data: 3 4 6 5 5 6 4
34502 071/300 Read block 1214 len 32768
34503 072/300 Read block 3084 len 16384
34504 073/300 Read block 2137 len 28672
34505 074/300 Read block 609 len 24576
34506 075/300 Read block 2118 len 40960
34507 076/300 Write block 2204 len 36864 data: 5 4 5 4 5 7 6 5 6
34508 077/300 Write block 3012 len 40960 data: 5 5 5 6 4 5 5 5 5 5
34509 078/300 Read block 2609 len 12288
34510 079/300 Write block 1036 len 32768 data: 4 6 6 7 9 7 8 8
34511 080/300 Write block 4331 len 24576 data: 4 5 8 7 7 10
34512 081/300 Read block 1204 len 40960
34513 082/300 Read block 4560 len 16384
34514 083/300 Read block 3046 len 28672
34515 084/300 Write block 4296 len 36864 data: 6 6 7 5 3 3 2 4 4
34516 085/300 Write block 2686 len 28672 data: 10 11 7 7 6 6 6
34517 086/300 Write block 681 len 40960 data: 7 6 6 5 4 3 3 3 4 5
34518 087/300 Flush
34519 088/300 Read block 4756 len 24576
34520 089/300 Write block 3404 len 4096 data: 3
34521 090/300 Write block 2294 len 40960 data: 4 4 4 5 5 4 4 4 5 5
34522 091/300 Write block 3484 len 4096 data: 7
34523 092/300 Read block 4868 len 24576
34524 093/300 Read block 3886 len 8192
34525 094/300 Read block 4171 len 12288
34526 095/300 Read block 3048 len 40960
34527 096/300 Read block 1776 len 28672
34528 097/300 Flush
34529 098/300 Flush
34530 099/300 Write block 1530 len 24576 data: 6 5 5 5 7 6
34531 100/300 Read block 15 len 20480
34532 101/300 Write block 1050 len 20480 data: 7 7 8 8 8
34533 102/300 Read block 1766 len 24576
34534 103/300 Read block 4003 len 20480
34535 104/300 Write block 1702 len 32768 data: 6 9 8 6 5 7 7 9
34536 105/300 Read block 1170 len 24576
34537 106/300 Read block 885 len 24576
34538 107/300 Read block 439 len 28672
34539 108/300 Flush
34540 109/300 Read block 660 len 4096
34541 110/300 Read block 1271 len 28672
34542 111/300 Flush
34543 112/300 Read block 4244 len 24576
34544 113/300 Read block 1407 len 40960
34545 114/300 Read block 3381 len 40960
34546 115/300 Read block 550 len 28672
34547 116/300 Read block 1463 len 12288
34548 117/300 Write block 4457 len 24576 data: 4 4 4 5 7 6
34549 118/300 Write block 2663 len 32768 data: 8 9 9 9 9 9 11 12
34550 119/300 Read block 2518 len 36864
34551 120/300 Write block 2730 len 20480 data: 10 7 7 8 7
34552 121/300 Write block 1076 len 36864 data: 5 5 6 5 7 6 6 5 4
34553 122/300 Read block 903 len 16384
34554 123/300 Read block 4860 len 32768
34555 124/300 Flush
34556 125/300 Write block 211 len 32768 data: 8 6 6 8 10 8 9 9
34557 126/300 Flush
34558 127/300 Write block 3623 len 36864 data: 7 7 7 6 6 6 6 4 5
34559 128/300 Read block 2086 len 28672
34560 129/300 Flush
34561 130/300 Write block 4927 len 20480 data: 7 5 4 5 5
34562 131/300 Read block 4207 len 24576
34563 132/300 Write block 558 len 28672 data: 6 6 5 7 7 7 7
34564 133/300 Read block 3093 len 16384
34565 134/300 Write block 1992 len 28672 data: 4 4 4 4 4 4 4
34566 135/300 Read block 3913 len 36864
34567 136/300 Write block 624 len 4096 data: 7
34568 137/300 Read block 4257 len 4096
34569 138/300 Flush
34570 139/300 Read block 4497 len 28672
34571 140/300 Write block 3053 len 28672 data: 4 4 5 6 3 4 4
34572 141/300 Write block 2814 len 36864 data: 8 8 8 6 7 8 7 6 5
34573 142/300 Read block 3235 len 36864
34574 143/300 Read block 1948 len 36864
34575 144/300 Write block 3 len 20480 data: 5 6 7 6 5
34576 145/300 Read block 2277 len 28672
34577 146/300 Flush
34578 147/300 Read block 4964 len 12288
34579 148/300 Write block 312 len 20480 data: 4 5 5 5 5
34580 149/300 Read block 2846 len 32768
34581 150/300 Read block 1964 len 4096
34582 151/300 Write block 3277 len 4096 data: 4
34583 152/300 Write block 827 len 20480 data: 4 3 6 6 6
34584 153/300 Write block 1179 len 32768 data: 11 9 8 6 5 7 6 6
34585 154/300 Flush
34586 155/300 Read block 4061 len 36864
34587 156/300 Read block 3747 len 16384
34588 157/300 Flush
34589 158/300 Flush
34590 159/300 Read block 4733 len 36864
34591 160/300 Write block 2250 len 8192 data: 3 3
34592 161/300 Read block 2304 len 40960
34593 162/300 Write block 3783 len 36864 data: 7 5 4 4 3 4 3 3 4
34594 163/300 Read block 3878 len 12288
34595 164/300 Write block 4764 len 28672 data: 6 5 5 5 6 7 10
34596 165/300 Write block 644 len 36864 data: 7 7 6 7 8 6 6 5 5
34597 166/300 Read block 4742 len 32768
34598 167/300 Read block 2916 len 8192
34599 168/300 Read block 1713 len 16384
34600 169/300 Read block 3726 len 4096
34601 170/300 Write block 746 len 12288 data: 10 9 12
34602 171/300 Flush
34603 172/300 Read block 1638 len 4096
34604 173/300 Write block 1265 len 16384 data: 7 7 6 6
34605 174/300 Read block 788 len 12288
34606 175/300 Write block 850 len 4096 data: 3
34607 176/300 Read block 28 len 4096
34608 177/300 Write block 1227 len 12288 data: 11 10 9
34609 178/300 Read block 2045 len 32768
34610 179/300 Read block 3281 len 40960
34611 180/300 Read block 3183 len 12288
34612 181/300 Read block 1895 len 20480
34613 182/300 Write block 4040 len 8192 data: 8 8
34614 183/300 Write block 1747 len 32768 data: 9 8 8 8 5 6 6 7
34615 184/300 Write block 3519 len 32768 data: 3 4 4 7 6 6 7 9
34616 185/300 Write block 1583 len 40960 data: 8 6 6 8 7 6 6 5 5 4
34617 186/300 Write block 4989 len 20480 data: 4 4 2 2 2
34618 187/300 Read block 3250 len 32768
34619 188/300 Read block 4832 len 28672
34620 189/300 Read block 3115 len 24576
34621 190/300 Read block 1214 len 20480
34622 191/300 Read block 1751 len 32768
34623 192/300 Write block 4 len 20480 data: 7 8 7 6 5
34624 193/300 Write block 1982 len 36864 data: 8 10 10 8 8 7 8 7 4
34625 194/300 Write block 3453 len 36864 data: 7 7 7 8 8 8 8 5 6
34626 195/300 Write block 1154 len 28672 data: 11 12 11 11 9 8 6
34627 196/300 Read block 1357 len 8192
34628 197/300 Write block 2590 len 24576 data: 4 6 5 5 5 5
34629 198/300 Read block 455 len 12288
34630 199/300 Write block 1518 len 24576 data: 6 4 5 6 7 6
34631 200/300 Write block 576 len 32768 data: 5 6 6 7 8 7 8 9
34632 201/300 Read block 1163 len 36864
34633 202/300 Write block 3906 len 4096 data: 7
34634 203/300 Write block 3050 len 12288 data: 4 4 3
34635 204/300 Read block 2226 len 24576
34636 205/300 Write block 3736 len 12288 data: 6 6 5
34637 206/300 Flush
34638 207/300 Write block 2583 len 24576 data: 8 8 7 7 6 5
34639 208/300 Flush
34640 209/300 Flush
34641 210/300 Write block 4253 len 16384 data: 5 4 5 4
34642 211/300 Read block 3173 len 4096
34643 212/300 Write block 3543 len 32768 data: 3 3 6 7 6 6 5 6
34644 213/300 Read block 3532 len 16384
34645 214/300 Read block 197 len 36864
34646 215/300 Write block 1398 len 36864 data: 3 6 7 5 6 5 5 5 4
34647 216/300 Read block 4290 len 20480
34648 217/300 Write block 8 len 24576 data: 6 6 7 6 6 4
34649 218/300 Read block 4610 len 16384
34650 219/300 Write block 2505 len 28672 data: 3 3 3 4 5 5 7
34651 220/300 Write block 2606 len 40960 data: 5 6 6 5 5 8 6 6 5 6
34652 221/300 Flush
34653 222/300 Read block 1253 len 12288
34654 223/300 Read block 2416 len 12288
34655 224/300 Read block 3000 len 28672
34656 225/300 Read block 202 len 36864
34657 226/300 Write block 2394 len 40960 data: 5 4 3 3 3 2 3 3 3 3
34658 227/300 Read block 3407 len 36864
34659 228/300 Read block 3581 len 32768
34660 229/300 Write block 2447 len 36864 data: 7 6 5 5 6 7 6 7 9
34661 230/300 Write block 997 len 36864 data: 5 5 6 6 7 7 6 5 5
34662 231/300 Read block 4314 len 32768
34663 232/300 Write block 2172 len 4096 data: 5
34664 233/300 Flush
34665 234/300 Read block 861 len 24576
34666 235/300 Read block 268 len 24576
34667 236/300 Write block 459 len 12288 data: 7 7 7
34668 237/300 Read block 3841 len 28672
34669 238/300 Read block 1142 len 8192
34670 239/300 Read block 4504 len 40960
34671 240/300 Read block 1688 len 16384
34672 241/300 Write block 3047 len 12288 data: 4 4 4
34673 242/300 Write block 961 len 36864 data: 4 5 5 5 6 6 5 5 4
34674 243/300 Write block 3878 len 40960 data: 5 5 4 5 6 7 5 5 5 5
34675 244/300 Read block 364 len 28672
34676 245/300 Read block 983 len 16384
34677 246/300 Read block 3779 len 36864
34678 247/300 Read block 920 len 8192
34679 248/300 Flush
34680 249/300 Write block 4969 len 28672 data: 10 9 11 10 9 8 6
34681 250/300 Read block 2228 len 16384
34682 251/300 Read block 4840 len 8192
34683 252/300 Write block 1276 len 16384 data: 5 3 4 4
34684 253/300 Write block 3318 len 32768 data: 5 6 8 9 10 10 10 9
34685 254/300 Write block 4346 len 28672 data: 9 10 8 8 7 7 8
34686 255/300 Flush
34687 256/300 Flush
34688 257/300 Flush
34689 258/300 Write block 3613 len 32768 data: 7 8 7 6 8 7 8 6
34690 259/300 Read block 1900 len 12288
34691 260/300 Read block 4014 len 40960
34692 261/300 Write block 2096 len 8192 data: 7 7
34693 262/300 Read block 1430 len 20480
34694 263/300 Flush
34695 264/300 Read block 3485 len 24576
34696 265/300 Read block 886 len 28672
34697 266/300 Read block 2766 len 16384
34698 267/300 Write block 1424 len 20480 data: 6 7 7 7 4
34699 268/300 Read block 1686 len 20480
34700 269/300 Flush
34701 270/300 Read block 3866 len 36864
34702 271/300 Read block 1038 len 4096
34703 272/300 Flush
34704 273/300 Read block 857 len 16384
34705 274/300 Read block 4747 len 36864
34706 275/300 Write block 4403 len 12288 data: 5 6 10
34707 276/300 Write block 652 len 28672 data: 6 5 5 5 5 6 7
34708 277/300 Read block 3170 len 24576
34709 278/300 Write block 3052 len 8192 data: 4 5
34710 279/300 Write block 2131 len 4096 data: 7
34711 280/300 Read block 2017 len 32768
34712 281/300 Write block 2798 len 16384 data: 6 7 7 7
34713 282/300 Write block 4926 len 28672 data: 8 8 6 5 6 6 4
34714 283/300 Write block 1277 len 4096 data: 4
34715 284/300 Write block 2440 len 32768 data: 6 6 8 7 7 7 7 8
34716 285/300 Write block 1485 len 32768 data: 9 9 8 10 11 12 10 10
34717 286/300 Read block 2267 len 28672
34718 287/300 Write block 1201 len 36864 data: 5 5 6 5 6 5 5 5 4
34719 288/300 Write block 1333 len 20480 data: 4 4 6 6 7
34720 289/300 Read block 337 len 12288
34721 290/300 Read block 597 len 16384
34722 291/300 Read block 3258 len 28672
34723 292/300 Read block 3795 len 4096
34724 293/300 Write block 2514 len 12288 data: 5 4 5
34725 294/300 Read block 3358 len 4096
34726 295/300 Read block 4464 len 24576
34727 296/300 Read block 4926 len 12288
34728 297/300 Write block 1616 len 36864 data: 5 5 4 4 5 5 4 4 7
34729 298/300 Read block 4546 len 32768
34730 299/300 Write block 2896 len 4096 data: 6
34731 300/300 Read block 103 len 40960
34732 301/300 Read block 4363 len 36864
34733 Main task has work to do, go find it
34734 got dsc Start(0)
34735 start 0
34736 Feb 28 06:08:43.430 INFO request completed, latency_us: 183, response_code: 204, uri: /start/cid/0, method: POST, req_id: 9906bd83-49c1-4854-85a3-3ca7f28651e4, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
34737 Replay: started 0, returned:Ok(())
34738 ----------------------------------------------------------------
34739 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34740 States: Offline Active Active
34741 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34742 9754 Acked Flush 0 Sent Done Done false
34743 9755 Acked Flush 0 Sent Done Done false
34744 9756 Acked Read 9 Sent Done Done false
34745 9757 Acked Write 9 Sent Done Done false
34746 9758 Acked Read 8 Sent Done Done false
34747 9759 Acked Write 7 Sent Done Done false
34748 9760 Acked Write 10 Sent Done Done false
34749 9761 Acked Write 1 Sent Done Done false
34750 9762 Acked Write 8 Sent Done Done false
34751 9763 Acked Flush 0 Sent Done Done false
34752 9764 Acked Write 7 Sent Done Done false
34753 9765 Acked Read 6 Sent Done Done false
34754 9766 Acked Write 2 Sent Done Done false
34755 9767 Acked Read 1 Sent Done Done false
34756 9768 Acked Write 3 Sent Done Done false
34757 9769 Acked Read 6 Sent Done Done false
34758 9770 Acked Write 8 Sent Done Done false
34759 9771 Acked Read 7 Sent Done Done false
34760 9772 Acked Write 4 Sent Done Done false
34761 9773 Acked Write 10 Sent Done Done false
34762 9774 Acked Read 4 Sent Done Done false
34763 9775 Acked Write 7 Sent Done Done false
34764 9776 Acked Write 1 Sent Done Done false
34765 9777 Acked Write 10 Sent Done Done false
34766 9778 Acked Read 9 Sent Done Done false
34767 9779 Acked Read 5 Sent Done Done false
34768 9780 Acked Write 2 Sent Done Done false
34769 9781 Acked Read 2 Sent Done Done false
34770 9782 Acked Flush 0 Sent Done Done false
34771 9783 Acked Read 4 Sent Done Done false
34772 9784 Acked Read 7 Sent Done Done false
34773 9785 Acked Read 6 Sent Done Done false
34774 9786 Acked Write 7 Sent Done Done false
34775 9787 Acked Read 1 Sent Done Done false
34776 9788 Acked Write 6 Sent Done Done false
34777 9789 Acked Flush 0 Sent Done Done false
34778 9790 Acked Read 10 Sent Done Done false
34779 9791 Acked Write 4 Sent Done Done false
34780 9792 Acked Read 9 Sent Done Done false
34781 9793 Acked Write 7 Sent Done Done false
34782 9794 Acked Read 7 Sent Done Done false
34783 9795 Acked Write 6 Sent Done Done false
34784 9796 Acked Read 2 Sent Done Done false
34785 9797 Acked Read 5 Sent Done Done false
34786 9798 Acked Write 2 Sent Done Done false
34787 9799 Acked Read 9 Sent Done Done false
34788 9800 Acked Read 7 Sent Done Done false
34789 9801 Acked Read 3 Sent Done Done false
34790 9802 Acked Write 9 Sent Done Done false
34791 9803 Acked Write 8 Sent Done Done false
34792 9804 Acked Read 10 Sent Done Done false
34793 9805 Acked Write 8 Sent Done Done false
34794 9806 Acked Write 3 Sent Done Done false
34795 9807 Acked Write 7 Sent Done Done false
34796 9808 Acked Write 8 Sent Done Done false
34797 9809 Acked Write 1 Sent Done Done false
34798 9810 Acked Write 8 Sent Done Done false
34799 9811 Acked Write 2 Sent Done Done false
34800 9812 Acked Read 1 Sent Done Done false
34801 9813 Acked Read 2 Sent Done Done false
34802 9814 Acked Flush 0 Sent Done Done false
34803 9815 Acked Write 1 Sent Done Done false
34804 9816 Acked Read 6 Sent Done Done false
34805 9817 Acked Read 7 Sent Done Done false
34806 9818 Acked Read 2 Sent Done Done false
34807 9819 Acked Write 5 Sent Done Done false
34808 9820 Acked Read 4 Sent Done Done false
34809 9821 Acked Write 9 Sent Done Done false
34810 9822 Acked Read 7 Sent Done Done false
34811 9823 Acked Write 7 Sent Done Done false
34812 9824 Acked Read 8 Sent Done Done false
34813 9825 Acked Read 4 Sent Done Done false
34814 9826 Acked Read 7 Sent Done Done false
34815 9827 Acked Read 6 Sent Done Done false
34816 9828 Acked Read 10 Sent Done Done false
34817 9829 Acked Write 9 Sent Done Done false
34818 9830 Acked Write 10 Sent Done Done false
34819 9831 Acked Read 3 Sent Done Done false
34820 9832 Acked Write 8 Sent Done Done false
34821 9833 Acked Write 6 Sent Done Done false
34822 9834 Acked Read 10 Sent Done Done false
34823 9835 Acked Read 4 Sent Done Done false
34824 9836 Acked Read 7 Sent Done Done false
34825 9837 Acked Write 9 Sent Done Done false
34826 9838 Acked Write 7 Sent Done Done false
34827 9839 Acked Write 10 Sent Done Done false
34828 9840 Acked Flush 0 Sent Done Done false
34829 9841 Acked Read 6 Sent Done Done false
34830 9842 Acked Write 1 Sent Done Done false
34831 9843 Acked Write 10 Sent Done Done false
34832 9844 Acked Write 1 Sent Done Done false
34833 9845 Acked Read 6 Sent Done Done false
34834 9846 Acked Read 2 Sent Done Done false
34835 9847 Acked Read 3 Sent Done Done false
34836 9848 Acked Read 10 Sent Done Done false
34837 9849 Acked Read 7 Sent Done Done false
34838 9850 Acked Flush 0 Sent Done Done false
34839 9851 Acked Flush 0 Sent Done Done false
34840 9852 Acked Write 6 Sent Done Done false
34841 9853 Acked Read 5 Sent Done Done false
34842 9854 Acked Write 5 Sent Done Done false
34843 9855 Acked Read 6 Sent Done Done false
34844 9856 Acked Read 5 Sent Done Done false
34845 9857 Acked Write 8 Sent Done Done false
34846 9858 Acked Read 6 Sent Done Done false
34847 9859 Acked Read 6 Sent Done Done false
34848 9860 Acked Read 7 Sent Done Done false
34849 9861 Acked Flush 0 Sent Done Done false
34850 9862 Acked Read 1 Sent Done Done false
34851 9863 Acked Read 7 Sent Done Done false
34852 9864 Acked Flush 0 Sent Done Done false
34853 9865 Acked Read 6 Sent Done Done false
34854 9866 Acked Read 10 Sent Done Done false
34855 9867 Acked Read 10 Sent Done Done false
34856 9868 Acked Read 7 Sent Done Done false
34857 9869 Acked Read 3 Sent Done Done false
34858 9870 Acked Write 6 Sent Done Done false
34859 9871 Acked Write 8 Sent Done Done false
34860 9872 Acked Read 9 Sent Done Done false
34861 9873 Acked Write 5 Sent Done Done false
34862 9874 Acked Write 9 Sent Done Done false
34863 9875 Acked Read 4 Sent Done Done false
34864 9876 Acked Read 8 Sent Done Done false
34865 9877 Acked Flush 0 Sent Done Done false
34866 9878 Acked Write 8 Sent Done Done false
34867 9879 Acked Flush 0 Sent Done Done false
34868 9880 Acked Write 9 Sent Done Done false
34869 9881 Acked Read 7 Sent Done Done false
34870 9882 Acked Flush 0 Sent Done Done false
34871 9883 Acked Write 5 Sent Done Done false
34872 9884 Acked Read 6 Sent Done Done false
34873 9885 Acked Write 7 Sent Done Done false
34874 9886 Acked Read 4 Sent Done Done false
34875 9887 Acked Write 7 Sent Done Done false
34876 9888 Acked Read 9 Sent Done Done false
34877 9889 Acked Write 1 Sent Done Done false
34878 9890 Acked Read 1 Sent Done Done false
34879 9891 Acked Flush 0 Sent Done Done false
34880 9892 Acked Read 7 Sent Done Done false
34881 9893 Acked Write 7 Sent Done Done false
34882 9894 Acked Write 9 Sent Done Done false
34883 9895 Acked Read 9 Sent Done Done false
34884 9896 Acked Read 9 Sent Done Done false
34885 9897 Acked Write 5 Sent Done Done false
34886 9898 Acked Read 7 Sent Done Done false
34887 9899 Acked Flush 0 Sent Done Done false
34888 9900 Acked Read 3 Sent Done Done false
34889 9901 Acked Write 5 Sent Done Done false
34890 9902 Acked Read 8 Sent Done Done false
34891 9903 Acked Read 1 Sent Done Done false
34892 9904 Acked Write 1 Sent Done Done false
34893 9905 Acked Write 5 Sent Done Done false
34894 9906 Acked Write 8 Sent Done Done false
34895 9907 Acked Flush 0 Sent Done Done false
34896 9908 Acked Read 9 Sent Done Done false
34897 9909 Acked Read 4 Sent Done Done false
34898 9910 Acked Flush 0 Sent Done Done false
34899 9911 Acked Flush 0 Sent Done Done false
34900 9912 Acked Read 9 Sent Done Done false
34901 9913 Acked Write 2 Sent Done Done false
34902 9914 Acked Read 10 Sent Done Done false
34903 9915 Acked Write 9 Sent Done Done false
34904 9916 Acked Read 3 Sent Done Done false
34905 9917 Acked Write 7 Sent Done Done false
34906 9918 Acked Write 9 Sent Done Done false
34907 9919 Acked Read 8 Sent Done Done false
34908 9920 Acked Read 2 Sent Done Done false
34909 9921 Acked Read 4 Sent Done Done false
34910 9922 Acked Read 1 Sent Done Done false
34911 9923 Acked Write 3 Sent Done Done false
34912 9924 Acked Flush 0 Sent Done Done false
34913 9925 Acked Read 1 Sent Done Done false
34914 9926 Acked Write 4 Sent Done Done false
34915 9927 Acked Read 3 Sent Done Done false
34916 9928 Acked Write 1 Sent Done Done false
34917 9929 Acked Read 1 Sent Done Done false
34918 9930 Acked Write 3 Sent Done Done false
34919 9931 Acked Read 8 Sent Done Done false
34920 9932 Acked Read 10 Sent Done Done false
34921 9933 Acked Read 3 Sent Done Done false
34922 9934 Acked Read 5 Sent Done Done false
34923 9935 Acked Write 2 Sent Done Done false
34924 9936 Acked Write 8 Sent Done Done false
34925 9937 Acked Write 8 Sent Done Done false
34926 9938 Acked Write 10 Sent Done Done false
34927 9939 Acked Write 5 Sent Done Done false
34928 9940 Acked Read 8 Sent Done Done false
34929 9941 Acked Read 7 Sent Done Done false
34930 9942 Acked Read 6 Sent Done Done false
34931 9943 Acked Read 5 Sent Done Done false
34932 9944 Acked Read 8 Sent Done Done false
34933 9945 Acked Write 5 Sent Done Done false
34934 9946 Acked Write 9 Sent Done Done false
34935 9947 Acked Write 9 Sent Done Done false
34936 9948 Acked Write 7 Sent Done Done false
34937 9949 Acked Read 2 Sent Done Done false
34938 9950 Acked Write 6 Sent Done Done false
34939 9951 Acked Read 3 Sent Done Done false
34940 9952 Acked Write 6 Sent Done Done false
34941 9953 Acked Write 8 Sent Done Done false
34942 9954 Acked Read 9 Sent Done Done false
34943 9955 Acked Write 1 Sent Done Done false
34944 9956 Acked Write 3 Sent Done Done false
34945 9957 Acked Read 6 Sent Done Done false
34946 9958 Acked Write 3 Sent Done Done false
34947 9959 Acked Flush 0 Sent Done Done false
34948 9960 Acked Write 6 Sent Done Done false
34949 9961 Acked Flush 0 Sent Done Done false
34950 9962 Acked Flush 0 Sent Done Done false
34951 9963 Acked Write 4 Sent Done Done false
34952 9964 Acked Read 1 Sent Done Done false
34953 9965 Acked Write 8 Sent Done Done false
34954 9966 Acked Read 4 Sent Done Done false
34955 9967 Acked Read 9 Sent Done Done false
34956 9968 Acked Write 9 Sent Done Done false
34957 9969 Acked Read 5 Sent Done Done false
34958 9970 Acked Write 6 Sent Done Done false
34959 9971 Acked Read 4 Sent Done Done false
34960 9972 Acked Write 7 Sent Done Done false
34961 9973 Acked Write 10 Sent Done Done false
34962 9974 Acked Flush 0 Sent Done Done false
34963 9975 Acked Read 3 Sent Done Done false
34964 9976 Acked Read 3 Sent Done Done false
34965 9977 Acked Read 7 Sent Done Done false
34966 9978 Acked Read 9 Sent Done Done false
34967 9979 Acked Write 10 Sent Done Done false
34968 9980 Acked Read 9 Sent Done Done false
34969 9981 Acked Read 8 Sent Done Done false
34970 9982 Acked Write 9 Sent Done Done false
34971 9983 Acked Write 9 Sent Done Done false
34972 9984 Acked Read 8 Sent Done Done false
34973 9985 Acked Write 1 Sent Done Done false
34974 9986 Acked Flush 0 Sent Done Done false
34975 9987 Acked Read 6 Sent Done Done false
34976 9988 Acked Read 6 Sent Done Done false
34977 9989 Acked Write 3 Sent Done Done false
34978 9990 Acked Read 7 Sent Done Done false
34979 9991 Acked Read 2 Sent Done Done false
34980 9992 Acked Read 10 Sent Done Done false
34981 9993 Acked Read 4 Sent Done Done false
34982 9994 Acked Write 3 Sent Done Done false
34983 9995 Acked Write 9 Sent Done Done false
34984 9996 Acked Write 10 Sent Done Done false
34985 9997 Acked Read 7 Sent Done Done false
34986 9998 Acked Read 4 Sent Done Done false
34987 9999 Acked Read 9 Sent Done Done false
34988 10000 Acked Read 2 Sent Done Done false
34989 10001 Acked Flush 0 Sent Done Done false
34990 10002 Acked Write 7 Sent Done Done false
34991 10003 Acked Read 4 Sent Done Done false
34992 10004 Acked Read 2 Sent Done Done false
34993 10005 Acked Write 4 Sent Done Done false
34994 10006 Acked Write 8 Sent Done Done false
34995 10007 Acked Write 7 Sent Done Done false
34996 10008 Acked Flush 0 Sent Done Done false
34997 10009 Acked Flush 0 Sent Done Done false
34998 10010 Acked Flush 0 Sent Done Done false
34999 10011 Acked Write 8 Sent Done Done false
35000 10012 Acked Read 3 Sent Done Done false
35001 10013 Acked Read 10 Sent Done Done false
35002 10014 Acked Write 2 Sent Done Done false
35003 10015 Acked Read 5 Sent Done Done false
35004 10016 Acked Flush 0 Sent Done Done false
35005 10017 Acked Read 6 Sent Done Done false
35006 10018 Acked Read 7 Sent Done Done false
35007 10019 Acked Read 4 Sent Done Done false
35008 10020 Acked Write 5 Sent Done Done false
35009 10021 Acked Read 5 Sent Done Done false
35010 10022 Acked Flush 0 Sent Done Done false
35011 10023 Acked Read 9 Sent Done Done false
35012 10024 Acked Read 1 Sent Done Done false
35013 10025 Acked Flush 0 Sent Done Done false
35014 10026 Acked Read 4 Sent Done Done false
35015 10027 Acked Read 9 Sent Done Done false
35016 10028 Acked Write 3 Sent Done Done false
35017 10029 Acked Write 7 Sent Done Done false
35018 10030 Acked Read 6 Sent Done Done false
35019 10031 Acked Write 2 Sent Done Done false
35020 10032 Acked Write 1 Sent Done Done false
35021 10033 Acked Read 8 Sent Done Done false
35022 10034 Acked Write 4 Sent Done Done false
35023 10035 Acked Write 7 Sent Done Done false
35024 10036 Acked Write 1 Sent Done Done false
35025 10037 Acked Write 8 Sent Done Done false
35026 10038 Acked Write 8 Sent Done Done false
35027 10039 Acked Read 7 Sent Done Done false
35028 10040 Acked Write 9 Sent Done Done false
35029 10041 Acked Write 5 Sent Done Done false
35030 10042 Acked Read 3 Sent Done Done false
35031 10043 Acked Read 4 Sent Done Done false
35032 10044 Acked Read 7 Sent Done Done false
35033 10045 Acked Read 1 Sent Done Done false
35034 10046 Acked Write 3 Sent Done Done false
35035 10047 Acked Read 1 Sent Done Done false
35036 10048 Acked Read 6 Sent Done Done false
35037 10049 Acked Read 3 Sent Done Done false
35038 10050 Acked Write 9 Sent Done Done false
35039 10051 Acked Read 8 Sent Done Done false
35040 10052 Acked Write 1 Sent Done Done false
35041 10053 Acked Read 10 Sent Done Done false
35042 10054 Acked Read 9 Sent Done Done false
35043 STATES DS:0 DS:1 DS:2 TOTAL
35044 Sent 301 0 0 301
35045 Done 0 301 301 602
35046 Skipped 0 0 0 0
35047 Error 0 0 0 0
35048 Last Flush: Some(JobId(9753)) Some(JobId(10025)) Some(JobId(10025))
35049 Downstairs last five completed: 9753 9752 9751 9750 9749
35050 Upstairs last five completed: 10054 10053 10052 10051 10050
35051 CLIENT: Up:0 ds:301 act:2
350522025-02-28T06:08:44.111ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
350532025-02-28T06:08:45.112ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
350542025-02-28T06:08:46.115ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
35055 [8810] got start action
35056 [8810] I am going to restart
35057 Starting downstairs at port 8810
35058 Make output file at "/tmp/dsc/downstairs-8810.txt"
35059 [8810][0] reports Starting
35060 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1241)
35061 [8810][0] reports Running
350622025-02-28T06:08:47.427ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
35063 ----------------------------------------------------------------
35064 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35065 States: Offline Active Active
35066 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35067 9754 Acked Flush 0 Sent Done Done false
35068 9755 Acked Flush 0 Sent Done Done false
35069 9756 Acked Read 9 Sent Done Done false
35070 9757 Acked Write 9 Sent Done Done false
35071 9758 Acked Read 8 Sent Done Done false
35072 9759 Acked Write 7 Sent Done Done false
35073 9760 Acked Write 10 Sent Done Done false
35074 9761 Acked Write 1 Sent Done Done false
35075 9762 Acked Write 8 Sent Done Done false
35076 9763 Acked Flush 0 Sent Done Done false
35077 9764 Acked Write 7 Sent Done Done false
35078 9765 Acked Read 6 Sent Done Done false
35079 9766 Acked Write 2 Sent Done Done false
35080 9767 Acked Read 1 Sent Done Done false
35081 9768 Acked Write 3 Sent Done Done false
35082 9769 Acked Read 6 Sent Done Done false
35083 9770 Acked Write 8 Sent Done Done false
35084 9771 Acked Read 7 Sent Done Done false
35085 9772 Acked Write 4 Sent Done Done false
35086 9773 Acked Write 10 Sent Done Done false
35087 9774 Acked Read 4 Sent Done Done false
35088 9775 Acked Write 7 Sent Done Done false
35089 9776 Acked Write 1 Sent Done Done false
35090 9777 Acked Write 10 Sent Done Done false
35091 9778 Acked Read 9 Sent Done Done false
35092 9779 Acked Read 5 Sent Done Done false
35093 9780 Acked Write 2 Sent Done Done false
35094 9781 Acked Read 2 Sent Done Done false
35095 9782 Acked Flush 0 Sent Done Done false
35096 9783 Acked Read 4 Sent Done Done false
35097 9784 Acked Read 7 Sent Done Done false
35098 9785 Acked Read 6 Sent Done Done false
35099 9786 Acked Write 7 Sent Done Done false
35100 9787 Acked Read 1 Sent Done Done false
35101 9788 Acked Write 6 Sent Done Done false
35102 9789 Acked Flush 0 Sent Done Done false
35103 9790 Acked Read 10 Sent Done Done false
35104 9791 Acked Write 4 Sent Done Done false
35105 9792 Acked Read 9 Sent Done Done false
35106 9793 Acked Write 7 Sent Done Done false
35107 9794 Acked Read 7 Sent Done Done false
35108 9795 Acked Write 6 Sent Done Done false
35109 9796 Acked Read 2 Sent Done Done false
35110 9797 Acked Read 5 Sent Done Done false
35111 9798 Acked Write 2 Sent Done Done false
35112 9799 Acked Read 9 Sent Done Done false
35113 9800 Acked Read 7 Sent Done Done false
35114 9801 Acked Read 3 Sent Done Done false
35115 9802 Acked Write 9 Sent Done Done false
35116 9803 Acked Write 8 Sent Done Done false
35117 9804 Acked Read 10 Sent Done Done false
35118 9805 Acked Write 8 Sent Done Done false
35119 9806 Acked Write 3 Sent Done Done false
35120 9807 Acked Write 7 Sent Done Done false
35121 9808 Acked Write 8 Sent Done Done false
35122 9809 Acked Write 1 Sent Done Done false
35123 9810 Acked Write 8 Sent Done Done false
35124 9811 Acked Write 2 Sent Done Done false
35125 9812 Acked Read 1 Sent Done Done false
35126 9813 Acked Read 2 Sent Done Done false
35127 9814 Acked Flush 0 Sent Done Done false
35128 9815 Acked Write 1 Sent Done Done false
35129 9816 Acked Read 6 Sent Done Done false
35130 9817 Acked Read 7 Sent Done Done false
35131 9818 Acked Read 2 Sent Done Done false
35132 9819 Acked Write 5 Sent Done Done false
35133 9820 Acked Read 4 Sent Done Done false
35134 9821 Acked Write 9 Sent Done Done false
35135 9822 Acked Read 7 Sent Done Done false
35136 9823 Acked Write 7 Sent Done Done false
35137 9824 Acked Read 8 Sent Done Done false
35138 9825 Acked Read 4 Sent Done Done false
35139 9826 Acked Read 7 Sent Done Done false
35140 9827 Acked Read 6 Sent Done Done false
35141 9828 Acked Read 10 Sent Done Done false
35142 9829 Acked Write 9 Sent Done Done false
35143 9830 Acked Write 10 Sent Done Done false
35144 9831 Acked Read 3 Sent Done Done false
35145 9832 Acked Write 8 Sent Done Done false
35146 9833 Acked Write 6 Sent Done Done false
35147 9834 Acked Read 10 Sent Done Done false
35148 9835 Acked Read 4 Sent Done Done false
35149 9836 Acked Read 7 Sent Done Done false
35150 9837 Acked Write 9 Sent Done Done false
35151 9838 Acked Write 7 Sent Done Done false
35152 9839 Acked Write 10 Sent Done Done false
35153 9840 Acked Flush 0 Sent Done Done false
35154 9841 Acked Read 6 Sent Done Done false
35155 9842 Acked Write 1 Sent Done Done false
35156 9843 Acked Write 10 Sent Done Done false
35157 9844 Acked Write 1 Sent Done Done false
35158 9845 Acked Read 6 Sent Done Done false
35159 9846 Acked Read 2 Sent Done Done false
35160 9847 Acked Read 3 Sent Done Done false
35161 9848 Acked Read 10 Sent Done Done false
35162 9849 Acked Read 7 Sent Done Done false
35163 9850 Acked Flush 0 Sent Done Done false
35164 9851 Acked Flush 0 Sent Done Done false
35165 9852 Acked Write 6 Sent Done Done false
35166 9853 Acked Read 5 Sent Done Done false
35167 9854 Acked Write 5 Sent Done Done false
35168 9855 Acked Read 6 Sent Done Done false
35169 9856 Acked Read 5 Sent Done Done false
35170 9857 Acked Write 8 Sent Done Done false
35171 9858 Acked Read 6 Sent Done Done false
35172 9859 Acked Read 6 Sent Done Done false
35173 9860 Acked Read 7 Sent Done Done false
35174 9861 Acked Flush 0 Sent Done Done false
35175 9862 Acked Read 1 Sent Done Done false
35176 9863 Acked Read 7 Sent Done Done false
35177 9864 Acked Flush 0 Sent Done Done false
35178 9865 Acked Read 6 Sent Done Done false
35179 9866 Acked Read 10 Sent Done Done false
35180 9867 Acked Read 10 Sent Done Done false
35181 9868 Acked Read 7 Sent Done Done false
35182 9869 Acked Read 3 Sent Done Done false
35183 9870 Acked Write 6 Sent Done Done false
35184 9871 Acked Write 8 Sent Done Done false
35185 9872 Acked Read 9 Sent Done Done false
35186 9873 Acked Write 5 Sent Done Done false
35187 9874 Acked Write 9 Sent Done Done false
35188 9875 Acked Read 4 Sent Done Done false
35189 9876 Acked Read 8 Sent Done Done false
35190 9877 Acked Flush 0 Sent Done Done false
35191 9878 Acked Write 8 Sent Done Done false
35192 9879 Acked Flush 0 Sent Done Done false
35193 9880 Acked Write 9 Sent Done Done false
35194 9881 Acked Read 7 Sent Done Done false
35195 9882 Acked Flush 0 Sent Done Done false
35196 9883 Acked Write 5 Sent Done Done false
35197 9884 Acked Read 6 Sent Done Done false
35198 9885 Acked Write 7 Sent Done Done false
35199 9886 Acked Read 4 Sent Done Done false
35200 9887 Acked Write 7 Sent Done Done false
35201 9888 Acked Read 9 Sent Done Done false
35202 9889 Acked Write 1 Sent Done Done false
35203 9890 Acked Read 1 Sent Done Done false
35204 9891 Acked Flush 0 Sent Done Done false
35205 9892 Acked Read 7 Sent Done Done false
35206 9893 Acked Write 7 Sent Done Done false
35207 9894 Acked Write 9 Sent Done Done false
35208 9895 Acked Read 9 Sent Done Done false
35209 9896 Acked Read 9 Sent Done Done false
35210 9897 Acked Write 5 Sent Done Done false
35211 9898 Acked Read 7 Sent Done Done false
35212 9899 Acked Flush 0 Sent Done Done false
35213 9900 Acked Read 3 Sent Done Done false
35214 9901 Acked Write 5 Sent Done Done false
35215 9902 Acked Read 8 Sent Done Done false
35216 9903 Acked Read 1 Sent Done Done false
35217 9904 Acked Write 1 Sent Done Done false
35218 9905 Acked Write 5 Sent Done Done false
35219 9906 Acked Write 8 Sent Done Done false
35220 9907 Acked Flush 0 Sent Done Done false
35221 9908 Acked Read 9 Sent Done Done false
35222 9909 Acked Read 4 Sent Done Done false
35223 9910 Acked Flush 0 Sent Done Done false
35224 9911 Acked Flush 0 Sent Done Done false
35225 9912 Acked Read 9 Sent Done Done false
35226 9913 Acked Write 2 Sent Done Done false
35227 9914 Acked Read 10 Sent Done Done false
35228 9915 Acked Write 9 Sent Done Done false
35229 9916 Acked Read 3 Sent Done Done false
35230 9917 Acked Write 7 Sent Done Done false
35231 9918 Acked Write 9 Sent Done Done false
35232 9919 Acked Read 8 Sent Done Done false
35233 9920 Acked Read 2 Sent Done Done false
35234 9921 Acked Read 4 Sent Done Done false
35235 9922 Acked Read 1 Sent Done Done false
35236 9923 Acked Write 3 Sent Done Done false
35237 9924 Acked Flush 0 Sent Done Done false
35238 9925 Acked Read 1 Sent Done Done false
35239 9926 Acked Write 4 Sent Done Done false
35240 9927 Acked Read 3 Sent Done Done false
35241 9928 Acked Write 1 Sent Done Done false
35242 9929 Acked Read 1 Sent Done Done false
35243 9930 Acked Write 3 Sent Done Done false
35244 9931 Acked Read 8 Sent Done Done false
35245 9932 Acked Read 10 Sent Done Done false
35246 9933 Acked Read 3 Sent Done Done false
35247 9934 Acked Read 5 Sent Done Done false
35248 9935 Acked Write 2 Sent Done Done false
35249 9936 Acked Write 8 Sent Done Done false
35250 9937 Acked Write 8 Sent Done Done false
35251 9938 Acked Write 10 Sent Done Done false
35252 9939 Acked Write 5 Sent Done Done false
35253 9940 Acked Read 8 Sent Done Done false
35254 9941 Acked Read 7 Sent Done Done false
35255 9942 Acked Read 6 Sent Done Done false
35256 9943 Acked Read 5 Sent Done Done false
35257 9944 Acked Read 8 Sent Done Done false
35258 9945 Acked Write 5 Sent Done Done false
35259 9946 Acked Write 9 Sent Done Done false
35260 9947 Acked Write 9 Sent Done Done false
35261 9948 Acked Write 7 Sent Done Done false
35262 9949 Acked Read 2 Sent Done Done false
35263 9950 Acked Write 6 Sent Done Done false
35264 9951 Acked Read 3 Sent Done Done false
35265 9952 Acked Write 6 Sent Done Done false
35266 9953 Acked Write 8 Sent Done Done false
35267 9954 Acked Read 9 Sent Done Done false
35268 9955 Acked Write 1 Sent Done Done false
35269 9956 Acked Write 3 Sent Done Done false
35270 9957 Acked Read 6 Sent Done Done false
35271 9958 Acked Write 3 Sent Done Done false
35272 9959 Acked Flush 0 Sent Done Done false
35273 9960 Acked Write 6 Sent Done Done false
35274 9961 Acked Flush 0 Sent Done Done false
35275 9962 Acked Flush 0 Sent Done Done false
35276 9963 Acked Write 4 Sent Done Done false
35277 9964 Acked Read 1 Sent Done Done false
35278 9965 Acked Write 8 Sent Done Done false
35279 9966 Acked Read 4 Sent Done Done false
35280 9967 Acked Read 9 Sent Done Done false
35281 9968 Acked Write 9 Sent Done Done false
35282 9969 Acked Read 5 Sent Done Done false
35283 9970 Acked Write 6 Sent Done Done false
35284 9971 Acked Read 4 Sent Done Done false
35285 9972 Acked Write 7 Sent Done Done false
35286 9973 Acked Write 10 Sent Done Done false
35287 9974 Acked Flush 0 Sent Done Done false
35288 9975 Acked Read 3 Sent Done Done false
35289 9976 Acked Read 3 Sent Done Done false
35290 9977 Acked Read 7 Sent Done Done false
35291 9978 Acked Read 9 Sent Done Done false
35292 9979 Acked Write 10 Sent Done Done false
35293 9980 Acked Read 9 Sent Done Done false
35294 9981 Acked Read 8 Sent Done Done false
35295 9982 Acked Write 9 Sent Done Done false
35296 9983 Acked Write 9 Sent Done Done false
35297 9984 Acked Read 8 Sent Done Done false
35298 9985 Acked Write 1 Sent Done Done false
35299 9986 Acked Flush 0 Sent Done Done false
35300 9987 Acked Read 6 Sent Done Done false
35301 9988 Acked Read 6 Sent Done Done false
35302 9989 Acked Write 3 Sent Done Done false
35303 9990 Acked Read 7 Sent Done Done false
35304 9991 Acked Read 2 Sent Done Done false
35305 9992 Acked Read 10 Sent Done Done false
35306 9993 Acked Read 4 Sent Done Done false
35307 9994 Acked Write 3 Sent Done Done false
35308 9995 Acked Write 9 Sent Done Done false
35309 9996 Acked Write 10 Sent Done Done false
35310 9997 Acked Read 7 Sent Done Done false
35311 9998 Acked Read 4 Sent Done Done false
35312 9999 Acked Read 9 Sent Done Done false
35313 10000 Acked Read 2 Sent Done Done false
35314 10001 Acked Flush 0 Sent Done Done false
35315 10002 Acked Write 7 Sent Done Done false
35316 10003 Acked Read 4 Sent Done Done false
35317 10004 Acked Read 2 Sent Done Done false
35318 10005 Acked Write 4 Sent Done Done false
35319 10006 Acked Write 8 Sent Done Done false
35320 10007 Acked Write 7 Sent Done Done false
35321 10008 Acked Flush 0 Sent Done Done false
35322 10009 Acked Flush 0 Sent Done Done false
35323 10010 Acked Flush 0 Sent Done Done false
35324 10011 Acked Write 8 Sent Done Done false
35325 10012 Acked Read 3 Sent Done Done false
35326 10013 Acked Read 10 Sent Done Done false
35327 10014 Acked Write 2 Sent Done Done false
35328 10015 Acked Read 5 Sent Done Done false
35329 10016 Acked Flush 0 Sent Done Done false
35330 10017 Acked Read 6 Sent Done Done false
35331 10018 Acked Read 7 Sent Done Done false
35332 10019 Acked Read 4 Sent Done Done false
35333 10020 Acked Write 5 Sent Done Done false
35334 10021 Acked Read 5 Sent Done Done false
35335 10022 Acked Flush 0 Sent Done Done false
35336 10023 Acked Read 9 Sent Done Done false
35337 10024 Acked Read 1 Sent Done Done false
35338 10025 Acked Flush 0 Sent Done Done false
35339 10026 Acked Read 4 Sent Done Done false
35340 10027 Acked Read 9 Sent Done Done false
35341 10028 Acked Write 3 Sent Done Done false
35342 10029 Acked Write 7 Sent Done Done false
35343 10030 Acked Read 6 Sent Done Done false
35344 10031 Acked Write 2 Sent Done Done false
35345 10032 Acked Write 1 Sent Done Done false
35346 10033 Acked Read 8 Sent Done Done false
35347 10034 Acked Write 4 Sent Done Done false
35348 10035 Acked Write 7 Sent Done Done false
35349 10036 Acked Write 1 Sent Done Done false
35350 10037 Acked Write 8 Sent Done Done false
35351 10038 Acked Write 8 Sent Done Done false
35352 10039 Acked Read 7 Sent Done Done false
35353 10040 Acked Write 9 Sent Done Done false
35354 10041 Acked Write 5 Sent Done Done false
35355 10042 Acked Read 3 Sent Done Done false
35356 10043 Acked Read 4 Sent Done Done false
35357 10044 Acked Read 7 Sent Done Done false
35358 10045 Acked Read 1 Sent Done Done false
35359 10046 Acked Write 3 Sent Done Done false
35360 10047 Acked Read 1 Sent Done Done false
35361 10048 Acked Read 6 Sent Done Done false
35362 10049 Acked Read 3 Sent Done Done false
35363 10050 Acked Write 9 Sent Done Done false
35364 10051 Acked Read 8 Sent Done Done false
35365 10052 Acked Write 1 Sent Done Done false
35366 10053 Acked Read 10 Sent Done Done false
35367 10054 Acked Read 9 Sent Done Done false
35368 STATES DS:0 DS:1 DS:2 TOTAL
35369 Sent 301 0 0 301
35370 Done 0 301 301 602
35371 Skipped 0 0 0 0
35372 Error 0 0 0 0
35373 Last Flush: Some(JobId(9753)) Some(JobId(10025)) Some(JobId(10025))
35374 Downstairs last five completed: 9753 9752 9751 9750 9749
35375 Upstairs last five completed: 10054 10053 10052 10051 10050
35376 CLIENT: Up:0 ds:301 act:2
353772025-02-28T06:08:48.119ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
353782025-02-28T06:08:49.121ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
353792025-02-28T06:08:49.181ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
353802025-02-28T06:08:49.181ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39060) = io task client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
353812025-02-28T06:08:49.183ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
353822025-02-28T06:08:49.183ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
353832025-02-28T06:08:49.183ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
353842025-02-28T06:08:49.183ZINFOcrucible: send last flush ID to this DS: Some(JobId(9753)) = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
353852025-02-28T06:08:49.183ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
353862025-02-28T06:08:49.183ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(9753)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
353872025-02-28T06:08:49.183ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(9753)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
35388 ----------------------------------------------------------------
35389 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35390 Downstairs last five completed: 10055 10054 10053 10052 10051
35391 Upstairs last five completed: 10055 10054 10053 10052 10051
35392 CLIENT: Up:0 ds:0 act:3
35393 Replay: All jobs finished
35394 Main task has work to do, go find it
35395 got dsc Stop(2)
35396 stop 2
35397 [8830] Got stop action so:false kr:false
35398 Feb 28 06:08:51.437 INFO request completed, latency_us: 394, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 2b316f51-2f4c-4d7d-9c53-15dbcb3a6db4, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
35399 Feb 28 06:08:51.438 INFO request completed, latency_us: 109, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9bb5706f-7d0f-4dab-bee5-520bdd8d94c5, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
354002025-02-28T06:08:51.438ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
354012025-02-28T06:08:51.438ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
354022025-02-28T06:08:51.438ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
354032025-02-28T06:08:51.438ZINFOcrucible: client task is exiting = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
35404 [8830] Exited with: ExitStatus(unix_wait_status(9))
35405 [8830][2] reports Exit
35406 Feb 28 06:08:55.440 INFO request completed, latency_us: 327, response_code: 200, uri: /state/cid/2, method: GET, req_id: 33993828-a554-4b03-91b5-7b177879233d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
35407 001/300 Flush
35408 002/300 Flush
35409 003/300 Read block 3032 len 8192
35410 004/300 Read block 4172 len 12288
35411 005/300 Read block 2116 len 28672
35412 006/300 Write block 1301 len 8192 data: 6 5
35413 007/300 Write block 2186 len 12288 data: 9 7 8
35414 008/300 Write block 4399 len 8192 data: 5 5
35415 009/300 Read block 4664 len 12288
35416 010/300 Write block 2229 len 8192 data: 7 9
35417 011/300 Read block 1855 len 16384
35418 012/300 Write block 1468 len 28672 data: 7 8 7 8 7 7 8
35419 013/300 Read block 214 len 24576
35420 014/300 Read block 1157 len 40960
35421 015/300 Write block 2314 len 16384 data: 6 5 5 6
35422 016/300 Read block 4182 len 36864
35423 017/300 Read block 1734 len 24576
35424 018/300 Read block 941 len 12288
35425 019/300 Read block 3642 len 32768
35426 020/300 Read block 3557 len 16384
35427 021/300 Write block 3163 len 28672 data: 3 4 6 6 6 6 6
35428 022/300 Write block 2854 len 20480 data: 7 7 7 7 7
35429 023/300 Read block 1240 len 8192
35430 024/300 Write block 4131 len 20480 data: 5 4 6 5 5
35431 025/300 Flush
35432 026/300 Read block 117 len 16384
35433 027/300 Write block 2481 len 24576 data: 6 5 7 8 8 7
35434 028/300 Read block 676 len 28672
35435 029/300 Read block 680 len 28672
35436 030/300 Write block 3480 len 40960 data: 10 9 8 8 8 6 7 5 5 6
35437 031/300 Flush
35438 032/300 Read block 3994 len 24576
35439 033/300 Flush
35440 034/300 Read block 3025 len 8192
35441 035/300 Read block 583 len 8192
35442 036/300 Read block 1304 len 16384
35443 037/300 Read block 1346 len 20480
35444 038/300 Read block 1237 len 36864
35445 039/300 Read block 3629 len 12288
35446 040/300 Write block 448 len 20480 data: 9 10 9 10 8
35447 041/300 Write block 4556 len 24576 data: 3 4 4 4 4 5
35448 042/300 Read block 1109 len 36864
35449 043/300 Write block 2707 len 24576 data: 7 7 7 5 5 6
35450 044/300 Read block 3536 len 40960
35451 045/300 Write block 4978 len 28672 data: 8 7 6 6 8 7 8
35452 046/300 Write block 4396 len 20480 data: 7 6 5 6 6
35453 047/300 Write block 1117 len 16384 data: 3 4 4 6
35454 048/300 Write block 532 len 36864 data: 5 5 6 5 4 5 4 5 4
35455 049/300 Read block 2656 len 16384
35456 050/300 Read block 4456 len 32768
35457 051/300 Write block 3097 len 40960 data: 6 6 6 6 8 7 9 8 9 8
35458 052/300 Write block 1015 len 4096 data: 6
35459 053/300 Write block 2831 len 36864 data: 3 2 5 4 5 5 6 6 6
35460 054/300 Write block 4340 len 40960 data: 8 7 10 9 9 8 10 11 9 9
35461 055/300 Read block 834 len 4096
35462 056/300 Read block 3441 len 4096
35463 057/300 Flush
35464 058/300 Read block 4932 len 20480
35465 059/300 Read block 2742 len 28672
35466 060/300 Read block 3063 len 4096
35467 061/300 Read block 252 len 8192
35468 062/300 Write block 3121 len 36864 data: 5 5 5 5 5 4 4 3 4
35469 063/300 Write block 1081 len 8192 data: 7 7
35470 064/300 Flush
35471 065/300 Read block 1123 len 12288
35472 066/300 Read block 1145 len 20480
35473 067/300 Read block 2430 len 16384
35474 068/300 Read block 412 len 36864
35475 069/300 Flush
35476 070/300 Read block 497 len 4096
35477 071/300 Read block 1018 len 36864
35478 072/300 Write block 2664 len 12288 data: 10 10 10
35479 073/300 Read block 4788 len 8192
35480 074/300 Flush
35481 075/300 Write block 75 len 32768 data: 11 9 9 8 10 9 11 8
35482 076/300 Read block 4462 len 36864
35483 077/300 Read block 2263 len 36864
35484 078/300 Read block 2260 len 28672
35485 079/300 Read block 2776 len 36864
35486 080/300 Read block 3299 len 28672
35487 081/300 Write block 2141 len 16384 data: 7 7 6 5
35488 082/300 Write block 102 len 16384 data: 4 6 6 5
35489 083/300 Read block 332 len 4096
35490 084/300 Read block 2770 len 16384
35491 085/300 Write block 4479 len 36864 data: 6 5 7 7 7 6 7 8 8
35492 086/300 Write block 2786 len 4096 data: 7
35493 087/300 Read block 2465 len 28672
35494 088/300 Read block 4943 len 8192
35495 089/300 Read block 3267 len 20480
35496 090/300 Read block 4219 len 36864
35497 091/300 Read block 216 len 12288
35498 092/300 Read block 3382 len 36864
35499 093/300 Read block 4498 len 8192
35500 094/300 Read block 3934 len 32768
35501 095/300 Read block 80 len 40960
35502 096/300 Write block 3005 len 16384 data: 3 5 5 5
35503 097/300 Read block 4082 len 12288
35504 098/300 Read block 1994 len 8192
35505 099/300 Write block 2756 len 36864 data: 6 4 6 8 7 6 5 5 5
35506 100/300 Write block 3705 len 40960 data: 6 7 9 7 6 5 5 3 2 3
35507 101/300 Write block 3911 len 8192 data: 5 6
35508 102/300 Flush
35509 103/300 Write block 953 len 28672 data: 5 5 4 4 4 4 4
35510 104/300 Read block 2451 len 28672
35511 105/300 Write block 844 len 16384 data: 7 6 5 2
35512 106/300 Read block 1191 len 8192
35513 107/300 Read block 109 len 28672
35514 108/300 Read block 1961 len 20480
35515 109/300 Read block 2933 len 8192
35516 110/300 Read block 489 len 28672
35517 111/300 Write block 4054 len 12288 data: 4 5 5
35518 112/300 Read block 1046 len 8192
35519 113/300 Write block 4900 len 36864 data: 7 6 7 7 6 5 5 4 4
35520 114/300 Read block 2166 len 8192
35521 115/300 Read block 396 len 36864
35522 116/300 Write block 2116 len 32768 data: 7 7 7 6 6 7 8 8
35523 117/300 Read block 4954 len 20480
35524 118/300 Read block 1114 len 20480
35525 119/300 Read block 1356 len 12288
35526 120/300 Read block 941 len 4096
35527 121/300 Read block 3932 len 8192
35528 122/300 Write block 4033 len 20480 data: 5 5 5 7 8
35529 123/300 Write block 1595 len 36864 data: 4 4 4 2 3 4 4 4 4
35530 124/300 Flush
35531 125/300 Write block 2803 len 32768 data: 6 6 7 7 7 6 6 5
35532 126/300 Write block 3989 len 28672 data: 6 6 6 5 6 7 6
35533 127/300 Write block 1921 len 20480 data: 5 3 3 3 4
35534 128/300 Read block 2032 len 16384
35535 129/300 Read block 1498 len 12288
35536 130/300 Write block 1130 len 36864 data: 8 9 8 7 9 9 10 10 8
35537 131/300 Read block 4785 len 24576
35538 132/300 Read block 3366 len 32768
35539 133/300 Read block 2864 len 32768
35540 134/300 Read block 216 len 28672
35541 135/300 Read block 4868 len 8192
35542 136/300 Read block 442 len 40960
35543 137/300 Write block 79 len 8192 data: 11 10
35544 138/300 Read block 4174 len 36864
35545 139/300 Write block 3664 len 12288 data: 5 5 5
35546 140/300 Read block 1584 len 12288
35547 141/300 Write block 3749 len 16384 data: 8 6 7 7
35548 142/300 Read block 4622 len 20480
35549 143/300 Read block 4359 len 20480
35550 144/300 Read block 2025 len 36864
35551 145/300 Read block 4984 len 40960
35552 146/300 Read block 916 len 36864
35553 147/300 Read block 2842 len 36864
35554 148/300 Write block 4220 len 20480 data: 4 4 3 4 5
35555 149/300 Write block 4057 len 24576 data: 4 6 7 7 5 7
35556 150/300 Read block 66 len 16384
35557 151/300 Read block 3720 len 40960
35558 152/300 Read block 3891 len 8192
35559 153/300 Write block 4504 len 24576 data: 6 6 8 10 8 7
35560 154/300 Read block 3903 len 24576
35561 155/300 Read block 4665 len 4096
35562 156/300 Read block 1944 len 40960
35563 157/300 Write block 3046 len 12288 data: 3 5 5
35564 158/300 Write block 1283 len 4096 data: 7
35565 159/300 Write block 4116 len 12288 data: 8 8 5
35566 160/300 Write block 4951 len 20480 data: 8 6 7 8 7
35567 161/300 Read block 1981 len 36864
35568 162/300 Write block 238 len 36864 data: 8 6 5 5 6 5 6 6 5
35569 163/300 Read block 3394 len 20480
35570 164/300 Read block 1505 len 28672
35571 165/300 Write block 244 len 36864 data: 7 7 6 4 4 5 5 6 5
35572 166/300 Read block 3773 len 20480
35573 167/300 Read block 228 len 24576
35574 168/300 Write block 414 len 4096 data: 10
35575 169/300 Read block 4464 len 20480
35576 170/300 Read block 1154 len 12288
35577 171/300 Read block 4263 len 8192
35578 172/300 Read block 2889 len 40960
35579 173/300 Write block 3605 len 16384 data: 6 6 7 6
35580 174/300 Read block 389 len 20480
35581 175/300 Write block 729 len 12288 data: 2 3 3
35582 176/300 Read block 2779 len 8192
35583 177/300 Read block 966 len 32768
35584 178/300 Write block 1695 len 16384 data: 5 6 6 5
35585 179/300 Write block 4432 len 20480 data: 6 5 4 5 7
35586 180/300 Read block 3144 len 12288
35587 181/300 Read block 4731 len 4096
35588 182/300 Write block 1420 len 40960 data: 6 7 6 6 7 8 8 8 5 3
35589 183/300 Read block 2566 len 20480
35590 184/300 Flush
35591 185/300 Write block 3490 len 24576 data: 4 4 5 6 6 7
35592 186/300 Flush
35593 187/300 Read block 4233 len 36864
35594 188/300 Read block 3781 len 28672
35595 189/300 Write block 2597 len 20480 data: 5 5 6 7 8
35596 190/300 Read block 3348 len 28672
35597 191/300 Read block 2345 len 20480
35598 192/300 Write block 2087 len 24576 data: 10 10 10 10 10 6
35599 193/300 Read block 118 len 36864
35600 194/300 Write block 1633 len 36864 data: 7 6 5 6 5 3 2 3 5
35601 195/300 Read block 1662 len 24576
35602 196/300 Read block 3739 len 8192
35603 197/300 Read block 2352 len 8192
35604 198/300 Write block 3607 len 36864 data: 8 7 5 7 8 8 8 9 8
35605 199/300 Read block 493 len 16384
35606 200/300 Read block 2756 len 28672
35607 201/300 Read block 2321 len 16384
35608 202/300 Read block 2301 len 8192
35609 203/300 Write block 222 len 12288 data: 7 6 5
35610 204/300 Read block 3373 len 40960
35611 205/300 Read block 1640 len 24576
35612 206/300 Write block 4302 len 8192 data: 3 5
35613 207/300 Write block 2417 len 20480 data: 5 6 7 6 6
35614 208/300 Write block 1869 len 4096 data: 10
35615 209/300 Read block 4109 len 8192
35616 210/300 Read block 480 len 24576
35617 211/300 Write block 1810 len 12288 data: 7 7 6
35618 212/300 Write block 3531 len 40960 data: 8 9 9 8 5 7 5 8 7 7
35619 213/300 Read block 894 len 4096
35620 214/300 Read block 760 len 32768
35621 215/300 Read block 3442 len 4096
35622 216/300 Read block 4777 len 28672
35623 217/300 Write block 442 len 4096 data: 12
35624 218/300 Read block 4011 len 32768
35625 219/300 Read block 1488 len 12288
35626 220/300 Write block 238 len 24576 data: 9 7 6 6 7 6
35627 221/300 Write block 3024 len 8192 data: 5 4
35628 222/300 Write block 4962 len 40960 data: 5 6 7 8 6 7 11 11 10 12
35629 223/300 Flush
35630 224/300 Read block 3582 len 8192
35631 225/300 Read block 4633 len 24576
35632 226/300 Write block 3603 len 20480 data: 4 5 7 7 9
35633 227/300 Read block 3142 len 20480
35634 228/300 Read block 304 len 28672
35635 229/300 Write block 2264 len 32768 data: 11 9 9 10 8 7 6 5
35636 230/300 Flush
35637 231/300 Read block 4182 len 4096
35638 232/300 Write block 2046 len 20480 data: 5 4 3 3 3
35639 233/300 Read block 3993 len 12288
35640 234/300 Read block 694 len 4096
35641 235/300 Write block 1161 len 40960 data: 6 6 6 9 8 8 8 7 9 8
35642 236/300 Read block 3069 len 36864
35643 237/300 Flush
35644 238/300 Write block 1163 len 20480 data: 7 10 9 9 9
35645 239/300 Read block 1444 len 36864
35646 240/300 Write block 475 len 32768 data: 8 9 8 8 7 8 8 8
35647 241/300 Read block 4842 len 12288
35648 242/300 Read block 1549 len 16384
35649 243/300 Read block 3752 len 8192
35650 244/300 Read block 2785 len 20480
35651 245/300 Write block 2004 len 36864 data: 7 6 6 5 4 3 3 4 6
35652 246/300 Read block 1944 len 20480
35653 247/300 Read block 4888 len 16384
35654 248/300 Read block 535 len 16384
35655 249/300 Write block 3014 len 12288 data: 6 7 5
35656 250/300 Flush
35657 251/300 Flush
35658 252/300 Read block 902 len 24576
35659 253/300 Read block 1544 len 20480
35660 254/300 Write block 2954 len 36864 data: 6 5 4 5 6 5 5 7 8
35661 255/300 Read block 2015 len 16384
35662 256/300 Write block 692 len 12288 data: 6 5 5
35663 257/300 Flush
35664 258/300 Write block 2506 len 12288 data: 4 4 5
35665 259/300 Write block 4636 len 28672 data: 4 6 6 7 7 5 7
35666 260/300 Write block 4837 len 32768 data: 8 9 8 8 8 9 8 7
35667 261/300 Flush
35668 262/300 Write block 2991 len 24576 data: 9 10 7 7 6 8
35669 263/300 Write block 3726 len 32768 data: 4 3 4 4 4 7 5 6
35670 264/300 Write block 2108 len 28672 data: 7 6 8 9 8 8 7
35671 265/300 Read block 1322 len 36864
35672 266/300 Write block 2329 len 12288 data: 5 4 4
35673 267/300 Write block 2528 len 8192 data: 7 9
35674 268/300 Read block 151 len 16384
35675 269/300 Read block 4141 len 12288
35676 270/300 Write block 840 len 32768 data: 4 5 5 5 8 7 6 3
35677 271/300 Flush
35678 272/300 Read block 619 len 12288
35679 273/300 Read block 285 len 40960
35680 274/300 Read block 2249 len 24576
35681 275/300 Read block 2420 len 16384
35682 276/300 Read block 2389 len 24576
35683 277/300 Read block 4315 len 24576
35684 278/300 Write block 4867 len 28672 data: 5 6 4 4 5 5 5
35685 279/300 Write block 657 len 36864 data: 7 8 8 8 8 10 9 9 9
35686 280/300 Read block 4477 len 12288
35687 281/300 Read block 2639 len 12288
35688 282/300 Write block 2086 len 16384 data: 10 11 11 11
35689 283/300 Read block 4903 len 16384
35690 284/300 Read block 1205 len 28672
35691 285/300 Write block 144 len 20480 data: 7 5 6 6 6
35692 286/300 Read block 683 len 20480
35693 287/300 Read block 157 len 28672
35694 288/300 Flush
35695 289/300 Write block 3731 len 20480 data: 8 6 7 8 8
35696 290/300 Read block 2805 len 8192
35697 291/300 Read block 4052 len 28672
35698 292/300 Write block 2985 len 4096 data: 4
35699 293/300 Read block 1835 len 32768
35700 294/300 Read block 456 len 28672
35701 295/300 Write block 684 len 40960 data: 6 5 4 4 4 5 6 5 7 6
35702 296/300 Read block 2915 len 4096
35703 297/300 Read block 1691 len 4096
35704 298/300 Read block 114 len 4096
35705 299/300 Read block 913 len 28672
35706 300/300 Read block 4052 len 36864
35707 301/300 Read block 1176 len 28672
35708 Main task has work to do, go find it
35709 got dsc Start(2)
35710 start 2
35711 Feb 28 06:08:55.692 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/2, method: POST, req_id: 63d592a5-5598-435b-a1c9-31b7df4f190d, remote_addr: 127.0.0.1:63908, local_addr: 127.0.0.1:9998
35712 Replay: started 2, returned:Ok(())
35713 ----------------------------------------------------------------
35714 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35715 States: Active Active Offline
35716 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35717 10056 Acked Flush 0 Done Done Sent false
35718 10057 Acked Flush 0 Done Done Sent false
35719 10058 Acked Read 2 Done Done Sent false
35720 10059 Acked Read 3 Done Done Sent false
35721 10060 Acked Read 7 Done Done Sent false
35722 10061 Acked Write 2 Done Done Sent false
35723 10062 Acked Write 3 Done Done Sent false
35724 10063 Acked Write 2 Done Done Sent false
35725 10064 Acked Read 3 Done Done Sent false
35726 10065 Acked Write 2 Done Done Sent false
35727 10066 Acked Read 4 Done Done Sent false
35728 10067 Acked Write 7 Done Done Sent false
35729 10068 Acked Read 6 Done Done Sent false
35730 10069 Acked Read 10 Done Done Sent false
35731 10070 Acked Write 4 Done Done Sent false
35732 10071 Acked Read 9 Done Done Sent false
35733 10072 Acked Read 6 Done Done Sent false
35734 10073 Acked Read 3 Done Done Sent false
35735 10074 Acked Read 8 Done Done Sent false
35736 10075 Acked Read 4 Done Done Sent false
35737 10076 Acked Write 7 Done Done Sent false
35738 10077 Acked Write 5 Done Done Sent false
35739 10078 Acked Read 2 Done Done Sent false
35740 10079 Acked Write 5 Done Done Sent false
35741 10080 Acked Flush 0 Done Done Sent false
35742 10081 Acked Read 4 Done Done Sent false
35743 10082 Acked Write 6 Done Done Sent false
35744 10083 Acked Read 7 Done Done Sent false
35745 10084 Acked Read 7 Done Done Sent false
35746 10085 Acked Write 10 Done Done Sent false
35747 10086 Acked Flush 0 Done Done Sent false
35748 10087 Acked Read 6 Done Done Sent false
35749 10088 Acked Flush 0 Done Done Sent false
35750 10089 Acked Read 2 Done Done Sent false
35751 10090 Acked Read 2 Done Done Sent false
35752 10091 Acked Read 4 Done Done Sent false
35753 10092 Acked Read 5 Done Done Sent false
35754 10093 Acked Read 9 Done Done Sent false
35755 10094 Acked Read 3 Done Done Sent false
35756 10095 Acked Write 5 Done Done Sent false
35757 10096 Acked Write 6 Done Done Sent false
35758 10097 Acked Read 9 Done Done Sent false
35759 10098 Acked Write 6 Done Done Sent false
35760 10099 Acked Read 10 Done Done Sent false
35761 10100 Acked Write 7 Done Done Sent false
35762 10101 Acked Write 5 Done Done Sent false
35763 10102 Acked Write 4 Done Done Sent false
35764 10103 Acked Write 9 Done Done Sent false
35765 10104 Acked Read 4 Done Done Sent false
35766 10105 Acked Read 8 Done Done Sent false
35767 10106 Acked Write 10 Done Done Sent false
35768 10107 Acked Write 1 Done Done Sent false
35769 10108 Acked Write 9 Done Done Sent false
35770 10109 Acked Write 10 Done Done Sent false
35771 10110 Acked Read 1 Done Done Sent false
35772 10111 Acked Read 1 Done Done Sent false
35773 10112 Acked Flush 0 Done Done Sent false
35774 10113 Acked Read 5 Done Done Sent false
35775 10114 Acked Read 7 Done Done Sent false
35776 10115 Acked Read 1 Done Done Sent false
35777 10116 Acked Read 2 Done Done Sent false
35778 10117 Acked Write 9 Done Done Sent false
35779 10118 Acked Write 2 Done Done Sent false
35780 10119 Acked Flush 0 Done Done Sent false
35781 10120 Acked Read 3 Done Done Sent false
35782 10121 Acked Read 5 Done Done Sent false
35783 10122 Acked Read 4 Done Done Sent false
35784 10123 Acked Read 9 Done Done Sent false
35785 10124 Acked Flush 0 Done Done Sent false
35786 10125 Acked Read 1 Done Done Sent false
35787 10126 Acked Read 9 Done Done Sent false
35788 10127 Acked Write 3 Done Done Sent false
35789 10128 Acked Read 2 Done Done Sent false
35790 10129 Acked Flush 0 Done Done Sent false
35791 10130 Acked Write 8 Done Done Sent false
35792 10131 Acked Read 9 Done Done Sent false
35793 10132 Acked Read 9 Done Done Sent false
35794 10133 Acked Read 7 Done Done Sent false
35795 10134 Acked Read 9 Done Done Sent false
35796 10135 Acked Read 7 Done Done Sent false
35797 10136 Acked Write 4 Done Done Sent false
35798 10137 Acked Write 4 Done Done Sent false
35799 10138 Acked Read 1 Done Done Sent false
35800 10139 Acked Read 4 Done Done Sent false
35801 10140 Acked Write 9 Done Done Sent false
35802 10141 Acked Write 1 Done Done Sent false
35803 10142 Acked Read 7 Done Done Sent false
35804 10143 Acked Read 2 Done Done Sent false
35805 10144 Acked Read 5 Done Done Sent false
35806 10145 Acked Read 9 Done Done Sent false
35807 10146 Acked Read 3 Done Done Sent false
35808 10147 Acked Read 9 Done Done Sent false
35809 10148 Acked Read 2 Done Done Sent false
35810 10149 Acked Read 8 Done Done Sent false
35811 10150 Acked Read 10 Done Done Sent false
35812 10151 Acked Write 4 Done Done Sent false
35813 10152 Acked Read 3 Done Done Sent false
35814 10153 Acked Read 2 Done Done Sent false
35815 10154 Acked Write 9 Done Done Sent false
35816 10155 Acked Write 10 Done Done Sent false
35817 10156 Acked Write 2 Done Done Sent false
35818 10157 Acked Flush 0 Done Done Sent false
35819 10158 Acked Write 7 Done Done Sent false
35820 10159 Acked Read 7 Done Done Sent false
35821 10160 Acked Write 4 Done Done Sent false
35822 10161 Acked Read 2 Done Done Sent false
35823 10162 Acked Read 7 Done Done Sent false
35824 10163 Acked Read 5 Done Done Sent false
35825 10164 Acked Read 2 Done Done Sent false
35826 10165 Acked Read 7 Done Done Sent false
35827 10166 Acked Write 3 Done Done Sent false
35828 10167 Acked Read 2 Done Done Sent false
35829 10168 Acked Write 9 Done Done Sent false
35830 10169 Acked Read 2 Done Done Sent false
35831 10170 Acked Read 9 Done Done Sent false
35832 10171 Acked Write 8 Done Done Sent false
35833 10172 Acked Read 5 Done Done Sent false
35834 10173 Acked Read 5 Done Done Sent false
35835 10174 Acked Read 3 Done Done Sent false
35836 10175 Acked Read 1 Done Done Sent false
35837 10176 Acked Read 2 Done Done Sent false
35838 10177 Acked Write 5 Done Done Sent false
35839 10178 Acked Write 9 Done Done Sent false
35840 10179 Acked Flush 0 Done Done Sent false
35841 10180 Acked Write 8 Done Done Sent false
35842 10181 Acked Write 7 Done Done Sent false
35843 10182 Acked Write 5 Done Done Sent false
35844 10183 Acked Read 4 Done Done Sent false
35845 10184 Acked Read 3 Done Done Sent false
35846 10185 Acked Write 9 Done Done Sent false
35847 10186 Acked Read 6 Done Done Sent false
35848 10187 Acked Read 8 Done Done Sent false
35849 10188 Acked Read 8 Done Done Sent false
35850 10189 Acked Read 7 Done Done Sent false
35851 10190 Acked Read 2 Done Done Sent false
35852 10191 Acked Read 10 Done Done Sent false
35853 10192 Acked Write 2 Done Done Sent false
35854 10193 Acked Read 9 Done Done Sent false
35855 10194 Acked Write 3 Done Done Sent false
35856 10195 Acked Read 3 Done Done Sent false
35857 10196 Acked Write 4 Done Done Sent false
35858 10197 Acked Read 5 Done Done Sent false
35859 10198 Acked Read 5 Done Done Sent false
35860 10199 Acked Read 9 Done Done Sent false
35861 10200 Acked Read 10 Done Done Sent false
35862 10201 Acked Read 9 Done Done Sent false
35863 10202 Acked Read 9 Done Done Sent false
35864 10203 Acked Write 5 Done Done Sent false
35865 10204 Acked Write 6 Done Done Sent false
35866 10205 Acked Read 4 Done Done Sent false
35867 10206 Acked Read 10 Done Done Sent false
35868 10207 Acked Read 2 Done Done Sent false
35869 10208 Acked Write 6 Done Done Sent false
35870 10209 Acked Read 6 Done Done Sent false
35871 10210 Acked Read 1 Done Done Sent false
35872 10211 Acked Read 10 Done Done Sent false
35873 10212 Acked Write 3 Done Done Sent false
35874 10213 Acked Write 1 Done Done Sent false
35875 10214 Acked Write 3 Done Done Sent false
35876 10215 Acked Write 5 Done Done Sent false
35877 10216 Acked Read 9 Done Done Sent false
35878 10217 Acked Write 9 Done Done Sent false
35879 10218 Acked Read 5 Done Done Sent false
35880 10219 Acked Read 7 Done Done Sent false
35881 10220 Acked Write 9 Done Done Sent false
35882 10221 Acked Read 5 Done Done Sent false
35883 10222 Acked Read 6 Done Done Sent false
35884 10223 Acked Write 1 Done Done Sent false
35885 10224 Acked Read 5 Done Done Sent false
35886 10225 Acked Read 3 Done Done Sent false
35887 10226 Acked Read 2 Done Done Sent false
35888 10227 Acked Read 10 Done Done Sent false
35889 10228 Acked Write 4 Done Done Sent false
35890 10229 Acked Read 5 Done Done Sent false
35891 10230 Acked Write 3 Done Done Sent false
35892 10231 Acked Read 2 Done Done Sent false
35893 10232 Acked Read 8 Done Done Sent false
35894 10233 Acked Write 4 Done Done Sent false
35895 10234 Acked Write 5 Done Done Sent false
35896 10235 Acked Read 3 Done Done Sent false
35897 10236 Acked Read 1 Done Done Sent false
35898 10237 Acked Write 10 Done Done Sent false
35899 10238 Acked Read 5 Done Done Sent false
35900 10239 Acked Flush 0 Done Done Sent false
35901 10240 Acked Write 6 Done Done Sent false
35902 10241 Acked Flush 0 Done Done Sent false
35903 10242 Acked Read 9 Done Done Sent false
35904 10243 Acked Read 7 Done Done Sent false
35905 10244 Acked Write 5 Done Done Sent false
35906 10245 Acked Read 7 Done Done Sent false
35907 10246 Acked Read 5 Done Done Sent false
35908 10247 Acked Write 6 Done Done Sent false
35909 10248 Acked Read 9 Done Done Sent false
35910 10249 Acked Write 9 Done Done Sent false
35911 10250 Acked Read 6 Done Done Sent false
35912 10251 Acked Read 2 Done Done Sent false
35913 10252 Acked Read 2 Done Done Sent false
35914 10253 Acked Write 9 Done Done Sent false
35915 10254 Acked Read 4 Done Done Sent false
35916 10255 Acked Read 7 Done Done Sent false
35917 10256 Acked Read 4 Done Done Sent false
35918 10257 Acked Read 2 Done Done Sent false
35919 10258 Acked Write 3 Done Done Sent false
35920 10259 Acked Read 10 Done Done Sent false
35921 10260 Acked Read 6 Done Done Sent false
35922 10261 Acked Write 2 Done Done Sent false
35923 10262 Acked Write 5 Done Done Sent false
35924 10263 Acked Write 1 Done Done Sent false
35925 10264 Acked Read 2 Done Done Sent false
35926 10265 Acked Read 6 Done Done Sent false
35927 10266 Acked Write 3 Done Done Sent false
35928 10267 Acked Write 10 Done Done Sent false
35929 10268 Acked Read 1 Done Done Sent false
35930 10269 Acked Read 8 Done Done Sent false
35931 10270 Acked Read 1 Done Done Sent false
35932 10271 Acked Read 7 Done Done Sent false
35933 10272 Acked Write 1 Done Done Sent false
35934 10273 Acked Read 8 Done Done Sent false
35935 10274 Acked Read 3 Done Done Sent false
35936 10275 Acked Write 6 Done Done Sent false
35937 10276 Acked Write 2 Done Done Sent false
35938 10277 Acked Write 10 Done Done Sent false
35939 10278 Acked Flush 0 Done Done Sent false
35940 10279 Acked Read 2 Done Done Sent false
35941 10280 Acked Read 6 Done Done Sent false
35942 10281 Acked Write 5 Done Done Sent false
35943 10282 Acked Read 5 Done Done Sent false
35944 10283 Acked Read 7 Done Done Sent false
35945 10284 Acked Write 8 Done Done Sent false
35946 10285 Acked Flush 0 Done Done Sent false
35947 10286 Acked Read 1 Done Done Sent false
35948 10287 Acked Write 5 Done Done Sent false
35949 10288 Acked Read 3 Done Done Sent false
35950 10289 Acked Read 1 Done Done Sent false
35951 10290 Acked Write 10 Done Done Sent false
35952 10291 Acked Read 9 Done Done Sent false
35953 10292 Acked Flush 0 Done Done Sent false
35954 10293 Acked Write 5 Done Done Sent false
35955 10294 Acked Read 9 Done Done Sent false
35956 10295 Acked Write 8 Done Done Sent false
35957 10296 Acked Read 3 Done Done Sent false
35958 10297 Acked Read 4 Done Done Sent false
35959 10298 Acked Read 2 Done Done Sent false
35960 10299 Acked Read 5 Done Done Sent false
35961 10300 Acked Write 9 Done Done Sent false
35962 10301 Acked Read 5 Done Done Sent false
35963 10302 Acked Read 4 Done Done Sent false
35964 10303 Acked Read 4 Done Done Sent false
35965 10304 Acked Write 3 Done Done Sent false
35966 10305 Acked Flush 0 Done Done Sent false
35967 10306 Acked Flush 0 Done Done Sent false
35968 10307 Acked Read 6 Done Done Sent false
35969 10308 Acked Read 5 Done Done Sent false
35970 10309 Acked Write 9 Done Done Sent false
35971 10310 Acked Read 4 Done Done Sent false
35972 10311 Acked Write 3 Done Done Sent false
35973 10312 Acked Flush 0 Done Done Sent false
35974 10313 Acked Write 3 Done Done Sent false
35975 10314 Acked Write 7 Done Done Sent false
35976 10315 Acked Write 8 Done Done Sent false
35977 10316 Acked Flush 0 Done Done Sent false
35978 10317 Acked Write 6 Done Done Sent false
35979 10318 Acked Write 8 Done Done Sent false
35980 10319 Acked Write 7 Done Done Sent false
35981 10320 Acked Read 9 Done Done Sent false
35982 10321 Acked Write 3 Done Done Sent false
35983 10322 Acked Write 2 Done Done Sent false
35984 10323 Acked Read 4 Done Done Sent false
35985 10324 Acked Read 3 Done Done Sent false
35986 10325 Acked Write 8 Done Done Sent false
35987 10326 Acked Flush 0 Done Done Sent false
35988 10327 Acked Read 3 Done Done Sent false
35989 10328 Acked Read 10 Done Done Sent false
35990 10329 Acked Read 6 Done Done Sent false
35991 10330 Acked Read 4 Done Done Sent false
35992 10331 Acked Read 6 Done Done Sent false
35993 10332 Acked Read 6 Done Done Sent false
35994 10333 Acked Write 7 Done Done Sent false
35995 10334 Acked Write 9 Done Done Sent false
35996 10335 Acked Read 3 Done Done Sent false
35997 10336 Acked Read 3 Done Done Sent false
35998 10337 Acked Write 4 Done Done Sent false
35999 10338 Acked Read 4 Done Done Sent false
36000 10339 Acked Read 7 Done Done Sent false
36001 10340 Acked Write 5 Done Done Sent false
36002 10341 Acked Read 5 Done Done Sent false
36003 10342 Acked Read 7 Done Done Sent false
36004 10343 Acked Flush 0 Done Done Sent false
36005 10344 Acked Write 5 Done Done Sent false
36006 10345 Acked Read 2 Done Done Sent false
36007 10346 Acked Read 7 Done Done Sent false
36008 10347 Acked Write 1 Done Done Sent false
36009 10348 Acked Read 8 Done Done Sent false
36010 10349 Acked Read 7 Done Done Sent false
36011 10350 Acked Write 10 Done Done Sent false
36012 10351 Acked Read 1 Done Done Sent false
36013 10352 Acked Read 1 Done Done Sent false
36014 10353 Acked Read 1 Done Done Sent false
36015 10354 Acked Read 7 Done Done Sent false
36016 10355 Acked Read 9 Done Done Sent false
36017 10356 Acked Read 7 Done Done Sent false
36018 STATES DS:0 DS:1 DS:2 TOTAL
36019 Sent 0 0 301 301
36020 Done 301 301 0 602
36021 Skipped 0 0 0 0
36022 Error 0 0 0 0
36023 Last Flush: Some(JobId(10343)) Some(JobId(10343)) Some(JobId(10055))
36024 Downstairs last five completed: 10055 10054 10053 10052 10051
36025 Upstairs last five completed: 10356 10355 10354 10353 10352
36026 CLIENT: Up:0 ds:301 act:2
360272025-02-28T06:08:56.194ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
360282025-02-28T06:08:57.136ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
360292025-02-28T06:08:58.138ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
360302025-02-28T06:08:59.140ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
36031 [8830] got start action
36032 [8830] I am going to restart
36033 Starting downstairs at port 8830
36034 Make output file at "/tmp/dsc/downstairs-8830.txt"
36035 [8830][2] reports Starting
36036 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1242)
36037 [8830][2] reports Running
36038 ----------------------------------------------------------------
36039 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
36040 States: Active Active Offline
36041 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36042 10056 Acked Flush 0 Done Done Sent false
36043 10057 Acked Flush 0 Done Done Sent false
36044 10058 Acked Read 2 Done Done Sent false
36045 10059 Acked Read 3 Done Done Sent false
36046 10060 Acked Read 7 Done Done Sent false
36047 10061 Acked Write 2 Done Done Sent false
36048 10062 Acked Write 3 Done Done Sent false
36049 10063 Acked Write 2 Done Done Sent false
36050 10064 Acked Read 3 Done Done Sent false
36051 10065 Acked Write 2 Done Done Sent false
36052 10066 Acked Read 4 Done Done Sent false
36053 10067 Acked Write 7 Done Done Sent false
36054 10068 Acked Read 6 Done Done Sent false
36055 10069 Acked Read 10 Done Done Sent false
36056 10070 Acked Write 4 Done Done Sent false
36057 10071 Acked Read 9 Done Done Sent false
36058 10072 Acked Read 6 Done Done Sent false
36059 10073 Acked Read 3 Done Done Sent false
36060 10074 Acked Read 8 Done Done Sent false
36061 10075 Acked Read 4 Done Done Sent false
36062 10076 Acked Write 7 Done Done Sent false
36063 10077 Acked Write 5 Done Done Sent false
36064 10078 Acked Read 2 Done Done Sent false
36065 10079 Acked Write 5 Done Done Sent false
36066 10080 Acked Flush 0 Done Done Sent false
36067 10081 Acked Read 4 Done Done Sent false
36068 10082 Acked Write 6 Done Done Sent false
36069 10083 Acked Read 7 Done Done Sent false
36070 10084 Acked Read 7 Done Done Sent false
36071 10085 Acked Write 10 Done Done Sent false
36072 10086 Acked Flush 0 Done Done Sent false
36073 10087 Acked Read 6 Done Done Sent false
36074 10088 Acked Flush 0 Done Done Sent false
36075 10089 Acked Read 2 Done Done Sent false
36076 10090 Acked Read 2 Done Done Sent false
36077 10091 Acked Read 4 Done Done Sent false
36078 10092 Acked Read 5 Done Done Sent false
36079 10093 Acked Read 9 Done Done Sent false
36080 10094 Acked Read 3 Done Done Sent false
36081 10095 Acked Write 5 Done Done Sent false
36082 10096 Acked Write 6 Done Done Sent false
36083 10097 Acked Read 9 Done Done Sent false
36084 10098 Acked Write 6 Done Done Sent false
36085 10099 Acked Read 10 Done Done Sent false
36086 10100 Acked Write 7 Done Done Sent false
36087 10101 Acked Write 5 Done Done Sent false
36088 10102 Acked Write 4 Done Done Sent false
36089 10103 Acked Write 9 Done Done Sent false
36090 10104 Acked Read 4 Done Done Sent false
36091 10105 Acked Read 8 Done Done Sent false
36092 10106 Acked Write 10 Done Done Sent false
36093 10107 Acked Write 1 Done Done Sent false
36094 10108 Acked Write 9 Done Done Sent false
36095 10109 Acked Write 10 Done Done Sent false
36096 10110 Acked Read 1 Done Done Sent false
36097 10111 Acked Read 1 Done Done Sent false
36098 10112 Acked Flush 0 Done Done Sent false
36099 10113 Acked Read 5 Done Done Sent false
36100 10114 Acked Read 7 Done Done Sent false
36101 10115 Acked Read 1 Done Done Sent false
36102 10116 Acked Read 2 Done Done Sent false
36103 10117 Acked Write 9 Done Done Sent false
36104 10118 Acked Write 2 Done Done Sent false
36105 10119 Acked Flush 0 Done Done Sent false
36106 10120 Acked Read 3 Done Done Sent false
36107 10121 Acked Read 5 Done Done Sent false
36108 10122 Acked Read 4 Done Done Sent false
36109 10123 Acked Read 9 Done Done Sent false
36110 10124 Acked Flush 0 Done Done Sent false
36111 10125 Acked Read 1 Done Done Sent false
36112 10126 Acked Read 9 Done Done Sent false
36113 10127 Acked Write 3 Done Done Sent false
36114 10128 Acked Read 2 Done Done Sent false
36115 10129 Acked Flush 0 Done Done Sent false
36116 10130 Acked Write 8 Done Done Sent false
36117 10131 Acked Read 9 Done Done Sent false
36118 10132 Acked Read 9 Done Done Sent false
36119 10133 Acked Read 7 Done Done Sent false
36120 10134 Acked Read 9 Done Done Sent false
36121 10135 Acked Read 7 Done Done Sent false
36122 10136 Acked Write 4 Done Done Sent false
36123 10137 Acked Write 4 Done Done Sent false
36124 10138 Acked Read 1 Done Done Sent false
36125 10139 Acked Read 4 Done Done Sent false
36126 10140 Acked Write 9 Done Done Sent false
36127 10141 Acked Write 1 Done Done Sent false
36128 10142 Acked Read 7 Done Done Sent false
36129 10143 Acked Read 2 Done Done Sent false
36130 10144 Acked Read 5 Done Done Sent false
36131 10145 Acked Read 9 Done Done Sent false
36132 10146 Acked Read 3 Done Done Sent false
36133 10147 Acked Read 9 Done Done Sent false
36134 10148 Acked Read 2 Done Done Sent false
36135 10149 Acked Read 8 Done Done Sent false
36136 10150 Acked Read 10 Done Done Sent false
36137 10151 Acked Write 4 Done Done Sent false
36138 10152 Acked Read 3 Done Done Sent false
36139 10153 Acked Read 2 Done Done Sent false
36140 10154 Acked Write 9 Done Done Sent false
36141 10155 Acked Write 10 Done Done Sent false
36142 10156 Acked Write 2 Done Done Sent false
36143 10157 Acked Flush 0 Done Done Sent false
36144 10158 Acked Write 7 Done Done Sent false
36145 10159 Acked Read 7 Done Done Sent false
36146 10160 Acked Write 4 Done Done Sent false
36147 10161 Acked Read 2 Done Done Sent false
36148 10162 Acked Read 7 Done Done Sent false
36149 10163 Acked Read 5 Done Done Sent false
36150 10164 Acked Read 2 Done Done Sent false
36151 10165 Acked Read 7 Done Done Sent false
36152 10166 Acked Write 3 Done Done Sent false
36153 10167 Acked Read 2 Done Done Sent false
36154 10168 Acked Write 9 Done Done Sent false
36155 10169 Acked Read 2 Done Done Sent false
36156 10170 Acked Read 9 Done Done Sent false
36157 10171 Acked Write 8 Done Done Sent false
36158 10172 Acked Read 5 Done Done Sent false
36159 10173 Acked Read 5 Done Done Sent false
36160 10174 Acked Read 3 Done Done Sent false
36161 10175 Acked Read 1 Done Done Sent false
36162 10176 Acked Read 2 Done Done Sent false
36163 10177 Acked Write 5 Done Done Sent false
36164 10178 Acked Write 9 Done Done Sent false
36165 10179 Acked Flush 0 Done Done Sent false
36166 10180 Acked Write 8 Done Done Sent false
36167 10181 Acked Write 7 Done Done Sent false
36168 10182 Acked Write 5 Done Done Sent false
36169 10183 Acked Read 4 Done Done Sent false
36170 10184 Acked Read 3 Done Done Sent false
36171 10185 Acked Write 9 Done Done Sent false
36172 10186 Acked Read 6 Done Done Sent false
36173 10187 Acked Read 8 Done Done Sent false
36174 10188 Acked Read 8 Done Done Sent false
36175 10189 Acked Read 7 Done Done Sent false
36176 10190 Acked Read 2 Done Done Sent false
36177 10191 Acked Read 10 Done Done Sent false
36178 10192 Acked Write 2 Done Done Sent false
36179 10193 Acked Read 9 Done Done Sent false
36180 10194 Acked Write 3 Done Done Sent false
36181 10195 Acked Read 3 Done Done Sent false
36182 10196 Acked Write 4 Done Done Sent false
36183 10197 Acked Read 5 Done Done Sent false
36184 10198 Acked Read 5 Done Done Sent false
36185 10199 Acked Read 9 Done Done Sent false
36186 10200 Acked Read 10 Done Done Sent false
36187 10201 Acked Read 9 Done Done Sent false
36188 10202 Acked Read 9 Done Done Sent false
36189 10203 Acked Write 5 Done Done Sent false
36190 10204 Acked Write 6 Done Done Sent false
36191 10205 Acked Read 4 Done Done Sent false
36192 10206 Acked Read 10 Done Done Sent false
36193 10207 Acked Read 2 Done Done Sent false
36194 10208 Acked Write 6 Done Done Sent false
36195 10209 Acked Read 6 Done Done Sent false
36196 10210 Acked Read 1 Done Done Sent false
36197 10211 Acked Read 10 Done Done Sent false
36198 10212 Acked Write 3 Done Done Sent false
36199 10213 Acked Write 1 Done Done Sent false
36200 10214 Acked Write 3 Done Done Sent false
36201 10215 Acked Write 5 Done Done Sent false
36202 10216 Acked Read 9 Done Done Sent false
36203 10217 Acked Write 9 Done Done Sent false
36204 10218 Acked Read 5 Done Done Sent false
36205 10219 Acked Read 7 Done Done Sent false
36206 10220 Acked Write 9 Done Done Sent false
36207 10221 Acked Read 5 Done Done Sent false
36208 10222 Acked Read 6 Done Done Sent false
36209 10223 Acked Write 1 Done Done Sent false
36210 10224 Acked Read 5 Done Done Sent false
36211 10225 Acked Read 3 Done Done Sent false
36212 10226 Acked Read 2 Done Done Sent false
36213 10227 Acked Read 10 Done Done Sent false
36214 10228 Acked Write 4 Done Done Sent false
36215 10229 Acked Read 5 Done Done Sent false
36216 10230 Acked Write 3 Done Done Sent false
36217 10231 Acked Read 2 Done Done Sent false
36218 10232 Acked Read 8 Done Done Sent false
36219 10233 Acked Write 4 Done Done Sent false
36220 10234 Acked Write 5 Done Done Sent false
36221 10235 Acked Read 3 Done Done Sent false
36222 10236 Acked Read 1 Done Done Sent false
36223 10237 Acked Write 10 Done Done Sent false
36224 10238 Acked Read 5 Done Done Sent false
36225 10239 Acked Flush 0 Done Done Sent false
36226 10240 Acked Write 6 Done Done Sent false
36227 10241 Acked Flush 0 Done Done Sent false
36228 10242 Acked Read 9 Done Done Sent false
36229 10243 Acked Read 7 Done Done Sent false
36230 10244 Acked Write 5 Done Done Sent false
36231 10245 Acked Read 7 Done Done Sent false
36232 10246 Acked Read 5 Done Done Sent false
36233 10247 Acked Write 6 Done Done Sent false
36234 10248 Acked Read 9 Done Done Sent false
36235 10249 Acked Write 9 Done Done Sent false
36236 10250 Acked Read 6 Done Done Sent false
36237 10251 Acked Read 2 Done Done Sent false
36238 10252 Acked Read 2 Done Done Sent false
36239 10253 Acked Write 9 Done Done Sent false
36240 10254 Acked Read 4 Done Done Sent false
36241 10255 Acked Read 7 Done Done Sent false
36242 10256 Acked Read 4 Done Done Sent false
36243 10257 Acked Read 2 Done Done Sent false
36244 10258 Acked Write 3 Done Done Sent false
36245 10259 Acked Read 10 Done Done Sent false
36246 10260 Acked Read 6 Done Done Sent false
36247 10261 Acked Write 2 Done Done Sent false
36248 10262 Acked Write 5 Done Done Sent false
36249 10263 Acked Write 1 Done Done Sent false
36250 10264 Acked Read 2 Done Done Sent false
36251 10265 Acked Read 6 Done Done Sent false
36252 10266 Acked Write 3 Done Done Sent false
36253 10267 Acked Write 10 Done Done Sent false
36254 10268 Acked Read 1 Done Done Sent false
36255 10269 Acked Read 8 Done Done Sent false
36256 10270 Acked Read 1 Done Done Sent false
36257 10271 Acked Read 7 Done Done Sent false
36258 10272 Acked Write 1 Done Done Sent false
36259 10273 Acked Read 8 Done Done Sent false
36260 10274 Acked Read 3 Done Done Sent false
36261 10275 Acked Write 6 Done Done Sent false
36262 10276 Acked Write 2 Done Done Sent false
36263 10277 Acked Write 10 Done Done Sent false
36264 10278 Acked Flush 0 Done Done Sent false
36265 10279 Acked Read 2 Done Done Sent false
36266 10280 Acked Read 6 Done Done Sent false
36267 10281 Acked Write 5 Done Done Sent false
36268 10282 Acked Read 5 Done Done Sent false
36269 10283 Acked Read 7 Done Done Sent false
36270 10284 Acked Write 8 Done Done Sent false
36271 10285 Acked Flush 0 Done Done Sent false
36272 10286 Acked Read 1 Done Done Sent false
36273 10287 Acked Write 5 Done Done Sent false
36274 10288 Acked Read 3 Done Done Sent false
36275 10289 Acked Read 1 Done Done Sent false
36276 10290 Acked Write 10 Done Done Sent false
36277 10291 Acked Read 9 Done Done Sent false
36278 10292 Acked Flush 0 Done Done Sent false
36279 10293 Acked Write 5 Done Done Sent false
36280 10294 Acked Read 9 Done Done Sent false
36281 10295 Acked Write 8 Done Done Sent false
36282 10296 Acked Read 3 Done Done Sent false
36283 10297 Acked Read 4 Done Done Sent false
36284 10298 Acked Read 2 Done Done Sent false
36285 10299 Acked Read 5 Done Done Sent false
36286 10300 Acked Write 9 Done Done Sent false
36287 10301 Acked Read 5 Done Done Sent false
36288 10302 Acked Read 4 Done Done Sent false
36289 10303 Acked Read 4 Done Done Sent false
36290 10304 Acked Write 3 Done Done Sent false
36291 10305 Acked Flush 0 Done Done Sent false
36292 10306 Acked Flush 0 Done Done Sent false
36293 10307 Acked Read 6 Done Done Sent false
36294 10308 Acked Read 5 Done Done Sent false
36295 10309 Acked Write 9 Done Done Sent false
36296 10310 Acked Read 4 Done Done Sent false
36297 10311 Acked Write 3 Done Done Sent false
36298 10312 Acked Flush 0 Done Done Sent false
36299 10313 Acked Write 3 Done Done Sent false
36300 10314 Acked Write 7 Done Done Sent false
36301 10315 Acked Write 8 Done Done Sent false
36302 10316 Acked Flush 0 Done Done Sent false
36303 10317 Acked Write 6 Done Done Sent false
36304 10318 Acked Write 8 Done Done Sent false
36305 10319 Acked Write 7 Done Done Sent false
36306 10320 Acked Read 9 Done Done Sent false
36307 10321 Acked Write 3 Done Done Sent false
36308 10322 Acked Write 2 Done Done Sent false
36309 10323 Acked Read 4 Done Done Sent false
36310 10324 Acked Read 3 Done Done Sent false
36311 10325 Acked Write 8 Done Done Sent false
36312 10326 Acked Flush 0 Done Done Sent false
36313 10327 Acked Read 3 Done Done Sent false
36314 10328 Acked Read 10 Done Done Sent false
36315 10329 Acked Read 6 Done Done Sent false
36316 10330 Acked Read 4 Done Done Sent false
36317 10331 Acked Read 6 Done Done Sent false
36318 10332 Acked Read 6 Done Done Sent false
36319 10333 Acked Write 7 Done Done Sent false
36320 10334 Acked Write 9 Done Done Sent false
36321 10335 Acked Read 3 Done Done Sent false
36322 10336 Acked Read 3 Done Done Sent false
36323 10337 Acked Write 4 Done Done Sent false
36324 10338 Acked Read 4 Done Done Sent false
36325 10339 Acked Read 7 Done Done Sent false
36326 10340 Acked Write 5 Done Done Sent false
36327 10341 Acked Read 5 Done Done Sent false
36328 10342 Acked Read 7 Done Done Sent false
36329 10343 Acked Flush 0 Done Done Sent false
36330 10344 Acked Write 5 Done Done Sent false
36331 10345 Acked Read 2 Done Done Sent false
36332 10346 Acked Read 7 Done Done Sent false
36333 10347 Acked Write 1 Done Done Sent false
36334 10348 Acked Read 8 Done Done Sent false
36335 10349 Acked Read 7 Done Done Sent false
36336 10350 Acked Write 10 Done Done Sent false
36337 10351 Acked Read 1 Done Done Sent false
36338 10352 Acked Read 1 Done Done Sent false
36339 10353 Acked Read 1 Done Done Sent false
36340 10354 Acked Read 7 Done Done Sent false
36341 10355 Acked Read 9 Done Done Sent false
36342 10356 Acked Read 7 Done Done Sent false
36343 STATES DS:0 DS:1 DS:2 TOTAL
36344 Sent 0 0 301 301
36345 Done 301 301 0 602
36346 Skipped 0 0 0 0
36347 Error 0 0 0 0
36348 Last Flush: Some(JobId(10343)) Some(JobId(10343)) Some(JobId(10055))
36349 Downstairs last five completed: 10055 10054 10053 10052 10051
36350 Upstairs last five completed: 10356 10355 10354 10353 10352
36351 CLIENT: Up:0 ds:301 act:2
363522025-02-28T06:09:01.143ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
363532025-02-28T06:09:01.439ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
363542025-02-28T06:09:01.439ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39127) = io task client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
363552025-02-28T06:09:01.440ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
363562025-02-28T06:09:01.440ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
363572025-02-28T06:09:01.440ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
363582025-02-28T06:09:01.440ZINFOcrucible: send last flush ID to this DS: Some(JobId(10055)) = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
363592025-02-28T06:09:01.440ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
363602025-02-28T06:09:01.440ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(10055)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
363612025-02-28T06:09:01.441ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(10055)) = downstairs session_id = 969fa3b2-3050-4644-bbdf-c5076af6b813
36362 ----------------------------------------------------------------
36363 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36364 Downstairs last five completed: 10357 10356 10355 10354 10353
36365 Upstairs last five completed: 10357 10356 10355 10354 10353
36366 CLIENT: Up:0 ds:0 act:3
36367 Replay: All jobs finished
36368 Test replay has completed
36369 Wrote out file "/tmp/test_replay_verify.log"
36370 CLIENT: Tests done. All submitted work has been ACK'd
36371 ----------------------------------------------------------------
36372 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36373 Downstairs last five completed: 10357 10356 10355 10354 10353
36374 Upstairs last five completed: 10357 10356 10355 10354 10353
36375 CLIENT: Up:0 ds:0 act:3
36376 CLIENT: All jobs finished, all DS active.
363772025-02-28T06:09:03.701ZWARNcrucible: Guest handle has been dropped
36378 Replays:30 time: 8:00
36379 Do final verify
36380 Feb 28 06:09:03.772 INFO accepted connection, remote_addr: 127.0.0.1:43386, local_addr: 127.0.0.1:9998
36381 Feb 28 06:09:03.772 INFO request completed, latency_us: 214, response_code: 200, uri: /regioninfo, method: GET, req_id: 6a42cb1a-9855-46a5-a4e7-b99efe138b9d, remote_addr: 127.0.0.1:43386, local_addr: 127.0.0.1:9998
363822025-02-28T06:09:03.773ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
36383 Feb 28 06:09:03.773 INFO request completed, latency_us: 89, response_code: 200, uri: /regioncount, method: GET, req_id: 690b5449-d7fd-4ab5-a79b-df6800be512f, remote_addr: 127.0.0.1:43386, local_addr: 127.0.0.1:9998
363842025-02-28T06:09:03.773ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
36385 Feb 28 06:09:03.774 INFO request completed, latency_us: 107, response_code: 200, uri: /port/cid/0, method: GET, req_id: 99792fae-7a1c-4ed5-b688-96c27dabb11f, remote_addr: 127.0.0.1:43386, local_addr: 127.0.0.1:9998
36386 Feb 28 06:09:03.774 INFO request completed, latency_us: 88, response_code: 200, uri: /port/cid/1, method: GET, req_id: f311a2bb-f438-4815-925c-fd0e72454bc9, remote_addr: 127.0.0.1:43386, local_addr: 127.0.0.1:9998
36387 Feb 28 06:09:03.775 INFO request completed, latency_us: 97, response_code: 200, uri: /port/cid/2, method: GET, req_id: 091e50b8-92f1-489b-a4c5-9c9bcaac40f4, remote_addr: 127.0.0.1:43386, local_addr: 127.0.0.1:9998
363882025-02-28T06:09:03.775ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
363892025-02-28T06:09:03.775ZINFOcrucible: Upstairs starts
363902025-02-28T06:09:03.775ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "0c103f9ae2997264530dbd443fadbdf83878dac4", git_commit_timestamp: "2025-02-28T05:45:10.000000000Z", git_branch: "renovate/uuid-1.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
363912025-02-28T06:09:03.775ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
363922025-02-28T06:09:03.775ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
363932025-02-28T06:09:03.776ZINFOcrucible: Crucible e43d6e3b-f875-4bf6-93ed-2a4796303496 has session id: c6ccf12c-5dc8-4d51-966e-c87d79da1b7f session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
363942025-02-28T06:09:03.776ZINFOcrucible: Upstairs opts: Upstairs UUID: e43d6e3b-f875-4bf6-93ed-2a4796303496, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
363952025-02-28T06:09:03.776ZINFOcrucible: Crucible stats registered with UUID: e43d6e3b-f875-4bf6-93ed-2a4796303496 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
363962025-02-28T06:09:03.776ZWARNcrucible: could not find Downstairs address for Nexus session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
363972025-02-28T06:09:03.777ZINFOcrucible: Downstairs in volume = 3 task = crutest
363982025-02-28T06:09:03.777ZINFOcrucible: The guest has requested activation with gen:3
363992025-02-28T06:09:03.777ZINFOcrucible: e43d6e3b-f875-4bf6-93ed-2a4796303496 active request set session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364002025-02-28T06:09:03.777ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364012025-02-28T06:09:03.777ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364022025-02-28T06:09:03.777ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364032025-02-28T06:09:03.777ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364042025-02-28T06:09:03.777ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364052025-02-28T06:09:03.777ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364062025-02-28T06:09:03.777ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364072025-02-28T06:09:03.777ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364082025-02-28T06:09:03.777ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364092025-02-28T06:09:03.777ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58207) = io task client = 1 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364102025-02-28T06:09:03.777ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36863) = io task client = 0 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364112025-02-28T06:09:03.777ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52451) = io task client = 2 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364122025-02-28T06:09:03.778ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364132025-02-28T06:09:03.778ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364142025-02-28T06:09:03.778ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364152025-02-28T06:09:03.778ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364162025-02-28T06:09:03.778ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364172025-02-28T06:09:03.778ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364182025-02-28T06:09:03.779ZINFOcrucible: Waiting for more clients to be ready session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364192025-02-28T06:09:03.779ZINFOcrucible: Waiting for more clients to be ready session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364202025-02-28T06:09:03.779ZINFOcrucible: [0]R flush_numbers[0..12]: [950, 960, 952, 922, 955, 944, 961, 950, 959, 949, 945, 955] = downstairs session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364212025-02-28T06:09:03.779ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364222025-02-28T06:09:03.779ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364232025-02-28T06:09:03.779ZINFOcrucible: [1]R flush_numbers[0..12]: [950, 960, 952, 922, 955, 944, 961, 950, 959, 949, 945, 955] = downstairs session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364242025-02-28T06:09:03.779ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364252025-02-28T06:09:03.779ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364262025-02-28T06:09:03.779ZINFOcrucible: [2]R flush_numbers[0..12]: [950, 960, 952, 922, 955, 944, 961, 950, 959, 949, 945, 955] = downstairs session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364272025-02-28T06:09:03.779ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364282025-02-28T06:09:03.779ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364292025-02-28T06:09:03.779ZINFOcrucible: Max found gen is 3 = downstairs session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
36430 {"msg":"Generation requested: 3 >= found:3","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
36431 ,"time":"2025-02-28T06:09:03.779877789Z","hostname":"w-01JN5J6TXF5NZGR4335G8W67DN","pid":1247,"":"downstairs","":"downstairs","session_id":"c6ccf12c-5dc8-4d51-966e-c87d79da1b7f"}
364322025-02-28T06:09:03.779ZINFOcrucible: Next flush: 962 = downstairs session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364332025-02-28T06:09:03.779ZINFOcrucible: All extents match = downstairs session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364342025-02-28T06:09:03.779ZINFOcrucible: No downstairs reconciliation required session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364352025-02-28T06:09:03.779ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364362025-02-28T06:09:03.779ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364372025-02-28T06:09:03.779ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364382025-02-28T06:09:03.779ZINFOcrucible: All required reconciliation work is completed session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364392025-02-28T06:09:03.779ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364402025-02-28T06:09:03.779ZINFOcrucible: e43d6e3b-f875-4bf6-93ed-2a4796303496 is now active with session: c6ccf12c-5dc8-4d51-966e-c87d79da1b7f session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364412025-02-28T06:09:03.780ZINFOcrucible: Set Active after no reconciliation session_id = c6ccf12c-5dc8-4d51-966e-c87d79da1b7f
364422025-02-28T06:09:03.780ZINFOcrucible: The guest has finished waiting for activation with:3
36443 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
36444 Loading write count information from file "/tmp/test_replay_verify.log"
36445 Read and Verify all blocks (0..5000 range:false)
36446 Wrote out file "/tmp/test_replay_verify.log"
36447 Verify test completed
36448 Wrote out file "/tmp/test_replay_verify.log"
36449 CLIENT: Tests done. All submitted work has been ACK'd
36450 ----------------------------------------------------------------
36451 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
36452 States: Active Active Active
36453 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36454 1000 Acked Read 256 Done Done Done false
36455 1001 Acked Read 256 Done Done Done false
36456 1002 Acked Read 256 Done Done Done false
36457 1003 Acked Read 256 Done Done Done false
36458 1004 Acked Read 256 Done Done Done false
36459 1005 Acked Read 256 Done Done Done false
36460 1006 Acked Read 256 Done Done Done false
36461 1007 Acked Read 256 Done Done Done false
36462 1008 Acked Read 256 Done Done Done false
36463 1009 Acked Read 256 Done Done Done false
36464 1010 Acked Read 256 Done Done Done false
36465 1011 Acked Read 256 Done Done Done false
36466 1012 Acked Read 256 Done Done Done false
36467 1013 Acked Read 256 Done Done Done false
36468 1014 Acked Read 256 Done Done Done false
36469 1015 Acked Read 256 Done Done Done false
36470 1016 Acked Read 256 Done Done Done false
36471 1017 Acked Read 136 Done Done Done false
36472 1018 Acked Read 256 Done Done Done false
36473 1019 Acked Read 256 Done Done Done false
36474 STATES DS:0 DS:1 DS:2 TOTAL
36475 Sent 0 0 0 0
36476 Done 20 20 20 60
36477 Skipped 0 0 0 0
36478 Error 0 0 0 0
36479 Last Flush: None None None
36480 Downstairs last five completed:
36481 Upstairs last five completed: 1019 1018 1017 1016 1015
36482 CLIENT: Up:0 ds:20 act:3
36483 ----------------------------------------------------------------
36484 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
36485 Downstairs last five completed: 1020 1019 1018 1017 1016
36486 Upstairs last five completed: 1020 1019 1018 1017 1016
36487 CLIENT: Up:0 ds:0 act:3
36488 CLIENT: All crucible jobs finished, exiting program
364892025-02-28T06:09:08.183ZWARNcrucible: Guest handle has been dropped
36490 Feb 28 06:09:08.247 INFO accepted connection, remote_addr: 127.0.0.1:49565, local_addr: 127.0.0.1:9998
36491 Main task has work to do, go find it
36492 got dsc Shutdown
36493 Shutdown
36494 Feb 28 06:09:08.247 INFO request completed, latency_us: 165, response_code: 204, uriShut it down
36495 :[8810] Disable keep_running
36496 /shutdown, method[8810] Got stop action so:false kr:false
36497 : POST, req_id: fdaa15a1-cb0e-42b4-b59d-a87000a72de1, remote_addr: 127.0.0.1:49565, local_addr: 127.0.0.1:9998
36498 [8830] Disable keep_running
36499 [8830] Got stop action so:false kr:false
36500 [8820] Disable keep_running
36501 [8820] Got stop action so:false kr:false
36502 [8810] Exited with: ExitStatus(unix_wait_status(9))
36503 [8820] Exited with: ExitStatus(unix_wait_status(9))
36504 [8830] Exited with: ExitStatus(unix_wait_status(9))
36505 [8810][0] reports Exit
36506 [8820][1] reports Exit
36507 [8830][2] reports Exit
36508 February 28, 2025 at 06:09:13 AM UTC Test ends with 0