Max level shown:
1 starting February 21, 2025 at 06:45:13 AM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.2266062
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.02164059
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.02079622
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start ds: 8820
14 start ds: 8830
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8820
17 Starting downstairs at port 8830
18 Make output file at "/tmp/dsc/downstairs-8830.txt"
19 Starting downstairs at port 8810
20 Make output file at "/tmp/dsc/downstairs-8810.txt"
21 Make output file at "/tmp/dsc/downstairs-8820.txt"
22 [8820][1] initial start wait reports Starting
23 [8830][2] initial start wait reports Starting
24 [8810][0] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1196)
26 [8810][0] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1197)
28 [8830][2] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1198)
30 [8820][1] initial start wait reports Running
31 All downstairs are running
32 Feb 21 06:45:13.391 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Feb 21 06:45:18.494 INFO accepted connection, remote_addr: 127.0.0.1:50620, local_addr: 127.0.0.1:9998
36 Feb 21 06:45:18.496 INFO request completed, latency_us: 658, response_code: 200, uri: /regioninfo, method: GET, req_id: d396fbc9-976f-45f0-8ed6-187d18fb1547, remote_addr: 127.0.0.1:50620, local_addr: 127.0.0.1:9998
37 Feb 21 06:45:18.498 INFO request completed, latency_us: 86, response_code: 200, uri: /regioncount, method: GET, req_id: 79620df2-153d-4c99-9811-3e7b708e4471, remote_addr: 127.0.0.1:50620, local_addr: 127.0.0.1:9998
382025-02-21T06:45:18.498ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-02-21T06:45:18.499ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Feb 21 06:45:18.499 INFO request completed, latency_us: 139, response_code: 200, uri: /port/cid/0, method: GET, req_id: 4d41f508-6154-4fec-9d6a-75f518de7853, remote_addr: 127.0.0.1:50620, local_addr: 127.0.0.1:9998
41 Feb 21 06:45:18.500 INFO request completed, latency_us: 87, response_code: 200, uri: /port/cid/1, method: GET, req_id: dd13e523-b077-462a-bbd1-6bcc097b72cf, remote_addr: 127.0.0.1:50620, local_addr: 127.0.0.1:9998
42 Feb 21 06:45:18.500 INFO request completed, latency_us: 87, response_code: 200, uri: /port/cid/2, method: GET, req_id: a0f65672-cbcb-42c0-a1e2-f99987b06b15, remote_addr: 127.0.0.1:50620, local_addr: 127.0.0.1:9998
432025-02-21T06:45:18.501ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-02-21T06:45:18.502ZINFOcrucible: Upstairs starts
452025-02-21T06:45:18.502ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "7ae555c374784c668297ef5478b8ff4aaa8d57b4", git_commit_timestamp: "2025-02-21T06:29:49.000000000Z", git_branch: "renovate/rust-1.x", rustc_semver: "1.85.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "4d91de4e48198da2e33413efdcd9cd2cc0c46688", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-02-21T06:45:18.502ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-02-21T06:45:18.502ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-02-21T06:45:18.503ZINFOcrucible: Crucible 2a249d34-ffd4-4a32-b96d-7e46f6dc469d has session id: f8e07b55-3e6f-450c-a99b-34f694047a0f session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
492025-02-21T06:45:18.504ZINFOcrucible: Upstairs opts: Upstairs UUID: 2a249d34-ffd4-4a32-b96d-7e46f6dc469d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
502025-02-21T06:45:18.504ZINFOcrucible: Crucible stats registered with UUID: 2a249d34-ffd4-4a32-b96d-7e46f6dc469d session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
512025-02-21T06:45:18.504ZWARNcrucible: could not find Downstairs address for Nexus session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
522025-02-21T06:45:18.506ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-02-21T06:45:18.506ZINFOcrucible: The guest has requested activation with gen:1
542025-02-21T06:45:18.506ZINFOcrucible: 2a249d34-ffd4-4a32-b96d-7e46f6dc469d active request set session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
552025-02-21T06:45:18.506ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
562025-02-21T06:45:18.506ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
572025-02-21T06:45:18.506ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
582025-02-21T06:45:18.506ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
592025-02-21T06:45:18.506ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
602025-02-21T06:45:18.506ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
612025-02-21T06:45:18.506ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
622025-02-21T06:45:18.506ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
632025-02-21T06:45:18.506ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
642025-02-21T06:45:18.507ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53079) = io task client = 0 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
652025-02-21T06:45:18.507ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43114) = io task client = 2 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
662025-02-21T06:45:18.507ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55760) = io task client = 1 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
672025-02-21T06:45:18.509ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
682025-02-21T06:45:18.509ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
692025-02-21T06:45:18.509ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
702025-02-21T06:45:18.510ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
712025-02-21T06:45:18.510ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
722025-02-21T06:45:18.510ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
732025-02-21T06:45:18.511ZINFOcrucible: Waiting for more clients to be ready session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
742025-02-21T06:45:18.511ZINFOcrucible: Waiting for more clients to be ready session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
752025-02-21T06:45:18.512ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
762025-02-21T06:45:18.512ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
772025-02-21T06:45:18.512ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
782025-02-21T06:45:18.512ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
792025-02-21T06:45:18.512ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
802025-02-21T06:45:18.512ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
812025-02-21T06:45:18.512ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
822025-02-21T06:45:18.512ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
832025-02-21T06:45:18.512ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
842025-02-21T06:45:18.512ZINFOcrucible: Max found gen is 1 = downstairs session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
852025-02-21T06:45:18.512ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
862025-02-21T06:45:18.512ZINFOcrucible: Next flush: 1 = downstairs session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
87 {"msg":"All extents match","v":0,"Activate successful
88 nameWait for a query_work_queue command to finish before sending IO
89 ":"crucible","level":30,"time":"2025-02-21T06:45:18.512207794Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1202,"":"downstairs","":"downstairs","session_id":"f8e07b55-3e6f-450c-a99b-34f694047a0f"}
902025-02-21T06:45:18.512ZINFOcrucible: No downstairs reconciliation required session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
912025-02-21T06:45:18.512ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
922025-02-21T06:45:18.512ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
932025-02-21T06:45:18.512ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
942025-02-21T06:45:18.512ZINFOcrucible: All required reconciliation work is completed session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
952025-02-21T06:45:18.512ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
962025-02-21T06:45:18.512ZINFOcrucible: 2a249d34-ffd4-4a32-b96d-7e46f6dc469d is now active with session: f8e07b55-3e6f-450c-a99b-34f694047a0f session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
972025-02-21T06:45:18.512ZINFOcrucible: Set Active after no reconciliation session_id = f8e07b55-3e6f-450c-a99b-34f694047a0f
982025-02-21T06:45:18.512ZINFOcrucible: The guest has finished waiting for activation with:1
99 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
100 Fill test
101 Read and Verify all blocks (0..5000 range:false)
102 Wrote out file "/tmp/test_replay_verify.log"
103 CLIENT: Tests done. All submitted work has been ACK'd
104 ----------------------------------------------------------------
105 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
106 States: Active Active Active
107 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
108 1021 Acked Read 256 Done Done Done false
109 1022 Acked Read 256 Done Done Done false
110 1023 Acked Read 256 Done Done Done false
111 1024 Acked Read 256 Done Done Done false
112 1025 Acked Read 256 Done Done Done false
113 1026 Acked Read 256 Done Done Done false
114 1027 Acked Read 256 Done Done Done false
115 1028 Acked Read 256 Done Done Done false
116 1029 Acked Read 256 Done Done Done false
117 1030 Acked Read 256 Done Done Done false
118 1031 Acked Read 256 Done Done Done false
119 1032 Acked Read 256 Done Done Done false
120 1033 Acked Read 256 Done Done Done false
121 1034 Acked Read 256 Done Done Done false
122 1035 Acked Read 256 Done Done Done false
123 1036 Acked Read 256 Done Done Done false
124 1037 Acked Read 256 Done Done Done false
125 1038 Acked Read 136 Done Done Done false
126 1039 Acked Read 256 Done Done Done false
127 1040 Acked Read 256 Done Done Done false
128 STATES DS:0 DS:1 DS:2 TOTAL
129 Sent 0 0 0 0
130 Done 20 20 20 60
131 Skipped 0 0 0 0
132 Error 0 0 0 0
133 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
134 Downstairs last five completed: 1020 1019 1018 1017 1016
135 Upstairs last five completed: 1040 1039 1038 1037 1036
136 CLIENT: Up:0 ds:20 act:3
137 ----------------------------------------------------------------
138 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
139 Downstairs last five completed: 1041 1040 1039 1038 1037
140 Upstairs last five completed: 1041 1040 1039 1038 1037
141 CLIENT: Up:0 ds:0 act:3
142 CLIENT: All crucible jobs finished, exiting program
1432025-02-21T06:45:23.083ZWARNcrucible: Guest handle has been dropped
144 Replay loop starts now February 21, 2025 at 06:45:23 AM UTC
145 Feb 21 06:45:23.159 INFO accepted connection, remote_addr: 127.0.0.1:64087, local_addr: 127.0.0.1:9998
146 Feb 21 06:45:23.159 INFO request completed, latency_us: 178, response_code: 200, uri: /regioninfo, method: GET, req_id: 91c31491-a5eb-4259-8113-10e25cc54989, remote_addr: 127.0.0.1:64087, local_addr: 127.0.0.1:9998
1472025-02-21T06:45:23.160ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
148 Feb 21 06:45:23.160 INFO request completed, latency_us: 109, response_code: 200, uri: /regioncount, method: GET, req_id: d3d88394-aee2-46db-91ca-a862dc4d8034, remote_addr: 127.0.0.1:64087, local_addr: 127.0.0.1:9998
1492025-02-21T06:45:23.161ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
150 Feb 21 06:45:23.161 INFO request completed, latency_us: 132, response_code: 200, uri: /port/cid/0, method: GET, req_id: 810129d9-8fc1-4813-9e03-77e67d833419, remote_addr: 127.0.0.1:64087, local_addr: 127.0.0.1:9998
151 Feb 21 06:45:23.162 INFO request completed, latency_us: 106, response_code: 200, uri: /port/cid/1, method: GET, req_id: 4e9815a9-53fb-424e-b885-23894c5b2325, remote_addr: 127.0.0.1:64087, local_addr: 127.0.0.1:9998
152 Feb 21 06:45:23.162 INFO request completed, latency_us: 89, response_code: 200, uri: /port/cid/2, method: GET, req_id: 799e3591-1b17-4fb8-9a06-e30197a0d629, remote_addr: 127.0.0.1:64087, local_addr: 127.0.0.1:9998
1532025-02-21T06:45:23.163ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1542025-02-21T06:45:23.163ZINFOcrucible: Upstairs starts
1552025-02-21T06:45:23.163ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "7ae555c374784c668297ef5478b8ff4aaa8d57b4", git_commit_timestamp: "2025-02-21T06:29:49.000000000Z", git_branch: "renovate/rust-1.x", rustc_semver: "1.85.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "4d91de4e48198da2e33413efdcd9cd2cc0c46688", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1562025-02-21T06:45:23.163ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1572025-02-21T06:45:23.163ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1582025-02-21T06:45:23.163ZINFOcrucible: Crucible 3199ea69-eac0-41e5-baa1-38f4d047d85f has session id: 3a757114-e39d-42e6-bc2b-0419bce3aed2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1592025-02-21T06:45:23.163ZINFOcrucible: Upstairs opts: Upstairs UUID: 3199ea69-eac0-41e5-baa1-38f4d047d85f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1602025-02-21T06:45:23.163ZINFOcrucible: Crucible stats registered with UUID: 3199ea69-eac0-41e5-baa1-38f4d047d85f session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1612025-02-21T06:45:23.164ZWARNcrucible: could not find Downstairs address for Nexus session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1622025-02-21T06:45:23.165ZINFOcrucible: Downstairs in volume = 3 task = crutest
1632025-02-21T06:45:23.165ZINFOcrucible: The guest has requested activation with gen:2
1642025-02-21T06:45:23.165ZINFOcrucible: 3199ea69-eac0-41e5-baa1-38f4d047d85f active request set session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1652025-02-21T06:45:23.165ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1662025-02-21T06:45:23.165ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1672025-02-21T06:45:23.165ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1682025-02-21T06:45:23.165ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1692025-02-21T06:45:23.165ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1702025-02-21T06:45:23.165ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1712025-02-21T06:45:23.165ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1722025-02-21T06:45:23.165ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1732025-02-21T06:45:23.165ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1742025-02-21T06:45:23.165ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59743) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1752025-02-21T06:45:23.165ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33466) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1762025-02-21T06:45:23.165ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39200) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1772025-02-21T06:45:23.166ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1782025-02-21T06:45:23.166ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1792025-02-21T06:45:23.166ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1802025-02-21T06:45:23.166ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1812025-02-21T06:45:23.167ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1822025-02-21T06:45:23.167ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1832025-02-21T06:45:23.167ZINFOcrucible: Waiting for more clients to be ready session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1842025-02-21T06:45:23.167ZINFOcrucible: Waiting for more clients to be ready session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1852025-02-21T06:45:23.167ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1862025-02-21T06:45:23.167ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1872025-02-21T06:45:23.167ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1882025-02-21T06:45:23.167ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1892025-02-21T06:45:23.167ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1902025-02-21T06:45:23.167ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1912025-02-21T06:45:23.167ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1922025-02-21T06:45:23.167ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1932025-02-21T06:45:23.167ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1942025-02-21T06:45:23.167ZINFOcrucible: Max found gen is 2 = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1952025-02-21T06:45:23.167ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1962025-02-21T06:45:23.168ZINFOcrucible: Next flush: 2 = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
197 {"msg":"All extents match","v":0,"name":"crucible","level":30Activate successful
198 ,Wait for a query_work_queue command to finish before sending IO
199 "time":"2025-02-21T06:45:23.168020811Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"downstairs","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
2002025-02-21T06:45:23.168ZINFOcrucible: No downstairs reconciliation required session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2012025-02-21T06:45:23.168ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2022025-02-21T06:45:23.168ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2032025-02-21T06:45:23.168ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2042025-02-21T06:45:23.168ZINFOcrucible: All required reconciliation work is completed session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2052025-02-21T06:45:23.168ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2062025-02-21T06:45:23.168ZINFOcrucible: 3199ea69-eac0-41e5-baa1-38f4d047d85f is now active with session: 3a757114-e39d-42e6-bc2b-0419bce3aed2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2072025-02-21T06:45:23.168ZINFOcrucible: Set Active after no reconciliation session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2082025-02-21T06:45:23.168ZINFOcrucible: The guest has finished waiting for activation with:2
209 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
210 Loading write count information from file "/tmp/test_replay_verify.log"
211 Feb 21 06:45:23.203 INFO accepted connection, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
212 Main task has work to do, go find it
213 got dsc Stop(0)
214 stop 0
215 Feb 21 06:45:23.204 INFO request completed, latency_us: 201, response_code: 204, uri: /stop/cid/0, method: POST, req_id: c40b16c6-4bf7-4931-b5b2-58ce121b7600, remote_addr: 127.0.0.1:53848, local_addr: 127.0.[8810] Got stop action so:false kr:false
216 0.1:9998
2172025-02-21T06:45:23.205ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2182025-02-21T06:45:23.205ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2192025-02-21T06:45:23.205ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
220 Feb 21 06:45:23.205 INFO request completed, latency_us: 165, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:45:23.205505334Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
221 response_code: 200, uri: /state/cid/0, method: GET, req_id: 939549a9-1b9e-4dbf-b8bb-04a1dd0ec74c, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
222 [8810] Exited with: ExitStatus(unix_wait_status(9))
223 [8810][0] reports Exit
224 Feb 21 06:45:27.207 INFO request completed, latency_us: 334, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2f6c4e50-df16-4f6f-b9e7-fbab65b2f9e5, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
225 001/300 Flush
226 002/300 Read block 1200 len 20480
227 003/300 Read block 1842 len 12288
228 004/300 Read block 1590 len 36864
229 005/300 Write block 1781 len 24576 data: 2 2 2 2 2 2
230 006/300 Write block 1846 len 8192 data: 2 2
231 007/300 Read block 4424 len 28672
232 008/300 Read block 3304 len 28672
233 009/300 Read block 205 len 40960
234 010/300 Write block 2964 len 8192 data: 2 2
235 011/300 Read block 708 len 20480
236 012/300 Read block 4112 len 16384
237 013/300 Write block 3411 len 28672 data: 2 2 2 2 2 2 2
238 014/300 Write block 1560 len 16384 data: 2 2 2 2
239 015/300 Flush
240 016/300 Read block 4596 len 28672
241 017/300 Read block 4446 len 12288
242 018/300 Read block 2993 len 4096
243 019/300 Read block 1282 len 16384
244 020/300 Read block 4180 len 8192
245 021/300 Read block 2654 len 24576
246 022/300 Flush
247 023/300 Read block 2314 len 32768
248 024/300 Write block 3950 len 4096 data: 2
249 025/300 Write block 4469 len 40960 data: 2 2 2 2 2 2 2 2 2 2
250 026/300 Read block 786 len 12288
251 027/300 Read block 2475 len 28672
252 028/300 Write block 2255 len 4096 data: 2
253 029/300 Read block 2840 len 8192
254 030/300 Write block 1131 len 36864 data: 2 2 2 2 2 2 2 2 2
255 031/300 Flush
256 032/300 Write block 1662 len 12288 data: 2 2 2
257 033/300 Read block 2967 len 16384
258 034/300 Read block 1218 len 32768
259 035/300 Write block 1463 len 24576 data: 2 2 2 2 2 2
260 036/300 Read block 4102 len 24576
261 037/300 Read block 3354 len 28672
262 038/300 Read block 3231 len 20480
263 039/300 Write block 4055 len 28672 data: 2 2 2 2 2 2 2
264 040/300 Read block 3499 len 28672
265 041/300 Write block 2163 len 36864 data: 2 2 2 2 2 2 2 2 2
266 042/300 Flush
267 043/300 Read block 113 len 40960
268 044/300 Write block 2226 len 12288 data: 2 2 2
269 045/300 Write block 1125 len 24576 data: 2 2 2 2 2 2
270 046/300 Read block 1993 len 28672
271 047/300 Read block 4820 len 32768
272 048/300 Write block 622 len 20480 data: 2 2 2 2 2
273 049/300 Flush
274 050/300 Read block 4323 len 4096
275 051/300 Flush
276 052/300 Write block 1209 len 40960 data: 2 2 2 2 2 2 2 2 2 2
277 053/300 Read block 2474 len 8192
278 054/300 Read block 1875 len 8192
279 055/300 Read block 2386 len 12288
280 056/300 Write block 2668 len 28672 data: 2 2 2 2 2 2 2
281 057/300 Read block 1265 len 32768
282 058/300 Write block 1185 len 36864 data: 2 2 2 2 2 2 2 2 2
283 059/300 Read block 2956 len 12288
284 060/300 Write block 3094 len 20480 data: 2 2 2 2 2
285 061/300 Write block 894 len 16384 data: 2 2 2 2
286 062/300 Read block 4187 len 36864
287 063/300 Read block 2499 len 8192
288 064/300 Read block 1762 len 12288
289 065/300 Read block 2306 len 24576
290 066/300 Write block 4508 len 20480 data: 2 2 2 2 2
291 067/300 Read block 4905 len 28672
292 068/300 Read block 4772 len 20480
293 069/300 Write block 638 len 12288 data: 2 2 2
294 070/300 Write block 1625 len 16384 data: 2 2 2 2
295 071/300 Read block 2682 len 8192
296 072/300 Read block 4216 len 8192
297 073/300 Write block 4092 len 20480 data: 2 2 2 2 2
298 074/300 Read block 4424 len 24576
299 075/300 Read block 3324 len 8192
300 076/300 Flush
301 077/300 Write block 105 len 36864 data: 2 2 2 2 2 2 2 2 2
302 078/300 Write block 3796 len 20480 data: 2 2 2 2 2
303 079/300 Read block 176 len 40960
304 080/300 Write block 4707 len 8192 data: 2 2
305 081/300 Write block 2261 len 24576 data: 2 2 2 2 2 2
306 082/300 Write block 4351 len 40960 data: 2 2 2 2 2 2 2 2 2 2
307 083/300 Read block 4864 len 16384
308 084/300 Write block 2239 len 32768 data: 2 2 2 2 2 2 2 2
309 085/300 Read block 4298 len 16384
310 086/300 Write block 2149 len 4096 data: 2
311 087/300 Read block 4218 len 28672
312 088/300 Flush
313 089/300 Read block 1612 len 32768
314 090/300 Write block 4044 len 4096 data: 2
315 091/300 Write block 3039 len 12288 data: 2 2 2
316 092/300 Read block 257 len 28672
317 093/300 Write block 4542 len 8192 data: 2 2
318 094/300 Flush
319 095/300 Write block 1007 len 12288 data: 2 2 2
320 096/300 Write block 1993 len 36864 data: 2 2 2 2 2 2 2 2 2
321 097/300 Flush
322 098/300 Flush
323 099/300 Read block 138 len 8192
324 100/300 Read block 4564 len 20480
325 101/300 Read block 62 len 20480
326 102/300 Read block 3562 len 28672
327 103/300 Write block 62 len 12288 data: 2 2 2
328 104/300 Write block 4985 len 28672 data: 2 2 2 2 2 2 2
329 105/300 Flush
330 106/300 Read block 109 len 40960
331 107/300 Write block 2585 len 24576 data: 2 2 2 2 2 2
332 108/300 Write block 3094 len 16384 data: 3 3 3 3
333 109/300 Write block 63 len 32768 data: 3 3 2 2 2 2 2 2
334 110/300 Read block 2093 len 32768
335 111/300 Read block 1699 len 4096
336 112/300 Read block 1782 len 16384
337 113/300 Read block 1667 len 16384
338 114/300 Flush
339 115/300 Read block 262 len 16384
340 116/300 Read block 2155 len 20480
341 117/300 Read block 3252 len 16384
342 118/300 Read block 1383 len 28672
343 119/300 Write block 4546 len 28672 data: 2 2 2 2 2 2 2
344 120/300 Write block 2480 len 12288 data: 2 2 2
345 121/300 Write block 4054 len 28672 data: 2 3 3 3 3 3 3
346 122/300 Read block 1150 len 16384
347 123/300 Write block 4645 len 32768 data: 2 2 2 2 2 2 2 2
348 124/300 Write block 2236 len 16384 data: 2 2 2 3
349 125/300 Read block 1307 len 20480
350 126/300 Read block 3490 len 32768
351 127/300 Write block 3582 len 36864 data: 2 2 2 2 2 2 2 2 2
352 128/300 Read block 1336 len 20480
353 129/300 Write block 425 len 32768 data: 2 2 2 2 2 2 2 2
354 130/300 Write block 1071 len 40960 data: 2 2 2 2 2 2 2 2 2 2
355 131/300 Read block 2188 len 4096
356 132/300 Write block 1043 len 16384 data: 2 2 2 2
357 133/300 Write block 998 len 32768 data: 2 2 2 2 2 2 2 2
358 134/300 Write block 2735 len 40960 data: 2 2 2 2 2 2 2 2 2 2
359 135/300 Read block 2962 len 12288
360 136/300 Write block 317 len 4096 data: 2
361 137/300 Write block 1967 len 20480 data: 2 2 2 2 2
362 138/300 Read block 4746 len 40960
363 139/300 Read block 4916 len 16384
364 140/300 Read block 414 len 16384
365 141/300 Write block 4105 len 4096 data: 2
366 142/300 Flush
367 143/300 Write block 2628 len 16384 data: 2 2 2 2
368 144/300 Read block 927 len 12288
369 145/300 Write block 215 len 20480 data: 2 2 2 2 2
370 146/300 Write block 463 len 4096 data: 2
371 147/300 Write block 814 len 28672 data: 2 2 2 2 2 2 2
372 148/300 Read block 4542 len 20480
373 149/300 Write block 4099 len 8192 data: 2 2
374 150/300 Write block 600 len 20480 data: 2 2 2 2 2
375 151/300 Read block 2939 len 32768
376 152/300 Read block 2876 len 12288
377 153/300 Read block 1300 len 8192
378 154/300 Write block 4006 len 28672 data: 2 2 2 2 2 2 2
379 155/300 Write block 3206 len 8192 data: 2 2
380 156/300 Write block 847 len 32768 data: 2 2 2 2 2 2 2 2
381 157/300 Read block 3294 len 24576
382 158/300 Read block 4721 len 32768
383 159/300 Write block 2003 len 24576 data: 2 2 2 2 2 2
384 160/300 Write block 4418 len 8192 data: 2 2
385 161/300 Write block 3947 len 28672 data: 2 2 2 3 2 2 2
386 162/300 Read block 4021 len 40960
387 163/300 Read block 2877 len 8192
388 164/300 Read block 924 len 36864
389 165/300 Write block 1115 len 20480 data: 2 2 2 2 2
390 166/300 Read block 2128 len 40960
391 167/300 Write block 1482 len 40960 data: 2 2 2 2 2 2 2 2 2 2
392 168/300 Write block 1376 len 28672 data: 2 2 2 2 2 2 2
393 169/300 Write block 639 len 12288 data: 3 3 2
394 170/300 Read block 1923 len 12288
395 171/300 Flush
396 172/300 Write block 669 len 36864 data: 2 2 2 2 2 2 2 2 2
397 173/300 Write block 3463 len 16384 data: 2 2 2 2
398 174/300 Read block 1989 len 28672
399 175/300 Write block 266 len 8192 data: 2 2
400 176/300 Write block 793 len 12288 data: 2 2 2
401 177/300 Read block 2978 len 36864
402 178/300 Read block 673 len 32768
403 179/300 Read block 2065 len 32768
404 180/300 Write block 1212 len 4096 data: 3
405 181/300 Read block 1604 len 8192
406 182/300 Read block 4120 len 8192
407 183/300 Read block 3962 len 24576
408 184/300 Flush
409 185/300 Flush
410 186/300 Write block 4980 len 40960 data: 2 2 2 2 2 3 3 3 3 3
411 187/300 Write block 2886 len 12288 data: 2 2 2
412 188/300 Write block 2175 len 4096 data: 2
413 189/300 Write block 3474 len 40960 data: 2 2 2 2 2 2 2 2 2 2
414 190/300 Read block 4100 len 40960
415 191/300 Write block 3408 len 40960 data: 2 2 2 3 3 3 3 3 3 3
416 192/300 Write block 1616 len 12288 data: 2 2 2
417 193/300 Flush
418 194/300 Read block 100 len 20480
419 195/300 Read block 4570 len 20480
420 196/300 Read block 2106 len 16384
421 197/300 Read block 3882 len 32768
422 198/300 Read block 1888 len 16384
423 199/300 Read block 4779 len 32768
424 200/300 Read block 3486 len 8192
425 201/300 Read block 1369 len 16384
426 202/300 Flush
427 203/300 Read block 1623 len 24576
428 204/300 Read block 2823 len 32768
429 205/300 Read block 3815 len 20480
430 206/300 Flush
431 207/300 Read block 3777 len 32768
432 208/300 Read block 4887 len 4096
433 209/300 Write block 591 len 12288 data: 2 2 2
434 210/300 Read block 2125 len 24576
435 211/300 Flush
436 212/300 Write block 1856 len 4096 data: 2
437 213/300 Read block 1890 len 40960
438 214/300 Flush
439 215/300 Read block 205 len 36864
440 216/300 Flush
441 217/300 Write block 2697 len 4096 data: 2
442 218/300 Read block 3646 len 28672
443 219/300 Read block 3869 len 28672
444 220/300 Read block 179 len 36864
445 221/300 Read block 4051 len 12288
446 222/300 Write block 4662 len 40960 data: 2 2 2 2 2 2 2 2 2 2
447 223/300 Read block 3898 len 40960
448 224/300 Write block 1530 len 32768 data: 2 2 2 2 2 2 2 2
449 225/300 Read block 3853 len 28672
450 226/300 Read block 3131 len 8192
451 227/300 Flush
452 228/300 Write block 4973 len 16384 data: 2 2 2 2
453 229/300 Write block 4949 len 20480 data: 2 2 2 2 2
454 230/300 Write block 3577 len 36864 data: 2 2 2 2 2 3 3 3 3
455 231/300 Read block 1059 len 4096
456 232/300 Read block 1738 len 28672
457 233/300 Read block 1999 len 4096
458 234/300 Flush
459 235/300 Write block 4872 len 36864 data: 2 2 2 2 2 2 2 2 2
460 236/300 Flush
461 237/300 Read block 1615 len 28672
462 238/300 Write block 3798 len 4096 data: 3
463 239/300 Read block 1769 len 40960
464 240/300 Write block 2274 len 40960 data: 2 2 2 2 2 2 2 2 2 2
465 241/300 Read block 4940 len 12288
466 242/300 Write block 904 len 4096 data: 2
467 243/300 Flush
468 244/300 Write block 1788 len 28672 data: 2 2 2 2 2 2 2
469 245/300 Read block 847 len 16384
470 246/300 Write block 1907 len 32768 data: 2 2 2 2 2 2 2 2
471 247/300 Read block 602 len 28672
472 248/300 Read block 4410 len 12288
473 249/300 Write block 1293 len 40960 data: 2 2 2 2 2 2 2 2 2 2
474 250/300 Read block 1959 len 4096
475 251/300 Read block 3102 len 16384
476 252/300 Read block 1887 len 8192
477 253/300 Read block 54 len 32768
478 254/300 Read block 689 len 40960
479 255/300 Read block 3596 len 32768
480 256/300 Write block 2473 len 40960 data: 2 2 2 2 2 2 2 3 3 3
481 257/300 Read block 2677 len 12288
482 258/300 Write block 441 len 4096 data: 2
483 259/300 Flush
484 260/300 Write block 1673 len 20480 data: 2 2 2 2 2
485 261/300 Read block 173 len 12288
486 262/300 Read block 860 len 28672
487 263/300 Read block 3370 len 16384
488 264/300 Write block 2948 len 40960 data: 2 2 2 2 2 2 2 2 2 2
489 265/300 Flush
490 266/300 Write block 2801 len 16384 data: 2 2 2 2
491 267/300 Read block 3748 len 36864
492 268/300 Write block 3573 len 28672 data: 2 2 2 2 3 3 3
493 269/300 Read block 1997 len 12288
494 270/300 Read block 4988 len 40960
495 271/300 Read block 4660 len 40960
496 272/300 Read block 3877 len 24576
497 273/300 Write block 776 len 12288 data: 2 2 2
498 274/300 Read block 4363 len 16384
499 275/300 Write block 3645 len 36864 data: 2 2 2 2 2 2 2 2 2
500 276/300 Read block 4759 len 40960
501 277/300 Read block 2762 len 36864
502 278/300 Write block 3984 len 8192 data: 2 2
503 279/300 Read block 473 len 40960
504 280/300 Flush
505 281/300 Write block 2975 len 32768 data: 2 2 2 2 2 2 2 2
506 282/300 Write block 4818 len 28672 data: 2 2 2 2 2 2 2
507 283/300 Read block 2984 len 36864
508 284/300 Read block 1445 len 8192
509 285/300 Write block 2214 len 36864 data: 2 2 2 2 2 2 2 2 2
510 286/300 Write block 753 len 28672 data: 2 2 2 2 2 2 2
511 287/300 Read block 860 len 36864
512 288/300 Read block 1111 len 28672
513 289/300 Read block 1629 len 4096
514 290/300 Read block 3181 len 8192
515 291/300 Read block 4393 len 36864
516 292/300 Read block 442 len 28672
517 293/300 Read block 3878 len 32768
518 294/300 Write block 3534 len 8192 data: 2 2
519 295/300 Read block 4771 len 32768
520 296/300 Read block 838 len 20480
521 297/300 Write block 2002 len 12288 data: 2 3 3
522 298/300 Write block 1741 len 16384 data: 2 2 2 2
523 299/300 Write block 3579 len 40960 data: 4 3 3 4 4 4 4 3 3 3
524 300/300 Read block 2971 len 36864
525 301/300 Write block 1053 len 8192 data: 2 2
526 Main task has work to do, go find it
527 got dsc Start(0)
528 start 0
529 Feb 21 06:45:27.457 INFO request completed, latency_us: 250, response_code: 204, uri: /start/cid/0, method: POST, req_id: 6f0307f0-7faf-4f27-b42c-a4309bdb6058, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
530 Replay: started 0, returned:Ok(())
531 ----------------------------------------------------------------
532 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
533 States: Offline Active Active
534 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
535 1000 Acked Flush 0 Sent Done Done false
536 1001 Acked Read 5 Sent Done Done false
537 1002 Acked Read 3 Sent Done Done false
538 1003 Acked Read 9 Sent Done Done false
539 1004 Acked Write 6 Sent Done Done false
540 1005 Acked Write 2 Sent Done Done false
541 1006 Acked Read 7 Sent Done Done false
542 1007 Acked Read 7 Sent Done Done false
543 1008 Acked Read 10 Sent Done Done false
544 1009 Acked Write 2 Sent Done Done false
545 1010 Acked Read 5 Sent Done Done false
546 1011 Acked Read 4 Sent Done Done false
547 1012 Acked Write 7 Sent Done Done false
548 1013 Acked Write 4 Sent Done Done false
549 1014 Acked Flush 0 Sent Done Done false
550 1015 Acked Read 7 Sent Done Done false
551 1016 Acked Read 3 Sent Done Done false
552 1017 Acked Read 1 Sent Done Done false
553 1018 Acked Read 4 Sent Done Done false
554 1019 Acked Read 2 Sent Done Done false
555 1020 Acked Read 6 Sent Done Done false
556 1021 Acked Flush 0 Sent Done Done false
557 1022 Acked Read 8 Sent Done Done false
558 1023 Acked Write 1 Sent Done Done false
559 1024 Acked Write 10 Sent Done Done false
560 1025 Acked Read 3 Sent Done Done false
561 1026 Acked Read 7 Sent Done Done false
562 1027 Acked Write 1 Sent Done Done false
563 1028 Acked Read 2 Sent Done Done false
564 1029 Acked Write 9 Sent Done Done false
565 1030 Acked Flush 0 Sent Done Done false
566 1031 Acked Write 3 Sent Done Done false
567 1032 Acked Read 4 Sent Done Done false
568 1033 Acked Read 8 Sent Done Done false
569 1034 Acked Write 6 Sent Done Done false
570 1035 Acked Read 6 Sent Done Done false
571 1036 Acked Read 7 Sent Done Done false
572 1037 Acked Read 5 Sent Done Done false
573 1038 Acked Write 7 Sent Done Done false
574 1039 Acked Read 7 Sent Done Done false
575 1040 Acked Write 9 Sent Done Done false
576 1041 Acked Flush 0 Sent Done Done false
577 1042 Acked Read 10 Sent Done Done false
578 1043 Acked Write 3 Sent Done Done false
579 1044 Acked Write 6 Sent Done Done false
580 1045 Acked Read 7 Sent Done Done false
581 1046 Acked Read 8 Sent Done Done false
582 1047 Acked Write 5 Sent Done Done false
583 1048 Acked Flush 0 Sent Done Done false
584 1049 Acked Read 1 Sent Done Done false
585 1050 Acked Flush 0 Sent Done Done false
586 1051 Acked Write 10 Sent Done Done false
587 1052 Acked Read 2 Sent Done Done false
588 1053 Acked Read 2 Sent Done Done false
589 1054 Acked Read 3 Sent Done Done false
590 1055 Acked Write 7 Sent Done Done false
591 1056 Acked Read 8 Sent Done Done false
592 1057 Acked Write 9 Sent Done Done false
593 1058 Acked Read 3 Sent Done Done false
594 1059 Acked Write 5 Sent Done Done false
595 1060 Acked Write 4 Sent Done Done false
596 1061 Acked Read 9 Sent Done Done false
597 1062 Acked Read 2 Sent Done Done false
598 1063 Acked Read 3 Sent Done Done false
599 1064 Acked Read 6 Sent Done Done false
600 1065 Acked Write 5 Sent Done Done false
601 1066 Acked Read 7 Sent Done Done false
602 1067 Acked Read 5 Sent Done Done false
603 1068 Acked Write 3 Sent Done Done false
604 1069 Acked Write 4 Sent Done Done false
605 1070 Acked Read 2 Sent Done Done false
606 1071 Acked Read 2 Sent Done Done false
607 1072 Acked Write 5 Sent Done Done false
608 1073 Acked Read 6 Sent Done Done false
609 1074 Acked Read 2 Sent Done Done false
610 1075 Acked Flush 0 Sent Done Done false
611 1076 Acked Write 9 Sent Done Done false
612 1077 Acked Write 5 Sent Done Done false
613 1078 Acked Read 10 Sent Done Done false
614 1079 Acked Write 2 Sent Done Done false
615 1080 Acked Write 6 Sent Done Done false
616 1081 Acked Write 10 Sent Done Done false
617 1082 Acked Read 4 Sent Done Done false
618 1083 Acked Write 8 Sent Done Done false
619 1084 Acked Read 4 Sent Done Done false
620 1085 Acked Write 1 Sent Done Done false
621 1086 Acked Read 7 Sent Done Done false
622 1087 Acked Flush 0 Sent Done Done false
623 1088 Acked Read 8 Sent Done Done false
624 1089 Acked Write 1 Sent Done Done false
625 1090 Acked Write 3 Sent Done Done false
626 1091 Acked Read 7 Sent Done Done false
627 1092 Acked Write 2 Sent Done Done false
628 1093 Acked Flush 0 Sent Done Done false
629 1094 Acked Write 3 Sent Done Done false
630 1095 Acked Write 9 Sent Done Done false
631 1096 Acked Flush 0 Sent Done Done false
632 1097 Acked Flush 0 Sent Done Done false
633 1098 Acked Read 2 Sent Done Done false
634 1099 Acked Read 5 Sent Done Done false
635 1100 Acked Read 5 Sent Done Done false
636 1101 Acked Read 7 Sent Done Done false
637 1102 Acked Write 3 Sent Done Done false
638 1103 Acked Write 7 Sent Done Done false
639 1104 Acked Flush 0 Sent Done Done false
640 1105 Acked Read 10 Sent Done Done false
641 1106 Acked Write 6 Sent Done Done false
642 1107 Acked Write 4 Sent Done Done false
643 1108 Acked Write 8 Sent Done Done false
644 1109 Acked Read 8 Sent Done Done false
645 1110 Acked Read 1 Sent Done Done false
646 1111 Acked Read 4 Sent Done Done false
647 1112 Acked Read 4 Sent Done Done false
648 1113 Acked Flush 0 Sent Done Done false
649 1114 Acked Read 4 Sent Done Done false
650 1115 Acked Read 5 Sent Done Done false
651 1116 Acked Read 4 Sent Done Done false
652 1117 Acked Read 7 Sent Done Done false
653 1118 Acked Write 7 Sent Done Done false
654 1119 Acked Write 3 Sent Done Done false
655 1120 Acked Write 7 Sent Done Done false
656 1121 Acked Read 4 Sent Done Done false
657 1122 Acked Write 8 Sent Done Done false
658 1123 Acked Write 4 Sent Done Done false
659 1124 Acked Read 5 Sent Done Done false
660 1125 Acked Read 8 Sent Done Done false
661 1126 Acked Write 9 Sent Done Done false
662 1127 Acked Read 5 Sent Done Done false
663 1128 Acked Write 8 Sent Done Done false
664 1129 Acked Write 10 Sent Done Done false
665 1130 Acked Read 1 Sent Done Done false
666 1131 Acked Write 4 Sent Done Done false
667 1132 Acked Write 8 Sent Done Done false
668 1133 Acked Write 10 Sent Done Done false
669 1134 Acked Read 3 Sent Done Done false
670 1135 Acked Write 1 Sent Done Done false
671 1136 Acked Write 5 Sent Done Done false
672 1137 Acked Read 10 Sent Done Done false
673 1138 Acked Read 4 Sent Done Done false
674 1139 Acked Read 4 Sent Done Done false
675 1140 Acked Write 1 Sent Done Done false
676 1141 Acked Flush 0 Sent Done Done false
677 1142 Acked Write 4 Sent Done Done false
678 1143 Acked Read 3 Sent Done Done false
679 1144 Acked Write 5 Sent Done Done false
680 1145 Acked Write 1 Sent Done Done false
681 1146 Acked Write 7 Sent Done Done false
682 1147 Acked Read 5 Sent Done Done false
683 1148 Acked Write 2 Sent Done Done false
684 1149 Acked Write 5 Sent Done Done false
685 1150 Acked Read 8 Sent Done Done false
686 1151 Acked Read 3 Sent Done Done false
687 1152 Acked Read 2 Sent Done Done false
688 1153 Acked Write 7 Sent Done Done false
689 1154 Acked Write 2 Sent Done Done false
690 1155 Acked Write 8 Sent Done Done false
691 1156 Acked Read 6 Sent Done Done false
692 1157 Acked Read 8 Sent Done Done false
693 1158 Acked Write 6 Sent Done Done false
694 1159 Acked Write 2 Sent Done Done false
695 1160 Acked Write 7 Sent Done Done false
696 1161 Acked Read 10 Sent Done Done false
697 1162 Acked Read 2 Sent Done Done false
698 1163 Acked Read 9 Sent Done Done false
699 1164 Acked Write 5 Sent Done Done false
700 1165 Acked Read 10 Sent Done Done false
701 1166 Acked Write 10 Sent Done Done false
702 1167 Acked Write 7 Sent Done Done false
703 1168 Acked Write 3 Sent Done Done false
704 1169 Acked Read 3 Sent Done Done false
705 1170 Acked Flush 0 Sent Done Done false
706 1171 Acked Write 9 Sent Done Done false
707 1172 Acked Write 4 Sent Done Done false
708 1173 Acked Read 7 Sent Done Done false
709 1174 Acked Write 2 Sent Done Done false
710 1175 Acked Write 3 Sent Done Done false
711 1176 Acked Read 9 Sent Done Done false
712 1177 Acked Read 8 Sent Done Done false
713 1178 Acked Read 8 Sent Done Done false
714 1179 Acked Write 1 Sent Done Done false
715 1180 Acked Read 2 Sent Done Done false
716 1181 Acked Read 2 Sent Done Done false
717 1182 Acked Read 6 Sent Done Done false
718 1183 Acked Flush 0 Sent Done Done false
719 1184 Acked Flush 0 Sent Done Done false
720 1185 Acked Write 10 Sent Done Done false
721 1186 Acked Write 3 Sent Done Done false
722 1187 Acked Write 1 Sent Done Done false
723 1188 Acked Write 10 Sent Done Done false
724 1189 Acked Read 10 Sent Done Done false
725 1190 Acked Write 10 Sent Done Done false
726 1191 Acked Write 3 Sent Done Done false
727 1192 Acked Flush 0 Sent Done Done false
728 1193 Acked Read 5 Sent Done Done false
729 1194 Acked Read 5 Sent Done Done false
730 1195 Acked Read 4 Sent Done Done false
731 1196 Acked Read 8 Sent Done Done false
732 1197 Acked Read 4 Sent Done Done false
733 1198 Acked Read 8 Sent Done Done false
734 1199 Acked Read 2 Sent Done Done false
735 1200 Acked Read 4 Sent Done Done false
736 1201 Acked Flush 0 Sent Done Done false
737 1202 Acked Read 6 Sent Done Done false
738 1203 Acked Read 8 Sent Done Done false
739 1204 Acked Read 5 Sent Done Done false
740 1205 Acked Flush 0 Sent Done Done false
741 1206 Acked Read 8 Sent Done Done false
742 1207 Acked Read 1 Sent Done Done false
743 1208 Acked Write 3 Sent Done Done false
744 1209 Acked Read 6 Sent Done Done false
745 1210 Acked Flush 0 Sent Done Done false
746 1211 Acked Write 1 Sent Done Done false
747 1212 Acked Read 10 Sent Done Done false
748 1213 Acked Flush 0 Sent Done Done false
749 1214 Acked Read 9 Sent Done Done false
750 1215 Acked Flush 0 Sent Done Done false
751 1216 Acked Write 1 Sent Done Done false
752 1217 Acked Read 7 Sent Done Done false
753 1218 Acked Read 7 Sent Done Done false
754 1219 Acked Read 9 Sent Done Done false
755 1220 Acked Read 3 Sent Done Done false
756 1221 Acked Write 10 Sent Done Done false
757 1222 Acked Read 10 Sent Done Done false
758 1223 Acked Write 8 Sent Done Done false
759 1224 Acked Read 7 Sent Done Done false
760 1225 Acked Read 2 Sent Done Done false
761 1226 Acked Flush 0 Sent Done Done false
762 1227 Acked Write 4 Sent Done Done false
763 1228 Acked Write 5 Sent Done Done false
764 1229 Acked Write 9 Sent Done Done false
765 1230 Acked Read 1 Sent Done Done false
766 1231 Acked Read 7 Sent Done Done false
767 1232 Acked Read 1 Sent Done Done false
768 1233 Acked Flush 0 Sent Done Done false
769 1234 Acked Write 9 Sent Done Done false
770 1235 Acked Flush 0 Sent Done Done false
771 1236 Acked Read 7 Sent Done Done false
772 1237 Acked Write 1 Sent Done Done false
773 1238 Acked Read 10 Sent Done Done false
774 1239 Acked Write 10 Sent Done Done false
775 1240 Acked Read 3 Sent Done Done false
776 1241 Acked Write 1 Sent Done Done false
777 1242 Acked Flush 0 Sent Done Done false
778 1243 Acked Write 7 Sent Done Done false
779 1244 Acked Read 4 Sent Done Done false
780 1245 Acked Write 8 Sent Done Done false
781 1246 Acked Read 7 Sent Done Done false
782 1247 Acked Read 3 Sent Done Done false
783 1248 Acked Write 10 Sent Done Done false
784 1249 Acked Read 1 Sent Done Done false
785 1250 Acked Read 4 Sent Done Done false
786 1251 Acked Read 2 Sent Done Done false
787 1252 Acked Read 8 Sent Done Done false
788 1253 Acked Read 10 Sent Done Done false
789 1254 Acked Read 8 Sent Done Done false
790 1255 Acked Write 10 Sent Done Done false
791 1256 Acked Read 3 Sent Done Done false
792 1257 Acked Write 1 Sent Done Done false
793 1258 Acked Flush 0 Sent Done Done false
794 1259 Acked Write 5 Sent Done Done false
795 1260 Acked Read 3 Sent Done Done false
796 1261 Acked Read 7 Sent Done Done false
797 1262 Acked Read 4 Sent Done Done false
798 1263 Acked Write 10 Sent Done Done false
799 1264 Acked Flush 0 Sent Done Done false
800 1265 Acked Write 4 Sent Done Done false
801 1266 Acked Read 9 Sent Done Done false
802 1267 Acked Write 7 Sent Done Done false
803 1268 Acked Read 3 Sent Done Done false
804 1269 Acked Read 10 Sent Done Done false
805 1270 Acked Read 10 Sent Done Done false
806 1271 Acked Read 6 Sent Done Done false
807 1272 Acked Write 3 Sent Done Done false
808 1273 Acked Read 4 Sent Done Done false
809 1274 Acked Write 9 Sent Done Done false
810 1275 Acked Read 10 Sent Done Done false
811 1276 Acked Read 9 Sent Done Done false
812 1277 Acked Write 2 Sent Done Done false
813 1278 Acked Read 10 Sent Done Done false
814 1279 Acked Flush 0 Sent Done Done false
815 1280 Acked Write 8 Sent Done Done false
816 1281 Acked Write 7 Sent Done Done false
817 1282 Acked Read 9 Sent Done Done false
818 1283 Acked Read 2 Sent Done Done false
819 1284 Acked Write 9 Sent Done Done false
820 1285 Acked Write 7 Sent Done Done false
821 1286 Acked Read 9 Sent Done Done false
822 1287 Acked Read 7 Sent Done Done false
823 1288 Acked Read 1 Sent Done Done false
824 1289 Acked Read 2 Sent Done Done false
825 1290 Acked Read 9 Sent Done Done false
826 1291 Acked Read 7 Sent Done Done false
827 1292 Acked Read 8 Sent Done Done false
828 1293 Acked Write 2 Sent Done Done false
829 1294 Acked Read 8 Sent Done Done false
830 1295 Acked Read 5 Sent Done Done false
831 1296 Acked Write 3 Sent Done Done false
832 1297 Acked Write 4 Sent Done Done false
833 1298 Acked Write 10 Sent Done Done false
834 1299 Acked Read 9 Sent Done Done false
835 1300 Acked Write 2 Sent Done Done false
836 STATES DS:0 DS:1 DS:2 TOTAL
837 Sent 301 0 0 301
838 Done 0 301 301 602
839 Skipped 0 0 0 0
840 Error 0 0 0 0
841 Last Flush: None Some(JobId(1279)) Some(JobId(1279))
842 Downstairs last five completed:
843 Upstairs last five completed: 1300 1299 1298 1297 1296
844 CLIENT: Up:0 ds:301 act:2
8452025-02-21T06:45:27.959ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
8462025-02-21T06:45:28.670ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
8472025-02-21T06:45:29.671ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
8482025-02-21T06:45:30.673ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
849 [8810] got start action
850 [8810] I am going to restart
851 Starting downstairs at port 8810
852 Make output file at "/tmp/dsc/downstairs-8810.txt"
853 [8810][0] reports Starting
854 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1207)
855 [8810][0] reports Running
856 ----------------------------------------------------------------
857 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
858 States: Offline Active Active
859 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
860 1000 Acked Flush 0 Sent Done Done false
861 1001 Acked Read 5 Sent Done Done false
862 1002 Acked Read 3 Sent Done Done false
863 1003 Acked Read 9 Sent Done Done false
864 1004 Acked Write 6 Sent Done Done false
865 1005 Acked Write 2 Sent Done Done false
866 1006 Acked Read 7 Sent Done Done false
867 1007 Acked Read 7 Sent Done Done false
868 1008 Acked Read 10 Sent Done Done false
869 1009 Acked Write 2 Sent Done Done false
870 1010 Acked Read 5 Sent Done Done false
871 1011 Acked Read 4 Sent Done Done false
872 1012 Acked Write 7 Sent Done Done false
873 1013 Acked Write 4 Sent Done Done false
874 1014 Acked Flush 0 Sent Done Done false
875 1015 Acked Read 7 Sent Done Done false
876 1016 Acked Read 3 Sent Done Done false
877 1017 Acked Read 1 Sent Done Done false
878 1018 Acked Read 4 Sent Done Done false
879 1019 Acked Read 2 Sent Done Done false
880 1020 Acked Read 6 Sent Done Done false
881 1021 Acked Flush 0 Sent Done Done false
882 1022 Acked Read 8 Sent Done Done false
883 1023 Acked Write 1 Sent Done Done false
884 1024 Acked Write 10 Sent Done Done false
885 1025 Acked Read 3 Sent Done Done false
886 1026 Acked Read 7 Sent Done Done false
887 1027 Acked Write 1 Sent Done Done false
888 1028 Acked Read 2 Sent Done Done false
889 1029 Acked Write 9 Sent Done Done false
890 1030 Acked Flush 0 Sent Done Done false
891 1031 Acked Write 3 Sent Done Done false
892 1032 Acked Read 4 Sent Done Done false
893 1033 Acked Read 8 Sent Done Done false
894 1034 Acked Write 6 Sent Done Done false
895 1035 Acked Read 6 Sent Done Done false
896 1036 Acked Read 7 Sent Done Done false
897 1037 Acked Read 5 Sent Done Done false
898 1038 Acked Write 7 Sent Done Done false
899 1039 Acked Read 7 Sent Done Done false
900 1040 Acked Write 9 Sent Done Done false
901 1041 Acked Flush 0 Sent Done Done false
902 1042 Acked Read 10 Sent Done Done false
903 1043 Acked Write 3 Sent Done Done false
904 1044 Acked Write 6 Sent Done Done false
905 1045 Acked Read 7 Sent Done Done false
906 1046 Acked Read 8 Sent Done Done false
907 1047 Acked Write 5 Sent Done Done false
908 1048 Acked Flush 0 Sent Done Done false
909 1049 Acked Read 1 Sent Done Done false
910 1050 Acked Flush 0 Sent Done Done false
911 1051 Acked Write 10 Sent Done Done false
912 1052 Acked Read 2 Sent Done Done false
913 1053 Acked Read 2 Sent Done Done false
914 1054 Acked Read 3 Sent Done Done false
915 1055 Acked Write 7 Sent Done Done false
916 1056 Acked Read 8 Sent Done Done false
917 1057 Acked Write 9 Sent Done Done false
918 1058 Acked Read 3 Sent Done Done false
919 1059 Acked Write 5 Sent Done Done false
920 1060 Acked Write 4 Sent Done Done false
921 1061 Acked Read 9 Sent Done Done false
922 1062 Acked Read 2 Sent Done Done false
923 1063 Acked Read 3 Sent Done Done false
924 1064 Acked Read 6 Sent Done Done false
925 1065 Acked Write 5 Sent Done Done false
926 1066 Acked Read 7 Sent Done Done false
927 1067 Acked Read 5 Sent Done Done false
928 1068 Acked Write 3 Sent Done Done false
929 1069 Acked Write 4 Sent Done Done false
930 1070 Acked Read 2 Sent Done Done false
931 1071 Acked Read 2 Sent Done Done false
932 1072 Acked Write 5 Sent Done Done false
933 1073 Acked Read 6 Sent Done Done false
934 1074 Acked Read 2 Sent Done Done false
935 1075 Acked Flush 0 Sent Done Done false
936 1076 Acked Write 9 Sent Done Done false
937 1077 Acked Write 5 Sent Done Done false
938 1078 Acked Read 10 Sent Done Done false
939 1079 Acked Write 2 Sent Done Done false
940 1080 Acked Write 6 Sent Done Done false
941 1081 Acked Write 10 Sent Done Done false
942 1082 Acked Read 4 Sent Done Done false
943 1083 Acked Write 8 Sent Done Done false
944 1084 Acked Read 4 Sent Done Done false
945 1085 Acked Write 1 Sent Done Done false
946 1086 Acked Read 7 Sent Done Done false
947 1087 Acked Flush 0 Sent Done Done false
948 1088 Acked Read 8 Sent Done Done false
949 1089 Acked Write 1 Sent Done Done false
950 1090 Acked Write 3 Sent Done Done false
951 1091 Acked Read 7 Sent Done Done false
952 1092 Acked Write 2 Sent Done Done false
953 1093 Acked Flush 0 Sent Done Done false
954 1094 Acked Write 3 Sent Done Done false
955 1095 Acked Write 9 Sent Done Done false
956 1096 Acked Flush 0 Sent Done Done false
957 1097 Acked Flush 0 Sent Done Done false
958 1098 Acked Read 2 Sent Done Done false
959 1099 Acked Read 5 Sent Done Done false
960 1100 Acked Read 5 Sent Done Done false
961 1101 Acked Read 7 Sent Done Done false
962 1102 Acked Write 3 Sent Done Done false
963 1103 Acked Write 7 Sent Done Done false
964 1104 Acked Flush 0 Sent Done Done false
965 1105 Acked Read 10 Sent Done Done false
966 1106 Acked Write 6 Sent Done Done false
967 1107 Acked Write 4 Sent Done Done false
968 1108 Acked Write 8 Sent Done Done false
969 1109 Acked Read 8 Sent Done Done false
970 1110 Acked Read 1 Sent Done Done false
971 1111 Acked Read 4 Sent Done Done false
972 1112 Acked Read 4 Sent Done Done false
973 1113 Acked Flush 0 Sent Done Done false
974 1114 Acked Read 4 Sent Done Done false
975 1115 Acked Read 5 Sent Done Done false
976 1116 Acked Read 4 Sent Done Done false
977 1117 Acked Read 7 Sent Done Done false
978 1118 Acked Write 7 Sent Done Done false
979 1119 Acked Write 3 Sent Done Done false
980 1120 Acked Write 7 Sent Done Done false
981 1121 Acked Read 4 Sent Done Done false
982 1122 Acked Write 8 Sent Done Done false
983 1123 Acked Write 4 Sent Done Done false
984 1124 Acked Read 5 Sent Done Done false
985 1125 Acked Read 8 Sent Done Done false
986 1126 Acked Write 9 Sent Done Done false
987 1127 Acked Read 5 Sent Done Done false
988 1128 Acked Write 8 Sent Done Done false
989 1129 Acked Write 10 Sent Done Done false
990 1130 Acked Read 1 Sent Done Done false
991 1131 Acked Write 4 Sent Done Done false
992 1132 Acked Write 8 Sent Done Done false
993 1133 Acked Write 10 Sent Done Done false
994 1134 Acked Read 3 Sent Done Done false
995 1135 Acked Write 1 Sent Done Done false
996 1136 Acked Write 5 Sent Done Done false
997 1137 Acked Read 10 Sent Done Done false
998 1138 Acked Read 4 Sent Done Done false
999 1139 Acked Read 4 Sent Done Done false
1000 1140 Acked Write 1 Sent Done Done false
1001 1141 Acked Flush 0 Sent Done Done false
1002 1142 Acked Write 4 Sent Done Done false
1003 1143 Acked Read 3 Sent Done Done false
1004 1144 Acked Write 5 Sent Done Done false
1005 1145 Acked Write 1 Sent Done Done false
1006 1146 Acked Write 7 Sent Done Done false
1007 1147 Acked Read 5 Sent Done Done false
1008 1148 Acked Write 2 Sent Done Done false
1009 1149 Acked Write 5 Sent Done Done false
1010 1150 Acked Read 8 Sent Done Done false
1011 1151 Acked Read 3 Sent Done Done false
1012 1152 Acked Read 2 Sent Done Done false
1013 1153 Acked Write 7 Sent Done Done false
1014 1154 Acked Write 2 Sent Done Done false
1015 1155 Acked Write 8 Sent Done Done false
1016 1156 Acked Read 6 Sent Done Done false
1017 1157 Acked Read 8 Sent Done Done false
1018 1158 Acked Write 6 Sent Done Done false
1019 1159 Acked Write 2 Sent Done Done false
1020 1160 Acked Write 7 Sent Done Done false
1021 1161 Acked Read 10 Sent Done Done false
1022 1162 Acked Read 2 Sent Done Done false
1023 1163 Acked Read 9 Sent Done Done false
1024 1164 Acked Write 5 Sent Done Done false
1025 1165 Acked Read 10 Sent Done Done false
1026 1166 Acked Write 10 Sent Done Done false
1027 1167 Acked Write 7 Sent Done Done false
1028 1168 Acked Write 3 Sent Done Done false
1029 1169 Acked Read 3 Sent Done Done false
1030 1170 Acked Flush 0 Sent Done Done false
1031 1171 Acked Write 9 Sent Done Done false
1032 1172 Acked Write 4 Sent Done Done false
1033 1173 Acked Read 7 Sent Done Done false
1034 1174 Acked Write 2 Sent Done Done false
1035 1175 Acked Write 3 Sent Done Done false
1036 1176 Acked Read 9 Sent Done Done false
1037 1177 Acked Read 8 Sent Done Done false
1038 1178 Acked Read 8 Sent Done Done false
1039 1179 Acked Write 1 Sent Done Done false
1040 1180 Acked Read 2 Sent Done Done false
1041 1181 Acked Read 2 Sent Done Done false
1042 1182 Acked Read 6 Sent Done Done false
1043 1183 Acked Flush 0 Sent Done Done false
1044 1184 Acked Flush 0 Sent Done Done false
1045 1185 Acked Write 10 Sent Done Done false
1046 1186 Acked Write 3 Sent Done Done false
1047 1187 Acked Write 1 Sent Done Done false
1048 1188 Acked Write 10 Sent Done Done false
1049 1189 Acked Read 10 Sent Done Done false
1050 1190 Acked Write 10 Sent Done Done false
1051 1191 Acked Write 3 Sent Done Done false
1052 1192 Acked Flush 0 Sent Done Done false
1053 1193 Acked Read 5 Sent Done Done false
1054 1194 Acked Read 5 Sent Done Done false
1055 1195 Acked Read 4 Sent Done Done false
1056 1196 Acked Read 8 Sent Done Done false
1057 1197 Acked Read 4 Sent Done Done false
1058 1198 Acked Read 8 Sent Done Done false
1059 1199 Acked Read 2 Sent Done Done false
1060 1200 Acked Read 4 Sent Done Done false
1061 1201 Acked Flush 0 Sent Done Done false
1062 1202 Acked Read 6 Sent Done Done false
1063 1203 Acked Read 8 Sent Done Done false
1064 1204 Acked Read 5 Sent Done Done false
1065 1205 Acked Flush 0 Sent Done Done false
1066 1206 Acked Read 8 Sent Done Done false
1067 1207 Acked Read 1 Sent Done Done false
1068 1208 Acked Write 3 Sent Done Done false
1069 1209 Acked Read 6 Sent Done Done false
1070 1210 Acked Flush 0 Sent Done Done false
1071 1211 Acked Write 1 Sent Done Done false
1072 1212 Acked Read 10 Sent Done Done false
1073 1213 Acked Flush 0 Sent Done Done false
1074 1214 Acked Read 9 Sent Done Done false
1075 1215 Acked Flush 0 Sent Done Done false
1076 1216 Acked Write 1 Sent Done Done false
1077 1217 Acked Read 7 Sent Done Done false
1078 1218 Acked Read 7 Sent Done Done false
1079 1219 Acked Read 9 Sent Done Done false
1080 1220 Acked Read 3 Sent Done Done false
1081 1221 Acked Write 10 Sent Done Done false
1082 1222 Acked Read 10 Sent Done Done false
1083 1223 Acked Write 8 Sent Done Done false
1084 1224 Acked Read 7 Sent Done Done false
1085 1225 Acked Read 2 Sent Done Done false
1086 1226 Acked Flush 0 Sent Done Done false
1087 1227 Acked Write 4 Sent Done Done false
1088 1228 Acked Write 5 Sent Done Done false
1089 1229 Acked Write 9 Sent Done Done false
1090 1230 Acked Read 1 Sent Done Done false
1091 1231 Acked Read 7 Sent Done Done false
1092 1232 Acked Read 1 Sent Done Done false
1093 1233 Acked Flush 0 Sent Done Done false
1094 1234 Acked Write 9 Sent Done Done false
1095 1235 Acked Flush 0 Sent Done Done false
1096 1236 Acked Read 7 Sent Done Done false
1097 1237 Acked Write 1 Sent Done Done false
1098 1238 Acked Read 10 Sent Done Done false
1099 1239 Acked Write 10 Sent Done Done false
1100 1240 Acked Read 3 Sent Done Done false
1101 1241 Acked Write 1 Sent Done Done false
1102 1242 Acked Flush 0 Sent Done Done false
1103 1243 Acked Write 7 Sent Done Done false
1104 1244 Acked Read 4 Sent Done Done false
1105 1245 Acked Write 8 Sent Done Done false
1106 1246 Acked Read 7 Sent Done Done false
1107 1247 Acked Read 3 Sent Done Done false
1108 1248 Acked Write 10 Sent Done Done false
1109 1249 Acked Read 1 Sent Done Done false
1110 1250 Acked Read 4 Sent Done Done false
1111 1251 Acked Read 2 Sent Done Done false
1112 1252 Acked Read 8 Sent Done Done false
1113 1253 Acked Read 10 Sent Done Done false
1114 1254 Acked Read 8 Sent Done Done false
1115 1255 Acked Write 10 Sent Done Done false
1116 1256 Acked Read 3 Sent Done Done false
1117 1257 Acked Write 1 Sent Done Done false
1118 1258 Acked Flush 0 Sent Done Done false
1119 1259 Acked Write 5 Sent Done Done false
1120 1260 Acked Read 3 Sent Done Done false
1121 1261 Acked Read 7 Sent Done Done false
1122 1262 Acked Read 4 Sent Done Done false
1123 1263 Acked Write 10 Sent Done Done false
1124 1264 Acked Flush 0 Sent Done Done false
1125 1265 Acked Write 4 Sent Done Done false
1126 1266 Acked Read 9 Sent Done Done false
1127 1267 Acked Write 7 Sent Done Done false
1128 1268 Acked Read 3 Sent Done Done false
1129 1269 Acked Read 10 Sent Done Done false
1130 1270 Acked Read 10 Sent Done Done false
1131 1271 Acked Read 6 Sent Done Done false
1132 1272 Acked Write 3 Sent Done Done false
1133 1273 Acked Read 4 Sent Done Done false
1134 1274 Acked Write 9 Sent Done Done false
1135 1275 Acked Read 10 Sent Done Done false
1136 1276 Acked Read 9 Sent Done Done false
1137 1277 Acked Write 2 Sent Done Done false
1138 1278 Acked Read 10 Sent Done Done false
1139 1279 Acked Flush 0 Sent Done Done false
1140 1280 Acked Write 8 Sent Done Done false
1141 1281 Acked Write 7 Sent Done Done false
1142 1282 Acked Read 9 Sent Done Done false
1143 1283 Acked Read 2 Sent Done Done false
1144 1284 Acked Write 9 Sent Done Done false
1145 1285 Acked Write 7 Sent Done Done false
1146 1286 Acked Read 9 Sent Done Done false
1147 1287 Acked Read 7 Sent Done Done false
1148 1288 Acked Read 1 Sent Done Done false
1149 1289 Acked Read 2 Sent Done Done false
1150 1290 Acked Read 9 Sent Done Done false
1151 1291 Acked Read 7 Sent Done Done false
1152 1292 Acked Read 8 Sent Done Done false
1153 1293 Acked Write 2 Sent Done Done false
1154 1294 Acked Read 8 Sent Done Done false
1155 1295 Acked Read 5 Sent Done Done false
1156 1296 Acked Write 3 Sent Done Done false
1157 1297 Acked Write 4 Sent Done Done false
1158 1298 Acked Write 10 Sent Done Done false
1159 1299 Acked Read 9 Sent Done Done false
1160 1300 Acked Write 2 Sent Done Done false
1161 STATES DS:0 DS:1 DS:2 TOTAL
1162 Sent 301 0 0 301
1163 Done 0 301 301 602
1164 Skipped 0 0 0 0
1165 Error 0 0 0 0
1166 Last Flush: None Some(JobId(1279)) Some(JobId(1279))
1167 Downstairs last five completed:
1168 Upstairs last five completed: 1300 1299 1298 1297 1296
1169 CLIENT: Up:0 ds:301 act:2
11702025-02-21T06:45:31.962ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11712025-02-21T06:45:32.676ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11722025-02-21T06:45:33.206ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11732025-02-21T06:45:33.206ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55161) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11742025-02-21T06:45:33.207ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11752025-02-21T06:45:33.207ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11762025-02-21T06:45:33.207ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11772025-02-21T06:45:33.207ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11782025-02-21T06:45:33.207ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11792025-02-21T06:45:33.207ZINFOcrucible: [0] client re-new 301 jobs since flush None = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11802025-02-21T06:45:33.208ZINFOcrucible: [0] Replayed 301 jobs since flush: None = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1181 ----------------------------------------------------------------
1182 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1183 Downstairs last five completed: 1301 1300 1299 1298 1297
1184 Upstairs last five completed: 1301 1300 1299 1298 1297
1185 CLIENT: Up:0 ds:0 act:3
1186 Replay: All jobs finished
1187 Main task has work to do, go find it
1188 got dsc Stop(0)
1189 stop 0
1190 Feb 21 06:45:35.463 INFO[8810] Got stop action so:false kr:false
1191 request completed, latency_us: 323, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 3849886e-8c8f-4108-aeb0-752cce9b48df, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
11922025-02-21T06:45:35.464ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11932025-02-21T06:45:35.464ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11942025-02-21T06:45:35.464ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1195 Feb 21 06:45:35.464 INFO request completed, latency_us: 162, response_code: 200, uri: /state/cid/0, method: GET, req_id: c8357aee-b676-403c-ab0f-a4dbbe6344be, remote_addr: 127.0.0.1:53848, local_addr: 127.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:45:35.464823246Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
1196 0.0.1:9998
1197 [8810] Exited with: ExitStatus(unix_wait_status(9))
1198 [8810][0] reports Exit
1199 Feb 21 06:45:39.467 INFO request completed, latency_us: 193, response_code: 200, uri: /state/cid/0, method: GET, req_id: b5926e92-ba84-4d5e-9216-8b9fa784dc22, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
1200 001/300 Read block 1095 len 32768
1201 002/300 Write block 2301 len 28672 data: 2 2 2 2 2 2 2
1202 003/300 Write block 1901 len 32768 data: 2 2 2 2 2 2 3 3
1203 004/300 Write block 4405 len 36864 data: 2 2 2 2 2 2 2 2 2
1204 005/300 Write block 1057 len 20480 data: 2 2 2 2 2
1205 006/300 Read block 536 len 36864
1206 007/300 Flush
1207 008/300 Read block 2315 len 36864
1208 009/300 Read block 263 len 8192
1209 010/300 Write block 4327 len 12288 data: 2 2 2
1210 011/300 Write block 2834 len 32768 data: 2 2 2 2 2 2 2 2
1211 012/300 Write block 1576 len 32768 data: 2 2 2 2 2 2 2 2
1212 013/300 Read block 4264 len 28672
1213 014/300 Read block 1291 len 12288
1214 015/300 Read block 519 len 36864
1215 016/300 Write block 3496 len 24576 data: 2 2 2 2 2 2
1216 017/300 Read block 2502 len 16384
1217 018/300 Read block 241 len 36864
1218 019/300 Read block 3715 len 20480
1219 020/300 Write block 901 len 24576 data: 2 2 2 3 2 2
1220 021/300 Write block 3010 len 16384 data: 2 2 2 2
1221 022/300 Read block 2200 len 20480
1222 023/300 Read block 2323 len 28672
1223 024/300 Write block 4742 len 8192 data: 2 2
1224 025/300 Read block 1879 len 24576
1225 026/300 Write block 2885 len 40960 data: 2 3 3 3 2 2 2 2 2 2
1226 027/300 Write block 2681 len 4096 data: 2
1227 028/300 Write block 1655 len 20480 data: 2 2 2 2 2
1228 029/300 Read block 3099 len 12288
1229 030/300 Read block 4328 len 36864
1230 031/300 Write block 804 len 12288 data: 2 2 2
1231 032/300 Read block 2296 len 24576
1232 033/300 Read block 2584 len 4096
1233 034/300 Flush
1234 035/300 Write block 2325 len 8192 data: 2 2
1235 036/300 Read block 764 len 8192
1236 037/300 Read block 1823 len 24576
1237 038/300 Write block 966 len 12288 data: 2 2 2
1238 039/300 Read block 3160 len 8192
1239 040/300 Read block 1383 len 12288
1240 041/300 Read block 747 len 4096
1241 042/300 Read block 4698 len 16384
1242 043/300 Write block 1109 len 28672 data: 2 2 2 2 2 2 3
1243 044/300 Read block 3645 len 28672
1244 045/300 Read block 3156 len 12288
1245 046/300 Read block 4728 len 4096
1246 047/300 Write block 1397 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1247 048/300 Write block 3055 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1248 049/300 Write block 3841 len 16384 data: 2 2 2 2
1249 050/300 Write block 4210 len 16384 data: 2 2 2 2
1250 051/300 Write block 2711 len 32768 data: 2 2 2 2 2 2 2 2
1251 052/300 Write block 2042 len 16384 data: 2 2 2 2
1252 053/300 Write block 861 len 8192 data: 2 2
1253 054/300 Write block 755 len 36864 data: 3 3 3 3 3 2 2 2 2
1254 055/300 Write block 4975 len 24576 data: 3 3 2 2 2 3
1255 056/300 Flush
1256 057/300 Read block 1753 len 4096
1257 058/300 Read block 305 len 8192
1258 059/300 Write block 4440 len 36864 data: 2 2 2 2 2 2 2 2 2
1259 060/300 Write block 3344 len 32768 data: 2 2 2 2 2 2 2 2
1260 061/300 Read block 3901 len 12288
1261 062/300 Read block 171 len 28672
1262 063/300 Write block 316 len 12288 data: 2 3 2
1263 064/300 Read block 2072 len 8192
1264 065/300 Write block 4393 len 20480 data: 2 2 2 2 2
1265 066/300 Read block 3394 len 24576
1266 067/300 Read block 1937 len 20480
1267 068/300 Write block 443 len 16384 data: 2 2 2 2
1268 069/300 Write block 3914 len 32768 data: 2 2 2 2 2 2 2 2
1269 070/300 Read block 4356 len 28672
1270 071/300 Read block 2127 len 12288
1271 072/300 Write block 216 len 28672 data: 3 3 3 3 2 2 2
1272 073/300 Write block 1838 len 24576 data: 2 2 2 2 2 2
1273 074/300 Read block 845 len 28672
1274 075/300 Write block 4479 len 4096 data: 2
1275 076/300 Write block 2623 len 40960 data: 2 2 2 2 2 3 3 3 3 2
1276 077/300 Read block 4613 len 32768
1277 078/300 Read block 1026 len 28672
1278 079/300 Read block 4721 len 32768
1279 080/300 Read block 147 len 20480
1280 081/300 Write block 685 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1281 082/300 Read block 522 len 32768
1282 083/300 Read block 2204 len 40960
1283 084/300 Read block 4726 len 24576
1284 085/300 Write block 2373 len 16384 data: 2 2 2 2
1285 086/300 Write block 2228 len 28672 data: 3 2 2 2 2 2 2
1286 087/300 Read block 197 len 20480
1287 088/300 Read block 2597 len 4096
1288 089/300 Read block 4453 len 4096
1289 090/300 Read block 4687 len 32768
1290 091/300 Write block 279 len 24576 data: 2 2 2 2 2 2
1291 092/300 Read block 3601 len 8192
1292 093/300 Write block 2486 len 16384 data: 2 2 2 2
1293 094/300 Flush
1294 095/300 Read block 3772 len 40960
1295 096/300 Flush
1296 097/300 Write block 4207 len 4096 data: 2
1297 098/300 Write block 1318 len 28672 data: 2 2 2 2 2 2 2
1298 099/300 Write block 1986 len 16384 data: 2 2 2 2
1299 100/300 Write block 2215 len 40960 data: 3 3 3 3 3 3 3 3 2 2
1300 101/300 Write block 3188 len 8192 data: 2 2
1301 102/300 Write block 1609 len 4096 data: 2
1302 103/300 Write block 3630 len 20480 data: 2 2 2 2 2
1303 104/300 Write block 4129 len 12288 data: 2 2 2
1304 105/300 Read block 3317 len 28672
1305 106/300 Write block 4707 len 8192 data: 3 3
1306 107/300 Write block 1985 len 16384 data: 2 3 3 3
1307 108/300 Write block 2973 len 4096 data: 2
1308 109/300 Read block 3045 len 36864
1309 110/300 Write block 870 len 8192 data: 2 2
1310 111/300 Read block 2273 len 28672
1311 112/300 Write block 1435 len 24576 data: 2 2 2 2 2 2
1312 113/300 Write block 3855 len 12288 data: 2 2 2
1313 114/300 Write block 500 len 16384 data: 2 2 2 2
1314 115/300 Write block 4204 len 36864 data: 2 2 2 3 2 2 3 3 3
1315 116/300 Write block 209 len 4096 data: 2
1316 117/300 Read block 3430 len 24576
1317 118/300 Read block 2849 len 20480
1318 119/300 Write block 3089 len 28672 data: 2 2 2 2 2 4 4
1319 120/300 Flush
1320 121/300 Read block 4706 len 40960
1321 122/300 Write block 2277 len 24576 data: 3 3 3 3 3 3
1322 123/300 Read block 160 len 40960
1323 124/300 Read block 2638 len 4096
1324 125/300 Flush
1325 126/300 Write block 1004 len 12288 data: 3 3 2
1326 127/300 Write block 3509 len 16384 data: 2 2 2 2
1327 128/300 Write block 1496 len 36864 data: 2 2 2 2 2 2 2 2 2
1328 129/300 Read block 3136 len 8192
1329 130/300 Read block 819 len 40960
1330 131/300 Write block 3354 len 12288 data: 2 2 2
1331 132/300 Flush
1332 133/300 Read block 4007 len 16384
1333 134/300 Read block 2702 len 32768
1334 135/300 Flush
1335 136/300 Read block 2981 len 40960
1336 137/300 Write block 2132 len 12288 data: 2 2 2
1337 138/300 Read block 3298 len 28672
1338 139/300 Read block 3802 len 8192
1339 140/300 Write block 970 len 32768 data: 2 2 2 2 2 2 2 2
1340 141/300 Read block 2436 len 40960
1341 142/300 Write block 2947 len 20480 data: 2 3 3 3 3
1342 143/300 Read block 3496 len 28672
1343 144/300 Write block 3193 len 16384 data: 2 2 2 2
1344 145/300 Write block 4598 len 20480 data: 2 2 2 2 2
1345 146/300 Read block 4831 len 40960
1346 147/300 Read block 2772 len 36864
1347 148/300 Write block 4707 len 32768 data: 4 4 2 2 2 2 2 2
1348 149/300 Read block 156 len 36864
1349 150/300 Flush
1350 151/300 Read block 4625 len 40960
1351 152/300 Read block 4722 len 40960
1352 153/300 Write block 412 len 4096 data: 2
1353 154/300 Flush
1354 155/300 Write block 3012 len 12288 data: 3 3 2
1355 156/300 Read block 1339 len 24576
1356 157/300 Write block 3080 len 24576 data: 2 2 2 2 2 2
1357 158/300 Flush
1358 159/300 Read block 4556 len 4096
1359 160/300 Read block 447 len 16384
1360 161/300 Read block 1085 len 12288
1361 162/300 Write block 3955 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1362 163/300 Read block 3647 len 36864
1363 164/300 Read block 916 len 8192
1364 165/300 Write block 2117 len 28672 data: 2 2 2 2 2 2 2
1365 166/300 Read block 201 len 8192
1366 167/300 Read block 3953 len 24576
1367 168/300 Read block 2636 len 12288
1368 169/300 Write block 2462 len 12288 data: 2 2 2
1369 170/300 Write block 3246 len 16384 data: 2 2 2 2
1370 171/300 Read block 1132 len 4096
1371 172/300 Flush
1372 173/300 Write block 1819 len 16384 data: 2 2 2 2
1373 174/300 Read block 1860 len 24576
1374 175/300 Write block 3987 len 32768 data: 2 2 2 2 2 2 2 2
1375 176/300 Read block 2159 len 20480
1376 177/300 Read block 1365 len 36864
1377 178/300 Read block 575 len 20480
1378 179/300 Flush
1379 180/300 Read block 3786 len 24576
1380 181/300 Write block 1748 len 8192 data: 2 2
1381 182/300 Read block 3733 len 40960
1382 183/300 Read block 2116 len 4096
1383 184/300 Flush
1384 185/300 Write block 3037 len 20480 data: 2 2 3 3 3
1385 186/300 Read block 1772 len 16384
1386 187/300 Read block 330 len 4096
1387 188/300 Read block 2292 len 36864
1388 189/300 Read block 4483 len 28672
1389 190/300 Flush
1390 191/300 Write block 1813 len 16384 data: 2 2 2 2
1391 192/300 Read block 1136 len 32768
1392 193/300 Read block 4632 len 20480
1393 194/300 Write block 2386 len 32768 data: 2 2 2 2 2 2 2 2
1394 195/300 Read block 3458 len 40960
1395 196/300 Write block 3392 len 8192 data: 2 2
1396 197/300 Write block 2641 len 28672 data: 2 2 2 2 2 2 2
1397 198/300 Write block 292 len 16384 data: 2 2 2 2
1398 199/300 Read block 3738 len 12288
1399 200/300 Read block 4782 len 24576
1400 201/300 Write block 1839 len 36864 data: 3 3 3 3 3 2 2 3 3
1401 202/300 Read block 3156 len 40960
1402 203/300 Read block 1941 len 4096
1403 204/300 Read block 3375 len 4096
1404 205/300 Read block 2016 len 8192
1405 206/300 Write block 4187 len 28672 data: 2 2 2 2 2 2 2
1406 207/300 Read block 3018 len 16384
1407 208/300 Read block 2617 len 24576
1408 209/300 Write block 1434 len 32768 data: 2 3 3 3 3 3 3 2
1409 210/300 Write block 2570 len 28672 data: 2 2 2 2 2 2 2
1410 211/300 Flush
1411 212/300 Write block 2390 len 12288 data: 3 3 3
1412 213/300 Read block 1645 len 36864
1413 214/300 Read block 65 len 16384
1414 215/300 Write block 1771 len 36864 data: 2 2 2 2 2 2 2 2 2
1415 216/300 Write block 190 len 8192 data: 2 2
1416 217/300 Read block 2334 len 12288
1417 218/300 Read block 3043 len 8192
1418 219/300 Write block 736 len 16384 data: 2 2 2 2
1419 220/300 Read block 3688 len 8192
1420 221/300 Write block 652 len 12288 data: 2 2 2
1421 222/300 Flush
1422 223/300 Read block 2780 len 12288
1423 224/300 Read block 3622 len 4096
1424 225/300 Write block 3140 len 32768 data: 2 2 2 2 2 2 2 2
1425 226/300 Read block 3548 len 32768
1426 227/300 Read block 4973 len 4096
1427 228/300 Read block 3522 len 40960
1428 229/300 Write block 3120 len 4096 data: 2
1429 230/300 Read block 4866 len 20480
1430 231/300 Read block 1835 len 8192
1431 232/300 Read block 1838 len 8192
1432 233/300 Write block 1643 len 12288 data: 2 2 2
1433 234/300 Write block 897 len 8192 data: 3 2
1434 235/300 Read block 3514 len 40960
1435 236/300 Read block 1987 len 24576
1436 237/300 Read block 4718 len 36864
1437 238/300 Read block 1143 len 36864
1438 239/300 Flush
1439 240/300 Write block 1258 len 36864 data: 2 2 2 2 2 2 2 2 2
1440 241/300 Write block 3928 len 24576 data: 2 2 2 2 2 2
1441 242/300 Write block 981 len 4096 data: 2
1442 243/300 Write block 4018 len 24576 data: 2 2 2 2 2 2
1443 244/300 Write block 3982 len 24576 data: 2 2 3 3 2 3
1444 245/300 Write block 3361 len 4096 data: 2
1445 246/300 Write block 4984 len 16384 data: 3 4 4 4
1446 247/300 Write block 2282 len 32768 data: 4 3 2 2 2 2 2 2
1447 248/300 Read block 3994 len 40960
1448 249/300 Read block 4573 len 24576
1449 250/300 Write block 2917 len 8192 data: 2 2
1450 251/300 Read block 1967 len 8192
1451 252/300 Write block 3033 len 16384 data: 2 2 2 2
1452 253/300 Write block 1840 len 4096 data: 4
1453 254/300 Write block 4879 len 24576 data: 3 3 2 2 2 2
1454 255/300 Write block 3340 len 16384 data: 2 2 2 2
1455 256/300 Read block 2827 len 8192
1456 257/300 Write block 4859 len 20480 data: 2 2 2 2 2
1457 258/300 Read block 4135 len 12288
1458 259/300 Read block 2204 len 8192
1459 260/300 Write block 4830 len 12288 data: 2 2 2
1460 261/300 Read block 681 len 8192
1461 262/300 Read block 3095 len 40960
1462 263/300 Write block 1386 len 28672 data: 2 2 2 2 2 2 2
1463 264/300 Read block 1584 len 20480
1464 265/300 Read block 1878 len 4096
1465 266/300 Write block 871 len 20480 data: 3 2 2 2 2
1466 267/300 Read block 2439 len 4096
1467 268/300 Read block 2934 len 20480
1468 269/300 Read block 2702 len 8192
1469 270/300 Write block 1879 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1470 271/300 Read block 711 len 20480
1471 272/300 Write block 4916 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1472 273/300 Read block 659 len 36864
1473 274/300 Write block 2782 len 16384 data: 2 2 2 2
1474 275/300 Write block 3821 len 12288 data: 2 2 2
1475 276/300 Write block 2604 len 24576 data: 2 2 2 2 2 2
1476 277/300 Write block 3872 len 8192 data: 2 2
1477 278/300 Read block 1990 len 12288
1478 279/300 Write block 2477 len 36864 data: 3 3 3 4 4 4 2 2 2
1479 280/300 Write block 4821 len 20480 data: 3 3 3 3 2
1480 281/300 Read block 1224 len 24576
1481 282/300 Read block 3016 len 8192
1482 283/300 Write block 4223 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1483 284/300 Write block 3263 len 16384 data: 2 2 2 2
1484 285/300 Read block 2464 len 28672
1485 286/300 Read block 709 len 4096
1486 287/300 Write block 1060 len 8192 data: 3 3
1487 288/300 Read block 1532 len 20480
1488 289/300 Flush
1489 290/300 Flush
1490 291/300 Write block 2423 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1491 292/300 Read block 880 len 16384
1492 293/300 Write block 2662 len 8192 data: 2 2
1493 294/300 Write block 4305 len 28672 data: 2 2 2 2 2 2 2
1494 295/300 Read block 804 len 32768
1495 296/300 Flush
1496 297/300 Read block 3891 len 4096
1497 298/300 Flush
1498 299/300 Read block 1100 len 40960
1499 300/300 Flush
1500 301/300 Read block 1270 len 40960
1501 Main task has work to do, go find it
1502 got dsc Start(0)
1503 start 0
1504 Feb 21 06:45:39.698 INFO request completed, latency_us: 231, response_code: 204, uri: /start/cid/0, method: POST, req_id: 4718c01b-f8a1-4472-9486-62f296ced3f0, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
1505 Replay: started 0, returned:Ok(())
1506 ----------------------------------------------------------------
1507 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1508 States: Offline Active Active
1509 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1510 1302 Acked Read 8 Sent Done Done false
1511 1303 Acked Write 7 Sent Done Done false
1512 1304 Acked Write 8 Sent Done Done false
1513 1305 Acked Write 9 Sent Done Done false
1514 1306 Acked Write 5 Sent Done Done false
1515 1307 Acked Read 9 Sent Done Done false
1516 1308 Acked Flush 0 Sent Done Done false
1517 1309 Acked Read 9 Sent Done Done false
1518 1310 Acked Read 2 Sent Done Done false
1519 1311 Acked Write 3 Sent Done Done false
1520 1312 Acked Write 8 Sent Done Done false
1521 1313 Acked Write 8 Sent Done Done false
1522 1314 Acked Read 7 Sent Done Done false
1523 1315 Acked Read 3 Sent Done Done false
1524 1316 Acked Read 9 Sent Done Done false
1525 1317 Acked Write 6 Sent Done Done false
1526 1318 Acked Read 4 Sent Done Done false
1527 1319 Acked Read 9 Sent Done Done false
1528 1320 Acked Read 5 Sent Done Done false
1529 1321 Acked Write 6 Sent Done Done false
1530 1322 Acked Write 4 Sent Done Done false
1531 1323 Acked Read 5 Sent Done Done false
1532 1324 Acked Read 7 Sent Done Done false
1533 1325 Acked Write 2 Sent Done Done false
1534 1326 Acked Read 6 Sent Done Done false
1535 1327 Acked Write 10 Sent Done Done false
1536 1328 Acked Write 1 Sent Done Done false
1537 1329 Acked Write 5 Sent Done Done false
1538 1330 Acked Read 3 Sent Done Done false
1539 1331 Acked Read 9 Sent Done Done false
1540 1332 Acked Write 3 Sent Done Done false
1541 1333 Acked Read 6 Sent Done Done false
1542 1334 Acked Read 1 Sent Done Done false
1543 1335 Acked Flush 0 Sent Done Done false
1544 1336 Acked Write 2 Sent Done Done false
1545 1337 Acked Read 2 Sent Done Done false
1546 1338 Acked Read 6 Sent Done Done false
1547 1339 Acked Write 3 Sent Done Done false
1548 1340 Acked Read 2 Sent Done Done false
1549 1341 Acked Read 3 Sent Done Done false
1550 1342 Acked Read 1 Sent Done Done false
1551 1343 Acked Read 4 Sent Done Done false
1552 1344 Acked Write 7 Sent Done Done false
1553 1345 Acked Read 7 Sent Done Done false
1554 1346 Acked Read 3 Sent Done Done false
1555 1347 Acked Read 1 Sent Done Done false
1556 1348 Acked Write 10 Sent Done Done false
1557 1349 Acked Write 10 Sent Done Done false
1558 1350 Acked Write 4 Sent Done Done false
1559 1351 Acked Write 4 Sent Done Done false
1560 1352 Acked Write 8 Sent Done Done false
1561 1353 Acked Write 4 Sent Done Done false
1562 1354 Acked Write 2 Sent Done Done false
1563 1355 Acked Write 9 Sent Done Done false
1564 1356 Acked Write 6 Sent Done Done false
1565 1357 Acked Flush 0 Sent Done Done false
1566 1358 Acked Read 1 Sent Done Done false
1567 1359 Acked Read 2 Sent Done Done false
1568 1360 Acked Write 9 Sent Done Done false
1569 1361 Acked Write 8 Sent Done Done false
1570 1362 Acked Read 3 Sent Done Done false
1571 1363 Acked Read 7 Sent Done Done false
1572 1364 Acked Write 3 Sent Done Done false
1573 1365 Acked Read 2 Sent Done Done false
1574 1366 Acked Write 5 Sent Done Done false
1575 1367 Acked Read 6 Sent Done Done false
1576 1368 Acked Read 5 Sent Done Done false
1577 1369 Acked Write 4 Sent Done Done false
1578 1370 Acked Write 8 Sent Done Done false
1579 1371 Acked Read 7 Sent Done Done false
1580 1372 Acked Read 3 Sent Done Done false
1581 1373 Acked Write 7 Sent Done Done false
1582 1374 Acked Write 6 Sent Done Done false
1583 1375 Acked Read 7 Sent Done Done false
1584 1376 Acked Write 1 Sent Done Done false
1585 1377 Acked Write 10 Sent Done Done false
1586 1378 Acked Read 8 Sent Done Done false
1587 1379 Acked Read 7 Sent Done Done false
1588 1380 Acked Read 8 Sent Done Done false
1589 1381 Acked Read 5 Sent Done Done false
1590 1382 Acked Write 10 Sent Done Done false
1591 1383 Acked Read 8 Sent Done Done false
1592 1384 Acked Read 10 Sent Done Done false
1593 1385 Acked Read 6 Sent Done Done false
1594 1386 Acked Write 4 Sent Done Done false
1595 1387 Acked Write 7 Sent Done Done false
1596 1388 Acked Read 5 Sent Done Done false
1597 1389 Acked Read 1 Sent Done Done false
1598 1390 Acked Read 1 Sent Done Done false
1599 1391 Acked Read 8 Sent Done Done false
1600 1392 Acked Write 6 Sent Done Done false
1601 1393 Acked Read 2 Sent Done Done false
1602 1394 Acked Write 4 Sent Done Done false
1603 1395 Acked Flush 0 Sent Done Done false
1604 1396 Acked Read 10 Sent Done Done false
1605 1397 Acked Flush 0 Sent Done Done false
1606 1398 Acked Write 1 Sent Done Done false
1607 1399 Acked Write 7 Sent Done Done false
1608 1400 Acked Write 4 Sent Done Done false
1609 1401 Acked Write 10 Sent Done Done false
1610 1402 Acked Write 2 Sent Done Done false
1611 1403 Acked Write 1 Sent Done Done false
1612 1404 Acked Write 5 Sent Done Done false
1613 1405 Acked Write 3 Sent Done Done false
1614 1406 Acked Read 7 Sent Done Done false
1615 1407 Acked Write 2 Sent Done Done false
1616 1408 Acked Write 4 Sent Done Done false
1617 1409 Acked Write 1 Sent Done Done false
1618 1410 Acked Read 9 Sent Done Done false
1619 1411 Acked Write 2 Sent Done Done false
1620 1412 Acked Read 7 Sent Done Done false
1621 1413 Acked Write 6 Sent Done Done false
1622 1414 Acked Write 3 Sent Done Done false
1623 1415 Acked Write 4 Sent Done Done false
1624 1416 Acked Write 9 Sent Done Done false
1625 1417 Acked Write 1 Sent Done Done false
1626 1418 Acked Read 6 Sent Done Done false
1627 1419 Acked Read 5 Sent Done Done false
1628 1420 Acked Write 7 Sent Done Done false
1629 1421 Acked Flush 0 Sent Done Done false
1630 1422 Acked Read 10 Sent Done Done false
1631 1423 Acked Write 6 Sent Done Done false
1632 1424 Acked Read 10 Sent Done Done false
1633 1425 Acked Read 1 Sent Done Done false
1634 1426 Acked Flush 0 Sent Done Done false
1635 1427 Acked Write 3 Sent Done Done false
1636 1428 Acked Write 4 Sent Done Done false
1637 1429 Acked Write 9 Sent Done Done false
1638 1430 Acked Read 2 Sent Done Done false
1639 1431 Acked Read 10 Sent Done Done false
1640 1432 Acked Write 3 Sent Done Done false
1641 1433 Acked Flush 0 Sent Done Done false
1642 1434 Acked Read 4 Sent Done Done false
1643 1435 Acked Read 8 Sent Done Done false
1644 1436 Acked Flush 0 Sent Done Done false
1645 1437 Acked Read 10 Sent Done Done false
1646 1438 Acked Write 3 Sent Done Done false
1647 1439 Acked Read 7 Sent Done Done false
1648 1440 Acked Read 2 Sent Done Done false
1649 1441 Acked Write 8 Sent Done Done false
1650 1442 Acked Read 10 Sent Done Done false
1651 1443 Acked Write 5 Sent Done Done false
1652 1444 Acked Read 7 Sent Done Done false
1653 1445 Acked Write 4 Sent Done Done false
1654 1446 Acked Write 5 Sent Done Done false
1655 1447 Acked Read 10 Sent Done Done false
1656 1448 Acked Read 9 Sent Done Done false
1657 1449 Acked Write 8 Sent Done Done false
1658 1450 Acked Read 9 Sent Done Done false
1659 1451 Acked Flush 0 Sent Done Done false
1660 1452 Acked Read 10 Sent Done Done false
1661 1453 Acked Read 10 Sent Done Done false
1662 1454 Acked Write 1 Sent Done Done false
1663 1455 Acked Flush 0 Sent Done Done false
1664 1456 Acked Write 3 Sent Done Done false
1665 1457 Acked Read 6 Sent Done Done false
1666 1458 Acked Write 6 Sent Done Done false
1667 1459 Acked Flush 0 Sent Done Done false
1668 1460 Acked Read 1 Sent Done Done false
1669 1461 Acked Read 4 Sent Done Done false
1670 1462 Acked Read 3 Sent Done Done false
1671 1463 Acked Write 10 Sent Done Done false
1672 1464 Acked Read 9 Sent Done Done false
1673 1465 Acked Read 2 Sent Done Done false
1674 1466 Acked Write 7 Sent Done Done false
1675 1467 Acked Read 2 Sent Done Done false
1676 1468 Acked Read 6 Sent Done Done false
1677 1469 Acked Read 3 Sent Done Done false
1678 1470 Acked Write 3 Sent Done Done false
1679 1471 Acked Write 4 Sent Done Done false
1680 1472 Acked Read 1 Sent Done Done false
1681 1473 Acked Flush 0 Sent Done Done false
1682 1474 Acked Write 4 Sent Done Done false
1683 1475 Acked Read 6 Sent Done Done false
1684 1476 Acked Write 8 Sent Done Done false
1685 1477 Acked Read 5 Sent Done Done false
1686 1478 Acked Read 9 Sent Done Done false
1687 1479 Acked Read 5 Sent Done Done false
1688 1480 Acked Flush 0 Sent Done Done false
1689 1481 Acked Read 6 Sent Done Done false
1690 1482 Acked Write 2 Sent Done Done false
1691 1483 Acked Read 10 Sent Done Done false
1692 1484 Acked Read 1 Sent Done Done false
1693 1485 Acked Flush 0 Sent Done Done false
1694 1486 Acked Write 5 Sent Done Done false
1695 1487 Acked Read 4 Sent Done Done false
1696 1488 Acked Read 1 Sent Done Done false
1697 1489 Acked Read 9 Sent Done Done false
1698 1490 Acked Read 7 Sent Done Done false
1699 1491 Acked Flush 0 Sent Done Done false
1700 1492 Acked Write 4 Sent Done Done false
1701 1493 Acked Read 8 Sent Done Done false
1702 1494 Acked Read 5 Sent Done Done false
1703 1495 Acked Write 8 Sent Done Done false
1704 1496 Acked Read 10 Sent Done Done false
1705 1497 Acked Write 2 Sent Done Done false
1706 1498 Acked Write 7 Sent Done Done false
1707 1499 Acked Write 4 Sent Done Done false
1708 1500 Acked Read 3 Sent Done Done false
1709 1501 Acked Read 6 Sent Done Done false
1710 1502 Acked Write 9 Sent Done Done false
1711 1503 Acked Read 10 Sent Done Done false
1712 1504 Acked Read 1 Sent Done Done false
1713 1505 Acked Read 1 Sent Done Done false
1714 1506 Acked Read 2 Sent Done Done false
1715 1507 Acked Write 7 Sent Done Done false
1716 1508 Acked Read 4 Sent Done Done false
1717 1509 Acked Read 6 Sent Done Done false
1718 1510 Acked Write 8 Sent Done Done false
1719 1511 Acked Write 7 Sent Done Done false
1720 1512 Acked Flush 0 Sent Done Done false
1721 1513 Acked Write 3 Sent Done Done false
1722 1514 Acked Read 9 Sent Done Done false
1723 1515 Acked Read 4 Sent Done Done false
1724 1516 Acked Write 9 Sent Done Done false
1725 1517 Acked Write 2 Sent Done Done false
1726 1518 Acked Read 3 Sent Done Done false
1727 1519 Acked Read 2 Sent Done Done false
1728 1520 Acked Write 4 Sent Done Done false
1729 1521 Acked Read 2 Sent Done Done false
1730 1522 Acked Write 3 Sent Done Done false
1731 1523 Acked Flush 0 Sent Done Done false
1732 1524 Acked Read 3 Sent Done Done false
1733 1525 Acked Read 1 Sent Done Done false
1734 1526 Acked Write 8 Sent Done Done false
1735 1527 Acked Read 8 Sent Done Done false
1736 1528 Acked Read 1 Sent Done Done false
1737 1529 Acked Read 10 Sent Done Done false
1738 1530 Acked Write 1 Sent Done Done false
1739 1531 Acked Read 5 Sent Done Done false
1740 1532 Acked Read 2 Sent Done Done false
1741 1533 Acked Read 2 Sent Done Done false
1742 1534 Acked Write 3 Sent Done Done false
1743 1535 Acked Write 2 Sent Done Done false
1744 1536 Acked Read 10 Sent Done Done false
1745 1537 Acked Read 6 Sent Done Done false
1746 1538 Acked Read 9 Sent Done Done false
1747 1539 Acked Read 9 Sent Done Done false
1748 1540 Acked Flush 0 Sent Done Done false
1749 1541 Acked Write 9 Sent Done Done false
1750 1542 Acked Write 6 Sent Done Done false
1751 1543 Acked Write 1 Sent Done Done false
1752 1544 Acked Write 6 Sent Done Done false
1753 1545 Acked Write 6 Sent Done Done false
1754 1546 Acked Write 1 Sent Done Done false
1755 1547 Acked Write 4 Sent Done Done false
1756 1548 Acked Write 8 Sent Done Done false
1757 1549 Acked Read 10 Sent Done Done false
1758 1550 Acked Read 6 Sent Done Done false
1759 1551 Acked Write 2 Sent Done Done false
1760 1552 Acked Read 2 Sent Done Done false
1761 1553 Acked Write 4 Sent Done Done false
1762 1554 Acked Write 1 Sent Done Done false
1763 1555 Acked Write 6 Sent Done Done false
1764 1556 Acked Write 4 Sent Done Done false
1765 1557 Acked Read 2 Sent Done Done false
1766 1558 Acked Write 5 Sent Done Done false
1767 1559 Acked Read 3 Sent Done Done false
1768 1560 Acked Read 2 Sent Done Done false
1769 1561 Acked Write 3 Sent Done Done false
1770 1562 Acked Read 2 Sent Done Done false
1771 1563 Acked Read 10 Sent Done Done false
1772 1564 Acked Write 7 Sent Done Done false
1773 1565 Acked Read 5 Sent Done Done false
1774 1566 Acked Read 1 Sent Done Done false
1775 1567 Acked Write 5 Sent Done Done false
1776 1568 Acked Read 1 Sent Done Done false
1777 1569 Acked Read 5 Sent Done Done false
1778 1570 Acked Read 2 Sent Done Done false
1779 1571 Acked Write 10 Sent Done Done false
1780 1572 Acked Read 5 Sent Done Done false
1781 1573 Acked Write 10 Sent Done Done false
1782 1574 Acked Read 9 Sent Done Done false
1783 1575 Acked Write 4 Sent Done Done false
1784 1576 Acked Write 3 Sent Done Done false
1785 1577 Acked Write 6 Sent Done Done false
1786 1578 Acked Write 2 Sent Done Done false
1787 1579 Acked Read 3 Sent Done Done false
1788 1580 Acked Write 9 Sent Done Done false
1789 1581 Acked Write 5 Sent Done Done false
1790 1582 Acked Read 6 Sent Done Done false
1791 1583 Acked Read 2 Sent Done Done false
1792 1584 Acked Write 10 Sent Done Done false
1793 1585 Acked Write 4 Sent Done Done false
1794 1586 Acked Read 7 Sent Done Done false
1795 1587 Acked Read 1 Sent Done Done false
1796 1588 Acked Write 2 Sent Done Done false
1797 1589 Acked Read 5 Sent Done Done false
1798 1590 Acked Flush 0 Sent Done Done false
1799 1591 Acked Flush 0 Sent Done Done false
1800 1592 Acked Write 10 Sent Done Done false
1801 1593 Acked Read 4 Sent Done Done false
1802 1594 Acked Write 2 Sent Done Done false
1803 1595 Acked Write 7 Sent Done Done false
1804 1596 Acked Read 8 Sent Done Done false
1805 1597 Acked Flush 0 Sent Done Done false
1806 1598 Acked Read 1 Sent Done Done false
1807 1599 Acked Flush 0 Sent Done Done false
1808 1600 Acked Read 10 Sent Done Done false
1809 1601 Acked Flush 0 Sent Done Done false
1810 1602 Acked Read 10 Sent Done Done false
1811 STATES DS:0 DS:1 DS:2 TOTAL
1812 Sent 301 0 0 301
1813 Done 0 301 301 602
1814 Skipped 0 0 0 0
1815 Error 0 0 0 0
1816 Last Flush: Some(JobId(1301)) Some(JobId(1601)) Some(JobId(1601))
1817 Downstairs last five completed: 1301 1300 1299 1298 1297
1818 Upstairs last five completed: 1602 1601 1600 1599 1598
1819 CLIENT: Up:0 ds:301 act:2
18202025-02-21T06:45:40.687ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1821 [8810] got start action
1822 [8810] I am going to restart
1823 Starting downstairs at port 8810
1824 Make output file at "/tmp/dsc/downstairs-8810.txt"
1825 [8810][0] reports Starting
1826 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1208)
1827 [8810][0] reports Running
18282025-02-21T06:45:41.689ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
18292025-02-21T06:45:42.690ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
18302025-02-21T06:45:43.691ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
1831 ----------------------------------------------------------------
1832 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1833 States: Offline Active Active
1834 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1835 1302 Acked Read 8 Sent Done Done false
1836 1303 Acked Write 7 Sent Done Done false
1837 1304 Acked Write 8 Sent Done Done false
1838 1305 Acked Write 9 Sent Done Done false
1839 1306 Acked Write 5 Sent Done Done false
1840 1307 Acked Read 9 Sent Done Done false
1841 1308 Acked Flush 0 Sent Done Done false
1842 1309 Acked Read 9 Sent Done Done false
1843 1310 Acked Read 2 Sent Done Done false
1844 1311 Acked Write 3 Sent Done Done false
1845 1312 Acked Write 8 Sent Done Done false
1846 1313 Acked Write 8 Sent Done Done false
1847 1314 Acked Read 7 Sent Done Done false
1848 1315 Acked Read 3 Sent Done Done false
1849 1316 Acked Read 9 Sent Done Done false
1850 1317 Acked Write 6 Sent Done Done false
1851 1318 Acked Read 4 Sent Done Done false
1852 1319 Acked Read 9 Sent Done Done false
1853 1320 Acked Read 5 Sent Done Done false
1854 1321 Acked Write 6 Sent Done Done false
1855 1322 Acked Write 4 Sent Done Done false
1856 1323 Acked Read 5 Sent Done Done false
1857 1324 Acked Read 7 Sent Done Done false
1858 1325 Acked Write 2 Sent Done Done false
1859 1326 Acked Read 6 Sent Done Done false
1860 1327 Acked Write 10 Sent Done Done false
1861 1328 Acked Write 1 Sent Done Done false
1862 1329 Acked Write 5 Sent Done Done false
1863 1330 Acked Read 3 Sent Done Done false
1864 1331 Acked Read 9 Sent Done Done false
1865 1332 Acked Write 3 Sent Done Done false
1866 1333 Acked Read 6 Sent Done Done false
1867 1334 Acked Read 1 Sent Done Done false
1868 1335 Acked Flush 0 Sent Done Done false
1869 1336 Acked Write 2 Sent Done Done false
1870 1337 Acked Read 2 Sent Done Done false
1871 1338 Acked Read 6 Sent Done Done false
1872 1339 Acked Write 3 Sent Done Done false
1873 1340 Acked Read 2 Sent Done Done false
1874 1341 Acked Read 3 Sent Done Done false
1875 1342 Acked Read 1 Sent Done Done false
1876 1343 Acked Read 4 Sent Done Done false
1877 1344 Acked Write 7 Sent Done Done false
1878 1345 Acked Read 7 Sent Done Done false
1879 1346 Acked Read 3 Sent Done Done false
1880 1347 Acked Read 1 Sent Done Done false
1881 1348 Acked Write 10 Sent Done Done false
1882 1349 Acked Write 10 Sent Done Done false
1883 1350 Acked Write 4 Sent Done Done false
1884 1351 Acked Write 4 Sent Done Done false
1885 1352 Acked Write 8 Sent Done Done false
1886 1353 Acked Write 4 Sent Done Done false
1887 1354 Acked Write 2 Sent Done Done false
1888 1355 Acked Write 9 Sent Done Done false
1889 1356 Acked Write 6 Sent Done Done false
1890 1357 Acked Flush 0 Sent Done Done false
1891 1358 Acked Read 1 Sent Done Done false
1892 1359 Acked Read 2 Sent Done Done false
1893 1360 Acked Write 9 Sent Done Done false
1894 1361 Acked Write 8 Sent Done Done false
1895 1362 Acked Read 3 Sent Done Done false
1896 1363 Acked Read 7 Sent Done Done false
1897 1364 Acked Write 3 Sent Done Done false
1898 1365 Acked Read 2 Sent Done Done false
1899 1366 Acked Write 5 Sent Done Done false
1900 1367 Acked Read 6 Sent Done Done false
1901 1368 Acked Read 5 Sent Done Done false
1902 1369 Acked Write 4 Sent Done Done false
1903 1370 Acked Write 8 Sent Done Done false
1904 1371 Acked Read 7 Sent Done Done false
1905 1372 Acked Read 3 Sent Done Done false
1906 1373 Acked Write 7 Sent Done Done false
1907 1374 Acked Write 6 Sent Done Done false
1908 1375 Acked Read 7 Sent Done Done false
1909 1376 Acked Write 1 Sent Done Done false
1910 1377 Acked Write 10 Sent Done Done false
1911 1378 Acked Read 8 Sent Done Done false
1912 1379 Acked Read 7 Sent Done Done false
1913 1380 Acked Read 8 Sent Done Done false
1914 1381 Acked Read 5 Sent Done Done false
1915 1382 Acked Write 10 Sent Done Done false
1916 1383 Acked Read 8 Sent Done Done false
1917 1384 Acked Read 10 Sent Done Done false
1918 1385 Acked Read 6 Sent Done Done false
1919 1386 Acked Write 4 Sent Done Done false
1920 1387 Acked Write 7 Sent Done Done false
1921 1388 Acked Read 5 Sent Done Done false
1922 1389 Acked Read 1 Sent Done Done false
1923 1390 Acked Read 1 Sent Done Done false
1924 1391 Acked Read 8 Sent Done Done false
1925 1392 Acked Write 6 Sent Done Done false
1926 1393 Acked Read 2 Sent Done Done false
1927 1394 Acked Write 4 Sent Done Done false
1928 1395 Acked Flush 0 Sent Done Done false
1929 1396 Acked Read 10 Sent Done Done false
1930 1397 Acked Flush 0 Sent Done Done false
1931 1398 Acked Write 1 Sent Done Done false
1932 1399 Acked Write 7 Sent Done Done false
1933 1400 Acked Write 4 Sent Done Done false
1934 1401 Acked Write 10 Sent Done Done false
1935 1402 Acked Write 2 Sent Done Done false
1936 1403 Acked Write 1 Sent Done Done false
1937 1404 Acked Write 5 Sent Done Done false
1938 1405 Acked Write 3 Sent Done Done false
1939 1406 Acked Read 7 Sent Done Done false
1940 1407 Acked Write 2 Sent Done Done false
1941 1408 Acked Write 4 Sent Done Done false
1942 1409 Acked Write 1 Sent Done Done false
1943 1410 Acked Read 9 Sent Done Done false
1944 1411 Acked Write 2 Sent Done Done false
1945 1412 Acked Read 7 Sent Done Done false
1946 1413 Acked Write 6 Sent Done Done false
1947 1414 Acked Write 3 Sent Done Done false
1948 1415 Acked Write 4 Sent Done Done false
1949 1416 Acked Write 9 Sent Done Done false
1950 1417 Acked Write 1 Sent Done Done false
1951 1418 Acked Read 6 Sent Done Done false
1952 1419 Acked Read 5 Sent Done Done false
1953 1420 Acked Write 7 Sent Done Done false
1954 1421 Acked Flush 0 Sent Done Done false
1955 1422 Acked Read 10 Sent Done Done false
1956 1423 Acked Write 6 Sent Done Done false
1957 1424 Acked Read 10 Sent Done Done false
1958 1425 Acked Read 1 Sent Done Done false
1959 1426 Acked Flush 0 Sent Done Done false
1960 1427 Acked Write 3 Sent Done Done false
1961 1428 Acked Write 4 Sent Done Done false
1962 1429 Acked Write 9 Sent Done Done false
1963 1430 Acked Read 2 Sent Done Done false
1964 1431 Acked Read 10 Sent Done Done false
1965 1432 Acked Write 3 Sent Done Done false
1966 1433 Acked Flush 0 Sent Done Done false
1967 1434 Acked Read 4 Sent Done Done false
1968 1435 Acked Read 8 Sent Done Done false
1969 1436 Acked Flush 0 Sent Done Done false
1970 1437 Acked Read 10 Sent Done Done false
1971 1438 Acked Write 3 Sent Done Done false
1972 1439 Acked Read 7 Sent Done Done false
1973 1440 Acked Read 2 Sent Done Done false
1974 1441 Acked Write 8 Sent Done Done false
1975 1442 Acked Read 10 Sent Done Done false
1976 1443 Acked Write 5 Sent Done Done false
1977 1444 Acked Read 7 Sent Done Done false
1978 1445 Acked Write 4 Sent Done Done false
1979 1446 Acked Write 5 Sent Done Done false
1980 1447 Acked Read 10 Sent Done Done false
1981 1448 Acked Read 9 Sent Done Done false
1982 1449 Acked Write 8 Sent Done Done false
1983 1450 Acked Read 9 Sent Done Done false
1984 1451 Acked Flush 0 Sent Done Done false
1985 1452 Acked Read 10 Sent Done Done false
1986 1453 Acked Read 10 Sent Done Done false
1987 1454 Acked Write 1 Sent Done Done false
1988 1455 Acked Flush 0 Sent Done Done false
1989 1456 Acked Write 3 Sent Done Done false
1990 1457 Acked Read 6 Sent Done Done false
1991 1458 Acked Write 6 Sent Done Done false
1992 1459 Acked Flush 0 Sent Done Done false
1993 1460 Acked Read 1 Sent Done Done false
1994 1461 Acked Read 4 Sent Done Done false
1995 1462 Acked Read 3 Sent Done Done false
1996 1463 Acked Write 10 Sent Done Done false
1997 1464 Acked Read 9 Sent Done Done false
1998 1465 Acked Read 2 Sent Done Done false
1999 1466 Acked Write 7 Sent Done Done false
2000 1467 Acked Read 2 Sent Done Done false
2001 1468 Acked Read 6 Sent Done Done false
2002 1469 Acked Read 3 Sent Done Done false
2003 1470 Acked Write 3 Sent Done Done false
2004 1471 Acked Write 4 Sent Done Done false
2005 1472 Acked Read 1 Sent Done Done false
2006 1473 Acked Flush 0 Sent Done Done false
2007 1474 Acked Write 4 Sent Done Done false
2008 1475 Acked Read 6 Sent Done Done false
2009 1476 Acked Write 8 Sent Done Done false
2010 1477 Acked Read 5 Sent Done Done false
2011 1478 Acked Read 9 Sent Done Done false
2012 1479 Acked Read 5 Sent Done Done false
2013 1480 Acked Flush 0 Sent Done Done false
2014 1481 Acked Read 6 Sent Done Done false
2015 1482 Acked Write 2 Sent Done Done false
2016 1483 Acked Read 10 Sent Done Done false
2017 1484 Acked Read 1 Sent Done Done false
2018 1485 Acked Flush 0 Sent Done Done false
2019 1486 Acked Write 5 Sent Done Done false
2020 1487 Acked Read 4 Sent Done Done false
2021 1488 Acked Read 1 Sent Done Done false
2022 1489 Acked Read 9 Sent Done Done false
2023 1490 Acked Read 7 Sent Done Done false
2024 1491 Acked Flush 0 Sent Done Done false
2025 1492 Acked Write 4 Sent Done Done false
2026 1493 Acked Read 8 Sent Done Done false
2027 1494 Acked Read 5 Sent Done Done false
2028 1495 Acked Write 8 Sent Done Done false
2029 1496 Acked Read 10 Sent Done Done false
2030 1497 Acked Write 2 Sent Done Done false
2031 1498 Acked Write 7 Sent Done Done false
2032 1499 Acked Write 4 Sent Done Done false
2033 1500 Acked Read 3 Sent Done Done false
2034 1501 Acked Read 6 Sent Done Done false
2035 1502 Acked Write 9 Sent Done Done false
2036 1503 Acked Read 10 Sent Done Done false
2037 1504 Acked Read 1 Sent Done Done false
2038 1505 Acked Read 1 Sent Done Done false
2039 1506 Acked Read 2 Sent Done Done false
2040 1507 Acked Write 7 Sent Done Done false
2041 1508 Acked Read 4 Sent Done Done false
2042 1509 Acked Read 6 Sent Done Done false
2043 1510 Acked Write 8 Sent Done Done false
2044 1511 Acked Write 7 Sent Done Done false
2045 1512 Acked Flush 0 Sent Done Done false
2046 1513 Acked Write 3 Sent Done Done false
2047 1514 Acked Read 9 Sent Done Done false
2048 1515 Acked Read 4 Sent Done Done false
2049 1516 Acked Write 9 Sent Done Done false
2050 1517 Acked Write 2 Sent Done Done false
2051 1518 Acked Read 3 Sent Done Done false
2052 1519 Acked Read 2 Sent Done Done false
2053 1520 Acked Write 4 Sent Done Done false
2054 1521 Acked Read 2 Sent Done Done false
2055 1522 Acked Write 3 Sent Done Done false
2056 1523 Acked Flush 0 Sent Done Done false
2057 1524 Acked Read 3 Sent Done Done false
2058 1525 Acked Read 1 Sent Done Done false
2059 1526 Acked Write 8 Sent Done Done false
2060 1527 Acked Read 8 Sent Done Done false
2061 1528 Acked Read 1 Sent Done Done false
2062 1529 Acked Read 10 Sent Done Done false
2063 1530 Acked Write 1 Sent Done Done false
2064 1531 Acked Read 5 Sent Done Done false
2065 1532 Acked Read 2 Sent Done Done false
2066 1533 Acked Read 2 Sent Done Done false
2067 1534 Acked Write 3 Sent Done Done false
2068 1535 Acked Write 2 Sent Done Done false
2069 1536 Acked Read 10 Sent Done Done false
2070 1537 Acked Read 6 Sent Done Done false
2071 1538 Acked Read 9 Sent Done Done false
2072 1539 Acked Read 9 Sent Done Done false
2073 1540 Acked Flush 0 Sent Done Done false
2074 1541 Acked Write 9 Sent Done Done false
2075 1542 Acked Write 6 Sent Done Done false
2076 1543 Acked Write 1 Sent Done Done false
2077 1544 Acked Write 6 Sent Done Done false
2078 1545 Acked Write 6 Sent Done Done false
2079 1546 Acked Write 1 Sent Done Done false
2080 1547 Acked Write 4 Sent Done Done false
2081 1548 Acked Write 8 Sent Done Done false
2082 1549 Acked Read 10 Sent Done Done false
2083 1550 Acked Read 6 Sent Done Done false
2084 1551 Acked Write 2 Sent Done Done false
2085 1552 Acked Read 2 Sent Done Done false
2086 1553 Acked Write 4 Sent Done Done false
2087 1554 Acked Write 1 Sent Done Done false
2088 1555 Acked Write 6 Sent Done Done false
2089 1556 Acked Write 4 Sent Done Done false
2090 1557 Acked Read 2 Sent Done Done false
2091 1558 Acked Write 5 Sent Done Done false
2092 1559 Acked Read 3 Sent Done Done false
2093 1560 Acked Read 2 Sent Done Done false
2094 1561 Acked Write 3 Sent Done Done false
2095 1562 Acked Read 2 Sent Done Done false
2096 1563 Acked Read 10 Sent Done Done false
2097 1564 Acked Write 7 Sent Done Done false
2098 1565 Acked Read 5 Sent Done Done false
2099 1566 Acked Read 1 Sent Done Done false
2100 1567 Acked Write 5 Sent Done Done false
2101 1568 Acked Read 1 Sent Done Done false
2102 1569 Acked Read 5 Sent Done Done false
2103 1570 Acked Read 2 Sent Done Done false
2104 1571 Acked Write 10 Sent Done Done false
2105 1572 Acked Read 5 Sent Done Done false
2106 1573 Acked Write 10 Sent Done Done false
2107 1574 Acked Read 9 Sent Done Done false
2108 1575 Acked Write 4 Sent Done Done false
2109 1576 Acked Write 3 Sent Done Done false
2110 1577 Acked Write 6 Sent Done Done false
2111 1578 Acked Write 2 Sent Done Done false
2112 1579 Acked Read 3 Sent Done Done false
2113 1580 Acked Write 9 Sent Done Done false
2114 1581 Acked Write 5 Sent Done Done false
2115 1582 Acked Read 6 Sent Done Done false
2116 1583 Acked Read 2 Sent Done Done false
2117 1584 Acked Write 10 Sent Done Done false
2118 1585 Acked Write 4 Sent Done Done false
2119 1586 Acked Read 7 Sent Done Done false
2120 1587 Acked Read 1 Sent Done Done false
2121 1588 Acked Write 2 Sent Done Done false
2122 1589 Acked Read 5 Sent Done Done false
2123 1590 Acked Flush 0 Sent Done Done false
2124 1591 Acked Flush 0 Sent Done Done false
2125 1592 Acked Write 10 Sent Done Done false
2126 1593 Acked Read 4 Sent Done Done false
2127 1594 Acked Write 2 Sent Done Done false
2128 1595 Acked Write 7 Sent Done Done false
2129 1596 Acked Read 8 Sent Done Done false
2130 1597 Acked Flush 0 Sent Done Done false
2131 1598 Acked Read 1 Sent Done Done false
2132 1599 Acked Flush 0 Sent Done Done false
2133 1600 Acked Read 10 Sent Done Done false
2134 1601 Acked Flush 0 Sent Done Done false
2135 1602 Acked Read 10 Sent Done Done false
2136 STATES DS:0 DS:1 DS:2 TOTAL
2137 Sent 301 0 0 301
2138 Done 0 301 301 602
2139 Skipped 0 0 0 0
2140 Error 0 0 0 0
2141 Last Flush: Some(JobId(1301)) Some(JobId(1601)) Some(JobId(1601))
2142 Downstairs last five completed: 1301 1300 1299 1298 1297
2143 Upstairs last five completed: 1602 1601 1600 1599 1598
2144 CLIENT: Up:0 ds:301 act:2
21452025-02-21T06:45:44.692ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21462025-02-21T06:45:45.466ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21472025-02-21T06:45:45.467ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65006) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21482025-02-21T06:45:45.468ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21492025-02-21T06:45:45.468ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21502025-02-21T06:45:45.468ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21512025-02-21T06:45:45.468ZINFOcrucible: send last flush ID to this DS: Some(JobId(1301)) = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21522025-02-21T06:45:45.468ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21532025-02-21T06:45:45.468ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(1301)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21542025-02-21T06:45:45.469ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(1301)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2155 ----------------------------------------------------------------
2156 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2157 Downstairs last five completed: 1603 1602 1601 1600 1599
2158 Upstairs last five completed: 1603 1602 1601 1600 1599
2159 CLIENT: Up:0 ds:0 act:3
2160 Replay: All jobs finished
2161 Main task has work to do, go find it
2162 got dsc Stop(1)
2163 stop 1
2164 Feb 21 06:45:47.706 INFO [8820] Got stop action so:false kr:false
2165 request completed, latency_us: 356, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 9195fbc9-541b-4cf5-a9dc-cb8576f56f8f, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
21662025-02-21T06:45:47.706ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21672025-02-21T06:45:47.707ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21682025-02-21T06:45:47.707ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2169 Feb 21 06:45:47.707 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/1, method: GET, req_id: d1a7dee2-7081-4007-97b9-01ec27c3d299, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
21702025-02-21T06:45:47.707ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2171 [8820] Exited with: ExitStatus(unix_wait_status(9))
2172 [8820][1] reports Exit
2173 Feb 21 06:45:51.709 INFO request completed, latency_us: 381, response_code: 200, uri: /state/cid/1, method: GET, req_id: c16de869-28ad-42a1-8b00-2132e5b3da3b, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
2174 001/300 Read block 1616 len 24576
2175 002/300 Read block 1533 len 32768
2176 003/300 Flush
2177 004/300 Write block 2806 len 12288 data: 2 2 2
2178 005/300 Write block 4696 len 16384 data: 2 2 2 2
2179 006/300 Write block 3508 len 32768 data: 2 3 3 3 3 2 2 2
2180 007/300 Write block 2632 len 24576 data: 3 2 2 2 2 2
2181 008/300 Write block 945 len 32768 data: 2 2 2 2 2 2 2 2
2182 009/300 Read block 1761 len 4096
2183 010/300 Write block 1352 len 8192 data: 2 2
2184 011/300 Read block 4737 len 40960
2185 012/300 Read block 3511 len 36864
2186 013/300 Write block 4040 len 32768 data: 2 2 2 2 3 2 2 2
2187 014/300 Read block 864 len 24576
2188 015/300 Write block 657 len 8192 data: 2 2
2189 016/300 Write block 4658 len 8192 data: 2 2
2190 017/300 Write block 4104 len 40960 data: 2 3 2 2 2 2 2 2 2 2
2191 018/300 Flush
2192 019/300 Flush
2193 020/300 Write block 563 len 16384 data: 2 2 2 2
2194 021/300 Write block 755 len 4096 data: 4
2195 022/300 Write block 3124 len 8192 data: 2 2
2196 023/300 Read block 3296 len 16384
2197 024/300 Read block 1847 len 20480
2198 025/300 Write block 4702 len 28672 data: 2 2 2 2 2 5 5
2199 026/300 Read block 4861 len 40960
2200 027/300 Write block 879 len 32768 data: 2 2 2 2 2 2 2 2
2201 028/300 Read block 2628 len 40960
2202 029/300 Read block 4407 len 28672
2203 030/300 Read block 471 len 8192
2204 031/300 Read block 2132 len 12288
2205 032/300 Write block 2198 len 4096 data: 2
2206 033/300 Read block 2660 len 4096
2207 034/300 Write block 4185 len 24576 data: 2 2 3 3 3 3
2208 035/300 Write block 372 len 24576 data: 2 2 2 2 2 2
2209 036/300 Write block 600 len 12288 data: 3 3 3
2210 037/300 Write block 3247 len 24576 data: 3 3 3 2 2 2
2211 038/300 Read block 4456 len 40960
2212 039/300 Write block 4170 len 24576 data: 2 2 2 2 2 2
2213 040/300 Read block 4984 len 20480
2214 041/300 Read block 397 len 4096
2215 042/300 Read block 2735 len 8192
2216 043/300 Read block 2836 len 36864
2217 044/300 Write block 1612 len 12288 data: 2 2 2
2218 045/300 Read block 125 len 8192
2219 046/300 Write block 3607 len 20480 data: 2 2 2 2 2
2220 047/300 Write block 4602 len 28672 data: 3 2 2 2 2 2 2
2221 048/300 Write block 141 len 20480 data: 2 2 2 2 2
2222 049/300 Write block 2817 len 20480 data: 2 2 2 2 2
2223 050/300 Read block 264 len 4096
2224 051/300 Write block 2187 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2225 052/300 Write block 1623 len 40960 data: 2 2 3 3 3 3 2 2 2 2
2226 053/300 Read block 304 len 28672
2227 054/300 Flush
2228 055/300 Write block 4923 len 8192 data: 3 3
2229 056/300 Read block 3035 len 36864
2230 057/300 Read block 3581 len 8192
2231 058/300 Read block 1967 len 40960
2232 059/300 Read block 2612 len 24576
2233 060/300 Write block 282 len 4096 data: 3
2234 061/300 Read block 2899 len 4096
2235 062/300 Write block 2290 len 8192 data: 2 2
2236 063/300 Write block 4959 len 32768 data: 2 2 2 2 2 2 2 2
2237 064/300 Write block 4292 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2238 065/300 Flush
2239 066/300 Read block 1717 len 12288
2240 067/300 Read block 4356 len 16384
2241 068/300 Write block 4382 len 32768 data: 2 2 2 2 2 2 2 2
2242 069/300 Read block 3743 len 16384
2243 070/300 Read block 1396 len 8192
2244 071/300 Read block 3637 len 32768
2245 072/300 Read block 3484 len 40960
2246 073/300 Flush
2247 074/300 Flush
2248 075/300 Write block 4914 len 24576 data: 2 2 3 3 3 3
2249 076/300 Write block 2347 len 32768 data: 2 2 2 2 2 2 2 2
2250 077/300 Read block 4262 len 40960
2251 078/300 Read block 1152 len 8192
2252 079/300 Read block 4516 len 36864
2253 080/300 Read block 1389 len 20480
2254 081/300 Read block 3901 len 36864
2255 082/300 Flush
2256 083/300 Write block 3696 len 8192 data: 2 2
2257 084/300 Read block 3430 len 16384
2258 085/300 Read block 1512 len 24576
2259 086/300 Write block 2105 len 24576 data: 2 2 2 2 2 2
2260 087/300 Read block 15 len 24576
2261 088/300 Write block 4255 len 8192 data: 2 2
2262 089/300 Read block 3178 len 32768
2263 090/300 Read block 905 len 28672
2264 091/300 Read block 2595 len 20480
2265 092/300 Write block 335 len 12288 data: 2 2 2
2266 093/300 Read block 1332 len 32768
2267 094/300 Read block 1576 len 36864
2268 095/300 Write block 4573 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2269 096/300 Write block 1697 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2270 097/300 Read block 3583 len 4096
2271 098/300 Write block 915 len 4096 data: 2
2272 099/300 Read block 4655 len 4096
2273 100/300 Write block 2309 len 8192 data: 2 2
2274 101/300 Flush
2275 102/300 Read block 975 len 24576
2276 103/300 Read block 1630 len 16384
2277 104/300 Read block 741 len 36864
2278 105/300 Flush
2279 106/300 Read block 1845 len 32768
2280 107/300 Write block 3425 len 32768 data: 2 2 2 2 2 2 2 2
2281 108/300 Write block 3496 len 20480 data: 3 3 3 3 3
2282 109/300 Read block 1278 len 40960
2283 110/300 Write block 4399 len 12288 data: 2 2 2
2284 111/300 Read block 3457 len 40960
2285 112/300 Read block 4448 len 4096
2286 113/300 Write block 2870 len 36864 data: 2 2 2 2 2 2 2 2 2
2287 114/300 Write block 513 len 4096 data: 2
2288 115/300 Write block 139 len 40960 data: 2 2 3 3 3 3 3 2 2 2
2289 116/300 Read block 1849 len 8192
2290 117/300 Flush
2291 118/300 Write block 2113 len 40960 data: 2 2 2 2 3 3 3 3 3 3
2292 119/300 Write block 3867 len 4096 data: 2
2293 120/300 Read block 555 len 12288
2294 121/300 Read block 1441 len 32768
2295 122/300 Read block 3590 len 20480
2296 123/300 Write block 697 len 4096 data: 2
2297 124/300 Write block 625 len 4096 data: 3
2298 125/300 Flush
2299 126/300 Read block 1555 len 28672
2300 127/300 Write block 4346 len 28672 data: 2 2 2 2 2 3 3
2301 128/300 Write block 1848 len 8192 data: 2 2
2302 129/300 Read block 4787 len 36864
2303 130/300 Read block 3865 len 40960
2304 131/300 Write block 2092 len 8192 data: 2 2
2305 132/300 Read block 767 len 32768
2306 133/300 Read block 4254 len 28672
2307 134/300 Read block 829 len 24576
2308 135/300 Read block 3218 len 12288
2309 136/300 Read block 1581 len 32768
2310 137/300 Write block 155 len 24576 data: 2 2 2 2 2 2
2311 138/300 Read block 1842 len 28672
2312 139/300 Read block 4574 len 8192
2313 140/300 Write block 4785 len 24576 data: 2 2 2 2 2 2
2314 141/300 Write block 3918 len 20480 data: 3 3 3 3 2
2315 142/300 Read block 2425 len 20480
2316 143/300 Read block 3541 len 36864
2317 144/300 Write block 476 len 8192 data: 2 2
2318 145/300 Write block 4333 len 20480 data: 2 2 2 2 2
2319 146/300 Flush
2320 147/300 Read block 2513 len 32768
2321 148/300 Write block 1798 len 32768 data: 2 2 2 2 2 2 2 2
2322 149/300 Read block 3555 len 16384
2323 150/300 Read block 741 len 12288
2324 151/300 Read block 2052 len 24576
2325 152/300 Read block 1343 len 28672
2326 153/300 Read block 671 len 4096
2327 154/300 Write block 2653 len 32768 data: 2 2 2 2 2 2 2 2
2328 155/300 Read block 155 len 24576
2329 156/300 Write block 2600 len 24576 data: 2 2 2 2 3 3
2330 157/300 Read block 2026 len 32768
2331 158/300 Read block 3317 len 8192
2332 159/300 Write block 576 len 8192 data: 2 2
2333 160/300 Write block 1811 len 28672 data: 2 2 3 3 3 3 2
2334 161/300 Read block 3342 len 4096
2335 162/300 Write block 1958 len 16384 data: 2 2 2 2
2336 163/300 Read block 4117 len 8192
2337 164/300 Write block 3976 len 8192 data: 2 2
2338 165/300 Read block 2864 len 20480
2339 166/300 Read block 3905 len 12288
2340 167/300 Read block 3862 len 8192
2341 168/300 Read block 2800 len 12288
2342 169/300 Write block 2284 len 4096 data: 3
2343 170/300 Write block 2539 len 16384 data: 2 2 2 2
2344 171/300 Read block 1181 len 20480
2345 172/300 Read block 1752 len 28672
2346 173/300 Write block 3772 len 24576 data: 2 2 2 2 2 2
2347 174/300 Write block 1049 len 32768 data: 2 2 2 2 3 3 2 2
2348 175/300 Write block 338 len 12288 data: 2 2 2
2349 176/300 Read block 4896 len 36864
2350 177/300 Read block 4849 len 36864
2351 178/300 Write block 2571 len 40960 data: 3 3 3 3 3 3 2 2 2 2
2352 179/300 Flush
2353 180/300 Write block 2093 len 36864 data: 3 2 2 2 2 2 2 2 2
2354 181/300 Flush
2355 182/300 Read block 4211 len 8192
2356 183/300 Read block 4772 len 36864
2357 184/300 Read block 1880 len 4096
2358 185/300 Read block 2223 len 12288
2359 186/300 Read block 1574 len 12288
2360 187/300 Write block 2123 len 36864 data: 3 2 2 2 2 2 2 2 2
2361 188/300 Read block 3303 len 24576
2362 189/300 Read block 1023 len 32768
2363 190/300 Read block 2917 len 20480
2364 191/300 Write block 1180 len 8192 data: 2 2
2365 192/300 Write block 4201 len 12288 data: 2 2 2
2366 193/300 Read block 922 len 36864
2367 194/300 Write block 3211 len 28672 data: 2 2 2 2 2 2 2
2368 195/300 Write block 4618 len 4096 data: 2
2369 196/300 Read block 2902 len 40960
2370 197/300 Read block 1166 len 28672
2371 198/300 Write block 1176 len 32768 data: 2 2 2 2 3 3 2 2
2372 199/300 Write block 4449 len 24576 data: 2 2 2 2 2 2
2373 200/300 Read block 3597 len 24576
2374 201/300 Read block 4776 len 32768
2375 202/300 Flush
2376 203/300 Read block 923 len 28672
2377 204/300 Read block 4547 len 4096
2378 205/300 Write block 2747 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2379 206/300 Flush
2380 207/300 Read block 4713 len 32768
2381 208/300 Read block 3355 len 24576
2382 209/300 Read block 2368 len 4096
2383 210/300 Read block 436 len 20480
2384 211/300 Read block 1892 len 20480
2385 212/300 Write block 954 len 32768 data: 2 2 2 2 2 2 2 2
2386 213/300 Write block 4038 len 16384 data: 2 2 3 3
2387 214/300 Read block 3115 len 32768
2388 215/300 Write block 2356 len 32768 data: 2 2 2 2 2 2 2 2
2389 216/300 Write block 1611 len 12288 data: 2 3 3
2390 217/300 Read block 2875 len 36864
2391 218/300 Write block 2959 len 20480 data: 2 2 2 2 2
2392 219/300 Read block 2844 len 40960
2393 220/300 Read block 4556 len 28672
2394 221/300 Read block 766 len 36864
2395 222/300 Read block 1036 len 12288
2396 223/300 Read block 2228 len 28672
2397 224/300 Read block 2077 len 12288
2398 225/300 Read block 2187 len 20480
2399 226/300 Read block 1339 len 20480
2400 227/300 Flush
2401 228/300 Flush
2402 229/300 Read block 2849 len 36864
2403 230/300 Write block 3347 len 36864 data: 3 3 3 3 3 2 2 3 3
2404 231/300 Write block 3977 len 8192 data: 3 2
2405 232/300 Write block 3910 len 12288 data: 2 2 2
2406 233/300 Read block 2302 len 32768
2407 234/300 Read block 4247 len 4096
2408 235/300 Flush
2409 236/300 Flush
2410 237/300 Write block 381 len 32768 data: 2 2 2 2 2 2 2 2
2411 238/300 Read block 2179 len 36864
2412 239/300 Read block 1954 len 12288
2413 240/300 Flush
2414 241/300 Write block 2751 len 16384 data: 3 3 3 3
2415 242/300 Write block 4614 len 20480 data: 2 2 2 2 3
2416 243/300 Write block 2986 len 24576 data: 2 2 2 2 2 2
2417 244/300 Read block 3725 len 16384
2418 245/300 Flush
2419 246/300 Read block 976 len 32768
2420 247/300 Read block 441 len 24576
2421 248/300 Write block 3737 len 32768 data: 2 2 2 2 2 2 2 2
2422 249/300 Read block 549 len 8192
2423 250/300 Write block 1505 len 8192 data: 2 2
2424 251/300 Read block 1975 len 40960
2425 252/300 Read block 1076 len 36864
2426 253/300 Write block 4312 len 36864 data: 2 2 2 2 2 2 2 2 2
2427 254/300 Write block 3120 len 28672 data: 3 2 2 2 3 3 2
2428 255/300 Read block 2912 len 32768
2429 256/300 Read block 2635 len 28672
2430 257/300 Write block 310 len 8192 data: 2 2
2431 258/300 Read block 357 len 36864
2432 259/300 Read block 2184 len 20480
2433 260/300 Write block 4032 len 24576 data: 2 2 2 2 2 2
2434 261/300 Read block 4290 len 28672
2435 262/300 Read block 305 len 40960
2436 263/300 Write block 2226 len 20480 data: 3 3 4 3 3
2437 264/300 Write block 4061 len 12288 data: 3 2 2
2438 265/300 Read block 1995 len 40960
2439 266/300 Write block 4858 len 40960 data: 2 3 3 3 3 3 2 2 2 2
2440 267/300 Read block 3327 len 8192
2441 268/300 Write block 585 len 28672 data: 2 2 2 2 2 2 3
2442 269/300 Read block 375 len 36864
2443 270/300 Write block 1525 len 12288 data: 2 2 2
2444 271/300 Read block 1677 len 8192
2445 272/300 Read block 411 len 24576
2446 273/300 Read block 622 len 4096
2447 274/300 Read block 180 len 16384
2448 275/300 Read block 1554 len 40960
2449 276/300 Write block 4730 len 4096 data: 2
2450 277/300 Read block 872 len 36864
2451 278/300 Flush
2452 279/300 Write block 4182 len 12288 data: 2 2 2
2453 280/300 Write block 3770 len 16384 data: 2 2 3 3
2454 281/300 Read block 1286 len 32768
2455 282/300 Read block 4792 len 16384
2456 283/300 Flush
2457 284/300 Write block 4625 len 4096 data: 2
2458 285/300 Flush
2459 286/300 Read block 4284 len 36864
2460 287/300 Write block 2686 len 20480 data: 2 2 2 2 2
2461 288/300 Read block 4118 len 20480
2462 289/300 Write block 976 len 40960 data: 3 3 2 2 2 3 2 2 2 2
2463 290/300 Read block 3924 len 40960
2464 291/300 Write block 3431 len 12288 data: 3 3 2
2465 292/300 Write block 4853 len 12288 data: 2 2 2
2466 293/300 Read block 2171 len 20480
2467 294/300 Read block 2951 len 8192
2468 295/300 Read block 2154 len 32768
2469 296/300 Read block 2614 len 20480
2470 297/300 Flush
2471 298/300 Write block 2421 len 16384 data: 2 2 3 3
2472 299/300 Read block 4690 len 20480
2473 300/300 Read block 494 len 12288
2474 301/300 Write block 3266 len 8192 data: 3 2
2475 Main task has work to do, go find it
2476 got dsc Start(1)
2477 start 1
2478 Feb 21 06:45:51.955 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/1, method: POST, req_id: 980826f4-96d7-4382-a1e0-027ef76589ff, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
2479 Replay: started 1, returned:Ok(())
2480 ----------------------------------------------------------------
2481 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2482 States: Active Offline Active
2483 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2484 1604 Acked Read 6 Done Sent Done false
2485 1605 Acked Read 8 Done Sent Done false
2486 1606 Acked Flush 0 Done Sent Done false
2487 1607 Acked Write 3 Done Sent Done false
2488 1608 Acked Write 4 Done Sent Done false
2489 1609 Acked Write 8 Done Sent Done false
2490 1610 Acked Write 6 Done Sent Done false
2491 1611 Acked Write 8 Done Sent Done false
2492 1612 Acked Read 1 Done Sent Done false
2493 1613 Acked Write 2 Done Sent Done false
2494 1614 Acked Read 10 Done Sent Done false
2495 1615 Acked Read 9 Done Sent Done false
2496 1616 Acked Write 8 Done Sent Done false
2497 1617 Acked Read 6 Done Sent Done false
2498 1618 Acked Write 2 Done Sent Done false
2499 1619 Acked Write 2 Done Sent Done false
2500 1620 Acked Write 10 Done Sent Done false
2501 1621 Acked Flush 0 Done Sent Done false
2502 1622 Acked Flush 0 Done Sent Done false
2503 1623 Acked Write 4 Done Sent Done false
2504 1624 Acked Write 1 Done Sent Done false
2505 1625 Acked Write 2 Done Sent Done false
2506 1626 Acked Read 4 Done Sent Done false
2507 1627 Acked Read 5 Done Sent Done false
2508 1628 Acked Write 7 Done Sent Done false
2509 1629 Acked Read 10 Done Sent Done false
2510 1630 Acked Write 8 Done Sent Done false
2511 1631 Acked Read 10 Done Sent Done false
2512 1632 Acked Read 7 Done Sent Done false
2513 1633 Acked Read 2 Done Sent Done false
2514 1634 Acked Read 3 Done Sent Done false
2515 1635 Acked Write 1 Done Sent Done false
2516 1636 Acked Read 1 Done Sent Done false
2517 1637 Acked Write 6 Done Sent Done false
2518 1638 Acked Write 6 Done Sent Done false
2519 1639 Acked Write 3 Done Sent Done false
2520 1640 Acked Write 6 Done Sent Done false
2521 1641 Acked Read 10 Done Sent Done false
2522 1642 Acked Write 6 Done Sent Done false
2523 1643 Acked Read 5 Done Sent Done false
2524 1644 Acked Read 1 Done Sent Done false
2525 1645 Acked Read 2 Done Sent Done false
2526 1646 Acked Read 9 Done Sent Done false
2527 1647 Acked Write 3 Done Sent Done false
2528 1648 Acked Read 2 Done Sent Done false
2529 1649 Acked Write 5 Done Sent Done false
2530 1650 Acked Write 7 Done Sent Done false
2531 1651 Acked Write 5 Done Sent Done false
2532 1652 Acked Write 5 Done Sent Done false
2533 1653 Acked Read 1 Done Sent Done false
2534 1654 Acked Write 10 Done Sent Done false
2535 1655 Acked Write 10 Done Sent Done false
2536 1656 Acked Read 7 Done Sent Done false
2537 1657 Acked Flush 0 Done Sent Done false
2538 1658 Acked Write 2 Done Sent Done false
2539 1659 Acked Read 9 Done Sent Done false
2540 1660 Acked Read 2 Done Sent Done false
2541 1661 Acked Read 10 Done Sent Done false
2542 1662 Acked Read 6 Done Sent Done false
2543 1663 Acked Write 1 Done Sent Done false
2544 1664 Acked Read 1 Done Sent Done false
2545 1665 Acked Write 2 Done Sent Done false
2546 1666 Acked Write 8 Done Sent Done false
2547 1667 Acked Write 10 Done Sent Done false
2548 1668 Acked Flush 0 Done Sent Done false
2549 1669 Acked Read 3 Done Sent Done false
2550 1670 Acked Read 4 Done Sent Done false
2551 1671 Acked Write 8 Done Sent Done false
2552 1672 Acked Read 4 Done Sent Done false
2553 1673 Acked Read 2 Done Sent Done false
2554 1674 Acked Read 8 Done Sent Done false
2555 1675 Acked Read 10 Done Sent Done false
2556 1676 Acked Flush 0 Done Sent Done false
2557 1677 Acked Flush 0 Done Sent Done false
2558 1678 Acked Write 6 Done Sent Done false
2559 1679 Acked Write 8 Done Sent Done false
2560 1680 Acked Read 10 Done Sent Done false
2561 1681 Acked Read 2 Done Sent Done false
2562 1682 Acked Read 9 Done Sent Done false
2563 1683 Acked Read 5 Done Sent Done false
2564 1684 Acked Read 9 Done Sent Done false
2565 1685 Acked Flush 0 Done Sent Done false
2566 1686 Acked Write 2 Done Sent Done false
2567 1687 Acked Read 4 Done Sent Done false
2568 1688 Acked Read 6 Done Sent Done false
2569 1689 Acked Write 6 Done Sent Done false
2570 1690 Acked Read 6 Done Sent Done false
2571 1691 Acked Write 2 Done Sent Done false
2572 1692 Acked Read 8 Done Sent Done false
2573 1693 Acked Read 7 Done Sent Done false
2574 1694 Acked Read 5 Done Sent Done false
2575 1695 Acked Write 3 Done Sent Done false
2576 1696 Acked Read 8 Done Sent Done false
2577 1697 Acked Read 9 Done Sent Done false
2578 1698 Acked Write 10 Done Sent Done false
2579 1699 Acked Write 10 Done Sent Done false
2580 1700 Acked Read 1 Done Sent Done false
2581 1701 Acked Write 1 Done Sent Done false
2582 1702 Acked Read 1 Done Sent Done false
2583 1703 Acked Write 2 Done Sent Done false
2584 1704 Acked Flush 0 Done Sent Done false
2585 1705 Acked Read 6 Done Sent Done false
2586 1706 Acked Read 4 Done Sent Done false
2587 1707 Acked Read 9 Done Sent Done false
2588 1708 Acked Flush 0 Done Sent Done false
2589 1709 Acked Read 8 Done Sent Done false
2590 1710 Acked Write 8 Done Sent Done false
2591 1711 Acked Write 5 Done Sent Done false
2592 1712 Acked Read 10 Done Sent Done false
2593 1713 Acked Write 3 Done Sent Done false
2594 1714 Acked Read 10 Done Sent Done false
2595 1715 Acked Read 1 Done Sent Done false
2596 1716 Acked Write 9 Done Sent Done false
2597 1717 Acked Write 1 Done Sent Done false
2598 1718 Acked Write 10 Done Sent Done false
2599 1719 Acked Read 2 Done Sent Done false
2600 1720 Acked Flush 0 Done Sent Done false
2601 1721 Acked Write 10 Done Sent Done false
2602 1722 Acked Write 1 Done Sent Done false
2603 1723 Acked Read 3 Done Sent Done false
2604 1724 Acked Read 8 Done Sent Done false
2605 1725 Acked Read 5 Done Sent Done false
2606 1726 Acked Write 1 Done Sent Done false
2607 1727 Acked Write 1 Done Sent Done false
2608 1728 Acked Flush 0 Done Sent Done false
2609 1729 Acked Read 7 Done Sent Done false
2610 1730 Acked Write 7 Done Sent Done false
2611 1731 Acked Write 2 Done Sent Done false
2612 1732 Acked Read 9 Done Sent Done false
2613 1733 Acked Read 10 Done Sent Done false
2614 1734 Acked Write 2 Done Sent Done false
2615 1735 Acked Read 8 Done Sent Done false
2616 1736 Acked Read 7 Done Sent Done false
2617 1737 Acked Read 6 Done Sent Done false
2618 1738 Acked Read 3 Done Sent Done false
2619 1739 Acked Read 8 Done Sent Done false
2620 1740 Acked Write 6 Done Sent Done false
2621 1741 Acked Read 7 Done Sent Done false
2622 1742 Acked Read 2 Done Sent Done false
2623 1743 Acked Write 6 Done Sent Done false
2624 1744 Acked Write 5 Done Sent Done false
2625 1745 Acked Read 5 Done Sent Done false
2626 1746 Acked Read 9 Done Sent Done false
2627 1747 Acked Write 2 Done Sent Done false
2628 1748 Acked Write 5 Done Sent Done false
2629 1749 Acked Flush 0 Done Sent Done false
2630 1750 Acked Read 8 Done Sent Done false
2631 1751 Acked Write 8 Done Sent Done false
2632 1752 Acked Read 4 Done Sent Done false
2633 1753 Acked Read 3 Done Sent Done false
2634 1754 Acked Read 6 Done Sent Done false
2635 1755 Acked Read 7 Done Sent Done false
2636 1756 Acked Read 1 Done Sent Done false
2637 1757 Acked Write 8 Done Sent Done false
2638 1758 Acked Read 6 Done Sent Done false
2639 1759 Acked Write 6 Done Sent Done false
2640 1760 Acked Read 8 Done Sent Done false
2641 1761 Acked Read 2 Done Sent Done false
2642 1762 Acked Write 2 Done Sent Done false
2643 1763 Acked Write 7 Done Sent Done false
2644 1764 Acked Read 1 Done Sent Done false
2645 1765 Acked Write 4 Done Sent Done false
2646 1766 Acked Read 2 Done Sent Done false
2647 1767 Acked Write 2 Done Sent Done false
2648 1768 Acked Read 5 Done Sent Done false
2649 1769 Acked Read 3 Done Sent Done false
2650 1770 Acked Read 2 Done Sent Done false
2651 1771 Acked Read 3 Done Sent Done false
2652 1772 Acked Write 1 Done Sent Done false
2653 1773 Acked Write 4 Done Sent Done false
2654 1774 Acked Read 5 Done Sent Done false
2655 1775 Acked Read 7 Done Sent Done false
2656 1776 Acked Write 6 Done Sent Done false
2657 1777 Acked Write 8 Done Sent Done false
2658 1778 Acked Write 3 Done Sent Done false
2659 1779 Acked Read 9 Done Sent Done false
2660 1780 Acked Read 9 Done Sent Done false
2661 1781 Acked Write 10 Done Sent Done false
2662 1782 Acked Flush 0 Done Sent Done false
2663 1783 Acked Write 9 Done Sent Done false
2664 1784 Acked Flush 0 Done Sent Done false
2665 1785 Acked Read 2 Done Sent Done false
2666 1786 Acked Read 9 Done Sent Done false
2667 1787 Acked Read 1 Done Sent Done false
2668 1788 Acked Read 3 Done Sent Done false
2669 1789 Acked Read 3 Done Sent Done false
2670 1790 Acked Write 9 Done Sent Done false
2671 1791 Acked Read 6 Done Sent Done false
2672 1792 Acked Read 8 Done Sent Done false
2673 1793 Acked Read 5 Done Sent Done false
2674 1794 Acked Write 2 Done Sent Done false
2675 1795 Acked Write 3 Done Sent Done false
2676 1796 Acked Read 9 Done Sent Done false
2677 1797 Acked Write 7 Done Sent Done false
2678 1798 Acked Write 1 Done Sent Done false
2679 1799 Acked Read 10 Done Sent Done false
2680 1800 Acked Read 7 Done Sent Done false
2681 1801 Acked Write 8 Done Sent Done false
2682 1802 Acked Write 6 Done Sent Done false
2683 1803 Acked Read 6 Done Sent Done false
2684 1804 Acked Read 8 Done Sent Done false
2685 1805 Acked Flush 0 Done Sent Done false
2686 1806 Acked Read 7 Done Sent Done false
2687 1807 Acked Read 1 Done Sent Done false
2688 1808 Acked Write 10 Done Sent Done false
2689 1809 Acked Flush 0 Done Sent Done false
2690 1810 Acked Read 8 Done Sent Done false
2691 1811 Acked Read 6 Done Sent Done false
2692 1812 Acked Read 1 Done Sent Done false
2693 1813 Acked Read 5 Done Sent Done false
2694 1814 Acked Read 5 Done Sent Done false
2695 1815 Acked Write 8 Done Sent Done false
2696 1816 Acked Write 4 Done Sent Done false
2697 1817 Acked Read 8 Done Sent Done false
2698 1818 Acked Write 8 Done Sent Done false
2699 1819 Acked Write 3 Done Sent Done false
2700 1820 Acked Read 9 Done Sent Done false
2701 1821 Acked Write 5 Done Sent Done false
2702 1822 Acked Read 10 Done Sent Done false
2703 1823 Acked Read 7 Done Sent Done false
2704 1824 Acked Read 9 Done Sent Done false
2705 1825 Acked Read 3 Done Sent Done false
2706 1826 Acked Read 7 Done Sent Done false
2707 1827 Acked Read 3 Done Sent Done false
2708 1828 Acked Read 5 Done Sent Done false
2709 1829 Acked Read 5 Done Sent Done false
2710 1830 Acked Flush 0 Done Sent Done false
2711 1831 Acked Flush 0 Done Sent Done false
2712 1832 Acked Read 9 Done Sent Done false
2713 1833 Acked Write 9 Done Sent Done false
2714 1834 Acked Write 2 Done Sent Done false
2715 1835 Acked Write 3 Done Sent Done false
2716 1836 Acked Read 8 Done Sent Done false
2717 1837 Acked Read 1 Done Sent Done false
2718 1838 Acked Flush 0 Done Sent Done false
2719 1839 Acked Flush 0 Done Sent Done false
2720 1840 Acked Write 8 Done Sent Done false
2721 1841 Acked Read 9 Done Sent Done false
2722 1842 Acked Read 3 Done Sent Done false
2723 1843 Acked Flush 0 Done Sent Done false
2724 1844 Acked Write 4 Done Sent Done false
2725 1845 Acked Write 5 Done Sent Done false
2726 1846 Acked Write 6 Done Sent Done false
2727 1847 Acked Read 4 Done Sent Done false
2728 1848 Acked Flush 0 Done Sent Done false
2729 1849 Acked Read 8 Done Sent Done false
2730 1850 Acked Read 6 Done Sent Done false
2731 1851 Acked Write 8 Done Sent Done false
2732 1852 Acked Read 2 Done Sent Done false
2733 1853 Acked Write 2 Done Sent Done false
2734 1854 Acked Read 10 Done Sent Done false
2735 1855 Acked Read 9 Done Sent Done false
2736 1856 Acked Write 9 Done Sent Done false
2737 1857 Acked Write 7 Done Sent Done false
2738 1858 Acked Read 8 Done Sent Done false
2739 1859 Acked Read 7 Done Sent Done false
2740 1860 Acked Write 2 Done Sent Done false
2741 1861 Acked Read 9 Done Sent Done false
2742 1862 Acked Read 5 Done Sent Done false
2743 1863 Acked Write 6 Done Sent Done false
2744 1864 Acked Read 7 Done Sent Done false
2745 1865 Acked Read 10 Done Sent Done false
2746 1866 Acked Write 5 Done Sent Done false
2747 1867 Acked Write 3 Done Sent Done false
2748 1868 Acked Read 10 Done Sent Done false
2749 1869 Acked Write 10 Done Sent Done false
2750 1870 Acked Read 2 Done Sent Done false
2751 1871 Acked Write 7 Done Sent Done false
2752 1872 Acked Read 9 Done Sent Done false
2753 1873 Acked Write 3 Done Sent Done false
2754 1874 Acked Read 2 Done Sent Done false
2755 1875 Acked Read 6 Done Sent Done false
2756 1876 Acked Read 1 Done Sent Done false
2757 1877 Acked Read 4 Done Sent Done false
2758 1878 Acked Read 10 Done Sent Done false
2759 1879 Acked Write 1 Done Sent Done false
2760 1880 Acked Read 9 Done Sent Done false
2761 1881 Acked Flush 0 Done Sent Done false
2762 1882 Acked Write 3 Done Sent Done false
2763 1883 Acked Write 4 Done Sent Done false
2764 1884 Acked Read 8 Done Sent Done false
2765 1885 Acked Read 4 Done Sent Done false
2766 1886 Acked Flush 0 Done Sent Done false
2767 1887 Acked Write 1 Done Sent Done false
2768 1888 Acked Flush 0 Done Sent Done false
2769 1889 Acked Read 9 Done Sent Done false
2770 1890 Acked Write 5 Done Sent Done false
2771 1891 Acked Read 5 Done Sent Done false
2772 1892 Acked Write 10 Done Sent Done false
2773 1893 Acked Read 10 Done Sent Done false
2774 1894 Acked Write 3 Done Sent Done false
2775 1895 Acked Write 3 Done Sent Done false
2776 1896 Acked Read 5 Done Sent Done false
2777 1897 Acked Read 2 Done Sent Done false
2778 1898 Acked Read 8 Done Sent Done false
2779 1899 Acked Read 5 Done Sent Done false
2780 1900 Acked Flush 0 Done Sent Done false
2781 1901 Acked Write 4 Done Sent Done false
2782 1902 Acked Read 5 Done Sent Done false
2783 1903 Acked Read 3 Done Sent Done false
2784 1904 Acked Write 2 Done Sent Done false
2785 STATES DS:0 DS:1 DS:2 TOTAL
2786 Sent 0 301 0 301
2787 Done 301 0 301 602
2788 Skipped 0 0 0 0
2789 Error 0 0 0 0
2790 Last Flush: Some(JobId(1900)) Some(JobId(1603)) Some(JobId(1900))
2791 Downstairs last five completed: 1603 1602 1601 1600 1599
2792 Upstairs last five completed: 1904 1903 1902 1901 1900
2793 CLIENT: Up:0 ds:301 act:2
27942025-02-21T06:45:52.704ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
27952025-02-21T06:45:53.705ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
27962025-02-21T06:45:54.707ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
2797 [8820] got start action
2798 [8820] I am going to restart
2799 Starting downstairs at port 8820
2800 Make output file at "/tmp/dsc/downstairs-8820.txt"
2801 [8820][1] reports Starting
2802 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1209)
2803 [8820][1] reports Running
2804 ----------------------------------------------------------------
2805 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2806 States: Active Offline Active
2807 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2808 1604 Acked Read 6 Done Sent Done false
2809 1605 Acked Read 8 Done Sent Done false
2810 1606 Acked Flush 0 Done Sent Done false
2811 1607 Acked Write 3 Done Sent Done false
2812 1608 Acked Write 4 Done Sent Done false
2813 1609 Acked Write 8 Done Sent Done false
2814 1610 Acked Write 6 Done Sent Done false
2815 1611 Acked Write 8 Done Sent Done false
2816 1612 Acked Read 1 Done Sent Done false
2817 1613 Acked Write 2 Done Sent Done false
2818 1614 Acked Read 10 Done Sent Done false
2819 1615 Acked Read 9 Done Sent Done false
2820 1616 Acked Write 8 Done Sent Done false
2821 1617 Acked Read 6 Done Sent Done false
2822 1618 Acked Write 2 Done Sent Done false
2823 1619 Acked Write 2 Done Sent Done false
2824 1620 Acked Write 10 Done Sent Done false
2825 1621 Acked Flush 0 Done Sent Done false
2826 1622 Acked Flush 0 Done Sent Done false
2827 1623 Acked Write 4 Done Sent Done false
2828 1624 Acked Write 1 Done Sent Done false
2829 1625 Acked Write 2 Done Sent Done false
2830 1626 Acked Read 4 Done Sent Done false
2831 1627 Acked Read 5 Done Sent Done false
2832 1628 Acked Write 7 Done Sent Done false
2833 1629 Acked Read 10 Done Sent Done false
2834 1630 Acked Write 8 Done Sent Done false
2835 1631 Acked Read 10 Done Sent Done false
2836 1632 Acked Read 7 Done Sent Done false
2837 1633 Acked Read 2 Done Sent Done false
2838 1634 Acked Read 3 Done Sent Done false
2839 1635 Acked Write 1 Done Sent Done false
2840 1636 Acked Read 1 Done Sent Done false
2841 1637 Acked Write 6 Done Sent Done false
2842 1638 Acked Write 6 Done Sent Done false
2843 1639 Acked Write 3 Done Sent Done false
2844 1640 Acked Write 6 Done Sent Done false
2845 1641 Acked Read 10 Done Sent Done false
2846 1642 Acked Write 6 Done Sent Done false
2847 1643 Acked Read 5 Done Sent Done false
2848 1644 Acked Read 1 Done Sent Done false
2849 1645 Acked Read 2 Done Sent Done false
2850 1646 Acked Read 9 Done Sent Done false
2851 1647 Acked Write 3 Done Sent Done false
2852 1648 Acked Read 2 Done Sent Done false
2853 1649 Acked Write 5 Done Sent Done false
2854 1650 Acked Write 7 Done Sent Done false
2855 1651 Acked Write 5 Done Sent Done false
2856 1652 Acked Write 5 Done Sent Done false
2857 1653 Acked Read 1 Done Sent Done false
2858 1654 Acked Write 10 Done Sent Done false
2859 1655 Acked Write 10 Done Sent Done false
2860 1656 Acked Read 7 Done Sent Done false
2861 1657 Acked Flush 0 Done Sent Done false
2862 1658 Acked Write 2 Done Sent Done false
2863 1659 Acked Read 9 Done Sent Done false
2864 1660 Acked Read 2 Done Sent Done false
2865 1661 Acked Read 10 Done Sent Done false
2866 1662 Acked Read 6 Done Sent Done false
2867 1663 Acked Write 1 Done Sent Done false
2868 1664 Acked Read 1 Done Sent Done false
2869 1665 Acked Write 2 Done Sent Done false
2870 1666 Acked Write 8 Done Sent Done false
2871 1667 Acked Write 10 Done Sent Done false
2872 1668 Acked Flush 0 Done Sent Done false
2873 1669 Acked Read 3 Done Sent Done false
2874 1670 Acked Read 4 Done Sent Done false
2875 1671 Acked Write 8 Done Sent Done false
2876 1672 Acked Read 4 Done Sent Done false
2877 1673 Acked Read 2 Done Sent Done false
2878 1674 Acked Read 8 Done Sent Done false
2879 1675 Acked Read 10 Done Sent Done false
2880 1676 Acked Flush 0 Done Sent Done false
2881 1677 Acked Flush 0 Done Sent Done false
2882 1678 Acked Write 6 Done Sent Done false
2883 1679 Acked Write 8 Done Sent Done false
2884 1680 Acked Read 10 Done Sent Done false
2885 1681 Acked Read 2 Done Sent Done false
2886 1682 Acked Read 9 Done Sent Done false
2887 1683 Acked Read 5 Done Sent Done false
2888 1684 Acked Read 9 Done Sent Done false
2889 1685 Acked Flush 0 Done Sent Done false
2890 1686 Acked Write 2 Done Sent Done false
2891 1687 Acked Read 4 Done Sent Done false
2892 1688 Acked Read 6 Done Sent Done false
2893 1689 Acked Write 6 Done Sent Done false
2894 1690 Acked Read 6 Done Sent Done false
2895 1691 Acked Write 2 Done Sent Done false
2896 1692 Acked Read 8 Done Sent Done false
2897 1693 Acked Read 7 Done Sent Done false
2898 1694 Acked Read 5 Done Sent Done false
2899 1695 Acked Write 3 Done Sent Done false
2900 1696 Acked Read 8 Done Sent Done false
2901 1697 Acked Read 9 Done Sent Done false
2902 1698 Acked Write 10 Done Sent Done false
2903 1699 Acked Write 10 Done Sent Done false
2904 1700 Acked Read 1 Done Sent Done false
2905 1701 Acked Write 1 Done Sent Done false
2906 1702 Acked Read 1 Done Sent Done false
2907 1703 Acked Write 2 Done Sent Done false
2908 1704 Acked Flush 0 Done Sent Done false
2909 1705 Acked Read 6 Done Sent Done false
2910 1706 Acked Read 4 Done Sent Done false
2911 1707 Acked Read 9 Done Sent Done false
2912 1708 Acked Flush 0 Done Sent Done false
2913 1709 Acked Read 8 Done Sent Done false
2914 1710 Acked Write 8 Done Sent Done false
2915 1711 Acked Write 5 Done Sent Done false
2916 1712 Acked Read 10 Done Sent Done false
2917 1713 Acked Write 3 Done Sent Done false
2918 1714 Acked Read 10 Done Sent Done false
2919 1715 Acked Read 1 Done Sent Done false
2920 1716 Acked Write 9 Done Sent Done false
2921 1717 Acked Write 1 Done Sent Done false
2922 1718 Acked Write 10 Done Sent Done false
2923 1719 Acked Read 2 Done Sent Done false
2924 1720 Acked Flush 0 Done Sent Done false
2925 1721 Acked Write 10 Done Sent Done false
2926 1722 Acked Write 1 Done Sent Done false
2927 1723 Acked Read 3 Done Sent Done false
2928 1724 Acked Read 8 Done Sent Done false
2929 1725 Acked Read 5 Done Sent Done false
2930 1726 Acked Write 1 Done Sent Done false
2931 1727 Acked Write 1 Done Sent Done false
2932 1728 Acked Flush 0 Done Sent Done false
2933 1729 Acked Read 7 Done Sent Done false
2934 1730 Acked Write 7 Done Sent Done false
2935 1731 Acked Write 2 Done Sent Done false
2936 1732 Acked Read 9 Done Sent Done false
2937 1733 Acked Read 10 Done Sent Done false
2938 1734 Acked Write 2 Done Sent Done false
2939 1735 Acked Read 8 Done Sent Done false
2940 1736 Acked Read 7 Done Sent Done false
2941 1737 Acked Read 6 Done Sent Done false
2942 1738 Acked Read 3 Done Sent Done false
2943 1739 Acked Read 8 Done Sent Done false
2944 1740 Acked Write 6 Done Sent Done false
2945 1741 Acked Read 7 Done Sent Done false
2946 1742 Acked Read 2 Done Sent Done false
2947 1743 Acked Write 6 Done Sent Done false
2948 1744 Acked Write 5 Done Sent Done false
2949 1745 Acked Read 5 Done Sent Done false
2950 1746 Acked Read 9 Done Sent Done false
2951 1747 Acked Write 2 Done Sent Done false
2952 1748 Acked Write 5 Done Sent Done false
2953 1749 Acked Flush 0 Done Sent Done false
2954 1750 Acked Read 8 Done Sent Done false
2955 1751 Acked Write 8 Done Sent Done false
2956 1752 Acked Read 4 Done Sent Done false
2957 1753 Acked Read 3 Done Sent Done false
2958 1754 Acked Read 6 Done Sent Done false
2959 1755 Acked Read 7 Done Sent Done false
2960 1756 Acked Read 1 Done Sent Done false
2961 1757 Acked Write 8 Done Sent Done false
2962 1758 Acked Read 6 Done Sent Done false
2963 1759 Acked Write 6 Done Sent Done false
2964 1760 Acked Read 8 Done Sent Done false
2965 1761 Acked Read 2 Done Sent Done false
2966 1762 Acked Write 2 Done Sent Done false
2967 1763 Acked Write 7 Done Sent Done false
2968 1764 Acked Read 1 Done Sent Done false
2969 1765 Acked Write 4 Done Sent Done false
2970 1766 Acked Read 2 Done Sent Done false
2971 1767 Acked Write 2 Done Sent Done false
2972 1768 Acked Read 5 Done Sent Done false
2973 1769 Acked Read 3 Done Sent Done false
2974 1770 Acked Read 2 Done Sent Done false
2975 1771 Acked Read 3 Done Sent Done false
2976 1772 Acked Write 1 Done Sent Done false
2977 1773 Acked Write 4 Done Sent Done false
2978 1774 Acked Read 5 Done Sent Done false
2979 1775 Acked Read 7 Done Sent Done false
2980 1776 Acked Write 6 Done Sent Done false
2981 1777 Acked Write 8 Done Sent Done false
2982 1778 Acked Write 3 Done Sent Done false
2983 1779 Acked Read 9 Done Sent Done false
2984 1780 Acked Read 9 Done Sent Done false
2985 1781 Acked Write 10 Done Sent Done false
2986 1782 Acked Flush 0 Done Sent Done false
2987 1783 Acked Write 9 Done Sent Done false
2988 1784 Acked Flush 0 Done Sent Done false
2989 1785 Acked Read 2 Done Sent Done false
2990 1786 Acked Read 9 Done Sent Done false
2991 1787 Acked Read 1 Done Sent Done false
2992 1788 Acked Read 3 Done Sent Done false
2993 1789 Acked Read 3 Done Sent Done false
2994 1790 Acked Write 9 Done Sent Done false
2995 1791 Acked Read 6 Done Sent Done false
2996 1792 Acked Read 8 Done Sent Done false
2997 1793 Acked Read 5 Done Sent Done false
2998 1794 Acked Write 2 Done Sent Done false
2999 1795 Acked Write 3 Done Sent Done false
3000 1796 Acked Read 9 Done Sent Done false
3001 1797 Acked Write 7 Done Sent Done false
3002 1798 Acked Write 1 Done Sent Done false
3003 1799 Acked Read 10 Done Sent Done false
3004 1800 Acked Read 7 Done Sent Done false
3005 1801 Acked Write 8 Done Sent Done false
3006 1802 Acked Write 6 Done Sent Done false
3007 1803 Acked Read 6 Done Sent Done false
3008 1804 Acked Read 8 Done Sent Done false
3009 1805 Acked Flush 0 Done Sent Done false
3010 1806 Acked Read 7 Done Sent Done false
3011 1807 Acked Read 1 Done Sent Done false
3012 1808 Acked Write 10 Done Sent Done false
3013 1809 Acked Flush 0 Done Sent Done false
3014 1810 Acked Read 8 Done Sent Done false
3015 1811 Acked Read 6 Done Sent Done false
3016 1812 Acked Read 1 Done Sent Done false
3017 1813 Acked Read 5 Done Sent Done false
3018 1814 Acked Read 5 Done Sent Done false
3019 1815 Acked Write 8 Done Sent Done false
3020 1816 Acked Write 4 Done Sent Done false
3021 1817 Acked Read 8 Done Sent Done false
3022 1818 Acked Write 8 Done Sent Done false
3023 1819 Acked Write 3 Done Sent Done false
3024 1820 Acked Read 9 Done Sent Done false
3025 1821 Acked Write 5 Done Sent Done false
3026 1822 Acked Read 10 Done Sent Done false
3027 1823 Acked Read 7 Done Sent Done false
3028 1824 Acked Read 9 Done Sent Done false
3029 1825 Acked Read 3 Done Sent Done false
3030 1826 Acked Read 7 Done Sent Done false
3031 1827 Acked Read 3 Done Sent Done false
3032 1828 Acked Read 5 Done Sent Done false
3033 1829 Acked Read 5 Done Sent Done false
3034 1830 Acked Flush 0 Done Sent Done false
3035 1831 Acked Flush 0 Done Sent Done false
3036 1832 Acked Read 9 Done Sent Done false
3037 1833 Acked Write 9 Done Sent Done false
3038 1834 Acked Write 2 Done Sent Done false
3039 1835 Acked Write 3 Done Sent Done false
3040 1836 Acked Read 8 Done Sent Done false
3041 1837 Acked Read 1 Done Sent Done false
3042 1838 Acked Flush 0 Done Sent Done false
3043 1839 Acked Flush 0 Done Sent Done false
3044 1840 Acked Write 8 Done Sent Done false
3045 1841 Acked Read 9 Done Sent Done false
3046 1842 Acked Read 3 Done Sent Done false
3047 1843 Acked Flush 0 Done Sent Done false
3048 1844 Acked Write 4 Done Sent Done false
3049 1845 Acked Write 5 Done Sent Done false
3050 1846 Acked Write 6 Done Sent Done false
3051 1847 Acked Read 4 Done Sent Done false
3052 1848 Acked Flush 0 Done Sent Done false
3053 1849 Acked Read 8 Done Sent Done false
3054 1850 Acked Read 6 Done Sent Done false
3055 1851 Acked Write 8 Done Sent Done false
3056 1852 Acked Read 2 Done Sent Done false
3057 1853 Acked Write 2 Done Sent Done false
3058 1854 Acked Read 10 Done Sent Done false
3059 1855 Acked Read 9 Done Sent Done false
3060 1856 Acked Write 9 Done Sent Done false
3061 1857 Acked Write 7 Done Sent Done false
3062 1858 Acked Read 8 Done Sent Done false
3063 1859 Acked Read 7 Done Sent Done false
3064 1860 Acked Write 2 Done Sent Done false
3065 1861 Acked Read 9 Done Sent Done false
3066 1862 Acked Read 5 Done Sent Done false
3067 1863 Acked Write 6 Done Sent Done false
3068 1864 Acked Read 7 Done Sent Done false
3069 1865 Acked Read 10 Done Sent Done false
3070 1866 Acked Write 5 Done Sent Done false
3071 1867 Acked Write 3 Done Sent Done false
3072 1868 Acked Read 10 Done Sent Done false
3073 1869 Acked Write 10 Done Sent Done false
3074 1870 Acked Read 2 Done Sent Done false
3075 1871 Acked Write 7 Done Sent Done false
3076 1872 Acked Read 9 Done Sent Done false
3077 1873 Acked Write 3 Done Sent Done false
3078 1874 Acked Read 2 Done Sent Done false
3079 1875 Acked Read 6 Done Sent Done false
3080 1876 Acked Read 1 Done Sent Done false
3081 1877 Acked Read 4 Done Sent Done false
3082 1878 Acked Read 10 Done Sent Done false
3083 1879 Acked Write 1 Done Sent Done false
3084 1880 Acked Read 9 Done Sent Done false
3085 1881 Acked Flush 0 Done Sent Done false
3086 1882 Acked Write 3 Done Sent Done false
3087 1883 Acked Write 4 Done Sent Done false
3088 1884 Acked Read 8 Done Sent Done false
3089 1885 Acked Read 4 Done Sent Done false
3090 1886 Acked Flush 0 Done Sent Done false
3091 1887 Acked Write 1 Done Sent Done false
3092 1888 Acked Flush 0 Done Sent Done false
3093 1889 Acked Read 9 Done Sent Done false
3094 1890 Acked Write 5 Done Sent Done false
3095 1891 Acked Read 5 Done Sent Done false
3096 1892 Acked Write 10 Done Sent Done false
3097 1893 Acked Read 10 Done Sent Done false
3098 1894 Acked Write 3 Done Sent Done false
3099 1895 Acked Write 3 Done Sent Done false
3100 1896 Acked Read 5 Done Sent Done false
3101 1897 Acked Read 2 Done Sent Done false
3102 1898 Acked Read 8 Done Sent Done false
3103 1899 Acked Read 5 Done Sent Done false
3104 1900 Acked Flush 0 Done Sent Done false
3105 1901 Acked Write 4 Done Sent Done false
3106 1902 Acked Read 5 Done Sent Done false
3107 1903 Acked Read 3 Done Sent Done false
3108 1904 Acked Write 2 Done Sent Done false
3109 STATES DS:0 DS:1 DS:2 TOTAL
3110 Sent 0 301 0 301
3111 Done 301 0 301 602
3112 Skipped 0 0 0 0
3113 Error 0 0 0 0
3114 Last Flush: Some(JobId(1900)) Some(JobId(1603)) Some(JobId(1900))
3115 Downstairs last five completed: 1603 1602 1601 1600 1599
3116 Upstairs last five completed: 1904 1903 1902 1901 1900
3117 CLIENT: Up:0 ds:301 act:2
31182025-02-21T06:45:56.710ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31192025-02-21T06:45:57.708ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31202025-02-21T06:45:57.708ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:32897) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31212025-02-21T06:45:57.709ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31222025-02-21T06:45:57.710ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31232025-02-21T06:45:57.710ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31242025-02-21T06:45:57.710ZINFOcrucible: send last flush ID to this DS: Some(JobId(1603)) = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31252025-02-21T06:45:57.710ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31262025-02-21T06:45:57.710ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(1603)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31272025-02-21T06:45:57.710ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(1603)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
3128 ----------------------------------------------------------------
3129 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3130 Downstairs last five completed: 1905 1904 1903 1902 1901
3131 Upstairs last five completed: 1905 1904 1903 1902 1901
3132 CLIENT: Up:0 ds:0 act:3
3133 Replay: All jobs finished
3134 Main task has work to do, go find it
3135 got dsc Stop(0)
3136 stop 0
3137 Feb 21 06:45:59.960 INFO request completed[8810] Got stop action so:false kr:false
3138 , latency_us: 346, response_code: 204, uri: /stop/cid/0, method: POST, req_id: f06bb134-892d-4956-8ffd-2cde4f8f2f19, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
3139 Feb 21 06:45:59.961 INFO request completed, latency_us: 155, response_code: 200, uri: /state/cid/0, method: GET, req_id{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:45:59.961507135Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
3140 : 01fedc94-8ead-4c29-abb7-cf4dfa5cde30, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
31412025-02-21T06:45:59.961ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31422025-02-21T06:45:59.961ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31432025-02-21T06:45:59.961ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
3144 [8810] Exited with: ExitStatus(unix_wait_status(9))
3145 [8810][0] reports Exit
3146 Feb 21 06:46:03.964 INFO request completed, latency_us: 241, response_code: 200, uri: /state/cid/0, method: GET, req_id: a0db736e-46da-45e3-99d0-62953452cce7, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
3147 001/300 Read block 2173 len 24576
3148 002/300 Read block 656 len 36864
3149 003/300 Read block 2699 len 24576
3150 004/300 Read block 1270 len 20480
3151 005/300 Read block 4406 len 36864
3152 006/300 Read block 1377 len 32768
3153 007/300 Read block 3423 len 12288
3154 008/300 Read block 3438 len 8192
3155 009/300 Read block 1465 len 40960
3156 010/300 Flush
3157 011/300 Write block 616 len 4096 data: 2
3158 012/300 Flush
3159 013/300 Read block 2355 len 28672
3160 014/300 Read block 1876 len 4096
3161 015/300 Read block 4274 len 16384
3162 016/300 Write block 2160 len 16384 data: 2 2 2 3
3163 017/300 Read block 2323 len 32768
3164 018/300 Read block 464 len 20480
3165 019/300 Read block 417 len 36864
3166 020/300 Read block 1907 len 32768
3167 021/300 Read block 3507 len 16384
3168 022/300 Write block 764 len 32768 data: 2 2 2 2 2 2 2 2
3169 023/300 Write block 3858 len 24576 data: 2 2 2 2 2 2
3170 024/300 Read block 1296 len 20480
3171 025/300 Read block 3127 len 36864
3172 026/300 Write block 500 len 32768 data: 3 3 3 3 2 2 2 2
3173 027/300 Read block 1880 len 4096
3174 028/300 Read block 197 len 20480
3175 029/300 Write block 4122 len 4096 data: 2
3176 030/300 Write block 4001 len 32768 data: 2 2 2 2 2 3 3 3
3177 031/300 Write block 3604 len 20480 data: 2 2 2 3 3
3178 032/300 Read block 4061 len 24576
3179 033/300 Flush
3180 034/300 Write block 1398 len 8192 data: 3 3
3181 035/300 Write block 902 len 16384 data: 3 3 4 3
3182 036/300 Read block 3084 len 24576
3183 037/300 Write block 2879 len 12288 data: 2 2 2
3184 038/300 Write block 4298 len 8192 data: 3 3
3185 039/300 Read block 829 len 12288
3186 040/300 Read block 1478 len 4096
3187 041/300 Read block 1787 len 16384
3188 042/300 Write block 3400 len 12288 data: 2 2 2
3189 043/300 Read block 3302 len 24576
3190 044/300 Write block 1984 len 40960 data: 2 3 4 4 4 3 2 2 2 3
3191 045/300 Read block 3763 len 32768
3192 046/300 Write block 1258 len 32768 data: 3 3 3 3 3 3 3 3
3193 047/300 Read block 559 len 32768
3194 048/300 Write block 3885 len 20480 data: 2 2 2 2 2
3195 049/300 Write block 545 len 36864 data: 2 2 2 2 2 2 2 2 2
3196 050/300 Read block 3497 len 36864
3197 051/300 Write block 1721 len 16384 data: 2 2 2 2
3198 052/300 Read block 561 len 16384
3199 053/300 Flush
3200 054/300 Write block 2066 len 28672 data: 2 2 2 2 2 2 2
3201 055/300 Read block 3233 len 20480
3202 056/300 Write block 3490 len 32768 data: 2 2 2 2 2 2 4 4
3203 057/300 Read block 529 len 4096
3204 058/300 Read block 2174 len 28672
3205 059/300 Flush
3206 060/300 Flush
3207 061/300 Read block 4412 len 12288
3208 062/300 Read block 3601 len 8192
3209 063/300 Read block 3151 len 4096
3210 064/300 Write block 3173 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3211 065/300 Write block 1846 len 32768 data: 4 4 3 3 2 2 2 2
3212 066/300 Read block 323 len 20480
3213 067/300 Read block 3563 len 16384
3214 068/300 Read block 44 len 40960
3215 069/300 Read block 3125 len 32768
3216 070/300 Read block 3232 len 8192
3217 071/300 Read block 1020 len 28672
3218 072/300 Read block 3833 len 12288
3219 073/300 Flush
3220 074/300 Read block 936 len 40960
3221 075/300 Write block 4212 len 12288 data: 4 3 2
3222 076/300 Write block 164 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3223 077/300 Read block 4287 len 12288
3224 078/300 Write block 424 len 40960 data: 2 3 3 3 3 3 3 3 3 2
3225 079/300 Write block 1663 len 28672 data: 3 3 2 2 2 2 2
3226 080/300 Read block 1619 len 12288
3227 081/300 Flush
3228 082/300 Write block 2515 len 16384 data: 2 2 2 2
3229 083/300 Read block 3252 len 40960
3230 084/300 Write block 2686 len 4096 data: 3
3231 085/300 Read block 3114 len 40960
3232 086/300 Write block 501 len 24576 data: 4 4 4 3 3 3
3233 087/300 Read block 368 len 40960
3234 088/300 Flush
3235 089/300 Read block 3928 len 40960
3236 090/300 Read block 3300 len 28672
3237 091/300 Write block 1758 len 24576 data: 2 2 2 2 2 2
3238 092/300 Write block 772 len 24576 data: 2 2 2 2 3 3
3239 093/300 Read block 1704 len 20480
3240 094/300 Read block 1897 len 8192
3241 095/300 Flush
3242 096/300 Write block 3816 len 32768 data: 2 2 2 2 2 3 3 3
3243 097/300 Flush
3244 098/300 Write block 1829 len 36864 data: 2 2 2 2 2 2 2 2 2
3245 099/300 Read block 619 len 32768
3246 100/300 Read block 38 len 40960
3247 101/300 Read block 1920 len 4096
3248 102/300 Write block 3099 len 20480 data: 2 2 2 2 2
3249 103/300 Flush
3250 104/300 Write block 4551 len 24576 data: 3 3 2 2 2 2
3251 105/300 Write block 1174 len 20480 data: 2 2 3 3 3
3252 106/300 Read block 3486 len 32768
3253 107/300 Read block 3674 len 24576
3254 108/300 Write block 2728 len 12288 data: 2 2 2
3255 109/300 Write block 4917 len 4096 data: 4
3256 110/300 Write block 1608 len 16384 data: 2 3 2 3
3257 111/300 Read block 557 len 28672
3258 112/300 Read block 608 len 20480
3259 113/300 Read block 4858 len 12288
3260 114/300 Write block 4120 len 16384 data: 2 2 3 2
3261 115/300 Read block 3908 len 8192
3262 116/300 Write block 2225 len 24576 data: 2 4 4 5 4 4
3263 117/300 Read block 3049 len 24576
3264 118/300 Read block 2860 len 8192
3265 119/300 Read block 642 len 4096
3266 120/300 Read block 2861 len 24576
3267 121/300 Read block 4342 len 32768
3268 122/300 Read block 2826 len 16384
3269 123/300 Read block 3788 len 20480
3270 124/300 Write block 4915 len 24576 data: 3 4 5 4 4 3
3271 125/300 Flush
3272 126/300 Write block 4647 len 40960 data: 3 3 3 3 3 3 2 2 2 2
3273 127/300 Read block 1067 len 40960
3274 128/300 Write block 4391 len 8192 data: 2 2
3275 129/300 Read block 664 len 16384
3276 130/300 Write block 2274 len 16384 data: 3 3 3 4
3277 131/300 Read block 2594 len 12288
3278 132/300 Read block 2504 len 40960
3279 133/300 Write block 1016 len 8192 data: 2 2
3280 134/300 Flush
3281 135/300 Read block 963 len 40960
3282 136/300 Write block 1607 len 24576 data: 2 3 4 3 4 4
3283 137/300 Read block 3913 len 24576
3284 138/300 Read block 4221 len 24576
3285 139/300 Write block 2036 len 28672 data: 2 2 2 2 2 2 3
3286 140/300 Read block 1950 len 16384
3287 141/300 Write block 2044 len 20480 data: 3 3 2 2 2
3288 142/300 Read block 2338 len 36864
3289 143/300 Write block 2495 len 12288 data: 2 2 2
3290 144/300 Flush
3291 145/300 Write block 4790 len 24576 data: 3 2 2 2 2 2
3292 146/300 Write block 1125 len 32768 data: 3 3 3 3 3 3 3 3
3293 147/300 Write block 3940 len 24576 data: 2 2 2 2 2 2
3294 148/300 Flush
3295 149/300 Read block 2504 len 36864
3296 150/300 Write block 1712 len 16384 data: 2 2 2 2
3297 151/300 Flush
3298 152/300 Read block 629 len 4096
3299 153/300 Write block 4993 len 28672 data: 2 2 2 2 2 2 2
3300 154/300 Read block 4302 len 8192
3301 155/300 Write block 2857 len 4096 data: 2
3302 156/300 Read block 1 len 12288
3303 157/300 Read block 1850 len 24576
3304 158/300 Write block 1538 len 8192 data: 2 2
3305 159/300 Write block 3975 len 40960 data: 2 3 4 3 2 2 2 3 3 4
3306 160/300 Read block 2626 len 12288
3307 161/300 Flush
3308 162/300 Write block 4547 len 28672 data: 3 3 3 3 4 4 3
3309 163/300 Flush
3310 164/300 Write block 2517 len 20480 data: 3 3 2 2 2
3311 165/300 Write block 2575 len 24576 data: 4 4 3 3 3 3
3312 166/300 Write block 4244 len 4096 data: 2
3313 167/300 Read block 1913 len 16384
3314 168/300 Read block 4755 len 40960
3315 169/300 Write block 4660 len 40960 data: 2 2 3 3 3 3 3 3 3 3
3316 170/300 Write block 1058 len 4096 data: 3
3317 171/300 Read block 3939 len 28672
3318 172/300 Read block 4694 len 12288
3319 173/300 Read block 2998 len 16384
3320 174/300 Write block 3103 len 12288 data: 3 2 2
3321 175/300 Write block 2701 len 4096 data: 2
3322 176/300 Write block 4308 len 20480 data: 3 3 3 3 3
3323 177/300 Write block 3283 len 28672 data: 2 2 2 2 2 2 2
3324 178/300 Read block 4823 len 24576
3325 179/300 Write block 2668 len 40960 data: 3 3 3 3 3 3 3 2 2 2
3326 180/300 Write block 4378 len 20480 data: 2 2 2 2 3
3327 181/300 Flush
3328 182/300 Read block 4825 len 20480
3329 183/300 Flush
3330 184/300 Flush
3331 185/300 Read block 2789 len 32768
3332 186/300 Read block 1565 len 8192
3333 187/300 Read block 751 len 28672
3334 188/300 Read block 4915 len 12288
3335 189/300 Write block 4368 len 12288 data: 2 2 2
3336 190/300 Write block 1126 len 32768 data: 4 4 4 4 4 4 4 3
3337 191/300 Flush
3338 192/300 Read block 2742 len 16384
3339 193/300 Read block 4590 len 24576
3340 194/300 Write block 2469 len 32768 data: 2 2 2 2 3 3 3 3
3341 195/300 Write block 3164 len 24576 data: 2 2 2 2 2 2
3342 196/300 Write block 1745 len 32768 data: 2 2 2 3 3 2 2 2
3343 197/300 Write block 1728 len 24576 data: 2 2 2 2 2 2
3344 198/300 Write block 2857 len 32768 data: 3 2 2 2 2 2 2 2
3345 199/300 Write block 1123 len 32768 data: 2 2 4 5 5 5 5 5
3346 200/300 Read block 4980 len 16384
3347 201/300 Read block 1643 len 40960
3348 202/300 Read block 634 len 32768
3349 203/300 Read block 163 len 28672
3350 204/300 Read block 1788 len 36864
3351 205/300 Flush
3352 206/300 Write block 2671 len 8192 data: 4 4
3353 207/300 Read block 2927 len 40960
3354 208/300 Read block 974 len 24576
3355 209/300 Read block 3181 len 12288
3356 210/300 Read block 2983 len 8192
3357 211/300 Flush
3358 212/300 Read block 2693 len 12288
3359 213/300 Write block 329 len 40960 data: 2 2 2 2 2 2 3 3 3 3
3360 214/300 Read block 3915 len 8192
3361 215/300 Read block 59 len 8192
3362 216/300 Write block 0 len 20480 data: 2 2 2 2 2
3363 217/300 Read block 4113 len 24576
3364 218/300 Read block 1387 len 24576
3365 219/300 Flush
3366 220/300 Write block 4897 len 24576 data: 2 2 2 2 2 2
3367 221/300 Read block 1586 len 20480
3368 222/300 Read block 2088 len 8192
3369 223/300 Read block 638 len 28672
3370 224/300 Read block 154 len 8192
3371 225/300 Read block 1790 len 32768
3372 226/300 Write block 1944 len 36864 data: 2 2 2 2 2 2 2 2 2
3373 227/300 Read block 1848 len 40960
3374 228/300 Write block 4985 len 20480 data: 5 5 5 4 4
3375 229/300 Read block 2360 len 24576
3376 230/300 Read block 2832 len 32768
3377 231/300 Read block 2281 len 20480
3378 232/300 Write block 4742 len 36864 data: 3 3 2 2 2 2 2 2 2
3379 233/300 Read block 3555 len 36864
3380 234/300 Read block 3839 len 8192
3381 235/300 Read block 1921 len 32768
3382 236/300 Write block 2400 len 20480 data: 2 2 2 2 2
3383 237/300 Read block 1407 len 24576
3384 238/300 Read block 580 len 32768
3385 239/300 Read block 15 len 20480
3386 240/300 Read block 741 len 4096
3387 241/300 Read block 4514 len 36864
3388 242/300 Flush
3389 243/300 Write block 3523 len 12288 data: 2 2 2
3390 244/300 Write block 1433 len 20480 data: 2 3 4 4 4
3391 245/300 Read block 3317 len 4096
3392 246/300 Write block 4565 len 36864 data: 2 2 2 2 2 2 2 2 3
3393 247/300 Flush
3394 248/300 Write block 4620 len 4096 data: 2
3395 249/300 Read block 3900 len 4096
3396 250/300 Write block 4778 len 8192 data: 2 2
3397 251/300 Read block 4303 len 20480
3398 252/300 Flush
3399 253/300 Read block 3492 len 24576
3400 254/300 Read block 2165 len 8192
3401 255/300 Flush
3402 256/300 Read block 1779 len 20480
3403 257/300 Read block 1263 len 8192
3404 258/300 Write block 1335 len 8192 data: 2 2
3405 259/300 Flush
3406 260/300 Write block 1103 len 16384 data: 2 2 2 2
3407 261/300 Write block 1684 len 12288 data: 2 2 2
3408 262/300 Flush
3409 263/300 Write block 4905 len 8192 data: 2 2
3410 264/300 Read block 4729 len 8192
3411 265/300 Read block 2402 len 12288
3412 266/300 Write block 2618 len 16384 data: 2 2 2 2
3413 267/300 Write block 1443 len 36864 data: 2 2 2 2 2 2 2 2 2
3414 268/300 Write block 2843 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3415 269/300 Read block 3731 len 12288
3416 270/300 Read block 4177 len 12288
3417 271/300 Write block 1418 len 28672 data: 2 2 2 2 2 2 2
3418 272/300 Write block 4895 len 28672 data: 2 2 3 3 3 3 3
3419 273/300 Read block 731 len 8192
3420 274/300 Read block 2626 len 36864
3421 275/300 Read block 3450 len 36864
3422 276/300 Write block 3817 len 40960 data: 3 3 3 3 4 4 4 2 2 2
3423 277/300 Read block 3049 len 8192
3424 278/300 Read block 3927 len 20480
3425 279/300 Write block 1492 len 16384 data: 2 2 2 2
3426 280/300 Write block 2877 len 40960 data: 3 3 3 3 3 2 2 2 3 4
3427 281/300 Write block 1716 len 16384 data: 2 2 2 2
3428 282/300 Read block 51 len 32768
3429 283/300 Read block 4836 len 4096
3430 284/300 Read block 3005 len 36864
3431 285/300 Read block 2129 len 36864
3432 286/300 Read block 2057 len 12288
3433 287/300 Write block 1556 len 4096 data: 2
3434 288/300 Read block 1327 len 40960
3435 289/300 Read block 384 len 12288
3436 290/300 Write block 822 len 20480 data: 2 2 2 2 2
3437 291/300 Read block 1295 len 8192
3438 292/300 Read block 173 len 24576
3439 293/300 Read block 3183 len 20480
3440 294/300 Read block 2200 len 36864
3441 295/300 Read block 3184 len 8192
3442 296/300 Flush
3443 297/300 Read block 2309 len 40960
3444 298/300 Write block 2668 len 40960 data: 4 4 4 5 5 4 4 3 3 3
3445 299/300 Write block 3582 len 24576 data: 5 5 5 5 4 4
3446 300/300 Write block 3812 len 24576 data: 2 2 2 2 3 4
3447 301/300 Write block 2574 len 4096 data: 4
3448 Main task has work to do, go find it
3449 got dsc Start(0)
3450 start 0
3451 Feb 21 06:46:04.210 INFO request completed, latency_us: 234, response_code: 204, uri: /start/cid/0, method: POST, req_id: ddbe4318-f7f4-44e1-8207-4daef9de25d0, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
3452 Replay: started 0, returned:Ok(())
3453 ----------------------------------------------------------------
3454 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3455 States: Offline Active Active
3456 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3457 1906 Acked Read 6 Sent Done Done false
3458 1907 Acked Read 9 Sent Done Done false
3459 1908 Acked Read 6 Sent Done Done false
3460 1909 Acked Read 5 Sent Done Done false
3461 1910 Acked Read 9 Sent Done Done false
3462 1911 Acked Read 8 Sent Done Done false
3463 1912 Acked Read 3 Sent Done Done false
3464 1913 Acked Read 2 Sent Done Done false
3465 1914 Acked Read 10 Sent Done Done false
3466 1915 Acked Flush 0 Sent Done Done false
3467 1916 Acked Write 1 Sent Done Done false
3468 1917 Acked Flush 0 Sent Done Done false
3469 1918 Acked Read 7 Sent Done Done false
3470 1919 Acked Read 1 Sent Done Done false
3471 1920 Acked Read 4 Sent Done Done false
3472 1921 Acked Write 4 Sent Done Done false
3473 1922 Acked Read 8 Sent Done Done false
3474 1923 Acked Read 5 Sent Done Done false
3475 1924 Acked Read 9 Sent Done Done false
3476 1925 Acked Read 8 Sent Done Done false
3477 1926 Acked Read 4 Sent Done Done false
3478 1927 Acked Write 8 Sent Done Done false
3479 1928 Acked Write 6 Sent Done Done false
3480 1929 Acked Read 5 Sent Done Done false
3481 1930 Acked Read 9 Sent Done Done false
3482 1931 Acked Write 8 Sent Done Done false
3483 1932 Acked Read 1 Sent Done Done false
3484 1933 Acked Read 5 Sent Done Done false
3485 1934 Acked Write 1 Sent Done Done false
3486 1935 Acked Write 8 Sent Done Done false
3487 1936 Acked Write 5 Sent Done Done false
3488 1937 Acked Read 6 Sent Done Done false
3489 1938 Acked Flush 0 Sent Done Done false
3490 1939 Acked Write 2 Sent Done Done false
3491 1940 Acked Write 4 Sent Done Done false
3492 1941 Acked Read 6 Sent Done Done false
3493 1942 Acked Write 3 Sent Done Done false
3494 1943 Acked Write 2 Sent Done Done false
3495 1944 Acked Read 3 Sent Done Done false
3496 1945 Acked Read 1 Sent Done Done false
3497 1946 Acked Read 4 Sent Done Done false
3498 1947 Acked Write 3 Sent Done Done false
3499 1948 Acked Read 6 Sent Done Done false
3500 1949 Acked Write 10 Sent Done Done false
3501 1950 Acked Read 8 Sent Done Done false
3502 1951 Acked Write 8 Sent Done Done false
3503 1952 Acked Read 8 Sent Done Done false
3504 1953 Acked Write 5 Sent Done Done false
3505 1954 Acked Write 9 Sent Done Done false
3506 1955 Acked Read 9 Sent Done Done false
3507 1956 Acked Write 4 Sent Done Done false
3508 1957 Acked Read 4 Sent Done Done false
3509 1958 Acked Flush 0 Sent Done Done false
3510 1959 Acked Write 7 Sent Done Done false
3511 1960 Acked Read 5 Sent Done Done false
3512 1961 Acked Write 8 Sent Done Done false
3513 1962 Acked Read 1 Sent Done Done false
3514 1963 Acked Read 7 Sent Done Done false
3515 1964 Acked Flush 0 Sent Done Done false
3516 1965 Acked Flush 0 Sent Done Done false
3517 1966 Acked Read 3 Sent Done Done false
3518 1967 Acked Read 2 Sent Done Done false
3519 1968 Acked Read 1 Sent Done Done false
3520 1969 Acked Write 10 Sent Done Done false
3521 1970 Acked Write 8 Sent Done Done false
3522 1971 Acked Read 5 Sent Done Done false
3523 1972 Acked Read 4 Sent Done Done false
3524 1973 Acked Read 10 Sent Done Done false
3525 1974 Acked Read 8 Sent Done Done false
3526 1975 Acked Read 2 Sent Done Done false
3527 1976 Acked Read 7 Sent Done Done false
3528 1977 Acked Read 3 Sent Done Done false
3529 1978 Acked Flush 0 Sent Done Done false
3530 1979 Acked Read 10 Sent Done Done false
3531 1980 Acked Write 3 Sent Done Done false
3532 1981 Acked Write 10 Sent Done Done false
3533 1982 Acked Read 3 Sent Done Done false
3534 1983 Acked Write 10 Sent Done Done false
3535 1984 Acked Write 7 Sent Done Done false
3536 1985 Acked Read 3 Sent Done Done false
3537 1986 Acked Flush 0 Sent Done Done false
3538 1987 Acked Write 4 Sent Done Done false
3539 1988 Acked Read 10 Sent Done Done false
3540 1989 Acked Write 1 Sent Done Done false
3541 1990 Acked Read 10 Sent Done Done false
3542 1991 Acked Write 6 Sent Done Done false
3543 1992 Acked Read 10 Sent Done Done false
3544 1993 Acked Flush 0 Sent Done Done false
3545 1994 Acked Read 10 Sent Done Done false
3546 1995 Acked Read 7 Sent Done Done false
3547 1996 Acked Write 6 Sent Done Done false
3548 1997 Acked Write 6 Sent Done Done false
3549 1998 Acked Read 5 Sent Done Done false
3550 1999 Acked Read 2 Sent Done Done false
3551 2000 Acked Flush 0 Sent Done Done false
3552 2001 Acked Write 8 Sent Done Done false
3553 2002 Acked Flush 0 Sent Done Done false
3554 2003 Acked Write 9 Sent Done Done false
3555 2004 Acked Read 8 Sent Done Done false
3556 2005 Acked Read 10 Sent Done Done false
3557 2006 Acked Read 1 Sent Done Done false
3558 2007 Acked Write 5 Sent Done Done false
3559 2008 Acked Flush 0 Sent Done Done false
3560 2009 Acked Write 6 Sent Done Done false
3561 2010 Acked Write 5 Sent Done Done false
3562 2011 Acked Read 8 Sent Done Done false
3563 2012 Acked Read 6 Sent Done Done false
3564 2013 Acked Write 3 Sent Done Done false
3565 2014 Acked Write 1 Sent Done Done false
3566 2015 Acked Write 4 Sent Done Done false
3567 2016 Acked Read 7 Sent Done Done false
3568 2017 Acked Read 5 Sent Done Done false
3569 2018 Acked Read 3 Sent Done Done false
3570 2019 Acked Write 4 Sent Done Done false
3571 2020 Acked Read 2 Sent Done Done false
3572 2021 Acked Write 6 Sent Done Done false
3573 2022 Acked Read 6 Sent Done Done false
3574 2023 Acked Read 2 Sent Done Done false
3575 2024 Acked Read 1 Sent Done Done false
3576 2025 Acked Read 6 Sent Done Done false
3577 2026 Acked Read 8 Sent Done Done false
3578 2027 Acked Read 4 Sent Done Done false
3579 2028 Acked Read 5 Sent Done Done false
3580 2029 Acked Write 6 Sent Done Done false
3581 2030 Acked Flush 0 Sent Done Done false
3582 2031 Acked Write 10 Sent Done Done false
3583 2032 Acked Read 10 Sent Done Done false
3584 2033 Acked Write 2 Sent Done Done false
3585 2034 Acked Read 4 Sent Done Done false
3586 2035 Acked Write 4 Sent Done Done false
3587 2036 Acked Read 3 Sent Done Done false
3588 2037 Acked Read 10 Sent Done Done false
3589 2038 Acked Write 2 Sent Done Done false
3590 2039 Acked Flush 0 Sent Done Done false
3591 2040 Acked Read 10 Sent Done Done false
3592 2041 Acked Write 6 Sent Done Done false
3593 2042 Acked Read 6 Sent Done Done false
3594 2043 Acked Read 6 Sent Done Done false
3595 2044 Acked Write 7 Sent Done Done false
3596 2045 Acked Read 4 Sent Done Done false
3597 2046 Acked Write 5 Sent Done Done false
3598 2047 Acked Read 9 Sent Done Done false
3599 2048 Acked Write 3 Sent Done Done false
3600 2049 Acked Flush 0 Sent Done Done false
3601 2050 Acked Write 6 Sent Done Done false
3602 2051 Acked Write 8 Sent Done Done false
3603 2052 Acked Write 6 Sent Done Done false
3604 2053 Acked Flush 0 Sent Done Done false
3605 2054 Acked Read 9 Sent Done Done false
3606 2055 Acked Write 4 Sent Done Done false
3607 2056 Acked Flush 0 Sent Done Done false
3608 2057 Acked Read 1 Sent Done Done false
3609 2058 Acked Write 7 Sent Done Done false
3610 2059 Acked Read 2 Sent Done Done false
3611 2060 Acked Write 1 Sent Done Done false
3612 2061 Acked Read 3 Sent Done Done false
3613 2062 Acked Read 6 Sent Done Done false
3614 2063 Acked Write 2 Sent Done Done false
3615 2064 Acked Write 10 Sent Done Done false
3616 2065 Acked Read 3 Sent Done Done false
3617 2066 Acked Flush 0 Sent Done Done false
3618 2067 Acked Write 7 Sent Done Done false
3619 2068 Acked Flush 0 Sent Done Done false
3620 2069 Acked Write 5 Sent Done Done false
3621 2070 Acked Write 6 Sent Done Done false
3622 2071 Acked Write 1 Sent Done Done false
3623 2072 Acked Read 4 Sent Done Done false
3624 2073 Acked Read 10 Sent Done Done false
3625 2074 Acked Write 10 Sent Done Done false
3626 2075 Acked Write 1 Sent Done Done false
3627 2076 Acked Read 7 Sent Done Done false
3628 2077 Acked Read 3 Sent Done Done false
3629 2078 Acked Read 4 Sent Done Done false
3630 2079 Acked Write 3 Sent Done Done false
3631 2080 Acked Write 1 Sent Done Done false
3632 2081 Acked Write 5 Sent Done Done false
3633 2082 Acked Write 7 Sent Done Done false
3634 2083 Acked Read 6 Sent Done Done false
3635 2084 Acked Write 10 Sent Done Done false
3636 2085 Acked Write 5 Sent Done Done false
3637 2086 Acked Flush 0 Sent Done Done false
3638 2087 Acked Read 5 Sent Done Done false
3639 2088 Acked Flush 0 Sent Done Done false
3640 2089 Acked Flush 0 Sent Done Done false
3641 2090 Acked Read 8 Sent Done Done false
3642 2091 Acked Read 2 Sent Done Done false
3643 2092 Acked Read 7 Sent Done Done false
3644 2093 Acked Read 3 Sent Done Done false
3645 2094 Acked Write 3 Sent Done Done false
3646 2095 Acked Write 8 Sent Done Done false
3647 2096 Acked Flush 0 Sent Done Done false
3648 2097 Acked Read 4 Sent Done Done false
3649 2098 Acked Read 6 Sent Done Done false
3650 2099 Acked Write 8 Sent Done Done false
3651 2100 Acked Write 6 Sent Done Done false
3652 2101 Acked Write 8 Sent Done Done false
3653 2102 Acked Write 6 Sent Done Done false
3654 2103 Acked Write 8 Sent Done Done false
3655 2104 Acked Write 8 Sent Done Done false
3656 2105 Acked Read 4 Sent Done Done false
3657 2106 Acked Read 10 Sent Done Done false
3658 2107 Acked Read 8 Sent Done Done false
3659 2108 Acked Read 7 Sent Done Done false
3660 2109 Acked Read 9 Sent Done Done false
3661 2110 Acked Flush 0 Sent Done Done false
3662 2111 Acked Write 2 Sent Done Done false
3663 2112 Acked Read 10 Sent Done Done false
3664 2113 Acked Read 6 Sent Done Done false
3665 2114 Acked Read 3 Sent Done Done false
3666 2115 Acked Read 2 Sent Done Done false
3667 2116 Acked Flush 0 Sent Done Done false
3668 2117 Acked Read 3 Sent Done Done false
3669 2118 Acked Write 10 Sent Done Done false
3670 2119 Acked Read 2 Sent Done Done false
3671 2120 Acked Read 2 Sent Done Done false
3672 2121 Acked Write 5 Sent Done Done false
3673 2122 Acked Read 6 Sent Done Done false
3674 2123 Acked Read 6 Sent Done Done false
3675 2124 Acked Flush 0 Sent Done Done false
3676 2125 Acked Write 6 Sent Done Done false
3677 2126 Acked Read 5 Sent Done Done false
3678 2127 Acked Read 2 Sent Done Done false
3679 2128 Acked Read 7 Sent Done Done false
3680 2129 Acked Read 2 Sent Done Done false
3681 2130 Acked Read 8 Sent Done Done false
3682 2131 Acked Write 9 Sent Done Done false
3683 2132 Acked Read 10 Sent Done Done false
3684 2133 Acked Write 5 Sent Done Done false
3685 2134 Acked Read 6 Sent Done Done false
3686 2135 Acked Read 8 Sent Done Done false
3687 2136 Acked Read 5 Sent Done Done false
3688 2137 Acked Write 9 Sent Done Done false
3689 2138 Acked Read 9 Sent Done Done false
3690 2139 Acked Read 2 Sent Done Done false
3691 2140 Acked Read 8 Sent Done Done false
3692 2141 Acked Write 5 Sent Done Done false
3693 2142 Acked Read 6 Sent Done Done false
3694 2143 Acked Read 8 Sent Done Done false
3695 2144 Acked Read 5 Sent Done Done false
3696 2145 Acked Read 1 Sent Done Done false
3697 2146 Acked Read 9 Sent Done Done false
3698 2147 Acked Flush 0 Sent Done Done false
3699 2148 Acked Write 3 Sent Done Done false
3700 2149 Acked Write 5 Sent Done Done false
3701 2150 Acked Read 1 Sent Done Done false
3702 2151 Acked Write 9 Sent Done Done false
3703 2152 Acked Flush 0 Sent Done Done false
3704 2153 Acked Write 1 Sent Done Done false
3705 2154 Acked Read 1 Sent Done Done false
3706 2155 Acked Write 2 Sent Done Done false
3707 2156 Acked Read 5 Sent Done Done false
3708 2157 Acked Flush 0 Sent Done Done false
3709 2158 Acked Read 6 Sent Done Done false
3710 2159 Acked Read 2 Sent Done Done false
3711 2160 Acked Flush 0 Sent Done Done false
3712 2161 Acked Read 5 Sent Done Done false
3713 2162 Acked Read 2 Sent Done Done false
3714 2163 Acked Write 2 Sent Done Done false
3715 2164 Acked Flush 0 Sent Done Done false
3716 2165 Acked Write 4 Sent Done Done false
3717 2166 Acked Write 3 Sent Done Done false
3718 2167 Acked Flush 0 Sent Done Done false
3719 2168 Acked Write 2 Sent Done Done false
3720 2169 Acked Read 2 Sent Done Done false
3721 2170 Acked Read 3 Sent Done Done false
3722 2171 Acked Write 4 Sent Done Done false
3723 2172 Acked Write 9 Sent Done Done false
3724 2173 Acked Write 10 Sent Done Done false
3725 2174 Acked Read 3 Sent Done Done false
3726 2175 Acked Read 3 Sent Done Done false
3727 2176 Acked Write 7 Sent Done Done false
3728 2177 Acked Write 7 Sent Done Done false
3729 2178 Acked Read 2 Sent Done Done false
3730 2179 Acked Read 9 Sent Done Done false
3731 2180 Acked Read 9 Sent Done Done false
3732 2181 Acked Write 10 Sent Done Done false
3733 2182 Acked Read 2 Sent Done Done false
3734 2183 Acked Read 5 Sent Done Done false
3735 2184 Acked Write 4 Sent Done Done false
3736 2185 Acked Write 10 Sent Done Done false
3737 2186 Acked Write 4 Sent Done Done false
3738 2187 Acked Read 8 Sent Done Done false
3739 2188 Acked Read 1 Sent Done Done false
3740 2189 Acked Read 9 Sent Done Done false
3741 2190 Acked Read 9 Sent Done Done false
3742 2191 Acked Read 3 Sent Done Done false
3743 2192 Acked Write 1 Sent Done Done false
3744 2193 Acked Read 10 Sent Done Done false
3745 2194 Acked Read 3 Sent Done Done false
3746 2195 Acked Write 5 Sent Done Done false
3747 2196 Acked Read 2 Sent Done Done false
3748 2197 Acked Read 6 Sent Done Done false
3749 2198 Acked Read 5 Sent Done Done false
3750 2199 Acked Read 9 Sent Done Done false
3751 2200 Acked Read 2 Sent Done Done false
3752 2201 Acked Flush 0 Sent Done Done false
3753 2202 Acked Read 10 Sent Done Done false
3754 2203 Acked Write 10 Sent Done Done false
3755 2204 Acked Write 6 Sent Sent Sent false
3756 2205 Acked Write 6 Sent Sent Sent false
3757 2206 Acked Write 1 Sent Sent Sent false
3758 STATES DS:0 DS:1 DS:2 TOTAL
3759 Sent 301 3 3 307
3760 Done 0 298 298 596
3761 Skipped 0 0 0 0
3762 Error 0 0 0 0
3763 Last Flush: Some(JobId(1905)) Some(JobId(2201)) Some(JobId(2201))
3764 Downstairs last five completed: 1905 1904 1903 1902 1901
3765 Upstairs last five completed: 2206 2205 2204 2203 2202
3766 CLIENT: Up:0 ds:301 act:2
37672025-02-21T06:46:04.723ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
37682025-02-21T06:46:05.724ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
3769 [8810] got start action
3770 [8810] I am going to restart
3771 Starting downstairs at port 8810
3772 Make output file at "/tmp/dsc/downstairs-8810.txt"
3773 [8810][0] reports Starting
3774 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1210)
3775 [8810][0] reports Running
37762025-02-21T06:46:06.725ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
3777 ----------------------------------------------------------------
3778 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3779 States: Offline Active Active
3780 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3781 1906 Acked Read 6 Sent Done Done false
3782 1907 Acked Read 9 Sent Done Done false
3783 1908 Acked Read 6 Sent Done Done false
3784 1909 Acked Read 5 Sent Done Done false
3785 1910 Acked Read 9 Sent Done Done false
3786 1911 Acked Read 8 Sent Done Done false
3787 1912 Acked Read 3 Sent Done Done false
3788 1913 Acked Read 2 Sent Done Done false
3789 1914 Acked Read 10 Sent Done Done false
3790 1915 Acked Flush 0 Sent Done Done false
3791 1916 Acked Write 1 Sent Done Done false
3792 1917 Acked Flush 0 Sent Done Done false
3793 1918 Acked Read 7 Sent Done Done false
3794 1919 Acked Read 1 Sent Done Done false
3795 1920 Acked Read 4 Sent Done Done false
3796 1921 Acked Write 4 Sent Done Done false
3797 1922 Acked Read 8 Sent Done Done false
3798 1923 Acked Read 5 Sent Done Done false
3799 1924 Acked Read 9 Sent Done Done false
3800 1925 Acked Read 8 Sent Done Done false
3801 1926 Acked Read 4 Sent Done Done false
3802 1927 Acked Write 8 Sent Done Done false
3803 1928 Acked Write 6 Sent Done Done false
3804 1929 Acked Read 5 Sent Done Done false
3805 1930 Acked Read 9 Sent Done Done false
3806 1931 Acked Write 8 Sent Done Done false
3807 1932 Acked Read 1 Sent Done Done false
3808 1933 Acked Read 5 Sent Done Done false
3809 1934 Acked Write 1 Sent Done Done false
3810 1935 Acked Write 8 Sent Done Done false
3811 1936 Acked Write 5 Sent Done Done false
3812 1937 Acked Read 6 Sent Done Done false
3813 1938 Acked Flush 0 Sent Done Done false
3814 1939 Acked Write 2 Sent Done Done false
3815 1940 Acked Write 4 Sent Done Done false
3816 1941 Acked Read 6 Sent Done Done false
3817 1942 Acked Write 3 Sent Done Done false
3818 1943 Acked Write 2 Sent Done Done false
3819 1944 Acked Read 3 Sent Done Done false
3820 1945 Acked Read 1 Sent Done Done false
3821 1946 Acked Read 4 Sent Done Done false
3822 1947 Acked Write 3 Sent Done Done false
3823 1948 Acked Read 6 Sent Done Done false
3824 1949 Acked Write 10 Sent Done Done false
3825 1950 Acked Read 8 Sent Done Done false
3826 1951 Acked Write 8 Sent Done Done false
3827 1952 Acked Read 8 Sent Done Done false
3828 1953 Acked Write 5 Sent Done Done false
3829 1954 Acked Write 9 Sent Done Done false
3830 1955 Acked Read 9 Sent Done Done false
3831 1956 Acked Write 4 Sent Done Done false
3832 1957 Acked Read 4 Sent Done Done false
3833 1958 Acked Flush 0 Sent Done Done false
3834 1959 Acked Write 7 Sent Done Done false
3835 1960 Acked Read 5 Sent Done Done false
3836 1961 Acked Write 8 Sent Done Done false
3837 1962 Acked Read 1 Sent Done Done false
3838 1963 Acked Read 7 Sent Done Done false
3839 1964 Acked Flush 0 Sent Done Done false
3840 1965 Acked Flush 0 Sent Done Done false
3841 1966 Acked Read 3 Sent Done Done false
3842 1967 Acked Read 2 Sent Done Done false
3843 1968 Acked Read 1 Sent Done Done false
3844 1969 Acked Write 10 Sent Done Done false
3845 1970 Acked Write 8 Sent Done Done false
3846 1971 Acked Read 5 Sent Done Done false
3847 1972 Acked Read 4 Sent Done Done false
3848 1973 Acked Read 10 Sent Done Done false
3849 1974 Acked Read 8 Sent Done Done false
3850 1975 Acked Read 2 Sent Done Done false
3851 1976 Acked Read 7 Sent Done Done false
3852 1977 Acked Read 3 Sent Done Done false
3853 1978 Acked Flush 0 Sent Done Done false
3854 1979 Acked Read 10 Sent Done Done false
3855 1980 Acked Write 3 Sent Done Done false
3856 1981 Acked Write 10 Sent Done Done false
3857 1982 Acked Read 3 Sent Done Done false
3858 1983 Acked Write 10 Sent Done Done false
3859 1984 Acked Write 7 Sent Done Done false
3860 1985 Acked Read 3 Sent Done Done false
3861 1986 Acked Flush 0 Sent Done Done false
3862 1987 Acked Write 4 Sent Done Done false
3863 1988 Acked Read 10 Sent Done Done false
3864 1989 Acked Write 1 Sent Done Done false
3865 1990 Acked Read 10 Sent Done Done false
3866 1991 Acked Write 6 Sent Done Done false
3867 1992 Acked Read 10 Sent Done Done false
3868 1993 Acked Flush 0 Sent Done Done false
3869 1994 Acked Read 10 Sent Done Done false
3870 1995 Acked Read 7 Sent Done Done false
3871 1996 Acked Write 6 Sent Done Done false
3872 1997 Acked Write 6 Sent Done Done false
3873 1998 Acked Read 5 Sent Done Done false
3874 1999 Acked Read 2 Sent Done Done false
3875 2000 Acked Flush 0 Sent Done Done false
3876 2001 Acked Write 8 Sent Done Done false
3877 2002 Acked Flush 0 Sent Done Done false
3878 2003 Acked Write 9 Sent Done Done false
3879 2004 Acked Read 8 Sent Done Done false
3880 2005 Acked Read 10 Sent Done Done false
3881 2006 Acked Read 1 Sent Done Done false
3882 2007 Acked Write 5 Sent Done Done false
3883 2008 Acked Flush 0 Sent Done Done false
3884 2009 Acked Write 6 Sent Done Done false
3885 2010 Acked Write 5 Sent Done Done false
3886 2011 Acked Read 8 Sent Done Done false
3887 2012 Acked Read 6 Sent Done Done false
3888 2013 Acked Write 3 Sent Done Done false
3889 2014 Acked Write 1 Sent Done Done false
3890 2015 Acked Write 4 Sent Done Done false
3891 2016 Acked Read 7 Sent Done Done false
3892 2017 Acked Read 5 Sent Done Done false
3893 2018 Acked Read 3 Sent Done Done false
3894 2019 Acked Write 4 Sent Done Done false
3895 2020 Acked Read 2 Sent Done Done false
3896 2021 Acked Write 6 Sent Done Done false
3897 2022 Acked Read 6 Sent Done Done false
3898 2023 Acked Read 2 Sent Done Done false
3899 2024 Acked Read 1 Sent Done Done false
3900 2025 Acked Read 6 Sent Done Done false
3901 2026 Acked Read 8 Sent Done Done false
3902 2027 Acked Read 4 Sent Done Done false
3903 2028 Acked Read 5 Sent Done Done false
3904 2029 Acked Write 6 Sent Done Done false
3905 2030 Acked Flush 0 Sent Done Done false
3906 2031 Acked Write 10 Sent Done Done false
3907 2032 Acked Read 10 Sent Done Done false
3908 2033 Acked Write 2 Sent Done Done false
3909 2034 Acked Read 4 Sent Done Done false
3910 2035 Acked Write 4 Sent Done Done false
3911 2036 Acked Read 3 Sent Done Done false
3912 2037 Acked Read 10 Sent Done Done false
3913 2038 Acked Write 2 Sent Done Done false
3914 2039 Acked Flush 0 Sent Done Done false
3915 2040 Acked Read 10 Sent Done Done false
3916 2041 Acked Write 6 Sent Done Done false
3917 2042 Acked Read 6 Sent Done Done false
3918 2043 Acked Read 6 Sent Done Done false
3919 2044 Acked Write 7 Sent Done Done false
3920 2045 Acked Read 4 Sent Done Done false
3921 2046 Acked Write 5 Sent Done Done false
3922 2047 Acked Read 9 Sent Done Done false
3923 2048 Acked Write 3 Sent Done Done false
3924 2049 Acked Flush 0 Sent Done Done false
3925 2050 Acked Write 6 Sent Done Done false
3926 2051 Acked Write 8 Sent Done Done false
3927 2052 Acked Write 6 Sent Done Done false
3928 2053 Acked Flush 0 Sent Done Done false
3929 2054 Acked Read 9 Sent Done Done false
3930 2055 Acked Write 4 Sent Done Done false
3931 2056 Acked Flush 0 Sent Done Done false
3932 2057 Acked Read 1 Sent Done Done false
3933 2058 Acked Write 7 Sent Done Done false
3934 2059 Acked Read 2 Sent Done Done false
3935 2060 Acked Write 1 Sent Done Done false
3936 2061 Acked Read 3 Sent Done Done false
3937 2062 Acked Read 6 Sent Done Done false
3938 2063 Acked Write 2 Sent Done Done false
3939 2064 Acked Write 10 Sent Done Done false
3940 2065 Acked Read 3 Sent Done Done false
3941 2066 Acked Flush 0 Sent Done Done false
3942 2067 Acked Write 7 Sent Done Done false
3943 2068 Acked Flush 0 Sent Done Done false
3944 2069 Acked Write 5 Sent Done Done false
3945 2070 Acked Write 6 Sent Done Done false
3946 2071 Acked Write 1 Sent Done Done false
3947 2072 Acked Read 4 Sent Done Done false
3948 2073 Acked Read 10 Sent Done Done false
3949 2074 Acked Write 10 Sent Done Done false
3950 2075 Acked Write 1 Sent Done Done false
3951 2076 Acked Read 7 Sent Done Done false
3952 2077 Acked Read 3 Sent Done Done false
3953 2078 Acked Read 4 Sent Done Done false
3954 2079 Acked Write 3 Sent Done Done false
3955 2080 Acked Write 1 Sent Done Done false
3956 2081 Acked Write 5 Sent Done Done false
3957 2082 Acked Write 7 Sent Done Done false
3958 2083 Acked Read 6 Sent Done Done false
3959 2084 Acked Write 10 Sent Done Done false
3960 2085 Acked Write 5 Sent Done Done false
3961 2086 Acked Flush 0 Sent Done Done false
3962 2087 Acked Read 5 Sent Done Done false
3963 2088 Acked Flush 0 Sent Done Done false
3964 2089 Acked Flush 0 Sent Done Done false
3965 2090 Acked Read 8 Sent Done Done false
3966 2091 Acked Read 2 Sent Done Done false
3967 2092 Acked Read 7 Sent Done Done false
3968 2093 Acked Read 3 Sent Done Done false
3969 2094 Acked Write 3 Sent Done Done false
3970 2095 Acked Write 8 Sent Done Done false
3971 2096 Acked Flush 0 Sent Done Done false
3972 2097 Acked Read 4 Sent Done Done false
3973 2098 Acked Read 6 Sent Done Done false
3974 2099 Acked Write 8 Sent Done Done false
3975 2100 Acked Write 6 Sent Done Done false
3976 2101 Acked Write 8 Sent Done Done false
3977 2102 Acked Write 6 Sent Done Done false
3978 2103 Acked Write 8 Sent Done Done false
3979 2104 Acked Write 8 Sent Done Done false
3980 2105 Acked Read 4 Sent Done Done false
3981 2106 Acked Read 10 Sent Done Done false
3982 2107 Acked Read 8 Sent Done Done false
3983 2108 Acked Read 7 Sent Done Done false
3984 2109 Acked Read 9 Sent Done Done false
3985 2110 Acked Flush 0 Sent Done Done false
3986 2111 Acked Write 2 Sent Done Done false
3987 2112 Acked Read 10 Sent Done Done false
3988 2113 Acked Read 6 Sent Done Done false
3989 2114 Acked Read 3 Sent Done Done false
3990 2115 Acked Read 2 Sent Done Done false
3991 2116 Acked Flush 0 Sent Done Done false
3992 2117 Acked Read 3 Sent Done Done false
3993 2118 Acked Write 10 Sent Done Done false
3994 2119 Acked Read 2 Sent Done Done false
3995 2120 Acked Read 2 Sent Done Done false
3996 2121 Acked Write 5 Sent Done Done false
3997 2122 Acked Read 6 Sent Done Done false
3998 2123 Acked Read 6 Sent Done Done false
3999 2124 Acked Flush 0 Sent Done Done false
4000 2125 Acked Write 6 Sent Done Done false
4001 2126 Acked Read 5 Sent Done Done false
4002 2127 Acked Read 2 Sent Done Done false
4003 2128 Acked Read 7 Sent Done Done false
4004 2129 Acked Read 2 Sent Done Done false
4005 2130 Acked Read 8 Sent Done Done false
4006 2131 Acked Write 9 Sent Done Done false
4007 2132 Acked Read 10 Sent Done Done false
4008 2133 Acked Write 5 Sent Done Done false
4009 2134 Acked Read 6 Sent Done Done false
4010 2135 Acked Read 8 Sent Done Done false
4011 2136 Acked Read 5 Sent Done Done false
4012 2137 Acked Write 9 Sent Done Done false
4013 2138 Acked Read 9 Sent Done Done false
4014 2139 Acked Read 2 Sent Done Done false
4015 2140 Acked Read 8 Sent Done Done false
4016 2141 Acked Write 5 Sent Done Done false
4017 2142 Acked Read 6 Sent Done Done false
4018 2143 Acked Read 8 Sent Done Done false
4019 2144 Acked Read 5 Sent Done Done false
4020 2145 Acked Read 1 Sent Done Done false
4021 2146 Acked Read 9 Sent Done Done false
4022 2147 Acked Flush 0 Sent Done Done false
4023 2148 Acked Write 3 Sent Done Done false
4024 2149 Acked Write 5 Sent Done Done false
4025 2150 Acked Read 1 Sent Done Done false
4026 2151 Acked Write 9 Sent Done Done false
4027 2152 Acked Flush 0 Sent Done Done false
4028 2153 Acked Write 1 Sent Done Done false
4029 2154 Acked Read 1 Sent Done Done false
4030 2155 Acked Write 2 Sent Done Done false
4031 2156 Acked Read 5 Sent Done Done false
4032 2157 Acked Flush 0 Sent Done Done false
4033 2158 Acked Read 6 Sent Done Done false
4034 2159 Acked Read 2 Sent Done Done false
4035 2160 Acked Flush 0 Sent Done Done false
4036 2161 Acked Read 5 Sent Done Done false
4037 2162 Acked Read 2 Sent Done Done false
4038 2163 Acked Write 2 Sent Done Done false
4039 2164 Acked Flush 0 Sent Done Done false
4040 2165 Acked Write 4 Sent Done Done false
4041 2166 Acked Write 3 Sent Done Done false
4042 2167 Acked Flush 0 Sent Done Done false
4043 2168 Acked Write 2 Sent Done Done false
4044 2169 Acked Read 2 Sent Done Done false
4045 2170 Acked Read 3 Sent Done Done false
4046 2171 Acked Write 4 Sent Done Done false
4047 2172 Acked Write 9 Sent Done Done false
4048 2173 Acked Write 10 Sent Done Done false
4049 2174 Acked Read 3 Sent Done Done false
4050 2175 Acked Read 3 Sent Done Done false
4051 2176 Acked Write 7 Sent Done Done false
4052 2177 Acked Write 7 Sent Done Done false
4053 2178 Acked Read 2 Sent Done Done false
4054 2179 Acked Read 9 Sent Done Done false
4055 2180 Acked Read 9 Sent Done Done false
4056 2181 Acked Write 10 Sent Done Done false
4057 2182 Acked Read 2 Sent Done Done false
4058 2183 Acked Read 5 Sent Done Done false
4059 2184 Acked Write 4 Sent Done Done false
4060 2185 Acked Write 10 Sent Done Done false
4061 2186 Acked Write 4 Sent Done Done false
4062 2187 Acked Read 8 Sent Done Done false
4063 2188 Acked Read 1 Sent Done Done false
4064 2189 Acked Read 9 Sent Done Done false
4065 2190 Acked Read 9 Sent Done Done false
4066 2191 Acked Read 3 Sent Done Done false
4067 2192 Acked Write 1 Sent Done Done false
4068 2193 Acked Read 10 Sent Done Done false
4069 2194 Acked Read 3 Sent Done Done false
4070 2195 Acked Write 5 Sent Done Done false
4071 2196 Acked Read 2 Sent Done Done false
4072 2197 Acked Read 6 Sent Done Done false
4073 2198 Acked Read 5 Sent Done Done false
4074 2199 Acked Read 9 Sent Done Done false
4075 2200 Acked Read 2 Sent Done Done false
4076 2201 Acked Flush 0 Sent Done Done false
4077 2202 Acked Read 10 Sent Done Done false
4078 2203 Acked Write 10 Sent Done Done false
4079 2204 Acked Write 6 Sent Done Done false
4080 2205 Acked Write 6 Sent Done Done false
4081 2206 Acked Write 1 Sent Done Done false
4082 STATES DS:0 DS:1 DS:2 TOTAL
4083 Sent 301 0 0 301
4084 Done 0 301 301 602
4085 Skipped 0 0 0 0
4086 Error 0 0 0 0
4087 Last Flush: Some(JobId(1905)) Some(JobId(2201)) Some(JobId(2201))
4088 Downstairs last five completed: 1905 1904 1903 1902 1901
4089 Upstairs last five completed: 2206 2205 2204 2203 2202
4090 CLIENT: Up:0 ds:301 act:2
40912025-02-21T06:46:08.728ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
40922025-02-21T06:46:09.730ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
40932025-02-21T06:46:09.962ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
40942025-02-21T06:46:09.962ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41182) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
40952025-02-21T06:46:09.963ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
40962025-02-21T06:46:09.964ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
40972025-02-21T06:46:09.964ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
40982025-02-21T06:46:09.964ZINFOcrucible: send last flush ID to this DS: Some(JobId(1905)) = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
40992025-02-21T06:46:09.964ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
41002025-02-21T06:46:09.964ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(1905)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
41012025-02-21T06:46:09.964ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(1905)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
4102 ----------------------------------------------------------------
4103 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4104 Downstairs last five completed: 2207 2206 2205 2204 2203
4105 Upstairs last five completed: 2207 2206 2205 2204 2203
4106 CLIENT: Up:0 ds:0 act:3
4107 Replay: All jobs finished
4108 Main task has work to do, go find it
4109 got dsc Stop(2)
4110 stop 2
4111 Feb 21 06:46:12.215 INFO request completed, latency_us: 386[8830] Got stop action so:false kr:false
4112 , response_code: 204, uri: /stop/cid/2, method: POST, req_id: 30dc7f54-eb87-4cc5-bd42-c438838d8442, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
41132025-02-21T06:46:12.216ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
41142025-02-21T06:46:12.216ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
41152025-02-21T06:46:12.216ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
4116 Feb 21 06:46:12.216 INFO request completed, latency_us: 164, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7478791b-2a42-4a47-8cf6-803b88c220fb, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
41172025-02-21T06:46:12.216ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
4118 [8830] Exited with: ExitStatus(unix_wait_status(9))
4119 [8830][2] reports Exit
4120 Feb 21 06:46:16.218 INFO request completed, latency_us: 232, response_code: 200, uri: /state/cid/2, method: GET, req_id: b7415ec2-c976-4c2e-86ab-fceb6544b51e, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
4121 001/300 Read block 3326 len 36864
4122 002/300 Flush
4123 003/300 Read block 1622 len 12288
4124 004/300 Write block 2038 len 24576 data: 3 3 3 3 4 3
4125 005/300 Read block 4515 len 24576
4126 006/300 Write block 3624 len 20480 data: 2 2 2 2 2
4127 007/300 Flush
4128 008/300 Write block 447 len 4096 data: 2
4129 009/300 Write block 74 len 24576 data: 2 2 2 2 2 2
4130 010/300 Write block 612 len 4096 data: 2
4131 011/300 Read block 2120 len 16384
4132 012/300 Read block 2179 len 36864
4133 013/300 Read block 4010 len 12288
4134 014/300 Read block 4151 len 28672
4135 015/300 Read block 1449 len 24576
4136 016/300 Read block 3867 len 28672
4137 017/300 Flush
4138 018/300 Write block 2339 len 40960 data: 2 2 2 2 2 2 2 2 3 3
4139 019/300 Read block 3060 len 4096
4140 020/300 Write block 4748 len 8192 data: 3 3
4141 021/300 Write block 3873 len 28672 data: 3 2 2 2 2 2 2
4142 022/300 Flush
4143 023/300 Write block 1173 len 16384 data: 2 3 3 4
4144 024/300 Write block 4738 len 20480 data: 2 2 2 2 4
4145 025/300 Flush
4146 026/300 Read block 1472 len 16384
4147 027/300 Read block 1488 len 32768
4148 028/300 Read block 3837 len 40960
4149 029/300 Read block 4691 len 28672
4150 030/300 Flush
4151 031/300 Write block 259 len 16384 data: 2 2 2 2
4152 032/300 Read block 4302 len 4096
4153 033/300 Write block 4698 len 20480 data: 3 3 2 2 3
4154 034/300 Write block 3862 len 28672 data: 3 3 2 2 2 3 2
4155 035/300 Write block 3614 len 8192 data: 2 2
4156 036/300 Write block 2714 len 32768 data: 3 3 3 3 3 2 2 2
4157 037/300 Write block 4273 len 20480 data: 2 2 2 2 2
4158 038/300 Write block 1811 len 36864 data: 3 3 4 4 4 4 3 2 3
4159 039/300 Flush
4160 040/300 Flush
4161 041/300 Read block 4811 len 40960
4162 042/300 Read block 4966 len 36864
4163 043/300 Read block 4390 len 36864
4164 044/300 Write block 1585 len 8192 data: 2 2
4165 045/300 Read block 1598 len 36864
4166 046/300 Read block 168 len 12288
4167 047/300 Read block 3669 len 28672
4168 048/300 Read block 1761 len 4096
4169 049/300 Read block 1476 len 8192
4170 050/300 Write block 779 len 8192 data: 2 2
4171 051/300 Read block 3033 len 8192
4172 052/300 Read block 4179 len 24576
4173 053/300 Read block 449 len 32768
4174 054/300 Read block 1584 len 28672
4175 055/300 Write block 2049 len 20480 data: 2 2 2 2 2
4176 056/300 Write block 2776 len 32768 data: 2 2 2 2 2 2 3 3
4177 057/300 Read block 2415 len 20480
4178 058/300 Read block 3738 len 20480
4179 059/300 Write block 1054 len 32768 data: 4 3 3 3 4 3 4 4
4180 060/300 Write block 983 len 24576 data: 3 3 3 2 2 2
4181 061/300 Write block 693 len 36864 data: 3 3 2 2 3 2 2 2 2
4182 062/300 Flush
4183 063/300 Write block 4093 len 36864 data: 3 3 3 3 2 2 3 3 2
4184 064/300 Read block 4255 len 24576
4185 065/300 Read block 2169 len 8192
4186 066/300 Read block 4722 len 36864
4187 067/300 Write block 3143 len 24576 data: 3 3 3 3 3 2
4188 068/300 Write block 781 len 16384 data: 2 2 2 2
4189 069/300 Flush
4190 070/300 Read block 43 len 12288
4191 071/300 Write block 3147 len 32768 data: 4 3 2 2 2 2 2 2
4192 072/300 Write block 2191 len 12288 data: 3 3 3
4193 073/300 Write block 439 len 32768 data: 2 2 3 2 3 3 3 3
4194 074/300 Read block 3226 len 24576
4195 075/300 Read block 2260 len 12288
4196 076/300 Write block 1025 len 36864 data: 2 2 2 2 2 2 2 2 2
4197 077/300 Read block 4276 len 40960
4198 078/300 Read block 644 len 40960
4199 079/300 Read block 2266 len 40960
4200 080/300 Write block 2187 len 12288 data: 3 3 3
4201 081/300 Write block 1762 len 28672 data: 3 3 2 2 2 2 2
4202 082/300 Read block 1358 len 12288
4203 083/300 Read block 4603 len 24576
4204 084/300 Write block 3585 len 8192 data: 6 5
4205 085/300 Write block 3045 len 4096 data: 2
4206 086/300 Read block 4120 len 40960
4207 087/300 Write block 99 len 16384 data: 2 2 2 2
4208 088/300 Read block 3343 len 8192
4209 089/300 Read block 1108 len 28672
4210 090/300 Write block 4788 len 4096 data: 3
4211 091/300 Read block 1751 len 24576
4212 092/300 Read block 1899 len 8192
4213 093/300 Read block 2198 len 24576
4214 094/300 Write block 3226 len 32768 data: 2 2 2 2 2 2 2 2
4215 095/300 Write block 3607 len 16384 data: 4 4 3 3
4216 096/300 Write block 1314 len 8192 data: 2 2
4217 097/300 Read block 3099 len 12288
4218 098/300 Write block 1893 len 28672 data: 2 2 2 2 2 2 2
4219 099/300 Flush
4220 100/300 Read block 4136 len 4096
4221 101/300 Write block 1609 len 16384 data: 5 4 5 5
4222 102/300 Read block 3501 len 32768
4223 103/300 Flush
4224 104/300 Read block 1209 len 24576
4225 105/300 Read block 405 len 12288
4226 106/300 Read block 4115 len 16384
4227 107/300 Flush
4228 108/300 Write block 1815 len 36864 data: 5 5 4 3 4 3 3 3 2
4229 109/300 Read block 598 len 28672
4230 110/300 Read block 2555 len 32768
4231 111/300 Write block 3169 len 36864 data: 3 2 2 2 3 3 3 3 3
4232 112/300 Read block 1475 len 36864
4233 113/300 Write block 1920 len 4096 data: 2
4234 114/300 Write block 198 len 28672 data: 2 2 2 2 2 2 2
4235 115/300 Read block 1298 len 28672
4236 116/300 Write block 1785 len 32768 data: 3 3 2 3 3 3 3 3
4237 117/300 Write block 1025 len 40960 data: 3 3 3 3 3 3 3 3 3 2
4238 118/300 Write block 2050 len 32768 data: 3 3 3 3 2 2 2 2
4239 119/300 Write block 3932 len 32768 data: 3 3 2 2 2 2 2 2
4240 120/300 Write block 4492 len 4096 data: 2
4241 121/300 Read block 3039 len 36864
4242 122/300 Write block 2133 len 32768 data: 3 3 2 2 2 2 2 2
4243 123/300 Write block 3785 len 32768 data: 2 2 2 2 2 2 2 2
4244 124/300 Write block 890 len 20480 data: 2 2 2 2 3
4245 125/300 Read block 3870 len 40960
4246 126/300 Read block 1054 len 4096
4247 127/300 Write block 2423 len 40960 data: 4 4 3 3 3 3 3 3 3 3
4248 128/300 Flush
4249 129/300 Write block 3175 len 4096 data: 4
4250 130/300 Read block 3982 len 36864
4251 131/300 Read block 2929 len 16384
4252 132/300 Read block 4213 len 8192
4253 133/300 Read block 2574 len 36864
4254 134/300 Read block 1542 len 8192
4255 135/300 Write block 1481 len 36864 data: 2 3 3 3 3 3 3 3 3
4256 136/300 Write block 3641 len 36864 data: 2 2 2 2 3 3 3 3 3
4257 137/300 Write block 2865 len 32768 data: 2 2 2 2 2 3 3 3
4258 138/300 Write block 4830 len 40960 data: 3 3 3 2 2 2 2 2 2 2
4259 139/300 Read block 2479 len 16384
4260 140/300 Write block 3338 len 28672 data: 2 2 3 3 3 3 3
4261 141/300 Write block 3976 len 16384 data: 4 5 4 3
4262 142/300 Read block 3993 len 28672
4263 143/300 Read block 2955 len 36864
4264 144/300 Read block 3649 len 16384
4265 145/300 Read block 195 len 40960
4266 146/300 Write block 3830 len 28672 data: 2 2 2 2 2 2 2
4267 147/300 Write block 3631 len 28672 data: 3 3 3 3 2 2 2
4268 148/300 Read block 703 len 28672
4269 149/300 Read block 4683 len 8192
4270 150/300 Read block 1770 len 20480
4271 151/300 Read block 4074 len 28672
4272 152/300 Write block 1219 len 28672 data: 2 2 2 2 2 2 2
4273 153/300 Flush
4274 154/300 Flush
4275 155/300 Write block 1862 len 16384 data: 2 2 2 2
4276 156/300 Read block 4914 len 28672
4277 157/300 Read block 533 len 24576
4278 158/300 Read block 846 len 16384
4279 159/300 Write block 2865 len 40960 data: 3 3 3 3 3 4 4 4 3 3
4280 160/300 Write block 1423 len 20480 data: 3 3 2 2 2
4281 161/300 Write block 503 len 16384 data: 5 4 4 4
4282 162/300 Read block 2473 len 8192
4283 163/300 Flush
4284 164/300 Read block 4942 len 16384
4285 165/300 Read block 2374 len 16384
4286 166/300 Read block 4375 len 32768
4287 167/300 Write block 4647 len 16384 data: 4 4 4 4
4288 168/300 Read block 3730 len 8192
4289 169/300 Write block 4601 len 4096 data: 3
4290 170/300 Read block 3322 len 16384
4291 171/300 Flush
4292 172/300 Read block 4529 len 16384
4293 173/300 Read block 2629 len 16384
4294 174/300 Write block 579 len 16384 data: 2 2 2 2
4295 175/300 Write block 2011 len 20480 data: 2 2 2 2 2
4296 176/300 Read block 4244 len 16384
4297 177/300 Read block 2883 len 32768
4298 178/300 Read block 2929 len 12288
4299 179/300 Read block 2480 len 20480
4300 180/300 Write block 741 len 8192 data: 2 2
4301 181/300 Write block 25 len 4096 data: 2
4302 182/300 Write block 3678 len 12288 data: 2 2 2
4303 183/300 Flush
4304 184/300 Read block 855 len 32768
4305 185/300 Read block 4398 len 24576
4306 186/300 Read block 1612 len 28672
4307 187/300 Flush
4308 188/300 Write block 3160 len 8192 data: 2 2
4309 189/300 Write block 4878 len 24576 data: 3 4 4 3 3 3
4310 190/300 Read block 675 len 32768
4311 191/300 Write block 177 len 36864 data: 2 2 2 2 2 2 2 2 2
4312 192/300 Write block 3842 len 36864 data: 3 3 3 2 2 2 2 2 2
4313 193/300 Write block 3830 len 20480 data: 3 3 3 3 3
4314 194/300 Read block 4878 len 4096
4315 195/300 Write block 3015 len 12288 data: 2 2 2
4316 196/300 Read block 4911 len 24576
4317 197/300 Write block 3048 len 4096 data: 2
4318 198/300 Read block 508 len 16384
4319 199/300 Write block 4872 len 36864 data: 3 3 3 3 3 3 4 5 5
4320 200/300 Read block 2499 len 20480
4321 201/300 Flush
4322 202/300 Write block 2419 len 12288 data: 2 2 3
4323 203/300 Write block 4532 len 4096 data: 2
4324 204/300 Read block 1193 len 20480
4325 205/300 Read block 4030 len 12288
4326 206/300 Write block 2613 len 40960 data: 2 2 2 2 2 3 3 3 3 2
4327 207/300 Read block 3625 len 36864
4328 208/300 Write block 3413 len 36864 data: 4 4 4 4 4 2 2 2 2
4329 209/300 Write block 765 len 16384 data: 3 3 3 3
4330 210/300 Write block 2259 len 12288 data: 2 2 3
4331 211/300 Read block 2970 len 24576
4332 212/300 Flush
4333 213/300 Write block 554 len 16384 data: 2 2 2 2
4334 214/300 Flush
4335 215/300 Read block 3238 len 20480
4336 216/300 Read block 1966 len 12288
4337 217/300 Read block 66 len 20480
4338 218/300 Write block 2312 len 28672 data: 2 2 2 2 2 2 2
4339 219/300 Read block 4510 len 20480
4340 220/300 Write block 3430 len 28672 data: 3 4 4 3 2 2 2
4341 221/300 Write block 4988 len 12288 data: 5 5 3
4342 222/300 Write block 3449 len 36864 data: 2 2 2 2 2 2 2 2 2
4343 223/300 Read block 1209 len 20480
4344 224/300 Read block 2525 len 12288
4345 225/300 Write block 4710 len 40960 data: 3 3 3 3 3 2 2 2 2 2
4346 226/300 Read block 272 len 8192
4347 227/300 Flush
4348 228/300 Read block 27 len 20480
4349 229/300 Write block 3036 len 20480 data: 3 3 3 4 4
4350 230/300 Read block 1850 len 28672
4351 231/300 Write block 759 len 12288 data: 4 3 3
4352 232/300 Write block 4073 len 24576 data: 2 2 2 2 2 2
4353 233/300 Write block 1811 len 4096 data: 4
4354 234/300 Write block 607 len 16384 data: 2 2 2 2
4355 235/300 Read block 2993 len 4096
4356 236/300 Read block 4989 len 12288
4357 237/300 Read block 1356 len 4096
4358 238/300 Read block 1351 len 16384
4359 239/300 Read block 4291 len 28672
4360 240/300 Write block 2360 len 28672 data: 3 3 3 3 2 2 2
4361 241/300 Read block 4256 len 28672
4362 242/300 Write block 4072 len 32768 data: 2 3 3 3 3 3 3 2
4363 243/300 Read block 1689 len 16384
4364 244/300 Read block 2619 len 4096
4365 245/300 Read block 1904 len 16384
4366 246/300 Write block 4531 len 12288 data: 2 3 2
4367 247/300 Write block 3834 len 40960 data: 4 3 3 2 2 2 2 3 4 4
4368 248/300 Read block 3130 len 16384
4369 249/300 Read block 1212 len 20480
4370 250/300 Write block 3820 len 12288 data: 4 5 5
4371 251/300 Read block 2462 len 12288
4372 252/300 Read block 206 len 8192
4373 253/300 Read block 1774 len 28672
4374 254/300 Write block 1530 len 16384 data: 3 3 3 3
4375 255/300 Read block 1135 len 28672
4376 256/300 Read block 1287 len 12288
4377 257/300 Read block 3666 len 8192
4378 258/300 Read block 1502 len 16384
4379 259/300 Read block 4020 len 4096
4380 260/300 Read block 2438 len 4096
4381 261/300 Read block 749 len 4096
4382 262/300 Flush
4383 263/300 Read block 3916 len 24576
4384 264/300 Write block 4566 len 20480 data: 3 3 3 3 3
4385 265/300 Read block 4396 len 12288
4386 266/300 Write block 3002 len 36864 data: 2 2 2 2 2 2 2 2 3
4387 267/300 Write block 1026 len 20480 data: 4 4 4 4 4
4388 268/300 Read block 1065 len 28672
4389 269/300 Flush
4390 270/300 Flush
4391 271/300 Write block 3884 len 28672 data: 2 3 3 3 3 3 2
4392 272/300 Read block 2876 len 40960
4393 273/300 Write block 1122 len 12288 data: 2 3 3
4394 274/300 Read block 2898 len 8192
4395 275/300 Read block 4220 len 12288
4396 276/300 Read block 3504 len 24576
4397 277/300 Read block 3381 len 36864
4398 278/300 Read block 355 len 12288
4399 279/300 Read block 3826 len 8192
4400 280/300 Read block 1275 len 36864
4401 281/300 Read block 548 len 24576
4402 282/300 Read block 156 len 12288
4403 283/300 Write block 233 len 16384 data: 2 2 2 2
4404 284/300 Write block 2854 len 28672 data: 2 2 2 4 3 3 3
4405 285/300 Read block 2533 len 12288
4406 286/300 Write block 18 len 12288 data: 2 2 2
4407 287/300 Read block 3529 len 40960
4408 288/300 Read block 4117 len 24576
4409 289/300 Read block 918 len 12288
4410 290/300 Read block 2326 len 32768
4411 291/300 Flush
4412 292/300 Write block 683 len 4096 data: 2
4413 293/300 Write block 53 len 28672 data: 2 2 2 2 2 2 2
4414 294/300 Read block 243 len 28672
4415 295/300 Write block 2876 len 20480 data: 3 4 4 4 4
4416 296/300 Read block 3135 len 20480
4417 297/300 Read block 3804 len 12288
4418 298/300 Write block 1613 len 32768 data: 4 3 2 3 3 3 2 2
4419 299/300 Write block 3162 len 36864 data: 2 2 3 3 3 3 3 4 3
4420 300/300 Write block 4610 len 4096 data: 2
4421 301/300 Write block 229 len 4096 data: 2
4422 Main task has work to do, go find it
4423 got dsc Start(2)
4424 start 2
4425 Feb 21 06:46:16.457 INFO request completed, latency_us: 292, response_code: 204, uri: /start/cid/2, method: POST, req_id: e85650a9-3372-4743-b4bb-c9c06e44077c, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
4426 Replay: started 2, returned:Ok(())
4427 ----------------------------------------------------------------
4428 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4429 States: Active Active Offline
4430 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4431 2208 Acked Read 9 Done Done Sent false
4432 2209 Acked Flush 0 Done Done Sent false
4433 2210 Acked Read 3 Done Done Sent false
4434 2211 Acked Write 6 Done Done Sent false
4435 2212 Acked Read 6 Done Done Sent false
4436 2213 Acked Write 5 Done Done Sent false
4437 2214 Acked Flush 0 Done Done Sent false
4438 2215 Acked Write 1 Done Done Sent false
4439 2216 Acked Write 6 Done Done Sent false
4440 2217 Acked Write 1 Done Done Sent false
4441 2218 Acked Read 4 Done Done Sent false
4442 2219 Acked Read 9 Done Done Sent false
4443 2220 Acked Read 3 Done Done Sent false
4444 2221 Acked Read 7 Done Done Sent false
4445 2222 Acked Read 6 Done Done Sent false
4446 2223 Acked Read 7 Done Done Sent false
4447 2224 Acked Flush 0 Done Done Sent false
4448 2225 Acked Write 10 Done Done Sent false
4449 2226 Acked Read 1 Done Done Sent false
4450 2227 Acked Write 2 Done Done Sent false
4451 2228 Acked Write 7 Done Done Sent false
4452 2229 Acked Flush 0 Done Done Sent false
4453 2230 Acked Write 4 Done Done Sent false
4454 2231 Acked Write 5 Done Done Sent false
4455 2232 Acked Flush 0 Done Done Sent false
4456 2233 Acked Read 4 Done Done Sent false
4457 2234 Acked Read 8 Done Done Sent false
4458 2235 Acked Read 10 Done Done Sent false
4459 2236 Acked Read 7 Done Done Sent false
4460 2237 Acked Flush 0 Done Done Sent false
4461 2238 Acked Write 4 Done Done Sent false
4462 2239 Acked Read 1 Done Done Sent false
4463 2240 Acked Write 5 Done Done Sent false
4464 2241 Acked Write 7 Done Done Sent false
4465 2242 Acked Write 2 Done Done Sent false
4466 2243 Acked Write 8 Done Done Sent false
4467 2244 Acked Write 5 Done Done Sent false
4468 2245 Acked Write 9 Done Done Sent false
4469 2246 Acked Flush 0 Done Done Sent false
4470 2247 Acked Flush 0 Done Done Sent false
4471 2248 Acked Read 10 Done Done Sent false
4472 2249 Acked Read 9 Done Done Sent false
4473 2250 Acked Read 9 Done Done Sent false
4474 2251 Acked Write 2 Done Done Sent false
4475 2252 Acked Read 9 Done Done Sent false
4476 2253 Acked Read 3 Done Done Sent false
4477 2254 Acked Read 7 Done Done Sent false
4478 2255 Acked Read 1 Done Done Sent false
4479 2256 Acked Read 2 Done Done Sent false
4480 2257 Acked Write 2 Done Done Sent false
4481 2258 Acked Read 2 Done Done Sent false
4482 2259 Acked Read 6 Done Done Sent false
4483 2260 Acked Read 8 Done Done Sent false
4484 2261 Acked Read 7 Done Done Sent false
4485 2262 Acked Write 5 Done Done Sent false
4486 2263 Acked Write 8 Done Done Sent false
4487 2264 Acked Read 5 Done Done Sent false
4488 2265 Acked Read 5 Done Done Sent false
4489 2266 Acked Write 8 Done Done Sent false
4490 2267 Acked Write 6 Done Done Sent false
4491 2268 Acked Write 9 Done Done Sent false
4492 2269 Acked Flush 0 Done Done Sent false
4493 2270 Acked Write 9 Done Done Sent false
4494 2271 Acked Read 6 Done Done Sent false
4495 2272 Acked Read 2 Done Done Sent false
4496 2273 Acked Read 9 Done Done Sent false
4497 2274 Acked Write 6 Done Done Sent false
4498 2275 Acked Write 4 Done Done Sent false
4499 2276 Acked Flush 0 Done Done Sent false
4500 2277 Acked Read 3 Done Done Sent false
4501 2278 Acked Write 8 Done Done Sent false
4502 2279 Acked Write 3 Done Done Sent false
4503 2280 Acked Write 8 Done Done Sent false
4504 2281 Acked Read 6 Done Done Sent false
4505 2282 Acked Read 3 Done Done Sent false
4506 2283 Acked Write 9 Done Done Sent false
4507 2284 Acked Read 10 Done Done Sent false
4508 2285 Acked Read 10 Done Done Sent false
4509 2286 Acked Read 10 Done Done Sent false
4510 2287 Acked Write 3 Done Done Sent false
4511 2288 Acked Write 7 Done Done Sent false
4512 2289 Acked Read 3 Done Done Sent false
4513 2290 Acked Read 6 Done Done Sent false
4514 2291 Acked Write 2 Done Done Sent false
4515 2292 Acked Write 1 Done Done Sent false
4516 2293 Acked Read 10 Done Done Sent false
4517 2294 Acked Write 4 Done Done Sent false
4518 2295 Acked Read 2 Done Done Sent false
4519 2296 Acked Read 7 Done Done Sent false
4520 2297 Acked Write 1 Done Done Sent false
4521 2298 Acked Read 6 Done Done Sent false
4522 2299 Acked Read 2 Done Done Sent false
4523 2300 Acked Read 6 Done Done Sent false
4524 2301 Acked Write 8 Done Done Sent false
4525 2302 Acked Write 4 Done Done Sent false
4526 2303 Acked Write 2 Done Done Sent false
4527 2304 Acked Read 3 Done Done Sent false
4528 2305 Acked Write 7 Done Done Sent false
4529 2306 Acked Flush 0 Done Done Sent false
4530 2307 Acked Read 1 Done Done Sent false
4531 2308 Acked Write 4 Done Done Sent false
4532 2309 Acked Read 8 Done Done Sent false
4533 2310 Acked Flush 0 Done Done Sent false
4534 2311 Acked Read 6 Done Done Sent false
4535 2312 Acked Read 3 Done Done Sent false
4536 2313 Acked Read 4 Done Done Sent false
4537 2314 Acked Flush 0 Done Done Sent false
4538 2315 Acked Write 9 Done Done Sent false
4539 2316 Acked Read 7 Done Done Sent false
4540 2317 Acked Read 8 Done Done Sent false
4541 2318 Acked Write 9 Done Done Sent false
4542 2319 Acked Read 9 Done Done Sent false
4543 2320 Acked Write 1 Done Done Sent false
4544 2321 Acked Write 7 Done Done Sent false
4545 2322 Acked Read 7 Done Done Sent false
4546 2323 Acked Write 8 Done Done Sent false
4547 2324 Acked Write 10 Done Done Sent false
4548 2325 Acked Write 8 Done Done Sent false
4549 2326 Acked Write 8 Done Done Sent false
4550 2327 Acked Write 1 Done Done Sent false
4551 2328 Acked Read 9 Done Done Sent false
4552 2329 Acked Write 8 Done Done Sent false
4553 2330 Acked Write 8 Done Done Sent false
4554 2331 Acked Write 5 Done Done Sent false
4555 2332 Acked Read 10 Done Done Sent false
4556 2333 Acked Read 1 Done Done Sent false
4557 2334 Acked Write 10 Done Done Sent false
4558 2335 Acked Flush 0 Done Done Sent false
4559 2336 Acked Write 1 Done Done Sent false
4560 2337 Acked Read 9 Done Done Sent false
4561 2338 Acked Read 4 Done Done Sent false
4562 2339 Acked Read 2 Done Done Sent false
4563 2340 Acked Read 9 Done Done Sent false
4564 2341 Acked Read 2 Done Done Sent false
4565 2342 Acked Write 9 Done Done Sent false
4566 2343 Acked Write 9 Done Done Sent false
4567 2344 Acked Write 8 Done Done Sent false
4568 2345 Acked Write 10 Done Done Sent false
4569 2346 Acked Read 4 Done Done Sent false
4570 2347 Acked Write 7 Done Done Sent false
4571 2348 Acked Write 4 Done Done Sent false
4572 2349 Acked Read 7 Done Done Sent false
4573 2350 Acked Read 9 Done Done Sent false
4574 2351 Acked Read 4 Done Done Sent false
4575 2352 Acked Read 10 Done Done Sent false
4576 2353 Acked Write 7 Done Done Sent false
4577 2354 Acked Write 7 Done Done Sent false
4578 2355 Acked Read 7 Done Done Sent false
4579 2356 Acked Read 2 Done Done Sent false
4580 2357 Acked Read 5 Done Done Sent false
4581 2358 Acked Read 7 Done Done Sent false
4582 2359 Acked Write 7 Done Done Sent false
4583 2360 Acked Flush 0 Done Done Sent false
4584 2361 Acked Flush 0 Done Done Sent false
4585 2362 Acked Write 4 Done Done Sent false
4586 2363 Acked Read 7 Done Done Sent false
4587 2364 Acked Read 6 Done Done Sent false
4588 2365 Acked Read 4 Done Done Sent false
4589 2366 Acked Write 10 Done Done Sent false
4590 2367 Acked Write 5 Done Done Sent false
4591 2368 Acked Write 4 Done Done Sent false
4592 2369 Acked Read 2 Done Done Sent false
4593 2370 Acked Flush 0 Done Done Sent false
4594 2371 Acked Read 4 Done Done Sent false
4595 2372 Acked Read 4 Done Done Sent false
4596 2373 Acked Read 8 Done Done Sent false
4597 2374 Acked Write 4 Done Done Sent false
4598 2375 Acked Read 2 Done Done Sent false
4599 2376 Acked Write 1 Done Done Sent false
4600 2377 Acked Read 4 Done Done Sent false
4601 2378 Acked Flush 0 Done Done Sent false
4602 2379 Acked Read 4 Done Done Sent false
4603 2380 Acked Read 4 Done Done Sent false
4604 2381 Acked Write 4 Done Done Sent false
4605 2382 Acked Write 5 Done Done Sent false
4606 2383 Acked Read 4 Done Done Sent false
4607 2384 Acked Read 8 Done Done Sent false
4608 2385 Acked Read 3 Done Done Sent false
4609 2386 Acked Read 5 Done Done Sent false
4610 2387 Acked Write 2 Done Done Sent false
4611 2388 Acked Write 1 Done Done Sent false
4612 2389 Acked Write 3 Done Done Sent false
4613 2390 Acked Flush 0 Done Done Sent false
4614 2391 Acked Read 8 Done Done Sent false
4615 2392 Acked Read 6 Done Done Sent false
4616 2393 Acked Read 7 Done Done Sent false
4617 2394 Acked Flush 0 Done Done Sent false
4618 2395 Acked Write 2 Done Done Sent false
4619 2396 Acked Write 6 Done Done Sent false
4620 2397 Acked Read 8 Done Done Sent false
4621 2398 Acked Write 9 Done Done Sent false
4622 2399 Acked Write 9 Done Done Sent false
4623 2400 Acked Write 5 Done Done Sent false
4624 2401 Acked Read 1 Done Done Sent false
4625 2402 Acked Write 3 Done Done Sent false
4626 2403 Acked Read 6 Done Done Sent false
4627 2404 Acked Write 1 Done Done Sent false
4628 2405 Acked Read 4 Done Done Sent false
4629 2406 Acked Write 9 Done Done Sent false
4630 2407 Acked Read 5 Done Done Sent false
4631 2408 Acked Flush 0 Done Done Sent false
4632 2409 Acked Write 3 Done Done Sent false
4633 2410 Acked Write 1 Done Done Sent false
4634 2411 Acked Read 5 Done Done Sent false
4635 2412 Acked Read 3 Done Done Sent false
4636 2413 Acked Write 10 Done Done Sent false
4637 2414 Acked Read 9 Done Done Sent false
4638 2415 Acked Write 9 Done Done Sent false
4639 2416 Acked Write 4 Done Done Sent false
4640 2417 Acked Write 3 Done Done Sent false
4641 2418 Acked Read 6 Done Done Sent false
4642 2419 Acked Flush 0 Done Done Sent false
4643 2420 Acked Write 4 Done Done Sent false
4644 2421 Acked Flush 0 Done Done Sent false
4645 2422 Acked Read 5 Done Done Sent false
4646 2423 Acked Read 3 Done Done Sent false
4647 2424 Acked Read 5 Done Done Sent false
4648 2425 Acked Write 7 Done Done Sent false
4649 2426 Acked Read 5 Done Done Sent false
4650 2427 Acked Write 7 Done Done Sent false
4651 2428 Acked Write 3 Done Done Sent false
4652 2429 Acked Write 9 Done Done Sent false
4653 2430 Acked Read 5 Done Done Sent false
4654 2431 Acked Read 3 Done Done Sent false
4655 2432 Acked Write 10 Done Done Sent false
4656 2433 Acked Read 2 Done Done Sent false
4657 2434 Acked Flush 0 Done Done Sent false
4658 2435 Acked Read 5 Done Done Sent false
4659 2436 Acked Write 5 Done Done Sent false
4660 2437 Acked Read 7 Done Done Sent false
4661 2438 Acked Write 3 Done Done Sent false
4662 2439 Acked Write 6 Done Done Sent false
4663 2440 Acked Write 1 Done Done Sent false
4664 2441 Acked Write 4 Done Done Sent false
4665 2442 Acked Read 1 Done Done Sent false
4666 2443 Acked Read 3 Done Done Sent false
4667 2444 Acked Read 1 Done Done Sent false
4668 2445 Acked Read 4 Done Done Sent false
4669 2446 Acked Read 7 Done Done Sent false
4670 2447 Acked Write 7 Done Done Sent false
4671 2448 Acked Read 7 Done Done Sent false
4672 2449 Acked Write 8 Done Done Sent false
4673 2450 Acked Read 4 Done Done Sent false
4674 2451 Acked Read 1 Done Done Sent false
4675 2452 Acked Read 4 Done Done Sent false
4676 2453 Acked Write 3 Done Done Sent false
4677 2454 Acked Write 10 Done Done Sent false
4678 2455 Acked Read 4 Done Done Sent false
4679 2456 Acked Read 5 Done Done Sent false
4680 2457 Acked Write 3 Done Done Sent false
4681 2458 Acked Read 3 Done Done Sent false
4682 2459 Acked Read 2 Done Done Sent false
4683 2460 Acked Read 7 Done Done Sent false
4684 2461 Acked Write 4 Done Done Sent false
4685 2462 Acked Read 7 Done Done Sent false
4686 2463 Acked Read 3 Done Done Sent false
4687 2464 Acked Read 2 Done Done Sent false
4688 2465 Acked Read 4 Done Done Sent false
4689 2466 Acked Read 1 Done Done Sent false
4690 2467 Acked Read 1 Done Done Sent false
4691 2468 Acked Read 1 Done Done Sent false
4692 2469 Acked Flush 0 Done Done Sent false
4693 2470 Acked Read 6 Done Done Sent false
4694 2471 Acked Write 5 Done Done Sent false
4695 2472 Acked Read 3 Done Done Sent false
4696 2473 Acked Write 9 Done Done Sent false
4697 2474 Acked Write 5 Done Done Sent false
4698 2475 Acked Read 7 Done Done Sent false
4699 2476 Acked Flush 0 Done Done Sent false
4700 2477 Acked Flush 0 Done Done Sent false
4701 2478 Acked Write 7 Done Done Sent false
4702 2479 Acked Read 10 Done Done Sent false
4703 2480 Acked Write 3 Done Done Sent false
4704 2481 Acked Read 2 Done Done Sent false
4705 2482 Acked Read 3 Done Done Sent false
4706 2483 Acked Read 6 Done Done Sent false
4707 2484 Acked Read 9 Done Done Sent false
4708 2485 Acked Read 3 Done Done Sent false
4709 2486 Acked Read 2 Done Done Sent false
4710 2487 Acked Read 9 Done Done Sent false
4711 2488 Acked Read 6 Done Done Sent false
4712 2489 Acked Read 3 Done Done Sent false
4713 2490 Acked Write 4 Done Done Sent false
4714 2491 Acked Write 7 Done Done Sent false
4715 2492 Acked Read 3 Done Done Sent false
4716 2493 Acked Write 3 Done Done Sent false
4717 2494 Acked Read 10 Done Done Sent false
4718 2495 Acked Read 6 Done Done Sent false
4719 2496 Acked Read 3 Done Done Sent false
4720 2497 Acked Read 8 Done Done Sent false
4721 2498 Acked Flush 0 Done Done Sent false
4722 2499 Acked Write 1 Done Done Sent false
4723 2500 Acked Write 7 Done Done Sent false
4724 2501 Acked Read 7 Done Done Sent false
4725 2502 Acked Write 5 Done Done Sent false
4726 2503 Acked Read 5 Done Done Sent false
4727 2504 Acked Read 3 Done Done Sent false
4728 2505 Acked Write 8 Done Done Sent false
4729 2506 Acked Write 9 Done Done Sent false
4730 2507 Acked Write 1 Done Done Sent false
4731 2508 Acked Write 1 Sent Done Sent false
4732 STATES DS:0 DS:1 DS:2 TOTAL
4733 Sent 1 0 301 302
4734 Done 300 301 0 601
4735 Skipped 0 0 0 0
4736 Error 0 0 0 0
4737 Last Flush: Some(JobId(2498)) Some(JobId(2498)) Some(JobId(2207))
4738 Downstairs last five completed: 2207 2206 2205 2204 2203
4739 Upstairs last five completed: 2508 2507 2506 2505 2504
4740 CLIENT: Up:0 ds:301 act:2
47412025-02-21T06:46:16.959ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
47422025-02-21T06:46:18.214ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
4743 [8830] got start action
4744 [8830] I am going to restart
4745 Starting downstairs at port 8830
4746 Make output file at "/tmp/dsc/downstairs-8830.txt"
4747 [8830][2] reports Starting
4748 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1211)
4749 [8830][2] reports Running
47502025-02-21T06:46:18.744ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
47512025-02-21T06:46:19.745ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
4752 ----------------------------------------------------------------
4753 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4754 States: Active Active Offline
4755 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4756 2208 Acked Read 9 Done Done Sent false
4757 2209 Acked Flush 0 Done Done Sent false
4758 2210 Acked Read 3 Done Done Sent false
4759 2211 Acked Write 6 Done Done Sent false
4760 2212 Acked Read 6 Done Done Sent false
4761 2213 Acked Write 5 Done Done Sent false
4762 2214 Acked Flush 0 Done Done Sent false
4763 2215 Acked Write 1 Done Done Sent false
4764 2216 Acked Write 6 Done Done Sent false
4765 2217 Acked Write 1 Done Done Sent false
4766 2218 Acked Read 4 Done Done Sent false
4767 2219 Acked Read 9 Done Done Sent false
4768 2220 Acked Read 3 Done Done Sent false
4769 2221 Acked Read 7 Done Done Sent false
4770 2222 Acked Read 6 Done Done Sent false
4771 2223 Acked Read 7 Done Done Sent false
4772 2224 Acked Flush 0 Done Done Sent false
4773 2225 Acked Write 10 Done Done Sent false
4774 2226 Acked Read 1 Done Done Sent false
4775 2227 Acked Write 2 Done Done Sent false
4776 2228 Acked Write 7 Done Done Sent false
4777 2229 Acked Flush 0 Done Done Sent false
4778 2230 Acked Write 4 Done Done Sent false
4779 2231 Acked Write 5 Done Done Sent false
4780 2232 Acked Flush 0 Done Done Sent false
4781 2233 Acked Read 4 Done Done Sent false
4782 2234 Acked Read 8 Done Done Sent false
4783 2235 Acked Read 10 Done Done Sent false
4784 2236 Acked Read 7 Done Done Sent false
4785 2237 Acked Flush 0 Done Done Sent false
4786 2238 Acked Write 4 Done Done Sent false
4787 2239 Acked Read 1 Done Done Sent false
4788 2240 Acked Write 5 Done Done Sent false
4789 2241 Acked Write 7 Done Done Sent false
4790 2242 Acked Write 2 Done Done Sent false
4791 2243 Acked Write 8 Done Done Sent false
4792 2244 Acked Write 5 Done Done Sent false
4793 2245 Acked Write 9 Done Done Sent false
4794 2246 Acked Flush 0 Done Done Sent false
4795 2247 Acked Flush 0 Done Done Sent false
4796 2248 Acked Read 10 Done Done Sent false
4797 2249 Acked Read 9 Done Done Sent false
4798 2250 Acked Read 9 Done Done Sent false
4799 2251 Acked Write 2 Done Done Sent false
4800 2252 Acked Read 9 Done Done Sent false
4801 2253 Acked Read 3 Done Done Sent false
4802 2254 Acked Read 7 Done Done Sent false
4803 2255 Acked Read 1 Done Done Sent false
4804 2256 Acked Read 2 Done Done Sent false
4805 2257 Acked Write 2 Done Done Sent false
4806 2258 Acked Read 2 Done Done Sent false
4807 2259 Acked Read 6 Done Done Sent false
4808 2260 Acked Read 8 Done Done Sent false
4809 2261 Acked Read 7 Done Done Sent false
4810 2262 Acked Write 5 Done Done Sent false
4811 2263 Acked Write 8 Done Done Sent false
4812 2264 Acked Read 5 Done Done Sent false
4813 2265 Acked Read 5 Done Done Sent false
4814 2266 Acked Write 8 Done Done Sent false
4815 2267 Acked Write 6 Done Done Sent false
4816 2268 Acked Write 9 Done Done Sent false
4817 2269 Acked Flush 0 Done Done Sent false
4818 2270 Acked Write 9 Done Done Sent false
4819 2271 Acked Read 6 Done Done Sent false
4820 2272 Acked Read 2 Done Done Sent false
4821 2273 Acked Read 9 Done Done Sent false
4822 2274 Acked Write 6 Done Done Sent false
4823 2275 Acked Write 4 Done Done Sent false
4824 2276 Acked Flush 0 Done Done Sent false
4825 2277 Acked Read 3 Done Done Sent false
4826 2278 Acked Write 8 Done Done Sent false
4827 2279 Acked Write 3 Done Done Sent false
4828 2280 Acked Write 8 Done Done Sent false
4829 2281 Acked Read 6 Done Done Sent false
4830 2282 Acked Read 3 Done Done Sent false
4831 2283 Acked Write 9 Done Done Sent false
4832 2284 Acked Read 10 Done Done Sent false
4833 2285 Acked Read 10 Done Done Sent false
4834 2286 Acked Read 10 Done Done Sent false
4835 2287 Acked Write 3 Done Done Sent false
4836 2288 Acked Write 7 Done Done Sent false
4837 2289 Acked Read 3 Done Done Sent false
4838 2290 Acked Read 6 Done Done Sent false
4839 2291 Acked Write 2 Done Done Sent false
4840 2292 Acked Write 1 Done Done Sent false
4841 2293 Acked Read 10 Done Done Sent false
4842 2294 Acked Write 4 Done Done Sent false
4843 2295 Acked Read 2 Done Done Sent false
4844 2296 Acked Read 7 Done Done Sent false
4845 2297 Acked Write 1 Done Done Sent false
4846 2298 Acked Read 6 Done Done Sent false
4847 2299 Acked Read 2 Done Done Sent false
4848 2300 Acked Read 6 Done Done Sent false
4849 2301 Acked Write 8 Done Done Sent false
4850 2302 Acked Write 4 Done Done Sent false
4851 2303 Acked Write 2 Done Done Sent false
4852 2304 Acked Read 3 Done Done Sent false
4853 2305 Acked Write 7 Done Done Sent false
4854 2306 Acked Flush 0 Done Done Sent false
4855 2307 Acked Read 1 Done Done Sent false
4856 2308 Acked Write 4 Done Done Sent false
4857 2309 Acked Read 8 Done Done Sent false
4858 2310 Acked Flush 0 Done Done Sent false
4859 2311 Acked Read 6 Done Done Sent false
4860 2312 Acked Read 3 Done Done Sent false
4861 2313 Acked Read 4 Done Done Sent false
4862 2314 Acked Flush 0 Done Done Sent false
4863 2315 Acked Write 9 Done Done Sent false
4864 2316 Acked Read 7 Done Done Sent false
4865 2317 Acked Read 8 Done Done Sent false
4866 2318 Acked Write 9 Done Done Sent false
4867 2319 Acked Read 9 Done Done Sent false
4868 2320 Acked Write 1 Done Done Sent false
4869 2321 Acked Write 7 Done Done Sent false
4870 2322 Acked Read 7 Done Done Sent false
4871 2323 Acked Write 8 Done Done Sent false
4872 2324 Acked Write 10 Done Done Sent false
4873 2325 Acked Write 8 Done Done Sent false
4874 2326 Acked Write 8 Done Done Sent false
4875 2327 Acked Write 1 Done Done Sent false
4876 2328 Acked Read 9 Done Done Sent false
4877 2329 Acked Write 8 Done Done Sent false
4878 2330 Acked Write 8 Done Done Sent false
4879 2331 Acked Write 5 Done Done Sent false
4880 2332 Acked Read 10 Done Done Sent false
4881 2333 Acked Read 1 Done Done Sent false
4882 2334 Acked Write 10 Done Done Sent false
4883 2335 Acked Flush 0 Done Done Sent false
4884 2336 Acked Write 1 Done Done Sent false
4885 2337 Acked Read 9 Done Done Sent false
4886 2338 Acked Read 4 Done Done Sent false
4887 2339 Acked Read 2 Done Done Sent false
4888 2340 Acked Read 9 Done Done Sent false
4889 2341 Acked Read 2 Done Done Sent false
4890 2342 Acked Write 9 Done Done Sent false
4891 2343 Acked Write 9 Done Done Sent false
4892 2344 Acked Write 8 Done Done Sent false
4893 2345 Acked Write 10 Done Done Sent false
4894 2346 Acked Read 4 Done Done Sent false
4895 2347 Acked Write 7 Done Done Sent false
4896 2348 Acked Write 4 Done Done Sent false
4897 2349 Acked Read 7 Done Done Sent false
4898 2350 Acked Read 9 Done Done Sent false
4899 2351 Acked Read 4 Done Done Sent false
4900 2352 Acked Read 10 Done Done Sent false
4901 2353 Acked Write 7 Done Done Sent false
4902 2354 Acked Write 7 Done Done Sent false
4903 2355 Acked Read 7 Done Done Sent false
4904 2356 Acked Read 2 Done Done Sent false
4905 2357 Acked Read 5 Done Done Sent false
4906 2358 Acked Read 7 Done Done Sent false
4907 2359 Acked Write 7 Done Done Sent false
4908 2360 Acked Flush 0 Done Done Sent false
4909 2361 Acked Flush 0 Done Done Sent false
4910 2362 Acked Write 4 Done Done Sent false
4911 2363 Acked Read 7 Done Done Sent false
4912 2364 Acked Read 6 Done Done Sent false
4913 2365 Acked Read 4 Done Done Sent false
4914 2366 Acked Write 10 Done Done Sent false
4915 2367 Acked Write 5 Done Done Sent false
4916 2368 Acked Write 4 Done Done Sent false
4917 2369 Acked Read 2 Done Done Sent false
4918 2370 Acked Flush 0 Done Done Sent false
4919 2371 Acked Read 4 Done Done Sent false
4920 2372 Acked Read 4 Done Done Sent false
4921 2373 Acked Read 8 Done Done Sent false
4922 2374 Acked Write 4 Done Done Sent false
4923 2375 Acked Read 2 Done Done Sent false
4924 2376 Acked Write 1 Done Done Sent false
4925 2377 Acked Read 4 Done Done Sent false
4926 2378 Acked Flush 0 Done Done Sent false
4927 2379 Acked Read 4 Done Done Sent false
4928 2380 Acked Read 4 Done Done Sent false
4929 2381 Acked Write 4 Done Done Sent false
4930 2382 Acked Write 5 Done Done Sent false
4931 2383 Acked Read 4 Done Done Sent false
4932 2384 Acked Read 8 Done Done Sent false
4933 2385 Acked Read 3 Done Done Sent false
4934 2386 Acked Read 5 Done Done Sent false
4935 2387 Acked Write 2 Done Done Sent false
4936 2388 Acked Write 1 Done Done Sent false
4937 2389 Acked Write 3 Done Done Sent false
4938 2390 Acked Flush 0 Done Done Sent false
4939 2391 Acked Read 8 Done Done Sent false
4940 2392 Acked Read 6 Done Done Sent false
4941 2393 Acked Read 7 Done Done Sent false
4942 2394 Acked Flush 0 Done Done Sent false
4943 2395 Acked Write 2 Done Done Sent false
4944 2396 Acked Write 6 Done Done Sent false
4945 2397 Acked Read 8 Done Done Sent false
4946 2398 Acked Write 9 Done Done Sent false
4947 2399 Acked Write 9 Done Done Sent false
4948 2400 Acked Write 5 Done Done Sent false
4949 2401 Acked Read 1 Done Done Sent false
4950 2402 Acked Write 3 Done Done Sent false
4951 2403 Acked Read 6 Done Done Sent false
4952 2404 Acked Write 1 Done Done Sent false
4953 2405 Acked Read 4 Done Done Sent false
4954 2406 Acked Write 9 Done Done Sent false
4955 2407 Acked Read 5 Done Done Sent false
4956 2408 Acked Flush 0 Done Done Sent false
4957 2409 Acked Write 3 Done Done Sent false
4958 2410 Acked Write 1 Done Done Sent false
4959 2411 Acked Read 5 Done Done Sent false
4960 2412 Acked Read 3 Done Done Sent false
4961 2413 Acked Write 10 Done Done Sent false
4962 2414 Acked Read 9 Done Done Sent false
4963 2415 Acked Write 9 Done Done Sent false
4964 2416 Acked Write 4 Done Done Sent false
4965 2417 Acked Write 3 Done Done Sent false
4966 2418 Acked Read 6 Done Done Sent false
4967 2419 Acked Flush 0 Done Done Sent false
4968 2420 Acked Write 4 Done Done Sent false
4969 2421 Acked Flush 0 Done Done Sent false
4970 2422 Acked Read 5 Done Done Sent false
4971 2423 Acked Read 3 Done Done Sent false
4972 2424 Acked Read 5 Done Done Sent false
4973 2425 Acked Write 7 Done Done Sent false
4974 2426 Acked Read 5 Done Done Sent false
4975 2427 Acked Write 7 Done Done Sent false
4976 2428 Acked Write 3 Done Done Sent false
4977 2429 Acked Write 9 Done Done Sent false
4978 2430 Acked Read 5 Done Done Sent false
4979 2431 Acked Read 3 Done Done Sent false
4980 2432 Acked Write 10 Done Done Sent false
4981 2433 Acked Read 2 Done Done Sent false
4982 2434 Acked Flush 0 Done Done Sent false
4983 2435 Acked Read 5 Done Done Sent false
4984 2436 Acked Write 5 Done Done Sent false
4985 2437 Acked Read 7 Done Done Sent false
4986 2438 Acked Write 3 Done Done Sent false
4987 2439 Acked Write 6 Done Done Sent false
4988 2440 Acked Write 1 Done Done Sent false
4989 2441 Acked Write 4 Done Done Sent false
4990 2442 Acked Read 1 Done Done Sent false
4991 2443 Acked Read 3 Done Done Sent false
4992 2444 Acked Read 1 Done Done Sent false
4993 2445 Acked Read 4 Done Done Sent false
4994 2446 Acked Read 7 Done Done Sent false
4995 2447 Acked Write 7 Done Done Sent false
4996 2448 Acked Read 7 Done Done Sent false
4997 2449 Acked Write 8 Done Done Sent false
4998 2450 Acked Read 4 Done Done Sent false
4999 2451 Acked Read 1 Done Done Sent false
5000 2452 Acked Read 4 Done Done Sent false
5001 2453 Acked Write 3 Done Done Sent false
5002 2454 Acked Write 10 Done Done Sent false
5003 2455 Acked Read 4 Done Done Sent false
5004 2456 Acked Read 5 Done Done Sent false
5005 2457 Acked Write 3 Done Done Sent false
5006 2458 Acked Read 3 Done Done Sent false
5007 2459 Acked Read 2 Done Done Sent false
5008 2460 Acked Read 7 Done Done Sent false
5009 2461 Acked Write 4 Done Done Sent false
5010 2462 Acked Read 7 Done Done Sent false
5011 2463 Acked Read 3 Done Done Sent false
5012 2464 Acked Read 2 Done Done Sent false
5013 2465 Acked Read 4 Done Done Sent false
5014 2466 Acked Read 1 Done Done Sent false
5015 2467 Acked Read 1 Done Done Sent false
5016 2468 Acked Read 1 Done Done Sent false
5017 2469 Acked Flush 0 Done Done Sent false
5018 2470 Acked Read 6 Done Done Sent false
5019 2471 Acked Write 5 Done Done Sent false
5020 2472 Acked Read 3 Done Done Sent false
5021 2473 Acked Write 9 Done Done Sent false
5022 2474 Acked Write 5 Done Done Sent false
5023 2475 Acked Read 7 Done Done Sent false
5024 2476 Acked Flush 0 Done Done Sent false
5025 2477 Acked Flush 0 Done Done Sent false
5026 2478 Acked Write 7 Done Done Sent false
5027 2479 Acked Read 10 Done Done Sent false
5028 2480 Acked Write 3 Done Done Sent false
5029 2481 Acked Read 2 Done Done Sent false
5030 2482 Acked Read 3 Done Done Sent false
5031 2483 Acked Read 6 Done Done Sent false
5032 2484 Acked Read 9 Done Done Sent false
5033 2485 Acked Read 3 Done Done Sent false
5034 2486 Acked Read 2 Done Done Sent false
5035 2487 Acked Read 9 Done Done Sent false
5036 2488 Acked Read 6 Done Done Sent false
5037 2489 Acked Read 3 Done Done Sent false
5038 2490 Acked Write 4 Done Done Sent false
5039 2491 Acked Write 7 Done Done Sent false
5040 2492 Acked Read 3 Done Done Sent false
5041 2493 Acked Write 3 Done Done Sent false
5042 2494 Acked Read 10 Done Done Sent false
5043 2495 Acked Read 6 Done Done Sent false
5044 2496 Acked Read 3 Done Done Sent false
5045 2497 Acked Read 8 Done Done Sent false
5046 2498 Acked Flush 0 Done Done Sent false
5047 2499 Acked Write 1 Done Done Sent false
5048 2500 Acked Write 7 Done Done Sent false
5049 2501 Acked Read 7 Done Done Sent false
5050 2502 Acked Write 5 Done Done Sent false
5051 2503 Acked Read 5 Done Done Sent false
5052 2504 Acked Read 3 Done Done Sent false
5053 2505 Acked Write 8 Done Done Sent false
5054 2506 Acked Write 9 Done Done Sent false
5055 2507 Acked Write 1 Done Done Sent false
5056 2508 Acked Write 1 Done Done Sent false
5057 STATES DS:0 DS:1 DS:2 TOTAL
5058 Sent 0 0 301 301
5059 Done 301 301 0 602
5060 Skipped 0 0 0 0
5061 Error 0 0 0 0
5062 Last Flush: Some(JobId(2498)) Some(JobId(2498)) Some(JobId(2207))
5063 Downstairs last five completed: 2207 2206 2205 2204 2203
5064 Upstairs last five completed: 2508 2507 2506 2505 2504
5065 CLIENT: Up:0 ds:301 act:2
50662025-02-21T06:46:20.961ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
50672025-02-21T06:46:21.748ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
50682025-02-21T06:46:22.217ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
50692025-02-21T06:46:22.217ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46466) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
50702025-02-21T06:46:22.218ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
50712025-02-21T06:46:22.218ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
50722025-02-21T06:46:22.218ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
50732025-02-21T06:46:22.218ZINFOcrucible: send last flush ID to this DS: Some(JobId(2207)) = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
50742025-02-21T06:46:22.218ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
50752025-02-21T06:46:22.218ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(2207)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
50762025-02-21T06:46:22.219ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(2207)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
5077 ----------------------------------------------------------------
5078 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5079 Downstairs last five completed: 2509 2508 2507 2506 2505
5080 Upstairs last five completed: 2509 2508 2507 2506 2505
5081 CLIENT: Up:0 ds:0 act:3
5082 Replay: All jobs finished
5083 Main task has work to do, go find it
5084 got dsc Stop(2)
5085 stop 2
5086 Feb 21 06:46:24.462 INFO request completed, latency_us: 358, response_code: 204, uri: /stop/cid/2, [8830] Got stop action so:false kr:false
5087 method: POST, req_id: 6c44b0e1-6e4f-4e89-8005-254347c028b8, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
5088 Feb 21 06:46:24.463 INFO request completed, latency_us: 150, response_code: 200, uri: /state/cid/2, method: GET, req_id: 64a0a363-1088-4da1-a8ef-3c6b99a3cb62, remote_addr: 127.0.0.1:53848, local_addr: 127{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:46:24.463455724Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
5089 .0.0.1:9998
50902025-02-21T06:46:24.463ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
50912025-02-21T06:46:24.463ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
50922025-02-21T06:46:24.463ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
5093 [8830] Exited with: ExitStatus(unix_wait_status(9))
5094 [8830][2] reports Exit
5095 Feb 21 06:46:28.465 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/2, method: GET, req_id: b735f45d-c610-49d5-96bf-f85f723620ef, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
5096 001/300 Read block 499 len 40960
5097 002/300 Read block 1180 len 28672
5098 003/300 Write block 2689 len 4096 data: 3
5099 004/300 Read block 797 len 20480
5100 005/300 Read block 3788 len 24576
5101 006/300 Write block 228 len 4096 data: 2
5102 007/300 Write block 3316 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5103 008/300 Read block 2544 len 24576
5104 009/300 Flush
5105 010/300 Read block 3662 len 4096
5106 011/300 Flush
5107 012/300 Write block 3912 len 20480 data: 3 2 3 3 3
5108 013/300 Write block 925 len 36864 data: 2 2 2 2 2 2 2 2 2
5109 014/300 Flush
5110 015/300 Read block 4781 len 24576
5111 016/300 Write block 3636 len 40960 data: 3 3 2 2 2 3 3 3 3 4
5112 017/300 Read block 3222 len 8192
5113 018/300 Write block 749 len 8192 data: 2 2
5114 019/300 Write block 2463 len 8192 data: 3 3
5115 020/300 Flush
5116 021/300 Write block 3854 len 12288 data: 2 3 3
5117 022/300 Write block 3796 len 32768 data: 3 3 4 3 3 2 2 2
5118 023/300 Read block 216 len 4096
5119 024/300 Read block 651 len 20480
5120 025/300 Read block 1994 len 16384
5121 026/300 Read block 502 len 28672
5122 027/300 Flush
5123 028/300 Read block 461 len 24576
5124 029/300 Read block 4299 len 4096
5125 030/300 Write block 1821 len 24576 data: 4 4 3 2 2 2
5126 031/300 Read block 268 len 20480
5127 032/300 Write block 781 len 20480 data: 3 3 3 3 2
5128 033/300 Write block 4739 len 28672 data: 3 3 3 5 4 3 3
5129 034/300 Write block 217 len 8192 data: 4 4
5130 035/300 Write block 728 len 8192 data: 2 2
5131 036/300 Write block 533 len 16384 data: 2 2 2 2
5132 037/300 Read block 3426 len 12288
5133 038/300 Write block 1500 len 40960 data: 3 3 3 3 3 3 3 2 2 2
5134 039/300 Write block 4004 len 40960 data: 3 3 4 4 4 3 3 3 3 2
5135 040/300 Read block 3515 len 24576
5136 041/300 Read block 256 len 20480
5137 042/300 Read block 3712 len 32768
5138 043/300 Write block 539 len 28672 data: 2 2 2 2 2 2 3
5139 044/300 Write block 426 len 16384 data: 4 4 4 4
5140 045/300 Read block 3062 len 40960
5141 046/300 Write block 3479 len 20480 data: 3 3 3 3 3
5142 047/300 Read block 4149 len 32768
5143 048/300 Read block 182 len 4096
5144 049/300 Write block 2629 len 24576 data: 4 4 4 4 3 3
5145 050/300 Write block 3632 len 32768 data: 4 4 4 3 4 4 3 3
5146 051/300 Read block 3700 len 32768
5147 052/300 Write block 184 len 40960 data: 3 3 2 2 2 2 3 3 2 2
5148 053/300 Read block 3682 len 8192
5149 054/300 Write block 3246 len 20480 data: 3 4 4 4 3
5150 055/300 Read block 1977 len 8192
5151 056/300 Write block 660 len 40960 data: 2 2 2 2 2 2 2 2 2 3
5152 057/300 Write block 1063 len 32768 data: 2 2 2 2 2 2 2 2
5153 058/300 Write block 3696 len 32768 data: 3 3 2 2 2 2 2 2
5154 059/300 Write block 398 len 36864 data: 2 2 2 2 2 2 2 2 2
5155 060/300 Flush
5156 061/300 Write block 2983 len 36864 data: 2 2 2 3 3 3 3 3 3
5157 062/300 Read block 1835 len 36864
5158 063/300 Read block 1801 len 24576
5159 064/300 Write block 4014 len 8192 data: 2 2
5160 065/300 Read block 3978 len 24576
5161 066/300 Write block 3941 len 40960 data: 3 3 3 3 3 2 3 3 3 4
5162 067/300 Read block 3589 len 24576
5163 068/300 Write block 1182 len 4096 data: 3
5164 069/300 Read block 4633 len 24576
5165 070/300 Read block 3087 len 24576
5166 071/300 Read block 321 len 32768
5167 072/300 Read block 1639 len 36864
5168 073/300 Write block 785 len 28672 data: 3 2 2 2 2 2 2
5169 074/300 Read block 3433 len 4096
5170 075/300 Write block 723 len 8192 data: 2 2
5171 076/300 Flush
5172 077/300 Read block 2387 len 36864
5173 078/300 Read block 1097 len 40960
5174 079/300 Read block 2161 len 32768
5175 080/300 Write block 955 len 40960 data: 3 3 3 3 3 3 3 2 2 2
5176 081/300 Read block 3593 len 12288
5177 082/300 Write block 1017 len 24576 data: 3 2 2 2 2 2
5178 083/300 Read block 1148 len 24576
5179 084/300 Write block 4823 len 28672 data: 4 4 3 2 2 2 2
5180 085/300 Read block 3133 len 8192
5181 086/300 Read block 4780 len 8192
5182 087/300 Read block 4430 len 8192
5183 088/300 Read block 1283 len 40960
5184 089/300 Read block 2325 len 8192
5185 090/300 Read block 3421 len 24576
5186 091/300 Read block 2551 len 8192
5187 092/300 Read block 587 len 4096
5188 093/300 Read block 1248 len 20480
5189 094/300 Write block 663 len 32768 data: 3 3 3 3 3 3 4 3
5190 095/300 Read block 4045 len 20480
5191 096/300 Write block 3940 len 12288 data: 3 4 4
5192 097/300 Read block 2585 len 20480
5193 098/300 Write block 4008 len 40960 data: 5 4 4 4 4 3 3 3 2 2
5194 099/300 Write block 1086 len 8192 data: 2 2
5195 100/300 Write block 3621 len 36864 data: 2 2 2 3 3 3 3 3 2
5196 101/300 Read block 3477 len 40960
5197 102/300 Read block 1300 len 8192
5198 103/300 Read block 4585 len 12288
5199 104/300 Write block 4925 len 8192 data: 3 2
5200 105/300 Read block 61 len 32768
5201 106/300 Read block 413 len 16384
5202 107/300 Read block 2505 len 20480
5203 108/300 Write block 4814 len 28672 data: 2 2 2 2 3 3 3
5204 109/300 Write block 2047 len 16384 data: 3 3 3 4
5205 110/300 Read block 4769 len 28672
5206 111/300 Read block 3311 len 4096
5207 112/300 Flush
5208 113/300 Read block 1100 len 20480
5209 114/300 Flush
5210 115/300 Read block 1158 len 28672
5211 116/300 Read block 2366 len 8192
5212 117/300 Read block 3914 len 16384
5213 118/300 Read block 3663 len 16384
5214 119/300 Read block 4464 len 16384
5215 120/300 Read block 2719 len 12288
5216 121/300 Write block 3700 len 4096 data: 3
5217 122/300 Write block 3766 len 36864 data: 2 2 2 2 3 3 4 4 3
5218 123/300 Read block 3839 len 28672
5219 124/300 Write block 761 len 20480 data: 4 3 3 3 4
5220 125/300 Write block 1747 len 40960 data: 3 4 4 3 3 3 2 2 2 2
5221 126/300 Read block 4637 len 16384
5222 127/300 Read block 4841 len 16384
5223 128/300 Read block 2950 len 36864
5224 129/300 Flush
5225 130/300 Read block 2950 len 36864
5226 131/300 Write block 1984 len 28672 data: 3 4 5 5 5 4 3
5227 132/300 Read block 1786 len 8192
5228 133/300 Read block 1877 len 8192
5229 134/300 Read block 4706 len 24576
5230 135/300 Flush
5231 136/300 Write block 3348 len 4096 data: 4
5232 137/300 Read block 3119 len 16384
5233 138/300 Read block 2441 len 16384
5234 139/300 Read block 1064 len 8192
5235 140/300 Read block 2106 len 20480
5236 141/300 Read block 2946 len 24576
5237 142/300 Write block 4934 len 28672 data: 2 2 2 2 2 2 2
5238 143/300 Flush
5239 144/300 Read block 215 len 36864
5240 145/300 Write block 3114 len 12288 data: 2 2 2
5241 146/300 Read block 2446 len 28672
5242 147/300 Write block 4568 len 8192 data: 4 4
5243 148/300 Read block 3010 len 36864
5244 149/300 Write block 1498 len 28672 data: 3 3 4 4 4 4 4
5245 150/300 Read block 2882 len 36864
5246 151/300 Read block 4632 len 20480
5247 152/300 Read block 2379 len 16384
5248 153/300 Read block 3452 len 16384
5249 154/300 Write block 4077 len 4096 data: 4
5250 155/300 Read block 890 len 4096
5251 156/300 Write block 842 len 12288 data: 2 2 2
5252 157/300 Read block 3373 len 36864
5253 158/300 Read block 2965 len 24576
5254 159/300 Write block 1408 len 20480 data: 2 2 2 2 2
5255 160/300 Read block 1578 len 40960
5256 161/300 Read block 1645 len 12288
5257 162/300 Write block 153 len 40960 data: 2 2 3 3 3 3 3 3 2 2
5258 163/300 Flush
5259 164/300 Write block 3515 len 28672 data: 3 2 2 2 2 2 2
5260 165/300 Read block 2704 len 28672
5261 166/300 Read block 2311 len 4096
5262 167/300 Read block 1939 len 28672
5263 168/300 Read block 726 len 12288
5264 169/300 Read block 215 len 16384
5265 170/300 Read block 4853 len 40960
5266 171/300 Write block 4758 len 12288 data: 2 2 2
5267 172/300 Read block 638 len 36864
5268 173/300 Write block 3419 len 12288 data: 3 3 3
5269 174/300 Read block 2143 len 12288
5270 175/300 Read block 3841 len 28672
5271 176/300 Write block 3422 len 32768 data: 2 2 2 3 3 3 3 3
5272 177/300 Read block 3646 len 12288
5273 178/300 Read block 961 len 32768
5274 179/300 Read block 1261 len 28672
5275 180/300 Read block 4209 len 36864
5276 181/300 Read block 559 len 36864
5277 182/300 Read block 4171 len 8192
5278 183/300 Read block 3950 len 36864
5279 184/300 Write block 3964 len 20480 data: 3 2 2 2 2
5280 185/300 Write block 1607 len 24576 data: 3 4 6 5 6 6
5281 186/300 Write block 2502 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5282 187/300 Read block 2807 len 12288
5283 188/300 Write block 537 len 36864 data: 2 2 3 3 3 3 3 3 4
5284 189/300 Flush
5285 190/300 Flush
5286 191/300 Write block 2746 len 32768 data: 2 3 3 3 3 4 4 4
5287 192/300 Read block 4276 len 40960
5288 193/300 Read block 1028 len 20480
5289 194/300 Read block 549 len 16384
5290 195/300 Write block 2593 len 24576 data: 2 2 2 2 2 2
5291 196/300 Write block 4824 len 40960 data: 5 4 3 3 3 3 4 4 4 3
5292 197/300 Read block 1026 len 12288
5293 198/300 Read block 1715 len 8192
5294 199/300 Read block 2962 len 16384
5295 200/300 Read block 827 len 24576
5296 201/300 Read block 3524 len 16384
5297 202/300 Read block 4558 len 24576
5298 203/300 Read block 2863 len 16384
5299 204/300 Read block 1355 len 4096
5300 205/300 Read block 1847 len 8192
5301 206/300 Read block 1749 len 40960
5302 207/300 Write block 4216 len 24576 data: 2 2 2 2 2 2
5303 208/300 Read block 4391 len 24576
5304 209/300 Write block 3224 len 4096 data: 2
5305 210/300 Write block 3840 len 32768 data: 3 4 5 5 4 3 3 3
5306 211/300 Write block 3261 len 16384 data: 2 2 3 3
5307 212/300 Write block 3148 len 8192 data: 4 3
5308 213/300 Write block 3987 len 28672 data: 4 3 3 3 3 3 3
5309 214/300 Read block 3053 len 4096
5310 215/300 Read block 330 len 12288
5311 216/300 Read block 4497 len 24576
5312 217/300 Flush
5313 218/300 Read block 2887 len 28672
5314 219/300 Flush
5315 220/300 Read block 107 len 4096
5316 221/300 Write block 2209 len 36864 data: 2 2 2 2 2 3 4 4 4
5317 222/300 Read block 2173 len 4096
5318 223/300 Read block 3058 len 16384
5319 224/300 Read block 863 len 20480
5320 225/300 Read block 4372 len 32768
5321 226/300 Read block 4608 len 12288
5322 227/300 Write block 3031 len 36864 data: 2 2 3 3 3 4 4 4 5
5323 228/300 Write block 1621 len 32768 data: 2 2 3 3 4 4 4 4
5324 229/300 Write block 2951 len 36864 data: 4 3 3 3 3 3 3 2 3
5325 230/300 Write block 3581 len 24576 data: 4 6 6 6 7 6
5326 231/300 Write block 2902 len 36864 data: 2 2 2 2 2 2 2 2 2
5327 232/300 Flush
5328 233/300 Read block 1047 len 12288
5329 234/300 Read block 3790 len 24576
5330 235/300 Write block 1329 len 16384 data: 2 2 2 2
5331 236/300 Read block 4349 len 12288
5332 237/300 Read block 3707 len 16384
5333 238/300 Read block 510 len 24576
5334 239/300 Write block 3161 len 32768 data: 3 3 3 4 4 4 4 4
5335 240/300 Write block 1203 len 16384 data: 2 2 2 2
5336 241/300 Flush
5337 242/300 Read block 2696 len 24576
5338 243/300 Read block 4675 len 20480
5339 244/300 Read block 3579 len 8192
5340 245/300 Read block 2852 len 28672
5341 246/300 Read block 283 len 24576
5342 247/300 Write block 2550 len 4096 data: 2
5343 248/300 Read block 2269 len 28672
5344 249/300 Read block 4513 len 20480
5345 250/300 Read block 4493 len 36864
5346 251/300 Write block 4325 len 12288 data: 2 2 3
5347 252/300 Read block 1296 len 32768
5348 253/300 Read block 1611 len 20480
5349 254/300 Read block 772 len 40960
5350 255/300 Write block 2915 len 40960 data: 2 2 3 3 2 2 2 2 2 2
5351 256/300 Read block 281 len 12288
5352 257/300 Read block 3806 len 40960
5353 258/300 Read block 4766 len 20480
5354 259/300 Write block 1708 len 8192 data: 2 2
5355 260/300 Read block 801 len 24576
5356 261/300 Write block 1352 len 16384 data: 3 3 2 2
5357 262/300 Read block 859 len 40960
5358 263/300 Write block 644 len 32768 data: 2 2 2 2 2 2 2 2
5359 264/300 Write block 369 len 16384 data: 2 2 2 3
5360 265/300 Write block 1213 len 8192 data: 3 3
5361 266/300 Write block 2188 len 12288 data: 4 4 3
5362 267/300 Write block 1018 len 16384 data: 3 3 3 3
5363 268/300 Write block 4728 len 32768 data: 2 2 3 2 2 2 2 2
5364 269/300 Write block 1374 len 28672 data: 2 2 3 3 3 3 3
5365 270/300 Read block 1598 len 36864
5366 271/300 Read block 4792 len 12288
5367 272/300 Read block 892 len 28672
5368 273/300 Read block 1161 len 16384
5369 274/300 Read block 4881 len 12288
5370 275/300 Read block 1797 len 24576
5371 276/300 Read block 3256 len 16384
5372 277/300 Flush
5373 278/300 Write block 2889 len 16384 data: 3 3 3 3
5374 279/300 Read block 3362 len 40960
5375 280/300 Read block 1749 len 40960
5376 281/300 Flush
5377 282/300 Read block 1431 len 16384
5378 283/300 Flush
5379 284/300 Read block 457 len 24576
5380 285/300 Flush
5381 286/300 Read block 4737 len 12288
5382 287/300 Read block 2521 len 40960
5383 288/300 Read block 3696 len 4096
5384 289/300 Write block 3287 len 8192 data: 3 3
5385 290/300 Read block 274 len 28672
5386 291/300 Read block 1193 len 36864
5387 292/300 Write block 2554 len 36864 data: 2 2 2 2 2 2 2 2 2
5388 293/300 Read block 2667 len 36864
5389 294/300 Write block 2752 len 24576 data: 5 5 4 3 3 2
5390 295/300 Flush
5391 296/300 Read block 2784 len 28672
5392 297/300 Read block 209 len 16384
5393 298/300 Read block 2608 len 8192
5394 299/300 Write block 2505 len 24576 data: 3 3 3 3 3 3
5395 300/300 Write block 235 len 32768 data: 3 3 2 2 2 2 2 2
5396 301/300 Read block 4262 len 12288
5397 Main task has work to do, go find it
5398 got dsc Start(2)
5399 start 2
5400 Feb 21 06:46:28.718 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/2, method: POST, req_id: 8b5a1144-c272-499a-8ad6-8894781220d5, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
5401 Replay: started 2, returned:Ok(())
5402 ----------------------------------------------------------------
5403 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5404 States: Active Active Offline
5405 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5406 2510 Acked Read 10 Done Done Sent false
5407 2511 Acked Read 7 Done Done Sent false
5408 2512 Acked Write 1 Done Done Sent false
5409 2513 Acked Read 5 Done Done Sent false
5410 2514 Acked Read 6 Done Done Sent false
5411 2515 Acked Write 1 Done Done Sent false
5412 2516 Acked Write 10 Done Done Sent false
5413 2517 Acked Read 6 Done Done Sent false
5414 2518 Acked Flush 0 Done Done Sent false
5415 2519 Acked Read 1 Done Done Sent false
5416 2520 Acked Flush 0 Done Done Sent false
5417 2521 Acked Write 5 Done Done Sent false
5418 2522 Acked Write 9 Done Done Sent false
5419 2523 Acked Flush 0 Done Done Sent false
5420 2524 Acked Read 6 Done Done Sent false
5421 2525 Acked Write 10 Done Done Sent false
5422 2526 Acked Read 2 Done Done Sent false
5423 2527 Acked Write 2 Done Done Sent false
5424 2528 Acked Write 2 Done Done Sent false
5425 2529 Acked Flush 0 Done Done Sent false
5426 2530 Acked Write 3 Done Done Sent false
5427 2531 Acked Write 8 Done Done Sent false
5428 2532 Acked Read 1 Done Done Sent false
5429 2533 Acked Read 5 Done Done Sent false
5430 2534 Acked Read 4 Done Done Sent false
5431 2535 Acked Read 7 Done Done Sent false
5432 2536 Acked Flush 0 Done Done Sent false
5433 2537 Acked Read 6 Done Done Sent false
5434 2538 Acked Read 1 Done Done Sent false
5435 2539 Acked Write 6 Done Done Sent false
5436 2540 Acked Read 5 Done Done Sent false
5437 2541 Acked Write 5 Done Done Sent false
5438 2542 Acked Write 7 Done Done Sent false
5439 2543 Acked Write 2 Done Done Sent false
5440 2544 Acked Write 2 Done Done Sent false
5441 2545 Acked Write 4 Done Done Sent false
5442 2546 Acked Read 3 Done Done Sent false
5443 2547 Acked Write 10 Done Done Sent false
5444 2548 Acked Write 10 Done Done Sent false
5445 2549 Acked Read 6 Done Done Sent false
5446 2550 Acked Read 5 Done Done Sent false
5447 2551 Acked Read 8 Done Done Sent false
5448 2552 Acked Write 7 Done Done Sent false
5449 2553 Acked Write 4 Done Done Sent false
5450 2554 Acked Read 10 Done Done Sent false
5451 2555 Acked Write 5 Done Done Sent false
5452 2556 Acked Read 8 Done Done Sent false
5453 2557 Acked Read 1 Done Done Sent false
5454 2558 Acked Write 6 Done Done Sent false
5455 2559 Acked Write 8 Done Done Sent false
5456 2560 Acked Read 8 Done Done Sent false
5457 2561 Acked Write 10 Done Done Sent false
5458 2562 Acked Read 2 Done Done Sent false
5459 2563 Acked Write 5 Done Done Sent false
5460 2564 Acked Read 2 Done Done Sent false
5461 2565 Acked Write 10 Done Done Sent false
5462 2566 Acked Write 8 Done Done Sent false
5463 2567 Acked Write 8 Done Done Sent false
5464 2568 Acked Write 9 Done Done Sent false
5465 2569 Acked Flush 0 Done Done Sent false
5466 2570 Acked Write 9 Done Done Sent false
5467 2571 Acked Read 9 Done Done Sent false
5468 2572 Acked Read 6 Done Done Sent false
5469 2573 Acked Write 2 Done Done Sent false
5470 2574 Acked Read 6 Done Done Sent false
5471 2575 Acked Write 10 Done Done Sent false
5472 2576 Acked Read 6 Done Done Sent false
5473 2577 Acked Write 1 Done Done Sent false
5474 2578 Acked Read 6 Done Done Sent false
5475 2579 Acked Read 6 Done Done Sent false
5476 2580 Acked Read 8 Done Done Sent false
5477 2581 Acked Read 9 Done Done Sent false
5478 2582 Acked Write 7 Done Done Sent false
5479 2583 Acked Read 1 Done Done Sent false
5480 2584 Acked Write 2 Done Done Sent false
5481 2585 Acked Flush 0 Done Done Sent false
5482 2586 Acked Read 9 Done Done Sent false
5483 2587 Acked Read 10 Done Done Sent false
5484 2588 Acked Read 8 Done Done Sent false
5485 2589 Acked Write 10 Done Done Sent false
5486 2590 Acked Read 3 Done Done Sent false
5487 2591 Acked Write 6 Done Done Sent false
5488 2592 Acked Read 6 Done Done Sent false
5489 2593 Acked Write 7 Done Done Sent false
5490 2594 Acked Read 2 Done Done Sent false
5491 2595 Acked Read 2 Done Done Sent false
5492 2596 Acked Read 2 Done Done Sent false
5493 2597 Acked Read 10 Done Done Sent false
5494 2598 Acked Read 2 Done Done Sent false
5495 2599 Acked Read 6 Done Done Sent false
5496 2600 Acked Read 2 Done Done Sent false
5497 2601 Acked Read 1 Done Done Sent false
5498 2602 Acked Read 5 Done Done Sent false
5499 2603 Acked Write 8 Done Done Sent false
5500 2604 Acked Read 5 Done Done Sent false
5501 2605 Acked Write 3 Done Done Sent false
5502 2606 Acked Read 5 Done Done Sent false
5503 2607 Acked Write 10 Done Done Sent false
5504 2608 Acked Write 2 Done Done Sent false
5505 2609 Acked Write 9 Done Done Sent false
5506 2610 Acked Read 10 Done Done Sent false
5507 2611 Acked Read 2 Done Done Sent false
5508 2612 Acked Read 3 Done Done Sent false
5509 2613 Acked Write 2 Done Done Sent false
5510 2614 Acked Read 8 Done Done Sent false
5511 2615 Acked Read 4 Done Done Sent false
5512 2616 Acked Read 5 Done Done Sent false
5513 2617 Acked Write 7 Done Done Sent false
5514 2618 Acked Write 4 Done Done Sent false
5515 2619 Acked Read 7 Done Done Sent false
5516 2620 Acked Read 1 Done Done Sent false
5517 2621 Acked Flush 0 Done Done Sent false
5518 2622 Acked Read 5 Done Done Sent false
5519 2623 Acked Flush 0 Done Done Sent false
5520 2624 Acked Read 7 Done Done Sent false
5521 2625 Acked Read 2 Done Done Sent false
5522 2626 Acked Read 4 Done Done Sent false
5523 2627 Acked Read 4 Done Done Sent false
5524 2628 Acked Read 4 Done Done Sent false
5525 2629 Acked Read 3 Done Done Sent false
5526 2630 Acked Write 1 Done Done Sent false
5527 2631 Acked Write 9 Done Done Sent false
5528 2632 Acked Read 7 Done Done Sent false
5529 2633 Acked Write 5 Done Done Sent false
5530 2634 Acked Write 10 Done Done Sent false
5531 2635 Acked Read 4 Done Done Sent false
5532 2636 Acked Read 4 Done Done Sent false
5533 2637 Acked Read 9 Done Done Sent false
5534 2638 Acked Flush 0 Done Done Sent false
5535 2639 Acked Read 9 Done Done Sent false
5536 2640 Acked Write 7 Done Done Sent false
5537 2641 Acked Read 2 Done Done Sent false
5538 2642 Acked Read 2 Done Done Sent false
5539 2643 Acked Read 6 Done Done Sent false
5540 2644 Acked Flush 0 Done Done Sent false
5541 2645 Acked Write 1 Done Done Sent false
5542 2646 Acked Read 4 Done Done Sent false
5543 2647 Acked Read 4 Done Done Sent false
5544 2648 Acked Read 2 Done Done Sent false
5545 2649 Acked Read 5 Done Done Sent false
5546 2650 Acked Read 6 Done Done Sent false
5547 2651 Acked Write 7 Done Done Sent false
5548 2652 Acked Flush 0 Done Done Sent false
5549 2653 Acked Read 9 Done Done Sent false
5550 2654 Acked Write 3 Done Done Sent false
5551 2655 Acked Read 7 Done Done Sent false
5552 2656 Acked Write 2 Done Done Sent false
5553 2657 Acked Read 9 Done Done Sent false
5554 2658 Acked Write 7 Done Done Sent false
5555 2659 Acked Read 9 Done Done Sent false
5556 2660 Acked Read 5 Done Done Sent false
5557 2661 Acked Read 4 Done Done Sent false
5558 2662 Acked Read 4 Done Done Sent false
5559 2663 Acked Write 1 Done Done Sent false
5560 2664 Acked Read 1 Done Done Sent false
5561 2665 Acked Write 3 Done Done Sent false
5562 2666 Acked Read 9 Done Done Sent false
5563 2667 Acked Read 6 Done Done Sent false
5564 2668 Acked Write 5 Done Done Sent false
5565 2669 Acked Read 10 Done Done Sent false
5566 2670 Acked Read 3 Done Done Sent false
5567 2671 Acked Write 10 Done Done Sent false
5568 2672 Acked Flush 0 Done Done Sent false
5569 2673 Acked Write 7 Done Done Sent false
5570 2674 Acked Read 7 Done Done Sent false
5571 2675 Acked Read 1 Done Done Sent false
5572 2676 Acked Read 7 Done Done Sent false
5573 2677 Acked Read 3 Done Done Sent false
5574 2678 Acked Read 4 Done Done Sent false
5575 2679 Acked Read 10 Done Done Sent false
5576 2680 Acked Write 3 Done Done Sent false
5577 2681 Acked Read 9 Done Done Sent false
5578 2682 Acked Write 3 Done Done Sent false
5579 2683 Acked Read 3 Done Done Sent false
5580 2684 Acked Read 7 Done Done Sent false
5581 2685 Acked Write 8 Done Done Sent false
5582 2686 Acked Read 3 Done Done Sent false
5583 2687 Acked Read 8 Done Done Sent false
5584 2688 Acked Read 7 Done Done Sent false
5585 2689 Acked Read 9 Done Done Sent false
5586 2690 Acked Read 9 Done Done Sent false
5587 2691 Acked Read 2 Done Done Sent false
5588 2692 Acked Read 9 Done Done Sent false
5589 2693 Acked Write 5 Done Done Sent false
5590 2694 Acked Write 6 Done Done Sent false
5591 2695 Acked Write 10 Done Done Sent false
5592 2696 Acked Read 3 Done Done Sent false
5593 2697 Acked Write 9 Done Done Sent false
5594 2698 Acked Flush 0 Done Done Sent false
5595 2699 Acked Flush 0 Done Done Sent false
5596 2700 Acked Write 8 Done Done Sent false
5597 2701 Acked Read 10 Done Done Sent false
5598 2702 Acked Read 5 Done Done Sent false
5599 2703 Acked Read 4 Done Done Sent false
5600 2704 Acked Write 6 Done Done Sent false
5601 2705 Acked Write 10 Done Done Sent false
5602 2706 Acked Read 3 Done Done Sent false
5603 2707 Acked Read 2 Done Done Sent false
5604 2708 Acked Read 4 Done Done Sent false
5605 2709 Acked Read 6 Done Done Sent false
5606 2710 Acked Read 4 Done Done Sent false
5607 2711 Acked Read 6 Done Done Sent false
5608 2712 Acked Read 4 Done Done Sent false
5609 2713 Acked Read 1 Done Done Sent false
5610 2714 Acked Read 2 Done Done Sent false
5611 2715 Acked Read 10 Done Done Sent false
5612 2716 Acked Write 6 Done Done Sent false
5613 2717 Acked Read 6 Done Done Sent false
5614 2718 Acked Write 1 Done Done Sent false
5615 2719 Acked Write 8 Done Done Sent false
5616 2720 Acked Write 4 Done Done Sent false
5617 2721 Acked Write 2 Done Done Sent false
5618 2722 Acked Write 7 Done Done Sent false
5619 2723 Acked Read 1 Done Done Sent false
5620 2724 Acked Read 3 Done Done Sent false
5621 2725 Acked Read 6 Done Done Sent false
5622 2726 Acked Flush 0 Done Done Sent false
5623 2727 Acked Read 7 Done Done Sent false
5624 2728 Acked Flush 0 Done Done Sent false
5625 2729 Acked Read 1 Done Done Sent false
5626 2730 Acked Write 9 Done Done Sent false
5627 2731 Acked Read 1 Done Done Sent false
5628 2732 Acked Read 4 Done Done Sent false
5629 2733 Acked Read 5 Done Done Sent false
5630 2734 Acked Read 8 Done Done Sent false
5631 2735 Acked Read 3 Done Done Sent false
5632 2736 Acked Write 9 Done Done Sent false
5633 2737 Acked Write 8 Done Done Sent false
5634 2738 Acked Write 9 Done Done Sent false
5635 2739 Acked Write 6 Done Done Sent false
5636 2740 Acked Write 9 Done Done Sent false
5637 2741 Acked Flush 0 Done Done Sent false
5638 2742 Acked Read 3 Done Done Sent false
5639 2743 Acked Read 6 Done Done Sent false
5640 2744 Acked Write 4 Done Done Sent false
5641 2745 Acked Read 3 Done Done Sent false
5642 2746 Acked Read 4 Done Done Sent false
5643 2747 Acked Read 6 Done Done Sent false
5644 2748 Acked Write 8 Done Done Sent false
5645 2749 Acked Write 4 Done Done Sent false
5646 2750 Acked Flush 0 Done Done Sent false
5647 2751 Acked Read 6 Done Done Sent false
5648 2752 Acked Read 5 Done Done Sent false
5649 2753 Acked Read 2 Done Done Sent false
5650 2754 Acked Read 7 Done Done Sent false
5651 2755 Acked Read 6 Done Done Sent false
5652 2756 Acked Write 1 Done Done Sent false
5653 2757 Acked Read 7 Done Done Sent false
5654 2758 Acked Read 5 Done Done Sent false
5655 2759 Acked Read 9 Done Done Sent false
5656 2760 Acked Write 3 Done Done Sent false
5657 2761 Acked Read 8 Done Done Sent false
5658 2762 Acked Read 5 Done Done Sent false
5659 2763 Acked Read 10 Done Done Sent false
5660 2764 Acked Write 10 Done Done Sent false
5661 2765 Acked Read 3 Done Done Sent false
5662 2766 Acked Read 10 Done Done Sent false
5663 2767 Acked Read 5 Done Done Sent false
5664 2768 Acked Write 2 Done Done Sent false
5665 2769 Acked Read 6 Done Done Sent false
5666 2770 Acked Write 4 Done Done Sent false
5667 2771 Acked Read 10 Done Done Sent false
5668 2772 Acked Write 8 Done Done Sent false
5669 2773 Acked Write 4 Done Done Sent false
5670 2774 Acked Write 2 Done Done Sent false
5671 2775 Acked Write 3 Done Done Sent false
5672 2776 Acked Write 4 Done Done Sent false
5673 2777 Acked Write 8 Done Done Sent false
5674 2778 Acked Write 7 Done Done Sent false
5675 2779 Acked Read 9 Done Done Sent false
5676 2780 Acked Read 3 Done Done Sent false
5677 2781 Acked Read 7 Done Done Sent false
5678 2782 Acked Read 4 Done Done Sent false
5679 2783 Acked Read 3 Done Done Sent false
5680 2784 Acked Read 6 Done Done Sent false
5681 2785 Acked Read 4 Done Done Sent false
5682 2786 Acked Flush 0 Done Done Sent false
5683 2787 Acked Write 4 Done Done Sent false
5684 2788 Acked Read 10 Done Done Sent false
5685 2789 Acked Read 10 Done Done Sent false
5686 2790 Acked Flush 0 Done Done Sent false
5687 2791 Acked Read 4 Done Done Sent false
5688 2792 Acked Flush 0 Done Done Sent false
5689 2793 Acked Read 6 Done Done Sent false
5690 2794 Acked Flush 0 Done Done Sent false
5691 2795 Acked Read 3 Done Done Sent false
5692 2796 Acked Read 10 Done Done Sent false
5693 2797 Acked Read 1 Done Done Sent false
5694 2798 Acked Write 2 Done Done Sent false
5695 2799 Acked Read 7 Done Done Sent false
5696 2800 Acked Read 9 Done Done Sent false
5697 2801 Acked Write 9 Done Done Sent false
5698 2802 Acked Read 9 Done Done Sent false
5699 2803 Acked Write 6 Done Done Sent false
5700 2804 Acked Flush 0 Done Done Sent false
5701 2805 Acked Read 7 Done Done Sent false
5702 2806 Acked Read 4 Done Done Sent false
5703 2807 Acked Read 2 Done Done Sent false
5704 2808 Acked Write 6 Done Done Sent false
5705 2809 Acked Write 8 Done Done Sent false
5706 2810 Acked Read 3 Done Done Sent false
5707 STATES DS:0 DS:1 DS:2 TOTAL
5708 Sent 0 0 301 301
5709 Done 301 301 0 602
5710 Skipped 0 0 0 0
5711 Error 0 0 0 0
5712 Last Flush: Some(JobId(2804)) Some(JobId(2804)) Some(JobId(2509))
5713 Downstairs last five completed: 2509 2508 2507 2506 2505
5714 Upstairs last five completed: 2810 2809 2808 2807 2806
5715 CLIENT: Up:0 ds:301 act:2
57162025-02-21T06:46:29.220ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
57172025-02-21T06:46:29.758ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
5718 [8830] got start action
5719 [8830] I am going to restart
5720 Starting downstairs at port 8830
5721 Make output file at "/tmp/dsc/downstairs-8830.txt"
5722 [8830][2] reports Starting
5723 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1212)
5724 [8830][2] reports Running
57252025-02-21T06:46:30.759ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
57262025-02-21T06:46:31.760ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
5727 ----------------------------------------------------------------
5728 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5729 States: Active Active Offline
5730 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5731 2510 Acked Read 10 Done Done Sent false
5732 2511 Acked Read 7 Done Done Sent false
5733 2512 Acked Write 1 Done Done Sent false
5734 2513 Acked Read 5 Done Done Sent false
5735 2514 Acked Read 6 Done Done Sent false
5736 2515 Acked Write 1 Done Done Sent false
5737 2516 Acked Write 10 Done Done Sent false
5738 2517 Acked Read 6 Done Done Sent false
5739 2518 Acked Flush 0 Done Done Sent false
5740 2519 Acked Read 1 Done Done Sent false
5741 2520 Acked Flush 0 Done Done Sent false
5742 2521 Acked Write 5 Done Done Sent false
5743 2522 Acked Write 9 Done Done Sent false
5744 2523 Acked Flush 0 Done Done Sent false
5745 2524 Acked Read 6 Done Done Sent false
5746 2525 Acked Write 10 Done Done Sent false
5747 2526 Acked Read 2 Done Done Sent false
5748 2527 Acked Write 2 Done Done Sent false
5749 2528 Acked Write 2 Done Done Sent false
5750 2529 Acked Flush 0 Done Done Sent false
5751 2530 Acked Write 3 Done Done Sent false
5752 2531 Acked Write 8 Done Done Sent false
5753 2532 Acked Read 1 Done Done Sent false
5754 2533 Acked Read 5 Done Done Sent false
5755 2534 Acked Read 4 Done Done Sent false
5756 2535 Acked Read 7 Done Done Sent false
5757 2536 Acked Flush 0 Done Done Sent false
5758 2537 Acked Read 6 Done Done Sent false
5759 2538 Acked Read 1 Done Done Sent false
5760 2539 Acked Write 6 Done Done Sent false
5761 2540 Acked Read 5 Done Done Sent false
5762 2541 Acked Write 5 Done Done Sent false
5763 2542 Acked Write 7 Done Done Sent false
5764 2543 Acked Write 2 Done Done Sent false
5765 2544 Acked Write 2 Done Done Sent false
5766 2545 Acked Write 4 Done Done Sent false
5767 2546 Acked Read 3 Done Done Sent false
5768 2547 Acked Write 10 Done Done Sent false
5769 2548 Acked Write 10 Done Done Sent false
5770 2549 Acked Read 6 Done Done Sent false
5771 2550 Acked Read 5 Done Done Sent false
5772 2551 Acked Read 8 Done Done Sent false
5773 2552 Acked Write 7 Done Done Sent false
5774 2553 Acked Write 4 Done Done Sent false
5775 2554 Acked Read 10 Done Done Sent false
5776 2555 Acked Write 5 Done Done Sent false
5777 2556 Acked Read 8 Done Done Sent false
5778 2557 Acked Read 1 Done Done Sent false
5779 2558 Acked Write 6 Done Done Sent false
5780 2559 Acked Write 8 Done Done Sent false
5781 2560 Acked Read 8 Done Done Sent false
5782 2561 Acked Write 10 Done Done Sent false
5783 2562 Acked Read 2 Done Done Sent false
5784 2563 Acked Write 5 Done Done Sent false
5785 2564 Acked Read 2 Done Done Sent false
5786 2565 Acked Write 10 Done Done Sent false
5787 2566 Acked Write 8 Done Done Sent false
5788 2567 Acked Write 8 Done Done Sent false
5789 2568 Acked Write 9 Done Done Sent false
5790 2569 Acked Flush 0 Done Done Sent false
5791 2570 Acked Write 9 Done Done Sent false
5792 2571 Acked Read 9 Done Done Sent false
5793 2572 Acked Read 6 Done Done Sent false
5794 2573 Acked Write 2 Done Done Sent false
5795 2574 Acked Read 6 Done Done Sent false
5796 2575 Acked Write 10 Done Done Sent false
5797 2576 Acked Read 6 Done Done Sent false
5798 2577 Acked Write 1 Done Done Sent false
5799 2578 Acked Read 6 Done Done Sent false
5800 2579 Acked Read 6 Done Done Sent false
5801 2580 Acked Read 8 Done Done Sent false
5802 2581 Acked Read 9 Done Done Sent false
5803 2582 Acked Write 7 Done Done Sent false
5804 2583 Acked Read 1 Done Done Sent false
5805 2584 Acked Write 2 Done Done Sent false
5806 2585 Acked Flush 0 Done Done Sent false
5807 2586 Acked Read 9 Done Done Sent false
5808 2587 Acked Read 10 Done Done Sent false
5809 2588 Acked Read 8 Done Done Sent false
5810 2589 Acked Write 10 Done Done Sent false
5811 2590 Acked Read 3 Done Done Sent false
5812 2591 Acked Write 6 Done Done Sent false
5813 2592 Acked Read 6 Done Done Sent false
5814 2593 Acked Write 7 Done Done Sent false
5815 2594 Acked Read 2 Done Done Sent false
5816 2595 Acked Read 2 Done Done Sent false
5817 2596 Acked Read 2 Done Done Sent false
5818 2597 Acked Read 10 Done Done Sent false
5819 2598 Acked Read 2 Done Done Sent false
5820 2599 Acked Read 6 Done Done Sent false
5821 2600 Acked Read 2 Done Done Sent false
5822 2601 Acked Read 1 Done Done Sent false
5823 2602 Acked Read 5 Done Done Sent false
5824 2603 Acked Write 8 Done Done Sent false
5825 2604 Acked Read 5 Done Done Sent false
5826 2605 Acked Write 3 Done Done Sent false
5827 2606 Acked Read 5 Done Done Sent false
5828 2607 Acked Write 10 Done Done Sent false
5829 2608 Acked Write 2 Done Done Sent false
5830 2609 Acked Write 9 Done Done Sent false
5831 2610 Acked Read 10 Done Done Sent false
5832 2611 Acked Read 2 Done Done Sent false
5833 2612 Acked Read 3 Done Done Sent false
5834 2613 Acked Write 2 Done Done Sent false
5835 2614 Acked Read 8 Done Done Sent false
5836 2615 Acked Read 4 Done Done Sent false
5837 2616 Acked Read 5 Done Done Sent false
5838 2617 Acked Write 7 Done Done Sent false
5839 2618 Acked Write 4 Done Done Sent false
5840 2619 Acked Read 7 Done Done Sent false
5841 2620 Acked Read 1 Done Done Sent false
5842 2621 Acked Flush 0 Done Done Sent false
5843 2622 Acked Read 5 Done Done Sent false
5844 2623 Acked Flush 0 Done Done Sent false
5845 2624 Acked Read 7 Done Done Sent false
5846 2625 Acked Read 2 Done Done Sent false
5847 2626 Acked Read 4 Done Done Sent false
5848 2627 Acked Read 4 Done Done Sent false
5849 2628 Acked Read 4 Done Done Sent false
5850 2629 Acked Read 3 Done Done Sent false
5851 2630 Acked Write 1 Done Done Sent false
5852 2631 Acked Write 9 Done Done Sent false
5853 2632 Acked Read 7 Done Done Sent false
5854 2633 Acked Write 5 Done Done Sent false
5855 2634 Acked Write 10 Done Done Sent false
5856 2635 Acked Read 4 Done Done Sent false
5857 2636 Acked Read 4 Done Done Sent false
5858 2637 Acked Read 9 Done Done Sent false
5859 2638 Acked Flush 0 Done Done Sent false
5860 2639 Acked Read 9 Done Done Sent false
5861 2640 Acked Write 7 Done Done Sent false
5862 2641 Acked Read 2 Done Done Sent false
5863 2642 Acked Read 2 Done Done Sent false
5864 2643 Acked Read 6 Done Done Sent false
5865 2644 Acked Flush 0 Done Done Sent false
5866 2645 Acked Write 1 Done Done Sent false
5867 2646 Acked Read 4 Done Done Sent false
5868 2647 Acked Read 4 Done Done Sent false
5869 2648 Acked Read 2 Done Done Sent false
5870 2649 Acked Read 5 Done Done Sent false
5871 2650 Acked Read 6 Done Done Sent false
5872 2651 Acked Write 7 Done Done Sent false
5873 2652 Acked Flush 0 Done Done Sent false
5874 2653 Acked Read 9 Done Done Sent false
5875 2654 Acked Write 3 Done Done Sent false
5876 2655 Acked Read 7 Done Done Sent false
5877 2656 Acked Write 2 Done Done Sent false
5878 2657 Acked Read 9 Done Done Sent false
5879 2658 Acked Write 7 Done Done Sent false
5880 2659 Acked Read 9 Done Done Sent false
5881 2660 Acked Read 5 Done Done Sent false
5882 2661 Acked Read 4 Done Done Sent false
5883 2662 Acked Read 4 Done Done Sent false
5884 2663 Acked Write 1 Done Done Sent false
5885 2664 Acked Read 1 Done Done Sent false
5886 2665 Acked Write 3 Done Done Sent false
5887 2666 Acked Read 9 Done Done Sent false
5888 2667 Acked Read 6 Done Done Sent false
5889 2668 Acked Write 5 Done Done Sent false
5890 2669 Acked Read 10 Done Done Sent false
5891 2670 Acked Read 3 Done Done Sent false
5892 2671 Acked Write 10 Done Done Sent false
5893 2672 Acked Flush 0 Done Done Sent false
5894 2673 Acked Write 7 Done Done Sent false
5895 2674 Acked Read 7 Done Done Sent false
5896 2675 Acked Read 1 Done Done Sent false
5897 2676 Acked Read 7 Done Done Sent false
5898 2677 Acked Read 3 Done Done Sent false
5899 2678 Acked Read 4 Done Done Sent false
5900 2679 Acked Read 10 Done Done Sent false
5901 2680 Acked Write 3 Done Done Sent false
5902 2681 Acked Read 9 Done Done Sent false
5903 2682 Acked Write 3 Done Done Sent false
5904 2683 Acked Read 3 Done Done Sent false
5905 2684 Acked Read 7 Done Done Sent false
5906 2685 Acked Write 8 Done Done Sent false
5907 2686 Acked Read 3 Done Done Sent false
5908 2687 Acked Read 8 Done Done Sent false
5909 2688 Acked Read 7 Done Done Sent false
5910 2689 Acked Read 9 Done Done Sent false
5911 2690 Acked Read 9 Done Done Sent false
5912 2691 Acked Read 2 Done Done Sent false
5913 2692 Acked Read 9 Done Done Sent false
5914 2693 Acked Write 5 Done Done Sent false
5915 2694 Acked Write 6 Done Done Sent false
5916 2695 Acked Write 10 Done Done Sent false
5917 2696 Acked Read 3 Done Done Sent false
5918 2697 Acked Write 9 Done Done Sent false
5919 2698 Acked Flush 0 Done Done Sent false
5920 2699 Acked Flush 0 Done Done Sent false
5921 2700 Acked Write 8 Done Done Sent false
5922 2701 Acked Read 10 Done Done Sent false
5923 2702 Acked Read 5 Done Done Sent false
5924 2703 Acked Read 4 Done Done Sent false
5925 2704 Acked Write 6 Done Done Sent false
5926 2705 Acked Write 10 Done Done Sent false
5927 2706 Acked Read 3 Done Done Sent false
5928 2707 Acked Read 2 Done Done Sent false
5929 2708 Acked Read 4 Done Done Sent false
5930 2709 Acked Read 6 Done Done Sent false
5931 2710 Acked Read 4 Done Done Sent false
5932 2711 Acked Read 6 Done Done Sent false
5933 2712 Acked Read 4 Done Done Sent false
5934 2713 Acked Read 1 Done Done Sent false
5935 2714 Acked Read 2 Done Done Sent false
5936 2715 Acked Read 10 Done Done Sent false
5937 2716 Acked Write 6 Done Done Sent false
5938 2717 Acked Read 6 Done Done Sent false
5939 2718 Acked Write 1 Done Done Sent false
5940 2719 Acked Write 8 Done Done Sent false
5941 2720 Acked Write 4 Done Done Sent false
5942 2721 Acked Write 2 Done Done Sent false
5943 2722 Acked Write 7 Done Done Sent false
5944 2723 Acked Read 1 Done Done Sent false
5945 2724 Acked Read 3 Done Done Sent false
5946 2725 Acked Read 6 Done Done Sent false
5947 2726 Acked Flush 0 Done Done Sent false
5948 2727 Acked Read 7 Done Done Sent false
5949 2728 Acked Flush 0 Done Done Sent false
5950 2729 Acked Read 1 Done Done Sent false
5951 2730 Acked Write 9 Done Done Sent false
5952 2731 Acked Read 1 Done Done Sent false
5953 2732 Acked Read 4 Done Done Sent false
5954 2733 Acked Read 5 Done Done Sent false
5955 2734 Acked Read 8 Done Done Sent false
5956 2735 Acked Read 3 Done Done Sent false
5957 2736 Acked Write 9 Done Done Sent false
5958 2737 Acked Write 8 Done Done Sent false
5959 2738 Acked Write 9 Done Done Sent false
5960 2739 Acked Write 6 Done Done Sent false
5961 2740 Acked Write 9 Done Done Sent false
5962 2741 Acked Flush 0 Done Done Sent false
5963 2742 Acked Read 3 Done Done Sent false
5964 2743 Acked Read 6 Done Done Sent false
5965 2744 Acked Write 4 Done Done Sent false
5966 2745 Acked Read 3 Done Done Sent false
5967 2746 Acked Read 4 Done Done Sent false
5968 2747 Acked Read 6 Done Done Sent false
5969 2748 Acked Write 8 Done Done Sent false
5970 2749 Acked Write 4 Done Done Sent false
5971 2750 Acked Flush 0 Done Done Sent false
5972 2751 Acked Read 6 Done Done Sent false
5973 2752 Acked Read 5 Done Done Sent false
5974 2753 Acked Read 2 Done Done Sent false
5975 2754 Acked Read 7 Done Done Sent false
5976 2755 Acked Read 6 Done Done Sent false
5977 2756 Acked Write 1 Done Done Sent false
5978 2757 Acked Read 7 Done Done Sent false
5979 2758 Acked Read 5 Done Done Sent false
5980 2759 Acked Read 9 Done Done Sent false
5981 2760 Acked Write 3 Done Done Sent false
5982 2761 Acked Read 8 Done Done Sent false
5983 2762 Acked Read 5 Done Done Sent false
5984 2763 Acked Read 10 Done Done Sent false
5985 2764 Acked Write 10 Done Done Sent false
5986 2765 Acked Read 3 Done Done Sent false
5987 2766 Acked Read 10 Done Done Sent false
5988 2767 Acked Read 5 Done Done Sent false
5989 2768 Acked Write 2 Done Done Sent false
5990 2769 Acked Read 6 Done Done Sent false
5991 2770 Acked Write 4 Done Done Sent false
5992 2771 Acked Read 10 Done Done Sent false
5993 2772 Acked Write 8 Done Done Sent false
5994 2773 Acked Write 4 Done Done Sent false
5995 2774 Acked Write 2 Done Done Sent false
5996 2775 Acked Write 3 Done Done Sent false
5997 2776 Acked Write 4 Done Done Sent false
5998 2777 Acked Write 8 Done Done Sent false
5999 2778 Acked Write 7 Done Done Sent false
6000 2779 Acked Read 9 Done Done Sent false
6001 2780 Acked Read 3 Done Done Sent false
6002 2781 Acked Read 7 Done Done Sent false
6003 2782 Acked Read 4 Done Done Sent false
6004 2783 Acked Read 3 Done Done Sent false
6005 2784 Acked Read 6 Done Done Sent false
6006 2785 Acked Read 4 Done Done Sent false
6007 2786 Acked Flush 0 Done Done Sent false
6008 2787 Acked Write 4 Done Done Sent false
6009 2788 Acked Read 10 Done Done Sent false
6010 2789 Acked Read 10 Done Done Sent false
6011 2790 Acked Flush 0 Done Done Sent false
6012 2791 Acked Read 4 Done Done Sent false
6013 2792 Acked Flush 0 Done Done Sent false
6014 2793 Acked Read 6 Done Done Sent false
6015 2794 Acked Flush 0 Done Done Sent false
6016 2795 Acked Read 3 Done Done Sent false
6017 2796 Acked Read 10 Done Done Sent false
6018 2797 Acked Read 1 Done Done Sent false
6019 2798 Acked Write 2 Done Done Sent false
6020 2799 Acked Read 7 Done Done Sent false
6021 2800 Acked Read 9 Done Done Sent false
6022 2801 Acked Write 9 Done Done Sent false
6023 2802 Acked Read 9 Done Done Sent false
6024 2803 Acked Write 6 Done Done Sent false
6025 2804 Acked Flush 0 Done Done Sent false
6026 2805 Acked Read 7 Done Done Sent false
6027 2806 Acked Read 4 Done Done Sent false
6028 2807 Acked Read 2 Done Done Sent false
6029 2808 Acked Write 6 Done Done Sent false
6030 2809 Acked Write 8 Done Done Sent false
6031 2810 Acked Read 3 Done Done Sent false
6032 STATES DS:0 DS:1 DS:2 TOTAL
6033 Sent 0 0 301 301
6034 Done 301 301 0 602
6035 Skipped 0 0 0 0
6036 Error 0 0 0 0
6037 Last Flush: Some(JobId(2804)) Some(JobId(2804)) Some(JobId(2509))
6038 Downstairs last five completed: 2509 2508 2507 2506 2505
6039 Upstairs last five completed: 2810 2809 2808 2807 2806
6040 CLIENT: Up:0 ds:301 act:2
60412025-02-21T06:46:33.222ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60422025-02-21T06:46:33.762ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60432025-02-21T06:46:34.464ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60442025-02-21T06:46:34.464ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46894) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60452025-02-21T06:46:34.465ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60462025-02-21T06:46:34.466ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60472025-02-21T06:46:34.466ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60482025-02-21T06:46:34.466ZINFOcrucible: send last flush ID to this DS: Some(JobId(2509)) = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60492025-02-21T06:46:34.466ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60502025-02-21T06:46:34.466ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(2509)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60512025-02-21T06:46:34.466ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(2509)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
6052 ----------------------------------------------------------------
6053 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6054 Downstairs last five completed: 2811 2810 2809 2808 2807
6055 Upstairs last five completed: 2811 2810 2809 2808 2807
6056 CLIENT: Up:0 ds:0 act:3
6057 Replay: All jobs finished
6058 Main task has work to do, go find it
6059 got dsc Stop(0)
6060 stop 0
6061 Feb 21 06:46:36.723 INFO request completed, latency_us: 389, response_code[8810] Got stop action so:false kr:false
6062 : 204, uri: /stop/cid/0, method: POST, req_id: b963f0e2-9356-4138-85ab-b01e9a49b5e3, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
60632025-02-21T06:46:36.724ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60642025-02-21T06:46:36.724ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60652025-02-21T06:46:36.724ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
60662025-02-21T06:46:36.724ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
6067 Feb 21 06:46:36.724 INFO request completed, latency_us: 139, response_code: 200, uri: /state/cid/0, method: GET, req_id: a2ee78a8-6145-4cfb-b3fe-b1f3a1521abd, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
6068 [8810] Exited with: ExitStatus(unix_wait_status(9))
6069 [8810][0] reports Exit
6070 Feb 21 06:46:40.726 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/0, method: GET, req_id: 362645c4-df63-43c2-87a5-1a226eb7a4be, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
6071 001/300 Write block 785 len 32768 data: 4 3 3 3 3 3 3 2
6072 002/300 Write block 4960 len 20480 data: 3 3 3 3 3
6073 003/300 Write block 672 len 4096 data: 3
6074 004/300 Write block 1418 len 28672 data: 3 3 3 3 3 4 4
6075 005/300 Read block 4605 len 24576
6076 006/300 Read block 1500 len 12288
6077 007/300 Write block 1018 len 16384 data: 4 4 4 4
6078 008/300 Read block 224 len 12288
6079 009/300 Read block 2269 len 8192
6080 010/300 Flush
6081 011/300 Read block 4386 len 24576
6082 012/300 Read block 4318 len 12288
6083 013/300 Read block 1066 len 36864
6084 014/300 Read block 4823 len 12288
6085 015/300 Read block 4429 len 32768
6086 016/300 Write block 3547 len 36864 data: 2 2 2 2 2 2 2 2 2
6087 017/300 Write block 2679 len 28672 data: 2 2 3 2 2 2 2
6088 018/300 Write block 2176 len 8192 data: 2 2
6089 019/300 Read block 1017 len 16384
6090 020/300 Write block 1810 len 12288 data: 2 5 4
6091 021/300 Read block 4937 len 28672
6092 022/300 Read block 4182 len 4096
6093 023/300 Read block 4025 len 32768
6094 024/300 Write block 3936 len 20480 data: 3 3 3 3 4
6095 025/300 Read block 2914 len 4096
6096 026/300 Write block 4464 len 40960 data: 2 2 2 2 2 3 3 3 3 3
6097 027/300 Write block 4016 len 16384 data: 3 3 3 3
6098 028/300 Write block 1426 len 32768 data: 3 3 2 2 2 2 2 3
6099 029/300 Read block 2782 len 8192
6100 030/300 Write block 2895 len 24576 data: 2 2 2 2 2 2
6101 031/300 Write block 1673 len 32768 data: 3 3 3 3 3 2 2 2
6102 032/300 Read block 3369 len 12288
6103 033/300 Write block 3625 len 36864 data: 4 4 4 4 3 3 4 5 5
6104 034/300 Read block 3021 len 8192
6105 035/300 Write block 734 len 28672 data: 2 2 3 3 3 3 2
6106 036/300 Read block 4390 len 24576
6107 037/300 Read block 2694 len 40960
6108 038/300 Read block 3898 len 12288
6109 039/300 Read block 3039 len 20480
6110 040/300 Read block 3971 len 36864
6111 041/300 Write block 4928 len 40960 data: 2 2 2 2 2 2 3 3 3 3
6112 042/300 Flush
6113 043/300 Write block 431 len 12288 data: 4 4 3
6114 044/300 Read block 879 len 32768
6115 045/300 Write block 3701 len 36864 data: 3 3 3 2 2 2 2 2 2
6116 046/300 Read block 3761 len 40960
6117 047/300 Read block 555 len 4096
6118 048/300 Write block 1490 len 20480 data: 3 3 3 3 3
6119 049/300 Read block 2902 len 24576
6120 050/300 Write block 1362 len 36864 data: 2 2 2 2 2 2 2 2 2
6121 051/300 Write block 1045 len 8192 data: 3 3
6122 052/300 Flush
6123 053/300 Read block 2153 len 20480
6124 054/300 Write block 3171 len 28672 data: 3 3 4 4 5 4 4
6125 055/300 Read block 2330 len 36864
6126 056/300 Write block 4476 len 36864 data: 3 3 3 3 2 2 2 2 2
6127 057/300 Write block 1768 len 12288 data: 3 2 2
6128 058/300 Write block 4389 len 36864 data: 3 2 3 3 3 3 3 3 3
6129 059/300 Read block 2053 len 16384
6130 060/300 Read block 313 len 4096
6131 061/300 Read block 2868 len 32768
6132 062/300 Write block 4144 len 16384 data: 2 2 2 2
6133 063/300 Write block 1860 len 28672 data: 2 2 3 3 3 3 2
6134 064/300 Write block 4452 len 12288 data: 3 3 3
6135 065/300 Read block 585 len 20480
6136 066/300 Read block 3121 len 20480
6137 067/300 Read block 1806 len 12288
6138 068/300 Write block 254 len 4096 data: 2
6139 069/300 Write block 4610 len 20480 data: 3 2 2 2 3
6140 070/300 Write block 2214 len 28672 data: 4 5 5 5 4 4 4
6141 071/300 Write block 4601 len 4096 data: 4
6142 072/300 Read block 68 len 36864
6143 073/300 Read block 2275 len 36864
6144 074/300 Write block 2001 len 32768 data: 3 3 4 4 3 3 3 3
6145 075/300 Read block 1626 len 16384
6146 076/300 Write block 2180 len 4096 data: 2
6147 077/300 Read block 199 len 20480
6148 078/300 Write block 1981 len 16384 data: 2 2 2 4
6149 079/300 Read block 4280 len 8192
6150 080/300 Read block 3910 len 20480
6151 081/300 Read block 3735 len 24576
6152 082/300 Read block 3786 len 12288
6153 083/300 Flush
6154 084/300 Write block 4121 len 20480 data: 3 4 3 2 2
6155 085/300 Flush
6156 086/300 Write block 1106 len 8192 data: 3 2
6157 087/300 Read block 1810 len 24576
6158 088/300 Write block 117 len 36864 data: 2 2 2 2 2 2 2 2 2
6159 089/300 Write block 1046 len 4096 data: 4
6160 090/300 Write block 4644 len 36864 data: 2 3 3 5 5 5 5 4 4
6161 091/300 Read block 3650 len 24576
6162 092/300 Write block 2603 len 16384 data: 3 4 4 3
6163 093/300 Read block 4154 len 16384
6164 094/300 Read block 2624 len 32768
6165 095/300 Write block 181 len 4096 data: 3
6166 096/300 Write block 2353 len 16384 data: 3 3 2 3
6167 097/300 Write block 646 len 28672 data: 3 3 3 3 3 3 3
6168 098/300 Write block 71 len 40960 data: 2 2 2 3 3 3 3 3 3 2
6169 099/300 Write block 3867 len 36864 data: 4 3 2 2 2 3 4 3 3
6170 100/300 Write block 4970 len 16384 data: 2 2 2 3
6171 101/300 Read block 4560 len 20480
6172 102/300 Read block 1623 len 32768
6173 103/300 Write block 4896 len 32768 data: 3 4 4 4 4 4 3 2
6174 104/300 Write block 2702 len 24576 data: 2 2 2 2 2 2
6175 105/300 Read block 3055 len 28672
6176 106/300 Read block 3035 len 24576
6177 107/300 Read block 1988 len 36864
6178 108/300 Read block 3601 len 12288
6179 109/300 Read block 4950 len 28672
6180 110/300 Read block 1341 len 20480
6181 111/300 Write block 1212 len 8192 data: 4 4
6182 112/300 Read block 1317 len 8192
6183 113/300 Write block 905 len 24576 data: 4 3 2 2 2 2
6184 114/300 Write block 2214 len 12288 data: 5 6 6
6185 115/300 Read block 1948 len 40960
6186 116/300 Write block 179 len 40960 data: 3 3 4 3 3 4 4 3 3 3
6187 117/300 Read block 1499 len 12288
6188 118/300 Read block 349 len 32768
6189 119/300 Flush
6190 120/300 Write block 3042 len 36864 data: 2 2 2 3 2 2 3 2 2
6191 121/300 Read block 2492 len 40960
6192 122/300 Read block 3840 len 8192
6193 123/300 Write block 392 len 20480 data: 2 2 2 2 2
6194 124/300 Read block 1547 len 40960
6195 125/300 Write block 3166 len 4096 data: 5
6196 126/300 Read block 4958 len 36864
6197 127/300 Write block 665 len 16384 data: 4 4 4 4
6198 128/300 Write block 4602 len 32768 data: 4 3 3 3 3 3 3 2
6199 129/300 Write block 2254 len 16384 data: 2 3 2 2
6200 130/300 Flush
6201 131/300 Read block 2923 len 36864
6202 132/300 Read block 2587 len 12288
6203 133/300 Read block 3818 len 24576
6204 134/300 Write block 1053 len 32768 data: 4 5 4 4 4 5 4 5
6205 135/300 Flush
6206 136/300 Read block 293 len 40960
6207 137/300 Write block 3861 len 20480 data: 3 4 4 3 3
6208 138/300 Read block 1138 len 12288
6209 139/300 Write block 2307 len 8192 data: 3 2
6210 140/300 Write block 2066 len 36864 data: 3 3 3 3 3 3 3 2 2
6211 141/300 Read block 2483 len 12288
6212 142/300 Read block 947 len 24576
6213 143/300 Write block 448 len 8192 data: 2 2
6214 144/300 Write block 2293 len 12288 data: 2 2 2
6215 145/300 Read block 4645 len 4096
6216 146/300 Read block 3217 len 16384
6217 147/300 Read block 1256 len 20480
6218 148/300 Write block 4913 len 20480 data: 2 3 4 5 6
6219 149/300 Write block 2374 len 20480 data: 3 3 3 2 2
6220 150/300 Write block 889 len 20480 data: 2 3 3 3 3
6221 151/300 Read block 1053 len 28672
6222 152/300 Write block 2066 len 16384 data: 4 4 4 4
6223 153/300 Flush
6224 154/300 Read block 4044 len 16384
6225 155/300 Flush
6226 156/300 Flush
6227 157/300 Write block 4197 len 12288 data: 2 2 2
6228 158/300 Write block 2649 len 4096 data: 2
6229 159/300 Read block 4816 len 12288
6230 160/300 Write block 2005 len 8192 data: 4 4
6231 161/300 Read block 3922 len 24576
6232 162/300 Flush
6233 163/300 Read block 4069 len 12288
6234 164/300 Write block 3772 len 16384 data: 5 5 4 3
6235 165/300 Read block 2765 len 32768
6236 166/300 Read block 3498 len 12288
6237 167/300 Write block 3552 len 16384 data: 3 3 3 3
6238 168/300 Read block 557 len 16384
6239 169/300 Write block 3461 len 12288 data: 2 2 3
6240 170/300 Read block 856 len 4096
6241 171/300 Read block 1332 len 36864
6242 172/300 Read block 4521 len 4096
6243 173/300 Read block 2137 len 24576
6244 174/300 Read block 3089 len 32768
6245 175/300 Read block 904 len 4096
6246 176/300 Read block 3689 len 8192
6247 177/300 Read block 4615 len 40960
6248 178/300 Read block 4553 len 36864
6249 179/300 Read block 2276 len 40960
6250 180/300 Write block 2883 len 16384 data: 3 3 4 5
6251 181/300 Write block 2976 len 12288 data: 3 3 3
6252 182/300 Read block 4624 len 16384
6253 183/300 Write block 1068 len 32768 data: 3 3 3 3 3 3 3 3
6254 184/300 Read block 2023 len 36864
6255 185/300 Write block 4810 len 40960 data: 2 2 2 2 3 3 3 3 4 4
6256 186/300 Flush
6257 187/300 Write block 4053 len 36864 data: 2 3 4 4 4 4 4 4 4
6258 188/300 Read block 4193 len 12288
6259 189/300 Write block 390 len 8192 data: 2 2
6260 190/300 Read block 4668 len 12288
6261 191/300 Read block 3052 len 36864
6262 192/300 Write block 3815 len 28672 data: 3 4 5 4 4 5 6
6263 193/300 Read block 791 len 8192
6264 194/300 Write block 304 len 12288 data: 2 2 2
6265 195/300 Flush
6266 196/300 Write block 4519 len 24576 data: 2 2 2 2 2 2
6267 197/300 Read block 3786 len 40960
6268 198/300 Write block 1813 len 20480 data: 5 5 6 6 5
6269 199/300 Write block 3076 len 8192 data: 2 2
6270 200/300 Read block 1253 len 28672
6271 201/300 Write block 349 len 28672 data: 2 2 2 2 2 2 2
6272 202/300 Write block 3564 len 36864 data: 2 2 2 2 2 2 2 2 2
6273 203/300 Flush
6274 204/300 Read block 1643 len 36864
6275 205/300 Read block 1747 len 24576
6276 206/300 Read block 3489 len 32768
6277 207/300 Write block 2450 len 8192 data: 2 2
6278 208/300 Read block 246 len 8192
6279 209/300 Read block 1401 len 16384
6280 210/300 Write block 716 len 36864 data: 2 2 2 2 2 2 2 3 3
6281 211/300 Write block 2234 len 8192 data: 3 2
6282 212/300 Read block 4498 len 16384
6283 213/300 Write block 3065 len 8192 data: 2 2
6284 214/300 Read block 4439 len 28672
6285 215/300 Write block 1370 len 20480 data: 3 2 2 2 3
6286 216/300 Write block 4090 len 16384 data: 2 2 3 4
6287 217/300 Read block 4951 len 24576
6288 218/300 Read block 2123 len 4096
6289 219/300 Read block 574 len 32768
6290 220/300 Read block 3524 len 20480
6291 221/300 Read block 3785 len 20480
6292 222/300 Write block 1217 len 40960 data: 3 3 3 3 3 3 3 3 3 2
6293 223/300 Read block 3371 len 4096
6294 224/300 Write block 2896 len 8192 data: 3 3
6295 225/300 Flush
6296 226/300 Read block 3822 len 32768
6297 227/300 Read block 3411 len 40960
6298 228/300 Read block 400 len 40960
6299 229/300 Read block 4582 len 32768
6300 230/300 Write block 1720 len 40960 data: 2 3 3 3 3 2 2 2 3 3
6301 231/300 Write block 13 len 16384 data: 2 2 2 2
6302 232/300 Read block 4178 len 32768
6303 233/300 Flush
6304 234/300 Read block 2209 len 36864
6305 235/300 Write block 1986 len 24576 data: 6 6 6 5 4 3
6306 236/300 Read block 692 len 28672
6307 237/300 Write block 927 len 12288 data: 3 3 3
6308 238/300 Write block 2275 len 32768 data: 4 4 5 4 4 4 4 5
6309 239/300 Write block 2980 len 32768 data: 3 3 3 3 3 3 4 4
6310 240/300 Read block 2327 len 32768
6311 241/300 Write block 42 len 20480 data: 2 2 2 2 2
6312 242/300 Read block 466 len 36864
6313 243/300 Read block 3129 len 16384
6314 244/300 Write block 3506 len 12288 data: 2 2 3
6315 245/300 Read block 3995 len 16384
6316 246/300 Write block 646 len 8192 data: 4 4
6317 247/300 Flush
6318 248/300 Read block 1873 len 40960
6319 249/300 Read block 4521 len 20480
6320 250/300 Write block 1824 len 4096 data: 3
6321 251/300 Write block 2201 len 32768 data: 2 2 2 2 2 2 2 2
6322 252/300 Write block 1882 len 28672 data: 3 3 3 3 3 3 3
6323 253/300 Write block 4906 len 8192 data: 3 2
6324 254/300 Read block 1259 len 24576
6325 255/300 Write block 3031 len 20480 data: 3 3 4 4 4
6326 256/300 Write block 4303 len 12288 data: 2 2 3
6327 257/300 Read block 4056 len 8192
6328 258/300 Read block 285 len 32768
6329 259/300 Write block 1533 len 20480 data: 4 3 3 3 3
6330 260/300 Read block 2781 len 24576
6331 261/300 Flush
6332 262/300 Write block 3877 len 36864 data: 3 3 3 2 2 2 2 3 4
6333 263/300 Read block 4591 len 40960
6334 264/300 Read block 1331 len 4096
6335 265/300 Read block 2232 len 36864
6336 266/300 Read block 2819 len 16384
6337 267/300 Read block 4422 len 32768
6338 268/300 Read block 759 len 12288
6339 269/300 Flush
6340 270/300 Write block 520 len 16384 data: 2 2 2 2
6341 271/300 Write block 1108 len 8192 data: 2 3
6342 272/300 Write block 3712 len 20480 data: 2 2 2 2 2
6343 273/300 Read block 1741 len 8192
6344 274/300 Read block 3726 len 8192
6345 275/300 Write block 1134 len 24576 data: 3 3 3 3 3 3
6346 276/300 Read block 75 len 8192
6347 277/300 Flush
6348 278/300 Read block 2610 len 8192
6349 279/300 Write block 453 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6350 280/300 Read block 3736 len 16384
6351 281/300 Write block 3584 len 40960 data: 7 8 7 5 4 3 3 2 2 2
6352 282/300 Read block 4415 len 16384
6353 283/300 Read block 484 len 12288
6354 284/300 Read block 3484 len 28672
6355 285/300 Read block 3182 len 32768
6356 286/300 Write block 3842 len 36864 data: 6 6 5 4 4 4 3 3 3
6357 287/300 Read block 2550 len 12288
6358 288/300 Write block 4888 len 20480 data: 2 2 2 2 2
6359 289/300 Read block 1686 len 24576
6360 290/300 Read block 3865 len 32768
6361 291/300 Write block 4430 len 16384 data: 2 2 2 2
6362 292/300 Flush
6363 293/300 Flush
6364 294/300 Flush
6365 295/300 Write block 1976 len 24576 data: 2 2 2 2 2 3
6366 296/300 Flush
6367 297/300 Read block 351 len 36864
6368 298/300 Read block 823 len 12288
6369 299/300 Write block 4740 len 12288 data: 4 4 6
6370 300/300 Read block 4605 len 28672
6371 301/300 Read block 4157 len 28672
6372 Main task has work to do, go find it
6373 got dsc Start(0)
6374 start 0
6375 Feb 21 06:46:40.968 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/0, method: POST, req_id: 322248f2-a719-402f-81ab-3de2c6660166, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
6376 Replay: started 0, returned:Ok(())
6377 ----------------------------------------------------------------
6378 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6379 States: Offline Active Active
6380 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6381 2812 Acked Write 8 Sent Done Done false
6382 2813 Acked Write 5 Sent Done Done false
6383 2814 Acked Write 1 Sent Done Done false
6384 2815 Acked Write 7 Sent Done Done false
6385 2816 Acked Read 6 Sent Done Done false
6386 2817 Acked Read 3 Sent Done Done false
6387 2818 Acked Write 4 Sent Done Done false
6388 2819 Acked Read 3 Sent Done Done false
6389 2820 Acked Read 2 Sent Done Done false
6390 2821 Acked Flush 0 Sent Done Done false
6391 2822 Acked Read 6 Sent Done Done false
6392 2823 Acked Read 3 Sent Done Done false
6393 2824 Acked Read 9 Sent Done Done false
6394 2825 Acked Read 3 Sent Done Done false
6395 2826 Acked Read 8 Sent Done Done false
6396 2827 Acked Write 9 Sent Done Done false
6397 2828 Acked Write 7 Sent Done Done false
6398 2829 Acked Write 2 Sent Done Done false
6399 2830 Acked Read 4 Sent Done Done false
6400 2831 Acked Write 3 Sent Done Done false
6401 2832 Acked Read 7 Sent Done Done false
6402 2833 Acked Read 1 Sent Done Done false
6403 2834 Acked Read 8 Sent Done Done false
6404 2835 Acked Write 5 Sent Done Done false
6405 2836 Acked Read 1 Sent Done Done false
6406 2837 Acked Write 10 Sent Done Done false
6407 2838 Acked Write 4 Sent Done Done false
6408 2839 Acked Write 8 Sent Done Done false
6409 2840 Acked Read 2 Sent Done Done false
6410 2841 Acked Write 6 Sent Done Done false
6411 2842 Acked Write 8 Sent Done Done false
6412 2843 Acked Read 3 Sent Done Done false
6413 2844 Acked Write 9 Sent Done Done false
6414 2845 Acked Read 2 Sent Done Done false
6415 2846 Acked Write 7 Sent Done Done false
6416 2847 Acked Read 6 Sent Done Done false
6417 2848 Acked Read 10 Sent Done Done false
6418 2849 Acked Read 3 Sent Done Done false
6419 2850 Acked Read 5 Sent Done Done false
6420 2851 Acked Read 9 Sent Done Done false
6421 2852 Acked Write 10 Sent Done Done false
6422 2853 Acked Flush 0 Sent Done Done false
6423 2854 Acked Write 3 Sent Done Done false
6424 2855 Acked Read 8 Sent Done Done false
6425 2856 Acked Write 9 Sent Done Done false
6426 2857 Acked Read 10 Sent Done Done false
6427 2858 Acked Read 1 Sent Done Done false
6428 2859 Acked Write 5 Sent Done Done false
6429 2860 Acked Read 6 Sent Done Done false
6430 2861 Acked Write 9 Sent Done Done false
6431 2862 Acked Write 2 Sent Done Done false
6432 2863 Acked Flush 0 Sent Done Done false
6433 2864 Acked Read 5 Sent Done Done false
6434 2865 Acked Write 7 Sent Done Done false
6435 2866 Acked Read 9 Sent Done Done false
6436 2867 Acked Write 9 Sent Done Done false
6437 2868 Acked Write 3 Sent Done Done false
6438 2869 Acked Write 9 Sent Done Done false
6439 2870 Acked Read 4 Sent Done Done false
6440 2871 Acked Read 1 Sent Done Done false
6441 2872 Acked Read 8 Sent Done Done false
6442 2873 Acked Write 4 Sent Done Done false
6443 2874 Acked Write 7 Sent Done Done false
6444 2875 Acked Write 3 Sent Done Done false
6445 2876 Acked Read 5 Sent Done Done false
6446 2877 Acked Read 5 Sent Done Done false
6447 2878 Acked Read 3 Sent Done Done false
6448 2879 Acked Write 1 Sent Done Done false
6449 2880 Acked Write 5 Sent Done Done false
6450 2881 Acked Write 7 Sent Done Done false
6451 2882 Acked Write 1 Sent Done Done false
6452 2883 Acked Read 9 Sent Done Done false
6453 2884 Acked Read 9 Sent Done Done false
6454 2885 Acked Write 8 Sent Done Done false
6455 2886 Acked Read 4 Sent Done Done false
6456 2887 Acked Write 1 Sent Done Done false
6457 2888 Acked Read 5 Sent Done Done false
6458 2889 Acked Write 4 Sent Done Done false
6459 2890 Acked Read 2 Sent Done Done false
6460 2891 Acked Read 5 Sent Done Done false
6461 2892 Acked Read 6 Sent Done Done false
6462 2893 Acked Read 3 Sent Done Done false
6463 2894 Acked Flush 0 Sent Done Done false
6464 2895 Acked Write 5 Sent Done Done false
6465 2896 Acked Flush 0 Sent Done Done false
6466 2897 Acked Write 2 Sent Done Done false
6467 2898 Acked Read 6 Sent Done Done false
6468 2899 Acked Write 9 Sent Done Done false
6469 2900 Acked Write 1 Sent Done Done false
6470 2901 Acked Write 9 Sent Done Done false
6471 2902 Acked Read 6 Sent Done Done false
6472 2903 Acked Write 4 Sent Done Done false
6473 2904 Acked Read 4 Sent Done Done false
6474 2905 Acked Read 8 Sent Done Done false
6475 2906 Acked Write 1 Sent Done Done false
6476 2907 Acked Write 4 Sent Done Done false
6477 2908 Acked Write 7 Sent Done Done false
6478 2909 Acked Write 10 Sent Done Done false
6479 2910 Acked Write 9 Sent Done Done false
6480 2911 Acked Write 4 Sent Done Done false
6481 2912 Acked Read 5 Sent Done Done false
6482 2913 Acked Read 8 Sent Done Done false
6483 2914 Acked Write 8 Sent Done Done false
6484 2915 Acked Write 6 Sent Done Done false
6485 2916 Acked Read 7 Sent Done Done false
6486 2917 Acked Read 6 Sent Done Done false
6487 2918 Acked Read 9 Sent Done Done false
6488 2919 Acked Read 3 Sent Done Done false
6489 2920 Acked Read 7 Sent Done Done false
6490 2921 Acked Read 5 Sent Done Done false
6491 2922 Acked Write 2 Sent Done Done false
6492 2923 Acked Read 2 Sent Done Done false
6493 2924 Acked Write 6 Sent Done Done false
6494 2925 Acked Write 3 Sent Done Done false
6495 2926 Acked Read 10 Sent Done Done false
6496 2927 Acked Write 10 Sent Done Done false
6497 2928 Acked Read 3 Sent Done Done false
6498 2929 Acked Read 8 Sent Done Done false
6499 2930 Acked Flush 0 Sent Done Done false
6500 2931 Acked Write 9 Sent Done Done false
6501 2932 Acked Read 10 Sent Done Done false
6502 2933 Acked Read 2 Sent Done Done false
6503 2934 Acked Write 5 Sent Done Done false
6504 2935 Acked Read 10 Sent Done Done false
6505 2936 Acked Write 1 Sent Done Done false
6506 2937 Acked Read 9 Sent Done Done false
6507 2938 Acked Write 4 Sent Done Done false
6508 2939 Acked Write 8 Sent Done Done false
6509 2940 Acked Write 4 Sent Done Done false
6510 2941 Acked Flush 0 Sent Done Done false
6511 2942 Acked Read 9 Sent Done Done false
6512 2943 Acked Read 3 Sent Done Done false
6513 2944 Acked Read 6 Sent Done Done false
6514 2945 Acked Write 8 Sent Done Done false
6515 2946 Acked Flush 0 Sent Done Done false
6516 2947 Acked Read 10 Sent Done Done false
6517 2948 Acked Write 5 Sent Done Done false
6518 2949 Acked Read 3 Sent Done Done false
6519 2950 Acked Write 2 Sent Done Done false
6520 2951 Acked Write 9 Sent Done Done false
6521 2952 Acked Read 3 Sent Done Done false
6522 2953 Acked Read 6 Sent Done Done false
6523 2954 Acked Write 2 Sent Done Done false
6524 2955 Acked Write 3 Sent Done Done false
6525 2956 Acked Read 1 Sent Done Done false
6526 2957 Acked Read 4 Sent Done Done false
6527 2958 Acked Read 5 Sent Done Done false
6528 2959 Acked Write 5 Sent Done Done false
6529 2960 Acked Write 5 Sent Done Done false
6530 2961 Acked Write 5 Sent Done Done false
6531 2962 Acked Read 7 Sent Done Done false
6532 2963 Acked Write 4 Sent Done Done false
6533 2964 Acked Flush 0 Sent Done Done false
6534 2965 Acked Read 4 Sent Done Done false
6535 2966 Acked Flush 0 Sent Done Done false
6536 2967 Acked Flush 0 Sent Done Done false
6537 2968 Acked Write 3 Sent Done Done false
6538 2969 Acked Write 1 Sent Done Done false
6539 2970 Acked Read 3 Sent Done Done false
6540 2971 Acked Write 2 Sent Done Done false
6541 2972 Acked Read 6 Sent Done Done false
6542 2973 Acked Flush 0 Sent Done Done false
6543 2974 Acked Read 3 Sent Done Done false
6544 2975 Acked Write 4 Sent Done Done false
6545 2976 Acked Read 8 Sent Done Done false
6546 2977 Acked Read 3 Sent Done Done false
6547 2978 Acked Write 4 Sent Done Done false
6548 2979 Acked Read 4 Sent Done Done false
6549 2980 Acked Write 3 Sent Done Done false
6550 2981 Acked Read 1 Sent Done Done false
6551 2982 Acked Read 9 Sent Done Done false
6552 2983 Acked Read 1 Sent Done Done false
6553 2984 Acked Read 6 Sent Done Done false
6554 2985 Acked Read 8 Sent Done Done false
6555 2986 Acked Read 1 Sent Done Done false
6556 2987 Acked Read 2 Sent Done Done false
6557 2988 Acked Read 10 Sent Done Done false
6558 2989 Acked Read 9 Sent Done Done false
6559 2990 Acked Read 10 Sent Done Done false
6560 2991 Acked Write 4 Sent Done Done false
6561 2992 Acked Write 3 Sent Done Done false
6562 2993 Acked Read 4 Sent Done Done false
6563 2994 Acked Write 8 Sent Done Done false
6564 2995 Acked Read 9 Sent Done Done false
6565 2996 Acked Write 10 Sent Done Done false
6566 2997 Acked Flush 0 Sent Done Done false
6567 2998 Acked Write 9 Sent Done Done false
6568 2999 Acked Read 3 Sent Done Done false
6569 3000 Acked Write 2 Sent Done Done false
6570 3001 Acked Read 3 Sent Done Done false
6571 3002 Acked Read 9 Sent Done Done false
6572 3003 Acked Write 7 Sent Done Done false
6573 3004 Acked Read 2 Sent Done Done false
6574 3005 Acked Write 3 Sent Done Done false
6575 3006 Acked Flush 0 Sent Done Done false
6576 3007 Acked Write 6 Sent Done Done false
6577 3008 Acked Read 10 Sent Done Done false
6578 3009 Acked Write 5 Sent Done Done false
6579 3010 Acked Write 2 Sent Done Done false
6580 3011 Acked Read 7 Sent Done Done false
6581 3012 Acked Write 7 Sent Done Done false
6582 3013 Acked Write 9 Sent Done Done false
6583 3014 Acked Flush 0 Sent Done Done false
6584 3015 Acked Read 9 Sent Done Done false
6585 3016 Acked Read 6 Sent Done Done false
6586 3017 Acked Read 8 Sent Done Done false
6587 3018 Acked Write 2 Sent Done Done false
6588 3019 Acked Read 2 Sent Done Done false
6589 3020 Acked Read 4 Sent Done Done false
6590 3021 Acked Write 9 Sent Done Done false
6591 3022 Acked Write 2 Sent Done Done false
6592 3023 Acked Read 4 Sent Done Done false
6593 3024 Acked Write 2 Sent Done Done false
6594 3025 Acked Read 7 Sent Done Done false
6595 3026 Acked Write 5 Sent Done Done false
6596 3027 Acked Write 4 Sent Done Done false
6597 3028 Acked Read 6 Sent Done Done false
6598 3029 Acked Read 1 Sent Done Done false
6599 3030 Acked Read 8 Sent Done Done false
6600 3031 Acked Read 5 Sent Done Done false
6601 3032 Acked Read 5 Sent Done Done false
6602 3033 Acked Write 10 Sent Done Done false
6603 3034 Acked Read 1 Sent Done Done false
6604 3035 Acked Write 2 Sent Done Done false
6605 3036 Acked Flush 0 Sent Done Done false
6606 3037 Acked Read 8 Sent Done Done false
6607 3038 Acked Read 10 Sent Done Done false
6608 3039 Acked Read 10 Sent Done Done false
6609 3040 Acked Read 8 Sent Done Done false
6610 3041 Acked Write 10 Sent Done Done false
6611 3042 Acked Write 4 Sent Done Done false
6612 3043 Acked Read 8 Sent Done Done false
6613 3044 Acked Flush 0 Sent Done Done false
6614 3045 Acked Read 9 Sent Done Done false
6615 3046 Acked Write 6 Sent Done Done false
6616 3047 Acked Read 7 Sent Done Done false
6617 3048 Acked Write 3 Sent Done Done false
6618 3049 Acked Write 8 Sent Done Done false
6619 3050 Acked Write 8 Sent Done Done false
6620 3051 Acked Read 8 Sent Done Done false
6621 3052 Acked Write 5 Sent Done Done false
6622 3053 Acked Read 9 Sent Done Done false
6623 3054 Acked Read 4 Sent Done Done false
6624 3055 Acked Write 3 Sent Done Done false
6625 3056 Acked Read 4 Sent Done Done false
6626 3057 Acked Write 2 Sent Done Done false
6627 3058 Acked Flush 0 Sent Done Done false
6628 3059 Acked Read 10 Sent Done Done false
6629 3060 Acked Read 5 Sent Done Done false
6630 3061 Acked Write 1 Sent Done Done false
6631 3062 Acked Write 8 Sent Done Done false
6632 3063 Acked Write 7 Sent Done Done false
6633 3064 Acked Write 2 Sent Done Done false
6634 3065 Acked Read 6 Sent Done Done false
6635 3066 Acked Write 5 Sent Done Done false
6636 3067 Acked Write 3 Sent Done Done false
6637 3068 Acked Read 2 Sent Done Done false
6638 3069 Acked Read 8 Sent Done Done false
6639 3070 Acked Write 5 Sent Done Done false
6640 3071 Acked Read 6 Sent Done Done false
6641 3072 Acked Flush 0 Sent Done Done false
6642 3073 Acked Write 9 Sent Done Done false
6643 3074 Acked Read 10 Sent Done Done false
6644 3075 Acked Read 1 Sent Done Done false
6645 3076 Acked Read 9 Sent Done Done false
6646 3077 Acked Read 4 Sent Done Done false
6647 3078 Acked Read 8 Sent Done Done false
6648 3079 Acked Read 3 Sent Done Done false
6649 3080 Acked Flush 0 Sent Done Done false
6650 3081 Acked Write 4 Sent Done Done false
6651 3082 Acked Write 2 Sent Done Done false
6652 3083 Acked Write 5 Sent Done Done false
6653 3084 Acked Read 2 Sent Done Done false
6654 3085 Acked Read 2 Sent Done Done false
6655 3086 Acked Write 6 Sent Done Done false
6656 3087 Acked Read 2 Sent Done Done false
6657 3088 Acked Flush 0 Sent Done Done false
6658 3089 Acked Read 2 Sent Done Done false
6659 3090 Acked Write 10 Sent Done Done false
6660 3091 Acked Read 4 Sent Done Done false
6661 3092 Acked Write 10 Sent Done Done false
6662 3093 Acked Read 4 Sent Done Done false
6663 3094 Acked Read 3 Sent Done Done false
6664 3095 Acked Read 7 Sent Done Done false
6665 3096 Acked Read 8 Sent Done Done false
6666 3097 Acked Write 9 Sent Done Done false
6667 3098 Acked Read 3 Sent Done Done false
6668 3099 Acked Write 5 Sent Done Done false
6669 3100 Acked Read 6 Sent Done Done false
6670 3101 Acked Read 8 Sent Done Done false
6671 3102 Acked Write 4 Sent Done Done false
6672 3103 Acked Flush 0 Sent Done Done false
6673 3104 Acked Flush 0 Sent Done Done false
6674 3105 Acked Flush 0 Sent Done Done false
6675 3106 Acked Write 6 Sent Done Done false
6676 3107 Acked Flush 0 Sent Done Done false
6677 3108 Acked Read 9 Sent Done Done false
6678 3109 Acked Read 3 Sent Done Done false
6679 3110 Acked Write 3 Sent Done Done false
6680 3111 Acked Read 7 Sent Done Done false
6681 3112 Acked Read 7 Sent Done Done false
6682 STATES DS:0 DS:1 DS:2 TOTAL
6683 Sent 301 0 0 301
6684 Done 0 301 301 602
6685 Skipped 0 0 0 0
6686 Error 0 0 0 0
6687 Last Flush: Some(JobId(2811)) Some(JobId(3107)) Some(JobId(3107))
6688 Downstairs last five completed: 2811 2810 2809 2808 2807
6689 Upstairs last five completed: 3112 3111 3110 3109 3108
6690 CLIENT: Up:0 ds:301 act:2
66912025-02-21T06:46:41.775ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
66922025-02-21T06:46:43.218ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
66932025-02-21T06:46:43.779ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
66942025-02-21T06:46:44.968ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
6695 ----------------------------------------------------------------
6696 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6697 States: Offline Active Active
6698 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6699 2812 Acked Write 8 Sent Done Done false
6700 2813 Acked Write 5 Sent Done Done false
6701 2814 Acked Write 1 Sent Done Done false
6702 2815 Acked Write 7 Sent Done Done false
6703 2816 Acked Read 6 Sent Done Done false
6704 2817 Acked Read 3 Sent Done Done false
6705 2818 Acked Write 4 Sent Done Done false
6706 2819 Acked Read 3 Sent Done Done false
6707 2820 Acked Read 2 Sent Done Done false
6708 2821 Acked Flush 0 Sent Done Done false
6709 2822 Acked Read 6 Sent Done Done false
6710 2823 Acked Read 3 Sent Done Done false
6711 2824 Acked Read 9 Sent Done Done false
6712 2825 Acked Read 3 Sent Done Done false
6713 2826 Acked Read 8 Sent Done Done false
6714 2827 Acked Write 9 Sent Done Done false
6715 2828 Acked Write 7 Sent Done Done false
6716 2829 Acked Write 2 Sent Done Done false
6717 2830 Acked Read 4 Sent Done Done false
6718 2831 Acked Write 3 Sent Done Done false
6719 2832 Acked Read 7 Sent Done Done false
6720 2833 Acked Read 1 Sent Done Done false
6721 2834 Acked Read 8 Sent Done Done false
6722 2835 Acked Write 5 Sent Done Done false
6723 2836 Acked Read 1 Sent Done Done false
6724 2837 Acked Write 10 Sent Done Done false
6725 2838 Acked Write 4 Sent Done Done false
6726 2839 Acked Write 8 Sent Done Done false
6727 2840 Acked Read 2 Sent Done Done false
6728 2841 Acked Write 6 Sent Done Done false
6729 2842 Acked Write 8 Sent Done Done false
6730 2843 Acked Read 3 Sent Done Done false
6731 2844 Acked Write 9 Sent Done Done false
6732 2845 Acked Read 2 Sent Done Done false
6733 2846 Acked Write 7 Sent Done Done false
6734 2847 Acked Read 6 Sent Done Done false
6735 2848 Acked Read 10 Sent Done Done false
6736 2849 Acked Read 3 Sent Done Done false
6737 2850 Acked Read 5 Sent Done Done false
6738 2851 Acked Read 9 Sent Done Done false
6739 2852 Acked Write 10 Sent Done Done false
6740 2853 Acked Flush 0 Sent Done Done false
6741 2854 Acked Write 3 Sent Done Done false
6742 2855 Acked Read 8 Sent Done Done false
6743 2856 Acked Write 9 Sent Done Done false
6744 2857 Acked Read 10 Sent Done Done false
6745 2858 Acked Read 1 Sent Done Done false
6746 2859 Acked Write 5 Sent Done Done false
6747 2860 Acked Read 6 Sent Done Done false
6748 2861 Acked Write 9 Sent Done Done false
6749 2862 Acked Write 2 Sent Done Done false
6750 2863 Acked Flush 0 Sent Done Done false
6751 2864 Acked Read 5 Sent Done Done false
6752 2865 Acked Write 7 Sent Done Done false
6753 2866 Acked Read 9 Sent Done Done false
6754 2867 Acked Write 9 Sent Done Done false
6755 2868 Acked Write 3 Sent Done Done false
6756 2869 Acked Write 9 Sent Done Done false
6757 2870 Acked Read 4 Sent Done Done false
6758 2871 Acked Read 1 Sent Done Done false
6759 2872 Acked Read 8 Sent Done Done false
6760 2873 Acked Write 4 Sent Done Done false
6761 2874 Acked Write 7 Sent Done Done false
6762 2875 Acked Write 3 Sent Done Done false
6763 2876 Acked Read 5 Sent Done Done false
6764 2877 Acked Read 5 Sent Done Done false
6765 2878 Acked Read 3 Sent Done Done false
6766 2879 Acked Write 1 Sent Done Done false
6767 2880 Acked Write 5 Sent Done Done false
6768 2881 Acked Write 7 Sent Done Done false
6769 2882 Acked Write 1 Sent Done Done false
6770 2883 Acked Read 9 Sent Done Done false
6771 2884 Acked Read 9 Sent Done Done false
6772 2885 Acked Write 8 Sent Done Done false
6773 2886 Acked Read 4 Sent Done Done false
6774 2887 Acked Write 1 Sent Done Done false
6775 2888 Acked Read 5 Sent Done Done false
6776 2889 Acked Write 4 Sent Done Done false
6777 2890 Acked Read 2 Sent Done Done false
6778 2891 Acked Read 5 Sent Done Done false
6779 2892 Acked Read 6 Sent Done Done false
6780 2893 Acked Read 3 Sent Done Done false
6781 2894 Acked Flush 0 Sent Done Done false
6782 2895 Acked Write 5 Sent Done Done false
6783 2896 Acked Flush 0 Sent Done Done false
6784 2897 Acked Write 2 Sent Done Done false
6785 2898 Acked Read 6 Sent Done Done false
6786 2899 Acked Write 9 Sent Done Done false
6787 2900 Acked Write 1 Sent Done Done false
6788 2901 Acked Write 9 Sent Done Done false
6789 2902 Acked Read 6 Sent Done Done false
6790 2903 Acked Write 4 Sent Done Done false
6791 2904 Acked Read 4 Sent Done Done false
6792 2905 Acked Read 8 Sent Done Done false
6793 2906 Acked Write 1 Sent Done Done false
6794 2907 Acked Write 4 Sent Done Done false
6795 2908 Acked Write 7 Sent Done Done false
6796 2909 Acked Write 10 Sent Done Done false
6797 2910 Acked Write 9 Sent Done Done false
6798 2911 Acked Write 4 Sent Done Done false
6799 2912 Acked Read 5 Sent Done Done false
6800 2913 Acked Read 8 Sent Done Done false
6801 2914 Acked Write 8 Sent Done Done false
6802 2915 Acked Write 6 Sent Done Done false
6803 2916 Acked Read 7 Sent Done Done false
6804 2917 Acked Read 6 Sent Done Done false
6805 2918 Acked Read 9 Sent Done Done false
6806 2919 Acked Read 3 Sent Done Done false
6807 2920 Acked Read 7 Sent Done Done false
6808 2921 Acked Read 5 Sent Done Done false
6809 2922 Acked Write 2 Sent Done Done false
6810 2923 Acked Read 2 Sent Done Done false
6811 2924 Acked Write 6 Sent Done Done false
6812 2925 Acked Write 3 Sent Done Done false
6813 2926 Acked Read 10 Sent Done Done false
6814 2927 Acked Write 10 Sent Done Done false
6815 2928 Acked Read 3 Sent Done Done false
6816 2929 Acked Read 8 Sent Done Done false
6817 2930 Acked Flush 0 Sent Done Done false
6818 2931 Acked Write 9 Sent Done Done false
6819 2932 Acked Read 10 Sent Done Done false
6820 2933 Acked Read 2 Sent Done Done false
6821 2934 Acked Write 5 Sent Done Done false
6822 2935 Acked Read 10 Sent Done Done false
6823 2936 Acked Write 1 Sent Done Done false
6824 2937 Acked Read 9 Sent Done Done false
6825 2938 Acked Write 4 Sent Done Done false
6826 2939 Acked Write 8 Sent Done Done false
6827 2940 Acked Write 4 Sent Done Done false
6828 2941 Acked Flush 0 Sent Done Done false
6829 2942 Acked Read 9 Sent Done Done false
6830 2943 Acked Read 3 Sent Done Done false
6831 2944 Acked Read 6 Sent Done Done false
6832 2945 Acked Write 8 Sent Done Done false
6833 2946 Acked Flush 0 Sent Done Done false
6834 2947 Acked Read 10 Sent Done Done false
6835 2948 Acked Write 5 Sent Done Done false
6836 2949 Acked Read 3 Sent Done Done false
6837 2950 Acked Write 2 Sent Done Done false
6838 2951 Acked Write 9 Sent Done Done false
6839 2952 Acked Read 3 Sent Done Done false
6840 2953 Acked Read 6 Sent Done Done false
6841 2954 Acked Write 2 Sent Done Done false
6842 2955 Acked Write 3 Sent Done Done false
6843 2956 Acked Read 1 Sent Done Done false
6844 2957 Acked Read 4 Sent Done Done false
6845 2958 Acked Read 5 Sent Done Done false
6846 2959 Acked Write 5 Sent Done Done false
6847 2960 Acked Write 5 Sent Done Done false
6848 2961 Acked Write 5 Sent Done Done false
6849 2962 Acked Read 7 Sent Done Done false
6850 2963 Acked Write 4 Sent Done Done false
6851 2964 Acked Flush 0 Sent Done Done false
6852 2965 Acked Read 4 Sent Done Done false
6853 2966 Acked Flush 0 Sent Done Done false
6854 2967 Acked Flush 0 Sent Done Done false
6855 2968 Acked Write 3 Sent Done Done false
6856 2969 Acked Write 1 Sent Done Done false
6857 2970 Acked Read 3 Sent Done Done false
6858 2971 Acked Write 2 Sent Done Done false
6859 2972 Acked Read 6 Sent Done Done false
6860 2973 Acked Flush 0 Sent Done Done false
6861 2974 Acked Read 3 Sent Done Done false
6862 2975 Acked Write 4 Sent Done Done false
6863 2976 Acked Read 8 Sent Done Done false
6864 2977 Acked Read 3 Sent Done Done false
6865 2978 Acked Write 4 Sent Done Done false
6866 2979 Acked Read 4 Sent Done Done false
6867 2980 Acked Write 3 Sent Done Done false
6868 2981 Acked Read 1 Sent Done Done false
6869 2982 Acked Read 9 Sent Done Done false
6870 2983 Acked Read 1 Sent Done Done false
6871 2984 Acked Read 6 Sent Done Done false
6872 2985 Acked Read 8 Sent Done Done false
6873 2986 Acked Read 1 Sent Done Done false
6874 2987 Acked Read 2 Sent Done Done false
6875 2988 Acked Read 10 Sent Done Done false
6876 2989 Acked Read 9 Sent Done Done false
6877 2990 Acked Read 10 Sent Done Done false
6878 2991 Acked Write 4 Sent Done Done false
6879 2992 Acked Write 3 Sent Done Done false
6880 2993 Acked Read 4 Sent Done Done false
6881 2994 Acked Write 8 Sent Done Done false
6882 2995 Acked Read 9 Sent Done Done false
6883 2996 Acked Write 10 Sent Done Done false
6884 2997 Acked Flush 0 Sent Done Done false
6885 2998 Acked Write 9 Sent Done Done false
6886 2999 Acked Read 3 Sent Done Done false
6887 3000 Acked Write 2 Sent Done Done false
6888 3001 Acked Read 3 Sent Done Done false
6889 3002 Acked Read 9 Sent Done Done false
6890 3003 Acked Write 7 Sent Done Done false
6891 3004 Acked Read 2 Sent Done Done false
6892 3005 Acked Write 3 Sent Done Done false
6893 3006 Acked Flush 0 Sent Done Done false
6894 3007 Acked Write 6 Sent Done Done false
6895 3008 Acked Read 10 Sent Done Done false
6896 3009 Acked Write 5 Sent Done Done false
6897 3010 Acked Write 2 Sent Done Done false
6898 3011 Acked Read 7 Sent Done Done false
6899 3012 Acked Write 7 Sent Done Done false
6900 3013 Acked Write 9 Sent Done Done false
6901 3014 Acked Flush 0 Sent Done Done false
6902 3015 Acked Read 9 Sent Done Done false
6903 3016 Acked Read 6 Sent Done Done false
6904 3017 Acked Read 8 Sent Done Done false
6905 3018 Acked Write 2 Sent Done Done false
6906 3019 Acked Read 2 Sent Done Done false
6907 3020 Acked Read 4 Sent Done Done false
6908 3021 Acked Write 9 Sent Done Done false
6909 3022 Acked Write 2 Sent Done Done false
6910 3023 Acked Read 4 Sent Done Done false
6911 3024 Acked Write 2 Sent Done Done false
6912 3025 Acked Read 7 Sent Done Done false
6913 3026 Acked Write 5 Sent Done Done false
6914 3027 Acked Write 4 Sent Done Done false
6915 3028 Acked Read 6 Sent Done Done false
6916 3029 Acked Read 1 Sent Done Done false
6917 3030 Acked Read 8 Sent Done Done false
6918 3031 Acked Read 5 Sent Done Done false
6919 3032 Acked Read 5 Sent Done Done false
6920 3033 Acked Write 10 Sent Done Done false
6921 3034 Acked Read 1 Sent Done Done false
6922 3035 Acked Write 2 Sent Done Done false
6923 3036 Acked Flush 0 Sent Done Done false
6924 3037 Acked Read 8 Sent Done Done false
6925 3038 Acked Read 10 Sent Done Done false
6926 3039 Acked Read 10 Sent Done Done false
6927 3040 Acked Read 8 Sent Done Done false
6928 3041 Acked Write 10 Sent Done Done false
6929 3042 Acked Write 4 Sent Done Done false
6930 3043 Acked Read 8 Sent Done Done false
6931 3044 Acked Flush 0 Sent Done Done false
6932 3045 Acked Read 9 Sent Done Done false
6933 3046 Acked Write 6 Sent Done Done false
6934 3047 Acked Read 7 Sent Done Done false
6935 3048 Acked Write 3 Sent Done Done false
6936 3049 Acked Write 8 Sent Done Done false
6937 3050 Acked Write 8 Sent Done Done false
6938 3051 Acked Read 8 Sent Done Done false
6939 3052 Acked Write 5 Sent Done Done false
6940 3053 Acked Read 9 Sent Done Done false
6941 3054 Acked Read 4 Sent Done Done false
6942 3055 Acked Write 3 Sent Done Done false
6943 3056 Acked Read 4 Sent Done Done false
6944 3057 Acked Write 2 Sent Done Done false
6945 3058 Acked Flush 0 Sent Done Done false
6946 3059 Acked Read 10 Sent Done Done false
6947 3060 Acked Read 5 Sent Done Done false
6948 3061 Acked Write 1 Sent Done Done false
6949 3062 Acked Write 8 Sent Done Done false
6950 3063 Acked Write 7 Sent Done Done false
6951 3064 Acked Write 2 Sent Done Done false
6952 3065 Acked Read 6 Sent Done Done false
6953 3066 Acked Write 5 Sent Done Done false
6954 3067 Acked Write 3 Sent Done Done false
6955 3068 Acked Read 2 Sent Done Done false
6956 3069 Acked Read 8 Sent Done Done false
6957 3070 Acked Write 5 Sent Done Done false
6958 3071 Acked Read 6 Sent Done Done false
6959 3072 Acked Flush 0 Sent Done Done false
6960 3073 Acked Write 9 Sent Done Done false
6961 3074 Acked Read 10 Sent Done Done false
6962 3075 Acked Read 1 Sent Done Done false
6963 3076 Acked Read 9 Sent Done Done false
6964 3077 Acked Read 4 Sent Done Done false
6965 3078 Acked Read 8 Sent Done Done false
6966 3079 Acked Read 3 Sent Done Done false
6967 3080 Acked Flush 0 Sent Done Done false
6968 3081 Acked Write 4 Sent Done Done false
6969 3082 Acked Write 2 Sent Done Done false
6970 3083 Acked Write 5 Sent Done Done false
6971 3084 Acked Read 2 Sent Done Done false
6972 3085 Acked Read 2 Sent Done Done false
6973 3086 Acked Write 6 Sent Done Done false
6974 3087 Acked Read 2 Sent Done Done false
6975 3088 Acked Flush 0 Sent Done Done false
6976 3089 Acked Read 2 Sent Done Done false
6977 3090 Acked Write 10 Sent Done Done false
6978 3091 Acked Read 4 Sent Done Done false
6979 3092 Acked Write 10 Sent Done Done false
6980 3093 Acked Read 4 Sent Done Done false
6981 3094 Acked Read 3 Sent Done Done false
6982 3095 Acked Read 7 Sent Done Done false
6983 3096 Acked Read 8 Sent Done Done false
6984 3097 Acked Write 9 Sent Done Done false
6985 3098 Acked Read 3 Sent Done Done false
6986 3099 Acked Write 5 Sent Done Done false
6987 3100 Acked Read 6 Sent Done Done false
6988 3101 Acked Read 8 Sent Done Done false
6989 3102 Acked Write 4 Sent Done Done false
6990 3103 Acked Flush 0 Sent Done Done false
6991 3104 Acked Flush 0 Sent Done Done false
6992 3105 Acked Flush 0 Sent Done Done false
6993 3106 Acked Write 6 Sent Done Done false
6994 3107 Acked Flush 0 Sent Done Done false
6995 3108 Acked Read 9 Sent Done Done false
6996 3109 Acked Read 3 Sent Done Done false
6997 3110 Acked Write 3 Sent Done Done false
6998 3111 Acked Read 7 Sent Done Done false
6999 3112 Acked Read 7 Sent Done Done false
7000 STATES DS:0 DS:1 DS:2 TOTAL
7001 Sent 301 0 0 301
7002 Done 0 301 301 602
7003 Skipped 0 0 0 0
7004 Error 0 0 0 0
7005 Last Flush: Some(JobId(2811)) Some(JobId(3107)) Some(JobId(3107))
7006 Downstairs last five completed: 2811 2810 2809 2808 2807
7007 Upstairs last five completed: 3112 3111 3110 3109 3108
7008 CLIENT: Up:0 ds:301 act:2
70092025-02-21T06:46:45.782ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
70102025-02-21T06:46:46.726ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
70112025-02-21T06:46:46.727ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
70122025-02-21T06:46:46.727ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
70132025-02-21T06:46:46.727ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
70142025-02-21T06:46:46.727ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
70152025-02-21T06:46:47.228ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
70162025-02-21T06:46:48.220ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
7017 [8810] got start action
7018 [8810] I am going to restart
7019 Starting downstairs at port 8810
7020 Make output file at "/tmp/dsc/downstairs-8810.txt"
7021 [8810][0] reports Starting
7022 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1213)
7023 [8810][0] reports Running
70242025-02-21T06:46:48.786ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
7025 ----------------------------------------------------------------
7026 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7027 States: Offline Active Active
7028 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7029 2812 Acked Write 8 Sent Done Done false
7030 2813 Acked Write 5 Sent Done Done false
7031 2814 Acked Write 1 Sent Done Done false
7032 2815 Acked Write 7 Sent Done Done false
7033 2816 Acked Read 6 Sent Done Done false
7034 2817 Acked Read 3 Sent Done Done false
7035 2818 Acked Write 4 Sent Done Done false
7036 2819 Acked Read 3 Sent Done Done false
7037 2820 Acked Read 2 Sent Done Done false
7038 2821 Acked Flush 0 Sent Done Done false
7039 2822 Acked Read 6 Sent Done Done false
7040 2823 Acked Read 3 Sent Done Done false
7041 2824 Acked Read 9 Sent Done Done false
7042 2825 Acked Read 3 Sent Done Done false
7043 2826 Acked Read 8 Sent Done Done false
7044 2827 Acked Write 9 Sent Done Done false
7045 2828 Acked Write 7 Sent Done Done false
7046 2829 Acked Write 2 Sent Done Done false
7047 2830 Acked Read 4 Sent Done Done false
7048 2831 Acked Write 3 Sent Done Done false
7049 2832 Acked Read 7 Sent Done Done false
7050 2833 Acked Read 1 Sent Done Done false
7051 2834 Acked Read 8 Sent Done Done false
7052 2835 Acked Write 5 Sent Done Done false
7053 2836 Acked Read 1 Sent Done Done false
7054 2837 Acked Write 10 Sent Done Done false
7055 2838 Acked Write 4 Sent Done Done false
7056 2839 Acked Write 8 Sent Done Done false
7057 2840 Acked Read 2 Sent Done Done false
7058 2841 Acked Write 6 Sent Done Done false
7059 2842 Acked Write 8 Sent Done Done false
7060 2843 Acked Read 3 Sent Done Done false
7061 2844 Acked Write 9 Sent Done Done false
7062 2845 Acked Read 2 Sent Done Done false
7063 2846 Acked Write 7 Sent Done Done false
7064 2847 Acked Read 6 Sent Done Done false
7065 2848 Acked Read 10 Sent Done Done false
7066 2849 Acked Read 3 Sent Done Done false
7067 2850 Acked Read 5 Sent Done Done false
7068 2851 Acked Read 9 Sent Done Done false
7069 2852 Acked Write 10 Sent Done Done false
7070 2853 Acked Flush 0 Sent Done Done false
7071 2854 Acked Write 3 Sent Done Done false
7072 2855 Acked Read 8 Sent Done Done false
7073 2856 Acked Write 9 Sent Done Done false
7074 2857 Acked Read 10 Sent Done Done false
7075 2858 Acked Read 1 Sent Done Done false
7076 2859 Acked Write 5 Sent Done Done false
7077 2860 Acked Read 6 Sent Done Done false
7078 2861 Acked Write 9 Sent Done Done false
7079 2862 Acked Write 2 Sent Done Done false
7080 2863 Acked Flush 0 Sent Done Done false
7081 2864 Acked Read 5 Sent Done Done false
7082 2865 Acked Write 7 Sent Done Done false
7083 2866 Acked Read 9 Sent Done Done false
7084 2867 Acked Write 9 Sent Done Done false
7085 2868 Acked Write 3 Sent Done Done false
7086 2869 Acked Write 9 Sent Done Done false
7087 2870 Acked Read 4 Sent Done Done false
7088 2871 Acked Read 1 Sent Done Done false
7089 2872 Acked Read 8 Sent Done Done false
7090 2873 Acked Write 4 Sent Done Done false
7091 2874 Acked Write 7 Sent Done Done false
7092 2875 Acked Write 3 Sent Done Done false
7093 2876 Acked Read 5 Sent Done Done false
7094 2877 Acked Read 5 Sent Done Done false
7095 2878 Acked Read 3 Sent Done Done false
7096 2879 Acked Write 1 Sent Done Done false
7097 2880 Acked Write 5 Sent Done Done false
7098 2881 Acked Write 7 Sent Done Done false
7099 2882 Acked Write 1 Sent Done Done false
7100 2883 Acked Read 9 Sent Done Done false
7101 2884 Acked Read 9 Sent Done Done false
7102 2885 Acked Write 8 Sent Done Done false
7103 2886 Acked Read 4 Sent Done Done false
7104 2887 Acked Write 1 Sent Done Done false
7105 2888 Acked Read 5 Sent Done Done false
7106 2889 Acked Write 4 Sent Done Done false
7107 2890 Acked Read 2 Sent Done Done false
7108 2891 Acked Read 5 Sent Done Done false
7109 2892 Acked Read 6 Sent Done Done false
7110 2893 Acked Read 3 Sent Done Done false
7111 2894 Acked Flush 0 Sent Done Done false
7112 2895 Acked Write 5 Sent Done Done false
7113 2896 Acked Flush 0 Sent Done Done false
7114 2897 Acked Write 2 Sent Done Done false
7115 2898 Acked Read 6 Sent Done Done false
7116 2899 Acked Write 9 Sent Done Done false
7117 2900 Acked Write 1 Sent Done Done false
7118 2901 Acked Write 9 Sent Done Done false
7119 2902 Acked Read 6 Sent Done Done false
7120 2903 Acked Write 4 Sent Done Done false
7121 2904 Acked Read 4 Sent Done Done false
7122 2905 Acked Read 8 Sent Done Done false
7123 2906 Acked Write 1 Sent Done Done false
7124 2907 Acked Write 4 Sent Done Done false
7125 2908 Acked Write 7 Sent Done Done false
7126 2909 Acked Write 10 Sent Done Done false
7127 2910 Acked Write 9 Sent Done Done false
7128 2911 Acked Write 4 Sent Done Done false
7129 2912 Acked Read 5 Sent Done Done false
7130 2913 Acked Read 8 Sent Done Done false
7131 2914 Acked Write 8 Sent Done Done false
7132 2915 Acked Write 6 Sent Done Done false
7133 2916 Acked Read 7 Sent Done Done false
7134 2917 Acked Read 6 Sent Done Done false
7135 2918 Acked Read 9 Sent Done Done false
7136 2919 Acked Read 3 Sent Done Done false
7137 2920 Acked Read 7 Sent Done Done false
7138 2921 Acked Read 5 Sent Done Done false
7139 2922 Acked Write 2 Sent Done Done false
7140 2923 Acked Read 2 Sent Done Done false
7141 2924 Acked Write 6 Sent Done Done false
7142 2925 Acked Write 3 Sent Done Done false
7143 2926 Acked Read 10 Sent Done Done false
7144 2927 Acked Write 10 Sent Done Done false
7145 2928 Acked Read 3 Sent Done Done false
7146 2929 Acked Read 8 Sent Done Done false
7147 2930 Acked Flush 0 Sent Done Done false
7148 2931 Acked Write 9 Sent Done Done false
7149 2932 Acked Read 10 Sent Done Done false
7150 2933 Acked Read 2 Sent Done Done false
7151 2934 Acked Write 5 Sent Done Done false
7152 2935 Acked Read 10 Sent Done Done false
7153 2936 Acked Write 1 Sent Done Done false
7154 2937 Acked Read 9 Sent Done Done false
7155 2938 Acked Write 4 Sent Done Done false
7156 2939 Acked Write 8 Sent Done Done false
7157 2940 Acked Write 4 Sent Done Done false
7158 2941 Acked Flush 0 Sent Done Done false
7159 2942 Acked Read 9 Sent Done Done false
7160 2943 Acked Read 3 Sent Done Done false
7161 2944 Acked Read 6 Sent Done Done false
7162 2945 Acked Write 8 Sent Done Done false
7163 2946 Acked Flush 0 Sent Done Done false
7164 2947 Acked Read 10 Sent Done Done false
7165 2948 Acked Write 5 Sent Done Done false
7166 2949 Acked Read 3 Sent Done Done false
7167 2950 Acked Write 2 Sent Done Done false
7168 2951 Acked Write 9 Sent Done Done false
7169 2952 Acked Read 3 Sent Done Done false
7170 2953 Acked Read 6 Sent Done Done false
7171 2954 Acked Write 2 Sent Done Done false
7172 2955 Acked Write 3 Sent Done Done false
7173 2956 Acked Read 1 Sent Done Done false
7174 2957 Acked Read 4 Sent Done Done false
7175 2958 Acked Read 5 Sent Done Done false
7176 2959 Acked Write 5 Sent Done Done false
7177 2960 Acked Write 5 Sent Done Done false
7178 2961 Acked Write 5 Sent Done Done false
7179 2962 Acked Read 7 Sent Done Done false
7180 2963 Acked Write 4 Sent Done Done false
7181 2964 Acked Flush 0 Sent Done Done false
7182 2965 Acked Read 4 Sent Done Done false
7183 2966 Acked Flush 0 Sent Done Done false
7184 2967 Acked Flush 0 Sent Done Done false
7185 2968 Acked Write 3 Sent Done Done false
7186 2969 Acked Write 1 Sent Done Done false
7187 2970 Acked Read 3 Sent Done Done false
7188 2971 Acked Write 2 Sent Done Done false
7189 2972 Acked Read 6 Sent Done Done false
7190 2973 Acked Flush 0 Sent Done Done false
7191 2974 Acked Read 3 Sent Done Done false
7192 2975 Acked Write 4 Sent Done Done false
7193 2976 Acked Read 8 Sent Done Done false
7194 2977 Acked Read 3 Sent Done Done false
7195 2978 Acked Write 4 Sent Done Done false
7196 2979 Acked Read 4 Sent Done Done false
7197 2980 Acked Write 3 Sent Done Done false
7198 2981 Acked Read 1 Sent Done Done false
7199 2982 Acked Read 9 Sent Done Done false
7200 2983 Acked Read 1 Sent Done Done false
7201 2984 Acked Read 6 Sent Done Done false
7202 2985 Acked Read 8 Sent Done Done false
7203 2986 Acked Read 1 Sent Done Done false
7204 2987 Acked Read 2 Sent Done Done false
7205 2988 Acked Read 10 Sent Done Done false
7206 2989 Acked Read 9 Sent Done Done false
7207 2990 Acked Read 10 Sent Done Done false
7208 2991 Acked Write 4 Sent Done Done false
7209 2992 Acked Write 3 Sent Done Done false
7210 2993 Acked Read 4 Sent Done Done false
7211 2994 Acked Write 8 Sent Done Done false
7212 2995 Acked Read 9 Sent Done Done false
7213 2996 Acked Write 10 Sent Done Done false
7214 2997 Acked Flush 0 Sent Done Done false
7215 2998 Acked Write 9 Sent Done Done false
7216 2999 Acked Read 3 Sent Done Done false
7217 3000 Acked Write 2 Sent Done Done false
7218 3001 Acked Read 3 Sent Done Done false
7219 3002 Acked Read 9 Sent Done Done false
7220 3003 Acked Write 7 Sent Done Done false
7221 3004 Acked Read 2 Sent Done Done false
7222 3005 Acked Write 3 Sent Done Done false
7223 3006 Acked Flush 0 Sent Done Done false
7224 3007 Acked Write 6 Sent Done Done false
7225 3008 Acked Read 10 Sent Done Done false
7226 3009 Acked Write 5 Sent Done Done false
7227 3010 Acked Write 2 Sent Done Done false
7228 3011 Acked Read 7 Sent Done Done false
7229 3012 Acked Write 7 Sent Done Done false
7230 3013 Acked Write 9 Sent Done Done false
7231 3014 Acked Flush 0 Sent Done Done false
7232 3015 Acked Read 9 Sent Done Done false
7233 3016 Acked Read 6 Sent Done Done false
7234 3017 Acked Read 8 Sent Done Done false
7235 3018 Acked Write 2 Sent Done Done false
7236 3019 Acked Read 2 Sent Done Done false
7237 3020 Acked Read 4 Sent Done Done false
7238 3021 Acked Write 9 Sent Done Done false
7239 3022 Acked Write 2 Sent Done Done false
7240 3023 Acked Read 4 Sent Done Done false
7241 3024 Acked Write 2 Sent Done Done false
7242 3025 Acked Read 7 Sent Done Done false
7243 3026 Acked Write 5 Sent Done Done false
7244 3027 Acked Write 4 Sent Done Done false
7245 3028 Acked Read 6 Sent Done Done false
7246 3029 Acked Read 1 Sent Done Done false
7247 3030 Acked Read 8 Sent Done Done false
7248 3031 Acked Read 5 Sent Done Done false
7249 3032 Acked Read 5 Sent Done Done false
7250 3033 Acked Write 10 Sent Done Done false
7251 3034 Acked Read 1 Sent Done Done false
7252 3035 Acked Write 2 Sent Done Done false
7253 3036 Acked Flush 0 Sent Done Done false
7254 3037 Acked Read 8 Sent Done Done false
7255 3038 Acked Read 10 Sent Done Done false
7256 3039 Acked Read 10 Sent Done Done false
7257 3040 Acked Read 8 Sent Done Done false
7258 3041 Acked Write 10 Sent Done Done false
7259 3042 Acked Write 4 Sent Done Done false
7260 3043 Acked Read 8 Sent Done Done false
7261 3044 Acked Flush 0 Sent Done Done false
7262 3045 Acked Read 9 Sent Done Done false
7263 3046 Acked Write 6 Sent Done Done false
7264 3047 Acked Read 7 Sent Done Done false
7265 3048 Acked Write 3 Sent Done Done false
7266 3049 Acked Write 8 Sent Done Done false
7267 3050 Acked Write 8 Sent Done Done false
7268 3051 Acked Read 8 Sent Done Done false
7269 3052 Acked Write 5 Sent Done Done false
7270 3053 Acked Read 9 Sent Done Done false
7271 3054 Acked Read 4 Sent Done Done false
7272 3055 Acked Write 3 Sent Done Done false
7273 3056 Acked Read 4 Sent Done Done false
7274 3057 Acked Write 2 Sent Done Done false
7275 3058 Acked Flush 0 Sent Done Done false
7276 3059 Acked Read 10 Sent Done Done false
7277 3060 Acked Read 5 Sent Done Done false
7278 3061 Acked Write 1 Sent Done Done false
7279 3062 Acked Write 8 Sent Done Done false
7280 3063 Acked Write 7 Sent Done Done false
7281 3064 Acked Write 2 Sent Done Done false
7282 3065 Acked Read 6 Sent Done Done false
7283 3066 Acked Write 5 Sent Done Done false
7284 3067 Acked Write 3 Sent Done Done false
7285 3068 Acked Read 2 Sent Done Done false
7286 3069 Acked Read 8 Sent Done Done false
7287 3070 Acked Write 5 Sent Done Done false
7288 3071 Acked Read 6 Sent Done Done false
7289 3072 Acked Flush 0 Sent Done Done false
7290 3073 Acked Write 9 Sent Done Done false
7291 3074 Acked Read 10 Sent Done Done false
7292 3075 Acked Read 1 Sent Done Done false
7293 3076 Acked Read 9 Sent Done Done false
7294 3077 Acked Read 4 Sent Done Done false
7295 3078 Acked Read 8 Sent Done Done false
7296 3079 Acked Read 3 Sent Done Done false
7297 3080 Acked Flush 0 Sent Done Done false
7298 3081 Acked Write 4 Sent Done Done false
7299 3082 Acked Write 2 Sent Done Done false
7300 3083 Acked Write 5 Sent Done Done false
7301 3084 Acked Read 2 Sent Done Done false
7302 3085 Acked Read 2 Sent Done Done false
7303 3086 Acked Write 6 Sent Done Done false
7304 3087 Acked Read 2 Sent Done Done false
7305 3088 Acked Flush 0 Sent Done Done false
7306 3089 Acked Read 2 Sent Done Done false
7307 3090 Acked Write 10 Sent Done Done false
7308 3091 Acked Read 4 Sent Done Done false
7309 3092 Acked Write 10 Sent Done Done false
7310 3093 Acked Read 4 Sent Done Done false
7311 3094 Acked Read 3 Sent Done Done false
7312 3095 Acked Read 7 Sent Done Done false
7313 3096 Acked Read 8 Sent Done Done false
7314 3097 Acked Write 9 Sent Done Done false
7315 3098 Acked Read 3 Sent Done Done false
7316 3099 Acked Write 5 Sent Done Done false
7317 3100 Acked Read 6 Sent Done Done false
7318 3101 Acked Read 8 Sent Done Done false
7319 3102 Acked Write 4 Sent Done Done false
7320 3103 Acked Flush 0 Sent Done Done false
7321 3104 Acked Flush 0 Sent Done Done false
7322 3105 Acked Flush 0 Sent Done Done false
7323 3106 Acked Write 6 Sent Done Done false
7324 3107 Acked Flush 0 Sent Done Done false
7325 3108 Acked Read 9 Sent Done Done false
7326 3109 Acked Read 3 Sent Done Done false
7327 3110 Acked Write 3 Sent Done Done false
7328 3111 Acked Read 7 Sent Done Done false
7329 3112 Acked Read 7 Sent Done Done false
7330 STATES DS:0 DS:1 DS:2 TOTAL
7331 Sent 301 0 0 301
7332 Done 0 301 301 602
7333 Skipped 0 0 0 0
7334 Error 0 0 0 0
7335 Last Flush: Some(JobId(2811)) Some(JobId(3107)) Some(JobId(3107))
7336 Downstairs last five completed: 2811 2810 2809 2808 2807
7337 Upstairs last five completed: 3112 3111 3110 3109 3108
7338 CLIENT: Up:0 ds:301 act:2
73392025-02-21T06:46:49.969ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
73402025-02-21T06:46:50.790ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
73412025-02-21T06:46:51.791ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
7342 ----------------------------------------------------------------
7343 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7344 States: Offline Active Active
7345 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7346 2812 Acked Write 8 Sent Done Done false
7347 2813 Acked Write 5 Sent Done Done false
7348 2814 Acked Write 1 Sent Done Done false
7349 2815 Acked Write 7 Sent Done Done false
7350 2816 Acked Read 6 Sent Done Done false
7351 2817 Acked Read 3 Sent Done Done false
7352 2818 Acked Write 4 Sent Done Done false
7353 2819 Acked Read 3 Sent Done Done false
7354 2820 Acked Read 2 Sent Done Done false
7355 2821 Acked Flush 0 Sent Done Done false
7356 2822 Acked Read 6 Sent Done Done false
7357 2823 Acked Read 3 Sent Done Done false
7358 2824 Acked Read 9 Sent Done Done false
7359 2825 Acked Read 3 Sent Done Done false
7360 2826 Acked Read 8 Sent Done Done false
7361 2827 Acked Write 9 Sent Done Done false
7362 2828 Acked Write 7 Sent Done Done false
7363 2829 Acked Write 2 Sent Done Done false
7364 2830 Acked Read 4 Sent Done Done false
7365 2831 Acked Write 3 Sent Done Done false
7366 2832 Acked Read 7 Sent Done Done false
7367 2833 Acked Read 1 Sent Done Done false
7368 2834 Acked Read 8 Sent Done Done false
7369 2835 Acked Write 5 Sent Done Done false
7370 2836 Acked Read 1 Sent Done Done false
7371 2837 Acked Write 10 Sent Done Done false
7372 2838 Acked Write 4 Sent Done Done false
7373 2839 Acked Write 8 Sent Done Done false
7374 2840 Acked Read 2 Sent Done Done false
7375 2841 Acked Write 6 Sent Done Done false
7376 2842 Acked Write 8 Sent Done Done false
7377 2843 Acked Read 3 Sent Done Done false
7378 2844 Acked Write 9 Sent Done Done false
7379 2845 Acked Read 2 Sent Done Done false
7380 2846 Acked Write 7 Sent Done Done false
7381 2847 Acked Read 6 Sent Done Done false
7382 2848 Acked Read 10 Sent Done Done false
7383 2849 Acked Read 3 Sent Done Done false
7384 2850 Acked Read 5 Sent Done Done false
7385 2851 Acked Read 9 Sent Done Done false
7386 2852 Acked Write 10 Sent Done Done false
7387 2853 Acked Flush 0 Sent Done Done false
7388 2854 Acked Write 3 Sent Done Done false
7389 2855 Acked Read 8 Sent Done Done false
7390 2856 Acked Write 9 Sent Done Done false
7391 2857 Acked Read 10 Sent Done Done false
7392 2858 Acked Read 1 Sent Done Done false
7393 2859 Acked Write 5 Sent Done Done false
7394 2860 Acked Read 6 Sent Done Done false
7395 2861 Acked Write 9 Sent Done Done false
7396 2862 Acked Write 2 Sent Done Done false
7397 2863 Acked Flush 0 Sent Done Done false
7398 2864 Acked Read 5 Sent Done Done false
7399 2865 Acked Write 7 Sent Done Done false
7400 2866 Acked Read 9 Sent Done Done false
7401 2867 Acked Write 9 Sent Done Done false
7402 2868 Acked Write 3 Sent Done Done false
7403 2869 Acked Write 9 Sent Done Done false
7404 2870 Acked Read 4 Sent Done Done false
7405 2871 Acked Read 1 Sent Done Done false
7406 2872 Acked Read 8 Sent Done Done false
7407 2873 Acked Write 4 Sent Done Done false
7408 2874 Acked Write 7 Sent Done Done false
7409 2875 Acked Write 3 Sent Done Done false
7410 2876 Acked Read 5 Sent Done Done false
7411 2877 Acked Read 5 Sent Done Done false
7412 2878 Acked Read 3 Sent Done Done false
7413 2879 Acked Write 1 Sent Done Done false
7414 2880 Acked Write 5 Sent Done Done false
7415 2881 Acked Write 7 Sent Done Done false
7416 2882 Acked Write 1 Sent Done Done false
7417 2883 Acked Read 9 Sent Done Done false
7418 2884 Acked Read 9 Sent Done Done false
7419 2885 Acked Write 8 Sent Done Done false
7420 2886 Acked Read 4 Sent Done Done false
7421 2887 Acked Write 1 Sent Done Done false
7422 2888 Acked Read 5 Sent Done Done false
7423 2889 Acked Write 4 Sent Done Done false
7424 2890 Acked Read 2 Sent Done Done false
7425 2891 Acked Read 5 Sent Done Done false
7426 2892 Acked Read 6 Sent Done Done false
7427 2893 Acked Read 3 Sent Done Done false
7428 2894 Acked Flush 0 Sent Done Done false
7429 2895 Acked Write 5 Sent Done Done false
7430 2896 Acked Flush 0 Sent Done Done false
7431 2897 Acked Write 2 Sent Done Done false
7432 2898 Acked Read 6 Sent Done Done false
7433 2899 Acked Write 9 Sent Done Done false
7434 2900 Acked Write 1 Sent Done Done false
7435 2901 Acked Write 9 Sent Done Done false
7436 2902 Acked Read 6 Sent Done Done false
7437 2903 Acked Write 4 Sent Done Done false
7438 2904 Acked Read 4 Sent Done Done false
7439 2905 Acked Read 8 Sent Done Done false
7440 2906 Acked Write 1 Sent Done Done false
7441 2907 Acked Write 4 Sent Done Done false
7442 2908 Acked Write 7 Sent Done Done false
7443 2909 Acked Write 10 Sent Done Done false
7444 2910 Acked Write 9 Sent Done Done false
7445 2911 Acked Write 4 Sent Done Done false
7446 2912 Acked Read 5 Sent Done Done false
7447 2913 Acked Read 8 Sent Done Done false
7448 2914 Acked Write 8 Sent Done Done false
7449 2915 Acked Write 6 Sent Done Done false
7450 2916 Acked Read 7 Sent Done Done false
7451 2917 Acked Read 6 Sent Done Done false
7452 2918 Acked Read 9 Sent Done Done false
7453 2919 Acked Read 3 Sent Done Done false
7454 2920 Acked Read 7 Sent Done Done false
7455 2921 Acked Read 5 Sent Done Done false
7456 2922 Acked Write 2 Sent Done Done false
7457 2923 Acked Read 2 Sent Done Done false
7458 2924 Acked Write 6 Sent Done Done false
7459 2925 Acked Write 3 Sent Done Done false
7460 2926 Acked Read 10 Sent Done Done false
7461 2927 Acked Write 10 Sent Done Done false
7462 2928 Acked Read 3 Sent Done Done false
7463 2929 Acked Read 8 Sent Done Done false
7464 2930 Acked Flush 0 Sent Done Done false
7465 2931 Acked Write 9 Sent Done Done false
7466 2932 Acked Read 10 Sent Done Done false
7467 2933 Acked Read 2 Sent Done Done false
7468 2934 Acked Write 5 Sent Done Done false
7469 2935 Acked Read 10 Sent Done Done false
7470 2936 Acked Write 1 Sent Done Done false
7471 2937 Acked Read 9 Sent Done Done false
7472 2938 Acked Write 4 Sent Done Done false
7473 2939 Acked Write 8 Sent Done Done false
7474 2940 Acked Write 4 Sent Done Done false
7475 2941 Acked Flush 0 Sent Done Done false
7476 2942 Acked Read 9 Sent Done Done false
7477 2943 Acked Read 3 Sent Done Done false
7478 2944 Acked Read 6 Sent Done Done false
7479 2945 Acked Write 8 Sent Done Done false
7480 2946 Acked Flush 0 Sent Done Done false
7481 2947 Acked Read 10 Sent Done Done false
7482 2948 Acked Write 5 Sent Done Done false
7483 2949 Acked Read 3 Sent Done Done false
7484 2950 Acked Write 2 Sent Done Done false
7485 2951 Acked Write 9 Sent Done Done false
7486 2952 Acked Read 3 Sent Done Done false
7487 2953 Acked Read 6 Sent Done Done false
7488 2954 Acked Write 2 Sent Done Done false
7489 2955 Acked Write 3 Sent Done Done false
7490 2956 Acked Read 1 Sent Done Done false
7491 2957 Acked Read 4 Sent Done Done false
7492 2958 Acked Read 5 Sent Done Done false
7493 2959 Acked Write 5 Sent Done Done false
7494 2960 Acked Write 5 Sent Done Done false
7495 2961 Acked Write 5 Sent Done Done false
7496 2962 Acked Read 7 Sent Done Done false
7497 2963 Acked Write 4 Sent Done Done false
7498 2964 Acked Flush 0 Sent Done Done false
7499 2965 Acked Read 4 Sent Done Done false
7500 2966 Acked Flush 0 Sent Done Done false
7501 2967 Acked Flush 0 Sent Done Done false
7502 2968 Acked Write 3 Sent Done Done false
7503 2969 Acked Write 1 Sent Done Done false
7504 2970 Acked Read 3 Sent Done Done false
7505 2971 Acked Write 2 Sent Done Done false
7506 2972 Acked Read 6 Sent Done Done false
7507 2973 Acked Flush 0 Sent Done Done false
7508 2974 Acked Read 3 Sent Done Done false
7509 2975 Acked Write 4 Sent Done Done false
7510 2976 Acked Read 8 Sent Done Done false
7511 2977 Acked Read 3 Sent Done Done false
7512 2978 Acked Write 4 Sent Done Done false
7513 2979 Acked Read 4 Sent Done Done false
7514 2980 Acked Write 3 Sent Done Done false
7515 2981 Acked Read 1 Sent Done Done false
7516 2982 Acked Read 9 Sent Done Done false
7517 2983 Acked Read 1 Sent Done Done false
7518 2984 Acked Read 6 Sent Done Done false
7519 2985 Acked Read 8 Sent Done Done false
7520 2986 Acked Read 1 Sent Done Done false
7521 2987 Acked Read 2 Sent Done Done false
7522 2988 Acked Read 10 Sent Done Done false
7523 2989 Acked Read 9 Sent Done Done false
7524 2990 Acked Read 10 Sent Done Done false
7525 2991 Acked Write 4 Sent Done Done false
7526 2992 Acked Write 3 Sent Done Done false
7527 2993 Acked Read 4 Sent Done Done false
7528 2994 Acked Write 8 Sent Done Done false
7529 2995 Acked Read 9 Sent Done Done false
7530 2996 Acked Write 10 Sent Done Done false
7531 2997 Acked Flush 0 Sent Done Done false
7532 2998 Acked Write 9 Sent Done Done false
7533 2999 Acked Read 3 Sent Done Done false
7534 3000 Acked Write 2 Sent Done Done false
7535 3001 Acked Read 3 Sent Done Done false
7536 3002 Acked Read 9 Sent Done Done false
7537 3003 Acked Write 7 Sent Done Done false
7538 3004 Acked Read 2 Sent Done Done false
7539 3005 Acked Write 3 Sent Done Done false
7540 3006 Acked Flush 0 Sent Done Done false
7541 3007 Acked Write 6 Sent Done Done false
7542 3008 Acked Read 10 Sent Done Done false
7543 3009 Acked Write 5 Sent Done Done false
7544 3010 Acked Write 2 Sent Done Done false
7545 3011 Acked Read 7 Sent Done Done false
7546 3012 Acked Write 7 Sent Done Done false
7547 3013 Acked Write 9 Sent Done Done false
7548 3014 Acked Flush 0 Sent Done Done false
7549 3015 Acked Read 9 Sent Done Done false
7550 3016 Acked Read 6 Sent Done Done false
7551 3017 Acked Read 8 Sent Done Done false
7552 3018 Acked Write 2 Sent Done Done false
7553 3019 Acked Read 2 Sent Done Done false
7554 3020 Acked Read 4 Sent Done Done false
7555 3021 Acked Write 9 Sent Done Done false
7556 3022 Acked Write 2 Sent Done Done false
7557 3023 Acked Read 4 Sent Done Done false
7558 3024 Acked Write 2 Sent Done Done false
7559 3025 Acked Read 7 Sent Done Done false
7560 3026 Acked Write 5 Sent Done Done false
7561 3027 Acked Write 4 Sent Done Done false
7562 3028 Acked Read 6 Sent Done Done false
7563 3029 Acked Read 1 Sent Done Done false
7564 3030 Acked Read 8 Sent Done Done false
7565 3031 Acked Read 5 Sent Done Done false
7566 3032 Acked Read 5 Sent Done Done false
7567 3033 Acked Write 10 Sent Done Done false
7568 3034 Acked Read 1 Sent Done Done false
7569 3035 Acked Write 2 Sent Done Done false
7570 3036 Acked Flush 0 Sent Done Done false
7571 3037 Acked Read 8 Sent Done Done false
7572 3038 Acked Read 10 Sent Done Done false
7573 3039 Acked Read 10 Sent Done Done false
7574 3040 Acked Read 8 Sent Done Done false
7575 3041 Acked Write 10 Sent Done Done false
7576 3042 Acked Write 4 Sent Done Done false
7577 3043 Acked Read 8 Sent Done Done false
7578 3044 Acked Flush 0 Sent Done Done false
7579 3045 Acked Read 9 Sent Done Done false
7580 3046 Acked Write 6 Sent Done Done false
7581 3047 Acked Read 7 Sent Done Done false
7582 3048 Acked Write 3 Sent Done Done false
7583 3049 Acked Write 8 Sent Done Done false
7584 3050 Acked Write 8 Sent Done Done false
7585 3051 Acked Read 8 Sent Done Done false
7586 3052 Acked Write 5 Sent Done Done false
7587 3053 Acked Read 9 Sent Done Done false
7588 3054 Acked Read 4 Sent Done Done false
7589 3055 Acked Write 3 Sent Done Done false
7590 3056 Acked Read 4 Sent Done Done false
7591 3057 Acked Write 2 Sent Done Done false
7592 3058 Acked Flush 0 Sent Done Done false
7593 3059 Acked Read 10 Sent Done Done false
7594 3060 Acked Read 5 Sent Done Done false
7595 3061 Acked Write 1 Sent Done Done false
7596 3062 Acked Write 8 Sent Done Done false
7597 3063 Acked Write 7 Sent Done Done false
7598 3064 Acked Write 2 Sent Done Done false
7599 3065 Acked Read 6 Sent Done Done false
7600 3066 Acked Write 5 Sent Done Done false
7601 3067 Acked Write 3 Sent Done Done false
7602 3068 Acked Read 2 Sent Done Done false
7603 3069 Acked Read 8 Sent Done Done false
7604 3070 Acked Write 5 Sent Done Done false
7605 3071 Acked Read 6 Sent Done Done false
7606 3072 Acked Flush 0 Sent Done Done false
7607 3073 Acked Write 9 Sent Done Done false
7608 3074 Acked Read 10 Sent Done Done false
7609 3075 Acked Read 1 Sent Done Done false
7610 3076 Acked Read 9 Sent Done Done false
7611 3077 Acked Read 4 Sent Done Done false
7612 3078 Acked Read 8 Sent Done Done false
7613 3079 Acked Read 3 Sent Done Done false
7614 3080 Acked Flush 0 Sent Done Done false
7615 3081 Acked Write 4 Sent Done Done false
7616 3082 Acked Write 2 Sent Done Done false
7617 3083 Acked Write 5 Sent Done Done false
7618 3084 Acked Read 2 Sent Done Done false
7619 3085 Acked Read 2 Sent Done Done false
7620 3086 Acked Write 6 Sent Done Done false
7621 3087 Acked Read 2 Sent Done Done false
7622 3088 Acked Flush 0 Sent Done Done false
7623 3089 Acked Read 2 Sent Done Done false
7624 3090 Acked Write 10 Sent Done Done false
7625 3091 Acked Read 4 Sent Done Done false
7626 3092 Acked Write 10 Sent Done Done false
7627 3093 Acked Read 4 Sent Done Done false
7628 3094 Acked Read 3 Sent Done Done false
7629 3095 Acked Read 7 Sent Done Done false
7630 3096 Acked Read 8 Sent Done Done false
7631 3097 Acked Write 9 Sent Done Done false
7632 3098 Acked Read 3 Sent Done Done false
7633 3099 Acked Write 5 Sent Done Done false
7634 3100 Acked Read 6 Sent Done Done false
7635 3101 Acked Read 8 Sent Done Done false
7636 3102 Acked Write 4 Sent Done Done false
7637 3103 Acked Flush 0 Sent Done Done false
7638 3104 Acked Flush 0 Sent Done Done false
7639 3105 Acked Flush 0 Sent Done Done false
7640 3106 Acked Write 6 Sent Done Done false
7641 3107 Acked Flush 0 Sent Done Done false
7642 3108 Acked Read 9 Sent Done Done false
7643 3109 Acked Read 3 Sent Done Done false
7644 3110 Acked Write 3 Sent Done Done false
7645 3111 Acked Read 7 Sent Done Done false
7646 3112 Acked Read 7 Sent Done Done false
7647 STATES DS:0 DS:1 DS:2 TOTAL
7648 Sent 301 0 0 301
7649 Done 0 301 301 602
7650 Skipped 0 0 0 0
7651 Error 0 0 0 0
7652 Last Flush: Some(JobId(2811)) Some(JobId(3107)) Some(JobId(3107))
7653 Downstairs last five completed: 2811 2810 2809 2808 2807
7654 Upstairs last five completed: 3112 3111 3110 3109 3108
7655 CLIENT: Up:0 ds:301 act:2
76562025-02-21T06:46:53.795ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
76572025-02-21T06:46:54.969ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
76582025-02-21T06:46:55.798ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
76592025-02-21T06:46:56.728ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
76602025-02-21T06:46:56.728ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58872) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
76612025-02-21T06:46:56.729ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
76622025-02-21T06:46:56.729ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
76632025-02-21T06:46:56.729ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
76642025-02-21T06:46:56.730ZINFOcrucible: send last flush ID to this DS: Some(JobId(2811)) = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
76652025-02-21T06:46:56.730ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
76662025-02-21T06:46:56.730ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2811)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
76672025-02-21T06:46:56.730ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2811)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
7668 ----------------------------------------------------------------
7669 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:5
7670 States: Active Active Active
7671 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7672 3108 Acked Read 9 Done Done Done true
7673 3109 Acked Read 3 Done Done Done true
7674 3110 Acked Write 3 Done Done Done true
7675 3111 Acked Read 7 Done Done Done true
7676 3112 Acked Read 7 Done Done Done true
7677 STATES DS:0 DS:1 DS:2 TOTAL
7678 Sent 0 0 0 0
7679 Done 5 5 5 15
7680 Skipped 0 0 0 0
7681 Error 0 0 0 0
7682 Last Flush: Some(JobId(3107)) Some(JobId(3107)) Some(JobId(3107))
7683 Downstairs last five completed: 3107 3106 3105 3104 3103
7684 Upstairs last five completed: 3112 3111 3110 3109 3108
7685 CLIENT: Up:0 ds:5 act:3
7686 ----------------------------------------------------------------
7687 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7688 Downstairs last five completed: 3113 3112 3111 3110 3109
7689 Upstairs last five completed: 3113 3112 3111 3110 3109
7690 CLIENT: Up:0 ds:0 act:3
7691 Replay: All jobs finished
7692 Main task has work to do, go find it
7693 got dsc Stop(1)
7694 stop 1
7695 Feb 21 06:47:00.980 INFO request completed, latency_us: 364, response_code: 204, uri: [8820] Got stop action so:false kr:false
7696 /stop/cid/1, method: POST, req_id: ae192a13-90e3-4b9e-a06f-8eb5e67652ef, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
7697 Feb 21 06:47:00.981 INFO request completed, latency_us{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:47:00.981765672Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
7698 : 122{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:47:00.981877742Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
7699 , response_code: 200, uri: /state/cid/1, method: GET, req_id: e1024135-e271-4617-973b-01f98229d3d4, {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:47:00.981905297Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
7700 remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:47:00.981976235Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
7701 1:9998
7702 [8820] Exited with: ExitStatus(unix_wait_status(9))
7703 [8820][1] reports Exit
7704 Feb 21 06:47:04.984 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/1, method: GET, req_id: 3d0a94f3-5d78-468e-9106-cf110317148d, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
7705 001/300 Write block 2933 len 28672 data: 2 2 2 2 2 2 2
7706 002/300 Read block 3117 len 32768
7707 003/300 Read block 3775 len 28672
7708 004/300 Write block 2952 len 16384 data: 4 4 4 4
7709 005/300 Write block 524 len 8192 data: 2 2
7710 006/300 Write block 2928 len 24576 data: 2 2 2 2 2 3
7711 007/300 Read block 2575 len 24576
7712 008/300 Read block 831 len 24576
7713 009/300 Write block 2626 len 16384 data: 3 3 4 5
7714 010/300 Write block 4776 len 8192 data: 2 2
7715 011/300 Read block 3143 len 28672
7716 012/300 Write block 2837 len 24576 data: 3 3 3 3 3 2
7717 013/300 Read block 1870 len 16384
7718 014/300 Read block 4915 len 8192
7719 015/300 Write block 374 len 36864 data: 3 3 3 3 2 2 2 3 3
7720 016/300 Write block 570 len 20480 data: 2 2 2 2 2
7721 017/300 Read block 1894 len 36864
7722 018/300 Read block 719 len 36864
7723 019/300 Read block 3784 len 4096
7724 020/300 Flush
7725 021/300 Write block 1536 len 40960 data: 4 4 3 3 2 2 2 2 2 2
7726 022/300 Read block 1790 len 36864
7727 023/300 Write block 2107 len 32768 data: 3 3 3 3 2 2 3 3
7728 024/300 Read block 4179 len 24576
7729 025/300 Read block 3035 len 28672
7730 026/300 Read block 4949 len 24576
7731 027/300 Flush
7732 028/300 Flush
7733 029/300 Write block 2758 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7734 030/300 Write block 4970 len 8192 data: 3 3
7735 031/300 Read block 3789 len 36864
7736 032/300 Flush
7737 033/300 Read block 184 len 28672
7738 034/300 Read block 3108 len 12288
7739 035/300 Read block 4340 len 16384
7740 036/300 Write block 2901 len 4096 data: 2
7741 037/300 Write block 2875 len 12288 data: 3 4 5
7742 038/300 Read block 1843 len 12288
7743 039/300 Read block 2150 len 24576
7744 040/300 Flush
7745 041/300 Write block 555 len 16384 data: 3 3 3 2
7746 042/300 Read block 3785 len 4096
7747 043/300 Read block 1141 len 12288
7748 044/300 Read block 1234 len 36864
7749 045/300 Write block 2471 len 40960 data: 3 3 4 4 4 4 4 4 4 5
7750 046/300 Read block 698 len 20480
7751 047/300 Write block 2582 len 20480 data: 2 2 2 3 3
7752 048/300 Flush
7753 049/300 Read block 1648 len 8192
7754 050/300 Read block 2591 len 24576
7755 051/300 Write block 2742 len 32768 data: 3 3 3 2 3 4 4 4
7756 052/300 Read block 4025 len 8192
7757 053/300 Write block 4436 len 8192 data: 2 2
7758 054/300 Read block 324 len 24576
7759 055/300 Read block 3819 len 8192
7760 056/300 Write block 208 len 16384 data: 2 3 2 2
7761 057/300 Write block 2658 len 32768 data: 3 3 3 2 3 3 2 2
7762 058/300 Read block 2443 len 16384
7763 059/300 Read block 4085 len 28672
7764 060/300 Write block 311 len 28672 data: 3 2 2 2 2 3 4
7765 061/300 Read block 2807 len 20480
7766 062/300 Read block 361 len 8192
7767 063/300 Read block 3496 len 16384
7768 064/300 Write block 2756 len 28672 data: 4 3 3 3 3 3 3
7769 065/300 Read block 1263 len 16384
7770 066/300 Read block 1978 len 40960
7771 067/300 Write block 3928 len 40960 data: 3 3 3 3 4 4 3 3 4 4
7772 068/300 Read block 3151 len 20480
7773 069/300 Write block 1472 len 24576 data: 2 2 2 2 2 2
7774 070/300 Write block 666 len 36864 data: 5 5 5 5 4 3 4 3 3
7775 071/300 Read block 1155 len 32768
7776 072/300 Read block 728 len 36864
7777 073/300 Write block 1123 len 32768 data: 4 4 5 6 6 6 6 6
7778 074/300 Read block 4220 len 12288
7779 075/300 Write block 3809 len 20480 data: 2 2 2 3 3
7780 076/300 Read block 4853 len 16384
7781 077/300 Read block 2639 len 36864
7782 078/300 Read block 2853 len 8192
7783 079/300 Read block 4328 len 24576
7784 080/300 Write block 2922 len 12288 data: 3 3 3
7785 081/300 Write block 533 len 4096 data: 3
7786 082/300 Write block 2600 len 8192 data: 3 3
7787 083/300 Read block 1403 len 20480
7788 084/300 Write block 4121 len 32768 data: 4 5 4 3 3 2 2 2
7789 085/300 Read block 3823 len 28672
7790 086/300 Read block 3704 len 20480
7791 087/300 Read block 1661 len 40960
7792 088/300 Read block 1357 len 16384
7793 089/300 Read block 4225 len 24576
7794 090/300 Write block 769 len 20480 data: 3 3 3 3 3
7795 091/300 Read block 484 len 16384
7796 092/300 Write block 1085 len 12288 data: 2 3 3
7797 093/300 Write block 4072 len 8192 data: 3 4
7798 094/300 Read block 3167 len 20480
7799 095/300 Flush
7800 096/300 Read block 474 len 16384
7801 097/300 Write block 1882 len 12288 data: 4 4 4
7802 098/300 Read block 2788 len 32768
7803 099/300 Read block 2459 len 36864
7804 100/300 Read block 843 len 4096
7805 101/300 Read block 2323 len 4096
7806 102/300 Read block 578 len 16384
7807 103/300 Read block 280 len 8192
7808 104/300 Write block 4156 len 8192 data: 2 2
7809 105/300 Write block 2235 len 16384 data: 3 3 3 3
7810 106/300 Read block 4933 len 16384
7811 107/300 Flush
7812 108/300 Read block 2618 len 28672
7813 109/300 Read block 3031 len 28672
7814 110/300 Write block 1911 len 24576 data: 3 3 3 3 2 2
7815 111/300 Read block 3289 len 16384
7816 112/300 Flush
7817 113/300 Read block 1017 len 8192
7818 114/300 Write block 456 len 20480 data: 3 3 3 3 3
7819 115/300 Flush
7820 116/300 Read block 1070 len 40960
7821 117/300 Write block 4538 len 4096 data: 2
7822 118/300 Read block 3186 len 8192
7823 119/300 Read block 3825 len 24576
7824 120/300 Write block 2188 len 20480 data: 5 5 4 4 4
7825 121/300 Write block 3005 len 32768 data: 3 3 3 3 3 4 3 4
7826 122/300 Read block 4801 len 32768
7827 123/300 Flush
7828 124/300 Read block 2316 len 8192
7829 125/300 Write block 3332 len 40960 data: 2 2 2 2 2 2 3 3 4 4
7830 126/300 Read block 4530 len 20480
7831 127/300 Write block 3438 len 36864 data: 2 2 2 2 2 2 2 2 2
7832 128/300 Flush
7833 129/300 Read block 1266 len 28672
7834 130/300 Write block 1522 len 32768 data: 2 2 2 3 3 3 2 2
7835 131/300 Read block 2418 len 12288
7836 132/300 Read block 387 len 40960
7837 133/300 Read block 3840 len 4096
7838 134/300 Read block 1584 len 32768
7839 135/300 Read block 2669 len 40960
7840 136/300 Read block 3939 len 20480
7841 137/300 Read block 1194 len 16384
7842 138/300 Read block 4077 len 40960
7843 139/300 Read block 1181 len 36864
7844 140/300 Read block 371 len 4096
7845 141/300 Flush
7846 142/300 Write block 1916 len 36864 data: 3 2 2 2 3 2 2 2 2
7847 143/300 Read block 948 len 8192
7848 144/300 Read block 2059 len 8192
7849 145/300 Read block 2740 len 24576
7850 146/300 Read block 851 len 20480
7851 147/300 Read block 3594 len 40960
7852 148/300 Read block 1644 len 20480
7853 149/300 Read block 4570 len 24576
7854 150/300 Write block 4334 len 16384 data: 3 3 3 3
7855 151/300 Read block 1781 len 24576
7856 152/300 Read block 4254 len 16384
7857 153/300 Write block 3508 len 28672 data: 4 4 4 4 4 3 3
7858 154/300 Write block 1399 len 40960 data: 4 3 3 3 3 3 3 3 2 3
7859 155/300 Read block 3840 len 16384
7860 156/300 Write block 2743 len 28672 data: 4 4 3 4 5 5 5
7861 157/300 Read block 2601 len 4096
7862 158/300 Write block 3347 len 36864 data: 4 5 4 4 4 3 3 4 4
7863 159/300 Read block 84 len 28672
7864 160/300 Write block 4681 len 20480 data: 2 2 2 2 2
7865 161/300 Read block 1679 len 28672
7866 162/300 Read block 48 len 4096
7867 163/300 Write block 137 len 24576 data: 2 2 3 3 4 4
7868 164/300 Read block 1424 len 12288
7869 165/300 Read block 4727 len 32768
7870 166/300 Write block 981 len 4096 data: 4
7871 167/300 Write block 4968 len 16384 data: 2 2 4 4
7872 168/300 Flush
7873 169/300 Write block 4543 len 4096 data: 3
7874 170/300 Write block 3342 len 12288 data: 4 4 4
7875 171/300 Read block 592 len 4096
7876 172/300 Write block 1035 len 8192 data: 2 2
7877 173/300 Read block 2001 len 12288
7878 174/300 Write block 3224 len 20480 data: 3 2 3 3 3
7879 175/300 Write block 1438 len 32768 data: 4 4 4 3 2 3 3 3
7880 176/300 Write block 1187 len 36864 data: 3 3 3 3 3 3 3 2 2
7881 177/300 Read block 1479 len 32768
7882 178/300 Write block 2188 len 40960 data: 6 6 5 5 5 4 3 3 3 2
7883 179/300 Read block 743 len 24576
7884 180/300 Write block 2025 len 12288 data: 2 2 2
7885 181/300 Write block 3252 len 8192 data: 3 2
7886 182/300 Read block 2164 len 16384
7887 183/300 Read block 4033 len 16384
7888 184/300 Write block 1237 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7889 185/300 Flush
7890 186/300 Write block 1288 len 36864 data: 2 2 2 2 2 3 3 3 3
7891 187/300 Write block 2233 len 32768 data: 3 4 4 4 4 4 4 3
7892 188/300 Read block 4307 len 24576
7893 189/300 Read block 534 len 28672
7894 190/300 Write block 505 len 8192 data: 5 5
7895 191/300 Flush
7896 192/300 Read block 165 len 8192
7897 193/300 Read block 3183 len 20480
7898 194/300 Write block 2950 len 28672 data: 4 5 5 5 5 5 4
7899 195/300 Write block 4624 len 20480 data: 2 3 2 2 2
7900 196/300 Write block 3267 len 36864 data: 3 2 2 2 2 2 2 2 2
7901 197/300 Read block 4248 len 12288
7902 198/300 Write block 2537 len 28672 data: 2 2 3 3 3 3 2
7903 199/300 Read block 2265 len 4096
7904 200/300 Read block 1280 len 28672
7905 201/300 Write block 2174 len 16384 data: 2 3 3 3
7906 202/300 Read block 2474 len 24576
7907 203/300 Write block 4771 len 36864 data: 2 2 2 2 2 3 3 3 3
7908 204/300 Write block 4290 len 24576 data: 2 2 3 3 3 3
7909 205/300 Write block 2152 len 4096 data: 2
7910 206/300 Write block 4810 len 32768 data: 3 3 3 3 4 4 4 4
7911 207/300 Read block 4306 len 20480
7912 208/300 Read block 2067 len 8192
7913 209/300 Read block 2179 len 36864
7914 210/300 Write block 3478 len 40960 data: 3 4 4 4 4 4 2 2 2 2
7915 211/300 Write block 2413 len 16384 data: 2 2 2 2
7916 212/300 Read block 3682 len 28672
7917 213/300 Write block 3249 len 32768 data: 5 4 3 4 3 2 2 2
7918 214/300 Read block 2418 len 40960
7919 215/300 Write block 4519 len 12288 data: 3 3 3
7920 216/300 Write block 1080 len 20480 data: 3 2 2 2 2
7921 217/300 Write block 4442 len 16384 data: 3 3 3 3
7922 218/300 Read block 4309 len 20480
7923 219/300 Read block 2927 len 4096
7924 220/300 Write block 1535 len 16384 data: 4 5 5 4
7925 221/300 Read block 3221 len 20480
7926 222/300 Read block 3172 len 28672
7927 223/300 Write block 1308 len 40960 data: 2 2 2 2 2 2 3 3 2 2
7928 224/300 Write block 1405 len 8192 data: 4 4
7929 225/300 Write block 523 len 24576 data: 3 3 3 2 2 2
7930 226/300 Read block 4417 len 8192
7931 227/300 Read block 1374 len 16384
7932 228/300 Read block 657 len 4096
7933 229/300 Read block 4529 len 24576
7934 230/300 Read block 2535 len 16384
7935 231/300 Flush
7936 232/300 Read block 1344 len 24576
7937 233/300 Write block 555 len 4096 data: 4
7938 234/300 Read block 3590 len 40960
7939 235/300 Write block 13 len 24576 data: 3 3 3 3 2 3
7940 236/300 Read block 158 len 32768
7941 237/300 Write block 4104 len 4096 data: 3
7942 238/300 Read block 2894 len 40960
7943 239/300 Write block 4794 len 40960 data: 3 3 2 2 2 2 2 2 2 2
7944 240/300 Read block 3303 len 24576
7945 241/300 Read block 1406 len 24576
7946 242/300 Write block 2618 len 32768 data: 4 4 4 4 3 3 3 3
7947 243/300 Read block 2786 len 32768
7948 244/300 Read block 2864 len 8192
7949 245/300 Read block 4386 len 40960
7950 246/300 Write block 202 len 4096 data: 3
7951 247/300 Read block 4564 len 40960
7952 248/300 Flush
7953 249/300 Read block 3678 len 36864
7954 250/300 Write block 2644 len 20480 data: 3 3 3 3 2
7955 251/300 Read block 4468 len 40960
7956 252/300 Flush
7957 253/300 Write block 2419 len 32768 data: 3 3 4 3 5 5 4 4
7958 254/300 Read block 1291 len 4096
7959 255/300 Write block 2762 len 40960 data: 4 3 3 3 3 3 2 2 2 2
7960 256/300 Read block 3415 len 8192
7961 257/300 Flush
7962 258/300 Read block 1214 len 20480
7963 259/300 Write block 1753 len 24576 data: 3 3 3 3 2 3
7964 260/300 Read block 4886 len 16384
7965 261/300 Read block 2420 len 28672
7966 262/300 Write block 1271 len 36864 data: 2 2 2 2 2 2 2 2 2
7967 263/300 Write block 969 len 8192 data: 2 3
7968 264/300 Read block 3251 len 28672
7969 265/300 Read block 3518 len 40960
7970 266/300 Read block 636 len 24576
7971 267/300 Read block 676 len 40960
7972 268/300 Write block 4271 len 20480 data: 2 2 3 3 3
7973 269/300 Flush
7974 270/300 Write block 1477 len 24576 data: 3 2 2 2 3 4
7975 271/300 Read block 1644 len 4096
7976 272/300 Read block 3793 len 12288
7977 273/300 Read block 2006 len 4096
7978 274/300 Write block 1533 len 12288 data: 5 4 5
7979 275/300 Read block 4807 len 36864
7980 276/300 Read block 2112 len 20480
7981 277/300 Write block 861 len 4096 data: 3
7982 278/300 Write block 3770 len 8192 data: 4 4
7983 279/300 Write block 4284 len 32768 data: 2 2 2 2 2 2 3 3
7984 280/300 Flush
7985 281/300 Read block 915 len 40960
7986 282/300 Read block 291 len 12288
7987 283/300 Write block 3610 len 20480 data: 4 3 2 2 3
7988 284/300 Read block 2251 len 20480
7989 285/300 Read block 3778 len 36864
7990 286/300 Read block 4018 len 32768
7991 287/300 Write block 1307 len 28672 data: 2 3 3 3 3 3 3
7992 288/300 Read block 4697 len 4096
7993 289/300 Write block 3906 len 32768 data: 2 2 2 2 3 3 4 3
7994 290/300 Read block 2550 len 16384
7995 291/300 Read block 3032 len 20480
7996 292/300 Flush
7997 293/300 Read block 3278 len 40960
7998 294/300 Read block 4851 len 4096
7999 295/300 Flush
8000 296/300 Read block 2779 len 32768
8001 297/300 Read block 2918 len 40960
8002 298/300 Read block 2119 len 24576
8003 299/300 Flush
8004 300/300 Read block 3461 len 28672
8005 301/300 Read block 3518 len 32768
8006 Main task has work to do, go find it
8007 got dsc Start(1)
8008 start 1
8009 Feb 21 06:47:05.236 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/1, method: POST, req_id: c6b57b69-364c-40d5-a527-cbcd9309e5f2, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
8010 Replay: started 1, returned:Ok(())
8011 ----------------------------------------------------------------
8012 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8013 States: Active Offline Active
8014 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8015 3114 Acked Write 7 Done Sent Done false
8016 3115 Acked Read 8 Done Sent Done false
8017 3116 Acked Read 7 Done Sent Done false
8018 3117 Acked Write 4 Done Sent Done false
8019 3118 Acked Write 2 Done Sent Done false
8020 3119 Acked Write 6 Done Sent Done false
8021 3120 Acked Read 6 Done Sent Done false
8022 3121 Acked Read 6 Done Sent Done false
8023 3122 Acked Write 4 Done Sent Done false
8024 3123 Acked Write 2 Done Sent Done false
8025 3124 Acked Read 7 Done Sent Done false
8026 3125 Acked Write 6 Done Sent Done false
8027 3126 Acked Read 4 Done Sent Done false
8028 3127 Acked Read 2 Done Sent Done false
8029 3128 Acked Write 9 Done Sent Done false
8030 3129 Acked Write 5 Done Sent Done false
8031 3130 Acked Read 9 Done Sent Done false
8032 3131 Acked Read 9 Done Sent Done false
8033 3132 Acked Read 1 Done Sent Done false
8034 3133 Acked Flush 0 Done Sent Done false
8035 3134 Acked Write 10 Done Sent Done false
8036 3135 Acked Read 9 Done Sent Done false
8037 3136 Acked Write 8 Done Sent Done false
8038 3137 Acked Read 6 Done Sent Done false
8039 3138 Acked Read 7 Done Sent Done false
8040 3139 Acked Read 6 Done Sent Done false
8041 3140 Acked Flush 0 Done Sent Done false
8042 3141 Acked Flush 0 Done Sent Done false
8043 3142 Acked Write 10 Done Sent Done false
8044 3143 Acked Write 2 Done Sent Done false
8045 3144 Acked Read 9 Done Sent Done false
8046 3145 Acked Flush 0 Done Sent Done false
8047 3146 Acked Read 7 Done Sent Done false
8048 3147 Acked Read 3 Done Sent Done false
8049 3148 Acked Read 4 Done Sent Done false
8050 3149 Acked Write 1 Done Sent Done false
8051 3150 Acked Write 3 Done Sent Done false
8052 3151 Acked Read 3 Done Sent Done false
8053 3152 Acked Read 6 Done Sent Done false
8054 3153 Acked Flush 0 Done Sent Done false
8055 3154 Acked Write 4 Done Sent Done false
8056 3155 Acked Read 1 Done Sent Done false
8057 3156 Acked Read 3 Done Sent Done false
8058 3157 Acked Read 9 Done Sent Done false
8059 3158 Acked Write 10 Done Sent Done false
8060 3159 Acked Read 5 Done Sent Done false
8061 3160 Acked Write 5 Done Sent Done false
8062 3161 Acked Flush 0 Done Sent Done false
8063 3162 Acked Read 2 Done Sent Done false
8064 3163 Acked Read 6 Done Sent Done false
8065 3164 Acked Write 8 Done Sent Done false
8066 3165 Acked Read 2 Done Sent Done false
8067 3166 Acked Write 2 Done Sent Done false
8068 3167 Acked Read 6 Done Sent Done false
8069 3168 Acked Read 2 Done Sent Done false
8070 3169 Acked Write 4 Done Sent Done false
8071 3170 Acked Write 8 Done Sent Done false
8072 3171 Acked Read 4 Done Sent Done false
8073 3172 Acked Read 7 Done Sent Done false
8074 3173 Acked Write 7 Done Sent Done false
8075 3174 Acked Read 5 Done Sent Done false
8076 3175 Acked Read 2 Done Sent Done false
8077 3176 Acked Read 4 Done Sent Done false
8078 3177 Acked Write 7 Done Sent Done false
8079 3178 Acked Read 4 Done Sent Done false
8080 3179 Acked Read 10 Done Sent Done false
8081 3180 Acked Write 10 Done Sent Done false
8082 3181 Acked Read 5 Done Sent Done false
8083 3182 Acked Write 6 Done Sent Done false
8084 3183 Acked Write 9 Done Sent Done false
8085 3184 Acked Read 8 Done Sent Done false
8086 3185 Acked Read 9 Done Sent Done false
8087 3186 Acked Write 8 Done Sent Done false
8088 3187 Acked Read 3 Done Sent Done false
8089 3188 Acked Write 5 Done Sent Done false
8090 3189 Acked Read 4 Done Sent Done false
8091 3190 Acked Read 9 Done Sent Done false
8092 3191 Acked Read 2 Done Sent Done false
8093 3192 Acked Read 6 Done Sent Done false
8094 3193 Acked Write 3 Done Sent Done false
8095 3194 Acked Write 1 Done Sent Done false
8096 3195 Acked Write 2 Done Sent Done false
8097 3196 Acked Read 5 Done Sent Done false
8098 3197 Acked Write 8 Done Sent Done false
8099 3198 Acked Read 7 Done Sent Done false
8100 3199 Acked Read 5 Done Sent Done false
8101 3200 Acked Read 10 Done Sent Done false
8102 3201 Acked Read 4 Done Sent Done false
8103 3202 Acked Read 6 Done Sent Done false
8104 3203 Acked Write 5 Done Sent Done false
8105 3204 Acked Read 4 Done Sent Done false
8106 3205 Acked Write 3 Done Sent Done false
8107 3206 Acked Write 2 Done Sent Done false
8108 3207 Acked Read 5 Done Sent Done false
8109 3208 Acked Flush 0 Done Sent Done false
8110 3209 Acked Read 4 Done Sent Done false
8111 3210 Acked Write 3 Done Sent Done false
8112 3211 Acked Read 8 Done Sent Done false
8113 3212 Acked Read 9 Done Sent Done false
8114 3213 Acked Read 1 Done Sent Done false
8115 3214 Acked Read 1 Done Sent Done false
8116 3215 Acked Read 4 Done Sent Done false
8117 3216 Acked Read 2 Done Sent Done false
8118 3217 Acked Write 2 Done Sent Done false
8119 3218 Acked Write 4 Done Sent Done false
8120 3219 Acked Read 4 Done Sent Done false
8121 3220 Acked Flush 0 Done Sent Done false
8122 3221 Acked Read 7 Done Sent Done false
8123 3222 Acked Read 7 Done Sent Done false
8124 3223 Acked Write 6 Done Sent Done false
8125 3224 Acked Read 4 Done Sent Done false
8126 3225 Acked Flush 0 Done Sent Done false
8127 3226 Acked Read 2 Done Sent Done false
8128 3227 Acked Write 5 Done Sent Done false
8129 3228 Acked Flush 0 Done Sent Done false
8130 3229 Acked Read 10 Done Sent Done false
8131 3230 Acked Write 1 Done Sent Done false
8132 3231 Acked Read 2 Done Sent Done false
8133 3232 Acked Read 6 Done Sent Done false
8134 3233 Acked Write 5 Done Sent Done false
8135 3234 Acked Write 8 Done Sent Done false
8136 3235 Acked Read 8 Done Sent Done false
8137 3236 Acked Flush 0 Done Sent Done false
8138 3237 Acked Read 2 Done Sent Done false
8139 3238 Acked Write 10 Done Sent Done false
8140 3239 Acked Read 5 Done Sent Done false
8141 3240 Acked Write 9 Done Sent Done false
8142 3241 Acked Flush 0 Done Sent Done false
8143 3242 Acked Read 7 Done Sent Done false
8144 3243 Acked Write 8 Done Sent Done false
8145 3244 Acked Read 3 Done Sent Done false
8146 3245 Acked Read 10 Done Sent Done false
8147 3246 Acked Read 1 Done Sent Done false
8148 3247 Acked Read 8 Done Sent Done false
8149 3248 Acked Read 10 Done Sent Done false
8150 3249 Acked Read 5 Done Sent Done false
8151 3250 Acked Read 4 Done Sent Done false
8152 3251 Acked Read 10 Done Sent Done false
8153 3252 Acked Read 9 Done Sent Done false
8154 3253 Acked Read 1 Done Sent Done false
8155 3254 Acked Flush 0 Done Sent Done false
8156 3255 Acked Write 9 Done Sent Done false
8157 3256 Acked Read 2 Done Sent Done false
8158 3257 Acked Read 2 Done Sent Done false
8159 3258 Acked Read 6 Done Sent Done false
8160 3259 Acked Read 5 Done Sent Done false
8161 3260 Acked Read 10 Done Sent Done false
8162 3261 Acked Read 5 Done Sent Done false
8163 3262 Acked Read 6 Done Sent Done false
8164 3263 Acked Write 4 Done Sent Done false
8165 3264 Acked Read 6 Done Sent Done false
8166 3265 Acked Read 4 Done Sent Done false
8167 3266 Acked Write 7 Done Sent Done false
8168 3267 Acked Write 10 Done Sent Done false
8169 3268 Acked Read 4 Done Sent Done false
8170 3269 Acked Write 7 Done Sent Done false
8171 3270 Acked Read 1 Done Sent Done false
8172 3271 Acked Write 9 Done Sent Done false
8173 3272 Acked Read 7 Done Sent Done false
8174 3273 Acked Write 5 Done Sent Done false
8175 3274 Acked Read 7 Done Sent Done false
8176 3275 Acked Read 1 Done Sent Done false
8177 3276 Acked Write 6 Done Sent Done false
8178 3277 Acked Read 3 Done Sent Done false
8179 3278 Acked Read 8 Done Sent Done false
8180 3279 Acked Write 1 Done Sent Done false
8181 3280 Acked Write 4 Done Sent Done false
8182 3281 Acked Flush 0 Done Sent Done false
8183 3282 Acked Write 1 Done Sent Done false
8184 3283 Acked Write 3 Done Sent Done false
8185 3284 Acked Read 1 Done Sent Done false
8186 3285 Acked Write 2 Done Sent Done false
8187 3286 Acked Read 3 Done Sent Done false
8188 3287 Acked Write 5 Done Sent Done false
8189 3288 Acked Write 8 Done Sent Done false
8190 3289 Acked Write 9 Done Sent Done false
8191 3290 Acked Read 8 Done Sent Done false
8192 3291 Acked Write 10 Done Sent Done false
8193 3292 Acked Read 6 Done Sent Done false
8194 3293 Acked Write 3 Done Sent Done false
8195 3294 Acked Write 2 Done Sent Done false
8196 3295 Acked Read 4 Done Sent Done false
8197 3296 Acked Read 4 Done Sent Done false
8198 3297 Acked Write 10 Done Sent Done false
8199 3298 Acked Flush 0 Done Sent Done false
8200 3299 Acked Write 9 Done Sent Done false
8201 3300 Acked Write 8 Done Sent Done false
8202 3301 Acked Read 6 Done Sent Done false
8203 3302 Acked Read 7 Done Sent Done false
8204 3303 Acked Write 2 Done Sent Done false
8205 3304 Acked Flush 0 Done Sent Done false
8206 3305 Acked Read 2 Done Sent Done false
8207 3306 Acked Read 5 Done Sent Done false
8208 3307 Acked Write 7 Done Sent Done false
8209 3308 Acked Write 5 Done Sent Done false
8210 3309 Acked Write 9 Done Sent Done false
8211 3310 Acked Read 3 Done Sent Done false
8212 3311 Acked Write 7 Done Sent Done false
8213 3312 Acked Read 1 Done Sent Done false
8214 3313 Acked Read 7 Done Sent Done false
8215 3314 Acked Write 4 Done Sent Done false
8216 3315 Acked Read 6 Done Sent Done false
8217 3316 Acked Write 9 Done Sent Done false
8218 3317 Acked Write 6 Done Sent Done false
8219 3318 Acked Write 1 Done Sent Done false
8220 3319 Acked Write 8 Done Sent Done false
8221 3320 Acked Read 5 Done Sent Done false
8222 3321 Acked Read 2 Done Sent Done false
8223 3322 Acked Read 9 Done Sent Done false
8224 3323 Acked Write 10 Done Sent Done false
8225 3324 Acked Write 4 Done Sent Done false
8226 3325 Acked Read 7 Done Sent Done false
8227 3326 Acked Write 8 Done Sent Done false
8228 3327 Acked Read 10 Done Sent Done false
8229 3328 Acked Write 3 Done Sent Done false
8230 3329 Acked Write 5 Done Sent Done false
8231 3330 Acked Write 4 Done Sent Done false
8232 3331 Acked Read 5 Done Sent Done false
8233 3332 Acked Read 1 Done Sent Done false
8234 3333 Acked Write 4 Done Sent Done false
8235 3334 Acked Read 5 Done Sent Done false
8236 3335 Acked Read 7 Done Sent Done false
8237 3336 Acked Write 10 Done Sent Done false
8238 3337 Acked Write 2 Done Sent Done false
8239 3338 Acked Write 6 Done Sent Done false
8240 3339 Acked Read 2 Done Sent Done false
8241 3340 Acked Read 4 Done Sent Done false
8242 3341 Acked Read 1 Done Sent Done false
8243 3342 Acked Read 6 Done Sent Done false
8244 3343 Acked Read 4 Done Sent Done false
8245 3344 Acked Flush 0 Done Sent Done false
8246 3345 Acked Read 6 Done Sent Done false
8247 3346 Acked Write 1 Done Sent Done false
8248 3347 Acked Read 10 Done Sent Done false
8249 3348 Acked Write 6 Done Sent Done false
8250 3349 Acked Read 8 Done Sent Done false
8251 3350 Acked Write 1 Done Sent Done false
8252 3351 Acked Read 10 Done Sent Done false
8253 3352 Acked Write 10 Done Sent Done false
8254 3353 Acked Read 6 Done Sent Done false
8255 3354 Acked Read 6 Done Sent Done false
8256 3355 Acked Write 8 Done Sent Done false
8257 3356 Acked Read 8 Done Sent Done false
8258 3357 Acked Read 2 Done Sent Done false
8259 3358 Acked Read 10 Done Sent Done false
8260 3359 Acked Write 1 Done Sent Done false
8261 3360 Acked Read 10 Done Sent Done false
8262 3361 Acked Flush 0 Done Sent Done false
8263 3362 Acked Read 9 Done Sent Done false
8264 3363 Acked Write 5 Done Sent Done false
8265 3364 Acked Read 10 Done Sent Done false
8266 3365 Acked Flush 0 Done Sent Done false
8267 3366 Acked Write 8 Done Sent Done false
8268 3367 Acked Read 1 Done Sent Done false
8269 3368 Acked Write 10 Done Sent Done false
8270 3369 Acked Read 2 Done Sent Done false
8271 3370 Acked Flush 0 Done Sent Done false
8272 3371 Acked Read 5 Done Sent Done false
8273 3372 Acked Write 6 Done Sent Done false
8274 3373 Acked Read 4 Done Sent Done false
8275 3374 Acked Read 7 Done Sent Done false
8276 3375 Acked Write 9 Done Sent Done false
8277 3376 Acked Write 2 Done Sent Done false
8278 3377 Acked Read 7 Done Sent Done false
8279 3378 Acked Read 10 Done Sent Done false
8280 3379 Acked Read 6 Done Sent Done false
8281 3380 Acked Read 10 Done Sent Done false
8282 3381 Acked Write 5 Done Sent Done false
8283 3382 Acked Flush 0 Done Sent Done false
8284 3383 Acked Write 6 Done Sent Done false
8285 3384 Acked Read 1 Done Sent Done false
8286 3385 Acked Read 3 Done Sent Done false
8287 3386 Acked Read 1 Done Sent Done false
8288 3387 Acked Write 3 Done Sent Done false
8289 3388 Acked Read 9 Done Sent Done false
8290 3389 Acked Read 5 Done Sent Done false
8291 3390 Acked Write 1 Done Sent Done false
8292 3391 Acked Write 2 Done Sent Done false
8293 3392 Acked Write 8 Done Sent Done false
8294 3393 Acked Flush 0 Done Sent Done false
8295 3394 Acked Read 10 Done Sent Done false
8296 3395 Acked Read 3 Done Sent Done false
8297 3396 Acked Write 5 Done Sent Done false
8298 3397 Acked Read 5 Done Sent Done false
8299 3398 Acked Read 9 Done Sent Done false
8300 3399 Acked Read 8 Done Sent Done false
8301 3400 Acked Write 7 Done Sent Done false
8302 3401 Acked Read 1 Done Sent Done false
8303 3402 Acked Write 8 Done Sent Done false
8304 3403 Acked Read 4 Done Sent Done false
8305 3404 Acked Read 5 Done Sent Done false
8306 3405 Acked Flush 0 Done Sent Done false
8307 3406 Acked Read 10 Done Sent Done false
8308 3407 Acked Read 1 Done Sent Done false
8309 3408 Acked Flush 0 Done Sent Done false
8310 3409 Acked Read 8 Done Sent Done false
8311 3410 Acked Read 10 Done Sent Done false
8312 3411 Acked Read 6 Done Sent Done false
8313 3412 Acked Flush 0 Done Sent Done false
8314 3413 Acked Read 7 Done Sent Done false
8315 3414 Acked Read 8 Done Sent Done false
8316 STATES DS:0 DS:1 DS:2 TOTAL
8317 Sent 0 301 0 301
8318 Done 301 0 301 602
8319 Skipped 0 0 0 0
8320 Error 0 0 0 0
8321 Last Flush: Some(JobId(3412)) Some(JobId(3113)) Some(JobId(3412))
8322 Downstairs last five completed: 3113 3112 3111 3110 3109
8323 Upstairs last five completed: 3414 3413 3412 3411 3410
8324 CLIENT: Up:0 ds:301 act:2
83252025-02-21T06:47:05.815ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
83262025-02-21T06:47:07.232ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
83272025-02-21T06:47:07.818ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
83282025-02-21T06:47:08.318ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
83292025-02-21T06:47:08.820ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
8330 ----------------------------------------------------------------
8331 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8332 States: Active Offline Active
8333 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8334 3114 Acked Write 7 Done Sent Done false
8335 3115 Acked Read 8 Done Sent Done false
8336 3116 Acked Read 7 Done Sent Done false
8337 3117 Acked Write 4 Done Sent Done false
8338 3118 Acked Write 2 Done Sent Done false
8339 3119 Acked Write 6 Done Sent Done false
8340 3120 Acked Read 6 Done Sent Done false
8341 3121 Acked Read 6 Done Sent Done false
8342 3122 Acked Write 4 Done Sent Done false
8343 3123 Acked Write 2 Done Sent Done false
8344 3124 Acked Read 7 Done Sent Done false
8345 3125 Acked Write 6 Done Sent Done false
8346 3126 Acked Read 4 Done Sent Done false
8347 3127 Acked Read 2 Done Sent Done false
8348 3128 Acked Write 9 Done Sent Done false
8349 3129 Acked Write 5 Done Sent Done false
8350 3130 Acked Read 9 Done Sent Done false
8351 3131 Acked Read 9 Done Sent Done false
8352 3132 Acked Read 1 Done Sent Done false
8353 3133 Acked Flush 0 Done Sent Done false
8354 3134 Acked Write 10 Done Sent Done false
8355 3135 Acked Read 9 Done Sent Done false
8356 3136 Acked Write 8 Done Sent Done false
8357 3137 Acked Read 6 Done Sent Done false
8358 3138 Acked Read 7 Done Sent Done false
8359 3139 Acked Read 6 Done Sent Done false
8360 3140 Acked Flush 0 Done Sent Done false
8361 3141 Acked Flush 0 Done Sent Done false
8362 3142 Acked Write 10 Done Sent Done false
8363 3143 Acked Write 2 Done Sent Done false
8364 3144 Acked Read 9 Done Sent Done false
8365 3145 Acked Flush 0 Done Sent Done false
8366 3146 Acked Read 7 Done Sent Done false
8367 3147 Acked Read 3 Done Sent Done false
8368 3148 Acked Read 4 Done Sent Done false
8369 3149 Acked Write 1 Done Sent Done false
8370 3150 Acked Write 3 Done Sent Done false
8371 3151 Acked Read 3 Done Sent Done false
8372 3152 Acked Read 6 Done Sent Done false
8373 3153 Acked Flush 0 Done Sent Done false
8374 3154 Acked Write 4 Done Sent Done false
8375 3155 Acked Read 1 Done Sent Done false
8376 3156 Acked Read 3 Done Sent Done false
8377 3157 Acked Read 9 Done Sent Done false
8378 3158 Acked Write 10 Done Sent Done false
8379 3159 Acked Read 5 Done Sent Done false
8380 3160 Acked Write 5 Done Sent Done false
8381 3161 Acked Flush 0 Done Sent Done false
8382 3162 Acked Read 2 Done Sent Done false
8383 3163 Acked Read 6 Done Sent Done false
8384 3164 Acked Write 8 Done Sent Done false
8385 3165 Acked Read 2 Done Sent Done false
8386 3166 Acked Write 2 Done Sent Done false
8387 3167 Acked Read 6 Done Sent Done false
8388 3168 Acked Read 2 Done Sent Done false
8389 3169 Acked Write 4 Done Sent Done false
8390 3170 Acked Write 8 Done Sent Done false
8391 3171 Acked Read 4 Done Sent Done false
8392 3172 Acked Read 7 Done Sent Done false
8393 3173 Acked Write 7 Done Sent Done false
8394 3174 Acked Read 5 Done Sent Done false
8395 3175 Acked Read 2 Done Sent Done false
8396 3176 Acked Read 4 Done Sent Done false
8397 3177 Acked Write 7 Done Sent Done false
8398 3178 Acked Read 4 Done Sent Done false
8399 3179 Acked Read 10 Done Sent Done false
8400 3180 Acked Write 10 Done Sent Done false
8401 3181 Acked Read 5 Done Sent Done false
8402 3182 Acked Write 6 Done Sent Done false
8403 3183 Acked Write 9 Done Sent Done false
8404 3184 Acked Read 8 Done Sent Done false
8405 3185 Acked Read 9 Done Sent Done false
8406 3186 Acked Write 8 Done Sent Done false
8407 3187 Acked Read 3 Done Sent Done false
8408 3188 Acked Write 5 Done Sent Done false
8409 3189 Acked Read 4 Done Sent Done false
8410 3190 Acked Read 9 Done Sent Done false
8411 3191 Acked Read 2 Done Sent Done false
8412 3192 Acked Read 6 Done Sent Done false
8413 3193 Acked Write 3 Done Sent Done false
8414 3194 Acked Write 1 Done Sent Done false
8415 3195 Acked Write 2 Done Sent Done false
8416 3196 Acked Read 5 Done Sent Done false
8417 3197 Acked Write 8 Done Sent Done false
8418 3198 Acked Read 7 Done Sent Done false
8419 3199 Acked Read 5 Done Sent Done false
8420 3200 Acked Read 10 Done Sent Done false
8421 3201 Acked Read 4 Done Sent Done false
8422 3202 Acked Read 6 Done Sent Done false
8423 3203 Acked Write 5 Done Sent Done false
8424 3204 Acked Read 4 Done Sent Done false
8425 3205 Acked Write 3 Done Sent Done false
8426 3206 Acked Write 2 Done Sent Done false
8427 3207 Acked Read 5 Done Sent Done false
8428 3208 Acked Flush 0 Done Sent Done false
8429 3209 Acked Read 4 Done Sent Done false
8430 3210 Acked Write 3 Done Sent Done false
8431 3211 Acked Read 8 Done Sent Done false
8432 3212 Acked Read 9 Done Sent Done false
8433 3213 Acked Read 1 Done Sent Done false
8434 3214 Acked Read 1 Done Sent Done false
8435 3215 Acked Read 4 Done Sent Done false
8436 3216 Acked Read 2 Done Sent Done false
8437 3217 Acked Write 2 Done Sent Done false
8438 3218 Acked Write 4 Done Sent Done false
8439 3219 Acked Read 4 Done Sent Done false
8440 3220 Acked Flush 0 Done Sent Done false
8441 3221 Acked Read 7 Done Sent Done false
8442 3222 Acked Read 7 Done Sent Done false
8443 3223 Acked Write 6 Done Sent Done false
8444 3224 Acked Read 4 Done Sent Done false
8445 3225 Acked Flush 0 Done Sent Done false
8446 3226 Acked Read 2 Done Sent Done false
8447 3227 Acked Write 5 Done Sent Done false
8448 3228 Acked Flush 0 Done Sent Done false
8449 3229 Acked Read 10 Done Sent Done false
8450 3230 Acked Write 1 Done Sent Done false
8451 3231 Acked Read 2 Done Sent Done false
8452 3232 Acked Read 6 Done Sent Done false
8453 3233 Acked Write 5 Done Sent Done false
8454 3234 Acked Write 8 Done Sent Done false
8455 3235 Acked Read 8 Done Sent Done false
8456 3236 Acked Flush 0 Done Sent Done false
8457 3237 Acked Read 2 Done Sent Done false
8458 3238 Acked Write 10 Done Sent Done false
8459 3239 Acked Read 5 Done Sent Done false
8460 3240 Acked Write 9 Done Sent Done false
8461 3241 Acked Flush 0 Done Sent Done false
8462 3242 Acked Read 7 Done Sent Done false
8463 3243 Acked Write 8 Done Sent Done false
8464 3244 Acked Read 3 Done Sent Done false
8465 3245 Acked Read 10 Done Sent Done false
8466 3246 Acked Read 1 Done Sent Done false
8467 3247 Acked Read 8 Done Sent Done false
8468 3248 Acked Read 10 Done Sent Done false
8469 3249 Acked Read 5 Done Sent Done false
8470 3250 Acked Read 4 Done Sent Done false
8471 3251 Acked Read 10 Done Sent Done false
8472 3252 Acked Read 9 Done Sent Done false
8473 3253 Acked Read 1 Done Sent Done false
8474 3254 Acked Flush 0 Done Sent Done false
8475 3255 Acked Write 9 Done Sent Done false
8476 3256 Acked Read 2 Done Sent Done false
8477 3257 Acked Read 2 Done Sent Done false
8478 3258 Acked Read 6 Done Sent Done false
8479 3259 Acked Read 5 Done Sent Done false
8480 3260 Acked Read 10 Done Sent Done false
8481 3261 Acked Read 5 Done Sent Done false
8482 3262 Acked Read 6 Done Sent Done false
8483 3263 Acked Write 4 Done Sent Done false
8484 3264 Acked Read 6 Done Sent Done false
8485 3265 Acked Read 4 Done Sent Done false
8486 3266 Acked Write 7 Done Sent Done false
8487 3267 Acked Write 10 Done Sent Done false
8488 3268 Acked Read 4 Done Sent Done false
8489 3269 Acked Write 7 Done Sent Done false
8490 3270 Acked Read 1 Done Sent Done false
8491 3271 Acked Write 9 Done Sent Done false
8492 3272 Acked Read 7 Done Sent Done false
8493 3273 Acked Write 5 Done Sent Done false
8494 3274 Acked Read 7 Done Sent Done false
8495 3275 Acked Read 1 Done Sent Done false
8496 3276 Acked Write 6 Done Sent Done false
8497 3277 Acked Read 3 Done Sent Done false
8498 3278 Acked Read 8 Done Sent Done false
8499 3279 Acked Write 1 Done Sent Done false
8500 3280 Acked Write 4 Done Sent Done false
8501 3281 Acked Flush 0 Done Sent Done false
8502 3282 Acked Write 1 Done Sent Done false
8503 3283 Acked Write 3 Done Sent Done false
8504 3284 Acked Read 1 Done Sent Done false
8505 3285 Acked Write 2 Done Sent Done false
8506 3286 Acked Read 3 Done Sent Done false
8507 3287 Acked Write 5 Done Sent Done false
8508 3288 Acked Write 8 Done Sent Done false
8509 3289 Acked Write 9 Done Sent Done false
8510 3290 Acked Read 8 Done Sent Done false
8511 3291 Acked Write 10 Done Sent Done false
8512 3292 Acked Read 6 Done Sent Done false
8513 3293 Acked Write 3 Done Sent Done false
8514 3294 Acked Write 2 Done Sent Done false
8515 3295 Acked Read 4 Done Sent Done false
8516 3296 Acked Read 4 Done Sent Done false
8517 3297 Acked Write 10 Done Sent Done false
8518 3298 Acked Flush 0 Done Sent Done false
8519 3299 Acked Write 9 Done Sent Done false
8520 3300 Acked Write 8 Done Sent Done false
8521 3301 Acked Read 6 Done Sent Done false
8522 3302 Acked Read 7 Done Sent Done false
8523 3303 Acked Write 2 Done Sent Done false
8524 3304 Acked Flush 0 Done Sent Done false
8525 3305 Acked Read 2 Done Sent Done false
8526 3306 Acked Read 5 Done Sent Done false
8527 3307 Acked Write 7 Done Sent Done false
8528 3308 Acked Write 5 Done Sent Done false
8529 3309 Acked Write 9 Done Sent Done false
8530 3310 Acked Read 3 Done Sent Done false
8531 3311 Acked Write 7 Done Sent Done false
8532 3312 Acked Read 1 Done Sent Done false
8533 3313 Acked Read 7 Done Sent Done false
8534 3314 Acked Write 4 Done Sent Done false
8535 3315 Acked Read 6 Done Sent Done false
8536 3316 Acked Write 9 Done Sent Done false
8537 3317 Acked Write 6 Done Sent Done false
8538 3318 Acked Write 1 Done Sent Done false
8539 3319 Acked Write 8 Done Sent Done false
8540 3320 Acked Read 5 Done Sent Done false
8541 3321 Acked Read 2 Done Sent Done false
8542 3322 Acked Read 9 Done Sent Done false
8543 3323 Acked Write 10 Done Sent Done false
8544 3324 Acked Write 4 Done Sent Done false
8545 3325 Acked Read 7 Done Sent Done false
8546 3326 Acked Write 8 Done Sent Done false
8547 3327 Acked Read 10 Done Sent Done false
8548 3328 Acked Write 3 Done Sent Done false
8549 3329 Acked Write 5 Done Sent Done false
8550 3330 Acked Write 4 Done Sent Done false
8551 3331 Acked Read 5 Done Sent Done false
8552 3332 Acked Read 1 Done Sent Done false
8553 3333 Acked Write 4 Done Sent Done false
8554 3334 Acked Read 5 Done Sent Done false
8555 3335 Acked Read 7 Done Sent Done false
8556 3336 Acked Write 10 Done Sent Done false
8557 3337 Acked Write 2 Done Sent Done false
8558 3338 Acked Write 6 Done Sent Done false
8559 3339 Acked Read 2 Done Sent Done false
8560 3340 Acked Read 4 Done Sent Done false
8561 3341 Acked Read 1 Done Sent Done false
8562 3342 Acked Read 6 Done Sent Done false
8563 3343 Acked Read 4 Done Sent Done false
8564 3344 Acked Flush 0 Done Sent Done false
8565 3345 Acked Read 6 Done Sent Done false
8566 3346 Acked Write 1 Done Sent Done false
8567 3347 Acked Read 10 Done Sent Done false
8568 3348 Acked Write 6 Done Sent Done false
8569 3349 Acked Read 8 Done Sent Done false
8570 3350 Acked Write 1 Done Sent Done false
8571 3351 Acked Read 10 Done Sent Done false
8572 3352 Acked Write 10 Done Sent Done false
8573 3353 Acked Read 6 Done Sent Done false
8574 3354 Acked Read 6 Done Sent Done false
8575 3355 Acked Write 8 Done Sent Done false
8576 3356 Acked Read 8 Done Sent Done false
8577 3357 Acked Read 2 Done Sent Done false
8578 3358 Acked Read 10 Done Sent Done false
8579 3359 Acked Write 1 Done Sent Done false
8580 3360 Acked Read 10 Done Sent Done false
8581 3361 Acked Flush 0 Done Sent Done false
8582 3362 Acked Read 9 Done Sent Done false
8583 3363 Acked Write 5 Done Sent Done false
8584 3364 Acked Read 10 Done Sent Done false
8585 3365 Acked Flush 0 Done Sent Done false
8586 3366 Acked Write 8 Done Sent Done false
8587 3367 Acked Read 1 Done Sent Done false
8588 3368 Acked Write 10 Done Sent Done false
8589 3369 Acked Read 2 Done Sent Done false
8590 3370 Acked Flush 0 Done Sent Done false
8591 3371 Acked Read 5 Done Sent Done false
8592 3372 Acked Write 6 Done Sent Done false
8593 3373 Acked Read 4 Done Sent Done false
8594 3374 Acked Read 7 Done Sent Done false
8595 3375 Acked Write 9 Done Sent Done false
8596 3376 Acked Write 2 Done Sent Done false
8597 3377 Acked Read 7 Done Sent Done false
8598 3378 Acked Read 10 Done Sent Done false
8599 3379 Acked Read 6 Done Sent Done false
8600 3380 Acked Read 10 Done Sent Done false
8601 3381 Acked Write 5 Done Sent Done false
8602 3382 Acked Flush 0 Done Sent Done false
8603 3383 Acked Write 6 Done Sent Done false
8604 3384 Acked Read 1 Done Sent Done false
8605 3385 Acked Read 3 Done Sent Done false
8606 3386 Acked Read 1 Done Sent Done false
8607 3387 Acked Write 3 Done Sent Done false
8608 3388 Acked Read 9 Done Sent Done false
8609 3389 Acked Read 5 Done Sent Done false
8610 3390 Acked Write 1 Done Sent Done false
8611 3391 Acked Write 2 Done Sent Done false
8612 3392 Acked Write 8 Done Sent Done false
8613 3393 Acked Flush 0 Done Sent Done false
8614 3394 Acked Read 10 Done Sent Done false
8615 3395 Acked Read 3 Done Sent Done false
8616 3396 Acked Write 5 Done Sent Done false
8617 3397 Acked Read 5 Done Sent Done false
8618 3398 Acked Read 9 Done Sent Done false
8619 3399 Acked Read 8 Done Sent Done false
8620 3400 Acked Write 7 Done Sent Done false
8621 3401 Acked Read 1 Done Sent Done false
8622 3402 Acked Write 8 Done Sent Done false
8623 3403 Acked Read 4 Done Sent Done false
8624 3404 Acked Read 5 Done Sent Done false
8625 3405 Acked Flush 0 Done Sent Done false
8626 3406 Acked Read 10 Done Sent Done false
8627 3407 Acked Read 1 Done Sent Done false
8628 3408 Acked Flush 0 Done Sent Done false
8629 3409 Acked Read 8 Done Sent Done false
8630 3410 Acked Read 10 Done Sent Done false
8631 3411 Acked Read 6 Done Sent Done false
8632 3412 Acked Flush 0 Done Sent Done false
8633 3413 Acked Read 7 Done Sent Done false
8634 3414 Acked Read 8 Done Sent Done false
8635 STATES DS:0 DS:1 DS:2 TOTAL
8636 Sent 0 301 0 301
8637 Done 301 0 301 602
8638 Skipped 0 0 0 0
8639 Error 0 0 0 0
8640 Last Flush: Some(JobId(3412)) Some(JobId(3113)) Some(JobId(3412))
8641 Downstairs last five completed: 3113 3112 3111 3110 3109
8642 Upstairs last five completed: 3414 3413 3412 3411 3410
8643 CLIENT: Up:0 ds:301 act:2
86442025-02-21T06:47:09.972ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
86452025-02-21T06:47:10.822ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
86462025-02-21T06:47:10.983ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
86472025-02-21T06:47:10.983ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
86482025-02-21T06:47:10.983ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
86492025-02-21T06:47:10.983ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
86502025-02-21T06:47:10.983ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
8651 [8820] got start action
8652 [8820] I am going to restart
8653 Starting downstairs at port 8820
8654 Make output file at "/tmp/dsc/downstairs-8820.txt"
8655 [8820][1] reports Starting
8656 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1214)
8657 [8820][1] reports Running
86582025-02-21T06:47:12.234ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
86592025-02-21T06:47:12.825ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
8660 ----------------------------------------------------------------
8661 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8662 States: Active Offline Active
8663 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8664 3114 Acked Write 7 Done Sent Done false
8665 3115 Acked Read 8 Done Sent Done false
8666 3116 Acked Read 7 Done Sent Done false
8667 3117 Acked Write 4 Done Sent Done false
8668 3118 Acked Write 2 Done Sent Done false
8669 3119 Acked Write 6 Done Sent Done false
8670 3120 Acked Read 6 Done Sent Done false
8671 3121 Acked Read 6 Done Sent Done false
8672 3122 Acked Write 4 Done Sent Done false
8673 3123 Acked Write 2 Done Sent Done false
8674 3124 Acked Read 7 Done Sent Done false
8675 3125 Acked Write 6 Done Sent Done false
8676 3126 Acked Read 4 Done Sent Done false
8677 3127 Acked Read 2 Done Sent Done false
8678 3128 Acked Write 9 Done Sent Done false
8679 3129 Acked Write 5 Done Sent Done false
8680 3130 Acked Read 9 Done Sent Done false
8681 3131 Acked Read 9 Done Sent Done false
8682 3132 Acked Read 1 Done Sent Done false
8683 3133 Acked Flush 0 Done Sent Done false
8684 3134 Acked Write 10 Done Sent Done false
8685 3135 Acked Read 9 Done Sent Done false
8686 3136 Acked Write 8 Done Sent Done false
8687 3137 Acked Read 6 Done Sent Done false
8688 3138 Acked Read 7 Done Sent Done false
8689 3139 Acked Read 6 Done Sent Done false
8690 3140 Acked Flush 0 Done Sent Done false
8691 3141 Acked Flush 0 Done Sent Done false
8692 3142 Acked Write 10 Done Sent Done false
8693 3143 Acked Write 2 Done Sent Done false
8694 3144 Acked Read 9 Done Sent Done false
8695 3145 Acked Flush 0 Done Sent Done false
8696 3146 Acked Read 7 Done Sent Done false
8697 3147 Acked Read 3 Done Sent Done false
8698 3148 Acked Read 4 Done Sent Done false
8699 3149 Acked Write 1 Done Sent Done false
8700 3150 Acked Write 3 Done Sent Done false
8701 3151 Acked Read 3 Done Sent Done false
8702 3152 Acked Read 6 Done Sent Done false
8703 3153 Acked Flush 0 Done Sent Done false
8704 3154 Acked Write 4 Done Sent Done false
8705 3155 Acked Read 1 Done Sent Done false
8706 3156 Acked Read 3 Done Sent Done false
8707 3157 Acked Read 9 Done Sent Done false
8708 3158 Acked Write 10 Done Sent Done false
8709 3159 Acked Read 5 Done Sent Done false
8710 3160 Acked Write 5 Done Sent Done false
8711 3161 Acked Flush 0 Done Sent Done false
8712 3162 Acked Read 2 Done Sent Done false
8713 3163 Acked Read 6 Done Sent Done false
8714 3164 Acked Write 8 Done Sent Done false
8715 3165 Acked Read 2 Done Sent Done false
8716 3166 Acked Write 2 Done Sent Done false
8717 3167 Acked Read 6 Done Sent Done false
8718 3168 Acked Read 2 Done Sent Done false
8719 3169 Acked Write 4 Done Sent Done false
8720 3170 Acked Write 8 Done Sent Done false
8721 3171 Acked Read 4 Done Sent Done false
8722 3172 Acked Read 7 Done Sent Done false
8723 3173 Acked Write 7 Done Sent Done false
8724 3174 Acked Read 5 Done Sent Done false
8725 3175 Acked Read 2 Done Sent Done false
8726 3176 Acked Read 4 Done Sent Done false
8727 3177 Acked Write 7 Done Sent Done false
8728 3178 Acked Read 4 Done Sent Done false
8729 3179 Acked Read 10 Done Sent Done false
8730 3180 Acked Write 10 Done Sent Done false
8731 3181 Acked Read 5 Done Sent Done false
8732 3182 Acked Write 6 Done Sent Done false
8733 3183 Acked Write 9 Done Sent Done false
8734 3184 Acked Read 8 Done Sent Done false
8735 3185 Acked Read 9 Done Sent Done false
8736 3186 Acked Write 8 Done Sent Done false
8737 3187 Acked Read 3 Done Sent Done false
8738 3188 Acked Write 5 Done Sent Done false
8739 3189 Acked Read 4 Done Sent Done false
8740 3190 Acked Read 9 Done Sent Done false
8741 3191 Acked Read 2 Done Sent Done false
8742 3192 Acked Read 6 Done Sent Done false
8743 3193 Acked Write 3 Done Sent Done false
8744 3194 Acked Write 1 Done Sent Done false
8745 3195 Acked Write 2 Done Sent Done false
8746 3196 Acked Read 5 Done Sent Done false
8747 3197 Acked Write 8 Done Sent Done false
8748 3198 Acked Read 7 Done Sent Done false
8749 3199 Acked Read 5 Done Sent Done false
8750 3200 Acked Read 10 Done Sent Done false
8751 3201 Acked Read 4 Done Sent Done false
8752 3202 Acked Read 6 Done Sent Done false
8753 3203 Acked Write 5 Done Sent Done false
8754 3204 Acked Read 4 Done Sent Done false
8755 3205 Acked Write 3 Done Sent Done false
8756 3206 Acked Write 2 Done Sent Done false
8757 3207 Acked Read 5 Done Sent Done false
8758 3208 Acked Flush 0 Done Sent Done false
8759 3209 Acked Read 4 Done Sent Done false
8760 3210 Acked Write 3 Done Sent Done false
8761 3211 Acked Read 8 Done Sent Done false
8762 3212 Acked Read 9 Done Sent Done false
8763 3213 Acked Read 1 Done Sent Done false
8764 3214 Acked Read 1 Done Sent Done false
8765 3215 Acked Read 4 Done Sent Done false
8766 3216 Acked Read 2 Done Sent Done false
8767 3217 Acked Write 2 Done Sent Done false
8768 3218 Acked Write 4 Done Sent Done false
8769 3219 Acked Read 4 Done Sent Done false
8770 3220 Acked Flush 0 Done Sent Done false
8771 3221 Acked Read 7 Done Sent Done false
8772 3222 Acked Read 7 Done Sent Done false
8773 3223 Acked Write 6 Done Sent Done false
8774 3224 Acked Read 4 Done Sent Done false
8775 3225 Acked Flush 0 Done Sent Done false
8776 3226 Acked Read 2 Done Sent Done false
8777 3227 Acked Write 5 Done Sent Done false
8778 3228 Acked Flush 0 Done Sent Done false
8779 3229 Acked Read 10 Done Sent Done false
8780 3230 Acked Write 1 Done Sent Done false
8781 3231 Acked Read 2 Done Sent Done false
8782 3232 Acked Read 6 Done Sent Done false
8783 3233 Acked Write 5 Done Sent Done false
8784 3234 Acked Write 8 Done Sent Done false
8785 3235 Acked Read 8 Done Sent Done false
8786 3236 Acked Flush 0 Done Sent Done false
8787 3237 Acked Read 2 Done Sent Done false
8788 3238 Acked Write 10 Done Sent Done false
8789 3239 Acked Read 5 Done Sent Done false
8790 3240 Acked Write 9 Done Sent Done false
8791 3241 Acked Flush 0 Done Sent Done false
8792 3242 Acked Read 7 Done Sent Done false
8793 3243 Acked Write 8 Done Sent Done false
8794 3244 Acked Read 3 Done Sent Done false
8795 3245 Acked Read 10 Done Sent Done false
8796 3246 Acked Read 1 Done Sent Done false
8797 3247 Acked Read 8 Done Sent Done false
8798 3248 Acked Read 10 Done Sent Done false
8799 3249 Acked Read 5 Done Sent Done false
8800 3250 Acked Read 4 Done Sent Done false
8801 3251 Acked Read 10 Done Sent Done false
8802 3252 Acked Read 9 Done Sent Done false
8803 3253 Acked Read 1 Done Sent Done false
8804 3254 Acked Flush 0 Done Sent Done false
8805 3255 Acked Write 9 Done Sent Done false
8806 3256 Acked Read 2 Done Sent Done false
8807 3257 Acked Read 2 Done Sent Done false
8808 3258 Acked Read 6 Done Sent Done false
8809 3259 Acked Read 5 Done Sent Done false
8810 3260 Acked Read 10 Done Sent Done false
8811 3261 Acked Read 5 Done Sent Done false
8812 3262 Acked Read 6 Done Sent Done false
8813 3263 Acked Write 4 Done Sent Done false
8814 3264 Acked Read 6 Done Sent Done false
8815 3265 Acked Read 4 Done Sent Done false
8816 3266 Acked Write 7 Done Sent Done false
8817 3267 Acked Write 10 Done Sent Done false
8818 3268 Acked Read 4 Done Sent Done false
8819 3269 Acked Write 7 Done Sent Done false
8820 3270 Acked Read 1 Done Sent Done false
8821 3271 Acked Write 9 Done Sent Done false
8822 3272 Acked Read 7 Done Sent Done false
8823 3273 Acked Write 5 Done Sent Done false
8824 3274 Acked Read 7 Done Sent Done false
8825 3275 Acked Read 1 Done Sent Done false
8826 3276 Acked Write 6 Done Sent Done false
8827 3277 Acked Read 3 Done Sent Done false
8828 3278 Acked Read 8 Done Sent Done false
8829 3279 Acked Write 1 Done Sent Done false
8830 3280 Acked Write 4 Done Sent Done false
8831 3281 Acked Flush 0 Done Sent Done false
8832 3282 Acked Write 1 Done Sent Done false
8833 3283 Acked Write 3 Done Sent Done false
8834 3284 Acked Read 1 Done Sent Done false
8835 3285 Acked Write 2 Done Sent Done false
8836 3286 Acked Read 3 Done Sent Done false
8837 3287 Acked Write 5 Done Sent Done false
8838 3288 Acked Write 8 Done Sent Done false
8839 3289 Acked Write 9 Done Sent Done false
8840 3290 Acked Read 8 Done Sent Done false
8841 3291 Acked Write 10 Done Sent Done false
8842 3292 Acked Read 6 Done Sent Done false
8843 3293 Acked Write 3 Done Sent Done false
8844 3294 Acked Write 2 Done Sent Done false
8845 3295 Acked Read 4 Done Sent Done false
8846 3296 Acked Read 4 Done Sent Done false
8847 3297 Acked Write 10 Done Sent Done false
8848 3298 Acked Flush 0 Done Sent Done false
8849 3299 Acked Write 9 Done Sent Done false
8850 3300 Acked Write 8 Done Sent Done false
8851 3301 Acked Read 6 Done Sent Done false
8852 3302 Acked Read 7 Done Sent Done false
8853 3303 Acked Write 2 Done Sent Done false
8854 3304 Acked Flush 0 Done Sent Done false
8855 3305 Acked Read 2 Done Sent Done false
8856 3306 Acked Read 5 Done Sent Done false
8857 3307 Acked Write 7 Done Sent Done false
8858 3308 Acked Write 5 Done Sent Done false
8859 3309 Acked Write 9 Done Sent Done false
8860 3310 Acked Read 3 Done Sent Done false
8861 3311 Acked Write 7 Done Sent Done false
8862 3312 Acked Read 1 Done Sent Done false
8863 3313 Acked Read 7 Done Sent Done false
8864 3314 Acked Write 4 Done Sent Done false
8865 3315 Acked Read 6 Done Sent Done false
8866 3316 Acked Write 9 Done Sent Done false
8867 3317 Acked Write 6 Done Sent Done false
8868 3318 Acked Write 1 Done Sent Done false
8869 3319 Acked Write 8 Done Sent Done false
8870 3320 Acked Read 5 Done Sent Done false
8871 3321 Acked Read 2 Done Sent Done false
8872 3322 Acked Read 9 Done Sent Done false
8873 3323 Acked Write 10 Done Sent Done false
8874 3324 Acked Write 4 Done Sent Done false
8875 3325 Acked Read 7 Done Sent Done false
8876 3326 Acked Write 8 Done Sent Done false
8877 3327 Acked Read 10 Done Sent Done false
8878 3328 Acked Write 3 Done Sent Done false
8879 3329 Acked Write 5 Done Sent Done false
8880 3330 Acked Write 4 Done Sent Done false
8881 3331 Acked Read 5 Done Sent Done false
8882 3332 Acked Read 1 Done Sent Done false
8883 3333 Acked Write 4 Done Sent Done false
8884 3334 Acked Read 5 Done Sent Done false
8885 3335 Acked Read 7 Done Sent Done false
8886 3336 Acked Write 10 Done Sent Done false
8887 3337 Acked Write 2 Done Sent Done false
8888 3338 Acked Write 6 Done Sent Done false
8889 3339 Acked Read 2 Done Sent Done false
8890 3340 Acked Read 4 Done Sent Done false
8891 3341 Acked Read 1 Done Sent Done false
8892 3342 Acked Read 6 Done Sent Done false
8893 3343 Acked Read 4 Done Sent Done false
8894 3344 Acked Flush 0 Done Sent Done false
8895 3345 Acked Read 6 Done Sent Done false
8896 3346 Acked Write 1 Done Sent Done false
8897 3347 Acked Read 10 Done Sent Done false
8898 3348 Acked Write 6 Done Sent Done false
8899 3349 Acked Read 8 Done Sent Done false
8900 3350 Acked Write 1 Done Sent Done false
8901 3351 Acked Read 10 Done Sent Done false
8902 3352 Acked Write 10 Done Sent Done false
8903 3353 Acked Read 6 Done Sent Done false
8904 3354 Acked Read 6 Done Sent Done false
8905 3355 Acked Write 8 Done Sent Done false
8906 3356 Acked Read 8 Done Sent Done false
8907 3357 Acked Read 2 Done Sent Done false
8908 3358 Acked Read 10 Done Sent Done false
8909 3359 Acked Write 1 Done Sent Done false
8910 3360 Acked Read 10 Done Sent Done false
8911 3361 Acked Flush 0 Done Sent Done false
8912 3362 Acked Read 9 Done Sent Done false
8913 3363 Acked Write 5 Done Sent Done false
8914 3364 Acked Read 10 Done Sent Done false
8915 3365 Acked Flush 0 Done Sent Done false
8916 3366 Acked Write 8 Done Sent Done false
8917 3367 Acked Read 1 Done Sent Done false
8918 3368 Acked Write 10 Done Sent Done false
8919 3369 Acked Read 2 Done Sent Done false
8920 3370 Acked Flush 0 Done Sent Done false
8921 3371 Acked Read 5 Done Sent Done false
8922 3372 Acked Write 6 Done Sent Done false
8923 3373 Acked Read 4 Done Sent Done false
8924 3374 Acked Read 7 Done Sent Done false
8925 3375 Acked Write 9 Done Sent Done false
8926 3376 Acked Write 2 Done Sent Done false
8927 3377 Acked Read 7 Done Sent Done false
8928 3378 Acked Read 10 Done Sent Done false
8929 3379 Acked Read 6 Done Sent Done false
8930 3380 Acked Read 10 Done Sent Done false
8931 3381 Acked Write 5 Done Sent Done false
8932 3382 Acked Flush 0 Done Sent Done false
8933 3383 Acked Write 6 Done Sent Done false
8934 3384 Acked Read 1 Done Sent Done false
8935 3385 Acked Read 3 Done Sent Done false
8936 3386 Acked Read 1 Done Sent Done false
8937 3387 Acked Write 3 Done Sent Done false
8938 3388 Acked Read 9 Done Sent Done false
8939 3389 Acked Read 5 Done Sent Done false
8940 3390 Acked Write 1 Done Sent Done false
8941 3391 Acked Write 2 Done Sent Done false
8942 3392 Acked Write 8 Done Sent Done false
8943 3393 Acked Flush 0 Done Sent Done false
8944 3394 Acked Read 10 Done Sent Done false
8945 3395 Acked Read 3 Done Sent Done false
8946 3396 Acked Write 5 Done Sent Done false
8947 3397 Acked Read 5 Done Sent Done false
8948 3398 Acked Read 9 Done Sent Done false
8949 3399 Acked Read 8 Done Sent Done false
8950 3400 Acked Write 7 Done Sent Done false
8951 3401 Acked Read 1 Done Sent Done false
8952 3402 Acked Write 8 Done Sent Done false
8953 3403 Acked Read 4 Done Sent Done false
8954 3404 Acked Read 5 Done Sent Done false
8955 3405 Acked Flush 0 Done Sent Done false
8956 3406 Acked Read 10 Done Sent Done false
8957 3407 Acked Read 1 Done Sent Done false
8958 3408 Acked Flush 0 Done Sent Done false
8959 3409 Acked Read 8 Done Sent Done false
8960 3410 Acked Read 10 Done Sent Done false
8961 3411 Acked Read 6 Done Sent Done false
8962 3412 Acked Flush 0 Done Sent Done false
8963 3413 Acked Read 7 Done Sent Done false
8964 3414 Acked Read 8 Done Sent Done false
8965 STATES DS:0 DS:1 DS:2 TOTAL
8966 Sent 0 301 0 301
8967 Done 301 0 301 602
8968 Skipped 0 0 0 0
8969 Error 0 0 0 0
8970 Last Flush: Some(JobId(3412)) Some(JobId(3113)) Some(JobId(3412))
8971 Downstairs last five completed: 3113 3112 3111 3110 3109
8972 Upstairs last five completed: 3414 3413 3412 3411 3410
8973 CLIENT: Up:0 ds:301 act:2
89742025-02-21T06:47:13.826ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
89752025-02-21T06:47:14.974ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
89762025-02-21T06:47:15.829ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
89772025-02-21T06:47:17.235ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
8978 ----------------------------------------------------------------
8979 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8980 States: Active Offline Active
8981 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8982 3114 Acked Write 7 Done Sent Done false
8983 3115 Acked Read 8 Done Sent Done false
8984 3116 Acked Read 7 Done Sent Done false
8985 3117 Acked Write 4 Done Sent Done false
8986 3118 Acked Write 2 Done Sent Done false
8987 3119 Acked Write 6 Done Sent Done false
8988 3120 Acked Read 6 Done Sent Done false
8989 3121 Acked Read 6 Done Sent Done false
8990 3122 Acked Write 4 Done Sent Done false
8991 3123 Acked Write 2 Done Sent Done false
8992 3124 Acked Read 7 Done Sent Done false
8993 3125 Acked Write 6 Done Sent Done false
8994 3126 Acked Read 4 Done Sent Done false
8995 3127 Acked Read 2 Done Sent Done false
8996 3128 Acked Write 9 Done Sent Done false
8997 3129 Acked Write 5 Done Sent Done false
8998 3130 Acked Read 9 Done Sent Done false
8999 3131 Acked Read 9 Done Sent Done false
9000 3132 Acked Read 1 Done Sent Done false
9001 3133 Acked Flush 0 Done Sent Done false
9002 3134 Acked Write 10 Done Sent Done false
9003 3135 Acked Read 9 Done Sent Done false
9004 3136 Acked Write 8 Done Sent Done false
9005 3137 Acked Read 6 Done Sent Done false
9006 3138 Acked Read 7 Done Sent Done false
9007 3139 Acked Read 6 Done Sent Done false
9008 3140 Acked Flush 0 Done Sent Done false
9009 3141 Acked Flush 0 Done Sent Done false
9010 3142 Acked Write 10 Done Sent Done false
9011 3143 Acked Write 2 Done Sent Done false
9012 3144 Acked Read 9 Done Sent Done false
9013 3145 Acked Flush 0 Done Sent Done false
9014 3146 Acked Read 7 Done Sent Done false
9015 3147 Acked Read 3 Done Sent Done false
9016 3148 Acked Read 4 Done Sent Done false
9017 3149 Acked Write 1 Done Sent Done false
9018 3150 Acked Write 3 Done Sent Done false
9019 3151 Acked Read 3 Done Sent Done false
9020 3152 Acked Read 6 Done Sent Done false
9021 3153 Acked Flush 0 Done Sent Done false
9022 3154 Acked Write 4 Done Sent Done false
9023 3155 Acked Read 1 Done Sent Done false
9024 3156 Acked Read 3 Done Sent Done false
9025 3157 Acked Read 9 Done Sent Done false
9026 3158 Acked Write 10 Done Sent Done false
9027 3159 Acked Read 5 Done Sent Done false
9028 3160 Acked Write 5 Done Sent Done false
9029 3161 Acked Flush 0 Done Sent Done false
9030 3162 Acked Read 2 Done Sent Done false
9031 3163 Acked Read 6 Done Sent Done false
9032 3164 Acked Write 8 Done Sent Done false
9033 3165 Acked Read 2 Done Sent Done false
9034 3166 Acked Write 2 Done Sent Done false
9035 3167 Acked Read 6 Done Sent Done false
9036 3168 Acked Read 2 Done Sent Done false
9037 3169 Acked Write 4 Done Sent Done false
9038 3170 Acked Write 8 Done Sent Done false
9039 3171 Acked Read 4 Done Sent Done false
9040 3172 Acked Read 7 Done Sent Done false
9041 3173 Acked Write 7 Done Sent Done false
9042 3174 Acked Read 5 Done Sent Done false
9043 3175 Acked Read 2 Done Sent Done false
9044 3176 Acked Read 4 Done Sent Done false
9045 3177 Acked Write 7 Done Sent Done false
9046 3178 Acked Read 4 Done Sent Done false
9047 3179 Acked Read 10 Done Sent Done false
9048 3180 Acked Write 10 Done Sent Done false
9049 3181 Acked Read 5 Done Sent Done false
9050 3182 Acked Write 6 Done Sent Done false
9051 3183 Acked Write 9 Done Sent Done false
9052 3184 Acked Read 8 Done Sent Done false
9053 3185 Acked Read 9 Done Sent Done false
9054 3186 Acked Write 8 Done Sent Done false
9055 3187 Acked Read 3 Done Sent Done false
9056 3188 Acked Write 5 Done Sent Done false
9057 3189 Acked Read 4 Done Sent Done false
9058 3190 Acked Read 9 Done Sent Done false
9059 3191 Acked Read 2 Done Sent Done false
9060 3192 Acked Read 6 Done Sent Done false
9061 3193 Acked Write 3 Done Sent Done false
9062 3194 Acked Write 1 Done Sent Done false
9063 3195 Acked Write 2 Done Sent Done false
9064 3196 Acked Read 5 Done Sent Done false
9065 3197 Acked Write 8 Done Sent Done false
9066 3198 Acked Read 7 Done Sent Done false
9067 3199 Acked Read 5 Done Sent Done false
9068 3200 Acked Read 10 Done Sent Done false
9069 3201 Acked Read 4 Done Sent Done false
9070 3202 Acked Read 6 Done Sent Done false
9071 3203 Acked Write 5 Done Sent Done false
9072 3204 Acked Read 4 Done Sent Done false
9073 3205 Acked Write 3 Done Sent Done false
9074 3206 Acked Write 2 Done Sent Done false
9075 3207 Acked Read 5 Done Sent Done false
9076 3208 Acked Flush 0 Done Sent Done false
9077 3209 Acked Read 4 Done Sent Done false
9078 3210 Acked Write 3 Done Sent Done false
9079 3211 Acked Read 8 Done Sent Done false
9080 3212 Acked Read 9 Done Sent Done false
9081 3213 Acked Read 1 Done Sent Done false
9082 3214 Acked Read 1 Done Sent Done false
9083 3215 Acked Read 4 Done Sent Done false
9084 3216 Acked Read 2 Done Sent Done false
9085 3217 Acked Write 2 Done Sent Done false
9086 3218 Acked Write 4 Done Sent Done false
9087 3219 Acked Read 4 Done Sent Done false
9088 3220 Acked Flush 0 Done Sent Done false
9089 3221 Acked Read 7 Done Sent Done false
9090 3222 Acked Read 7 Done Sent Done false
9091 3223 Acked Write 6 Done Sent Done false
9092 3224 Acked Read 4 Done Sent Done false
9093 3225 Acked Flush 0 Done Sent Done false
9094 3226 Acked Read 2 Done Sent Done false
9095 3227 Acked Write 5 Done Sent Done false
9096 3228 Acked Flush 0 Done Sent Done false
9097 3229 Acked Read 10 Done Sent Done false
9098 3230 Acked Write 1 Done Sent Done false
9099 3231 Acked Read 2 Done Sent Done false
9100 3232 Acked Read 6 Done Sent Done false
9101 3233 Acked Write 5 Done Sent Done false
9102 3234 Acked Write 8 Done Sent Done false
9103 3235 Acked Read 8 Done Sent Done false
9104 3236 Acked Flush 0 Done Sent Done false
9105 3237 Acked Read 2 Done Sent Done false
9106 3238 Acked Write 10 Done Sent Done false
9107 3239 Acked Read 5 Done Sent Done false
9108 3240 Acked Write 9 Done Sent Done false
9109 3241 Acked Flush 0 Done Sent Done false
9110 3242 Acked Read 7 Done Sent Done false
9111 3243 Acked Write 8 Done Sent Done false
9112 3244 Acked Read 3 Done Sent Done false
9113 3245 Acked Read 10 Done Sent Done false
9114 3246 Acked Read 1 Done Sent Done false
9115 3247 Acked Read 8 Done Sent Done false
9116 3248 Acked Read 10 Done Sent Done false
9117 3249 Acked Read 5 Done Sent Done false
9118 3250 Acked Read 4 Done Sent Done false
9119 3251 Acked Read 10 Done Sent Done false
9120 3252 Acked Read 9 Done Sent Done false
9121 3253 Acked Read 1 Done Sent Done false
9122 3254 Acked Flush 0 Done Sent Done false
9123 3255 Acked Write 9 Done Sent Done false
9124 3256 Acked Read 2 Done Sent Done false
9125 3257 Acked Read 2 Done Sent Done false
9126 3258 Acked Read 6 Done Sent Done false
9127 3259 Acked Read 5 Done Sent Done false
9128 3260 Acked Read 10 Done Sent Done false
9129 3261 Acked Read 5 Done Sent Done false
9130 3262 Acked Read 6 Done Sent Done false
9131 3263 Acked Write 4 Done Sent Done false
9132 3264 Acked Read 6 Done Sent Done false
9133 3265 Acked Read 4 Done Sent Done false
9134 3266 Acked Write 7 Done Sent Done false
9135 3267 Acked Write 10 Done Sent Done false
9136 3268 Acked Read 4 Done Sent Done false
9137 3269 Acked Write 7 Done Sent Done false
9138 3270 Acked Read 1 Done Sent Done false
9139 3271 Acked Write 9 Done Sent Done false
9140 3272 Acked Read 7 Done Sent Done false
9141 3273 Acked Write 5 Done Sent Done false
9142 3274 Acked Read 7 Done Sent Done false
9143 3275 Acked Read 1 Done Sent Done false
9144 3276 Acked Write 6 Done Sent Done false
9145 3277 Acked Read 3 Done Sent Done false
9146 3278 Acked Read 8 Done Sent Done false
9147 3279 Acked Write 1 Done Sent Done false
9148 3280 Acked Write 4 Done Sent Done false
9149 3281 Acked Flush 0 Done Sent Done false
9150 3282 Acked Write 1 Done Sent Done false
9151 3283 Acked Write 3 Done Sent Done false
9152 3284 Acked Read 1 Done Sent Done false
9153 3285 Acked Write 2 Done Sent Done false
9154 3286 Acked Read 3 Done Sent Done false
9155 3287 Acked Write 5 Done Sent Done false
9156 3288 Acked Write 8 Done Sent Done false
9157 3289 Acked Write 9 Done Sent Done false
9158 3290 Acked Read 8 Done Sent Done false
9159 3291 Acked Write 10 Done Sent Done false
9160 3292 Acked Read 6 Done Sent Done false
9161 3293 Acked Write 3 Done Sent Done false
9162 3294 Acked Write 2 Done Sent Done false
9163 3295 Acked Read 4 Done Sent Done false
9164 3296 Acked Read 4 Done Sent Done false
9165 3297 Acked Write 10 Done Sent Done false
9166 3298 Acked Flush 0 Done Sent Done false
9167 3299 Acked Write 9 Done Sent Done false
9168 3300 Acked Write 8 Done Sent Done false
9169 3301 Acked Read 6 Done Sent Done false
9170 3302 Acked Read 7 Done Sent Done false
9171 3303 Acked Write 2 Done Sent Done false
9172 3304 Acked Flush 0 Done Sent Done false
9173 3305 Acked Read 2 Done Sent Done false
9174 3306 Acked Read 5 Done Sent Done false
9175 3307 Acked Write 7 Done Sent Done false
9176 3308 Acked Write 5 Done Sent Done false
9177 3309 Acked Write 9 Done Sent Done false
9178 3310 Acked Read 3 Done Sent Done false
9179 3311 Acked Write 7 Done Sent Done false
9180 3312 Acked Read 1 Done Sent Done false
9181 3313 Acked Read 7 Done Sent Done false
9182 3314 Acked Write 4 Done Sent Done false
9183 3315 Acked Read 6 Done Sent Done false
9184 3316 Acked Write 9 Done Sent Done false
9185 3317 Acked Write 6 Done Sent Done false
9186 3318 Acked Write 1 Done Sent Done false
9187 3319 Acked Write 8 Done Sent Done false
9188 3320 Acked Read 5 Done Sent Done false
9189 3321 Acked Read 2 Done Sent Done false
9190 3322 Acked Read 9 Done Sent Done false
9191 3323 Acked Write 10 Done Sent Done false
9192 3324 Acked Write 4 Done Sent Done false
9193 3325 Acked Read 7 Done Sent Done false
9194 3326 Acked Write 8 Done Sent Done false
9195 3327 Acked Read 10 Done Sent Done false
9196 3328 Acked Write 3 Done Sent Done false
9197 3329 Acked Write 5 Done Sent Done false
9198 3330 Acked Write 4 Done Sent Done false
9199 3331 Acked Read 5 Done Sent Done false
9200 3332 Acked Read 1 Done Sent Done false
9201 3333 Acked Write 4 Done Sent Done false
9202 3334 Acked Read 5 Done Sent Done false
9203 3335 Acked Read 7 Done Sent Done false
9204 3336 Acked Write 10 Done Sent Done false
9205 3337 Acked Write 2 Done Sent Done false
9206 3338 Acked Write 6 Done Sent Done false
9207 3339 Acked Read 2 Done Sent Done false
9208 3340 Acked Read 4 Done Sent Done false
9209 3341 Acked Read 1 Done Sent Done false
9210 3342 Acked Read 6 Done Sent Done false
9211 3343 Acked Read 4 Done Sent Done false
9212 3344 Acked Flush 0 Done Sent Done false
9213 3345 Acked Read 6 Done Sent Done false
9214 3346 Acked Write 1 Done Sent Done false
9215 3347 Acked Read 10 Done Sent Done false
9216 3348 Acked Write 6 Done Sent Done false
9217 3349 Acked Read 8 Done Sent Done false
9218 3350 Acked Write 1 Done Sent Done false
9219 3351 Acked Read 10 Done Sent Done false
9220 3352 Acked Write 10 Done Sent Done false
9221 3353 Acked Read 6 Done Sent Done false
9222 3354 Acked Read 6 Done Sent Done false
9223 3355 Acked Write 8 Done Sent Done false
9224 3356 Acked Read 8 Done Sent Done false
9225 3357 Acked Read 2 Done Sent Done false
9226 3358 Acked Read 10 Done Sent Done false
9227 3359 Acked Write 1 Done Sent Done false
9228 3360 Acked Read 10 Done Sent Done false
9229 3361 Acked Flush 0 Done Sent Done false
9230 3362 Acked Read 9 Done Sent Done false
9231 3363 Acked Write 5 Done Sent Done false
9232 3364 Acked Read 10 Done Sent Done false
9233 3365 Acked Flush 0 Done Sent Done false
9234 3366 Acked Write 8 Done Sent Done false
9235 3367 Acked Read 1 Done Sent Done false
9236 3368 Acked Write 10 Done Sent Done false
9237 3369 Acked Read 2 Done Sent Done false
9238 3370 Acked Flush 0 Done Sent Done false
9239 3371 Acked Read 5 Done Sent Done false
9240 3372 Acked Write 6 Done Sent Done false
9241 3373 Acked Read 4 Done Sent Done false
9242 3374 Acked Read 7 Done Sent Done false
9243 3375 Acked Write 9 Done Sent Done false
9244 3376 Acked Write 2 Done Sent Done false
9245 3377 Acked Read 7 Done Sent Done false
9246 3378 Acked Read 10 Done Sent Done false
9247 3379 Acked Read 6 Done Sent Done false
9248 3380 Acked Read 10 Done Sent Done false
9249 3381 Acked Write 5 Done Sent Done false
9250 3382 Acked Flush 0 Done Sent Done false
9251 3383 Acked Write 6 Done Sent Done false
9252 3384 Acked Read 1 Done Sent Done false
9253 3385 Acked Read 3 Done Sent Done false
9254 3386 Acked Read 1 Done Sent Done false
9255 3387 Acked Write 3 Done Sent Done false
9256 3388 Acked Read 9 Done Sent Done false
9257 3389 Acked Read 5 Done Sent Done false
9258 3390 Acked Write 1 Done Sent Done false
9259 3391 Acked Write 2 Done Sent Done false
9260 3392 Acked Write 8 Done Sent Done false
9261 3393 Acked Flush 0 Done Sent Done false
9262 3394 Acked Read 10 Done Sent Done false
9263 3395 Acked Read 3 Done Sent Done false
9264 3396 Acked Write 5 Done Sent Done false
9265 3397 Acked Read 5 Done Sent Done false
9266 3398 Acked Read 9 Done Sent Done false
9267 3399 Acked Read 8 Done Sent Done false
9268 3400 Acked Write 7 Done Sent Done false
9269 3401 Acked Read 1 Done Sent Done false
9270 3402 Acked Write 8 Done Sent Done false
9271 3403 Acked Read 4 Done Sent Done false
9272 3404 Acked Read 5 Done Sent Done false
9273 3405 Acked Flush 0 Done Sent Done false
9274 3406 Acked Read 10 Done Sent Done false
9275 3407 Acked Read 1 Done Sent Done false
9276 3408 Acked Flush 0 Done Sent Done false
9277 3409 Acked Read 8 Done Sent Done false
9278 3410 Acked Read 10 Done Sent Done false
9279 3411 Acked Read 6 Done Sent Done false
9280 3412 Acked Flush 0 Done Sent Done false
9281 3413 Acked Read 7 Done Sent Done false
9282 3414 Acked Read 8 Done Sent Done false
9283 STATES DS:0 DS:1 DS:2 TOTAL
9284 Sent 0 301 0 301
9285 Done 301 0 301 602
9286 Skipped 0 0 0 0
9287 Error 0 0 0 0
9288 Last Flush: Some(JobId(3412)) Some(JobId(3113)) Some(JobId(3412))
9289 Downstairs last five completed: 3113 3112 3111 3110 3109
9290 Upstairs last five completed: 3414 3413 3412 3411 3410
9291 CLIENT: Up:0 ds:301 act:2
92922025-02-21T06:47:17.832ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
92932025-02-21T06:47:18.833ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
92942025-02-21T06:47:19.333ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
92952025-02-21T06:47:19.975ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
92962025-02-21T06:47:20.837ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
92972025-02-21T06:47:20.984ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
92982025-02-21T06:47:20.984ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54410) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
92992025-02-21T06:47:20.985ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
93002025-02-21T06:47:20.986ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
93012025-02-21T06:47:20.986ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
93022025-02-21T06:47:20.986ZINFOcrucible: send last flush ID to this DS: Some(JobId(3113)) = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
93032025-02-21T06:47:20.986ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
93042025-02-21T06:47:20.986ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(3113)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
93052025-02-21T06:47:20.986ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(3113)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
9306 ----------------------------------------------------------------
9307 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:2
9308 States: Active Active Active
9309 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9310 3413 Acked Read 7 Done Done Done true
9311 3414 Acked Read 8 Done Done Done true
9312 STATES DS:0 DS:1 DS:2 TOTAL
9313 Sent 0 0 0 0
9314 Done 2 2 2 6
9315 Skipped 0 0 0 0
9316 Error 0 0 0 0
9317 Last Flush: Some(JobId(3412)) Some(JobId(3412)) Some(JobId(3412))
9318 Downstairs last five completed: 3412 3411 3410 3409 3408
9319 Upstairs last five completed: 3414 3413 3412 3411 3410
9320 CLIENT: Up:0 ds:2 act:3
9321 ----------------------------------------------------------------
9322 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9323 Downstairs last five completed: 3415 3414 3413 3412 3411
9324 Upstairs last five completed: 3415 3414 3413 3412 3411
9325 CLIENT: Up:0 ds:0 act:3
9326 Replay: All jobs finished
9327 Main task has work to do, go find it
9328 got dsc Stop(0)
9329 stop 0
9330 [8810] Got stop action so:false kr:false
9331 Feb 21 06:47:25.248 INFO request completed, latency_us: 378, response_code: 204, uri: /stop/cid/0, method: POST, req_id: bb7bee81-6d4a-4100-8a42-5f9da120a26a, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
93322025-02-21T06:47:25.249ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
9333 Feb 21 06:47:25.249 INFO request completed, latency_us: 127, response_code: 200, uri: /state/cid/0, method: GET, req_id: 864975b7-c99d-4fb3-86fb-712585cc2b47, remote_addr: 127.0.0.1:53848, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:47:25.249639181Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
9334 local_addr: 127.0.0.1:9998
93352025-02-21T06:47:25.249ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
93362025-02-21T06:47:25.249ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
9337 [8810] Exited with: ExitStatus(unix_wait_status(9))
9338 [8810][0] reports Exit
9339 Feb 21 06:47:29.252 INFO request completed, latency_us: 363, response_code: 200, uri: /state/cid/0, method: GET, req_id: fcf3d81f-fff3-4de1-a17c-1bcce2b89e31, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
9340 001/300 Read block 2778 len 20480
9341 002/300 Read block 3999 len 4096
9342 003/300 Read block 1242 len 16384
9343 004/300 Flush
9344 005/300 Read block 619 len 32768
9345 006/300 Read block 788 len 28672
9346 007/300 Read block 932 len 28672
9347 008/300 Write block 2135 len 4096 data: 3
9348 009/300 Read block 4982 len 24576
9349 010/300 Read block 4316 len 16384
9350 011/300 Read block 4166 len 24576
9351 012/300 Read block 3050 len 8192
9352 013/300 Read block 1780 len 12288
9353 014/300 Flush
9354 015/300 Write block 3584 len 36864 data: 8 9 8 6 5 4 4 3 3
9355 016/300 Read block 2353 len 32768
9356 017/300 Read block 1724 len 12288
9357 018/300 Write block 562 len 20480 data: 2 3 3 3 3
9358 019/300 Write block 2948 len 36864 data: 4 4 5 6 6 6 6 6 5
9359 020/300 Read block 4944 len 36864
9360 021/300 Write block 2584 len 24576 data: 3 4 4 3 3 3
9361 022/300 Write block 4144 len 16384 data: 3 3 3 3
9362 023/300 Read block 3968 len 20480
9363 024/300 Flush
9364 025/300 Write block 4834 len 12288 data: 3 3 3
9365 026/300 Write block 2849 len 36864 data: 3 3 3 3 2 3 3 3 5
9366 027/300 Read block 3558 len 40960
9367 028/300 Read block 3993 len 20480
9368 029/300 Read block 100 len 16384
9369 030/300 Read block 4185 len 4096
9370 031/300 Read block 4164 len 36864
9371 032/300 Flush
9372 033/300 Read block 1999 len 8192
9373 034/300 Read block 1092 len 16384
9374 035/300 Read block 1662 len 16384
9375 036/300 Read block 4560 len 36864
9376 037/300 Flush
9377 038/300 Write block 3658 len 16384 data: 2 2 2 2
9378 039/300 Write block 3132 len 32768 data: 2 2 2 2 2 2 2 2
9379 040/300 Flush
9380 041/300 Read block 781 len 32768
9381 042/300 Read block 940 len 20480
9382 043/300 Flush
9383 044/300 Flush
9384 045/300 Write block 2175 len 4096 data: 4
9385 046/300 Read block 2406 len 32768
9386 047/300 Write block 1216 len 4096 data: 3
9387 048/300 Write block 2909 len 16384 data: 3 3 2 2
9388 049/300 Flush
9389 050/300 Flush
9390 051/300 Write block 2429 len 40960 data: 4 4 4 4 2 2 2 2 2 2
9391 052/300 Write block 2524 len 8192 data: 2 2
9392 053/300 Flush
9393 054/300 Read block 3959 len 4096
9394 055/300 Read block 1095 len 16384
9395 056/300 Write block 4469 len 40960 data: 4 4 4 4 4 3 3 4 4 4
9396 057/300 Write block 3722 len 36864 data: 2 2 2 2 2 2 2 2 2
9397 058/300 Read block 1890 len 8192
9398 059/300 Write block 4855 len 16384 data: 3 2 2 3
9399 060/300 Write block 4271 len 28672 data: 3 3 4 4 4 3 3
9400 061/300 Read block 1470 len 28672
9401 062/300 Write block 3600 len 12288 data: 2 2 2
9402 063/300 Read block 1715 len 32768
9403 064/300 Read block 1847 len 8192
9404 065/300 Read block 1922 len 4096
9405 066/300 Read block 1033 len 40960
9406 067/300 Write block 4761 len 36864 data: 2 2 2 2 2 2 2 2 2
9407 068/300 Read block 3082 len 4096
9408 069/300 Write block 4616 len 40960 data: 3 3 4 2 3 2 2 2 3 4
9409 070/300 Write block 42 len 20480 data: 3 3 3 3 3
9410 071/300 Read block 3042 len 24576
9411 072/300 Write block 1820 len 16384 data: 4 5 5 4
9412 073/300 Write block 3130 len 40960 data: 2 2 3 3 3 3 3 3 3 3
9413 074/300 Read block 2633 len 24576
9414 075/300 Flush
9415 076/300 Flush
9416 077/300 Read block 17 len 4096
9417 078/300 Write block 381 len 12288 data: 4 4 3
9418 079/300 Write block 3219 len 16384 data: 2 2 2 2
9419 080/300 Read block 3614 len 36864
9420 081/300 Write block 3728 len 12288 data: 3 3 3
9421 082/300 Read block 860 len 32768
9422 083/300 Write block 782 len 24576 data: 4 4 4 5 4 4
9423 084/300 Read block 1470 len 4096
9424 085/300 Read block 9 len 8192
9425 086/300 Write block 4646 len 12288 data: 4 6 6
9426 087/300 Read block 599 len 24576
9427 088/300 Write block 1694 len 4096 data: 2
9428 089/300 Read block 1487 len 16384
9429 090/300 Read block 1290 len 12288
9430 091/300 Read block 1916 len 24576
9431 092/300 Write block 4596 len 4096 data: 2
9432 093/300 Write block 4675 len 16384 data: 2 2 2 2
9433 094/300 Read block 2628 len 12288
9434 095/300 Write block 1115 len 28672 data: 4 3 3 3 3 2 2
9435 096/300 Read block 4450 len 32768
9436 097/300 Flush
9437 098/300 Read block 565 len 16384
9438 099/300 Read block 2509 len 28672
9439 100/300 Write block 2721 len 28672 data: 3 2 2 2 2 2 2
9440 101/300 Write block 470 len 4096 data: 2
9441 102/300 Read block 2611 len 12288
9442 103/300 Read block 3184 len 16384
9443 104/300 Flush
9444 105/300 Read block 4565 len 28672
9445 106/300 Read block 4297 len 12288
9446 107/300 Read block 2775 len 12288
9447 108/300 Read block 4988 len 12288
9448 109/300 Read block 1392 len 4096
9449 110/300 Flush
9450 111/300 Read block 3473 len 12288
9451 112/300 Write block 4101 len 36864 data: 3 2 2 4 4 3 3 3 3
9452 113/300 Write block 4410 len 40960 data: 3 3 3 3 2 2 2 2 3 3
9453 114/300 Read block 1678 len 20480
9454 115/300 Write block 1630 len 20480 data: 3 3 3 2 2
9455 116/300 Write block 570 len 4096 data: 3
9456 117/300 Read block 3460 len 36864
9457 118/300 Read block 3444 len 16384
9458 119/300 Write block 1825 len 24576 data: 3 3 2 2 3 3
9459 120/300 Write block 3997 len 16384 data: 2 2 2 2
9460 121/300 Read block 3413 len 40960
9461 122/300 Write block 1087 len 36864 data: 4 2 2 2 2 2 2 2 2
9462 123/300 Write block 1452 len 40960 data: 2 2 2 2 2 2 2 2 2 2
9463 124/300 Write block 925 len 24576 data: 3 3 4 4 4 3
9464 125/300 Write block 709 len 4096 data: 2
9465 126/300 Read block 870 len 36864
9466 127/300 Read block 1911 len 8192
9467 128/300 Read block 711 len 28672
9468 129/300 Write block 1723 len 28672 data: 4 4 3 3 3 4 4
9469 130/300 Flush
9470 131/300 Write block 4560 len 8192 data: 2 2
9471 132/300 Read block 3417 len 28672
9472 133/300 Write block 2021 len 32768 data: 2 2 2 2 3 3 3 2
9473 134/300 Write block 3873 len 8192 data: 5 4
9474 135/300 Write block 4703 len 4096 data: 3
9475 136/300 Read block 3067 len 20480
9476 137/300 Read block 1579 len 36864
9477 138/300 Write block 3062 len 20480 data: 3 3 3 3 3
9478 139/300 Write block 4941 len 24576 data: 2 2 2 2 2 2
9479 140/300 Read block 3400 len 4096
9480 141/300 Read block 4348 len 24576
9481 142/300 Read block 3588 len 32768
9482 143/300 Write block 2836 len 20480 data: 3 4 4 4 4
9483 144/300 Flush
9484 145/300 Read block 3442 len 32768
9485 146/300 Read block 2924 len 4096
9486 147/300 Write block 3814 len 32768 data: 3 4 5 6 5 5 6 7
9487 148/300 Write block 2282 len 4096 data: 6
9488 149/300 Write block 3118 len 12288 data: 2 2 4
9489 150/300 Read block 412 len 8192
9490 151/300 Write block 4114 len 40960 data: 2 2 2 2 2 2 3 5 6 5
9491 152/300 Read block 737 len 8192
9492 153/300 Read block 2466 len 32768
9493 154/300 Write block 3664 len 40960 data: 2 2 2 2 2 2 2 2 2 2
9494 155/300 Write block 3677 len 12288 data: 2 3 3
9495 156/300 Read block 3329 len 28672
9496 157/300 Read block 1442 len 24576
9497 158/300 Flush
9498 159/300 Write block 1196 len 24576 data: 2 2 2 2 2 2
9499 160/300 Read block 2420 len 24576
9500 161/300 Read block 150 len 32768
9501 162/300 Read block 808 len 12288
9502 163/300 Flush
9503 164/300 Read block 1566 len 20480
9504 165/300 Write block 490 len 16384 data: 2 2 2 2
9505 166/300 Write block 4622 len 40960 data: 3 3 4 5 3 3 3 2 2 2
9506 167/300 Read block 88 len 8192
9507 168/300 Read block 2560 len 20480
9508 169/300 Read block 2820 len 4096
9509 170/300 Read block 184 len 40960
9510 171/300 Read block 2131 len 4096
9511 172/300 Write block 1047 len 8192 data: 2 2
9512 173/300 Write block 3565 len 12288 data: 3 3 3
9513 174/300 Read block 3760 len 8192
9514 175/300 Read block 4690 len 20480
9515 176/300 Write block 564 len 16384 data: 4 4 4 2
9516 177/300 Write block 2678 len 12288 data: 2 3 3
9517 178/300 Read block 2915 len 40960
9518 179/300 Write block 3132 len 36864 data: 4 4 4 4 4 4 4 4 3
9519 180/300 Flush
9520 181/300 Read block 2333 len 28672
9521 182/300 Write block 2214 len 32768 data: 6 7 7 6 5 5 5 4
9522 183/300 Read block 3843 len 16384
9523 184/300 Read block 2618 len 4096
9524 185/300 Read block 2344 len 32768
9525 186/300 Read block 404 len 8192
9526 187/300 Write block 3331 len 32768 data: 2 3 3 3 3 3 3 4
9527 188/300 Read block 3640 len 28672
9528 189/300 Write block 2678 len 12288 data: 3 4 4
9529 190/300 Read block 1525 len 8192
9530 191/300 Write block 1828 len 8192 data: 3 4
9531 192/300 Write block 312 len 16384 data: 3 3 3 3
9532 193/300 Flush
9533 194/300 Write block 2901 len 24576 data: 3 3 3 3 3 3
9534 195/300 Write block 4203 len 32768 data: 3 3 3 3 4 3 3 4
9535 196/300 Write block 3243 len 20480 data: 2 2 2 4 5
9536 197/300 Read block 1322 len 4096
9537 198/300 Flush
9538 199/300 Write block 2341 len 28672 data: 3 3 3 3 3 3 4
9539 200/300 Read block 1147 len 16384
9540 201/300 Write block 4781 len 4096 data: 2
9541 202/300 Read block 1757 len 36864
9542 203/300 Read block 4227 len 40960
9543 204/300 Flush
9544 205/300 Read block 1893 len 4096
9545 206/300 Write block 430 len 28672 data: 4 5 5 4 2 2 2
9546 207/300 Write block 3503 len 40960 data: 2 2 2 3 3 5 5 5 5 5
9547 208/300 Flush
9548 209/300 Read block 1798 len 28672
9549 210/300 Read block 188 len 20480
9550 211/300 Read block 186 len 24576
9551 212/300 Read block 4173 len 8192
9552 213/300 Write block 669 len 12288 data: 6 5 4
9553 214/300 Read block 2553 len 28672
9554 215/300 Flush
9555 216/300 Write block 1806 len 4096 data: 2
9556 217/300 Write block 4367 len 40960 data: 2 3 3 3 2 2 2 2 2 2
9557 218/300 Write block 3125 len 36864 data: 4 3 2 2 2 3 3 5 5
9558 219/300 Write block 2785 len 24576 data: 3 2 2 2 2 2
9559 220/300 Write block 1910 len 4096 data: 3
9560 221/300 Read block 3389 len 12288
9561 222/300 Flush
9562 223/300 Read block 1051 len 40960
9563 224/300 Read block 4727 len 36864
9564 225/300 Read block 1154 len 28672
9565 226/300 Read block 4438 len 32768
9566 227/300 Write block 4432 len 16384 data: 3 3 2 2
9567 228/300 Read block 3629 len 8192
9568 229/300 Write block 3595 len 36864 data: 2 2 2 2 2 3 3 3 2
9569 230/300 Read block 3888 len 36864
9570 231/300 Read block 86 len 4096
9571 232/300 Write block 1962 len 12288 data: 2 2 2
9572 233/300 Read block 2978 len 20480
9573 234/300 Read block 3969 len 24576
9574 235/300 Write block 1612 len 36864 data: 7 5 4 3 4 4 4 3 3
9575 236/300 Read block 1032 len 8192
9576 237/300 Write block 4094 len 4096 data: 4
9577 238/300 Read block 1632 len 32768
9578 239/300 Read block 259 len 12288
9579 240/300 Read block 3023 len 20480
9580 241/300 Flush
9581 242/300 Write block 4185 len 4096 data: 3
9582 243/300 Write block 2961 len 40960 data: 3 3 3 3 3 2 2 2 2 2
9583 244/300 Write block 1854 len 32768 data: 2 2 3 2 2 2 3 3
9584 245/300 Write block 206 len 32768 data: 2 2 3 4 3 3 2 2
9585 246/300 Read block 3629 len 36864
9586 247/300 Read block 2464 len 24576
9587 248/300 Write block 2187 len 12288 data: 4 7 7
9588 249/300 Read block 4891 len 16384
9589 250/300 Flush
9590 251/300 Write block 1775 len 40960 data: 3 3 3 3 3 2 3 3 3 3
9591 252/300 Read block 4716 len 4096
9592 253/300 Read block 4421 len 36864
9593 254/300 Write block 1902 len 16384 data: 3 3 3 3
9594 255/300 Read block 3568 len 12288
9595 256/300 Flush
9596 257/300 Read block 2148 len 28672
9597 258/300 Read block 1465 len 24576
9598 259/300 Write block 1102 len 8192 data: 2 3
9599 260/300 Read block 2338 len 32768
9600 261/300 Flush
9601 262/300 Read block 2853 len 40960
9602 263/300 Write block 115 len 24576 data: 2 2 3 3 3 3
9603 264/300 Read block 4482 len 12288
9604 265/300 Read block 997 len 12288
9605 266/300 Read block 3973 len 4096
9606 267/300 Write block 3702 len 36864 data: 4 4 3 3 3 3 3 3 2
9607 268/300 Write block 3306 len 8192 data: 2 2
9608 269/300 Read block 544 len 8192
9609 270/300 Read block 3399 len 20480
9610 271/300 Write block 3508 len 40960 data: 6 6 6 6 6 4 4 4 3 3
9611 272/300 Read block 3458 len 40960
9612 273/300 Flush
9613 274/300 Read block 3832 len 16384
9614 275/300 Read block 476 len 20480
9615 276/300 Flush
9616 277/300 Write block 4880 len 32768 data: 6 4 4 4 3 2 2 2
9617 278/300 Write block 3386 len 32768 data: 2 2 2 2 2 2 3 3
9618 279/300 Read block 4598 len 40960
9619 280/300 Write block 4670 len 28672 data: 3 3 2 2 2 3 3
9620 281/300 Read block 2481 len 16384
9621 282/300 Flush
9622 283/300 Write block 73 len 36864 data: 3 4 4 4 4 4 4 3 2
9623 284/300 Write block 2049 len 36864 data: 4 5 4 4 4 3 3 3 3
9624 285/300 Read block 199 len 20480
9625 286/300 Write block 3136 len 32768 data: 5 5 5 5 4 3 3 4
9626 287/300 Flush
9627 288/300 Flush
9628 289/300 Read block 2578 len 36864
9629 290/300 Write block 2388 len 40960 data: 3 3 4 4 4 3 2 2 2 2
9630 291/300 Write block 1407 len 8192 data: 3 4
9631 292/300 Read block 250 len 36864
9632 293/300 Read block 1949 len 40960
9633 294/300 Read block 3172 len 8192
9634 295/300 Read block 3564 len 16384
9635 296/300 Write block 1655 len 20480 data: 3 3 3 3 3
9636 297/300 Flush
9637 298/300 Write block 1955 len 8192 data: 2 2
9638 299/300 Flush
9639 300/300 Write block 3425 len 36864 data: 4 4 4 4 4 4 5 5 4
9640 301/300 Read block 650 len 40960
9641 Main task has work to do, go find it
9642 got dsc Start(0)
9643 start 0
9644 Feb 21 06:47:29.488 INFO request completed, latency_us: 295, response_code: 204, uri: /start/cid/0, method: POST, req_id: 7676add2-0cbc-41f4-8f38-61d7552b2389, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
9645 Replay: started 0, returned:Ok(())
9646 ----------------------------------------------------------------
9647 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9648 States: Offline Active Active
9649 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9650 3416 Acked Read 5 Sent Done Done false
9651 3417 Acked Read 1 Sent Done Done false
9652 3418 Acked Read 4 Sent Done Done false
9653 3419 Acked Flush 0 Sent Done Done false
9654 3420 Acked Read 8 Sent Done Done false
9655 3421 Acked Read 7 Sent Done Done false
9656 3422 Acked Read 7 Sent Done Done false
9657 3423 Acked Write 1 Sent Done Done false
9658 3424 Acked Read 6 Sent Done Done false
9659 3425 Acked Read 4 Sent Done Done false
9660 3426 Acked Read 6 Sent Done Done false
9661 3427 Acked Read 2 Sent Done Done false
9662 3428 Acked Read 3 Sent Done Done false
9663 3429 Acked Flush 0 Sent Done Done false
9664 3430 Acked Write 9 Sent Done Done false
9665 3431 Acked Read 8 Sent Done Done false
9666 3432 Acked Read 3 Sent Done Done false
9667 3433 Acked Write 5 Sent Done Done false
9668 3434 Acked Write 9 Sent Done Done false
9669 3435 Acked Read 9 Sent Done Done false
9670 3436 Acked Write 6 Sent Done Done false
9671 3437 Acked Write 4 Sent Done Done false
9672 3438 Acked Read 5 Sent Done Done false
9673 3439 Acked Flush 0 Sent Done Done false
9674 3440 Acked Write 3 Sent Done Done false
9675 3441 Acked Write 9 Sent Done Done false
9676 3442 Acked Read 10 Sent Done Done false
9677 3443 Acked Read 5 Sent Done Done false
9678 3444 Acked Read 4 Sent Done Done false
9679 3445 Acked Read 1 Sent Done Done false
9680 3446 Acked Read 9 Sent Done Done false
9681 3447 Acked Flush 0 Sent Done Done false
9682 3448 Acked Read 2 Sent Done Done false
9683 3449 Acked Read 4 Sent Done Done false
9684 3450 Acked Read 4 Sent Done Done false
9685 3451 Acked Read 9 Sent Done Done false
9686 3452 Acked Flush 0 Sent Done Done false
9687 3453 Acked Write 4 Sent Done Done false
9688 3454 Acked Write 8 Sent Done Done false
9689 3455 Acked Flush 0 Sent Done Done false
9690 3456 Acked Read 8 Sent Done Done false
9691 3457 Acked Read 5 Sent Done Done false
9692 3458 Acked Flush 0 Sent Done Done false
9693 3459 Acked Flush 0 Sent Done Done false
9694 3460 Acked Write 1 Sent Done Done false
9695 3461 Acked Read 8 Sent Done Done false
9696 3462 Acked Write 1 Sent Done Done false
9697 3463 Acked Write 4 Sent Done Done false
9698 3464 Acked Flush 0 Sent Done Done false
9699 3465 Acked Flush 0 Sent Done Done false
9700 3466 Acked Write 10 Sent Done Done false
9701 3467 Acked Write 2 Sent Done Done false
9702 3468 Acked Flush 0 Sent Done Done false
9703 3469 Acked Read 1 Sent Done Done false
9704 3470 Acked Read 4 Sent Done Done false
9705 3471 Acked Write 10 Sent Done Done false
9706 3472 Acked Write 9 Sent Done Done false
9707 3473 Acked Read 2 Sent Done Done false
9708 3474 Acked Write 4 Sent Done Done false
9709 3475 Acked Write 7 Sent Done Done false
9710 3476 Acked Read 7 Sent Done Done false
9711 3477 Acked Write 3 Sent Done Done false
9712 3478 Acked Read 8 Sent Done Done false
9713 3479 Acked Read 2 Sent Done Done false
9714 3480 Acked Read 1 Sent Done Done false
9715 3481 Acked Read 10 Sent Done Done false
9716 3482 Acked Write 9 Sent Done Done false
9717 3483 Acked Read 1 Sent Done Done false
9718 3484 Acked Write 10 Sent Done Done false
9719 3485 Acked Write 5 Sent Done Done false
9720 3486 Acked Read 6 Sent Done Done false
9721 3487 Acked Write 4 Sent Done Done false
9722 3488 Acked Write 10 Sent Done Done false
9723 3489 Acked Read 6 Sent Done Done false
9724 3490 Acked Flush 0 Sent Done Done false
9725 3491 Acked Flush 0 Sent Done Done false
9726 3492 Acked Read 1 Sent Done Done false
9727 3493 Acked Write 3 Sent Done Done false
9728 3494 Acked Write 4 Sent Done Done false
9729 3495 Acked Read 9 Sent Done Done false
9730 3496 Acked Write 3 Sent Done Done false
9731 3497 Acked Read 8 Sent Done Done false
9732 3498 Acked Write 6 Sent Done Done false
9733 3499 Acked Read 1 Sent Done Done false
9734 3500 Acked Read 2 Sent Done Done false
9735 3501 Acked Write 3 Sent Done Done false
9736 3502 Acked Read 6 Sent Done Done false
9737 3503 Acked Write 1 Sent Done Done false
9738 3504 Acked Read 4 Sent Done Done false
9739 3505 Acked Read 3 Sent Done Done false
9740 3506 Acked Read 6 Sent Done Done false
9741 3507 Acked Write 1 Sent Done Done false
9742 3508 Acked Write 4 Sent Done Done false
9743 3509 Acked Read 3 Sent Done Done false
9744 3510 Acked Write 7 Sent Done Done false
9745 3511 Acked Read 8 Sent Done Done false
9746 3512 Acked Flush 0 Sent Done Done false
9747 3513 Acked Read 4 Sent Done Done false
9748 3514 Acked Read 7 Sent Done Done false
9749 3515 Acked Write 7 Sent Done Done false
9750 3516 Acked Write 1 Sent Done Done false
9751 3517 Acked Read 3 Sent Done Done false
9752 3518 Acked Read 4 Sent Done Done false
9753 3519 Acked Flush 0 Sent Done Done false
9754 3520 Acked Read 7 Sent Done Done false
9755 3521 Acked Read 3 Sent Done Done false
9756 3522 Acked Read 3 Sent Done Done false
9757 3523 Acked Read 3 Sent Done Done false
9758 3524 Acked Read 1 Sent Done Done false
9759 3525 Acked Flush 0 Sent Done Done false
9760 3526 Acked Read 3 Sent Done Done false
9761 3527 Acked Write 9 Sent Done Done false
9762 3528 Acked Write 10 Sent Done Done false
9763 3529 Acked Read 5 Sent Done Done false
9764 3530 Acked Write 5 Sent Done Done false
9765 3531 Acked Write 1 Sent Done Done false
9766 3532 Acked Read 9 Sent Done Done false
9767 3533 Acked Read 4 Sent Done Done false
9768 3534 Acked Write 6 Sent Done Done false
9769 3535 Acked Write 4 Sent Done Done false
9770 3536 Acked Read 10 Sent Done Done false
9771 3537 Acked Write 9 Sent Done Done false
9772 3538 Acked Write 10 Sent Done Done false
9773 3539 Acked Write 6 Sent Done Done false
9774 3540 Acked Write 1 Sent Done Done false
9775 3541 Acked Read 9 Sent Done Done false
9776 3542 Acked Read 2 Sent Done Done false
9777 3543 Acked Read 7 Sent Done Done false
9778 3544 Acked Write 7 Sent Done Done false
9779 3545 Acked Flush 0 Sent Done Done false
9780 3546 Acked Write 2 Sent Done Done false
9781 3547 Acked Read 7 Sent Done Done false
9782 3548 Acked Write 8 Sent Done Done false
9783 3549 Acked Write 2 Sent Done Done false
9784 3550 Acked Write 1 Sent Done Done false
9785 3551 Acked Read 5 Sent Done Done false
9786 3552 Acked Read 9 Sent Done Done false
9787 3553 Acked Write 5 Sent Done Done false
9788 3554 Acked Write 6 Sent Done Done false
9789 3555 Acked Read 1 Sent Done Done false
9790 3556 Acked Read 6 Sent Done Done false
9791 3557 Acked Read 8 Sent Done Done false
9792 3558 Acked Write 5 Sent Done Done false
9793 3559 Acked Flush 0 Sent Done Done false
9794 3560 Acked Read 8 Sent Done Done false
9795 3561 Acked Read 1 Sent Done Done false
9796 3562 Acked Write 8 Sent Done Done false
9797 3563 Acked Write 1 Sent Done Done false
9798 3564 Acked Write 3 Sent Done Done false
9799 3565 Acked Read 2 Sent Done Done false
9800 3566 Acked Write 10 Sent Done Done false
9801 3567 Acked Read 2 Sent Done Done false
9802 3568 Acked Read 8 Sent Done Done false
9803 3569 Acked Write 10 Sent Done Done false
9804 3570 Acked Write 3 Sent Done Done false
9805 3571 Acked Read 7 Sent Done Done false
9806 3572 Acked Read 6 Sent Done Done false
9807 3573 Acked Flush 0 Sent Done Done false
9808 3574 Acked Write 6 Sent Done Done false
9809 3575 Acked Read 6 Sent Done Done false
9810 3576 Acked Read 8 Sent Done Done false
9811 3577 Acked Read 3 Sent Done Done false
9812 3578 Acked Flush 0 Sent Done Done false
9813 3579 Acked Read 5 Sent Done Done false
9814 3580 Acked Write 4 Sent Done Done false
9815 3581 Acked Write 10 Sent Done Done false
9816 3582 Acked Read 2 Sent Done Done false
9817 3583 Acked Read 5 Sent Done Done false
9818 3584 Acked Read 1 Sent Done Done false
9819 3585 Acked Read 10 Sent Done Done false
9820 3586 Acked Read 1 Sent Done Done false
9821 3587 Acked Write 2 Sent Done Done false
9822 3588 Acked Write 3 Sent Done Done false
9823 3589 Acked Read 2 Sent Done Done false
9824 3590 Acked Read 5 Sent Done Done false
9825 3591 Acked Write 4 Sent Done Done false
9826 3592 Acked Write 3 Sent Done Done false
9827 3593 Acked Read 10 Sent Done Done false
9828 3594 Acked Write 9 Sent Done Done false
9829 3595 Acked Flush 0 Sent Done Done false
9830 3596 Acked Read 7 Sent Done Done false
9831 3597 Acked Write 8 Sent Done Done false
9832 3598 Acked Read 4 Sent Done Done false
9833 3599 Acked Read 1 Sent Done Done false
9834 3600 Acked Read 8 Sent Done Done false
9835 3601 Acked Read 2 Sent Done Done false
9836 3602 Acked Write 8 Sent Done Done false
9837 3603 Acked Read 7 Sent Done Done false
9838 3604 Acked Write 3 Sent Done Done false
9839 3605 Acked Read 2 Sent Done Done false
9840 3606 Acked Write 2 Sent Done Done false
9841 3607 Acked Write 4 Sent Done Done false
9842 3608 Acked Flush 0 Sent Done Done false
9843 3609 Acked Write 6 Sent Done Done false
9844 3610 Acked Write 8 Sent Done Done false
9845 3611 Acked Write 5 Sent Done Done false
9846 3612 Acked Read 1 Sent Done Done false
9847 3613 Acked Flush 0 Sent Done Done false
9848 3614 Acked Write 7 Sent Done Done false
9849 3615 Acked Read 4 Sent Done Done false
9850 3616 Acked Write 1 Sent Done Done false
9851 3617 Acked Read 9 Sent Done Done false
9852 3618 Acked Read 10 Sent Done Done false
9853 3619 Acked Flush 0 Sent Done Done false
9854 3620 Acked Read 1 Sent Done Done false
9855 3621 Acked Write 7 Sent Done Done false
9856 3622 Acked Write 10 Sent Done Done false
9857 3623 Acked Flush 0 Sent Done Done false
9858 3624 Acked Read 7 Sent Done Done false
9859 3625 Acked Read 5 Sent Done Done false
9860 3626 Acked Read 6 Sent Done Done false
9861 3627 Acked Read 2 Sent Done Done false
9862 3628 Acked Write 3 Sent Done Done false
9863 3629 Acked Read 7 Sent Done Done false
9864 3630 Acked Flush 0 Sent Done Done false
9865 3631 Acked Write 1 Sent Done Done false
9866 3632 Acked Write 10 Sent Done Done false
9867 3633 Acked Write 9 Sent Done Done false
9868 3634 Acked Write 6 Sent Done Done false
9869 3635 Acked Write 1 Sent Done Done false
9870 3636 Acked Read 3 Sent Done Done false
9871 3637 Acked Flush 0 Sent Done Done false
9872 3638 Acked Read 10 Sent Done Done false
9873 3639 Acked Read 9 Sent Done Done false
9874 3640 Acked Read 7 Sent Done Done false
9875 3641 Acked Read 8 Sent Done Done false
9876 3642 Acked Write 4 Sent Done Done false
9877 3643 Acked Read 2 Sent Done Done false
9878 3644 Acked Write 9 Sent Done Done false
9879 3645 Acked Read 9 Sent Done Done false
9880 3646 Acked Read 1 Sent Done Done false
9881 3647 Acked Write 3 Sent Done Done false
9882 3648 Acked Read 5 Sent Done Done false
9883 3649 Acked Read 6 Sent Done Done false
9884 3650 Acked Write 9 Sent Done Done false
9885 3651 Acked Read 2 Sent Done Done false
9886 3652 Acked Write 1 Sent Done Done false
9887 3653 Acked Read 8 Sent Done Done false
9888 3654 Acked Read 3 Sent Done Done false
9889 3655 Acked Read 5 Sent Done Done false
9890 3656 Acked Flush 0 Sent Done Done false
9891 3657 Acked Write 1 Sent Done Done false
9892 3658 Acked Write 10 Sent Done Done false
9893 3659 Acked Write 8 Sent Done Done false
9894 3660 Acked Write 8 Sent Done Done false
9895 3661 Acked Read 9 Sent Done Done false
9896 3662 Acked Read 6 Sent Done Done false
9897 3663 Acked Write 3 Sent Done Done false
9898 3664 Acked Read 4 Sent Done Done false
9899 3665 Acked Flush 0 Sent Done Done false
9900 3666 Acked Write 10 Sent Done Done false
9901 3667 Acked Read 1 Sent Done Done false
9902 3668 Acked Read 9 Sent Done Done false
9903 3669 Acked Write 4 Sent Done Done false
9904 3670 Acked Read 3 Sent Done Done false
9905 3671 Acked Flush 0 Sent Done Done false
9906 3672 Acked Read 7 Sent Done Done false
9907 3673 Acked Read 6 Sent Done Done false
9908 3674 Acked Write 2 Sent Done Done false
9909 3675 Acked Read 8 Sent Done Done false
9910 3676 Acked Flush 0 Sent Done Done false
9911 3677 Acked Read 10 Sent Done Done false
9912 3678 Acked Write 6 Sent Done Done false
9913 3679 Acked Read 3 Sent Done Done false
9914 3680 Acked Read 3 Sent Done Done false
9915 3681 Acked Read 1 Sent Done Done false
9916 3682 Acked Write 9 Sent Done Done false
9917 3683 Acked Write 2 Sent Done Done false
9918 3684 Acked Read 2 Sent Done Done false
9919 3685 Acked Read 5 Sent Done Done false
9920 3686 Acked Write 10 Sent Done Done false
9921 3687 Acked Read 10 Sent Done Done false
9922 3688 Acked Flush 0 Sent Done Done false
9923 3689 Acked Read 4 Sent Done Done false
9924 3690 Acked Read 5 Sent Done Done false
9925 3691 Acked Flush 0 Sent Done Done false
9926 3692 Acked Write 8 Sent Done Done false
9927 3693 Acked Write 8 Sent Done Done false
9928 3694 Acked Read 10 Sent Done Done false
9929 3695 Acked Write 7 Sent Done Done false
9930 3696 Acked Read 4 Sent Done Done false
9931 3697 Acked Flush 0 Sent Done Done false
9932 3698 Acked Write 9 Sent Done Done false
9933 3699 Acked Write 9 Sent Done Done false
9934 3700 Acked Read 5 Sent Done Done false
9935 3701 Acked Write 8 Sent Done Done false
9936 3702 Acked Flush 0 Sent Done Done false
9937 3703 Acked Flush 0 Sent Done Done false
9938 3704 Acked Read 9 Sent Done Done false
9939 3705 Acked Write 10 Sent Done Done false
9940 3706 Acked Write 2 Sent Done Done false
9941 3707 Acked Read 9 Sent Done Done false
9942 3708 Acked Read 10 Sent Done Done false
9943 3709 Acked Read 2 Sent Done Done false
9944 3710 Acked Read 4 Sent Done Done false
9945 3711 Acked Write 5 Sent Done Done false
9946 3712 Acked Flush 0 Sent Done Done false
9947 3713 Acked Write 2 Sent Done Done false
9948 3714 Acked Flush 0 Sent Done Done false
9949 3715 Acked Write 9 Sent Done Done false
9950 3716 Acked Read 10 Sent Done Done false
9951 STATES DS:0 DS:1 DS:2 TOTAL
9952 Sent 301 0 0 301
9953 Done 0 301 301 602
9954 Skipped 0 0 0 0
9955 Error 0 0 0 0
9956 Last Flush: Some(JobId(3415)) Some(JobId(3714)) Some(JobId(3714))
9957 Downstairs last five completed: 3415 3414 3413 3412 3411
9958 Upstairs last five completed: 3716 3715 3714 3713 3712
9959 CLIENT: Up:0 ds:301 act:2
99602025-02-21T06:47:29.989ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
99612025-02-21T06:47:30.851ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
99622025-02-21T06:47:31.853ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
99632025-02-21T06:47:32.854ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
9964 [8810] got start action
9965 [8810] I am going to restart
9966 Starting downstairs at port 8810
9967 Make output file at "/tmp/dsc/downstairs-8810.txt"
9968 [8810][0] reports Starting
9969 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1215)
9970 [8810][0] reports Running
9971 ----------------------------------------------------------------
9972 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9973 States: Offline Active Active
9974 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9975 3416 Acked Read 5 Sent Done Done false
9976 3417 Acked Read 1 Sent Done Done false
9977 3418 Acked Read 4 Sent Done Done false
9978 3419 Acked Flush 0 Sent Done Done false
9979 3420 Acked Read 8 Sent Done Done false
9980 3421 Acked Read 7 Sent Done Done false
9981 3422 Acked Read 7 Sent Done Done false
9982 3423 Acked Write 1 Sent Done Done false
9983 3424 Acked Read 6 Sent Done Done false
9984 3425 Acked Read 4 Sent Done Done false
9985 3426 Acked Read 6 Sent Done Done false
9986 3427 Acked Read 2 Sent Done Done false
9987 3428 Acked Read 3 Sent Done Done false
9988 3429 Acked Flush 0 Sent Done Done false
9989 3430 Acked Write 9 Sent Done Done false
9990 3431 Acked Read 8 Sent Done Done false
9991 3432 Acked Read 3 Sent Done Done false
9992 3433 Acked Write 5 Sent Done Done false
9993 3434 Acked Write 9 Sent Done Done false
9994 3435 Acked Read 9 Sent Done Done false
9995 3436 Acked Write 6 Sent Done Done false
9996 3437 Acked Write 4 Sent Done Done false
9997 3438 Acked Read 5 Sent Done Done false
9998 3439 Acked Flush 0 Sent Done Done false
9999 3440 Acked Write 3 Sent Done Done false
10000 3441 Acked Write 9 Sent Done Done false
10001 3442 Acked Read 10 Sent Done Done false
10002 3443 Acked Read 5 Sent Done Done false
10003 3444 Acked Read 4 Sent Done Done false
10004 3445 Acked Read 1 Sent Done Done false
10005 3446 Acked Read 9 Sent Done Done false
10006 3447 Acked Flush 0 Sent Done Done false
10007 3448 Acked Read 2 Sent Done Done false
10008 3449 Acked Read 4 Sent Done Done false
10009 3450 Acked Read 4 Sent Done Done false
10010 3451 Acked Read 9 Sent Done Done false
10011 3452 Acked Flush 0 Sent Done Done false
10012 3453 Acked Write 4 Sent Done Done false
10013 3454 Acked Write 8 Sent Done Done false
10014 3455 Acked Flush 0 Sent Done Done false
10015 3456 Acked Read 8 Sent Done Done false
10016 3457 Acked Read 5 Sent Done Done false
10017 3458 Acked Flush 0 Sent Done Done false
10018 3459 Acked Flush 0 Sent Done Done false
10019 3460 Acked Write 1 Sent Done Done false
10020 3461 Acked Read 8 Sent Done Done false
10021 3462 Acked Write 1 Sent Done Done false
10022 3463 Acked Write 4 Sent Done Done false
10023 3464 Acked Flush 0 Sent Done Done false
10024 3465 Acked Flush 0 Sent Done Done false
10025 3466 Acked Write 10 Sent Done Done false
10026 3467 Acked Write 2 Sent Done Done false
10027 3468 Acked Flush 0 Sent Done Done false
10028 3469 Acked Read 1 Sent Done Done false
10029 3470 Acked Read 4 Sent Done Done false
10030 3471 Acked Write 10 Sent Done Done false
10031 3472 Acked Write 9 Sent Done Done false
10032 3473 Acked Read 2 Sent Done Done false
10033 3474 Acked Write 4 Sent Done Done false
10034 3475 Acked Write 7 Sent Done Done false
10035 3476 Acked Read 7 Sent Done Done false
10036 3477 Acked Write 3 Sent Done Done false
10037 3478 Acked Read 8 Sent Done Done false
10038 3479 Acked Read 2 Sent Done Done false
10039 3480 Acked Read 1 Sent Done Done false
10040 3481 Acked Read 10 Sent Done Done false
10041 3482 Acked Write 9 Sent Done Done false
10042 3483 Acked Read 1 Sent Done Done false
10043 3484 Acked Write 10 Sent Done Done false
10044 3485 Acked Write 5 Sent Done Done false
10045 3486 Acked Read 6 Sent Done Done false
10046 3487 Acked Write 4 Sent Done Done false
10047 3488 Acked Write 10 Sent Done Done false
10048 3489 Acked Read 6 Sent Done Done false
10049 3490 Acked Flush 0 Sent Done Done false
10050 3491 Acked Flush 0 Sent Done Done false
10051 3492 Acked Read 1 Sent Done Done false
10052 3493 Acked Write 3 Sent Done Done false
10053 3494 Acked Write 4 Sent Done Done false
10054 3495 Acked Read 9 Sent Done Done false
10055 3496 Acked Write 3 Sent Done Done false
10056 3497 Acked Read 8 Sent Done Done false
10057 3498 Acked Write 6 Sent Done Done false
10058 3499 Acked Read 1 Sent Done Done false
10059 3500 Acked Read 2 Sent Done Done false
10060 3501 Acked Write 3 Sent Done Done false
10061 3502 Acked Read 6 Sent Done Done false
10062 3503 Acked Write 1 Sent Done Done false
10063 3504 Acked Read 4 Sent Done Done false
10064 3505 Acked Read 3 Sent Done Done false
10065 3506 Acked Read 6 Sent Done Done false
10066 3507 Acked Write 1 Sent Done Done false
10067 3508 Acked Write 4 Sent Done Done false
10068 3509 Acked Read 3 Sent Done Done false
10069 3510 Acked Write 7 Sent Done Done false
10070 3511 Acked Read 8 Sent Done Done false
10071 3512 Acked Flush 0 Sent Done Done false
10072 3513 Acked Read 4 Sent Done Done false
10073 3514 Acked Read 7 Sent Done Done false
10074 3515 Acked Write 7 Sent Done Done false
10075 3516 Acked Write 1 Sent Done Done false
10076 3517 Acked Read 3 Sent Done Done false
10077 3518 Acked Read 4 Sent Done Done false
10078 3519 Acked Flush 0 Sent Done Done false
10079 3520 Acked Read 7 Sent Done Done false
10080 3521 Acked Read 3 Sent Done Done false
10081 3522 Acked Read 3 Sent Done Done false
10082 3523 Acked Read 3 Sent Done Done false
10083 3524 Acked Read 1 Sent Done Done false
10084 3525 Acked Flush 0 Sent Done Done false
10085 3526 Acked Read 3 Sent Done Done false
10086 3527 Acked Write 9 Sent Done Done false
10087 3528 Acked Write 10 Sent Done Done false
10088 3529 Acked Read 5 Sent Done Done false
10089 3530 Acked Write 5 Sent Done Done false
10090 3531 Acked Write 1 Sent Done Done false
10091 3532 Acked Read 9 Sent Done Done false
10092 3533 Acked Read 4 Sent Done Done false
10093 3534 Acked Write 6 Sent Done Done false
10094 3535 Acked Write 4 Sent Done Done false
10095 3536 Acked Read 10 Sent Done Done false
10096 3537 Acked Write 9 Sent Done Done false
10097 3538 Acked Write 10 Sent Done Done false
10098 3539 Acked Write 6 Sent Done Done false
10099 3540 Acked Write 1 Sent Done Done false
10100 3541 Acked Read 9 Sent Done Done false
10101 3542 Acked Read 2 Sent Done Done false
10102 3543 Acked Read 7 Sent Done Done false
10103 3544 Acked Write 7 Sent Done Done false
10104 3545 Acked Flush 0 Sent Done Done false
10105 3546 Acked Write 2 Sent Done Done false
10106 3547 Acked Read 7 Sent Done Done false
10107 3548 Acked Write 8 Sent Done Done false
10108 3549 Acked Write 2 Sent Done Done false
10109 3550 Acked Write 1 Sent Done Done false
10110 3551 Acked Read 5 Sent Done Done false
10111 3552 Acked Read 9 Sent Done Done false
10112 3553 Acked Write 5 Sent Done Done false
10113 3554 Acked Write 6 Sent Done Done false
10114 3555 Acked Read 1 Sent Done Done false
10115 3556 Acked Read 6 Sent Done Done false
10116 3557 Acked Read 8 Sent Done Done false
10117 3558 Acked Write 5 Sent Done Done false
10118 3559 Acked Flush 0 Sent Done Done false
10119 3560 Acked Read 8 Sent Done Done false
10120 3561 Acked Read 1 Sent Done Done false
10121 3562 Acked Write 8 Sent Done Done false
10122 3563 Acked Write 1 Sent Done Done false
10123 3564 Acked Write 3 Sent Done Done false
10124 3565 Acked Read 2 Sent Done Done false
10125 3566 Acked Write 10 Sent Done Done false
10126 3567 Acked Read 2 Sent Done Done false
10127 3568 Acked Read 8 Sent Done Done false
10128 3569 Acked Write 10 Sent Done Done false
10129 3570 Acked Write 3 Sent Done Done false
10130 3571 Acked Read 7 Sent Done Done false
10131 3572 Acked Read 6 Sent Done Done false
10132 3573 Acked Flush 0 Sent Done Done false
10133 3574 Acked Write 6 Sent Done Done false
10134 3575 Acked Read 6 Sent Done Done false
10135 3576 Acked Read 8 Sent Done Done false
10136 3577 Acked Read 3 Sent Done Done false
10137 3578 Acked Flush 0 Sent Done Done false
10138 3579 Acked Read 5 Sent Done Done false
10139 3580 Acked Write 4 Sent Done Done false
10140 3581 Acked Write 10 Sent Done Done false
10141 3582 Acked Read 2 Sent Done Done false
10142 3583 Acked Read 5 Sent Done Done false
10143 3584 Acked Read 1 Sent Done Done false
10144 3585 Acked Read 10 Sent Done Done false
10145 3586 Acked Read 1 Sent Done Done false
10146 3587 Acked Write 2 Sent Done Done false
10147 3588 Acked Write 3 Sent Done Done false
10148 3589 Acked Read 2 Sent Done Done false
10149 3590 Acked Read 5 Sent Done Done false
10150 3591 Acked Write 4 Sent Done Done false
10151 3592 Acked Write 3 Sent Done Done false
10152 3593 Acked Read 10 Sent Done Done false
10153 3594 Acked Write 9 Sent Done Done false
10154 3595 Acked Flush 0 Sent Done Done false
10155 3596 Acked Read 7 Sent Done Done false
10156 3597 Acked Write 8 Sent Done Done false
10157 3598 Acked Read 4 Sent Done Done false
10158 3599 Acked Read 1 Sent Done Done false
10159 3600 Acked Read 8 Sent Done Done false
10160 3601 Acked Read 2 Sent Done Done false
10161 3602 Acked Write 8 Sent Done Done false
10162 3603 Acked Read 7 Sent Done Done false
10163 3604 Acked Write 3 Sent Done Done false
10164 3605 Acked Read 2 Sent Done Done false
10165 3606 Acked Write 2 Sent Done Done false
10166 3607 Acked Write 4 Sent Done Done false
10167 3608 Acked Flush 0 Sent Done Done false
10168 3609 Acked Write 6 Sent Done Done false
10169 3610 Acked Write 8 Sent Done Done false
10170 3611 Acked Write 5 Sent Done Done false
10171 3612 Acked Read 1 Sent Done Done false
10172 3613 Acked Flush 0 Sent Done Done false
10173 3614 Acked Write 7 Sent Done Done false
10174 3615 Acked Read 4 Sent Done Done false
10175 3616 Acked Write 1 Sent Done Done false
10176 3617 Acked Read 9 Sent Done Done false
10177 3618 Acked Read 10 Sent Done Done false
10178 3619 Acked Flush 0 Sent Done Done false
10179 3620 Acked Read 1 Sent Done Done false
10180 3621 Acked Write 7 Sent Done Done false
10181 3622 Acked Write 10 Sent Done Done false
10182 3623 Acked Flush 0 Sent Done Done false
10183 3624 Acked Read 7 Sent Done Done false
10184 3625 Acked Read 5 Sent Done Done false
10185 3626 Acked Read 6 Sent Done Done false
10186 3627 Acked Read 2 Sent Done Done false
10187 3628 Acked Write 3 Sent Done Done false
10188 3629 Acked Read 7 Sent Done Done false
10189 3630 Acked Flush 0 Sent Done Done false
10190 3631 Acked Write 1 Sent Done Done false
10191 3632 Acked Write 10 Sent Done Done false
10192 3633 Acked Write 9 Sent Done Done false
10193 3634 Acked Write 6 Sent Done Done false
10194 3635 Acked Write 1 Sent Done Done false
10195 3636 Acked Read 3 Sent Done Done false
10196 3637 Acked Flush 0 Sent Done Done false
10197 3638 Acked Read 10 Sent Done Done false
10198 3639 Acked Read 9 Sent Done Done false
10199 3640 Acked Read 7 Sent Done Done false
10200 3641 Acked Read 8 Sent Done Done false
10201 3642 Acked Write 4 Sent Done Done false
10202 3643 Acked Read 2 Sent Done Done false
10203 3644 Acked Write 9 Sent Done Done false
10204 3645 Acked Read 9 Sent Done Done false
10205 3646 Acked Read 1 Sent Done Done false
10206 3647 Acked Write 3 Sent Done Done false
10207 3648 Acked Read 5 Sent Done Done false
10208 3649 Acked Read 6 Sent Done Done false
10209 3650 Acked Write 9 Sent Done Done false
10210 3651 Acked Read 2 Sent Done Done false
10211 3652 Acked Write 1 Sent Done Done false
10212 3653 Acked Read 8 Sent Done Done false
10213 3654 Acked Read 3 Sent Done Done false
10214 3655 Acked Read 5 Sent Done Done false
10215 3656 Acked Flush 0 Sent Done Done false
10216 3657 Acked Write 1 Sent Done Done false
10217 3658 Acked Write 10 Sent Done Done false
10218 3659 Acked Write 8 Sent Done Done false
10219 3660 Acked Write 8 Sent Done Done false
10220 3661 Acked Read 9 Sent Done Done false
10221 3662 Acked Read 6 Sent Done Done false
10222 3663 Acked Write 3 Sent Done Done false
10223 3664 Acked Read 4 Sent Done Done false
10224 3665 Acked Flush 0 Sent Done Done false
10225 3666 Acked Write 10 Sent Done Done false
10226 3667 Acked Read 1 Sent Done Done false
10227 3668 Acked Read 9 Sent Done Done false
10228 3669 Acked Write 4 Sent Done Done false
10229 3670 Acked Read 3 Sent Done Done false
10230 3671 Acked Flush 0 Sent Done Done false
10231 3672 Acked Read 7 Sent Done Done false
10232 3673 Acked Read 6 Sent Done Done false
10233 3674 Acked Write 2 Sent Done Done false
10234 3675 Acked Read 8 Sent Done Done false
10235 3676 Acked Flush 0 Sent Done Done false
10236 3677 Acked Read 10 Sent Done Done false
10237 3678 Acked Write 6 Sent Done Done false
10238 3679 Acked Read 3 Sent Done Done false
10239 3680 Acked Read 3 Sent Done Done false
10240 3681 Acked Read 1 Sent Done Done false
10241 3682 Acked Write 9 Sent Done Done false
10242 3683 Acked Write 2 Sent Done Done false
10243 3684 Acked Read 2 Sent Done Done false
10244 3685 Acked Read 5 Sent Done Done false
10245 3686 Acked Write 10 Sent Done Done false
10246 3687 Acked Read 10 Sent Done Done false
10247 3688 Acked Flush 0 Sent Done Done false
10248 3689 Acked Read 4 Sent Done Done false
10249 3690 Acked Read 5 Sent Done Done false
10250 3691 Acked Flush 0 Sent Done Done false
10251 3692 Acked Write 8 Sent Done Done false
10252 3693 Acked Write 8 Sent Done Done false
10253 3694 Acked Read 10 Sent Done Done false
10254 3695 Acked Write 7 Sent Done Done false
10255 3696 Acked Read 4 Sent Done Done false
10256 3697 Acked Flush 0 Sent Done Done false
10257 3698 Acked Write 9 Sent Done Done false
10258 3699 Acked Write 9 Sent Done Done false
10259 3700 Acked Read 5 Sent Done Done false
10260 3701 Acked Write 8 Sent Done Done false
10261 3702 Acked Flush 0 Sent Done Done false
10262 3703 Acked Flush 0 Sent Done Done false
10263 3704 Acked Read 9 Sent Done Done false
10264 3705 Acked Write 10 Sent Done Done false
10265 3706 Acked Write 2 Sent Done Done false
10266 3707 Acked Read 9 Sent Done Done false
10267 3708 Acked Read 10 Sent Done Done false
10268 3709 Acked Read 2 Sent Done Done false
10269 3710 Acked Read 4 Sent Done Done false
10270 3711 Acked Write 5 Sent Done Done false
10271 3712 Acked Flush 0 Sent Done Done false
10272 3713 Acked Write 2 Sent Done Done false
10273 3714 Acked Flush 0 Sent Done Done false
10274 3715 Acked Write 9 Sent Done Done false
10275 3716 Acked Read 10 Sent Done Done false
10276 STATES DS:0 DS:1 DS:2 TOTAL
10277 Sent 301 0 0 301
10278 Done 0 301 301 602
10279 Skipped 0 0 0 0
10280 Error 0 0 0 0
10281 Last Flush: Some(JobId(3415)) Some(JobId(3714)) Some(JobId(3714))
10282 Downstairs last five completed: 3415 3414 3413 3412 3411
10283 Upstairs last five completed: 3716 3715 3714 3713 3712
10284 CLIENT: Up:0 ds:301 act:2
102852025-02-21T06:47:33.992ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
102862025-02-21T06:47:34.978ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
102872025-02-21T06:47:35.250ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
102882025-02-21T06:47:35.250ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49053) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
102892025-02-21T06:47:35.251ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
102902025-02-21T06:47:35.252ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
102912025-02-21T06:47:35.252ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
102922025-02-21T06:47:35.252ZINFOcrucible: send last flush ID to this DS: Some(JobId(3415)) = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
102932025-02-21T06:47:35.252ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
102942025-02-21T06:47:35.252ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(3415)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
102952025-02-21T06:47:35.252ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(3415)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
10296 ----------------------------------------------------------------
10297 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10298 Downstairs last five completed: 3717 3716 3715 3714 3713
10299 Upstairs last five completed: 3717 3716 3715 3714 3713
10300 CLIENT: Up:0 ds:0 act:3
10301 Replay: All jobs finished
10302 Main task has work to do, go find it
10303 got dsc Stop(0)
10304 stop 0
10305 [8810] Got stop action so:false kr:false
10306 Feb 21 06:47:37.493 INFO request completed, latency_us: 375, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 4e26e209-6517-42b8-85f8-5dc62b2d51fb, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
103072025-02-21T06:47:37.494ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
103082025-02-21T06:47:37.494ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
103092025-02-21T06:47:37.494ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
10310 Feb 21 06:47:37.494 INFO request completed, latency_us: 156, response_code: 200, uri: /state/cid/0, method: GET, req_id: ecda4195-7b0d-4f3a-a505-534394cd15b8, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
103112025-02-21T06:47:37.494ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
10312 [8810] Exited with: ExitStatus(unix_wait_status(9))
10313 [8810][0] reports Exit
10314 Feb 21 06:47:41.496 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/0, method: GET, req_id: f0b01177-9bcc-4b93-ba29-eabb11a7741c, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
10315 001/300 Read block 4130 len 4096
10316 002/300 Read block 4717 len 32768
10317 003/300 Write block 492 len 4096 data: 3
10318 004/300 Write block 1661 len 20480 data: 2 3 4 4 3
10319 005/300 Read block 2934 len 40960
10320 006/300 Write block 4023 len 16384 data: 3 2 2 2
10321 007/300 Write block 2784 len 40960 data: 3 4 3 3 3 3 3 2 2 2
10322 008/300 Write block 1484 len 12288 data: 4 4 4
10323 009/300 Read block 3784 len 12288
10324 010/300 Read block 4200 len 16384
10325 011/300 Read block 326 len 32768
10326 012/300 Read block 3448 len 8192
10327 013/300 Flush
10328 014/300 Read block 733 len 36864
10329 015/300 Flush
10330 016/300 Read block 723 len 8192
10331 017/300 Flush
10332 018/300 Write block 4394 len 28672 data: 4 4 4 4 2 3 3
10333 019/300 Write block 340 len 4096 data: 3
10334 020/300 Write block 4355 len 24576 data: 3 3 3 3 3 3
10335 021/300 Read block 1035 len 4096
10336 022/300 Read block 3900 len 36864
10337 023/300 Write block 4503 len 36864 data: 2 2 2 2 2 3 3 3 3
10338 024/300 Write block 583 len 8192 data: 2 2
10339 025/300 Read block 3684 len 20480
10340 026/300 Read block 1989 len 24576
10341 027/300 Flush
10342 028/300 Flush
10343 029/300 Write block 3194 len 36864 data: 3 3 3 2 2 2 2 2 2
10344 030/300 Read block 1722 len 20480
10345 031/300 Read block 3481 len 4096
10346 032/300 Write block 2886 len 40960 data: 6 4 4 4 4 4 4 3 3 3
10347 033/300 Write block 3897 len 8192 data: 2 2
10348 034/300 Write block 3810 len 28672 data: 3 3 4 4 4 5 6
10349 035/300 Write block 4815 len 32768 data: 5 5 5 5 5 4 4 4
10350 036/300 Write block 2088 len 24576 data: 2 2 2 2 3 4
10351 037/300 Write block 1135 len 32768 data: 4 4 4 4 4 2 2 2
10352 038/300 Read block 3524 len 28672
10353 039/300 Write block 238 len 24576 data: 3 3 3 3 3 2
10354 040/300 Read block 1931 len 24576
10355 041/300 Read block 2693 len 12288
10356 042/300 Write block 4874 len 28672 data: 4 4 4 4 5 6 7
10357 043/300 Read block 3216 len 4096
10358 044/300 Read block 218 len 16384
10359 045/300 Read block 1500 len 20480
10360 046/300 Read block 3154 len 4096
10361 047/300 Read block 1906 len 28672
10362 048/300 Read block 402 len 24576
10363 049/300 Read block 741 len 12288
10364 050/300 Read block 101 len 32768
10365 051/300 Read block 409 len 40960
10366 052/300 Read block 3350 len 28672
10367 053/300 Read block 145 len 24576
10368 054/300 Read block 1445 len 32768
10369 055/300 Write block 2107 len 4096 data: 4
10370 056/300 Read block 3815 len 24576
10371 057/300 Read block 2201 len 8192
10372 058/300 Write block 3591 len 24576 data: 4 4 3 2 3 3
10373 059/300 Read block 217 len 32768
10374 060/300 Write block 2193 len 36864 data: 5 4 4 4 3 3 2 2 3
10375 061/300 Write block 4032 len 32768 data: 3 3 3 3 3 3 3 3
10376 062/300 Write block 2794 len 12288 data: 2 2 2
10377 063/300 Read block 1463 len 16384
10378 064/300 Read block 2187 len 32768
10379 065/300 Flush
10380 066/300 Write block 253 len 40960 data: 2 3 2 2 2 2 3 3 3 3
10381 067/300 Write block 4918 len 12288 data: 5 5 4
10382 068/300 Read block 4530 len 12288
10383 069/300 Write block 2816 len 8192 data: 2 3
10384 070/300 Flush
10385 071/300 Write block 1426 len 4096 data: 4
10386 072/300 Read block 2381 len 36864
10387 073/300 Write block 2643 len 36864 data: 3 4 4 4 4 3 3 2 2
10388 074/300 Write block 3563 len 40960 data: 2 3 4 4 4 3 3 3 3 3
10389 075/300 Write block 4555 len 32768 data: 3 3 2 2 2 3 3 2
10390 076/300 Write block 450 len 8192 data: 2 2
10391 077/300 Write block 4017 len 40960 data: 4 4 4 3 3 3 4 3 3 3
10392 078/300 Read block 3434 len 16384
10393 079/300 Write block 4280 len 16384 data: 2 2 2 2
10394 080/300 Flush
10395 081/300 Write block 655 len 12288 data: 2 2 3
10396 082/300 Flush
10397 083/300 Read block 1343 len 24576
10398 084/300 Flush
10399 085/300 Flush
10400 086/300 Write block 2231 len 12288 data: 3 3 4
10401 087/300 Write block 1187 len 12288 data: 4 4 4
10402 088/300 Read block 432 len 28672
10403 089/300 Write block 1032 len 4096 data: 4
10404 090/300 Read block 685 len 20480
10405 091/300 Flush
10406 092/300 Flush
10407 093/300 Flush
10408 094/300 Read block 3266 len 32768
10409 095/300 Flush
10410 096/300 Write block 4221 len 32768 data: 3 2 3 3 3 3 3 3
10411 097/300 Read block 3498 len 32768
10412 098/300 Write block 2738 len 12288 data: 3 3 3
10413 099/300 Read block 87 len 12288
10414 100/300 Flush
10415 101/300 Read block 2261 len 8192
10416 102/300 Write block 1530 len 20480 data: 4 4 4 6 5
10417 103/300 Read block 1168 len 20480
10418 104/300 Read block 2538 len 40960
10419 105/300 Read block 4313 len 20480
10420 106/300 Flush
10421 107/300 Read block 4967 len 24576
10422 108/300 Write block 2572 len 32768 data: 4 4 5 5 5 4 4 4
10423 109/300 Read block 2143 len 16384
10424 110/300 Read block 888 len 20480
10425 111/300 Write block 1013 len 40960 data: 2 2 2 3 4 5 5 5 5 3
10426 112/300 Write block 4445 len 36864 data: 4 3 3 3 3 3 3 4 4
10427 113/300 Flush
10428 114/300 Write block 1144 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10429 115/300 Read block 1695 len 24576
10430 116/300 Write block 2509 len 4096 data: 4
10431 117/300 Write block 3220 len 24576 data: 3 3 3 2 4 3
10432 118/300 Write block 3417 len 16384 data: 5 3 4 4
10433 119/300 Write block 3764 len 16384 data: 2 2 3 3
10434 120/300 Read block 378 len 36864
10435 121/300 Read block 2618 len 12288
10436 122/300 Read block 4940 len 20480
10437 123/300 Write block 4104 len 8192 data: 5 5
10438 124/300 Flush
10439 125/300 Read block 2050 len 4096
10440 126/300 Read block 4009 len 40960
10441 127/300 Read block 2541 len 28672
10442 128/300 Write block 3826 len 40960 data: 3 2 2 2 4 4 4 4 5 4
10443 129/300 Write block 1346 len 12288 data: 2 2 2
10444 130/300 Read block 4470 len 32768
10445 131/300 Read block 3490 len 12288
10446 132/300 Read block 1027 len 36864
10447 133/300 Read block 3995 len 20480
10448 134/300 Read block 124 len 8192
10449 135/300 Write block 1420 len 32768 data: 4 4 4 5 5 3 5 4
10450 136/300 Write block 1447 len 20480 data: 3 3 3 3 3
10451 137/300 Write block 1289 len 32768 data: 3 3 3 3 4 4 4 4
10452 138/300 Read block 1509 len 32768
10453 139/300 Write block 3273 len 4096 data: 3
10454 140/300 Write block 2704 len 40960 data: 3 3 3 3 2 2 2 3 3 3
10455 141/300 Write block 3260 len 32768 data: 2 3 3 4 4 3 4 4
10456 142/300 Write block 1227 len 36864 data: 2 2 2 2 2 2 2 2 2
10457 143/300 Flush
10458 144/300 Write block 1978 len 16384 data: 3 3 3 4
10459 145/300 Read block 3682 len 16384
10460 146/300 Flush
10461 147/300 Write block 1868 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10462 148/300 Flush
10463 149/300 Write block 454 len 32768 data: 3 3 4 4 4 4 4 3
10464 150/300 Read block 4761 len 20480
10465 151/300 Read block 3128 len 8192
10466 152/300 Read block 2469 len 8192
10467 153/300 Write block 287 len 36864 data: 2 2 2 2 2 3 3 3 3
10468 154/300 Read block 3532 len 12288
10469 155/300 Read block 3889 len 40960
10470 156/300 Write block 3247 len 20480 data: 6 5 6 5 4
10471 157/300 Write block 4156 len 20480 data: 3 3 2 2 2
10472 158/300 Read block 2739 len 4096
10473 159/300 Write block 28 len 12288 data: 2 2 2
10474 160/300 Read block 2724 len 20480
10475 161/300 Write block 805 len 24576 data: 3 3 2 2 2 2
10476 162/300 Read block 212 len 28672
10477 163/300 Write block 789 len 36864 data: 4 4 4 3 3 3 3 2 2
10478 164/300 Write block 1439 len 24576 data: 5 5 4 3 4 4
10479 165/300 Read block 3910 len 32768
10480 166/300 Write block 4312 len 12288 data: 4 3 3
10481 167/300 Write block 2045 len 4096 data: 4
10482 168/300 Write block 2552 len 8192 data: 2 2
10483 169/300 Write block 253 len 4096 data: 3
10484 170/300 Flush
10485 171/300 Write block 3485 len 32768 data: 3 3 3 2 2 3 3 3
10486 172/300 Read block 2984 len 36864
10487 173/300 Write block 1574 len 32768 data: 2 2 3 3 3 3 3 3
10488 174/300 Write block 1819 len 40960 data: 5 5 6 6 5 4 4 4 3 4
10489 175/300 Read block 410 len 24576
10490 176/300 Read block 2226 len 4096
10491 177/300 Write block 1599 len 24576 data: 2 2 2 2 2 2
10492 178/300 Read block 1405 len 4096
10493 179/300 Read block 2715 len 36864
10494 180/300 Write block 1142 len 32768 data: 3 2 3 3 3 3 3 3
10495 181/300 Read block 4742 len 24576
10496 182/300 Write block 4211 len 28672 data: 4 5 4 3 2 3 3
10497 183/300 Write block 1991 len 8192 data: 4 3
10498 184/300 Read block 2076 len 16384
10499 185/300 Flush
10500 186/300 Read block 1093 len 4096
10501 187/300 Write block 77 len 24576 data: 5 5 5 4 3 2
10502 188/300 Flush
10503 189/300 Read block 1630 len 24576
10504 190/300 Write block 3606 len 8192 data: 3 5
10505 191/300 Write block 2558 len 36864 data: 3 3 3 3 3 2 2 2 2
10506 192/300 Write block 1088 len 32768 data: 3 3 3 3 3 3 3 3
10507 193/300 Write block 4881 len 40960 data: 5 5 5 4 3 3 3 3 3 3
10508 194/300 Write block 2572 len 12288 data: 5 5 6
10509 195/300 Read block 2071 len 12288
10510 196/300 Write block 313 len 28672 data: 4 4 4 4 5 3 2
10511 197/300 Read block 713 len 32768
10512 198/300 Write block 2546 len 4096 data: 2
10513 199/300 Write block 2778 len 40960 data: 3 3 3 3 4 4 4 5 4 4
10514 200/300 Read block 1179 len 24576
10515 201/300 Flush
10516 202/300 Read block 2023 len 8192
10517 203/300 Read block 3221 len 40960
10518 204/300 Flush
10519 205/300 Write block 972 len 20480 data: 3 3 3 3 4
10520 206/300 Read block 55 len 40960
10521 207/300 Read block 2334 len 20480
10522 208/300 Read block 3862 len 28672
10523 209/300 Write block 3794 len 28672 data: 2 2 4 4 5 4 4
10524 210/300 Read block 1274 len 16384
10525 211/300 Write block 4479 len 36864 data: 4 3 3 3 3 3 2 2 2
10526 212/300 Write block 483 len 12288 data: 2 2 2
10527 213/300 Write block 3220 len 24576 data: 4 4 4 3 5 4
10528 214/300 Write block 2236 len 4096 data: 5
10529 215/300 Read block 2603 len 36864
10530 216/300 Write block 4915 len 12288 data: 5 6 7
10531 217/300 Write block 2177 len 12288 data: 4 2 2
10532 218/300 Read block 2565 len 28672
10533 219/300 Read block 3908 len 4096
10534 220/300 Write block 3537 len 4096 data: 2
10535 221/300 Write block 4411 len 32768 data: 4 4 4 3 3 3 3 4
10536 222/300 Write block 377 len 40960 data: 4 3 3 3 5 5 4 3 3 3
10537 223/300 Read block 898 len 20480
10538 224/300 Read block 3056 len 40960
10539 225/300 Read block 4670 len 32768
10540 226/300 Read block 1239 len 24576
10541 227/300 Read block 4177 len 16384
10542 228/300 Read block 718 len 12288
10543 229/300 Flush
10544 230/300 Read block 2304 len 20480
10545 231/300 Write block 1498 len 24576 data: 4 4 5 5 5 5
10546 232/300 Read block 4934 len 20480
10547 233/300 Read block 2514 len 40960
10548 234/300 Write block 2681 len 24576 data: 4 3 3 3 3 4
10549 235/300 Flush
10550 236/300 Read block 864 len 36864
10551 237/300 Write block 2443 len 36864 data: 2 2 2 2 2 2 2 3 3
10552 238/300 Write block 671 len 36864 data: 5 5 4 4 3 3 3 2 2
10553 239/300 Read block 3047 len 40960
10554 240/300 Read block 4790 len 36864
10555 241/300 Read block 3037 len 20480
10556 242/300 Flush
10557 243/300 Flush
10558 244/300 Write block 121 len 8192 data: 3 3
10559 245/300 Read block 1677 len 4096
10560 246/300 Flush
10561 247/300 Write block 172 len 16384 data: 3 3 2 2
10562 248/300 Read block 2040 len 24576
10563 249/300 Flush
10564 250/300 Read block 2625 len 36864
10565 251/300 Read block 3582 len 32768
10566 252/300 Read block 1829 len 32768
10567 253/300 Write block 1226 len 36864 data: 3 3 3 3 3 3 3 3 3
10568 254/300 Write block 2157 len 32768 data: 2 2 2 3 3 3 4 3
10569 255/300 Flush
10570 256/300 Read block 3174 len 28672
10571 257/300 Read block 184 len 24576
10572 258/300 Flush
10573 259/300 Write block 1260 len 24576 data: 4 4 4 4 4 4
10574 260/300 Flush
10575 261/300 Read block 2651 len 16384
10576 262/300 Write block 1952 len 24576 data: 3 2 2 3 3 2
10577 263/300 Write block 495 len 40960 data: 2 2 2 2 2 4 5 5 6 5
10578 264/300 Read block 3167 len 8192
10579 265/300 Read block 276 len 16384
10580 266/300 Read block 4888 len 32768
10581 267/300 Read block 2636 len 36864
10582 268/300 Read block 2202 len 8192
10583 269/300 Flush
10584 270/300 Read block 1681 len 24576
10585 271/300 Read block 2041 len 40960
10586 272/300 Read block 3544 len 32768
10587 273/300 Read block 862 len 36864
10588 274/300 Write block 4382 len 4096 data: 4
10589 275/300 Read block 3028 len 8192
10590 276/300 Read block 1104 len 40960
10591 277/300 Read block 2705 len 20480
10592 278/300 Write block 512 len 24576 data: 2 3 2 2 2 2
10593 279/300 Write block 4157 len 8192 data: 4 3
10594 280/300 Read block 1288 len 32768
10595 281/300 Write block 4686 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10596 282/300 Read block 65 len 28672
10597 283/300 Read block 4387 len 16384
10598 284/300 Write block 2032 len 28672 data: 2 2 2 2 3 3 4
10599 285/300 Write block 3135 len 8192 data: 5 6
10600 286/300 Write block 1605 len 32768 data: 2 2 4 5 7 6 7 8
10601 287/300 Read block 3963 len 16384
10602 288/300 Write block 899 len 40960 data: 2 2 3 4 4 5 5 4 3 3
10603 289/300 Read block 1696 len 28672
10604 290/300 Flush
10605 291/300 Read block 4488 len 28672
10606 292/300 Read block 3395 len 8192
10607 293/300 Write block 3467 len 16384 data: 2 2 2 2
10608 294/300 Read block 575 len 8192
10609 295/300 Write block 3224 len 40960 data: 6 5 4 4 4 3 3 3 3 3
10610 296/300 Read block 1716 len 32768
10611 297/300 Read block 2756 len 40960
10612 298/300 Write block 1419 len 12288 data: 4 5 5
10613 299/300 Read block 3562 len 24576
10614 300/300 Write block 3743 len 12288 data: 3 3 2
10615 301/300 Write block 2856 len 20480 data: 4 6 4 4 4
10616 Main task has work to do, go find it
10617 got dsc Start(0)
10618 start 0
10619 Feb 21 06:47:41.736 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/0, method: POST, req_id: d29f964c-9832-4ea1-8c00-a09a44b3b93b, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
10620 Replay: started 0, returned:Ok(())
10621 ----------------------------------------------------------------
10622 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10623 States: Offline Active Active
10624 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10625 3718 Acked Read 1 Sent Done Done false
10626 3719 Acked Read 8 Sent Done Done false
10627 3720 Acked Write 1 Sent Done Done false
10628 3721 Acked Write 5 Sent Done Done false
10629 3722 Acked Read 10 Sent Done Done false
10630 3723 Acked Write 4 Sent Done Done false
10631 3724 Acked Write 10 Sent Done Done false
10632 3725 Acked Write 3 Sent Done Done false
10633 3726 Acked Read 3 Sent Done Done false
10634 3727 Acked Read 4 Sent Done Done false
10635 3728 Acked Read 8 Sent Done Done false
10636 3729 Acked Read 2 Sent Done Done false
10637 3730 Acked Flush 0 Sent Done Done false
10638 3731 Acked Read 9 Sent Done Done false
10639 3732 Acked Flush 0 Sent Done Done false
10640 3733 Acked Read 2 Sent Done Done false
10641 3734 Acked Flush 0 Sent Done Done false
10642 3735 Acked Write 7 Sent Done Done false
10643 3736 Acked Write 1 Sent Done Done false
10644 3737 Acked Write 6 Sent Done Done false
10645 3738 Acked Read 1 Sent Done Done false
10646 3739 Acked Read 9 Sent Done Done false
10647 3740 Acked Write 9 Sent Done Done false
10648 3741 Acked Write 2 Sent Done Done false
10649 3742 Acked Read 5 Sent Done Done false
10650 3743 Acked Read 6 Sent Done Done false
10651 3744 Acked Flush 0 Sent Done Done false
10652 3745 Acked Flush 0 Sent Done Done false
10653 3746 Acked Write 9 Sent Done Done false
10654 3747 Acked Read 5 Sent Done Done false
10655 3748 Acked Read 1 Sent Done Done false
10656 3749 Acked Write 10 Sent Done Done false
10657 3750 Acked Write 2 Sent Done Done false
10658 3751 Acked Write 7 Sent Done Done false
10659 3752 Acked Write 8 Sent Done Done false
10660 3753 Acked Write 6 Sent Done Done false
10661 3754 Acked Write 8 Sent Done Done false
10662 3755 Acked Read 7 Sent Done Done false
10663 3756 Acked Write 6 Sent Done Done false
10664 3757 Acked Read 6 Sent Done Done false
10665 3758 Acked Read 3 Sent Done Done false
10666 3759 Acked Write 7 Sent Done Done false
10667 3760 Acked Read 1 Sent Done Done false
10668 3761 Acked Read 4 Sent Done Done false
10669 3762 Acked Read 5 Sent Done Done false
10670 3763 Acked Read 1 Sent Done Done false
10671 3764 Acked Read 7 Sent Done Done false
10672 3765 Acked Read 6 Sent Done Done false
10673 3766 Acked Read 3 Sent Done Done false
10674 3767 Acked Read 8 Sent Done Done false
10675 3768 Acked Read 10 Sent Done Done false
10676 3769 Acked Read 7 Sent Done Done false
10677 3770 Acked Read 6 Sent Done Done false
10678 3771 Acked Read 8 Sent Done Done false
10679 3772 Acked Write 1 Sent Done Done false
10680 3773 Acked Read 6 Sent Done Done false
10681 3774 Acked Read 2 Sent Done Done false
10682 3775 Acked Write 6 Sent Done Done false
10683 3776 Acked Read 8 Sent Done Done false
10684 3777 Acked Write 9 Sent Done Done false
10685 3778 Acked Write 8 Sent Done Done false
10686 3779 Acked Write 3 Sent Done Done false
10687 3780 Acked Read 4 Sent Done Done false
10688 3781 Acked Read 8 Sent Done Done false
10689 3782 Acked Flush 0 Sent Done Done false
10690 3783 Acked Write 10 Sent Done Done false
10691 3784 Acked Write 3 Sent Done Done false
10692 3785 Acked Read 3 Sent Done Done false
10693 3786 Acked Write 2 Sent Done Done false
10694 3787 Acked Flush 0 Sent Done Done false
10695 3788 Acked Write 1 Sent Done Done false
10696 3789 Acked Read 9 Sent Done Done false
10697 3790 Acked Write 9 Sent Done Done false
10698 3791 Acked Write 10 Sent Done Done false
10699 3792 Acked Write 8 Sent Done Done false
10700 3793 Acked Write 2 Sent Done Done false
10701 3794 Acked Write 10 Sent Done Done false
10702 3795 Acked Read 4 Sent Done Done false
10703 3796 Acked Write 4 Sent Done Done false
10704 3797 Acked Flush 0 Sent Done Done false
10705 3798 Acked Write 3 Sent Done Done false
10706 3799 Acked Flush 0 Sent Done Done false
10707 3800 Acked Read 6 Sent Done Done false
10708 3801 Acked Flush 0 Sent Done Done false
10709 3802 Acked Flush 0 Sent Done Done false
10710 3803 Acked Write 3 Sent Done Done false
10711 3804 Acked Write 3 Sent Done Done false
10712 3805 Acked Read 7 Sent Done Done false
10713 3806 Acked Write 1 Sent Done Done false
10714 3807 Acked Read 5 Sent Done Done false
10715 3808 Acked Flush 0 Sent Done Done false
10716 3809 Acked Flush 0 Sent Done Done false
10717 3810 Acked Flush 0 Sent Done Done false
10718 3811 Acked Read 8 Sent Done Done false
10719 3812 Acked Flush 0 Sent Done Done false
10720 3813 Acked Write 8 Sent Done Done false
10721 3814 Acked Read 8 Sent Done Done false
10722 3815 Acked Write 3 Sent Done Done false
10723 3816 Acked Read 3 Sent Done Done false
10724 3817 Acked Flush 0 Sent Done Done false
10725 3818 Acked Read 2 Sent Done Done false
10726 3819 Acked Write 5 Sent Done Done false
10727 3820 Acked Read 5 Sent Done Done false
10728 3821 Acked Read 10 Sent Done Done false
10729 3822 Acked Read 5 Sent Done Done false
10730 3823 Acked Flush 0 Sent Done Done false
10731 3824 Acked Read 6 Sent Done Done false
10732 3825 Acked Write 8 Sent Done Done false
10733 3826 Acked Read 4 Sent Done Done false
10734 3827 Acked Read 5 Sent Done Done false
10735 3828 Acked Write 10 Sent Done Done false
10736 3829 Acked Write 9 Sent Done Done false
10737 3830 Acked Flush 0 Sent Done Done false
10738 3831 Acked Write 10 Sent Done Done false
10739 3832 Acked Read 6 Sent Done Done false
10740 3833 Acked Write 1 Sent Done Done false
10741 3834 Acked Write 6 Sent Done Done false
10742 3835 Acked Write 4 Sent Done Done false
10743 3836 Acked Write 4 Sent Done Done false
10744 3837 Acked Read 9 Sent Done Done false
10745 3838 Acked Read 3 Sent Done Done false
10746 3839 Acked Read 5 Sent Done Done false
10747 3840 Acked Write 2 Sent Done Done false
10748 3841 Acked Flush 0 Sent Done Done false
10749 3842 Acked Read 1 Sent Done Done false
10750 3843 Acked Read 10 Sent Done Done false
10751 3844 Acked Read 7 Sent Done Done false
10752 3845 Acked Write 10 Sent Done Done false
10753 3846 Acked Write 3 Sent Done Done false
10754 3847 Acked Read 8 Sent Done Done false
10755 3848 Acked Read 3 Sent Done Done false
10756 3849 Acked Read 9 Sent Done Done false
10757 3850 Acked Read 5 Sent Done Done false
10758 3851 Acked Read 2 Sent Done Done false
10759 3852 Acked Write 8 Sent Done Done false
10760 3853 Acked Write 5 Sent Done Done false
10761 3854 Acked Write 8 Sent Done Done false
10762 3855 Acked Read 8 Sent Done Done false
10763 3856 Acked Write 1 Sent Done Done false
10764 3857 Acked Write 10 Sent Done Done false
10765 3858 Acked Write 8 Sent Done Done false
10766 3859 Acked Write 9 Sent Done Done false
10767 3860 Acked Flush 0 Sent Done Done false
10768 3861 Acked Write 4 Sent Done Done false
10769 3862 Acked Read 4 Sent Done Done false
10770 3863 Acked Flush 0 Sent Done Done false
10771 3864 Acked Write 10 Sent Done Done false
10772 3865 Acked Flush 0 Sent Done Done false
10773 3866 Acked Write 8 Sent Done Done false
10774 3867 Acked Read 5 Sent Done Done false
10775 3868 Acked Read 2 Sent Done Done false
10776 3869 Acked Read 2 Sent Done Done false
10777 3870 Acked Write 9 Sent Done Done false
10778 3871 Acked Read 3 Sent Done Done false
10779 3872 Acked Read 10 Sent Done Done false
10780 3873 Acked Write 5 Sent Done Done false
10781 3874 Acked Write 5 Sent Done Done false
10782 3875 Acked Read 1 Sent Done Done false
10783 3876 Acked Write 3 Sent Done Done false
10784 3877 Acked Read 5 Sent Done Done false
10785 3878 Acked Write 6 Sent Done Done false
10786 3879 Acked Read 7 Sent Done Done false
10787 3880 Acked Write 9 Sent Done Done false
10788 3881 Acked Write 6 Sent Done Done false
10789 3882 Acked Read 8 Sent Done Done false
10790 3883 Acked Write 3 Sent Done Done false
10791 3884 Acked Write 1 Sent Done Done false
10792 3885 Acked Write 2 Sent Done Done false
10793 3886 Acked Write 1 Sent Done Done false
10794 3887 Acked Flush 0 Sent Done Done false
10795 3888 Acked Write 8 Sent Done Done false
10796 3889 Acked Read 9 Sent Done Done false
10797 3890 Acked Write 8 Sent Done Done false
10798 3891 Acked Write 10 Sent Done Done false
10799 3892 Acked Read 6 Sent Done Done false
10800 3893 Acked Read 1 Sent Done Done false
10801 3894 Acked Write 6 Sent Done Done false
10802 3895 Acked Read 1 Sent Done Done false
10803 3896 Acked Read 9 Sent Done Done false
10804 3897 Acked Write 8 Sent Done Done false
10805 3898 Acked Read 6 Sent Done Done false
10806 3899 Acked Write 7 Sent Done Done false
10807 3900 Acked Write 2 Sent Done Done false
10808 3901 Acked Read 4 Sent Done Done false
10809 3902 Acked Flush 0 Sent Done Done false
10810 3903 Acked Read 1 Sent Done Done false
10811 3904 Acked Write 6 Sent Done Done false
10812 3905 Acked Flush 0 Sent Done Done false
10813 3906 Acked Read 6 Sent Done Done false
10814 3907 Acked Write 2 Sent Done Done false
10815 3908 Acked Write 9 Sent Done Done false
10816 3909 Acked Write 8 Sent Done Done false
10817 3910 Acked Write 10 Sent Done Done false
10818 3911 Acked Write 3 Sent Done Done false
10819 3912 Acked Read 3 Sent Done Done false
10820 3913 Acked Write 7 Sent Done Done false
10821 3914 Acked Read 8 Sent Done Done false
10822 3915 Acked Write 1 Sent Done Done false
10823 3916 Acked Write 10 Sent Done Done false
10824 3917 Acked Read 6 Sent Done Done false
10825 3918 Acked Flush 0 Sent Done Done false
10826 3919 Acked Read 2 Sent Done Done false
10827 3920 Acked Read 10 Sent Done Done false
10828 3921 Acked Flush 0 Sent Done Done false
10829 3922 Acked Write 5 Sent Done Done false
10830 3923 Acked Read 10 Sent Done Done false
10831 3924 Acked Read 5 Sent Done Done false
10832 3925 Acked Read 7 Sent Done Done false
10833 3926 Acked Write 7 Sent Done Done false
10834 3927 Acked Read 4 Sent Done Done false
10835 3928 Acked Write 9 Sent Done Done false
10836 3929 Acked Write 3 Sent Done Done false
10837 3930 Acked Write 6 Sent Done Done false
10838 3931 Acked Write 1 Sent Done Done false
10839 3932 Acked Read 9 Sent Done Done false
10840 3933 Acked Write 3 Sent Done Done false
10841 3934 Acked Write 3 Sent Done Done false
10842 3935 Acked Read 7 Sent Done Done false
10843 3936 Acked Read 1 Sent Done Done false
10844 3937 Acked Write 1 Sent Done Done false
10845 3938 Acked Write 8 Sent Done Done false
10846 3939 Acked Write 10 Sent Done Done false
10847 3940 Acked Read 5 Sent Done Done false
10848 3941 Acked Read 10 Sent Done Done false
10849 3942 Acked Read 8 Sent Done Done false
10850 3943 Acked Read 6 Sent Done Done false
10851 3944 Acked Read 4 Sent Done Done false
10852 3945 Acked Read 3 Sent Done Done false
10853 3946 Acked Flush 0 Sent Done Done false
10854 3947 Acked Read 5 Sent Done Done false
10855 3948 Acked Write 6 Sent Done Done false
10856 3949 Acked Read 5 Sent Done Done false
10857 3950 Acked Read 10 Sent Done Done false
10858 3951 Acked Write 6 Sent Done Done false
10859 3952 Acked Flush 0 Sent Done Done false
10860 3953 Acked Read 9 Sent Done Done false
10861 3954 Acked Write 9 Sent Done Done false
10862 3955 Acked Write 9 Sent Done Done false
10863 3956 Acked Read 10 Sent Done Done false
10864 3957 Acked Read 9 Sent Done Done false
10865 3958 Acked Read 5 Sent Done Done false
10866 3959 Acked Flush 0 Sent Done Done false
10867 3960 Acked Flush 0 Sent Done Done false
10868 3961 Acked Write 2 Sent Done Done false
10869 3962 Acked Read 1 Sent Done Done false
10870 3963 Acked Flush 0 Sent Done Done false
10871 3964 Acked Write 4 Sent Done Done false
10872 3965 Acked Read 6 Sent Done Done false
10873 3966 Acked Flush 0 Sent Done Done false
10874 3967 Acked Read 9 Sent Done Done false
10875 3968 Acked Read 8 Sent Done Done false
10876 3969 Acked Read 8 Sent Done Done false
10877 3970 Acked Write 9 Sent Done Done false
10878 3971 Acked Write 8 Sent Done Done false
10879 3972 Acked Flush 0 Sent Done Done false
10880 3973 Acked Read 7 Sent Done Done false
10881 3974 Acked Read 6 Sent Done Done false
10882 3975 Acked Flush 0 Sent Done Done false
10883 3976 Acked Write 6 Sent Done Done false
10884 3977 Acked Flush 0 Sent Done Done false
10885 3978 Acked Read 4 Sent Done Done false
10886 3979 Acked Write 6 Sent Done Done false
10887 3980 Acked Write 10 Sent Done Done false
10888 3981 Acked Read 2 Sent Done Done false
10889 3982 Acked Read 4 Sent Done Done false
10890 3983 Acked Read 8 Sent Done Done false
10891 3984 Acked Read 9 Sent Done Done false
10892 3985 Acked Read 2 Sent Done Done false
10893 3986 Acked Flush 0 Sent Done Done false
10894 3987 Acked Read 6 Sent Done Done false
10895 3988 Acked Read 10 Sent Done Done false
10896 3989 Acked Read 8 Sent Done Done false
10897 3990 Acked Read 9 Sent Done Done false
10898 3991 Acked Write 1 Sent Done Done false
10899 3992 Acked Read 2 Sent Done Done false
10900 3993 Acked Read 10 Sent Done Done false
10901 3994 Acked Read 5 Sent Done Done false
10902 3995 Acked Write 6 Sent Done Done false
10903 3996 Acked Write 2 Sent Done Done false
10904 3997 Acked Read 8 Sent Done Done false
10905 3998 Acked Write 10 Sent Done Done false
10906 3999 Acked Read 7 Sent Done Done false
10907 4000 Acked Read 4 Sent Done Done false
10908 4001 Acked Write 7 Sent Done Done false
10909 4002 Acked Write 2 Sent Done Done false
10910 4003 Acked Write 8 Sent Done Done false
10911 4004 Acked Read 4 Sent Done Done false
10912 4005 Acked Write 10 Sent Done Done false
10913 4006 Acked Read 7 Sent Done Done false
10914 4007 Acked Flush 0 Sent Done Done false
10915 4008 Acked Read 7 Sent Done Done false
10916 4009 Acked Read 2 Sent Done Done false
10917 4010 Acked Write 4 Sent Done Done false
10918 4011 Acked Read 2 Sent Done Done false
10919 4012 Acked Write 10 Sent Done Done false
10920 4013 Acked Read 8 Sent Done Done false
10921 4014 Acked Read 10 Sent Done Done false
10922 4015 Acked Write 3 Sent Done Done false
10923 4016 Acked Read 6 Sent Done Done false
10924 4017 Acked Write 3 Sent Done Done false
10925 4018 Acked Write 5 Sent Done Done false
10926 STATES DS:0 DS:1 DS:2 TOTAL
10927 Sent 301 0 0 301
10928 Done 0 301 301 602
10929 Skipped 0 0 0 0
10930 Error 0 0 0 0
10931 Last Flush: Some(JobId(3717)) Some(JobId(4007)) Some(JobId(4007))
10932 Downstairs last five completed: 3717 3716 3715 3714 3713
10933 Upstairs last five completed: 4018 4017 4016 4015 4014
10934 CLIENT: Up:0 ds:301 act:2
109352025-02-21T06:47:42.238ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
109362025-02-21T06:47:42.872ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
109372025-02-21T06:47:43.372ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
109382025-02-21T06:47:43.874ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
109392025-02-21T06:47:44.980ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
10940 ----------------------------------------------------------------
10941 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10942 States: Offline Active Active
10943 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10944 3718 Acked Read 1 Sent Done Done false
10945 3719 Acked Read 8 Sent Done Done false
10946 3720 Acked Write 1 Sent Done Done false
10947 3721 Acked Write 5 Sent Done Done false
10948 3722 Acked Read 10 Sent Done Done false
10949 3723 Acked Write 4 Sent Done Done false
10950 3724 Acked Write 10 Sent Done Done false
10951 3725 Acked Write 3 Sent Done Done false
10952 3726 Acked Read 3 Sent Done Done false
10953 3727 Acked Read 4 Sent Done Done false
10954 3728 Acked Read 8 Sent Done Done false
10955 3729 Acked Read 2 Sent Done Done false
10956 3730 Acked Flush 0 Sent Done Done false
10957 3731 Acked Read 9 Sent Done Done false
10958 3732 Acked Flush 0 Sent Done Done false
10959 3733 Acked Read 2 Sent Done Done false
10960 3734 Acked Flush 0 Sent Done Done false
10961 3735 Acked Write 7 Sent Done Done false
10962 3736 Acked Write 1 Sent Done Done false
10963 3737 Acked Write 6 Sent Done Done false
10964 3738 Acked Read 1 Sent Done Done false
10965 3739 Acked Read 9 Sent Done Done false
10966 3740 Acked Write 9 Sent Done Done false
10967 3741 Acked Write 2 Sent Done Done false
10968 3742 Acked Read 5 Sent Done Done false
10969 3743 Acked Read 6 Sent Done Done false
10970 3744 Acked Flush 0 Sent Done Done false
10971 3745 Acked Flush 0 Sent Done Done false
10972 3746 Acked Write 9 Sent Done Done false
10973 3747 Acked Read 5 Sent Done Done false
10974 3748 Acked Read 1 Sent Done Done false
10975 3749 Acked Write 10 Sent Done Done false
10976 3750 Acked Write 2 Sent Done Done false
10977 3751 Acked Write 7 Sent Done Done false
10978 3752 Acked Write 8 Sent Done Done false
10979 3753 Acked Write 6 Sent Done Done false
10980 3754 Acked Write 8 Sent Done Done false
10981 3755 Acked Read 7 Sent Done Done false
10982 3756 Acked Write 6 Sent Done Done false
10983 3757 Acked Read 6 Sent Done Done false
10984 3758 Acked Read 3 Sent Done Done false
10985 3759 Acked Write 7 Sent Done Done false
10986 3760 Acked Read 1 Sent Done Done false
10987 3761 Acked Read 4 Sent Done Done false
10988 3762 Acked Read 5 Sent Done Done false
10989 3763 Acked Read 1 Sent Done Done false
10990 3764 Acked Read 7 Sent Done Done false
10991 3765 Acked Read 6 Sent Done Done false
10992 3766 Acked Read 3 Sent Done Done false
10993 3767 Acked Read 8 Sent Done Done false
10994 3768 Acked Read 10 Sent Done Done false
10995 3769 Acked Read 7 Sent Done Done false
10996 3770 Acked Read 6 Sent Done Done false
10997 3771 Acked Read 8 Sent Done Done false
10998 3772 Acked Write 1 Sent Done Done false
10999 3773 Acked Read 6 Sent Done Done false
11000 3774 Acked Read 2 Sent Done Done false
11001 3775 Acked Write 6 Sent Done Done false
11002 3776 Acked Read 8 Sent Done Done false
11003 3777 Acked Write 9 Sent Done Done false
11004 3778 Acked Write 8 Sent Done Done false
11005 3779 Acked Write 3 Sent Done Done false
11006 3780 Acked Read 4 Sent Done Done false
11007 3781 Acked Read 8 Sent Done Done false
11008 3782 Acked Flush 0 Sent Done Done false
11009 3783 Acked Write 10 Sent Done Done false
11010 3784 Acked Write 3 Sent Done Done false
11011 3785 Acked Read 3 Sent Done Done false
11012 3786 Acked Write 2 Sent Done Done false
11013 3787 Acked Flush 0 Sent Done Done false
11014 3788 Acked Write 1 Sent Done Done false
11015 3789 Acked Read 9 Sent Done Done false
11016 3790 Acked Write 9 Sent Done Done false
11017 3791 Acked Write 10 Sent Done Done false
11018 3792 Acked Write 8 Sent Done Done false
11019 3793 Acked Write 2 Sent Done Done false
11020 3794 Acked Write 10 Sent Done Done false
11021 3795 Acked Read 4 Sent Done Done false
11022 3796 Acked Write 4 Sent Done Done false
11023 3797 Acked Flush 0 Sent Done Done false
11024 3798 Acked Write 3 Sent Done Done false
11025 3799 Acked Flush 0 Sent Done Done false
11026 3800 Acked Read 6 Sent Done Done false
11027 3801 Acked Flush 0 Sent Done Done false
11028 3802 Acked Flush 0 Sent Done Done false
11029 3803 Acked Write 3 Sent Done Done false
11030 3804 Acked Write 3 Sent Done Done false
11031 3805 Acked Read 7 Sent Done Done false
11032 3806 Acked Write 1 Sent Done Done false
11033 3807 Acked Read 5 Sent Done Done false
11034 3808 Acked Flush 0 Sent Done Done false
11035 3809 Acked Flush 0 Sent Done Done false
11036 3810 Acked Flush 0 Sent Done Done false
11037 3811 Acked Read 8 Sent Done Done false
11038 3812 Acked Flush 0 Sent Done Done false
11039 3813 Acked Write 8 Sent Done Done false
11040 3814 Acked Read 8 Sent Done Done false
11041 3815 Acked Write 3 Sent Done Done false
11042 3816 Acked Read 3 Sent Done Done false
11043 3817 Acked Flush 0 Sent Done Done false
11044 3818 Acked Read 2 Sent Done Done false
11045 3819 Acked Write 5 Sent Done Done false
11046 3820 Acked Read 5 Sent Done Done false
11047 3821 Acked Read 10 Sent Done Done false
11048 3822 Acked Read 5 Sent Done Done false
11049 3823 Acked Flush 0 Sent Done Done false
11050 3824 Acked Read 6 Sent Done Done false
11051 3825 Acked Write 8 Sent Done Done false
11052 3826 Acked Read 4 Sent Done Done false
11053 3827 Acked Read 5 Sent Done Done false
11054 3828 Acked Write 10 Sent Done Done false
11055 3829 Acked Write 9 Sent Done Done false
11056 3830 Acked Flush 0 Sent Done Done false
11057 3831 Acked Write 10 Sent Done Done false
11058 3832 Acked Read 6 Sent Done Done false
11059 3833 Acked Write 1 Sent Done Done false
11060 3834 Acked Write 6 Sent Done Done false
11061 3835 Acked Write 4 Sent Done Done false
11062 3836 Acked Write 4 Sent Done Done false
11063 3837 Acked Read 9 Sent Done Done false
11064 3838 Acked Read 3 Sent Done Done false
11065 3839 Acked Read 5 Sent Done Done false
11066 3840 Acked Write 2 Sent Done Done false
11067 3841 Acked Flush 0 Sent Done Done false
11068 3842 Acked Read 1 Sent Done Done false
11069 3843 Acked Read 10 Sent Done Done false
11070 3844 Acked Read 7 Sent Done Done false
11071 3845 Acked Write 10 Sent Done Done false
11072 3846 Acked Write 3 Sent Done Done false
11073 3847 Acked Read 8 Sent Done Done false
11074 3848 Acked Read 3 Sent Done Done false
11075 3849 Acked Read 9 Sent Done Done false
11076 3850 Acked Read 5 Sent Done Done false
11077 3851 Acked Read 2 Sent Done Done false
11078 3852 Acked Write 8 Sent Done Done false
11079 3853 Acked Write 5 Sent Done Done false
11080 3854 Acked Write 8 Sent Done Done false
11081 3855 Acked Read 8 Sent Done Done false
11082 3856 Acked Write 1 Sent Done Done false
11083 3857 Acked Write 10 Sent Done Done false
11084 3858 Acked Write 8 Sent Done Done false
11085 3859 Acked Write 9 Sent Done Done false
11086 3860 Acked Flush 0 Sent Done Done false
11087 3861 Acked Write 4 Sent Done Done false
11088 3862 Acked Read 4 Sent Done Done false
11089 3863 Acked Flush 0 Sent Done Done false
11090 3864 Acked Write 10 Sent Done Done false
11091 3865 Acked Flush 0 Sent Done Done false
11092 3866 Acked Write 8 Sent Done Done false
11093 3867 Acked Read 5 Sent Done Done false
11094 3868 Acked Read 2 Sent Done Done false
11095 3869 Acked Read 2 Sent Done Done false
11096 3870 Acked Write 9 Sent Done Done false
11097 3871 Acked Read 3 Sent Done Done false
11098 3872 Acked Read 10 Sent Done Done false
11099 3873 Acked Write 5 Sent Done Done false
11100 3874 Acked Write 5 Sent Done Done false
11101 3875 Acked Read 1 Sent Done Done false
11102 3876 Acked Write 3 Sent Done Done false
11103 3877 Acked Read 5 Sent Done Done false
11104 3878 Acked Write 6 Sent Done Done false
11105 3879 Acked Read 7 Sent Done Done false
11106 3880 Acked Write 9 Sent Done Done false
11107 3881 Acked Write 6 Sent Done Done false
11108 3882 Acked Read 8 Sent Done Done false
11109 3883 Acked Write 3 Sent Done Done false
11110 3884 Acked Write 1 Sent Done Done false
11111 3885 Acked Write 2 Sent Done Done false
11112 3886 Acked Write 1 Sent Done Done false
11113 3887 Acked Flush 0 Sent Done Done false
11114 3888 Acked Write 8 Sent Done Done false
11115 3889 Acked Read 9 Sent Done Done false
11116 3890 Acked Write 8 Sent Done Done false
11117 3891 Acked Write 10 Sent Done Done false
11118 3892 Acked Read 6 Sent Done Done false
11119 3893 Acked Read 1 Sent Done Done false
11120 3894 Acked Write 6 Sent Done Done false
11121 3895 Acked Read 1 Sent Done Done false
11122 3896 Acked Read 9 Sent Done Done false
11123 3897 Acked Write 8 Sent Done Done false
11124 3898 Acked Read 6 Sent Done Done false
11125 3899 Acked Write 7 Sent Done Done false
11126 3900 Acked Write 2 Sent Done Done false
11127 3901 Acked Read 4 Sent Done Done false
11128 3902 Acked Flush 0 Sent Done Done false
11129 3903 Acked Read 1 Sent Done Done false
11130 3904 Acked Write 6 Sent Done Done false
11131 3905 Acked Flush 0 Sent Done Done false
11132 3906 Acked Read 6 Sent Done Done false
11133 3907 Acked Write 2 Sent Done Done false
11134 3908 Acked Write 9 Sent Done Done false
11135 3909 Acked Write 8 Sent Done Done false
11136 3910 Acked Write 10 Sent Done Done false
11137 3911 Acked Write 3 Sent Done Done false
11138 3912 Acked Read 3 Sent Done Done false
11139 3913 Acked Write 7 Sent Done Done false
11140 3914 Acked Read 8 Sent Done Done false
11141 3915 Acked Write 1 Sent Done Done false
11142 3916 Acked Write 10 Sent Done Done false
11143 3917 Acked Read 6 Sent Done Done false
11144 3918 Acked Flush 0 Sent Done Done false
11145 3919 Acked Read 2 Sent Done Done false
11146 3920 Acked Read 10 Sent Done Done false
11147 3921 Acked Flush 0 Sent Done Done false
11148 3922 Acked Write 5 Sent Done Done false
11149 3923 Acked Read 10 Sent Done Done false
11150 3924 Acked Read 5 Sent Done Done false
11151 3925 Acked Read 7 Sent Done Done false
11152 3926 Acked Write 7 Sent Done Done false
11153 3927 Acked Read 4 Sent Done Done false
11154 3928 Acked Write 9 Sent Done Done false
11155 3929 Acked Write 3 Sent Done Done false
11156 3930 Acked Write 6 Sent Done Done false
11157 3931 Acked Write 1 Sent Done Done false
11158 3932 Acked Read 9 Sent Done Done false
11159 3933 Acked Write 3 Sent Done Done false
11160 3934 Acked Write 3 Sent Done Done false
11161 3935 Acked Read 7 Sent Done Done false
11162 3936 Acked Read 1 Sent Done Done false
11163 3937 Acked Write 1 Sent Done Done false
11164 3938 Acked Write 8 Sent Done Done false
11165 3939 Acked Write 10 Sent Done Done false
11166 3940 Acked Read 5 Sent Done Done false
11167 3941 Acked Read 10 Sent Done Done false
11168 3942 Acked Read 8 Sent Done Done false
11169 3943 Acked Read 6 Sent Done Done false
11170 3944 Acked Read 4 Sent Done Done false
11171 3945 Acked Read 3 Sent Done Done false
11172 3946 Acked Flush 0 Sent Done Done false
11173 3947 Acked Read 5 Sent Done Done false
11174 3948 Acked Write 6 Sent Done Done false
11175 3949 Acked Read 5 Sent Done Done false
11176 3950 Acked Read 10 Sent Done Done false
11177 3951 Acked Write 6 Sent Done Done false
11178 3952 Acked Flush 0 Sent Done Done false
11179 3953 Acked Read 9 Sent Done Done false
11180 3954 Acked Write 9 Sent Done Done false
11181 3955 Acked Write 9 Sent Done Done false
11182 3956 Acked Read 10 Sent Done Done false
11183 3957 Acked Read 9 Sent Done Done false
11184 3958 Acked Read 5 Sent Done Done false
11185 3959 Acked Flush 0 Sent Done Done false
11186 3960 Acked Flush 0 Sent Done Done false
11187 3961 Acked Write 2 Sent Done Done false
11188 3962 Acked Read 1 Sent Done Done false
11189 3963 Acked Flush 0 Sent Done Done false
11190 3964 Acked Write 4 Sent Done Done false
11191 3965 Acked Read 6 Sent Done Done false
11192 3966 Acked Flush 0 Sent Done Done false
11193 3967 Acked Read 9 Sent Done Done false
11194 3968 Acked Read 8 Sent Done Done false
11195 3969 Acked Read 8 Sent Done Done false
11196 3970 Acked Write 9 Sent Done Done false
11197 3971 Acked Write 8 Sent Done Done false
11198 3972 Acked Flush 0 Sent Done Done false
11199 3973 Acked Read 7 Sent Done Done false
11200 3974 Acked Read 6 Sent Done Done false
11201 3975 Acked Flush 0 Sent Done Done false
11202 3976 Acked Write 6 Sent Done Done false
11203 3977 Acked Flush 0 Sent Done Done false
11204 3978 Acked Read 4 Sent Done Done false
11205 3979 Acked Write 6 Sent Done Done false
11206 3980 Acked Write 10 Sent Done Done false
11207 3981 Acked Read 2 Sent Done Done false
11208 3982 Acked Read 4 Sent Done Done false
11209 3983 Acked Read 8 Sent Done Done false
11210 3984 Acked Read 9 Sent Done Done false
11211 3985 Acked Read 2 Sent Done Done false
11212 3986 Acked Flush 0 Sent Done Done false
11213 3987 Acked Read 6 Sent Done Done false
11214 3988 Acked Read 10 Sent Done Done false
11215 3989 Acked Read 8 Sent Done Done false
11216 3990 Acked Read 9 Sent Done Done false
11217 3991 Acked Write 1 Sent Done Done false
11218 3992 Acked Read 2 Sent Done Done false
11219 3993 Acked Read 10 Sent Done Done false
11220 3994 Acked Read 5 Sent Done Done false
11221 3995 Acked Write 6 Sent Done Done false
11222 3996 Acked Write 2 Sent Done Done false
11223 3997 Acked Read 8 Sent Done Done false
11224 3998 Acked Write 10 Sent Done Done false
11225 3999 Acked Read 7 Sent Done Done false
11226 4000 Acked Read 4 Sent Done Done false
11227 4001 Acked Write 7 Sent Done Done false
11228 4002 Acked Write 2 Sent Done Done false
11229 4003 Acked Write 8 Sent Done Done false
11230 4004 Acked Read 4 Sent Done Done false
11231 4005 Acked Write 10 Sent Done Done false
11232 4006 Acked Read 7 Sent Done Done false
11233 4007 Acked Flush 0 Sent Done Done false
11234 4008 Acked Read 7 Sent Done Done false
11235 4009 Acked Read 2 Sent Done Done false
11236 4010 Acked Write 4 Sent Done Done false
11237 4011 Acked Read 2 Sent Done Done false
11238 4012 Acked Write 10 Sent Done Done false
11239 4013 Acked Read 8 Sent Done Done false
11240 4014 Acked Read 10 Sent Done Done false
11241 4015 Acked Write 3 Sent Done Done false
11242 4016 Acked Read 6 Sent Done Done false
11243 4017 Acked Write 3 Sent Done Done false
11244 4018 Acked Write 5 Sent Done Done false
11245 STATES DS:0 DS:1 DS:2 TOTAL
11246 Sent 301 0 0 301
11247 Done 0 301 301 602
11248 Skipped 0 0 0 0
11249 Error 0 0 0 0
11250 Last Flush: Some(JobId(3717)) Some(JobId(4007)) Some(JobId(4007))
11251 Downstairs last five completed: 3717 3716 3715 3714 3713
11252 Upstairs last five completed: 4018 4017 4016 4015 4014
11253 CLIENT: Up:0 ds:301 act:2
112542025-02-21T06:47:46.877ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
112552025-02-21T06:47:47.495ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
112562025-02-21T06:47:47.495ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
112572025-02-21T06:47:47.495ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
112582025-02-21T06:47:47.495ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
112592025-02-21T06:47:47.496ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
112602025-02-21T06:47:47.997ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
112612025-02-21T06:47:48.880ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
112622025-02-21T06:47:49.381ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11263 ----------------------------------------------------------------
11264 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11265 States: Offline Active Active
11266 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11267 3718 Acked Read 1 Sent Done Done false
11268 3719 Acked Read 8 Sent Done Done false
11269 3720 Acked Write 1 Sent Done Done false
11270 3721 Acked Write 5 Sent Done Done false
11271 3722 Acked Read 10 Sent Done Done false
11272 3723 Acked Write 4 Sent Done Done false
11273 3724 Acked Write 10 Sent Done Done false
11274 3725 Acked Write 3 Sent Done Done false
11275 3726 Acked Read 3 Sent Done Done false
11276 3727 Acked Read 4 Sent Done Done false
11277 3728 Acked Read 8 Sent Done Done false
11278 3729 Acked Read 2 Sent Done Done false
11279 3730 Acked Flush 0 Sent Done Done false
11280 3731 Acked Read 9 Sent Done Done false
11281 3732 Acked Flush 0 Sent Done Done false
11282 3733 Acked Read 2 Sent Done Done false
11283 3734 Acked Flush 0 Sent Done Done false
11284 3735 Acked Write 7 Sent Done Done false
11285 3736 Acked Write 1 Sent Done Done false
11286 3737 Acked Write 6 Sent Done Done false
11287 3738 Acked Read 1 Sent Done Done false
11288 3739 Acked Read 9 Sent Done Done false
11289 3740 Acked Write 9 Sent Done Done false
11290 3741 Acked Write 2 Sent Done Done false
11291 3742 Acked Read 5 Sent Done Done false
11292 3743 Acked Read 6 Sent Done Done false
11293 3744 Acked Flush 0 Sent Done Done false
11294 3745 Acked Flush 0 Sent Done Done false
11295 3746 Acked Write 9 Sent Done Done false
11296 3747 Acked Read 5 Sent Done Done false
11297 3748 Acked Read 1 Sent Done Done false
11298 3749 Acked Write 10 Sent Done Done false
11299 3750 Acked Write 2 Sent Done Done false
11300 3751 Acked Write 7 Sent Done Done false
11301 3752 Acked Write 8 Sent Done Done false
11302 3753 Acked Write 6 Sent Done Done false
11303 3754 Acked Write 8 Sent Done Done false
11304 3755 Acked Read 7 Sent Done Done false
11305 3756 Acked Write 6 Sent Done Done false
11306 3757 Acked Read 6 Sent Done Done false
11307 3758 Acked Read 3 Sent Done Done false
11308 3759 Acked Write 7 Sent Done Done false
11309 3760 Acked Read 1 Sent Done Done false
11310 3761 Acked Read 4 Sent Done Done false
11311 3762 Acked Read 5 Sent Done Done false
11312 3763 Acked Read 1 Sent Done Done false
11313 3764 Acked Read 7 Sent Done Done false
11314 3765 Acked Read 6 Sent Done Done false
11315 3766 Acked Read 3 Sent Done Done false
11316 3767 Acked Read 8 Sent Done Done false
11317 3768 Acked Read 10 Sent Done Done false
11318 3769 Acked Read 7 Sent Done Done false
11319 3770 Acked Read 6 Sent Done Done false
11320 3771 Acked Read 8 Sent Done Done false
11321 3772 Acked Write 1 Sent Done Done false
11322 3773 Acked Read 6 Sent Done Done false
11323 3774 Acked Read 2 Sent Done Done false
11324 3775 Acked Write 6 Sent Done Done false
11325 3776 Acked Read 8 Sent Done Done false
11326 3777 Acked Write 9 Sent Done Done false
11327 3778 Acked Write 8 Sent Done Done false
11328 3779 Acked Write 3 Sent Done Done false
11329 3780 Acked Read 4 Sent Done Done false
11330 3781 Acked Read 8 Sent Done Done false
11331 3782 Acked Flush 0 Sent Done Done false
11332 3783 Acked Write 10 Sent Done Done false
11333 3784 Acked Write 3 Sent Done Done false
11334 3785 Acked Read 3 Sent Done Done false
11335 3786 Acked Write 2 Sent Done Done false
11336 3787 Acked Flush 0 Sent Done Done false
11337 3788 Acked Write 1 Sent Done Done false
11338 3789 Acked Read 9 Sent Done Done false
11339 3790 Acked Write 9 Sent Done Done false
11340 3791 Acked Write 10 Sent Done Done false
11341 3792 Acked Write 8 Sent Done Done false
11342 3793 Acked Write 2 Sent Done Done false
11343 3794 Acked Write 10 Sent Done Done false
11344 3795 Acked Read 4 Sent Done Done false
11345 3796 Acked Write 4 Sent Done Done false
11346 3797 Acked Flush 0 Sent Done Done false
11347 3798 Acked Write 3 Sent Done Done false
11348 3799 Acked Flush 0 Sent Done Done false
11349 3800 Acked Read 6 Sent Done Done false
11350 3801 Acked Flush 0 Sent Done Done false
11351 3802 Acked Flush 0 Sent Done Done false
11352 3803 Acked Write 3 Sent Done Done false
11353 3804 Acked Write 3 Sent Done Done false
11354 3805 Acked Read 7 Sent Done Done false
11355 3806 Acked Write 1 Sent Done Done false
11356 3807 Acked Read 5 Sent Done Done false
11357 3808 Acked Flush 0 Sent Done Done false
11358 3809 Acked Flush 0 Sent Done Done false
11359 3810 Acked Flush 0 Sent Done Done false
11360 3811 Acked Read 8 Sent Done Done false
11361 3812 Acked Flush 0 Sent Done Done false
11362 3813 Acked Write 8 Sent Done Done false
11363 3814 Acked Read 8 Sent Done Done false
11364 3815 Acked Write 3 Sent Done Done false
11365 3816 Acked Read 3 Sent Done Done false
11366 3817 Acked Flush 0 Sent Done Done false
11367 3818 Acked Read 2 Sent Done Done false
11368 3819 Acked Write 5 Sent Done Done false
11369 3820 Acked Read 5 Sent Done Done false
11370 3821 Acked Read 10 Sent Done Done false
11371 3822 Acked Read 5 Sent Done Done false
11372 3823 Acked Flush 0 Sent Done Done false
11373 3824 Acked Read 6 Sent Done Done false
11374 3825 Acked Write 8 Sent Done Done false
11375 3826 Acked Read 4 Sent Done Done false
11376 3827 Acked Read 5 Sent Done Done false
11377 3828 Acked Write 10 Sent Done Done false
11378 3829 Acked Write 9 Sent Done Done false
11379 3830 Acked Flush 0 Sent Done Done false
11380 3831 Acked Write 10 Sent Done Done false
11381 3832 Acked Read 6 Sent Done Done false
11382 3833 Acked Write 1 Sent Done Done false
11383 3834 Acked Write 6 Sent Done Done false
11384 3835 Acked Write 4 Sent Done Done false
11385 3836 Acked Write 4 Sent Done Done false
11386 3837 Acked Read 9 Sent Done Done false
11387 3838 Acked Read 3 Sent Done Done false
11388 3839 Acked Read 5 Sent Done Done false
11389 3840 Acked Write 2 Sent Done Done false
11390 3841 Acked Flush 0 Sent Done Done false
11391 3842 Acked Read 1 Sent Done Done false
11392 3843 Acked Read 10 Sent Done Done false
11393 3844 Acked Read 7 Sent Done Done false
11394 3845 Acked Write 10 Sent Done Done false
11395 3846 Acked Write 3 Sent Done Done false
11396 3847 Acked Read 8 Sent Done Done false
11397 3848 Acked Read 3 Sent Done Done false
11398 3849 Acked Read 9 Sent Done Done false
11399 3850 Acked Read 5 Sent Done Done false
11400 3851 Acked Read 2 Sent Done Done false
11401 3852 Acked Write 8 Sent Done Done false
11402 3853 Acked Write 5 Sent Done Done false
11403 3854 Acked Write 8 Sent Done Done false
11404 3855 Acked Read 8 Sent Done Done false
11405 3856 Acked Write 1 Sent Done Done false
11406 3857 Acked Write 10 Sent Done Done false
11407 3858 Acked Write 8 Sent Done Done false
11408 3859 Acked Write 9 Sent Done Done false
11409 3860 Acked Flush 0 Sent Done Done false
11410 3861 Acked Write 4 Sent Done Done false
11411 3862 Acked Read 4 Sent Done Done false
11412 3863 Acked Flush 0 Sent Done Done false
11413 3864 Acked Write 10 Sent Done Done false
11414 3865 Acked Flush 0 Sent Done Done false
11415 3866 Acked Write 8 Sent Done Done false
11416 3867 Acked Read 5 Sent Done Done false
11417 3868 Acked Read 2 Sent Done Done false
11418 3869 Acked Read 2 Sent Done Done false
11419 3870 Acked Write 9 Sent Done Done false
11420 3871 Acked Read 3 Sent Done Done false
11421 3872 Acked Read 10 Sent Done Done false
11422 3873 Acked Write 5 Sent Done Done false
11423 3874 Acked Write 5 Sent Done Done false
11424 3875 Acked Read 1 Sent Done Done false
11425 3876 Acked Write 3 Sent Done Done false
11426 3877 Acked Read 5 Sent Done Done false
11427 3878 Acked Write 6 Sent Done Done false
11428 3879 Acked Read 7 Sent Done Done false
11429 3880 Acked Write 9 Sent Done Done false
11430 3881 Acked Write 6 Sent Done Done false
11431 3882 Acked Read 8 Sent Done Done false
11432 3883 Acked Write 3 Sent Done Done false
11433 3884 Acked Write 1 Sent Done Done false
11434 3885 Acked Write 2 Sent Done Done false
11435 3886 Acked Write 1 Sent Done Done false
11436 3887 Acked Flush 0 Sent Done Done false
11437 3888 Acked Write 8 Sent Done Done false
11438 3889 Acked Read 9 Sent Done Done false
11439 3890 Acked Write 8 Sent Done Done false
11440 3891 Acked Write 10 Sent Done Done false
11441 3892 Acked Read 6 Sent Done Done false
11442 3893 Acked Read 1 Sent Done Done false
11443 3894 Acked Write 6 Sent Done Done false
11444 3895 Acked Read 1 Sent Done Done false
11445 3896 Acked Read 9 Sent Done Done false
11446 3897 Acked Write 8 Sent Done Done false
11447 3898 Acked Read 6 Sent Done Done false
11448 3899 Acked Write 7 Sent Done Done false
11449 3900 Acked Write 2 Sent Done Done false
11450 3901 Acked Read 4 Sent Done Done false
11451 3902 Acked Flush 0 Sent Done Done false
11452 3903 Acked Read 1 Sent Done Done false
11453 3904 Acked Write 6 Sent Done Done false
11454 3905 Acked Flush 0 Sent Done Done false
11455 3906 Acked Read 6 Sent Done Done false
11456 3907 Acked Write 2 Sent Done Done false
11457 3908 Acked Write 9 Sent Done Done false
11458 3909 Acked Write 8 Sent Done Done false
11459 3910 Acked Write 10 Sent Done Done false
11460 3911 Acked Write 3 Sent Done Done false
11461 3912 Acked Read 3 Sent Done Done false
11462 3913 Acked Write 7 Sent Done Done false
11463 3914 Acked Read 8 Sent Done Done false
11464 3915 Acked Write 1 Sent Done Done false
11465 3916 Acked Write 10 Sent Done Done false
11466 3917 Acked Read 6 Sent Done Done false
11467 3918 Acked Flush 0 Sent Done Done false
11468 3919 Acked Read 2 Sent Done Done false
11469 3920 Acked Read 10 Sent Done Done false
11470 3921 Acked Flush 0 Sent Done Done false
11471 3922 Acked Write 5 Sent Done Done false
11472 3923 Acked Read 10 Sent Done Done false
11473 3924 Acked Read 5 Sent Done Done false
11474 3925 Acked Read 7 Sent Done Done false
11475 3926 Acked Write 7 Sent Done Done false
11476 3927 Acked Read 4 Sent Done Done false
11477 3928 Acked Write 9 Sent Done Done false
11478 3929 Acked Write 3 Sent Done Done false
11479 3930 Acked Write 6 Sent Done Done false
11480 3931 Acked Write 1 Sent Done Done false
11481 3932 Acked Read 9 Sent Done Done false
11482 3933 Acked Write 3 Sent Done Done false
11483 3934 Acked Write 3 Sent Done Done false
11484 3935 Acked Read 7 Sent Done Done false
11485 3936 Acked Read 1 Sent Done Done false
11486 3937 Acked Write 1 Sent Done Done false
11487 3938 Acked Write 8 Sent Done Done false
11488 3939 Acked Write 10 Sent Done Done false
11489 3940 Acked Read 5 Sent Done Done false
11490 3941 Acked Read 10 Sent Done Done false
11491 3942 Acked Read 8 Sent Done Done false
11492 3943 Acked Read 6 Sent Done Done false
11493 3944 Acked Read 4 Sent Done Done false
11494 3945 Acked Read 3 Sent Done Done false
11495 3946 Acked Flush 0 Sent Done Done false
11496 3947 Acked Read 5 Sent Done Done false
11497 3948 Acked Write 6 Sent Done Done false
11498 3949 Acked Read 5 Sent Done Done false
11499 3950 Acked Read 10 Sent Done Done false
11500 3951 Acked Write 6 Sent Done Done false
11501 3952 Acked Flush 0 Sent Done Done false
11502 3953 Acked Read 9 Sent Done Done false
11503 3954 Acked Write 9 Sent Done Done false
11504 3955 Acked Write 9 Sent Done Done false
11505 3956 Acked Read 10 Sent Done Done false
11506 3957 Acked Read 9 Sent Done Done false
11507 3958 Acked Read 5 Sent Done Done false
11508 3959 Acked Flush 0 Sent Done Done false
11509 3960 Acked Flush 0 Sent Done Done false
11510 3961 Acked Write 2 Sent Done Done false
11511 3962 Acked Read 1 Sent Done Done false
11512 3963 Acked Flush 0 Sent Done Done false
11513 3964 Acked Write 4 Sent Done Done false
11514 3965 Acked Read 6 Sent Done Done false
11515 3966 Acked Flush 0 Sent Done Done false
11516 3967 Acked Read 9 Sent Done Done false
11517 3968 Acked Read 8 Sent Done Done false
11518 3969 Acked Read 8 Sent Done Done false
11519 3970 Acked Write 9 Sent Done Done false
11520 3971 Acked Write 8 Sent Done Done false
11521 3972 Acked Flush 0 Sent Done Done false
11522 3973 Acked Read 7 Sent Done Done false
11523 3974 Acked Read 6 Sent Done Done false
11524 3975 Acked Flush 0 Sent Done Done false
11525 3976 Acked Write 6 Sent Done Done false
11526 3977 Acked Flush 0 Sent Done Done false
11527 3978 Acked Read 4 Sent Done Done false
11528 3979 Acked Write 6 Sent Done Done false
11529 3980 Acked Write 10 Sent Done Done false
11530 3981 Acked Read 2 Sent Done Done false
11531 3982 Acked Read 4 Sent Done Done false
11532 3983 Acked Read 8 Sent Done Done false
11533 3984 Acked Read 9 Sent Done Done false
11534 3985 Acked Read 2 Sent Done Done false
11535 3986 Acked Flush 0 Sent Done Done false
11536 3987 Acked Read 6 Sent Done Done false
11537 3988 Acked Read 10 Sent Done Done false
11538 3989 Acked Read 8 Sent Done Done false
11539 3990 Acked Read 9 Sent Done Done false
11540 3991 Acked Write 1 Sent Done Done false
11541 3992 Acked Read 2 Sent Done Done false
11542 3993 Acked Read 10 Sent Done Done false
11543 3994 Acked Read 5 Sent Done Done false
11544 3995 Acked Write 6 Sent Done Done false
11545 3996 Acked Write 2 Sent Done Done false
11546 3997 Acked Read 8 Sent Done Done false
11547 3998 Acked Write 10 Sent Done Done false
11548 3999 Acked Read 7 Sent Done Done false
11549 4000 Acked Read 4 Sent Done Done false
11550 4001 Acked Write 7 Sent Done Done false
11551 4002 Acked Write 2 Sent Done Done false
11552 4003 Acked Write 8 Sent Done Done false
11553 4004 Acked Read 4 Sent Done Done false
11554 4005 Acked Write 10 Sent Done Done false
11555 4006 Acked Read 7 Sent Done Done false
11556 4007 Acked Flush 0 Sent Done Done false
11557 4008 Acked Read 7 Sent Done Done false
11558 4009 Acked Read 2 Sent Done Done false
11559 4010 Acked Write 4 Sent Done Done false
11560 4011 Acked Read 2 Sent Done Done false
11561 4012 Acked Write 10 Sent Done Done false
11562 4013 Acked Read 8 Sent Done Done false
11563 4014 Acked Read 10 Sent Done Done false
11564 4015 Acked Write 3 Sent Done Done false
11565 4016 Acked Read 6 Sent Done Done false
11566 4017 Acked Write 3 Sent Done Done false
11567 4018 Acked Write 5 Sent Done Done false
11568 STATES DS:0 DS:1 DS:2 TOTAL
11569 Sent 301 0 0 301
11570 Done 0 301 301 602
11571 Skipped 0 0 0 0
11572 Error 0 0 0 0
11573 Last Flush: Some(JobId(3717)) Some(JobId(4007)) Some(JobId(4007))
11574 Downstairs last five completed: 3717 3716 3715 3714 3713
11575 Upstairs last five completed: 4018 4017 4016 4015 4014
11576 CLIENT: Up:0 ds:301 act:2
115772025-02-21T06:47:50.242ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
115782025-02-21T06:47:50.883ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
115792025-02-21T06:47:51.885ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
115802025-02-21T06:47:52.886ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11581 [8810] got start action
11582 [8810] I am going to restart
11583 Starting downstairs at port 8810
11584 Make output file at "/tmp/dsc/downstairs-8810.txt"
11585 [8810][0] reports Starting
11586 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1216)
11587 [8810][0] reports Running
11588 ----------------------------------------------------------------
11589 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11590 States: Offline Active Active
11591 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11592 3718 Acked Read 1 Sent Done Done false
11593 3719 Acked Read 8 Sent Done Done false
11594 3720 Acked Write 1 Sent Done Done false
11595 3721 Acked Write 5 Sent Done Done false
11596 3722 Acked Read 10 Sent Done Done false
11597 3723 Acked Write 4 Sent Done Done false
11598 3724 Acked Write 10 Sent Done Done false
11599 3725 Acked Write 3 Sent Done Done false
11600 3726 Acked Read 3 Sent Done Done false
11601 3727 Acked Read 4 Sent Done Done false
11602 3728 Acked Read 8 Sent Done Done false
11603 3729 Acked Read 2 Sent Done Done false
11604 3730 Acked Flush 0 Sent Done Done false
11605 3731 Acked Read 9 Sent Done Done false
11606 3732 Acked Flush 0 Sent Done Done false
11607 3733 Acked Read 2 Sent Done Done false
11608 3734 Acked Flush 0 Sent Done Done false
11609 3735 Acked Write 7 Sent Done Done false
11610 3736 Acked Write 1 Sent Done Done false
11611 3737 Acked Write 6 Sent Done Done false
11612 3738 Acked Read 1 Sent Done Done false
11613 3739 Acked Read 9 Sent Done Done false
11614 3740 Acked Write 9 Sent Done Done false
11615 3741 Acked Write 2 Sent Done Done false
11616 3742 Acked Read 5 Sent Done Done false
11617 3743 Acked Read 6 Sent Done Done false
11618 3744 Acked Flush 0 Sent Done Done false
11619 3745 Acked Flush 0 Sent Done Done false
11620 3746 Acked Write 9 Sent Done Done false
11621 3747 Acked Read 5 Sent Done Done false
11622 3748 Acked Read 1 Sent Done Done false
11623 3749 Acked Write 10 Sent Done Done false
11624 3750 Acked Write 2 Sent Done Done false
11625 3751 Acked Write 7 Sent Done Done false
11626 3752 Acked Write 8 Sent Done Done false
11627 3753 Acked Write 6 Sent Done Done false
11628 3754 Acked Write 8 Sent Done Done false
11629 3755 Acked Read 7 Sent Done Done false
11630 3756 Acked Write 6 Sent Done Done false
11631 3757 Acked Read 6 Sent Done Done false
11632 3758 Acked Read 3 Sent Done Done false
11633 3759 Acked Write 7 Sent Done Done false
11634 3760 Acked Read 1 Sent Done Done false
11635 3761 Acked Read 4 Sent Done Done false
11636 3762 Acked Read 5 Sent Done Done false
11637 3763 Acked Read 1 Sent Done Done false
11638 3764 Acked Read 7 Sent Done Done false
11639 3765 Acked Read 6 Sent Done Done false
11640 3766 Acked Read 3 Sent Done Done false
11641 3767 Acked Read 8 Sent Done Done false
11642 3768 Acked Read 10 Sent Done Done false
11643 3769 Acked Read 7 Sent Done Done false
11644 3770 Acked Read 6 Sent Done Done false
11645 3771 Acked Read 8 Sent Done Done false
11646 3772 Acked Write 1 Sent Done Done false
11647 3773 Acked Read 6 Sent Done Done false
11648 3774 Acked Read 2 Sent Done Done false
11649 3775 Acked Write 6 Sent Done Done false
11650 3776 Acked Read 8 Sent Done Done false
11651 3777 Acked Write 9 Sent Done Done false
11652 3778 Acked Write 8 Sent Done Done false
11653 3779 Acked Write 3 Sent Done Done false
11654 3780 Acked Read 4 Sent Done Done false
11655 3781 Acked Read 8 Sent Done Done false
11656 3782 Acked Flush 0 Sent Done Done false
11657 3783 Acked Write 10 Sent Done Done false
11658 3784 Acked Write 3 Sent Done Done false
11659 3785 Acked Read 3 Sent Done Done false
11660 3786 Acked Write 2 Sent Done Done false
11661 3787 Acked Flush 0 Sent Done Done false
11662 3788 Acked Write 1 Sent Done Done false
11663 3789 Acked Read 9 Sent Done Done false
11664 3790 Acked Write 9 Sent Done Done false
11665 3791 Acked Write 10 Sent Done Done false
11666 3792 Acked Write 8 Sent Done Done false
11667 3793 Acked Write 2 Sent Done Done false
11668 3794 Acked Write 10 Sent Done Done false
11669 3795 Acked Read 4 Sent Done Done false
11670 3796 Acked Write 4 Sent Done Done false
11671 3797 Acked Flush 0 Sent Done Done false
11672 3798 Acked Write 3 Sent Done Done false
11673 3799 Acked Flush 0 Sent Done Done false
11674 3800 Acked Read 6 Sent Done Done false
11675 3801 Acked Flush 0 Sent Done Done false
11676 3802 Acked Flush 0 Sent Done Done false
11677 3803 Acked Write 3 Sent Done Done false
11678 3804 Acked Write 3 Sent Done Done false
11679 3805 Acked Read 7 Sent Done Done false
11680 3806 Acked Write 1 Sent Done Done false
11681 3807 Acked Read 5 Sent Done Done false
11682 3808 Acked Flush 0 Sent Done Done false
11683 3809 Acked Flush 0 Sent Done Done false
11684 3810 Acked Flush 0 Sent Done Done false
11685 3811 Acked Read 8 Sent Done Done false
11686 3812 Acked Flush 0 Sent Done Done false
11687 3813 Acked Write 8 Sent Done Done false
11688 3814 Acked Read 8 Sent Done Done false
11689 3815 Acked Write 3 Sent Done Done false
11690 3816 Acked Read 3 Sent Done Done false
11691 3817 Acked Flush 0 Sent Done Done false
11692 3818 Acked Read 2 Sent Done Done false
11693 3819 Acked Write 5 Sent Done Done false
11694 3820 Acked Read 5 Sent Done Done false
11695 3821 Acked Read 10 Sent Done Done false
11696 3822 Acked Read 5 Sent Done Done false
11697 3823 Acked Flush 0 Sent Done Done false
11698 3824 Acked Read 6 Sent Done Done false
11699 3825 Acked Write 8 Sent Done Done false
11700 3826 Acked Read 4 Sent Done Done false
11701 3827 Acked Read 5 Sent Done Done false
11702 3828 Acked Write 10 Sent Done Done false
11703 3829 Acked Write 9 Sent Done Done false
11704 3830 Acked Flush 0 Sent Done Done false
11705 3831 Acked Write 10 Sent Done Done false
11706 3832 Acked Read 6 Sent Done Done false
11707 3833 Acked Write 1 Sent Done Done false
11708 3834 Acked Write 6 Sent Done Done false
11709 3835 Acked Write 4 Sent Done Done false
11710 3836 Acked Write 4 Sent Done Done false
11711 3837 Acked Read 9 Sent Done Done false
11712 3838 Acked Read 3 Sent Done Done false
11713 3839 Acked Read 5 Sent Done Done false
11714 3840 Acked Write 2 Sent Done Done false
11715 3841 Acked Flush 0 Sent Done Done false
11716 3842 Acked Read 1 Sent Done Done false
11717 3843 Acked Read 10 Sent Done Done false
11718 3844 Acked Read 7 Sent Done Done false
11719 3845 Acked Write 10 Sent Done Done false
11720 3846 Acked Write 3 Sent Done Done false
11721 3847 Acked Read 8 Sent Done Done false
11722 3848 Acked Read 3 Sent Done Done false
11723 3849 Acked Read 9 Sent Done Done false
11724 3850 Acked Read 5 Sent Done Done false
11725 3851 Acked Read 2 Sent Done Done false
11726 3852 Acked Write 8 Sent Done Done false
11727 3853 Acked Write 5 Sent Done Done false
11728 3854 Acked Write 8 Sent Done Done false
11729 3855 Acked Read 8 Sent Done Done false
11730 3856 Acked Write 1 Sent Done Done false
11731 3857 Acked Write 10 Sent Done Done false
11732 3858 Acked Write 8 Sent Done Done false
11733 3859 Acked Write 9 Sent Done Done false
11734 3860 Acked Flush 0 Sent Done Done false
11735 3861 Acked Write 4 Sent Done Done false
11736 3862 Acked Read 4 Sent Done Done false
11737 3863 Acked Flush 0 Sent Done Done false
11738 3864 Acked Write 10 Sent Done Done false
11739 3865 Acked Flush 0 Sent Done Done false
11740 3866 Acked Write 8 Sent Done Done false
11741 3867 Acked Read 5 Sent Done Done false
11742 3868 Acked Read 2 Sent Done Done false
11743 3869 Acked Read 2 Sent Done Done false
11744 3870 Acked Write 9 Sent Done Done false
11745 3871 Acked Read 3 Sent Done Done false
11746 3872 Acked Read 10 Sent Done Done false
11747 3873 Acked Write 5 Sent Done Done false
11748 3874 Acked Write 5 Sent Done Done false
11749 3875 Acked Read 1 Sent Done Done false
11750 3876 Acked Write 3 Sent Done Done false
11751 3877 Acked Read 5 Sent Done Done false
11752 3878 Acked Write 6 Sent Done Done false
11753 3879 Acked Read 7 Sent Done Done false
11754 3880 Acked Write 9 Sent Done Done false
11755 3881 Acked Write 6 Sent Done Done false
11756 3882 Acked Read 8 Sent Done Done false
11757 3883 Acked Write 3 Sent Done Done false
11758 3884 Acked Write 1 Sent Done Done false
11759 3885 Acked Write 2 Sent Done Done false
11760 3886 Acked Write 1 Sent Done Done false
11761 3887 Acked Flush 0 Sent Done Done false
11762 3888 Acked Write 8 Sent Done Done false
11763 3889 Acked Read 9 Sent Done Done false
11764 3890 Acked Write 8 Sent Done Done false
11765 3891 Acked Write 10 Sent Done Done false
11766 3892 Acked Read 6 Sent Done Done false
11767 3893 Acked Read 1 Sent Done Done false
11768 3894 Acked Write 6 Sent Done Done false
11769 3895 Acked Read 1 Sent Done Done false
11770 3896 Acked Read 9 Sent Done Done false
11771 3897 Acked Write 8 Sent Done Done false
11772 3898 Acked Read 6 Sent Done Done false
11773 3899 Acked Write 7 Sent Done Done false
11774 3900 Acked Write 2 Sent Done Done false
11775 3901 Acked Read 4 Sent Done Done false
11776 3902 Acked Flush 0 Sent Done Done false
11777 3903 Acked Read 1 Sent Done Done false
11778 3904 Acked Write 6 Sent Done Done false
11779 3905 Acked Flush 0 Sent Done Done false
11780 3906 Acked Read 6 Sent Done Done false
11781 3907 Acked Write 2 Sent Done Done false
11782 3908 Acked Write 9 Sent Done Done false
11783 3909 Acked Write 8 Sent Done Done false
11784 3910 Acked Write 10 Sent Done Done false
11785 3911 Acked Write 3 Sent Done Done false
11786 3912 Acked Read 3 Sent Done Done false
11787 3913 Acked Write 7 Sent Done Done false
11788 3914 Acked Read 8 Sent Done Done false
11789 3915 Acked Write 1 Sent Done Done false
11790 3916 Acked Write 10 Sent Done Done false
11791 3917 Acked Read 6 Sent Done Done false
11792 3918 Acked Flush 0 Sent Done Done false
11793 3919 Acked Read 2 Sent Done Done false
11794 3920 Acked Read 10 Sent Done Done false
11795 3921 Acked Flush 0 Sent Done Done false
11796 3922 Acked Write 5 Sent Done Done false
11797 3923 Acked Read 10 Sent Done Done false
11798 3924 Acked Read 5 Sent Done Done false
11799 3925 Acked Read 7 Sent Done Done false
11800 3926 Acked Write 7 Sent Done Done false
11801 3927 Acked Read 4 Sent Done Done false
11802 3928 Acked Write 9 Sent Done Done false
11803 3929 Acked Write 3 Sent Done Done false
11804 3930 Acked Write 6 Sent Done Done false
11805 3931 Acked Write 1 Sent Done Done false
11806 3932 Acked Read 9 Sent Done Done false
11807 3933 Acked Write 3 Sent Done Done false
11808 3934 Acked Write 3 Sent Done Done false
11809 3935 Acked Read 7 Sent Done Done false
11810 3936 Acked Read 1 Sent Done Done false
11811 3937 Acked Write 1 Sent Done Done false
11812 3938 Acked Write 8 Sent Done Done false
11813 3939 Acked Write 10 Sent Done Done false
11814 3940 Acked Read 5 Sent Done Done false
11815 3941 Acked Read 10 Sent Done Done false
11816 3942 Acked Read 8 Sent Done Done false
11817 3943 Acked Read 6 Sent Done Done false
11818 3944 Acked Read 4 Sent Done Done false
11819 3945 Acked Read 3 Sent Done Done false
11820 3946 Acked Flush 0 Sent Done Done false
11821 3947 Acked Read 5 Sent Done Done false
11822 3948 Acked Write 6 Sent Done Done false
11823 3949 Acked Read 5 Sent Done Done false
11824 3950 Acked Read 10 Sent Done Done false
11825 3951 Acked Write 6 Sent Done Done false
11826 3952 Acked Flush 0 Sent Done Done false
11827 3953 Acked Read 9 Sent Done Done false
11828 3954 Acked Write 9 Sent Done Done false
11829 3955 Acked Write 9 Sent Done Done false
11830 3956 Acked Read 10 Sent Done Done false
11831 3957 Acked Read 9 Sent Done Done false
11832 3958 Acked Read 5 Sent Done Done false
11833 3959 Acked Flush 0 Sent Done Done false
11834 3960 Acked Flush 0 Sent Done Done false
11835 3961 Acked Write 2 Sent Done Done false
11836 3962 Acked Read 1 Sent Done Done false
11837 3963 Acked Flush 0 Sent Done Done false
11838 3964 Acked Write 4 Sent Done Done false
11839 3965 Acked Read 6 Sent Done Done false
11840 3966 Acked Flush 0 Sent Done Done false
11841 3967 Acked Read 9 Sent Done Done false
11842 3968 Acked Read 8 Sent Done Done false
11843 3969 Acked Read 8 Sent Done Done false
11844 3970 Acked Write 9 Sent Done Done false
11845 3971 Acked Write 8 Sent Done Done false
11846 3972 Acked Flush 0 Sent Done Done false
11847 3973 Acked Read 7 Sent Done Done false
11848 3974 Acked Read 6 Sent Done Done false
11849 3975 Acked Flush 0 Sent Done Done false
11850 3976 Acked Write 6 Sent Done Done false
11851 3977 Acked Flush 0 Sent Done Done false
11852 3978 Acked Read 4 Sent Done Done false
11853 3979 Acked Write 6 Sent Done Done false
11854 3980 Acked Write 10 Sent Done Done false
11855 3981 Acked Read 2 Sent Done Done false
11856 3982 Acked Read 4 Sent Done Done false
11857 3983 Acked Read 8 Sent Done Done false
11858 3984 Acked Read 9 Sent Done Done false
11859 3985 Acked Read 2 Sent Done Done false
11860 3986 Acked Flush 0 Sent Done Done false
11861 3987 Acked Read 6 Sent Done Done false
11862 3988 Acked Read 10 Sent Done Done false
11863 3989 Acked Read 8 Sent Done Done false
11864 3990 Acked Read 9 Sent Done Done false
11865 3991 Acked Write 1 Sent Done Done false
11866 3992 Acked Read 2 Sent Done Done false
11867 3993 Acked Read 10 Sent Done Done false
11868 3994 Acked Read 5 Sent Done Done false
11869 3995 Acked Write 6 Sent Done Done false
11870 3996 Acked Write 2 Sent Done Done false
11871 3997 Acked Read 8 Sent Done Done false
11872 3998 Acked Write 10 Sent Done Done false
11873 3999 Acked Read 7 Sent Done Done false
11874 4000 Acked Read 4 Sent Done Done false
11875 4001 Acked Write 7 Sent Done Done false
11876 4002 Acked Write 2 Sent Done Done false
11877 4003 Acked Write 8 Sent Done Done false
11878 4004 Acked Read 4 Sent Done Done false
11879 4005 Acked Write 10 Sent Done Done false
11880 4006 Acked Read 7 Sent Done Done false
11881 4007 Acked Flush 0 Sent Done Done false
11882 4008 Acked Read 7 Sent Done Done false
11883 4009 Acked Read 2 Sent Done Done false
11884 4010 Acked Write 4 Sent Done Done false
11885 4011 Acked Read 2 Sent Done Done false
11886 4012 Acked Write 10 Sent Done Done false
11887 4013 Acked Read 8 Sent Done Done false
11888 4014 Acked Read 10 Sent Done Done false
11889 4015 Acked Write 3 Sent Done Done false
11890 4016 Acked Read 6 Sent Done Done false
11891 4017 Acked Write 3 Sent Done Done false
11892 4018 Acked Write 5 Sent Done Done false
11893 STATES DS:0 DS:1 DS:2 TOTAL
11894 Sent 301 0 0 301
11895 Done 0 301 301 602
11896 Skipped 0 0 0 0
11897 Error 0 0 0 0
11898 Last Flush: Some(JobId(3717)) Some(JobId(4007)) Some(JobId(4007))
11899 Downstairs last five completed: 3717 3716 3715 3714 3713
11900 Upstairs last five completed: 4018 4017 4016 4015 4014
11901 CLIENT: Up:0 ds:301 act:2
119022025-02-21T06:47:54.245ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119032025-02-21T06:47:54.982ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119042025-02-21T06:47:55.891ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119052025-02-21T06:47:56.892ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119062025-02-21T06:47:57.497ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119072025-02-21T06:47:57.497ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38726) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119082025-02-21T06:47:57.498ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119092025-02-21T06:47:57.499ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119102025-02-21T06:47:57.499ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119112025-02-21T06:47:57.499ZINFOcrucible: send last flush ID to this DS: Some(JobId(3717)) = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119122025-02-21T06:47:57.499ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119132025-02-21T06:47:57.499ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(3717)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119142025-02-21T06:47:57.499ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(3717)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11915 ----------------------------------------------------------------
11916 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:11
11917 States: Active Active Active
11918 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11919 4008 Acked Read 7 Done Done Done true
11920 4009 Acked Read 2 Done Done Done true
11921 4010 Acked Write 4 Done Done Done true
11922 4011 Acked Read 2 Done Done Done true
11923 4012 Acked Write 10 Done Done Done true
11924 4013 Acked Read 8 Done Done Done true
11925 4014 Acked Read 10 Done Done Done true
11926 4015 Acked Write 3 Done Done Done true
11927 4016 Acked Read 6 Done Done Done true
11928 4017 Acked Write 3 Done Done Done true
11929 4018 Acked Write 5 Done Done Done true
11930 STATES DS:0 DS:1 DS:2 TOTAL
11931 Sent 0 0 0 0
11932 Done 11 11 11 33
11933 Skipped 0 0 0 0
11934 Error 0 0 0 0
11935 Last Flush: Some(JobId(4007)) Some(JobId(4007)) Some(JobId(4007))
11936 Downstairs last five completed: 4007 4006 4005 4004 4003
11937 Upstairs last five completed: 4018 4017 4016 4015 4014
11938 CLIENT: Up:0 ds:11 act:3
11939 ----------------------------------------------------------------
11940 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11941 Downstairs last five completed: 4019 4018 4017 4016 4015
11942 Upstairs last five completed: 4019 4018 4017 4016 4015
11943 CLIENT: Up:0 ds:0 act:3
11944 Replay: All jobs finished
11945 Main task has work to do, go find it
11946 got dsc Stop(0)
11947 stop 0
11948 [8810] Got stop action so:false kr:false
11949 Feb 21 06:48:01.748 INFO request completed, latency_us: 351, response_code: 204, uri: /stop/cid/0, method: POST, req_id: d8819926-a6d5-47ec-bbe8-84885d3f91bb, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
119502025-02-21T06:48:01.749ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119512025-02-21T06:48:01.749ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
119522025-02-21T06:48:01.749ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
11953 Feb 21 06:48:01.749 INFO request completed, latency_us: 167, response_code: 200{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:48:01.749695832Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
11954 , uri: /state/cid/0, method: GET, req_id: 8e1eb6bd-efd9-455f-add9-693afd505a9c, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
11955 [8810] Exited with: ExitStatus(unix_wait_status(9))
11956 [8810][0] reports Exit
11957 Feb 21 06:48:05.752 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5c958349-2763-4531-a62b-843dcacda9b7, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
11958 001/300 Read block 3230 len 8192
11959 002/300 Flush
11960 003/300 Flush
11961 004/300 Read block 3900 len 12288
11962 005/300 Read block 3658 len 8192
11963 006/300 Read block 4750 len 24576
11964 007/300 Read block 473 len 4096
11965 008/300 Read block 3705 len 8192
11966 009/300 Read block 849 len 4096
11967 010/300 Read block 97 len 28672
11968 011/300 Flush
11969 012/300 Read block 4619 len 32768
11970 013/300 Read block 803 len 4096
11971 014/300 Write block 2939 len 32768 data: 3 2 2 2 2 2 2 2
11972 015/300 Read block 1581 len 32768
11973 016/300 Write block 3113 len 12288 data: 2 3 3
11974 017/300 Write block 2655 len 16384 data: 3 3 3 4
11975 018/300 Read block 1185 len 12288
11976 019/300 Write block 1603 len 20480 data: 3 3 3 3 5
11977 020/300 Flush
11978 021/300 Write block 1214 len 20480 data: 4 3 4 4 4
11979 022/300 Write block 1439 len 40960 data: 6 6 5 4 5 5 4 3 4 4
11980 023/300 Write block 1492 len 28672 data: 4 4 4 3 3 3 5
11981 024/300 Read block 2780 len 16384
11982 025/300 Write block 2925 len 32768 data: 2 2 2 3 3 3 3 3
11983 026/300 Read block 1588 len 40960
11984 027/300 Write block 1556 len 8192 data: 3 2
11985 028/300 Read block 2150 len 24576
11986 029/300 Write block 1601 len 24576 data: 3 3 4 4 4 4
11987 030/300 Write block 1213 len 36864 data: 5 5 4 5 5 5 4 4 4
11988 031/300 Read block 4528 len 20480
11989 032/300 Read block 3123 len 24576
11990 033/300 Read block 367 len 4096
11991 034/300 Read block 1564 len 40960
11992 035/300 Write block 407 len 4096 data: 2
11993 036/300 Read block 484 len 24576
11994 037/300 Flush
11995 038/300 Flush
11996 039/300 Read block 378 len 32768
11997 040/300 Read block 2490 len 36864
11998 041/300 Read block 806 len 32768
11999 042/300 Read block 3801 len 8192
12000 043/300 Read block 2202 len 32768
12001 044/300 Read block 3555 len 8192
12002 045/300 Read block 4092 len 8192
12003 046/300 Read block 484 len 24576
12004 047/300 Read block 3680 len 12288
12005 048/300 Write block 944 len 28672 data: 2 3 3 3 3 3 3
12006 049/300 Flush
12007 050/300 Read block 1461 len 36864
12008 051/300 Read block 848 len 20480
12009 052/300 Read block 2597 len 32768
12010 053/300 Write block 3403 len 32768 data: 2 2 2 2 2 3 3 3
12011 054/300 Write block 4188 len 24576 data: 4 4 4 3 3 3
12012 055/300 Read block 491 len 20480
12013 056/300 Write block 1928 len 32768 data: 2 2 2 2 2 2 2 2
12014 057/300 Write block 1870 len 8192 data: 3 3
12015 058/300 Read block 956 len 40960
12016 059/300 Write block 2154 len 12288 data: 2 2 2
12017 060/300 Flush
12018 061/300 Read block 957 len 36864
12019 062/300 Read block 1222 len 40960
12020 063/300 Read block 172 len 24576
12021 064/300 Read block 1448 len 28672
12022 065/300 Write block 1343 len 36864 data: 2 2 2 3 3 3 2 2 2
12023 066/300 Write block 2303 len 28672 data: 3 3 3 3 4 3 3
12024 067/300 Read block 2421 len 20480
12025 068/300 Flush
12026 069/300 Write block 4165 len 12288 data: 2 2 2
12027 070/300 Write block 4622 len 36864 data: 4 4 5 6 4 4 4 3 3
12028 071/300 Read block 1133 len 36864
12029 072/300 Read block 807 len 8192
12030 073/300 Read block 2246 len 4096
12031 074/300 Read block 1687 len 32768
12032 075/300 Write block 211 len 24576 data: 4 3 3 2 3 4
12033 076/300 Read block 915 len 28672
12034 077/300 Flush
12035 078/300 Read block 132 len 12288
12036 079/300 Read block 3662 len 28672
12037 080/300 Write block 4112 len 4096 data: 3
12038 081/300 Read block 3695 len 8192
12039 082/300 Read block 2616 len 12288
12040 083/300 Write block 4820 len 16384 data: 5 5 5 5
12041 084/300 Write block 1109 len 12288 data: 4 3 3
12042 085/300 Write block 4843 len 8192 data: 2 2
12043 086/300 Read block 1773 len 32768
12044 087/300 Read block 1956 len 8192
12045 088/300 Write block 2335 len 12288 data: 2 2 2
12046 089/300 Read block 3693 len 12288
12047 090/300 Flush
12048 091/300 Read block 4088 len 36864
12049 092/300 Read block 4184 len 36864
12050 093/300 Read block 320 len 24576
12051 094/300 Read block 2419 len 4096
12052 095/300 Read block 915 len 4096
12053 096/300 Read block 1992 len 12288
12054 097/300 Write block 4565 len 8192 data: 3 4
12055 098/300 Read block 808 len 28672
12056 099/300 Read block 1079 len 40960
12057 100/300 Write block 862 len 32768 data: 3 2 2 2 2 2 2 2
12058 101/300 Read block 3031 len 28672
12059 102/300 Write block 4181 len 36864 data: 2 3 3 3 4 3 4 5 5
12060 103/300 Read block 3448 len 32768
12061 104/300 Read block 1589 len 32768
12062 105/300 Flush
12063 106/300 Write block 972 len 20480 data: 4 4 4 4 5
12064 107/300 Read block 2336 len 24576
12065 108/300 Read block 3404 len 28672
12066 109/300 Write block 3007 len 4096 data: 4
12067 110/300 Read block 3251 len 36864
12068 111/300 Write block 3319 len 4096 data: 3
12069 112/300 Read block 3210 len 28672
12070 113/300 Write block 1018 len 40960 data: 6 6 6 6 4 2 2 4 5 5
12071 114/300 Read block 1885 len 36864
12072 115/300 Read block 3784 len 8192
12073 116/300 Read block 2489 len 8192
12074 117/300 Write block 3010 len 8192 data: 5 4
12075 118/300 Read block 4950 len 32768
12076 119/300 Read block 258 len 4096
12077 120/300 Write block 872 len 8192 data: 3 3
12078 121/300 Write block 943 len 28672 data: 2 3 4 4 4 4 4
12079 122/300 Flush
12080 123/300 Read block 1654 len 36864
12081 124/300 Read block 4423 len 16384
12082 125/300 Read block 2880 len 24576
12083 126/300 Write block 132 len 20480 data: 2 2 2 2 2
12084 127/300 Write block 4570 len 8192 data: 4 3
12085 128/300 Write block 1622 len 24576 data: 3 4 4 5 5 5
12086 129/300 Write block 120 len 20480 data: 4 4 4 3 3
12087 130/300 Flush
12088 131/300 Write block 2678 len 40960 data: 4 5 5 5 4 4 4 4 5 3
12089 132/300 Read block 1746 len 20480
12090 133/300 Write block 1096 len 28672 data: 2 2 2 2 2 2 3
12091 134/300 Read block 3213 len 20480
12092 135/300 Flush
12093 136/300 Write block 30 len 40960 data: 3 2 2 2 2 2 2 2 2 2
12094 137/300 Write block 2758 len 32768 data: 4 4 4 4 5 4 4 4
12095 138/300 Read block 2460 len 40960
12096 139/300 Write block 1119 len 40960 data: 4 3 3 3 5 5 6 7 7 7
12097 140/300 Read block 2243 len 16384
12098 141/300 Read block 301 len 20480
12099 142/300 Read block 1030 len 20480
12100 143/300 Read block 4369 len 20480
12101 144/300 Read block 4925 len 20480
12102 145/300 Read block 1968 len 40960
12103 146/300 Read block 1334 len 28672
12104 147/300 Read block 3921 len 16384
12105 148/300 Write block 4120 len 8192 data: 4 6
12106 149/300 Write block 1238 len 36864 data: 3 3 3 3 3 3 3 3 3
12107 150/300 Write block 3467 len 8192 data: 3 3
12108 151/300 Read block 3326 len 8192
12109 152/300 Write block 3635 len 16384 data: 4 5 5 4
12110 153/300 Flush
12111 154/300 Write block 3974 len 32768 data: 2 3 5 6 5 4 3 3
12112 155/300 Read block 3620 len 12288
12113 156/300 Write block 1797 len 36864 data: 2 3 3 3 3 3 3 3 3
12114 157/300 Read block 3068 len 36864
12115 158/300 Read block 2605 len 20480
12116 159/300 Read block 3590 len 28672
12117 160/300 Read block 3843 len 28672
12118 161/300 Write block 82 len 16384 data: 3 2 2 2
12119 162/300 Write block 443 len 12288 data: 4 4 4
12120 163/300 Read block 3639 len 36864
12121 164/300 Read block 2062 len 12288
12122 165/300 Flush
12123 166/300 Flush
12124 167/300 Write block 4728 len 24576 data: 3 3 4 3 3 3
12125 168/300 Read block 4367 len 12288
12126 169/300 Write block 2181 len 12288 data: 2 2 2
12127 170/300 Write block 4906 len 24576 data: 4 3 2 2 2 2
12128 171/300 Write block 2544 len 16384 data: 2 2 3 2
12129 172/300 Read block 2895 len 16384
12130 173/300 Read block 4441 len 12288
12131 174/300 Read block 1533 len 12288
12132 175/300 Read block 2875 len 20480
12133 176/300 Flush
12134 177/300 Read block 2736 len 40960
12135 178/300 Write block 2411 len 4096 data: 2
12136 179/300 Flush
12137 180/300 Write block 4077 len 24576 data: 5 4 3 2 2 2
12138 181/300 Write block 3555 len 20480 data: 4 2 2 2 2
12139 182/300 Write block 4927 len 8192 data: 2 3
12140 183/300 Write block 1203 len 12288 data: 3 3 3
12141 184/300 Flush
12142 185/300 Write block 698 len 12288 data: 3 3 3
12143 186/300 Read block 2519 len 8192
12144 187/300 Flush
12145 188/300 Write block 2882 len 12288 data: 3 4 4
12146 189/300 Write block 3667 len 20480 data: 3 3 3 3 3
12147 190/300 Write block 3816 len 8192 data: 7 7
12148 191/300 Flush
12149 192/300 Write block 3835 len 28672 data: 5 4 3 3 3 4 5
12150 193/300 Write block 3605 len 28672 data: 3 4 6 5 4 5 4
12151 194/300 Read block 2916 len 4096
12152 195/300 Read block 4366 len 20480
12153 196/300 Read block 810 len 4096
12154 197/300 Flush
12155 198/300 Write block 4669 len 36864 data: 4 4 4 3 3 3 4 4 3
12156 199/300 Read block 784 len 40960
12157 200/300 Read block 4805 len 32768
12158 201/300 Write block 1170 len 4096 data: 2
12159 202/300 Flush
12160 203/300 Write block 4916 len 28672 data: 7 8 6 6 5 3 3
12161 204/300 Read block 3076 len 16384
12162 205/300 Write block 1310 len 40960 data: 4 4 4 4 4 4 3 3 3 3
12163 206/300 Read block 2759 len 20480
12164 207/300 Read block 4015 len 24576
12165 208/300 Write block 4814 len 20480 data: 5 6 6 6 6
12166 209/300 Read block 1132 len 36864
12167 210/300 Write block 4319 len 20480 data: 3 3 2 2 2
12168 211/300 Flush
12169 212/300 Write block 2413 len 8192 data: 3 3
12170 213/300 Read block 4001 len 4096
12171 214/300 Read block 1686 len 32768
12172 215/300 Read block 1661 len 12288
12173 216/300 Write block 1061 len 20480 data: 5 2 3 3 3
12174 217/300 Read block 2087 len 24576
12175 218/300 Flush
12176 219/300 Read block 687 len 16384
12177 220/300 Read block 1449 len 8192
12178 221/300 Write block 248 len 28672 data: 2 2 2 2 2 4 4
12179 222/300 Read block 1603 len 20480
12180 223/300 Write block 3907 len 24576 data: 3 3 3 4 4 5
12181 224/300 Read block 1030 len 4096
12182 225/300 Read block 2899 len 28672
12183 226/300 Read block 1827 len 36864
12184 227/300 Read block 2114 len 28672
12185 228/300 Read block 2080 len 32768
12186 229/300 Write block 3865 len 40960 data: 4 3 5 4 3 3 3 4 6 5
12187 230/300 Read block 4610 len 24576
12188 231/300 Write block 2297 len 4096 data: 2
12189 232/300 Write block 3459 len 8192 data: 2 2
12190 233/300 Flush
12191 234/300 Read block 2261 len 8192
12192 235/300 Write block 4897 len 36864 data: 5 5 5 5 5 4 3 2 3
12193 236/300 Write block 3906 len 36864 data: 3 4 4 4 5 5 6 4 4
12194 237/300 Read block 608 len 32768
12195 238/300 Flush
12196 239/300 Read block 1493 len 12288
12197 240/300 Write block 3100 len 36864 data: 3 3 3 4 3 3 2 2 2
12198 241/300 Write block 3710 len 36864 data: 3 2 3 3 3 3 3 2 2
12199 242/300 Write block 2570 len 12288 data: 3 4 6
12200 243/300 Read block 2481 len 32768
12201 244/300 Write block 1740 len 36864 data: 2 3 3 3 3 3 3 4 5
12202 245/300 Read block 3686 len 8192
12203 246/300 Flush
12204 247/300 Flush
12205 248/300 Read block 1321 len 36864
12206 249/300 Write block 327 len 16384 data: 2 2 3 3
12207 250/300 Flush
12208 251/300 Read block 2873 len 12288
12209 252/300 Read block 4664 len 36864
12210 253/300 Read block 2437 len 24576
12211 254/300 Write block 3762 len 24576 data: 2 2 3 3 4 4
12212 255/300 Write block 1169 len 32768 data: 2 3 2 2 3 4 4 5
12213 256/300 Write block 1382 len 4096 data: 3
12214 257/300 Write block 4036 len 12288 data: 4 4 4
12215 258/300 Read block 3374 len 20480
12216 259/300 Read block 3144 len 20480
12217 260/300 Read block 4695 len 32768
12218 261/300 Write block 2604 len 16384 data: 5 5 4 3
12219 262/300 Write block 4514 len 12288 data: 2 2 2
12220 263/300 Read block 559 len 12288
12221 264/300 Write block 2507 len 4096 data: 4
12222 265/300 Write block 3678 len 4096 data: 4
12223 266/300 Write block 4917 len 36864 data: 9 7 7 6 4 4 4 4 4
12224 267/300 Write block 2495 len 8192 data: 3 3
12225 268/300 Read block 1089 len 32768
12226 269/300 Write block 2871 len 24576 data: 5 5 4 4 4 5
12227 270/300 Write block 1166 len 36864 data: 2 2 2 3 4 3 3 4 5
12228 271/300 Write block 4569 len 20480 data: 5 5 4 3 4
12229 272/300 Write block 3767 len 40960 data: 5 3 3 5 5 6 6 5 4 3
12230 273/300 Write block 4083 len 16384 data: 2 2 2 2
12231 274/300 Write block 4535 len 8192 data: 2 2
12232 275/300 Write block 3571 len 12288 data: 4 4 3
12233 276/300 Write block 4536 len 8192 data: 3 2
12234 277/300 Read block 3714 len 12288
12235 278/300 Write block 4426 len 12288 data: 2 2 2
12236 279/300 Write block 2162 len 24576 data: 4 5 4 3 3 3
12237 280/300 Read block 4015 len 36864
12238 281/300 Write block 4975 len 4096 data: 4
12239 282/300 Write block 670 len 36864 data: 6 6 6 5 5 4 4 4 3
12240 283/300 Read block 148 len 32768
12241 284/300 Read block 3580 len 24576
12242 285/300 Read block 2570 len 24576
12243 286/300 Write block 2424 len 12288 data: 6 5 5
12244 287/300 Read block 2193 len 4096
12245 288/300 Read block 4457 len 16384
12246 289/300 Flush
12247 290/300 Read block 4677 len 4096
12248 291/300 Flush
12249 292/300 Read block 4392 len 12288
12250 293/300 Flush
12251 294/300 Read block 4267 len 4096
12252 295/300 Write block 2259 len 28672 data: 3 3 4 3 3 3 3
12253 296/300 Flush
12254 297/300 Read block 3878 len 16384
12255 298/300 Read block 780 len 28672
12256 299/300 Read block 1045 len 20480
12257 300/300 Read block 4211 len 8192
12258 301/300 Read block 305 len 24576
12259 Main task has work to do, go find it
12260 got dsc Start(0)
12261 start 0
12262 Feb 21 06:48:05.990 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/0, method: POST, req_id: 9f9116eb-4389-4e8e-b9f0-1df6d2c23056, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
12263 Replay: started 0, returned:Ok(())
12264 ----------------------------------------------------------------
12265 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12266 States: Offline Active Active
12267 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12268 4020 Acked Read 2 Sent Done Done false
12269 4021 Acked Flush 0 Sent Done Done false
12270 4022 Acked Flush 0 Sent Done Done false
12271 4023 Acked Read 3 Sent Done Done false
12272 4024 Acked Read 2 Sent Done Done false
12273 4025 Acked Read 6 Sent Done Done false
12274 4026 Acked Read 1 Sent Done Done false
12275 4027 Acked Read 2 Sent Done Done false
12276 4028 Acked Read 1 Sent Done Done false
12277 4029 Acked Read 7 Sent Done Done false
12278 4030 Acked Flush 0 Sent Done Done false
12279 4031 Acked Read 8 Sent Done Done false
12280 4032 Acked Read 1 Sent Done Done false
12281 4033 Acked Write 8 Sent Done Done false
12282 4034 Acked Read 8 Sent Done Done false
12283 4035 Acked Write 3 Sent Done Done false
12284 4036 Acked Write 4 Sent Done Done false
12285 4037 Acked Read 3 Sent Done Done false
12286 4038 Acked Write 5 Sent Done Done false
12287 4039 Acked Flush 0 Sent Done Done false
12288 4040 Acked Write 5 Sent Done Done false
12289 4041 Acked Write 10 Sent Done Done false
12290 4042 Acked Write 7 Sent Done Done false
12291 4043 Acked Read 4 Sent Done Done false
12292 4044 Acked Write 8 Sent Done Done false
12293 4045 Acked Read 10 Sent Done Done false
12294 4046 Acked Write 2 Sent Done Done false
12295 4047 Acked Read 6 Sent Done Done false
12296 4048 Acked Write 6 Sent Done Done false
12297 4049 Acked Write 9 Sent Done Done false
12298 4050 Acked Read 5 Sent Done Done false
12299 4051 Acked Read 6 Sent Done Done false
12300 4052 Acked Read 1 Sent Done Done false
12301 4053 Acked Read 10 Sent Done Done false
12302 4054 Acked Write 1 Sent Done Done false
12303 4055 Acked Read 6 Sent Done Done false
12304 4056 Acked Flush 0 Sent Done Done false
12305 4057 Acked Flush 0 Sent Done Done false
12306 4058 Acked Read 8 Sent Done Done false
12307 4059 Acked Read 9 Sent Done Done false
12308 4060 Acked Read 8 Sent Done Done false
12309 4061 Acked Read 2 Sent Done Done false
12310 4062 Acked Read 8 Sent Done Done false
12311 4063 Acked Read 2 Sent Done Done false
12312 4064 Acked Read 2 Sent Done Done false
12313 4065 Acked Read 6 Sent Done Done false
12314 4066 Acked Read 3 Sent Done Done false
12315 4067 Acked Write 7 Sent Done Done false
12316 4068 Acked Flush 0 Sent Done Done false
12317 4069 Acked Read 9 Sent Done Done false
12318 4070 Acked Read 5 Sent Done Done false
12319 4071 Acked Read 8 Sent Done Done false
12320 4072 Acked Write 8 Sent Done Done false
12321 4073 Acked Write 6 Sent Done Done false
12322 4074 Acked Read 5 Sent Done Done false
12323 4075 Acked Write 8 Sent Done Done false
12324 4076 Acked Write 2 Sent Done Done false
12325 4077 Acked Read 10 Sent Done Done false
12326 4078 Acked Write 3 Sent Done Done false
12327 4079 Acked Flush 0 Sent Done Done false
12328 4080 Acked Read 9 Sent Done Done false
12329 4081 Acked Read 10 Sent Done Done false
12330 4082 Acked Read 6 Sent Done Done false
12331 4083 Acked Read 7 Sent Done Done false
12332 4084 Acked Write 9 Sent Done Done false
12333 4085 Acked Write 7 Sent Done Done false
12334 4086 Acked Read 5 Sent Done Done false
12335 4087 Acked Flush 0 Sent Done Done false
12336 4088 Acked Write 3 Sent Done Done false
12337 4089 Acked Write 9 Sent Done Done false
12338 4090 Acked Read 9 Sent Done Done false
12339 4091 Acked Read 2 Sent Done Done false
12340 4092 Acked Read 1 Sent Done Done false
12341 4093 Acked Read 8 Sent Done Done false
12342 4094 Acked Write 6 Sent Done Done false
12343 4095 Acked Read 7 Sent Done Done false
12344 4096 Acked Flush 0 Sent Done Done false
12345 4097 Acked Read 3 Sent Done Done false
12346 4098 Acked Read 7 Sent Done Done false
12347 4099 Acked Write 1 Sent Done Done false
12348 4100 Acked Read 2 Sent Done Done false
12349 4101 Acked Read 3 Sent Done Done false
12350 4102 Acked Write 4 Sent Done Done false
12351 4103 Acked Write 3 Sent Done Done false
12352 4104 Acked Write 2 Sent Done Done false
12353 4105 Acked Read 8 Sent Done Done false
12354 4106 Acked Read 2 Sent Done Done false
12355 4107 Acked Write 3 Sent Done Done false
12356 4108 Acked Read 3 Sent Done Done false
12357 4109 Acked Flush 0 Sent Done Done false
12358 4110 Acked Read 9 Sent Done Done false
12359 4111 Acked Read 9 Sent Done Done false
12360 4112 Acked Read 6 Sent Done Done false
12361 4113 Acked Read 1 Sent Done Done false
12362 4114 Acked Read 1 Sent Done Done false
12363 4115 Acked Read 3 Sent Done Done false
12364 4116 Acked Write 2 Sent Done Done false
12365 4117 Acked Read 7 Sent Done Done false
12366 4118 Acked Read 10 Sent Done Done false
12367 4119 Acked Write 8 Sent Done Done false
12368 4120 Acked Read 7 Sent Done Done false
12369 4121 Acked Write 9 Sent Done Done false
12370 4122 Acked Read 8 Sent Done Done false
12371 4123 Acked Read 8 Sent Done Done false
12372 4124 Acked Flush 0 Sent Done Done false
12373 4125 Acked Write 5 Sent Done Done false
12374 4126 Acked Read 6 Sent Done Done false
12375 4127 Acked Read 7 Sent Done Done false
12376 4128 Acked Write 1 Sent Done Done false
12377 4129 Acked Read 9 Sent Done Done false
12378 4130 Acked Write 1 Sent Done Done false
12379 4131 Acked Read 7 Sent Done Done false
12380 4132 Acked Write 10 Sent Done Done false
12381 4133 Acked Read 9 Sent Done Done false
12382 4134 Acked Read 2 Sent Done Done false
12383 4135 Acked Read 2 Sent Done Done false
12384 4136 Acked Write 2 Sent Done Done false
12385 4137 Acked Read 8 Sent Done Done false
12386 4138 Acked Read 1 Sent Done Done false
12387 4139 Acked Write 2 Sent Done Done false
12388 4140 Acked Write 7 Sent Done Done false
12389 4141 Acked Flush 0 Sent Done Done false
12390 4142 Acked Read 9 Sent Done Done false
12391 4143 Acked Read 4 Sent Done Done false
12392 4144 Acked Read 6 Sent Done Done false
12393 4145 Acked Write 5 Sent Done Done false
12394 4146 Acked Write 2 Sent Done Done false
12395 4147 Acked Write 6 Sent Done Done false
12396 4148 Acked Write 5 Sent Done Done false
12397 4149 Acked Flush 0 Sent Done Done false
12398 4150 Acked Write 10 Sent Done Done false
12399 4151 Acked Read 5 Sent Done Done false
12400 4152 Acked Write 7 Sent Done Done false
12401 4153 Acked Read 5 Sent Done Done false
12402 4154 Acked Flush 0 Sent Done Done false
12403 4155 Acked Write 10 Sent Done Done false
12404 4156 Acked Write 8 Sent Done Done false
12405 4157 Acked Read 10 Sent Done Done false
12406 4158 Acked Write 10 Sent Done Done false
12407 4159 Acked Read 4 Sent Done Done false
12408 4160 Acked Read 5 Sent Done Done false
12409 4161 Acked Read 5 Sent Done Done false
12410 4162 Acked Read 5 Sent Done Done false
12411 4163 Acked Read 5 Sent Done Done false
12412 4164 Acked Read 10 Sent Done Done false
12413 4165 Acked Read 7 Sent Done Done false
12414 4166 Acked Read 4 Sent Done Done false
12415 4167 Acked Write 2 Sent Done Done false
12416 4168 Acked Write 9 Sent Done Done false
12417 4169 Acked Write 2 Sent Done Done false
12418 4170 Acked Read 2 Sent Done Done false
12419 4171 Acked Write 4 Sent Done Done false
12420 4172 Acked Flush 0 Sent Done Done false
12421 4173 Acked Write 8 Sent Done Done false
12422 4174 Acked Read 3 Sent Done Done false
12423 4175 Acked Write 9 Sent Done Done false
12424 4176 Acked Read 9 Sent Done Done false
12425 4177 Acked Read 5 Sent Done Done false
12426 4178 Acked Read 7 Sent Done Done false
12427 4179 Acked Read 7 Sent Done Done false
12428 4180 Acked Write 4 Sent Done Done false
12429 4181 Acked Write 3 Sent Done Done false
12430 4182 Acked Read 9 Sent Done Done false
12431 4183 Acked Read 3 Sent Done Done false
12432 4184 Acked Flush 0 Sent Done Done false
12433 4185 Acked Flush 0 Sent Done Done false
12434 4186 Acked Write 6 Sent Done Done false
12435 4187 Acked Read 3 Sent Done Done false
12436 4188 Acked Write 3 Sent Done Done false
12437 4189 Acked Write 6 Sent Done Done false
12438 4190 Acked Write 4 Sent Done Done false
12439 4191 Acked Read 4 Sent Done Done false
12440 4192 Acked Read 3 Sent Done Done false
12441 4193 Acked Read 3 Sent Done Done false
12442 4194 Acked Read 5 Sent Done Done false
12443 4195 Acked Flush 0 Sent Done Done false
12444 4196 Acked Read 10 Sent Done Done false
12445 4197 Acked Write 1 Sent Done Done false
12446 4198 Acked Flush 0 Sent Done Done false
12447 4199 Acked Write 6 Sent Done Done false
12448 4200 Acked Write 5 Sent Done Done false
12449 4201 Acked Write 2 Sent Done Done false
12450 4202 Acked Write 3 Sent Done Done false
12451 4203 Acked Flush 0 Sent Done Done false
12452 4204 Acked Write 3 Sent Done Done false
12453 4205 Acked Read 2 Sent Done Done false
12454 4206 Acked Flush 0 Sent Done Done false
12455 4207 Acked Write 3 Sent Done Done false
12456 4208 Acked Write 5 Sent Done Done false
12457 4209 Acked Write 2 Sent Done Done false
12458 4210 Acked Flush 0 Sent Done Done false
12459 4211 Acked Write 7 Sent Done Done false
12460 4212 Acked Write 7 Sent Done Done false
12461 4213 Acked Read 1 Sent Done Done false
12462 4214 Acked Read 5 Sent Done Done false
12463 4215 Acked Read 1 Sent Done Done false
12464 4216 Acked Flush 0 Sent Done Done false
12465 4217 Acked Write 9 Sent Done Done false
12466 4218 Acked Read 10 Sent Done Done false
12467 4219 Acked Read 8 Sent Done Done false
12468 4220 Acked Write 1 Sent Done Done false
12469 4221 Acked Flush 0 Sent Done Done false
12470 4222 Acked Write 7 Sent Done Done false
12471 4223 Acked Read 4 Sent Done Done false
12472 4224 Acked Write 10 Sent Done Done false
12473 4225 Acked Read 5 Sent Done Done false
12474 4226 Acked Read 6 Sent Done Done false
12475 4227 Acked Write 5 Sent Done Done false
12476 4228 Acked Read 9 Sent Done Done false
12477 4229 Acked Write 5 Sent Done Done false
12478 4230 Acked Flush 0 Sent Done Done false
12479 4231 Acked Write 2 Sent Done Done false
12480 4232 Acked Read 1 Sent Done Done false
12481 4233 Acked Read 8 Sent Done Done false
12482 4234 Acked Read 3 Sent Done Done false
12483 4235 Acked Write 5 Sent Done Done false
12484 4236 Acked Read 6 Sent Done Done false
12485 4237 Acked Flush 0 Sent Done Done false
12486 4238 Acked Read 4 Sent Done Done false
12487 4239 Acked Read 2 Sent Done Done false
12488 4240 Acked Write 7 Sent Done Done false
12489 4241 Acked Read 5 Sent Done Done false
12490 4242 Acked Write 6 Sent Done Done false
12491 4243 Acked Read 1 Sent Done Done false
12492 4244 Acked Read 7 Sent Done Done false
12493 4245 Acked Read 9 Sent Done Done false
12494 4246 Acked Read 7 Sent Done Done false
12495 4247 Acked Read 8 Sent Done Done false
12496 4248 Acked Write 10 Sent Done Done false
12497 4249 Acked Read 6 Sent Done Done false
12498 4250 Acked Write 1 Sent Done Done false
12499 4251 Acked Write 2 Sent Done Done false
12500 4252 Acked Flush 0 Sent Done Done false
12501 4253 Acked Read 2 Sent Done Done false
12502 4254 Acked Write 9 Sent Done Done false
12503 4255 Acked Write 9 Sent Done Done false
12504 4256 Acked Read 8 Sent Done Done false
12505 4257 Acked Flush 0 Sent Done Done false
12506 4258 Acked Read 3 Sent Done Done false
12507 4259 Acked Write 9 Sent Done Done false
12508 4260 Acked Write 9 Sent Done Done false
12509 4261 Acked Write 3 Sent Done Done false
12510 4262 Acked Read 8 Sent Done Done false
12511 4263 Acked Write 9 Sent Done Done false
12512 4264 Acked Read 2 Sent Done Done false
12513 4265 Acked Flush 0 Sent Done Done false
12514 4266 Acked Flush 0 Sent Done Done false
12515 4267 Acked Read 9 Sent Done Done false
12516 4268 Acked Write 4 Sent Done Done false
12517 4269 Acked Flush 0 Sent Done Done false
12518 4270 Acked Read 3 Sent Done Done false
12519 4271 Acked Read 9 Sent Done Done false
12520 4272 Acked Read 6 Sent Done Done false
12521 4273 Acked Write 6 Sent Done Done false
12522 4274 Acked Write 8 Sent Done Done false
12523 4275 Acked Write 1 Sent Done Done false
12524 4276 Acked Write 3 Sent Done Done false
12525 4277 Acked Read 5 Sent Done Done false
12526 4278 Acked Read 5 Sent Done Done false
12527 4279 Acked Read 8 Sent Done Done false
12528 4280 Acked Write 4 Sent Done Done false
12529 4281 Acked Write 3 Sent Done Done false
12530 4282 Acked Read 3 Sent Done Done false
12531 4283 Acked Write 1 Sent Done Done false
12532 4284 Acked Write 1 Sent Done Done false
12533 4285 Acked Write 9 Sent Done Done false
12534 4286 Acked Write 2 Sent Done Done false
12535 4287 Acked Read 8 Sent Done Done false
12536 4288 Acked Write 6 Sent Done Done false
12537 4289 Acked Write 9 Sent Done Done false
12538 4290 Acked Write 5 Sent Done Done false
12539 4291 Acked Write 10 Sent Done Done false
12540 4292 Acked Write 4 Sent Done Done false
12541 4293 Acked Write 2 Sent Done Done false
12542 4294 Acked Write 3 Sent Done Done false
12543 4295 Acked Write 2 Sent Done Done false
12544 4296 Acked Read 3 Sent Done Done false
12545 4297 Acked Write 3 Sent Done Done false
12546 4298 Acked Write 6 Sent Done Done false
12547 4299 Acked Read 9 Sent Done Done false
12548 4300 Acked Write 1 Sent Done Done false
12549 4301 Acked Write 9 Sent Done Done false
12550 4302 Acked Read 8 Sent Done Done false
12551 4303 Acked Read 6 Sent Done Done false
12552 4304 Acked Read 6 Sent Done Done false
12553 4305 Acked Write 3 Sent Done Done false
12554 4306 Acked Read 1 Sent Done Done false
12555 4307 Acked Read 4 Sent Done Done false
12556 4308 Acked Flush 0 Sent Done Done false
12557 4309 Acked Read 1 Sent Done Done false
12558 4310 Acked Flush 0 Sent Done Done false
12559 4311 Acked Read 3 Sent Done Done false
12560 4312 Acked Flush 0 Sent Done Done false
12561 4313 Acked Read 1 Sent Done Done false
12562 4314 Acked Write 7 Sent Done Done false
12563 4315 Acked Flush 0 Sent Done Done false
12564 4316 Acked Read 4 Sent Done Done false
12565 4317 Acked Read 7 Sent Done Done false
12566 4318 Acked Read 5 Sent Done Done false
12567 4319 Acked Read 2 Sent Done Done false
12568 4320 Acked Read 6 Sent Done Done false
12569 STATES DS:0 DS:1 DS:2 TOTAL
12570 Sent 301 0 0 301
12571 Done 0 301 301 602
12572 Skipped 0 0 0 0
12573 Error 0 0 0 0
12574 Last Flush: Some(JobId(4019)) Some(JobId(4315)) Some(JobId(4315))
12575 Downstairs last five completed: 4019 4018 4017 4016 4015
12576 Upstairs last five completed: 4320 4319 4318 4317 4316
12577 CLIENT: Up:0 ds:301 act:2
125782025-02-21T06:48:06.906ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
125792025-02-21T06:48:07.908ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
125802025-02-21T06:48:08.910ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
125812025-02-21T06:48:09.985ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
12582 ----------------------------------------------------------------
12583 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12584 States: Offline Active Active
12585 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12586 4020 Acked Read 2 Sent Done Done false
12587 4021 Acked Flush 0 Sent Done Done false
12588 4022 Acked Flush 0 Sent Done Done false
12589 4023 Acked Read 3 Sent Done Done false
12590 4024 Acked Read 2 Sent Done Done false
12591 4025 Acked Read 6 Sent Done Done false
12592 4026 Acked Read 1 Sent Done Done false
12593 4027 Acked Read 2 Sent Done Done false
12594 4028 Acked Read 1 Sent Done Done false
12595 4029 Acked Read 7 Sent Done Done false
12596 4030 Acked Flush 0 Sent Done Done false
12597 4031 Acked Read 8 Sent Done Done false
12598 4032 Acked Read 1 Sent Done Done false
12599 4033 Acked Write 8 Sent Done Done false
12600 4034 Acked Read 8 Sent Done Done false
12601 4035 Acked Write 3 Sent Done Done false
12602 4036 Acked Write 4 Sent Done Done false
12603 4037 Acked Read 3 Sent Done Done false
12604 4038 Acked Write 5 Sent Done Done false
12605 4039 Acked Flush 0 Sent Done Done false
12606 4040 Acked Write 5 Sent Done Done false
12607 4041 Acked Write 10 Sent Done Done false
12608 4042 Acked Write 7 Sent Done Done false
12609 4043 Acked Read 4 Sent Done Done false
12610 4044 Acked Write 8 Sent Done Done false
12611 4045 Acked Read 10 Sent Done Done false
12612 4046 Acked Write 2 Sent Done Done false
12613 4047 Acked Read 6 Sent Done Done false
12614 4048 Acked Write 6 Sent Done Done false
12615 4049 Acked Write 9 Sent Done Done false
12616 4050 Acked Read 5 Sent Done Done false
12617 4051 Acked Read 6 Sent Done Done false
12618 4052 Acked Read 1 Sent Done Done false
12619 4053 Acked Read 10 Sent Done Done false
12620 4054 Acked Write 1 Sent Done Done false
12621 4055 Acked Read 6 Sent Done Done false
12622 4056 Acked Flush 0 Sent Done Done false
12623 4057 Acked Flush 0 Sent Done Done false
12624 4058 Acked Read 8 Sent Done Done false
12625 4059 Acked Read 9 Sent Done Done false
12626 4060 Acked Read 8 Sent Done Done false
12627 4061 Acked Read 2 Sent Done Done false
12628 4062 Acked Read 8 Sent Done Done false
12629 4063 Acked Read 2 Sent Done Done false
12630 4064 Acked Read 2 Sent Done Done false
12631 4065 Acked Read 6 Sent Done Done false
12632 4066 Acked Read 3 Sent Done Done false
12633 4067 Acked Write 7 Sent Done Done false
12634 4068 Acked Flush 0 Sent Done Done false
12635 4069 Acked Read 9 Sent Done Done false
12636 4070 Acked Read 5 Sent Done Done false
12637 4071 Acked Read 8 Sent Done Done false
12638 4072 Acked Write 8 Sent Done Done false
12639 4073 Acked Write 6 Sent Done Done false
12640 4074 Acked Read 5 Sent Done Done false
12641 4075 Acked Write 8 Sent Done Done false
12642 4076 Acked Write 2 Sent Done Done false
12643 4077 Acked Read 10 Sent Done Done false
12644 4078 Acked Write 3 Sent Done Done false
12645 4079 Acked Flush 0 Sent Done Done false
12646 4080 Acked Read 9 Sent Done Done false
12647 4081 Acked Read 10 Sent Done Done false
12648 4082 Acked Read 6 Sent Done Done false
12649 4083 Acked Read 7 Sent Done Done false
12650 4084 Acked Write 9 Sent Done Done false
12651 4085 Acked Write 7 Sent Done Done false
12652 4086 Acked Read 5 Sent Done Done false
12653 4087 Acked Flush 0 Sent Done Done false
12654 4088 Acked Write 3 Sent Done Done false
12655 4089 Acked Write 9 Sent Done Done false
12656 4090 Acked Read 9 Sent Done Done false
12657 4091 Acked Read 2 Sent Done Done false
12658 4092 Acked Read 1 Sent Done Done false
12659 4093 Acked Read 8 Sent Done Done false
12660 4094 Acked Write 6 Sent Done Done false
12661 4095 Acked Read 7 Sent Done Done false
12662 4096 Acked Flush 0 Sent Done Done false
12663 4097 Acked Read 3 Sent Done Done false
12664 4098 Acked Read 7 Sent Done Done false
12665 4099 Acked Write 1 Sent Done Done false
12666 4100 Acked Read 2 Sent Done Done false
12667 4101 Acked Read 3 Sent Done Done false
12668 4102 Acked Write 4 Sent Done Done false
12669 4103 Acked Write 3 Sent Done Done false
12670 4104 Acked Write 2 Sent Done Done false
12671 4105 Acked Read 8 Sent Done Done false
12672 4106 Acked Read 2 Sent Done Done false
12673 4107 Acked Write 3 Sent Done Done false
12674 4108 Acked Read 3 Sent Done Done false
12675 4109 Acked Flush 0 Sent Done Done false
12676 4110 Acked Read 9 Sent Done Done false
12677 4111 Acked Read 9 Sent Done Done false
12678 4112 Acked Read 6 Sent Done Done false
12679 4113 Acked Read 1 Sent Done Done false
12680 4114 Acked Read 1 Sent Done Done false
12681 4115 Acked Read 3 Sent Done Done false
12682 4116 Acked Write 2 Sent Done Done false
12683 4117 Acked Read 7 Sent Done Done false
12684 4118 Acked Read 10 Sent Done Done false
12685 4119 Acked Write 8 Sent Done Done false
12686 4120 Acked Read 7 Sent Done Done false
12687 4121 Acked Write 9 Sent Done Done false
12688 4122 Acked Read 8 Sent Done Done false
12689 4123 Acked Read 8 Sent Done Done false
12690 4124 Acked Flush 0 Sent Done Done false
12691 4125 Acked Write 5 Sent Done Done false
12692 4126 Acked Read 6 Sent Done Done false
12693 4127 Acked Read 7 Sent Done Done false
12694 4128 Acked Write 1 Sent Done Done false
12695 4129 Acked Read 9 Sent Done Done false
12696 4130 Acked Write 1 Sent Done Done false
12697 4131 Acked Read 7 Sent Done Done false
12698 4132 Acked Write 10 Sent Done Done false
12699 4133 Acked Read 9 Sent Done Done false
12700 4134 Acked Read 2 Sent Done Done false
12701 4135 Acked Read 2 Sent Done Done false
12702 4136 Acked Write 2 Sent Done Done false
12703 4137 Acked Read 8 Sent Done Done false
12704 4138 Acked Read 1 Sent Done Done false
12705 4139 Acked Write 2 Sent Done Done false
12706 4140 Acked Write 7 Sent Done Done false
12707 4141 Acked Flush 0 Sent Done Done false
12708 4142 Acked Read 9 Sent Done Done false
12709 4143 Acked Read 4 Sent Done Done false
12710 4144 Acked Read 6 Sent Done Done false
12711 4145 Acked Write 5 Sent Done Done false
12712 4146 Acked Write 2 Sent Done Done false
12713 4147 Acked Write 6 Sent Done Done false
12714 4148 Acked Write 5 Sent Done Done false
12715 4149 Acked Flush 0 Sent Done Done false
12716 4150 Acked Write 10 Sent Done Done false
12717 4151 Acked Read 5 Sent Done Done false
12718 4152 Acked Write 7 Sent Done Done false
12719 4153 Acked Read 5 Sent Done Done false
12720 4154 Acked Flush 0 Sent Done Done false
12721 4155 Acked Write 10 Sent Done Done false
12722 4156 Acked Write 8 Sent Done Done false
12723 4157 Acked Read 10 Sent Done Done false
12724 4158 Acked Write 10 Sent Done Done false
12725 4159 Acked Read 4 Sent Done Done false
12726 4160 Acked Read 5 Sent Done Done false
12727 4161 Acked Read 5 Sent Done Done false
12728 4162 Acked Read 5 Sent Done Done false
12729 4163 Acked Read 5 Sent Done Done false
12730 4164 Acked Read 10 Sent Done Done false
12731 4165 Acked Read 7 Sent Done Done false
12732 4166 Acked Read 4 Sent Done Done false
12733 4167 Acked Write 2 Sent Done Done false
12734 4168 Acked Write 9 Sent Done Done false
12735 4169 Acked Write 2 Sent Done Done false
12736 4170 Acked Read 2 Sent Done Done false
12737 4171 Acked Write 4 Sent Done Done false
12738 4172 Acked Flush 0 Sent Done Done false
12739 4173 Acked Write 8 Sent Done Done false
12740 4174 Acked Read 3 Sent Done Done false
12741 4175 Acked Write 9 Sent Done Done false
12742 4176 Acked Read 9 Sent Done Done false
12743 4177 Acked Read 5 Sent Done Done false
12744 4178 Acked Read 7 Sent Done Done false
12745 4179 Acked Read 7 Sent Done Done false
12746 4180 Acked Write 4 Sent Done Done false
12747 4181 Acked Write 3 Sent Done Done false
12748 4182 Acked Read 9 Sent Done Done false
12749 4183 Acked Read 3 Sent Done Done false
12750 4184 Acked Flush 0 Sent Done Done false
12751 4185 Acked Flush 0 Sent Done Done false
12752 4186 Acked Write 6 Sent Done Done false
12753 4187 Acked Read 3 Sent Done Done false
12754 4188 Acked Write 3 Sent Done Done false
12755 4189 Acked Write 6 Sent Done Done false
12756 4190 Acked Write 4 Sent Done Done false
12757 4191 Acked Read 4 Sent Done Done false
12758 4192 Acked Read 3 Sent Done Done false
12759 4193 Acked Read 3 Sent Done Done false
12760 4194 Acked Read 5 Sent Done Done false
12761 4195 Acked Flush 0 Sent Done Done false
12762 4196 Acked Read 10 Sent Done Done false
12763 4197 Acked Write 1 Sent Done Done false
12764 4198 Acked Flush 0 Sent Done Done false
12765 4199 Acked Write 6 Sent Done Done false
12766 4200 Acked Write 5 Sent Done Done false
12767 4201 Acked Write 2 Sent Done Done false
12768 4202 Acked Write 3 Sent Done Done false
12769 4203 Acked Flush 0 Sent Done Done false
12770 4204 Acked Write 3 Sent Done Done false
12771 4205 Acked Read 2 Sent Done Done false
12772 4206 Acked Flush 0 Sent Done Done false
12773 4207 Acked Write 3 Sent Done Done false
12774 4208 Acked Write 5 Sent Done Done false
12775 4209 Acked Write 2 Sent Done Done false
12776 4210 Acked Flush 0 Sent Done Done false
12777 4211 Acked Write 7 Sent Done Done false
12778 4212 Acked Write 7 Sent Done Done false
12779 4213 Acked Read 1 Sent Done Done false
12780 4214 Acked Read 5 Sent Done Done false
12781 4215 Acked Read 1 Sent Done Done false
12782 4216 Acked Flush 0 Sent Done Done false
12783 4217 Acked Write 9 Sent Done Done false
12784 4218 Acked Read 10 Sent Done Done false
12785 4219 Acked Read 8 Sent Done Done false
12786 4220 Acked Write 1 Sent Done Done false
12787 4221 Acked Flush 0 Sent Done Done false
12788 4222 Acked Write 7 Sent Done Done false
12789 4223 Acked Read 4 Sent Done Done false
12790 4224 Acked Write 10 Sent Done Done false
12791 4225 Acked Read 5 Sent Done Done false
12792 4226 Acked Read 6 Sent Done Done false
12793 4227 Acked Write 5 Sent Done Done false
12794 4228 Acked Read 9 Sent Done Done false
12795 4229 Acked Write 5 Sent Done Done false
12796 4230 Acked Flush 0 Sent Done Done false
12797 4231 Acked Write 2 Sent Done Done false
12798 4232 Acked Read 1 Sent Done Done false
12799 4233 Acked Read 8 Sent Done Done false
12800 4234 Acked Read 3 Sent Done Done false
12801 4235 Acked Write 5 Sent Done Done false
12802 4236 Acked Read 6 Sent Done Done false
12803 4237 Acked Flush 0 Sent Done Done false
12804 4238 Acked Read 4 Sent Done Done false
12805 4239 Acked Read 2 Sent Done Done false
12806 4240 Acked Write 7 Sent Done Done false
12807 4241 Acked Read 5 Sent Done Done false
12808 4242 Acked Write 6 Sent Done Done false
12809 4243 Acked Read 1 Sent Done Done false
12810 4244 Acked Read 7 Sent Done Done false
12811 4245 Acked Read 9 Sent Done Done false
12812 4246 Acked Read 7 Sent Done Done false
12813 4247 Acked Read 8 Sent Done Done false
12814 4248 Acked Write 10 Sent Done Done false
12815 4249 Acked Read 6 Sent Done Done false
12816 4250 Acked Write 1 Sent Done Done false
12817 4251 Acked Write 2 Sent Done Done false
12818 4252 Acked Flush 0 Sent Done Done false
12819 4253 Acked Read 2 Sent Done Done false
12820 4254 Acked Write 9 Sent Done Done false
12821 4255 Acked Write 9 Sent Done Done false
12822 4256 Acked Read 8 Sent Done Done false
12823 4257 Acked Flush 0 Sent Done Done false
12824 4258 Acked Read 3 Sent Done Done false
12825 4259 Acked Write 9 Sent Done Done false
12826 4260 Acked Write 9 Sent Done Done false
12827 4261 Acked Write 3 Sent Done Done false
12828 4262 Acked Read 8 Sent Done Done false
12829 4263 Acked Write 9 Sent Done Done false
12830 4264 Acked Read 2 Sent Done Done false
12831 4265 Acked Flush 0 Sent Done Done false
12832 4266 Acked Flush 0 Sent Done Done false
12833 4267 Acked Read 9 Sent Done Done false
12834 4268 Acked Write 4 Sent Done Done false
12835 4269 Acked Flush 0 Sent Done Done false
12836 4270 Acked Read 3 Sent Done Done false
12837 4271 Acked Read 9 Sent Done Done false
12838 4272 Acked Read 6 Sent Done Done false
12839 4273 Acked Write 6 Sent Done Done false
12840 4274 Acked Write 8 Sent Done Done false
12841 4275 Acked Write 1 Sent Done Done false
12842 4276 Acked Write 3 Sent Done Done false
12843 4277 Acked Read 5 Sent Done Done false
12844 4278 Acked Read 5 Sent Done Done false
12845 4279 Acked Read 8 Sent Done Done false
12846 4280 Acked Write 4 Sent Done Done false
12847 4281 Acked Write 3 Sent Done Done false
12848 4282 Acked Read 3 Sent Done Done false
12849 4283 Acked Write 1 Sent Done Done false
12850 4284 Acked Write 1 Sent Done Done false
12851 4285 Acked Write 9 Sent Done Done false
12852 4286 Acked Write 2 Sent Done Done false
12853 4287 Acked Read 8 Sent Done Done false
12854 4288 Acked Write 6 Sent Done Done false
12855 4289 Acked Write 9 Sent Done Done false
12856 4290 Acked Write 5 Sent Done Done false
12857 4291 Acked Write 10 Sent Done Done false
12858 4292 Acked Write 4 Sent Done Done false
12859 4293 Acked Write 2 Sent Done Done false
12860 4294 Acked Write 3 Sent Done Done false
12861 4295 Acked Write 2 Sent Done Done false
12862 4296 Acked Read 3 Sent Done Done false
12863 4297 Acked Write 3 Sent Done Done false
12864 4298 Acked Write 6 Sent Done Done false
12865 4299 Acked Read 9 Sent Done Done false
12866 4300 Acked Write 1 Sent Done Done false
12867 4301 Acked Write 9 Sent Done Done false
12868 4302 Acked Read 8 Sent Done Done false
12869 4303 Acked Read 6 Sent Done Done false
12870 4304 Acked Read 6 Sent Done Done false
12871 4305 Acked Write 3 Sent Done Done false
12872 4306 Acked Read 1 Sent Done Done false
12873 4307 Acked Read 4 Sent Done Done false
12874 4308 Acked Flush 0 Sent Done Done false
12875 4309 Acked Read 1 Sent Done Done false
12876 4310 Acked Flush 0 Sent Done Done false
12877 4311 Acked Read 3 Sent Done Done false
12878 4312 Acked Flush 0 Sent Done Done false
12879 4313 Acked Read 1 Sent Done Done false
12880 4314 Acked Write 7 Sent Done Done false
12881 4315 Acked Flush 0 Sent Done Done false
12882 4316 Acked Read 4 Sent Done Done false
12883 4317 Acked Read 7 Sent Done Done false
12884 4318 Acked Read 5 Sent Done Done false
12885 4319 Acked Read 2 Sent Done Done false
12886 4320 Acked Read 6 Sent Done Done false
12887 STATES DS:0 DS:1 DS:2 TOTAL
12888 Sent 301 0 0 301
12889 Done 0 301 301 602
12890 Skipped 0 0 0 0
12891 Error 0 0 0 0
12892 Last Flush: Some(JobId(4019)) Some(JobId(4315)) Some(JobId(4315))
12893 Downstairs last five completed: 4019 4018 4017 4016 4015
12894 Upstairs last five completed: 4320 4319 4318 4317 4316
12895 CLIENT: Up:0 ds:301 act:2
128962025-02-21T06:48:10.914ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
128972025-02-21T06:48:11.750ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
128982025-02-21T06:48:11.750ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
128992025-02-21T06:48:11.750ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
129002025-02-21T06:48:11.750ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
129012025-02-21T06:48:11.750ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
12902 [8810] got start action
12903 [8810] I am going to restart
12904 Starting downstairs at port 8810
12905 Make output file at "/tmp/dsc/downstairs-8810.txt"
12906 [8810][0] reports Starting
12907 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1217)
12908 [8810][0] reports Running
129092025-02-21T06:48:12.251ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
129102025-02-21T06:48:12.918ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
129112025-02-21T06:48:13.920ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
12912 ----------------------------------------------------------------
12913 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12914 States: Offline Active Active
12915 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12916 4020 Acked Read 2 Sent Done Done false
12917 4021 Acked Flush 0 Sent Done Done false
12918 4022 Acked Flush 0 Sent Done Done false
12919 4023 Acked Read 3 Sent Done Done false
12920 4024 Acked Read 2 Sent Done Done false
12921 4025 Acked Read 6 Sent Done Done false
12922 4026 Acked Read 1 Sent Done Done false
12923 4027 Acked Read 2 Sent Done Done false
12924 4028 Acked Read 1 Sent Done Done false
12925 4029 Acked Read 7 Sent Done Done false
12926 4030 Acked Flush 0 Sent Done Done false
12927 4031 Acked Read 8 Sent Done Done false
12928 4032 Acked Read 1 Sent Done Done false
12929 4033 Acked Write 8 Sent Done Done false
12930 4034 Acked Read 8 Sent Done Done false
12931 4035 Acked Write 3 Sent Done Done false
12932 4036 Acked Write 4 Sent Done Done false
12933 4037 Acked Read 3 Sent Done Done false
12934 4038 Acked Write 5 Sent Done Done false
12935 4039 Acked Flush 0 Sent Done Done false
12936 4040 Acked Write 5 Sent Done Done false
12937 4041 Acked Write 10 Sent Done Done false
12938 4042 Acked Write 7 Sent Done Done false
12939 4043 Acked Read 4 Sent Done Done false
12940 4044 Acked Write 8 Sent Done Done false
12941 4045 Acked Read 10 Sent Done Done false
12942 4046 Acked Write 2 Sent Done Done false
12943 4047 Acked Read 6 Sent Done Done false
12944 4048 Acked Write 6 Sent Done Done false
12945 4049 Acked Write 9 Sent Done Done false
12946 4050 Acked Read 5 Sent Done Done false
12947 4051 Acked Read 6 Sent Done Done false
12948 4052 Acked Read 1 Sent Done Done false
12949 4053 Acked Read 10 Sent Done Done false
12950 4054 Acked Write 1 Sent Done Done false
12951 4055 Acked Read 6 Sent Done Done false
12952 4056 Acked Flush 0 Sent Done Done false
12953 4057 Acked Flush 0 Sent Done Done false
12954 4058 Acked Read 8 Sent Done Done false
12955 4059 Acked Read 9 Sent Done Done false
12956 4060 Acked Read 8 Sent Done Done false
12957 4061 Acked Read 2 Sent Done Done false
12958 4062 Acked Read 8 Sent Done Done false
12959 4063 Acked Read 2 Sent Done Done false
12960 4064 Acked Read 2 Sent Done Done false
12961 4065 Acked Read 6 Sent Done Done false
12962 4066 Acked Read 3 Sent Done Done false
12963 4067 Acked Write 7 Sent Done Done false
12964 4068 Acked Flush 0 Sent Done Done false
12965 4069 Acked Read 9 Sent Done Done false
12966 4070 Acked Read 5 Sent Done Done false
12967 4071 Acked Read 8 Sent Done Done false
12968 4072 Acked Write 8 Sent Done Done false
12969 4073 Acked Write 6 Sent Done Done false
12970 4074 Acked Read 5 Sent Done Done false
12971 4075 Acked Write 8 Sent Done Done false
12972 4076 Acked Write 2 Sent Done Done false
12973 4077 Acked Read 10 Sent Done Done false
12974 4078 Acked Write 3 Sent Done Done false
12975 4079 Acked Flush 0 Sent Done Done false
12976 4080 Acked Read 9 Sent Done Done false
12977 4081 Acked Read 10 Sent Done Done false
12978 4082 Acked Read 6 Sent Done Done false
12979 4083 Acked Read 7 Sent Done Done false
12980 4084 Acked Write 9 Sent Done Done false
12981 4085 Acked Write 7 Sent Done Done false
12982 4086 Acked Read 5 Sent Done Done false
12983 4087 Acked Flush 0 Sent Done Done false
12984 4088 Acked Write 3 Sent Done Done false
12985 4089 Acked Write 9 Sent Done Done false
12986 4090 Acked Read 9 Sent Done Done false
12987 4091 Acked Read 2 Sent Done Done false
12988 4092 Acked Read 1 Sent Done Done false
12989 4093 Acked Read 8 Sent Done Done false
12990 4094 Acked Write 6 Sent Done Done false
12991 4095 Acked Read 7 Sent Done Done false
12992 4096 Acked Flush 0 Sent Done Done false
12993 4097 Acked Read 3 Sent Done Done false
12994 4098 Acked Read 7 Sent Done Done false
12995 4099 Acked Write 1 Sent Done Done false
12996 4100 Acked Read 2 Sent Done Done false
12997 4101 Acked Read 3 Sent Done Done false
12998 4102 Acked Write 4 Sent Done Done false
12999 4103 Acked Write 3 Sent Done Done false
13000 4104 Acked Write 2 Sent Done Done false
13001 4105 Acked Read 8 Sent Done Done false
13002 4106 Acked Read 2 Sent Done Done false
13003 4107 Acked Write 3 Sent Done Done false
13004 4108 Acked Read 3 Sent Done Done false
13005 4109 Acked Flush 0 Sent Done Done false
13006 4110 Acked Read 9 Sent Done Done false
13007 4111 Acked Read 9 Sent Done Done false
13008 4112 Acked Read 6 Sent Done Done false
13009 4113 Acked Read 1 Sent Done Done false
13010 4114 Acked Read 1 Sent Done Done false
13011 4115 Acked Read 3 Sent Done Done false
13012 4116 Acked Write 2 Sent Done Done false
13013 4117 Acked Read 7 Sent Done Done false
13014 4118 Acked Read 10 Sent Done Done false
13015 4119 Acked Write 8 Sent Done Done false
13016 4120 Acked Read 7 Sent Done Done false
13017 4121 Acked Write 9 Sent Done Done false
13018 4122 Acked Read 8 Sent Done Done false
13019 4123 Acked Read 8 Sent Done Done false
13020 4124 Acked Flush 0 Sent Done Done false
13021 4125 Acked Write 5 Sent Done Done false
13022 4126 Acked Read 6 Sent Done Done false
13023 4127 Acked Read 7 Sent Done Done false
13024 4128 Acked Write 1 Sent Done Done false
13025 4129 Acked Read 9 Sent Done Done false
13026 4130 Acked Write 1 Sent Done Done false
13027 4131 Acked Read 7 Sent Done Done false
13028 4132 Acked Write 10 Sent Done Done false
13029 4133 Acked Read 9 Sent Done Done false
13030 4134 Acked Read 2 Sent Done Done false
13031 4135 Acked Read 2 Sent Done Done false
13032 4136 Acked Write 2 Sent Done Done false
13033 4137 Acked Read 8 Sent Done Done false
13034 4138 Acked Read 1 Sent Done Done false
13035 4139 Acked Write 2 Sent Done Done false
13036 4140 Acked Write 7 Sent Done Done false
13037 4141 Acked Flush 0 Sent Done Done false
13038 4142 Acked Read 9 Sent Done Done false
13039 4143 Acked Read 4 Sent Done Done false
13040 4144 Acked Read 6 Sent Done Done false
13041 4145 Acked Write 5 Sent Done Done false
13042 4146 Acked Write 2 Sent Done Done false
13043 4147 Acked Write 6 Sent Done Done false
13044 4148 Acked Write 5 Sent Done Done false
13045 4149 Acked Flush 0 Sent Done Done false
13046 4150 Acked Write 10 Sent Done Done false
13047 4151 Acked Read 5 Sent Done Done false
13048 4152 Acked Write 7 Sent Done Done false
13049 4153 Acked Read 5 Sent Done Done false
13050 4154 Acked Flush 0 Sent Done Done false
13051 4155 Acked Write 10 Sent Done Done false
13052 4156 Acked Write 8 Sent Done Done false
13053 4157 Acked Read 10 Sent Done Done false
13054 4158 Acked Write 10 Sent Done Done false
13055 4159 Acked Read 4 Sent Done Done false
13056 4160 Acked Read 5 Sent Done Done false
13057 4161 Acked Read 5 Sent Done Done false
13058 4162 Acked Read 5 Sent Done Done false
13059 4163 Acked Read 5 Sent Done Done false
13060 4164 Acked Read 10 Sent Done Done false
13061 4165 Acked Read 7 Sent Done Done false
13062 4166 Acked Read 4 Sent Done Done false
13063 4167 Acked Write 2 Sent Done Done false
13064 4168 Acked Write 9 Sent Done Done false
13065 4169 Acked Write 2 Sent Done Done false
13066 4170 Acked Read 2 Sent Done Done false
13067 4171 Acked Write 4 Sent Done Done false
13068 4172 Acked Flush 0 Sent Done Done false
13069 4173 Acked Write 8 Sent Done Done false
13070 4174 Acked Read 3 Sent Done Done false
13071 4175 Acked Write 9 Sent Done Done false
13072 4176 Acked Read 9 Sent Done Done false
13073 4177 Acked Read 5 Sent Done Done false
13074 4178 Acked Read 7 Sent Done Done false
13075 4179 Acked Read 7 Sent Done Done false
13076 4180 Acked Write 4 Sent Done Done false
13077 4181 Acked Write 3 Sent Done Done false
13078 4182 Acked Read 9 Sent Done Done false
13079 4183 Acked Read 3 Sent Done Done false
13080 4184 Acked Flush 0 Sent Done Done false
13081 4185 Acked Flush 0 Sent Done Done false
13082 4186 Acked Write 6 Sent Done Done false
13083 4187 Acked Read 3 Sent Done Done false
13084 4188 Acked Write 3 Sent Done Done false
13085 4189 Acked Write 6 Sent Done Done false
13086 4190 Acked Write 4 Sent Done Done false
13087 4191 Acked Read 4 Sent Done Done false
13088 4192 Acked Read 3 Sent Done Done false
13089 4193 Acked Read 3 Sent Done Done false
13090 4194 Acked Read 5 Sent Done Done false
13091 4195 Acked Flush 0 Sent Done Done false
13092 4196 Acked Read 10 Sent Done Done false
13093 4197 Acked Write 1 Sent Done Done false
13094 4198 Acked Flush 0 Sent Done Done false
13095 4199 Acked Write 6 Sent Done Done false
13096 4200 Acked Write 5 Sent Done Done false
13097 4201 Acked Write 2 Sent Done Done false
13098 4202 Acked Write 3 Sent Done Done false
13099 4203 Acked Flush 0 Sent Done Done false
13100 4204 Acked Write 3 Sent Done Done false
13101 4205 Acked Read 2 Sent Done Done false
13102 4206 Acked Flush 0 Sent Done Done false
13103 4207 Acked Write 3 Sent Done Done false
13104 4208 Acked Write 5 Sent Done Done false
13105 4209 Acked Write 2 Sent Done Done false
13106 4210 Acked Flush 0 Sent Done Done false
13107 4211 Acked Write 7 Sent Done Done false
13108 4212 Acked Write 7 Sent Done Done false
13109 4213 Acked Read 1 Sent Done Done false
13110 4214 Acked Read 5 Sent Done Done false
13111 4215 Acked Read 1 Sent Done Done false
13112 4216 Acked Flush 0 Sent Done Done false
13113 4217 Acked Write 9 Sent Done Done false
13114 4218 Acked Read 10 Sent Done Done false
13115 4219 Acked Read 8 Sent Done Done false
13116 4220 Acked Write 1 Sent Done Done false
13117 4221 Acked Flush 0 Sent Done Done false
13118 4222 Acked Write 7 Sent Done Done false
13119 4223 Acked Read 4 Sent Done Done false
13120 4224 Acked Write 10 Sent Done Done false
13121 4225 Acked Read 5 Sent Done Done false
13122 4226 Acked Read 6 Sent Done Done false
13123 4227 Acked Write 5 Sent Done Done false
13124 4228 Acked Read 9 Sent Done Done false
13125 4229 Acked Write 5 Sent Done Done false
13126 4230 Acked Flush 0 Sent Done Done false
13127 4231 Acked Write 2 Sent Done Done false
13128 4232 Acked Read 1 Sent Done Done false
13129 4233 Acked Read 8 Sent Done Done false
13130 4234 Acked Read 3 Sent Done Done false
13131 4235 Acked Write 5 Sent Done Done false
13132 4236 Acked Read 6 Sent Done Done false
13133 4237 Acked Flush 0 Sent Done Done false
13134 4238 Acked Read 4 Sent Done Done false
13135 4239 Acked Read 2 Sent Done Done false
13136 4240 Acked Write 7 Sent Done Done false
13137 4241 Acked Read 5 Sent Done Done false
13138 4242 Acked Write 6 Sent Done Done false
13139 4243 Acked Read 1 Sent Done Done false
13140 4244 Acked Read 7 Sent Done Done false
13141 4245 Acked Read 9 Sent Done Done false
13142 4246 Acked Read 7 Sent Done Done false
13143 4247 Acked Read 8 Sent Done Done false
13144 4248 Acked Write 10 Sent Done Done false
13145 4249 Acked Read 6 Sent Done Done false
13146 4250 Acked Write 1 Sent Done Done false
13147 4251 Acked Write 2 Sent Done Done false
13148 4252 Acked Flush 0 Sent Done Done false
13149 4253 Acked Read 2 Sent Done Done false
13150 4254 Acked Write 9 Sent Done Done false
13151 4255 Acked Write 9 Sent Done Done false
13152 4256 Acked Read 8 Sent Done Done false
13153 4257 Acked Flush 0 Sent Done Done false
13154 4258 Acked Read 3 Sent Done Done false
13155 4259 Acked Write 9 Sent Done Done false
13156 4260 Acked Write 9 Sent Done Done false
13157 4261 Acked Write 3 Sent Done Done false
13158 4262 Acked Read 8 Sent Done Done false
13159 4263 Acked Write 9 Sent Done Done false
13160 4264 Acked Read 2 Sent Done Done false
13161 4265 Acked Flush 0 Sent Done Done false
13162 4266 Acked Flush 0 Sent Done Done false
13163 4267 Acked Read 9 Sent Done Done false
13164 4268 Acked Write 4 Sent Done Done false
13165 4269 Acked Flush 0 Sent Done Done false
13166 4270 Acked Read 3 Sent Done Done false
13167 4271 Acked Read 9 Sent Done Done false
13168 4272 Acked Read 6 Sent Done Done false
13169 4273 Acked Write 6 Sent Done Done false
13170 4274 Acked Write 8 Sent Done Done false
13171 4275 Acked Write 1 Sent Done Done false
13172 4276 Acked Write 3 Sent Done Done false
13173 4277 Acked Read 5 Sent Done Done false
13174 4278 Acked Read 5 Sent Done Done false
13175 4279 Acked Read 8 Sent Done Done false
13176 4280 Acked Write 4 Sent Done Done false
13177 4281 Acked Write 3 Sent Done Done false
13178 4282 Acked Read 3 Sent Done Done false
13179 4283 Acked Write 1 Sent Done Done false
13180 4284 Acked Write 1 Sent Done Done false
13181 4285 Acked Write 9 Sent Done Done false
13182 4286 Acked Write 2 Sent Done Done false
13183 4287 Acked Read 8 Sent Done Done false
13184 4288 Acked Write 6 Sent Done Done false
13185 4289 Acked Write 9 Sent Done Done false
13186 4290 Acked Write 5 Sent Done Done false
13187 4291 Acked Write 10 Sent Done Done false
13188 4292 Acked Write 4 Sent Done Done false
13189 4293 Acked Write 2 Sent Done Done false
13190 4294 Acked Write 3 Sent Done Done false
13191 4295 Acked Write 2 Sent Done Done false
13192 4296 Acked Read 3 Sent Done Done false
13193 4297 Acked Write 3 Sent Done Done false
13194 4298 Acked Write 6 Sent Done Done false
13195 4299 Acked Read 9 Sent Done Done false
13196 4300 Acked Write 1 Sent Done Done false
13197 4301 Acked Write 9 Sent Done Done false
13198 4302 Acked Read 8 Sent Done Done false
13199 4303 Acked Read 6 Sent Done Done false
13200 4304 Acked Read 6 Sent Done Done false
13201 4305 Acked Write 3 Sent Done Done false
13202 4306 Acked Read 1 Sent Done Done false
13203 4307 Acked Read 4 Sent Done Done false
13204 4308 Acked Flush 0 Sent Done Done false
13205 4309 Acked Read 1 Sent Done Done false
13206 4310 Acked Flush 0 Sent Done Done false
13207 4311 Acked Read 3 Sent Done Done false
13208 4312 Acked Flush 0 Sent Done Done false
13209 4313 Acked Read 1 Sent Done Done false
13210 4314 Acked Write 7 Sent Done Done false
13211 4315 Acked Flush 0 Sent Done Done false
13212 4316 Acked Read 4 Sent Done Done false
13213 4317 Acked Read 7 Sent Done Done false
13214 4318 Acked Read 5 Sent Done Done false
13215 4319 Acked Read 2 Sent Done Done false
13216 4320 Acked Read 6 Sent Done Done false
13217 STATES DS:0 DS:1 DS:2 TOTAL
13218 Sent 301 0 0 301
13219 Done 0 301 301 602
13220 Skipped 0 0 0 0
13221 Error 0 0 0 0
13222 Last Flush: Some(JobId(4019)) Some(JobId(4315)) Some(JobId(4315))
13223 Downstairs last five completed: 4019 4018 4017 4016 4015
13224 Upstairs last five completed: 4320 4319 4318 4317 4316
13225 CLIENT: Up:0 ds:301 act:2
132262025-02-21T06:48:14.987ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
132272025-02-21T06:48:15.923ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
132282025-02-21T06:48:16.925ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
132292025-02-21T06:48:17.927ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
13230 ----------------------------------------------------------------
13231 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13232 States: Offline Active Active
13233 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13234 4020 Acked Read 2 Sent Done Done false
13235 4021 Acked Flush 0 Sent Done Done false
13236 4022 Acked Flush 0 Sent Done Done false
13237 4023 Acked Read 3 Sent Done Done false
13238 4024 Acked Read 2 Sent Done Done false
13239 4025 Acked Read 6 Sent Done Done false
13240 4026 Acked Read 1 Sent Done Done false
13241 4027 Acked Read 2 Sent Done Done false
13242 4028 Acked Read 1 Sent Done Done false
13243 4029 Acked Read 7 Sent Done Done false
13244 4030 Acked Flush 0 Sent Done Done false
13245 4031 Acked Read 8 Sent Done Done false
13246 4032 Acked Read 1 Sent Done Done false
13247 4033 Acked Write 8 Sent Done Done false
13248 4034 Acked Read 8 Sent Done Done false
13249 4035 Acked Write 3 Sent Done Done false
13250 4036 Acked Write 4 Sent Done Done false
13251 4037 Acked Read 3 Sent Done Done false
13252 4038 Acked Write 5 Sent Done Done false
13253 4039 Acked Flush 0 Sent Done Done false
13254 4040 Acked Write 5 Sent Done Done false
13255 4041 Acked Write 10 Sent Done Done false
13256 4042 Acked Write 7 Sent Done Done false
13257 4043 Acked Read 4 Sent Done Done false
13258 4044 Acked Write 8 Sent Done Done false
13259 4045 Acked Read 10 Sent Done Done false
13260 4046 Acked Write 2 Sent Done Done false
13261 4047 Acked Read 6 Sent Done Done false
13262 4048 Acked Write 6 Sent Done Done false
13263 4049 Acked Write 9 Sent Done Done false
13264 4050 Acked Read 5 Sent Done Done false
13265 4051 Acked Read 6 Sent Done Done false
13266 4052 Acked Read 1 Sent Done Done false
13267 4053 Acked Read 10 Sent Done Done false
13268 4054 Acked Write 1 Sent Done Done false
13269 4055 Acked Read 6 Sent Done Done false
13270 4056 Acked Flush 0 Sent Done Done false
13271 4057 Acked Flush 0 Sent Done Done false
13272 4058 Acked Read 8 Sent Done Done false
13273 4059 Acked Read 9 Sent Done Done false
13274 4060 Acked Read 8 Sent Done Done false
13275 4061 Acked Read 2 Sent Done Done false
13276 4062 Acked Read 8 Sent Done Done false
13277 4063 Acked Read 2 Sent Done Done false
13278 4064 Acked Read 2 Sent Done Done false
13279 4065 Acked Read 6 Sent Done Done false
13280 4066 Acked Read 3 Sent Done Done false
13281 4067 Acked Write 7 Sent Done Done false
13282 4068 Acked Flush 0 Sent Done Done false
13283 4069 Acked Read 9 Sent Done Done false
13284 4070 Acked Read 5 Sent Done Done false
13285 4071 Acked Read 8 Sent Done Done false
13286 4072 Acked Write 8 Sent Done Done false
13287 4073 Acked Write 6 Sent Done Done false
13288 4074 Acked Read 5 Sent Done Done false
13289 4075 Acked Write 8 Sent Done Done false
13290 4076 Acked Write 2 Sent Done Done false
13291 4077 Acked Read 10 Sent Done Done false
13292 4078 Acked Write 3 Sent Done Done false
13293 4079 Acked Flush 0 Sent Done Done false
13294 4080 Acked Read 9 Sent Done Done false
13295 4081 Acked Read 10 Sent Done Done false
13296 4082 Acked Read 6 Sent Done Done false
13297 4083 Acked Read 7 Sent Done Done false
13298 4084 Acked Write 9 Sent Done Done false
13299 4085 Acked Write 7 Sent Done Done false
13300 4086 Acked Read 5 Sent Done Done false
13301 4087 Acked Flush 0 Sent Done Done false
13302 4088 Acked Write 3 Sent Done Done false
13303 4089 Acked Write 9 Sent Done Done false
13304 4090 Acked Read 9 Sent Done Done false
13305 4091 Acked Read 2 Sent Done Done false
13306 4092 Acked Read 1 Sent Done Done false
13307 4093 Acked Read 8 Sent Done Done false
13308 4094 Acked Write 6 Sent Done Done false
13309 4095 Acked Read 7 Sent Done Done false
13310 4096 Acked Flush 0 Sent Done Done false
13311 4097 Acked Read 3 Sent Done Done false
13312 4098 Acked Read 7 Sent Done Done false
13313 4099 Acked Write 1 Sent Done Done false
13314 4100 Acked Read 2 Sent Done Done false
13315 4101 Acked Read 3 Sent Done Done false
13316 4102 Acked Write 4 Sent Done Done false
13317 4103 Acked Write 3 Sent Done Done false
13318 4104 Acked Write 2 Sent Done Done false
13319 4105 Acked Read 8 Sent Done Done false
13320 4106 Acked Read 2 Sent Done Done false
13321 4107 Acked Write 3 Sent Done Done false
13322 4108 Acked Read 3 Sent Done Done false
13323 4109 Acked Flush 0 Sent Done Done false
13324 4110 Acked Read 9 Sent Done Done false
13325 4111 Acked Read 9 Sent Done Done false
13326 4112 Acked Read 6 Sent Done Done false
13327 4113 Acked Read 1 Sent Done Done false
13328 4114 Acked Read 1 Sent Done Done false
13329 4115 Acked Read 3 Sent Done Done false
13330 4116 Acked Write 2 Sent Done Done false
13331 4117 Acked Read 7 Sent Done Done false
13332 4118 Acked Read 10 Sent Done Done false
13333 4119 Acked Write 8 Sent Done Done false
13334 4120 Acked Read 7 Sent Done Done false
13335 4121 Acked Write 9 Sent Done Done false
13336 4122 Acked Read 8 Sent Done Done false
13337 4123 Acked Read 8 Sent Done Done false
13338 4124 Acked Flush 0 Sent Done Done false
13339 4125 Acked Write 5 Sent Done Done false
13340 4126 Acked Read 6 Sent Done Done false
13341 4127 Acked Read 7 Sent Done Done false
13342 4128 Acked Write 1 Sent Done Done false
13343 4129 Acked Read 9 Sent Done Done false
13344 4130 Acked Write 1 Sent Done Done false
13345 4131 Acked Read 7 Sent Done Done false
13346 4132 Acked Write 10 Sent Done Done false
13347 4133 Acked Read 9 Sent Done Done false
13348 4134 Acked Read 2 Sent Done Done false
13349 4135 Acked Read 2 Sent Done Done false
13350 4136 Acked Write 2 Sent Done Done false
13351 4137 Acked Read 8 Sent Done Done false
13352 4138 Acked Read 1 Sent Done Done false
13353 4139 Acked Write 2 Sent Done Done false
13354 4140 Acked Write 7 Sent Done Done false
13355 4141 Acked Flush 0 Sent Done Done false
13356 4142 Acked Read 9 Sent Done Done false
13357 4143 Acked Read 4 Sent Done Done false
13358 4144 Acked Read 6 Sent Done Done false
13359 4145 Acked Write 5 Sent Done Done false
13360 4146 Acked Write 2 Sent Done Done false
13361 4147 Acked Write 6 Sent Done Done false
13362 4148 Acked Write 5 Sent Done Done false
13363 4149 Acked Flush 0 Sent Done Done false
13364 4150 Acked Write 10 Sent Done Done false
13365 4151 Acked Read 5 Sent Done Done false
13366 4152 Acked Write 7 Sent Done Done false
13367 4153 Acked Read 5 Sent Done Done false
13368 4154 Acked Flush 0 Sent Done Done false
13369 4155 Acked Write 10 Sent Done Done false
13370 4156 Acked Write 8 Sent Done Done false
13371 4157 Acked Read 10 Sent Done Done false
13372 4158 Acked Write 10 Sent Done Done false
13373 4159 Acked Read 4 Sent Done Done false
13374 4160 Acked Read 5 Sent Done Done false
13375 4161 Acked Read 5 Sent Done Done false
13376 4162 Acked Read 5 Sent Done Done false
13377 4163 Acked Read 5 Sent Done Done false
13378 4164 Acked Read 10 Sent Done Done false
13379 4165 Acked Read 7 Sent Done Done false
13380 4166 Acked Read 4 Sent Done Done false
13381 4167 Acked Write 2 Sent Done Done false
13382 4168 Acked Write 9 Sent Done Done false
13383 4169 Acked Write 2 Sent Done Done false
13384 4170 Acked Read 2 Sent Done Done false
13385 4171 Acked Write 4 Sent Done Done false
13386 4172 Acked Flush 0 Sent Done Done false
13387 4173 Acked Write 8 Sent Done Done false
13388 4174 Acked Read 3 Sent Done Done false
13389 4175 Acked Write 9 Sent Done Done false
13390 4176 Acked Read 9 Sent Done Done false
13391 4177 Acked Read 5 Sent Done Done false
13392 4178 Acked Read 7 Sent Done Done false
13393 4179 Acked Read 7 Sent Done Done false
13394 4180 Acked Write 4 Sent Done Done false
13395 4181 Acked Write 3 Sent Done Done false
13396 4182 Acked Read 9 Sent Done Done false
13397 4183 Acked Read 3 Sent Done Done false
13398 4184 Acked Flush 0 Sent Done Done false
13399 4185 Acked Flush 0 Sent Done Done false
13400 4186 Acked Write 6 Sent Done Done false
13401 4187 Acked Read 3 Sent Done Done false
13402 4188 Acked Write 3 Sent Done Done false
13403 4189 Acked Write 6 Sent Done Done false
13404 4190 Acked Write 4 Sent Done Done false
13405 4191 Acked Read 4 Sent Done Done false
13406 4192 Acked Read 3 Sent Done Done false
13407 4193 Acked Read 3 Sent Done Done false
13408 4194 Acked Read 5 Sent Done Done false
13409 4195 Acked Flush 0 Sent Done Done false
13410 4196 Acked Read 10 Sent Done Done false
13411 4197 Acked Write 1 Sent Done Done false
13412 4198 Acked Flush 0 Sent Done Done false
13413 4199 Acked Write 6 Sent Done Done false
13414 4200 Acked Write 5 Sent Done Done false
13415 4201 Acked Write 2 Sent Done Done false
13416 4202 Acked Write 3 Sent Done Done false
13417 4203 Acked Flush 0 Sent Done Done false
13418 4204 Acked Write 3 Sent Done Done false
13419 4205 Acked Read 2 Sent Done Done false
13420 4206 Acked Flush 0 Sent Done Done false
13421 4207 Acked Write 3 Sent Done Done false
13422 4208 Acked Write 5 Sent Done Done false
13423 4209 Acked Write 2 Sent Done Done false
13424 4210 Acked Flush 0 Sent Done Done false
13425 4211 Acked Write 7 Sent Done Done false
13426 4212 Acked Write 7 Sent Done Done false
13427 4213 Acked Read 1 Sent Done Done false
13428 4214 Acked Read 5 Sent Done Done false
13429 4215 Acked Read 1 Sent Done Done false
13430 4216 Acked Flush 0 Sent Done Done false
13431 4217 Acked Write 9 Sent Done Done false
13432 4218 Acked Read 10 Sent Done Done false
13433 4219 Acked Read 8 Sent Done Done false
13434 4220 Acked Write 1 Sent Done Done false
13435 4221 Acked Flush 0 Sent Done Done false
13436 4222 Acked Write 7 Sent Done Done false
13437 4223 Acked Read 4 Sent Done Done false
13438 4224 Acked Write 10 Sent Done Done false
13439 4225 Acked Read 5 Sent Done Done false
13440 4226 Acked Read 6 Sent Done Done false
13441 4227 Acked Write 5 Sent Done Done false
13442 4228 Acked Read 9 Sent Done Done false
13443 4229 Acked Write 5 Sent Done Done false
13444 4230 Acked Flush 0 Sent Done Done false
13445 4231 Acked Write 2 Sent Done Done false
13446 4232 Acked Read 1 Sent Done Done false
13447 4233 Acked Read 8 Sent Done Done false
13448 4234 Acked Read 3 Sent Done Done false
13449 4235 Acked Write 5 Sent Done Done false
13450 4236 Acked Read 6 Sent Done Done false
13451 4237 Acked Flush 0 Sent Done Done false
13452 4238 Acked Read 4 Sent Done Done false
13453 4239 Acked Read 2 Sent Done Done false
13454 4240 Acked Write 7 Sent Done Done false
13455 4241 Acked Read 5 Sent Done Done false
13456 4242 Acked Write 6 Sent Done Done false
13457 4243 Acked Read 1 Sent Done Done false
13458 4244 Acked Read 7 Sent Done Done false
13459 4245 Acked Read 9 Sent Done Done false
13460 4246 Acked Read 7 Sent Done Done false
13461 4247 Acked Read 8 Sent Done Done false
13462 4248 Acked Write 10 Sent Done Done false
13463 4249 Acked Read 6 Sent Done Done false
13464 4250 Acked Write 1 Sent Done Done false
13465 4251 Acked Write 2 Sent Done Done false
13466 4252 Acked Flush 0 Sent Done Done false
13467 4253 Acked Read 2 Sent Done Done false
13468 4254 Acked Write 9 Sent Done Done false
13469 4255 Acked Write 9 Sent Done Done false
13470 4256 Acked Read 8 Sent Done Done false
13471 4257 Acked Flush 0 Sent Done Done false
13472 4258 Acked Read 3 Sent Done Done false
13473 4259 Acked Write 9 Sent Done Done false
13474 4260 Acked Write 9 Sent Done Done false
13475 4261 Acked Write 3 Sent Done Done false
13476 4262 Acked Read 8 Sent Done Done false
13477 4263 Acked Write 9 Sent Done Done false
13478 4264 Acked Read 2 Sent Done Done false
13479 4265 Acked Flush 0 Sent Done Done false
13480 4266 Acked Flush 0 Sent Done Done false
13481 4267 Acked Read 9 Sent Done Done false
13482 4268 Acked Write 4 Sent Done Done false
13483 4269 Acked Flush 0 Sent Done Done false
13484 4270 Acked Read 3 Sent Done Done false
13485 4271 Acked Read 9 Sent Done Done false
13486 4272 Acked Read 6 Sent Done Done false
13487 4273 Acked Write 6 Sent Done Done false
13488 4274 Acked Write 8 Sent Done Done false
13489 4275 Acked Write 1 Sent Done Done false
13490 4276 Acked Write 3 Sent Done Done false
13491 4277 Acked Read 5 Sent Done Done false
13492 4278 Acked Read 5 Sent Done Done false
13493 4279 Acked Read 8 Sent Done Done false
13494 4280 Acked Write 4 Sent Done Done false
13495 4281 Acked Write 3 Sent Done Done false
13496 4282 Acked Read 3 Sent Done Done false
13497 4283 Acked Write 1 Sent Done Done false
13498 4284 Acked Write 1 Sent Done Done false
13499 4285 Acked Write 9 Sent Done Done false
13500 4286 Acked Write 2 Sent Done Done false
13501 4287 Acked Read 8 Sent Done Done false
13502 4288 Acked Write 6 Sent Done Done false
13503 4289 Acked Write 9 Sent Done Done false
13504 4290 Acked Write 5 Sent Done Done false
13505 4291 Acked Write 10 Sent Done Done false
13506 4292 Acked Write 4 Sent Done Done false
13507 4293 Acked Write 2 Sent Done Done false
13508 4294 Acked Write 3 Sent Done Done false
13509 4295 Acked Write 2 Sent Done Done false
13510 4296 Acked Read 3 Sent Done Done false
13511 4297 Acked Write 3 Sent Done Done false
13512 4298 Acked Write 6 Sent Done Done false
13513 4299 Acked Read 9 Sent Done Done false
13514 4300 Acked Write 1 Sent Done Done false
13515 4301 Acked Write 9 Sent Done Done false
13516 4302 Acked Read 8 Sent Done Done false
13517 4303 Acked Read 6 Sent Done Done false
13518 4304 Acked Read 6 Sent Done Done false
13519 4305 Acked Write 3 Sent Done Done false
13520 4306 Acked Read 1 Sent Done Done false
13521 4307 Acked Read 4 Sent Done Done false
13522 4308 Acked Flush 0 Sent Done Done false
13523 4309 Acked Read 1 Sent Done Done false
13524 4310 Acked Flush 0 Sent Done Done false
13525 4311 Acked Read 3 Sent Done Done false
13526 4312 Acked Flush 0 Sent Done Done false
13527 4313 Acked Read 1 Sent Done Done false
13528 4314 Acked Write 7 Sent Done Done false
13529 4315 Acked Flush 0 Sent Done Done false
13530 4316 Acked Read 4 Sent Done Done false
13531 4317 Acked Read 7 Sent Done Done false
13532 4318 Acked Read 5 Sent Done Done false
13533 4319 Acked Read 2 Sent Done Done false
13534 4320 Acked Read 6 Sent Done Done false
13535 STATES DS:0 DS:1 DS:2 TOTAL
13536 Sent 301 0 0 301
13537 Done 0 301 301 602
13538 Skipped 0 0 0 0
13539 Error 0 0 0 0
13540 Last Flush: Some(JobId(4019)) Some(JobId(4315)) Some(JobId(4315))
13541 Downstairs last five completed: 4019 4018 4017 4016 4015
13542 Upstairs last five completed: 4320 4319 4318 4317 4316
13543 CLIENT: Up:0 ds:301 act:2
135442025-02-21T06:48:18.928ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135452025-02-21T06:48:19.988ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135462025-02-21T06:48:20.932ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135472025-02-21T06:48:21.751ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135482025-02-21T06:48:21.751ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65305) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135492025-02-21T06:48:21.752ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135502025-02-21T06:48:21.753ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135512025-02-21T06:48:21.753ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135522025-02-21T06:48:21.753ZINFOcrucible: send last flush ID to this DS: Some(JobId(4019)) = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135532025-02-21T06:48:21.753ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135542025-02-21T06:48:21.753ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4019)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135552025-02-21T06:48:21.753ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4019)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
13556 ----------------------------------------------------------------
13557 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:5
13558 States: Active Active Active
13559 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13560 4316 Acked Read 4 Done Done Done true
13561 4317 Acked Read 7 Done Done Done true
13562 4318 Acked Read 5 Done Done Done true
13563 4319 Acked Read 2 Done Done Done true
13564 4320 Acked Read 6 Done Done Done true
13565 STATES DS:0 DS:1 DS:2 TOTAL
13566 Sent 0 0 0 0
13567 Done 5 5 5 15
13568 Skipped 0 0 0 0
13569 Error 0 0 0 0
13570 Last Flush: Some(JobId(4315)) Some(JobId(4315)) Some(JobId(4315))
13571 Downstairs last five completed: 4315 4314 4313 4312 4311
13572 Upstairs last five completed: 4320 4319 4318 4317 4316
13573 CLIENT: Up:0 ds:5 act:3
13574 ----------------------------------------------------------------
13575 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13576 Downstairs last five completed: 4321 4320 4319 4318 4317
13577 Upstairs last five completed: 4321 4320 4319 4318 4317
13578 CLIENT: Up:0 ds:0 act:3
13579 Replay: All jobs finished
13580 Main task has work to do, go find it
13581 got dsc Stop(1)
13582 stop 1
13583 [8820] Got stop action so:false kr:false
13584 Feb 21 06:48:26.001 INFO request completed, latency_us: 359, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 7004e674-b885-44c0-b857-e2ec7e9f98b8, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
135852025-02-21T06:48:26.001ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135862025-02-21T06:48:26.001ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135872025-02-21T06:48:26.002ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
135882025-02-21T06:48:26.002ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
13589 Feb 21 06:48:26.002 INFO request completed, latency_us: 122, response_code: 200, uri: /state/cid/1, method: GET, req_id: 71a0c826-01f8-40d7-9820-3d5d524737e4, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
13590 [8820] Exited with: ExitStatus(unix_wait_status(9))
13591 [8820][1] reports Exit
13592 Feb 21 06:48:30.003 INFO request completed, latency_us: 192, response_code: 200, uri: /state/cid/1, method: GET, req_id: d3dfb108-900b-43e2-af81-ed5b91ddc5b7, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
13593 001/300 Read block 2053 len 24576
13594 002/300 Read block 469 len 8192
13595 003/300 Write block 1889 len 40960 data: 2 2 2 2 3 3 3 3 3 3
13596 004/300 Read block 4728 len 20480
13597 005/300 Write block 3265 len 8192 data: 4 5
13598 006/300 Write block 2068 len 36864 data: 5 5 4 4 4 3 3 2 2
13599 007/300 Read block 310 len 28672
13600 008/300 Read block 4746 len 12288
13601 009/300 Read block 294 len 4096
13602 010/300 Write block 64 len 24576 data: 4 3 3 3 3 3
13603 011/300 Write block 4077 len 4096 data: 6
13604 012/300 Read block 3143 len 4096
13605 013/300 Read block 4172 len 28672
13606 014/300 Read block 1366 len 8192
13607 015/300 Write block 4225 len 20480 data: 4 4 4 4 3
13608 016/300 Write block 737 len 20480 data: 4 4 4 3 3
13609 017/300 Flush
13610 018/300 Read block 167 len 32768
13611 019/300 Read block 162 len 40960
13612 020/300 Write block 4651 len 20480 data: 5 5 3 3 3
13613 021/300 Read block 866 len 4096
13614 022/300 Flush
13615 023/300 Write block 4330 len 12288 data: 2 2 2
13616 024/300 Flush
13617 025/300 Read block 684 len 36864
13618 026/300 Flush
13619 027/300 Read block 2155 len 8192
13620 028/300 Write block 1562 len 12288 data: 3 3 2
13621 029/300 Read block 821 len 24576
13622 030/300 Write block 3167 len 36864 data: 5 5 5 4 4 4 5 5 6
13623 031/300 Write block 4342 len 8192 data: 2 2
13624 032/300 Read block 339 len 4096
13625 033/300 Read block 3515 len 28672
13626 034/300 Read block 3816 len 24576
13627 035/300 Read block 2697 len 36864
13628 036/300 Flush
13629 037/300 Read block 3235 len 16384
13630 038/300 Write block 3321 len 24576 data: 3 3 3 3 3 2
13631 039/300 Read block 2499 len 20480
13632 040/300 Write block 599 len 20480 data: 2 4 4 4 3
13633 041/300 Write block 1877 len 8192 data: 3 2
13634 042/300 Write block 1938 len 40960 data: 2 2 2 2 2 2 3 3 3 3
13635 043/300 Write block 1940 len 24576 data: 3 3 3 3 4 4
13636 044/300 Write block 86 len 28672 data: 2 2 2 2 2 2 2
13637 045/300 Read block 1109 len 40960
13638 046/300 Read block 1666 len 28672
13639 047/300 Write block 274 len 4096 data: 2
13640 048/300 Read block 2818 len 12288
13641 049/300 Read block 3 len 24576
13642 050/300 Read block 4873 len 36864
13643 051/300 Read block 2788 len 16384
13644 052/300 Write block 4418 len 12288 data: 5 4 2
13645 053/300 Write block 1728 len 24576 data: 5 5 3 3 3 3
13646 054/300 Read block 3315 len 40960
13647 055/300 Read block 2384 len 16384
13648 056/300 Read block 2286 len 8192
13649 057/300 Write block 783 len 24576 data: 5 5 6 5 5 4
13650 058/300 Read block 330 len 24576
13651 059/300 Read block 4964 len 12288
13652 060/300 Read block 4809 len 4096
13653 061/300 Write block 2757 len 8192 data: 4 5
13654 062/300 Write block 2797 len 40960 data: 2 2 2 2 3 3 3 3 2 3
13655 063/300 Flush
13656 064/300 Read block 1031 len 36864
13657 065/300 Write block 4427 len 8192 data: 3 3
13658 066/300 Read block 1661 len 36864
13659 067/300 Flush
13660 068/300 Write block 3770 len 28672 data: 6 6 7 7 6 5 4
13661 069/300 Write block 3474 len 4096 data: 3
13662 070/300 Read block 1590 len 40960
13663 071/300 Read block 1535 len 16384
13664 072/300 Write block 922 len 12288 data: 2 2 2
13665 073/300 Write block 4612 len 32768 data: 3 3 4 3 4 4 5 3
13666 074/300 Write block 3272 len 16384 data: 3 4 3 3
13667 075/300 Flush
13668 076/300 Write block 3819 len 8192 data: 6 7
13669 077/300 Read block 4884 len 4096
13670 078/300 Read block 3691 len 32768
13671 079/300 Read block 986 len 36864
13672 080/300 Read block 3631 len 4096
13673 081/300 Read block 4093 len 20480
13674 082/300 Write block 2850 len 4096 data: 4
13675 083/300 Read block 4713 len 16384
13676 084/300 Flush
13677 085/300 Write block 368 len 40960 data: 2 3 3 3 4 3 4 4 4 5
13678 086/300 Write block 4865 len 12288 data: 3 3 3
13679 087/300 Write block 4233 len 20480 data: 2 2 2 2 2
13680 088/300 Read block 2336 len 24576
13681 089/300 Read block 2020 len 8192
13682 090/300 Read block 4503 len 28672
13683 091/300 Read block 1221 len 4096
13684 092/300 Write block 2136 len 36864 data: 3 3 3 3 3 2 2 2 2
13685 093/300 Write block 3028 len 12288 data: 2 2 2
13686 094/300 Read block 128 len 36864
13687 095/300 Write block 3098 len 24576 data: 3 3 4 4 4 5
13688 096/300 Read block 1122 len 20480
13689 097/300 Write block 1608 len 36864 data: 6 8 7 8 9 6 5 4 5
13690 098/300 Read block 1111 len 12288
13691 099/300 Write block 4108 len 36864 data: 4 4 3 3 4 3 3 3 3
13692 100/300 Write block 4655 len 24576 data: 4 3 2 3 3 3
13693 101/300 Read block 932 len 20480
13694 102/300 Write block 2302 len 32768 data: 3 4 4 4 4 5 4 4
13695 103/300 Flush
13696 104/300 Write block 1920 len 36864 data: 4 3 3 3 3 2 2 2 3
13697 105/300 Write block 725 len 4096 data: 2
13698 106/300 Read block 1954 len 8192
13699 107/300 Read block 2250 len 40960
13700 108/300 Read block 4132 len 4096
13701 109/300 Write block 4838 len 4096 data: 3
13702 110/300 Write block 1560 len 8192 data: 3 3
13703 111/300 Write block 3544 len 32768 data: 2 2 2 3 3 3 3 3
13704 112/300 Write block 1009 len 24576 data: 3 2 2 2 3 3
13705 113/300 Read block 2657 len 16384
13706 114/300 Write block 781 len 36864 data: 4 5 6 6 7 6 6 5 5
13707 115/300 Read block 3043 len 36864
13708 116/300 Read block 2373 len 40960
13709 117/300 Write block 3129 len 16384 data: 3 4 4 6
13710 118/300 Read block 4480 len 4096
13711 119/300 Flush
13712 120/300 Write block 3455 len 28672 data: 3 3 3 2 3 3 3
13713 121/300 Flush
13714 122/300 Read block 475 len 20480
13715 123/300 Read block 350 len 28672
13716 124/300 Write block 497 len 12288 data: 3 3 3
13717 125/300 Flush
13718 126/300 Read block 2601 len 16384
13719 127/300 Read block 3567 len 32768
13720 128/300 Read block 1387 len 20480
13721 129/300 Read block 133 len 8192
13722 130/300 Write block 746 len 16384 data: 2 2 2 3
13723 131/300 Read block 2504 len 32768
13724 132/300 Read block 4334 len 12288
13725 133/300 Write block 981 len 32768 data: 5 3 4 4 4 3 3 3
13726 134/300 Read block 4334 len 8192
13727 135/300 Flush
13728 136/300 Read block 2413 len 32768
13729 137/300 Read block 2935 len 16384
13730 138/300 Read block 3532 len 40960
13731 139/300 Read block 3858 len 36864
13732 140/300 Read block 682 len 40960
13733 141/300 Write block 1752 len 28672 data: 4 4 4 4 4 3 4
13734 142/300 Read block 2315 len 8192
13735 143/300 Write block 1928 len 12288 data: 4 3 3
13736 144/300 Write block 323 len 20480 data: 2 2 2 2 3
13737 145/300 Write block 3540 len 40960 data: 2 2 2 2 3 3 3 4 4 4
13738 146/300 Write block 2277 len 12288 data: 6 5 5
13739 147/300 Write block 677 len 32768 data: 5 4 3 2 2 2 3 2
13740 148/300 Read block 2229 len 28672
13741 149/300 Write block 3734 len 16384 data: 2 2 2 3
13742 150/300 Write block 2019 len 36864 data: 2 2 3 3 3 3 4 4 4
13743 151/300 Read block 3874 len 8192
13744 152/300 Read block 3592 len 4096
13745 153/300 Write block 811 len 16384 data: 2 2 2 3
13746 154/300 Read block 1881 len 40960
13747 155/300 Write block 2965 len 16384 data: 4 3 3 3
13748 156/300 Read block 1761 len 36864
13749 157/300 Write block 1875 len 12288 data: 3 3 4
13750 158/300 Read block 1261 len 8192
13751 159/300 Write block 3541 len 32768 data: 3 3 3 4 4 4 5 5
13752 160/300 Write block 158 len 24576 data: 4 4 4 3 3 2
13753 161/300 Write block 2992 len 32768 data: 2 2 2 2 2 2 2 2
13754 162/300 Write block 1525 len 40960 data: 4 4 4 3 3 5 5 5 7 6
13755 163/300 Flush
13756 164/300 Read block 1446 len 28672
13757 165/300 Write block 3445 len 32768 data: 3 3 2 2 3 3 3 3
13758 166/300 Read block 3088 len 36864
13759 167/300 Write block 1190 len 8192 data: 4 4
13760 168/300 Read block 3113 len 12288
13761 169/300 Read block 4688 len 24576
13762 170/300 Read block 2911 len 8192
13763 171/300 Write block 975 len 16384 data: 5 6 4 3
13764 172/300 Flush
13765 173/300 Read block 2404 len 36864
13766 174/300 Read block 183 len 12288
13767 175/300 Write block 2693 len 28672 data: 2 2 2 2 3 2 2
13768 176/300 Write block 1909 len 32768 data: 3 4 4 4 4 4 3 4
13769 177/300 Read block 3486 len 20480
13770 178/300 Read block 3897 len 16384
13771 179/300 Write block 1684 len 40960 data: 3 3 3 2 2 2 2 2 2 2
13772 180/300 Read block 3810 len 20480
13773 181/300 Read block 4764 len 8192
13774 182/300 Read block 4305 len 32768
13775 183/300 Flush
13776 184/300 Write block 1017 len 36864 data: 5 7 7 7 7 5 3 3 5
13777 185/300 Write block 271 len 28672 data: 2 2 2 3 2 2 2
13778 186/300 Flush
13779 187/300 Read block 1138 len 16384
13780 188/300 Read block 3977 len 20480
13781 189/300 Read block 730 len 20480
13782 190/300 Read block 937 len 32768
13783 191/300 Write block 122 len 40960 data: 5 4 4 3 2 2 2 2 2 2
13784 192/300 Write block 957 len 36864 data: 4 4 4 4 4 3 3 3 2
13785 193/300 Read block 995 len 40960
13786 194/300 Read block 3923 len 12288
13787 195/300 Read block 784 len 40960
13788 196/300 Read block 1515 len 16384
13789 197/300 Read block 2911 len 36864
13790 198/300 Write block 4003 len 24576 data: 3 4 4 5 5 6
13791 199/300 Write block 2280 len 16384 data: 5 5 7 4
13792 200/300 Read block 3733 len 36864
13793 201/300 Flush
13794 202/300 Read block 617 len 12288
13795 203/300 Read block 1556 len 36864
13796 204/300 Write block 3197 len 8192 data: 3 3
13797 205/300 Read block 2619 len 40960
13798 206/300 Read block 2419 len 36864
13799 207/300 Read block 3155 len 24576
13800 208/300 Read block 3679 len 12288
13801 209/300 Write block 4357 len 4096 data: 4
13802 210/300 Read block 4358 len 16384
13803 211/300 Read block 2915 len 32768
13804 212/300 Read block 4471 len 28672
13805 213/300 Read block 3345 len 32768
13806 214/300 Read block 4021 len 28672
13807 215/300 Read block 799 len 36864
13808 216/300 Write block 1478 len 12288 data: 3 3 3
13809 217/300 Flush
13810 218/300 Read block 3620 len 16384
13811 219/300 Read block 1203 len 16384
13812 220/300 Read block 1555 len 40960
13813 221/300 Write block 1199 len 4096 data: 3
13814 222/300 Read block 659 len 12288
13815 223/300 Read block 1112 len 32768
13816 224/300 Write block 3067 len 4096 data: 2
13817 225/300 Write block 1625 len 32768 data: 6 6 6 5 3 4 4 4
13818 226/300 Read block 2798 len 20480
13819 227/300 Read block 1283 len 8192
13820 228/300 Read block 3775 len 24576
13821 229/300 Read block 2614 len 12288
13822 230/300 Read block 2716 len 12288
13823 231/300 Read block 883 len 32768
13824 232/300 Read block 1378 len 36864
13825 233/300 Read block 1402 len 32768
13826 234/300 Read block 866 len 28672
13827 235/300 Write block 2123 len 12288 data: 4 3 3
13828 236/300 Read block 1822 len 28672
13829 237/300 Read block 4305 len 4096
13830 238/300 Write block 4591 len 8192 data: 2 2
13831 239/300 Read block 4422 len 24576
13832 240/300 Read block 326 len 4096
13833 241/300 Flush
13834 242/300 Write block 1170 len 16384 data: 5 4 4 5
13835 243/300 Write block 1792 len 28672 data: 4 3 3 2 2 3 4
13836 244/300 Write block 3103 len 8192 data: 6 4
13837 245/300 Read block 4582 len 24576
13838 246/300 Flush
13839 247/300 Write block 2457 len 40960 data: 2 2 2 2 2 3 4 4 2 2
13840 248/300 Write block 3075 len 24576 data: 2 3 3 2 2 3
13841 249/300 Read block 4290 len 4096
13842 250/300 Write block 1807 len 36864 data: 2 2 2 3 6 5 6 6 7
13843 251/300 Read block 2620 len 16384
13844 252/300 Write block 2726 len 20480 data: 3 3 3 3 3
13845 253/300 Read block 1677 len 8192
13846 254/300 Read block 628 len 16384
13847 255/300 Read block 1000 len 24576
13848 256/300 Write block 1418 len 36864 data: 4 5 6 6 5 6 6 4 6
13849 257/300 Read block 284 len 12288
13850 258/300 Flush
13851 259/300 Read block 928 len 4096
13852 260/300 Read block 4285 len 4096
13853 261/300 Read block 612 len 36864
13854 262/300 Write block 4860 len 16384 data: 4 4 4 4
13855 263/300 Read block 3028 len 4096
13856 264/300 Read block 2367 len 32768
13857 265/300 Read block 24 len 28672
13858 266/300 Write block 2158 len 8192 data: 3 3
13859 267/300 Write block 156 len 16384 data: 4 4 5 5
13860 268/300 Read block 2183 len 40960
13861 269/300 Read block 1064 len 36864
13862 270/300 Flush
13863 271/300 Write block 1386 len 36864 data: 3 3 3 3 3 3 3 2 2
13864 272/300 Read block 1185 len 24576
13865 273/300 Write block 1495 len 32768 data: 4 4 4 6 5 6 6 6
13866 274/300 Read block 464 len 28672
13867 275/300 Read block 1345 len 24576
13868 276/300 Read block 3129 len 28672
13869 277/300 Read block 1484 len 36864
13870 278/300 Read block 131 len 4096
13871 279/300 Read block 1091 len 40960
13872 280/300 Write block 3551 len 20480 data: 4 4 4 4 5
13873 281/300 Write block 2927 len 24576 data: 3 4 4 4 4 4
13874 282/300 Write block 1701 len 16384 data: 3 3 3 3
13875 283/300 Write block 3227 len 36864 data: 5 5 4 4 4 4 4 2 2
13876 284/300 Read block 4649 len 36864
13877 285/300 Read block 3632 len 20480
13878 286/300 Write block 4465 len 24576 data: 3 3 3 3 5 5
13879 287/300 Write block 3710 len 8192 data: 4 3
13880 288/300 Read block 3948 len 32768
13881 289/300 Read block 4250 len 32768
13882 290/300 Read block 2990 len 4096
13883 291/300 Write block 2097 len 24576 data: 3 3 3 3 3 2
13884 292/300 Write block 3620 len 8192 data: 2 3
13885 293/300 Read block 1231 len 8192
13886 294/300 Flush
13887 295/300 Read block 859 len 28672
13888 296/300 Write block 2605 len 8192 data: 6 5
13889 297/300 Read block 1554 len 12288
13890 298/300 Read block 3440 len 16384
13891 299/300 Read block 3539 len 12288
13892 300/300 Flush
13893 301/300 Write block 2443 len 20480 data: 3 3 3 3 3
13894 Main task has work to do, go find it
13895 got dsc Start(1)
13896 start 1
13897 Feb 21 06:48:30.253 INFO request completed, latency_us: 263, response_code: 204, uri: /start/cid/1, method: POST, req_id: 27f85393-57c7-4d68-b29d-73923a758c2e, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
13898 Replay: started 1, returned:Ok(())
13899 ----------------------------------------------------------------
13900 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13901 States: Active Offline Active
13902 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13903 4322 Acked Read 6 Done Sent Done false
13904 4323 Acked Read 2 Done Sent Done false
13905 4324 Acked Write 10 Done Sent Done false
13906 4325 Acked Read 5 Done Sent Done false
13907 4326 Acked Write 2 Done Sent Done false
13908 4327 Acked Write 9 Done Sent Done false
13909 4328 Acked Read 7 Done Sent Done false
13910 4329 Acked Read 3 Done Sent Done false
13911 4330 Acked Read 1 Done Sent Done false
13912 4331 Acked Write 6 Done Sent Done false
13913 4332 Acked Write 1 Done Sent Done false
13914 4333 Acked Read 1 Done Sent Done false
13915 4334 Acked Read 7 Done Sent Done false
13916 4335 Acked Read 2 Done Sent Done false
13917 4336 Acked Write 5 Done Sent Done false
13918 4337 Acked Write 5 Done Sent Done false
13919 4338 Acked Flush 0 Done Sent Done false
13920 4339 Acked Read 8 Done Sent Done false
13921 4340 Acked Read 10 Done Sent Done false
13922 4341 Acked Write 5 Done Sent Done false
13923 4342 Acked Read 1 Done Sent Done false
13924 4343 Acked Flush 0 Done Sent Done false
13925 4344 Acked Write 3 Done Sent Done false
13926 4345 Acked Flush 0 Done Sent Done false
13927 4346 Acked Read 9 Done Sent Done false
13928 4347 Acked Flush 0 Done Sent Done false
13929 4348 Acked Read 2 Done Sent Done false
13930 4349 Acked Write 3 Done Sent Done false
13931 4350 Acked Read 6 Done Sent Done false
13932 4351 Acked Write 9 Done Sent Done false
13933 4352 Acked Write 2 Done Sent Done false
13934 4353 Acked Read 1 Done Sent Done false
13935 4354 Acked Read 7 Done Sent Done false
13936 4355 Acked Read 6 Done Sent Done false
13937 4356 Acked Read 9 Done Sent Done false
13938 4357 Acked Flush 0 Done Sent Done false
13939 4358 Acked Read 4 Done Sent Done false
13940 4359 Acked Write 6 Done Sent Done false
13941 4360 Acked Read 5 Done Sent Done false
13942 4361 Acked Write 5 Done Sent Done false
13943 4362 Acked Write 2 Done Sent Done false
13944 4363 Acked Write 10 Done Sent Done false
13945 4364 Acked Write 6 Done Sent Done false
13946 4365 Acked Write 7 Done Sent Done false
13947 4366 Acked Read 10 Done Sent Done false
13948 4367 Acked Read 7 Done Sent Done false
13949 4368 Acked Write 1 Done Sent Done false
13950 4369 Acked Read 3 Done Sent Done false
13951 4370 Acked Read 6 Done Sent Done false
13952 4371 Acked Read 9 Done Sent Done false
13953 4372 Acked Read 4 Done Sent Done false
13954 4373 Acked Write 3 Done Sent Done false
13955 4374 Acked Write 6 Done Sent Done false
13956 4375 Acked Read 10 Done Sent Done false
13957 4376 Acked Read 4 Done Sent Done false
13958 4377 Acked Read 2 Done Sent Done false
13959 4378 Acked Write 6 Done Sent Done false
13960 4379 Acked Read 6 Done Sent Done false
13961 4380 Acked Read 3 Done Sent Done false
13962 4381 Acked Read 1 Done Sent Done false
13963 4382 Acked Write 2 Done Sent Done false
13964 4383 Acked Write 10 Done Sent Done false
13965 4384 Acked Flush 0 Done Sent Done false
13966 4385 Acked Read 9 Done Sent Done false
13967 4386 Acked Write 2 Done Sent Done false
13968 4387 Acked Read 9 Done Sent Done false
13969 4388 Acked Flush 0 Done Sent Done false
13970 4389 Acked Write 7 Done Sent Done false
13971 4390 Acked Write 1 Done Sent Done false
13972 4391 Acked Read 10 Done Sent Done false
13973 4392 Acked Read 4 Done Sent Done false
13974 4393 Acked Write 3 Done Sent Done false
13975 4394 Acked Write 8 Done Sent Done false
13976 4395 Acked Write 4 Done Sent Done false
13977 4396 Acked Flush 0 Done Sent Done false
13978 4397 Acked Write 2 Done Sent Done false
13979 4398 Acked Read 1 Done Sent Done false
13980 4399 Acked Read 8 Done Sent Done false
13981 4400 Acked Read 9 Done Sent Done false
13982 4401 Acked Read 1 Done Sent Done false
13983 4402 Acked Read 5 Done Sent Done false
13984 4403 Acked Write 1 Done Sent Done false
13985 4404 Acked Read 4 Done Sent Done false
13986 4405 Acked Flush 0 Done Sent Done false
13987 4406 Acked Write 10 Done Sent Done false
13988 4407 Acked Write 3 Done Sent Done false
13989 4408 Acked Write 5 Done Sent Done false
13990 4409 Acked Read 6 Done Sent Done false
13991 4410 Acked Read 2 Done Sent Done false
13992 4411 Acked Read 7 Done Sent Done false
13993 4412 Acked Read 1 Done Sent Done false
13994 4413 Acked Write 9 Done Sent Done false
13995 4414 Acked Write 3 Done Sent Done false
13996 4415 Acked Read 9 Done Sent Done false
13997 4416 Acked Write 6 Done Sent Done false
13998 4417 Acked Read 5 Done Sent Done false
13999 4418 Acked Write 9 Done Sent Done false
14000 4419 Acked Read 3 Done Sent Done false
14001 4420 Acked Write 9 Done Sent Done false
14002 4421 Acked Write 6 Done Sent Done false
14003 4422 Acked Read 5 Done Sent Done false
14004 4423 Acked Write 8 Done Sent Done false
14005 4424 Acked Flush 0 Done Sent Done false
14006 4425 Acked Write 9 Done Sent Done false
14007 4426 Acked Write 1 Done Sent Done false
14008 4427 Acked Read 2 Done Sent Done false
14009 4428 Acked Read 10 Done Sent Done false
14010 4429 Acked Read 1 Done Sent Done false
14011 4430 Acked Write 1 Done Sent Done false
14012 4431 Acked Write 2 Done Sent Done false
14013 4432 Acked Write 8 Done Sent Done false
14014 4433 Acked Write 6 Done Sent Done false
14015 4434 Acked Read 4 Done Sent Done false
14016 4435 Acked Write 9 Done Sent Done false
14017 4436 Acked Read 9 Done Sent Done false
14018 4437 Acked Read 10 Done Sent Done false
14019 4438 Acked Write 4 Done Sent Done false
14020 4439 Acked Read 1 Done Sent Done false
14021 4440 Acked Flush 0 Done Sent Done false
14022 4441 Acked Write 7 Done Sent Done false
14023 4442 Acked Flush 0 Done Sent Done false
14024 4443 Acked Read 5 Done Sent Done false
14025 4444 Acked Read 7 Done Sent Done false
14026 4445 Acked Write 3 Done Sent Done false
14027 4446 Acked Flush 0 Done Sent Done false
14028 4447 Acked Read 4 Done Sent Done false
14029 4448 Acked Read 8 Done Sent Done false
14030 4449 Acked Read 5 Done Sent Done false
14031 4450 Acked Read 2 Done Sent Done false
14032 4451 Acked Write 4 Done Sent Done false
14033 4452 Acked Read 8 Done Sent Done false
14034 4453 Acked Read 3 Done Sent Done false
14035 4454 Acked Write 8 Done Sent Done false
14036 4455 Acked Read 2 Done Sent Done false
14037 4456 Acked Flush 0 Done Sent Done false
14038 4457 Acked Read 8 Done Sent Done false
14039 4458 Acked Read 4 Done Sent Done false
14040 4459 Acked Read 10 Done Sent Done false
14041 4460 Acked Read 9 Done Sent Done false
14042 4461 Acked Read 10 Done Sent Done false
14043 4462 Acked Write 7 Done Sent Done false
14044 4463 Acked Read 2 Done Sent Done false
14045 4464 Acked Write 3 Done Sent Done false
14046 4465 Acked Write 5 Done Sent Done false
14047 4466 Acked Write 10 Done Sent Done false
14048 4467 Acked Write 3 Done Sent Done false
14049 4468 Acked Write 8 Done Sent Done false
14050 4469 Acked Read 7 Done Sent Done false
14051 4470 Acked Write 4 Done Sent Done false
14052 4471 Acked Write 9 Done Sent Done false
14053 4472 Acked Read 2 Done Sent Done false
14054 4473 Acked Read 1 Done Sent Done false
14055 4474 Acked Write 4 Done Sent Done false
14056 4475 Acked Read 10 Done Sent Done false
14057 4476 Acked Write 4 Done Sent Done false
14058 4477 Acked Read 9 Done Sent Done false
14059 4478 Acked Write 3 Done Sent Done false
14060 4479 Acked Read 2 Done Sent Done false
14061 4480 Acked Write 8 Done Sent Done false
14062 4481 Acked Write 6 Done Sent Done false
14063 4482 Acked Write 8 Done Sent Done false
14064 4483 Acked Write 10 Done Sent Done false
14065 4484 Acked Flush 0 Done Sent Done false
14066 4485 Acked Read 7 Done Sent Done false
14067 4486 Acked Write 8 Done Sent Done false
14068 4487 Acked Read 9 Done Sent Done false
14069 4488 Acked Write 2 Done Sent Done false
14070 4489 Acked Read 3 Done Sent Done false
14071 4490 Acked Read 6 Done Sent Done false
14072 4491 Acked Read 2 Done Sent Done false
14073 4492 Acked Write 4 Done Sent Done false
14074 4493 Acked Flush 0 Done Sent Done false
14075 4494 Acked Read 9 Done Sent Done false
14076 4495 Acked Read 3 Done Sent Done false
14077 4496 Acked Write 7 Done Sent Done false
14078 4497 Acked Write 8 Done Sent Done false
14079 4498 Acked Read 5 Done Sent Done false
14080 4499 Acked Read 4 Done Sent Done false
14081 4500 Acked Write 10 Done Sent Done false
14082 4501 Acked Read 5 Done Sent Done false
14083 4502 Acked Read 2 Done Sent Done false
14084 4503 Acked Read 8 Done Sent Done false
14085 4504 Acked Flush 0 Done Sent Done false
14086 4505 Acked Write 9 Done Sent Done false
14087 4506 Acked Write 7 Done Sent Done false
14088 4507 Acked Flush 0 Done Sent Done false
14089 4508 Acked Read 4 Done Sent Done false
14090 4509 Acked Read 5 Done Sent Done false
14091 4510 Acked Read 5 Done Sent Done false
14092 4511 Acked Read 8 Done Sent Done false
14093 4512 Acked Write 10 Done Sent Done false
14094 4513 Acked Write 9 Done Sent Done false
14095 4514 Acked Read 10 Done Sent Done false
14096 4515 Acked Read 3 Done Sent Done false
14097 4516 Acked Read 10 Done Sent Done false
14098 4517 Acked Read 4 Done Sent Done false
14099 4518 Acked Read 9 Done Sent Done false
14100 4519 Acked Write 6 Done Sent Done false
14101 4520 Acked Write 4 Done Sent Done false
14102 4521 Acked Read 9 Done Sent Done false
14103 4522 Acked Flush 0 Done Sent Done false
14104 4523 Acked Read 3 Done Sent Done false
14105 4524 Acked Read 9 Done Sent Done false
14106 4525 Acked Write 2 Done Sent Done false
14107 4526 Acked Read 10 Done Sent Done false
14108 4527 Acked Read 9 Done Sent Done false
14109 4528 Acked Read 6 Done Sent Done false
14110 4529 Acked Read 3 Done Sent Done false
14111 4530 Acked Write 1 Done Sent Done false
14112 4531 Acked Read 4 Done Sent Done false
14113 4532 Acked Read 8 Done Sent Done false
14114 4533 Acked Read 7 Done Sent Done false
14115 4534 Acked Read 8 Done Sent Done false
14116 4535 Acked Read 7 Done Sent Done false
14117 4536 Acked Read 9 Done Sent Done false
14118 4537 Acked Write 3 Done Sent Done false
14119 4538 Acked Flush 0 Done Sent Done false
14120 4539 Acked Read 4 Done Sent Done false
14121 4540 Acked Read 4 Done Sent Done false
14122 4541 Acked Read 10 Done Sent Done false
14123 4542 Acked Write 1 Done Sent Done false
14124 4543 Acked Read 3 Done Sent Done false
14125 4544 Acked Read 8 Done Sent Done false
14126 4545 Acked Write 1 Done Sent Done false
14127 4546 Acked Write 8 Done Sent Done false
14128 4547 Acked Read 5 Done Sent Done false
14129 4548 Acked Read 2 Done Sent Done false
14130 4549 Acked Read 6 Done Sent Done false
14131 4550 Acked Read 3 Done Sent Done false
14132 4551 Acked Read 3 Done Sent Done false
14133 4552 Acked Read 8 Done Sent Done false
14134 4553 Acked Read 9 Done Sent Done false
14135 4554 Acked Read 8 Done Sent Done false
14136 4555 Acked Read 7 Done Sent Done false
14137 4556 Acked Write 3 Done Sent Done false
14138 4557 Acked Read 7 Done Sent Done false
14139 4558 Acked Read 1 Done Sent Done false
14140 4559 Acked Write 2 Done Sent Done false
14141 4560 Acked Read 6 Done Sent Done false
14142 4561 Acked Read 1 Done Sent Done false
14143 4562 Acked Flush 0 Done Sent Done false
14144 4563 Acked Write 4 Done Sent Done false
14145 4564 Acked Write 7 Done Sent Done false
14146 4565 Acked Write 2 Done Sent Done false
14147 4566 Acked Read 6 Done Sent Done false
14148 4567 Acked Flush 0 Done Sent Done false
14149 4568 Acked Write 10 Done Sent Done false
14150 4569 Acked Write 6 Done Sent Done false
14151 4570 Acked Read 1 Done Sent Done false
14152 4571 Acked Write 9 Done Sent Done false
14153 4572 Acked Read 4 Done Sent Done false
14154 4573 Acked Write 5 Done Sent Done false
14155 4574 Acked Read 2 Done Sent Done false
14156 4575 Acked Read 4 Done Sent Done false
14157 4576 Acked Read 6 Done Sent Done false
14158 4577 Acked Write 9 Done Sent Done false
14159 4578 Acked Read 3 Done Sent Done false
14160 4579 Acked Flush 0 Done Sent Done false
14161 4580 Acked Read 1 Done Sent Done false
14162 4581 Acked Read 1 Done Sent Done false
14163 4582 Acked Read 9 Done Sent Done false
14164 4583 Acked Write 4 Done Sent Done false
14165 4584 Acked Read 1 Done Sent Done false
14166 4585 Acked Read 8 Done Sent Done false
14167 4586 Acked Read 7 Done Sent Done false
14168 4587 Acked Write 2 Done Sent Done false
14169 4588 Acked Write 4 Done Sent Done false
14170 4589 Acked Read 10 Done Sent Done false
14171 4590 Acked Read 9 Done Sent Done false
14172 4591 Acked Flush 0 Done Sent Done false
14173 4592 Acked Write 9 Done Sent Done false
14174 4593 Acked Read 6 Done Sent Done false
14175 4594 Acked Write 8 Done Sent Done false
14176 4595 Acked Read 7 Done Sent Done false
14177 4596 Acked Read 6 Done Sent Done false
14178 4597 Acked Read 7 Done Sent Done false
14179 4598 Acked Read 9 Done Sent Done false
14180 4599 Acked Read 1 Done Sent Done false
14181 4600 Acked Read 10 Done Sent Done false
14182 4601 Acked Write 5 Done Sent Done false
14183 4602 Acked Write 6 Done Sent Done false
14184 4603 Acked Write 4 Done Sent Done false
14185 4604 Acked Write 9 Done Sent Done false
14186 4605 Acked Read 9 Done Sent Done false
14187 4606 Acked Read 5 Done Sent Done false
14188 4607 Acked Write 6 Done Sent Done false
14189 4608 Acked Write 2 Done Sent Done false
14190 4609 Acked Read 8 Done Sent Done false
14191 4610 Acked Read 8 Done Sent Done false
14192 4611 Acked Read 1 Done Sent Done false
14193 4612 Acked Write 6 Done Sent Done false
14194 4613 Acked Write 2 Done Sent Done false
14195 4614 Acked Read 2 Done Sent Done false
14196 4615 Acked Flush 0 Done Sent Done false
14197 4616 Acked Read 7 Done Sent Done false
14198 4617 Acked Write 2 Done Sent Done false
14199 4618 Acked Read 3 Done Sent Done false
14200 4619 Acked Read 4 Done Sent Done false
14201 4620 Acked Read 3 Done Sent Done false
14202 4621 Acked Flush 0 Done Sent Done false
14203 4622 Acked Write 5 Done Sent Done false
14204 STATES DS:0 DS:1 DS:2 TOTAL
14205 Sent 0 301 0 301
14206 Done 301 0 301 602
14207 Skipped 0 0 0 0
14208 Error 0 0 0 0
14209 Last Flush: Some(JobId(4621)) Some(JobId(4321)) Some(JobId(4621))
14210 Downstairs last five completed: 4321 4320 4319 4318 4317
14211 Upstairs last five completed: 4622 4621 4620 4619 4618
14212 CLIENT: Up:0 ds:301 act:2
142132025-02-21T06:48:30.948ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
14214 [8820] got start action
14215 [8820] I am going to restart
14216 Starting downstairs at port 8820
14217 Make output file at "/tmp/dsc/downstairs-8820.txt"
14218 [8820][1] reports Starting
14219 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1218)
14220 [8820][1] reports Running
142212025-02-21T06:48:32.254ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
142222025-02-21T06:48:32.950ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
142232025-02-21T06:48:33.951ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
14224 ----------------------------------------------------------------
14225 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14226 States: Active Offline Active
14227 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14228 4322 Acked Read 6 Done Sent Done false
14229 4323 Acked Read 2 Done Sent Done false
14230 4324 Acked Write 10 Done Sent Done false
14231 4325 Acked Read 5 Done Sent Done false
14232 4326 Acked Write 2 Done Sent Done false
14233 4327 Acked Write 9 Done Sent Done false
14234 4328 Acked Read 7 Done Sent Done false
14235 4329 Acked Read 3 Done Sent Done false
14236 4330 Acked Read 1 Done Sent Done false
14237 4331 Acked Write 6 Done Sent Done false
14238 4332 Acked Write 1 Done Sent Done false
14239 4333 Acked Read 1 Done Sent Done false
14240 4334 Acked Read 7 Done Sent Done false
14241 4335 Acked Read 2 Done Sent Done false
14242 4336 Acked Write 5 Done Sent Done false
14243 4337 Acked Write 5 Done Sent Done false
14244 4338 Acked Flush 0 Done Sent Done false
14245 4339 Acked Read 8 Done Sent Done false
14246 4340 Acked Read 10 Done Sent Done false
14247 4341 Acked Write 5 Done Sent Done false
14248 4342 Acked Read 1 Done Sent Done false
14249 4343 Acked Flush 0 Done Sent Done false
14250 4344 Acked Write 3 Done Sent Done false
14251 4345 Acked Flush 0 Done Sent Done false
14252 4346 Acked Read 9 Done Sent Done false
14253 4347 Acked Flush 0 Done Sent Done false
14254 4348 Acked Read 2 Done Sent Done false
14255 4349 Acked Write 3 Done Sent Done false
14256 4350 Acked Read 6 Done Sent Done false
14257 4351 Acked Write 9 Done Sent Done false
14258 4352 Acked Write 2 Done Sent Done false
14259 4353 Acked Read 1 Done Sent Done false
14260 4354 Acked Read 7 Done Sent Done false
14261 4355 Acked Read 6 Done Sent Done false
14262 4356 Acked Read 9 Done Sent Done false
14263 4357 Acked Flush 0 Done Sent Done false
14264 4358 Acked Read 4 Done Sent Done false
14265 4359 Acked Write 6 Done Sent Done false
14266 4360 Acked Read 5 Done Sent Done false
14267 4361 Acked Write 5 Done Sent Done false
14268 4362 Acked Write 2 Done Sent Done false
14269 4363 Acked Write 10 Done Sent Done false
14270 4364 Acked Write 6 Done Sent Done false
14271 4365 Acked Write 7 Done Sent Done false
14272 4366 Acked Read 10 Done Sent Done false
14273 4367 Acked Read 7 Done Sent Done false
14274 4368 Acked Write 1 Done Sent Done false
14275 4369 Acked Read 3 Done Sent Done false
14276 4370 Acked Read 6 Done Sent Done false
14277 4371 Acked Read 9 Done Sent Done false
14278 4372 Acked Read 4 Done Sent Done false
14279 4373 Acked Write 3 Done Sent Done false
14280 4374 Acked Write 6 Done Sent Done false
14281 4375 Acked Read 10 Done Sent Done false
14282 4376 Acked Read 4 Done Sent Done false
14283 4377 Acked Read 2 Done Sent Done false
14284 4378 Acked Write 6 Done Sent Done false
14285 4379 Acked Read 6 Done Sent Done false
14286 4380 Acked Read 3 Done Sent Done false
14287 4381 Acked Read 1 Done Sent Done false
14288 4382 Acked Write 2 Done Sent Done false
14289 4383 Acked Write 10 Done Sent Done false
14290 4384 Acked Flush 0 Done Sent Done false
14291 4385 Acked Read 9 Done Sent Done false
14292 4386 Acked Write 2 Done Sent Done false
14293 4387 Acked Read 9 Done Sent Done false
14294 4388 Acked Flush 0 Done Sent Done false
14295 4389 Acked Write 7 Done Sent Done false
14296 4390 Acked Write 1 Done Sent Done false
14297 4391 Acked Read 10 Done Sent Done false
14298 4392 Acked Read 4 Done Sent Done false
14299 4393 Acked Write 3 Done Sent Done false
14300 4394 Acked Write 8 Done Sent Done false
14301 4395 Acked Write 4 Done Sent Done false
14302 4396 Acked Flush 0 Done Sent Done false
14303 4397 Acked Write 2 Done Sent Done false
14304 4398 Acked Read 1 Done Sent Done false
14305 4399 Acked Read 8 Done Sent Done false
14306 4400 Acked Read 9 Done Sent Done false
14307 4401 Acked Read 1 Done Sent Done false
14308 4402 Acked Read 5 Done Sent Done false
14309 4403 Acked Write 1 Done Sent Done false
14310 4404 Acked Read 4 Done Sent Done false
14311 4405 Acked Flush 0 Done Sent Done false
14312 4406 Acked Write 10 Done Sent Done false
14313 4407 Acked Write 3 Done Sent Done false
14314 4408 Acked Write 5 Done Sent Done false
14315 4409 Acked Read 6 Done Sent Done false
14316 4410 Acked Read 2 Done Sent Done false
14317 4411 Acked Read 7 Done Sent Done false
14318 4412 Acked Read 1 Done Sent Done false
14319 4413 Acked Write 9 Done Sent Done false
14320 4414 Acked Write 3 Done Sent Done false
14321 4415 Acked Read 9 Done Sent Done false
14322 4416 Acked Write 6 Done Sent Done false
14323 4417 Acked Read 5 Done Sent Done false
14324 4418 Acked Write 9 Done Sent Done false
14325 4419 Acked Read 3 Done Sent Done false
14326 4420 Acked Write 9 Done Sent Done false
14327 4421 Acked Write 6 Done Sent Done false
14328 4422 Acked Read 5 Done Sent Done false
14329 4423 Acked Write 8 Done Sent Done false
14330 4424 Acked Flush 0 Done Sent Done false
14331 4425 Acked Write 9 Done Sent Done false
14332 4426 Acked Write 1 Done Sent Done false
14333 4427 Acked Read 2 Done Sent Done false
14334 4428 Acked Read 10 Done Sent Done false
14335 4429 Acked Read 1 Done Sent Done false
14336 4430 Acked Write 1 Done Sent Done false
14337 4431 Acked Write 2 Done Sent Done false
14338 4432 Acked Write 8 Done Sent Done false
14339 4433 Acked Write 6 Done Sent Done false
14340 4434 Acked Read 4 Done Sent Done false
14341 4435 Acked Write 9 Done Sent Done false
14342 4436 Acked Read 9 Done Sent Done false
14343 4437 Acked Read 10 Done Sent Done false
14344 4438 Acked Write 4 Done Sent Done false
14345 4439 Acked Read 1 Done Sent Done false
14346 4440 Acked Flush 0 Done Sent Done false
14347 4441 Acked Write 7 Done Sent Done false
14348 4442 Acked Flush 0 Done Sent Done false
14349 4443 Acked Read 5 Done Sent Done false
14350 4444 Acked Read 7 Done Sent Done false
14351 4445 Acked Write 3 Done Sent Done false
14352 4446 Acked Flush 0 Done Sent Done false
14353 4447 Acked Read 4 Done Sent Done false
14354 4448 Acked Read 8 Done Sent Done false
14355 4449 Acked Read 5 Done Sent Done false
14356 4450 Acked Read 2 Done Sent Done false
14357 4451 Acked Write 4 Done Sent Done false
14358 4452 Acked Read 8 Done Sent Done false
14359 4453 Acked Read 3 Done Sent Done false
14360 4454 Acked Write 8 Done Sent Done false
14361 4455 Acked Read 2 Done Sent Done false
14362 4456 Acked Flush 0 Done Sent Done false
14363 4457 Acked Read 8 Done Sent Done false
14364 4458 Acked Read 4 Done Sent Done false
14365 4459 Acked Read 10 Done Sent Done false
14366 4460 Acked Read 9 Done Sent Done false
14367 4461 Acked Read 10 Done Sent Done false
14368 4462 Acked Write 7 Done Sent Done false
14369 4463 Acked Read 2 Done Sent Done false
14370 4464 Acked Write 3 Done Sent Done false
14371 4465 Acked Write 5 Done Sent Done false
14372 4466 Acked Write 10 Done Sent Done false
14373 4467 Acked Write 3 Done Sent Done false
14374 4468 Acked Write 8 Done Sent Done false
14375 4469 Acked Read 7 Done Sent Done false
14376 4470 Acked Write 4 Done Sent Done false
14377 4471 Acked Write 9 Done Sent Done false
14378 4472 Acked Read 2 Done Sent Done false
14379 4473 Acked Read 1 Done Sent Done false
14380 4474 Acked Write 4 Done Sent Done false
14381 4475 Acked Read 10 Done Sent Done false
14382 4476 Acked Write 4 Done Sent Done false
14383 4477 Acked Read 9 Done Sent Done false
14384 4478 Acked Write 3 Done Sent Done false
14385 4479 Acked Read 2 Done Sent Done false
14386 4480 Acked Write 8 Done Sent Done false
14387 4481 Acked Write 6 Done Sent Done false
14388 4482 Acked Write 8 Done Sent Done false
14389 4483 Acked Write 10 Done Sent Done false
14390 4484 Acked Flush 0 Done Sent Done false
14391 4485 Acked Read 7 Done Sent Done false
14392 4486 Acked Write 8 Done Sent Done false
14393 4487 Acked Read 9 Done Sent Done false
14394 4488 Acked Write 2 Done Sent Done false
14395 4489 Acked Read 3 Done Sent Done false
14396 4490 Acked Read 6 Done Sent Done false
14397 4491 Acked Read 2 Done Sent Done false
14398 4492 Acked Write 4 Done Sent Done false
14399 4493 Acked Flush 0 Done Sent Done false
14400 4494 Acked Read 9 Done Sent Done false
14401 4495 Acked Read 3 Done Sent Done false
14402 4496 Acked Write 7 Done Sent Done false
14403 4497 Acked Write 8 Done Sent Done false
14404 4498 Acked Read 5 Done Sent Done false
14405 4499 Acked Read 4 Done Sent Done false
14406 4500 Acked Write 10 Done Sent Done false
14407 4501 Acked Read 5 Done Sent Done false
14408 4502 Acked Read 2 Done Sent Done false
14409 4503 Acked Read 8 Done Sent Done false
14410 4504 Acked Flush 0 Done Sent Done false
14411 4505 Acked Write 9 Done Sent Done false
14412 4506 Acked Write 7 Done Sent Done false
14413 4507 Acked Flush 0 Done Sent Done false
14414 4508 Acked Read 4 Done Sent Done false
14415 4509 Acked Read 5 Done Sent Done false
14416 4510 Acked Read 5 Done Sent Done false
14417 4511 Acked Read 8 Done Sent Done false
14418 4512 Acked Write 10 Done Sent Done false
14419 4513 Acked Write 9 Done Sent Done false
14420 4514 Acked Read 10 Done Sent Done false
14421 4515 Acked Read 3 Done Sent Done false
14422 4516 Acked Read 10 Done Sent Done false
14423 4517 Acked Read 4 Done Sent Done false
14424 4518 Acked Read 9 Done Sent Done false
14425 4519 Acked Write 6 Done Sent Done false
14426 4520 Acked Write 4 Done Sent Done false
14427 4521 Acked Read 9 Done Sent Done false
14428 4522 Acked Flush 0 Done Sent Done false
14429 4523 Acked Read 3 Done Sent Done false
14430 4524 Acked Read 9 Done Sent Done false
14431 4525 Acked Write 2 Done Sent Done false
14432 4526 Acked Read 10 Done Sent Done false
14433 4527 Acked Read 9 Done Sent Done false
14434 4528 Acked Read 6 Done Sent Done false
14435 4529 Acked Read 3 Done Sent Done false
14436 4530 Acked Write 1 Done Sent Done false
14437 4531 Acked Read 4 Done Sent Done false
14438 4532 Acked Read 8 Done Sent Done false
14439 4533 Acked Read 7 Done Sent Done false
14440 4534 Acked Read 8 Done Sent Done false
14441 4535 Acked Read 7 Done Sent Done false
14442 4536 Acked Read 9 Done Sent Done false
14443 4537 Acked Write 3 Done Sent Done false
14444 4538 Acked Flush 0 Done Sent Done false
14445 4539 Acked Read 4 Done Sent Done false
14446 4540 Acked Read 4 Done Sent Done false
14447 4541 Acked Read 10 Done Sent Done false
14448 4542 Acked Write 1 Done Sent Done false
14449 4543 Acked Read 3 Done Sent Done false
14450 4544 Acked Read 8 Done Sent Done false
14451 4545 Acked Write 1 Done Sent Done false
14452 4546 Acked Write 8 Done Sent Done false
14453 4547 Acked Read 5 Done Sent Done false
14454 4548 Acked Read 2 Done Sent Done false
14455 4549 Acked Read 6 Done Sent Done false
14456 4550 Acked Read 3 Done Sent Done false
14457 4551 Acked Read 3 Done Sent Done false
14458 4552 Acked Read 8 Done Sent Done false
14459 4553 Acked Read 9 Done Sent Done false
14460 4554 Acked Read 8 Done Sent Done false
14461 4555 Acked Read 7 Done Sent Done false
14462 4556 Acked Write 3 Done Sent Done false
14463 4557 Acked Read 7 Done Sent Done false
14464 4558 Acked Read 1 Done Sent Done false
14465 4559 Acked Write 2 Done Sent Done false
14466 4560 Acked Read 6 Done Sent Done false
14467 4561 Acked Read 1 Done Sent Done false
14468 4562 Acked Flush 0 Done Sent Done false
14469 4563 Acked Write 4 Done Sent Done false
14470 4564 Acked Write 7 Done Sent Done false
14471 4565 Acked Write 2 Done Sent Done false
14472 4566 Acked Read 6 Done Sent Done false
14473 4567 Acked Flush 0 Done Sent Done false
14474 4568 Acked Write 10 Done Sent Done false
14475 4569 Acked Write 6 Done Sent Done false
14476 4570 Acked Read 1 Done Sent Done false
14477 4571 Acked Write 9 Done Sent Done false
14478 4572 Acked Read 4 Done Sent Done false
14479 4573 Acked Write 5 Done Sent Done false
14480 4574 Acked Read 2 Done Sent Done false
14481 4575 Acked Read 4 Done Sent Done false
14482 4576 Acked Read 6 Done Sent Done false
14483 4577 Acked Write 9 Done Sent Done false
14484 4578 Acked Read 3 Done Sent Done false
14485 4579 Acked Flush 0 Done Sent Done false
14486 4580 Acked Read 1 Done Sent Done false
14487 4581 Acked Read 1 Done Sent Done false
14488 4582 Acked Read 9 Done Sent Done false
14489 4583 Acked Write 4 Done Sent Done false
14490 4584 Acked Read 1 Done Sent Done false
14491 4585 Acked Read 8 Done Sent Done false
14492 4586 Acked Read 7 Done Sent Done false
14493 4587 Acked Write 2 Done Sent Done false
14494 4588 Acked Write 4 Done Sent Done false
14495 4589 Acked Read 10 Done Sent Done false
14496 4590 Acked Read 9 Done Sent Done false
14497 4591 Acked Flush 0 Done Sent Done false
14498 4592 Acked Write 9 Done Sent Done false
14499 4593 Acked Read 6 Done Sent Done false
14500 4594 Acked Write 8 Done Sent Done false
14501 4595 Acked Read 7 Done Sent Done false
14502 4596 Acked Read 6 Done Sent Done false
14503 4597 Acked Read 7 Done Sent Done false
14504 4598 Acked Read 9 Done Sent Done false
14505 4599 Acked Read 1 Done Sent Done false
14506 4600 Acked Read 10 Done Sent Done false
14507 4601 Acked Write 5 Done Sent Done false
14508 4602 Acked Write 6 Done Sent Done false
14509 4603 Acked Write 4 Done Sent Done false
14510 4604 Acked Write 9 Done Sent Done false
14511 4605 Acked Read 9 Done Sent Done false
14512 4606 Acked Read 5 Done Sent Done false
14513 4607 Acked Write 6 Done Sent Done false
14514 4608 Acked Write 2 Done Sent Done false
14515 4609 Acked Read 8 Done Sent Done false
14516 4610 Acked Read 8 Done Sent Done false
14517 4611 Acked Read 1 Done Sent Done false
14518 4612 Acked Write 6 Done Sent Done false
14519 4613 Acked Write 2 Done Sent Done false
14520 4614 Acked Read 2 Done Sent Done false
14521 4615 Acked Flush 0 Done Sent Done false
14522 4616 Acked Read 7 Done Sent Done false
14523 4617 Acked Write 2 Done Sent Done false
14524 4618 Acked Read 3 Done Sent Done false
14525 4619 Acked Read 4 Done Sent Done false
14526 4620 Acked Read 3 Done Sent Done false
14527 4621 Acked Flush 0 Done Sent Done false
14528 4622 Acked Write 5 Done Sent Done false
14529 STATES DS:0 DS:1 DS:2 TOTAL
14530 Sent 0 301 0 301
14531 Done 301 0 301 602
14532 Skipped 0 0 0 0
14533 Error 0 0 0 0
14534 Last Flush: Some(JobId(4621)) Some(JobId(4321)) Some(JobId(4621))
14535 Downstairs last five completed: 4321 4320 4319 4318 4317
14536 Upstairs last five completed: 4622 4621 4620 4619 4618
14537 CLIENT: Up:0 ds:301 act:2
145382025-02-21T06:48:34.992ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
145392025-02-21T06:48:35.955ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
145402025-02-21T06:48:36.003ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
145412025-02-21T06:48:36.003ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:35120) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
145422025-02-21T06:48:36.004ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
145432025-02-21T06:48:36.005ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
145442025-02-21T06:48:36.005ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
145452025-02-21T06:48:36.005ZINFOcrucible: send last flush ID to this DS: Some(JobId(4321)) = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
145462025-02-21T06:48:36.005ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
145472025-02-21T06:48:36.005ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4321)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
145482025-02-21T06:48:36.005ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4321)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
14549 ----------------------------------------------------------------
14550 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14551 Downstairs last five completed: 4623 4622 4621 4620 4619
14552 Upstairs last five completed: 4623 4622 4621 4620 4619
14553 CLIENT: Up:0 ds:0 act:3
14554 Replay: All jobs finished
14555 Main task has work to do, go find it
14556 got dsc Stop(2)
14557 stop 2
14558 Feb 21 06:48:38.259[8830] Got stop action so:false kr:false
14559 INFO request completed, latency_us: 415, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 3890d019-4e3c-40e5-883a-6b29871badba, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
145602025-02-21T06:48:38.260ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
145612025-02-21T06:48:38.260ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
145622025-02-21T06:48:38.260ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
14563 Feb 21 06:48:38.260 INFO {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:48:38.260651271Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
14564 request completed, latency_us: 148, response_code: 200, uri: /state/cid/2, method: GET, req_id: 15eb3635-5efa-43ce-8d50-58f8cadab0d8, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
14565 [8830] Exited with: ExitStatus(unix_wait_status(9))
14566 [8830][2] reports Exit
14567 Feb 21 06:48:42.263 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/2, method: GET, req_id: f516d016-6c3d-4ad9-8868-95c4825b8318, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
14568 001/300 Write block 4421 len 20480 data: 2 2 2 2 2
14569 002/300 Read block 938 len 8192
14570 003/300 Write block 524 len 8192 data: 4 4
14571 004/300 Read block 987 len 8192
14572 005/300 Read block 3771 len 12288
14573 006/300 Read block 3047 len 40960
14574 007/300 Read block 1226 len 4096
14575 008/300 Read block 93 len 24576
14576 009/300 Write block 347 len 4096 data: 2
14577 010/300 Write block 3906 len 4096 data: 4
14578 011/300 Read block 3409 len 24576
14579 012/300 Read block 4669 len 16384
14580 013/300 Read block 3443 len 24576
14581 014/300 Write block 1259 len 4096 data: 4
14582 015/300 Write block 1638 len 12288 data: 2 2 2
14583 016/300 Write block 3254 len 4096 data: 3
14584 017/300 Write block 738 len 28672 data: 5 5 4 4 3 2 2
14585 018/300 Read block 2837 len 8192
14586 019/300 Read block 2768 len 16384
14587 020/300 Read block 4021 len 40960
14588 021/300 Read block 2478 len 4096
14589 022/300 Read block 4381 len 36864
14590 023/300 Write block 3306 len 4096 data: 3
14591 024/300 Read block 3703 len 12288
14592 025/300 Read block 4299 len 28672
14593 026/300 Read block 705 len 36864
14594 027/300 Read block 1969 len 24576
14595 028/300 Write block 2392 len 12288 data: 5 4 3
14596 029/300 Read block 914 len 40960
14597 030/300 Read block 230 len 4096
14598 031/300 Read block 1901 len 20480
14599 032/300 Read block 976 len 40960
14600 033/300 Flush
14601 034/300 Write block 2038 len 4096 data: 5
14602 035/300 Read block 2411 len 12288
14603 036/300 Read block 4729 len 40960
14604 037/300 Read block 3878 len 12288
14605 038/300 Write block 4500 len 8192 data: 2 2
14606 039/300 Read block 3046 len 20480
14607 040/300 Read block 3641 len 40960
14608 041/300 Read block 1915 len 40960
14609 042/300 Read block 4548 len 32768
14610 043/300 Read block 2030 len 36864
14611 044/300 Flush
14612 045/300 Write block 2848 len 16384 data: 3 4 5 4
14613 046/300 Flush
14614 047/300 Read block 1954 len 28672
14615 048/300 Read block 1367 len 12288
14616 049/300 Write block 1356 len 16384 data: 2 2 2 2
14617 050/300 Write block 986 len 12288 data: 4 4 4
14618 051/300 Write block 2519 len 32768 data: 3 3 3 2 2 3 3 2
14619 052/300 Write block 4725 len 36864 data: 2 2 2 4 4 5 4 4 4
14620 053/300 Read block 1701 len 8192
14621 054/300 Read block 2819 len 20480
14622 055/300 Write block 214 len 20480 data: 3 4 5 5 5
14623 056/300 Write block 3155 len 32768 data: 2 2 2 2 2 3 4 4
14624 057/300 Read block 2904 len 28672
14625 058/300 Write block 2342 len 4096 data: 4
14626 059/300 Read block 4091 len 32768
14627 060/300 Read block 3658 len 24576
14628 061/300 Write block 2850 len 40960 data: 6 5 4 3 4 4 5 7 5 5
14629 062/300 Write block 2353 len 12288 data: 4 4 3
14630 063/300 Read block 4109 len 12288
14631 064/300 Read block 3600 len 12288
14632 065/300 Write block 1424 len 20480 data: 7 5 7 5 3
14633 066/300 Flush
14634 067/300 Write block 3952 len 24576 data: 3 3 2 3 3 3
14635 068/300 Read block 2383 len 4096
14636 069/300 Read block 660 len 36864
14637 070/300 Write block 4968 len 24576 data: 3 3 5 5 3 4
14638 071/300 Read block 98 len 40960
14639 072/300 Flush
14640 073/300 Write block 4915 len 36864 data: 6 8 10 8 8 7 5 5 5
14641 074/300 Read block 4086 len 28672
14642 075/300 Write block 4215 len 36864 data: 3 4 4 3 3 3 4 3 4
14643 076/300 Write block 1825 len 16384 data: 5 5 4 5
14644 077/300 Read block 2156 len 36864
14645 078/300 Write block 1462 len 36864 data: 2 3 3 3 3 3 3 2 2
14646 079/300 Write block 1125 len 36864 data: 7 8 8 8 7 7 5 5 4
14647 080/300 Write block 4729 len 20480 data: 5 6 5 5 5
14648 081/300 Read block 2199 len 16384
14649 082/300 Read block 2586 len 12288
14650 083/300 Read block 2724 len 8192
14651 084/300 Read block 4879 len 12288
14652 085/300 Read block 4154 len 20480
14653 086/300 Write block 2505 len 32768 data: 4 4 5 4 5 4 3 2
14654 087/300 Write block 3681 len 4096 data: 2
14655 088/300 Read block 3171 len 12288
14656 089/300 Write block 1746 len 20480 data: 4 5 6 5 4
14657 090/300 Read block 3868 len 12288
14658 091/300 Write block 4052 len 28672 data: 2 3 4 5 5 5 5
14659 092/300 Read block 3533 len 4096
14660 093/300 Read block 2030 len 4096
14661 094/300 Read block 4196 len 40960
14662 095/300 Flush
14663 096/300 Read block 4771 len 24576
14664 097/300 Read block 3111 len 32768
14665 098/300 Read block 2956 len 4096
14666 099/300 Read block 3670 len 24576
14667 100/300 Write block 3436 len 40960 data: 3 2 3 3 3 3 3 3 3 4
14668 101/300 Flush
14669 102/300 Read block 4180 len 36864
14670 103/300 Write block 2191 len 4096 data: 6
14671 104/300 Read block 4330 len 20480
14672 105/300 Read block 4121 len 36864
14673 106/300 Read block 680 len 8192
14674 107/300 Write block 970 len 40960 data: 4 3 5 5 5 6 7 5 4 3
14675 108/300 Read block 1339 len 12288
14676 109/300 Write block 428 len 28672 data: 5 5 5 6 6 5 3
14677 110/300 Read block 1268 len 36864
14678 111/300 Write block 3823 len 36864 data: 5 3 3 4 3 3 3 5 5
14679 112/300 Write block 1006 len 24576 data: 3 3 3 4 3 3
14680 113/300 Write block 415 len 12288 data: 2 2 2
14681 114/300 Flush
14682 115/300 Read block 3763 len 24576
14683 116/300 Write block 2853 len 36864 data: 4 5 5 6 8 6 6 5 3
14684 117/300 Read block 4966 len 16384
14685 118/300 Read block 1880 len 36864
14686 119/300 Read block 4458 len 16384
14687 120/300 Read block 1300 len 4096
14688 121/300 Read block 847 len 16384
14689 122/300 Read block 4880 len 28672
14690 123/300 Flush
14691 124/300 Read block 4169 len 16384
14692 125/300 Read block 3140 len 8192
14693 126/300 Write block 1195 len 28672 data: 3 3 3 3 4 3 3
14694 127/300 Read block 1998 len 32768
14695 128/300 Read block 1204 len 12288
14696 129/300 Read block 3055 len 12288
14697 130/300 Write block 4073 len 8192 data: 5 4
14698 131/300 Flush
14699 132/300 Read block 152 len 16384
14700 133/300 Read block 970 len 12288
14701 134/300 Write block 3494 len 12288 data: 3 3 5
14702 135/300 Flush
14703 136/300 Read block 3329 len 16384
14704 137/300 Write block 3248 len 12288 data: 6 7 6
14705 138/300 Read block 860 len 16384
14706 139/300 Read block 1416 len 40960
14707 140/300 Read block 591 len 8192
14708 141/300 Read block 2287 len 16384
14709 142/300 Write block 870 len 16384 data: 3 4 4 4
14710 143/300 Flush
14711 144/300 Write block 4913 len 36864 data: 3 4 7 9 11 9 9 8 6
14712 145/300 Write block 2662 len 32768 data: 4 4 3 3 2 2 5 5
14713 146/300 Read block 1806 len 20480
14714 147/300 Read block 225 len 40960
14715 148/300 Read block 375 len 8192
14716 149/300 Read block 4524 len 24576
14717 150/300 Write block 4649 len 24576 data: 6 6 6 6 4 4
14718 151/300 Read block 3910 len 28672
14719 152/300 Write block 543 len 24576 data: 4 4 5 3 3 3
14720 153/300 Flush
14721 154/300 Read block 4487 len 28672
14722 155/300 Read block 836 len 12288
14723 156/300 Read block 4648 len 24576
14724 157/300 Read block 512 len 12288
14725 158/300 Flush
14726 159/300 Read block 2748 len 40960
14727 160/300 Read block 3981 len 12288
14728 161/300 Write block 3489 len 12288 data: 3 4 4
14729 162/300 Read block 430 len 28672
14730 163/300 Read block 2716 len 4096
14731 164/300 Read block 299 len 36864
14732 165/300 Write block 3312 len 32768 data: 2 2 2 2 3 3 3 4
14733 166/300 Write block 445 len 24576 data: 5 4 3 3 3 3
14734 167/300 Write block 4908 len 24576 data: 3 3 3 3 2 4
14735 168/300 Write block 2199 len 24576 data: 3 3 4 3 3 3
14736 169/300 Read block 1599 len 32768
14737 170/300 Flush
14738 171/300 Read block 4800 len 12288
14739 172/300 Read block 796 len 16384
14740 173/300 Write block 1871 len 16384 data: 4 3 3 3
14741 174/300 Read block 655 len 28672
14742 175/300 Write block 4283 len 8192 data: 3 3
14743 176/300 Flush
14744 177/300 Read block 4722 len 32768
14745 178/300 Write block 3579 len 32768 data: 5 4 5 7 7 9 10 9
14746 179/300 Read block 3391 len 16384
14747 180/300 Read block 2521 len 4096
14748 181/300 Write block 2328 len 12288 data: 2 2 2
14749 182/300 Write block 4176 len 16384 data: 2 2 2 2
14750 183/300 Write block 3345 len 40960 data: 3 3 5 6 5 5 5 4 4 5
14751 184/300 Write block 2336 len 12288 data: 3 3 2
14752 185/300 Read block 4343 len 28672
14753 186/300 Write block 4068 len 16384 data: 2 2 2 2
14754 187/300 Read block 1988 len 36864
14755 188/300 Write block 909 len 36864 data: 3 3 2 2 2 2 3 2 2
14756 189/300 Read block 4280 len 36864
14757 190/300 Read block 758 len 12288
14758 191/300 Write block 1453 len 36864 data: 3 3 3 3 3 3 3 3 3
14759 192/300 Write block 737 len 20480 data: 5 6 6 5 5
14760 193/300 Write block 3475 len 12288 data: 3 3 3
14761 194/300 Write block 2745 len 4096 data: 4
14762 195/300 Write block 3581 len 4096 data: 6
14763 196/300 Read block 4177 len 40960
14764 197/300 Read block 4201 len 8192
14765 198/300 Read block 3445 len 40960
14766 199/300 Read block 4519 len 16384
14767 200/300 Read block 4360 len 8192
14768 201/300 Read block 2288 len 8192
14769 202/300 Write block 210 len 36864 data: 4 5 4 4 4 5 6 6 6
14770 203/300 Read block 4818 len 4096
14771 204/300 Write block 2491 len 20480 data: 2 2 2 2 4
14772 205/300 Read block 4729 len 12288
14773 206/300 Flush
14774 207/300 Read block 3068 len 16384
14775 208/300 Flush
14776 209/300 Write block 463 len 16384 data: 3 2 2 2
14777 210/300 Read block 2930 len 20480
14778 211/300 Read block 1214 len 20480
14779 212/300 Read block 3883 len 8192
14780 213/300 Read block 3827 len 12288
14781 214/300 Write block 4193 len 40960 data: 4 2 2 2 3 3 3 2 3 3
14782 215/300 Write block 1951 len 32768 data: 3 4 3 3 4 4 3 3
14783 216/300 Write block 4710 len 40960 data: 4 4 4 4 4 3 3 3 3 3
14784 217/300 Flush
14785 218/300 Read block 3845 len 16384
14786 219/300 Read block 3730 len 8192
14787 220/300 Read block 1506 len 36864
14788 221/300 Read block 55 len 12288
14789 222/300 Flush
14790 223/300 Flush
14791 224/300 Read block 2543 len 16384
14792 225/300 Write block 3405 len 8192 data: 3 3
14793 226/300 Write block 2486 len 12288 data: 3 3 3
14794 227/300 Write block 3652 len 12288 data: 3 3 2
14795 228/300 Read block 527 len 24576
14796 229/300 Read block 3219 len 40960
14797 230/300 Write block 1164 len 36864 data: 2 2 3 3 3 4 6 5 5
14798 231/300 Read block 1839 len 36864
14799 232/300 Write block 813 len 24576 data: 3 4 3 3 3 3
14800 233/300 Write block 2575 len 32768 data: 6 6 5 5 5 4 2 3
14801 234/300 Write block 843 len 28672 data: 3 3 2 2 3 3 3
14802 235/300 Write block 3048 len 36864 data: 4 3 3 2 2 2 2 3 3
14803 236/300 Write block 3935 len 8192 data: 4 5
14804 237/300 Read block 315 len 8192
14805 238/300 Write block 427 len 28672 data: 5 6 6 6 7 7 6
14806 239/300 Read block 3007 len 4096
14807 240/300 Write block 3415 len 40960 data: 5 5 6 4 5 5 4 3 3 3
14808 241/300 Read block 3859 len 4096
14809 242/300 Read block 2451 len 40960
14810 243/300 Read block 938 len 32768
14811 244/300 Read block 4103 len 28672
14812 245/300 Read block 185 len 24576
14813 246/300 Write block 4950 len 12288 data: 3 3 3
14814 247/300 Write block 2473 len 4096 data: 5
14815 248/300 Write block 381 len 24576 data: 6 6 5 4 4 4
14816 249/300 Read block 4528 len 20480
14817 250/300 Read block 379 len 28672
14818 251/300 Write block 1008 len 16384 data: 4 5 4 4
14819 252/300 Read block 3052 len 24576
14820 253/300 Write block 1658 len 16384 data: 4 4 2 3
14821 254/300 Flush
14822 255/300 Flush
14823 256/300 Write block 212 len 36864 data: 5 5 5 6 7 7 7 4 3
14824 257/300 Read block 2279 len 32768
14825 258/300 Write block 1762 len 32768 data: 4 4 3 3 3 3 4 3
14826 259/300 Write block 3371 len 28672 data: 2 2 2 2 2 2 2
14827 260/300 Flush
14828 261/300 Read block 1973 len 4096
14829 262/300 Write block 1975 len 28672 data: 2 3 3 4 4 4 5
14830 263/300 Read block 878 len 16384
14831 264/300 Write block 103 len 40960 data: 2 2 3 3 3 3 3 3 3 3
14832 265/300 Read block 3557 len 36864
14833 266/300 Write block 677 len 20480 data: 6 5 4 3 3
14834 267/300 Write block 774 len 28672 data: 3 3 4 4 3 3 3
14835 268/300 Write block 2574 len 8192 data: 7 7
14836 269/300 Write block 580 len 16384 data: 3 3 3 3
14837 270/300 Read block 4521 len 4096
14838 271/300 Read block 871 len 28672
14839 272/300 Read block 2650 len 32768
14840 273/300 Write block 55 len 28672 data: 3 3 3 3 3 2 2
14841 274/300 Read block 1982 len 32768
14842 275/300 Read block 1173 len 8192
14843 276/300 Write block 3547 len 8192 data: 6 6
14844 277/300 Write block 1611 len 4096 data: 9
14845 278/300 Write block 460 len 28672 data: 5 4 3 4 3 3 3
14846 279/300 Write block 1031 len 32768 data: 4 5 4 3 3 3 2 2
14847 280/300 Write block 4353 len 36864 data: 3 3 4 4 5 4 4 4 2
14848 281/300 Read block 3459 len 28672
14849 282/300 Read block 3443 len 4096
14850 283/300 Write block 995 len 20480 data: 2 2 2 3 3
14851 284/300 Flush
14852 285/300 Flush
14853 286/300 Write block 834 len 28672 data: 2 2 2 2 2 2 2
14854 287/300 Write block 3187 len 24576 data: 2 3 3 2 2 2
14855 288/300 Write block 4355 len 16384 data: 5 5 6 5
14856 289/300 Write block 1714 len 12288 data: 3 3 3
14857 290/300 Read block 4990 len 40960
14858 291/300 Read block 4441 len 16384
14859 292/300 Flush
14860 293/300 Read block 540 len 4096
14861 294/300 Write block 1666 len 12288 data: 3 3 3
14862 295/300 Write block 2104 len 12288 data: 2 3 3
14863 296/300 Write block 1805 len 28672 data: 4 3 3 3 3 4 7
14864 297/300 Write block 3257 len 8192 data: 2 2
14865 298/300 Write block 3337 len 12288 data: 4 5 4
14866 299/300 Write block 1451 len 24576 data: 4 3 4 4 4 4
14867 300/300 Read block 1207 len 40960
14868 301/300 Flush
14869 Main task has work to do, go find it
14870 got dsc Start(2)
14871 start 2
14872 Feb 21 06:48:42.502 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/2, method: POST, req_id: f25e1365-3d89-495c-a53e-2aba2b263152, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
14873 Replay: started 2, returned:Ok(())
14874 ----------------------------------------------------------------
14875 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14876 States: Active Active Offline
14877 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14878 4624 Acked Write 5 Done Done Sent false
14879 4625 Acked Read 2 Done Done Sent false
14880 4626 Acked Write 2 Done Done Sent false
14881 4627 Acked Read 2 Done Done Sent false
14882 4628 Acked Read 3 Done Done Sent false
14883 4629 Acked Read 10 Done Done Sent false
14884 4630 Acked Read 1 Done Done Sent false
14885 4631 Acked Read 6 Done Done Sent false
14886 4632 Acked Write 1 Done Done Sent false
14887 4633 Acked Write 1 Done Done Sent false
14888 4634 Acked Read 6 Done Done Sent false
14889 4635 Acked Read 4 Done Done Sent false
14890 4636 Acked Read 6 Done Done Sent false
14891 4637 Acked Write 1 Done Done Sent false
14892 4638 Acked Write 3 Done Done Sent false
14893 4639 Acked Write 1 Done Done Sent false
14894 4640 Acked Write 7 Done Done Sent false
14895 4641 Acked Read 2 Done Done Sent false
14896 4642 Acked Read 4 Done Done Sent false
14897 4643 Acked Read 10 Done Done Sent false
14898 4644 Acked Read 1 Done Done Sent false
14899 4645 Acked Read 9 Done Done Sent false
14900 4646 Acked Write 1 Done Done Sent false
14901 4647 Acked Read 3 Done Done Sent false
14902 4648 Acked Read 7 Done Done Sent false
14903 4649 Acked Read 9 Done Done Sent false
14904 4650 Acked Read 6 Done Done Sent false
14905 4651 Acked Write 3 Done Done Sent false
14906 4652 Acked Read 10 Done Done Sent false
14907 4653 Acked Read 1 Done Done Sent false
14908 4654 Acked Read 5 Done Done Sent false
14909 4655 Acked Read 10 Done Done Sent false
14910 4656 Acked Flush 0 Done Done Sent false
14911 4657 Acked Write 1 Done Done Sent false
14912 4658 Acked Read 3 Done Done Sent false
14913 4659 Acked Read 10 Done Done Sent false
14914 4660 Acked Read 3 Done Done Sent false
14915 4661 Acked Write 2 Done Done Sent false
14916 4662 Acked Read 5 Done Done Sent false
14917 4663 Acked Read 10 Done Done Sent false
14918 4664 Acked Read 10 Done Done Sent false
14919 4665 Acked Read 8 Done Done Sent false
14920 4666 Acked Read 9 Done Done Sent false
14921 4667 Acked Flush 0 Done Done Sent false
14922 4668 Acked Write 4 Done Done Sent false
14923 4669 Acked Flush 0 Done Done Sent false
14924 4670 Acked Read 7 Done Done Sent false
14925 4671 Acked Read 3 Done Done Sent false
14926 4672 Acked Write 4 Done Done Sent false
14927 4673 Acked Write 3 Done Done Sent false
14928 4674 Acked Write 8 Done Done Sent false
14929 4675 Acked Write 9 Done Done Sent false
14930 4676 Acked Read 2 Done Done Sent false
14931 4677 Acked Read 5 Done Done Sent false
14932 4678 Acked Write 5 Done Done Sent false
14933 4679 Acked Write 8 Done Done Sent false
14934 4680 Acked Read 7 Done Done Sent false
14935 4681 Acked Write 1 Done Done Sent false
14936 4682 Acked Read 8 Done Done Sent false
14937 4683 Acked Read 6 Done Done Sent false
14938 4684 Acked Write 10 Done Done Sent false
14939 4685 Acked Write 3 Done Done Sent false
14940 4686 Acked Read 3 Done Done Sent false
14941 4687 Acked Read 3 Done Done Sent false
14942 4688 Acked Write 5 Done Done Sent false
14943 4689 Acked Flush 0 Done Done Sent false
14944 4690 Acked Write 6 Done Done Sent false
14945 4691 Acked Read 1 Done Done Sent false
14946 4692 Acked Read 9 Done Done Sent false
14947 4693 Acked Write 6 Done Done Sent false
14948 4694 Acked Read 10 Done Done Sent false
14949 4695 Acked Flush 0 Done Done Sent false
14950 4696 Acked Write 9 Done Done Sent false
14951 4697 Acked Read 7 Done Done Sent false
14952 4698 Acked Write 9 Done Done Sent false
14953 4699 Acked Write 4 Done Done Sent false
14954 4700 Acked Read 9 Done Done Sent false
14955 4701 Acked Write 9 Done Done Sent false
14956 4702 Acked Write 9 Done Done Sent false
14957 4703 Acked Write 5 Done Done Sent false
14958 4704 Acked Read 4 Done Done Sent false
14959 4705 Acked Read 3 Done Done Sent false
14960 4706 Acked Read 2 Done Done Sent false
14961 4707 Acked Read 3 Done Done Sent false
14962 4708 Acked Read 5 Done Done Sent false
14963 4709 Acked Write 8 Done Done Sent false
14964 4710 Acked Write 1 Done Done Sent false
14965 4711 Acked Read 3 Done Done Sent false
14966 4712 Acked Write 5 Done Done Sent false
14967 4713 Acked Read 3 Done Done Sent false
14968 4714 Acked Write 7 Done Done Sent false
14969 4715 Acked Read 1 Done Done Sent false
14970 4716 Acked Read 1 Done Done Sent false
14971 4717 Acked Read 10 Done Done Sent false
14972 4718 Acked Flush 0 Done Done Sent false
14973 4719 Acked Read 6 Done Done Sent false
14974 4720 Acked Read 8 Done Done Sent false
14975 4721 Acked Read 1 Done Done Sent false
14976 4722 Acked Read 6 Done Done Sent false
14977 4723 Acked Write 10 Done Done Sent false
14978 4724 Acked Flush 0 Done Done Sent false
14979 4725 Acked Read 9 Done Done Sent false
14980 4726 Acked Write 1 Done Done Sent false
14981 4727 Acked Read 5 Done Done Sent false
14982 4728 Acked Read 9 Done Done Sent false
14983 4729 Acked Read 2 Done Done Sent false
14984 4730 Acked Write 10 Done Done Sent false
14985 4731 Acked Read 3 Done Done Sent false
14986 4732 Acked Write 7 Done Done Sent false
14987 4733 Acked Read 9 Done Done Sent false
14988 4734 Acked Write 9 Done Done Sent false
14989 4735 Acked Write 6 Done Done Sent false
14990 4736 Acked Write 3 Done Done Sent false
14991 4737 Acked Flush 0 Done Done Sent false
14992 4738 Acked Read 6 Done Done Sent false
14993 4739 Acked Write 9 Done Done Sent false
14994 4740 Acked Read 4 Done Done Sent false
14995 4741 Acked Read 9 Done Done Sent false
14996 4742 Acked Read 4 Done Done Sent false
14997 4743 Acked Read 1 Done Done Sent false
14998 4744 Acked Read 4 Done Done Sent false
14999 4745 Acked Read 7 Done Done Sent false
15000 4746 Acked Flush 0 Done Done Sent false
15001 4747 Acked Read 4 Done Done Sent false
15002 4748 Acked Read 2 Done Done Sent false
15003 4749 Acked Write 7 Done Done Sent false
15004 4750 Acked Read 8 Done Done Sent false
15005 4751 Acked Read 3 Done Done Sent false
15006 4752 Acked Read 3 Done Done Sent false
15007 4753 Acked Write 2 Done Done Sent false
15008 4754 Acked Flush 0 Done Done Sent false
15009 4755 Acked Read 4 Done Done Sent false
15010 4756 Acked Read 3 Done Done Sent false
15011 4757 Acked Write 3 Done Done Sent false
15012 4758 Acked Flush 0 Done Done Sent false
15013 4759 Acked Read 4 Done Done Sent false
15014 4760 Acked Write 3 Done Done Sent false
15015 4761 Acked Read 4 Done Done Sent false
15016 4762 Acked Read 10 Done Done Sent false
15017 4763 Acked Read 2 Done Done Sent false
15018 4764 Acked Read 4 Done Done Sent false
15019 4765 Acked Write 4 Done Done Sent false
15020 4766 Acked Flush 0 Done Done Sent false
15021 4767 Acked Write 9 Done Done Sent false
15022 4768 Acked Write 8 Done Done Sent false
15023 4769 Acked Read 5 Done Done Sent false
15024 4770 Acked Read 10 Done Done Sent false
15025 4771 Acked Read 2 Done Done Sent false
15026 4772 Acked Read 6 Done Done Sent false
15027 4773 Acked Write 6 Done Done Sent false
15028 4774 Acked Read 7 Done Done Sent false
15029 4775 Acked Write 6 Done Done Sent false
15030 4776 Acked Flush 0 Done Done Sent false
15031 4777 Acked Read 7 Done Done Sent false
15032 4778 Acked Read 3 Done Done Sent false
15033 4779 Acked Read 6 Done Done Sent false
15034 4780 Acked Read 3 Done Done Sent false
15035 4781 Acked Flush 0 Done Done Sent false
15036 4782 Acked Read 10 Done Done Sent false
15037 4783 Acked Read 3 Done Done Sent false
15038 4784 Acked Write 3 Done Done Sent false
15039 4785 Acked Read 7 Done Done Sent false
15040 4786 Acked Read 1 Done Done Sent false
15041 4787 Acked Read 9 Done Done Sent false
15042 4788 Acked Write 8 Done Done Sent false
15043 4789 Acked Write 6 Done Done Sent false
15044 4790 Acked Write 6 Done Done Sent false
15045 4791 Acked Write 6 Done Done Sent false
15046 4792 Acked Read 8 Done Done Sent false
15047 4793 Acked Flush 0 Done Done Sent false
15048 4794 Acked Read 3 Done Done Sent false
15049 4795 Acked Read 4 Done Done Sent false
15050 4796 Acked Write 4 Done Done Sent false
15051 4797 Acked Read 7 Done Done Sent false
15052 4798 Acked Write 2 Done Done Sent false
15053 4799 Acked Flush 0 Done Done Sent false
15054 4800 Acked Read 8 Done Done Sent false
15055 4801 Acked Write 8 Done Done Sent false
15056 4802 Acked Read 4 Done Done Sent false
15057 4803 Acked Read 1 Done Done Sent false
15058 4804 Acked Write 3 Done Done Sent false
15059 4805 Acked Write 4 Done Done Sent false
15060 4806 Acked Write 10 Done Done Sent false
15061 4807 Acked Write 3 Done Done Sent false
15062 4808 Acked Read 7 Done Done Sent false
15063 4809 Acked Write 4 Done Done Sent false
15064 4810 Acked Read 9 Done Done Sent false
15065 4811 Acked Write 9 Done Done Sent false
15066 4812 Acked Read 9 Done Done Sent false
15067 4813 Acked Read 3 Done Done Sent false
15068 4814 Acked Write 9 Done Done Sent false
15069 4815 Acked Write 5 Done Done Sent false
15070 4816 Acked Write 3 Done Done Sent false
15071 4817 Acked Write 1 Done Done Sent false
15072 4818 Acked Write 1 Done Done Sent false
15073 4819 Acked Read 10 Done Done Sent false
15074 4820 Acked Read 2 Done Done Sent false
15075 4821 Acked Read 10 Done Done Sent false
15076 4822 Acked Read 4 Done Done Sent false
15077 4823 Acked Read 2 Done Done Sent false
15078 4824 Acked Read 2 Done Done Sent false
15079 4825 Acked Write 9 Done Done Sent false
15080 4826 Acked Read 1 Done Done Sent false
15081 4827 Acked Write 5 Done Done Sent false
15082 4828 Acked Read 3 Done Done Sent false
15083 4829 Acked Flush 0 Done Done Sent false
15084 4830 Acked Read 4 Done Done Sent false
15085 4831 Acked Flush 0 Done Done Sent false
15086 4832 Acked Write 4 Done Done Sent false
15087 4833 Acked Read 5 Done Done Sent false
15088 4834 Acked Read 5 Done Done Sent false
15089 4835 Acked Read 2 Done Done Sent false
15090 4836 Acked Read 3 Done Done Sent false
15091 4837 Acked Write 10 Done Done Sent false
15092 4838 Acked Write 8 Done Done Sent false
15093 4839 Acked Write 10 Done Done Sent false
15094 4840 Acked Flush 0 Done Done Sent false
15095 4841 Acked Read 4 Done Done Sent false
15096 4842 Acked Read 2 Done Done Sent false
15097 4843 Acked Read 9 Done Done Sent false
15098 4844 Acked Read 3 Done Done Sent false
15099 4845 Acked Flush 0 Done Done Sent false
15100 4846 Acked Flush 0 Done Done Sent false
15101 4847 Acked Read 4 Done Done Sent false
15102 4848 Acked Write 2 Done Done Sent false
15103 4849 Acked Write 3 Done Done Sent false
15104 4850 Acked Write 3 Done Done Sent false
15105 4851 Acked Read 6 Done Done Sent false
15106 4852 Acked Read 10 Done Done Sent false
15107 4853 Acked Write 9 Done Done Sent false
15108 4854 Acked Read 9 Done Done Sent false
15109 4855 Acked Write 6 Done Done Sent false
15110 4856 Acked Write 8 Done Done Sent false
15111 4857 Acked Write 7 Done Done Sent false
15112 4858 Acked Write 9 Done Done Sent false
15113 4859 Acked Write 2 Done Done Sent false
15114 4860 Acked Read 2 Done Done Sent false
15115 4861 Acked Write 7 Done Done Sent false
15116 4862 Acked Read 1 Done Done Sent false
15117 4863 Acked Write 10 Done Done Sent false
15118 4864 Acked Read 1 Done Done Sent false
15119 4865 Acked Read 10 Done Done Sent false
15120 4866 Acked Read 8 Done Done Sent false
15121 4867 Acked Read 7 Done Done Sent false
15122 4868 Acked Read 6 Done Done Sent false
15123 4869 Acked Write 3 Done Done Sent false
15124 4870 Acked Write 1 Done Done Sent false
15125 4871 Acked Write 6 Done Done Sent false
15126 4872 Acked Read 5 Done Done Sent false
15127 4873 Acked Read 7 Done Done Sent false
15128 4874 Acked Write 4 Done Done Sent false
15129 4875 Acked Read 6 Done Done Sent false
15130 4876 Acked Write 4 Done Done Sent false
15131 4877 Acked Flush 0 Done Done Sent false
15132 4878 Acked Flush 0 Done Done Sent false
15133 4879 Acked Write 9 Done Done Sent false
15134 4880 Acked Read 8 Done Done Sent false
15135 4881 Acked Write 8 Done Done Sent false
15136 4882 Acked Write 7 Done Done Sent false
15137 4883 Acked Flush 0 Done Done Sent false
15138 4884 Acked Read 1 Done Done Sent false
15139 4885 Acked Write 7 Done Done Sent false
15140 4886 Acked Read 4 Done Done Sent false
15141 4887 Acked Write 10 Done Done Sent false
15142 4888 Acked Read 9 Done Done Sent false
15143 4889 Acked Write 5 Done Done Sent false
15144 4890 Acked Write 7 Done Done Sent false
15145 4891 Acked Write 2 Done Done Sent false
15146 4892 Acked Write 4 Done Done Sent false
15147 4893 Acked Read 1 Done Done Sent false
15148 4894 Acked Read 7 Done Done Sent false
15149 4895 Acked Read 8 Done Done Sent false
15150 4896 Acked Write 7 Done Done Sent false
15151 4897 Acked Read 8 Done Done Sent false
15152 4898 Acked Read 2 Done Done Sent false
15153 4899 Acked Write 2 Done Done Sent false
15154 4900 Acked Write 1 Done Done Sent false
15155 4901 Acked Write 7 Done Done Sent false
15156 4902 Acked Write 8 Done Done Sent false
15157 4903 Acked Write 9 Done Done Sent false
15158 4904 Acked Read 7 Done Done Sent false
15159 4905 Acked Read 1 Done Done Sent false
15160 4906 Acked Write 5 Done Done Sent false
15161 4907 Acked Flush 0 Done Done Sent false
15162 4908 Acked Flush 0 Done Done Sent false
15163 4909 Acked Write 7 Done Done Sent false
15164 4910 Acked Write 6 Done Done Sent false
15165 4911 Acked Write 4 Done Done Sent false
15166 4912 Acked Write 3 Done Done Sent false
15167 4913 Acked Read 10 Done Done Sent false
15168 4914 Acked Read 4 Done Done Sent false
15169 4915 Acked Flush 0 Done Done Sent false
15170 4916 Acked Read 1 Done Done Sent false
15171 4917 Acked Write 3 Done Done Sent false
15172 4918 Acked Write 3 Done Done Sent false
15173 4919 Acked Write 7 Done Done Sent false
15174 4920 Acked Write 2 Done Done Sent false
15175 4921 Acked Write 3 Done Done Sent false
15176 4922 Acked Write 6 Done Done Sent false
15177 4923 Acked Read 10 Done Done Sent false
15178 4924 Acked Flush 0 Done Done Sent false
15179 STATES DS:0 DS:1 DS:2 TOTAL
15180 Sent 0 0 301 301
15181 Done 301 301 0 602
15182 Skipped 0 0 0 0
15183 Error 0 0 0 0
15184 Last Flush: Some(JobId(4924)) Some(JobId(4924)) Some(JobId(4623))
15185 Downstairs last five completed: 4623 4622 4621 4620 4619
15186 Upstairs last five completed: 4924 4923 4922 4921 4920
15187 CLIENT: Up:0 ds:301 act:2
151882025-02-21T06:48:43.004ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
151892025-02-21T06:48:43.966ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
15190 [8830] got start action
15191 [8830] I am going to restart
15192 Starting downstairs at port 8830
15193 Make output file at "/tmp/dsc/downstairs-8830.txt"
15194 [8830][2] reports Starting
15195 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1219)
15196 [8830][2] reports Running
151972025-02-21T06:48:44.967ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
151982025-02-21T06:48:45.968ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
15199 ----------------------------------------------------------------
15200 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15201 States: Active Active Offline
15202 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15203 4624 Acked Write 5 Done Done Sent false
15204 4625 Acked Read 2 Done Done Sent false
15205 4626 Acked Write 2 Done Done Sent false
15206 4627 Acked Read 2 Done Done Sent false
15207 4628 Acked Read 3 Done Done Sent false
15208 4629 Acked Read 10 Done Done Sent false
15209 4630 Acked Read 1 Done Done Sent false
15210 4631 Acked Read 6 Done Done Sent false
15211 4632 Acked Write 1 Done Done Sent false
15212 4633 Acked Write 1 Done Done Sent false
15213 4634 Acked Read 6 Done Done Sent false
15214 4635 Acked Read 4 Done Done Sent false
15215 4636 Acked Read 6 Done Done Sent false
15216 4637 Acked Write 1 Done Done Sent false
15217 4638 Acked Write 3 Done Done Sent false
15218 4639 Acked Write 1 Done Done Sent false
15219 4640 Acked Write 7 Done Done Sent false
15220 4641 Acked Read 2 Done Done Sent false
15221 4642 Acked Read 4 Done Done Sent false
15222 4643 Acked Read 10 Done Done Sent false
15223 4644 Acked Read 1 Done Done Sent false
15224 4645 Acked Read 9 Done Done Sent false
15225 4646 Acked Write 1 Done Done Sent false
15226 4647 Acked Read 3 Done Done Sent false
15227 4648 Acked Read 7 Done Done Sent false
15228 4649 Acked Read 9 Done Done Sent false
15229 4650 Acked Read 6 Done Done Sent false
15230 4651 Acked Write 3 Done Done Sent false
15231 4652 Acked Read 10 Done Done Sent false
15232 4653 Acked Read 1 Done Done Sent false
15233 4654 Acked Read 5 Done Done Sent false
15234 4655 Acked Read 10 Done Done Sent false
15235 4656 Acked Flush 0 Done Done Sent false
15236 4657 Acked Write 1 Done Done Sent false
15237 4658 Acked Read 3 Done Done Sent false
15238 4659 Acked Read 10 Done Done Sent false
15239 4660 Acked Read 3 Done Done Sent false
15240 4661 Acked Write 2 Done Done Sent false
15241 4662 Acked Read 5 Done Done Sent false
15242 4663 Acked Read 10 Done Done Sent false
15243 4664 Acked Read 10 Done Done Sent false
15244 4665 Acked Read 8 Done Done Sent false
15245 4666 Acked Read 9 Done Done Sent false
15246 4667 Acked Flush 0 Done Done Sent false
15247 4668 Acked Write 4 Done Done Sent false
15248 4669 Acked Flush 0 Done Done Sent false
15249 4670 Acked Read 7 Done Done Sent false
15250 4671 Acked Read 3 Done Done Sent false
15251 4672 Acked Write 4 Done Done Sent false
15252 4673 Acked Write 3 Done Done Sent false
15253 4674 Acked Write 8 Done Done Sent false
15254 4675 Acked Write 9 Done Done Sent false
15255 4676 Acked Read 2 Done Done Sent false
15256 4677 Acked Read 5 Done Done Sent false
15257 4678 Acked Write 5 Done Done Sent false
15258 4679 Acked Write 8 Done Done Sent false
15259 4680 Acked Read 7 Done Done Sent false
15260 4681 Acked Write 1 Done Done Sent false
15261 4682 Acked Read 8 Done Done Sent false
15262 4683 Acked Read 6 Done Done Sent false
15263 4684 Acked Write 10 Done Done Sent false
15264 4685 Acked Write 3 Done Done Sent false
15265 4686 Acked Read 3 Done Done Sent false
15266 4687 Acked Read 3 Done Done Sent false
15267 4688 Acked Write 5 Done Done Sent false
15268 4689 Acked Flush 0 Done Done Sent false
15269 4690 Acked Write 6 Done Done Sent false
15270 4691 Acked Read 1 Done Done Sent false
15271 4692 Acked Read 9 Done Done Sent false
15272 4693 Acked Write 6 Done Done Sent false
15273 4694 Acked Read 10 Done Done Sent false
15274 4695 Acked Flush 0 Done Done Sent false
15275 4696 Acked Write 9 Done Done Sent false
15276 4697 Acked Read 7 Done Done Sent false
15277 4698 Acked Write 9 Done Done Sent false
15278 4699 Acked Write 4 Done Done Sent false
15279 4700 Acked Read 9 Done Done Sent false
15280 4701 Acked Write 9 Done Done Sent false
15281 4702 Acked Write 9 Done Done Sent false
15282 4703 Acked Write 5 Done Done Sent false
15283 4704 Acked Read 4 Done Done Sent false
15284 4705 Acked Read 3 Done Done Sent false
15285 4706 Acked Read 2 Done Done Sent false
15286 4707 Acked Read 3 Done Done Sent false
15287 4708 Acked Read 5 Done Done Sent false
15288 4709 Acked Write 8 Done Done Sent false
15289 4710 Acked Write 1 Done Done Sent false
15290 4711 Acked Read 3 Done Done Sent false
15291 4712 Acked Write 5 Done Done Sent false
15292 4713 Acked Read 3 Done Done Sent false
15293 4714 Acked Write 7 Done Done Sent false
15294 4715 Acked Read 1 Done Done Sent false
15295 4716 Acked Read 1 Done Done Sent false
15296 4717 Acked Read 10 Done Done Sent false
15297 4718 Acked Flush 0 Done Done Sent false
15298 4719 Acked Read 6 Done Done Sent false
15299 4720 Acked Read 8 Done Done Sent false
15300 4721 Acked Read 1 Done Done Sent false
15301 4722 Acked Read 6 Done Done Sent false
15302 4723 Acked Write 10 Done Done Sent false
15303 4724 Acked Flush 0 Done Done Sent false
15304 4725 Acked Read 9 Done Done Sent false
15305 4726 Acked Write 1 Done Done Sent false
15306 4727 Acked Read 5 Done Done Sent false
15307 4728 Acked Read 9 Done Done Sent false
15308 4729 Acked Read 2 Done Done Sent false
15309 4730 Acked Write 10 Done Done Sent false
15310 4731 Acked Read 3 Done Done Sent false
15311 4732 Acked Write 7 Done Done Sent false
15312 4733 Acked Read 9 Done Done Sent false
15313 4734 Acked Write 9 Done Done Sent false
15314 4735 Acked Write 6 Done Done Sent false
15315 4736 Acked Write 3 Done Done Sent false
15316 4737 Acked Flush 0 Done Done Sent false
15317 4738 Acked Read 6 Done Done Sent false
15318 4739 Acked Write 9 Done Done Sent false
15319 4740 Acked Read 4 Done Done Sent false
15320 4741 Acked Read 9 Done Done Sent false
15321 4742 Acked Read 4 Done Done Sent false
15322 4743 Acked Read 1 Done Done Sent false
15323 4744 Acked Read 4 Done Done Sent false
15324 4745 Acked Read 7 Done Done Sent false
15325 4746 Acked Flush 0 Done Done Sent false
15326 4747 Acked Read 4 Done Done Sent false
15327 4748 Acked Read 2 Done Done Sent false
15328 4749 Acked Write 7 Done Done Sent false
15329 4750 Acked Read 8 Done Done Sent false
15330 4751 Acked Read 3 Done Done Sent false
15331 4752 Acked Read 3 Done Done Sent false
15332 4753 Acked Write 2 Done Done Sent false
15333 4754 Acked Flush 0 Done Done Sent false
15334 4755 Acked Read 4 Done Done Sent false
15335 4756 Acked Read 3 Done Done Sent false
15336 4757 Acked Write 3 Done Done Sent false
15337 4758 Acked Flush 0 Done Done Sent false
15338 4759 Acked Read 4 Done Done Sent false
15339 4760 Acked Write 3 Done Done Sent false
15340 4761 Acked Read 4 Done Done Sent false
15341 4762 Acked Read 10 Done Done Sent false
15342 4763 Acked Read 2 Done Done Sent false
15343 4764 Acked Read 4 Done Done Sent false
15344 4765 Acked Write 4 Done Done Sent false
15345 4766 Acked Flush 0 Done Done Sent false
15346 4767 Acked Write 9 Done Done Sent false
15347 4768 Acked Write 8 Done Done Sent false
15348 4769 Acked Read 5 Done Done Sent false
15349 4770 Acked Read 10 Done Done Sent false
15350 4771 Acked Read 2 Done Done Sent false
15351 4772 Acked Read 6 Done Done Sent false
15352 4773 Acked Write 6 Done Done Sent false
15353 4774 Acked Read 7 Done Done Sent false
15354 4775 Acked Write 6 Done Done Sent false
15355 4776 Acked Flush 0 Done Done Sent false
15356 4777 Acked Read 7 Done Done Sent false
15357 4778 Acked Read 3 Done Done Sent false
15358 4779 Acked Read 6 Done Done Sent false
15359 4780 Acked Read 3 Done Done Sent false
15360 4781 Acked Flush 0 Done Done Sent false
15361 4782 Acked Read 10 Done Done Sent false
15362 4783 Acked Read 3 Done Done Sent false
15363 4784 Acked Write 3 Done Done Sent false
15364 4785 Acked Read 7 Done Done Sent false
15365 4786 Acked Read 1 Done Done Sent false
15366 4787 Acked Read 9 Done Done Sent false
15367 4788 Acked Write 8 Done Done Sent false
15368 4789 Acked Write 6 Done Done Sent false
15369 4790 Acked Write 6 Done Done Sent false
15370 4791 Acked Write 6 Done Done Sent false
15371 4792 Acked Read 8 Done Done Sent false
15372 4793 Acked Flush 0 Done Done Sent false
15373 4794 Acked Read 3 Done Done Sent false
15374 4795 Acked Read 4 Done Done Sent false
15375 4796 Acked Write 4 Done Done Sent false
15376 4797 Acked Read 7 Done Done Sent false
15377 4798 Acked Write 2 Done Done Sent false
15378 4799 Acked Flush 0 Done Done Sent false
15379 4800 Acked Read 8 Done Done Sent false
15380 4801 Acked Write 8 Done Done Sent false
15381 4802 Acked Read 4 Done Done Sent false
15382 4803 Acked Read 1 Done Done Sent false
15383 4804 Acked Write 3 Done Done Sent false
15384 4805 Acked Write 4 Done Done Sent false
15385 4806 Acked Write 10 Done Done Sent false
15386 4807 Acked Write 3 Done Done Sent false
15387 4808 Acked Read 7 Done Done Sent false
15388 4809 Acked Write 4 Done Done Sent false
15389 4810 Acked Read 9 Done Done Sent false
15390 4811 Acked Write 9 Done Done Sent false
15391 4812 Acked Read 9 Done Done Sent false
15392 4813 Acked Read 3 Done Done Sent false
15393 4814 Acked Write 9 Done Done Sent false
15394 4815 Acked Write 5 Done Done Sent false
15395 4816 Acked Write 3 Done Done Sent false
15396 4817 Acked Write 1 Done Done Sent false
15397 4818 Acked Write 1 Done Done Sent false
15398 4819 Acked Read 10 Done Done Sent false
15399 4820 Acked Read 2 Done Done Sent false
15400 4821 Acked Read 10 Done Done Sent false
15401 4822 Acked Read 4 Done Done Sent false
15402 4823 Acked Read 2 Done Done Sent false
15403 4824 Acked Read 2 Done Done Sent false
15404 4825 Acked Write 9 Done Done Sent false
15405 4826 Acked Read 1 Done Done Sent false
15406 4827 Acked Write 5 Done Done Sent false
15407 4828 Acked Read 3 Done Done Sent false
15408 4829 Acked Flush 0 Done Done Sent false
15409 4830 Acked Read 4 Done Done Sent false
15410 4831 Acked Flush 0 Done Done Sent false
15411 4832 Acked Write 4 Done Done Sent false
15412 4833 Acked Read 5 Done Done Sent false
15413 4834 Acked Read 5 Done Done Sent false
15414 4835 Acked Read 2 Done Done Sent false
15415 4836 Acked Read 3 Done Done Sent false
15416 4837 Acked Write 10 Done Done Sent false
15417 4838 Acked Write 8 Done Done Sent false
15418 4839 Acked Write 10 Done Done Sent false
15419 4840 Acked Flush 0 Done Done Sent false
15420 4841 Acked Read 4 Done Done Sent false
15421 4842 Acked Read 2 Done Done Sent false
15422 4843 Acked Read 9 Done Done Sent false
15423 4844 Acked Read 3 Done Done Sent false
15424 4845 Acked Flush 0 Done Done Sent false
15425 4846 Acked Flush 0 Done Done Sent false
15426 4847 Acked Read 4 Done Done Sent false
15427 4848 Acked Write 2 Done Done Sent false
15428 4849 Acked Write 3 Done Done Sent false
15429 4850 Acked Write 3 Done Done Sent false
15430 4851 Acked Read 6 Done Done Sent false
15431 4852 Acked Read 10 Done Done Sent false
15432 4853 Acked Write 9 Done Done Sent false
15433 4854 Acked Read 9 Done Done Sent false
15434 4855 Acked Write 6 Done Done Sent false
15435 4856 Acked Write 8 Done Done Sent false
15436 4857 Acked Write 7 Done Done Sent false
15437 4858 Acked Write 9 Done Done Sent false
15438 4859 Acked Write 2 Done Done Sent false
15439 4860 Acked Read 2 Done Done Sent false
15440 4861 Acked Write 7 Done Done Sent false
15441 4862 Acked Read 1 Done Done Sent false
15442 4863 Acked Write 10 Done Done Sent false
15443 4864 Acked Read 1 Done Done Sent false
15444 4865 Acked Read 10 Done Done Sent false
15445 4866 Acked Read 8 Done Done Sent false
15446 4867 Acked Read 7 Done Done Sent false
15447 4868 Acked Read 6 Done Done Sent false
15448 4869 Acked Write 3 Done Done Sent false
15449 4870 Acked Write 1 Done Done Sent false
15450 4871 Acked Write 6 Done Done Sent false
15451 4872 Acked Read 5 Done Done Sent false
15452 4873 Acked Read 7 Done Done Sent false
15453 4874 Acked Write 4 Done Done Sent false
15454 4875 Acked Read 6 Done Done Sent false
15455 4876 Acked Write 4 Done Done Sent false
15456 4877 Acked Flush 0 Done Done Sent false
15457 4878 Acked Flush 0 Done Done Sent false
15458 4879 Acked Write 9 Done Done Sent false
15459 4880 Acked Read 8 Done Done Sent false
15460 4881 Acked Write 8 Done Done Sent false
15461 4882 Acked Write 7 Done Done Sent false
15462 4883 Acked Flush 0 Done Done Sent false
15463 4884 Acked Read 1 Done Done Sent false
15464 4885 Acked Write 7 Done Done Sent false
15465 4886 Acked Read 4 Done Done Sent false
15466 4887 Acked Write 10 Done Done Sent false
15467 4888 Acked Read 9 Done Done Sent false
15468 4889 Acked Write 5 Done Done Sent false
15469 4890 Acked Write 7 Done Done Sent false
15470 4891 Acked Write 2 Done Done Sent false
15471 4892 Acked Write 4 Done Done Sent false
15472 4893 Acked Read 1 Done Done Sent false
15473 4894 Acked Read 7 Done Done Sent false
15474 4895 Acked Read 8 Done Done Sent false
15475 4896 Acked Write 7 Done Done Sent false
15476 4897 Acked Read 8 Done Done Sent false
15477 4898 Acked Read 2 Done Done Sent false
15478 4899 Acked Write 2 Done Done Sent false
15479 4900 Acked Write 1 Done Done Sent false
15480 4901 Acked Write 7 Done Done Sent false
15481 4902 Acked Write 8 Done Done Sent false
15482 4903 Acked Write 9 Done Done Sent false
15483 4904 Acked Read 7 Done Done Sent false
15484 4905 Acked Read 1 Done Done Sent false
15485 4906 Acked Write 5 Done Done Sent false
15486 4907 Acked Flush 0 Done Done Sent false
15487 4908 Acked Flush 0 Done Done Sent false
15488 4909 Acked Write 7 Done Done Sent false
15489 4910 Acked Write 6 Done Done Sent false
15490 4911 Acked Write 4 Done Done Sent false
15491 4912 Acked Write 3 Done Done Sent false
15492 4913 Acked Read 10 Done Done Sent false
15493 4914 Acked Read 4 Done Done Sent false
15494 4915 Acked Flush 0 Done Done Sent false
15495 4916 Acked Read 1 Done Done Sent false
15496 4917 Acked Write 3 Done Done Sent false
15497 4918 Acked Write 3 Done Done Sent false
15498 4919 Acked Write 7 Done Done Sent false
15499 4920 Acked Write 2 Done Done Sent false
15500 4921 Acked Write 3 Done Done Sent false
15501 4922 Acked Write 6 Done Done Sent false
15502 4923 Acked Read 10 Done Done Sent false
15503 4924 Acked Flush 0 Done Done Sent false
15504 STATES DS:0 DS:1 DS:2 TOTAL
15505 Sent 0 0 301 301
15506 Done 301 301 0 602
15507 Skipped 0 0 0 0
15508 Error 0 0 0 0
15509 Last Flush: Some(JobId(4924)) Some(JobId(4924)) Some(JobId(4623))
15510 Downstairs last five completed: 4623 4622 4621 4620 4619
15511 Upstairs last five completed: 4924 4923 4922 4921 4920
15512 CLIENT: Up:0 ds:301 act:2
155132025-02-21T06:48:47.258ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155142025-02-21T06:48:47.972ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155152025-02-21T06:48:48.261ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155162025-02-21T06:48:48.261ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64310) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155172025-02-21T06:48:48.262ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155182025-02-21T06:48:48.263ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155192025-02-21T06:48:48.263ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155202025-02-21T06:48:48.263ZINFOcrucible: send last flush ID to this DS: Some(JobId(4623)) = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155212025-02-21T06:48:48.263ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155222025-02-21T06:48:48.263ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4623)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155232025-02-21T06:48:48.263ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4623)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
15524 ----------------------------------------------------------------
15525 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15526 Downstairs last five completed: 4924 4923 4922 4921 4920
15527 Upstairs last five completed: 4924 4923 4922 4921 4920
15528 CLIENT: Up:0 ds:0 act:3
15529 Replay: All jobs finished
15530 Main task has work to do, go find it
15531 got dsc Stop(1)
15532 stop 1
15533 Feb 21 06:48:50.507[8820] Got stop action so:false kr:false
15534 INFO request completed, latency_us: 328, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 55d5b2bc-9df3-45c9-b441-72f53d75b1e3, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
15535 Feb 21 06:48:50.508 INFO request completed, latency_us: 117, response_code: 200, uri: /state/cid/1, method: GET, req_id: d9510843-2c03-4585-9851-4712439e814d, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
155362025-02-21T06:48:50.508ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155372025-02-21T06:48:50.508ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155382025-02-21T06:48:50.508ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
155392025-02-21T06:48:50.508ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
15540 [8820] Exited with: ExitStatus(unix_wait_status(9))
15541 [8820][1] reports Exit
15542 Feb 21 06:48:54.510 INFO request completed, latency_us: 227, response_code: 200, uri: /state/cid/1, method: GET, req_id: b26b5a55-3f7a-404e-bd4d-f778056f2b44, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
15543 001/300 Read block 1752 len 16384
15544 002/300 Read block 120 len 12288
15545 003/300 Read block 433 len 12288
15546 004/300 Read block 2333 len 20480
15547 005/300 Write block 224 len 36864 data: 2 2 2 2 3 3 2 2 2
15548 006/300 Read block 3853 len 24576
15549 007/300 Write block 2274 len 40960 data: 4 5 5 7 6 6 6 6 8 5
15550 008/300 Read block 3340 len 20480
15551 009/300 Flush
15552 010/300 Write block 192 len 28672 data: 3 3 2 2 2 2 3
15553 011/300 Read block 2000 len 20480
15554 012/300 Read block 794 len 24576
15555 013/300 Read block 1814 len 16384
15556 014/300 Read block 2721 len 16384
15557 015/300 Write block 4974 len 16384 data: 3 5 4 3
15558 016/300 Read block 4874 len 40960
15559 017/300 Write block 1276 len 20480 data: 3 3 3 3 2
15560 018/300 Flush
15561 019/300 Read block 3588 len 8192
15562 020/300 Read block 2271 len 12288
15563 021/300 Flush
15564 022/300 Read block 1000 len 28672
15565 023/300 Write block 4828 len 28672 data: 4 4 5 5 5 4 4
15566 024/300 Write block 1494 len 16384 data: 5 5 5 5
15567 025/300 Write block 957 len 40960 data: 5 5 5 5 5 4 4 4 3 3
15568 026/300 Write block 3507 len 28672 data: 4 7 7 7 7 7 5
15569 027/300 Write block 3520 len 8192 data: 3 3
15570 028/300 Read block 3046 len 20480
15571 029/300 Write block 2123 len 36864 data: 5 4 4 3 3 3 3 3 3
15572 030/300 Write block 95 len 36864 data: 2 2 2 2 3 3 3 3 3
15573 031/300 Write block 2455 len 32768 data: 2 2 3 3 3 3 3 4
15574 032/300 Read block 2787 len 8192
15575 033/300 Write block 2390 len 20480 data: 5 5 6 5 4
15576 034/300 Write block 3007 len 28672 data: 5 4 4 6 5 5 4
15577 035/300 Write block 3668 len 4096 data: 4
15578 036/300 Write block 4963 len 32768 data: 4 4 3 3 2 4 4 6
15579 037/300 Write block 462 len 8192 data: 4 5
15580 038/300 Read block 3184 len 4096
15581 039/300 Write block 1622 len 4096 data: 4
15582 040/300 Write block 995 len 32768 data: 3 3 3 4 4 3 3 3
15583 041/300 Flush
15584 042/300 Write block 1005 len 4096 data: 4
15585 043/300 Read block 4454 len 40960
15586 044/300 Write block 3528 len 4096 data: 2
15587 045/300 Read block 2341 len 36864
15588 046/300 Flush
15589 047/300 Write block 2665 len 12288 data: 4 3 3
15590 048/300 Read block 1121 len 24576
15591 049/300 Read block 2265 len 12288
15592 050/300 Flush
15593 051/300 Read block 116 len 40960
15594 052/300 Read block 1802 len 8192
15595 053/300 Write block 4415 len 40960 data: 4 4 4 6 5 3 3 3 3 3
15596 054/300 Flush
15597 055/300 Read block 1176 len 40960
15598 056/300 Flush
15599 057/300 Write block 2069 len 36864 data: 6 5 5 5 4 4 3 3 2
15600 058/300 Read block 2813 len 16384
15601 059/300 Write block 2904 len 36864 data: 4 4 4 3 3 4 4 3 3
15602 060/300 Write block 38 len 24576 data: 3 3 2 2 4 4
15603 061/300 Read block 1250 len 40960
15604 062/300 Read block 3701 len 32768
15605 063/300 Read block 2209 len 12288
15606 064/300 Read block 2584 len 36864
15607 065/300 Write block 2715 len 12288 data: 4 4 4
15608 066/300 Write block 723 len 40960 data: 4 4 3 2 2 3 3 2 2 2
15609 067/300 Read block 1215 len 28672
15610 068/300 Read block 664 len 24576
15611 069/300 Read block 4703 len 20480
15612 070/300 Read block 4516 len 24576
15613 071/300 Read block 4362 len 40960
15614 072/300 Write block 4768 len 4096 data: 3
15615 073/300 Write block 189 len 24576 data: 3 4 4 4 4 3
15616 074/300 Read block 4899 len 24576
15617 075/300 Write block 2850 len 8192 data: 7 6
15618 076/300 Read block 1817 len 24576
15619 077/300 Write block 2512 len 32768 data: 3 2 2 3 3 4 4 4
15620 078/300 Read block 4577 len 4096
15621 079/300 Read block 1327 len 40960
15622 080/300 Write block 4627 len 4096 data: 5
15623 081/300 Write block 411 len 16384 data: 2 3 2 2
15624 082/300 Read block 4990 len 16384
15625 083/300 Read block 3354 len 12288
15626 084/300 Write block 3812 len 36864 data: 5 5 5 6 8 8 6 7 8
15627 085/300 Read block 4602 len 28672
15628 086/300 Read block 4366 len 32768
15629 087/300 Read block 2430 len 40960
15630 088/300 Write block 2588 len 32768 data: 4 4 3 2 2 3 3 3
15631 089/300 Read block 227 len 28672
15632 090/300 Read block 1252 len 12288
15633 091/300 Write block 2270 len 4096 data: 2
15634 092/300 Write block 2804 len 16384 data: 4 3 4 3
15635 093/300 Read block 297 len 36864
15636 094/300 Write block 2668 len 16384 data: 6 6 5 6
15637 095/300 Read block 2529 len 20480
15638 096/300 Write block 1331 len 8192 data: 3 3
15639 097/300 Read block 271 len 32768
15640 098/300 Read block 3113 len 4096
15641 099/300 Write block 1520 len 16384 data: 2 2 3 3
15642 100/300 Read block 3569 len 36864
15643 101/300 Read block 3191 len 28672
15644 102/300 Write block 465 len 36864 data: 4 4 2 2 2 3 2 2 2
15645 103/300 Read block 3123 len 8192
15646 104/300 Flush
15647 105/300 Write block 3176 len 32768 data: 5 5 3 3 3 3 3 2
15648 106/300 Read block 2608 len 20480
15649 107/300 Write block 2245 len 36864 data: 3 3 2 2 2 2 2 2 2
15650 108/300 Read block 2607 len 32768
15651 109/300 Read block 3133 len 40960
15652 110/300 Read block 2558 len 40960
15653 111/300 Write block 1888 len 28672 data: 4 3 3 3 3 4 4
15654 112/300 Write block 2787 len 24576 data: 5 4 4 4 3 3
15655 113/300 Write block 184 len 20480 data: 5 5 4 4 4
15656 114/300 Flush
15657 115/300 Read block 2307 len 20480
15658 116/300 Read block 3313 len 12288
15659 117/300 Write block 4782 len 16384 data: 2 2 2 3
15660 118/300 Read block 3836 len 40960
15661 119/300 Read block 3083 len 32768
15662 120/300 Read block 478 len 20480
15663 121/300 Write block 4685 len 20480 data: 3 3 3 3 3
15664 122/300 Read block 1771 len 40960
15665 123/300 Read block 877 len 24576
15666 124/300 Write block 4361 len 24576 data: 3 2 2 2 2 2
15667 125/300 Read block 1302 len 8192
15668 126/300 Write block 1450 len 36864 data: 4 5 4 5 5 5 5 4 4
15669 127/300 Write block 1783 len 40960 data: 4 4 4 4 3 4 4 4 4 5
15670 128/300 Write block 2609 len 8192 data: 3 2
15671 129/300 Write block 3135 len 32768 data: 6 7 6 6 6 5 4 4
15672 130/300 Read block 3132 len 12288
15673 131/300 Read block 4891 len 40960
15674 132/300 Write block 3963 len 36864 data: 3 4 3 3 3 3 2 2 2
15675 133/300 Read block 4760 len 8192
15676 134/300 Write block 2821 len 12288 data: 3 2 2
15677 135/300 Write block 517 len 24576 data: 3 2 2 3 3 3
15678 136/300 Flush
15679 137/300 Read block 571 len 16384
15680 138/300 Read block 2711 len 32768
15681 139/300 Read block 4756 len 12288
15682 140/300 Read block 2677 len 16384
15683 141/300 Read block 3154 len 28672
15684 142/300 Write block 687 len 16384 data: 3 3 3 3
15685 143/300 Write block 1768 len 24576 data: 5 4 3 3 3 3
15686 144/300 Write block 503 len 32768 data: 7 6 6 6 3 2 2 2
15687 145/300 Read block 4455 len 16384
15688 146/300 Write block 1902 len 16384 data: 4 4 4 4
15689 147/300 Write block 2500 len 8192 data: 2 2
15690 148/300 Read block 393 len 8192
15691 149/300 Write block 3581 len 32768 data: 7 8 8 10 11 10 7 6
15692 150/300 Write block 24 len 32768 data: 2 3 2 2 3 3 4 3
15693 151/300 Write block 2566 len 32768 data: 3 2 2 2 4 5 7 6
15694 152/300 Write block 2017 len 16384 data: 2 2 3 3
15695 153/300 Flush
15696 154/300 Read block 3377 len 16384
15697 155/300 Read block 4626 len 20480
15698 156/300 Write block 676 len 28672 data: 5 7 6 5 4 4 3
15699 157/300 Read block 3615 len 32768
15700 158/300 Read block 4400 len 40960
15701 159/300 Flush
15702 160/300 Write block 2303 len 4096 data: 5
15703 161/300 Read block 1340 len 36864
15704 162/300 Write block 585 len 16384 data: 3 3 3 3
15705 163/300 Read block 1084 len 20480
15706 164/300 Write block 3068 len 12288 data: 2 2 2
15707 165/300 Write block 4521 len 24576 data: 4 3 3 3 2 2
15708 166/300 Write block 730 len 36864 data: 3 3 3 2 3 3 4 6 7
15709 167/300 Flush
15710 168/300 Flush
15711 169/300 Read block 1230 len 28672
15712 170/300 Read block 2906 len 24576
15713 171/300 Read block 3878 len 28672
15714 172/300 Read block 273 len 36864
15715 173/300 Flush
15716 174/300 Read block 1775 len 40960
15717 175/300 Write block 1869 len 28672 data: 3 4 5 4 4 4 4
15718 176/300 Read block 783 len 36864
15719 177/300 Flush
15720 178/300 Read block 2620 len 28672
15721 179/300 Write block 3767 len 12288 data: 6 4 4
15722 180/300 Write block 2288 len 24576 data: 3 3 3 3 2 3
15723 181/300 Read block 3833 len 20480
15724 182/300 Read block 3552 len 32768
15725 183/300 Read block 3514 len 16384
15726 184/300 Write block 2952 len 8192 data: 7 7
15727 185/300 Write block 4374 len 40960 data: 3 3 3 2 3 3 3 3 5 3
15728 186/300 Write block 1726 len 4096 data: 4
15729 187/300 Read block 3215 len 8192
15730 188/300 Read block 1477 len 12288
15731 189/300 Write block 1770 len 24576 data: 4 4 4 4 3 4
15732 190/300 Read block 2107 len 20480
15733 191/300 Read block 1714 len 24576
15734 192/300 Write block 545 len 20480 data: 6 4 4 4 3
15735 193/300 Write block 2186 len 16384 data: 2 5 8 8
15736 194/300 Write block 4385 len 28672 data: 3 3 3 3 4 3 4
15737 195/300 Read block 1547 len 4096
15738 196/300 Read block 4091 len 32768
15739 197/300 Read block 1188 len 12288
15740 198/300 Flush
15741 199/300 Write block 3283 len 36864 data: 3 3 3 3 4 4 3 2 2
15742 200/300 Read block 3862 len 28672
15743 201/300 Write block 1680 len 16384 data: 3 2 2 2
15744 202/300 Read block 2478 len 40960
15745 203/300 Read block 2790 len 12288
15746 204/300 Read block 4658 len 32768
15747 205/300 Write block 1715 len 16384 data: 4 4 3 3
15748 206/300 Read block 1185 len 40960
15749 207/300 Write block 1684 len 40960 data: 4 4 4 3 3 3 3 3 3 3
15750 208/300 Read block 4042 len 20480
15751 209/300 Read block 470 len 12288
15752 210/300 Write block 4748 len 32768 data: 4 4 3 2 2 2 2 2
15753 211/300 Write block 877 len 12288 data: 2 2 3
15754 212/300 Write block 2814 len 28672 data: 2 2 3 4 3 3 3
15755 213/300 Read block 1289 len 8192
15756 214/300 Read block 3145 len 12288
15757 215/300 Read block 3335 len 4096
15758 216/300 Write block 1946 len 12288 data: 4 4 3
15759 217/300 Write block 884 len 40960 data: 3 3 3 2 2 3 4 4 4 4
15760 218/300 Write block 2325 len 36864 data: 3 3 2 3 3 3 2 2 2
15761 219/300 Write block 2075 len 28672 data: 4 4 3 2 2 2 2
15762 220/300 Write block 2745 len 32768 data: 5 5 6 6 6 4 5 6
15763 221/300 Write block 102 len 36864 data: 4 4 3 4 4 4 4 4 4
15764 222/300 Read block 3864 len 28672
15765 223/300 Write block 4641 len 28672 data: 2 2 2 3 4 5 7
15766 224/300 Write block 3021 len 36864 data: 2 2 2 2 2 2 2 3 3
15767 225/300 Write block 4564 len 40960 data: 2 4 5 4 5 6 6 5 4 5
15768 226/300 Read block 1318 len 20480
15769 227/300 Read block 3834 len 8192
15770 228/300 Write block 4460 len 12288 data: 2 2 2
15771 229/300 Read block 493 len 40960
15772 230/300 Read block 3866 len 40960
15773 231/300 Flush
15774 232/300 Flush
15775 233/300 Write block 1852 len 16384 data: 3 3 3 3
15776 234/300 Flush
15777 235/300 Read block 4146 len 36864
15778 236/300 Read block 830 len 32768
15779 237/300 Write block 2572 len 12288 data: 8 7 8
15780 238/300 Read block 4458 len 4096
15781 239/300 Write block 792 len 16384 data: 4 4 4 4
15782 240/300 Write block 233 len 40960 data: 3 3 4 4 3 4 4 4 4 4
15783 241/300 Read block 1567 len 4096
15784 242/300 Read block 760 len 32768
15785 243/300 Write block 3531 len 12288 data: 2 2 2
15786 244/300 Write block 718 len 16384 data: 3 3 3 3
15787 245/300 Read block 4219 len 28672
15788 246/300 Read block 2644 len 28672
15789 247/300 Read block 1564 len 16384
15790 248/300 Read block 3716 len 4096
15791 249/300 Read block 4257 len 20480
15792 250/300 Write block 4756 len 32768 data: 2 2 3 3 3 3 3 3
15793 251/300 Write block 3341 len 16384 data: 5 5 5 5
15794 252/300 Write block 4471 len 24576 data: 5 5 5 4 4 5
15795 253/300 Read block 2507 len 32768
15796 254/300 Write block 2931 len 8192 data: 5 5
15797 255/300 Write block 1238 len 4096 data: 4
15798 256/300 Read block 4513 len 20480
15799 257/300 Write block 51 len 8192 data: 2 2
15800 258/300 Read block 2004 len 4096
15801 259/300 Write block 1972 len 12288 data: 2 2 2
15802 260/300 Write block 4329 len 12288 data: 3 3 3
15803 261/300 Read block 3642 len 24576
15804 262/300 Write block 1396 len 24576 data: 2 3 4 5 4 4
15805 263/300 Write block 105 len 8192 data: 5 5
15806 264/300 Read block 1762 len 16384
15807 265/300 Read block 2740 len 12288
15808 266/300 Read block 3553 len 36864
15809 267/300 Read block 887 len 40960
15810 268/300 Read block 342 len 36864
15811 269/300 Read block 4242 len 8192
15812 270/300 Read block 1980 len 20480
15813 271/300 Read block 1862 len 40960
15814 272/300 Read block 2345 len 12288
15815 273/300 Write block 4576 len 8192 data: 3 3
15816 274/300 Write block 2878 len 4096 data: 5
15817 275/300 Write block 785 len 12288 data: 8 7 7
15818 276/300 Read block 2771 len 36864
15819 277/300 Read block 1295 len 16384
15820 278/300 Flush
15821 279/300 Read block 1643 len 24576
15822 280/300 Read block 4720 len 20480
15823 281/300 Write block 1908 len 28672 data: 4 4 5 5 5 5 5
15824 282/300 Write block 1708 len 12288 data: 3 3 2
15825 283/300 Write block 701 len 12288 data: 3 2 2
15826 284/300 Write block 4651 len 28672 data: 7 7 5 5 5 4 3
15827 285/300 Read block 688 len 32768
15828 286/300 Read block 3110 len 16384
15829 287/300 Write block 428 len 40960 data: 7 7 7 8 8 7 4 3 3 2
15830 288/300 Read block 4661 len 28672
15831 289/300 Write block 2744 len 40960 data: 5 6 6 7 7 7 5 6 7 6
15832 290/300 Flush
15833 291/300 Read block 3804 len 36864
15834 292/300 Flush
15835 293/300 Read block 2313 len 8192
15836 294/300 Read block 2418 len 4096
15837 295/300 Read block 334 len 4096
15838 296/300 Read block 3779 len 40960
15839 297/300 Read block 101 len 28672
15840 298/300 Read block 3259 len 4096
15841 299/300 Write block 2660 len 4096 data: 4
15842 300/300 Write block 3815 len 40960 data: 7 9 9 7 8 9 8 6 6 4
15843 301/300 Read block 3367 len 32768
15844 Main task has work to do, go find it
15845 got dsc Start(1)
15846 start 1
15847 Feb 21 06:48:54.755 INFO request completed, latency_us: 244, response_code: 204, uri: /start/cid/1, method: POST, req_id: 4b7fb854-a672-486b-9cd2-a39fc724b3cb, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
15848 Replay: started 1, returned:Ok(())
15849 ----------------------------------------------------------------
15850 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15851 States: Active Offline Active
15852 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15853 4925 Acked Read 4 Done Sent Done false
15854 4926 Acked Read 3 Done Sent Done false
15855 4927 Acked Read 3 Done Sent Done false
15856 4928 Acked Read 5 Done Sent Done false
15857 4929 Acked Write 9 Done Sent Done false
15858 4930 Acked Read 6 Done Sent Done false
15859 4931 Acked Write 10 Done Sent Done false
15860 4932 Acked Read 5 Done Sent Done false
15861 4933 Acked Flush 0 Done Sent Done false
15862 4934 Acked Write 7 Done Sent Done false
15863 4935 Acked Read 5 Done Sent Done false
15864 4936 Acked Read 6 Done Sent Done false
15865 4937 Acked Read 4 Done Sent Done false
15866 4938 Acked Read 4 Done Sent Done false
15867 4939 Acked Write 4 Done Sent Done false
15868 4940 Acked Read 10 Done Sent Done false
15869 4941 Acked Write 5 Done Sent Done false
15870 4942 Acked Flush 0 Done Sent Done false
15871 4943 Acked Read 2 Done Sent Done false
15872 4944 Acked Read 3 Done Sent Done false
15873 4945 Acked Flush 0 Done Sent Done false
15874 4946 Acked Read 7 Done Sent Done false
15875 4947 Acked Write 7 Done Sent Done false
15876 4948 Acked Write 4 Done Sent Done false
15877 4949 Acked Write 10 Done Sent Done false
15878 4950 Acked Write 7 Done Sent Done false
15879 4951 Acked Write 2 Done Sent Done false
15880 4952 Acked Read 5 Done Sent Done false
15881 4953 Acked Write 9 Done Sent Done false
15882 4954 Acked Write 9 Done Sent Done false
15883 4955 Acked Write 8 Done Sent Done false
15884 4956 Acked Read 2 Done Sent Done false
15885 4957 Acked Write 5 Done Sent Done false
15886 4958 Acked Write 7 Done Sent Done false
15887 4959 Acked Write 1 Done Sent Done false
15888 4960 Acked Write 8 Done Sent Done false
15889 4961 Acked Write 2 Done Sent Done false
15890 4962 Acked Read 1 Done Sent Done false
15891 4963 Acked Write 1 Done Sent Done false
15892 4964 Acked Write 8 Done Sent Done false
15893 4965 Acked Flush 0 Done Sent Done false
15894 4966 Acked Write 1 Done Sent Done false
15895 4967 Acked Read 10 Done Sent Done false
15896 4968 Acked Write 1 Done Sent Done false
15897 4969 Acked Read 9 Done Sent Done false
15898 4970 Acked Flush 0 Done Sent Done false
15899 4971 Acked Write 3 Done Sent Done false
15900 4972 Acked Read 6 Done Sent Done false
15901 4973 Acked Read 3 Done Sent Done false
15902 4974 Acked Flush 0 Done Sent Done false
15903 4975 Acked Read 10 Done Sent Done false
15904 4976 Acked Read 2 Done Sent Done false
15905 4977 Acked Write 10 Done Sent Done false
15906 4978 Acked Flush 0 Done Sent Done false
15907 4979 Acked Read 10 Done Sent Done false
15908 4980 Acked Flush 0 Done Sent Done false
15909 4981 Acked Write 9 Done Sent Done false
15910 4982 Acked Read 4 Done Sent Done false
15911 4983 Acked Write 9 Done Sent Done false
15912 4984 Acked Write 6 Done Sent Done false
15913 4985 Acked Read 10 Done Sent Done false
15914 4986 Acked Read 8 Done Sent Done false
15915 4987 Acked Read 3 Done Sent Done false
15916 4988 Acked Read 9 Done Sent Done false
15917 4989 Acked Write 3 Done Sent Done false
15918 4990 Acked Write 10 Done Sent Done false
15919 4991 Acked Read 7 Done Sent Done false
15920 4992 Acked Read 6 Done Sent Done false
15921 4993 Acked Read 5 Done Sent Done false
15922 4994 Acked Read 6 Done Sent Done false
15923 4995 Acked Read 10 Done Sent Done false
15924 4996 Acked Write 1 Done Sent Done false
15925 4997 Acked Write 6 Done Sent Done false
15926 4998 Acked Read 6 Done Sent Done false
15927 4999 Acked Write 2 Done Sent Done false
15928 5000 Acked Read 6 Done Sent Done false
15929 5001 Acked Write 8 Done Sent Done false
15930 5002 Acked Read 1 Done Sent Done false
15931 5003 Acked Read 10 Done Sent Done false
15932 5004 Acked Write 1 Done Sent Done false
15933 5005 Acked Write 4 Done Sent Done false
15934 5006 Acked Read 4 Done Sent Done false
15935 5007 Acked Read 3 Done Sent Done false
15936 5008 Acked Write 9 Done Sent Done false
15937 5009 Acked Read 7 Done Sent Done false
15938 5010 Acked Read 8 Done Sent Done false
15939 5011 Acked Read 10 Done Sent Done false
15940 5012 Acked Write 8 Done Sent Done false
15941 5013 Acked Read 7 Done Sent Done false
15942 5014 Acked Read 3 Done Sent Done false
15943 5015 Acked Write 1 Done Sent Done false
15944 5016 Acked Write 4 Done Sent Done false
15945 5017 Acked Read 9 Done Sent Done false
15946 5018 Acked Write 4 Done Sent Done false
15947 5019 Acked Read 5 Done Sent Done false
15948 5020 Acked Write 2 Done Sent Done false
15949 5021 Acked Read 8 Done Sent Done false
15950 5022 Acked Read 1 Done Sent Done false
15951 5023 Acked Write 4 Done Sent Done false
15952 5024 Acked Read 9 Done Sent Done false
15953 5025 Acked Read 7 Done Sent Done false
15954 5026 Acked Write 9 Done Sent Done false
15955 5027 Acked Read 2 Done Sent Done false
15956 5028 Acked Flush 0 Done Sent Done false
15957 5029 Acked Write 8 Done Sent Done false
15958 5030 Acked Read 5 Done Sent Done false
15959 5031 Acked Write 9 Done Sent Done false
15960 5032 Acked Read 8 Done Sent Done false
15961 5033 Acked Read 10 Done Sent Done false
15962 5034 Acked Read 10 Done Sent Done false
15963 5035 Acked Write 7 Done Sent Done false
15964 5036 Acked Write 6 Done Sent Done false
15965 5037 Acked Write 5 Done Sent Done false
15966 5038 Acked Flush 0 Done Sent Done false
15967 5039 Acked Read 5 Done Sent Done false
15968 5040 Acked Read 3 Done Sent Done false
15969 5041 Acked Write 4 Done Sent Done false
15970 5042 Acked Read 10 Done Sent Done false
15971 5043 Acked Read 8 Done Sent Done false
15972 5044 Acked Read 5 Done Sent Done false
15973 5045 Acked Write 5 Done Sent Done false
15974 5046 Acked Read 10 Done Sent Done false
15975 5047 Acked Read 6 Done Sent Done false
15976 5048 Acked Write 6 Done Sent Done false
15977 5049 Acked Read 2 Done Sent Done false
15978 5050 Acked Write 9 Done Sent Done false
15979 5051 Acked Write 10 Done Sent Done false
15980 5052 Acked Write 2 Done Sent Done false
15981 5053 Acked Write 8 Done Sent Done false
15982 5054 Acked Read 3 Done Sent Done false
15983 5055 Acked Read 10 Done Sent Done false
15984 5056 Acked Write 9 Done Sent Done false
15985 5057 Acked Read 2 Done Sent Done false
15986 5058 Acked Write 3 Done Sent Done false
15987 5059 Acked Write 6 Done Sent Done false
15988 5060 Acked Flush 0 Done Sent Done false
15989 5061 Acked Read 4 Done Sent Done false
15990 5062 Acked Read 8 Done Sent Done false
15991 5063 Acked Read 3 Done Sent Done false
15992 5064 Acked Read 4 Done Sent Done false
15993 5065 Acked Read 7 Done Sent Done false
15994 5066 Acked Write 4 Done Sent Done false
15995 5067 Acked Write 6 Done Sent Done false
15996 5068 Acked Write 8 Done Sent Done false
15997 5069 Acked Read 4 Done Sent Done false
15998 5070 Acked Write 4 Done Sent Done false
15999 5071 Acked Write 2 Done Sent Done false
16000 5072 Acked Read 2 Done Sent Done false
16001 5073 Acked Write 8 Done Sent Done false
16002 5074 Acked Write 8 Done Sent Done false
16003 5075 Acked Write 8 Done Sent Done false
16004 5076 Acked Write 4 Done Sent Done false
16005 5077 Acked Flush 0 Done Sent Done false
16006 5078 Acked Read 4 Done Sent Done false
16007 5079 Acked Read 5 Done Sent Done false
16008 5080 Acked Write 7 Done Sent Done false
16009 5081 Acked Read 8 Done Sent Done false
16010 5082 Acked Read 10 Done Sent Done false
16011 5083 Acked Flush 0 Done Sent Done false
16012 5084 Acked Write 1 Done Sent Done false
16013 5085 Acked Read 9 Done Sent Done false
16014 5086 Acked Write 4 Done Sent Done false
16015 5087 Acked Read 5 Done Sent Done false
16016 5088 Acked Write 3 Done Sent Done false
16017 5089 Acked Write 6 Done Sent Done false
16018 5090 Acked Write 9 Done Sent Done false
16019 5091 Acked Flush 0 Done Sent Done false
16020 5092 Acked Flush 0 Done Sent Done false
16021 5093 Acked Read 7 Done Sent Done false
16022 5094 Acked Read 6 Done Sent Done false
16023 5095 Acked Read 7 Done Sent Done false
16024 5096 Acked Read 9 Done Sent Done false
16025 5097 Acked Flush 0 Done Sent Done false
16026 5098 Acked Read 10 Done Sent Done false
16027 5099 Acked Write 7 Done Sent Done false
16028 5100 Acked Read 9 Done Sent Done false
16029 5101 Acked Flush 0 Done Sent Done false
16030 5102 Acked Read 7 Done Sent Done false
16031 5103 Acked Write 3 Done Sent Done false
16032 5104 Acked Write 6 Done Sent Done false
16033 5105 Acked Read 5 Done Sent Done false
16034 5106 Acked Read 8 Done Sent Done false
16035 5107 Acked Read 4 Done Sent Done false
16036 5108 Acked Write 2 Done Sent Done false
16037 5109 Acked Write 10 Done Sent Done false
16038 5110 Acked Write 1 Done Sent Done false
16039 5111 Acked Read 2 Done Sent Done false
16040 5112 Acked Read 3 Done Sent Done false
16041 5113 Acked Write 6 Done Sent Done false
16042 5114 Acked Read 5 Done Sent Done false
16043 5115 Acked Read 6 Done Sent Done false
16044 5116 Acked Write 5 Done Sent Done false
16045 5117 Acked Write 4 Done Sent Done false
16046 5118 Acked Write 7 Done Sent Done false
16047 5119 Acked Read 1 Done Sent Done false
16048 5120 Acked Read 8 Done Sent Done false
16049 5121 Acked Read 3 Done Sent Done false
16050 5122 Acked Flush 0 Done Sent Done false
16051 5123 Acked Write 9 Done Sent Done false
16052 5124 Acked Read 7 Done Sent Done false
16053 5125 Acked Write 4 Done Sent Done false
16054 5126 Acked Read 10 Done Sent Done false
16055 5127 Acked Read 3 Done Sent Done false
16056 5128 Acked Read 8 Done Sent Done false
16057 5129 Acked Write 4 Done Sent Done false
16058 5130 Acked Read 10 Done Sent Done false
16059 5131 Acked Write 10 Done Sent Done false
16060 5132 Acked Read 5 Done Sent Done false
16061 5133 Acked Read 3 Done Sent Done false
16062 5134 Acked Write 8 Done Sent Done false
16063 5135 Acked Write 3 Done Sent Done false
16064 5136 Acked Write 7 Done Sent Done false
16065 5137 Acked Read 2 Done Sent Done false
16066 5138 Acked Read 3 Done Sent Done false
16067 5139 Acked Read 1 Done Sent Done false
16068 5140 Acked Write 3 Done Sent Done false
16069 5141 Acked Write 10 Done Sent Done false
16070 5142 Acked Write 9 Done Sent Done false
16071 5143 Acked Write 7 Done Sent Done false
16072 5144 Acked Write 8 Done Sent Done false
16073 5145 Acked Write 9 Done Sent Done false
16074 5146 Acked Read 7 Done Sent Done false
16075 5147 Acked Write 7 Done Sent Done false
16076 5148 Acked Write 9 Done Sent Done false
16077 5149 Acked Write 10 Done Sent Done false
16078 5150 Acked Read 5 Done Sent Done false
16079 5151 Acked Read 2 Done Sent Done false
16080 5152 Acked Write 3 Done Sent Done false
16081 5153 Acked Read 10 Done Sent Done false
16082 5154 Acked Read 10 Done Sent Done false
16083 5155 Acked Flush 0 Done Sent Done false
16084 5156 Acked Flush 0 Done Sent Done false
16085 5157 Acked Write 4 Done Sent Done false
16086 5158 Acked Flush 0 Done Sent Done false
16087 5159 Acked Read 9 Done Sent Done false
16088 5160 Acked Read 8 Done Sent Done false
16089 5161 Acked Write 3 Done Sent Done false
16090 5162 Acked Read 1 Done Sent Done false
16091 5163 Acked Write 4 Done Sent Done false
16092 5164 Acked Write 10 Done Sent Done false
16093 5165 Acked Read 1 Done Sent Done false
16094 5166 Acked Read 8 Done Sent Done false
16095 5167 Acked Write 3 Done Sent Done false
16096 5168 Acked Write 4 Done Sent Done false
16097 5169 Acked Read 7 Done Sent Done false
16098 5170 Acked Read 7 Done Sent Done false
16099 5171 Acked Read 4 Done Sent Done false
16100 5172 Acked Read 1 Done Sent Done false
16101 5173 Acked Read 5 Done Sent Done false
16102 5174 Acked Write 8 Done Sent Done false
16103 5175 Acked Write 4 Done Sent Done false
16104 5176 Acked Write 6 Done Sent Done false
16105 5177 Acked Read 8 Done Sent Done false
16106 5178 Acked Write 2 Done Sent Done false
16107 5179 Acked Write 1 Done Sent Done false
16108 5180 Acked Read 5 Done Sent Done false
16109 5181 Acked Write 2 Done Sent Done false
16110 5182 Acked Read 1 Done Sent Done false
16111 5183 Acked Write 3 Done Sent Done false
16112 5184 Acked Write 3 Done Sent Done false
16113 5185 Acked Read 6 Done Sent Done false
16114 5186 Acked Write 6 Done Sent Done false
16115 5187 Acked Write 2 Done Sent Done false
16116 5188 Acked Read 4 Done Sent Done false
16117 5189 Acked Read 3 Done Sent Done false
16118 5190 Acked Read 9 Done Sent Done false
16119 5191 Acked Read 10 Done Sent Done false
16120 5192 Acked Read 9 Done Sent Done false
16121 5193 Acked Read 2 Done Sent Done false
16122 5194 Acked Read 5 Done Sent Done false
16123 5195 Acked Read 10 Done Sent Done false
16124 5196 Acked Read 3 Done Sent Done false
16125 5197 Acked Write 2 Done Sent Done false
16126 5198 Acked Write 1 Done Sent Done false
16127 5199 Acked Write 3 Done Sent Done false
16128 5200 Acked Read 9 Done Sent Done false
16129 5201 Acked Read 4 Done Sent Done false
16130 5202 Acked Flush 0 Done Sent Done false
16131 5203 Acked Read 6 Done Sent Done false
16132 5204 Acked Read 5 Done Sent Done false
16133 5205 Acked Write 7 Done Sent Done false
16134 5206 Acked Write 3 Done Sent Done false
16135 5207 Acked Write 3 Done Sent Done false
16136 5208 Acked Write 7 Done Sent Done false
16137 5209 Acked Read 8 Done Sent Done false
16138 5210 Acked Read 4 Done Sent Done false
16139 5211 Acked Write 10 Done Sent Done false
16140 5212 Acked Read 7 Done Sent Done false
16141 5213 Acked Write 10 Done Sent Done false
16142 5214 Acked Flush 0 Done Sent Done false
16143 5215 Acked Read 9 Done Sent Done false
16144 5216 Acked Flush 0 Done Sent Done false
16145 5217 Acked Read 2 Done Sent Done false
16146 5218 Acked Read 1 Done Sent Done false
16147 5219 Acked Read 1 Done Sent Done false
16148 5220 Acked Read 10 Done Sent Done false
16149 5221 Acked Read 7 Done Sent Done false
16150 5222 Acked Read 1 Done Sent Done false
16151 5223 Acked Write 1 Done Sent Done false
16152 5224 Acked Write 10 Done Sent Done false
16153 5225 Acked Read 8 Done Sent Done false
16154 STATES DS:0 DS:1 DS:2 TOTAL
16155 Sent 0 301 0 301
16156 Done 301 0 301 602
16157 Skipped 0 0 0 0
16158 Error 0 0 0 0
16159 Last Flush: Some(JobId(5216)) Some(JobId(4924)) Some(JobId(5216))
16160 Downstairs last five completed: 4924 4923 4922 4921 4920
16161 Upstairs last five completed: 5225 5224 5223 5222 5221
16162 CLIENT: Up:0 ds:301 act:2
161632025-02-21T06:48:55.258ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
161642025-02-21T06:48:55.985ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
16165 [8820] got start action
16166 [8820] I am going to restart
16167 Starting downstairs at port 8820
16168 Make output file at "/tmp/dsc/downstairs-8820.txt"
16169 [8820][1] reports Starting
16170 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1220)
16171 [8820][1] reports Running
161722025-02-21T06:48:57.260ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
161732025-02-21T06:48:57.989ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
16174 ----------------------------------------------------------------
16175 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16176 States: Active Offline Active
16177 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16178 4925 Acked Read 4 Done Sent Done false
16179 4926 Acked Read 3 Done Sent Done false
16180 4927 Acked Read 3 Done Sent Done false
16181 4928 Acked Read 5 Done Sent Done false
16182 4929 Acked Write 9 Done Sent Done false
16183 4930 Acked Read 6 Done Sent Done false
16184 4931 Acked Write 10 Done Sent Done false
16185 4932 Acked Read 5 Done Sent Done false
16186 4933 Acked Flush 0 Done Sent Done false
16187 4934 Acked Write 7 Done Sent Done false
16188 4935 Acked Read 5 Done Sent Done false
16189 4936 Acked Read 6 Done Sent Done false
16190 4937 Acked Read 4 Done Sent Done false
16191 4938 Acked Read 4 Done Sent Done false
16192 4939 Acked Write 4 Done Sent Done false
16193 4940 Acked Read 10 Done Sent Done false
16194 4941 Acked Write 5 Done Sent Done false
16195 4942 Acked Flush 0 Done Sent Done false
16196 4943 Acked Read 2 Done Sent Done false
16197 4944 Acked Read 3 Done Sent Done false
16198 4945 Acked Flush 0 Done Sent Done false
16199 4946 Acked Read 7 Done Sent Done false
16200 4947 Acked Write 7 Done Sent Done false
16201 4948 Acked Write 4 Done Sent Done false
16202 4949 Acked Write 10 Done Sent Done false
16203 4950 Acked Write 7 Done Sent Done false
16204 4951 Acked Write 2 Done Sent Done false
16205 4952 Acked Read 5 Done Sent Done false
16206 4953 Acked Write 9 Done Sent Done false
16207 4954 Acked Write 9 Done Sent Done false
16208 4955 Acked Write 8 Done Sent Done false
16209 4956 Acked Read 2 Done Sent Done false
16210 4957 Acked Write 5 Done Sent Done false
16211 4958 Acked Write 7 Done Sent Done false
16212 4959 Acked Write 1 Done Sent Done false
16213 4960 Acked Write 8 Done Sent Done false
16214 4961 Acked Write 2 Done Sent Done false
16215 4962 Acked Read 1 Done Sent Done false
16216 4963 Acked Write 1 Done Sent Done false
16217 4964 Acked Write 8 Done Sent Done false
16218 4965 Acked Flush 0 Done Sent Done false
16219 4966 Acked Write 1 Done Sent Done false
16220 4967 Acked Read 10 Done Sent Done false
16221 4968 Acked Write 1 Done Sent Done false
16222 4969 Acked Read 9 Done Sent Done false
16223 4970 Acked Flush 0 Done Sent Done false
16224 4971 Acked Write 3 Done Sent Done false
16225 4972 Acked Read 6 Done Sent Done false
16226 4973 Acked Read 3 Done Sent Done false
16227 4974 Acked Flush 0 Done Sent Done false
16228 4975 Acked Read 10 Done Sent Done false
16229 4976 Acked Read 2 Done Sent Done false
16230 4977 Acked Write 10 Done Sent Done false
16231 4978 Acked Flush 0 Done Sent Done false
16232 4979 Acked Read 10 Done Sent Done false
16233 4980 Acked Flush 0 Done Sent Done false
16234 4981 Acked Write 9 Done Sent Done false
16235 4982 Acked Read 4 Done Sent Done false
16236 4983 Acked Write 9 Done Sent Done false
16237 4984 Acked Write 6 Done Sent Done false
16238 4985 Acked Read 10 Done Sent Done false
16239 4986 Acked Read 8 Done Sent Done false
16240 4987 Acked Read 3 Done Sent Done false
16241 4988 Acked Read 9 Done Sent Done false
16242 4989 Acked Write 3 Done Sent Done false
16243 4990 Acked Write 10 Done Sent Done false
16244 4991 Acked Read 7 Done Sent Done false
16245 4992 Acked Read 6 Done Sent Done false
16246 4993 Acked Read 5 Done Sent Done false
16247 4994 Acked Read 6 Done Sent Done false
16248 4995 Acked Read 10 Done Sent Done false
16249 4996 Acked Write 1 Done Sent Done false
16250 4997 Acked Write 6 Done Sent Done false
16251 4998 Acked Read 6 Done Sent Done false
16252 4999 Acked Write 2 Done Sent Done false
16253 5000 Acked Read 6 Done Sent Done false
16254 5001 Acked Write 8 Done Sent Done false
16255 5002 Acked Read 1 Done Sent Done false
16256 5003 Acked Read 10 Done Sent Done false
16257 5004 Acked Write 1 Done Sent Done false
16258 5005 Acked Write 4 Done Sent Done false
16259 5006 Acked Read 4 Done Sent Done false
16260 5007 Acked Read 3 Done Sent Done false
16261 5008 Acked Write 9 Done Sent Done false
16262 5009 Acked Read 7 Done Sent Done false
16263 5010 Acked Read 8 Done Sent Done false
16264 5011 Acked Read 10 Done Sent Done false
16265 5012 Acked Write 8 Done Sent Done false
16266 5013 Acked Read 7 Done Sent Done false
16267 5014 Acked Read 3 Done Sent Done false
16268 5015 Acked Write 1 Done Sent Done false
16269 5016 Acked Write 4 Done Sent Done false
16270 5017 Acked Read 9 Done Sent Done false
16271 5018 Acked Write 4 Done Sent Done false
16272 5019 Acked Read 5 Done Sent Done false
16273 5020 Acked Write 2 Done Sent Done false
16274 5021 Acked Read 8 Done Sent Done false
16275 5022 Acked Read 1 Done Sent Done false
16276 5023 Acked Write 4 Done Sent Done false
16277 5024 Acked Read 9 Done Sent Done false
16278 5025 Acked Read 7 Done Sent Done false
16279 5026 Acked Write 9 Done Sent Done false
16280 5027 Acked Read 2 Done Sent Done false
16281 5028 Acked Flush 0 Done Sent Done false
16282 5029 Acked Write 8 Done Sent Done false
16283 5030 Acked Read 5 Done Sent Done false
16284 5031 Acked Write 9 Done Sent Done false
16285 5032 Acked Read 8 Done Sent Done false
16286 5033 Acked Read 10 Done Sent Done false
16287 5034 Acked Read 10 Done Sent Done false
16288 5035 Acked Write 7 Done Sent Done false
16289 5036 Acked Write 6 Done Sent Done false
16290 5037 Acked Write 5 Done Sent Done false
16291 5038 Acked Flush 0 Done Sent Done false
16292 5039 Acked Read 5 Done Sent Done false
16293 5040 Acked Read 3 Done Sent Done false
16294 5041 Acked Write 4 Done Sent Done false
16295 5042 Acked Read 10 Done Sent Done false
16296 5043 Acked Read 8 Done Sent Done false
16297 5044 Acked Read 5 Done Sent Done false
16298 5045 Acked Write 5 Done Sent Done false
16299 5046 Acked Read 10 Done Sent Done false
16300 5047 Acked Read 6 Done Sent Done false
16301 5048 Acked Write 6 Done Sent Done false
16302 5049 Acked Read 2 Done Sent Done false
16303 5050 Acked Write 9 Done Sent Done false
16304 5051 Acked Write 10 Done Sent Done false
16305 5052 Acked Write 2 Done Sent Done false
16306 5053 Acked Write 8 Done Sent Done false
16307 5054 Acked Read 3 Done Sent Done false
16308 5055 Acked Read 10 Done Sent Done false
16309 5056 Acked Write 9 Done Sent Done false
16310 5057 Acked Read 2 Done Sent Done false
16311 5058 Acked Write 3 Done Sent Done false
16312 5059 Acked Write 6 Done Sent Done false
16313 5060 Acked Flush 0 Done Sent Done false
16314 5061 Acked Read 4 Done Sent Done false
16315 5062 Acked Read 8 Done Sent Done false
16316 5063 Acked Read 3 Done Sent Done false
16317 5064 Acked Read 4 Done Sent Done false
16318 5065 Acked Read 7 Done Sent Done false
16319 5066 Acked Write 4 Done Sent Done false
16320 5067 Acked Write 6 Done Sent Done false
16321 5068 Acked Write 8 Done Sent Done false
16322 5069 Acked Read 4 Done Sent Done false
16323 5070 Acked Write 4 Done Sent Done false
16324 5071 Acked Write 2 Done Sent Done false
16325 5072 Acked Read 2 Done Sent Done false
16326 5073 Acked Write 8 Done Sent Done false
16327 5074 Acked Write 8 Done Sent Done false
16328 5075 Acked Write 8 Done Sent Done false
16329 5076 Acked Write 4 Done Sent Done false
16330 5077 Acked Flush 0 Done Sent Done false
16331 5078 Acked Read 4 Done Sent Done false
16332 5079 Acked Read 5 Done Sent Done false
16333 5080 Acked Write 7 Done Sent Done false
16334 5081 Acked Read 8 Done Sent Done false
16335 5082 Acked Read 10 Done Sent Done false
16336 5083 Acked Flush 0 Done Sent Done false
16337 5084 Acked Write 1 Done Sent Done false
16338 5085 Acked Read 9 Done Sent Done false
16339 5086 Acked Write 4 Done Sent Done false
16340 5087 Acked Read 5 Done Sent Done false
16341 5088 Acked Write 3 Done Sent Done false
16342 5089 Acked Write 6 Done Sent Done false
16343 5090 Acked Write 9 Done Sent Done false
16344 5091 Acked Flush 0 Done Sent Done false
16345 5092 Acked Flush 0 Done Sent Done false
16346 5093 Acked Read 7 Done Sent Done false
16347 5094 Acked Read 6 Done Sent Done false
16348 5095 Acked Read 7 Done Sent Done false
16349 5096 Acked Read 9 Done Sent Done false
16350 5097 Acked Flush 0 Done Sent Done false
16351 5098 Acked Read 10 Done Sent Done false
16352 5099 Acked Write 7 Done Sent Done false
16353 5100 Acked Read 9 Done Sent Done false
16354 5101 Acked Flush 0 Done Sent Done false
16355 5102 Acked Read 7 Done Sent Done false
16356 5103 Acked Write 3 Done Sent Done false
16357 5104 Acked Write 6 Done Sent Done false
16358 5105 Acked Read 5 Done Sent Done false
16359 5106 Acked Read 8 Done Sent Done false
16360 5107 Acked Read 4 Done Sent Done false
16361 5108 Acked Write 2 Done Sent Done false
16362 5109 Acked Write 10 Done Sent Done false
16363 5110 Acked Write 1 Done Sent Done false
16364 5111 Acked Read 2 Done Sent Done false
16365 5112 Acked Read 3 Done Sent Done false
16366 5113 Acked Write 6 Done Sent Done false
16367 5114 Acked Read 5 Done Sent Done false
16368 5115 Acked Read 6 Done Sent Done false
16369 5116 Acked Write 5 Done Sent Done false
16370 5117 Acked Write 4 Done Sent Done false
16371 5118 Acked Write 7 Done Sent Done false
16372 5119 Acked Read 1 Done Sent Done false
16373 5120 Acked Read 8 Done Sent Done false
16374 5121 Acked Read 3 Done Sent Done false
16375 5122 Acked Flush 0 Done Sent Done false
16376 5123 Acked Write 9 Done Sent Done false
16377 5124 Acked Read 7 Done Sent Done false
16378 5125 Acked Write 4 Done Sent Done false
16379 5126 Acked Read 10 Done Sent Done false
16380 5127 Acked Read 3 Done Sent Done false
16381 5128 Acked Read 8 Done Sent Done false
16382 5129 Acked Write 4 Done Sent Done false
16383 5130 Acked Read 10 Done Sent Done false
16384 5131 Acked Write 10 Done Sent Done false
16385 5132 Acked Read 5 Done Sent Done false
16386 5133 Acked Read 3 Done Sent Done false
16387 5134 Acked Write 8 Done Sent Done false
16388 5135 Acked Write 3 Done Sent Done false
16389 5136 Acked Write 7 Done Sent Done false
16390 5137 Acked Read 2 Done Sent Done false
16391 5138 Acked Read 3 Done Sent Done false
16392 5139 Acked Read 1 Done Sent Done false
16393 5140 Acked Write 3 Done Sent Done false
16394 5141 Acked Write 10 Done Sent Done false
16395 5142 Acked Write 9 Done Sent Done false
16396 5143 Acked Write 7 Done Sent Done false
16397 5144 Acked Write 8 Done Sent Done false
16398 5145 Acked Write 9 Done Sent Done false
16399 5146 Acked Read 7 Done Sent Done false
16400 5147 Acked Write 7 Done Sent Done false
16401 5148 Acked Write 9 Done Sent Done false
16402 5149 Acked Write 10 Done Sent Done false
16403 5150 Acked Read 5 Done Sent Done false
16404 5151 Acked Read 2 Done Sent Done false
16405 5152 Acked Write 3 Done Sent Done false
16406 5153 Acked Read 10 Done Sent Done false
16407 5154 Acked Read 10 Done Sent Done false
16408 5155 Acked Flush 0 Done Sent Done false
16409 5156 Acked Flush 0 Done Sent Done false
16410 5157 Acked Write 4 Done Sent Done false
16411 5158 Acked Flush 0 Done Sent Done false
16412 5159 Acked Read 9 Done Sent Done false
16413 5160 Acked Read 8 Done Sent Done false
16414 5161 Acked Write 3 Done Sent Done false
16415 5162 Acked Read 1 Done Sent Done false
16416 5163 Acked Write 4 Done Sent Done false
16417 5164 Acked Write 10 Done Sent Done false
16418 5165 Acked Read 1 Done Sent Done false
16419 5166 Acked Read 8 Done Sent Done false
16420 5167 Acked Write 3 Done Sent Done false
16421 5168 Acked Write 4 Done Sent Done false
16422 5169 Acked Read 7 Done Sent Done false
16423 5170 Acked Read 7 Done Sent Done false
16424 5171 Acked Read 4 Done Sent Done false
16425 5172 Acked Read 1 Done Sent Done false
16426 5173 Acked Read 5 Done Sent Done false
16427 5174 Acked Write 8 Done Sent Done false
16428 5175 Acked Write 4 Done Sent Done false
16429 5176 Acked Write 6 Done Sent Done false
16430 5177 Acked Read 8 Done Sent Done false
16431 5178 Acked Write 2 Done Sent Done false
16432 5179 Acked Write 1 Done Sent Done false
16433 5180 Acked Read 5 Done Sent Done false
16434 5181 Acked Write 2 Done Sent Done false
16435 5182 Acked Read 1 Done Sent Done false
16436 5183 Acked Write 3 Done Sent Done false
16437 5184 Acked Write 3 Done Sent Done false
16438 5185 Acked Read 6 Done Sent Done false
16439 5186 Acked Write 6 Done Sent Done false
16440 5187 Acked Write 2 Done Sent Done false
16441 5188 Acked Read 4 Done Sent Done false
16442 5189 Acked Read 3 Done Sent Done false
16443 5190 Acked Read 9 Done Sent Done false
16444 5191 Acked Read 10 Done Sent Done false
16445 5192 Acked Read 9 Done Sent Done false
16446 5193 Acked Read 2 Done Sent Done false
16447 5194 Acked Read 5 Done Sent Done false
16448 5195 Acked Read 10 Done Sent Done false
16449 5196 Acked Read 3 Done Sent Done false
16450 5197 Acked Write 2 Done Sent Done false
16451 5198 Acked Write 1 Done Sent Done false
16452 5199 Acked Write 3 Done Sent Done false
16453 5200 Acked Read 9 Done Sent Done false
16454 5201 Acked Read 4 Done Sent Done false
16455 5202 Acked Flush 0 Done Sent Done false
16456 5203 Acked Read 6 Done Sent Done false
16457 5204 Acked Read 5 Done Sent Done false
16458 5205 Acked Write 7 Done Sent Done false
16459 5206 Acked Write 3 Done Sent Done false
16460 5207 Acked Write 3 Done Sent Done false
16461 5208 Acked Write 7 Done Sent Done false
16462 5209 Acked Read 8 Done Sent Done false
16463 5210 Acked Read 4 Done Sent Done false
16464 5211 Acked Write 10 Done Sent Done false
16465 5212 Acked Read 7 Done Sent Done false
16466 5213 Acked Write 10 Done Sent Done false
16467 5214 Acked Flush 0 Done Sent Done false
16468 5215 Acked Read 9 Done Sent Done false
16469 5216 Acked Flush 0 Done Sent Done false
16470 5217 Acked Read 2 Done Sent Done false
16471 5218 Acked Read 1 Done Sent Done false
16472 5219 Acked Read 1 Done Sent Done false
16473 5220 Acked Read 10 Done Sent Done false
16474 5221 Acked Read 7 Done Sent Done false
16475 5222 Acked Read 1 Done Sent Done false
16476 5223 Acked Write 1 Done Sent Done false
16477 5224 Acked Write 10 Done Sent Done false
16478 5225 Acked Read 8 Done Sent Done false
16479 STATES DS:0 DS:1 DS:2 TOTAL
16480 Sent 0 301 0 301
16481 Done 301 0 301 602
16482 Skipped 0 0 0 0
16483 Error 0 0 0 0
16484 Last Flush: Some(JobId(5216)) Some(JobId(4924)) Some(JobId(5216))
16485 Downstairs last five completed: 4924 4923 4922 4921 4920
16486 Upstairs last five completed: 5225 5224 5223 5222 5221
16487 CLIENT: Up:0 ds:301 act:2
164882025-02-21T06:48:59.260ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
164892025-02-21T06:48:59.991ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
164902025-02-21T06:49:00.509ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
164912025-02-21T06:49:00.509ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46731) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
164922025-02-21T06:49:00.510ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
164932025-02-21T06:49:00.511ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
164942025-02-21T06:49:00.511ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
164952025-02-21T06:49:00.511ZINFOcrucible: send last flush ID to this DS: Some(JobId(4924)) = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
164962025-02-21T06:49:00.511ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
164972025-02-21T06:49:00.511ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4924)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
164982025-02-21T06:49:00.511ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4924)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
16499 ----------------------------------------------------------------
16500 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16501 Downstairs last five completed: 5226 5225 5224 5223 5222
16502 Upstairs last five completed: 5226 5225 5224 5223 5222
16503 CLIENT: Up:0 ds:0 act:3
16504 Replay: All jobs finished
16505 Main task has work to do, go find it
16506 got dsc Stop(2)
16507 stop 2
16508 [8830] Got stop action so:false kr:false
16509 Feb 21 06:49:02.761 INFO request completed, latency_us: 341, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c9c31934-d517-4c09-b26d-a5a25ff569f6, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
16510 Feb 21 06:49:02.762 INFO request completed, latency_us: 107, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4340eb22-1f07-4a06-b394-85210780405b, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
165112025-02-21T06:49:02.762ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
165122025-02-21T06:49:02.762ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
165132025-02-21T06:49:02.762ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
165142025-02-21T06:49:02.762ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
16515 [8830] Exited with: ExitStatus(unix_wait_status(9))
16516 [8830][2] reports Exit
16517 Feb 21 06:49:06.763 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5f9709ef-64f0-4c46-93f7-f079b14036a4, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
16518 001/300 Read block 2566 len 32768
16519 002/300 Read block 1087 len 20480
16520 003/300 Read block 2121 len 28672
16521 004/300 Read block 447 len 16384
16522 005/300 Write block 3116 len 4096 data: 3
16523 006/300 Write block 245 len 20480 data: 2 2 2 3 3
16524 007/300 Flush
16525 008/300 Read block 4632 len 8192
16526 009/300 Write block 1352 len 24576 data: 4 4 3 3 3 3
16527 010/300 Read block 4738 len 16384
16528 011/300 Write block 2312 len 32768 data: 3 3 3 3 3 3 3 2
16529 012/300 Write block 641 len 16384 data: 3 2 2 3
16530 013/300 Read block 4389 len 12288
16531 014/300 Write block 630 len 28672 data: 2 2 2 2 2 2 2
16532 015/300 Read block 2941 len 40960
16533 016/300 Write block 3476 len 24576 data: 4 4 4 5 5 5
16534 017/300 Read block 3647 len 28672
16535 018/300 Read block 3236 len 8192
16536 019/300 Flush
16537 020/300 Write block 2223 len 20480 data: 3 3 3 5 5
16538 021/300 Write block 2233 len 32768 data: 5 5 5 6 5 5 5 4
16539 022/300 Read block 1497 len 8192
16540 023/300 Read block 752 len 8192
16541 024/300 Read block 3188 len 4096
16542 025/300 Read block 1653 len 36864
16543 026/300 Write block 3298 len 28672 data: 2 2 2 2 2 2 2
16544 027/300 Write block 1724 len 12288 data: 5 4 5
16545 028/300 Write block 1759 len 16384 data: 3 3 3 5
16546 029/300 Flush
16547 030/300 Write block 4570 len 40960 data: 7 6 5 6 3 3 4 4 3 3
16548 031/300 Read block 4744 len 4096
16549 032/300 Flush
16550 033/300 Flush
16551 034/300 Read block 2733 len 20480
16552 035/300 Read block 4769 len 4096
16553 036/300 Write block 2180 len 36864 data: 3 3 3 3 2 2 3 6 9
16554 037/300 Read block 3568 len 20480
16555 038/300 Read block 2284 len 16384
16556 039/300 Read block 1086 len 24576
16557 040/300 Read block 2079 len 40960
16558 041/300 Read block 3268 len 12288
16559 042/300 Write block 1048 len 8192 data: 3 3
16560 043/300 Write block 1095 len 8192 data: 4 3
16561 044/300 Write block 2420 len 40960 data: 4 5 4 6 7 6 6 4 4 5
16562 045/300 Read block 3993 len 32768
16563 046/300 Read block 3919 len 40960
16564 047/300 Read block 2664 len 4096
16565 048/300 Write block 2768 len 12288 data: 3 3 3
16566 049/300 Write block 3431 len 40960 data: 6 6 5 3 3 4 3 4 4 4
16567 050/300 Write block 1399 len 8192 data: 6 5
16568 051/300 Read block 1274 len 8192
16569 052/300 Write block 3228 len 16384 data: 6 5 5 5
16570 053/300 Read block 726 len 8192
16571 054/300 Read block 3724 len 12288
16572 055/300 Write block 867 len 20480 data: 3 3 3 4 5
16573 056/300 Write block 1908 len 28672 data: 5 5 6 6 6 6 6
16574 057/300 Read block 2346 len 40960
16575 058/300 Write block 586 len 8192 data: 4 4
16576 059/300 Write block 58 len 28672 data: 4 4 3 3 3 4 5
16577 060/300 Write block 1925 len 40960 data: 3 3 3 5 4 4 3 3 3 3
16578 061/300 Read block 3261 len 36864
16579 062/300 Write block 2560 len 20480 data: 4 4 4 3 3
16580 063/300 Read block 4372 len 36864
16581 064/300 Read block 3426 len 36864
16582 065/300 Read block 3233 len 16384
16583 066/300 Write block 1817 len 16384 data: 6 4 6 6
16584 067/300 Read block 516 len 40960
16585 068/300 Read block 3028 len 36864
16586 069/300 Write block 1505 len 28672 data: 4 4 3 3 3 2 2
16587 070/300 Read block 3705 len 4096
16588 071/300 Read block 582 len 36864
16589 072/300 Read block 3438 len 8192
16590 073/300 Flush
16591 074/300 Read block 3903 len 28672
16592 075/300 Read block 1917 len 36864
16593 076/300 Write block 1618 len 32768 data: 5 4 4 3 5 5 5 7
16594 077/300 Read block 4179 len 24576
16595 078/300 Read block 190 len 32768
16596 079/300 Read block 4502 len 16384
16597 080/300 Read block 742 len 16384
16598 081/300 Read block 2309 len 28672
16599 082/300 Read block 1065 len 24576
16600 083/300 Read block 2497 len 36864
16601 084/300 Write block 3681 len 20480 data: 3 2 2 2 2
16602 085/300 Write block 1683 len 16384 data: 3 5 5 5
16603 086/300 Flush
16604 087/300 Read block 4162 len 32768
16605 088/300 Read block 3599 len 12288
16606 089/300 Read block 4138 len 24576
16607 090/300 Read block 1310 len 12288
16608 091/300 Write block 2607 len 20480 data: 4 3 4 3 2
16609 092/300 Write block 710 len 16384 data: 2 2 2 2
16610 093/300 Write block 4796 len 36864 data: 3 3 3 3 3 3 3 3 2
16611 094/300 Read block 3543 len 28672
16612 095/300 Read block 1416 len 12288
16613 096/300 Write block 3840 len 36864 data: 5 6 7 7 6 5 5 5 4
16614 097/300 Write block 1066 len 36864 data: 3 3 4 4 4 4 4 4 4
16615 098/300 Read block 641 len 8192
16616 099/300 Read block 1457 len 32768
16617 100/300 Write block 4728 len 32768 data: 5 6 7 6 6 6 3 3
16618 101/300 Read block 3092 len 4096
16619 102/300 Write block 3135 len 28672 data: 7 8 7 7 7 6 5
16620 103/300 Write block 4447 len 24576 data: 4 4 4 4 4 5
16621 104/300 Read block 3416 len 16384
16622 105/300 Read block 4552 len 32768
16623 106/300 Write block 3925 len 24576 data: 2 2 2 4 4 4
16624 107/300 Write block 3030 len 16384 data: 3 4 4 5
16625 108/300 Write block 4004 len 32768 data: 5 5 6 6 7 5 5 5
16626 109/300 Write block 2248 len 12288 data: 3 3 3
16627 110/300 Flush
16628 111/300 Read block 665 len 28672
16629 112/300 Read block 255 len 8192
16630 113/300 Read block 1600 len 12288
16631 114/300 Read block 1981 len 28672
16632 115/300 Read block 259 len 8192
16633 116/300 Read block 4908 len 24576
16634 117/300 Flush
16635 118/300 Flush
16636 119/300 Write block 433 len 20480 data: 8 5 4 4 3
16637 120/300 Write block 4648 len 16384 data: 7 7 7 8
16638 121/300 Write block 4344 len 40960 data: 2 2 3 3 3 3 3 4 4 4
16639 122/300 Read block 302 len 20480
16640 123/300 Write block 4425 len 16384 data: 3 3 4 4
16641 124/300 Flush
16642 125/300 Flush
16643 126/300 Write block 4723 len 16384 data: 2 2 3 3
16644 127/300 Write block 1232 len 24576 data: 4 4 4 3 2 3
16645 128/300 Flush
16646 129/300 Write block 3363 len 28672 data: 2 2 2 2 2 2 2
16647 130/300 Write block 1067 len 32768 data: 4 5 5 5 5 5 5 5
16648 131/300 Write block 252 len 8192 data: 3 5
16649 132/300 Write block 3913 len 32768 data: 5 5 4 4 3 4 4 4
16650 133/300 Read block 4652 len 4096
16651 134/300 Write block 4596 len 28672 data: 3 2 3 3 3 5 5
16652 135/300 Write block 3009 len 16384 data: 5 7 6 6
16653 136/300 Write block 114 len 8192 data: 2 3
16654 137/300 Write block 4808 len 4096 data: 2
16655 138/300 Write block 3585 len 16384 data: 12 11 8 7
16656 139/300 Write block 1593 len 8192 data: 2 2
16657 140/300 Write block 787 len 40960 data: 8 6 6 5 5 5 5 5 5 3
16658 141/300 Flush
16659 142/300 Write block 2953 len 4096 data: 8
16660 143/300 Read block 82 len 40960
16661 144/300 Write block 918 len 8192 data: 2 2
16662 145/300 Write block 4053 len 36864 data: 4 5 6 6 6 6 5 5 5
16663 146/300 Read block 4244 len 12288
16664 147/300 Write block 2912 len 16384 data: 4 2 2 3
16665 148/300 Read block 3245 len 28672
16666 149/300 Write block 3732 len 8192 data: 2 2
16667 150/300 Read block 3294 len 16384
16668 151/300 Read block 4773 len 4096
16669 152/300 Read block 3158 len 20480
16670 153/300 Write block 131 len 28672 data: 3 3 3 3 3 3 3
16671 154/300 Read block 1829 len 16384
16672 155/300 Write block 4003 len 28672 data: 4 6 6 7 7 8 6
16673 156/300 Write block 3185 len 12288 data: 2 2 3
16674 157/300 Write block 368 len 20480 data: 3 4 4 4 5
16675 158/300 Read block 490 len 28672
16676 159/300 Read block 403 len 40960
16677 160/300 Write block 1758 len 32768 data: 5 4 4 4 6 5 4 4
16678 161/300 Write block 4481 len 36864 data: 4 4 4 4 3 3 3 2 2
16679 162/300 Read block 269 len 4096
16680 163/300 Write block 1831 len 40960 data: 3 3 3 3 3 3 3 3 4 5
16681 164/300 Read block 4012 len 8192
16682 165/300 Read block 626 len 12288
16683 166/300 Read block 3440 len 4096
16684 167/300 Write block 219 len 28672 data: 5 4 3 3 2 3 3
16685 168/300 Write block 1462 len 20480 data: 3 4 4 4 4
16686 169/300 Write block 3761 len 12288 data: 2 3 3
16687 170/300 Write block 58 len 12288 data: 5 5 4
16688 171/300 Read block 2540 len 24576
16689 172/300 Write block 4218 len 12288 data: 4 4 4
16690 173/300 Write block 757 len 16384 data: 4 4 5 4
16691 174/300 Write block 1365 len 16384 data: 3 3 3 3
16692 175/300 Write block 666 len 28672 data: 6 6 6 7 7 7 7
16693 176/300 Write block 3150 len 8192 data: 3 3
16694 177/300 Flush
16695 178/300 Flush
16696 179/300 Read block 85 len 28672
16697 180/300 Write block 3200 len 36864 data: 3 3 3 2 2 2 3 3 2
16698 181/300 Write block 1353 len 4096 data: 5
16699 182/300 Write block 389 len 8192 data: 2 3
16700 183/300 Write block 2494 len 32768 data: 3 5 4 3 2 2 3 3
16701 184/300 Write block 936 len 28672 data: 2 2 2 2 2 2 2
16702 185/300 Read block 763 len 36864
16703 186/300 Write block 4206 len 32768 data: 4 5 4 4 5 5 6 5
16704 187/300 Read block 2349 len 4096
16705 188/300 Read block 1066 len 12288
16706 189/300 Read block 3728 len 40960
16707 190/300 Read block 4574 len 12288
16708 191/300 Write block 624 len 12288 data: 3 4 3
16709 192/300 Read block 571 len 12288
16710 193/300 Read block 2309 len 16384
16711 194/300 Read block 3177 len 4096
16712 195/300 Write block 3676 len 40960 data: 2 3 5 4 3 4 3 3 3 3
16713 196/300 Write block 4876 len 4096 data: 5
16714 197/300 Write block 3390 len 28672 data: 3 3 4 4 2 2 2
16715 198/300 Write block 4749 len 8192 data: 5 4
16716 199/300 Write block 4472 len 16384 data: 6 6 5 5
16717 200/300 Write block 1329 len 12288 data: 3 3 4
16718 201/300 Read block 2568 len 40960
16719 202/300 Write block 4395 len 8192 data: 5 5
16720 203/300 Read block 391 len 28672
16721 204/300 Read block 2200 len 32768
16722 205/300 Write block 4033 len 16384 data: 4 4 4 5
16723 206/300 Read block 2909 len 40960
16724 207/300 Flush
16725 208/300 Write block 3007 len 8192 data: 6 5
16726 209/300 Write block 3235 len 36864 data: 3 2 2 2 2 2 2 2 3
16727 210/300 Flush
16728 211/300 Write block 2937 len 24576 data: 3 3 4 3 3 3
16729 212/300 Read block 3258 len 8192
16730 213/300 Read block 1390 len 8192
16731 214/300 Flush
16732 215/300 Read block 452 len 12288
16733 216/300 Write block 1755 len 32768 data: 5 5 4 6 5 5 5 7
16734 217/300 Read block 389 len 20480
16735 218/300 Write block 4134 len 24576 data: 2 2 2 2 2 2
16736 219/300 Read block 2706 len 4096
16737 220/300 Read block 3925 len 24576
16738 221/300 Flush
16739 222/300 Read block 2251 len 12288
16740 223/300 Read block 163 len 12288
16741 224/300 Write block 2096 len 20480 data: 3 4 4 4 4
16742 225/300 Read block 924 len 36864
16743 226/300 Read block 4326 len 32768
16744 227/300 Flush
16745 228/300 Write block 4680 len 8192 data: 2 3
16746 229/300 Read block 1256 len 20480
16747 230/300 Write block 3915 len 40960 data: 5 5 4 5 5 5 4 3 2 2
16748 231/300 Read block 2052 len 16384
16749 232/300 Flush
16750 233/300 Write block 1355 len 16384 data: 4 4 4 3
16751 234/300 Read block 69 len 8192
16752 235/300 Read block 3858 len 28672
16753 236/300 Read block 2677 len 36864
16754 237/300 Write block 4702 len 20480 data: 4 4 3 3 3
16755 238/300 Write block 1390 len 40960 data: 4 4 4 3 3 2 3 4 5 7
16756 239/300 Flush
16757 240/300 Read block 3267 len 4096
16758 241/300 Write block 870 len 8192 data: 5 6
16759 242/300 Write block 3752 len 40960 data: 2 2 2 2 2 2 2 2 2 3
16760 243/300 Write block 473 len 28672 data: 3 2 2 3 3 2 2
16761 244/300 Write block 2751 len 8192 data: 7 8
16762 245/300 Read block 1998 len 4096
16763 246/300 Write block 1272 len 12288 data: 3 3 3
16764 247/300 Read block 2414 len 8192
16765 248/300 Read block 4032 len 28672
16766 249/300 Write block 3982 len 32768 data: 4 4 5 4 3 5 4 4
16767 250/300 Read block 1164 len 36864
16768 251/300 Read block 3203 len 12288
16769 252/300 Flush
16770 253/300 Write block 1272 len 32768 data: 4 4 4 3 4 4 4 4
16771 254/300 Read block 4482 len 28672
16772 255/300 Read block 160 len 40960
16773 256/300 Flush
16774 257/300 Write block 1632 len 20480 data: 5 3 3 2 2
16775 258/300 Flush
16776 259/300 Write block 4833 len 8192 data: 5 5
16777 260/300 Write block 3364 len 8192 data: 3 3
16778 261/300 Write block 2296 len 36864 data: 2 3 2 2 2 3 4 6 5
16779 262/300 Write block 4773 len 24576 data: 3 3 3 4 4 4
16780 263/300 Write block 4851 len 16384 data: 2 2 3 3
16781 264/300 Write block 4032 len 8192 data: 4 5
16782 265/300 Write block 3557 len 32768 data: 3 3 3 2 2 2 3 4
16783 266/300 Read block 1621 len 40960
16784 267/300 Read block 1750 len 20480
16785 268/300 Read block 2735 len 24576
16786 269/300 Write block 984 len 16384 data: 5 5 5 5
16787 270/300 Read block 4867 len 24576
16788 271/300 Read block 3641 len 16384
16789 272/300 Read block 54 len 12288
16790 273/300 Read block 1604 len 32768
16791 274/300 Write block 3811 len 4096 data: 4
16792 275/300 Read block 4617 len 20480
16793 276/300 Flush
16794 277/300 Flush
16795 278/300 Read block 4909 len 24576
16796 279/300 Write block 2775 len 40960 data: 2 3 3 4 4 4 4 5 5 5
16797 280/300 Read block 3984 len 16384
16798 281/300 Write block 3019 len 32768 data: 2 2 3 3 3 3 3 3
16799 282/300 Write block 201 len 32768 data: 3 4 3 3 2 3 3 4
16800 283/300 Read block 4310 len 40960
16801 284/300 Read block 3106 len 8192
16802 285/300 Write block 1652 len 32768 data: 2 2 2 4 4 4 5 5
16803 286/300 Write block 2311 len 16384 data: 2 4 4 4
16804 287/300 Write block 2695 len 32768 data: 3 3 4 3 3 2 3 3
16805 288/300 Read block 584 len 12288
16806 289/300 Write block 529 len 12288 data: 2 2 2
16807 290/300 Read block 2284 len 12288
16808 291/300 Write block 1273 len 36864 data: 5 5 4 5 5 5 5 3 2
16809 292/300 Write block 2458 len 32768 data: 4 4 4 4 5 5 5 3
16810 293/300 Read block 1579 len 4096
16811 294/300 Write block 4899 len 4096 data: 6
16812 295/300 Read block 3757 len 32768
16813 296/300 Write block 4433 len 28672 data: 4 3 3 3 3 2 2
16814 297/300 Read block 2121 len 40960
16815 298/300 Read block 4412 len 20480
16816 299/300 Read block 1788 len 32768
16817 300/300 Read block 3388 len 32768
16818 301/300 Read block 766 len 20480
16819 Main task has work to do, go find it
16820 got dsc Start(2)
16821 start 2
16822 Feb 21 06:49:06.992 INFO request completed, latency_us: 235, response_code: 204, uri: /start/cid/2, method: POST, req_id: 96b68a2f-6067-4fea-96db-b01189c7e754, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
16823 Replay: started 2, returned:Ok(())
16824 ----------------------------------------------------------------
16825 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16826 States: Active Active Offline
16827 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16828 5227 Acked Read 8 Done Done Sent false
16829 5228 Acked Read 5 Done Done Sent false
16830 5229 Acked Read 7 Done Done Sent false
16831 5230 Acked Read 4 Done Done Sent false
16832 5231 Acked Write 1 Done Done Sent false
16833 5232 Acked Write 5 Done Done Sent false
16834 5233 Acked Flush 0 Done Done Sent false
16835 5234 Acked Read 2 Done Done Sent false
16836 5235 Acked Write 6 Done Done Sent false
16837 5236 Acked Read 4 Done Done Sent false
16838 5237 Acked Write 8 Done Done Sent false
16839 5238 Acked Write 4 Done Done Sent false
16840 5239 Acked Read 3 Done Done Sent false
16841 5240 Acked Write 7 Done Done Sent false
16842 5241 Acked Read 10 Done Done Sent false
16843 5242 Acked Write 6 Done Done Sent false
16844 5243 Acked Read 7 Done Done Sent false
16845 5244 Acked Read 2 Done Done Sent false
16846 5245 Acked Flush 0 Done Done Sent false
16847 5246 Acked Write 5 Done Done Sent false
16848 5247 Acked Write 8 Done Done Sent false
16849 5248 Acked Read 2 Done Done Sent false
16850 5249 Acked Read 2 Done Done Sent false
16851 5250 Acked Read 1 Done Done Sent false
16852 5251 Acked Read 9 Done Done Sent false
16853 5252 Acked Write 7 Done Done Sent false
16854 5253 Acked Write 3 Done Done Sent false
16855 5254 Acked Write 4 Done Done Sent false
16856 5255 Acked Flush 0 Done Done Sent false
16857 5256 Acked Write 10 Done Done Sent false
16858 5257 Acked Read 1 Done Done Sent false
16859 5258 Acked Flush 0 Done Done Sent false
16860 5259 Acked Flush 0 Done Done Sent false
16861 5260 Acked Read 5 Done Done Sent false
16862 5261 Acked Read 1 Done Done Sent false
16863 5262 Acked Write 9 Done Done Sent false
16864 5263 Acked Read 5 Done Done Sent false
16865 5264 Acked Read 4 Done Done Sent false
16866 5265 Acked Read 6 Done Done Sent false
16867 5266 Acked Read 10 Done Done Sent false
16868 5267 Acked Read 3 Done Done Sent false
16869 5268 Acked Write 2 Done Done Sent false
16870 5269 Acked Write 2 Done Done Sent false
16871 5270 Acked Write 10 Done Done Sent false
16872 5271 Acked Read 8 Done Done Sent false
16873 5272 Acked Read 10 Done Done Sent false
16874 5273 Acked Read 1 Done Done Sent false
16875 5274 Acked Write 3 Done Done Sent false
16876 5275 Acked Write 10 Done Done Sent false
16877 5276 Acked Write 2 Done Done Sent false
16878 5277 Acked Read 2 Done Done Sent false
16879 5278 Acked Write 4 Done Done Sent false
16880 5279 Acked Read 2 Done Done Sent false
16881 5280 Acked Read 3 Done Done Sent false
16882 5281 Acked Write 5 Done Done Sent false
16883 5282 Acked Write 7 Done Done Sent false
16884 5283 Acked Read 10 Done Done Sent false
16885 5284 Acked Write 2 Done Done Sent false
16886 5285 Acked Write 7 Done Done Sent false
16887 5286 Acked Write 10 Done Done Sent false
16888 5287 Acked Read 9 Done Done Sent false
16889 5288 Acked Write 5 Done Done Sent false
16890 5289 Acked Read 9 Done Done Sent false
16891 5290 Acked Read 9 Done Done Sent false
16892 5291 Acked Read 4 Done Done Sent false
16893 5292 Acked Write 4 Done Done Sent false
16894 5293 Acked Read 10 Done Done Sent false
16895 5294 Acked Read 9 Done Done Sent false
16896 5295 Acked Write 7 Done Done Sent false
16897 5296 Acked Read 1 Done Done Sent false
16898 5297 Acked Read 9 Done Done Sent false
16899 5298 Acked Read 2 Done Done Sent false
16900 5299 Acked Flush 0 Done Done Sent false
16901 5300 Acked Read 7 Done Done Sent false
16902 5301 Acked Read 9 Done Done Sent false
16903 5302 Acked Write 8 Done Done Sent false
16904 5303 Acked Read 6 Done Done Sent false
16905 5304 Acked Read 8 Done Done Sent false
16906 5305 Acked Read 4 Done Done Sent false
16907 5306 Acked Read 4 Done Done Sent false
16908 5307 Acked Read 7 Done Done Sent false
16909 5308 Acked Read 6 Done Done Sent false
16910 5309 Acked Read 9 Done Done Sent false
16911 5310 Acked Write 5 Done Done Sent false
16912 5311 Acked Write 4 Done Done Sent false
16913 5312 Acked Flush 0 Done Done Sent false
16914 5313 Acked Read 8 Done Done Sent false
16915 5314 Acked Read 3 Done Done Sent false
16916 5315 Acked Read 6 Done Done Sent false
16917 5316 Acked Read 3 Done Done Sent false
16918 5317 Acked Write 5 Done Done Sent false
16919 5318 Acked Write 4 Done Done Sent false
16920 5319 Acked Write 9 Done Done Sent false
16921 5320 Acked Read 7 Done Done Sent false
16922 5321 Acked Read 3 Done Done Sent false
16923 5322 Acked Write 9 Done Done Sent false
16924 5323 Acked Write 9 Done Done Sent false
16925 5324 Acked Read 2 Done Done Sent false
16926 5325 Acked Read 8 Done Done Sent false
16927 5326 Acked Write 8 Done Done Sent false
16928 5327 Acked Read 1 Done Done Sent false
16929 5328 Acked Write 7 Done Done Sent false
16930 5329 Acked Write 6 Done Done Sent false
16931 5330 Acked Read 4 Done Done Sent false
16932 5331 Acked Read 8 Done Done Sent false
16933 5332 Acked Write 6 Done Done Sent false
16934 5333 Acked Write 4 Done Done Sent false
16935 5334 Acked Write 8 Done Done Sent false
16936 5335 Acked Write 3 Done Done Sent false
16937 5336 Acked Flush 0 Done Done Sent false
16938 5337 Acked Read 7 Done Done Sent false
16939 5338 Acked Read 2 Done Done Sent false
16940 5339 Acked Read 3 Done Done Sent false
16941 5340 Acked Read 7 Done Done Sent false
16942 5341 Acked Read 2 Done Done Sent false
16943 5342 Acked Read 6 Done Done Sent false
16944 5343 Acked Flush 0 Done Done Sent false
16945 5344 Acked Flush 0 Done Done Sent false
16946 5345 Acked Write 5 Done Done Sent false
16947 5346 Acked Write 4 Done Done Sent false
16948 5347 Acked Write 10 Done Done Sent false
16949 5348 Acked Read 5 Done Done Sent false
16950 5349 Acked Write 4 Done Done Sent false
16951 5350 Acked Flush 0 Done Done Sent false
16952 5351 Acked Flush 0 Done Done Sent false
16953 5352 Acked Write 4 Done Done Sent false
16954 5353 Acked Write 6 Done Done Sent false
16955 5354 Acked Flush 0 Done Done Sent false
16956 5355 Acked Write 7 Done Done Sent false
16957 5356 Acked Write 8 Done Done Sent false
16958 5357 Acked Write 2 Done Done Sent false
16959 5358 Acked Write 8 Done Done Sent false
16960 5359 Acked Read 1 Done Done Sent false
16961 5360 Acked Write 7 Done Done Sent false
16962 5361 Acked Write 4 Done Done Sent false
16963 5362 Acked Write 2 Done Done Sent false
16964 5363 Acked Write 1 Done Done Sent false
16965 5364 Acked Write 4 Done Done Sent false
16966 5365 Acked Write 2 Done Done Sent false
16967 5366 Acked Write 10 Done Done Sent false
16968 5367 Acked Flush 0 Done Done Sent false
16969 5368 Acked Write 1 Done Done Sent false
16970 5369 Acked Read 10 Done Done Sent false
16971 5370 Acked Write 2 Done Done Sent false
16972 5371 Acked Write 9 Done Done Sent false
16973 5372 Acked Read 3 Done Done Sent false
16974 5373 Acked Write 4 Done Done Sent false
16975 5374 Acked Read 7 Done Done Sent false
16976 5375 Acked Write 2 Done Done Sent false
16977 5376 Acked Read 4 Done Done Sent false
16978 5377 Acked Read 1 Done Done Sent false
16979 5378 Acked Read 5 Done Done Sent false
16980 5379 Acked Write 7 Done Done Sent false
16981 5380 Acked Read 4 Done Done Sent false
16982 5381 Acked Write 7 Done Done Sent false
16983 5382 Acked Write 3 Done Done Sent false
16984 5383 Acked Write 5 Done Done Sent false
16985 5384 Acked Read 7 Done Done Sent false
16986 5385 Acked Read 10 Done Done Sent false
16987 5386 Acked Write 8 Done Done Sent false
16988 5387 Acked Write 9 Done Done Sent false
16989 5388 Acked Read 1 Done Done Sent false
16990 5389 Acked Write 10 Done Done Sent false
16991 5390 Acked Read 2 Done Done Sent false
16992 5391 Acked Read 3 Done Done Sent false
16993 5392 Acked Read 1 Done Done Sent false
16994 5393 Acked Write 7 Done Done Sent false
16995 5394 Acked Write 5 Done Done Sent false
16996 5395 Acked Write 3 Done Done Sent false
16997 5396 Acked Write 3 Done Done Sent false
16998 5397 Acked Read 6 Done Done Sent false
16999 5398 Acked Write 3 Done Done Sent false
17000 5399 Acked Write 4 Done Done Sent false
17001 5400 Acked Write 4 Done Done Sent false
17002 5401 Acked Write 7 Done Done Sent false
17003 5402 Acked Write 2 Done Done Sent false
17004 5403 Acked Flush 0 Done Done Sent false
17005 5404 Acked Flush 0 Done Done Sent false
17006 5405 Acked Read 7 Done Done Sent false
17007 5406 Acked Write 9 Done Done Sent false
17008 5407 Acked Write 1 Done Done Sent false
17009 5408 Acked Write 2 Done Done Sent false
17010 5409 Acked Write 8 Done Done Sent false
17011 5410 Acked Write 7 Done Done Sent false
17012 5411 Acked Read 9 Done Done Sent false
17013 5412 Acked Write 8 Done Done Sent false
17014 5413 Acked Read 1 Done Done Sent false
17015 5414 Acked Read 3 Done Done Sent false
17016 5415 Acked Read 10 Done Done Sent false
17017 5416 Acked Read 3 Done Done Sent false
17018 5417 Acked Write 3 Done Done Sent false
17019 5418 Acked Read 3 Done Done Sent false
17020 5419 Acked Read 4 Done Done Sent false
17021 5420 Acked Read 1 Done Done Sent false
17022 5421 Acked Write 10 Done Done Sent false
17023 5422 Acked Write 1 Done Done Sent false
17024 5423 Acked Write 7 Done Done Sent false
17025 5424 Acked Write 2 Done Done Sent false
17026 5425 Acked Write 4 Done Done Sent false
17027 5426 Acked Write 3 Done Done Sent false
17028 5427 Acked Read 10 Done Done Sent false
17029 5428 Acked Write 2 Done Done Sent false
17030 5429 Acked Read 7 Done Done Sent false
17031 5430 Acked Read 8 Done Done Sent false
17032 5431 Acked Write 4 Done Done Sent false
17033 5432 Acked Read 10 Done Done Sent false
17034 5433 Acked Flush 0 Done Done Sent false
17035 5434 Acked Write 2 Done Done Sent false
17036 5435 Acked Write 9 Done Done Sent false
17037 5436 Acked Flush 0 Done Done Sent false
17038 5437 Acked Write 6 Done Done Sent false
17039 5438 Acked Read 2 Done Done Sent false
17040 5439 Acked Read 2 Done Done Sent false
17041 5440 Acked Flush 0 Done Done Sent false
17042 5441 Acked Read 3 Done Done Sent false
17043 5442 Acked Write 8 Done Done Sent false
17044 5443 Acked Read 5 Done Done Sent false
17045 5444 Acked Write 6 Done Done Sent false
17046 5445 Acked Read 1 Done Done Sent false
17047 5446 Acked Read 6 Done Done Sent false
17048 5447 Acked Flush 0 Done Done Sent false
17049 5448 Acked Read 3 Done Done Sent false
17050 5449 Acked Read 3 Done Done Sent false
17051 5450 Acked Write 5 Done Done Sent false
17052 5451 Acked Read 9 Done Done Sent false
17053 5452 Acked Read 8 Done Done Sent false
17054 5453 Acked Flush 0 Done Done Sent false
17055 5454 Acked Write 2 Done Done Sent false
17056 5455 Acked Read 5 Done Done Sent false
17057 5456 Acked Write 10 Done Done Sent false
17058 5457 Acked Read 4 Done Done Sent false
17059 5458 Acked Flush 0 Done Done Sent false
17060 5459 Acked Write 4 Done Done Sent false
17061 5460 Acked Read 2 Done Done Sent false
17062 5461 Acked Read 7 Done Done Sent false
17063 5462 Acked Read 9 Done Done Sent false
17064 5463 Acked Write 5 Done Done Sent false
17065 5464 Acked Write 10 Done Done Sent false
17066 5465 Acked Flush 0 Done Done Sent false
17067 5466 Acked Read 1 Done Done Sent false
17068 5467 Acked Write 2 Done Done Sent false
17069 5468 Acked Write 10 Done Done Sent false
17070 5469 Acked Write 7 Done Done Sent false
17071 5470 Acked Write 2 Done Done Sent false
17072 5471 Acked Read 1 Done Done Sent false
17073 5472 Acked Write 3 Done Done Sent false
17074 5473 Acked Read 2 Done Done Sent false
17075 5474 Acked Read 7 Done Done Sent false
17076 5475 Acked Write 8 Done Done Sent false
17077 5476 Acked Read 9 Done Done Sent false
17078 5477 Acked Read 3 Done Done Sent false
17079 5478 Acked Flush 0 Done Done Sent false
17080 5479 Acked Write 8 Done Done Sent false
17081 5480 Acked Read 7 Done Done Sent false
17082 5481 Acked Read 10 Done Done Sent false
17083 5482 Acked Flush 0 Done Done Sent false
17084 5483 Acked Write 5 Done Done Sent false
17085 5484 Acked Flush 0 Done Done Sent false
17086 5485 Acked Write 2 Done Done Sent false
17087 5486 Acked Write 2 Done Done Sent false
17088 5487 Acked Write 9 Done Done Sent false
17089 5488 Acked Write 6 Done Done Sent false
17090 5489 Acked Write 4 Done Done Sent false
17091 5490 Acked Write 2 Done Done Sent false
17092 5491 Acked Write 8 Done Done Sent false
17093 5492 Acked Read 10 Done Done Sent false
17094 5493 Acked Read 5 Done Done Sent false
17095 5494 Acked Read 6 Done Done Sent false
17096 5495 Acked Write 4 Done Done Sent false
17097 5496 Acked Read 6 Done Done Sent false
17098 5497 Acked Read 4 Done Done Sent false
17099 5498 Acked Read 3 Done Done Sent false
17100 5499 Acked Read 8 Done Done Sent false
17101 5500 Acked Write 1 Done Done Sent false
17102 5501 Acked Read 5 Done Done Sent false
17103 5502 Acked Flush 0 Done Done Sent false
17104 5503 Acked Flush 0 Done Done Sent false
17105 5504 Acked Read 6 Done Done Sent false
17106 5505 Acked Write 10 Done Done Sent false
17107 5506 Acked Read 4 Done Done Sent false
17108 5507 Acked Write 8 Done Done Sent false
17109 5508 Acked Write 8 Done Done Sent false
17110 5509 Acked Read 10 Done Done Sent false
17111 5510 Acked Read 2 Done Done Sent false
17112 5511 Acked Write 8 Done Done Sent false
17113 5512 Acked Write 4 Done Done Sent false
17114 5513 Acked Write 8 Done Done Sent false
17115 5514 Acked Read 3 Done Done Sent false
17116 5515 Acked Write 3 Done Done Sent false
17117 5516 Acked Read 3 Done Done Sent false
17118 5517 Acked Write 9 Done Done Sent false
17119 5518 Acked Write 8 Done Done Sent false
17120 5519 Acked Read 1 Done Done Sent false
17121 5520 Acked Write 1 Done Done Sent false
17122 5521 Acked Read 8 Done Done Sent false
17123 5522 Acked Write 7 Done Done Sent false
17124 5523 Acked Read 10 Done Done Sent false
17125 5524 Acked Read 5 Done Done Sent false
17126 5525 Acked Read 8 Done Done Sent false
17127 5526 Acked Read 8 Done Done Sent false
17128 5527 Acked Read 5 Done Done Sent false
17129 STATES DS:0 DS:1 DS:2 TOTAL
17130 Sent 0 0 301 301
17131 Done 301 301 0 602
17132 Skipped 0 0 0 0
17133 Error 0 0 0 0
17134 Last Flush: Some(JobId(5503)) Some(JobId(5503)) Some(JobId(5226))
17135 Downstairs last five completed: 5226 5225 5224 5223 5222
17136 Upstairs last five completed: 5527 5526 5525 5524 5523
17137 CLIENT: Up:0 ds:301 act:2
171382025-02-21T06:49:07.495ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
171392025-02-21T06:49:08.005ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
171402025-02-21T06:49:09.006ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
171412025-02-21T06:49:10.007ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
17142 ----------------------------------------------------------------
17143 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17144 States: Active Active Offline
17145 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17146 5227 Acked Read 8 Done Done Sent false
17147 5228 Acked Read 5 Done Done Sent false
17148 5229 Acked Read 7 Done Done Sent false
17149 5230 Acked Read 4 Done Done Sent false
17150 5231 Acked Write 1 Done Done Sent false
17151 5232 Acked Write 5 Done Done Sent false
17152 5233 Acked Flush 0 Done Done Sent false
17153 5234 Acked Read 2 Done Done Sent false
17154 5235 Acked Write 6 Done Done Sent false
17155 5236 Acked Read 4 Done Done Sent false
17156 5237 Acked Write 8 Done Done Sent false
17157 5238 Acked Write 4 Done Done Sent false
17158 5239 Acked Read 3 Done Done Sent false
17159 5240 Acked Write 7 Done Done Sent false
17160 5241 Acked Read 10 Done Done Sent false
17161 5242 Acked Write 6 Done Done Sent false
17162 5243 Acked Read 7 Done Done Sent false
17163 5244 Acked Read 2 Done Done Sent false
17164 5245 Acked Flush 0 Done Done Sent false
17165 5246 Acked Write 5 Done Done Sent false
17166 5247 Acked Write 8 Done Done Sent false
17167 5248 Acked Read 2 Done Done Sent false
17168 5249 Acked Read 2 Done Done Sent false
17169 5250 Acked Read 1 Done Done Sent false
17170 5251 Acked Read 9 Done Done Sent false
17171 5252 Acked Write 7 Done Done Sent false
17172 5253 Acked Write 3 Done Done Sent false
17173 5254 Acked Write 4 Done Done Sent false
17174 5255 Acked Flush 0 Done Done Sent false
17175 5256 Acked Write 10 Done Done Sent false
17176 5257 Acked Read 1 Done Done Sent false
17177 5258 Acked Flush 0 Done Done Sent false
17178 5259 Acked Flush 0 Done Done Sent false
17179 5260 Acked Read 5 Done Done Sent false
17180 5261 Acked Read 1 Done Done Sent false
17181 5262 Acked Write 9 Done Done Sent false
17182 5263 Acked Read 5 Done Done Sent false
17183 5264 Acked Read 4 Done Done Sent false
17184 5265 Acked Read 6 Done Done Sent false
17185 5266 Acked Read 10 Done Done Sent false
17186 5267 Acked Read 3 Done Done Sent false
17187 5268 Acked Write 2 Done Done Sent false
17188 5269 Acked Write 2 Done Done Sent false
17189 5270 Acked Write 10 Done Done Sent false
17190 5271 Acked Read 8 Done Done Sent false
17191 5272 Acked Read 10 Done Done Sent false
17192 5273 Acked Read 1 Done Done Sent false
17193 5274 Acked Write 3 Done Done Sent false
17194 5275 Acked Write 10 Done Done Sent false
17195 5276 Acked Write 2 Done Done Sent false
17196 5277 Acked Read 2 Done Done Sent false
17197 5278 Acked Write 4 Done Done Sent false
17198 5279 Acked Read 2 Done Done Sent false
17199 5280 Acked Read 3 Done Done Sent false
17200 5281 Acked Write 5 Done Done Sent false
17201 5282 Acked Write 7 Done Done Sent false
17202 5283 Acked Read 10 Done Done Sent false
17203 5284 Acked Write 2 Done Done Sent false
17204 5285 Acked Write 7 Done Done Sent false
17205 5286 Acked Write 10 Done Done Sent false
17206 5287 Acked Read 9 Done Done Sent false
17207 5288 Acked Write 5 Done Done Sent false
17208 5289 Acked Read 9 Done Done Sent false
17209 5290 Acked Read 9 Done Done Sent false
17210 5291 Acked Read 4 Done Done Sent false
17211 5292 Acked Write 4 Done Done Sent false
17212 5293 Acked Read 10 Done Done Sent false
17213 5294 Acked Read 9 Done Done Sent false
17214 5295 Acked Write 7 Done Done Sent false
17215 5296 Acked Read 1 Done Done Sent false
17216 5297 Acked Read 9 Done Done Sent false
17217 5298 Acked Read 2 Done Done Sent false
17218 5299 Acked Flush 0 Done Done Sent false
17219 5300 Acked Read 7 Done Done Sent false
17220 5301 Acked Read 9 Done Done Sent false
17221 5302 Acked Write 8 Done Done Sent false
17222 5303 Acked Read 6 Done Done Sent false
17223 5304 Acked Read 8 Done Done Sent false
17224 5305 Acked Read 4 Done Done Sent false
17225 5306 Acked Read 4 Done Done Sent false
17226 5307 Acked Read 7 Done Done Sent false
17227 5308 Acked Read 6 Done Done Sent false
17228 5309 Acked Read 9 Done Done Sent false
17229 5310 Acked Write 5 Done Done Sent false
17230 5311 Acked Write 4 Done Done Sent false
17231 5312 Acked Flush 0 Done Done Sent false
17232 5313 Acked Read 8 Done Done Sent false
17233 5314 Acked Read 3 Done Done Sent false
17234 5315 Acked Read 6 Done Done Sent false
17235 5316 Acked Read 3 Done Done Sent false
17236 5317 Acked Write 5 Done Done Sent false
17237 5318 Acked Write 4 Done Done Sent false
17238 5319 Acked Write 9 Done Done Sent false
17239 5320 Acked Read 7 Done Done Sent false
17240 5321 Acked Read 3 Done Done Sent false
17241 5322 Acked Write 9 Done Done Sent false
17242 5323 Acked Write 9 Done Done Sent false
17243 5324 Acked Read 2 Done Done Sent false
17244 5325 Acked Read 8 Done Done Sent false
17245 5326 Acked Write 8 Done Done Sent false
17246 5327 Acked Read 1 Done Done Sent false
17247 5328 Acked Write 7 Done Done Sent false
17248 5329 Acked Write 6 Done Done Sent false
17249 5330 Acked Read 4 Done Done Sent false
17250 5331 Acked Read 8 Done Done Sent false
17251 5332 Acked Write 6 Done Done Sent false
17252 5333 Acked Write 4 Done Done Sent false
17253 5334 Acked Write 8 Done Done Sent false
17254 5335 Acked Write 3 Done Done Sent false
17255 5336 Acked Flush 0 Done Done Sent false
17256 5337 Acked Read 7 Done Done Sent false
17257 5338 Acked Read 2 Done Done Sent false
17258 5339 Acked Read 3 Done Done Sent false
17259 5340 Acked Read 7 Done Done Sent false
17260 5341 Acked Read 2 Done Done Sent false
17261 5342 Acked Read 6 Done Done Sent false
17262 5343 Acked Flush 0 Done Done Sent false
17263 5344 Acked Flush 0 Done Done Sent false
17264 5345 Acked Write 5 Done Done Sent false
17265 5346 Acked Write 4 Done Done Sent false
17266 5347 Acked Write 10 Done Done Sent false
17267 5348 Acked Read 5 Done Done Sent false
17268 5349 Acked Write 4 Done Done Sent false
17269 5350 Acked Flush 0 Done Done Sent false
17270 5351 Acked Flush 0 Done Done Sent false
17271 5352 Acked Write 4 Done Done Sent false
17272 5353 Acked Write 6 Done Done Sent false
17273 5354 Acked Flush 0 Done Done Sent false
17274 5355 Acked Write 7 Done Done Sent false
17275 5356 Acked Write 8 Done Done Sent false
17276 5357 Acked Write 2 Done Done Sent false
17277 5358 Acked Write 8 Done Done Sent false
17278 5359 Acked Read 1 Done Done Sent false
17279 5360 Acked Write 7 Done Done Sent false
17280 5361 Acked Write 4 Done Done Sent false
17281 5362 Acked Write 2 Done Done Sent false
17282 5363 Acked Write 1 Done Done Sent false
17283 5364 Acked Write 4 Done Done Sent false
17284 5365 Acked Write 2 Done Done Sent false
17285 5366 Acked Write 10 Done Done Sent false
17286 5367 Acked Flush 0 Done Done Sent false
17287 5368 Acked Write 1 Done Done Sent false
17288 5369 Acked Read 10 Done Done Sent false
17289 5370 Acked Write 2 Done Done Sent false
17290 5371 Acked Write 9 Done Done Sent false
17291 5372 Acked Read 3 Done Done Sent false
17292 5373 Acked Write 4 Done Done Sent false
17293 5374 Acked Read 7 Done Done Sent false
17294 5375 Acked Write 2 Done Done Sent false
17295 5376 Acked Read 4 Done Done Sent false
17296 5377 Acked Read 1 Done Done Sent false
17297 5378 Acked Read 5 Done Done Sent false
17298 5379 Acked Write 7 Done Done Sent false
17299 5380 Acked Read 4 Done Done Sent false
17300 5381 Acked Write 7 Done Done Sent false
17301 5382 Acked Write 3 Done Done Sent false
17302 5383 Acked Write 5 Done Done Sent false
17303 5384 Acked Read 7 Done Done Sent false
17304 5385 Acked Read 10 Done Done Sent false
17305 5386 Acked Write 8 Done Done Sent false
17306 5387 Acked Write 9 Done Done Sent false
17307 5388 Acked Read 1 Done Done Sent false
17308 5389 Acked Write 10 Done Done Sent false
17309 5390 Acked Read 2 Done Done Sent false
17310 5391 Acked Read 3 Done Done Sent false
17311 5392 Acked Read 1 Done Done Sent false
17312 5393 Acked Write 7 Done Done Sent false
17313 5394 Acked Write 5 Done Done Sent false
17314 5395 Acked Write 3 Done Done Sent false
17315 5396 Acked Write 3 Done Done Sent false
17316 5397 Acked Read 6 Done Done Sent false
17317 5398 Acked Write 3 Done Done Sent false
17318 5399 Acked Write 4 Done Done Sent false
17319 5400 Acked Write 4 Done Done Sent false
17320 5401 Acked Write 7 Done Done Sent false
17321 5402 Acked Write 2 Done Done Sent false
17322 5403 Acked Flush 0 Done Done Sent false
17323 5404 Acked Flush 0 Done Done Sent false
17324 5405 Acked Read 7 Done Done Sent false
17325 5406 Acked Write 9 Done Done Sent false
17326 5407 Acked Write 1 Done Done Sent false
17327 5408 Acked Write 2 Done Done Sent false
17328 5409 Acked Write 8 Done Done Sent false
17329 5410 Acked Write 7 Done Done Sent false
17330 5411 Acked Read 9 Done Done Sent false
17331 5412 Acked Write 8 Done Done Sent false
17332 5413 Acked Read 1 Done Done Sent false
17333 5414 Acked Read 3 Done Done Sent false
17334 5415 Acked Read 10 Done Done Sent false
17335 5416 Acked Read 3 Done Done Sent false
17336 5417 Acked Write 3 Done Done Sent false
17337 5418 Acked Read 3 Done Done Sent false
17338 5419 Acked Read 4 Done Done Sent false
17339 5420 Acked Read 1 Done Done Sent false
17340 5421 Acked Write 10 Done Done Sent false
17341 5422 Acked Write 1 Done Done Sent false
17342 5423 Acked Write 7 Done Done Sent false
17343 5424 Acked Write 2 Done Done Sent false
17344 5425 Acked Write 4 Done Done Sent false
17345 5426 Acked Write 3 Done Done Sent false
17346 5427 Acked Read 10 Done Done Sent false
17347 5428 Acked Write 2 Done Done Sent false
17348 5429 Acked Read 7 Done Done Sent false
17349 5430 Acked Read 8 Done Done Sent false
17350 5431 Acked Write 4 Done Done Sent false
17351 5432 Acked Read 10 Done Done Sent false
17352 5433 Acked Flush 0 Done Done Sent false
17353 5434 Acked Write 2 Done Done Sent false
17354 5435 Acked Write 9 Done Done Sent false
17355 5436 Acked Flush 0 Done Done Sent false
17356 5437 Acked Write 6 Done Done Sent false
17357 5438 Acked Read 2 Done Done Sent false
17358 5439 Acked Read 2 Done Done Sent false
17359 5440 Acked Flush 0 Done Done Sent false
17360 5441 Acked Read 3 Done Done Sent false
17361 5442 Acked Write 8 Done Done Sent false
17362 5443 Acked Read 5 Done Done Sent false
17363 5444 Acked Write 6 Done Done Sent false
17364 5445 Acked Read 1 Done Done Sent false
17365 5446 Acked Read 6 Done Done Sent false
17366 5447 Acked Flush 0 Done Done Sent false
17367 5448 Acked Read 3 Done Done Sent false
17368 5449 Acked Read 3 Done Done Sent false
17369 5450 Acked Write 5 Done Done Sent false
17370 5451 Acked Read 9 Done Done Sent false
17371 5452 Acked Read 8 Done Done Sent false
17372 5453 Acked Flush 0 Done Done Sent false
17373 5454 Acked Write 2 Done Done Sent false
17374 5455 Acked Read 5 Done Done Sent false
17375 5456 Acked Write 10 Done Done Sent false
17376 5457 Acked Read 4 Done Done Sent false
17377 5458 Acked Flush 0 Done Done Sent false
17378 5459 Acked Write 4 Done Done Sent false
17379 5460 Acked Read 2 Done Done Sent false
17380 5461 Acked Read 7 Done Done Sent false
17381 5462 Acked Read 9 Done Done Sent false
17382 5463 Acked Write 5 Done Done Sent false
17383 5464 Acked Write 10 Done Done Sent false
17384 5465 Acked Flush 0 Done Done Sent false
17385 5466 Acked Read 1 Done Done Sent false
17386 5467 Acked Write 2 Done Done Sent false
17387 5468 Acked Write 10 Done Done Sent false
17388 5469 Acked Write 7 Done Done Sent false
17389 5470 Acked Write 2 Done Done Sent false
17390 5471 Acked Read 1 Done Done Sent false
17391 5472 Acked Write 3 Done Done Sent false
17392 5473 Acked Read 2 Done Done Sent false
17393 5474 Acked Read 7 Done Done Sent false
17394 5475 Acked Write 8 Done Done Sent false
17395 5476 Acked Read 9 Done Done Sent false
17396 5477 Acked Read 3 Done Done Sent false
17397 5478 Acked Flush 0 Done Done Sent false
17398 5479 Acked Write 8 Done Done Sent false
17399 5480 Acked Read 7 Done Done Sent false
17400 5481 Acked Read 10 Done Done Sent false
17401 5482 Acked Flush 0 Done Done Sent false
17402 5483 Acked Write 5 Done Done Sent false
17403 5484 Acked Flush 0 Done Done Sent false
17404 5485 Acked Write 2 Done Done Sent false
17405 5486 Acked Write 2 Done Done Sent false
17406 5487 Acked Write 9 Done Done Sent false
17407 5488 Acked Write 6 Done Done Sent false
17408 5489 Acked Write 4 Done Done Sent false
17409 5490 Acked Write 2 Done Done Sent false
17410 5491 Acked Write 8 Done Done Sent false
17411 5492 Acked Read 10 Done Done Sent false
17412 5493 Acked Read 5 Done Done Sent false
17413 5494 Acked Read 6 Done Done Sent false
17414 5495 Acked Write 4 Done Done Sent false
17415 5496 Acked Read 6 Done Done Sent false
17416 5497 Acked Read 4 Done Done Sent false
17417 5498 Acked Read 3 Done Done Sent false
17418 5499 Acked Read 8 Done Done Sent false
17419 5500 Acked Write 1 Done Done Sent false
17420 5501 Acked Read 5 Done Done Sent false
17421 5502 Acked Flush 0 Done Done Sent false
17422 5503 Acked Flush 0 Done Done Sent false
17423 5504 Acked Read 6 Done Done Sent false
17424 5505 Acked Write 10 Done Done Sent false
17425 5506 Acked Read 4 Done Done Sent false
17426 5507 Acked Write 8 Done Done Sent false
17427 5508 Acked Write 8 Done Done Sent false
17428 5509 Acked Read 10 Done Done Sent false
17429 5510 Acked Read 2 Done Done Sent false
17430 5511 Acked Write 8 Done Done Sent false
17431 5512 Acked Write 4 Done Done Sent false
17432 5513 Acked Write 8 Done Done Sent false
17433 5514 Acked Read 3 Done Done Sent false
17434 5515 Acked Write 3 Done Done Sent false
17435 5516 Acked Read 3 Done Done Sent false
17436 5517 Acked Write 9 Done Done Sent false
17437 5518 Acked Write 8 Done Done Sent false
17438 5519 Acked Read 1 Done Done Sent false
17439 5520 Acked Write 1 Done Done Sent false
17440 5521 Acked Read 8 Done Done Sent false
17441 5522 Acked Write 7 Done Done Sent false
17442 5523 Acked Read 10 Done Done Sent false
17443 5524 Acked Read 5 Done Done Sent false
17444 5525 Acked Read 8 Done Done Sent false
17445 5526 Acked Read 8 Done Done Sent false
17446 5527 Acked Read 5 Done Done Sent false
17447 STATES DS:0 DS:1 DS:2 TOTAL
17448 Sent 0 0 301 301
17449 Done 301 301 0 602
17450 Skipped 0 0 0 0
17451 Error 0 0 0 0
17452 Last Flush: Some(JobId(5503)) Some(JobId(5503)) Some(JobId(5226))
17453 Downstairs last five completed: 5226 5225 5224 5223 5222
17454 Upstairs last five completed: 5527 5526 5525 5524 5523
17455 CLIENT: Up:0 ds:301 act:2
174562025-02-21T06:49:11.497ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
174572025-02-21T06:49:12.263ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
174582025-02-21T06:49:12.763ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
174592025-02-21T06:49:12.763ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
174602025-02-21T06:49:12.763ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
174612025-02-21T06:49:12.763ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
174622025-02-21T06:49:12.763ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
17463 [8830] got start action
17464 [8830] I am going to restart
17465 Starting downstairs at port 8830
17466 Make output file at "/tmp/dsc/downstairs-8830.txt"
17467 [8830][2] reports Starting
17468 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1221)
17469 [8830][2] reports Running
174702025-02-21T06:49:13.265ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
174712025-02-21T06:49:14.012ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
17472 ----------------------------------------------------------------
17473 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17474 States: Active Active Offline
17475 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17476 5227 Acked Read 8 Done Done Sent false
17477 5228 Acked Read 5 Done Done Sent false
17478 5229 Acked Read 7 Done Done Sent false
17479 5230 Acked Read 4 Done Done Sent false
17480 5231 Acked Write 1 Done Done Sent false
17481 5232 Acked Write 5 Done Done Sent false
17482 5233 Acked Flush 0 Done Done Sent false
17483 5234 Acked Read 2 Done Done Sent false
17484 5235 Acked Write 6 Done Done Sent false
17485 5236 Acked Read 4 Done Done Sent false
17486 5237 Acked Write 8 Done Done Sent false
17487 5238 Acked Write 4 Done Done Sent false
17488 5239 Acked Read 3 Done Done Sent false
17489 5240 Acked Write 7 Done Done Sent false
17490 5241 Acked Read 10 Done Done Sent false
17491 5242 Acked Write 6 Done Done Sent false
17492 5243 Acked Read 7 Done Done Sent false
17493 5244 Acked Read 2 Done Done Sent false
17494 5245 Acked Flush 0 Done Done Sent false
17495 5246 Acked Write 5 Done Done Sent false
17496 5247 Acked Write 8 Done Done Sent false
17497 5248 Acked Read 2 Done Done Sent false
17498 5249 Acked Read 2 Done Done Sent false
17499 5250 Acked Read 1 Done Done Sent false
17500 5251 Acked Read 9 Done Done Sent false
17501 5252 Acked Write 7 Done Done Sent false
17502 5253 Acked Write 3 Done Done Sent false
17503 5254 Acked Write 4 Done Done Sent false
17504 5255 Acked Flush 0 Done Done Sent false
17505 5256 Acked Write 10 Done Done Sent false
17506 5257 Acked Read 1 Done Done Sent false
17507 5258 Acked Flush 0 Done Done Sent false
17508 5259 Acked Flush 0 Done Done Sent false
17509 5260 Acked Read 5 Done Done Sent false
17510 5261 Acked Read 1 Done Done Sent false
17511 5262 Acked Write 9 Done Done Sent false
17512 5263 Acked Read 5 Done Done Sent false
17513 5264 Acked Read 4 Done Done Sent false
17514 5265 Acked Read 6 Done Done Sent false
17515 5266 Acked Read 10 Done Done Sent false
17516 5267 Acked Read 3 Done Done Sent false
17517 5268 Acked Write 2 Done Done Sent false
17518 5269 Acked Write 2 Done Done Sent false
17519 5270 Acked Write 10 Done Done Sent false
17520 5271 Acked Read 8 Done Done Sent false
17521 5272 Acked Read 10 Done Done Sent false
17522 5273 Acked Read 1 Done Done Sent false
17523 5274 Acked Write 3 Done Done Sent false
17524 5275 Acked Write 10 Done Done Sent false
17525 5276 Acked Write 2 Done Done Sent false
17526 5277 Acked Read 2 Done Done Sent false
17527 5278 Acked Write 4 Done Done Sent false
17528 5279 Acked Read 2 Done Done Sent false
17529 5280 Acked Read 3 Done Done Sent false
17530 5281 Acked Write 5 Done Done Sent false
17531 5282 Acked Write 7 Done Done Sent false
17532 5283 Acked Read 10 Done Done Sent false
17533 5284 Acked Write 2 Done Done Sent false
17534 5285 Acked Write 7 Done Done Sent false
17535 5286 Acked Write 10 Done Done Sent false
17536 5287 Acked Read 9 Done Done Sent false
17537 5288 Acked Write 5 Done Done Sent false
17538 5289 Acked Read 9 Done Done Sent false
17539 5290 Acked Read 9 Done Done Sent false
17540 5291 Acked Read 4 Done Done Sent false
17541 5292 Acked Write 4 Done Done Sent false
17542 5293 Acked Read 10 Done Done Sent false
17543 5294 Acked Read 9 Done Done Sent false
17544 5295 Acked Write 7 Done Done Sent false
17545 5296 Acked Read 1 Done Done Sent false
17546 5297 Acked Read 9 Done Done Sent false
17547 5298 Acked Read 2 Done Done Sent false
17548 5299 Acked Flush 0 Done Done Sent false
17549 5300 Acked Read 7 Done Done Sent false
17550 5301 Acked Read 9 Done Done Sent false
17551 5302 Acked Write 8 Done Done Sent false
17552 5303 Acked Read 6 Done Done Sent false
17553 5304 Acked Read 8 Done Done Sent false
17554 5305 Acked Read 4 Done Done Sent false
17555 5306 Acked Read 4 Done Done Sent false
17556 5307 Acked Read 7 Done Done Sent false
17557 5308 Acked Read 6 Done Done Sent false
17558 5309 Acked Read 9 Done Done Sent false
17559 5310 Acked Write 5 Done Done Sent false
17560 5311 Acked Write 4 Done Done Sent false
17561 5312 Acked Flush 0 Done Done Sent false
17562 5313 Acked Read 8 Done Done Sent false
17563 5314 Acked Read 3 Done Done Sent false
17564 5315 Acked Read 6 Done Done Sent false
17565 5316 Acked Read 3 Done Done Sent false
17566 5317 Acked Write 5 Done Done Sent false
17567 5318 Acked Write 4 Done Done Sent false
17568 5319 Acked Write 9 Done Done Sent false
17569 5320 Acked Read 7 Done Done Sent false
17570 5321 Acked Read 3 Done Done Sent false
17571 5322 Acked Write 9 Done Done Sent false
17572 5323 Acked Write 9 Done Done Sent false
17573 5324 Acked Read 2 Done Done Sent false
17574 5325 Acked Read 8 Done Done Sent false
17575 5326 Acked Write 8 Done Done Sent false
17576 5327 Acked Read 1 Done Done Sent false
17577 5328 Acked Write 7 Done Done Sent false
17578 5329 Acked Write 6 Done Done Sent false
17579 5330 Acked Read 4 Done Done Sent false
17580 5331 Acked Read 8 Done Done Sent false
17581 5332 Acked Write 6 Done Done Sent false
17582 5333 Acked Write 4 Done Done Sent false
17583 5334 Acked Write 8 Done Done Sent false
17584 5335 Acked Write 3 Done Done Sent false
17585 5336 Acked Flush 0 Done Done Sent false
17586 5337 Acked Read 7 Done Done Sent false
17587 5338 Acked Read 2 Done Done Sent false
17588 5339 Acked Read 3 Done Done Sent false
17589 5340 Acked Read 7 Done Done Sent false
17590 5341 Acked Read 2 Done Done Sent false
17591 5342 Acked Read 6 Done Done Sent false
17592 5343 Acked Flush 0 Done Done Sent false
17593 5344 Acked Flush 0 Done Done Sent false
17594 5345 Acked Write 5 Done Done Sent false
17595 5346 Acked Write 4 Done Done Sent false
17596 5347 Acked Write 10 Done Done Sent false
17597 5348 Acked Read 5 Done Done Sent false
17598 5349 Acked Write 4 Done Done Sent false
17599 5350 Acked Flush 0 Done Done Sent false
17600 5351 Acked Flush 0 Done Done Sent false
17601 5352 Acked Write 4 Done Done Sent false
17602 5353 Acked Write 6 Done Done Sent false
17603 5354 Acked Flush 0 Done Done Sent false
17604 5355 Acked Write 7 Done Done Sent false
17605 5356 Acked Write 8 Done Done Sent false
17606 5357 Acked Write 2 Done Done Sent false
17607 5358 Acked Write 8 Done Done Sent false
17608 5359 Acked Read 1 Done Done Sent false
17609 5360 Acked Write 7 Done Done Sent false
17610 5361 Acked Write 4 Done Done Sent false
17611 5362 Acked Write 2 Done Done Sent false
17612 5363 Acked Write 1 Done Done Sent false
17613 5364 Acked Write 4 Done Done Sent false
17614 5365 Acked Write 2 Done Done Sent false
17615 5366 Acked Write 10 Done Done Sent false
17616 5367 Acked Flush 0 Done Done Sent false
17617 5368 Acked Write 1 Done Done Sent false
17618 5369 Acked Read 10 Done Done Sent false
17619 5370 Acked Write 2 Done Done Sent false
17620 5371 Acked Write 9 Done Done Sent false
17621 5372 Acked Read 3 Done Done Sent false
17622 5373 Acked Write 4 Done Done Sent false
17623 5374 Acked Read 7 Done Done Sent false
17624 5375 Acked Write 2 Done Done Sent false
17625 5376 Acked Read 4 Done Done Sent false
17626 5377 Acked Read 1 Done Done Sent false
17627 5378 Acked Read 5 Done Done Sent false
17628 5379 Acked Write 7 Done Done Sent false
17629 5380 Acked Read 4 Done Done Sent false
17630 5381 Acked Write 7 Done Done Sent false
17631 5382 Acked Write 3 Done Done Sent false
17632 5383 Acked Write 5 Done Done Sent false
17633 5384 Acked Read 7 Done Done Sent false
17634 5385 Acked Read 10 Done Done Sent false
17635 5386 Acked Write 8 Done Done Sent false
17636 5387 Acked Write 9 Done Done Sent false
17637 5388 Acked Read 1 Done Done Sent false
17638 5389 Acked Write 10 Done Done Sent false
17639 5390 Acked Read 2 Done Done Sent false
17640 5391 Acked Read 3 Done Done Sent false
17641 5392 Acked Read 1 Done Done Sent false
17642 5393 Acked Write 7 Done Done Sent false
17643 5394 Acked Write 5 Done Done Sent false
17644 5395 Acked Write 3 Done Done Sent false
17645 5396 Acked Write 3 Done Done Sent false
17646 5397 Acked Read 6 Done Done Sent false
17647 5398 Acked Write 3 Done Done Sent false
17648 5399 Acked Write 4 Done Done Sent false
17649 5400 Acked Write 4 Done Done Sent false
17650 5401 Acked Write 7 Done Done Sent false
17651 5402 Acked Write 2 Done Done Sent false
17652 5403 Acked Flush 0 Done Done Sent false
17653 5404 Acked Flush 0 Done Done Sent false
17654 5405 Acked Read 7 Done Done Sent false
17655 5406 Acked Write 9 Done Done Sent false
17656 5407 Acked Write 1 Done Done Sent false
17657 5408 Acked Write 2 Done Done Sent false
17658 5409 Acked Write 8 Done Done Sent false
17659 5410 Acked Write 7 Done Done Sent false
17660 5411 Acked Read 9 Done Done Sent false
17661 5412 Acked Write 8 Done Done Sent false
17662 5413 Acked Read 1 Done Done Sent false
17663 5414 Acked Read 3 Done Done Sent false
17664 5415 Acked Read 10 Done Done Sent false
17665 5416 Acked Read 3 Done Done Sent false
17666 5417 Acked Write 3 Done Done Sent false
17667 5418 Acked Read 3 Done Done Sent false
17668 5419 Acked Read 4 Done Done Sent false
17669 5420 Acked Read 1 Done Done Sent false
17670 5421 Acked Write 10 Done Done Sent false
17671 5422 Acked Write 1 Done Done Sent false
17672 5423 Acked Write 7 Done Done Sent false
17673 5424 Acked Write 2 Done Done Sent false
17674 5425 Acked Write 4 Done Done Sent false
17675 5426 Acked Write 3 Done Done Sent false
17676 5427 Acked Read 10 Done Done Sent false
17677 5428 Acked Write 2 Done Done Sent false
17678 5429 Acked Read 7 Done Done Sent false
17679 5430 Acked Read 8 Done Done Sent false
17680 5431 Acked Write 4 Done Done Sent false
17681 5432 Acked Read 10 Done Done Sent false
17682 5433 Acked Flush 0 Done Done Sent false
17683 5434 Acked Write 2 Done Done Sent false
17684 5435 Acked Write 9 Done Done Sent false
17685 5436 Acked Flush 0 Done Done Sent false
17686 5437 Acked Write 6 Done Done Sent false
17687 5438 Acked Read 2 Done Done Sent false
17688 5439 Acked Read 2 Done Done Sent false
17689 5440 Acked Flush 0 Done Done Sent false
17690 5441 Acked Read 3 Done Done Sent false
17691 5442 Acked Write 8 Done Done Sent false
17692 5443 Acked Read 5 Done Done Sent false
17693 5444 Acked Write 6 Done Done Sent false
17694 5445 Acked Read 1 Done Done Sent false
17695 5446 Acked Read 6 Done Done Sent false
17696 5447 Acked Flush 0 Done Done Sent false
17697 5448 Acked Read 3 Done Done Sent false
17698 5449 Acked Read 3 Done Done Sent false
17699 5450 Acked Write 5 Done Done Sent false
17700 5451 Acked Read 9 Done Done Sent false
17701 5452 Acked Read 8 Done Done Sent false
17702 5453 Acked Flush 0 Done Done Sent false
17703 5454 Acked Write 2 Done Done Sent false
17704 5455 Acked Read 5 Done Done Sent false
17705 5456 Acked Write 10 Done Done Sent false
17706 5457 Acked Read 4 Done Done Sent false
17707 5458 Acked Flush 0 Done Done Sent false
17708 5459 Acked Write 4 Done Done Sent false
17709 5460 Acked Read 2 Done Done Sent false
17710 5461 Acked Read 7 Done Done Sent false
17711 5462 Acked Read 9 Done Done Sent false
17712 5463 Acked Write 5 Done Done Sent false
17713 5464 Acked Write 10 Done Done Sent false
17714 5465 Acked Flush 0 Done Done Sent false
17715 5466 Acked Read 1 Done Done Sent false
17716 5467 Acked Write 2 Done Done Sent false
17717 5468 Acked Write 10 Done Done Sent false
17718 5469 Acked Write 7 Done Done Sent false
17719 5470 Acked Write 2 Done Done Sent false
17720 5471 Acked Read 1 Done Done Sent false
17721 5472 Acked Write 3 Done Done Sent false
17722 5473 Acked Read 2 Done Done Sent false
17723 5474 Acked Read 7 Done Done Sent false
17724 5475 Acked Write 8 Done Done Sent false
17725 5476 Acked Read 9 Done Done Sent false
17726 5477 Acked Read 3 Done Done Sent false
17727 5478 Acked Flush 0 Done Done Sent false
17728 5479 Acked Write 8 Done Done Sent false
17729 5480 Acked Read 7 Done Done Sent false
17730 5481 Acked Read 10 Done Done Sent false
17731 5482 Acked Flush 0 Done Done Sent false
17732 5483 Acked Write 5 Done Done Sent false
17733 5484 Acked Flush 0 Done Done Sent false
17734 5485 Acked Write 2 Done Done Sent false
17735 5486 Acked Write 2 Done Done Sent false
17736 5487 Acked Write 9 Done Done Sent false
17737 5488 Acked Write 6 Done Done Sent false
17738 5489 Acked Write 4 Done Done Sent false
17739 5490 Acked Write 2 Done Done Sent false
17740 5491 Acked Write 8 Done Done Sent false
17741 5492 Acked Read 10 Done Done Sent false
17742 5493 Acked Read 5 Done Done Sent false
17743 5494 Acked Read 6 Done Done Sent false
17744 5495 Acked Write 4 Done Done Sent false
17745 5496 Acked Read 6 Done Done Sent false
17746 5497 Acked Read 4 Done Done Sent false
17747 5498 Acked Read 3 Done Done Sent false
17748 5499 Acked Read 8 Done Done Sent false
17749 5500 Acked Write 1 Done Done Sent false
17750 5501 Acked Read 5 Done Done Sent false
17751 5502 Acked Flush 0 Done Done Sent false
17752 5503 Acked Flush 0 Done Done Sent false
17753 5504 Acked Read 6 Done Done Sent false
17754 5505 Acked Write 10 Done Done Sent false
17755 5506 Acked Read 4 Done Done Sent false
17756 5507 Acked Write 8 Done Done Sent false
17757 5508 Acked Write 8 Done Done Sent false
17758 5509 Acked Read 10 Done Done Sent false
17759 5510 Acked Read 2 Done Done Sent false
17760 5511 Acked Write 8 Done Done Sent false
17761 5512 Acked Write 4 Done Done Sent false
17762 5513 Acked Write 8 Done Done Sent false
17763 5514 Acked Read 3 Done Done Sent false
17764 5515 Acked Write 3 Done Done Sent false
17765 5516 Acked Read 3 Done Done Sent false
17766 5517 Acked Write 9 Done Done Sent false
17767 5518 Acked Write 8 Done Done Sent false
17768 5519 Acked Read 1 Done Done Sent false
17769 5520 Acked Write 1 Done Done Sent false
17770 5521 Acked Read 8 Done Done Sent false
17771 5522 Acked Write 7 Done Done Sent false
17772 5523 Acked Read 10 Done Done Sent false
17773 5524 Acked Read 5 Done Done Sent false
17774 5525 Acked Read 8 Done Done Sent false
17775 5526 Acked Read 8 Done Done Sent false
17776 5527 Acked Read 5 Done Done Sent false
17777 STATES DS:0 DS:1 DS:2 TOTAL
17778 Sent 0 0 301 301
17779 Done 301 301 0 602
17780 Skipped 0 0 0 0
17781 Error 0 0 0 0
17782 Last Flush: Some(JobId(5503)) Some(JobId(5503)) Some(JobId(5226))
17783 Downstairs last five completed: 5226 5225 5224 5223 5222
17784 Upstairs last five completed: 5527 5526 5525 5524 5523
17785 CLIENT: Up:0 ds:301 act:2
177862025-02-21T06:49:15.500ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
177872025-02-21T06:49:16.014ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
177882025-02-21T06:49:17.264ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
177892025-02-21T06:49:18.017ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
17790 ----------------------------------------------------------------
17791 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17792 States: Active Active Offline
17793 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17794 5227 Acked Read 8 Done Done Sent false
17795 5228 Acked Read 5 Done Done Sent false
17796 5229 Acked Read 7 Done Done Sent false
17797 5230 Acked Read 4 Done Done Sent false
17798 5231 Acked Write 1 Done Done Sent false
17799 5232 Acked Write 5 Done Done Sent false
17800 5233 Acked Flush 0 Done Done Sent false
17801 5234 Acked Read 2 Done Done Sent false
17802 5235 Acked Write 6 Done Done Sent false
17803 5236 Acked Read 4 Done Done Sent false
17804 5237 Acked Write 8 Done Done Sent false
17805 5238 Acked Write 4 Done Done Sent false
17806 5239 Acked Read 3 Done Done Sent false
17807 5240 Acked Write 7 Done Done Sent false
17808 5241 Acked Read 10 Done Done Sent false
17809 5242 Acked Write 6 Done Done Sent false
17810 5243 Acked Read 7 Done Done Sent false
17811 5244 Acked Read 2 Done Done Sent false
17812 5245 Acked Flush 0 Done Done Sent false
17813 5246 Acked Write 5 Done Done Sent false
17814 5247 Acked Write 8 Done Done Sent false
17815 5248 Acked Read 2 Done Done Sent false
17816 5249 Acked Read 2 Done Done Sent false
17817 5250 Acked Read 1 Done Done Sent false
17818 5251 Acked Read 9 Done Done Sent false
17819 5252 Acked Write 7 Done Done Sent false
17820 5253 Acked Write 3 Done Done Sent false
17821 5254 Acked Write 4 Done Done Sent false
17822 5255 Acked Flush 0 Done Done Sent false
17823 5256 Acked Write 10 Done Done Sent false
17824 5257 Acked Read 1 Done Done Sent false
17825 5258 Acked Flush 0 Done Done Sent false
17826 5259 Acked Flush 0 Done Done Sent false
17827 5260 Acked Read 5 Done Done Sent false
17828 5261 Acked Read 1 Done Done Sent false
17829 5262 Acked Write 9 Done Done Sent false
17830 5263 Acked Read 5 Done Done Sent false
17831 5264 Acked Read 4 Done Done Sent false
17832 5265 Acked Read 6 Done Done Sent false
17833 5266 Acked Read 10 Done Done Sent false
17834 5267 Acked Read 3 Done Done Sent false
17835 5268 Acked Write 2 Done Done Sent false
17836 5269 Acked Write 2 Done Done Sent false
17837 5270 Acked Write 10 Done Done Sent false
17838 5271 Acked Read 8 Done Done Sent false
17839 5272 Acked Read 10 Done Done Sent false
17840 5273 Acked Read 1 Done Done Sent false
17841 5274 Acked Write 3 Done Done Sent false
17842 5275 Acked Write 10 Done Done Sent false
17843 5276 Acked Write 2 Done Done Sent false
17844 5277 Acked Read 2 Done Done Sent false
17845 5278 Acked Write 4 Done Done Sent false
17846 5279 Acked Read 2 Done Done Sent false
17847 5280 Acked Read 3 Done Done Sent false
17848 5281 Acked Write 5 Done Done Sent false
17849 5282 Acked Write 7 Done Done Sent false
17850 5283 Acked Read 10 Done Done Sent false
17851 5284 Acked Write 2 Done Done Sent false
17852 5285 Acked Write 7 Done Done Sent false
17853 5286 Acked Write 10 Done Done Sent false
17854 5287 Acked Read 9 Done Done Sent false
17855 5288 Acked Write 5 Done Done Sent false
17856 5289 Acked Read 9 Done Done Sent false
17857 5290 Acked Read 9 Done Done Sent false
17858 5291 Acked Read 4 Done Done Sent false
17859 5292 Acked Write 4 Done Done Sent false
17860 5293 Acked Read 10 Done Done Sent false
17861 5294 Acked Read 9 Done Done Sent false
17862 5295 Acked Write 7 Done Done Sent false
17863 5296 Acked Read 1 Done Done Sent false
17864 5297 Acked Read 9 Done Done Sent false
17865 5298 Acked Read 2 Done Done Sent false
17866 5299 Acked Flush 0 Done Done Sent false
17867 5300 Acked Read 7 Done Done Sent false
17868 5301 Acked Read 9 Done Done Sent false
17869 5302 Acked Write 8 Done Done Sent false
17870 5303 Acked Read 6 Done Done Sent false
17871 5304 Acked Read 8 Done Done Sent false
17872 5305 Acked Read 4 Done Done Sent false
17873 5306 Acked Read 4 Done Done Sent false
17874 5307 Acked Read 7 Done Done Sent false
17875 5308 Acked Read 6 Done Done Sent false
17876 5309 Acked Read 9 Done Done Sent false
17877 5310 Acked Write 5 Done Done Sent false
17878 5311 Acked Write 4 Done Done Sent false
17879 5312 Acked Flush 0 Done Done Sent false
17880 5313 Acked Read 8 Done Done Sent false
17881 5314 Acked Read 3 Done Done Sent false
17882 5315 Acked Read 6 Done Done Sent false
17883 5316 Acked Read 3 Done Done Sent false
17884 5317 Acked Write 5 Done Done Sent false
17885 5318 Acked Write 4 Done Done Sent false
17886 5319 Acked Write 9 Done Done Sent false
17887 5320 Acked Read 7 Done Done Sent false
17888 5321 Acked Read 3 Done Done Sent false
17889 5322 Acked Write 9 Done Done Sent false
17890 5323 Acked Write 9 Done Done Sent false
17891 5324 Acked Read 2 Done Done Sent false
17892 5325 Acked Read 8 Done Done Sent false
17893 5326 Acked Write 8 Done Done Sent false
17894 5327 Acked Read 1 Done Done Sent false
17895 5328 Acked Write 7 Done Done Sent false
17896 5329 Acked Write 6 Done Done Sent false
17897 5330 Acked Read 4 Done Done Sent false
17898 5331 Acked Read 8 Done Done Sent false
17899 5332 Acked Write 6 Done Done Sent false
17900 5333 Acked Write 4 Done Done Sent false
17901 5334 Acked Write 8 Done Done Sent false
17902 5335 Acked Write 3 Done Done Sent false
17903 5336 Acked Flush 0 Done Done Sent false
17904 5337 Acked Read 7 Done Done Sent false
17905 5338 Acked Read 2 Done Done Sent false
17906 5339 Acked Read 3 Done Done Sent false
17907 5340 Acked Read 7 Done Done Sent false
17908 5341 Acked Read 2 Done Done Sent false
17909 5342 Acked Read 6 Done Done Sent false
17910 5343 Acked Flush 0 Done Done Sent false
17911 5344 Acked Flush 0 Done Done Sent false
17912 5345 Acked Write 5 Done Done Sent false
17913 5346 Acked Write 4 Done Done Sent false
17914 5347 Acked Write 10 Done Done Sent false
17915 5348 Acked Read 5 Done Done Sent false
17916 5349 Acked Write 4 Done Done Sent false
17917 5350 Acked Flush 0 Done Done Sent false
17918 5351 Acked Flush 0 Done Done Sent false
17919 5352 Acked Write 4 Done Done Sent false
17920 5353 Acked Write 6 Done Done Sent false
17921 5354 Acked Flush 0 Done Done Sent false
17922 5355 Acked Write 7 Done Done Sent false
17923 5356 Acked Write 8 Done Done Sent false
17924 5357 Acked Write 2 Done Done Sent false
17925 5358 Acked Write 8 Done Done Sent false
17926 5359 Acked Read 1 Done Done Sent false
17927 5360 Acked Write 7 Done Done Sent false
17928 5361 Acked Write 4 Done Done Sent false
17929 5362 Acked Write 2 Done Done Sent false
17930 5363 Acked Write 1 Done Done Sent false
17931 5364 Acked Write 4 Done Done Sent false
17932 5365 Acked Write 2 Done Done Sent false
17933 5366 Acked Write 10 Done Done Sent false
17934 5367 Acked Flush 0 Done Done Sent false
17935 5368 Acked Write 1 Done Done Sent false
17936 5369 Acked Read 10 Done Done Sent false
17937 5370 Acked Write 2 Done Done Sent false
17938 5371 Acked Write 9 Done Done Sent false
17939 5372 Acked Read 3 Done Done Sent false
17940 5373 Acked Write 4 Done Done Sent false
17941 5374 Acked Read 7 Done Done Sent false
17942 5375 Acked Write 2 Done Done Sent false
17943 5376 Acked Read 4 Done Done Sent false
17944 5377 Acked Read 1 Done Done Sent false
17945 5378 Acked Read 5 Done Done Sent false
17946 5379 Acked Write 7 Done Done Sent false
17947 5380 Acked Read 4 Done Done Sent false
17948 5381 Acked Write 7 Done Done Sent false
17949 5382 Acked Write 3 Done Done Sent false
17950 5383 Acked Write 5 Done Done Sent false
17951 5384 Acked Read 7 Done Done Sent false
17952 5385 Acked Read 10 Done Done Sent false
17953 5386 Acked Write 8 Done Done Sent false
17954 5387 Acked Write 9 Done Done Sent false
17955 5388 Acked Read 1 Done Done Sent false
17956 5389 Acked Write 10 Done Done Sent false
17957 5390 Acked Read 2 Done Done Sent false
17958 5391 Acked Read 3 Done Done Sent false
17959 5392 Acked Read 1 Done Done Sent false
17960 5393 Acked Write 7 Done Done Sent false
17961 5394 Acked Write 5 Done Done Sent false
17962 5395 Acked Write 3 Done Done Sent false
17963 5396 Acked Write 3 Done Done Sent false
17964 5397 Acked Read 6 Done Done Sent false
17965 5398 Acked Write 3 Done Done Sent false
17966 5399 Acked Write 4 Done Done Sent false
17967 5400 Acked Write 4 Done Done Sent false
17968 5401 Acked Write 7 Done Done Sent false
17969 5402 Acked Write 2 Done Done Sent false
17970 5403 Acked Flush 0 Done Done Sent false
17971 5404 Acked Flush 0 Done Done Sent false
17972 5405 Acked Read 7 Done Done Sent false
17973 5406 Acked Write 9 Done Done Sent false
17974 5407 Acked Write 1 Done Done Sent false
17975 5408 Acked Write 2 Done Done Sent false
17976 5409 Acked Write 8 Done Done Sent false
17977 5410 Acked Write 7 Done Done Sent false
17978 5411 Acked Read 9 Done Done Sent false
17979 5412 Acked Write 8 Done Done Sent false
17980 5413 Acked Read 1 Done Done Sent false
17981 5414 Acked Read 3 Done Done Sent false
17982 5415 Acked Read 10 Done Done Sent false
17983 5416 Acked Read 3 Done Done Sent false
17984 5417 Acked Write 3 Done Done Sent false
17985 5418 Acked Read 3 Done Done Sent false
17986 5419 Acked Read 4 Done Done Sent false
17987 5420 Acked Read 1 Done Done Sent false
17988 5421 Acked Write 10 Done Done Sent false
17989 5422 Acked Write 1 Done Done Sent false
17990 5423 Acked Write 7 Done Done Sent false
17991 5424 Acked Write 2 Done Done Sent false
17992 5425 Acked Write 4 Done Done Sent false
17993 5426 Acked Write 3 Done Done Sent false
17994 5427 Acked Read 10 Done Done Sent false
17995 5428 Acked Write 2 Done Done Sent false
17996 5429 Acked Read 7 Done Done Sent false
17997 5430 Acked Read 8 Done Done Sent false
17998 5431 Acked Write 4 Done Done Sent false
17999 5432 Acked Read 10 Done Done Sent false
18000 5433 Acked Flush 0 Done Done Sent false
18001 5434 Acked Write 2 Done Done Sent false
18002 5435 Acked Write 9 Done Done Sent false
18003 5436 Acked Flush 0 Done Done Sent false
18004 5437 Acked Write 6 Done Done Sent false
18005 5438 Acked Read 2 Done Done Sent false
18006 5439 Acked Read 2 Done Done Sent false
18007 5440 Acked Flush 0 Done Done Sent false
18008 5441 Acked Read 3 Done Done Sent false
18009 5442 Acked Write 8 Done Done Sent false
18010 5443 Acked Read 5 Done Done Sent false
18011 5444 Acked Write 6 Done Done Sent false
18012 5445 Acked Read 1 Done Done Sent false
18013 5446 Acked Read 6 Done Done Sent false
18014 5447 Acked Flush 0 Done Done Sent false
18015 5448 Acked Read 3 Done Done Sent false
18016 5449 Acked Read 3 Done Done Sent false
18017 5450 Acked Write 5 Done Done Sent false
18018 5451 Acked Read 9 Done Done Sent false
18019 5452 Acked Read 8 Done Done Sent false
18020 5453 Acked Flush 0 Done Done Sent false
18021 5454 Acked Write 2 Done Done Sent false
18022 5455 Acked Read 5 Done Done Sent false
18023 5456 Acked Write 10 Done Done Sent false
18024 5457 Acked Read 4 Done Done Sent false
18025 5458 Acked Flush 0 Done Done Sent false
18026 5459 Acked Write 4 Done Done Sent false
18027 5460 Acked Read 2 Done Done Sent false
18028 5461 Acked Read 7 Done Done Sent false
18029 5462 Acked Read 9 Done Done Sent false
18030 5463 Acked Write 5 Done Done Sent false
18031 5464 Acked Write 10 Done Done Sent false
18032 5465 Acked Flush 0 Done Done Sent false
18033 5466 Acked Read 1 Done Done Sent false
18034 5467 Acked Write 2 Done Done Sent false
18035 5468 Acked Write 10 Done Done Sent false
18036 5469 Acked Write 7 Done Done Sent false
18037 5470 Acked Write 2 Done Done Sent false
18038 5471 Acked Read 1 Done Done Sent false
18039 5472 Acked Write 3 Done Done Sent false
18040 5473 Acked Read 2 Done Done Sent false
18041 5474 Acked Read 7 Done Done Sent false
18042 5475 Acked Write 8 Done Done Sent false
18043 5476 Acked Read 9 Done Done Sent false
18044 5477 Acked Read 3 Done Done Sent false
18045 5478 Acked Flush 0 Done Done Sent false
18046 5479 Acked Write 8 Done Done Sent false
18047 5480 Acked Read 7 Done Done Sent false
18048 5481 Acked Read 10 Done Done Sent false
18049 5482 Acked Flush 0 Done Done Sent false
18050 5483 Acked Write 5 Done Done Sent false
18051 5484 Acked Flush 0 Done Done Sent false
18052 5485 Acked Write 2 Done Done Sent false
18053 5486 Acked Write 2 Done Done Sent false
18054 5487 Acked Write 9 Done Done Sent false
18055 5488 Acked Write 6 Done Done Sent false
18056 5489 Acked Write 4 Done Done Sent false
18057 5490 Acked Write 2 Done Done Sent false
18058 5491 Acked Write 8 Done Done Sent false
18059 5492 Acked Read 10 Done Done Sent false
18060 5493 Acked Read 5 Done Done Sent false
18061 5494 Acked Read 6 Done Done Sent false
18062 5495 Acked Write 4 Done Done Sent false
18063 5496 Acked Read 6 Done Done Sent false
18064 5497 Acked Read 4 Done Done Sent false
18065 5498 Acked Read 3 Done Done Sent false
18066 5499 Acked Read 8 Done Done Sent false
18067 5500 Acked Write 1 Done Done Sent false
18068 5501 Acked Read 5 Done Done Sent false
18069 5502 Acked Flush 0 Done Done Sent false
18070 5503 Acked Flush 0 Done Done Sent false
18071 5504 Acked Read 6 Done Done Sent false
18072 5505 Acked Write 10 Done Done Sent false
18073 5506 Acked Read 4 Done Done Sent false
18074 5507 Acked Write 8 Done Done Sent false
18075 5508 Acked Write 8 Done Done Sent false
18076 5509 Acked Read 10 Done Done Sent false
18077 5510 Acked Read 2 Done Done Sent false
18078 5511 Acked Write 8 Done Done Sent false
18079 5512 Acked Write 4 Done Done Sent false
18080 5513 Acked Write 8 Done Done Sent false
18081 5514 Acked Read 3 Done Done Sent false
18082 5515 Acked Write 3 Done Done Sent false
18083 5516 Acked Read 3 Done Done Sent false
18084 5517 Acked Write 9 Done Done Sent false
18085 5518 Acked Write 8 Done Done Sent false
18086 5519 Acked Read 1 Done Done Sent false
18087 5520 Acked Write 1 Done Done Sent false
18088 5521 Acked Read 8 Done Done Sent false
18089 5522 Acked Write 7 Done Done Sent false
18090 5523 Acked Read 10 Done Done Sent false
18091 5524 Acked Read 5 Done Done Sent false
18092 5525 Acked Read 8 Done Done Sent false
18093 5526 Acked Read 8 Done Done Sent false
18094 5527 Acked Read 5 Done Done Sent false
18095 STATES DS:0 DS:1 DS:2 TOTAL
18096 Sent 0 0 301 301
18097 Done 301 301 0 602
18098 Skipped 0 0 0 0
18099 Error 0 0 0 0
18100 Last Flush: Some(JobId(5503)) Some(JobId(5503)) Some(JobId(5226))
18101 Downstairs last five completed: 5226 5225 5224 5223 5222
18102 Upstairs last five completed: 5527 5526 5525 5524 5523
18103 CLIENT: Up:0 ds:301 act:2
181042025-02-21T06:49:19.502ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181052025-02-21T06:49:20.020ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181062025-02-21T06:49:21.022ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181072025-02-21T06:49:22.264ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181082025-02-21T06:49:22.765ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181092025-02-21T06:49:22.766ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42444) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181102025-02-21T06:49:22.767ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181112025-02-21T06:49:22.767ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181122025-02-21T06:49:22.767ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181132025-02-21T06:49:22.767ZINFOcrucible: send last flush ID to this DS: Some(JobId(5226)) = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181142025-02-21T06:49:22.767ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181152025-02-21T06:49:22.767ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(5226)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181162025-02-21T06:49:22.768ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(5226)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
18117 ----------------------------------------------------------------
18118 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:24
18119 States: Active Active Active
18120 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18121 5504 Acked Read 6 Done Done Done true
18122 5505 Acked Write 10 Done Done Done true
18123 5506 Acked Read 4 Done Done Done true
18124 5507 Acked Write 8 Done Done Done true
18125 5508 Acked Write 8 Done Done Done true
18126 5509 Acked Read 10 Done Done Done true
18127 5510 Acked Read 2 Done Done Done true
18128 5511 Acked Write 8 Done Done Done true
18129 5512 Acked Write 4 Done Done Done true
18130 5513 Acked Write 8 Done Done Done true
18131 5514 Acked Read 3 Done Done Done true
18132 5515 Acked Write 3 Done Done Done true
18133 5516 Acked Read 3 Done Done Done true
18134 5517 Acked Write 9 Done Done Done true
18135 5518 Acked Write 8 Done Done Done true
18136 5519 Acked Read 1 Done Done Done true
18137 5520 Acked Write 1 Done Done Done true
18138 5521 Acked Read 8 Done Done Done true
18139 5522 Acked Write 7 Done Done Done true
18140 5523 Acked Read 10 Done Done Done true
18141 5524 Acked Read 5 Done Done Done true
18142 5525 Acked Read 8 Done Done Done true
18143 5526 Acked Read 8 Done Done Done true
18144 5527 Acked Read 5 Done Done Done true
18145 STATES DS:0 DS:1 DS:2 TOTAL
18146 Sent 0 0 0 0
18147 Done 24 24 24 72
18148 Skipped 0 0 0 0
18149 Error 0 0 0 0
18150 Last Flush: Some(JobId(5503)) Some(JobId(5503)) Some(JobId(5503))
18151 Downstairs last five completed: 5503 5502 5501 5500 5499
18152 Upstairs last five completed: 5527 5526 5525 5524 5523
18153 CLIENT: Up:0 ds:24 act:3
18154 ----------------------------------------------------------------
18155 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18156 Downstairs last five completed: 5528 5527 5526 5525 5524
18157 Upstairs last five completed: 5528 5527 5526 5525 5524
18158 CLIENT: Up:0 ds:0 act:3
18159 Replay: All jobs finished
18160 Main task has work to do, go find it
18161 got dsc Stop(0)
18162 stop 0
18163 Feb 21 06:49:27.004 INFO request completed, latency_us: 356[8810] Got stop action so:false kr:false
18164 , response_code: 204, uri: /stop/cid/0, method: POST, req_id: 0f55c455-e50e-450e-8957-bd90fb944334, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
181652025-02-21T06:49:27.004ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
181662025-02-21T06:49:27.004ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
18167 Feb 21 06:49:27.004 INFO request completed, latency_us: 140, response_code{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:49:27.004940649Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
18168 : 200, uri: /state/cid/0, method: GET, req_id: 19655bdd-885f-4daf-8d06-329c9f4303bd, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
181692025-02-21T06:49:27.005ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
18170 [8810] Exited with: ExitStatus(unix_wait_status(9))
18171 [8810][0] reports Exit
18172 Feb 21 06:49:31.007 INFO request completed, latency_us: 220, response_code: 200, uri: /state/cid/0, method: GET, req_id: 26fa32f0-9bd5-4f61-8662-d5ada2bfd102, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
18173 001/300 Read block 4208 len 4096
18174 002/300 Write block 1987 len 24576 data: 7 7 6 5 5 4
18175 003/300 Write block 4853 len 8192 data: 4 4
18176 004/300 Read block 3356 len 36864
18177 005/300 Write block 4572 len 4096 data: 6
18178 006/300 Read block 3747 len 36864
18179 007/300 Flush
18180 008/300 Flush
18181 009/300 Write block 780 len 4096 data: 4
18182 010/300 Read block 2153 len 20480
18183 011/300 Read block 4485 len 12288
18184 012/300 Read block 2800 len 32768
18185 013/300 Read block 4920 len 36864
18186 014/300 Read block 3040 len 36864
18187 015/300 Read block 4278 len 32768
18188 016/300 Read block 1395 len 28672
18189 017/300 Write block 3654 len 24576 data: 3 2 2 2 3 3
18190 018/300 Read block 927 len 4096
18191 019/300 Write block 3308 len 28672 data: 2 2 2 2 3 3 3
18192 020/300 Read block 2260 len 8192
18193 021/300 Read block 4626 len 28672
18194 022/300 Read block 3312 len 16384
18195 023/300 Read block 358 len 16384
18196 024/300 Read block 4875 len 28672
18197 025/300 Flush
18198 026/300 Write block 2262 len 32768 data: 4 4 4 4 3 2 2 2
18199 027/300 Flush
18200 028/300 Read block 1251 len 40960
18201 029/300 Write block 16 len 20480 data: 4 3 4 3 3
18202 030/300 Read block 4832 len 8192
18203 031/300 Write block 4513 len 8192 data: 2 3
18204 032/300 Write block 4279 len 40960 data: 2 3 3 3 4 4 3 3 3 3
18205 033/300 Read block 1443 len 32768
18206 034/300 Read block 4109 len 8192
18207 035/300 Read block 2099 len 32768
18208 036/300 Read block 2264 len 8192
18209 037/300 Write block 1463 len 12288 data: 5 5 5
18210 038/300 Write block 4077 len 16384 data: 7 5 4 3
18211 039/300 Write block 2903 len 24576 data: 4 5 5 5 4 4
18212 040/300 Flush
18213 041/300 Write block 524 len 8192 data: 5 5
18214 042/300 Read block 2974 len 4096
18215 043/300 Read block 1173 len 40960
18216 044/300 Write block 3115 len 36864 data: 4 4 2 3 3 5 3 3 3
18217 045/300 Read block 3687 len 20480
18218 046/300 Read block 3066 len 8192
18219 047/300 Read block 4712 len 24576
18220 048/300 Write block 3400 len 28672 data: 3 3 3 3 3 4 4
18221 049/300 Write block 4586 len 28672 data: 2 2 2 2 2 3 3
18222 050/300 Read block 1971 len 4096
18223 051/300 Read block 2478 len 4096
18224 052/300 Write block 415 len 28672 data: 3 3 3 2 2 2 2
18225 053/300 Write block 3642 len 36864 data: 4 4 4 5 4 4 4 4 3
18226 054/300 Write block 3079 len 32768 data: 3 4 3 3 3 3 3 2
18227 055/300 Read block 2769 len 32768
18228 056/300 Flush
18229 057/300 Write block 2557 len 36864 data: 3 4 4 5 5 5 4 4 3
18230 058/300 Write block 2006 len 40960 data: 5 4 4 2 2 3 3 3 3 3
18231 059/300 Read block 4934 len 32768
18232 060/300 Read block 3864 len 28672
18233 061/300 Read block 1764 len 40960
18234 062/300 Read block 485 len 32768
18235 063/300 Write block 4441 len 16384 data: 3 4 4 4
18236 064/300 Read block 39 len 24576
18237 065/300 Write block 2377 len 28672 data: 3 3 2 2 2 2 2
18238 066/300 Write block 1051 len 40960 data: 3 3 5 6 5 5 5 6 5 6
18239 067/300 Write block 2174 len 32768 data: 3 5 4 5 3 3 4 4
18240 068/300 Read block 3727 len 32768
18241 069/300 Read block 188 len 8192
18242 070/300 Write block 1962 len 8192 data: 3 3
18243 071/300 Write block 1569 len 12288 data: 2 2 2
18244 072/300 Flush
18245 073/300 Write block 3470 len 28672 data: 3 2 2 2 4 4 5
18246 074/300 Read block 599 len 40960
18247 075/300 Write block 4780 len 12288 data: 2 3 3
18248 076/300 Read block 2578 len 36864
18249 077/300 Read block 1198 len 12288
18250 078/300 Write block 952 len 36864 data: 3 2 3 4 4 6 6 6 6
18251 079/300 Read block 2849 len 12288
18252 080/300 Read block 754 len 28672
18253 081/300 Write block 1041 len 4096 data: 2
18254 082/300 Read block 3646 len 24576
18255 083/300 Read block 2976 len 40960
18256 084/300 Read block 2420 len 20480
18257 085/300 Write block 168 len 40960 data: 3 3 3 3 4 4 3 3 2 3
18258 086/300 Read block 1255 len 28672
18259 087/300 Write block 1890 len 16384 data: 4 4 4 5
18260 088/300 Write block 621 len 4096 data: 2
18261 089/300 Write block 3769 len 28672 data: 5 7 7 8 8 7 6
18262 090/300 Read block 539 len 36864
18263 091/300 Read block 2496 len 4096
18264 092/300 Write block 1551 len 40960 data: 2 2 2 2 2 4 3 2 2 4
18265 093/300 Read block 4958 len 4096
18266 094/300 Flush
18267 095/300 Read block 2207 len 20480
18268 096/300 Write block 1053 len 24576 data: 6 7 6 6 6 7
18269 097/300 Write block 1177 len 24576 data: 4 4 3 4 4 4
18270 098/300 Flush
18271 099/300 Read block 2792 len 12288
18272 100/300 Write block 1759 len 16384 data: 6 6 6 8
18273 101/300 Read block 4029 len 12288
18274 102/300 Write block 3953 len 32768 data: 4 3 4 4 4 3 3 3
18275 103/300 Read block 2431 len 24576
18276 104/300 Flush
18277 105/300 Write block 972 len 8192 data: 6 6
18278 106/300 Read block 4097 len 20480
18279 107/300 Read block 4850 len 20480
18280 108/300 Read block 1129 len 12288
18281 109/300 Write block 1880 len 16384 data: 3 3 5 5
18282 110/300 Read block 3297 len 8192
18283 111/300 Write block 1855 len 40960 data: 4 4 3 3 3 4 4 4 4 4
18284 112/300 Write block 3018 len 40960 data: 2 3 3 4 4 4 4 4 4 3
18285 113/300 Read block 972 len 20480
18286 114/300 Read block 804 len 40960
18287 115/300 Write block 3937 len 24576 data: 5 4 4 5 5 5
18288 116/300 Read block 3375 len 20480
18289 117/300 Read block 3957 len 28672
18290 118/300 Read block 757 len 24576
18291 119/300 Read block 4978 len 40960
18292 120/300 Flush
18293 121/300 Write block 1309 len 36864 data: 4 5 5 5 5 5 5 4 4
18294 122/300 Flush
18295 123/300 Flush
18296 124/300 Read block 4030 len 16384
18297 125/300 Read block 411 len 28672
18298 126/300 Read block 4862 len 24576
18299 127/300 Write block 2743 len 36864 data: 5 6 7 7 8 8 8 6 8
18300 128/300 Read block 4709 len 16384
18301 129/300 Read block 2014 len 32768
18302 130/300 Flush
18303 131/300 Read block 135 len 24576
18304 132/300 Read block 4042 len 20480
18305 133/300 Flush
18306 134/300 Read block 2174 len 24576
18307 135/300 Read block 2232 len 40960
18308 136/300 Write block 2854 len 36864 data: 6 6 7 9 7 7 6 4 3
18309 137/300 Read block 196 len 20480
18310 138/300 Write block 2000 len 4096 data: 3
18311 139/300 Read block 2164 len 40960
18312 140/300 Read block 498 len 32768
18313 141/300 Flush
18314 142/300 Write block 816 len 4096 data: 4
18315 143/300 Write block 3717 len 28672 data: 3 3 2 2 2 3 3
18316 144/300 Write block 1333 len 20480 data: 2 2 3 3 2
18317 145/300 Flush
18318 146/300 Read block 3389 len 20480
18319 147/300 Write block 3471 len 20480 data: 3 3 3 5 5
18320 148/300 Write block 2692 len 32768 data: 2 3 3 4 4 5 4 4
18321 149/300 Read block 4658 len 24576
18322 150/300 Write block 3712 len 4096 data: 4
18323 151/300 Read block 1767 len 4096
18324 152/300 Read block 4888 len 36864
18325 153/300 Read block 735 len 32768
18326 154/300 Read block 3735 len 32768
18327 155/300 Write block 4455 len 40960 data: 2 2 2 2 2 3 3 3 2 3
18328 156/300 Flush
18329 157/300 Write block 4301 len 28672 data: 3 2 3 3 4 3 3
18330 158/300 Flush
18331 159/300 Write block 576 len 40960 data: 3 3 2 3 4 4 4 4 3 4
18332 160/300 Flush
18333 161/300 Write block 2625 len 12288 data: 4 4 4
18334 162/300 Read block 3817 len 12288
18335 163/300 Read block 285 len 20480
18336 164/300 Read block 104 len 20480
18337 165/300 Write block 4490 len 28672 data: 2 2 3 2 2 2 2
18338 166/300 Write block 4958 len 24576 data: 2 3 4 4 4 5
18339 167/300 Read block 2340 len 20480
18340 168/300 Read block 2340 len 12288
18341 169/300 Read block 2113 len 32768
18342 170/300 Read block 358 len 8192
18343 171/300 Write block 924 len 36864 data: 3 4 4 5 5 5 4 3 3
18344 172/300 Read block 4728 len 40960
18345 173/300 Write block 4424 len 20480 data: 4 4 4 5 5
18346 174/300 Flush
18347 175/300 Write block 2905 len 8192 data: 6 6
18348 176/300 Read block 4990 len 24576
18349 177/300 Read block 850 len 4096
18350 178/300 Write block 1799 len 32768 data: 4 4 4 4 4 4 5 4
18351 179/300 Write block 3366 len 8192 data: 3 3
18352 180/300 Read block 3692 len 36864
18353 181/300 Read block 8 len 24576
18354 182/300 Read block 4690 len 12288
18355 183/300 Read block 300 len 4096
18356 184/300 Write block 1792 len 20480 data: 6 4 4 3 3
18357 185/300 Flush
18358 186/300 Read block 455 len 4096
18359 187/300 Read block 2474 len 24576
18360 188/300 Flush
18361 189/300 Flush
18362 190/300 Write block 1633 len 32768 data: 4 4 3 3 2 3 3 3
18363 191/300 Flush
18364 192/300 Read block 2287 len 12288
18365 193/300 Read block 4026 len 40960
18366 194/300 Read block 2889 len 8192
18367 195/300 Write block 1874 len 16384 data: 5 5 4 5
18368 196/300 Read block 2224 len 40960
18369 197/300 Read block 1616 len 16384
18370 198/300 Read block 1891 len 4096
18371 199/300 Read block 3583 len 12288
18372 200/300 Read block 902 len 20480
18373 201/300 Read block 4422 len 36864
18374 202/300 Read block 1303 len 40960
18375 203/300 Read block 2598 len 16384
18376 204/300 Write block 1618 len 36864 data: 6 5 5 4 6 6 6 8 7
18377 205/300 Read block 980 len 20480
18378 206/300 Write block 4936 len 20480 data: 4 4 3 3 3
18379 207/300 Write block 2630 len 40960 data: 5 5 5 4 4 3 3 3 2 2
18380 208/300 Read block 1237 len 32768
18381 209/300 Read block 220 len 8192
18382 210/300 Write block 3070 len 24576 data: 3 2 2 2 2 3
18383 211/300 Read block 1605 len 4096
18384 212/300 Read block 3605 len 8192
18385 213/300 Read block 884 len 16384
18386 214/300 Write block 1908 len 8192 data: 6 6
18387 215/300 Write block 4195 len 4096 data: 3
18388 216/300 Read block 112 len 16384
18389 217/300 Write block 4798 len 36864 data: 4 4 4 4 4 4 3 2 2
18390 218/300 Read block 2514 len 36864
18391 219/300 Write block 1811 len 12288 data: 8 6 7
18392 220/300 Read block 2532 len 36864
18393 221/300 Write block 3971 len 24576 data: 3 2 2 3 4 6
18394 222/300 Read block 1583 len 12288
18395 223/300 Read block 3623 len 40960
18396 224/300 Read block 4117 len 36864
18397 225/300 Read block 359 len 28672
18398 226/300 Write block 2884 len 4096 data: 5
18399 227/300 Read block 4162 len 4096
18400 228/300 Flush
18401 229/300 Write block 2042 len 20480 data: 5 4 4 5 3
18402 230/300 Write block 4443 len 8192 data: 5 5
18403 231/300 Flush
18404 232/300 Read block 1180 len 24576
18405 233/300 Write block 438 len 16384 data: 2 3 3 4
18406 234/300 Read block 3832 len 36864
18407 235/300 Read block 4354 len 24576
18408 236/300 Flush
18409 237/300 Write block 3137 len 16384 data: 8 8 8 7
18410 238/300 Flush
18411 239/300 Write block 3184 len 32768 data: 2 3 3 4 4 4 3 3
18412 240/300 Read block 4014 len 36864
18413 241/300 Flush
18414 242/300 Read block 1364 len 40960
18415 243/300 Write block 804 len 24576 data: 3 4 4 3 3 3
18416 244/300 Write block 970 len 8192 data: 5 4
18417 245/300 Write block 4122 len 24576 data: 7 6 4 4 3 3
18418 246/300 Write block 988 len 28672 data: 5 2 2 2 2 2 2
18419 247/300 Flush
18420 248/300 Read block 1007 len 4096
18421 249/300 Write block 1163 len 20480 data: 2 3 3 4 4
18422 250/300 Write block 4586 len 20480 data: 3 3 3 3 3
18423 251/300 Write block 617 len 8192 data: 2 2
18424 252/300 Read block 3954 len 12288
18425 253/300 Flush
18426 254/300 Read block 3199 len 40960
18427 255/300 Read block 460 len 32768
18428 256/300 Write block 3041 len 16384 data: 4 3 3 3
18429 257/300 Write block 4474 len 16384 data: 6 6 6 5
18430 258/300 Write block 3750 len 8192 data: 2 2
18431 259/300 Write block 3551 len 20480 data: 5 5 5 5 6
18432 260/300 Read block 3801 len 16384
18433 261/300 Write block 1563 len 24576 data: 4 3 2 2 2 2
18434 262/300 Write block 3780 len 32768 data: 2 2 2 2 2 3 3 3
18435 263/300 Read block 625 len 28672
18436 264/300 Write block 2073 len 40960 data: 5 5 5 5 4 3 3 3 3 2
18437 265/300 Write block 2410 len 28672 data: 2 3 2 4 4 3 3
18438 266/300 Read block 1203 len 20480
18439 267/300 Read block 497 len 8192
18440 268/300 Write block 4913 len 24576 data: 5 5 8 10 12 10
18441 269/300 Read block 4827 len 16384
18442 270/300 Read block 347 len 4096
18443 271/300 Read block 4065 len 12288
18444 272/300 Flush
18445 273/300 Read block 4659 len 4096
18446 274/300 Flush
18447 275/300 Write block 3493 len 8192 data: 3 4
18448 276/300 Write block 1878 len 4096 data: 3
18449 277/300 Flush
18450 278/300 Read block 4566 len 36864
18451 279/300 Read block 4060 len 36864
18452 280/300 Read block 1736 len 36864
18453 281/300 Write block 4400 len 4096 data: 4
18454 282/300 Write block 241 len 28672 data: 5 5 3 2 3 3 3
18455 283/300 Write block 4724 len 12288 data: 3 4 4
18456 284/300 Write block 1675 len 36864 data: 4 4 4 3 3 4 3 3 4
18457 285/300 Read block 4906 len 40960
18458 286/300 Read block 2365 len 32768
18459 287/300 Write block 4238 len 28672 data: 2 2 2 2 2 2 3
18460 288/300 Write block 4671 len 32768 data: 5 4 4 4 5 5 4 3
18461 289/300 Flush
18462 290/300 Read block 979 len 24576
18463 291/300 Write block 3968 len 20480 data: 4 3 3 4 3
18464 292/300 Write block 2589 len 12288 data: 5 4 3
18465 293/300 Write block 1819 len 24576 data: 7 7 7 7 6 5
18466 294/300 Read block 4368 len 24576
18467 295/300 Read block 1065 len 40960
18468 296/300 Write block 1211 len 12288 data: 3 5 6
18469 297/300 Read block 3404 len 36864
18470 298/300 Read block 3119 len 24576
18471 299/300 Read block 3721 len 4096
18472 300/300 Flush
18473 301/300 Flush
18474 Main task has work to do, go find it
18475 got dsc Start(0)
18476 start 0
18477 Feb 21 06:49:31.252 INFO request completed, latency_us: 236, response_code: 204, uri: /start/cid/0, method: POST, req_id: af9007fe-8282-4535-ad46-639e8a97a6e7, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
18478 Replay: started 0, returned:Ok(())
18479 ----------------------------------------------------------------
18480 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18481 States: Offline Active Active
18482 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18483 5529 Acked Read 1 Sent Done Done false
18484 5530 Acked Write 6 Sent Done Done false
18485 5531 Acked Write 2 Sent Done Done false
18486 5532 Acked Read 9 Sent Done Done false
18487 5533 Acked Write 1 Sent Done Done false
18488 5534 Acked Read 9 Sent Done Done false
18489 5535 Acked Flush 0 Sent Done Done false
18490 5536 Acked Flush 0 Sent Done Done false
18491 5537 Acked Write 1 Sent Done Done false
18492 5538 Acked Read 5 Sent Done Done false
18493 5539 Acked Read 3 Sent Done Done false
18494 5540 Acked Read 8 Sent Done Done false
18495 5541 Acked Read 9 Sent Done Done false
18496 5542 Acked Read 9 Sent Done Done false
18497 5543 Acked Read 8 Sent Done Done false
18498 5544 Acked Read 7 Sent Done Done false
18499 5545 Acked Write 6 Sent Done Done false
18500 5546 Acked Read 1 Sent Done Done false
18501 5547 Acked Write 7 Sent Done Done false
18502 5548 Acked Read 2 Sent Done Done false
18503 5549 Acked Read 7 Sent Done Done false
18504 5550 Acked Read 4 Sent Done Done false
18505 5551 Acked Read 4 Sent Done Done false
18506 5552 Acked Read 7 Sent Done Done false
18507 5553 Acked Flush 0 Sent Done Done false
18508 5554 Acked Write 8 Sent Done Done false
18509 5555 Acked Flush 0 Sent Done Done false
18510 5556 Acked Read 10 Sent Done Done false
18511 5557 Acked Write 5 Sent Done Done false
18512 5558 Acked Read 2 Sent Done Done false
18513 5559 Acked Write 2 Sent Done Done false
18514 5560 Acked Write 10 Sent Done Done false
18515 5561 Acked Read 8 Sent Done Done false
18516 5562 Acked Read 2 Sent Done Done false
18517 5563 Acked Read 8 Sent Done Done false
18518 5564 Acked Read 2 Sent Done Done false
18519 5565 Acked Write 3 Sent Done Done false
18520 5566 Acked Write 4 Sent Done Done false
18521 5567 Acked Write 6 Sent Done Done false
18522 5568 Acked Flush 0 Sent Done Done false
18523 5569 Acked Write 2 Sent Done Done false
18524 5570 Acked Read 1 Sent Done Done false
18525 5571 Acked Read 10 Sent Done Done false
18526 5572 Acked Write 9 Sent Done Done false
18527 5573 Acked Read 5 Sent Done Done false
18528 5574 Acked Read 2 Sent Done Done false
18529 5575 Acked Read 6 Sent Done Done false
18530 5576 Acked Write 7 Sent Done Done false
18531 5577 Acked Write 7 Sent Done Done false
18532 5578 Acked Read 1 Sent Done Done false
18533 5579 Acked Read 1 Sent Done Done false
18534 5580 Acked Write 7 Sent Done Done false
18535 5581 Acked Write 9 Sent Done Done false
18536 5582 Acked Write 8 Sent Done Done false
18537 5583 Acked Read 8 Sent Done Done false
18538 5584 Acked Flush 0 Sent Done Done false
18539 5585 Acked Write 9 Sent Done Done false
18540 5586 Acked Write 10 Sent Done Done false
18541 5587 Acked Read 8 Sent Done Done false
18542 5588 Acked Read 7 Sent Done Done false
18543 5589 Acked Read 10 Sent Done Done false
18544 5590 Acked Read 8 Sent Done Done false
18545 5591 Acked Write 4 Sent Done Done false
18546 5592 Acked Read 6 Sent Done Done false
18547 5593 Acked Write 7 Sent Done Done false
18548 5594 Acked Write 10 Sent Done Done false
18549 5595 Acked Write 8 Sent Done Done false
18550 5596 Acked Read 8 Sent Done Done false
18551 5597 Acked Read 2 Sent Done Done false
18552 5598 Acked Write 2 Sent Done Done false
18553 5599 Acked Write 3 Sent Done Done false
18554 5600 Acked Flush 0 Sent Done Done false
18555 5601 Acked Write 7 Sent Done Done false
18556 5602 Acked Read 10 Sent Done Done false
18557 5603 Acked Write 3 Sent Done Done false
18558 5604 Acked Read 9 Sent Done Done false
18559 5605 Acked Read 3 Sent Done Done false
18560 5606 Acked Write 9 Sent Done Done false
18561 5607 Acked Read 3 Sent Done Done false
18562 5608 Acked Read 7 Sent Done Done false
18563 5609 Acked Write 1 Sent Done Done false
18564 5610 Acked Read 6 Sent Done Done false
18565 5611 Acked Read 10 Sent Done Done false
18566 5612 Acked Read 5 Sent Done Done false
18567 5613 Acked Write 10 Sent Done Done false
18568 5614 Acked Read 7 Sent Done Done false
18569 5615 Acked Write 4 Sent Done Done false
18570 5616 Acked Write 1 Sent Done Done false
18571 5617 Acked Write 7 Sent Done Done false
18572 5618 Acked Read 9 Sent Done Done false
18573 5619 Acked Read 1 Sent Done Done false
18574 5620 Acked Write 10 Sent Done Done false
18575 5621 Acked Read 1 Sent Done Done false
18576 5622 Acked Flush 0 Sent Done Done false
18577 5623 Acked Read 5 Sent Done Done false
18578 5624 Acked Write 6 Sent Done Done false
18579 5625 Acked Write 6 Sent Done Done false
18580 5626 Acked Flush 0 Sent Done Done false
18581 5627 Acked Read 3 Sent Done Done false
18582 5628 Acked Write 4 Sent Done Done false
18583 5629 Acked Read 3 Sent Done Done false
18584 5630 Acked Write 8 Sent Done Done false
18585 5631 Acked Read 6 Sent Done Done false
18586 5632 Acked Flush 0 Sent Done Done false
18587 5633 Acked Write 2 Sent Done Done false
18588 5634 Acked Read 5 Sent Done Done false
18589 5635 Acked Read 5 Sent Done Done false
18590 5636 Acked Read 3 Sent Done Done false
18591 5637 Acked Write 4 Sent Done Done false
18592 5638 Acked Read 2 Sent Done Done false
18593 5639 Acked Write 10 Sent Done Done false
18594 5640 Acked Write 10 Sent Done Done false
18595 5641 Acked Read 5 Sent Done Done false
18596 5642 Acked Read 10 Sent Done Done false
18597 5643 Acked Write 6 Sent Done Done false
18598 5644 Acked Read 5 Sent Done Done false
18599 5645 Acked Read 7 Sent Done Done false
18600 5646 Acked Read 6 Sent Done Done false
18601 5647 Acked Read 10 Sent Done Done false
18602 5648 Acked Flush 0 Sent Done Done false
18603 5649 Acked Write 9 Sent Done Done false
18604 5650 Acked Flush 0 Sent Done Done false
18605 5651 Acked Flush 0 Sent Done Done false
18606 5652 Acked Read 4 Sent Done Done false
18607 5653 Acked Read 7 Sent Done Done false
18608 5654 Acked Read 6 Sent Done Done false
18609 5655 Acked Write 9 Sent Done Done false
18610 5656 Acked Read 4 Sent Done Done false
18611 5657 Acked Read 8 Sent Done Done false
18612 5658 Acked Flush 0 Sent Done Done false
18613 5659 Acked Read 6 Sent Done Done false
18614 5660 Acked Read 5 Sent Done Done false
18615 5661 Acked Flush 0 Sent Done Done false
18616 5662 Acked Read 6 Sent Done Done false
18617 5663 Acked Read 10 Sent Done Done false
18618 5664 Acked Write 9 Sent Done Done false
18619 5665 Acked Read 5 Sent Done Done false
18620 5666 Acked Write 1 Sent Done Done false
18621 5667 Acked Read 10 Sent Done Done false
18622 5668 Acked Read 8 Sent Done Done false
18623 5669 Acked Flush 0 Sent Done Done false
18624 5670 Acked Write 1 Sent Done Done false
18625 5671 Acked Write 7 Sent Done Done false
18626 5672 Acked Write 5 Sent Done Done false
18627 5673 Acked Flush 0 Sent Done Done false
18628 5674 Acked Read 5 Sent Done Done false
18629 5675 Acked Write 5 Sent Done Done false
18630 5676 Acked Write 8 Sent Done Done false
18631 5677 Acked Read 6 Sent Done Done false
18632 5678 Acked Write 1 Sent Done Done false
18633 5679 Acked Read 1 Sent Done Done false
18634 5680 Acked Read 9 Sent Done Done false
18635 5681 Acked Read 8 Sent Done Done false
18636 5682 Acked Read 8 Sent Done Done false
18637 5683 Acked Write 10 Sent Done Done false
18638 5684 Acked Flush 0 Sent Done Done false
18639 5685 Acked Write 7 Sent Done Done false
18640 5686 Acked Flush 0 Sent Done Done false
18641 5687 Acked Write 10 Sent Done Done false
18642 5688 Acked Flush 0 Sent Done Done false
18643 5689 Acked Write 3 Sent Done Done false
18644 5690 Acked Read 3 Sent Done Done false
18645 5691 Acked Read 5 Sent Done Done false
18646 5692 Acked Read 5 Sent Done Done false
18647 5693 Acked Write 7 Sent Done Done false
18648 5694 Acked Write 6 Sent Done Done false
18649 5695 Acked Read 5 Sent Done Done false
18650 5696 Acked Read 3 Sent Done Done false
18651 5697 Acked Read 8 Sent Done Done false
18652 5698 Acked Read 2 Sent Done Done false
18653 5699 Acked Write 9 Sent Done Done false
18654 5700 Acked Read 10 Sent Done Done false
18655 5701 Acked Write 5 Sent Done Done false
18656 5702 Acked Flush 0 Sent Done Done false
18657 5703 Acked Write 2 Sent Done Done false
18658 5704 Acked Read 6 Sent Done Done false
18659 5705 Acked Read 1 Sent Done Done false
18660 5706 Acked Write 8 Sent Done Done false
18661 5707 Acked Write 2 Sent Done Done false
18662 5708 Acked Read 9 Sent Done Done false
18663 5709 Acked Read 6 Sent Done Done false
18664 5710 Acked Read 3 Sent Done Done false
18665 5711 Acked Read 1 Sent Done Done false
18666 5712 Acked Write 5 Sent Done Done false
18667 5713 Acked Flush 0 Sent Done Done false
18668 5714 Acked Read 1 Sent Done Done false
18669 5715 Acked Read 6 Sent Done Done false
18670 5716 Acked Flush 0 Sent Done Done false
18671 5717 Acked Flush 0 Sent Done Done false
18672 5718 Acked Write 8 Sent Done Done false
18673 5719 Acked Flush 0 Sent Done Done false
18674 5720 Acked Read 3 Sent Done Done false
18675 5721 Acked Read 10 Sent Done Done false
18676 5722 Acked Read 2 Sent Done Done false
18677 5723 Acked Write 4 Sent Done Done false
18678 5724 Acked Read 10 Sent Done Done false
18679 5725 Acked Read 4 Sent Done Done false
18680 5726 Acked Read 1 Sent Done Done false
18681 5727 Acked Read 3 Sent Done Done false
18682 5728 Acked Read 5 Sent Done Done false
18683 5729 Acked Read 9 Sent Done Done false
18684 5730 Acked Read 10 Sent Done Done false
18685 5731 Acked Read 4 Sent Done Done false
18686 5732 Acked Write 9 Sent Done Done false
18687 5733 Acked Read 5 Sent Done Done false
18688 5734 Acked Write 5 Sent Done Done false
18689 5735 Acked Write 10 Sent Done Done false
18690 5736 Acked Read 8 Sent Done Done false
18691 5737 Acked Read 2 Sent Done Done false
18692 5738 Acked Write 6 Sent Done Done false
18693 5739 Acked Read 1 Sent Done Done false
18694 5740 Acked Read 2 Sent Done Done false
18695 5741 Acked Read 4 Sent Done Done false
18696 5742 Acked Write 2 Sent Done Done false
18697 5743 Acked Write 1 Sent Done Done false
18698 5744 Acked Read 4 Sent Done Done false
18699 5745 Acked Write 9 Sent Done Done false
18700 5746 Acked Read 9 Sent Done Done false
18701 5747 Acked Write 3 Sent Done Done false
18702 5748 Acked Read 9 Sent Done Done false
18703 5749 Acked Write 6 Sent Done Done false
18704 5750 Acked Read 3 Sent Done Done false
18705 5751 Acked Read 10 Sent Done Done false
18706 5752 Acked Read 9 Sent Done Done false
18707 5753 Acked Read 7 Sent Done Done false
18708 5754 Acked Write 1 Sent Done Done false
18709 5755 Acked Read 1 Sent Done Done false
18710 5756 Acked Flush 0 Sent Done Done false
18711 5757 Acked Write 5 Sent Done Done false
18712 5758 Acked Write 2 Sent Done Done false
18713 5759 Acked Flush 0 Sent Done Done false
18714 5760 Acked Read 6 Sent Done Done false
18715 5761 Acked Write 4 Sent Done Done false
18716 5762 Acked Read 9 Sent Done Done false
18717 5763 Acked Read 6 Sent Done Done false
18718 5764 Acked Flush 0 Sent Done Done false
18719 5765 Acked Write 4 Sent Done Done false
18720 5766 Acked Flush 0 Sent Done Done false
18721 5767 Acked Write 8 Sent Done Done false
18722 5768 Acked Read 9 Sent Done Done false
18723 5769 Acked Flush 0 Sent Done Done false
18724 5770 Acked Read 10 Sent Done Done false
18725 5771 Acked Write 6 Sent Done Done false
18726 5772 Acked Write 2 Sent Done Done false
18727 5773 Acked Write 6 Sent Done Done false
18728 5774 Acked Write 7 Sent Done Done false
18729 5775 Acked Flush 0 Sent Done Done false
18730 5776 Acked Read 1 Sent Done Done false
18731 5777 Acked Write 5 Sent Done Done false
18732 5778 Acked Write 5 Sent Done Done false
18733 5779 Acked Write 2 Sent Done Done false
18734 5780 Acked Read 3 Sent Done Done false
18735 5781 Acked Flush 0 Sent Done Done false
18736 5782 Acked Read 10 Sent Done Done false
18737 5783 Acked Read 8 Sent Done Done false
18738 5784 Acked Write 4 Sent Done Done false
18739 5785 Acked Write 4 Sent Done Done false
18740 5786 Acked Write 2 Sent Done Done false
18741 5787 Acked Write 5 Sent Done Done false
18742 5788 Acked Read 4 Sent Done Done false
18743 5789 Acked Write 6 Sent Done Done false
18744 5790 Acked Write 8 Sent Done Done false
18745 5791 Acked Read 7 Sent Done Done false
18746 5792 Acked Write 10 Sent Done Done false
18747 5793 Acked Write 7 Sent Done Done false
18748 5794 Acked Read 5 Sent Done Done false
18749 5795 Acked Read 2 Sent Done Done false
18750 5796 Acked Write 6 Sent Done Done false
18751 5797 Acked Read 4 Sent Done Done false
18752 5798 Acked Read 1 Sent Done Done false
18753 5799 Acked Read 3 Sent Done Done false
18754 5800 Acked Flush 0 Sent Done Done false
18755 5801 Acked Read 1 Sent Done Done false
18756 5802 Acked Flush 0 Sent Done Done false
18757 5803 Acked Write 2 Sent Done Done false
18758 5804 Acked Write 1 Sent Done Done false
18759 5805 Acked Flush 0 Sent Done Done false
18760 5806 Acked Read 9 Sent Done Done false
18761 5807 Acked Read 9 Sent Done Done false
18762 5808 Acked Read 9 Sent Done Done false
18763 5809 Acked Write 1 Sent Done Done false
18764 5810 Acked Write 7 Sent Done Done false
18765 5811 Acked Write 3 Sent Done Done false
18766 5812 Acked Write 9 Sent Done Done false
18767 5813 Acked Read 10 Sent Done Done false
18768 5814 Acked Read 8 Sent Done Done false
18769 5815 Acked Write 7 Sent Done Done false
18770 5816 Acked Write 8 Sent Done Done false
18771 5817 Acked Flush 0 Sent Done Done false
18772 5818 Acked Read 6 Sent Done Done false
18773 5819 Acked Write 5 Sent Done Done false
18774 5820 Acked Write 3 Sent Done Done false
18775 5821 Acked Write 6 Sent Done Done false
18776 5822 Acked Read 6 Sent Done Done false
18777 5823 Acked Read 10 Sent Done Done false
18778 5824 Acked Write 3 Sent Done Done false
18779 5825 Acked Read 9 Sent Done Done false
18780 5826 Acked Read 6 Sent Done Done false
18781 5827 Acked Read 1 Sent Done Done false
18782 5828 Acked Flush 0 Sent Done Done false
18783 5829 Acked Flush 0 Sent Done Done false
18784 STATES DS:0 DS:1 DS:2 TOTAL
18785 Sent 301 0 0 301
18786 Done 0 301 301 602
18787 Skipped 0 0 0 0
18788 Error 0 0 0 0
18789 Last Flush: Some(JobId(5528)) Some(JobId(5829)) Some(JobId(5829))
18790 Downstairs last five completed: 5528 5527 5526 5525 5524
18791 Upstairs last five completed: 5829 5828 5827 5826 5825
18792 CLIENT: Up:0 ds:301 act:2
187932025-02-21T06:49:32.042ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
187942025-02-21T06:49:33.269ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
187952025-02-21T06:49:34.046ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
18796 [8810] got start action
18797 [8810] I am going to restart
18798 Starting downstairs at port 8810
18799 Make output file at "/tmp/dsc/downstairs-8810.txt"
18800 [8810][0] reports Starting
18801 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1222)
18802 [8810][0] reports Running
188032025-02-21T06:49:35.048ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
18804 ----------------------------------------------------------------
18805 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18806 States: Offline Active Active
18807 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18808 5529 Acked Read 1 Sent Done Done false
18809 5530 Acked Write 6 Sent Done Done false
18810 5531 Acked Write 2 Sent Done Done false
18811 5532 Acked Read 9 Sent Done Done false
18812 5533 Acked Write 1 Sent Done Done false
18813 5534 Acked Read 9 Sent Done Done false
18814 5535 Acked Flush 0 Sent Done Done false
18815 5536 Acked Flush 0 Sent Done Done false
18816 5537 Acked Write 1 Sent Done Done false
18817 5538 Acked Read 5 Sent Done Done false
18818 5539 Acked Read 3 Sent Done Done false
18819 5540 Acked Read 8 Sent Done Done false
18820 5541 Acked Read 9 Sent Done Done false
18821 5542 Acked Read 9 Sent Done Done false
18822 5543 Acked Read 8 Sent Done Done false
18823 5544 Acked Read 7 Sent Done Done false
18824 5545 Acked Write 6 Sent Done Done false
18825 5546 Acked Read 1 Sent Done Done false
18826 5547 Acked Write 7 Sent Done Done false
18827 5548 Acked Read 2 Sent Done Done false
18828 5549 Acked Read 7 Sent Done Done false
18829 5550 Acked Read 4 Sent Done Done false
18830 5551 Acked Read 4 Sent Done Done false
18831 5552 Acked Read 7 Sent Done Done false
18832 5553 Acked Flush 0 Sent Done Done false
18833 5554 Acked Write 8 Sent Done Done false
18834 5555 Acked Flush 0 Sent Done Done false
18835 5556 Acked Read 10 Sent Done Done false
18836 5557 Acked Write 5 Sent Done Done false
18837 5558 Acked Read 2 Sent Done Done false
18838 5559 Acked Write 2 Sent Done Done false
18839 5560 Acked Write 10 Sent Done Done false
18840 5561 Acked Read 8 Sent Done Done false
18841 5562 Acked Read 2 Sent Done Done false
18842 5563 Acked Read 8 Sent Done Done false
18843 5564 Acked Read 2 Sent Done Done false
18844 5565 Acked Write 3 Sent Done Done false
18845 5566 Acked Write 4 Sent Done Done false
18846 5567 Acked Write 6 Sent Done Done false
18847 5568 Acked Flush 0 Sent Done Done false
18848 5569 Acked Write 2 Sent Done Done false
18849 5570 Acked Read 1 Sent Done Done false
18850 5571 Acked Read 10 Sent Done Done false
18851 5572 Acked Write 9 Sent Done Done false
18852 5573 Acked Read 5 Sent Done Done false
18853 5574 Acked Read 2 Sent Done Done false
18854 5575 Acked Read 6 Sent Done Done false
18855 5576 Acked Write 7 Sent Done Done false
18856 5577 Acked Write 7 Sent Done Done false
18857 5578 Acked Read 1 Sent Done Done false
18858 5579 Acked Read 1 Sent Done Done false
18859 5580 Acked Write 7 Sent Done Done false
18860 5581 Acked Write 9 Sent Done Done false
18861 5582 Acked Write 8 Sent Done Done false
18862 5583 Acked Read 8 Sent Done Done false
18863 5584 Acked Flush 0 Sent Done Done false
18864 5585 Acked Write 9 Sent Done Done false
18865 5586 Acked Write 10 Sent Done Done false
18866 5587 Acked Read 8 Sent Done Done false
18867 5588 Acked Read 7 Sent Done Done false
18868 5589 Acked Read 10 Sent Done Done false
18869 5590 Acked Read 8 Sent Done Done false
18870 5591 Acked Write 4 Sent Done Done false
18871 5592 Acked Read 6 Sent Done Done false
18872 5593 Acked Write 7 Sent Done Done false
18873 5594 Acked Write 10 Sent Done Done false
18874 5595 Acked Write 8 Sent Done Done false
18875 5596 Acked Read 8 Sent Done Done false
18876 5597 Acked Read 2 Sent Done Done false
18877 5598 Acked Write 2 Sent Done Done false
18878 5599 Acked Write 3 Sent Done Done false
18879 5600 Acked Flush 0 Sent Done Done false
18880 5601 Acked Write 7 Sent Done Done false
18881 5602 Acked Read 10 Sent Done Done false
18882 5603 Acked Write 3 Sent Done Done false
18883 5604 Acked Read 9 Sent Done Done false
18884 5605 Acked Read 3 Sent Done Done false
18885 5606 Acked Write 9 Sent Done Done false
18886 5607 Acked Read 3 Sent Done Done false
18887 5608 Acked Read 7 Sent Done Done false
18888 5609 Acked Write 1 Sent Done Done false
18889 5610 Acked Read 6 Sent Done Done false
18890 5611 Acked Read 10 Sent Done Done false
18891 5612 Acked Read 5 Sent Done Done false
18892 5613 Acked Write 10 Sent Done Done false
18893 5614 Acked Read 7 Sent Done Done false
18894 5615 Acked Write 4 Sent Done Done false
18895 5616 Acked Write 1 Sent Done Done false
18896 5617 Acked Write 7 Sent Done Done false
18897 5618 Acked Read 9 Sent Done Done false
18898 5619 Acked Read 1 Sent Done Done false
18899 5620 Acked Write 10 Sent Done Done false
18900 5621 Acked Read 1 Sent Done Done false
18901 5622 Acked Flush 0 Sent Done Done false
18902 5623 Acked Read 5 Sent Done Done false
18903 5624 Acked Write 6 Sent Done Done false
18904 5625 Acked Write 6 Sent Done Done false
18905 5626 Acked Flush 0 Sent Done Done false
18906 5627 Acked Read 3 Sent Done Done false
18907 5628 Acked Write 4 Sent Done Done false
18908 5629 Acked Read 3 Sent Done Done false
18909 5630 Acked Write 8 Sent Done Done false
18910 5631 Acked Read 6 Sent Done Done false
18911 5632 Acked Flush 0 Sent Done Done false
18912 5633 Acked Write 2 Sent Done Done false
18913 5634 Acked Read 5 Sent Done Done false
18914 5635 Acked Read 5 Sent Done Done false
18915 5636 Acked Read 3 Sent Done Done false
18916 5637 Acked Write 4 Sent Done Done false
18917 5638 Acked Read 2 Sent Done Done false
18918 5639 Acked Write 10 Sent Done Done false
18919 5640 Acked Write 10 Sent Done Done false
18920 5641 Acked Read 5 Sent Done Done false
18921 5642 Acked Read 10 Sent Done Done false
18922 5643 Acked Write 6 Sent Done Done false
18923 5644 Acked Read 5 Sent Done Done false
18924 5645 Acked Read 7 Sent Done Done false
18925 5646 Acked Read 6 Sent Done Done false
18926 5647 Acked Read 10 Sent Done Done false
18927 5648 Acked Flush 0 Sent Done Done false
18928 5649 Acked Write 9 Sent Done Done false
18929 5650 Acked Flush 0 Sent Done Done false
18930 5651 Acked Flush 0 Sent Done Done false
18931 5652 Acked Read 4 Sent Done Done false
18932 5653 Acked Read 7 Sent Done Done false
18933 5654 Acked Read 6 Sent Done Done false
18934 5655 Acked Write 9 Sent Done Done false
18935 5656 Acked Read 4 Sent Done Done false
18936 5657 Acked Read 8 Sent Done Done false
18937 5658 Acked Flush 0 Sent Done Done false
18938 5659 Acked Read 6 Sent Done Done false
18939 5660 Acked Read 5 Sent Done Done false
18940 5661 Acked Flush 0 Sent Done Done false
18941 5662 Acked Read 6 Sent Done Done false
18942 5663 Acked Read 10 Sent Done Done false
18943 5664 Acked Write 9 Sent Done Done false
18944 5665 Acked Read 5 Sent Done Done false
18945 5666 Acked Write 1 Sent Done Done false
18946 5667 Acked Read 10 Sent Done Done false
18947 5668 Acked Read 8 Sent Done Done false
18948 5669 Acked Flush 0 Sent Done Done false
18949 5670 Acked Write 1 Sent Done Done false
18950 5671 Acked Write 7 Sent Done Done false
18951 5672 Acked Write 5 Sent Done Done false
18952 5673 Acked Flush 0 Sent Done Done false
18953 5674 Acked Read 5 Sent Done Done false
18954 5675 Acked Write 5 Sent Done Done false
18955 5676 Acked Write 8 Sent Done Done false
18956 5677 Acked Read 6 Sent Done Done false
18957 5678 Acked Write 1 Sent Done Done false
18958 5679 Acked Read 1 Sent Done Done false
18959 5680 Acked Read 9 Sent Done Done false
18960 5681 Acked Read 8 Sent Done Done false
18961 5682 Acked Read 8 Sent Done Done false
18962 5683 Acked Write 10 Sent Done Done false
18963 5684 Acked Flush 0 Sent Done Done false
18964 5685 Acked Write 7 Sent Done Done false
18965 5686 Acked Flush 0 Sent Done Done false
18966 5687 Acked Write 10 Sent Done Done false
18967 5688 Acked Flush 0 Sent Done Done false
18968 5689 Acked Write 3 Sent Done Done false
18969 5690 Acked Read 3 Sent Done Done false
18970 5691 Acked Read 5 Sent Done Done false
18971 5692 Acked Read 5 Sent Done Done false
18972 5693 Acked Write 7 Sent Done Done false
18973 5694 Acked Write 6 Sent Done Done false
18974 5695 Acked Read 5 Sent Done Done false
18975 5696 Acked Read 3 Sent Done Done false
18976 5697 Acked Read 8 Sent Done Done false
18977 5698 Acked Read 2 Sent Done Done false
18978 5699 Acked Write 9 Sent Done Done false
18979 5700 Acked Read 10 Sent Done Done false
18980 5701 Acked Write 5 Sent Done Done false
18981 5702 Acked Flush 0 Sent Done Done false
18982 5703 Acked Write 2 Sent Done Done false
18983 5704 Acked Read 6 Sent Done Done false
18984 5705 Acked Read 1 Sent Done Done false
18985 5706 Acked Write 8 Sent Done Done false
18986 5707 Acked Write 2 Sent Done Done false
18987 5708 Acked Read 9 Sent Done Done false
18988 5709 Acked Read 6 Sent Done Done false
18989 5710 Acked Read 3 Sent Done Done false
18990 5711 Acked Read 1 Sent Done Done false
18991 5712 Acked Write 5 Sent Done Done false
18992 5713 Acked Flush 0 Sent Done Done false
18993 5714 Acked Read 1 Sent Done Done false
18994 5715 Acked Read 6 Sent Done Done false
18995 5716 Acked Flush 0 Sent Done Done false
18996 5717 Acked Flush 0 Sent Done Done false
18997 5718 Acked Write 8 Sent Done Done false
18998 5719 Acked Flush 0 Sent Done Done false
18999 5720 Acked Read 3 Sent Done Done false
19000 5721 Acked Read 10 Sent Done Done false
19001 5722 Acked Read 2 Sent Done Done false
19002 5723 Acked Write 4 Sent Done Done false
19003 5724 Acked Read 10 Sent Done Done false
19004 5725 Acked Read 4 Sent Done Done false
19005 5726 Acked Read 1 Sent Done Done false
19006 5727 Acked Read 3 Sent Done Done false
19007 5728 Acked Read 5 Sent Done Done false
19008 5729 Acked Read 9 Sent Done Done false
19009 5730 Acked Read 10 Sent Done Done false
19010 5731 Acked Read 4 Sent Done Done false
19011 5732 Acked Write 9 Sent Done Done false
19012 5733 Acked Read 5 Sent Done Done false
19013 5734 Acked Write 5 Sent Done Done false
19014 5735 Acked Write 10 Sent Done Done false
19015 5736 Acked Read 8 Sent Done Done false
19016 5737 Acked Read 2 Sent Done Done false
19017 5738 Acked Write 6 Sent Done Done false
19018 5739 Acked Read 1 Sent Done Done false
19019 5740 Acked Read 2 Sent Done Done false
19020 5741 Acked Read 4 Sent Done Done false
19021 5742 Acked Write 2 Sent Done Done false
19022 5743 Acked Write 1 Sent Done Done false
19023 5744 Acked Read 4 Sent Done Done false
19024 5745 Acked Write 9 Sent Done Done false
19025 5746 Acked Read 9 Sent Done Done false
19026 5747 Acked Write 3 Sent Done Done false
19027 5748 Acked Read 9 Sent Done Done false
19028 5749 Acked Write 6 Sent Done Done false
19029 5750 Acked Read 3 Sent Done Done false
19030 5751 Acked Read 10 Sent Done Done false
19031 5752 Acked Read 9 Sent Done Done false
19032 5753 Acked Read 7 Sent Done Done false
19033 5754 Acked Write 1 Sent Done Done false
19034 5755 Acked Read 1 Sent Done Done false
19035 5756 Acked Flush 0 Sent Done Done false
19036 5757 Acked Write 5 Sent Done Done false
19037 5758 Acked Write 2 Sent Done Done false
19038 5759 Acked Flush 0 Sent Done Done false
19039 5760 Acked Read 6 Sent Done Done false
19040 5761 Acked Write 4 Sent Done Done false
19041 5762 Acked Read 9 Sent Done Done false
19042 5763 Acked Read 6 Sent Done Done false
19043 5764 Acked Flush 0 Sent Done Done false
19044 5765 Acked Write 4 Sent Done Done false
19045 5766 Acked Flush 0 Sent Done Done false
19046 5767 Acked Write 8 Sent Done Done false
19047 5768 Acked Read 9 Sent Done Done false
19048 5769 Acked Flush 0 Sent Done Done false
19049 5770 Acked Read 10 Sent Done Done false
19050 5771 Acked Write 6 Sent Done Done false
19051 5772 Acked Write 2 Sent Done Done false
19052 5773 Acked Write 6 Sent Done Done false
19053 5774 Acked Write 7 Sent Done Done false
19054 5775 Acked Flush 0 Sent Done Done false
19055 5776 Acked Read 1 Sent Done Done false
19056 5777 Acked Write 5 Sent Done Done false
19057 5778 Acked Write 5 Sent Done Done false
19058 5779 Acked Write 2 Sent Done Done false
19059 5780 Acked Read 3 Sent Done Done false
19060 5781 Acked Flush 0 Sent Done Done false
19061 5782 Acked Read 10 Sent Done Done false
19062 5783 Acked Read 8 Sent Done Done false
19063 5784 Acked Write 4 Sent Done Done false
19064 5785 Acked Write 4 Sent Done Done false
19065 5786 Acked Write 2 Sent Done Done false
19066 5787 Acked Write 5 Sent Done Done false
19067 5788 Acked Read 4 Sent Done Done false
19068 5789 Acked Write 6 Sent Done Done false
19069 5790 Acked Write 8 Sent Done Done false
19070 5791 Acked Read 7 Sent Done Done false
19071 5792 Acked Write 10 Sent Done Done false
19072 5793 Acked Write 7 Sent Done Done false
19073 5794 Acked Read 5 Sent Done Done false
19074 5795 Acked Read 2 Sent Done Done false
19075 5796 Acked Write 6 Sent Done Done false
19076 5797 Acked Read 4 Sent Done Done false
19077 5798 Acked Read 1 Sent Done Done false
19078 5799 Acked Read 3 Sent Done Done false
19079 5800 Acked Flush 0 Sent Done Done false
19080 5801 Acked Read 1 Sent Done Done false
19081 5802 Acked Flush 0 Sent Done Done false
19082 5803 Acked Write 2 Sent Done Done false
19083 5804 Acked Write 1 Sent Done Done false
19084 5805 Acked Flush 0 Sent Done Done false
19085 5806 Acked Read 9 Sent Done Done false
19086 5807 Acked Read 9 Sent Done Done false
19087 5808 Acked Read 9 Sent Done Done false
19088 5809 Acked Write 1 Sent Done Done false
19089 5810 Acked Write 7 Sent Done Done false
19090 5811 Acked Write 3 Sent Done Done false
19091 5812 Acked Write 9 Sent Done Done false
19092 5813 Acked Read 10 Sent Done Done false
19093 5814 Acked Read 8 Sent Done Done false
19094 5815 Acked Write 7 Sent Done Done false
19095 5816 Acked Write 8 Sent Done Done false
19096 5817 Acked Flush 0 Sent Done Done false
19097 5818 Acked Read 6 Sent Done Done false
19098 5819 Acked Write 5 Sent Done Done false
19099 5820 Acked Write 3 Sent Done Done false
19100 5821 Acked Write 6 Sent Done Done false
19101 5822 Acked Read 6 Sent Done Done false
19102 5823 Acked Read 10 Sent Done Done false
19103 5824 Acked Write 3 Sent Done Done false
19104 5825 Acked Read 9 Sent Done Done false
19105 5826 Acked Read 6 Sent Done Done false
19106 5827 Acked Read 1 Sent Done Done false
19107 5828 Acked Flush 0 Sent Done Done false
19108 5829 Acked Flush 0 Sent Done Done false
19109 STATES DS:0 DS:1 DS:2 TOTAL
19110 Sent 301 0 0 301
19111 Done 0 301 301 602
19112 Skipped 0 0 0 0
19113 Error 0 0 0 0
19114 Last Flush: Some(JobId(5528)) Some(JobId(5829)) Some(JobId(5829))
19115 Downstairs last five completed: 5528 5527 5526 5525 5524
19116 Upstairs last five completed: 5829 5828 5827 5826 5825
19117 CLIENT: Up:0 ds:301 act:2
191182025-02-21T06:49:36.050ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
191192025-02-21T06:49:37.006ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
191202025-02-21T06:49:37.006ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:35440) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
191212025-02-21T06:49:37.007ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
191222025-02-21T06:49:37.008ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
191232025-02-21T06:49:37.008ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
191242025-02-21T06:49:37.008ZINFOcrucible: send last flush ID to this DS: Some(JobId(5528)) = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
191252025-02-21T06:49:37.008ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
191262025-02-21T06:49:37.008ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(5528)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
191272025-02-21T06:49:37.008ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(5528)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
19128 ----------------------------------------------------------------
19129 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19130 Downstairs last five completed: 5829 5828 5827 5826 5825
19131 Upstairs last five completed: 5829 5828 5827 5826 5825
19132 CLIENT: Up:0 ds:0 act:3
19133 Replay: All jobs finished
19134 Main task has work to do, go find it
19135 got dsc Stop(1)
19136 stop 1
19137 [8820] Got stop action so:false kr:false
19138 Feb 21 06:49:39.258 INFO request completed, latency_us: 339, response_code: 204, uri: /stop/cid/1, method: POST, req_id: cbb33f86-7201-4f9b-89e9-9e2c23aceef9, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
191392025-02-21T06:49:39.258ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
191402025-02-21T06:49:39.258ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
191412025-02-21T06:49:39.258ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
19142 Feb 21 06:49:39.258 INFO request completed, latency_us: 138, response_code: 200, uri: /state/cid/1, method:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:49:39.258982773Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
19143 GET, req_id: ef70a68d-8207-45b5-a270-f4582fe0288a, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
19144 [8820] Exited with: ExitStatus(unix_wait_status(9))
19145 [8820][1] reports Exit
19146 Feb 21 06:49:43.261 INFO request completed, latency_us: 264, response_code: 200, uri: /state/cid/1, method: GET, req_id: c6cdab59-d4b7-4042-8d0d-88fede3c9093, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
19147 001/300 Read block 4056 len 8192
19148 002/300 Write block 471 len 8192 data: 3 3
19149 003/300 Read block 2128 len 8192
19150 004/300 Read block 3681 len 32768
19151 005/300 Write block 137 len 24576 data: 4 3 4 4 5 5
19152 006/300 Read block 4553 len 32768
19153 007/300 Read block 1695 len 12288
19154 008/300 Write block 2416 len 12288 data: 4 2 2
19155 009/300 Read block 3847 len 16384
19156 010/300 Read block 3712 len 40960
19157 011/300 Write block 2016 len 20480 data: 2 3 3 4 4
19158 012/300 Read block 4142 len 36864
19159 013/300 Read block 4031 len 32768
19160 014/300 Flush
19161 015/300 Write block 3918 len 32768 data: 6 6 6 5 4 3 3 3
19162 016/300 Write block 1778 len 24576 data: 4 4 3 4 4 5
19163 017/300 Read block 3445 len 36864
19164 018/300 Write block 2343 len 4096 data: 4
19165 019/300 Write block 1349 len 36864 data: 3 3 3 5 6 4 5 5 5
19166 020/300 Flush
19167 021/300 Write block 3783 len 28672 data: 3 3 4 4 4 3 3
19168 022/300 Read block 2638 len 36864
19169 023/300 Read block 4382 len 4096
19170 024/300 Read block 2821 len 32768
19171 025/300 Write block 4338 len 12288 data: 2 2 2
19172 026/300 Write block 3266 len 24576 data: 6 5 3 3 3 3
19173 027/300 Read block 772 len 28672
19174 028/300 Read block 339 len 16384
19175 029/300 Write block 2015 len 4096 data: 4
19176 030/300 Write block 1044 len 28672 data: 3 4 5 3 4 4 3
19177 031/300 Write block 2704 len 40960 data: 4 4 4 4 3 3 3 4 4 4
19178 032/300 Write block 4383 len 20480 data: 4 3 4 4 4
19179 033/300 Read block 2626 len 24576
19180 034/300 Write block 311 len 24576 data: 4 4 5 5 5 5
19181 035/300 Read block 3227 len 36864
19182 036/300 Read block 846 len 12288
19183 037/300 Write block 2394 len 16384 data: 5 3 3 3
19184 038/300 Read block 1078 len 12288
19185 039/300 Read block 4472 len 20480
19186 040/300 Write block 2782 len 24576 data: 6 6 6 6 5 6
19187 041/300 Write block 2959 len 8192 data: 4 3
19188 042/300 Write block 394 len 24576 data: 3 3 3 2 3 3
19189 043/300 Write block 2070 len 16384 data: 6 6 6 6
19190 044/300 Write block 3942 len 8192 data: 6 4
19191 045/300 Write block 1461 len 24576 data: 4 4 6 6 6 5
19192 046/300 Write block 4620 len 40960 data: 4 3 5 5 6 7 5 6 5 4
19193 047/300 Write block 3246 len 28672 data: 5 7 7 8 7 5 5
19194 048/300 Read block 4126 len 12288
19195 049/300 Flush
19196 050/300 Read block 4843 len 36864
19197 051/300 Write block 1047 len 20480 data: 4 5 5 4 4
19198 052/300 Read block 759 len 20480
19199 053/300 Write block 3288 len 28672 data: 5 4 3 3 2 2 2
19200 054/300 Read block 986 len 16384
19201 055/300 Write block 2615 len 24576 data: 3 3 3 5 5 5
19202 056/300 Read block 4914 len 40960
19203 057/300 Read block 656 len 4096
19204 058/300 Read block 3802 len 28672
19205 059/300 Flush
19206 060/300 Write block 3721 len 36864 data: 3 4 4 3 3 3 3 4 4
19207 061/300 Read block 658 len 36864
19208 062/300 Read block 1511 len 24576
19209 063/300 Read block 3907 len 16384
19210 064/300 Write block 3878 len 36864 data: 4 4 3 3 3 3 4 5 4
19211 065/300 Write block 1859 len 4096 data: 4
19212 066/300 Write block 2616 len 24576 data: 4 4 6 6 6 5
19213 067/300 Flush
19214 068/300 Read block 837 len 32768
19215 069/300 Write block 3870 len 28672 data: 4 4 5 7 6 4 3
19216 070/300 Read block 4746 len 12288
19217 071/300 Write block 2110 len 36864 data: 4 3 3 4 4 3 3 4 4
19218 072/300 Write block 1191 len 8192 data: 5 4
19219 073/300 Read block 425 len 20480
19220 074/300 Read block 1735 len 4096
19221 075/300 Read block 3113 len 40960
19222 076/300 Read block 3407 len 4096
19223 077/300 Write block 4264 len 32768 data: 2 2 2 2 2 2 2 4
19224 078/300 Flush
19225 079/300 Read block 1579 len 20480
19226 080/300 Write block 4852 len 20480 data: 3 5 5 4 3
19227 081/300 Read block 1369 len 32768
19228 082/300 Write block 2226 len 12288 data: 6 6 6
19229 083/300 Flush
19230 084/300 Read block 2570 len 20480
19231 085/300 Read block 3256 len 12288
19232 086/300 Read block 725 len 4096
19233 087/300 Read block 889 len 20480
19234 088/300 Write block 4467 len 16384 data: 4 4 6 6
19235 089/300 Read block 27 len 8192
19236 090/300 Write block 214 len 20480 data: 6 7 8 8 8
19237 091/300 Read block 4755 len 4096
19238 092/300 Flush
19239 093/300 Write block 2401 len 40960 data: 3 3 3 3 2 2 2 2 2 3
19240 094/300 Flush
19241 095/300 Write block 3410 len 28672 data: 4 4 4 5 5 6 6
19242 096/300 Read block 62 len 4096
19243 097/300 Read block 3907 len 20480
19244 098/300 Read block 4979 len 12288
19245 099/300 Read block 4405 len 12288
19246 100/300 Read block 2000 len 20480
19247 101/300 Write block 4786 len 36864 data: 3 3 4 3 4 3 3 3 4
19248 102/300 Write block 3473 len 32768 data: 4 6 6 6 5 5 6 6
19249 103/300 Write block 2748 len 4096 data: 9
19250 104/300 Read block 3887 len 32768
19251 105/300 Read block 1826 len 40960
19252 106/300 Read block 4581 len 40960
19253 107/300 Read block 1237 len 32768
19254 108/300 Read block 1453 len 20480
19255 109/300 Flush
19256 110/300 Read block 3598 len 8192
19257 111/300 Flush
19258 112/300 Write block 3652 len 36864 data: 4 4 4 3 3 3 4 4 3
19259 113/300 Write block 1170 len 20480 data: 7 6 6 6 6
19260 114/300 Read block 3610 len 16384
19261 115/300 Read block 4059 len 16384
19262 116/300 Read block 3904 len 16384
19263 117/300 Flush
19264 118/300 Read block 2589 len 32768
19265 119/300 Write block 3280 len 28672 data: 2 2 2 4 4 4 4
19266 120/300 Read block 3768 len 20480
19267 121/300 Write block 633 len 28672 data: 3 3 3 3 2 3 4
19268 122/300 Read block 3823 len 16384
19269 123/300 Read block 1882 len 32768
19270 124/300 Read block 1028 len 32768
19271 125/300 Write block 4107 len 24576 data: 4 5 5 4 4 5
19272 126/300 Read block 70 len 4096
19273 127/300 Read block 3933 len 36864
19274 128/300 Write block 4001 len 12288 data: 3 3 5
19275 129/300 Read block 2463 len 8192
19276 130/300 Flush
19277 131/300 Write block 3948 len 24576 data: 4 4 5 3 4 5
19278 132/300 Flush
19279 133/300 Read block 64 len 8192
19280 134/300 Flush
19281 135/300 Read block 1566 len 8192
19282 136/300 Flush
19283 137/300 Write block 1776 len 32768 data: 4 4 5 5 4 5 5 6
19284 138/300 Write block 2034 len 12288 data: 3 3 4
19285 139/300 Write block 2678 len 24576 data: 5 6 6 6 5 5
19286 140/300 Write block 4021 len 12288 data: 4 4 5
19287 141/300 Write block 2137 len 24576 data: 4 4 4 4 3 3
19288 142/300 Write block 2906 len 4096 data: 7
19289 143/300 Read block 3993 len 24576
19290 144/300 Write block 3107 len 32768 data: 3 3 2 2 2 2 3 4
19291 145/300 Read block 887 len 24576
19292 146/300 Write block 1025 len 32768 data: 6 6 6 5 5 5 5 6
19293 147/300 Read block 614 len 12288
19294 148/300 Read block 2244 len 8192
19295 149/300 Read block 2916 len 20480
19296 150/300 Write block 2656 len 12288 data: 4 4 5
19297 151/300 Flush
19298 152/300 Flush
19299 153/300 Read block 4551 len 28672
19300 154/300 Read block 1517 len 20480
19301 155/300 Read block 2688 len 20480
19302 156/300 Read block 3948 len 20480
19303 157/300 Write block 3242 len 40960 data: 3 4 3 3 6 8 8 9 8 6
19304 158/300 Flush
19305 159/300 Read block 4728 len 16384
19306 160/300 Write block 3926 len 36864 data: 3 3 5 5 5 4 5 5 4
19307 161/300 Read block 464 len 36864
19308 162/300 Write block 589 len 36864 data: 3 3 4 3 3 2 2 2 2
19309 163/300 Write block 3452 len 20480 data: 4 3 3 4 4
19310 164/300 Read block 130 len 24576
19311 165/300 Read block 1037 len 12288
19312 166/300 Read block 4951 len 24576
19313 167/300 Read block 775 len 20480
19314 168/300 Read block 707 len 8192
19315 169/300 Flush
19316 170/300 Write block 3754 len 20480 data: 3 3 3 3 3
19317 171/300 Read block 807 len 8192
19318 172/300 Read block 363 len 16384
19319 173/300 Read block 2895 len 16384
19320 174/300 Read block 1786 len 32768
19321 175/300 Write block 236 len 20480 data: 5 4 5 5 5
19322 176/300 Read block 2055 len 4096
19323 177/300 Read block 1075 len 24576
19324 178/300 Flush
19325 179/300 Write block 2410 len 24576 data: 4 4 3 5 5 4
19326 180/300 Read block 4697 len 36864
19327 181/300 Read block 437 len 4096
19328 182/300 Write block 3415 len 12288 data: 7 7 7
19329 183/300 Read block 556 len 8192
19330 184/300 Read block 902 len 24576
19331 185/300 Read block 3827 len 8192
19332 186/300 Read block 3935 len 4096
19333 187/300 Write block 2008 len 16384 data: 5 3 3 4
19334 188/300 Read block 3763 len 32768
19335 189/300 Read block 4659 len 40960
19336 190/300 Read block 1372 len 20480
19337 191/300 Read block 1977 len 24576
19338 192/300 Write block 3350 len 4096 data: 6
19339 193/300 Read block 1119 len 16384
19340 194/300 Write block 2320 len 28672 data: 2 2 2 2 2 4 4
19341 195/300 Read block 4183 len 4096
19342 196/300 Write block 3085 len 12288 data: 4 3 2
19343 197/300 Write block 180 len 28672 data: 4 5 4 4 6 6 5
19344 198/300 Write block 3599 len 8192 data: 3 4
19345 199/300 Read block 2002 len 12288
19346 200/300 Flush
19347 201/300 Write block 4771 len 4096 data: 3
19348 202/300 Read block 2180 len 36864
19349 203/300 Read block 411 len 24576
19350 204/300 Write block 2173 len 4096 data: 2
19351 205/300 Flush
19352 206/300 Read block 946 len 40960
19353 207/300 Write block 1927 len 4096 data: 4
19354 208/300 Read block 996 len 4096
19355 209/300 Write block 569 len 32768 data: 2 4 3 3 3 3 2 4
19356 210/300 Write block 516 len 36864 data: 3 4 3 3 4 4 4 4 6
19357 211/300 Flush
19358 212/300 Read block 4197 len 32768
19359 213/300 Read block 2746 len 12288
19360 214/300 Write block 342 len 4096 data: 2
19361 215/300 Read block 3267 len 16384
19362 216/300 Write block 1771 len 16384 data: 5 5 5 4
19363 217/300 Read block 510 len 36864
19364 218/300 Write block 754 len 8192 data: 3 5
19365 219/300 Read block 2127 len 40960
19366 220/300 Read block 4837 len 24576
19367 221/300 Write block 2508 len 40960 data: 5 6 5 4 4 3 3 4 4 5
19368 222/300 Read block 639 len 20480
19369 223/300 Write block 1259 len 20480 data: 5 5 5 5 5
19370 224/300 Read block 4831 len 12288
19371 225/300 Read block 501 len 4096
19372 226/300 Write block 4674 len 28672 data: 5 6 6 5 4 2 3
19373 227/300 Read block 6 len 4096
19374 228/300 Write block 81 len 28672 data: 4 4 3 3 3 3 3
19375 229/300 Read block 1685 len 40960
19376 230/300 Write block 2226 len 40960 data: 7 7 7 5 5 4 4 6 6 6
19377 231/300 Flush
19378 232/300 Read block 4544 len 12288
19379 233/300 Write block 4523 len 32768 data: 4 4 3 3 2 2 2 2
19380 234/300 Read block 3121 len 4096
19381 235/300 Read block 4573 len 12288
19382 236/300 Read block 4484 len 24576
19383 237/300 Flush
19384 238/300 Flush
19385 239/300 Write block 554 len 40960 data: 3 5 4 4 3 2 2 2 3 4
19386 240/300 Read block 476 len 40960
19387 241/300 Read block 1195 len 4096
19388 242/300 Flush
19389 243/300 Read block 3115 len 28672
19390 244/300 Write block 4614 len 12288 data: 5 4 5
19391 245/300 Write block 2443 len 36864 data: 4 4 4 4 4 3 3 4 4
19392 246/300 Write block 4693 len 16384 data: 3 3 3 3
19393 247/300 Write block 1603 len 12288 data: 5 5 5
19394 248/300 Read block 1127 len 20480
19395 249/300 Write block 4869 len 8192 data: 2 2
19396 250/300 Write block 2459 len 36864 data: 5 5 5 6 6 6 4 3 2
19397 251/300 Read block 2249 len 40960
19398 252/300 Read block 1313 len 12288
19399 253/300 Read block 1519 len 28672
19400 254/300 Read block 521 len 40960
19401 255/300 Write block 658 len 32768 data: 3 2 3 3 3 4 4 5
19402 256/300 Write block 3076 len 12288 data: 4 4 3
19403 257/300 Read block 4341 len 16384
19404 258/300 Write block 2880 len 32768 data: 5 4 4 5 6 5 7 5
19405 259/300 Write block 2100 len 20480 data: 5 4 3 2 3
19406 260/300 Write block 3372 len 16384 data: 3 3 3 3
19407 261/300 Read block 3739 len 28672
19408 262/300 Read block 4414 len 16384
19409 263/300 Write block 4519 len 20480 data: 4 4 5 4 5
19410 264/300 Read block 1046 len 20480
19411 265/300 Write block 3524 len 36864 data: 3 3 2 2 3 2 2 3 3
19412 266/300 Read block 3284 len 24576
19413 267/300 Read block 2945 len 20480
19414 268/300 Read block 3216 len 32768
19415 269/300 Write block 3795 len 24576 data: 3 5 5 6 5 5
19416 270/300 Read block 1433 len 12288
19417 271/300 Read block 4104 len 36864
19418 272/300 Write block 3337 len 20480 data: 5 6 5 5 6
19419 273/300 Read block 1949 len 28672
19420 274/300 Write block 1426 len 8192 data: 8 6
19421 275/300 Write block 4339 len 8192 data: 3 3
19422 276/300 Write block 3386 len 4096 data: 3
19423 277/300 Read block 3940 len 20480
19424 278/300 Read block 2313 len 24576
19425 279/300 Read block 4822 len 16384
19426 280/300 Read block 3184 len 36864
19427 281/300 Write block 4308 len 16384 data: 4 4 4 4
19428 282/300 Read block 2660 len 32768
19429 283/300 Write block 1762 len 8192 data: 9 6
19430 284/300 Read block 1055 len 16384
19431 285/300 Read block 3502 len 8192
19432 286/300 Read block 3483 len 40960
19433 287/300 Read block 4293 len 32768
19434 288/300 Read block 4822 len 12288
19435 289/300 Write block 3132 len 4096 data: 7
19436 290/300 Write block 2146 len 8192 data: 2 2
19437 291/300 Write block 2809 len 28672 data: 2 2 2 2 2 3 3
19438 292/300 Flush
19439 293/300 Read block 4872 len 24576
19440 294/300 Read block 3965 len 12288
19441 295/300 Flush
19442 296/300 Read block 1721 len 32768
19443 297/300 Read block 265 len 28672
19444 298/300 Flush
19445 299/300 Read block 3202 len 16384
19446 300/300 Read block 3794 len 40960
19447 301/300 Read block 304 len 28672
19448 Main task has work to do, go find it
19449 got dsc Start(1)
19450 start 1
19451 Feb 21 06:49:43.508 INFO request completed, latency_us: 235, response_code: 204, uri: /start/cid/1, method: POST, req_id: 895a4d32-d59a-47a6-905c-9a28f71f1359, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
19452 Replay: started 1, returned:Ok(())
19453 ----------------------------------------------------------------
19454 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19455 States: Active Offline Active
19456 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19457 5830 Acked Read 2 Done Sent Done false
19458 5831 Acked Write 2 Done Sent Done false
19459 5832 Acked Read 2 Done Sent Done false
19460 5833 Acked Read 8 Done Sent Done false
19461 5834 Acked Write 6 Done Sent Done false
19462 5835 Acked Read 8 Done Sent Done false
19463 5836 Acked Read 3 Done Sent Done false
19464 5837 Acked Write 3 Done Sent Done false
19465 5838 Acked Read 4 Done Sent Done false
19466 5839 Acked Read 10 Done Sent Done false
19467 5840 Acked Write 5 Done Sent Done false
19468 5841 Acked Read 9 Done Sent Done false
19469 5842 Acked Read 8 Done Sent Done false
19470 5843 Acked Flush 0 Done Sent Done false
19471 5844 Acked Write 8 Done Sent Done false
19472 5845 Acked Write 6 Done Sent Done false
19473 5846 Acked Read 9 Done Sent Done false
19474 5847 Acked Write 1 Done Sent Done false
19475 5848 Acked Write 9 Done Sent Done false
19476 5849 Acked Flush 0 Done Sent Done false
19477 5850 Acked Write 7 Done Sent Done false
19478 5851 Acked Read 9 Done Sent Done false
19479 5852 Acked Read 1 Done Sent Done false
19480 5853 Acked Read 8 Done Sent Done false
19481 5854 Acked Write 3 Done Sent Done false
19482 5855 Acked Write 6 Done Sent Done false
19483 5856 Acked Read 7 Done Sent Done false
19484 5857 Acked Read 4 Done Sent Done false
19485 5858 Acked Write 1 Done Sent Done false
19486 5859 Acked Write 7 Done Sent Done false
19487 5860 Acked Write 10 Done Sent Done false
19488 5861 Acked Write 5 Done Sent Done false
19489 5862 Acked Read 6 Done Sent Done false
19490 5863 Acked Write 6 Done Sent Done false
19491 5864 Acked Read 9 Done Sent Done false
19492 5865 Acked Read 3 Done Sent Done false
19493 5866 Acked Write 4 Done Sent Done false
19494 5867 Acked Read 3 Done Sent Done false
19495 5868 Acked Read 5 Done Sent Done false
19496 5869 Acked Write 6 Done Sent Done false
19497 5870 Acked Write 2 Done Sent Done false
19498 5871 Acked Write 6 Done Sent Done false
19499 5872 Acked Write 4 Done Sent Done false
19500 5873 Acked Write 2 Done Sent Done false
19501 5874 Acked Write 6 Done Sent Done false
19502 5875 Acked Write 10 Done Sent Done false
19503 5876 Acked Write 7 Done Sent Done false
19504 5877 Acked Read 3 Done Sent Done false
19505 5878 Acked Flush 0 Done Sent Done false
19506 5879 Acked Read 9 Done Sent Done false
19507 5880 Acked Write 5 Done Sent Done false
19508 5881 Acked Read 5 Done Sent Done false
19509 5882 Acked Write 7 Done Sent Done false
19510 5883 Acked Read 4 Done Sent Done false
19511 5884 Acked Write 6 Done Sent Done false
19512 5885 Acked Read 10 Done Sent Done false
19513 5886 Acked Read 1 Done Sent Done false
19514 5887 Acked Read 7 Done Sent Done false
19515 5888 Acked Flush 0 Done Sent Done false
19516 5889 Acked Write 9 Done Sent Done false
19517 5890 Acked Read 9 Done Sent Done false
19518 5891 Acked Read 6 Done Sent Done false
19519 5892 Acked Read 4 Done Sent Done false
19520 5893 Acked Write 9 Done Sent Done false
19521 5894 Acked Write 1 Done Sent Done false
19522 5895 Acked Write 6 Done Sent Done false
19523 5896 Acked Flush 0 Done Sent Done false
19524 5897 Acked Read 8 Done Sent Done false
19525 5898 Acked Write 7 Done Sent Done false
19526 5899 Acked Read 3 Done Sent Done false
19527 5900 Acked Write 9 Done Sent Done false
19528 5901 Acked Write 2 Done Sent Done false
19529 5902 Acked Read 5 Done Sent Done false
19530 5903 Acked Read 1 Done Sent Done false
19531 5904 Acked Read 10 Done Sent Done false
19532 5905 Acked Read 1 Done Sent Done false
19533 5906 Acked Write 8 Done Sent Done false
19534 5907 Acked Flush 0 Done Sent Done false
19535 5908 Acked Read 5 Done Sent Done false
19536 5909 Acked Write 5 Done Sent Done false
19537 5910 Acked Read 8 Done Sent Done false
19538 5911 Acked Write 3 Done Sent Done false
19539 5912 Acked Flush 0 Done Sent Done false
19540 5913 Acked Read 5 Done Sent Done false
19541 5914 Acked Read 3 Done Sent Done false
19542 5915 Acked Read 1 Done Sent Done false
19543 5916 Acked Read 5 Done Sent Done false
19544 5917 Acked Write 4 Done Sent Done false
19545 5918 Acked Read 2 Done Sent Done false
19546 5919 Acked Write 5 Done Sent Done false
19547 5920 Acked Read 1 Done Sent Done false
19548 5921 Acked Flush 0 Done Sent Done false
19549 5922 Acked Write 10 Done Sent Done false
19550 5923 Acked Flush 0 Done Sent Done false
19551 5924 Acked Write 7 Done Sent Done false
19552 5925 Acked Read 1 Done Sent Done false
19553 5926 Acked Read 5 Done Sent Done false
19554 5927 Acked Read 3 Done Sent Done false
19555 5928 Acked Read 3 Done Sent Done false
19556 5929 Acked Read 5 Done Sent Done false
19557 5930 Acked Write 9 Done Sent Done false
19558 5931 Acked Write 8 Done Sent Done false
19559 5932 Acked Write 1 Done Sent Done false
19560 5933 Acked Read 8 Done Sent Done false
19561 5934 Acked Read 10 Done Sent Done false
19562 5935 Acked Read 10 Done Sent Done false
19563 5936 Acked Read 8 Done Sent Done false
19564 5937 Acked Read 5 Done Sent Done false
19565 5938 Acked Flush 0 Done Sent Done false
19566 5939 Acked Read 2 Done Sent Done false
19567 5940 Acked Flush 0 Done Sent Done false
19568 5941 Acked Write 9 Done Sent Done false
19569 5942 Acked Write 5 Done Sent Done false
19570 5943 Acked Read 4 Done Sent Done false
19571 5944 Acked Read 4 Done Sent Done false
19572 5945 Acked Read 4 Done Sent Done false
19573 5946 Acked Flush 0 Done Sent Done false
19574 5947 Acked Read 8 Done Sent Done false
19575 5948 Acked Write 7 Done Sent Done false
19576 5949 Acked Read 5 Done Sent Done false
19577 5950 Acked Write 7 Done Sent Done false
19578 5951 Acked Read 4 Done Sent Done false
19579 5952 Acked Read 8 Done Sent Done false
19580 5953 Acked Read 8 Done Sent Done false
19581 5954 Acked Write 6 Done Sent Done false
19582 5955 Acked Read 1 Done Sent Done false
19583 5956 Acked Read 9 Done Sent Done false
19584 5957 Acked Write 3 Done Sent Done false
19585 5958 Acked Read 2 Done Sent Done false
19586 5959 Acked Flush 0 Done Sent Done false
19587 5960 Acked Write 6 Done Sent Done false
19588 5961 Acked Flush 0 Done Sent Done false
19589 5962 Acked Read 2 Done Sent Done false
19590 5963 Acked Flush 0 Done Sent Done false
19591 5964 Acked Read 2 Done Sent Done false
19592 5965 Acked Flush 0 Done Sent Done false
19593 5966 Acked Write 8 Done Sent Done false
19594 5967 Acked Write 3 Done Sent Done false
19595 5968 Acked Write 6 Done Sent Done false
19596 5969 Acked Write 3 Done Sent Done false
19597 5970 Acked Write 6 Done Sent Done false
19598 5971 Acked Write 1 Done Sent Done false
19599 5972 Acked Read 6 Done Sent Done false
19600 5973 Acked Write 8 Done Sent Done false
19601 5974 Acked Read 6 Done Sent Done false
19602 5975 Acked Write 8 Done Sent Done false
19603 5976 Acked Read 3 Done Sent Done false
19604 5977 Acked Read 2 Done Sent Done false
19605 5978 Acked Read 5 Done Sent Done false
19606 5979 Acked Write 3 Done Sent Done false
19607 5980 Acked Flush 0 Done Sent Done false
19608 5981 Acked Flush 0 Done Sent Done false
19609 5982 Acked Read 7 Done Sent Done false
19610 5983 Acked Read 5 Done Sent Done false
19611 5984 Acked Read 5 Done Sent Done false
19612 5985 Acked Read 5 Done Sent Done false
19613 5986 Acked Write 10 Done Sent Done false
19614 5987 Acked Flush 0 Done Sent Done false
19615 5988 Acked Read 4 Done Sent Done false
19616 5989 Acked Write 9 Done Sent Done false
19617 5990 Acked Read 9 Done Sent Done false
19618 5991 Acked Write 9 Done Sent Done false
19619 5992 Acked Write 5 Done Sent Done false
19620 5993 Acked Read 6 Done Sent Done false
19621 5994 Acked Read 3 Done Sent Done false
19622 5995 Acked Read 6 Done Sent Done false
19623 5996 Acked Read 5 Done Sent Done false
19624 5997 Acked Read 2 Done Sent Done false
19625 5998 Acked Flush 0 Done Sent Done false
19626 5999 Acked Write 5 Done Sent Done false
19627 6000 Acked Read 2 Done Sent Done false
19628 6001 Acked Read 4 Done Sent Done false
19629 6002 Acked Read 4 Done Sent Done false
19630 6003 Acked Read 8 Done Sent Done false
19631 6004 Acked Write 5 Done Sent Done false
19632 6005 Acked Read 1 Done Sent Done false
19633 6006 Acked Read 6 Done Sent Done false
19634 6007 Acked Flush 0 Done Sent Done false
19635 6008 Acked Write 6 Done Sent Done false
19636 6009 Acked Read 9 Done Sent Done false
19637 6010 Acked Read 1 Done Sent Done false
19638 6011 Acked Write 3 Done Sent Done false
19639 6012 Acked Read 2 Done Sent Done false
19640 6013 Acked Read 6 Done Sent Done false
19641 6014 Acked Read 2 Done Sent Done false
19642 6015 Acked Read 1 Done Sent Done false
19643 6016 Acked Write 4 Done Sent Done false
19644 6017 Acked Read 8 Done Sent Done false
19645 6018 Acked Read 10 Done Sent Done false
19646 6019 Acked Read 5 Done Sent Done false
19647 6020 Acked Read 6 Done Sent Done false
19648 6021 Acked Write 1 Done Sent Done false
19649 6022 Acked Read 4 Done Sent Done false
19650 6023 Acked Write 7 Done Sent Done false
19651 6024 Acked Read 1 Done Sent Done false
19652 6025 Acked Write 3 Done Sent Done false
19653 6026 Acked Write 7 Done Sent Done false
19654 6027 Acked Write 2 Done Sent Done false
19655 6028 Acked Read 3 Done Sent Done false
19656 6029 Acked Flush 0 Done Sent Done false
19657 6030 Acked Write 1 Done Sent Done false
19658 6031 Acked Read 9 Done Sent Done false
19659 6032 Acked Read 6 Done Sent Done false
19660 6033 Acked Write 1 Done Sent Done false
19661 6034 Acked Flush 0 Done Sent Done false
19662 6035 Acked Read 10 Done Sent Done false
19663 6036 Acked Write 1 Done Sent Done false
19664 6037 Acked Read 1 Done Sent Done false
19665 6038 Acked Write 8 Done Sent Done false
19666 6039 Acked Write 9 Done Sent Done false
19667 6040 Acked Flush 0 Done Sent Done false
19668 6041 Acked Read 8 Done Sent Done false
19669 6042 Acked Read 3 Done Sent Done false
19670 6043 Acked Write 1 Done Sent Done false
19671 6044 Acked Read 4 Done Sent Done false
19672 6045 Acked Write 4 Done Sent Done false
19673 6046 Acked Read 9 Done Sent Done false
19674 6047 Acked Write 2 Done Sent Done false
19675 6048 Acked Read 10 Done Sent Done false
19676 6049 Acked Read 6 Done Sent Done false
19677 6050 Acked Write 10 Done Sent Done false
19678 6051 Acked Read 5 Done Sent Done false
19679 6052 Acked Write 5 Done Sent Done false
19680 6053 Acked Read 3 Done Sent Done false
19681 6054 Acked Read 1 Done Sent Done false
19682 6055 Acked Write 7 Done Sent Done false
19683 6056 Acked Read 1 Done Sent Done false
19684 6057 Acked Write 7 Done Sent Done false
19685 6058 Acked Read 10 Done Sent Done false
19686 6059 Acked Write 10 Done Sent Done false
19687 6060 Acked Flush 0 Done Sent Done false
19688 6061 Acked Read 3 Done Sent Done false
19689 6062 Acked Write 8 Done Sent Done false
19690 6063 Acked Read 1 Done Sent Done false
19691 6064 Acked Read 3 Done Sent Done false
19692 6065 Acked Read 6 Done Sent Done false
19693 6066 Acked Flush 0 Done Sent Done false
19694 6067 Acked Flush 0 Done Sent Done false
19695 6068 Acked Write 10 Done Sent Done false
19696 6069 Acked Read 10 Done Sent Done false
19697 6070 Acked Read 1 Done Sent Done false
19698 6071 Acked Flush 0 Done Sent Done false
19699 6072 Acked Read 7 Done Sent Done false
19700 6073 Acked Write 3 Done Sent Done false
19701 6074 Acked Write 9 Done Sent Done false
19702 6075 Acked Write 4 Done Sent Done false
19703 6076 Acked Write 3 Done Sent Done false
19704 6077 Acked Read 5 Done Sent Done false
19705 6078 Acked Write 2 Done Sent Done false
19706 6079 Acked Write 9 Done Sent Done false
19707 6080 Acked Read 10 Done Sent Done false
19708 6081 Acked Read 3 Done Sent Done false
19709 6082 Acked Read 7 Done Sent Done false
19710 6083 Acked Read 10 Done Sent Done false
19711 6084 Acked Write 8 Done Sent Done false
19712 6085 Acked Write 3 Done Sent Done false
19713 6086 Acked Read 4 Done Sent Done false
19714 6087 Acked Write 8 Done Sent Done false
19715 6088 Acked Write 5 Done Sent Done false
19716 6089 Acked Write 4 Done Sent Done false
19717 6090 Acked Read 7 Done Sent Done false
19718 6091 Acked Read 4 Done Sent Done false
19719 6092 Acked Write 5 Done Sent Done false
19720 6093 Acked Read 5 Done Sent Done false
19721 6094 Acked Write 9 Done Sent Done false
19722 6095 Acked Read 6 Done Sent Done false
19723 6096 Acked Read 5 Done Sent Done false
19724 6097 Acked Read 8 Done Sent Done false
19725 6098 Acked Write 6 Done Sent Done false
19726 6099 Acked Read 3 Done Sent Done false
19727 6100 Acked Read 9 Done Sent Done false
19728 6101 Acked Write 5 Done Sent Done false
19729 6102 Acked Read 7 Done Sent Done false
19730 6103 Acked Write 2 Done Sent Done false
19731 6104 Acked Write 2 Done Sent Done false
19732 6105 Acked Write 1 Done Sent Done false
19733 6106 Acked Read 5 Done Sent Done false
19734 6107 Acked Read 6 Done Sent Done false
19735 6108 Acked Read 4 Done Sent Done false
19736 6109 Acked Read 9 Done Sent Done false
19737 6110 Acked Write 4 Done Sent Done false
19738 6111 Acked Read 8 Done Sent Done false
19739 6112 Acked Write 2 Done Sent Done false
19740 6113 Acked Read 4 Done Sent Done false
19741 6114 Acked Read 2 Done Sent Done false
19742 6115 Acked Read 10 Done Sent Done false
19743 6116 Acked Read 8 Done Sent Done false
19744 6117 Acked Read 3 Done Sent Done false
19745 6118 Acked Write 1 Done Sent Done false
19746 6119 Acked Write 2 Done Sent Done false
19747 6120 Acked Write 7 Done Sent Done false
19748 6121 Acked Flush 0 Done Sent Done false
19749 6122 Acked Read 6 Done Sent Done false
19750 6123 Acked Read 3 Done Sent Done false
19751 6124 Acked Flush 0 Done Sent Done false
19752 6125 Acked Read 8 Done Sent Done false
19753 6126 Acked Read 7 Done Sent Done false
19754 6127 Acked Flush 0 Done Sent Done false
19755 6128 Acked Read 4 Done Sent Done false
19756 6129 Acked Read 10 Done Sent Done false
19757 6130 Acked Read 7 Done Sent Done false
19758 STATES DS:0 DS:1 DS:2 TOTAL
19759 Sent 0 301 0 301
19760 Done 301 0 301 602
19761 Skipped 0 0 0 0
19762 Error 0 0 0 0
19763 Last Flush: Some(JobId(6127)) Some(JobId(5829)) Some(JobId(6127))
19764 Downstairs last five completed: 5829 5828 5827 5826 5825
19765 Upstairs last five completed: 6130 6129 6128 6127 6126
19766 CLIENT: Up:0 ds:301 act:2
197672025-02-21T06:49:44.064ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
197682025-02-21T06:49:45.067ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
19769 [8820] got start action
19770 [8820] I am going to restart
19771 Starting downstairs at port 8820
19772 Make output file at "/tmp/dsc/downstairs-8820.txt"
19773 [8820][1] reports Starting
19774 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1223)
19775 [8820][1] reports Running
197762025-02-21T06:49:46.068ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
197772025-02-21T06:49:47.509ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
19778 ----------------------------------------------------------------
19779 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19780 States: Active Offline Active
19781 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19782 5830 Acked Read 2 Done Sent Done false
19783 5831 Acked Write 2 Done Sent Done false
19784 5832 Acked Read 2 Done Sent Done false
19785 5833 Acked Read 8 Done Sent Done false
19786 5834 Acked Write 6 Done Sent Done false
19787 5835 Acked Read 8 Done Sent Done false
19788 5836 Acked Read 3 Done Sent Done false
19789 5837 Acked Write 3 Done Sent Done false
19790 5838 Acked Read 4 Done Sent Done false
19791 5839 Acked Read 10 Done Sent Done false
19792 5840 Acked Write 5 Done Sent Done false
19793 5841 Acked Read 9 Done Sent Done false
19794 5842 Acked Read 8 Done Sent Done false
19795 5843 Acked Flush 0 Done Sent Done false
19796 5844 Acked Write 8 Done Sent Done false
19797 5845 Acked Write 6 Done Sent Done false
19798 5846 Acked Read 9 Done Sent Done false
19799 5847 Acked Write 1 Done Sent Done false
19800 5848 Acked Write 9 Done Sent Done false
19801 5849 Acked Flush 0 Done Sent Done false
19802 5850 Acked Write 7 Done Sent Done false
19803 5851 Acked Read 9 Done Sent Done false
19804 5852 Acked Read 1 Done Sent Done false
19805 5853 Acked Read 8 Done Sent Done false
19806 5854 Acked Write 3 Done Sent Done false
19807 5855 Acked Write 6 Done Sent Done false
19808 5856 Acked Read 7 Done Sent Done false
19809 5857 Acked Read 4 Done Sent Done false
19810 5858 Acked Write 1 Done Sent Done false
19811 5859 Acked Write 7 Done Sent Done false
19812 5860 Acked Write 10 Done Sent Done false
19813 5861 Acked Write 5 Done Sent Done false
19814 5862 Acked Read 6 Done Sent Done false
19815 5863 Acked Write 6 Done Sent Done false
19816 5864 Acked Read 9 Done Sent Done false
19817 5865 Acked Read 3 Done Sent Done false
19818 5866 Acked Write 4 Done Sent Done false
19819 5867 Acked Read 3 Done Sent Done false
19820 5868 Acked Read 5 Done Sent Done false
19821 5869 Acked Write 6 Done Sent Done false
19822 5870 Acked Write 2 Done Sent Done false
19823 5871 Acked Write 6 Done Sent Done false
19824 5872 Acked Write 4 Done Sent Done false
19825 5873 Acked Write 2 Done Sent Done false
19826 5874 Acked Write 6 Done Sent Done false
19827 5875 Acked Write 10 Done Sent Done false
19828 5876 Acked Write 7 Done Sent Done false
19829 5877 Acked Read 3 Done Sent Done false
19830 5878 Acked Flush 0 Done Sent Done false
19831 5879 Acked Read 9 Done Sent Done false
19832 5880 Acked Write 5 Done Sent Done false
19833 5881 Acked Read 5 Done Sent Done false
19834 5882 Acked Write 7 Done Sent Done false
19835 5883 Acked Read 4 Done Sent Done false
19836 5884 Acked Write 6 Done Sent Done false
19837 5885 Acked Read 10 Done Sent Done false
19838 5886 Acked Read 1 Done Sent Done false
19839 5887 Acked Read 7 Done Sent Done false
19840 5888 Acked Flush 0 Done Sent Done false
19841 5889 Acked Write 9 Done Sent Done false
19842 5890 Acked Read 9 Done Sent Done false
19843 5891 Acked Read 6 Done Sent Done false
19844 5892 Acked Read 4 Done Sent Done false
19845 5893 Acked Write 9 Done Sent Done false
19846 5894 Acked Write 1 Done Sent Done false
19847 5895 Acked Write 6 Done Sent Done false
19848 5896 Acked Flush 0 Done Sent Done false
19849 5897 Acked Read 8 Done Sent Done false
19850 5898 Acked Write 7 Done Sent Done false
19851 5899 Acked Read 3 Done Sent Done false
19852 5900 Acked Write 9 Done Sent Done false
19853 5901 Acked Write 2 Done Sent Done false
19854 5902 Acked Read 5 Done Sent Done false
19855 5903 Acked Read 1 Done Sent Done false
19856 5904 Acked Read 10 Done Sent Done false
19857 5905 Acked Read 1 Done Sent Done false
19858 5906 Acked Write 8 Done Sent Done false
19859 5907 Acked Flush 0 Done Sent Done false
19860 5908 Acked Read 5 Done Sent Done false
19861 5909 Acked Write 5 Done Sent Done false
19862 5910 Acked Read 8 Done Sent Done false
19863 5911 Acked Write 3 Done Sent Done false
19864 5912 Acked Flush 0 Done Sent Done false
19865 5913 Acked Read 5 Done Sent Done false
19866 5914 Acked Read 3 Done Sent Done false
19867 5915 Acked Read 1 Done Sent Done false
19868 5916 Acked Read 5 Done Sent Done false
19869 5917 Acked Write 4 Done Sent Done false
19870 5918 Acked Read 2 Done Sent Done false
19871 5919 Acked Write 5 Done Sent Done false
19872 5920 Acked Read 1 Done Sent Done false
19873 5921 Acked Flush 0 Done Sent Done false
19874 5922 Acked Write 10 Done Sent Done false
19875 5923 Acked Flush 0 Done Sent Done false
19876 5924 Acked Write 7 Done Sent Done false
19877 5925 Acked Read 1 Done Sent Done false
19878 5926 Acked Read 5 Done Sent Done false
19879 5927 Acked Read 3 Done Sent Done false
19880 5928 Acked Read 3 Done Sent Done false
19881 5929 Acked Read 5 Done Sent Done false
19882 5930 Acked Write 9 Done Sent Done false
19883 5931 Acked Write 8 Done Sent Done false
19884 5932 Acked Write 1 Done Sent Done false
19885 5933 Acked Read 8 Done Sent Done false
19886 5934 Acked Read 10 Done Sent Done false
19887 5935 Acked Read 10 Done Sent Done false
19888 5936 Acked Read 8 Done Sent Done false
19889 5937 Acked Read 5 Done Sent Done false
19890 5938 Acked Flush 0 Done Sent Done false
19891 5939 Acked Read 2 Done Sent Done false
19892 5940 Acked Flush 0 Done Sent Done false
19893 5941 Acked Write 9 Done Sent Done false
19894 5942 Acked Write 5 Done Sent Done false
19895 5943 Acked Read 4 Done Sent Done false
19896 5944 Acked Read 4 Done Sent Done false
19897 5945 Acked Read 4 Done Sent Done false
19898 5946 Acked Flush 0 Done Sent Done false
19899 5947 Acked Read 8 Done Sent Done false
19900 5948 Acked Write 7 Done Sent Done false
19901 5949 Acked Read 5 Done Sent Done false
19902 5950 Acked Write 7 Done Sent Done false
19903 5951 Acked Read 4 Done Sent Done false
19904 5952 Acked Read 8 Done Sent Done false
19905 5953 Acked Read 8 Done Sent Done false
19906 5954 Acked Write 6 Done Sent Done false
19907 5955 Acked Read 1 Done Sent Done false
19908 5956 Acked Read 9 Done Sent Done false
19909 5957 Acked Write 3 Done Sent Done false
19910 5958 Acked Read 2 Done Sent Done false
19911 5959 Acked Flush 0 Done Sent Done false
19912 5960 Acked Write 6 Done Sent Done false
19913 5961 Acked Flush 0 Done Sent Done false
19914 5962 Acked Read 2 Done Sent Done false
19915 5963 Acked Flush 0 Done Sent Done false
19916 5964 Acked Read 2 Done Sent Done false
19917 5965 Acked Flush 0 Done Sent Done false
19918 5966 Acked Write 8 Done Sent Done false
19919 5967 Acked Write 3 Done Sent Done false
19920 5968 Acked Write 6 Done Sent Done false
19921 5969 Acked Write 3 Done Sent Done false
19922 5970 Acked Write 6 Done Sent Done false
19923 5971 Acked Write 1 Done Sent Done false
19924 5972 Acked Read 6 Done Sent Done false
19925 5973 Acked Write 8 Done Sent Done false
19926 5974 Acked Read 6 Done Sent Done false
19927 5975 Acked Write 8 Done Sent Done false
19928 5976 Acked Read 3 Done Sent Done false
19929 5977 Acked Read 2 Done Sent Done false
19930 5978 Acked Read 5 Done Sent Done false
19931 5979 Acked Write 3 Done Sent Done false
19932 5980 Acked Flush 0 Done Sent Done false
19933 5981 Acked Flush 0 Done Sent Done false
19934 5982 Acked Read 7 Done Sent Done false
19935 5983 Acked Read 5 Done Sent Done false
19936 5984 Acked Read 5 Done Sent Done false
19937 5985 Acked Read 5 Done Sent Done false
19938 5986 Acked Write 10 Done Sent Done false
19939 5987 Acked Flush 0 Done Sent Done false
19940 5988 Acked Read 4 Done Sent Done false
19941 5989 Acked Write 9 Done Sent Done false
19942 5990 Acked Read 9 Done Sent Done false
19943 5991 Acked Write 9 Done Sent Done false
19944 5992 Acked Write 5 Done Sent Done false
19945 5993 Acked Read 6 Done Sent Done false
19946 5994 Acked Read 3 Done Sent Done false
19947 5995 Acked Read 6 Done Sent Done false
19948 5996 Acked Read 5 Done Sent Done false
19949 5997 Acked Read 2 Done Sent Done false
19950 5998 Acked Flush 0 Done Sent Done false
19951 5999 Acked Write 5 Done Sent Done false
19952 6000 Acked Read 2 Done Sent Done false
19953 6001 Acked Read 4 Done Sent Done false
19954 6002 Acked Read 4 Done Sent Done false
19955 6003 Acked Read 8 Done Sent Done false
19956 6004 Acked Write 5 Done Sent Done false
19957 6005 Acked Read 1 Done Sent Done false
19958 6006 Acked Read 6 Done Sent Done false
19959 6007 Acked Flush 0 Done Sent Done false
19960 6008 Acked Write 6 Done Sent Done false
19961 6009 Acked Read 9 Done Sent Done false
19962 6010 Acked Read 1 Done Sent Done false
19963 6011 Acked Write 3 Done Sent Done false
19964 6012 Acked Read 2 Done Sent Done false
19965 6013 Acked Read 6 Done Sent Done false
19966 6014 Acked Read 2 Done Sent Done false
19967 6015 Acked Read 1 Done Sent Done false
19968 6016 Acked Write 4 Done Sent Done false
19969 6017 Acked Read 8 Done Sent Done false
19970 6018 Acked Read 10 Done Sent Done false
19971 6019 Acked Read 5 Done Sent Done false
19972 6020 Acked Read 6 Done Sent Done false
19973 6021 Acked Write 1 Done Sent Done false
19974 6022 Acked Read 4 Done Sent Done false
19975 6023 Acked Write 7 Done Sent Done false
19976 6024 Acked Read 1 Done Sent Done false
19977 6025 Acked Write 3 Done Sent Done false
19978 6026 Acked Write 7 Done Sent Done false
19979 6027 Acked Write 2 Done Sent Done false
19980 6028 Acked Read 3 Done Sent Done false
19981 6029 Acked Flush 0 Done Sent Done false
19982 6030 Acked Write 1 Done Sent Done false
19983 6031 Acked Read 9 Done Sent Done false
19984 6032 Acked Read 6 Done Sent Done false
19985 6033 Acked Write 1 Done Sent Done false
19986 6034 Acked Flush 0 Done Sent Done false
19987 6035 Acked Read 10 Done Sent Done false
19988 6036 Acked Write 1 Done Sent Done false
19989 6037 Acked Read 1 Done Sent Done false
19990 6038 Acked Write 8 Done Sent Done false
19991 6039 Acked Write 9 Done Sent Done false
19992 6040 Acked Flush 0 Done Sent Done false
19993 6041 Acked Read 8 Done Sent Done false
19994 6042 Acked Read 3 Done Sent Done false
19995 6043 Acked Write 1 Done Sent Done false
19996 6044 Acked Read 4 Done Sent Done false
19997 6045 Acked Write 4 Done Sent Done false
19998 6046 Acked Read 9 Done Sent Done false
19999 6047 Acked Write 2 Done Sent Done false
20000 6048 Acked Read 10 Done Sent Done false
20001 6049 Acked Read 6 Done Sent Done false
20002 6050 Acked Write 10 Done Sent Done false
20003 6051 Acked Read 5 Done Sent Done false
20004 6052 Acked Write 5 Done Sent Done false
20005 6053 Acked Read 3 Done Sent Done false
20006 6054 Acked Read 1 Done Sent Done false
20007 6055 Acked Write 7 Done Sent Done false
20008 6056 Acked Read 1 Done Sent Done false
20009 6057 Acked Write 7 Done Sent Done false
20010 6058 Acked Read 10 Done Sent Done false
20011 6059 Acked Write 10 Done Sent Done false
20012 6060 Acked Flush 0 Done Sent Done false
20013 6061 Acked Read 3 Done Sent Done false
20014 6062 Acked Write 8 Done Sent Done false
20015 6063 Acked Read 1 Done Sent Done false
20016 6064 Acked Read 3 Done Sent Done false
20017 6065 Acked Read 6 Done Sent Done false
20018 6066 Acked Flush 0 Done Sent Done false
20019 6067 Acked Flush 0 Done Sent Done false
20020 6068 Acked Write 10 Done Sent Done false
20021 6069 Acked Read 10 Done Sent Done false
20022 6070 Acked Read 1 Done Sent Done false
20023 6071 Acked Flush 0 Done Sent Done false
20024 6072 Acked Read 7 Done Sent Done false
20025 6073 Acked Write 3 Done Sent Done false
20026 6074 Acked Write 9 Done Sent Done false
20027 6075 Acked Write 4 Done Sent Done false
20028 6076 Acked Write 3 Done Sent Done false
20029 6077 Acked Read 5 Done Sent Done false
20030 6078 Acked Write 2 Done Sent Done false
20031 6079 Acked Write 9 Done Sent Done false
20032 6080 Acked Read 10 Done Sent Done false
20033 6081 Acked Read 3 Done Sent Done false
20034 6082 Acked Read 7 Done Sent Done false
20035 6083 Acked Read 10 Done Sent Done false
20036 6084 Acked Write 8 Done Sent Done false
20037 6085 Acked Write 3 Done Sent Done false
20038 6086 Acked Read 4 Done Sent Done false
20039 6087 Acked Write 8 Done Sent Done false
20040 6088 Acked Write 5 Done Sent Done false
20041 6089 Acked Write 4 Done Sent Done false
20042 6090 Acked Read 7 Done Sent Done false
20043 6091 Acked Read 4 Done Sent Done false
20044 6092 Acked Write 5 Done Sent Done false
20045 6093 Acked Read 5 Done Sent Done false
20046 6094 Acked Write 9 Done Sent Done false
20047 6095 Acked Read 6 Done Sent Done false
20048 6096 Acked Read 5 Done Sent Done false
20049 6097 Acked Read 8 Done Sent Done false
20050 6098 Acked Write 6 Done Sent Done false
20051 6099 Acked Read 3 Done Sent Done false
20052 6100 Acked Read 9 Done Sent Done false
20053 6101 Acked Write 5 Done Sent Done false
20054 6102 Acked Read 7 Done Sent Done false
20055 6103 Acked Write 2 Done Sent Done false
20056 6104 Acked Write 2 Done Sent Done false
20057 6105 Acked Write 1 Done Sent Done false
20058 6106 Acked Read 5 Done Sent Done false
20059 6107 Acked Read 6 Done Sent Done false
20060 6108 Acked Read 4 Done Sent Done false
20061 6109 Acked Read 9 Done Sent Done false
20062 6110 Acked Write 4 Done Sent Done false
20063 6111 Acked Read 8 Done Sent Done false
20064 6112 Acked Write 2 Done Sent Done false
20065 6113 Acked Read 4 Done Sent Done false
20066 6114 Acked Read 2 Done Sent Done false
20067 6115 Acked Read 10 Done Sent Done false
20068 6116 Acked Read 8 Done Sent Done false
20069 6117 Acked Read 3 Done Sent Done false
20070 6118 Acked Write 1 Done Sent Done false
20071 6119 Acked Write 2 Done Sent Done false
20072 6120 Acked Write 7 Done Sent Done false
20073 6121 Acked Flush 0 Done Sent Done false
20074 6122 Acked Read 6 Done Sent Done false
20075 6123 Acked Read 3 Done Sent Done false
20076 6124 Acked Flush 0 Done Sent Done false
20077 6125 Acked Read 8 Done Sent Done false
20078 6126 Acked Read 7 Done Sent Done false
20079 6127 Acked Flush 0 Done Sent Done false
20080 6128 Acked Read 4 Done Sent Done false
20081 6129 Acked Read 10 Done Sent Done false
20082 6130 Acked Read 7 Done Sent Done false
20083 STATES DS:0 DS:1 DS:2 TOTAL
20084 Sent 0 301 0 301
20085 Done 301 0 301 602
20086 Skipped 0 0 0 0
20087 Error 0 0 0 0
20088 Last Flush: Some(JobId(6127)) Some(JobId(5829)) Some(JobId(6127))
20089 Downstairs last five completed: 5829 5828 5827 5826 5825
20090 Upstairs last five completed: 6130 6129 6128 6127 6126
20091 CLIENT: Up:0 ds:301 act:2
200922025-02-21T06:49:48.274ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
200932025-02-21T06:49:49.072ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
200942025-02-21T06:49:49.260ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
200952025-02-21T06:49:49.260ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61553) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
200962025-02-21T06:49:49.261ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
200972025-02-21T06:49:49.262ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
200982025-02-21T06:49:49.262ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
200992025-02-21T06:49:49.262ZINFOcrucible: send last flush ID to this DS: Some(JobId(5829)) = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
201002025-02-21T06:49:49.262ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
201012025-02-21T06:49:49.262ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5829)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
201022025-02-21T06:49:49.262ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5829)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
20103 ----------------------------------------------------------------
20104 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20105 Downstairs last five completed: 6131 6130 6129 6128 6127
20106 Upstairs last five completed: 6131 6130 6129 6128 6127
20107 CLIENT: Up:0 ds:0 act:3
20108 Replay: All jobs finished
20109 Main task has work to do, go find it
20110 got dsc Stop(2)
20111 stop 2
20112 Feb 21 06:49:51.514 INFO request completed, latency_us: 412, response_code: 204[8830] Got stop action so:false kr:false
20113 , uri: /stop/cid/2, method: POST, req_id: fd4f37af-b813-420c-877e-a2d3481e579c, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
201142025-02-21T06:49:51.514ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
201152025-02-21T06:49:51.514ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
201162025-02-21T06:49:51.514ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
201172025-02-21T06:49:51.515ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
20118 Feb 21 06:49:51.515 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/2, method: GET, req_id: f83ba107-a7ef-4c0e-9920-2d52b95db0bc, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
20119 [8830] Exited with: ExitStatus(unix_wait_status(9))
20120 [8830][2] reports Exit
20121 Feb 21 06:49:55.516 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2c4bf67b-2282-402b-a29e-c37b939cfa54, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
20122 001/300 Write block 4107 len 12288 data: 5 6 6
20123 002/300 Read block 3852 len 32768
20124 003/300 Flush
20125 004/300 Write block 1074 len 24576 data: 6 4 3 3 3 3
20126 005/300 Read block 2727 len 16384
20127 006/300 Read block 2777 len 4096
20128 007/300 Write block 1946 len 28672 data: 5 5 4 3 3 4 5
20129 008/300 Write block 2289 len 12288 data: 4 4 4
20130 009/300 Read block 384 len 8192
20131 010/300 Write block 136 len 8192 data: 4 5
20132 011/300 Write block 639 len 16384 data: 5 4 4 3
20133 012/300 Read block 3360 len 20480
20134 013/300 Write block 2177 len 4096 data: 6
20135 014/300 Write block 3607 len 24576 data: 7 6 5 6 5 3
20136 015/300 Read block 1662 len 16384
20137 016/300 Read block 758 len 36864
20138 017/300 Read block 255 len 36864
20139 018/300 Read block 4072 len 12288
20140 019/300 Write block 2931 len 32768 data: 6 6 4 3 3 3 4 4
20141 020/300 Read block 4580 len 4096
20142 021/300 Write block 4534 len 40960 data: 2 3 4 3 3 2 2 2 3 4
20143 022/300 Write block 2706 len 4096 data: 5
20144 023/300 Read block 4972 len 32768
20145 024/300 Read block 3086 len 36864
20146 025/300 Read block 3372 len 8192
20147 026/300 Read block 691 len 4096
20148 027/300 Read block 267 len 4096
20149 028/300 Read block 2244 len 32768
20150 029/300 Write block 2875 len 20480 data: 5 6 6 6 5
20151 030/300 Read block 1255 len 36864
20152 031/300 Read block 1379 len 12288
20153 032/300 Read block 1086 len 4096
20154 033/300 Read block 4832 len 8192
20155 034/300 Flush
20156 035/300 Read block 1012 len 12288
20157 036/300 Write block 3494 len 36864 data: 5 4 6 5 4 4 4 3 2
20158 037/300 Write block 115 len 8192 data: 4 3
20159 038/300 Read block 3306 len 28672
20160 039/300 Write block 3028 len 36864 data: 4 4 4 5 5 6 5 5 5
20161 040/300 Write block 2863 len 32768 data: 3 3 4 4 4 4 4 5
20162 041/300 Read block 4812 len 40960
20163 042/300 Write block 4427 len 4096 data: 6
20164 043/300 Read block 2528 len 8192
20165 044/300 Read block 3438 len 8192
20166 045/300 Write block 726 len 40960 data: 3 3 4 4 4 4 4 3 4 4
20167 046/300 Flush
20168 047/300 Read block 3488 len 24576
20169 048/300 Read block 4113 len 36864
20170 049/300 Read block 429 len 16384
20171 050/300 Write block 1448 len 20480 data: 5 4 5 6 5
20172 051/300 Read block 85 len 16384
20173 052/300 Write block 4901 len 24576 data: 6 5 4 3 4 5
20174 053/300 Read block 2737 len 28672
20175 054/300 Read block 3728 len 4096
20176 055/300 Write block 3213 len 32768 data: 3 3 3 3 3 2 3 5
20177 056/300 Write block 536 len 4096 data: 3
20178 057/300 Read block 2768 len 36864
20179 058/300 Write block 4827 len 20480 data: 4 5 5 6 6
20180 059/300 Read block 4414 len 8192
20181 060/300 Write block 2121 len 12288 data: 4 4 6
20182 061/300 Read block 2497 len 24576
20183 062/300 Read block 213 len 8192
20184 063/300 Read block 2150 len 16384
20185 064/300 Write block 3147 len 36864 data: 5 5 4 4 4 3 3 3 3
20186 065/300 Read block 2977 len 8192
20187 066/300 Write block 4722 len 4096 data: 2
20188 067/300 Read block 4885 len 28672
20189 068/300 Write block 1595 len 28672 data: 2 2 2 2 3 3 4
20190 069/300 Read block 334 len 28672
20191 070/300 Read block 415 len 24576
20192 071/300 Read block 2861 len 36864
20193 072/300 Write block 1211 len 40960 data: 4 6 7 6 5 6 6 6 5 5
20194 073/300 Read block 3299 len 24576
20195 074/300 Read block 2094 len 24576
20196 075/300 Read block 3695 len 32768
20197 076/300 Flush
20198 077/300 Flush
20199 078/300 Write block 2879 len 12288 data: 6 6 5
20200 079/300 Read block 2690 len 4096
20201 080/300 Flush
20202 081/300 Flush
20203 082/300 Write block 3105 len 4096 data: 4
20204 083/300 Read block 1763 len 20480
20205 084/300 Flush
20206 085/300 Read block 4381 len 20480
20207 086/300 Read block 4224 len 24576
20208 087/300 Read block 1750 len 28672
20209 088/300 Write block 3804 len 28672 data: 2 2 2 2 2 3 4
20210 089/300 Flush
20211 090/300 Write block 888 len 36864 data: 3 4 5 5 5 5 4 3 3
20212 091/300 Write block 3432 len 20480 data: 7 6 4 4 5
20213 092/300 Read block 1433 len 20480
20214 093/300 Write block 1532 len 28672 data: 6 8 7 6 6 6 5
20215 094/300 Read block 1052 len 28672
20216 095/300 Write block 1655 len 28672 data: 5 5 5 6 6 3 4
20217 096/300 Write block 366 len 20480 data: 2 2 4 5 5
20218 097/300 Write block 1515 len 4096 data: 2
20219 098/300 Flush
20220 099/300 Flush
20221 100/300 Read block 1387 len 16384
20222 101/300 Write block 2086 len 8192 data: 2 2
20223 102/300 Write block 3573 len 20480 data: 4 3 3 3 4
20224 103/300 Read block 4152 len 4096
20225 104/300 Read block 3228 len 32768
20226 105/300 Read block 827 len 28672
20227 106/300 Flush
20228 107/300 Flush
20229 108/300 Read block 3142 len 24576
20230 109/300 Read block 4158 len 12288
20231 110/300 Read block 4034 len 20480
20232 111/300 Read block 4624 len 40960
20233 112/300 Read block 4168 len 24576
20234 113/300 Read block 1759 len 28672
20235 114/300 Write block 4608 len 20480 data: 4 3 4 3 4
20236 115/300 Write block 4758 len 36864 data: 4 4 4 4 4 4 3 3 3
20237 116/300 Read block 4195 len 40960
20238 117/300 Flush
20239 118/300 Write block 692 len 20480 data: 3 4 4 3 3
20240 119/300 Write block 538 len 28672 data: 3 4 4 4 4 5 5
20241 120/300 Read block 755 len 40960
20242 121/300 Flush
20243 122/300 Read block 1609 len 36864
20244 123/300 Read block 1145 len 12288
20245 124/300 Read block 3552 len 40960
20246 125/300 Read block 2213 len 16384
20247 126/300 Write block 3025 len 24576 data: 5 5 4 5 5 5
20248 127/300 Write block 3078 len 16384 data: 4 4 5 4
20249 128/300 Read block 437 len 20480
20250 129/300 Read block 927 len 40960
20251 130/300 Write block 3097 len 36864 data: 4 4 4 5 5 5 7 5 5
20252 131/300 Write block 3747 len 20480 data: 2 2 2 3 3
20253 132/300 Read block 1855 len 24576
20254 133/300 Write block 3390 len 16384 data: 4 4 5 5
20255 134/300 Read block 903 len 32768
20256 135/300 Write block 826 len 20480 data: 3 2 2 2 2
20257 136/300 Read block 3864 len 24576
20258 137/300 Write block 1245 len 24576 data: 4 4 2 2 2 2
20259 138/300 Read block 1925 len 36864
20260 139/300 Read block 3313 len 8192
20261 140/300 Read block 2915 len 36864
20262 141/300 Read block 4585 len 4096
20263 142/300 Write block 2382 len 28672 data: 3 3 2 2 3 3 4
20264 143/300 Read block 2791 len 16384
20265 144/300 Write block 304 len 24576 data: 3 3 3 2 2 2
20266 145/300 Read block 1715 len 24576
20267 146/300 Read block 2829 len 4096
20268 147/300 Read block 3581 len 12288
20269 148/300 Write block 969 len 12288 data: 3 6 5
20270 149/300 Read block 1478 len 36864
20271 150/300 Write block 3385 len 24576 data: 2 4 3 3 3 5
20272 151/300 Read block 2770 len 20480
20273 152/300 Read block 3685 len 28672
20274 153/300 Write block 3061 len 20480 data: 3 4 4 4 4
20275 154/300 Read block 4906 len 24576
20276 155/300 Write block 3127 len 20480 data: 3 3 4 5 5
20277 156/300 Read block 3440 len 28672
20278 157/300 Read block 1429 len 28672
20279 158/300 Write block 3311 len 32768 data: 3 4 4 4 3 4 4 4
20280 159/300 Write block 366 len 16384 data: 3 3 5 6
20281 160/300 Flush
20282 161/300 Read block 1922 len 32768
20283 162/300 Read block 4013 len 16384
20284 163/300 Write block 3875 len 12288 data: 5 4 4
20285 164/300 Read block 3701 len 20480
20286 165/300 Read block 1828 len 32768
20287 166/300 Read block 2926 len 8192
20288 167/300 Write block 3671 len 8192 data: 4 3
20289 168/300 Write block 382 len 40960 data: 7 6 5 5 5 3 3 3 4 3
20290 169/300 Write block 2207 len 24576 data: 3 3 3 3 3 3
20291 170/300 Read block 5 len 20480
20292 171/300 Read block 3059 len 12288
20293 172/300 Read block 3799 len 40960
20294 173/300 Read block 4165 len 20480
20295 174/300 Write block 4362 len 8192 data: 3 3
20296 175/300 Read block 1689 len 8192
20297 176/300 Write block 1864 len 4096 data: 5
20298 177/300 Read block 2630 len 28672
20299 178/300 Read block 2080 len 16384
20300 179/300 Flush
20301 180/300 Read block 4189 len 24576
20302 181/300 Read block 392 len 32768
20303 182/300 Read block 1734 len 16384
20304 183/300 Read block 2627 len 36864
20305 184/300 Read block 326 len 12288
20306 185/300 Write block 405 len 28672 data: 3 3 3 2 2 2 3
20307 186/300 Flush
20308 187/300 Write block 472 len 20480 data: 4 4 3 3 4
20309 188/300 Flush
20310 189/300 Read block 2263 len 24576
20311 190/300 Read block 545 len 12288
20312 191/300 Write block 2038 len 12288 data: 6 4 4
20313 192/300 Write block 2179 len 32768 data: 4 5 5 4 4 3 3 4
20314 193/300 Read block 2234 len 8192
20315 194/300 Write block 3655 len 32768 data: 4 4 4 5 5 4 3 2
20316 195/300 Read block 1464 len 12288
20317 196/300 Write block 3885 len 12288 data: 6 5 4
20318 197/300 Write block 300 len 28672 data: 2 2 2 2 4 4 4
20319 198/300 Flush
20320 199/300 Read block 4689 len 28672
20321 200/300 Read block 4276 len 36864
20322 201/300 Read block 3085 len 28672
20323 202/300 Flush
20324 203/300 Write block 1168 len 32768 data: 4 5 8 7 7 7 7 5
20325 204/300 Read block 1375 len 24576
20326 205/300 Read block 1272 len 28672
20327 206/300 Read block 4636 len 28672
20328 207/300 Read block 745 len 16384
20329 208/300 Read block 87 len 24576
20330 209/300 Flush
20331 210/300 Read block 4945 len 36864
20332 211/300 Read block 777 len 36864
20333 212/300 Read block 3712 len 24576
20334 213/300 Read block 4113 len 36864
20335 214/300 Read block 489 len 16384
20336 215/300 Read block 4080 len 12288
20337 216/300 Write block 4283 len 24576 data: 5 5 4 4 4 4
20338 217/300 Write block 1964 len 36864 data: 3 2 2 3 3 3 3 3 3
20339 218/300 Write block 4181 len 20480 data: 3 4 4 4 5
20340 219/300 Read block 1331 len 28672
20341 220/300 Read block 2759 len 16384
20342 221/300 Read block 1010 len 16384
20343 222/300 Read block 993 len 32768
20344 223/300 Read block 3721 len 40960
20345 224/300 Write block 4944 len 36864 data: 3 3 3 2 2 3 4 4 4
20346 225/300 Write block 2378 len 12288 data: 4 3 3
20347 226/300 Write block 2675 len 40960 data: 4 4 4 6 7 7 7 6 6 5
20348 227/300 Read block 3858 len 8192
20349 228/300 Read block 2278 len 4096
20350 229/300 Read block 1885 len 12288
20351 230/300 Read block 2794 len 16384
20352 231/300 Write block 229 len 16384 data: 4 3 3 3
20353 232/300 Flush
20354 233/300 Write block 526 len 20480 data: 3 3 3 3 3
20355 234/300 Read block 3775 len 4096
20356 235/300 Write block 3748 len 32768 data: 3 3 4 4 3 3 4 4
20357 236/300 Read block 325 len 24576
20358 237/300 Read block 1733 len 40960
20359 238/300 Flush
20360 239/300 Write block 2660 len 32768 data: 5 3 5 5 4 5 4 4
20361 240/300 Read block 3654 len 32768
20362 241/300 Read block 1742 len 4096
20363 242/300 Read block 2368 len 12288
20364 243/300 Flush
20365 244/300 Read block 1347 len 8192
20366 245/300 Write block 379 len 28672 data: 4 4 7 8 7 6 6
20367 246/300 Write block 4636 len 16384 data: 2 2 2 2
20368 247/300 Flush
20369 248/300 Flush
20370 249/300 Write block 1274 len 20480 data: 6 5 6 6 6
20371 250/300 Read block 1578 len 32768
20372 251/300 Write block 3460 len 16384 data: 4 4 3 4
20373 252/300 Read block 53 len 28672
20374 253/300 Flush
20375 254/300 Write block 317 len 28672 data: 6 4 3 2 2 2 3
20376 255/300 Write block 2412 len 24576 data: 4 6 6 5 5 3
20377 256/300 Read block 4403 len 20480
20378 257/300 Read block 2235 len 36864
20379 258/300 Write block 836 len 32768 data: 3 3 3 3 3 2 3 4
20380 259/300 Write block 3018 len 20480 data: 3 4 4 5 5
20381 260/300 Write block 1475 len 28672 data: 3 3 4 4 4 4 4
20382 261/300 Read block 3925 len 40960
20383 262/300 Write block 2170 len 4096 data: 3
20384 263/300 Read block 4039 len 36864
20385 264/300 Read block 892 len 16384
20386 265/300 Write block 2035 len 40960 data: 4 5 4 7 5 5 4 6 5 5
20387 266/300 Read block 1945 len 8192
20388 267/300 Write block 173 len 16384 data: 5 4 4 3
20389 268/300 Flush
20390 269/300 Write block 3081 len 32768 data: 5 4 4 4 5 4 3 2
20391 270/300 Read block 3430 len 20480
20392 271/300 Read block 180 len 40960
20393 272/300 Read block 4871 len 16384
20394 273/300 Write block 2070 len 8192 data: 7 7
20395 274/300 Read block 4587 len 40960
20396 275/300 Read block 1328 len 4096
20397 276/300 Read block 1762 len 36864
20398 277/300 Read block 247 len 8192
20399 278/300 Read block 348 len 16384
20400 279/300 Write block 2337 len 24576 data: 4 3 3 3 4 5
20401 280/300 Read block 2662 len 20480
20402 281/300 Flush
20403 282/300 Write block 2856 len 28672 data: 8 10 8 8 7 5 4
20404 283/300 Write block 3129 len 36864 data: 5 6 6 8 6 5 8 9 9
20405 284/300 Read block 3731 len 28672
20406 285/300 Read block 3031 len 24576
20407 286/300 Read block 2217 len 8192
20408 287/300 Write block 1948 len 20480 data: 5 4 4 5 6
20409 288/300 Write block 2964 len 4096 data: 4
20410 289/300 Write block 2788 len 4096 data: 5
20411 290/300 Read block 1486 len 32768
20412 291/300 Write block 3421 len 12288 data: 5 4 4
20413 292/300 Read block 2898 len 16384
20414 293/300 Write block 1166 len 4096 data: 5
20415 294/300 Read block 299 len 4096
20416 295/300 Flush
20417 296/300 Read block 501 len 36864
20418 297/300 Read block 3471 len 32768
20419 298/300 Read block 3829 len 28672
20420 299/300 Flush
20421 300/300 Write block 3134 len 32768 data: 6 9 10 10 9 9 8 6
20422 301/300 Write block 1049 len 20480 data: 6 5 5 4 7
20423 Main task has work to do, go find it
20424 got dsc Start(2)
20425 start 2
20426 Feb 21 06:49:55.766 INFO request completed, latency_us: 272, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1895c85c-03f2-4d7f-a367-2fef5247c424, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
20427 Replay: started 2, returned:Ok(())
20428 ----------------------------------------------------------------
20429 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20430 States: Active Active Offline
20431 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20432 6132 Acked Write 3 Done Done Sent false
20433 6133 Acked Read 8 Done Done Sent false
20434 6134 Acked Flush 0 Done Done Sent false
20435 6135 Acked Write 6 Done Done Sent false
20436 6136 Acked Read 4 Done Done Sent false
20437 6137 Acked Read 1 Done Done Sent false
20438 6138 Acked Write 7 Done Done Sent false
20439 6139 Acked Write 3 Done Done Sent false
20440 6140 Acked Read 2 Done Done Sent false
20441 6141 Acked Write 2 Done Done Sent false
20442 6142 Acked Write 4 Done Done Sent false
20443 6143 Acked Read 5 Done Done Sent false
20444 6144 Acked Write 1 Done Done Sent false
20445 6145 Acked Write 6 Done Done Sent false
20446 6146 Acked Read 4 Done Done Sent false
20447 6147 Acked Read 9 Done Done Sent false
20448 6148 Acked Read 9 Done Done Sent false
20449 6149 Acked Read 3 Done Done Sent false
20450 6150 Acked Write 8 Done Done Sent false
20451 6151 Acked Read 1 Done Done Sent false
20452 6152 Acked Write 10 Done Done Sent false
20453 6153 Acked Write 1 Done Done Sent false
20454 6154 Acked Read 8 Done Done Sent false
20455 6155 Acked Read 9 Done Done Sent false
20456 6156 Acked Read 2 Done Done Sent false
20457 6157 Acked Read 1 Done Done Sent false
20458 6158 Acked Read 1 Done Done Sent false
20459 6159 Acked Read 8 Done Done Sent false
20460 6160 Acked Write 5 Done Done Sent false
20461 6161 Acked Read 9 Done Done Sent false
20462 6162 Acked Read 3 Done Done Sent false
20463 6163 Acked Read 1 Done Done Sent false
20464 6164 Acked Read 2 Done Done Sent false
20465 6165 Acked Flush 0 Done Done Sent false
20466 6166 Acked Read 3 Done Done Sent false
20467 6167 Acked Write 9 Done Done Sent false
20468 6168 Acked Write 2 Done Done Sent false
20469 6169 Acked Read 7 Done Done Sent false
20470 6170 Acked Write 9 Done Done Sent false
20471 6171 Acked Write 8 Done Done Sent false
20472 6172 Acked Read 10 Done Done Sent false
20473 6173 Acked Write 1 Done Done Sent false
20474 6174 Acked Read 2 Done Done Sent false
20475 6175 Acked Read 2 Done Done Sent false
20476 6176 Acked Write 10 Done Done Sent false
20477 6177 Acked Flush 0 Done Done Sent false
20478 6178 Acked Read 6 Done Done Sent false
20479 6179 Acked Read 9 Done Done Sent false
20480 6180 Acked Read 4 Done Done Sent false
20481 6181 Acked Write 5 Done Done Sent false
20482 6182 Acked Read 4 Done Done Sent false
20483 6183 Acked Write 6 Done Done Sent false
20484 6184 Acked Read 7 Done Done Sent false
20485 6185 Acked Read 1 Done Done Sent false
20486 6186 Acked Write 8 Done Done Sent false
20487 6187 Acked Write 1 Done Done Sent false
20488 6188 Acked Read 9 Done Done Sent false
20489 6189 Acked Write 5 Done Done Sent false
20490 6190 Acked Read 2 Done Done Sent false
20491 6191 Acked Write 3 Done Done Sent false
20492 6192 Acked Read 6 Done Done Sent false
20493 6193 Acked Read 2 Done Done Sent false
20494 6194 Acked Read 4 Done Done Sent false
20495 6195 Acked Write 9 Done Done Sent false
20496 6196 Acked Read 2 Done Done Sent false
20497 6197 Acked Write 1 Done Done Sent false
20498 6198 Acked Read 7 Done Done Sent false
20499 6199 Acked Write 7 Done Done Sent false
20500 6200 Acked Read 7 Done Done Sent false
20501 6201 Acked Read 6 Done Done Sent false
20502 6202 Acked Read 9 Done Done Sent false
20503 6203 Acked Write 10 Done Done Sent false
20504 6204 Acked Read 6 Done Done Sent false
20505 6205 Acked Read 6 Done Done Sent false
20506 6206 Acked Read 8 Done Done Sent false
20507 6207 Acked Flush 0 Done Done Sent false
20508 6208 Acked Flush 0 Done Done Sent false
20509 6209 Acked Write 3 Done Done Sent false
20510 6210 Acked Read 1 Done Done Sent false
20511 6211 Acked Flush 0 Done Done Sent false
20512 6212 Acked Flush 0 Done Done Sent false
20513 6213 Acked Write 1 Done Done Sent false
20514 6214 Acked Read 5 Done Done Sent false
20515 6215 Acked Flush 0 Done Done Sent false
20516 6216 Acked Read 5 Done Done Sent false
20517 6217 Acked Read 6 Done Done Sent false
20518 6218 Acked Read 7 Done Done Sent false
20519 6219 Acked Write 7 Done Done Sent false
20520 6220 Acked Flush 0 Done Done Sent false
20521 6221 Acked Write 9 Done Done Sent false
20522 6222 Acked Write 5 Done Done Sent false
20523 6223 Acked Read 5 Done Done Sent false
20524 6224 Acked Write 7 Done Done Sent false
20525 6225 Acked Read 7 Done Done Sent false
20526 6226 Acked Write 7 Done Done Sent false
20527 6227 Acked Write 5 Done Done Sent false
20528 6228 Acked Write 1 Done Done Sent false
20529 6229 Acked Flush 0 Done Done Sent false
20530 6230 Acked Flush 0 Done Done Sent false
20531 6231 Acked Read 4 Done Done Sent false
20532 6232 Acked Write 2 Done Done Sent false
20533 6233 Acked Write 5 Done Done Sent false
20534 6234 Acked Read 1 Done Done Sent false
20535 6235 Acked Read 8 Done Done Sent false
20536 6236 Acked Read 7 Done Done Sent false
20537 6237 Acked Flush 0 Done Done Sent false
20538 6238 Acked Flush 0 Done Done Sent false
20539 6239 Acked Read 6 Done Done Sent false
20540 6240 Acked Read 3 Done Done Sent false
20541 6241 Acked Read 5 Done Done Sent false
20542 6242 Acked Read 10 Done Done Sent false
20543 6243 Acked Read 6 Done Done Sent false
20544 6244 Acked Read 7 Done Done Sent false
20545 6245 Acked Write 5 Done Done Sent false
20546 6246 Acked Write 9 Done Done Sent false
20547 6247 Acked Read 10 Done Done Sent false
20548 6248 Acked Flush 0 Done Done Sent false
20549 6249 Acked Write 5 Done Done Sent false
20550 6250 Acked Write 7 Done Done Sent false
20551 6251 Acked Read 10 Done Done Sent false
20552 6252 Acked Flush 0 Done Done Sent false
20553 6253 Acked Read 9 Done Done Sent false
20554 6254 Acked Read 3 Done Done Sent false
20555 6255 Acked Read 10 Done Done Sent false
20556 6256 Acked Read 4 Done Done Sent false
20557 6257 Acked Write 6 Done Done Sent false
20558 6258 Acked Write 4 Done Done Sent false
20559 6259 Acked Read 5 Done Done Sent false
20560 6260 Acked Read 10 Done Done Sent false
20561 6261 Acked Write 9 Done Done Sent false
20562 6262 Acked Write 5 Done Done Sent false
20563 6263 Acked Read 6 Done Done Sent false
20564 6264 Acked Write 4 Done Done Sent false
20565 6265 Acked Read 8 Done Done Sent false
20566 6266 Acked Write 5 Done Done Sent false
20567 6267 Acked Read 6 Done Done Sent false
20568 6268 Acked Write 6 Done Done Sent false
20569 6269 Acked Read 9 Done Done Sent false
20570 6270 Acked Read 2 Done Done Sent false
20571 6271 Acked Read 9 Done Done Sent false
20572 6272 Acked Read 1 Done Done Sent false
20573 6273 Acked Write 7 Done Done Sent false
20574 6274 Acked Read 4 Done Done Sent false
20575 6275 Acked Write 6 Done Done Sent false
20576 6276 Acked Read 6 Done Done Sent false
20577 6277 Acked Read 1 Done Done Sent false
20578 6278 Acked Read 3 Done Done Sent false
20579 6279 Acked Write 3 Done Done Sent false
20580 6280 Acked Read 9 Done Done Sent false
20581 6281 Acked Write 6 Done Done Sent false
20582 6282 Acked Read 5 Done Done Sent false
20583 6283 Acked Read 7 Done Done Sent false
20584 6284 Acked Write 5 Done Done Sent false
20585 6285 Acked Read 6 Done Done Sent false
20586 6286 Acked Write 5 Done Done Sent false
20587 6287 Acked Read 7 Done Done Sent false
20588 6288 Acked Read 7 Done Done Sent false
20589 6289 Acked Write 8 Done Done Sent false
20590 6290 Acked Write 4 Done Done Sent false
20591 6291 Acked Flush 0 Done Done Sent false
20592 6292 Acked Read 8 Done Done Sent false
20593 6293 Acked Read 4 Done Done Sent false
20594 6294 Acked Write 3 Done Done Sent false
20595 6295 Acked Read 5 Done Done Sent false
20596 6296 Acked Read 8 Done Done Sent false
20597 6297 Acked Read 2 Done Done Sent false
20598 6298 Acked Write 2 Done Done Sent false
20599 6299 Acked Write 10 Done Done Sent false
20600 6300 Acked Write 6 Done Done Sent false
20601 6301 Acked Read 5 Done Done Sent false
20602 6302 Acked Read 3 Done Done Sent false
20603 6303 Acked Read 10 Done Done Sent false
20604 6304 Acked Read 5 Done Done Sent false
20605 6305 Acked Write 2 Done Done Sent false
20606 6306 Acked Read 2 Done Done Sent false
20607 6307 Acked Write 1 Done Done Sent false
20608 6308 Acked Read 7 Done Done Sent false
20609 6309 Acked Read 4 Done Done Sent false
20610 6310 Acked Flush 0 Done Done Sent false
20611 6311 Acked Read 6 Done Done Sent false
20612 6312 Acked Read 8 Done Done Sent false
20613 6313 Acked Read 4 Done Done Sent false
20614 6314 Acked Read 9 Done Done Sent false
20615 6315 Acked Read 3 Done Done Sent false
20616 6316 Acked Write 7 Done Done Sent false
20617 6317 Acked Flush 0 Done Done Sent false
20618 6318 Acked Write 5 Done Done Sent false
20619 6319 Acked Flush 0 Done Done Sent false
20620 6320 Acked Read 6 Done Done Sent false
20621 6321 Acked Read 3 Done Done Sent false
20622 6322 Acked Write 3 Done Done Sent false
20623 6323 Acked Write 8 Done Done Sent false
20624 6324 Acked Read 2 Done Done Sent false
20625 6325 Acked Write 8 Done Done Sent false
20626 6326 Acked Read 3 Done Done Sent false
20627 6327 Acked Write 3 Done Done Sent false
20628 6328 Acked Write 7 Done Done Sent false
20629 6329 Acked Flush 0 Done Done Sent false
20630 6330 Acked Read 7 Done Done Sent false
20631 6331 Acked Read 9 Done Done Sent false
20632 6332 Acked Read 7 Done Done Sent false
20633 6333 Acked Flush 0 Done Done Sent false
20634 6334 Acked Write 8 Done Done Sent false
20635 6335 Acked Read 6 Done Done Sent false
20636 6336 Acked Read 7 Done Done Sent false
20637 6337 Acked Read 7 Done Done Sent false
20638 6338 Acked Read 4 Done Done Sent false
20639 6339 Acked Read 6 Done Done Sent false
20640 6340 Acked Flush 0 Done Done Sent false
20641 6341 Acked Read 9 Done Done Sent false
20642 6342 Acked Read 9 Done Done Sent false
20643 6343 Acked Read 6 Done Done Sent false
20644 6344 Acked Read 9 Done Done Sent false
20645 6345 Acked Read 4 Done Done Sent false
20646 6346 Acked Read 3 Done Done Sent false
20647 6347 Acked Write 6 Done Done Sent false
20648 6348 Acked Write 9 Done Done Sent false
20649 6349 Acked Write 5 Done Done Sent false
20650 6350 Acked Read 7 Done Done Sent false
20651 6351 Acked Read 4 Done Done Sent false
20652 6352 Acked Read 4 Done Done Sent false
20653 6353 Acked Read 8 Done Done Sent false
20654 6354 Acked Read 10 Done Done Sent false
20655 6355 Acked Write 9 Done Done Sent false
20656 6356 Acked Write 3 Done Done Sent false
20657 6357 Acked Write 10 Done Done Sent false
20658 6358 Acked Read 2 Done Done Sent false
20659 6359 Acked Read 1 Done Done Sent false
20660 6360 Acked Read 3 Done Done Sent false
20661 6361 Acked Read 4 Done Done Sent false
20662 6362 Acked Write 4 Done Done Sent false
20663 6363 Acked Flush 0 Done Done Sent false
20664 6364 Acked Write 5 Done Done Sent false
20665 6365 Acked Read 1 Done Done Sent false
20666 6366 Acked Write 8 Done Done Sent false
20667 6367 Acked Read 6 Done Done Sent false
20668 6368 Acked Read 10 Done Done Sent false
20669 6369 Acked Flush 0 Done Done Sent false
20670 6370 Acked Write 8 Done Done Sent false
20671 6371 Acked Read 8 Done Done Sent false
20672 6372 Acked Read 1 Done Done Sent false
20673 6373 Acked Read 3 Done Done Sent false
20674 6374 Acked Flush 0 Done Done Sent false
20675 6375 Acked Read 2 Done Done Sent false
20676 6376 Acked Write 7 Done Done Sent false
20677 6377 Acked Write 4 Done Done Sent false
20678 6378 Acked Flush 0 Done Done Sent false
20679 6379 Acked Flush 0 Done Done Sent false
20680 6380 Acked Write 5 Done Done Sent false
20681 6381 Acked Read 8 Done Done Sent false
20682 6382 Acked Write 4 Done Done Sent false
20683 6383 Acked Read 7 Done Done Sent false
20684 6384 Acked Flush 0 Done Done Sent false
20685 6385 Acked Write 7 Done Done Sent false
20686 6386 Acked Write 6 Done Done Sent false
20687 6387 Acked Read 5 Done Done Sent false
20688 6388 Acked Read 9 Done Done Sent false
20689 6389 Acked Write 8 Done Done Sent false
20690 6390 Acked Write 5 Done Done Sent false
20691 6391 Acked Write 7 Done Done Sent false
20692 6392 Acked Read 10 Done Done Sent false
20693 6393 Acked Write 1 Done Done Sent false
20694 6394 Acked Read 9 Done Done Sent false
20695 6395 Acked Read 4 Done Done Sent false
20696 6396 Acked Write 10 Done Done Sent false
20697 6397 Acked Read 2 Done Done Sent false
20698 6398 Acked Write 4 Done Done Sent false
20699 6399 Acked Flush 0 Done Done Sent false
20700 6400 Acked Write 8 Done Done Sent false
20701 6401 Acked Read 5 Done Done Sent false
20702 6402 Acked Read 10 Done Done Sent false
20703 6403 Acked Read 4 Done Done Sent false
20704 6404 Acked Write 2 Done Done Sent false
20705 6405 Acked Read 10 Done Done Sent false
20706 6406 Acked Read 1 Done Done Sent false
20707 6407 Acked Read 9 Done Done Sent false
20708 6408 Acked Read 2 Done Done Sent false
20709 6409 Acked Read 4 Done Done Sent false
20710 6410 Acked Write 6 Done Done Sent false
20711 6411 Acked Read 5 Done Done Sent false
20712 6412 Acked Flush 0 Done Done Sent false
20713 6413 Acked Write 7 Done Done Sent false
20714 6414 Acked Write 9 Done Done Sent false
20715 6415 Acked Read 7 Done Done Sent false
20716 6416 Acked Read 6 Done Done Sent false
20717 6417 Acked Read 2 Done Done Sent false
20718 6418 Acked Write 5 Done Done Sent false
20719 6419 Acked Write 1 Done Done Sent false
20720 6420 Acked Write 1 Done Done Sent false
20721 6421 Acked Read 8 Done Done Sent false
20722 6422 Acked Write 3 Done Done Sent false
20723 6423 Acked Read 4 Done Done Sent false
20724 6424 Acked Write 1 Done Done Sent false
20725 6425 Acked Read 1 Done Done Sent false
20726 6426 Acked Flush 0 Done Done Sent false
20727 6427 Acked Read 9 Done Done Sent false
20728 6428 Acked Read 8 Done Done Sent false
20729 6429 Acked Read 7 Done Done Sent false
20730 6430 Acked Flush 0 Done Done Sent false
20731 6431 Acked Write 8 Done Done Sent false
20732 6432 Acked Write 5 Done Done Sent false
20733 STATES DS:0 DS:1 DS:2 TOTAL
20734 Sent 0 0 301 301
20735 Done 301 301 0 602
20736 Skipped 0 0 0 0
20737 Error 0 0 0 0
20738 Last Flush: Some(JobId(6430)) Some(JobId(6430)) Some(JobId(6131))
20739 Downstairs last five completed: 6131 6130 6129 6128 6127
20740 Upstairs last five completed: 6432 6431 6430 6429 6428
20741 CLIENT: Up:0 ds:301 act:2
207422025-02-21T06:49:56.268ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
207432025-02-21T06:49:57.511ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
20744 [8830] got start action
20745 [8830] I am going to restart
20746 Starting downstairs at port 8830
20747 Make output file at "/tmp/dsc/downstairs-8830.txt"
20748 [8830][2] reports Starting
20749 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1224)
20750 [8830][2] reports Running
207512025-02-21T06:49:58.086ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
207522025-02-21T06:49:59.087ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
20753 ----------------------------------------------------------------
20754 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20755 States: Active Active Offline
20756 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20757 6132 Acked Write 3 Done Done Sent false
20758 6133 Acked Read 8 Done Done Sent false
20759 6134 Acked Flush 0 Done Done Sent false
20760 6135 Acked Write 6 Done Done Sent false
20761 6136 Acked Read 4 Done Done Sent false
20762 6137 Acked Read 1 Done Done Sent false
20763 6138 Acked Write 7 Done Done Sent false
20764 6139 Acked Write 3 Done Done Sent false
20765 6140 Acked Read 2 Done Done Sent false
20766 6141 Acked Write 2 Done Done Sent false
20767 6142 Acked Write 4 Done Done Sent false
20768 6143 Acked Read 5 Done Done Sent false
20769 6144 Acked Write 1 Done Done Sent false
20770 6145 Acked Write 6 Done Done Sent false
20771 6146 Acked Read 4 Done Done Sent false
20772 6147 Acked Read 9 Done Done Sent false
20773 6148 Acked Read 9 Done Done Sent false
20774 6149 Acked Read 3 Done Done Sent false
20775 6150 Acked Write 8 Done Done Sent false
20776 6151 Acked Read 1 Done Done Sent false
20777 6152 Acked Write 10 Done Done Sent false
20778 6153 Acked Write 1 Done Done Sent false
20779 6154 Acked Read 8 Done Done Sent false
20780 6155 Acked Read 9 Done Done Sent false
20781 6156 Acked Read 2 Done Done Sent false
20782 6157 Acked Read 1 Done Done Sent false
20783 6158 Acked Read 1 Done Done Sent false
20784 6159 Acked Read 8 Done Done Sent false
20785 6160 Acked Write 5 Done Done Sent false
20786 6161 Acked Read 9 Done Done Sent false
20787 6162 Acked Read 3 Done Done Sent false
20788 6163 Acked Read 1 Done Done Sent false
20789 6164 Acked Read 2 Done Done Sent false
20790 6165 Acked Flush 0 Done Done Sent false
20791 6166 Acked Read 3 Done Done Sent false
20792 6167 Acked Write 9 Done Done Sent false
20793 6168 Acked Write 2 Done Done Sent false
20794 6169 Acked Read 7 Done Done Sent false
20795 6170 Acked Write 9 Done Done Sent false
20796 6171 Acked Write 8 Done Done Sent false
20797 6172 Acked Read 10 Done Done Sent false
20798 6173 Acked Write 1 Done Done Sent false
20799 6174 Acked Read 2 Done Done Sent false
20800 6175 Acked Read 2 Done Done Sent false
20801 6176 Acked Write 10 Done Done Sent false
20802 6177 Acked Flush 0 Done Done Sent false
20803 6178 Acked Read 6 Done Done Sent false
20804 6179 Acked Read 9 Done Done Sent false
20805 6180 Acked Read 4 Done Done Sent false
20806 6181 Acked Write 5 Done Done Sent false
20807 6182 Acked Read 4 Done Done Sent false
20808 6183 Acked Write 6 Done Done Sent false
20809 6184 Acked Read 7 Done Done Sent false
20810 6185 Acked Read 1 Done Done Sent false
20811 6186 Acked Write 8 Done Done Sent false
20812 6187 Acked Write 1 Done Done Sent false
20813 6188 Acked Read 9 Done Done Sent false
20814 6189 Acked Write 5 Done Done Sent false
20815 6190 Acked Read 2 Done Done Sent false
20816 6191 Acked Write 3 Done Done Sent false
20817 6192 Acked Read 6 Done Done Sent false
20818 6193 Acked Read 2 Done Done Sent false
20819 6194 Acked Read 4 Done Done Sent false
20820 6195 Acked Write 9 Done Done Sent false
20821 6196 Acked Read 2 Done Done Sent false
20822 6197 Acked Write 1 Done Done Sent false
20823 6198 Acked Read 7 Done Done Sent false
20824 6199 Acked Write 7 Done Done Sent false
20825 6200 Acked Read 7 Done Done Sent false
20826 6201 Acked Read 6 Done Done Sent false
20827 6202 Acked Read 9 Done Done Sent false
20828 6203 Acked Write 10 Done Done Sent false
20829 6204 Acked Read 6 Done Done Sent false
20830 6205 Acked Read 6 Done Done Sent false
20831 6206 Acked Read 8 Done Done Sent false
20832 6207 Acked Flush 0 Done Done Sent false
20833 6208 Acked Flush 0 Done Done Sent false
20834 6209 Acked Write 3 Done Done Sent false
20835 6210 Acked Read 1 Done Done Sent false
20836 6211 Acked Flush 0 Done Done Sent false
20837 6212 Acked Flush 0 Done Done Sent false
20838 6213 Acked Write 1 Done Done Sent false
20839 6214 Acked Read 5 Done Done Sent false
20840 6215 Acked Flush 0 Done Done Sent false
20841 6216 Acked Read 5 Done Done Sent false
20842 6217 Acked Read 6 Done Done Sent false
20843 6218 Acked Read 7 Done Done Sent false
20844 6219 Acked Write 7 Done Done Sent false
20845 6220 Acked Flush 0 Done Done Sent false
20846 6221 Acked Write 9 Done Done Sent false
20847 6222 Acked Write 5 Done Done Sent false
20848 6223 Acked Read 5 Done Done Sent false
20849 6224 Acked Write 7 Done Done Sent false
20850 6225 Acked Read 7 Done Done Sent false
20851 6226 Acked Write 7 Done Done Sent false
20852 6227 Acked Write 5 Done Done Sent false
20853 6228 Acked Write 1 Done Done Sent false
20854 6229 Acked Flush 0 Done Done Sent false
20855 6230 Acked Flush 0 Done Done Sent false
20856 6231 Acked Read 4 Done Done Sent false
20857 6232 Acked Write 2 Done Done Sent false
20858 6233 Acked Write 5 Done Done Sent false
20859 6234 Acked Read 1 Done Done Sent false
20860 6235 Acked Read 8 Done Done Sent false
20861 6236 Acked Read 7 Done Done Sent false
20862 6237 Acked Flush 0 Done Done Sent false
20863 6238 Acked Flush 0 Done Done Sent false
20864 6239 Acked Read 6 Done Done Sent false
20865 6240 Acked Read 3 Done Done Sent false
20866 6241 Acked Read 5 Done Done Sent false
20867 6242 Acked Read 10 Done Done Sent false
20868 6243 Acked Read 6 Done Done Sent false
20869 6244 Acked Read 7 Done Done Sent false
20870 6245 Acked Write 5 Done Done Sent false
20871 6246 Acked Write 9 Done Done Sent false
20872 6247 Acked Read 10 Done Done Sent false
20873 6248 Acked Flush 0 Done Done Sent false
20874 6249 Acked Write 5 Done Done Sent false
20875 6250 Acked Write 7 Done Done Sent false
20876 6251 Acked Read 10 Done Done Sent false
20877 6252 Acked Flush 0 Done Done Sent false
20878 6253 Acked Read 9 Done Done Sent false
20879 6254 Acked Read 3 Done Done Sent false
20880 6255 Acked Read 10 Done Done Sent false
20881 6256 Acked Read 4 Done Done Sent false
20882 6257 Acked Write 6 Done Done Sent false
20883 6258 Acked Write 4 Done Done Sent false
20884 6259 Acked Read 5 Done Done Sent false
20885 6260 Acked Read 10 Done Done Sent false
20886 6261 Acked Write 9 Done Done Sent false
20887 6262 Acked Write 5 Done Done Sent false
20888 6263 Acked Read 6 Done Done Sent false
20889 6264 Acked Write 4 Done Done Sent false
20890 6265 Acked Read 8 Done Done Sent false
20891 6266 Acked Write 5 Done Done Sent false
20892 6267 Acked Read 6 Done Done Sent false
20893 6268 Acked Write 6 Done Done Sent false
20894 6269 Acked Read 9 Done Done Sent false
20895 6270 Acked Read 2 Done Done Sent false
20896 6271 Acked Read 9 Done Done Sent false
20897 6272 Acked Read 1 Done Done Sent false
20898 6273 Acked Write 7 Done Done Sent false
20899 6274 Acked Read 4 Done Done Sent false
20900 6275 Acked Write 6 Done Done Sent false
20901 6276 Acked Read 6 Done Done Sent false
20902 6277 Acked Read 1 Done Done Sent false
20903 6278 Acked Read 3 Done Done Sent false
20904 6279 Acked Write 3 Done Done Sent false
20905 6280 Acked Read 9 Done Done Sent false
20906 6281 Acked Write 6 Done Done Sent false
20907 6282 Acked Read 5 Done Done Sent false
20908 6283 Acked Read 7 Done Done Sent false
20909 6284 Acked Write 5 Done Done Sent false
20910 6285 Acked Read 6 Done Done Sent false
20911 6286 Acked Write 5 Done Done Sent false
20912 6287 Acked Read 7 Done Done Sent false
20913 6288 Acked Read 7 Done Done Sent false
20914 6289 Acked Write 8 Done Done Sent false
20915 6290 Acked Write 4 Done Done Sent false
20916 6291 Acked Flush 0 Done Done Sent false
20917 6292 Acked Read 8 Done Done Sent false
20918 6293 Acked Read 4 Done Done Sent false
20919 6294 Acked Write 3 Done Done Sent false
20920 6295 Acked Read 5 Done Done Sent false
20921 6296 Acked Read 8 Done Done Sent false
20922 6297 Acked Read 2 Done Done Sent false
20923 6298 Acked Write 2 Done Done Sent false
20924 6299 Acked Write 10 Done Done Sent false
20925 6300 Acked Write 6 Done Done Sent false
20926 6301 Acked Read 5 Done Done Sent false
20927 6302 Acked Read 3 Done Done Sent false
20928 6303 Acked Read 10 Done Done Sent false
20929 6304 Acked Read 5 Done Done Sent false
20930 6305 Acked Write 2 Done Done Sent false
20931 6306 Acked Read 2 Done Done Sent false
20932 6307 Acked Write 1 Done Done Sent false
20933 6308 Acked Read 7 Done Done Sent false
20934 6309 Acked Read 4 Done Done Sent false
20935 6310 Acked Flush 0 Done Done Sent false
20936 6311 Acked Read 6 Done Done Sent false
20937 6312 Acked Read 8 Done Done Sent false
20938 6313 Acked Read 4 Done Done Sent false
20939 6314 Acked Read 9 Done Done Sent false
20940 6315 Acked Read 3 Done Done Sent false
20941 6316 Acked Write 7 Done Done Sent false
20942 6317 Acked Flush 0 Done Done Sent false
20943 6318 Acked Write 5 Done Done Sent false
20944 6319 Acked Flush 0 Done Done Sent false
20945 6320 Acked Read 6 Done Done Sent false
20946 6321 Acked Read 3 Done Done Sent false
20947 6322 Acked Write 3 Done Done Sent false
20948 6323 Acked Write 8 Done Done Sent false
20949 6324 Acked Read 2 Done Done Sent false
20950 6325 Acked Write 8 Done Done Sent false
20951 6326 Acked Read 3 Done Done Sent false
20952 6327 Acked Write 3 Done Done Sent false
20953 6328 Acked Write 7 Done Done Sent false
20954 6329 Acked Flush 0 Done Done Sent false
20955 6330 Acked Read 7 Done Done Sent false
20956 6331 Acked Read 9 Done Done Sent false
20957 6332 Acked Read 7 Done Done Sent false
20958 6333 Acked Flush 0 Done Done Sent false
20959 6334 Acked Write 8 Done Done Sent false
20960 6335 Acked Read 6 Done Done Sent false
20961 6336 Acked Read 7 Done Done Sent false
20962 6337 Acked Read 7 Done Done Sent false
20963 6338 Acked Read 4 Done Done Sent false
20964 6339 Acked Read 6 Done Done Sent false
20965 6340 Acked Flush 0 Done Done Sent false
20966 6341 Acked Read 9 Done Done Sent false
20967 6342 Acked Read 9 Done Done Sent false
20968 6343 Acked Read 6 Done Done Sent false
20969 6344 Acked Read 9 Done Done Sent false
20970 6345 Acked Read 4 Done Done Sent false
20971 6346 Acked Read 3 Done Done Sent false
20972 6347 Acked Write 6 Done Done Sent false
20973 6348 Acked Write 9 Done Done Sent false
20974 6349 Acked Write 5 Done Done Sent false
20975 6350 Acked Read 7 Done Done Sent false
20976 6351 Acked Read 4 Done Done Sent false
20977 6352 Acked Read 4 Done Done Sent false
20978 6353 Acked Read 8 Done Done Sent false
20979 6354 Acked Read 10 Done Done Sent false
20980 6355 Acked Write 9 Done Done Sent false
20981 6356 Acked Write 3 Done Done Sent false
20982 6357 Acked Write 10 Done Done Sent false
20983 6358 Acked Read 2 Done Done Sent false
20984 6359 Acked Read 1 Done Done Sent false
20985 6360 Acked Read 3 Done Done Sent false
20986 6361 Acked Read 4 Done Done Sent false
20987 6362 Acked Write 4 Done Done Sent false
20988 6363 Acked Flush 0 Done Done Sent false
20989 6364 Acked Write 5 Done Done Sent false
20990 6365 Acked Read 1 Done Done Sent false
20991 6366 Acked Write 8 Done Done Sent false
20992 6367 Acked Read 6 Done Done Sent false
20993 6368 Acked Read 10 Done Done Sent false
20994 6369 Acked Flush 0 Done Done Sent false
20995 6370 Acked Write 8 Done Done Sent false
20996 6371 Acked Read 8 Done Done Sent false
20997 6372 Acked Read 1 Done Done Sent false
20998 6373 Acked Read 3 Done Done Sent false
20999 6374 Acked Flush 0 Done Done Sent false
21000 6375 Acked Read 2 Done Done Sent false
21001 6376 Acked Write 7 Done Done Sent false
21002 6377 Acked Write 4 Done Done Sent false
21003 6378 Acked Flush 0 Done Done Sent false
21004 6379 Acked Flush 0 Done Done Sent false
21005 6380 Acked Write 5 Done Done Sent false
21006 6381 Acked Read 8 Done Done Sent false
21007 6382 Acked Write 4 Done Done Sent false
21008 6383 Acked Read 7 Done Done Sent false
21009 6384 Acked Flush 0 Done Done Sent false
21010 6385 Acked Write 7 Done Done Sent false
21011 6386 Acked Write 6 Done Done Sent false
21012 6387 Acked Read 5 Done Done Sent false
21013 6388 Acked Read 9 Done Done Sent false
21014 6389 Acked Write 8 Done Done Sent false
21015 6390 Acked Write 5 Done Done Sent false
21016 6391 Acked Write 7 Done Done Sent false
21017 6392 Acked Read 10 Done Done Sent false
21018 6393 Acked Write 1 Done Done Sent false
21019 6394 Acked Read 9 Done Done Sent false
21020 6395 Acked Read 4 Done Done Sent false
21021 6396 Acked Write 10 Done Done Sent false
21022 6397 Acked Read 2 Done Done Sent false
21023 6398 Acked Write 4 Done Done Sent false
21024 6399 Acked Flush 0 Done Done Sent false
21025 6400 Acked Write 8 Done Done Sent false
21026 6401 Acked Read 5 Done Done Sent false
21027 6402 Acked Read 10 Done Done Sent false
21028 6403 Acked Read 4 Done Done Sent false
21029 6404 Acked Write 2 Done Done Sent false
21030 6405 Acked Read 10 Done Done Sent false
21031 6406 Acked Read 1 Done Done Sent false
21032 6407 Acked Read 9 Done Done Sent false
21033 6408 Acked Read 2 Done Done Sent false
21034 6409 Acked Read 4 Done Done Sent false
21035 6410 Acked Write 6 Done Done Sent false
21036 6411 Acked Read 5 Done Done Sent false
21037 6412 Acked Flush 0 Done Done Sent false
21038 6413 Acked Write 7 Done Done Sent false
21039 6414 Acked Write 9 Done Done Sent false
21040 6415 Acked Read 7 Done Done Sent false
21041 6416 Acked Read 6 Done Done Sent false
21042 6417 Acked Read 2 Done Done Sent false
21043 6418 Acked Write 5 Done Done Sent false
21044 6419 Acked Write 1 Done Done Sent false
21045 6420 Acked Write 1 Done Done Sent false
21046 6421 Acked Read 8 Done Done Sent false
21047 6422 Acked Write 3 Done Done Sent false
21048 6423 Acked Read 4 Done Done Sent false
21049 6424 Acked Write 1 Done Done Sent false
21050 6425 Acked Read 1 Done Done Sent false
21051 6426 Acked Flush 0 Done Done Sent false
21052 6427 Acked Read 9 Done Done Sent false
21053 6428 Acked Read 8 Done Done Sent false
21054 6429 Acked Read 7 Done Done Sent false
21055 6430 Acked Flush 0 Done Done Sent false
21056 6431 Acked Write 8 Done Done Sent false
21057 6432 Acked Write 5 Done Done Sent false
21058 STATES DS:0 DS:1 DS:2 TOTAL
21059 Sent 0 0 301 301
21060 Done 301 301 0 602
21061 Skipped 0 0 0 0
21062 Error 0 0 0 0
21063 Last Flush: Some(JobId(6430)) Some(JobId(6430)) Some(JobId(6131))
21064 Downstairs last five completed: 6131 6130 6129 6128 6127
21065 Upstairs last five completed: 6432 6431 6430 6429 6428
21066 CLIENT: Up:0 ds:301 act:2
210672025-02-21T06:50:00.270ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
210682025-02-21T06:50:01.091ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
210692025-02-21T06:50:01.516ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
210702025-02-21T06:50:01.516ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60335) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
210712025-02-21T06:50:01.517ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
210722025-02-21T06:50:01.518ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
210732025-02-21T06:50:01.518ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
210742025-02-21T06:50:01.518ZINFOcrucible: send last flush ID to this DS: Some(JobId(6131)) = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
210752025-02-21T06:50:01.518ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
210762025-02-21T06:50:01.518ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6131)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
210772025-02-21T06:50:01.518ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6131)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21078 ----------------------------------------------------------------
21079 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21080 Downstairs last five completed: 6433 6432 6431 6430 6429
21081 Upstairs last five completed: 6433 6432 6431 6430 6429
21082 CLIENT: Up:0 ds:0 act:3
21083 Replay: All jobs finished
21084 Main task has work to do, go find it
21085 got dsc Stop(1)
21086 stop 1
21087 [8820] Got stop action so:false kr:false
21088 Feb 21 06:50:03.772 INFO request completed, latency_us: 367, response_code: 204, uri: /stop/cid/1, method: POST, req_id: c089d3f5-1f8c-4355-b80f-d17f23002176, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
210892025-02-21T06:50:03.772ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
210902025-02-21T06:50:03.772ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
210912025-02-21T06:50:03.772ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21092 Feb 21 06:50:03.772 INFO{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:50:03.772978425Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
21093 request completed, latency_us: 159, response_code: 200, uri: /state/cid/1, method: GET, req_id: 98e6aaa5-097d-43d5-99ba-81959a0ee283, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
21094 [8820] Exited with: ExitStatus(unix_wait_status(9))
21095 [8820][1] reports Exit
21096 Feb 21 06:50:07.775 INFO request completed, latency_us: 198, response_code: 200, uri: /state/cid/1, method: GET, req_id: 94f96e0f-6527-418d-ac66-1bd095d06de5, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
21097 001/300 Read block 1821 len 16384
21098 002/300 Read block 1021 len 28672
21099 003/300 Write block 2905 len 32768 data: 7 8 5 5 5 5 4 5
21100 004/300 Read block 1431 len 32768
21101 005/300 Read block 865 len 20480
21102 006/300 Read block 1615 len 40960
21103 007/300 Read block 1293 len 32768
21104 008/300 Read block 2904 len 36864
21105 009/300 Read block 1154 len 20480
21106 010/300 Flush
21107 011/300 Read block 4358 len 4096
21108 012/300 Read block 4675 len 28672
21109 013/300 Read block 1131 len 20480
21110 014/300 Write block 406 len 12288 data: 4 4 3
21111 015/300 Read block 2021 len 12288
21112 016/300 Flush
21113 017/300 Write block 2258 len 12288 data: 2 4 4
21114 018/300 Read block 615 len 20480
21115 019/300 Read block 533 len 28672
21116 020/300 Read block 1577 len 16384
21117 021/300 Read block 922 len 4096
21118 022/300 Write block 914 len 4096 data: 3
21119 023/300 Write block 575 len 20480 data: 3 5 4 3 4
21120 024/300 Read block 493 len 12288
21121 025/300 Read block 2975 len 36864
21122 026/300 Write block 4745 len 24576 data: 4 3 3 5 6 5
21123 027/300 Read block 1359 len 16384
21124 028/300 Read block 1943 len 12288
21125 029/300 Read block 3845 len 28672
21126 030/300 Write block 1047 len 40960 data: 5 6 7 6 6 5 8 8 7 7
21127 031/300 Write block 3567 len 12288 data: 5 4 4
21128 032/300 Read block 4686 len 16384
21129 033/300 Write block 4903 len 4096 data: 5
21130 034/300 Read block 2932 len 40960
21131 035/300 Write block 4415 len 4096 data: 5
21132 036/300 Write block 901 len 28672 data: 4 5 5 6 6 5 4
21133 037/300 Flush
21134 038/300 Read block 3582 len 40960
21135 039/300 Read block 1242 len 32768
21136 040/300 Flush
21137 041/300 Read block 2478 len 28672
21138 042/300 Read block 3055 len 40960
21139 043/300 Write block 1955 len 12288 data: 5 5 4
21140 044/300 Write block 3315 len 4096 data: 4
21141 045/300 Write block 4582 len 20480 data: 3 2 2 2 4
21142 046/300 Write block 3118 len 20480 data: 4 4 6 4 4
21143 047/300 Read block 4301 len 8192
21144 048/300 Read block 930 len 16384
21145 049/300 Read block 2012 len 32768
21146 050/300 Write block 3861 len 16384 data: 4 5 5 4
21147 051/300 Write block 244 len 16384 data: 3 4 4 4
21148 052/300 Read block 3390 len 8192
21149 053/300 Write block 2984 len 36864 data: 4 4 5 5 4 4 4 4 3
21150 054/300 Read block 941 len 24576
21151 055/300 Read block 2129 len 40960
21152 056/300 Read block 4432 len 36864
21153 057/300 Read block 2931 len 36864
21154 058/300 Read block 2093 len 40960
21155 059/300 Read block 2332 len 28672
21156 060/300 Write block 4410 len 12288 data: 4 5 5
21157 061/300 Read block 4198 len 4096
21158 062/300 Read block 3320 len 28672
21159 063/300 Read block 3469 len 36864
21160 064/300 Flush
21161 065/300 Read block 1934 len 20480
21162 066/300 Flush
21163 067/300 Write block 4355 len 32768 data: 6 6 7 6 5 5 4 4
21164 068/300 Read block 611 len 4096
21165 069/300 Read block 2574 len 28672
21166 070/300 Flush
21167 071/300 Write block 4111 len 40960 data: 5 6 4 4 4 4 3 3 3 5
21168 072/300 Read block 1326 len 4096
21169 073/300 Read block 4831 len 24576
21170 074/300 Write block 2951 len 28672 data: 7 8 9 7 7 6 4
21171 075/300 Write block 330 len 40960 data: 4 3 3 3 3 4 4 4 4 3
21172 076/300 Write block 4954 len 8192 data: 2 2
21173 077/300 Read block 2948 len 28672
21174 078/300 Write block 2974 len 24576 data: 2 3 4 4 4 3
21175 079/300 Flush
21176 080/300 Read block 2932 len 8192
21177 081/300 Flush
21178 082/300 Write block 1301 len 24576 data: 3 3 2 2 2 2
21179 083/300 Write block 4144 len 12288 data: 4 4 4
21180 084/300 Write block 4110 len 12288 data: 5 6 7
21181 085/300 Read block 2527 len 40960
21182 086/300 Flush
21183 087/300 Write block 2333 len 32768 data: 3 2 3 4 5 4 4 4
21184 088/300 Read block 2310 len 32768
21185 089/300 Write block 1536 len 4096 data: 7
21186 090/300 Read block 1490 len 36864
21187 091/300 Read block 191 len 8192
21188 092/300 Flush
21189 093/300 Write block 1577 len 12288 data: 4 4 4
21190 094/300 Write block 3102 len 24576 data: 6 8 6 6 3 4
21191 095/300 Read block 1571 len 36864
21192 096/300 Read block 3211 len 4096
21193 097/300 Read block 1604 len 12288
21194 098/300 Read block 2231 len 12288
21195 099/300 Write block 1499 len 12288 data: 6 7 7
21196 100/300 Write block 2759 len 28672 data: 5 5 5 6 5 5 5
21197 101/300 Read block 573 len 12288
21198 102/300 Write block 3224 len 12288 data: 7 6 5
21199 103/300 Read block 114 len 28672
21200 104/300 Write block 185 len 20480 data: 7 6 5 5 4
21201 105/300 Write block 606 len 28672 data: 2 3 3 3 3 2 3
21202 106/300 Read block 1421 len 36864
21203 107/300 Flush
21204 108/300 Write block 1551 len 4096 data: 3
21205 109/300 Read block 2487 len 28672
21206 110/300 Write block 3526 len 24576 data: 3 3 4 3 3 4
21207 111/300 Write block 2514 len 20480 data: 4 5 5 6 5
21208 112/300 Read block 3456 len 36864
21209 113/300 Read block 2725 len 8192
21210 114/300 Write block 3399 len 40960 data: 2 4 4 4 4 4 5 5 3 4
21211 115/300 Flush
21212 116/300 Read block 2327 len 20480
21213 117/300 Flush
21214 118/300 Flush
21215 119/300 Read block 4881 len 16384
21216 120/300 Write block 1709 len 12288 data: 4 3 2
21217 121/300 Flush
21218 122/300 Write block 472 len 28672 data: 5 5 4 4 5 4 3
21219 123/300 Write block 4329 len 12288 data: 4 4 4
21220 124/300 Read block 4540 len 16384
21221 125/300 Read block 1199 len 12288
21222 126/300 Read block 2836 len 12288
21223 127/300 Write block 2607 len 16384 data: 5 4 5 4
21224 128/300 Write block 2916 len 4096 data: 3
21225 129/300 Write block 4698 len 36864 data: 4 4 3 3 5 5 4 4 4
21226 130/300 Write block 2900 len 12288 data: 3 4 4
21227 131/300 Read block 361 len 12288
21228 132/300 Read block 144 len 40960
21229 133/300 Write block 92 len 40960 data: 3 2 2 3 3 3 3 4 4 4
21230 134/300 Flush
21231 135/300 Write block 3270 len 40960 data: 4 4 4 5 4 4 2 2 2 2
21232 136/300 Read block 1661 len 36864
21233 137/300 Read block 3218 len 12288
21234 138/300 Flush
21235 139/300 Write block 1516 len 40960 data: 2 2 2 2 3 3 4 4 3 5
21236 140/300 Read block 370 len 28672
21237 141/300 Read block 2522 len 8192
21238 142/300 Flush
21239 143/300 Write block 3050 len 16384 data: 4 3 3 3
21240 144/300 Read block 701 len 8192
21241 145/300 Write block 4672 len 20480 data: 5 5 6 7 7
21242 146/300 Flush
21243 147/300 Flush
21244 148/300 Write block 1546 len 12288 data: 2 2 2
21245 149/300 Read block 3122 len 20480
21246 150/300 Read block 241 len 20480
21247 151/300 Flush
21248 152/300 Flush
21249 153/300 Flush
21250 154/300 Write block 3629 len 8192 data: 4 4
21251 155/300 Read block 2116 len 16384
21252 156/300 Write block 353 len 8192 data: 3 3
21253 157/300 Write block 703 len 20480 data: 3 2 2 2 2
21254 158/300 Read block 2564 len 32768
21255 159/300 Read block 2428 len 24576
21256 160/300 Read block 2487 len 20480
21257 161/300 Write block 3238 len 32768 data: 3 3 3 3 4 5 4 4
21258 162/300 Flush
21259 163/300 Write block 4554 len 40960 data: 3 4 4 3 3 3 4 4 3 2
21260 164/300 Write block 3178 len 4096 data: 4
21261 165/300 Write block 98 len 36864 data: 4 5 5 5 5 5 4 6 6
21262 166/300 Read block 711 len 16384
21263 167/300 Read block 4124 len 20480
21264 168/300 Read block 3431 len 28672
21265 169/300 Write block 3144 len 20480 data: 4 4 4 6 6
21266 170/300 Flush
21267 171/300 Read block 3032 len 4096
21268 172/300 Write block 2326 len 16384 data: 5 3 4 4
21269 173/300 Write block 3167 len 4096 data: 6
21270 174/300 Read block 4355 len 16384
21271 175/300 Read block 4146 len 40960
21272 176/300 Read block 2043 len 28672
21273 177/300 Write block 1097 len 28672 data: 3 3 3 3 3 4 4
21274 178/300 Read block 130 len 28672
21275 179/300 Write block 532 len 36864 data: 2 4 3 3 4 3 4 5 5
21276 180/300 Write block 1665 len 28672 data: 4 4 4 4 3 2 2
21277 181/300 Read block 4 len 20480
21278 182/300 Read block 1792 len 32768
21279 183/300 Flush
21280 184/300 Write block 1175 len 40960 data: 6 6 5 5 4 5 5 5 3 2
21281 185/300 Write block 3381 len 12288 data: 2 2 2
21282 186/300 Read block 2378 len 20480
21283 187/300 Write block 4806 len 32768 data: 3 2 3 2 4 4 4 4
21284 188/300 Read block 1333 len 36864
21285 189/300 Write block 295 len 36864 data: 4 2 2 2 2 3 3 3 3
21286 190/300 Read block 102 len 40960
21287 191/300 Read block 2006 len 16384
21288 192/300 Write block 2242 len 40960 data: 3 3 3 4 4 3 4 4 4 3
21289 193/300 Write block 4674 len 16384 data: 7 8 8 6
21290 194/300 Write block 3018 len 8192 data: 4 5
21291 195/300 Write block 2008 len 40960 data: 6 4 4 5 4 4 4 5 3 4
21292 196/300 Read block 384 len 20480
21293 197/300 Read block 3235 len 32768
21294 198/300 Read block 794 len 32768
21295 199/300 Write block 2976 len 8192 data: 5 5
21296 200/300 Read block 228 len 16384
21297 201/300 Write block 922 len 20480 data: 3 3 4 5 5
21298 202/300 Write block 3087 len 36864 data: 4 3 3 3 3 3 3 5 5
21299 203/300 Read block 2415 len 12288
21300 204/300 Read block 1124 len 8192
21301 205/300 Flush
21302 206/300 Flush
21303 207/300 Read block 1224 len 36864
21304 208/300 Write block 4418 len 16384 data: 7 6 4 4
21305 209/300 Read block 1244 len 24576
21306 210/300 Write block 1893 len 16384 data: 6 5 4 4
21307 211/300 Read block 4841 len 24576
21308 212/300 Read block 357 len 16384
21309 213/300 Read block 4927 len 4096
21310 214/300 Read block 4108 len 24576
21311 215/300 Flush
21312 216/300 Read block 3359 len 8192
21313 217/300 Write block 2676 len 28672 data: 5 5 7 8 8 8 7
21314 218/300 Read block 2015 len 4096
21315 219/300 Flush
21316 220/300 Read block 2437 len 32768
21317 221/300 Write block 4879 len 28672 data: 7 8 6 6 6 5 4
21318 222/300 Flush
21319 223/300 Write block 3206 len 20480 data: 4 4 3 2 2
21320 224/300 Write block 1352 len 24576 data: 6 7 5 6 6 6
21321 225/300 Read block 2839 len 4096
21322 226/300 Write block 4811 len 40960 data: 5 5 5 6 7 7 7 7 6 6
21323 227/300 Read block 2321 len 36864
21324 228/300 Read block 1212 len 36864
21325 229/300 Read block 2017 len 20480
21326 230/300 Write block 3239 len 12288 data: 4 4 4
21327 231/300 Write block 1665 len 8192 data: 5 5
21328 232/300 Flush
21329 233/300 Flush
21330 234/300 Write block 1692 len 12288 data: 4 4 3
21331 235/300 Read block 359 len 36864
21332 236/300 Read block 559 len 40960
21333 237/300 Read block 3089 len 28672
21334 238/300 Write block 2794 len 40960 data: 3 3 3 3 3 3 3 4 4 4
21335 239/300 Read block 136 len 4096
21336 240/300 Write block 427 len 24576 data: 6 8 8 8 9 9
21337 241/300 Write block 4834 len 32768 data: 6 4 4 3 4 3 2 2
21338 242/300 Write block 4487 len 40960 data: 4 3 3 3 3 4 3 3 3 3
21339 243/300 Flush
21340 244/300 Write block 1156 len 20480 data: 2 2 2 2 2
21341 245/300 Write block 3163 len 32768 data: 4 5 5 6 7 6 6 5
21342 246/300 Read block 4457 len 4096
21343 247/300 Flush
21344 248/300 Read block 4615 len 20480
21345 249/300 Flush
21346 250/300 Write block 3837 len 20480 data: 4 4 4 6 7
21347 251/300 Write block 4808 len 24576 data: 4 3 5 6 6 6
21348 252/300 Write block 367 len 32768 data: 4 6 7 6 5 6 4 5
21349 253/300 Read block 937 len 40960
21350 254/300 Write block 737 len 28672 data: 7 8 7 6 6 4 3
21351 255/300 Write block 28 len 36864 data: 4 4 5 4 3 3 3 3 3
21352 256/300 Flush
21353 257/300 Read block 4243 len 4096
21354 258/300 Read block 3334 len 16384
21355 259/300 Write block 4901 len 20480 data: 7 6 6 4 5
21356 260/300 Write block 3821 len 28672 data: 9 7 7 5 4 5 4
21357 261/300 Read block 338 len 32768
21358 262/300 Write block 3135 len 40960 data: 10 11 11 10 10 9 7 5 5 5
21359 263/300 Read block 4688 len 32768
21360 264/300 Write block 1205 len 20480 data: 4 3 2 2 3
21361 265/300 Write block 3775 len 32768 data: 7 5 3 2 2 3 3 3
21362 266/300 Read block 1375 len 40960
21363 267/300 Write block 1377 len 16384 data: 4 4 4 4
21364 268/300 Read block 3182 len 20480
21365 269/300 Write block 1736 len 36864 data: 2 2 2 2 3 4 4 4 4
21366 270/300 Read block 464 len 20480
21367 271/300 Write block 4064 len 36864 data: 2 2 2 2 3 3 3 3 4
21368 272/300 Flush
21369 273/300 Write block 2649 len 36864 data: 4 3 3 2 3 3 4 5 5
21370 274/300 Read block 3112 len 12288
21371 275/300 Flush
21372 276/300 Write block 2696 len 32768 data: 5 6 5 5 3 4 4 3
21373 277/300 Write block 4939 len 8192 data: 4 4
21374 278/300 Write block 3109 len 40960 data: 3 3 3 3 4 5 5 5 3 5
21375 279/300 Read block 2851 len 8192
21376 280/300 Read block 4033 len 36864
21377 281/300 Read block 3598 len 20480
21378 282/300 Read block 676 len 24576
21379 283/300 Read block 4149 len 20480
21380 284/300 Read block 4081 len 28672
21381 285/300 Write block 3364 len 24576 data: 4 4 4 4 3 3
21382 286/300 Write block 220 len 20480 data: 5 4 4 3 4
21383 287/300 Read block 106 len 8192
21384 288/300 Write block 4137 len 28672 data: 3 3 3 2 2 2 2
21385 289/300 Write block 720 len 24576 data: 4 4 3 5 5 4
21386 290/300 Write block 3901 len 16384 data: 2 2 2 2
21387 291/300 Read block 4187 len 36864
21388 292/300 Read block 1703 len 24576
21389 293/300 Read block 4067 len 32768
21390 294/300 Read block 1134 len 12288
21391 295/300 Write block 3031 len 40960 data: 6 6 7 6 6 6 5 5 6 5
21392 296/300 Flush
21393 297/300 Read block 253 len 20480
21394 298/300 Read block 2699 len 24576
21395 299/300 Write block 3693 len 40960 data: 2 2 2 4 4 3 3 4 4 5
21396 300/300 Read block 2967 len 40960
21397 301/300 Read block 1339 len 16384
21398 Main task has work to do, go find it
21399 got dsc Start(1)
21400 start 1
21401 Feb 21 06:50:08.019 INFO request completed, latency_us: 227, response_code: 204, uri: /start/cid/1, method: POST, req_id: 0dea3d6d-a9b3-4791-9f76-28a8efae5831, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
21402 Replay: started 1, returned:Ok(())
21403 ----------------------------------------------------------------
21404 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21405 States: Active Offline Active
21406 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21407 6434 Acked Read 4 Done Sent Done false
21408 6435 Acked Read 7 Done Sent Done false
21409 6436 Acked Write 8 Done Sent Done false
21410 6437 Acked Read 8 Done Sent Done false
21411 6438 Acked Read 5 Done Sent Done false
21412 6439 Acked Read 10 Done Sent Done false
21413 6440 Acked Read 8 Done Sent Done false
21414 6441 Acked Read 9 Done Sent Done false
21415 6442 Acked Read 5 Done Sent Done false
21416 6443 Acked Flush 0 Done Sent Done false
21417 6444 Acked Read 1 Done Sent Done false
21418 6445 Acked Read 7 Done Sent Done false
21419 6446 Acked Read 5 Done Sent Done false
21420 6447 Acked Write 3 Done Sent Done false
21421 6448 Acked Read 3 Done Sent Done false
21422 6449 Acked Flush 0 Done Sent Done false
21423 6450 Acked Write 3 Done Sent Done false
21424 6451 Acked Read 5 Done Sent Done false
21425 6452 Acked Read 7 Done Sent Done false
21426 6453 Acked Read 4 Done Sent Done false
21427 6454 Acked Read 1 Done Sent Done false
21428 6455 Acked Write 1 Done Sent Done false
21429 6456 Acked Write 5 Done Sent Done false
21430 6457 Acked Read 3 Done Sent Done false
21431 6458 Acked Read 9 Done Sent Done false
21432 6459 Acked Write 6 Done Sent Done false
21433 6460 Acked Read 4 Done Sent Done false
21434 6461 Acked Read 3 Done Sent Done false
21435 6462 Acked Read 7 Done Sent Done false
21436 6463 Acked Write 10 Done Sent Done false
21437 6464 Acked Write 3 Done Sent Done false
21438 6465 Acked Read 4 Done Sent Done false
21439 6466 Acked Write 1 Done Sent Done false
21440 6467 Acked Read 10 Done Sent Done false
21441 6468 Acked Write 1 Done Sent Done false
21442 6469 Acked Write 7 Done Sent Done false
21443 6470 Acked Flush 0 Done Sent Done false
21444 6471 Acked Read 10 Done Sent Done false
21445 6472 Acked Read 8 Done Sent Done false
21446 6473 Acked Flush 0 Done Sent Done false
21447 6474 Acked Read 7 Done Sent Done false
21448 6475 Acked Read 10 Done Sent Done false
21449 6476 Acked Write 3 Done Sent Done false
21450 6477 Acked Write 1 Done Sent Done false
21451 6478 Acked Write 5 Done Sent Done false
21452 6479 Acked Write 5 Done Sent Done false
21453 6480 Acked Read 2 Done Sent Done false
21454 6481 Acked Read 4 Done Sent Done false
21455 6482 Acked Read 8 Done Sent Done false
21456 6483 Acked Write 4 Done Sent Done false
21457 6484 Acked Write 4 Done Sent Done false
21458 6485 Acked Read 2 Done Sent Done false
21459 6486 Acked Write 9 Done Sent Done false
21460 6487 Acked Read 6 Done Sent Done false
21461 6488 Acked Read 10 Done Sent Done false
21462 6489 Acked Read 9 Done Sent Done false
21463 6490 Acked Read 9 Done Sent Done false
21464 6491 Acked Read 10 Done Sent Done false
21465 6492 Acked Read 7 Done Sent Done false
21466 6493 Acked Write 3 Done Sent Done false
21467 6494 Acked Read 1 Done Sent Done false
21468 6495 Acked Read 7 Done Sent Done false
21469 6496 Acked Read 9 Done Sent Done false
21470 6497 Acked Flush 0 Done Sent Done false
21471 6498 Acked Read 5 Done Sent Done false
21472 6499 Acked Flush 0 Done Sent Done false
21473 6500 Acked Write 8 Done Sent Done false
21474 6501 Acked Read 1 Done Sent Done false
21475 6502 Acked Read 7 Done Sent Done false
21476 6503 Acked Flush 0 Done Sent Done false
21477 6504 Acked Write 10 Done Sent Done false
21478 6505 Acked Read 1 Done Sent Done false
21479 6506 Acked Read 6 Done Sent Done false
21480 6507 Acked Write 7 Done Sent Done false
21481 6508 Acked Write 10 Done Sent Done false
21482 6509 Acked Write 2 Done Sent Done false
21483 6510 Acked Read 7 Done Sent Done false
21484 6511 Acked Write 6 Done Sent Done false
21485 6512 Acked Flush 0 Done Sent Done false
21486 6513 Acked Read 2 Done Sent Done false
21487 6514 Acked Flush 0 Done Sent Done false
21488 6515 Acked Write 6 Done Sent Done false
21489 6516 Acked Write 3 Done Sent Done false
21490 6517 Acked Write 3 Done Sent Done false
21491 6518 Acked Read 10 Done Sent Done false
21492 6519 Acked Flush 0 Done Sent Done false
21493 6520 Acked Write 8 Done Sent Done false
21494 6521 Acked Read 8 Done Sent Done false
21495 6522 Acked Write 1 Done Sent Done false
21496 6523 Acked Read 9 Done Sent Done false
21497 6524 Acked Read 2 Done Sent Done false
21498 6525 Acked Flush 0 Done Sent Done false
21499 6526 Acked Write 3 Done Sent Done false
21500 6527 Acked Write 6 Done Sent Done false
21501 6528 Acked Read 9 Done Sent Done false
21502 6529 Acked Read 1 Done Sent Done false
21503 6530 Acked Read 3 Done Sent Done false
21504 6531 Acked Read 3 Done Sent Done false
21505 6532 Acked Write 3 Done Sent Done false
21506 6533 Acked Write 7 Done Sent Done false
21507 6534 Acked Read 3 Done Sent Done false
21508 6535 Acked Write 3 Done Sent Done false
21509 6536 Acked Read 7 Done Sent Done false
21510 6537 Acked Write 5 Done Sent Done false
21511 6538 Acked Write 7 Done Sent Done false
21512 6539 Acked Read 9 Done Sent Done false
21513 6540 Acked Flush 0 Done Sent Done false
21514 6541 Acked Write 1 Done Sent Done false
21515 6542 Acked Read 7 Done Sent Done false
21516 6543 Acked Write 6 Done Sent Done false
21517 6544 Acked Write 5 Done Sent Done false
21518 6545 Acked Read 9 Done Sent Done false
21519 6546 Acked Read 2 Done Sent Done false
21520 6547 Acked Write 10 Done Sent Done false
21521 6548 Acked Flush 0 Done Sent Done false
21522 6549 Acked Read 5 Done Sent Done false
21523 6550 Acked Flush 0 Done Sent Done false
21524 6551 Acked Flush 0 Done Sent Done false
21525 6552 Acked Read 4 Done Sent Done false
21526 6553 Acked Write 3 Done Sent Done false
21527 6554 Acked Flush 0 Done Sent Done false
21528 6555 Acked Write 7 Done Sent Done false
21529 6556 Acked Write 3 Done Sent Done false
21530 6557 Acked Read 4 Done Sent Done false
21531 6558 Acked Read 3 Done Sent Done false
21532 6559 Acked Read 3 Done Sent Done false
21533 6560 Acked Write 4 Done Sent Done false
21534 6561 Acked Write 1 Done Sent Done false
21535 6562 Acked Write 9 Done Sent Done false
21536 6563 Acked Write 3 Done Sent Done false
21537 6564 Acked Read 3 Done Sent Done false
21538 6565 Acked Read 10 Done Sent Done false
21539 6566 Acked Write 10 Done Sent Done false
21540 6567 Acked Flush 0 Done Sent Done false
21541 6568 Acked Write 10 Done Sent Done false
21542 6569 Acked Read 9 Done Sent Done false
21543 6570 Acked Read 3 Done Sent Done false
21544 6571 Acked Flush 0 Done Sent Done false
21545 6572 Acked Write 10 Done Sent Done false
21546 6573 Acked Read 7 Done Sent Done false
21547 6574 Acked Read 2 Done Sent Done false
21548 6575 Acked Flush 0 Done Sent Done false
21549 6576 Acked Write 4 Done Sent Done false
21550 6577 Acked Read 2 Done Sent Done false
21551 6578 Acked Write 5 Done Sent Done false
21552 6579 Acked Flush 0 Done Sent Done false
21553 6580 Acked Flush 0 Done Sent Done false
21554 6581 Acked Write 3 Done Sent Done false
21555 6582 Acked Read 5 Done Sent Done false
21556 6583 Acked Read 5 Done Sent Done false
21557 6584 Acked Flush 0 Done Sent Done false
21558 6585 Acked Flush 0 Done Sent Done false
21559 6586 Acked Flush 0 Done Sent Done false
21560 6587 Acked Write 2 Done Sent Done false
21561 6588 Acked Read 4 Done Sent Done false
21562 6589 Acked Write 2 Done Sent Done false
21563 6590 Acked Write 5 Done Sent Done false
21564 6591 Acked Read 8 Done Sent Done false
21565 6592 Acked Read 6 Done Sent Done false
21566 6593 Acked Read 5 Done Sent Done false
21567 6594 Acked Write 8 Done Sent Done false
21568 6595 Acked Flush 0 Done Sent Done false
21569 6596 Acked Write 10 Done Sent Done false
21570 6597 Acked Write 1 Done Sent Done false
21571 6598 Acked Write 9 Done Sent Done false
21572 6599 Acked Read 4 Done Sent Done false
21573 6600 Acked Read 5 Done Sent Done false
21574 6601 Acked Read 7 Done Sent Done false
21575 6602 Acked Write 5 Done Sent Done false
21576 6603 Acked Flush 0 Done Sent Done false
21577 6604 Acked Read 1 Done Sent Done false
21578 6605 Acked Write 4 Done Sent Done false
21579 6606 Acked Write 1 Done Sent Done false
21580 6607 Acked Read 4 Done Sent Done false
21581 6608 Acked Read 10 Done Sent Done false
21582 6609 Acked Read 7 Done Sent Done false
21583 6610 Acked Write 7 Done Sent Done false
21584 6611 Acked Read 7 Done Sent Done false
21585 6612 Acked Write 9 Done Sent Done false
21586 6613 Acked Write 7 Done Sent Done false
21587 6614 Acked Read 5 Done Sent Done false
21588 6615 Acked Read 8 Done Sent Done false
21589 6616 Acked Flush 0 Done Sent Done false
21590 6617 Acked Write 10 Done Sent Done false
21591 6618 Acked Write 3 Done Sent Done false
21592 6619 Acked Read 5 Done Sent Done false
21593 6620 Acked Write 8 Done Sent Done false
21594 6621 Acked Read 9 Done Sent Done false
21595 6622 Acked Write 9 Done Sent Done false
21596 6623 Acked Read 10 Done Sent Done false
21597 6624 Acked Read 4 Done Sent Done false
21598 6625 Acked Write 10 Done Sent Done false
21599 6626 Acked Write 4 Done Sent Done false
21600 6627 Acked Write 2 Done Sent Done false
21601 6628 Acked Write 10 Done Sent Done false
21602 6629 Acked Read 5 Done Sent Done false
21603 6630 Acked Read 8 Done Sent Done false
21604 6631 Acked Read 8 Done Sent Done false
21605 6632 Acked Write 2 Done Sent Done false
21606 6633 Acked Read 4 Done Sent Done false
21607 6634 Acked Write 5 Done Sent Done false
21608 6635 Acked Write 9 Done Sent Done false
21609 6636 Acked Read 3 Done Sent Done false
21610 6637 Acked Read 2 Done Sent Done false
21611 6638 Acked Flush 0 Done Sent Done false
21612 6639 Acked Flush 0 Done Sent Done false
21613 6640 Acked Read 9 Done Sent Done false
21614 6641 Acked Write 4 Done Sent Done false
21615 6642 Acked Read 6 Done Sent Done false
21616 6643 Acked Write 4 Done Sent Done false
21617 6644 Acked Read 6 Done Sent Done false
21618 6645 Acked Read 4 Done Sent Done false
21619 6646 Acked Read 1 Done Sent Done false
21620 6647 Acked Read 6 Done Sent Done false
21621 6648 Acked Flush 0 Done Sent Done false
21622 6649 Acked Read 2 Done Sent Done false
21623 6650 Acked Write 7 Done Sent Done false
21624 6651 Acked Read 1 Done Sent Done false
21625 6652 Acked Flush 0 Done Sent Done false
21626 6653 Acked Read 8 Done Sent Done false
21627 6654 Acked Write 7 Done Sent Done false
21628 6655 Acked Flush 0 Done Sent Done false
21629 6656 Acked Write 5 Done Sent Done false
21630 6657 Acked Write 6 Done Sent Done false
21631 6658 Acked Read 1 Done Sent Done false
21632 6659 Acked Write 10 Done Sent Done false
21633 6660 Acked Read 9 Done Sent Done false
21634 6661 Acked Read 9 Done Sent Done false
21635 6662 Acked Read 5 Done Sent Done false
21636 6663 Acked Write 3 Done Sent Done false
21637 6664 Acked Write 2 Done Sent Done false
21638 6665 Acked Flush 0 Done Sent Done false
21639 6666 Acked Flush 0 Done Sent Done false
21640 6667 Acked Write 3 Done Sent Done false
21641 6668 Acked Read 9 Done Sent Done false
21642 6669 Acked Read 10 Done Sent Done false
21643 6670 Acked Read 7 Done Sent Done false
21644 6671 Acked Write 10 Done Sent Done false
21645 6672 Acked Read 1 Done Sent Done false
21646 6673 Acked Write 6 Done Sent Done false
21647 6674 Acked Write 8 Done Sent Done false
21648 6675 Acked Write 10 Done Sent Done false
21649 6676 Acked Flush 0 Done Sent Done false
21650 6677 Acked Write 5 Done Sent Done false
21651 6678 Acked Write 8 Done Sent Done false
21652 6679 Acked Read 1 Done Sent Done false
21653 6680 Acked Flush 0 Done Sent Done false
21654 6681 Acked Read 5 Done Sent Done false
21655 6682 Acked Flush 0 Done Sent Done false
21656 6683 Acked Write 5 Done Sent Done false
21657 6684 Acked Write 6 Done Sent Done false
21658 6685 Acked Write 8 Done Sent Done false
21659 6686 Acked Read 10 Done Sent Done false
21660 6687 Acked Write 7 Done Sent Done false
21661 6688 Acked Write 9 Done Sent Done false
21662 6689 Acked Flush 0 Done Sent Done false
21663 6690 Acked Read 1 Done Sent Done false
21664 6691 Acked Read 4 Done Sent Done false
21665 6692 Acked Write 5 Done Sent Done false
21666 6693 Acked Write 7 Done Sent Done false
21667 6694 Acked Read 8 Done Sent Done false
21668 6695 Acked Write 10 Done Sent Done false
21669 6696 Acked Read 8 Done Sent Done false
21670 6697 Acked Write 5 Done Sent Done false
21671 6698 Acked Write 8 Done Sent Done false
21672 6699 Acked Read 10 Done Sent Done false
21673 6700 Acked Write 4 Done Sent Done false
21674 6701 Acked Read 5 Done Sent Done false
21675 6702 Acked Write 9 Done Sent Done false
21676 6703 Acked Read 5 Done Sent Done false
21677 6704 Acked Write 9 Done Sent Done false
21678 6705 Acked Flush 0 Done Sent Done false
21679 6706 Acked Write 9 Done Sent Done false
21680 6707 Acked Read 3 Done Sent Done false
21681 6708 Acked Flush 0 Done Sent Done false
21682 6709 Acked Write 8 Done Sent Done false
21683 6710 Acked Write 2 Done Sent Done false
21684 6711 Acked Write 10 Done Sent Done false
21685 6712 Acked Read 2 Done Sent Done false
21686 6713 Acked Read 9 Done Sent Done false
21687 6714 Acked Read 5 Done Sent Done false
21688 6715 Acked Read 6 Done Sent Done false
21689 6716 Acked Read 5 Done Sent Done false
21690 6717 Acked Read 7 Done Sent Done false
21691 6718 Acked Write 6 Done Sent Done false
21692 6719 Acked Write 5 Done Sent Done false
21693 6720 Acked Read 2 Done Sent Done false
21694 6721 Acked Write 7 Done Sent Done false
21695 6722 Acked Write 6 Done Sent Done false
21696 6723 Acked Write 4 Done Sent Done false
21697 6724 Acked Read 9 Done Sent Done false
21698 6725 Acked Read 6 Done Sent Done false
21699 6726 Acked Read 8 Done Sent Done false
21700 6727 Acked Read 3 Done Sent Done false
21701 6728 Acked Write 10 Done Sent Done false
21702 6729 Acked Flush 0 Done Sent Done false
21703 6730 Acked Read 5 Done Sent Done false
21704 6731 Acked Read 6 Done Sent Done false
21705 6732 Acked Write 10 Done Sent Done false
21706 6733 Acked Read 10 Done Sent Done false
21707 6734 Acked Read 4 Done Sent Done false
21708 STATES DS:0 DS:1 DS:2 TOTAL
21709 Sent 0 301 0 301
21710 Done 301 0 301 602
21711 Skipped 0 0 0 0
21712 Error 0 0 0 0
21713 Last Flush: Some(JobId(6729)) Some(JobId(6433)) Some(JobId(6729))
21714 Downstairs last five completed: 6433 6432 6431 6430 6429
21715 Upstairs last five completed: 6734 6733 6732 6731 6730
21716 CLIENT: Up:0 ds:301 act:2
217172025-02-21T06:50:08.521ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
217182025-02-21T06:50:09.101ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21719 [8820] got start action
21720 [8820] I am going to restart
21721 Starting downstairs at port 8820
21722 Make output file at "/tmp/dsc/downstairs-8820.txt"
21723 [8820][1] reports Starting
21724 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1225)
21725 [8820][1] reports Running
217262025-02-21T06:50:10.103ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
217272025-02-21T06:50:11.105ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
21728 ----------------------------------------------------------------
21729 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21730 States: Active Offline Active
21731 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21732 6434 Acked Read 4 Done Sent Done false
21733 6435 Acked Read 7 Done Sent Done false
21734 6436 Acked Write 8 Done Sent Done false
21735 6437 Acked Read 8 Done Sent Done false
21736 6438 Acked Read 5 Done Sent Done false
21737 6439 Acked Read 10 Done Sent Done false
21738 6440 Acked Read 8 Done Sent Done false
21739 6441 Acked Read 9 Done Sent Done false
21740 6442 Acked Read 5 Done Sent Done false
21741 6443 Acked Flush 0 Done Sent Done false
21742 6444 Acked Read 1 Done Sent Done false
21743 6445 Acked Read 7 Done Sent Done false
21744 6446 Acked Read 5 Done Sent Done false
21745 6447 Acked Write 3 Done Sent Done false
21746 6448 Acked Read 3 Done Sent Done false
21747 6449 Acked Flush 0 Done Sent Done false
21748 6450 Acked Write 3 Done Sent Done false
21749 6451 Acked Read 5 Done Sent Done false
21750 6452 Acked Read 7 Done Sent Done false
21751 6453 Acked Read 4 Done Sent Done false
21752 6454 Acked Read 1 Done Sent Done false
21753 6455 Acked Write 1 Done Sent Done false
21754 6456 Acked Write 5 Done Sent Done false
21755 6457 Acked Read 3 Done Sent Done false
21756 6458 Acked Read 9 Done Sent Done false
21757 6459 Acked Write 6 Done Sent Done false
21758 6460 Acked Read 4 Done Sent Done false
21759 6461 Acked Read 3 Done Sent Done false
21760 6462 Acked Read 7 Done Sent Done false
21761 6463 Acked Write 10 Done Sent Done false
21762 6464 Acked Write 3 Done Sent Done false
21763 6465 Acked Read 4 Done Sent Done false
21764 6466 Acked Write 1 Done Sent Done false
21765 6467 Acked Read 10 Done Sent Done false
21766 6468 Acked Write 1 Done Sent Done false
21767 6469 Acked Write 7 Done Sent Done false
21768 6470 Acked Flush 0 Done Sent Done false
21769 6471 Acked Read 10 Done Sent Done false
21770 6472 Acked Read 8 Done Sent Done false
21771 6473 Acked Flush 0 Done Sent Done false
21772 6474 Acked Read 7 Done Sent Done false
21773 6475 Acked Read 10 Done Sent Done false
21774 6476 Acked Write 3 Done Sent Done false
21775 6477 Acked Write 1 Done Sent Done false
21776 6478 Acked Write 5 Done Sent Done false
21777 6479 Acked Write 5 Done Sent Done false
21778 6480 Acked Read 2 Done Sent Done false
21779 6481 Acked Read 4 Done Sent Done false
21780 6482 Acked Read 8 Done Sent Done false
21781 6483 Acked Write 4 Done Sent Done false
21782 6484 Acked Write 4 Done Sent Done false
21783 6485 Acked Read 2 Done Sent Done false
21784 6486 Acked Write 9 Done Sent Done false
21785 6487 Acked Read 6 Done Sent Done false
21786 6488 Acked Read 10 Done Sent Done false
21787 6489 Acked Read 9 Done Sent Done false
21788 6490 Acked Read 9 Done Sent Done false
21789 6491 Acked Read 10 Done Sent Done false
21790 6492 Acked Read 7 Done Sent Done false
21791 6493 Acked Write 3 Done Sent Done false
21792 6494 Acked Read 1 Done Sent Done false
21793 6495 Acked Read 7 Done Sent Done false
21794 6496 Acked Read 9 Done Sent Done false
21795 6497 Acked Flush 0 Done Sent Done false
21796 6498 Acked Read 5 Done Sent Done false
21797 6499 Acked Flush 0 Done Sent Done false
21798 6500 Acked Write 8 Done Sent Done false
21799 6501 Acked Read 1 Done Sent Done false
21800 6502 Acked Read 7 Done Sent Done false
21801 6503 Acked Flush 0 Done Sent Done false
21802 6504 Acked Write 10 Done Sent Done false
21803 6505 Acked Read 1 Done Sent Done false
21804 6506 Acked Read 6 Done Sent Done false
21805 6507 Acked Write 7 Done Sent Done false
21806 6508 Acked Write 10 Done Sent Done false
21807 6509 Acked Write 2 Done Sent Done false
21808 6510 Acked Read 7 Done Sent Done false
21809 6511 Acked Write 6 Done Sent Done false
21810 6512 Acked Flush 0 Done Sent Done false
21811 6513 Acked Read 2 Done Sent Done false
21812 6514 Acked Flush 0 Done Sent Done false
21813 6515 Acked Write 6 Done Sent Done false
21814 6516 Acked Write 3 Done Sent Done false
21815 6517 Acked Write 3 Done Sent Done false
21816 6518 Acked Read 10 Done Sent Done false
21817 6519 Acked Flush 0 Done Sent Done false
21818 6520 Acked Write 8 Done Sent Done false
21819 6521 Acked Read 8 Done Sent Done false
21820 6522 Acked Write 1 Done Sent Done false
21821 6523 Acked Read 9 Done Sent Done false
21822 6524 Acked Read 2 Done Sent Done false
21823 6525 Acked Flush 0 Done Sent Done false
21824 6526 Acked Write 3 Done Sent Done false
21825 6527 Acked Write 6 Done Sent Done false
21826 6528 Acked Read 9 Done Sent Done false
21827 6529 Acked Read 1 Done Sent Done false
21828 6530 Acked Read 3 Done Sent Done false
21829 6531 Acked Read 3 Done Sent Done false
21830 6532 Acked Write 3 Done Sent Done false
21831 6533 Acked Write 7 Done Sent Done false
21832 6534 Acked Read 3 Done Sent Done false
21833 6535 Acked Write 3 Done Sent Done false
21834 6536 Acked Read 7 Done Sent Done false
21835 6537 Acked Write 5 Done Sent Done false
21836 6538 Acked Write 7 Done Sent Done false
21837 6539 Acked Read 9 Done Sent Done false
21838 6540 Acked Flush 0 Done Sent Done false
21839 6541 Acked Write 1 Done Sent Done false
21840 6542 Acked Read 7 Done Sent Done false
21841 6543 Acked Write 6 Done Sent Done false
21842 6544 Acked Write 5 Done Sent Done false
21843 6545 Acked Read 9 Done Sent Done false
21844 6546 Acked Read 2 Done Sent Done false
21845 6547 Acked Write 10 Done Sent Done false
21846 6548 Acked Flush 0 Done Sent Done false
21847 6549 Acked Read 5 Done Sent Done false
21848 6550 Acked Flush 0 Done Sent Done false
21849 6551 Acked Flush 0 Done Sent Done false
21850 6552 Acked Read 4 Done Sent Done false
21851 6553 Acked Write 3 Done Sent Done false
21852 6554 Acked Flush 0 Done Sent Done false
21853 6555 Acked Write 7 Done Sent Done false
21854 6556 Acked Write 3 Done Sent Done false
21855 6557 Acked Read 4 Done Sent Done false
21856 6558 Acked Read 3 Done Sent Done false
21857 6559 Acked Read 3 Done Sent Done false
21858 6560 Acked Write 4 Done Sent Done false
21859 6561 Acked Write 1 Done Sent Done false
21860 6562 Acked Write 9 Done Sent Done false
21861 6563 Acked Write 3 Done Sent Done false
21862 6564 Acked Read 3 Done Sent Done false
21863 6565 Acked Read 10 Done Sent Done false
21864 6566 Acked Write 10 Done Sent Done false
21865 6567 Acked Flush 0 Done Sent Done false
21866 6568 Acked Write 10 Done Sent Done false
21867 6569 Acked Read 9 Done Sent Done false
21868 6570 Acked Read 3 Done Sent Done false
21869 6571 Acked Flush 0 Done Sent Done false
21870 6572 Acked Write 10 Done Sent Done false
21871 6573 Acked Read 7 Done Sent Done false
21872 6574 Acked Read 2 Done Sent Done false
21873 6575 Acked Flush 0 Done Sent Done false
21874 6576 Acked Write 4 Done Sent Done false
21875 6577 Acked Read 2 Done Sent Done false
21876 6578 Acked Write 5 Done Sent Done false
21877 6579 Acked Flush 0 Done Sent Done false
21878 6580 Acked Flush 0 Done Sent Done false
21879 6581 Acked Write 3 Done Sent Done false
21880 6582 Acked Read 5 Done Sent Done false
21881 6583 Acked Read 5 Done Sent Done false
21882 6584 Acked Flush 0 Done Sent Done false
21883 6585 Acked Flush 0 Done Sent Done false
21884 6586 Acked Flush 0 Done Sent Done false
21885 6587 Acked Write 2 Done Sent Done false
21886 6588 Acked Read 4 Done Sent Done false
21887 6589 Acked Write 2 Done Sent Done false
21888 6590 Acked Write 5 Done Sent Done false
21889 6591 Acked Read 8 Done Sent Done false
21890 6592 Acked Read 6 Done Sent Done false
21891 6593 Acked Read 5 Done Sent Done false
21892 6594 Acked Write 8 Done Sent Done false
21893 6595 Acked Flush 0 Done Sent Done false
21894 6596 Acked Write 10 Done Sent Done false
21895 6597 Acked Write 1 Done Sent Done false
21896 6598 Acked Write 9 Done Sent Done false
21897 6599 Acked Read 4 Done Sent Done false
21898 6600 Acked Read 5 Done Sent Done false
21899 6601 Acked Read 7 Done Sent Done false
21900 6602 Acked Write 5 Done Sent Done false
21901 6603 Acked Flush 0 Done Sent Done false
21902 6604 Acked Read 1 Done Sent Done false
21903 6605 Acked Write 4 Done Sent Done false
21904 6606 Acked Write 1 Done Sent Done false
21905 6607 Acked Read 4 Done Sent Done false
21906 6608 Acked Read 10 Done Sent Done false
21907 6609 Acked Read 7 Done Sent Done false
21908 6610 Acked Write 7 Done Sent Done false
21909 6611 Acked Read 7 Done Sent Done false
21910 6612 Acked Write 9 Done Sent Done false
21911 6613 Acked Write 7 Done Sent Done false
21912 6614 Acked Read 5 Done Sent Done false
21913 6615 Acked Read 8 Done Sent Done false
21914 6616 Acked Flush 0 Done Sent Done false
21915 6617 Acked Write 10 Done Sent Done false
21916 6618 Acked Write 3 Done Sent Done false
21917 6619 Acked Read 5 Done Sent Done false
21918 6620 Acked Write 8 Done Sent Done false
21919 6621 Acked Read 9 Done Sent Done false
21920 6622 Acked Write 9 Done Sent Done false
21921 6623 Acked Read 10 Done Sent Done false
21922 6624 Acked Read 4 Done Sent Done false
21923 6625 Acked Write 10 Done Sent Done false
21924 6626 Acked Write 4 Done Sent Done false
21925 6627 Acked Write 2 Done Sent Done false
21926 6628 Acked Write 10 Done Sent Done false
21927 6629 Acked Read 5 Done Sent Done false
21928 6630 Acked Read 8 Done Sent Done false
21929 6631 Acked Read 8 Done Sent Done false
21930 6632 Acked Write 2 Done Sent Done false
21931 6633 Acked Read 4 Done Sent Done false
21932 6634 Acked Write 5 Done Sent Done false
21933 6635 Acked Write 9 Done Sent Done false
21934 6636 Acked Read 3 Done Sent Done false
21935 6637 Acked Read 2 Done Sent Done false
21936 6638 Acked Flush 0 Done Sent Done false
21937 6639 Acked Flush 0 Done Sent Done false
21938 6640 Acked Read 9 Done Sent Done false
21939 6641 Acked Write 4 Done Sent Done false
21940 6642 Acked Read 6 Done Sent Done false
21941 6643 Acked Write 4 Done Sent Done false
21942 6644 Acked Read 6 Done Sent Done false
21943 6645 Acked Read 4 Done Sent Done false
21944 6646 Acked Read 1 Done Sent Done false
21945 6647 Acked Read 6 Done Sent Done false
21946 6648 Acked Flush 0 Done Sent Done false
21947 6649 Acked Read 2 Done Sent Done false
21948 6650 Acked Write 7 Done Sent Done false
21949 6651 Acked Read 1 Done Sent Done false
21950 6652 Acked Flush 0 Done Sent Done false
21951 6653 Acked Read 8 Done Sent Done false
21952 6654 Acked Write 7 Done Sent Done false
21953 6655 Acked Flush 0 Done Sent Done false
21954 6656 Acked Write 5 Done Sent Done false
21955 6657 Acked Write 6 Done Sent Done false
21956 6658 Acked Read 1 Done Sent Done false
21957 6659 Acked Write 10 Done Sent Done false
21958 6660 Acked Read 9 Done Sent Done false
21959 6661 Acked Read 9 Done Sent Done false
21960 6662 Acked Read 5 Done Sent Done false
21961 6663 Acked Write 3 Done Sent Done false
21962 6664 Acked Write 2 Done Sent Done false
21963 6665 Acked Flush 0 Done Sent Done false
21964 6666 Acked Flush 0 Done Sent Done false
21965 6667 Acked Write 3 Done Sent Done false
21966 6668 Acked Read 9 Done Sent Done false
21967 6669 Acked Read 10 Done Sent Done false
21968 6670 Acked Read 7 Done Sent Done false
21969 6671 Acked Write 10 Done Sent Done false
21970 6672 Acked Read 1 Done Sent Done false
21971 6673 Acked Write 6 Done Sent Done false
21972 6674 Acked Write 8 Done Sent Done false
21973 6675 Acked Write 10 Done Sent Done false
21974 6676 Acked Flush 0 Done Sent Done false
21975 6677 Acked Write 5 Done Sent Done false
21976 6678 Acked Write 8 Done Sent Done false
21977 6679 Acked Read 1 Done Sent Done false
21978 6680 Acked Flush 0 Done Sent Done false
21979 6681 Acked Read 5 Done Sent Done false
21980 6682 Acked Flush 0 Done Sent Done false
21981 6683 Acked Write 5 Done Sent Done false
21982 6684 Acked Write 6 Done Sent Done false
21983 6685 Acked Write 8 Done Sent Done false
21984 6686 Acked Read 10 Done Sent Done false
21985 6687 Acked Write 7 Done Sent Done false
21986 6688 Acked Write 9 Done Sent Done false
21987 6689 Acked Flush 0 Done Sent Done false
21988 6690 Acked Read 1 Done Sent Done false
21989 6691 Acked Read 4 Done Sent Done false
21990 6692 Acked Write 5 Done Sent Done false
21991 6693 Acked Write 7 Done Sent Done false
21992 6694 Acked Read 8 Done Sent Done false
21993 6695 Acked Write 10 Done Sent Done false
21994 6696 Acked Read 8 Done Sent Done false
21995 6697 Acked Write 5 Done Sent Done false
21996 6698 Acked Write 8 Done Sent Done false
21997 6699 Acked Read 10 Done Sent Done false
21998 6700 Acked Write 4 Done Sent Done false
21999 6701 Acked Read 5 Done Sent Done false
22000 6702 Acked Write 9 Done Sent Done false
22001 6703 Acked Read 5 Done Sent Done false
22002 6704 Acked Write 9 Done Sent Done false
22003 6705 Acked Flush 0 Done Sent Done false
22004 6706 Acked Write 9 Done Sent Done false
22005 6707 Acked Read 3 Done Sent Done false
22006 6708 Acked Flush 0 Done Sent Done false
22007 6709 Acked Write 8 Done Sent Done false
22008 6710 Acked Write 2 Done Sent Done false
22009 6711 Acked Write 10 Done Sent Done false
22010 6712 Acked Read 2 Done Sent Done false
22011 6713 Acked Read 9 Done Sent Done false
22012 6714 Acked Read 5 Done Sent Done false
22013 6715 Acked Read 6 Done Sent Done false
22014 6716 Acked Read 5 Done Sent Done false
22015 6717 Acked Read 7 Done Sent Done false
22016 6718 Acked Write 6 Done Sent Done false
22017 6719 Acked Write 5 Done Sent Done false
22018 6720 Acked Read 2 Done Sent Done false
22019 6721 Acked Write 7 Done Sent Done false
22020 6722 Acked Write 6 Done Sent Done false
22021 6723 Acked Write 4 Done Sent Done false
22022 6724 Acked Read 9 Done Sent Done false
22023 6725 Acked Read 6 Done Sent Done false
22024 6726 Acked Read 8 Done Sent Done false
22025 6727 Acked Read 3 Done Sent Done false
22026 6728 Acked Write 10 Done Sent Done false
22027 6729 Acked Flush 0 Done Sent Done false
22028 6730 Acked Read 5 Done Sent Done false
22029 6731 Acked Read 6 Done Sent Done false
22030 6732 Acked Write 10 Done Sent Done false
22031 6733 Acked Read 10 Done Sent Done false
22032 6734 Acked Read 4 Done Sent Done false
22033 STATES DS:0 DS:1 DS:2 TOTAL
22034 Sent 0 301 0 301
22035 Done 301 0 301 602
22036 Skipped 0 0 0 0
22037 Error 0 0 0 0
22038 Last Flush: Some(JobId(6729)) Some(JobId(6433)) Some(JobId(6729))
22039 Downstairs last five completed: 6433 6432 6431 6430 6429
22040 Upstairs last five completed: 6734 6733 6732 6731 6730
22041 CLIENT: Up:0 ds:301 act:2
220422025-02-21T06:50:12.523ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
220432025-02-21T06:50:13.109ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
220442025-02-21T06:50:13.774ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
220452025-02-21T06:50:13.774ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60894) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
220462025-02-21T06:50:13.775ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
220472025-02-21T06:50:13.775ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
220482025-02-21T06:50:13.775ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
220492025-02-21T06:50:13.775ZINFOcrucible: send last flush ID to this DS: Some(JobId(6433)) = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
220502025-02-21T06:50:13.775ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
220512025-02-21T06:50:13.775ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(6433)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
220522025-02-21T06:50:13.776ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(6433)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
22053 ----------------------------------------------------------------
22054 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22055 Downstairs last five completed: 6735 6734 6733 6732 6731
22056 Upstairs last five completed: 6735 6734 6733 6732 6731
22057 CLIENT: Up:0 ds:0 act:3
22058 Replay: All jobs finished
22059 Main task has work to do, go find it
22060 got dsc Stop(0)
22061 stop 0
22062 Feb 21 06:50:16.024 [8810] Got stop action so:false kr:false
22063 INFO request completed, latency_us: 373, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 98b32ab9-3e0a-4c87-b55f-45dff47c1aab, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
220642025-02-21T06:50:16.025ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
22065 Feb 21 06:50:16.025 INFO request completed, latency_us: 135, response_code: 200, uri: /state/cid/0, method: GET, req_id{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:50:16.025542188Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
22066 : 987c8d51-f28d-43a3-a2a2-d83c441daea6, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:50:16.025575533Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
22067 1:9998
220682025-02-21T06:50:16.025ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
22069 [8810] Exited with: ExitStatus(unix_wait_status(9))
22070 [8810][0] reports Exit
22071 Feb 21 06:50:20.028 INFO request completed, latency_us: 245, response_code: 200, uri: /state/cid/0, method: GET, req_id: 3ea9832e-212f-476f-9b08-bdfe68688185, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
22072 001/300 Write block 4649 len 32768 data: 8 8 9 8 6 6 6 5
22073 002/300 Read block 249 len 8192
22074 003/300 Read block 2558 len 12288
22075 004/300 Read block 3924 len 8192
22076 005/300 Read block 1037 len 24576
22077 006/300 Read block 535 len 40960
22078 007/300 Write block 608 len 32768 data: 4 4 4 3 4 2 2 2
22079 008/300 Write block 2630 len 12288 data: 6 6 6
22080 009/300 Read block 2516 len 36864
22081 010/300 Write block 829 len 20480 data: 3 3 2 2 2
22082 011/300 Write block 2598 len 36864 data: 3 2 4 4 3 4 6 7 6
22083 012/300 Read block 4251 len 24576
22084 013/300 Read block 3726 len 36864
22085 014/300 Write block 4652 len 12288 data: 9 7 7
22086 015/300 Write block 3507 len 24576 data: 5 8 8 8 8 8
22087 016/300 Write block 2071 len 4096 data: 8
22088 017/300 Read block 269 len 32768
22089 018/300 Read block 1511 len 28672
22090 019/300 Read block 604 len 4096
22091 020/300 Write block 2620 len 16384 data: 7 6 4 4
22092 021/300 Write block 1475 len 32768 data: 4 4 5 5 5 5 5 5
22093 022/300 Write block 3136 len 36864 data: 12 12 11 11 10 8 6 6 6
22094 023/300 Flush
22095 024/300 Read block 3457 len 24576
22096 025/300 Read block 4761 len 28672
22097 026/300 Write block 1679 len 8192 data: 4 5
22098 027/300 Flush
22099 028/300 Read block 43 len 8192
22100 029/300 Write block 4476 len 12288 data: 7 6 5
22101 030/300 Read block 3904 len 32768
22102 031/300 Write block 1744 len 40960 data: 5 4 5 6 7 6 5 4 5 5
22103 032/300 Read block 570 len 8192
22104 033/300 Write block 4798 len 20480 data: 5 5 5 5 5
22105 034/300 Flush
22106 035/300 Write block 3566 len 8192 data: 5 6
22107 036/300 Read block 3881 len 40960
22108 037/300 Write block 307 len 12288 data: 3 3 3
22109 038/300 Write block 3130 len 16384 data: 7 7 9 7
22110 039/300 Write block 272 len 12288 data: 3 3 4
22111 040/300 Read block 1016 len 8192
22112 041/300 Write block 1493 len 20480 data: 5 6 6 6 6
22113 042/300 Read block 1454 len 4096
22114 043/300 Read block 4649 len 24576
22115 044/300 Write block 524 len 20480 data: 7 6 4 4 4
22116 045/300 Read block 2891 len 28672
22117 046/300 Read block 2818 len 40960
22118 047/300 Read block 2330 len 8192
22119 048/300 Read block 170 len 32768
22120 049/300 Write block 3719 len 8192 data: 3 3
22121 050/300 Read block 3199 len 24576
22122 051/300 Read block 1608 len 12288
22123 052/300 Write block 1813 len 32768 data: 8 7 8 7 7 5 8 8
22124 053/300 Write block 4608 len 24576 data: 5 4 5 4 5 4
22125 054/300 Flush
22126 055/300 Write block 3509 len 24576 data: 9 9 9 9 6 5
22127 056/300 Read block 2688 len 24576
22128 057/300 Write block 1092 len 28672 data: 4 4 4 5 4 4 4
22129 058/300 Write block 4161 len 16384 data: 2 2 2 2
22130 059/300 Read block 669 len 40960
22131 060/300 Read block 574 len 8192
22132 061/300 Read block 2542 len 36864
22133 062/300 Write block 1145 len 40960 data: 4 4 4 4 4 3 3 3 3 2
22134 063/300 Read block 4502 len 16384
22135 064/300 Read block 2440 len 4096
22136 065/300 Read block 3186 len 8192
22137 066/300 Write block 3968 len 20480 data: 5 4 4 5 4
22138 067/300 Write block 388 len 40960 data: 4 4 5 4 3 3 4 4 4 3
22139 068/300 Flush
22140 069/300 Write block 339 len 4096 data: 4
22141 070/300 Read block 540 len 20480
22142 071/300 Read block 2070 len 4096
22143 072/300 Read block 3197 len 36864
22144 073/300 Write block 3352 len 32768 data: 5 5 6 5 3 2 2 2
22145 074/300 Read block 2599 len 36864
22146 075/300 Write block 1506 len 12288 data: 5 4 4
22147 076/300 Flush
22148 077/300 Read block 937 len 36864
22149 078/300 Write block 32 len 28672 data: 4 4 4 4 4 3 4
22150 079/300 Read block 4691 len 40960
22151 080/300 Read block 1922 len 20480
22152 081/300 Read block 2217 len 16384
22153 082/300 Write block 2972 len 40960 data: 2 3 3 4 6 6 5 4 4 4
22154 083/300 Write block 721 len 32768 data: 5 4 6 6 5 4 4 5
22155 084/300 Read block 1174 len 32768
22156 085/300 Read block 3716 len 16384
22157 086/300 Flush
22158 087/300 Read block 226 len 36864
22159 088/300 Read block 2756 len 40960
22160 089/300 Write block 3124 len 12288 data: 4 5 4
22161 090/300 Write block 613 len 40960 data: 3 3 3 3 3 3 2 2 3 3
22162 091/300 Read block 919 len 36864
22163 092/300 Flush
22164 093/300 Flush
22165 094/300 Read block 3388 len 36864
22166 095/300 Read block 1166 len 28672
22167 096/300 Read block 409 len 28672
22168 097/300 Read block 597 len 36864
22169 098/300 Read block 3677 len 32768
22170 099/300 Write block 736 len 12288 data: 5 8 9
22171 100/300 Write block 1683 len 24576 data: 5 6 6 6 4 4
22172 101/300 Read block 893 len 28672
22173 102/300 Write block 77 len 24576 data: 6 6 6 5 5 5
22174 103/300 Write block 2358 len 32768 data: 3 3 4 4 4 4 3 3
22175 104/300 Read block 3017 len 40960
22176 105/300 Read block 2947 len 28672
22177 106/300 Write block 3485 len 8192 data: 4 4
22178 107/300 Flush
22179 108/300 Read block 3106 len 20480
22180 109/300 Flush
22181 110/300 Read block 4288 len 20480
22182 111/300 Read block 3380 len 4096
22183 112/300 Read block 2604 len 36864
22184 113/300 Read block 3793 len 12288
22185 114/300 Read block 4671 len 4096
22186 115/300 Flush
22187 116/300 Write block 2667 len 4096 data: 5
22188 117/300 Read block 2736 len 4096
22189 118/300 Read block 1985 len 24576
22190 119/300 Write block 582 len 36864 data: 5 5 4 5 5 5 4 4 4
22191 120/300 Read block 4537 len 20480
22192 121/300 Read block 2370 len 40960
22193 122/300 Write block 591 len 4096 data: 5
22194 123/300 Write block 171 len 24576 data: 4 5 6 5 5 4
22195 124/300 Write block 3236 len 4096 data: 3
22196 125/300 Read block 510 len 16384
22197 126/300 Flush
22198 127/300 Read block 4481 len 8192
22199 128/300 Write block 3278 len 36864 data: 3 3 3 3 3 5 5 5 5
22200 129/300 Flush
22201 130/300 Read block 1139 len 28672
22202 131/300 Read block 4280 len 16384
22203 132/300 Read block 3045 len 16384
22204 133/300 Write block 1992 len 4096 data: 5
22205 134/300 Flush
22206 135/300 Write block 1186 len 40960 data: 3 5 5 5 5 6 5 4 3 4
22207 136/300 Flush
22208 137/300 Write block 3500 len 40960 data: 5 4 3 3 3 3 4 6 9 10
22209 138/300 Write block 3973 len 4096 data: 3
22210 139/300 Read block 3790 len 12288
22211 140/300 Write block 4732 len 40960 data: 7 7 4 4 2 2 3 4 5 5
22212 141/300 Write block 4121 len 40960 data: 7 8 7 5 5 4 4 3 3 3
22213 142/300 Read block 896 len 28672
22214 143/300 Write block 1979 len 24576 data: 5 5 6 3 3 5
22215 144/300 Write block 1135 len 28672 data: 5 5 5 5 5 3 3
22216 145/300 Write block 3320 len 28672 data: 3 4 4 4 4 4 3
22217 146/300 Read block 138 len 4096
22218 147/300 Write block 163 len 28672 data: 3 3 3 3 3 4 4
22219 148/300 Read block 628 len 32768
22220 149/300 Read block 4195 len 12288
22221 150/300 Read block 2243 len 8192
22222 151/300 Write block 4828 len 32768 data: 6 6 7 7 6 6 7 5
22223 152/300 Flush
22224 153/300 Read block 2922 len 40960
22225 154/300 Flush
22226 155/300 Write block 3173 len 16384 data: 6 6 7 6
22227 156/300 Write block 3238 len 8192 data: 4 5
22228 157/300 Read block 4025 len 8192
22229 158/300 Write block 2909 len 24576 data: 6 6 5 6 3 3
22230 159/300 Write block 1528 len 12288 data: 4 4 6
22231 160/300 Read block 4746 len 28672
22232 161/300 Write block 3713 len 12288 data: 4 4 4
22233 162/300 Read block 2350 len 40960
22234 163/300 Read block 1618 len 32768
22235 164/300 Read block 665 len 4096
22236 165/300 Flush
22237 166/300 Write block 4588 len 28672 data: 4 4 4 4 4 2 2
22238 167/300 Flush
22239 168/300 Read block 3550 len 28672
22240 169/300 Write block 3539 len 12288 data: 2 3 4
22241 170/300 Read block 731 len 40960
22242 171/300 Read block 4500 len 8192
22243 172/300 Write block 3810 len 32768 data: 5 5 6 6 6 8 10 10
22244 173/300 Read block 2524 len 32768
22245 174/300 Write block 3069 len 28672 data: 3 4 3 3 3 3 4
22246 175/300 Read block 4292 len 32768
22247 176/300 Write block 3799 len 40960 data: 6 6 3 3 3 3 3 3 3 3
22248 177/300 Read block 3564 len 12288
22249 178/300 Read block 2775 len 32768
22250 179/300 Read block 4968 len 4096
22251 180/300 Read block 4899 len 32768
22252 181/300 Write block 4583 len 24576 data: 3 3 3 5 4 5
22253 182/300 Flush
22254 183/300 Read block 4786 len 12288
22255 184/300 Read block 4307 len 36864
22256 185/300 Read block 4995 len 12288
22257 186/300 Write block 2129 len 28672 data: 4 4 4 3 4 4 4
22258 187/300 Write block 4144 len 8192 data: 5 5
22259 188/300 Write block 3592 len 20480 data: 5 4 3 4 4
22260 189/300 Write block 650 len 36864 data: 4 4 4 3 3 3 3 4 4
22261 190/300 Flush
22262 191/300 Read block 3759 len 12288
22263 192/300 Read block 4569 len 24576
22264 193/300 Read block 3208 len 36864
22265 194/300 Read block 1024 len 28672
22266 195/300 Write block 2030 len 28672 data: 2 2 3 3 4 5 6
22267 196/300 Read block 2441 len 20480
22268 197/300 Read block 145 len 12288
22269 198/300 Read block 3754 len 12288
22270 199/300 Write block 3058 len 24576 data: 3 3 3 4 5 5
22271 200/300 Write block 225 len 4096 data: 4
22272 201/300 Read block 2653 len 36864
22273 202/300 Read block 2731 len 12288
22274 203/300 Flush
22275 204/300 Read block 2706 len 20480
22276 205/300 Flush
22277 206/300 Read block 409 len 4096
22278 207/300 Write block 3378 len 32768 data: 2 2 2 3 3 3 2 3
22279 208/300 Write block 2666 len 20480 data: 5 6 7 7 6
22280 209/300 Read block 4301 len 16384
22281 210/300 Write block 945 len 4096 data: 5
22282 211/300 Read block 3835 len 4096
22283 212/300 Write block 4503 len 40960 data: 3 3 3 3 3 4 4 4 4 3
22284 213/300 Read block 1943 len 32768
22285 214/300 Write block 4900 len 36864 data: 6 8 7 7 5 6 6 4 4
22286 215/300 Write block 4367 len 12288 data: 3 4 4
22287 216/300 Write block 2850 len 4096 data: 8
22288 217/300 Read block 272 len 40960
22289 218/300 Read block 587 len 40960
22290 219/300 Read block 1247 len 36864
22291 220/300 Read block 1926 len 8192
22292 221/300 Write block 2361 len 20480 data: 5 5 5 4 4
22293 222/300 Write block 1198 len 8192 data: 4 5
22294 223/300 Flush
22295 224/300 Flush
22296 225/300 Read block 2031 len 8192
22297 226/300 Flush
22298 227/300 Read block 1339 len 16384
22299 228/300 Read block 2002 len 40960
22300 229/300 Write block 3464 len 24576 data: 3 3 3 4 4 3
22301 230/300 Read block 1526 len 8192
22302 231/300 Read block 3524 len 8192
22303 232/300 Write block 4266 len 28672 data: 3 3 3 3 3 5 4
22304 233/300 Read block 756 len 4096
22305 234/300 Read block 3790 len 40960
22306 235/300 Write block 1036 len 4096 data: 4
22307 236/300 Write block 314 len 28672 data: 6 6 6 7 5 4 3
22308 237/300 Write block 1377 len 28672 data: 5 5 5 5 3 4 2
22309 238/300 Flush
22310 239/300 Read block 4102 len 40960
22311 240/300 Write block 2670 len 16384 data: 7 7 6 5
22312 241/300 Flush
22313 242/300 Write block 4789 len 12288 data: 4 5 4
22314 243/300 Flush
22315 244/300 Read block 1525 len 28672
22316 245/300 Read block 4743 len 40960
22317 246/300 Write block 1463 len 32768 data: 7 7 7 6 4 4 3 3
22318 247/300 Write block 45 len 20480 data: 4 4 2 2 2
22319 248/300 Write block 3798 len 28672 data: 7 7 7 4 4 4 4
22320 249/300 Read block 2579 len 12288
22321 250/300 Write block 1569 len 20480 data: 3 3 3 2 2
22322 251/300 Write block 1551 len 36864 data: 4 3 3 3 3 5 4 3 3
22323 252/300 Read block 4354 len 36864
22324 253/300 Write block 3479 len 32768 data: 7 7 6 5 5 3 5 5
22325 254/300 Write block 443 len 12288 data: 5 5 6
22326 255/300 Read block 1452 len 28672
22327 256/300 Write block 399 len 8192 data: 4 3
22328 257/300 Read block 2184 len 16384
22329 258/300 Write block 2405 len 4096 data: 3
22330 259/300 Read block 4550 len 40960
22331 260/300 Write block 4058 len 28672 data: 7 6 6 6 3 3 3
22332 261/300 Write block 473 len 36864 data: 6 5 5 6 5 4 3 2 2
22333 262/300 Read block 4220 len 28672
22334 263/300 Read block 283 len 32768
22335 264/300 Read block 4081 len 20480
22336 265/300 Read block 4167 len 36864
22337 266/300 Write block 4963 len 16384 data: 6 5 4 4
22338 267/300 Flush
22339 268/300 Read block 1445 len 28672
22340 269/300 Write block 2219 len 4096 data: 6
22341 270/300 Read block 4686 len 20480
22342 271/300 Flush
22343 272/300 Read block 141 len 40960
22344 273/300 Read block 1759 len 32768
22345 274/300 Write block 1970 len 16384 data: 4 4 4 3
22346 275/300 Read block 237 len 12288
22347 276/300 Write block 4276 len 40960 data: 4 4 2 3 4 4 4 6 6 5
22348 277/300 Flush
22349 278/300 Flush
22350 279/300 Write block 24 len 32768 data: 3 4 3 3 5 5 6 5
22351 280/300 Read block 1155 len 12288
22352 281/300 Read block 4171 len 8192
22353 282/300 Write block 3988 len 4096 data: 5
22354 283/300 Flush
22355 284/300 Read block 2845 len 32768
22356 285/300 Read block 1325 len 24576
22357 286/300 Write block 1464 len 4096 data: 8
22358 287/300 Flush
22359 288/300 Read block 2468 len 32768
22360 289/300 Write block 3290 len 24576 data: 4 4 3 3 3 2
22361 290/300 Flush
22362 291/300 Read block 3798 len 28672
22363 292/300 Read block 3875 len 24576
22364 293/300 Read block 3718 len 40960
22365 294/300 Write block 1642 len 32768 data: 2 3 3 3 2 2 2 2
22366 295/300 Read block 1802 len 40960
22367 296/300 Write block 2240 len 16384 data: 5 3 4 4
22368 297/300 Read block 4830 len 32768
22369 298/300 Read block 2276 len 12288
22370 299/300 Read block 2626 len 20480
22371 300/300 Write block 129 len 28672 data: 3 3 4 4 4 4 4
22372 301/300 Read block 445 len 16384
22373 Main task has work to do, go find it
22374 got dsc Start(0)
22375 start 0
22376 Feb 21 06:50:20.274 INFO request completed, latency_us: 240, response_code: 204, uri: /start/cid/0, method: POST, req_id: 77880794-1a9d-4421-894c-5c4949b72d5e, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
22377 Replay: started 0, returned:Ok(())
22378 ----------------------------------------------------------------
22379 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22380 States: Offline Active Active
22381 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22382 6736 Acked Write 8 Sent Done Done false
22383 6737 Acked Read 2 Sent Done Done false
22384 6738 Acked Read 3 Sent Done Done false
22385 6739 Acked Read 2 Sent Done Done false
22386 6740 Acked Read 6 Sent Done Done false
22387 6741 Acked Read 10 Sent Done Done false
22388 6742 Acked Write 8 Sent Done Done false
22389 6743 Acked Write 3 Sent Done Done false
22390 6744 Acked Read 9 Sent Done Done false
22391 6745 Acked Write 5 Sent Done Done false
22392 6746 Acked Write 9 Sent Done Done false
22393 6747 Acked Read 6 Sent Done Done false
22394 6748 Acked Read 9 Sent Done Done false
22395 6749 Acked Write 3 Sent Done Done false
22396 6750 Acked Write 6 Sent Done Done false
22397 6751 Acked Write 1 Sent Done Done false
22398 6752 Acked Read 8 Sent Done Done false
22399 6753 Acked Read 7 Sent Done Done false
22400 6754 Acked Read 1 Sent Done Done false
22401 6755 Acked Write 4 Sent Done Done false
22402 6756 Acked Write 8 Sent Done Done false
22403 6757 Acked Write 9 Sent Done Done false
22404 6758 Acked Flush 0 Sent Done Done false
22405 6759 Acked Read 6 Sent Done Done false
22406 6760 Acked Read 7 Sent Done Done false
22407 6761 Acked Write 2 Sent Done Done false
22408 6762 Acked Flush 0 Sent Done Done false
22409 6763 Acked Read 2 Sent Done Done false
22410 6764 Acked Write 3 Sent Done Done false
22411 6765 Acked Read 8 Sent Done Done false
22412 6766 Acked Write 10 Sent Done Done false
22413 6767 Acked Read 2 Sent Done Done false
22414 6768 Acked Write 5 Sent Done Done false
22415 6769 Acked Flush 0 Sent Done Done false
22416 6770 Acked Write 2 Sent Done Done false
22417 6771 Acked Read 10 Sent Done Done false
22418 6772 Acked Write 3 Sent Done Done false
22419 6773 Acked Write 4 Sent Done Done false
22420 6774 Acked Write 3 Sent Done Done false
22421 6775 Acked Read 2 Sent Done Done false
22422 6776 Acked Write 5 Sent Done Done false
22423 6777 Acked Read 1 Sent Done Done false
22424 6778 Acked Read 6 Sent Done Done false
22425 6779 Acked Write 5 Sent Done Done false
22426 6780 Acked Read 7 Sent Done Done false
22427 6781 Acked Read 10 Sent Done Done false
22428 6782 Acked Read 2 Sent Done Done false
22429 6783 Acked Read 8 Sent Done Done false
22430 6784 Acked Write 2 Sent Done Done false
22431 6785 Acked Read 6 Sent Done Done false
22432 6786 Acked Read 3 Sent Done Done false
22433 6787 Acked Write 8 Sent Done Done false
22434 6788 Acked Write 6 Sent Done Done false
22435 6789 Acked Flush 0 Sent Done Done false
22436 6790 Acked Write 6 Sent Done Done false
22437 6791 Acked Read 6 Sent Done Done false
22438 6792 Acked Write 7 Sent Done Done false
22439 6793 Acked Write 4 Sent Done Done false
22440 6794 Acked Read 10 Sent Done Done false
22441 6795 Acked Read 2 Sent Done Done false
22442 6796 Acked Read 9 Sent Done Done false
22443 6797 Acked Write 10 Sent Done Done false
22444 6798 Acked Read 4 Sent Done Done false
22445 6799 Acked Read 1 Sent Done Done false
22446 6800 Acked Read 2 Sent Done Done false
22447 6801 Acked Write 5 Sent Done Done false
22448 6802 Acked Write 10 Sent Done Done false
22449 6803 Acked Flush 0 Sent Done Done false
22450 6804 Acked Write 1 Sent Done Done false
22451 6805 Acked Read 5 Sent Done Done false
22452 6806 Acked Read 1 Sent Done Done false
22453 6807 Acked Read 9 Sent Done Done false
22454 6808 Acked Write 8 Sent Done Done false
22455 6809 Acked Read 9 Sent Done Done false
22456 6810 Acked Write 3 Sent Done Done false
22457 6811 Acked Flush 0 Sent Done Done false
22458 6812 Acked Read 9 Sent Done Done false
22459 6813 Acked Write 7 Sent Done Done false
22460 6814 Acked Read 10 Sent Done Done false
22461 6815 Acked Read 5 Sent Done Done false
22462 6816 Acked Read 4 Sent Done Done false
22463 6817 Acked Write 10 Sent Done Done false
22464 6818 Acked Write 8 Sent Done Done false
22465 6819 Acked Read 8 Sent Done Done false
22466 6820 Acked Read 4 Sent Done Done false
22467 6821 Acked Flush 0 Sent Done Done false
22468 6822 Acked Read 9 Sent Done Done false
22469 6823 Acked Read 10 Sent Done Done false
22470 6824 Acked Write 3 Sent Done Done false
22471 6825 Acked Write 10 Sent Done Done false
22472 6826 Acked Read 9 Sent Done Done false
22473 6827 Acked Flush 0 Sent Done Done false
22474 6828 Acked Flush 0 Sent Done Done false
22475 6829 Acked Read 9 Sent Done Done false
22476 6830 Acked Read 7 Sent Done Done false
22477 6831 Acked Read 7 Sent Done Done false
22478 6832 Acked Read 9 Sent Done Done false
22479 6833 Acked Read 8 Sent Done Done false
22480 6834 Acked Write 3 Sent Done Done false
22481 6835 Acked Write 6 Sent Done Done false
22482 6836 Acked Read 7 Sent Done Done false
22483 6837 Acked Write 6 Sent Done Done false
22484 6838 Acked Write 8 Sent Done Done false
22485 6839 Acked Read 10 Sent Done Done false
22486 6840 Acked Read 7 Sent Done Done false
22487 6841 Acked Write 2 Sent Done Done false
22488 6842 Acked Flush 0 Sent Done Done false
22489 6843 Acked Read 5 Sent Done Done false
22490 6844 Acked Flush 0 Sent Done Done false
22491 6845 Acked Read 5 Sent Done Done false
22492 6846 Acked Read 1 Sent Done Done false
22493 6847 Acked Read 9 Sent Done Done false
22494 6848 Acked Read 3 Sent Done Done false
22495 6849 Acked Read 1 Sent Done Done false
22496 6850 Acked Flush 0 Sent Done Done false
22497 6851 Acked Write 1 Sent Done Done false
22498 6852 Acked Read 1 Sent Done Done false
22499 6853 Acked Read 6 Sent Done Done false
22500 6854 Acked Write 9 Sent Done Done false
22501 6855 Acked Read 5 Sent Done Done false
22502 6856 Acked Read 10 Sent Done Done false
22503 6857 Acked Write 1 Sent Done Done false
22504 6858 Acked Write 6 Sent Done Done false
22505 6859 Acked Write 1 Sent Done Done false
22506 6860 Acked Read 4 Sent Done Done false
22507 6861 Acked Flush 0 Sent Done Done false
22508 6862 Acked Read 2 Sent Done Done false
22509 6863 Acked Write 9 Sent Done Done false
22510 6864 Acked Flush 0 Sent Done Done false
22511 6865 Acked Read 7 Sent Done Done false
22512 6866 Acked Read 4 Sent Done Done false
22513 6867 Acked Read 4 Sent Done Done false
22514 6868 Acked Write 1 Sent Done Done false
22515 6869 Acked Flush 0 Sent Done Done false
22516 6870 Acked Write 10 Sent Done Done false
22517 6871 Acked Flush 0 Sent Done Done false
22518 6872 Acked Write 10 Sent Done Done false
22519 6873 Acked Write 1 Sent Done Done false
22520 6874 Acked Read 3 Sent Done Done false
22521 6875 Acked Write 10 Sent Done Done false
22522 6876 Acked Write 10 Sent Done Done false
22523 6877 Acked Read 7 Sent Done Done false
22524 6878 Acked Write 6 Sent Done Done false
22525 6879 Acked Write 7 Sent Done Done false
22526 6880 Acked Write 7 Sent Done Done false
22527 6881 Acked Read 1 Sent Done Done false
22528 6882 Acked Write 7 Sent Done Done false
22529 6883 Acked Read 8 Sent Done Done false
22530 6884 Acked Read 3 Sent Done Done false
22531 6885 Acked Read 2 Sent Done Done false
22532 6886 Acked Write 8 Sent Done Done false
22533 6887 Acked Flush 0 Sent Done Done false
22534 6888 Acked Read 10 Sent Done Done false
22535 6889 Acked Flush 0 Sent Done Done false
22536 6890 Acked Write 4 Sent Done Done false
22537 6891 Acked Write 2 Sent Done Done false
22538 6892 Acked Read 2 Sent Done Done false
22539 6893 Acked Write 6 Sent Done Done false
22540 6894 Acked Write 3 Sent Done Done false
22541 6895 Acked Read 7 Sent Done Done false
22542 6896 Acked Write 3 Sent Done Done false
22543 6897 Acked Read 10 Sent Done Done false
22544 6898 Acked Read 8 Sent Done Done false
22545 6899 Acked Read 1 Sent Done Done false
22546 6900 Acked Flush 0 Sent Done Done false
22547 6901 Acked Write 7 Sent Done Done false
22548 6902 Acked Flush 0 Sent Done Done false
22549 6903 Acked Read 7 Sent Done Done false
22550 6904 Acked Write 3 Sent Done Done false
22551 6905 Acked Read 10 Sent Done Done false
22552 6906 Acked Read 2 Sent Done Done false
22553 6907 Acked Write 8 Sent Done Done false
22554 6908 Acked Read 8 Sent Done Done false
22555 6909 Acked Write 7 Sent Done Done false
22556 6910 Acked Read 8 Sent Done Done false
22557 6911 Acked Write 10 Sent Done Done false
22558 6912 Acked Read 3 Sent Done Done false
22559 6913 Acked Read 8 Sent Done Done false
22560 6914 Acked Read 1 Sent Done Done false
22561 6915 Acked Read 8 Sent Done Done false
22562 6916 Acked Write 6 Sent Done Done false
22563 6917 Acked Flush 0 Sent Done Done false
22564 6918 Acked Read 3 Sent Done Done false
22565 6919 Acked Read 9 Sent Done Done false
22566 6920 Acked Read 3 Sent Done Done false
22567 6921 Acked Write 7 Sent Done Done false
22568 6922 Acked Write 2 Sent Done Done false
22569 6923 Acked Write 5 Sent Done Done false
22570 6924 Acked Write 9 Sent Done Done false
22571 6925 Acked Flush 0 Sent Done Done false
22572 6926 Acked Read 3 Sent Done Done false
22573 6927 Acked Read 6 Sent Done Done false
22574 6928 Acked Read 9 Sent Done Done false
22575 6929 Acked Read 7 Sent Done Done false
22576 6930 Acked Write 7 Sent Done Done false
22577 6931 Acked Read 5 Sent Done Done false
22578 6932 Acked Read 3 Sent Done Done false
22579 6933 Acked Read 3 Sent Done Done false
22580 6934 Acked Write 6 Sent Done Done false
22581 6935 Acked Write 1 Sent Done Done false
22582 6936 Acked Read 9 Sent Done Done false
22583 6937 Acked Read 3 Sent Done Done false
22584 6938 Acked Flush 0 Sent Done Done false
22585 6939 Acked Read 5 Sent Done Done false
22586 6940 Acked Flush 0 Sent Done Done false
22587 6941 Acked Read 1 Sent Done Done false
22588 6942 Acked Write 8 Sent Done Done false
22589 6943 Acked Write 5 Sent Done Done false
22590 6944 Acked Read 4 Sent Done Done false
22591 6945 Acked Write 1 Sent Done Done false
22592 6946 Acked Read 1 Sent Done Done false
22593 6947 Acked Write 10 Sent Done Done false
22594 6948 Acked Read 8 Sent Done Done false
22595 6949 Acked Write 9 Sent Done Done false
22596 6950 Acked Write 3 Sent Done Done false
22597 6951 Acked Write 1 Sent Done Done false
22598 6952 Acked Read 10 Sent Done Done false
22599 6953 Acked Read 10 Sent Done Done false
22600 6954 Acked Read 9 Sent Done Done false
22601 6955 Acked Read 2 Sent Done Done false
22602 6956 Acked Write 5 Sent Done Done false
22603 6957 Acked Write 2 Sent Done Done false
22604 6958 Acked Flush 0 Sent Done Done false
22605 6959 Acked Flush 0 Sent Done Done false
22606 6960 Acked Read 2 Sent Done Done false
22607 6961 Acked Flush 0 Sent Done Done false
22608 6962 Acked Read 4 Sent Done Done false
22609 6963 Acked Read 10 Sent Done Done false
22610 6964 Acked Write 6 Sent Done Done false
22611 6965 Acked Read 2 Sent Done Done false
22612 6966 Acked Read 2 Sent Done Done false
22613 6967 Acked Write 7 Sent Done Done false
22614 6968 Acked Read 1 Sent Done Done false
22615 6969 Acked Read 10 Sent Done Done false
22616 6970 Acked Write 1 Sent Done Done false
22617 6971 Acked Write 7 Sent Done Done false
22618 6972 Acked Write 7 Sent Done Done false
22619 6973 Acked Flush 0 Sent Done Done false
22620 6974 Acked Read 10 Sent Done Done false
22621 6975 Acked Write 4 Sent Done Done false
22622 6976 Acked Flush 0 Sent Done Done false
22623 6977 Acked Write 3 Sent Done Done false
22624 6978 Acked Flush 0 Sent Done Done false
22625 6979 Acked Read 7 Sent Done Done false
22626 6980 Acked Read 10 Sent Done Done false
22627 6981 Acked Write 8 Sent Done Done false
22628 6982 Acked Write 5 Sent Done Done false
22629 6983 Acked Write 7 Sent Done Done false
22630 6984 Acked Read 3 Sent Done Done false
22631 6985 Acked Write 5 Sent Done Done false
22632 6986 Acked Write 9 Sent Done Done false
22633 6987 Acked Read 9 Sent Done Done false
22634 6988 Acked Write 8 Sent Done Done false
22635 6989 Acked Write 3 Sent Done Done false
22636 6990 Acked Read 7 Sent Done Done false
22637 6991 Acked Write 2 Sent Done Done false
22638 6992 Acked Read 4 Sent Done Done false
22639 6993 Acked Write 1 Sent Done Done false
22640 6994 Acked Read 10 Sent Done Done false
22641 6995 Acked Write 7 Sent Done Done false
22642 6996 Acked Write 9 Sent Done Done false
22643 6997 Acked Read 7 Sent Done Done false
22644 6998 Acked Read 8 Sent Done Done false
22645 6999 Acked Read 5 Sent Done Done false
22646 7000 Acked Read 9 Sent Done Done false
22647 7001 Acked Write 4 Sent Done Done false
22648 7002 Acked Flush 0 Sent Done Done false
22649 7003 Acked Read 7 Sent Done Done false
22650 7004 Acked Write 1 Sent Done Done false
22651 7005 Acked Read 5 Sent Done Done false
22652 7006 Acked Flush 0 Sent Done Done false
22653 7007 Acked Read 10 Sent Done Done false
22654 7008 Acked Read 8 Sent Done Done false
22655 7009 Acked Write 4 Sent Done Done false
22656 7010 Acked Read 3 Sent Done Done false
22657 7011 Acked Write 10 Sent Done Done false
22658 7012 Acked Flush 0 Sent Done Done false
22659 7013 Acked Flush 0 Sent Done Done false
22660 7014 Acked Write 8 Sent Done Done false
22661 7015 Acked Read 3 Sent Done Done false
22662 7016 Acked Read 2 Sent Done Done false
22663 7017 Acked Write 1 Sent Done Done false
22664 7018 Acked Flush 0 Sent Done Done false
22665 7019 Acked Read 8 Sent Done Done false
22666 7020 Acked Read 6 Sent Done Done false
22667 7021 Acked Write 1 Sent Done Done false
22668 7022 Acked Flush 0 Sent Done Done false
22669 7023 Acked Read 8 Sent Done Done false
22670 7024 Acked Write 6 Sent Done Done false
22671 7025 Acked Flush 0 Sent Done Done false
22672 7026 Acked Read 7 Sent Done Done false
22673 7027 Acked Read 6 Sent Done Done false
22674 7028 Acked Read 10 Sent Done Done false
22675 7029 Acked Write 8 Sent Done Done false
22676 7030 Acked Read 10 Sent Done Done false
22677 7031 Acked Write 4 Sent Done Done false
22678 7032 Acked Read 8 Sent Done Done false
22679 7033 Acked Read 3 Sent Done Done false
22680 7034 Acked Read 5 Sent Done Done false
22681 7035 Acked Write 7 Sent Done Done false
22682 7036 Acked Read 4 Sent Done Done false
22683 STATES DS:0 DS:1 DS:2 TOTAL
22684 Sent 301 0 0 301
22685 Done 0 301 301 602
22686 Skipped 0 0 0 0
22687 Error 0 0 0 0
22688 Last Flush: Some(JobId(6735)) Some(JobId(7025)) Some(JobId(7025))
22689 Downstairs last five completed: 6735 6734 6733 6732 6731
22690 Upstairs last five completed: 7036 7035 7034 7033 7032
22691 CLIENT: Up:0 ds:301 act:2
226922025-02-21T06:50:21.122ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
226932025-02-21T06:50:22.124ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
226942025-02-21T06:50:23.124ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
22695 [8810] got start action
22696 [8810] I am going to restart
22697 Starting downstairs at port 8810
22698 Make output file at "/tmp/dsc/downstairs-8810.txt"
22699 [8810][0] reports Starting
22700 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1226)
22701 [8810][0] reports Running
22702 {"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40----------------------------------------------------------------
22703 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22704 States: Offline Active Active
22705 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22706 6736 Acked Write 8," Senttime":" Done Done2025-02-21T06:50:24.276949039Z false
22707 "," 6737 Acked Read 2hostname Sent" Done:" Done false
22708 w-01JMKKZ32D872C7C41J12FXWY5" 6738 Acked Read 3, Sent" Donepid Done" false
22709 : 6739 Acked Read 21206 Sent Done Done false
22710 , 6740 Acked Read 6" Sent Donesession_id" Done: false
22711 " 6741 Acked Read 103a757114-e39d-42e6-bc2b-0419bce3aed2" Sent Done Done false}
22712 
22713 6742 Acked Write 8 Sent Done Done false
22714 6743 Acked Write 3 Sent Done Done false
22715 6744 Acked Read 9 Sent Done Done false
22716 6745 Acked Write 5 Sent Done Done false
22717 6746 Acked Write 9 Sent Done Done false
22718 6747 Acked Read 6 Sent Done Done false
22719 6748 Acked Read 9 Sent Done Done false
22720 6749 Acked Write 3 Sent Done Done false
22721 6750 Acked Write 6 Sent Done Done false
22722 6751 Acked Write 1 Sent Done Done false
22723 6752 Acked Read 8 Sent Done Done false
22724 6753 Acked Read 7 Sent Done Done false
22725 6754 Acked Read 1 Sent Done Done false
22726 6755 Acked Write 4 Sent Done Done false
22727 6756 Acked Write 8 Sent Done Done false
22728 6757 Acked Write 9 Sent Done Done false
22729 6758 Acked Flush 0 Sent Done Done false
22730 6759 Acked Read 6 Sent Done Done false
22731 6760 Acked Read 7 Sent Done Done false
22732 6761 Acked Write 2 Sent Done Done false
22733 6762 Acked Flush 0 Sent Done Done false
22734 6763 Acked Read 2 Sent Done Done false
22735 6764 Acked Write 3 Sent Done Done false
22736 6765 Acked Read 8 Sent Done Done false
22737 6766 Acked Write 10 Sent Done Done false
22738 6767 Acked Read 2 Sent Done Done false
22739 6768 Acked Write 5 Sent Done Done false
22740 6769 Acked Flush 0 Sent Done Done false
22741 6770 Acked Write 2 Sent Done Done false
22742 6771 Acked Read 10 Sent Done Done false
22743 6772 Acked Write 3 Sent Done Done false
22744 6773 Acked Write 4 Sent Done Done false
22745 6774 Acked Write 3 Sent Done Done false
22746 6775 Acked Read 2 Sent Done Done false
22747 6776 Acked Write 5 Sent Done Done false
22748 6777 Acked Read 1 Sent Done Done false
22749 6778 Acked Read 6 Sent Done Done false
22750 6779 Acked Write 5 Sent Done Done false
22751 6780 Acked Read 7 Sent Done Done false
22752 6781 Acked Read 10 Sent Done Done false
22753 6782 Acked Read 2 Sent Done Done false
22754 6783 Acked Read 8 Sent Done Done false
22755 6784 Acked Write 2 Sent Done Done false
22756 6785 Acked Read 6 Sent Done Done false
22757 6786 Acked Read 3 Sent Done Done false
22758 6787 Acked Write 8 Sent Done Done false
22759 6788 Acked Write 6 Sent Done Done false
22760 6789 Acked Flush 0 Sent Done Done false
22761 6790 Acked Write 6 Sent Done Done false
22762 6791 Acked Read 6 Sent Done Done false
22763 6792 Acked Write 7 Sent Done Done false
22764 6793 Acked Write 4 Sent Done Done false
22765 6794 Acked Read 10 Sent Done Done false
22766 6795 Acked Read 2 Sent Done Done false
22767 6796 Acked Read 9 Sent Done Done false
22768 6797 Acked Write 10 Sent Done Done false
22769 6798 Acked Read 4 Sent Done Done false
22770 6799 Acked Read 1 Sent Done Done false
22771 6800 Acked Read 2 Sent Done Done false
22772 6801 Acked Write 5 Sent Done Done false
22773 6802 Acked Write 10 Sent Done Done false
22774 6803 Acked Flush 0 Sent Done Done false
22775 6804 Acked Write 1 Sent Done Done false
22776 6805 Acked Read 5 Sent Done Done false
22777 6806 Acked Read 1 Sent Done Done false
22778 6807 Acked Read 9 Sent Done Done false
22779 6808 Acked Write 8 Sent Done Done false
22780 6809 Acked Read 9 Sent Done Done false
22781 6810 Acked Write 3 Sent Done Done false
22782 6811 Acked Flush 0 Sent Done Done false
22783 6812 Acked Read 9 Sent Done Done false
22784 6813 Acked Write 7 Sent Done Done false
22785 6814 Acked Read 10 Sent Done Done false
22786 6815 Acked Read 5 Sent Done Done false
22787 6816 Acked Read 4 Sent Done Done false
22788 6817 Acked Write 10 Sent Done Done false
22789 6818 Acked Write 8 Sent Done Done false
22790 6819 Acked Read 8 Sent Done Done false
22791 6820 Acked Read 4 Sent Done Done false
22792 6821 Acked Flush 0 Sent Done Done false
22793 6822 Acked Read 9 Sent Done Done false
22794 6823 Acked Read 10 Sent Done Done false
22795 6824 Acked Write 3 Sent Done Done false
22796 6825 Acked Write 10 Sent Done Done false
22797 6826 Acked Read 9 Sent Done Done false
22798 6827 Acked Flush 0 Sent Done Done false
22799 6828 Acked Flush 0 Sent Done Done false
22800 6829 Acked Read 9 Sent Done Done false
22801 6830 Acked Read 7 Sent Done Done false
22802 6831 Acked Read 7 Sent Done Done false
22803 6832 Acked Read 9 Sent Done Done false
22804 6833 Acked Read 8 Sent Done Done false
22805 6834 Acked Write 3 Sent Done Done false
22806 6835 Acked Write 6 Sent Done Done false
22807 6836 Acked Read 7 Sent Done Done false
22808 6837 Acked Write 6 Sent Done Done false
22809 6838 Acked Write 8 Sent Done Done false
22810 6839 Acked Read 10 Sent Done Done false
22811 6840 Acked Read 7 Sent Done Done false
22812 6841 Acked Write 2 Sent Done Done false
22813 6842 Acked Flush 0 Sent Done Done false
22814 6843 Acked Read 5 Sent Done Done false
22815 6844 Acked Flush 0 Sent Done Done false
22816 6845 Acked Read 5 Sent Done Done false
22817 6846 Acked Read 1 Sent Done Done false
22818 6847 Acked Read 9 Sent Done Done false
22819 6848 Acked Read 3 Sent Done Done false
22820 6849 Acked Read 1 Sent Done Done false
22821 6850 Acked Flush 0 Sent Done Done false
22822 6851 Acked Write 1 Sent Done Done false
22823 6852 Acked Read 1 Sent Done Done false
22824 6853 Acked Read 6 Sent Done Done false
22825 6854 Acked Write 9 Sent Done Done false
22826 6855 Acked Read 5 Sent Done Done false
22827 6856 Acked Read 10 Sent Done Done false
22828 6857 Acked Write 1 Sent Done Done false
22829 6858 Acked Write 6 Sent Done Done false
22830 6859 Acked Write 1 Sent Done Done false
22831 6860 Acked Read 4 Sent Done Done false
22832 6861 Acked Flush 0 Sent Done Done false
22833 6862 Acked Read 2 Sent Done Done false
22834 6863 Acked Write 9 Sent Done Done false
22835 6864 Acked Flush 0 Sent Done Done false
22836 6865 Acked Read 7 Sent Done Done false
22837 6866 Acked Read 4 Sent Done Done false
22838 6867 Acked Read 4 Sent Done Done false
22839 6868 Acked Write 1 Sent Done Done false
22840 6869 Acked Flush 0 Sent Done Done false
22841 6870 Acked Write 10 Sent Done Done false
22842 6871 Acked Flush 0 Sent Done Done false
22843 6872 Acked Write 10 Sent Done Done false
22844 6873 Acked Write 1 Sent Done Done false
22845 6874 Acked Read 3 Sent Done Done false
22846 6875 Acked Write 10 Sent Done Done false
22847 6876 Acked Write 10 Sent Done Done false
22848 6877 Acked Read 7 Sent Done Done false
22849 6878 Acked Write 6 Sent Done Done false
22850 6879 Acked Write 7 Sent Done Done false
22851 6880 Acked Write 7 Sent Done Done false
22852 6881 Acked Read 1 Sent Done Done false
22853 6882 Acked Write 7 Sent Done Done false
22854 6883 Acked Read 8 Sent Done Done false
22855 6884 Acked Read 3 Sent Done Done false
22856 6885 Acked Read 2 Sent Done Done false
22857 6886 Acked Write 8 Sent Done Done false
22858 6887 Acked Flush 0 Sent Done Done false
22859 6888 Acked Read 10 Sent Done Done false
22860 6889 Acked Flush 0 Sent Done Done false
22861 6890 Acked Write 4 Sent Done Done false
22862 6891 Acked Write 2 Sent Done Done false
22863 6892 Acked Read 2 Sent Done Done false
22864 6893 Acked Write 6 Sent Done Done false
22865 6894 Acked Write 3 Sent Done Done false
22866 6895 Acked Read 7 Sent Done Done false
22867 6896 Acked Write 3 Sent Done Done false
22868 6897 Acked Read 10 Sent Done Done false
22869 6898 Acked Read 8 Sent Done Done false
22870 6899 Acked Read 1 Sent Done Done false
22871 6900 Acked Flush 0 Sent Done Done false
22872 6901 Acked Write 7 Sent Done Done false
22873 6902 Acked Flush 0 Sent Done Done false
22874 6903 Acked Read 7 Sent Done Done false
22875 6904 Acked Write 3 Sent Done Done false
22876 6905 Acked Read 10 Sent Done Done false
22877 6906 Acked Read 2 Sent Done Done false
22878 6907 Acked Write 8 Sent Done Done false
22879 6908 Acked Read 8 Sent Done Done false
22880 6909 Acked Write 7 Sent Done Done false
22881 6910 Acked Read 8 Sent Done Done false
22882 6911 Acked Write 10 Sent Done Done false
22883 6912 Acked Read 3 Sent Done Done false
22884 6913 Acked Read 8 Sent Done Done false
22885 6914 Acked Read 1 Sent Done Done false
22886 6915 Acked Read 8 Sent Done Done false
22887 6916 Acked Write 6 Sent Done Done false
22888 6917 Acked Flush 0 Sent Done Done false
22889 6918 Acked Read 3 Sent Done Done false
22890 6919 Acked Read 9 Sent Done Done false
22891 6920 Acked Read 3 Sent Done Done false
22892 6921 Acked Write 7 Sent Done Done false
22893 6922 Acked Write 2 Sent Done Done false
22894 6923 Acked Write 5 Sent Done Done false
22895 6924 Acked Write 9 Sent Done Done false
22896 6925 Acked Flush 0 Sent Done Done false
22897 6926 Acked Read 3 Sent Done Done false
22898 6927 Acked Read 6 Sent Done Done false
22899 6928 Acked Read 9 Sent Done Done false
22900 6929 Acked Read 7 Sent Done Done false
22901 6930 Acked Write 7 Sent Done Done false
22902 6931 Acked Read 5 Sent Done Done false
22903 6932 Acked Read 3 Sent Done Done false
22904 6933 Acked Read 3 Sent Done Done false
22905 6934 Acked Write 6 Sent Done Done false
22906 6935 Acked Write 1 Sent Done Done false
22907 6936 Acked Read 9 Sent Done Done false
22908 6937 Acked Read 3 Sent Done Done false
22909 6938 Acked Flush 0 Sent Done Done false
22910 6939 Acked Read 5 Sent Done Done false
22911 6940 Acked Flush 0 Sent Done Done false
22912 6941 Acked Read 1 Sent Done Done false
22913 6942 Acked Write 8 Sent Done Done false
22914 6943 Acked Write 5 Sent Done Done false
22915 6944 Acked Read 4 Sent Done Done false
22916 6945 Acked Write 1 Sent Done Done false
22917 6946 Acked Read 1 Sent Done Done false
22918 6947 Acked Write 10 Sent Done Done false
22919 6948 Acked Read 8 Sent Done Done false
22920 6949 Acked Write 9 Sent Done Done false
22921 6950 Acked Write 3 Sent Done Done false
22922 6951 Acked Write 1 Sent Done Done false
22923 6952 Acked Read 10 Sent Done Done false
22924 6953 Acked Read 10 Sent Done Done false
22925 6954 Acked Read 9 Sent Done Done false
22926 6955 Acked Read 2 Sent Done Done false
22927 6956 Acked Write 5 Sent Done Done false
22928 6957 Acked Write 2 Sent Done Done false
22929 6958 Acked Flush 0 Sent Done Done false
22930 6959 Acked Flush 0 Sent Done Done false
22931 6960 Acked Read 2 Sent Done Done false
22932 6961 Acked Flush 0 Sent Done Done false
22933 6962 Acked Read 4 Sent Done Done false
22934 6963 Acked Read 10 Sent Done Done false
22935 6964 Acked Write 6 Sent Done Done false
22936 6965 Acked Read 2 Sent Done Done false
22937 6966 Acked Read 2 Sent Done Done false
22938 6967 Acked Write 7 Sent Done Done false
22939 6968 Acked Read 1 Sent Done Done false
22940 6969 Acked Read 10 Sent Done Done false
22941 6970 Acked Write 1 Sent Done Done false
22942 6971 Acked Write 7 Sent Done Done false
22943 6972 Acked Write 7 Sent Done Done false
22944 6973 Acked Flush 0 Sent Done Done false
22945 6974 Acked Read 10 Sent Done Done false
22946 6975 Acked Write 4 Sent Done Done false
22947 6976 Acked Flush 0 Sent Done Done false
22948 6977 Acked Write 3 Sent Done Done false
22949 6978 Acked Flush 0 Sent Done Done false
22950 6979 Acked Read 7 Sent Done Done false
22951 6980 Acked Read 10 Sent Done Done false
22952 6981 Acked Write 8 Sent Done Done false
22953 6982 Acked Write 5 Sent Done Done false
22954 6983 Acked Write 7 Sent Done Done false
22955 6984 Acked Read 3 Sent Done Done false
22956 6985 Acked Write 5 Sent Done Done false
22957 6986 Acked Write 9 Sent Done Done false
22958 6987 Acked Read 9 Sent Done Done false
22959 6988 Acked Write 8 Sent Done Done false
22960 6989 Acked Write 3 Sent Done Done false
22961 6990 Acked Read 7 Sent Done Done false
22962 6991 Acked Write 2 Sent Done Done false
22963 6992 Acked Read 4 Sent Done Done false
22964 6993 Acked Write 1 Sent Done Done false
22965 6994 Acked Read 10 Sent Done Done false
22966 6995 Acked Write 7 Sent Done Done false
22967 6996 Acked Write 9 Sent Done Done false
22968 6997 Acked Read 7 Sent Done Done false
22969 6998 Acked Read 8 Sent Done Done false
22970 6999 Acked Read 5 Sent Done Done false
22971 7000 Acked Read 9 Sent Done Done false
22972 7001 Acked Write 4 Sent Done Done false
22973 7002 Acked Flush 0 Sent Done Done false
22974 7003 Acked Read 7 Sent Done Done false
22975 7004 Acked Write 1 Sent Done Done false
22976 7005 Acked Read 5 Sent Done Done false
22977 7006 Acked Flush 0 Sent Done Done false
22978 7007 Acked Read 10 Sent Done Done false
22979 7008 Acked Read 8 Sent Done Done false
22980 7009 Acked Write 4 Sent Done Done false
22981 7010 Acked Read 3 Sent Done Done false
22982 7011 Acked Write 10 Sent Done Done false
22983 7012 Acked Flush 0 Sent Done Done false
22984 7013 Acked Flush 0 Sent Done Done false
22985 7014 Acked Write 8 Sent Done Done false
22986 7015 Acked Read 3 Sent Done Done false
22987 7016 Acked Read 2 Sent Done Done false
22988 7017 Acked Write 1 Sent Done Done false
22989 7018 Acked Flush 0 Sent Done Done false
22990 7019 Acked Read 8 Sent Done Done false
22991 7020 Acked Read 6 Sent Done Done false
22992 7021 Acked Write 1 Sent Done Done false
22993 7022 Acked Flush 0 Sent Done Done false
22994 7023 Acked Read 8 Sent Done Done false
22995 7024 Acked Write 6 Sent Done Done false
22996 7025 Acked Flush 0 Sent Done Done false
22997 7026 Acked Read 7 Sent Done Done false
22998 7027 Acked Read 6 Sent Done Done false
22999 7028 Acked Read 10 Sent Done Done false
23000 7029 Acked Write 8 Sent Done Done false
23001 7030 Acked Read 10 Sent Done Done false
23002 7031 Acked Write 4 Sent Done Done false
23003 7032 Acked Read 8 Sent Done Done false
23004 7033 Acked Read 3 Sent Done Done false
23005 7034 Acked Read 5 Sent Done Done false
23006 7035 Acked Write 7 Sent Done Done false
23007 7036 Acked Read 4 Sent Done Done false
23008 STATES DS:0 DS:1 DS:2 TOTAL
23009 Sent 301 0 0 301
23010 Done 0 301 301 602
23011 Skipped 0 0 0 0
23012 Error 0 0 0 0
23013 Last Flush: Some(JobId(6735)) Some(JobId(7025)) Some(JobId(7025))
23014 Downstairs last five completed: 6735 6734 6733 6732 6731
23015 Upstairs last five completed: 7036 7035 7034 7033 7032
23016 CLIENT: Up:0 ds:301 act:2
230172025-02-21T06:50:25.127ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
230182025-02-21T06:50:26.026ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
230192025-02-21T06:50:26.027ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51544) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
230202025-02-21T06:50:26.028ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
230212025-02-21T06:50:26.029ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
230222025-02-21T06:50:26.029ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
230232025-02-21T06:50:26.029ZINFOcrucible: send last flush ID to this DS: Some(JobId(6735)) = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
230242025-02-21T06:50:26.029ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
230252025-02-21T06:50:26.029ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(6735)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
230262025-02-21T06:50:26.029ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(6735)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
23027 ----------------------------------------------------------------
23028 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23029 Downstairs last five completed: 7037 7036 7035 7034 7033
23030 Upstairs last five completed: 7037 7036 7035 7034 7033
23031 CLIENT: Up:0 ds:0 act:3
23032 Replay: All jobs finished
23033 Main task has work to do, go find it
23034 got dsc Stop(1)
23035 stop 1
23036 [8820] Got stop action so:false kr:false
23037 Feb 21 06:50:28.279 INFO request completed, latency_us: 367, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 03304726-567d-4e03-a595-2f73270e3227, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
230382025-02-21T06:50:28.280ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
23039 Feb 21 06:50:28.280 INFO{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:50:28.28031993Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
23040 request completed, latency_us: 161, response_code: 200, uri{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:50:28.280353374Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
23041 : /state/cid/1, method: GET, req_id: 1f435ca0-ef9d-45c2-8f3b-c2bad9da4071, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
230422025-02-21T06:50:28.280ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
23043 [8820] Exited with: ExitStatus(unix_wait_status(9))
23044 [8820][1] reports Exit
23045 Feb 21 06:50:32.282 INFO request completed, latency_us: 226, response_code: 200, uri: /state/cid/1, method: GET, req_id: dfd79ac7-1454-4781-9e6a-a48b8c5c19d7, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
23046 001/300 Read block 211 len 20480
23047 002/300 Write block 1850 len 24576 data: 3 3 4 4 4 5
23048 003/300 Write block 1848 len 8192 data: 4 4
23049 004/300 Write block 173 len 28672 data: 7 6 6 5 4 3 4
23050 005/300 Read block 2939 len 28672
23051 006/300 Write block 2066 len 8192 data: 5 5
23052 007/300 Write block 3169 len 24576 data: 7 6 5 5 7 7
23053 008/300 Read block 3053 len 36864
23054 009/300 Write block 3821 len 16384 data: 10 8 8 6
23055 010/300 Read block 97 len 28672
23056 011/300 Read block 1988 len 20480
23057 012/300 Read block 4173 len 16384
23058 013/300 Read block 3147 len 36864
23059 014/300 Write block 2200 len 4096 data: 4
23060 015/300 Read block 754 len 4096
23061 016/300 Write block 3032 len 28672 data: 7 8 7 7 7 6 6
23062 017/300 Read block 1103 len 40960
23063 018/300 Flush
23064 019/300 Read block 1335 len 4096
23065 020/300 Write block 4035 len 28672 data: 5 6 5 5 4 4 4
23066 021/300 Read block 1311 len 36864
23067 022/300 Write block 1646 len 24576 data: 3 3 3 3 2 2
23068 023/300 Write block 3031 len 40960 data: 7 8 9 8 8 8 7 7 7 6
23069 024/300 Flush
23070 025/300 Read block 1766 len 28672
23071 026/300 Read block 1586 len 4096
23072 027/300 Read block 3710 len 40960
23073 028/300 Read block 1571 len 8192
23074 029/300 Write block 4737 len 4096 data: 3
23075 030/300 Write block 753 len 12288 data: 3 4 6
23076 031/300 Read block 3144 len 8192
23077 032/300 Flush
23078 033/300 Read block 1256 len 12288
23079 034/300 Read block 445 len 12288
23080 035/300 Read block 2478 len 8192
23081 036/300 Read block 3265 len 24576
23082 037/300 Flush
23083 038/300 Read block 2810 len 8192
23084 039/300 Write block 4004 len 24576 data: 7 7 8 8 9 7
23085 040/300 Read block 323 len 32768
23086 041/300 Read block 8 len 36864
23087 042/300 Write block 2320 len 28672 data: 3 3 3 3 3 5 6
23088 043/300 Read block 2292 len 8192
23089 044/300 Read block 4239 len 28672
23090 045/300 Write block 2272 len 8192 data: 2 2
23091 046/300 Read block 4812 len 36864
23092 047/300 Read block 4473 len 28672
23093 048/300 Write block 4836 len 32768 data: 5 4 5 4 3 3 2 3
23094 049/300 Write block 834 len 12288 data: 3 3 4
23095 050/300 Write block 4362 len 24576 data: 5 4 3 3 3 4
23096 051/300 Write block 1665 len 12288 data: 6 6 5
23097 052/300 Read block 418 len 28672
23098 053/300 Read block 3988 len 16384
23099 054/300 Write block 4030 len 40960 data: 2 2 5 6 5 6 7 6 6 5
23100 055/300 Read block 1252 len 40960
23101 056/300 Write block 1482 len 20480 data: 6 4 5 5 5
23102 057/300 Write block 3335 len 12288 data: 4 4 6
23103 058/300 Read block 87 len 40960
23104 059/300 Write block 524 len 24576 data: 8 7 5 5 5 4
23105 060/300 Write block 4202 len 20480 data: 4 4 4 4 5
23106 061/300 Write block 3923 len 28672 data: 4 4 4 4 4 6 6
23107 062/300 Read block 3917 len 20480
23108 063/300 Write block 1945 len 12288 data: 5 6 6
23109 064/300 Write block 2752 len 16384 data: 9 7 5 4
23110 065/300 Write block 3776 len 12288 data: 6 4 3
23111 066/300 Read block 173 len 36864
23112 067/300 Read block 3192 len 32768
23113 068/300 Read block 3122 len 8192
23114 069/300 Write block 1972 len 16384 data: 5 4 3 3
23115 070/300 Write block 2209 len 12288 data: 4 4 4
23116 071/300 Read block 1704 len 40960
23117 072/300 Write block 2647 len 4096 data: 5
23118 073/300 Read block 3321 len 36864
23119 074/300 Read block 3382 len 8192
23120 075/300 Flush
23121 076/300 Read block 644 len 8192
23122 077/300 Read block 2655 len 32768
23123 078/300 Write block 342 len 4096 data: 3
23124 079/300 Read block 4485 len 20480
23125 080/300 Read block 1233 len 4096
23126 081/300 Read block 3613 len 20480
23127 082/300 Read block 1248 len 28672
23128 083/300 Write block 221 len 24576 data: 5 5 4 5 5 3
23129 084/300 Read block 4494 len 36864
23130 085/300 Read block 188 len 40960
23131 086/300 Write block 1538 len 4096 data: 6
23132 087/300 Read block 3772 len 24576
23133 088/300 Write block 4625 len 8192 data: 8 6
23134 089/300 Read block 1083 len 32768
23135 090/300 Flush
23136 091/300 Write block 2181 len 4096 data: 6
23137 092/300 Read block 331 len 4096
23138 093/300 Write block 2238 len 20480 data: 6 6 6 4 5
23139 094/300 Flush
23140 095/300 Read block 860 len 12288
23141 096/300 Read block 1226 len 40960
23142 097/300 Read block 2680 len 40960
23143 098/300 Write block 1029 len 40960 data: 6 6 6 7 5 4 4 5 3 3
23144 099/300 Read block 2862 len 28672
23145 100/300 Read block 1168 len 4096
23146 101/300 Read block 1825 len 32768
23147 102/300 Flush
23148 103/300 Flush
23149 104/300 Read block 763 len 28672
23150 105/300 Read block 2123 len 36864
23151 106/300 Read block 1639 len 16384
23152 107/300 Write block 3979 len 16384 data: 5 4 4 5
23153 108/300 Read block 4036 len 36864
23154 109/300 Write block 932 len 12288 data: 4 3 2
23155 110/300 Read block 2896 len 32768
23156 111/300 Read block 711 len 4096
23157 112/300 Read block 726 len 20480
23158 113/300 Read block 1909 len 8192
23159 114/300 Write block 3972 len 4096 data: 5
23160 115/300 Write block 2659 len 16384 data: 4 6 4 6
23161 116/300 Write block 1087 len 36864 data: 5 4 4 4 4 5 5 5 6
23162 117/300 Read block 1047 len 32768
23163 118/300 Read block 407 len 36864
23164 119/300 Write block 2871 len 8192 data: 6 6
23165 120/300 Flush
23166 121/300 Write block 281 len 4096 data: 3
23167 122/300 Flush
23168 123/300 Read block 3728 len 36864
23169 124/300 Read block 559 len 28672
23170 125/300 Read block 3872 len 4096
23171 126/300 Write block 2018 len 36864 data: 4 5 5 4 4 4 4 5 5
23172 127/300 Read block 2079 len 36864
23173 128/300 Read block 4973 len 40960
23174 129/300 Write block 913 len 40960 data: 3 4 4 3 3 3 3 2 2 4
23175 130/300 Read block 4265 len 28672
23176 131/300 Read block 4094 len 12288
23177 132/300 Write block 4355 len 28672 data: 7 7 8 7 6 6 5
23178 133/300 Read block 2191 len 12288
23179 134/300 Read block 1716 len 8192
23180 135/300 Write block 2800 len 40960 data: 4 5 5 5 5 4 5 4 3 3
23181 136/300 Read block 2610 len 24576
23182 137/300 Write block 255 len 28672 data: 3 3 3 3 4 4 4
23183 138/300 Write block 3780 len 4096 data: 4
23184 139/300 Read block 95 len 28672
23185 140/300 Write block 354 len 32768 data: 4 3 2 2 2 2 2 2
23186 141/300 Read block 632 len 24576
23187 142/300 Flush
23188 143/300 Write block 2536 len 36864 data: 2 3 3 4 4 4 4 3 3
23189 144/300 Write block 4212 len 20480 data: 7 6 4 4 5
23190 145/300 Read block 4241 len 28672
23191 146/300 Read block 2543 len 8192
23192 147/300 Write block 4279 len 20480 data: 4 5 5 5 7
23193 148/300 Read block 175 len 12288
23194 149/300 Read block 4616 len 16384
23195 150/300 Write block 640 len 12288 data: 5 5 4
23196 151/300 Read block 1914 len 28672
23197 152/300 Write block 2811 len 28672 data: 3 3 3 4 4 4 5
23198 153/300 Write block 4671 len 24576 data: 6 6 6 8 9 9
23199 154/300 Flush
23200 155/300 Read block 2009 len 40960
23201 156/300 Read block 230 len 8192
23202 157/300 Write block 4342 len 16384 data: 3 3 3 3
23203 158/300 Read block 2464 len 8192
23204 159/300 Write block 1779 len 20480 data: 6 5 6 6 7
23205 160/300 Read block 3775 len 20480
23206 161/300 Read block 3977 len 20480
23207 162/300 Read block 1017 len 28672
23208 163/300 Read block 3799 len 24576
23209 164/300 Write block 1882 len 4096 data: 6
23210 165/300 Write block 642 len 24576 data: 5 3 4 3 5 5
23211 166/300 Flush
23212 167/300 Write block 2136 len 32768 data: 4 5 5 5 5 4 4 3
23213 168/300 Write block 955 len 12288 data: 5 5 7
23214 169/300 Read block 2315 len 20480
23215 170/300 Read block 577 len 40960
23216 171/300 Write block 3314 len 32768 data: 5 5 5 5 5 5 4 5
23217 172/300 Read block 4132 len 28672
23218 173/300 Read block 4482 len 32768
23219 174/300 Read block 4836 len 40960
23220 175/300 Flush
23221 176/300 Write block 351 len 24576 data: 3 3 4 5 4 3
23222 177/300 Read block 2102 len 8192
23223 178/300 Write block 1030 len 28672 data: 7 7 8 6 5 5 6
23224 179/300 Write block 1887 len 12288 data: 4 5 4
23225 180/300 Write block 138 len 40960 data: 4 5 5 6 6 4 4 4 3 3
23226 181/300 Write block 3499 len 20480 data: 5 6 5 4 4
23227 182/300 Write block 2213 len 20480 data: 3 7 8 8 7
23228 183/300 Read block 2773 len 36864
23229 184/300 Flush
23230 185/300 Write block 2261 len 4096 data: 5
23231 186/300 Read block 2447 len 4096
23232 187/300 Read block 337 len 36864
23233 188/300 Write block 4517 len 40960 data: 2 2 5 5 6 5 6 5 4 4
23234 189/300 Read block 2008 len 40960
23235 190/300 Read block 1980 len 20480
23236 191/300 Write block 2837 len 40960 data: 5 5 5 5 4 3 3 3 3 3
23237 192/300 Write block 464 len 8192 data: 4 5
23238 193/300 Read block 3521 len 12288
23239 194/300 Write block 3699 len 32768 data: 4 5 5 6 5 4 4 4
23240 195/300 Write block 3046 len 8192 data: 3 3
23241 196/300 Read block 1870 len 28672
23242 197/300 Read block 4086 len 40960
23243 198/300 Write block 294 len 36864 data: 4 5 3 3 3 3 4 4 4
23244 199/300 Read block 3370 len 8192
23245 200/300 Write block 3096 len 24576 data: 4 5 5 5 6 6
23246 201/300 Read block 3528 len 28672
23247 202/300 Write block 532 len 28672 data: 3 5 4 4 5 4 5
23248 203/300 Read block 3386 len 4096
23249 204/300 Read block 3542 len 8192
23250 205/300 Read block 4921 len 8192
23251 206/300 Read block 2697 len 16384
23252 207/300 Write block 3277 len 12288 data: 3 4 4
23253 208/300 Write block 2488 len 12288 data: 4 3 2
23254 209/300 Read block 4566 len 36864
23255 210/300 Read block 465 len 24576
23256 211/300 Read block 3487 len 20480
23257 212/300 Read block 2485 len 36864
23258 213/300 Write block 4370 len 4096 data: 4
23259 214/300 Read block 3340 len 16384
23260 215/300 Write block 321 len 36864 data: 3 3 4 3 3 3 4 3 4
23261 216/300 Write block 2950 len 12288 data: 6 8 9
23262 217/300 Write block 3536 len 20480 data: 2 3 2 3 4
23263 218/300 Flush
23264 219/300 Read block 2540 len 12288
23265 220/300 Write block 2771 len 12288 data: 3 2 2
23266 221/300 Write block 1921 len 4096 data: 4
23267 222/300 Read block 4099 len 8192
23268 223/300 Read block 3244 len 12288
23269 224/300 Write block 2489 len 20480 data: 4 3 3 3 3
23270 225/300 Write block 4758 len 24576 data: 5 5 5 5 5 5
23271 226/300 Read block 1924 len 40960
23272 227/300 Read block 4343 len 12288
23273 228/300 Read block 814 len 32768
23274 229/300 Read block 3631 len 16384
23275 230/300 Read block 1088 len 12288
23276 231/300 Read block 2947 len 20480
23277 232/300 Read block 4015 len 20480
23278 233/300 Write block 995 len 12288 data: 4 4 4
23279 234/300 Write block 4673 len 16384 data: 7 9 10 10
23280 235/300 Read block 3198 len 24576
23281 236/300 Flush
23282 237/300 Flush
23283 238/300 Read block 2210 len 4096
23284 239/300 Read block 4529 len 40960
23285 240/300 Read block 4666 len 32768
23286 241/300 Flush
23287 242/300 Read block 1844 len 24576
23288 243/300 Read block 4822 len 40960
23289 244/300 Read block 3904 len 28672
23290 245/300 Write block 203 len 12288 data: 4 4 3
23291 246/300 Read block 4805 len 24576
23292 247/300 Write block 3961 len 28672 data: 3 3 4 5 4 4 4
23293 248/300 Read block 1615 len 40960
23294 249/300 Read block 1750 len 28672
23295 250/300 Flush
23296 251/300 Read block 4280 len 28672
23297 252/300 Read block 700 len 40960
23298 253/300 Read block 2624 len 32768
23299 254/300 Read block 929 len 40960
23300 255/300 Read block 4972 len 32768
23301 256/300 Write block 1270 len 36864 data: 2 3 5 6 7 6 7 7 7
23302 257/300 Write block 4171 len 20480 data: 3 3 3 3 3
23303 258/300 Write block 4614 len 32768 data: 6 5 6 5 6 4 5 4
23304 259/300 Read block 2133 len 24576
23305 260/300 Read block 4923 len 4096
23306 261/300 Write block 3460 len 28672 data: 5 5 4 5 4 4 4
23307 262/300 Read block 2806 len 24576
23308 263/300 Write block 3953 len 24576 data: 6 4 5 5 5 4
23309 264/300 Read block 762 len 40960
23310 265/300 Write block 2939 len 4096 data: 5
23311 266/300 Read block 1125 len 36864
23312 267/300 Flush
23313 268/300 Read block 4621 len 12288
23314 269/300 Read block 1318 len 28672
23315 270/300 Read block 1910 len 4096
23316 271/300 Write block 2048 len 8192 data: 4 5
23317 272/300 Write block 4012 len 16384 data: 5 4 4 4
23318 273/300 Write block 2560 len 24576 data: 6 6 6 5 5 4
23319 274/300 Write block 1080 len 20480 data: 4 3 3 3 3
23320 275/300 Flush
23321 276/300 Read block 2922 len 28672
23322 277/300 Read block 694 len 36864
23323 278/300 Write block 100 len 24576 data: 6 6 6 6 5 7
23324 279/300 Write block 483 len 32768 data: 3 3 3 2 2 2 2 3
23325 280/300 Read block 1407 len 36864
23326 281/300 Write block 922 len 36864 data: 5 4 5 6 6 6 6 6 5
23327 282/300 Read block 1349 len 28672
23328 283/300 Read block 502 len 36864
23329 284/300 Write block 4370 len 16384 data: 5 3 3 3
23330 285/300 Write block 934 len 32768 data: 3 2 3 3 3 3 3 3
23331 286/300 Write block 4933 len 32768 data: 3 4 4 5 5 4 5 5
23332 287/300 Read block 666 len 24576
23333 288/300 Read block 3964 len 4096
23334 289/300 Write block 3462 len 28672 data: 5 6 5 5 5 5 5
23335 290/300 Write block 4181 len 4096 data: 4
23336 291/300 Write block 4095 len 24576 data: 4 4 3 3 4 4
23337 292/300 Read block 1809 len 12288
23338 293/300 Read block 4539 len 24576
23339 294/300 Write block 1836 len 24576 data: 4 4 4 5 6 4
23340 295/300 Read block 4484 len 36864
23341 296/300 Write block 436 len 28672 data: 5 4 3 4 4 5 3
23342 297/300 Write block 3040 len 40960 data: 7 5 4 4 4 4 4 4 5 4
23343 298/300 Read block 2082 len 16384
23344 299/300 Flush
23345 300/300 Write block 4275 len 8192 data: 5 5
23346 301/300 Write block 3687 len 24576 data: 2 2 2 2 2 2
23347 Main task has work to do, go find it
23348 got dsc Start(1)
23349 start 1
23350 Feb 21 06:50:32.533 INFO request completed, latency_us: 231, response_code: 204, uri: /start/cid/1, method: POST, req_id: 685e30cd-d9d6-45ca-a411-9b506895644d, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
23351 Replay: started 1, returned:Ok(())
23352 ----------------------------------------------------------------
23353 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23354 States: Active Offline Active
23355 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23356 7038 Acked Read 5 Done Sent Done false
23357 7039 Acked Write 6 Done Sent Done false
23358 7040 Acked Write 2 Done Sent Done false
23359 7041 Acked Write 7 Done Sent Done false
23360 7042 Acked Read 7 Done Sent Done false
23361 7043 Acked Write 2 Done Sent Done false
23362 7044 Acked Write 6 Done Sent Done false
23363 7045 Acked Read 9 Done Sent Done false
23364 7046 Acked Write 4 Done Sent Done false
23365 7047 Acked Read 7 Done Sent Done false
23366 7048 Acked Read 5 Done Sent Done false
23367 7049 Acked Read 4 Done Sent Done false
23368 7050 Acked Read 9 Done Sent Done false
23369 7051 Acked Write 1 Done Sent Done false
23370 7052 Acked Read 1 Done Sent Done false
23371 7053 Acked Write 7 Done Sent Done false
23372 7054 Acked Read 10 Done Sent Done false
23373 7055 Acked Flush 0 Done Sent Done false
23374 7056 Acked Read 1 Done Sent Done false
23375 7057 Acked Write 7 Done Sent Done false
23376 7058 Acked Read 9 Done Sent Done false
23377 7059 Acked Write 6 Done Sent Done false
23378 7060 Acked Write 10 Done Sent Done false
23379 7061 Acked Flush 0 Done Sent Done false
23380 7062 Acked Read 7 Done Sent Done false
23381 7063 Acked Read 1 Done Sent Done false
23382 7064 Acked Read 10 Done Sent Done false
23383 7065 Acked Read 2 Done Sent Done false
23384 7066 Acked Write 1 Done Sent Done false
23385 7067 Acked Write 3 Done Sent Done false
23386 7068 Acked Read 2 Done Sent Done false
23387 7069 Acked Flush 0 Done Sent Done false
23388 7070 Acked Read 3 Done Sent Done false
23389 7071 Acked Read 3 Done Sent Done false
23390 7072 Acked Read 2 Done Sent Done false
23391 7073 Acked Read 6 Done Sent Done false
23392 7074 Acked Flush 0 Done Sent Done false
23393 7075 Acked Read 2 Done Sent Done false
23394 7076 Acked Write 6 Done Sent Done false
23395 7077 Acked Read 8 Done Sent Done false
23396 7078 Acked Read 9 Done Sent Done false
23397 7079 Acked Write 7 Done Sent Done false
23398 7080 Acked Read 2 Done Sent Done false
23399 7081 Acked Read 7 Done Sent Done false
23400 7082 Acked Write 2 Done Sent Done false
23401 7083 Acked Read 9 Done Sent Done false
23402 7084 Acked Read 7 Done Sent Done false
23403 7085 Acked Write 8 Done Sent Done false
23404 7086 Acked Write 3 Done Sent Done false
23405 7087 Acked Write 6 Done Sent Done false
23406 7088 Acked Write 3 Done Sent Done false
23407 7089 Acked Read 7 Done Sent Done false
23408 7090 Acked Read 4 Done Sent Done false
23409 7091 Acked Write 10 Done Sent Done false
23410 7092 Acked Read 10 Done Sent Done false
23411 7093 Acked Write 5 Done Sent Done false
23412 7094 Acked Write 3 Done Sent Done false
23413 7095 Acked Read 10 Done Sent Done false
23414 7096 Acked Write 6 Done Sent Done false
23415 7097 Acked Write 5 Done Sent Done false
23416 7098 Acked Write 7 Done Sent Done false
23417 7099 Acked Read 5 Done Sent Done false
23418 7100 Acked Write 3 Done Sent Done false
23419 7101 Acked Write 4 Done Sent Done false
23420 7102 Acked Write 3 Done Sent Done false
23421 7103 Acked Read 9 Done Sent Done false
23422 7104 Acked Read 8 Done Sent Done false
23423 7105 Acked Read 2 Done Sent Done false
23424 7106 Acked Write 4 Done Sent Done false
23425 7107 Acked Write 3 Done Sent Done false
23426 7108 Acked Read 10 Done Sent Done false
23427 7109 Acked Write 1 Done Sent Done false
23428 7110 Acked Read 9 Done Sent Done false
23429 7111 Acked Read 2 Done Sent Done false
23430 7112 Acked Flush 0 Done Sent Done false
23431 7113 Acked Read 2 Done Sent Done false
23432 7114 Acked Read 8 Done Sent Done false
23433 7115 Acked Write 1 Done Sent Done false
23434 7116 Acked Read 5 Done Sent Done false
23435 7117 Acked Read 1 Done Sent Done false
23436 7118 Acked Read 5 Done Sent Done false
23437 7119 Acked Read 7 Done Sent Done false
23438 7120 Acked Write 6 Done Sent Done false
23439 7121 Acked Read 9 Done Sent Done false
23440 7122 Acked Read 10 Done Sent Done false
23441 7123 Acked Write 1 Done Sent Done false
23442 7124 Acked Read 6 Done Sent Done false
23443 7125 Acked Write 2 Done Sent Done false
23444 7126 Acked Read 8 Done Sent Done false
23445 7127 Acked Flush 0 Done Sent Done false
23446 7128 Acked Write 1 Done Sent Done false
23447 7129 Acked Read 1 Done Sent Done false
23448 7130 Acked Write 5 Done Sent Done false
23449 7131 Acked Flush 0 Done Sent Done false
23450 7132 Acked Read 3 Done Sent Done false
23451 7133 Acked Read 10 Done Sent Done false
23452 7134 Acked Read 10 Done Sent Done false
23453 7135 Acked Write 10 Done Sent Done false
23454 7136 Acked Read 7 Done Sent Done false
23455 7137 Acked Read 1 Done Sent Done false
23456 7138 Acked Read 8 Done Sent Done false
23457 7139 Acked Flush 0 Done Sent Done false
23458 7140 Acked Flush 0 Done Sent Done false
23459 7141 Acked Read 7 Done Sent Done false
23460 7142 Acked Read 9 Done Sent Done false
23461 7143 Acked Read 4 Done Sent Done false
23462 7144 Acked Write 4 Done Sent Done false
23463 7145 Acked Read 9 Done Sent Done false
23464 7146 Acked Write 3 Done Sent Done false
23465 7147 Acked Read 8 Done Sent Done false
23466 7148 Acked Read 1 Done Sent Done false
23467 7149 Acked Read 5 Done Sent Done false
23468 7150 Acked Read 2 Done Sent Done false
23469 7151 Acked Write 1 Done Sent Done false
23470 7152 Acked Write 4 Done Sent Done false
23471 7153 Acked Write 9 Done Sent Done false
23472 7154 Acked Read 8 Done Sent Done false
23473 7155 Acked Read 9 Done Sent Done false
23474 7156 Acked Write 2 Done Sent Done false
23475 7157 Acked Flush 0 Done Sent Done false
23476 7158 Acked Write 1 Done Sent Done false
23477 7159 Acked Flush 0 Done Sent Done false
23478 7160 Acked Read 9 Done Sent Done false
23479 7161 Acked Read 7 Done Sent Done false
23480 7162 Acked Read 1 Done Sent Done false
23481 7163 Acked Write 9 Done Sent Done false
23482 7164 Acked Read 9 Done Sent Done false
23483 7165 Acked Read 10 Done Sent Done false
23484 7166 Acked Write 10 Done Sent Done false
23485 7167 Acked Read 7 Done Sent Done false
23486 7168 Acked Read 3 Done Sent Done false
23487 7169 Acked Write 7 Done Sent Done false
23488 7170 Acked Read 3 Done Sent Done false
23489 7171 Acked Read 2 Done Sent Done false
23490 7172 Acked Write 10 Done Sent Done false
23491 7173 Acked Read 6 Done Sent Done false
23492 7174 Acked Write 7 Done Sent Done false
23493 7175 Acked Write 1 Done Sent Done false
23494 7176 Acked Read 7 Done Sent Done false
23495 7177 Acked Write 8 Done Sent Done false
23496 7178 Acked Read 6 Done Sent Done false
23497 7179 Acked Flush 0 Done Sent Done false
23498 7180 Acked Write 9 Done Sent Done false
23499 7181 Acked Write 5 Done Sent Done false
23500 7182 Acked Read 7 Done Sent Done false
23501 7183 Acked Read 2 Done Sent Done false
23502 7184 Acked Write 5 Done Sent Done false
23503 7185 Acked Read 3 Done Sent Done false
23504 7186 Acked Read 4 Done Sent Done false
23505 7187 Acked Write 3 Done Sent Done false
23506 7188 Acked Read 7 Done Sent Done false
23507 7189 Acked Write 7 Done Sent Done false
23508 7190 Acked Write 6 Done Sent Done false
23509 7191 Acked Flush 0 Done Sent Done false
23510 7192 Acked Read 10 Done Sent Done false
23511 7193 Acked Read 2 Done Sent Done false
23512 7194 Acked Write 4 Done Sent Done false
23513 7195 Acked Read 2 Done Sent Done false
23514 7196 Acked Write 5 Done Sent Done false
23515 7197 Acked Read 5 Done Sent Done false
23516 7198 Acked Read 5 Done Sent Done false
23517 7199 Acked Read 7 Done Sent Done false
23518 7200 Acked Read 6 Done Sent Done false
23519 7201 Acked Write 1 Done Sent Done false
23520 7202 Acked Write 6 Done Sent Done false
23521 7203 Acked Flush 0 Done Sent Done false
23522 7204 Acked Write 8 Done Sent Done false
23523 7205 Acked Write 3 Done Sent Done false
23524 7206 Acked Read 5 Done Sent Done false
23525 7207 Acked Read 10 Done Sent Done false
23526 7208 Acked Write 8 Done Sent Done false
23527 7209 Acked Read 7 Done Sent Done false
23528 7210 Acked Read 8 Done Sent Done false
23529 7211 Acked Read 10 Done Sent Done false
23530 7212 Acked Flush 0 Done Sent Done false
23531 7213 Acked Write 6 Done Sent Done false
23532 7214 Acked Read 2 Done Sent Done false
23533 7215 Acked Write 7 Done Sent Done false
23534 7216 Acked Write 3 Done Sent Done false
23535 7217 Acked Write 10 Done Sent Done false
23536 7218 Acked Write 5 Done Sent Done false
23537 7219 Acked Write 5 Done Sent Done false
23538 7220 Acked Read 9 Done Sent Done false
23539 7221 Acked Flush 0 Done Sent Done false
23540 7222 Acked Write 1 Done Sent Done false
23541 7223 Acked Read 1 Done Sent Done false
23542 7224 Acked Read 9 Done Sent Done false
23543 7225 Acked Write 10 Done Sent Done false
23544 7226 Acked Read 10 Done Sent Done false
23545 7227 Acked Read 5 Done Sent Done false
23546 7228 Acked Write 10 Done Sent Done false
23547 7229 Acked Write 2 Done Sent Done false
23548 7230 Acked Read 3 Done Sent Done false
23549 7231 Acked Write 8 Done Sent Done false
23550 7232 Acked Write 2 Done Sent Done false
23551 7233 Acked Read 7 Done Sent Done false
23552 7234 Acked Read 10 Done Sent Done false
23553 7235 Acked Write 9 Done Sent Done false
23554 7236 Acked Read 2 Done Sent Done false
23555 7237 Acked Write 6 Done Sent Done false
23556 7238 Acked Read 7 Done Sent Done false
23557 7239 Acked Write 7 Done Sent Done false
23558 7240 Acked Read 1 Done Sent Done false
23559 7241 Acked Read 2 Done Sent Done false
23560 7242 Acked Read 2 Done Sent Done false
23561 7243 Acked Read 4 Done Sent Done false
23562 7244 Acked Write 3 Done Sent Done false
23563 7245 Acked Write 3 Done Sent Done false
23564 7246 Acked Read 9 Done Sent Done false
23565 7247 Acked Read 6 Done Sent Done false
23566 7248 Acked Read 5 Done Sent Done false
23567 7249 Acked Read 9 Done Sent Done false
23568 7250 Acked Write 1 Done Sent Done false
23569 7251 Acked Read 4 Done Sent Done false
23570 7252 Acked Write 9 Done Sent Done false
23571 7253 Acked Write 3 Done Sent Done false
23572 7254 Acked Write 5 Done Sent Done false
23573 7255 Acked Flush 0 Done Sent Done false
23574 7256 Acked Read 3 Done Sent Done false
23575 7257 Acked Write 3 Done Sent Done false
23576 7258 Acked Write 1 Done Sent Done false
23577 7259 Acked Read 2 Done Sent Done false
23578 7260 Acked Read 3 Done Sent Done false
23579 7261 Acked Write 5 Done Sent Done false
23580 7262 Acked Write 6 Done Sent Done false
23581 7263 Acked Read 10 Done Sent Done false
23582 7264 Acked Read 3 Done Sent Done false
23583 7265 Acked Read 8 Done Sent Done false
23584 7266 Acked Read 4 Done Sent Done false
23585 7267 Acked Read 3 Done Sent Done false
23586 7268 Acked Read 5 Done Sent Done false
23587 7269 Acked Read 5 Done Sent Done false
23588 7270 Acked Write 3 Done Sent Done false
23589 7271 Acked Write 4 Done Sent Done false
23590 7272 Acked Read 6 Done Sent Done false
23591 7273 Acked Flush 0 Done Sent Done false
23592 7274 Acked Flush 0 Done Sent Done false
23593 7275 Acked Read 1 Done Sent Done false
23594 7276 Acked Read 10 Done Sent Done false
23595 7277 Acked Read 8 Done Sent Done false
23596 7278 Acked Flush 0 Done Sent Done false
23597 7279 Acked Read 6 Done Sent Done false
23598 7280 Acked Read 10 Done Sent Done false
23599 7281 Acked Read 7 Done Sent Done false
23600 7282 Acked Write 3 Done Sent Done false
23601 7283 Acked Read 6 Done Sent Done false
23602 7284 Acked Write 7 Done Sent Done false
23603 7285 Acked Read 10 Done Sent Done false
23604 7286 Acked Read 7 Done Sent Done false
23605 7287 Acked Flush 0 Done Sent Done false
23606 7288 Acked Read 7 Done Sent Done false
23607 7289 Acked Read 10 Done Sent Done false
23608 7290 Acked Read 8 Done Sent Done false
23609 7291 Acked Read 10 Done Sent Done false
23610 7292 Acked Read 8 Done Sent Done false
23611 7293 Acked Write 9 Done Sent Done false
23612 7294 Acked Write 5 Done Sent Done false
23613 7295 Acked Write 8 Done Sent Done false
23614 7296 Acked Read 6 Done Sent Done false
23615 7297 Acked Read 1 Done Sent Done false
23616 7298 Acked Write 7 Done Sent Done false
23617 7299 Acked Read 6 Done Sent Done false
23618 7300 Acked Write 6 Done Sent Done false
23619 7301 Acked Read 10 Done Sent Done false
23620 7302 Acked Write 1 Done Sent Done false
23621 7303 Acked Read 9 Done Sent Done false
23622 7304 Acked Flush 0 Done Sent Done false
23623 7305 Acked Read 3 Done Sent Done false
23624 7306 Acked Read 7 Done Sent Done false
23625 7307 Acked Read 1 Done Sent Done false
23626 7308 Acked Write 2 Done Sent Done false
23627 7309 Acked Write 4 Done Sent Done false
23628 7310 Acked Write 6 Done Sent Done false
23629 7311 Acked Write 5 Done Sent Done false
23630 7312 Acked Flush 0 Done Sent Done false
23631 7313 Acked Read 7 Done Sent Done false
23632 7314 Acked Read 9 Done Sent Done false
23633 7315 Acked Write 6 Done Sent Done false
23634 7316 Acked Write 8 Done Sent Done false
23635 7317 Acked Read 9 Done Sent Done false
23636 7318 Acked Write 9 Done Sent Done false
23637 7319 Acked Read 7 Done Sent Done false
23638 7320 Acked Read 9 Done Sent Done false
23639 7321 Acked Write 4 Done Sent Done false
23640 7322 Acked Write 8 Done Sent Done false
23641 7323 Acked Write 8 Done Sent Done false
23642 7324 Acked Read 6 Done Sent Done false
23643 7325 Acked Read 1 Done Sent Done false
23644 7326 Acked Write 7 Done Sent Done false
23645 7327 Acked Write 1 Done Sent Done false
23646 7328 Acked Write 6 Done Sent Done false
23647 7329 Acked Read 3 Done Sent Done false
23648 7330 Acked Read 6 Done Sent Done false
23649 7331 Acked Write 6 Done Sent Done false
23650 7332 Acked Read 9 Done Sent Done false
23651 7333 Acked Write 7 Done Sent Done false
23652 7334 Acked Write 10 Done Sent Done false
23653 7335 Acked Read 4 Done Sent Done false
23654 7336 Acked Flush 0 Done Sent Done false
23655 7337 Acked Write 2 Done Sent Done false
23656 7338 Acked Write 6 Done Sent Done false
23657 STATES DS:0 DS:1 DS:2 TOTAL
23658 Sent 0 301 0 301
23659 Done 301 0 301 602
23660 Skipped 0 0 0 0
23661 Error 0 0 0 0
23662 Last Flush: Some(JobId(7336)) Some(JobId(7037)) Some(JobId(7336))
23663 Downstairs last five completed: 7037 7036 7035 7034 7033
23664 Upstairs last five completed: 7338 7337 7336 7335 7334
23665 CLIENT: Up:0 ds:301 act:2
236662025-02-21T06:50:33.137ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
236672025-02-21T06:50:34.139ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
23668 [8820] got start action
23669 [8820] I am going to restart
23670 Starting downstairs at port 8820
23671 Make output file at "/tmp/dsc/downstairs-8820.txt"
23672 [8820][1] reports Starting
23673 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1227)
23674 [8820][1] reports Running
236752025-02-21T06:50:35.140ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
23676 ----------------------------------------------------------------
23677 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23678 States: Active Offline Active
23679 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23680 7038 Acked Read 5 Done Sent Done false
23681 7039 Acked Write 6 Done Sent Done false
23682 7040 Acked Write 2 Done Sent Done false
23683 7041 Acked Write 7 Done Sent Done false
23684 7042 Acked Read 7 Done Sent Done false
23685 7043 Acked Write 2 Done Sent Done false
23686 7044 Acked Write 6 Done Sent Done false
23687 7045 Acked Read 9 Done Sent Done false
23688 7046 Acked Write 4 Done Sent Done false
23689 7047 Acked Read 7 Done Sent Done false
23690 7048 Acked Read 5 Done Sent Done false
23691 7049 Acked Read 4 Done Sent Done false
23692 7050 Acked Read 9 Done Sent Done false
23693 7051 Acked Write 1 Done Sent Done false
23694 7052 Acked Read 1 Done Sent Done false
23695 7053 Acked Write 7 Done Sent Done false
23696 7054 Acked Read 10 Done Sent Done false
23697 7055 Acked Flush 0 Done Sent Done false
23698 7056 Acked Read 1 Done Sent Done false
23699 7057 Acked Write 7 Done Sent Done false
23700 7058 Acked Read 9 Done Sent Done false
23701 7059 Acked Write 6 Done Sent Done false
23702 7060 Acked Write 10 Done Sent Done false
23703 7061 Acked Flush 0 Done Sent Done false
23704 7062 Acked Read 7 Done Sent Done false
23705 7063 Acked Read 1 Done Sent Done false
23706 7064 Acked Read 10 Done Sent Done false
23707 7065 Acked Read 2 Done Sent Done false
23708 7066 Acked Write 1 Done Sent Done false
23709 7067 Acked Write 3 Done Sent Done false
23710 7068 Acked Read 2 Done Sent Done false
23711 7069 Acked Flush 0 Done Sent Done false
23712 7070 Acked Read 3 Done Sent Done false
23713 7071 Acked Read 3 Done Sent Done false
23714 7072 Acked Read 2 Done Sent Done false
23715 7073 Acked Read 6 Done Sent Done false
23716 7074 Acked Flush 0 Done Sent Done false
23717 7075 Acked Read 2 Done Sent Done false
23718 7076 Acked Write 6 Done Sent Done false
23719 7077 Acked Read 8 Done Sent Done false
23720 7078 Acked Read 9 Done Sent Done false
23721 7079 Acked Write 7 Done Sent Done false
23722 7080 Acked Read 2 Done Sent Done false
23723 7081 Acked Read 7 Done Sent Done false
23724 7082 Acked Write 2 Done Sent Done false
23725 7083 Acked Read 9 Done Sent Done false
23726 7084 Acked Read 7 Done Sent Done false
23727 7085 Acked Write 8 Done Sent Done false
23728 7086 Acked Write 3 Done Sent Done false
23729 7087 Acked Write 6 Done Sent Done false
23730 7088 Acked Write 3 Done Sent Done false
23731 7089 Acked Read 7 Done Sent Done false
23732 7090 Acked Read 4 Done Sent Done false
23733 7091 Acked Write 10 Done Sent Done false
23734 7092 Acked Read 10 Done Sent Done false
23735 7093 Acked Write 5 Done Sent Done false
23736 7094 Acked Write 3 Done Sent Done false
23737 7095 Acked Read 10 Done Sent Done false
23738 7096 Acked Write 6 Done Sent Done false
23739 7097 Acked Write 5 Done Sent Done false
23740 7098 Acked Write 7 Done Sent Done false
23741 7099 Acked Read 5 Done Sent Done false
23742 7100 Acked Write 3 Done Sent Done false
23743 7101 Acked Write 4 Done Sent Done false
23744 7102 Acked Write 3 Done Sent Done false
23745 7103 Acked Read 9 Done Sent Done false
23746 7104 Acked Read 8 Done Sent Done false
23747 7105 Acked Read 2 Done Sent Done false
23748 7106 Acked Write 4 Done Sent Done false
23749 7107 Acked Write 3 Done Sent Done false
23750 7108 Acked Read 10 Done Sent Done false
23751 7109 Acked Write 1 Done Sent Done false
23752 7110 Acked Read 9 Done Sent Done false
23753 7111 Acked Read 2 Done Sent Done false
23754 7112 Acked Flush 0 Done Sent Done false
23755 7113 Acked Read 2 Done Sent Done false
23756 7114 Acked Read 8 Done Sent Done false
23757 7115 Acked Write 1 Done Sent Done false
23758 7116 Acked Read 5 Done Sent Done false
23759 7117 Acked Read 1 Done Sent Done false
23760 7118 Acked Read 5 Done Sent Done false
23761 7119 Acked Read 7 Done Sent Done false
23762 7120 Acked Write 6 Done Sent Done false
23763 7121 Acked Read 9 Done Sent Done false
23764 7122 Acked Read 10 Done Sent Done false
23765 7123 Acked Write 1 Done Sent Done false
23766 7124 Acked Read 6 Done Sent Done false
23767 7125 Acked Write 2 Done Sent Done false
23768 7126 Acked Read 8 Done Sent Done false
23769 7127 Acked Flush 0 Done Sent Done false
23770 7128 Acked Write 1 Done Sent Done false
23771 7129 Acked Read 1 Done Sent Done false
23772 7130 Acked Write 5 Done Sent Done false
23773 7131 Acked Flush 0 Done Sent Done false
23774 7132 Acked Read 3 Done Sent Done false
23775 7133 Acked Read 10 Done Sent Done false
23776 7134 Acked Read 10 Done Sent Done false
23777 7135 Acked Write 10 Done Sent Done false
23778 7136 Acked Read 7 Done Sent Done false
23779 7137 Acked Read 1 Done Sent Done false
23780 7138 Acked Read 8 Done Sent Done false
23781 7139 Acked Flush 0 Done Sent Done false
23782 7140 Acked Flush 0 Done Sent Done false
23783 7141 Acked Read 7 Done Sent Done false
23784 7142 Acked Read 9 Done Sent Done false
23785 7143 Acked Read 4 Done Sent Done false
23786 7144 Acked Write 4 Done Sent Done false
23787 7145 Acked Read 9 Done Sent Done false
23788 7146 Acked Write 3 Done Sent Done false
23789 7147 Acked Read 8 Done Sent Done false
23790 7148 Acked Read 1 Done Sent Done false
23791 7149 Acked Read 5 Done Sent Done false
23792 7150 Acked Read 2 Done Sent Done false
23793 7151 Acked Write 1 Done Sent Done false
23794 7152 Acked Write 4 Done Sent Done false
23795 7153 Acked Write 9 Done Sent Done false
23796 7154 Acked Read 8 Done Sent Done false
23797 7155 Acked Read 9 Done Sent Done false
23798 7156 Acked Write 2 Done Sent Done false
23799 7157 Acked Flush 0 Done Sent Done false
23800 7158 Acked Write 1 Done Sent Done false
23801 7159 Acked Flush 0 Done Sent Done false
23802 7160 Acked Read 9 Done Sent Done false
23803 7161 Acked Read 7 Done Sent Done false
23804 7162 Acked Read 1 Done Sent Done false
23805 7163 Acked Write 9 Done Sent Done false
23806 7164 Acked Read 9 Done Sent Done false
23807 7165 Acked Read 10 Done Sent Done false
23808 7166 Acked Write 10 Done Sent Done false
23809 7167 Acked Read 7 Done Sent Done false
23810 7168 Acked Read 3 Done Sent Done false
23811 7169 Acked Write 7 Done Sent Done false
23812 7170 Acked Read 3 Done Sent Done false
23813 7171 Acked Read 2 Done Sent Done false
23814 7172 Acked Write 10 Done Sent Done false
23815 7173 Acked Read 6 Done Sent Done false
23816 7174 Acked Write 7 Done Sent Done false
23817 7175 Acked Write 1 Done Sent Done false
23818 7176 Acked Read 7 Done Sent Done false
23819 7177 Acked Write 8 Done Sent Done false
23820 7178 Acked Read 6 Done Sent Done false
23821 7179 Acked Flush 0 Done Sent Done false
23822 7180 Acked Write 9 Done Sent Done false
23823 7181 Acked Write 5 Done Sent Done false
23824 7182 Acked Read 7 Done Sent Done false
23825 7183 Acked Read 2 Done Sent Done false
23826 7184 Acked Write 5 Done Sent Done false
23827 7185 Acked Read 3 Done Sent Done false
23828 7186 Acked Read 4 Done Sent Done false
23829 7187 Acked Write 3 Done Sent Done false
23830 7188 Acked Read 7 Done Sent Done false
23831 7189 Acked Write 7 Done Sent Done false
23832 7190 Acked Write 6 Done Sent Done false
23833 7191 Acked Flush 0 Done Sent Done false
23834 7192 Acked Read 10 Done Sent Done false
23835 7193 Acked Read 2 Done Sent Done false
23836 7194 Acked Write 4 Done Sent Done false
23837 7195 Acked Read 2 Done Sent Done false
23838 7196 Acked Write 5 Done Sent Done false
23839 7197 Acked Read 5 Done Sent Done false
23840 7198 Acked Read 5 Done Sent Done false
23841 7199 Acked Read 7 Done Sent Done false
23842 7200 Acked Read 6 Done Sent Done false
23843 7201 Acked Write 1 Done Sent Done false
23844 7202 Acked Write 6 Done Sent Done false
23845 7203 Acked Flush 0 Done Sent Done false
23846 7204 Acked Write 8 Done Sent Done false
23847 7205 Acked Write 3 Done Sent Done false
23848 7206 Acked Read 5 Done Sent Done false
23849 7207 Acked Read 10 Done Sent Done false
23850 7208 Acked Write 8 Done Sent Done false
23851 7209 Acked Read 7 Done Sent Done false
23852 7210 Acked Read 8 Done Sent Done false
23853 7211 Acked Read 10 Done Sent Done false
23854 7212 Acked Flush 0 Done Sent Done false
23855 7213 Acked Write 6 Done Sent Done false
23856 7214 Acked Read 2 Done Sent Done false
23857 7215 Acked Write 7 Done Sent Done false
23858 7216 Acked Write 3 Done Sent Done false
23859 7217 Acked Write 10 Done Sent Done false
23860 7218 Acked Write 5 Done Sent Done false
23861 7219 Acked Write 5 Done Sent Done false
23862 7220 Acked Read 9 Done Sent Done false
23863 7221 Acked Flush 0 Done Sent Done false
23864 7222 Acked Write 1 Done Sent Done false
23865 7223 Acked Read 1 Done Sent Done false
23866 7224 Acked Read 9 Done Sent Done false
23867 7225 Acked Write 10 Done Sent Done false
23868 7226 Acked Read 10 Done Sent Done false
23869 7227 Acked Read 5 Done Sent Done false
23870 7228 Acked Write 10 Done Sent Done false
23871 7229 Acked Write 2 Done Sent Done false
23872 7230 Acked Read 3 Done Sent Done false
23873 7231 Acked Write 8 Done Sent Done false
23874 7232 Acked Write 2 Done Sent Done false
23875 7233 Acked Read 7 Done Sent Done false
23876 7234 Acked Read 10 Done Sent Done false
23877 7235 Acked Write 9 Done Sent Done false
23878 7236 Acked Read 2 Done Sent Done false
23879 7237 Acked Write 6 Done Sent Done false
23880 7238 Acked Read 7 Done Sent Done false
23881 7239 Acked Write 7 Done Sent Done false
23882 7240 Acked Read 1 Done Sent Done false
23883 7241 Acked Read 2 Done Sent Done false
23884 7242 Acked Read 2 Done Sent Done false
23885 7243 Acked Read 4 Done Sent Done false
23886 7244 Acked Write 3 Done Sent Done false
23887 7245 Acked Write 3 Done Sent Done false
23888 7246 Acked Read 9 Done Sent Done false
23889 7247 Acked Read 6 Done Sent Done false
23890 7248 Acked Read 5 Done Sent Done false
23891 7249 Acked Read 9 Done Sent Done false
23892 7250 Acked Write 1 Done Sent Done false
23893 7251 Acked Read 4 Done Sent Done false
23894 7252 Acked Write 9 Done Sent Done false
23895 7253 Acked Write 3 Done Sent Done false
23896 7254 Acked Write 5 Done Sent Done false
23897 7255 Acked Flush 0 Done Sent Done false
23898 7256 Acked Read 3 Done Sent Done false
23899 7257 Acked Write 3 Done Sent Done false
23900 7258 Acked Write 1 Done Sent Done false
23901 7259 Acked Read 2 Done Sent Done false
23902 7260 Acked Read 3 Done Sent Done false
23903 7261 Acked Write 5 Done Sent Done false
23904 7262 Acked Write 6 Done Sent Done false
23905 7263 Acked Read 10 Done Sent Done false
23906 7264 Acked Read 3 Done Sent Done false
23907 7265 Acked Read 8 Done Sent Done false
23908 7266 Acked Read 4 Done Sent Done false
23909 7267 Acked Read 3 Done Sent Done false
23910 7268 Acked Read 5 Done Sent Done false
23911 7269 Acked Read 5 Done Sent Done false
23912 7270 Acked Write 3 Done Sent Done false
23913 7271 Acked Write 4 Done Sent Done false
23914 7272 Acked Read 6 Done Sent Done false
23915 7273 Acked Flush 0 Done Sent Done false
23916 7274 Acked Flush 0 Done Sent Done false
23917 7275 Acked Read 1 Done Sent Done false
23918 7276 Acked Read 10 Done Sent Done false
23919 7277 Acked Read 8 Done Sent Done false
23920 7278 Acked Flush 0 Done Sent Done false
23921 7279 Acked Read 6 Done Sent Done false
23922 7280 Acked Read 10 Done Sent Done false
23923 7281 Acked Read 7 Done Sent Done false
23924 7282 Acked Write 3 Done Sent Done false
23925 7283 Acked Read 6 Done Sent Done false
23926 7284 Acked Write 7 Done Sent Done false
23927 7285 Acked Read 10 Done Sent Done false
23928 7286 Acked Read 7 Done Sent Done false
23929 7287 Acked Flush 0 Done Sent Done false
23930 7288 Acked Read 7 Done Sent Done false
23931 7289 Acked Read 10 Done Sent Done false
23932 7290 Acked Read 8 Done Sent Done false
23933 7291 Acked Read 10 Done Sent Done false
23934 7292 Acked Read 8 Done Sent Done false
23935 7293 Acked Write 9 Done Sent Done false
23936 7294 Acked Write 5 Done Sent Done false
23937 7295 Acked Write 8 Done Sent Done false
23938 7296 Acked Read 6 Done Sent Done false
23939 7297 Acked Read 1 Done Sent Done false
23940 7298 Acked Write 7 Done Sent Done false
23941 7299 Acked Read 6 Done Sent Done false
23942 7300 Acked Write 6 Done Sent Done false
23943 7301 Acked Read 10 Done Sent Done false
23944 7302 Acked Write 1 Done Sent Done false
23945 7303 Acked Read 9 Done Sent Done false
23946 7304 Acked Flush 0 Done Sent Done false
23947 7305 Acked Read 3 Done Sent Done false
23948 7306 Acked Read 7 Done Sent Done false
23949 7307 Acked Read 1 Done Sent Done false
23950 7308 Acked Write 2 Done Sent Done false
23951 7309 Acked Write 4 Done Sent Done false
23952 7310 Acked Write 6 Done Sent Done false
23953 7311 Acked Write 5 Done Sent Done false
23954 7312 Acked Flush 0 Done Sent Done false
23955 7313 Acked Read 7 Done Sent Done false
23956 7314 Acked Read 9 Done Sent Done false
23957 7315 Acked Write 6 Done Sent Done false
23958 7316 Acked Write 8 Done Sent Done false
23959 7317 Acked Read 9 Done Sent Done false
23960 7318 Acked Write 9 Done Sent Done false
23961 7319 Acked Read 7 Done Sent Done false
23962 7320 Acked Read 9 Done Sent Done false
23963 7321 Acked Write 4 Done Sent Done false
23964 7322 Acked Write 8 Done Sent Done false
23965 7323 Acked Write 8 Done Sent Done false
23966 7324 Acked Read 6 Done Sent Done false
23967 7325 Acked Read 1 Done Sent Done false
23968 7326 Acked Write 7 Done Sent Done false
23969 7327 Acked Write 1 Done Sent Done false
23970 7328 Acked Write 6 Done Sent Done false
23971 7329 Acked Read 3 Done Sent Done false
23972 7330 Acked Read 6 Done Sent Done false
23973 7331 Acked Write 6 Done Sent Done false
23974 7332 Acked Read 9 Done Sent Done false
23975 7333 Acked Write 7 Done Sent Done false
23976 7334 Acked Write 10 Done Sent Done false
23977 7335 Acked Read 4 Done Sent Done false
23978 7336 Acked Flush 0 Done Sent Done false
23979 7337 Acked Write 2 Done Sent Done false
23980 7338 Acked Write 6 Done Sent Done false
23981 STATES DS:0 DS:1 DS:2 TOTAL
23982 Sent 0 301 0 301
23983 Done 301 0 301 602
23984 Skipped 0 0 0 0
23985 Error 0 0 0 0
23986 Last Flush: Some(JobId(7336)) Some(JobId(7037)) Some(JobId(7336))
23987 Downstairs last five completed: 7037 7036 7035 7034 7033
23988 Upstairs last five completed: 7338 7337 7336 7335 7334
23989 CLIENT: Up:0 ds:301 act:2
239902025-02-21T06:50:37.143ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
239912025-02-21T06:50:38.145ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
239922025-02-21T06:50:38.281ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
239932025-02-21T06:50:38.282ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53173) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
239942025-02-21T06:50:38.283ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
239952025-02-21T06:50:38.283ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
239962025-02-21T06:50:38.283ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
239972025-02-21T06:50:38.283ZINFOcrucible: send last flush ID to this DS: Some(JobId(7037)) = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
239982025-02-21T06:50:38.283ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
239992025-02-21T06:50:38.283ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7037)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
240002025-02-21T06:50:38.283ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7037)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
24001 ----------------------------------------------------------------
24002 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24003 Downstairs last five completed: 7339 7338 7337 7336 7335
24004 Upstairs last five completed: 7339 7338 7337 7336 7335
24005 CLIENT: Up:0 ds:0 act:3
24006 Replay: All jobs finished
24007 Main task has work to do, go find it
24008 got dsc Stop(2)
24009 stop 2
24010 Feb 21 06:50:40.538 INFO request completed[8830] Got stop action so:false kr:false
24011 , latency_us: 367, response_code: 204, uri: /stop/cid/2, method: POST, req_id: fc9a74c8-05d9-449d-929f-6e664a5fc855, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
240122025-02-21T06:50:40.539ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
240132025-02-21T06:50:40.539ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
24014 Feb 21 06:50:40.539 INFO request completed, latency_us: 158, response_code:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:50:40.539420853Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
24015 200, uri: /state/cid/2, method: GET, req_id: d3440e6f-70ff-4d5c-954f-902f1cf552f7, remote_addr: 127{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:50:40.539460866Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
24016 .0.0.1:53848, local_addr: 127.0.0.1:9998
24017 [8830] Exited with: ExitStatus(unix_wait_status(9))
24018 [8830][2] reports Exit
24019 Feb 21 06:50:44.541 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4f8a6ac2-423c-4859-8186-8acd5fb9e856, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
24020 001/300 Read block 2068 len 40960
24021 002/300 Flush
24022 003/300 Read block 166 len 4096
24023 004/300 Write block 2147 len 20480 data: 3 2 3 2 2
24024 005/300 Read block 66 len 40960
24025 006/300 Read block 1504 len 36864
24026 007/300 Read block 2298 len 24576
24027 008/300 Read block 3577 len 8192
24028 009/300 Read block 151 len 12288
24029 010/300 Read block 3683 len 40960
24030 011/300 Write block 1769 len 12288 data: 5 5 6
24031 012/300 Write block 4963 len 12288 data: 7 6 5
24032 013/300 Write block 2320 len 32768 data: 4 4 4 4 4 6 7 4
24033 014/300 Read block 515 len 12288
24034 015/300 Flush
24035 016/300 Read block 2200 len 40960
24036 017/300 Read block 3247 len 16384
24037 018/300 Read block 4742 len 36864
24038 019/300 Write block 777 len 20480 data: 5 4 4 5 5
24039 020/300 Flush
24040 021/300 Read block 1776 len 8192
24041 022/300 Read block 71 len 20480
24042 023/300 Write block 1538 len 24576 data: 7 4 3 3 3 3
24043 024/300 Read block 3250 len 36864
24044 025/300 Flush
24045 026/300 Write block 1251 len 8192 data: 2 2
24046 027/300 Write block 3266 len 20480 data: 7 6 4 4 5
24047 028/300 Write block 3986 len 36864 data: 4 6 6 5 4 4 4 4 3
24048 029/300 Read block 3250 len 28672
24049 030/300 Write block 3129 len 32768 data: 6 8 8 10 8 7 11 13
24050 031/300 Read block 108 len 32768
24051 032/300 Read block 871 len 28672
24052 033/300 Read block 1201 len 8192
24053 034/300 Read block 629 len 20480
24054 035/300 Flush
24055 036/300 Read block 4067 len 36864
24056 037/300 Write block 2822 len 32768 data: 3 3 2 2 2 2 2 2
24057 038/300 Read block 3277 len 24576
24058 039/300 Flush
24059 040/300 Read block 3495 len 4096
24060 041/300 Read block 4849 len 36864
24061 042/300 Read block 548 len 32768
24062 043/300 Write block 3324 len 32768 data: 5 5 4 2 2 2 2 3
24063 044/300 Write block 3340 len 24576 data: 6 7 6 6 6 4
24064 045/300 Write block 3078 len 16384 data: 5 5 6 6
24065 046/300 Read block 738 len 16384
24066 047/300 Write block 3666 len 4096 data: 3
24067 048/300 Read block 2931 len 4096
24068 049/300 Read block 4807 len 20480
24069 050/300 Write block 3338 len 24576 data: 7 6 7 8 7 7
24070 051/300 Read block 4419 len 24576
24071 052/300 Read block 3163 len 32768
24072 053/300 Flush
24073 054/300 Read block 2195 len 40960
24074 055/300 Flush
24075 056/300 Write block 1573 len 40960 data: 3 3 3 4 5 5 5 4 4 3
24076 057/300 Read block 1185 len 36864
24077 058/300 Read block 4976 len 32768
24078 059/300 Read block 1524 len 12288
24079 060/300 Write block 2715 len 28672 data: 5 5 5 4 3 3 4
24080 061/300 Read block 2682 len 40960
24081 062/300 Flush
24082 063/300 Read block 202 len 8192
24083 064/300 Read block 4841 len 20480
24084 065/300 Read block 4 len 32768
24085 066/300 Write block 1471 len 12288 data: 2 3 3
24086 067/300 Read block 330 len 8192
24087 068/300 Write block 2029 len 20480 data: 2 3 3 4 4
24088 069/300 Read block 25 len 12288
24089 070/300 Write block 93 len 32768 data: 3 3 4 4 4 5 6 7
24090 071/300 Flush
24091 072/300 Read block 1752 len 16384
24092 073/300 Flush
24093 074/300 Write block 1737 len 12288 data: 3 3 3
24094 075/300 Read block 2662 len 32768
24095 076/300 Write block 2169 len 8192 data: 3 4
24096 077/300 Read block 1275 len 28672
24097 078/300 Read block 3125 len 12288
24098 079/300 Write block 4799 len 8192 data: 6 6
24099 080/300 Write block 871 len 16384 data: 7 5 5 3
24100 081/300 Read block 546 len 40960
24101 082/300 Read block 1311 len 16384
24102 083/300 Flush
24103 084/300 Read block 467 len 8192
24104 085/300 Write block 2008 len 16384 data: 7 5 5 6
24105 086/300 Flush
24106 087/300 Read block 1384 len 32768
24107 088/300 Read block 323 len 4096
24108 089/300 Read block 2682 len 40960
24109 090/300 Read block 3998 len 28672
24110 091/300 Read block 3995 len 8192
24111 092/300 Read block 523 len 20480
24112 093/300 Read block 1927 len 8192
24113 094/300 Read block 4497 len 40960
24114 095/300 Flush
24115 096/300 Read block 1628 len 36864
24116 097/300 Read block 4406 len 32768
24117 098/300 Write block 4312 len 8192 data: 5 4
24118 099/300 Write block 443 len 12288 data: 6 6 7
24119 100/300 Read block 4767 len 8192
24120 101/300 Read block 247 len 36864
24121 102/300 Read block 4728 len 12288
24122 103/300 Flush
24123 104/300 Write block 682 len 36864 data: 4 4 3 3 3 4 4 4 4
24124 105/300 Write block 215 len 12288 data: 8 9 9
24125 106/300 Write block 2254 len 12288 data: 3 4 3
24126 107/300 Read block 4520 len 32768
24127 108/300 Read block 974 len 32768
24128 109/300 Write block 4738 len 40960 data: 4 5 6 6 7 5 4 5 4 4
24129 110/300 Flush
24130 111/300 Read block 3624 len 16384
24131 112/300 Read block 1613 len 16384
24132 113/300 Write block 475 len 40960 data: 6 7 6 5 4 3 3 2 4 4
24133 114/300 Read block 2289 len 16384
24134 115/300 Read block 366 len 12288
24135 116/300 Write block 3711 len 28672 data: 4 5 5 5 5 4 4
24136 117/300 Write block 2939 len 12288 data: 6 4 4
24137 118/300 Write block 3840 len 12288 data: 7 8 8
24138 119/300 Write block 4267 len 20480 data: 4 4 4 4 6
24139 120/300 Write block 652 len 20480 data: 5 4 4 4 4
24140 121/300 Read block 4481 len 36864
24141 122/300 Write block 2483 len 36864 data: 3 3 3 4 4 5 5 4 4
24142 123/300 Read block 4492 len 36864
24143 124/300 Write block 2270 len 36864 data: 3 2 3 3 5 6 6 8 7
24144 125/300 Read block 1758 len 4096
24145 126/300 Write block 3670 len 4096 data: 4
24146 127/300 Write block 1660 len 12288 data: 4 5 4
24147 128/300 Write block 2896 len 20480 data: 4 4 3 3 4
24148 129/300 Write block 4750 len 40960 data: 6 3 3 3 3 3 3 3 6 6
24149 130/300 Read block 1000 len 32768
24150 131/300 Flush
24151 132/300 Read block 1524 len 12288
24152 133/300 Read block 138 len 16384
24153 134/300 Read block 822 len 4096
24154 135/300 Write block 3171 len 4096 data: 6
24155 136/300 Read block 186 len 16384
24156 137/300 Write block 2154 len 32768 data: 3 3 3 3 4 4 4 4
24157 138/300 Read block 1662 len 16384
24158 139/300 Read block 3885 len 8192
24159 140/300 Write block 4577 len 4096 data: 5
24160 141/300 Read block 4493 len 40960
24161 142/300 Read block 1654 len 40960
24162 143/300 Read block 4075 len 20480
24163 144/300 Read block 2527 len 28672
24164 145/300 Read block 1509 len 32768
24165 146/300 Read block 4028 len 24576
24166 147/300 Write block 2529 len 24576 data: 2 2 2 2 2 2
24167 148/300 Flush
24168 149/300 Write block 794 len 24576 data: 6 6 4 3 2 2
24169 150/300 Write block 1155 len 40960 data: 2 3 3 3 3 3 2 2 3 4
24170 151/300 Write block 3772 len 20480 data: 9 9 8 8 7
24171 152/300 Read block 951 len 16384
24172 153/300 Read block 1548 len 4096
24173 154/300 Read block 3922 len 40960
24174 155/300 Read block 2033 len 36864
24175 156/300 Write block 4394 len 8192 data: 5 6
24176 157/300 Read block 3539 len 32768
24177 158/300 Write block 196 len 20480 data: 3 3 4 3 3
24178 159/300 Read block 452 len 16384
24179 160/300 Write block 2499 len 28672 data: 3 4 4 3 3 3 5
24180 161/300 Flush
24181 162/300 Flush
24182 163/300 Write block 252 len 40960 data: 4 6 5 4 4 4 4 5 5 5
24183 164/300 Read block 1439 len 32768
24184 165/300 Write block 132 len 32768 data: 5 5 5 5 5 6 5 6
24185 166/300 Read block 174 len 4096
24186 167/300 Write block 3516 len 12288 data: 4 4 3
24187 168/300 Write block 2715 len 16384 data: 6 6 6 5
24188 169/300 Write block 3355 len 24576 data: 6 4 3 3 3 2
24189 170/300 Write block 2343 len 28672 data: 5 4 4 4 5 4 3
24190 171/300 Read block 1692 len 32768
24191 172/300 Read block 3060 len 36864
24192 173/300 Write block 3220 len 16384 data: 6 5 5 4
24193 174/300 Write block 293 len 40960 data: 4 5 6 4 4 4 4 5 5 5
24194 175/300 Write block 2587 len 8192 data: 4 5
24195 176/300 Read block 4640 len 4096
24196 177/300 Read block 4473 len 36864
24197 178/300 Write block 3025 len 36864 data: 6 6 5 6 6 6 8 9 10
24198 179/300 Write block 2453 len 20480 data: 2 2 3 3 4
24199 180/300 Read block 2252 len 20480
24200 181/300 Flush
24201 182/300 Read block 3761 len 40960
24202 183/300 Read block 4398 len 40960
24203 184/300 Write block 4869 len 40960 data: 3 3 2 4 4 5 5 6 5 6
24204 185/300 Read block 1238 len 28672
24205 186/300 Read block 2596 len 20480
24206 187/300 Write block 868 len 24576 data: 4 4 6 8 6 6
24207 188/300 Flush
24208 189/300 Read block 2861 len 28672
24209 190/300 Write block 4334 len 20480 data: 4 4 4 4 3
24210 191/300 Read block 3691 len 12288
24211 192/300 Write block 2865 len 28672 data: 5 5 5 5 5 6 7
24212 193/300 Read block 2486 len 32768
24213 194/300 Read block 371 len 8192
24214 195/300 Write block 1670 len 40960 data: 3 3 2 4 4 5 5 5 4 5
24215 196/300 Write block 4659 len 24576 data: 4 4 3 4 4 4
24216 197/300 Read block 2272 len 28672
24217 198/300 Read block 1574 len 28672
24218 199/300 Write block 3192 len 16384 data: 3 3 4 4
24219 200/300 Read block 58 len 40960
24220 201/300 Write block 11 len 36864 data: 2 2 4 4 4 5 4 5 4
24221 202/300 Read block 3563 len 20480
24222 203/300 Write block 4449 len 4096 data: 5
24223 204/300 Write block 662 len 28672 data: 4 5 5 6 7 7 7
24224 205/300 Write block 4633 len 8192 data: 2 2
24225 206/300 Read block 4523 len 8192
24226 207/300 Read block 4056 len 24576
24227 208/300 Write block 1980 len 4096 data: 6
24228 209/300 Write block 4494 len 32768 data: 4 4 4 2 2 2 3 3
24229 210/300 Write block 3340 len 12288 data: 8 9 8
24230 211/300 Read block 813 len 12288
24231 212/300 Write block 2976 len 36864 data: 7 7 6 5 5 5 4 4 5
24232 213/300 Write block 2296 len 36864 data: 3 4 3 3 3 4 5 7 6
24233 214/300 Write block 1854 len 16384 data: 5 6 5 4
24234 215/300 Flush
24235 216/300 Flush
24236 217/300 Write block 43 len 12288 data: 5 4 5
24237 218/300 Write block 3869 len 20480 data: 4 5 5 6 8
24238 219/300 Read block 2220 len 16384
24239 220/300 Write block 3048 len 28672 data: 6 5 5 4 4 4 3
24240 221/300 Write block 2780 len 8192 data: 5 5
24241 222/300 Write block 219 len 36864 data: 6 6 6 6 5 6 6 4 3
24242 223/300 Read block 1370 len 4096
24243 224/300 Write block 4766 len 8192 data: 4 3
24244 225/300 Write block 1197 len 36864 data: 4 5 6 4 4 2 4 4 5
24245 226/300 Read block 2790 len 16384
24246 227/300 Write block 4480 len 36864 data: 4 5 5 5 5 4 4 5 4
24247 228/300 Read block 3815 len 32768
24248 229/300 Write block 2735 len 28672 data: 3 3 3 4 4 4 3
24249 230/300 Flush
24250 231/300 Read block 255 len 12288
24251 232/300 Flush
24252 233/300 Read block 1321 len 28672
24253 234/300 Flush
24254 235/300 Read block 817 len 16384
24255 236/300 Read block 3678 len 20480
24256 237/300 Write block 3890 len 12288 data: 3 2 2
24257 238/300 Write block 1449 len 24576 data: 5 6 7 6 6 6
24258 239/300 Read block 267 len 16384
24259 240/300 Read block 3902 len 36864
24260 241/300 Read block 952 len 24576
24261 242/300 Read block 1916 len 32768
24262 243/300 Flush
24263 244/300 Write block 2678 len 4096 data: 8
24264 245/300 Read block 1942 len 8192
24265 246/300 Read block 202 len 12288
24266 247/300 Write block 355 len 28672 data: 5 4 3 3 3 3 3
24267 248/300 Read block 4200 len 24576
24268 249/300 Write block 3270 len 28672 data: 6 5 5 6 5 5 3
24269 250/300 Write block 1921 len 16384 data: 5 4 4 4
24270 251/300 Read block 1520 len 40960
24271 252/300 Write block 3103 len 24576 data: 9 7 7 4 5 4
24272 253/300 Write block 3497 len 32768 data: 6 5 6 7 6 5 5 4
24273 254/300 Read block 168 len 32768
24274 255/300 Read block 3100 len 8192
24275 256/300 Write block 2361 len 32768 data: 6 6 6 5 5 3 2 2
24276 257/300 Read block 3778 len 32768
24277 258/300 Read block 4908 len 28672
24278 259/300 Write block 3086 len 28672 data: 5 5 4 4 4 4 4
24279 260/300 Write block 1487 len 28672 data: 4 4 4 4 4 5 6
24280 261/300 Read block 945 len 36864
24281 262/300 Write block 2437 len 16384 data: 3 3 2 2
24282 263/300 Flush
24283 264/300 Write block 2873 len 24576 data: 5 5 6 7 7 7
24284 265/300 Write block 1067 len 28672 data: 5 6 6 6 6 6 6
24285 266/300 Read block 2026 len 16384
24286 267/300 Write block 2343 len 4096 data: 6
24287 268/300 Flush
24288 269/300 Read block 1817 len 8192
24289 270/300 Write block 4602 len 36864 data: 6 4 4 4 4 4 6 5 6
24290 271/300 Write block 4985 len 8192 data: 6 6
24291 272/300 Write block 987 len 40960 data: 6 6 3 3 3 3 3 3 5 5
24292 273/300 Write block 4550 len 16384 data: 4 5 5 4
24293 274/300 Write block 3169 len 28672 data: 8 7 7 6 8 8 8
24294 275/300 Write block 3639 len 16384 data: 4 3 4 5
24295 276/300 Write block 272 len 20480 data: 4 4 5 3 3
24296 277/300 Read block 2595 len 36864
24297 278/300 Write block 4716 len 28672 data: 4 4 4 4 2 2 3
24298 279/300 Write block 4097 len 32768 data: 4 4 5 5 4 3 3 6
24299 280/300 Read block 170 len 24576
24300 281/300 Write block 1418 len 36864 data: 5 6 7 7 6 7 8 6 9
24301 282/300 Write block 2651 len 36864 data: 4 3 4 4 5 6 6 6 5
24302 283/300 Write block 4666 len 16384 data: 4 4 4 5
24303 284/300 Write block 4299 len 4096 data: 4
24304 285/300 Read block 4417 len 28672
24305 286/300 Write block 2849 len 32768 data: 5 9 7 5 5 7 7 9
24306 287/300 Write block 613 len 36864 data: 4 4 4 4 4 4 3 3 4
24307 288/300 Read block 4799 len 40960
24308 289/300 Flush
24309 290/300 Read block 4153 len 32768
24310 291/300 Read block 2460 len 32768
24311 292/300 Write block 153 len 8192 data: 3 3
24312 293/300 Read block 4760 len 36864
24313 294/300 Write block 3187 len 36864 data: 5 5 5 4 4 4 4 5 5
24314 295/300 Write block 877 len 28672 data: 3 3 4 3 3 3 3
24315 296/300 Write block 4117 len 40960 data: 4 4 4 6 8 9 8 6 6 5
24316 297/300 Read block 3120 len 40960
24317 298/300 Read block 2341 len 40960
24318 299/300 Read block 4046 len 24576
24319 300/300 Read block 968 len 16384
24320 301/300 Flush
24321 Main task has work to do, go find it
24322 got dsc Start(2)
24323 start 2
24324 Feb 21 06:50:44.786 INFO request completed, latency_us: 216, response_code: 204, uri: /start/cid/2, method: POST, req_id: c4638b4a-11bf-413b-99cb-720e5d17f423, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
24325 Replay: started 2, returned:Ok(())
24326 ----------------------------------------------------------------
24327 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24328 States: Active Active Offline
24329 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24330 7340 Acked Read 10 Done Done Sent false
24331 7341 Acked Flush 0 Done Done Sent false
24332 7342 Acked Read 1 Done Done Sent false
24333 7343 Acked Write 5 Done Done Sent false
24334 7344 Acked Read 10 Done Done Sent false
24335 7345 Acked Read 9 Done Done Sent false
24336 7346 Acked Read 6 Done Done Sent false
24337 7347 Acked Read 2 Done Done Sent false
24338 7348 Acked Read 3 Done Done Sent false
24339 7349 Acked Read 10 Done Done Sent false
24340 7350 Acked Write 3 Done Done Sent false
24341 7351 Acked Write 3 Done Done Sent false
24342 7352 Acked Write 8 Done Done Sent false
24343 7353 Acked Read 3 Done Done Sent false
24344 7354 Acked Flush 0 Done Done Sent false
24345 7355 Acked Read 10 Done Done Sent false
24346 7356 Acked Read 4 Done Done Sent false
24347 7357 Acked Read 9 Done Done Sent false
24348 7358 Acked Write 5 Done Done Sent false
24349 7359 Acked Flush 0 Done Done Sent false
24350 7360 Acked Read 2 Done Done Sent false
24351 7361 Acked Read 5 Done Done Sent false
24352 7362 Acked Write 6 Done Done Sent false
24353 7363 Acked Read 9 Done Done Sent false
24354 7364 Acked Flush 0 Done Done Sent false
24355 7365 Acked Write 2 Done Done Sent false
24356 7366 Acked Write 5 Done Done Sent false
24357 7367 Acked Write 9 Done Done Sent false
24358 7368 Acked Read 7 Done Done Sent false
24359 7369 Acked Write 8 Done Done Sent false
24360 7370 Acked Read 8 Done Done Sent false
24361 7371 Acked Read 7 Done Done Sent false
24362 7372 Acked Read 2 Done Done Sent false
24363 7373 Acked Read 5 Done Done Sent false
24364 7374 Acked Flush 0 Done Done Sent false
24365 7375 Acked Read 9 Done Done Sent false
24366 7376 Acked Write 8 Done Done Sent false
24367 7377 Acked Read 6 Done Done Sent false
24368 7378 Acked Flush 0 Done Done Sent false
24369 7379 Acked Read 1 Done Done Sent false
24370 7380 Acked Read 9 Done Done Sent false
24371 7381 Acked Read 8 Done Done Sent false
24372 7382 Acked Write 8 Done Done Sent false
24373 7383 Acked Write 6 Done Done Sent false
24374 7384 Acked Write 4 Done Done Sent false
24375 7385 Acked Read 4 Done Done Sent false
24376 7386 Acked Write 1 Done Done Sent false
24377 7387 Acked Read 1 Done Done Sent false
24378 7388 Acked Read 5 Done Done Sent false
24379 7389 Acked Write 6 Done Done Sent false
24380 7390 Acked Read 6 Done Done Sent false
24381 7391 Acked Read 8 Done Done Sent false
24382 7392 Acked Flush 0 Done Done Sent false
24383 7393 Acked Read 10 Done Done Sent false
24384 7394 Acked Flush 0 Done Done Sent false
24385 7395 Acked Write 10 Done Done Sent false
24386 7396 Acked Read 9 Done Done Sent false
24387 7397 Acked Read 8 Done Done Sent false
24388 7398 Acked Read 3 Done Done Sent false
24389 7399 Acked Write 7 Done Done Sent false
24390 7400 Acked Read 10 Done Done Sent false
24391 7401 Acked Flush 0 Done Done Sent false
24392 7402 Acked Read 2 Done Done Sent false
24393 7403 Acked Read 5 Done Done Sent false
24394 7404 Acked Read 8 Done Done Sent false
24395 7405 Acked Write 3 Done Done Sent false
24396 7406 Acked Read 2 Done Done Sent false
24397 7407 Acked Write 5 Done Done Sent false
24398 7408 Acked Read 3 Done Done Sent false
24399 7409 Acked Write 8 Done Done Sent false
24400 7410 Acked Flush 0 Done Done Sent false
24401 7411 Acked Read 4 Done Done Sent false
24402 7412 Acked Flush 0 Done Done Sent false
24403 7413 Acked Write 3 Done Done Sent false
24404 7414 Acked Read 8 Done Done Sent false
24405 7415 Acked Write 2 Done Done Sent false
24406 7416 Acked Read 7 Done Done Sent false
24407 7417 Acked Read 3 Done Done Sent false
24408 7418 Acked Write 2 Done Done Sent false
24409 7419 Acked Write 4 Done Done Sent false
24410 7420 Acked Read 10 Done Done Sent false
24411 7421 Acked Read 4 Done Done Sent false
24412 7422 Acked Flush 0 Done Done Sent false
24413 7423 Acked Read 2 Done Done Sent false
24414 7424 Acked Write 4 Done Done Sent false
24415 7425 Acked Flush 0 Done Done Sent false
24416 7426 Acked Read 8 Done Done Sent false
24417 7427 Acked Read 1 Done Done Sent false
24418 7428 Acked Read 10 Done Done Sent false
24419 7429 Acked Read 7 Done Done Sent false
24420 7430 Acked Read 2 Done Done Sent false
24421 7431 Acked Read 5 Done Done Sent false
24422 7432 Acked Read 2 Done Done Sent false
24423 7433 Acked Read 10 Done Done Sent false
24424 7434 Acked Flush 0 Done Done Sent false
24425 7435 Acked Read 9 Done Done Sent false
24426 7436 Acked Read 8 Done Done Sent false
24427 7437 Acked Write 2 Done Done Sent false
24428 7438 Acked Write 3 Done Done Sent false
24429 7439 Acked Read 2 Done Done Sent false
24430 7440 Acked Read 9 Done Done Sent false
24431 7441 Acked Read 3 Done Done Sent false
24432 7442 Acked Flush 0 Done Done Sent false
24433 7443 Acked Write 9 Done Done Sent false
24434 7444 Acked Write 3 Done Done Sent false
24435 7445 Acked Write 3 Done Done Sent false
24436 7446 Acked Read 8 Done Done Sent false
24437 7447 Acked Read 8 Done Done Sent false
24438 7448 Acked Write 10 Done Done Sent false
24439 7449 Acked Flush 0 Done Done Sent false
24440 7450 Acked Read 4 Done Done Sent false
24441 7451 Acked Read 4 Done Done Sent false
24442 7452 Acked Write 10 Done Done Sent false
24443 7453 Acked Read 4 Done Done Sent false
24444 7454 Acked Read 3 Done Done Sent false
24445 7455 Acked Write 7 Done Done Sent false
24446 7456 Acked Write 3 Done Done Sent false
24447 7457 Acked Write 3 Done Done Sent false
24448 7458 Acked Write 5 Done Done Sent false
24449 7459 Acked Write 5 Done Done Sent false
24450 7460 Acked Read 9 Done Done Sent false
24451 7461 Acked Write 9 Done Done Sent false
24452 7462 Acked Read 9 Done Done Sent false
24453 7463 Acked Write 9 Done Done Sent false
24454 7464 Acked Read 1 Done Done Sent false
24455 7465 Acked Write 1 Done Done Sent false
24456 7466 Acked Write 3 Done Done Sent false
24457 7467 Acked Write 5 Done Done Sent false
24458 7468 Acked Write 10 Done Done Sent false
24459 7469 Acked Read 8 Done Done Sent false
24460 7470 Acked Flush 0 Done Done Sent false
24461 7471 Acked Read 3 Done Done Sent false
24462 7472 Acked Read 4 Done Done Sent false
24463 7473 Acked Read 1 Done Done Sent false
24464 7474 Acked Write 1 Done Done Sent false
24465 7475 Acked Read 4 Done Done Sent false
24466 7476 Acked Write 8 Done Done Sent false
24467 7477 Acked Read 4 Done Done Sent false
24468 7478 Acked Read 2 Done Done Sent false
24469 7479 Acked Write 1 Done Done Sent false
24470 7480 Acked Read 10 Done Done Sent false
24471 7481 Acked Read 10 Done Done Sent false
24472 7482 Acked Read 5 Done Done Sent false
24473 7483 Acked Read 7 Done Done Sent false
24474 7484 Acked Read 8 Done Done Sent false
24475 7485 Acked Read 6 Done Done Sent false
24476 7486 Acked Write 6 Done Done Sent false
24477 7487 Acked Flush 0 Done Done Sent false
24478 7488 Acked Write 6 Done Done Sent false
24479 7489 Acked Write 10 Done Done Sent false
24480 7490 Acked Write 5 Done Done Sent false
24481 7491 Acked Read 4 Done Done Sent false
24482 7492 Acked Read 1 Done Done Sent false
24483 7493 Acked Read 10 Done Done Sent false
24484 7494 Acked Read 9 Done Done Sent false
24485 7495 Acked Write 2 Done Done Sent false
24486 7496 Acked Read 8 Done Done Sent false
24487 7497 Acked Write 5 Done Done Sent false
24488 7498 Acked Read 4 Done Done Sent false
24489 7499 Acked Write 7 Done Done Sent false
24490 7500 Acked Flush 0 Done Done Sent false
24491 7501 Acked Flush 0 Done Done Sent false
24492 7502 Acked Write 10 Done Done Sent false
24493 7503 Acked Read 8 Done Done Sent false
24494 7504 Acked Write 8 Done Done Sent false
24495 7505 Acked Read 1 Done Done Sent false
24496 7506 Acked Write 3 Done Done Sent false
24497 7507 Acked Write 4 Done Done Sent false
24498 7508 Acked Write 6 Done Done Sent false
24499 7509 Acked Write 7 Done Done Sent false
24500 7510 Acked Read 8 Done Done Sent false
24501 7511 Acked Read 9 Done Done Sent false
24502 7512 Acked Write 4 Done Done Sent false
24503 7513 Acked Write 10 Done Done Sent false
24504 7514 Acked Write 2 Done Done Sent false
24505 7515 Acked Read 1 Done Done Sent false
24506 7516 Acked Read 9 Done Done Sent false
24507 7517 Acked Write 9 Done Done Sent false
24508 7518 Acked Write 5 Done Done Sent false
24509 7519 Acked Read 5 Done Done Sent false
24510 7520 Acked Flush 0 Done Done Sent false
24511 7521 Acked Read 10 Done Done Sent false
24512 7522 Acked Read 10 Done Done Sent false
24513 7523 Acked Write 10 Done Done Sent false
24514 7524 Acked Read 7 Done Done Sent false
24515 7525 Acked Read 5 Done Done Sent false
24516 7526 Acked Write 6 Done Done Sent false
24517 7527 Acked Flush 0 Done Done Sent false
24518 7528 Acked Read 7 Done Done Sent false
24519 7529 Acked Write 5 Done Done Sent false
24520 7530 Acked Read 3 Done Done Sent false
24521 7531 Acked Write 7 Done Done Sent false
24522 7532 Acked Read 8 Done Done Sent false
24523 7533 Acked Read 2 Done Done Sent false
24524 7534 Acked Write 10 Done Done Sent false
24525 7535 Acked Write 6 Done Done Sent false
24526 7536 Acked Read 7 Done Done Sent false
24527 7537 Acked Read 7 Done Done Sent false
24528 7538 Acked Write 4 Done Done Sent false
24529 7539 Acked Read 10 Done Done Sent false
24530 7540 Acked Write 9 Done Done Sent false
24531 7541 Acked Read 5 Done Done Sent false
24532 7542 Acked Write 1 Done Done Sent false
24533 7543 Acked Write 7 Done Done Sent false
24534 7544 Acked Write 2 Done Done Sent false
24535 7545 Acked Read 2 Done Done Sent false
24536 7546 Acked Read 6 Done Done Sent false
24537 7547 Acked Write 1 Done Done Sent false
24538 7548 Acked Write 8 Done Done Sent false
24539 7549 Acked Write 3 Done Done Sent false
24540 7550 Acked Read 3 Done Done Sent false
24541 7551 Acked Write 9 Done Done Sent false
24542 7552 Acked Write 9 Done Done Sent false
24543 7553 Acked Write 4 Done Done Sent false
24544 7554 Acked Flush 0 Done Done Sent false
24545 7555 Acked Flush 0 Done Done Sent false
24546 7556 Acked Write 3 Done Done Sent false
24547 7557 Acked Write 5 Done Done Sent false
24548 7558 Acked Read 4 Done Done Sent false
24549 7559 Acked Write 7 Done Done Sent false
24550 7560 Acked Write 2 Done Done Sent false
24551 7561 Acked Write 9 Done Done Sent false
24552 7562 Acked Read 1 Done Done Sent false
24553 7563 Acked Write 2 Done Done Sent false
24554 7564 Acked Write 9 Done Done Sent false
24555 7565 Acked Read 4 Done Done Sent false
24556 7566 Acked Write 9 Done Done Sent false
24557 7567 Acked Read 8 Done Done Sent false
24558 7568 Acked Write 7 Done Done Sent false
24559 7569 Acked Flush 0 Done Done Sent false
24560 7570 Acked Read 3 Done Done Sent false
24561 7571 Acked Flush 0 Done Done Sent false
24562 7572 Acked Read 7 Done Done Sent false
24563 7573 Acked Flush 0 Done Done Sent false
24564 7574 Acked Read 4 Done Done Sent false
24565 7575 Acked Read 5 Done Done Sent false
24566 7576 Acked Write 3 Done Done Sent false
24567 7577 Acked Write 6 Done Done Sent false
24568 7578 Acked Read 4 Done Done Sent false
24569 7579 Acked Read 9 Done Done Sent false
24570 7580 Acked Read 6 Done Done Sent false
24571 7581 Acked Read 8 Done Done Sent false
24572 7582 Acked Flush 0 Done Done Sent false
24573 7583 Acked Write 1 Done Done Sent false
24574 7584 Acked Read 2 Done Done Sent false
24575 7585 Acked Read 3 Done Done Sent false
24576 7586 Acked Write 7 Done Done Sent false
24577 7587 Acked Read 6 Done Done Sent false
24578 7588 Acked Write 7 Done Done Sent false
24579 7589 Acked Write 4 Done Done Sent false
24580 7590 Acked Read 10 Done Done Sent false
24581 7591 Acked Write 6 Done Done Sent false
24582 7592 Acked Write 8 Done Done Sent false
24583 7593 Acked Read 8 Done Done Sent false
24584 7594 Acked Read 2 Done Done Sent false
24585 7595 Acked Write 8 Done Done Sent false
24586 7596 Acked Read 8 Done Done Sent false
24587 7597 Acked Read 7 Done Done Sent false
24588 7598 Acked Write 7 Done Done Sent false
24589 7599 Acked Write 7 Done Done Sent false
24590 7600 Acked Read 9 Done Done Sent false
24591 7601 Acked Write 4 Done Done Sent false
24592 7602 Acked Flush 0 Done Done Sent false
24593 7603 Acked Write 6 Done Done Sent false
24594 7604 Acked Write 7 Done Done Sent false
24595 7605 Acked Read 4 Done Done Sent false
24596 7606 Acked Write 1 Done Done Sent false
24597 7607 Acked Flush 0 Done Done Sent false
24598 7608 Acked Read 2 Done Done Sent false
24599 7609 Acked Write 9 Done Done Sent false
24600 7610 Acked Write 2 Done Done Sent false
24601 7611 Acked Write 10 Done Done Sent false
24602 7612 Acked Write 4 Done Done Sent false
24603 7613 Acked Write 7 Done Done Sent false
24604 7614 Acked Write 4 Done Done Sent false
24605 7615 Acked Write 5 Done Done Sent false
24606 7616 Acked Read 9 Done Done Sent false
24607 7617 Acked Write 7 Done Done Sent false
24608 7618 Acked Write 8 Done Done Sent false
24609 7619 Acked Read 6 Done Done Sent false
24610 7620 Acked Write 9 Done Done Sent false
24611 7621 Acked Write 9 Done Done Sent false
24612 7622 Acked Write 4 Done Done Sent false
24613 7623 Acked Write 1 Done Done Sent false
24614 7624 Acked Read 7 Done Done Sent false
24615 7625 Acked Write 8 Done Done Sent false
24616 7626 Acked Write 9 Done Done Sent false
24617 7627 Acked Read 10 Done Done Sent false
24618 7628 Acked Flush 0 Done Done Sent false
24619 7629 Acked Read 8 Done Done Sent false
24620 7630 Acked Read 8 Done Done Sent false
24621 7631 Acked Write 2 Done Done Sent false
24622 7632 Acked Read 9 Done Done Sent false
24623 7633 Acked Write 9 Done Done Sent false
24624 7634 Acked Write 7 Done Done Sent false
24625 7635 Acked Write 10 Done Done Sent false
24626 7636 Acked Read 10 Done Done Sent false
24627 7637 Acked Read 10 Done Done Sent false
24628 7638 Acked Read 6 Done Done Sent false
24629 7639 Acked Read 4 Done Done Sent false
24630 7640 Acked Flush 0 Done Done Sent false
24631 STATES DS:0 DS:1 DS:2 TOTAL
24632 Sent 0 0 301 301
24633 Done 301 301 0 602
24634 Skipped 0 0 0 0
24635 Error 0 0 0 0
24636 Last Flush: Some(JobId(7640)) Some(JobId(7640)) Some(JobId(7339))
24637 Downstairs last five completed: 7339 7338 7337 7336 7335
24638 Upstairs last five completed: 7640 7639 7638 7637 7636
24639 CLIENT: Up:0 ds:301 act:2
246402025-02-21T06:50:45.289ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
246412025-02-21T06:50:46.532ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
24642 [8830] got start action
24643 [8830] I am going to restart
24644 Starting downstairs at port 8830
24645 Make output file at "/tmp/dsc/downstairs-8830.txt"
24646 [8830][2] reports Starting
24647 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1228)
24648 [8830][2] reports Running
246492025-02-21T06:50:47.155ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
246502025-02-21T06:50:48.157ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
24651 ----------------------------------------------------------------
24652 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24653 States: Active Active Offline
24654 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24655 7340 Acked Read 10 Done Done Sent false
24656 7341 Acked Flush 0 Done Done Sent false
24657 7342 Acked Read 1 Done Done Sent false
24658 7343 Acked Write 5 Done Done Sent false
24659 7344 Acked Read 10 Done Done Sent false
24660 7345 Acked Read 9 Done Done Sent false
24661 7346 Acked Read 6 Done Done Sent false
24662 7347 Acked Read 2 Done Done Sent false
24663 7348 Acked Read 3 Done Done Sent false
24664 7349 Acked Read 10 Done Done Sent false
24665 7350 Acked Write 3 Done Done Sent false
24666 7351 Acked Write 3 Done Done Sent false
24667 7352 Acked Write 8 Done Done Sent false
24668 7353 Acked Read 3 Done Done Sent false
24669 7354 Acked Flush 0 Done Done Sent false
24670 7355 Acked Read 10 Done Done Sent false
24671 7356 Acked Read 4 Done Done Sent false
24672 7357 Acked Read 9 Done Done Sent false
24673 7358 Acked Write 5 Done Done Sent false
24674 7359 Acked Flush 0 Done Done Sent false
24675 7360 Acked Read 2 Done Done Sent false
24676 7361 Acked Read 5 Done Done Sent false
24677 7362 Acked Write 6 Done Done Sent false
24678 7363 Acked Read 9 Done Done Sent false
24679 7364 Acked Flush 0 Done Done Sent false
24680 7365 Acked Write 2 Done Done Sent false
24681 7366 Acked Write 5 Done Done Sent false
24682 7367 Acked Write 9 Done Done Sent false
24683 7368 Acked Read 7 Done Done Sent false
24684 7369 Acked Write 8 Done Done Sent false
24685 7370 Acked Read 8 Done Done Sent false
24686 7371 Acked Read 7 Done Done Sent false
24687 7372 Acked Read 2 Done Done Sent false
24688 7373 Acked Read 5 Done Done Sent false
24689 7374 Acked Flush 0 Done Done Sent false
24690 7375 Acked Read 9 Done Done Sent false
24691 7376 Acked Write 8 Done Done Sent false
24692 7377 Acked Read 6 Done Done Sent false
24693 7378 Acked Flush 0 Done Done Sent false
24694 7379 Acked Read 1 Done Done Sent false
24695 7380 Acked Read 9 Done Done Sent false
24696 7381 Acked Read 8 Done Done Sent false
24697 7382 Acked Write 8 Done Done Sent false
24698 7383 Acked Write 6 Done Done Sent false
24699 7384 Acked Write 4 Done Done Sent false
24700 7385 Acked Read 4 Done Done Sent false
24701 7386 Acked Write 1 Done Done Sent false
24702 7387 Acked Read 1 Done Done Sent false
24703 7388 Acked Read 5 Done Done Sent false
24704 7389 Acked Write 6 Done Done Sent false
24705 7390 Acked Read 6 Done Done Sent false
24706 7391 Acked Read 8 Done Done Sent false
24707 7392 Acked Flush 0 Done Done Sent false
24708 7393 Acked Read 10 Done Done Sent false
24709 7394 Acked Flush 0 Done Done Sent false
24710 7395 Acked Write 10 Done Done Sent false
24711 7396 Acked Read 9 Done Done Sent false
24712 7397 Acked Read 8 Done Done Sent false
24713 7398 Acked Read 3 Done Done Sent false
24714 7399 Acked Write 7 Done Done Sent false
24715 7400 Acked Read 10 Done Done Sent false
24716 7401 Acked Flush 0 Done Done Sent false
24717 7402 Acked Read 2 Done Done Sent false
24718 7403 Acked Read 5 Done Done Sent false
24719 7404 Acked Read 8 Done Done Sent false
24720 7405 Acked Write 3 Done Done Sent false
24721 7406 Acked Read 2 Done Done Sent false
24722 7407 Acked Write 5 Done Done Sent false
24723 7408 Acked Read 3 Done Done Sent false
24724 7409 Acked Write 8 Done Done Sent false
24725 7410 Acked Flush 0 Done Done Sent false
24726 7411 Acked Read 4 Done Done Sent false
24727 7412 Acked Flush 0 Done Done Sent false
24728 7413 Acked Write 3 Done Done Sent false
24729 7414 Acked Read 8 Done Done Sent false
24730 7415 Acked Write 2 Done Done Sent false
24731 7416 Acked Read 7 Done Done Sent false
24732 7417 Acked Read 3 Done Done Sent false
24733 7418 Acked Write 2 Done Done Sent false
24734 7419 Acked Write 4 Done Done Sent false
24735 7420 Acked Read 10 Done Done Sent false
24736 7421 Acked Read 4 Done Done Sent false
24737 7422 Acked Flush 0 Done Done Sent false
24738 7423 Acked Read 2 Done Done Sent false
24739 7424 Acked Write 4 Done Done Sent false
24740 7425 Acked Flush 0 Done Done Sent false
24741 7426 Acked Read 8 Done Done Sent false
24742 7427 Acked Read 1 Done Done Sent false
24743 7428 Acked Read 10 Done Done Sent false
24744 7429 Acked Read 7 Done Done Sent false
24745 7430 Acked Read 2 Done Done Sent false
24746 7431 Acked Read 5 Done Done Sent false
24747 7432 Acked Read 2 Done Done Sent false
24748 7433 Acked Read 10 Done Done Sent false
24749 7434 Acked Flush 0 Done Done Sent false
24750 7435 Acked Read 9 Done Done Sent false
24751 7436 Acked Read 8 Done Done Sent false
24752 7437 Acked Write 2 Done Done Sent false
24753 7438 Acked Write 3 Done Done Sent false
24754 7439 Acked Read 2 Done Done Sent false
24755 7440 Acked Read 9 Done Done Sent false
24756 7441 Acked Read 3 Done Done Sent false
24757 7442 Acked Flush 0 Done Done Sent false
24758 7443 Acked Write 9 Done Done Sent false
24759 7444 Acked Write 3 Done Done Sent false
24760 7445 Acked Write 3 Done Done Sent false
24761 7446 Acked Read 8 Done Done Sent false
24762 7447 Acked Read 8 Done Done Sent false
24763 7448 Acked Write 10 Done Done Sent false
24764 7449 Acked Flush 0 Done Done Sent false
24765 7450 Acked Read 4 Done Done Sent false
24766 7451 Acked Read 4 Done Done Sent false
24767 7452 Acked Write 10 Done Done Sent false
24768 7453 Acked Read 4 Done Done Sent false
24769 7454 Acked Read 3 Done Done Sent false
24770 7455 Acked Write 7 Done Done Sent false
24771 7456 Acked Write 3 Done Done Sent false
24772 7457 Acked Write 3 Done Done Sent false
24773 7458 Acked Write 5 Done Done Sent false
24774 7459 Acked Write 5 Done Done Sent false
24775 7460 Acked Read 9 Done Done Sent false
24776 7461 Acked Write 9 Done Done Sent false
24777 7462 Acked Read 9 Done Done Sent false
24778 7463 Acked Write 9 Done Done Sent false
24779 7464 Acked Read 1 Done Done Sent false
24780 7465 Acked Write 1 Done Done Sent false
24781 7466 Acked Write 3 Done Done Sent false
24782 7467 Acked Write 5 Done Done Sent false
24783 7468 Acked Write 10 Done Done Sent false
24784 7469 Acked Read 8 Done Done Sent false
24785 7470 Acked Flush 0 Done Done Sent false
24786 7471 Acked Read 3 Done Done Sent false
24787 7472 Acked Read 4 Done Done Sent false
24788 7473 Acked Read 1 Done Done Sent false
24789 7474 Acked Write 1 Done Done Sent false
24790 7475 Acked Read 4 Done Done Sent false
24791 7476 Acked Write 8 Done Done Sent false
24792 7477 Acked Read 4 Done Done Sent false
24793 7478 Acked Read 2 Done Done Sent false
24794 7479 Acked Write 1 Done Done Sent false
24795 7480 Acked Read 10 Done Done Sent false
24796 7481 Acked Read 10 Done Done Sent false
24797 7482 Acked Read 5 Done Done Sent false
24798 7483 Acked Read 7 Done Done Sent false
24799 7484 Acked Read 8 Done Done Sent false
24800 7485 Acked Read 6 Done Done Sent false
24801 7486 Acked Write 6 Done Done Sent false
24802 7487 Acked Flush 0 Done Done Sent false
24803 7488 Acked Write 6 Done Done Sent false
24804 7489 Acked Write 10 Done Done Sent false
24805 7490 Acked Write 5 Done Done Sent false
24806 7491 Acked Read 4 Done Done Sent false
24807 7492 Acked Read 1 Done Done Sent false
24808 7493 Acked Read 10 Done Done Sent false
24809 7494 Acked Read 9 Done Done Sent false
24810 7495 Acked Write 2 Done Done Sent false
24811 7496 Acked Read 8 Done Done Sent false
24812 7497 Acked Write 5 Done Done Sent false
24813 7498 Acked Read 4 Done Done Sent false
24814 7499 Acked Write 7 Done Done Sent false
24815 7500 Acked Flush 0 Done Done Sent false
24816 7501 Acked Flush 0 Done Done Sent false
24817 7502 Acked Write 10 Done Done Sent false
24818 7503 Acked Read 8 Done Done Sent false
24819 7504 Acked Write 8 Done Done Sent false
24820 7505 Acked Read 1 Done Done Sent false
24821 7506 Acked Write 3 Done Done Sent false
24822 7507 Acked Write 4 Done Done Sent false
24823 7508 Acked Write 6 Done Done Sent false
24824 7509 Acked Write 7 Done Done Sent false
24825 7510 Acked Read 8 Done Done Sent false
24826 7511 Acked Read 9 Done Done Sent false
24827 7512 Acked Write 4 Done Done Sent false
24828 7513 Acked Write 10 Done Done Sent false
24829 7514 Acked Write 2 Done Done Sent false
24830 7515 Acked Read 1 Done Done Sent false
24831 7516 Acked Read 9 Done Done Sent false
24832 7517 Acked Write 9 Done Done Sent false
24833 7518 Acked Write 5 Done Done Sent false
24834 7519 Acked Read 5 Done Done Sent false
24835 7520 Acked Flush 0 Done Done Sent false
24836 7521 Acked Read 10 Done Done Sent false
24837 7522 Acked Read 10 Done Done Sent false
24838 7523 Acked Write 10 Done Done Sent false
24839 7524 Acked Read 7 Done Done Sent false
24840 7525 Acked Read 5 Done Done Sent false
24841 7526 Acked Write 6 Done Done Sent false
24842 7527 Acked Flush 0 Done Done Sent false
24843 7528 Acked Read 7 Done Done Sent false
24844 7529 Acked Write 5 Done Done Sent false
24845 7530 Acked Read 3 Done Done Sent false
24846 7531 Acked Write 7 Done Done Sent false
24847 7532 Acked Read 8 Done Done Sent false
24848 7533 Acked Read 2 Done Done Sent false
24849 7534 Acked Write 10 Done Done Sent false
24850 7535 Acked Write 6 Done Done Sent false
24851 7536 Acked Read 7 Done Done Sent false
24852 7537 Acked Read 7 Done Done Sent false
24853 7538 Acked Write 4 Done Done Sent false
24854 7539 Acked Read 10 Done Done Sent false
24855 7540 Acked Write 9 Done Done Sent false
24856 7541 Acked Read 5 Done Done Sent false
24857 7542 Acked Write 1 Done Done Sent false
24858 7543 Acked Write 7 Done Done Sent false
24859 7544 Acked Write 2 Done Done Sent false
24860 7545 Acked Read 2 Done Done Sent false
24861 7546 Acked Read 6 Done Done Sent false
24862 7547 Acked Write 1 Done Done Sent false
24863 7548 Acked Write 8 Done Done Sent false
24864 7549 Acked Write 3 Done Done Sent false
24865 7550 Acked Read 3 Done Done Sent false
24866 7551 Acked Write 9 Done Done Sent false
24867 7552 Acked Write 9 Done Done Sent false
24868 7553 Acked Write 4 Done Done Sent false
24869 7554 Acked Flush 0 Done Done Sent false
24870 7555 Acked Flush 0 Done Done Sent false
24871 7556 Acked Write 3 Done Done Sent false
24872 7557 Acked Write 5 Done Done Sent false
24873 7558 Acked Read 4 Done Done Sent false
24874 7559 Acked Write 7 Done Done Sent false
24875 7560 Acked Write 2 Done Done Sent false
24876 7561 Acked Write 9 Done Done Sent false
24877 7562 Acked Read 1 Done Done Sent false
24878 7563 Acked Write 2 Done Done Sent false
24879 7564 Acked Write 9 Done Done Sent false
24880 7565 Acked Read 4 Done Done Sent false
24881 7566 Acked Write 9 Done Done Sent false
24882 7567 Acked Read 8 Done Done Sent false
24883 7568 Acked Write 7 Done Done Sent false
24884 7569 Acked Flush 0 Done Done Sent false
24885 7570 Acked Read 3 Done Done Sent false
24886 7571 Acked Flush 0 Done Done Sent false
24887 7572 Acked Read 7 Done Done Sent false
24888 7573 Acked Flush 0 Done Done Sent false
24889 7574 Acked Read 4 Done Done Sent false
24890 7575 Acked Read 5 Done Done Sent false
24891 7576 Acked Write 3 Done Done Sent false
24892 7577 Acked Write 6 Done Done Sent false
24893 7578 Acked Read 4 Done Done Sent false
24894 7579 Acked Read 9 Done Done Sent false
24895 7580 Acked Read 6 Done Done Sent false
24896 7581 Acked Read 8 Done Done Sent false
24897 7582 Acked Flush 0 Done Done Sent false
24898 7583 Acked Write 1 Done Done Sent false
24899 7584 Acked Read 2 Done Done Sent false
24900 7585 Acked Read 3 Done Done Sent false
24901 7586 Acked Write 7 Done Done Sent false
24902 7587 Acked Read 6 Done Done Sent false
24903 7588 Acked Write 7 Done Done Sent false
24904 7589 Acked Write 4 Done Done Sent false
24905 7590 Acked Read 10 Done Done Sent false
24906 7591 Acked Write 6 Done Done Sent false
24907 7592 Acked Write 8 Done Done Sent false
24908 7593 Acked Read 8 Done Done Sent false
24909 7594 Acked Read 2 Done Done Sent false
24910 7595 Acked Write 8 Done Done Sent false
24911 7596 Acked Read 8 Done Done Sent false
24912 7597 Acked Read 7 Done Done Sent false
24913 7598 Acked Write 7 Done Done Sent false
24914 7599 Acked Write 7 Done Done Sent false
24915 7600 Acked Read 9 Done Done Sent false
24916 7601 Acked Write 4 Done Done Sent false
24917 7602 Acked Flush 0 Done Done Sent false
24918 7603 Acked Write 6 Done Done Sent false
24919 7604 Acked Write 7 Done Done Sent false
24920 7605 Acked Read 4 Done Done Sent false
24921 7606 Acked Write 1 Done Done Sent false
24922 7607 Acked Flush 0 Done Done Sent false
24923 7608 Acked Read 2 Done Done Sent false
24924 7609 Acked Write 9 Done Done Sent false
24925 7610 Acked Write 2 Done Done Sent false
24926 7611 Acked Write 10 Done Done Sent false
24927 7612 Acked Write 4 Done Done Sent false
24928 7613 Acked Write 7 Done Done Sent false
24929 7614 Acked Write 4 Done Done Sent false
24930 7615 Acked Write 5 Done Done Sent false
24931 7616 Acked Read 9 Done Done Sent false
24932 7617 Acked Write 7 Done Done Sent false
24933 7618 Acked Write 8 Done Done Sent false
24934 7619 Acked Read 6 Done Done Sent false
24935 7620 Acked Write 9 Done Done Sent false
24936 7621 Acked Write 9 Done Done Sent false
24937 7622 Acked Write 4 Done Done Sent false
24938 7623 Acked Write 1 Done Done Sent false
24939 7624 Acked Read 7 Done Done Sent false
24940 7625 Acked Write 8 Done Done Sent false
24941 7626 Acked Write 9 Done Done Sent false
24942 7627 Acked Read 10 Done Done Sent false
24943 7628 Acked Flush 0 Done Done Sent false
24944 7629 Acked Read 8 Done Done Sent false
24945 7630 Acked Read 8 Done Done Sent false
24946 7631 Acked Write 2 Done Done Sent false
24947 7632 Acked Read 9 Done Done Sent false
24948 7633 Acked Write 9 Done Done Sent false
24949 7634 Acked Write 7 Done Done Sent false
24950 7635 Acked Write 10 Done Done Sent false
24951 7636 Acked Read 10 Done Done Sent false
24952 7637 Acked Read 10 Done Done Sent false
24953 7638 Acked Read 6 Done Done Sent false
24954 7639 Acked Read 4 Done Done Sent false
24955 7640 Acked Flush 0 Done Done Sent false
24956 STATES DS:0 DS:1 DS:2 TOTAL
24957 Sent 0 0 301 301
24958 Done 301 301 0 602
24959 Skipped 0 0 0 0
24960 Error 0 0 0 0
24961 Last Flush: Some(JobId(7640)) Some(JobId(7640)) Some(JobId(7339))
24962 Downstairs last five completed: 7339 7338 7337 7336 7335
24963 Upstairs last five completed: 7640 7639 7638 7637 7636
24964 CLIENT: Up:0 ds:301 act:2
249652025-02-21T06:50:49.290ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249662025-02-21T06:50:50.159ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249672025-02-21T06:50:50.540ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249682025-02-21T06:50:50.540ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51997) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249692025-02-21T06:50:50.541ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249702025-02-21T06:50:50.542ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249712025-02-21T06:50:50.542ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249722025-02-21T06:50:50.542ZINFOcrucible: send last flush ID to this DS: Some(JobId(7339)) = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249732025-02-21T06:50:50.542ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249742025-02-21T06:50:50.542ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7339)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249752025-02-21T06:50:50.542ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7339)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
24976 ----------------------------------------------------------------
24977 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24978 Downstairs last five completed: 7640 7639 7638 7637 7636
24979 Upstairs last five completed: 7640 7639 7638 7637 7636
24980 CLIENT: Up:0 ds:0 act:3
24981 Replay: All jobs finished
24982 Main task has work to do, go find it
24983 got dsc Stop(2)
24984 stop 2
24985 Feb 21 06:50:52.792 INFO request completed, [8830] Got stop action so:false kr:false
24986 latency_us: 379, response_code: 204, uri: /stop/cid/2, method: POST, req_id: cea4b44c-90db-4ffd-816f-eb947b065f75, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
24987 Feb 21 06:50:52.793 INFO request completed, latency_us: 121, response_code: 200, uri: /state/cid/2, method: GET, req_id: 54e224a2-aecb-4c31-89ba-ec7c7c1d357b, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
249882025-02-21T06:50:52.793ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249892025-02-21T06:50:52.793ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249902025-02-21T06:50:52.793ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
249912025-02-21T06:50:52.793ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
24992 [8830] Exited with: ExitStatus(unix_wait_status(9))
24993 [8830][2] reports Exit
24994 Feb 21 06:50:56.794 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/2, method: GET, req_id: 128d6b7d-2d57-4c64-a819-e8737108b7a8, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
24995 001/300 Read block 497 len 40960
24996 002/300 Read block 27 len 20480
24997 003/300 Write block 2562 len 4096 data: 7
24998 004/300 Flush
24999 005/300 Read block 3519 len 8192
25000 006/300 Read block 1497 len 12288
25001 007/300 Write block 3164 len 28672 data: 6 6 7 8 7 9 8
25002 008/300 Read block 2390 len 20480
25003 009/300 Read block 1648 len 20480
25004 010/300 Read block 1651 len 8192
25005 011/300 Flush
25006 012/300 Write block 3751 len 24576 data: 5 4 4 5 5 4
25007 013/300 Read block 2589 len 12288
25008 014/300 Read block 2923 len 36864
25009 015/300 Read block 1958 len 28672
25010 016/300 Read block 2515 len 8192
25011 017/300 Write block 1332 len 28672 data: 4 3 3 4 4 3 2
25012 018/300 Write block 2348 len 32768 data: 5 4 3 3 3 5 5 4
25013 019/300 Read block 3240 len 40960
25014 020/300 Read block 541 len 40960
25015 021/300 Read block 4118 len 28672
25016 022/300 Write block 640 len 36864 data: 6 6 6 4 5 4 6 6 4
25017 023/300 Read block 607 len 24576
25018 024/300 Write block 3646 len 20480 data: 5 5 5 5 4
25019 025/300 Write block 4782 len 24576 data: 4 3 3 4 4 4
25020 026/300 Read block 2120 len 4096
25021 027/300 Read block 1354 len 8192
25022 028/300 Read block 3948 len 24576
25023 029/300 Write block 113 len 12288 data: 3 3 5
25024 030/300 Read block 4776 len 32768
25025 031/300 Read block 4079 len 28672
25026 032/300 Write block 3990 len 36864 data: 5 5 5 5 4 2 2 3 3
25027 033/300 Read block 476 len 12288
25028 034/300 Read block 2975 len 36864
25029 035/300 Read block 826 len 40960
25030 036/300 Read block 3149 len 28672
25031 037/300 Read block 51 len 12288
25032 038/300 Read block 1401 len 4096
25033 039/300 Write block 230 len 4096 data: 4
25034 040/300 Read block 2603 len 28672
25035 041/300 Write block 1327 len 4096 data: 2
25036 042/300 Read block 4232 len 24576
25037 043/300 Read block 4862 len 16384
25038 044/300 Write block 4951 len 16384 data: 5 5 3 3
25039 045/300 Write block 4030 len 32768 data: 3 3 6 7 6 7 8 7
25040 046/300 Write block 187 len 4096 data: 6
25041 047/300 Flush
25042 048/300 Write block 2629 len 4096 data: 6
25043 049/300 Read block 4562 len 16384
25044 050/300 Read block 2419 len 12288
25045 051/300 Write block 2530 len 8192 data: 3 3
25046 052/300 Write block 4305 len 40960 data: 5 4 4 5 5 5 5 6 5 4
25047 053/300 Write block 1323 len 40960 data: 3 3 2 2 3 2 4 4 5 5
25048 054/300 Write block 578 len 12288 data: 4 5 5
25049 055/300 Read block 2006 len 4096
25050 056/300 Read block 308 len 36864
25051 057/300 Read block 3426 len 8192
25052 058/300 Read block 379 len 4096
25053 059/300 Read block 939 len 12288
25054 060/300 Read block 265 len 12288
25055 061/300 Read block 4892 len 32768
25056 062/300 Write block 2806 len 4096 data: 6
25057 063/300 Write block 3034 len 32768 data: 9 9 9 8 8 8 8 6
25058 064/300 Read block 695 len 40960
25059 065/300 Read block 3677 len 36864
25060 066/300 Read block 4173 len 12288
25061 067/300 Read block 1721 len 24576
25062 068/300 Flush
25063 069/300 Read block 3040 len 20480
25064 070/300 Write block 4037 len 24576 data: 8 7 6 5 5 3
25065 071/300 Write block 2557 len 36864 data: 4 5 5 7 7 8 6 6 5
25066 072/300 Read block 3915 len 28672
25067 073/300 Flush
25068 074/300 Write block 1937 len 20480 data: 2 3 3 4 4
25069 075/300 Read block 216 len 40960
25070 076/300 Flush
25071 077/300 Flush
25072 078/300 Flush
25073 079/300 Write block 2565 len 4096 data: 6
25074 080/300 Write block 4133 len 4096 data: 2
25075 081/300 Read block 2576 len 36864
25076 082/300 Write block 652 len 24576 data: 6 5 5 5 5 5
25077 083/300 Write block 2718 len 32768 data: 6 4 4 5 3 3 3 3
25078 084/300 Read block 3941 len 36864
25079 085/300 Write block 4228 len 20480 data: 5 4 3 3 3
25080 086/300 Read block 2291 len 36864
25081 087/300 Write block 3135 len 36864 data: 12 14 13 12 12 11 9 7 7
25082 088/300 Write block 4566 len 40960 data: 6 5 6 7 8 7 7 7 4 4
25083 089/300 Write block 4850 len 12288 data: 2 3 4
25084 090/300 Read block 1548 len 28672
25085 091/300 Read block 419 len 24576
25086 092/300 Write block 4751 len 4096 data: 4
25087 093/300 Write block 3662 len 40960 data: 3 2 3 3 4 4 5 4 5 5
25088 094/300 Read block 4010 len 8192
25089 095/300 Read block 3061 len 36864
25090 096/300 Read block 2950 len 8192
25091 097/300 Write block 3667 len 16384 data: 5 6 5 6
25092 098/300 Read block 4813 len 32768
25093 099/300 Read block 2112 len 32768
25094 100/300 Read block 3825 len 20480
25095 101/300 Write block 684 len 24576 data: 4 4 4 5 5 5
25096 102/300 Read block 4090 len 36864
25097 103/300 Write block 4237 len 32768 data: 3 3 3 3 3 3 3 4
25098 104/300 Read block 3217 len 8192
25099 105/300 Read block 2597 len 4096
25100 106/300 Read block 1378 len 16384
25101 107/300 Read block 2248 len 28672
25102 108/300 Write block 2173 len 24576 data: 3 4 6 5 7 4
25103 109/300 Read block 2366 len 12288
25104 110/300 Read block 1528 len 36864
25105 111/300 Write block 1441 len 4096 data: 6
25106 112/300 Write block 122 len 8192 data: 6 5
25107 113/300 Read block 4605 len 8192
25108 114/300 Read block 3027 len 12288
25109 115/300 Read block 3859 len 32768
25110 116/300 Write block 1823 len 20480 data: 7 6 6 6 5
25111 117/300 Write block 3087 len 16384 data: 6 5 5 5
25112 118/300 Read block 2191 len 16384
25113 119/300 Write block 2363 len 20480 data: 7 6 6 4 3
25114 120/300 Write block 4901 len 16384 data: 9 8 8 6
25115 121/300 Read block 4062 len 4096
25116 122/300 Flush
25117 123/300 Flush
25118 124/300 Read block 2622 len 36864
25119 125/300 Read block 972 len 32768
25120 126/300 Flush
25121 127/300 Read block 2314 len 32768
25122 128/300 Write block 2278 len 24576 data: 8 7 7 7 9 6
25123 129/300 Flush
25124 130/300 Write block 4939 len 24576 data: 6 6 3 3 3 4
25125 131/300 Read block 535 len 32768
25126 132/300 Read block 888 len 36864
25127 133/300 Write block 677 len 16384 data: 8 7 6 5
25128 134/300 Write block 534 len 32768 data: 5 5 6 5 6 6 6 5
25129 135/300 Read block 1626 len 4096
25130 136/300 Write block 2668 len 12288 data: 8 8 8
25131 137/300 Write block 2036 len 24576 data: 7 5 8 6 6 5
25132 138/300 Read block 106 len 20480
25133 139/300 Flush
25134 140/300 Write block 668 len 12288 data: 8 8 8
25135 141/300 Read block 3922 len 4096
25136 142/300 Flush
25137 143/300 Write block 1008 len 8192 data: 5 6
25138 144/300 Flush
25139 145/300 Read block 3693 len 16384
25140 146/300 Read block 774 len 40960
25141 147/300 Read block 3832 len 12288
25142 148/300 Read block 3552 len 12288
25143 149/300 Read block 353 len 24576
25144 150/300 Write block 2939 len 20480 data: 7 5 5 4 3
25145 151/300 Write block 165 len 20480 data: 4 4 4 5 5
25146 152/300 Read block 4372 len 12288
25147 153/300 Write block 1402 len 40960 data: 4 4 4 5 5 4 5 3 3 3
25148 154/300 Write block 3059 len 16384 data: 4 4 5 6
25149 155/300 Flush
25150 156/300 Read block 4706 len 4096
25151 157/300 Read block 2812 len 16384
25152 158/300 Flush
25153 159/300 Read block 130 len 12288
25154 160/300 Write block 388 len 24576 data: 5 5 6 5 4 4
25155 161/300 Read block 4793 len 8192
25156 162/300 Read block 1598 len 20480
25157 163/300 Read block 615 len 32768
25158 164/300 Flush
25159 165/300 Write block 850 len 36864 data: 3 3 3 3 3 2 2 2 2
25160 166/300 Read block 4594 len 32768
25161 167/300 Read block 3050 len 40960
25162 168/300 Read block 4226 len 28672
25163 169/300 Flush
25164 170/300 Read block 152 len 24576
25165 171/300 Write block 320 len 8192 data: 4 4
25166 172/300 Read block 275 len 4096
25167 173/300 Read block 2524 len 40960
25168 174/300 Write block 724 len 4096 data: 7
25169 175/300 Write block 4130 len 16384 data: 4 3 2 3
25170 176/300 Write block 952 len 24576 data: 4 3 4 6 6 8
25171 177/300 Read block 3704 len 16384
25172 178/300 Write block 4021 len 4096 data: 5
25173 179/300 Write block 1187 len 8192 data: 6 6
25174 180/300 Read block 289 len 36864
25175 181/300 Write block 587 len 32768 data: 6 5 5 5 6 4 4 3
25176 182/300 Read block 42 len 16384
25177 183/300 Flush
25178 184/300 Read block 2356 len 40960
25179 185/300 Read block 4707 len 4096
25180 186/300 Read block 2625 len 40960
25181 187/300 Write block 4013 len 32768 data: 5 5 5 4 5 5 5 4
25182 188/300 Write block 302 len 4096 data: 6
25183 189/300 Write block 2340 len 32768 data: 5 5 6 7 5 5 5 6
25184 190/300 Write block 204 len 24576 data: 5 4 4 4 5 5
25185 191/300 Read block 1383 len 8192
25186 192/300 Read block 3297 len 8192
25187 193/300 Read block 1270 len 16384
25188 194/300 Write block 113 len 4096 data: 4
25189 195/300 Read block 4282 len 12288
25190 196/300 Write block 2393 len 20480 data: 6 6 4 4 4
25191 197/300 Write block 4138 len 36864 data: 4 4 3 3 3 3 6 6 5
25192 198/300 Read block 644 len 12288
25193 199/300 Read block 4167 len 28672
25194 200/300 Write block 2635 len 40960 data: 4 4 4 3 3 2 3 3 4 5
25195 201/300 Flush
25196 202/300 Read block 256 len 28672
25197 203/300 Write block 1084 len 20480 data: 4 3 4 6 5
25198 204/300 Read block 4430 len 20480
25199 205/300 Write block 2442 len 16384 data: 2 5 5 5
25200 206/300 Read block 4830 len 20480
25201 207/300 Read block 3475 len 24576
25202 208/300 Write block 1969 len 24576 data: 4 5 5 6 5 4
25203 209/300 Write block 4086 len 12288 data: 3 2 2
25204 210/300 Flush
25205 211/300 Write block 4735 len 20480 data: 5 3 4 5 6
25206 212/300 Flush
25207 213/300 Read block 4451 len 28672
25208 214/300 Read block 2436 len 12288
25209 215/300 Read block 3691 len 28672
25210 216/300 Read block 967 len 4096
25211 217/300 Write block 1742 len 4096 data: 5
25212 218/300 Write block 2333 len 36864 data: 4 3 4 5 6 5 5 6 6
25213 219/300 Write block 4240 len 8192 data: 4 4
25214 220/300 Write block 53 len 4096 data: 3
25215 221/300 Write block 1488 len 16384 data: 5 5 5 5
25216 222/300 Write block 2717 len 20480 data: 7 7 5 5 6
25217 223/300 Read block 1064 len 24576
25218 224/300 Write block 1377 len 40960 data: 6 6 6 6 4 5 3 2 2 4
25219 225/300 Write block 350 len 4096 data: 3
25220 226/300 Read block 495 len 28672
25221 227/300 Write block 4558 len 16384 data: 4 4 5 5
25222 228/300 Read block 2000 len 32768
25223 229/300 Read block 3788 len 4096
25224 230/300 Read block 3772 len 16384
25225 231/300 Flush
25226 232/300 Read block 527 len 20480
25227 233/300 Write block 4155 len 40960 data: 2 4 5 4 3 3 3 3 3 3
25228 234/300 Write block 3776 len 16384 data: 8 5 4 3
25229 235/300 Write block 1198 len 20480 data: 6 7 5 5 3
25230 236/300 Write block 4642 len 24576 data: 3 3 4 5 6 8
25231 237/300 Flush
25232 238/300 Write block 907 len 32768 data: 5 4 4 4 3 3 4 5
25233 239/300 Read block 4484 len 12288
25234 240/300 Read block 181 len 20480
25235 241/300 Write block 3059 len 36864 data: 5 5 6 7 6 5 5 4 3
25236 242/300 Read block 3725 len 20480
25237 243/300 Read block 1130 len 24576
25238 244/300 Write block 1551 len 20480 data: 5 4 4 4 4
25239 245/300 Write block 3777 len 4096 data: 6
25240 246/300 Flush
25241 247/300 Read block 374 len 40960
25242 248/300 Write block 1304 len 8192 data: 3 3
25243 249/300 Read block 3000 len 32768
25244 250/300 Read block 1744 len 12288
25245 251/300 Read block 2865 len 28672
25246 252/300 Read block 375 len 28672
25247 253/300 Read block 3277 len 8192
25248 254/300 Read block 2151 len 16384
25249 255/300 Write block 77 len 32768 data: 7 7 7 6 6 6 4 4
25250 256/300 Flush
25251 257/300 Read block 2016 len 12288
25252 258/300 Write block 4901 len 4096 data: 10
25253 259/300 Write block 1806 len 16384 data: 5 4 4 4
25254 260/300 Read block 3836 len 28672
25255 261/300 Flush
25256 262/300 Read block 13 len 32768
25257 263/300 Write block 1608 len 28672 data: 7 9 8 10 10 7 6
25258 264/300 Write block 3519 len 16384 data: 3 4 4 2
25259 265/300 Read block 4311 len 28672
25260 266/300 Write block 3758 len 32768 data: 4 3 3 4 4 4 4 4
25261 267/300 Write block 3289 len 12288 data: 5 5 5
25262 268/300 Flush
25263 269/300 Flush
25264 270/300 Read block 3227 len 8192
25265 271/300 Read block 2367 len 20480
25266 272/300 Write block 3074 len 24576 data: 4 5 5 5 6 6
25267 273/300 Read block 248 len 24576
25268 274/300 Read block 2289 len 24576
25269 275/300 Read block 2093 len 24576
25270 276/300 Read block 2134 len 4096
25271 277/300 Write block 856 len 12288 data: 3 3 3
25272 278/300 Flush
25273 279/300 Read block 3734 len 36864
25274 280/300 Read block 1853 len 32768
25275 281/300 Read block 4666 len 24576
25276 282/300 Flush
25277 283/300 Read block 1815 len 36864
25278 284/300 Flush
25279 285/300 Write block 163 len 36864 data: 4 4 5 5 5 6 6 4 5
25280 286/300 Write block 436 len 36864 data: 6 5 4 5 5 6 4 7 7
25281 287/300 Read block 1446 len 28672
25282 288/300 Write block 3618 len 4096 data: 2
25283 289/300 Write block 2625 len 20480 data: 5 5 5 5 7
25284 290/300 Write block 3207 len 32768 data: 5 4 3 3 3 3 4 4
25285 291/300 Write block 519 len 32768 data: 4 5 5 5 5 9 8 6
25286 292/300 Read block 3078 len 24576
25287 293/300 Read block 3501 len 36864
25288 294/300 Read block 721 len 16384
25289 295/300 Write block 713 len 16384 data: 3 2 2 3
25290 296/300 Read block 307 len 36864
25291 297/300 Write block 3966 len 40960 data: 5 5 6 5 5 6 6 4 4 5
25292 298/300 Flush
25293 299/300 Read block 4050 len 16384
25294 300/300 Flush
25295 301/300 Read block 767 len 40960
25296 Main task has work to do, go find it
25297 got dsc Start(2)
25298 start 2
25299 Feb 21 06:50:57.037 INFO request completed, latency_us: 241, response_code: 204, uri: /start/cid/2, method: POST, req_id: c4456017-9258-4d87-9998-c1bd2fd11c9d, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
25300 Replay: started 2, returned:Ok(())
25301 ----------------------------------------------------------------
25302 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25303 States: Active Active Offline
25304 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25305 7641 Acked Read 10 Done Done Sent false
25306 7642 Acked Read 5 Done Done Sent false
25307 7643 Acked Write 1 Done Done Sent false
25308 7644 Acked Flush 0 Done Done Sent false
25309 7645 Acked Read 2 Done Done Sent false
25310 7646 Acked Read 3 Done Done Sent false
25311 7647 Acked Write 7 Done Done Sent false
25312 7648 Acked Read 5 Done Done Sent false
25313 7649 Acked Read 5 Done Done Sent false
25314 7650 Acked Read 2 Done Done Sent false
25315 7651 Acked Flush 0 Done Done Sent false
25316 7652 Acked Write 6 Done Done Sent false
25317 7653 Acked Read 3 Done Done Sent false
25318 7654 Acked Read 9 Done Done Sent false
25319 7655 Acked Read 7 Done Done Sent false
25320 7656 Acked Read 2 Done Done Sent false
25321 7657 Acked Write 7 Done Done Sent false
25322 7658 Acked Write 8 Done Done Sent false
25323 7659 Acked Read 10 Done Done Sent false
25324 7660 Acked Read 10 Done Done Sent false
25325 7661 Acked Read 7 Done Done Sent false
25326 7662 Acked Write 9 Done Done Sent false
25327 7663 Acked Read 6 Done Done Sent false
25328 7664 Acked Write 5 Done Done Sent false
25329 7665 Acked Write 6 Done Done Sent false
25330 7666 Acked Read 1 Done Done Sent false
25331 7667 Acked Read 2 Done Done Sent false
25332 7668 Acked Read 6 Done Done Sent false
25333 7669 Acked Write 3 Done Done Sent false
25334 7670 Acked Read 8 Done Done Sent false
25335 7671 Acked Read 7 Done Done Sent false
25336 7672 Acked Write 9 Done Done Sent false
25337 7673 Acked Read 3 Done Done Sent false
25338 7674 Acked Read 9 Done Done Sent false
25339 7675 Acked Read 10 Done Done Sent false
25340 7676 Acked Read 7 Done Done Sent false
25341 7677 Acked Read 3 Done Done Sent false
25342 7678 Acked Read 1 Done Done Sent false
25343 7679 Acked Write 1 Done Done Sent false
25344 7680 Acked Read 7 Done Done Sent false
25345 7681 Acked Write 1 Done Done Sent false
25346 7682 Acked Read 6 Done Done Sent false
25347 7683 Acked Read 4 Done Done Sent false
25348 7684 Acked Write 4 Done Done Sent false
25349 7685 Acked Write 8 Done Done Sent false
25350 7686 Acked Write 1 Done Done Sent false
25351 7687 Acked Flush 0 Done Done Sent false
25352 7688 Acked Write 1 Done Done Sent false
25353 7689 Acked Read 4 Done Done Sent false
25354 7690 Acked Read 3 Done Done Sent false
25355 7691 Acked Write 2 Done Done Sent false
25356 7692 Acked Write 10 Done Done Sent false
25357 7693 Acked Write 10 Done Done Sent false
25358 7694 Acked Write 3 Done Done Sent false
25359 7695 Acked Read 1 Done Done Sent false
25360 7696 Acked Read 9 Done Done Sent false
25361 7697 Acked Read 2 Done Done Sent false
25362 7698 Acked Read 1 Done Done Sent false
25363 7699 Acked Read 3 Done Done Sent false
25364 7700 Acked Read 3 Done Done Sent false
25365 7701 Acked Read 8 Done Done Sent false
25366 7702 Acked Write 1 Done Done Sent false
25367 7703 Acked Write 8 Done Done Sent false
25368 7704 Acked Read 10 Done Done Sent false
25369 7705 Acked Read 9 Done Done Sent false
25370 7706 Acked Read 3 Done Done Sent false
25371 7707 Acked Read 6 Done Done Sent false
25372 7708 Acked Flush 0 Done Done Sent false
25373 7709 Acked Read 5 Done Done Sent false
25374 7710 Acked Write 6 Done Done Sent false
25375 7711 Acked Write 9 Done Done Sent false
25376 7712 Acked Read 7 Done Done Sent false
25377 7713 Acked Flush 0 Done Done Sent false
25378 7714 Acked Write 5 Done Done Sent false
25379 7715 Acked Read 10 Done Done Sent false
25380 7716 Acked Flush 0 Done Done Sent false
25381 7717 Acked Flush 0 Done Done Sent false
25382 7718 Acked Flush 0 Done Done Sent false
25383 7719 Acked Write 1 Done Done Sent false
25384 7720 Acked Write 1 Done Done Sent false
25385 7721 Acked Read 9 Done Done Sent false
25386 7722 Acked Write 6 Done Done Sent false
25387 7723 Acked Write 8 Done Done Sent false
25388 7724 Acked Read 9 Done Done Sent false
25389 7725 Acked Write 5 Done Done Sent false
25390 7726 Acked Read 9 Done Done Sent false
25391 7727 Acked Write 9 Done Done Sent false
25392 7728 Acked Write 10 Done Done Sent false
25393 7729 Acked Write 3 Done Done Sent false
25394 7730 Acked Read 7 Done Done Sent false
25395 7731 Acked Read 6 Done Done Sent false
25396 7732 Acked Write 1 Done Done Sent false
25397 7733 Acked Write 10 Done Done Sent false
25398 7734 Acked Read 2 Done Done Sent false
25399 7735 Acked Read 9 Done Done Sent false
25400 7736 Acked Read 2 Done Done Sent false
25401 7737 Acked Write 4 Done Done Sent false
25402 7738 Acked Read 8 Done Done Sent false
25403 7739 Acked Read 8 Done Done Sent false
25404 7740 Acked Read 5 Done Done Sent false
25405 7741 Acked Write 6 Done Done Sent false
25406 7742 Acked Read 9 Done Done Sent false
25407 7743 Acked Write 8 Done Done Sent false
25408 7744 Acked Read 2 Done Done Sent false
25409 7745 Acked Read 1 Done Done Sent false
25410 7746 Acked Read 4 Done Done Sent false
25411 7747 Acked Read 7 Done Done Sent false
25412 7748 Acked Write 6 Done Done Sent false
25413 7749 Acked Read 3 Done Done Sent false
25414 7750 Acked Read 9 Done Done Sent false
25415 7751 Acked Write 1 Done Done Sent false
25416 7752 Acked Write 2 Done Done Sent false
25417 7753 Acked Read 2 Done Done Sent false
25418 7754 Acked Read 3 Done Done Sent false
25419 7755 Acked Read 8 Done Done Sent false
25420 7756 Acked Write 5 Done Done Sent false
25421 7757 Acked Write 4 Done Done Sent false
25422 7758 Acked Read 4 Done Done Sent false
25423 7759 Acked Write 5 Done Done Sent false
25424 7760 Acked Write 4 Done Done Sent false
25425 7761 Acked Read 1 Done Done Sent false
25426 7762 Acked Flush 0 Done Done Sent false
25427 7763 Acked Flush 0 Done Done Sent false
25428 7764 Acked Read 9 Done Done Sent false
25429 7765 Acked Read 8 Done Done Sent false
25430 7766 Acked Flush 0 Done Done Sent false
25431 7767 Acked Read 8 Done Done Sent false
25432 7768 Acked Write 6 Done Done Sent false
25433 7769 Acked Flush 0 Done Done Sent false
25434 7770 Acked Write 6 Done Done Sent false
25435 7771 Acked Read 8 Done Done Sent false
25436 7772 Acked Read 9 Done Done Sent false
25437 7773 Acked Write 4 Done Done Sent false
25438 7774 Acked Write 8 Done Done Sent false
25439 7775 Acked Read 1 Done Done Sent false
25440 7776 Acked Write 3 Done Done Sent false
25441 7777 Acked Write 6 Done Done Sent false
25442 7778 Acked Read 5 Done Done Sent false
25443 7779 Acked Flush 0 Done Done Sent false
25444 7780 Acked Write 3 Done Done Sent false
25445 7781 Acked Read 1 Done Done Sent false
25446 7782 Acked Flush 0 Done Done Sent false
25447 7783 Acked Write 2 Done Done Sent false
25448 7784 Acked Flush 0 Done Done Sent false
25449 7785 Acked Read 4 Done Done Sent false
25450 7786 Acked Read 10 Done Done Sent false
25451 7787 Acked Read 3 Done Done Sent false
25452 7788 Acked Read 3 Done Done Sent false
25453 7789 Acked Read 6 Done Done Sent false
25454 7790 Acked Write 5 Done Done Sent false
25455 7791 Acked Write 5 Done Done Sent false
25456 7792 Acked Read 3 Done Done Sent false
25457 7793 Acked Write 10 Done Done Sent false
25458 7794 Acked Write 4 Done Done Sent false
25459 7795 Acked Flush 0 Done Done Sent false
25460 7796 Acked Read 1 Done Done Sent false
25461 7797 Acked Read 4 Done Done Sent false
25462 7798 Acked Flush 0 Done Done Sent false
25463 7799 Acked Read 3 Done Done Sent false
25464 7800 Acked Write 6 Done Done Sent false
25465 7801 Acked Read 2 Done Done Sent false
25466 7802 Acked Read 5 Done Done Sent false
25467 7803 Acked Read 8 Done Done Sent false
25468 7804 Acked Flush 0 Done Done Sent false
25469 7805 Acked Write 9 Done Done Sent false
25470 7806 Acked Read 8 Done Done Sent false
25471 7807 Acked Read 10 Done Done Sent false
25472 7808 Acked Read 7 Done Done Sent false
25473 7809 Acked Flush 0 Done Done Sent false
25474 7810 Acked Read 6 Done Done Sent false
25475 7811 Acked Write 2 Done Done Sent false
25476 7812 Acked Read 1 Done Done Sent false
25477 7813 Acked Read 10 Done Done Sent false
25478 7814 Acked Write 1 Done Done Sent false
25479 7815 Acked Write 4 Done Done Sent false
25480 7816 Acked Write 6 Done Done Sent false
25481 7817 Acked Read 4 Done Done Sent false
25482 7818 Acked Write 1 Done Done Sent false
25483 7819 Acked Write 2 Done Done Sent false
25484 7820 Acked Read 9 Done Done Sent false
25485 7821 Acked Write 8 Done Done Sent false
25486 7822 Acked Read 4 Done Done Sent false
25487 7823 Acked Flush 0 Done Done Sent false
25488 7824 Acked Read 10 Done Done Sent false
25489 7825 Acked Read 1 Done Done Sent false
25490 7826 Acked Read 10 Done Done Sent false
25491 7827 Acked Write 8 Done Done Sent false
25492 7828 Acked Write 1 Done Done Sent false
25493 7829 Acked Write 8 Done Done Sent false
25494 7830 Acked Write 6 Done Done Sent false
25495 7831 Acked Read 2 Done Done Sent false
25496 7832 Acked Read 2 Done Done Sent false
25497 7833 Acked Read 4 Done Done Sent false
25498 7834 Acked Write 1 Done Done Sent false
25499 7835 Acked Read 3 Done Done Sent false
25500 7836 Acked Write 5 Done Done Sent false
25501 7837 Acked Write 9 Done Done Sent false
25502 7838 Acked Read 3 Done Done Sent false
25503 7839 Acked Read 7 Done Done Sent false
25504 7840 Acked Write 10 Done Done Sent false
25505 7841 Acked Flush 0 Done Done Sent false
25506 7842 Acked Read 7 Done Done Sent false
25507 7843 Acked Write 5 Done Done Sent false
25508 7844 Acked Read 5 Done Done Sent false
25509 7845 Acked Write 4 Done Done Sent false
25510 7846 Acked Read 5 Done Done Sent false
25511 7847 Acked Read 6 Done Done Sent false
25512 7848 Acked Write 6 Done Done Sent false
25513 7849 Acked Write 3 Done Done Sent false
25514 7850 Acked Flush 0 Done Done Sent false
25515 7851 Acked Write 5 Done Done Sent false
25516 7852 Acked Flush 0 Done Done Sent false
25517 7853 Acked Read 7 Done Done Sent false
25518 7854 Acked Read 3 Done Done Sent false
25519 7855 Acked Read 7 Done Done Sent false
25520 7856 Acked Read 1 Done Done Sent false
25521 7857 Acked Write 1 Done Done Sent false
25522 7858 Acked Write 9 Done Done Sent false
25523 7859 Acked Write 2 Done Done Sent false
25524 7860 Acked Write 1 Done Done Sent false
25525 7861 Acked Write 4 Done Done Sent false
25526 7862 Acked Write 5 Done Done Sent false
25527 7863 Acked Read 6 Done Done Sent false
25528 7864 Acked Write 10 Done Done Sent false
25529 7865 Acked Write 1 Done Done Sent false
25530 7866 Acked Read 7 Done Done Sent false
25531 7867 Acked Write 4 Done Done Sent false
25532 7868 Acked Read 8 Done Done Sent false
25533 7869 Acked Read 1 Done Done Sent false
25534 7870 Acked Read 4 Done Done Sent false
25535 7871 Acked Flush 0 Done Done Sent false
25536 7872 Acked Read 5 Done Done Sent false
25537 7873 Acked Write 10 Done Done Sent false
25538 7874 Acked Write 4 Done Done Sent false
25539 7875 Acked Write 5 Done Done Sent false
25540 7876 Acked Write 6 Done Done Sent false
25541 7877 Acked Flush 0 Done Done Sent false
25542 7878 Acked Write 8 Done Done Sent false
25543 7879 Acked Read 3 Done Done Sent false
25544 7880 Acked Read 5 Done Done Sent false
25545 7881 Acked Write 9 Done Done Sent false
25546 7882 Acked Read 5 Done Done Sent false
25547 7883 Acked Read 6 Done Done Sent false
25548 7884 Acked Write 5 Done Done Sent false
25549 7885 Acked Write 1 Done Done Sent false
25550 7886 Acked Flush 0 Done Done Sent false
25551 7887 Acked Read 10 Done Done Sent false
25552 7888 Acked Write 2 Done Done Sent false
25553 7889 Acked Read 8 Done Done Sent false
25554 7890 Acked Read 3 Done Done Sent false
25555 7891 Acked Read 7 Done Done Sent false
25556 7892 Acked Read 7 Done Done Sent false
25557 7893 Acked Read 2 Done Done Sent false
25558 7894 Acked Read 4 Done Done Sent false
25559 7895 Acked Write 8 Done Done Sent false
25560 7896 Acked Flush 0 Done Done Sent false
25561 7897 Acked Read 3 Done Done Sent false
25562 7898 Acked Write 1 Done Done Sent false
25563 7899 Acked Write 4 Done Done Sent false
25564 7900 Acked Read 7 Done Done Sent false
25565 7901 Acked Flush 0 Done Done Sent false
25566 7902 Acked Read 8 Done Done Sent false
25567 7903 Acked Write 7 Done Done Sent false
25568 7904 Acked Write 4 Done Done Sent false
25569 7905 Acked Read 7 Done Done Sent false
25570 7906 Acked Write 8 Done Done Sent false
25571 7907 Acked Write 3 Done Done Sent false
25572 7908 Acked Flush 0 Done Done Sent false
25573 7909 Acked Flush 0 Done Done Sent false
25574 7910 Acked Read 2 Done Done Sent false
25575 7911 Acked Read 5 Done Done Sent false
25576 7912 Acked Write 6 Done Done Sent false
25577 7913 Acked Read 6 Done Done Sent false
25578 7914 Acked Read 6 Done Done Sent false
25579 7915 Acked Read 6 Done Done Sent false
25580 7916 Acked Read 1 Done Done Sent false
25581 7917 Acked Write 3 Done Done Sent false
25582 7918 Acked Flush 0 Done Done Sent false
25583 7919 Acked Read 9 Done Done Sent false
25584 7920 Acked Read 8 Done Done Sent false
25585 7921 Acked Read 6 Done Done Sent false
25586 7922 Acked Flush 0 Done Done Sent false
25587 7923 Acked Read 9 Done Done Sent false
25588 7924 Acked Flush 0 Done Done Sent false
25589 7925 Acked Write 9 Done Done Sent false
25590 7926 Acked Write 9 Done Done Sent false
25591 7927 Acked Read 7 Done Done Sent false
25592 7928 Acked Write 1 Done Done Sent false
25593 7929 Acked Write 5 Done Done Sent false
25594 7930 Acked Write 8 Done Done Sent false
25595 7931 Acked Write 8 Done Done Sent false
25596 7932 Acked Read 6 Done Done Sent false
25597 7933 Acked Read 9 Done Done Sent false
25598 7934 Acked Read 4 Done Done Sent false
25599 7935 Acked Write 4 Done Done Sent false
25600 7936 Acked Read 9 Done Done Sent false
25601 7937 Acked Write 10 Done Done Sent false
25602 7938 Acked Flush 0 Done Done Sent false
25603 7939 Acked Read 4 Done Done Sent false
25604 7940 Acked Flush 0 Done Done Sent false
25605 7941 Acked Read 10 Done Done Sent false
25606 STATES DS:0 DS:1 DS:2 TOTAL
25607 Sent 0 0 301 301
25608 Done 301 301 0 602
25609 Skipped 0 0 0 0
25610 Error 0 0 0 0
25611 Last Flush: Some(JobId(7940)) Some(JobId(7940)) Some(JobId(7640))
25612 Downstairs last five completed: 7640 7639 7638 7637 7636
25613 Upstairs last five completed: 7941 7940 7939 7938 7937
25614 CLIENT: Up:0 ds:301 act:2
256152025-02-21T06:50:57.539ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
256162025-02-21T06:50:58.172ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
25617 [8830] got start action
25618 [8830] I am going to restart
25619 Starting downstairs at port 8830
25620 Make output file at "/tmp/dsc/downstairs-8830.txt"
25621 [8830][2] reports Starting
25622 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1229)
25623 [8830][2] reports Running
256242025-02-21T06:50:59.174ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
256252025-02-21T06:51:00.176ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
25626 ----------------------------------------------------------------
25627 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25628 States: Active Active Offline
25629 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25630 7641 Acked Read 10 Done Done Sent false
25631 7642 Acked Read 5 Done Done Sent false
25632 7643 Acked Write 1 Done Done Sent false
25633 7644 Acked Flush 0 Done Done Sent false
25634 7645 Acked Read 2 Done Done Sent false
25635 7646 Acked Read 3 Done Done Sent false
25636 7647 Acked Write 7 Done Done Sent false
25637 7648 Acked Read 5 Done Done Sent false
25638 7649 Acked Read 5 Done Done Sent false
25639 7650 Acked Read 2 Done Done Sent false
25640 7651 Acked Flush 0 Done Done Sent false
25641 7652 Acked Write 6 Done Done Sent false
25642 7653 Acked Read 3 Done Done Sent false
25643 7654 Acked Read 9 Done Done Sent false
25644 7655 Acked Read 7 Done Done Sent false
25645 7656 Acked Read 2 Done Done Sent false
25646 7657 Acked Write 7 Done Done Sent false
25647 7658 Acked Write 8 Done Done Sent false
25648 7659 Acked Read 10 Done Done Sent false
25649 7660 Acked Read 10 Done Done Sent false
25650 7661 Acked Read 7 Done Done Sent false
25651 7662 Acked Write 9 Done Done Sent false
25652 7663 Acked Read 6 Done Done Sent false
25653 7664 Acked Write 5 Done Done Sent false
25654 7665 Acked Write 6 Done Done Sent false
25655 7666 Acked Read 1 Done Done Sent false
25656 7667 Acked Read 2 Done Done Sent false
25657 7668 Acked Read 6 Done Done Sent false
25658 7669 Acked Write 3 Done Done Sent false
25659 7670 Acked Read 8 Done Done Sent false
25660 7671 Acked Read 7 Done Done Sent false
25661 7672 Acked Write 9 Done Done Sent false
25662 7673 Acked Read 3 Done Done Sent false
25663 7674 Acked Read 9 Done Done Sent false
25664 7675 Acked Read 10 Done Done Sent false
25665 7676 Acked Read 7 Done Done Sent false
25666 7677 Acked Read 3 Done Done Sent false
25667 7678 Acked Read 1 Done Done Sent false
25668 7679 Acked Write 1 Done Done Sent false
25669 7680 Acked Read 7 Done Done Sent false
25670 7681 Acked Write 1 Done Done Sent false
25671 7682 Acked Read 6 Done Done Sent false
25672 7683 Acked Read 4 Done Done Sent false
25673 7684 Acked Write 4 Done Done Sent false
25674 7685 Acked Write 8 Done Done Sent false
25675 7686 Acked Write 1 Done Done Sent false
25676 7687 Acked Flush 0 Done Done Sent false
25677 7688 Acked Write 1 Done Done Sent false
25678 7689 Acked Read 4 Done Done Sent false
25679 7690 Acked Read 3 Done Done Sent false
25680 7691 Acked Write 2 Done Done Sent false
25681 7692 Acked Write 10 Done Done Sent false
25682 7693 Acked Write 10 Done Done Sent false
25683 7694 Acked Write 3 Done Done Sent false
25684 7695 Acked Read 1 Done Done Sent false
25685 7696 Acked Read 9 Done Done Sent false
25686 7697 Acked Read 2 Done Done Sent false
25687 7698 Acked Read 1 Done Done Sent false
25688 7699 Acked Read 3 Done Done Sent false
25689 7700 Acked Read 3 Done Done Sent false
25690 7701 Acked Read 8 Done Done Sent false
25691 7702 Acked Write 1 Done Done Sent false
25692 7703 Acked Write 8 Done Done Sent false
25693 7704 Acked Read 10 Done Done Sent false
25694 7705 Acked Read 9 Done Done Sent false
25695 7706 Acked Read 3 Done Done Sent false
25696 7707 Acked Read 6 Done Done Sent false
25697 7708 Acked Flush 0 Done Done Sent false
25698 7709 Acked Read 5 Done Done Sent false
25699 7710 Acked Write 6 Done Done Sent false
25700 7711 Acked Write 9 Done Done Sent false
25701 7712 Acked Read 7 Done Done Sent false
25702 7713 Acked Flush 0 Done Done Sent false
25703 7714 Acked Write 5 Done Done Sent false
25704 7715 Acked Read 10 Done Done Sent false
25705 7716 Acked Flush 0 Done Done Sent false
25706 7717 Acked Flush 0 Done Done Sent false
25707 7718 Acked Flush 0 Done Done Sent false
25708 7719 Acked Write 1 Done Done Sent false
25709 7720 Acked Write 1 Done Done Sent false
25710 7721 Acked Read 9 Done Done Sent false
25711 7722 Acked Write 6 Done Done Sent false
25712 7723 Acked Write 8 Done Done Sent false
25713 7724 Acked Read 9 Done Done Sent false
25714 7725 Acked Write 5 Done Done Sent false
25715 7726 Acked Read 9 Done Done Sent false
25716 7727 Acked Write 9 Done Done Sent false
25717 7728 Acked Write 10 Done Done Sent false
25718 7729 Acked Write 3 Done Done Sent false
25719 7730 Acked Read 7 Done Done Sent false
25720 7731 Acked Read 6 Done Done Sent false
25721 7732 Acked Write 1 Done Done Sent false
25722 7733 Acked Write 10 Done Done Sent false
25723 7734 Acked Read 2 Done Done Sent false
25724 7735 Acked Read 9 Done Done Sent false
25725 7736 Acked Read 2 Done Done Sent false
25726 7737 Acked Write 4 Done Done Sent false
25727 7738 Acked Read 8 Done Done Sent false
25728 7739 Acked Read 8 Done Done Sent false
25729 7740 Acked Read 5 Done Done Sent false
25730 7741 Acked Write 6 Done Done Sent false
25731 7742 Acked Read 9 Done Done Sent false
25732 7743 Acked Write 8 Done Done Sent false
25733 7744 Acked Read 2 Done Done Sent false
25734 7745 Acked Read 1 Done Done Sent false
25735 7746 Acked Read 4 Done Done Sent false
25736 7747 Acked Read 7 Done Done Sent false
25737 7748 Acked Write 6 Done Done Sent false
25738 7749 Acked Read 3 Done Done Sent false
25739 7750 Acked Read 9 Done Done Sent false
25740 7751 Acked Write 1 Done Done Sent false
25741 7752 Acked Write 2 Done Done Sent false
25742 7753 Acked Read 2 Done Done Sent false
25743 7754 Acked Read 3 Done Done Sent false
25744 7755 Acked Read 8 Done Done Sent false
25745 7756 Acked Write 5 Done Done Sent false
25746 7757 Acked Write 4 Done Done Sent false
25747 7758 Acked Read 4 Done Done Sent false
25748 7759 Acked Write 5 Done Done Sent false
25749 7760 Acked Write 4 Done Done Sent false
25750 7761 Acked Read 1 Done Done Sent false
25751 7762 Acked Flush 0 Done Done Sent false
25752 7763 Acked Flush 0 Done Done Sent false
25753 7764 Acked Read 9 Done Done Sent false
25754 7765 Acked Read 8 Done Done Sent false
25755 7766 Acked Flush 0 Done Done Sent false
25756 7767 Acked Read 8 Done Done Sent false
25757 7768 Acked Write 6 Done Done Sent false
25758 7769 Acked Flush 0 Done Done Sent false
25759 7770 Acked Write 6 Done Done Sent false
25760 7771 Acked Read 8 Done Done Sent false
25761 7772 Acked Read 9 Done Done Sent false
25762 7773 Acked Write 4 Done Done Sent false
25763 7774 Acked Write 8 Done Done Sent false
25764 7775 Acked Read 1 Done Done Sent false
25765 7776 Acked Write 3 Done Done Sent false
25766 7777 Acked Write 6 Done Done Sent false
25767 7778 Acked Read 5 Done Done Sent false
25768 7779 Acked Flush 0 Done Done Sent false
25769 7780 Acked Write 3 Done Done Sent false
25770 7781 Acked Read 1 Done Done Sent false
25771 7782 Acked Flush 0 Done Done Sent false
25772 7783 Acked Write 2 Done Done Sent false
25773 7784 Acked Flush 0 Done Done Sent false
25774 7785 Acked Read 4 Done Done Sent false
25775 7786 Acked Read 10 Done Done Sent false
25776 7787 Acked Read 3 Done Done Sent false
25777 7788 Acked Read 3 Done Done Sent false
25778 7789 Acked Read 6 Done Done Sent false
25779 7790 Acked Write 5 Done Done Sent false
25780 7791 Acked Write 5 Done Done Sent false
25781 7792 Acked Read 3 Done Done Sent false
25782 7793 Acked Write 10 Done Done Sent false
25783 7794 Acked Write 4 Done Done Sent false
25784 7795 Acked Flush 0 Done Done Sent false
25785 7796 Acked Read 1 Done Done Sent false
25786 7797 Acked Read 4 Done Done Sent false
25787 7798 Acked Flush 0 Done Done Sent false
25788 7799 Acked Read 3 Done Done Sent false
25789 7800 Acked Write 6 Done Done Sent false
25790 7801 Acked Read 2 Done Done Sent false
25791 7802 Acked Read 5 Done Done Sent false
25792 7803 Acked Read 8 Done Done Sent false
25793 7804 Acked Flush 0 Done Done Sent false
25794 7805 Acked Write 9 Done Done Sent false
25795 7806 Acked Read 8 Done Done Sent false
25796 7807 Acked Read 10 Done Done Sent false
25797 7808 Acked Read 7 Done Done Sent false
25798 7809 Acked Flush 0 Done Done Sent false
25799 7810 Acked Read 6 Done Done Sent false
25800 7811 Acked Write 2 Done Done Sent false
25801 7812 Acked Read 1 Done Done Sent false
25802 7813 Acked Read 10 Done Done Sent false
25803 7814 Acked Write 1 Done Done Sent false
25804 7815 Acked Write 4 Done Done Sent false
25805 7816 Acked Write 6 Done Done Sent false
25806 7817 Acked Read 4 Done Done Sent false
25807 7818 Acked Write 1 Done Done Sent false
25808 7819 Acked Write 2 Done Done Sent false
25809 7820 Acked Read 9 Done Done Sent false
25810 7821 Acked Write 8 Done Done Sent false
25811 7822 Acked Read 4 Done Done Sent false
25812 7823 Acked Flush 0 Done Done Sent false
25813 7824 Acked Read 10 Done Done Sent false
25814 7825 Acked Read 1 Done Done Sent false
25815 7826 Acked Read 10 Done Done Sent false
25816 7827 Acked Write 8 Done Done Sent false
25817 7828 Acked Write 1 Done Done Sent false
25818 7829 Acked Write 8 Done Done Sent false
25819 7830 Acked Write 6 Done Done Sent false
25820 7831 Acked Read 2 Done Done Sent false
25821 7832 Acked Read 2 Done Done Sent false
25822 7833 Acked Read 4 Done Done Sent false
25823 7834 Acked Write 1 Done Done Sent false
25824 7835 Acked Read 3 Done Done Sent false
25825 7836 Acked Write 5 Done Done Sent false
25826 7837 Acked Write 9 Done Done Sent false
25827 7838 Acked Read 3 Done Done Sent false
25828 7839 Acked Read 7 Done Done Sent false
25829 7840 Acked Write 10 Done Done Sent false
25830 7841 Acked Flush 0 Done Done Sent false
25831 7842 Acked Read 7 Done Done Sent false
25832 7843 Acked Write 5 Done Done Sent false
25833 7844 Acked Read 5 Done Done Sent false
25834 7845 Acked Write 4 Done Done Sent false
25835 7846 Acked Read 5 Done Done Sent false
25836 7847 Acked Read 6 Done Done Sent false
25837 7848 Acked Write 6 Done Done Sent false
25838 7849 Acked Write 3 Done Done Sent false
25839 7850 Acked Flush 0 Done Done Sent false
25840 7851 Acked Write 5 Done Done Sent false
25841 7852 Acked Flush 0 Done Done Sent false
25842 7853 Acked Read 7 Done Done Sent false
25843 7854 Acked Read 3 Done Done Sent false
25844 7855 Acked Read 7 Done Done Sent false
25845 7856 Acked Read 1 Done Done Sent false
25846 7857 Acked Write 1 Done Done Sent false
25847 7858 Acked Write 9 Done Done Sent false
25848 7859 Acked Write 2 Done Done Sent false
25849 7860 Acked Write 1 Done Done Sent false
25850 7861 Acked Write 4 Done Done Sent false
25851 7862 Acked Write 5 Done Done Sent false
25852 7863 Acked Read 6 Done Done Sent false
25853 7864 Acked Write 10 Done Done Sent false
25854 7865 Acked Write 1 Done Done Sent false
25855 7866 Acked Read 7 Done Done Sent false
25856 7867 Acked Write 4 Done Done Sent false
25857 7868 Acked Read 8 Done Done Sent false
25858 7869 Acked Read 1 Done Done Sent false
25859 7870 Acked Read 4 Done Done Sent false
25860 7871 Acked Flush 0 Done Done Sent false
25861 7872 Acked Read 5 Done Done Sent false
25862 7873 Acked Write 10 Done Done Sent false
25863 7874 Acked Write 4 Done Done Sent false
25864 7875 Acked Write 5 Done Done Sent false
25865 7876 Acked Write 6 Done Done Sent false
25866 7877 Acked Flush 0 Done Done Sent false
25867 7878 Acked Write 8 Done Done Sent false
25868 7879 Acked Read 3 Done Done Sent false
25869 7880 Acked Read 5 Done Done Sent false
25870 7881 Acked Write 9 Done Done Sent false
25871 7882 Acked Read 5 Done Done Sent false
25872 7883 Acked Read 6 Done Done Sent false
25873 7884 Acked Write 5 Done Done Sent false
25874 7885 Acked Write 1 Done Done Sent false
25875 7886 Acked Flush 0 Done Done Sent false
25876 7887 Acked Read 10 Done Done Sent false
25877 7888 Acked Write 2 Done Done Sent false
25878 7889 Acked Read 8 Done Done Sent false
25879 7890 Acked Read 3 Done Done Sent false
25880 7891 Acked Read 7 Done Done Sent false
25881 7892 Acked Read 7 Done Done Sent false
25882 7893 Acked Read 2 Done Done Sent false
25883 7894 Acked Read 4 Done Done Sent false
25884 7895 Acked Write 8 Done Done Sent false
25885 7896 Acked Flush 0 Done Done Sent false
25886 7897 Acked Read 3 Done Done Sent false
25887 7898 Acked Write 1 Done Done Sent false
25888 7899 Acked Write 4 Done Done Sent false
25889 7900 Acked Read 7 Done Done Sent false
25890 7901 Acked Flush 0 Done Done Sent false
25891 7902 Acked Read 8 Done Done Sent false
25892 7903 Acked Write 7 Done Done Sent false
25893 7904 Acked Write 4 Done Done Sent false
25894 7905 Acked Read 7 Done Done Sent false
25895 7906 Acked Write 8 Done Done Sent false
25896 7907 Acked Write 3 Done Done Sent false
25897 7908 Acked Flush 0 Done Done Sent false
25898 7909 Acked Flush 0 Done Done Sent false
25899 7910 Acked Read 2 Done Done Sent false
25900 7911 Acked Read 5 Done Done Sent false
25901 7912 Acked Write 6 Done Done Sent false
25902 7913 Acked Read 6 Done Done Sent false
25903 7914 Acked Read 6 Done Done Sent false
25904 7915 Acked Read 6 Done Done Sent false
25905 7916 Acked Read 1 Done Done Sent false
25906 7917 Acked Write 3 Done Done Sent false
25907 7918 Acked Flush 0 Done Done Sent false
25908 7919 Acked Read 9 Done Done Sent false
25909 7920 Acked Read 8 Done Done Sent false
25910 7921 Acked Read 6 Done Done Sent false
25911 7922 Acked Flush 0 Done Done Sent false
25912 7923 Acked Read 9 Done Done Sent false
25913 7924 Acked Flush 0 Done Done Sent false
25914 7925 Acked Write 9 Done Done Sent false
25915 7926 Acked Write 9 Done Done Sent false
25916 7927 Acked Read 7 Done Done Sent false
25917 7928 Acked Write 1 Done Done Sent false
25918 7929 Acked Write 5 Done Done Sent false
25919 7930 Acked Write 8 Done Done Sent false
25920 7931 Acked Write 8 Done Done Sent false
25921 7932 Acked Read 6 Done Done Sent false
25922 7933 Acked Read 9 Done Done Sent false
25923 7934 Acked Read 4 Done Done Sent false
25924 7935 Acked Write 4 Done Done Sent false
25925 7936 Acked Read 9 Done Done Sent false
25926 7937 Acked Write 10 Done Done Sent false
25927 7938 Acked Flush 0 Done Done Sent false
25928 7939 Acked Read 4 Done Done Sent false
25929 7940 Acked Flush 0 Done Done Sent false
25930 7941 Acked Read 10 Done Done Sent false
25931 STATES DS:0 DS:1 DS:2 TOTAL
25932 Sent 0 0 301 301
25933 Done 301 301 0 602
25934 Skipped 0 0 0 0
25935 Error 0 0 0 0
25936 Last Flush: Some(JobId(7940)) Some(JobId(7940)) Some(JobId(7640))
25937 Downstairs last five completed: 7640 7639 7638 7637 7636
25938 Upstairs last five completed: 7941 7940 7939 7938 7937
25939 CLIENT: Up:0 ds:301 act:2
259402025-02-21T06:51:01.541ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259412025-02-21T06:51:02.180ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259422025-02-21T06:51:02.794ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259432025-02-21T06:51:02.794ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55458) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259442025-02-21T06:51:02.795ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259452025-02-21T06:51:02.795ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259462025-02-21T06:51:02.795ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259472025-02-21T06:51:02.795ZINFOcrucible: send last flush ID to this DS: Some(JobId(7640)) = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259482025-02-21T06:51:02.795ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259492025-02-21T06:51:02.795ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7640)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259502025-02-21T06:51:02.796ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7640)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
25951 ----------------------------------------------------------------
25952 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25953 Downstairs last five completed: 7942 7941 7940 7939 7938
25954 Upstairs last five completed: 7942 7941 7940 7939 7938
25955 CLIENT: Up:0 ds:0 act:3
25956 Replay: All jobs finished
25957 Main task has work to do, go find it
25958 got dsc Stop(0)
25959 stop 0
25960 [8810] Got stop action so:false kr:false
25961 Feb 21 06:51:05.042 INFO request completed, latency_us: 364, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 5fc2db8a-d8e6-48b5-ad97-2cb3765a1e9d, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
25962 Feb 21 06:51:05.043 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/0, method: GET, req_id: 131a4110-161b-422c-b84d-d5c3c1fe01d1, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
259632025-02-21T06:51:05.043ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259642025-02-21T06:51:05.043ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259652025-02-21T06:51:05.043ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
259662025-02-21T06:51:05.043ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
25967 [8810] Exited with: ExitStatus(unix_wait_status(9))
25968 [8810][0] reports Exit
25969 Feb 21 06:51:09.045 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/0, method: GET, req_id: 51e0c26b-fd85-4ce1-84a1-a79d5a07d684, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
25970 001/300 Write block 4992 len 4096 data: 2
25971 002/300 Read block 1991 len 12288
25972 003/300 Read block 2132 len 32768
25973 004/300 Read block 2441 len 40960
25974 005/300 Write block 2103 len 32768 data: 3 4 4 4 5 4 4 5
25975 006/300 Read block 3991 len 4096
25976 007/300 Flush
25977 008/300 Read block 2117 len 36864
25978 009/300 Read block 1356 len 28672
25979 010/300 Write block 3480 len 32768 data: 8 7 6 6 4 6 6 4
25980 011/300 Flush
25981 012/300 Read block 4887 len 24576
25982 013/300 Flush
25983 014/300 Write block 2283 len 24576 data: 7 4 3 3 3 4
25984 015/300 Read block 2992 len 16384
25985 016/300 Read block 668 len 8192
25986 017/300 Flush
25987 018/300 Read block 914 len 12288
25988 019/300 Read block 4820 len 20480
25989 020/300 Write block 3868 len 32768 data: 5 5 6 6 7 9 7 6
25990 021/300 Read block 3391 len 32768
25991 022/300 Write block 763 len 28672 data: 4 4 5 4 4 4 4
25992 023/300 Write block 1319 len 36864 data: 4 3 3 3 4 4 3 3 4
25993 024/300 Read block 83 len 24576
25994 025/300 Read block 1396 len 12288
25995 026/300 Read block 4436 len 12288
25996 027/300 Write block 1254 len 20480 data: 2 2 2 2 4
25997 028/300 Read block 1273 len 32768
25998 029/300 Read block 3367 len 36864
25999 030/300 Flush
26000 031/300 Read block 4794 len 16384
26001 032/300 Read block 1554 len 24576
26002 033/300 Read block 4947 len 36864
26003 034/300 Read block 4937 len 24576
26004 035/300 Read block 655 len 32768
26005 036/300 Read block 4488 len 20480
26006 037/300 Write block 1855 len 4096 data: 7
26007 038/300 Write block 3252 len 36864 data: 6 4 4 3 3 3 3 2 3
26008 039/300 Write block 1415 len 4096 data: 2
26009 040/300 Write block 1837 len 40960 data: 5 5 6 7 5 4 4 3 3 5
26010 041/300 Write block 4348 len 40960 data: 4 4 4 5 5 5 4 8 8 9
26011 042/300 Read block 1423 len 12288
26012 043/300 Read block 3004 len 28672
26013 044/300 Write block 543 len 24576 data: 6 6 7 5 5 5
26014 045/300 Read block 2835 len 8192
26015 046/300 Read block 4741 len 4096
26016 047/300 Write block 3604 len 4096 data: 3
26017 048/300 Read block 4128 len 24576
26018 049/300 Read block 2208 len 36864
26019 050/300 Read block 4712 len 16384
26020 051/300 Read block 1705 len 36864
26021 052/300 Read block 1828 len 32768
26022 053/300 Write block 3238 len 16384 data: 5 6 5 5
26023 054/300 Read block 3696 len 28672
26024 055/300 Write block 786 len 20480 data: 8 9 7 7 6
26025 056/300 Read block 3181 len 4096
26026 057/300 Write block 1388 len 36864 data: 4 4 5 5 5 4 4 3 4
26027 058/300 Read block 3400 len 20480
26028 059/300 Write block 3433 len 28672 data: 7 5 5 6 4 5 5
26029 060/300 Write block 1708 len 32768 data: 4 5 4 3 3 3 4 5
26030 061/300 Read block 2004 len 32768
26031 062/300 Write block 609 len 36864 data: 5 5 4 5 5 5 5 5 5
26032 063/300 Read block 1975 len 36864
26033 064/300 Write block 3703 len 40960 data: 6 5 5 5 4 4 4 5 5 6
26034 065/300 Read block 2765 len 36864
26035 066/300 Read block 2623 len 8192
26036 067/300 Read block 1708 len 8192
26037 068/300 Write block 186 len 4096 data: 7
26038 069/300 Flush
26039 070/300 Read block 2999 len 8192
26040 071/300 Write block 2208 len 8192 data: 4 5
26041 072/300 Write block 2235 len 32768 data: 7 7 6 7 7 7 5 6
26042 073/300 Read block 1541 len 24576
26043 074/300 Write block 4794 len 32768 data: 5 4 4 4 6 7 7 6
26044 075/300 Write block 1465 len 32768 data: 8 7 5 5 4 4 3 4
26045 076/300 Read block 2833 len 12288
26046 077/300 Write block 4289 len 4096 data: 3
26047 078/300 Read block 4073 len 16384
26048 079/300 Read block 3997 len 28672
26049 080/300 Read block 1796 len 24576
26050 081/300 Read block 3294 len 20480
26051 082/300 Read block 394 len 36864
26052 083/300 Write block 3202 len 24576 data: 4 3 3 3 5 6
26053 084/300 Write block 2174 len 24576 data: 5 7 6 8 5 5
26054 085/300 Write block 1135 len 16384 data: 6 6 6 6
26055 086/300 Read block 4194 len 40960
26056 087/300 Read block 2532 len 20480
26057 088/300 Read block 1034 len 4096
26058 089/300 Read block 4322 len 32768
26059 090/300 Write block 152 len 28672 data: 2 4 4 4 5 5 6
26060 091/300 Read block 4395 len 36864
26061 092/300 Write block 438 len 4096 data: 5
26062 093/300 Read block 1510 len 16384
26063 094/300 Write block 40 len 32768 data: 3 3 5 6 5 6 5 3
26064 095/300 Read block 402 len 40960
26065 096/300 Read block 1122 len 28672
26066 097/300 Read block 2083 len 8192
26067 098/300 Write block 3497 len 40960 data: 7 6 7 8 7 6 6 5 4 5
26068 099/300 Write block 4450 len 24576 data: 5 5 6 5 4 3
26069 100/300 Flush
26070 101/300 Flush
26071 102/300 Read block 3814 len 32768
26072 103/300 Read block 1940 len 20480
26073 104/300 Write block 4615 len 36864 data: 6 7 6 7 5 6 5 6 6
26074 105/300 Read block 237 len 24576
26075 106/300 Flush
26076 107/300 Read block 2426 len 8192
26077 108/300 Read block 3188 len 28672
26078 109/300 Write block 3017 len 24576 data: 3 5 6 5 6 6
26079 110/300 Read block 2695 len 24576
26080 111/300 Write block 3092 len 4096 data: 5
26081 112/300 Write block 1292 len 24576 data: 4 5 5 5 5 3
26082 113/300 Read block 1949 len 36864
26083 114/300 Read block 1550 len 8192
26084 115/300 Write block 269 len 4096 data: 2
26085 116/300 Read block 3230 len 32768
26086 117/300 Read block 1902 len 20480
26087 118/300 Read block 3375 len 32768
26088 119/300 Write block 1638 len 36864 data: 4 4 4 2 3 4 4 4 4
26089 120/300 Read block 4980 len 28672
26090 121/300 Read block 3459 len 28672
26091 122/300 Write block 3238 len 36864 data: 6 7 6 6 5 6 5 5 7
26092 123/300 Write block 286 len 36864 data: 2 3 3 3 3 3 4 5 6
26093 124/300 Write block 1642 len 28672 data: 4 5 5 5 5 4 4
26094 125/300 Write block 1070 len 40960 data: 7 7 7 7 7 5 4 4 4 4
26095 126/300 Flush
26096 127/300 Flush
26097 128/300 Read block 3091 len 12288
26098 129/300 Write block 1701 len 24576 data: 4 4 4 4 3 3
26099 130/300 Read block 4544 len 20480
26100 131/300 Write block 2862 len 20480 data: 5 4 4 6 6
26101 132/300 Write block 2542 len 8192 data: 5 4
26102 133/300 Write block 4835 len 24576 data: 6 6 5 6 5 4
26103 134/300 Flush
26104 135/300 Read block 2574 len 40960
26105 136/300 Write block 3491 len 16384 data: 5 4 4 6
26106 137/300 Read block 4920 len 32768
26107 138/300 Read block 25 len 40960
26108 139/300 Read block 3783 len 8192
26109 140/300 Read block 4220 len 4096
26110 141/300 Read block 2604 len 12288
26111 142/300 Read block 4454 len 40960
26112 143/300 Read block 2032 len 28672
26113 144/300 Read block 2556 len 16384
26114 145/300 Write block 3259 len 20480 data: 3 4 4 4 5
26115 146/300 Write block 4349 len 24576 data: 5 5 6 6 6 5
26116 147/300 Read block 1579 len 24576
26117 148/300 Read block 1233 len 8192
26118 149/300 Read block 4753 len 40960
26119 150/300 Write block 1064 len 4096 data: 4
26120 151/300 Read block 3249 len 36864
26121 152/300 Read block 1541 len 8192
26122 153/300 Read block 858 len 20480
26123 154/300 Write block 3910 len 28672 data: 6 6 7 6 6 6 6
26124 155/300 Read block 3274 len 8192
26125 156/300 Flush
26126 157/300 Read block 1819 len 16384
26127 158/300 Write block 4592 len 12288 data: 5 3 3
26128 159/300 Read block 1316 len 20480
26129 160/300 Read block 399 len 32768
26130 161/300 Read block 2183 len 24576
26131 162/300 Flush
26132 163/300 Read block 2973 len 16384
26133 164/300 Read block 3682 len 4096
26134 165/300 Read block 1769 len 8192
26135 166/300 Read block 1000 len 28672
26136 167/300 Read block 117 len 32768
26137 168/300 Flush
26138 169/300 Flush
26139 170/300 Write block 2088 len 24576 data: 3 3 3 3 4 5
26140 171/300 Read block 2829 len 40960
26141 172/300 Flush
26142 173/300 Read block 3227 len 4096
26143 174/300 Write block 3635 len 32768 data: 5 6 6 5 5 4 5 6
26144 175/300 Read block 636 len 40960
26145 176/300 Read block 1970 len 16384
26146 177/300 Read block 612 len 24576
26147 178/300 Read block 601 len 36864
26148 179/300 Read block 186 len 12288
26149 180/300 Write block 977 len 28672 data: 6 5 4 3 6 4 5
26150 181/300 Read block 539 len 20480
26151 182/300 Write block 2344 len 24576 data: 6 6 6 7 6 5
26152 183/300 Write block 2033 len 20480 data: 5 5 6 8 6
26153 184/300 Write block 4568 len 12288 data: 7 8 9
26154 185/300 Write block 3942 len 28672 data: 7 5 4 4 3 4 5
26155 186/300 Write block 520 len 36864 data: 6 6 6 6 10 9 7 6 6
26156 187/300 Read block 4871 len 24576
26157 188/300 Read block 1596 len 24576
26158 189/300 Write block 4362 len 8192 data: 6 5
26159 190/300 Read block 1709 len 32768
26160 191/300 Write block 2988 len 32768 data: 5 5 5 5 4 3 3 3
26161 192/300 Read block 759 len 16384
26162 193/300 Read block 375 len 4096
26163 194/300 Read block 2486 len 16384
26164 195/300 Write block 576 len 8192 data: 6 5
26165 196/300 Read block 3841 len 32768
26166 197/300 Write block 1996 len 8192 data: 3 3
26167 198/300 Read block 4114 len 16384
26168 199/300 Write block 4412 len 16384 data: 6 5 4 6
26169 200/300 Write block 4230 len 32768 data: 4 4 4 3 3 3 3 4
26170 201/300 Write block 1308 len 20480 data: 4 5 6 6 6
26171 202/300 Read block 3768 len 32768
26172 203/300 Write block 1003 len 8192 data: 3 4
26173 204/300 Read block 3739 len 16384
26174 205/300 Read block 1413 len 16384
26175 206/300 Read block 3109 len 40960
26176 207/300 Write block 2011 len 32768 data: 7 5 5 5 6 4 5 5
26177 208/300 Write block 3507 len 40960 data: 7 10 11 10 10 10 7 6 5 5
26178 209/300 Read block 2980 len 32768
26179 210/300 Read block 4440 len 24576
26180 211/300 Flush
26181 212/300 Read block 3492 len 4096
26182 213/300 Write block 3478 len 40960 data: 6 8 9 8 7 7 5 7 7 5
26183 214/300 Write block 249 len 8192 data: 4 3
26184 215/300 Read block 2802 len 16384
26185 216/300 Read block 4125 len 12288
26186 217/300 Write block 3899 len 16384 data: 2 2 3 3
26187 218/300 Write block 2524 len 24576 data: 4 4 3 2 2 3
26188 219/300 Write block 2169 len 4096 data: 4
26189 220/300 Write block 1245 len 20480 data: 5 5 3 3 3
26190 221/300 Flush
26191 222/300 Read block 2777 len 28672
26192 223/300 Read block 1597 len 12288
26193 224/300 Write block 3514 len 28672 data: 7 6 6 5 4 4 5
26194 225/300 Read block 1286 len 12288
26195 226/300 Write block 1788 len 28672 data: 5 5 5 5 7 5 5
26196 227/300 Read block 131 len 16384
26197 228/300 Read block 878 len 24576
26198 229/300 Read block 872 len 4096
26199 230/300 Flush
26200 231/300 Write block 3483 len 40960 data: 8 6 8 8 6 3 4 5 6 5
26201 232/300 Read block 4654 len 36864
26202 233/300 Read block 2600 len 20480
26203 234/300 Read block 3907 len 16384
26204 235/300 Read block 2246 len 12288
26205 236/300 Write block 896 len 24576 data: 4 4 3 3 3 5
26206 237/300 Flush
26207 238/300 Write block 2283 len 24576 data: 8 5 4 4 4 5
26208 239/300 Read block 2235 len 24576
26209 240/300 Read block 248 len 36864
26210 241/300 Read block 2060 len 8192
26211 242/300 Read block 4401 len 20480
26212 243/300 Read block 3177 len 8192
26213 244/300 Read block 4241 len 24576
26214 245/300 Write block 285 len 28672 data: 2 3 4 4 4 4 4
26215 246/300 Flush
26216 247/300 Read block 97 len 12288
26217 248/300 Read block 1476 len 36864
26218 249/300 Write block 689 len 40960 data: 6 5 3 4 5 5 4 4 4 4
26219 250/300 Write block 1438 len 12288 data: 5 7 7
26220 251/300 Read block 4676 len 20480
26221 252/300 Flush
26222 253/300 Read block 3741 len 40960
26223 254/300 Read block 4476 len 40960
26224 255/300 Read block 1377 len 16384
26225 256/300 Write block 4039 len 40960 data: 7 6 6 4 3 4 3 3 3 2
26226 257/300 Read block 4303 len 28672
26227 258/300 Read block 4834 len 24576
26228 259/300 Flush
26229 260/300 Write block 4996 len 8192 data: 3 3
26230 261/300 Read block 2972 len 20480
26231 262/300 Write block 229 len 16384 data: 5 5 4 4
26232 263/300 Read block 858 len 36864
26233 264/300 Read block 3622 len 16384
26234 265/300 Flush
26235 266/300 Read block 1678 len 24576
26236 267/300 Read block 4523 len 32768
26237 268/300 Read block 2195 len 8192
26238 269/300 Read block 3716 len 28672
26239 270/300 Read block 1649 len 36864
26240 271/300 Read block 864 len 20480
26241 272/300 Read block 1907 len 28672
26242 273/300 Read block 27 len 20480
26243 274/300 Read block 4661 len 32768
26244 275/300 Write block 2126 len 12288 data: 4 4 4
26245 276/300 Read block 3252 len 28672
26246 277/300 Write block 1820 len 16384 data: 9 8 8 8
26247 278/300 Write block 2072 len 12288 data: 7 7 6
26248 279/300 Write block 3043 len 36864 data: 5 5 5 5 5 7 6 6 5
26249 280/300 Read block 3198 len 40960
26250 281/300 Write block 336 len 8192 data: 5 5
26251 282/300 Read block 4870 len 8192
26252 283/300 Read block 588 len 40960
26253 284/300 Write block 3345 len 40960 data: 5 4 6 7 6 7 6 6 6 7
26254 285/300 Write block 404 len 16384 data: 3 4 5 5
26255 286/300 Write block 1627 len 12288 data: 7 6 4
26256 287/300 Write block 829 len 40960 data: 4 4 3 3 3 4 4 5 4 4
26257 288/300 Read block 346 len 40960
26258 289/300 Write block 675 len 36864 data: 5 6 9 8 7 6 5 5 5
26259 290/300 Write block 1795 len 20480 data: 4 4 4 5 5
26260 291/300 Write block 4072 len 24576 data: 5 6 5 4 4 8
26261 292/300 Write block 3944 len 20480 data: 5 5 4 5 6
26262 293/300 Read block 308 len 16384
26263 294/300 Read block 1228 len 28672
26264 295/300 Write block 4836 len 12288 data: 7 6 7
26265 296/300 Read block 3130 len 16384
26266 297/300 Read block 2939 len 20480
26267 298/300 Read block 1154 len 36864
26268 299/300 Flush
26269 300/300 Read block 3228 len 24576
26270 301/300 Read block 3206 len 28672
26271 Main task has work to do, go find it
26272 got dsc Start(0)
26273 start 0
26274 Feb 21 06:51:09.306 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/0, method: POST, req_id: e2967371-7a70-4c5b-8753-8a13bec399ca, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
26275 Replay: started 0, returned:Ok(())
26276 ----------------------------------------------------------------
26277 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26278 States: Offline Active Active
26279 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26280 7943 Acked Write 1 Sent Done Done false
26281 7944 Acked Read 3 Sent Done Done false
26282 7945 Acked Read 8 Sent Done Done false
26283 7946 Acked Read 10 Sent Done Done false
26284 7947 Acked Write 8 Sent Done Done false
26285 7948 Acked Read 1 Sent Done Done false
26286 7949 Acked Flush 0 Sent Done Done false
26287 7950 Acked Read 9 Sent Done Done false
26288 7951 Acked Read 7 Sent Done Done false
26289 7952 Acked Write 8 Sent Done Done false
26290 7953 Acked Flush 0 Sent Done Done false
26291 7954 Acked Read 6 Sent Done Done false
26292 7955 Acked Flush 0 Sent Done Done false
26293 7956 Acked Write 6 Sent Done Done false
26294 7957 Acked Read 4 Sent Done Done false
26295 7958 Acked Read 2 Sent Done Done false
26296 7959 Acked Flush 0 Sent Done Done false
26297 7960 Acked Read 3 Sent Done Done false
26298 7961 Acked Read 5 Sent Done Done false
26299 7962 Acked Write 8 Sent Done Done false
26300 7963 Acked Read 8 Sent Done Done false
26301 7964 Acked Write 7 Sent Done Done false
26302 7965 Acked Write 9 Sent Done Done false
26303 7966 Acked Read 6 Sent Done Done false
26304 7967 Acked Read 3 Sent Done Done false
26305 7968 Acked Read 3 Sent Done Done false
26306 7969 Acked Write 5 Sent Done Done false
26307 7970 Acked Read 8 Sent Done Done false
26308 7971 Acked Read 9 Sent Done Done false
26309 7972 Acked Flush 0 Sent Done Done false
26310 7973 Acked Read 4 Sent Done Done false
26311 7974 Acked Read 6 Sent Done Done false
26312 7975 Acked Read 9 Sent Done Done false
26313 7976 Acked Read 6 Sent Done Done false
26314 7977 Acked Read 8 Sent Done Done false
26315 7978 Acked Read 5 Sent Done Done false
26316 7979 Acked Write 1 Sent Done Done false
26317 7980 Acked Write 9 Sent Done Done false
26318 7981 Acked Write 1 Sent Done Done false
26319 7982 Acked Write 10 Sent Done Done false
26320 7983 Acked Write 10 Sent Done Done false
26321 7984 Acked Read 3 Sent Done Done false
26322 7985 Acked Read 7 Sent Done Done false
26323 7986 Acked Write 6 Sent Done Done false
26324 7987 Acked Read 2 Sent Done Done false
26325 7988 Acked Read 1 Sent Done Done false
26326 7989 Acked Write 1 Sent Done Done false
26327 7990 Acked Read 6 Sent Done Done false
26328 7991 Acked Read 9 Sent Done Done false
26329 7992 Acked Read 4 Sent Done Done false
26330 7993 Acked Read 9 Sent Done Done false
26331 7994 Acked Read 8 Sent Done Done false
26332 7995 Acked Write 4 Sent Done Done false
26333 7996 Acked Read 7 Sent Done Done false
26334 7997 Acked Write 5 Sent Done Done false
26335 7998 Acked Read 1 Sent Done Done false
26336 7999 Acked Write 9 Sent Done Done false
26337 8000 Acked Read 5 Sent Done Done false
26338 8001 Acked Write 7 Sent Done Done false
26339 8002 Acked Write 8 Sent Done Done false
26340 8003 Acked Read 8 Sent Done Done false
26341 8004 Acked Write 9 Sent Done Done false
26342 8005 Acked Read 9 Sent Done Done false
26343 8006 Acked Write 10 Sent Done Done false
26344 8007 Acked Read 9 Sent Done Done false
26345 8008 Acked Read 2 Sent Done Done false
26346 8009 Acked Read 2 Sent Done Done false
26347 8010 Acked Write 1 Sent Done Done false
26348 8011 Acked Flush 0 Sent Done Done false
26349 8012 Acked Read 2 Sent Done Done false
26350 8013 Acked Write 2 Sent Done Done false
26351 8014 Acked Write 8 Sent Done Done false
26352 8015 Acked Read 6 Sent Done Done false
26353 8016 Acked Write 8 Sent Done Done false
26354 8017 Acked Write 8 Sent Done Done false
26355 8018 Acked Read 3 Sent Done Done false
26356 8019 Acked Write 1 Sent Done Done false
26357 8020 Acked Read 4 Sent Done Done false
26358 8021 Acked Read 7 Sent Done Done false
26359 8022 Acked Read 6 Sent Done Done false
26360 8023 Acked Read 5 Sent Done Done false
26361 8024 Acked Read 9 Sent Done Done false
26362 8025 Acked Write 6 Sent Done Done false
26363 8026 Acked Write 6 Sent Done Done false
26364 8027 Acked Write 4 Sent Done Done false
26365 8028 Acked Read 10 Sent Done Done false
26366 8029 Acked Read 5 Sent Done Done false
26367 8030 Acked Read 1 Sent Done Done false
26368 8031 Acked Read 8 Sent Done Done false
26369 8032 Acked Write 7 Sent Done Done false
26370 8033 Acked Read 9 Sent Done Done false
26371 8034 Acked Write 1 Sent Done Done false
26372 8035 Acked Read 4 Sent Done Done false
26373 8036 Acked Write 8 Sent Done Done false
26374 8037 Acked Read 10 Sent Done Done false
26375 8038 Acked Read 7 Sent Done Done false
26376 8039 Acked Read 2 Sent Done Done false
26377 8040 Acked Write 10 Sent Done Done false
26378 8041 Acked Write 6 Sent Done Done false
26379 8042 Acked Flush 0 Sent Done Done false
26380 8043 Acked Flush 0 Sent Done Done false
26381 8044 Acked Read 8 Sent Done Done false
26382 8045 Acked Read 5 Sent Done Done false
26383 8046 Acked Write 9 Sent Done Done false
26384 8047 Acked Read 6 Sent Done Done false
26385 8048 Acked Flush 0 Sent Done Done false
26386 8049 Acked Read 2 Sent Done Done false
26387 8050 Acked Read 7 Sent Done Done false
26388 8051 Acked Write 6 Sent Done Done false
26389 8052 Acked Read 6 Sent Done Done false
26390 8053 Acked Write 1 Sent Done Done false
26391 8054 Acked Write 6 Sent Done Done false
26392 8055 Acked Read 9 Sent Done Done false
26393 8056 Acked Read 2 Sent Done Done false
26394 8057 Acked Write 1 Sent Done Done false
26395 8058 Acked Read 8 Sent Done Done false
26396 8059 Acked Read 5 Sent Done Done false
26397 8060 Acked Read 8 Sent Done Done false
26398 8061 Acked Write 9 Sent Done Done false
26399 8062 Acked Read 7 Sent Done Done false
26400 8063 Acked Read 7 Sent Done Done false
26401 8064 Acked Write 9 Sent Done Done false
26402 8065 Acked Write 9 Sent Done Done false
26403 8066 Acked Write 7 Sent Done Done false
26404 8067 Acked Write 10 Sent Done Done false
26405 8068 Acked Flush 0 Sent Done Done false
26406 8069 Acked Flush 0 Sent Done Done false
26407 8070 Acked Read 3 Sent Done Done false
26408 8071 Acked Write 6 Sent Done Done false
26409 8072 Acked Read 5 Sent Done Done false
26410 8073 Acked Write 5 Sent Done Done false
26411 8074 Acked Write 2 Sent Done Done false
26412 8075 Acked Write 6 Sent Done Done false
26413 8076 Acked Flush 0 Sent Done Done false
26414 8077 Acked Read 10 Sent Done Done false
26415 8078 Acked Write 4 Sent Done Done false
26416 8079 Acked Read 8 Sent Done Done false
26417 8080 Acked Read 10 Sent Done Done false
26418 8081 Acked Read 2 Sent Done Done false
26419 8082 Acked Read 1 Sent Done Done false
26420 8083 Acked Read 3 Sent Done Done false
26421 8084 Acked Read 10 Sent Done Done false
26422 8085 Acked Read 7 Sent Done Done false
26423 8086 Acked Read 4 Sent Done Done false
26424 8087 Acked Write 5 Sent Done Done false
26425 8088 Acked Write 6 Sent Done Done false
26426 8089 Acked Read 6 Sent Done Done false
26427 8090 Acked Read 2 Sent Done Done false
26428 8091 Acked Read 10 Sent Done Done false
26429 8092 Acked Write 1 Sent Done Done false
26430 8093 Acked Read 9 Sent Done Done false
26431 8094 Acked Read 2 Sent Done Done false
26432 8095 Acked Read 5 Sent Done Done false
26433 8096 Acked Write 7 Sent Done Done false
26434 8097 Acked Read 2 Sent Done Done false
26435 8098 Acked Flush 0 Sent Done Done false
26436 8099 Acked Read 4 Sent Done Done false
26437 8100 Acked Write 3 Sent Done Done false
26438 8101 Acked Read 5 Sent Done Done false
26439 8102 Acked Read 8 Sent Done Done false
26440 8103 Acked Read 6 Sent Done Done false
26441 8104 Acked Flush 0 Sent Done Done false
26442 8105 Acked Read 4 Sent Done Done false
26443 8106 Acked Read 1 Sent Done Done false
26444 8107 Acked Read 2 Sent Done Done false
26445 8108 Acked Read 7 Sent Done Done false
26446 8109 Acked Read 8 Sent Done Done false
26447 8110 Acked Flush 0 Sent Done Done false
26448 8111 Acked Flush 0 Sent Done Done false
26449 8112 Acked Write 6 Sent Done Done false
26450 8113 Acked Read 10 Sent Done Done false
26451 8114 Acked Flush 0 Sent Done Done false
26452 8115 Acked Read 1 Sent Done Done false
26453 8116 Acked Write 8 Sent Done Done false
26454 8117 Acked Read 10 Sent Done Done false
26455 8118 Acked Read 4 Sent Done Done false
26456 8119 Acked Read 6 Sent Done Done false
26457 8120 Acked Read 9 Sent Done Done false
26458 8121 Acked Read 3 Sent Done Done false
26459 8122 Acked Write 7 Sent Done Done false
26460 8123 Acked Read 5 Sent Done Done false
26461 8124 Acked Write 6 Sent Done Done false
26462 8125 Acked Write 5 Sent Done Done false
26463 8126 Acked Write 3 Sent Done Done false
26464 8127 Acked Write 7 Sent Done Done false
26465 8128 Acked Write 9 Sent Done Done false
26466 8129 Acked Read 6 Sent Done Done false
26467 8130 Acked Read 6 Sent Done Done false
26468 8131 Acked Write 2 Sent Done Done false
26469 8132 Acked Read 8 Sent Done Done false
26470 8133 Acked Write 8 Sent Done Done false
26471 8134 Acked Read 4 Sent Done Done false
26472 8135 Acked Read 1 Sent Done Done false
26473 8136 Acked Read 4 Sent Done Done false
26474 8137 Acked Write 2 Sent Done Done false
26475 8138 Acked Read 8 Sent Done Done false
26476 8139 Acked Write 2 Sent Done Done false
26477 8140 Acked Read 4 Sent Done Done false
26478 8141 Acked Write 4 Sent Done Done false
26479 8142 Acked Write 8 Sent Done Done false
26480 8143 Acked Write 5 Sent Done Done false
26481 8144 Acked Read 8 Sent Done Done false
26482 8145 Acked Write 2 Sent Done Done false
26483 8146 Acked Read 4 Sent Done Done false
26484 8147 Acked Read 4 Sent Done Done false
26485 8148 Acked Read 10 Sent Done Done false
26486 8149 Acked Write 8 Sent Done Done false
26487 8150 Acked Write 10 Sent Done Done false
26488 8151 Acked Read 8 Sent Done Done false
26489 8152 Acked Read 6 Sent Done Done false
26490 8153 Acked Flush 0 Sent Done Done false
26491 8154 Acked Read 1 Sent Done Done false
26492 8155 Acked Write 10 Sent Done Done false
26493 8156 Acked Write 2 Sent Done Done false
26494 8157 Acked Read 4 Sent Done Done false
26495 8158 Acked Read 3 Sent Done Done false
26496 8159 Acked Write 4 Sent Done Done false
26497 8160 Acked Write 6 Sent Done Done false
26498 8161 Acked Write 1 Sent Done Done false
26499 8162 Acked Write 5 Sent Done Done false
26500 8163 Acked Flush 0 Sent Done Done false
26501 8164 Acked Read 7 Sent Done Done false
26502 8165 Acked Read 3 Sent Done Done false
26503 8166 Acked Write 7 Sent Done Done false
26504 8167 Acked Read 3 Sent Done Done false
26505 8168 Acked Write 7 Sent Done Done false
26506 8169 Acked Read 4 Sent Done Done false
26507 8170 Acked Read 6 Sent Done Done false
26508 8171 Acked Read 1 Sent Done Done false
26509 8172 Acked Flush 0 Sent Done Done false
26510 8173 Acked Write 10 Sent Done Done false
26511 8174 Acked Read 9 Sent Done Done false
26512 8175 Acked Read 5 Sent Done Done false
26513 8176 Acked Read 4 Sent Done Done false
26514 8177 Acked Read 3 Sent Done Done false
26515 8178 Acked Write 6 Sent Done Done false
26516 8179 Acked Flush 0 Sent Done Done false
26517 8180 Acked Write 6 Sent Done Done false
26518 8181 Acked Read 6 Sent Done Done false
26519 8182 Acked Read 9 Sent Done Done false
26520 8183 Acked Read 2 Sent Done Done false
26521 8184 Acked Read 5 Sent Done Done false
26522 8185 Acked Read 2 Sent Done Done false
26523 8186 Acked Read 6 Sent Done Done false
26524 8187 Acked Write 7 Sent Done Done false
26525 8188 Acked Flush 0 Sent Done Done false
26526 8189 Acked Read 3 Sent Done Done false
26527 8190 Acked Read 9 Sent Done Done false
26528 8191 Acked Write 10 Sent Done Done false
26529 8192 Acked Write 3 Sent Done Done false
26530 8193 Acked Read 5 Sent Done Done false
26531 8194 Acked Flush 0 Sent Done Done false
26532 8195 Acked Read 10 Sent Done Done false
26533 8196 Acked Read 10 Sent Done Done false
26534 8197 Acked Read 4 Sent Done Done false
26535 8198 Acked Write 10 Sent Done Done false
26536 8199 Acked Read 7 Sent Done Done false
26537 8200 Acked Read 6 Sent Done Done false
26538 8201 Acked Flush 0 Sent Done Done false
26539 8202 Acked Write 2 Sent Done Done false
26540 8203 Acked Read 5 Sent Done Done false
26541 8204 Acked Write 4 Sent Done Done false
26542 8205 Acked Read 9 Sent Done Done false
26543 8206 Acked Read 4 Sent Done Done false
26544 8207 Acked Flush 0 Sent Done Done false
26545 8208 Acked Read 6 Sent Done Done false
26546 8209 Acked Read 8 Sent Done Done false
26547 8210 Acked Read 2 Sent Done Done false
26548 8211 Acked Read 7 Sent Done Done false
26549 8212 Acked Read 9 Sent Done Done false
26550 8213 Acked Read 5 Sent Done Done false
26551 8214 Acked Read 7 Sent Done Done false
26552 8215 Acked Read 5 Sent Done Done false
26553 8216 Acked Read 8 Sent Done Done false
26554 8217 Acked Write 3 Sent Done Done false
26555 8218 Acked Read 7 Sent Done Done false
26556 8219 Acked Write 4 Sent Done Done false
26557 8220 Acked Write 3 Sent Done Done false
26558 8221 Acked Write 9 Sent Done Done false
26559 8222 Acked Read 10 Sent Done Done false
26560 8223 Acked Write 2 Sent Done Done false
26561 8224 Acked Read 2 Sent Done Done false
26562 8225 Acked Read 10 Sent Done Done false
26563 8226 Acked Write 10 Sent Done Done false
26564 8227 Acked Write 4 Sent Done Done false
26565 8228 Acked Write 3 Sent Done Done false
26566 8229 Acked Write 10 Sent Done Done false
26567 8230 Acked Read 10 Sent Done Done false
26568 8231 Acked Write 9 Sent Done Done false
26569 8232 Acked Write 5 Sent Done Done false
26570 8233 Acked Write 6 Sent Done Done false
26571 8234 Acked Write 5 Sent Done Done false
26572 8235 Acked Read 4 Sent Done Done false
26573 8236 Acked Read 7 Sent Done Done false
26574 8237 Acked Write 3 Sent Done Done false
26575 8238 Acked Read 4 Sent Done Done false
26576 8239 Acked Read 5 Sent Done Done false
26577 8240 Acked Read 9 Sent Done Done false
26578 8241 Acked Flush 0 Sent Done Done false
26579 8242 Acked Read 6 Sent Done Done false
26580 8243 Acked Read 7 Sent Done Done false
26581 STATES DS:0 DS:1 DS:2 TOTAL
26582 Sent 301 0 0 301
26583 Done 0 301 301 602
26584 Skipped 0 0 0 0
26585 Error 0 0 0 0
26586 Last Flush: Some(JobId(7942)) Some(JobId(8241)) Some(JobId(8241))
26587 Downstairs last five completed: 7942 7941 7940 7939 7938
26588 Upstairs last five completed: 8243 8242 8241 8240 8239
26589 CLIENT: Up:0 ds:301 act:2
265902025-02-21T06:51:10.192ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
265912025-02-21T06:51:11.194ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
265922025-02-21T06:51:12.195ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
26593 ----------------------------------------------------------------
26594 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26595 States: Offline Active Active
26596 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26597 7943 Acked Write 1 Sent Done Done false
26598 7944 Acked Read 3 Sent Done Done false
26599 7945 Acked Read 8 Sent Done Done false
26600 7946 Acked Read 10 Sent Done Done false
26601 7947 Acked Write 8 Sent Done Done false
26602 7948 Acked Read 1 Sent Done Done false
26603 7949 Acked Flush 0 Sent Done Done false
26604 7950 Acked Read 9 Sent Done Done false
26605 7951 Acked Read 7 Sent Done Done false
26606 7952 Acked Write 8 Sent Done Done false
26607 7953 Acked Flush 0 Sent Done Done false
26608 7954 Acked Read 6 Sent Done Done false
26609 7955 Acked Flush 0 Sent Done Done false
26610 7956 Acked Write 6 Sent Done Done false
26611 7957 Acked Read 4 Sent Done Done false
26612 7958 Acked Read 2 Sent Done Done false
26613 7959 Acked Flush 0 Sent Done Done false
26614 7960 Acked Read 3 Sent Done Done false
26615 7961 Acked Read 5 Sent Done Done false
26616 7962 Acked Write 8 Sent Done Done false
26617 7963 Acked Read 8 Sent Done Done false
26618 7964 Acked Write 7 Sent Done Done false
26619 7965 Acked Write 9 Sent Done Done false
26620 7966 Acked Read 6 Sent Done Done false
26621 7967 Acked Read 3 Sent Done Done false
26622 7968 Acked Read 3 Sent Done Done false
26623 7969 Acked Write 5 Sent Done Done false
26624 7970 Acked Read 8 Sent Done Done false
26625 7971 Acked Read 9 Sent Done Done false
26626 7972 Acked Flush 0 Sent Done Done false
26627 7973 Acked Read 4 Sent Done Done false
26628 7974 Acked Read 6 Sent Done Done false
26629 7975 Acked Read 9 Sent Done Done false
26630 7976 Acked Read 6 Sent Done Done false
26631 7977 Acked Read 8 Sent Done Done false
26632 7978 Acked Read 5 Sent Done Done false
26633 7979 Acked Write 1 Sent Done Done false
26634 7980 Acked Write 9 Sent Done Done false
26635 7981 Acked Write 1 Sent Done Done false
26636 7982 Acked Write 10 Sent Done Done false
26637 7983 Acked Write 10 Sent Done Done false
26638 7984 Acked Read 3 Sent Done Done false
26639 7985 Acked Read 7 Sent Done Done false
26640 7986 Acked Write 6 Sent Done Done false
26641 7987 Acked Read 2 Sent Done Done false
26642 7988 Acked Read 1 Sent Done Done false
26643 7989 Acked Write 1 Sent Done Done false
26644 7990 Acked Read 6 Sent Done Done false
26645 7991 Acked Read 9 Sent Done Done false
26646 7992 Acked Read 4 Sent Done Done false
26647 7993 Acked Read 9 Sent Done Done false
26648 7994 Acked Read 8 Sent Done Done false
26649 7995 Acked Write 4 Sent Done Done false
26650 7996 Acked Read 7 Sent Done Done false
26651 7997 Acked Write 5 Sent Done Done false
26652 7998 Acked Read 1 Sent Done Done false
26653 7999 Acked Write 9 Sent Done Done false
26654 8000 Acked Read 5 Sent Done Done false
26655 8001 Acked Write 7 Sent Done Done false
26656 8002 Acked Write 8 Sent Done Done false
26657 8003 Acked Read 8 Sent Done Done false
26658 8004 Acked Write 9 Sent Done Done false
26659 8005 Acked Read 9 Sent Done Done false
26660 8006 Acked Write 10 Sent Done Done false
26661 8007 Acked Read 9 Sent Done Done false
26662 8008 Acked Read 2 Sent Done Done false
26663 8009 Acked Read 2 Sent Done Done false
26664 8010 Acked Write 1 Sent Done Done false
26665 8011 Acked Flush 0 Sent Done Done false
26666 8012 Acked Read 2 Sent Done Done false
26667 8013 Acked Write 2 Sent Done Done false
26668 8014 Acked Write 8 Sent Done Done false
26669 8015 Acked Read 6 Sent Done Done false
26670 8016 Acked Write 8 Sent Done Done false
26671 8017 Acked Write 8 Sent Done Done false
26672 8018 Acked Read 3 Sent Done Done false
26673 8019 Acked Write 1 Sent Done Done false
26674 8020 Acked Read 4 Sent Done Done false
26675 8021 Acked Read 7 Sent Done Done false
26676 8022 Acked Read 6 Sent Done Done false
26677 8023 Acked Read 5 Sent Done Done false
26678 8024 Acked Read 9 Sent Done Done false
26679 8025 Acked Write 6 Sent Done Done false
26680 8026 Acked Write 6 Sent Done Done false
26681 8027 Acked Write 4 Sent Done Done false
26682 8028 Acked Read 10 Sent Done Done false
26683 8029 Acked Read 5 Sent Done Done false
26684 8030 Acked Read 1 Sent Done Done false
26685 8031 Acked Read 8 Sent Done Done false
26686 8032 Acked Write 7 Sent Done Done false
26687 8033 Acked Read 9 Sent Done Done false
26688 8034 Acked Write 1 Sent Done Done false
26689 8035 Acked Read 4 Sent Done Done false
26690 8036 Acked Write 8 Sent Done Done false
26691 8037 Acked Read 10 Sent Done Done false
26692 8038 Acked Read 7 Sent Done Done false
26693 8039 Acked Read 2 Sent Done Done false
26694 8040 Acked Write 10 Sent Done Done false
26695 8041 Acked Write 6 Sent Done Done false
26696 8042 Acked Flush 0 Sent Done Done false
26697 8043 Acked Flush 0 Sent Done Done false
26698 8044 Acked Read 8 Sent Done Done false
26699 8045 Acked Read 5 Sent Done Done false
26700 8046 Acked Write 9 Sent Done Done false
26701 8047 Acked Read 6 Sent Done Done false
26702 8048 Acked Flush 0 Sent Done Done false
26703 8049 Acked Read 2 Sent Done Done false
26704 8050 Acked Read 7 Sent Done Done false
26705 8051 Acked Write 6 Sent Done Done false
26706 8052 Acked Read 6 Sent Done Done false
26707 8053 Acked Write 1 Sent Done Done false
26708 8054 Acked Write 6 Sent Done Done false
26709 8055 Acked Read 9 Sent Done Done false
26710 8056 Acked Read 2 Sent Done Done false
26711 8057 Acked Write 1 Sent Done Done false
26712 8058 Acked Read 8 Sent Done Done false
26713 8059 Acked Read 5 Sent Done Done false
26714 8060 Acked Read 8 Sent Done Done false
26715 8061 Acked Write 9 Sent Done Done false
26716 8062 Acked Read 7 Sent Done Done false
26717 8063 Acked Read 7 Sent Done Done false
26718 8064 Acked Write 9 Sent Done Done false
26719 8065 Acked Write 9 Sent Done Done false
26720 8066 Acked Write 7 Sent Done Done false
26721 8067 Acked Write 10 Sent Done Done false
26722 8068 Acked Flush 0 Sent Done Done false
26723 8069 Acked Flush 0 Sent Done Done false
26724 8070 Acked Read 3 Sent Done Done false
26725 8071 Acked Write 6 Sent Done Done false
26726 8072 Acked Read 5 Sent Done Done false
26727 8073 Acked Write 5 Sent Done Done false
26728 8074 Acked Write 2 Sent Done Done false
26729 8075 Acked Write 6 Sent Done Done false
26730 8076 Acked Flush 0 Sent Done Done false
26731 8077 Acked Read 10 Sent Done Done false
26732 8078 Acked Write 4 Sent Done Done false
26733 8079 Acked Read 8 Sent Done Done false
26734 8080 Acked Read 10 Sent Done Done false
26735 8081 Acked Read 2 Sent Done Done false
26736 8082 Acked Read 1 Sent Done Done false
26737 8083 Acked Read 3 Sent Done Done false
26738 8084 Acked Read 10 Sent Done Done false
26739 8085 Acked Read 7 Sent Done Done false
26740 8086 Acked Read 4 Sent Done Done false
26741 8087 Acked Write 5 Sent Done Done false
26742 8088 Acked Write 6 Sent Done Done false
26743 8089 Acked Read 6 Sent Done Done false
26744 8090 Acked Read 2 Sent Done Done false
26745 8091 Acked Read 10 Sent Done Done false
26746 8092 Acked Write 1 Sent Done Done false
26747 8093 Acked Read 9 Sent Done Done false
26748 8094 Acked Read 2 Sent Done Done false
26749 8095 Acked Read 5 Sent Done Done false
26750 8096 Acked Write 7 Sent Done Done false
26751 8097 Acked Read 2 Sent Done Done false
26752 8098 Acked Flush 0 Sent Done Done false
26753 8099 Acked Read 4 Sent Done Done false
26754 8100 Acked Write 3 Sent Done Done false
26755 8101 Acked Read 5 Sent Done Done false
26756 8102 Acked Read 8 Sent Done Done false
26757 8103 Acked Read 6 Sent Done Done false
26758 8104 Acked Flush 0 Sent Done Done false
26759 8105 Acked Read 4 Sent Done Done false
26760 8106 Acked Read 1 Sent Done Done false
26761 8107 Acked Read 2 Sent Done Done false
26762 8108 Acked Read 7 Sent Done Done false
26763 8109 Acked Read 8 Sent Done Done false
26764 8110 Acked Flush 0 Sent Done Done false
26765 8111 Acked Flush 0 Sent Done Done false
26766 8112 Acked Write 6 Sent Done Done false
26767 8113 Acked Read 10 Sent Done Done false
26768 8114 Acked Flush 0 Sent Done Done false
26769 8115 Acked Read 1 Sent Done Done false
26770 8116 Acked Write 8 Sent Done Done false
26771 8117 Acked Read 10 Sent Done Done false
26772 8118 Acked Read 4 Sent Done Done false
26773 8119 Acked Read 6 Sent Done Done false
26774 8120 Acked Read 9 Sent Done Done false
26775 8121 Acked Read 3 Sent Done Done false
26776 8122 Acked Write 7 Sent Done Done false
26777 8123 Acked Read 5 Sent Done Done false
26778 8124 Acked Write 6 Sent Done Done false
26779 8125 Acked Write 5 Sent Done Done false
26780 8126 Acked Write 3 Sent Done Done false
26781 8127 Acked Write 7 Sent Done Done false
26782 8128 Acked Write 9 Sent Done Done false
26783 8129 Acked Read 6 Sent Done Done false
26784 8130 Acked Read 6 Sent Done Done false
26785 8131 Acked Write 2 Sent Done Done false
26786 8132 Acked Read 8 Sent Done Done false
26787 8133 Acked Write 8 Sent Done Done false
26788 8134 Acked Read 4 Sent Done Done false
26789 8135 Acked Read 1 Sent Done Done false
26790 8136 Acked Read 4 Sent Done Done false
26791 8137 Acked Write 2 Sent Done Done false
26792 8138 Acked Read 8 Sent Done Done false
26793 8139 Acked Write 2 Sent Done Done false
26794 8140 Acked Read 4 Sent Done Done false
26795 8141 Acked Write 4 Sent Done Done false
26796 8142 Acked Write 8 Sent Done Done false
26797 8143 Acked Write 5 Sent Done Done false
26798 8144 Acked Read 8 Sent Done Done false
26799 8145 Acked Write 2 Sent Done Done false
26800 8146 Acked Read 4 Sent Done Done false
26801 8147 Acked Read 4 Sent Done Done false
26802 8148 Acked Read 10 Sent Done Done false
26803 8149 Acked Write 8 Sent Done Done false
26804 8150 Acked Write 10 Sent Done Done false
26805 8151 Acked Read 8 Sent Done Done false
26806 8152 Acked Read 6 Sent Done Done false
26807 8153 Acked Flush 0 Sent Done Done false
26808 8154 Acked Read 1 Sent Done Done false
26809 8155 Acked Write 10 Sent Done Done false
26810 8156 Acked Write 2 Sent Done Done false
26811 8157 Acked Read 4 Sent Done Done false
26812 8158 Acked Read 3 Sent Done Done false
26813 8159 Acked Write 4 Sent Done Done false
26814 8160 Acked Write 6 Sent Done Done false
26815 8161 Acked Write 1 Sent Done Done false
26816 8162 Acked Write 5 Sent Done Done false
26817 8163 Acked Flush 0 Sent Done Done false
26818 8164 Acked Read 7 Sent Done Done false
26819 8165 Acked Read 3 Sent Done Done false
26820 8166 Acked Write 7 Sent Done Done false
26821 8167 Acked Read 3 Sent Done Done false
26822 8168 Acked Write 7 Sent Done Done false
26823 8169 Acked Read 4 Sent Done Done false
26824 8170 Acked Read 6 Sent Done Done false
26825 8171 Acked Read 1 Sent Done Done false
26826 8172 Acked Flush 0 Sent Done Done false
26827 8173 Acked Write 10 Sent Done Done false
26828 8174 Acked Read 9 Sent Done Done false
26829 8175 Acked Read 5 Sent Done Done false
26830 8176 Acked Read 4 Sent Done Done false
26831 8177 Acked Read 3 Sent Done Done false
26832 8178 Acked Write 6 Sent Done Done false
26833 8179 Acked Flush 0 Sent Done Done false
26834 8180 Acked Write 6 Sent Done Done false
26835 8181 Acked Read 6 Sent Done Done false
26836 8182 Acked Read 9 Sent Done Done false
26837 8183 Acked Read 2 Sent Done Done false
26838 8184 Acked Read 5 Sent Done Done false
26839 8185 Acked Read 2 Sent Done Done false
26840 8186 Acked Read 6 Sent Done Done false
26841 8187 Acked Write 7 Sent Done Done false
26842 8188 Acked Flush 0 Sent Done Done false
26843 8189 Acked Read 3 Sent Done Done false
26844 8190 Acked Read 9 Sent Done Done false
26845 8191 Acked Write 10 Sent Done Done false
26846 8192 Acked Write 3 Sent Done Done false
26847 8193 Acked Read 5 Sent Done Done false
26848 8194 Acked Flush 0 Sent Done Done false
26849 8195 Acked Read 10 Sent Done Done false
26850 8196 Acked Read 10 Sent Done Done false
26851 8197 Acked Read 4 Sent Done Done false
26852 8198 Acked Write 10 Sent Done Done false
26853 8199 Acked Read 7 Sent Done Done false
26854 8200 Acked Read 6 Sent Done Done false
26855 8201 Acked Flush 0 Sent Done Done false
26856 8202 Acked Write 2 Sent Done Done false
26857 8203 Acked Read 5 Sent Done Done false
26858 8204 Acked Write 4 Sent Done Done false
26859 8205 Acked Read 9 Sent Done Done false
26860 8206 Acked Read 4 Sent Done Done false
26861 8207 Acked Flush 0 Sent Done Done false
26862 8208 Acked Read 6 Sent Done Done false
26863 8209 Acked Read 8 Sent Done Done false
26864 8210 Acked Read 2 Sent Done Done false
26865 8211 Acked Read 7 Sent Done Done false
26866 8212 Acked Read 9 Sent Done Done false
26867 8213 Acked Read 5 Sent Done Done false
26868 8214 Acked Read 7 Sent Done Done false
26869 8215 Acked Read 5 Sent Done Done false
26870 8216 Acked Read 8 Sent Done Done false
26871 8217 Acked Write 3 Sent Done Done false
26872 8218 Acked Read 7 Sent Done Done false
26873 8219 Acked Write 4 Sent Done Done false
26874 8220 Acked Write 3 Sent Done Done false
26875 8221 Acked Write 9 Sent Done Done false
26876 8222 Acked Read 10 Sent Done Done false
26877 8223 Acked Write 2 Sent Done Done false
26878 8224 Acked Read 2 Sent Done Done false
26879 8225 Acked Read 10 Sent Done Done false
26880 8226 Acked Write 10 Sent Done Done false
26881 8227 Acked Write 4 Sent Done Done false
26882 8228 Acked Write 3 Sent Done Done false
26883 8229 Acked Write 10 Sent Done Done false
26884 8230 Acked Read 10 Sent Done Done false
26885 8231 Acked Write 9 Sent Done Done false
26886 8232 Acked Write 5 Sent Done Done false
26887 8233 Acked Write 6 Sent Done Done false
26888 8234 Acked Write 5 Sent Done Done false
26889 8235 Acked Read 4 Sent Done Done false
26890 8236 Acked Read 7 Sent Done Done false
26891 8237 Acked Write 3 Sent Done Done false
26892 8238 Acked Read 4 Sent Done Done false
26893 8239 Acked Read 5 Sent Done Done false
26894 8240 Acked Read 9 Sent Done Done false
26895 8241 Acked Flush 0 Sent Done Done false
26896 8242 Acked Read 6 Sent Done Done false
26897 8243 Acked Read 7 Sent Done Done false
26898 STATES DS:0 DS:1 DS:2 TOTAL
26899 Sent 301 0 0 301
26900 Done 0 301 301 602
26901 Skipped 0 0 0 0
26902 Error 0 0 0 0
26903 Last Flush: Some(JobId(7942)) Some(JobId(8241)) Some(JobId(8241))
26904 Downstairs last five completed: 7942 7941 7940 7939 7938
26905 Upstairs last five completed: 8243 8242 8241 8240 8239
26906 CLIENT: Up:0 ds:301 act:2
269072025-02-21T06:51:14.198ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
269082025-02-21T06:51:15.045ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
269092025-02-21T06:51:15.045ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
269102025-02-21T06:51:15.045ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
269112025-02-21T06:51:15.045ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
269122025-02-21T06:51:15.045ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
269132025-02-21T06:51:15.546ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
269142025-02-21T06:51:16.201ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
269152025-02-21T06:51:17.201ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
26916 ----------------------------------------------------------------
26917 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26918 States: Offline Active Active
26919 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26920 7943 Acked Write 1 Sent Done Done false
26921 7944 Acked Read 3 Sent Done Done false
26922 7945 Acked Read 8 Sent Done Done false
26923 7946 Acked Read 10 Sent Done Done false
26924 7947 Acked Write 8 Sent Done Done false
26925 7948 Acked Read 1 Sent Done Done false
26926 7949 Acked Flush 0 Sent Done Done false
26927 7950 Acked Read 9 Sent Done Done false
26928 7951 Acked Read 7 Sent Done Done false
26929 7952 Acked Write 8 Sent Done Done false
26930 7953 Acked Flush 0 Sent Done Done false
26931 7954 Acked Read 6 Sent Done Done false
26932 7955 Acked Flush 0 Sent Done Done false
26933 7956 Acked Write 6 Sent Done Done false
26934 7957 Acked Read 4 Sent Done Done false
26935 7958 Acked Read 2 Sent Done Done false
26936 7959 Acked Flush 0 Sent Done Done false
26937 7960 Acked Read 3 Sent Done Done false
26938 7961 Acked Read 5 Sent Done Done false
26939 7962 Acked Write 8 Sent Done Done false
26940 7963 Acked Read 8 Sent Done Done false
26941 7964 Acked Write 7 Sent Done Done false
26942 7965 Acked Write 9 Sent Done Done false
26943 7966 Acked Read 6 Sent Done Done false
26944 7967 Acked Read 3 Sent Done Done false
26945 7968 Acked Read 3 Sent Done Done false
26946 7969 Acked Write 5 Sent Done Done false
26947 7970 Acked Read 8 Sent Done Done false
26948 7971 Acked Read 9 Sent Done Done false
26949 7972 Acked Flush 0 Sent Done Done false
26950 7973 Acked Read 4 Sent Done Done false
26951 7974 Acked Read 6 Sent Done Done false
26952 7975 Acked Read 9 Sent Done Done false
26953 7976 Acked Read 6 Sent Done Done false
26954 7977 Acked Read 8 Sent Done Done false
26955 7978 Acked Read 5 Sent Done Done false
26956 7979 Acked Write 1 Sent Done Done false
26957 7980 Acked Write 9 Sent Done Done false
26958 7981 Acked Write 1 Sent Done Done false
26959 7982 Acked Write 10 Sent Done Done false
26960 7983 Acked Write 10 Sent Done Done false
26961 7984 Acked Read 3 Sent Done Done false
26962 7985 Acked Read 7 Sent Done Done false
26963 7986 Acked Write 6 Sent Done Done false
26964 7987 Acked Read 2 Sent Done Done false
26965 7988 Acked Read 1 Sent Done Done false
26966 7989 Acked Write 1 Sent Done Done false
26967 7990 Acked Read 6 Sent Done Done false
26968 7991 Acked Read 9 Sent Done Done false
26969 7992 Acked Read 4 Sent Done Done false
26970 7993 Acked Read 9 Sent Done Done false
26971 7994 Acked Read 8 Sent Done Done false
26972 7995 Acked Write 4 Sent Done Done false
26973 7996 Acked Read 7 Sent Done Done false
26974 7997 Acked Write 5 Sent Done Done false
26975 7998 Acked Read 1 Sent Done Done false
26976 7999 Acked Write 9 Sent Done Done false
26977 8000 Acked Read 5 Sent Done Done false
26978 8001 Acked Write 7 Sent Done Done false
26979 8002 Acked Write 8 Sent Done Done false
26980 8003 Acked Read 8 Sent Done Done false
26981 8004 Acked Write 9 Sent Done Done false
26982 8005 Acked Read 9 Sent Done Done false
26983 8006 Acked Write 10 Sent Done Done false
26984 8007 Acked Read 9 Sent Done Done false
26985 8008 Acked Read 2 Sent Done Done false
26986 8009 Acked Read 2 Sent Done Done false
26987 8010 Acked Write 1 Sent Done Done false
26988 8011 Acked Flush 0 Sent Done Done false
26989 8012 Acked Read 2 Sent Done Done false
26990 8013 Acked Write 2 Sent Done Done false
26991 8014 Acked Write 8 Sent Done Done false
26992 8015 Acked Read 6 Sent Done Done false
26993 8016 Acked Write 8 Sent Done Done false
26994 8017 Acked Write 8 Sent Done Done false
26995 8018 Acked Read 3 Sent Done Done false
26996 8019 Acked Write 1 Sent Done Done false
26997 8020 Acked Read 4 Sent Done Done false
26998 8021 Acked Read 7 Sent Done Done false
26999 8022 Acked Read 6 Sent Done Done false
27000 8023 Acked Read 5 Sent Done Done false
27001 8024 Acked Read 9 Sent Done Done false
27002 8025 Acked Write 6 Sent Done Done false
27003 8026 Acked Write 6 Sent Done Done false
27004 8027 Acked Write 4 Sent Done Done false
27005 8028 Acked Read 10 Sent Done Done false
27006 8029 Acked Read 5 Sent Done Done false
27007 8030 Acked Read 1 Sent Done Done false
27008 8031 Acked Read 8 Sent Done Done false
27009 8032 Acked Write 7 Sent Done Done false
27010 8033 Acked Read 9 Sent Done Done false
27011 8034 Acked Write 1 Sent Done Done false
27012 8035 Acked Read 4 Sent Done Done false
27013 8036 Acked Write 8 Sent Done Done false
27014 8037 Acked Read 10 Sent Done Done false
27015 8038 Acked Read 7 Sent Done Done false
27016 8039 Acked Read 2 Sent Done Done false
27017 8040 Acked Write 10 Sent Done Done false
27018 8041 Acked Write 6 Sent Done Done false
27019 8042 Acked Flush 0 Sent Done Done false
27020 8043 Acked Flush 0 Sent Done Done false
27021 8044 Acked Read 8 Sent Done Done false
27022 8045 Acked Read 5 Sent Done Done false
27023 8046 Acked Write 9 Sent Done Done false
27024 8047 Acked Read 6 Sent Done Done false
27025 8048 Acked Flush 0 Sent Done Done false
27026 8049 Acked Read 2 Sent Done Done false
27027 8050 Acked Read 7 Sent Done Done false
27028 8051 Acked Write 6 Sent Done Done false
27029 8052 Acked Read 6 Sent Done Done false
27030 8053 Acked Write 1 Sent Done Done false
27031 8054 Acked Write 6 Sent Done Done false
27032 8055 Acked Read 9 Sent Done Done false
27033 8056 Acked Read 2 Sent Done Done false
27034 8057 Acked Write 1 Sent Done Done false
27035 8058 Acked Read 8 Sent Done Done false
27036 8059 Acked Read 5 Sent Done Done false
27037 8060 Acked Read 8 Sent Done Done false
27038 8061 Acked Write 9 Sent Done Done false
27039 8062 Acked Read 7 Sent Done Done false
27040 8063 Acked Read 7 Sent Done Done false
27041 8064 Acked Write 9 Sent Done Done false
27042 8065 Acked Write 9 Sent Done Done false
27043 8066 Acked Write 7 Sent Done Done false
27044 8067 Acked Write 10 Sent Done Done false
27045 8068 Acked Flush 0 Sent Done Done false
27046 8069 Acked Flush 0 Sent Done Done false
27047 8070 Acked Read 3 Sent Done Done false
27048 8071 Acked Write 6 Sent Done Done false
27049 8072 Acked Read 5 Sent Done Done false
27050 8073 Acked Write 5 Sent Done Done false
27051 8074 Acked Write 2 Sent Done Done false
27052 8075 Acked Write 6 Sent Done Done false
27053 8076 Acked Flush 0 Sent Done Done false
27054 8077 Acked Read 10 Sent Done Done false
27055 8078 Acked Write 4 Sent Done Done false
27056 8079 Acked Read 8 Sent Done Done false
27057 8080 Acked Read 10 Sent Done Done false
27058 8081 Acked Read 2 Sent Done Done false
27059 8082 Acked Read 1 Sent Done Done false
27060 8083 Acked Read 3 Sent Done Done false
27061 8084 Acked Read 10 Sent Done Done false
27062 8085 Acked Read 7 Sent Done Done false
27063 8086 Acked Read 4 Sent Done Done false
27064 8087 Acked Write 5 Sent Done Done false
27065 8088 Acked Write 6 Sent Done Done false
27066 8089 Acked Read 6 Sent Done Done false
27067 8090 Acked Read 2 Sent Done Done false
27068 8091 Acked Read 10 Sent Done Done false
27069 8092 Acked Write 1 Sent Done Done false
27070 8093 Acked Read 9 Sent Done Done false
27071 8094 Acked Read 2 Sent Done Done false
27072 8095 Acked Read 5 Sent Done Done false
27073 8096 Acked Write 7 Sent Done Done false
27074 8097 Acked Read 2 Sent Done Done false
27075 8098 Acked Flush 0 Sent Done Done false
27076 8099 Acked Read 4 Sent Done Done false
27077 8100 Acked Write 3 Sent Done Done false
27078 8101 Acked Read 5 Sent Done Done false
27079 8102 Acked Read 8 Sent Done Done false
27080 8103 Acked Read 6 Sent Done Done false
27081 8104 Acked Flush 0 Sent Done Done false
27082 8105 Acked Read 4 Sent Done Done false
27083 8106 Acked Read 1 Sent Done Done false
27084 8107 Acked Read 2 Sent Done Done false
27085 8108 Acked Read 7 Sent Done Done false
27086 8109 Acked Read 8 Sent Done Done false
27087 8110 Acked Flush 0 Sent Done Done false
27088 8111 Acked Flush 0 Sent Done Done false
27089 8112 Acked Write 6 Sent Done Done false
27090 8113 Acked Read 10 Sent Done Done false
27091 8114 Acked Flush 0 Sent Done Done false
27092 8115 Acked Read 1 Sent Done Done false
27093 8116 Acked Write 8 Sent Done Done false
27094 8117 Acked Read 10 Sent Done Done false
27095 8118 Acked Read 4 Sent Done Done false
27096 8119 Acked Read 6 Sent Done Done false
27097 8120 Acked Read 9 Sent Done Done false
27098 8121 Acked Read 3 Sent Done Done false
27099 8122 Acked Write 7 Sent Done Done false
27100 8123 Acked Read 5 Sent Done Done false
27101 8124 Acked Write 6 Sent Done Done false
27102 8125 Acked Write 5 Sent Done Done false
27103 8126 Acked Write 3 Sent Done Done false
27104 8127 Acked Write 7 Sent Done Done false
27105 8128 Acked Write 9 Sent Done Done false
27106 8129 Acked Read 6 Sent Done Done false
27107 8130 Acked Read 6 Sent Done Done false
27108 8131 Acked Write 2 Sent Done Done false
27109 8132 Acked Read 8 Sent Done Done false
27110 8133 Acked Write 8 Sent Done Done false
27111 8134 Acked Read 4 Sent Done Done false
27112 8135 Acked Read 1 Sent Done Done false
27113 8136 Acked Read 4 Sent Done Done false
27114 8137 Acked Write 2 Sent Done Done false
27115 8138 Acked Read 8 Sent Done Done false
27116 8139 Acked Write 2 Sent Done Done false
27117 8140 Acked Read 4 Sent Done Done false
27118 8141 Acked Write 4 Sent Done Done false
27119 8142 Acked Write 8 Sent Done Done false
27120 8143 Acked Write 5 Sent Done Done false
27121 8144 Acked Read 8 Sent Done Done false
27122 8145 Acked Write 2 Sent Done Done false
27123 8146 Acked Read 4 Sent Done Done false
27124 8147 Acked Read 4 Sent Done Done false
27125 8148 Acked Read 10 Sent Done Done false
27126 8149 Acked Write 8 Sent Done Done false
27127 8150 Acked Write 10 Sent Done Done false
27128 8151 Acked Read 8 Sent Done Done false
27129 8152 Acked Read 6 Sent Done Done false
27130 8153 Acked Flush 0 Sent Done Done false
27131 8154 Acked Read 1 Sent Done Done false
27132 8155 Acked Write 10 Sent Done Done false
27133 8156 Acked Write 2 Sent Done Done false
27134 8157 Acked Read 4 Sent Done Done false
27135 8158 Acked Read 3 Sent Done Done false
27136 8159 Acked Write 4 Sent Done Done false
27137 8160 Acked Write 6 Sent Done Done false
27138 8161 Acked Write 1 Sent Done Done false
27139 8162 Acked Write 5 Sent Done Done false
27140 8163 Acked Flush 0 Sent Done Done false
27141 8164 Acked Read 7 Sent Done Done false
27142 8165 Acked Read 3 Sent Done Done false
27143 8166 Acked Write 7 Sent Done Done false
27144 8167 Acked Read 3 Sent Done Done false
27145 8168 Acked Write 7 Sent Done Done false
27146 8169 Acked Read 4 Sent Done Done false
27147 8170 Acked Read 6 Sent Done Done false
27148 8171 Acked Read 1 Sent Done Done false
27149 8172 Acked Flush 0 Sent Done Done false
27150 8173 Acked Write 10 Sent Done Done false
27151 8174 Acked Read 9 Sent Done Done false
27152 8175 Acked Read 5 Sent Done Done false
27153 8176 Acked Read 4 Sent Done Done false
27154 8177 Acked Read 3 Sent Done Done false
27155 8178 Acked Write 6 Sent Done Done false
27156 8179 Acked Flush 0 Sent Done Done false
27157 8180 Acked Write 6 Sent Done Done false
27158 8181 Acked Read 6 Sent Done Done false
27159 8182 Acked Read 9 Sent Done Done false
27160 8183 Acked Read 2 Sent Done Done false
27161 8184 Acked Read 5 Sent Done Done false
27162 8185 Acked Read 2 Sent Done Done false
27163 8186 Acked Read 6 Sent Done Done false
27164 8187 Acked Write 7 Sent Done Done false
27165 8188 Acked Flush 0 Sent Done Done false
27166 8189 Acked Read 3 Sent Done Done false
27167 8190 Acked Read 9 Sent Done Done false
27168 8191 Acked Write 10 Sent Done Done false
27169 8192 Acked Write 3 Sent Done Done false
27170 8193 Acked Read 5 Sent Done Done false
27171 8194 Acked Flush 0 Sent Done Done false
27172 8195 Acked Read 10 Sent Done Done false
27173 8196 Acked Read 10 Sent Done Done false
27174 8197 Acked Read 4 Sent Done Done false
27175 8198 Acked Write 10 Sent Done Done false
27176 8199 Acked Read 7 Sent Done Done false
27177 8200 Acked Read 6 Sent Done Done false
27178 8201 Acked Flush 0 Sent Done Done false
27179 8202 Acked Write 2 Sent Done Done false
27180 8203 Acked Read 5 Sent Done Done false
27181 8204 Acked Write 4 Sent Done Done false
27182 8205 Acked Read 9 Sent Done Done false
27183 8206 Acked Read 4 Sent Done Done false
27184 8207 Acked Flush 0 Sent Done Done false
27185 8208 Acked Read 6 Sent Done Done false
27186 8209 Acked Read 8 Sent Done Done false
27187 8210 Acked Read 2 Sent Done Done false
27188 8211 Acked Read 7 Sent Done Done false
27189 8212 Acked Read 9 Sent Done Done false
27190 8213 Acked Read 5 Sent Done Done false
27191 8214 Acked Read 7 Sent Done Done false
27192 8215 Acked Read 5 Sent Done Done false
27193 8216 Acked Read 8 Sent Done Done false
27194 8217 Acked Write 3 Sent Done Done false
27195 8218 Acked Read 7 Sent Done Done false
27196 8219 Acked Write 4 Sent Done Done false
27197 8220 Acked Write 3 Sent Done Done false
27198 8221 Acked Write 9 Sent Done Done false
27199 8222 Acked Read 10 Sent Done Done false
27200 8223 Acked Write 2 Sent Done Done false
27201 8224 Acked Read 2 Sent Done Done false
27202 8225 Acked Read 10 Sent Done Done false
27203 8226 Acked Write 10 Sent Done Done false
27204 8227 Acked Write 4 Sent Done Done false
27205 8228 Acked Write 3 Sent Done Done false
27206 8229 Acked Write 10 Sent Done Done false
27207 8230 Acked Read 10 Sent Done Done false
27208 8231 Acked Write 9 Sent Done Done false
27209 8232 Acked Write 5 Sent Done Done false
27210 8233 Acked Write 6 Sent Done Done false
27211 8234 Acked Write 5 Sent Done Done false
27212 8235 Acked Read 4 Sent Done Done false
27213 8236 Acked Read 7 Sent Done Done false
27214 8237 Acked Write 3 Sent Done Done false
27215 8238 Acked Read 4 Sent Done Done false
27216 8239 Acked Read 5 Sent Done Done false
27217 8240 Acked Read 9 Sent Done Done false
27218 8241 Acked Flush 0 Sent Done Done false
27219 8242 Acked Read 6 Sent Done Done false
27220 8243 Acked Read 7 Sent Done Done false
27221 STATES DS:0 DS:1 DS:2 TOTAL
27222 Sent 301 0 0 301
27223 Done 0 301 301 602
27224 Skipped 0 0 0 0
27225 Error 0 0 0 0
27226 Last Flush: Some(JobId(7942)) Some(JobId(8241)) Some(JobId(8241))
27227 Downstairs last five completed: 7942 7941 7940 7939 7938
27228 Upstairs last five completed: 8243 8242 8241 8240 8239
27229 CLIENT: Up:0 ds:301 act:2
27230 [8810] got start action
27231 [8810] I am going to restart
27232 Starting downstairs at port 8810
27233 Make output file at "/tmp/dsc/downstairs-8810.txt"
27234 [8810][0] reports Starting
27235 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1230)
27236 [8810][0] reports Running
272372025-02-21T06:51:19.205ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
272382025-02-21T06:51:20.206ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
272392025-02-21T06:51:21.207ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
27240 ----------------------------------------------------------------
27241 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27242 States: Offline Active Active
27243 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27244 7943 Acked Write 1 Sent Done Done false
27245 7944 Acked Read 3 Sent Done Done false
27246 7945 Acked Read 8 Sent Done Done false
27247 7946 Acked Read 10 Sent Done Done false
27248 7947 Acked Write 8 Sent Done Done false
27249 7948 Acked Read 1 Sent Done Done false
27250 7949 Acked Flush 0 Sent Done Done false
27251 7950 Acked Read 9 Sent Done Done false
27252 7951 Acked Read 7 Sent Done Done false
27253 7952 Acked Write 8 Sent Done Done false
27254 7953 Acked Flush 0 Sent Done Done false
27255 7954 Acked Read 6 Sent Done Done false
27256 7955 Acked Flush 0 Sent Done Done false
27257 7956 Acked Write 6 Sent Done Done false
27258 7957 Acked Read 4 Sent Done Done false
27259 7958 Acked Read 2 Sent Done Done false
27260 7959 Acked Flush 0 Sent Done Done false
27261 7960 Acked Read 3 Sent Done Done false
27262 7961 Acked Read 5 Sent Done Done false
27263 7962 Acked Write 8 Sent Done Done false
27264 7963 Acked Read 8 Sent Done Done false
27265 7964 Acked Write 7 Sent Done Done false
27266 7965 Acked Write 9 Sent Done Done false
27267 7966 Acked Read 6 Sent Done Done false
27268 7967 Acked Read 3 Sent Done Done false
27269 7968 Acked Read 3 Sent Done Done false
27270 7969 Acked Write 5 Sent Done Done false
27271 7970 Acked Read 8 Sent Done Done false
27272 7971 Acked Read 9 Sent Done Done false
27273 7972 Acked Flush 0 Sent Done Done false
27274 7973 Acked Read 4 Sent Done Done false
27275 7974 Acked Read 6 Sent Done Done false
27276 7975 Acked Read 9 Sent Done Done false
27277 7976 Acked Read 6 Sent Done Done false
27278 7977 Acked Read 8 Sent Done Done false
27279 7978 Acked Read 5 Sent Done Done false
27280 7979 Acked Write 1 Sent Done Done false
27281 7980 Acked Write 9 Sent Done Done false
27282 7981 Acked Write 1 Sent Done Done false
27283 7982 Acked Write 10 Sent Done Done false
27284 7983 Acked Write 10 Sent Done Done false
27285 7984 Acked Read 3 Sent Done Done false
27286 7985 Acked Read 7 Sent Done Done false
27287 7986 Acked Write 6 Sent Done Done false
27288 7987 Acked Read 2 Sent Done Done false
27289 7988 Acked Read 1 Sent Done Done false
27290 7989 Acked Write 1 Sent Done Done false
27291 7990 Acked Read 6 Sent Done Done false
27292 7991 Acked Read 9 Sent Done Done false
27293 7992 Acked Read 4 Sent Done Done false
27294 7993 Acked Read 9 Sent Done Done false
27295 7994 Acked Read 8 Sent Done Done false
27296 7995 Acked Write 4 Sent Done Done false
27297 7996 Acked Read 7 Sent Done Done false
27298 7997 Acked Write 5 Sent Done Done false
27299 7998 Acked Read 1 Sent Done Done false
27300 7999 Acked Write 9 Sent Done Done false
27301 8000 Acked Read 5 Sent Done Done false
27302 8001 Acked Write 7 Sent Done Done false
27303 8002 Acked Write 8 Sent Done Done false
27304 8003 Acked Read 8 Sent Done Done false
27305 8004 Acked Write 9 Sent Done Done false
27306 8005 Acked Read 9 Sent Done Done false
27307 8006 Acked Write 10 Sent Done Done false
27308 8007 Acked Read 9 Sent Done Done false
27309 8008 Acked Read 2 Sent Done Done false
27310 8009 Acked Read 2 Sent Done Done false
27311 8010 Acked Write 1 Sent Done Done false
27312 8011 Acked Flush 0 Sent Done Done false
27313 8012 Acked Read 2 Sent Done Done false
27314 8013 Acked Write 2 Sent Done Done false
27315 8014 Acked Write 8 Sent Done Done false
27316 8015 Acked Read 6 Sent Done Done false
27317 8016 Acked Write 8 Sent Done Done false
27318 8017 Acked Write 8 Sent Done Done false
27319 8018 Acked Read 3 Sent Done Done false
27320 8019 Acked Write 1 Sent Done Done false
27321 8020 Acked Read 4 Sent Done Done false
27322 8021 Acked Read 7 Sent Done Done false
27323 8022 Acked Read 6 Sent Done Done false
27324 8023 Acked Read 5 Sent Done Done false
27325 8024 Acked Read 9 Sent Done Done false
27326 8025 Acked Write 6 Sent Done Done false
27327 8026 Acked Write 6 Sent Done Done false
27328 8027 Acked Write 4 Sent Done Done false
27329 8028 Acked Read 10 Sent Done Done false
27330 8029 Acked Read 5 Sent Done Done false
27331 8030 Acked Read 1 Sent Done Done false
27332 8031 Acked Read 8 Sent Done Done false
27333 8032 Acked Write 7 Sent Done Done false
27334 8033 Acked Read 9 Sent Done Done false
27335 8034 Acked Write 1 Sent Done Done false
27336 8035 Acked Read 4 Sent Done Done false
27337 8036 Acked Write 8 Sent Done Done false
27338 8037 Acked Read 10 Sent Done Done false
27339 8038 Acked Read 7 Sent Done Done false
27340 8039 Acked Read 2 Sent Done Done false
27341 8040 Acked Write 10 Sent Done Done false
27342 8041 Acked Write 6 Sent Done Done false
27343 8042 Acked Flush 0 Sent Done Done false
27344 8043 Acked Flush 0 Sent Done Done false
27345 8044 Acked Read 8 Sent Done Done false
27346 8045 Acked Read 5 Sent Done Done false
27347 8046 Acked Write 9 Sent Done Done false
27348 8047 Acked Read 6 Sent Done Done false
27349 8048 Acked Flush 0 Sent Done Done false
27350 8049 Acked Read 2 Sent Done Done false
27351 8050 Acked Read 7 Sent Done Done false
27352 8051 Acked Write 6 Sent Done Done false
27353 8052 Acked Read 6 Sent Done Done false
27354 8053 Acked Write 1 Sent Done Done false
27355 8054 Acked Write 6 Sent Done Done false
27356 8055 Acked Read 9 Sent Done Done false
27357 8056 Acked Read 2 Sent Done Done false
27358 8057 Acked Write 1 Sent Done Done false
27359 8058 Acked Read 8 Sent Done Done false
27360 8059 Acked Read 5 Sent Done Done false
27361 8060 Acked Read 8 Sent Done Done false
27362 8061 Acked Write 9 Sent Done Done false
27363 8062 Acked Read 7 Sent Done Done false
27364 8063 Acked Read 7 Sent Done Done false
27365 8064 Acked Write 9 Sent Done Done false
27366 8065 Acked Write 9 Sent Done Done false
27367 8066 Acked Write 7 Sent Done Done false
27368 8067 Acked Write 10 Sent Done Done false
27369 8068 Acked Flush 0 Sent Done Done false
27370 8069 Acked Flush 0 Sent Done Done false
27371 8070 Acked Read 3 Sent Done Done false
27372 8071 Acked Write 6 Sent Done Done false
27373 8072 Acked Read 5 Sent Done Done false
27374 8073 Acked Write 5 Sent Done Done false
27375 8074 Acked Write 2 Sent Done Done false
27376 8075 Acked Write 6 Sent Done Done false
27377 8076 Acked Flush 0 Sent Done Done false
27378 8077 Acked Read 10 Sent Done Done false
27379 8078 Acked Write 4 Sent Done Done false
27380 8079 Acked Read 8 Sent Done Done false
27381 8080 Acked Read 10 Sent Done Done false
27382 8081 Acked Read 2 Sent Done Done false
27383 8082 Acked Read 1 Sent Done Done false
27384 8083 Acked Read 3 Sent Done Done false
27385 8084 Acked Read 10 Sent Done Done false
27386 8085 Acked Read 7 Sent Done Done false
27387 8086 Acked Read 4 Sent Done Done false
27388 8087 Acked Write 5 Sent Done Done false
27389 8088 Acked Write 6 Sent Done Done false
27390 8089 Acked Read 6 Sent Done Done false
27391 8090 Acked Read 2 Sent Done Done false
27392 8091 Acked Read 10 Sent Done Done false
27393 8092 Acked Write 1 Sent Done Done false
27394 8093 Acked Read 9 Sent Done Done false
27395 8094 Acked Read 2 Sent Done Done false
27396 8095 Acked Read 5 Sent Done Done false
27397 8096 Acked Write 7 Sent Done Done false
27398 8097 Acked Read 2 Sent Done Done false
27399 8098 Acked Flush 0 Sent Done Done false
27400 8099 Acked Read 4 Sent Done Done false
27401 8100 Acked Write 3 Sent Done Done false
27402 8101 Acked Read 5 Sent Done Done false
27403 8102 Acked Read 8 Sent Done Done false
27404 8103 Acked Read 6 Sent Done Done false
27405 8104 Acked Flush 0 Sent Done Done false
27406 8105 Acked Read 4 Sent Done Done false
27407 8106 Acked Read 1 Sent Done Done false
27408 8107 Acked Read 2 Sent Done Done false
27409 8108 Acked Read 7 Sent Done Done false
27410 8109 Acked Read 8 Sent Done Done false
27411 8110 Acked Flush 0 Sent Done Done false
27412 8111 Acked Flush 0 Sent Done Done false
27413 8112 Acked Write 6 Sent Done Done false
27414 8113 Acked Read 10 Sent Done Done false
27415 8114 Acked Flush 0 Sent Done Done false
27416 8115 Acked Read 1 Sent Done Done false
27417 8116 Acked Write 8 Sent Done Done false
27418 8117 Acked Read 10 Sent Done Done false
27419 8118 Acked Read 4 Sent Done Done false
27420 8119 Acked Read 6 Sent Done Done false
27421 8120 Acked Read 9 Sent Done Done false
27422 8121 Acked Read 3 Sent Done Done false
27423 8122 Acked Write 7 Sent Done Done false
27424 8123 Acked Read 5 Sent Done Done false
27425 8124 Acked Write 6 Sent Done Done false
27426 8125 Acked Write 5 Sent Done Done false
27427 8126 Acked Write 3 Sent Done Done false
27428 8127 Acked Write 7 Sent Done Done false
27429 8128 Acked Write 9 Sent Done Done false
27430 8129 Acked Read 6 Sent Done Done false
27431 8130 Acked Read 6 Sent Done Done false
27432 8131 Acked Write 2 Sent Done Done false
27433 8132 Acked Read 8 Sent Done Done false
27434 8133 Acked Write 8 Sent Done Done false
27435 8134 Acked Read 4 Sent Done Done false
27436 8135 Acked Read 1 Sent Done Done false
27437 8136 Acked Read 4 Sent Done Done false
27438 8137 Acked Write 2 Sent Done Done false
27439 8138 Acked Read 8 Sent Done Done false
27440 8139 Acked Write 2 Sent Done Done false
27441 8140 Acked Read 4 Sent Done Done false
27442 8141 Acked Write 4 Sent Done Done false
27443 8142 Acked Write 8 Sent Done Done false
27444 8143 Acked Write 5 Sent Done Done false
27445 8144 Acked Read 8 Sent Done Done false
27446 8145 Acked Write 2 Sent Done Done false
27447 8146 Acked Read 4 Sent Done Done false
27448 8147 Acked Read 4 Sent Done Done false
27449 8148 Acked Read 10 Sent Done Done false
27450 8149 Acked Write 8 Sent Done Done false
27451 8150 Acked Write 10 Sent Done Done false
27452 8151 Acked Read 8 Sent Done Done false
27453 8152 Acked Read 6 Sent Done Done false
27454 8153 Acked Flush 0 Sent Done Done false
27455 8154 Acked Read 1 Sent Done Done false
27456 8155 Acked Write 10 Sent Done Done false
27457 8156 Acked Write 2 Sent Done Done false
27458 8157 Acked Read 4 Sent Done Done false
27459 8158 Acked Read 3 Sent Done Done false
27460 8159 Acked Write 4 Sent Done Done false
27461 8160 Acked Write 6 Sent Done Done false
27462 8161 Acked Write 1 Sent Done Done false
27463 8162 Acked Write 5 Sent Done Done false
27464 8163 Acked Flush 0 Sent Done Done false
27465 8164 Acked Read 7 Sent Done Done false
27466 8165 Acked Read 3 Sent Done Done false
27467 8166 Acked Write 7 Sent Done Done false
27468 8167 Acked Read 3 Sent Done Done false
27469 8168 Acked Write 7 Sent Done Done false
27470 8169 Acked Read 4 Sent Done Done false
27471 8170 Acked Read 6 Sent Done Done false
27472 8171 Acked Read 1 Sent Done Done false
27473 8172 Acked Flush 0 Sent Done Done false
27474 8173 Acked Write 10 Sent Done Done false
27475 8174 Acked Read 9 Sent Done Done false
27476 8175 Acked Read 5 Sent Done Done false
27477 8176 Acked Read 4 Sent Done Done false
27478 8177 Acked Read 3 Sent Done Done false
27479 8178 Acked Write 6 Sent Done Done false
27480 8179 Acked Flush 0 Sent Done Done false
27481 8180 Acked Write 6 Sent Done Done false
27482 8181 Acked Read 6 Sent Done Done false
27483 8182 Acked Read 9 Sent Done Done false
27484 8183 Acked Read 2 Sent Done Done false
27485 8184 Acked Read 5 Sent Done Done false
27486 8185 Acked Read 2 Sent Done Done false
27487 8186 Acked Read 6 Sent Done Done false
27488 8187 Acked Write 7 Sent Done Done false
27489 8188 Acked Flush 0 Sent Done Done false
27490 8189 Acked Read 3 Sent Done Done false
27491 8190 Acked Read 9 Sent Done Done false
27492 8191 Acked Write 10 Sent Done Done false
27493 8192 Acked Write 3 Sent Done Done false
27494 8193 Acked Read 5 Sent Done Done false
27495 8194 Acked Flush 0 Sent Done Done false
27496 8195 Acked Read 10 Sent Done Done false
27497 8196 Acked Read 10 Sent Done Done false
27498 8197 Acked Read 4 Sent Done Done false
27499 8198 Acked Write 10 Sent Done Done false
27500 8199 Acked Read 7 Sent Done Done false
27501 8200 Acked Read 6 Sent Done Done false
27502 8201 Acked Flush 0 Sent Done Done false
27503 8202 Acked Write 2 Sent Done Done false
27504 8203 Acked Read 5 Sent Done Done false
27505 8204 Acked Write 4 Sent Done Done false
27506 8205 Acked Read 9 Sent Done Done false
27507 8206 Acked Read 4 Sent Done Done false
27508 8207 Acked Flush 0 Sent Done Done false
27509 8208 Acked Read 6 Sent Done Done false
27510 8209 Acked Read 8 Sent Done Done false
27511 8210 Acked Read 2 Sent Done Done false
27512 8211 Acked Read 7 Sent Done Done false
27513 8212 Acked Read 9 Sent Done Done false
27514 8213 Acked Read 5 Sent Done Done false
27515 8214 Acked Read 7 Sent Done Done false
27516 8215 Acked Read 5 Sent Done Done false
27517 8216 Acked Read 8 Sent Done Done false
27518 8217 Acked Write 3 Sent Done Done false
27519 8218 Acked Read 7 Sent Done Done false
27520 8219 Acked Write 4 Sent Done Done false
27521 8220 Acked Write 3 Sent Done Done false
27522 8221 Acked Write 9 Sent Done Done false
27523 8222 Acked Read 10 Sent Done Done false
27524 8223 Acked Write 2 Sent Done Done false
27525 8224 Acked Read 2 Sent Done Done false
27526 8225 Acked Read 10 Sent Done Done false
27527 8226 Acked Write 10 Sent Done Done false
27528 8227 Acked Write 4 Sent Done Done false
27529 8228 Acked Write 3 Sent Done Done false
27530 8229 Acked Write 10 Sent Done Done false
27531 8230 Acked Read 10 Sent Done Done false
27532 8231 Acked Write 9 Sent Done Done false
27533 8232 Acked Write 5 Sent Done Done false
27534 8233 Acked Write 6 Sent Done Done false
27535 8234 Acked Write 5 Sent Done Done false
27536 8235 Acked Read 4 Sent Done Done false
27537 8236 Acked Read 7 Sent Done Done false
27538 8237 Acked Write 3 Sent Done Done false
27539 8238 Acked Read 4 Sent Done Done false
27540 8239 Acked Read 5 Sent Done Done false
27541 8240 Acked Read 9 Sent Done Done false
27542 8241 Acked Flush 0 Sent Done Done false
27543 8242 Acked Read 6 Sent Done Done false
27544 8243 Acked Read 7 Sent Done Done false
27545 STATES DS:0 DS:1 DS:2 TOTAL
27546 Sent 301 0 0 301
27547 Done 0 301 301 602
27548 Skipped 0 0 0 0
27549 Error 0 0 0 0
27550 Last Flush: Some(JobId(7942)) Some(JobId(8241)) Some(JobId(8241))
27551 Downstairs last five completed: 7942 7941 7940 7939 7938
27552 Upstairs last five completed: 8243 8242 8241 8240 8239
27553 CLIENT: Up:0 ds:301 act:2
275542025-02-21T06:51:22.209ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
275552025-02-21T06:51:24.213ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
275562025-02-21T06:51:25.046ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
275572025-02-21T06:51:25.046ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36313) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
275582025-02-21T06:51:25.047ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
275592025-02-21T06:51:25.048ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
275602025-02-21T06:51:25.048ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
275612025-02-21T06:51:25.048ZINFOcrucible: send last flush ID to this DS: Some(JobId(7942)) = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
275622025-02-21T06:51:25.048ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
275632025-02-21T06:51:25.048ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(7942)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
275642025-02-21T06:51:25.048ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(7942)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
27565 ----------------------------------------------------------------
27566 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:2
27567 States: Active Active Active
27568 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27569 8242 Acked Read 6 Done Done Done true
27570 8243 Acked Read 7 Done Done Done true
27571 STATES DS:0 DS:1 DS:2 TOTAL
27572 Sent 0 0 0 0
27573 Done 2 2 2 6
27574 Skipped 0 0 0 0
27575 Error 0 0 0 0
27576 Last Flush: Some(JobId(8241)) Some(JobId(8241)) Some(JobId(8241))
27577 Downstairs last five completed: 8241 8240 8239 8238 8237
27578 Upstairs last five completed: 8243 8242 8241 8240 8239
27579 CLIENT: Up:0 ds:2 act:3
27580 ----------------------------------------------------------------
27581 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27582 Downstairs last five completed: 8244 8243 8242 8241 8240
27583 Upstairs last five completed: 8244 8243 8242 8241 8240
27584 CLIENT: Up:0 ds:0 act:3
27585 Replay: All jobs finished
27586 Main task has work to do, go find it
27587 got dsc Stop(2)
27588 stop 2
27589 [8830] Got stop action so:false kr:false
27590 Feb 21 06:51:29.316 INFO request completed, latency_us: 326, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 03f87cc3-cb72-4aca-bbd9-ab4546fe463a, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
275912025-02-21T06:51:29.317ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
275922025-02-21T06:51:29.317ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
275932025-02-21T06:51:29.317ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
27594 Feb 21 06:51:29.317 INFO request completed, latency_us: 183, response_code: 200, uri: /state/cid/2, method: GET, req_id: aeeb2ca8-fd4a-4b40-8d91-c9a7270905f6, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
275952025-02-21T06:51:29.317ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
27596 [8830] Exited with: ExitStatus(unix_wait_status(9))
27597 [8830][2] reports Exit
27598 Feb 21 06:51:33.319 INFO request completed, latency_us: 198, response_code: 200, uri: /state/cid/2, method: GET, req_id: 536348cd-8d1c-4133-b153-79d31cd77e84, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
27599 001/300 Read block 4322 len 28672
27600 002/300 Read block 3062 len 8192
27601 003/300 Read block 719 len 8192
27602 004/300 Write block 4688 len 28672 data: 4 4 3 3 3 4 4
27603 005/300 Read block 4520 len 36864
27604 006/300 Read block 1202 len 12288
27605 007/300 Write block 677 len 32768 data: 10 9 8 7 6 6 6 5
27606 008/300 Write block 2136 len 20480 data: 5 6 6 6 6
27607 009/300 Read block 3287 len 12288
27608 010/300 Read block 2814 len 28672
27609 011/300 Write block 4812 len 4096 data: 7
27610 012/300 Read block 529 len 20480
27611 013/300 Read block 2985 len 4096
27612 014/300 Write block 4409 len 40960 data: 3 5 6 7 6 5 7 5 5 8
27613 015/300 Flush
27614 016/300 Read block 3122 len 32768
27615 017/300 Read block 2095 len 32768
27616 018/300 Flush
27617 019/300 Read block 801 len 28672
27618 020/300 Write block 1902 len 32768 data: 5 5 5 5 3 4 7 7
27619 021/300 Read block 3618 len 28672
27620 022/300 Read block 985 len 36864
27621 023/300 Flush
27622 024/300 Read block 525 len 28672
27623 025/300 Read block 2671 len 20480
27624 026/300 Flush
27625 027/300 Write block 4169 len 32768 data: 2 3 4 4 4 4 4 3
27626 028/300 Read block 2890 len 8192
27627 029/300 Write block 4490 len 40960 data: 4 4 5 4 5 5 5 3 3 3
27628 030/300 Flush
27629 031/300 Read block 2546 len 40960
27630 032/300 Write block 975 len 16384 data: 7 8 7 6
27631 033/300 Write block 1066 len 40960 data: 4 6 7 7 8 8 8 8 8 6
27632 034/300 Read block 611 len 20480
27633 035/300 Flush
27634 036/300 Write block 1522 len 40960 data: 5 5 4 6 5 5 5 5 7 6
27635 037/300 Read block 2905 len 36864
27636 038/300 Read block 853 len 28672
27637 039/300 Read block 1867 len 16384
27638 040/300 Read block 545 len 16384
27639 041/300 Write block 724 len 8192 data: 8 6
27640 042/300 Read block 1671 len 24576
27641 043/300 Write block 1875 len 20480 data: 6 5 6 4 3
27642 044/300 Read block 3786 len 32768
27643 045/300 Write block 4022 len 20480 data: 5 6 4 4 4
27644 046/300 Write block 675 len 8192 data: 6 7
27645 047/300 Read block 1309 len 40960
27646 048/300 Flush
27647 049/300 Read block 1265 len 40960
27648 050/300 Flush
27649 051/300 Write block 1288 len 20480 data: 3 4 4 4 5
27650 052/300 Write block 114 len 40960 data: 4 6 4 4 4 4 5 5 7 6
27651 053/300 Write block 2166 len 40960 data: 4 4 3 5 5 3 2 4 6 8
27652 054/300 Write block 3067 len 8192 data: 4 3
27653 055/300 Write block 88 len 36864 data: 3 3 3 3 4 4 4 5 5
27654 056/300 Write block 2866 len 4096 data: 7
27655 057/300 Read block 2669 len 28672
27656 058/300 Read block 836 len 8192
27657 059/300 Write block 4372 len 16384 data: 4 4 4 4
27658 060/300 Write block 2020 len 28672 data: 6 5 5 5 5 6 6
27659 061/300 Read block 1849 len 12288
27660 062/300 Read block 2747 len 40960
27661 063/300 Write block 3095 len 16384 data: 6 5 6 6
27662 064/300 Write block 1416 len 8192 data: 2 2
27663 065/300 Write block 1493 len 32768 data: 7 7 7 7 7 7 7 8
27664 066/300 Write block 3123 len 8192 data: 4 5
27665 067/300 Write block 1410 len 20480 data: 4 4 3 2 2
27666 068/300 Read block 760 len 12288
27667 069/300 Read block 4938 len 40960
27668 070/300 Read block 3688 len 24576
27669 071/300 Write block 881 len 16384 data: 4 4 4 4
27670 072/300 Write block 4132 len 8192 data: 3 4
27671 073/300 Write block 302 len 20480 data: 7 4 5 5 5
27672 074/300 Read block 3336 len 36864
27673 075/300 Flush
27674 076/300 Write block 1045 len 12288 data: 5 6 6
27675 077/300 Read block 2393 len 8192
27676 078/300 Read block 2708 len 36864
27677 079/300 Write block 400 len 24576 data: 4 3 3 3 4 5
27678 080/300 Write block 234 len 8192 data: 4 5
27679 081/300 Write block 3657 len 36864 data: 5 6 6 5 4 4 3 4 4
27680 082/300 Read block 1727 len 40960
27681 083/300 Read block 635 len 8192
27682 084/300 Read block 2162 len 20480
27683 085/300 Write block 4917 len 32768 data: 13 11 10 9 7 6 6 5
27684 086/300 Write block 3444 len 20480 data: 4 5 4 3 3
27685 087/300 Write block 318 len 24576 data: 6 5 5 5 4 5
27686 088/300 Read block 225 len 12288
27687 089/300 Write block 3412 len 16384 data: 5 6 6 8
27688 090/300 Read block 2563 len 32768
27689 091/300 Write block 1664 len 28672 data: 5 7 7 6 5 4 4
27690 092/300 Read block 3414 len 40960
27691 093/300 Write block 3586 len 32768 data: 12 9 8 5 5 5 6 5
27692 094/300 Read block 2107 len 32768
27693 095/300 Write block 537 len 16384 data: 6 7 7 7
27694 096/300 Write block 511 len 16384 data: 2 3 4 3
27695 097/300 Write block 4737 len 16384 data: 5 6 7 7
27696 098/300 Flush
27697 099/300 Write block 3381 len 28672 data: 4 4 4 3 4 5 4
27698 100/300 Write block 1493 len 20480 data: 8 8 8 8 8
27699 101/300 Write block 2478 len 36864 data: 5 5 6 5 5 4 4 4 5
27700 102/300 Write block 4015 len 16384 data: 6 5 6 6
27701 103/300 Write block 1969 len 8192 data: 5 6
27702 104/300 Write block 1729 len 4096 data: 6
27703 105/300 Read block 1942 len 40960
27704 106/300 Flush
27705 107/300 Read block 97 len 16384
27706 108/300 Flush
27707 109/300 Flush
27708 110/300 Write block 4733 len 32768 data: 8 5 6 4 6 7 8 8
27709 111/300 Write block 3064 len 32768 data: 6 6 5 5 4 4 5 4
27710 112/300 Read block 1974 len 28672
27711 113/300 Read block 570 len 12288
27712 114/300 Flush
27713 115/300 Read block 719 len 24576
27714 116/300 Read block 624 len 4096
27715 117/300 Write block 113 len 36864 data: 5 5 7 5 5 5 5 6 6
27716 118/300 Write block 4615 len 24576 data: 7 8 7 8 6 7
27717 119/300 Read block 838 len 36864
27718 120/300 Read block 421 len 24576
27719 121/300 Read block 3807 len 28672
27720 122/300 Read block 2346 len 16384
27721 123/300 Flush
27722 124/300 Write block 2199 len 36864 data: 4 5 5 4 4 4 3 3 4
27723 125/300 Read block 4623 len 24576
27724 126/300 Read block 3540 len 28672
27725 127/300 Read block 657 len 20480
27726 128/300 Write block 834 len 40960 data: 5 5 6 5 5 4 4 3 4 5
27727 129/300 Read block 1025 len 20480
27728 130/300 Write block 4243 len 32768 data: 4 5 2 2 2 2 2 2
27729 131/300 Read block 1529 len 40960
27730 132/300 Read block 542 len 28672
27731 133/300 Read block 3091 len 32768
27732 134/300 Flush
27733 135/300 Read block 4337 len 4096
27734 136/300 Write block 4587 len 36864 data: 5 6 5 5 5 6 4 4 2
27735 137/300 Read block 4592 len 40960
27736 138/300 Read block 3867 len 32768
27737 139/300 Read block 4752 len 24576
27738 140/300 Read block 4008 len 28672
27739 141/300 Write block 3872 len 28672 data: 8 10 8 7 5 5 5
27740 142/300 Read block 2517 len 16384
27741 143/300 Read block 1318 len 36864
27742 144/300 Read block 3976 len 36864
27743 145/300 Read block 4559 len 8192
27744 146/300 Read block 1340 len 20480
27745 147/300 Read block 2711 len 4096
27746 148/300 Read block 3864 len 32768
27747 149/300 Write block 89 len 16384 data: 4 4 4 5
27748 150/300 Write block 816 len 4096 data: 5
27749 151/300 Read block 545 len 24576
27750 152/300 Read block 144 len 40960
27751 153/300 Write block 4371 len 20480 data: 4 5 5 5 5
27752 154/300 Read block 3856 len 12288
27753 155/300 Write block 833 len 12288 data: 4 6 6
27754 156/300 Write block 1247 len 40960 data: 4 4 4 3 3 3 2 3 3 3
27755 157/300 Read block 2998 len 24576
27756 158/300 Write block 3596 len 36864 data: 5 3 3 4 5 4 4 3 4
27757 159/300 Write block 1803 len 28672 data: 5 5 6 6 5 5 5
27758 160/300 Write block 4331 len 16384 data: 5 3 3 5
27759 161/300 Read block 2724 len 40960
27760 162/300 Read block 3547 len 20480
27761 163/300 Read block 4237 len 28672
27762 164/300 Write block 725 len 8192 data: 7 5
27763 165/300 Write block 4672 len 4096 data: 7
27764 166/300 Read block 1858 len 4096
27765 167/300 Read block 3380 len 4096
27766 168/300 Read block 363 len 24576
27767 169/300 Read block 3756 len 16384
27768 170/300 Read block 4984 len 4096
27769 171/300 Read block 604 len 4096
27770 172/300 Write block 1959 len 40960 data: 3 3 3 4 4 4 3 3 4 4
27771 173/300 Write block 4972 len 40960 data: 4 5 4 6 5 4 3 3 4 3
27772 174/300 Read block 1479 len 12288
27773 175/300 Write block 1859 len 20480 data: 5 5 5 5 5
27774 176/300 Read block 1882 len 16384
27775 177/300 Write block 1690 len 20480 data: 4 4 5 5 4
27776 178/300 Write block 399 len 4096 data: 5
27777 179/300 Read block 795 len 12288
27778 180/300 Write block 4709 len 12288 data: 3 5 5
27779 181/300 Write block 1258 len 28672 data: 5 6 6 6 6 6 5
27780 182/300 Write block 4751 len 12288 data: 5 4 4
27781 183/300 Read block 228 len 12288
27782 184/300 Read block 4102 len 24576
27783 185/300 Read block 2149 len 28672
27784 186/300 Write block 3166 len 12288 data: 8 9 8
27785 187/300 Read block 1733 len 20480
27786 188/300 Read block 4464 len 4096
27787 189/300 Write block 4406 len 24576 data: 3 3 3 4 6 7
27788 190/300 Read block 3508 len 4096
27789 191/300 Flush
27790 192/300 Write block 1995 len 4096 data: 3
27791 193/300 Write block 876 len 16384 data: 2 4 4 5
27792 194/300 Read block 691 len 12288
27793 195/300 Flush
27794 196/300 Write block 3303 len 28672 data: 3 3 2 4 3 3 3
27795 197/300 Read block 4593 len 12288
27796 198/300 Read block 3924 len 36864
27797 199/300 Read block 1401 len 20480
27798 200/300 Read block 4764 len 8192
27799 201/300 Read block 1594 len 16384
27800 202/300 Flush
27801 203/300 Read block 3532 len 28672
27802 204/300 Read block 1148 len 20480
27803 205/300 Read block 1261 len 28672
27804 206/300 Flush
27805 207/300 Read block 896 len 16384
27806 208/300 Read block 1411 len 36864
27807 209/300 Write block 3603 len 40960 data: 4 5 4 5 8 7 6 7 6 4
27808 210/300 Read block 4293 len 20480
27809 211/300 Write block 2030 len 28672 data: 4 4 5 6 6 7 9
27810 212/300 Write block 3436 len 36864 data: 7 5 6 6 5 4 4 4 5
27811 213/300 Write block 2055 len 40960 data: 4 4 4 2 2 2 2 2 2 2
27812 214/300 Flush
27813 215/300 Write block 3987 len 4096 data: 7
27814 216/300 Write block 425 len 40960 data: 4 5 7 9 9 9 10 10 9 6
27815 217/300 Write block 2552 len 24576 data: 3 3 3 3 3 5
27816 218/300 Flush
27817 219/300 Write block 3901 len 32768 data: 4 4 3 3 2 5 5 5
27818 220/300 Read block 4823 len 12288
27819 221/300 Read block 3335 len 24576
27820 222/300 Write block 4130 len 16384 data: 5 4 4 5
27821 223/300 Write block 2631 len 12288 data: 7 7 5
27822 224/300 Write block 1447 len 24576 data: 5 6 6 7 8 7
27823 225/300 Write block 2326 len 28672 data: 8 5 5 5 4 3 3
27824 226/300 Read block 3947 len 28672
27825 227/300 Read block 1604 len 28672
27826 228/300 Read block 2664 len 36864
27827 229/300 Write block 4387 len 4096 data: 5
27828 230/300 Read block 2639 len 32768
27829 231/300 Write block 3479 len 8192 data: 9 10
27830 232/300 Read block 3422 len 20480
27831 233/300 Read block 220 len 40960
27832 234/300 Write block 2815 len 20480 data: 5 5 6 4 4
27833 235/300 Read block 2722 len 16384
27834 236/300 Read block 522 len 20480
27835 237/300 Read block 1520 len 28672
27836 238/300 Write block 2739 len 8192 data: 5 5
27837 239/300 Read block 2550 len 32768
27838 240/300 Read block 2097 len 28672
27839 241/300 Read block 2429 len 32768
27840 242/300 Read block 1443 len 12288
27841 243/300 Read block 1717 len 4096
27842 244/300 Read block 1604 len 12288
27843 245/300 Write block 2713 len 40960 data: 5 4 7 7 8 8 6 6 7 4
27844 246/300 Write block 4806 len 8192 data: 4 3
27845 247/300 Read block 3755 len 4096
27846 248/300 Flush
27847 249/300 Read block 3023 len 8192
27848 250/300 Flush
27849 251/300 Read block 1252 len 28672
27850 252/300 Write block 1127 len 16384 data: 9 9 8 8
27851 253/300 Flush
27852 254/300 Flush
27853 255/300 Write block 2103 len 4096 data: 4
27854 256/300 Write block 4816 len 40960 data: 8 8 8 7 7 6 6 6 6 5
27855 257/300 Read block 4369 len 12288
27856 258/300 Flush
27857 259/300 Read block 3886 len 4096
27858 260/300 Write block 1893 len 8192 data: 7 6
27859 261/300 Write block 1117 len 24576 data: 4 4 5 4 4 4
27860 262/300 Flush
27861 263/300 Write block 4010 len 36864 data: 6 6 6 6 6 7 6 7 7
27862 264/300 Write block 4417 len 16384 data: 6 9 7 5
27863 265/300 Write block 4698 len 28672 data: 5 5 4 4 6 6 5
27864 266/300 Write block 1556 len 12288 data: 6 5 4
27865 267/300 Write block 2253 len 24576 data: 3 4 5 4 3 3
27866 268/300 Write block 2878 len 32768 data: 8 7 7 6 5 6 7 6
27867 269/300 Flush
27868 270/300 Read block 2999 len 36864
27869 271/300 Read block 844 len 12288
27870 272/300 Write block 1922 len 28672 data: 5 5 5 4 4 5 6
27871 273/300 Read block 4276 len 20480
27872 274/300 Write block 4664 len 36864 data: 5 4 5 5 5 6 5 7 8
27873 275/300 Read block 3986 len 40960
27874 276/300 Read block 3964 len 32768
27875 277/300 Read block 800 len 36864
27876 278/300 Read block 1542 len 20480
27877 279/300 Write block 1119 len 40960 data: 6 5 5 5 6 6 8 9 10 10
27878 280/300 Flush
27879 281/300 Read block 1656 len 24576
27880 282/300 Flush
27881 283/300 Read block 4083 len 40960
27882 284/300 Read block 3306 len 32768
27883 285/300 Flush
27884 286/300 Flush
27885 287/300 Read block 86 len 12288
27886 288/300 Write block 1045 len 24576 data: 6 7 7 7 8 7
27887 289/300 Write block 1804 len 8192 data: 6 7
27888 290/300 Read block 4991 len 16384
27889 291/300 Read block 2510 len 8192
27890 292/300 Read block 3372 len 32768
27891 293/300 Write block 4331 len 8192 data: 6 4
27892 294/300 Read block 4952 len 8192
27893 295/300 Read block 432 len 4096
27894 296/300 Read block 1251 len 24576
27895 297/300 Write block 346 len 36864 data: 2 3 2 3 4 4 4 5 6
27896 298/300 Flush
27897 299/300 Read block 1912 len 36864
27898 300/300 Write block 4388 len 8192 data: 4 5
27899 301/300 Write block 3273 len 36864 data: 7 6 6 4 4 5 5 4 4
27900 Main task has work to do, go find it
27901 got dsc Start(2)
27902 start 2
27903 Feb 21 06:51:33.563 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5c60f1a8-c178-41a6-a495-79691c20a770, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
27904 Replay: started 2, returned:Ok(())
27905 ----------------------------------------------------------------
27906 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27907 States: Active Active Offline
27908 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27909 8245 Acked Read 7 Done Done Sent false
27910 8246 Acked Read 2 Done Done Sent false
27911 8247 Acked Read 2 Done Done Sent false
27912 8248 Acked Write 7 Done Done Sent false
27913 8249 Acked Read 9 Done Done Sent false
27914 8250 Acked Read 3 Done Done Sent false
27915 8251 Acked Write 8 Done Done Sent false
27916 8252 Acked Write 5 Done Done Sent false
27917 8253 Acked Read 3 Done Done Sent false
27918 8254 Acked Read 7 Done Done Sent false
27919 8255 Acked Write 1 Done Done Sent false
27920 8256 Acked Read 5 Done Done Sent false
27921 8257 Acked Read 1 Done Done Sent false
27922 8258 Acked Write 10 Done Done Sent false
27923 8259 Acked Flush 0 Done Done Sent false
27924 8260 Acked Read 8 Done Done Sent false
27925 8261 Acked Read 8 Done Done Sent false
27926 8262 Acked Flush 0 Done Done Sent false
27927 8263 Acked Read 7 Done Done Sent false
27928 8264 Acked Write 8 Done Done Sent false
27929 8265 Acked Read 7 Done Done Sent false
27930 8266 Acked Read 9 Done Done Sent false
27931 8267 Acked Flush 0 Done Done Sent false
27932 8268 Acked Read 7 Done Done Sent false
27933 8269 Acked Read 5 Done Done Sent false
27934 8270 Acked Flush 0 Done Done Sent false
27935 8271 Acked Write 8 Done Done Sent false
27936 8272 Acked Read 2 Done Done Sent false
27937 8273 Acked Write 10 Done Done Sent false
27938 8274 Acked Flush 0 Done Done Sent false
27939 8275 Acked Read 10 Done Done Sent false
27940 8276 Acked Write 4 Done Done Sent false
27941 8277 Acked Write 10 Done Done Sent false
27942 8278 Acked Read 5 Done Done Sent false
27943 8279 Acked Flush 0 Done Done Sent false
27944 8280 Acked Write 10 Done Done Sent false
27945 8281 Acked Read 9 Done Done Sent false
27946 8282 Acked Read 7 Done Done Sent false
27947 8283 Acked Read 4 Done Done Sent false
27948 8284 Acked Read 4 Done Done Sent false
27949 8285 Acked Write 2 Done Done Sent false
27950 8286 Acked Read 6 Done Done Sent false
27951 8287 Acked Write 5 Done Done Sent false
27952 8288 Acked Read 8 Done Done Sent false
27953 8289 Acked Write 5 Done Done Sent false
27954 8290 Acked Write 2 Done Done Sent false
27955 8291 Acked Read 10 Done Done Sent false
27956 8292 Acked Flush 0 Done Done Sent false
27957 8293 Acked Read 10 Done Done Sent false
27958 8294 Acked Flush 0 Done Done Sent false
27959 8295 Acked Write 5 Done Done Sent false
27960 8296 Acked Write 10 Done Done Sent false
27961 8297 Acked Write 10 Done Done Sent false
27962 8298 Acked Write 2 Done Done Sent false
27963 8299 Acked Write 9 Done Done Sent false
27964 8300 Acked Write 1 Done Done Sent false
27965 8301 Acked Read 7 Done Done Sent false
27966 8302 Acked Read 2 Done Done Sent false
27967 8303 Acked Write 4 Done Done Sent false
27968 8304 Acked Write 7 Done Done Sent false
27969 8305 Acked Read 3 Done Done Sent false
27970 8306 Acked Read 10 Done Done Sent false
27971 8307 Acked Write 4 Done Done Sent false
27972 8308 Acked Write 2 Done Done Sent false
27973 8309 Acked Write 8 Done Done Sent false
27974 8310 Acked Write 2 Done Done Sent false
27975 8311 Acked Write 5 Done Done Sent false
27976 8312 Acked Read 3 Done Done Sent false
27977 8313 Acked Read 10 Done Done Sent false
27978 8314 Acked Read 6 Done Done Sent false
27979 8315 Acked Write 4 Done Done Sent false
27980 8316 Acked Write 2 Done Done Sent false
27981 8317 Acked Write 5 Done Done Sent false
27982 8318 Acked Read 9 Done Done Sent false
27983 8319 Acked Flush 0 Done Done Sent false
27984 8320 Acked Write 3 Done Done Sent false
27985 8321 Acked Read 2 Done Done Sent false
27986 8322 Acked Read 9 Done Done Sent false
27987 8323 Acked Write 6 Done Done Sent false
27988 8324 Acked Write 2 Done Done Sent false
27989 8325 Acked Write 9 Done Done Sent false
27990 8326 Acked Read 10 Done Done Sent false
27991 8327 Acked Read 2 Done Done Sent false
27992 8328 Acked Read 5 Done Done Sent false
27993 8329 Acked Write 8 Done Done Sent false
27994 8330 Acked Write 5 Done Done Sent false
27995 8331 Acked Write 6 Done Done Sent false
27996 8332 Acked Read 3 Done Done Sent false
27997 8333 Acked Write 4 Done Done Sent false
27998 8334 Acked Read 8 Done Done Sent false
27999 8335 Acked Write 7 Done Done Sent false
28000 8336 Acked Read 10 Done Done Sent false
28001 8337 Acked Write 8 Done Done Sent false
28002 8338 Acked Read 8 Done Done Sent false
28003 8339 Acked Write 4 Done Done Sent false
28004 8340 Acked Write 4 Done Done Sent false
28005 8341 Acked Write 4 Done Done Sent false
28006 8342 Acked Flush 0 Done Done Sent false
28007 8343 Acked Write 7 Done Done Sent false
28008 8344 Acked Write 5 Done Done Sent false
28009 8345 Acked Write 9 Done Done Sent false
28010 8346 Acked Write 4 Done Done Sent false
28011 8347 Acked Write 2 Done Done Sent false
28012 8348 Acked Write 1 Done Done Sent false
28013 8349 Acked Read 10 Done Done Sent false
28014 8350 Acked Flush 0 Done Done Sent false
28015 8351 Acked Read 4 Done Done Sent false
28016 8352 Acked Flush 0 Done Done Sent false
28017 8353 Acked Flush 0 Done Done Sent false
28018 8354 Acked Write 8 Done Done Sent false
28019 8355 Acked Write 8 Done Done Sent false
28020 8356 Acked Read 7 Done Done Sent false
28021 8357 Acked Read 3 Done Done Sent false
28022 8358 Acked Flush 0 Done Done Sent false
28023 8359 Acked Read 6 Done Done Sent false
28024 8360 Acked Read 1 Done Done Sent false
28025 8361 Acked Write 9 Done Done Sent false
28026 8362 Acked Write 6 Done Done Sent false
28027 8363 Acked Read 9 Done Done Sent false
28028 8364 Acked Read 6 Done Done Sent false
28029 8365 Acked Read 7 Done Done Sent false
28030 8366 Acked Read 4 Done Done Sent false
28031 8367 Acked Flush 0 Done Done Sent false
28032 8368 Acked Write 9 Done Done Sent false
28033 8369 Acked Read 6 Done Done Sent false
28034 8370 Acked Read 7 Done Done Sent false
28035 8371 Acked Read 5 Done Done Sent false
28036 8372 Acked Write 10 Done Done Sent false
28037 8373 Acked Read 5 Done Done Sent false
28038 8374 Acked Write 8 Done Done Sent false
28039 8375 Acked Read 10 Done Done Sent false
28040 8376 Acked Read 7 Done Done Sent false
28041 8377 Acked Read 8 Done Done Sent false
28042 8378 Acked Flush 0 Done Done Sent false
28043 8379 Acked Read 1 Done Done Sent false
28044 8380 Acked Write 9 Done Done Sent false
28045 8381 Acked Read 10 Done Done Sent false
28046 8382 Acked Read 8 Done Done Sent false
28047 8383 Acked Read 6 Done Done Sent false
28048 8384 Acked Read 7 Done Done Sent false
28049 8385 Acked Write 7 Done Done Sent false
28050 8386 Acked Read 4 Done Done Sent false
28051 8387 Acked Read 9 Done Done Sent false
28052 8388 Acked Read 9 Done Done Sent false
28053 8389 Acked Read 2 Done Done Sent false
28054 8390 Acked Read 5 Done Done Sent false
28055 8391 Acked Read 1 Done Done Sent false
28056 8392 Acked Read 8 Done Done Sent false
28057 8393 Acked Write 4 Done Done Sent false
28058 8394 Acked Write 1 Done Done Sent false
28059 8395 Acked Read 6 Done Done Sent false
28060 8396 Acked Read 10 Done Done Sent false
28061 8397 Acked Write 5 Done Done Sent false
28062 8398 Acked Read 3 Done Done Sent false
28063 8399 Acked Write 3 Done Done Sent false
28064 8400 Acked Write 10 Done Done Sent false
28065 8401 Acked Read 6 Done Done Sent false
28066 8402 Acked Write 9 Done Done Sent false
28067 8403 Acked Write 7 Done Done Sent false
28068 8404 Acked Write 4 Done Done Sent false
28069 8405 Acked Read 10 Done Done Sent false
28070 8406 Acked Read 5 Done Done Sent false
28071 8407 Acked Read 7 Done Done Sent false
28072 8408 Acked Write 2 Done Done Sent false
28073 8409 Acked Write 1 Done Done Sent false
28074 8410 Acked Read 1 Done Done Sent false
28075 8411 Acked Read 1 Done Done Sent false
28076 8412 Acked Read 6 Done Done Sent false
28077 8413 Acked Read 4 Done Done Sent false
28078 8414 Acked Read 1 Done Done Sent false
28079 8415 Acked Read 1 Done Done Sent false
28080 8416 Acked Write 10 Done Done Sent false
28081 8417 Acked Write 10 Done Done Sent false
28082 8418 Acked Read 3 Done Done Sent false
28083 8419 Acked Write 5 Done Done Sent false
28084 8420 Acked Read 4 Done Done Sent false
28085 8421 Acked Write 5 Done Done Sent false
28086 8422 Acked Write 1 Done Done Sent false
28087 8423 Acked Read 3 Done Done Sent false
28088 8424 Acked Write 3 Done Done Sent false
28089 8425 Acked Write 7 Done Done Sent false
28090 8426 Acked Write 3 Done Done Sent false
28091 8427 Acked Read 3 Done Done Sent false
28092 8428 Acked Read 6 Done Done Sent false
28093 8429 Acked Read 7 Done Done Sent false
28094 8430 Acked Write 3 Done Done Sent false
28095 8431 Acked Read 5 Done Done Sent false
28096 8432 Acked Read 1 Done Done Sent false
28097 8433 Acked Write 6 Done Done Sent false
28098 8434 Acked Read 1 Done Done Sent false
28099 8435 Acked Flush 0 Done Done Sent false
28100 8436 Acked Write 1 Done Done Sent false
28101 8437 Acked Write 4 Done Done Sent false
28102 8438 Acked Read 3 Done Done Sent false
28103 8439 Acked Flush 0 Done Done Sent false
28104 8440 Acked Write 7 Done Done Sent false
28105 8441 Acked Read 3 Done Done Sent false
28106 8442 Acked Read 9 Done Done Sent false
28107 8443 Acked Read 5 Done Done Sent false
28108 8444 Acked Read 2 Done Done Sent false
28109 8445 Acked Read 4 Done Done Sent false
28110 8446 Acked Flush 0 Done Done Sent false
28111 8447 Acked Read 7 Done Done Sent false
28112 8448 Acked Read 5 Done Done Sent false
28113 8449 Acked Read 7 Done Done Sent false
28114 8450 Acked Flush 0 Done Done Sent false
28115 8451 Acked Read 4 Done Done Sent false
28116 8452 Acked Read 9 Done Done Sent false
28117 8453 Acked Write 10 Done Done Sent false
28118 8454 Acked Read 5 Done Done Sent false
28119 8455 Acked Write 7 Done Done Sent false
28120 8456 Acked Write 9 Done Done Sent false
28121 8457 Acked Write 10 Done Done Sent false
28122 8458 Acked Flush 0 Done Done Sent false
28123 8459 Acked Write 1 Done Done Sent false
28124 8460 Acked Write 10 Done Done Sent false
28125 8461 Acked Write 6 Done Done Sent false
28126 8462 Acked Flush 0 Done Done Sent false
28127 8463 Acked Write 8 Done Done Sent false
28128 8464 Acked Read 3 Done Done Sent false
28129 8465 Acked Read 6 Done Done Sent false
28130 8466 Acked Write 4 Done Done Sent false
28131 8467 Acked Write 3 Done Done Sent false
28132 8468 Acked Write 6 Done Done Sent false
28133 8469 Acked Write 7 Done Done Sent false
28134 8470 Acked Read 7 Done Done Sent false
28135 8471 Acked Read 7 Done Done Sent false
28136 8472 Acked Read 9 Done Done Sent false
28137 8473 Acked Write 1 Done Done Sent false
28138 8474 Acked Read 8 Done Done Sent false
28139 8475 Acked Write 2 Done Done Sent false
28140 8476 Acked Read 5 Done Done Sent false
28141 8477 Acked Read 10 Done Done Sent false
28142 8478 Acked Write 5 Done Done Sent false
28143 8479 Acked Read 4 Done Done Sent false
28144 8480 Acked Read 5 Done Done Sent false
28145 8481 Acked Read 7 Done Done Sent false
28146 8482 Acked Write 2 Done Done Sent false
28147 8483 Acked Read 8 Done Done Sent false
28148 8484 Acked Read 7 Done Done Sent false
28149 8485 Acked Read 8 Done Done Sent false
28150 8486 Acked Read 3 Done Done Sent false
28151 8487 Acked Read 1 Done Done Sent false
28152 8488 Acked Read 3 Done Done Sent false
28153 8489 Acked Write 10 Done Done Sent false
28154 8490 Acked Write 2 Done Done Sent false
28155 8491 Acked Read 1 Done Done Sent false
28156 8492 Acked Flush 0 Done Done Sent false
28157 8493 Acked Read 2 Done Done Sent false
28158 8494 Acked Flush 0 Done Done Sent false
28159 8495 Acked Read 7 Done Done Sent false
28160 8496 Acked Write 4 Done Done Sent false
28161 8497 Acked Flush 0 Done Done Sent false
28162 8498 Acked Flush 0 Done Done Sent false
28163 8499 Acked Write 1 Done Done Sent false
28164 8500 Acked Write 10 Done Done Sent false
28165 8501 Acked Read 3 Done Done Sent false
28166 8502 Acked Flush 0 Done Done Sent false
28167 8503 Acked Read 1 Done Done Sent false
28168 8504 Acked Write 2 Done Done Sent false
28169 8505 Acked Write 6 Done Done Sent false
28170 8506 Acked Flush 0 Done Done Sent false
28171 8507 Acked Write 9 Done Done Sent false
28172 8508 Acked Write 4 Done Done Sent false
28173 8509 Acked Write 7 Done Done Sent false
28174 8510 Acked Write 3 Done Done Sent false
28175 8511 Acked Write 6 Done Done Sent false
28176 8512 Acked Write 8 Done Done Sent false
28177 8513 Acked Flush 0 Done Done Sent false
28178 8514 Acked Read 9 Done Done Sent false
28179 8515 Acked Read 3 Done Done Sent false
28180 8516 Acked Write 7 Done Done Sent false
28181 8517 Acked Read 5 Done Done Sent false
28182 8518 Acked Write 9 Done Done Sent false
28183 8519 Acked Read 10 Done Done Sent false
28184 8520 Acked Read 8 Done Done Sent false
28185 8521 Acked Read 9 Done Done Sent false
28186 8522 Acked Read 5 Done Done Sent false
28187 8523 Acked Write 10 Done Done Sent false
28188 8524 Acked Flush 0 Done Done Sent false
28189 8525 Acked Read 6 Done Done Sent false
28190 8526 Acked Flush 0 Done Done Sent false
28191 8527 Acked Read 10 Done Done Sent false
28192 8528 Acked Read 8 Done Done Sent false
28193 8529 Acked Flush 0 Done Done Sent false
28194 8530 Acked Flush 0 Done Done Sent false
28195 8531 Acked Read 3 Done Done Sent false
28196 8532 Acked Write 6 Done Done Sent false
28197 8533 Acked Write 2 Done Done Sent false
28198 8534 Acked Read 4 Done Done Sent false
28199 8535 Acked Read 2 Done Done Sent false
28200 8536 Acked Read 8 Done Done Sent false
28201 8537 Acked Write 2 Done Done Sent false
28202 8538 Acked Read 2 Done Done Sent false
28203 8539 Acked Read 1 Done Done Sent false
28204 8540 Acked Read 6 Done Done Sent false
28205 8541 Acked Write 9 Done Done Sent false
28206 8542 Acked Flush 0 Done Done Sent false
28207 8543 Acked Read 9 Done Done Sent false
28208 8544 Acked Write 2 Done Done Sent false
28209 8545 Acked Write 9 Done Done Sent false
28210 STATES DS:0 DS:1 DS:2 TOTAL
28211 Sent 0 0 301 301
28212 Done 301 301 0 602
28213 Skipped 0 0 0 0
28214 Error 0 0 0 0
28215 Last Flush: Some(JobId(8542)) Some(JobId(8542)) Some(JobId(8244))
28216 Downstairs last five completed: 8244 8243 8242 8241 8240
28217 Upstairs last five completed: 8545 8544 8543 8542 8541
28218 CLIENT: Up:0 ds:301 act:2
282192025-02-21T06:51:34.231ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
28220 [8830] got start action
28221 [8830] I am going to restart
28222 Starting downstairs at port 8830
28223 Make output file at "/tmp/dsc/downstairs-8830.txt"
28224 [8830][2] reports Starting
28225 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1231)
28226 [8830][2] reports Running
282272025-02-21T06:51:35.550ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
282282025-02-21T06:51:36.232ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
282292025-02-21T06:51:37.233ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
28230 ----------------------------------------------------------------
28231 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28232 States: Active Active Offline
28233 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28234 8245 Acked Read 7 Done Done Sent false
28235 8246 Acked Read 2 Done Done Sent false
28236 8247 Acked Read 2 Done Done Sent false
28237 8248 Acked Write 7 Done Done Sent false
28238 8249 Acked Read 9 Done Done Sent false
28239 8250 Acked Read 3 Done Done Sent false
28240 8251 Acked Write 8 Done Done Sent false
28241 8252 Acked Write 5 Done Done Sent false
28242 8253 Acked Read 3 Done Done Sent false
28243 8254 Acked Read 7 Done Done Sent false
28244 8255 Acked Write 1 Done Done Sent false
28245 8256 Acked Read 5 Done Done Sent false
28246 8257 Acked Read 1 Done Done Sent false
28247 8258 Acked Write 10 Done Done Sent false
28248 8259 Acked Flush 0 Done Done Sent false
28249 8260 Acked Read 8 Done Done Sent false
28250 8261 Acked Read 8 Done Done Sent false
28251 8262 Acked Flush 0 Done Done Sent false
28252 8263 Acked Read 7 Done Done Sent false
28253 8264 Acked Write 8 Done Done Sent false
28254 8265 Acked Read 7 Done Done Sent false
28255 8266 Acked Read 9 Done Done Sent false
28256 8267 Acked Flush 0 Done Done Sent false
28257 8268 Acked Read 7 Done Done Sent false
28258 8269 Acked Read 5 Done Done Sent false
28259 8270 Acked Flush 0 Done Done Sent false
28260 8271 Acked Write 8 Done Done Sent false
28261 8272 Acked Read 2 Done Done Sent false
28262 8273 Acked Write 10 Done Done Sent false
28263 8274 Acked Flush 0 Done Done Sent false
28264 8275 Acked Read 10 Done Done Sent false
28265 8276 Acked Write 4 Done Done Sent false
28266 8277 Acked Write 10 Done Done Sent false
28267 8278 Acked Read 5 Done Done Sent false
28268 8279 Acked Flush 0 Done Done Sent false
28269 8280 Acked Write 10 Done Done Sent false
28270 8281 Acked Read 9 Done Done Sent false
28271 8282 Acked Read 7 Done Done Sent false
28272 8283 Acked Read 4 Done Done Sent false
28273 8284 Acked Read 4 Done Done Sent false
28274 8285 Acked Write 2 Done Done Sent false
28275 8286 Acked Read 6 Done Done Sent false
28276 8287 Acked Write 5 Done Done Sent false
28277 8288 Acked Read 8 Done Done Sent false
28278 8289 Acked Write 5 Done Done Sent false
28279 8290 Acked Write 2 Done Done Sent false
28280 8291 Acked Read 10 Done Done Sent false
28281 8292 Acked Flush 0 Done Done Sent false
28282 8293 Acked Read 10 Done Done Sent false
28283 8294 Acked Flush 0 Done Done Sent false
28284 8295 Acked Write 5 Done Done Sent false
28285 8296 Acked Write 10 Done Done Sent false
28286 8297 Acked Write 10 Done Done Sent false
28287 8298 Acked Write 2 Done Done Sent false
28288 8299 Acked Write 9 Done Done Sent false
28289 8300 Acked Write 1 Done Done Sent false
28290 8301 Acked Read 7 Done Done Sent false
28291 8302 Acked Read 2 Done Done Sent false
28292 8303 Acked Write 4 Done Done Sent false
28293 8304 Acked Write 7 Done Done Sent false
28294 8305 Acked Read 3 Done Done Sent false
28295 8306 Acked Read 10 Done Done Sent false
28296 8307 Acked Write 4 Done Done Sent false
28297 8308 Acked Write 2 Done Done Sent false
28298 8309 Acked Write 8 Done Done Sent false
28299 8310 Acked Write 2 Done Done Sent false
28300 8311 Acked Write 5 Done Done Sent false
28301 8312 Acked Read 3 Done Done Sent false
28302 8313 Acked Read 10 Done Done Sent false
28303 8314 Acked Read 6 Done Done Sent false
28304 8315 Acked Write 4 Done Done Sent false
28305 8316 Acked Write 2 Done Done Sent false
28306 8317 Acked Write 5 Done Done Sent false
28307 8318 Acked Read 9 Done Done Sent false
28308 8319 Acked Flush 0 Done Done Sent false
28309 8320 Acked Write 3 Done Done Sent false
28310 8321 Acked Read 2 Done Done Sent false
28311 8322 Acked Read 9 Done Done Sent false
28312 8323 Acked Write 6 Done Done Sent false
28313 8324 Acked Write 2 Done Done Sent false
28314 8325 Acked Write 9 Done Done Sent false
28315 8326 Acked Read 10 Done Done Sent false
28316 8327 Acked Read 2 Done Done Sent false
28317 8328 Acked Read 5 Done Done Sent false
28318 8329 Acked Write 8 Done Done Sent false
28319 8330 Acked Write 5 Done Done Sent false
28320 8331 Acked Write 6 Done Done Sent false
28321 8332 Acked Read 3 Done Done Sent false
28322 8333 Acked Write 4 Done Done Sent false
28323 8334 Acked Read 8 Done Done Sent false
28324 8335 Acked Write 7 Done Done Sent false
28325 8336 Acked Read 10 Done Done Sent false
28326 8337 Acked Write 8 Done Done Sent false
28327 8338 Acked Read 8 Done Done Sent false
28328 8339 Acked Write 4 Done Done Sent false
28329 8340 Acked Write 4 Done Done Sent false
28330 8341 Acked Write 4 Done Done Sent false
28331 8342 Acked Flush 0 Done Done Sent false
28332 8343 Acked Write 7 Done Done Sent false
28333 8344 Acked Write 5 Done Done Sent false
28334 8345 Acked Write 9 Done Done Sent false
28335 8346 Acked Write 4 Done Done Sent false
28336 8347 Acked Write 2 Done Done Sent false
28337 8348 Acked Write 1 Done Done Sent false
28338 8349 Acked Read 10 Done Done Sent false
28339 8350 Acked Flush 0 Done Done Sent false
28340 8351 Acked Read 4 Done Done Sent false
28341 8352 Acked Flush 0 Done Done Sent false
28342 8353 Acked Flush 0 Done Done Sent false
28343 8354 Acked Write 8 Done Done Sent false
28344 8355 Acked Write 8 Done Done Sent false
28345 8356 Acked Read 7 Done Done Sent false
28346 8357 Acked Read 3 Done Done Sent false
28347 8358 Acked Flush 0 Done Done Sent false
28348 8359 Acked Read 6 Done Done Sent false
28349 8360 Acked Read 1 Done Done Sent false
28350 8361 Acked Write 9 Done Done Sent false
28351 8362 Acked Write 6 Done Done Sent false
28352 8363 Acked Read 9 Done Done Sent false
28353 8364 Acked Read 6 Done Done Sent false
28354 8365 Acked Read 7 Done Done Sent false
28355 8366 Acked Read 4 Done Done Sent false
28356 8367 Acked Flush 0 Done Done Sent false
28357 8368 Acked Write 9 Done Done Sent false
28358 8369 Acked Read 6 Done Done Sent false
28359 8370 Acked Read 7 Done Done Sent false
28360 8371 Acked Read 5 Done Done Sent false
28361 8372 Acked Write 10 Done Done Sent false
28362 8373 Acked Read 5 Done Done Sent false
28363 8374 Acked Write 8 Done Done Sent false
28364 8375 Acked Read 10 Done Done Sent false
28365 8376 Acked Read 7 Done Done Sent false
28366 8377 Acked Read 8 Done Done Sent false
28367 8378 Acked Flush 0 Done Done Sent false
28368 8379 Acked Read 1 Done Done Sent false
28369 8380 Acked Write 9 Done Done Sent false
28370 8381 Acked Read 10 Done Done Sent false
28371 8382 Acked Read 8 Done Done Sent false
28372 8383 Acked Read 6 Done Done Sent false
28373 8384 Acked Read 7 Done Done Sent false
28374 8385 Acked Write 7 Done Done Sent false
28375 8386 Acked Read 4 Done Done Sent false
28376 8387 Acked Read 9 Done Done Sent false
28377 8388 Acked Read 9 Done Done Sent false
28378 8389 Acked Read 2 Done Done Sent false
28379 8390 Acked Read 5 Done Done Sent false
28380 8391 Acked Read 1 Done Done Sent false
28381 8392 Acked Read 8 Done Done Sent false
28382 8393 Acked Write 4 Done Done Sent false
28383 8394 Acked Write 1 Done Done Sent false
28384 8395 Acked Read 6 Done Done Sent false
28385 8396 Acked Read 10 Done Done Sent false
28386 8397 Acked Write 5 Done Done Sent false
28387 8398 Acked Read 3 Done Done Sent false
28388 8399 Acked Write 3 Done Done Sent false
28389 8400 Acked Write 10 Done Done Sent false
28390 8401 Acked Read 6 Done Done Sent false
28391 8402 Acked Write 9 Done Done Sent false
28392 8403 Acked Write 7 Done Done Sent false
28393 8404 Acked Write 4 Done Done Sent false
28394 8405 Acked Read 10 Done Done Sent false
28395 8406 Acked Read 5 Done Done Sent false
28396 8407 Acked Read 7 Done Done Sent false
28397 8408 Acked Write 2 Done Done Sent false
28398 8409 Acked Write 1 Done Done Sent false
28399 8410 Acked Read 1 Done Done Sent false
28400 8411 Acked Read 1 Done Done Sent false
28401 8412 Acked Read 6 Done Done Sent false
28402 8413 Acked Read 4 Done Done Sent false
28403 8414 Acked Read 1 Done Done Sent false
28404 8415 Acked Read 1 Done Done Sent false
28405 8416 Acked Write 10 Done Done Sent false
28406 8417 Acked Write 10 Done Done Sent false
28407 8418 Acked Read 3 Done Done Sent false
28408 8419 Acked Write 5 Done Done Sent false
28409 8420 Acked Read 4 Done Done Sent false
28410 8421 Acked Write 5 Done Done Sent false
28411 8422 Acked Write 1 Done Done Sent false
28412 8423 Acked Read 3 Done Done Sent false
28413 8424 Acked Write 3 Done Done Sent false
28414 8425 Acked Write 7 Done Done Sent false
28415 8426 Acked Write 3 Done Done Sent false
28416 8427 Acked Read 3 Done Done Sent false
28417 8428 Acked Read 6 Done Done Sent false
28418 8429 Acked Read 7 Done Done Sent false
28419 8430 Acked Write 3 Done Done Sent false
28420 8431 Acked Read 5 Done Done Sent false
28421 8432 Acked Read 1 Done Done Sent false
28422 8433 Acked Write 6 Done Done Sent false
28423 8434 Acked Read 1 Done Done Sent false
28424 8435 Acked Flush 0 Done Done Sent false
28425 8436 Acked Write 1 Done Done Sent false
28426 8437 Acked Write 4 Done Done Sent false
28427 8438 Acked Read 3 Done Done Sent false
28428 8439 Acked Flush 0 Done Done Sent false
28429 8440 Acked Write 7 Done Done Sent false
28430 8441 Acked Read 3 Done Done Sent false
28431 8442 Acked Read 9 Done Done Sent false
28432 8443 Acked Read 5 Done Done Sent false
28433 8444 Acked Read 2 Done Done Sent false
28434 8445 Acked Read 4 Done Done Sent false
28435 8446 Acked Flush 0 Done Done Sent false
28436 8447 Acked Read 7 Done Done Sent false
28437 8448 Acked Read 5 Done Done Sent false
28438 8449 Acked Read 7 Done Done Sent false
28439 8450 Acked Flush 0 Done Done Sent false
28440 8451 Acked Read 4 Done Done Sent false
28441 8452 Acked Read 9 Done Done Sent false
28442 8453 Acked Write 10 Done Done Sent false
28443 8454 Acked Read 5 Done Done Sent false
28444 8455 Acked Write 7 Done Done Sent false
28445 8456 Acked Write 9 Done Done Sent false
28446 8457 Acked Write 10 Done Done Sent false
28447 8458 Acked Flush 0 Done Done Sent false
28448 8459 Acked Write 1 Done Done Sent false
28449 8460 Acked Write 10 Done Done Sent false
28450 8461 Acked Write 6 Done Done Sent false
28451 8462 Acked Flush 0 Done Done Sent false
28452 8463 Acked Write 8 Done Done Sent false
28453 8464 Acked Read 3 Done Done Sent false
28454 8465 Acked Read 6 Done Done Sent false
28455 8466 Acked Write 4 Done Done Sent false
28456 8467 Acked Write 3 Done Done Sent false
28457 8468 Acked Write 6 Done Done Sent false
28458 8469 Acked Write 7 Done Done Sent false
28459 8470 Acked Read 7 Done Done Sent false
28460 8471 Acked Read 7 Done Done Sent false
28461 8472 Acked Read 9 Done Done Sent false
28462 8473 Acked Write 1 Done Done Sent false
28463 8474 Acked Read 8 Done Done Sent false
28464 8475 Acked Write 2 Done Done Sent false
28465 8476 Acked Read 5 Done Done Sent false
28466 8477 Acked Read 10 Done Done Sent false
28467 8478 Acked Write 5 Done Done Sent false
28468 8479 Acked Read 4 Done Done Sent false
28469 8480 Acked Read 5 Done Done Sent false
28470 8481 Acked Read 7 Done Done Sent false
28471 8482 Acked Write 2 Done Done Sent false
28472 8483 Acked Read 8 Done Done Sent false
28473 8484 Acked Read 7 Done Done Sent false
28474 8485 Acked Read 8 Done Done Sent false
28475 8486 Acked Read 3 Done Done Sent false
28476 8487 Acked Read 1 Done Done Sent false
28477 8488 Acked Read 3 Done Done Sent false
28478 8489 Acked Write 10 Done Done Sent false
28479 8490 Acked Write 2 Done Done Sent false
28480 8491 Acked Read 1 Done Done Sent false
28481 8492 Acked Flush 0 Done Done Sent false
28482 8493 Acked Read 2 Done Done Sent false
28483 8494 Acked Flush 0 Done Done Sent false
28484 8495 Acked Read 7 Done Done Sent false
28485 8496 Acked Write 4 Done Done Sent false
28486 8497 Acked Flush 0 Done Done Sent false
28487 8498 Acked Flush 0 Done Done Sent false
28488 8499 Acked Write 1 Done Done Sent false
28489 8500 Acked Write 10 Done Done Sent false
28490 8501 Acked Read 3 Done Done Sent false
28491 8502 Acked Flush 0 Done Done Sent false
28492 8503 Acked Read 1 Done Done Sent false
28493 8504 Acked Write 2 Done Done Sent false
28494 8505 Acked Write 6 Done Done Sent false
28495 8506 Acked Flush 0 Done Done Sent false
28496 8507 Acked Write 9 Done Done Sent false
28497 8508 Acked Write 4 Done Done Sent false
28498 8509 Acked Write 7 Done Done Sent false
28499 8510 Acked Write 3 Done Done Sent false
28500 8511 Acked Write 6 Done Done Sent false
28501 8512 Acked Write 8 Done Done Sent false
28502 8513 Acked Flush 0 Done Done Sent false
28503 8514 Acked Read 9 Done Done Sent false
28504 8515 Acked Read 3 Done Done Sent false
28505 8516 Acked Write 7 Done Done Sent false
28506 8517 Acked Read 5 Done Done Sent false
28507 8518 Acked Write 9 Done Done Sent false
28508 8519 Acked Read 10 Done Done Sent false
28509 8520 Acked Read 8 Done Done Sent false
28510 8521 Acked Read 9 Done Done Sent false
28511 8522 Acked Read 5 Done Done Sent false
28512 8523 Acked Write 10 Done Done Sent false
28513 8524 Acked Flush 0 Done Done Sent false
28514 8525 Acked Read 6 Done Done Sent false
28515 8526 Acked Flush 0 Done Done Sent false
28516 8527 Acked Read 10 Done Done Sent false
28517 8528 Acked Read 8 Done Done Sent false
28518 8529 Acked Flush 0 Done Done Sent false
28519 8530 Acked Flush 0 Done Done Sent false
28520 8531 Acked Read 3 Done Done Sent false
28521 8532 Acked Write 6 Done Done Sent false
28522 8533 Acked Write 2 Done Done Sent false
28523 8534 Acked Read 4 Done Done Sent false
28524 8535 Acked Read 2 Done Done Sent false
28525 8536 Acked Read 8 Done Done Sent false
28526 8537 Acked Write 2 Done Done Sent false
28527 8538 Acked Read 2 Done Done Sent false
28528 8539 Acked Read 1 Done Done Sent false
28529 8540 Acked Read 6 Done Done Sent false
28530 8541 Acked Write 9 Done Done Sent false
28531 8542 Acked Flush 0 Done Done Sent false
28532 8543 Acked Read 9 Done Done Sent false
28533 8544 Acked Write 2 Done Done Sent false
28534 8545 Acked Write 9 Done Done Sent false
28535 STATES DS:0 DS:1 DS:2 TOTAL
28536 Sent 0 0 301 301
28537 Done 301 301 0 602
28538 Skipped 0 0 0 0
28539 Error 0 0 0 0
28540 Last Flush: Some(JobId(8542)) Some(JobId(8542)) Some(JobId(8244))
28541 Downstairs last five completed: 8244 8243 8242 8241 8240
28542 Upstairs last five completed: 8545 8544 8543 8542 8541
28543 CLIENT: Up:0 ds:301 act:2
285442025-02-21T06:51:38.235ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285452025-02-21T06:51:39.237ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285462025-02-21T06:51:39.318ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285472025-02-21T06:51:39.318ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52840) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285482025-02-21T06:51:39.319ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285492025-02-21T06:51:39.320ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285502025-02-21T06:51:39.320ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285512025-02-21T06:51:39.320ZINFOcrucible: send last flush ID to this DS: Some(JobId(8244)) = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285522025-02-21T06:51:39.320ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285532025-02-21T06:51:39.320ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(8244)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285542025-02-21T06:51:39.320ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(8244)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
28555 ----------------------------------------------------------------
28556 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28557 Downstairs last five completed: 8546 8545 8544 8543 8542
28558 Upstairs last five completed: 8546 8545 8544 8543 8542
28559 CLIENT: Up:0 ds:0 act:3
28560 Replay: All jobs finished
28561 Main task has work to do, go find it
28562 got dsc Stop(2)
28563 stop 2
28564 Feb 21 06:51:41.569 INFO request completed, latency_us: 354, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 0b88b3b6-c4bf-4bd2-8161-88879ee7639e, remote_addr: [8830] Got stop action so:false kr:false
28565 127.0.0.1:53848, local_addr: 127.0.0.1:9998
28566 Feb 21 06:51:41.569 INFO request completed, latency_us: 109, response_code: 200, uri: /state/cid/2, method: GET, req_id: b0d4dce2-e4c1-4640-8272-6ae13a3324fe, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
285672025-02-21T06:51:41.569ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285682025-02-21T06:51:41.569ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285692025-02-21T06:51:41.569ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
285702025-02-21T06:51:41.569ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
28571 [8830] Exited with: ExitStatus(unix_wait_status(9))
28572 [8830][2] reports Exit
28573 Feb 21 06:51:45.571 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/2, method: GET, req_id: be187e43-13b4-4314-bcdc-f740170a914b, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
28574 001/300 Write block 806 len 12288 data: 5 4 4
28575 002/300 Write block 4409 len 20480 data: 5 7 8 8 7
28576 003/300 Write block 3066 len 8192 data: 6 6
28577 004/300 Flush
28578 005/300 Read block 2226 len 40960
28579 006/300 Read block 3660 len 36864
28580 007/300 Read block 3376 len 12288
28581 008/300 Write block 3300 len 12288 data: 3 3 3
28582 009/300 Flush
28583 010/300 Write block 2702 len 36864 data: 5 4 5 5 6 5 4 4 4
28584 011/300 Read block 1870 len 28672
28585 012/300 Read block 3091 len 40960
28586 013/300 Write block 1460 len 40960 data: 4 5 5 8 9 9 8 6 6 5
28587 014/300 Read block 4722 len 28672
28588 015/300 Read block 72 len 40960
28589 016/300 Read block 4904 len 28672
28590 017/300 Write block 4488 len 4096 data: 5
28591 018/300 Read block 3272 len 20480
28592 019/300 Write block 3423 len 28672 data: 5 4 5 5 5 5 5
28593 020/300 Read block 1104 len 12288
28594 021/300 Flush
28595 022/300 Write block 4284 len 16384 data: 7 6 5 5
28596 023/300 Read block 1596 len 28672
28597 024/300 Write block 376 len 32768 data: 5 6 4 5 5 8 9 8
28598 025/300 Write block 2701 len 28672 data: 5 6 5 6 6 7 6
28599 026/300 Read block 3691 len 12288
28600 027/300 Read block 1324 len 16384
28601 028/300 Write block 1424 len 20480 data: 9 7 10 7 4
28602 029/300 Read block 4717 len 8192
28603 030/300 Write block 4262 len 32768 data: 2 2 3 3 4 5 5 5
28604 031/300 Read block 4035 len 32768
28605 032/300 Write block 3787 len 32768 data: 5 4 4 3 3 3 2 3
28606 033/300 Write block 2509 len 28672 data: 7 6 5 5 4 5 6
28607 034/300 Read block 1029 len 4096
28608 035/300 Write block 2748 len 20480 data: 10 9 7 9 10
28609 036/300 Read block 1053 len 16384
28610 037/300 Read block 896 len 24576
28611 038/300 Read block 4623 len 8192
28612 039/300 Read block 103 len 28672
28613 040/300 Read block 1498 len 36864
28614 041/300 Write block 1559 len 12288 data: 4 5 4
28615 042/300 Write block 1240 len 32768 data: 4 4 4 4 4 6 6 5
28616 043/300 Read block 4376 len 40960
28617 044/300 Read block 1362 len 8192
28618 045/300 Write block 4116 len 36864 data: 5 5 5 5 7 9 10 9 7
28619 046/300 Read block 4126 len 36864
28620 047/300 Read block 1339 len 28672
28621 048/300 Read block 1827 len 24576
28622 049/300 Read block 684 len 16384
28623 050/300 Write block 4813 len 40960 data: 7 7 8 9 9 9 8 8 7 7
28624 051/300 Read block 601 len 40960
28625 052/300 Read block 3479 len 20480
28626 053/300 Write block 284 len 40960 data: 3 3 4 5 5 5 5 5 5 6
28627 054/300 Write block 4407 len 20480 data: 4 4 6 8 9
28628 055/300 Write block 1490 len 16384 data: 6 6 6 9
28629 056/300 Read block 4889 len 4096
28630 057/300 Read block 2886 len 4096
28631 058/300 Write block 2622 len 8192 data: 5 5
28632 059/300 Write block 1259 len 20480 data: 7 7 7 7 7
28633 060/300 Write block 3731 len 40960 data: 2 3 3 3 3 3 4 3 3 3
28634 061/300 Read block 2519 len 28672
28635 062/300 Read block 2884 len 20480
28636 063/300 Read block 4486 len 8192
28637 064/300 Read block 2091 len 28672
28638 065/300 Write block 4409 len 40960 data: 7 9 10 9 8 6 8 6 7 10
28639 066/300 Write block 2185 len 24576 data: 4 5 7 10 9 6
28640 067/300 Read block 1377 len 28672
28641 068/300 Write block 3506 len 40960 data: 6 8 11 12 11 11 11 8 8 7
28642 069/300 Write block 3309 len 12288 data: 4 3 4
28643 070/300 Read block 2852 len 20480
28644 071/300 Write block 479 len 40960 data: 5 4 4 3 5 5 4 3 3 3
28645 072/300 Write block 3321 len 32768 data: 6 5 5 6 6 5 3 3
28646 073/300 Read block 4897 len 40960
28647 074/300 Read block 2623 len 36864
28648 075/300 Read block 3505 len 16384
28649 076/300 Read block 3127 len 8192
28650 077/300 Write block 1826 len 8192 data: 7 6
28651 078/300 Read block 4913 len 36864
28652 079/300 Write block 3275 len 36864 data: 7 5 5 6 6 5 5 4 6
28653 080/300 Read block 245 len 36864
28654 081/300 Write block 2021 len 8192 data: 6 6
28655 082/300 Write block 209 len 24576 data: 6 5 6 6 6 7
28656 083/300 Flush
28657 084/300 Write block 419 len 28672 data: 3 3 3 2 2 3 5
28658 085/300 Read block 2685 len 40960
28659 086/300 Read block 2562 len 24576
28660 087/300 Write block 298 len 32768 data: 5 5 6 6 8 5 6 6
28661 088/300 Write block 3960 len 12288 data: 4 4 4
28662 089/300 Read block 1237 len 8192
28663 090/300 Read block 2151 len 28672
28664 091/300 Write block 4471 len 36864 data: 6 7 7 7 7 8 7 6 5
28665 092/300 Read block 3269 len 32768
28666 093/300 Read block 3223 len 32768
28667 094/300 Read block 896 len 4096
28668 095/300 Read block 660 len 32768
28669 096/300 Read block 3458 len 12288
28670 097/300 Flush
28671 098/300 Write block 3226 len 4096 data: 6
28672 099/300 Write block 4884 len 32768 data: 6 5 4 4 4 4 4 3
28673 100/300 Write block 893 len 8192 data: 6 5
28674 101/300 Read block 2438 len 4096
28675 102/300 Flush
28676 103/300 Read block 4943 len 28672
28677 104/300 Read block 784 len 28672
28678 105/300 Write block 3905 len 28672 data: 3 6 6 6 5 7 7
28679 106/300 Write block 2303 len 36864 data: 8 7 5 5 6 5 5 3 3
28680 107/300 Read block 1628 len 40960
28681 108/300 Read block 1979 len 28672
28682 109/300 Flush
28683 110/300 Read block 4810 len 20480
28684 111/300 Read block 740 len 36864
28685 112/300 Read block 1419 len 4096
28686 113/300 Write block 4197 len 36864 data: 4 4 4 3 4 5 5 5 5
28687 114/300 Flush
28688 115/300 Flush
28689 116/300 Write block 852 len 28672 data: 4 4 4 3 4 4 4
28690 117/300 Read block 1966 len 24576
28691 118/300 Write block 2299 len 16384 data: 4 4 5 6
28692 119/300 Write block 4447 len 32768 data: 5 5 6 6 6 7 6 5
28693 120/300 Read block 3358 len 4096
28694 121/300 Flush
28695 122/300 Write block 1261 len 40960 data: 8 8 8 6 5 3 2 2 2 3
28696 123/300 Read block 722 len 40960
28697 124/300 Read block 4258 len 24576
28698 125/300 Write block 3418 len 16384 data: 5 6 6 6
28699 126/300 Write block 2112 len 4096 data: 4
28700 127/300 Read block 300 len 28672
28701 128/300 Read block 3943 len 16384
28702 129/300 Read block 4124 len 40960
28703 130/300 Read block 3330 len 20480
28704 131/300 Write block 355 len 20480 data: 6 5 4 4 4
28705 132/300 Read block 3577 len 16384
28706 133/300 Read block 733 len 8192
28707 134/300 Flush
28708 135/300 Write block 4062 len 12288 data: 4 4 4
28709 136/300 Write block 4059 len 16384 data: 7 7 7 5
28710 137/300 Write block 4627 len 16384 data: 7 6 5 4
28711 138/300 Flush
28712 139/300 Read block 3764 len 24576
28713 140/300 Read block 2309 len 28672
28714 141/300 Write block 2502 len 40960 data: 4 4 4 6 5 6 6 8 7 6
28715 142/300 Flush
28716 143/300 Write block 353 len 28672 data: 6 7 7 6 5 5 5
28717 144/300 Read block 4772 len 24576
28718 145/300 Write block 2371 len 32768 data: 2 2 3 4 4 4 4 5
28719 146/300 Flush
28720 147/300 Write block 4124 len 24576 data: 8 7 6 5 4 4
28721 148/300 Write block 4105 len 36864 data: 6 4 6 7 7 6 7 8 5
28722 149/300 Read block 790 len 32768
28723 150/300 Write block 1438 len 32768 data: 6 8 8 7 5 6 6 5
28724 151/300 Read block 702 len 8192
28725 152/300 Write block 3781 len 8192 data: 4 4
28726 153/300 Write block 607 len 20480 data: 4 5 6 6 5
28727 154/300 Write block 3109 len 28672 data: 4 4 4 4 5 6 6
28728 155/300 Write block 4081 len 40960 data: 3 3 3 3 3 4 3 3 2 3
28729 156/300 Read block 1924 len 20480
28730 157/300 Read block 2026 len 8192
28731 158/300 Write block 3802 len 40960 data: 5 5 5 4 4 4 4 4 6 6
28732 159/300 Write block 101 len 16384 data: 7 7 7 6
28733 160/300 Flush
28734 161/300 Read block 444 len 28672
28735 162/300 Read block 410 len 32768
28736 163/300 Read block 1102 len 40960
28737 164/300 Read block 1650 len 4096
28738 165/300 Write block 1464 len 36864 data: 10 10 9 7 7 6 5 4 5
28739 166/300 Write block 1055 len 20480 data: 8 8 7 8 6
28740 167/300 Write block 4253 len 8192 data: 2 2
28741 168/300 Read block 4323 len 32768
28742 169/300 Write block 1678 len 12288 data: 5 6 6
28743 170/300 Write block 2557 len 4096 data: 6
28744 171/300 Write block 3948 len 40960 data: 7 5 6 4 5 7 5 6 6 6
28745 172/300 Write block 4950 len 20480 data: 5 6 6 4 4
28746 173/300 Read block 1309 len 16384
28747 174/300 Read block 3337 len 36864
28748 175/300 Write block 2892 len 12288 data: 5 4 4
28749 176/300 Write block 2333 len 40960 data: 5 4 5 6 7 6 6 7 7 7
28750 177/300 Read block 4502 len 24576
28751 178/300 Read block 3979 len 32768
28752 179/300 Read block 431 len 12288
28753 180/300 Read block 1504 len 8192
28754 181/300 Flush
28755 182/300 Flush
28756 183/300 Read block 125 len 32768
28757 184/300 Write block 1540 len 28672 data: 4 4 4 4 3 3 3
28758 185/300 Write block 3750 len 12288 data: 5 6 5
28759 186/300 Read block 1069 len 28672
28760 187/300 Read block 1911 len 24576
28761 188/300 Flush
28762 189/300 Read block 4355 len 32768
28763 190/300 Write block 3060 len 28672 data: 6 7 8 7 7 7 7
28764 191/300 Read block 2787 len 40960
28765 192/300 Read block 3384 len 24576
28766 193/300 Read block 4596 len 16384
28767 194/300 Write block 219 len 28672 data: 7 7 7 7 6 7 7
28768 195/300 Read block 2788 len 12288
28769 196/300 Write block 472 len 40960 data: 6 7 6 7 8 7 6 6 5 5
28770 197/300 Flush
28771 198/300 Read block 163 len 20480
28772 199/300 Write block 2759 len 36864 data: 6 6 6 7 6 6 6 4 4
28773 200/300 Write block 824 len 32768 data: 3 3 4 3 3 5 5 4
28774 201/300 Write block 1252 len 20480 data: 4 3 4 4 4
28775 202/300 Read block 1671 len 28672
28776 203/300 Read block 4970 len 16384
28777 204/300 Write block 3976 len 16384 data: 7 7 6 6
28778 205/300 Write block 1824 len 16384 data: 7 7 8 7
28779 206/300 Read block 3120 len 12288
28780 207/300 Read block 544 len 12288
28781 208/300 Write block 3580 len 4096 data: 5
28782 209/300 Read block 813 len 32768
28783 210/300 Read block 4825 len 12288
28784 211/300 Write block 4037 len 12288 data: 9 8 8
28785 212/300 Read block 2024 len 40960
28786 213/300 Flush
28787 214/300 Read block 187 len 4096
28788 215/300 Read block 4718 len 24576
28789 216/300 Flush
28790 217/300 Read block 802 len 4096
28791 218/300 Read block 4882 len 16384
28792 219/300 Write block 3978 len 28672 data: 7 7 5 5 6 5 6
28793 220/300 Write block 4697 len 24576 data: 3 6 6 5 5 7
28794 221/300 Read block 299 len 36864
28795 222/300 Write block 422 len 20480 data: 3 3 4 6 6
28796 223/300 Write block 1121 len 4096 data: 6
28797 224/300 Write block 2276 len 36864 data: 7 9 9 8 8 8 10 9 6
28798 225/300 Write block 4692 len 4096 data: 4
28799 226/300 Read block 2229 len 16384
28800 227/300 Write block 51 len 36864 data: 3 3 4 3 4 4 4 6 6
28801 228/300 Read block 3847 len 20480
28802 229/300 Write block 3805 len 20480 data: 5 5 5 5 5
28803 230/300 Write block 1641 len 20480 data: 3 5 6 6 6
28804 231/300 Read block 2667 len 8192
28805 232/300 Write block 1002 len 12288 data: 4 4 5
28806 233/300 Write block 2344 len 8192 data: 7 7
28807 234/300 Read block 2229 len 40960
28808 235/300 Write block 4345 len 28672 data: 4 4 4 5 6 6 7
28809 236/300 Flush
28810 237/300 Write block 92 len 16384 data: 6 5 5 6
28811 238/300 Write block 2489 len 40960 data: 6 5 5 4 4 4 6 5 4 3
28812 239/300 Write block 1974 len 16384 data: 5 4 4 4
28813 240/300 Read block 3507 len 28672
28814 241/300 Read block 1670 len 24576
28815 242/300 Read block 231 len 24576
28816 243/300 Write block 2036 len 12288 data: 10 7 9
28817 244/300 Read block 339 len 8192
28818 245/300 Read block 4829 len 16384
28819 246/300 Write block 4014 len 16384 data: 7 8 7 8
28820 247/300 Flush
28821 248/300 Read block 4279 len 24576
28822 249/300 Read block 2261 len 40960
28823 250/300 Read block 3779 len 36864
28824 251/300 Write block 4772 len 28672 data: 3 4 4 4 5 5 5
28825 252/300 Write block 4773 len 8192 data: 5 5
28826 253/300 Write block 3974 len 36864 data: 5 6 8 8 8 8 6 6 7
28827 254/300 Read block 4863 len 4096
28828 255/300 Write block 2363 len 40960 data: 8 7 7 5 4 3 2 2 3 3
28829 256/300 Read block 1784 len 16384
28830 257/300 Write block 3420 len 32768 data: 7 7 5 6 5 6 6 6
28831 258/300 Write block 4956 len 8192 data: 2 2
28832 259/300 Write block 1502 len 40960 data: 7 6 5 5 6 5 5 4 3 3
28833 260/300 Read block 476 len 24576
28834 261/300 Read block 3066 len 12288
28835 262/300 Write block 3255 len 20480 data: 4 4 4 4 4
28836 263/300 Flush
28837 264/300 Read block 719 len 8192
28838 265/300 Write block 4958 len 4096 data: 3
28839 266/300 Read block 2549 len 28672
28840 267/300 Read block 209 len 12288
28841 268/300 Flush
28842 269/300 Write block 2965 len 12288 data: 5 4 4
28843 270/300 Write block 3698 len 40960 data: 4 5 6 6 7 7 6 6 6 5
28844 271/300 Read block 4357 len 16384
28845 272/300 Write block 258 len 20480 data: 5 6 6 6 4
28846 273/300 Write block 2674 len 32768 data: 5 5 6 6 9 9 9 9
28847 274/300 Read block 1749 len 20480
28848 275/300 Read block 834 len 36864
28849 276/300 Write block 450 len 16384 data: 4 3 2 3
28850 277/300 Write block 2 len 32768 data: 3 3 3 2 2 2 2 2
28851 278/300 Write block 4004 len 20480 data: 8 8 9 9 10
28852 279/300 Read block 167 len 4096
28853 280/300 Read block 3791 len 24576
28854 281/300 Write block 2325 len 28672 data: 7 9 6 6 6 5 4
28855 282/300 Read block 1922 len 28672
28856 283/300 Write block 1912 len 20480 data: 7 7 7 4 5
28857 284/300 Write block 4863 len 40960 data: 5 3 4 4 4 2 4 4 3 5
28858 285/300 Read block 1992 len 20480
28859 286/300 Read block 986 len 40960
28860 287/300 Read block 4285 len 40960
28861 288/300 Write block 4603 len 36864 data: 5 5 5 5 5 7 6 7 5
28862 289/300 Read block 2165 len 28672
28863 290/300 Read block 384 len 16384
28864 291/300 Read block 1681 len 12288
28865 292/300 Write block 3334 len 36864 data: 4 5 5 7 8 7 9 10 9
28866 293/300 Read block 156 len 40960
28867 294/300 Read block 1877 len 40960
28868 295/300 Read block 2028 len 16384
28869 296/300 Read block 2221 len 12288
28870 297/300 Write block 3892 len 4096 data: 3
28871 298/300 Write block 2326 len 32768 data: 10 7 7 7 6 5 4 6
28872 299/300 Read block 3567 len 16384
28873 300/300 Write block 989 len 12288 data: 4 4 4
28874 301/300 Flush
28875 Main task has work to do, go find it
28876 got dsc Start(2)
28877 start 2
28878 Feb 21 06:51:45.822 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/2, method: POST, req_id: e560b873-c865-4566-a966-40cf8ed8af04, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
28879 Replay: started 2, returned:Ok(())
28880 ----------------------------------------------------------------
28881 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28882 States: Active Active Offline
28883 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28884 8547 Acked Write 3 Done Done Sent false
28885 8548 Acked Write 5 Done Done Sent false
28886 8549 Acked Write 2 Done Done Sent false
28887 8550 Acked Flush 0 Done Done Sent false
28888 8551 Acked Read 10 Done Done Sent false
28889 8552 Acked Read 9 Done Done Sent false
28890 8553 Acked Read 3 Done Done Sent false
28891 8554 Acked Write 3 Done Done Sent false
28892 8555 Acked Flush 0 Done Done Sent false
28893 8556 Acked Write 9 Done Done Sent false
28894 8557 Acked Read 7 Done Done Sent false
28895 8558 Acked Read 10 Done Done Sent false
28896 8559 Acked Write 10 Done Done Sent false
28897 8560 Acked Read 7 Done Done Sent false
28898 8561 Acked Read 10 Done Done Sent false
28899 8562 Acked Read 7 Done Done Sent false
28900 8563 Acked Write 1 Done Done Sent false
28901 8564 Acked Read 5 Done Done Sent false
28902 8565 Acked Write 7 Done Done Sent false
28903 8566 Acked Read 3 Done Done Sent false
28904 8567 Acked Flush 0 Done Done Sent false
28905 8568 Acked Write 4 Done Done Sent false
28906 8569 Acked Read 7 Done Done Sent false
28907 8570 Acked Write 8 Done Done Sent false
28908 8571 Acked Write 7 Done Done Sent false
28909 8572 Acked Read 3 Done Done Sent false
28910 8573 Acked Read 4 Done Done Sent false
28911 8574 Acked Write 5 Done Done Sent false
28912 8575 Acked Read 2 Done Done Sent false
28913 8576 Acked Write 8 Done Done Sent false
28914 8577 Acked Read 8 Done Done Sent false
28915 8578 Acked Write 8 Done Done Sent false
28916 8579 Acked Write 7 Done Done Sent false
28917 8580 Acked Read 1 Done Done Sent false
28918 8581 Acked Write 5 Done Done Sent false
28919 8582 Acked Read 4 Done Done Sent false
28920 8583 Acked Read 6 Done Done Sent false
28921 8584 Acked Read 2 Done Done Sent false
28922 8585 Acked Read 7 Done Done Sent false
28923 8586 Acked Read 9 Done Done Sent false
28924 8587 Acked Write 3 Done Done Sent false
28925 8588 Acked Write 8 Done Done Sent false
28926 8589 Acked Read 10 Done Done Sent false
28927 8590 Acked Read 2 Done Done Sent false
28928 8591 Acked Write 9 Done Done Sent false
28929 8592 Acked Read 9 Done Done Sent false
28930 8593 Acked Read 7 Done Done Sent false
28931 8594 Acked Read 6 Done Done Sent false
28932 8595 Acked Read 4 Done Done Sent false
28933 8596 Acked Write 10 Done Done Sent false
28934 8597 Acked Read 10 Done Done Sent false
28935 8598 Acked Read 5 Done Done Sent false
28936 8599 Acked Write 10 Done Done Sent false
28937 8600 Acked Write 5 Done Done Sent false
28938 8601 Acked Write 4 Done Done Sent false
28939 8602 Acked Read 1 Done Done Sent false
28940 8603 Acked Read 1 Done Done Sent false
28941 8604 Acked Write 2 Done Done Sent false
28942 8605 Acked Write 5 Done Done Sent false
28943 8606 Acked Write 10 Done Done Sent false
28944 8607 Acked Read 7 Done Done Sent false
28945 8608 Acked Read 5 Done Done Sent false
28946 8609 Acked Read 2 Done Done Sent false
28947 8610 Acked Read 7 Done Done Sent false
28948 8611 Acked Write 10 Done Done Sent false
28949 8612 Acked Write 6 Done Done Sent false
28950 8613 Acked Read 7 Done Done Sent false
28951 8614 Acked Write 10 Done Done Sent false
28952 8615 Acked Write 3 Done Done Sent false
28953 8616 Acked Read 5 Done Done Sent false
28954 8617 Acked Write 10 Done Done Sent false
28955 8618 Acked Write 8 Done Done Sent false
28956 8619 Acked Read 10 Done Done Sent false
28957 8620 Acked Read 9 Done Done Sent false
28958 8621 Acked Read 4 Done Done Sent false
28959 8622 Acked Read 2 Done Done Sent false
28960 8623 Acked Write 2 Done Done Sent false
28961 8624 Acked Read 9 Done Done Sent false
28962 8625 Acked Write 9 Done Done Sent false
28963 8626 Acked Read 9 Done Done Sent false
28964 8627 Acked Write 2 Done Done Sent false
28965 8628 Acked Write 6 Done Done Sent false
28966 8629 Acked Flush 0 Done Done Sent false
28967 8630 Acked Write 7 Done Done Sent false
28968 8631 Acked Read 10 Done Done Sent false
28969 8632 Acked Read 6 Done Done Sent false
28970 8633 Acked Write 8 Done Done Sent false
28971 8634 Acked Write 3 Done Done Sent false
28972 8635 Acked Read 2 Done Done Sent false
28973 8636 Acked Read 7 Done Done Sent false
28974 8637 Acked Write 9 Done Done Sent false
28975 8638 Acked Read 8 Done Done Sent false
28976 8639 Acked Read 8 Done Done Sent false
28977 8640 Acked Read 1 Done Done Sent false
28978 8641 Acked Read 8 Done Done Sent false
28979 8642 Acked Read 3 Done Done Sent false
28980 8643 Acked Flush 0 Done Done Sent false
28981 8644 Acked Write 1 Done Done Sent false
28982 8645 Acked Write 8 Done Done Sent false
28983 8646 Acked Write 2 Done Done Sent false
28984 8647 Acked Read 1 Done Done Sent false
28985 8648 Acked Flush 0 Done Done Sent false
28986 8649 Acked Read 7 Done Done Sent false
28987 8650 Acked Read 7 Done Done Sent false
28988 8651 Acked Write 7 Done Done Sent false
28989 8652 Acked Write 9 Done Done Sent false
28990 8653 Acked Read 10 Done Done Sent false
28991 8654 Acked Read 7 Done Done Sent false
28992 8655 Acked Flush 0 Done Done Sent false
28993 8656 Acked Read 5 Done Done Sent false
28994 8657 Acked Read 9 Done Done Sent false
28995 8658 Acked Read 1 Done Done Sent false
28996 8659 Acked Write 9 Done Done Sent false
28997 8660 Acked Flush 0 Done Done Sent false
28998 8661 Acked Flush 0 Done Done Sent false
28999 8662 Acked Write 7 Done Done Sent false
29000 8663 Acked Read 6 Done Done Sent false
29001 8664 Acked Write 4 Done Done Sent false
29002 8665 Acked Write 8 Done Done Sent false
29003 8666 Acked Read 1 Done Done Sent false
29004 8667 Acked Flush 0 Done Done Sent false
29005 8668 Acked Write 10 Done Done Sent false
29006 8669 Acked Read 10 Done Done Sent false
29007 8670 Acked Read 6 Done Done Sent false
29008 8671 Acked Write 4 Done Done Sent false
29009 8672 Acked Write 1 Done Done Sent false
29010 8673 Acked Read 7 Done Done Sent false
29011 8674 Acked Read 4 Done Done Sent false
29012 8675 Acked Read 10 Done Done Sent false
29013 8676 Acked Read 5 Done Done Sent false
29014 8677 Acked Write 5 Done Done Sent false
29015 8678 Acked Read 4 Done Done Sent false
29016 8679 Acked Read 2 Done Done Sent false
29017 8680 Acked Flush 0 Done Done Sent false
29018 8681 Acked Write 3 Done Done Sent false
29019 8682 Acked Write 4 Done Done Sent false
29020 8683 Acked Write 4 Done Done Sent false
29021 8684 Acked Flush 0 Done Done Sent false
29022 8685 Acked Read 6 Done Done Sent false
29023 8686 Acked Read 7 Done Done Sent false
29024 8687 Acked Write 10 Done Done Sent false
29025 8688 Acked Flush 0 Done Done Sent false
29026 8689 Acked Write 7 Done Done Sent false
29027 8690 Acked Read 6 Done Done Sent false
29028 8691 Acked Write 8 Done Done Sent false
29029 8692 Acked Flush 0 Done Done Sent false
29030 8693 Acked Write 6 Done Done Sent false
29031 8694 Acked Write 9 Done Done Sent false
29032 8695 Acked Read 8 Done Done Sent false
29033 8696 Acked Write 8 Done Done Sent false
29034 8697 Acked Read 2 Done Done Sent false
29035 8698 Acked Write 2 Done Done Sent false
29036 8699 Acked Write 5 Done Done Sent false
29037 8700 Acked Write 7 Done Done Sent false
29038 8701 Acked Write 10 Done Done Sent false
29039 8702 Acked Read 5 Done Done Sent false
29040 8703 Acked Read 2 Done Done Sent false
29041 8704 Acked Write 10 Done Done Sent false
29042 8705 Acked Write 4 Done Done Sent false
29043 8706 Acked Flush 0 Done Done Sent false
29044 8707 Acked Read 7 Done Done Sent false
29045 8708 Acked Read 8 Done Done Sent false
29046 8709 Acked Read 10 Done Done Sent false
29047 8710 Acked Read 1 Done Done Sent false
29048 8711 Acked Write 9 Done Done Sent false
29049 8712 Acked Write 5 Done Done Sent false
29050 8713 Acked Write 2 Done Done Sent false
29051 8714 Acked Read 8 Done Done Sent false
29052 8715 Acked Write 3 Done Done Sent false
29053 8716 Acked Write 1 Done Done Sent false
29054 8717 Acked Write 10 Done Done Sent false
29055 8718 Acked Write 5 Done Done Sent false
29056 8719 Acked Read 4 Done Done Sent false
29057 8720 Acked Read 9 Done Done Sent false
29058 8721 Acked Write 3 Done Done Sent false
29059 8722 Acked Write 10 Done Done Sent false
29060 8723 Acked Read 6 Done Done Sent false
29061 8724 Acked Read 8 Done Done Sent false
29062 8725 Acked Read 3 Done Done Sent false
29063 8726 Acked Read 2 Done Done Sent false
29064 8727 Acked Flush 0 Done Done Sent false
29065 8728 Acked Flush 0 Done Done Sent false
29066 8729 Acked Read 8 Done Done Sent false
29067 8730 Acked Write 7 Done Done Sent false
29068 8731 Acked Write 3 Done Done Sent false
29069 8732 Acked Read 7 Done Done Sent false
29070 8733 Acked Read 6 Done Done Sent false
29071 8734 Acked Flush 0 Done Done Sent false
29072 8735 Acked Read 8 Done Done Sent false
29073 8736 Acked Write 7 Done Done Sent false
29074 8737 Acked Read 10 Done Done Sent false
29075 8738 Acked Read 6 Done Done Sent false
29076 8739 Acked Read 4 Done Done Sent false
29077 8740 Acked Write 7 Done Done Sent false
29078 8741 Acked Read 3 Done Done Sent false
29079 8742 Acked Write 10 Done Done Sent false
29080 8743 Acked Flush 0 Done Done Sent false
29081 8744 Acked Read 5 Done Done Sent false
29082 8745 Acked Write 9 Done Done Sent false
29083 8746 Acked Write 8 Done Done Sent false
29084 8747 Acked Write 5 Done Done Sent false
29085 8748 Acked Read 7 Done Done Sent false
29086 8749 Acked Read 4 Done Done Sent false
29087 8750 Acked Write 4 Done Done Sent false
29088 8751 Acked Write 4 Done Done Sent false
29089 8752 Acked Read 3 Done Done Sent false
29090 8753 Acked Read 3 Done Done Sent false
29091 8754 Acked Write 1 Done Done Sent false
29092 8755 Acked Read 8 Done Done Sent false
29093 8756 Acked Read 3 Done Done Sent false
29094 8757 Acked Write 3 Done Done Sent false
29095 8758 Acked Read 10 Done Done Sent false
29096 8759 Acked Flush 0 Done Done Sent false
29097 8760 Acked Read 1 Done Done Sent false
29098 8761 Acked Read 6 Done Done Sent false
29099 8762 Acked Flush 0 Done Done Sent false
29100 8763 Acked Read 1 Done Done Sent false
29101 8764 Acked Read 4 Done Done Sent false
29102 8765 Acked Write 7 Done Done Sent false
29103 8766 Acked Write 6 Done Done Sent false
29104 8767 Acked Read 9 Done Done Sent false
29105 8768 Acked Write 5 Done Done Sent false
29106 8769 Acked Write 1 Done Done Sent false
29107 8770 Acked Write 9 Done Done Sent false
29108 8771 Acked Write 1 Done Done Sent false
29109 8772 Acked Read 4 Done Done Sent false
29110 8773 Acked Write 9 Done Done Sent false
29111 8774 Acked Read 5 Done Done Sent false
29112 8775 Acked Write 5 Done Done Sent false
29113 8776 Acked Write 5 Done Done Sent false
29114 8777 Acked Read 2 Done Done Sent false
29115 8778 Acked Write 3 Done Done Sent false
29116 8779 Acked Write 2 Done Done Sent false
29117 8780 Acked Read 10 Done Done Sent false
29118 8781 Acked Write 7 Done Done Sent false
29119 8782 Acked Flush 0 Done Done Sent false
29120 8783 Acked Write 4 Done Done Sent false
29121 8784 Acked Write 10 Done Done Sent false
29122 8785 Acked Write 4 Done Done Sent false
29123 8786 Acked Read 7 Done Done Sent false
29124 8787 Acked Read 6 Done Done Sent false
29125 8788 Acked Read 6 Done Done Sent false
29126 8789 Acked Write 3 Done Done Sent false
29127 8790 Acked Read 2 Done Done Sent false
29128 8791 Acked Read 4 Done Done Sent false
29129 8792 Acked Write 4 Done Done Sent false
29130 8793 Acked Flush 0 Done Done Sent false
29131 8794 Acked Read 6 Done Done Sent false
29132 8795 Acked Read 10 Done Done Sent false
29133 8796 Acked Read 9 Done Done Sent false
29134 8797 Acked Write 7 Done Done Sent false
29135 8798 Acked Write 2 Done Done Sent false
29136 8799 Acked Write 9 Done Done Sent false
29137 8800 Acked Read 1 Done Done Sent false
29138 8801 Acked Write 10 Done Done Sent false
29139 8802 Acked Read 4 Done Done Sent false
29140 8803 Acked Write 8 Done Done Sent false
29141 8804 Acked Write 2 Done Done Sent false
29142 8805 Acked Write 10 Done Done Sent false
29143 8806 Acked Read 6 Done Done Sent false
29144 8807 Acked Read 3 Done Done Sent false
29145 8808 Acked Write 5 Done Done Sent false
29146 8809 Acked Flush 0 Done Done Sent false
29147 8810 Acked Read 2 Done Done Sent false
29148 8811 Acked Write 1 Done Done Sent false
29149 8812 Acked Read 7 Done Done Sent false
29150 8813 Acked Read 3 Done Done Sent false
29151 8814 Acked Flush 0 Done Done Sent false
29152 8815 Acked Write 3 Done Done Sent false
29153 8816 Acked Write 10 Done Done Sent false
29154 8817 Acked Read 4 Done Done Sent false
29155 8818 Acked Write 5 Done Done Sent false
29156 8819 Acked Write 8 Done Done Sent false
29157 8820 Acked Read 5 Done Done Sent false
29158 8821 Acked Read 9 Done Done Sent false
29159 8822 Acked Write 4 Done Done Sent false
29160 8823 Acked Write 8 Done Done Sent false
29161 8824 Acked Write 5 Done Done Sent false
29162 8825 Acked Read 1 Done Done Sent false
29163 8826 Acked Read 6 Done Done Sent false
29164 8827 Acked Write 7 Done Done Sent false
29165 8828 Acked Read 7 Done Done Sent false
29166 8829 Acked Write 5 Done Done Sent false
29167 8830 Acked Write 10 Done Done Sent false
29168 8831 Acked Read 5 Done Done Sent false
29169 8832 Acked Read 10 Done Done Sent false
29170 8833 Acked Read 10 Done Done Sent false
29171 8834 Acked Write 9 Done Done Sent false
29172 8835 Acked Read 7 Done Done Sent false
29173 8836 Acked Read 4 Done Done Sent false
29174 8837 Acked Read 3 Done Done Sent false
29175 8838 Acked Write 9 Done Done Sent false
29176 8839 Acked Read 10 Done Done Sent false
29177 8840 Acked Read 10 Done Done Sent false
29178 8841 Acked Read 4 Done Done Sent false
29179 8842 Acked Read 3 Done Done Sent false
29180 8843 Acked Write 1 Done Done Sent false
29181 8844 Acked Write 8 Done Done Sent false
29182 8845 Acked Read 4 Done Done Sent false
29183 8846 Acked Write 3 Done Done Sent false
29184 8847 Acked Flush 0 Done Done Sent false
29185 STATES DS:0 DS:1 DS:2 TOTAL
29186 Sent 0 0 301 301
29187 Done 301 301 0 602
29188 Skipped 0 0 0 0
29189 Error 0 0 0 0
29190 Last Flush: Some(JobId(8847)) Some(JobId(8847)) Some(JobId(8546))
29191 Downstairs last five completed: 8546 8545 8544 8543 8542
29192 Upstairs last five completed: 8847 8846 8845 8844 8843
29193 CLIENT: Up:0 ds:301 act:2
291942025-02-21T06:51:46.324ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
291952025-02-21T06:51:47.248ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
291962025-02-21T06:51:48.249ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
291972025-02-21T06:51:49.250ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
29198 ----------------------------------------------------------------
29199 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29200 States: Active Active Offline
29201 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29202 8547 Acked Write 3 Done Done Sent false
29203 8548 Acked Write 5 Done Done Sent false
29204 8549 Acked Write 2 Done Done Sent false
29205 8550 Acked Flush 0 Done Done Sent false
29206 8551 Acked Read 10 Done Done Sent false
29207 8552 Acked Read 9 Done Done Sent false
29208 8553 Acked Read 3 Done Done Sent false
29209 8554 Acked Write 3 Done Done Sent false
29210 8555 Acked Flush 0 Done Done Sent false
29211 8556 Acked Write 9 Done Done Sent false
29212 8557 Acked Read 7 Done Done Sent false
29213 8558 Acked Read 10 Done Done Sent false
29214 8559 Acked Write 10 Done Done Sent false
29215 8560 Acked Read 7 Done Done Sent false
29216 8561 Acked Read 10 Done Done Sent false
29217 8562 Acked Read 7 Done Done Sent false
29218 8563 Acked Write 1 Done Done Sent false
29219 8564 Acked Read 5 Done Done Sent false
29220 8565 Acked Write 7 Done Done Sent false
29221 8566 Acked Read 3 Done Done Sent false
29222 8567 Acked Flush 0 Done Done Sent false
29223 8568 Acked Write 4 Done Done Sent false
29224 8569 Acked Read 7 Done Done Sent false
29225 8570 Acked Write 8 Done Done Sent false
29226 8571 Acked Write 7 Done Done Sent false
29227 8572 Acked Read 3 Done Done Sent false
29228 8573 Acked Read 4 Done Done Sent false
29229 8574 Acked Write 5 Done Done Sent false
29230 8575 Acked Read 2 Done Done Sent false
29231 8576 Acked Write 8 Done Done Sent false
29232 8577 Acked Read 8 Done Done Sent false
29233 8578 Acked Write 8 Done Done Sent false
29234 8579 Acked Write 7 Done Done Sent false
29235 8580 Acked Read 1 Done Done Sent false
29236 8581 Acked Write 5 Done Done Sent false
29237 8582 Acked Read 4 Done Done Sent false
29238 8583 Acked Read 6 Done Done Sent false
29239 8584 Acked Read 2 Done Done Sent false
29240 8585 Acked Read 7 Done Done Sent false
29241 8586 Acked Read 9 Done Done Sent false
29242 8587 Acked Write 3 Done Done Sent false
29243 8588 Acked Write 8 Done Done Sent false
29244 8589 Acked Read 10 Done Done Sent false
29245 8590 Acked Read 2 Done Done Sent false
29246 8591 Acked Write 9 Done Done Sent false
29247 8592 Acked Read 9 Done Done Sent false
29248 8593 Acked Read 7 Done Done Sent false
29249 8594 Acked Read 6 Done Done Sent false
29250 8595 Acked Read 4 Done Done Sent false
29251 8596 Acked Write 10 Done Done Sent false
29252 8597 Acked Read 10 Done Done Sent false
29253 8598 Acked Read 5 Done Done Sent false
29254 8599 Acked Write 10 Done Done Sent false
29255 8600 Acked Write 5 Done Done Sent false
29256 8601 Acked Write 4 Done Done Sent false
29257 8602 Acked Read 1 Done Done Sent false
29258 8603 Acked Read 1 Done Done Sent false
29259 8604 Acked Write 2 Done Done Sent false
29260 8605 Acked Write 5 Done Done Sent false
29261 8606 Acked Write 10 Done Done Sent false
29262 8607 Acked Read 7 Done Done Sent false
29263 8608 Acked Read 5 Done Done Sent false
29264 8609 Acked Read 2 Done Done Sent false
29265 8610 Acked Read 7 Done Done Sent false
29266 8611 Acked Write 10 Done Done Sent false
29267 8612 Acked Write 6 Done Done Sent false
29268 8613 Acked Read 7 Done Done Sent false
29269 8614 Acked Write 10 Done Done Sent false
29270 8615 Acked Write 3 Done Done Sent false
29271 8616 Acked Read 5 Done Done Sent false
29272 8617 Acked Write 10 Done Done Sent false
29273 8618 Acked Write 8 Done Done Sent false
29274 8619 Acked Read 10 Done Done Sent false
29275 8620 Acked Read 9 Done Done Sent false
29276 8621 Acked Read 4 Done Done Sent false
29277 8622 Acked Read 2 Done Done Sent false
29278 8623 Acked Write 2 Done Done Sent false
29279 8624 Acked Read 9 Done Done Sent false
29280 8625 Acked Write 9 Done Done Sent false
29281 8626 Acked Read 9 Done Done Sent false
29282 8627 Acked Write 2 Done Done Sent false
29283 8628 Acked Write 6 Done Done Sent false
29284 8629 Acked Flush 0 Done Done Sent false
29285 8630 Acked Write 7 Done Done Sent false
29286 8631 Acked Read 10 Done Done Sent false
29287 8632 Acked Read 6 Done Done Sent false
29288 8633 Acked Write 8 Done Done Sent false
29289 8634 Acked Write 3 Done Done Sent false
29290 8635 Acked Read 2 Done Done Sent false
29291 8636 Acked Read 7 Done Done Sent false
29292 8637 Acked Write 9 Done Done Sent false
29293 8638 Acked Read 8 Done Done Sent false
29294 8639 Acked Read 8 Done Done Sent false
29295 8640 Acked Read 1 Done Done Sent false
29296 8641 Acked Read 8 Done Done Sent false
29297 8642 Acked Read 3 Done Done Sent false
29298 8643 Acked Flush 0 Done Done Sent false
29299 8644 Acked Write 1 Done Done Sent false
29300 8645 Acked Write 8 Done Done Sent false
29301 8646 Acked Write 2 Done Done Sent false
29302 8647 Acked Read 1 Done Done Sent false
29303 8648 Acked Flush 0 Done Done Sent false
29304 8649 Acked Read 7 Done Done Sent false
29305 8650 Acked Read 7 Done Done Sent false
29306 8651 Acked Write 7 Done Done Sent false
29307 8652 Acked Write 9 Done Done Sent false
29308 8653 Acked Read 10 Done Done Sent false
29309 8654 Acked Read 7 Done Done Sent false
29310 8655 Acked Flush 0 Done Done Sent false
29311 8656 Acked Read 5 Done Done Sent false
29312 8657 Acked Read 9 Done Done Sent false
29313 8658 Acked Read 1 Done Done Sent false
29314 8659 Acked Write 9 Done Done Sent false
29315 8660 Acked Flush 0 Done Done Sent false
29316 8661 Acked Flush 0 Done Done Sent false
29317 8662 Acked Write 7 Done Done Sent false
29318 8663 Acked Read 6 Done Done Sent false
29319 8664 Acked Write 4 Done Done Sent false
29320 8665 Acked Write 8 Done Done Sent false
29321 8666 Acked Read 1 Done Done Sent false
29322 8667 Acked Flush 0 Done Done Sent false
29323 8668 Acked Write 10 Done Done Sent false
29324 8669 Acked Read 10 Done Done Sent false
29325 8670 Acked Read 6 Done Done Sent false
29326 8671 Acked Write 4 Done Done Sent false
29327 8672 Acked Write 1 Done Done Sent false
29328 8673 Acked Read 7 Done Done Sent false
29329 8674 Acked Read 4 Done Done Sent false
29330 8675 Acked Read 10 Done Done Sent false
29331 8676 Acked Read 5 Done Done Sent false
29332 8677 Acked Write 5 Done Done Sent false
29333 8678 Acked Read 4 Done Done Sent false
29334 8679 Acked Read 2 Done Done Sent false
29335 8680 Acked Flush 0 Done Done Sent false
29336 8681 Acked Write 3 Done Done Sent false
29337 8682 Acked Write 4 Done Done Sent false
29338 8683 Acked Write 4 Done Done Sent false
29339 8684 Acked Flush 0 Done Done Sent false
29340 8685 Acked Read 6 Done Done Sent false
29341 8686 Acked Read 7 Done Done Sent false
29342 8687 Acked Write 10 Done Done Sent false
29343 8688 Acked Flush 0 Done Done Sent false
29344 8689 Acked Write 7 Done Done Sent false
29345 8690 Acked Read 6 Done Done Sent false
29346 8691 Acked Write 8 Done Done Sent false
29347 8692 Acked Flush 0 Done Done Sent false
29348 8693 Acked Write 6 Done Done Sent false
29349 8694 Acked Write 9 Done Done Sent false
29350 8695 Acked Read 8 Done Done Sent false
29351 8696 Acked Write 8 Done Done Sent false
29352 8697 Acked Read 2 Done Done Sent false
29353 8698 Acked Write 2 Done Done Sent false
29354 8699 Acked Write 5 Done Done Sent false
29355 8700 Acked Write 7 Done Done Sent false
29356 8701 Acked Write 10 Done Done Sent false
29357 8702 Acked Read 5 Done Done Sent false
29358 8703 Acked Read 2 Done Done Sent false
29359 8704 Acked Write 10 Done Done Sent false
29360 8705 Acked Write 4 Done Done Sent false
29361 8706 Acked Flush 0 Done Done Sent false
29362 8707 Acked Read 7 Done Done Sent false
29363 8708 Acked Read 8 Done Done Sent false
29364 8709 Acked Read 10 Done Done Sent false
29365 8710 Acked Read 1 Done Done Sent false
29366 8711 Acked Write 9 Done Done Sent false
29367 8712 Acked Write 5 Done Done Sent false
29368 8713 Acked Write 2 Done Done Sent false
29369 8714 Acked Read 8 Done Done Sent false
29370 8715 Acked Write 3 Done Done Sent false
29371 8716 Acked Write 1 Done Done Sent false
29372 8717 Acked Write 10 Done Done Sent false
29373 8718 Acked Write 5 Done Done Sent false
29374 8719 Acked Read 4 Done Done Sent false
29375 8720 Acked Read 9 Done Done Sent false
29376 8721 Acked Write 3 Done Done Sent false
29377 8722 Acked Write 10 Done Done Sent false
29378 8723 Acked Read 6 Done Done Sent false
29379 8724 Acked Read 8 Done Done Sent false
29380 8725 Acked Read 3 Done Done Sent false
29381 8726 Acked Read 2 Done Done Sent false
29382 8727 Acked Flush 0 Done Done Sent false
29383 8728 Acked Flush 0 Done Done Sent false
29384 8729 Acked Read 8 Done Done Sent false
29385 8730 Acked Write 7 Done Done Sent false
29386 8731 Acked Write 3 Done Done Sent false
29387 8732 Acked Read 7 Done Done Sent false
29388 8733 Acked Read 6 Done Done Sent false
29389 8734 Acked Flush 0 Done Done Sent false
29390 8735 Acked Read 8 Done Done Sent false
29391 8736 Acked Write 7 Done Done Sent false
29392 8737 Acked Read 10 Done Done Sent false
29393 8738 Acked Read 6 Done Done Sent false
29394 8739 Acked Read 4 Done Done Sent false
29395 8740 Acked Write 7 Done Done Sent false
29396 8741 Acked Read 3 Done Done Sent false
29397 8742 Acked Write 10 Done Done Sent false
29398 8743 Acked Flush 0 Done Done Sent false
29399 8744 Acked Read 5 Done Done Sent false
29400 8745 Acked Write 9 Done Done Sent false
29401 8746 Acked Write 8 Done Done Sent false
29402 8747 Acked Write 5 Done Done Sent false
29403 8748 Acked Read 7 Done Done Sent false
29404 8749 Acked Read 4 Done Done Sent false
29405 8750 Acked Write 4 Done Done Sent false
29406 8751 Acked Write 4 Done Done Sent false
29407 8752 Acked Read 3 Done Done Sent false
29408 8753 Acked Read 3 Done Done Sent false
29409 8754 Acked Write 1 Done Done Sent false
29410 8755 Acked Read 8 Done Done Sent false
29411 8756 Acked Read 3 Done Done Sent false
29412 8757 Acked Write 3 Done Done Sent false
29413 8758 Acked Read 10 Done Done Sent false
29414 8759 Acked Flush 0 Done Done Sent false
29415 8760 Acked Read 1 Done Done Sent false
29416 8761 Acked Read 6 Done Done Sent false
29417 8762 Acked Flush 0 Done Done Sent false
29418 8763 Acked Read 1 Done Done Sent false
29419 8764 Acked Read 4 Done Done Sent false
29420 8765 Acked Write 7 Done Done Sent false
29421 8766 Acked Write 6 Done Done Sent false
29422 8767 Acked Read 9 Done Done Sent false
29423 8768 Acked Write 5 Done Done Sent false
29424 8769 Acked Write 1 Done Done Sent false
29425 8770 Acked Write 9 Done Done Sent false
29426 8771 Acked Write 1 Done Done Sent false
29427 8772 Acked Read 4 Done Done Sent false
29428 8773 Acked Write 9 Done Done Sent false
29429 8774 Acked Read 5 Done Done Sent false
29430 8775 Acked Write 5 Done Done Sent false
29431 8776 Acked Write 5 Done Done Sent false
29432 8777 Acked Read 2 Done Done Sent false
29433 8778 Acked Write 3 Done Done Sent false
29434 8779 Acked Write 2 Done Done Sent false
29435 8780 Acked Read 10 Done Done Sent false
29436 8781 Acked Write 7 Done Done Sent false
29437 8782 Acked Flush 0 Done Done Sent false
29438 8783 Acked Write 4 Done Done Sent false
29439 8784 Acked Write 10 Done Done Sent false
29440 8785 Acked Write 4 Done Done Sent false
29441 8786 Acked Read 7 Done Done Sent false
29442 8787 Acked Read 6 Done Done Sent false
29443 8788 Acked Read 6 Done Done Sent false
29444 8789 Acked Write 3 Done Done Sent false
29445 8790 Acked Read 2 Done Done Sent false
29446 8791 Acked Read 4 Done Done Sent false
29447 8792 Acked Write 4 Done Done Sent false
29448 8793 Acked Flush 0 Done Done Sent false
29449 8794 Acked Read 6 Done Done Sent false
29450 8795 Acked Read 10 Done Done Sent false
29451 8796 Acked Read 9 Done Done Sent false
29452 8797 Acked Write 7 Done Done Sent false
29453 8798 Acked Write 2 Done Done Sent false
29454 8799 Acked Write 9 Done Done Sent false
29455 8800 Acked Read 1 Done Done Sent false
29456 8801 Acked Write 10 Done Done Sent false
29457 8802 Acked Read 4 Done Done Sent false
29458 8803 Acked Write 8 Done Done Sent false
29459 8804 Acked Write 2 Done Done Sent false
29460 8805 Acked Write 10 Done Done Sent false
29461 8806 Acked Read 6 Done Done Sent false
29462 8807 Acked Read 3 Done Done Sent false
29463 8808 Acked Write 5 Done Done Sent false
29464 8809 Acked Flush 0 Done Done Sent false
29465 8810 Acked Read 2 Done Done Sent false
29466 8811 Acked Write 1 Done Done Sent false
29467 8812 Acked Read 7 Done Done Sent false
29468 8813 Acked Read 3 Done Done Sent false
29469 8814 Acked Flush 0 Done Done Sent false
29470 8815 Acked Write 3 Done Done Sent false
29471 8816 Acked Write 10 Done Done Sent false
29472 8817 Acked Read 4 Done Done Sent false
29473 8818 Acked Write 5 Done Done Sent false
29474 8819 Acked Write 8 Done Done Sent false
29475 8820 Acked Read 5 Done Done Sent false
29476 8821 Acked Read 9 Done Done Sent false
29477 8822 Acked Write 4 Done Done Sent false
29478 8823 Acked Write 8 Done Done Sent false
29479 8824 Acked Write 5 Done Done Sent false
29480 8825 Acked Read 1 Done Done Sent false
29481 8826 Acked Read 6 Done Done Sent false
29482 8827 Acked Write 7 Done Done Sent false
29483 8828 Acked Read 7 Done Done Sent false
29484 8829 Acked Write 5 Done Done Sent false
29485 8830 Acked Write 10 Done Done Sent false
29486 8831 Acked Read 5 Done Done Sent false
29487 8832 Acked Read 10 Done Done Sent false
29488 8833 Acked Read 10 Done Done Sent false
29489 8834 Acked Write 9 Done Done Sent false
29490 8835 Acked Read 7 Done Done Sent false
29491 8836 Acked Read 4 Done Done Sent false
29492 8837 Acked Read 3 Done Done Sent false
29493 8838 Acked Write 9 Done Done Sent false
29494 8839 Acked Read 10 Done Done Sent false
29495 8840 Acked Read 10 Done Done Sent false
29496 8841 Acked Read 4 Done Done Sent false
29497 8842 Acked Read 3 Done Done Sent false
29498 8843 Acked Write 1 Done Done Sent false
29499 8844 Acked Write 8 Done Done Sent false
29500 8845 Acked Read 4 Done Done Sent false
29501 8846 Acked Write 3 Done Done Sent false
29502 8847 Acked Flush 0 Done Done Sent false
29503 STATES DS:0 DS:1 DS:2 TOTAL
29504 Sent 0 0 301 301
29505 Done 301 301 0 602
29506 Skipped 0 0 0 0
29507 Error 0 0 0 0
29508 Last Flush: Some(JobId(8847)) Some(JobId(8847)) Some(JobId(8546))
29509 Downstairs last five completed: 8546 8545 8544 8543 8542
29510 Upstairs last five completed: 8847 8846 8845 8844 8843
29511 CLIENT: Up:0 ds:301 act:2
295122025-02-21T06:51:50.554ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
295132025-02-21T06:51:51.253ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
295142025-02-21T06:51:51.572ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
295152025-02-21T06:51:51.572ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
295162025-02-21T06:51:51.572ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
295172025-02-21T06:51:51.572ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
295182025-02-21T06:51:51.572ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
295192025-02-21T06:51:52.255ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
295202025-02-21T06:51:53.256ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
29521 [8830] got start action
29522 [8830] I am going to restart
29523 Starting downstairs at port 8830
29524 Make output file at "/tmp/dsc/downstairs-8830.txt"
29525 [8830][2] reports Starting
29526 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1232)
29527 [8830][2] reports Running
29528 ----------------------------------------------------------------
29529 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29530 States: Active Active Offline
29531 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29532 8547 Acked Write 3 Done Done Sent false
29533 8548 Acked Write 5 Done Done Sent false
29534 8549 Acked Write 2 Done Done Sent false
29535 8550 Acked Flush 0 Done Done Sent false
29536 8551 Acked Read 10 Done Done Sent false
29537 8552 Acked Read 9 Done Done Sent false
29538 8553 Acked Read 3 Done Done Sent false
29539 8554 Acked Write 3 Done Done Sent false
29540 8555 Acked Flush 0 Done Done Sent false
29541 8556 Acked Write 9 Done Done Sent false
29542 8557 Acked Read 7 Done Done Sent false
29543 8558 Acked Read 10 Done Done Sent false
29544 8559 Acked Write 10 Done Done Sent false
29545 8560 Acked Read 7 Done Done Sent false
29546 8561 Acked Read 10 Done Done Sent false
29547 8562 Acked Read 7 Done Done Sent false
29548 8563 Acked Write 1 Done Done Sent false
29549 8564 Acked Read 5 Done Done Sent false
29550 8565 Acked Write 7 Done Done Sent false
29551 8566 Acked Read 3 Done Done Sent false
29552 8567 Acked Flush 0 Done Done Sent false
29553 8568 Acked Write 4 Done Done Sent false
29554 8569 Acked Read 7 Done Done Sent false
29555 8570 Acked Write 8 Done Done Sent false
29556 8571 Acked Write 7 Done Done Sent false
29557 8572 Acked Read 3 Done Done Sent false
29558 8573 Acked Read 4 Done Done Sent false
29559 8574 Acked Write 5 Done Done Sent false
29560 8575 Acked Read 2 Done Done Sent false
29561 8576 Acked Write 8 Done Done Sent false
29562 8577 Acked Read 8 Done Done Sent false
29563 8578 Acked Write 8 Done Done Sent false
29564 8579 Acked Write 7 Done Done Sent false
29565 8580 Acked Read 1 Done Done Sent false
29566 8581 Acked Write 5 Done Done Sent false
29567 8582 Acked Read 4 Done Done Sent false
29568 8583 Acked Read 6 Done Done Sent false
29569 8584 Acked Read 2 Done Done Sent false
29570 8585 Acked Read 7 Done Done Sent false
29571 8586 Acked Read 9 Done Done Sent false
29572 8587 Acked Write 3 Done Done Sent false
29573 8588 Acked Write 8 Done Done Sent false
29574 8589 Acked Read 10 Done Done Sent false
29575 8590 Acked Read 2 Done Done Sent false
29576 8591 Acked Write 9 Done Done Sent false
29577 8592 Acked Read 9 Done Done Sent false
29578 8593 Acked Read 7 Done Done Sent false
29579 8594 Acked Read 6 Done Done Sent false
29580 8595 Acked Read 4 Done Done Sent false
29581 8596 Acked Write 10 Done Done Sent false
29582 8597 Acked Read 10 Done Done Sent false
29583 8598 Acked Read 5 Done Done Sent false
29584 8599 Acked Write 10 Done Done Sent false
29585 8600 Acked Write 5 Done Done Sent false
29586 8601 Acked Write 4 Done Done Sent false
29587 8602 Acked Read 1 Done Done Sent false
29588 8603 Acked Read 1 Done Done Sent false
29589 8604 Acked Write 2 Done Done Sent false
29590 8605 Acked Write 5 Done Done Sent false
29591 8606 Acked Write 10 Done Done Sent false
29592 8607 Acked Read 7 Done Done Sent false
29593 8608 Acked Read 5 Done Done Sent false
29594 8609 Acked Read 2 Done Done Sent false
29595 8610 Acked Read 7 Done Done Sent false
29596 8611 Acked Write 10 Done Done Sent false
29597 8612 Acked Write 6 Done Done Sent false
29598 8613 Acked Read 7 Done Done Sent false
29599 8614 Acked Write 10 Done Done Sent false
29600 8615 Acked Write 3 Done Done Sent false
29601 8616 Acked Read 5 Done Done Sent false
29602 8617 Acked Write 10 Done Done Sent false
29603 8618 Acked Write 8 Done Done Sent false
29604 8619 Acked Read 10 Done Done Sent false
29605 8620 Acked Read 9 Done Done Sent false
29606 8621 Acked Read 4 Done Done Sent false
29607 8622 Acked Read 2 Done Done Sent false
29608 8623 Acked Write 2 Done Done Sent false
29609 8624 Acked Read 9 Done Done Sent false
29610 8625 Acked Write 9 Done Done Sent false
29611 8626 Acked Read 9 Done Done Sent false
29612 8627 Acked Write 2 Done Done Sent false
29613 8628 Acked Write 6 Done Done Sent false
29614 8629 Acked Flush 0 Done Done Sent false
29615 8630 Acked Write 7 Done Done Sent false
29616 8631 Acked Read 10 Done Done Sent false
29617 8632 Acked Read 6 Done Done Sent false
29618 8633 Acked Write 8 Done Done Sent false
29619 8634 Acked Write 3 Done Done Sent false
29620 8635 Acked Read 2 Done Done Sent false
29621 8636 Acked Read 7 Done Done Sent false
29622 8637 Acked Write 9 Done Done Sent false
29623 8638 Acked Read 8 Done Done Sent false
29624 8639 Acked Read 8 Done Done Sent false
29625 8640 Acked Read 1 Done Done Sent false
29626 8641 Acked Read 8 Done Done Sent false
29627 8642 Acked Read 3 Done Done Sent false
29628 8643 Acked Flush 0 Done Done Sent false
29629 8644 Acked Write 1 Done Done Sent false
29630 8645 Acked Write 8 Done Done Sent false
29631 8646 Acked Write 2 Done Done Sent false
29632 8647 Acked Read 1 Done Done Sent false
29633 8648 Acked Flush 0 Done Done Sent false
29634 8649 Acked Read 7 Done Done Sent false
29635 8650 Acked Read 7 Done Done Sent false
29636 8651 Acked Write 7 Done Done Sent false
29637 8652 Acked Write 9 Done Done Sent false
29638 8653 Acked Read 10 Done Done Sent false
29639 8654 Acked Read 7 Done Done Sent false
29640 8655 Acked Flush 0 Done Done Sent false
29641 8656 Acked Read 5 Done Done Sent false
29642 8657 Acked Read 9 Done Done Sent false
29643 8658 Acked Read 1 Done Done Sent false
29644 8659 Acked Write 9 Done Done Sent false
29645 8660 Acked Flush 0 Done Done Sent false
29646 8661 Acked Flush 0 Done Done Sent false
29647 8662 Acked Write 7 Done Done Sent false
29648 8663 Acked Read 6 Done Done Sent false
29649 8664 Acked Write 4 Done Done Sent false
29650 8665 Acked Write 8 Done Done Sent false
29651 8666 Acked Read 1 Done Done Sent false
29652 8667 Acked Flush 0 Done Done Sent false
29653 8668 Acked Write 10 Done Done Sent false
29654 8669 Acked Read 10 Done Done Sent false
29655 8670 Acked Read 6 Done Done Sent false
29656 8671 Acked Write 4 Done Done Sent false
29657 8672 Acked Write 1 Done Done Sent false
29658 8673 Acked Read 7 Done Done Sent false
29659 8674 Acked Read 4 Done Done Sent false
29660 8675 Acked Read 10 Done Done Sent false
29661 8676 Acked Read 5 Done Done Sent false
29662 8677 Acked Write 5 Done Done Sent false
29663 8678 Acked Read 4 Done Done Sent false
29664 8679 Acked Read 2 Done Done Sent false
29665 8680 Acked Flush 0 Done Done Sent false
29666 8681 Acked Write 3 Done Done Sent false
29667 8682 Acked Write 4 Done Done Sent false
29668 8683 Acked Write 4 Done Done Sent false
29669 8684 Acked Flush 0 Done Done Sent false
29670 8685 Acked Read 6 Done Done Sent false
29671 8686 Acked Read 7 Done Done Sent false
29672 8687 Acked Write 10 Done Done Sent false
29673 8688 Acked Flush 0 Done Done Sent false
29674 8689 Acked Write 7 Done Done Sent false
29675 8690 Acked Read 6 Done Done Sent false
29676 8691 Acked Write 8 Done Done Sent false
29677 8692 Acked Flush 0 Done Done Sent false
29678 8693 Acked Write 6 Done Done Sent false
29679 8694 Acked Write 9 Done Done Sent false
29680 8695 Acked Read 8 Done Done Sent false
29681 8696 Acked Write 8 Done Done Sent false
29682 8697 Acked Read 2 Done Done Sent false
29683 8698 Acked Write 2 Done Done Sent false
29684 8699 Acked Write 5 Done Done Sent false
29685 8700 Acked Write 7 Done Done Sent false
29686 8701 Acked Write 10 Done Done Sent false
29687 8702 Acked Read 5 Done Done Sent false
29688 8703 Acked Read 2 Done Done Sent false
29689 8704 Acked Write 10 Done Done Sent false
29690 8705 Acked Write 4 Done Done Sent false
29691 8706 Acked Flush 0 Done Done Sent false
29692 8707 Acked Read 7 Done Done Sent false
29693 8708 Acked Read 8 Done Done Sent false
29694 8709 Acked Read 10 Done Done Sent false
29695 8710 Acked Read 1 Done Done Sent false
29696 8711 Acked Write 9 Done Done Sent false
29697 8712 Acked Write 5 Done Done Sent false
29698 8713 Acked Write 2 Done Done Sent false
29699 8714 Acked Read 8 Done Done Sent false
29700 8715 Acked Write 3 Done Done Sent false
29701 8716 Acked Write 1 Done Done Sent false
29702 8717 Acked Write 10 Done Done Sent false
29703 8718 Acked Write 5 Done Done Sent false
29704 8719 Acked Read 4 Done Done Sent false
29705 8720 Acked Read 9 Done Done Sent false
29706 8721 Acked Write 3 Done Done Sent false
29707 8722 Acked Write 10 Done Done Sent false
29708 8723 Acked Read 6 Done Done Sent false
29709 8724 Acked Read 8 Done Done Sent false
29710 8725 Acked Read 3 Done Done Sent false
29711 8726 Acked Read 2 Done Done Sent false
29712 8727 Acked Flush 0 Done Done Sent false
29713 8728 Acked Flush 0 Done Done Sent false
29714 8729 Acked Read 8 Done Done Sent false
29715 8730 Acked Write 7 Done Done Sent false
29716 8731 Acked Write 3 Done Done Sent false
29717 8732 Acked Read 7 Done Done Sent false
29718 8733 Acked Read 6 Done Done Sent false
29719 8734 Acked Flush 0 Done Done Sent false
29720 8735 Acked Read 8 Done Done Sent false
29721 8736 Acked Write 7 Done Done Sent false
29722 8737 Acked Read 10 Done Done Sent false
29723 8738 Acked Read 6 Done Done Sent false
29724 8739 Acked Read 4 Done Done Sent false
29725 8740 Acked Write 7 Done Done Sent false
29726 8741 Acked Read 3 Done Done Sent false
29727 8742 Acked Write 10 Done Done Sent false
29728 8743 Acked Flush 0 Done Done Sent false
29729 8744 Acked Read 5 Done Done Sent false
29730 8745 Acked Write 9 Done Done Sent false
29731 8746 Acked Write 8 Done Done Sent false
29732 8747 Acked Write 5 Done Done Sent false
29733 8748 Acked Read 7 Done Done Sent false
29734 8749 Acked Read 4 Done Done Sent false
29735 8750 Acked Write 4 Done Done Sent false
29736 8751 Acked Write 4 Done Done Sent false
29737 8752 Acked Read 3 Done Done Sent false
29738 8753 Acked Read 3 Done Done Sent false
29739 8754 Acked Write 1 Done Done Sent false
29740 8755 Acked Read 8 Done Done Sent false
29741 8756 Acked Read 3 Done Done Sent false
29742 8757 Acked Write 3 Done Done Sent false
29743 8758 Acked Read 10 Done Done Sent false
29744 8759 Acked Flush 0 Done Done Sent false
29745 8760 Acked Read 1 Done Done Sent false
29746 8761 Acked Read 6 Done Done Sent false
29747 8762 Acked Flush 0 Done Done Sent false
29748 8763 Acked Read 1 Done Done Sent false
29749 8764 Acked Read 4 Done Done Sent false
29750 8765 Acked Write 7 Done Done Sent false
29751 8766 Acked Write 6 Done Done Sent false
29752 8767 Acked Read 9 Done Done Sent false
29753 8768 Acked Write 5 Done Done Sent false
29754 8769 Acked Write 1 Done Done Sent false
29755 8770 Acked Write 9 Done Done Sent false
29756 8771 Acked Write 1 Done Done Sent false
29757 8772 Acked Read 4 Done Done Sent false
29758 8773 Acked Write 9 Done Done Sent false
29759 8774 Acked Read 5 Done Done Sent false
29760 8775 Acked Write 5 Done Done Sent false
29761 8776 Acked Write 5 Done Done Sent false
29762 8777 Acked Read 2 Done Done Sent false
29763 8778 Acked Write 3 Done Done Sent false
29764 8779 Acked Write 2 Done Done Sent false
29765 8780 Acked Read 10 Done Done Sent false
29766 8781 Acked Write 7 Done Done Sent false
29767 8782 Acked Flush 0 Done Done Sent false
29768 8783 Acked Write 4 Done Done Sent false
29769 8784 Acked Write 10 Done Done Sent false
29770 8785 Acked Write 4 Done Done Sent false
29771 8786 Acked Read 7 Done Done Sent false
29772 8787 Acked Read 6 Done Done Sent false
29773 8788 Acked Read 6 Done Done Sent false
29774 8789 Acked Write 3 Done Done Sent false
29775 8790 Acked Read 2 Done Done Sent false
29776 8791 Acked Read 4 Done Done Sent false
29777 8792 Acked Write 4 Done Done Sent false
29778 8793 Acked Flush 0 Done Done Sent false
29779 8794 Acked Read 6 Done Done Sent false
29780 8795 Acked Read 10 Done Done Sent false
29781 8796 Acked Read 9 Done Done Sent false
29782 8797 Acked Write 7 Done Done Sent false
29783 8798 Acked Write 2 Done Done Sent false
29784 8799 Acked Write 9 Done Done Sent false
29785 8800 Acked Read 1 Done Done Sent false
29786 8801 Acked Write 10 Done Done Sent false
29787 8802 Acked Read 4 Done Done Sent false
29788 8803 Acked Write 8 Done Done Sent false
29789 8804 Acked Write 2 Done Done Sent false
29790 8805 Acked Write 10 Done Done Sent false
29791 8806 Acked Read 6 Done Done Sent false
29792 8807 Acked Read 3 Done Done Sent false
29793 8808 Acked Write 5 Done Done Sent false
29794 8809 Acked Flush 0 Done Done Sent false
29795 8810 Acked Read 2 Done Done Sent false
29796 8811 Acked Write 1 Done Done Sent false
29797 8812 Acked Read 7 Done Done Sent false
29798 8813 Acked Read 3 Done Done Sent false
29799 8814 Acked Flush 0 Done Done Sent false
29800 8815 Acked Write 3 Done Done Sent false
29801 8816 Acked Write 10 Done Done Sent false
29802 8817 Acked Read 4 Done Done Sent false
29803 8818 Acked Write 5 Done Done Sent false
29804 8819 Acked Write 8 Done Done Sent false
29805 8820 Acked Read 5 Done Done Sent false
29806 8821 Acked Read 9 Done Done Sent false
29807 8822 Acked Write 4 Done Done Sent false
29808 8823 Acked Write 8 Done Done Sent false
29809 8824 Acked Write 5 Done Done Sent false
29810 8825 Acked Read 1 Done Done Sent false
29811 8826 Acked Read 6 Done Done Sent false
29812 8827 Acked Write 7 Done Done Sent false
29813 8828 Acked Read 7 Done Done Sent false
29814 8829 Acked Write 5 Done Done Sent false
29815 8830 Acked Write 10 Done Done Sent false
29816 8831 Acked Read 5 Done Done Sent false
29817 8832 Acked Read 10 Done Done Sent false
29818 8833 Acked Read 10 Done Done Sent false
29819 8834 Acked Write 9 Done Done Sent false
29820 8835 Acked Read 7 Done Done Sent false
29821 8836 Acked Read 4 Done Done Sent false
29822 8837 Acked Read 3 Done Done Sent false
29823 8838 Acked Write 9 Done Done Sent false
29824 8839 Acked Read 10 Done Done Sent false
29825 8840 Acked Read 10 Done Done Sent false
29826 8841 Acked Read 4 Done Done Sent false
29827 8842 Acked Read 3 Done Done Sent false
29828 8843 Acked Write 1 Done Done Sent false
29829 8844 Acked Write 8 Done Done Sent false
29830 8845 Acked Read 4 Done Done Sent false
29831 8846 Acked Write 3 Done Done Sent false
29832 8847 Acked Flush 0 Done Done Sent false
29833 STATES DS:0 DS:1 DS:2 TOTAL
29834 Sent 0 0 301 301
29835 Done 301 301 0 602
29836 Skipped 0 0 0 0
29837 Error 0 0 0 0
29838 Last Flush: Some(JobId(8847)) Some(JobId(8847)) Some(JobId(8546))
29839 Downstairs last five completed: 8546 8545 8544 8543 8542
29840 Upstairs last five completed: 8847 8846 8845 8844 8843
29841 CLIENT: Up:0 ds:301 act:2
298422025-02-21T06:51:54.328ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
298432025-02-21T06:51:55.556ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
298442025-02-21T06:51:56.261ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
298452025-02-21T06:51:57.262ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
29846 ----------------------------------------------------------------
29847 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29848 States: Active Active Offline
29849 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29850 8547 Acked Write 3 Done Done Sent false
29851 8548 Acked Write 5 Done Done Sent false
29852 8549 Acked Write 2 Done Done Sent false
29853 8550 Acked Flush 0 Done Done Sent false
29854 8551 Acked Read 10 Done Done Sent false
29855 8552 Acked Read 9 Done Done Sent false
29856 8553 Acked Read 3 Done Done Sent false
29857 8554 Acked Write 3 Done Done Sent false
29858 8555 Acked Flush 0 Done Done Sent false
29859 8556 Acked Write 9 Done Done Sent false
29860 8557 Acked Read 7 Done Done Sent false
29861 8558 Acked Read 10 Done Done Sent false
29862 8559 Acked Write 10 Done Done Sent false
29863 8560 Acked Read 7 Done Done Sent false
29864 8561 Acked Read 10 Done Done Sent false
29865 8562 Acked Read 7 Done Done Sent false
29866 8563 Acked Write 1 Done Done Sent false
29867 8564 Acked Read 5 Done Done Sent false
29868 8565 Acked Write 7 Done Done Sent false
29869 8566 Acked Read 3 Done Done Sent false
29870 8567 Acked Flush 0 Done Done Sent false
29871 8568 Acked Write 4 Done Done Sent false
29872 8569 Acked Read 7 Done Done Sent false
29873 8570 Acked Write 8 Done Done Sent false
29874 8571 Acked Write 7 Done Done Sent false
29875 8572 Acked Read 3 Done Done Sent false
29876 8573 Acked Read 4 Done Done Sent false
29877 8574 Acked Write 5 Done Done Sent false
29878 8575 Acked Read 2 Done Done Sent false
29879 8576 Acked Write 8 Done Done Sent false
29880 8577 Acked Read 8 Done Done Sent false
29881 8578 Acked Write 8 Done Done Sent false
29882 8579 Acked Write 7 Done Done Sent false
29883 8580 Acked Read 1 Done Done Sent false
29884 8581 Acked Write 5 Done Done Sent false
29885 8582 Acked Read 4 Done Done Sent false
29886 8583 Acked Read 6 Done Done Sent false
29887 8584 Acked Read 2 Done Done Sent false
29888 8585 Acked Read 7 Done Done Sent false
29889 8586 Acked Read 9 Done Done Sent false
29890 8587 Acked Write 3 Done Done Sent false
29891 8588 Acked Write 8 Done Done Sent false
29892 8589 Acked Read 10 Done Done Sent false
29893 8590 Acked Read 2 Done Done Sent false
29894 8591 Acked Write 9 Done Done Sent false
29895 8592 Acked Read 9 Done Done Sent false
29896 8593 Acked Read 7 Done Done Sent false
29897 8594 Acked Read 6 Done Done Sent false
29898 8595 Acked Read 4 Done Done Sent false
29899 8596 Acked Write 10 Done Done Sent false
29900 8597 Acked Read 10 Done Done Sent false
29901 8598 Acked Read 5 Done Done Sent false
29902 8599 Acked Write 10 Done Done Sent false
29903 8600 Acked Write 5 Done Done Sent false
29904 8601 Acked Write 4 Done Done Sent false
29905 8602 Acked Read 1 Done Done Sent false
29906 8603 Acked Read 1 Done Done Sent false
29907 8604 Acked Write 2 Done Done Sent false
29908 8605 Acked Write 5 Done Done Sent false
29909 8606 Acked Write 10 Done Done Sent false
29910 8607 Acked Read 7 Done Done Sent false
29911 8608 Acked Read 5 Done Done Sent false
29912 8609 Acked Read 2 Done Done Sent false
29913 8610 Acked Read 7 Done Done Sent false
29914 8611 Acked Write 10 Done Done Sent false
29915 8612 Acked Write 6 Done Done Sent false
29916 8613 Acked Read 7 Done Done Sent false
29917 8614 Acked Write 10 Done Done Sent false
29918 8615 Acked Write 3 Done Done Sent false
29919 8616 Acked Read 5 Done Done Sent false
29920 8617 Acked Write 10 Done Done Sent false
29921 8618 Acked Write 8 Done Done Sent false
29922 8619 Acked Read 10 Done Done Sent false
29923 8620 Acked Read 9 Done Done Sent false
29924 8621 Acked Read 4 Done Done Sent false
29925 8622 Acked Read 2 Done Done Sent false
29926 8623 Acked Write 2 Done Done Sent false
29927 8624 Acked Read 9 Done Done Sent false
29928 8625 Acked Write 9 Done Done Sent false
29929 8626 Acked Read 9 Done Done Sent false
29930 8627 Acked Write 2 Done Done Sent false
29931 8628 Acked Write 6 Done Done Sent false
29932 8629 Acked Flush 0 Done Done Sent false
29933 8630 Acked Write 7 Done Done Sent false
29934 8631 Acked Read 10 Done Done Sent false
29935 8632 Acked Read 6 Done Done Sent false
29936 8633 Acked Write 8 Done Done Sent false
29937 8634 Acked Write 3 Done Done Sent false
29938 8635 Acked Read 2 Done Done Sent false
29939 8636 Acked Read 7 Done Done Sent false
29940 8637 Acked Write 9 Done Done Sent false
29941 8638 Acked Read 8 Done Done Sent false
29942 8639 Acked Read 8 Done Done Sent false
29943 8640 Acked Read 1 Done Done Sent false
29944 8641 Acked Read 8 Done Done Sent false
29945 8642 Acked Read 3 Done Done Sent false
29946 8643 Acked Flush 0 Done Done Sent false
29947 8644 Acked Write 1 Done Done Sent false
29948 8645 Acked Write 8 Done Done Sent false
29949 8646 Acked Write 2 Done Done Sent false
29950 8647 Acked Read 1 Done Done Sent false
29951 8648 Acked Flush 0 Done Done Sent false
29952 8649 Acked Read 7 Done Done Sent false
29953 8650 Acked Read 7 Done Done Sent false
29954 8651 Acked Write 7 Done Done Sent false
29955 8652 Acked Write 9 Done Done Sent false
29956 8653 Acked Read 10 Done Done Sent false
29957 8654 Acked Read 7 Done Done Sent false
29958 8655 Acked Flush 0 Done Done Sent false
29959 8656 Acked Read 5 Done Done Sent false
29960 8657 Acked Read 9 Done Done Sent false
29961 8658 Acked Read 1 Done Done Sent false
29962 8659 Acked Write 9 Done Done Sent false
29963 8660 Acked Flush 0 Done Done Sent false
29964 8661 Acked Flush 0 Done Done Sent false
29965 8662 Acked Write 7 Done Done Sent false
29966 8663 Acked Read 6 Done Done Sent false
29967 8664 Acked Write 4 Done Done Sent false
29968 8665 Acked Write 8 Done Done Sent false
29969 8666 Acked Read 1 Done Done Sent false
29970 8667 Acked Flush 0 Done Done Sent false
29971 8668 Acked Write 10 Done Done Sent false
29972 8669 Acked Read 10 Done Done Sent false
29973 8670 Acked Read 6 Done Done Sent false
29974 8671 Acked Write 4 Done Done Sent false
29975 8672 Acked Write 1 Done Done Sent false
29976 8673 Acked Read 7 Done Done Sent false
29977 8674 Acked Read 4 Done Done Sent false
29978 8675 Acked Read 10 Done Done Sent false
29979 8676 Acked Read 5 Done Done Sent false
29980 8677 Acked Write 5 Done Done Sent false
29981 8678 Acked Read 4 Done Done Sent false
29982 8679 Acked Read 2 Done Done Sent false
29983 8680 Acked Flush 0 Done Done Sent false
29984 8681 Acked Write 3 Done Done Sent false
29985 8682 Acked Write 4 Done Done Sent false
29986 8683 Acked Write 4 Done Done Sent false
29987 8684 Acked Flush 0 Done Done Sent false
29988 8685 Acked Read 6 Done Done Sent false
29989 8686 Acked Read 7 Done Done Sent false
29990 8687 Acked Write 10 Done Done Sent false
29991 8688 Acked Flush 0 Done Done Sent false
29992 8689 Acked Write 7 Done Done Sent false
29993 8690 Acked Read 6 Done Done Sent false
29994 8691 Acked Write 8 Done Done Sent false
29995 8692 Acked Flush 0 Done Done Sent false
29996 8693 Acked Write 6 Done Done Sent false
29997 8694 Acked Write 9 Done Done Sent false
29998 8695 Acked Read 8 Done Done Sent false
29999 8696 Acked Write 8 Done Done Sent false
30000 8697 Acked Read 2 Done Done Sent false
30001 8698 Acked Write 2 Done Done Sent false
30002 8699 Acked Write 5 Done Done Sent false
30003 8700 Acked Write 7 Done Done Sent false
30004 8701 Acked Write 10 Done Done Sent false
30005 8702 Acked Read 5 Done Done Sent false
30006 8703 Acked Read 2 Done Done Sent false
30007 8704 Acked Write 10 Done Done Sent false
30008 8705 Acked Write 4 Done Done Sent false
30009 8706 Acked Flush 0 Done Done Sent false
30010 8707 Acked Read 7 Done Done Sent false
30011 8708 Acked Read 8 Done Done Sent false
30012 8709 Acked Read 10 Done Done Sent false
30013 8710 Acked Read 1 Done Done Sent false
30014 8711 Acked Write 9 Done Done Sent false
30015 8712 Acked Write 5 Done Done Sent false
30016 8713 Acked Write 2 Done Done Sent false
30017 8714 Acked Read 8 Done Done Sent false
30018 8715 Acked Write 3 Done Done Sent false
30019 8716 Acked Write 1 Done Done Sent false
30020 8717 Acked Write 10 Done Done Sent false
30021 8718 Acked Write 5 Done Done Sent false
30022 8719 Acked Read 4 Done Done Sent false
30023 8720 Acked Read 9 Done Done Sent false
30024 8721 Acked Write 3 Done Done Sent false
30025 8722 Acked Write 10 Done Done Sent false
30026 8723 Acked Read 6 Done Done Sent false
30027 8724 Acked Read 8 Done Done Sent false
30028 8725 Acked Read 3 Done Done Sent false
30029 8726 Acked Read 2 Done Done Sent false
30030 8727 Acked Flush 0 Done Done Sent false
30031 8728 Acked Flush 0 Done Done Sent false
30032 8729 Acked Read 8 Done Done Sent false
30033 8730 Acked Write 7 Done Done Sent false
30034 8731 Acked Write 3 Done Done Sent false
30035 8732 Acked Read 7 Done Done Sent false
30036 8733 Acked Read 6 Done Done Sent false
30037 8734 Acked Flush 0 Done Done Sent false
30038 8735 Acked Read 8 Done Done Sent false
30039 8736 Acked Write 7 Done Done Sent false
30040 8737 Acked Read 10 Done Done Sent false
30041 8738 Acked Read 6 Done Done Sent false
30042 8739 Acked Read 4 Done Done Sent false
30043 8740 Acked Write 7 Done Done Sent false
30044 8741 Acked Read 3 Done Done Sent false
30045 8742 Acked Write 10 Done Done Sent false
30046 8743 Acked Flush 0 Done Done Sent false
30047 8744 Acked Read 5 Done Done Sent false
30048 8745 Acked Write 9 Done Done Sent false
30049 8746 Acked Write 8 Done Done Sent false
30050 8747 Acked Write 5 Done Done Sent false
30051 8748 Acked Read 7 Done Done Sent false
30052 8749 Acked Read 4 Done Done Sent false
30053 8750 Acked Write 4 Done Done Sent false
30054 8751 Acked Write 4 Done Done Sent false
30055 8752 Acked Read 3 Done Done Sent false
30056 8753 Acked Read 3 Done Done Sent false
30057 8754 Acked Write 1 Done Done Sent false
30058 8755 Acked Read 8 Done Done Sent false
30059 8756 Acked Read 3 Done Done Sent false
30060 8757 Acked Write 3 Done Done Sent false
30061 8758 Acked Read 10 Done Done Sent false
30062 8759 Acked Flush 0 Done Done Sent false
30063 8760 Acked Read 1 Done Done Sent false
30064 8761 Acked Read 6 Done Done Sent false
30065 8762 Acked Flush 0 Done Done Sent false
30066 8763 Acked Read 1 Done Done Sent false
30067 8764 Acked Read 4 Done Done Sent false
30068 8765 Acked Write 7 Done Done Sent false
30069 8766 Acked Write 6 Done Done Sent false
30070 8767 Acked Read 9 Done Done Sent false
30071 8768 Acked Write 5 Done Done Sent false
30072 8769 Acked Write 1 Done Done Sent false
30073 8770 Acked Write 9 Done Done Sent false
30074 8771 Acked Write 1 Done Done Sent false
30075 8772 Acked Read 4 Done Done Sent false
30076 8773 Acked Write 9 Done Done Sent false
30077 8774 Acked Read 5 Done Done Sent false
30078 8775 Acked Write 5 Done Done Sent false
30079 8776 Acked Write 5 Done Done Sent false
30080 8777 Acked Read 2 Done Done Sent false
30081 8778 Acked Write 3 Done Done Sent false
30082 8779 Acked Write 2 Done Done Sent false
30083 8780 Acked Read 10 Done Done Sent false
30084 8781 Acked Write 7 Done Done Sent false
30085 8782 Acked Flush 0 Done Done Sent false
30086 8783 Acked Write 4 Done Done Sent false
30087 8784 Acked Write 10 Done Done Sent false
30088 8785 Acked Write 4 Done Done Sent false
30089 8786 Acked Read 7 Done Done Sent false
30090 8787 Acked Read 6 Done Done Sent false
30091 8788 Acked Read 6 Done Done Sent false
30092 8789 Acked Write 3 Done Done Sent false
30093 8790 Acked Read 2 Done Done Sent false
30094 8791 Acked Read 4 Done Done Sent false
30095 8792 Acked Write 4 Done Done Sent false
30096 8793 Acked Flush 0 Done Done Sent false
30097 8794 Acked Read 6 Done Done Sent false
30098 8795 Acked Read 10 Done Done Sent false
30099 8796 Acked Read 9 Done Done Sent false
30100 8797 Acked Write 7 Done Done Sent false
30101 8798 Acked Write 2 Done Done Sent false
30102 8799 Acked Write 9 Done Done Sent false
30103 8800 Acked Read 1 Done Done Sent false
30104 8801 Acked Write 10 Done Done Sent false
30105 8802 Acked Read 4 Done Done Sent false
30106 8803 Acked Write 8 Done Done Sent false
30107 8804 Acked Write 2 Done Done Sent false
30108 8805 Acked Write 10 Done Done Sent false
30109 8806 Acked Read 6 Done Done Sent false
30110 8807 Acked Read 3 Done Done Sent false
30111 8808 Acked Write 5 Done Done Sent false
30112 8809 Acked Flush 0 Done Done Sent false
30113 8810 Acked Read 2 Done Done Sent false
30114 8811 Acked Write 1 Done Done Sent false
30115 8812 Acked Read 7 Done Done Sent false
30116 8813 Acked Read 3 Done Done Sent false
30117 8814 Acked Flush 0 Done Done Sent false
30118 8815 Acked Write 3 Done Done Sent false
30119 8816 Acked Write 10 Done Done Sent false
30120 8817 Acked Read 4 Done Done Sent false
30121 8818 Acked Write 5 Done Done Sent false
30122 8819 Acked Write 8 Done Done Sent false
30123 8820 Acked Read 5 Done Done Sent false
30124 8821 Acked Read 9 Done Done Sent false
30125 8822 Acked Write 4 Done Done Sent false
30126 8823 Acked Write 8 Done Done Sent false
30127 8824 Acked Write 5 Done Done Sent false
30128 8825 Acked Read 1 Done Done Sent false
30129 8826 Acked Read 6 Done Done Sent false
30130 8827 Acked Write 7 Done Done Sent false
30131 8828 Acked Read 7 Done Done Sent false
30132 8829 Acked Write 5 Done Done Sent false
30133 8830 Acked Write 10 Done Done Sent false
30134 8831 Acked Read 5 Done Done Sent false
30135 8832 Acked Read 10 Done Done Sent false
30136 8833 Acked Read 10 Done Done Sent false
30137 8834 Acked Write 9 Done Done Sent false
30138 8835 Acked Read 7 Done Done Sent false
30139 8836 Acked Read 4 Done Done Sent false
30140 8837 Acked Read 3 Done Done Sent false
30141 8838 Acked Write 9 Done Done Sent false
30142 8839 Acked Read 10 Done Done Sent false
30143 8840 Acked Read 10 Done Done Sent false
30144 8841 Acked Read 4 Done Done Sent false
30145 8842 Acked Read 3 Done Done Sent false
30146 8843 Acked Write 1 Done Done Sent false
30147 8844 Acked Write 8 Done Done Sent false
30148 8845 Acked Read 4 Done Done Sent false
30149 8846 Acked Write 3 Done Done Sent false
30150 8847 Acked Flush 0 Done Done Sent false
30151 STATES DS:0 DS:1 DS:2 TOTAL
30152 Sent 0 0 301 301
30153 Done 301 301 0 602
30154 Skipped 0 0 0 0
30155 Error 0 0 0 0
30156 Last Flush: Some(JobId(8847)) Some(JobId(8847)) Some(JobId(8546))
30157 Downstairs last five completed: 8546 8545 8544 8543 8542
30158 Upstairs last five completed: 8847 8846 8845 8844 8843
30159 CLIENT: Up:0 ds:301 act:2
301602025-02-21T06:51:59.265ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301612025-02-21T06:52:00.557ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301622025-02-21T06:52:01.268ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301632025-02-21T06:52:01.573ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301642025-02-21T06:52:01.573ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42086) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301652025-02-21T06:52:01.574ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301662025-02-21T06:52:01.575ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301672025-02-21T06:52:01.575ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301682025-02-21T06:52:01.575ZINFOcrucible: send last flush ID to this DS: Some(JobId(8546)) = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301692025-02-21T06:52:01.575ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301702025-02-21T06:52:01.575ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(8546)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301712025-02-21T06:52:01.575ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(8546)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
30172 ----------------------------------------------------------------
30173 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30174 Downstairs last five completed: 8847 8846 8845 8844 8843
30175 Upstairs last five completed: 8847 8846 8845 8844 8843
30176 CLIENT: Up:0 ds:0 act:3
30177 Replay: All jobs finished
30178 Main task has work to do, go find it
30179 got dsc Stop(0)
30180 stop 0
30181 Feb 21 06:52:01.831 INFO request completed, latency_us:[8810] Got stop action so:false kr:false
30182 364, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 57e7021c-ed09-4083-94e8-e241c0fb3726, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
30183 Feb 21 06:52:01.832 INFO request completed, latency_us: 124, response_code: 200, uri: /state/cid/0, method: GET, req_id: 311a360f-8e60-46c5-b91e-21f511f529d7, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
301842025-02-21T06:52:01.832ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301852025-02-21T06:52:01.832ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301862025-02-21T06:52:01.832ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
301872025-02-21T06:52:01.832ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
30188 [8810] Exited with: ExitStatus(unix_wait_status(9))
30189 [8810][0] reports Exit
30190 Feb 21 06:52:05.835 INFO request completed, latency_us: 233, response_code: 200, uri: /state/cid/0, method: GET, req_id: ff284ca2-2e91-4f62-916b-42ddafd95d6c, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
30191 001/300 Flush
30192 002/300 Flush
30193 003/300 Read block 1937 len 12288
30194 004/300 Read block 4228 len 20480
30195 005/300 Write block 1774 len 36864 data: 5 5 5 5 6 7 6 7 7
30196 006/300 Read block 1958 len 20480
30197 007/300 Read block 4915 len 20480
30198 008/300 Read block 4524 len 16384
30199 009/300 Flush
30200 010/300 Read block 216 len 16384
30201 011/300 Read block 4381 len 20480
30202 012/300 Read block 508 len 32768
30203 013/300 Read block 1614 len 4096
30204 014/300 Read block 21 len 40960
30205 015/300 Flush
30206 016/300 Read block 3399 len 28672
30207 017/300 Write block 507 len 20480 data: 4 3 3 3 3
30208 018/300 Flush
30209 019/300 Read block 1657 len 16384
30210 020/300 Flush
30211 021/300 Read block 2763 len 4096
30212 022/300 Read block 4375 len 8192
30213 023/300 Read block 4169 len 40960
30214 024/300 Write block 2406 len 40960 data: 3 3 3 3 5 5 5 7 7 6
30215 025/300 Read block 1835 len 4096
30216 026/300 Write block 2321 len 28672 data: 5 5 5 5 8 11 8
30217 027/300 Write block 2261 len 24576 data: 6 5 5 5 5 4
30218 028/300 Read block 4268 len 8192
30219 029/300 Write block 4816 len 24576 data: 10 10 10 9 9 8
30220 030/300 Read block 303 len 32768
30221 031/300 Write block 3317 len 16384 data: 6 6 6 5
30222 032/300 Flush
30223 033/300 Read block 429 len 20480
30224 034/300 Read block 4156 len 40960
30225 035/300 Read block 4872 len 16384
30226 036/300 Read block 1841 len 28672
30227 037/300 Read block 1667 len 20480
30228 038/300 Write block 1191 len 32768 data: 7 6 5 4 5 4 5 7
30229 039/300 Flush
30230 040/300 Read block 4015 len 12288
30231 041/300 Write block 3121 len 20480 data: 5 5 5 6 6
30232 042/300 Flush
30233 043/300 Write block 4010 len 36864 data: 7 7 7 7 8 9 8 9 8
30234 044/300 Write block 2815 len 8192 data: 6 6
30235 045/300 Read block 710 len 8192
30236 046/300 Write block 482 len 4096 data: 4
30237 047/300 Write block 4064 len 36864 data: 5 3 3 3 4 4 4 4 6
30238 048/300 Read block 3551 len 36864
30239 049/300 Read block 657 len 4096
30240 050/300 Write block 3380 len 24576 data: 3 5 5 5 4 5
30241 051/300 Read block 2270 len 40960
30242 052/300 Write block 1554 len 4096 data: 5
30243 053/300 Read block 2629 len 20480
30244 054/300 Read block 4236 len 32768
30245 055/300 Write block 2368 len 20480 data: 4 3 3 4 4
30246 056/300 Read block 124 len 40960
30247 057/300 Write block 4553 len 20480 data: 5 4 5 5 4
30248 058/300 Write block 303 len 16384 data: 6 7 7 6
30249 059/300 Flush
30250 060/300 Write block 4964 len 24576 data: 7 6 5 3 5 5
30251 061/300 Flush
30252 062/300 Read block 992 len 16384
30253 063/300 Write block 500 len 16384 data: 5 6 6 8
30254 064/300 Write block 1698 len 4096 data: 3
30255 065/300 Write block 2727 len 32768 data: 4 4 4 4 2 2 2 2
30256 066/300 Write block 3588 len 24576 data: 9 6 6 6 7 6
30257 067/300 Write block 955 len 4096 data: 7
30258 068/300 Read block 2257 len 24576
30259 069/300 Read block 2186 len 24576
30260 070/300 Write block 3652 len 24576 data: 5 5 5 5 5 6
30261 071/300 Write block 3566 len 24576 data: 6 7 5 5 4 5
30262 072/300 Write block 1730 len 16384 data: 4 4 4 4
30263 073/300 Read block 1870 len 40960
30264 074/300 Flush
30265 075/300 Write block 1360 len 12288 data: 2 2 3
30266 076/300 Read block 2014 len 12288
30267 077/300 Write block 2399 len 16384 data: 2 3 4 4
30268 078/300 Read block 2747 len 12288
30269 079/300 Write block 3111 len 4096 data: 5
30270 080/300 Read block 4251 len 8192
30271 081/300 Flush
30272 082/300 Read block 376 len 32768
30273 083/300 Read block 4642 len 40960
30274 084/300 Read block 757 len 16384
30275 085/300 Write block 4470 len 40960 data: 7 7 8 8 8 8 9 8 7 6
30276 086/300 Write block 3415 len 28672 data: 9 8 8 6 7 8 8
30277 087/300 Write block 1179 len 12288 data: 5 6 6
30278 088/300 Read block 1262 len 20480
30279 089/300 Write block 1993 len 4096 data: 4
30280 090/300 Write block 893 len 28672 data: 7 6 4 5 5 4 4
30281 091/300 Read block 3930 len 4096
30282 092/300 Write block 3877 len 8192 data: 6 6
30283 093/300 Flush
30284 094/300 Read block 4567 len 28672
30285 095/300 Write block 3260 len 28672 data: 5 5 5 6 5 5 8
30286 096/300 Read block 4807 len 40960
30287 097/300 Read block 206 len 32768
30288 098/300 Read block 1135 len 36864
30289 099/300 Read block 2836 len 12288
30290 100/300 Write block 1972 len 12288 data: 7 6 6
30291 101/300 Write block 2814 len 8192 data: 5 7
30292 102/300 Read block 4924 len 36864
30293 103/300 Write block 525 len 12288 data: 10 8 7
30294 104/300 Write block 876 len 20480 data: 3 5 5 6 4
30295 105/300 Write block 1610 len 32768 data: 9 11 11 8 7 5 6 5
30296 106/300 Read block 3826 len 36864
30297 107/300 Read block 856 len 16384
30298 108/300 Write block 1054 len 28672 data: 9 9 9 8 9 7 7
30299 109/300 Write block 3111 len 36864 data: 6 5 6 7 7 6 4 6 5
30300 110/300 Write block 519 len 40960 data: 5 7 7 7 7 11 11 9 8 7
30301 111/300 Read block 1290 len 40960
30302 112/300 Read block 4031 len 40960
30303 113/300 Write block 712 len 36864 data: 3 4 3 3 4 3 4 4 5
30304 114/300 Read block 859 len 28672
30305 115/300 Write block 2205 len 32768 data: 4 4 5 5 6 5 5 4
30306 116/300 Write block 1317 len 8192 data: 5 4
30307 117/300 Read block 4924 len 40960
30308 118/300 Read block 4773 len 32768
30309 119/300 Write block 1601 len 32768 data: 5 4 6 6 6 5 6 8
30310 120/300 Write block 60 len 24576 data: 5 4 4 5 6 4
30311 121/300 Write block 3077 len 40960 data: 6 7 7 7 7 5 5 5 6 6
30312 122/300 Read block 422 len 12288
30313 123/300 Write block 1033 len 4096 data: 7
30314 124/300 Read block 3627 len 20480
30315 125/300 Flush
30316 126/300 Write block 1506 len 16384 data: 7 6 6 5
30317 127/300 Write block 1322 len 4096 data: 4
30318 128/300 Flush
30319 129/300 Read block 4807 len 8192
30320 130/300 Read block 3043 len 20480
30321 131/300 Read block 4807 len 36864
30322 132/300 Read block 4169 len 12288
30323 133/300 Read block 2830 len 32768
30324 134/300 Read block 3474 len 36864
30325 135/300 Write block 4033 len 40960 data: 8 7 8 9 10 9 9 7 7 5
30326 136/300 Write block 2583 len 8192 data: 3 4
30327 137/300 Flush
30328 138/300 Read block 4648 len 32768
30329 139/300 Write block 1535 len 12288 data: 7 8 7
30330 140/300 Write block 1246 len 20480 data: 7 6 5 5 4
30331 141/300 Read block 3765 len 8192
30332 142/300 Write block 3321 len 40960 data: 7 6 6 7 7 6 4 4 3 3
30333 143/300 Write block 4097 len 12288 data: 5 5 6
30334 144/300 Read block 1152 len 40960
30335 145/300 Read block 2902 len 12288
30336 146/300 Read block 2905 len 8192
30337 147/300 Read block 1185 len 40960
30338 148/300 Read block 3147 len 40960
30339 149/300 Write block 1615 len 20480 data: 6 7 6 7 6
30340 150/300 Read block 2037 len 24576
30341 151/300 Read block 3672 len 32768
30342 152/300 Read block 4159 len 28672
30343 153/300 Write block 220 len 12288 data: 8 8 8
30344 154/300 Read block 1043 len 8192
30345 155/300 Write block 4585 len 12288 data: 4 6 6
30346 156/300 Read block 4302 len 12288
30347 157/300 Read block 598 len 16384
30348 158/300 Write block 775 len 20480 data: 4 5 6 5 5
30349 159/300 Read block 3512 len 36864
30350 160/300 Read block 3374 len 32768
30351 161/300 Read block 4850 len 32768
30352 162/300 Read block 419 len 16384
30353 163/300 Write block 3154 len 36864 data: 4 4 3 3 3 3 4 5 5
30354 164/300 Read block 1359 len 24576
30355 165/300 Read block 3482 len 32768
30356 166/300 Read block 212 len 12288
30357 167/300 Read block 390 len 36864
30358 168/300 Read block 2873 len 20480
30359 169/300 Write block 3645 len 36864 data: 6 6 6 6 6 5 3 6 6
30360 170/300 Write block 147 len 8192 data: 4 3
30361 171/300 Read block 1196 len 28672
30362 172/300 Read block 1682 len 40960
30363 173/300 Write block 3142 len 4096 data: 8
30364 174/300 Read block 663 len 24576
30365 175/300 Read block 2840 len 24576
30366 176/300 Read block 4769 len 40960
30367 177/300 Read block 3525 len 40960
30368 178/300 Read block 2756 len 16384
30369 179/300 Read block 1737 len 40960
30370 180/300 Write block 3193 len 40960 data: 5 6 6 4 4 4 3 4 4 5
30371 181/300 Read block 74 len 24576
30372 182/300 Read block 1927 len 36864
30373 183/300 Write block 1691 len 8192 data: 5 6
30374 184/300 Write block 3199 len 24576 data: 4 5 5 6 4 4
30375 185/300 Flush
30376 186/300 Write block 86 len 4096 data: 4
30377 187/300 Read block 1166 len 20480
30378 188/300 Read block 3109 len 20480
30379 189/300 Read block 3016 len 12288
30380 190/300 Read block 4584 len 8192
30381 191/300 Read block 2872 len 40960
30382 192/300 Read block 4586 len 20480
30383 193/300 Read block 3007 len 8192
30384 194/300 Write block 4785 len 28672 data: 5 5 5 5 5 6 5
30385 195/300 Flush
30386 196/300 Write block 678 len 16384 data: 10 9 8 7
30387 197/300 Read block 1931 len 32768
30388 198/300 Write block 1394 len 40960 data: 5 4 5 5 6 8 6 5 5 5
30389 199/300 Write block 198 len 20480 data: 5 4 4 4 5
30390 200/300 Read block 4561 len 12288
30391 201/300 Write block 2273 len 40960 data: 4 6 7 8 10 10 9 9 9 11
30392 202/300 Write block 1106 len 20480 data: 4 3 3 5 4
30393 203/300 Write block 913 len 12288 data: 5 6 5
30394 204/300 Write block 4009 len 36864 data: 8 8 8 8 8 9 10 9 10
30395 205/300 Write block 1389 len 12288 data: 5 6 6
30396 206/300 Write block 4027 len 20480 data: 2 2 2 4 4
30397 207/300 Write block 1324 len 40960 data: 5 4 4 5 3 5 5 6 6 4
30398 208/300 Read block 109 len 28672
30399 209/300 Read block 4291 len 28672
30400 210/300 Write block 3290 len 20480 data: 6 6 4 4 4
30401 211/300 Flush
30402 212/300 Read block 2980 len 16384
30403 213/300 Read block 2564 len 24576
30404 214/300 Flush
30405 215/300 Write block 3752 len 40960 data: 6 5 6 6 5 4 5 4 4 5
30406 216/300 Read block 1967 len 16384
30407 217/300 Write block 4086 len 8192 data: 5 4
30408 218/300 Write block 3293 len 32768 data: 5 5 3 2 2 3 3 4
30409 219/300 Write block 3248 len 8192 data: 9 10
30410 220/300 Write block 269 len 4096 data: 3
30411 221/300 Write block 2728 len 12288 data: 5 5 5
30412 222/300 Read block 1014 len 32768
30413 223/300 Write block 4566 len 36864 data: 7 6 8 9 10 8 8 8 5
30414 224/300 Read block 165 len 24576
30415 225/300 Write block 24 len 20480 data: 4 5 4 4 6
30416 226/300 Read block 642 len 28672
30417 227/300 Read block 179 len 36864
30418 228/300 Read block 3721 len 40960
30419 229/300 Write block 4781 len 12288 data: 4 5 4
30420 230/300 Read block 1812 len 28672
30421 231/300 Read block 25 len 24576
30422 232/300 Read block 4523 len 28672
30423 233/300 Write block 3616 len 8192 data: 2 2
30424 234/300 Write block 1942 len 32768 data: 4 4 5 6 7 7 6 5
30425 235/300 Write block 33 len 20480 data: 5 5 5 5 4
30426 236/300 Write block 2864 len 20480 data: 5 7 8 6 6
30427 237/300 Flush
30428 238/300 Write block 2014 len 24576 data: 6 7 5 6 6 6
30429 239/300 Write block 4837 len 36864 data: 7 8 6 5 4 3 4 3 2
30430 240/300 Read block 4089 len 40960
30431 241/300 Read block 340 len 32768
30432 242/300 Read block 1247 len 20480
30433 243/300 Read block 3036 len 20480
30434 244/300 Write block 20 len 36864 data: 4 2 2 2 5 6 5 5 7
30435 245/300 Read block 1910 len 28672
30436 246/300 Read block 4930 len 32768
30437 247/300 Read block 3071 len 40960
30438 248/300 Write block 4643 len 8192 data: 4 5
30439 249/300 Write block 2935 len 36864 data: 4 4 5 5 8 6 6 5 4
30440 250/300 Write block 1167 len 36864 data: 5 5 6 9 8 8 8 8 7
30441 251/300 Read block 1591 len 40960
30442 252/300 Write block 4314 len 12288 data: 5 3 3
30443 253/300 Write block 3096 len 12288 data: 6 7 7
30444 254/300 Read block 4073 len 20480
30445 255/300 Read block 1331 len 32768
30446 256/300 Read block 4023 len 32768
30447 257/300 Write block 1757 len 32768 data: 5 7 7 7 7 10 7 5
30448 258/300 Write block 255 len 8192 data: 5 5
30449 259/300 Read block 3384 len 24576
30450 260/300 Read block 823 len 12288
30451 261/300 Read block 4844 len 12288
30452 262/300 Write block 492 len 12288 data: 4 3 2
30453 263/300 Write block 502 len 12288 data: 7 9 7
30454 264/300 Write block 1160 len 20480 data: 4 3 3 4 5
30455 265/300 Read block 4534 len 16384
30456 266/300 Read block 4192 len 8192
30457 267/300 Read block 2620 len 8192
30458 268/300 Write block 1404 len 8192 data: 5 6
30459 269/300 Write block 3662 len 8192 data: 5 4
30460 270/300 Write block 3840 len 40960 data: 8 9 9 8 7 6 6 6 5 4
30461 271/300 Read block 2375 len 8192
30462 272/300 Read block 4935 len 24576
30463 273/300 Read block 1985 len 20480
30464 274/300 Read block 1242 len 20480
30465 275/300 Write block 104 len 36864 data: 7 8 7 5 5 5 5 4 4
30466 276/300 Read block 3130 len 32768
30467 277/300 Write block 1665 len 20480 data: 8 8 7 6 5
30468 278/300 Write block 4888 len 36864 data: 5 5 5 4 3 2 2 3 4
30469 279/300 Read block 3813 len 8192
30470 280/300 Read block 386 len 8192
30471 281/300 Write block 87 len 24576 data: 4 4 5 5 5 7
30472 282/300 Write block 2672 len 40960 data: 7 6 6 6 7 7 10 10 10 10
30473 283/300 Write block 3059 len 32768 data: 6 7 8 9 8 8 8 8
30474 284/300 Write block 1629 len 12288 data: 5 5 5
30475 285/300 Flush
30476 286/300 Read block 3085 len 16384
30477 287/300 Write block 1695 len 16384 data: 2 2 3 4
30478 288/300 Write block 3016 len 36864 data: 3 4 6 7 6 7 7 5 5
30479 289/300 Read block 2042 len 24576
30480 290/300 Read block 2360 len 40960
30481 291/300 Read block 3756 len 8192
30482 292/300 Write block 3695 len 32768 data: 3 5 5 5 6 7 7 8
30483 293/300 Write block 3360 len 12288 data: 3 3 2
30484 294/300 Flush
30485 295/300 Read block 3812 len 40960
30486 296/300 Write block 4453 len 36864 data: 7 6 4 3 3 3 3 4 4
30487 297/300 Read block 3570 len 20480
30488 298/300 Read block 4744 len 32768
30489 299/300 Write block 3720 len 24576 data: 4 4 5 5 4 4
30490 300/300 Write block 3886 len 24576 data: 6 5 4 4 4 3
30491 301/300 Read block 1656 len 40960
30492 Main task has work to do, go find it
30493 got dsc Start(0)
30494 start 0
30495 Feb 21 06:52:06.083 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/0, method: POST, req_id: e06d1625-4ae9-4154-b05c-e89d3a854fec, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
30496 Replay: started 0, returned:Ok(())
30497 ----------------------------------------------------------------
30498 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30499 States: Offline Active Active
30500 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30501 8848 Acked Flush 0 Sent Done Done false
30502 8849 Acked Flush 0 Sent Done Done false
30503 8850 Acked Read 3 Sent Done Done false
30504 8851 Acked Read 5 Sent Done Done false
30505 8852 Acked Write 9 Sent Done Done false
30506 8853 Acked Read 5 Sent Done Done false
30507 8854 Acked Read 5 Sent Done Done false
30508 8855 Acked Read 4 Sent Done Done false
30509 8856 Acked Flush 0 Sent Done Done false
30510 8857 Acked Read 4 Sent Done Done false
30511 8858 Acked Read 5 Sent Done Done false
30512 8859 Acked Read 8 Sent Done Done false
30513 8860 Acked Read 1 Sent Done Done false
30514 8861 Acked Read 10 Sent Done Done false
30515 8862 Acked Flush 0 Sent Done Done false
30516 8863 Acked Read 7 Sent Done Done false
30517 8864 Acked Write 5 Sent Done Done false
30518 8865 Acked Flush 0 Sent Done Done false
30519 8866 Acked Read 4 Sent Done Done false
30520 8867 Acked Flush 0 Sent Done Done false
30521 8868 Acked Read 1 Sent Done Done false
30522 8869 Acked Read 2 Sent Done Done false
30523 8870 Acked Read 10 Sent Done Done false
30524 8871 Acked Write 10 Sent Done Done false
30525 8872 Acked Read 1 Sent Done Done false
30526 8873 Acked Write 7 Sent Done Done false
30527 8874 Acked Write 6 Sent Done Done false
30528 8875 Acked Read 2 Sent Done Done false
30529 8876 Acked Write 6 Sent Done Done false
30530 8877 Acked Read 8 Sent Done Done false
30531 8878 Acked Write 4 Sent Done Done false
30532 8879 Acked Flush 0 Sent Done Done false
30533 8880 Acked Read 5 Sent Done Done false
30534 8881 Acked Read 10 Sent Done Done false
30535 8882 Acked Read 4 Sent Done Done false
30536 8883 Acked Read 7 Sent Done Done false
30537 8884 Acked Read 5 Sent Done Done false
30538 8885 Acked Write 8 Sent Done Done false
30539 8886 Acked Flush 0 Sent Done Done false
30540 8887 Acked Read 3 Sent Done Done false
30541 8888 Acked Write 5 Sent Done Done false
30542 8889 Acked Flush 0 Sent Done Done false
30543 8890 Acked Write 9 Sent Done Done false
30544 8891 Acked Write 2 Sent Done Done false
30545 8892 Acked Read 2 Sent Done Done false
30546 8893 Acked Write 1 Sent Done Done false
30547 8894 Acked Write 9 Sent Done Done false
30548 8895 Acked Read 9 Sent Done Done false
30549 8896 Acked Read 1 Sent Done Done false
30550 8897 Acked Write 6 Sent Done Done false
30551 8898 Acked Read 10 Sent Done Done false
30552 8899 Acked Write 1 Sent Done Done false
30553 8900 Acked Read 5 Sent Done Done false
30554 8901 Acked Read 8 Sent Done Done false
30555 8902 Acked Write 5 Sent Done Done false
30556 8903 Acked Read 10 Sent Done Done false
30557 8904 Acked Write 5 Sent Done Done false
30558 8905 Acked Write 4 Sent Done Done false
30559 8906 Acked Flush 0 Sent Done Done false
30560 8907 Acked Write 6 Sent Done Done false
30561 8908 Acked Flush 0 Sent Done Done false
30562 8909 Acked Read 4 Sent Done Done false
30563 8910 Acked Write 4 Sent Done Done false
30564 8911 Acked Write 1 Sent Done Done false
30565 8912 Acked Write 8 Sent Done Done false
30566 8913 Acked Write 6 Sent Done Done false
30567 8914 Acked Write 1 Sent Done Done false
30568 8915 Acked Read 6 Sent Done Done false
30569 8916 Acked Read 6 Sent Done Done false
30570 8917 Acked Write 6 Sent Done Done false
30571 8918 Acked Write 6 Sent Done Done false
30572 8919 Acked Write 4 Sent Done Done false
30573 8920 Acked Read 10 Sent Done Done false
30574 8921 Acked Flush 0 Sent Done Done false
30575 8922 Acked Write 3 Sent Done Done false
30576 8923 Acked Read 3 Sent Done Done false
30577 8924 Acked Write 4 Sent Done Done false
30578 8925 Acked Read 3 Sent Done Done false
30579 8926 Acked Write 1 Sent Done Done false
30580 8927 Acked Read 2 Sent Done Done false
30581 8928 Acked Flush 0 Sent Done Done false
30582 8929 Acked Read 8 Sent Done Done false
30583 8930 Acked Read 10 Sent Done Done false
30584 8931 Acked Read 4 Sent Done Done false
30585 8932 Acked Write 10 Sent Done Done false
30586 8933 Acked Write 7 Sent Done Done false
30587 8934 Acked Write 3 Sent Done Done false
30588 8935 Acked Read 5 Sent Done Done false
30589 8936 Acked Write 1 Sent Done Done false
30590 8937 Acked Write 7 Sent Done Done false
30591 8938 Acked Read 1 Sent Done Done false
30592 8939 Acked Write 2 Sent Done Done false
30593 8940 Acked Flush 0 Sent Done Done false
30594 8941 Acked Read 7 Sent Done Done false
30595 8942 Acked Write 7 Sent Done Done false
30596 8943 Acked Read 10 Sent Done Done false
30597 8944 Acked Read 8 Sent Done Done false
30598 8945 Acked Read 9 Sent Done Done false
30599 8946 Acked Read 3 Sent Done Done false
30600 8947 Acked Write 3 Sent Done Done false
30601 8948 Acked Write 2 Sent Done Done false
30602 8949 Acked Read 9 Sent Done Done false
30603 8950 Acked Write 3 Sent Done Done false
30604 8951 Acked Write 5 Sent Done Done false
30605 8952 Acked Write 8 Sent Done Done false
30606 8953 Acked Read 9 Sent Done Done false
30607 8954 Acked Read 4 Sent Done Done false
30608 8955 Acked Write 7 Sent Done Done false
30609 8956 Acked Write 9 Sent Done Done false
30610 8957 Acked Write 10 Sent Done Done false
30611 8958 Acked Read 10 Sent Done Done false
30612 8959 Acked Read 10 Sent Done Done false
30613 8960 Acked Write 9 Sent Done Done false
30614 8961 Acked Read 7 Sent Done Done false
30615 8962 Acked Write 8 Sent Done Done false
30616 8963 Acked Write 2 Sent Done Done false
30617 8964 Acked Read 10 Sent Done Done false
30618 8965 Acked Read 8 Sent Done Done false
30619 8966 Acked Write 8 Sent Done Done false
30620 8967 Acked Write 6 Sent Done Done false
30621 8968 Acked Write 10 Sent Done Done false
30622 8969 Acked Read 3 Sent Done Done false
30623 8970 Acked Write 1 Sent Done Done false
30624 8971 Acked Read 5 Sent Done Done false
30625 8972 Acked Flush 0 Sent Done Done false
30626 8973 Acked Write 4 Sent Done Done false
30627 8974 Acked Write 1 Sent Done Done false
30628 8975 Acked Flush 0 Sent Done Done false
30629 8976 Acked Read 2 Sent Done Done false
30630 8977 Acked Read 5 Sent Done Done false
30631 8978 Acked Read 9 Sent Done Done false
30632 8979 Acked Read 3 Sent Done Done false
30633 8980 Acked Read 8 Sent Done Done false
30634 8981 Acked Read 9 Sent Done Done false
30635 8982 Acked Write 10 Sent Done Done false
30636 8983 Acked Write 2 Sent Done Done false
30637 8984 Acked Flush 0 Sent Done Done false
30638 8985 Acked Read 8 Sent Done Done false
30639 8986 Acked Write 3 Sent Done Done false
30640 8987 Acked Write 5 Sent Done Done false
30641 8988 Acked Read 2 Sent Done Done false
30642 8989 Acked Write 10 Sent Done Done false
30643 8990 Acked Write 3 Sent Done Done false
30644 8991 Acked Read 10 Sent Done Done false
30645 8992 Acked Read 3 Sent Done Done false
30646 8993 Acked Read 2 Sent Done Done false
30647 8994 Acked Read 10 Sent Done Done false
30648 8995 Acked Read 10 Sent Done Done false
30649 8996 Acked Write 5 Sent Done Done false
30650 8997 Acked Read 6 Sent Done Done false
30651 8998 Acked Read 8 Sent Done Done false
30652 8999 Acked Read 7 Sent Done Done false
30653 9000 Acked Write 3 Sent Done Done false
30654 9001 Acked Read 2 Sent Done Done false
30655 9002 Acked Write 3 Sent Done Done false
30656 9003 Acked Read 3 Sent Done Done false
30657 9004 Acked Read 4 Sent Done Done false
30658 9005 Acked Write 5 Sent Done Done false
30659 9006 Acked Read 9 Sent Done Done false
30660 9007 Acked Read 8 Sent Done Done false
30661 9008 Acked Read 8 Sent Done Done false
30662 9009 Acked Read 4 Sent Done Done false
30663 9010 Acked Write 9 Sent Done Done false
30664 9011 Acked Read 6 Sent Done Done false
30665 9012 Acked Read 8 Sent Done Done false
30666 9013 Acked Read 3 Sent Done Done false
30667 9014 Acked Read 9 Sent Done Done false
30668 9015 Acked Read 5 Sent Done Done false
30669 9016 Acked Write 9 Sent Done Done false
30670 9017 Acked Write 2 Sent Done Done false
30671 9018 Acked Read 7 Sent Done Done false
30672 9019 Acked Read 10 Sent Done Done false
30673 9020 Acked Write 1 Sent Done Done false
30674 9021 Acked Read 6 Sent Done Done false
30675 9022 Acked Read 6 Sent Done Done false
30676 9023 Acked Read 10 Sent Done Done false
30677 9024 Acked Read 10 Sent Done Done false
30678 9025 Acked Read 4 Sent Done Done false
30679 9026 Acked Read 10 Sent Done Done false
30680 9027 Acked Write 10 Sent Done Done false
30681 9028 Acked Read 6 Sent Done Done false
30682 9029 Acked Read 9 Sent Done Done false
30683 9030 Acked Write 2 Sent Done Done false
30684 9031 Acked Write 6 Sent Done Done false
30685 9032 Acked Flush 0 Sent Done Done false
30686 9033 Acked Write 1 Sent Done Done false
30687 9034 Acked Read 5 Sent Done Done false
30688 9035 Acked Read 5 Sent Done Done false
30689 9036 Acked Read 3 Sent Done Done false
30690 9037 Acked Read 2 Sent Done Done false
30691 9038 Acked Read 10 Sent Done Done false
30692 9039 Acked Read 5 Sent Done Done false
30693 9040 Acked Read 2 Sent Done Done false
30694 9041 Acked Write 7 Sent Done Done false
30695 9042 Acked Flush 0 Sent Done Done false
30696 9043 Acked Write 4 Sent Done Done false
30697 9044 Acked Read 8 Sent Done Done false
30698 9045 Acked Write 10 Sent Done Done false
30699 9046 Acked Write 5 Sent Done Done false
30700 9047 Acked Read 3 Sent Done Done false
30701 9048 Acked Write 10 Sent Done Done false
30702 9049 Acked Write 5 Sent Done Done false
30703 9050 Acked Write 3 Sent Done Done false
30704 9051 Acked Write 9 Sent Done Done false
30705 9052 Acked Write 3 Sent Done Done false
30706 9053 Acked Write 5 Sent Done Done false
30707 9054 Acked Write 10 Sent Done Done false
30708 9055 Acked Read 7 Sent Done Done false
30709 9056 Acked Read 7 Sent Done Done false
30710 9057 Acked Write 5 Sent Done Done false
30711 9058 Acked Flush 0 Sent Done Done false
30712 9059 Acked Read 4 Sent Done Done false
30713 9060 Acked Read 6 Sent Done Done false
30714 9061 Acked Flush 0 Sent Done Done false
30715 9062 Acked Write 10 Sent Done Done false
30716 9063 Acked Read 4 Sent Done Done false
30717 9064 Acked Write 2 Sent Done Done false
30718 9065 Acked Write 8 Sent Done Done false
30719 9066 Acked Write 2 Sent Done Done false
30720 9067 Acked Write 1 Sent Done Done false
30721 9068 Acked Write 3 Sent Done Done false
30722 9069 Acked Read 8 Sent Done Done false
30723 9070 Acked Write 9 Sent Done Done false
30724 9071 Acked Read 6 Sent Done Done false
30725 9072 Acked Write 5 Sent Done Done false
30726 9073 Acked Read 7 Sent Done Done false
30727 9074 Acked Read 9 Sent Done Done false
30728 9075 Acked Read 10 Sent Done Done false
30729 9076 Acked Write 3 Sent Done Done false
30730 9077 Acked Read 7 Sent Done Done false
30731 9078 Acked Read 6 Sent Done Done false
30732 9079 Acked Read 7 Sent Done Done false
30733 9080 Acked Write 2 Sent Done Done false
30734 9081 Acked Write 8 Sent Done Done false
30735 9082 Acked Write 5 Sent Done Done false
30736 9083 Acked Write 5 Sent Done Done false
30737 9084 Acked Flush 0 Sent Done Done false
30738 9085 Acked Write 6 Sent Done Done false
30739 9086 Acked Write 9 Sent Done Done false
30740 9087 Acked Read 10 Sent Done Done false
30741 9088 Acked Read 8 Sent Done Done false
30742 9089 Acked Read 5 Sent Done Done false
30743 9090 Acked Read 5 Sent Done Done false
30744 9091 Acked Write 9 Sent Done Done false
30745 9092 Acked Read 7 Sent Done Done false
30746 9093 Acked Read 8 Sent Done Done false
30747 9094 Acked Read 10 Sent Done Done false
30748 9095 Acked Write 2 Sent Done Done false
30749 9096 Acked Write 9 Sent Done Done false
30750 9097 Acked Write 9 Sent Done Done false
30751 9098 Acked Read 10 Sent Done Done false
30752 9099 Acked Write 3 Sent Done Done false
30753 9100 Acked Write 3 Sent Done Done false
30754 9101 Acked Read 5 Sent Done Done false
30755 9102 Acked Read 8 Sent Done Done false
30756 9103 Acked Read 8 Sent Done Done false
30757 9104 Acked Write 8 Sent Done Done false
30758 9105 Acked Write 2 Sent Done Done false
30759 9106 Acked Read 6 Sent Done Done false
30760 9107 Acked Read 3 Sent Done Done false
30761 9108 Acked Read 3 Sent Done Done false
30762 9109 Acked Write 3 Sent Done Done false
30763 9110 Acked Write 3 Sent Done Done false
30764 9111 Acked Write 5 Sent Done Done false
30765 9112 Acked Read 4 Sent Done Done false
30766 9113 Acked Read 2 Sent Done Done false
30767 9114 Acked Read 2 Sent Done Done false
30768 9115 Acked Write 2 Sent Done Done false
30769 9116 Acked Write 2 Sent Done Done false
30770 9117 Acked Write 10 Sent Done Done false
30771 9118 Acked Read 2 Sent Done Done false
30772 9119 Acked Read 6 Sent Done Done false
30773 9120 Acked Read 5 Sent Done Done false
30774 9121 Acked Read 5 Sent Done Done false
30775 9122 Acked Write 9 Sent Done Done false
30776 9123 Acked Read 8 Sent Done Done false
30777 9124 Acked Write 5 Sent Done Done false
30778 9125 Acked Write 9 Sent Done Done false
30779 9126 Acked Read 2 Sent Done Done false
30780 9127 Acked Read 2 Sent Done Done false
30781 9128 Acked Write 6 Sent Done Done false
30782 9129 Acked Write 10 Sent Done Done false
30783 9130 Acked Write 8 Sent Done Done false
30784 9131 Acked Write 3 Sent Done Done false
30785 9132 Acked Flush 0 Sent Done Done false
30786 9133 Acked Read 4 Sent Done Done false
30787 9134 Acked Write 4 Sent Done Done false
30788 9135 Acked Write 9 Sent Done Done false
30789 9136 Acked Read 6 Sent Done Done false
30790 9137 Acked Read 10 Sent Done Done false
30791 9138 Acked Read 2 Sent Done Done false
30792 9139 Acked Write 8 Sent Done Done false
30793 9140 Acked Write 3 Sent Done Done false
30794 9141 Acked Flush 0 Sent Done Done false
30795 9142 Acked Read 10 Sent Done Done false
30796 9143 Acked Write 9 Sent Done Done false
30797 9144 Acked Read 5 Sent Done Done false
30798 9145 Acked Read 8 Sent Done Done false
30799 9146 Acked Write 6 Sent Done Done false
30800 9147 Acked Write 6 Sent Done Done false
30801 9148 Acked Read 10 Sent Done Done false
30802 STATES DS:0 DS:1 DS:2 TOTAL
30803 Sent 301 0 0 301
30804 Done 0 301 301 602
30805 Skipped 0 0 0 0
30806 Error 0 0 0 0
30807 Last Flush: Some(JobId(8847)) Some(JobId(9141)) Some(JobId(9141))
30808 Downstairs last five completed: 8847 8846 8845 8844 8843
30809 Upstairs last five completed: 9148 9147 9146 9145 9144
30810 CLIENT: Up:0 ds:301 act:2
308112025-02-21T06:52:07.279ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
308122025-02-21T06:52:08.278ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
308132025-02-21T06:52:09.281ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
308142025-02-21T06:52:09.781ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
30815 ----------------------------------------------------------------
30816 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30817 States: Offline Active Active
30818 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30819 8848 Acked Flush 0 Sent Done Done false
30820 8849 Acked Flush 0 Sent Done Done false
30821 8850 Acked Read 3 Sent Done Done false
30822 8851 Acked Read 5 Sent Done Done false
30823 8852 Acked Write 9 Sent Done Done false
30824 8853 Acked Read 5 Sent Done Done false
30825 8854 Acked Read 5 Sent Done Done false
30826 8855 Acked Read 4 Sent Done Done false
30827 8856 Acked Flush 0 Sent Done Done false
30828 8857 Acked Read 4 Sent Done Done false
30829 8858 Acked Read 5 Sent Done Done false
30830 8859 Acked Read 8 Sent Done Done false
30831 8860 Acked Read 1 Sent Done Done false
30832 8861 Acked Read 10 Sent Done Done false
30833 8862 Acked Flush 0 Sent Done Done false
30834 8863 Acked Read 7 Sent Done Done false
30835 8864 Acked Write 5 Sent Done Done false
30836 8865 Acked Flush 0 Sent Done Done false
30837 8866 Acked Read 4 Sent Done Done false
30838 8867 Acked Flush 0 Sent Done Done false
30839 8868 Acked Read 1 Sent Done Done false
30840 8869 Acked Read 2 Sent Done Done false
30841 8870 Acked Read 10 Sent Done Done false
30842 8871 Acked Write 10 Sent Done Done false
30843 8872 Acked Read 1 Sent Done Done false
30844 8873 Acked Write 7 Sent Done Done false
30845 8874 Acked Write 6 Sent Done Done false
30846 8875 Acked Read 2 Sent Done Done false
30847 8876 Acked Write 6 Sent Done Done false
30848 8877 Acked Read 8 Sent Done Done false
30849 8878 Acked Write 4 Sent Done Done false
30850 8879 Acked Flush 0 Sent Done Done false
30851 8880 Acked Read 5 Sent Done Done false
30852 8881 Acked Read 10 Sent Done Done false
30853 8882 Acked Read 4 Sent Done Done false
30854 8883 Acked Read 7 Sent Done Done false
30855 8884 Acked Read 5 Sent Done Done false
30856 8885 Acked Write 8 Sent Done Done false
30857 8886 Acked Flush 0 Sent Done Done false
30858 8887 Acked Read 3 Sent Done Done false
30859 8888 Acked Write 5 Sent Done Done false
30860 8889 Acked Flush 0 Sent Done Done false
30861 8890 Acked Write 9 Sent Done Done false
30862 8891 Acked Write 2 Sent Done Done false
30863 8892 Acked Read 2 Sent Done Done false
30864 8893 Acked Write 1 Sent Done Done false
30865 8894 Acked Write 9 Sent Done Done false
30866 8895 Acked Read 9 Sent Done Done false
30867 8896 Acked Read 1 Sent Done Done false
30868 8897 Acked Write 6 Sent Done Done false
30869 8898 Acked Read 10 Sent Done Done false
30870 8899 Acked Write 1 Sent Done Done false
30871 8900 Acked Read 5 Sent Done Done false
30872 8901 Acked Read 8 Sent Done Done false
30873 8902 Acked Write 5 Sent Done Done false
30874 8903 Acked Read 10 Sent Done Done false
30875 8904 Acked Write 5 Sent Done Done false
30876 8905 Acked Write 4 Sent Done Done false
30877 8906 Acked Flush 0 Sent Done Done false
30878 8907 Acked Write 6 Sent Done Done false
30879 8908 Acked Flush 0 Sent Done Done false
30880 8909 Acked Read 4 Sent Done Done false
30881 8910 Acked Write 4 Sent Done Done false
30882 8911 Acked Write 1 Sent Done Done false
30883 8912 Acked Write 8 Sent Done Done false
30884 8913 Acked Write 6 Sent Done Done false
30885 8914 Acked Write 1 Sent Done Done false
30886 8915 Acked Read 6 Sent Done Done false
30887 8916 Acked Read 6 Sent Done Done false
30888 8917 Acked Write 6 Sent Done Done false
30889 8918 Acked Write 6 Sent Done Done false
30890 8919 Acked Write 4 Sent Done Done false
30891 8920 Acked Read 10 Sent Done Done false
30892 8921 Acked Flush 0 Sent Done Done false
30893 8922 Acked Write 3 Sent Done Done false
30894 8923 Acked Read 3 Sent Done Done false
30895 8924 Acked Write 4 Sent Done Done false
30896 8925 Acked Read 3 Sent Done Done false
30897 8926 Acked Write 1 Sent Done Done false
30898 8927 Acked Read 2 Sent Done Done false
30899 8928 Acked Flush 0 Sent Done Done false
30900 8929 Acked Read 8 Sent Done Done false
30901 8930 Acked Read 10 Sent Done Done false
30902 8931 Acked Read 4 Sent Done Done false
30903 8932 Acked Write 10 Sent Done Done false
30904 8933 Acked Write 7 Sent Done Done false
30905 8934 Acked Write 3 Sent Done Done false
30906 8935 Acked Read 5 Sent Done Done false
30907 8936 Acked Write 1 Sent Done Done false
30908 8937 Acked Write 7 Sent Done Done false
30909 8938 Acked Read 1 Sent Done Done false
30910 8939 Acked Write 2 Sent Done Done false
30911 8940 Acked Flush 0 Sent Done Done false
30912 8941 Acked Read 7 Sent Done Done false
30913 8942 Acked Write 7 Sent Done Done false
30914 8943 Acked Read 10 Sent Done Done false
30915 8944 Acked Read 8 Sent Done Done false
30916 8945 Acked Read 9 Sent Done Done false
30917 8946 Acked Read 3 Sent Done Done false
30918 8947 Acked Write 3 Sent Done Done false
30919 8948 Acked Write 2 Sent Done Done false
30920 8949 Acked Read 9 Sent Done Done false
30921 8950 Acked Write 3 Sent Done Done false
30922 8951 Acked Write 5 Sent Done Done false
30923 8952 Acked Write 8 Sent Done Done false
30924 8953 Acked Read 9 Sent Done Done false
30925 8954 Acked Read 4 Sent Done Done false
30926 8955 Acked Write 7 Sent Done Done false
30927 8956 Acked Write 9 Sent Done Done false
30928 8957 Acked Write 10 Sent Done Done false
30929 8958 Acked Read 10 Sent Done Done false
30930 8959 Acked Read 10 Sent Done Done false
30931 8960 Acked Write 9 Sent Done Done false
30932 8961 Acked Read 7 Sent Done Done false
30933 8962 Acked Write 8 Sent Done Done false
30934 8963 Acked Write 2 Sent Done Done false
30935 8964 Acked Read 10 Sent Done Done false
30936 8965 Acked Read 8 Sent Done Done false
30937 8966 Acked Write 8 Sent Done Done false
30938 8967 Acked Write 6 Sent Done Done false
30939 8968 Acked Write 10 Sent Done Done false
30940 8969 Acked Read 3 Sent Done Done false
30941 8970 Acked Write 1 Sent Done Done false
30942 8971 Acked Read 5 Sent Done Done false
30943 8972 Acked Flush 0 Sent Done Done false
30944 8973 Acked Write 4 Sent Done Done false
30945 8974 Acked Write 1 Sent Done Done false
30946 8975 Acked Flush 0 Sent Done Done false
30947 8976 Acked Read 2 Sent Done Done false
30948 8977 Acked Read 5 Sent Done Done false
30949 8978 Acked Read 9 Sent Done Done false
30950 8979 Acked Read 3 Sent Done Done false
30951 8980 Acked Read 8 Sent Done Done false
30952 8981 Acked Read 9 Sent Done Done false
30953 8982 Acked Write 10 Sent Done Done false
30954 8983 Acked Write 2 Sent Done Done false
30955 8984 Acked Flush 0 Sent Done Done false
30956 8985 Acked Read 8 Sent Done Done false
30957 8986 Acked Write 3 Sent Done Done false
30958 8987 Acked Write 5 Sent Done Done false
30959 8988 Acked Read 2 Sent Done Done false
30960 8989 Acked Write 10 Sent Done Done false
30961 8990 Acked Write 3 Sent Done Done false
30962 8991 Acked Read 10 Sent Done Done false
30963 8992 Acked Read 3 Sent Done Done false
30964 8993 Acked Read 2 Sent Done Done false
30965 8994 Acked Read 10 Sent Done Done false
30966 8995 Acked Read 10 Sent Done Done false
30967 8996 Acked Write 5 Sent Done Done false
30968 8997 Acked Read 6 Sent Done Done false
30969 8998 Acked Read 8 Sent Done Done false
30970 8999 Acked Read 7 Sent Done Done false
30971 9000 Acked Write 3 Sent Done Done false
30972 9001 Acked Read 2 Sent Done Done false
30973 9002 Acked Write 3 Sent Done Done false
30974 9003 Acked Read 3 Sent Done Done false
30975 9004 Acked Read 4 Sent Done Done false
30976 9005 Acked Write 5 Sent Done Done false
30977 9006 Acked Read 9 Sent Done Done false
30978 9007 Acked Read 8 Sent Done Done false
30979 9008 Acked Read 8 Sent Done Done false
30980 9009 Acked Read 4 Sent Done Done false
30981 9010 Acked Write 9 Sent Done Done false
30982 9011 Acked Read 6 Sent Done Done false
30983 9012 Acked Read 8 Sent Done Done false
30984 9013 Acked Read 3 Sent Done Done false
30985 9014 Acked Read 9 Sent Done Done false
30986 9015 Acked Read 5 Sent Done Done false
30987 9016 Acked Write 9 Sent Done Done false
30988 9017 Acked Write 2 Sent Done Done false
30989 9018 Acked Read 7 Sent Done Done false
30990 9019 Acked Read 10 Sent Done Done false
30991 9020 Acked Write 1 Sent Done Done false
30992 9021 Acked Read 6 Sent Done Done false
30993 9022 Acked Read 6 Sent Done Done false
30994 9023 Acked Read 10 Sent Done Done false
30995 9024 Acked Read 10 Sent Done Done false
30996 9025 Acked Read 4 Sent Done Done false
30997 9026 Acked Read 10 Sent Done Done false
30998 9027 Acked Write 10 Sent Done Done false
30999 9028 Acked Read 6 Sent Done Done false
31000 9029 Acked Read 9 Sent Done Done false
31001 9030 Acked Write 2 Sent Done Done false
31002 9031 Acked Write 6 Sent Done Done false
31003 9032 Acked Flush 0 Sent Done Done false
31004 9033 Acked Write 1 Sent Done Done false
31005 9034 Acked Read 5 Sent Done Done false
31006 9035 Acked Read 5 Sent Done Done false
31007 9036 Acked Read 3 Sent Done Done false
31008 9037 Acked Read 2 Sent Done Done false
31009 9038 Acked Read 10 Sent Done Done false
31010 9039 Acked Read 5 Sent Done Done false
31011 9040 Acked Read 2 Sent Done Done false
31012 9041 Acked Write 7 Sent Done Done false
31013 9042 Acked Flush 0 Sent Done Done false
31014 9043 Acked Write 4 Sent Done Done false
31015 9044 Acked Read 8 Sent Done Done false
31016 9045 Acked Write 10 Sent Done Done false
31017 9046 Acked Write 5 Sent Done Done false
31018 9047 Acked Read 3 Sent Done Done false
31019 9048 Acked Write 10 Sent Done Done false
31020 9049 Acked Write 5 Sent Done Done false
31021 9050 Acked Write 3 Sent Done Done false
31022 9051 Acked Write 9 Sent Done Done false
31023 9052 Acked Write 3 Sent Done Done false
31024 9053 Acked Write 5 Sent Done Done false
31025 9054 Acked Write 10 Sent Done Done false
31026 9055 Acked Read 7 Sent Done Done false
31027 9056 Acked Read 7 Sent Done Done false
31028 9057 Acked Write 5 Sent Done Done false
31029 9058 Acked Flush 0 Sent Done Done false
31030 9059 Acked Read 4 Sent Done Done false
31031 9060 Acked Read 6 Sent Done Done false
31032 9061 Acked Flush 0 Sent Done Done false
31033 9062 Acked Write 10 Sent Done Done false
31034 9063 Acked Read 4 Sent Done Done false
31035 9064 Acked Write 2 Sent Done Done false
31036 9065 Acked Write 8 Sent Done Done false
31037 9066 Acked Write 2 Sent Done Done false
31038 9067 Acked Write 1 Sent Done Done false
31039 9068 Acked Write 3 Sent Done Done false
31040 9069 Acked Read 8 Sent Done Done false
31041 9070 Acked Write 9 Sent Done Done false
31042 9071 Acked Read 6 Sent Done Done false
31043 9072 Acked Write 5 Sent Done Done false
31044 9073 Acked Read 7 Sent Done Done false
31045 9074 Acked Read 9 Sent Done Done false
31046 9075 Acked Read 10 Sent Done Done false
31047 9076 Acked Write 3 Sent Done Done false
31048 9077 Acked Read 7 Sent Done Done false
31049 9078 Acked Read 6 Sent Done Done false
31050 9079 Acked Read 7 Sent Done Done false
31051 9080 Acked Write 2 Sent Done Done false
31052 9081 Acked Write 8 Sent Done Done false
31053 9082 Acked Write 5 Sent Done Done false
31054 9083 Acked Write 5 Sent Done Done false
31055 9084 Acked Flush 0 Sent Done Done false
31056 9085 Acked Write 6 Sent Done Done false
31057 9086 Acked Write 9 Sent Done Done false
31058 9087 Acked Read 10 Sent Done Done false
31059 9088 Acked Read 8 Sent Done Done false
31060 9089 Acked Read 5 Sent Done Done false
31061 9090 Acked Read 5 Sent Done Done false
31062 9091 Acked Write 9 Sent Done Done false
31063 9092 Acked Read 7 Sent Done Done false
31064 9093 Acked Read 8 Sent Done Done false
31065 9094 Acked Read 10 Sent Done Done false
31066 9095 Acked Write 2 Sent Done Done false
31067 9096 Acked Write 9 Sent Done Done false
31068 9097 Acked Write 9 Sent Done Done false
31069 9098 Acked Read 10 Sent Done Done false
31070 9099 Acked Write 3 Sent Done Done false
31071 9100 Acked Write 3 Sent Done Done false
31072 9101 Acked Read 5 Sent Done Done false
31073 9102 Acked Read 8 Sent Done Done false
31074 9103 Acked Read 8 Sent Done Done false
31075 9104 Acked Write 8 Sent Done Done false
31076 9105 Acked Write 2 Sent Done Done false
31077 9106 Acked Read 6 Sent Done Done false
31078 9107 Acked Read 3 Sent Done Done false
31079 9108 Acked Read 3 Sent Done Done false
31080 9109 Acked Write 3 Sent Done Done false
31081 9110 Acked Write 3 Sent Done Done false
31082 9111 Acked Write 5 Sent Done Done false
31083 9112 Acked Read 4 Sent Done Done false
31084 9113 Acked Read 2 Sent Done Done false
31085 9114 Acked Read 2 Sent Done Done false
31086 9115 Acked Write 2 Sent Done Done false
31087 9116 Acked Write 2 Sent Done Done false
31088 9117 Acked Write 10 Sent Done Done false
31089 9118 Acked Read 2 Sent Done Done false
31090 9119 Acked Read 6 Sent Done Done false
31091 9120 Acked Read 5 Sent Done Done false
31092 9121 Acked Read 5 Sent Done Done false
31093 9122 Acked Write 9 Sent Done Done false
31094 9123 Acked Read 8 Sent Done Done false
31095 9124 Acked Write 5 Sent Done Done false
31096 9125 Acked Write 9 Sent Done Done false
31097 9126 Acked Read 2 Sent Done Done false
31098 9127 Acked Read 2 Sent Done Done false
31099 9128 Acked Write 6 Sent Done Done false
31100 9129 Acked Write 10 Sent Done Done false
31101 9130 Acked Write 8 Sent Done Done false
31102 9131 Acked Write 3 Sent Done Done false
31103 9132 Acked Flush 0 Sent Done Done false
31104 9133 Acked Read 4 Sent Done Done false
31105 9134 Acked Write 4 Sent Done Done false
31106 9135 Acked Write 9 Sent Done Done false
31107 9136 Acked Read 6 Sent Done Done false
31108 9137 Acked Read 10 Sent Done Done false
31109 9138 Acked Read 2 Sent Done Done false
31110 9139 Acked Write 8 Sent Done Done false
31111 9140 Acked Write 3 Sent Done Done false
31112 9141 Acked Flush 0 Sent Done Done false
31113 9142 Acked Read 10 Sent Done Done false
31114 9143 Acked Write 9 Sent Done Done false
31115 9144 Acked Read 5 Sent Done Done false
31116 9145 Acked Read 8 Sent Done Done false
31117 9146 Acked Write 6 Sent Done Done false
31118 9147 Acked Write 6 Sent Done Done false
31119 9148 Acked Read 10 Sent Done Done false
31120 STATES DS:0 DS:1 DS:2 TOTAL
31121 Sent 301 0 0 301
31122 Done 0 301 301 602
31123 Skipped 0 0 0 0
31124 Error 0 0 0 0
31125 Last Flush: Some(JobId(8847)) Some(JobId(9141)) Some(JobId(9141))
31126 Downstairs last five completed: 8847 8846 8845 8844 8843
31127 Upstairs last five completed: 9148 9147 9146 9145 9144
31128 CLIENT: Up:0 ds:301 act:2
311292025-02-21T06:52:10.588ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
311302025-02-21T06:52:11.285ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
311312025-02-21T06:52:11.833ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
311322025-02-21T06:52:11.833ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
311332025-02-21T06:52:11.833ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
311342025-02-21T06:52:11.833ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
311352025-02-21T06:52:11.833ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31136 [8810] got start action
31137 [8810] I am going to restart
31138 Starting downstairs at port 8810
31139 Make output file at "/tmp/dsc/downstairs-8810.txt"
31140 [8810][0] reports Starting
31141 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1233)
31142 [8810][0] reports Running
311432025-02-21T06:52:12.334ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
311442025-02-21T06:52:13.288ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31145 ----------------------------------------------------------------
31146 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31147 States: Offline Active Active
31148 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31149 8848 Acked Flush 0 Sent Done Done false
31150 8849 Acked Flush 0 Sent Done Done false
31151 8850 Acked Read 3 Sent Done Done false
31152 8851 Acked Read 5 Sent Done Done false
31153 8852 Acked Write 9 Sent Done Done false
31154 8853 Acked Read 5 Sent Done Done false
31155 8854 Acked Read 5 Sent Done Done false
31156 8855 Acked Read 4 Sent Done Done false
31157 8856 Acked Flush 0 Sent Done Done false
31158 8857 Acked Read 4 Sent Done Done false
31159 8858 Acked Read 5 Sent Done Done false
31160 8859 Acked Read 8 Sent Done Done false
31161 8860 Acked Read 1 Sent Done Done false
31162 8861 Acked Read 10 Sent Done Done false
31163 8862 Acked Flush 0 Sent Done Done false
31164 8863 Acked Read 7 Sent Done Done false
31165 8864 Acked Write 5 Sent Done Done false
31166 8865 Acked Flush 0 Sent Done Done false
31167 8866 Acked Read 4 Sent Done Done false
31168 8867 Acked Flush 0 Sent Done Done false
31169 8868 Acked Read 1 Sent Done Done false
31170 8869 Acked Read 2 Sent Done Done false
31171 8870 Acked Read 10 Sent Done Done false
31172 8871 Acked Write 10 Sent Done Done false
31173 8872 Acked Read 1 Sent Done Done false
31174 8873 Acked Write 7 Sent Done Done false
31175 8874 Acked Write 6 Sent Done Done false
31176 8875 Acked Read 2 Sent Done Done false
31177 8876 Acked Write 6 Sent Done Done false
31178 8877 Acked Read 8 Sent Done Done false
31179 8878 Acked Write 4 Sent Done Done false
31180 8879 Acked Flush 0 Sent Done Done false
31181 8880 Acked Read 5 Sent Done Done false
31182 8881 Acked Read 10 Sent Done Done false
31183 8882 Acked Read 4 Sent Done Done false
31184 8883 Acked Read 7 Sent Done Done false
31185 8884 Acked Read 5 Sent Done Done false
31186 8885 Acked Write 8 Sent Done Done false
31187 8886 Acked Flush 0 Sent Done Done false
31188 8887 Acked Read 3 Sent Done Done false
31189 8888 Acked Write 5 Sent Done Done false
31190 8889 Acked Flush 0 Sent Done Done false
31191 8890 Acked Write 9 Sent Done Done false
31192 8891 Acked Write 2 Sent Done Done false
31193 8892 Acked Read 2 Sent Done Done false
31194 8893 Acked Write 1 Sent Done Done false
31195 8894 Acked Write 9 Sent Done Done false
31196 8895 Acked Read 9 Sent Done Done false
31197 8896 Acked Read 1 Sent Done Done false
31198 8897 Acked Write 6 Sent Done Done false
31199 8898 Acked Read 10 Sent Done Done false
31200 8899 Acked Write 1 Sent Done Done false
31201 8900 Acked Read 5 Sent Done Done false
31202 8901 Acked Read 8 Sent Done Done false
31203 8902 Acked Write 5 Sent Done Done false
31204 8903 Acked Read 10 Sent Done Done false
31205 8904 Acked Write 5 Sent Done Done false
31206 8905 Acked Write 4 Sent Done Done false
31207 8906 Acked Flush 0 Sent Done Done false
31208 8907 Acked Write 6 Sent Done Done false
31209 8908 Acked Flush 0 Sent Done Done false
31210 8909 Acked Read 4 Sent Done Done false
31211 8910 Acked Write 4 Sent Done Done false
31212 8911 Acked Write 1 Sent Done Done false
31213 8912 Acked Write 8 Sent Done Done false
31214 8913 Acked Write 6 Sent Done Done false
31215 8914 Acked Write 1 Sent Done Done false
31216 8915 Acked Read 6 Sent Done Done false
31217 8916 Acked Read 6 Sent Done Done false
31218 8917 Acked Write 6 Sent Done Done false
31219 8918 Acked Write 6 Sent Done Done false
31220 8919 Acked Write 4 Sent Done Done false
31221 8920 Acked Read 10 Sent Done Done false
31222 8921 Acked Flush 0 Sent Done Done false
31223 8922 Acked Write 3 Sent Done Done false
31224 8923 Acked Read 3 Sent Done Done false
31225 8924 Acked Write 4 Sent Done Done false
31226 8925 Acked Read 3 Sent Done Done false
31227 8926 Acked Write 1 Sent Done Done false
31228 8927 Acked Read 2 Sent Done Done false
31229 8928 Acked Flush 0 Sent Done Done false
31230 8929 Acked Read 8 Sent Done Done false
31231 8930 Acked Read 10 Sent Done Done false
31232 8931 Acked Read 4 Sent Done Done false
31233 8932 Acked Write 10 Sent Done Done false
31234 8933 Acked Write 7 Sent Done Done false
31235 8934 Acked Write 3 Sent Done Done false
31236 8935 Acked Read 5 Sent Done Done false
31237 8936 Acked Write 1 Sent Done Done false
31238 8937 Acked Write 7 Sent Done Done false
31239 8938 Acked Read 1 Sent Done Done false
31240 8939 Acked Write 2 Sent Done Done false
31241 8940 Acked Flush 0 Sent Done Done false
31242 8941 Acked Read 7 Sent Done Done false
31243 8942 Acked Write 7 Sent Done Done false
31244 8943 Acked Read 10 Sent Done Done false
31245 8944 Acked Read 8 Sent Done Done false
31246 8945 Acked Read 9 Sent Done Done false
31247 8946 Acked Read 3 Sent Done Done false
31248 8947 Acked Write 3 Sent Done Done false
31249 8948 Acked Write 2 Sent Done Done false
31250 8949 Acked Read 9 Sent Done Done false
31251 8950 Acked Write 3 Sent Done Done false
31252 8951 Acked Write 5 Sent Done Done false
31253 8952 Acked Write 8 Sent Done Done false
31254 8953 Acked Read 9 Sent Done Done false
31255 8954 Acked Read 4 Sent Done Done false
31256 8955 Acked Write 7 Sent Done Done false
31257 8956 Acked Write 9 Sent Done Done false
31258 8957 Acked Write 10 Sent Done Done false
31259 8958 Acked Read 10 Sent Done Done false
31260 8959 Acked Read 10 Sent Done Done false
31261 8960 Acked Write 9 Sent Done Done false
31262 8961 Acked Read 7 Sent Done Done false
31263 8962 Acked Write 8 Sent Done Done false
31264 8963 Acked Write 2 Sent Done Done false
31265 8964 Acked Read 10 Sent Done Done false
31266 8965 Acked Read 8 Sent Done Done false
31267 8966 Acked Write 8 Sent Done Done false
31268 8967 Acked Write 6 Sent Done Done false
31269 8968 Acked Write 10 Sent Done Done false
31270 8969 Acked Read 3 Sent Done Done false
31271 8970 Acked Write 1 Sent Done Done false
31272 8971 Acked Read 5 Sent Done Done false
31273 8972 Acked Flush 0 Sent Done Done false
31274 8973 Acked Write 4 Sent Done Done false
31275 8974 Acked Write 1 Sent Done Done false
31276 8975 Acked Flush 0 Sent Done Done false
31277 8976 Acked Read 2 Sent Done Done false
31278 8977 Acked Read 5 Sent Done Done false
31279 8978 Acked Read 9 Sent Done Done false
31280 8979 Acked Read 3 Sent Done Done false
31281 8980 Acked Read 8 Sent Done Done false
31282 8981 Acked Read 9 Sent Done Done false
31283 8982 Acked Write 10 Sent Done Done false
31284 8983 Acked Write 2 Sent Done Done false
31285 8984 Acked Flush 0 Sent Done Done false
31286 8985 Acked Read 8 Sent Done Done false
31287 8986 Acked Write 3 Sent Done Done false
31288 8987 Acked Write 5 Sent Done Done false
31289 8988 Acked Read 2 Sent Done Done false
31290 8989 Acked Write 10 Sent Done Done false
31291 8990 Acked Write 3 Sent Done Done false
31292 8991 Acked Read 10 Sent Done Done false
31293 8992 Acked Read 3 Sent Done Done false
31294 8993 Acked Read 2 Sent Done Done false
31295 8994 Acked Read 10 Sent Done Done false
31296 8995 Acked Read 10 Sent Done Done false
31297 8996 Acked Write 5 Sent Done Done false
31298 8997 Acked Read 6 Sent Done Done false
31299 8998 Acked Read 8 Sent Done Done false
31300 8999 Acked Read 7 Sent Done Done false
31301 9000 Acked Write 3 Sent Done Done false
31302 9001 Acked Read 2 Sent Done Done false
31303 9002 Acked Write 3 Sent Done Done false
31304 9003 Acked Read 3 Sent Done Done false
31305 9004 Acked Read 4 Sent Done Done false
31306 9005 Acked Write 5 Sent Done Done false
31307 9006 Acked Read 9 Sent Done Done false
31308 9007 Acked Read 8 Sent Done Done false
31309 9008 Acked Read 8 Sent Done Done false
31310 9009 Acked Read 4 Sent Done Done false
31311 9010 Acked Write 9 Sent Done Done false
31312 9011 Acked Read 6 Sent Done Done false
31313 9012 Acked Read 8 Sent Done Done false
31314 9013 Acked Read 3 Sent Done Done false
31315 9014 Acked Read 9 Sent Done Done false
31316 9015 Acked Read 5 Sent Done Done false
31317 9016 Acked Write 9 Sent Done Done false
31318 9017 Acked Write 2 Sent Done Done false
31319 9018 Acked Read 7 Sent Done Done false
31320 9019 Acked Read 10 Sent Done Done false
31321 9020 Acked Write 1 Sent Done Done false
31322 9021 Acked Read 6 Sent Done Done false
31323 9022 Acked Read 6 Sent Done Done false
31324 9023 Acked Read 10 Sent Done Done false
31325 9024 Acked Read 10 Sent Done Done false
31326 9025 Acked Read 4 Sent Done Done false
31327 9026 Acked Read 10 Sent Done Done false
31328 9027 Acked Write 10 Sent Done Done false
31329 9028 Acked Read 6 Sent Done Done false
31330 9029 Acked Read 9 Sent Done Done false
31331 9030 Acked Write 2 Sent Done Done false
31332 9031 Acked Write 6 Sent Done Done false
31333 9032 Acked Flush 0 Sent Done Done false
31334 9033 Acked Write 1 Sent Done Done false
31335 9034 Acked Read 5 Sent Done Done false
31336 9035 Acked Read 5 Sent Done Done false
31337 9036 Acked Read 3 Sent Done Done false
31338 9037 Acked Read 2 Sent Done Done false
31339 9038 Acked Read 10 Sent Done Done false
31340 9039 Acked Read 5 Sent Done Done false
31341 9040 Acked Read 2 Sent Done Done false
31342 9041 Acked Write 7 Sent Done Done false
31343 9042 Acked Flush 0 Sent Done Done false
31344 9043 Acked Write 4 Sent Done Done false
31345 9044 Acked Read 8 Sent Done Done false
31346 9045 Acked Write 10 Sent Done Done false
31347 9046 Acked Write 5 Sent Done Done false
31348 9047 Acked Read 3 Sent Done Done false
31349 9048 Acked Write 10 Sent Done Done false
31350 9049 Acked Write 5 Sent Done Done false
31351 9050 Acked Write 3 Sent Done Done false
31352 9051 Acked Write 9 Sent Done Done false
31353 9052 Acked Write 3 Sent Done Done false
31354 9053 Acked Write 5 Sent Done Done false
31355 9054 Acked Write 10 Sent Done Done false
31356 9055 Acked Read 7 Sent Done Done false
31357 9056 Acked Read 7 Sent Done Done false
31358 9057 Acked Write 5 Sent Done Done false
31359 9058 Acked Flush 0 Sent Done Done false
31360 9059 Acked Read 4 Sent Done Done false
31361 9060 Acked Read 6 Sent Done Done false
31362 9061 Acked Flush 0 Sent Done Done false
31363 9062 Acked Write 10 Sent Done Done false
31364 9063 Acked Read 4 Sent Done Done false
31365 9064 Acked Write 2 Sent Done Done false
31366 9065 Acked Write 8 Sent Done Done false
31367 9066 Acked Write 2 Sent Done Done false
31368 9067 Acked Write 1 Sent Done Done false
31369 9068 Acked Write 3 Sent Done Done false
31370 9069 Acked Read 8 Sent Done Done false
31371 9070 Acked Write 9 Sent Done Done false
31372 9071 Acked Read 6 Sent Done Done false
31373 9072 Acked Write 5 Sent Done Done false
31374 9073 Acked Read 7 Sent Done Done false
31375 9074 Acked Read 9 Sent Done Done false
31376 9075 Acked Read 10 Sent Done Done false
31377 9076 Acked Write 3 Sent Done Done false
31378 9077 Acked Read 7 Sent Done Done false
31379 9078 Acked Read 6 Sent Done Done false
31380 9079 Acked Read 7 Sent Done Done false
31381 9080 Acked Write 2 Sent Done Done false
31382 9081 Acked Write 8 Sent Done Done false
31383 9082 Acked Write 5 Sent Done Done false
31384 9083 Acked Write 5 Sent Done Done false
31385 9084 Acked Flush 0 Sent Done Done false
31386 9085 Acked Write 6 Sent Done Done false
31387 9086 Acked Write 9 Sent Done Done false
31388 9087 Acked Read 10 Sent Done Done false
31389 9088 Acked Read 8 Sent Done Done false
31390 9089 Acked Read 5 Sent Done Done false
31391 9090 Acked Read 5 Sent Done Done false
31392 9091 Acked Write 9 Sent Done Done false
31393 9092 Acked Read 7 Sent Done Done false
31394 9093 Acked Read 8 Sent Done Done false
31395 9094 Acked Read 10 Sent Done Done false
31396 9095 Acked Write 2 Sent Done Done false
31397 9096 Acked Write 9 Sent Done Done false
31398 9097 Acked Write 9 Sent Done Done false
31399 9098 Acked Read 10 Sent Done Done false
31400 9099 Acked Write 3 Sent Done Done false
31401 9100 Acked Write 3 Sent Done Done false
31402 9101 Acked Read 5 Sent Done Done false
31403 9102 Acked Read 8 Sent Done Done false
31404 9103 Acked Read 8 Sent Done Done false
31405 9104 Acked Write 8 Sent Done Done false
31406 9105 Acked Write 2 Sent Done Done false
31407 9106 Acked Read 6 Sent Done Done false
31408 9107 Acked Read 3 Sent Done Done false
31409 9108 Acked Read 3 Sent Done Done false
31410 9109 Acked Write 3 Sent Done Done false
31411 9110 Acked Write 3 Sent Done Done false
31412 9111 Acked Write 5 Sent Done Done false
31413 9112 Acked Read 4 Sent Done Done false
31414 9113 Acked Read 2 Sent Done Done false
31415 9114 Acked Read 2 Sent Done Done false
31416 9115 Acked Write 2 Sent Done Done false
31417 9116 Acked Write 2 Sent Done Done false
31418 9117 Acked Write 10 Sent Done Done false
31419 9118 Acked Read 2 Sent Done Done false
31420 9119 Acked Read 6 Sent Done Done false
31421 9120 Acked Read 5 Sent Done Done false
31422 9121 Acked Read 5 Sent Done Done false
31423 9122 Acked Write 9 Sent Done Done false
31424 9123 Acked Read 8 Sent Done Done false
31425 9124 Acked Write 5 Sent Done Done false
31426 9125 Acked Write 9 Sent Done Done false
31427 9126 Acked Read 2 Sent Done Done false
31428 9127 Acked Read 2 Sent Done Done false
31429 9128 Acked Write 6 Sent Done Done false
31430 9129 Acked Write 10 Sent Done Done false
31431 9130 Acked Write 8 Sent Done Done false
31432 9131 Acked Write 3 Sent Done Done false
31433 9132 Acked Flush 0 Sent Done Done false
31434 9133 Acked Read 4 Sent Done Done false
31435 9134 Acked Write 4 Sent Done Done false
31436 9135 Acked Write 9 Sent Done Done false
31437 9136 Acked Read 6 Sent Done Done false
31438 9137 Acked Read 10 Sent Done Done false
31439 9138 Acked Read 2 Sent Done Done false
31440 9139 Acked Write 8 Sent Done Done false
31441 9140 Acked Write 3 Sent Done Done false
31442 9141 Acked Flush 0 Sent Done Done false
31443 9142 Acked Read 10 Sent Done Done false
31444 9143 Acked Write 9 Sent Done Done false
31445 9144 Acked Read 5 Sent Done Done false
31446 9145 Acked Read 8 Sent Done Done false
31447 9146 Acked Write 6 Sent Done Done false
31448 9147 Acked Write 6 Sent Done Done false
31449 9148 Acked Read 10 Sent Done Done false
31450 STATES DS:0 DS:1 DS:2 TOTAL
31451 Sent 301 0 0 301
31452 Done 0 301 301 602
31453 Skipped 0 0 0 0
31454 Error 0 0 0 0
31455 Last Flush: Some(JobId(8847)) Some(JobId(9141)) Some(JobId(9141))
31456 Downstairs last five completed: 8847 8846 8845 8844 8843
31457 Upstairs last five completed: 9148 9147 9146 9145 9144
31458 CLIENT: Up:0 ds:301 act:2
314592025-02-21T06:52:14.590ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
314602025-02-21T06:52:15.291ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
314612025-02-21T06:52:16.293ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
314622025-02-21T06:52:17.295ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31463 ----------------------------------------------------------------
31464 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31465 States: Offline Active Active
31466 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31467 8848 Acked Flush 0 Sent Done Done false
31468 8849 Acked Flush 0 Sent Done Done false
31469 8850 Acked Read 3 Sent Done Done false
31470 8851 Acked Read 5 Sent Done Done false
31471 8852 Acked Write 9 Sent Done Done false
31472 8853 Acked Read 5 Sent Done Done false
31473 8854 Acked Read 5 Sent Done Done false
31474 8855 Acked Read 4 Sent Done Done false
31475 8856 Acked Flush 0 Sent Done Done false
31476 8857 Acked Read 4 Sent Done Done false
31477 8858 Acked Read 5 Sent Done Done false
31478 8859 Acked Read 8 Sent Done Done false
31479 8860 Acked Read 1 Sent Done Done false
31480 8861 Acked Read 10 Sent Done Done false
31481 8862 Acked Flush 0 Sent Done Done false
31482 8863 Acked Read 7 Sent Done Done false
31483 8864 Acked Write 5 Sent Done Done false
31484 8865 Acked Flush 0 Sent Done Done false
31485 8866 Acked Read 4 Sent Done Done false
31486 8867 Acked Flush 0 Sent Done Done false
31487 8868 Acked Read 1 Sent Done Done false
31488 8869 Acked Read 2 Sent Done Done false
31489 8870 Acked Read 10 Sent Done Done false
31490 8871 Acked Write 10 Sent Done Done false
31491 8872 Acked Read 1 Sent Done Done false
31492 8873 Acked Write 7 Sent Done Done false
31493 8874 Acked Write 6 Sent Done Done false
31494 8875 Acked Read 2 Sent Done Done false
31495 8876 Acked Write 6 Sent Done Done false
31496 8877 Acked Read 8 Sent Done Done false
31497 8878 Acked Write 4 Sent Done Done false
31498 8879 Acked Flush 0 Sent Done Done false
31499 8880 Acked Read 5 Sent Done Done false
31500 8881 Acked Read 10 Sent Done Done false
31501 8882 Acked Read 4 Sent Done Done false
31502 8883 Acked Read 7 Sent Done Done false
31503 8884 Acked Read 5 Sent Done Done false
31504 8885 Acked Write 8 Sent Done Done false
31505 8886 Acked Flush 0 Sent Done Done false
31506 8887 Acked Read 3 Sent Done Done false
31507 8888 Acked Write 5 Sent Done Done false
31508 8889 Acked Flush 0 Sent Done Done false
31509 8890 Acked Write 9 Sent Done Done false
31510 8891 Acked Write 2 Sent Done Done false
31511 8892 Acked Read 2 Sent Done Done false
31512 8893 Acked Write 1 Sent Done Done false
31513 8894 Acked Write 9 Sent Done Done false
31514 8895 Acked Read 9 Sent Done Done false
31515 8896 Acked Read 1 Sent Done Done false
31516 8897 Acked Write 6 Sent Done Done false
31517 8898 Acked Read 10 Sent Done Done false
31518 8899 Acked Write 1 Sent Done Done false
31519 8900 Acked Read 5 Sent Done Done false
31520 8901 Acked Read 8 Sent Done Done false
31521 8902 Acked Write 5 Sent Done Done false
31522 8903 Acked Read 10 Sent Done Done false
31523 8904 Acked Write 5 Sent Done Done false
31524 8905 Acked Write 4 Sent Done Done false
31525 8906 Acked Flush 0 Sent Done Done false
31526 8907 Acked Write 6 Sent Done Done false
31527 8908 Acked Flush 0 Sent Done Done false
31528 8909 Acked Read 4 Sent Done Done false
31529 8910 Acked Write 4 Sent Done Done false
31530 8911 Acked Write 1 Sent Done Done false
31531 8912 Acked Write 8 Sent Done Done false
31532 8913 Acked Write 6 Sent Done Done false
31533 8914 Acked Write 1 Sent Done Done false
31534 8915 Acked Read 6 Sent Done Done false
31535 8916 Acked Read 6 Sent Done Done false
31536 8917 Acked Write 6 Sent Done Done false
31537 8918 Acked Write 6 Sent Done Done false
31538 8919 Acked Write 4 Sent Done Done false
31539 8920 Acked Read 10 Sent Done Done false
31540 8921 Acked Flush 0 Sent Done Done false
31541 8922 Acked Write 3 Sent Done Done false
31542 8923 Acked Read 3 Sent Done Done false
31543 8924 Acked Write 4 Sent Done Done false
31544 8925 Acked Read 3 Sent Done Done false
31545 8926 Acked Write 1 Sent Done Done false
31546 8927 Acked Read 2 Sent Done Done false
31547 8928 Acked Flush 0 Sent Done Done false
31548 8929 Acked Read 8 Sent Done Done false
31549 8930 Acked Read 10 Sent Done Done false
31550 8931 Acked Read 4 Sent Done Done false
31551 8932 Acked Write 10 Sent Done Done false
31552 8933 Acked Write 7 Sent Done Done false
31553 8934 Acked Write 3 Sent Done Done false
31554 8935 Acked Read 5 Sent Done Done false
31555 8936 Acked Write 1 Sent Done Done false
31556 8937 Acked Write 7 Sent Done Done false
31557 8938 Acked Read 1 Sent Done Done false
31558 8939 Acked Write 2 Sent Done Done false
31559 8940 Acked Flush 0 Sent Done Done false
31560 8941 Acked Read 7 Sent Done Done false
31561 8942 Acked Write 7 Sent Done Done false
31562 8943 Acked Read 10 Sent Done Done false
31563 8944 Acked Read 8 Sent Done Done false
31564 8945 Acked Read 9 Sent Done Done false
31565 8946 Acked Read 3 Sent Done Done false
31566 8947 Acked Write 3 Sent Done Done false
31567 8948 Acked Write 2 Sent Done Done false
31568 8949 Acked Read 9 Sent Done Done false
31569 8950 Acked Write 3 Sent Done Done false
31570 8951 Acked Write 5 Sent Done Done false
31571 8952 Acked Write 8 Sent Done Done false
31572 8953 Acked Read 9 Sent Done Done false
31573 8954 Acked Read 4 Sent Done Done false
31574 8955 Acked Write 7 Sent Done Done false
31575 8956 Acked Write 9 Sent Done Done false
31576 8957 Acked Write 10 Sent Done Done false
31577 8958 Acked Read 10 Sent Done Done false
31578 8959 Acked Read 10 Sent Done Done false
31579 8960 Acked Write 9 Sent Done Done false
31580 8961 Acked Read 7 Sent Done Done false
31581 8962 Acked Write 8 Sent Done Done false
31582 8963 Acked Write 2 Sent Done Done false
31583 8964 Acked Read 10 Sent Done Done false
31584 8965 Acked Read 8 Sent Done Done false
31585 8966 Acked Write 8 Sent Done Done false
31586 8967 Acked Write 6 Sent Done Done false
31587 8968 Acked Write 10 Sent Done Done false
31588 8969 Acked Read 3 Sent Done Done false
31589 8970 Acked Write 1 Sent Done Done false
31590 8971 Acked Read 5 Sent Done Done false
31591 8972 Acked Flush 0 Sent Done Done false
31592 8973 Acked Write 4 Sent Done Done false
31593 8974 Acked Write 1 Sent Done Done false
31594 8975 Acked Flush 0 Sent Done Done false
31595 8976 Acked Read 2 Sent Done Done false
31596 8977 Acked Read 5 Sent Done Done false
31597 8978 Acked Read 9 Sent Done Done false
31598 8979 Acked Read 3 Sent Done Done false
31599 8980 Acked Read 8 Sent Done Done false
31600 8981 Acked Read 9 Sent Done Done false
31601 8982 Acked Write 10 Sent Done Done false
31602 8983 Acked Write 2 Sent Done Done false
31603 8984 Acked Flush 0 Sent Done Done false
31604 8985 Acked Read 8 Sent Done Done false
31605 8986 Acked Write 3 Sent Done Done false
31606 8987 Acked Write 5 Sent Done Done false
31607 8988 Acked Read 2 Sent Done Done false
31608 8989 Acked Write 10 Sent Done Done false
31609 8990 Acked Write 3 Sent Done Done false
31610 8991 Acked Read 10 Sent Done Done false
31611 8992 Acked Read 3 Sent Done Done false
31612 8993 Acked Read 2 Sent Done Done false
31613 8994 Acked Read 10 Sent Done Done false
31614 8995 Acked Read 10 Sent Done Done false
31615 8996 Acked Write 5 Sent Done Done false
31616 8997 Acked Read 6 Sent Done Done false
31617 8998 Acked Read 8 Sent Done Done false
31618 8999 Acked Read 7 Sent Done Done false
31619 9000 Acked Write 3 Sent Done Done false
31620 9001 Acked Read 2 Sent Done Done false
31621 9002 Acked Write 3 Sent Done Done false
31622 9003 Acked Read 3 Sent Done Done false
31623 9004 Acked Read 4 Sent Done Done false
31624 9005 Acked Write 5 Sent Done Done false
31625 9006 Acked Read 9 Sent Done Done false
31626 9007 Acked Read 8 Sent Done Done false
31627 9008 Acked Read 8 Sent Done Done false
31628 9009 Acked Read 4 Sent Done Done false
31629 9010 Acked Write 9 Sent Done Done false
31630 9011 Acked Read 6 Sent Done Done false
31631 9012 Acked Read 8 Sent Done Done false
31632 9013 Acked Read 3 Sent Done Done false
31633 9014 Acked Read 9 Sent Done Done false
31634 9015 Acked Read 5 Sent Done Done false
31635 9016 Acked Write 9 Sent Done Done false
31636 9017 Acked Write 2 Sent Done Done false
31637 9018 Acked Read 7 Sent Done Done false
31638 9019 Acked Read 10 Sent Done Done false
31639 9020 Acked Write 1 Sent Done Done false
31640 9021 Acked Read 6 Sent Done Done false
31641 9022 Acked Read 6 Sent Done Done false
31642 9023 Acked Read 10 Sent Done Done false
31643 9024 Acked Read 10 Sent Done Done false
31644 9025 Acked Read 4 Sent Done Done false
31645 9026 Acked Read 10 Sent Done Done false
31646 9027 Acked Write 10 Sent Done Done false
31647 9028 Acked Read 6 Sent Done Done false
31648 9029 Acked Read 9 Sent Done Done false
31649 9030 Acked Write 2 Sent Done Done false
31650 9031 Acked Write 6 Sent Done Done false
31651 9032 Acked Flush 0 Sent Done Done false
31652 9033 Acked Write 1 Sent Done Done false
31653 9034 Acked Read 5 Sent Done Done false
31654 9035 Acked Read 5 Sent Done Done false
31655 9036 Acked Read 3 Sent Done Done false
31656 9037 Acked Read 2 Sent Done Done false
31657 9038 Acked Read 10 Sent Done Done false
31658 9039 Acked Read 5 Sent Done Done false
31659 9040 Acked Read 2 Sent Done Done false
31660 9041 Acked Write 7 Sent Done Done false
31661 9042 Acked Flush 0 Sent Done Done false
31662 9043 Acked Write 4 Sent Done Done false
31663 9044 Acked Read 8 Sent Done Done false
31664 9045 Acked Write 10 Sent Done Done false
31665 9046 Acked Write 5 Sent Done Done false
31666 9047 Acked Read 3 Sent Done Done false
31667 9048 Acked Write 10 Sent Done Done false
31668 9049 Acked Write 5 Sent Done Done false
31669 9050 Acked Write 3 Sent Done Done false
31670 9051 Acked Write 9 Sent Done Done false
31671 9052 Acked Write 3 Sent Done Done false
31672 9053 Acked Write 5 Sent Done Done false
31673 9054 Acked Write 10 Sent Done Done false
31674 9055 Acked Read 7 Sent Done Done false
31675 9056 Acked Read 7 Sent Done Done false
31676 9057 Acked Write 5 Sent Done Done false
31677 9058 Acked Flush 0 Sent Done Done false
31678 9059 Acked Read 4 Sent Done Done false
31679 9060 Acked Read 6 Sent Done Done false
31680 9061 Acked Flush 0 Sent Done Done false
31681 9062 Acked Write 10 Sent Done Done false
31682 9063 Acked Read 4 Sent Done Done false
31683 9064 Acked Write 2 Sent Done Done false
31684 9065 Acked Write 8 Sent Done Done false
31685 9066 Acked Write 2 Sent Done Done false
31686 9067 Acked Write 1 Sent Done Done false
31687 9068 Acked Write 3 Sent Done Done false
31688 9069 Acked Read 8 Sent Done Done false
31689 9070 Acked Write 9 Sent Done Done false
31690 9071 Acked Read 6 Sent Done Done false
31691 9072 Acked Write 5 Sent Done Done false
31692 9073 Acked Read 7 Sent Done Done false
31693 9074 Acked Read 9 Sent Done Done false
31694 9075 Acked Read 10 Sent Done Done false
31695 9076 Acked Write 3 Sent Done Done false
31696 9077 Acked Read 7 Sent Done Done false
31697 9078 Acked Read 6 Sent Done Done false
31698 9079 Acked Read 7 Sent Done Done false
31699 9080 Acked Write 2 Sent Done Done false
31700 9081 Acked Write 8 Sent Done Done false
31701 9082 Acked Write 5 Sent Done Done false
31702 9083 Acked Write 5 Sent Done Done false
31703 9084 Acked Flush 0 Sent Done Done false
31704 9085 Acked Write 6 Sent Done Done false
31705 9086 Acked Write 9 Sent Done Done false
31706 9087 Acked Read 10 Sent Done Done false
31707 9088 Acked Read 8 Sent Done Done false
31708 9089 Acked Read 5 Sent Done Done false
31709 9090 Acked Read 5 Sent Done Done false
31710 9091 Acked Write 9 Sent Done Done false
31711 9092 Acked Read 7 Sent Done Done false
31712 9093 Acked Read 8 Sent Done Done false
31713 9094 Acked Read 10 Sent Done Done false
31714 9095 Acked Write 2 Sent Done Done false
31715 9096 Acked Write 9 Sent Done Done false
31716 9097 Acked Write 9 Sent Done Done false
31717 9098 Acked Read 10 Sent Done Done false
31718 9099 Acked Write 3 Sent Done Done false
31719 9100 Acked Write 3 Sent Done Done false
31720 9101 Acked Read 5 Sent Done Done false
31721 9102 Acked Read 8 Sent Done Done false
31722 9103 Acked Read 8 Sent Done Done false
31723 9104 Acked Write 8 Sent Done Done false
31724 9105 Acked Write 2 Sent Done Done false
31725 9106 Acked Read 6 Sent Done Done false
31726 9107 Acked Read 3 Sent Done Done false
31727 9108 Acked Read 3 Sent Done Done false
31728 9109 Acked Write 3 Sent Done Done false
31729 9110 Acked Write 3 Sent Done Done false
31730 9111 Acked Write 5 Sent Done Done false
31731 9112 Acked Read 4 Sent Done Done false
31732 9113 Acked Read 2 Sent Done Done false
31733 9114 Acked Read 2 Sent Done Done false
31734 9115 Acked Write 2 Sent Done Done false
31735 9116 Acked Write 2 Sent Done Done false
31736 9117 Acked Write 10 Sent Done Done false
31737 9118 Acked Read 2 Sent Done Done false
31738 9119 Acked Read 6 Sent Done Done false
31739 9120 Acked Read 5 Sent Done Done false
31740 9121 Acked Read 5 Sent Done Done false
31741 9122 Acked Write 9 Sent Done Done false
31742 9123 Acked Read 8 Sent Done Done false
31743 9124 Acked Write 5 Sent Done Done false
31744 9125 Acked Write 9 Sent Done Done false
31745 9126 Acked Read 2 Sent Done Done false
31746 9127 Acked Read 2 Sent Done Done false
31747 9128 Acked Write 6 Sent Done Done false
31748 9129 Acked Write 10 Sent Done Done false
31749 9130 Acked Write 8 Sent Done Done false
31750 9131 Acked Write 3 Sent Done Done false
31751 9132 Acked Flush 0 Sent Done Done false
31752 9133 Acked Read 4 Sent Done Done false
31753 9134 Acked Write 4 Sent Done Done false
31754 9135 Acked Write 9 Sent Done Done false
31755 9136 Acked Read 6 Sent Done Done false
31756 9137 Acked Read 10 Sent Done Done false
31757 9138 Acked Read 2 Sent Done Done false
31758 9139 Acked Write 8 Sent Done Done false
31759 9140 Acked Write 3 Sent Done Done false
31760 9141 Acked Flush 0 Sent Done Done false
31761 9142 Acked Read 10 Sent Done Done false
31762 9143 Acked Write 9 Sent Done Done false
31763 9144 Acked Read 5 Sent Done Done false
31764 9145 Acked Read 8 Sent Done Done false
31765 9146 Acked Write 6 Sent Done Done false
31766 9147 Acked Write 6 Sent Done Done false
31767 9148 Acked Read 10 Sent Done Done false
31768 STATES DS:0 DS:1 DS:2 TOTAL
31769 Sent 301 0 0 301
31770 Done 0 301 301 602
31771 Skipped 0 0 0 0
31772 Error 0 0 0 0
31773 Last Flush: Some(JobId(8847)) Some(JobId(9141)) Some(JobId(9141))
31774 Downstairs last five completed: 8847 8846 8845 8844 8843
31775 Upstairs last five completed: 9148 9147 9146 9145 9144
31776 CLIENT: Up:0 ds:301 act:2
317772025-02-21T06:52:19.297ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
317782025-02-21T06:52:20.299ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
317792025-02-21T06:52:21.301ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
317802025-02-21T06:52:21.834ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
317812025-02-21T06:52:21.834ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62543) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
317822025-02-21T06:52:21.835ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
317832025-02-21T06:52:21.835ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
317842025-02-21T06:52:21.835ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
317852025-02-21T06:52:21.835ZINFOcrucible: send last flush ID to this DS: Some(JobId(8847)) = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
317862025-02-21T06:52:21.835ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
317872025-02-21T06:52:21.835ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(8847)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
317882025-02-21T06:52:21.835ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(8847)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
31789 ----------------------------------------------------------------
31790 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:7
31791 States: Active Active Active
31792 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31793 9142 Acked Read 10 Done Done Done true
31794 9143 Acked Write 9 Done Done Done true
31795 9144 Acked Read 5 Done Done Done true
31796 9145 Acked Read 8 Done Done Done true
31797 9146 Acked Write 6 Done Done Done true
31798 9147 Acked Write 6 Done Done Done true
31799 9148 Acked Read 10 Done Done Done true
31800 STATES DS:0 DS:1 DS:2 TOTAL
31801 Sent 0 0 0 0
31802 Done 7 7 7 21
31803 Skipped 0 0 0 0
31804 Error 0 0 0 0
31805 Last Flush: Some(JobId(9141)) Some(JobId(9141)) Some(JobId(9141))
31806 Downstairs last five completed: 9141 9140 9139 9138 9137
31807 Upstairs last five completed: 9148 9147 9146 9145 9144
31808 CLIENT: Up:0 ds:7 act:3
31809 ----------------------------------------------------------------
31810 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31811 Downstairs last five completed: 9149 9148 9147 9146 9145
31812 Upstairs last five completed: 9149 9148 9147 9146 9145
31813 CLIENT: Up:0 ds:0 act:3
31814 Replay: All jobs finished
31815 Main task has work to do, go find it
31816 got dsc Stop(0)
31817 stop 0
31818 [8810] Got stop action so:false kr:false
31819 Feb 21 06:52:26.094 INFO request completed, latency_us: 426, response_code: 204, uri: /stop/cid/0, method: POST, req_id: a8d9e296-31bc-4150-b19f-acbbb79791ef, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
31820 Feb 21 06:52:26.095 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:52:26.095477185Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
31821 INFO request completed, latency_us: 134, response_code: 200, uri: /state/cid/0, method: GET, req_id:{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:52:26.095633528Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
31822 cab572db-c1aa-4e7a-a5e9-c0ab5294a515, remote_addr: 127.0.0{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:52:26.095691427Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
31823 .1:53848, local_addr: 127.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:26.095722702Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
31824 0.0.1:9998
31825 [8810] Exited with: ExitStatus(unix_wait_status(9))
31826 [8810][0] reports Exit
31827 Feb 21 06:52:30.098 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/0, method: GET, req_id: 729299f4-d636-4b3f-b177-4eded97a98bd, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
31828 001/300 Read block 1404 len 12288
31829 002/300 Write block 3996 len 16384 data: 3 4 4 3
31830 003/300 Read block 2095 len 24576
31831 004/300 Write block 4753 len 24576 data: 5 4 4 4 4 7
31832 005/300 Read block 336 len 36864
31833 006/300 Read block 3088 len 20480
31834 007/300 Write block 1212 len 4096 data: 7
31835 008/300 Flush
31836 009/300 Write block 3323 len 4096 data: 7
31837 010/300 Flush
31838 011/300 Write block 2152 len 24576 data: 3 2 4 4 4 4
31839 012/300 Read block 2855 len 16384
31840 013/300 Read block 2829 len 8192
31841 014/300 Read block 3039 len 20480
31842 015/300 Write block 1301 len 20480 data: 4 4 3 4 4
31843 016/300 Flush
31844 017/300 Write block 3900 len 40960 data: 3 5 5 4 4 4 7 7 7 6
31845 018/300 Write block 3254 len 8192 data: 5 5
31846 019/300 Read block 1 len 20480
31847 020/300 Read block 1422 len 16384
31848 021/300 Read block 2197 len 32768
31849 022/300 Write block 4833 len 4096 data: 7
31850 023/300 Read block 49 len 4096
31851 024/300 Read block 2525 len 40960
31852 025/300 Write block 3206 len 20480 data: 6 7 5 4 4
31853 026/300 Flush
31854 027/300 Read block 1278 len 24576
31855 028/300 Read block 463 len 16384
31856 029/300 Write block 4199 len 20480 data: 5 4 5 6 6
31857 030/300 Write block 281 len 24576 data: 4 4 3 4 4 5
31858 031/300 Flush
31859 032/300 Write block 3658 len 12288 data: 7 7 6
31860 033/300 Write block 4907 len 16384 data: 5 5 4 4
31861 034/300 Write block 482 len 36864 data: 5 6 6 5 4 4 4 3 4
31862 035/300 Write block 2991 len 12288 data: 6 5 4
31863 036/300 Write block 2949 len 20480 data: 5 7 9 10 10
31864 037/300 Write block 227 len 4096 data: 4
31865 038/300 Write block 3100 len 8192 data: 7 7
31866 039/300 Write block 4445 len 36864 data: 5 4 6 6 7 7 7 8 8
31867 040/300 Write block 357 len 4096 data: 6
31868 041/300 Read block 1222 len 4096
31869 042/300 Read block 2081 len 36864
31870 043/300 Write block 1339 len 40960 data: 2 2 2 2 3 3 3 4 4 4
31871 044/300 Read block 966 len 8192
31872 045/300 Read block 314 len 20480
31873 046/300 Read block 452 len 32768
31874 047/300 Write block 168 len 36864 data: 7 7 5 6 6 8 7 7 6
31875 048/300 Read block 3886 len 12288
31876 049/300 Write block 2304 len 4096 data: 8
31877 050/300 Read block 62 len 24576
31878 051/300 Write block 3270 len 32768 data: 7 6 6 8 7 8 6 6
31879 052/300 Write block 962 len 28672 data: 5 5 5 4 4 3 3
31880 053/300 Read block 3511 len 36864
31881 054/300 Read block 714 len 4096
31882 055/300 Read block 2767 len 20480
31883 056/300 Write block 4018 len 8192 data: 9 6
31884 057/300 Write block 915 len 16384 data: 6 4 4 4
31885 058/300 Write block 4078 len 32768 data: 6 5 4 4 4 4 4 4
31886 059/300 Read block 3805 len 36864
31887 060/300 Write block 2922 len 36864 data: 4 4 4 3 3 4 5 5 5
31888 061/300 Write block 3159 len 12288 data: 4 5 6
31889 062/300 Read block 4384 len 40960
31890 063/300 Write block 2632 len 40960 data: 8 6 5 5 5 5 4 4 3 4
31891 064/300 Read block 1869 len 24576
31892 065/300 Flush
31893 066/300 Read block 511 len 12288
31894 067/300 Read block 2539 len 24576
31895 068/300 Read block 2059 len 32768
31896 069/300 Flush
31897 070/300 Read block 4915 len 4096
31898 071/300 Flush
31899 072/300 Write block 4492 len 32768 data: 6 5 6 6 6 4 4 4
31900 073/300 Read block 1070 len 12288
31901 074/300 Read block 3214 len 12288
31902 075/300 Write block 3700 len 12288 data: 8 8 9
31903 076/300 Read block 1682 len 36864
31904 077/300 Flush
31905 078/300 Read block 328 len 12288
31906 079/300 Read block 3048 len 4096
31907 080/300 Read block 4437 len 40960
31908 081/300 Write block 915 len 8192 data: 7 5
31909 082/300 Read block 4525 len 32768
31910 083/300 Write block 1459 len 32768 data: 4 5 6 6 9 11 11 10
31911 084/300 Write block 3643 len 8192 data: 5 5
31912 085/300 Write block 358 len 8192 data: 6 6
31913 086/300 Write block 1483 len 8192 data: 5 6
31914 087/300 Flush
31915 088/300 Read block 1001 len 36864
31916 089/300 Flush
31917 090/300 Read block 2253 len 8192
31918 091/300 Write block 4619 len 28672 data: 7 8 6 7 7 7 9
31919 092/300 Write block 1509 len 20480 data: 6 4 4 2 2
31920 093/300 Read block 185 len 36864
31921 094/300 Read block 269 len 32768
31922 095/300 Read block 4792 len 24576
31923 096/300 Read block 3965 len 24576
31924 097/300 Write block 2079 len 24576 data: 4 4 4 3 2 2
31925 098/300 Write block 2045 len 16384 data: 6 4 4 5
31926 099/300 Read block 3913 len 20480
31927 100/300 Write block 4880 len 8192 data: 9 7
31928 101/300 Write block 3085 len 20480 data: 7 7 7 6 6
31929 102/300 Write block 2746 len 12288 data: 8 9 11
31930 103/300 Write block 2950 len 32768 data: 8 10 11 11 8 8 7 5
31931 104/300 Read block 3825 len 20480
31932 105/300 Write block 2597 len 32768 data: 3 4 3 5 5 4 5 7
31933 106/300 Write block 3348 len 20480 data: 8 7 8 7 7
31934 107/300 Write block 4959 len 36864 data: 4 5 5 5 8 8 7 6 4
31935 108/300 Write block 3324 len 40960 data: 8 8 7 5 5 4 4 4 4 4
31936 109/300 Read block 1879 len 32768
31937 110/300 Write block 659 len 8192 data: 3 4
31938 111/300 Read block 3792 len 24576
31939 112/300 Write block 755 len 12288 data: 7 4 5
31940 113/300 Read block 2803 len 20480
31941 114/300 Write block 1647 len 40960 data: 5 5 4 3 3 3 3 3 6 6
31942 115/300 Read block 4243 len 12288
31943 116/300 Write block 3812 len 40960 data: 7 7 7 9 11 11 8 9 10 11
31944 117/300 Flush
31945 118/300 Read block 39 len 24576
31946 119/300 Read block 3120 len 36864
31947 120/300 Write block 503 len 8192 data: 10 8
31948 121/300 Read block 2203 len 20480
31949 122/300 Write block 2685 len 28672 data: 5 6 4 3 4 3 2
31950 123/300 Write block 3800 len 4096 data: 8
31951 124/300 Write block 1083 len 24576 data: 4 5 4 5 7 6
31952 125/300 Write block 4729 len 32768 data: 7 8 7 8 9 6 7 5
31953 126/300 Read block 3354 len 4096
31954 127/300 Read block 4121 len 40960
31955 128/300 Write block 596 len 16384 data: 3 3 2 3
31956 129/300 Read block 2492 len 20480
31957 130/300 Read block 4125 len 4096
31958 131/300 Read block 2019 len 36864
31959 132/300 Flush
31960 133/300 Read block 2119 len 12288
31961 134/300 Write block 3782 len 36864 data: 5 4 4 5 5 6 5 5 4
31962 135/300 Read block 1435 len 28672
31963 136/300 Write block 1811 len 12288 data: 9 7 9
31964 137/300 Read block 3825 len 40960
31965 138/300 Read block 1071 len 28672
31966 139/300 Read block 3312 len 12288
31967 140/300 Write block 3534 len 28672 data: 3 3 3 4 3 4 5
31968 141/300 Read block 3593 len 24576
31969 142/300 Write block 723 len 8192 data: 7 9
31970 143/300 Write block 2607 len 8192 data: 6 5
31971 144/300 Read block 712 len 12288
31972 145/300 Flush
31973 146/300 Write block 4732 len 32768 data: 9 10 7 8 6 7 8 9
31974 147/300 Write block 2507 len 16384 data: 7 7 9 8
31975 148/300 Read block 4081 len 40960
31976 149/300 Write block 3661 len 36864 data: 5 6 5 5 5 5 6 7 6
31977 150/300 Write block 3542 len 28672 data: 4 4 5 5 5 7 7
31978 151/300 Read block 1661 len 36864
31979 152/300 Write block 3363 len 16384 data: 3 5 5 5
31980 153/300 Read block 1342 len 8192
31981 154/300 Read block 1092 len 20480
31982 155/300 Write block 3804 len 16384 data: 6 6 6 6
31983 156/300 Write block 1540 len 24576 data: 5 5 5 5 4 4
31984 157/300 Read block 2365 len 36864
31985 158/300 Read block 416 len 16384
31986 159/300 Read block 281 len 32768
31987 160/300 Read block 4343 len 28672
31988 161/300 Read block 3644 len 4096
31989 162/300 Read block 2471 len 16384
31990 163/300 Read block 980 len 24576
31991 164/300 Read block 2715 len 32768
31992 165/300 Write block 2599 len 20480 data: 4 6 6 5 6
31993 166/300 Read block 2632 len 12288
31994 167/300 Write block 1799 len 40960 data: 6 5 5 5 6 7 8 7 6 6
31995 168/300 Flush
31996 169/300 Read block 525 len 4096
31997 170/300 Write block 4107 len 16384 data: 7 8 8 7
31998 171/300 Read block 1257 len 36864
31999 172/300 Read block 3629 len 32768
32000 173/300 Read block 801 len 36864
32001 174/300 Read block 16 len 20480
32002 175/300 Read block 3960 len 32768
32003 176/300 Write block 3392 len 24576 data: 6 6 3 3 3 2
32004 177/300 Read block 822 len 40960
32005 178/300 Read block 814 len 4096
32006 179/300 Read block 537 len 20480
32007 180/300 Write block 4696 len 40960 data: 4 4 7 7 6 6 8 7 6 5
32008 181/300 Flush
32009 182/300 Write block 1947 len 24576 data: 8 7 6 5 6 7
32010 183/300 Read block 4991 len 28672
32011 184/300 Read block 621 len 8192
32012 185/300 Write block 3413 len 8192 data: 7 7
32013 186/300 Read block 3892 len 28672
32014 187/300 Read block 4158 len 24576
32015 188/300 Read block 2687 len 40960
32016 189/300 Read block 1871 len 4096
32017 190/300 Write block 4439 len 20480 data: 3 3 4 5 6
32018 191/300 Read block 1811 len 20480
32019 192/300 Read block 807 len 40960
32020 193/300 Read block 626 len 12288
32021 194/300 Write block 3798 len 36864 data: 8 8 9 5 6 6 7 7 7
32022 195/300 Read block 2629 len 8192
32023 196/300 Write block 1440 len 36864 data: 9 8 6 7 7 6 4 6 7
32024 197/300 Write block 1440 len 28672 data: 10 9 7 8 8 7 5
32025 198/300 Write block 4472 len 12288 data: 9 9 9
32026 199/300 Write block 4841 len 36864 data: 5 4 5 4 3 2 2 2 2
32027 200/300 Read block 4212 len 16384
32028 201/300 Write block 3255 len 4096 data: 6
32029 202/300 Write block 408 len 32768 data: 4 3 3 4 4 3 3 4
32030 203/300 Flush
32031 204/300 Write block 4142 len 32768 data: 4 4 7 7 6 4 2 2
32032 205/300 Read block 4029 len 16384
32033 206/300 Write block 4586 len 20480 data: 7 7 7 6 6
32034 207/300 Write block 1618 len 28672 data: 8 7 6 5 7 7 7
32035 208/300 Flush
32036 209/300 Write block 2393 len 24576 data: 7 7 5 5 5 2
32037 210/300 Write block 4443 len 28672 data: 7 6 6 5 7 7 8
32038 211/300 Read block 112 len 16384
32039 212/300 Read block 3392 len 28672
32040 213/300 Read block 944 len 8192
32041 214/300 Write block 4233 len 16384 data: 4 4 4 4
32042 215/300 Read block 1501 len 12288
32043 216/300 Write block 474 len 28672 data: 7 8 9 8 7 7 6
32044 217/300 Write block 3583 len 40960 data: 9 11 13 13 10 10 7 7 7 8
32045 218/300 Read block 4960 len 40960
32046 219/300 Read block 177 len 12288
32047 220/300 Write block 3785 len 4096 data: 6
32048 221/300 Write block 2285 len 28672 data: 5 5 5 6 5 5 5
32049 222/300 Write block 1983 len 8192 data: 4 6
32050 223/300 Read block 4553 len 36864
32051 224/300 Read block 3674 len 28672
32052 225/300 Write block 2986 len 28672 data: 6 6 6 6 6 7 6
32053 226/300 Flush
32054 227/300 Read block 4572 len 36864
32055 228/300 Write block 579 len 36864 data: 6 6 5 6 6 5 6 6 7
32056 229/300 Read block 3956 len 24576
32057 230/300 Read block 1833 len 4096
32058 231/300 Write block 4931 len 24576 data: 3 3 4 5 5 6
32059 232/300 Write block 817 len 20480 data: 4 4 3 3 2
32060 233/300 Read block 1822 len 12288
32061 234/300 Flush
32062 235/300 Write block 511 len 28672 data: 4 4 5 4 3 4 5
32063 236/300 Flush
32064 237/300 Read block 4603 len 12288
32065 238/300 Read block 4407 len 8192
32066 239/300 Read block 1493 len 28672
32067 240/300 Flush
32068 241/300 Write block 3417 len 28672 data: 9 7 8 9 9 6 7
32069 242/300 Read block 1512 len 4096
32070 243/300 Write block 4801 len 40960 data: 7 6 5 4 3 5 4 5 4 6
32071 244/300 Read block 2845 len 8192
32072 245/300 Write block 4823 len 24576 data: 7 7 6 4 5 7
32073 246/300 Flush
32074 247/300 Write block 2787 len 4096 data: 7
32075 248/300 Write block 687 len 4096 data: 6
32076 249/300 Read block 1339 len 20480
32077 250/300 Read block 4008 len 8192
32078 251/300 Read block 1343 len 8192
32079 252/300 Read block 4381 len 20480
32080 253/300 Write block 2392 len 28672 data: 7 8 8 6 6 6 3
32081 254/300 Read block 1801 len 28672
32082 255/300 Write block 1853 len 24576 data: 5 6 8 6 5 4
32083 256/300 Write block 2120 len 16384 data: 4 5 5 7
32084 257/300 Write block 515 len 28672 data: 4 5 6 4 6 8 8
32085 258/300 Write block 4708 len 36864 data: 6 4 6 6 5 5 5 4 5
32086 259/300 Read block 3070 len 20480
32087 260/300 Write block 3718 len 32768 data: 4 4 5 5 6 6 5 5
32088 261/300 Write block 1350 len 8192 data: 4 4
32089 262/300 Read block 714 len 4096
32090 263/300 Write block 2327 len 20480 data: 9 8 8 7 6
32091 264/300 Read block 1084 len 20480
32092 265/300 Read block 4698 len 28672
32093 266/300 Read block 2396 len 24576
32094 267/300 Read block 4354 len 8192
32095 268/300 Read block 3165 len 16384
32096 269/300 Write block 2872 len 4096 data: 7
32097 270/300 Read block 2752 len 16384
32098 271/300 Write block 3118 len 4096 data: 7
32099 272/300 Read block 450 len 8192
32100 273/300 Write block 24 len 12288 data: 6 7 6
32101 274/300 Read block 2592 len 20480
32102 275/300 Read block 4928 len 36864
32103 276/300 Write block 4053 len 12288 data: 5 6 7
32104 277/300 Write block 119 len 24576 data: 6 7 7 8 7 5
32105 278/300 Flush
32106 279/300 Write block 1212 len 4096 data: 8
32107 280/300 Read block 4685 len 12288
32108 281/300 Read block 460 len 12288
32109 282/300 Read block 2300 len 36864
32110 283/300 Read block 4274 len 40960
32111 284/300 Write block 4590 len 40960 data: 7 6 7 5 5 3 4 3 4 4
32112 285/300 Read block 1245 len 4096
32113 286/300 Write block 4544 len 24576 data: 2 2 3 4 4 4
32114 287/300 Write block 2040 len 4096 data: 7
32115 288/300 Write block 1706 len 8192 data: 4 2
32116 289/300 Write block 4261 len 40960 data: 2 3 3 4 4 5 6 6 6 5
32117 290/300 Read block 4340 len 32768
32118 291/300 Flush
32119 292/300 Write block 4338 len 36864 data: 4 4 4 2 4 4 4 5 5
32120 293/300 Read block 1597 len 28672
32121 294/300 Write block 4864 len 36864 data: 4 5 5 5 3 5 5 4 6
32122 295/300 Read block 4095 len 20480
32123 296/300 Read block 3301 len 32768
32124 297/300 Write block 1752 len 36864 data: 6 6 5 6 6 6 8 8 8
32125 298/300 Write block 3489 len 12288 data: 5 6 7
32126 299/300 Write block 1466 len 36864 data: 11 8 8 7 6 5 6 4 3
32127 300/300 Read block 262 len 24576
32128 301/300 Read block 2979 len 4096
32129 Main task has work to do, go find it
32130 got dsc Start(0)
32131 start 0
32132 Feb 21 06:52:30.336 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/0, method: POST, req_id: 0a73d57b-19f0-40b3-943a-f3f2c1c12f02, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
32133 Replay: started 0, returned:Ok(())
32134 ----------------------------------------------------------------
32135 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32136 States: Offline Active Active
32137 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32138 9150 Acked Read 3 Sent Done Done false
32139 9151 Acked Write 4 Sent Done Done false
32140 9152 Acked Read 6 Sent Done Done false
32141 9153 Acked Write 6 Sent Done Done false
32142 9154 Acked Read 9 Sent Done Done false
32143 9155 Acked Read 5 Sent Done Done false
32144 9156 Acked Write 1 Sent Done Done false
32145 9157 Acked Flush 0 Sent Done Done false
32146 9158 Acked Write 1 Sent Done Done false
32147 9159 Acked Flush 0 Sent Done Done false
32148 9160 Acked Write 6 Sent Done Done false
32149 9161 Acked Read 4 Sent Done Done false
32150 9162 Acked Read 2 Sent Done Done false
32151 9163 Acked Read 5 Sent Done Done false
32152 9164 Acked Write 5 Sent Done Done false
32153 9165 Acked Flush 0 Sent Done Done false
32154 9166 Acked Write 10 Sent Done Done false
32155 9167 Acked Write 2 Sent Done Done false
32156 9168 Acked Read 5 Sent Done Done false
32157 9169 Acked Read 4 Sent Done Done false
32158 9170 Acked Read 8 Sent Done Done false
32159 9171 Acked Write 1 Sent Done Done false
32160 9172 Acked Read 1 Sent Done Done false
32161 9173 Acked Read 10 Sent Done Done false
32162 9174 Acked Write 5 Sent Done Done false
32163 9175 Acked Flush 0 Sent Done Done false
32164 9176 Acked Read 6 Sent Done Done false
32165 9177 Acked Read 4 Sent Done Done false
32166 9178 Acked Write 5 Sent Done Done false
32167 9179 Acked Write 6 Sent Done Done false
32168 9180 Acked Flush 0 Sent Done Done false
32169 9181 Acked Write 3 Sent Done Done false
32170 9182 Acked Write 4 Sent Done Done false
32171 9183 Acked Write 9 Sent Done Done false
32172 9184 Acked Write 3 Sent Done Done false
32173 9185 Acked Write 5 Sent Done Done false
32174 9186 Acked Write 1 Sent Done Done false
32175 9187 Acked Write 2 Sent Done Done false
32176 9188 Acked Write 9 Sent Done Done false
32177 9189 Acked Write 1 Sent Done Done false
32178 9190 Acked Read 1 Sent Done Done false
32179 9191 Acked Read 9 Sent Done Done false
32180 9192 Acked Write 10 Sent Done Done false
32181 9193 Acked Read 2 Sent Done Done false
32182 9194 Acked Read 5 Sent Done Done false
32183 9195 Acked Read 8 Sent Done Done false
32184 9196 Acked Write 9 Sent Done Done false
32185 9197 Acked Read 3 Sent Done Done false
32186 9198 Acked Write 1 Sent Done Done false
32187 9199 Acked Read 6 Sent Done Done false
32188 9200 Acked Write 8 Sent Done Done false
32189 9201 Acked Write 7 Sent Done Done false
32190 9202 Acked Read 9 Sent Done Done false
32191 9203 Acked Read 1 Sent Done Done false
32192 9204 Acked Read 5 Sent Done Done false
32193 9205 Acked Write 2 Sent Done Done false
32194 9206 Acked Write 4 Sent Done Done false
32195 9207 Acked Write 8 Sent Done Done false
32196 9208 Acked Read 9 Sent Done Done false
32197 9209 Acked Write 9 Sent Done Done false
32198 9210 Acked Write 3 Sent Done Done false
32199 9211 Acked Read 10 Sent Done Done false
32200 9212 Acked Write 10 Sent Done Done false
32201 9213 Acked Read 6 Sent Done Done false
32202 9214 Acked Flush 0 Sent Done Done false
32203 9215 Acked Read 3 Sent Done Done false
32204 9216 Acked Read 6 Sent Done Done false
32205 9217 Acked Read 8 Sent Done Done false
32206 9218 Acked Flush 0 Sent Done Done false
32207 9219 Acked Read 1 Sent Done Done false
32208 9220 Acked Flush 0 Sent Done Done false
32209 9221 Acked Write 8 Sent Done Done false
32210 9222 Acked Read 3 Sent Done Done false
32211 9223 Acked Read 3 Sent Done Done false
32212 9224 Acked Write 3 Sent Done Done false
32213 9225 Acked Read 9 Sent Done Done false
32214 9226 Acked Flush 0 Sent Done Done false
32215 9227 Acked Read 3 Sent Done Done false
32216 9228 Acked Read 1 Sent Done Done false
32217 9229 Acked Read 10 Sent Done Done false
32218 9230 Acked Write 2 Sent Done Done false
32219 9231 Acked Read 8 Sent Done Done false
32220 9232 Acked Write 8 Sent Done Done false
32221 9233 Acked Write 2 Sent Done Done false
32222 9234 Acked Write 2 Sent Done Done false
32223 9235 Acked Write 2 Sent Done Done false
32224 9236 Acked Flush 0 Sent Done Done false
32225 9237 Acked Read 9 Sent Done Done false
32226 9238 Acked Flush 0 Sent Done Done false
32227 9239 Acked Read 2 Sent Done Done false
32228 9240 Acked Write 7 Sent Done Done false
32229 9241 Acked Write 5 Sent Done Done false
32230 9242 Acked Read 9 Sent Done Done false
32231 9243 Acked Read 8 Sent Done Done false
32232 9244 Acked Read 6 Sent Done Done false
32233 9245 Acked Read 6 Sent Done Done false
32234 9246 Acked Write 6 Sent Done Done false
32235 9247 Acked Write 4 Sent Done Done false
32236 9248 Acked Read 5 Sent Done Done false
32237 9249 Acked Write 2 Sent Done Done false
32238 9250 Acked Write 5 Sent Done Done false
32239 9251 Acked Write 3 Sent Done Done false
32240 9252 Acked Write 8 Sent Done Done false
32241 9253 Acked Read 5 Sent Done Done false
32242 9254 Acked Write 8 Sent Done Done false
32243 9255 Acked Write 5 Sent Done Done false
32244 9256 Acked Write 9 Sent Done Done false
32245 9257 Acked Write 10 Sent Done Done false
32246 9258 Acked Read 8 Sent Done Done false
32247 9259 Acked Write 2 Sent Done Done false
32248 9260 Acked Read 6 Sent Done Done false
32249 9261 Acked Write 3 Sent Done Done false
32250 9262 Acked Read 5 Sent Done Done false
32251 9263 Acked Write 10 Sent Done Done false
32252 9264 Acked Read 3 Sent Done Done false
32253 9265 Acked Write 10 Sent Done Done false
32254 9266 Acked Flush 0 Sent Done Done false
32255 9267 Acked Read 6 Sent Done Done false
32256 9268 Acked Read 9 Sent Done Done false
32257 9269 Acked Write 2 Sent Done Done false
32258 9270 Acked Read 5 Sent Done Done false
32259 9271 Acked Write 7 Sent Done Done false
32260 9272 Acked Write 1 Sent Done Done false
32261 9273 Acked Write 6 Sent Done Done false
32262 9274 Acked Write 8 Sent Done Done false
32263 9275 Acked Read 1 Sent Done Done false
32264 9276 Acked Read 10 Sent Done Done false
32265 9277 Acked Write 4 Sent Done Done false
32266 9278 Acked Read 5 Sent Done Done false
32267 9279 Acked Read 1 Sent Done Done false
32268 9280 Acked Read 9 Sent Done Done false
32269 9281 Acked Flush 0 Sent Done Done false
32270 9282 Acked Read 3 Sent Done Done false
32271 9283 Acked Write 9 Sent Done Done false
32272 9284 Acked Read 7 Sent Done Done false
32273 9285 Acked Write 3 Sent Done Done false
32274 9286 Acked Read 10 Sent Done Done false
32275 9287 Acked Read 7 Sent Done Done false
32276 9288 Acked Read 3 Sent Done Done false
32277 9289 Acked Write 7 Sent Done Done false
32278 9290 Acked Read 6 Sent Done Done false
32279 9291 Acked Write 2 Sent Done Done false
32280 9292 Acked Write 2 Sent Done Done false
32281 9293 Acked Read 3 Sent Done Done false
32282 9294 Acked Flush 0 Sent Done Done false
32283 9295 Acked Write 8 Sent Done Done false
32284 9296 Acked Write 4 Sent Done Done false
32285 9297 Acked Read 10 Sent Done Done false
32286 9298 Acked Write 9 Sent Done Done false
32287 9299 Acked Write 7 Sent Done Done false
32288 9300 Acked Read 9 Sent Done Done false
32289 9301 Acked Write 4 Sent Done Done false
32290 9302 Acked Read 2 Sent Done Done false
32291 9303 Acked Read 5 Sent Done Done false
32292 9304 Acked Write 4 Sent Done Done false
32293 9305 Acked Write 6 Sent Done Done false
32294 9306 Acked Read 9 Sent Done Done false
32295 9307 Acked Read 4 Sent Done Done false
32296 9308 Acked Read 8 Sent Done Done false
32297 9309 Acked Read 7 Sent Done Done false
32298 9310 Acked Read 1 Sent Done Done false
32299 9311 Acked Read 4 Sent Done Done false
32300 9312 Acked Read 6 Sent Done Done false
32301 9313 Acked Read 8 Sent Done Done false
32302 9314 Acked Write 5 Sent Done Done false
32303 9315 Acked Read 3 Sent Done Done false
32304 9316 Acked Write 10 Sent Done Done false
32305 9317 Acked Flush 0 Sent Done Done false
32306 9318 Acked Read 1 Sent Done Done false
32307 9319 Acked Write 4 Sent Done Done false
32308 9320 Acked Read 9 Sent Done Done false
32309 9321 Acked Read 8 Sent Done Done false
32310 9322 Acked Read 9 Sent Done Done false
32311 9323 Acked Read 5 Sent Done Done false
32312 9324 Acked Read 8 Sent Done Done false
32313 9325 Acked Write 6 Sent Done Done false
32314 9326 Acked Read 10 Sent Done Done false
32315 9327 Acked Read 1 Sent Done Done false
32316 9328 Acked Read 5 Sent Done Done false
32317 9329 Acked Write 10 Sent Done Done false
32318 9330 Acked Flush 0 Sent Done Done false
32319 9331 Acked Write 6 Sent Done Done false
32320 9332 Acked Read 7 Sent Done Done false
32321 9333 Acked Read 2 Sent Done Done false
32322 9334 Acked Write 2 Sent Done Done false
32323 9335 Acked Read 7 Sent Done Done false
32324 9336 Acked Read 6 Sent Done Done false
32325 9337 Acked Read 10 Sent Done Done false
32326 9338 Acked Read 1 Sent Done Done false
32327 9339 Acked Write 5 Sent Done Done false
32328 9340 Acked Read 5 Sent Done Done false
32329 9341 Acked Read 10 Sent Done Done false
32330 9342 Acked Read 3 Sent Done Done false
32331 9343 Acked Write 9 Sent Done Done false
32332 9344 Acked Read 2 Sent Done Done false
32333 9345 Acked Write 9 Sent Done Done false
32334 9346 Acked Write 7 Sent Done Done false
32335 9347 Acked Write 3 Sent Done Done false
32336 9348 Acked Write 9 Sent Done Done false
32337 9349 Acked Read 4 Sent Done Done false
32338 9350 Acked Write 1 Sent Done Done false
32339 9351 Acked Write 8 Sent Done Done false
32340 9352 Acked Flush 0 Sent Done Done false
32341 9353 Acked Write 8 Sent Done Done false
32342 9354 Acked Read 4 Sent Done Done false
32343 9355 Acked Write 5 Sent Done Done false
32344 9356 Acked Write 7 Sent Done Done false
32345 9357 Acked Flush 0 Sent Done Done false
32346 9358 Acked Write 6 Sent Done Done false
32347 9359 Acked Write 7 Sent Done Done false
32348 9360 Acked Read 4 Sent Done Done false
32349 9361 Acked Read 7 Sent Done Done false
32350 9362 Acked Read 2 Sent Done Done false
32351 9363 Acked Write 4 Sent Done Done false
32352 9364 Acked Read 3 Sent Done Done false
32353 9365 Acked Write 7 Sent Done Done false
32354 9366 Acked Write 10 Sent Done Done false
32355 9367 Acked Read 10 Sent Done Done false
32356 9368 Acked Read 3 Sent Done Done false
32357 9369 Acked Write 1 Sent Done Done false
32358 9370 Acked Write 7 Sent Done Done false
32359 9371 Acked Write 2 Sent Done Done false
32360 9372 Acked Read 9 Sent Done Done false
32361 9373 Acked Read 7 Sent Done Done false
32362 9374 Acked Write 7 Sent Done Done false
32363 9375 Acked Flush 0 Sent Done Done false
32364 9376 Acked Read 9 Sent Done Done false
32365 9377 Acked Write 9 Sent Done Done false
32366 9378 Acked Read 6 Sent Done Done false
32367 9379 Acked Read 1 Sent Done Done false
32368 9380 Acked Write 6 Sent Done Done false
32369 9381 Acked Write 5 Sent Done Done false
32370 9382 Acked Read 3 Sent Done Done false
32371 9383 Acked Flush 0 Sent Done Done false
32372 9384 Acked Write 7 Sent Done Done false
32373 9385 Acked Flush 0 Sent Done Done false
32374 9386 Acked Read 3 Sent Done Done false
32375 9387 Acked Read 2 Sent Done Done false
32376 9388 Acked Read 7 Sent Done Done false
32377 9389 Acked Flush 0 Sent Done Done false
32378 9390 Acked Write 7 Sent Done Done false
32379 9391 Acked Read 1 Sent Done Done false
32380 9392 Acked Write 10 Sent Done Done false
32381 9393 Acked Read 2 Sent Done Done false
32382 9394 Acked Write 6 Sent Done Done false
32383 9395 Acked Flush 0 Sent Done Done false
32384 9396 Acked Write 1 Sent Done Done false
32385 9397 Acked Write 1 Sent Done Done false
32386 9398 Acked Read 5 Sent Done Done false
32387 9399 Acked Read 2 Sent Done Done false
32388 9400 Acked Read 2 Sent Done Done false
32389 9401 Acked Read 5 Sent Done Done false
32390 9402 Acked Write 7 Sent Done Done false
32391 9403 Acked Read 7 Sent Done Done false
32392 9404 Acked Write 6 Sent Done Done false
32393 9405 Acked Write 4 Sent Done Done false
32394 9406 Acked Write 7 Sent Done Done false
32395 9407 Acked Write 9 Sent Done Done false
32396 9408 Acked Read 5 Sent Done Done false
32397 9409 Acked Write 8 Sent Done Done false
32398 9410 Acked Write 2 Sent Done Done false
32399 9411 Acked Read 1 Sent Done Done false
32400 9412 Acked Write 5 Sent Done Done false
32401 9413 Acked Read 5 Sent Done Done false
32402 9414 Acked Read 7 Sent Done Done false
32403 9415 Acked Read 6 Sent Done Done false
32404 9416 Acked Read 2 Sent Done Done false
32405 9417 Acked Read 4 Sent Done Done false
32406 9418 Acked Write 1 Sent Done Done false
32407 9419 Acked Read 4 Sent Done Done false
32408 9420 Acked Write 1 Sent Done Done false
32409 9421 Acked Read 2 Sent Done Done false
32410 9422 Acked Write 3 Sent Done Done false
32411 9423 Acked Read 5 Sent Done Done false
32412 9424 Acked Read 9 Sent Done Done false
32413 9425 Acked Write 3 Sent Done Done false
32414 9426 Acked Write 6 Sent Done Done false
32415 9427 Acked Flush 0 Sent Done Done false
32416 9428 Acked Write 1 Sent Done Done false
32417 9429 Acked Read 3 Sent Done Done false
32418 9430 Acked Read 3 Sent Done Done false
32419 9431 Acked Read 9 Sent Done Done false
32420 9432 Acked Read 10 Sent Done Done false
32421 9433 Acked Write 10 Sent Done Done false
32422 9434 Acked Read 1 Sent Done Done false
32423 9435 Acked Write 6 Sent Done Done false
32424 9436 Acked Write 1 Sent Done Done false
32425 9437 Acked Write 2 Sent Done Done false
32426 9438 Acked Write 10 Sent Done Done false
32427 9439 Acked Read 8 Sent Done Done false
32428 9440 Acked Flush 0 Sent Done Done false
32429 9441 Acked Write 9 Sent Done Done false
32430 9442 Acked Read 7 Sent Done Done false
32431 9443 Acked Write 9 Sent Done Done false
32432 9444 Acked Read 5 Sent Done Done false
32433 9445 Acked Read 8 Sent Done Done false
32434 9446 Acked Write 9 Sent Done Done false
32435 9447 Acked Write 3 Sent Done Done false
32436 9448 Acked Write 9 Sent Done Done false
32437 9449 Acked Read 6 Sent Done Done false
32438 9450 Acked Read 1 Sent Done Done false
32439 STATES DS:0 DS:1 DS:2 TOTAL
32440 Sent 301 0 0 301
32441 Done 0 301 301 602
32442 Skipped 0 0 0 0
32443 Error 0 0 0 0
32444 Last Flush: Some(JobId(9149)) Some(JobId(9440)) Some(JobId(9440))
32445 Downstairs last five completed: 9149 9148 9147 9146 9145
32446 Upstairs last five completed: 9450 9449 9448 9447 9446
32447 CLIENT: Up:0 ds:301 act:2
324482025-02-21T06:52:31.313ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
324492025-02-21T06:52:32.315ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
324502025-02-21T06:52:33.806ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
324512025-02-21T06:52:34.318ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
32452 ----------------------------------------------------------------
32453 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32454 States: Offline Active Active
32455 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32456 9150 Acked Read 3 Sent Done Done false
32457 9151 Acked Write 4 Sent Done Done false
32458 9152 Acked Read 6 Sent Done Done false
32459 9153 Acked Write 6 Sent Done Done false
32460 9154 Acked Read 9 Sent Done Done false
32461 9155 Acked Read 5 Sent Done Done false
32462 9156 Acked Write 1 Sent Done Done false
32463 9157 Acked Flush 0 Sent Done Done false
32464 9158 Acked Write 1 Sent Done Done false
32465 9159 Acked Flush 0 Sent Done Done false
32466 9160 Acked Write 6 Sent Done Done false
32467 9161 Acked Read 4 Sent Done Done false
32468 9162 Acked Read 2 Sent Done Done false
32469 9163 Acked Read 5 Sent Done Done false
32470 9164 Acked Write 5 Sent Done Done false
32471 9165 Acked Flush 0 Sent Done Done false
32472 9166 Acked Write 10 Sent Done Done false
32473 9167 Acked Write 2 Sent Done Done false
32474 9168 Acked Read 5 Sent Done Done false
32475 9169 Acked Read 4 Sent Done Done false
32476 9170 Acked Read 8 Sent Done Done false
32477 9171 Acked Write 1 Sent Done Done false
32478 9172 Acked Read 1 Sent Done Done false
32479 9173 Acked Read 10 Sent Done Done false
32480 9174 Acked Write 5 Sent Done Done false
32481 9175 Acked Flush 0 Sent Done Done false
32482 9176 Acked Read 6 Sent Done Done false
32483 9177 Acked Read 4 Sent Done Done false
32484 9178 Acked Write 5 Sent Done Done false
32485 9179 Acked Write 6 Sent Done Done false
32486 9180 Acked Flush 0 Sent Done Done false
32487 9181 Acked Write 3 Sent Done Done false
32488 9182 Acked Write 4 Sent Done Done false
32489 9183 Acked Write 9 Sent Done Done false
32490 9184 Acked Write 3 Sent Done Done false
32491 9185 Acked Write 5 Sent Done Done false
32492 9186 Acked Write 1 Sent Done Done false
32493 9187 Acked Write 2 Sent Done Done false
32494 9188 Acked Write 9 Sent Done Done false
32495 9189 Acked Write 1 Sent Done Done false
32496 9190 Acked Read 1 Sent Done Done false
32497 9191 Acked Read 9 Sent Done Done false
32498 9192 Acked Write 10 Sent Done Done false
32499 9193 Acked Read 2 Sent Done Done false
32500 9194 Acked Read 5 Sent Done Done false
32501 9195 Acked Read 8 Sent Done Done false
32502 9196 Acked Write 9 Sent Done Done false
32503 9197 Acked Read 3 Sent Done Done false
32504 9198 Acked Write 1 Sent Done Done false
32505 9199 Acked Read 6 Sent Done Done false
32506 9200 Acked Write 8 Sent Done Done false
32507 9201 Acked Write 7 Sent Done Done false
32508 9202 Acked Read 9 Sent Done Done false
32509 9203 Acked Read 1 Sent Done Done false
32510 9204 Acked Read 5 Sent Done Done false
32511 9205 Acked Write 2 Sent Done Done false
32512 9206 Acked Write 4 Sent Done Done false
32513 9207 Acked Write 8 Sent Done Done false
32514 9208 Acked Read 9 Sent Done Done false
32515 9209 Acked Write 9 Sent Done Done false
32516 9210 Acked Write 3 Sent Done Done false
32517 9211 Acked Read 10 Sent Done Done false
32518 9212 Acked Write 10 Sent Done Done false
32519 9213 Acked Read 6 Sent Done Done false
32520 9214 Acked Flush 0 Sent Done Done false
32521 9215 Acked Read 3 Sent Done Done false
32522 9216 Acked Read 6 Sent Done Done false
32523 9217 Acked Read 8 Sent Done Done false
32524 9218 Acked Flush 0 Sent Done Done false
32525 9219 Acked Read 1 Sent Done Done false
32526 9220 Acked Flush 0 Sent Done Done false
32527 9221 Acked Write 8 Sent Done Done false
32528 9222 Acked Read 3 Sent Done Done false
32529 9223 Acked Read 3 Sent Done Done false
32530 9224 Acked Write 3 Sent Done Done false
32531 9225 Acked Read 9 Sent Done Done false
32532 9226 Acked Flush 0 Sent Done Done false
32533 9227 Acked Read 3 Sent Done Done false
32534 9228 Acked Read 1 Sent Done Done false
32535 9229 Acked Read 10 Sent Done Done false
32536 9230 Acked Write 2 Sent Done Done false
32537 9231 Acked Read 8 Sent Done Done false
32538 9232 Acked Write 8 Sent Done Done false
32539 9233 Acked Write 2 Sent Done Done false
32540 9234 Acked Write 2 Sent Done Done false
32541 9235 Acked Write 2 Sent Done Done false
32542 9236 Acked Flush 0 Sent Done Done false
32543 9237 Acked Read 9 Sent Done Done false
32544 9238 Acked Flush 0 Sent Done Done false
32545 9239 Acked Read 2 Sent Done Done false
32546 9240 Acked Write 7 Sent Done Done false
32547 9241 Acked Write 5 Sent Done Done false
32548 9242 Acked Read 9 Sent Done Done false
32549 9243 Acked Read 8 Sent Done Done false
32550 9244 Acked Read 6 Sent Done Done false
32551 9245 Acked Read 6 Sent Done Done false
32552 9246 Acked Write 6 Sent Done Done false
32553 9247 Acked Write 4 Sent Done Done false
32554 9248 Acked Read 5 Sent Done Done false
32555 9249 Acked Write 2 Sent Done Done false
32556 9250 Acked Write 5 Sent Done Done false
32557 9251 Acked Write 3 Sent Done Done false
32558 9252 Acked Write 8 Sent Done Done false
32559 9253 Acked Read 5 Sent Done Done false
32560 9254 Acked Write 8 Sent Done Done false
32561 9255 Acked Write 5 Sent Done Done false
32562 9256 Acked Write 9 Sent Done Done false
32563 9257 Acked Write 10 Sent Done Done false
32564 9258 Acked Read 8 Sent Done Done false
32565 9259 Acked Write 2 Sent Done Done false
32566 9260 Acked Read 6 Sent Done Done false
32567 9261 Acked Write 3 Sent Done Done false
32568 9262 Acked Read 5 Sent Done Done false
32569 9263 Acked Write 10 Sent Done Done false
32570 9264 Acked Read 3 Sent Done Done false
32571 9265 Acked Write 10 Sent Done Done false
32572 9266 Acked Flush 0 Sent Done Done false
32573 9267 Acked Read 6 Sent Done Done false
32574 9268 Acked Read 9 Sent Done Done false
32575 9269 Acked Write 2 Sent Done Done false
32576 9270 Acked Read 5 Sent Done Done false
32577 9271 Acked Write 7 Sent Done Done false
32578 9272 Acked Write 1 Sent Done Done false
32579 9273 Acked Write 6 Sent Done Done false
32580 9274 Acked Write 8 Sent Done Done false
32581 9275 Acked Read 1 Sent Done Done false
32582 9276 Acked Read 10 Sent Done Done false
32583 9277 Acked Write 4 Sent Done Done false
32584 9278 Acked Read 5 Sent Done Done false
32585 9279 Acked Read 1 Sent Done Done false
32586 9280 Acked Read 9 Sent Done Done false
32587 9281 Acked Flush 0 Sent Done Done false
32588 9282 Acked Read 3 Sent Done Done false
32589 9283 Acked Write 9 Sent Done Done false
32590 9284 Acked Read 7 Sent Done Done false
32591 9285 Acked Write 3 Sent Done Done false
32592 9286 Acked Read 10 Sent Done Done false
32593 9287 Acked Read 7 Sent Done Done false
32594 9288 Acked Read 3 Sent Done Done false
32595 9289 Acked Write 7 Sent Done Done false
32596 9290 Acked Read 6 Sent Done Done false
32597 9291 Acked Write 2 Sent Done Done false
32598 9292 Acked Write 2 Sent Done Done false
32599 9293 Acked Read 3 Sent Done Done false
32600 9294 Acked Flush 0 Sent Done Done false
32601 9295 Acked Write 8 Sent Done Done false
32602 9296 Acked Write 4 Sent Done Done false
32603 9297 Acked Read 10 Sent Done Done false
32604 9298 Acked Write 9 Sent Done Done false
32605 9299 Acked Write 7 Sent Done Done false
32606 9300 Acked Read 9 Sent Done Done false
32607 9301 Acked Write 4 Sent Done Done false
32608 9302 Acked Read 2 Sent Done Done false
32609 9303 Acked Read 5 Sent Done Done false
32610 9304 Acked Write 4 Sent Done Done false
32611 9305 Acked Write 6 Sent Done Done false
32612 9306 Acked Read 9 Sent Done Done false
32613 9307 Acked Read 4 Sent Done Done false
32614 9308 Acked Read 8 Sent Done Done false
32615 9309 Acked Read 7 Sent Done Done false
32616 9310 Acked Read 1 Sent Done Done false
32617 9311 Acked Read 4 Sent Done Done false
32618 9312 Acked Read 6 Sent Done Done false
32619 9313 Acked Read 8 Sent Done Done false
32620 9314 Acked Write 5 Sent Done Done false
32621 9315 Acked Read 3 Sent Done Done false
32622 9316 Acked Write 10 Sent Done Done false
32623 9317 Acked Flush 0 Sent Done Done false
32624 9318 Acked Read 1 Sent Done Done false
32625 9319 Acked Write 4 Sent Done Done false
32626 9320 Acked Read 9 Sent Done Done false
32627 9321 Acked Read 8 Sent Done Done false
32628 9322 Acked Read 9 Sent Done Done false
32629 9323 Acked Read 5 Sent Done Done false
32630 9324 Acked Read 8 Sent Done Done false
32631 9325 Acked Write 6 Sent Done Done false
32632 9326 Acked Read 10 Sent Done Done false
32633 9327 Acked Read 1 Sent Done Done false
32634 9328 Acked Read 5 Sent Done Done false
32635 9329 Acked Write 10 Sent Done Done false
32636 9330 Acked Flush 0 Sent Done Done false
32637 9331 Acked Write 6 Sent Done Done false
32638 9332 Acked Read 7 Sent Done Done false
32639 9333 Acked Read 2 Sent Done Done false
32640 9334 Acked Write 2 Sent Done Done false
32641 9335 Acked Read 7 Sent Done Done false
32642 9336 Acked Read 6 Sent Done Done false
32643 9337 Acked Read 10 Sent Done Done false
32644 9338 Acked Read 1 Sent Done Done false
32645 9339 Acked Write 5 Sent Done Done false
32646 9340 Acked Read 5 Sent Done Done false
32647 9341 Acked Read 10 Sent Done Done false
32648 9342 Acked Read 3 Sent Done Done false
32649 9343 Acked Write 9 Sent Done Done false
32650 9344 Acked Read 2 Sent Done Done false
32651 9345 Acked Write 9 Sent Done Done false
32652 9346 Acked Write 7 Sent Done Done false
32653 9347 Acked Write 3 Sent Done Done false
32654 9348 Acked Write 9 Sent Done Done false
32655 9349 Acked Read 4 Sent Done Done false
32656 9350 Acked Write 1 Sent Done Done false
32657 9351 Acked Write 8 Sent Done Done false
32658 9352 Acked Flush 0 Sent Done Done false
32659 9353 Acked Write 8 Sent Done Done false
32660 9354 Acked Read 4 Sent Done Done false
32661 9355 Acked Write 5 Sent Done Done false
32662 9356 Acked Write 7 Sent Done Done false
32663 9357 Acked Flush 0 Sent Done Done false
32664 9358 Acked Write 6 Sent Done Done false
32665 9359 Acked Write 7 Sent Done Done false
32666 9360 Acked Read 4 Sent Done Done false
32667 9361 Acked Read 7 Sent Done Done false
32668 9362 Acked Read 2 Sent Done Done false
32669 9363 Acked Write 4 Sent Done Done false
32670 9364 Acked Read 3 Sent Done Done false
32671 9365 Acked Write 7 Sent Done Done false
32672 9366 Acked Write 10 Sent Done Done false
32673 9367 Acked Read 10 Sent Done Done false
32674 9368 Acked Read 3 Sent Done Done false
32675 9369 Acked Write 1 Sent Done Done false
32676 9370 Acked Write 7 Sent Done Done false
32677 9371 Acked Write 2 Sent Done Done false
32678 9372 Acked Read 9 Sent Done Done false
32679 9373 Acked Read 7 Sent Done Done false
32680 9374 Acked Write 7 Sent Done Done false
32681 9375 Acked Flush 0 Sent Done Done false
32682 9376 Acked Read 9 Sent Done Done false
32683 9377 Acked Write 9 Sent Done Done false
32684 9378 Acked Read 6 Sent Done Done false
32685 9379 Acked Read 1 Sent Done Done false
32686 9380 Acked Write 6 Sent Done Done false
32687 9381 Acked Write 5 Sent Done Done false
32688 9382 Acked Read 3 Sent Done Done false
32689 9383 Acked Flush 0 Sent Done Done false
32690 9384 Acked Write 7 Sent Done Done false
32691 9385 Acked Flush 0 Sent Done Done false
32692 9386 Acked Read 3 Sent Done Done false
32693 9387 Acked Read 2 Sent Done Done false
32694 9388 Acked Read 7 Sent Done Done false
32695 9389 Acked Flush 0 Sent Done Done false
32696 9390 Acked Write 7 Sent Done Done false
32697 9391 Acked Read 1 Sent Done Done false
32698 9392 Acked Write 10 Sent Done Done false
32699 9393 Acked Read 2 Sent Done Done false
32700 9394 Acked Write 6 Sent Done Done false
32701 9395 Acked Flush 0 Sent Done Done false
32702 9396 Acked Write 1 Sent Done Done false
32703 9397 Acked Write 1 Sent Done Done false
32704 9398 Acked Read 5 Sent Done Done false
32705 9399 Acked Read 2 Sent Done Done false
32706 9400 Acked Read 2 Sent Done Done false
32707 9401 Acked Read 5 Sent Done Done false
32708 9402 Acked Write 7 Sent Done Done false
32709 9403 Acked Read 7 Sent Done Done false
32710 9404 Acked Write 6 Sent Done Done false
32711 9405 Acked Write 4 Sent Done Done false
32712 9406 Acked Write 7 Sent Done Done false
32713 9407 Acked Write 9 Sent Done Done false
32714 9408 Acked Read 5 Sent Done Done false
32715 9409 Acked Write 8 Sent Done Done false
32716 9410 Acked Write 2 Sent Done Done false
32717 9411 Acked Read 1 Sent Done Done false
32718 9412 Acked Write 5 Sent Done Done false
32719 9413 Acked Read 5 Sent Done Done false
32720 9414 Acked Read 7 Sent Done Done false
32721 9415 Acked Read 6 Sent Done Done false
32722 9416 Acked Read 2 Sent Done Done false
32723 9417 Acked Read 4 Sent Done Done false
32724 9418 Acked Write 1 Sent Done Done false
32725 9419 Acked Read 4 Sent Done Done false
32726 9420 Acked Write 1 Sent Done Done false
32727 9421 Acked Read 2 Sent Done Done false
32728 9422 Acked Write 3 Sent Done Done false
32729 9423 Acked Read 5 Sent Done Done false
32730 9424 Acked Read 9 Sent Done Done false
32731 9425 Acked Write 3 Sent Done Done false
32732 9426 Acked Write 6 Sent Done Done false
32733 9427 Acked Flush 0 Sent Done Done false
32734 9428 Acked Write 1 Sent Done Done false
32735 9429 Acked Read 3 Sent Done Done false
32736 9430 Acked Read 3 Sent Done Done false
32737 9431 Acked Read 9 Sent Done Done false
32738 9432 Acked Read 10 Sent Done Done false
32739 9433 Acked Write 10 Sent Done Done false
32740 9434 Acked Read 1 Sent Done Done false
32741 9435 Acked Write 6 Sent Done Done false
32742 9436 Acked Write 1 Sent Done Done false
32743 9437 Acked Write 2 Sent Done Done false
32744 9438 Acked Write 10 Sent Done Done false
32745 9439 Acked Read 8 Sent Done Done false
32746 9440 Acked Flush 0 Sent Done Done false
32747 9441 Acked Write 9 Sent Done Done false
32748 9442 Acked Read 7 Sent Done Done false
32749 9443 Acked Write 9 Sent Done Done false
32750 9444 Acked Read 5 Sent Done Done false
32751 9445 Acked Read 8 Sent Done Done false
32752 9446 Acked Write 9 Sent Done Done false
32753 9447 Acked Write 3 Sent Done Done false
32754 9448 Acked Write 9 Sent Done Done false
32755 9449 Acked Read 6 Sent Done Done false
32756 9450 Acked Read 1 Sent Done Done false
32757 STATES DS:0 DS:1 DS:2 TOTAL
32758 Sent 301 0 0 301
32759 Done 0 301 301 602
32760 Skipped 0 0 0 0
32761 Error 0 0 0 0
32762 Last Flush: Some(JobId(9149)) Some(JobId(9440)) Some(JobId(9440))
32763 Downstairs last five completed: 9149 9148 9147 9146 9145
32764 Upstairs last five completed: 9450 9449 9448 9447 9446
32765 CLIENT: Up:0 ds:301 act:2
327662025-02-21T06:52:35.320ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
327672025-02-21T06:52:36.096ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
327682025-02-21T06:52:36.096ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
327692025-02-21T06:52:36.096ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
327702025-02-21T06:52:36.096ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
327712025-02-21T06:52:36.096ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
32772 [8810] got start action
32773 [8810] I am going to restart
32774 Starting downstairs at port 8810
32775 Make output file at "/tmp/dsc/downstairs-8810.txt"
32776 [8810][0] reports Starting
32777 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1234)
32778 [8810][0] reports Running
327792025-02-21T06:52:37.324ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
32780 ----------------------------------------------------------------
32781 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32782 States: Offline Active Active
32783 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32784 9150 Acked Read 3 Sent Done Done false
32785 9151 Acked Write 4 Sent Done Done false
32786 9152 Acked Read 6 Sent Done Done false
32787 9153 Acked Write 6 Sent Done Done false
32788 9154 Acked Read 9 Sent Done Done false
32789 9155 Acked Read 5 Sent Done Done false
32790 9156 Acked Write 1 Sent Done Done false
32791 9157 Acked Flush 0 Sent Done Done false
32792 9158 Acked Write 1 Sent Done Done false
32793 9159 Acked Flush 0 Sent Done Done false
32794 9160 Acked Write 6 Sent Done Done false
32795 9161 Acked Read 4 Sent Done Done false
32796 9162 Acked Read 2 Sent Done Done false
32797 9163 Acked Read 5 Sent Done Done false
32798 9164 Acked Write 5 Sent Done Done false
32799 9165 Acked Flush 0 Sent Done Done false
32800 9166 Acked Write 10 Sent Done Done false
32801 9167 Acked Write 2 Sent Done Done false
32802 9168 Acked Read 5 Sent Done Done false
32803 9169 Acked Read 4 Sent Done Done false
32804 9170 Acked Read 8 Sent Done Done false
32805 9171 Acked Write 1 Sent Done Done false
32806 9172 Acked Read 1 Sent Done Done false
32807 9173 Acked Read 10 Sent Done Done false
32808 9174 Acked Write 5 Sent Done Done false
32809 9175 Acked Flush 0 Sent Done Done false
32810 9176 Acked Read 6 Sent Done Done false
32811 9177 Acked Read 4 Sent Done Done false
32812 9178 Acked Write 5 Sent Done Done false
32813 9179 Acked Write 6 Sent Done Done false
32814 9180 Acked Flush 0 Sent Done Done false
32815 9181 Acked Write 3 Sent Done Done false
32816 9182 Acked Write 4 Sent Done Done false
32817 9183 Acked Write 9 Sent Done Done false
32818 9184 Acked Write 3 Sent Done Done false
32819 9185 Acked Write 5 Sent Done Done false
32820 9186 Acked Write 1 Sent Done Done false
32821 9187 Acked Write 2 Sent Done Done false
32822 9188 Acked Write 9 Sent Done Done false
32823 9189 Acked Write 1 Sent Done Done false
32824 9190 Acked Read 1 Sent Done Done false
32825 9191 Acked Read 9 Sent Done Done false
32826 9192 Acked Write 10 Sent Done Done false
32827 9193 Acked Read 2 Sent Done Done false
32828 9194 Acked Read 5 Sent Done Done false
32829 9195 Acked Read 8 Sent Done Done false
32830 9196 Acked Write 9 Sent Done Done false
32831 9197 Acked Read 3 Sent Done Done false
32832 9198 Acked Write 1 Sent Done Done false
32833 9199 Acked Read 6 Sent Done Done false
32834 9200 Acked Write 8 Sent Done Done false
32835 9201 Acked Write 7 Sent Done Done false
32836 9202 Acked Read 9 Sent Done Done false
32837 9203 Acked Read 1 Sent Done Done false
32838 9204 Acked Read 5 Sent Done Done false
32839 9205 Acked Write 2 Sent Done Done false
32840 9206 Acked Write 4 Sent Done Done false
32841 9207 Acked Write 8 Sent Done Done false
32842 9208 Acked Read 9 Sent Done Done false
32843 9209 Acked Write 9 Sent Done Done false
32844 9210 Acked Write 3 Sent Done Done false
32845 9211 Acked Read 10 Sent Done Done false
32846 9212 Acked Write 10 Sent Done Done false
32847 9213 Acked Read 6 Sent Done Done false
32848 9214 Acked Flush 0 Sent Done Done false
32849 9215 Acked Read 3 Sent Done Done false
32850 9216 Acked Read 6 Sent Done Done false
32851 9217 Acked Read 8 Sent Done Done false
32852 9218 Acked Flush 0 Sent Done Done false
32853 9219 Acked Read 1 Sent Done Done false
32854 9220 Acked Flush 0 Sent Done Done false
32855 9221 Acked Write 8 Sent Done Done false
32856 9222 Acked Read 3 Sent Done Done false
32857 9223 Acked Read 3 Sent Done Done false
32858 9224 Acked Write 3 Sent Done Done false
32859 9225 Acked Read 9 Sent Done Done false
32860 9226 Acked Flush 0 Sent Done Done false
32861 9227 Acked Read 3 Sent Done Done false
32862 9228 Acked Read 1 Sent Done Done false
32863 9229 Acked Read 10 Sent Done Done false
32864 9230 Acked Write 2 Sent Done Done false
32865 9231 Acked Read 8 Sent Done Done false
32866 9232 Acked Write 8 Sent Done Done false
32867 9233 Acked Write 2 Sent Done Done false
32868 9234 Acked Write 2 Sent Done Done false
32869 9235 Acked Write 2 Sent Done Done false
32870 9236 Acked Flush 0 Sent Done Done false
32871 9237 Acked Read 9 Sent Done Done false
32872 9238 Acked Flush 0 Sent Done Done false
32873 9239 Acked Read 2 Sent Done Done false
32874 9240 Acked Write 7 Sent Done Done false
32875 9241 Acked Write 5 Sent Done Done false
32876 9242 Acked Read 9 Sent Done Done false
32877 9243 Acked Read 8 Sent Done Done false
32878 9244 Acked Read 6 Sent Done Done false
32879 9245 Acked Read 6 Sent Done Done false
32880 9246 Acked Write 6 Sent Done Done false
32881 9247 Acked Write 4 Sent Done Done false
32882 9248 Acked Read 5 Sent Done Done false
32883 9249 Acked Write 2 Sent Done Done false
32884 9250 Acked Write 5 Sent Done Done false
32885 9251 Acked Write 3 Sent Done Done false
32886 9252 Acked Write 8 Sent Done Done false
32887 9253 Acked Read 5 Sent Done Done false
32888 9254 Acked Write 8 Sent Done Done false
32889 9255 Acked Write 5 Sent Done Done false
32890 9256 Acked Write 9 Sent Done Done false
32891 9257 Acked Write 10 Sent Done Done false
32892 9258 Acked Read 8 Sent Done Done false
32893 9259 Acked Write 2 Sent Done Done false
32894 9260 Acked Read 6 Sent Done Done false
32895 9261 Acked Write 3 Sent Done Done false
32896 9262 Acked Read 5 Sent Done Done false
32897 9263 Acked Write 10 Sent Done Done false
32898 9264 Acked Read 3 Sent Done Done false
32899 9265 Acked Write 10 Sent Done Done false
32900 9266 Acked Flush 0 Sent Done Done false
32901 9267 Acked Read 6 Sent Done Done false
32902 9268 Acked Read 9 Sent Done Done false
32903 9269 Acked Write 2 Sent Done Done false
32904 9270 Acked Read 5 Sent Done Done false
32905 9271 Acked Write 7 Sent Done Done false
32906 9272 Acked Write 1 Sent Done Done false
32907 9273 Acked Write 6 Sent Done Done false
32908 9274 Acked Write 8 Sent Done Done false
32909 9275 Acked Read 1 Sent Done Done false
32910 9276 Acked Read 10 Sent Done Done false
32911 9277 Acked Write 4 Sent Done Done false
32912 9278 Acked Read 5 Sent Done Done false
32913 9279 Acked Read 1 Sent Done Done false
32914 9280 Acked Read 9 Sent Done Done false
32915 9281 Acked Flush 0 Sent Done Done false
32916 9282 Acked Read 3 Sent Done Done false
32917 9283 Acked Write 9 Sent Done Done false
32918 9284 Acked Read 7 Sent Done Done false
32919 9285 Acked Write 3 Sent Done Done false
32920 9286 Acked Read 10 Sent Done Done false
32921 9287 Acked Read 7 Sent Done Done false
32922 9288 Acked Read 3 Sent Done Done false
32923 9289 Acked Write 7 Sent Done Done false
32924 9290 Acked Read 6 Sent Done Done false
32925 9291 Acked Write 2 Sent Done Done false
32926 9292 Acked Write 2 Sent Done Done false
32927 9293 Acked Read 3 Sent Done Done false
32928 9294 Acked Flush 0 Sent Done Done false
32929 9295 Acked Write 8 Sent Done Done false
32930 9296 Acked Write 4 Sent Done Done false
32931 9297 Acked Read 10 Sent Done Done false
32932 9298 Acked Write 9 Sent Done Done false
32933 9299 Acked Write 7 Sent Done Done false
32934 9300 Acked Read 9 Sent Done Done false
32935 9301 Acked Write 4 Sent Done Done false
32936 9302 Acked Read 2 Sent Done Done false
32937 9303 Acked Read 5 Sent Done Done false
32938 9304 Acked Write 4 Sent Done Done false
32939 9305 Acked Write 6 Sent Done Done false
32940 9306 Acked Read 9 Sent Done Done false
32941 9307 Acked Read 4 Sent Done Done false
32942 9308 Acked Read 8 Sent Done Done false
32943 9309 Acked Read 7 Sent Done Done false
32944 9310 Acked Read 1 Sent Done Done false
32945 9311 Acked Read 4 Sent Done Done false
32946 9312 Acked Read 6 Sent Done Done false
32947 9313 Acked Read 8 Sent Done Done false
32948 9314 Acked Write 5 Sent Done Done false
32949 9315 Acked Read 3 Sent Done Done false
32950 9316 Acked Write 10 Sent Done Done false
32951 9317 Acked Flush 0 Sent Done Done false
32952 9318 Acked Read 1 Sent Done Done false
32953 9319 Acked Write 4 Sent Done Done false
32954 9320 Acked Read 9 Sent Done Done false
32955 9321 Acked Read 8 Sent Done Done false
32956 9322 Acked Read 9 Sent Done Done false
32957 9323 Acked Read 5 Sent Done Done false
32958 9324 Acked Read 8 Sent Done Done false
32959 9325 Acked Write 6 Sent Done Done false
32960 9326 Acked Read 10 Sent Done Done false
32961 9327 Acked Read 1 Sent Done Done false
32962 9328 Acked Read 5 Sent Done Done false
32963 9329 Acked Write 10 Sent Done Done false
32964 9330 Acked Flush 0 Sent Done Done false
32965 9331 Acked Write 6 Sent Done Done false
32966 9332 Acked Read 7 Sent Done Done false
32967 9333 Acked Read 2 Sent Done Done false
32968 9334 Acked Write 2 Sent Done Done false
32969 9335 Acked Read 7 Sent Done Done false
32970 9336 Acked Read 6 Sent Done Done false
32971 9337 Acked Read 10 Sent Done Done false
32972 9338 Acked Read 1 Sent Done Done false
32973 9339 Acked Write 5 Sent Done Done false
32974 9340 Acked Read 5 Sent Done Done false
32975 9341 Acked Read 10 Sent Done Done false
32976 9342 Acked Read 3 Sent Done Done false
32977 9343 Acked Write 9 Sent Done Done false
32978 9344 Acked Read 2 Sent Done Done false
32979 9345 Acked Write 9 Sent Done Done false
32980 9346 Acked Write 7 Sent Done Done false
32981 9347 Acked Write 3 Sent Done Done false
32982 9348 Acked Write 9 Sent Done Done false
32983 9349 Acked Read 4 Sent Done Done false
32984 9350 Acked Write 1 Sent Done Done false
32985 9351 Acked Write 8 Sent Done Done false
32986 9352 Acked Flush 0 Sent Done Done false
32987 9353 Acked Write 8 Sent Done Done false
32988 9354 Acked Read 4 Sent Done Done false
32989 9355 Acked Write 5 Sent Done Done false
32990 9356 Acked Write 7 Sent Done Done false
32991 9357 Acked Flush 0 Sent Done Done false
32992 9358 Acked Write 6 Sent Done Done false
32993 9359 Acked Write 7 Sent Done Done false
32994 9360 Acked Read 4 Sent Done Done false
32995 9361 Acked Read 7 Sent Done Done false
32996 9362 Acked Read 2 Sent Done Done false
32997 9363 Acked Write 4 Sent Done Done false
32998 9364 Acked Read 3 Sent Done Done false
32999 9365 Acked Write 7 Sent Done Done false
33000 9366 Acked Write 10 Sent Done Done false
33001 9367 Acked Read 10 Sent Done Done false
33002 9368 Acked Read 3 Sent Done Done false
33003 9369 Acked Write 1 Sent Done Done false
33004 9370 Acked Write 7 Sent Done Done false
33005 9371 Acked Write 2 Sent Done Done false
33006 9372 Acked Read 9 Sent Done Done false
33007 9373 Acked Read 7 Sent Done Done false
33008 9374 Acked Write 7 Sent Done Done false
33009 9375 Acked Flush 0 Sent Done Done false
33010 9376 Acked Read 9 Sent Done Done false
33011 9377 Acked Write 9 Sent Done Done false
33012 9378 Acked Read 6 Sent Done Done false
33013 9379 Acked Read 1 Sent Done Done false
33014 9380 Acked Write 6 Sent Done Done false
33015 9381 Acked Write 5 Sent Done Done false
33016 9382 Acked Read 3 Sent Done Done false
33017 9383 Acked Flush 0 Sent Done Done false
33018 9384 Acked Write 7 Sent Done Done false
33019 9385 Acked Flush 0 Sent Done Done false
33020 9386 Acked Read 3 Sent Done Done false
33021 9387 Acked Read 2 Sent Done Done false
33022 9388 Acked Read 7 Sent Done Done false
33023 9389 Acked Flush 0 Sent Done Done false
33024 9390 Acked Write 7 Sent Done Done false
33025 9391 Acked Read 1 Sent Done Done false
33026 9392 Acked Write 10 Sent Done Done false
33027 9393 Acked Read 2 Sent Done Done false
33028 9394 Acked Write 6 Sent Done Done false
33029 9395 Acked Flush 0 Sent Done Done false
33030 9396 Acked Write 1 Sent Done Done false
33031 9397 Acked Write 1 Sent Done Done false
33032 9398 Acked Read 5 Sent Done Done false
33033 9399 Acked Read 2 Sent Done Done false
33034 9400 Acked Read 2 Sent Done Done false
33035 9401 Acked Read 5 Sent Done Done false
33036 9402 Acked Write 7 Sent Done Done false
33037 9403 Acked Read 7 Sent Done Done false
33038 9404 Acked Write 6 Sent Done Done false
33039 9405 Acked Write 4 Sent Done Done false
33040 9406 Acked Write 7 Sent Done Done false
33041 9407 Acked Write 9 Sent Done Done false
33042 9408 Acked Read 5 Sent Done Done false
33043 9409 Acked Write 8 Sent Done Done false
33044 9410 Acked Write 2 Sent Done Done false
33045 9411 Acked Read 1 Sent Done Done false
33046 9412 Acked Write 5 Sent Done Done false
33047 9413 Acked Read 5 Sent Done Done false
33048 9414 Acked Read 7 Sent Done Done false
33049 9415 Acked Read 6 Sent Done Done false
33050 9416 Acked Read 2 Sent Done Done false
33051 9417 Acked Read 4 Sent Done Done false
33052 9418 Acked Write 1 Sent Done Done false
33053 9419 Acked Read 4 Sent Done Done false
33054 9420 Acked Write 1 Sent Done Done false
33055 9421 Acked Read 2 Sent Done Done false
33056 9422 Acked Write 3 Sent Done Done false
33057 9423 Acked Read 5 Sent Done Done false
33058 9424 Acked Read 9 Sent Done Done false
33059 9425 Acked Write 3 Sent Done Done false
33060 9426 Acked Write 6 Sent Done Done false
33061 9427 Acked Flush 0 Sent Done Done false
33062 9428 Acked Write 1 Sent Done Done false
33063 9429 Acked Read 3 Sent Done Done false
33064 9430 Acked Read 3 Sent Done Done false
33065 9431 Acked Read 9 Sent Done Done false
33066 9432 Acked Read 10 Sent Done Done false
33067 9433 Acked Write 10 Sent Done Done false
33068 9434 Acked Read 1 Sent Done Done false
33069 9435 Acked Write 6 Sent Done Done false
33070 9436 Acked Write 1 Sent Done Done false
33071 9437 Acked Write 2 Sent Done Done false
33072 9438 Acked Write 10 Sent Done Done false
33073 9439 Acked Read 8 Sent Done Done false
33074 9440 Acked Flush 0 Sent Done Done false
33075 9441 Acked Write 9 Sent Done Done false
33076 9442 Acked Read 7 Sent Done Done false
33077 9443 Acked Write 9 Sent Done Done false
33078 9444 Acked Read 5 Sent Done Done false
33079 9445 Acked Read 8 Sent Done Done false
33080 9446 Acked Write 9 Sent Done Done false
33081 9447 Acked Write 3 Sent Done Done false
33082 9448 Acked Write 9 Sent Done Done false
33083 9449 Acked Read 6 Sent Done Done false
33084 9450 Acked Read 1 Sent Done Done false
33085 STATES DS:0 DS:1 DS:2 TOTAL
33086 Sent 301 0 0 301
33087 Done 0 301 301 602
33088 Skipped 0 0 0 0
33089 Error 0 0 0 0
33090 Last Flush: Some(JobId(9149)) Some(JobId(9440)) Some(JobId(9440))
33091 Downstairs last five completed: 9149 9148 9147 9146 9145
33092 Upstairs last five completed: 9450 9449 9448 9447 9446
33093 CLIENT: Up:0 ds:301 act:2
330942025-02-21T06:52:39.327ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
330952025-02-21T06:52:40.329ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
330962025-02-21T06:52:41.330ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
330972025-02-21T06:52:42.331ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
33098 ----------------------------------------------------------------
33099 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33100 States: Offline Active Active
33101 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33102 9150 Acked Read 3 Sent Done Done false
33103 9151 Acked Write 4 Sent Done Done false
33104 9152 Acked Read 6 Sent Done Done false
33105 9153 Acked Write 6 Sent Done Done false
33106 9154 Acked Read 9 Sent Done Done false
33107 9155 Acked Read 5 Sent Done Done false
33108 9156 Acked Write 1 Sent Done Done false
33109 9157 Acked Flush 0 Sent Done Done false
33110 9158 Acked Write 1 Sent Done Done false
33111 9159 Acked Flush 0 Sent Done Done false
33112 9160 Acked Write 6 Sent Done Done false
33113 9161 Acked Read 4 Sent Done Done false
33114 9162 Acked Read 2 Sent Done Done false
33115 9163 Acked Read 5 Sent Done Done false
33116 9164 Acked Write 5 Sent Done Done false
33117 9165 Acked Flush 0 Sent Done Done false
33118 9166 Acked Write 10 Sent Done Done false
33119 9167 Acked Write 2 Sent Done Done false
33120 9168 Acked Read 5 Sent Done Done false
33121 9169 Acked Read 4 Sent Done Done false
33122 9170 Acked Read 8 Sent Done Done false
33123 9171 Acked Write 1 Sent Done Done false
33124 9172 Acked Read 1 Sent Done Done false
33125 9173 Acked Read 10 Sent Done Done false
33126 9174 Acked Write 5 Sent Done Done false
33127 9175 Acked Flush 0 Sent Done Done false
33128 9176 Acked Read 6 Sent Done Done false
33129 9177 Acked Read 4 Sent Done Done false
33130 9178 Acked Write 5 Sent Done Done false
33131 9179 Acked Write 6 Sent Done Done false
33132 9180 Acked Flush 0 Sent Done Done false
33133 9181 Acked Write 3 Sent Done Done false
33134 9182 Acked Write 4 Sent Done Done false
33135 9183 Acked Write 9 Sent Done Done false
33136 9184 Acked Write 3 Sent Done Done false
33137 9185 Acked Write 5 Sent Done Done false
33138 9186 Acked Write 1 Sent Done Done false
33139 9187 Acked Write 2 Sent Done Done false
33140 9188 Acked Write 9 Sent Done Done false
33141 9189 Acked Write 1 Sent Done Done false
33142 9190 Acked Read 1 Sent Done Done false
33143 9191 Acked Read 9 Sent Done Done false
33144 9192 Acked Write 10 Sent Done Done false
33145 9193 Acked Read 2 Sent Done Done false
33146 9194 Acked Read 5 Sent Done Done false
33147 9195 Acked Read 8 Sent Done Done false
33148 9196 Acked Write 9 Sent Done Done false
33149 9197 Acked Read 3 Sent Done Done false
33150 9198 Acked Write 1 Sent Done Done false
33151 9199 Acked Read 6 Sent Done Done false
33152 9200 Acked Write 8 Sent Done Done false
33153 9201 Acked Write 7 Sent Done Done false
33154 9202 Acked Read 9 Sent Done Done false
33155 9203 Acked Read 1 Sent Done Done false
33156 9204 Acked Read 5 Sent Done Done false
33157 9205 Acked Write 2 Sent Done Done false
33158 9206 Acked Write 4 Sent Done Done false
33159 9207 Acked Write 8 Sent Done Done false
33160 9208 Acked Read 9 Sent Done Done false
33161 9209 Acked Write 9 Sent Done Done false
33162 9210 Acked Write 3 Sent Done Done false
33163 9211 Acked Read 10 Sent Done Done false
33164 9212 Acked Write 10 Sent Done Done false
33165 9213 Acked Read 6 Sent Done Done false
33166 9214 Acked Flush 0 Sent Done Done false
33167 9215 Acked Read 3 Sent Done Done false
33168 9216 Acked Read 6 Sent Done Done false
33169 9217 Acked Read 8 Sent Done Done false
33170 9218 Acked Flush 0 Sent Done Done false
33171 9219 Acked Read 1 Sent Done Done false
33172 9220 Acked Flush 0 Sent Done Done false
33173 9221 Acked Write 8 Sent Done Done false
33174 9222 Acked Read 3 Sent Done Done false
33175 9223 Acked Read 3 Sent Done Done false
33176 9224 Acked Write 3 Sent Done Done false
33177 9225 Acked Read 9 Sent Done Done false
33178 9226 Acked Flush 0 Sent Done Done false
33179 9227 Acked Read 3 Sent Done Done false
33180 9228 Acked Read 1 Sent Done Done false
33181 9229 Acked Read 10 Sent Done Done false
33182 9230 Acked Write 2 Sent Done Done false
33183 9231 Acked Read 8 Sent Done Done false
33184 9232 Acked Write 8 Sent Done Done false
33185 9233 Acked Write 2 Sent Done Done false
33186 9234 Acked Write 2 Sent Done Done false
33187 9235 Acked Write 2 Sent Done Done false
33188 9236 Acked Flush 0 Sent Done Done false
33189 9237 Acked Read 9 Sent Done Done false
33190 9238 Acked Flush 0 Sent Done Done false
33191 9239 Acked Read 2 Sent Done Done false
33192 9240 Acked Write 7 Sent Done Done false
33193 9241 Acked Write 5 Sent Done Done false
33194 9242 Acked Read 9 Sent Done Done false
33195 9243 Acked Read 8 Sent Done Done false
33196 9244 Acked Read 6 Sent Done Done false
33197 9245 Acked Read 6 Sent Done Done false
33198 9246 Acked Write 6 Sent Done Done false
33199 9247 Acked Write 4 Sent Done Done false
33200 9248 Acked Read 5 Sent Done Done false
33201 9249 Acked Write 2 Sent Done Done false
33202 9250 Acked Write 5 Sent Done Done false
33203 9251 Acked Write 3 Sent Done Done false
33204 9252 Acked Write 8 Sent Done Done false
33205 9253 Acked Read 5 Sent Done Done false
33206 9254 Acked Write 8 Sent Done Done false
33207 9255 Acked Write 5 Sent Done Done false
33208 9256 Acked Write 9 Sent Done Done false
33209 9257 Acked Write 10 Sent Done Done false
33210 9258 Acked Read 8 Sent Done Done false
33211 9259 Acked Write 2 Sent Done Done false
33212 9260 Acked Read 6 Sent Done Done false
33213 9261 Acked Write 3 Sent Done Done false
33214 9262 Acked Read 5 Sent Done Done false
33215 9263 Acked Write 10 Sent Done Done false
33216 9264 Acked Read 3 Sent Done Done false
33217 9265 Acked Write 10 Sent Done Done false
33218 9266 Acked Flush 0 Sent Done Done false
33219 9267 Acked Read 6 Sent Done Done false
33220 9268 Acked Read 9 Sent Done Done false
33221 9269 Acked Write 2 Sent Done Done false
33222 9270 Acked Read 5 Sent Done Done false
33223 9271 Acked Write 7 Sent Done Done false
33224 9272 Acked Write 1 Sent Done Done false
33225 9273 Acked Write 6 Sent Done Done false
33226 9274 Acked Write 8 Sent Done Done false
33227 9275 Acked Read 1 Sent Done Done false
33228 9276 Acked Read 10 Sent Done Done false
33229 9277 Acked Write 4 Sent Done Done false
33230 9278 Acked Read 5 Sent Done Done false
33231 9279 Acked Read 1 Sent Done Done false
33232 9280 Acked Read 9 Sent Done Done false
33233 9281 Acked Flush 0 Sent Done Done false
33234 9282 Acked Read 3 Sent Done Done false
33235 9283 Acked Write 9 Sent Done Done false
33236 9284 Acked Read 7 Sent Done Done false
33237 9285 Acked Write 3 Sent Done Done false
33238 9286 Acked Read 10 Sent Done Done false
33239 9287 Acked Read 7 Sent Done Done false
33240 9288 Acked Read 3 Sent Done Done false
33241 9289 Acked Write 7 Sent Done Done false
33242 9290 Acked Read 6 Sent Done Done false
33243 9291 Acked Write 2 Sent Done Done false
33244 9292 Acked Write 2 Sent Done Done false
33245 9293 Acked Read 3 Sent Done Done false
33246 9294 Acked Flush 0 Sent Done Done false
33247 9295 Acked Write 8 Sent Done Done false
33248 9296 Acked Write 4 Sent Done Done false
33249 9297 Acked Read 10 Sent Done Done false
33250 9298 Acked Write 9 Sent Done Done false
33251 9299 Acked Write 7 Sent Done Done false
33252 9300 Acked Read 9 Sent Done Done false
33253 9301 Acked Write 4 Sent Done Done false
33254 9302 Acked Read 2 Sent Done Done false
33255 9303 Acked Read 5 Sent Done Done false
33256 9304 Acked Write 4 Sent Done Done false
33257 9305 Acked Write 6 Sent Done Done false
33258 9306 Acked Read 9 Sent Done Done false
33259 9307 Acked Read 4 Sent Done Done false
33260 9308 Acked Read 8 Sent Done Done false
33261 9309 Acked Read 7 Sent Done Done false
33262 9310 Acked Read 1 Sent Done Done false
33263 9311 Acked Read 4 Sent Done Done false
33264 9312 Acked Read 6 Sent Done Done false
33265 9313 Acked Read 8 Sent Done Done false
33266 9314 Acked Write 5 Sent Done Done false
33267 9315 Acked Read 3 Sent Done Done false
33268 9316 Acked Write 10 Sent Done Done false
33269 9317 Acked Flush 0 Sent Done Done false
33270 9318 Acked Read 1 Sent Done Done false
33271 9319 Acked Write 4 Sent Done Done false
33272 9320 Acked Read 9 Sent Done Done false
33273 9321 Acked Read 8 Sent Done Done false
33274 9322 Acked Read 9 Sent Done Done false
33275 9323 Acked Read 5 Sent Done Done false
33276 9324 Acked Read 8 Sent Done Done false
33277 9325 Acked Write 6 Sent Done Done false
33278 9326 Acked Read 10 Sent Done Done false
33279 9327 Acked Read 1 Sent Done Done false
33280 9328 Acked Read 5 Sent Done Done false
33281 9329 Acked Write 10 Sent Done Done false
33282 9330 Acked Flush 0 Sent Done Done false
33283 9331 Acked Write 6 Sent Done Done false
33284 9332 Acked Read 7 Sent Done Done false
33285 9333 Acked Read 2 Sent Done Done false
33286 9334 Acked Write 2 Sent Done Done false
33287 9335 Acked Read 7 Sent Done Done false
33288 9336 Acked Read 6 Sent Done Done false
33289 9337 Acked Read 10 Sent Done Done false
33290 9338 Acked Read 1 Sent Done Done false
33291 9339 Acked Write 5 Sent Done Done false
33292 9340 Acked Read 5 Sent Done Done false
33293 9341 Acked Read 10 Sent Done Done false
33294 9342 Acked Read 3 Sent Done Done false
33295 9343 Acked Write 9 Sent Done Done false
33296 9344 Acked Read 2 Sent Done Done false
33297 9345 Acked Write 9 Sent Done Done false
33298 9346 Acked Write 7 Sent Done Done false
33299 9347 Acked Write 3 Sent Done Done false
33300 9348 Acked Write 9 Sent Done Done false
33301 9349 Acked Read 4 Sent Done Done false
33302 9350 Acked Write 1 Sent Done Done false
33303 9351 Acked Write 8 Sent Done Done false
33304 9352 Acked Flush 0 Sent Done Done false
33305 9353 Acked Write 8 Sent Done Done false
33306 9354 Acked Read 4 Sent Done Done false
33307 9355 Acked Write 5 Sent Done Done false
33308 9356 Acked Write 7 Sent Done Done false
33309 9357 Acked Flush 0 Sent Done Done false
33310 9358 Acked Write 6 Sent Done Done false
33311 9359 Acked Write 7 Sent Done Done false
33312 9360 Acked Read 4 Sent Done Done false
33313 9361 Acked Read 7 Sent Done Done false
33314 9362 Acked Read 2 Sent Done Done false
33315 9363 Acked Write 4 Sent Done Done false
33316 9364 Acked Read 3 Sent Done Done false
33317 9365 Acked Write 7 Sent Done Done false
33318 9366 Acked Write 10 Sent Done Done false
33319 9367 Acked Read 10 Sent Done Done false
33320 9368 Acked Read 3 Sent Done Done false
33321 9369 Acked Write 1 Sent Done Done false
33322 9370 Acked Write 7 Sent Done Done false
33323 9371 Acked Write 2 Sent Done Done false
33324 9372 Acked Read 9 Sent Done Done false
33325 9373 Acked Read 7 Sent Done Done false
33326 9374 Acked Write 7 Sent Done Done false
33327 9375 Acked Flush 0 Sent Done Done false
33328 9376 Acked Read 9 Sent Done Done false
33329 9377 Acked Write 9 Sent Done Done false
33330 9378 Acked Read 6 Sent Done Done false
33331 9379 Acked Read 1 Sent Done Done false
33332 9380 Acked Write 6 Sent Done Done false
33333 9381 Acked Write 5 Sent Done Done false
33334 9382 Acked Read 3 Sent Done Done false
33335 9383 Acked Flush 0 Sent Done Done false
33336 9384 Acked Write 7 Sent Done Done false
33337 9385 Acked Flush 0 Sent Done Done false
33338 9386 Acked Read 3 Sent Done Done false
33339 9387 Acked Read 2 Sent Done Done false
33340 9388 Acked Read 7 Sent Done Done false
33341 9389 Acked Flush 0 Sent Done Done false
33342 9390 Acked Write 7 Sent Done Done false
33343 9391 Acked Read 1 Sent Done Done false
33344 9392 Acked Write 10 Sent Done Done false
33345 9393 Acked Read 2 Sent Done Done false
33346 9394 Acked Write 6 Sent Done Done false
33347 9395 Acked Flush 0 Sent Done Done false
33348 9396 Acked Write 1 Sent Done Done false
33349 9397 Acked Write 1 Sent Done Done false
33350 9398 Acked Read 5 Sent Done Done false
33351 9399 Acked Read 2 Sent Done Done false
33352 9400 Acked Read 2 Sent Done Done false
33353 9401 Acked Read 5 Sent Done Done false
33354 9402 Acked Write 7 Sent Done Done false
33355 9403 Acked Read 7 Sent Done Done false
33356 9404 Acked Write 6 Sent Done Done false
33357 9405 Acked Write 4 Sent Done Done false
33358 9406 Acked Write 7 Sent Done Done false
33359 9407 Acked Write 9 Sent Done Done false
33360 9408 Acked Read 5 Sent Done Done false
33361 9409 Acked Write 8 Sent Done Done false
33362 9410 Acked Write 2 Sent Done Done false
33363 9411 Acked Read 1 Sent Done Done false
33364 9412 Acked Write 5 Sent Done Done false
33365 9413 Acked Read 5 Sent Done Done false
33366 9414 Acked Read 7 Sent Done Done false
33367 9415 Acked Read 6 Sent Done Done false
33368 9416 Acked Read 2 Sent Done Done false
33369 9417 Acked Read 4 Sent Done Done false
33370 9418 Acked Write 1 Sent Done Done false
33371 9419 Acked Read 4 Sent Done Done false
33372 9420 Acked Write 1 Sent Done Done false
33373 9421 Acked Read 2 Sent Done Done false
33374 9422 Acked Write 3 Sent Done Done false
33375 9423 Acked Read 5 Sent Done Done false
33376 9424 Acked Read 9 Sent Done Done false
33377 9425 Acked Write 3 Sent Done Done false
33378 9426 Acked Write 6 Sent Done Done false
33379 9427 Acked Flush 0 Sent Done Done false
33380 9428 Acked Write 1 Sent Done Done false
33381 9429 Acked Read 3 Sent Done Done false
33382 9430 Acked Read 3 Sent Done Done false
33383 9431 Acked Read 9 Sent Done Done false
33384 9432 Acked Read 10 Sent Done Done false
33385 9433 Acked Write 10 Sent Done Done false
33386 9434 Acked Read 1 Sent Done Done false
33387 9435 Acked Write 6 Sent Done Done false
33388 9436 Acked Write 1 Sent Done Done false
33389 9437 Acked Write 2 Sent Done Done false
33390 9438 Acked Write 10 Sent Done Done false
33391 9439 Acked Read 8 Sent Done Done false
33392 9440 Acked Flush 0 Sent Done Done false
33393 9441 Acked Write 9 Sent Done Done false
33394 9442 Acked Read 7 Sent Done Done false
33395 9443 Acked Write 9 Sent Done Done false
33396 9444 Acked Read 5 Sent Done Done false
33397 9445 Acked Read 8 Sent Done Done false
33398 9446 Acked Write 9 Sent Done Done false
33399 9447 Acked Write 3 Sent Done Done false
33400 9448 Acked Write 9 Sent Done Done false
33401 9449 Acked Read 6 Sent Done Done false
33402 9450 Acked Read 1 Sent Done Done false
33403 STATES DS:0 DS:1 DS:2 TOTAL
33404 Sent 301 0 0 301
33405 Done 0 301 301 602
33406 Skipped 0 0 0 0
33407 Error 0 0 0 0
33408 Last Flush: Some(JobId(9149)) Some(JobId(9440)) Some(JobId(9440))
33409 Downstairs last five completed: 9149 9148 9147 9146 9145
33410 Upstairs last five completed: 9450 9449 9448 9447 9446
33411 CLIENT: Up:0 ds:301 act:2
334122025-02-21T06:52:43.808ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334132025-02-21T06:52:44.334ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334142025-02-21T06:52:45.335ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334152025-02-21T06:52:46.097ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334162025-02-21T06:52:46.098ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:45808) = io task client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334172025-02-21T06:52:46.099ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334182025-02-21T06:52:46.099ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334192025-02-21T06:52:46.099ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334202025-02-21T06:52:46.099ZINFOcrucible: send last flush ID to this DS: Some(JobId(9149)) = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334212025-02-21T06:52:46.099ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334222025-02-21T06:52:46.099ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(9149)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334232025-02-21T06:52:46.100ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(9149)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
33424 ----------------------------------------------------------------
33425 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:10
33426 States: Active Active Active
33427 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33428 9441 Acked Write 9 Done Done Done true
33429 9442 Acked Read 7 Done Done Done true
33430 9443 Acked Write 9 Done Done Done true
33431 9444 Acked Read 5 Done Done Done true
33432 9445 Acked Read 8 Done Done Done true
33433 9446 Acked Write 9 Done Done Done true
33434 9447 Acked Write 3 Done Done Done true
33435 9448 Acked Write 9 Done Done Done true
33436 9449 Acked Read 6 Done Done Done true
33437 9450 Acked Read 1 Done Done Done true
33438 STATES DS:0 DS:1 DS:2 TOTAL
33439 Sent 0 0 0 0
33440 Done 10 10 10 30
33441 Skipped 0 0 0 0
33442 Error 0 0 0 0
33443 Last Flush: Some(JobId(9440)) Some(JobId(9440)) Some(JobId(9440))
33444 Downstairs last five completed: 9440 9439 9438 9437 9436
33445 Upstairs last five completed: 9450 9449 9448 9447 9446
33446 CLIENT: Up:0 ds:10 act:3
33447 ----------------------------------------------------------------
33448 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33449 Downstairs last five completed: 9451 9450 9449 9448 9447
33450 Upstairs last five completed: 9451 9450 9449 9448 9447
33451 CLIENT: Up:0 ds:0 act:3
33452 Replay: All jobs finished
33453 Main task has work to do, go find it
33454 got dsc Stop(1)
33455 stop 1
33456 [8820] Got stop action so:false kr:false
33457 Feb 21 06:52:50.347 INFO request completed, latency_us: 378, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 198245d2-fe76-492f-9297-1bd903ccd9a5, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
33458 Feb 21 06:52:50.348 INFO request completed, latency_us: 172, response_code: 200, uri: /state/cid/1, method: GET, req_id: d1548939-1745-4f23-99a4-ff77f02eb263, remote_addr: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:52:50.348779289Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
33459 127.0.0.1:53848, local_addr: 127.0.0.1:9998
334602025-02-21T06:52:50.348ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334612025-02-21T06:52:50.348ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
334622025-02-21T06:52:50.348ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
33463 [8820] Exited with: ExitStatus(unix_wait_status(9))
33464 [8820][1] reports Exit
33465 Feb 21 06:52:54.351 INFO request completed, latency_us: 220, response_code: 200, uri: /state/cid/1, method: GET, req_id: 67f6f27c-8adc-43ab-ae41-eaf2d564c919, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
33466 001/300 Write block 785 len 24576 data: 9 9 10 8 8 7
33467 002/300 Read block 2777 len 4096
33468 003/300 Read block 141 len 8192
33469 004/300 Write block 4576 len 8192 data: 5 6
33470 005/300 Write block 2155 len 40960 data: 5 5 5 5 5 5 5 5 6 5
33471 006/300 Read block 3736 len 20480
33472 007/300 Read block 3471 len 28672
33473 008/300 Read block 2128 len 20480
33474 009/300 Read block 2691 len 12288
33475 010/300 Read block 3336 len 16384
33476 011/300 Write block 1226 len 16384 data: 4 4 4 4
33477 012/300 Read block 3883 len 12288
33478 013/300 Write block 91 len 4096 data: 6
33479 014/300 Write block 949 len 36864 data: 5 4 3 5 4 5 8 7 9
33480 015/300 Read block 4468 len 4096
33481 016/300 Write block 2960 len 40960 data: 4 4 4 4 5 6 5 5 4 3
33482 017/300 Read block 3958 len 4096
33483 018/300 Flush
33484 019/300 Write block 1611 len 8192 data: 12 12
33485 020/300 Read block 3334 len 12288
33486 021/300 Flush
33487 022/300 Read block 4402 len 4096
33488 023/300 Read block 361 len 40960
33489 024/300 Read block 3942 len 4096
33490 025/300 Read block 3847 len 32768
33491 026/300 Read block 4078 len 40960
33492 027/300 Write block 2623 len 36864 data: 6 4 6 6 6 6 8 7 8
33493 028/300 Read block 3540 len 36864
33494 029/300 Read block 4765 len 12288
33495 030/300 Read block 1907 len 20480
33496 031/300 Flush
33497 032/300 Write block 3840 len 16384 data: 9 10 10 9
33498 033/300 Read block 1242 len 16384
33499 034/300 Read block 3430 len 36864
33500 035/300 Flush
33501 036/300 Write block 1707 len 32768 data: 3 5 6 5 4 4 4 5
33502 037/300 Write block 3354 len 20480 data: 8 7 5 4 4
33503 038/300 Read block 511 len 4096
33504 039/300 Write block 852 len 8192 data: 5 5
33505 040/300 Write block 1267 len 4096 data: 3
33506 041/300 Read block 3987 len 36864
33507 042/300 Write block 1257 len 28672 data: 3 6 8 8 9 9 9
33508 043/300 Read block 3225 len 4096
33509 044/300 Read block 1235 len 24576
33510 045/300 Read block 1080 len 40960
33511 046/300 Read block 758 len 12288
33512 047/300 Write block 940 len 40960 data: 4 4 3 3 4 6 5 5 5 6
33513 048/300 Write block 992 len 40960 data: 4 4 4 6 6 5 5 5 4 4
33514 049/300 Read block 2205 len 36864
33515 050/300 Read block 4877 len 36864
33516 051/300 Read block 4002 len 24576
33517 052/300 Write block 244 len 36864 data: 4 5 5 5 4 5 4 3 5
33518 053/300 Read block 3023 len 8192
33519 054/300 Write block 3466 len 4096 data: 6
33520 055/300 Flush
33521 056/300 Read block 3088 len 28672
33522 057/300 Write block 2639 len 40960 data: 5 4 5 4 5 6 5 5 6 4
33523 058/300 Write block 2141 len 36864 data: 5 5 4 3 2 3 4 3 4
33524 059/300 Write block 2716 len 32768 data: 8 9 9 7 7 8 5 4
33525 060/300 Write block 1468 len 4096 data: 9
33526 061/300 Flush
33527 062/300 Write block 2096 len 40960 data: 4 5 5 5 6 5 4 5 5 5
33528 063/300 Read block 4055 len 36864
33529 064/300 Read block 1675 len 28672
33530 065/300 Read block 3715 len 28672
33531 066/300 Read block 4567 len 8192
33532 067/300 Read block 2007 len 36864
33533 068/300 Flush
33534 069/300 Read block 783 len 12288
33535 070/300 Read block 3977 len 32768
33536 071/300 Write block 4233 len 28672 data: 5 5 5 5 5 4 4
33537 072/300 Write block 38 len 28672 data: 5 4 4 4 6 7 6
33538 073/300 Read block 1899 len 36864
33539 074/300 Write block 3271 len 8192 data: 7 7
33540 075/300 Read block 2163 len 40960
33541 076/300 Write block 554 len 4096 data: 4
33542 077/300 Write block 4564 len 36864 data: 3 5 8 7 9 10 11 9 9
33543 078/300 Read block 290 len 24576
33544 079/300 Read block 4092 len 24576
33545 080/300 Read block 4071 len 40960
33546 081/300 Write block 502 len 8192 data: 8 11
33547 082/300 Read block 4467 len 20480
33548 083/300 Read block 7 len 32768
33549 084/300 Read block 3628 len 20480
33550 085/300 Read block 3640 len 28672
33551 086/300 Write block 4783 len 16384 data: 5 4 6 6
33552 087/300 Write block 3798 len 20480 data: 9 9 10 6 7
33553 088/300 Read block 3776 len 32768
33554 089/300 Read block 563 len 28672
33555 090/300 Read block 3617 len 32768
33556 091/300 Write block 1263 len 12288 data: 10 7 6
33557 092/300 Write block 2713 len 16384 data: 6 5 8 9
33558 093/300 Read block 2240 len 36864
33559 094/300 Flush
33560 095/300 Write block 4516 len 16384 data: 3 3 3 6
33561 096/300 Read block 3040 len 36864
33562 097/300 Read block 4924 len 24576
33563 098/300 Write block 434 len 12288 data: 7 5 7
33564 099/300 Read block 2488 len 36864
33565 100/300 Write block 9 len 24576 data: 3 2 3 3 5 5
33566 101/300 Write block 1682 len 32768 data: 4 6 7 7 7 5 5 4
33567 102/300 Write block 4110 len 4096 data: 8
33568 103/300 Read block 3340 len 36864
33569 104/300 Write block 2148 len 36864 data: 4 5 3 3 4 3 5 6 6
33570 105/300 Write block 193 len 36864 data: 5 4 3 4 4 6 5 5 5
33571 106/300 Write block 1665 len 8192 data: 9 9
33572 107/300 Write block 3653 len 28672 data: 7 6 6 6 7 8 8
33573 108/300 Read block 2794 len 12288
33574 109/300 Flush
33575 110/300 Read block 1599 len 36864
33576 111/300 Read block 1575 len 32768
33577 112/300 Read block 3539 len 12288
33578 113/300 Read block 2723 len 24576
33579 114/300 Write block 1470 len 24576 data: 7 6 7 5 4 5
33580 115/300 Read block 4992 len 28672
33581 116/300 Read block 1849 len 24576
33582 117/300 Write block 4574 len 20480 data: 6 5 6 7 4
33583 118/300 Read block 1463 len 4096
33584 119/300 Read block 4076 len 8192
33585 120/300 Read block 2838 len 36864
33586 121/300 Read block 494 len 28672
33587 122/300 Write block 4011 len 12288 data: 9 9 9
33588 123/300 Flush
33589 124/300 Read block 1704 len 40960
33590 125/300 Read block 3514 len 8192
33591 126/300 Read block 3635 len 28672
33592 127/300 Read block 3684 len 12288
33593 128/300 Read block 4546 len 32768
33594 129/300 Read block 674 len 28672
33595 130/300 Read block 2260 len 32768
33596 131/300 Read block 3149 len 4096
33597 132/300 Read block 2556 len 36864
33598 133/300 Write block 1422 len 16384 data: 7 8 10 8
33599 134/300 Read block 4448 len 20480
33600 135/300 Flush
33601 136/300 Read block 188 len 12288
33602 137/300 Read block 3184 len 16384
33603 138/300 Read block 4316 len 32768
33604 139/300 Write block 3440 len 20480 data: 6 5 5 5 6
33605 140/300 Read block 3051 len 12288
33606 141/300 Flush
33607 142/300 Write block 1696 len 40960 data: 3 4 5 3 3 5 5 5 5 4
33608 143/300 Read block 1426 len 24576
33609 144/300 Read block 1481 len 40960
33610 145/300 Read block 1568 len 28672
33611 146/300 Read block 937 len 28672
33612 147/300 Read block 3092 len 36864
33613 148/300 Write block 2893 len 32768 data: 5 5 4 5 5 4 4 5
33614 149/300 Read block 177 len 8192
33615 150/300 Write block 842 len 28672 data: 5 6 4 3 3 4 4
33616 151/300 Write block 4145 len 32768 data: 8 7 5 3 3 2 2 2
33617 152/300 Write block 81 len 24576 data: 7 7 5 5 4 5
33618 153/300 Write block 3626 len 12288 data: 5 5 5
33619 154/300 Read block 2637 len 4096
33620 155/300 Flush
33621 156/300 Write block 2289 len 28672 data: 6 6 6 3 4 3 3
33622 157/300 Write block 3540 len 28672 data: 6 5 5 5 6 6 6
33623 158/300 Read block 1403 len 40960
33624 159/300 Write block 3361 len 40960 data: 4 3 4 6 6 6 5 4 4 2
33625 160/300 Flush
33626 161/300 Write block 2771 len 8192 data: 4 3
33627 162/300 Flush
33628 163/300 Write block 2069 len 20480 data: 7 8 9 8 8
33629 164/300 Read block 3241 len 16384
33630 165/300 Read block 3181 len 24576
33631 166/300 Read block 2196 len 40960
33632 167/300 Write block 3607 len 12288 data: 9 8 7
33633 168/300 Write block 1682 len 36864 data: 5 7 8 8 8 6 6 5 5
33634 169/300 Write block 3044 len 4096 data: 6
33635 170/300 Read block 2291 len 32768
33636 171/300 Read block 160 len 40960
33637 172/300 Write block 1893 len 4096 data: 8
33638 173/300 Write block 4325 len 4096 data: 3
33639 174/300 Read block 660 len 28672
33640 175/300 Flush
33641 176/300 Read block 2779 len 28672
33642 177/300 Write block 198 len 40960 data: 7 6 6 6 6 5 6 5 5 5
33643 178/300 Read block 4993 len 28672
33644 179/300 Write block 2362 len 36864 data: 7 9 8 8 6 5 5 4 4
33645 180/300 Read block 1641 len 28672
33646 181/300 Write block 4527 len 4096 data: 3
33647 182/300 Flush
33648 183/300 Read block 3290 len 8192
33649 184/300 Flush
33650 185/300 Read block 4965 len 28672
33651 186/300 Write block 3831 len 16384 data: 6 5 5 6
33652 187/300 Read block 3364 len 40960
33653 188/300 Read block 1153 len 8192
33654 189/300 Read block 3762 len 16384
33655 190/300 Read block 3419 len 40960
33656 191/300 Write block 3934 len 36864 data: 5 5 6 6 5 5 6 6 8
33657 192/300 Write block 1874 len 36864 data: 6 7 6 7 5 4 4 4 7
33658 193/300 Write block 1948 len 20480 data: 8 7 6 7 8
33659 194/300 Flush
33660 195/300 Write block 1459 len 32768 data: 5 6 7 7 10 12 12 12
33661 196/300 Read block 33 len 16384
33662 197/300 Read block 3143 len 36864
33663 198/300 Read block 2310 len 20480
33664 199/300 Read block 802 len 32768
33665 200/300 Read block 2598 len 24576
33666 201/300 Read block 2295 len 8192
33667 202/300 Write block 1275 len 8192 data: 7 8
33668 203/300 Flush
33669 204/300 Read block 3075 len 28672
33670 205/300 Read block 2807 len 12288
33671 206/300 Read block 928 len 12288
33672 207/300 Write block 1622 len 12288 data: 8 8 8
33673 208/300 Write block 2387 len 24576 data: 4 5 4 6 6 8
33674 209/300 Read block 3452 len 40960
33675 210/300 Read block 3062 len 4096
33676 211/300 Read block 1005 len 28672
33677 212/300 Read block 1610 len 36864
33678 213/300 Read block 1670 len 40960
33679 214/300 Read block 4125 len 12288
33680 215/300 Read block 2275 len 24576
33681 216/300 Read block 549 len 40960
33682 217/300 Flush
33683 218/300 Read block 2538 len 28672
33684 219/300 Read block 2197 len 28672
33685 220/300 Write block 2793 len 12288 data: 3 4 4
33686 221/300 Write block 1032 len 40960 data: 9 8 6 6 7 4 4 2 2 3
33687 222/300 Read block 1438 len 16384
33688 223/300 Write block 786 len 24576 data: 10 11 9 9 8 6
33689 224/300 Read block 2197 len 8192
33690 225/300 Read block 3061 len 40960
33691 226/300 Write block 2226 len 24576 data: 8 8 8 6 6 5
33692 227/300 Flush
33693 228/300 Flush
33694 229/300 Write block 3894 len 8192 data: 2 2
33695 230/300 Read block 3759 len 8192
33696 231/300 Write block 4469 len 28672 data: 7 8 8 10 10 10 9
33697 232/300 Read block 616 len 40960
33698 233/300 Write block 4631 len 28672 data: 3 2 3 3 2 3 3
33699 234/300 Write block 3569 len 24576 data: 6 5 6 5 5 4
33700 235/300 Read block 3253 len 4096
33701 236/300 Read block 554 len 12288
33702 237/300 Write block 4757 len 16384 data: 5 8 7 6
33703 238/300 Write block 28 len 24576 data: 8 6 7 6 5 6
33704 239/300 Flush
33705 240/300 Read block 858 len 8192
33706 241/300 Write block 4710 len 36864 data: 7 7 6 6 6 5 6 5 5
33707 242/300 Write block 2604 len 36864 data: 8 8 7 7 6 6 5 3 2
33708 243/300 Read block 697 len 12288
33709 244/300 Read block 4033 len 8192
33710 245/300 Write block 4051 len 40960 data: 2 3 6 7 8 7 7 8 8 8
33711 246/300 Write block 1493 len 40960 data: 10 9 9 9 9 8 8 9 8 8
33712 247/300 Read block 1162 len 8192
33713 248/300 Read block 256 len 20480
33714 249/300 Write block 91 len 4096 data: 7
33715 250/300 Read block 3750 len 12288
33716 251/300 Read block 2406 len 28672
33717 252/300 Read block 630 len 28672
33718 253/300 Read block 1993 len 8192
33719 254/300 Read block 3302 len 20480
33720 255/300 Write block 2149 len 40960 data: 6 4 4 5 4 6 7 7 6 6
33721 256/300 Flush
33722 257/300 Read block 1602 len 12288
33723 258/300 Read block 469 len 32768
33724 259/300 Read block 4229 len 8192
33725 260/300 Read block 121 len 36864
33726 261/300 Read block 1780 len 16384
33727 262/300 Write block 3034 len 28672 data: 10 10 10 9 9 9 9
33728 263/300 Write block 1202 len 8192 data: 4 5
33729 264/300 Read block 2665 len 28672
33730 265/300 Read block 1037 len 36864
33731 266/300 Flush
33732 267/300 Read block 3718 len 36864
33733 268/300 Read block 3107 len 16384
33734 269/300 Read block 1377 len 16384
33735 270/300 Flush
33736 271/300 Read block 726 len 32768
33737 272/300 Write block 2905 len 32768 data: 8 9 6 6 7 7 6 7
33738 273/300 Read block 4573 len 24576
33739 274/300 Flush
33740 275/300 Read block 800 len 36864
33741 276/300 Write block 4901 len 24576 data: 11 9 9 7 7 7
33742 277/300 Read block 4247 len 12288
33743 278/300 Write block 1168 len 8192 data: 6 7
33744 279/300 Flush
33745 280/300 Read block 3203 len 32768
33746 281/300 Read block 1903 len 36864
33747 282/300 Read block 480 len 40960
33748 283/300 Read block 1160 len 28672
33749 284/300 Write block 1469 len 20480 data: 8 8 7 8 6
33750 285/300 Read block 1976 len 12288
33751 286/300 Read block 3077 len 4096
33752 287/300 Write block 13 len 4096 data: 6
33753 288/300 Write block 2112 len 16384 data: 5 5 5 4
33754 289/300 Flush
33755 290/300 Read block 377 len 28672
33756 291/300 Write block 597 len 28672 data: 4 3 4 5 5 5 4
33757 292/300 Write block 2947 len 32768 data: 3 5 6 9 11 12 12 9
33758 293/300 Read block 4656 len 8192
33759 294/300 Read block 4035 len 32768
33760 295/300 Read block 3996 len 40960
33761 296/300 Read block 1199 len 36864
33762 297/300 Read block 1641 len 16384
33763 298/300 Read block 2046 len 32768
33764 299/300 Read block 246 len 4096
33765 300/300 Write block 1816 len 36864 data: 8 8 6 9 10 9 9 9 8
33766 301/300 Write block 3850 len 24576 data: 4 2 2 2 3 4
33767 Main task has work to do, go find it
33768 got dsc Start(1)
33769 start 1
33770 Feb 21 06:52:54.612 INFO request completed, latency_us: 225, response_code: 204, uri: /start/cid/1, method: POST, req_id: 22bc175a-9107-478e-b7de-e522668500d4, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
33771 Replay: started 1, returned:Ok(())
33772 ----------------------------------------------------------------
33773 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33774 States: Active Offline Active
33775 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33776 9452 Acked Write 6 Done Sent Done false
33777 9453 Acked Read 1 Done Sent Done false
33778 9454 Acked Read 2 Done Sent Done false
33779 9455 Acked Write 2 Done Sent Done false
33780 9456 Acked Write 10 Done Sent Done false
33781 9457 Acked Read 5 Done Sent Done false
33782 9458 Acked Read 7 Done Sent Done false
33783 9459 Acked Read 5 Done Sent Done false
33784 9460 Acked Read 3 Done Sent Done false
33785 9461 Acked Read 4 Done Sent Done false
33786 9462 Acked Write 4 Done Sent Done false
33787 9463 Acked Read 3 Done Sent Done false
33788 9464 Acked Write 1 Done Sent Done false
33789 9465 Acked Write 9 Done Sent Done false
33790 9466 Acked Read 1 Done Sent Done false
33791 9467 Acked Write 10 Done Sent Done false
33792 9468 Acked Read 1 Done Sent Done false
33793 9469 Acked Flush 0 Done Sent Done false
33794 9470 Acked Write 2 Done Sent Done false
33795 9471 Acked Read 3 Done Sent Done false
33796 9472 Acked Flush 0 Done Sent Done false
33797 9473 Acked Read 1 Done Sent Done false
33798 9474 Acked Read 10 Done Sent Done false
33799 9475 Acked Read 1 Done Sent Done false
33800 9476 Acked Read 8 Done Sent Done false
33801 9477 Acked Read 10 Done Sent Done false
33802 9478 Acked Write 9 Done Sent Done false
33803 9479 Acked Read 9 Done Sent Done false
33804 9480 Acked Read 3 Done Sent Done false
33805 9481 Acked Read 5 Done Sent Done false
33806 9482 Acked Flush 0 Done Sent Done false
33807 9483 Acked Write 4 Done Sent Done false
33808 9484 Acked Read 4 Done Sent Done false
33809 9485 Acked Read 9 Done Sent Done false
33810 9486 Acked Flush 0 Done Sent Done false
33811 9487 Acked Write 8 Done Sent Done false
33812 9488 Acked Write 5 Done Sent Done false
33813 9489 Acked Read 1 Done Sent Done false
33814 9490 Acked Write 2 Done Sent Done false
33815 9491 Acked Write 1 Done Sent Done false
33816 9492 Acked Read 9 Done Sent Done false
33817 9493 Acked Write 7 Done Sent Done false
33818 9494 Acked Read 1 Done Sent Done false
33819 9495 Acked Read 6 Done Sent Done false
33820 9496 Acked Read 10 Done Sent Done false
33821 9497 Acked Read 3 Done Sent Done false
33822 9498 Acked Write 10 Done Sent Done false
33823 9499 Acked Write 10 Done Sent Done false
33824 9500 Acked Read 9 Done Sent Done false
33825 9501 Acked Read 9 Done Sent Done false
33826 9502 Acked Read 6 Done Sent Done false
33827 9503 Acked Write 9 Done Sent Done false
33828 9504 Acked Read 2 Done Sent Done false
33829 9505 Acked Write 1 Done Sent Done false
33830 9506 Acked Flush 0 Done Sent Done false
33831 9507 Acked Read 7 Done Sent Done false
33832 9508 Acked Write 10 Done Sent Done false
33833 9509 Acked Write 9 Done Sent Done false
33834 9510 Acked Write 8 Done Sent Done false
33835 9511 Acked Write 1 Done Sent Done false
33836 9512 Acked Flush 0 Done Sent Done false
33837 9513 Acked Write 10 Done Sent Done false
33838 9514 Acked Read 9 Done Sent Done false
33839 9515 Acked Read 7 Done Sent Done false
33840 9516 Acked Read 7 Done Sent Done false
33841 9517 Acked Read 2 Done Sent Done false
33842 9518 Acked Read 9 Done Sent Done false
33843 9519 Acked Flush 0 Done Sent Done false
33844 9520 Acked Read 3 Done Sent Done false
33845 9521 Acked Read 8 Done Sent Done false
33846 9522 Acked Write 7 Done Sent Done false
33847 9523 Acked Write 7 Done Sent Done false
33848 9524 Acked Read 9 Done Sent Done false
33849 9525 Acked Write 2 Done Sent Done false
33850 9526 Acked Read 10 Done Sent Done false
33851 9527 Acked Write 1 Done Sent Done false
33852 9528 Acked Write 9 Done Sent Done false
33853 9529 Acked Read 6 Done Sent Done false
33854 9530 Acked Read 6 Done Sent Done false
33855 9531 Acked Read 10 Done Sent Done false
33856 9532 Acked Write 2 Done Sent Done false
33857 9533 Acked Read 5 Done Sent Done false
33858 9534 Acked Read 8 Done Sent Done false
33859 9535 Acked Read 5 Done Sent Done false
33860 9536 Acked Read 7 Done Sent Done false
33861 9537 Acked Write 4 Done Sent Done false
33862 9538 Acked Write 5 Done Sent Done false
33863 9539 Acked Read 8 Done Sent Done false
33864 9540 Acked Read 7 Done Sent Done false
33865 9541 Acked Read 8 Done Sent Done false
33866 9542 Acked Write 3 Done Sent Done false
33867 9543 Acked Write 4 Done Sent Done false
33868 9544 Acked Read 9 Done Sent Done false
33869 9545 Acked Flush 0 Done Sent Done false
33870 9546 Acked Write 4 Done Sent Done false
33871 9547 Acked Read 9 Done Sent Done false
33872 9548 Acked Read 6 Done Sent Done false
33873 9549 Acked Write 3 Done Sent Done false
33874 9550 Acked Read 9 Done Sent Done false
33875 9551 Acked Write 6 Done Sent Done false
33876 9552 Acked Write 8 Done Sent Done false
33877 9553 Acked Write 1 Done Sent Done false
33878 9554 Acked Read 9 Done Sent Done false
33879 9555 Acked Write 9 Done Sent Done false
33880 9556 Acked Write 9 Done Sent Done false
33881 9557 Acked Write 2 Done Sent Done false
33882 9558 Acked Write 7 Done Sent Done false
33883 9559 Acked Read 3 Done Sent Done false
33884 9560 Acked Flush 0 Done Sent Done false
33885 9561 Acked Read 9 Done Sent Done false
33886 9562 Acked Read 8 Done Sent Done false
33887 9563 Acked Read 3 Done Sent Done false
33888 9564 Acked Read 6 Done Sent Done false
33889 9565 Acked Write 6 Done Sent Done false
33890 9566 Acked Read 7 Done Sent Done false
33891 9567 Acked Read 6 Done Sent Done false
33892 9568 Acked Write 5 Done Sent Done false
33893 9569 Acked Read 1 Done Sent Done false
33894 9570 Acked Read 2 Done Sent Done false
33895 9571 Acked Read 9 Done Sent Done false
33896 9572 Acked Read 7 Done Sent Done false
33897 9573 Acked Write 3 Done Sent Done false
33898 9574 Acked Flush 0 Done Sent Done false
33899 9575 Acked Read 10 Done Sent Done false
33900 9576 Acked Read 2 Done Sent Done false
33901 9577 Acked Read 7 Done Sent Done false
33902 9578 Acked Read 3 Done Sent Done false
33903 9579 Acked Read 8 Done Sent Done false
33904 9580 Acked Read 7 Done Sent Done false
33905 9581 Acked Read 8 Done Sent Done false
33906 9582 Acked Read 1 Done Sent Done false
33907 9583 Acked Read 9 Done Sent Done false
33908 9584 Acked Write 4 Done Sent Done false
33909 9585 Acked Read 5 Done Sent Done false
33910 9586 Acked Flush 0 Done Sent Done false
33911 9587 Acked Read 3 Done Sent Done false
33912 9588 Acked Read 4 Done Sent Done false
33913 9589 Acked Read 8 Done Sent Done false
33914 9590 Acked Write 5 Done Sent Done false
33915 9591 Acked Read 3 Done Sent Done false
33916 9592 Acked Flush 0 Done Sent Done false
33917 9593 Acked Write 10 Done Sent Done false
33918 9594 Acked Read 6 Done Sent Done false
33919 9595 Acked Read 10 Done Sent Done false
33920 9596 Acked Read 7 Done Sent Done false
33921 9597 Acked Read 7 Done Sent Done false
33922 9598 Acked Read 9 Done Sent Done false
33923 9599 Acked Write 8 Done Sent Done false
33924 9600 Acked Read 2 Done Sent Done false
33925 9601 Acked Write 7 Done Sent Done false
33926 9602 Acked Write 8 Done Sent Done false
33927 9603 Acked Write 6 Done Sent Done false
33928 9604 Acked Write 3 Done Sent Done false
33929 9605 Acked Read 1 Done Sent Done false
33930 9606 Acked Flush 0 Done Sent Done false
33931 9607 Acked Write 7 Done Sent Done false
33932 9608 Acked Write 7 Done Sent Done false
33933 9609 Acked Read 10 Done Sent Done false
33934 9610 Acked Write 10 Done Sent Done false
33935 9611 Acked Flush 0 Done Sent Done false
33936 9612 Acked Write 2 Done Sent Done false
33937 9613 Acked Flush 0 Done Sent Done false
33938 9614 Acked Write 5 Done Sent Done false
33939 9615 Acked Read 4 Done Sent Done false
33940 9616 Acked Read 6 Done Sent Done false
33941 9617 Acked Read 10 Done Sent Done false
33942 9618 Acked Write 3 Done Sent Done false
33943 9619 Acked Write 9 Done Sent Done false
33944 9620 Acked Write 1 Done Sent Done false
33945 9621 Acked Read 8 Done Sent Done false
33946 9622 Acked Read 10 Done Sent Done false
33947 9623 Acked Write 1 Done Sent Done false
33948 9624 Acked Write 1 Done Sent Done false
33949 9625 Acked Read 7 Done Sent Done false
33950 9626 Acked Flush 0 Done Sent Done false
33951 9627 Acked Read 7 Done Sent Done false
33952 9628 Acked Write 10 Done Sent Done false
33953 9629 Acked Read 7 Done Sent Done false
33954 9630 Acked Write 9 Done Sent Done false
33955 9631 Acked Read 7 Done Sent Done false
33956 9632 Acked Write 1 Done Sent Done false
33957 9633 Acked Flush 0 Done Sent Done false
33958 9634 Acked Read 2 Done Sent Done false
33959 9635 Acked Flush 0 Done Sent Done false
33960 9636 Acked Read 7 Done Sent Done false
33961 9637 Acked Write 4 Done Sent Done false
33962 9638 Acked Read 10 Done Sent Done false
33963 9639 Acked Read 2 Done Sent Done false
33964 9640 Acked Read 4 Done Sent Done false
33965 9641 Acked Read 10 Done Sent Done false
33966 9642 Acked Write 9 Done Sent Done false
33967 9643 Acked Write 9 Done Sent Done false
33968 9644 Acked Write 5 Done Sent Done false
33969 9645 Acked Flush 0 Done Sent Done false
33970 9646 Acked Write 8 Done Sent Done false
33971 9647 Acked Read 4 Done Sent Done false
33972 9648 Acked Read 9 Done Sent Done false
33973 9649 Acked Read 5 Done Sent Done false
33974 9650 Acked Read 8 Done Sent Done false
33975 9651 Acked Read 6 Done Sent Done false
33976 9652 Acked Read 2 Done Sent Done false
33977 9653 Acked Write 2 Done Sent Done false
33978 9654 Acked Flush 0 Done Sent Done false
33979 9655 Acked Read 7 Done Sent Done false
33980 9656 Acked Read 3 Done Sent Done false
33981 9657 Acked Read 3 Done Sent Done false
33982 9658 Acked Write 3 Done Sent Done false
33983 9659 Acked Write 6 Done Sent Done false
33984 9660 Acked Read 10 Done Sent Done false
33985 9661 Acked Read 1 Done Sent Done false
33986 9662 Acked Read 7 Done Sent Done false
33987 9663 Acked Read 9 Done Sent Done false
33988 9664 Acked Read 10 Done Sent Done false
33989 9665 Acked Read 3 Done Sent Done false
33990 9666 Acked Read 6 Done Sent Done false
33991 9667 Acked Read 10 Done Sent Done false
33992 9668 Acked Flush 0 Done Sent Done false
33993 9669 Acked Read 7 Done Sent Done false
33994 9670 Acked Read 7 Done Sent Done false
33995 9671 Acked Write 3 Done Sent Done false
33996 9672 Acked Write 10 Done Sent Done false
33997 9673 Acked Read 4 Done Sent Done false
33998 9674 Acked Write 6 Done Sent Done false
33999 9675 Acked Read 2 Done Sent Done false
34000 9676 Acked Read 10 Done Sent Done false
34001 9677 Acked Write 6 Done Sent Done false
34002 9678 Acked Flush 0 Done Sent Done false
34003 9679 Acked Flush 0 Done Sent Done false
34004 9680 Acked Write 2 Done Sent Done false
34005 9681 Acked Read 2 Done Sent Done false
34006 9682 Acked Write 7 Done Sent Done false
34007 9683 Acked Read 10 Done Sent Done false
34008 9684 Acked Write 7 Done Sent Done false
34009 9685 Acked Write 6 Done Sent Done false
34010 9686 Acked Read 1 Done Sent Done false
34011 9687 Acked Read 3 Done Sent Done false
34012 9688 Acked Write 4 Done Sent Done false
34013 9689 Acked Write 6 Done Sent Done false
34014 9690 Acked Flush 0 Done Sent Done false
34015 9691 Acked Read 2 Done Sent Done false
34016 9692 Acked Write 9 Done Sent Done false
34017 9693 Acked Write 9 Done Sent Done false
34018 9694 Acked Read 3 Done Sent Done false
34019 9695 Acked Read 2 Done Sent Done false
34020 9696 Acked Write 10 Done Sent Done false
34021 9697 Acked Write 10 Done Sent Done false
34022 9698 Acked Read 2 Done Sent Done false
34023 9699 Acked Read 5 Done Sent Done false
34024 9700 Acked Write 1 Done Sent Done false
34025 9701 Acked Read 3 Done Sent Done false
34026 9702 Acked Read 7 Done Sent Done false
34027 9703 Acked Read 7 Done Sent Done false
34028 9704 Acked Read 2 Done Sent Done false
34029 9705 Acked Read 5 Done Sent Done false
34030 9706 Acked Write 10 Done Sent Done false
34031 9707 Acked Flush 0 Done Sent Done false
34032 9708 Acked Read 3 Done Sent Done false
34033 9709 Acked Read 8 Done Sent Done false
34034 9710 Acked Read 2 Done Sent Done false
34035 9711 Acked Read 9 Done Sent Done false
34036 9712 Acked Read 4 Done Sent Done false
34037 9713 Acked Write 7 Done Sent Done false
34038 9714 Acked Write 2 Done Sent Done false
34039 9715 Acked Read 7 Done Sent Done false
34040 9716 Acked Read 9 Done Sent Done false
34041 9717 Acked Flush 0 Done Sent Done false
34042 9718 Acked Read 9 Done Sent Done false
34043 9719 Acked Read 4 Done Sent Done false
34044 9720 Acked Read 4 Done Sent Done false
34045 9721 Acked Flush 0 Done Sent Done false
34046 9722 Acked Read 8 Done Sent Done false
34047 9723 Acked Write 8 Done Sent Done false
34048 9724 Acked Read 6 Done Sent Done false
34049 9725 Acked Flush 0 Done Sent Done false
34050 9726 Acked Read 9 Done Sent Done false
34051 9727 Acked Write 6 Done Sent Done false
34052 9728 Acked Read 3 Done Sent Done false
34053 9729 Acked Write 2 Done Sent Done false
34054 9730 Acked Flush 0 Done Sent Done false
34055 9731 Acked Read 8 Done Sent Done false
34056 9732 Acked Read 9 Done Sent Done false
34057 9733 Acked Read 10 Done Sent Done false
34058 9734 Acked Read 7 Done Sent Done false
34059 9735 Acked Write 5 Done Sent Done false
34060 9736 Acked Read 3 Done Sent Done false
34061 9737 Acked Read 1 Done Sent Done false
34062 9738 Acked Write 1 Done Sent Done false
34063 9739 Acked Write 4 Done Sent Done false
34064 9740 Acked Flush 0 Done Sent Done false
34065 9741 Acked Read 7 Done Sent Done false
34066 9742 Acked Write 7 Done Sent Done false
34067 9743 Acked Write 8 Done Sent Done false
34068 9744 Acked Read 2 Done Sent Done false
34069 9745 Acked Read 8 Done Sent Done false
34070 9746 Acked Read 10 Done Sent Done false
34071 9747 Acked Read 9 Done Sent Done false
34072 9748 Acked Read 4 Done Sent Done false
34073 9749 Acked Read 8 Done Sent Done false
34074 9750 Acked Read 1 Done Sent Done false
34075 9751 Acked Write 9 Done Sent Done false
34076 9752 Acked Write 6 Done Sent Done false
34077 STATES DS:0 DS:1 DS:2 TOTAL
34078 Sent 0 301 0 301
34079 Done 301 0 301 602
34080 Skipped 0 0 0 0
34081 Error 0 0 0 0
34082 Last Flush: Some(JobId(9740)) Some(JobId(9451)) Some(JobId(9740))
34083 Downstairs last five completed: 9451 9450 9449 9448 9447
34084 Upstairs last five completed: 9752 9751 9750 9749 9748
34085 CLIENT: Up:0 ds:301 act:2
340862025-02-21T06:52:55.351ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
340872025-02-21T06:52:57.355ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
340882025-02-21T06:52:58.357ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
34089 [8820] got start action
34090 [8820] I am going to restart
34091 Starting downstairs at port 8820
34092 Make output file at "/tmp/dsc/downstairs-8820.txt"
34093 [8820][1] reports Starting
34094 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1235)
34095 [8820][1] reports Running
34096 ----------------------------------------------------------------
34097 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34098 States: Active Offline Active
34099 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34100 9452 Acked Write 6 Done Sent Done false
34101 9453 Acked Read 1 Done Sent Done false
34102 9454 Acked Read 2 Done Sent Done false
34103 9455 Acked Write 2 Done Sent Done false
34104 9456 Acked Write 10 Done Sent Done false
34105 9457 Acked Read 5 Done Sent Done false
34106 9458 Acked Read 7 Done Sent Done false
34107 9459 Acked Read 5 Done Sent Done false
34108 9460 Acked Read 3 Done Sent Done false
34109 9461 Acked Read 4 Done Sent Done false
34110 9462 Acked Write 4 Done Sent Done false
34111 9463 Acked Read 3 Done Sent Done false
34112 9464 Acked Write 1 Done Sent Done false
34113 9465 Acked Write 9 Done Sent Done false
34114 9466 Acked Read 1 Done Sent Done false
34115 9467 Acked Write 10 Done Sent Done false
34116 9468 Acked Read 1 Done Sent Done false
34117 9469 Acked Flush 0 Done Sent Done false
34118 9470 Acked Write 2 Done Sent Done false
34119 9471 Acked Read 3 Done Sent Done false
34120 9472 Acked Flush 0 Done Sent Done false
34121 9473 Acked Read 1 Done Sent Done false
34122 9474 Acked Read 10 Done Sent Done false
34123 9475 Acked Read 1 Done Sent Done false
34124 9476 Acked Read 8 Done Sent Done false
34125 9477 Acked Read 10 Done Sent Done false
34126 9478 Acked Write 9 Done Sent Done false
34127 9479 Acked Read 9 Done Sent Done false
34128 9480 Acked Read 3 Done Sent Done false
34129 9481 Acked Read 5 Done Sent Done false
34130 9482 Acked Flush 0 Done Sent Done false
34131 9483 Acked Write 4 Done Sent Done false
34132 9484 Acked Read 4 Done Sent Done false
34133 9485 Acked Read 9 Done Sent Done false
34134 9486 Acked Flush 0 Done Sent Done false
34135 9487 Acked Write 8 Done Sent Done false
34136 9488 Acked Write 5 Done Sent Done false
34137 9489 Acked Read 1 Done Sent Done false
34138 9490 Acked Write 2 Done Sent Done false
34139 9491 Acked Write 1 Done Sent Done false
34140 9492 Acked Read 9 Done Sent Done false
34141 9493 Acked Write 7 Done Sent Done false
34142 9494 Acked Read 1 Done Sent Done false
34143 9495 Acked Read 6 Done Sent Done false
34144 9496 Acked Read 10 Done Sent Done false
34145 9497 Acked Read 3 Done Sent Done false
34146 9498 Acked Write 10 Done Sent Done false
34147 9499 Acked Write 10 Done Sent Done false
34148 9500 Acked Read 9 Done Sent Done false
34149 9501 Acked Read 9 Done Sent Done false
34150 9502 Acked Read 6 Done Sent Done false
34151 9503 Acked Write 9 Done Sent Done false
34152 9504 Acked Read 2 Done Sent Done false
34153 9505 Acked Write 1 Done Sent Done false
34154 9506 Acked Flush 0 Done Sent Done false
34155 9507 Acked Read 7 Done Sent Done false
34156 9508 Acked Write 10 Done Sent Done false
34157 9509 Acked Write 9 Done Sent Done false
34158 9510 Acked Write 8 Done Sent Done false
34159 9511 Acked Write 1 Done Sent Done false
34160 9512 Acked Flush 0 Done Sent Done false
34161 9513 Acked Write 10 Done Sent Done false
34162 9514 Acked Read 9 Done Sent Done false
34163 9515 Acked Read 7 Done Sent Done false
34164 9516 Acked Read 7 Done Sent Done false
34165 9517 Acked Read 2 Done Sent Done false
34166 9518 Acked Read 9 Done Sent Done false
34167 9519 Acked Flush 0 Done Sent Done false
34168 9520 Acked Read 3 Done Sent Done false
34169 9521 Acked Read 8 Done Sent Done false
34170 9522 Acked Write 7 Done Sent Done false
34171 9523 Acked Write 7 Done Sent Done false
34172 9524 Acked Read 9 Done Sent Done false
34173 9525 Acked Write 2 Done Sent Done false
34174 9526 Acked Read 10 Done Sent Done false
34175 9527 Acked Write 1 Done Sent Done false
34176 9528 Acked Write 9 Done Sent Done false
34177 9529 Acked Read 6 Done Sent Done false
34178 9530 Acked Read 6 Done Sent Done false
34179 9531 Acked Read 10 Done Sent Done false
34180 9532 Acked Write 2 Done Sent Done false
34181 9533 Acked Read 5 Done Sent Done false
34182 9534 Acked Read 8 Done Sent Done false
34183 9535 Acked Read 5 Done Sent Done false
34184 9536 Acked Read 7 Done Sent Done false
34185 9537 Acked Write 4 Done Sent Done false
34186 9538 Acked Write 5 Done Sent Done false
34187 9539 Acked Read 8 Done Sent Done false
34188 9540 Acked Read 7 Done Sent Done false
34189 9541 Acked Read 8 Done Sent Done false
34190 9542 Acked Write 3 Done Sent Done false
34191 9543 Acked Write 4 Done Sent Done false
34192 9544 Acked Read 9 Done Sent Done false
34193 9545 Acked Flush 0 Done Sent Done false
34194 9546 Acked Write 4 Done Sent Done false
34195 9547 Acked Read 9 Done Sent Done false
34196 9548 Acked Read 6 Done Sent Done false
34197 9549 Acked Write 3 Done Sent Done false
34198 9550 Acked Read 9 Done Sent Done false
34199 9551 Acked Write 6 Done Sent Done false
34200 9552 Acked Write 8 Done Sent Done false
34201 9553 Acked Write 1 Done Sent Done false
34202 9554 Acked Read 9 Done Sent Done false
34203 9555 Acked Write 9 Done Sent Done false
34204 9556 Acked Write 9 Done Sent Done false
34205 9557 Acked Write 2 Done Sent Done false
34206 9558 Acked Write 7 Done Sent Done false
34207 9559 Acked Read 3 Done Sent Done false
34208 9560 Acked Flush 0 Done Sent Done false
34209 9561 Acked Read 9 Done Sent Done false
34210 9562 Acked Read 8 Done Sent Done false
34211 9563 Acked Read 3 Done Sent Done false
34212 9564 Acked Read 6 Done Sent Done false
34213 9565 Acked Write 6 Done Sent Done false
34214 9566 Acked Read 7 Done Sent Done false
34215 9567 Acked Read 6 Done Sent Done false
34216 9568 Acked Write 5 Done Sent Done false
34217 9569 Acked Read 1 Done Sent Done false
34218 9570 Acked Read 2 Done Sent Done false
34219 9571 Acked Read 9 Done Sent Done false
34220 9572 Acked Read 7 Done Sent Done false
34221 9573 Acked Write 3 Done Sent Done false
34222 9574 Acked Flush 0 Done Sent Done false
34223 9575 Acked Read 10 Done Sent Done false
34224 9576 Acked Read 2 Done Sent Done false
34225 9577 Acked Read 7 Done Sent Done false
34226 9578 Acked Read 3 Done Sent Done false
34227 9579 Acked Read 8 Done Sent Done false
34228 9580 Acked Read 7 Done Sent Done false
34229 9581 Acked Read 8 Done Sent Done false
34230 9582 Acked Read 1 Done Sent Done false
34231 9583 Acked Read 9 Done Sent Done false
34232 9584 Acked Write 4 Done Sent Done false
34233 9585 Acked Read 5 Done Sent Done false
34234 9586 Acked Flush 0 Done Sent Done false
34235 9587 Acked Read 3 Done Sent Done false
34236 9588 Acked Read 4 Done Sent Done false
34237 9589 Acked Read 8 Done Sent Done false
34238 9590 Acked Write 5 Done Sent Done false
34239 9591 Acked Read 3 Done Sent Done false
34240 9592 Acked Flush 0 Done Sent Done false
34241 9593 Acked Write 10 Done Sent Done false
34242 9594 Acked Read 6 Done Sent Done false
34243 9595 Acked Read 10 Done Sent Done false
34244 9596 Acked Read 7 Done Sent Done false
34245 9597 Acked Read 7 Done Sent Done false
34246 9598 Acked Read 9 Done Sent Done false
34247 9599 Acked Write 8 Done Sent Done false
34248 9600 Acked Read 2 Done Sent Done false
34249 9601 Acked Write 7 Done Sent Done false
34250 9602 Acked Write 8 Done Sent Done false
34251 9603 Acked Write 6 Done Sent Done false
34252 9604 Acked Write 3 Done Sent Done false
34253 9605 Acked Read 1 Done Sent Done false
34254 9606 Acked Flush 0 Done Sent Done false
34255 9607 Acked Write 7 Done Sent Done false
34256 9608 Acked Write 7 Done Sent Done false
34257 9609 Acked Read 10 Done Sent Done false
34258 9610 Acked Write 10 Done Sent Done false
34259 9611 Acked Flush 0 Done Sent Done false
34260 9612 Acked Write 2 Done Sent Done false
34261 9613 Acked Flush 0 Done Sent Done false
34262 9614 Acked Write 5 Done Sent Done false
34263 9615 Acked Read 4 Done Sent Done false
34264 9616 Acked Read 6 Done Sent Done false
34265 9617 Acked Read 10 Done Sent Done false
34266 9618 Acked Write 3 Done Sent Done false
34267 9619 Acked Write 9 Done Sent Done false
34268 9620 Acked Write 1 Done Sent Done false
34269 9621 Acked Read 8 Done Sent Done false
34270 9622 Acked Read 10 Done Sent Done false
34271 9623 Acked Write 1 Done Sent Done false
34272 9624 Acked Write 1 Done Sent Done false
34273 9625 Acked Read 7 Done Sent Done false
34274 9626 Acked Flush 0 Done Sent Done false
34275 9627 Acked Read 7 Done Sent Done false
34276 9628 Acked Write 10 Done Sent Done false
34277 9629 Acked Read 7 Done Sent Done false
34278 9630 Acked Write 9 Done Sent Done false
34279 9631 Acked Read 7 Done Sent Done false
34280 9632 Acked Write 1 Done Sent Done false
34281 9633 Acked Flush 0 Done Sent Done false
34282 9634 Acked Read 2 Done Sent Done false
34283 9635 Acked Flush 0 Done Sent Done false
34284 9636 Acked Read 7 Done Sent Done false
34285 9637 Acked Write 4 Done Sent Done false
34286 9638 Acked Read 10 Done Sent Done false
34287 9639 Acked Read 2 Done Sent Done false
34288 9640 Acked Read 4 Done Sent Done false
34289 9641 Acked Read 10 Done Sent Done false
34290 9642 Acked Write 9 Done Sent Done false
34291 9643 Acked Write 9 Done Sent Done false
34292 9644 Acked Write 5 Done Sent Done false
34293 9645 Acked Flush 0 Done Sent Done false
34294 9646 Acked Write 8 Done Sent Done false
34295 9647 Acked Read 4 Done Sent Done false
34296 9648 Acked Read 9 Done Sent Done false
34297 9649 Acked Read 5 Done Sent Done false
34298 9650 Acked Read 8 Done Sent Done false
34299 9651 Acked Read 6 Done Sent Done false
34300 9652 Acked Read 2 Done Sent Done false
34301 9653 Acked Write 2 Done Sent Done false
34302 9654 Acked Flush 0 Done Sent Done false
34303 9655 Acked Read 7 Done Sent Done false
34304 9656 Acked Read 3 Done Sent Done false
34305 9657 Acked Read 3 Done Sent Done false
34306 9658 Acked Write 3 Done Sent Done false
34307 9659 Acked Write 6 Done Sent Done false
34308 9660 Acked Read 10 Done Sent Done false
34309 9661 Acked Read 1 Done Sent Done false
34310 9662 Acked Read 7 Done Sent Done false
34311 9663 Acked Read 9 Done Sent Done false
34312 9664 Acked Read 10 Done Sent Done false
34313 9665 Acked Read 3 Done Sent Done false
34314 9666 Acked Read 6 Done Sent Done false
34315 9667 Acked Read 10 Done Sent Done false
34316 9668 Acked Flush 0 Done Sent Done false
34317 9669 Acked Read 7 Done Sent Done false
34318 9670 Acked Read 7 Done Sent Done false
34319 9671 Acked Write 3 Done Sent Done false
34320 9672 Acked Write 10 Done Sent Done false
34321 9673 Acked Read 4 Done Sent Done false
34322 9674 Acked Write 6 Done Sent Done false
34323 9675 Acked Read 2 Done Sent Done false
34324 9676 Acked Read 10 Done Sent Done false
34325 9677 Acked Write 6 Done Sent Done false
34326 9678 Acked Flush 0 Done Sent Done false
34327 9679 Acked Flush 0 Done Sent Done false
34328 9680 Acked Write 2 Done Sent Done false
34329 9681 Acked Read 2 Done Sent Done false
34330 9682 Acked Write 7 Done Sent Done false
34331 9683 Acked Read 10 Done Sent Done false
34332 9684 Acked Write 7 Done Sent Done false
34333 9685 Acked Write 6 Done Sent Done false
34334 9686 Acked Read 1 Done Sent Done false
34335 9687 Acked Read 3 Done Sent Done false
34336 9688 Acked Write 4 Done Sent Done false
34337 9689 Acked Write 6 Done Sent Done false
34338 9690 Acked Flush 0 Done Sent Done false
34339 9691 Acked Read 2 Done Sent Done false
34340 9692 Acked Write 9 Done Sent Done false
34341 9693 Acked Write 9 Done Sent Done false
34342 9694 Acked Read 3 Done Sent Done false
34343 9695 Acked Read 2 Done Sent Done false
34344 9696 Acked Write 10 Done Sent Done false
34345 9697 Acked Write 10 Done Sent Done false
34346 9698 Acked Read 2 Done Sent Done false
34347 9699 Acked Read 5 Done Sent Done false
34348 9700 Acked Write 1 Done Sent Done false
34349 9701 Acked Read 3 Done Sent Done false
34350 9702 Acked Read 7 Done Sent Done false
34351 9703 Acked Read 7 Done Sent Done false
34352 9704 Acked Read 2 Done Sent Done false
34353 9705 Acked Read 5 Done Sent Done false
34354 9706 Acked Write 10 Done Sent Done false
34355 9707 Acked Flush 0 Done Sent Done false
34356 9708 Acked Read 3 Done Sent Done false
34357 9709 Acked Read 8 Done Sent Done false
34358 9710 Acked Read 2 Done Sent Done false
34359 9711 Acked Read 9 Done Sent Done false
34360 9712 Acked Read 4 Done Sent Done false
34361 9713 Acked Write 7 Done Sent Done false
34362 9714 Acked Write 2 Done Sent Done false
34363 9715 Acked Read 7 Done Sent Done false
34364 9716 Acked Read 9 Done Sent Done false
34365 9717 Acked Flush 0 Done Sent Done false
34366 9718 Acked Read 9 Done Sent Done false
34367 9719 Acked Read 4 Done Sent Done false
34368 9720 Acked Read 4 Done Sent Done false
34369 9721 Acked Flush 0 Done Sent Done false
34370 9722 Acked Read 8 Done Sent Done false
34371 9723 Acked Write 8 Done Sent Done false
34372 9724 Acked Read 6 Done Sent Done false
34373 9725 Acked Flush 0 Done Sent Done false
34374 9726 Acked Read 9 Done Sent Done false
34375 9727 Acked Write 6 Done Sent Done false
34376 9728 Acked Read 3 Done Sent Done false
34377 9729 Acked Write 2 Done Sent Done false
34378 9730 Acked Flush 0 Done Sent Done false
34379 9731 Acked Read 8 Done Sent Done false
34380 9732 Acked Read 9 Done Sent Done false
34381 9733 Acked Read 10 Done Sent Done false
34382 9734 Acked Read 7 Done Sent Done false
34383 9735 Acked Write 5 Done Sent Done false
34384 9736 Acked Read 3 Done Sent Done false
34385 9737 Acked Read 1 Done Sent Done false
34386 9738 Acked Write 1 Done Sent Done false
34387 9739 Acked Write 4 Done Sent Done false
34388 9740 Acked Flush 0 Done Sent Done false
34389 9741 Acked Read 7 Done Sent Done false
34390 9742 Acked Write 7 Done Sent Done false
34391 9743 Acked Write 8 Done Sent Done false
34392 9744 Acked Read 2 Done Sent Done false
34393 9745 Acked Read 8 Done Sent Done false
34394 9746 Acked Read 10 Done Sent Done false
34395 9747 Acked Read 9 Done Sent Done false
34396 9748 Acked Read 4 Done Sent Done false
34397 9749 Acked Read 8 Done Sent Done false
34398 9750 Acked Read 1 Done Sent Done false
34399 9751 Acked Write 9 Done Sent Done false
34400 9752 Acked Write 6 Done Sent Done false
34401 STATES DS:0 DS:1 DS:2 TOTAL
34402 Sent 0 301 0 301
34403 Done 301 0 301 602
34404 Skipped 0 0 0 0
34405 Error 0 0 0 0
34406 Last Flush: Some(JobId(9740)) Some(JobId(9451)) Some(JobId(9740))
34407 Downstairs last five completed: 9451 9450 9449 9448 9447
34408 Upstairs last five completed: 9752 9751 9750 9749 9748
34409 CLIENT: Up:0 ds:301 act:2
344102025-02-21T06:52:59.358ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
344112025-02-21T06:53:00.350ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
344122025-02-21T06:53:00.350ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58679) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
344132025-02-21T06:53:00.351ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
344142025-02-21T06:53:00.352ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
344152025-02-21T06:53:00.352ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
344162025-02-21T06:53:00.352ZINFOcrucible: send last flush ID to this DS: Some(JobId(9451)) = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
344172025-02-21T06:53:00.352ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
344182025-02-21T06:53:00.352ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(9451)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
344192025-02-21T06:53:00.352ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(9451)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
34420 ----------------------------------------------------------------
34421 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
34422 Downstairs last five completed: 9753 9752 9751 9750 9749
34423 Upstairs last five completed: 9753 9752 9751 9750 9749
34424 CLIENT: Up:0 ds:0 act:3
34425 Replay: All jobs finished
34426 Main task has work to do, go find it
34427 got dsc Stop(1)
34428 stop 1
34429 Feb 21 06:53:02.617[8820] Got stop action so:false kr:false
34430 INFO request completed, latency_us: 367, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 400eb8a4-b076-4fc3-a2df-e67109b06e9e, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
344312025-02-21T06:53:02.618ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
34432 Feb 21 06:53:02.618 INFO request completed, latency_us: 159, response_code: 200, uri: /state/cid/1, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:53:02.618678994Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
34433 method: GET, req_id:{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:53:02.618736904Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
34434 430b6a41-f083-4312-8ea1-c43b909c0464, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:53:02.618807761Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"3a757114-e39d-42e6-bc2b-0419bce3aed2"}
34435 9998
34436 [8820] Exited with: ExitStatus(unix_wait_status(9))
34437 [8820][1] reports Exit
34438 Feb 21 06:53:06.620 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4b919865-0884-4551-ba63-90a7a482d2b4, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
34439 001/300 Read block 2571 len 8192
34440 002/300 Write block 2626 len 12288 data: 7 7 7
34441 003/300 Flush
34442 004/300 Write block 2121 len 32768 data: 6 6 8 5 5 5 5 5
34443 005/300 Write block 1457 len 16384 data: 5 5 6 7
34444 006/300 Read block 457 len 4096
34445 007/300 Write block 2227 len 4096 data: 9
34446 008/300 Read block 294 len 4096
34447 009/300 Write block 3076 len 40960 data: 6 7 8 8 8 8 6 6 6 8
34448 010/300 Write block 2646 len 4096 data: 6
34449 011/300 Read block 1213 len 20480
34450 012/300 Read block 4366 len 12288
34451 013/300 Flush
34452 014/300 Write block 1424 len 36864 data: 11 9 11 8 5 3 3 3 3
34453 015/300 Write block 1208 len 20480 data: 3 4 3 5 9
34454 016/300 Write block 839 len 24576 data: 5 5 4 6 7 5
34455 017/300 Read block 2657 len 40960
34456 018/300 Read block 1878 len 40960
34457 019/300 Flush
34458 020/300 Read block 3959 len 8192
34459 021/300 Write block 3956 len 8192 data: 7 7
34460 022/300 Write block 2467 len 4096 data: 3
34461 023/300 Read block 264 len 32768
34462 024/300 Write block 2126 len 12288 data: 6 6 6
34463 025/300 Write block 4432 len 12288 data: 4 5 4
34464 026/300 Flush
34465 027/300 Read block 2859 len 16384
34466 028/300 Flush
34467 029/300 Write block 760 len 24576 data: 5 5 4 5 5 6
34468 030/300 Read block 3294 len 8192
34469 031/300 Write block 4317 len 20480 data: 3 3 4 4 3
34470 032/300 Read block 3008 len 20480
34471 033/300 Write block 4244 len 36864 data: 6 3 3 3 3 3 3 2 2
34472 034/300 Read block 2481 len 20480
34473 035/300 Write block 2151 len 16384 data: 5 6 5 7
34474 036/300 Read block 1794 len 36864
34475 037/300 Read block 4134 len 8192
34476 038/300 Write block 4162 len 24576 data: 4 4 4 3 3 3
34477 039/300 Write block 2396 len 16384 data: 7 7 4 3
34478 040/300 Flush
34479 041/300 Write block 2964 len 12288 data: 6 7 6
34480 042/300 Write block 270 len 36864 data: 2 3 5 5 6 4 4 3 2
34481 043/300 Write block 3194 len 4096 data: 7
34482 044/300 Read block 3423 len 24576
34483 045/300 Read block 2944 len 12288
34484 046/300 Write block 4762 len 28672 data: 6 6 4 4 5 4 4
34485 047/300 Write block 3872 len 8192 data: 9 11
34486 048/300 Flush
34487 049/300 Read block 3061 len 16384
34488 050/300 Write block 2342 len 40960 data: 8 8 8 8 7 8 7 6 4 4
34489 051/300 Write block 647 len 28672 data: 7 5 4 5 5 7 6
34490 052/300 Write block 4197 len 32768 data: 5 5 6 5 6 7 7 6
34491 053/300 Write block 4609 len 40960 data: 7 8 6 6 5 7 8 9 8 9
34492 054/300 Write block 2519 len 20480 data: 5 4 4 3 3
34493 055/300 Write block 57 len 4096 data: 5
34494 056/300 Read block 4731 len 12288
34495 057/300 Write block 282 len 4096 data: 5
34496 058/300 Read block 887 len 20480
34497 059/300 Write block 2562 len 20480 data: 9 7 7 7 4
34498 060/300 Read block 848 len 40960
34499 061/300 Write block 2802 len 28672 data: 6 6 6 5 7 5 4
34500 062/300 Flush
34501 063/300 Write block 4925 len 16384 data: 5 3 3 4
34502 064/300 Write block 2071 len 24576 data: 10 9 9 7 6 6
34503 065/300 Write block 4277 len 32768 data: 5 3 5 6 6 6 8 8
34504 066/300 Read block 4425 len 20480
34505 067/300 Read block 2038 len 4096
34506 068/300 Read block 3041 len 40960
34507 069/300 Write block 3219 len 20480 data: 4 7 6 6 5
34508 070/300 Flush
34509 071/300 Read block 3394 len 8192
34510 072/300 Write block 2180 len 28672 data: 6 7 5 5 4 5 6
34511 073/300 Flush
34512 074/300 Write block 102 len 28672 data: 8 8 8 9 8 6 6
34513 075/300 Read block 2715 len 12288
34514 076/300 Write block 1326 len 4096 data: 5
34515 077/300 Read block 1220 len 20480
34516 078/300 Read block 4356 len 32768
34517 079/300 Write block 1065 len 4096 data: 4
34518 080/300 Read block 214 len 16384
34519 081/300 Read block 4991 len 24576
34520 082/300 Read block 3688 len 40960
34521 083/300 Write block 4386 len 32768 data: 5 6 5 6 4 5 4 4
34522 084/300 Read block 1705 len 36864
34523 085/300 Read block 4494 len 20480
34524 086/300 Flush
34525 087/300 Read block 2760 len 4096
34526 088/300 Read block 1619 len 32768
34527 089/300 Read block 3771 len 8192
34528 090/300 Write block 1646 len 28672 data: 6 6 6 5 4 4 4
34529 091/300 Flush
34530 092/300 Read block 2407 len 16384
34531 093/300 Read block 2753 len 36864
34532 094/300 Read block 2737 len 24576
34533 095/300 Write block 1176 len 40960 data: 7 6 6 6 7 7 6 4 3 3
34534 096/300 Read block 423 len 28672
34535 097/300 Flush
34536 098/300 Write block 2784 len 40960 data: 7 7 6 8 6 5 5 4 4 4
34537 099/300 Read block 502 len 8192
34538 100/300 Read block 75 len 40960
34539 101/300 Flush
34540 102/300 Write block 4586 len 28672 data: 8 8 8 7 8 7 8
34541 103/300 Write block 2278 len 12288 data: 11 10 10
34542 104/300 Read block 3611 len 16384
34543 105/300 Flush
34544 106/300 Flush
34545 107/300 Write block 87 len 12288 data: 5 5 6
34546 108/300 Write block 3565 len 40960 data: 5 7 8 6 7 6 7 6 6 5
34547 109/300 Read block 3357 len 24576
34548 110/300 Write block 1039 len 12288 data: 3 3 4
34549 111/300 Write block 3660 len 20480 data: 7 6 7 6 6
34550 112/300 Write block 858 len 4096 data: 5
34551 113/300 Read block 3655 len 28672
34552 114/300 Read block 4555 len 20480
34553 115/300 Read block 4728 len 16384
34554 116/300 Read block 4226 len 4096
34555 117/300 Read block 3141 len 40960
34556 118/300 Flush
34557 119/300 Write block 2019 len 16384 data: 7 7 7 7
34558 120/300 Read block 3683 len 16384
34559 121/300 Read block 3981 len 16384
34560 122/300 Read block 3509 len 40960
34561 123/300 Read block 797 len 32768
34562 124/300 Write block 411 len 32768 data: 5 5 4 4 5 4 4 3
34563 125/300 Write block 2761 len 24576 data: 7 8 7 7 7 5
34564 126/300 Write block 3651 len 32768 data: 4 7 8 7 7 7 8 9
34565 127/300 Write block 3102 len 28672 data: 7 10 8 8 5 6 5
34566 128/300 Read block 2704 len 24576
34567 129/300 Read block 4118 len 12288
34568 130/300 Write block 2475 len 20480 data: 5 5 5 6 6
34569 131/300 Read block 961 len 32768
34570 132/300 Write block 2320 len 28672 data: 5 6 6 6 6 9 12
34571 133/300 Write block 2355 len 12288 data: 5 4 3
34572 134/300 Write block 3872 len 20480 data: 10 12 9 8 6
34573 135/300 Read block 529 len 8192
34574 136/300 Flush
34575 137/300 Write block 1900 len 12288 data: 2 3 6
34576 138/300 Write block 1187 len 4096 data: 7
34577 139/300 Read block 1543 len 12288
34578 140/300 Flush
34579 141/300 Flush
34580 142/300 Write block 1230 len 36864 data: 4 4 5 5 5 4 3 4 5
34581 143/300 Read block 2526 len 36864
34582 144/300 Write block 4835 len 20480 data: 7 8 8 9 7
34583 145/300 Write block 2431 len 40960 data: 5 5 3 3 3 3 4 4 3 3
34584 146/300 Read block 1659 len 36864
34585 147/300 Read block 2855 len 20480
34586 148/300 Read block 80 len 36864
34587 149/300 Read block 827 len 24576
34588 150/300 Write block 1284 len 28672 data: 2 2 2 2 4 5 5
34589 151/300 Read block 162 len 12288
34590 152/300 Read block 850 len 20480
34591 153/300 Read block 4499 len 4096
34592 154/300 Read block 790 len 36864
34593 155/300 Read block 2305 len 24576
34594 156/300 Write block 556 len 4096 data: 5
34595 157/300 Read block 4462 len 24576
34596 158/300 Read block 3934 len 8192
34597 159/300 Write block 2119 len 40960 data: 4 5 7 7 9 6 6 7 7 7
34598 160/300 Read block 649 len 16384
34599 161/300 Read block 3601 len 4096
34600 162/300 Write block 1271 len 4096 data: 4
34601 163/300 Read block 4704 len 28672
34602 164/300 Write block 1912 len 32768 data: 8 8 8 5 6 3 3 3
34603 165/300 Write block 2241 len 24576 data: 6 7 5 4 5 5
34604 166/300 Write block 2715 len 12288 data: 9 10 10
34605 167/300 Read block 1952 len 12288
34606 168/300 Read block 4871 len 20480
34607 169/300 Write block 1059 len 32768 data: 8 8 6 3 4 5 5 5
34608 170/300 Write block 1787 len 40960 data: 4 6 6 6 6 8 6 6 5 5
34609 171/300 Read block 124 len 8192
34610 172/300 Flush
34611 173/300 Read block 4188 len 20480
34612 174/300 Read block 4882 len 28672
34613 175/300 Read block 4816 len 40960
34614 176/300 Write block 1571 len 8192 data: 4 3
34615 177/300 Read block 4123 len 16384
34616 178/300 Read block 3331 len 40960
34617 179/300 Write block 175 len 8192 data: 8 7
34618 180/300 Write block 3930 len 16384 data: 6 5 6 6
34619 181/300 Flush
34620 182/300 Write block 3200 len 12288 data: 6 6 7
34621 183/300 Write block 3435 len 20480 data: 6 8 6 7 7
34622 184/300 Write block 1707 len 36864 data: 4 6 7 6 5 5 5 6 6
34623 185/300 Write block 4756 len 12288 data: 5 6 9
34624 186/300 Write block 1475 len 12288 data: 6 5 6
34625 187/300 Write block 3075 len 32768 data: 6 7 8 9 9 9 9 7
34626 188/300 Write block 4178 len 12288 data: 3 3 2
34627 189/300 Read block 288 len 36864
34628 190/300 Write block 3087 len 20480 data: 8 7 7 6 5
34629 191/300 Read block 4008 len 24576
34630 192/300 Write block 2446 len 8192 data: 5 5
34631 193/300 Write block 4008 len 28672 data: 11 9 9 10 10 10 10
34632 194/300 Flush
34633 195/300 Read block 2463 len 12288
34634 196/300 Read block 874 len 24576
34635 197/300 Read block 1611 len 36864
34636 198/300 Write block 533 len 36864 data: 6 6 6 7 7 8 8 8 6
34637 199/300 Write block 3957 len 24576 data: 8 5 4 5 5 5
34638 200/300 Read block 3582 len 8192
34639 201/300 Write block 3836 len 28672 data: 5 5 5 5 10 11 11
34640 202/300 Write block 1064 len 4096 data: 6
34641 203/300 Write block 2693 len 28672 data: 4 4 5 6 7 6 6
34642 204/300 Write block 4318 len 40960 data: 4 5 5 4 3 3 2 4 3 4
34643 205/300 Read block 2142 len 40960
34644 206/300 Read block 4466 len 40960
34645 207/300 Write block 3338 len 8192 data: 9 8
34646 208/300 Read block 1880 len 8192
34647 209/300 Read block 3988 len 20480
34648 210/300 Read block 1903 len 8192
34649 211/300 Read block 2807 len 40960
34650 212/300 Read block 2085 len 32768
34651 213/300 Write block 1721 len 4096 data: 4
34652 214/300 Write block 4532 len 16384 data: 4 3 3 4
34653 215/300 Flush
34654 216/300 Read block 1043 len 12288
34655 217/300 Write block 3045 len 16384 data: 6 6 6 8
34656 218/300 Read block 1342 len 36864
34657 219/300 Write block 2943 len 24576 data: 5 3 3 3 4 6
34658 220/300 Write block 31 len 24576 data: 7 6 7 6 6 6
34659 221/300 Read block 791 len 28672
34660 222/300 Write block 3873 len 32768 data: 13 10 9 7 7 7 5 4
34661 223/300 Read block 2594 len 24576
34662 224/300 Read block 1855 len 40960
34663 225/300 Flush
34664 226/300 Write block 3300 len 40960 data: 5 4 4 4 4 3 5 4 4 5
34665 227/300 Read block 4258 len 12288
34666 228/300 Write block 3100 len 32768 data: 8 8 8 11 9 9 6 7
34667 229/300 Write block 2305 len 28672 data: 6 6 7 6 6 4 4
34668 230/300 Read block 2029 len 32768
34669 231/300 Write block 4196 len 8192 data: 3 6
34670 232/300 Read block 1599 len 16384
34671 233/300 Write block 134 len 32768 data: 6 6 6 7 6 7 6 7
34672 234/300 Read block 4292 len 24576
34673 235/300 Read block 3011 len 28672
34674 236/300 Read block 4844 len 32768
34675 237/300 Read block 2697 len 12288
34676 238/300 Write block 2482 len 20480 data: 6 5 5 5 6
34677 239/300 Write block 2371 len 4096 data: 5
34678 240/300 Read block 4681 len 24576
34679 241/300 Read block 1973 len 12288
34680 242/300 Write block 2440 len 4096 data: 4
34681 243/300 Read block 2014 len 32768
34682 244/300 Read block 1000 len 4096
34683 245/300 Write block 3401 len 4096 data: 5
34684 246/300 Read block 4180 len 28672
34685 247/300 Read block 941 len 32768
34686 248/300 Read block 3493 len 36864
34687 249/300 Flush
34688 250/300 Write block 842 len 4096 data: 7
34689 251/300 Write block 3750 len 20480 data: 6 7 7 6 7
34690 252/300 Read block 1838 len 16384
34691 253/300 Read block 208 len 32768
34692 254/300 Write block 4427 len 4096 data: 7
34693 255/300 Read block 3122 len 8192
34694 256/300 Write block 3377 len 12288 data: 3 3 3
34695 257/300 Read block 3973 len 16384
34696 258/300 Write block 2736 len 12288 data: 4 4 5
34697 259/300 Read block 626 len 36864
34698 260/300 Write block 43 len 28672 data: 8 7 7 6 4 3 3
34699 261/300 Read block 1457 len 16384
34700 262/300 Write block 4054 len 8192 data: 8 9
34701 263/300 Write block 3218 len 36864 data: 3 5 8 7 7 6 8 7 7
34702 264/300 Read block 2884 len 24576
34703 265/300 Read block 1996 len 8192
34704 266/300 Write block 3369 len 36864 data: 5 3 3 4 4 4 4 3 4
34705 267/300 Write block 1844 len 20480 data: 4 4 6 5 5
34706 268/300 Write block 1277 len 8192 data: 8 8
34707 269/300 Write block 1743 len 40960 data: 5 6 5 6 7 8 7 6 5 7
34708 270/300 Write block 2497 len 28672 data: 5 4 4 5 5 5 5
34709 271/300 Read block 1292 len 36864
34710 272/300 Read block 4580 len 32768
34711 273/300 Read block 2449 len 40960
34712 274/300 Read block 1547 len 4096
34713 275/300 Write block 907 len 4096 data: 6
34714 276/300 Flush
34715 277/300 Read block 2347 len 24576
34716 278/300 Read block 2528 len 4096
34717 279/300 Write block 990 len 40960 data: 5 5 5 5 5 7 7 6 6 6
34718 280/300 Write block 4535 len 16384 data: 5 5 4 4
34719 281/300 Write block 3522 len 4096 data: 3
34720 282/300 Read block 1367 len 20480
34721 283/300 Read block 3303 len 20480
34722 284/300 Read block 4802 len 8192
34723 285/300 Flush
34724 286/300 Write block 3785 len 36864 data: 7 6 7 6 6 5 4 4 3
34725 287/300 Read block 2695 len 28672
34726 288/300 Write block 1091 len 36864 data: 5 6 6 6 7 5 5 5 4
34727 289/300 Write block 4379 len 20480 data: 4 4 4 6 5
34728 290/300 Flush
34729 291/300 Read block 4011 len 24576
34730 292/300 Flush
34731 293/300 Read block 4374 len 4096
34732 294/300 Write block 4583 len 32768 data: 4 4 5 9 9 9 8 9
34733 295/300 Write block 4700 len 20480 data: 7 7 9 8 7
34734 296/300 Write block 3185 len 16384 data: 4 4 6 6
34735 297/300 Read block 1942 len 24576
34736 298/300 Write block 298 len 32768 data: 6 6 7 7 9 7 8 8
34737 299/300 Write block 3571 len 32768 data: 8 7 7 6 4 4 5 4
34738 300/300 Read block 148 len 28672
34739 301/300 Read block 3376 len 24576
34740 Main task has work to do, go find it
34741 got dsc Start(1)
34742 start 1
34743 Feb 21 06:53:06.851 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/1, method: POST, req_id: b9d1cbd2-6d02-4cf4-8a74-ccc3be2a1b2a, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
34744 Replay: started 1, returned:Ok(())
34745 ----------------------------------------------------------------
34746 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34747 States: Active Offline Active
34748 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34749 9754 Acked Read 2 Done Sent Done false
34750 9755 Acked Write 3 Done Sent Done false
34751 9756 Acked Flush 0 Done Sent Done false
34752 9757 Acked Write 8 Done Sent Done false
34753 9758 Acked Write 4 Done Sent Done false
34754 9759 Acked Read 1 Done Sent Done false
34755 9760 Acked Write 1 Done Sent Done false
34756 9761 Acked Read 1 Done Sent Done false
34757 9762 Acked Write 10 Done Sent Done false
34758 9763 Acked Write 1 Done Sent Done false
34759 9764 Acked Read 5 Done Sent Done false
34760 9765 Acked Read 3 Done Sent Done false
34761 9766 Acked Flush 0 Done Sent Done false
34762 9767 Acked Write 9 Done Sent Done false
34763 9768 Acked Write 5 Done Sent Done false
34764 9769 Acked Write 6 Done Sent Done false
34765 9770 Acked Read 10 Done Sent Done false
34766 9771 Acked Read 10 Done Sent Done false
34767 9772 Acked Flush 0 Done Sent Done false
34768 9773 Acked Read 2 Done Sent Done false
34769 9774 Acked Write 2 Done Sent Done false
34770 9775 Acked Write 1 Done Sent Done false
34771 9776 Acked Read 8 Done Sent Done false
34772 9777 Acked Write 3 Done Sent Done false
34773 9778 Acked Write 3 Done Sent Done false
34774 9779 Acked Flush 0 Done Sent Done false
34775 9780 Acked Read 4 Done Sent Done false
34776 9781 Acked Flush 0 Done Sent Done false
34777 9782 Acked Write 6 Done Sent Done false
34778 9783 Acked Read 2 Done Sent Done false
34779 9784 Acked Write 5 Done Sent Done false
34780 9785 Acked Read 5 Done Sent Done false
34781 9786 Acked Write 9 Done Sent Done false
34782 9787 Acked Read 5 Done Sent Done false
34783 9788 Acked Write 4 Done Sent Done false
34784 9789 Acked Read 9 Done Sent Done false
34785 9790 Acked Read 2 Done Sent Done false
34786 9791 Acked Write 6 Done Sent Done false
34787 9792 Acked Write 4 Done Sent Done false
34788 9793 Acked Flush 0 Done Sent Done false
34789 9794 Acked Write 3 Done Sent Done false
34790 9795 Acked Write 9 Done Sent Done false
34791 9796 Acked Write 1 Done Sent Done false
34792 9797 Acked Read 6 Done Sent Done false
34793 9798 Acked Read 3 Done Sent Done false
34794 9799 Acked Write 7 Done Sent Done false
34795 9800 Acked Write 2 Done Sent Done false
34796 9801 Acked Flush 0 Done Sent Done false
34797 9802 Acked Read 4 Done Sent Done false
34798 9803 Acked Write 10 Done Sent Done false
34799 9804 Acked Write 7 Done Sent Done false
34800 9805 Acked Write 8 Done Sent Done false
34801 9806 Acked Write 10 Done Sent Done false
34802 9807 Acked Write 5 Done Sent Done false
34803 9808 Acked Write 1 Done Sent Done false
34804 9809 Acked Read 3 Done Sent Done false
34805 9810 Acked Write 1 Done Sent Done false
34806 9811 Acked Read 5 Done Sent Done false
34807 9812 Acked Write 5 Done Sent Done false
34808 9813 Acked Read 10 Done Sent Done false
34809 9814 Acked Write 7 Done Sent Done false
34810 9815 Acked Flush 0 Done Sent Done false
34811 9816 Acked Write 4 Done Sent Done false
34812 9817 Acked Write 6 Done Sent Done false
34813 9818 Acked Write 8 Done Sent Done false
34814 9819 Acked Read 5 Done Sent Done false
34815 9820 Acked Read 1 Done Sent Done false
34816 9821 Acked Read 10 Done Sent Done false
34817 9822 Acked Write 5 Done Sent Done false
34818 9823 Acked Flush 0 Done Sent Done false
34819 9824 Acked Read 2 Done Sent Done false
34820 9825 Acked Write 7 Done Sent Done false
34821 9826 Acked Flush 0 Done Sent Done false
34822 9827 Acked Write 7 Done Sent Done false
34823 9828 Acked Read 3 Done Sent Done false
34824 9829 Acked Write 1 Done Sent Done false
34825 9830 Acked Read 5 Done Sent Done false
34826 9831 Acked Read 8 Done Sent Done false
34827 9832 Acked Write 1 Done Sent Done false
34828 9833 Acked Read 4 Done Sent Done false
34829 9834 Acked Read 6 Done Sent Done false
34830 9835 Acked Read 10 Done Sent Done false
34831 9836 Acked Write 8 Done Sent Done false
34832 9837 Acked Read 9 Done Sent Done false
34833 9838 Acked Read 5 Done Sent Done false
34834 9839 Acked Flush 0 Done Sent Done false
34835 9840 Acked Read 1 Done Sent Done false
34836 9841 Acked Read 8 Done Sent Done false
34837 9842 Acked Read 2 Done Sent Done false
34838 9843 Acked Write 7 Done Sent Done false
34839 9844 Acked Flush 0 Done Sent Done false
34840 9845 Acked Read 4 Done Sent Done false
34841 9846 Acked Read 9 Done Sent Done false
34842 9847 Acked Read 6 Done Sent Done false
34843 9848 Acked Write 10 Done Sent Done false
34844 9849 Acked Read 7 Done Sent Done false
34845 9850 Acked Flush 0 Done Sent Done false
34846 9851 Acked Write 10 Done Sent Done false
34847 9852 Acked Read 2 Done Sent Done false
34848 9853 Acked Read 10 Done Sent Done false
34849 9854 Acked Flush 0 Done Sent Done false
34850 9855 Acked Write 7 Done Sent Done false
34851 9856 Acked Write 3 Done Sent Done false
34852 9857 Acked Read 4 Done Sent Done false
34853 9858 Acked Flush 0 Done Sent Done false
34854 9859 Acked Flush 0 Done Sent Done false
34855 9860 Acked Write 3 Done Sent Done false
34856 9861 Acked Write 10 Done Sent Done false
34857 9862 Acked Read 6 Done Sent Done false
34858 9863 Acked Write 3 Done Sent Done false
34859 9864 Acked Write 5 Done Sent Done false
34860 9865 Acked Write 1 Done Sent Done false
34861 9866 Acked Read 7 Done Sent Done false
34862 9867 Acked Read 5 Done Sent Done false
34863 9868 Acked Read 4 Done Sent Done false
34864 9869 Acked Read 1 Done Sent Done false
34865 9870 Acked Read 10 Done Sent Done false
34866 9871 Acked Flush 0 Done Sent Done false
34867 9872 Acked Write 4 Done Sent Done false
34868 9873 Acked Read 4 Done Sent Done false
34869 9874 Acked Read 4 Done Sent Done false
34870 9875 Acked Read 10 Done Sent Done false
34871 9876 Acked Read 8 Done Sent Done false
34872 9877 Acked Write 8 Done Sent Done false
34873 9878 Acked Write 6 Done Sent Done false
34874 9879 Acked Write 8 Done Sent Done false
34875 9880 Acked Write 7 Done Sent Done false
34876 9881 Acked Read 6 Done Sent Done false
34877 9882 Acked Read 3 Done Sent Done false
34878 9883 Acked Write 5 Done Sent Done false
34879 9884 Acked Read 8 Done Sent Done false
34880 9885 Acked Write 7 Done Sent Done false
34881 9886 Acked Write 3 Done Sent Done false
34882 9887 Acked Write 5 Done Sent Done false
34883 9888 Acked Read 2 Done Sent Done false
34884 9889 Acked Flush 0 Done Sent Done false
34885 9890 Acked Write 3 Done Sent Done false
34886 9891 Acked Write 1 Done Sent Done false
34887 9892 Acked Read 3 Done Sent Done false
34888 9893 Acked Flush 0 Done Sent Done false
34889 9894 Acked Flush 0 Done Sent Done false
34890 9895 Acked Write 9 Done Sent Done false
34891 9896 Acked Read 9 Done Sent Done false
34892 9897 Acked Write 5 Done Sent Done false
34893 9898 Acked Write 10 Done Sent Done false
34894 9899 Acked Read 9 Done Sent Done false
34895 9900 Acked Read 5 Done Sent Done false
34896 9901 Acked Read 9 Done Sent Done false
34897 9902 Acked Read 6 Done Sent Done false
34898 9903 Acked Write 7 Done Sent Done false
34899 9904 Acked Read 3 Done Sent Done false
34900 9905 Acked Read 5 Done Sent Done false
34901 9906 Acked Read 1 Done Sent Done false
34902 9907 Acked Read 9 Done Sent Done false
34903 9908 Acked Read 6 Done Sent Done false
34904 9909 Acked Write 1 Done Sent Done false
34905 9910 Acked Read 6 Done Sent Done false
34906 9911 Acked Read 2 Done Sent Done false
34907 9912 Acked Write 10 Done Sent Done false
34908 9913 Acked Read 4 Done Sent Done false
34909 9914 Acked Read 1 Done Sent Done false
34910 9915 Acked Write 1 Done Sent Done false
34911 9916 Acked Read 7 Done Sent Done false
34912 9917 Acked Write 8 Done Sent Done false
34913 9918 Acked Write 6 Done Sent Done false
34914 9919 Acked Write 3 Done Sent Done false
34915 9920 Acked Read 3 Done Sent Done false
34916 9921 Acked Read 5 Done Sent Done false
34917 9922 Acked Write 8 Done Sent Done false
34918 9923 Acked Write 10 Done Sent Done false
34919 9924 Acked Read 2 Done Sent Done false
34920 9925 Acked Flush 0 Done Sent Done false
34921 9926 Acked Read 5 Done Sent Done false
34922 9927 Acked Read 7 Done Sent Done false
34923 9928 Acked Read 10 Done Sent Done false
34924 9929 Acked Write 2 Done Sent Done false
34925 9930 Acked Read 4 Done Sent Done false
34926 9931 Acked Read 10 Done Sent Done false
34927 9932 Acked Write 2 Done Sent Done false
34928 9933 Acked Write 4 Done Sent Done false
34929 9934 Acked Flush 0 Done Sent Done false
34930 9935 Acked Write 3 Done Sent Done false
34931 9936 Acked Write 5 Done Sent Done false
34932 9937 Acked Write 9 Done Sent Done false
34933 9938 Acked Write 3 Done Sent Done false
34934 9939 Acked Write 3 Done Sent Done false
34935 9940 Acked Write 8 Done Sent Done false
34936 9941 Acked Write 3 Done Sent Done false
34937 9942 Acked Read 9 Done Sent Done false
34938 9943 Acked Write 5 Done Sent Done false
34939 9944 Acked Read 6 Done Sent Done false
34940 9945 Acked Write 2 Done Sent Done false
34941 9946 Acked Write 7 Done Sent Done false
34942 9947 Acked Flush 0 Done Sent Done false
34943 9948 Acked Read 3 Done Sent Done false
34944 9949 Acked Read 6 Done Sent Done false
34945 9950 Acked Read 9 Done Sent Done false
34946 9951 Acked Write 9 Done Sent Done false
34947 9952 Acked Write 6 Done Sent Done false
34948 9953 Acked Read 2 Done Sent Done false
34949 9954 Acked Write 7 Done Sent Done false
34950 9955 Acked Write 1 Done Sent Done false
34951 9956 Acked Write 7 Done Sent Done false
34952 9957 Acked Write 10 Done Sent Done false
34953 9958 Acked Read 10 Done Sent Done false
34954 9959 Acked Read 10 Done Sent Done false
34955 9960 Acked Write 2 Done Sent Done false
34956 9961 Acked Read 2 Done Sent Done false
34957 9962 Acked Read 5 Done Sent Done false
34958 9963 Acked Read 2 Done Sent Done false
34959 9964 Acked Read 10 Done Sent Done false
34960 9965 Acked Read 8 Done Sent Done false
34961 9966 Acked Write 1 Done Sent Done false
34962 9967 Acked Write 4 Done Sent Done false
34963 9968 Acked Flush 0 Done Sent Done false
34964 9969 Acked Read 3 Done Sent Done false
34965 9970 Acked Write 4 Done Sent Done false
34966 9971 Acked Read 9 Done Sent Done false
34967 9972 Acked Write 6 Done Sent Done false
34968 9973 Acked Write 6 Done Sent Done false
34969 9974 Acked Read 7 Done Sent Done false
34970 9975 Acked Write 8 Done Sent Done false
34971 9976 Acked Read 6 Done Sent Done false
34972 9977 Acked Read 10 Done Sent Done false
34973 9978 Acked Flush 0 Done Sent Done false
34974 9979 Acked Write 10 Done Sent Done false
34975 9980 Acked Read 3 Done Sent Done false
34976 9981 Acked Write 8 Done Sent Done false
34977 9982 Acked Write 7 Done Sent Done false
34978 9983 Acked Read 8 Done Sent Done false
34979 9984 Acked Write 2 Done Sent Done false
34980 9985 Acked Read 4 Done Sent Done false
34981 9986 Acked Write 8 Done Sent Done false
34982 9987 Acked Read 6 Done Sent Done false
34983 9988 Acked Read 7 Done Sent Done false
34984 9989 Acked Read 8 Done Sent Done false
34985 9990 Acked Read 3 Done Sent Done false
34986 9991 Acked Write 5 Done Sent Done false
34987 9992 Acked Write 1 Done Sent Done false
34988 9993 Acked Read 6 Done Sent Done false
34989 9994 Acked Read 3 Done Sent Done false
34990 9995 Acked Write 1 Done Sent Done false
34991 9996 Acked Read 8 Done Sent Done false
34992 9997 Acked Read 1 Done Sent Done false
34993 9998 Acked Write 1 Done Sent Done false
34994 9999 Acked Read 7 Done Sent Done false
34995 10000 Acked Read 8 Done Sent Done false
34996 10001 Acked Read 9 Done Sent Done false
34997 10002 Acked Flush 0 Done Sent Done false
34998 10003 Acked Write 1 Done Sent Done false
34999 10004 Acked Write 5 Done Sent Done false
35000 10005 Acked Read 4 Done Sent Done false
35001 10006 Acked Read 8 Done Sent Done false
35002 10007 Acked Write 1 Done Sent Done false
35003 10008 Acked Read 2 Done Sent Done false
35004 10009 Acked Write 3 Done Sent Done false
35005 10010 Acked Read 4 Done Sent Done false
35006 10011 Acked Write 3 Done Sent Done false
35007 10012 Acked Read 9 Done Sent Done false
35008 10013 Acked Write 7 Done Sent Done false
35009 10014 Acked Read 4 Done Sent Done false
35010 10015 Acked Write 2 Done Sent Done false
35011 10016 Acked Write 9 Done Sent Done false
35012 10017 Acked Read 6 Done Sent Done false
35013 10018 Acked Read 2 Done Sent Done false
35014 10019 Acked Write 9 Done Sent Done false
35015 10020 Acked Write 5 Done Sent Done false
35016 10021 Acked Write 2 Done Sent Done false
35017 10022 Acked Write 10 Done Sent Done false
35018 10023 Acked Write 7 Done Sent Done false
35019 10024 Acked Read 9 Done Sent Done false
35020 10025 Acked Read 8 Done Sent Done false
35021 10026 Acked Read 10 Done Sent Done false
35022 10027 Acked Read 1 Done Sent Done false
35023 10028 Acked Write 1 Done Sent Done false
35024 10029 Acked Flush 0 Done Sent Done false
35025 10030 Acked Read 6 Done Sent Done false
35026 10031 Acked Read 1 Done Sent Done false
35027 10032 Acked Write 10 Done Sent Done false
35028 10033 Acked Write 4 Done Sent Done false
35029 10034 Acked Write 1 Done Sent Done false
35030 10035 Acked Read 5 Done Sent Done false
35031 10036 Acked Read 5 Done Sent Done false
35032 10037 Acked Read 2 Done Sent Done false
35033 10038 Acked Flush 0 Done Sent Done false
35034 10039 Acked Write 9 Done Sent Done false
35035 10040 Acked Read 7 Done Sent Done false
35036 10041 Acked Write 9 Done Sent Done false
35037 10042 Acked Write 5 Done Sent Done false
35038 10043 Acked Flush 0 Done Sent Done false
35039 10044 Acked Read 6 Done Sent Done false
35040 10045 Acked Flush 0 Done Sent Done false
35041 10046 Acked Read 1 Done Sent Done false
35042 10047 Acked Write 8 Done Sent Done false
35043 10048 Acked Write 5 Done Sent Done false
35044 10049 Acked Write 4 Done Sent Done false
35045 10050 Acked Read 6 Done Sent Done false
35046 10051 Acked Write 8 Done Sent Done false
35047 10052 Acked Write 8 Done Sent Done false
35048 10053 Acked Read 7 Done Sent Done false
35049 10054 Acked Read 6 Done Sent Done false
35050 STATES DS:0 DS:1 DS:2 TOTAL
35051 Sent 0 301 0 301
35052 Done 301 0 301 602
35053 Skipped 0 0 0 0
35054 Error 0 0 0 0
35055 Last Flush: Some(JobId(10045)) Some(JobId(9753)) Some(JobId(10045))
35056 Downstairs last five completed: 9753 9752 9751 9750 9749
35057 Upstairs last five completed: 10054 10053 10052 10051 10050
35058 CLIENT: Up:0 ds:301 act:2
350592025-02-21T06:53:07.368ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
350602025-02-21T06:53:08.370ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
35061 [8820] got start action
35062 [8820] I am going to restart
35063 Starting downstairs at port 8820
35064 Make output file at "/tmp/dsc/downstairs-8820.txt"
35065 [8820][1] reports Starting
35066 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1236)
35067 [8820][1] reports Running
350682025-02-21T06:53:09.372ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
350692025-02-21T06:53:10.374ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
35070 ----------------------------------------------------------------
35071 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35072 States: Active Offline Active
35073 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35074 9754 Acked Read 2 Done Sent Done false
35075 9755 Acked Write 3 Done Sent Done false
35076 9756 Acked Flush 0 Done Sent Done false
35077 9757 Acked Write 8 Done Sent Done false
35078 9758 Acked Write 4 Done Sent Done false
35079 9759 Acked Read 1 Done Sent Done false
35080 9760 Acked Write 1 Done Sent Done false
35081 9761 Acked Read 1 Done Sent Done false
35082 9762 Acked Write 10 Done Sent Done false
35083 9763 Acked Write 1 Done Sent Done false
35084 9764 Acked Read 5 Done Sent Done false
35085 9765 Acked Read 3 Done Sent Done false
35086 9766 Acked Flush 0 Done Sent Done false
35087 9767 Acked Write 9 Done Sent Done false
35088 9768 Acked Write 5 Done Sent Done false
35089 9769 Acked Write 6 Done Sent Done false
35090 9770 Acked Read 10 Done Sent Done false
35091 9771 Acked Read 10 Done Sent Done false
35092 9772 Acked Flush 0 Done Sent Done false
35093 9773 Acked Read 2 Done Sent Done false
35094 9774 Acked Write 2 Done Sent Done false
35095 9775 Acked Write 1 Done Sent Done false
35096 9776 Acked Read 8 Done Sent Done false
35097 9777 Acked Write 3 Done Sent Done false
35098 9778 Acked Write 3 Done Sent Done false
35099 9779 Acked Flush 0 Done Sent Done false
35100 9780 Acked Read 4 Done Sent Done false
35101 9781 Acked Flush 0 Done Sent Done false
35102 9782 Acked Write 6 Done Sent Done false
35103 9783 Acked Read 2 Done Sent Done false
35104 9784 Acked Write 5 Done Sent Done false
35105 9785 Acked Read 5 Done Sent Done false
35106 9786 Acked Write 9 Done Sent Done false
35107 9787 Acked Read 5 Done Sent Done false
35108 9788 Acked Write 4 Done Sent Done false
35109 9789 Acked Read 9 Done Sent Done false
35110 9790 Acked Read 2 Done Sent Done false
35111 9791 Acked Write 6 Done Sent Done false
35112 9792 Acked Write 4 Done Sent Done false
35113 9793 Acked Flush 0 Done Sent Done false
35114 9794 Acked Write 3 Done Sent Done false
35115 9795 Acked Write 9 Done Sent Done false
35116 9796 Acked Write 1 Done Sent Done false
35117 9797 Acked Read 6 Done Sent Done false
35118 9798 Acked Read 3 Done Sent Done false
35119 9799 Acked Write 7 Done Sent Done false
35120 9800 Acked Write 2 Done Sent Done false
35121 9801 Acked Flush 0 Done Sent Done false
35122 9802 Acked Read 4 Done Sent Done false
35123 9803 Acked Write 10 Done Sent Done false
35124 9804 Acked Write 7 Done Sent Done false
35125 9805 Acked Write 8 Done Sent Done false
35126 9806 Acked Write 10 Done Sent Done false
35127 9807 Acked Write 5 Done Sent Done false
35128 9808 Acked Write 1 Done Sent Done false
35129 9809 Acked Read 3 Done Sent Done false
35130 9810 Acked Write 1 Done Sent Done false
35131 9811 Acked Read 5 Done Sent Done false
35132 9812 Acked Write 5 Done Sent Done false
35133 9813 Acked Read 10 Done Sent Done false
35134 9814 Acked Write 7 Done Sent Done false
35135 9815 Acked Flush 0 Done Sent Done false
35136 9816 Acked Write 4 Done Sent Done false
35137 9817 Acked Write 6 Done Sent Done false
35138 9818 Acked Write 8 Done Sent Done false
35139 9819 Acked Read 5 Done Sent Done false
35140 9820 Acked Read 1 Done Sent Done false
35141 9821 Acked Read 10 Done Sent Done false
35142 9822 Acked Write 5 Done Sent Done false
35143 9823 Acked Flush 0 Done Sent Done false
35144 9824 Acked Read 2 Done Sent Done false
35145 9825 Acked Write 7 Done Sent Done false
35146 9826 Acked Flush 0 Done Sent Done false
35147 9827 Acked Write 7 Done Sent Done false
35148 9828 Acked Read 3 Done Sent Done false
35149 9829 Acked Write 1 Done Sent Done false
35150 9830 Acked Read 5 Done Sent Done false
35151 9831 Acked Read 8 Done Sent Done false
35152 9832 Acked Write 1 Done Sent Done false
35153 9833 Acked Read 4 Done Sent Done false
35154 9834 Acked Read 6 Done Sent Done false
35155 9835 Acked Read 10 Done Sent Done false
35156 9836 Acked Write 8 Done Sent Done false
35157 9837 Acked Read 9 Done Sent Done false
35158 9838 Acked Read 5 Done Sent Done false
35159 9839 Acked Flush 0 Done Sent Done false
35160 9840 Acked Read 1 Done Sent Done false
35161 9841 Acked Read 8 Done Sent Done false
35162 9842 Acked Read 2 Done Sent Done false
35163 9843 Acked Write 7 Done Sent Done false
35164 9844 Acked Flush 0 Done Sent Done false
35165 9845 Acked Read 4 Done Sent Done false
35166 9846 Acked Read 9 Done Sent Done false
35167 9847 Acked Read 6 Done Sent Done false
35168 9848 Acked Write 10 Done Sent Done false
35169 9849 Acked Read 7 Done Sent Done false
35170 9850 Acked Flush 0 Done Sent Done false
35171 9851 Acked Write 10 Done Sent Done false
35172 9852 Acked Read 2 Done Sent Done false
35173 9853 Acked Read 10 Done Sent Done false
35174 9854 Acked Flush 0 Done Sent Done false
35175 9855 Acked Write 7 Done Sent Done false
35176 9856 Acked Write 3 Done Sent Done false
35177 9857 Acked Read 4 Done Sent Done false
35178 9858 Acked Flush 0 Done Sent Done false
35179 9859 Acked Flush 0 Done Sent Done false
35180 9860 Acked Write 3 Done Sent Done false
35181 9861 Acked Write 10 Done Sent Done false
35182 9862 Acked Read 6 Done Sent Done false
35183 9863 Acked Write 3 Done Sent Done false
35184 9864 Acked Write 5 Done Sent Done false
35185 9865 Acked Write 1 Done Sent Done false
35186 9866 Acked Read 7 Done Sent Done false
35187 9867 Acked Read 5 Done Sent Done false
35188 9868 Acked Read 4 Done Sent Done false
35189 9869 Acked Read 1 Done Sent Done false
35190 9870 Acked Read 10 Done Sent Done false
35191 9871 Acked Flush 0 Done Sent Done false
35192 9872 Acked Write 4 Done Sent Done false
35193 9873 Acked Read 4 Done Sent Done false
35194 9874 Acked Read 4 Done Sent Done false
35195 9875 Acked Read 10 Done Sent Done false
35196 9876 Acked Read 8 Done Sent Done false
35197 9877 Acked Write 8 Done Sent Done false
35198 9878 Acked Write 6 Done Sent Done false
35199 9879 Acked Write 8 Done Sent Done false
35200 9880 Acked Write 7 Done Sent Done false
35201 9881 Acked Read 6 Done Sent Done false
35202 9882 Acked Read 3 Done Sent Done false
35203 9883 Acked Write 5 Done Sent Done false
35204 9884 Acked Read 8 Done Sent Done false
35205 9885 Acked Write 7 Done Sent Done false
35206 9886 Acked Write 3 Done Sent Done false
35207 9887 Acked Write 5 Done Sent Done false
35208 9888 Acked Read 2 Done Sent Done false
35209 9889 Acked Flush 0 Done Sent Done false
35210 9890 Acked Write 3 Done Sent Done false
35211 9891 Acked Write 1 Done Sent Done false
35212 9892 Acked Read 3 Done Sent Done false
35213 9893 Acked Flush 0 Done Sent Done false
35214 9894 Acked Flush 0 Done Sent Done false
35215 9895 Acked Write 9 Done Sent Done false
35216 9896 Acked Read 9 Done Sent Done false
35217 9897 Acked Write 5 Done Sent Done false
35218 9898 Acked Write 10 Done Sent Done false
35219 9899 Acked Read 9 Done Sent Done false
35220 9900 Acked Read 5 Done Sent Done false
35221 9901 Acked Read 9 Done Sent Done false
35222 9902 Acked Read 6 Done Sent Done false
35223 9903 Acked Write 7 Done Sent Done false
35224 9904 Acked Read 3 Done Sent Done false
35225 9905 Acked Read 5 Done Sent Done false
35226 9906 Acked Read 1 Done Sent Done false
35227 9907 Acked Read 9 Done Sent Done false
35228 9908 Acked Read 6 Done Sent Done false
35229 9909 Acked Write 1 Done Sent Done false
35230 9910 Acked Read 6 Done Sent Done false
35231 9911 Acked Read 2 Done Sent Done false
35232 9912 Acked Write 10 Done Sent Done false
35233 9913 Acked Read 4 Done Sent Done false
35234 9914 Acked Read 1 Done Sent Done false
35235 9915 Acked Write 1 Done Sent Done false
35236 9916 Acked Read 7 Done Sent Done false
35237 9917 Acked Write 8 Done Sent Done false
35238 9918 Acked Write 6 Done Sent Done false
35239 9919 Acked Write 3 Done Sent Done false
35240 9920 Acked Read 3 Done Sent Done false
35241 9921 Acked Read 5 Done Sent Done false
35242 9922 Acked Write 8 Done Sent Done false
35243 9923 Acked Write 10 Done Sent Done false
35244 9924 Acked Read 2 Done Sent Done false
35245 9925 Acked Flush 0 Done Sent Done false
35246 9926 Acked Read 5 Done Sent Done false
35247 9927 Acked Read 7 Done Sent Done false
35248 9928 Acked Read 10 Done Sent Done false
35249 9929 Acked Write 2 Done Sent Done false
35250 9930 Acked Read 4 Done Sent Done false
35251 9931 Acked Read 10 Done Sent Done false
35252 9932 Acked Write 2 Done Sent Done false
35253 9933 Acked Write 4 Done Sent Done false
35254 9934 Acked Flush 0 Done Sent Done false
35255 9935 Acked Write 3 Done Sent Done false
35256 9936 Acked Write 5 Done Sent Done false
35257 9937 Acked Write 9 Done Sent Done false
35258 9938 Acked Write 3 Done Sent Done false
35259 9939 Acked Write 3 Done Sent Done false
35260 9940 Acked Write 8 Done Sent Done false
35261 9941 Acked Write 3 Done Sent Done false
35262 9942 Acked Read 9 Done Sent Done false
35263 9943 Acked Write 5 Done Sent Done false
35264 9944 Acked Read 6 Done Sent Done false
35265 9945 Acked Write 2 Done Sent Done false
35266 9946 Acked Write 7 Done Sent Done false
35267 9947 Acked Flush 0 Done Sent Done false
35268 9948 Acked Read 3 Done Sent Done false
35269 9949 Acked Read 6 Done Sent Done false
35270 9950 Acked Read 9 Done Sent Done false
35271 9951 Acked Write 9 Done Sent Done false
35272 9952 Acked Write 6 Done Sent Done false
35273 9953 Acked Read 2 Done Sent Done false
35274 9954 Acked Write 7 Done Sent Done false
35275 9955 Acked Write 1 Done Sent Done false
35276 9956 Acked Write 7 Done Sent Done false
35277 9957 Acked Write 10 Done Sent Done false
35278 9958 Acked Read 10 Done Sent Done false
35279 9959 Acked Read 10 Done Sent Done false
35280 9960 Acked Write 2 Done Sent Done false
35281 9961 Acked Read 2 Done Sent Done false
35282 9962 Acked Read 5 Done Sent Done false
35283 9963 Acked Read 2 Done Sent Done false
35284 9964 Acked Read 10 Done Sent Done false
35285 9965 Acked Read 8 Done Sent Done false
35286 9966 Acked Write 1 Done Sent Done false
35287 9967 Acked Write 4 Done Sent Done false
35288 9968 Acked Flush 0 Done Sent Done false
35289 9969 Acked Read 3 Done Sent Done false
35290 9970 Acked Write 4 Done Sent Done false
35291 9971 Acked Read 9 Done Sent Done false
35292 9972 Acked Write 6 Done Sent Done false
35293 9973 Acked Write 6 Done Sent Done false
35294 9974 Acked Read 7 Done Sent Done false
35295 9975 Acked Write 8 Done Sent Done false
35296 9976 Acked Read 6 Done Sent Done false
35297 9977 Acked Read 10 Done Sent Done false
35298 9978 Acked Flush 0 Done Sent Done false
35299 9979 Acked Write 10 Done Sent Done false
35300 9980 Acked Read 3 Done Sent Done false
35301 9981 Acked Write 8 Done Sent Done false
35302 9982 Acked Write 7 Done Sent Done false
35303 9983 Acked Read 8 Done Sent Done false
35304 9984 Acked Write 2 Done Sent Done false
35305 9985 Acked Read 4 Done Sent Done false
35306 9986 Acked Write 8 Done Sent Done false
35307 9987 Acked Read 6 Done Sent Done false
35308 9988 Acked Read 7 Done Sent Done false
35309 9989 Acked Read 8 Done Sent Done false
35310 9990 Acked Read 3 Done Sent Done false
35311 9991 Acked Write 5 Done Sent Done false
35312 9992 Acked Write 1 Done Sent Done false
35313 9993 Acked Read 6 Done Sent Done false
35314 9994 Acked Read 3 Done Sent Done false
35315 9995 Acked Write 1 Done Sent Done false
35316 9996 Acked Read 8 Done Sent Done false
35317 9997 Acked Read 1 Done Sent Done false
35318 9998 Acked Write 1 Done Sent Done false
35319 9999 Acked Read 7 Done Sent Done false
35320 10000 Acked Read 8 Done Sent Done false
35321 10001 Acked Read 9 Done Sent Done false
35322 10002 Acked Flush 0 Done Sent Done false
35323 10003 Acked Write 1 Done Sent Done false
35324 10004 Acked Write 5 Done Sent Done false
35325 10005 Acked Read 4 Done Sent Done false
35326 10006 Acked Read 8 Done Sent Done false
35327 10007 Acked Write 1 Done Sent Done false
35328 10008 Acked Read 2 Done Sent Done false
35329 10009 Acked Write 3 Done Sent Done false
35330 10010 Acked Read 4 Done Sent Done false
35331 10011 Acked Write 3 Done Sent Done false
35332 10012 Acked Read 9 Done Sent Done false
35333 10013 Acked Write 7 Done Sent Done false
35334 10014 Acked Read 4 Done Sent Done false
35335 10015 Acked Write 2 Done Sent Done false
35336 10016 Acked Write 9 Done Sent Done false
35337 10017 Acked Read 6 Done Sent Done false
35338 10018 Acked Read 2 Done Sent Done false
35339 10019 Acked Write 9 Done Sent Done false
35340 10020 Acked Write 5 Done Sent Done false
35341 10021 Acked Write 2 Done Sent Done false
35342 10022 Acked Write 10 Done Sent Done false
35343 10023 Acked Write 7 Done Sent Done false
35344 10024 Acked Read 9 Done Sent Done false
35345 10025 Acked Read 8 Done Sent Done false
35346 10026 Acked Read 10 Done Sent Done false
35347 10027 Acked Read 1 Done Sent Done false
35348 10028 Acked Write 1 Done Sent Done false
35349 10029 Acked Flush 0 Done Sent Done false
35350 10030 Acked Read 6 Done Sent Done false
35351 10031 Acked Read 1 Done Sent Done false
35352 10032 Acked Write 10 Done Sent Done false
35353 10033 Acked Write 4 Done Sent Done false
35354 10034 Acked Write 1 Done Sent Done false
35355 10035 Acked Read 5 Done Sent Done false
35356 10036 Acked Read 5 Done Sent Done false
35357 10037 Acked Read 2 Done Sent Done false
35358 10038 Acked Flush 0 Done Sent Done false
35359 10039 Acked Write 9 Done Sent Done false
35360 10040 Acked Read 7 Done Sent Done false
35361 10041 Acked Write 9 Done Sent Done false
35362 10042 Acked Write 5 Done Sent Done false
35363 10043 Acked Flush 0 Done Sent Done false
35364 10044 Acked Read 6 Done Sent Done false
35365 10045 Acked Flush 0 Done Sent Done false
35366 10046 Acked Read 1 Done Sent Done false
35367 10047 Acked Write 8 Done Sent Done false
35368 10048 Acked Write 5 Done Sent Done false
35369 10049 Acked Write 4 Done Sent Done false
35370 10050 Acked Read 6 Done Sent Done false
35371 10051 Acked Write 8 Done Sent Done false
35372 10052 Acked Write 8 Done Sent Done false
35373 10053 Acked Read 7 Done Sent Done false
35374 10054 Acked Read 6 Done Sent Done false
35375 STATES DS:0 DS:1 DS:2 TOTAL
35376 Sent 0 301 0 301
35377 Done 301 0 301 602
35378 Skipped 0 0 0 0
35379 Error 0 0 0 0
35380 Last Flush: Some(JobId(10045)) Some(JobId(9753)) Some(JobId(10045))
35381 Downstairs last five completed: 9753 9752 9751 9750 9749
35382 Upstairs last five completed: 10054 10053 10052 10051 10050
35383 CLIENT: Up:0 ds:301 act:2
353842025-02-21T06:53:12.376ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
353852025-02-21T06:53:12.620ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
353862025-02-21T06:53:12.620ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50943) = io task client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
353872025-02-21T06:53:12.621ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
353882025-02-21T06:53:12.621ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
353892025-02-21T06:53:12.621ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
353902025-02-21T06:53:12.621ZINFOcrucible: send last flush ID to this DS: Some(JobId(9753)) = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
353912025-02-21T06:53:12.621ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
353922025-02-21T06:53:12.621ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(9753)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
353932025-02-21T06:53:12.622ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(9753)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
35394 ----------------------------------------------------------------
35395 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35396 Downstairs last five completed: 10055 10054 10053 10052 10051
35397 Upstairs last five completed: 10055 10054 10053 10052 10051
35398 CLIENT: Up:0 ds:0 act:3
35399 Replay: All jobs finished
35400 Main task has work to do, go find it
35401 got dsc Stop(2)
35402 stop 2
35403 [8830] Got stop action so:false kr:false
35404 Feb 21 06:53:14.857 INFO request completed, latency_us: 385, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 82bd5c0a-f867-424c-bd80-e4a7ec6b3696, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
354052025-02-21T06:53:14.858ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
354062025-02-21T06:53:14.858ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
354072025-02-21T06:53:14.858ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
35408 Feb 21 06:53:14.858 INFO request completed, latency_us: 167, response_code: 200, uri: /state/cid/2, method: GET, req_id: 34b09b79-6a38-4eb0-9f1d-61773871baec, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
354092025-02-21T06:53:14.858ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
35410 [8830] Exited with: ExitStatus(unix_wait_status(9))
35411 [8830][2] reports Exit
35412 Feb 21 06:53:18.860 INFO request completed, latency_us: 196, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0fff6675-4f65-45a7-bf40-058472399320, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
35413 001/300 Flush
35414 002/300 Write block 1441 len 16384 data: 10 8 9 9
35415 003/300 Read block 2482 len 28672
35416 004/300 Read block 4823 len 24576
35417 005/300 Write block 3306 len 28672 data: 6 5 5 6 4 5 5
35418 006/300 Read block 2505 len 12288
35419 007/300 Read block 575 len 16384
35420 008/300 Write block 560 len 24576 data: 3 3 4 5 5 5
35421 009/300 Flush
35422 010/300 Read block 3302 len 24576
35423 011/300 Flush
35424 012/300 Flush
35425 013/300 Read block 2169 len 16384
35426 014/300 Write block 3208 len 40960 data: 6 5 5 4 4 5 5 4 4 4
35427 015/300 Write block 634 len 4096 data: 4
35428 016/300 Read block 2473 len 36864
35429 017/300 Read block 3557 len 32768
35430 018/300 Read block 3213 len 28672
35431 019/300 Read block 1685 len 4096
35432 020/300 Read block 3625 len 24576
35433 021/300 Flush
35434 022/300 Flush
35435 023/300 Write block 4501 len 40960 data: 4 2 4 4 4 4 4 5 5 5
35436 024/300 Read block 2419 len 32768
35437 025/300 Flush
35438 026/300 Read block 1485 len 8192
35439 027/300 Write block 4139 len 20480 data: 5 4 4 5 5
35440 028/300 Read block 3030 len 16384
35441 029/300 Write block 2745 len 12288 data: 8 9 10
35442 030/300 Read block 1773 len 24576
35443 031/300 Write block 4439 len 8192 data: 4 4
35444 032/300 Write block 1239 len 4096 data: 4
35445 033/300 Write block 801 len 32768 data: 2 2 2 4 5 6 5 5
35446 034/300 Read block 3122 len 24576
35447 035/300 Write block 1738 len 12288 data: 4 4 4
35448 036/300 Read block 2452 len 16384
35449 037/300 Write block 311 len 4096 data: 5
35450 038/300 Flush
35451 039/300 Write block 4830 len 40960 data: 8 8 7 8 8 8 9 9 10 8
35452 040/300 Read block 987 len 4096
35453 041/300 Flush
35454 042/300 Flush
35455 043/300 Read block 3460 len 8192
35456 044/300 Write block 1788 len 40960 data: 7 7 7 7 9 7 7 6 6 5
35457 045/300 Read block 1976 len 28672
35458 046/300 Flush
35459 047/300 Read block 2232 len 32768
35460 048/300 Write block 1082 len 40960 data: 4 5 6 5 6 8 7 5 5 6
35461 049/300 Flush
35462 050/300 Read block 2974 len 16384
35463 051/300 Write block 2733 len 4096 data: 3
35464 052/300 Read block 1073 len 36864
35465 053/300 Read block 2060 len 16384
35466 054/300 Write block 3538 len 24576 data: 4 5 7 6 6 6
35467 055/300 Read block 3199 len 20480
35468 056/300 Read block 3546 len 32768
35469 057/300 Write block 4784 len 16384 data: 5 7 7 6
35470 058/300 Write block 3367 len 8192 data: 6 5
35471 059/300 Flush
35472 060/300 Read block 4353 len 12288
35473 061/300 Write block 3781 len 24576 data: 5 6 5 5 8 7
35474 062/300 Write block 1170 len 32768 data: 10 9 9 9 9 8 8 7
35475 063/300 Write block 1070 len 20480 data: 9 9 9 9 9
35476 064/300 Write block 2702 len 8192 data: 7 6
35477 065/300 Write block 3217 len 36864 data: 5 4 6 9 8 8 7 9 8
35478 066/300 Flush
35479 067/300 Flush
35480 068/300 Read block 1597 len 8192
35481 069/300 Write block 2669 len 40960 data: 9 9 8 8 7 7 7 8 8 11
35482 070/300 Write block 3449 len 8192 data: 4 4
35483 071/300 Flush
35484 072/300 Read block 407 len 24576
35485 073/300 Read block 4874 len 24576
35486 074/300 Read block 4443 len 40960
35487 075/300 Flush
35488 076/300 Read block 4813 len 16384
35489 077/300 Write block 4218 len 36864 data: 5 5 5 5 4 5 4 5 5
35490 078/300 Write block 1686 len 12288 data: 9 7 7
35491 079/300 Write block 1577 len 16384 data: 6 6 6 5
35492 080/300 Write block 1394 len 12288 data: 6 5 6
35493 081/300 Read block 1063 len 8192
35494 082/300 Write block 1573 len 24576 data: 4 4 4 5 7 7
35495 083/300 Read block 1705 len 36864
35496 084/300 Read block 2911 len 28672
35497 085/300 Read block 2478 len 40960
35498 086/300 Read block 3439 len 12288
35499 087/300 Read block 574 len 28672
35500 088/300 Write block 1906 len 20480 data: 4 5 8 8 7
35501 089/300 Write block 4096 len 12288 data: 5 6 6
35502 090/300 Write block 1237 len 36864 data: 5 6 5 5 5 5 5 5 7
35503 091/300 Read block 4973 len 24576
35504 092/300 Write block 3100 len 24576 data: 9 9 9 12 10 10
35505 093/300 Write block 4515 len 24576 data: 3 4 4 4 7 6
35506 094/300 Flush
35507 095/300 Write block 3063 len 28672 data: 9 9 9 9 7 5 5
35508 096/300 Read block 4147 len 32768
35509 097/300 Write block 918 len 32768 data: 5 4 3 3 6 5 6 7
35510 098/300 Flush
35511 099/300 Write block 3185 len 12288 data: 5 5 7
35512 100/300 Write block 1265 len 40960 data: 7 4 4 3 3 4 5 6 7 8
35513 101/300 Read block 4337 len 36864
35514 102/300 Write block 3224 len 28672 data: 10 9 8 6 7 6 6
35515 103/300 Write block 1596 len 4096 data: 3
35516 104/300 Write block 2034 len 24576 data: 7 8 11 8 10 7
35517 105/300 Write block 4091 len 36864 data: 3 4 5 5 5 6 7 7 7
35518 106/300 Read block 1144 len 20480
35519 107/300 Read block 4378 len 40960
35520 108/300 Write block 3059 len 40960 data: 7 8 9 10 10 10 10 10 8 6
35521 109/300 Write block 3574 len 28672 data: 7 5 5 6 5 6 6
35522 110/300 Read block 1094 len 28672
35523 111/300 Write block 305 len 28672 data: 9 7 4 4 4 3 6
35524 112/300 Read block 4675 len 16384
35525 113/300 Read block 2106 len 12288
35526 114/300 Read block 2542 len 12288
35527 115/300 Read block 168 len 4096
35528 116/300 Write block 3559 len 36864 data: 4 3 3 3 4 5 6 8 9
35529 117/300 Read block 1393 len 8192
35530 118/300 Read block 4073 len 4096
35531 119/300 Write block 3532 len 20480 data: 4 3 4 4 4
35532 120/300 Read block 1494 len 16384
35533 121/300 Read block 2570 len 8192
35534 122/300 Read block 1985 len 24576
35535 123/300 Write block 1053 len 40960 data: 9 10 10 10 9 10 9 9 7 4
35536 124/300 Write block 3960 len 36864 data: 6 6 6 5 6 5 6 6 7
35537 125/300 Write block 1502 len 36864 data: 9 7 6 6 8 7 7 7 5
35538 126/300 Read block 3818 len 24576
35539 127/300 Read block 3645 len 36864
35540 128/300 Read block 1328 len 8192
35541 129/300 Read block 248 len 20480
35542 130/300 Read block 3724 len 24576
35543 131/300 Read block 3888 len 36864
35544 132/300 Flush
35545 133/300 Read block 3098 len 16384
35546 134/300 Read block 956 len 12288
35547 135/300 Read block 4435 len 12288
35548 136/300 Flush
35549 137/300 Read block 3672 len 28672
35550 138/300 Write block 3799 len 8192 data: 10 11
35551 139/300 Read block 4056 len 40960
35552 140/300 Read block 2942 len 8192
35553 141/300 Read block 2562 len 20480
35554 142/300 Read block 1426 len 16384
35555 143/300 Read block 3772 len 32768
35556 144/300 Read block 4379 len 24576
35557 145/300 Write block 2554 len 12288 data: 4 4 4
35558 146/300 Read block 4982 len 16384
35559 147/300 Write block 4771 len 28672 data: 4 4 6 6 5 6 6
35560 148/300 Flush
35561 149/300 Read block 3637 len 12288
35562 150/300 Read block 2091 len 20480
35563 151/300 Read block 2961 len 16384
35564 152/300 Write block 2959 len 12288 data: 5 5 5
35565 153/300 Read block 3155 len 16384
35566 154/300 Write block 4907 len 32768 data: 6 6 5 5 4 3 6 6
35567 155/300 Read block 3672 len 16384
35568 156/300 Read block 3216 len 8192
35569 157/300 Write block 2236 len 4096 data: 8
35570 158/300 Write block 2568 len 8192 data: 3 3
35571 159/300 Flush
35572 160/300 Write block 4803 len 4096 data: 6
35573 161/300 Read block 4821 len 20480
35574 162/300 Write block 2578 len 36864 data: 6 6 5 3 4 4 5 5 5
35575 163/300 Read block 1760 len 24576
35576 164/300 Write block 4262 len 4096 data: 4
35577 165/300 Read block 4701 len 20480
35578 166/300 Read block 1150 len 16384
35579 167/300 Read block 582 len 20480
35580 168/300 Read block 2242 len 4096
35581 169/300 Read block 465 len 32768
35582 170/300 Read block 3704 len 28672
35583 171/300 Flush
35584 172/300 Read block 4521 len 32768
35585 173/300 Read block 3010 len 16384
35586 174/300 Read block 2891 len 12288
35587 175/300 Write block 615 len 28672 data: 6 6 6 5 4 4 5
35588 176/300 Write block 4094 len 12288 data: 6 6 7
35589 177/300 Read block 1938 len 20480
35590 178/300 Write block 2999 len 4096 data: 3
35591 179/300 Read block 3517 len 4096
35592 180/300 Write block 2101 len 12288 data: 6 5 6
35593 181/300 Read block 3747 len 20480
35594 182/300 Read block 2432 len 20480
35595 183/300 Read block 3181 len 20480
35596 184/300 Read block 974 len 40960
35597 185/300 Read block 78 len 32768
35598 186/300 Write block 35 len 16384 data: 7 7 5 6
35599 187/300 Read block 1988 len 40960
35600 188/300 Write block 729 len 8192 data: 5 5
35601 189/300 Read block 2710 len 24576
35602 190/300 Read block 603 len 12288
35603 191/300 Write block 3992 len 20480 data: 6 6 5 3 4
35604 192/300 Read block 2855 len 12288
35605 193/300 Read block 2848 len 8192
35606 194/300 Read block 798 len 8192
35607 195/300 Write block 2484 len 20480 data: 6 6 7 5 6
35608 196/300 Write block 694 len 24576 data: 6 5 5 5 5 4
35609 197/300 Read block 4113 len 16384
35610 198/300 Write block 583 len 16384 data: 7 6 7 7
35611 199/300 Write block 4826 len 8192 data: 5 6
35612 200/300 Read block 261 len 16384
35613 201/300 Read block 3196 len 20480
35614 202/300 Read block 2519 len 24576
35615 203/300 Write block 3054 len 24576 data: 4 4 4 3 4 8
35616 204/300 Flush
35617 205/300 Read block 1539 len 20480
35618 206/300 Flush
35619 207/300 Read block 4515 len 40960
35620 208/300 Read block 4943 len 24576
35621 209/300 Read block 1521 len 24576
35622 210/300 Read block 3218 len 8192
35623 211/300 Write block 3108 len 20480 data: 6 5 5 7 6
35624 212/300 Write block 4961 len 32768 data: 6 6 9 9 8 7 5 6
35625 213/300 Read block 3359 len 4096
35626 214/300 Read block 2087 len 24576
35627 215/300 Write block 3738 len 12288 data: 4 4 4
35628 216/300 Write block 4159 len 40960 data: 4 4 4 5 5 5 4 4 4 2
35629 217/300 Write block 3984 len 16384 data: 7 5 5 8
35630 218/300 Flush
35631 219/300 Read block 4327 len 8192
35632 220/300 Read block 1936 len 16384
35633 221/300 Read block 3115 len 28672
35634 222/300 Read block 3328 len 24576
35635 223/300 Write block 697 len 24576 data: 6 6 5 4 4 3
35636 224/300 Read block 688 len 36864
35637 225/300 Write block 3722 len 24576 data: 7 7 6 6 4 4
35638 226/300 Write block 4753 len 40960 data: 6 5 5 6 7 10 8 7 6 7
35639 227/300 Flush
35640 228/300 Write block 3372 len 28672 data: 5 5 5 5 4 5 4
35641 229/300 Read block 4614 len 20480
35642 230/300 Read block 4883 len 16384
35643 231/300 Write block 997 len 40960 data: 7 7 7 5 5 5 5 6 5 4
35644 232/300 Write block 543 len 36864 data: 7 7 8 6 6 6 4 3 3
35645 233/300 Read block 4676 len 20480
35646 234/300 Write block 2287 len 20480 data: 6 7 7 7 7
35647 235/300 Read block 4698 len 20480
35648 236/300 Write block 3276 len 16384 data: 7 7 7 7
35649 237/300 Write block 928 len 12288 data: 7 7 6
35650 238/300 Flush
35651 239/300 Read block 3821 len 32768
35652 240/300 Read block 2572 len 4096
35653 241/300 Write block 3665 len 28672 data: 6 6 7 8 7 7 6
35654 242/300 Read block 2 len 8192
35655 243/300 Read block 3215 len 24576
35656 244/300 Write block 2824 len 32768 data: 3 3 3 3 3 3 2 2
35657 245/300 Read block 3893 len 16384
35658 246/300 Read block 583 len 16384
35659 247/300 Write block 514 len 40960 data: 5 5 6 7 5 7 9 9 8 8
35660 248/300 Read block 704 len 8192
35661 249/300 Write block 1740 len 24576 data: 5 5 6 6 7 6
35662 250/300 Read block 930 len 20480
35663 251/300 Write block 940 len 20480 data: 5 5 4 4 5
35664 252/300 Write block 380 len 32768 data: 6 9 10 9 7 7 6 4
35665 253/300 Write block 1374 len 32768 data: 4 3 4 7 7 7 7 5
35666 254/300 Write block 30 len 4096 data: 8
35667 255/300 Read block 4064 len 36864
35668 256/300 Read block 1218 len 16384
35669 257/300 Read block 743 len 12288
35670 258/300 Write block 4337 len 12288 data: 5 5 5
35671 259/300 Write block 1107 len 16384 data: 4 4 6 5
35672 260/300 Write block 136 len 12288 data: 7 8 7
35673 261/300 Write block 127 len 8192 data: 3 3
35674 262/300 Write block 4617 len 4096 data: 9
35675 263/300 Write block 1580 len 36864 data: 6 5 4 3 2 3 3 2 2
35676 264/300 Write block 2501 len 24576 data: 6 6 6 5 7 6
35677 265/300 Read block 2039 len 32768
35678 266/300 Read block 1103 len 4096
35679 267/300 Flush
35680 268/300 Read block 901 len 24576
35681 269/300 Write block 2258 len 16384 data: 4 5 5 7
35682 270/300 Write block 1703 len 4096 data: 6
35683 271/300 Flush
35684 272/300 Flush
35685 273/300 Write block 3148 len 32768 data: 7 5 5 5 4 4 5 5
35686 274/300 Read block 87 len 20480
35687 275/300 Read block 3874 len 12288
35688 276/300 Write block 762 len 16384 data: 5 6 6 7
35689 277/300 Read block 847 len 20480
35690 278/300 Read block 4089 len 40960
35691 279/300 Flush
35692 280/300 Read block 2851 len 20480
35693 281/300 Read block 303 len 20480
35694 282/300 Write block 3826 len 20480 data: 6 5 4 4 6
35695 283/300 Read block 1139 len 32768
35696 284/300 Write block 3128 len 16384 data: 4 7 9 9
35697 285/300 Read block 4272 len 32768
35698 286/300 Flush
35699 287/300 Write block 4370 len 16384 data: 6 5 6 6
35700 288/300 Flush
35701 289/300 Read block 215 len 36864
35702 290/300 Write block 2579 len 8192 data: 7 6
35703 291/300 Write block 1545 len 40960 data: 5 4 3 3 2 2 6 5 5 6
35704 292/300 Write block 3350 len 12288 data: 9 8 8
35705 293/300 Write block 3191 len 4096 data: 5
35706 294/300 Read block 1714 len 12288
35707 295/300 Flush
35708 296/300 Read block 1433 len 36864
35709 297/300 Flush
35710 298/300 Read block 1648 len 28672
35711 299/300 Read block 2982 len 24576
35712 300/300 Read block 2826 len 20480
35713 301/300 Read block 1588 len 12288
35714 Main task has work to do, go find it
35715 got dsc Start(2)
35716 start 2
35717 Feb 21 06:53:19.098 INFO request completed, latency_us: 256, response_code: 204, uri: /start/cid/2, method: POST, req_id: d29d98d8-805f-41f5-a881-6de493e88495, remote_addr: 127.0.0.1:53848, local_addr: 127.0.0.1:9998
35718 Replay: started 2, returned:Ok(())
35719 ----------------------------------------------------------------
35720 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35721 States: Active Active Offline
35722 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35723 10056 Acked Flush 0 Done Done Sent false
35724 10057 Acked Write 4 Done Done Sent false
35725 10058 Acked Read 7 Done Done Sent false
35726 10059 Acked Read 6 Done Done Sent false
35727 10060 Acked Write 7 Done Done Sent false
35728 10061 Acked Read 3 Done Done Sent false
35729 10062 Acked Read 4 Done Done Sent false
35730 10063 Acked Write 6 Done Done Sent false
35731 10064 Acked Flush 0 Done Done Sent false
35732 10065 Acked Read 6 Done Done Sent false
35733 10066 Acked Flush 0 Done Done Sent false
35734 10067 Acked Flush 0 Done Done Sent false
35735 10068 Acked Read 4 Done Done Sent false
35736 10069 Acked Write 10 Done Done Sent false
35737 10070 Acked Write 1 Done Done Sent false
35738 10071 Acked Read 9 Done Done Sent false
35739 10072 Acked Read 8 Done Done Sent false
35740 10073 Acked Read 7 Done Done Sent false
35741 10074 Acked Read 1 Done Done Sent false
35742 10075 Acked Read 6 Done Done Sent false
35743 10076 Acked Flush 0 Done Done Sent false
35744 10077 Acked Flush 0 Done Done Sent false
35745 10078 Acked Write 10 Done Done Sent false
35746 10079 Acked Read 8 Done Done Sent false
35747 10080 Acked Flush 0 Done Done Sent false
35748 10081 Acked Read 2 Done Done Sent false
35749 10082 Acked Write 5 Done Done Sent false
35750 10083 Acked Read 4 Done Done Sent false
35751 10084 Acked Write 3 Done Done Sent false
35752 10085 Acked Read 6 Done Done Sent false
35753 10086 Acked Write 2 Done Done Sent false
35754 10087 Acked Write 1 Done Done Sent false
35755 10088 Acked Write 8 Done Done Sent false
35756 10089 Acked Read 6 Done Done Sent false
35757 10090 Acked Write 3 Done Done Sent false
35758 10091 Acked Read 4 Done Done Sent false
35759 10092 Acked Write 1 Done Done Sent false
35760 10093 Acked Flush 0 Done Done Sent false
35761 10094 Acked Write 10 Done Done Sent false
35762 10095 Acked Read 1 Done Done Sent false
35763 10096 Acked Flush 0 Done Done Sent false
35764 10097 Acked Flush 0 Done Done Sent false
35765 10098 Acked Read 2 Done Done Sent false
35766 10099 Acked Write 10 Done Done Sent false
35767 10100 Acked Read 7 Done Done Sent false
35768 10101 Acked Flush 0 Done Done Sent false
35769 10102 Acked Read 8 Done Done Sent false
35770 10103 Acked Write 10 Done Done Sent false
35771 10104 Acked Flush 0 Done Done Sent false
35772 10105 Acked Read 4 Done Done Sent false
35773 10106 Acked Write 1 Done Done Sent false
35774 10107 Acked Read 9 Done Done Sent false
35775 10108 Acked Read 4 Done Done Sent false
35776 10109 Acked Write 6 Done Done Sent false
35777 10110 Acked Read 5 Done Done Sent false
35778 10111 Acked Read 8 Done Done Sent false
35779 10112 Acked Write 4 Done Done Sent false
35780 10113 Acked Write 2 Done Done Sent false
35781 10114 Acked Flush 0 Done Done Sent false
35782 10115 Acked Read 3 Done Done Sent false
35783 10116 Acked Write 6 Done Done Sent false
35784 10117 Acked Write 8 Done Done Sent false
35785 10118 Acked Write 5 Done Done Sent false
35786 10119 Acked Write 2 Done Done Sent false
35787 10120 Acked Write 9 Done Done Sent false
35788 10121 Acked Flush 0 Done Done Sent false
35789 10122 Acked Flush 0 Done Done Sent false
35790 10123 Acked Read 2 Done Done Sent false
35791 10124 Acked Write 10 Done Done Sent false
35792 10125 Acked Write 2 Done Done Sent false
35793 10126 Acked Flush 0 Done Done Sent false
35794 10127 Acked Read 6 Done Done Sent false
35795 10128 Acked Read 6 Done Done Sent false
35796 10129 Acked Read 10 Done Done Sent false
35797 10130 Acked Flush 0 Done Done Sent false
35798 10131 Acked Read 4 Done Done Sent false
35799 10132 Acked Write 9 Done Done Sent false
35800 10133 Acked Write 3 Done Done Sent false
35801 10134 Acked Write 4 Done Done Sent false
35802 10135 Acked Write 3 Done Done Sent false
35803 10136 Acked Read 2 Done Done Sent false
35804 10137 Acked Write 6 Done Done Sent false
35805 10138 Acked Read 9 Done Done Sent false
35806 10139 Acked Read 7 Done Done Sent false
35807 10140 Acked Read 10 Done Done Sent false
35808 10141 Acked Read 3 Done Done Sent false
35809 10142 Acked Read 7 Done Done Sent false
35810 10143 Acked Write 5 Done Done Sent false
35811 10144 Acked Write 3 Done Done Sent false
35812 10145 Acked Write 9 Done Done Sent false
35813 10146 Acked Read 6 Done Done Sent false
35814 10147 Acked Write 6 Done Done Sent false
35815 10148 Acked Write 6 Done Done Sent false
35816 10149 Acked Flush 0 Done Done Sent false
35817 10150 Acked Write 7 Done Done Sent false
35818 10151 Acked Read 8 Done Done Sent false
35819 10152 Acked Write 8 Done Done Sent false
35820 10153 Acked Flush 0 Done Done Sent false
35821 10154 Acked Write 3 Done Done Sent false
35822 10155 Acked Write 10 Done Done Sent false
35823 10156 Acked Read 9 Done Done Sent false
35824 10157 Acked Write 7 Done Done Sent false
35825 10158 Acked Write 1 Done Done Sent false
35826 10159 Acked Write 6 Done Done Sent false
35827 10160 Acked Write 9 Done Done Sent false
35828 10161 Acked Read 5 Done Done Sent false
35829 10162 Acked Read 10 Done Done Sent false
35830 10163 Acked Write 10 Done Done Sent false
35831 10164 Acked Write 7 Done Done Sent false
35832 10165 Acked Read 7 Done Done Sent false
35833 10166 Acked Write 7 Done Done Sent false
35834 10167 Acked Read 4 Done Done Sent false
35835 10168 Acked Read 3 Done Done Sent false
35836 10169 Acked Read 3 Done Done Sent false
35837 10170 Acked Read 1 Done Done Sent false
35838 10171 Acked Write 9 Done Done Sent false
35839 10172 Acked Read 2 Done Done Sent false
35840 10173 Acked Read 1 Done Done Sent false
35841 10174 Acked Write 5 Done Done Sent false
35842 10175 Acked Read 4 Done Done Sent false
35843 10176 Acked Read 2 Done Done Sent false
35844 10177 Acked Read 6 Done Done Sent false
35845 10178 Acked Write 10 Done Done Sent false
35846 10179 Acked Write 9 Done Done Sent false
35847 10180 Acked Write 9 Done Done Sent false
35848 10181 Acked Read 6 Done Done Sent false
35849 10182 Acked Read 9 Done Done Sent false
35850 10183 Acked Read 2 Done Done Sent false
35851 10184 Acked Read 5 Done Done Sent false
35852 10185 Acked Read 6 Done Done Sent false
35853 10186 Acked Read 9 Done Done Sent false
35854 10187 Acked Flush 0 Done Done Sent false
35855 10188 Acked Read 4 Done Done Sent false
35856 10189 Acked Read 3 Done Done Sent false
35857 10190 Acked Read 3 Done Done Sent false
35858 10191 Acked Flush 0 Done Done Sent false
35859 10192 Acked Read 7 Done Done Sent false
35860 10193 Acked Write 2 Done Done Sent false
35861 10194 Acked Read 10 Done Done Sent false
35862 10195 Acked Read 2 Done Done Sent false
35863 10196 Acked Read 5 Done Done Sent false
35864 10197 Acked Read 4 Done Done Sent false
35865 10198 Acked Read 8 Done Done Sent false
35866 10199 Acked Read 6 Done Done Sent false
35867 10200 Acked Write 3 Done Done Sent false
35868 10201 Acked Read 4 Done Done Sent false
35869 10202 Acked Write 7 Done Done Sent false
35870 10203 Acked Flush 0 Done Done Sent false
35871 10204 Acked Read 3 Done Done Sent false
35872 10205 Acked Read 5 Done Done Sent false
35873 10206 Acked Read 4 Done Done Sent false
35874 10207 Acked Write 3 Done Done Sent false
35875 10208 Acked Read 4 Done Done Sent false
35876 10209 Acked Write 8 Done Done Sent false
35877 10210 Acked Read 4 Done Done Sent false
35878 10211 Acked Read 2 Done Done Sent false
35879 10212 Acked Write 1 Done Done Sent false
35880 10213 Acked Write 2 Done Done Sent false
35881 10214 Acked Flush 0 Done Done Sent false
35882 10215 Acked Write 1 Done Done Sent false
35883 10216 Acked Read 5 Done Done Sent false
35884 10217 Acked Write 9 Done Done Sent false
35885 10218 Acked Read 6 Done Done Sent false
35886 10219 Acked Write 1 Done Done Sent false
35887 10220 Acked Read 5 Done Done Sent false
35888 10221 Acked Read 4 Done Done Sent false
35889 10222 Acked Read 5 Done Done Sent false
35890 10223 Acked Read 1 Done Done Sent false
35891 10224 Acked Read 8 Done Done Sent false
35892 10225 Acked Read 7 Done Done Sent false
35893 10226 Acked Flush 0 Done Done Sent false
35894 10227 Acked Read 8 Done Done Sent false
35895 10228 Acked Read 4 Done Done Sent false
35896 10229 Acked Read 3 Done Done Sent false
35897 10230 Acked Write 7 Done Done Sent false
35898 10231 Acked Write 3 Done Done Sent false
35899 10232 Acked Read 5 Done Done Sent false
35900 10233 Acked Write 1 Done Done Sent false
35901 10234 Acked Read 1 Done Done Sent false
35902 10235 Acked Write 3 Done Done Sent false
35903 10236 Acked Read 5 Done Done Sent false
35904 10237 Acked Read 5 Done Done Sent false
35905 10238 Acked Read 5 Done Done Sent false
35906 10239 Acked Read 10 Done Done Sent false
35907 10240 Acked Read 8 Done Done Sent false
35908 10241 Acked Write 4 Done Done Sent false
35909 10242 Acked Read 10 Done Done Sent false
35910 10243 Acked Write 2 Done Done Sent false
35911 10244 Acked Read 6 Done Done Sent false
35912 10245 Acked Read 3 Done Done Sent false
35913 10246 Acked Write 5 Done Done Sent false
35914 10247 Acked Read 3 Done Done Sent false
35915 10248 Acked Read 2 Done Done Sent false
35916 10249 Acked Read 2 Done Done Sent false
35917 10250 Acked Write 5 Done Done Sent false
35918 10251 Acked Write 6 Done Done Sent false
35919 10252 Acked Read 4 Done Done Sent false
35920 10253 Acked Write 4 Done Done Sent false
35921 10254 Acked Write 2 Done Done Sent false
35922 10255 Acked Read 4 Done Done Sent false
35923 10256 Acked Read 5 Done Done Sent false
35924 10257 Acked Read 6 Done Done Sent false
35925 10258 Acked Write 6 Done Done Sent false
35926 10259 Acked Flush 0 Done Done Sent false
35927 10260 Acked Read 5 Done Done Sent false
35928 10261 Acked Flush 0 Done Done Sent false
35929 10262 Acked Read 10 Done Done Sent false
35930 10263 Acked Read 6 Done Done Sent false
35931 10264 Acked Read 6 Done Done Sent false
35932 10265 Acked Read 2 Done Done Sent false
35933 10266 Acked Write 5 Done Done Sent false
35934 10267 Acked Write 8 Done Done Sent false
35935 10268 Acked Read 1 Done Done Sent false
35936 10269 Acked Read 6 Done Done Sent false
35937 10270 Acked Write 3 Done Done Sent false
35938 10271 Acked Write 10 Done Done Sent false
35939 10272 Acked Write 4 Done Done Sent false
35940 10273 Acked Flush 0 Done Done Sent false
35941 10274 Acked Read 2 Done Done Sent false
35942 10275 Acked Read 4 Done Done Sent false
35943 10276 Acked Read 7 Done Done Sent false
35944 10277 Acked Read 6 Done Done Sent false
35945 10278 Acked Write 6 Done Done Sent false
35946 10279 Acked Read 9 Done Done Sent false
35947 10280 Acked Write 6 Done Done Sent false
35948 10281 Acked Write 10 Done Done Sent false
35949 10282 Acked Flush 0 Done Done Sent false
35950 10283 Acked Write 7 Done Done Sent false
35951 10284 Acked Read 5 Done Done Sent false
35952 10285 Acked Read 4 Done Done Sent false
35953 10286 Acked Write 10 Done Done Sent false
35954 10287 Acked Write 9 Done Done Sent false
35955 10288 Acked Read 5 Done Done Sent false
35956 10289 Acked Write 5 Done Done Sent false
35957 10290 Acked Read 5 Done Done Sent false
35958 10291 Acked Write 4 Done Done Sent false
35959 10292 Acked Write 3 Done Done Sent false
35960 10293 Acked Flush 0 Done Done Sent false
35961 10294 Acked Read 8 Done Done Sent false
35962 10295 Acked Read 1 Done Done Sent false
35963 10296 Acked Write 7 Done Done Sent false
35964 10297 Acked Read 2 Done Done Sent false
35965 10298 Acked Read 6 Done Done Sent false
35966 10299 Acked Write 8 Done Done Sent false
35967 10300 Acked Read 4 Done Done Sent false
35968 10301 Acked Read 4 Done Done Sent false
35969 10302 Acked Write 10 Done Done Sent false
35970 10303 Acked Read 2 Done Done Sent false
35971 10304 Acked Write 6 Done Done Sent false
35972 10305 Acked Read 5 Done Done Sent false
35973 10306 Acked Write 5 Done Done Sent false
35974 10307 Acked Write 8 Done Done Sent false
35975 10308 Acked Write 8 Done Done Sent false
35976 10309 Acked Write 1 Done Done Sent false
35977 10310 Acked Read 9 Done Done Sent false
35978 10311 Acked Read 4 Done Done Sent false
35979 10312 Acked Read 3 Done Done Sent false
35980 10313 Acked Write 3 Done Done Sent false
35981 10314 Acked Write 4 Done Done Sent false
35982 10315 Acked Write 3 Done Done Sent false
35983 10316 Acked Write 2 Done Done Sent false
35984 10317 Acked Write 1 Done Done Sent false
35985 10318 Acked Write 9 Done Done Sent false
35986 10319 Acked Write 6 Done Done Sent false
35987 10320 Acked Read 8 Done Done Sent false
35988 10321 Acked Read 1 Done Done Sent false
35989 10322 Acked Flush 0 Done Done Sent false
35990 10323 Acked Read 6 Done Done Sent false
35991 10324 Acked Write 4 Done Done Sent false
35992 10325 Acked Write 1 Done Done Sent false
35993 10326 Acked Flush 0 Done Done Sent false
35994 10327 Acked Flush 0 Done Done Sent false
35995 10328 Acked Write 8 Done Done Sent false
35996 10329 Acked Read 5 Done Done Sent false
35997 10330 Acked Read 3 Done Done Sent false
35998 10331 Acked Write 4 Done Done Sent false
35999 10332 Acked Read 5 Done Done Sent false
36000 10333 Acked Read 10 Done Done Sent false
36001 10334 Acked Flush 0 Done Done Sent false
36002 10335 Acked Read 5 Done Done Sent false
36003 10336 Acked Read 5 Done Done Sent false
36004 10337 Acked Write 5 Done Done Sent false
36005 10338 Acked Read 8 Done Done Sent false
36006 10339 Acked Write 4 Done Done Sent false
36007 10340 Acked Read 8 Done Done Sent false
36008 10341 Acked Flush 0 Done Done Sent false
36009 10342 Acked Write 4 Done Done Sent false
36010 10343 Acked Flush 0 Done Done Sent false
36011 10344 Acked Read 9 Done Done Sent false
36012 10345 Acked Write 2 Done Done Sent false
36013 10346 Acked Write 10 Done Done Sent false
36014 10347 Acked Write 3 Done Done Sent false
36015 10348 Acked Write 1 Done Done Sent false
36016 10349 Acked Read 3 Done Done Sent false
36017 10350 Acked Flush 0 Done Done Sent false
36018 10351 Acked Read 9 Done Done Sent false
36019 10352 Acked Flush 0 Done Done Sent false
36020 10353 Acked Read 7 Done Done Sent false
36021 10354 Acked Read 6 Done Done Sent false
36022 10355 Acked Read 5 Done Done Sent false
36023 10356 Acked Read 3 Done Done Sent false
36024 STATES DS:0 DS:1 DS:2 TOTAL
36025 Sent 0 0 301 301
36026 Done 301 301 0 602
36027 Skipped 0 0 0 0
36028 Error 0 0 0 0
36029 Last Flush: Some(JobId(10352)) Some(JobId(10352)) Some(JobId(10055))
36030 Downstairs last five completed: 10055 10054 10053 10052 10051
36031 Upstairs last five completed: 10356 10355 10354 10353 10352
36032 CLIENT: Up:0 ds:301 act:2
360332025-02-21T06:53:19.600ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
360342025-02-21T06:53:20.386ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
36035 [8830] got start action
36036 [8830] I am going to restart
36037 Starting downstairs at port 8830
36038 Make output file at "/tmp/dsc/downstairs-8830.txt"
36039 [8830][2] reports Starting
36040 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1237)
36041 [8830][2] reports Running
360422025-02-21T06:53:21.606ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
360432025-02-21T06:53:22.388ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
36044 ----------------------------------------------------------------
36045 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
36046 States: Active Active Offline
36047 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36048 10056 Acked Flush 0 Done Done Sent false
36049 10057 Acked Write 4 Done Done Sent false
36050 10058 Acked Read 7 Done Done Sent false
36051 10059 Acked Read 6 Done Done Sent false
36052 10060 Acked Write 7 Done Done Sent false
36053 10061 Acked Read 3 Done Done Sent false
36054 10062 Acked Read 4 Done Done Sent false
36055 10063 Acked Write 6 Done Done Sent false
36056 10064 Acked Flush 0 Done Done Sent false
36057 10065 Acked Read 6 Done Done Sent false
36058 10066 Acked Flush 0 Done Done Sent false
36059 10067 Acked Flush 0 Done Done Sent false
36060 10068 Acked Read 4 Done Done Sent false
36061 10069 Acked Write 10 Done Done Sent false
36062 10070 Acked Write 1 Done Done Sent false
36063 10071 Acked Read 9 Done Done Sent false
36064 10072 Acked Read 8 Done Done Sent false
36065 10073 Acked Read 7 Done Done Sent false
36066 10074 Acked Read 1 Done Done Sent false
36067 10075 Acked Read 6 Done Done Sent false
36068 10076 Acked Flush 0 Done Done Sent false
36069 10077 Acked Flush 0 Done Done Sent false
36070 10078 Acked Write 10 Done Done Sent false
36071 10079 Acked Read 8 Done Done Sent false
36072 10080 Acked Flush 0 Done Done Sent false
36073 10081 Acked Read 2 Done Done Sent false
36074 10082 Acked Write 5 Done Done Sent false
36075 10083 Acked Read 4 Done Done Sent false
36076 10084 Acked Write 3 Done Done Sent false
36077 10085 Acked Read 6 Done Done Sent false
36078 10086 Acked Write 2 Done Done Sent false
36079 10087 Acked Write 1 Done Done Sent false
36080 10088 Acked Write 8 Done Done Sent false
36081 10089 Acked Read 6 Done Done Sent false
36082 10090 Acked Write 3 Done Done Sent false
36083 10091 Acked Read 4 Done Done Sent false
36084 10092 Acked Write 1 Done Done Sent false
36085 10093 Acked Flush 0 Done Done Sent false
36086 10094 Acked Write 10 Done Done Sent false
36087 10095 Acked Read 1 Done Done Sent false
36088 10096 Acked Flush 0 Done Done Sent false
36089 10097 Acked Flush 0 Done Done Sent false
36090 10098 Acked Read 2 Done Done Sent false
36091 10099 Acked Write 10 Done Done Sent false
36092 10100 Acked Read 7 Done Done Sent false
36093 10101 Acked Flush 0 Done Done Sent false
36094 10102 Acked Read 8 Done Done Sent false
36095 10103 Acked Write 10 Done Done Sent false
36096 10104 Acked Flush 0 Done Done Sent false
36097 10105 Acked Read 4 Done Done Sent false
36098 10106 Acked Write 1 Done Done Sent false
36099 10107 Acked Read 9 Done Done Sent false
36100 10108 Acked Read 4 Done Done Sent false
36101 10109 Acked Write 6 Done Done Sent false
36102 10110 Acked Read 5 Done Done Sent false
36103 10111 Acked Read 8 Done Done Sent false
36104 10112 Acked Write 4 Done Done Sent false
36105 10113 Acked Write 2 Done Done Sent false
36106 10114 Acked Flush 0 Done Done Sent false
36107 10115 Acked Read 3 Done Done Sent false
36108 10116 Acked Write 6 Done Done Sent false
36109 10117 Acked Write 8 Done Done Sent false
36110 10118 Acked Write 5 Done Done Sent false
36111 10119 Acked Write 2 Done Done Sent false
36112 10120 Acked Write 9 Done Done Sent false
36113 10121 Acked Flush 0 Done Done Sent false
36114 10122 Acked Flush 0 Done Done Sent false
36115 10123 Acked Read 2 Done Done Sent false
36116 10124 Acked Write 10 Done Done Sent false
36117 10125 Acked Write 2 Done Done Sent false
36118 10126 Acked Flush 0 Done Done Sent false
36119 10127 Acked Read 6 Done Done Sent false
36120 10128 Acked Read 6 Done Done Sent false
36121 10129 Acked Read 10 Done Done Sent false
36122 10130 Acked Flush 0 Done Done Sent false
36123 10131 Acked Read 4 Done Done Sent false
36124 10132 Acked Write 9 Done Done Sent false
36125 10133 Acked Write 3 Done Done Sent false
36126 10134 Acked Write 4 Done Done Sent false
36127 10135 Acked Write 3 Done Done Sent false
36128 10136 Acked Read 2 Done Done Sent false
36129 10137 Acked Write 6 Done Done Sent false
36130 10138 Acked Read 9 Done Done Sent false
36131 10139 Acked Read 7 Done Done Sent false
36132 10140 Acked Read 10 Done Done Sent false
36133 10141 Acked Read 3 Done Done Sent false
36134 10142 Acked Read 7 Done Done Sent false
36135 10143 Acked Write 5 Done Done Sent false
36136 10144 Acked Write 3 Done Done Sent false
36137 10145 Acked Write 9 Done Done Sent false
36138 10146 Acked Read 6 Done Done Sent false
36139 10147 Acked Write 6 Done Done Sent false
36140 10148 Acked Write 6 Done Done Sent false
36141 10149 Acked Flush 0 Done Done Sent false
36142 10150 Acked Write 7 Done Done Sent false
36143 10151 Acked Read 8 Done Done Sent false
36144 10152 Acked Write 8 Done Done Sent false
36145 10153 Acked Flush 0 Done Done Sent false
36146 10154 Acked Write 3 Done Done Sent false
36147 10155 Acked Write 10 Done Done Sent false
36148 10156 Acked Read 9 Done Done Sent false
36149 10157 Acked Write 7 Done Done Sent false
36150 10158 Acked Write 1 Done Done Sent false
36151 10159 Acked Write 6 Done Done Sent false
36152 10160 Acked Write 9 Done Done Sent false
36153 10161 Acked Read 5 Done Done Sent false
36154 10162 Acked Read 10 Done Done Sent false
36155 10163 Acked Write 10 Done Done Sent false
36156 10164 Acked Write 7 Done Done Sent false
36157 10165 Acked Read 7 Done Done Sent false
36158 10166 Acked Write 7 Done Done Sent false
36159 10167 Acked Read 4 Done Done Sent false
36160 10168 Acked Read 3 Done Done Sent false
36161 10169 Acked Read 3 Done Done Sent false
36162 10170 Acked Read 1 Done Done Sent false
36163 10171 Acked Write 9 Done Done Sent false
36164 10172 Acked Read 2 Done Done Sent false
36165 10173 Acked Read 1 Done Done Sent false
36166 10174 Acked Write 5 Done Done Sent false
36167 10175 Acked Read 4 Done Done Sent false
36168 10176 Acked Read 2 Done Done Sent false
36169 10177 Acked Read 6 Done Done Sent false
36170 10178 Acked Write 10 Done Done Sent false
36171 10179 Acked Write 9 Done Done Sent false
36172 10180 Acked Write 9 Done Done Sent false
36173 10181 Acked Read 6 Done Done Sent false
36174 10182 Acked Read 9 Done Done Sent false
36175 10183 Acked Read 2 Done Done Sent false
36176 10184 Acked Read 5 Done Done Sent false
36177 10185 Acked Read 6 Done Done Sent false
36178 10186 Acked Read 9 Done Done Sent false
36179 10187 Acked Flush 0 Done Done Sent false
36180 10188 Acked Read 4 Done Done Sent false
36181 10189 Acked Read 3 Done Done Sent false
36182 10190 Acked Read 3 Done Done Sent false
36183 10191 Acked Flush 0 Done Done Sent false
36184 10192 Acked Read 7 Done Done Sent false
36185 10193 Acked Write 2 Done Done Sent false
36186 10194 Acked Read 10 Done Done Sent false
36187 10195 Acked Read 2 Done Done Sent false
36188 10196 Acked Read 5 Done Done Sent false
36189 10197 Acked Read 4 Done Done Sent false
36190 10198 Acked Read 8 Done Done Sent false
36191 10199 Acked Read 6 Done Done Sent false
36192 10200 Acked Write 3 Done Done Sent false
36193 10201 Acked Read 4 Done Done Sent false
36194 10202 Acked Write 7 Done Done Sent false
36195 10203 Acked Flush 0 Done Done Sent false
36196 10204 Acked Read 3 Done Done Sent false
36197 10205 Acked Read 5 Done Done Sent false
36198 10206 Acked Read 4 Done Done Sent false
36199 10207 Acked Write 3 Done Done Sent false
36200 10208 Acked Read 4 Done Done Sent false
36201 10209 Acked Write 8 Done Done Sent false
36202 10210 Acked Read 4 Done Done Sent false
36203 10211 Acked Read 2 Done Done Sent false
36204 10212 Acked Write 1 Done Done Sent false
36205 10213 Acked Write 2 Done Done Sent false
36206 10214 Acked Flush 0 Done Done Sent false
36207 10215 Acked Write 1 Done Done Sent false
36208 10216 Acked Read 5 Done Done Sent false
36209 10217 Acked Write 9 Done Done Sent false
36210 10218 Acked Read 6 Done Done Sent false
36211 10219 Acked Write 1 Done Done Sent false
36212 10220 Acked Read 5 Done Done Sent false
36213 10221 Acked Read 4 Done Done Sent false
36214 10222 Acked Read 5 Done Done Sent false
36215 10223 Acked Read 1 Done Done Sent false
36216 10224 Acked Read 8 Done Done Sent false
36217 10225 Acked Read 7 Done Done Sent false
36218 10226 Acked Flush 0 Done Done Sent false
36219 10227 Acked Read 8 Done Done Sent false
36220 10228 Acked Read 4 Done Done Sent false
36221 10229 Acked Read 3 Done Done Sent false
36222 10230 Acked Write 7 Done Done Sent false
36223 10231 Acked Write 3 Done Done Sent false
36224 10232 Acked Read 5 Done Done Sent false
36225 10233 Acked Write 1 Done Done Sent false
36226 10234 Acked Read 1 Done Done Sent false
36227 10235 Acked Write 3 Done Done Sent false
36228 10236 Acked Read 5 Done Done Sent false
36229 10237 Acked Read 5 Done Done Sent false
36230 10238 Acked Read 5 Done Done Sent false
36231 10239 Acked Read 10 Done Done Sent false
36232 10240 Acked Read 8 Done Done Sent false
36233 10241 Acked Write 4 Done Done Sent false
36234 10242 Acked Read 10 Done Done Sent false
36235 10243 Acked Write 2 Done Done Sent false
36236 10244 Acked Read 6 Done Done Sent false
36237 10245 Acked Read 3 Done Done Sent false
36238 10246 Acked Write 5 Done Done Sent false
36239 10247 Acked Read 3 Done Done Sent false
36240 10248 Acked Read 2 Done Done Sent false
36241 10249 Acked Read 2 Done Done Sent false
36242 10250 Acked Write 5 Done Done Sent false
36243 10251 Acked Write 6 Done Done Sent false
36244 10252 Acked Read 4 Done Done Sent false
36245 10253 Acked Write 4 Done Done Sent false
36246 10254 Acked Write 2 Done Done Sent false
36247 10255 Acked Read 4 Done Done Sent false
36248 10256 Acked Read 5 Done Done Sent false
36249 10257 Acked Read 6 Done Done Sent false
36250 10258 Acked Write 6 Done Done Sent false
36251 10259 Acked Flush 0 Done Done Sent false
36252 10260 Acked Read 5 Done Done Sent false
36253 10261 Acked Flush 0 Done Done Sent false
36254 10262 Acked Read 10 Done Done Sent false
36255 10263 Acked Read 6 Done Done Sent false
36256 10264 Acked Read 6 Done Done Sent false
36257 10265 Acked Read 2 Done Done Sent false
36258 10266 Acked Write 5 Done Done Sent false
36259 10267 Acked Write 8 Done Done Sent false
36260 10268 Acked Read 1 Done Done Sent false
36261 10269 Acked Read 6 Done Done Sent false
36262 10270 Acked Write 3 Done Done Sent false
36263 10271 Acked Write 10 Done Done Sent false
36264 10272 Acked Write 4 Done Done Sent false
36265 10273 Acked Flush 0 Done Done Sent false
36266 10274 Acked Read 2 Done Done Sent false
36267 10275 Acked Read 4 Done Done Sent false
36268 10276 Acked Read 7 Done Done Sent false
36269 10277 Acked Read 6 Done Done Sent false
36270 10278 Acked Write 6 Done Done Sent false
36271 10279 Acked Read 9 Done Done Sent false
36272 10280 Acked Write 6 Done Done Sent false
36273 10281 Acked Write 10 Done Done Sent false
36274 10282 Acked Flush 0 Done Done Sent false
36275 10283 Acked Write 7 Done Done Sent false
36276 10284 Acked Read 5 Done Done Sent false
36277 10285 Acked Read 4 Done Done Sent false
36278 10286 Acked Write 10 Done Done Sent false
36279 10287 Acked Write 9 Done Done Sent false
36280 10288 Acked Read 5 Done Done Sent false
36281 10289 Acked Write 5 Done Done Sent false
36282 10290 Acked Read 5 Done Done Sent false
36283 10291 Acked Write 4 Done Done Sent false
36284 10292 Acked Write 3 Done Done Sent false
36285 10293 Acked Flush 0 Done Done Sent false
36286 10294 Acked Read 8 Done Done Sent false
36287 10295 Acked Read 1 Done Done Sent false
36288 10296 Acked Write 7 Done Done Sent false
36289 10297 Acked Read 2 Done Done Sent false
36290 10298 Acked Read 6 Done Done Sent false
36291 10299 Acked Write 8 Done Done Sent false
36292 10300 Acked Read 4 Done Done Sent false
36293 10301 Acked Read 4 Done Done Sent false
36294 10302 Acked Write 10 Done Done Sent false
36295 10303 Acked Read 2 Done Done Sent false
36296 10304 Acked Write 6 Done Done Sent false
36297 10305 Acked Read 5 Done Done Sent false
36298 10306 Acked Write 5 Done Done Sent false
36299 10307 Acked Write 8 Done Done Sent false
36300 10308 Acked Write 8 Done Done Sent false
36301 10309 Acked Write 1 Done Done Sent false
36302 10310 Acked Read 9 Done Done Sent false
36303 10311 Acked Read 4 Done Done Sent false
36304 10312 Acked Read 3 Done Done Sent false
36305 10313 Acked Write 3 Done Done Sent false
36306 10314 Acked Write 4 Done Done Sent false
36307 10315 Acked Write 3 Done Done Sent false
36308 10316 Acked Write 2 Done Done Sent false
36309 10317 Acked Write 1 Done Done Sent false
36310 10318 Acked Write 9 Done Done Sent false
36311 10319 Acked Write 6 Done Done Sent false
36312 10320 Acked Read 8 Done Done Sent false
36313 10321 Acked Read 1 Done Done Sent false
36314 10322 Acked Flush 0 Done Done Sent false
36315 10323 Acked Read 6 Done Done Sent false
36316 10324 Acked Write 4 Done Done Sent false
36317 10325 Acked Write 1 Done Done Sent false
36318 10326 Acked Flush 0 Done Done Sent false
36319 10327 Acked Flush 0 Done Done Sent false
36320 10328 Acked Write 8 Done Done Sent false
36321 10329 Acked Read 5 Done Done Sent false
36322 10330 Acked Read 3 Done Done Sent false
36323 10331 Acked Write 4 Done Done Sent false
36324 10332 Acked Read 5 Done Done Sent false
36325 10333 Acked Read 10 Done Done Sent false
36326 10334 Acked Flush 0 Done Done Sent false
36327 10335 Acked Read 5 Done Done Sent false
36328 10336 Acked Read 5 Done Done Sent false
36329 10337 Acked Write 5 Done Done Sent false
36330 10338 Acked Read 8 Done Done Sent false
36331 10339 Acked Write 4 Done Done Sent false
36332 10340 Acked Read 8 Done Done Sent false
36333 10341 Acked Flush 0 Done Done Sent false
36334 10342 Acked Write 4 Done Done Sent false
36335 10343 Acked Flush 0 Done Done Sent false
36336 10344 Acked Read 9 Done Done Sent false
36337 10345 Acked Write 2 Done Done Sent false
36338 10346 Acked Write 10 Done Done Sent false
36339 10347 Acked Write 3 Done Done Sent false
36340 10348 Acked Write 1 Done Done Sent false
36341 10349 Acked Read 3 Done Done Sent false
36342 10350 Acked Flush 0 Done Done Sent false
36343 10351 Acked Read 9 Done Done Sent false
36344 10352 Acked Flush 0 Done Done Sent false
36345 10353 Acked Read 7 Done Done Sent false
36346 10354 Acked Read 6 Done Done Sent false
36347 10355 Acked Read 5 Done Done Sent false
36348 10356 Acked Read 3 Done Done Sent false
36349 STATES DS:0 DS:1 DS:2 TOTAL
36350 Sent 0 0 301 301
36351 Done 301 301 0 602
36352 Skipped 0 0 0 0
36353 Error 0 0 0 0
36354 Last Flush: Some(JobId(10352)) Some(JobId(10352)) Some(JobId(10055))
36355 Downstairs last five completed: 10055 10054 10053 10052 10051
36356 Upstairs last five completed: 10356 10355 10354 10353 10352
36357 CLIENT: Up:0 ds:301 act:2
363582025-02-21T06:53:23.602ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
363592025-02-21T06:53:24.391ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
363602025-02-21T06:53:24.859ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
363612025-02-21T06:53:24.859ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34770) = io task client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
363622025-02-21T06:53:24.860ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
363632025-02-21T06:53:24.860ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
363642025-02-21T06:53:24.860ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
363652025-02-21T06:53:24.860ZINFOcrucible: send last flush ID to this DS: Some(JobId(10055)) = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
363662025-02-21T06:53:24.860ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
363672025-02-21T06:53:24.860ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(10055)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
363682025-02-21T06:53:24.861ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(10055)) = downstairs session_id = 3a757114-e39d-42e6-bc2b-0419bce3aed2
36369 ----------------------------------------------------------------
36370 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36371 Downstairs last five completed: 10357 10356 10355 10354 10353
36372 Upstairs last five completed: 10357 10356 10355 10354 10353
36373 CLIENT: Up:0 ds:0 act:3
36374 Replay: All jobs finished
36375 Test replay has completed
36376 Wrote out file "/tmp/test_replay_verify.log"
36377 CLIENT: Tests done. All submitted work has been ACK'd
36378 ----------------------------------------------------------------
36379 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36380 Downstairs last five completed: 10357 10356 10355 10354 10353
36381 Upstairs last five completed: 10357 10356 10355 10354 10353
36382 CLIENT: Up:0 ds:0 act:3
36383 CLIENT: All jobs finished, all DS active.
36384 Replays:30 time: 8:04
36385 Do final verify
36386 Feb 21 06:53:27.179 INFO accepted connection, remote_addr: 127.0.0.1:51760, local_addr: 127.0.0.1:9998
36387 Feb 21 06:53:27.180 INFO request completed, latency_us: 198, response_code: 200, uri: /regioninfo, method: GET, req_id: dad4c917-0bb6-411a-87bc-5a31d94ab330, remote_addr: 127.0.0.1:51760, local_addr: 127.0.0.1:9998
363882025-02-21T06:53:27.180ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
36389 Feb 21 06:53:27.181 INFO request completed, latency_us: 91, response_code: 200, uri: /regioncount, method: GET, req_id: c48c849d-271d-4167-881e-35dc6c0a4d54, remote_addr: 127.0.0.1:51760, local_addr: 127.0.0.1:9998
363902025-02-21T06:53:27.181ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
36391 Feb 21 06:53:27.181 INFO request completed, latency_us: 105, response_code: 200, uri: /port/cid/0, method: GET, req_id: eb695c4d-2ed5-45fc-8c29-a2f3ee11e00c, remote_addr: 127.0.0.1:51760, local_addr: 127.0.0.1:9998
36392 Feb 21 06:53:27.182 INFO request completed, latency_us: 117, response_code: 200, uri: /port/cid/1, method: GET, req_id: 38dd9d00-7d08-43b6-93a1-80792001bebc, remote_addr: 127.0.0.1:51760, local_addr: 127.0.0.1:9998
36393 Feb 21 06:53:27.182 INFO request completed, latency_us: 122, response_code: 200, uri: /port/cid/2, method: GET, req_id: d0aad481-c0ab-4bbb-8935-0f988f86a989, remote_addr: 127.0.0.1:51760, local_addr: 127.0.0.1:9998
363942025-02-21T06:53:27.183ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
363952025-02-21T06:53:27.183ZINFOcrucible: Upstairs starts
363962025-02-21T06:53:27.183ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "7ae555c374784c668297ef5478b8ff4aaa8d57b4", git_commit_timestamp: "2025-02-21T06:29:49.000000000Z", git_branch: "renovate/rust-1.x", rustc_semver: "1.85.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "4d91de4e48198da2e33413efdcd9cd2cc0c46688", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
363972025-02-21T06:53:27.183ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
363982025-02-21T06:53:27.183ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
363992025-02-21T06:53:27.184ZINFOcrucible: Crucible 0f1d5e09-097e-416c-ad35-3da02e731a31 has session id: 2389ad2b-b86b-4295-add2-5736429d0282 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364002025-02-21T06:53:27.184ZINFOcrucible: Upstairs opts: Upstairs UUID: 0f1d5e09-097e-416c-ad35-3da02e731a31, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364012025-02-21T06:53:27.184ZINFOcrucible: Crucible stats registered with UUID: 0f1d5e09-097e-416c-ad35-3da02e731a31 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364022025-02-21T06:53:27.184ZWARNcrucible: could not find Downstairs address for Nexus session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364032025-02-21T06:53:27.185ZINFOcrucible: Downstairs in volume = 3 task = crutest
364042025-02-21T06:53:27.185ZINFOcrucible: The guest has requested activation with gen:3
364052025-02-21T06:53:27.185ZINFOcrucible: 0f1d5e09-097e-416c-ad35-3da02e731a31 active request set session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364062025-02-21T06:53:27.185ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364072025-02-21T06:53:27.185ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364082025-02-21T06:53:27.185ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364092025-02-21T06:53:27.185ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364102025-02-21T06:53:27.185ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364112025-02-21T06:53:27.185ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364122025-02-21T06:53:27.185ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364132025-02-21T06:53:27.185ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364142025-02-21T06:53:27.185ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364152025-02-21T06:53:27.185ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42671) = io task client = 1 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364162025-02-21T06:53:27.185ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37475) = io task client = 0 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364172025-02-21T06:53:27.185ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36263) = io task client = 2 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364182025-02-21T06:53:27.186ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364192025-02-21T06:53:27.186ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364202025-02-21T06:53:27.186ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364212025-02-21T06:53:27.187ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364222025-02-21T06:53:27.187ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364232025-02-21T06:53:27.187ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364242025-02-21T06:53:27.187ZINFOcrucible: Waiting for more clients to be ready session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364252025-02-21T06:53:27.187ZINFOcrucible: Waiting for more clients to be ready session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364262025-02-21T06:53:27.187ZINFOcrucible: [0]R flush_numbers[0..12]: [957, 957, 927, 957, 914, 957, 955, 960, 935, 957, 956, 957] = downstairs session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364272025-02-21T06:53:27.188ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364282025-02-21T06:53:27.188ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364292025-02-21T06:53:27.188ZINFOcrucible: [1]R flush_numbers[0..12]: [957, 957, 927, 957, 914, 957, 955, 960, 935, 957, 956, 957] = downstairs session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364302025-02-21T06:53:27.188ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364312025-02-21T06:53:27.188ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364322025-02-21T06:53:27.188ZINFOcrucible: [2]R flush_numbers[0..12]: [957, 957, 927, 957, 914, 957, 955, 960, 935, 957, 956, 957] = downstairs session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364332025-02-21T06:53:27.188ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364342025-02-21T06:53:27.188ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364352025-02-21T06:53:27.188ZINFOcrucible: Max found gen is 3 = downstairs session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364362025-02-21T06:53:27.188ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364372025-02-21T06:53:27.188ZINFOcrucible: Next flush: 964 = downstairs session_id = 2389ad2b-b86b-4295-add2-5736429d0282
36438 {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:53:27.188150862Z","hostname":"w-01JMKKZ32D872C7C41J12FXWY5","pid":1242,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO
36439 2389ad2b-b86b-4295-add2-5736429d0282"}
364402025-02-21T06:53:27.188ZINFOcrucible: No downstairs reconciliation required session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364412025-02-21T06:53:27.188ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364422025-02-21T06:53:27.188ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364432025-02-21T06:53:27.188ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364442025-02-21T06:53:27.188ZINFOcrucible: All required reconciliation work is completed session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364452025-02-21T06:53:27.188ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364462025-02-21T06:53:27.188ZINFOcrucible: 0f1d5e09-097e-416c-ad35-3da02e731a31 is now active with session: 2389ad2b-b86b-4295-add2-5736429d0282 session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364472025-02-21T06:53:27.188ZINFOcrucible: Set Active after no reconciliation session_id = 2389ad2b-b86b-4295-add2-5736429d0282
364482025-02-21T06:53:27.188ZINFOcrucible: The guest has finished waiting for activation with:3
36449 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
36450 Loading write count information from file "/tmp/test_replay_verify.log"
36451 Read and Verify all blocks (0..5000 range:false)
36452 Wrote out file "/tmp/test_replay_verify.log"
36453 Verify test completed
36454 Wrote out file "/tmp/test_replay_verify.log"
36455 CLIENT: Tests done. All submitted work has been ACK'd
36456 ----------------------------------------------------------------
36457 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
36458 States: Active Active Active
36459 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36460 1000 Acked Read 256 Done Done Done false
36461 1001 Acked Read 256 Done Done Done false
36462 1002 Acked Read 256 Done Done Done false
36463 1003 Acked Read 256 Done Done Done false
36464 1004 Acked Read 256 Done Done Done false
36465 1005 Acked Read 256 Done Done Done false
36466 1006 Acked Read 256 Done Done Done false
36467 1007 Acked Read 256 Done Done Done false
36468 1008 Acked Read 256 Done Done Done false
36469 1009 Acked Read 256 Done Done Done false
36470 1010 Acked Read 256 Done Done Done false
36471 1011 Acked Read 256 Done Done Done false
36472 1012 Acked Read 256 Done Done Done false
36473 1013 Acked Read 256 Done Done Done false
36474 1014 Acked Read 256 Done Done Done false
36475 1015 Acked Read 256 Done Done Done false
36476 1016 Acked Read 256 Done Done Done false
36477 1017 Acked Read 256 Done Done Done false
36478 1018 Acked Read 136 Done Done Done false
36479 1019 Acked Read 256 Done Done Done false
36480 STATES DS:0 DS:1 DS:2 TOTAL
36481 Sent 0 0 0 0
36482 Done 20 20 20 60
36483 Skipped 0 0 0 0
36484 Error 0 0 0 0
36485 Last Flush: None None None
36486 Downstairs last five completed:
36487 Upstairs last five completed: 1019 1018 1017 1016 1015
36488 CLIENT: Up:0 ds:20 act:3
36489 ----------------------------------------------------------------
36490 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
36491 Downstairs last five completed: 1020 1019 1018 1017 1016
36492 Upstairs last five completed: 1020 1019 1018 1017 1016
36493 CLIENT: Up:0 ds:0 act:3
36494 CLIENT: All crucible jobs finished, exiting program
36495 Feb 21 06:53:31.639 INFO accepted connection, remote_addr: 127.0.0.1:57926, local_addr: 127.0.0.1:9998
36496 Main task has work to do, go find it
36497 got dsc Shutdown
36498 Shutdown
36499 Feb 21 06:53:31.640 INFO request completed, latency_us: 171Shut it down
36500 , [8810] Disable keep_running
36501 response_code[8810] Got stop action so:false kr:false
36502 : 204, uri: /shutdown, method: POST, [8820] Disable keep_running
36503 req_id[8820] Got stop action so:false kr:false
36504 :[8830] Disable keep_running
36505 [8830] Got stop action so:false kr:false
36506 67099756-85c3-4118-bcc2-5070b037ace6, remote_addr: 127.0.0.1:57926, local_addr: 127.0.0.1:9998
36507 [8820] Exited with: ExitStatus(unix_wait_status(9))
36508 [8820][1] reports Exit
36509 [8830] Exited with: ExitStatus(unix_wait_status(9))
36510 [8830][2] reports Exit
36511 [8810] Exited with: ExitStatus(unix_wait_status(9))
36512 [8810][0] reports Exit
36513 February 21, 2025 at 06:53:36 AM UTC Test ends with 0