Max level shown:
1 starting February 25, 2025 at 05:18:07 PM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.2761817
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.024977393
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.024998073
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start ds: 8820
14 start ds: 8830
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8830
17 Starting downstairs at port 8820
18 Make output file at "/tmp/dsc/downstairs-8830.txt"
19 Make output file at "/tmp/dsc/downstairs-8820.txt"
20 Starting downstairs at port 8810
21 Make output file at "/tmp/dsc/downstairs-8810.txt"
22 [8830][2] initial start wait reports Starting
23 [8820][1] initial start wait reports Starting
24 [8810][0] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1198)
26 [8820][1] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1199)
28 [8810][0] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1200)
30 [8830][2] initial start wait reports Running
31 All downstairs are running
32 Feb 25 17:18:08.090 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Feb 25 17:18:13.154 INFO accepted connection, remote_addr: 127.0.0.1:43344, local_addr: 127.0.0.1:9998
36 Feb 25 17:18:13.157 INFO request completed, latency_us: 630, response_code: 200, uri: /regioninfo, method: GET, req_id: c9afa166-4337-4ba5-8c6c-b4520956a322, remote_addr: 127.0.0.1:43344, local_addr: 127.0.0.1:9998
37 Feb 25 17:18:13.158 INFO request completed, latency_us: 106, response_code: 200, uri: /regioncount, method: GET, req_id: f2063b6e-fdaa-48f6-8ce8-380b2123bd22, remote_addr: 127.0.0.1:43344, local_addr: 127.0.0.1:9998
382025-02-25T17:18:13.158ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-02-25T17:18:13.159ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Feb 25 17:18:13.159 INFO request completed, latency_us: 120, response_code: 200, uri: /port/cid/0, method: GET, req_id: 941c375e-cc0c-4983-80c3-9a0c47be644d, remote_addr: 127.0.0.1:43344, local_addr: 127.0.0.1:9998
41 Feb 25 17:18:13.160 INFO request completed, latency_us: 95, response_code: 200, uri: /port/cid/1, method: GET, req_id: cfede4c9-7828-490a-ac2a-4d73da07f5db, remote_addr: 127.0.0.1:43344, local_addr: 127.0.0.1:9998
42 Feb 25 17:18:13.160 INFO request completed, latency_us: 113, response_code: 200, uri: /port/cid/2, method: GET, req_id: cc360ace-8416-437b-8123-865549ad8acb, remote_addr: 127.0.0.1:43344, local_addr: 127.0.0.1:9998
432025-02-25T17:18:13.161ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-02-25T17:18:13.161ZINFOcrucible: Upstairs starts
452025-02-25T17:18:13.161ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "47aa4c863f974975b86a01cc6893b0f87d94aae1", git_commit_timestamp: "2025-02-25T16:27:18.000000000Z", git_branch: "renovate/thiserror-1.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-02-25T17:18:13.161ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-02-25T17:18:13.161ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-02-25T17:18:13.162ZINFOcrucible: Crucible 051f99c0-8a60-4d9c-bfdc-c0fa410942f6 has session id: b92e395c-4fdc-4ac6-b47e-b184953199d8 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
492025-02-25T17:18:13.162ZINFOcrucible: Upstairs opts: Upstairs UUID: 051f99c0-8a60-4d9c-bfdc-c0fa410942f6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
502025-02-25T17:18:13.162ZINFOcrucible: Crucible stats registered with UUID: 051f99c0-8a60-4d9c-bfdc-c0fa410942f6 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
512025-02-25T17:18:13.162ZWARNcrucible: could not find Downstairs address for Nexus session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
522025-02-25T17:18:13.164ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-02-25T17:18:13.164ZINFOcrucible: The guest has requested activation with gen:1
542025-02-25T17:18:13.164ZINFOcrucible: 051f99c0-8a60-4d9c-bfdc-c0fa410942f6 active request set session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
552025-02-25T17:18:13.164ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
562025-02-25T17:18:13.164ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
572025-02-25T17:18:13.164ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
582025-02-25T17:18:13.164ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
592025-02-25T17:18:13.164ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
602025-02-25T17:18:13.164ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
612025-02-25T17:18:13.164ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
622025-02-25T17:18:13.164ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
632025-02-25T17:18:13.164ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
642025-02-25T17:18:13.164ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58915) = io task client = 1 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
652025-02-25T17:18:13.164ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63879) = io task client = 0 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
662025-02-25T17:18:13.164ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42684) = io task client = 2 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
672025-02-25T17:18:13.166ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
682025-02-25T17:18:13.166ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
692025-02-25T17:18:13.166ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
702025-02-25T17:18:13.167ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
712025-02-25T17:18:13.167ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
722025-02-25T17:18:13.167ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
732025-02-25T17:18:13.168ZINFOcrucible: Waiting for more clients to be ready session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
742025-02-25T17:18:13.168ZINFOcrucible: Waiting for more clients to be ready session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
752025-02-25T17:18:13.168ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
762025-02-25T17:18:13.168ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
772025-02-25T17:18:13.168ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
782025-02-25T17:18:13.168ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
792025-02-25T17:18:13.168ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
802025-02-25T17:18:13.168ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
812025-02-25T17:18:13.168ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
822025-02-25T17:18:13.168ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
832025-02-25T17:18:13.168ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
842025-02-25T17:18:13.168ZINFOcrucible: Max found gen is 1 = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
852025-02-25T17:18:13.168ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
862025-02-25T17:18:13.168ZINFOcrucible: Next flush: 1 = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
872025-02-25T17:18:13.168ZINFOcrucible: All extents match = downstairs session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
882025-02-25T17:18:13.169ZINFOcrucible: No downstairs reconciliation required session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
892025-02-25T17:18:13.169ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
902025-02-25T17:18:13.169ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
91 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-02-25T17:18:13.169062839Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1204,"client":"2","":"Activate successful
92 downstairsWait for a query_work_queue command to finish before sending IO
93 ","session_id":"b92e395c-4fdc-4ac6-b47e-b184953199d8"}
942025-02-25T17:18:13.169ZINFOcrucible: All required reconciliation work is completed session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
952025-02-25T17:18:13.169ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
962025-02-25T17:18:13.169ZINFOcrucible: 051f99c0-8a60-4d9c-bfdc-c0fa410942f6 is now active with session: b92e395c-4fdc-4ac6-b47e-b184953199d8 session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
972025-02-25T17:18:13.169ZINFOcrucible: Set Active after no reconciliation session_id = b92e395c-4fdc-4ac6-b47e-b184953199d8
982025-02-25T17:18:13.169ZINFOcrucible: The guest has finished waiting for activation with:1
99 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
100 Fill test
101 Read and Verify all blocks (0..5000 range:false)
102 Wrote out file "/tmp/test_replay_verify.log"
103 CLIENT: Tests done. All submitted work has been ACK'd
104 ----------------------------------------------------------------
105 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
106 States: Active Active Active
107 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
108 1021 Acked Read 256 Done Done Done false
109 1022 Acked Read 256 Done Done Done false
110 1023 Acked Read 256 Done Done Done false
111 1024 Acked Read 256 Done Done Done false
112 1025 Acked Read 256 Done Done Done false
113 1026 Acked Read 256 Done Done Done false
114 1027 Acked Read 256 Done Done Done false
115 1028 Acked Read 256 Done Done Done false
116 1029 Acked Read 256 Done Done Done false
117 1030 Acked Read 256 Done Done Done false
118 1031 Acked Read 256 Done Done Done false
119 1032 Acked Read 256 Done Done Done false
120 1033 Acked Read 256 Done Done Done false
121 1034 Acked Read 256 Done Done Done false
122 1035 Acked Read 256 Done Done Done false
123 1036 Acked Read 256 Done Done Done false
124 1037 Acked Read 256 Done Done Done false
125 1038 Acked Read 256 Done Done Done false
126 1039 Acked Read 256 Done Done Done false
127 1040 Acked Read 136 Done Done Done false
128 STATES DS:0 DS:1 DS:2 TOTAL
129 Sent 0 0 0 0
130 Done 20 20 20 60
131 Skipped 0 0 0 0
132 Error 0 0 0 0
133 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
134 Downstairs last five completed: 1020 1019 1018 1017 1016
135 Upstairs last five completed: 1040 1039 1038 1037 1036
136 CLIENT: Up:0 ds:20 act:3
137 ----------------------------------------------------------------
138 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
139 Downstairs last five completed: 1041 1040 1039 1038 1037
140 Upstairs last five completed: 1041 1040 1039 1038 1037
141 CLIENT: Up:0 ds:0 act:3
142 CLIENT: All crucible jobs finished, exiting program
1432025-02-25T17:18:17.715ZWARNcrucible: Guest handle has been dropped
144 Replay loop starts now February 25, 2025 at 05:18:17 PM UTC
145 Feb 25 17:18:17.790 INFO accepted connection, remote_addr: 127.0.0.1:46253, local_addr: 127.0.0.1:9998
146 Feb 25 17:18:17.791 INFO request completed, latency_us: 171, response_code: 200, uri: /regioninfo, method: GET, req_id: 50e3889a-1443-44e6-b087-13e5d9e30a21, remote_addr: 127.0.0.1:46253, local_addr: 127.0.0.1:9998
1472025-02-25T17:18:17.792ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
148 Feb 25 17:18:17.792 INFO request completed, latency_us: 99, response_code: 200, uri: /regioncount, method: GET, req_id: 2b33ce31-2941-42f9-9ada-3d6ecbcf0c61, remote_addr: 127.0.0.1:46253, local_addr: 127.0.0.1:9998
1492025-02-25T17:18:17.792ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
150 Feb 25 17:18:17.793 INFO request completed, latency_us: 135, response_code: 200, uri: /port/cid/0, method: GET, req_id: 5479a36c-ce09-44ef-af41-7a44ac3b056c, remote_addr: 127.0.0.1:46253, local_addr: 127.0.0.1:9998
151 Feb 25 17:18:17.793 INFO request completed, latency_us: 111, response_code: 200, uri: /port/cid/1, method: GET, req_id: 7011702a-e2aa-4f68-9438-51129b91811e, remote_addr: 127.0.0.1:46253, local_addr: 127.0.0.1:9998
152 Feb 25 17:18:17.794 INFO request completed, latency_us: 96, response_code: 200, uri: /port/cid/2, method: GET, req_id: 41d94ecb-8714-4b8f-b5c9-09c7a3ecd2eb, remote_addr: 127.0.0.1:46253, local_addr: 127.0.0.1:9998
1532025-02-25T17:18:17.794ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1542025-02-25T17:18:17.794ZINFOcrucible: Upstairs starts
1552025-02-25T17:18:17.794ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "47aa4c863f974975b86a01cc6893b0f87d94aae1", git_commit_timestamp: "2025-02-25T16:27:18.000000000Z", git_branch: "renovate/thiserror-1.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1562025-02-25T17:18:17.794ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1572025-02-25T17:18:17.794ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1582025-02-25T17:18:17.795ZINFOcrucible: Crucible 933aae41-749e-42f6-ae8e-398fc5b1a5e9 has session id: df7322ab-c489-4662-a11c-3ea4e180b6a3 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1592025-02-25T17:18:17.795ZINFOcrucible: Upstairs opts: Upstairs UUID: 933aae41-749e-42f6-ae8e-398fc5b1a5e9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1602025-02-25T17:18:17.795ZINFOcrucible: Crucible stats registered with UUID: 933aae41-749e-42f6-ae8e-398fc5b1a5e9 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1612025-02-25T17:18:17.795ZWARNcrucible: could not find Downstairs address for Nexus session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1622025-02-25T17:18:17.796ZINFOcrucible: Downstairs in volume = 3 task = crutest
1632025-02-25T17:18:17.796ZINFOcrucible: The guest has requested activation with gen:2
1642025-02-25T17:18:17.796ZINFOcrucible: 933aae41-749e-42f6-ae8e-398fc5b1a5e9 active request set session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1652025-02-25T17:18:17.796ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1662025-02-25T17:18:17.796ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1672025-02-25T17:18:17.796ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1682025-02-25T17:18:17.796ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1692025-02-25T17:18:17.796ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1702025-02-25T17:18:17.797ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1712025-02-25T17:18:17.797ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1722025-02-25T17:18:17.797ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1732025-02-25T17:18:17.797ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1742025-02-25T17:18:17.797ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56167) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1752025-02-25T17:18:17.797ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53685) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1762025-02-25T17:18:17.797ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53483) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1772025-02-25T17:18:17.798ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1782025-02-25T17:18:17.798ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1792025-02-25T17:18:17.798ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1802025-02-25T17:18:17.798ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1812025-02-25T17:18:17.798ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1822025-02-25T17:18:17.798ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1832025-02-25T17:18:17.799ZINFOcrucible: Waiting for more clients to be ready session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1842025-02-25T17:18:17.799ZINFOcrucible: Waiting for more clients to be ready session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1852025-02-25T17:18:17.799ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1862025-02-25T17:18:17.799ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1872025-02-25T17:18:17.799ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1882025-02-25T17:18:17.799ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1892025-02-25T17:18:17.799ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1902025-02-25T17:18:17.799ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1912025-02-25T17:18:17.799ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1922025-02-25T17:18:17.799ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1932025-02-25T17:18:17.799ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1942025-02-25T17:18:17.799ZINFOcrucible: Max found gen is 2 = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
195 {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30Activate successful
196 ,Wait for a query_work_queue command to finish before sending IO
197 "time":"2025-02-25T17:18:17.799747086Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"downstairs","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
1982025-02-25T17:18:17.799ZINFOcrucible: Next flush: 2 = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1992025-02-25T17:18:17.799ZINFOcrucible: All extents match = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2002025-02-25T17:18:17.799ZINFOcrucible: No downstairs reconciliation required session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2012025-02-25T17:18:17.799ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2022025-02-25T17:18:17.799ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2032025-02-25T17:18:17.799ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2042025-02-25T17:18:17.799ZINFOcrucible: All required reconciliation work is completed session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2052025-02-25T17:18:17.799ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
206 {"msg":"933aae41-749e-42f6-ae8e-398fc5b1a5e9 is now active with session: df7322ab-c489-4662-a11c-3ea4e180b6a3","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
207 ,"time":"2025-02-25T17:18:17.799927881Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
2082025-02-25T17:18:17.799ZINFOcrucible: Set Active after no reconciliation session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2092025-02-25T17:18:17.799ZINFOcrucible: The guest has finished waiting for activation with:2
210 Loading write count information from file "/tmp/test_replay_verify.log"
211 Feb 25 17:18:17.834 INFO accepted connection, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
212 Main task has work to do, go find it
213 got dsc Stop(0)
214 stop 0
215 Feb 25 17:18:17.834 INFO request completed, latency_us: 251, response_code: 204, uri: /stop/cid/0, method:[8810] Got stop action so:false kr:false
216 POST, req_id: ff9c663d-5444-498c-a363-fc377f54beeb, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
2172025-02-25T17:18:17.835ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2182025-02-25T17:18:17.835ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2192025-02-25T17:18:17.835ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2202025-02-25T17:18:17.835ZINFOcrucible: client task is exiting = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
221 Feb 25 17:18:17.835 INFO request completed, latency_us: 173, response_code: 200, uri: /state/cid/0, method: GET, req_id: ac779bd3-4715-4e7f-81aa-472361d76905, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
222 [8810] Exited with: ExitStatus(unix_wait_status(9))
223 [8810][0] reports Exit
224 Feb 25 17:18:21.838 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/0, method: GET, req_id: 8a9d615e-4221-4fcf-8ac5-e112ee664217, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
225 001/300 Read block 3894 len 16384
226 002/300 Read block 2222 len 8192
227 003/300 Read block 4909 len 36864
228 004/300 Write block 2765 len 24576 data: 2 2 2 2 2 2
229 005/300 Write block 1592 len 40960 data: 2 2 2 2 2 2 2 2 2 2
230 006/300 Read block 4447 len 8192
231 007/300 Write block 3682 len 32768 data: 2 2 2 2 2 2 2 2
232 008/300 Read block 96 len 24576
233 009/300 Read block 4324 len 28672
234 010/300 Read block 514 len 8192
235 011/300 Read block 1086 len 12288
236 012/300 Write block 932 len 12288 data: 2 2 2
237 013/300 Read block 1280 len 36864
238 014/300 Write block 1070 len 40960 data: 2 2 2 2 2 2 2 2 2 2
239 015/300 Write block 2344 len 16384 data: 2 2 2 2
240 016/300 Read block 4090 len 36864
241 017/300 Write block 1189 len 8192 data: 2 2
242 018/300 Read block 2414 len 20480
243 019/300 Read block 1394 len 12288
244 020/300 Write block 4355 len 8192 data: 2 2
245 021/300 Write block 4557 len 36864 data: 2 2 2 2 2 2 2 2 2
246 022/300 Write block 2694 len 36864 data: 2 2 2 2 2 2 2 2 2
247 023/300 Read block 4494 len 40960
248 024/300 Write block 748 len 32768 data: 2 2 2 2 2 2 2 2
249 025/300 Write block 2510 len 28672 data: 2 2 2 2 2 2 2
250 026/300 Read block 1540 len 20480
251 027/300 Write block 4176 len 20480 data: 2 2 2 2 2
252 028/300 Read block 744 len 20480
253 029/300 Write block 864 len 16384 data: 2 2 2 2
254 030/300 Write block 3258 len 8192 data: 2 2
255 031/300 Read block 4874 len 12288
256 032/300 Read block 3037 len 40960
257 033/300 Write block 4882 len 16384 data: 2 2 2 2
258 034/300 Write block 3930 len 8192 data: 2 2
259 035/300 Read block 66 len 24576
260 036/300 Read block 2434 len 20480
261 037/300 Read block 1164 len 20480
262 038/300 Write block 1495 len 12288 data: 2 2 2
263 039/300 Read block 4154 len 32768
264 040/300 Write block 2082 len 28672 data: 2 2 2 2 2 2 2
265 041/300 Flush
266 042/300 Read block 3073 len 28672
267 043/300 Read block 8 len 4096
268 044/300 Write block 749 len 16384 data: 3 3 3 3
269 045/300 Write block 3717 len 40960 data: 2 2 2 2 2 2 2 2 2 2
270 046/300 Read block 4907 len 36864
271 047/300 Read block 665 len 32768
272 048/300 Read block 2260 len 4096
273 049/300 Write block 4299 len 12288 data: 2 2 2
274 050/300 Read block 4689 len 12288
275 051/300 Write block 265 len 40960 data: 2 2 2 2 2 2 2 2 2 2
276 052/300 Write block 2217 len 28672 data: 2 2 2 2 2 2 2
277 053/300 Write block 3584 len 28672 data: 2 2 2 2 2 2 2
278 054/300 Read block 773 len 8192
279 055/300 Read block 1977 len 12288
280 056/300 Read block 4894 len 12288
281 057/300 Write block 405 len 28672 data: 2 2 2 2 2 2 2
282 058/300 Write block 2318 len 8192 data: 2 2
283 059/300 Read block 4851 len 20480
284 060/300 Read block 3093 len 20480
285 061/300 Read block 2870 len 4096
286 062/300 Write block 1079 len 32768 data: 3 2 2 2 2 2 2 2
287 063/300 Read block 2049 len 20480
288 064/300 Write block 3698 len 32768 data: 2 2 2 2 2 2 2 2
289 065/300 Write block 1945 len 40960 data: 2 2 2 2 2 2 2 2 2 2
290 066/300 Write block 2111 len 16384 data: 2 2 2 2
291 067/300 Write block 4531 len 4096 data: 2
292 068/300 Read block 601 len 40960
293 069/300 Write block 2598 len 16384 data: 2 2 2 2
294 070/300 Write block 4384 len 4096 data: 2
295 071/300 Read block 4521 len 4096
296 072/300 Write block 1176 len 28672 data: 2 2 2 2 2 2 2
297 073/300 Write block 3091 len 40960 data: 2 2 2 2 2 2 2 2 2 2
298 074/300 Write block 2923 len 12288 data: 2 2 2
299 075/300 Read block 403 len 32768
300 076/300 Read block 1643 len 32768
301 077/300 Read block 2032 len 8192
302 078/300 Write block 1296 len 24576 data: 2 2 2 2 2 2
303 079/300 Write block 832 len 36864 data: 2 2 2 2 2 2 2 2 2
304 080/300 Read block 2643 len 8192
305 081/300 Read block 1809 len 4096
306 082/300 Read block 512 len 16384
307 083/300 Write block 3669 len 32768 data: 2 2 2 2 2 2 2 2
308 084/300 Read block 3084 len 4096
309 085/300 Write block 3792 len 32768 data: 2 2 2 2 2 2 2 2
310 086/300 Write block 2857 len 16384 data: 2 2 2 2
311 087/300 Read block 2976 len 12288
312 088/300 Write block 3080 len 20480 data: 2 2 2 2 2
313 089/300 Read block 4029 len 36864
314 090/300 Write block 4212 len 4096 data: 2
315 091/300 Write block 3815 len 36864 data: 2 2 2 2 2 2 2 2 2
316 092/300 Write block 2607 len 16384 data: 2 2 2 2
317 093/300 Read block 2949 len 24576
318 094/300 Read block 941 len 16384
319 095/300 Read block 2136 len 40960
320 096/300 Read block 377 len 28672
321 097/300 Write block 4606 len 12288 data: 2 2 2
322 098/300 Write block 1450 len 12288 data: 2 2 2
323 099/300 Write block 1700 len 28672 data: 2 2 2 2 2 2 2
324 100/300 Write block 1025 len 40960 data: 2 2 2 2 2 2 2 2 2 2
325 101/300 Read block 4562 len 24576
326 102/300 Read block 4682 len 40960
327 103/300 Read block 4353 len 36864
328 104/300 Read block 1184 len 40960
329 105/300 Read block 4415 len 4096
330 106/300 Read block 644 len 40960
331 107/300 Read block 853 len 4096
332 108/300 Write block 1679 len 16384 data: 2 2 2 2
333 109/300 Write block 2355 len 16384 data: 2 2 2 2
334 110/300 Write block 4916 len 12288 data: 2 2 2
335 111/300 Read block 3490 len 40960
336 112/300 Read block 1759 len 40960
337 113/300 Flush
338 114/300 Read block 1089 len 20480
339 115/300 Read block 1818 len 28672
340 116/300 Read block 3608 len 32768
341 117/300 Write block 2523 len 12288 data: 2 2 2
342 118/300 Flush
343 119/300 Write block 2065 len 8192 data: 2 2
344 120/300 Read block 4743 len 4096
345 121/300 Flush
346 122/300 Read block 1141 len 20480
347 123/300 Read block 4709 len 40960
348 124/300 Write block 193 len 24576 data: 2 2 2 2 2 2
349 125/300 Write block 4809 len 32768 data: 2 2 2 2 2 2 2 2
350 126/300 Read block 821 len 32768
351 127/300 Flush
352 128/300 Write block 484 len 24576 data: 2 2 2 2 2 2
353 129/300 Write block 1196 len 12288 data: 2 2 2
354 130/300 Read block 170 len 40960
355 131/300 Read block 3845 len 12288
356 132/300 Read block 2263 len 8192
357 133/300 Write block 966 len 12288 data: 2 2 2
358 134/300 Read block 3979 len 20480
359 135/300 Flush
360 136/300 Read block 1649 len 40960
361 137/300 Flush
362 138/300 Read block 1136 len 40960
363 139/300 Read block 3066 len 32768
364 140/300 Read block 1142 len 4096
365 141/300 Flush
366 142/300 Write block 1667 len 40960 data: 2 2 2 2 2 2 2 2 2 2
367 143/300 Read block 112 len 24576
368 144/300 Flush
369 145/300 Read block 1016 len 8192
370 146/300 Read block 150 len 32768
371 147/300 Flush
372 148/300 Read block 403 len 12288
373 149/300 Read block 1436 len 20480
374 150/300 Read block 4393 len 12288
375 151/300 Write block 4202 len 24576 data: 2 2 2 2 2 2
376 152/300 Read block 2173 len 16384
377 153/300 Read block 1427 len 20480
378 154/300 Read block 1055 len 40960
379 155/300 Write block 4962 len 36864 data: 2 2 2 2 2 2 2 2 2
380 156/300 Write block 3284 len 16384 data: 2 2 2 2
381 157/300 Flush
382 158/300 Read block 2258 len 4096
383 159/300 Read block 3596 len 32768
384 160/300 Read block 320 len 4096
385 161/300 Flush
386 162/300 Write block 4889 len 12288 data: 2 2 2
387 163/300 Read block 3944 len 4096
388 164/300 Write block 4336 len 20480 data: 2 2 2 2 2
389 165/300 Write block 287 len 4096 data: 2
390 166/300 Flush
391 167/300 Read block 3390 len 20480
392 168/300 Write block 176 len 40960 data: 2 2 2 2 2 2 2 2 2 2
393 169/300 Read block 4860 len 20480
394 170/300 Read block 3496 len 24576
395 171/300 Read block 157 len 16384
396 172/300 Read block 4070 len 8192
397 173/300 Read block 2369 len 40960
398 174/300 Read block 2084 len 36864
399 175/300 Read block 3648 len 40960
400 176/300 Read block 3726 len 12288
401 177/300 Read block 3244 len 36864
402 178/300 Write block 2924 len 16384 data: 3 3 2 2
403 179/300 Read block 4328 len 16384
404 180/300 Write block 3897 len 12288 data: 2 2 2
405 181/300 Write block 1166 len 28672 data: 2 2 2 2 2 2 2
406 182/300 Read block 2167 len 12288
407 183/300 Read block 3812 len 4096
408 184/300 Write block 2332 len 28672 data: 2 2 2 2 2 2 2
409 185/300 Read block 3356 len 12288
410 186/300 Read block 1396 len 28672
411 187/300 Read block 3883 len 32768
412 188/300 Write block 4661 len 24576 data: 2 2 2 2 2 2
413 189/300 Write block 247 len 40960 data: 2 2 2 2 2 2 2 2 2 2
414 190/300 Read block 4751 len 16384
415 191/300 Write block 68 len 36864 data: 2 2 2 2 2 2 2 2 2
416 192/300 Write block 4316 len 4096 data: 2
417 193/300 Write block 1145 len 24576 data: 2 2 2 2 2 2
418 194/300 Read block 1728 len 40960
419 195/300 Write block 150 len 4096 data: 2
420 196/300 Write block 4421 len 20480 data: 2 2 2 2 2
421 197/300 Write block 3918 len 32768 data: 2 2 2 2 2 2 2 2
422 198/300 Read block 263 len 40960
423 199/300 Write block 2946 len 12288 data: 2 2 2
424 200/300 Flush
425 201/300 Read block 4182 len 4096
426 202/300 Flush
427 203/300 Read block 1336 len 40960
428 204/300 Flush
429 205/300 Write block 4762 len 40960 data: 2 2 2 2 2 2 2 2 2 2
430 206/300 Read block 1565 len 32768
431 207/300 Read block 3330 len 8192
432 208/300 Read block 3690 len 40960
433 209/300 Flush
434 210/300 Write block 1257 len 36864 data: 2 2 2 2 2 2 2 2 2
435 211/300 Read block 525 len 24576
436 212/300 Read block 3836 len 8192
437 213/300 Read block 1316 len 40960
438 214/300 Read block 2894 len 16384
439 215/300 Write block 1539 len 16384 data: 2 2 2 2
440 216/300 Flush
441 217/300 Read block 1253 len 8192
442 218/300 Write block 4312 len 12288 data: 2 2 2
443 219/300 Write block 2131 len 20480 data: 2 2 2 2 2
444 220/300 Flush
445 221/300 Write block 3396 len 28672 data: 2 2 2 2 2 2 2
446 222/300 Flush
447 223/300 Read block 3534 len 32768
448 224/300 Write block 2459 len 16384 data: 2 2 2 2
449 225/300 Write block 1047 len 16384 data: 2 2 2 2
450 226/300 Read block 1804 len 40960
451 227/300 Write block 2517 len 24576 data: 2 2 2 2 2 2
452 228/300 Write block 294 len 4096 data: 2
453 229/300 Read block 3537 len 32768
454 230/300 Read block 1061 len 16384
455 231/300 Write block 4784 len 28672 data: 2 2 2 2 2 2 2
456 232/300 Write block 4143 len 16384 data: 2 2 2 2
457 233/300 Read block 2296 len 28672
458 234/300 Read block 4364 len 20480
459 235/300 Write block 2621 len 24576 data: 2 2 2 2 2 2
460 236/300 Read block 1759 len 4096
461 237/300 Read block 1194 len 8192
462 238/300 Read block 2078 len 8192
463 239/300 Read block 1558 len 20480
464 240/300 Read block 1619 len 8192
465 241/300 Write block 3057 len 32768 data: 2 2 2 2 2 2 2 2
466 242/300 Read block 4069 len 12288
467 243/300 Write block 3762 len 12288 data: 2 2 2
468 244/300 Read block 4606 len 12288
469 245/300 Write block 4263 len 32768 data: 2 2 2 2 2 2 2 2
470 246/300 Read block 2917 len 28672
471 247/300 Read block 162 len 36864
472 248/300 Read block 65 len 28672
473 249/300 Read block 2331 len 24576
474 250/300 Write block 4015 len 32768 data: 2 2 2 2 2 2 2 2
475 251/300 Read block 3499 len 12288
476 252/300 Write block 4170 len 8192 data: 2 2
477 253/300 Read block 3164 len 32768
478 254/300 Read block 4763 len 8192
479 255/300 Write block 2405 len 4096 data: 2
480 256/300 Write block 3819 len 20480 data: 3 3 3 3 3
481 257/300 Write block 1234 len 20480 data: 2 2 2 2 2
482 258/300 Read block 1965 len 32768
483 259/300 Read block 2735 len 12288
484 260/300 Read block 4849 len 8192
485 261/300 Read block 364 len 32768
486 262/300 Read block 2398 len 20480
487 263/300 Read block 2567 len 12288
488 264/300 Read block 2633 len 28672
489 265/300 Read block 770 len 4096
490 266/300 Read block 2926 len 32768
491 267/300 Read block 3298 len 12288
492 268/300 Write block 1439 len 40960 data: 2 2 2 2 2 2 2 2 2 2
493 269/300 Write block 1836 len 36864 data: 2 2 2 2 2 2 2 2 2
494 270/300 Write block 3510 len 24576 data: 2 2 2 2 2 2
495 271/300 Read block 134 len 36864
496 272/300 Write block 304 len 28672 data: 2 2 2 2 2 2 2
497 273/300 Read block 1352 len 24576
498 274/300 Read block 4187 len 8192
499 275/300 Read block 174 len 32768
500 276/300 Write block 4750 len 40960 data: 2 2 2 2 2 2 2 2 2 2
501 277/300 Write block 1058 len 24576 data: 2 2 2 2 2 2
502 278/300 Read block 3549 len 24576
503 279/300 Read block 3552 len 40960
504 280/300 Read block 2596 len 32768
505 281/300 Read block 2521 len 20480
506 282/300 Write block 3484 len 40960 data: 2 2 2 2 2 2 2 2 2 2
507 283/300 Write block 466 len 8192 data: 2 2
508 284/300 Flush
509 285/300 Write block 1041 len 24576 data: 2 2 2 2 2 2
510 286/300 Write block 4516 len 20480 data: 2 2 2 2 2
511 287/300 Read block 1729 len 20480
512 288/300 Read block 4392 len 24576
513 289/300 Read block 2276 len 28672
514 290/300 Write block 2196 len 36864 data: 2 2 2 2 2 2 2 2 2
515 291/300 Write block 1307 len 32768 data: 2 2 2 2 2 2 2 2
516 292/300 Flush
517 293/300 Read block 2322 len 16384
518 294/300 Flush
519 295/300 Write block 1823 len 40960 data: 2 2 2 2 2 2 2 2 2 2
520 296/300 Read block 4553 len 36864
521 297/300 Read block 4117 len 4096
522 298/300 Read block 4947 len 28672
523 299/300 Read block 988 len 20480
524 300/300 Write block 107 len 16384 data: 2 2 2 2
525 301/300 Read block 408 len 36864
526 Main task has work to do, go find it
527 got dsc Start(0)
528 start 0
529 Feb 25 17:18:22.101 INFO request completed, latency_us: 253, response_code: 204, uri: /start/cid/0, method: POST, req_id: 594f300c-c1e5-4026-b7f0-d793f4d2d090, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
530 Replay: started 0, returned:Ok(())
531 ----------------------------------------------------------------
532 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
533 States: Offline Active Active
534 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
535 1000 Acked Read 4 Sent Done Done false
536 1001 Acked Read 2 Sent Done Done false
537 1002 Acked Read 9 Sent Done Done false
538 1003 Acked Write 6 Sent Done Done false
539 1004 Acked Write 10 Sent Done Done false
540 1005 Acked Read 2 Sent Done Done false
541 1006 Acked Write 8 Sent Done Done false
542 1007 Acked Read 6 Sent Done Done false
543 1008 Acked Read 7 Sent Done Done false
544 1009 Acked Read 2 Sent Done Done false
545 1010 Acked Read 3 Sent Done Done false
546 1011 Acked Write 3 Sent Done Done false
547 1012 Acked Read 9 Sent Done Done false
548 1013 Acked Write 10 Sent Done Done false
549 1014 Acked Write 4 Sent Done Done false
550 1015 Acked Read 9 Sent Done Done false
551 1016 Acked Write 2 Sent Done Done false
552 1017 Acked Read 5 Sent Done Done false
553 1018 Acked Read 3 Sent Done Done false
554 1019 Acked Write 2 Sent Done Done false
555 1020 Acked Write 9 Sent Done Done false
556 1021 Acked Write 9 Sent Done Done false
557 1022 Acked Read 10 Sent Done Done false
558 1023 Acked Write 8 Sent Done Done false
559 1024 Acked Write 7 Sent Done Done false
560 1025 Acked Read 5 Sent Done Done false
561 1026 Acked Write 5 Sent Done Done false
562 1027 Acked Read 5 Sent Done Done false
563 1028 Acked Write 4 Sent Done Done false
564 1029 Acked Write 2 Sent Done Done false
565 1030 Acked Read 3 Sent Done Done false
566 1031 Acked Read 10 Sent Done Done false
567 1032 Acked Write 4 Sent Done Done false
568 1033 Acked Write 2 Sent Done Done false
569 1034 Acked Read 6 Sent Done Done false
570 1035 Acked Read 5 Sent Done Done false
571 1036 Acked Read 5 Sent Done Done false
572 1037 Acked Write 3 Sent Done Done false
573 1038 Acked Read 8 Sent Done Done false
574 1039 Acked Write 7 Sent Done Done false
575 1040 Acked Flush 0 Sent Done Done false
576 1041 Acked Read 7 Sent Done Done false
577 1042 Acked Read 1 Sent Done Done false
578 1043 Acked Write 4 Sent Done Done false
579 1044 Acked Write 10 Sent Done Done false
580 1045 Acked Read 9 Sent Done Done false
581 1046 Acked Read 8 Sent Done Done false
582 1047 Acked Read 1 Sent Done Done false
583 1048 Acked Write 3 Sent Done Done false
584 1049 Acked Read 3 Sent Done Done false
585 1050 Acked Write 10 Sent Done Done false
586 1051 Acked Write 7 Sent Done Done false
587 1052 Acked Write 7 Sent Done Done false
588 1053 Acked Read 2 Sent Done Done false
589 1054 Acked Read 3 Sent Done Done false
590 1055 Acked Read 3 Sent Done Done false
591 1056 Acked Write 7 Sent Done Done false
592 1057 Acked Write 2 Sent Done Done false
593 1058 Acked Read 5 Sent Done Done false
594 1059 Acked Read 5 Sent Done Done false
595 1060 Acked Read 1 Sent Done Done false
596 1061 Acked Write 8 Sent Done Done false
597 1062 Acked Read 5 Sent Done Done false
598 1063 Acked Write 8 Sent Done Done false
599 1064 Acked Write 10 Sent Done Done false
600 1065 Acked Write 4 Sent Done Done false
601 1066 Acked Write 1 Sent Done Done false
602 1067 Acked Read 10 Sent Done Done false
603 1068 Acked Write 4 Sent Done Done false
604 1069 Acked Write 1 Sent Done Done false
605 1070 Acked Read 1 Sent Done Done false
606 1071 Acked Write 7 Sent Done Done false
607 1072 Acked Write 10 Sent Done Done false
608 1073 Acked Write 3 Sent Done Done false
609 1074 Acked Read 8 Sent Done Done false
610 1075 Acked Read 8 Sent Done Done false
611 1076 Acked Read 2 Sent Done Done false
612 1077 Acked Write 6 Sent Done Done false
613 1078 Acked Write 9 Sent Done Done false
614 1079 Acked Read 2 Sent Done Done false
615 1080 Acked Read 1 Sent Done Done false
616 1081 Acked Read 4 Sent Done Done false
617 1082 Acked Write 8 Sent Done Done false
618 1083 Acked Read 1 Sent Done Done false
619 1084 Acked Write 8 Sent Done Done false
620 1085 Acked Write 4 Sent Done Done false
621 1086 Acked Read 3 Sent Done Done false
622 1087 Acked Write 5 Sent Done Done false
623 1088 Acked Read 9 Sent Done Done false
624 1089 Acked Write 1 Sent Done Done false
625 1090 Acked Write 9 Sent Done Done false
626 1091 Acked Write 4 Sent Done Done false
627 1092 Acked Read 6 Sent Done Done false
628 1093 Acked Read 4 Sent Done Done false
629 1094 Acked Read 10 Sent Done Done false
630 1095 Acked Read 7 Sent Done Done false
631 1096 Acked Write 3 Sent Done Done false
632 1097 Acked Write 3 Sent Done Done false
633 1098 Acked Write 7 Sent Done Done false
634 1099 Acked Write 10 Sent Done Done false
635 1100 Acked Read 6 Sent Done Done false
636 1101 Acked Read 10 Sent Done Done false
637 1102 Acked Read 9 Sent Done Done false
638 1103 Acked Read 10 Sent Done Done false
639 1104 Acked Read 1 Sent Done Done false
640 1105 Acked Read 10 Sent Done Done false
641 1106 Acked Read 1 Sent Done Done false
642 1107 Acked Write 4 Sent Done Done false
643 1108 Acked Write 4 Sent Done Done false
644 1109 Acked Write 3 Sent Done Done false
645 1110 Acked Read 10 Sent Done Done false
646 1111 Acked Read 10 Sent Done Done false
647 1112 Acked Flush 0 Sent Done Done false
648 1113 Acked Read 5 Sent Done Done false
649 1114 Acked Read 7 Sent Done Done false
650 1115 Acked Read 8 Sent Done Done false
651 1116 Acked Write 3 Sent Done Done false
652 1117 Acked Flush 0 Sent Done Done false
653 1118 Acked Write 2 Sent Done Done false
654 1119 Acked Read 1 Sent Done Done false
655 1120 Acked Flush 0 Sent Done Done false
656 1121 Acked Read 5 Sent Done Done false
657 1122 Acked Read 10 Sent Done Done false
658 1123 Acked Write 6 Sent Done Done false
659 1124 Acked Write 8 Sent Done Done false
660 1125 Acked Read 8 Sent Done Done false
661 1126 Acked Flush 0 Sent Done Done false
662 1127 Acked Write 6 Sent Done Done false
663 1128 Acked Write 3 Sent Done Done false
664 1129 Acked Read 10 Sent Done Done false
665 1130 Acked Read 3 Sent Done Done false
666 1131 Acked Read 2 Sent Done Done false
667 1132 Acked Write 3 Sent Done Done false
668 1133 Acked Read 5 Sent Done Done false
669 1134 Acked Flush 0 Sent Done Done false
670 1135 Acked Read 10 Sent Done Done false
671 1136 Acked Flush 0 Sent Done Done false
672 1137 Acked Read 10 Sent Done Done false
673 1138 Acked Read 8 Sent Done Done false
674 1139 Acked Read 1 Sent Done Done false
675 1140 Acked Flush 0 Sent Done Done false
676 1141 Acked Write 10 Sent Done Done false
677 1142 Acked Read 6 Sent Done Done false
678 1143 Acked Flush 0 Sent Done Done false
679 1144 Acked Read 2 Sent Done Done false
680 1145 Acked Read 8 Sent Done Done false
681 1146 Acked Flush 0 Sent Done Done false
682 1147 Acked Read 3 Sent Done Done false
683 1148 Acked Read 5 Sent Done Done false
684 1149 Acked Read 3 Sent Done Done false
685 1150 Acked Write 6 Sent Done Done false
686 1151 Acked Read 4 Sent Done Done false
687 1152 Acked Read 5 Sent Done Done false
688 1153 Acked Read 10 Sent Done Done false
689 1154 Acked Write 9 Sent Done Done false
690 1155 Acked Write 4 Sent Done Done false
691 1156 Acked Flush 0 Sent Done Done false
692 1157 Acked Read 1 Sent Done Done false
693 1158 Acked Read 8 Sent Done Done false
694 1159 Acked Read 1 Sent Done Done false
695 1160 Acked Flush 0 Sent Done Done false
696 1161 Acked Write 3 Sent Done Done false
697 1162 Acked Read 1 Sent Done Done false
698 1163 Acked Write 5 Sent Done Done false
699 1164 Acked Write 1 Sent Done Done false
700 1165 Acked Flush 0 Sent Done Done false
701 1166 Acked Read 5 Sent Done Done false
702 1167 Acked Write 10 Sent Done Done false
703 1168 Acked Read 5 Sent Done Done false
704 1169 Acked Read 6 Sent Done Done false
705 1170 Acked Read 4 Sent Done Done false
706 1171 Acked Read 2 Sent Done Done false
707 1172 Acked Read 10 Sent Done Done false
708 1173 Acked Read 9 Sent Done Done false
709 1174 Acked Read 10 Sent Done Done false
710 1175 Acked Read 3 Sent Done Done false
711 1176 Acked Read 9 Sent Done Done false
712 1177 Acked Write 4 Sent Done Done false
713 1178 Acked Read 4 Sent Done Done false
714 1179 Acked Write 3 Sent Done Done false
715 1180 Acked Write 7 Sent Done Done false
716 1181 Acked Read 3 Sent Done Done false
717 1182 Acked Read 1 Sent Done Done false
718 1183 Acked Write 7 Sent Done Done false
719 1184 Acked Read 3 Sent Done Done false
720 1185 Acked Read 7 Sent Done Done false
721 1186 Acked Read 8 Sent Done Done false
722 1187 Acked Write 6 Sent Done Done false
723 1188 Acked Write 10 Sent Done Done false
724 1189 Acked Read 4 Sent Done Done false
725 1190 Acked Write 9 Sent Done Done false
726 1191 Acked Write 1 Sent Done Done false
727 1192 Acked Write 6 Sent Done Done false
728 1193 Acked Read 10 Sent Done Done false
729 1194 Acked Write 1 Sent Done Done false
730 1195 Acked Write 5 Sent Done Done false
731 1196 Acked Write 8 Sent Done Done false
732 1197 Acked Read 10 Sent Done Done false
733 1198 Acked Write 3 Sent Done Done false
734 1199 Acked Flush 0 Sent Done Done false
735 1200 Acked Read 1 Sent Done Done false
736 1201 Acked Flush 0 Sent Done Done false
737 1202 Acked Read 10 Sent Done Done false
738 1203 Acked Flush 0 Sent Done Done false
739 1204 Acked Write 10 Sent Done Done false
740 1205 Acked Read 8 Sent Done Done false
741 1206 Acked Read 2 Sent Done Done false
742 1207 Acked Read 10 Sent Done Done false
743 1208 Acked Flush 0 Sent Done Done false
744 1209 Acked Write 9 Sent Done Done false
745 1210 Acked Read 6 Sent Done Done false
746 1211 Acked Read 2 Sent Done Done false
747 1212 Acked Read 10 Sent Done Done false
748 1213 Acked Read 4 Sent Done Done false
749 1214 Acked Write 4 Sent Done Done false
750 1215 Acked Flush 0 Sent Done Done false
751 1216 Acked Read 2 Sent Done Done false
752 1217 Acked Write 3 Sent Done Done false
753 1218 Acked Write 5 Sent Done Done false
754 1219 Acked Flush 0 Sent Done Done false
755 1220 Acked Write 7 Sent Done Done false
756 1221 Acked Flush 0 Sent Done Done false
757 1222 Acked Read 8 Sent Done Done false
758 1223 Acked Write 4 Sent Done Done false
759 1224 Acked Write 4 Sent Done Done false
760 1225 Acked Read 10 Sent Done Done false
761 1226 Acked Write 6 Sent Done Done false
762 1227 Acked Write 1 Sent Done Done false
763 1228 Acked Read 8 Sent Done Done false
764 1229 Acked Read 4 Sent Done Done false
765 1230 Acked Write 7 Sent Done Done false
766 1231 Acked Write 4 Sent Done Done false
767 1232 Acked Read 7 Sent Done Done false
768 1233 Acked Read 5 Sent Done Done false
769 1234 Acked Write 6 Sent Done Done false
770 1235 Acked Read 1 Sent Done Done false
771 1236 Acked Read 2 Sent Done Done false
772 1237 Acked Read 2 Sent Done Done false
773 1238 Acked Read 5 Sent Done Done false
774 1239 Acked Read 2 Sent Done Done false
775 1240 Acked Write 8 Sent Done Done false
776 1241 Acked Read 3 Sent Done Done false
777 1242 Acked Write 3 Sent Done Done false
778 1243 Acked Read 3 Sent Done Done false
779 1244 Acked Write 8 Sent Done Done false
780 1245 Acked Read 7 Sent Done Done false
781 1246 Acked Read 9 Sent Done Done false
782 1247 Acked Read 7 Sent Done Done false
783 1248 Acked Read 6 Sent Done Done false
784 1249 Acked Write 8 Sent Done Done false
785 1250 Acked Read 3 Sent Done Done false
786 1251 Acked Write 2 Sent Done Done false
787 1252 Acked Read 8 Sent Done Done false
788 1253 Acked Read 2 Sent Done Done false
789 1254 Acked Write 1 Sent Done Done false
790 1255 Acked Write 5 Sent Done Done false
791 1256 Acked Write 5 Sent Done Done false
792 1257 Acked Read 8 Sent Done Done false
793 1258 Acked Read 3 Sent Done Done false
794 1259 Acked Read 2 Sent Done Done false
795 1260 Acked Read 8 Sent Done Done false
796 1261 Acked Read 5 Sent Done Done false
797 1262 Acked Read 3 Sent Done Done false
798 1263 Acked Read 7 Sent Done Done false
799 1264 Acked Read 1 Sent Done Done false
800 1265 Acked Read 8 Sent Done Done false
801 1266 Acked Read 3 Sent Done Done false
802 1267 Acked Write 10 Sent Done Done false
803 1268 Acked Write 9 Sent Done Done false
804 1269 Acked Write 6 Sent Done Done false
805 1270 Acked Read 9 Sent Done Done false
806 1271 Acked Write 7 Sent Done Done false
807 1272 Acked Read 6 Sent Done Done false
808 1273 Acked Read 2 Sent Done Done false
809 1274 Acked Read 8 Sent Done Done false
810 1275 Acked Write 10 Sent Done Done false
811 1276 Acked Write 6 Sent Done Done false
812 1277 Acked Read 6 Sent Done Done false
813 1278 Acked Read 10 Sent Done Done false
814 1279 Acked Read 8 Sent Done Done false
815 1280 Acked Read 5 Sent Done Done false
816 1281 Acked Write 10 Sent Done Done false
817 1282 Acked Write 2 Sent Done Done false
818 1283 Acked Flush 0 Sent Done Done false
819 1284 Acked Write 6 Sent Done Done false
820 1285 Acked Write 5 Sent Done Done false
821 1286 Acked Read 5 Sent Done Done false
822 1287 Acked Read 6 Sent Done Done false
823 1288 Acked Read 7 Sent Done Done false
824 1289 Acked Write 9 Sent Done Done false
825 1290 Acked Write 8 Sent Done Done false
826 1291 Acked Flush 0 Sent Done Done false
827 1292 Acked Read 4 Sent Done Done false
828 1293 Acked Flush 0 Sent Done Done false
829 1294 Acked Write 10 Sent Done Done false
830 1295 Acked Read 9 Sent Done Done false
831 1296 Acked Read 1 Sent Done Done false
832 1297 Acked Read 7 Sent Done Done false
833 1298 Acked Read 5 Sent Done Done false
834 1299 Acked Write 4 Sent Done Done false
835 1300 Acked Read 9 Sent Done Done false
836 STATES DS:0 DS:1 DS:2 TOTAL
837 Sent 301 0 0 301
838 Done 0 301 301 602
839 Skipped 0 0 0 0
840 Error 0 0 0 0
841 Last Flush: None Some(JobId(1293)) Some(JobId(1293))
842 Downstairs last five completed:
843 Upstairs last five completed: 1300 1299 1298 1297 1296
844 CLIENT: Up:0 ds:301 act:2
8452025-02-25T17:18:22.604ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
8462025-02-25T17:18:23.306ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
847 [8810] got start action
848 [8810] I am going to restart
849 Starting downstairs at port 8810
850 Make output file at "/tmp/dsc/downstairs-8810.txt"
851 [8810][0] reports Starting
852 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1209)
853 [8810][0] reports Running
8542025-02-25T17:18:24.308ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
8552025-02-25T17:18:25.310ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
856 ----------------------------------------------------------------
857 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
858 States: Offline Active Active
859 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
860 1000 Acked Read 4 Sent Done Done false
861 1001 Acked Read 2 Sent Done Done false
862 1002 Acked Read 9 Sent Done Done false
863 1003 Acked Write 6 Sent Done Done false
864 1004 Acked Write 10 Sent Done Done false
865 1005 Acked Read 2 Sent Done Done false
866 1006 Acked Write 8 Sent Done Done false
867 1007 Acked Read 6 Sent Done Done false
868 1008 Acked Read 7 Sent Done Done false
869 1009 Acked Read 2 Sent Done Done false
870 1010 Acked Read 3 Sent Done Done false
871 1011 Acked Write 3 Sent Done Done false
872 1012 Acked Read 9 Sent Done Done false
873 1013 Acked Write 10 Sent Done Done false
874 1014 Acked Write 4 Sent Done Done false
875 1015 Acked Read 9 Sent Done Done false
876 1016 Acked Write 2 Sent Done Done false
877 1017 Acked Read 5 Sent Done Done false
878 1018 Acked Read 3 Sent Done Done false
879 1019 Acked Write 2 Sent Done Done false
880 1020 Acked Write 9 Sent Done Done false
881 1021 Acked Write 9 Sent Done Done false
882 1022 Acked Read 10 Sent Done Done false
883 1023 Acked Write 8 Sent Done Done false
884 1024 Acked Write 7 Sent Done Done false
885 1025 Acked Read 5 Sent Done Done false
886 1026 Acked Write 5 Sent Done Done false
887 1027 Acked Read 5 Sent Done Done false
888 1028 Acked Write 4 Sent Done Done false
889 1029 Acked Write 2 Sent Done Done false
890 1030 Acked Read 3 Sent Done Done false
891 1031 Acked Read 10 Sent Done Done false
892 1032 Acked Write 4 Sent Done Done false
893 1033 Acked Write 2 Sent Done Done false
894 1034 Acked Read 6 Sent Done Done false
895 1035 Acked Read 5 Sent Done Done false
896 1036 Acked Read 5 Sent Done Done false
897 1037 Acked Write 3 Sent Done Done false
898 1038 Acked Read 8 Sent Done Done false
899 1039 Acked Write 7 Sent Done Done false
900 1040 Acked Flush 0 Sent Done Done false
901 1041 Acked Read 7 Sent Done Done false
902 1042 Acked Read 1 Sent Done Done false
903 1043 Acked Write 4 Sent Done Done false
904 1044 Acked Write 10 Sent Done Done false
905 1045 Acked Read 9 Sent Done Done false
906 1046 Acked Read 8 Sent Done Done false
907 1047 Acked Read 1 Sent Done Done false
908 1048 Acked Write 3 Sent Done Done false
909 1049 Acked Read 3 Sent Done Done false
910 1050 Acked Write 10 Sent Done Done false
911 1051 Acked Write 7 Sent Done Done false
912 1052 Acked Write 7 Sent Done Done false
913 1053 Acked Read 2 Sent Done Done false
914 1054 Acked Read 3 Sent Done Done false
915 1055 Acked Read 3 Sent Done Done false
916 1056 Acked Write 7 Sent Done Done false
917 1057 Acked Write 2 Sent Done Done false
918 1058 Acked Read 5 Sent Done Done false
919 1059 Acked Read 5 Sent Done Done false
920 1060 Acked Read 1 Sent Done Done false
921 1061 Acked Write 8 Sent Done Done false
922 1062 Acked Read 5 Sent Done Done false
923 1063 Acked Write 8 Sent Done Done false
924 1064 Acked Write 10 Sent Done Done false
925 1065 Acked Write 4 Sent Done Done false
926 1066 Acked Write 1 Sent Done Done false
927 1067 Acked Read 10 Sent Done Done false
928 1068 Acked Write 4 Sent Done Done false
929 1069 Acked Write 1 Sent Done Done false
930 1070 Acked Read 1 Sent Done Done false
931 1071 Acked Write 7 Sent Done Done false
932 1072 Acked Write 10 Sent Done Done false
933 1073 Acked Write 3 Sent Done Done false
934 1074 Acked Read 8 Sent Done Done false
935 1075 Acked Read 8 Sent Done Done false
936 1076 Acked Read 2 Sent Done Done false
937 1077 Acked Write 6 Sent Done Done false
938 1078 Acked Write 9 Sent Done Done false
939 1079 Acked Read 2 Sent Done Done false
940 1080 Acked Read 1 Sent Done Done false
941 1081 Acked Read 4 Sent Done Done false
942 1082 Acked Write 8 Sent Done Done false
943 1083 Acked Read 1 Sent Done Done false
944 1084 Acked Write 8 Sent Done Done false
945 1085 Acked Write 4 Sent Done Done false
946 1086 Acked Read 3 Sent Done Done false
947 1087 Acked Write 5 Sent Done Done false
948 1088 Acked Read 9 Sent Done Done false
949 1089 Acked Write 1 Sent Done Done false
950 1090 Acked Write 9 Sent Done Done false
951 1091 Acked Write 4 Sent Done Done false
952 1092 Acked Read 6 Sent Done Done false
953 1093 Acked Read 4 Sent Done Done false
954 1094 Acked Read 10 Sent Done Done false
955 1095 Acked Read 7 Sent Done Done false
956 1096 Acked Write 3 Sent Done Done false
957 1097 Acked Write 3 Sent Done Done false
958 1098 Acked Write 7 Sent Done Done false
959 1099 Acked Write 10 Sent Done Done false
960 1100 Acked Read 6 Sent Done Done false
961 1101 Acked Read 10 Sent Done Done false
962 1102 Acked Read 9 Sent Done Done false
963 1103 Acked Read 10 Sent Done Done false
964 1104 Acked Read 1 Sent Done Done false
965 1105 Acked Read 10 Sent Done Done false
966 1106 Acked Read 1 Sent Done Done false
967 1107 Acked Write 4 Sent Done Done false
968 1108 Acked Write 4 Sent Done Done false
969 1109 Acked Write 3 Sent Done Done false
970 1110 Acked Read 10 Sent Done Done false
971 1111 Acked Read 10 Sent Done Done false
972 1112 Acked Flush 0 Sent Done Done false
973 1113 Acked Read 5 Sent Done Done false
974 1114 Acked Read 7 Sent Done Done false
975 1115 Acked Read 8 Sent Done Done false
976 1116 Acked Write 3 Sent Done Done false
977 1117 Acked Flush 0 Sent Done Done false
978 1118 Acked Write 2 Sent Done Done false
979 1119 Acked Read 1 Sent Done Done false
980 1120 Acked Flush 0 Sent Done Done false
981 1121 Acked Read 5 Sent Done Done false
982 1122 Acked Read 10 Sent Done Done false
983 1123 Acked Write 6 Sent Done Done false
984 1124 Acked Write 8 Sent Done Done false
985 1125 Acked Read 8 Sent Done Done false
986 1126 Acked Flush 0 Sent Done Done false
987 1127 Acked Write 6 Sent Done Done false
988 1128 Acked Write 3 Sent Done Done false
989 1129 Acked Read 10 Sent Done Done false
990 1130 Acked Read 3 Sent Done Done false
991 1131 Acked Read 2 Sent Done Done false
992 1132 Acked Write 3 Sent Done Done false
993 1133 Acked Read 5 Sent Done Done false
994 1134 Acked Flush 0 Sent Done Done false
995 1135 Acked Read 10 Sent Done Done false
996 1136 Acked Flush 0 Sent Done Done false
997 1137 Acked Read 10 Sent Done Done false
998 1138 Acked Read 8 Sent Done Done false
999 1139 Acked Read 1 Sent Done Done false
1000 1140 Acked Flush 0 Sent Done Done false
1001 1141 Acked Write 10 Sent Done Done false
1002 1142 Acked Read 6 Sent Done Done false
1003 1143 Acked Flush 0 Sent Done Done false
1004 1144 Acked Read 2 Sent Done Done false
1005 1145 Acked Read 8 Sent Done Done false
1006 1146 Acked Flush 0 Sent Done Done false
1007 1147 Acked Read 3 Sent Done Done false
1008 1148 Acked Read 5 Sent Done Done false
1009 1149 Acked Read 3 Sent Done Done false
1010 1150 Acked Write 6 Sent Done Done false
1011 1151 Acked Read 4 Sent Done Done false
1012 1152 Acked Read 5 Sent Done Done false
1013 1153 Acked Read 10 Sent Done Done false
1014 1154 Acked Write 9 Sent Done Done false
1015 1155 Acked Write 4 Sent Done Done false
1016 1156 Acked Flush 0 Sent Done Done false
1017 1157 Acked Read 1 Sent Done Done false
1018 1158 Acked Read 8 Sent Done Done false
1019 1159 Acked Read 1 Sent Done Done false
1020 1160 Acked Flush 0 Sent Done Done false
1021 1161 Acked Write 3 Sent Done Done false
1022 1162 Acked Read 1 Sent Done Done false
1023 1163 Acked Write 5 Sent Done Done false
1024 1164 Acked Write 1 Sent Done Done false
1025 1165 Acked Flush 0 Sent Done Done false
1026 1166 Acked Read 5 Sent Done Done false
1027 1167 Acked Write 10 Sent Done Done false
1028 1168 Acked Read 5 Sent Done Done false
1029 1169 Acked Read 6 Sent Done Done false
1030 1170 Acked Read 4 Sent Done Done false
1031 1171 Acked Read 2 Sent Done Done false
1032 1172 Acked Read 10 Sent Done Done false
1033 1173 Acked Read 9 Sent Done Done false
1034 1174 Acked Read 10 Sent Done Done false
1035 1175 Acked Read 3 Sent Done Done false
1036 1176 Acked Read 9 Sent Done Done false
1037 1177 Acked Write 4 Sent Done Done false
1038 1178 Acked Read 4 Sent Done Done false
1039 1179 Acked Write 3 Sent Done Done false
1040 1180 Acked Write 7 Sent Done Done false
1041 1181 Acked Read 3 Sent Done Done false
1042 1182 Acked Read 1 Sent Done Done false
1043 1183 Acked Write 7 Sent Done Done false
1044 1184 Acked Read 3 Sent Done Done false
1045 1185 Acked Read 7 Sent Done Done false
1046 1186 Acked Read 8 Sent Done Done false
1047 1187 Acked Write 6 Sent Done Done false
1048 1188 Acked Write 10 Sent Done Done false
1049 1189 Acked Read 4 Sent Done Done false
1050 1190 Acked Write 9 Sent Done Done false
1051 1191 Acked Write 1 Sent Done Done false
1052 1192 Acked Write 6 Sent Done Done false
1053 1193 Acked Read 10 Sent Done Done false
1054 1194 Acked Write 1 Sent Done Done false
1055 1195 Acked Write 5 Sent Done Done false
1056 1196 Acked Write 8 Sent Done Done false
1057 1197 Acked Read 10 Sent Done Done false
1058 1198 Acked Write 3 Sent Done Done false
1059 1199 Acked Flush 0 Sent Done Done false
1060 1200 Acked Read 1 Sent Done Done false
1061 1201 Acked Flush 0 Sent Done Done false
1062 1202 Acked Read 10 Sent Done Done false
1063 1203 Acked Flush 0 Sent Done Done false
1064 1204 Acked Write 10 Sent Done Done false
1065 1205 Acked Read 8 Sent Done Done false
1066 1206 Acked Read 2 Sent Done Done false
1067 1207 Acked Read 10 Sent Done Done false
1068 1208 Acked Flush 0 Sent Done Done false
1069 1209 Acked Write 9 Sent Done Done false
1070 1210 Acked Read 6 Sent Done Done false
1071 1211 Acked Read 2 Sent Done Done false
1072 1212 Acked Read 10 Sent Done Done false
1073 1213 Acked Read 4 Sent Done Done false
1074 1214 Acked Write 4 Sent Done Done false
1075 1215 Acked Flush 0 Sent Done Done false
1076 1216 Acked Read 2 Sent Done Done false
1077 1217 Acked Write 3 Sent Done Done false
1078 1218 Acked Write 5 Sent Done Done false
1079 1219 Acked Flush 0 Sent Done Done false
1080 1220 Acked Write 7 Sent Done Done false
1081 1221 Acked Flush 0 Sent Done Done false
1082 1222 Acked Read 8 Sent Done Done false
1083 1223 Acked Write 4 Sent Done Done false
1084 1224 Acked Write 4 Sent Done Done false
1085 1225 Acked Read 10 Sent Done Done false
1086 1226 Acked Write 6 Sent Done Done false
1087 1227 Acked Write 1 Sent Done Done false
1088 1228 Acked Read 8 Sent Done Done false
1089 1229 Acked Read 4 Sent Done Done false
1090 1230 Acked Write 7 Sent Done Done false
1091 1231 Acked Write 4 Sent Done Done false
1092 1232 Acked Read 7 Sent Done Done false
1093 1233 Acked Read 5 Sent Done Done false
1094 1234 Acked Write 6 Sent Done Done false
1095 1235 Acked Read 1 Sent Done Done false
1096 1236 Acked Read 2 Sent Done Done false
1097 1237 Acked Read 2 Sent Done Done false
1098 1238 Acked Read 5 Sent Done Done false
1099 1239 Acked Read 2 Sent Done Done false
1100 1240 Acked Write 8 Sent Done Done false
1101 1241 Acked Read 3 Sent Done Done false
1102 1242 Acked Write 3 Sent Done Done false
1103 1243 Acked Read 3 Sent Done Done false
1104 1244 Acked Write 8 Sent Done Done false
1105 1245 Acked Read 7 Sent Done Done false
1106 1246 Acked Read 9 Sent Done Done false
1107 1247 Acked Read 7 Sent Done Done false
1108 1248 Acked Read 6 Sent Done Done false
1109 1249 Acked Write 8 Sent Done Done false
1110 1250 Acked Read 3 Sent Done Done false
1111 1251 Acked Write 2 Sent Done Done false
1112 1252 Acked Read 8 Sent Done Done false
1113 1253 Acked Read 2 Sent Done Done false
1114 1254 Acked Write 1 Sent Done Done false
1115 1255 Acked Write 5 Sent Done Done false
1116 1256 Acked Write 5 Sent Done Done false
1117 1257 Acked Read 8 Sent Done Done false
1118 1258 Acked Read 3 Sent Done Done false
1119 1259 Acked Read 2 Sent Done Done false
1120 1260 Acked Read 8 Sent Done Done false
1121 1261 Acked Read 5 Sent Done Done false
1122 1262 Acked Read 3 Sent Done Done false
1123 1263 Acked Read 7 Sent Done Done false
1124 1264 Acked Read 1 Sent Done Done false
1125 1265 Acked Read 8 Sent Done Done false
1126 1266 Acked Read 3 Sent Done Done false
1127 1267 Acked Write 10 Sent Done Done false
1128 1268 Acked Write 9 Sent Done Done false
1129 1269 Acked Write 6 Sent Done Done false
1130 1270 Acked Read 9 Sent Done Done false
1131 1271 Acked Write 7 Sent Done Done false
1132 1272 Acked Read 6 Sent Done Done false
1133 1273 Acked Read 2 Sent Done Done false
1134 1274 Acked Read 8 Sent Done Done false
1135 1275 Acked Write 10 Sent Done Done false
1136 1276 Acked Write 6 Sent Done Done false
1137 1277 Acked Read 6 Sent Done Done false
1138 1278 Acked Read 10 Sent Done Done false
1139 1279 Acked Read 8 Sent Done Done false
1140 1280 Acked Read 5 Sent Done Done false
1141 1281 Acked Write 10 Sent Done Done false
1142 1282 Acked Write 2 Sent Done Done false
1143 1283 Acked Flush 0 Sent Done Done false
1144 1284 Acked Write 6 Sent Done Done false
1145 1285 Acked Write 5 Sent Done Done false
1146 1286 Acked Read 5 Sent Done Done false
1147 1287 Acked Read 6 Sent Done Done false
1148 1288 Acked Read 7 Sent Done Done false
1149 1289 Acked Write 9 Sent Done Done false
1150 1290 Acked Write 8 Sent Done Done false
1151 1291 Acked Flush 0 Sent Done Done false
1152 1292 Acked Read 4 Sent Done Done false
1153 1293 Acked Flush 0 Sent Done Done false
1154 1294 Acked Write 10 Sent Done Done false
1155 1295 Acked Read 9 Sent Done Done false
1156 1296 Acked Read 1 Sent Done Done false
1157 1297 Acked Read 7 Sent Done Done false
1158 1298 Acked Read 5 Sent Done Done false
1159 1299 Acked Write 4 Sent Done Done false
1160 1300 Acked Read 9 Sent Done Done false
1161 STATES DS:0 DS:1 DS:2 TOTAL
1162 Sent 301 0 0 301
1163 Done 0 301 301 602
1164 Skipped 0 0 0 0
1165 Error 0 0 0 0
1166 Last Flush: None Some(JobId(1293)) Some(JobId(1293))
1167 Downstairs last five completed:
1168 Upstairs last five completed: 1300 1299 1298 1297 1296
1169 CLIENT: Up:0 ds:301 act:2
11702025-02-25T17:18:26.607ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11712025-02-25T17:18:27.314ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11722025-02-25T17:18:27.836ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11732025-02-25T17:18:27.836ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58252) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11742025-02-25T17:18:27.837ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11752025-02-25T17:18:27.838ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11762025-02-25T17:18:27.838ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11772025-02-25T17:18:27.838ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11782025-02-25T17:18:27.838ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11792025-02-25T17:18:27.838ZINFOcrucible: [0] client re-new 301 jobs since flush None = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11802025-02-25T17:18:27.838ZINFOcrucible: [0] Replayed 301 jobs since flush: None = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1181 ----------------------------------------------------------------
1182 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1183 Downstairs last five completed: 1301 1300 1299 1298 1297
1184 Upstairs last five completed: 1301 1300 1299 1298 1297
1185 CLIENT: Up:0 ds:0 act:3
1186 Replay: All jobs finished
1187 Main task has work to do, go find it
1188 got dsc Stop(0)
1189 stop 0
1190 Feb 25 17:18:30.108 INFO request completed, latency_us: 393, response_code: 204, uri[8810] Got stop action so:false kr:false
1191 : /stop/cid/0, method: POST, req_id: b92bd15c-cc4f-4ab9-ae6a-952777a6183d, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
11922025-02-25T17:18:30.108ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11932025-02-25T17:18:30.108ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11942025-02-25T17:18:30.109ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11952025-02-25T17:18:30.109ZINFOcrucible: client task is exiting = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1196 Feb 25 17:18:30.109 INFO request completed, latency_us: 304, response_code: 200, uri: /state/cid/0, method: GET, req_id: 45a9047d-192b-4556-9eeb-95982e3ab0c8, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
1197 [8810] Exited with: ExitStatus(unix_wait_status(9))
1198 [8810][0] reports Exit
1199 Feb 25 17:18:34.111 INFO request completed, latency_us: 216, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5730fb0b-eb94-41c0-b91d-485ac625b9d3, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
1200 001/300 Read block 4682 len 24576
1201 002/300 Read block 3835 len 36864
1202 003/300 Write block 4707 len 8192 data: 2 2
1203 004/300 Read block 2940 len 28672
1204 005/300 Read block 146 len 24576
1205 006/300 Flush
1206 007/300 Read block 4572 len 36864
1207 008/300 Write block 3854 len 16384 data: 2 2 2 2
1208 009/300 Read block 1197 len 24576
1209 010/300 Read block 1002 len 8192
1210 011/300 Write block 3283 len 36864 data: 2 3 3 3 3 2 2 2 2
1211 012/300 Write block 3321 len 12288 data: 2 2 2
1212 013/300 Read block 1130 len 32768
1213 014/300 Write block 3788 len 16384 data: 2 2 2 2
1214 015/300 Read block 4657 len 32768
1215 016/300 Write block 2128 len 36864 data: 2 2 2 3 3 3 3 3 2
1216 017/300 Read block 3280 len 12288
1217 018/300 Read block 689 len 20480
1218 019/300 Flush
1219 020/300 Write block 143 len 24576 data: 2 2 2 2 2 2
1220 021/300 Read block 1740 len 8192
1221 022/300 Read block 3542 len 32768
1222 023/300 Write block 962 len 36864 data: 2 2 2 2 3 3 3 2 2
1223 024/300 Flush
1224 025/300 Read block 4727 len 32768
1225 026/300 Read block 1885 len 36864
1226 027/300 Write block 927 len 28672 data: 2 2 2 2 2 3 3
1227 028/300 Write block 1646 len 24576 data: 2 2 2 2 2 2
1228 029/300 Flush
1229 030/300 Write block 4061 len 20480 data: 2 2 2 2 2
1230 031/300 Write block 4718 len 20480 data: 2 2 2 2 2
1231 032/300 Write block 1998 len 4096 data: 2
1232 033/300 Read block 3486 len 12288
1233 034/300 Read block 4290 len 4096
1234 035/300 Write block 3937 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1235 036/300 Write block 3516 len 32768 data: 2 2 2 2 2 2 2 2
1236 037/300 Read block 4468 len 40960
1237 038/300 Write block 893 len 12288 data: 2 2 2
1238 039/300 Read block 630 len 36864
1239 040/300 Write block 2805 len 28672 data: 2 2 2 2 2 2 2
1240 041/300 Write block 729 len 24576 data: 2 2 2 2 2 2
1241 042/300 Write block 3210 len 16384 data: 2 2 2 2
1242 043/300 Write block 3042 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1243 044/300 Flush
1244 045/300 Write block 2781 len 16384 data: 2 2 2 2
1245 046/300 Read block 3814 len 36864
1246 047/300 Read block 1585 len 40960
1247 048/300 Read block 141 len 24576
1248 049/300 Read block 2756 len 20480
1249 050/300 Read block 4266 len 12288
1250 051/300 Read block 1619 len 40960
1251 052/300 Read block 1893 len 12288
1252 053/300 Write block 2541 len 12288 data: 2 2 2
1253 054/300 Read block 873 len 28672
1254 055/300 Write block 1385 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1255 056/300 Write block 1427 len 28672 data: 2 2 2 2 2 2 2
1256 057/300 Read block 4541 len 36864
1257 058/300 Read block 2322 len 40960
1258 059/300 Write block 3159 len 20480 data: 2 2 2 2 2
1259 060/300 Read block 3155 len 4096
1260 061/300 Read block 4518 len 24576
1261 062/300 Read block 1508 len 28672
1262 063/300 Write block 3028 len 12288 data: 2 2 2
1263 064/300 Write block 3140 len 20480 data: 2 2 2 2 2
1264 065/300 Flush
1265 066/300 Write block 330 len 36864 data: 2 2 2 2 2 2 2 2 2
1266 067/300 Write block 1889 len 16384 data: 2 2 2 2
1267 068/300 Write block 963 len 24576 data: 3 3 3 4 4 4
1268 069/300 Read block 1867 len 4096
1269 070/300 Write block 356 len 8192 data: 2 2
1270 071/300 Write block 2282 len 20480 data: 2 2 2 2 2
1271 072/300 Flush
1272 073/300 Read block 4301 len 4096
1273 074/300 Read block 2107 len 4096
1274 075/300 Write block 4980 len 4096 data: 2
1275 076/300 Write block 874 len 16384 data: 2 2 2 2
1276 077/300 Read block 4851 len 20480
1277 078/300 Read block 3967 len 24576
1278 079/300 Write block 1157 len 12288 data: 2 2 2
1279 080/300 Write block 786 len 32768 data: 2 2 2 2 2 2 2 2
1280 081/300 Write block 1837 len 16384 data: 3 3 3 3
1281 082/300 Read block 1702 len 28672
1282 083/300 Write block 1062 len 4096 data: 3
1283 084/300 Read block 1236 len 4096
1284 085/300 Write block 108 len 40960 data: 3 3 3 2 2 2 2 2 2 2
1285 086/300 Read block 4042 len 8192
1286 087/300 Write block 1241 len 32768 data: 2 2 2 2 2 2 2 2
1287 088/300 Write block 2558 len 8192 data: 2 2
1288 089/300 Write block 1465 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1289 090/300 Read block 3575 len 32768
1290 091/300 Read block 1756 len 8192
1291 092/300 Read block 4828 len 32768
1292 093/300 Read block 2909 len 40960
1293 094/300 Write block 489 len 36864 data: 3 2 2 2 2 2 2 2 2
1294 095/300 Write block 4812 len 12288 data: 3 3 3
1295 096/300 Write block 270 len 28672 data: 3 3 3 3 3 2 2
1296 097/300 Flush
1297 098/300 Write block 4939 len 16384 data: 2 2 2 2
1298 099/300 Write block 4233 len 16384 data: 2 2 2 2
1299 100/300 Read block 2227 len 36864
1300 101/300 Read block 4056 len 24576
1301 102/300 Write block 4811 len 4096 data: 3
1302 103/300 Read block 132 len 40960
1303 104/300 Write block 1974 len 24576 data: 2 2 2 2 2 2
1304 105/300 Read block 1710 len 12288
1305 106/300 Read block 3026 len 28672
1306 107/300 Write block 533 len 32768 data: 2 2 2 2 2 2 2 2
1307 108/300 Write block 4701 len 32768 data: 2 2 2 2 2 2 3 3
1308 109/300 Read block 4628 len 16384
1309 110/300 Read block 2342 len 40960
1310 111/300 Read block 2730 len 20480
1311 112/300 Write block 925 len 8192 data: 2 2
1312 113/300 Write block 1853 len 12288 data: 2 2 2
1313 114/300 Read block 734 len 8192
1314 115/300 Write block 3635 len 12288 data: 2 2 2
1315 116/300 Write block 3028 len 28672 data: 3 3 3 2 2 2 2
1316 117/300 Read block 411 len 4096
1317 118/300 Write block 2303 len 8192 data: 2 2
1318 119/300 Read block 1036 len 16384
1319 120/300 Write block 4748 len 8192 data: 2 2
1320 121/300 Write block 1904 len 32768 data: 2 2 2 2 2 2 2 2
1321 122/300 Read block 2201 len 32768
1322 123/300 Write block 18 len 32768 data: 2 2 2 2 2 2 2 2
1323 124/300 Flush
1324 125/300 Flush
1325 126/300 Write block 249 len 8192 data: 3 3
1326 127/300 Read block 4954 len 4096
1327 128/300 Flush
1328 129/300 Write block 3408 len 20480 data: 2 2 2 2 2
1329 130/300 Read block 4297 len 20480
1330 131/300 Read block 2269 len 20480
1331 132/300 Read block 684 len 8192
1332 133/300 Write block 3469 len 8192 data: 2 2
1333 134/300 Read block 1443 len 4096
1334 135/300 Write block 1748 len 36864 data: 2 2 2 2 2 2 2 2 2
1335 136/300 Write block 1436 len 20480 data: 2 2 2 3 3
1336 137/300 Write block 3360 len 16384 data: 2 2 2 2
1337 138/300 Read block 3129 len 40960
1338 139/300 Read block 2053 len 16384
1339 140/300 Write block 3911 len 24576 data: 2 2 2 2 2 2
1340 141/300 Read block 630 len 16384
1341 142/300 Write block 279 len 12288 data: 2 2 2
1342 143/300 Flush
1343 144/300 Write block 1960 len 8192 data: 2 2
1344 145/300 Write block 4720 len 20480 data: 3 3 3 2 2
1345 146/300 Write block 709 len 8192 data: 2 2
1346 147/300 Write block 2439 len 8192 data: 2 2
1347 148/300 Write block 2900 len 32768 data: 2 2 2 2 2 2 2 2
1348 149/300 Write block 2858 len 36864 data: 3 3 3 2 2 2 2 2 2
1349 150/300 Write block 418 len 12288 data: 2 2 2
1350 151/300 Write block 3352 len 4096 data: 2
1351 152/300 Read block 1984 len 24576
1352 153/300 Write block 4752 len 32768 data: 3 3 3 3 3 3 3 3
1353 154/300 Read block 3191 len 32768
1354 155/300 Read block 662 len 40960
1355 156/300 Read block 4593 len 8192
1356 157/300 Read block 1234 len 8192
1357 158/300 Read block 2180 len 32768
1358 159/300 Write block 1719 len 16384 data: 2 2 2 2
1359 160/300 Write block 2320 len 28672 data: 2 2 2 2 2 2 2
1360 161/300 Flush
1361 162/300 Read block 3790 len 8192
1362 163/300 Flush
1363 164/300 Write block 1763 len 32768 data: 2 2 2 2 2 2 2 2
1364 165/300 Flush
1365 166/300 Read block 1813 len 36864
1366 167/300 Flush
1367 168/300 Write block 1899 len 28672 data: 2 2 2 2 2 3 3
1368 169/300 Read block 2988 len 16384
1369 170/300 Read block 158 len 16384
1370 171/300 Flush
1371 172/300 Read block 4745 len 28672
1372 173/300 Read block 2006 len 16384
1373 174/300 Read block 1290 len 12288
1374 175/300 Read block 405 len 12288
1375 176/300 Read block 3391 len 8192
1376 177/300 Write block 1581 len 32768 data: 2 2 2 2 2 2 2 2
1377 178/300 Write block 3547 len 32768 data: 2 2 2 2 2 2 2 2
1378 179/300 Flush
1379 180/300 Read block 3636 len 28672
1380 181/300 Read block 4133 len 36864
1381 182/300 Write block 4120 len 20480 data: 2 2 2 2 2
1382 183/300 Read block 895 len 8192
1383 184/300 Read block 1868 len 20480
1384 185/300 Read block 4790 len 24576
1385 186/300 Read block 3054 len 12288
1386 187/300 Read block 1938 len 28672
1387 188/300 Write block 4156 len 24576 data: 2 2 2 2 2 2
1388 189/300 Flush
1389 190/300 Read block 2525 len 4096
1390 191/300 Write block 1787 len 4096 data: 2
1391 192/300 Read block 4386 len 40960
1392 193/300 Write block 4798 len 20480 data: 2 2 2 2 2
1393 194/300 Write block 3665 len 24576 data: 2 2 2 2 3 3
1394 195/300 Write block 4279 len 28672 data: 2 2 2 2 2 2 2
1395 196/300 Read block 4176 len 36864
1396 197/300 Read block 1882 len 40960
1397 198/300 Read block 936 len 4096
1398 199/300 Read block 959 len 12288
1399 200/300 Read block 1502 len 32768
1400 201/300 Write block 1645 len 32768 data: 2 3 3 3 3 3 3 2
1401 202/300 Write block 2767 len 40960 data: 3 3 3 3 2 2 2 2 2 2
1402 203/300 Flush
1403 204/300 Write block 2009 len 16384 data: 2 2 2 2
1404 205/300 Read block 2974 len 24576
1405 206/300 Flush
1406 207/300 Read block 3997 len 8192
1407 208/300 Read block 2923 len 36864
1408 209/300 Write block 4613 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1409 210/300 Read block 1696 len 4096
1410 211/300 Flush
1411 212/300 Write block 422 len 12288 data: 2 2 2
1412 213/300 Read block 4722 len 24576
1413 214/300 Read block 3102 len 8192
1414 215/300 Read block 1976 len 36864
1415 216/300 Read block 34 len 40960
1416 217/300 Read block 143 len 12288
1417 218/300 Read block 4293 len 28672
1418 219/300 Read block 429 len 32768
1419 220/300 Write block 2852 len 32768 data: 2 2 2 2 2 3 4 4
1420 221/300 Read block 2174 len 28672
1421 222/300 Write block 3719 len 4096 data: 3
1422 223/300 Read block 3698 len 8192
1423 224/300 Read block 222 len 32768
1424 225/300 Read block 3874 len 28672
1425 226/300 Write block 992 len 20480 data: 2 2 2 2 2
1426 227/300 Write block 4345 len 28672 data: 2 2 2 2 2 2 2
1427 228/300 Read block 4688 len 36864
1428 229/300 Read block 2087 len 20480
1429 230/300 Write block 1620 len 28672 data: 2 2 2 2 2 2 2
1430 231/300 Write block 1779 len 4096 data: 2
1431 232/300 Write block 4378 len 16384 data: 2 2 2 2
1432 233/300 Write block 3294 len 16384 data: 2 2 2 2
1433 234/300 Read block 2532 len 8192
1434 235/300 Flush
1435 236/300 Write block 1235 len 16384 data: 3 3 3 3
1436 237/300 Read block 232 len 16384
1437 238/300 Write block 3199 len 36864 data: 2 2 2 2 2 2 2 2 2
1438 239/300 Read block 3023 len 20480
1439 240/300 Write block 1523 len 36864 data: 2 2 2 2 2 2 2 2 2
1440 241/300 Write block 613 len 16384 data: 2 2 2 2
1441 242/300 Read block 665 len 28672
1442 243/300 Read block 2254 len 16384
1443 244/300 Flush
1444 245/300 Write block 4803 len 16384 data: 2 2 2 2
1445 246/300 Write block 2121 len 36864 data: 2 2 2 2 2 2 2 3 3
1446 247/300 Write block 380 len 8192 data: 2 2
1447 248/300 Read block 3743 len 8192
1448 249/300 Read block 3373 len 24576
1449 250/300 Read block 4058 len 40960
1450 251/300 Read block 2754 len 16384
1451 252/300 Write block 1974 len 12288 data: 3 3 3
1452 253/300 Read block 1228 len 32768
1453 254/300 Write block 3828 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1454 255/300 Write block 3653 len 36864 data: 2 2 2 2 2 2 2 2 2
1455 256/300 Read block 668 len 24576
1456 257/300 Flush
1457 258/300 Read block 334 len 20480
1458 259/300 Read block 3028 len 28672
1459 260/300 Write block 4870 len 8192 data: 2 2
1460 261/300 Read block 2401 len 12288
1461 262/300 Write block 4288 len 36864 data: 2 2 2 2 2 2 2 2 2
1462 263/300 Read block 4934 len 24576
1463 264/300 Read block 3226 len 20480
1464 265/300 Write block 700 len 8192 data: 2 2
1465 266/300 Read block 2640 len 32768
1466 267/300 Read block 3497 len 4096
1467 268/300 Write block 1929 len 36864 data: 2 2 2 2 2 2 2 2 2
1468 269/300 Read block 4263 len 20480
1469 270/300 Flush
1470 271/300 Read block 1282 len 36864
1471 272/300 Read block 1692 len 24576
1472 273/300 Read block 3058 len 12288
1473 274/300 Write block 580 len 8192 data: 2 2
1474 275/300 Read block 830 len 32768
1475 276/300 Read block 2481 len 8192
1476 277/300 Read block 1470 len 8192
1477 278/300 Write block 377 len 16384 data: 2 2 2 3
1478 279/300 Read block 3288 len 20480
1479 280/300 Read block 1700 len 4096
1480 281/300 Write block 2961 len 36864 data: 2 2 2 2 2 2 2 2 2
1481 282/300 Write block 1734 len 16384 data: 2 2 2 2
1482 283/300 Flush
1483 284/300 Write block 974 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1484 285/300 Write block 2172 len 12288 data: 2 2 2
1485 286/300 Write block 829 len 36864 data: 2 2 2 3 3 3 3 3 3
1486 287/300 Read block 3083 len 24576
1487 288/300 Write block 2211 len 4096 data: 2
1488 289/300 Write block 2352 len 16384 data: 2 2 2 3
1489 290/300 Write block 886 len 24576 data: 2 2 2 2 2 2
1490 291/300 Read block 4578 len 28672
1491 292/300 Read block 2941 len 40960
1492 293/300 Flush
1493 294/300 Write block 2457 len 28672 data: 2 2 3 3 3 3 2
1494 295/300 Read block 2386 len 32768
1495 296/300 Read block 3377 len 36864
1496 297/300 Write block 315 len 4096 data: 2
1497 298/300 Write block 176 len 16384 data: 3 3 3 3
1498 299/300 Read block 1499 len 40960
1499 300/300 Read block 3200 len 4096
1500 301/300 Write block 2517 len 8192 data: 3 3
1501 Main task has work to do, go find it
1502 got dsc Start(0)
1503 start 0
1504 Feb 25 17:18:34.354 INFO request completed, latency_us: 225, response_code: 204, uri: /start/cid/0, method: POST, req_id: 8d6ac4f3-e40d-4831-b6cb-3c8ec3685bfc, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
1505 Replay: started 0, returned:Ok(())
1506 ----------------------------------------------------------------
1507 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1508 States: Offline Active Active
1509 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1510 1302 Acked Read 6 Sent Done Done false
1511 1303 Acked Read 9 Sent Done Done false
1512 1304 Acked Write 2 Sent Done Done false
1513 1305 Acked Read 7 Sent Done Done false
1514 1306 Acked Read 6 Sent Done Done false
1515 1307 Acked Flush 0 Sent Done Done false
1516 1308 Acked Read 9 Sent Done Done false
1517 1309 Acked Write 4 Sent Done Done false
1518 1310 Acked Read 6 Sent Done Done false
1519 1311 Acked Read 2 Sent Done Done false
1520 1312 Acked Write 9 Sent Done Done false
1521 1313 Acked Write 3 Sent Done Done false
1522 1314 Acked Read 8 Sent Done Done false
1523 1315 Acked Write 4 Sent Done Done false
1524 1316 Acked Read 8 Sent Done Done false
1525 1317 Acked Write 9 Sent Done Done false
1526 1318 Acked Read 3 Sent Done Done false
1527 1319 Acked Read 5 Sent Done Done false
1528 1320 Acked Flush 0 Sent Done Done false
1529 1321 Acked Write 6 Sent Done Done false
1530 1322 Acked Read 2 Sent Done Done false
1531 1323 Acked Read 8 Sent Done Done false
1532 1324 Acked Write 9 Sent Done Done false
1533 1325 Acked Flush 0 Sent Done Done false
1534 1326 Acked Read 8 Sent Done Done false
1535 1327 Acked Read 9 Sent Done Done false
1536 1328 Acked Write 7 Sent Done Done false
1537 1329 Acked Write 6 Sent Done Done false
1538 1330 Acked Flush 0 Sent Done Done false
1539 1331 Acked Write 5 Sent Done Done false
1540 1332 Acked Write 5 Sent Done Done false
1541 1333 Acked Write 1 Sent Done Done false
1542 1334 Acked Read 3 Sent Done Done false
1543 1335 Acked Read 1 Sent Done Done false
1544 1336 Acked Write 10 Sent Done Done false
1545 1337 Acked Write 8 Sent Done Done false
1546 1338 Acked Read 10 Sent Done Done false
1547 1339 Acked Write 3 Sent Done Done false
1548 1340 Acked Read 9 Sent Done Done false
1549 1341 Acked Write 7 Sent Done Done false
1550 1342 Acked Write 6 Sent Done Done false
1551 1343 Acked Write 4 Sent Done Done false
1552 1344 Acked Write 10 Sent Done Done false
1553 1345 Acked Flush 0 Sent Done Done false
1554 1346 Acked Write 4 Sent Done Done false
1555 1347 Acked Read 9 Sent Done Done false
1556 1348 Acked Read 10 Sent Done Done false
1557 1349 Acked Read 6 Sent Done Done false
1558 1350 Acked Read 5 Sent Done Done false
1559 1351 Acked Read 3 Sent Done Done false
1560 1352 Acked Read 10 Sent Done Done false
1561 1353 Acked Read 3 Sent Done Done false
1562 1354 Acked Write 3 Sent Done Done false
1563 1355 Acked Read 7 Sent Done Done false
1564 1356 Acked Write 10 Sent Done Done false
1565 1357 Acked Write 7 Sent Done Done false
1566 1358 Acked Read 9 Sent Done Done false
1567 1359 Acked Read 10 Sent Done Done false
1568 1360 Acked Write 5 Sent Done Done false
1569 1361 Acked Read 1 Sent Done Done false
1570 1362 Acked Read 6 Sent Done Done false
1571 1363 Acked Read 7 Sent Done Done false
1572 1364 Acked Write 3 Sent Done Done false
1573 1365 Acked Write 5 Sent Done Done false
1574 1366 Acked Flush 0 Sent Done Done false
1575 1367 Acked Write 9 Sent Done Done false
1576 1368 Acked Write 4 Sent Done Done false
1577 1369 Acked Write 6 Sent Done Done false
1578 1370 Acked Read 1 Sent Done Done false
1579 1371 Acked Write 2 Sent Done Done false
1580 1372 Acked Write 5 Sent Done Done false
1581 1373 Acked Flush 0 Sent Done Done false
1582 1374 Acked Read 1 Sent Done Done false
1583 1375 Acked Read 1 Sent Done Done false
1584 1376 Acked Write 1 Sent Done Done false
1585 1377 Acked Write 4 Sent Done Done false
1586 1378 Acked Read 5 Sent Done Done false
1587 1379 Acked Read 6 Sent Done Done false
1588 1380 Acked Write 3 Sent Done Done false
1589 1381 Acked Write 8 Sent Done Done false
1590 1382 Acked Write 4 Sent Done Done false
1591 1383 Acked Read 7 Sent Done Done false
1592 1384 Acked Write 1 Sent Done Done false
1593 1385 Acked Read 1 Sent Done Done false
1594 1386 Acked Write 10 Sent Done Done false
1595 1387 Acked Read 2 Sent Done Done false
1596 1388 Acked Write 8 Sent Done Done false
1597 1389 Acked Write 2 Sent Done Done false
1598 1390 Acked Write 10 Sent Done Done false
1599 1391 Acked Read 8 Sent Done Done false
1600 1392 Acked Read 2 Sent Done Done false
1601 1393 Acked Read 8 Sent Done Done false
1602 1394 Acked Read 10 Sent Done Done false
1603 1395 Acked Write 9 Sent Done Done false
1604 1396 Acked Write 3 Sent Done Done false
1605 1397 Acked Write 7 Sent Done Done false
1606 1398 Acked Flush 0 Sent Done Done false
1607 1399 Acked Write 4 Sent Done Done false
1608 1400 Acked Write 4 Sent Done Done false
1609 1401 Acked Read 9 Sent Done Done false
1610 1402 Acked Read 6 Sent Done Done false
1611 1403 Acked Write 1 Sent Done Done false
1612 1404 Acked Read 10 Sent Done Done false
1613 1405 Acked Write 6 Sent Done Done false
1614 1406 Acked Read 3 Sent Done Done false
1615 1407 Acked Read 7 Sent Done Done false
1616 1408 Acked Write 8 Sent Done Done false
1617 1409 Acked Write 8 Sent Done Done false
1618 1410 Acked Read 4 Sent Done Done false
1619 1411 Acked Read 10 Sent Done Done false
1620 1412 Acked Read 5 Sent Done Done false
1621 1413 Acked Write 2 Sent Done Done false
1622 1414 Acked Write 3 Sent Done Done false
1623 1415 Acked Read 2 Sent Done Done false
1624 1416 Acked Write 3 Sent Done Done false
1625 1417 Acked Write 7 Sent Done Done false
1626 1418 Acked Read 1 Sent Done Done false
1627 1419 Acked Write 2 Sent Done Done false
1628 1420 Acked Read 4 Sent Done Done false
1629 1421 Acked Write 2 Sent Done Done false
1630 1422 Acked Write 8 Sent Done Done false
1631 1423 Acked Read 8 Sent Done Done false
1632 1424 Acked Write 8 Sent Done Done false
1633 1425 Acked Flush 0 Sent Done Done false
1634 1426 Acked Flush 0 Sent Done Done false
1635 1427 Acked Write 2 Sent Done Done false
1636 1428 Acked Read 1 Sent Done Done false
1637 1429 Acked Flush 0 Sent Done Done false
1638 1430 Acked Write 5 Sent Done Done false
1639 1431 Acked Read 5 Sent Done Done false
1640 1432 Acked Read 5 Sent Done Done false
1641 1433 Acked Read 2 Sent Done Done false
1642 1434 Acked Write 2 Sent Done Done false
1643 1435 Acked Read 1 Sent Done Done false
1644 1436 Acked Write 9 Sent Done Done false
1645 1437 Acked Write 5 Sent Done Done false
1646 1438 Acked Write 4 Sent Done Done false
1647 1439 Acked Read 10 Sent Done Done false
1648 1440 Acked Read 4 Sent Done Done false
1649 1441 Acked Write 6 Sent Done Done false
1650 1442 Acked Read 4 Sent Done Done false
1651 1443 Acked Write 3 Sent Done Done false
1652 1444 Acked Flush 0 Sent Done Done false
1653 1445 Acked Write 2 Sent Done Done false
1654 1446 Acked Write 5 Sent Done Done false
1655 1447 Acked Write 2 Sent Done Done false
1656 1448 Acked Write 2 Sent Done Done false
1657 1449 Acked Write 8 Sent Done Done false
1658 1450 Acked Write 9 Sent Done Done false
1659 1451 Acked Write 3 Sent Done Done false
1660 1452 Acked Write 1 Sent Done Done false
1661 1453 Acked Read 6 Sent Done Done false
1662 1454 Acked Write 8 Sent Done Done false
1663 1455 Acked Read 8 Sent Done Done false
1664 1456 Acked Read 10 Sent Done Done false
1665 1457 Acked Read 2 Sent Done Done false
1666 1458 Acked Read 2 Sent Done Done false
1667 1459 Acked Read 8 Sent Done Done false
1668 1460 Acked Write 4 Sent Done Done false
1669 1461 Acked Write 7 Sent Done Done false
1670 1462 Acked Flush 0 Sent Done Done false
1671 1463 Acked Read 2 Sent Done Done false
1672 1464 Acked Flush 0 Sent Done Done false
1673 1465 Acked Write 8 Sent Done Done false
1674 1466 Acked Flush 0 Sent Done Done false
1675 1467 Acked Read 9 Sent Done Done false
1676 1468 Acked Flush 0 Sent Done Done false
1677 1469 Acked Write 7 Sent Done Done false
1678 1470 Acked Read 4 Sent Done Done false
1679 1471 Acked Read 4 Sent Done Done false
1680 1472 Acked Flush 0 Sent Done Done false
1681 1473 Acked Read 7 Sent Done Done false
1682 1474 Acked Read 4 Sent Done Done false
1683 1475 Acked Read 3 Sent Done Done false
1684 1476 Acked Read 3 Sent Done Done false
1685 1477 Acked Read 2 Sent Done Done false
1686 1478 Acked Write 8 Sent Done Done false
1687 1479 Acked Write 8 Sent Done Done false
1688 1480 Acked Flush 0 Sent Done Done false
1689 1481 Acked Read 7 Sent Done Done false
1690 1482 Acked Read 9 Sent Done Done false
1691 1483 Acked Write 5 Sent Done Done false
1692 1484 Acked Read 2 Sent Done Done false
1693 1485 Acked Read 5 Sent Done Done false
1694 1486 Acked Read 6 Sent Done Done false
1695 1487 Acked Read 3 Sent Done Done false
1696 1488 Acked Read 7 Sent Done Done false
1697 1489 Acked Write 6 Sent Done Done false
1698 1490 Acked Flush 0 Sent Done Done false
1699 1491 Acked Read 1 Sent Done Done false
1700 1492 Acked Write 1 Sent Done Done false
1701 1493 Acked Read 10 Sent Done Done false
1702 1494 Acked Write 5 Sent Done Done false
1703 1495 Acked Write 6 Sent Done Done false
1704 1496 Acked Write 7 Sent Done Done false
1705 1497 Acked Read 9 Sent Done Done false
1706 1498 Acked Read 10 Sent Done Done false
1707 1499 Acked Read 1 Sent Done Done false
1708 1500 Acked Read 3 Sent Done Done false
1709 1501 Acked Read 8 Sent Done Done false
1710 1502 Acked Write 8 Sent Done Done false
1711 1503 Acked Write 10 Sent Done Done false
1712 1504 Acked Flush 0 Sent Done Done false
1713 1505 Acked Write 4 Sent Done Done false
1714 1506 Acked Read 6 Sent Done Done false
1715 1507 Acked Flush 0 Sent Done Done false
1716 1508 Acked Read 2 Sent Done Done false
1717 1509 Acked Read 9 Sent Done Done false
1718 1510 Acked Write 10 Sent Done Done false
1719 1511 Acked Read 1 Sent Done Done false
1720 1512 Acked Flush 0 Sent Done Done false
1721 1513 Acked Write 3 Sent Done Done false
1722 1514 Acked Read 6 Sent Done Done false
1723 1515 Acked Read 2 Sent Done Done false
1724 1516 Acked Read 9 Sent Done Done false
1725 1517 Acked Read 10 Sent Done Done false
1726 1518 Acked Read 3 Sent Done Done false
1727 1519 Acked Read 7 Sent Done Done false
1728 1520 Acked Read 8 Sent Done Done false
1729 1521 Acked Write 8 Sent Done Done false
1730 1522 Acked Read 7 Sent Done Done false
1731 1523 Acked Write 1 Sent Done Done false
1732 1524 Acked Read 2 Sent Done Done false
1733 1525 Acked Read 8 Sent Done Done false
1734 1526 Acked Read 7 Sent Done Done false
1735 1527 Acked Write 5 Sent Done Done false
1736 1528 Acked Write 7 Sent Done Done false
1737 1529 Acked Read 9 Sent Done Done false
1738 1530 Acked Read 5 Sent Done Done false
1739 1531 Acked Write 7 Sent Done Done false
1740 1532 Acked Write 1 Sent Done Done false
1741 1533 Acked Write 4 Sent Done Done false
1742 1534 Acked Write 4 Sent Done Done false
1743 1535 Acked Read 2 Sent Done Done false
1744 1536 Acked Flush 0 Sent Done Done false
1745 1537 Acked Write 4 Sent Done Done false
1746 1538 Acked Read 4 Sent Done Done false
1747 1539 Acked Write 9 Sent Done Done false
1748 1540 Acked Read 5 Sent Done Done false
1749 1541 Acked Write 9 Sent Done Done false
1750 1542 Acked Write 4 Sent Done Done false
1751 1543 Acked Read 7 Sent Done Done false
1752 1544 Acked Read 4 Sent Done Done false
1753 1545 Acked Flush 0 Sent Done Done false
1754 1546 Acked Write 4 Sent Done Done false
1755 1547 Acked Write 9 Sent Done Done false
1756 1548 Acked Write 2 Sent Done Done false
1757 1549 Acked Read 2 Sent Done Done false
1758 1550 Acked Read 6 Sent Done Done false
1759 1551 Acked Read 10 Sent Done Done false
1760 1552 Acked Read 4 Sent Done Done false
1761 1553 Acked Write 3 Sent Done Done false
1762 1554 Acked Read 8 Sent Done Done false
1763 1555 Acked Write 10 Sent Done Done false
1764 1556 Acked Write 9 Sent Done Done false
1765 1557 Acked Read 6 Sent Done Done false
1766 1558 Acked Flush 0 Sent Done Done false
1767 1559 Acked Read 5 Sent Done Done false
1768 1560 Acked Read 7 Sent Done Done false
1769 1561 Acked Write 2 Sent Done Done false
1770 1562 Acked Read 3 Sent Done Done false
1771 1563 Acked Write 9 Sent Done Done false
1772 1564 Acked Read 6 Sent Done Done false
1773 1565 Acked Read 5 Sent Done Done false
1774 1566 Acked Write 2 Sent Done Done false
1775 1567 Acked Read 8 Sent Done Done false
1776 1568 Acked Read 1 Sent Done Done false
1777 1569 Acked Write 9 Sent Done Done false
1778 1570 Acked Read 5 Sent Done Done false
1779 1571 Acked Flush 0 Sent Done Done false
1780 1572 Acked Read 9 Sent Done Done false
1781 1573 Acked Read 6 Sent Done Done false
1782 1574 Acked Read 3 Sent Done Done false
1783 1575 Acked Write 2 Sent Done Done false
1784 1576 Acked Read 8 Sent Done Done false
1785 1577 Acked Read 2 Sent Done Done false
1786 1578 Acked Read 2 Sent Done Done false
1787 1579 Acked Write 4 Sent Done Done false
1788 1580 Acked Read 5 Sent Done Done false
1789 1581 Acked Read 1 Sent Done Done false
1790 1582 Acked Write 9 Sent Done Done false
1791 1583 Acked Write 4 Sent Done Done false
1792 1584 Acked Flush 0 Sent Done Done false
1793 1585 Acked Write 10 Sent Done Done false
1794 1586 Acked Write 3 Sent Done Done false
1795 1587 Acked Write 9 Sent Done Done false
1796 1588 Acked Read 6 Sent Done Done false
1797 1589 Acked Write 1 Sent Done Done false
1798 1590 Acked Write 4 Sent Done Done false
1799 1591 Acked Write 6 Sent Done Done false
1800 1592 Acked Read 7 Sent Done Done false
1801 1593 Acked Read 10 Sent Done Done false
1802 1594 Acked Flush 0 Sent Done Done false
1803 1595 Acked Write 7 Sent Done Done false
1804 1596 Acked Read 8 Sent Done Done false
1805 1597 Acked Read 9 Sent Done Done false
1806 1598 Acked Write 1 Sent Done Done false
1807 1599 Acked Write 4 Sent Done Done false
1808 1600 Acked Read 10 Sent Done Done false
1809 1601 Acked Read 1 Sent Done Done false
1810 1602 Acked Write 2 Sent Done Done false
1811 STATES DS:0 DS:1 DS:2 TOTAL
1812 Sent 301 0 0 301
1813 Done 0 301 301 602
1814 Skipped 0 0 0 0
1815 Error 0 0 0 0
1816 Last Flush: Some(JobId(1301)) Some(JobId(1594)) Some(JobId(1594))
1817 Downstairs last five completed: 1301 1300 1299 1298 1297
1818 Upstairs last five completed: 1602 1601 1600 1599 1598
1819 CLIENT: Up:0 ds:301 act:2
18202025-02-25T17:18:35.327ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
18212025-02-25T17:18:35.828ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
18222025-02-25T17:18:36.329ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
18232025-02-25T17:18:36.830ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
18242025-02-25T17:18:37.331ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1825 [8810] got start action
1826 [8810] I am going to restart
1827 Starting downstairs at port 8810
1828 Make output file at "/tmp/dsc/downstairs-8810.txt"
1829 [8810][0] reports Starting
1830 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1210)
1831 [8810][0] reports Running
18322025-02-25T17:18:38.333ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
1833 ----------------------------------------------------------------
1834 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1835 States: Offline Active Active
1836 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1837 1302 Acked Read 6 Sent Done Done false
1838 1303 Acked Read 9 Sent Done Done false
1839 1304 Acked Write 2 Sent Done Done false
1840 1305 Acked Read 7 Sent Done Done false
1841 1306 Acked Read 6 Sent Done Done false
1842 1307 Acked Flush 0 Sent Done Done false
1843 1308 Acked Read 9 Sent Done Done false
1844 1309 Acked Write 4 Sent Done Done false
1845 1310 Acked Read 6 Sent Done Done false
1846 1311 Acked Read 2 Sent Done Done false
1847 1312 Acked Write 9 Sent Done Done false
1848 1313 Acked Write 3 Sent Done Done false
1849 1314 Acked Read 8 Sent Done Done false
1850 1315 Acked Write 4 Sent Done Done false
1851 1316 Acked Read 8 Sent Done Done false
1852 1317 Acked Write 9 Sent Done Done false
1853 1318 Acked Read 3 Sent Done Done false
1854 1319 Acked Read 5 Sent Done Done false
1855 1320 Acked Flush 0 Sent Done Done false
1856 1321 Acked Write 6 Sent Done Done false
1857 1322 Acked Read 2 Sent Done Done false
1858 1323 Acked Read 8 Sent Done Done false
1859 1324 Acked Write 9 Sent Done Done false
1860 1325 Acked Flush 0 Sent Done Done false
1861 1326 Acked Read 8 Sent Done Done false
1862 1327 Acked Read 9 Sent Done Done false
1863 1328 Acked Write 7 Sent Done Done false
1864 1329 Acked Write 6 Sent Done Done false
1865 1330 Acked Flush 0 Sent Done Done false
1866 1331 Acked Write 5 Sent Done Done false
1867 1332 Acked Write 5 Sent Done Done false
1868 1333 Acked Write 1 Sent Done Done false
1869 1334 Acked Read 3 Sent Done Done false
1870 1335 Acked Read 1 Sent Done Done false
1871 1336 Acked Write 10 Sent Done Done false
1872 1337 Acked Write 8 Sent Done Done false
1873 1338 Acked Read 10 Sent Done Done false
1874 1339 Acked Write 3 Sent Done Done false
1875 1340 Acked Read 9 Sent Done Done false
1876 1341 Acked Write 7 Sent Done Done false
1877 1342 Acked Write 6 Sent Done Done false
1878 1343 Acked Write 4 Sent Done Done false
1879 1344 Acked Write 10 Sent Done Done false
1880 1345 Acked Flush 0 Sent Done Done false
1881 1346 Acked Write 4 Sent Done Done false
1882 1347 Acked Read 9 Sent Done Done false
1883 1348 Acked Read 10 Sent Done Done false
1884 1349 Acked Read 6 Sent Done Done false
1885 1350 Acked Read 5 Sent Done Done false
1886 1351 Acked Read 3 Sent Done Done false
1887 1352 Acked Read 10 Sent Done Done false
1888 1353 Acked Read 3 Sent Done Done false
1889 1354 Acked Write 3 Sent Done Done false
1890 1355 Acked Read 7 Sent Done Done false
1891 1356 Acked Write 10 Sent Done Done false
1892 1357 Acked Write 7 Sent Done Done false
1893 1358 Acked Read 9 Sent Done Done false
1894 1359 Acked Read 10 Sent Done Done false
1895 1360 Acked Write 5 Sent Done Done false
1896 1361 Acked Read 1 Sent Done Done false
1897 1362 Acked Read 6 Sent Done Done false
1898 1363 Acked Read 7 Sent Done Done false
1899 1364 Acked Write 3 Sent Done Done false
1900 1365 Acked Write 5 Sent Done Done false
1901 1366 Acked Flush 0 Sent Done Done false
1902 1367 Acked Write 9 Sent Done Done false
1903 1368 Acked Write 4 Sent Done Done false
1904 1369 Acked Write 6 Sent Done Done false
1905 1370 Acked Read 1 Sent Done Done false
1906 1371 Acked Write 2 Sent Done Done false
1907 1372 Acked Write 5 Sent Done Done false
1908 1373 Acked Flush 0 Sent Done Done false
1909 1374 Acked Read 1 Sent Done Done false
1910 1375 Acked Read 1 Sent Done Done false
1911 1376 Acked Write 1 Sent Done Done false
1912 1377 Acked Write 4 Sent Done Done false
1913 1378 Acked Read 5 Sent Done Done false
1914 1379 Acked Read 6 Sent Done Done false
1915 1380 Acked Write 3 Sent Done Done false
1916 1381 Acked Write 8 Sent Done Done false
1917 1382 Acked Write 4 Sent Done Done false
1918 1383 Acked Read 7 Sent Done Done false
1919 1384 Acked Write 1 Sent Done Done false
1920 1385 Acked Read 1 Sent Done Done false
1921 1386 Acked Write 10 Sent Done Done false
1922 1387 Acked Read 2 Sent Done Done false
1923 1388 Acked Write 8 Sent Done Done false
1924 1389 Acked Write 2 Sent Done Done false
1925 1390 Acked Write 10 Sent Done Done false
1926 1391 Acked Read 8 Sent Done Done false
1927 1392 Acked Read 2 Sent Done Done false
1928 1393 Acked Read 8 Sent Done Done false
1929 1394 Acked Read 10 Sent Done Done false
1930 1395 Acked Write 9 Sent Done Done false
1931 1396 Acked Write 3 Sent Done Done false
1932 1397 Acked Write 7 Sent Done Done false
1933 1398 Acked Flush 0 Sent Done Done false
1934 1399 Acked Write 4 Sent Done Done false
1935 1400 Acked Write 4 Sent Done Done false
1936 1401 Acked Read 9 Sent Done Done false
1937 1402 Acked Read 6 Sent Done Done false
1938 1403 Acked Write 1 Sent Done Done false
1939 1404 Acked Read 10 Sent Done Done false
1940 1405 Acked Write 6 Sent Done Done false
1941 1406 Acked Read 3 Sent Done Done false
1942 1407 Acked Read 7 Sent Done Done false
1943 1408 Acked Write 8 Sent Done Done false
1944 1409 Acked Write 8 Sent Done Done false
1945 1410 Acked Read 4 Sent Done Done false
1946 1411 Acked Read 10 Sent Done Done false
1947 1412 Acked Read 5 Sent Done Done false
1948 1413 Acked Write 2 Sent Done Done false
1949 1414 Acked Write 3 Sent Done Done false
1950 1415 Acked Read 2 Sent Done Done false
1951 1416 Acked Write 3 Sent Done Done false
1952 1417 Acked Write 7 Sent Done Done false
1953 1418 Acked Read 1 Sent Done Done false
1954 1419 Acked Write 2 Sent Done Done false
1955 1420 Acked Read 4 Sent Done Done false
1956 1421 Acked Write 2 Sent Done Done false
1957 1422 Acked Write 8 Sent Done Done false
1958 1423 Acked Read 8 Sent Done Done false
1959 1424 Acked Write 8 Sent Done Done false
1960 1425 Acked Flush 0 Sent Done Done false
1961 1426 Acked Flush 0 Sent Done Done false
1962 1427 Acked Write 2 Sent Done Done false
1963 1428 Acked Read 1 Sent Done Done false
1964 1429 Acked Flush 0 Sent Done Done false
1965 1430 Acked Write 5 Sent Done Done false
1966 1431 Acked Read 5 Sent Done Done false
1967 1432 Acked Read 5 Sent Done Done false
1968 1433 Acked Read 2 Sent Done Done false
1969 1434 Acked Write 2 Sent Done Done false
1970 1435 Acked Read 1 Sent Done Done false
1971 1436 Acked Write 9 Sent Done Done false
1972 1437 Acked Write 5 Sent Done Done false
1973 1438 Acked Write 4 Sent Done Done false
1974 1439 Acked Read 10 Sent Done Done false
1975 1440 Acked Read 4 Sent Done Done false
1976 1441 Acked Write 6 Sent Done Done false
1977 1442 Acked Read 4 Sent Done Done false
1978 1443 Acked Write 3 Sent Done Done false
1979 1444 Acked Flush 0 Sent Done Done false
1980 1445 Acked Write 2 Sent Done Done false
1981 1446 Acked Write 5 Sent Done Done false
1982 1447 Acked Write 2 Sent Done Done false
1983 1448 Acked Write 2 Sent Done Done false
1984 1449 Acked Write 8 Sent Done Done false
1985 1450 Acked Write 9 Sent Done Done false
1986 1451 Acked Write 3 Sent Done Done false
1987 1452 Acked Write 1 Sent Done Done false
1988 1453 Acked Read 6 Sent Done Done false
1989 1454 Acked Write 8 Sent Done Done false
1990 1455 Acked Read 8 Sent Done Done false
1991 1456 Acked Read 10 Sent Done Done false
1992 1457 Acked Read 2 Sent Done Done false
1993 1458 Acked Read 2 Sent Done Done false
1994 1459 Acked Read 8 Sent Done Done false
1995 1460 Acked Write 4 Sent Done Done false
1996 1461 Acked Write 7 Sent Done Done false
1997 1462 Acked Flush 0 Sent Done Done false
1998 1463 Acked Read 2 Sent Done Done false
1999 1464 Acked Flush 0 Sent Done Done false
2000 1465 Acked Write 8 Sent Done Done false
2001 1466 Acked Flush 0 Sent Done Done false
2002 1467 Acked Read 9 Sent Done Done false
2003 1468 Acked Flush 0 Sent Done Done false
2004 1469 Acked Write 7 Sent Done Done false
2005 1470 Acked Read 4 Sent Done Done false
2006 1471 Acked Read 4 Sent Done Done false
2007 1472 Acked Flush 0 Sent Done Done false
2008 1473 Acked Read 7 Sent Done Done false
2009 1474 Acked Read 4 Sent Done Done false
2010 1475 Acked Read 3 Sent Done Done false
2011 1476 Acked Read 3 Sent Done Done false
2012 1477 Acked Read 2 Sent Done Done false
2013 1478 Acked Write 8 Sent Done Done false
2014 1479 Acked Write 8 Sent Done Done false
2015 1480 Acked Flush 0 Sent Done Done false
2016 1481 Acked Read 7 Sent Done Done false
2017 1482 Acked Read 9 Sent Done Done false
2018 1483 Acked Write 5 Sent Done Done false
2019 1484 Acked Read 2 Sent Done Done false
2020 1485 Acked Read 5 Sent Done Done false
2021 1486 Acked Read 6 Sent Done Done false
2022 1487 Acked Read 3 Sent Done Done false
2023 1488 Acked Read 7 Sent Done Done false
2024 1489 Acked Write 6 Sent Done Done false
2025 1490 Acked Flush 0 Sent Done Done false
2026 1491 Acked Read 1 Sent Done Done false
2027 1492 Acked Write 1 Sent Done Done false
2028 1493 Acked Read 10 Sent Done Done false
2029 1494 Acked Write 5 Sent Done Done false
2030 1495 Acked Write 6 Sent Done Done false
2031 1496 Acked Write 7 Sent Done Done false
2032 1497 Acked Read 9 Sent Done Done false
2033 1498 Acked Read 10 Sent Done Done false
2034 1499 Acked Read 1 Sent Done Done false
2035 1500 Acked Read 3 Sent Done Done false
2036 1501 Acked Read 8 Sent Done Done false
2037 1502 Acked Write 8 Sent Done Done false
2038 1503 Acked Write 10 Sent Done Done false
2039 1504 Acked Flush 0 Sent Done Done false
2040 1505 Acked Write 4 Sent Done Done false
2041 1506 Acked Read 6 Sent Done Done false
2042 1507 Acked Flush 0 Sent Done Done false
2043 1508 Acked Read 2 Sent Done Done false
2044 1509 Acked Read 9 Sent Done Done false
2045 1510 Acked Write 10 Sent Done Done false
2046 1511 Acked Read 1 Sent Done Done false
2047 1512 Acked Flush 0 Sent Done Done false
2048 1513 Acked Write 3 Sent Done Done false
2049 1514 Acked Read 6 Sent Done Done false
2050 1515 Acked Read 2 Sent Done Done false
2051 1516 Acked Read 9 Sent Done Done false
2052 1517 Acked Read 10 Sent Done Done false
2053 1518 Acked Read 3 Sent Done Done false
2054 1519 Acked Read 7 Sent Done Done false
2055 1520 Acked Read 8 Sent Done Done false
2056 1521 Acked Write 8 Sent Done Done false
2057 1522 Acked Read 7 Sent Done Done false
2058 1523 Acked Write 1 Sent Done Done false
2059 1524 Acked Read 2 Sent Done Done false
2060 1525 Acked Read 8 Sent Done Done false
2061 1526 Acked Read 7 Sent Done Done false
2062 1527 Acked Write 5 Sent Done Done false
2063 1528 Acked Write 7 Sent Done Done false
2064 1529 Acked Read 9 Sent Done Done false
2065 1530 Acked Read 5 Sent Done Done false
2066 1531 Acked Write 7 Sent Done Done false
2067 1532 Acked Write 1 Sent Done Done false
2068 1533 Acked Write 4 Sent Done Done false
2069 1534 Acked Write 4 Sent Done Done false
2070 1535 Acked Read 2 Sent Done Done false
2071 1536 Acked Flush 0 Sent Done Done false
2072 1537 Acked Write 4 Sent Done Done false
2073 1538 Acked Read 4 Sent Done Done false
2074 1539 Acked Write 9 Sent Done Done false
2075 1540 Acked Read 5 Sent Done Done false
2076 1541 Acked Write 9 Sent Done Done false
2077 1542 Acked Write 4 Sent Done Done false
2078 1543 Acked Read 7 Sent Done Done false
2079 1544 Acked Read 4 Sent Done Done false
2080 1545 Acked Flush 0 Sent Done Done false
2081 1546 Acked Write 4 Sent Done Done false
2082 1547 Acked Write 9 Sent Done Done false
2083 1548 Acked Write 2 Sent Done Done false
2084 1549 Acked Read 2 Sent Done Done false
2085 1550 Acked Read 6 Sent Done Done false
2086 1551 Acked Read 10 Sent Done Done false
2087 1552 Acked Read 4 Sent Done Done false
2088 1553 Acked Write 3 Sent Done Done false
2089 1554 Acked Read 8 Sent Done Done false
2090 1555 Acked Write 10 Sent Done Done false
2091 1556 Acked Write 9 Sent Done Done false
2092 1557 Acked Read 6 Sent Done Done false
2093 1558 Acked Flush 0 Sent Done Done false
2094 1559 Acked Read 5 Sent Done Done false
2095 1560 Acked Read 7 Sent Done Done false
2096 1561 Acked Write 2 Sent Done Done false
2097 1562 Acked Read 3 Sent Done Done false
2098 1563 Acked Write 9 Sent Done Done false
2099 1564 Acked Read 6 Sent Done Done false
2100 1565 Acked Read 5 Sent Done Done false
2101 1566 Acked Write 2 Sent Done Done false
2102 1567 Acked Read 8 Sent Done Done false
2103 1568 Acked Read 1 Sent Done Done false
2104 1569 Acked Write 9 Sent Done Done false
2105 1570 Acked Read 5 Sent Done Done false
2106 1571 Acked Flush 0 Sent Done Done false
2107 1572 Acked Read 9 Sent Done Done false
2108 1573 Acked Read 6 Sent Done Done false
2109 1574 Acked Read 3 Sent Done Done false
2110 1575 Acked Write 2 Sent Done Done false
2111 1576 Acked Read 8 Sent Done Done false
2112 1577 Acked Read 2 Sent Done Done false
2113 1578 Acked Read 2 Sent Done Done false
2114 1579 Acked Write 4 Sent Done Done false
2115 1580 Acked Read 5 Sent Done Done false
2116 1581 Acked Read 1 Sent Done Done false
2117 1582 Acked Write 9 Sent Done Done false
2118 1583 Acked Write 4 Sent Done Done false
2119 1584 Acked Flush 0 Sent Done Done false
2120 1585 Acked Write 10 Sent Done Done false
2121 1586 Acked Write 3 Sent Done Done false
2122 1587 Acked Write 9 Sent Done Done false
2123 1588 Acked Read 6 Sent Done Done false
2124 1589 Acked Write 1 Sent Done Done false
2125 1590 Acked Write 4 Sent Done Done false
2126 1591 Acked Write 6 Sent Done Done false
2127 1592 Acked Read 7 Sent Done Done false
2128 1593 Acked Read 10 Sent Done Done false
2129 1594 Acked Flush 0 Sent Done Done false
2130 1595 Acked Write 7 Sent Done Done false
2131 1596 Acked Read 8 Sent Done Done false
2132 1597 Acked Read 9 Sent Done Done false
2133 1598 Acked Write 1 Sent Done Done false
2134 1599 Acked Write 4 Sent Done Done false
2135 1600 Acked Read 10 Sent Done Done false
2136 1601 Acked Read 1 Sent Done Done false
2137 1602 Acked Write 2 Sent Done Done false
2138 STATES DS:0 DS:1 DS:2 TOTAL
2139 Sent 301 0 0 301
2140 Done 0 301 301 602
2141 Skipped 0 0 0 0
2142 Error 0 0 0 0
2143 Last Flush: Some(JobId(1301)) Some(JobId(1594)) Some(JobId(1594))
2144 Downstairs last five completed: 1301 1300 1299 1298 1297
2145 Upstairs last five completed: 1602 1601 1600 1599 1598
2146 CLIENT: Up:0 ds:301 act:2
21472025-02-25T17:18:39.335ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21482025-02-25T17:18:40.110ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21492025-02-25T17:18:40.110ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41972) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21502025-02-25T17:18:40.111ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21512025-02-25T17:18:40.112ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21522025-02-25T17:18:40.112ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21532025-02-25T17:18:40.112ZINFOcrucible: send last flush ID to this DS: Some(JobId(1301)) = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21542025-02-25T17:18:40.112ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21552025-02-25T17:18:40.112ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(1301)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21562025-02-25T17:18:40.112ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(1301)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2157 ----------------------------------------------------------------
2158 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2159 Downstairs last five completed: 1603 1602 1601 1600 1599
2160 Upstairs last five completed: 1603 1602 1601 1600 1599
2161 CLIENT: Up:0 ds:0 act:3
2162 Replay: All jobs finished
2163 Main task has work to do, go find it
2164 got dsc Stop(2)
2165 stop 2
2166 Feb 25 17:18:42.359 INFO request completed, latency_us: 387, response_code: 204, uri: /stop/cid/2, method: POST, req_id: d7d3e02b-94cc-4a77-88d3-fd2c0edcf58f, remote_addr: [8830] Got stop action so:false kr:false
2167 127.0.0.1:52576, local_addr: 127.0.0.1:9998
2168 Feb 25 17:18:42.360 INFO request completed, latency_us: 156, response_code: 200, uri: /state/cid/2, method: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:18:42.360407253Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
2169 GET, req_id: 47ae647e-832f-4c4e-b16a-b7fc3db91767, remote_addr{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:18:42.360487912Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
2170 : 127.0.0.1:52576, {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:18:42.360544264Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
2171 local_addr: 127.0.0.1:9998
21722025-02-25T17:18:42.360ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2173 [8830] Exited with: ExitStatus(unix_wait_status(9))
2174 [8830][2] reports Exit
2175 Feb 25 17:18:46.362 INFO request completed, latency_us: 327, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1475f3a7-28d3-437d-a1af-3297eeb54d93, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
2176 001/300 Write block 2664 len 36864 data: 2 2 2 2 2 2 2 2 2
2177 002/300 Write block 1320 len 20480 data: 2 2 2 2 2
2178 003/300 Read block 4407 len 16384
2179 004/300 Read block 872 len 28672
2180 005/300 Write block 3936 len 40960 data: 2 3 3 3 3 3 3 3 3 3
2181 006/300 Write block 1793 len 12288 data: 2 2 2
2182 007/300 Write block 939 len 24576 data: 2 2 2 2 2 2
2183 008/300 Flush
2184 009/300 Write block 2210 len 20480 data: 2 3 2 2 2
2185 010/300 Read block 3797 len 28672
2186 011/300 Read block 2651 len 32768
2187 012/300 Read block 4146 len 4096
2188 013/300 Write block 3811 len 32768 data: 2 2 2 2 3 3 3 3
2189 014/300 Read block 767 len 40960
2190 015/300 Write block 3132 len 40960 data: 2 2 2 2 2 2 2 2 3 3
2191 016/300 Read block 4630 len 36864
2192 017/300 Flush
2193 018/300 Write block 1336 len 36864 data: 2 2 2 2 2 2 2 2 2
2194 019/300 Write block 2445 len 24576 data: 2 2 2 2 2 2
2195 020/300 Write block 1483 len 32768 data: 2 2 2 2 2 2 2 2
2196 021/300 Read block 3451 len 20480
2197 022/300 Read block 4865 len 8192
2198 023/300 Write block 3714 len 36864 data: 2 2 2 3 3 4 3 3 3
2199 024/300 Read block 359 len 4096
2200 025/300 Write block 4264 len 36864 data: 3 3 3 3 3 3 3 2 2
2201 026/300 Read block 1634 len 28672
2202 027/300 Read block 4093 len 20480
2203 028/300 Write block 4127 len 36864 data: 2 2 2 2 2 2 2 2 2
2204 029/300 Read block 1239 len 28672
2205 030/300 Write block 2835 len 24576 data: 2 2 2 2 2 2
2206 031/300 Read block 613 len 16384
2207 032/300 Read block 3064 len 4096
2208 033/300 Flush
2209 034/300 Write block 99 len 12288 data: 2 2 2
2210 035/300 Read block 1239 len 36864
2211 036/300 Read block 4300 len 28672
2212 037/300 Write block 290 len 8192 data: 2 2
2213 038/300 Read block 182 len 32768
2214 039/300 Write block 4134 len 12288 data: 3 3 2
2215 040/300 Read block 4899 len 16384
2216 041/300 Flush
2217 042/300 Write block 3162 len 24576 data: 3 3 2 2 2 2
2218 043/300 Write block 3847 len 40960 data: 2 2 2 2 2 2 2 3 3 3
2219 044/300 Flush
2220 045/300 Write block 3263 len 12288 data: 2 2 2
2221 046/300 Write block 4389 len 24576 data: 2 2 2 2 2 2
2222 047/300 Read block 1347 len 36864
2223 048/300 Write block 1091 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2224 049/300 Read block 3178 len 12288
2225 050/300 Write block 627 len 28672 data: 2 2 2 2 2 2 2
2226 051/300 Read block 3394 len 4096
2227 052/300 Write block 3790 len 32768 data: 3 3 3 3 3 3 3 3
2228 053/300 Write block 2957 len 20480 data: 2 2 2 2 3
2229 054/300 Read block 4790 len 20480
2230 055/300 Read block 3281 len 36864
2231 056/300 Flush
2232 057/300 Read block 2193 len 20480
2233 058/300 Read block 1002 len 20480
2234 059/300 Write block 3223 len 36864 data: 2 2 2 2 2 2 2 2 2
2235 060/300 Read block 2424 len 28672
2236 061/300 Read block 4861 len 24576
2237 062/300 Write block 3584 len 32768 data: 3 3 3 3 3 3 3 2
2238 063/300 Write block 4422 len 8192 data: 3 3
2239 064/300 Read block 3909 len 8192
2240 065/300 Write block 4322 len 4096 data: 2
2241 066/300 Read block 1346 len 36864
2242 067/300 Write block 1291 len 16384 data: 2 2 2 2
2243 068/300 Read block 1863 len 32768
2244 069/300 Read block 3200 len 24576
2245 070/300 Read block 2478 len 32768
2246 071/300 Read block 1576 len 32768
2247 072/300 Read block 1559 len 24576
2248 073/300 Write block 787 len 28672 data: 3 3 3 3 3 3 3
2249 074/300 Read block 4767 len 16384
2250 075/300 Read block 470 len 36864
2251 076/300 Read block 1299 len 28672
2252 077/300 Write block 991 len 24576 data: 2 3 3 3 3 3
2253 078/300 Read block 1925 len 36864
2254 079/300 Write block 1371 len 20480 data: 2 2 2 2 2
2255 080/300 Read block 1924 len 28672
2256 081/300 Flush
2257 082/300 Write block 2572 len 12288 data: 2 2 2
2258 083/300 Flush
2259 084/300 Write block 216 len 28672 data: 2 2 2 2 2 2 2
2260 085/300 Flush
2261 086/300 Write block 1338 len 36864 data: 3 3 3 3 3 3 3 2 2
2262 087/300 Write block 3544 len 12288 data: 2 2 2
2263 088/300 Flush
2264 089/300 Write block 2877 len 4096 data: 2
2265 090/300 Read block 3526 len 8192
2266 091/300 Read block 277 len 24576
2267 092/300 Read block 3854 len 36864
2268 093/300 Read block 4164 len 16384
2269 094/300 Read block 514 len 32768
2270 095/300 Write block 4227 len 4096 data: 2
2271 096/300 Read block 4133 len 32768
2272 097/300 Write block 1987 len 12288 data: 2 2 2
2273 098/300 Read block 377 len 24576
2274 099/300 Write block 57 len 12288 data: 2 2 2
2275 100/300 Write block 36 len 12288 data: 2 2 2
2276 101/300 Read block 3891 len 24576
2277 102/300 Write block 2329 len 28672 data: 2 2 2 3 3 3 3
2278 103/300 Write block 4531 len 20480 data: 3 2 2 2 2
2279 104/300 Write block 3728 len 8192 data: 2 2
2280 105/300 Write block 3930 len 20480 data: 3 3 2 2 2
2281 106/300 Write block 3931 len 8192 data: 4 3
2282 107/300 Write block 2672 len 28672 data: 3 2 2 2 2 2 2
2283 108/300 Read block 4725 len 12288
2284 109/300 Write block 422 len 16384 data: 3 3 3 2
2285 110/300 Read block 1186 len 40960
2286 111/300 Read block 1259 len 28672
2287 112/300 Read block 1878 len 12288
2288 113/300 Write block 1975 len 36864 data: 4 4 3 3 3 2 2 2 2
2289 114/300 Write block 1712 len 36864 data: 2 2 2 2 2 2 2 3 3
2290 115/300 Read block 3415 len 12288
2291 116/300 Read block 1458 len 32768
2292 117/300 Read block 3669 len 24576
2293 118/300 Write block 1916 len 12288 data: 2 2 2
2294 119/300 Write block 3617 len 16384 data: 2 2 2 2
2295 120/300 Flush
2296 121/300 Read block 1682 len 16384
2297 122/300 Read block 3465 len 40960
2298 123/300 Read block 863 len 20480
2299 124/300 Flush
2300 125/300 Flush
2301 126/300 Flush
2302 127/300 Read block 2919 len 4096
2303 128/300 Write block 4974 len 16384 data: 2 2 2 2
2304 129/300 Write block 3352 len 36864 data: 3 2 2 2 2 2 2 2 3
2305 130/300 Write block 1813 len 32768 data: 2 2 2 2 2 2 2 2
2306 131/300 Write block 3299 len 20480 data: 2 2 2 2 2
2307 132/300 Read block 4742 len 20480
2308 133/300 Write block 209 len 16384 data: 2 2 2 2
2309 134/300 Read block 4038 len 16384
2310 135/300 Write block 315 len 36864 data: 3 2 2 2 2 2 2 2 2
2311 136/300 Write block 3002 len 36864 data: 2 2 2 2 2 2 2 2 2
2312 137/300 Flush
2313 138/300 Read block 982 len 40960
2314 139/300 Flush
2315 140/300 Read block 4458 len 12288
2316 141/300 Read block 3220 len 8192
2317 142/300 Read block 2390 len 16384
2318 143/300 Write block 1754 len 16384 data: 3 3 3 2
2319 144/300 Flush
2320 145/300 Read block 3977 len 28672
2321 146/300 Read block 1244 len 12288
2322 147/300 Read block 3114 len 20480
2323 148/300 Write block 1477 len 28672 data: 2 2 2 2 2 2 3
2324 149/300 Read block 1281 len 32768
2325 150/300 Read block 4966 len 28672
2326 151/300 Read block 253 len 16384
2327 152/300 Read block 855 len 12288
2328 153/300 Read block 1907 len 40960
2329 154/300 Write block 2733 len 32768 data: 2 2 2 2 2 2 2 2
2330 155/300 Write block 4085 len 8192 data: 2 2
2331 156/300 Read block 637 len 8192
2332 157/300 Write block 1808 len 8192 data: 2 2
2333 158/300 Read block 4135 len 12288
2334 159/300 Read block 1976 len 40960
2335 160/300 Read block 3721 len 12288
2336 161/300 Read block 3241 len 40960
2337 162/300 Read block 3739 len 24576
2338 163/300 Write block 3899 len 8192 data: 3 2
2339 164/300 Read block 3636 len 36864
2340 165/300 Read block 2595 len 20480
2341 166/300 Write block 2743 len 36864 data: 2 2 2 2 2 2 2 2 2
2342 167/300 Read block 2257 len 32768
2343 168/300 Read block 4079 len 16384
2344 169/300 Write block 3508 len 4096 data: 2
2345 170/300 Write block 3725 len 24576 data: 3 3 2 3 3 2
2346 171/300 Read block 2793 len 8192
2347 172/300 Read block 2486 len 32768
2348 173/300 Flush
2349 174/300 Read block 55 len 28672
2350 175/300 Read block 1601 len 28672
2351 176/300 Write block 4666 len 36864 data: 3 2 2 2 2 2 2 2 2
2352 177/300 Write block 1167 len 16384 data: 3 3 3 3
2353 178/300 Read block 3598 len 28672
2354 179/300 Write block 4337 len 36864 data: 3 3 3 3 2 2 2 2 3
2355 180/300 Read block 890 len 20480
2356 181/300 Write block 1530 len 24576 data: 3 3 2 2 2 2
2357 182/300 Write block 4240 len 4096 data: 2
2358 183/300 Write block 1628 len 8192 data: 2 2
2359 184/300 Write block 3082 len 20480 data: 3 3 3 2 2
2360 185/300 Write block 3601 len 36864 data: 2 2 2 2 2 2 2 2 2
2361 186/300 Write block 202 len 28672 data: 2 2 2 2 2 2 2
2362 187/300 Read block 2978 len 28672
2363 188/300 Read block 2645 len 20480
2364 189/300 Read block 4935 len 20480
2365 190/300 Read block 3483 len 16384
2366 191/300 Read block 4100 len 12288
2367 192/300 Write block 591 len 32768 data: 2 2 2 2 2 2 2 2
2368 193/300 Write block 4661 len 40960 data: 3 3 3 3 3 4 3 3 3 3
2369 194/300 Read block 1556 len 36864
2370 195/300 Flush
2371 196/300 Write block 1191 len 20480 data: 2 2 2 2 2
2372 197/300 Read block 1284 len 20480
2373 198/300 Read block 2659 len 24576
2374 199/300 Write block 4746 len 36864 data: 2 2 3 3 3 3 4 4 4
2375 200/300 Write block 1064 len 24576 data: 2 2 2 2 2 2
2376 201/300 Read block 1426 len 8192
2377 202/300 Read block 3878 len 24576
2378 203/300 Read block 2128 len 24576
2379 204/300 Read block 1110 len 40960
2380 205/300 Write block 4968 len 4096 data: 3
2381 206/300 Flush
2382 207/300 Read block 2220 len 28672
2383 208/300 Read block 3301 len 36864
2384 209/300 Write block 3001 len 32768 data: 2 3 3 3 3 3 3 3
2385 210/300 Read block 1879 len 28672
2386 211/300 Read block 3381 len 4096
2387 212/300 Read block 3272 len 20480
2388 213/300 Read block 600 len 40960
2389 214/300 Read block 2115 len 12288
2390 215/300 Write block 2871 len 12288 data: 2 2 2
2391 216/300 Flush
2392 217/300 Write block 177 len 4096 data: 4
2393 218/300 Read block 2452 len 36864
2394 219/300 Read block 2041 len 36864
2395 220/300 Read block 659 len 4096
2396 221/300 Read block 2731 len 4096
2397 222/300 Read block 2567 len 24576
2398 223/300 Read block 1081 len 32768
2399 224/300 Write block 1459 len 4096 data: 2
2400 225/300 Write block 2998 len 20480 data: 2 2 2 3 4
2401 226/300 Read block 2627 len 32768
2402 227/300 Write block 4238 len 36864 data: 2 2 3 2 2 2 2 2 2
2403 228/300 Write block 1710 len 40960 data: 2 2 3 3 3 3 3 3 3 4
2404 229/300 Write block 266 len 12288 data: 3 3 3
2405 230/300 Flush
2406 231/300 Read block 3468 len 20480
2407 232/300 Read block 1254 len 4096
2408 233/300 Read block 3703 len 16384
2409 234/300 Flush
2410 235/300 Read block 423 len 12288
2411 236/300 Write block 4208 len 32768 data: 2 2 2 2 3 2 2 2
2412 237/300 Flush
2413 238/300 Read block 995 len 40960
2414 239/300 Write block 3904 len 24576 data: 2 2 2 2 2 2
2415 240/300 Read block 289 len 40960
2416 241/300 Write block 1668 len 32768 data: 3 3 3 3 3 3 3 3
2417 242/300 Write block 3301 len 20480 data: 3 3 3 2 2
2418 243/300 Read block 1813 len 32768
2419 244/300 Read block 3435 len 4096
2420 245/300 Write block 2994 len 8192 data: 2 2
2421 246/300 Read block 2085 len 16384
2422 247/300 Write block 886 len 40960 data: 3 3 3 3 3 3 2 3 3 3
2423 248/300 Write block 4628 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2424 249/300 Read block 1478 len 20480
2425 250/300 Write block 2261 len 20480 data: 2 2 2 2 2
2426 251/300 Write block 4022 len 28672 data: 3 2 2 2 2 2 2
2427 252/300 Flush
2428 253/300 Write block 4587 len 24576 data: 2 2 2 2 2 2
2429 254/300 Write block 3053 len 20480 data: 2 2 2 2 3
2430 255/300 Write block 4431 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2431 256/300 Read block 3419 len 8192
2432 257/300 Write block 1641 len 36864 data: 2 2 2 2 3 4 4 4 4
2433 258/300 Write block 4849 len 16384 data: 2 2 2 2
2434 259/300 Write block 4696 len 36864 data: 2 2 2 2 2 3 3 3 3
2435 260/300 Read block 3042 len 40960
2436 261/300 Write block 1602 len 8192 data: 2 2
2437 262/300 Write block 1954 len 36864 data: 3 2 2 2 2 2 3 3 2
2438 263/300 Read block 4425 len 12288
2439 264/300 Flush
2440 265/300 Write block 4197 len 20480 data: 2 2 2 2 2
2441 266/300 Flush
2442 267/300 Write block 2902 len 4096 data: 3
2443 268/300 Write block 4846 len 20480 data: 2 2 2 3 3
2444 269/300 Write block 4347 len 8192 data: 3 3
2445 270/300 Read block 717 len 4096
2446 271/300 Read block 2876 len 36864
2447 272/300 Flush
2448 273/300 Write block 2893 len 20480 data: 2 2 2 2 2
2449 274/300 Write block 4021 len 40960 data: 3 4 3 3 3 3 3 3 2 2
2450 275/300 Write block 2734 len 28672 data: 3 3 3 3 3 3 3
2451 276/300 Write block 4798 len 28672 data: 3 3 3 3 3 3 3
2452 277/300 Write block 2323 len 4096 data: 3
2453 278/300 Flush
2454 279/300 Write block 2918 len 28672 data: 2 2 2 2 2 3 4
2455 280/300 Read block 3816 len 40960
2456 281/300 Write block 1314 len 12288 data: 3 2 2
2457 282/300 Read block 498 len 36864
2458 283/300 Write block 3876 len 24576 data: 2 2 2 2 2 2
2459 284/300 Write block 1287 len 32768 data: 2 2 2 2 3 3 3 3
2460 285/300 Read block 2411 len 4096
2461 286/300 Write block 580 len 28672 data: 3 3 2 2 2 2 2
2462 287/300 Read block 3096 len 8192
2463 288/300 Write block 4123 len 36864 data: 3 3 2 2 3 3 3 3 3
2464 289/300 Write block 3346 len 16384 data: 2 2 2 2
2465 290/300 Read block 2458 len 36864
2466 291/300 Flush
2467 292/300 Read block 3500 len 32768
2468 293/300 Write block 1169 len 8192 data: 4 4
2469 294/300 Flush
2470 295/300 Read block 2973 len 20480
2471 296/300 Flush
2472 297/300 Read block 1910 len 40960
2473 298/300 Write block 2291 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2474 299/300 Write block 561 len 20480 data: 2 2 2 2 2
2475 300/300 Read block 697 len 8192
2476 301/300 Read block 732 len 12288
2477 Main task has work to do, go find it
2478 got dsc Start(2)
2479 start 2
2480 Feb 25 17:18:46.615 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/2, method: POST, req_id: a77a014b-6ad1-42cd-8f10-701a88bf0e4f, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
2481 Replay: started 2, returned:Ok(())
2482 ----------------------------------------------------------------
2483 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2484 States: Active Active Offline
2485 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2486 1604 Acked Write 9 Done Done Sent false
2487 1605 Acked Write 5 Done Done Sent false
2488 1606 Acked Read 4 Done Done Sent false
2489 1607 Acked Read 7 Done Done Sent false
2490 1608 Acked Write 10 Done Done Sent false
2491 1609 Acked Write 3 Done Done Sent false
2492 1610 Acked Write 6 Done Done Sent false
2493 1611 Acked Flush 0 Done Done Sent false
2494 1612 Acked Write 5 Done Done Sent false
2495 1613 Acked Read 7 Done Done Sent false
2496 1614 Acked Read 8 Done Done Sent false
2497 1615 Acked Read 1 Done Done Sent false
2498 1616 Acked Write 8 Done Done Sent false
2499 1617 Acked Read 10 Done Done Sent false
2500 1618 Acked Write 10 Done Done Sent false
2501 1619 Acked Read 9 Done Done Sent false
2502 1620 Acked Flush 0 Done Done Sent false
2503 1621 Acked Write 9 Done Done Sent false
2504 1622 Acked Write 6 Done Done Sent false
2505 1623 Acked Write 8 Done Done Sent false
2506 1624 Acked Read 5 Done Done Sent false
2507 1625 Acked Read 2 Done Done Sent false
2508 1626 Acked Write 9 Done Done Sent false
2509 1627 Acked Read 1 Done Done Sent false
2510 1628 Acked Write 9 Done Done Sent false
2511 1629 Acked Read 7 Done Done Sent false
2512 1630 Acked Read 5 Done Done Sent false
2513 1631 Acked Write 9 Done Done Sent false
2514 1632 Acked Read 7 Done Done Sent false
2515 1633 Acked Write 6 Done Done Sent false
2516 1634 Acked Read 4 Done Done Sent false
2517 1635 Acked Read 1 Done Done Sent false
2518 1636 Acked Flush 0 Done Done Sent false
2519 1637 Acked Write 3 Done Done Sent false
2520 1638 Acked Read 9 Done Done Sent false
2521 1639 Acked Read 7 Done Done Sent false
2522 1640 Acked Write 2 Done Done Sent false
2523 1641 Acked Read 8 Done Done Sent false
2524 1642 Acked Write 3 Done Done Sent false
2525 1643 Acked Read 4 Done Done Sent false
2526 1644 Acked Flush 0 Done Done Sent false
2527 1645 Acked Write 6 Done Done Sent false
2528 1646 Acked Write 10 Done Done Sent false
2529 1647 Acked Flush 0 Done Done Sent false
2530 1648 Acked Write 3 Done Done Sent false
2531 1649 Acked Write 6 Done Done Sent false
2532 1650 Acked Read 9 Done Done Sent false
2533 1651 Acked Write 10 Done Done Sent false
2534 1652 Acked Read 3 Done Done Sent false
2535 1653 Acked Write 7 Done Done Sent false
2536 1654 Acked Read 1 Done Done Sent false
2537 1655 Acked Write 8 Done Done Sent false
2538 1656 Acked Write 5 Done Done Sent false
2539 1657 Acked Read 5 Done Done Sent false
2540 1658 Acked Read 9 Done Done Sent false
2541 1659 Acked Flush 0 Done Done Sent false
2542 1660 Acked Read 5 Done Done Sent false
2543 1661 Acked Read 5 Done Done Sent false
2544 1662 Acked Write 9 Done Done Sent false
2545 1663 Acked Read 7 Done Done Sent false
2546 1664 Acked Read 6 Done Done Sent false
2547 1665 Acked Write 8 Done Done Sent false
2548 1666 Acked Write 2 Done Done Sent false
2549 1667 Acked Read 2 Done Done Sent false
2550 1668 Acked Write 1 Done Done Sent false
2551 1669 Acked Read 9 Done Done Sent false
2552 1670 Acked Write 4 Done Done Sent false
2553 1671 Acked Read 8 Done Done Sent false
2554 1672 Acked Read 6 Done Done Sent false
2555 1673 Acked Read 8 Done Done Sent false
2556 1674 Acked Read 8 Done Done Sent false
2557 1675 Acked Read 6 Done Done Sent false
2558 1676 Acked Write 7 Done Done Sent false
2559 1677 Acked Read 4 Done Done Sent false
2560 1678 Acked Read 9 Done Done Sent false
2561 1679 Acked Read 7 Done Done Sent false
2562 1680 Acked Write 6 Done Done Sent false
2563 1681 Acked Read 9 Done Done Sent false
2564 1682 Acked Write 5 Done Done Sent false
2565 1683 Acked Read 7 Done Done Sent false
2566 1684 Acked Flush 0 Done Done Sent false
2567 1685 Acked Write 3 Done Done Sent false
2568 1686 Acked Flush 0 Done Done Sent false
2569 1687 Acked Write 7 Done Done Sent false
2570 1688 Acked Flush 0 Done Done Sent false
2571 1689 Acked Write 9 Done Done Sent false
2572 1690 Acked Write 3 Done Done Sent false
2573 1691 Acked Flush 0 Done Done Sent false
2574 1692 Acked Write 1 Done Done Sent false
2575 1693 Acked Read 2 Done Done Sent false
2576 1694 Acked Read 6 Done Done Sent false
2577 1695 Acked Read 9 Done Done Sent false
2578 1696 Acked Read 4 Done Done Sent false
2579 1697 Acked Read 8 Done Done Sent false
2580 1698 Acked Write 1 Done Done Sent false
2581 1699 Acked Read 8 Done Done Sent false
2582 1700 Acked Write 3 Done Done Sent false
2583 1701 Acked Read 6 Done Done Sent false
2584 1702 Acked Write 3 Done Done Sent false
2585 1703 Acked Write 3 Done Done Sent false
2586 1704 Acked Read 6 Done Done Sent false
2587 1705 Acked Write 7 Done Done Sent false
2588 1706 Acked Write 5 Done Done Sent false
2589 1707 Acked Write 2 Done Done Sent false
2590 1708 Acked Write 5 Done Done Sent false
2591 1709 Acked Write 2 Done Done Sent false
2592 1710 Acked Write 7 Done Done Sent false
2593 1711 Acked Read 3 Done Done Sent false
2594 1712 Acked Write 4 Done Done Sent false
2595 1713 Acked Read 10 Done Done Sent false
2596 1714 Acked Read 7 Done Done Sent false
2597 1715 Acked Read 3 Done Done Sent false
2598 1716 Acked Write 9 Done Done Sent false
2599 1717 Acked Write 9 Done Done Sent false
2600 1718 Acked Read 3 Done Done Sent false
2601 1719 Acked Read 8 Done Done Sent false
2602 1720 Acked Read 6 Done Done Sent false
2603 1721 Acked Write 3 Done Done Sent false
2604 1722 Acked Write 4 Done Done Sent false
2605 1723 Acked Flush 0 Done Done Sent false
2606 1724 Acked Read 4 Done Done Sent false
2607 1725 Acked Read 10 Done Done Sent false
2608 1726 Acked Read 5 Done Done Sent false
2609 1727 Acked Flush 0 Done Done Sent false
2610 1728 Acked Flush 0 Done Done Sent false
2611 1729 Acked Flush 0 Done Done Sent false
2612 1730 Acked Read 1 Done Done Sent false
2613 1731 Acked Write 4 Done Done Sent false
2614 1732 Acked Write 9 Done Done Sent false
2615 1733 Acked Write 8 Done Done Sent false
2616 1734 Acked Write 5 Done Done Sent false
2617 1735 Acked Read 5 Done Done Sent false
2618 1736 Acked Write 4 Done Done Sent false
2619 1737 Acked Read 4 Done Done Sent false
2620 1738 Acked Write 9 Done Done Sent false
2621 1739 Acked Write 9 Done Done Sent false
2622 1740 Acked Flush 0 Done Done Sent false
2623 1741 Acked Read 10 Done Done Sent false
2624 1742 Acked Flush 0 Done Done Sent false
2625 1743 Acked Read 3 Done Done Sent false
2626 1744 Acked Read 2 Done Done Sent false
2627 1745 Acked Read 4 Done Done Sent false
2628 1746 Acked Write 4 Done Done Sent false
2629 1747 Acked Flush 0 Done Done Sent false
2630 1748 Acked Read 7 Done Done Sent false
2631 1749 Acked Read 3 Done Done Sent false
2632 1750 Acked Read 5 Done Done Sent false
2633 1751 Acked Write 7 Done Done Sent false
2634 1752 Acked Read 8 Done Done Sent false
2635 1753 Acked Read 7 Done Done Sent false
2636 1754 Acked Read 4 Done Done Sent false
2637 1755 Acked Read 3 Done Done Sent false
2638 1756 Acked Read 10 Done Done Sent false
2639 1757 Acked Write 8 Done Done Sent false
2640 1758 Acked Write 2 Done Done Sent false
2641 1759 Acked Read 2 Done Done Sent false
2642 1760 Acked Write 2 Done Done Sent false
2643 1761 Acked Read 3 Done Done Sent false
2644 1762 Acked Read 10 Done Done Sent false
2645 1763 Acked Read 3 Done Done Sent false
2646 1764 Acked Read 10 Done Done Sent false
2647 1765 Acked Read 6 Done Done Sent false
2648 1766 Acked Write 2 Done Done Sent false
2649 1767 Acked Read 9 Done Done Sent false
2650 1768 Acked Read 5 Done Done Sent false
2651 1769 Acked Write 9 Done Done Sent false
2652 1770 Acked Read 8 Done Done Sent false
2653 1771 Acked Read 4 Done Done Sent false
2654 1772 Acked Write 1 Done Done Sent false
2655 1773 Acked Write 6 Done Done Sent false
2656 1774 Acked Read 2 Done Done Sent false
2657 1775 Acked Read 8 Done Done Sent false
2658 1776 Acked Flush 0 Done Done Sent false
2659 1777 Acked Read 7 Done Done Sent false
2660 1778 Acked Read 7 Done Done Sent false
2661 1779 Acked Write 9 Done Done Sent false
2662 1780 Acked Write 4 Done Done Sent false
2663 1781 Acked Read 7 Done Done Sent false
2664 1782 Acked Write 9 Done Done Sent false
2665 1783 Acked Read 5 Done Done Sent false
2666 1784 Acked Write 6 Done Done Sent false
2667 1785 Acked Write 1 Done Done Sent false
2668 1786 Acked Write 2 Done Done Sent false
2669 1787 Acked Write 5 Done Done Sent false
2670 1788 Acked Write 9 Done Done Sent false
2671 1789 Acked Write 7 Done Done Sent false
2672 1790 Acked Read 7 Done Done Sent false
2673 1791 Acked Read 5 Done Done Sent false
2674 1792 Acked Read 5 Done Done Sent false
2675 1793 Acked Read 4 Done Done Sent false
2676 1794 Acked Read 3 Done Done Sent false
2677 1795 Acked Write 8 Done Done Sent false
2678 1796 Acked Write 10 Done Done Sent false
2679 1797 Acked Read 9 Done Done Sent false
2680 1798 Acked Flush 0 Done Done Sent false
2681 1799 Acked Write 5 Done Done Sent false
2682 1800 Acked Read 5 Done Done Sent false
2683 1801 Acked Read 6 Done Done Sent false
2684 1802 Acked Write 9 Done Done Sent false
2685 1803 Acked Write 6 Done Done Sent false
2686 1804 Acked Read 2 Done Done Sent false
2687 1805 Acked Read 6 Done Done Sent false
2688 1806 Acked Read 6 Done Done Sent false
2689 1807 Acked Read 10 Done Done Sent false
2690 1808 Acked Write 1 Done Done Sent false
2691 1809 Acked Flush 0 Done Done Sent false
2692 1810 Acked Read 7 Done Done Sent false
2693 1811 Acked Read 9 Done Done Sent false
2694 1812 Acked Write 8 Done Done Sent false
2695 1813 Acked Read 7 Done Done Sent false
2696 1814 Acked Read 1 Done Done Sent false
2697 1815 Acked Read 5 Done Done Sent false
2698 1816 Acked Read 10 Done Done Sent false
2699 1817 Acked Read 3 Done Done Sent false
2700 1818 Acked Write 3 Done Done Sent false
2701 1819 Acked Flush 0 Done Done Sent false
2702 1820 Acked Write 1 Done Done Sent false
2703 1821 Acked Read 9 Done Done Sent false
2704 1822 Acked Read 9 Done Done Sent false
2705 1823 Acked Read 1 Done Done Sent false
2706 1824 Acked Read 1 Done Done Sent false
2707 1825 Acked Read 6 Done Done Sent false
2708 1826 Acked Read 8 Done Done Sent false
2709 1827 Acked Write 1 Done Done Sent false
2710 1828 Acked Write 5 Done Done Sent false
2711 1829 Acked Read 8 Done Done Sent false
2712 1830 Acked Write 9 Done Done Sent false
2713 1831 Acked Write 10 Done Done Sent false
2714 1832 Acked Write 3 Done Done Sent false
2715 1833 Acked Flush 0 Done Done Sent false
2716 1834 Acked Read 5 Done Done Sent false
2717 1835 Acked Read 1 Done Done Sent false
2718 1836 Acked Read 4 Done Done Sent false
2719 1837 Acked Flush 0 Done Done Sent false
2720 1838 Acked Read 3 Done Done Sent false
2721 1839 Acked Write 8 Done Done Sent false
2722 1840 Acked Flush 0 Done Done Sent false
2723 1841 Acked Read 10 Done Done Sent false
2724 1842 Acked Write 6 Done Done Sent false
2725 1843 Acked Read 10 Done Done Sent false
2726 1844 Acked Write 8 Done Done Sent false
2727 1845 Acked Write 5 Done Done Sent false
2728 1846 Acked Read 8 Done Done Sent false
2729 1847 Acked Read 1 Done Done Sent false
2730 1848 Acked Write 2 Done Done Sent false
2731 1849 Acked Read 4 Done Done Sent false
2732 1850 Acked Write 10 Done Done Sent false
2733 1851 Acked Write 10 Done Done Sent false
2734 1852 Acked Read 5 Done Done Sent false
2735 1853 Acked Write 5 Done Done Sent false
2736 1854 Acked Write 7 Done Done Sent false
2737 1855 Acked Flush 0 Done Done Sent false
2738 1856 Acked Write 6 Done Done Sent false
2739 1857 Acked Write 5 Done Done Sent false
2740 1858 Acked Write 10 Done Done Sent false
2741 1859 Acked Read 2 Done Done Sent false
2742 1860 Acked Write 9 Done Done Sent false
2743 1861 Acked Write 4 Done Done Sent false
2744 1862 Acked Write 9 Done Done Sent false
2745 1863 Acked Read 10 Done Done Sent false
2746 1864 Acked Write 2 Done Done Sent false
2747 1865 Acked Write 9 Done Done Sent false
2748 1866 Acked Read 3 Done Done Sent false
2749 1867 Acked Flush 0 Done Done Sent false
2750 1868 Acked Write 5 Done Done Sent false
2751 1869 Acked Flush 0 Done Done Sent false
2752 1870 Acked Write 1 Done Done Sent false
2753 1871 Acked Write 5 Done Done Sent false
2754 1872 Acked Write 2 Done Done Sent false
2755 1873 Acked Read 1 Done Done Sent false
2756 1874 Acked Read 9 Done Done Sent false
2757 1875 Acked Flush 0 Done Done Sent false
2758 1876 Acked Write 5 Done Done Sent false
2759 1877 Acked Write 10 Done Done Sent false
2760 1878 Acked Write 7 Done Done Sent false
2761 1879 Acked Write 7 Done Done Sent false
2762 1880 Acked Write 1 Done Done Sent false
2763 1881 Acked Flush 0 Done Done Sent false
2764 1882 Acked Write 7 Done Done Sent false
2765 1883 Acked Read 10 Done Done Sent false
2766 1884 Acked Write 3 Done Done Sent false
2767 1885 Acked Read 9 Done Done Sent false
2768 1886 Acked Write 6 Done Done Sent false
2769 1887 Acked Write 8 Done Done Sent false
2770 1888 Acked Read 1 Done Done Sent false
2771 1889 Acked Write 7 Done Done Sent false
2772 1890 Acked Read 2 Done Done Sent false
2773 1891 Acked Write 9 Done Done Sent false
2774 1892 Acked Write 4 Done Done Sent false
2775 1893 Acked Read 9 Done Done Sent false
2776 1894 Acked Flush 0 Done Done Sent false
2777 1895 Acked Read 8 Done Done Sent false
2778 1896 Acked Write 2 Done Done Sent false
2779 1897 Acked Flush 0 Done Done Sent false
2780 1898 Acked Read 5 Done Done Sent false
2781 1899 Acked Flush 0 Done Done Sent false
2782 1900 Acked Read 10 Done Done Sent false
2783 1901 Acked Write 10 Done Done Sent false
2784 1902 Acked Write 5 Done Done Sent false
2785 1903 Acked Read 2 Done Done Sent false
2786 1904 Acked Read 3 Done Done Sent false
2787 STATES DS:0 DS:1 DS:2 TOTAL
2788 Sent 0 0 301 301
2789 Done 301 301 0 602
2790 Skipped 0 0 0 0
2791 Error 0 0 0 0
2792 Last Flush: Some(JobId(1899)) Some(JobId(1899)) Some(JobId(1603))
2793 Downstairs last five completed: 1603 1602 1601 1600 1599
2794 Upstairs last five completed: 1904 1903 1902 1901 1900
2795 CLIENT: Up:0 ds:301 act:2
27962025-02-25T17:18:47.347ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
27972025-02-25T17:18:48.348ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2798 [8830] got start action
2799 [8830] I am going to restart
2800 Starting downstairs at port 8830
2801 Make output file at "/tmp/dsc/downstairs-8830.txt"
2802 [8830][2] reports Starting
2803 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1211)
2804 [8830][2] reports Running
28052025-02-25T17:18:49.349ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
28062025-02-25T17:18:49.850ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
28072025-02-25T17:18:50.615ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
2808 ----------------------------------------------------------------
2809 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2810 States: Active Active Offline
2811 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2812 1604 Acked Write 9 Done Done Sent false
2813 1605 Acked Write 5 Done Done Sent false
2814 1606 Acked Read 4 Done Done Sent false
2815 1607 Acked Read 7 Done Done Sent false
2816 1608 Acked Write 10 Done Done Sent false
2817 1609 Acked Write 3 Done Done Sent false
2818 1610 Acked Write 6 Done Done Sent false
2819 1611 Acked Flush 0 Done Done Sent false
2820 1612 Acked Write 5 Done Done Sent false
2821 1613 Acked Read 7 Done Done Sent false
2822 1614 Acked Read 8 Done Done Sent false
2823 1615 Acked Read 1 Done Done Sent false
2824 1616 Acked Write 8 Done Done Sent false
2825 1617 Acked Read 10 Done Done Sent false
2826 1618 Acked Write 10 Done Done Sent false
2827 1619 Acked Read 9 Done Done Sent false
2828 1620 Acked Flush 0 Done Done Sent false
2829 1621 Acked Write 9 Done Done Sent false
2830 1622 Acked Write 6 Done Done Sent false
2831 1623 Acked Write 8 Done Done Sent false
2832 1624 Acked Read 5 Done Done Sent false
2833 1625 Acked Read 2 Done Done Sent false
2834 1626 Acked Write 9 Done Done Sent false
2835 1627 Acked Read 1 Done Done Sent false
2836 1628 Acked Write 9 Done Done Sent false
2837 1629 Acked Read 7 Done Done Sent false
2838 1630 Acked Read 5 Done Done Sent false
2839 1631 Acked Write 9 Done Done Sent false
2840 1632 Acked Read 7 Done Done Sent false
2841 1633 Acked Write 6 Done Done Sent false
2842 1634 Acked Read 4 Done Done Sent false
2843 1635 Acked Read 1 Done Done Sent false
2844 1636 Acked Flush 0 Done Done Sent false
2845 1637 Acked Write 3 Done Done Sent false
2846 1638 Acked Read 9 Done Done Sent false
2847 1639 Acked Read 7 Done Done Sent false
2848 1640 Acked Write 2 Done Done Sent false
2849 1641 Acked Read 8 Done Done Sent false
2850 1642 Acked Write 3 Done Done Sent false
2851 1643 Acked Read 4 Done Done Sent false
2852 1644 Acked Flush 0 Done Done Sent false
2853 1645 Acked Write 6 Done Done Sent false
2854 1646 Acked Write 10 Done Done Sent false
2855 1647 Acked Flush 0 Done Done Sent false
2856 1648 Acked Write 3 Done Done Sent false
2857 1649 Acked Write 6 Done Done Sent false
2858 1650 Acked Read 9 Done Done Sent false
2859 1651 Acked Write 10 Done Done Sent false
2860 1652 Acked Read 3 Done Done Sent false
2861 1653 Acked Write 7 Done Done Sent false
2862 1654 Acked Read 1 Done Done Sent false
2863 1655 Acked Write 8 Done Done Sent false
2864 1656 Acked Write 5 Done Done Sent false
2865 1657 Acked Read 5 Done Done Sent false
2866 1658 Acked Read 9 Done Done Sent false
2867 1659 Acked Flush 0 Done Done Sent false
2868 1660 Acked Read 5 Done Done Sent false
2869 1661 Acked Read 5 Done Done Sent false
2870 1662 Acked Write 9 Done Done Sent false
2871 1663 Acked Read 7 Done Done Sent false
2872 1664 Acked Read 6 Done Done Sent false
2873 1665 Acked Write 8 Done Done Sent false
2874 1666 Acked Write 2 Done Done Sent false
2875 1667 Acked Read 2 Done Done Sent false
2876 1668 Acked Write 1 Done Done Sent false
2877 1669 Acked Read 9 Done Done Sent false
2878 1670 Acked Write 4 Done Done Sent false
2879 1671 Acked Read 8 Done Done Sent false
2880 1672 Acked Read 6 Done Done Sent false
2881 1673 Acked Read 8 Done Done Sent false
2882 1674 Acked Read 8 Done Done Sent false
2883 1675 Acked Read 6 Done Done Sent false
2884 1676 Acked Write 7 Done Done Sent false
2885 1677 Acked Read 4 Done Done Sent false
2886 1678 Acked Read 9 Done Done Sent false
2887 1679 Acked Read 7 Done Done Sent false
2888 1680 Acked Write 6 Done Done Sent false
2889 1681 Acked Read 9 Done Done Sent false
2890 1682 Acked Write 5 Done Done Sent false
2891 1683 Acked Read 7 Done Done Sent false
2892 1684 Acked Flush 0 Done Done Sent false
2893 1685 Acked Write 3 Done Done Sent false
2894 1686 Acked Flush 0 Done Done Sent false
2895 1687 Acked Write 7 Done Done Sent false
2896 1688 Acked Flush 0 Done Done Sent false
2897 1689 Acked Write 9 Done Done Sent false
2898 1690 Acked Write 3 Done Done Sent false
2899 1691 Acked Flush 0 Done Done Sent false
2900 1692 Acked Write 1 Done Done Sent false
2901 1693 Acked Read 2 Done Done Sent false
2902 1694 Acked Read 6 Done Done Sent false
2903 1695 Acked Read 9 Done Done Sent false
2904 1696 Acked Read 4 Done Done Sent false
2905 1697 Acked Read 8 Done Done Sent false
2906 1698 Acked Write 1 Done Done Sent false
2907 1699 Acked Read 8 Done Done Sent false
2908 1700 Acked Write 3 Done Done Sent false
2909 1701 Acked Read 6 Done Done Sent false
2910 1702 Acked Write 3 Done Done Sent false
2911 1703 Acked Write 3 Done Done Sent false
2912 1704 Acked Read 6 Done Done Sent false
2913 1705 Acked Write 7 Done Done Sent false
2914 1706 Acked Write 5 Done Done Sent false
2915 1707 Acked Write 2 Done Done Sent false
2916 1708 Acked Write 5 Done Done Sent false
2917 1709 Acked Write 2 Done Done Sent false
2918 1710 Acked Write 7 Done Done Sent false
2919 1711 Acked Read 3 Done Done Sent false
2920 1712 Acked Write 4 Done Done Sent false
2921 1713 Acked Read 10 Done Done Sent false
2922 1714 Acked Read 7 Done Done Sent false
2923 1715 Acked Read 3 Done Done Sent false
2924 1716 Acked Write 9 Done Done Sent false
2925 1717 Acked Write 9 Done Done Sent false
2926 1718 Acked Read 3 Done Done Sent false
2927 1719 Acked Read 8 Done Done Sent false
2928 1720 Acked Read 6 Done Done Sent false
2929 1721 Acked Write 3 Done Done Sent false
2930 1722 Acked Write 4 Done Done Sent false
2931 1723 Acked Flush 0 Done Done Sent false
2932 1724 Acked Read 4 Done Done Sent false
2933 1725 Acked Read 10 Done Done Sent false
2934 1726 Acked Read 5 Done Done Sent false
2935 1727 Acked Flush 0 Done Done Sent false
2936 1728 Acked Flush 0 Done Done Sent false
2937 1729 Acked Flush 0 Done Done Sent false
2938 1730 Acked Read 1 Done Done Sent false
2939 1731 Acked Write 4 Done Done Sent false
2940 1732 Acked Write 9 Done Done Sent false
2941 1733 Acked Write 8 Done Done Sent false
2942 1734 Acked Write 5 Done Done Sent false
2943 1735 Acked Read 5 Done Done Sent false
2944 1736 Acked Write 4 Done Done Sent false
2945 1737 Acked Read 4 Done Done Sent false
2946 1738 Acked Write 9 Done Done Sent false
2947 1739 Acked Write 9 Done Done Sent false
2948 1740 Acked Flush 0 Done Done Sent false
2949 1741 Acked Read 10 Done Done Sent false
2950 1742 Acked Flush 0 Done Done Sent false
2951 1743 Acked Read 3 Done Done Sent false
2952 1744 Acked Read 2 Done Done Sent false
2953 1745 Acked Read 4 Done Done Sent false
2954 1746 Acked Write 4 Done Done Sent false
2955 1747 Acked Flush 0 Done Done Sent false
2956 1748 Acked Read 7 Done Done Sent false
2957 1749 Acked Read 3 Done Done Sent false
2958 1750 Acked Read 5 Done Done Sent false
2959 1751 Acked Write 7 Done Done Sent false
2960 1752 Acked Read 8 Done Done Sent false
2961 1753 Acked Read 7 Done Done Sent false
2962 1754 Acked Read 4 Done Done Sent false
2963 1755 Acked Read 3 Done Done Sent false
2964 1756 Acked Read 10 Done Done Sent false
2965 1757 Acked Write 8 Done Done Sent false
2966 1758 Acked Write 2 Done Done Sent false
2967 1759 Acked Read 2 Done Done Sent false
2968 1760 Acked Write 2 Done Done Sent false
2969 1761 Acked Read 3 Done Done Sent false
2970 1762 Acked Read 10 Done Done Sent false
2971 1763 Acked Read 3 Done Done Sent false
2972 1764 Acked Read 10 Done Done Sent false
2973 1765 Acked Read 6 Done Done Sent false
2974 1766 Acked Write 2 Done Done Sent false
2975 1767 Acked Read 9 Done Done Sent false
2976 1768 Acked Read 5 Done Done Sent false
2977 1769 Acked Write 9 Done Done Sent false
2978 1770 Acked Read 8 Done Done Sent false
2979 1771 Acked Read 4 Done Done Sent false
2980 1772 Acked Write 1 Done Done Sent false
2981 1773 Acked Write 6 Done Done Sent false
2982 1774 Acked Read 2 Done Done Sent false
2983 1775 Acked Read 8 Done Done Sent false
2984 1776 Acked Flush 0 Done Done Sent false
2985 1777 Acked Read 7 Done Done Sent false
2986 1778 Acked Read 7 Done Done Sent false
2987 1779 Acked Write 9 Done Done Sent false
2988 1780 Acked Write 4 Done Done Sent false
2989 1781 Acked Read 7 Done Done Sent false
2990 1782 Acked Write 9 Done Done Sent false
2991 1783 Acked Read 5 Done Done Sent false
2992 1784 Acked Write 6 Done Done Sent false
2993 1785 Acked Write 1 Done Done Sent false
2994 1786 Acked Write 2 Done Done Sent false
2995 1787 Acked Write 5 Done Done Sent false
2996 1788 Acked Write 9 Done Done Sent false
2997 1789 Acked Write 7 Done Done Sent false
2998 1790 Acked Read 7 Done Done Sent false
2999 1791 Acked Read 5 Done Done Sent false
3000 1792 Acked Read 5 Done Done Sent false
3001 1793 Acked Read 4 Done Done Sent false
3002 1794 Acked Read 3 Done Done Sent false
3003 1795 Acked Write 8 Done Done Sent false
3004 1796 Acked Write 10 Done Done Sent false
3005 1797 Acked Read 9 Done Done Sent false
3006 1798 Acked Flush 0 Done Done Sent false
3007 1799 Acked Write 5 Done Done Sent false
3008 1800 Acked Read 5 Done Done Sent false
3009 1801 Acked Read 6 Done Done Sent false
3010 1802 Acked Write 9 Done Done Sent false
3011 1803 Acked Write 6 Done Done Sent false
3012 1804 Acked Read 2 Done Done Sent false
3013 1805 Acked Read 6 Done Done Sent false
3014 1806 Acked Read 6 Done Done Sent false
3015 1807 Acked Read 10 Done Done Sent false
3016 1808 Acked Write 1 Done Done Sent false
3017 1809 Acked Flush 0 Done Done Sent false
3018 1810 Acked Read 7 Done Done Sent false
3019 1811 Acked Read 9 Done Done Sent false
3020 1812 Acked Write 8 Done Done Sent false
3021 1813 Acked Read 7 Done Done Sent false
3022 1814 Acked Read 1 Done Done Sent false
3023 1815 Acked Read 5 Done Done Sent false
3024 1816 Acked Read 10 Done Done Sent false
3025 1817 Acked Read 3 Done Done Sent false
3026 1818 Acked Write 3 Done Done Sent false
3027 1819 Acked Flush 0 Done Done Sent false
3028 1820 Acked Write 1 Done Done Sent false
3029 1821 Acked Read 9 Done Done Sent false
3030 1822 Acked Read 9 Done Done Sent false
3031 1823 Acked Read 1 Done Done Sent false
3032 1824 Acked Read 1 Done Done Sent false
3033 1825 Acked Read 6 Done Done Sent false
3034 1826 Acked Read 8 Done Done Sent false
3035 1827 Acked Write 1 Done Done Sent false
3036 1828 Acked Write 5 Done Done Sent false
3037 1829 Acked Read 8 Done Done Sent false
3038 1830 Acked Write 9 Done Done Sent false
3039 1831 Acked Write 10 Done Done Sent false
3040 1832 Acked Write 3 Done Done Sent false
3041 1833 Acked Flush 0 Done Done Sent false
3042 1834 Acked Read 5 Done Done Sent false
3043 1835 Acked Read 1 Done Done Sent false
3044 1836 Acked Read 4 Done Done Sent false
3045 1837 Acked Flush 0 Done Done Sent false
3046 1838 Acked Read 3 Done Done Sent false
3047 1839 Acked Write 8 Done Done Sent false
3048 1840 Acked Flush 0 Done Done Sent false
3049 1841 Acked Read 10 Done Done Sent false
3050 1842 Acked Write 6 Done Done Sent false
3051 1843 Acked Read 10 Done Done Sent false
3052 1844 Acked Write 8 Done Done Sent false
3053 1845 Acked Write 5 Done Done Sent false
3054 1846 Acked Read 8 Done Done Sent false
3055 1847 Acked Read 1 Done Done Sent false
3056 1848 Acked Write 2 Done Done Sent false
3057 1849 Acked Read 4 Done Done Sent false
3058 1850 Acked Write 10 Done Done Sent false
3059 1851 Acked Write 10 Done Done Sent false
3060 1852 Acked Read 5 Done Done Sent false
3061 1853 Acked Write 5 Done Done Sent false
3062 1854 Acked Write 7 Done Done Sent false
3063 1855 Acked Flush 0 Done Done Sent false
3064 1856 Acked Write 6 Done Done Sent false
3065 1857 Acked Write 5 Done Done Sent false
3066 1858 Acked Write 10 Done Done Sent false
3067 1859 Acked Read 2 Done Done Sent false
3068 1860 Acked Write 9 Done Done Sent false
3069 1861 Acked Write 4 Done Done Sent false
3070 1862 Acked Write 9 Done Done Sent false
3071 1863 Acked Read 10 Done Done Sent false
3072 1864 Acked Write 2 Done Done Sent false
3073 1865 Acked Write 9 Done Done Sent false
3074 1866 Acked Read 3 Done Done Sent false
3075 1867 Acked Flush 0 Done Done Sent false
3076 1868 Acked Write 5 Done Done Sent false
3077 1869 Acked Flush 0 Done Done Sent false
3078 1870 Acked Write 1 Done Done Sent false
3079 1871 Acked Write 5 Done Done Sent false
3080 1872 Acked Write 2 Done Done Sent false
3081 1873 Acked Read 1 Done Done Sent false
3082 1874 Acked Read 9 Done Done Sent false
3083 1875 Acked Flush 0 Done Done Sent false
3084 1876 Acked Write 5 Done Done Sent false
3085 1877 Acked Write 10 Done Done Sent false
3086 1878 Acked Write 7 Done Done Sent false
3087 1879 Acked Write 7 Done Done Sent false
3088 1880 Acked Write 1 Done Done Sent false
3089 1881 Acked Flush 0 Done Done Sent false
3090 1882 Acked Write 7 Done Done Sent false
3091 1883 Acked Read 10 Done Done Sent false
3092 1884 Acked Write 3 Done Done Sent false
3093 1885 Acked Read 9 Done Done Sent false
3094 1886 Acked Write 6 Done Done Sent false
3095 1887 Acked Write 8 Done Done Sent false
3096 1888 Acked Read 1 Done Done Sent false
3097 1889 Acked Write 7 Done Done Sent false
3098 1890 Acked Read 2 Done Done Sent false
3099 1891 Acked Write 9 Done Done Sent false
3100 1892 Acked Write 4 Done Done Sent false
3101 1893 Acked Read 9 Done Done Sent false
3102 1894 Acked Flush 0 Done Done Sent false
3103 1895 Acked Read 8 Done Done Sent false
3104 1896 Acked Write 2 Done Done Sent false
3105 1897 Acked Flush 0 Done Done Sent false
3106 1898 Acked Read 5 Done Done Sent false
3107 1899 Acked Flush 0 Done Done Sent false
3108 1900 Acked Read 10 Done Done Sent false
3109 1901 Acked Write 10 Done Done Sent false
3110 1902 Acked Write 5 Done Done Sent false
3111 1903 Acked Read 2 Done Done Sent false
3112 1904 Acked Read 3 Done Done Sent false
3113 STATES DS:0 DS:1 DS:2 TOTAL
3114 Sent 0 0 301 301
3115 Done 301 301 0 602
3116 Skipped 0 0 0 0
3117 Error 0 0 0 0
3118 Last Flush: Some(JobId(1899)) Some(JobId(1899)) Some(JobId(1603))
3119 Downstairs last five completed: 1603 1602 1601 1600 1599
3120 Upstairs last five completed: 1904 1903 1902 1901 1900
3121 CLIENT: Up:0 ds:301 act:2
31222025-02-25T17:18:51.353ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31232025-02-25T17:18:52.354ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31242025-02-25T17:18:52.361ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31252025-02-25T17:18:52.361ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65042) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31262025-02-25T17:18:52.362ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31272025-02-25T17:18:52.363ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31282025-02-25T17:18:52.363ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31292025-02-25T17:18:52.363ZINFOcrucible: send last flush ID to this DS: Some(JobId(1603)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31302025-02-25T17:18:52.363ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31312025-02-25T17:18:52.363ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1603)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31322025-02-25T17:18:52.363ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1603)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
3133 ----------------------------------------------------------------
3134 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3135 Downstairs last five completed: 1905 1904 1903 1902 1901
3136 Upstairs last five completed: 1905 1904 1903 1902 1901
3137 CLIENT: Up:0 ds:0 act:3
3138 Replay: All jobs finished
3139 Main task has work to do, go find it
3140 got dsc Stop(0)
3141 stop 0
3142 Feb 25 17:18:54.620 INFO request completed, latency_us: 381, response_code: [8810] Got stop action so:false kr:false
3143 204, uri: /stop/cid/0, method: POST, req_id: c7750f6a-d1a4-4914-9eda-a9be9ece6d69, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
3144 Feb 25 17:18:54.621 INFO request completed, latency_us: 131, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5aa244ba-e9a8-4319-9731-035176f952e2, remote_addr: 127.0.0.1:52576, local_addr: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:18:54.621304771Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
3145 127.0.0.1:9998
31462025-02-25T17:18:54.621ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31472025-02-25T17:18:54.621ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31482025-02-25T17:18:54.621ZINFOcrucible: client task is exiting = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
3149 [8810] Exited with: ExitStatus(unix_wait_status(9))
3150 [8810][0] reports Exit
3151 Feb 25 17:18:58.623 INFO request completed, latency_us: 223, response_code: 200, uri: /state/cid/0, method: GET, req_id: b1a0e0b2-fea3-4c18-bd99-3acef05c619c, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
3152 001/300 Read block 4163 len 20480
3153 002/300 Read block 1905 len 16384
3154 003/300 Read block 3318 len 32768
3155 004/300 Read block 3285 len 12288
3156 005/300 Flush
3157 006/300 Flush
3158 007/300 Write block 1399 len 16384 data: 2 2 2 2
3159 008/300 Write block 4961 len 20480 data: 2 3 3 3 3
3160 009/300 Read block 3240 len 8192
3161 010/300 Flush
3162 011/300 Read block 1911 len 40960
3163 012/300 Read block 2542 len 36864
3164 013/300 Write block 3203 len 16384 data: 3 3 3 3
3165 014/300 Flush
3166 015/300 Read block 1393 len 40960
3167 016/300 Write block 2257 len 4096 data: 2
3168 017/300 Write block 3964 len 24576 data: 2 2 2 2 2 2
3169 018/300 Read block 4752 len 24576
3170 019/300 Write block 1604 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3171 020/300 Read block 4310 len 20480
3172 021/300 Write block 4421 len 8192 data: 3 4
3173 022/300 Read block 4837 len 40960
3174 023/300 Write block 1510 len 4096 data: 2
3175 024/300 Write block 355 len 28672 data: 2 3 3 2 2 2 2
3176 025/300 Write block 4768 len 16384 data: 3 3 3 3
3177 026/300 Write block 3776 len 8192 data: 2 2
3178 027/300 Flush
3179 028/300 Read block 563 len 24576
3180 029/300 Read block 3611 len 28672
3181 030/300 Read block 2108 len 12288
3182 031/300 Read block 1995 len 4096
3183 032/300 Read block 4798 len 32768
3184 033/300 Write block 3076 len 4096 data: 2
3185 034/300 Write block 1844 len 12288 data: 3 2 2
3186 035/300 Read block 1056 len 16384
3187 036/300 Write block 2211 len 40960 data: 4 3 3 3 2 2 3 3 3 3
3188 037/300 Read block 25 len 24576
3189 038/300 Flush
3190 039/300 Flush
3191 040/300 Read block 4528 len 28672
3192 041/300 Write block 4310 len 28672 data: 2 2 3 3 3 2 3
3193 042/300 Read block 424 len 20480
3194 043/300 Read block 2126 len 20480
3195 044/300 Write block 999 len 28672 data: 2 2 2 2 2 2 2
3196 045/300 Read block 3657 len 4096
3197 046/300 Read block 2783 len 8192
3198 047/300 Write block 1589 len 28672 data: 2 2 2 3 3 3 3
3199 048/300 Write block 4502 len 4096 data: 2
3200 049/300 Flush
3201 050/300 Write block 1935 len 28672 data: 3 3 3 2 2 2 2
3202 051/300 Read block 932 len 28672
3203 052/300 Read block 418 len 28672
3204 053/300 Write block 2581 len 16384 data: 2 2 2 2
3205 054/300 Read block 1176 len 8192
3206 055/300 Read block 2334 len 36864
3207 056/300 Read block 4521 len 4096
3208 057/300 Read block 741 len 20480
3209 058/300 Write block 4787 len 20480 data: 3 3 3 3 2
3210 059/300 Write block 3734 len 4096 data: 2
3211 060/300 Read block 352 len 24576
3212 061/300 Read block 240 len 40960
3213 062/300 Write block 1068 len 16384 data: 3 3 3 3
3214 063/300 Write block 1939 len 8192 data: 3 3
3215 064/300 Write block 2327 len 32768 data: 2 2 3 3 3 4 4 4
3216 065/300 Write block 897 len 32768 data: 2 2 2 2 2 2 2 2
3217 066/300 Write block 2868 len 32768 data: 2 2 2 3 3 3 2 2
3218 067/300 Read block 3696 len 40960
3219 068/300 Read block 1703 len 32768
3220 069/300 Read block 313 len 32768
3221 070/300 Read block 4556 len 24576
3222 071/300 Flush
3223 072/300 Read block 1205 len 16384
3224 073/300 Read block 280 len 24576
3225 074/300 Read block 1100 len 4096
3226 075/300 Write block 2663 len 28672 data: 2 3 3 3 3 3 3
3227 076/300 Read block 4806 len 40960
3228 077/300 Write block 3642 len 28672 data: 2 2 2 2 2 2 2
3229 078/300 Write block 2351 len 8192 data: 2 3
3230 079/300 Write block 3380 len 8192 data: 2 2
3231 080/300 Write block 718 len 16384 data: 2 2 2 2
3232 081/300 Read block 1288 len 32768
3233 082/300 Write block 4301 len 20480 data: 3 2 2 2 2
3234 083/300 Read block 324 len 16384
3235 084/300 Read block 3106 len 40960
3236 085/300 Read block 3297 len 12288
3237 086/300 Read block 3648 len 12288
3238 087/300 Read block 802 len 20480
3239 088/300 Read block 1490 len 40960
3240 089/300 Read block 1141 len 36864
3241 090/300 Read block 4211 len 12288
3242 091/300 Read block 266 len 40960
3243 092/300 Read block 4122 len 4096
3244 093/300 Write block 906 len 8192 data: 2 2
3245 094/300 Write block 209 len 40960 data: 3 3 3 3 2 2 2 3 3 3
3246 095/300 Write block 1040 len 36864 data: 2 3 3 3 3 3 3 3 3
3247 096/300 Read block 716 len 16384
3248 097/300 Write block 2901 len 32768 data: 3 4 3 3 3 3 3 2
3249 098/300 Flush
3250 099/300 Write block 1413 len 12288 data: 2 2 2
3251 100/300 Write block 16 len 4096 data: 2
3252 101/300 Read block 689 len 28672
3253 102/300 Read block 2017 len 8192
3254 103/300 Read block 1687 len 12288
3255 104/300 Write block 4869 len 36864 data: 2 3 3 2 2 2 2 2 2
3256 105/300 Read block 149 len 8192
3257 106/300 Read block 863 len 4096
3258 107/300 Read block 2697 len 28672
3259 108/300 Read block 3880 len 36864
3260 109/300 Read block 2360 len 8192
3261 110/300 Read block 3887 len 24576
3262 111/300 Read block 1275 len 28672
3263 112/300 Flush
3264 113/300 Read block 3385 len 8192
3265 114/300 Flush
3266 115/300 Flush
3267 116/300 Read block 852 len 16384
3268 117/300 Read block 4812 len 36864
3269 118/300 Flush
3270 119/300 Write block 3401 len 40960 data: 3 3 2 2 2 2 2 3 3 3
3271 120/300 Read block 4230 len 36864
3272 121/300 Write block 861 len 16384 data: 2 2 2 3
3273 122/300 Read block 2946 len 20480
3274 123/300 Write block 2505 len 28672 data: 2 2 2 2 2 3 3
3275 124/300 Read block 4462 len 12288
3276 125/300 Write block 1210 len 4096 data: 2
3277 126/300 Read block 4014 len 32768
3278 127/300 Read block 2121 len 4096
3279 128/300 Write block 2598 len 32768 data: 3 3 3 3 2 2 2 2
3280 129/300 Write block 1666 len 12288 data: 2 3 4
3281 130/300 Write block 1931 len 32768 data: 3 3 3 3 4 4 4 3
3282 131/300 Write block 1685 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3283 132/300 Write block 1177 len 40960 data: 3 3 3 3 3 3 2 2 2 2
3284 133/300 Read block 1035 len 32768
3285 134/300 Read block 2377 len 36864
3286 135/300 Read block 864 len 8192
3287 136/300 Read block 3861 len 4096
3288 137/300 Read block 2609 len 36864
3289 138/300 Read block 1220 len 16384
3290 139/300 Read block 2331 len 32768
3291 140/300 Read block 4200 len 16384
3292 141/300 Read block 4378 len 28672
3293 142/300 Write block 4939 len 16384 data: 3 3 3 3
3294 143/300 Read block 3988 len 16384
3295 144/300 Write block 3709 len 8192 data: 2 2
3296 145/300 Read block 1680 len 24576
3297 146/300 Write block 3384 len 16384 data: 2 2 2 2
3298 147/300 Read block 2664 len 36864
3299 148/300 Flush
3300 149/300 Read block 107 len 32768
3301 150/300 Write block 4575 len 32768 data: 2 2 2 2 2 2 2 2
3302 151/300 Write block 3624 len 16384 data: 2 2 2 2
3303 152/300 Read block 3960 len 8192
3304 153/300 Write block 3896 len 4096 data: 2
3305 154/300 Read block 3314 len 32768
3306 155/300 Read block 994 len 16384
3307 156/300 Write block 190 len 4096 data: 2
3308 157/300 Read block 3649 len 4096
3309 158/300 Write block 4399 len 32768 data: 2 2 2 2 2 2 2 2
3310 159/300 Write block 951 len 4096 data: 2
3311 160/300 Read block 3550 len 8192
3312 161/300 Read block 4412 len 32768
3313 162/300 Read block 1117 len 28672
3314 163/300 Read block 365 len 24576
3315 164/300 Read block 4246 len 36864
3316 165/300 Read block 521 len 20480
3317 166/300 Write block 4340 len 28672 data: 4 3 3 3 3 4 3
3318 167/300 Write block 4249 len 8192 data: 2 2
3319 168/300 Read block 4659 len 20480
3320 169/300 Read block 3144 len 32768
3321 170/300 Read block 412 len 12288
3322 171/300 Write block 3795 len 16384 data: 4 4 4 3
3323 172/300 Write block 3432 len 32768 data: 2 2 2 2 2 2 2 2
3324 173/300 Write block 3947 len 16384 data: 2 2 2 2
3325 174/300 Read block 372 len 36864
3326 175/300 Flush
3327 176/300 Read block 4214 len 28672
3328 177/300 Write block 2137 len 36864 data: 2 2 2 2 2 2 2 2 2
3329 178/300 Read block 4797 len 40960
3330 179/300 Read block 850 len 8192
3331 180/300 Write block 1930 len 12288 data: 3 4 4
3332 181/300 Write block 63 len 4096 data: 2
3333 182/300 Flush
3334 183/300 Flush
3335 184/300 Read block 3993 len 24576
3336 185/300 Read block 3963 len 20480
3337 186/300 Write block 2191 len 12288 data: 2 2 2
3338 187/300 Read block 2707 len 32768
3339 188/300 Read block 559 len 4096
3340 189/300 Read block 3482 len 12288
3341 190/300 Read block 4561 len 36864
3342 191/300 Write block 2728 len 16384 data: 2 2 2 2
3343 192/300 Read block 4723 len 40960
3344 193/300 Read block 3259 len 24576
3345 194/300 Read block 4619 len 36864
3346 195/300 Write block 1382 len 40960 data: 2 2 2 3 3 3 3 3 3 3
3347 196/300 Write block 419 len 28672 data: 3 3 2 4 4 4 3
3348 197/300 Read block 2558 len 24576
3349 198/300 Write block 3805 len 20480 data: 2 2 2 2 2
3350 199/300 Read block 932 len 8192
3351 200/300 Flush
3352 201/300 Read block 2411 len 24576
3353 202/300 Read block 3768 len 12288
3354 203/300 Write block 4758 len 36864 data: 4 4 2 2 3 3 3 3 3
3355 204/300 Write block 2369 len 12288 data: 2 2 2
3356 205/300 Read block 4070 len 24576
3357 206/300 Write block 4683 len 8192 data: 2 2
3358 207/300 Read block 3453 len 4096
3359 208/300 Read block 2969 len 24576
3360 209/300 Write block 545 len 32768 data: 2 2 2 2 2 2 2 2
3361 210/300 Read block 4849 len 32768
3362 211/300 Read block 165 len 12288
3363 212/300 Write block 2632 len 8192 data: 2 2
3364 213/300 Read block 4212 len 36864
3365 214/300 Read block 1205 len 32768
3366 215/300 Flush
3367 216/300 Write block 3531 len 4096 data: 2
3368 217/300 Read block 2932 len 28672
3369 218/300 Read block 2284 len 36864
3370 219/300 Read block 2713 len 4096
3371 220/300 Write block 4637 len 12288 data: 3 2 2
3372 221/300 Read block 4488 len 40960
3373 222/300 Write block 4228 len 28672 data: 2 2 2 2 2 3 3
3374 223/300 Read block 792 len 8192
3375 224/300 Flush
3376 225/300 Read block 4395 len 36864
3377 226/300 Read block 3939 len 20480
3378 227/300 Read block 3864 len 24576
3379 228/300 Flush
3380 229/300 Read block 679 len 16384
3381 230/300 Read block 4774 len 8192
3382 231/300 Read block 1455 len 36864
3383 232/300 Read block 304 len 20480
3384 233/300 Read block 729 len 36864
3385 234/300 Read block 4895 len 12288
3386 235/300 Read block 3865 len 32768
3387 236/300 Read block 2306 len 4096
3388 237/300 Read block 575 len 4096
3389 238/300 Write block 179 len 32768 data: 4 3 3 3 3 3 3 2
3390 239/300 Read block 4930 len 4096
3391 240/300 Read block 142 len 24576
3392 241/300 Read block 4954 len 32768
3393 242/300 Write block 3752 len 36864 data: 2 2 2 2 2 2 2 2 2
3394 243/300 Read block 154 len 28672
3395 244/300 Write block 2323 len 32768 data: 4 3 3 3 3 3 4 4
3396 245/300 Read block 4713 len 36864
3397 246/300 Write block 1712 len 36864 data: 4 4 4 4 4 4 4 5 4
3398 247/300 Flush
3399 248/300 Read block 4859 len 4096
3400 249/300 Read block 2336 len 36864
3401 250/300 Read block 358 len 36864
3402 251/300 Write block 618 len 32768 data: 2 2 2 2 2 2 2 2
3403 252/300 Read block 3927 len 40960
3404 253/300 Flush
3405 254/300 Write block 1944 len 28672 data: 2 3 3 3 3 3 3
3406 255/300 Write block 1053 len 20480 data: 2 2 2 2 2
3407 256/300 Write block 2247 len 20480 data: 2 2 2 2 2
3408 257/300 Read block 4651 len 16384
3409 258/300 Read block 2326 len 28672
3410 259/300 Write block 4767 len 24576 data: 3 4 4 4 4 2
3411 260/300 Write block 1910 len 4096 data: 3
3412 261/300 Flush
3413 262/300 Write block 1630 len 24576 data: 2 2 2 2 2 2
3414 263/300 Read block 4574 len 24576
3415 264/300 Read block 1278 len 4096
3416 265/300 Write block 1532 len 28672 data: 3 3 3 3 2 2 2
3417 266/300 Write block 1503 len 28672 data: 2 2 2 2 2 2 2
3418 267/300 Read block 803 len 4096
3419 268/300 Read block 3493 len 24576
3420 269/300 Read block 3010 len 12288
3421 270/300 Read block 4843 len 28672
3422 271/300 Flush
3423 272/300 Flush
3424 273/300 Write block 1106 len 32768 data: 2 2 2 2 2 2 2 2
3425 274/300 Read block 1070 len 20480
3426 275/300 Read block 124 len 4096
3427 276/300 Read block 1645 len 24576
3428 277/300 Flush
3429 278/300 Read block 2997 len 16384
3430 279/300 Flush
3431 280/300 Write block 2599 len 4096 data: 4
3432 281/300 Write block 1092 len 4096 data: 3
3433 282/300 Read block 3658 len 8192
3434 283/300 Write block 1388 len 12288 data: 4 4 4
3435 284/300 Flush
3436 285/300 Write block 3939 len 40960 data: 4 4 4 4 4 4 4 3 3 3
3437 286/300 Write block 2153 len 36864 data: 2 2 2 2 2 2 2 2 2
3438 287/300 Read block 713 len 36864
3439 288/300 Read block 2107 len 12288
3440 289/300 Write block 3731 len 36864 data: 2 2 2 3 2 2 2 2 2
3441 290/300 Write block 2850 len 40960 data: 2 2 3 3 3 3 3 4 5 5
3442 291/300 Write block 4766 len 24576 data: 4 4 5 5 5 5
3443 292/300 Read block 1737 len 32768
3444 293/300 Read block 512 len 36864
3445 294/300 Flush
3446 295/300 Read block 477 len 36864
3447 296/300 Flush
3448 297/300 Write block 2881 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3449 298/300 Read block 4041 len 20480
3450 299/300 Read block 1117 len 8192
3451 300/300 Write block 1004 len 8192 data: 3 3
3452 301/300 Write block 741 len 28672 data: 2 2 2 2 2 2 2
3453 Main task has work to do, go find it
3454 got dsc Start(0)
3455 start 0
3456 Feb 25 17:18:58.884 INFO request completed, latency_us: 227, response_code: 204, uri: /start/cid/0, method: POST, req_id: f8ce1cff-5d70-4004-89aa-c94275fcfd46, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
3457 Replay: started 0, returned:Ok(())
3458 ----------------------------------------------------------------
3459 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3460 States: Offline Active Active
3461 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3462 1906 Acked Read 5 Sent Done Done false
3463 1907 Acked Read 4 Sent Done Done false
3464 1908 Acked Read 8 Sent Done Done false
3465 1909 Acked Read 3 Sent Done Done false
3466 1910 Acked Flush 0 Sent Done Done false
3467 1911 Acked Flush 0 Sent Done Done false
3468 1912 Acked Write 4 Sent Done Done false
3469 1913 Acked Write 5 Sent Done Done false
3470 1914 Acked Read 2 Sent Done Done false
3471 1915 Acked Flush 0 Sent Done Done false
3472 1916 Acked Read 10 Sent Done Done false
3473 1917 Acked Read 9 Sent Done Done false
3474 1918 Acked Write 4 Sent Done Done false
3475 1919 Acked Flush 0 Sent Done Done false
3476 1920 Acked Read 10 Sent Done Done false
3477 1921 Acked Write 1 Sent Done Done false
3478 1922 Acked Write 6 Sent Done Done false
3479 1923 Acked Read 6 Sent Done Done false
3480 1924 Acked Write 10 Sent Done Done false
3481 1925 Acked Read 5 Sent Done Done false
3482 1926 Acked Write 2 Sent Done Done false
3483 1927 Acked Read 10 Sent Done Done false
3484 1928 Acked Write 1 Sent Done Done false
3485 1929 Acked Write 7 Sent Done Done false
3486 1930 Acked Write 4 Sent Done Done false
3487 1931 Acked Write 2 Sent Done Done false
3488 1932 Acked Flush 0 Sent Done Done false
3489 1933 Acked Read 6 Sent Done Done false
3490 1934 Acked Read 7 Sent Done Done false
3491 1935 Acked Read 3 Sent Done Done false
3492 1936 Acked Read 1 Sent Done Done false
3493 1937 Acked Read 8 Sent Done Done false
3494 1938 Acked Write 1 Sent Done Done false
3495 1939 Acked Write 3 Sent Done Done false
3496 1940 Acked Read 4 Sent Done Done false
3497 1941 Acked Write 10 Sent Done Done false
3498 1942 Acked Read 6 Sent Done Done false
3499 1943 Acked Flush 0 Sent Done Done false
3500 1944 Acked Flush 0 Sent Done Done false
3501 1945 Acked Read 7 Sent Done Done false
3502 1946 Acked Write 7 Sent Done Done false
3503 1947 Acked Read 5 Sent Done Done false
3504 1948 Acked Read 5 Sent Done Done false
3505 1949 Acked Write 7 Sent Done Done false
3506 1950 Acked Read 1 Sent Done Done false
3507 1951 Acked Read 2 Sent Done Done false
3508 1952 Acked Write 7 Sent Done Done false
3509 1953 Acked Write 1 Sent Done Done false
3510 1954 Acked Flush 0 Sent Done Done false
3511 1955 Acked Write 7 Sent Done Done false
3512 1956 Acked Read 7 Sent Done Done false
3513 1957 Acked Read 7 Sent Done Done false
3514 1958 Acked Write 4 Sent Done Done false
3515 1959 Acked Read 2 Sent Done Done false
3516 1960 Acked Read 9 Sent Done Done false
3517 1961 Acked Read 1 Sent Done Done false
3518 1962 Acked Read 5 Sent Done Done false
3519 1963 Acked Write 5 Sent Done Done false
3520 1964 Acked Write 1 Sent Done Done false
3521 1965 Acked Read 6 Sent Done Done false
3522 1966 Acked Read 10 Sent Done Done false
3523 1967 Acked Write 4 Sent Done Done false
3524 1968 Acked Write 2 Sent Done Done false
3525 1969 Acked Write 8 Sent Done Done false
3526 1970 Acked Write 8 Sent Done Done false
3527 1971 Acked Write 8 Sent Done Done false
3528 1972 Acked Read 10 Sent Done Done false
3529 1973 Acked Read 8 Sent Done Done false
3530 1974 Acked Read 8 Sent Done Done false
3531 1975 Acked Read 6 Sent Done Done false
3532 1976 Acked Flush 0 Sent Done Done false
3533 1977 Acked Read 4 Sent Done Done false
3534 1978 Acked Read 6 Sent Done Done false
3535 1979 Acked Read 1 Sent Done Done false
3536 1980 Acked Write 7 Sent Done Done false
3537 1981 Acked Read 10 Sent Done Done false
3538 1982 Acked Write 7 Sent Done Done false
3539 1983 Acked Write 2 Sent Done Done false
3540 1984 Acked Write 2 Sent Done Done false
3541 1985 Acked Write 4 Sent Done Done false
3542 1986 Acked Read 8 Sent Done Done false
3543 1987 Acked Write 5 Sent Done Done false
3544 1988 Acked Read 4 Sent Done Done false
3545 1989 Acked Read 10 Sent Done Done false
3546 1990 Acked Read 3 Sent Done Done false
3547 1991 Acked Read 3 Sent Done Done false
3548 1992 Acked Read 5 Sent Done Done false
3549 1993 Acked Read 10 Sent Done Done false
3550 1994 Acked Read 9 Sent Done Done false
3551 1995 Acked Read 3 Sent Done Done false
3552 1996 Acked Read 10 Sent Done Done false
3553 1997 Acked Read 1 Sent Done Done false
3554 1998 Acked Write 2 Sent Done Done false
3555 1999 Acked Write 10 Sent Done Done false
3556 2000 Acked Write 9 Sent Done Done false
3557 2001 Acked Read 4 Sent Done Done false
3558 2002 Acked Write 8 Sent Done Done false
3559 2003 Acked Flush 0 Sent Done Done false
3560 2004 Acked Write 3 Sent Done Done false
3561 2005 Acked Write 1 Sent Done Done false
3562 2006 Acked Read 7 Sent Done Done false
3563 2007 Acked Read 2 Sent Done Done false
3564 2008 Acked Read 3 Sent Done Done false
3565 2009 Acked Write 9 Sent Done Done false
3566 2010 Acked Read 2 Sent Done Done false
3567 2011 Acked Read 1 Sent Done Done false
3568 2012 Acked Read 7 Sent Done Done false
3569 2013 Acked Read 9 Sent Done Done false
3570 2014 Acked Read 2 Sent Done Done false
3571 2015 Acked Read 6 Sent Done Done false
3572 2016 Acked Read 7 Sent Done Done false
3573 2017 Acked Flush 0 Sent Done Done false
3574 2018 Acked Read 2 Sent Done Done false
3575 2019 Acked Flush 0 Sent Done Done false
3576 2020 Acked Flush 0 Sent Done Done false
3577 2021 Acked Read 4 Sent Done Done false
3578 2022 Acked Read 9 Sent Done Done false
3579 2023 Acked Flush 0 Sent Done Done false
3580 2024 Acked Write 10 Sent Done Done false
3581 2025 Acked Read 9 Sent Done Done false
3582 2026 Acked Write 4 Sent Done Done false
3583 2027 Acked Read 5 Sent Done Done false
3584 2028 Acked Write 7 Sent Done Done false
3585 2029 Acked Read 3 Sent Done Done false
3586 2030 Acked Write 1 Sent Done Done false
3587 2031 Acked Read 8 Sent Done Done false
3588 2032 Acked Read 1 Sent Done Done false
3589 2033 Acked Write 8 Sent Done Done false
3590 2034 Acked Write 3 Sent Done Done false
3591 2035 Acked Write 8 Sent Done Done false
3592 2036 Acked Write 10 Sent Done Done false
3593 2037 Acked Write 10 Sent Done Done false
3594 2038 Acked Read 8 Sent Done Done false
3595 2039 Acked Read 9 Sent Done Done false
3596 2040 Acked Read 2 Sent Done Done false
3597 2041 Acked Read 1 Sent Done Done false
3598 2042 Acked Read 9 Sent Done Done false
3599 2043 Acked Read 4 Sent Done Done false
3600 2044 Acked Read 8 Sent Done Done false
3601 2045 Acked Read 4 Sent Done Done false
3602 2046 Acked Read 7 Sent Done Done false
3603 2047 Acked Write 4 Sent Done Done false
3604 2048 Acked Read 4 Sent Done Done false
3605 2049 Acked Write 2 Sent Done Done false
3606 2050 Acked Read 6 Sent Done Done false
3607 2051 Acked Write 4 Sent Done Done false
3608 2052 Acked Read 9 Sent Done Done false
3609 2053 Acked Flush 0 Sent Done Done false
3610 2054 Acked Read 8 Sent Done Done false
3611 2055 Acked Write 8 Sent Done Done false
3612 2056 Acked Write 4 Sent Done Done false
3613 2057 Acked Read 2 Sent Done Done false
3614 2058 Acked Write 1 Sent Done Done false
3615 2059 Acked Read 8 Sent Done Done false
3616 2060 Acked Read 4 Sent Done Done false
3617 2061 Acked Write 1 Sent Done Done false
3618 2062 Acked Read 1 Sent Done Done false
3619 2063 Acked Write 8 Sent Done Done false
3620 2064 Acked Write 1 Sent Done Done false
3621 2065 Acked Read 2 Sent Done Done false
3622 2066 Acked Read 8 Sent Done Done false
3623 2067 Acked Read 7 Sent Done Done false
3624 2068 Acked Read 6 Sent Done Done false
3625 2069 Acked Read 9 Sent Done Done false
3626 2070 Acked Read 5 Sent Done Done false
3627 2071 Acked Write 7 Sent Done Done false
3628 2072 Acked Write 2 Sent Done Done false
3629 2073 Acked Read 5 Sent Done Done false
3630 2074 Acked Read 8 Sent Done Done false
3631 2075 Acked Read 3 Sent Done Done false
3632 2076 Acked Write 4 Sent Done Done false
3633 2077 Acked Write 8 Sent Done Done false
3634 2078 Acked Write 4 Sent Done Done false
3635 2079 Acked Read 9 Sent Done Done false
3636 2080 Acked Flush 0 Sent Done Done false
3637 2081 Acked Read 7 Sent Done Done false
3638 2082 Acked Write 9 Sent Done Done false
3639 2083 Acked Read 10 Sent Done Done false
3640 2084 Acked Read 2 Sent Done Done false
3641 2085 Acked Write 3 Sent Done Done false
3642 2086 Acked Write 1 Sent Done Done false
3643 2087 Acked Flush 0 Sent Done Done false
3644 2088 Acked Flush 0 Sent Done Done false
3645 2089 Acked Read 6 Sent Done Done false
3646 2090 Acked Read 5 Sent Done Done false
3647 2091 Acked Write 3 Sent Done Done false
3648 2092 Acked Read 8 Sent Done Done false
3649 2093 Acked Read 1 Sent Done Done false
3650 2094 Acked Read 3 Sent Done Done false
3651 2095 Acked Read 9 Sent Done Done false
3652 2096 Acked Write 4 Sent Done Done false
3653 2097 Acked Read 10 Sent Done Done false
3654 2098 Acked Read 6 Sent Done Done false
3655 2099 Acked Read 9 Sent Done Done false
3656 2100 Acked Write 10 Sent Done Done false
3657 2101 Acked Write 7 Sent Done Done false
3658 2102 Acked Read 6 Sent Done Done false
3659 2103 Acked Write 5 Sent Done Done false
3660 2104 Acked Read 2 Sent Done Done false
3661 2105 Acked Flush 0 Sent Done Done false
3662 2106 Acked Read 6 Sent Done Done false
3663 2107 Acked Read 3 Sent Done Done false
3664 2108 Acked Write 9 Sent Done Done false
3665 2109 Acked Write 3 Sent Done Done false
3666 2110 Acked Read 6 Sent Done Done false
3667 2111 Acked Write 2 Sent Done Done false
3668 2112 Acked Read 1 Sent Done Done false
3669 2113 Acked Read 6 Sent Done Done false
3670 2114 Acked Write 8 Sent Done Done false
3671 2115 Acked Read 8 Sent Done Done false
3672 2116 Acked Read 3 Sent Done Done false
3673 2117 Acked Write 2 Sent Done Done false
3674 2118 Acked Read 9 Sent Done Done false
3675 2119 Acked Read 8 Sent Done Done false
3676 2120 Acked Flush 0 Sent Done Done false
3677 2121 Acked Write 1 Sent Done Done false
3678 2122 Acked Read 7 Sent Done Done false
3679 2123 Acked Read 9 Sent Done Done false
3680 2124 Acked Read 1 Sent Done Done false
3681 2125 Acked Write 3 Sent Done Done false
3682 2126 Acked Read 10 Sent Done Done false
3683 2127 Acked Write 7 Sent Done Done false
3684 2128 Acked Read 2 Sent Done Done false
3685 2129 Acked Flush 0 Sent Done Done false
3686 2130 Acked Read 9 Sent Done Done false
3687 2131 Acked Read 5 Sent Done Done false
3688 2132 Acked Read 6 Sent Done Done false
3689 2133 Acked Flush 0 Sent Done Done false
3690 2134 Acked Read 4 Sent Done Done false
3691 2135 Acked Read 2 Sent Done Done false
3692 2136 Acked Read 9 Sent Done Done false
3693 2137 Acked Read 5 Sent Done Done false
3694 2138 Acked Read 9 Sent Done Done false
3695 2139 Acked Read 3 Sent Done Done false
3696 2140 Acked Read 8 Sent Done Done false
3697 2141 Acked Read 1 Sent Done Done false
3698 2142 Acked Read 1 Sent Done Done false
3699 2143 Acked Write 8 Sent Done Done false
3700 2144 Acked Read 1 Sent Done Done false
3701 2145 Acked Read 6 Sent Done Done false
3702 2146 Acked Read 8 Sent Done Done false
3703 2147 Acked Write 9 Sent Done Done false
3704 2148 Acked Read 7 Sent Done Done false
3705 2149 Acked Write 8 Sent Done Done false
3706 2150 Acked Read 9 Sent Done Done false
3707 2151 Acked Write 9 Sent Done Done false
3708 2152 Acked Flush 0 Sent Done Done false
3709 2153 Acked Read 1 Sent Done Done false
3710 2154 Acked Read 9 Sent Done Done false
3711 2155 Acked Read 9 Sent Done Done false
3712 2156 Acked Write 8 Sent Done Done false
3713 2157 Acked Read 10 Sent Done Done false
3714 2158 Acked Flush 0 Sent Done Done false
3715 2159 Acked Write 7 Sent Done Done false
3716 2160 Acked Write 5 Sent Done Done false
3717 2161 Acked Write 5 Sent Done Done false
3718 2162 Acked Read 4 Sent Done Done false
3719 2163 Acked Read 7 Sent Done Done false
3720 2164 Acked Write 6 Sent Done Done false
3721 2165 Acked Write 1 Sent Done Done false
3722 2166 Acked Flush 0 Sent Done Done false
3723 2167 Acked Write 6 Sent Done Done false
3724 2168 Acked Read 6 Sent Done Done false
3725 2169 Acked Read 1 Sent Done Done false
3726 2170 Acked Write 7 Sent Done Done false
3727 2171 Acked Write 7 Sent Done Done false
3728 2172 Acked Read 1 Sent Done Done false
3729 2173 Acked Read 6 Sent Done Done false
3730 2174 Acked Read 3 Sent Done Done false
3731 2175 Acked Read 7 Sent Done Done false
3732 2176 Acked Flush 0 Sent Done Done false
3733 2177 Acked Flush 0 Sent Done Done false
3734 2178 Acked Write 8 Sent Done Done false
3735 2179 Acked Read 5 Sent Done Done false
3736 2180 Acked Read 1 Sent Done Done false
3737 2181 Acked Read 6 Sent Done Done false
3738 2182 Acked Flush 0 Sent Done Done false
3739 2183 Acked Read 4 Sent Done Done false
3740 2184 Acked Flush 0 Sent Done Done false
3741 2185 Acked Write 1 Sent Done Done false
3742 2186 Acked Write 1 Sent Done Done false
3743 2187 Acked Read 2 Sent Done Done false
3744 2188 Acked Write 3 Sent Done Done false
3745 2189 Acked Flush 0 Sent Done Done false
3746 2190 Acked Write 10 Sent Done Done false
3747 2191 Acked Write 9 Sent Done Done false
3748 2192 Acked Read 9 Sent Done Done false
3749 2193 Acked Read 3 Sent Done Done false
3750 2194 Acked Write 9 Sent Done Done false
3751 2195 Acked Write 10 Sent Done Done false
3752 2196 Acked Write 6 Sent Done Done false
3753 2197 Acked Read 8 Sent Done Done false
3754 2198 Acked Read 9 Sent Done Done false
3755 2199 Acked Flush 0 Sent Done Done false
3756 2200 Acked Read 9 Sent Done Done false
3757 2201 Acked Flush 0 Sent Done Done false
3758 2202 Acked Write 10 Sent Done Done false
3759 2203 Acked Read 5 Sent Done Done false
3760 2204 Acked Read 2 Sent Done Done false
3761 2205 Acked Write 2 Sent Done Done false
3762 2206 Acked Write 7 Sent Done Done false
3763 STATES DS:0 DS:1 DS:2 TOTAL
3764 Sent 301 0 0 301
3765 Done 0 301 301 602
3766 Skipped 0 0 0 0
3767 Error 0 0 0 0
3768 Last Flush: Some(JobId(1905)) Some(JobId(2201)) Some(JobId(2201))
3769 Downstairs last five completed: 1905 1904 1903 1902 1901
3770 Upstairs last five completed: 2206 2205 2204 2203 2202
3771 CLIENT: Up:0 ds:301 act:2
37722025-02-25T17:18:59.386ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
37732025-02-25T17:19:00.364ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
3774 [8810] got start action
3775 [8810] I am going to restart
3776 Starting downstairs at port 8810
3777 Make output file at "/tmp/dsc/downstairs-8810.txt"
3778 [8810][0] reports Starting
3779 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1212)
3780 [8810][0] reports Running
37812025-02-25T17:19:01.366ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
3782 ----------------------------------------------------------------
3783 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3784 States: Offline Active Active
3785 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3786 1906 Acked Read 5 Sent Done Done false
3787 1907 Acked Read 4 Sent Done Done false
3788 1908 Acked Read 8 Sent Done Done false
3789 1909 Acked Read 3 Sent Done Done false
3790 1910 Acked Flush 0 Sent Done Done false
3791 1911 Acked Flush 0 Sent Done Done false
3792 1912 Acked Write 4 Sent Done Done false
3793 1913 Acked Write 5 Sent Done Done false
3794 1914 Acked Read 2 Sent Done Done false
3795 1915 Acked Flush 0 Sent Done Done false
3796 1916 Acked Read 10 Sent Done Done false
3797 1917 Acked Read 9 Sent Done Done false
3798 1918 Acked Write 4 Sent Done Done false
3799 1919 Acked Flush 0 Sent Done Done false
3800 1920 Acked Read 10 Sent Done Done false
3801 1921 Acked Write 1 Sent Done Done false
3802 1922 Acked Write 6 Sent Done Done false
3803 1923 Acked Read 6 Sent Done Done false
3804 1924 Acked Write 10 Sent Done Done false
3805 1925 Acked Read 5 Sent Done Done false
3806 1926 Acked Write 2 Sent Done Done false
3807 1927 Acked Read 10 Sent Done Done false
3808 1928 Acked Write 1 Sent Done Done false
3809 1929 Acked Write 7 Sent Done Done false
3810 1930 Acked Write 4 Sent Done Done false
3811 1931 Acked Write 2 Sent Done Done false
3812 1932 Acked Flush 0 Sent Done Done false
3813 1933 Acked Read 6 Sent Done Done false
3814 1934 Acked Read 7 Sent Done Done false
3815 1935 Acked Read 3 Sent Done Done false
3816 1936 Acked Read 1 Sent Done Done false
3817 1937 Acked Read 8 Sent Done Done false
3818 1938 Acked Write 1 Sent Done Done false
3819 1939 Acked Write 3 Sent Done Done false
3820 1940 Acked Read 4 Sent Done Done false
3821 1941 Acked Write 10 Sent Done Done false
3822 1942 Acked Read 6 Sent Done Done false
3823 1943 Acked Flush 0 Sent Done Done false
3824 1944 Acked Flush 0 Sent Done Done false
3825 1945 Acked Read 7 Sent Done Done false
3826 1946 Acked Write 7 Sent Done Done false
3827 1947 Acked Read 5 Sent Done Done false
3828 1948 Acked Read 5 Sent Done Done false
3829 1949 Acked Write 7 Sent Done Done false
3830 1950 Acked Read 1 Sent Done Done false
3831 1951 Acked Read 2 Sent Done Done false
3832 1952 Acked Write 7 Sent Done Done false
3833 1953 Acked Write 1 Sent Done Done false
3834 1954 Acked Flush 0 Sent Done Done false
3835 1955 Acked Write 7 Sent Done Done false
3836 1956 Acked Read 7 Sent Done Done false
3837 1957 Acked Read 7 Sent Done Done false
3838 1958 Acked Write 4 Sent Done Done false
3839 1959 Acked Read 2 Sent Done Done false
3840 1960 Acked Read 9 Sent Done Done false
3841 1961 Acked Read 1 Sent Done Done false
3842 1962 Acked Read 5 Sent Done Done false
3843 1963 Acked Write 5 Sent Done Done false
3844 1964 Acked Write 1 Sent Done Done false
3845 1965 Acked Read 6 Sent Done Done false
3846 1966 Acked Read 10 Sent Done Done false
3847 1967 Acked Write 4 Sent Done Done false
3848 1968 Acked Write 2 Sent Done Done false
3849 1969 Acked Write 8 Sent Done Done false
3850 1970 Acked Write 8 Sent Done Done false
3851 1971 Acked Write 8 Sent Done Done false
3852 1972 Acked Read 10 Sent Done Done false
3853 1973 Acked Read 8 Sent Done Done false
3854 1974 Acked Read 8 Sent Done Done false
3855 1975 Acked Read 6 Sent Done Done false
3856 1976 Acked Flush 0 Sent Done Done false
3857 1977 Acked Read 4 Sent Done Done false
3858 1978 Acked Read 6 Sent Done Done false
3859 1979 Acked Read 1 Sent Done Done false
3860 1980 Acked Write 7 Sent Done Done false
3861 1981 Acked Read 10 Sent Done Done false
3862 1982 Acked Write 7 Sent Done Done false
3863 1983 Acked Write 2 Sent Done Done false
3864 1984 Acked Write 2 Sent Done Done false
3865 1985 Acked Write 4 Sent Done Done false
3866 1986 Acked Read 8 Sent Done Done false
3867 1987 Acked Write 5 Sent Done Done false
3868 1988 Acked Read 4 Sent Done Done false
3869 1989 Acked Read 10 Sent Done Done false
3870 1990 Acked Read 3 Sent Done Done false
3871 1991 Acked Read 3 Sent Done Done false
3872 1992 Acked Read 5 Sent Done Done false
3873 1993 Acked Read 10 Sent Done Done false
3874 1994 Acked Read 9 Sent Done Done false
3875 1995 Acked Read 3 Sent Done Done false
3876 1996 Acked Read 10 Sent Done Done false
3877 1997 Acked Read 1 Sent Done Done false
3878 1998 Acked Write 2 Sent Done Done false
3879 1999 Acked Write 10 Sent Done Done false
3880 2000 Acked Write 9 Sent Done Done false
3881 2001 Acked Read 4 Sent Done Done false
3882 2002 Acked Write 8 Sent Done Done false
3883 2003 Acked Flush 0 Sent Done Done false
3884 2004 Acked Write 3 Sent Done Done false
3885 2005 Acked Write 1 Sent Done Done false
3886 2006 Acked Read 7 Sent Done Done false
3887 2007 Acked Read 2 Sent Done Done false
3888 2008 Acked Read 3 Sent Done Done false
3889 2009 Acked Write 9 Sent Done Done false
3890 2010 Acked Read 2 Sent Done Done false
3891 2011 Acked Read 1 Sent Done Done false
3892 2012 Acked Read 7 Sent Done Done false
3893 2013 Acked Read 9 Sent Done Done false
3894 2014 Acked Read 2 Sent Done Done false
3895 2015 Acked Read 6 Sent Done Done false
3896 2016 Acked Read 7 Sent Done Done false
3897 2017 Acked Flush 0 Sent Done Done false
3898 2018 Acked Read 2 Sent Done Done false
3899 2019 Acked Flush 0 Sent Done Done false
3900 2020 Acked Flush 0 Sent Done Done false
3901 2021 Acked Read 4 Sent Done Done false
3902 2022 Acked Read 9 Sent Done Done false
3903 2023 Acked Flush 0 Sent Done Done false
3904 2024 Acked Write 10 Sent Done Done false
3905 2025 Acked Read 9 Sent Done Done false
3906 2026 Acked Write 4 Sent Done Done false
3907 2027 Acked Read 5 Sent Done Done false
3908 2028 Acked Write 7 Sent Done Done false
3909 2029 Acked Read 3 Sent Done Done false
3910 2030 Acked Write 1 Sent Done Done false
3911 2031 Acked Read 8 Sent Done Done false
3912 2032 Acked Read 1 Sent Done Done false
3913 2033 Acked Write 8 Sent Done Done false
3914 2034 Acked Write 3 Sent Done Done false
3915 2035 Acked Write 8 Sent Done Done false
3916 2036 Acked Write 10 Sent Done Done false
3917 2037 Acked Write 10 Sent Done Done false
3918 2038 Acked Read 8 Sent Done Done false
3919 2039 Acked Read 9 Sent Done Done false
3920 2040 Acked Read 2 Sent Done Done false
3921 2041 Acked Read 1 Sent Done Done false
3922 2042 Acked Read 9 Sent Done Done false
3923 2043 Acked Read 4 Sent Done Done false
3924 2044 Acked Read 8 Sent Done Done false
3925 2045 Acked Read 4 Sent Done Done false
3926 2046 Acked Read 7 Sent Done Done false
3927 2047 Acked Write 4 Sent Done Done false
3928 2048 Acked Read 4 Sent Done Done false
3929 2049 Acked Write 2 Sent Done Done false
3930 2050 Acked Read 6 Sent Done Done false
3931 2051 Acked Write 4 Sent Done Done false
3932 2052 Acked Read 9 Sent Done Done false
3933 2053 Acked Flush 0 Sent Done Done false
3934 2054 Acked Read 8 Sent Done Done false
3935 2055 Acked Write 8 Sent Done Done false
3936 2056 Acked Write 4 Sent Done Done false
3937 2057 Acked Read 2 Sent Done Done false
3938 2058 Acked Write 1 Sent Done Done false
3939 2059 Acked Read 8 Sent Done Done false
3940 2060 Acked Read 4 Sent Done Done false
3941 2061 Acked Write 1 Sent Done Done false
3942 2062 Acked Read 1 Sent Done Done false
3943 2063 Acked Write 8 Sent Done Done false
3944 2064 Acked Write 1 Sent Done Done false
3945 2065 Acked Read 2 Sent Done Done false
3946 2066 Acked Read 8 Sent Done Done false
3947 2067 Acked Read 7 Sent Done Done false
3948 2068 Acked Read 6 Sent Done Done false
3949 2069 Acked Read 9 Sent Done Done false
3950 2070 Acked Read 5 Sent Done Done false
3951 2071 Acked Write 7 Sent Done Done false
3952 2072 Acked Write 2 Sent Done Done false
3953 2073 Acked Read 5 Sent Done Done false
3954 2074 Acked Read 8 Sent Done Done false
3955 2075 Acked Read 3 Sent Done Done false
3956 2076 Acked Write 4 Sent Done Done false
3957 2077 Acked Write 8 Sent Done Done false
3958 2078 Acked Write 4 Sent Done Done false
3959 2079 Acked Read 9 Sent Done Done false
3960 2080 Acked Flush 0 Sent Done Done false
3961 2081 Acked Read 7 Sent Done Done false
3962 2082 Acked Write 9 Sent Done Done false
3963 2083 Acked Read 10 Sent Done Done false
3964 2084 Acked Read 2 Sent Done Done false
3965 2085 Acked Write 3 Sent Done Done false
3966 2086 Acked Write 1 Sent Done Done false
3967 2087 Acked Flush 0 Sent Done Done false
3968 2088 Acked Flush 0 Sent Done Done false
3969 2089 Acked Read 6 Sent Done Done false
3970 2090 Acked Read 5 Sent Done Done false
3971 2091 Acked Write 3 Sent Done Done false
3972 2092 Acked Read 8 Sent Done Done false
3973 2093 Acked Read 1 Sent Done Done false
3974 2094 Acked Read 3 Sent Done Done false
3975 2095 Acked Read 9 Sent Done Done false
3976 2096 Acked Write 4 Sent Done Done false
3977 2097 Acked Read 10 Sent Done Done false
3978 2098 Acked Read 6 Sent Done Done false
3979 2099 Acked Read 9 Sent Done Done false
3980 2100 Acked Write 10 Sent Done Done false
3981 2101 Acked Write 7 Sent Done Done false
3982 2102 Acked Read 6 Sent Done Done false
3983 2103 Acked Write 5 Sent Done Done false
3984 2104 Acked Read 2 Sent Done Done false
3985 2105 Acked Flush 0 Sent Done Done false
3986 2106 Acked Read 6 Sent Done Done false
3987 2107 Acked Read 3 Sent Done Done false
3988 2108 Acked Write 9 Sent Done Done false
3989 2109 Acked Write 3 Sent Done Done false
3990 2110 Acked Read 6 Sent Done Done false
3991 2111 Acked Write 2 Sent Done Done false
3992 2112 Acked Read 1 Sent Done Done false
3993 2113 Acked Read 6 Sent Done Done false
3994 2114 Acked Write 8 Sent Done Done false
3995 2115 Acked Read 8 Sent Done Done false
3996 2116 Acked Read 3 Sent Done Done false
3997 2117 Acked Write 2 Sent Done Done false
3998 2118 Acked Read 9 Sent Done Done false
3999 2119 Acked Read 8 Sent Done Done false
4000 2120 Acked Flush 0 Sent Done Done false
4001 2121 Acked Write 1 Sent Done Done false
4002 2122 Acked Read 7 Sent Done Done false
4003 2123 Acked Read 9 Sent Done Done false
4004 2124 Acked Read 1 Sent Done Done false
4005 2125 Acked Write 3 Sent Done Done false
4006 2126 Acked Read 10 Sent Done Done false
4007 2127 Acked Write 7 Sent Done Done false
4008 2128 Acked Read 2 Sent Done Done false
4009 2129 Acked Flush 0 Sent Done Done false
4010 2130 Acked Read 9 Sent Done Done false
4011 2131 Acked Read 5 Sent Done Done false
4012 2132 Acked Read 6 Sent Done Done false
4013 2133 Acked Flush 0 Sent Done Done false
4014 2134 Acked Read 4 Sent Done Done false
4015 2135 Acked Read 2 Sent Done Done false
4016 2136 Acked Read 9 Sent Done Done false
4017 2137 Acked Read 5 Sent Done Done false
4018 2138 Acked Read 9 Sent Done Done false
4019 2139 Acked Read 3 Sent Done Done false
4020 2140 Acked Read 8 Sent Done Done false
4021 2141 Acked Read 1 Sent Done Done false
4022 2142 Acked Read 1 Sent Done Done false
4023 2143 Acked Write 8 Sent Done Done false
4024 2144 Acked Read 1 Sent Done Done false
4025 2145 Acked Read 6 Sent Done Done false
4026 2146 Acked Read 8 Sent Done Done false
4027 2147 Acked Write 9 Sent Done Done false
4028 2148 Acked Read 7 Sent Done Done false
4029 2149 Acked Write 8 Sent Done Done false
4030 2150 Acked Read 9 Sent Done Done false
4031 2151 Acked Write 9 Sent Done Done false
4032 2152 Acked Flush 0 Sent Done Done false
4033 2153 Acked Read 1 Sent Done Done false
4034 2154 Acked Read 9 Sent Done Done false
4035 2155 Acked Read 9 Sent Done Done false
4036 2156 Acked Write 8 Sent Done Done false
4037 2157 Acked Read 10 Sent Done Done false
4038 2158 Acked Flush 0 Sent Done Done false
4039 2159 Acked Write 7 Sent Done Done false
4040 2160 Acked Write 5 Sent Done Done false
4041 2161 Acked Write 5 Sent Done Done false
4042 2162 Acked Read 4 Sent Done Done false
4043 2163 Acked Read 7 Sent Done Done false
4044 2164 Acked Write 6 Sent Done Done false
4045 2165 Acked Write 1 Sent Done Done false
4046 2166 Acked Flush 0 Sent Done Done false
4047 2167 Acked Write 6 Sent Done Done false
4048 2168 Acked Read 6 Sent Done Done false
4049 2169 Acked Read 1 Sent Done Done false
4050 2170 Acked Write 7 Sent Done Done false
4051 2171 Acked Write 7 Sent Done Done false
4052 2172 Acked Read 1 Sent Done Done false
4053 2173 Acked Read 6 Sent Done Done false
4054 2174 Acked Read 3 Sent Done Done false
4055 2175 Acked Read 7 Sent Done Done false
4056 2176 Acked Flush 0 Sent Done Done false
4057 2177 Acked Flush 0 Sent Done Done false
4058 2178 Acked Write 8 Sent Done Done false
4059 2179 Acked Read 5 Sent Done Done false
4060 2180 Acked Read 1 Sent Done Done false
4061 2181 Acked Read 6 Sent Done Done false
4062 2182 Acked Flush 0 Sent Done Done false
4063 2183 Acked Read 4 Sent Done Done false
4064 2184 Acked Flush 0 Sent Done Done false
4065 2185 Acked Write 1 Sent Done Done false
4066 2186 Acked Write 1 Sent Done Done false
4067 2187 Acked Read 2 Sent Done Done false
4068 2188 Acked Write 3 Sent Done Done false
4069 2189 Acked Flush 0 Sent Done Done false
4070 2190 Acked Write 10 Sent Done Done false
4071 2191 Acked Write 9 Sent Done Done false
4072 2192 Acked Read 9 Sent Done Done false
4073 2193 Acked Read 3 Sent Done Done false
4074 2194 Acked Write 9 Sent Done Done false
4075 2195 Acked Write 10 Sent Done Done false
4076 2196 Acked Write 6 Sent Done Done false
4077 2197 Acked Read 8 Sent Done Done false
4078 2198 Acked Read 9 Sent Done Done false
4079 2199 Acked Flush 0 Sent Done Done false
4080 2200 Acked Read 9 Sent Done Done false
4081 2201 Acked Flush 0 Sent Done Done false
4082 2202 Acked Write 10 Sent Done Done false
4083 2203 Acked Read 5 Sent Done Done false
4084 2204 Acked Read 2 Sent Done Done false
4085 2205 Acked Write 2 Sent Done Done false
4086 2206 Acked Write 7 Sent Done Done false
4087 STATES DS:0 DS:1 DS:2 TOTAL
4088 Sent 301 0 0 301
4089 Done 0 301 301 602
4090 Skipped 0 0 0 0
4091 Error 0 0 0 0
4092 Last Flush: Some(JobId(1905)) Some(JobId(2201)) Some(JobId(2201))
4093 Downstairs last five completed: 1905 1904 1903 1902 1901
4094 Upstairs last five completed: 2206 2205 2204 2203 2202
4095 CLIENT: Up:0 ds:301 act:2
40962025-02-25T17:19:03.388ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
40972025-02-25T17:19:04.372ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
40982025-02-25T17:19:04.622ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
40992025-02-25T17:19:04.622ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60877) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
41002025-02-25T17:19:04.624ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
41012025-02-25T17:19:04.624ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
41022025-02-25T17:19:04.624ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
41032025-02-25T17:19:04.624ZINFOcrucible: send last flush ID to this DS: Some(JobId(1905)) = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
41042025-02-25T17:19:04.624ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
41052025-02-25T17:19:04.624ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(1905)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
41062025-02-25T17:19:04.625ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(1905)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
4107 ----------------------------------------------------------------
4108 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4109 Downstairs last five completed: 2207 2206 2205 2204 2203
4110 Upstairs last five completed: 2207 2206 2205 2204 2203
4111 CLIENT: Up:0 ds:0 act:3
4112 Replay: All jobs finished
4113 Main task has work to do, go find it
4114 got dsc Stop(2)
4115 stop 2
4116 Feb 25 17:19:06.889 INFO request completed, latency_us: 376[8830] Got stop action so:false kr:false
4117 , response_code: 204, uri: /stop/cid/2, method: POST, req_id: bc370f1c-624d-424d-ab32-d0413903c1ad, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
41182025-02-25T17:19:06.890ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
41192025-02-25T17:19:06.890ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
41202025-02-25T17:19:06.890ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
4121 Feb 25 17:19:06.890 INFO request completed, latency_us: 163, response_code: 200, uri: /state/cid/2, method: GET, req_id: acaf0a54-7897-4a9d-bea3-fdbcfcf661d8, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
41222025-02-25T17:19:06.890ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
4123 [8830] Exited with: ExitStatus(unix_wait_status(9))
4124 [8830][2] reports Exit
4125 Feb 25 17:19:10.891 INFO request completed, latency_us: 245, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5f03b201-b4b6-4d6a-b58d-821e35864ad6, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
4126 001/300 Flush
4127 002/300 Write block 3486 len 28672 data: 3 3 3 3 3 3 3
4128 003/300 Read block 2916 len 20480
4129 004/300 Read block 3348 len 16384
4130 005/300 Write block 1450 len 12288 data: 3 3 3
4131 006/300 Write block 3110 len 36864 data: 2 2 2 2 2 2 2 2 2
4132 007/300 Read block 3066 len 12288
4133 008/300 Write block 3540 len 16384 data: 2 2 2 2
4134 009/300 Write block 3581 len 16384 data: 2 2 2 4
4135 010/300 Read block 459 len 12288
4136 011/300 Write block 2057 len 20480 data: 2 2 2 2 2
4137 012/300 Flush
4138 013/300 Read block 1153 len 24576
4139 014/300 Read block 489 len 28672
4140 015/300 Read block 382 len 32768
4141 016/300 Write block 1519 len 8192 data: 2 2
4142 017/300 Write block 4195 len 8192 data: 2 2
4143 018/300 Read block 3449 len 12288
4144 019/300 Write block 2453 len 28672 data: 2 2 2 2 3 3 4
4145 020/300 Write block 663 len 8192 data: 2 2
4146 021/300 Read block 3650 len 40960
4147 022/300 Flush
4148 023/300 Write block 480 len 24576 data: 2 2 2 2 3 3
4149 024/300 Write block 2928 len 4096 data: 2
4150 025/300 Read block 799 len 24576
4151 026/300 Read block 1244 len 12288
4152 027/300 Write block 3068 len 16384 data: 2 2 2 2
4153 028/300 Read block 3678 len 16384
4154 029/300 Write block 3353 len 32768 data: 3 3 3 3 3 3 3 4
4155 030/300 Read block 1441 len 16384
4156 031/300 Read block 2018 len 28672
4157 032/300 Read block 223 len 36864
4158 033/300 Read block 735 len 28672
4159 034/300 Write block 3683 len 36864 data: 3 3 3 3 3 3 3 2 2
4160 035/300 Read block 3573 len 4096
4161 036/300 Read block 4090 len 40960
4162 037/300 Read block 2361 len 8192
4163 038/300 Write block 1039 len 36864 data: 2 3 4 4 4 4 4 4 4
4164 039/300 Flush
4165 040/300 Write block 2450 len 36864 data: 3 2 2 3 3 3 3 4 4
4166 041/300 Read block 296 len 28672
4167 042/300 Write block 674 len 24576 data: 2 2 2 2 2 2
4168 043/300 Write block 3446 len 20480 data: 2 2 2 2 2
4169 044/300 Read block 1837 len 32768
4170 045/300 Read block 791 len 32768
4171 046/300 Write block 2660 len 28672 data: 2 2 2 3 4 4 4
4172 047/300 Read block 1567 len 8192
4173 048/300 Flush
4174 049/300 Read block 2965 len 20480
4175 050/300 Write block 1721 len 4096 data: 3
4176 051/300 Write block 508 len 12288 data: 2 2 2
4177 052/300 Read block 327 len 16384
4178 053/300 Write block 3535 len 24576 data: 2 2 2 2 2 3
4179 054/300 Read block 1536 len 16384
4180 055/300 Read block 1416 len 40960
4181 056/300 Read block 1409 len 32768
4182 057/300 Read block 474 len 20480
4183 058/300 Write block 3941 len 20480 data: 5 5 5 5 5
4184 059/300 Read block 567 len 36864
4185 060/300 Write block 2103 len 24576 data: 2 2 2 2 2 2
4186 061/300 Read block 970 len 28672
4187 062/300 Write block 3400 len 12288 data: 3 4 4
4188 063/300 Read block 910 len 12288
4189 064/300 Read block 196 len 4096
4190 065/300 Write block 776 len 28672 data: 2 2 2 2 2 2 2
4191 066/300 Flush
4192 067/300 Write block 94 len 24576 data: 2 2 2 2 2 3
4193 068/300 Read block 3711 len 40960
4194 069/300 Write block 4989 len 28672 data: 2 2 2 2 2 2 2
4195 070/300 Read block 526 len 16384
4196 071/300 Write block 2822 len 32768 data: 2 2 2 2 2 2 2 2
4197 072/300 Write block 3814 len 20480 data: 3 4 4 4 4
4198 073/300 Read block 2626 len 36864
4199 074/300 Write block 4941 len 24576 data: 4 4 2 2 2 2
4200 075/300 Write block 4562 len 8192 data: 3 3
4201 076/300 Write block 275 len 28672 data: 3 3 2 2 3 3 3
4202 077/300 Write block 4107 len 32768 data: 2 2 2 2 2 2 2 2
4203 078/300 Read block 3639 len 16384
4204 079/300 Read block 2684 len 28672
4205 080/300 Read block 619 len 40960
4206 081/300 Read block 2881 len 36864
4207 082/300 Read block 702 len 40960
4208 083/300 Flush
4209 084/300 Flush
4210 085/300 Write block 4822 len 32768 data: 2 2 2 2 2 2 2 2
4211 086/300 Read block 4319 len 4096
4212 087/300 Write block 3382 len 20480 data: 2 2 3 3 3
4213 088/300 Read block 219 len 24576
4214 089/300 Read block 4521 len 8192
4215 090/300 Flush
4216 091/300 Read block 4924 len 8192
4217 092/300 Read block 4514 len 28672
4218 093/300 Flush
4219 094/300 Read block 1968 len 4096
4220 095/300 Read block 4588 len 24576
4221 096/300 Read block 4275 len 24576
4222 097/300 Write block 1401 len 20480 data: 3 3 2 2 2
4223 098/300 Write block 4530 len 16384 data: 2 4 3 3
4224 099/300 Read block 3650 len 36864
4225 100/300 Read block 4907 len 4096
4226 101/300 Read block 1121 len 32768
4227 102/300 Read block 4578 len 12288
4228 103/300 Write block 2216 len 40960 data: 3 4 4 4 4 3 3 3 2 2
4229 104/300 Read block 2193 len 28672
4230 105/300 Read block 837 len 4096
4231 106/300 Read block 2587 len 28672
4232 107/300 Write block 3991 len 28672 data: 2 2 2 2 2 2 2
4233 108/300 Write block 883 len 24576 data: 2 2 2 4 4 4
4234 109/300 Read block 1180 len 28672
4235 110/300 Write block 4854 len 36864 data: 2 2 2 2 2 2 2 2 2
4236 111/300 Read block 3730 len 4096
4237 112/300 Read block 3067 len 16384
4238 113/300 Read block 1911 len 16384
4239 114/300 Read block 350 len 40960
4240 115/300 Read block 3865 len 24576
4241 116/300 Write block 821 len 20480 data: 2 2 2 2 2
4242 117/300 Read block 3612 len 20480
4243 118/300 Flush
4244 119/300 Read block 861 len 16384
4245 120/300 Flush
4246 121/300 Read block 3604 len 20480
4247 122/300 Read block 1851 len 12288
4248 123/300 Read block 4103 len 24576
4249 124/300 Write block 4078 len 8192 data: 2 2
4250 125/300 Read block 3325 len 8192
4251 126/300 Read block 4744 len 20480
4252 127/300 Write block 3118 len 36864 data: 3 2 2 2 2 2 2 2 2
4253 128/300 Write block 2830 len 36864 data: 2 2 2 2 2 3 3 3 3
4254 129/300 Flush
4255 130/300 Read block 2880 len 12288
4256 131/300 Read block 256 len 32768
4257 132/300 Write block 124 len 28672 data: 2 2 2 2 2 2 2
4258 133/300 Read block 1584 len 12288
4259 134/300 Write block 2998 len 36864 data: 3 3 3 4 5 4 4 4 4
4260 135/300 Read block 326 len 32768
4261 136/300 Write block 3876 len 20480 data: 3 3 3 3 3
4262 137/300 Write block 2330 len 4096 data: 5
4263 138/300 Read block 3143 len 40960
4264 139/300 Write block 1151 len 32768 data: 2 2 2 2 2 2 3 3
4265 140/300 Write block 3658 len 32768 data: 3 3 3 3 2 2 2 3
4266 141/300 Read block 1001 len 8192
4267 142/300 Write block 2092 len 20480 data: 2 2 2 2 2
4268 143/300 Flush
4269 144/300 Read block 3251 len 40960
4270 145/300 Read block 893 len 40960
4271 146/300 Read block 16 len 40960
4272 147/300 Write block 1062 len 12288 data: 4 3 3
4273 148/300 Read block 4142 len 28672
4274 149/300 Read block 2171 len 28672
4275 150/300 Read block 1779 len 16384
4276 151/300 Read block 230 len 20480
4277 152/300 Read block 1708 len 24576
4278 153/300 Write block 539 len 28672 data: 3 3 2 2 2 2 3
4279 154/300 Read block 2220 len 28672
4280 155/300 Write block 1710 len 8192 data: 3 3
4281 156/300 Read block 2487 len 36864
4282 157/300 Read block 315 len 20480
4283 158/300 Read block 4039 len 12288
4284 159/300 Write block 1467 len 40960 data: 3 3 3 3 3 3 3 3 2 2
4285 160/300 Flush
4286 161/300 Read block 3189 len 36864
4287 162/300 Write block 3909 len 12288 data: 3 2 3
4288 163/300 Write block 314 len 12288 data: 2 4 3
4289 164/300 Read block 2067 len 8192
4290 165/300 Read block 1941 len 8192
4291 166/300 Read block 4236 len 28672
4292 167/300 Write block 2433 len 28672 data: 2 2 2 2 2 2 3
4293 168/300 Read block 1401 len 4096
4294 169/300 Read block 2084 len 4096
4295 170/300 Write block 1991 len 8192 data: 2 2
4296 171/300 Read block 4403 len 36864
4297 172/300 Write block 1189 len 28672 data: 3 3 3 3 3 3 3
4298 173/300 Write block 621 len 24576 data: 3 3 3 3 3 2
4299 174/300 Write block 4443 len 8192 data: 2 2
4300 175/300 Read block 4685 len 8192
4301 176/300 Read block 1261 len 36864
4302 177/300 Flush
4303 178/300 Read block 3406 len 4096
4304 179/300 Read block 1721 len 20480
4305 180/300 Read block 2563 len 24576
4306 181/300 Write block 2477 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4307 182/300 Read block 2412 len 32768
4308 183/300 Read block 2748 len 20480
4309 184/300 Write block 4533 len 20480 data: 4 3 3 2 2
4310 185/300 Write block 3402 len 4096 data: 5
4311 186/300 Read block 3695 len 20480
4312 187/300 Write block 2938 len 40960 data: 2 2 2 2 2 2 2 2 3 3
4313 188/300 Write block 2199 len 8192 data: 3 3
4314 189/300 Write block 1348 len 36864 data: 2 2 2 2 2 2 2 2 2
4315 190/300 Read block 3228 len 12288
4316 191/300 Read block 4588 len 28672
4317 192/300 Read block 4582 len 40960
4318 193/300 Write block 1062 len 4096 data: 5
4319 194/300 Write block 3313 len 16384 data: 2 2 2 2
4320 195/300 Write block 1018 len 32768 data: 2 2 2 2 2 2 2 3
4321 196/300 Write block 1873 len 8192 data: 2 2
4322 197/300 Write block 922 len 12288 data: 2 2 2
4323 198/300 Read block 1823 len 36864
4324 199/300 Read block 1418 len 8192
4325 200/300 Write block 1375 len 12288 data: 3 2 2
4326 201/300 Read block 3944 len 32768
4327 202/300 Write block 214 len 8192 data: 3 3
4328 203/300 Read block 147 len 20480
4329 204/300 Write block 2538 len 36864 data: 2 2 2 3 3 3 2 2 2
4330 205/300 Read block 658 len 28672
4331 206/300 Write block 2280 len 20480 data: 2 2 3 3 3
4332 207/300 Write block 4133 len 20480 data: 3 4 4 3 2
4333 208/300 Read block 1305 len 28672
4334 209/300 Read block 2648 len 24576
4335 210/300 Write block 1625 len 28672 data: 3 3 2 3 3 3 3
4336 211/300 Write block 3416 len 20480 data: 2 2 2 2 2
4337 212/300 Read block 1558 len 4096
4338 213/300 Flush
4339 214/300 Read block 2018 len 24576
4340 215/300 Write block 4671 len 40960 data: 3 3 3 3 2 2 2 2 2 2
4341 216/300 Flush
4342 217/300 Read block 3082 len 28672
4343 218/300 Write block 43 len 16384 data: 2 2 2 2
4344 219/300 Read block 1279 len 24576
4345 220/300 Write block 1522 len 28672 data: 2 3 3 3 3 3 3
4346 221/300 Read block 1694 len 24576
4347 222/300 Read block 371 len 32768
4348 223/300 Write block 3995 len 32768 data: 3 3 3 2 2 2 2 2
4349 224/300 Flush
4350 225/300 Write block 710 len 28672 data: 3 2 2 2 2 2 2
4351 226/300 Write block 4187 len 24576 data: 2 2 2 2 2 2
4352 227/300 Read block 4678 len 20480
4353 228/300 Write block 2625 len 28672 data: 3 3 2 2 2 2 2
4354 229/300 Read block 2608 len 20480
4355 230/300 Read block 1401 len 40960
4356 231/300 Read block 4306 len 20480
4357 232/300 Write block 2116 len 4096 data: 2
4358 233/300 Read block 122 len 32768
4359 234/300 Flush
4360 235/300 Write block 518 len 20480 data: 2 2 2 2 2
4361 236/300 Flush
4362 237/300 Read block 518 len 16384
4363 238/300 Read block 1371 len 12288
4364 239/300 Read block 2514 len 4096
4365 240/300 Read block 4955 len 12288
4366 241/300 Read block 4637 len 24576
4367 242/300 Read block 4148 len 32768
4368 243/300 Flush
4369 244/300 Read block 8 len 16384
4370 245/300 Write block 3176 len 16384 data: 2 2 2 2
4371 246/300 Write block 663 len 20480 data: 3 3 2 2 2
4372 247/300 Read block 3070 len 20480
4373 248/300 Read block 2915 len 32768
4374 249/300 Read block 116 len 8192
4375 250/300 Read block 1509 len 36864
4376 251/300 Write block 2943 len 8192 data: 3 3
4377 252/300 Flush
4378 253/300 Write block 4219 len 4096 data: 2
4379 254/300 Read block 373 len 16384
4380 255/300 Write block 1110 len 20480 data: 3 3 3 3 2
4381 256/300 Write block 4634 len 28672 data: 3 3 3 4 3 3 2
4382 257/300 Read block 4266 len 24576
4383 258/300 Flush
4384 259/300 Flush
4385 260/300 Read block 396 len 32768
4386 261/300 Read block 679 len 16384
4387 262/300 Read block 4469 len 16384
4388 263/300 Read block 2696 len 36864
4389 264/300 Write block 3520 len 20480 data: 3 3 3 3 2
4390 265/300 Write block 983 len 28672 data: 3 2 2 2 2 2 2
4391 266/300 Read block 2409 len 4096
4392 267/300 Read block 3559 len 32768
4393 268/300 Flush
4394 269/300 Write block 3912 len 8192 data: 3 3
4395 270/300 Write block 1927 len 40960 data: 2 2 3 4 5 5 4 4 5 5
4396 271/300 Read block 1141 len 4096
4397 272/300 Read block 2041 len 36864
4398 273/300 Write block 4840 len 28672 data: 2 2 2 2 2 2 3
4399 274/300 Read block 1211 len 8192
4400 275/300 Read block 235 len 20480
4401 276/300 Read block 3166 len 32768
4402 277/300 Read block 2096 len 28672
4403 278/300 Read block 3239 len 4096
4404 279/300 Write block 4491 len 16384 data: 2 2 2 2
4405 280/300 Read block 396 len 28672
4406 281/300 Read block 3593 len 24576
4407 282/300 Read block 2183 len 32768
4408 283/300 Read block 2893 len 36864
4409 284/300 Read block 873 len 4096
4410 285/300 Read block 2815 len 36864
4411 286/300 Read block 682 len 28672
4412 287/300 Write block 2572 len 32768 data: 3 3 3 2 2 2 2 2
4413 288/300 Read block 2466 len 12288
4414 289/300 Read block 2113 len 40960
4415 290/300 Read block 1611 len 24576
4416 291/300 Flush
4417 292/300 Flush
4418 293/300 Read block 1710 len 16384
4419 294/300 Write block 4206 len 12288 data: 3 3 3
4420 295/300 Write block 1553 len 24576 data: 2 2 2 2 2 2
4421 296/300 Read block 1370 len 24576
4422 297/300 Write block 1430 len 8192 data: 3 3
4423 298/300 Write block 678 len 36864 data: 3 3 2 2 2 2 2 2 2
4424 299/300 Read block 4574 len 36864
4425 300/300 Read block 2401 len 12288
4426 301/300 Read block 2679 len 24576
4427 Main task has work to do, go find it
4428 got dsc Start(2)
4429 start 2
4430 Feb 25 17:19:11.155 INFO request completed, latency_us: 232, response_code: 204, uri: /start/cid/2, method: POST, req_id: d9303b87-edfb-419e-8343-5e31519c26b8, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
4431 Replay: started 2, returned:Ok(())
4432 ----------------------------------------------------------------
4433 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4434 States: Active Active Offline
4435 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4436 2208 Acked Flush 0 Done Done Sent false
4437 2209 Acked Write 7 Done Done Sent false
4438 2210 Acked Read 5 Done Done Sent false
4439 2211 Acked Read 4 Done Done Sent false
4440 2212 Acked Write 3 Done Done Sent false
4441 2213 Acked Write 9 Done Done Sent false
4442 2214 Acked Read 3 Done Done Sent false
4443 2215 Acked Write 4 Done Done Sent false
4444 2216 Acked Write 4 Done Done Sent false
4445 2217 Acked Read 3 Done Done Sent false
4446 2218 Acked Write 5 Done Done Sent false
4447 2219 Acked Flush 0 Done Done Sent false
4448 2220 Acked Read 6 Done Done Sent false
4449 2221 Acked Read 7 Done Done Sent false
4450 2222 Acked Read 8 Done Done Sent false
4451 2223 Acked Write 2 Done Done Sent false
4452 2224 Acked Write 2 Done Done Sent false
4453 2225 Acked Read 3 Done Done Sent false
4454 2226 Acked Write 7 Done Done Sent false
4455 2227 Acked Write 2 Done Done Sent false
4456 2228 Acked Read 10 Done Done Sent false
4457 2229 Acked Flush 0 Done Done Sent false
4458 2230 Acked Write 6 Done Done Sent false
4459 2231 Acked Write 1 Done Done Sent false
4460 2232 Acked Read 6 Done Done Sent false
4461 2233 Acked Read 3 Done Done Sent false
4462 2234 Acked Write 4 Done Done Sent false
4463 2235 Acked Read 4 Done Done Sent false
4464 2236 Acked Write 8 Done Done Sent false
4465 2237 Acked Read 4 Done Done Sent false
4466 2238 Acked Read 7 Done Done Sent false
4467 2239 Acked Read 9 Done Done Sent false
4468 2240 Acked Read 7 Done Done Sent false
4469 2241 Acked Write 9 Done Done Sent false
4470 2242 Acked Read 1 Done Done Sent false
4471 2243 Acked Read 10 Done Done Sent false
4472 2244 Acked Read 2 Done Done Sent false
4473 2245 Acked Write 9 Done Done Sent false
4474 2246 Acked Flush 0 Done Done Sent false
4475 2247 Acked Write 9 Done Done Sent false
4476 2248 Acked Read 7 Done Done Sent false
4477 2249 Acked Write 6 Done Done Sent false
4478 2250 Acked Write 5 Done Done Sent false
4479 2251 Acked Read 8 Done Done Sent false
4480 2252 Acked Read 8 Done Done Sent false
4481 2253 Acked Write 7 Done Done Sent false
4482 2254 Acked Read 2 Done Done Sent false
4483 2255 Acked Flush 0 Done Done Sent false
4484 2256 Acked Read 5 Done Done Sent false
4485 2257 Acked Write 1 Done Done Sent false
4486 2258 Acked Write 3 Done Done Sent false
4487 2259 Acked Read 4 Done Done Sent false
4488 2260 Acked Write 6 Done Done Sent false
4489 2261 Acked Read 4 Done Done Sent false
4490 2262 Acked Read 10 Done Done Sent false
4491 2263 Acked Read 8 Done Done Sent false
4492 2264 Acked Read 5 Done Done Sent false
4493 2265 Acked Write 5 Done Done Sent false
4494 2266 Acked Read 9 Done Done Sent false
4495 2267 Acked Write 6 Done Done Sent false
4496 2268 Acked Read 7 Done Done Sent false
4497 2269 Acked Write 3 Done Done Sent false
4498 2270 Acked Read 3 Done Done Sent false
4499 2271 Acked Read 1 Done Done Sent false
4500 2272 Acked Write 7 Done Done Sent false
4501 2273 Acked Flush 0 Done Done Sent false
4502 2274 Acked Write 6 Done Done Sent false
4503 2275 Acked Read 10 Done Done Sent false
4504 2276 Acked Write 7 Done Done Sent false
4505 2277 Acked Read 4 Done Done Sent false
4506 2278 Acked Write 8 Done Done Sent false
4507 2279 Acked Write 5 Done Done Sent false
4508 2280 Acked Read 9 Done Done Sent false
4509 2281 Acked Write 6 Done Done Sent false
4510 2282 Acked Write 2 Done Done Sent false
4511 2283 Acked Write 7 Done Done Sent false
4512 2284 Acked Write 8 Done Done Sent false
4513 2285 Acked Read 4 Done Done Sent false
4514 2286 Acked Read 7 Done Done Sent false
4515 2287 Acked Read 10 Done Done Sent false
4516 2288 Acked Read 9 Done Done Sent false
4517 2289 Acked Read 10 Done Done Sent false
4518 2290 Acked Flush 0 Done Done Sent false
4519 2291 Acked Flush 0 Done Done Sent false
4520 2292 Acked Write 8 Done Done Sent false
4521 2293 Acked Read 1 Done Done Sent false
4522 2294 Acked Write 5 Done Done Sent false
4523 2295 Acked Read 6 Done Done Sent false
4524 2296 Acked Read 2 Done Done Sent false
4525 2297 Acked Flush 0 Done Done Sent false
4526 2298 Acked Read 2 Done Done Sent false
4527 2299 Acked Read 7 Done Done Sent false
4528 2300 Acked Flush 0 Done Done Sent false
4529 2301 Acked Read 1 Done Done Sent false
4530 2302 Acked Read 6 Done Done Sent false
4531 2303 Acked Read 6 Done Done Sent false
4532 2304 Acked Write 5 Done Done Sent false
4533 2305 Acked Write 4 Done Done Sent false
4534 2306 Acked Read 9 Done Done Sent false
4535 2307 Acked Read 1 Done Done Sent false
4536 2308 Acked Read 8 Done Done Sent false
4537 2309 Acked Read 3 Done Done Sent false
4538 2310 Acked Write 10 Done Done Sent false
4539 2311 Acked Read 7 Done Done Sent false
4540 2312 Acked Read 1 Done Done Sent false
4541 2313 Acked Read 7 Done Done Sent false
4542 2314 Acked Write 7 Done Done Sent false
4543 2315 Acked Write 6 Done Done Sent false
4544 2316 Acked Read 7 Done Done Sent false
4545 2317 Acked Write 9 Done Done Sent false
4546 2318 Acked Read 1 Done Done Sent false
4547 2319 Acked Read 4 Done Done Sent false
4548 2320 Acked Read 4 Done Done Sent false
4549 2321 Acked Read 10 Done Done Sent false
4550 2322 Acked Read 6 Done Done Sent false
4551 2323 Acked Write 5 Done Done Sent false
4552 2324 Acked Read 5 Done Done Sent false
4553 2325 Acked Flush 0 Done Done Sent false
4554 2326 Acked Read 4 Done Done Sent false
4555 2327 Acked Flush 0 Done Done Sent false
4556 2328 Acked Read 5 Done Done Sent false
4557 2329 Acked Read 3 Done Done Sent false
4558 2330 Acked Read 6 Done Done Sent false
4559 2331 Acked Write 2 Done Done Sent false
4560 2332 Acked Read 2 Done Done Sent false
4561 2333 Acked Read 5 Done Done Sent false
4562 2334 Acked Write 9 Done Done Sent false
4563 2335 Acked Write 9 Done Done Sent false
4564 2336 Acked Flush 0 Done Done Sent false
4565 2337 Acked Read 3 Done Done Sent false
4566 2338 Acked Read 8 Done Done Sent false
4567 2339 Acked Write 7 Done Done Sent false
4568 2340 Acked Read 3 Done Done Sent false
4569 2341 Acked Write 9 Done Done Sent false
4570 2342 Acked Read 8 Done Done Sent false
4571 2343 Acked Write 5 Done Done Sent false
4572 2344 Acked Write 1 Done Done Sent false
4573 2345 Acked Read 10 Done Done Sent false
4574 2346 Acked Write 8 Done Done Sent false
4575 2347 Acked Write 8 Done Done Sent false
4576 2348 Acked Read 2 Done Done Sent false
4577 2349 Acked Write 5 Done Done Sent false
4578 2350 Acked Flush 0 Done Done Sent false
4579 2351 Acked Read 10 Done Done Sent false
4580 2352 Acked Read 10 Done Done Sent false
4581 2353 Acked Read 10 Done Done Sent false
4582 2354 Acked Write 3 Done Done Sent false
4583 2355 Acked Read 7 Done Done Sent false
4584 2356 Acked Read 7 Done Done Sent false
4585 2357 Acked Read 4 Done Done Sent false
4586 2358 Acked Read 5 Done Done Sent false
4587 2359 Acked Read 6 Done Done Sent false
4588 2360 Acked Write 7 Done Done Sent false
4589 2361 Acked Read 7 Done Done Sent false
4590 2362 Acked Write 2 Done Done Sent false
4591 2363 Acked Read 9 Done Done Sent false
4592 2364 Acked Read 5 Done Done Sent false
4593 2365 Acked Read 3 Done Done Sent false
4594 2366 Acked Write 10 Done Done Sent false
4595 2367 Acked Flush 0 Done Done Sent false
4596 2368 Acked Read 9 Done Done Sent false
4597 2369 Acked Write 3 Done Done Sent false
4598 2370 Acked Write 3 Done Done Sent false
4599 2371 Acked Read 2 Done Done Sent false
4600 2372 Acked Read 2 Done Done Sent false
4601 2373 Acked Read 7 Done Done Sent false
4602 2374 Acked Write 7 Done Done Sent false
4603 2375 Acked Read 1 Done Done Sent false
4604 2376 Acked Read 1 Done Done Sent false
4605 2377 Acked Write 2 Done Done Sent false
4606 2378 Acked Read 9 Done Done Sent false
4607 2379 Acked Write 7 Done Done Sent false
4608 2380 Acked Write 6 Done Done Sent false
4609 2381 Acked Write 2 Done Done Sent false
4610 2382 Acked Read 2 Done Done Sent false
4611 2383 Acked Read 9 Done Done Sent false
4612 2384 Acked Flush 0 Done Done Sent false
4613 2385 Acked Read 1 Done Done Sent false
4614 2386 Acked Read 5 Done Done Sent false
4615 2387 Acked Read 6 Done Done Sent false
4616 2388 Acked Write 10 Done Done Sent false
4617 2389 Acked Read 8 Done Done Sent false
4618 2390 Acked Read 5 Done Done Sent false
4619 2391 Acked Write 5 Done Done Sent false
4620 2392 Acked Write 1 Done Done Sent false
4621 2393 Acked Read 5 Done Done Sent false
4622 2394 Acked Write 10 Done Done Sent false
4623 2395 Acked Write 2 Done Done Sent false
4624 2396 Acked Write 9 Done Done Sent false
4625 2397 Acked Read 3 Done Done Sent false
4626 2398 Acked Read 7 Done Done Sent false
4627 2399 Acked Read 10 Done Done Sent false
4628 2400 Acked Write 1 Done Done Sent false
4629 2401 Acked Write 4 Done Done Sent false
4630 2402 Acked Write 8 Done Done Sent false
4631 2403 Acked Write 2 Done Done Sent false
4632 2404 Acked Write 3 Done Done Sent false
4633 2405 Acked Read 9 Done Done Sent false
4634 2406 Acked Read 2 Done Done Sent false
4635 2407 Acked Write 3 Done Done Sent false
4636 2408 Acked Read 8 Done Done Sent false
4637 2409 Acked Write 2 Done Done Sent false
4638 2410 Acked Read 5 Done Done Sent false
4639 2411 Acked Write 9 Done Done Sent false
4640 2412 Acked Read 7 Done Done Sent false
4641 2413 Acked Write 5 Done Done Sent false
4642 2414 Acked Write 5 Done Done Sent false
4643 2415 Acked Read 7 Done Done Sent false
4644 2416 Acked Read 6 Done Done Sent false
4645 2417 Acked Write 7 Done Done Sent false
4646 2418 Acked Write 5 Done Done Sent false
4647 2419 Acked Read 1 Done Done Sent false
4648 2420 Acked Flush 0 Done Done Sent false
4649 2421 Acked Read 6 Done Done Sent false
4650 2422 Acked Write 10 Done Done Sent false
4651 2423 Acked Flush 0 Done Done Sent false
4652 2424 Acked Read 7 Done Done Sent false
4653 2425 Acked Write 4 Done Done Sent false
4654 2426 Acked Read 6 Done Done Sent false
4655 2427 Acked Write 7 Done Done Sent false
4656 2428 Acked Read 6 Done Done Sent false
4657 2429 Acked Read 8 Done Done Sent false
4658 2430 Acked Write 8 Done Done Sent false
4659 2431 Acked Flush 0 Done Done Sent false
4660 2432 Acked Write 7 Done Done Sent false
4661 2433 Acked Write 6 Done Done Sent false
4662 2434 Acked Read 5 Done Done Sent false
4663 2435 Acked Write 7 Done Done Sent false
4664 2436 Acked Read 5 Done Done Sent false
4665 2437 Acked Read 10 Done Done Sent false
4666 2438 Acked Read 5 Done Done Sent false
4667 2439 Acked Write 1 Done Done Sent false
4668 2440 Acked Read 8 Done Done Sent false
4669 2441 Acked Flush 0 Done Done Sent false
4670 2442 Acked Write 5 Done Done Sent false
4671 2443 Acked Flush 0 Done Done Sent false
4672 2444 Acked Read 4 Done Done Sent false
4673 2445 Acked Read 3 Done Done Sent false
4674 2446 Acked Read 1 Done Done Sent false
4675 2447 Acked Read 3 Done Done Sent false
4676 2448 Acked Read 6 Done Done Sent false
4677 2449 Acked Read 8 Done Done Sent false
4678 2450 Acked Flush 0 Done Done Sent false
4679 2451 Acked Read 4 Done Done Sent false
4680 2452 Acked Write 4 Done Done Sent false
4681 2453 Acked Write 5 Done Done Sent false
4682 2454 Acked Read 5 Done Done Sent false
4683 2455 Acked Read 8 Done Done Sent false
4684 2456 Acked Read 2 Done Done Sent false
4685 2457 Acked Read 9 Done Done Sent false
4686 2458 Acked Write 2 Done Done Sent false
4687 2459 Acked Flush 0 Done Done Sent false
4688 2460 Acked Write 1 Done Done Sent false
4689 2461 Acked Read 4 Done Done Sent false
4690 2462 Acked Write 5 Done Done Sent false
4691 2463 Acked Write 7 Done Done Sent false
4692 2464 Acked Read 6 Done Done Sent false
4693 2465 Acked Flush 0 Done Done Sent false
4694 2466 Acked Flush 0 Done Done Sent false
4695 2467 Acked Read 8 Done Done Sent false
4696 2468 Acked Read 4 Done Done Sent false
4697 2469 Acked Read 4 Done Done Sent false
4698 2470 Acked Read 9 Done Done Sent false
4699 2471 Acked Write 5 Done Done Sent false
4700 2472 Acked Write 7 Done Done Sent false
4701 2473 Acked Read 1 Done Done Sent false
4702 2474 Acked Read 8 Done Done Sent false
4703 2475 Acked Flush 0 Done Done Sent false
4704 2476 Acked Write 2 Done Done Sent false
4705 2477 Acked Write 10 Done Done Sent false
4706 2478 Acked Read 1 Done Done Sent false
4707 2479 Acked Read 9 Done Done Sent false
4708 2480 Acked Write 7 Done Done Sent false
4709 2481 Acked Read 2 Done Done Sent false
4710 2482 Acked Read 5 Done Done Sent false
4711 2483 Acked Read 8 Done Done Sent false
4712 2484 Acked Read 7 Done Done Sent false
4713 2485 Acked Read 1 Done Done Sent false
4714 2486 Acked Write 4 Done Done Sent false
4715 2487 Acked Read 7 Done Done Sent false
4716 2488 Acked Read 6 Done Done Sent false
4717 2489 Acked Read 8 Done Done Sent false
4718 2490 Acked Read 9 Done Done Sent false
4719 2491 Acked Read 1 Done Done Sent false
4720 2492 Acked Read 9 Done Done Sent false
4721 2493 Acked Read 7 Done Done Sent false
4722 2494 Acked Write 8 Done Done Sent false
4723 2495 Acked Read 3 Done Done Sent false
4724 2496 Acked Read 10 Done Done Sent false
4725 2497 Acked Read 6 Done Done Sent false
4726 2498 Acked Flush 0 Done Done Sent false
4727 2499 Acked Flush 0 Done Done Sent false
4728 2500 Acked Read 4 Done Done Sent false
4729 2501 Acked Write 3 Done Done Sent false
4730 2502 Acked Write 6 Done Done Sent false
4731 2503 Acked Read 6 Done Done Sent false
4732 2504 Acked Write 2 Done Done Sent false
4733 2505 Acked Write 9 Done Done Sent false
4734 2506 Acked Read 9 Done Done Sent false
4735 2507 Acked Read 3 Done Done Sent false
4736 2508 Acked Read 6 Done Done Sent false
4737 STATES DS:0 DS:1 DS:2 TOTAL
4738 Sent 0 0 301 301
4739 Done 301 301 0 602
4740 Skipped 0 0 0 0
4741 Error 0 0 0 0
4742 Last Flush: Some(JobId(2499)) Some(JobId(2499)) Some(JobId(2207))
4743 Downstairs last five completed: 2207 2206 2205 2204 2203
4744 Upstairs last five completed: 2508 2507 2506 2505 2504
4745 CLIENT: Up:0 ds:301 act:2
47462025-02-25T17:19:11.657ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
47472025-02-25T17:19:12.382ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
47482025-02-25T17:19:13.383ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
47492025-02-25T17:19:14.384ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
4750 [8830] got start action
4751 [8830] I am going to restart
4752 Starting downstairs at port 8830
4753 Make output file at "/tmp/dsc/downstairs-8830.txt"
4754 [8830][2] reports Starting
4755 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1213)
4756 [8830][2] reports Running
4757 ----------------------------------------------------------------
4758 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4759 States: Active Active Offline
4760 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4761 2208 Acked Flush 0 Done Done Sent false
4762 2209 Acked Write 7 Done Done Sent false
4763 2210 Acked Read 5 Done Done Sent false
4764 2211 Acked Read 4 Done Done Sent false
4765 2212 Acked Write 3 Done Done Sent false
4766 2213 Acked Write 9 Done Done Sent false
4767 2214 Acked Read 3 Done Done Sent false
4768 2215 Acked Write 4 Done Done Sent false
4769 2216 Acked Write 4 Done Done Sent false
4770 2217 Acked Read 3 Done Done Sent false
4771 2218 Acked Write 5 Done Done Sent false
4772 2219 Acked Flush 0 Done Done Sent false
4773 2220 Acked Read 6 Done Done Sent false
4774 2221 Acked Read 7 Done Done Sent false
4775 2222 Acked Read 8 Done Done Sent false
4776 2223 Acked Write 2 Done Done Sent false
4777 2224 Acked Write 2 Done Done Sent false
4778 2225 Acked Read 3 Done Done Sent false
4779 2226 Acked Write 7 Done Done Sent false
4780 2227 Acked Write 2 Done Done Sent false
4781 2228 Acked Read 10 Done Done Sent false
4782 2229 Acked Flush 0 Done Done Sent false
4783 2230 Acked Write 6 Done Done Sent false
4784 2231 Acked Write 1 Done Done Sent false
4785 2232 Acked Read 6 Done Done Sent false
4786 2233 Acked Read 3 Done Done Sent false
4787 2234 Acked Write 4 Done Done Sent false
4788 2235 Acked Read 4 Done Done Sent false
4789 2236 Acked Write 8 Done Done Sent false
4790 2237 Acked Read 4 Done Done Sent false
4791 2238 Acked Read 7 Done Done Sent false
4792 2239 Acked Read 9 Done Done Sent false
4793 2240 Acked Read 7 Done Done Sent false
4794 2241 Acked Write 9 Done Done Sent false
4795 2242 Acked Read 1 Done Done Sent false
4796 2243 Acked Read 10 Done Done Sent false
4797 2244 Acked Read 2 Done Done Sent false
4798 2245 Acked Write 9 Done Done Sent false
4799 2246 Acked Flush 0 Done Done Sent false
4800 2247 Acked Write 9 Done Done Sent false
4801 2248 Acked Read 7 Done Done Sent false
4802 2249 Acked Write 6 Done Done Sent false
4803 2250 Acked Write 5 Done Done Sent false
4804 2251 Acked Read 8 Done Done Sent false
4805 2252 Acked Read 8 Done Done Sent false
4806 2253 Acked Write 7 Done Done Sent false
4807 2254 Acked Read 2 Done Done Sent false
4808 2255 Acked Flush 0 Done Done Sent false
4809 2256 Acked Read 5 Done Done Sent false
4810 2257 Acked Write 1 Done Done Sent false
4811 2258 Acked Write 3 Done Done Sent false
4812 2259 Acked Read 4 Done Done Sent false
4813 2260 Acked Write 6 Done Done Sent false
4814 2261 Acked Read 4 Done Done Sent false
4815 2262 Acked Read 10 Done Done Sent false
4816 2263 Acked Read 8 Done Done Sent false
4817 2264 Acked Read 5 Done Done Sent false
4818 2265 Acked Write 5 Done Done Sent false
4819 2266 Acked Read 9 Done Done Sent false
4820 2267 Acked Write 6 Done Done Sent false
4821 2268 Acked Read 7 Done Done Sent false
4822 2269 Acked Write 3 Done Done Sent false
4823 2270 Acked Read 3 Done Done Sent false
4824 2271 Acked Read 1 Done Done Sent false
4825 2272 Acked Write 7 Done Done Sent false
4826 2273 Acked Flush 0 Done Done Sent false
4827 2274 Acked Write 6 Done Done Sent false
4828 2275 Acked Read 10 Done Done Sent false
4829 2276 Acked Write 7 Done Done Sent false
4830 2277 Acked Read 4 Done Done Sent false
4831 2278 Acked Write 8 Done Done Sent false
4832 2279 Acked Write 5 Done Done Sent false
4833 2280 Acked Read 9 Done Done Sent false
4834 2281 Acked Write 6 Done Done Sent false
4835 2282 Acked Write 2 Done Done Sent false
4836 2283 Acked Write 7 Done Done Sent false
4837 2284 Acked Write 8 Done Done Sent false
4838 2285 Acked Read 4 Done Done Sent false
4839 2286 Acked Read 7 Done Done Sent false
4840 2287 Acked Read 10 Done Done Sent false
4841 2288 Acked Read 9 Done Done Sent false
4842 2289 Acked Read 10 Done Done Sent false
4843 2290 Acked Flush 0 Done Done Sent false
4844 2291 Acked Flush 0 Done Done Sent false
4845 2292 Acked Write 8 Done Done Sent false
4846 2293 Acked Read 1 Done Done Sent false
4847 2294 Acked Write 5 Done Done Sent false
4848 2295 Acked Read 6 Done Done Sent false
4849 2296 Acked Read 2 Done Done Sent false
4850 2297 Acked Flush 0 Done Done Sent false
4851 2298 Acked Read 2 Done Done Sent false
4852 2299 Acked Read 7 Done Done Sent false
4853 2300 Acked Flush 0 Done Done Sent false
4854 2301 Acked Read 1 Done Done Sent false
4855 2302 Acked Read 6 Done Done Sent false
4856 2303 Acked Read 6 Done Done Sent false
4857 2304 Acked Write 5 Done Done Sent false
4858 2305 Acked Write 4 Done Done Sent false
4859 2306 Acked Read 9 Done Done Sent false
4860 2307 Acked Read 1 Done Done Sent false
4861 2308 Acked Read 8 Done Done Sent false
4862 2309 Acked Read 3 Done Done Sent false
4863 2310 Acked Write 10 Done Done Sent false
4864 2311 Acked Read 7 Done Done Sent false
4865 2312 Acked Read 1 Done Done Sent false
4866 2313 Acked Read 7 Done Done Sent false
4867 2314 Acked Write 7 Done Done Sent false
4868 2315 Acked Write 6 Done Done Sent false
4869 2316 Acked Read 7 Done Done Sent false
4870 2317 Acked Write 9 Done Done Sent false
4871 2318 Acked Read 1 Done Done Sent false
4872 2319 Acked Read 4 Done Done Sent false
4873 2320 Acked Read 4 Done Done Sent false
4874 2321 Acked Read 10 Done Done Sent false
4875 2322 Acked Read 6 Done Done Sent false
4876 2323 Acked Write 5 Done Done Sent false
4877 2324 Acked Read 5 Done Done Sent false
4878 2325 Acked Flush 0 Done Done Sent false
4879 2326 Acked Read 4 Done Done Sent false
4880 2327 Acked Flush 0 Done Done Sent false
4881 2328 Acked Read 5 Done Done Sent false
4882 2329 Acked Read 3 Done Done Sent false
4883 2330 Acked Read 6 Done Done Sent false
4884 2331 Acked Write 2 Done Done Sent false
4885 2332 Acked Read 2 Done Done Sent false
4886 2333 Acked Read 5 Done Done Sent false
4887 2334 Acked Write 9 Done Done Sent false
4888 2335 Acked Write 9 Done Done Sent false
4889 2336 Acked Flush 0 Done Done Sent false
4890 2337 Acked Read 3 Done Done Sent false
4891 2338 Acked Read 8 Done Done Sent false
4892 2339 Acked Write 7 Done Done Sent false
4893 2340 Acked Read 3 Done Done Sent false
4894 2341 Acked Write 9 Done Done Sent false
4895 2342 Acked Read 8 Done Done Sent false
4896 2343 Acked Write 5 Done Done Sent false
4897 2344 Acked Write 1 Done Done Sent false
4898 2345 Acked Read 10 Done Done Sent false
4899 2346 Acked Write 8 Done Done Sent false
4900 2347 Acked Write 8 Done Done Sent false
4901 2348 Acked Read 2 Done Done Sent false
4902 2349 Acked Write 5 Done Done Sent false
4903 2350 Acked Flush 0 Done Done Sent false
4904 2351 Acked Read 10 Done Done Sent false
4905 2352 Acked Read 10 Done Done Sent false
4906 2353 Acked Read 10 Done Done Sent false
4907 2354 Acked Write 3 Done Done Sent false
4908 2355 Acked Read 7 Done Done Sent false
4909 2356 Acked Read 7 Done Done Sent false
4910 2357 Acked Read 4 Done Done Sent false
4911 2358 Acked Read 5 Done Done Sent false
4912 2359 Acked Read 6 Done Done Sent false
4913 2360 Acked Write 7 Done Done Sent false
4914 2361 Acked Read 7 Done Done Sent false
4915 2362 Acked Write 2 Done Done Sent false
4916 2363 Acked Read 9 Done Done Sent false
4917 2364 Acked Read 5 Done Done Sent false
4918 2365 Acked Read 3 Done Done Sent false
4919 2366 Acked Write 10 Done Done Sent false
4920 2367 Acked Flush 0 Done Done Sent false
4921 2368 Acked Read 9 Done Done Sent false
4922 2369 Acked Write 3 Done Done Sent false
4923 2370 Acked Write 3 Done Done Sent false
4924 2371 Acked Read 2 Done Done Sent false
4925 2372 Acked Read 2 Done Done Sent false
4926 2373 Acked Read 7 Done Done Sent false
4927 2374 Acked Write 7 Done Done Sent false
4928 2375 Acked Read 1 Done Done Sent false
4929 2376 Acked Read 1 Done Done Sent false
4930 2377 Acked Write 2 Done Done Sent false
4931 2378 Acked Read 9 Done Done Sent false
4932 2379 Acked Write 7 Done Done Sent false
4933 2380 Acked Write 6 Done Done Sent false
4934 2381 Acked Write 2 Done Done Sent false
4935 2382 Acked Read 2 Done Done Sent false
4936 2383 Acked Read 9 Done Done Sent false
4937 2384 Acked Flush 0 Done Done Sent false
4938 2385 Acked Read 1 Done Done Sent false
4939 2386 Acked Read 5 Done Done Sent false
4940 2387 Acked Read 6 Done Done Sent false
4941 2388 Acked Write 10 Done Done Sent false
4942 2389 Acked Read 8 Done Done Sent false
4943 2390 Acked Read 5 Done Done Sent false
4944 2391 Acked Write 5 Done Done Sent false
4945 2392 Acked Write 1 Done Done Sent false
4946 2393 Acked Read 5 Done Done Sent false
4947 2394 Acked Write 10 Done Done Sent false
4948 2395 Acked Write 2 Done Done Sent false
4949 2396 Acked Write 9 Done Done Sent false
4950 2397 Acked Read 3 Done Done Sent false
4951 2398 Acked Read 7 Done Done Sent false
4952 2399 Acked Read 10 Done Done Sent false
4953 2400 Acked Write 1 Done Done Sent false
4954 2401 Acked Write 4 Done Done Sent false
4955 2402 Acked Write 8 Done Done Sent false
4956 2403 Acked Write 2 Done Done Sent false
4957 2404 Acked Write 3 Done Done Sent false
4958 2405 Acked Read 9 Done Done Sent false
4959 2406 Acked Read 2 Done Done Sent false
4960 2407 Acked Write 3 Done Done Sent false
4961 2408 Acked Read 8 Done Done Sent false
4962 2409 Acked Write 2 Done Done Sent false
4963 2410 Acked Read 5 Done Done Sent false
4964 2411 Acked Write 9 Done Done Sent false
4965 2412 Acked Read 7 Done Done Sent false
4966 2413 Acked Write 5 Done Done Sent false
4967 2414 Acked Write 5 Done Done Sent false
4968 2415 Acked Read 7 Done Done Sent false
4969 2416 Acked Read 6 Done Done Sent false
4970 2417 Acked Write 7 Done Done Sent false
4971 2418 Acked Write 5 Done Done Sent false
4972 2419 Acked Read 1 Done Done Sent false
4973 2420 Acked Flush 0 Done Done Sent false
4974 2421 Acked Read 6 Done Done Sent false
4975 2422 Acked Write 10 Done Done Sent false
4976 2423 Acked Flush 0 Done Done Sent false
4977 2424 Acked Read 7 Done Done Sent false
4978 2425 Acked Write 4 Done Done Sent false
4979 2426 Acked Read 6 Done Done Sent false
4980 2427 Acked Write 7 Done Done Sent false
4981 2428 Acked Read 6 Done Done Sent false
4982 2429 Acked Read 8 Done Done Sent false
4983 2430 Acked Write 8 Done Done Sent false
4984 2431 Acked Flush 0 Done Done Sent false
4985 2432 Acked Write 7 Done Done Sent false
4986 2433 Acked Write 6 Done Done Sent false
4987 2434 Acked Read 5 Done Done Sent false
4988 2435 Acked Write 7 Done Done Sent false
4989 2436 Acked Read 5 Done Done Sent false
4990 2437 Acked Read 10 Done Done Sent false
4991 2438 Acked Read 5 Done Done Sent false
4992 2439 Acked Write 1 Done Done Sent false
4993 2440 Acked Read 8 Done Done Sent false
4994 2441 Acked Flush 0 Done Done Sent false
4995 2442 Acked Write 5 Done Done Sent false
4996 2443 Acked Flush 0 Done Done Sent false
4997 2444 Acked Read 4 Done Done Sent false
4998 2445 Acked Read 3 Done Done Sent false
4999 2446 Acked Read 1 Done Done Sent false
5000 2447 Acked Read 3 Done Done Sent false
5001 2448 Acked Read 6 Done Done Sent false
5002 2449 Acked Read 8 Done Done Sent false
5003 2450 Acked Flush 0 Done Done Sent false
5004 2451 Acked Read 4 Done Done Sent false
5005 2452 Acked Write 4 Done Done Sent false
5006 2453 Acked Write 5 Done Done Sent false
5007 2454 Acked Read 5 Done Done Sent false
5008 2455 Acked Read 8 Done Done Sent false
5009 2456 Acked Read 2 Done Done Sent false
5010 2457 Acked Read 9 Done Done Sent false
5011 2458 Acked Write 2 Done Done Sent false
5012 2459 Acked Flush 0 Done Done Sent false
5013 2460 Acked Write 1 Done Done Sent false
5014 2461 Acked Read 4 Done Done Sent false
5015 2462 Acked Write 5 Done Done Sent false
5016 2463 Acked Write 7 Done Done Sent false
5017 2464 Acked Read 6 Done Done Sent false
5018 2465 Acked Flush 0 Done Done Sent false
5019 2466 Acked Flush 0 Done Done Sent false
5020 2467 Acked Read 8 Done Done Sent false
5021 2468 Acked Read 4 Done Done Sent false
5022 2469 Acked Read 4 Done Done Sent false
5023 2470 Acked Read 9 Done Done Sent false
5024 2471 Acked Write 5 Done Done Sent false
5025 2472 Acked Write 7 Done Done Sent false
5026 2473 Acked Read 1 Done Done Sent false
5027 2474 Acked Read 8 Done Done Sent false
5028 2475 Acked Flush 0 Done Done Sent false
5029 2476 Acked Write 2 Done Done Sent false
5030 2477 Acked Write 10 Done Done Sent false
5031 2478 Acked Read 1 Done Done Sent false
5032 2479 Acked Read 9 Done Done Sent false
5033 2480 Acked Write 7 Done Done Sent false
5034 2481 Acked Read 2 Done Done Sent false
5035 2482 Acked Read 5 Done Done Sent false
5036 2483 Acked Read 8 Done Done Sent false
5037 2484 Acked Read 7 Done Done Sent false
5038 2485 Acked Read 1 Done Done Sent false
5039 2486 Acked Write 4 Done Done Sent false
5040 2487 Acked Read 7 Done Done Sent false
5041 2488 Acked Read 6 Done Done Sent false
5042 2489 Acked Read 8 Done Done Sent false
5043 2490 Acked Read 9 Done Done Sent false
5044 2491 Acked Read 1 Done Done Sent false
5045 2492 Acked Read 9 Done Done Sent false
5046 2493 Acked Read 7 Done Done Sent false
5047 2494 Acked Write 8 Done Done Sent false
5048 2495 Acked Read 3 Done Done Sent false
5049 2496 Acked Read 10 Done Done Sent false
5050 2497 Acked Read 6 Done Done Sent false
5051 2498 Acked Flush 0 Done Done Sent false
5052 2499 Acked Flush 0 Done Done Sent false
5053 2500 Acked Read 4 Done Done Sent false
5054 2501 Acked Write 3 Done Done Sent false
5055 2502 Acked Write 6 Done Done Sent false
5056 2503 Acked Read 6 Done Done Sent false
5057 2504 Acked Write 2 Done Done Sent false
5058 2505 Acked Write 9 Done Done Sent false
5059 2506 Acked Read 9 Done Done Sent false
5060 2507 Acked Read 3 Done Done Sent false
5061 2508 Acked Read 6 Done Done Sent false
5062 STATES DS:0 DS:1 DS:2 TOTAL
5063 Sent 0 0 301 301
5064 Done 301 301 0 602
5065 Skipped 0 0 0 0
5066 Error 0 0 0 0
5067 Last Flush: Some(JobId(2499)) Some(JobId(2499)) Some(JobId(2207))
5068 Downstairs last five completed: 2207 2206 2205 2204 2203
5069 Upstairs last five completed: 2508 2507 2506 2505 2504
5070 CLIENT: Up:0 ds:301 act:2
50712025-02-25T17:19:15.659ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
50722025-02-25T17:19:16.388ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
50732025-02-25T17:19:16.891ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
50742025-02-25T17:19:16.891ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43040) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
50752025-02-25T17:19:16.892ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
50762025-02-25T17:19:16.893ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
50772025-02-25T17:19:16.893ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
50782025-02-25T17:19:16.893ZINFOcrucible: send last flush ID to this DS: Some(JobId(2207)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
50792025-02-25T17:19:16.893ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
50802025-02-25T17:19:16.893ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(2207)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
50812025-02-25T17:19:16.893ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(2207)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
5082 ----------------------------------------------------------------
5083 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5084 Downstairs last five completed: 2509 2508 2507 2506 2505
5085 Upstairs last five completed: 2509 2508 2507 2506 2505
5086 CLIENT: Up:0 ds:0 act:3
5087 Replay: All jobs finished
5088 Main task has work to do, go find it
5089 got dsc Stop(2)
5090 stop 2
5091 Feb 25 17:19:19.161 INFO request completed, latency_us: 422, response_code: 204, uri: /stop/cid/2, method[8830] Got stop action so:false kr:false
5092 : POST, req_id: 83c8b958-1ebc-4960-a281-21abdee13e55, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
50932025-02-25T17:19:19.161ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
50942025-02-25T17:19:19.161ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
50952025-02-25T17:19:19.161ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
5096 Feb 25 17:19:19.161 INFO {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T17:19:19.161919851Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
5097 request completed, latency_us: 194, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6b6ad398-86c4-4ac2-a553-42f332be7961, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
5098 [8830] Exited with: ExitStatus(unix_wait_status(9))
5099 [8830][2] reports Exit
5100 Feb 25 17:19:23.164 INFO request completed, latency_us: 188, response_code: 200, uri: /state/cid/2, method: GET, req_id: 744d53e9-a2a6-4e41-9be2-5b09a76a1e1d, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
5101 001/300 Write block 869 len 16384 data: 2 2 2 2
5102 002/300 Flush
5103 003/300 Read block 2911 len 8192
5104 004/300 Read block 4576 len 40960
5105 005/300 Read block 4593 len 32768
5106 006/300 Read block 1083 len 40960
5107 007/300 Read block 4444 len 36864
5108 008/300 Write block 1549 len 20480 data: 2 2 2 2 3
5109 009/300 Read block 1038 len 20480
5110 010/300 Read block 1756 len 28672
5111 011/300 Write block 642 len 8192 data: 2 2
5112 012/300 Read block 4059 len 8192
5113 013/300 Read block 922 len 12288
5114 014/300 Read block 3139 len 4096
5115 015/300 Read block 1997 len 16384
5116 016/300 Write block 2737 len 40960 data: 4 4 4 4 2 2 3 3 3 3
5117 017/300 Write block 4308 len 8192 data: 2 2
5118 018/300 Write block 3549 len 32768 data: 3 3 3 3 3 3 2 2
5119 019/300 Write block 3945 len 28672 data: 6 4 4 4 3 3 2
5120 020/300 Read block 4991 len 32768
5121 021/300 Write block 183 len 24576 data: 4 4 4 3 2 2
5122 022/300 Flush
5123 023/300 Read block 2219 len 36864
5124 024/300 Write block 4061 len 28672 data: 3 3 3 3 3 2 2
5125 025/300 Flush
5126 026/300 Read block 1943 len 20480
5127 027/300 Read block 2191 len 20480
5128 028/300 Read block 1185 len 20480
5129 029/300 Read block 4566 len 20480
5130 030/300 Read block 3635 len 8192
5131 031/300 Read block 4447 len 40960
5132 032/300 Write block 4595 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5133 033/300 Flush
5134 034/300 Read block 4284 len 36864
5135 035/300 Write block 3713 len 24576 data: 2 3 3 3 4 4
5136 036/300 Read block 1685 len 20480
5137 037/300 Read block 2426 len 28672
5138 038/300 Write block 109 len 40960 data: 4 4 3 3 3 3 3 3 3 2
5139 039/300 Flush
5140 040/300 Write block 748 len 4096 data: 3
5141 041/300 Write block 3399 len 8192 data: 3 4
5142 042/300 Read block 4907 len 28672
5143 043/300 Write block 80 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5144 044/300 Write block 1592 len 8192 data: 4 4
5145 045/300 Flush
5146 046/300 Read block 526 len 4096
5147 047/300 Read block 803 len 36864
5148 048/300 Write block 3893 len 36864 data: 2 2 2 3 3 3 4 3 2
5149 049/300 Write block 193 len 36864 data: 3 3 3 3 3 3 2 2 2
5150 050/300 Read block 3737 len 8192
5151 051/300 Read block 861 len 16384
5152 052/300 Read block 2538 len 12288
5153 053/300 Write block 157 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5154 054/300 Write block 3476 len 28672 data: 2 2 2 2 2 2 2
5155 055/300 Flush
5156 056/300 Write block 2653 len 12288 data: 2 2 2
5157 057/300 Read block 674 len 32768
5158 058/300 Write block 3142 len 12288 data: 3 3 3
5159 059/300 Write block 3106 len 24576 data: 2 2 2 2 3 3
5160 060/300 Read block 987 len 4096
5161 061/300 Write block 4193 len 28672 data: 2 2 3 3 3 3 3
5162 062/300 Read block 3281 len 16384
5163 063/300 Write block 4180 len 24576 data: 3 2 2 2 2 2
5164 064/300 Write block 2454 len 16384 data: 4 4 4 5
5165 065/300 Read block 2779 len 4096
5166 066/300 Read block 1850 len 8192
5167 067/300 Write block 3559 len 32768 data: 2 2 2 2 2 2 2 2
5168 068/300 Write block 3121 len 12288 data: 3 3 3
5169 069/300 Flush
5170 070/300 Write block 3501 len 32768 data: 2 2 2 2 2 2 2 3
5171 071/300 Write block 3662 len 40960 data: 3 3 3 4 3 3 3 4 4 3
5172 072/300 Flush
5173 073/300 Read block 4443 len 24576
5174 074/300 Write block 2430 len 40960 data: 2 2 2 3 3 3 3 3 3 4
5175 075/300 Read block 399 len 36864
5176 076/300 Write block 1426 len 24576 data: 2 3 3 3 4 4
5177 077/300 Read block 2332 len 4096
5178 078/300 Write block 646 len 4096 data: 2
5179 079/300 Write block 641 len 16384 data: 2 3 3 2
5180 080/300 Read block 1297 len 4096
5181 081/300 Read block 4196 len 36864
5182 082/300 Flush
5183 083/300 Read block 3428 len 24576
5184 084/300 Read block 4470 len 12288
5185 085/300 Read block 3210 len 20480
5186 086/300 Read block 3910 len 36864
5187 087/300 Write block 4595 len 4096 data: 3
5188 088/300 Read block 3366 len 28672
5189 089/300 Flush
5190 090/300 Read block 2767 len 12288
5191 091/300 Read block 4125 len 36864
5192 092/300 Read block 315 len 28672
5193 093/300 Write block 3035 len 28672 data: 2 2 2 2 2 2 2
5194 094/300 Write block 1573 len 36864 data: 2 2 2 2 2 2 2 2 3
5195 095/300 Read block 367 len 20480
5196 096/300 Write block 747 len 36864 data: 3 4 4 4 4 4 3 3 3
5197 097/300 Write block 43 len 20480 data: 3 3 3 3 2
5198 098/300 Read block 1958 len 36864
5199 099/300 Read block 394 len 20480
5200 100/300 Read block 136 len 4096
5201 101/300 Write block 4808 len 8192 data: 2 3
5202 102/300 Write block 4642 len 12288 data: 2 2 2
5203 103/300 Write block 2468 len 24576 data: 2 2 2 2 2 2
5204 104/300 Flush
5205 105/300 Read block 4157 len 4096
5206 106/300 Read block 3039 len 12288
5207 107/300 Write block 3201 len 16384 data: 3 3 4 4
5208 108/300 Flush
5209 109/300 Read block 662 len 4096
5210 110/300 Write block 3550 len 28672 data: 4 4 4 4 4 3 3
5211 111/300 Read block 662 len 32768
5212 112/300 Read block 1191 len 36864
5213 113/300 Flush
5214 114/300 Write block 3812 len 20480 data: 3 3 4 5 5
5215 115/300 Flush
5216 116/300 Write block 3060 len 12288 data: 3 3 3
5217 117/300 Read block 4056 len 36864
5218 118/300 Write block 2743 len 40960 data: 4 4 4 4 3 3 3 3 3 2
5219 119/300 Read block 832 len 20480
5220 120/300 Write block 705 len 16384 data: 2 2 2 2
5221 121/300 Flush
5222 122/300 Read block 409 len 20480
5223 123/300 Read block 3382 len 32768
5224 124/300 Read block 4040 len 40960
5225 125/300 Write block 3681 len 16384 data: 2 3 4 4
5226 126/300 Write block 2857 len 8192 data: 5 6
5227 127/300 Write block 4766 len 12288 data: 5 5 6
5228 128/300 Read block 1284 len 32768
5229 129/300 Flush
5230 130/300 Flush
5231 131/300 Read block 3816 len 8192
5232 132/300 Read block 2098 len 40960
5233 133/300 Write block 2625 len 40960 data: 4 4 3 3 3 3 3 3 3 2
5234 134/300 Write block 3853 len 16384 data: 3 4 4 4
5235 135/300 Read block 1819 len 12288
5236 136/300 Read block 2081 len 24576
5237 137/300 Write block 513 len 16384 data: 2 2 2 2
5238 138/300 Read block 2570 len 16384
5239 139/300 Read block 140 len 28672
5240 140/300 Write block 567 len 36864 data: 2 2 2 2 2 2 2 2 2
5241 141/300 Write block 4167 len 16384 data: 2 2 2 3
5242 142/300 Flush
5243 143/300 Read block 1554 len 4096
5244 144/300 Write block 1338 len 12288 data: 4 4 4
5245 145/300 Write block 3481 len 20480 data: 3 3 2 3 3
5246 146/300 Read block 1959 len 24576
5247 147/300 Flush
5248 148/300 Read block 3807 len 32768
5249 149/300 Flush
5250 150/300 Write block 288 len 24576 data: 2 2 3 3 2 2
5251 151/300 Write block 4591 len 24576 data: 3 3 2 2 4 3
5252 152/300 Read block 3732 len 40960
5253 153/300 Write block 1923 len 16384 data: 2 2 2 2
5254 154/300 Read block 2885 len 8192
5255 155/300 Write block 4201 len 20480 data: 3 3 3 3 3
5256 156/300 Read block 3772 len 16384
5257 157/300 Flush
5258 158/300 Write block 3821 len 24576 data: 4 4 4 2 2 2
5259 159/300 Write block 2080 len 28672 data: 2 2 3 3 3 3 3
5260 160/300 Read block 4203 len 24576
5261 161/300 Write block 1606 len 32768 data: 3 3 3 3 3 3 3 3
5262 162/300 Write block 4729 len 32768 data: 2 2 2 2 2 2 2 2
5263 163/300 Read block 721 len 8192
5264 164/300 Write block 3952 len 24576 data: 2 2 2 2 2 2
5265 165/300 Write block 711 len 8192 data: 3 3
5266 166/300 Read block 4658 len 32768
5267 167/300 Write block 3283 len 4096 data: 3
5268 168/300 Write block 4366 len 28672 data: 2 2 2 2 2 2 2
5269 169/300 Write block 2117 len 28672 data: 2 2 2 2 3 3 3
5270 170/300 Read block 4007 len 28672
5271 171/300 Read block 1745 len 36864
5272 172/300 Read block 2642 len 8192
5273 173/300 Write block 4700 len 28672 data: 3 4 4 4 4 3 3
5274 174/300 Read block 1347 len 28672
5275 175/300 Read block 4620 len 24576
5276 176/300 Write block 1395 len 24576 data: 2 2 2 2 3 3
5277 177/300 Read block 3267 len 36864
5278 178/300 Write block 1732 len 40960 data: 2 2 3 3 3 3 2 2 2 2
5279 179/300 Read block 4029 len 40960
5280 180/300 Read block 2989 len 36864
5281 181/300 Write block 597 len 40960 data: 3 3 2 2 2 2 2 2 2 2
5282 182/300 Write block 187 len 12288 data: 3 3 2
5283 183/300 Write block 2992 len 4096 data: 2
5284 184/300 Write block 2037 len 36864 data: 2 2 2 2 2 2 2 2 2
5285 185/300 Read block 2708 len 36864
5286 186/300 Write block 3041 len 40960 data: 3 3 3 3 3 3 3 3 3 3
5287 187/300 Flush
5288 188/300 Write block 3730 len 40960 data: 3 3 3 3 4 3 3 3 3 3
5289 189/300 Write block 349 len 16384 data: 2 2 2 2
5290 190/300 Write block 2030 len 24576 data: 2 2 2 2 2 2
5291 191/300 Flush
5292 192/300 Write block 3296 len 24576 data: 3 3 2 3 3 4
5293 193/300 Read block 4425 len 36864
5294 194/300 Write block 2411 len 20480 data: 2 2 2 2 2
5295 195/300 Read block 1860 len 12288
5296 196/300 Write block 3392 len 16384 data: 2 2 2 2
5297 197/300 Write block 3352 len 20480 data: 4 4 4 4 4
5298 198/300 Write block 3240 len 16384 data: 2 2 2 2
5299 199/300 Flush
5300 200/300 Flush
5301 201/300 Write block 4391 len 24576 data: 3 3 3 3 2 2
5302 202/300 Flush
5303 203/300 Write block 4515 len 4096 data: 2
5304 204/300 Flush
5305 205/300 Write block 3304 len 4096 data: 3
5306 206/300 Read block 2405 len 28672
5307 207/300 Write block 2903 len 24576 data: 4 4 4 4 4 3
5308 208/300 Write block 4253 len 4096 data: 2
5309 209/300 Write block 3910 len 36864 data: 3 4 4 4 3 3 3 2 3
5310 210/300 Read block 4486 len 24576
5311 211/300 Flush
5312 212/300 Read block 3778 len 8192
5313 213/300 Read block 3246 len 36864
5314 214/300 Write block 4761 len 24576 data: 3 4 4 4 4 6
5315 215/300 Read block 2777 len 32768
5316 216/300 Write block 4697 len 20480 data: 3 3 3 4 5
5317 217/300 Write block 732 len 32768 data: 3 3 3 2 2 2 2 2
5318 218/300 Write block 3460 len 4096 data: 2
5319 219/300 Write block 3620 len 8192 data: 3 2
5320 220/300 Read block 1519 len 12288
5321 221/300 Write block 4429 len 8192 data: 2 2
5322 222/300 Write block 4264 len 24576 data: 4 4 4 4 4 4
5323 223/300 Write block 4812 len 40960 data: 4 4 4 3 3 2 2 2 2 2
5324 224/300 Read block 3442 len 28672
5325 225/300 Read block 3616 len 24576
5326 226/300 Read block 3296 len 24576
5327 227/300 Write block 2021 len 4096 data: 2
5328 228/300 Read block 1969 len 20480
5329 229/300 Read block 2123 len 24576
5330 230/300 Read block 1034 len 16384
5331 231/300 Write block 2050 len 8192 data: 2 2
5332 232/300 Write block 3821 len 4096 data: 5
5333 233/300 Read block 2115 len 32768
5334 234/300 Write block 4303 len 36864 data: 3 3 3 2 2 3 3 3 3
5335 235/300 Write block 198 len 40960 data: 4 3 3 3 3 3 3 3 3 3
5336 236/300 Read block 2542 len 32768
5337 237/300 Flush
5338 238/300 Write block 834 len 24576 data: 4 4 4 4 3 3
5339 239/300 Read block 3284 len 4096
5340 240/300 Read block 605 len 12288
5341 241/300 Read block 3887 len 4096
5342 242/300 Write block 1807 len 36864 data: 2 3 3 2 2 2 3 3 3
5343 243/300 Write block 162 len 12288 data: 3 3 3
5344 244/300 Write block 3724 len 36864 data: 3 4 4 3 4 4 4 4 4
5345 245/300 Write block 1771 len 16384 data: 2 2 2 2
5346 246/300 Read block 2596 len 28672
5347 247/300 Read block 2814 len 24576
5348 248/300 Flush
5349 249/300 Read block 3006 len 4096
5350 250/300 Read block 1486 len 24576
5351 251/300 Write block 728 len 4096 data: 2
5352 252/300 Write block 2807 len 24576 data: 3 3 3 3 3 2
5353 253/300 Read block 4908 len 12288
5354 254/300 Write block 3819 len 4096 data: 4
5355 255/300 Write block 2031 len 36864 data: 3 3 3 3 3 2 3 3 3
5356 256/300 Flush
5357 257/300 Write block 3948 len 28672 data: 5 4 4 3 3 3 3
5358 258/300 Read block 2126 len 4096
5359 259/300 Read block 30 len 20480
5360 260/300 Write block 453 len 20480 data: 2 2 2 2 2
5361 261/300 Write block 1095 len 24576 data: 3 3 3 3 3 3
5362 262/300 Read block 615 len 12288
5363 263/300 Flush
5364 264/300 Write block 590 len 32768 data: 2 3 3 3 3 3 3 4
5365 265/300 Read block 1240 len 4096
5366 266/300 Read block 959 len 16384
5367 267/300 Write block 416 len 28672 data: 2 2 3 4 4 3 5
5368 268/300 Read block 2360 len 16384
5369 269/300 Flush
5370 270/300 Read block 2783 len 8192
5371 271/300 Write block 1088 len 40960 data: 2 2 2 3 4 3 3 4 4 4
5372 272/300 Flush
5373 273/300 Write block 519 len 24576 data: 3 3 3 3 2 2
5374 274/300 Read block 1176 len 32768
5375 275/300 Read block 457 len 32768
5376 276/300 Read block 4081 len 28672
5377 277/300 Read block 991 len 12288
5378 278/300 Write block 1527 len 16384 data: 4 4 3 4
5379 279/300 Write block 1719 len 28672 data: 6 5 4 3 2 2 2
5380 280/300 Write block 3547 len 32768 data: 3 3 4 5 5 5 5 5
5381 281/300 Write block 2138 len 28672 data: 3 3 3 3 3 3 3
5382 282/300 Write block 3685 len 16384 data: 4 4 4 4
5383 283/300 Read block 1342 len 20480
5384 284/300 Flush
5385 285/300 Write block 2875 len 28672 data: 3 2 3 2 2 2 3
5386 286/300 Write block 3704 len 40960 data: 3 3 2 2 2 3 3 2 2 3
5387 287/300 Write block 3568 len 4096 data: 2
5388 288/300 Write block 4686 len 4096 data: 2
5389 289/300 Read block 2103 len 16384
5390 290/300 Write block 2214 len 24576 data: 4 3 4 5 5 5
5391 291/300 Write block 3226 len 8192 data: 3 3
5392 292/300 Write block 13 len 24576 data: 2 2 2 3 2 3
5393 293/300 Read block 494 len 36864
5394 294/300 Read block 847 len 24576
5395 295/300 Read block 1551 len 16384
5396 296/300 Read block 219 len 36864
5397 297/300 Write block 4545 len 36864 data: 2 2 2 2 2 2 2 2 2
5398 298/300 Write block 832 len 16384 data: 4 4 5 5
5399 299/300 Write block 3592 len 36864 data: 2 2 2 2 2 2 2 2 2
5400 300/300 Read block 587 len 4096
5401 301/300 Read block 4631 len 4096
5402 Main task has work to do, go find it
5403 got dsc Start(2)
5404 start 2
5405 Feb 25 17:19:23.402 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5f9b8af2-1981-4ee4-aba0-0437a85f115f, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
5406 Replay: started 2, returned:Ok(())
5407 ----------------------------------------------------------------
5408 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5409 States: Active Active Offline
5410 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5411 2510 Acked Write 4 Done Done Sent false
5412 2511 Acked Flush 0 Done Done Sent false
5413 2512 Acked Read 2 Done Done Sent false
5414 2513 Acked Read 10 Done Done Sent false
5415 2514 Acked Read 8 Done Done Sent false
5416 2515 Acked Read 10 Done Done Sent false
5417 2516 Acked Read 9 Done Done Sent false
5418 2517 Acked Write 5 Done Done Sent false
5419 2518 Acked Read 5 Done Done Sent false
5420 2519 Acked Read 7 Done Done Sent false
5421 2520 Acked Write 2 Done Done Sent false
5422 2521 Acked Read 2 Done Done Sent false
5423 2522 Acked Read 3 Done Done Sent false
5424 2523 Acked Read 1 Done Done Sent false
5425 2524 Acked Read 4 Done Done Sent false
5426 2525 Acked Write 10 Done Done Sent false
5427 2526 Acked Write 2 Done Done Sent false
5428 2527 Acked Write 8 Done Done Sent false
5429 2528 Acked Write 7 Done Done Sent false
5430 2529 Acked Read 8 Done Done Sent false
5431 2530 Acked Write 6 Done Done Sent false
5432 2531 Acked Flush 0 Done Done Sent false
5433 2532 Acked Read 9 Done Done Sent false
5434 2533 Acked Write 7 Done Done Sent false
5435 2534 Acked Flush 0 Done Done Sent false
5436 2535 Acked Read 5 Done Done Sent false
5437 2536 Acked Read 5 Done Done Sent false
5438 2537 Acked Read 5 Done Done Sent false
5439 2538 Acked Read 5 Done Done Sent false
5440 2539 Acked Read 2 Done Done Sent false
5441 2540 Acked Read 10 Done Done Sent false
5442 2541 Acked Write 10 Done Done Sent false
5443 2542 Acked Flush 0 Done Done Sent false
5444 2543 Acked Read 9 Done Done Sent false
5445 2544 Acked Write 6 Done Done Sent false
5446 2545 Acked Read 5 Done Done Sent false
5447 2546 Acked Read 7 Done Done Sent false
5448 2547 Acked Write 10 Done Done Sent false
5449 2548 Acked Flush 0 Done Done Sent false
5450 2549 Acked Write 1 Done Done Sent false
5451 2550 Acked Write 2 Done Done Sent false
5452 2551 Acked Read 7 Done Done Sent false
5453 2552 Acked Write 10 Done Done Sent false
5454 2553 Acked Write 2 Done Done Sent false
5455 2554 Acked Flush 0 Done Done Sent false
5456 2555 Acked Read 1 Done Done Sent false
5457 2556 Acked Read 9 Done Done Sent false
5458 2557 Acked Write 9 Done Done Sent false
5459 2558 Acked Write 9 Done Done Sent false
5460 2559 Acked Read 2 Done Done Sent false
5461 2560 Acked Read 4 Done Done Sent false
5462 2561 Acked Read 3 Done Done Sent false
5463 2562 Acked Write 10 Done Done Sent false
5464 2563 Acked Write 7 Done Done Sent false
5465 2564 Acked Flush 0 Done Done Sent false
5466 2565 Acked Write 3 Done Done Sent false
5467 2566 Acked Read 8 Done Done Sent false
5468 2567 Acked Write 3 Done Done Sent false
5469 2568 Acked Write 6 Done Done Sent false
5470 2569 Acked Read 1 Done Done Sent false
5471 2570 Acked Write 7 Done Done Sent false
5472 2571 Acked Read 4 Done Done Sent false
5473 2572 Acked Write 6 Done Done Sent false
5474 2573 Acked Write 4 Done Done Sent false
5475 2574 Acked Read 1 Done Done Sent false
5476 2575 Acked Read 2 Done Done Sent false
5477 2576 Acked Write 8 Done Done Sent false
5478 2577 Acked Write 3 Done Done Sent false
5479 2578 Acked Flush 0 Done Done Sent false
5480 2579 Acked Write 8 Done Done Sent false
5481 2580 Acked Write 10 Done Done Sent false
5482 2581 Acked Flush 0 Done Done Sent false
5483 2582 Acked Read 6 Done Done Sent false
5484 2583 Acked Write 10 Done Done Sent false
5485 2584 Acked Read 9 Done Done Sent false
5486 2585 Acked Write 6 Done Done Sent false
5487 2586 Acked Read 1 Done Done Sent false
5488 2587 Acked Write 1 Done Done Sent false
5489 2588 Acked Write 4 Done Done Sent false
5490 2589 Acked Read 1 Done Done Sent false
5491 2590 Acked Read 9 Done Done Sent false
5492 2591 Acked Flush 0 Done Done Sent false
5493 2592 Acked Read 6 Done Done Sent false
5494 2593 Acked Read 3 Done Done Sent false
5495 2594 Acked Read 5 Done Done Sent false
5496 2595 Acked Read 9 Done Done Sent false
5497 2596 Acked Write 1 Done Done Sent false
5498 2597 Acked Read 7 Done Done Sent false
5499 2598 Acked Flush 0 Done Done Sent false
5500 2599 Acked Read 3 Done Done Sent false
5501 2600 Acked Read 9 Done Done Sent false
5502 2601 Acked Read 7 Done Done Sent false
5503 2602 Acked Write 7 Done Done Sent false
5504 2603 Acked Write 9 Done Done Sent false
5505 2604 Acked Read 5 Done Done Sent false
5506 2605 Acked Write 9 Done Done Sent false
5507 2606 Acked Write 5 Done Done Sent false
5508 2607 Acked Read 9 Done Done Sent false
5509 2608 Acked Read 5 Done Done Sent false
5510 2609 Acked Read 1 Done Done Sent false
5511 2610 Acked Write 2 Done Done Sent false
5512 2611 Acked Write 3 Done Done Sent false
5513 2612 Acked Write 6 Done Done Sent false
5514 2613 Acked Flush 0 Done Done Sent false
5515 2614 Acked Read 1 Done Done Sent false
5516 2615 Acked Read 3 Done Done Sent false
5517 2616 Acked Write 4 Done Done Sent false
5518 2617 Acked Flush 0 Done Done Sent false
5519 2618 Acked Read 1 Done Done Sent false
5520 2619 Acked Write 7 Done Done Sent false
5521 2620 Acked Read 8 Done Done Sent false
5522 2621 Acked Read 9 Done Done Sent false
5523 2622 Acked Flush 0 Done Done Sent false
5524 2623 Acked Write 5 Done Done Sent false
5525 2624 Acked Flush 0 Done Done Sent false
5526 2625 Acked Write 3 Done Done Sent false
5527 2626 Acked Read 9 Done Done Sent false
5528 2627 Acked Write 10 Done Done Sent false
5529 2628 Acked Read 5 Done Done Sent false
5530 2629 Acked Write 4 Done Done Sent false
5531 2630 Acked Flush 0 Done Done Sent false
5532 2631 Acked Read 5 Done Done Sent false
5533 2632 Acked Read 8 Done Done Sent false
5534 2633 Acked Read 10 Done Done Sent false
5535 2634 Acked Write 4 Done Done Sent false
5536 2635 Acked Write 2 Done Done Sent false
5537 2636 Acked Write 3 Done Done Sent false
5538 2637 Acked Read 8 Done Done Sent false
5539 2638 Acked Flush 0 Done Done Sent false
5540 2639 Acked Flush 0 Done Done Sent false
5541 2640 Acked Read 2 Done Done Sent false
5542 2641 Acked Read 10 Done Done Sent false
5543 2642 Acked Write 10 Done Done Sent false
5544 2643 Acked Write 4 Done Done Sent false
5545 2644 Acked Read 3 Done Done Sent false
5546 2645 Acked Read 6 Done Done Sent false
5547 2646 Acked Write 4 Done Done Sent false
5548 2647 Acked Read 4 Done Done Sent false
5549 2648 Acked Read 7 Done Done Sent false
5550 2649 Acked Write 9 Done Done Sent false
5551 2650 Acked Write 4 Done Done Sent false
5552 2651 Acked Flush 0 Done Done Sent false
5553 2652 Acked Read 1 Done Done Sent false
5554 2653 Acked Write 3 Done Done Sent false
5555 2654 Acked Write 5 Done Done Sent false
5556 2655 Acked Read 6 Done Done Sent false
5557 2656 Acked Flush 0 Done Done Sent false
5558 2657 Acked Read 8 Done Done Sent false
5559 2658 Acked Flush 0 Done Done Sent false
5560 2659 Acked Write 6 Done Done Sent false
5561 2660 Acked Write 6 Done Done Sent false
5562 2661 Acked Read 10 Done Done Sent false
5563 2662 Acked Write 4 Done Done Sent false
5564 2663 Acked Read 2 Done Done Sent false
5565 2664 Acked Write 5 Done Done Sent false
5566 2665 Acked Read 4 Done Done Sent false
5567 2666 Acked Flush 0 Done Done Sent false
5568 2667 Acked Write 6 Done Done Sent false
5569 2668 Acked Write 7 Done Done Sent false
5570 2669 Acked Read 6 Done Done Sent false
5571 2670 Acked Write 8 Done Done Sent false
5572 2671 Acked Write 8 Done Done Sent false
5573 2672 Acked Read 2 Done Done Sent false
5574 2673 Acked Write 6 Done Done Sent false
5575 2674 Acked Write 2 Done Done Sent false
5576 2675 Acked Read 8 Done Done Sent false
5577 2676 Acked Write 1 Done Done Sent false
5578 2677 Acked Write 7 Done Done Sent false
5579 2678 Acked Write 7 Done Done Sent false
5580 2679 Acked Read 7 Done Done Sent false
5581 2680 Acked Read 9 Done Done Sent false
5582 2681 Acked Read 2 Done Done Sent false
5583 2682 Acked Write 7 Done Done Sent false
5584 2683 Acked Read 7 Done Done Sent false
5585 2684 Acked Read 6 Done Done Sent false
5586 2685 Acked Write 6 Done Done Sent false
5587 2686 Acked Read 9 Done Done Sent false
5588 2687 Acked Write 10 Done Done Sent false
5589 2688 Acked Read 10 Done Done Sent false
5590 2689 Acked Read 9 Done Done Sent false
5591 2690 Acked Write 10 Done Done Sent false
5592 2691 Acked Write 3 Done Done Sent false
5593 2692 Acked Write 1 Done Done Sent false
5594 2693 Acked Write 9 Done Done Sent false
5595 2694 Acked Read 9 Done Done Sent false
5596 2695 Acked Write 10 Done Done Sent false
5597 2696 Acked Flush 0 Done Done Sent false
5598 2697 Acked Write 10 Done Done Sent false
5599 2698 Acked Write 4 Done Done Sent false
5600 2699 Acked Write 6 Done Done Sent false
5601 2700 Acked Flush 0 Done Done Sent false
5602 2701 Acked Write 6 Done Done Sent false
5603 2702 Acked Read 9 Done Done Sent false
5604 2703 Acked Write 5 Done Done Sent false
5605 2704 Acked Read 3 Done Done Sent false
5606 2705 Acked Write 4 Done Done Sent false
5607 2706 Acked Write 5 Done Done Sent false
5608 2707 Acked Write 4 Done Done Sent false
5609 2708 Acked Flush 0 Done Done Sent false
5610 2709 Acked Flush 0 Done Done Sent false
5611 2710 Acked Write 6 Done Done Sent false
5612 2711 Acked Flush 0 Done Done Sent false
5613 2712 Acked Write 1 Done Done Sent false
5614 2713 Acked Flush 0 Done Done Sent false
5615 2714 Acked Write 1 Done Done Sent false
5616 2715 Acked Read 7 Done Done Sent false
5617 2716 Acked Write 6 Done Done Sent false
5618 2717 Acked Write 1 Done Done Sent false
5619 2718 Acked Write 9 Done Done Sent false
5620 2719 Acked Read 6 Done Done Sent false
5621 2720 Acked Flush 0 Done Done Sent false
5622 2721 Acked Read 2 Done Done Sent false
5623 2722 Acked Read 9 Done Done Sent false
5624 2723 Acked Write 6 Done Done Sent false
5625 2724 Acked Read 8 Done Done Sent false
5626 2725 Acked Write 5 Done Done Sent false
5627 2726 Acked Write 8 Done Done Sent false
5628 2727 Acked Write 1 Done Done Sent false
5629 2728 Acked Write 2 Done Done Sent false
5630 2729 Acked Read 3 Done Done Sent false
5631 2730 Acked Write 2 Done Done Sent false
5632 2731 Acked Write 6 Done Done Sent false
5633 2732 Acked Write 10 Done Done Sent false
5634 2733 Acked Read 7 Done Done Sent false
5635 2734 Acked Read 6 Done Done Sent false
5636 2735 Acked Read 6 Done Done Sent false
5637 2736 Acked Write 1 Done Done Sent false
5638 2737 Acked Read 5 Done Done Sent false
5639 2738 Acked Read 6 Done Done Sent false
5640 2739 Acked Read 4 Done Done Sent false
5641 2740 Acked Write 2 Done Done Sent false
5642 2741 Acked Write 1 Done Done Sent false
5643 2742 Acked Read 8 Done Done Sent false
5644 2743 Acked Write 9 Done Done Sent false
5645 2744 Acked Write 10 Done Done Sent false
5646 2745 Acked Read 8 Done Done Sent false
5647 2746 Acked Flush 0 Done Done Sent false
5648 2747 Acked Write 6 Done Done Sent false
5649 2748 Acked Read 1 Done Done Sent false
5650 2749 Acked Read 3 Done Done Sent false
5651 2750 Acked Read 1 Done Done Sent false
5652 2751 Acked Write 9 Done Done Sent false
5653 2752 Acked Write 3 Done Done Sent false
5654 2753 Acked Write 9 Done Done Sent false
5655 2754 Acked Write 4 Done Done Sent false
5656 2755 Acked Read 7 Done Done Sent false
5657 2756 Acked Read 6 Done Done Sent false
5658 2757 Acked Flush 0 Done Done Sent false
5659 2758 Acked Read 1 Done Done Sent false
5660 2759 Acked Read 6 Done Done Sent false
5661 2760 Acked Write 1 Done Done Sent false
5662 2761 Acked Write 6 Done Done Sent false
5663 2762 Acked Read 3 Done Done Sent false
5664 2763 Acked Write 1 Done Done Sent false
5665 2764 Acked Write 9 Done Done Sent false
5666 2765 Acked Flush 0 Done Done Sent false
5667 2766 Acked Write 7 Done Done Sent false
5668 2767 Acked Read 1 Done Done Sent false
5669 2768 Acked Read 5 Done Done Sent false
5670 2769 Acked Write 5 Done Done Sent false
5671 2770 Acked Write 6 Done Done Sent false
5672 2771 Acked Read 3 Done Done Sent false
5673 2772 Acked Flush 0 Done Done Sent false
5674 2773 Acked Write 8 Done Done Sent false
5675 2774 Acked Read 1 Done Done Sent false
5676 2775 Acked Read 4 Done Done Sent false
5677 2776 Acked Write 7 Done Done Sent false
5678 2777 Acked Read 4 Done Done Sent false
5679 2778 Acked Flush 0 Done Done Sent false
5680 2779 Acked Read 2 Done Done Sent false
5681 2780 Acked Write 10 Done Done Sent false
5682 2781 Acked Flush 0 Done Done Sent false
5683 2782 Acked Write 6 Done Done Sent false
5684 2783 Acked Read 8 Done Done Sent false
5685 2784 Acked Read 8 Done Done Sent false
5686 2785 Acked Read 7 Done Done Sent false
5687 2786 Acked Read 3 Done Done Sent false
5688 2787 Acked Write 4 Done Done Sent false
5689 2788 Acked Write 7 Done Done Sent false
5690 2789 Acked Write 8 Done Done Sent false
5691 2790 Acked Write 7 Done Done Sent false
5692 2791 Acked Write 4 Done Done Sent false
5693 2792 Acked Read 5 Done Done Sent false
5694 2793 Acked Flush 0 Done Done Sent false
5695 2794 Acked Write 7 Done Done Sent false
5696 2795 Acked Write 10 Done Done Sent false
5697 2796 Acked Write 1 Done Done Sent false
5698 2797 Acked Write 1 Done Done Sent false
5699 2798 Acked Read 4 Done Done Sent false
5700 2799 Acked Write 6 Done Done Sent false
5701 2800 Acked Write 2 Done Done Sent false
5702 2801 Acked Write 6 Done Done Sent false
5703 2802 Acked Read 9 Done Done Sent false
5704 2803 Acked Read 6 Done Done Sent false
5705 2804 Acked Read 4 Done Done Sent false
5706 2805 Acked Read 9 Done Done Sent false
5707 2806 Acked Write 9 Done Done Sent false
5708 2807 Acked Write 4 Done Done Sent false
5709 2808 Acked Write 9 Done Done Sent false
5710 2809 Acked Read 1 Done Done Sent false
5711 2810 Acked Read 1 Done Done Sent false
5712 STATES DS:0 DS:1 DS:2 TOTAL
5713 Sent 0 0 301 301
5714 Done 301 301 0 602
5715 Skipped 0 0 0 0
5716 Error 0 0 0 0
5717 Last Flush: Some(JobId(2793)) Some(JobId(2793)) Some(JobId(2509))
5718 Downstairs last five completed: 2509 2508 2507 2506 2505
5719 Upstairs last five completed: 2810 2809 2808 2807 2806
5720 CLIENT: Up:0 ds:301 act:2
57212025-02-25T17:19:24.399ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
57222025-02-25T17:19:25.401ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
57232025-02-25T17:19:26.403ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
5724 [8830] got start action
5725 [8830] I am going to restart
5726 Starting downstairs at port 8830
5727 Make output file at "/tmp/dsc/downstairs-8830.txt"
5728 [8830][2] reports Starting
5729 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1214)
5730 [8830][2] reports Running
5731 ----------------------------------------------------------------
5732 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5733 States: Active Active Offline
5734 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5735 2510 Acked Write 4 Done Done Sent false
5736 2511 Acked Flush 0 Done Done Sent false
5737 2512 Acked Read 2 Done Done Sent false
5738 2513 Acked Read 10 Done Done Sent false
5739 2514 Acked Read 8 Done Done Sent false
5740 2515 Acked Read 10 Done Done Sent false
5741 2516 Acked Read 9 Done Done Sent false
5742 2517 Acked Write 5 Done Done Sent false
5743 2518 Acked Read 5 Done Done Sent false
5744 2519 Acked Read 7 Done Done Sent false
5745 2520 Acked Write 2 Done Done Sent false
5746 2521 Acked Read 2 Done Done Sent false
5747 2522 Acked Read 3 Done Done Sent false
5748 2523 Acked Read 1 Done Done Sent false
5749 2524 Acked Read 4 Done Done Sent false
5750 2525 Acked Write 10 Done Done Sent false
5751 2526 Acked Write 2 Done Done Sent false
5752 2527 Acked Write 8 Done Done Sent false
5753 2528 Acked Write 7 Done Done Sent false
5754 2529 Acked Read 8 Done Done Sent false
5755 2530 Acked Write 6 Done Done Sent false
5756 2531 Acked Flush 0 Done Done Sent false
5757 2532 Acked Read 9 Done Done Sent false
5758 2533 Acked Write 7 Done Done Sent false
5759 2534 Acked Flush 0 Done Done Sent false
5760 2535 Acked Read 5 Done Done Sent false
5761 2536 Acked Read 5 Done Done Sent false
5762 2537 Acked Read 5 Done Done Sent false
5763 2538 Acked Read 5 Done Done Sent false
5764 2539 Acked Read 2 Done Done Sent false
5765 2540 Acked Read 10 Done Done Sent false
5766 2541 Acked Write 10 Done Done Sent false
5767 2542 Acked Flush 0 Done Done Sent false
5768 2543 Acked Read 9 Done Done Sent false
5769 2544 Acked Write 6 Done Done Sent false
5770 2545 Acked Read 5 Done Done Sent false
5771 2546 Acked Read 7 Done Done Sent false
5772 2547 Acked Write 10 Done Done Sent false
5773 2548 Acked Flush 0 Done Done Sent false
5774 2549 Acked Write 1 Done Done Sent false
5775 2550 Acked Write 2 Done Done Sent false
5776 2551 Acked Read 7 Done Done Sent false
5777 2552 Acked Write 10 Done Done Sent false
5778 2553 Acked Write 2 Done Done Sent false
5779 2554 Acked Flush 0 Done Done Sent false
5780 2555 Acked Read 1 Done Done Sent false
5781 2556 Acked Read 9 Done Done Sent false
5782 2557 Acked Write 9 Done Done Sent false
5783 2558 Acked Write 9 Done Done Sent false
5784 2559 Acked Read 2 Done Done Sent false
5785 2560 Acked Read 4 Done Done Sent false
5786 2561 Acked Read 3 Done Done Sent false
5787 2562 Acked Write 10 Done Done Sent false
5788 2563 Acked Write 7 Done Done Sent false
5789 2564 Acked Flush 0 Done Done Sent false
5790 2565 Acked Write 3 Done Done Sent false
5791 2566 Acked Read 8 Done Done Sent false
5792 2567 Acked Write 3 Done Done Sent false
5793 2568 Acked Write 6 Done Done Sent false
5794 2569 Acked Read 1 Done Done Sent false
5795 2570 Acked Write 7 Done Done Sent false
5796 2571 Acked Read 4 Done Done Sent false
5797 2572 Acked Write 6 Done Done Sent false
5798 2573 Acked Write 4 Done Done Sent false
5799 2574 Acked Read 1 Done Done Sent false
5800 2575 Acked Read 2 Done Done Sent false
5801 2576 Acked Write 8 Done Done Sent false
5802 2577 Acked Write 3 Done Done Sent false
5803 2578 Acked Flush 0 Done Done Sent false
5804 2579 Acked Write 8 Done Done Sent false
5805 2580 Acked Write 10 Done Done Sent false
5806 2581 Acked Flush 0 Done Done Sent false
5807 2582 Acked Read 6 Done Done Sent false
5808 2583 Acked Write 10 Done Done Sent false
5809 2584 Acked Read 9 Done Done Sent false
5810 2585 Acked Write 6 Done Done Sent false
5811 2586 Acked Read 1 Done Done Sent false
5812 2587 Acked Write 1 Done Done Sent false
5813 2588 Acked Write 4 Done Done Sent false
5814 2589 Acked Read 1 Done Done Sent false
5815 2590 Acked Read 9 Done Done Sent false
5816 2591 Acked Flush 0 Done Done Sent false
5817 2592 Acked Read 6 Done Done Sent false
5818 2593 Acked Read 3 Done Done Sent false
5819 2594 Acked Read 5 Done Done Sent false
5820 2595 Acked Read 9 Done Done Sent false
5821 2596 Acked Write 1 Done Done Sent false
5822 2597 Acked Read 7 Done Done Sent false
5823 2598 Acked Flush 0 Done Done Sent false
5824 2599 Acked Read 3 Done Done Sent false
5825 2600 Acked Read 9 Done Done Sent false
5826 2601 Acked Read 7 Done Done Sent false
5827 2602 Acked Write 7 Done Done Sent false
5828 2603 Acked Write 9 Done Done Sent false
5829 2604 Acked Read 5 Done Done Sent false
5830 2605 Acked Write 9 Done Done Sent false
5831 2606 Acked Write 5 Done Done Sent false
5832 2607 Acked Read 9 Done Done Sent false
5833 2608 Acked Read 5 Done Done Sent false
5834 2609 Acked Read 1 Done Done Sent false
5835 2610 Acked Write 2 Done Done Sent false
5836 2611 Acked Write 3 Done Done Sent false
5837 2612 Acked Write 6 Done Done Sent false
5838 2613 Acked Flush 0 Done Done Sent false
5839 2614 Acked Read 1 Done Done Sent false
5840 2615 Acked Read 3 Done Done Sent false
5841 2616 Acked Write 4 Done Done Sent false
5842 2617 Acked Flush 0 Done Done Sent false
5843 2618 Acked Read 1 Done Done Sent false
5844 2619 Acked Write 7 Done Done Sent false
5845 2620 Acked Read 8 Done Done Sent false
5846 2621 Acked Read 9 Done Done Sent false
5847 2622 Acked Flush 0 Done Done Sent false
5848 2623 Acked Write 5 Done Done Sent false
5849 2624 Acked Flush 0 Done Done Sent false
5850 2625 Acked Write 3 Done Done Sent false
5851 2626 Acked Read 9 Done Done Sent false
5852 2627 Acked Write 10 Done Done Sent false
5853 2628 Acked Read 5 Done Done Sent false
5854 2629 Acked Write 4 Done Done Sent false
5855 2630 Acked Flush 0 Done Done Sent false
5856 2631 Acked Read 5 Done Done Sent false
5857 2632 Acked Read 8 Done Done Sent false
5858 2633 Acked Read 10 Done Done Sent false
5859 2634 Acked Write 4 Done Done Sent false
5860 2635 Acked Write 2 Done Done Sent false
5861 2636 Acked Write 3 Done Done Sent false
5862 2637 Acked Read 8 Done Done Sent false
5863 2638 Acked Flush 0 Done Done Sent false
5864 2639 Acked Flush 0 Done Done Sent false
5865 2640 Acked Read 2 Done Done Sent false
5866 2641 Acked Read 10 Done Done Sent false
5867 2642 Acked Write 10 Done Done Sent false
5868 2643 Acked Write 4 Done Done Sent false
5869 2644 Acked Read 3 Done Done Sent false
5870 2645 Acked Read 6 Done Done Sent false
5871 2646 Acked Write 4 Done Done Sent false
5872 2647 Acked Read 4 Done Done Sent false
5873 2648 Acked Read 7 Done Done Sent false
5874 2649 Acked Write 9 Done Done Sent false
5875 2650 Acked Write 4 Done Done Sent false
5876 2651 Acked Flush 0 Done Done Sent false
5877 2652 Acked Read 1 Done Done Sent false
5878 2653 Acked Write 3 Done Done Sent false
5879 2654 Acked Write 5 Done Done Sent false
5880 2655 Acked Read 6 Done Done Sent false
5881 2656 Acked Flush 0 Done Done Sent false
5882 2657 Acked Read 8 Done Done Sent false
5883 2658 Acked Flush 0 Done Done Sent false
5884 2659 Acked Write 6 Done Done Sent false
5885 2660 Acked Write 6 Done Done Sent false
5886 2661 Acked Read 10 Done Done Sent false
5887 2662 Acked Write 4 Done Done Sent false
5888 2663 Acked Read 2 Done Done Sent false
5889 2664 Acked Write 5 Done Done Sent false
5890 2665 Acked Read 4 Done Done Sent false
5891 2666 Acked Flush 0 Done Done Sent false
5892 2667 Acked Write 6 Done Done Sent false
5893 2668 Acked Write 7 Done Done Sent false
5894 2669 Acked Read 6 Done Done Sent false
5895 2670 Acked Write 8 Done Done Sent false
5896 2671 Acked Write 8 Done Done Sent false
5897 2672 Acked Read 2 Done Done Sent false
5898 2673 Acked Write 6 Done Done Sent false
5899 2674 Acked Write 2 Done Done Sent false
5900 2675 Acked Read 8 Done Done Sent false
5901 2676 Acked Write 1 Done Done Sent false
5902 2677 Acked Write 7 Done Done Sent false
5903 2678 Acked Write 7 Done Done Sent false
5904 2679 Acked Read 7 Done Done Sent false
5905 2680 Acked Read 9 Done Done Sent false
5906 2681 Acked Read 2 Done Done Sent false
5907 2682 Acked Write 7 Done Done Sent false
5908 2683 Acked Read 7 Done Done Sent false
5909 2684 Acked Read 6 Done Done Sent false
5910 2685 Acked Write 6 Done Done Sent false
5911 2686 Acked Read 9 Done Done Sent false
5912 2687 Acked Write 10 Done Done Sent false
5913 2688 Acked Read 10 Done Done Sent false
5914 2689 Acked Read 9 Done Done Sent false
5915 2690 Acked Write 10 Done Done Sent false
5916 2691 Acked Write 3 Done Done Sent false
5917 2692 Acked Write 1 Done Done Sent false
5918 2693 Acked Write 9 Done Done Sent false
5919 2694 Acked Read 9 Done Done Sent false
5920 2695 Acked Write 10 Done Done Sent false
5921 2696 Acked Flush 0 Done Done Sent false
5922 2697 Acked Write 10 Done Done Sent false
5923 2698 Acked Write 4 Done Done Sent false
5924 2699 Acked Write 6 Done Done Sent false
5925 2700 Acked Flush 0 Done Done Sent false
5926 2701 Acked Write 6 Done Done Sent false
5927 2702 Acked Read 9 Done Done Sent false
5928 2703 Acked Write 5 Done Done Sent false
5929 2704 Acked Read 3 Done Done Sent false
5930 2705 Acked Write 4 Done Done Sent false
5931 2706 Acked Write 5 Done Done Sent false
5932 2707 Acked Write 4 Done Done Sent false
5933 2708 Acked Flush 0 Done Done Sent false
5934 2709 Acked Flush 0 Done Done Sent false
5935 2710 Acked Write 6 Done Done Sent false
5936 2711 Acked Flush 0 Done Done Sent false
5937 2712 Acked Write 1 Done Done Sent false
5938 2713 Acked Flush 0 Done Done Sent false
5939 2714 Acked Write 1 Done Done Sent false
5940 2715 Acked Read 7 Done Done Sent false
5941 2716 Acked Write 6 Done Done Sent false
5942 2717 Acked Write 1 Done Done Sent false
5943 2718 Acked Write 9 Done Done Sent false
5944 2719 Acked Read 6 Done Done Sent false
5945 2720 Acked Flush 0 Done Done Sent false
5946 2721 Acked Read 2 Done Done Sent false
5947 2722 Acked Read 9 Done Done Sent false
5948 2723 Acked Write 6 Done Done Sent false
5949 2724 Acked Read 8 Done Done Sent false
5950 2725 Acked Write 5 Done Done Sent false
5951 2726 Acked Write 8 Done Done Sent false
5952 2727 Acked Write 1 Done Done Sent false
5953 2728 Acked Write 2 Done Done Sent false
5954 2729 Acked Read 3 Done Done Sent false
5955 2730 Acked Write 2 Done Done Sent false
5956 2731 Acked Write 6 Done Done Sent false
5957 2732 Acked Write 10 Done Done Sent false
5958 2733 Acked Read 7 Done Done Sent false
5959 2734 Acked Read 6 Done Done Sent false
5960 2735 Acked Read 6 Done Done Sent false
5961 2736 Acked Write 1 Done Done Sent false
5962 2737 Acked Read 5 Done Done Sent false
5963 2738 Acked Read 6 Done Done Sent false
5964 2739 Acked Read 4 Done Done Sent false
5965 2740 Acked Write 2 Done Done Sent false
5966 2741 Acked Write 1 Done Done Sent false
5967 2742 Acked Read 8 Done Done Sent false
5968 2743 Acked Write 9 Done Done Sent false
5969 2744 Acked Write 10 Done Done Sent false
5970 2745 Acked Read 8 Done Done Sent false
5971 2746 Acked Flush 0 Done Done Sent false
5972 2747 Acked Write 6 Done Done Sent false
5973 2748 Acked Read 1 Done Done Sent false
5974 2749 Acked Read 3 Done Done Sent false
5975 2750 Acked Read 1 Done Done Sent false
5976 2751 Acked Write 9 Done Done Sent false
5977 2752 Acked Write 3 Done Done Sent false
5978 2753 Acked Write 9 Done Done Sent false
5979 2754 Acked Write 4 Done Done Sent false
5980 2755 Acked Read 7 Done Done Sent false
5981 2756 Acked Read 6 Done Done Sent false
5982 2757 Acked Flush 0 Done Done Sent false
5983 2758 Acked Read 1 Done Done Sent false
5984 2759 Acked Read 6 Done Done Sent false
5985 2760 Acked Write 1 Done Done Sent false
5986 2761 Acked Write 6 Done Done Sent false
5987 2762 Acked Read 3 Done Done Sent false
5988 2763 Acked Write 1 Done Done Sent false
5989 2764 Acked Write 9 Done Done Sent false
5990 2765 Acked Flush 0 Done Done Sent false
5991 2766 Acked Write 7 Done Done Sent false
5992 2767 Acked Read 1 Done Done Sent false
5993 2768 Acked Read 5 Done Done Sent false
5994 2769 Acked Write 5 Done Done Sent false
5995 2770 Acked Write 6 Done Done Sent false
5996 2771 Acked Read 3 Done Done Sent false
5997 2772 Acked Flush 0 Done Done Sent false
5998 2773 Acked Write 8 Done Done Sent false
5999 2774 Acked Read 1 Done Done Sent false
6000 2775 Acked Read 4 Done Done Sent false
6001 2776 Acked Write 7 Done Done Sent false
6002 2777 Acked Read 4 Done Done Sent false
6003 2778 Acked Flush 0 Done Done Sent false
6004 2779 Acked Read 2 Done Done Sent false
6005 2780 Acked Write 10 Done Done Sent false
6006 2781 Acked Flush 0 Done Done Sent false
6007 2782 Acked Write 6 Done Done Sent false
6008 2783 Acked Read 8 Done Done Sent false
6009 2784 Acked Read 8 Done Done Sent false
6010 2785 Acked Read 7 Done Done Sent false
6011 2786 Acked Read 3 Done Done Sent false
6012 2787 Acked Write 4 Done Done Sent false
6013 2788 Acked Write 7 Done Done Sent false
6014 2789 Acked Write 8 Done Done Sent false
6015 2790 Acked Write 7 Done Done Sent false
6016 2791 Acked Write 4 Done Done Sent false
6017 2792 Acked Read 5 Done Done Sent false
6018 2793 Acked Flush 0 Done Done Sent false
6019 2794 Acked Write 7 Done Done Sent false
6020 2795 Acked Write 10 Done Done Sent false
6021 2796 Acked Write 1 Done Done Sent false
6022 2797 Acked Write 1 Done Done Sent false
6023 2798 Acked Read 4 Done Done Sent false
6024 2799 Acked Write 6 Done Done Sent false
6025 2800 Acked Write 2 Done Done Sent false
6026 2801 Acked Write 6 Done Done Sent false
6027 2802 Acked Read 9 Done Done Sent false
6028 2803 Acked Read 6 Done Done Sent false
6029 2804 Acked Read 4 Done Done Sent false
6030 2805 Acked Read 9 Done Done Sent false
6031 2806 Acked Write 9 Done Done Sent false
6032 2807 Acked Write 4 Done Done Sent false
6033 2808 Acked Write 9 Done Done Sent false
6034 2809 Acked Read 1 Done Done Sent false
6035 2810 Acked Read 1 Done Done Sent false
6036 STATES DS:0 DS:1 DS:2 TOTAL
6037 Sent 0 0 301 301
6038 Done 301 301 0 602
6039 Skipped 0 0 0 0
6040 Error 0 0 0 0
6041 Last Flush: Some(JobId(2793)) Some(JobId(2793)) Some(JobId(2509))
6042 Downstairs last five completed: 2509 2508 2507 2506 2505
6043 Upstairs last five completed: 2810 2809 2808 2807 2806
6044 CLIENT: Up:0 ds:301 act:2
60452025-02-25T17:19:28.406ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
60462025-02-25T17:19:29.163ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
60472025-02-25T17:19:29.164ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42555) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
60482025-02-25T17:19:29.165ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
60492025-02-25T17:19:29.165ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
60502025-02-25T17:19:29.165ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
60512025-02-25T17:19:29.165ZINFOcrucible: send last flush ID to this DS: Some(JobId(2509)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
60522025-02-25T17:19:29.165ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
60532025-02-25T17:19:29.165ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(2509)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
60542025-02-25T17:19:29.166ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(2509)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
6055 ----------------------------------------------------------------
6056 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6057 Downstairs last five completed: 2811 2810 2809 2808 2807
6058 Upstairs last five completed: 2811 2810 2809 2808 2807
6059 CLIENT: Up:0 ds:0 act:3
6060 Replay: All jobs finished
6061 Main task has work to do, go find it
6062 got dsc Stop(2)
6063 stop 2
6064 Feb 25 17:19:31.407 INFO request completed, latency_us: 380, response_code[8830] Got stop action so:false kr:false
6065 : 204, uri: /stop/cid/2, method: POST, req_id: cdb23f3b-68c0-401c-aaae-b6367283be9b, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
6066 Feb 25 17:19:31.408 INFO request completed, latency_us: 163, response_code: 200, uri: /state/cid/2, method: GET, req_id: 260de837-a738-4065-afd9-a5f37039b6c8, remote_addr: 127.0.0.1:52576, local_addr: 127{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:19:31.408086377Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
6067 .0.0.1:{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:19:31.408180204Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
6068 9998
60692025-02-25T17:19:31.408ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
60702025-02-25T17:19:31.408ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
6071 [8830] Exited with: ExitStatus(unix_wait_status(9))
6072 [8830][2] reports Exit
6073 Feb 25 17:19:35.410 INFO request completed, latency_us: 226, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4625960e-98bc-453d-b6a4-e1179d62482c, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
6074 001/300 Write block 4235 len 32768 data: 3 3 2 3 3 4 3 3
6075 002/300 Read block 2303 len 24576
6076 003/300 Write block 2177 len 12288 data: 2 2 2
6077 004/300 Read block 4867 len 8192
6078 005/300 Read block 1482 len 36864
6079 006/300 Flush
6080 007/300 Read block 303 len 12288
6081 008/300 Read block 4284 len 40960
6082 009/300 Flush
6083 010/300 Flush
6084 011/300 Read block 2050 len 8192
6085 012/300 Write block 2965 len 20480 data: 3 3 3 3 3
6086 013/300 Write block 378 len 8192 data: 3 3
6087 014/300 Read block 2756 len 4096
6088 015/300 Read block 417 len 16384
6089 016/300 Read block 4551 len 40960
6090 017/300 Read block 1661 len 4096
6091 018/300 Read block 2316 len 4096
6092 019/300 Write block 4333 len 28672 data: 2 2 2 3 4 4 4
6093 020/300 Read block 395 len 36864
6094 021/300 Read block 188 len 16384
6095 022/300 Read block 1447 len 16384
6096 023/300 Write block 2786 len 8192 data: 2 2
6097 024/300 Flush
6098 025/300 Write block 1614 len 24576 data: 2 2 2 2 2 2
6099 026/300 Flush
6100 027/300 Write block 2386 len 32768 data: 2 2 2 2 2 2 2 2
6101 028/300 Flush
6102 029/300 Read block 2025 len 36864
6103 030/300 Write block 4854 len 12288 data: 3 3 3
6104 031/300 Read block 4433 len 32768
6105 032/300 Write block 993 len 28672 data: 4 4 4 4 2 2 3
6106 033/300 Write block 1015 len 4096 data: 2
6107 034/300 Read block 1328 len 40960
6108 035/300 Read block 4423 len 12288
6109 036/300 Read block 1407 len 32768
6110 037/300 Read block 4037 len 16384
6111 038/300 Read block 1518 len 12288
6112 039/300 Write block 2092 len 40960 data: 3 3 3 3 3 2 2 2 2 2
6113 040/300 Write block 1022 len 16384 data: 3 3 3 4
6114 041/300 Flush
6115 042/300 Write block 3751 len 8192 data: 2 3
6116 043/300 Read block 3037 len 12288
6117 044/300 Write block 1875 len 8192 data: 2 2
6118 045/300 Read block 3286 len 16384
6119 046/300 Read block 2596 len 16384
6120 047/300 Write block 3638 len 36864 data: 2 2 2 2 3 3 3 3 3
6121 048/300 Write block 1106 len 4096 data: 3
6122 049/300 Write block 3302 len 32768 data: 4 4 4 3 2 2 2 2
6123 050/300 Write block 2974 len 24576 data: 2 2 2 2 2 2
6124 051/300 Write block 1908 len 32768 data: 3 3 4 3 2 2 2 2
6125 052/300 Write block 651 len 24576 data: 2 2 2 2 2 2
6126 053/300 Read block 422 len 4096
6127 054/300 Flush
6128 055/300 Read block 1352 len 24576
6129 056/300 Write block 4628 len 28672 data: 3 3 3 3 3 3 4
6130 057/300 Write block 871 len 4096 data: 3
6131 058/300 Read block 3552 len 36864
6132 059/300 Read block 3983 len 36864
6133 060/300 Read block 2132 len 8192
6134 061/300 Write block 2331 len 32768 data: 4 5 5 5 4 3 3 3
6135 062/300 Read block 4766 len 36864
6136 063/300 Flush
6137 064/300 Write block 264 len 16384 data: 2 3 4 4
6138 065/300 Read block 4058 len 20480
6139 066/300 Flush
6140 067/300 Write block 1939 len 40960 data: 4 4 3 2 2 3 4 4 4 4
6141 068/300 Write block 4489 len 20480 data: 2 2 3 3 3
6142 069/300 Read block 1863 len 4096
6143 070/300 Write block 833 len 40960 data: 5 6 6 5 5 4 4 3 2 2
6144 071/300 Read block 4168 len 40960
6145 072/300 Read block 3300 len 4096
6146 073/300 Read block 4981 len 40960
6147 074/300 Read block 2068 len 32768
6148 075/300 Read block 1597 len 20480
6149 076/300 Read block 2182 len 20480
6150 077/300 Write block 2589 len 32768 data: 2 2 2 2 2 2 2 2
6151 078/300 Flush
6152 079/300 Read block 2606 len 8192
6153 080/300 Read block 1032 len 16384
6154 081/300 Write block 1326 len 28672 data: 2 2 2 2 2 2 2
6155 082/300 Flush
6156 083/300 Write block 4355 len 16384 data: 3 3 2 2
6157 084/300 Write block 2518 len 4096 data: 4
6158 085/300 Read block 4520 len 24576
6159 086/300 Read block 3376 len 32768
6160 087/300 Read block 3454 len 20480
6161 088/300 Read block 3878 len 16384
6162 089/300 Write block 1290 len 8192 data: 3 4
6163 090/300 Write block 4457 len 32768 data: 2 2 2 2 2 2 2 2
6164 091/300 Read block 4035 len 32768
6165 092/300 Read block 51 len 20480
6166 093/300 Flush
6167 094/300 Write block 2930 len 32768 data: 2 2 2 2 2 2 2 2
6168 095/300 Read block 3831 len 4096
6169 096/300 Read block 3570 len 16384
6170 097/300 Read block 1345 len 12288
6171 098/300 Write block 86 len 8192 data: 3 3
6172 099/300 Write block 3102 len 16384 data: 2 2 2 2
6173 100/300 Write block 906 len 36864 data: 3 3 2 2 2 2 2 2 2
6174 101/300 Read block 61 len 8192
6175 102/300 Read block 3210 len 32768
6176 103/300 Read block 1136 len 40960
6177 104/300 Read block 689 len 28672
6178 105/300 Read block 4423 len 36864
6179 106/300 Read block 2948 len 28672
6180 107/300 Read block 2758 len 36864
6181 108/300 Write block 4987 len 20480 data: 2 2 3 3 3
6182 109/300 Read block 3096 len 32768
6183 110/300 Write block 3991 len 4096 data: 3
6184 111/300 Read block 3617 len 20480
6185 112/300 Read block 696 len 32768
6186 113/300 Read block 3747 len 24576
6187 114/300 Read block 258 len 24576
6188 115/300 Write block 45 len 36864 data: 4 4 3 2 2 2 2 2 2
6189 116/300 Read block 1266 len 36864
6190 117/300 Read block 745 len 36864
6191 118/300 Read block 697 len 40960
6192 119/300 Read block 4135 len 12288
6193 120/300 Read block 1178 len 28672
6194 121/300 Write block 4509 len 28672 data: 2 2 2 2 2 2 3
6195 122/300 Write block 4538 len 20480 data: 2 2 2 2 2
6196 123/300 Write block 4239 len 32768 data: 4 5 4 4 3 3 3 3
6197 124/300 Read block 3998 len 24576
6198 125/300 Write block 3427 len 20480 data: 2 2 2 2 2
6199 126/300 Read block 2491 len 28672
6200 127/300 Read block 3307 len 40960
6201 128/300 Read block 480 len 24576
6202 129/300 Read block 1146 len 28672
6203 130/300 Write block 877 len 4096 data: 3
6204 131/300 Write block 4619 len 4096 data: 3
6205 132/300 Write block 3086 len 24576 data: 3 2 2 2 2 3
6206 133/300 Write block 398 len 32768 data: 2 2 2 2 2 2 2 3
6207 134/300 Write block 415 len 4096 data: 2
6208 135/300 Write block 2797 len 16384 data: 2 2 2 2
6209 136/300 Write block 2898 len 32768 data: 2 2 3 4 5 5 5 5
6210 137/300 Write block 4094 len 8192 data: 2 2
6211 138/300 Read block 730 len 36864
6212 139/300 Read block 4050 len 36864
6213 140/300 Read block 726 len 36864
6214 141/300 Flush
6215 142/300 Write block 4960 len 36864 data: 2 3 4 4 4 4 3 3 4
6216 143/300 Read block 3369 len 4096
6217 144/300 Flush
6218 145/300 Read block 1447 len 40960
6219 146/300 Write block 4068 len 16384 data: 2 2 2 2
6220 147/300 Read block 2042 len 20480
6221 148/300 Read block 885 len 24576
6222 149/300 Write block 438 len 24576 data: 2 2 2 2 2 2
6223 150/300 Read block 718 len 16384
6224 151/300 Write block 3860 len 28672 data: 2 2 2 2 2 2 2
6225 152/300 Read block 3648 len 40960
6226 153/300 Write block 1245 len 8192 data: 3 3
6227 154/300 Write block 3758 len 40960 data: 3 3 3 2 3 3 3 2 2 2
6228 155/300 Write block 922 len 40960 data: 3 3 3 3 3 3 3 3 3 3
6229 156/300 Read block 824 len 12288
6230 157/300 Read block 4133 len 24576
6231 158/300 Read block 1804 len 28672
6232 159/300 Read block 4228 len 8192
6233 160/300 Write block 2026 len 24576 data: 2 2 2 2 3 4
6234 161/300 Read block 670 len 8192
6235 162/300 Write block 3637 len 8192 data: 3 3
6236 163/300 Read block 426 len 40960
6237 164/300 Read block 2384 len 4096
6238 165/300 Write block 4239 len 4096 data: 5
6239 166/300 Read block 1947 len 16384
6240 167/300 Write block 4463 len 24576 data: 3 3 2 2 2 2
6241 168/300 Read block 3312 len 24576
6242 169/300 Flush
6243 170/300 Read block 1959 len 12288
6244 171/300 Read block 783 len 40960
6245 172/300 Write block 2618 len 8192 data: 2 2
6246 173/300 Read block 3632 len 24576
6247 174/300 Read block 3455 len 8192
6248 175/300 Read block 2991 len 24576
6249 176/300 Read block 1633 len 24576
6250 177/300 Read block 4409 len 16384
6251 178/300 Write block 4896 len 36864 data: 2 2 2 2 2 2 2 2 2
6252 179/300 Read block 163 len 32768
6253 180/300 Write block 4445 len 16384 data: 2 2 2 2
6254 181/300 Read block 1683 len 28672
6255 182/300 Read block 2698 len 28672
6256 183/300 Write block 1382 len 28672 data: 3 3 3 4 4 4 5
6257 184/300 Write block 3606 len 36864 data: 3 3 3 3 2 2 2 2 2
6258 185/300 Read block 3908 len 12288
6259 186/300 Read block 3739 len 12288
6260 187/300 Write block 3423 len 20480 data: 2 2 2 2 3
6261 188/300 Flush
6262 189/300 Write block 4620 len 28672 data: 3 3 3 2 2 2 2
6263 190/300 Read block 4394 len 20480
6264 191/300 Write block 2690 len 4096 data: 2
6265 192/300 Read block 4330 len 36864
6266 193/300 Write block 269 len 28672 data: 3 4 4 4 4 4 4
6267 194/300 Read block 1548 len 8192
6268 195/300 Flush
6269 196/300 Read block 2535 len 36864
6270 197/300 Read block 1903 len 36864
6271 198/300 Read block 1101 len 12288
6272 199/300 Write block 4300 len 20480 data: 3 4 3 4 4
6273 200/300 Read block 3807 len 32768
6274 201/300 Read block 4390 len 4096
6275 202/300 Write block 2964 len 28672 data: 3 4 4 4 4 4 2
6276 203/300 Read block 381 len 36864
6277 204/300 Write block 2609 len 12288 data: 3 3 2
6278 205/300 Write block 4491 len 40960 data: 4 4 4 3 2 2 2 2 2 2
6279 206/300 Write block 2456 len 28672 data: 5 6 5 5 4 4 4
6280 207/300 Read block 991 len 20480
6281 208/300 Write block 1212 len 24576 data: 2 2 2 2 2 2
6282 209/300 Read block 4655 len 16384
6283 210/300 Write block 1178 len 32768 data: 4 4 4 4 4 3 3 3
6284 211/300 Read block 3869 len 20480
6285 212/300 Read block 4671 len 16384
6286 213/300 Flush
6287 214/300 Read block 2142 len 20480
6288 215/300 Read block 2346 len 20480
6289 216/300 Read block 2183 len 24576
6290 217/300 Flush
6291 218/300 Write block 4025 len 40960 data: 4 4 4 4 3 3 2 2 2 2
6292 219/300 Write block 2994 len 12288 data: 3 3 2
6293 220/300 Write block 688 len 20480 data: 2 2 2 2 2
6294 221/300 Write block 1978 len 40960 data: 4 4 3 3 3 3 2 2 2 3
6295 222/300 Read block 2966 len 20480
6296 223/300 Write block 2966 len 20480 data: 5 5 5 5 3
6297 224/300 Write block 26 len 24576 data: 2 2 2 2 2 2
6298 225/300 Read block 1064 len 28672
6299 226/300 Read block 699 len 8192
6300 227/300 Write block 3023 len 4096 data: 2
6301 228/300 Write block 4195 len 20480 data: 4 4 4 4 4
6302 229/300 Read block 3047 len 4096
6303 230/300 Write block 411 len 20480 data: 3 2 2 2 3
6304 231/300 Read block 550 len 4096
6305 232/300 Flush
6306 233/300 Read block 2715 len 28672
6307 234/300 Write block 4351 len 24576 data: 3 2 2 2 4 4
6308 235/300 Write block 4961 len 4096 data: 4
6309 236/300 Read block 1263 len 12288
6310 237/300 Write block 69 len 28672 data: 3 3 3 3 3 3 3
6311 238/300 Write block 2508 len 8192 data: 3 3
6312 239/300 Read block 1879 len 8192
6313 240/300 Write block 2528 len 16384 data: 2 2 2 2
6314 241/300 Write block 4530 len 36864 data: 3 5 4 5 4 4 3 3 3
6315 242/300 Read block 1646 len 12288
6316 243/300 Read block 4106 len 12288
6317 244/300 Write block 4277 len 4096 data: 2
6318 245/300 Flush
6319 246/300 Read block 2755 len 28672
6320 247/300 Read block 1880 len 8192
6321 248/300 Read block 938 len 36864
6322 249/300 Read block 733 len 12288
6323 250/300 Read block 2085 len 8192
6324 251/300 Read block 759 len 32768
6325 252/300 Write block 1767 len 32768 data: 3 3 3 3 3 3 3 3
6326 253/300 Write block 4183 len 16384 data: 3 3 3 2
6327 254/300 Write block 3538 len 20480 data: 3 3 4 3 3
6328 255/300 Read block 1648 len 36864
6329 256/300 Read block 2950 len 24576
6330 257/300 Read block 4220 len 40960
6331 258/300 Write block 3388 len 12288 data: 2 2 2
6332 259/300 Read block 4425 len 32768
6333 260/300 Read block 132 len 4096
6334 261/300 Write block 4325 len 28672 data: 2 2 2 2 2 2 2
6335 262/300 Write block 4102 len 20480 data: 2 2 2 2 2
6336 263/300 Read block 4462 len 12288
6337 264/300 Flush
6338 265/300 Write block 4661 len 16384 data: 4 4 4 4
6339 266/300 Read block 3179 len 20480
6340 267/300 Write block 4691 len 12288 data: 2 2 2
6341 268/300 Write block 1909 len 24576 data: 4 5 4 3 3 3
6342 269/300 Write block 2798 len 8192 data: 3 3
6343 270/300 Flush
6344 271/300 Write block 1988 len 16384 data: 3 3 2 3
6345 272/300 Read block 262 len 36864
6346 273/300 Read block 807 len 32768
6347 274/300 Read block 2531 len 12288
6348 275/300 Write block 4933 len 36864 data: 2 2 2 2 2 2 4 4 5
6349 276/300 Write block 372 len 16384 data: 2 2 2 2
6350 277/300 Read block 3456 len 40960
6351 278/300 Write block 4102 len 4096 data: 3
6352 279/300 Write block 4224 len 8192 data: 2 2
6353 280/300 Write block 1254 len 40960 data: 2 2 2 3 3 3 3 3 3 3
6354 281/300 Read block 4528 len 28672
6355 282/300 Read block 4191 len 4096
6356 283/300 Read block 573 len 4096
6357 284/300 Read block 1340 len 28672
6358 285/300 Write block 2093 len 32768 data: 4 4 4 4 3 3 3 3
6359 286/300 Read block 4004 len 28672
6360 287/300 Write block 39 len 12288 data: 2 2 2
6361 288/300 Read block 1376 len 40960
6362 289/300 Write block 2457 len 20480 data: 7 6 6 5 5
6363 290/300 Write block 2189 len 28672 data: 2 2 3 3 3 2 2
6364 291/300 Read block 4600 len 24576
6365 292/300 Write block 1197 len 32768 data: 3 3 2 2 2 2 2 2
6366 293/300 Write block 68 len 28672 data: 3 4 4 4 4 4 4
6367 294/300 Read block 3172 len 40960
6368 295/300 Read block 3651 len 12288
6369 296/300 Write block 3872 len 12288 data: 2 2 2
6370 297/300 Write block 4488 len 32768 data: 2 3 3 5 5 5 4 3
6371 298/300 Read block 3125 len 20480
6372 299/300 Write block 855 len 20480 data: 2 2 2 2 2
6373 300/300 Read block 599 len 32768
6374 301/300 Read block 1692 len 8192
6375 Main task has work to do, go find it
6376 got dsc Start(2)
6377 start 2
6378 Feb 25 17:19:35.669 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/2, method: POST, req_id: cf5f5ca6-61de-4345-85b7-d33a1a0917d1, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
6379 Replay: started 2, returned:Ok(())
6380 ----------------------------------------------------------------
6381 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6382 States: Active Active Offline
6383 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6384 2812 Acked Write 8 Done Done Sent false
6385 2813 Acked Read 6 Done Done Sent false
6386 2814 Acked Write 3 Done Done Sent false
6387 2815 Acked Read 2 Done Done Sent false
6388 2816 Acked Read 9 Done Done Sent false
6389 2817 Acked Flush 0 Done Done Sent false
6390 2818 Acked Read 3 Done Done Sent false
6391 2819 Acked Read 10 Done Done Sent false
6392 2820 Acked Flush 0 Done Done Sent false
6393 2821 Acked Flush 0 Done Done Sent false
6394 2822 Acked Read 2 Done Done Sent false
6395 2823 Acked Write 5 Done Done Sent false
6396 2824 Acked Write 2 Done Done Sent false
6397 2825 Acked Read 1 Done Done Sent false
6398 2826 Acked Read 4 Done Done Sent false
6399 2827 Acked Read 10 Done Done Sent false
6400 2828 Acked Read 1 Done Done Sent false
6401 2829 Acked Read 1 Done Done Sent false
6402 2830 Acked Write 7 Done Done Sent false
6403 2831 Acked Read 9 Done Done Sent false
6404 2832 Acked Read 4 Done Done Sent false
6405 2833 Acked Read 4 Done Done Sent false
6406 2834 Acked Write 2 Done Done Sent false
6407 2835 Acked Flush 0 Done Done Sent false
6408 2836 Acked Write 6 Done Done Sent false
6409 2837 Acked Flush 0 Done Done Sent false
6410 2838 Acked Write 8 Done Done Sent false
6411 2839 Acked Flush 0 Done Done Sent false
6412 2840 Acked Read 9 Done Done Sent false
6413 2841 Acked Write 3 Done Done Sent false
6414 2842 Acked Read 8 Done Done Sent false
6415 2843 Acked Write 7 Done Done Sent false
6416 2844 Acked Write 1 Done Done Sent false
6417 2845 Acked Read 10 Done Done Sent false
6418 2846 Acked Read 3 Done Done Sent false
6419 2847 Acked Read 8 Done Done Sent false
6420 2848 Acked Read 4 Done Done Sent false
6421 2849 Acked Read 3 Done Done Sent false
6422 2850 Acked Write 10 Done Done Sent false
6423 2851 Acked Write 4 Done Done Sent false
6424 2852 Acked Flush 0 Done Done Sent false
6425 2853 Acked Write 2 Done Done Sent false
6426 2854 Acked Read 3 Done Done Sent false
6427 2855 Acked Write 2 Done Done Sent false
6428 2856 Acked Read 4 Done Done Sent false
6429 2857 Acked Read 4 Done Done Sent false
6430 2858 Acked Write 9 Done Done Sent false
6431 2859 Acked Write 1 Done Done Sent false
6432 2860 Acked Write 8 Done Done Sent false
6433 2861 Acked Write 6 Done Done Sent false
6434 2862 Acked Write 8 Done Done Sent false
6435 2863 Acked Write 6 Done Done Sent false
6436 2864 Acked Read 1 Done Done Sent false
6437 2865 Acked Flush 0 Done Done Sent false
6438 2866 Acked Read 6 Done Done Sent false
6439 2867 Acked Write 7 Done Done Sent false
6440 2868 Acked Write 1 Done Done Sent false
6441 2869 Acked Read 9 Done Done Sent false
6442 2870 Acked Read 9 Done Done Sent false
6443 2871 Acked Read 2 Done Done Sent false
6444 2872 Acked Write 8 Done Done Sent false
6445 2873 Acked Read 9 Done Done Sent false
6446 2874 Acked Flush 0 Done Done Sent false
6447 2875 Acked Write 4 Done Done Sent false
6448 2876 Acked Read 5 Done Done Sent false
6449 2877 Acked Flush 0 Done Done Sent false
6450 2878 Acked Write 10 Done Done Sent false
6451 2879 Acked Write 5 Done Done Sent false
6452 2880 Acked Read 1 Done Done Sent false
6453 2881 Acked Write 10 Done Done Sent false
6454 2882 Acked Read 10 Done Done Sent false
6455 2883 Acked Read 1 Done Done Sent false
6456 2884 Acked Read 10 Done Done Sent false
6457 2885 Acked Read 8 Done Done Sent false
6458 2886 Acked Read 5 Done Done Sent false
6459 2887 Acked Read 5 Done Done Sent false
6460 2888 Acked Write 8 Done Done Sent false
6461 2889 Acked Flush 0 Done Done Sent false
6462 2890 Acked Read 2 Done Done Sent false
6463 2891 Acked Read 4 Done Done Sent false
6464 2892 Acked Write 7 Done Done Sent false
6465 2893 Acked Flush 0 Done Done Sent false
6466 2894 Acked Write 4 Done Done Sent false
6467 2895 Acked Write 1 Done Done Sent false
6468 2896 Acked Read 6 Done Done Sent false
6469 2897 Acked Read 8 Done Done Sent false
6470 2898 Acked Read 5 Done Done Sent false
6471 2899 Acked Read 4 Done Done Sent false
6472 2900 Acked Write 2 Done Done Sent false
6473 2901 Acked Write 8 Done Done Sent false
6474 2902 Acked Read 8 Done Done Sent false
6475 2903 Acked Read 5 Done Done Sent false
6476 2904 Acked Flush 0 Done Done Sent false
6477 2905 Acked Write 8 Done Done Sent false
6478 2906 Acked Read 1 Done Done Sent false
6479 2907 Acked Read 4 Done Done Sent false
6480 2908 Acked Read 3 Done Done Sent false
6481 2909 Acked Write 2 Done Done Sent false
6482 2910 Acked Write 4 Done Done Sent false
6483 2911 Acked Write 9 Done Done Sent false
6484 2912 Acked Read 2 Done Done Sent false
6485 2913 Acked Read 8 Done Done Sent false
6486 2914 Acked Read 10 Done Done Sent false
6487 2915 Acked Read 7 Done Done Sent false
6488 2916 Acked Read 9 Done Done Sent false
6489 2917 Acked Read 7 Done Done Sent false
6490 2918 Acked Read 9 Done Done Sent false
6491 2919 Acked Write 5 Done Done Sent false
6492 2920 Acked Read 8 Done Done Sent false
6493 2921 Acked Write 1 Done Done Sent false
6494 2922 Acked Read 5 Done Done Sent false
6495 2923 Acked Read 8 Done Done Sent false
6496 2924 Acked Read 6 Done Done Sent false
6497 2925 Acked Read 6 Done Done Sent false
6498 2926 Acked Write 9 Done Done Sent false
6499 2927 Acked Read 9 Done Done Sent false
6500 2928 Acked Read 9 Done Done Sent false
6501 2929 Acked Read 10 Done Done Sent false
6502 2930 Acked Read 3 Done Done Sent false
6503 2931 Acked Read 7 Done Done Sent false
6504 2932 Acked Write 7 Done Done Sent false
6505 2933 Acked Write 5 Done Done Sent false
6506 2934 Acked Write 8 Done Done Sent false
6507 2935 Acked Read 6 Done Done Sent false
6508 2936 Acked Write 5 Done Done Sent false
6509 2937 Acked Read 7 Done Done Sent false
6510 2938 Acked Read 10 Done Done Sent false
6511 2939 Acked Read 6 Done Done Sent false
6512 2940 Acked Read 7 Done Done Sent false
6513 2941 Acked Write 1 Done Done Sent false
6514 2942 Acked Write 1 Done Done Sent false
6515 2943 Acked Write 6 Done Done Sent false
6516 2944 Acked Write 8 Done Done Sent false
6517 2945 Acked Write 1 Done Done Sent false
6518 2946 Acked Write 4 Done Done Sent false
6519 2947 Acked Write 8 Done Done Sent false
6520 2948 Acked Write 2 Done Done Sent false
6521 2949 Acked Read 9 Done Done Sent false
6522 2950 Acked Read 9 Done Done Sent false
6523 2951 Acked Read 9 Done Done Sent false
6524 2952 Acked Flush 0 Done Done Sent false
6525 2953 Acked Write 9 Done Done Sent false
6526 2954 Acked Read 1 Done Done Sent false
6527 2955 Acked Flush 0 Done Done Sent false
6528 2956 Acked Read 10 Done Done Sent false
6529 2957 Acked Write 4 Done Done Sent false
6530 2958 Acked Read 5 Done Done Sent false
6531 2959 Acked Read 6 Done Done Sent false
6532 2960 Acked Write 6 Done Done Sent false
6533 2961 Acked Read 4 Done Done Sent false
6534 2962 Acked Write 7 Done Done Sent false
6535 2963 Acked Read 10 Done Done Sent false
6536 2964 Acked Write 2 Done Done Sent false
6537 2965 Acked Write 10 Done Done Sent false
6538 2966 Acked Write 10 Done Done Sent false
6539 2967 Acked Read 3 Done Done Sent false
6540 2968 Acked Read 6 Done Done Sent false
6541 2969 Acked Read 7 Done Done Sent false
6542 2970 Acked Read 2 Done Done Sent false
6543 2971 Acked Write 6 Done Done Sent false
6544 2972 Acked Read 2 Done Done Sent false
6545 2973 Acked Write 2 Done Done Sent false
6546 2974 Acked Read 10 Done Done Sent false
6547 2975 Acked Read 1 Done Done Sent false
6548 2976 Acked Write 1 Done Done Sent false
6549 2977 Acked Read 4 Done Done Sent false
6550 2978 Acked Write 6 Done Done Sent false
6551 2979 Acked Read 6 Done Done Sent false
6552 2980 Acked Flush 0 Done Done Sent false
6553 2981 Acked Read 3 Done Done Sent false
6554 2982 Acked Read 10 Done Done Sent false
6555 2983 Acked Write 2 Done Done Sent false
6556 2984 Acked Read 6 Done Done Sent false
6557 2985 Acked Read 2 Done Done Sent false
6558 2986 Acked Read 6 Done Done Sent false
6559 2987 Acked Read 6 Done Done Sent false
6560 2988 Acked Read 4 Done Done Sent false
6561 2989 Acked Write 9 Done Done Sent false
6562 2990 Acked Read 8 Done Done Sent false
6563 2991 Acked Write 4 Done Done Sent false
6564 2992 Acked Read 7 Done Done Sent false
6565 2993 Acked Read 7 Done Done Sent false
6566 2994 Acked Write 7 Done Done Sent false
6567 2995 Acked Write 9 Done Done Sent false
6568 2996 Acked Read 3 Done Done Sent false
6569 2997 Acked Read 3 Done Done Sent false
6570 2998 Acked Write 5 Done Done Sent false
6571 2999 Acked Flush 0 Done Done Sent false
6572 3000 Acked Write 7 Done Done Sent false
6573 3001 Acked Read 5 Done Done Sent false
6574 3002 Acked Write 1 Done Done Sent false
6575 3003 Acked Read 9 Done Done Sent false
6576 3004 Acked Write 7 Done Done Sent false
6577 3005 Acked Read 2 Done Done Sent false
6578 3006 Acked Flush 0 Done Done Sent false
6579 3007 Acked Read 9 Done Done Sent false
6580 3008 Acked Read 9 Done Done Sent false
6581 3009 Acked Read 3 Done Done Sent false
6582 3010 Acked Write 5 Done Done Sent false
6583 3011 Acked Read 8 Done Done Sent false
6584 3012 Acked Read 1 Done Done Sent false
6585 3013 Acked Write 7 Done Done Sent false
6586 3014 Acked Read 9 Done Done Sent false
6587 3015 Acked Write 3 Done Done Sent false
6588 3016 Acked Write 10 Done Done Sent false
6589 3017 Acked Write 7 Done Done Sent false
6590 3018 Acked Read 5 Done Done Sent false
6591 3019 Acked Write 6 Done Done Sent false
6592 3020 Acked Read 4 Done Done Sent false
6593 3021 Acked Write 8 Done Done Sent false
6594 3022 Acked Read 5 Done Done Sent false
6595 3023 Acked Read 4 Done Done Sent false
6596 3024 Acked Flush 0 Done Done Sent false
6597 3025 Acked Read 5 Done Done Sent false
6598 3026 Acked Read 5 Done Done Sent false
6599 3027 Acked Read 6 Done Done Sent false
6600 3028 Acked Flush 0 Done Done Sent false
6601 3029 Acked Write 10 Done Done Sent false
6602 3030 Acked Write 3 Done Done Sent false
6603 3031 Acked Write 5 Done Done Sent false
6604 3032 Acked Write 10 Done Done Sent false
6605 3033 Acked Read 5 Done Done Sent false
6606 3034 Acked Write 5 Done Done Sent false
6607 3035 Acked Write 6 Done Done Sent false
6608 3036 Acked Read 7 Done Done Sent false
6609 3037 Acked Read 2 Done Done Sent false
6610 3038 Acked Write 1 Done Done Sent false
6611 3039 Acked Write 5 Done Done Sent false
6612 3040 Acked Read 1 Done Done Sent false
6613 3041 Acked Write 5 Done Done Sent false
6614 3042 Acked Read 1 Done Done Sent false
6615 3043 Acked Flush 0 Done Done Sent false
6616 3044 Acked Read 7 Done Done Sent false
6617 3045 Acked Write 6 Done Done Sent false
6618 3046 Acked Write 1 Done Done Sent false
6619 3047 Acked Read 3 Done Done Sent false
6620 3048 Acked Write 7 Done Done Sent false
6621 3049 Acked Write 2 Done Done Sent false
6622 3050 Acked Read 2 Done Done Sent false
6623 3051 Acked Write 4 Done Done Sent false
6624 3052 Acked Write 9 Done Done Sent false
6625 3053 Acked Read 3 Done Done Sent false
6626 3054 Acked Read 3 Done Done Sent false
6627 3055 Acked Write 1 Done Done Sent false
6628 3056 Acked Flush 0 Done Done Sent false
6629 3057 Acked Read 7 Done Done Sent false
6630 3058 Acked Read 2 Done Done Sent false
6631 3059 Acked Read 9 Done Done Sent false
6632 3060 Acked Read 3 Done Done Sent false
6633 3061 Acked Read 2 Done Done Sent false
6634 3062 Acked Read 8 Done Done Sent false
6635 3063 Acked Write 8 Done Done Sent false
6636 3064 Acked Write 4 Done Done Sent false
6637 3065 Acked Write 5 Done Done Sent false
6638 3066 Acked Read 9 Done Done Sent false
6639 3067 Acked Read 6 Done Done Sent false
6640 3068 Acked Read 10 Done Done Sent false
6641 3069 Acked Write 3 Done Done Sent false
6642 3070 Acked Read 8 Done Done Sent false
6643 3071 Acked Read 1 Done Done Sent false
6644 3072 Acked Write 7 Done Done Sent false
6645 3073 Acked Write 5 Done Done Sent false
6646 3074 Acked Read 3 Done Done Sent false
6647 3075 Acked Flush 0 Done Done Sent false
6648 3076 Acked Write 4 Done Done Sent false
6649 3077 Acked Read 5 Done Done Sent false
6650 3078 Acked Write 3 Done Done Sent false
6651 3079 Acked Write 6 Done Done Sent false
6652 3080 Acked Write 2 Done Done Sent false
6653 3081 Acked Flush 0 Done Done Sent false
6654 3082 Acked Write 4 Done Done Sent false
6655 3083 Acked Read 9 Done Done Sent false
6656 3084 Acked Read 8 Done Done Sent false
6657 3085 Acked Read 3 Done Done Sent false
6658 3086 Acked Write 9 Done Done Sent false
6659 3087 Acked Write 4 Done Done Sent false
6660 3088 Acked Read 10 Done Done Sent false
6661 3089 Acked Write 1 Done Done Sent false
6662 3090 Acked Write 2 Done Done Sent false
6663 3091 Acked Write 10 Done Done Sent false
6664 3092 Acked Read 7 Done Done Sent false
6665 3093 Acked Read 1 Done Done Sent false
6666 3094 Acked Read 1 Done Done Sent false
6667 3095 Acked Read 7 Done Done Sent false
6668 3096 Acked Write 8 Done Done Sent false
6669 3097 Acked Read 7 Done Done Sent false
6670 3098 Acked Write 3 Done Done Sent false
6671 3099 Acked Read 10 Done Done Sent false
6672 3100 Acked Write 5 Done Done Sent false
6673 3101 Acked Write 7 Done Done Sent false
6674 3102 Acked Read 6 Done Done Sent false
6675 3103 Acked Write 8 Done Done Sent false
6676 3104 Acked Write 7 Done Done Sent false
6677 3105 Acked Read 10 Done Done Sent false
6678 3106 Acked Read 3 Done Done Sent false
6679 3107 Acked Write 3 Done Done Sent false
6680 3108 Acked Write 8 Done Done Sent false
6681 3109 Acked Read 5 Done Done Sent false
6682 3110 Acked Write 5 Done Done Sent false
6683 3111 Acked Read 8 Done Done Sent false
6684 3112 Acked Read 2 Done Done Sent false
6685 STATES DS:0 DS:1 DS:2 TOTAL
6686 Sent 0 0 301 301
6687 Done 301 301 0 602
6688 Skipped 0 0 0 0
6689 Error 0 0 0 0
6690 Last Flush: Some(JobId(3081)) Some(JobId(3081)) Some(JobId(2811))
6691 Downstairs last five completed: 2811 2810 2809 2808 2807
6692 Upstairs last five completed: 3112 3111 3110 3109 3108
6693 CLIENT: Up:0 ds:301 act:2
66942025-02-25T17:19:36.418ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
66952025-02-25T17:19:37.420ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
66962025-02-25T17:19:38.421ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
66972025-02-25T17:19:39.422ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
6698 ----------------------------------------------------------------
6699 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6700 States: Active Active Offline
6701 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6702 2812 Acked Write 8 Done Done Sent false
6703 2813 Acked Read 6 Done Done Sent false
6704 2814 Acked Write 3 Done Done Sent false
6705 2815 Acked Read 2 Done Done Sent false
6706 2816 Acked Read 9 Done Done Sent false
6707 2817 Acked Flush 0 Done Done Sent false
6708 2818 Acked Read 3 Done Done Sent false
6709 2819 Acked Read 10 Done Done Sent false
6710 2820 Acked Flush 0 Done Done Sent false
6711 2821 Acked Flush 0 Done Done Sent false
6712 2822 Acked Read 2 Done Done Sent false
6713 2823 Acked Write 5 Done Done Sent false
6714 2824 Acked Write 2 Done Done Sent false
6715 2825 Acked Read 1 Done Done Sent false
6716 2826 Acked Read 4 Done Done Sent false
6717 2827 Acked Read 10 Done Done Sent false
6718 2828 Acked Read 1 Done Done Sent false
6719 2829 Acked Read 1 Done Done Sent false
6720 2830 Acked Write 7 Done Done Sent false
6721 2831 Acked Read 9 Done Done Sent false
6722 2832 Acked Read 4 Done Done Sent false
6723 2833 Acked Read 4 Done Done Sent false
6724 2834 Acked Write 2 Done Done Sent false
6725 2835 Acked Flush 0 Done Done Sent false
6726 2836 Acked Write 6 Done Done Sent false
6727 2837 Acked Flush 0 Done Done Sent false
6728 2838 Acked Write 8 Done Done Sent false
6729 2839 Acked Flush 0 Done Done Sent false
6730 2840 Acked Read 9 Done Done Sent false
6731 2841 Acked Write 3 Done Done Sent false
6732 2842 Acked Read 8 Done Done Sent false
6733 2843 Acked Write 7 Done Done Sent false
6734 2844 Acked Write 1 Done Done Sent false
6735 2845 Acked Read 10 Done Done Sent false
6736 2846 Acked Read 3 Done Done Sent false
6737 2847 Acked Read 8 Done Done Sent false
6738 2848 Acked Read 4 Done Done Sent false
6739 2849 Acked Read 3 Done Done Sent false
6740 2850 Acked Write 10 Done Done Sent false
6741 2851 Acked Write 4 Done Done Sent false
6742 2852 Acked Flush 0 Done Done Sent false
6743 2853 Acked Write 2 Done Done Sent false
6744 2854 Acked Read 3 Done Done Sent false
6745 2855 Acked Write 2 Done Done Sent false
6746 2856 Acked Read 4 Done Done Sent false
6747 2857 Acked Read 4 Done Done Sent false
6748 2858 Acked Write 9 Done Done Sent false
6749 2859 Acked Write 1 Done Done Sent false
6750 2860 Acked Write 8 Done Done Sent false
6751 2861 Acked Write 6 Done Done Sent false
6752 2862 Acked Write 8 Done Done Sent false
6753 2863 Acked Write 6 Done Done Sent false
6754 2864 Acked Read 1 Done Done Sent false
6755 2865 Acked Flush 0 Done Done Sent false
6756 2866 Acked Read 6 Done Done Sent false
6757 2867 Acked Write 7 Done Done Sent false
6758 2868 Acked Write 1 Done Done Sent false
6759 2869 Acked Read 9 Done Done Sent false
6760 2870 Acked Read 9 Done Done Sent false
6761 2871 Acked Read 2 Done Done Sent false
6762 2872 Acked Write 8 Done Done Sent false
6763 2873 Acked Read 9 Done Done Sent false
6764 2874 Acked Flush 0 Done Done Sent false
6765 2875 Acked Write 4 Done Done Sent false
6766 2876 Acked Read 5 Done Done Sent false
6767 2877 Acked Flush 0 Done Done Sent false
6768 2878 Acked Write 10 Done Done Sent false
6769 2879 Acked Write 5 Done Done Sent false
6770 2880 Acked Read 1 Done Done Sent false
6771 2881 Acked Write 10 Done Done Sent false
6772 2882 Acked Read 10 Done Done Sent false
6773 2883 Acked Read 1 Done Done Sent false
6774 2884 Acked Read 10 Done Done Sent false
6775 2885 Acked Read 8 Done Done Sent false
6776 2886 Acked Read 5 Done Done Sent false
6777 2887 Acked Read 5 Done Done Sent false
6778 2888 Acked Write 8 Done Done Sent false
6779 2889 Acked Flush 0 Done Done Sent false
6780 2890 Acked Read 2 Done Done Sent false
6781 2891 Acked Read 4 Done Done Sent false
6782 2892 Acked Write 7 Done Done Sent false
6783 2893 Acked Flush 0 Done Done Sent false
6784 2894 Acked Write 4 Done Done Sent false
6785 2895 Acked Write 1 Done Done Sent false
6786 2896 Acked Read 6 Done Done Sent false
6787 2897 Acked Read 8 Done Done Sent false
6788 2898 Acked Read 5 Done Done Sent false
6789 2899 Acked Read 4 Done Done Sent false
6790 2900 Acked Write 2 Done Done Sent false
6791 2901 Acked Write 8 Done Done Sent false
6792 2902 Acked Read 8 Done Done Sent false
6793 2903 Acked Read 5 Done Done Sent false
6794 2904 Acked Flush 0 Done Done Sent false
6795 2905 Acked Write 8 Done Done Sent false
6796 2906 Acked Read 1 Done Done Sent false
6797 2907 Acked Read 4 Done Done Sent false
6798 2908 Acked Read 3 Done Done Sent false
6799 2909 Acked Write 2 Done Done Sent false
6800 2910 Acked Write 4 Done Done Sent false
6801 2911 Acked Write 9 Done Done Sent false
6802 2912 Acked Read 2 Done Done Sent false
6803 2913 Acked Read 8 Done Done Sent false
6804 2914 Acked Read 10 Done Done Sent false
6805 2915 Acked Read 7 Done Done Sent false
6806 2916 Acked Read 9 Done Done Sent false
6807 2917 Acked Read 7 Done Done Sent false
6808 2918 Acked Read 9 Done Done Sent false
6809 2919 Acked Write 5 Done Done Sent false
6810 2920 Acked Read 8 Done Done Sent false
6811 2921 Acked Write 1 Done Done Sent false
6812 2922 Acked Read 5 Done Done Sent false
6813 2923 Acked Read 8 Done Done Sent false
6814 2924 Acked Read 6 Done Done Sent false
6815 2925 Acked Read 6 Done Done Sent false
6816 2926 Acked Write 9 Done Done Sent false
6817 2927 Acked Read 9 Done Done Sent false
6818 2928 Acked Read 9 Done Done Sent false
6819 2929 Acked Read 10 Done Done Sent false
6820 2930 Acked Read 3 Done Done Sent false
6821 2931 Acked Read 7 Done Done Sent false
6822 2932 Acked Write 7 Done Done Sent false
6823 2933 Acked Write 5 Done Done Sent false
6824 2934 Acked Write 8 Done Done Sent false
6825 2935 Acked Read 6 Done Done Sent false
6826 2936 Acked Write 5 Done Done Sent false
6827 2937 Acked Read 7 Done Done Sent false
6828 2938 Acked Read 10 Done Done Sent false
6829 2939 Acked Read 6 Done Done Sent false
6830 2940 Acked Read 7 Done Done Sent false
6831 2941 Acked Write 1 Done Done Sent false
6832 2942 Acked Write 1 Done Done Sent false
6833 2943 Acked Write 6 Done Done Sent false
6834 2944 Acked Write 8 Done Done Sent false
6835 2945 Acked Write 1 Done Done Sent false
6836 2946 Acked Write 4 Done Done Sent false
6837 2947 Acked Write 8 Done Done Sent false
6838 2948 Acked Write 2 Done Done Sent false
6839 2949 Acked Read 9 Done Done Sent false
6840 2950 Acked Read 9 Done Done Sent false
6841 2951 Acked Read 9 Done Done Sent false
6842 2952 Acked Flush 0 Done Done Sent false
6843 2953 Acked Write 9 Done Done Sent false
6844 2954 Acked Read 1 Done Done Sent false
6845 2955 Acked Flush 0 Done Done Sent false
6846 2956 Acked Read 10 Done Done Sent false
6847 2957 Acked Write 4 Done Done Sent false
6848 2958 Acked Read 5 Done Done Sent false
6849 2959 Acked Read 6 Done Done Sent false
6850 2960 Acked Write 6 Done Done Sent false
6851 2961 Acked Read 4 Done Done Sent false
6852 2962 Acked Write 7 Done Done Sent false
6853 2963 Acked Read 10 Done Done Sent false
6854 2964 Acked Write 2 Done Done Sent false
6855 2965 Acked Write 10 Done Done Sent false
6856 2966 Acked Write 10 Done Done Sent false
6857 2967 Acked Read 3 Done Done Sent false
6858 2968 Acked Read 6 Done Done Sent false
6859 2969 Acked Read 7 Done Done Sent false
6860 2970 Acked Read 2 Done Done Sent false
6861 2971 Acked Write 6 Done Done Sent false
6862 2972 Acked Read 2 Done Done Sent false
6863 2973 Acked Write 2 Done Done Sent false
6864 2974 Acked Read 10 Done Done Sent false
6865 2975 Acked Read 1 Done Done Sent false
6866 2976 Acked Write 1 Done Done Sent false
6867 2977 Acked Read 4 Done Done Sent false
6868 2978 Acked Write 6 Done Done Sent false
6869 2979 Acked Read 6 Done Done Sent false
6870 2980 Acked Flush 0 Done Done Sent false
6871 2981 Acked Read 3 Done Done Sent false
6872 2982 Acked Read 10 Done Done Sent false
6873 2983 Acked Write 2 Done Done Sent false
6874 2984 Acked Read 6 Done Done Sent false
6875 2985 Acked Read 2 Done Done Sent false
6876 2986 Acked Read 6 Done Done Sent false
6877 2987 Acked Read 6 Done Done Sent false
6878 2988 Acked Read 4 Done Done Sent false
6879 2989 Acked Write 9 Done Done Sent false
6880 2990 Acked Read 8 Done Done Sent false
6881 2991 Acked Write 4 Done Done Sent false
6882 2992 Acked Read 7 Done Done Sent false
6883 2993 Acked Read 7 Done Done Sent false
6884 2994 Acked Write 7 Done Done Sent false
6885 2995 Acked Write 9 Done Done Sent false
6886 2996 Acked Read 3 Done Done Sent false
6887 2997 Acked Read 3 Done Done Sent false
6888 2998 Acked Write 5 Done Done Sent false
6889 2999 Acked Flush 0 Done Done Sent false
6890 3000 Acked Write 7 Done Done Sent false
6891 3001 Acked Read 5 Done Done Sent false
6892 3002 Acked Write 1 Done Done Sent false
6893 3003 Acked Read 9 Done Done Sent false
6894 3004 Acked Write 7 Done Done Sent false
6895 3005 Acked Read 2 Done Done Sent false
6896 3006 Acked Flush 0 Done Done Sent false
6897 3007 Acked Read 9 Done Done Sent false
6898 3008 Acked Read 9 Done Done Sent false
6899 3009 Acked Read 3 Done Done Sent false
6900 3010 Acked Write 5 Done Done Sent false
6901 3011 Acked Read 8 Done Done Sent false
6902 3012 Acked Read 1 Done Done Sent false
6903 3013 Acked Write 7 Done Done Sent false
6904 3014 Acked Read 9 Done Done Sent false
6905 3015 Acked Write 3 Done Done Sent false
6906 3016 Acked Write 10 Done Done Sent false
6907 3017 Acked Write 7 Done Done Sent false
6908 3018 Acked Read 5 Done Done Sent false
6909 3019 Acked Write 6 Done Done Sent false
6910 3020 Acked Read 4 Done Done Sent false
6911 3021 Acked Write 8 Done Done Sent false
6912 3022 Acked Read 5 Done Done Sent false
6913 3023 Acked Read 4 Done Done Sent false
6914 3024 Acked Flush 0 Done Done Sent false
6915 3025 Acked Read 5 Done Done Sent false
6916 3026 Acked Read 5 Done Done Sent false
6917 3027 Acked Read 6 Done Done Sent false
6918 3028 Acked Flush 0 Done Done Sent false
6919 3029 Acked Write 10 Done Done Sent false
6920 3030 Acked Write 3 Done Done Sent false
6921 3031 Acked Write 5 Done Done Sent false
6922 3032 Acked Write 10 Done Done Sent false
6923 3033 Acked Read 5 Done Done Sent false
6924 3034 Acked Write 5 Done Done Sent false
6925 3035 Acked Write 6 Done Done Sent false
6926 3036 Acked Read 7 Done Done Sent false
6927 3037 Acked Read 2 Done Done Sent false
6928 3038 Acked Write 1 Done Done Sent false
6929 3039 Acked Write 5 Done Done Sent false
6930 3040 Acked Read 1 Done Done Sent false
6931 3041 Acked Write 5 Done Done Sent false
6932 3042 Acked Read 1 Done Done Sent false
6933 3043 Acked Flush 0 Done Done Sent false
6934 3044 Acked Read 7 Done Done Sent false
6935 3045 Acked Write 6 Done Done Sent false
6936 3046 Acked Write 1 Done Done Sent false
6937 3047 Acked Read 3 Done Done Sent false
6938 3048 Acked Write 7 Done Done Sent false
6939 3049 Acked Write 2 Done Done Sent false
6940 3050 Acked Read 2 Done Done Sent false
6941 3051 Acked Write 4 Done Done Sent false
6942 3052 Acked Write 9 Done Done Sent false
6943 3053 Acked Read 3 Done Done Sent false
6944 3054 Acked Read 3 Done Done Sent false
6945 3055 Acked Write 1 Done Done Sent false
6946 3056 Acked Flush 0 Done Done Sent false
6947 3057 Acked Read 7 Done Done Sent false
6948 3058 Acked Read 2 Done Done Sent false
6949 3059 Acked Read 9 Done Done Sent false
6950 3060 Acked Read 3 Done Done Sent false
6951 3061 Acked Read 2 Done Done Sent false
6952 3062 Acked Read 8 Done Done Sent false
6953 3063 Acked Write 8 Done Done Sent false
6954 3064 Acked Write 4 Done Done Sent false
6955 3065 Acked Write 5 Done Done Sent false
6956 3066 Acked Read 9 Done Done Sent false
6957 3067 Acked Read 6 Done Done Sent false
6958 3068 Acked Read 10 Done Done Sent false
6959 3069 Acked Write 3 Done Done Sent false
6960 3070 Acked Read 8 Done Done Sent false
6961 3071 Acked Read 1 Done Done Sent false
6962 3072 Acked Write 7 Done Done Sent false
6963 3073 Acked Write 5 Done Done Sent false
6964 3074 Acked Read 3 Done Done Sent false
6965 3075 Acked Flush 0 Done Done Sent false
6966 3076 Acked Write 4 Done Done Sent false
6967 3077 Acked Read 5 Done Done Sent false
6968 3078 Acked Write 3 Done Done Sent false
6969 3079 Acked Write 6 Done Done Sent false
6970 3080 Acked Write 2 Done Done Sent false
6971 3081 Acked Flush 0 Done Done Sent false
6972 3082 Acked Write 4 Done Done Sent false
6973 3083 Acked Read 9 Done Done Sent false
6974 3084 Acked Read 8 Done Done Sent false
6975 3085 Acked Read 3 Done Done Sent false
6976 3086 Acked Write 9 Done Done Sent false
6977 3087 Acked Write 4 Done Done Sent false
6978 3088 Acked Read 10 Done Done Sent false
6979 3089 Acked Write 1 Done Done Sent false
6980 3090 Acked Write 2 Done Done Sent false
6981 3091 Acked Write 10 Done Done Sent false
6982 3092 Acked Read 7 Done Done Sent false
6983 3093 Acked Read 1 Done Done Sent false
6984 3094 Acked Read 1 Done Done Sent false
6985 3095 Acked Read 7 Done Done Sent false
6986 3096 Acked Write 8 Done Done Sent false
6987 3097 Acked Read 7 Done Done Sent false
6988 3098 Acked Write 3 Done Done Sent false
6989 3099 Acked Read 10 Done Done Sent false
6990 3100 Acked Write 5 Done Done Sent false
6991 3101 Acked Write 7 Done Done Sent false
6992 3102 Acked Read 6 Done Done Sent false
6993 3103 Acked Write 8 Done Done Sent false
6994 3104 Acked Write 7 Done Done Sent false
6995 3105 Acked Read 10 Done Done Sent false
6996 3106 Acked Read 3 Done Done Sent false
6997 3107 Acked Write 3 Done Done Sent false
6998 3108 Acked Write 8 Done Done Sent false
6999 3109 Acked Read 5 Done Done Sent false
7000 3110 Acked Write 5 Done Done Sent false
7001 3111 Acked Read 8 Done Done Sent false
7002 3112 Acked Read 2 Done Done Sent false
7003 STATES DS:0 DS:1 DS:2 TOTAL
7004 Sent 0 0 301 301
7005 Done 301 301 0 602
7006 Skipped 0 0 0 0
7007 Error 0 0 0 0
7008 Last Flush: Some(JobId(3081)) Some(JobId(3081)) Some(JobId(2811))
7009 Downstairs last five completed: 2811 2810 2809 2808 2807
7010 Upstairs last five completed: 3112 3111 3110 3109 3108
7011 CLIENT: Up:0 ds:301 act:2
70122025-02-25T17:19:40.424ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
70132025-02-25T17:19:41.409ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
70142025-02-25T17:19:41.409ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
70152025-02-25T17:19:41.409ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
70162025-02-25T17:19:41.409ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
70172025-02-25T17:19:41.409ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
7018 [8830] got start action
7019 [8830] I am going to restart
7020 Starting downstairs at port 8830
7021 Make output file at "/tmp/dsc/downstairs-8830.txt"
7022 [8830][2] reports Starting
7023 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1215)
7024 [8830][2] reports Running
70252025-02-25T17:19:41.910ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
70262025-02-25T17:19:42.428ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
70272025-02-25T17:19:43.429ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
7028 ----------------------------------------------------------------
7029 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7030 States: Active Active Offline
7031 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7032 2812 Acked Write 8 Done Done Sent false
7033 2813 Acked Read 6 Done Done Sent false
7034 2814 Acked Write 3 Done Done Sent false
7035 2815 Acked Read 2 Done Done Sent false
7036 2816 Acked Read 9 Done Done Sent false
7037 2817 Acked Flush 0 Done Done Sent false
7038 2818 Acked Read 3 Done Done Sent false
7039 2819 Acked Read 10 Done Done Sent false
7040 2820 Acked Flush 0 Done Done Sent false
7041 2821 Acked Flush 0 Done Done Sent false
7042 2822 Acked Read 2 Done Done Sent false
7043 2823 Acked Write 5 Done Done Sent false
7044 2824 Acked Write 2 Done Done Sent false
7045 2825 Acked Read 1 Done Done Sent false
7046 2826 Acked Read 4 Done Done Sent false
7047 2827 Acked Read 10 Done Done Sent false
7048 2828 Acked Read 1 Done Done Sent false
7049 2829 Acked Read 1 Done Done Sent false
7050 2830 Acked Write 7 Done Done Sent false
7051 2831 Acked Read 9 Done Done Sent false
7052 2832 Acked Read 4 Done Done Sent false
7053 2833 Acked Read 4 Done Done Sent false
7054 2834 Acked Write 2 Done Done Sent false
7055 2835 Acked Flush 0 Done Done Sent false
7056 2836 Acked Write 6 Done Done Sent false
7057 2837 Acked Flush 0 Done Done Sent false
7058 2838 Acked Write 8 Done Done Sent false
7059 2839 Acked Flush 0 Done Done Sent false
7060 2840 Acked Read 9 Done Done Sent false
7061 2841 Acked Write 3 Done Done Sent false
7062 2842 Acked Read 8 Done Done Sent false
7063 2843 Acked Write 7 Done Done Sent false
7064 2844 Acked Write 1 Done Done Sent false
7065 2845 Acked Read 10 Done Done Sent false
7066 2846 Acked Read 3 Done Done Sent false
7067 2847 Acked Read 8 Done Done Sent false
7068 2848 Acked Read 4 Done Done Sent false
7069 2849 Acked Read 3 Done Done Sent false
7070 2850 Acked Write 10 Done Done Sent false
7071 2851 Acked Write 4 Done Done Sent false
7072 2852 Acked Flush 0 Done Done Sent false
7073 2853 Acked Write 2 Done Done Sent false
7074 2854 Acked Read 3 Done Done Sent false
7075 2855 Acked Write 2 Done Done Sent false
7076 2856 Acked Read 4 Done Done Sent false
7077 2857 Acked Read 4 Done Done Sent false
7078 2858 Acked Write 9 Done Done Sent false
7079 2859 Acked Write 1 Done Done Sent false
7080 2860 Acked Write 8 Done Done Sent false
7081 2861 Acked Write 6 Done Done Sent false
7082 2862 Acked Write 8 Done Done Sent false
7083 2863 Acked Write 6 Done Done Sent false
7084 2864 Acked Read 1 Done Done Sent false
7085 2865 Acked Flush 0 Done Done Sent false
7086 2866 Acked Read 6 Done Done Sent false
7087 2867 Acked Write 7 Done Done Sent false
7088 2868 Acked Write 1 Done Done Sent false
7089 2869 Acked Read 9 Done Done Sent false
7090 2870 Acked Read 9 Done Done Sent false
7091 2871 Acked Read 2 Done Done Sent false
7092 2872 Acked Write 8 Done Done Sent false
7093 2873 Acked Read 9 Done Done Sent false
7094 2874 Acked Flush 0 Done Done Sent false
7095 2875 Acked Write 4 Done Done Sent false
7096 2876 Acked Read 5 Done Done Sent false
7097 2877 Acked Flush 0 Done Done Sent false
7098 2878 Acked Write 10 Done Done Sent false
7099 2879 Acked Write 5 Done Done Sent false
7100 2880 Acked Read 1 Done Done Sent false
7101 2881 Acked Write 10 Done Done Sent false
7102 2882 Acked Read 10 Done Done Sent false
7103 2883 Acked Read 1 Done Done Sent false
7104 2884 Acked Read 10 Done Done Sent false
7105 2885 Acked Read 8 Done Done Sent false
7106 2886 Acked Read 5 Done Done Sent false
7107 2887 Acked Read 5 Done Done Sent false
7108 2888 Acked Write 8 Done Done Sent false
7109 2889 Acked Flush 0 Done Done Sent false
7110 2890 Acked Read 2 Done Done Sent false
7111 2891 Acked Read 4 Done Done Sent false
7112 2892 Acked Write 7 Done Done Sent false
7113 2893 Acked Flush 0 Done Done Sent false
7114 2894 Acked Write 4 Done Done Sent false
7115 2895 Acked Write 1 Done Done Sent false
7116 2896 Acked Read 6 Done Done Sent false
7117 2897 Acked Read 8 Done Done Sent false
7118 2898 Acked Read 5 Done Done Sent false
7119 2899 Acked Read 4 Done Done Sent false
7120 2900 Acked Write 2 Done Done Sent false
7121 2901 Acked Write 8 Done Done Sent false
7122 2902 Acked Read 8 Done Done Sent false
7123 2903 Acked Read 5 Done Done Sent false
7124 2904 Acked Flush 0 Done Done Sent false
7125 2905 Acked Write 8 Done Done Sent false
7126 2906 Acked Read 1 Done Done Sent false
7127 2907 Acked Read 4 Done Done Sent false
7128 2908 Acked Read 3 Done Done Sent false
7129 2909 Acked Write 2 Done Done Sent false
7130 2910 Acked Write 4 Done Done Sent false
7131 2911 Acked Write 9 Done Done Sent false
7132 2912 Acked Read 2 Done Done Sent false
7133 2913 Acked Read 8 Done Done Sent false
7134 2914 Acked Read 10 Done Done Sent false
7135 2915 Acked Read 7 Done Done Sent false
7136 2916 Acked Read 9 Done Done Sent false
7137 2917 Acked Read 7 Done Done Sent false
7138 2918 Acked Read 9 Done Done Sent false
7139 2919 Acked Write 5 Done Done Sent false
7140 2920 Acked Read 8 Done Done Sent false
7141 2921 Acked Write 1 Done Done Sent false
7142 2922 Acked Read 5 Done Done Sent false
7143 2923 Acked Read 8 Done Done Sent false
7144 2924 Acked Read 6 Done Done Sent false
7145 2925 Acked Read 6 Done Done Sent false
7146 2926 Acked Write 9 Done Done Sent false
7147 2927 Acked Read 9 Done Done Sent false
7148 2928 Acked Read 9 Done Done Sent false
7149 2929 Acked Read 10 Done Done Sent false
7150 2930 Acked Read 3 Done Done Sent false
7151 2931 Acked Read 7 Done Done Sent false
7152 2932 Acked Write 7 Done Done Sent false
7153 2933 Acked Write 5 Done Done Sent false
7154 2934 Acked Write 8 Done Done Sent false
7155 2935 Acked Read 6 Done Done Sent false
7156 2936 Acked Write 5 Done Done Sent false
7157 2937 Acked Read 7 Done Done Sent false
7158 2938 Acked Read 10 Done Done Sent false
7159 2939 Acked Read 6 Done Done Sent false
7160 2940 Acked Read 7 Done Done Sent false
7161 2941 Acked Write 1 Done Done Sent false
7162 2942 Acked Write 1 Done Done Sent false
7163 2943 Acked Write 6 Done Done Sent false
7164 2944 Acked Write 8 Done Done Sent false
7165 2945 Acked Write 1 Done Done Sent false
7166 2946 Acked Write 4 Done Done Sent false
7167 2947 Acked Write 8 Done Done Sent false
7168 2948 Acked Write 2 Done Done Sent false
7169 2949 Acked Read 9 Done Done Sent false
7170 2950 Acked Read 9 Done Done Sent false
7171 2951 Acked Read 9 Done Done Sent false
7172 2952 Acked Flush 0 Done Done Sent false
7173 2953 Acked Write 9 Done Done Sent false
7174 2954 Acked Read 1 Done Done Sent false
7175 2955 Acked Flush 0 Done Done Sent false
7176 2956 Acked Read 10 Done Done Sent false
7177 2957 Acked Write 4 Done Done Sent false
7178 2958 Acked Read 5 Done Done Sent false
7179 2959 Acked Read 6 Done Done Sent false
7180 2960 Acked Write 6 Done Done Sent false
7181 2961 Acked Read 4 Done Done Sent false
7182 2962 Acked Write 7 Done Done Sent false
7183 2963 Acked Read 10 Done Done Sent false
7184 2964 Acked Write 2 Done Done Sent false
7185 2965 Acked Write 10 Done Done Sent false
7186 2966 Acked Write 10 Done Done Sent false
7187 2967 Acked Read 3 Done Done Sent false
7188 2968 Acked Read 6 Done Done Sent false
7189 2969 Acked Read 7 Done Done Sent false
7190 2970 Acked Read 2 Done Done Sent false
7191 2971 Acked Write 6 Done Done Sent false
7192 2972 Acked Read 2 Done Done Sent false
7193 2973 Acked Write 2 Done Done Sent false
7194 2974 Acked Read 10 Done Done Sent false
7195 2975 Acked Read 1 Done Done Sent false
7196 2976 Acked Write 1 Done Done Sent false
7197 2977 Acked Read 4 Done Done Sent false
7198 2978 Acked Write 6 Done Done Sent false
7199 2979 Acked Read 6 Done Done Sent false
7200 2980 Acked Flush 0 Done Done Sent false
7201 2981 Acked Read 3 Done Done Sent false
7202 2982 Acked Read 10 Done Done Sent false
7203 2983 Acked Write 2 Done Done Sent false
7204 2984 Acked Read 6 Done Done Sent false
7205 2985 Acked Read 2 Done Done Sent false
7206 2986 Acked Read 6 Done Done Sent false
7207 2987 Acked Read 6 Done Done Sent false
7208 2988 Acked Read 4 Done Done Sent false
7209 2989 Acked Write 9 Done Done Sent false
7210 2990 Acked Read 8 Done Done Sent false
7211 2991 Acked Write 4 Done Done Sent false
7212 2992 Acked Read 7 Done Done Sent false
7213 2993 Acked Read 7 Done Done Sent false
7214 2994 Acked Write 7 Done Done Sent false
7215 2995 Acked Write 9 Done Done Sent false
7216 2996 Acked Read 3 Done Done Sent false
7217 2997 Acked Read 3 Done Done Sent false
7218 2998 Acked Write 5 Done Done Sent false
7219 2999 Acked Flush 0 Done Done Sent false
7220 3000 Acked Write 7 Done Done Sent false
7221 3001 Acked Read 5 Done Done Sent false
7222 3002 Acked Write 1 Done Done Sent false
7223 3003 Acked Read 9 Done Done Sent false
7224 3004 Acked Write 7 Done Done Sent false
7225 3005 Acked Read 2 Done Done Sent false
7226 3006 Acked Flush 0 Done Done Sent false
7227 3007 Acked Read 9 Done Done Sent false
7228 3008 Acked Read 9 Done Done Sent false
7229 3009 Acked Read 3 Done Done Sent false
7230 3010 Acked Write 5 Done Done Sent false
7231 3011 Acked Read 8 Done Done Sent false
7232 3012 Acked Read 1 Done Done Sent false
7233 3013 Acked Write 7 Done Done Sent false
7234 3014 Acked Read 9 Done Done Sent false
7235 3015 Acked Write 3 Done Done Sent false
7236 3016 Acked Write 10 Done Done Sent false
7237 3017 Acked Write 7 Done Done Sent false
7238 3018 Acked Read 5 Done Done Sent false
7239 3019 Acked Write 6 Done Done Sent false
7240 3020 Acked Read 4 Done Done Sent false
7241 3021 Acked Write 8 Done Done Sent false
7242 3022 Acked Read 5 Done Done Sent false
7243 3023 Acked Read 4 Done Done Sent false
7244 3024 Acked Flush 0 Done Done Sent false
7245 3025 Acked Read 5 Done Done Sent false
7246 3026 Acked Read 5 Done Done Sent false
7247 3027 Acked Read 6 Done Done Sent false
7248 3028 Acked Flush 0 Done Done Sent false
7249 3029 Acked Write 10 Done Done Sent false
7250 3030 Acked Write 3 Done Done Sent false
7251 3031 Acked Write 5 Done Done Sent false
7252 3032 Acked Write 10 Done Done Sent false
7253 3033 Acked Read 5 Done Done Sent false
7254 3034 Acked Write 5 Done Done Sent false
7255 3035 Acked Write 6 Done Done Sent false
7256 3036 Acked Read 7 Done Done Sent false
7257 3037 Acked Read 2 Done Done Sent false
7258 3038 Acked Write 1 Done Done Sent false
7259 3039 Acked Write 5 Done Done Sent false
7260 3040 Acked Read 1 Done Done Sent false
7261 3041 Acked Write 5 Done Done Sent false
7262 3042 Acked Read 1 Done Done Sent false
7263 3043 Acked Flush 0 Done Done Sent false
7264 3044 Acked Read 7 Done Done Sent false
7265 3045 Acked Write 6 Done Done Sent false
7266 3046 Acked Write 1 Done Done Sent false
7267 3047 Acked Read 3 Done Done Sent false
7268 3048 Acked Write 7 Done Done Sent false
7269 3049 Acked Write 2 Done Done Sent false
7270 3050 Acked Read 2 Done Done Sent false
7271 3051 Acked Write 4 Done Done Sent false
7272 3052 Acked Write 9 Done Done Sent false
7273 3053 Acked Read 3 Done Done Sent false
7274 3054 Acked Read 3 Done Done Sent false
7275 3055 Acked Write 1 Done Done Sent false
7276 3056 Acked Flush 0 Done Done Sent false
7277 3057 Acked Read 7 Done Done Sent false
7278 3058 Acked Read 2 Done Done Sent false
7279 3059 Acked Read 9 Done Done Sent false
7280 3060 Acked Read 3 Done Done Sent false
7281 3061 Acked Read 2 Done Done Sent false
7282 3062 Acked Read 8 Done Done Sent false
7283 3063 Acked Write 8 Done Done Sent false
7284 3064 Acked Write 4 Done Done Sent false
7285 3065 Acked Write 5 Done Done Sent false
7286 3066 Acked Read 9 Done Done Sent false
7287 3067 Acked Read 6 Done Done Sent false
7288 3068 Acked Read 10 Done Done Sent false
7289 3069 Acked Write 3 Done Done Sent false
7290 3070 Acked Read 8 Done Done Sent false
7291 3071 Acked Read 1 Done Done Sent false
7292 3072 Acked Write 7 Done Done Sent false
7293 3073 Acked Write 5 Done Done Sent false
7294 3074 Acked Read 3 Done Done Sent false
7295 3075 Acked Flush 0 Done Done Sent false
7296 3076 Acked Write 4 Done Done Sent false
7297 3077 Acked Read 5 Done Done Sent false
7298 3078 Acked Write 3 Done Done Sent false
7299 3079 Acked Write 6 Done Done Sent false
7300 3080 Acked Write 2 Done Done Sent false
7301 3081 Acked Flush 0 Done Done Sent false
7302 3082 Acked Write 4 Done Done Sent false
7303 3083 Acked Read 9 Done Done Sent false
7304 3084 Acked Read 8 Done Done Sent false
7305 3085 Acked Read 3 Done Done Sent false
7306 3086 Acked Write 9 Done Done Sent false
7307 3087 Acked Write 4 Done Done Sent false
7308 3088 Acked Read 10 Done Done Sent false
7309 3089 Acked Write 1 Done Done Sent false
7310 3090 Acked Write 2 Done Done Sent false
7311 3091 Acked Write 10 Done Done Sent false
7312 3092 Acked Read 7 Done Done Sent false
7313 3093 Acked Read 1 Done Done Sent false
7314 3094 Acked Read 1 Done Done Sent false
7315 3095 Acked Read 7 Done Done Sent false
7316 3096 Acked Write 8 Done Done Sent false
7317 3097 Acked Read 7 Done Done Sent false
7318 3098 Acked Write 3 Done Done Sent false
7319 3099 Acked Read 10 Done Done Sent false
7320 3100 Acked Write 5 Done Done Sent false
7321 3101 Acked Write 7 Done Done Sent false
7322 3102 Acked Read 6 Done Done Sent false
7323 3103 Acked Write 8 Done Done Sent false
7324 3104 Acked Write 7 Done Done Sent false
7325 3105 Acked Read 10 Done Done Sent false
7326 3106 Acked Read 3 Done Done Sent false
7327 3107 Acked Write 3 Done Done Sent false
7328 3108 Acked Write 8 Done Done Sent false
7329 3109 Acked Read 5 Done Done Sent false
7330 3110 Acked Write 5 Done Done Sent false
7331 3111 Acked Read 8 Done Done Sent false
7332 3112 Acked Read 2 Done Done Sent false
7333 STATES DS:0 DS:1 DS:2 TOTAL
7334 Sent 0 0 301 301
7335 Done 301 301 0 602
7336 Skipped 0 0 0 0
7337 Error 0 0 0 0
7338 Last Flush: Some(JobId(3081)) Some(JobId(3081)) Some(JobId(2811))
7339 Downstairs last five completed: 2811 2810 2809 2808 2807
7340 Upstairs last five completed: 3112 3111 3110 3109 3108
7341 CLIENT: Up:0 ds:301 act:2
73422025-02-25T17:19:44.430ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
73432025-02-25T17:19:45.432ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
73442025-02-25T17:19:46.434ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
73452025-02-25T17:19:47.436ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
7346 ----------------------------------------------------------------
7347 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7348 States: Active Active Offline
7349 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7350 2812 Acked Write 8 Done Done Sent false
7351 2813 Acked Read 6 Done Done Sent false
7352 2814 Acked Write 3 Done Done Sent false
7353 2815 Acked Read 2 Done Done Sent false
7354 2816 Acked Read 9 Done Done Sent false
7355 2817 Acked Flush 0 Done Done Sent false
7356 2818 Acked Read 3 Done Done Sent false
7357 2819 Acked Read 10 Done Done Sent false
7358 2820 Acked Flush 0 Done Done Sent false
7359 2821 Acked Flush 0 Done Done Sent false
7360 2822 Acked Read 2 Done Done Sent false
7361 2823 Acked Write 5 Done Done Sent false
7362 2824 Acked Write 2 Done Done Sent false
7363 2825 Acked Read 1 Done Done Sent false
7364 2826 Acked Read 4 Done Done Sent false
7365 2827 Acked Read 10 Done Done Sent false
7366 2828 Acked Read 1 Done Done Sent false
7367 2829 Acked Read 1 Done Done Sent false
7368 2830 Acked Write 7 Done Done Sent false
7369 2831 Acked Read 9 Done Done Sent false
7370 2832 Acked Read 4 Done Done Sent false
7371 2833 Acked Read 4 Done Done Sent false
7372 2834 Acked Write 2 Done Done Sent false
7373 2835 Acked Flush 0 Done Done Sent false
7374 2836 Acked Write 6 Done Done Sent false
7375 2837 Acked Flush 0 Done Done Sent false
7376 2838 Acked Write 8 Done Done Sent false
7377 2839 Acked Flush 0 Done Done Sent false
7378 2840 Acked Read 9 Done Done Sent false
7379 2841 Acked Write 3 Done Done Sent false
7380 2842 Acked Read 8 Done Done Sent false
7381 2843 Acked Write 7 Done Done Sent false
7382 2844 Acked Write 1 Done Done Sent false
7383 2845 Acked Read 10 Done Done Sent false
7384 2846 Acked Read 3 Done Done Sent false
7385 2847 Acked Read 8 Done Done Sent false
7386 2848 Acked Read 4 Done Done Sent false
7387 2849 Acked Read 3 Done Done Sent false
7388 2850 Acked Write 10 Done Done Sent false
7389 2851 Acked Write 4 Done Done Sent false
7390 2852 Acked Flush 0 Done Done Sent false
7391 2853 Acked Write 2 Done Done Sent false
7392 2854 Acked Read 3 Done Done Sent false
7393 2855 Acked Write 2 Done Done Sent false
7394 2856 Acked Read 4 Done Done Sent false
7395 2857 Acked Read 4 Done Done Sent false
7396 2858 Acked Write 9 Done Done Sent false
7397 2859 Acked Write 1 Done Done Sent false
7398 2860 Acked Write 8 Done Done Sent false
7399 2861 Acked Write 6 Done Done Sent false
7400 2862 Acked Write 8 Done Done Sent false
7401 2863 Acked Write 6 Done Done Sent false
7402 2864 Acked Read 1 Done Done Sent false
7403 2865 Acked Flush 0 Done Done Sent false
7404 2866 Acked Read 6 Done Done Sent false
7405 2867 Acked Write 7 Done Done Sent false
7406 2868 Acked Write 1 Done Done Sent false
7407 2869 Acked Read 9 Done Done Sent false
7408 2870 Acked Read 9 Done Done Sent false
7409 2871 Acked Read 2 Done Done Sent false
7410 2872 Acked Write 8 Done Done Sent false
7411 2873 Acked Read 9 Done Done Sent false
7412 2874 Acked Flush 0 Done Done Sent false
7413 2875 Acked Write 4 Done Done Sent false
7414 2876 Acked Read 5 Done Done Sent false
7415 2877 Acked Flush 0 Done Done Sent false
7416 2878 Acked Write 10 Done Done Sent false
7417 2879 Acked Write 5 Done Done Sent false
7418 2880 Acked Read 1 Done Done Sent false
7419 2881 Acked Write 10 Done Done Sent false
7420 2882 Acked Read 10 Done Done Sent false
7421 2883 Acked Read 1 Done Done Sent false
7422 2884 Acked Read 10 Done Done Sent false
7423 2885 Acked Read 8 Done Done Sent false
7424 2886 Acked Read 5 Done Done Sent false
7425 2887 Acked Read 5 Done Done Sent false
7426 2888 Acked Write 8 Done Done Sent false
7427 2889 Acked Flush 0 Done Done Sent false
7428 2890 Acked Read 2 Done Done Sent false
7429 2891 Acked Read 4 Done Done Sent false
7430 2892 Acked Write 7 Done Done Sent false
7431 2893 Acked Flush 0 Done Done Sent false
7432 2894 Acked Write 4 Done Done Sent false
7433 2895 Acked Write 1 Done Done Sent false
7434 2896 Acked Read 6 Done Done Sent false
7435 2897 Acked Read 8 Done Done Sent false
7436 2898 Acked Read 5 Done Done Sent false
7437 2899 Acked Read 4 Done Done Sent false
7438 2900 Acked Write 2 Done Done Sent false
7439 2901 Acked Write 8 Done Done Sent false
7440 2902 Acked Read 8 Done Done Sent false
7441 2903 Acked Read 5 Done Done Sent false
7442 2904 Acked Flush 0 Done Done Sent false
7443 2905 Acked Write 8 Done Done Sent false
7444 2906 Acked Read 1 Done Done Sent false
7445 2907 Acked Read 4 Done Done Sent false
7446 2908 Acked Read 3 Done Done Sent false
7447 2909 Acked Write 2 Done Done Sent false
7448 2910 Acked Write 4 Done Done Sent false
7449 2911 Acked Write 9 Done Done Sent false
7450 2912 Acked Read 2 Done Done Sent false
7451 2913 Acked Read 8 Done Done Sent false
7452 2914 Acked Read 10 Done Done Sent false
7453 2915 Acked Read 7 Done Done Sent false
7454 2916 Acked Read 9 Done Done Sent false
7455 2917 Acked Read 7 Done Done Sent false
7456 2918 Acked Read 9 Done Done Sent false
7457 2919 Acked Write 5 Done Done Sent false
7458 2920 Acked Read 8 Done Done Sent false
7459 2921 Acked Write 1 Done Done Sent false
7460 2922 Acked Read 5 Done Done Sent false
7461 2923 Acked Read 8 Done Done Sent false
7462 2924 Acked Read 6 Done Done Sent false
7463 2925 Acked Read 6 Done Done Sent false
7464 2926 Acked Write 9 Done Done Sent false
7465 2927 Acked Read 9 Done Done Sent false
7466 2928 Acked Read 9 Done Done Sent false
7467 2929 Acked Read 10 Done Done Sent false
7468 2930 Acked Read 3 Done Done Sent false
7469 2931 Acked Read 7 Done Done Sent false
7470 2932 Acked Write 7 Done Done Sent false
7471 2933 Acked Write 5 Done Done Sent false
7472 2934 Acked Write 8 Done Done Sent false
7473 2935 Acked Read 6 Done Done Sent false
7474 2936 Acked Write 5 Done Done Sent false
7475 2937 Acked Read 7 Done Done Sent false
7476 2938 Acked Read 10 Done Done Sent false
7477 2939 Acked Read 6 Done Done Sent false
7478 2940 Acked Read 7 Done Done Sent false
7479 2941 Acked Write 1 Done Done Sent false
7480 2942 Acked Write 1 Done Done Sent false
7481 2943 Acked Write 6 Done Done Sent false
7482 2944 Acked Write 8 Done Done Sent false
7483 2945 Acked Write 1 Done Done Sent false
7484 2946 Acked Write 4 Done Done Sent false
7485 2947 Acked Write 8 Done Done Sent false
7486 2948 Acked Write 2 Done Done Sent false
7487 2949 Acked Read 9 Done Done Sent false
7488 2950 Acked Read 9 Done Done Sent false
7489 2951 Acked Read 9 Done Done Sent false
7490 2952 Acked Flush 0 Done Done Sent false
7491 2953 Acked Write 9 Done Done Sent false
7492 2954 Acked Read 1 Done Done Sent false
7493 2955 Acked Flush 0 Done Done Sent false
7494 2956 Acked Read 10 Done Done Sent false
7495 2957 Acked Write 4 Done Done Sent false
7496 2958 Acked Read 5 Done Done Sent false
7497 2959 Acked Read 6 Done Done Sent false
7498 2960 Acked Write 6 Done Done Sent false
7499 2961 Acked Read 4 Done Done Sent false
7500 2962 Acked Write 7 Done Done Sent false
7501 2963 Acked Read 10 Done Done Sent false
7502 2964 Acked Write 2 Done Done Sent false
7503 2965 Acked Write 10 Done Done Sent false
7504 2966 Acked Write 10 Done Done Sent false
7505 2967 Acked Read 3 Done Done Sent false
7506 2968 Acked Read 6 Done Done Sent false
7507 2969 Acked Read 7 Done Done Sent false
7508 2970 Acked Read 2 Done Done Sent false
7509 2971 Acked Write 6 Done Done Sent false
7510 2972 Acked Read 2 Done Done Sent false
7511 2973 Acked Write 2 Done Done Sent false
7512 2974 Acked Read 10 Done Done Sent false
7513 2975 Acked Read 1 Done Done Sent false
7514 2976 Acked Write 1 Done Done Sent false
7515 2977 Acked Read 4 Done Done Sent false
7516 2978 Acked Write 6 Done Done Sent false
7517 2979 Acked Read 6 Done Done Sent false
7518 2980 Acked Flush 0 Done Done Sent false
7519 2981 Acked Read 3 Done Done Sent false
7520 2982 Acked Read 10 Done Done Sent false
7521 2983 Acked Write 2 Done Done Sent false
7522 2984 Acked Read 6 Done Done Sent false
7523 2985 Acked Read 2 Done Done Sent false
7524 2986 Acked Read 6 Done Done Sent false
7525 2987 Acked Read 6 Done Done Sent false
7526 2988 Acked Read 4 Done Done Sent false
7527 2989 Acked Write 9 Done Done Sent false
7528 2990 Acked Read 8 Done Done Sent false
7529 2991 Acked Write 4 Done Done Sent false
7530 2992 Acked Read 7 Done Done Sent false
7531 2993 Acked Read 7 Done Done Sent false
7532 2994 Acked Write 7 Done Done Sent false
7533 2995 Acked Write 9 Done Done Sent false
7534 2996 Acked Read 3 Done Done Sent false
7535 2997 Acked Read 3 Done Done Sent false
7536 2998 Acked Write 5 Done Done Sent false
7537 2999 Acked Flush 0 Done Done Sent false
7538 3000 Acked Write 7 Done Done Sent false
7539 3001 Acked Read 5 Done Done Sent false
7540 3002 Acked Write 1 Done Done Sent false
7541 3003 Acked Read 9 Done Done Sent false
7542 3004 Acked Write 7 Done Done Sent false
7543 3005 Acked Read 2 Done Done Sent false
7544 3006 Acked Flush 0 Done Done Sent false
7545 3007 Acked Read 9 Done Done Sent false
7546 3008 Acked Read 9 Done Done Sent false
7547 3009 Acked Read 3 Done Done Sent false
7548 3010 Acked Write 5 Done Done Sent false
7549 3011 Acked Read 8 Done Done Sent false
7550 3012 Acked Read 1 Done Done Sent false
7551 3013 Acked Write 7 Done Done Sent false
7552 3014 Acked Read 9 Done Done Sent false
7553 3015 Acked Write 3 Done Done Sent false
7554 3016 Acked Write 10 Done Done Sent false
7555 3017 Acked Write 7 Done Done Sent false
7556 3018 Acked Read 5 Done Done Sent false
7557 3019 Acked Write 6 Done Done Sent false
7558 3020 Acked Read 4 Done Done Sent false
7559 3021 Acked Write 8 Done Done Sent false
7560 3022 Acked Read 5 Done Done Sent false
7561 3023 Acked Read 4 Done Done Sent false
7562 3024 Acked Flush 0 Done Done Sent false
7563 3025 Acked Read 5 Done Done Sent false
7564 3026 Acked Read 5 Done Done Sent false
7565 3027 Acked Read 6 Done Done Sent false
7566 3028 Acked Flush 0 Done Done Sent false
7567 3029 Acked Write 10 Done Done Sent false
7568 3030 Acked Write 3 Done Done Sent false
7569 3031 Acked Write 5 Done Done Sent false
7570 3032 Acked Write 10 Done Done Sent false
7571 3033 Acked Read 5 Done Done Sent false
7572 3034 Acked Write 5 Done Done Sent false
7573 3035 Acked Write 6 Done Done Sent false
7574 3036 Acked Read 7 Done Done Sent false
7575 3037 Acked Read 2 Done Done Sent false
7576 3038 Acked Write 1 Done Done Sent false
7577 3039 Acked Write 5 Done Done Sent false
7578 3040 Acked Read 1 Done Done Sent false
7579 3041 Acked Write 5 Done Done Sent false
7580 3042 Acked Read 1 Done Done Sent false
7581 3043 Acked Flush 0 Done Done Sent false
7582 3044 Acked Read 7 Done Done Sent false
7583 3045 Acked Write 6 Done Done Sent false
7584 3046 Acked Write 1 Done Done Sent false
7585 3047 Acked Read 3 Done Done Sent false
7586 3048 Acked Write 7 Done Done Sent false
7587 3049 Acked Write 2 Done Done Sent false
7588 3050 Acked Read 2 Done Done Sent false
7589 3051 Acked Write 4 Done Done Sent false
7590 3052 Acked Write 9 Done Done Sent false
7591 3053 Acked Read 3 Done Done Sent false
7592 3054 Acked Read 3 Done Done Sent false
7593 3055 Acked Write 1 Done Done Sent false
7594 3056 Acked Flush 0 Done Done Sent false
7595 3057 Acked Read 7 Done Done Sent false
7596 3058 Acked Read 2 Done Done Sent false
7597 3059 Acked Read 9 Done Done Sent false
7598 3060 Acked Read 3 Done Done Sent false
7599 3061 Acked Read 2 Done Done Sent false
7600 3062 Acked Read 8 Done Done Sent false
7601 3063 Acked Write 8 Done Done Sent false
7602 3064 Acked Write 4 Done Done Sent false
7603 3065 Acked Write 5 Done Done Sent false
7604 3066 Acked Read 9 Done Done Sent false
7605 3067 Acked Read 6 Done Done Sent false
7606 3068 Acked Read 10 Done Done Sent false
7607 3069 Acked Write 3 Done Done Sent false
7608 3070 Acked Read 8 Done Done Sent false
7609 3071 Acked Read 1 Done Done Sent false
7610 3072 Acked Write 7 Done Done Sent false
7611 3073 Acked Write 5 Done Done Sent false
7612 3074 Acked Read 3 Done Done Sent false
7613 3075 Acked Flush 0 Done Done Sent false
7614 3076 Acked Write 4 Done Done Sent false
7615 3077 Acked Read 5 Done Done Sent false
7616 3078 Acked Write 3 Done Done Sent false
7617 3079 Acked Write 6 Done Done Sent false
7618 3080 Acked Write 2 Done Done Sent false
7619 3081 Acked Flush 0 Done Done Sent false
7620 3082 Acked Write 4 Done Done Sent false
7621 3083 Acked Read 9 Done Done Sent false
7622 3084 Acked Read 8 Done Done Sent false
7623 3085 Acked Read 3 Done Done Sent false
7624 3086 Acked Write 9 Done Done Sent false
7625 3087 Acked Write 4 Done Done Sent false
7626 3088 Acked Read 10 Done Done Sent false
7627 3089 Acked Write 1 Done Done Sent false
7628 3090 Acked Write 2 Done Done Sent false
7629 3091 Acked Write 10 Done Done Sent false
7630 3092 Acked Read 7 Done Done Sent false
7631 3093 Acked Read 1 Done Done Sent false
7632 3094 Acked Read 1 Done Done Sent false
7633 3095 Acked Read 7 Done Done Sent false
7634 3096 Acked Write 8 Done Done Sent false
7635 3097 Acked Read 7 Done Done Sent false
7636 3098 Acked Write 3 Done Done Sent false
7637 3099 Acked Read 10 Done Done Sent false
7638 3100 Acked Write 5 Done Done Sent false
7639 3101 Acked Write 7 Done Done Sent false
7640 3102 Acked Read 6 Done Done Sent false
7641 3103 Acked Write 8 Done Done Sent false
7642 3104 Acked Write 7 Done Done Sent false
7643 3105 Acked Read 10 Done Done Sent false
7644 3106 Acked Read 3 Done Done Sent false
7645 3107 Acked Write 3 Done Done Sent false
7646 3108 Acked Write 8 Done Done Sent false
7647 3109 Acked Read 5 Done Done Sent false
7648 3110 Acked Write 5 Done Done Sent false
7649 3111 Acked Read 8 Done Done Sent false
7650 3112 Acked Read 2 Done Done Sent false
7651 STATES DS:0 DS:1 DS:2 TOTAL
7652 Sent 0 0 301 301
7653 Done 301 301 0 602
7654 Skipped 0 0 0 0
7655 Error 0 0 0 0
7656 Last Flush: Some(JobId(3081)) Some(JobId(3081)) Some(JobId(2811))
7657 Downstairs last five completed: 2811 2810 2809 2808 2807
7658 Upstairs last five completed: 3112 3111 3110 3109 3108
7659 CLIENT: Up:0 ds:301 act:2
76602025-02-25T17:19:48.438ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
76612025-02-25T17:19:49.440ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
76622025-02-25T17:19:50.441ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
76632025-02-25T17:19:51.410ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
76642025-02-25T17:19:51.410ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:35716) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
76652025-02-25T17:19:51.412ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
76662025-02-25T17:19:51.412ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
76672025-02-25T17:19:51.412ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
76682025-02-25T17:19:51.412ZINFOcrucible: send last flush ID to this DS: Some(JobId(2811)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
76692025-02-25T17:19:51.412ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
76702025-02-25T17:19:51.412ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(2811)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
76712025-02-25T17:19:51.412ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(2811)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
7672 ----------------------------------------------------------------
7673 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:31
7674 States: Active Active Active
7675 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7676 3082 Acked Write 4 Done Done Done true
7677 3083 Acked Read 9 Done Done Done true
7678 3084 Acked Read 8 Done Done Done true
7679 3085 Acked Read 3 Done Done Done true
7680 3086 Acked Write 9 Done Done Done true
7681 3087 Acked Write 4 Done Done Done true
7682 3088 Acked Read 10 Done Done Done true
7683 3089 Acked Write 1 Done Done Done true
7684 3090 Acked Write 2 Done Done Done true
7685 3091 Acked Write 10 Done Done Done true
7686 3092 Acked Read 7 Done Done Done true
7687 3093 Acked Read 1 Done Done Done true
7688 3094 Acked Read 1 Done Done Done true
7689 3095 Acked Read 7 Done Done Done true
7690 3096 Acked Write 8 Done Done Done true
7691 3097 Acked Read 7 Done Done Done true
7692 3098 Acked Write 3 Done Done Done true
7693 3099 Acked Read 10 Done Done Done true
7694 3100 Acked Write 5 Done Done Done true
7695 3101 Acked Write 7 Done Done Done true
7696 3102 Acked Read 6 Done Done Done true
7697 3103 Acked Write 8 Done Done Done true
7698 3104 Acked Write 7 Done Done Done true
7699 3105 Acked Read 10 Done Done Done true
7700 3106 Acked Read 3 Done Done Done true
7701 3107 Acked Write 3 Done Done Done true
7702 3108 Acked Write 8 Done Done Done true
7703 3109 Acked Read 5 Done Done Done true
7704 3110 Acked Write 5 Done Done Done true
7705 3111 Acked Read 8 Done Done Done true
7706 3112 Acked Read 2 Done Done Done true
7707 STATES DS:0 DS:1 DS:2 TOTAL
7708 Sent 0 0 0 0
7709 Done 31 31 31 93
7710 Skipped 0 0 0 0
7711 Error 0 0 0 0
7712 Last Flush: Some(JobId(3081)) Some(JobId(3081)) Some(JobId(3081))
7713 Downstairs last five completed: 3081 3080 3079 3078 3077
7714 Upstairs last five completed: 3112 3111 3110 3109 3108
7715 CLIENT: Up:0 ds:31 act:3
7716 ----------------------------------------------------------------
7717 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7718 Downstairs last five completed: 3113 3112 3111 3110 3109
7719 Upstairs last five completed: 3113 3112 3111 3110 3109
7720 CLIENT: Up:0 ds:0 act:3
7721 Replay: All jobs finished
7722 Main task has work to do, go find it
7723 got dsc Stop(2)
7724 stop 2
7725 Feb 25 17:19:55.680 INFO request completed, latency_us: 397, response_code: 204, uri: [8830] Got stop action so:false kr:false
7726 /stop/cid/2, method: POST, req_id: 54a306c4-c037-437f-ab92-8e3b00e226a3, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
77272025-02-25T17:19:55.681ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
7728 Feb 25 17:19:55.681 INFO request completed, latency_us: 122, response_code: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:19:55.68179622Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
7729 200, uri: /state/cid/2, method: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:19:55.681832434Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
7730 GET, req_id: c0b0e278-e6f9-410a-afa0-994b5373e514, remote_addr: 127.0.0.1:52576, local_addr{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T17:19:55.681929151Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
7731 : 127.0.0.1:9998
7732 [8830] Exited with: ExitStatus(unix_wait_status(9))
7733 [8830][2] reports Exit
7734 Feb 25 17:19:59.684 INFO request completed, latency_us: 216, response_code: 200, uri: /state/cid/2, method: GET, req_id: 16cc0343-9ed9-4c7a-88bc-2109695fae2e, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
7735 001/300 Flush
7736 002/300 Read block 2602 len 12288
7737 003/300 Read block 2836 len 24576
7738 004/300 Flush
7739 005/300 Read block 527 len 32768
7740 006/300 Write block 2394 len 24576 data: 2 2 2 2 2 2
7741 007/300 Read block 3140 len 36864
7742 008/300 Write block 3182 len 16384 data: 2 2 2 2
7743 009/300 Write block 4602 len 16384 data: 3 3 3 2
7744 010/300 Flush
7745 011/300 Read block 4237 len 32768
7746 012/300 Read block 1342 len 16384
7747 013/300 Write block 1357 len 4096 data: 2
7748 014/300 Read block 4531 len 32768
7749 015/300 Read block 4887 len 28672
7750 016/300 Flush
7751 017/300 Write block 681 len 32768 data: 3 3 3 3 3 3 2 3
7752 018/300 Write block 3417 len 4096 data: 3
7753 019/300 Write block 3046 len 24576 data: 4 4 4 4 4 3
7754 020/300 Read block 529 len 28672
7755 021/300 Read block 2932 len 40960
7756 022/300 Read block 3672 len 12288
7757 023/300 Write block 4461 len 36864 data: 3 3 4 4 3 3 3 3 2
7758 024/300 Write block 3839 len 12288 data: 2 2 2
7759 025/300 Flush
7760 026/300 Flush
7761 027/300 Write block 983 len 16384 data: 4 3 3 3
7762 028/300 Write block 2628 len 12288 data: 4 4 4
7763 029/300 Write block 2536 len 28672 data: 2 2 3 3 3 4 4
7764 030/300 Write block 2240 len 16384 data: 2 2 2 2
7765 031/300 Read block 3330 len 4096
7766 032/300 Write block 1052 len 16384 data: 2 3 3 3
7767 033/300 Flush
7768 034/300 Write block 796 len 16384 data: 2 2 2 2
7769 035/300 Read block 3132 len 4096
7770 036/300 Write block 1067 len 36864 data: 3 4 4 4 4 3 3 3 3
7771 037/300 Read block 3279 len 28672
7772 038/300 Read block 4116 len 32768
7773 039/300 Read block 3345 len 40960
7774 040/300 Read block 3862 len 16384
7775 041/300 Write block 1405 len 20480 data: 3 2 2 2 2
7776 042/300 Write block 2406 len 12288 data: 2 2 2
7777 043/300 Write block 2460 len 32768 data: 6 6 5 3 2 2 2 2
7778 044/300 Write block 4243 len 40960 data: 4 4 4 4 2 2 3 3 2 2
7779 045/300 Write block 52 len 4096 data: 3
7780 046/300 Write block 531 len 24576 data: 2 2 3 3 3 3
7781 047/300 Read block 3669 len 16384
7782 048/300 Read block 1208 len 12288
7783 049/300 Read block 1855 len 28672
7784 050/300 Read block 423 len 36864
7785 051/300 Flush
7786 052/300 Write block 1613 len 12288 data: 4 3 3
7787 053/300 Flush
7788 054/300 Write block 228 len 24576 data: 2 2 2 2 2 2
7789 055/300 Flush
7790 056/300 Write block 4222 len 28672 data: 2 2 3 3 2 3 3
7791 057/300 Write block 850 len 32768 data: 2 2 2 2 2 3 3 3
7792 058/300 Flush
7793 059/300 Read block 1340 len 12288
7794 060/300 Write block 3213 len 40960 data: 3 2 2 2 2 2 2 2 2 2
7795 061/300 Read block 2016 len 16384
7796 062/300 Read block 4770 len 8192
7797 063/300 Write block 1422 len 4096 data: 2
7798 064/300 Read block 1973 len 16384
7799 065/300 Read block 233 len 40960
7800 066/300 Write block 1247 len 8192 data: 3 3
7801 067/300 Read block 4277 len 32768
7802 068/300 Read block 894 len 28672
7803 069/300 Read block 122 len 24576
7804 070/300 Read block 2918 len 32768
7805 071/300 Write block 3341 len 4096 data: 2
7806 072/300 Write block 2527 len 8192 data: 2 3
7807 073/300 Read block 954 len 28672
7808 074/300 Write block 4554 len 16384 data: 2 2 2 3
7809 075/300 Write block 1489 len 20480 data: 3 3 2 2 2
7810 076/300 Flush
7811 077/300 Write block 893 len 28672 data: 4 4 4 2 3 3 3
7812 078/300 Read block 1981 len 8192
7813 079/300 Write block 3654 len 24576 data: 3 3 3 3 4 4
7814 080/300 Write block 4515 len 32768 data: 4 3 3 3 3 3 2 2
7815 081/300 Read block 400 len 36864
7816 082/300 Read block 3827 len 12288
7817 083/300 Write block 3947 len 4096 data: 5
7818 084/300 Write block 154 len 20480 data: 2 2 2 3 3
7819 085/300 Write block 4663 len 16384 data: 5 5 4 5
7820 086/300 Write block 1893 len 36864 data: 2 2 2 2 2 2 3 3 3
7821 087/300 Read block 3634 len 40960
7822 088/300 Read block 949 len 8192
7823 089/300 Write block 129 len 16384 data: 3 3 2 2
7824 090/300 Read block 1049 len 32768
7825 091/300 Read block 2158 len 20480
7826 092/300 Write block 2830 len 20480 data: 3 3 3 3 3
7827 093/300 Read block 3756 len 36864
7828 094/300 Read block 1872 len 36864
7829 095/300 Read block 2527 len 32768
7830 096/300 Write block 4353 len 36864 data: 3 3 5 5 3 3 2 2 2
7831 097/300 Read block 1233 len 36864
7832 098/300 Write block 2197 len 28672 data: 3 3 4 4 3 3 3
7833 099/300 Read block 868 len 8192
7834 100/300 Write block 4954 len 4096 data: 2
7835 101/300 Read block 3729 len 20480
7836 102/300 Write block 4822 len 16384 data: 3 3 3 3
7837 103/300 Read block 4548 len 40960
7838 104/300 Write block 3365 len 28672 data: 2 2 2 2 2 2 2
7839 105/300 Read block 3060 len 16384
7840 106/300 Flush
7841 107/300 Read block 3440 len 8192
7842 108/300 Write block 1511 len 36864 data: 2 2 2 2 2 2 2 2 3
7843 109/300 Write block 1109 len 12288 data: 3 4 4
7844 110/300 Read block 101 len 40960
7845 111/300 Write block 3448 len 40960 data: 3 3 3 2 2 2 2 2 2 2
7846 112/300 Write block 3492 len 28672 data: 4 3 2 2 2 2 2
7847 113/300 Flush
7848 114/300 Write block 762 len 16384 data: 2 2 2 2
7849 115/300 Read block 577 len 32768
7850 116/300 Write block 2489 len 20480 data: 2 2 2 2 2
7851 117/300 Write block 688 len 8192 data: 4 3
7852 118/300 Read block 2987 len 36864
7853 119/300 Read block 2538 len 16384
7854 120/300 Write block 4456 len 4096 data: 2
7855 121/300 Read block 2617 len 20480
7856 122/300 Write block 1996 len 28672 data: 2 2 3 2 2 2 2
7857 123/300 Read block 2204 len 36864
7858 124/300 Write block 1963 len 4096 data: 2
7859 125/300 Read block 693 len 32768
7860 126/300 Read block 3167 len 28672
7861 127/300 Read block 1372 len 20480
7862 128/300 Read block 2419 len 24576
7863 129/300 Read block 3867 len 36864
7864 130/300 Read block 3892 len 16384
7865 131/300 Write block 1779 len 8192 data: 3 2
7866 132/300 Write block 2894 len 40960 data: 3 3 3 3 3 3 4 5 6 6
7867 133/300 Read block 3230 len 36864
7868 134/300 Read block 2430 len 16384
7869 135/300 Write block 4844 len 40960 data: 3 3 4 3 3 4 4 3 3 2
7870 136/300 Write block 1602 len 32768 data: 3 3 3 3 4 4 4 4
7871 137/300 Write block 4630 len 24576 data: 4 4 4 4 5 4
7872 138/300 Read block 2962 len 4096
7873 139/300 Write block 2565 len 8192 data: 2 2
7874 140/300 Flush
7875 141/300 Flush
7876 142/300 Write block 4812 len 40960 data: 5 5 5 4 4 3 3 3 3 3
7877 143/300 Read block 1577 len 4096
7878 144/300 Write block 4042 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7879 145/300 Read block 4752 len 24576
7880 146/300 Read block 2780 len 24576
7881 147/300 Read block 479 len 16384
7882 148/300 Write block 1772 len 24576 data: 4 4 4 2 2 2
7883 149/300 Write block 345 len 4096 data: 2
7884 150/300 Flush
7885 151/300 Flush
7886 152/300 Read block 2489 len 16384
7887 153/300 Read block 3920 len 36864
7888 154/300 Flush
7889 155/300 Flush
7890 156/300 Write block 3244 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7891 157/300 Write block 2359 len 36864 data: 2 2 2 2 2 2 2 2 2
7892 158/300 Write block 2601 len 12288 data: 4 3 3
7893 159/300 Flush
7894 160/300 Read block 236 len 40960
7895 161/300 Write block 2206 len 32768 data: 2 2 2 2 3 5 4 4
7896 162/300 Write block 3519 len 4096 data: 3
7897 163/300 Read block 2827 len 16384
7898 164/300 Read block 4242 len 4096
7899 165/300 Read block 3464 len 40960
7900 166/300 Write block 3116 len 12288 data: 3 3 4
7901 167/300 Read block 3126 len 32768
7902 168/300 Read block 3244 len 16384
7903 169/300 Read block 4755 len 12288
7904 170/300 Read block 2918 len 40960
7905 171/300 Write block 1722 len 4096 data: 4
7906 172/300 Read block 3600 len 4096
7907 173/300 Read block 1241 len 40960
7908 174/300 Write block 3206 len 28672 data: 4 3 2 2 3 3 3
7909 175/300 Read block 3786 len 16384
7910 176/300 Read block 4589 len 4096
7911 177/300 Read block 3210 len 4096
7912 178/300 Read block 1057 len 36864
7913 179/300 Flush
7914 180/300 Write block 1442 len 32768 data: 3 3 3 3 3 3 3 2
7915 181/300 Write block 4776 len 20480 data: 2 2 2 2 2
7916 182/300 Read block 4588 len 4096
7917 183/300 Write block 3002 len 4096 data: 6
7918 184/300 Read block 3822 len 40960
7919 185/300 Read block 4705 len 40960
7920 186/300 Read block 3713 len 24576
7921 187/300 Read block 1507 len 8192
7922 188/300 Write block 1440 len 28672 data: 4 3 4 4 4 4 4
7923 189/300 Read block 2616 len 16384
7924 190/300 Write block 1284 len 12288 data: 2 2 2
7925 191/300 Read block 1622 len 8192
7926 192/300 Read block 2551 len 40960
7927 193/300 Flush
7928 194/300 Write block 3002 len 36864 data: 7 5 5 5 5 4 4 3 3
7929 195/300 Read block 1840 len 8192
7930 196/300 Flush
7931 197/300 Write block 3497 len 4096 data: 3
7932 198/300 Write block 3066 len 32768 data: 2 2 3 3 3 3 2 2
7933 199/300 Write block 631 len 4096 data: 3
7934 200/300 Read block 276 len 16384
7935 201/300 Read block 943 len 8192
7936 202/300 Read block 2178 len 20480
7937 203/300 Read block 4395 len 40960
7938 204/300 Read block 4186 len 36864
7939 205/300 Read block 4272 len 8192
7940 206/300 Read block 2633 len 16384
7941 207/300 Flush
7942 208/300 Write block 2566 len 28672 data: 3 2 2 2 2 2 4
7943 209/300 Read block 1006 len 4096
7944 210/300 Read block 3237 len 24576
7945 211/300 Read block 4226 len 8192
7946 212/300 Flush
7947 213/300 Write block 2405 len 12288 data: 3 3 3
7948 214/300 Read block 2367 len 36864
7949 215/300 Read block 2313 len 40960
7950 216/300 Read block 1405 len 8192
7951 217/300 Write block 2037 len 8192 data: 4 4
7952 218/300 Read block 2340 len 8192
7953 219/300 Write block 2113 len 28672 data: 3 3 2 3 3 3 3
7954 220/300 Write block 4270 len 20480 data: 4 3 3 2 2
7955 221/300 Read block 4520 len 16384
7956 222/300 Read block 111 len 24576
7957 223/300 Write block 3837 len 24576 data: 3 2 3 3 3 2
7958 224/300 Read block 2022 len 4096
7959 225/300 Read block 4792 len 4096
7960 226/300 Write block 2314 len 16384 data: 2 2 2 2
7961 227/300 Write block 1031 len 4096 data: 3
7962 228/300 Read block 3420 len 32768
7963 229/300 Read block 3524 len 24576
7964 230/300 Read block 1985 len 4096
7965 231/300 Write block 678 len 32768 data: 4 4 3 4 4 4 4 4
7966 232/300 Read block 2953 len 8192
7967 233/300 Read block 4335 len 28672
7968 234/300 Write block 27 len 20480 data: 3 3 3 3 3
7969 235/300 Read block 4325 len 20480
7970 236/300 Write block 2499 len 28672 data: 2 2 2 2 2 2 3
7971 237/300 Read block 2249 len 4096
7972 238/300 Read block 965 len 20480
7973 239/300 Read block 4893 len 12288
7974 240/300 Read block 3855 len 32768
7975 241/300 Read block 1019 len 12288
7976 242/300 Read block 2236 len 32768
7977 243/300 Read block 1664 len 28672
7978 244/300 Write block 2128 len 12288 data: 4 4 3
7979 245/300 Flush
7980 246/300 Read block 3453 len 40960
7981 247/300 Read block 298 len 36864
7982 248/300 Write block 2348 len 36864 data: 2 2 2 3 4 3 3 4 3
7983 249/300 Read block 3670 len 20480
7984 250/300 Write block 3745 len 20480 data: 2 2 2 2 2
7985 251/300 Write block 1381 len 12288 data: 2 4 4
7986 252/300 Read block 487 len 16384
7987 253/300 Write block 3217 len 12288 data: 3 3 3
7988 254/300 Write block 478 len 8192 data: 2 2
7989 255/300 Write block 4817 len 36864 data: 4 4 4 4 4 4 4 4 4
7990 256/300 Write block 1434 len 20480 data: 2 2 3 3 3
7991 257/300 Read block 416 len 12288
7992 258/300 Read block 3003 len 16384
7993 259/300 Read block 4617 len 20480
7994 260/300 Read block 4448 len 8192
7995 261/300 Write block 4459 len 40960 data: 3 3 4 4 5 5 4 4 4 4
7996 262/300 Read block 4345 len 40960
7997 263/300 Read block 2655 len 20480
7998 264/300 Read block 1368 len 16384
7999 265/300 Read block 794 len 12288
8000 266/300 Write block 4487 len 12288 data: 2 3 4
8001 267/300 Read block 2295 len 36864
8002 268/300 Write block 727 len 16384 data: 2 3 3 3
8003 269/300 Write block 2967 len 36864 data: 6 6 6 4 2 2 2 3 3
8004 270/300 Write block 761 len 32768 data: 2 3 3 3 3 2 2 2
8005 271/300 Write block 1157 len 40960 data: 4 4 3 2 2 2 2 2 2 3
8006 272/300 Read block 3051 len 16384
8007 273/300 Flush
8008 274/300 Write block 1616 len 20480 data: 3 3 3 3 3
8009 275/300 Read block 1586 len 20480
8010 276/300 Write block 2517 len 16384 data: 4 5 3 3
8011 277/300 Read block 459 len 40960
8012 278/300 Write block 1111 len 4096 data: 5
8013 279/300 Read block 3062 len 36864
8014 280/300 Read block 918 len 4096
8015 281/300 Read block 4860 len 32768
8016 282/300 Write block 696 len 16384 data: 2 2 2 2
8017 283/300 Read block 2263 len 12288
8018 284/300 Write block 2575 len 8192 data: 3 3
8019 285/300 Read block 3708 len 8192
8020 286/300 Write block 4612 len 32768 data: 2 3 3 3 3 3 3 4
8021 287/300 Read block 3229 len 40960
8022 288/300 Read block 3209 len 8192
8023 289/300 Read block 2698 len 28672
8024 290/300 Read block 3196 len 28672
8025 291/300 Write block 3990 len 20480 data: 2 4 3 3 3
8026 292/300 Read block 3949 len 32768
8027 293/300 Read block 501 len 36864
8028 294/300 Write block 3561 len 12288 data: 3 3 3
8029 295/300 Read block 1286 len 20480
8030 296/300 Read block 3124 len 12288
8031 297/300 Read block 1008 len 40960
8032 298/300 Read block 377 len 36864
8033 299/300 Write block 1906 len 12288 data: 3 3 4
8034 300/300 Read block 1129 len 4096
8035 301/300 Read block 2549 len 36864
8036 Main task has work to do, go find it
8037 got dsc Start(2)
8038 start 2
8039 Feb 25 17:19:59.942 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/2, method: POST, req_id: ba574631-349d-4901-90f7-0246dcb6e512, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
8040 Replay: started 2, returned:Ok(())
8041 ----------------------------------------------------------------
8042 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8043 States: Active Active Offline
8044 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8045 3114 Acked Flush 0 Done Done Sent false
8046 3115 Acked Read 3 Done Done Sent false
8047 3116 Acked Read 6 Done Done Sent false
8048 3117 Acked Flush 0 Done Done Sent false
8049 3118 Acked Read 8 Done Done Sent false
8050 3119 Acked Write 6 Done Done Sent false
8051 3120 Acked Read 9 Done Done Sent false
8052 3121 Acked Write 4 Done Done Sent false
8053 3122 Acked Write 4 Done Done Sent false
8054 3123 Acked Flush 0 Done Done Sent false
8055 3124 Acked Read 8 Done Done Sent false
8056 3125 Acked Read 4 Done Done Sent false
8057 3126 Acked Write 1 Done Done Sent false
8058 3127 Acked Read 8 Done Done Sent false
8059 3128 Acked Read 7 Done Done Sent false
8060 3129 Acked Flush 0 Done Done Sent false
8061 3130 Acked Write 8 Done Done Sent false
8062 3131 Acked Write 1 Done Done Sent false
8063 3132 Acked Write 6 Done Done Sent false
8064 3133 Acked Read 7 Done Done Sent false
8065 3134 Acked Read 10 Done Done Sent false
8066 3135 Acked Read 3 Done Done Sent false
8067 3136 Acked Write 9 Done Done Sent false
8068 3137 Acked Write 3 Done Done Sent false
8069 3138 Acked Flush 0 Done Done Sent false
8070 3139 Acked Flush 0 Done Done Sent false
8071 3140 Acked Write 4 Done Done Sent false
8072 3141 Acked Write 3 Done Done Sent false
8073 3142 Acked Write 7 Done Done Sent false
8074 3143 Acked Write 4 Done Done Sent false
8075 3144 Acked Read 1 Done Done Sent false
8076 3145 Acked Write 4 Done Done Sent false
8077 3146 Acked Flush 0 Done Done Sent false
8078 3147 Acked Write 4 Done Done Sent false
8079 3148 Acked Read 1 Done Done Sent false
8080 3149 Acked Write 9 Done Done Sent false
8081 3150 Acked Read 7 Done Done Sent false
8082 3151 Acked Read 8 Done Done Sent false
8083 3152 Acked Read 10 Done Done Sent false
8084 3153 Acked Read 4 Done Done Sent false
8085 3154 Acked Write 5 Done Done Sent false
8086 3155 Acked Write 3 Done Done Sent false
8087 3156 Acked Write 8 Done Done Sent false
8088 3157 Acked Write 10 Done Done Sent false
8089 3158 Acked Write 1 Done Done Sent false
8090 3159 Acked Write 6 Done Done Sent false
8091 3160 Acked Read 4 Done Done Sent false
8092 3161 Acked Read 3 Done Done Sent false
8093 3162 Acked Read 7 Done Done Sent false
8094 3163 Acked Read 9 Done Done Sent false
8095 3164 Acked Flush 0 Done Done Sent false
8096 3165 Acked Write 3 Done Done Sent false
8097 3166 Acked Flush 0 Done Done Sent false
8098 3167 Acked Write 6 Done Done Sent false
8099 3168 Acked Flush 0 Done Done Sent false
8100 3169 Acked Write 7 Done Done Sent false
8101 3170 Acked Write 8 Done Done Sent false
8102 3171 Acked Flush 0 Done Done Sent false
8103 3172 Acked Read 3 Done Done Sent false
8104 3173 Acked Write 10 Done Done Sent false
8105 3174 Acked Read 4 Done Done Sent false
8106 3175 Acked Read 2 Done Done Sent false
8107 3176 Acked Write 1 Done Done Sent false
8108 3177 Acked Read 4 Done Done Sent false
8109 3178 Acked Read 10 Done Done Sent false
8110 3179 Acked Write 2 Done Done Sent false
8111 3180 Acked Read 8 Done Done Sent false
8112 3181 Acked Read 7 Done Done Sent false
8113 3182 Acked Read 6 Done Done Sent false
8114 3183 Acked Read 8 Done Done Sent false
8115 3184 Acked Write 1 Done Done Sent false
8116 3185 Acked Write 2 Done Done Sent false
8117 3186 Acked Read 7 Done Done Sent false
8118 3187 Acked Write 4 Done Done Sent false
8119 3188 Acked Write 5 Done Done Sent false
8120 3189 Acked Flush 0 Done Done Sent false
8121 3190 Acked Write 7 Done Done Sent false
8122 3191 Acked Read 2 Done Done Sent false
8123 3192 Acked Write 6 Done Done Sent false
8124 3193 Acked Write 8 Done Done Sent false
8125 3194 Acked Read 9 Done Done Sent false
8126 3195 Acked Read 3 Done Done Sent false
8127 3196 Acked Write 1 Done Done Sent false
8128 3197 Acked Write 5 Done Done Sent false
8129 3198 Acked Write 4 Done Done Sent false
8130 3199 Acked Write 9 Done Done Sent false
8131 3200 Acked Read 10 Done Done Sent false
8132 3201 Acked Read 2 Done Done Sent false
8133 3202 Acked Write 4 Done Done Sent false
8134 3203 Acked Read 8 Done Done Sent false
8135 3204 Acked Read 5 Done Done Sent false
8136 3205 Acked Write 5 Done Done Sent false
8137 3206 Acked Read 9 Done Done Sent false
8138 3207 Acked Read 9 Done Done Sent false
8139 3208 Acked Read 8 Done Done Sent false
8140 3209 Acked Write 9 Done Done Sent false
8141 3210 Acked Read 9 Done Done Sent false
8142 3211 Acked Write 7 Done Done Sent false
8143 3212 Acked Read 2 Done Done Sent false
8144 3213 Acked Write 1 Done Done Sent false
8145 3214 Acked Read 5 Done Done Sent false
8146 3215 Acked Write 4 Done Done Sent false
8147 3216 Acked Read 10 Done Done Sent false
8148 3217 Acked Write 7 Done Done Sent false
8149 3218 Acked Read 4 Done Done Sent false
8150 3219 Acked Flush 0 Done Done Sent false
8151 3220 Acked Read 2 Done Done Sent false
8152 3221 Acked Write 9 Done Done Sent false
8153 3222 Acked Write 3 Done Done Sent false
8154 3223 Acked Read 10 Done Done Sent false
8155 3224 Acked Write 10 Done Done Sent false
8156 3225 Acked Write 7 Done Done Sent false
8157 3226 Acked Flush 0 Done Done Sent false
8158 3227 Acked Write 4 Done Done Sent false
8159 3228 Acked Read 8 Done Done Sent false
8160 3229 Acked Write 5 Done Done Sent false
8161 3230 Acked Write 2 Done Done Sent false
8162 3231 Acked Read 9 Done Done Sent false
8163 3232 Acked Read 4 Done Done Sent false
8164 3233 Acked Write 1 Done Done Sent false
8165 3234 Acked Read 5 Done Done Sent false
8166 3235 Acked Write 7 Done Done Sent false
8167 3236 Acked Read 9 Done Done Sent false
8168 3237 Acked Write 1 Done Done Sent false
8169 3238 Acked Read 8 Done Done Sent false
8170 3239 Acked Read 7 Done Done Sent false
8171 3240 Acked Read 5 Done Done Sent false
8172 3241 Acked Read 6 Done Done Sent false
8173 3242 Acked Read 9 Done Done Sent false
8174 3243 Acked Read 4 Done Done Sent false
8175 3244 Acked Write 2 Done Done Sent false
8176 3245 Acked Write 10 Done Done Sent false
8177 3246 Acked Read 9 Done Done Sent false
8178 3247 Acked Read 4 Done Done Sent false
8179 3248 Acked Write 10 Done Done Sent false
8180 3249 Acked Write 8 Done Done Sent false
8181 3250 Acked Write 6 Done Done Sent false
8182 3251 Acked Read 1 Done Done Sent false
8183 3252 Acked Write 2 Done Done Sent false
8184 3253 Acked Flush 0 Done Done Sent false
8185 3254 Acked Flush 0 Done Done Sent false
8186 3255 Acked Write 10 Done Done Sent false
8187 3256 Acked Read 1 Done Done Sent false
8188 3257 Acked Write 10 Done Done Sent false
8189 3258 Acked Read 6 Done Done Sent false
8190 3259 Acked Read 6 Done Done Sent false
8191 3260 Acked Read 4 Done Done Sent false
8192 3261 Acked Write 6 Done Done Sent false
8193 3262 Acked Write 1 Done Done Sent false
8194 3263 Acked Flush 0 Done Done Sent false
8195 3264 Acked Flush 0 Done Done Sent false
8196 3265 Acked Read 4 Done Done Sent false
8197 3266 Acked Read 9 Done Done Sent false
8198 3267 Acked Flush 0 Done Done Sent false
8199 3268 Acked Flush 0 Done Done Sent false
8200 3269 Acked Write 10 Done Done Sent false
8201 3270 Acked Write 9 Done Done Sent false
8202 3271 Acked Write 3 Done Done Sent false
8203 3272 Acked Flush 0 Done Done Sent false
8204 3273 Acked Read 10 Done Done Sent false
8205 3274 Acked Write 8 Done Done Sent false
8206 3275 Acked Write 1 Done Done Sent false
8207 3276 Acked Read 4 Done Done Sent false
8208 3277 Acked Read 1 Done Done Sent false
8209 3278 Acked Read 10 Done Done Sent false
8210 3279 Acked Write 3 Done Done Sent false
8211 3280 Acked Read 8 Done Done Sent false
8212 3281 Acked Read 4 Done Done Sent false
8213 3282 Acked Read 3 Done Done Sent false
8214 3283 Acked Read 10 Done Done Sent false
8215 3284 Acked Write 1 Done Done Sent false
8216 3285 Acked Read 1 Done Done Sent false
8217 3286 Acked Read 10 Done Done Sent false
8218 3287 Acked Write 7 Done Done Sent false
8219 3288 Acked Read 4 Done Done Sent false
8220 3289 Acked Read 1 Done Done Sent false
8221 3290 Acked Read 1 Done Done Sent false
8222 3291 Acked Read 9 Done Done Sent false
8223 3292 Acked Flush 0 Done Done Sent false
8224 3293 Acked Write 8 Done Done Sent false
8225 3294 Acked Write 5 Done Done Sent false
8226 3295 Acked Read 1 Done Done Sent false
8227 3296 Acked Write 1 Done Done Sent false
8228 3297 Acked Read 10 Done Done Sent false
8229 3298 Acked Read 10 Done Done Sent false
8230 3299 Acked Read 6 Done Done Sent false
8231 3300 Acked Read 2 Done Done Sent false
8232 3301 Acked Write 7 Done Done Sent false
8233 3302 Acked Read 4 Done Done Sent false
8234 3303 Acked Write 3 Done Done Sent false
8235 3304 Acked Read 2 Done Done Sent false
8236 3305 Acked Read 10 Done Done Sent false
8237 3306 Acked Flush 0 Done Done Sent false
8238 3307 Acked Write 9 Done Done Sent false
8239 3308 Acked Read 2 Done Done Sent false
8240 3309 Acked Flush 0 Done Done Sent false
8241 3310 Acked Write 1 Done Done Sent false
8242 3311 Acked Write 8 Done Done Sent false
8243 3312 Acked Write 1 Done Done Sent false
8244 3313 Acked Read 4 Done Done Sent false
8245 3314 Acked Read 2 Done Done Sent false
8246 3315 Acked Read 5 Done Done Sent false
8247 3316 Acked Read 10 Done Done Sent false
8248 3317 Acked Read 9 Done Done Sent false
8249 3318 Acked Read 2 Done Done Sent false
8250 3319 Acked Read 4 Done Done Sent false
8251 3320 Acked Flush 0 Done Done Sent false
8252 3321 Acked Write 7 Done Done Sent false
8253 3322 Acked Read 1 Done Done Sent false
8254 3323 Acked Read 6 Done Done Sent false
8255 3324 Acked Read 2 Done Done Sent false
8256 3325 Acked Flush 0 Done Done Sent false
8257 3326 Acked Write 3 Done Done Sent false
8258 3327 Acked Read 9 Done Done Sent false
8259 3328 Acked Read 10 Done Done Sent false
8260 3329 Acked Read 2 Done Done Sent false
8261 3330 Acked Write 2 Done Done Sent false
8262 3331 Acked Read 2 Done Done Sent false
8263 3332 Acked Write 7 Done Done Sent false
8264 3333 Acked Write 5 Done Done Sent false
8265 3334 Acked Read 4 Done Done Sent false
8266 3335 Acked Read 6 Done Done Sent false
8267 3336 Acked Write 6 Done Done Sent false
8268 3337 Acked Read 1 Done Done Sent false
8269 3338 Acked Read 1 Done Done Sent false
8270 3339 Acked Write 4 Done Done Sent false
8271 3340 Acked Write 1 Done Done Sent false
8272 3341 Acked Read 8 Done Done Sent false
8273 3342 Acked Read 6 Done Done Sent false
8274 3343 Acked Read 1 Done Done Sent false
8275 3344 Acked Write 8 Done Done Sent false
8276 3345 Acked Read 2 Done Done Sent false
8277 3346 Acked Read 7 Done Done Sent false
8278 3347 Acked Write 5 Done Done Sent false
8279 3348 Acked Read 5 Done Done Sent false
8280 3349 Acked Write 7 Done Done Sent false
8281 3350 Acked Read 1 Done Done Sent false
8282 3351 Acked Read 5 Done Done Sent false
8283 3352 Acked Read 3 Done Done Sent false
8284 3353 Acked Read 8 Done Done Sent false
8285 3354 Acked Read 3 Done Done Sent false
8286 3355 Acked Read 8 Done Done Sent false
8287 3356 Acked Read 7 Done Done Sent false
8288 3357 Acked Write 3 Done Done Sent false
8289 3358 Acked Flush 0 Done Done Sent false
8290 3359 Acked Read 10 Done Done Sent false
8291 3360 Acked Read 9 Done Done Sent false
8292 3361 Acked Write 9 Done Done Sent false
8293 3362 Acked Read 5 Done Done Sent false
8294 3363 Acked Write 5 Done Done Sent false
8295 3364 Acked Write 3 Done Done Sent false
8296 3365 Acked Read 4 Done Done Sent false
8297 3366 Acked Write 3 Done Done Sent false
8298 3367 Acked Write 2 Done Done Sent false
8299 3368 Acked Write 9 Done Done Sent false
8300 3369 Acked Write 5 Done Done Sent false
8301 3370 Acked Read 3 Done Done Sent false
8302 3371 Acked Read 4 Done Done Sent false
8303 3372 Acked Read 5 Done Done Sent false
8304 3373 Acked Read 2 Done Done Sent false
8305 3374 Acked Write 10 Done Done Sent false
8306 3375 Acked Read 10 Done Done Sent false
8307 3376 Acked Read 5 Done Done Sent false
8308 3377 Acked Read 4 Done Done Sent false
8309 3378 Acked Read 3 Done Done Sent false
8310 3379 Acked Write 3 Done Done Sent false
8311 3380 Acked Read 9 Done Done Sent false
8312 3381 Acked Write 4 Done Done Sent false
8313 3382 Acked Write 9 Done Done Sent false
8314 3383 Acked Write 8 Done Done Sent false
8315 3384 Acked Write 10 Done Done Sent false
8316 3385 Acked Read 4 Done Done Sent false
8317 3386 Acked Flush 0 Done Done Sent false
8318 3387 Acked Write 5 Done Done Sent false
8319 3388 Acked Read 5 Done Done Sent false
8320 3389 Acked Write 4 Done Done Sent false
8321 3390 Acked Read 10 Done Done Sent false
8322 3391 Acked Write 1 Done Done Sent false
8323 3392 Acked Read 9 Done Done Sent false
8324 3393 Acked Read 1 Done Done Sent false
8325 3394 Acked Read 8 Done Done Sent false
8326 3395 Acked Write 4 Done Done Sent false
8327 3396 Acked Read 3 Done Done Sent false
8328 3397 Acked Write 2 Done Done Sent false
8329 3398 Acked Read 2 Done Done Sent false
8330 3399 Acked Write 8 Done Done Sent false
8331 3400 Acked Read 10 Done Done Sent false
8332 3401 Acked Read 2 Done Done Sent false
8333 3402 Acked Read 7 Done Done Sent false
8334 3403 Acked Read 7 Done Done Sent false
8335 3404 Acked Write 5 Done Done Sent false
8336 3405 Acked Read 8 Done Done Sent false
8337 3406 Acked Read 9 Done Done Sent false
8338 3407 Acked Write 3 Done Done Sent false
8339 3408 Acked Read 5 Done Done Sent false
8340 3409 Acked Read 3 Done Done Sent false
8341 3410 Acked Read 10 Done Done Sent false
8342 3411 Acked Read 9 Done Done Sent false
8343 3412 Acked Write 3 Done Done Sent false
8344 3413 Acked Read 1 Done Done Sent false
8345 3414 Acked Read 9 Done Done Sent false
8346 STATES DS:0 DS:1 DS:2 TOTAL
8347 Sent 0 0 301 301
8348 Done 301 301 0 602
8349 Skipped 0 0 0 0
8350 Error 0 0 0 0
8351 Last Flush: Some(JobId(3386)) Some(JobId(3386)) Some(JobId(3113))
8352 Downstairs last five completed: 3113 3112 3111 3110 3109
8353 Upstairs last five completed: 3414 3413 3412 3411 3410
8354 CLIENT: Up:0 ds:301 act:2
83552025-02-25T17:20:00.453ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
83562025-02-25T17:20:01.454ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
83572025-02-25T17:20:02.455ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
83582025-02-25T17:20:03.457ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
8359 ----------------------------------------------------------------
8360 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8361 States: Active Active Offline
8362 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8363 3114 Acked Flush 0 Done Done Sent false
8364 3115 Acked Read 3 Done Done Sent false
8365 3116 Acked Read 6 Done Done Sent false
8366 3117 Acked Flush 0 Done Done Sent false
8367 3118 Acked Read 8 Done Done Sent false
8368 3119 Acked Write 6 Done Done Sent false
8369 3120 Acked Read 9 Done Done Sent false
8370 3121 Acked Write 4 Done Done Sent false
8371 3122 Acked Write 4 Done Done Sent false
8372 3123 Acked Flush 0 Done Done Sent false
8373 3124 Acked Read 8 Done Done Sent false
8374 3125 Acked Read 4 Done Done Sent false
8375 3126 Acked Write 1 Done Done Sent false
8376 3127 Acked Read 8 Done Done Sent false
8377 3128 Acked Read 7 Done Done Sent false
8378 3129 Acked Flush 0 Done Done Sent false
8379 3130 Acked Write 8 Done Done Sent false
8380 3131 Acked Write 1 Done Done Sent false
8381 3132 Acked Write 6 Done Done Sent false
8382 3133 Acked Read 7 Done Done Sent false
8383 3134 Acked Read 10 Done Done Sent false
8384 3135 Acked Read 3 Done Done Sent false
8385 3136 Acked Write 9 Done Done Sent false
8386 3137 Acked Write 3 Done Done Sent false
8387 3138 Acked Flush 0 Done Done Sent false
8388 3139 Acked Flush 0 Done Done Sent false
8389 3140 Acked Write 4 Done Done Sent false
8390 3141 Acked Write 3 Done Done Sent false
8391 3142 Acked Write 7 Done Done Sent false
8392 3143 Acked Write 4 Done Done Sent false
8393 3144 Acked Read 1 Done Done Sent false
8394 3145 Acked Write 4 Done Done Sent false
8395 3146 Acked Flush 0 Done Done Sent false
8396 3147 Acked Write 4 Done Done Sent false
8397 3148 Acked Read 1 Done Done Sent false
8398 3149 Acked Write 9 Done Done Sent false
8399 3150 Acked Read 7 Done Done Sent false
8400 3151 Acked Read 8 Done Done Sent false
8401 3152 Acked Read 10 Done Done Sent false
8402 3153 Acked Read 4 Done Done Sent false
8403 3154 Acked Write 5 Done Done Sent false
8404 3155 Acked Write 3 Done Done Sent false
8405 3156 Acked Write 8 Done Done Sent false
8406 3157 Acked Write 10 Done Done Sent false
8407 3158 Acked Write 1 Done Done Sent false
8408 3159 Acked Write 6 Done Done Sent false
8409 3160 Acked Read 4 Done Done Sent false
8410 3161 Acked Read 3 Done Done Sent false
8411 3162 Acked Read 7 Done Done Sent false
8412 3163 Acked Read 9 Done Done Sent false
8413 3164 Acked Flush 0 Done Done Sent false
8414 3165 Acked Write 3 Done Done Sent false
8415 3166 Acked Flush 0 Done Done Sent false
8416 3167 Acked Write 6 Done Done Sent false
8417 3168 Acked Flush 0 Done Done Sent false
8418 3169 Acked Write 7 Done Done Sent false
8419 3170 Acked Write 8 Done Done Sent false
8420 3171 Acked Flush 0 Done Done Sent false
8421 3172 Acked Read 3 Done Done Sent false
8422 3173 Acked Write 10 Done Done Sent false
8423 3174 Acked Read 4 Done Done Sent false
8424 3175 Acked Read 2 Done Done Sent false
8425 3176 Acked Write 1 Done Done Sent false
8426 3177 Acked Read 4 Done Done Sent false
8427 3178 Acked Read 10 Done Done Sent false
8428 3179 Acked Write 2 Done Done Sent false
8429 3180 Acked Read 8 Done Done Sent false
8430 3181 Acked Read 7 Done Done Sent false
8431 3182 Acked Read 6 Done Done Sent false
8432 3183 Acked Read 8 Done Done Sent false
8433 3184 Acked Write 1 Done Done Sent false
8434 3185 Acked Write 2 Done Done Sent false
8435 3186 Acked Read 7 Done Done Sent false
8436 3187 Acked Write 4 Done Done Sent false
8437 3188 Acked Write 5 Done Done Sent false
8438 3189 Acked Flush 0 Done Done Sent false
8439 3190 Acked Write 7 Done Done Sent false
8440 3191 Acked Read 2 Done Done Sent false
8441 3192 Acked Write 6 Done Done Sent false
8442 3193 Acked Write 8 Done Done Sent false
8443 3194 Acked Read 9 Done Done Sent false
8444 3195 Acked Read 3 Done Done Sent false
8445 3196 Acked Write 1 Done Done Sent false
8446 3197 Acked Write 5 Done Done Sent false
8447 3198 Acked Write 4 Done Done Sent false
8448 3199 Acked Write 9 Done Done Sent false
8449 3200 Acked Read 10 Done Done Sent false
8450 3201 Acked Read 2 Done Done Sent false
8451 3202 Acked Write 4 Done Done Sent false
8452 3203 Acked Read 8 Done Done Sent false
8453 3204 Acked Read 5 Done Done Sent false
8454 3205 Acked Write 5 Done Done Sent false
8455 3206 Acked Read 9 Done Done Sent false
8456 3207 Acked Read 9 Done Done Sent false
8457 3208 Acked Read 8 Done Done Sent false
8458 3209 Acked Write 9 Done Done Sent false
8459 3210 Acked Read 9 Done Done Sent false
8460 3211 Acked Write 7 Done Done Sent false
8461 3212 Acked Read 2 Done Done Sent false
8462 3213 Acked Write 1 Done Done Sent false
8463 3214 Acked Read 5 Done Done Sent false
8464 3215 Acked Write 4 Done Done Sent false
8465 3216 Acked Read 10 Done Done Sent false
8466 3217 Acked Write 7 Done Done Sent false
8467 3218 Acked Read 4 Done Done Sent false
8468 3219 Acked Flush 0 Done Done Sent false
8469 3220 Acked Read 2 Done Done Sent false
8470 3221 Acked Write 9 Done Done Sent false
8471 3222 Acked Write 3 Done Done Sent false
8472 3223 Acked Read 10 Done Done Sent false
8473 3224 Acked Write 10 Done Done Sent false
8474 3225 Acked Write 7 Done Done Sent false
8475 3226 Acked Flush 0 Done Done Sent false
8476 3227 Acked Write 4 Done Done Sent false
8477 3228 Acked Read 8 Done Done Sent false
8478 3229 Acked Write 5 Done Done Sent false
8479 3230 Acked Write 2 Done Done Sent false
8480 3231 Acked Read 9 Done Done Sent false
8481 3232 Acked Read 4 Done Done Sent false
8482 3233 Acked Write 1 Done Done Sent false
8483 3234 Acked Read 5 Done Done Sent false
8484 3235 Acked Write 7 Done Done Sent false
8485 3236 Acked Read 9 Done Done Sent false
8486 3237 Acked Write 1 Done Done Sent false
8487 3238 Acked Read 8 Done Done Sent false
8488 3239 Acked Read 7 Done Done Sent false
8489 3240 Acked Read 5 Done Done Sent false
8490 3241 Acked Read 6 Done Done Sent false
8491 3242 Acked Read 9 Done Done Sent false
8492 3243 Acked Read 4 Done Done Sent false
8493 3244 Acked Write 2 Done Done Sent false
8494 3245 Acked Write 10 Done Done Sent false
8495 3246 Acked Read 9 Done Done Sent false
8496 3247 Acked Read 4 Done Done Sent false
8497 3248 Acked Write 10 Done Done Sent false
8498 3249 Acked Write 8 Done Done Sent false
8499 3250 Acked Write 6 Done Done Sent false
8500 3251 Acked Read 1 Done Done Sent false
8501 3252 Acked Write 2 Done Done Sent false
8502 3253 Acked Flush 0 Done Done Sent false
8503 3254 Acked Flush 0 Done Done Sent false
8504 3255 Acked Write 10 Done Done Sent false
8505 3256 Acked Read 1 Done Done Sent false
8506 3257 Acked Write 10 Done Done Sent false
8507 3258 Acked Read 6 Done Done Sent false
8508 3259 Acked Read 6 Done Done Sent false
8509 3260 Acked Read 4 Done Done Sent false
8510 3261 Acked Write 6 Done Done Sent false
8511 3262 Acked Write 1 Done Done Sent false
8512 3263 Acked Flush 0 Done Done Sent false
8513 3264 Acked Flush 0 Done Done Sent false
8514 3265 Acked Read 4 Done Done Sent false
8515 3266 Acked Read 9 Done Done Sent false
8516 3267 Acked Flush 0 Done Done Sent false
8517 3268 Acked Flush 0 Done Done Sent false
8518 3269 Acked Write 10 Done Done Sent false
8519 3270 Acked Write 9 Done Done Sent false
8520 3271 Acked Write 3 Done Done Sent false
8521 3272 Acked Flush 0 Done Done Sent false
8522 3273 Acked Read 10 Done Done Sent false
8523 3274 Acked Write 8 Done Done Sent false
8524 3275 Acked Write 1 Done Done Sent false
8525 3276 Acked Read 4 Done Done Sent false
8526 3277 Acked Read 1 Done Done Sent false
8527 3278 Acked Read 10 Done Done Sent false
8528 3279 Acked Write 3 Done Done Sent false
8529 3280 Acked Read 8 Done Done Sent false
8530 3281 Acked Read 4 Done Done Sent false
8531 3282 Acked Read 3 Done Done Sent false
8532 3283 Acked Read 10 Done Done Sent false
8533 3284 Acked Write 1 Done Done Sent false
8534 3285 Acked Read 1 Done Done Sent false
8535 3286 Acked Read 10 Done Done Sent false
8536 3287 Acked Write 7 Done Done Sent false
8537 3288 Acked Read 4 Done Done Sent false
8538 3289 Acked Read 1 Done Done Sent false
8539 3290 Acked Read 1 Done Done Sent false
8540 3291 Acked Read 9 Done Done Sent false
8541 3292 Acked Flush 0 Done Done Sent false
8542 3293 Acked Write 8 Done Done Sent false
8543 3294 Acked Write 5 Done Done Sent false
8544 3295 Acked Read 1 Done Done Sent false
8545 3296 Acked Write 1 Done Done Sent false
8546 3297 Acked Read 10 Done Done Sent false
8547 3298 Acked Read 10 Done Done Sent false
8548 3299 Acked Read 6 Done Done Sent false
8549 3300 Acked Read 2 Done Done Sent false
8550 3301 Acked Write 7 Done Done Sent false
8551 3302 Acked Read 4 Done Done Sent false
8552 3303 Acked Write 3 Done Done Sent false
8553 3304 Acked Read 2 Done Done Sent false
8554 3305 Acked Read 10 Done Done Sent false
8555 3306 Acked Flush 0 Done Done Sent false
8556 3307 Acked Write 9 Done Done Sent false
8557 3308 Acked Read 2 Done Done Sent false
8558 3309 Acked Flush 0 Done Done Sent false
8559 3310 Acked Write 1 Done Done Sent false
8560 3311 Acked Write 8 Done Done Sent false
8561 3312 Acked Write 1 Done Done Sent false
8562 3313 Acked Read 4 Done Done Sent false
8563 3314 Acked Read 2 Done Done Sent false
8564 3315 Acked Read 5 Done Done Sent false
8565 3316 Acked Read 10 Done Done Sent false
8566 3317 Acked Read 9 Done Done Sent false
8567 3318 Acked Read 2 Done Done Sent false
8568 3319 Acked Read 4 Done Done Sent false
8569 3320 Acked Flush 0 Done Done Sent false
8570 3321 Acked Write 7 Done Done Sent false
8571 3322 Acked Read 1 Done Done Sent false
8572 3323 Acked Read 6 Done Done Sent false
8573 3324 Acked Read 2 Done Done Sent false
8574 3325 Acked Flush 0 Done Done Sent false
8575 3326 Acked Write 3 Done Done Sent false
8576 3327 Acked Read 9 Done Done Sent false
8577 3328 Acked Read 10 Done Done Sent false
8578 3329 Acked Read 2 Done Done Sent false
8579 3330 Acked Write 2 Done Done Sent false
8580 3331 Acked Read 2 Done Done Sent false
8581 3332 Acked Write 7 Done Done Sent false
8582 3333 Acked Write 5 Done Done Sent false
8583 3334 Acked Read 4 Done Done Sent false
8584 3335 Acked Read 6 Done Done Sent false
8585 3336 Acked Write 6 Done Done Sent false
8586 3337 Acked Read 1 Done Done Sent false
8587 3338 Acked Read 1 Done Done Sent false
8588 3339 Acked Write 4 Done Done Sent false
8589 3340 Acked Write 1 Done Done Sent false
8590 3341 Acked Read 8 Done Done Sent false
8591 3342 Acked Read 6 Done Done Sent false
8592 3343 Acked Read 1 Done Done Sent false
8593 3344 Acked Write 8 Done Done Sent false
8594 3345 Acked Read 2 Done Done Sent false
8595 3346 Acked Read 7 Done Done Sent false
8596 3347 Acked Write 5 Done Done Sent false
8597 3348 Acked Read 5 Done Done Sent false
8598 3349 Acked Write 7 Done Done Sent false
8599 3350 Acked Read 1 Done Done Sent false
8600 3351 Acked Read 5 Done Done Sent false
8601 3352 Acked Read 3 Done Done Sent false
8602 3353 Acked Read 8 Done Done Sent false
8603 3354 Acked Read 3 Done Done Sent false
8604 3355 Acked Read 8 Done Done Sent false
8605 3356 Acked Read 7 Done Done Sent false
8606 3357 Acked Write 3 Done Done Sent false
8607 3358 Acked Flush 0 Done Done Sent false
8608 3359 Acked Read 10 Done Done Sent false
8609 3360 Acked Read 9 Done Done Sent false
8610 3361 Acked Write 9 Done Done Sent false
8611 3362 Acked Read 5 Done Done Sent false
8612 3363 Acked Write 5 Done Done Sent false
8613 3364 Acked Write 3 Done Done Sent false
8614 3365 Acked Read 4 Done Done Sent false
8615 3366 Acked Write 3 Done Done Sent false
8616 3367 Acked Write 2 Done Done Sent false
8617 3368 Acked Write 9 Done Done Sent false
8618 3369 Acked Write 5 Done Done Sent false
8619 3370 Acked Read 3 Done Done Sent false
8620 3371 Acked Read 4 Done Done Sent false
8621 3372 Acked Read 5 Done Done Sent false
8622 3373 Acked Read 2 Done Done Sent false
8623 3374 Acked Write 10 Done Done Sent false
8624 3375 Acked Read 10 Done Done Sent false
8625 3376 Acked Read 5 Done Done Sent false
8626 3377 Acked Read 4 Done Done Sent false
8627 3378 Acked Read 3 Done Done Sent false
8628 3379 Acked Write 3 Done Done Sent false
8629 3380 Acked Read 9 Done Done Sent false
8630 3381 Acked Write 4 Done Done Sent false
8631 3382 Acked Write 9 Done Done Sent false
8632 3383 Acked Write 8 Done Done Sent false
8633 3384 Acked Write 10 Done Done Sent false
8634 3385 Acked Read 4 Done Done Sent false
8635 3386 Acked Flush 0 Done Done Sent false
8636 3387 Acked Write 5 Done Done Sent false
8637 3388 Acked Read 5 Done Done Sent false
8638 3389 Acked Write 4 Done Done Sent false
8639 3390 Acked Read 10 Done Done Sent false
8640 3391 Acked Write 1 Done Done Sent false
8641 3392 Acked Read 9 Done Done Sent false
8642 3393 Acked Read 1 Done Done Sent false
8643 3394 Acked Read 8 Done Done Sent false
8644 3395 Acked Write 4 Done Done Sent false
8645 3396 Acked Read 3 Done Done Sent false
8646 3397 Acked Write 2 Done Done Sent false
8647 3398 Acked Read 2 Done Done Sent false
8648 3399 Acked Write 8 Done Done Sent false
8649 3400 Acked Read 10 Done Done Sent false
8650 3401 Acked Read 2 Done Done Sent false
8651 3402 Acked Read 7 Done Done Sent false
8652 3403 Acked Read 7 Done Done Sent false
8653 3404 Acked Write 5 Done Done Sent false
8654 3405 Acked Read 8 Done Done Sent false
8655 3406 Acked Read 9 Done Done Sent false
8656 3407 Acked Write 3 Done Done Sent false
8657 3408 Acked Read 5 Done Done Sent false
8658 3409 Acked Read 3 Done Done Sent false
8659 3410 Acked Read 10 Done Done Sent false
8660 3411 Acked Read 9 Done Done Sent false
8661 3412 Acked Write 3 Done Done Sent false
8662 3413 Acked Read 1 Done Done Sent false
8663 3414 Acked Read 9 Done Done Sent false
8664 STATES DS:0 DS:1 DS:2 TOTAL
8665 Sent 0 0 301 301
8666 Done 301 301 0 602
8667 Skipped 0 0 0 0
8668 Error 0 0 0 0
8669 Last Flush: Some(JobId(3386)) Some(JobId(3386)) Some(JobId(3113))
8670 Downstairs last five completed: 3113 3112 3111 3110 3109
8671 Upstairs last five completed: 3414 3413 3412 3411 3410
8672 CLIENT: Up:0 ds:301 act:2
86732025-02-25T17:20:04.459ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
86742025-02-25T17:20:05.460ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
86752025-02-25T17:20:05.683ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
86762025-02-25T17:20:05.683ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
86772025-02-25T17:20:05.683ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
86782025-02-25T17:20:05.683ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
86792025-02-25T17:20:05.683ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
8680 [8830] got start action
8681 [8830] I am going to restart
8682 Starting downstairs at port 8830
8683 Make output file at "/tmp/dsc/downstairs-8830.txt"
8684 [8830][2] reports Starting
8685 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1216)
8686 [8830][2] reports Running
86872025-02-25T17:20:06.462ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
86882025-02-25T17:20:07.464ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
8689 ----------------------------------------------------------------
8690 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8691 States: Active Active Offline
8692 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8693 3114 Acked Flush 0 Done Done Sent false
8694 3115 Acked Read 3 Done Done Sent false
8695 3116 Acked Read 6 Done Done Sent false
8696 3117 Acked Flush 0 Done Done Sent false
8697 3118 Acked Read 8 Done Done Sent false
8698 3119 Acked Write 6 Done Done Sent false
8699 3120 Acked Read 9 Done Done Sent false
8700 3121 Acked Write 4 Done Done Sent false
8701 3122 Acked Write 4 Done Done Sent false
8702 3123 Acked Flush 0 Done Done Sent false
8703 3124 Acked Read 8 Done Done Sent false
8704 3125 Acked Read 4 Done Done Sent false
8705 3126 Acked Write 1 Done Done Sent false
8706 3127 Acked Read 8 Done Done Sent false
8707 3128 Acked Read 7 Done Done Sent false
8708 3129 Acked Flush 0 Done Done Sent false
8709 3130 Acked Write 8 Done Done Sent false
8710 3131 Acked Write 1 Done Done Sent false
8711 3132 Acked Write 6 Done Done Sent false
8712 3133 Acked Read 7 Done Done Sent false
8713 3134 Acked Read 10 Done Done Sent false
8714 3135 Acked Read 3 Done Done Sent false
8715 3136 Acked Write 9 Done Done Sent false
8716 3137 Acked Write 3 Done Done Sent false
8717 3138 Acked Flush 0 Done Done Sent false
8718 3139 Acked Flush 0 Done Done Sent false
8719 3140 Acked Write 4 Done Done Sent false
8720 3141 Acked Write 3 Done Done Sent false
8721 3142 Acked Write 7 Done Done Sent false
8722 3143 Acked Write 4 Done Done Sent false
8723 3144 Acked Read 1 Done Done Sent false
8724 3145 Acked Write 4 Done Done Sent false
8725 3146 Acked Flush 0 Done Done Sent false
8726 3147 Acked Write 4 Done Done Sent false
8727 3148 Acked Read 1 Done Done Sent false
8728 3149 Acked Write 9 Done Done Sent false
8729 3150 Acked Read 7 Done Done Sent false
8730 3151 Acked Read 8 Done Done Sent false
8731 3152 Acked Read 10 Done Done Sent false
8732 3153 Acked Read 4 Done Done Sent false
8733 3154 Acked Write 5 Done Done Sent false
8734 3155 Acked Write 3 Done Done Sent false
8735 3156 Acked Write 8 Done Done Sent false
8736 3157 Acked Write 10 Done Done Sent false
8737 3158 Acked Write 1 Done Done Sent false
8738 3159 Acked Write 6 Done Done Sent false
8739 3160 Acked Read 4 Done Done Sent false
8740 3161 Acked Read 3 Done Done Sent false
8741 3162 Acked Read 7 Done Done Sent false
8742 3163 Acked Read 9 Done Done Sent false
8743 3164 Acked Flush 0 Done Done Sent false
8744 3165 Acked Write 3 Done Done Sent false
8745 3166 Acked Flush 0 Done Done Sent false
8746 3167 Acked Write 6 Done Done Sent false
8747 3168 Acked Flush 0 Done Done Sent false
8748 3169 Acked Write 7 Done Done Sent false
8749 3170 Acked Write 8 Done Done Sent false
8750 3171 Acked Flush 0 Done Done Sent false
8751 3172 Acked Read 3 Done Done Sent false
8752 3173 Acked Write 10 Done Done Sent false
8753 3174 Acked Read 4 Done Done Sent false
8754 3175 Acked Read 2 Done Done Sent false
8755 3176 Acked Write 1 Done Done Sent false
8756 3177 Acked Read 4 Done Done Sent false
8757 3178 Acked Read 10 Done Done Sent false
8758 3179 Acked Write 2 Done Done Sent false
8759 3180 Acked Read 8 Done Done Sent false
8760 3181 Acked Read 7 Done Done Sent false
8761 3182 Acked Read 6 Done Done Sent false
8762 3183 Acked Read 8 Done Done Sent false
8763 3184 Acked Write 1 Done Done Sent false
8764 3185 Acked Write 2 Done Done Sent false
8765 3186 Acked Read 7 Done Done Sent false
8766 3187 Acked Write 4 Done Done Sent false
8767 3188 Acked Write 5 Done Done Sent false
8768 3189 Acked Flush 0 Done Done Sent false
8769 3190 Acked Write 7 Done Done Sent false
8770 3191 Acked Read 2 Done Done Sent false
8771 3192 Acked Write 6 Done Done Sent false
8772 3193 Acked Write 8 Done Done Sent false
8773 3194 Acked Read 9 Done Done Sent false
8774 3195 Acked Read 3 Done Done Sent false
8775 3196 Acked Write 1 Done Done Sent false
8776 3197 Acked Write 5 Done Done Sent false
8777 3198 Acked Write 4 Done Done Sent false
8778 3199 Acked Write 9 Done Done Sent false
8779 3200 Acked Read 10 Done Done Sent false
8780 3201 Acked Read 2 Done Done Sent false
8781 3202 Acked Write 4 Done Done Sent false
8782 3203 Acked Read 8 Done Done Sent false
8783 3204 Acked Read 5 Done Done Sent false
8784 3205 Acked Write 5 Done Done Sent false
8785 3206 Acked Read 9 Done Done Sent false
8786 3207 Acked Read 9 Done Done Sent false
8787 3208 Acked Read 8 Done Done Sent false
8788 3209 Acked Write 9 Done Done Sent false
8789 3210 Acked Read 9 Done Done Sent false
8790 3211 Acked Write 7 Done Done Sent false
8791 3212 Acked Read 2 Done Done Sent false
8792 3213 Acked Write 1 Done Done Sent false
8793 3214 Acked Read 5 Done Done Sent false
8794 3215 Acked Write 4 Done Done Sent false
8795 3216 Acked Read 10 Done Done Sent false
8796 3217 Acked Write 7 Done Done Sent false
8797 3218 Acked Read 4 Done Done Sent false
8798 3219 Acked Flush 0 Done Done Sent false
8799 3220 Acked Read 2 Done Done Sent false
8800 3221 Acked Write 9 Done Done Sent false
8801 3222 Acked Write 3 Done Done Sent false
8802 3223 Acked Read 10 Done Done Sent false
8803 3224 Acked Write 10 Done Done Sent false
8804 3225 Acked Write 7 Done Done Sent false
8805 3226 Acked Flush 0 Done Done Sent false
8806 3227 Acked Write 4 Done Done Sent false
8807 3228 Acked Read 8 Done Done Sent false
8808 3229 Acked Write 5 Done Done Sent false
8809 3230 Acked Write 2 Done Done Sent false
8810 3231 Acked Read 9 Done Done Sent false
8811 3232 Acked Read 4 Done Done Sent false
8812 3233 Acked Write 1 Done Done Sent false
8813 3234 Acked Read 5 Done Done Sent false
8814 3235 Acked Write 7 Done Done Sent false
8815 3236 Acked Read 9 Done Done Sent false
8816 3237 Acked Write 1 Done Done Sent false
8817 3238 Acked Read 8 Done Done Sent false
8818 3239 Acked Read 7 Done Done Sent false
8819 3240 Acked Read 5 Done Done Sent false
8820 3241 Acked Read 6 Done Done Sent false
8821 3242 Acked Read 9 Done Done Sent false
8822 3243 Acked Read 4 Done Done Sent false
8823 3244 Acked Write 2 Done Done Sent false
8824 3245 Acked Write 10 Done Done Sent false
8825 3246 Acked Read 9 Done Done Sent false
8826 3247 Acked Read 4 Done Done Sent false
8827 3248 Acked Write 10 Done Done Sent false
8828 3249 Acked Write 8 Done Done Sent false
8829 3250 Acked Write 6 Done Done Sent false
8830 3251 Acked Read 1 Done Done Sent false
8831 3252 Acked Write 2 Done Done Sent false
8832 3253 Acked Flush 0 Done Done Sent false
8833 3254 Acked Flush 0 Done Done Sent false
8834 3255 Acked Write 10 Done Done Sent false
8835 3256 Acked Read 1 Done Done Sent false
8836 3257 Acked Write 10 Done Done Sent false
8837 3258 Acked Read 6 Done Done Sent false
8838 3259 Acked Read 6 Done Done Sent false
8839 3260 Acked Read 4 Done Done Sent false
8840 3261 Acked Write 6 Done Done Sent false
8841 3262 Acked Write 1 Done Done Sent false
8842 3263 Acked Flush 0 Done Done Sent false
8843 3264 Acked Flush 0 Done Done Sent false
8844 3265 Acked Read 4 Done Done Sent false
8845 3266 Acked Read 9 Done Done Sent false
8846 3267 Acked Flush 0 Done Done Sent false
8847 3268 Acked Flush 0 Done Done Sent false
8848 3269 Acked Write 10 Done Done Sent false
8849 3270 Acked Write 9 Done Done Sent false
8850 3271 Acked Write 3 Done Done Sent false
8851 3272 Acked Flush 0 Done Done Sent false
8852 3273 Acked Read 10 Done Done Sent false
8853 3274 Acked Write 8 Done Done Sent false
8854 3275 Acked Write 1 Done Done Sent false
8855 3276 Acked Read 4 Done Done Sent false
8856 3277 Acked Read 1 Done Done Sent false
8857 3278 Acked Read 10 Done Done Sent false
8858 3279 Acked Write 3 Done Done Sent false
8859 3280 Acked Read 8 Done Done Sent false
8860 3281 Acked Read 4 Done Done Sent false
8861 3282 Acked Read 3 Done Done Sent false
8862 3283 Acked Read 10 Done Done Sent false
8863 3284 Acked Write 1 Done Done Sent false
8864 3285 Acked Read 1 Done Done Sent false
8865 3286 Acked Read 10 Done Done Sent false
8866 3287 Acked Write 7 Done Done Sent false
8867 3288 Acked Read 4 Done Done Sent false
8868 3289 Acked Read 1 Done Done Sent false
8869 3290 Acked Read 1 Done Done Sent false
8870 3291 Acked Read 9 Done Done Sent false
8871 3292 Acked Flush 0 Done Done Sent false
8872 3293 Acked Write 8 Done Done Sent false
8873 3294 Acked Write 5 Done Done Sent false
8874 3295 Acked Read 1 Done Done Sent false
8875 3296 Acked Write 1 Done Done Sent false
8876 3297 Acked Read 10 Done Done Sent false
8877 3298 Acked Read 10 Done Done Sent false
8878 3299 Acked Read 6 Done Done Sent false
8879 3300 Acked Read 2 Done Done Sent false
8880 3301 Acked Write 7 Done Done Sent false
8881 3302 Acked Read 4 Done Done Sent false
8882 3303 Acked Write 3 Done Done Sent false
8883 3304 Acked Read 2 Done Done Sent false
8884 3305 Acked Read 10 Done Done Sent false
8885 3306 Acked Flush 0 Done Done Sent false
8886 3307 Acked Write 9 Done Done Sent false
8887 3308 Acked Read 2 Done Done Sent false
8888 3309 Acked Flush 0 Done Done Sent false
8889 3310 Acked Write 1 Done Done Sent false
8890 3311 Acked Write 8 Done Done Sent false
8891 3312 Acked Write 1 Done Done Sent false
8892 3313 Acked Read 4 Done Done Sent false
8893 3314 Acked Read 2 Done Done Sent false
8894 3315 Acked Read 5 Done Done Sent false
8895 3316 Acked Read 10 Done Done Sent false
8896 3317 Acked Read 9 Done Done Sent false
8897 3318 Acked Read 2 Done Done Sent false
8898 3319 Acked Read 4 Done Done Sent false
8899 3320 Acked Flush 0 Done Done Sent false
8900 3321 Acked Write 7 Done Done Sent false
8901 3322 Acked Read 1 Done Done Sent false
8902 3323 Acked Read 6 Done Done Sent false
8903 3324 Acked Read 2 Done Done Sent false
8904 3325 Acked Flush 0 Done Done Sent false
8905 3326 Acked Write 3 Done Done Sent false
8906 3327 Acked Read 9 Done Done Sent false
8907 3328 Acked Read 10 Done Done Sent false
8908 3329 Acked Read 2 Done Done Sent false
8909 3330 Acked Write 2 Done Done Sent false
8910 3331 Acked Read 2 Done Done Sent false
8911 3332 Acked Write 7 Done Done Sent false
8912 3333 Acked Write 5 Done Done Sent false
8913 3334 Acked Read 4 Done Done Sent false
8914 3335 Acked Read 6 Done Done Sent false
8915 3336 Acked Write 6 Done Done Sent false
8916 3337 Acked Read 1 Done Done Sent false
8917 3338 Acked Read 1 Done Done Sent false
8918 3339 Acked Write 4 Done Done Sent false
8919 3340 Acked Write 1 Done Done Sent false
8920 3341 Acked Read 8 Done Done Sent false
8921 3342 Acked Read 6 Done Done Sent false
8922 3343 Acked Read 1 Done Done Sent false
8923 3344 Acked Write 8 Done Done Sent false
8924 3345 Acked Read 2 Done Done Sent false
8925 3346 Acked Read 7 Done Done Sent false
8926 3347 Acked Write 5 Done Done Sent false
8927 3348 Acked Read 5 Done Done Sent false
8928 3349 Acked Write 7 Done Done Sent false
8929 3350 Acked Read 1 Done Done Sent false
8930 3351 Acked Read 5 Done Done Sent false
8931 3352 Acked Read 3 Done Done Sent false
8932 3353 Acked Read 8 Done Done Sent false
8933 3354 Acked Read 3 Done Done Sent false
8934 3355 Acked Read 8 Done Done Sent false
8935 3356 Acked Read 7 Done Done Sent false
8936 3357 Acked Write 3 Done Done Sent false
8937 3358 Acked Flush 0 Done Done Sent false
8938 3359 Acked Read 10 Done Done Sent false
8939 3360 Acked Read 9 Done Done Sent false
8940 3361 Acked Write 9 Done Done Sent false
8941 3362 Acked Read 5 Done Done Sent false
8942 3363 Acked Write 5 Done Done Sent false
8943 3364 Acked Write 3 Done Done Sent false
8944 3365 Acked Read 4 Done Done Sent false
8945 3366 Acked Write 3 Done Done Sent false
8946 3367 Acked Write 2 Done Done Sent false
8947 3368 Acked Write 9 Done Done Sent false
8948 3369 Acked Write 5 Done Done Sent false
8949 3370 Acked Read 3 Done Done Sent false
8950 3371 Acked Read 4 Done Done Sent false
8951 3372 Acked Read 5 Done Done Sent false
8952 3373 Acked Read 2 Done Done Sent false
8953 3374 Acked Write 10 Done Done Sent false
8954 3375 Acked Read 10 Done Done Sent false
8955 3376 Acked Read 5 Done Done Sent false
8956 3377 Acked Read 4 Done Done Sent false
8957 3378 Acked Read 3 Done Done Sent false
8958 3379 Acked Write 3 Done Done Sent false
8959 3380 Acked Read 9 Done Done Sent false
8960 3381 Acked Write 4 Done Done Sent false
8961 3382 Acked Write 9 Done Done Sent false
8962 3383 Acked Write 8 Done Done Sent false
8963 3384 Acked Write 10 Done Done Sent false
8964 3385 Acked Read 4 Done Done Sent false
8965 3386 Acked Flush 0 Done Done Sent false
8966 3387 Acked Write 5 Done Done Sent false
8967 3388 Acked Read 5 Done Done Sent false
8968 3389 Acked Write 4 Done Done Sent false
8969 3390 Acked Read 10 Done Done Sent false
8970 3391 Acked Write 1 Done Done Sent false
8971 3392 Acked Read 9 Done Done Sent false
8972 3393 Acked Read 1 Done Done Sent false
8973 3394 Acked Read 8 Done Done Sent false
8974 3395 Acked Write 4 Done Done Sent false
8975 3396 Acked Read 3 Done Done Sent false
8976 3397 Acked Write 2 Done Done Sent false
8977 3398 Acked Read 2 Done Done Sent false
8978 3399 Acked Write 8 Done Done Sent false
8979 3400 Acked Read 10 Done Done Sent false
8980 3401 Acked Read 2 Done Done Sent false
8981 3402 Acked Read 7 Done Done Sent false
8982 3403 Acked Read 7 Done Done Sent false
8983 3404 Acked Write 5 Done Done Sent false
8984 3405 Acked Read 8 Done Done Sent false
8985 3406 Acked Read 9 Done Done Sent false
8986 3407 Acked Write 3 Done Done Sent false
8987 3408 Acked Read 5 Done Done Sent false
8988 3409 Acked Read 3 Done Done Sent false
8989 3410 Acked Read 10 Done Done Sent false
8990 3411 Acked Read 9 Done Done Sent false
8991 3412 Acked Write 3 Done Done Sent false
8992 3413 Acked Read 1 Done Done Sent false
8993 3414 Acked Read 9 Done Done Sent false
8994 STATES DS:0 DS:1 DS:2 TOTAL
8995 Sent 0 0 301 301
8996 Done 301 301 0 602
8997 Skipped 0 0 0 0
8998 Error 0 0 0 0
8999 Last Flush: Some(JobId(3386)) Some(JobId(3386)) Some(JobId(3113))
9000 Downstairs last five completed: 3113 3112 3111 3110 3109
9001 Upstairs last five completed: 3414 3413 3412 3411 3410
9002 CLIENT: Up:0 ds:301 act:2
90032025-02-25T17:20:08.465ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
90042025-02-25T17:20:09.466ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
90052025-02-25T17:20:10.467ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
90062025-02-25T17:20:10.968ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
90072025-02-25T17:20:11.469ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
9008 ----------------------------------------------------------------
9009 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9010 States: Active Active Offline
9011 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9012 3114 Acked Flush 0 Done Done Sent false
9013 3115 Acked Read 3 Done Done Sent false
9014 3116 Acked Read 6 Done Done Sent false
9015 3117 Acked Flush 0 Done Done Sent false
9016 3118 Acked Read 8 Done Done Sent false
9017 3119 Acked Write 6 Done Done Sent false
9018 3120 Acked Read 9 Done Done Sent false
9019 3121 Acked Write 4 Done Done Sent false
9020 3122 Acked Write 4 Done Done Sent false
9021 3123 Acked Flush 0 Done Done Sent false
9022 3124 Acked Read 8 Done Done Sent false
9023 3125 Acked Read 4 Done Done Sent false
9024 3126 Acked Write 1 Done Done Sent false
9025 3127 Acked Read 8 Done Done Sent false
9026 3128 Acked Read 7 Done Done Sent false
9027 3129 Acked Flush 0 Done Done Sent false
9028 3130 Acked Write 8 Done Done Sent false
9029 3131 Acked Write 1 Done Done Sent false
9030 3132 Acked Write 6 Done Done Sent false
9031 3133 Acked Read 7 Done Done Sent false
9032 3134 Acked Read 10 Done Done Sent false
9033 3135 Acked Read 3 Done Done Sent false
9034 3136 Acked Write 9 Done Done Sent false
9035 3137 Acked Write 3 Done Done Sent false
9036 3138 Acked Flush 0 Done Done Sent false
9037 3139 Acked Flush 0 Done Done Sent false
9038 3140 Acked Write 4 Done Done Sent false
9039 3141 Acked Write 3 Done Done Sent false
9040 3142 Acked Write 7 Done Done Sent false
9041 3143 Acked Write 4 Done Done Sent false
9042 3144 Acked Read 1 Done Done Sent false
9043 3145 Acked Write 4 Done Done Sent false
9044 3146 Acked Flush 0 Done Done Sent false
9045 3147 Acked Write 4 Done Done Sent false
9046 3148 Acked Read 1 Done Done Sent false
9047 3149 Acked Write 9 Done Done Sent false
9048 3150 Acked Read 7 Done Done Sent false
9049 3151 Acked Read 8 Done Done Sent false
9050 3152 Acked Read 10 Done Done Sent false
9051 3153 Acked Read 4 Done Done Sent false
9052 3154 Acked Write 5 Done Done Sent false
9053 3155 Acked Write 3 Done Done Sent false
9054 3156 Acked Write 8 Done Done Sent false
9055 3157 Acked Write 10 Done Done Sent false
9056 3158 Acked Write 1 Done Done Sent false
9057 3159 Acked Write 6 Done Done Sent false
9058 3160 Acked Read 4 Done Done Sent false
9059 3161 Acked Read 3 Done Done Sent false
9060 3162 Acked Read 7 Done Done Sent false
9061 3163 Acked Read 9 Done Done Sent false
9062 3164 Acked Flush 0 Done Done Sent false
9063 3165 Acked Write 3 Done Done Sent false
9064 3166 Acked Flush 0 Done Done Sent false
9065 3167 Acked Write 6 Done Done Sent false
9066 3168 Acked Flush 0 Done Done Sent false
9067 3169 Acked Write 7 Done Done Sent false
9068 3170 Acked Write 8 Done Done Sent false
9069 3171 Acked Flush 0 Done Done Sent false
9070 3172 Acked Read 3 Done Done Sent false
9071 3173 Acked Write 10 Done Done Sent false
9072 3174 Acked Read 4 Done Done Sent false
9073 3175 Acked Read 2 Done Done Sent false
9074 3176 Acked Write 1 Done Done Sent false
9075 3177 Acked Read 4 Done Done Sent false
9076 3178 Acked Read 10 Done Done Sent false
9077 3179 Acked Write 2 Done Done Sent false
9078 3180 Acked Read 8 Done Done Sent false
9079 3181 Acked Read 7 Done Done Sent false
9080 3182 Acked Read 6 Done Done Sent false
9081 3183 Acked Read 8 Done Done Sent false
9082 3184 Acked Write 1 Done Done Sent false
9083 3185 Acked Write 2 Done Done Sent false
9084 3186 Acked Read 7 Done Done Sent false
9085 3187 Acked Write 4 Done Done Sent false
9086 3188 Acked Write 5 Done Done Sent false
9087 3189 Acked Flush 0 Done Done Sent false
9088 3190 Acked Write 7 Done Done Sent false
9089 3191 Acked Read 2 Done Done Sent false
9090 3192 Acked Write 6 Done Done Sent false
9091 3193 Acked Write 8 Done Done Sent false
9092 3194 Acked Read 9 Done Done Sent false
9093 3195 Acked Read 3 Done Done Sent false
9094 3196 Acked Write 1 Done Done Sent false
9095 3197 Acked Write 5 Done Done Sent false
9096 3198 Acked Write 4 Done Done Sent false
9097 3199 Acked Write 9 Done Done Sent false
9098 3200 Acked Read 10 Done Done Sent false
9099 3201 Acked Read 2 Done Done Sent false
9100 3202 Acked Write 4 Done Done Sent false
9101 3203 Acked Read 8 Done Done Sent false
9102 3204 Acked Read 5 Done Done Sent false
9103 3205 Acked Write 5 Done Done Sent false
9104 3206 Acked Read 9 Done Done Sent false
9105 3207 Acked Read 9 Done Done Sent false
9106 3208 Acked Read 8 Done Done Sent false
9107 3209 Acked Write 9 Done Done Sent false
9108 3210 Acked Read 9 Done Done Sent false
9109 3211 Acked Write 7 Done Done Sent false
9110 3212 Acked Read 2 Done Done Sent false
9111 3213 Acked Write 1 Done Done Sent false
9112 3214 Acked Read 5 Done Done Sent false
9113 3215 Acked Write 4 Done Done Sent false
9114 3216 Acked Read 10 Done Done Sent false
9115 3217 Acked Write 7 Done Done Sent false
9116 3218 Acked Read 4 Done Done Sent false
9117 3219 Acked Flush 0 Done Done Sent false
9118 3220 Acked Read 2 Done Done Sent false
9119 3221 Acked Write 9 Done Done Sent false
9120 3222 Acked Write 3 Done Done Sent false
9121 3223 Acked Read 10 Done Done Sent false
9122 3224 Acked Write 10 Done Done Sent false
9123 3225 Acked Write 7 Done Done Sent false
9124 3226 Acked Flush 0 Done Done Sent false
9125 3227 Acked Write 4 Done Done Sent false
9126 3228 Acked Read 8 Done Done Sent false
9127 3229 Acked Write 5 Done Done Sent false
9128 3230 Acked Write 2 Done Done Sent false
9129 3231 Acked Read 9 Done Done Sent false
9130 3232 Acked Read 4 Done Done Sent false
9131 3233 Acked Write 1 Done Done Sent false
9132 3234 Acked Read 5 Done Done Sent false
9133 3235 Acked Write 7 Done Done Sent false
9134 3236 Acked Read 9 Done Done Sent false
9135 3237 Acked Write 1 Done Done Sent false
9136 3238 Acked Read 8 Done Done Sent false
9137 3239 Acked Read 7 Done Done Sent false
9138 3240 Acked Read 5 Done Done Sent false
9139 3241 Acked Read 6 Done Done Sent false
9140 3242 Acked Read 9 Done Done Sent false
9141 3243 Acked Read 4 Done Done Sent false
9142 3244 Acked Write 2 Done Done Sent false
9143 3245 Acked Write 10 Done Done Sent false
9144 3246 Acked Read 9 Done Done Sent false
9145 3247 Acked Read 4 Done Done Sent false
9146 3248 Acked Write 10 Done Done Sent false
9147 3249 Acked Write 8 Done Done Sent false
9148 3250 Acked Write 6 Done Done Sent false
9149 3251 Acked Read 1 Done Done Sent false
9150 3252 Acked Write 2 Done Done Sent false
9151 3253 Acked Flush 0 Done Done Sent false
9152 3254 Acked Flush 0 Done Done Sent false
9153 3255 Acked Write 10 Done Done Sent false
9154 3256 Acked Read 1 Done Done Sent false
9155 3257 Acked Write 10 Done Done Sent false
9156 3258 Acked Read 6 Done Done Sent false
9157 3259 Acked Read 6 Done Done Sent false
9158 3260 Acked Read 4 Done Done Sent false
9159 3261 Acked Write 6 Done Done Sent false
9160 3262 Acked Write 1 Done Done Sent false
9161 3263 Acked Flush 0 Done Done Sent false
9162 3264 Acked Flush 0 Done Done Sent false
9163 3265 Acked Read 4 Done Done Sent false
9164 3266 Acked Read 9 Done Done Sent false
9165 3267 Acked Flush 0 Done Done Sent false
9166 3268 Acked Flush 0 Done Done Sent false
9167 3269 Acked Write 10 Done Done Sent false
9168 3270 Acked Write 9 Done Done Sent false
9169 3271 Acked Write 3 Done Done Sent false
9170 3272 Acked Flush 0 Done Done Sent false
9171 3273 Acked Read 10 Done Done Sent false
9172 3274 Acked Write 8 Done Done Sent false
9173 3275 Acked Write 1 Done Done Sent false
9174 3276 Acked Read 4 Done Done Sent false
9175 3277 Acked Read 1 Done Done Sent false
9176 3278 Acked Read 10 Done Done Sent false
9177 3279 Acked Write 3 Done Done Sent false
9178 3280 Acked Read 8 Done Done Sent false
9179 3281 Acked Read 4 Done Done Sent false
9180 3282 Acked Read 3 Done Done Sent false
9181 3283 Acked Read 10 Done Done Sent false
9182 3284 Acked Write 1 Done Done Sent false
9183 3285 Acked Read 1 Done Done Sent false
9184 3286 Acked Read 10 Done Done Sent false
9185 3287 Acked Write 7 Done Done Sent false
9186 3288 Acked Read 4 Done Done Sent false
9187 3289 Acked Read 1 Done Done Sent false
9188 3290 Acked Read 1 Done Done Sent false
9189 3291 Acked Read 9 Done Done Sent false
9190 3292 Acked Flush 0 Done Done Sent false
9191 3293 Acked Write 8 Done Done Sent false
9192 3294 Acked Write 5 Done Done Sent false
9193 3295 Acked Read 1 Done Done Sent false
9194 3296 Acked Write 1 Done Done Sent false
9195 3297 Acked Read 10 Done Done Sent false
9196 3298 Acked Read 10 Done Done Sent false
9197 3299 Acked Read 6 Done Done Sent false
9198 3300 Acked Read 2 Done Done Sent false
9199 3301 Acked Write 7 Done Done Sent false
9200 3302 Acked Read 4 Done Done Sent false
9201 3303 Acked Write 3 Done Done Sent false
9202 3304 Acked Read 2 Done Done Sent false
9203 3305 Acked Read 10 Done Done Sent false
9204 3306 Acked Flush 0 Done Done Sent false
9205 3307 Acked Write 9 Done Done Sent false
9206 3308 Acked Read 2 Done Done Sent false
9207 3309 Acked Flush 0 Done Done Sent false
9208 3310 Acked Write 1 Done Done Sent false
9209 3311 Acked Write 8 Done Done Sent false
9210 3312 Acked Write 1 Done Done Sent false
9211 3313 Acked Read 4 Done Done Sent false
9212 3314 Acked Read 2 Done Done Sent false
9213 3315 Acked Read 5 Done Done Sent false
9214 3316 Acked Read 10 Done Done Sent false
9215 3317 Acked Read 9 Done Done Sent false
9216 3318 Acked Read 2 Done Done Sent false
9217 3319 Acked Read 4 Done Done Sent false
9218 3320 Acked Flush 0 Done Done Sent false
9219 3321 Acked Write 7 Done Done Sent false
9220 3322 Acked Read 1 Done Done Sent false
9221 3323 Acked Read 6 Done Done Sent false
9222 3324 Acked Read 2 Done Done Sent false
9223 3325 Acked Flush 0 Done Done Sent false
9224 3326 Acked Write 3 Done Done Sent false
9225 3327 Acked Read 9 Done Done Sent false
9226 3328 Acked Read 10 Done Done Sent false
9227 3329 Acked Read 2 Done Done Sent false
9228 3330 Acked Write 2 Done Done Sent false
9229 3331 Acked Read 2 Done Done Sent false
9230 3332 Acked Write 7 Done Done Sent false
9231 3333 Acked Write 5 Done Done Sent false
9232 3334 Acked Read 4 Done Done Sent false
9233 3335 Acked Read 6 Done Done Sent false
9234 3336 Acked Write 6 Done Done Sent false
9235 3337 Acked Read 1 Done Done Sent false
9236 3338 Acked Read 1 Done Done Sent false
9237 3339 Acked Write 4 Done Done Sent false
9238 3340 Acked Write 1 Done Done Sent false
9239 3341 Acked Read 8 Done Done Sent false
9240 3342 Acked Read 6 Done Done Sent false
9241 3343 Acked Read 1 Done Done Sent false
9242 3344 Acked Write 8 Done Done Sent false
9243 3345 Acked Read 2 Done Done Sent false
9244 3346 Acked Read 7 Done Done Sent false
9245 3347 Acked Write 5 Done Done Sent false
9246 3348 Acked Read 5 Done Done Sent false
9247 3349 Acked Write 7 Done Done Sent false
9248 3350 Acked Read 1 Done Done Sent false
9249 3351 Acked Read 5 Done Done Sent false
9250 3352 Acked Read 3 Done Done Sent false
9251 3353 Acked Read 8 Done Done Sent false
9252 3354 Acked Read 3 Done Done Sent false
9253 3355 Acked Read 8 Done Done Sent false
9254 3356 Acked Read 7 Done Done Sent false
9255 3357 Acked Write 3 Done Done Sent false
9256 3358 Acked Flush 0 Done Done Sent false
9257 3359 Acked Read 10 Done Done Sent false
9258 3360 Acked Read 9 Done Done Sent false
9259 3361 Acked Write 9 Done Done Sent false
9260 3362 Acked Read 5 Done Done Sent false
9261 3363 Acked Write 5 Done Done Sent false
9262 3364 Acked Write 3 Done Done Sent false
9263 3365 Acked Read 4 Done Done Sent false
9264 3366 Acked Write 3 Done Done Sent false
9265 3367 Acked Write 2 Done Done Sent false
9266 3368 Acked Write 9 Done Done Sent false
9267 3369 Acked Write 5 Done Done Sent false
9268 3370 Acked Read 3 Done Done Sent false
9269 3371 Acked Read 4 Done Done Sent false
9270 3372 Acked Read 5 Done Done Sent false
9271 3373 Acked Read 2 Done Done Sent false
9272 3374 Acked Write 10 Done Done Sent false
9273 3375 Acked Read 10 Done Done Sent false
9274 3376 Acked Read 5 Done Done Sent false
9275 3377 Acked Read 4 Done Done Sent false
9276 3378 Acked Read 3 Done Done Sent false
9277 3379 Acked Write 3 Done Done Sent false
9278 3380 Acked Read 9 Done Done Sent false
9279 3381 Acked Write 4 Done Done Sent false
9280 3382 Acked Write 9 Done Done Sent false
9281 3383 Acked Write 8 Done Done Sent false
9282 3384 Acked Write 10 Done Done Sent false
9283 3385 Acked Read 4 Done Done Sent false
9284 3386 Acked Flush 0 Done Done Sent false
9285 3387 Acked Write 5 Done Done Sent false
9286 3388 Acked Read 5 Done Done Sent false
9287 3389 Acked Write 4 Done Done Sent false
9288 3390 Acked Read 10 Done Done Sent false
9289 3391 Acked Write 1 Done Done Sent false
9290 3392 Acked Read 9 Done Done Sent false
9291 3393 Acked Read 1 Done Done Sent false
9292 3394 Acked Read 8 Done Done Sent false
9293 3395 Acked Write 4 Done Done Sent false
9294 3396 Acked Read 3 Done Done Sent false
9295 3397 Acked Write 2 Done Done Sent false
9296 3398 Acked Read 2 Done Done Sent false
9297 3399 Acked Write 8 Done Done Sent false
9298 3400 Acked Read 10 Done Done Sent false
9299 3401 Acked Read 2 Done Done Sent false
9300 3402 Acked Read 7 Done Done Sent false
9301 3403 Acked Read 7 Done Done Sent false
9302 3404 Acked Write 5 Done Done Sent false
9303 3405 Acked Read 8 Done Done Sent false
9304 3406 Acked Read 9 Done Done Sent false
9305 3407 Acked Write 3 Done Done Sent false
9306 3408 Acked Read 5 Done Done Sent false
9307 3409 Acked Read 3 Done Done Sent false
9308 3410 Acked Read 10 Done Done Sent false
9309 3411 Acked Read 9 Done Done Sent false
9310 3412 Acked Write 3 Done Done Sent false
9311 3413 Acked Read 1 Done Done Sent false
9312 3414 Acked Read 9 Done Done Sent false
9313 STATES DS:0 DS:1 DS:2 TOTAL
9314 Sent 0 0 301 301
9315 Done 301 301 0 602
9316 Skipped 0 0 0 0
9317 Error 0 0 0 0
9318 Last Flush: Some(JobId(3386)) Some(JobId(3386)) Some(JobId(3113))
9319 Downstairs last five completed: 3113 3112 3111 3110 3109
9320 Upstairs last five completed: 3414 3413 3412 3411 3410
9321 CLIENT: Up:0 ds:301 act:2
93222025-02-25T17:20:12.471ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93232025-02-25T17:20:13.472ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93242025-02-25T17:20:14.474ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93252025-02-25T17:20:15.475ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93262025-02-25T17:20:15.684ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93272025-02-25T17:20:15.685ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39628) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93282025-02-25T17:20:15.686ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93292025-02-25T17:20:15.686ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93302025-02-25T17:20:15.686ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93312025-02-25T17:20:15.686ZINFOcrucible: send last flush ID to this DS: Some(JobId(3113)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93322025-02-25T17:20:15.686ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93332025-02-25T17:20:15.686ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3113)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93342025-02-25T17:20:15.687ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3113)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
9335 ----------------------------------------------------------------
9336 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:28
9337 States: Active Active Active
9338 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9339 3387 Acked Write 5 Done Done Done true
9340 3388 Acked Read 5 Done Done Done true
9341 3389 Acked Write 4 Done Done Done true
9342 3390 Acked Read 10 Done Done Done true
9343 3391 Acked Write 1 Done Done Done true
9344 3392 Acked Read 9 Done Done Done true
9345 3393 Acked Read 1 Done Done Done true
9346 3394 Acked Read 8 Done Done Done true
9347 3395 Acked Write 4 Done Done Done true
9348 3396 Acked Read 3 Done Done Done true
9349 3397 Acked Write 2 Done Done Done true
9350 3398 Acked Read 2 Done Done Done true
9351 3399 Acked Write 8 Done Done Done true
9352 3400 Acked Read 10 Done Done Done true
9353 3401 Acked Read 2 Done Done Done true
9354 3402 Acked Read 7 Done Done Done true
9355 3403 Acked Read 7 Done Done Done true
9356 3404 Acked Write 5 Done Done Done true
9357 3405 Acked Read 8 Done Done Done true
9358 3406 Acked Read 9 Done Done Done true
9359 3407 Acked Write 3 Done Done Done true
9360 3408 Acked Read 5 Done Done Done true
9361 3409 Acked Read 3 Done Done Done true
9362 3410 Acked Read 10 Done Done Done true
9363 3411 Acked Read 9 Done Done Done true
9364 3412 Acked Write 3 Done Done Done true
9365 3413 Acked Read 1 Done Done Done true
9366 3414 Acked Read 9 Done Done Done true
9367 STATES DS:0 DS:1 DS:2 TOTAL
9368 Sent 0 0 0 0
9369 Done 28 28 28 84
9370 Skipped 0 0 0 0
9371 Error 0 0 0 0
9372 Last Flush: Some(JobId(3386)) Some(JobId(3386)) Some(JobId(3386))
9373 Downstairs last five completed: 3386 3385 3384 3383 3382
9374 Upstairs last five completed: 3414 3413 3412 3411 3410
9375 CLIENT: Up:0 ds:28 act:3
9376 ----------------------------------------------------------------
9377 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9378 Downstairs last five completed: 3415 3414 3413 3412 3411
9379 Upstairs last five completed: 3415 3414 3413 3412 3411
9380 CLIENT: Up:0 ds:0 act:3
9381 Replay: All jobs finished
9382 Main task has work to do, go find it
9383 got dsc Stop(2)
9384 stop 2
9385 [8830] Got stop action so:false kr:false
9386 Feb 25 17:20:19.952 INFO request completed, latency_us: 397, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 1bd5e7f4-c834-4640-8337-087402fef6ac, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
9387 Feb 25 17:20:19.953 INFO request completed, latency_us: 140, response_code: 200, uri: /state/cid/2, method: GET, req_id: 141d246c-e363-4981-a0d4-a32a538f8322, remote_addr: 127.0.0.1:52576, local_addr{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:20:19.953447147Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
9388 : 127.0.0.1{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:20:19.953500919Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
9389 :9998
93902025-02-25T17:20:19.953ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
93912025-02-25T17:20:19.953ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
9392 [8830] Exited with: ExitStatus(unix_wait_status(9))
9393 [8830][2] reports Exit
9394 Feb 25 17:20:23.955 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/2, method: GET, req_id: 387c3d3a-49bd-4ebf-9f0e-58103da91516, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
9395 001/300 Read block 2550 len 32768
9396 002/300 Read block 362 len 32768
9397 003/300 Read block 2149 len 28672
9398 004/300 Write block 916 len 8192 data: 2 2
9399 005/300 Read block 1551 len 20480
9400 006/300 Write block 377 len 24576 data: 3 4 4 4 3 2
9401 007/300 Read block 4118 len 36864
9402 008/300 Read block 4141 len 36864
9403 009/300 Read block 3476 len 32768
9404 010/300 Flush
9405 011/300 Read block 3899 len 16384
9406 012/300 Write block 613 len 4096 data: 3
9407 013/300 Read block 4622 len 28672
9408 014/300 Flush
9409 015/300 Read block 734 len 16384
9410 016/300 Write block 1291 len 12288 data: 5 4 4
9411 017/300 Read block 2383 len 28672
9412 018/300 Write block 1394 len 36864 data: 3 3 3 3 3 4 4 4 4
9413 019/300 Write block 3496 len 28672 data: 3 4 3 2 2 3 3
9414 020/300 Read block 1900 len 12288
9415 021/300 Read block 1262 len 32768
9416 022/300 Read block 490 len 32768
9417 023/300 Read block 4843 len 24576
9418 024/300 Write block 4716 len 12288 data: 2 2 3
9419 025/300 Read block 4134 len 36864
9420 026/300 Write block 3667 len 4096 data: 4
9421 027/300 Write block 2727 len 32768 data: 2 3 3 3 3 2 3 4
9422 028/300 Read block 3 len 12288
9423 029/300 Write block 229 len 36864 data: 3 3 3 3 3 2 2 2 2
9424 030/300 Read block 3508 len 36864
9425 031/300 Read block 2096 len 28672
9426 032/300 Write block 1998 len 20480 data: 4 3 3 3 3
9427 033/300 Write block 2791 len 32768 data: 2 2 2 2 2 2 3 4
9428 034/300 Read block 4087 len 12288
9429 035/300 Write block 3152 len 28672 data: 2 2 2 2 2 2 2
9430 036/300 Read block 879 len 12288
9431 037/300 Write block 2937 len 8192 data: 3 3
9432 038/300 Read block 2212 len 32768
9433 039/300 Write block 1358 len 36864 data: 2 2 2 2 2 2 2 2 2
9434 040/300 Read block 2322 len 12288
9435 041/300 Read block 3402 len 32768
9436 042/300 Read block 4507 len 8192
9437 043/300 Read block 2196 len 12288
9438 044/300 Flush
9439 045/300 Write block 934 len 4096 data: 3
9440 046/300 Read block 3599 len 24576
9441 047/300 Write block 4164 len 40960 data: 2 2 2 3 3 3 4 3 2 2
9442 048/300 Read block 2834 len 20480
9443 049/300 Write block 2626 len 20480 data: 5 4 5 5 5
9444 050/300 Read block 2066 len 40960
9445 051/300 Read block 3504 len 4096
9446 052/300 Read block 3315 len 4096
9447 053/300 Flush
9448 054/300 Flush
9449 055/300 Write block 737 len 28672 data: 3 3 3 2 3 3 3
9450 056/300 Flush
9451 057/300 Read block 2612 len 12288
9452 058/300 Read block 1595 len 24576
9453 059/300 Write block 1910 len 4096 data: 6
9454 060/300 Write block 1362 len 32768 data: 3 3 3 3 3 2 2 2
9455 061/300 Read block 2414 len 36864
9456 062/300 Read block 2816 len 32768
9457 063/300 Write block 3107 len 36864 data: 3 3 3 4 4 3 3 3 3
9458 064/300 Write block 2254 len 40960 data: 2 2 2 3 2 2 2 3 3 3
9459 065/300 Flush
9460 066/300 Flush
9461 067/300 Write block 2590 len 24576 data: 3 3 3 3 3 3
9462 068/300 Write block 388 len 24576 data: 2 2 2 2 2 2
9463 069/300 Read block 594 len 28672
9464 070/300 Read block 277 len 28672
9465 071/300 Read block 3951 len 24576
9466 072/300 Write block 2303 len 12288 data: 3 3 2
9467 073/300 Read block 3038 len 16384
9468 074/300 Read block 3737 len 20480
9469 075/300 Write block 3769 len 24576 data: 2 2 2 2 2 2
9470 076/300 Write block 12 len 32768 data: 2 3 3 3 4 3 4 3
9471 077/300 Write block 74 len 12288 data: 5 4 3
9472 078/300 Read block 4006 len 20480
9473 079/300 Write block 1486 len 4096 data: 3
9474 080/300 Read block 4557 len 32768
9475 081/300 Read block 4391 len 40960
9476 082/300 Read block 2141 len 24576
9477 083/300 Write block 4922 len 20480 data: 2 2 2 2 2
9478 084/300 Write block 2637 len 12288 data: 2 2 2
9479 085/300 Read block 2099 len 4096
9480 086/300 Read block 720 len 32768
9481 087/300 Write block 3633 len 8192 data: 2 2
9482 088/300 Write block 2704 len 24576 data: 2 2 2 2 2 2
9483 089/300 Write block 3538 len 24576 data: 4 4 5 4 4 3
9484 090/300 Flush
9485 091/300 Write block 1999 len 8192 data: 4 4
9486 092/300 Write block 2786 len 32768 data: 3 3 2 2 2 3 3 3
9487 093/300 Read block 3666 len 28672
9488 094/300 Read block 649 len 16384
9489 095/300 Read block 1196 len 8192
9490 096/300 Read block 858 len 40960
9491 097/300 Write block 849 len 16384 data: 2 3 3 3
9492 098/300 Write block 3010 len 32768 data: 4 2 2 2 2 2 2 2
9493 099/300 Read block 3276 len 4096
9494 100/300 Read block 2802 len 40960
9495 101/300 Read block 2618 len 28672
9496 102/300 Flush
9497 103/300 Write block 410 len 28672 data: 3 4 3 3 3 4 3
9498 104/300 Write block 870 len 16384 data: 3 4 3 2
9499 105/300 Write block 2825 len 40960 data: 3 3 3 3 3 4 4 4 4 4
9500 106/300 Write block 333 len 20480 data: 3 3 3 3 3
9501 107/300 Read block 4425 len 40960
9502 108/300 Flush
9503 109/300 Read block 207 len 8192
9504 110/300 Write block 2553 len 40960 data: 2 2 2 2 2 3 3 2 2 2
9505 111/300 Read block 2010 len 24576
9506 112/300 Write block 1375 len 40960 data: 4 3 3 2 2 2 3 5 5 4
9507 113/300 Read block 1524 len 28672
9508 114/300 Read block 1616 len 28672
9509 115/300 Read block 3052 len 20480
9510 116/300 Read block 822 len 8192
9511 117/300 Write block 4154 len 4096 data: 2
9512 118/300 Read block 3759 len 12288
9513 119/300 Write block 636 len 28672 data: 2 2 2 2 2 3 4
9514 120/300 Flush
9515 121/300 Flush
9516 122/300 Read block 1145 len 8192
9517 123/300 Read block 248 len 24576
9518 124/300 Read block 1358 len 8192
9519 125/300 Write block 3378 len 16384 data: 2 2 3 3
9520 126/300 Read block 1415 len 4096
9521 127/300 Read block 577 len 32768
9522 128/300 Read block 4971 len 32768
9523 129/300 Read block 142 len 12288
9524 130/300 Flush
9525 131/300 Write block 4077 len 32768 data: 2 3 3 2 2 2 2 2
9526 132/300 Flush
9527 133/300 Write block 1364 len 16384 data: 4 4 4 3
9528 134/300 Write block 40 len 28672 data: 3 3 2 4 4 5 5
9529 135/300 Flush
9530 136/300 Write block 4138 len 20480 data: 2 2 2 2 2
9531 137/300 Read block 1801 len 40960
9532 138/300 Read block 3186 len 40960
9533 139/300 Read block 3903 len 36864
9534 140/300 Read block 2044 len 32768
9535 141/300 Read block 926 len 4096
9536 142/300 Write block 4346 len 16384 data: 4 4 4 3
9537 143/300 Flush
9538 144/300 Read block 4515 len 32768
9539 145/300 Write block 1056 len 8192 data: 3 3
9540 146/300 Read block 4200 len 12288
9541 147/300 Read block 79 len 4096
9542 148/300 Write block 2875 len 28672 data: 4 3 4 3 3 3 4
9543 149/300 Write block 2101 len 12288 data: 3 2 3
9544 150/300 Read block 3333 len 16384
9545 151/300 Write block 1477 len 36864 data: 3 3 3 3 3 3 4 3 3
9546 152/300 Write block 2453 len 36864 data: 4 5 5 6 8 7 7 7 7
9547 153/300 Flush
9548 154/300 Write block 1360 len 40960 data: 3 3 4 4 5 5 5 4 3 3
9549 155/300 Read block 3256 len 40960
9550 156/300 Read block 4315 len 28672
9551 157/300 Read block 4041 len 24576
9552 158/300 Write block 4287 len 20480 data: 2 3 3 3 3
9553 159/300 Write block 1320 len 24576 data: 3 3 3 3 3 2
9554 160/300 Flush
9555 161/300 Read block 1098 len 12288
9556 162/300 Write block 3708 len 16384 data: 3 4 4 3
9557 163/300 Write block 366 len 28672 data: 2 2 2 2 2 2 3
9558 164/300 Flush
9559 165/300 Write block 895 len 8192 data: 5 3
9560 166/300 Write block 1635 len 20480 data: 3 2 2 2 2
9561 167/300 Read block 298 len 36864
9562 168/300 Read block 945 len 28672
9563 169/300 Write block 59 len 12288 data: 3 2 2
9564 170/300 Read block 906 len 28672
9565 171/300 Read block 1334 len 12288
9566 172/300 Write block 626 len 36864 data: 3 3 3 3 3 4 3 3 2
9567 173/300 Read block 2986 len 24576
9568 174/300 Flush
9569 175/300 Flush
9570 176/300 Read block 338 len 12288
9571 177/300 Write block 2210 len 24576 data: 4 6 5 5 5 4
9572 178/300 Read block 4897 len 40960
9573 179/300 Write block 1865 len 4096 data: 2
9574 180/300 Read block 209 len 20480
9575 181/300 Flush
9576 182/300 Write block 2715 len 24576 data: 2 2 2 2 2 2
9577 183/300 Read block 1857 len 8192
9578 184/300 Read block 3102 len 12288
9579 185/300 Flush
9580 186/300 Read block 2750 len 4096
9581 187/300 Write block 1174 len 28672 data: 2 2 3 4 5 5 5
9582 188/300 Read block 3676 len 28672
9583 189/300 Write block 3937 len 4096 data: 4
9584 190/300 Read block 1702 len 24576
9585 191/300 Write block 2945 len 36864 data: 3 4 4 3 2 2 2 2 2
9586 192/300 Write block 4537 len 8192 data: 4 4
9587 193/300 Read block 152 len 8192
9588 194/300 Read block 1902 len 12288
9589 195/300 Flush
9590 196/300 Write block 2664 len 24576 data: 5 5 5 4 4 4
9591 197/300 Read block 2117 len 8192
9592 198/300 Write block 4247 len 12288 data: 3 3 4
9593 199/300 Read block 2866 len 4096
9594 200/300 Write block 2677 len 40960 data: 3 3 2 2 2 2 2 2 2 2
9595 201/300 Read block 402 len 32768
9596 202/300 Read block 1737 len 20480
9597 203/300 Read block 1333 len 24576
9598 204/300 Flush
9599 205/300 Write block 1702 len 24576 data: 3 3 3 3 3 2
9600 206/300 Write block 2319 len 28672 data: 3 3 3 3 5 4 4
9601 207/300 Flush
9602 208/300 Write block 2578 len 32768 data: 3 3 2 3 3 3 3 2
9603 209/300 Read block 1017 len 28672
9604 210/300 Write block 3179 len 12288 data: 3 2 2
9605 211/300 Read block 2106 len 20480
9606 212/300 Read block 2507 len 28672
9607 213/300 Write block 2143 len 28672 data: 4 4 3 2 2 2 2
9608 214/300 Write block 2461 len 24576 data: 8 6 4 3 3 3
9609 215/300 Write block 3218 len 28672 data: 4 4 3 3 3 3 3
9610 216/300 Read block 3392 len 40960
9611 217/300 Write block 1868 len 20480 data: 2 2 2 2 2
9612 218/300 Read block 702 len 4096
9613 219/300 Read block 4994 len 24576
9614 220/300 Read block 1347 len 16384
9615 221/300 Read block 716 len 20480
9616 222/300 Write block 2043 len 4096 data: 3
9617 223/300 Read block 4555 len 24576
9618 224/300 Read block 3639 len 36864
9619 225/300 Read block 2990 len 12288
9620 226/300 Read block 4442 len 4096
9621 227/300 Write block 4662 len 20480 data: 5 6 6 5 6
9622 228/300 Read block 4268 len 4096
9623 229/300 Flush
9624 230/300 Write block 1012 len 12288 data: 2 2 2
9625 231/300 Write block 3710 len 32768 data: 5 4 3 4 4 4 4 5
9626 232/300 Read block 3585 len 24576
9627 233/300 Write block 4578 len 36864 data: 3 3 3 3 3 2 2 2 2
9628 234/300 Write block 1485 len 40960 data: 4 4 3 3 4 4 3 3 3 2
9629 235/300 Read block 3342 len 28672
9630 236/300 Read block 4800 len 24576
9631 237/300 Read block 3700 len 40960
9632 238/300 Read block 626 len 8192
9633 239/300 Read block 4651 len 20480
9634 240/300 Read block 747 len 28672
9635 241/300 Read block 3210 len 40960
9636 242/300 Read block 4009 len 32768
9637 243/300 Read block 2053 len 4096
9638 244/300 Write block 1326 len 16384 data: 3 3 3 3
9639 245/300 Read block 2948 len 20480
9640 246/300 Write block 2347 len 28672 data: 3 3 3 3 4 5 4
9641 247/300 Flush
9642 248/300 Write block 3728 len 28672 data: 5 5 5 5 5 4 5
9643 249/300 Write block 4941 len 16384 data: 6 5 3 3
9644 250/300 Read block 4158 len 16384
9645 251/300 Read block 2115 len 36864
9646 252/300 Write block 4089 len 28672 data: 2 2 2 2 2 3 3
9647 253/300 Write block 316 len 12288 data: 4 3 3
9648 254/300 Read block 50 len 16384
9649 255/300 Write block 876 len 40960 data: 3 4 2 2 2 2 2 3 3 3
9650 256/300 Read block 2017 len 8192
9651 257/300 Write block 3593 len 8192 data: 3 3
9652 258/300 Read block 3515 len 12288
9653 259/300 Read block 3403 len 4096
9654 260/300 Read block 2555 len 32768
9655 261/300 Read block 3705 len 40960
9656 262/300 Write block 2388 len 32768 data: 3 3 3 3 3 3 3 3
9657 263/300 Read block 167 len 28672
9658 264/300 Read block 817 len 8192
9659 265/300 Write block 4193 len 28672 data: 3 3 5 5 5 5 5
9660 266/300 Read block 4694 len 24576
9661 267/300 Read block 2813 len 16384
9662 268/300 Read block 3232 len 8192
9663 269/300 Write block 2771 len 24576 data: 3 3 3 3 3 3
9664 270/300 Read block 2662 len 4096
9665 271/300 Write block 31 len 32768 data: 4 2 2 2 2 3 3 3
9666 272/300 Read block 2928 len 12288
9667 273/300 Write block 1800 len 16384 data: 2 2 2 2
9668 274/300 Write block 700 len 12288 data: 3 3 2
9669 275/300 Flush
9670 276/300 Read block 3970 len 28672
9671 277/300 Read block 1545 len 28672
9672 278/300 Write block 3456 len 32768 data: 3 3 2 2 3 2 2 2
9673 279/300 Read block 1284 len 36864
9674 280/300 Write block 3942 len 36864 data: 6 6 6 7 5 6 6 5 5
9675 281/300 Read block 4681 len 4096
9676 282/300 Write block 1193 len 24576 data: 4 4 4 3 4 4
9677 283/300 Read block 732 len 36864
9678 284/300 Write block 690 len 20480 data: 3 3 3 2 2
9679 285/300 Read block 1007 len 20480
9680 286/300 Read block 2550 len 4096
9681 287/300 Read block 2334 len 12288
9682 288/300 Read block 1333 len 24576
9683 289/300 Write block 2578 len 20480 data: 4 4 3 4 4
9684 290/300 Flush
9685 291/300 Read block 4471 len 24576
9686 292/300 Read block 657 len 28672
9687 293/300 Flush
9688 294/300 Read block 1335 len 40960
9689 295/300 Read block 316 len 12288
9690 296/300 Read block 4005 len 16384
9691 297/300 Flush
9692 298/300 Read block 4826 len 20480
9693 299/300 Read block 2446 len 4096
9694 300/300 Write block 621 len 40960 data: 4 4 4 4 4 4 4 4 4 4
9695 301/300 Read block 1065 len 16384
9696 Main task has work to do, go find it
9697 got dsc Start(2)
9698 start 2
9699 Feb 25 17:20:24.216 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/2, method: POST, req_id: 299fac7e-5b57-4ea8-966f-ae57e4456887, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
9700 Replay: started 2, returned:Ok(())
9701 ----------------------------------------------------------------
9702 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9703 States: Active Active Offline
9704 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9705 3416 Acked Read 8 Done Done Sent false
9706 3417 Acked Read 8 Done Done Sent false
9707 3418 Acked Read 7 Done Done Sent false
9708 3419 Acked Write 2 Done Done Sent false
9709 3420 Acked Read 5 Done Done Sent false
9710 3421 Acked Write 6 Done Done Sent false
9711 3422 Acked Read 9 Done Done Sent false
9712 3423 Acked Read 9 Done Done Sent false
9713 3424 Acked Read 8 Done Done Sent false
9714 3425 Acked Flush 0 Done Done Sent false
9715 3426 Acked Read 4 Done Done Sent false
9716 3427 Acked Write 1 Done Done Sent false
9717 3428 Acked Read 7 Done Done Sent false
9718 3429 Acked Flush 0 Done Done Sent false
9719 3430 Acked Read 4 Done Done Sent false
9720 3431 Acked Write 3 Done Done Sent false
9721 3432 Acked Read 7 Done Done Sent false
9722 3433 Acked Write 9 Done Done Sent false
9723 3434 Acked Write 7 Done Done Sent false
9724 3435 Acked Read 3 Done Done Sent false
9725 3436 Acked Read 8 Done Done Sent false
9726 3437 Acked Read 8 Done Done Sent false
9727 3438 Acked Read 6 Done Done Sent false
9728 3439 Acked Write 3 Done Done Sent false
9729 3440 Acked Read 9 Done Done Sent false
9730 3441 Acked Write 1 Done Done Sent false
9731 3442 Acked Write 8 Done Done Sent false
9732 3443 Acked Read 3 Done Done Sent false
9733 3444 Acked Write 9 Done Done Sent false
9734 3445 Acked Read 9 Done Done Sent false
9735 3446 Acked Read 7 Done Done Sent false
9736 3447 Acked Write 5 Done Done Sent false
9737 3448 Acked Write 8 Done Done Sent false
9738 3449 Acked Read 3 Done Done Sent false
9739 3450 Acked Write 7 Done Done Sent false
9740 3451 Acked Read 3 Done Done Sent false
9741 3452 Acked Write 2 Done Done Sent false
9742 3453 Acked Read 8 Done Done Sent false
9743 3454 Acked Write 9 Done Done Sent false
9744 3455 Acked Read 3 Done Done Sent false
9745 3456 Acked Read 8 Done Done Sent false
9746 3457 Acked Read 2 Done Done Sent false
9747 3458 Acked Read 3 Done Done Sent false
9748 3459 Acked Flush 0 Done Done Sent false
9749 3460 Acked Write 1 Done Done Sent false
9750 3461 Acked Read 6 Done Done Sent false
9751 3462 Acked Write 10 Done Done Sent false
9752 3463 Acked Read 5 Done Done Sent false
9753 3464 Acked Write 5 Done Done Sent false
9754 3465 Acked Read 10 Done Done Sent false
9755 3466 Acked Read 1 Done Done Sent false
9756 3467 Acked Read 1 Done Done Sent false
9757 3468 Acked Flush 0 Done Done Sent false
9758 3469 Acked Flush 0 Done Done Sent false
9759 3470 Acked Write 7 Done Done Sent false
9760 3471 Acked Flush 0 Done Done Sent false
9761 3472 Acked Read 3 Done Done Sent false
9762 3473 Acked Read 6 Done Done Sent false
9763 3474 Acked Write 1 Done Done Sent false
9764 3475 Acked Write 8 Done Done Sent false
9765 3476 Acked Read 9 Done Done Sent false
9766 3477 Acked Read 8 Done Done Sent false
9767 3478 Acked Write 9 Done Done Sent false
9768 3479 Acked Write 10 Done Done Sent false
9769 3480 Acked Flush 0 Done Done Sent false
9770 3481 Acked Flush 0 Done Done Sent false
9771 3482 Acked Write 6 Done Done Sent false
9772 3483 Acked Write 6 Done Done Sent false
9773 3484 Acked Read 7 Done Done Sent false
9774 3485 Acked Read 7 Done Done Sent false
9775 3486 Acked Read 6 Done Done Sent false
9776 3487 Acked Write 3 Done Done Sent false
9777 3488 Acked Read 4 Done Done Sent false
9778 3489 Acked Read 5 Done Done Sent false
9779 3490 Acked Write 6 Done Done Sent false
9780 3491 Acked Write 8 Done Done Sent false
9781 3492 Acked Write 3 Done Done Sent false
9782 3493 Acked Read 5 Done Done Sent false
9783 3494 Acked Write 1 Done Done Sent false
9784 3495 Acked Read 8 Done Done Sent false
9785 3496 Acked Read 10 Done Done Sent false
9786 3497 Acked Read 6 Done Done Sent false
9787 3498 Acked Write 5 Done Done Sent false
9788 3499 Acked Write 3 Done Done Sent false
9789 3500 Acked Read 1 Done Done Sent false
9790 3501 Acked Read 8 Done Done Sent false
9791 3502 Acked Write 2 Done Done Sent false
9792 3503 Acked Write 6 Done Done Sent false
9793 3504 Acked Write 6 Done Done Sent false
9794 3505 Acked Flush 0 Done Done Sent false
9795 3506 Acked Write 2 Done Done Sent false
9796 3507 Acked Write 8 Done Done Sent false
9797 3508 Acked Read 7 Done Done Sent false
9798 3509 Acked Read 4 Done Done Sent false
9799 3510 Acked Read 2 Done Done Sent false
9800 3511 Acked Read 10 Done Done Sent false
9801 3512 Acked Write 4 Done Done Sent false
9802 3513 Acked Write 8 Done Done Sent false
9803 3514 Acked Read 1 Done Done Sent false
9804 3515 Acked Read 10 Done Done Sent false
9805 3516 Acked Read 7 Done Done Sent false
9806 3517 Acked Flush 0 Done Done Sent false
9807 3518 Acked Write 7 Done Done Sent false
9808 3519 Acked Write 4 Done Done Sent false
9809 3520 Acked Write 10 Done Done Sent false
9810 3521 Acked Write 5 Done Done Sent false
9811 3522 Acked Read 10 Done Done Sent false
9812 3523 Acked Flush 0 Done Done Sent false
9813 3524 Acked Read 2 Done Done Sent false
9814 3525 Acked Write 10 Done Done Sent false
9815 3526 Acked Read 6 Done Done Sent false
9816 3527 Acked Write 10 Done Done Sent false
9817 3528 Acked Read 7 Done Done Sent false
9818 3529 Acked Read 7 Done Done Sent false
9819 3530 Acked Read 5 Done Done Sent false
9820 3531 Acked Read 2 Done Done Sent false
9821 3532 Acked Write 1 Done Done Sent false
9822 3533 Acked Read 3 Done Done Sent false
9823 3534 Acked Write 7 Done Done Sent false
9824 3535 Acked Flush 0 Done Done Sent false
9825 3536 Acked Flush 0 Done Done Sent false
9826 3537 Acked Read 2 Done Done Sent false
9827 3538 Acked Read 6 Done Done Sent false
9828 3539 Acked Read 2 Done Done Sent false
9829 3540 Acked Write 4 Done Done Sent false
9830 3541 Acked Read 1 Done Done Sent false
9831 3542 Acked Read 8 Done Done Sent false
9832 3543 Acked Read 8 Done Done Sent false
9833 3544 Acked Read 3 Done Done Sent false
9834 3545 Acked Flush 0 Done Done Sent false
9835 3546 Acked Write 8 Done Done Sent false
9836 3547 Acked Flush 0 Done Done Sent false
9837 3548 Acked Write 4 Done Done Sent false
9838 3549 Acked Write 7 Done Done Sent false
9839 3550 Acked Flush 0 Done Done Sent false
9840 3551 Acked Write 5 Done Done Sent false
9841 3552 Acked Read 10 Done Done Sent false
9842 3553 Acked Read 10 Done Done Sent false
9843 3554 Acked Read 9 Done Done Sent false
9844 3555 Acked Read 8 Done Done Sent false
9845 3556 Acked Read 1 Done Done Sent false
9846 3557 Acked Write 4 Done Done Sent false
9847 3558 Acked Flush 0 Done Done Sent false
9848 3559 Acked Read 8 Done Done Sent false
9849 3560 Acked Write 2 Done Done Sent false
9850 3561 Acked Read 3 Done Done Sent false
9851 3562 Acked Read 1 Done Done Sent false
9852 3563 Acked Write 7 Done Done Sent false
9853 3564 Acked Write 3 Done Done Sent false
9854 3565 Acked Read 4 Done Done Sent false
9855 3566 Acked Write 9 Done Done Sent false
9856 3567 Acked Write 9 Done Done Sent false
9857 3568 Acked Flush 0 Done Done Sent false
9858 3569 Acked Write 10 Done Done Sent false
9859 3570 Acked Read 10 Done Done Sent false
9860 3571 Acked Read 7 Done Done Sent false
9861 3572 Acked Read 6 Done Done Sent false
9862 3573 Acked Write 5 Done Done Sent false
9863 3574 Acked Write 6 Done Done Sent false
9864 3575 Acked Flush 0 Done Done Sent false
9865 3576 Acked Read 3 Done Done Sent false
9866 3577 Acked Write 4 Done Done Sent false
9867 3578 Acked Write 7 Done Done Sent false
9868 3579 Acked Flush 0 Done Done Sent false
9869 3580 Acked Write 2 Done Done Sent false
9870 3581 Acked Write 5 Done Done Sent false
9871 3582 Acked Read 9 Done Done Sent false
9872 3583 Acked Read 7 Done Done Sent false
9873 3584 Acked Write 3 Done Done Sent false
9874 3585 Acked Read 7 Done Done Sent false
9875 3586 Acked Read 3 Done Done Sent false
9876 3587 Acked Write 9 Done Done Sent false
9877 3588 Acked Read 6 Done Done Sent false
9878 3589 Acked Flush 0 Done Done Sent false
9879 3590 Acked Flush 0 Done Done Sent false
9880 3591 Acked Read 3 Done Done Sent false
9881 3592 Acked Write 6 Done Done Sent false
9882 3593 Acked Read 10 Done Done Sent false
9883 3594 Acked Write 1 Done Done Sent false
9884 3595 Acked Read 5 Done Done Sent false
9885 3596 Acked Flush 0 Done Done Sent false
9886 3597 Acked Write 6 Done Done Sent false
9887 3598 Acked Read 2 Done Done Sent false
9888 3599 Acked Read 3 Done Done Sent false
9889 3600 Acked Flush 0 Done Done Sent false
9890 3601 Acked Read 1 Done Done Sent false
9891 3602 Acked Write 7 Done Done Sent false
9892 3603 Acked Read 7 Done Done Sent false
9893 3604 Acked Write 1 Done Done Sent false
9894 3605 Acked Read 6 Done Done Sent false
9895 3606 Acked Write 9 Done Done Sent false
9896 3607 Acked Write 2 Done Done Sent false
9897 3608 Acked Read 2 Done Done Sent false
9898 3609 Acked Read 3 Done Done Sent false
9899 3610 Acked Flush 0 Done Done Sent false
9900 3611 Acked Write 6 Done Done Sent false
9901 3612 Acked Read 2 Done Done Sent false
9902 3613 Acked Write 3 Done Done Sent false
9903 3614 Acked Read 1 Done Done Sent false
9904 3615 Acked Write 10 Done Done Sent false
9905 3616 Acked Read 8 Done Done Sent false
9906 3617 Acked Read 5 Done Done Sent false
9907 3618 Acked Read 6 Done Done Sent false
9908 3619 Acked Flush 0 Done Done Sent false
9909 3620 Acked Write 6 Done Done Sent false
9910 3621 Acked Write 7 Done Done Sent false
9911 3622 Acked Flush 0 Done Done Sent false
9912 3623 Acked Write 8 Done Done Sent false
9913 3624 Acked Read 7 Done Done Sent false
9914 3625 Acked Write 3 Done Done Sent false
9915 3626 Acked Read 5 Done Done Sent false
9916 3627 Acked Read 7 Done Done Sent false
9917 3628 Acked Write 7 Done Done Sent false
9918 3629 Acked Write 6 Done Done Sent false
9919 3630 Acked Write 7 Done Done Sent false
9920 3631 Acked Read 10 Done Done Sent false
9921 3632 Acked Write 5 Done Done Sent false
9922 3633 Acked Read 1 Done Done Sent false
9923 3634 Acked Read 6 Done Done Sent false
9924 3635 Acked Read 4 Done Done Sent false
9925 3636 Acked Read 5 Done Done Sent false
9926 3637 Acked Write 1 Done Done Sent false
9927 3638 Acked Read 6 Done Done Sent false
9928 3639 Acked Read 9 Done Done Sent false
9929 3640 Acked Read 3 Done Done Sent false
9930 3641 Acked Read 1 Done Done Sent false
9931 3642 Acked Write 5 Done Done Sent false
9932 3643 Acked Read 1 Done Done Sent false
9933 3644 Acked Flush 0 Done Done Sent false
9934 3645 Acked Write 3 Done Done Sent false
9935 3646 Acked Write 8 Done Done Sent false
9936 3647 Acked Read 6 Done Done Sent false
9937 3648 Acked Write 9 Done Done Sent false
9938 3649 Acked Write 10 Done Done Sent false
9939 3650 Acked Read 7 Done Done Sent false
9940 3651 Acked Read 6 Done Done Sent false
9941 3652 Acked Read 10 Done Done Sent false
9942 3653 Acked Read 2 Done Done Sent false
9943 3654 Acked Read 5 Done Done Sent false
9944 3655 Acked Read 7 Done Done Sent false
9945 3656 Acked Read 10 Done Done Sent false
9946 3657 Acked Read 8 Done Done Sent false
9947 3658 Acked Read 1 Done Done Sent false
9948 3659 Acked Write 4 Done Done Sent false
9949 3660 Acked Read 5 Done Done Sent false
9950 3661 Acked Write 7 Done Done Sent false
9951 3662 Acked Flush 0 Done Done Sent false
9952 3663 Acked Write 7 Done Done Sent false
9953 3664 Acked Write 4 Done Done Sent false
9954 3665 Acked Read 4 Done Done Sent false
9955 3666 Acked Read 9 Done Done Sent false
9956 3667 Acked Write 7 Done Done Sent false
9957 3668 Acked Write 3 Done Done Sent false
9958 3669 Acked Read 4 Done Done Sent false
9959 3670 Acked Write 10 Done Done Sent false
9960 3671 Acked Read 2 Done Done Sent false
9961 3672 Acked Write 2 Done Done Sent false
9962 3673 Acked Read 3 Done Done Sent false
9963 3674 Acked Read 1 Done Done Sent false
9964 3675 Acked Read 8 Done Done Sent false
9965 3676 Acked Read 10 Done Done Sent false
9966 3677 Acked Write 8 Done Done Sent false
9967 3678 Acked Read 7 Done Done Sent false
9968 3679 Acked Read 2 Done Done Sent false
9969 3680 Acked Write 7 Done Done Sent false
9970 3681 Acked Read 6 Done Done Sent false
9971 3682 Acked Read 4 Done Done Sent false
9972 3683 Acked Read 2 Done Done Sent false
9973 3684 Acked Write 6 Done Done Sent false
9974 3685 Acked Read 1 Done Done Sent false
9975 3686 Acked Write 8 Done Done Sent false
9976 3687 Acked Read 3 Done Done Sent false
9977 3688 Acked Write 4 Done Done Sent false
9978 3689 Acked Write 3 Done Done Sent false
9979 3690 Acked Flush 0 Done Done Sent false
9980 3691 Acked Read 7 Done Done Sent false
9981 3692 Acked Read 7 Done Done Sent false
9982 3693 Acked Write 8 Done Done Sent false
9983 3694 Acked Read 9 Done Done Sent false
9984 3695 Acked Write 9 Done Done Sent false
9985 3696 Acked Read 1 Done Done Sent false
9986 3697 Acked Write 6 Done Done Sent false
9987 3698 Acked Read 9 Done Done Sent false
9988 3699 Acked Write 5 Done Done Sent false
9989 3700 Acked Read 5 Done Done Sent false
9990 3701 Acked Read 1 Done Done Sent false
9991 3702 Acked Read 3 Done Done Sent false
9992 3703 Acked Read 6 Done Done Sent false
9993 3704 Acked Write 5 Done Done Sent false
9994 3705 Acked Flush 0 Done Done Sent false
9995 3706 Acked Read 6 Done Done Sent false
9996 3707 Acked Read 7 Done Done Sent false
9997 3708 Acked Flush 0 Done Done Sent false
9998 3709 Acked Read 10 Done Done Sent false
9999 3710 Acked Read 3 Done Done Sent false
10000 3711 Acked Read 4 Done Done Sent false
10001 3712 Acked Flush 0 Done Done Sent false
10002 3713 Acked Read 5 Done Done Sent false
10003 3714 Acked Read 1 Done Done Sent false
10004 3715 Acked Write 10 Done Done Sent false
10005 3716 Acked Read 4 Done Done Sent false
10006 STATES DS:0 DS:1 DS:2 TOTAL
10007 Sent 0 0 301 301
10008 Done 301 301 0 602
10009 Skipped 0 0 0 0
10010 Error 0 0 0 0
10011 Last Flush: Some(JobId(3712)) Some(JobId(3712)) Some(JobId(3415))
10012 Downstairs last five completed: 3415 3414 3413 3412 3411
10013 Upstairs last five completed: 3716 3715 3714 3713 3712
10014 CLIENT: Up:0 ds:301 act:2
100152025-02-25T17:20:25.490ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
10016 [8830] got start action
10017 [8830] I am going to restart
10018 Starting downstairs at port 8830
10019 Make output file at "/tmp/dsc/downstairs-8830.txt"
10020 [8830][2] reports Starting
10021 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1217)
10022 [8830][2] reports Running
100232025-02-25T17:20:26.492ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
100242025-02-25T17:20:27.494ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
10025 ----------------------------------------------------------------
10026 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10027 States: Active Active Offline
10028 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10029 3416 Acked Read 8 Done Done Sent false
10030 3417 Acked Read 8 Done Done Sent false
10031 3418 Acked Read 7 Done Done Sent false
10032 3419 Acked Write 2 Done Done Sent false
10033 3420 Acked Read 5 Done Done Sent false
10034 3421 Acked Write 6 Done Done Sent false
10035 3422 Acked Read 9 Done Done Sent false
10036 3423 Acked Read 9 Done Done Sent false
10037 3424 Acked Read 8 Done Done Sent false
10038 3425 Acked Flush 0 Done Done Sent false
10039 3426 Acked Read 4 Done Done Sent false
10040 3427 Acked Write 1 Done Done Sent false
10041 3428 Acked Read 7 Done Done Sent false
10042 3429 Acked Flush 0 Done Done Sent false
10043 3430 Acked Read 4 Done Done Sent false
10044 3431 Acked Write 3 Done Done Sent false
10045 3432 Acked Read 7 Done Done Sent false
10046 3433 Acked Write 9 Done Done Sent false
10047 3434 Acked Write 7 Done Done Sent false
10048 3435 Acked Read 3 Done Done Sent false
10049 3436 Acked Read 8 Done Done Sent false
10050 3437 Acked Read 8 Done Done Sent false
10051 3438 Acked Read 6 Done Done Sent false
10052 3439 Acked Write 3 Done Done Sent false
10053 3440 Acked Read 9 Done Done Sent false
10054 3441 Acked Write 1 Done Done Sent false
10055 3442 Acked Write 8 Done Done Sent false
10056 3443 Acked Read 3 Done Done Sent false
10057 3444 Acked Write 9 Done Done Sent false
10058 3445 Acked Read 9 Done Done Sent false
10059 3446 Acked Read 7 Done Done Sent false
10060 3447 Acked Write 5 Done Done Sent false
10061 3448 Acked Write 8 Done Done Sent false
10062 3449 Acked Read 3 Done Done Sent false
10063 3450 Acked Write 7 Done Done Sent false
10064 3451 Acked Read 3 Done Done Sent false
10065 3452 Acked Write 2 Done Done Sent false
10066 3453 Acked Read 8 Done Done Sent false
10067 3454 Acked Write 9 Done Done Sent false
10068 3455 Acked Read 3 Done Done Sent false
10069 3456 Acked Read 8 Done Done Sent false
10070 3457 Acked Read 2 Done Done Sent false
10071 3458 Acked Read 3 Done Done Sent false
10072 3459 Acked Flush 0 Done Done Sent false
10073 3460 Acked Write 1 Done Done Sent false
10074 3461 Acked Read 6 Done Done Sent false
10075 3462 Acked Write 10 Done Done Sent false
10076 3463 Acked Read 5 Done Done Sent false
10077 3464 Acked Write 5 Done Done Sent false
10078 3465 Acked Read 10 Done Done Sent false
10079 3466 Acked Read 1 Done Done Sent false
10080 3467 Acked Read 1 Done Done Sent false
10081 3468 Acked Flush 0 Done Done Sent false
10082 3469 Acked Flush 0 Done Done Sent false
10083 3470 Acked Write 7 Done Done Sent false
10084 3471 Acked Flush 0 Done Done Sent false
10085 3472 Acked Read 3 Done Done Sent false
10086 3473 Acked Read 6 Done Done Sent false
10087 3474 Acked Write 1 Done Done Sent false
10088 3475 Acked Write 8 Done Done Sent false
10089 3476 Acked Read 9 Done Done Sent false
10090 3477 Acked Read 8 Done Done Sent false
10091 3478 Acked Write 9 Done Done Sent false
10092 3479 Acked Write 10 Done Done Sent false
10093 3480 Acked Flush 0 Done Done Sent false
10094 3481 Acked Flush 0 Done Done Sent false
10095 3482 Acked Write 6 Done Done Sent false
10096 3483 Acked Write 6 Done Done Sent false
10097 3484 Acked Read 7 Done Done Sent false
10098 3485 Acked Read 7 Done Done Sent false
10099 3486 Acked Read 6 Done Done Sent false
10100 3487 Acked Write 3 Done Done Sent false
10101 3488 Acked Read 4 Done Done Sent false
10102 3489 Acked Read 5 Done Done Sent false
10103 3490 Acked Write 6 Done Done Sent false
10104 3491 Acked Write 8 Done Done Sent false
10105 3492 Acked Write 3 Done Done Sent false
10106 3493 Acked Read 5 Done Done Sent false
10107 3494 Acked Write 1 Done Done Sent false
10108 3495 Acked Read 8 Done Done Sent false
10109 3496 Acked Read 10 Done Done Sent false
10110 3497 Acked Read 6 Done Done Sent false
10111 3498 Acked Write 5 Done Done Sent false
10112 3499 Acked Write 3 Done Done Sent false
10113 3500 Acked Read 1 Done Done Sent false
10114 3501 Acked Read 8 Done Done Sent false
10115 3502 Acked Write 2 Done Done Sent false
10116 3503 Acked Write 6 Done Done Sent false
10117 3504 Acked Write 6 Done Done Sent false
10118 3505 Acked Flush 0 Done Done Sent false
10119 3506 Acked Write 2 Done Done Sent false
10120 3507 Acked Write 8 Done Done Sent false
10121 3508 Acked Read 7 Done Done Sent false
10122 3509 Acked Read 4 Done Done Sent false
10123 3510 Acked Read 2 Done Done Sent false
10124 3511 Acked Read 10 Done Done Sent false
10125 3512 Acked Write 4 Done Done Sent false
10126 3513 Acked Write 8 Done Done Sent false
10127 3514 Acked Read 1 Done Done Sent false
10128 3515 Acked Read 10 Done Done Sent false
10129 3516 Acked Read 7 Done Done Sent false
10130 3517 Acked Flush 0 Done Done Sent false
10131 3518 Acked Write 7 Done Done Sent false
10132 3519 Acked Write 4 Done Done Sent false
10133 3520 Acked Write 10 Done Done Sent false
10134 3521 Acked Write 5 Done Done Sent false
10135 3522 Acked Read 10 Done Done Sent false
10136 3523 Acked Flush 0 Done Done Sent false
10137 3524 Acked Read 2 Done Done Sent false
10138 3525 Acked Write 10 Done Done Sent false
10139 3526 Acked Read 6 Done Done Sent false
10140 3527 Acked Write 10 Done Done Sent false
10141 3528 Acked Read 7 Done Done Sent false
10142 3529 Acked Read 7 Done Done Sent false
10143 3530 Acked Read 5 Done Done Sent false
10144 3531 Acked Read 2 Done Done Sent false
10145 3532 Acked Write 1 Done Done Sent false
10146 3533 Acked Read 3 Done Done Sent false
10147 3534 Acked Write 7 Done Done Sent false
10148 3535 Acked Flush 0 Done Done Sent false
10149 3536 Acked Flush 0 Done Done Sent false
10150 3537 Acked Read 2 Done Done Sent false
10151 3538 Acked Read 6 Done Done Sent false
10152 3539 Acked Read 2 Done Done Sent false
10153 3540 Acked Write 4 Done Done Sent false
10154 3541 Acked Read 1 Done Done Sent false
10155 3542 Acked Read 8 Done Done Sent false
10156 3543 Acked Read 8 Done Done Sent false
10157 3544 Acked Read 3 Done Done Sent false
10158 3545 Acked Flush 0 Done Done Sent false
10159 3546 Acked Write 8 Done Done Sent false
10160 3547 Acked Flush 0 Done Done Sent false
10161 3548 Acked Write 4 Done Done Sent false
10162 3549 Acked Write 7 Done Done Sent false
10163 3550 Acked Flush 0 Done Done Sent false
10164 3551 Acked Write 5 Done Done Sent false
10165 3552 Acked Read 10 Done Done Sent false
10166 3553 Acked Read 10 Done Done Sent false
10167 3554 Acked Read 9 Done Done Sent false
10168 3555 Acked Read 8 Done Done Sent false
10169 3556 Acked Read 1 Done Done Sent false
10170 3557 Acked Write 4 Done Done Sent false
10171 3558 Acked Flush 0 Done Done Sent false
10172 3559 Acked Read 8 Done Done Sent false
10173 3560 Acked Write 2 Done Done Sent false
10174 3561 Acked Read 3 Done Done Sent false
10175 3562 Acked Read 1 Done Done Sent false
10176 3563 Acked Write 7 Done Done Sent false
10177 3564 Acked Write 3 Done Done Sent false
10178 3565 Acked Read 4 Done Done Sent false
10179 3566 Acked Write 9 Done Done Sent false
10180 3567 Acked Write 9 Done Done Sent false
10181 3568 Acked Flush 0 Done Done Sent false
10182 3569 Acked Write 10 Done Done Sent false
10183 3570 Acked Read 10 Done Done Sent false
10184 3571 Acked Read 7 Done Done Sent false
10185 3572 Acked Read 6 Done Done Sent false
10186 3573 Acked Write 5 Done Done Sent false
10187 3574 Acked Write 6 Done Done Sent false
10188 3575 Acked Flush 0 Done Done Sent false
10189 3576 Acked Read 3 Done Done Sent false
10190 3577 Acked Write 4 Done Done Sent false
10191 3578 Acked Write 7 Done Done Sent false
10192 3579 Acked Flush 0 Done Done Sent false
10193 3580 Acked Write 2 Done Done Sent false
10194 3581 Acked Write 5 Done Done Sent false
10195 3582 Acked Read 9 Done Done Sent false
10196 3583 Acked Read 7 Done Done Sent false
10197 3584 Acked Write 3 Done Done Sent false
10198 3585 Acked Read 7 Done Done Sent false
10199 3586 Acked Read 3 Done Done Sent false
10200 3587 Acked Write 9 Done Done Sent false
10201 3588 Acked Read 6 Done Done Sent false
10202 3589 Acked Flush 0 Done Done Sent false
10203 3590 Acked Flush 0 Done Done Sent false
10204 3591 Acked Read 3 Done Done Sent false
10205 3592 Acked Write 6 Done Done Sent false
10206 3593 Acked Read 10 Done Done Sent false
10207 3594 Acked Write 1 Done Done Sent false
10208 3595 Acked Read 5 Done Done Sent false
10209 3596 Acked Flush 0 Done Done Sent false
10210 3597 Acked Write 6 Done Done Sent false
10211 3598 Acked Read 2 Done Done Sent false
10212 3599 Acked Read 3 Done Done Sent false
10213 3600 Acked Flush 0 Done Done Sent false
10214 3601 Acked Read 1 Done Done Sent false
10215 3602 Acked Write 7 Done Done Sent false
10216 3603 Acked Read 7 Done Done Sent false
10217 3604 Acked Write 1 Done Done Sent false
10218 3605 Acked Read 6 Done Done Sent false
10219 3606 Acked Write 9 Done Done Sent false
10220 3607 Acked Write 2 Done Done Sent false
10221 3608 Acked Read 2 Done Done Sent false
10222 3609 Acked Read 3 Done Done Sent false
10223 3610 Acked Flush 0 Done Done Sent false
10224 3611 Acked Write 6 Done Done Sent false
10225 3612 Acked Read 2 Done Done Sent false
10226 3613 Acked Write 3 Done Done Sent false
10227 3614 Acked Read 1 Done Done Sent false
10228 3615 Acked Write 10 Done Done Sent false
10229 3616 Acked Read 8 Done Done Sent false
10230 3617 Acked Read 5 Done Done Sent false
10231 3618 Acked Read 6 Done Done Sent false
10232 3619 Acked Flush 0 Done Done Sent false
10233 3620 Acked Write 6 Done Done Sent false
10234 3621 Acked Write 7 Done Done Sent false
10235 3622 Acked Flush 0 Done Done Sent false
10236 3623 Acked Write 8 Done Done Sent false
10237 3624 Acked Read 7 Done Done Sent false
10238 3625 Acked Write 3 Done Done Sent false
10239 3626 Acked Read 5 Done Done Sent false
10240 3627 Acked Read 7 Done Done Sent false
10241 3628 Acked Write 7 Done Done Sent false
10242 3629 Acked Write 6 Done Done Sent false
10243 3630 Acked Write 7 Done Done Sent false
10244 3631 Acked Read 10 Done Done Sent false
10245 3632 Acked Write 5 Done Done Sent false
10246 3633 Acked Read 1 Done Done Sent false
10247 3634 Acked Read 6 Done Done Sent false
10248 3635 Acked Read 4 Done Done Sent false
10249 3636 Acked Read 5 Done Done Sent false
10250 3637 Acked Write 1 Done Done Sent false
10251 3638 Acked Read 6 Done Done Sent false
10252 3639 Acked Read 9 Done Done Sent false
10253 3640 Acked Read 3 Done Done Sent false
10254 3641 Acked Read 1 Done Done Sent false
10255 3642 Acked Write 5 Done Done Sent false
10256 3643 Acked Read 1 Done Done Sent false
10257 3644 Acked Flush 0 Done Done Sent false
10258 3645 Acked Write 3 Done Done Sent false
10259 3646 Acked Write 8 Done Done Sent false
10260 3647 Acked Read 6 Done Done Sent false
10261 3648 Acked Write 9 Done Done Sent false
10262 3649 Acked Write 10 Done Done Sent false
10263 3650 Acked Read 7 Done Done Sent false
10264 3651 Acked Read 6 Done Done Sent false
10265 3652 Acked Read 10 Done Done Sent false
10266 3653 Acked Read 2 Done Done Sent false
10267 3654 Acked Read 5 Done Done Sent false
10268 3655 Acked Read 7 Done Done Sent false
10269 3656 Acked Read 10 Done Done Sent false
10270 3657 Acked Read 8 Done Done Sent false
10271 3658 Acked Read 1 Done Done Sent false
10272 3659 Acked Write 4 Done Done Sent false
10273 3660 Acked Read 5 Done Done Sent false
10274 3661 Acked Write 7 Done Done Sent false
10275 3662 Acked Flush 0 Done Done Sent false
10276 3663 Acked Write 7 Done Done Sent false
10277 3664 Acked Write 4 Done Done Sent false
10278 3665 Acked Read 4 Done Done Sent false
10279 3666 Acked Read 9 Done Done Sent false
10280 3667 Acked Write 7 Done Done Sent false
10281 3668 Acked Write 3 Done Done Sent false
10282 3669 Acked Read 4 Done Done Sent false
10283 3670 Acked Write 10 Done Done Sent false
10284 3671 Acked Read 2 Done Done Sent false
10285 3672 Acked Write 2 Done Done Sent false
10286 3673 Acked Read 3 Done Done Sent false
10287 3674 Acked Read 1 Done Done Sent false
10288 3675 Acked Read 8 Done Done Sent false
10289 3676 Acked Read 10 Done Done Sent false
10290 3677 Acked Write 8 Done Done Sent false
10291 3678 Acked Read 7 Done Done Sent false
10292 3679 Acked Read 2 Done Done Sent false
10293 3680 Acked Write 7 Done Done Sent false
10294 3681 Acked Read 6 Done Done Sent false
10295 3682 Acked Read 4 Done Done Sent false
10296 3683 Acked Read 2 Done Done Sent false
10297 3684 Acked Write 6 Done Done Sent false
10298 3685 Acked Read 1 Done Done Sent false
10299 3686 Acked Write 8 Done Done Sent false
10300 3687 Acked Read 3 Done Done Sent false
10301 3688 Acked Write 4 Done Done Sent false
10302 3689 Acked Write 3 Done Done Sent false
10303 3690 Acked Flush 0 Done Done Sent false
10304 3691 Acked Read 7 Done Done Sent false
10305 3692 Acked Read 7 Done Done Sent false
10306 3693 Acked Write 8 Done Done Sent false
10307 3694 Acked Read 9 Done Done Sent false
10308 3695 Acked Write 9 Done Done Sent false
10309 3696 Acked Read 1 Done Done Sent false
10310 3697 Acked Write 6 Done Done Sent false
10311 3698 Acked Read 9 Done Done Sent false
10312 3699 Acked Write 5 Done Done Sent false
10313 3700 Acked Read 5 Done Done Sent false
10314 3701 Acked Read 1 Done Done Sent false
10315 3702 Acked Read 3 Done Done Sent false
10316 3703 Acked Read 6 Done Done Sent false
10317 3704 Acked Write 5 Done Done Sent false
10318 3705 Acked Flush 0 Done Done Sent false
10319 3706 Acked Read 6 Done Done Sent false
10320 3707 Acked Read 7 Done Done Sent false
10321 3708 Acked Flush 0 Done Done Sent false
10322 3709 Acked Read 10 Done Done Sent false
10323 3710 Acked Read 3 Done Done Sent false
10324 3711 Acked Read 4 Done Done Sent false
10325 3712 Acked Flush 0 Done Done Sent false
10326 3713 Acked Read 5 Done Done Sent false
10327 3714 Acked Read 1 Done Done Sent false
10328 3715 Acked Write 10 Done Done Sent false
10329 3716 Acked Read 4 Done Done Sent false
10330 STATES DS:0 DS:1 DS:2 TOTAL
10331 Sent 0 0 301 301
10332 Done 301 301 0 602
10333 Skipped 0 0 0 0
10334 Error 0 0 0 0
10335 Last Flush: Some(JobId(3712)) Some(JobId(3712)) Some(JobId(3415))
10336 Downstairs last five completed: 3415 3414 3413 3412 3411
10337 Upstairs last five completed: 3716 3715 3714 3713 3712
10338 CLIENT: Up:0 ds:301 act:2
103392025-02-25T17:20:28.720ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
103402025-02-25T17:20:29.497ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
103412025-02-25T17:20:29.954ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
103422025-02-25T17:20:29.954ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56887) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
103432025-02-25T17:20:29.955ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
103442025-02-25T17:20:29.956ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
103452025-02-25T17:20:29.956ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
103462025-02-25T17:20:29.956ZINFOcrucible: send last flush ID to this DS: Some(JobId(3415)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
103472025-02-25T17:20:29.956ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
103482025-02-25T17:20:29.956ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3415)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
103492025-02-25T17:20:29.956ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3415)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
10350 ----------------------------------------------------------------
10351 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10352 Downstairs last five completed: 3717 3716 3715 3714 3713
10353 Upstairs last five completed: 3717 3716 3715 3714 3713
10354 CLIENT: Up:0 ds:0 act:3
10355 Replay: All jobs finished
10356 Main task has work to do, go find it
10357 got dsc Stop(1)
10358 stop 1
10359 Feb 25 17:20:32.221 INFO request completed, latency_us: 408, response_code: 204[8820] Got stop action so:false kr:false
10360 , uri: /stop/cid/1, method: POST, req_id: f4b2b5ae-face-4d5e-aca4-c2f757d46190, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
103612025-02-25T17:20:32.222ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
103622025-02-25T17:20:32.222ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
103632025-02-25T17:20:32.222ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
10364 Feb 25 17:20:32.222{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T17:20:32.222378033Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
10365 INFO request completed, latency_us: 185, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2812b743-a661-48e1-a79f-952e1d209dc5, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
10366 [8820] Exited with: ExitStatus(unix_wait_status(9))
10367 [8820][1] reports Exit
10368 Feb 25 17:20:36.224 INFO request completed, latency_us: 222, response_code: 200, uri: /state/cid/1, method: GET, req_id: ea7bb481-e8eb-40fa-a9b9-42ea79ba24e2, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
10369 001/300 Read block 3503 len 4096
10370 002/300 Read block 3988 len 28672
10371 003/300 Read block 2934 len 32768
10372 004/300 Write block 3649 len 32768 data: 2 2 2 2 3 4 4 4
10373 005/300 Read block 2385 len 8192
10374 006/300 Read block 762 len 12288
10375 007/300 Write block 3698 len 20480 data: 3 3 3 3 3
10376 008/300 Write block 4385 len 16384 data: 2 2 2 2
10377 009/300 Read block 587 len 12288
10378 010/300 Read block 65 len 12288
10379 011/300 Read block 3770 len 24576
10380 012/300 Read block 55 len 36864
10381 013/300 Write block 4336 len 16384 data: 4 5 5 5
10382 014/300 Read block 4534 len 16384
10383 015/300 Read block 4348 len 8192
10384 016/300 Read block 117 len 12288
10385 017/300 Write block 882 len 20480 data: 3 4 4 4 5
10386 018/300 Read block 1729 len 36864
10387 019/300 Write block 3646 len 32768 data: 4 3 3 3 3 3 3 4
10388 020/300 Write block 2716 len 20480 data: 3 3 3 3 3
10389 021/300 Write block 2042 len 12288 data: 3 4 3
10390 022/300 Read block 3017 len 20480
10391 023/300 Read block 1347 len 36864
10392 024/300 Read block 111 len 32768
10393 025/300 Read block 4021 len 4096
10394 026/300 Read block 1620 len 12288
10395 027/300 Read block 2124 len 8192
10396 028/300 Write block 1999 len 36864 data: 5 5 4 4 2 2 2 2 2
10397 029/300 Write block 3135 len 24576 data: 3 3 3 3 3 4
10398 030/300 Read block 3681 len 32768
10399 031/300 Write block 1248 len 16384 data: 4 2 2 2
10400 032/300 Read block 3495 len 24576
10401 033/300 Read block 2400 len 24576
10402 034/300 Read block 3212 len 40960
10403 035/300 Flush
10404 036/300 Flush
10405 037/300 Read block 3796 len 36864
10406 038/300 Write block 2917 len 20480 data: 2 3 3 3 3
10407 039/300 Write block 2482 len 24576 data: 3 3 3 3 3 2
10408 040/300 Write block 2004 len 32768 data: 3 3 3 3 2 3 3 3
10409 041/300 Read block 660 len 40960
10410 042/300 Write block 3810 len 16384 data: 2 3 4 4
10411 043/300 Write block 1178 len 4096 data: 6
10412 044/300 Write block 2032 len 20480 data: 4 4 4 4 3
10413 045/300 Write block 2505 len 12288 data: 4 3 3
10414 046/300 Read block 3853 len 36864
10415 047/300 Write block 2392 len 12288 data: 4 4 4
10416 048/300 Write block 2250 len 8192 data: 3 3
10417 049/300 Read block 4359 len 20480
10418 050/300 Write block 3099 len 24576 data: 3 3 2 3 3 3
10419 051/300 Flush
10420 052/300 Write block 4946 len 28672 data: 3 2 2 2 2 2 2
10421 053/300 Write block 683 len 12288 data: 5 5 5
10422 054/300 Write block 3126 len 8192 data: 3 2
10423 055/300 Read block 1752 len 8192
10424 056/300 Read block 1198 len 8192
10425 057/300 Write block 1461 len 40960 data: 2 2 2 2 3 3 4 4 4 4
10426 058/300 Write block 2545 len 16384 data: 3 3 2 2
10427 059/300 Write block 2550 len 12288 data: 2 2 2
10428 060/300 Flush
10429 061/300 Write block 4561 len 32768 data: 3 4 4 3 3 2 2 2
10430 062/300 Read block 2252 len 40960
10431 063/300 Write block 3699 len 36864 data: 4 4 4 4 3 4 4 3 3
10432 064/300 Write block 3902 len 20480 data: 2 2 3 3 3
10433 065/300 Read block 2925 len 16384
10434 066/300 Read block 3064 len 32768
10435 067/300 Write block 833 len 16384 data: 6 7 7 6
10436 068/300 Read block 999 len 8192
10437 069/300 Read block 2720 len 16384
10438 070/300 Write block 334 len 36864 data: 4 4 4 4 3 2 2 2 2
10439 071/300 Write block 1580 len 16384 data: 3 4 3 3
10440 072/300 Read block 3621 len 40960
10441 073/300 Write block 2787 len 28672 data: 4 3 3 3 4 4 4
10442 074/300 Flush
10443 075/300 Write block 3349 len 40960 data: 3 2 2 5 5 5 5 5 4 4
10444 076/300 Write block 4022 len 16384 data: 5 4 4 5
10445 077/300 Read block 2065 len 24576
10446 078/300 Read block 763 len 16384
10447 079/300 Write block 2018 len 12288 data: 2 2 2
10448 080/300 Write block 4559 len 8192 data: 3 3
10449 081/300 Flush
10450 082/300 Write block 2988 len 36864 data: 2 2 2 2 3 2 4 4 3
10451 083/300 Read block 2962 len 28672
10452 084/300 Read block 4194 len 20480
10453 085/300 Write block 254 len 8192 data: 3 3
10454 086/300 Write block 950 len 32768 data: 2 3 2 2 2 2 2 2
10455 087/300 Write block 1426 len 28672 data: 3 4 4 4 5 5 3
10456 088/300 Read block 2552 len 20480
10457 089/300 Flush
10458 090/300 Write block 239 len 32768 data: 2 2 2 2 2 2 2 2
10459 091/300 Read block 2085 len 20480
10460 092/300 Write block 1104 len 12288 data: 2 2 4
10461 093/300 Read block 1803 len 4096
10462 094/300 Read block 866 len 28672
10463 095/300 Read block 3278 len 16384
10464 096/300 Write block 356 len 24576 data: 4 4 3 3 3 3
10465 097/300 Read block 2173 len 4096
10466 098/300 Write block 4599 len 28672 data: 3 3 3 4 4 4 3
10467 099/300 Write block 1705 len 8192 data: 4 4
10468 100/300 Read block 4347 len 16384
10469 101/300 Write block 161 len 16384 data: 3 4 4 4
10470 102/300 Read block 954 len 20480
10471 103/300 Write block 2962 len 20480 data: 3 3 4 5 6
10472 104/300 Read block 3830 len 4096
10473 105/300 Write block 2551 len 16384 data: 3 3 3 3
10474 106/300 Flush
10475 107/300 Read block 2924 len 20480
10476 108/300 Write block 3311 len 20480 data: 2 2 3 3 3
10477 109/300 Read block 1371 len 8192
10478 110/300 Read block 1645 len 20480
10479 111/300 Read block 4018 len 24576
10480 112/300 Read block 4921 len 4096
10481 113/300 Write block 3774 len 36864 data: 3 2 3 3 2 2 2 2 2
10482 114/300 Write block 3813 len 8192 data: 5 5
10483 115/300 Write block 441 len 8192 data: 3 3
10484 116/300 Read block 4143 len 32768
10485 117/300 Write block 2773 len 16384 data: 4 4 4 4
10486 118/300 Read block 43 len 36864
10487 119/300 Read block 3852 len 8192
10488 120/300 Write block 2176 len 12288 data: 2 3 3
10489 121/300 Read block 1528 len 8192
10490 122/300 Read block 2179 len 12288
10491 123/300 Flush
10492 124/300 Read block 3110 len 8192
10493 125/300 Write block 224 len 4096 data: 2
10494 126/300 Write block 4386 len 28672 data: 3 3 3 3 3 4 4
10495 127/300 Flush
10496 128/300 Write block 1219 len 24576 data: 2 2 2 2 2 2
10497 129/300 Read block 836 len 4096
10498 130/300 Read block 394 len 20480
10499 131/300 Write block 3766 len 32768 data: 3 3 2 3 3 3 3 3
10500 132/300 Read block 2225 len 24576
10501 133/300 Write block 2941 len 12288 data: 3 3 4
10502 134/300 Read block 2286 len 8192
10503 135/300 Read block 3661 len 4096
10504 136/300 Write block 2036 len 40960 data: 4 5 5 4 3 3 4 5 4 3
10505 137/300 Read block 1190 len 16384
10506 138/300 Write block 2794 len 24576 data: 3 3 3 4 5 4
10507 139/300 Write block 3120 len 20480 data: 3 4 4 4 3
10508 140/300 Write block 4623 len 28672 data: 3 3 3 3 2 4 4
10509 141/300 Read block 4526 len 32768
10510 142/300 Flush
10511 143/300 Write block 4172 len 20480 data: 3 3 2 2 3
10512 144/300 Write block 4643 len 4096 data: 3
10513 145/300 Write block 1087 len 8192 data: 2 3
10514 146/300 Read block 4544 len 12288
10515 147/300 Flush
10516 148/300 Write block 322 len 32768 data: 3 3 2 2 2 2 2 2
10517 149/300 Write block 3689 len 4096 data: 4
10518 150/300 Write block 748 len 8192 data: 5 5
10519 151/300 Flush
10520 152/300 Read block 981 len 12288
10521 153/300 Read block 1012 len 8192
10522 154/300 Read block 4032 len 12288
10523 155/300 Write block 573 len 40960 data: 3 3 3 2 2 2 2 4 4 3
10524 156/300 Read block 1873 len 12288
10525 157/300 Write block 617 len 28672 data: 2 3 3 3 5 5 5
10526 158/300 Write block 4672 len 8192 data: 4 4
10527 159/300 Read block 3390 len 36864
10528 160/300 Read block 3590 len 8192
10529 161/300 Write block 3736 len 36864 data: 4 4 4 4 2 2 2 2 2
10530 162/300 Read block 3158 len 12288
10531 163/300 Read block 33 len 24576
10532 164/300 Read block 2783 len 8192
10533 165/300 Read block 4724 len 8192
10534 166/300 Write block 4157 len 12288 data: 3 3 3
10535 167/300 Read block 1231 len 36864
10536 168/300 Write block 401 len 28672 data: 3 3 3 3 4 3 3
10537 169/300 Read block 3770 len 32768
10538 170/300 Read block 2663 len 8192
10539 171/300 Write block 2751 len 20480 data: 4 3 2 2 2
10540 172/300 Read block 2913 len 16384
10541 173/300 Read block 1854 len 12288
10542 174/300 Read block 1260 len 28672
10543 175/300 Write block 1274 len 12288 data: 2 2 2
10544 176/300 Read block 4418 len 28672
10545 177/300 Read block 4631 len 16384
10546 178/300 Read block 1898 len 4096
10547 179/300 Write block 4148 len 12288 data: 2 2 2
10548 180/300 Read block 4879 len 8192
10549 181/300 Write block 3271 len 20480 data: 2 2 2 2 2
10550 182/300 Read block 915 len 24576
10551 183/300 Read block 4281 len 24576
10552 184/300 Read block 990 len 20480
10553 185/300 Read block 383 len 36864
10554 186/300 Read block 4602 len 8192
10555 187/300 Read block 3630 len 32768
10556 188/300 Read block 1186 len 32768
10557 189/300 Write block 1933 len 4096 data: 5
10558 190/300 Write block 476 len 32768 data: 2 2 3 3 3 3 3 3
10559 191/300 Read block 264 len 20480
10560 192/300 Write block 3668 len 12288 data: 4 5 5
10561 193/300 Write block 1380 len 4096 data: 3
10562 194/300 Read block 4616 len 24576
10563 195/300 Write block 2881 len 24576 data: 5 3 3 3 3 3
10564 196/300 Read block 1901 len 36864
10565 197/300 Read block 612 len 12288
10566 198/300 Write block 2302 len 40960 data: 2 4 4 3 2 2 2 2 2 2
10567 199/300 Read block 2343 len 36864
10568 200/300 Read block 3142 len 40960
10569 201/300 Read block 1850 len 32768
10570 202/300 Read block 2418 len 12288
10571 203/300 Write block 2269 len 4096 data: 2
10572 204/300 Read block 484 len 36864
10573 205/300 Read block 2334 len 4096
10574 206/300 Read block 1128 len 12288
10575 207/300 Read block 2135 len 12288
10576 208/300 Read block 1779 len 12288
10577 209/300 Read block 4050 len 20480
10578 210/300 Read block 1302 len 40960
10579 211/300 Write block 2763 len 40960 data: 2 2 3 3 4 4 4 4 4 4
10580 212/300 Read block 4510 len 4096
10581 213/300 Read block 3260 len 24576
10582 214/300 Write block 662 len 24576 data: 2 4 4 3 3 3
10583 215/300 Read block 3195 len 8192
10584 216/300 Read block 1770 len 8192
10585 217/300 Write block 3073 len 32768 data: 3 2 2 3 2 2 2 3
10586 218/300 Read block 3424 len 28672
10587 219/300 Read block 2747 len 32768
10588 220/300 Write block 660 len 8192 data: 2 2
10589 221/300 Flush
10590 222/300 Flush
10591 223/300 Read block 2017 len 12288
10592 224/300 Flush
10593 225/300 Read block 4526 len 28672
10594 226/300 Read block 255 len 8192
10595 227/300 Write block 2422 len 16384 data: 2 2 2 2
10596 228/300 Read block 2038 len 12288
10597 229/300 Write block 1133 len 4096 data: 2
10598 230/300 Write block 1064 len 12288 data: 4 3 3
10599 231/300 Read block 438 len 16384
10600 232/300 Write block 4158 len 12288 data: 4 4 3
10601 233/300 Read block 3609 len 28672
10602 234/300 Flush
10603 235/300 Read block 621 len 32768
10604 236/300 Read block 4275 len 12288
10605 237/300 Write block 3998 len 20480 data: 3 3 3 3 3
10606 238/300 Write block 3487 len 24576 data: 4 4 4 4 4 5
10607 239/300 Read block 1704 len 28672
10608 240/300 Read block 4858 len 16384
10609 241/300 Write block 4974 len 12288 data: 3 3 3
10610 242/300 Read block 3414 len 16384
10611 243/300 Read block 3802 len 12288
10612 244/300 Write block 3655 len 40960 data: 5 5 4 5 5 4 4 4 4 4
10613 245/300 Write block 124 len 20480 data: 3 3 3 3 3
10614 246/300 Write block 2392 len 20480 data: 5 5 5 4 3
10615 247/300 Read block 1522 len 32768
10616 248/300 Write block 4931 len 40960 data: 2 2 3 3 3 3 3 3 5 5
10617 249/300 Write block 4452 len 12288 data: 2 2 2
10618 250/300 Write block 2404 len 20480 data: 2 4 4 4 3
10619 251/300 Write block 4032 len 4096 data: 3
10620 252/300 Read block 2622 len 16384
10621 253/300 Read block 4621 len 24576
10622 254/300 Write block 1697 len 16384 data: 2 2 2 3
10623 255/300 Write block 1554 len 24576 data: 3 3 3 3 3 2
10624 256/300 Read block 1433 len 4096
10625 257/300 Read block 3399 len 20480
10626 258/300 Read block 3357 len 36864
10627 259/300 Read block 4965 len 16384
10628 260/300 Read block 1750 len 12288
10629 261/300 Read block 3914 len 4096
10630 262/300 Read block 2216 len 4096
10631 263/300 Read block 892 len 4096
10632 264/300 Read block 1363 len 32768
10633 265/300 Write block 307 len 24576 data: 3 3 3 3 2 2
10634 266/300 Read block 3734 len 32768
10635 267/300 Read block 540 len 32768
10636 268/300 Write block 3525 len 8192 data: 2 2
10637 269/300 Write block 2298 len 20480 data: 3 3 3 2 3
10638 270/300 Flush
10639 271/300 Write block 833 len 36864 data: 7 8 8 7 6 5 5 4 3
10640 272/300 Read block 3113 len 20480
10641 273/300 Write block 1625 len 16384 data: 4 4 3 4
10642 274/300 Read block 4150 len 12288
10643 275/300 Read block 1396 len 12288
10644 276/300 Flush
10645 277/300 Read block 4551 len 12288
10646 278/300 Write block 3081 len 16384 data: 3 4 4 4
10647 279/300 Flush
10648 280/300 Read block 1713 len 12288
10649 281/300 Read block 445 len 28672
10650 282/300 Write block 304 len 4096 data: 3
10651 283/300 Write block 97 len 16384 data: 3 3 4 3
10652 284/300 Read block 4922 len 8192
10653 285/300 Read block 4126 len 8192
10654 286/300 Read block 4558 len 8192
10655 287/300 Read block 3547 len 20480
10656 288/300 Flush
10657 289/300 Read block 1292 len 4096
10658 290/300 Flush
10659 291/300 Write block 3350 len 12288 data: 3 3 6
10660 292/300 Write block 3766 len 36864 data: 4 4 3 4 4 4 4 4 4
10661 293/300 Write block 2493 len 40960 data: 3 2 2 2 2 2 3 3 3 3
10662 294/300 Read block 822 len 36864
10663 295/300 Write block 2901 len 36864 data: 6 7 7 6 6 5 5 4 2
10664 296/300 Read block 3513 len 24576
10665 297/300 Write block 741 len 12288 data: 4 4 4
10666 298/300 Write block 1487 len 12288 data: 4 4 5
10667 299/300 Read block 4962 len 16384
10668 300/300 Read block 352 len 20480
10669 301/300 Write block 3986 len 4096 data: 2
10670 Main task has work to do, go find it
10671 got dsc Start(1)
10672 start 1
10673 Feb 25 17:20:36.469 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/1, method: POST, req_id: e7fac3a4-b88b-4291-9b5f-28b10919e0c1, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
10674 Replay: started 1, returned:Ok(())
10675 ----------------------------------------------------------------
10676 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10677 States: Active Offline Active
10678 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10679 3718 Acked Read 1 Done Sent Done false
10680 3719 Acked Read 7 Done Sent Done false
10681 3720 Acked Read 8 Done Sent Done false
10682 3721 Acked Write 8 Done Sent Done false
10683 3722 Acked Read 2 Done Sent Done false
10684 3723 Acked Read 3 Done Sent Done false
10685 3724 Acked Write 5 Done Sent Done false
10686 3725 Acked Write 4 Done Sent Done false
10687 3726 Acked Read 3 Done Sent Done false
10688 3727 Acked Read 3 Done Sent Done false
10689 3728 Acked Read 6 Done Sent Done false
10690 3729 Acked Read 9 Done Sent Done false
10691 3730 Acked Write 4 Done Sent Done false
10692 3731 Acked Read 4 Done Sent Done false
10693 3732 Acked Read 2 Done Sent Done false
10694 3733 Acked Read 3 Done Sent Done false
10695 3734 Acked Write 5 Done Sent Done false
10696 3735 Acked Read 9 Done Sent Done false
10697 3736 Acked Write 8 Done Sent Done false
10698 3737 Acked Write 5 Done Sent Done false
10699 3738 Acked Write 3 Done Sent Done false
10700 3739 Acked Read 5 Done Sent Done false
10701 3740 Acked Read 9 Done Sent Done false
10702 3741 Acked Read 8 Done Sent Done false
10703 3742 Acked Read 1 Done Sent Done false
10704 3743 Acked Read 3 Done Sent Done false
10705 3744 Acked Read 2 Done Sent Done false
10706 3745 Acked Write 9 Done Sent Done false
10707 3746 Acked Write 6 Done Sent Done false
10708 3747 Acked Read 8 Done Sent Done false
10709 3748 Acked Write 4 Done Sent Done false
10710 3749 Acked Read 6 Done Sent Done false
10711 3750 Acked Read 6 Done Sent Done false
10712 3751 Acked Read 10 Done Sent Done false
10713 3752 Acked Flush 0 Done Sent Done false
10714 3753 Acked Flush 0 Done Sent Done false
10715 3754 Acked Read 9 Done Sent Done false
10716 3755 Acked Write 5 Done Sent Done false
10717 3756 Acked Write 6 Done Sent Done false
10718 3757 Acked Write 8 Done Sent Done false
10719 3758 Acked Read 10 Done Sent Done false
10720 3759 Acked Write 4 Done Sent Done false
10721 3760 Acked Write 1 Done Sent Done false
10722 3761 Acked Write 5 Done Sent Done false
10723 3762 Acked Write 3 Done Sent Done false
10724 3763 Acked Read 9 Done Sent Done false
10725 3764 Acked Write 3 Done Sent Done false
10726 3765 Acked Write 2 Done Sent Done false
10727 3766 Acked Read 5 Done Sent Done false
10728 3767 Acked Write 6 Done Sent Done false
10729 3768 Acked Flush 0 Done Sent Done false
10730 3769 Acked Write 7 Done Sent Done false
10731 3770 Acked Write 3 Done Sent Done false
10732 3771 Acked Write 2 Done Sent Done false
10733 3772 Acked Read 2 Done Sent Done false
10734 3773 Acked Read 2 Done Sent Done false
10735 3774 Acked Write 10 Done Sent Done false
10736 3775 Acked Write 4 Done Sent Done false
10737 3776 Acked Write 3 Done Sent Done false
10738 3777 Acked Flush 0 Done Sent Done false
10739 3778 Acked Write 8 Done Sent Done false
10740 3779 Acked Read 10 Done Sent Done false
10741 3780 Acked Write 9 Done Sent Done false
10742 3781 Acked Write 5 Done Sent Done false
10743 3782 Acked Read 4 Done Sent Done false
10744 3783 Acked Read 8 Done Sent Done false
10745 3784 Acked Write 4 Done Sent Done false
10746 3785 Acked Read 2 Done Sent Done false
10747 3786 Acked Read 4 Done Sent Done false
10748 3787 Acked Write 9 Done Sent Done false
10749 3788 Acked Write 4 Done Sent Done false
10750 3789 Acked Read 10 Done Sent Done false
10751 3790 Acked Write 7 Done Sent Done false
10752 3791 Acked Flush 0 Done Sent Done false
10753 3792 Acked Write 10 Done Sent Done false
10754 3793 Acked Write 4 Done Sent Done false
10755 3794 Acked Read 6 Done Sent Done false
10756 3795 Acked Read 4 Done Sent Done false
10757 3796 Acked Write 3 Done Sent Done false
10758 3797 Acked Write 2 Done Sent Done false
10759 3798 Acked Flush 0 Done Sent Done false
10760 3799 Acked Write 9 Done Sent Done false
10761 3800 Acked Read 7 Done Sent Done false
10762 3801 Acked Read 5 Done Sent Done false
10763 3802 Acked Write 2 Done Sent Done false
10764 3803 Acked Write 8 Done Sent Done false
10765 3804 Acked Write 7 Done Sent Done false
10766 3805 Acked Read 5 Done Sent Done false
10767 3806 Acked Flush 0 Done Sent Done false
10768 3807 Acked Write 8 Done Sent Done false
10769 3808 Acked Read 5 Done Sent Done false
10770 3809 Acked Write 3 Done Sent Done false
10771 3810 Acked Read 1 Done Sent Done false
10772 3811 Acked Read 7 Done Sent Done false
10773 3812 Acked Read 4 Done Sent Done false
10774 3813 Acked Write 6 Done Sent Done false
10775 3814 Acked Read 1 Done Sent Done false
10776 3815 Acked Write 7 Done Sent Done false
10777 3816 Acked Write 2 Done Sent Done false
10778 3817 Acked Read 4 Done Sent Done false
10779 3818 Acked Write 4 Done Sent Done false
10780 3819 Acked Read 5 Done Sent Done false
10781 3820 Acked Write 5 Done Sent Done false
10782 3821 Acked Read 1 Done Sent Done false
10783 3822 Acked Write 4 Done Sent Done false
10784 3823 Acked Flush 0 Done Sent Done false
10785 3824 Acked Read 5 Done Sent Done false
10786 3825 Acked Write 5 Done Sent Done false
10787 3826 Acked Read 2 Done Sent Done false
10788 3827 Acked Read 5 Done Sent Done false
10789 3828 Acked Read 6 Done Sent Done false
10790 3829 Acked Read 1 Done Sent Done false
10791 3830 Acked Write 9 Done Sent Done false
10792 3831 Acked Write 2 Done Sent Done false
10793 3832 Acked Write 2 Done Sent Done false
10794 3833 Acked Read 8 Done Sent Done false
10795 3834 Acked Write 4 Done Sent Done false
10796 3835 Acked Read 9 Done Sent Done false
10797 3836 Acked Read 2 Done Sent Done false
10798 3837 Acked Write 3 Done Sent Done false
10799 3838 Acked Read 2 Done Sent Done false
10800 3839 Acked Read 3 Done Sent Done false
10801 3840 Acked Flush 0 Done Sent Done false
10802 3841 Acked Read 2 Done Sent Done false
10803 3842 Acked Write 1 Done Sent Done false
10804 3843 Acked Write 7 Done Sent Done false
10805 3844 Acked Flush 0 Done Sent Done false
10806 3845 Acked Write 6 Done Sent Done false
10807 3846 Acked Read 1 Done Sent Done false
10808 3847 Acked Read 5 Done Sent Done false
10809 3848 Acked Write 8 Done Sent Done false
10810 3849 Acked Read 6 Done Sent Done false
10811 3850 Acked Write 3 Done Sent Done false
10812 3851 Acked Read 2 Done Sent Done false
10813 3852 Acked Read 1 Done Sent Done false
10814 3853 Acked Write 10 Done Sent Done false
10815 3854 Acked Read 4 Done Sent Done false
10816 3855 Acked Write 6 Done Sent Done false
10817 3856 Acked Write 5 Done Sent Done false
10818 3857 Acked Write 7 Done Sent Done false
10819 3858 Acked Read 8 Done Sent Done false
10820 3859 Acked Flush 0 Done Sent Done false
10821 3860 Acked Write 5 Done Sent Done false
10822 3861 Acked Write 1 Done Sent Done false
10823 3862 Acked Write 2 Done Sent Done false
10824 3863 Acked Read 3 Done Sent Done false
10825 3864 Acked Flush 0 Done Sent Done false
10826 3865 Acked Write 8 Done Sent Done false
10827 3866 Acked Write 1 Done Sent Done false
10828 3867 Acked Write 2 Done Sent Done false
10829 3868 Acked Flush 0 Done Sent Done false
10830 3869 Acked Read 3 Done Sent Done false
10831 3870 Acked Read 2 Done Sent Done false
10832 3871 Acked Read 3 Done Sent Done false
10833 3872 Acked Write 10 Done Sent Done false
10834 3873 Acked Read 3 Done Sent Done false
10835 3874 Acked Write 7 Done Sent Done false
10836 3875 Acked Write 2 Done Sent Done false
10837 3876 Acked Read 9 Done Sent Done false
10838 3877 Acked Read 2 Done Sent Done false
10839 3878 Acked Write 9 Done Sent Done false
10840 3879 Acked Read 3 Done Sent Done false
10841 3880 Acked Read 6 Done Sent Done false
10842 3881 Acked Read 2 Done Sent Done false
10843 3882 Acked Read 2 Done Sent Done false
10844 3883 Acked Write 3 Done Sent Done false
10845 3884 Acked Read 9 Done Sent Done false
10846 3885 Acked Write 7 Done Sent Done false
10847 3886 Acked Read 8 Done Sent Done false
10848 3887 Acked Read 2 Done Sent Done false
10849 3888 Acked Write 5 Done Sent Done false
10850 3889 Acked Read 4 Done Sent Done false
10851 3890 Acked Read 3 Done Sent Done false
10852 3891 Acked Read 7 Done Sent Done false
10853 3892 Acked Write 3 Done Sent Done false
10854 3893 Acked Read 7 Done Sent Done false
10855 3894 Acked Read 4 Done Sent Done false
10856 3895 Acked Read 1 Done Sent Done false
10857 3896 Acked Write 3 Done Sent Done false
10858 3897 Acked Read 2 Done Sent Done false
10859 3898 Acked Write 5 Done Sent Done false
10860 3899 Acked Read 6 Done Sent Done false
10861 3900 Acked Read 6 Done Sent Done false
10862 3901 Acked Read 5 Done Sent Done false
10863 3902 Acked Read 9 Done Sent Done false
10864 3903 Acked Read 2 Done Sent Done false
10865 3904 Acked Read 8 Done Sent Done false
10866 3905 Acked Read 8 Done Sent Done false
10867 3906 Acked Write 1 Done Sent Done false
10868 3907 Acked Write 8 Done Sent Done false
10869 3908 Acked Read 5 Done Sent Done false
10870 3909 Acked Write 3 Done Sent Done false
10871 3910 Acked Write 1 Done Sent Done false
10872 3911 Acked Read 6 Done Sent Done false
10873 3912 Acked Write 6 Done Sent Done false
10874 3913 Acked Read 9 Done Sent Done false
10875 3914 Acked Read 3 Done Sent Done false
10876 3915 Acked Write 10 Done Sent Done false
10877 3916 Acked Read 9 Done Sent Done false
10878 3917 Acked Read 10 Done Sent Done false
10879 3918 Acked Read 8 Done Sent Done false
10880 3919 Acked Read 3 Done Sent Done false
10881 3920 Acked Write 1 Done Sent Done false
10882 3921 Acked Read 9 Done Sent Done false
10883 3922 Acked Read 1 Done Sent Done false
10884 3923 Acked Read 3 Done Sent Done false
10885 3924 Acked Read 3 Done Sent Done false
10886 3925 Acked Read 3 Done Sent Done false
10887 3926 Acked Read 5 Done Sent Done false
10888 3927 Acked Read 10 Done Sent Done false
10889 3928 Acked Write 10 Done Sent Done false
10890 3929 Acked Read 1 Done Sent Done false
10891 3930 Acked Read 6 Done Sent Done false
10892 3931 Acked Write 6 Done Sent Done false
10893 3932 Acked Read 2 Done Sent Done false
10894 3933 Acked Read 2 Done Sent Done false
10895 3934 Acked Write 8 Done Sent Done false
10896 3935 Acked Read 7 Done Sent Done false
10897 3936 Acked Read 8 Done Sent Done false
10898 3937 Acked Write 2 Done Sent Done false
10899 3938 Acked Flush 0 Done Sent Done false
10900 3939 Acked Flush 0 Done Sent Done false
10901 3940 Acked Read 3 Done Sent Done false
10902 3941 Acked Flush 0 Done Sent Done false
10903 3942 Acked Read 7 Done Sent Done false
10904 3943 Acked Read 2 Done Sent Done false
10905 3944 Acked Write 4 Done Sent Done false
10906 3945 Acked Read 3 Done Sent Done false
10907 3946 Acked Write 1 Done Sent Done false
10908 3947 Acked Write 3 Done Sent Done false
10909 3948 Acked Read 4 Done Sent Done false
10910 3949 Acked Write 3 Done Sent Done false
10911 3950 Acked Read 7 Done Sent Done false
10912 3951 Acked Flush 0 Done Sent Done false
10913 3952 Acked Read 8 Done Sent Done false
10914 3953 Acked Read 3 Done Sent Done false
10915 3954 Acked Write 5 Done Sent Done false
10916 3955 Acked Write 6 Done Sent Done false
10917 3956 Acked Read 7 Done Sent Done false
10918 3957 Acked Read 4 Done Sent Done false
10919 3958 Acked Write 3 Done Sent Done false
10920 3959 Acked Read 4 Done Sent Done false
10921 3960 Acked Read 3 Done Sent Done false
10922 3961 Acked Write 10 Done Sent Done false
10923 3962 Acked Write 5 Done Sent Done false
10924 3963 Acked Write 5 Done Sent Done false
10925 3964 Acked Read 8 Done Sent Done false
10926 3965 Acked Write 10 Done Sent Done false
10927 3966 Acked Write 3 Done Sent Done false
10928 3967 Acked Write 5 Done Sent Done false
10929 3968 Acked Write 1 Done Sent Done false
10930 3969 Acked Read 4 Done Sent Done false
10931 3970 Acked Read 6 Done Sent Done false
10932 3971 Acked Write 4 Done Sent Done false
10933 3972 Acked Write 6 Done Sent Done false
10934 3973 Acked Read 1 Done Sent Done false
10935 3974 Acked Read 5 Done Sent Done false
10936 3975 Acked Read 9 Done Sent Done false
10937 3976 Acked Read 4 Done Sent Done false
10938 3977 Acked Read 3 Done Sent Done false
10939 3978 Acked Read 1 Done Sent Done false
10940 3979 Acked Read 1 Done Sent Done false
10941 3980 Acked Read 1 Done Sent Done false
10942 3981 Acked Read 8 Done Sent Done false
10943 3982 Acked Write 6 Done Sent Done false
10944 3983 Acked Read 8 Done Sent Done false
10945 3984 Acked Read 8 Done Sent Done false
10946 3985 Acked Write 2 Done Sent Done false
10947 3986 Acked Write 5 Done Sent Done false
10948 3987 Acked Flush 0 Done Sent Done false
10949 3988 Acked Write 9 Done Sent Done false
10950 3989 Acked Read 5 Done Sent Done false
10951 3990 Acked Write 4 Done Sent Done false
10952 3991 Acked Read 3 Done Sent Done false
10953 3992 Acked Read 3 Done Sent Done false
10954 3993 Acked Flush 0 Done Sent Done false
10955 3994 Acked Read 3 Done Sent Done false
10956 3995 Acked Write 4 Done Sent Done false
10957 3996 Acked Flush 0 Done Sent Done false
10958 3997 Acked Read 3 Done Sent Done false
10959 3998 Acked Read 7 Done Sent Done false
10960 3999 Acked Write 1 Done Sent Done false
10961 4000 Acked Write 4 Done Sent Done false
10962 4001 Acked Read 2 Done Sent Done false
10963 4002 Acked Read 2 Done Sent Done false
10964 4003 Acked Read 2 Done Sent Done false
10965 4004 Acked Read 5 Done Sent Done false
10966 4005 Acked Flush 0 Done Sent Done false
10967 4006 Acked Read 1 Done Sent Done false
10968 4007 Acked Flush 0 Done Sent Done false
10969 4008 Acked Write 3 Done Sent Done false
10970 4009 Acked Write 9 Done Sent Done false
10971 4010 Acked Write 10 Done Sent Done false
10972 4011 Acked Read 9 Done Sent Done false
10973 4012 Acked Write 9 Done Sent Done false
10974 4013 Acked Read 6 Done Sent Done false
10975 4014 Acked Write 3 Done Sent Done false
10976 4015 Acked Write 3 Done Sent Done false
10977 4016 Acked Read 4 Done Sent Done false
10978 4017 Acked Read 5 Done Sent Done false
10979 4018 Acked Write 1 Done Sent Done false
10980 STATES DS:0 DS:1 DS:2 TOTAL
10981 Sent 0 301 0 301
10982 Done 301 0 301 602
10983 Skipped 0 0 0 0
10984 Error 0 0 0 0
10985 Last Flush: Some(JobId(4007)) Some(JobId(3717)) Some(JobId(4007))
10986 Downstairs last five completed: 3717 3716 3715 3714 3713
10987 Upstairs last five completed: 4018 4017 4016 4015 4014
10988 CLIENT: Up:0 ds:301 act:2
109892025-02-25T17:20:36.971ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
109902025-02-25T17:20:37.508ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
109912025-02-25T17:20:38.509ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
109922025-02-25T17:20:39.509ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
10993 ----------------------------------------------------------------
10994 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10995 States: Active Offline Active
10996 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10997 3718 Acked Read 1 Done Sent Done false
10998 3719 Acked Read 7 Done Sent Done false
10999 3720 Acked Read 8 Done Sent Done false
11000 3721 Acked Write 8 Done Sent Done false
11001 3722 Acked Read 2 Done Sent Done false
11002 3723 Acked Read 3 Done Sent Done false
11003 3724 Acked Write 5 Done Sent Done false
11004 3725 Acked Write 4 Done Sent Done false
11005 3726 Acked Read 3 Done Sent Done false
11006 3727 Acked Read 3 Done Sent Done false
11007 3728 Acked Read 6 Done Sent Done false
11008 3729 Acked Read 9 Done Sent Done false
11009 3730 Acked Write 4 Done Sent Done false
11010 3731 Acked Read 4 Done Sent Done false
11011 3732 Acked Read 2 Done Sent Done false
11012 3733 Acked Read 3 Done Sent Done false
11013 3734 Acked Write 5 Done Sent Done false
11014 3735 Acked Read 9 Done Sent Done false
11015 3736 Acked Write 8 Done Sent Done false
11016 3737 Acked Write 5 Done Sent Done false
11017 3738 Acked Write 3 Done Sent Done false
11018 3739 Acked Read 5 Done Sent Done false
11019 3740 Acked Read 9 Done Sent Done false
11020 3741 Acked Read 8 Done Sent Done false
11021 3742 Acked Read 1 Done Sent Done false
11022 3743 Acked Read 3 Done Sent Done false
11023 3744 Acked Read 2 Done Sent Done false
11024 3745 Acked Write 9 Done Sent Done false
11025 3746 Acked Write 6 Done Sent Done false
11026 3747 Acked Read 8 Done Sent Done false
11027 3748 Acked Write 4 Done Sent Done false
11028 3749 Acked Read 6 Done Sent Done false
11029 3750 Acked Read 6 Done Sent Done false
11030 3751 Acked Read 10 Done Sent Done false
11031 3752 Acked Flush 0 Done Sent Done false
11032 3753 Acked Flush 0 Done Sent Done false
11033 3754 Acked Read 9 Done Sent Done false
11034 3755 Acked Write 5 Done Sent Done false
11035 3756 Acked Write 6 Done Sent Done false
11036 3757 Acked Write 8 Done Sent Done false
11037 3758 Acked Read 10 Done Sent Done false
11038 3759 Acked Write 4 Done Sent Done false
11039 3760 Acked Write 1 Done Sent Done false
11040 3761 Acked Write 5 Done Sent Done false
11041 3762 Acked Write 3 Done Sent Done false
11042 3763 Acked Read 9 Done Sent Done false
11043 3764 Acked Write 3 Done Sent Done false
11044 3765 Acked Write 2 Done Sent Done false
11045 3766 Acked Read 5 Done Sent Done false
11046 3767 Acked Write 6 Done Sent Done false
11047 3768 Acked Flush 0 Done Sent Done false
11048 3769 Acked Write 7 Done Sent Done false
11049 3770 Acked Write 3 Done Sent Done false
11050 3771 Acked Write 2 Done Sent Done false
11051 3772 Acked Read 2 Done Sent Done false
11052 3773 Acked Read 2 Done Sent Done false
11053 3774 Acked Write 10 Done Sent Done false
11054 3775 Acked Write 4 Done Sent Done false
11055 3776 Acked Write 3 Done Sent Done false
11056 3777 Acked Flush 0 Done Sent Done false
11057 3778 Acked Write 8 Done Sent Done false
11058 3779 Acked Read 10 Done Sent Done false
11059 3780 Acked Write 9 Done Sent Done false
11060 3781 Acked Write 5 Done Sent Done false
11061 3782 Acked Read 4 Done Sent Done false
11062 3783 Acked Read 8 Done Sent Done false
11063 3784 Acked Write 4 Done Sent Done false
11064 3785 Acked Read 2 Done Sent Done false
11065 3786 Acked Read 4 Done Sent Done false
11066 3787 Acked Write 9 Done Sent Done false
11067 3788 Acked Write 4 Done Sent Done false
11068 3789 Acked Read 10 Done Sent Done false
11069 3790 Acked Write 7 Done Sent Done false
11070 3791 Acked Flush 0 Done Sent Done false
11071 3792 Acked Write 10 Done Sent Done false
11072 3793 Acked Write 4 Done Sent Done false
11073 3794 Acked Read 6 Done Sent Done false
11074 3795 Acked Read 4 Done Sent Done false
11075 3796 Acked Write 3 Done Sent Done false
11076 3797 Acked Write 2 Done Sent Done false
11077 3798 Acked Flush 0 Done Sent Done false
11078 3799 Acked Write 9 Done Sent Done false
11079 3800 Acked Read 7 Done Sent Done false
11080 3801 Acked Read 5 Done Sent Done false
11081 3802 Acked Write 2 Done Sent Done false
11082 3803 Acked Write 8 Done Sent Done false
11083 3804 Acked Write 7 Done Sent Done false
11084 3805 Acked Read 5 Done Sent Done false
11085 3806 Acked Flush 0 Done Sent Done false
11086 3807 Acked Write 8 Done Sent Done false
11087 3808 Acked Read 5 Done Sent Done false
11088 3809 Acked Write 3 Done Sent Done false
11089 3810 Acked Read 1 Done Sent Done false
11090 3811 Acked Read 7 Done Sent Done false
11091 3812 Acked Read 4 Done Sent Done false
11092 3813 Acked Write 6 Done Sent Done false
11093 3814 Acked Read 1 Done Sent Done false
11094 3815 Acked Write 7 Done Sent Done false
11095 3816 Acked Write 2 Done Sent Done false
11096 3817 Acked Read 4 Done Sent Done false
11097 3818 Acked Write 4 Done Sent Done false
11098 3819 Acked Read 5 Done Sent Done false
11099 3820 Acked Write 5 Done Sent Done false
11100 3821 Acked Read 1 Done Sent Done false
11101 3822 Acked Write 4 Done Sent Done false
11102 3823 Acked Flush 0 Done Sent Done false
11103 3824 Acked Read 5 Done Sent Done false
11104 3825 Acked Write 5 Done Sent Done false
11105 3826 Acked Read 2 Done Sent Done false
11106 3827 Acked Read 5 Done Sent Done false
11107 3828 Acked Read 6 Done Sent Done false
11108 3829 Acked Read 1 Done Sent Done false
11109 3830 Acked Write 9 Done Sent Done false
11110 3831 Acked Write 2 Done Sent Done false
11111 3832 Acked Write 2 Done Sent Done false
11112 3833 Acked Read 8 Done Sent Done false
11113 3834 Acked Write 4 Done Sent Done false
11114 3835 Acked Read 9 Done Sent Done false
11115 3836 Acked Read 2 Done Sent Done false
11116 3837 Acked Write 3 Done Sent Done false
11117 3838 Acked Read 2 Done Sent Done false
11118 3839 Acked Read 3 Done Sent Done false
11119 3840 Acked Flush 0 Done Sent Done false
11120 3841 Acked Read 2 Done Sent Done false
11121 3842 Acked Write 1 Done Sent Done false
11122 3843 Acked Write 7 Done Sent Done false
11123 3844 Acked Flush 0 Done Sent Done false
11124 3845 Acked Write 6 Done Sent Done false
11125 3846 Acked Read 1 Done Sent Done false
11126 3847 Acked Read 5 Done Sent Done false
11127 3848 Acked Write 8 Done Sent Done false
11128 3849 Acked Read 6 Done Sent Done false
11129 3850 Acked Write 3 Done Sent Done false
11130 3851 Acked Read 2 Done Sent Done false
11131 3852 Acked Read 1 Done Sent Done false
11132 3853 Acked Write 10 Done Sent Done false
11133 3854 Acked Read 4 Done Sent Done false
11134 3855 Acked Write 6 Done Sent Done false
11135 3856 Acked Write 5 Done Sent Done false
11136 3857 Acked Write 7 Done Sent Done false
11137 3858 Acked Read 8 Done Sent Done false
11138 3859 Acked Flush 0 Done Sent Done false
11139 3860 Acked Write 5 Done Sent Done false
11140 3861 Acked Write 1 Done Sent Done false
11141 3862 Acked Write 2 Done Sent Done false
11142 3863 Acked Read 3 Done Sent Done false
11143 3864 Acked Flush 0 Done Sent Done false
11144 3865 Acked Write 8 Done Sent Done false
11145 3866 Acked Write 1 Done Sent Done false
11146 3867 Acked Write 2 Done Sent Done false
11147 3868 Acked Flush 0 Done Sent Done false
11148 3869 Acked Read 3 Done Sent Done false
11149 3870 Acked Read 2 Done Sent Done false
11150 3871 Acked Read 3 Done Sent Done false
11151 3872 Acked Write 10 Done Sent Done false
11152 3873 Acked Read 3 Done Sent Done false
11153 3874 Acked Write 7 Done Sent Done false
11154 3875 Acked Write 2 Done Sent Done false
11155 3876 Acked Read 9 Done Sent Done false
11156 3877 Acked Read 2 Done Sent Done false
11157 3878 Acked Write 9 Done Sent Done false
11158 3879 Acked Read 3 Done Sent Done false
11159 3880 Acked Read 6 Done Sent Done false
11160 3881 Acked Read 2 Done Sent Done false
11161 3882 Acked Read 2 Done Sent Done false
11162 3883 Acked Write 3 Done Sent Done false
11163 3884 Acked Read 9 Done Sent Done false
11164 3885 Acked Write 7 Done Sent Done false
11165 3886 Acked Read 8 Done Sent Done false
11166 3887 Acked Read 2 Done Sent Done false
11167 3888 Acked Write 5 Done Sent Done false
11168 3889 Acked Read 4 Done Sent Done false
11169 3890 Acked Read 3 Done Sent Done false
11170 3891 Acked Read 7 Done Sent Done false
11171 3892 Acked Write 3 Done Sent Done false
11172 3893 Acked Read 7 Done Sent Done false
11173 3894 Acked Read 4 Done Sent Done false
11174 3895 Acked Read 1 Done Sent Done false
11175 3896 Acked Write 3 Done Sent Done false
11176 3897 Acked Read 2 Done Sent Done false
11177 3898 Acked Write 5 Done Sent Done false
11178 3899 Acked Read 6 Done Sent Done false
11179 3900 Acked Read 6 Done Sent Done false
11180 3901 Acked Read 5 Done Sent Done false
11181 3902 Acked Read 9 Done Sent Done false
11182 3903 Acked Read 2 Done Sent Done false
11183 3904 Acked Read 8 Done Sent Done false
11184 3905 Acked Read 8 Done Sent Done false
11185 3906 Acked Write 1 Done Sent Done false
11186 3907 Acked Write 8 Done Sent Done false
11187 3908 Acked Read 5 Done Sent Done false
11188 3909 Acked Write 3 Done Sent Done false
11189 3910 Acked Write 1 Done Sent Done false
11190 3911 Acked Read 6 Done Sent Done false
11191 3912 Acked Write 6 Done Sent Done false
11192 3913 Acked Read 9 Done Sent Done false
11193 3914 Acked Read 3 Done Sent Done false
11194 3915 Acked Write 10 Done Sent Done false
11195 3916 Acked Read 9 Done Sent Done false
11196 3917 Acked Read 10 Done Sent Done false
11197 3918 Acked Read 8 Done Sent Done false
11198 3919 Acked Read 3 Done Sent Done false
11199 3920 Acked Write 1 Done Sent Done false
11200 3921 Acked Read 9 Done Sent Done false
11201 3922 Acked Read 1 Done Sent Done false
11202 3923 Acked Read 3 Done Sent Done false
11203 3924 Acked Read 3 Done Sent Done false
11204 3925 Acked Read 3 Done Sent Done false
11205 3926 Acked Read 5 Done Sent Done false
11206 3927 Acked Read 10 Done Sent Done false
11207 3928 Acked Write 10 Done Sent Done false
11208 3929 Acked Read 1 Done Sent Done false
11209 3930 Acked Read 6 Done Sent Done false
11210 3931 Acked Write 6 Done Sent Done false
11211 3932 Acked Read 2 Done Sent Done false
11212 3933 Acked Read 2 Done Sent Done false
11213 3934 Acked Write 8 Done Sent Done false
11214 3935 Acked Read 7 Done Sent Done false
11215 3936 Acked Read 8 Done Sent Done false
11216 3937 Acked Write 2 Done Sent Done false
11217 3938 Acked Flush 0 Done Sent Done false
11218 3939 Acked Flush 0 Done Sent Done false
11219 3940 Acked Read 3 Done Sent Done false
11220 3941 Acked Flush 0 Done Sent Done false
11221 3942 Acked Read 7 Done Sent Done false
11222 3943 Acked Read 2 Done Sent Done false
11223 3944 Acked Write 4 Done Sent Done false
11224 3945 Acked Read 3 Done Sent Done false
11225 3946 Acked Write 1 Done Sent Done false
11226 3947 Acked Write 3 Done Sent Done false
11227 3948 Acked Read 4 Done Sent Done false
11228 3949 Acked Write 3 Done Sent Done false
11229 3950 Acked Read 7 Done Sent Done false
11230 3951 Acked Flush 0 Done Sent Done false
11231 3952 Acked Read 8 Done Sent Done false
11232 3953 Acked Read 3 Done Sent Done false
11233 3954 Acked Write 5 Done Sent Done false
11234 3955 Acked Write 6 Done Sent Done false
11235 3956 Acked Read 7 Done Sent Done false
11236 3957 Acked Read 4 Done Sent Done false
11237 3958 Acked Write 3 Done Sent Done false
11238 3959 Acked Read 4 Done Sent Done false
11239 3960 Acked Read 3 Done Sent Done false
11240 3961 Acked Write 10 Done Sent Done false
11241 3962 Acked Write 5 Done Sent Done false
11242 3963 Acked Write 5 Done Sent Done false
11243 3964 Acked Read 8 Done Sent Done false
11244 3965 Acked Write 10 Done Sent Done false
11245 3966 Acked Write 3 Done Sent Done false
11246 3967 Acked Write 5 Done Sent Done false
11247 3968 Acked Write 1 Done Sent Done false
11248 3969 Acked Read 4 Done Sent Done false
11249 3970 Acked Read 6 Done Sent Done false
11250 3971 Acked Write 4 Done Sent Done false
11251 3972 Acked Write 6 Done Sent Done false
11252 3973 Acked Read 1 Done Sent Done false
11253 3974 Acked Read 5 Done Sent Done false
11254 3975 Acked Read 9 Done Sent Done false
11255 3976 Acked Read 4 Done Sent Done false
11256 3977 Acked Read 3 Done Sent Done false
11257 3978 Acked Read 1 Done Sent Done false
11258 3979 Acked Read 1 Done Sent Done false
11259 3980 Acked Read 1 Done Sent Done false
11260 3981 Acked Read 8 Done Sent Done false
11261 3982 Acked Write 6 Done Sent Done false
11262 3983 Acked Read 8 Done Sent Done false
11263 3984 Acked Read 8 Done Sent Done false
11264 3985 Acked Write 2 Done Sent Done false
11265 3986 Acked Write 5 Done Sent Done false
11266 3987 Acked Flush 0 Done Sent Done false
11267 3988 Acked Write 9 Done Sent Done false
11268 3989 Acked Read 5 Done Sent Done false
11269 3990 Acked Write 4 Done Sent Done false
11270 3991 Acked Read 3 Done Sent Done false
11271 3992 Acked Read 3 Done Sent Done false
11272 3993 Acked Flush 0 Done Sent Done false
11273 3994 Acked Read 3 Done Sent Done false
11274 3995 Acked Write 4 Done Sent Done false
11275 3996 Acked Flush 0 Done Sent Done false
11276 3997 Acked Read 3 Done Sent Done false
11277 3998 Acked Read 7 Done Sent Done false
11278 3999 Acked Write 1 Done Sent Done false
11279 4000 Acked Write 4 Done Sent Done false
11280 4001 Acked Read 2 Done Sent Done false
11281 4002 Acked Read 2 Done Sent Done false
11282 4003 Acked Read 2 Done Sent Done false
11283 4004 Acked Read 5 Done Sent Done false
11284 4005 Acked Flush 0 Done Sent Done false
11285 4006 Acked Read 1 Done Sent Done false
11286 4007 Acked Flush 0 Done Sent Done false
11287 4008 Acked Write 3 Done Sent Done false
11288 4009 Acked Write 9 Done Sent Done false
11289 4010 Acked Write 10 Done Sent Done false
11290 4011 Acked Read 9 Done Sent Done false
11291 4012 Acked Write 9 Done Sent Done false
11292 4013 Acked Read 6 Done Sent Done false
11293 4014 Acked Write 3 Done Sent Done false
11294 4015 Acked Write 3 Done Sent Done false
11295 4016 Acked Read 4 Done Sent Done false
11296 4017 Acked Read 5 Done Sent Done false
11297 4018 Acked Write 1 Done Sent Done false
11298 STATES DS:0 DS:1 DS:2 TOTAL
11299 Sent 0 301 0 301
11300 Done 301 0 301 602
11301 Skipped 0 0 0 0
11302 Error 0 0 0 0
11303 Last Flush: Some(JobId(4007)) Some(JobId(3717)) Some(JobId(4007))
11304 Downstairs last five completed: 3717 3716 3715 3714 3713
11305 Upstairs last five completed: 4018 4017 4016 4015 4014
11306 CLIENT: Up:0 ds:301 act:2
113072025-02-25T17:20:40.973ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
113082025-02-25T17:20:41.511ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
113092025-02-25T17:20:42.223ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
113102025-02-25T17:20:42.223ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
113112025-02-25T17:20:42.223ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
113122025-02-25T17:20:42.223ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
113132025-02-25T17:20:42.223ZINFOcrucible: client task is exiting = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
113142025-02-25T17:20:42.724ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
113152025-02-25T17:20:43.514ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11316 [8820] got start action
11317 [8820] I am going to restart
11318 Starting downstairs at port 8820
11319 Make output file at "/tmp/dsc/downstairs-8820.txt"
11320 [8820][1] reports Starting
11321 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1218)
11322 [8820][1] reports Running
11323 ----------------------------------------------------------------
11324 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11325 States: Active Offline Active
11326 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11327 3718 Acked Read 1 Done Sent Done false
11328 3719 Acked Read 7 Done Sent Done false
11329 3720 Acked Read 8 Done Sent Done false
11330 3721 Acked Write 8 Done Sent Done false
11331 3722 Acked Read 2 Done Sent Done false
11332 3723 Acked Read 3 Done Sent Done false
11333 3724 Acked Write 5 Done Sent Done false
11334 3725 Acked Write 4 Done Sent Done false
11335 3726 Acked Read 3 Done Sent Done false
11336 3727 Acked Read 3 Done Sent Done false
11337 3728 Acked Read 6 Done Sent Done false
11338 3729 Acked Read 9 Done Sent Done false
11339 3730 Acked Write 4 Done Sent Done false
11340 3731 Acked Read 4 Done Sent Done false
11341 3732 Acked Read 2 Done Sent Done false
11342 3733 Acked Read 3 Done Sent Done false
11343 3734 Acked Write 5 Done Sent Done false
11344 3735 Acked Read 9 Done Sent Done false
11345 3736 Acked Write 8 Done Sent Done false
11346 3737 Acked Write 5 Done Sent Done false
11347 3738 Acked Write 3 Done Sent Done false
11348 3739 Acked Read 5 Done Sent Done false
11349 3740 Acked Read 9 Done Sent Done false
11350 3741 Acked Read 8 Done Sent Done false
11351 3742 Acked Read 1 Done Sent Done false
11352 3743 Acked Read 3 Done Sent Done false
11353 3744 Acked Read 2 Done Sent Done false
11354 3745 Acked Write 9 Done Sent Done false
11355 3746 Acked Write 6 Done Sent Done false
11356 3747 Acked Read 8 Done Sent Done false
11357 3748 Acked Write 4 Done Sent Done false
11358 3749 Acked Read 6 Done Sent Done false
11359 3750 Acked Read 6 Done Sent Done false
11360 3751 Acked Read 10 Done Sent Done false
11361 3752 Acked Flush 0 Done Sent Done false
11362 3753 Acked Flush 0 Done Sent Done false
11363 3754 Acked Read 9 Done Sent Done false
11364 3755 Acked Write 5 Done Sent Done false
11365 3756 Acked Write 6 Done Sent Done false
11366 3757 Acked Write 8 Done Sent Done false
11367 3758 Acked Read 10 Done Sent Done false
11368 3759 Acked Write 4 Done Sent Done false
11369 3760 Acked Write 1 Done Sent Done false
11370 3761 Acked Write 5 Done Sent Done false
11371 3762 Acked Write 3 Done Sent Done false
11372 3763 Acked Read 9 Done Sent Done false
11373 3764 Acked Write 3 Done Sent Done false
11374 3765 Acked Write 2 Done Sent Done false
11375 3766 Acked Read 5 Done Sent Done false
11376 3767 Acked Write 6 Done Sent Done false
11377 3768 Acked Flush 0 Done Sent Done false
11378 3769 Acked Write 7 Done Sent Done false
11379 3770 Acked Write 3 Done Sent Done false
11380 3771 Acked Write 2 Done Sent Done false
11381 3772 Acked Read 2 Done Sent Done false
11382 3773 Acked Read 2 Done Sent Done false
11383 3774 Acked Write 10 Done Sent Done false
11384 3775 Acked Write 4 Done Sent Done false
11385 3776 Acked Write 3 Done Sent Done false
11386 3777 Acked Flush 0 Done Sent Done false
11387 3778 Acked Write 8 Done Sent Done false
11388 3779 Acked Read 10 Done Sent Done false
11389 3780 Acked Write 9 Done Sent Done false
11390 3781 Acked Write 5 Done Sent Done false
11391 3782 Acked Read 4 Done Sent Done false
11392 3783 Acked Read 8 Done Sent Done false
11393 3784 Acked Write 4 Done Sent Done false
11394 3785 Acked Read 2 Done Sent Done false
11395 3786 Acked Read 4 Done Sent Done false
11396 3787 Acked Write 9 Done Sent Done false
11397 3788 Acked Write 4 Done Sent Done false
11398 3789 Acked Read 10 Done Sent Done false
11399 3790 Acked Write 7 Done Sent Done false
11400 3791 Acked Flush 0 Done Sent Done false
11401 3792 Acked Write 10 Done Sent Done false
11402 3793 Acked Write 4 Done Sent Done false
11403 3794 Acked Read 6 Done Sent Done false
11404 3795 Acked Read 4 Done Sent Done false
11405 3796 Acked Write 3 Done Sent Done false
11406 3797 Acked Write 2 Done Sent Done false
11407 3798 Acked Flush 0 Done Sent Done false
11408 3799 Acked Write 9 Done Sent Done false
11409 3800 Acked Read 7 Done Sent Done false
11410 3801 Acked Read 5 Done Sent Done false
11411 3802 Acked Write 2 Done Sent Done false
11412 3803 Acked Write 8 Done Sent Done false
11413 3804 Acked Write 7 Done Sent Done false
11414 3805 Acked Read 5 Done Sent Done false
11415 3806 Acked Flush 0 Done Sent Done false
11416 3807 Acked Write 8 Done Sent Done false
11417 3808 Acked Read 5 Done Sent Done false
11418 3809 Acked Write 3 Done Sent Done false
11419 3810 Acked Read 1 Done Sent Done false
11420 3811 Acked Read 7 Done Sent Done false
11421 3812 Acked Read 4 Done Sent Done false
11422 3813 Acked Write 6 Done Sent Done false
11423 3814 Acked Read 1 Done Sent Done false
11424 3815 Acked Write 7 Done Sent Done false
11425 3816 Acked Write 2 Done Sent Done false
11426 3817 Acked Read 4 Done Sent Done false
11427 3818 Acked Write 4 Done Sent Done false
11428 3819 Acked Read 5 Done Sent Done false
11429 3820 Acked Write 5 Done Sent Done false
11430 3821 Acked Read 1 Done Sent Done false
11431 3822 Acked Write 4 Done Sent Done false
11432 3823 Acked Flush 0 Done Sent Done false
11433 3824 Acked Read 5 Done Sent Done false
11434 3825 Acked Write 5 Done Sent Done false
11435 3826 Acked Read 2 Done Sent Done false
11436 3827 Acked Read 5 Done Sent Done false
11437 3828 Acked Read 6 Done Sent Done false
11438 3829 Acked Read 1 Done Sent Done false
11439 3830 Acked Write 9 Done Sent Done false
11440 3831 Acked Write 2 Done Sent Done false
11441 3832 Acked Write 2 Done Sent Done false
11442 3833 Acked Read 8 Done Sent Done false
11443 3834 Acked Write 4 Done Sent Done false
11444 3835 Acked Read 9 Done Sent Done false
11445 3836 Acked Read 2 Done Sent Done false
11446 3837 Acked Write 3 Done Sent Done false
11447 3838 Acked Read 2 Done Sent Done false
11448 3839 Acked Read 3 Done Sent Done false
11449 3840 Acked Flush 0 Done Sent Done false
11450 3841 Acked Read 2 Done Sent Done false
11451 3842 Acked Write 1 Done Sent Done false
11452 3843 Acked Write 7 Done Sent Done false
11453 3844 Acked Flush 0 Done Sent Done false
11454 3845 Acked Write 6 Done Sent Done false
11455 3846 Acked Read 1 Done Sent Done false
11456 3847 Acked Read 5 Done Sent Done false
11457 3848 Acked Write 8 Done Sent Done false
11458 3849 Acked Read 6 Done Sent Done false
11459 3850 Acked Write 3 Done Sent Done false
11460 3851 Acked Read 2 Done Sent Done false
11461 3852 Acked Read 1 Done Sent Done false
11462 3853 Acked Write 10 Done Sent Done false
11463 3854 Acked Read 4 Done Sent Done false
11464 3855 Acked Write 6 Done Sent Done false
11465 3856 Acked Write 5 Done Sent Done false
11466 3857 Acked Write 7 Done Sent Done false
11467 3858 Acked Read 8 Done Sent Done false
11468 3859 Acked Flush 0 Done Sent Done false
11469 3860 Acked Write 5 Done Sent Done false
11470 3861 Acked Write 1 Done Sent Done false
11471 3862 Acked Write 2 Done Sent Done false
11472 3863 Acked Read 3 Done Sent Done false
11473 3864 Acked Flush 0 Done Sent Done false
11474 3865 Acked Write 8 Done Sent Done false
11475 3866 Acked Write 1 Done Sent Done false
11476 3867 Acked Write 2 Done Sent Done false
11477 3868 Acked Flush 0 Done Sent Done false
11478 3869 Acked Read 3 Done Sent Done false
11479 3870 Acked Read 2 Done Sent Done false
11480 3871 Acked Read 3 Done Sent Done false
11481 3872 Acked Write 10 Done Sent Done false
11482 3873 Acked Read 3 Done Sent Done false
11483 3874 Acked Write 7 Done Sent Done false
11484 3875 Acked Write 2 Done Sent Done false
11485 3876 Acked Read 9 Done Sent Done false
11486 3877 Acked Read 2 Done Sent Done false
11487 3878 Acked Write 9 Done Sent Done false
11488 3879 Acked Read 3 Done Sent Done false
11489 3880 Acked Read 6 Done Sent Done false
11490 3881 Acked Read 2 Done Sent Done false
11491 3882 Acked Read 2 Done Sent Done false
11492 3883 Acked Write 3 Done Sent Done false
11493 3884 Acked Read 9 Done Sent Done false
11494 3885 Acked Write 7 Done Sent Done false
11495 3886 Acked Read 8 Done Sent Done false
11496 3887 Acked Read 2 Done Sent Done false
11497 3888 Acked Write 5 Done Sent Done false
11498 3889 Acked Read 4 Done Sent Done false
11499 3890 Acked Read 3 Done Sent Done false
11500 3891 Acked Read 7 Done Sent Done false
11501 3892 Acked Write 3 Done Sent Done false
11502 3893 Acked Read 7 Done Sent Done false
11503 3894 Acked Read 4 Done Sent Done false
11504 3895 Acked Read 1 Done Sent Done false
11505 3896 Acked Write 3 Done Sent Done false
11506 3897 Acked Read 2 Done Sent Done false
11507 3898 Acked Write 5 Done Sent Done false
11508 3899 Acked Read 6 Done Sent Done false
11509 3900 Acked Read 6 Done Sent Done false
11510 3901 Acked Read 5 Done Sent Done false
11511 3902 Acked Read 9 Done Sent Done false
11512 3903 Acked Read 2 Done Sent Done false
11513 3904 Acked Read 8 Done Sent Done false
11514 3905 Acked Read 8 Done Sent Done false
11515 3906 Acked Write 1 Done Sent Done false
11516 3907 Acked Write 8 Done Sent Done false
11517 3908 Acked Read 5 Done Sent Done false
11518 3909 Acked Write 3 Done Sent Done false
11519 3910 Acked Write 1 Done Sent Done false
11520 3911 Acked Read 6 Done Sent Done false
11521 3912 Acked Write 6 Done Sent Done false
11522 3913 Acked Read 9 Done Sent Done false
11523 3914 Acked Read 3 Done Sent Done false
11524 3915 Acked Write 10 Done Sent Done false
11525 3916 Acked Read 9 Done Sent Done false
11526 3917 Acked Read 10 Done Sent Done false
11527 3918 Acked Read 8 Done Sent Done false
11528 3919 Acked Read 3 Done Sent Done false
11529 3920 Acked Write 1 Done Sent Done false
11530 3921 Acked Read 9 Done Sent Done false
11531 3922 Acked Read 1 Done Sent Done false
11532 3923 Acked Read 3 Done Sent Done false
11533 3924 Acked Read 3 Done Sent Done false
11534 3925 Acked Read 3 Done Sent Done false
11535 3926 Acked Read 5 Done Sent Done false
11536 3927 Acked Read 10 Done Sent Done false
11537 3928 Acked Write 10 Done Sent Done false
11538 3929 Acked Read 1 Done Sent Done false
11539 3930 Acked Read 6 Done Sent Done false
11540 3931 Acked Write 6 Done Sent Done false
11541 3932 Acked Read 2 Done Sent Done false
11542 3933 Acked Read 2 Done Sent Done false
11543 3934 Acked Write 8 Done Sent Done false
11544 3935 Acked Read 7 Done Sent Done false
11545 3936 Acked Read 8 Done Sent Done false
11546 3937 Acked Write 2 Done Sent Done false
11547 3938 Acked Flush 0 Done Sent Done false
11548 3939 Acked Flush 0 Done Sent Done false
11549 3940 Acked Read 3 Done Sent Done false
11550 3941 Acked Flush 0 Done Sent Done false
11551 3942 Acked Read 7 Done Sent Done false
11552 3943 Acked Read 2 Done Sent Done false
11553 3944 Acked Write 4 Done Sent Done false
11554 3945 Acked Read 3 Done Sent Done false
11555 3946 Acked Write 1 Done Sent Done false
11556 3947 Acked Write 3 Done Sent Done false
11557 3948 Acked Read 4 Done Sent Done false
11558 3949 Acked Write 3 Done Sent Done false
11559 3950 Acked Read 7 Done Sent Done false
11560 3951 Acked Flush 0 Done Sent Done false
11561 3952 Acked Read 8 Done Sent Done false
11562 3953 Acked Read 3 Done Sent Done false
11563 3954 Acked Write 5 Done Sent Done false
11564 3955 Acked Write 6 Done Sent Done false
11565 3956 Acked Read 7 Done Sent Done false
11566 3957 Acked Read 4 Done Sent Done false
11567 3958 Acked Write 3 Done Sent Done false
11568 3959 Acked Read 4 Done Sent Done false
11569 3960 Acked Read 3 Done Sent Done false
11570 3961 Acked Write 10 Done Sent Done false
11571 3962 Acked Write 5 Done Sent Done false
11572 3963 Acked Write 5 Done Sent Done false
11573 3964 Acked Read 8 Done Sent Done false
11574 3965 Acked Write 10 Done Sent Done false
11575 3966 Acked Write 3 Done Sent Done false
11576 3967 Acked Write 5 Done Sent Done false
11577 3968 Acked Write 1 Done Sent Done false
11578 3969 Acked Read 4 Done Sent Done false
11579 3970 Acked Read 6 Done Sent Done false
11580 3971 Acked Write 4 Done Sent Done false
11581 3972 Acked Write 6 Done Sent Done false
11582 3973 Acked Read 1 Done Sent Done false
11583 3974 Acked Read 5 Done Sent Done false
11584 3975 Acked Read 9 Done Sent Done false
11585 3976 Acked Read 4 Done Sent Done false
11586 3977 Acked Read 3 Done Sent Done false
11587 3978 Acked Read 1 Done Sent Done false
11588 3979 Acked Read 1 Done Sent Done false
11589 3980 Acked Read 1 Done Sent Done false
11590 3981 Acked Read 8 Done Sent Done false
11591 3982 Acked Write 6 Done Sent Done false
11592 3983 Acked Read 8 Done Sent Done false
11593 3984 Acked Read 8 Done Sent Done false
11594 3985 Acked Write 2 Done Sent Done false
11595 3986 Acked Write 5 Done Sent Done false
11596 3987 Acked Flush 0 Done Sent Done false
11597 3988 Acked Write 9 Done Sent Done false
11598 3989 Acked Read 5 Done Sent Done false
11599 3990 Acked Write 4 Done Sent Done false
11600 3991 Acked Read 3 Done Sent Done false
11601 3992 Acked Read 3 Done Sent Done false
11602 3993 Acked Flush 0 Done Sent Done false
11603 3994 Acked Read 3 Done Sent Done false
11604 3995 Acked Write 4 Done Sent Done false
11605 3996 Acked Flush 0 Done Sent Done false
11606 3997 Acked Read 3 Done Sent Done false
11607 3998 Acked Read 7 Done Sent Done false
11608 3999 Acked Write 1 Done Sent Done false
11609 4000 Acked Write 4 Done Sent Done false
11610 4001 Acked Read 2 Done Sent Done false
11611 4002 Acked Read 2 Done Sent Done false
11612 4003 Acked Read 2 Done Sent Done false
11613 4004 Acked Read 5 Done Sent Done false
11614 4005 Acked Flush 0 Done Sent Done false
11615 4006 Acked Read 1 Done Sent Done false
11616 4007 Acked Flush 0 Done Sent Done false
11617 4008 Acked Write 3 Done Sent Done false
11618 4009 Acked Write 9 Done Sent Done false
11619 4010 Acked Write 10 Done Sent Done false
11620 4011 Acked Read 9 Done Sent Done false
11621 4012 Acked Write 9 Done Sent Done false
11622 4013 Acked Read 6 Done Sent Done false
11623 4014 Acked Write 3 Done Sent Done false
11624 4015 Acked Write 3 Done Sent Done false
11625 4016 Acked Read 4 Done Sent Done false
11626 4017 Acked Read 5 Done Sent Done false
11627 4018 Acked Write 1 Done Sent Done false
11628 STATES DS:0 DS:1 DS:2 TOTAL
11629 Sent 0 301 0 301
11630 Done 301 0 301 602
11631 Skipped 0 0 0 0
11632 Error 0 0 0 0
11633 Last Flush: Some(JobId(4007)) Some(JobId(3717)) Some(JobId(4007))
11634 Downstairs last five completed: 3717 3716 3715 3714 3713
11635 Upstairs last five completed: 4018 4017 4016 4015 4014
11636 CLIENT: Up:0 ds:301 act:2
116372025-02-25T17:20:45.518ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
116382025-02-25T17:20:46.019ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
116392025-02-25T17:20:46.520ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
116402025-02-25T17:20:47.021ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
116412025-02-25T17:20:47.522ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
116422025-02-25T17:20:48.023ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11643 ----------------------------------------------------------------
11644 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11645 States: Active Offline Active
11646 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11647 3718 Acked Read 1 Done Sent Done false
11648 3719 Acked Read 7 Done Sent Done false
11649 3720 Acked Read 8 Done Sent Done false
11650 3721 Acked Write 8 Done Sent Done false
11651 3722 Acked Read 2 Done Sent Done false
11652 3723 Acked Read 3 Done Sent Done false
11653 3724 Acked Write 5 Done Sent Done false
11654 3725 Acked Write 4 Done Sent Done false
11655 3726 Acked Read 3 Done Sent Done false
11656 3727 Acked Read 3 Done Sent Done false
11657 3728 Acked Read 6 Done Sent Done false
11658 3729 Acked Read 9 Done Sent Done false
11659 3730 Acked Write 4 Done Sent Done false
11660 3731 Acked Read 4 Done Sent Done false
11661 3732 Acked Read 2 Done Sent Done false
11662 3733 Acked Read 3 Done Sent Done false
11663 3734 Acked Write 5 Done Sent Done false
11664 3735 Acked Read 9 Done Sent Done false
11665 3736 Acked Write 8 Done Sent Done false
11666 3737 Acked Write 5 Done Sent Done false
11667 3738 Acked Write 3 Done Sent Done false
11668 3739 Acked Read 5 Done Sent Done false
11669 3740 Acked Read 9 Done Sent Done false
11670 3741 Acked Read 8 Done Sent Done false
11671 3742 Acked Read 1 Done Sent Done false
11672 3743 Acked Read 3 Done Sent Done false
11673 3744 Acked Read 2 Done Sent Done false
11674 3745 Acked Write 9 Done Sent Done false
11675 3746 Acked Write 6 Done Sent Done false
11676 3747 Acked Read 8 Done Sent Done false
11677 3748 Acked Write 4 Done Sent Done false
11678 3749 Acked Read 6 Done Sent Done false
11679 3750 Acked Read 6 Done Sent Done false
11680 3751 Acked Read 10 Done Sent Done false
11681 3752 Acked Flush 0 Done Sent Done false
11682 3753 Acked Flush 0 Done Sent Done false
11683 3754 Acked Read 9 Done Sent Done false
11684 3755 Acked Write 5 Done Sent Done false
11685 3756 Acked Write 6 Done Sent Done false
11686 3757 Acked Write 8 Done Sent Done false
11687 3758 Acked Read 10 Done Sent Done false
11688 3759 Acked Write 4 Done Sent Done false
11689 3760 Acked Write 1 Done Sent Done false
11690 3761 Acked Write 5 Done Sent Done false
11691 3762 Acked Write 3 Done Sent Done false
11692 3763 Acked Read 9 Done Sent Done false
11693 3764 Acked Write 3 Done Sent Done false
11694 3765 Acked Write 2 Done Sent Done false
11695 3766 Acked Read 5 Done Sent Done false
11696 3767 Acked Write 6 Done Sent Done false
11697 3768 Acked Flush 0 Done Sent Done false
11698 3769 Acked Write 7 Done Sent Done false
11699 3770 Acked Write 3 Done Sent Done false
11700 3771 Acked Write 2 Done Sent Done false
11701 3772 Acked Read 2 Done Sent Done false
11702 3773 Acked Read 2 Done Sent Done false
11703 3774 Acked Write 10 Done Sent Done false
11704 3775 Acked Write 4 Done Sent Done false
11705 3776 Acked Write 3 Done Sent Done false
11706 3777 Acked Flush 0 Done Sent Done false
11707 3778 Acked Write 8 Done Sent Done false
11708 3779 Acked Read 10 Done Sent Done false
11709 3780 Acked Write 9 Done Sent Done false
11710 3781 Acked Write 5 Done Sent Done false
11711 3782 Acked Read 4 Done Sent Done false
11712 3783 Acked Read 8 Done Sent Done false
11713 3784 Acked Write 4 Done Sent Done false
11714 3785 Acked Read 2 Done Sent Done false
11715 3786 Acked Read 4 Done Sent Done false
11716 3787 Acked Write 9 Done Sent Done false
11717 3788 Acked Write 4 Done Sent Done false
11718 3789 Acked Read 10 Done Sent Done false
11719 3790 Acked Write 7 Done Sent Done false
11720 3791 Acked Flush 0 Done Sent Done false
11721 3792 Acked Write 10 Done Sent Done false
11722 3793 Acked Write 4 Done Sent Done false
11723 3794 Acked Read 6 Done Sent Done false
11724 3795 Acked Read 4 Done Sent Done false
11725 3796 Acked Write 3 Done Sent Done false
11726 3797 Acked Write 2 Done Sent Done false
11727 3798 Acked Flush 0 Done Sent Done false
11728 3799 Acked Write 9 Done Sent Done false
11729 3800 Acked Read 7 Done Sent Done false
11730 3801 Acked Read 5 Done Sent Done false
11731 3802 Acked Write 2 Done Sent Done false
11732 3803 Acked Write 8 Done Sent Done false
11733 3804 Acked Write 7 Done Sent Done false
11734 3805 Acked Read 5 Done Sent Done false
11735 3806 Acked Flush 0 Done Sent Done false
11736 3807 Acked Write 8 Done Sent Done false
11737 3808 Acked Read 5 Done Sent Done false
11738 3809 Acked Write 3 Done Sent Done false
11739 3810 Acked Read 1 Done Sent Done false
11740 3811 Acked Read 7 Done Sent Done false
11741 3812 Acked Read 4 Done Sent Done false
11742 3813 Acked Write 6 Done Sent Done false
11743 3814 Acked Read 1 Done Sent Done false
11744 3815 Acked Write 7 Done Sent Done false
11745 3816 Acked Write 2 Done Sent Done false
11746 3817 Acked Read 4 Done Sent Done false
11747 3818 Acked Write 4 Done Sent Done false
11748 3819 Acked Read 5 Done Sent Done false
11749 3820 Acked Write 5 Done Sent Done false
11750 3821 Acked Read 1 Done Sent Done false
11751 3822 Acked Write 4 Done Sent Done false
11752 3823 Acked Flush 0 Done Sent Done false
11753 3824 Acked Read 5 Done Sent Done false
11754 3825 Acked Write 5 Done Sent Done false
11755 3826 Acked Read 2 Done Sent Done false
11756 3827 Acked Read 5 Done Sent Done false
11757 3828 Acked Read 6 Done Sent Done false
11758 3829 Acked Read 1 Done Sent Done false
11759 3830 Acked Write 9 Done Sent Done false
11760 3831 Acked Write 2 Done Sent Done false
11761 3832 Acked Write 2 Done Sent Done false
11762 3833 Acked Read 8 Done Sent Done false
11763 3834 Acked Write 4 Done Sent Done false
11764 3835 Acked Read 9 Done Sent Done false
11765 3836 Acked Read 2 Done Sent Done false
11766 3837 Acked Write 3 Done Sent Done false
11767 3838 Acked Read 2 Done Sent Done false
11768 3839 Acked Read 3 Done Sent Done false
11769 3840 Acked Flush 0 Done Sent Done false
11770 3841 Acked Read 2 Done Sent Done false
11771 3842 Acked Write 1 Done Sent Done false
11772 3843 Acked Write 7 Done Sent Done false
11773 3844 Acked Flush 0 Done Sent Done false
11774 3845 Acked Write 6 Done Sent Done false
11775 3846 Acked Read 1 Done Sent Done false
11776 3847 Acked Read 5 Done Sent Done false
11777 3848 Acked Write 8 Done Sent Done false
11778 3849 Acked Read 6 Done Sent Done false
11779 3850 Acked Write 3 Done Sent Done false
11780 3851 Acked Read 2 Done Sent Done false
11781 3852 Acked Read 1 Done Sent Done false
11782 3853 Acked Write 10 Done Sent Done false
11783 3854 Acked Read 4 Done Sent Done false
11784 3855 Acked Write 6 Done Sent Done false
11785 3856 Acked Write 5 Done Sent Done false
11786 3857 Acked Write 7 Done Sent Done false
11787 3858 Acked Read 8 Done Sent Done false
11788 3859 Acked Flush 0 Done Sent Done false
11789 3860 Acked Write 5 Done Sent Done false
11790 3861 Acked Write 1 Done Sent Done false
11791 3862 Acked Write 2 Done Sent Done false
11792 3863 Acked Read 3 Done Sent Done false
11793 3864 Acked Flush 0 Done Sent Done false
11794 3865 Acked Write 8 Done Sent Done false
11795 3866 Acked Write 1 Done Sent Done false
11796 3867 Acked Write 2 Done Sent Done false
11797 3868 Acked Flush 0 Done Sent Done false
11798 3869 Acked Read 3 Done Sent Done false
11799 3870 Acked Read 2 Done Sent Done false
11800 3871 Acked Read 3 Done Sent Done false
11801 3872 Acked Write 10 Done Sent Done false
11802 3873 Acked Read 3 Done Sent Done false
11803 3874 Acked Write 7 Done Sent Done false
11804 3875 Acked Write 2 Done Sent Done false
11805 3876 Acked Read 9 Done Sent Done false
11806 3877 Acked Read 2 Done Sent Done false
11807 3878 Acked Write 9 Done Sent Done false
11808 3879 Acked Read 3 Done Sent Done false
11809 3880 Acked Read 6 Done Sent Done false
11810 3881 Acked Read 2 Done Sent Done false
11811 3882 Acked Read 2 Done Sent Done false
11812 3883 Acked Write 3 Done Sent Done false
11813 3884 Acked Read 9 Done Sent Done false
11814 3885 Acked Write 7 Done Sent Done false
11815 3886 Acked Read 8 Done Sent Done false
11816 3887 Acked Read 2 Done Sent Done false
11817 3888 Acked Write 5 Done Sent Done false
11818 3889 Acked Read 4 Done Sent Done false
11819 3890 Acked Read 3 Done Sent Done false
11820 3891 Acked Read 7 Done Sent Done false
11821 3892 Acked Write 3 Done Sent Done false
11822 3893 Acked Read 7 Done Sent Done false
11823 3894 Acked Read 4 Done Sent Done false
11824 3895 Acked Read 1 Done Sent Done false
11825 3896 Acked Write 3 Done Sent Done false
11826 3897 Acked Read 2 Done Sent Done false
11827 3898 Acked Write 5 Done Sent Done false
11828 3899 Acked Read 6 Done Sent Done false
11829 3900 Acked Read 6 Done Sent Done false
11830 3901 Acked Read 5 Done Sent Done false
11831 3902 Acked Read 9 Done Sent Done false
11832 3903 Acked Read 2 Done Sent Done false
11833 3904 Acked Read 8 Done Sent Done false
11834 3905 Acked Read 8 Done Sent Done false
11835 3906 Acked Write 1 Done Sent Done false
11836 3907 Acked Write 8 Done Sent Done false
11837 3908 Acked Read 5 Done Sent Done false
11838 3909 Acked Write 3 Done Sent Done false
11839 3910 Acked Write 1 Done Sent Done false
11840 3911 Acked Read 6 Done Sent Done false
11841 3912 Acked Write 6 Done Sent Done false
11842 3913 Acked Read 9 Done Sent Done false
11843 3914 Acked Read 3 Done Sent Done false
11844 3915 Acked Write 10 Done Sent Done false
11845 3916 Acked Read 9 Done Sent Done false
11846 3917 Acked Read 10 Done Sent Done false
11847 3918 Acked Read 8 Done Sent Done false
11848 3919 Acked Read 3 Done Sent Done false
11849 3920 Acked Write 1 Done Sent Done false
11850 3921 Acked Read 9 Done Sent Done false
11851 3922 Acked Read 1 Done Sent Done false
11852 3923 Acked Read 3 Done Sent Done false
11853 3924 Acked Read 3 Done Sent Done false
11854 3925 Acked Read 3 Done Sent Done false
11855 3926 Acked Read 5 Done Sent Done false
11856 3927 Acked Read 10 Done Sent Done false
11857 3928 Acked Write 10 Done Sent Done false
11858 3929 Acked Read 1 Done Sent Done false
11859 3930 Acked Read 6 Done Sent Done false
11860 3931 Acked Write 6 Done Sent Done false
11861 3932 Acked Read 2 Done Sent Done false
11862 3933 Acked Read 2 Done Sent Done false
11863 3934 Acked Write 8 Done Sent Done false
11864 3935 Acked Read 7 Done Sent Done false
11865 3936 Acked Read 8 Done Sent Done false
11866 3937 Acked Write 2 Done Sent Done false
11867 3938 Acked Flush 0 Done Sent Done false
11868 3939 Acked Flush 0 Done Sent Done false
11869 3940 Acked Read 3 Done Sent Done false
11870 3941 Acked Flush 0 Done Sent Done false
11871 3942 Acked Read 7 Done Sent Done false
11872 3943 Acked Read 2 Done Sent Done false
11873 3944 Acked Write 4 Done Sent Done false
11874 3945 Acked Read 3 Done Sent Done false
11875 3946 Acked Write 1 Done Sent Done false
11876 3947 Acked Write 3 Done Sent Done false
11877 3948 Acked Read 4 Done Sent Done false
11878 3949 Acked Write 3 Done Sent Done false
11879 3950 Acked Read 7 Done Sent Done false
11880 3951 Acked Flush 0 Done Sent Done false
11881 3952 Acked Read 8 Done Sent Done false
11882 3953 Acked Read 3 Done Sent Done false
11883 3954 Acked Write 5 Done Sent Done false
11884 3955 Acked Write 6 Done Sent Done false
11885 3956 Acked Read 7 Done Sent Done false
11886 3957 Acked Read 4 Done Sent Done false
11887 3958 Acked Write 3 Done Sent Done false
11888 3959 Acked Read 4 Done Sent Done false
11889 3960 Acked Read 3 Done Sent Done false
11890 3961 Acked Write 10 Done Sent Done false
11891 3962 Acked Write 5 Done Sent Done false
11892 3963 Acked Write 5 Done Sent Done false
11893 3964 Acked Read 8 Done Sent Done false
11894 3965 Acked Write 10 Done Sent Done false
11895 3966 Acked Write 3 Done Sent Done false
11896 3967 Acked Write 5 Done Sent Done false
11897 3968 Acked Write 1 Done Sent Done false
11898 3969 Acked Read 4 Done Sent Done false
11899 3970 Acked Read 6 Done Sent Done false
11900 3971 Acked Write 4 Done Sent Done false
11901 3972 Acked Write 6 Done Sent Done false
11902 3973 Acked Read 1 Done Sent Done false
11903 3974 Acked Read 5 Done Sent Done false
11904 3975 Acked Read 9 Done Sent Done false
11905 3976 Acked Read 4 Done Sent Done false
11906 3977 Acked Read 3 Done Sent Done false
11907 3978 Acked Read 1 Done Sent Done false
11908 3979 Acked Read 1 Done Sent Done false
11909 3980 Acked Read 1 Done Sent Done false
11910 3981 Acked Read 8 Done Sent Done false
11911 3982 Acked Write 6 Done Sent Done false
11912 3983 Acked Read 8 Done Sent Done false
11913 3984 Acked Read 8 Done Sent Done false
11914 3985 Acked Write 2 Done Sent Done false
11915 3986 Acked Write 5 Done Sent Done false
11916 3987 Acked Flush 0 Done Sent Done false
11917 3988 Acked Write 9 Done Sent Done false
11918 3989 Acked Read 5 Done Sent Done false
11919 3990 Acked Write 4 Done Sent Done false
11920 3991 Acked Read 3 Done Sent Done false
11921 3992 Acked Read 3 Done Sent Done false
11922 3993 Acked Flush 0 Done Sent Done false
11923 3994 Acked Read 3 Done Sent Done false
11924 3995 Acked Write 4 Done Sent Done false
11925 3996 Acked Flush 0 Done Sent Done false
11926 3997 Acked Read 3 Done Sent Done false
11927 3998 Acked Read 7 Done Sent Done false
11928 3999 Acked Write 1 Done Sent Done false
11929 4000 Acked Write 4 Done Sent Done false
11930 4001 Acked Read 2 Done Sent Done false
11931 4002 Acked Read 2 Done Sent Done false
11932 4003 Acked Read 2 Done Sent Done false
11933 4004 Acked Read 5 Done Sent Done false
11934 4005 Acked Flush 0 Done Sent Done false
11935 4006 Acked Read 1 Done Sent Done false
11936 4007 Acked Flush 0 Done Sent Done false
11937 4008 Acked Write 3 Done Sent Done false
11938 4009 Acked Write 9 Done Sent Done false
11939 4010 Acked Write 10 Done Sent Done false
11940 4011 Acked Read 9 Done Sent Done false
11941 4012 Acked Write 9 Done Sent Done false
11942 4013 Acked Read 6 Done Sent Done false
11943 4014 Acked Write 3 Done Sent Done false
11944 4015 Acked Write 3 Done Sent Done false
11945 4016 Acked Read 4 Done Sent Done false
11946 4017 Acked Read 5 Done Sent Done false
11947 4018 Acked Write 1 Done Sent Done false
11948 STATES DS:0 DS:1 DS:2 TOTAL
11949 Sent 0 301 0 301
11950 Done 301 0 301 602
11951 Skipped 0 0 0 0
11952 Error 0 0 0 0
11953 Last Flush: Some(JobId(4007)) Some(JobId(3717)) Some(JobId(4007))
11954 Downstairs last five completed: 3717 3716 3715 3714 3713
11955 Upstairs last five completed: 4018 4017 4016 4015 4014
11956 CLIENT: Up:0 ds:301 act:2
119572025-02-25T17:20:48.977ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
119582025-02-25T17:20:49.525ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
119592025-02-25T17:20:50.526ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
119602025-02-25T17:20:51.527ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
119612025-02-25T17:20:52.224ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
119622025-02-25T17:20:52.224ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60328) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
119632025-02-25T17:20:52.225ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
119642025-02-25T17:20:52.226ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
119652025-02-25T17:20:52.226ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
119662025-02-25T17:20:52.226ZINFOcrucible: send last flush ID to this DS: Some(JobId(3717)) = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
119672025-02-25T17:20:52.226ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
119682025-02-25T17:20:52.226ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(3717)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
119692025-02-25T17:20:52.226ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(3717)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
11970 ----------------------------------------------------------------
11971 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:11
11972 States: Active Active Active
11973 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11974 4008 Acked Write 3 Done Done Done true
11975 4009 Acked Write 9 Done Done Done true
11976 4010 Acked Write 10 Done Done Done true
11977 4011 Acked Read 9 Done Done Done true
11978 4012 Acked Write 9 Done Done Done true
11979 4013 Acked Read 6 Done Done Done true
11980 4014 Acked Write 3 Done Done Done true
11981 4015 Acked Write 3 Done Done Done true
11982 4016 Acked Read 4 Done Done Done true
11983 4017 Acked Read 5 Done Done Done true
11984 4018 Acked Write 1 Done Done Done true
11985 STATES DS:0 DS:1 DS:2 TOTAL
11986 Sent 0 0 0 0
11987 Done 11 11 11 33
11988 Skipped 0 0 0 0
11989 Error 0 0 0 0
11990 Last Flush: Some(JobId(4007)) Some(JobId(4007)) Some(JobId(4007))
11991 Downstairs last five completed: 4007 4006 4005 4004 4003
11992 Upstairs last five completed: 4018 4017 4016 4015 4014
11993 CLIENT: Up:0 ds:11 act:3
11994 ----------------------------------------------------------------
11995 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11996 Downstairs last five completed: 4019 4018 4017 4016 4015
11997 Upstairs last five completed: 4019 4018 4017 4016 4015
11998 CLIENT: Up:0 ds:0 act:3
11999 Replay: All jobs finished
12000 Main task has work to do, go find it
12001 got dsc Stop(1)
12002 stop 1
12003 [8820] Got stop action so:false kr:false
12004 Feb 25 17:20:56.479 INFO request completed, latency_us: 349, response_code: 204, uri: /stop/cid/1, method: POST, req_id: a51b4036-c600-41f7-aee6-be8f79c73d85, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
120052025-02-25T17:20:56.480ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
120062025-02-25T17:20:56.480ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
12007 Feb 25 17:20:56.480 {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:20:56.48065416Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
12008 INFO request completed, latency_us: 162, response_code: 200, uri: /state/cid/1, method: GET, req_id: 23105d4c-2668-4221-b36e-a6dda462f96f, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
120092025-02-25T17:20:56.480ZINFOcrucible: client task is exiting = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
12010 [8820] Exited with: ExitStatus(unix_wait_status(9))
12011 [8820][1] reports Exit
12012 Feb 25 17:21:00.483 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/1, method: GET, req_id: c60e6a96-0c7b-455c-ad06-7caf74c49a2d, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
12013 001/300 Write block 64 len 40960 data: 2 2 2 2 4 5 5 5 5 5
12014 002/300 Read block 4280 len 36864
12015 003/300 Read block 2617 len 8192
12016 004/300 Flush
12017 005/300 Read block 4638 len 12288
12018 006/300 Read block 3568 len 32768
12019 007/300 Write block 2279 len 4096 data: 2
12020 008/300 Write block 4024 len 28672 data: 5 6 5 5 5 4 4
12021 009/300 Read block 835 len 16384
12022 010/300 Flush
12023 011/300 Read block 1399 len 36864
12024 012/300 Read block 4160 len 20480
12025 013/300 Read block 4529 len 24576
12026 014/300 Write block 627 len 24576 data: 5 5 5 5 5 4
12027 015/300 Read block 129 len 40960
12028 016/300 Read block 1727 len 32768
12029 017/300 Flush
12030 018/300 Read block 1796 len 32768
12031 019/300 Write block 3368 len 20480 data: 3 3 3 3 2
12032 020/300 Write block 489 len 40960 data: 4 3 3 3 3 3 3 3 3 2
12033 021/300 Write block 1994 len 12288 data: 2 2 3
12034 022/300 Read block 4046 len 4096
12035 023/300 Write block 4002 len 36864 data: 4 2 2 2 2 2 2 2 2
12036 024/300 Read block 4634 len 28672
12037 025/300 Write block 584 len 28672 data: 3 3 3 2 2 2 3
12038 026/300 Read block 2413 len 8192
12039 027/300 Read block 3084 len 4096
12040 028/300 Read block 399 len 20480
12041 029/300 Write block 4989 len 8192 data: 4 4
12042 030/300 Write block 3428 len 16384 data: 3 3 3 3
12043 031/300 Read block 3541 len 4096
12044 032/300 Write block 4862 len 28672 data: 3 2 2 2 2 2 2
12045 033/300 Write block 318 len 16384 data: 4 3 3 3
12046 034/300 Write block 432 len 12288 data: 2 2 2
12047 035/300 Read block 3067 len 36864
12048 036/300 Read block 290 len 16384
12049 037/300 Read block 869 len 12288
12050 038/300 Read block 181 len 40960
12051 039/300 Write block 1921 len 4096 data: 2
12052 040/300 Read block 3127 len 8192
12053 041/300 Read block 4956 len 28672
12054 042/300 Write block 1232 len 24576 data: 2 2 3 4 4 4
12055 043/300 Flush
12056 044/300 Read block 3856 len 32768
12057 045/300 Write block 4841 len 36864 data: 3 3 3 4 4 5 4 4 5
12058 046/300 Read block 1035 len 24576
12059 047/300 Read block 4195 len 16384
12060 048/300 Write block 3401 len 20480 data: 5 6 3 3 3
12061 049/300 Write block 1631 len 40960 data: 4 3 3 3 4 3 3 3 3 2
12062 050/300 Read block 2913 len 20480
12063 051/300 Write block 565 len 20480 data: 3 2 3 3 3
12064 052/300 Read block 1747 len 32768
12065 053/300 Read block 3432 len 40960
12066 054/300 Write block 1087 len 8192 data: 3 4
12067 055/300 Write block 100 len 24576 data: 4 3 2 2 2 2
12068 056/300 Write block 1456 len 36864 data: 2 2 2 3 2 3 3 3 3
12069 057/300 Write block 3454 len 20480 data: 3 3 4 4 3
12070 058/300 Read block 1969 len 28672
12071 059/300 Write block 1090 len 16384 data: 3 4 5 4
12072 060/300 Read block 4401 len 28672
12073 061/300 Read block 989 len 32768
12074 062/300 Write block 2706 len 32768 data: 3 3 3 3 2 2 2 2
12075 063/300 Read block 3908 len 32768
12076 064/300 Read block 2643 len 12288
12077 065/300 Write block 3332 len 40960 data: 2 2 2 2 2 2 2 2 2 3
12078 066/300 Write block 1019 len 40960 data: 3 3 3 4 4 4 5 3 3 3
12079 067/300 Read block 1309 len 4096
12080 068/300 Flush
12081 069/300 Read block 360 len 4096
12082 070/300 Flush
12083 071/300 Write block 2736 len 24576 data: 4 5 5 5 5 3
12084 072/300 Flush
12085 073/300 Write block 1250 len 36864 data: 3 3 2 2 3 3 3 4 4
12086 074/300 Write block 2998 len 12288 data: 4 4 4
12087 075/300 Read block 1346 len 32768
12088 076/300 Write block 3219 len 36864 data: 5 4 4 4 4 4 3 4 4
12089 077/300 Read block 4809 len 20480
12090 078/300 Write block 2469 len 4096 data: 3
12091 079/300 Read block 1135 len 36864
12092 080/300 Write block 3866 len 40960 data: 3 2 2 2 2 2 3 3 3 2
12093 081/300 Write block 3601 len 36864 data: 3 3 3 3 3 4 4 4 4
12094 082/300 Write block 1700 len 20480 data: 4 3 4 4 4
12095 083/300 Read block 4735 len 40960
12096 084/300 Read block 2730 len 8192
12097 085/300 Write block 2079 len 40960 data: 2 3 3 4 4 4 4 4 3 3
12098 086/300 Write block 2156 len 8192 data: 3 3
12099 087/300 Read block 2587 len 16384
12100 088/300 Read block 1830 len 20480
12101 089/300 Write block 1090 len 4096 data: 4
12102 090/300 Read block 3909 len 32768
12103 091/300 Write block 2183 len 12288 data: 2 2 2
12104 092/300 Read block 4539 len 40960
12105 093/300 Read block 4585 len 28672
12106 094/300 Write block 1723 len 32768 data: 3 3 3 2 2 2 2 2
12107 095/300 Read block 3883 len 24576
12108 096/300 Write block 779 len 32768 data: 3 3 3 3 2 2 2 3
12109 097/300 Read block 3837 len 16384
12110 098/300 Write block 4806 len 40960 data: 3 2 3 4 3 4 6 6 6 5
12111 099/300 Read block 4757 len 36864
12112 100/300 Read block 4165 len 28672
12113 101/300 Read block 684 len 8192
12114 102/300 Read block 1471 len 40960
12115 103/300 Write block 154 len 20480 data: 3 3 3 4 4
12116 104/300 Read block 2035 len 20480
12117 105/300 Write block 867 len 16384 data: 3 2 3 4
12118 106/300 Write block 1645 len 24576 data: 4 5 5 5 5 4
12119 107/300 Flush
12120 108/300 Write block 129 len 20480 data: 4 4 3 3 2
12121 109/300 Read block 3055 len 40960
12122 110/300 Write block 920 len 24576 data: 2 2 4 4 4 4
12123 111/300 Read block 557 len 36864
12124 112/300 Flush
12125 113/300 Write block 4515 len 32768 data: 5 4 4 4 4 4 3 3
12126 114/300 Write block 1452 len 32768 data: 4 2 2 2 3 3 3 4
12127 115/300 Read block 4524 len 32768
12128 116/300 Read block 3509 len 20480
12129 117/300 Read block 4835 len 8192
12130 118/300 Flush
12131 119/300 Read block 2073 len 36864
12132 120/300 Write block 564 len 28672 data: 3 4 3 4 4 4 3
12133 121/300 Read block 3877 len 4096
12134 122/300 Write block 2560 len 4096 data: 3
12135 123/300 Read block 1186 len 20480
12136 124/300 Read block 1246 len 4096
12137 125/300 Read block 2272 len 36864
12138 126/300 Write block 551 len 40960 data: 3 3 2 2 2 2 2 2 2 2
12139 127/300 Flush
12140 128/300 Write block 1626 len 40960 data: 5 4 5 4 4 5 4 4 4 5
12141 129/300 Read block 1436 len 16384
12142 130/300 Flush
12143 131/300 Write block 4810 len 24576 data: 4 5 7 7 7 6
12144 132/300 Read block 4934 len 12288
12145 133/300 Read block 1963 len 40960
12146 134/300 Read block 1382 len 12288
12147 135/300 Read block 918 len 24576
12148 136/300 Write block 716 len 8192 data: 3 2
12149 137/300 Write block 2195 len 28672 data: 3 3 4 4 5 5 4
12150 138/300 Write block 2713 len 20480 data: 3 2 3 4 4
12151 139/300 Read block 896 len 12288
12152 140/300 Write block 3389 len 24576 data: 3 3 2 3 3 3
12153 141/300 Read block 72 len 12288
12154 142/300 Read block 3531 len 40960
12155 143/300 Write block 944 len 12288 data: 3 2 2
12156 144/300 Write block 4635 len 16384 data: 5 4 5 4
12157 145/300 Write block 1175 len 4096 data: 3
12158 146/300 Read block 2846 len 4096
12159 147/300 Write block 2396 len 20480 data: 4 3 3 3 2
12160 148/300 Read block 3623 len 8192
12161 149/300 Read block 4393 len 32768
12162 150/300 Read block 3606 len 28672
12163 151/300 Write block 3214 len 24576 data: 3 3 3 4 5 6
12164 152/300 Write block 4413 len 16384 data: 2 2 2 2
12165 153/300 Write block 3776 len 36864 data: 4 4 3 3 3 3 3 2 2
12166 154/300 Read block 2748 len 28672
12167 155/300 Read block 477 len 32768
12168 156/300 Flush
12169 157/300 Read block 885 len 4096
12170 158/300 Write block 1172 len 20480 data: 3 2 3 4 4
12171 159/300 Read block 288 len 12288
12172 160/300 Write block 2813 len 8192 data: 2 2
12173 161/300 Write block 4776 len 28672 data: 3 3 3 3 3 2 2
12174 162/300 Read block 2223 len 4096
12175 163/300 Read block 2527 len 20480
12176 164/300 Read block 4785 len 28672
12177 165/300 Read block 3613 len 8192
12178 166/300 Read block 914 len 36864
12179 167/300 Read block 2010 len 20480
12180 168/300 Write block 1261 len 24576 data: 4 4 4 3 3 2
12181 169/300 Read block 3442 len 12288
12182 170/300 Read block 3466 len 28672
12183 171/300 Read block 579 len 36864
12184 172/300 Write block 2133 len 24576 data: 4 4 4 3 3 4
12185 173/300 Read block 4616 len 8192
12186 174/300 Write block 4390 len 36864 data: 4 5 5 4 4 3 3 2 2
12187 175/300 Write block 3920 len 16384 data: 3 3 3 3
12188 176/300 Write block 4965 len 4096 data: 5
12189 177/300 Read block 3510 len 4096
12190 178/300 Read block 4434 len 12288
12191 179/300 Read block 1551 len 4096
12192 180/300 Read block 1292 len 16384
12193 181/300 Read block 1177 len 4096
12194 182/300 Flush
12195 183/300 Write block 2820 len 32768 data: 2 2 3 3 3 4 4 4
12196 184/300 Read block 1313 len 16384
12197 185/300 Write block 4358 len 32768 data: 4 3 3 3 2 2 2 2
12198 186/300 Read block 1775 len 20480
12199 187/300 Flush
12200 188/300 Read block 4957 len 32768
12201 189/300 Read block 2664 len 36864
12202 190/300 Read block 43 len 28672
12203 191/300 Flush
12204 192/300 Read block 4687 len 36864
12205 193/300 Read block 1794 len 36864
12206 194/300 Read block 3577 len 36864
12207 195/300 Write block 4865 len 8192 data: 3 3
12208 196/300 Flush
12209 197/300 Flush
12210 198/300 Flush
12211 199/300 Write block 4708 len 32768 data: 4 2 2 2 2 2 2 2
12212 200/300 Read block 3925 len 12288
12213 201/300 Write block 628 len 28672 data: 6 6 6 6 5 4 3
12214 202/300 Read block 1724 len 4096
12215 203/300 Flush
12216 204/300 Read block 2882 len 16384
12217 205/300 Write block 3665 len 16384 data: 5 4 5 5
12218 206/300 Read block 307 len 32768
12219 207/300 Write block 1044 len 16384 data: 5 5 5 5
12220 208/300 Read block 3004 len 24576
12221 209/300 Write block 2503 len 4096 data: 3
12222 210/300 Flush
12223 211/300 Read block 3568 len 12288
12224 212/300 Read block 1321 len 16384
12225 213/300 Write block 2926 len 8192 data: 3 3
12226 214/300 Read block 3107 len 28672
12227 215/300 Flush
12228 216/300 Read block 3271 len 28672
12229 217/300 Write block 3966 len 16384 data: 3 3 3 3
12230 218/300 Write block 3193 len 28672 data: 2 2 2 2 2 2 3
12231 219/300 Write block 4260 len 12288 data: 2 2 2
12232 220/300 Write block 3162 len 8192 data: 4 4
12233 221/300 Read block 2092 len 32768
12234 222/300 Write block 2835 len 20480 data: 4 4 4 4 3
12235 223/300 Write block 2090 len 40960 data: 2 2 4 5 5 5 5 4 4 4
12236 224/300 Write block 1531 len 16384 data: 4 4 4 4
12237 225/300 Read block 886 len 40960
12238 226/300 Read block 4497 len 24576
12239 227/300 Read block 2855 len 36864
12240 228/300 Write block 1234 len 32768 data: 4 5 5 5 4 2 2 3
12241 229/300 Read block 347 len 40960
12242 230/300 Read block 646 len 12288
12243 231/300 Write block 3674 len 36864 data: 3 3 3 2 2 2 2 3 4
12244 232/300 Read block 2814 len 12288
12245 233/300 Write block 1895 len 24576 data: 3 3 3 3 4 4
12246 234/300 Write block 2008 len 4096 data: 3
12247 235/300 Read block 1618 len 32768
12248 236/300 Write block 962 len 40960 data: 3 4 4 4 5 5 5 3 3 2
12249 237/300 Read block 431 len 24576
12250 238/300 Write block 4597 len 28672 data: 3 3 4 4 4 5 5
12251 239/300 Read block 1142 len 8192
12252 240/300 Write block 4956 len 4096 data: 2
12253 241/300 Write block 2859 len 36864 data: 6 4 3 3 3 3 3 3 2
12254 242/300 Write block 3265 len 40960 data: 3 2 2 2 2 2 3 3 3 3
12255 243/300 Read block 3074 len 40960
12256 244/300 Read block 599 len 28672
12257 245/300 Write block 2065 len 32768 data: 3 3 2 2 2 2 2 2
12258 246/300 Read block 3777 len 36864
12259 247/300 Read block 3964 len 4096
12260 248/300 Write block 2211 len 32768 data: 7 6 6 6 5 5 6 6
12261 249/300 Write block 2055 len 24576 data: 2 2 3 3 3 3
12262 250/300 Read block 4679 len 4096
12263 251/300 Read block 2843 len 4096
12264 252/300 Write block 2012 len 28672 data: 3 2 2 2 2 2 3
12265 253/300 Read block 2913 len 24576
12266 254/300 Write block 2426 len 24576 data: 2 2 2 2 3 3
12267 255/300 Write block 4434 len 12288 data: 3 3 3
12268 256/300 Read block 180 len 16384
12269 257/300 Write block 4318 len 4096 data: 2
12270 258/300 Write block 3780 len 40960 data: 4 4 4 3 3 2 2 2 3 3
12271 259/300 Write block 2190 len 12288 data: 3 4 4
12272 260/300 Write block 2277 len 20480 data: 2 2 3 3 3
12273 261/300 Write block 2502 len 16384 data: 4 4 3 5
12274 262/300 Read block 1201 len 16384
12275 263/300 Read block 1924 len 12288
12276 264/300 Write block 3225 len 4096 data: 4
12277 265/300 Read block 3820 len 32768
12278 266/300 Write block 3793 len 16384 data: 4 4 5 5
12279 267/300 Flush
12280 268/300 Write block 366 len 12288 data: 3 3 3
12281 269/300 Write block 299 len 4096 data: 2
12282 270/300 Read block 3863 len 8192
12283 271/300 Read block 4987 len 12288
12284 272/300 Read block 2334 len 8192
12285 273/300 Write block 3475 len 28672 data: 2 3 3 3 3 3 4
12286 274/300 Read block 1322 len 20480
12287 275/300 Read block 497 len 20480
12288 276/300 Write block 1288 len 16384 data: 3 3 4 6
12289 277/300 Write block 1157 len 4096 data: 5
12290 278/300 Read block 2340 len 24576
12291 279/300 Read block 1982 len 24576
12292 280/300 Read block 649 len 36864
12293 281/300 Read block 4125 len 40960
12294 282/300 Read block 2249 len 4096
12295 283/300 Write block 525 len 8192 data: 2 2
12296 284/300 Read block 2463 len 36864
12297 285/300 Write block 3301 len 36864 data: 5 5 5 5 4 3 3 3 3
12298 286/300 Read block 499 len 16384
12299 287/300 Write block 414 len 40960 data: 4 5 4 3 4 5 5 4 6 5
12300 288/300 Read block 2391 len 12288
12301 289/300 Write block 4635 len 12288 data: 6 5 6
12302 290/300 Read block 2453 len 8192
12303 291/300 Read block 4165 len 24576
12304 292/300 Read block 1765 len 4096
12305 293/300 Read block 727 len 12288
12306 294/300 Read block 1130 len 20480
12307 295/300 Read block 3574 len 28672
12308 296/300 Read block 1230 len 8192
12309 297/300 Write block 2371 len 32768 data: 3 2 2 2 2 2 2 2
12310 298/300 Read block 3123 len 24576
12311 299/300 Read block 2874 len 4096
12312 300/300 Read block 4946 len 32768
12313 301/300 Read block 3688 len 24576
12314 Main task has work to do, go find it
12315 got dsc Start(1)
12316 start 1
12317 Feb 25 17:21:00.737 INFO request completed, latency_us: 226, response_code: 204, uri: /start/cid/1, method: POST, req_id: 86a37862-da95-473a-8eb9-2b1657bc74f8, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
12318 Replay: started 1, returned:Ok(())
12319 ----------------------------------------------------------------
12320 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12321 States: Active Offline Active
12322 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12323 4020 Acked Write 10 Done Sent Done false
12324 4021 Acked Read 9 Done Sent Done false
12325 4022 Acked Read 2 Done Sent Done false
12326 4023 Acked Flush 0 Done Sent Done false
12327 4024 Acked Read 3 Done Sent Done false
12328 4025 Acked Read 8 Done Sent Done false
12329 4026 Acked Write 1 Done Sent Done false
12330 4027 Acked Write 7 Done Sent Done false
12331 4028 Acked Read 4 Done Sent Done false
12332 4029 Acked Flush 0 Done Sent Done false
12333 4030 Acked Read 9 Done Sent Done false
12334 4031 Acked Read 5 Done Sent Done false
12335 4032 Acked Read 6 Done Sent Done false
12336 4033 Acked Write 6 Done Sent Done false
12337 4034 Acked Read 10 Done Sent Done false
12338 4035 Acked Read 8 Done Sent Done false
12339 4036 Acked Flush 0 Done Sent Done false
12340 4037 Acked Read 8 Done Sent Done false
12341 4038 Acked Write 5 Done Sent Done false
12342 4039 Acked Write 10 Done Sent Done false
12343 4040 Acked Write 3 Done Sent Done false
12344 4041 Acked Read 1 Done Sent Done false
12345 4042 Acked Write 9 Done Sent Done false
12346 4043 Acked Read 7 Done Sent Done false
12347 4044 Acked Write 7 Done Sent Done false
12348 4045 Acked Read 2 Done Sent Done false
12349 4046 Acked Read 1 Done Sent Done false
12350 4047 Acked Read 5 Done Sent Done false
12351 4048 Acked Write 2 Done Sent Done false
12352 4049 Acked Write 4 Done Sent Done false
12353 4050 Acked Read 1 Done Sent Done false
12354 4051 Acked Write 7 Done Sent Done false
12355 4052 Acked Write 4 Done Sent Done false
12356 4053 Acked Write 3 Done Sent Done false
12357 4054 Acked Read 9 Done Sent Done false
12358 4055 Acked Read 4 Done Sent Done false
12359 4056 Acked Read 3 Done Sent Done false
12360 4057 Acked Read 10 Done Sent Done false
12361 4058 Acked Write 1 Done Sent Done false
12362 4059 Acked Read 2 Done Sent Done false
12363 4060 Acked Read 7 Done Sent Done false
12364 4061 Acked Write 6 Done Sent Done false
12365 4062 Acked Flush 0 Done Sent Done false
12366 4063 Acked Read 8 Done Sent Done false
12367 4064 Acked Write 9 Done Sent Done false
12368 4065 Acked Read 6 Done Sent Done false
12369 4066 Acked Read 4 Done Sent Done false
12370 4067 Acked Write 5 Done Sent Done false
12371 4068 Acked Write 10 Done Sent Done false
12372 4069 Acked Read 5 Done Sent Done false
12373 4070 Acked Write 5 Done Sent Done false
12374 4071 Acked Read 8 Done Sent Done false
12375 4072 Acked Read 10 Done Sent Done false
12376 4073 Acked Write 2 Done Sent Done false
12377 4074 Acked Write 6 Done Sent Done false
12378 4075 Acked Write 9 Done Sent Done false
12379 4076 Acked Write 5 Done Sent Done false
12380 4077 Acked Read 7 Done Sent Done false
12381 4078 Acked Write 4 Done Sent Done false
12382 4079 Acked Read 7 Done Sent Done false
12383 4080 Acked Read 8 Done Sent Done false
12384 4081 Acked Write 8 Done Sent Done false
12385 4082 Acked Read 8 Done Sent Done false
12386 4083 Acked Read 3 Done Sent Done false
12387 4084 Acked Write 10 Done Sent Done false
12388 4085 Acked Write 10 Done Sent Done false
12389 4086 Acked Read 1 Done Sent Done false
12390 4087 Acked Flush 0 Done Sent Done false
12391 4088 Acked Read 1 Done Sent Done false
12392 4089 Acked Flush 0 Done Sent Done false
12393 4090 Acked Write 6 Done Sent Done false
12394 4091 Acked Flush 0 Done Sent Done false
12395 4092 Acked Write 9 Done Sent Done false
12396 4093 Acked Write 3 Done Sent Done false
12397 4094 Acked Read 8 Done Sent Done false
12398 4095 Acked Write 9 Done Sent Done false
12399 4096 Acked Read 5 Done Sent Done false
12400 4097 Acked Write 1 Done Sent Done false
12401 4098 Acked Read 9 Done Sent Done false
12402 4099 Acked Write 10 Done Sent Done false
12403 4100 Acked Write 9 Done Sent Done false
12404 4101 Acked Write 5 Done Sent Done false
12405 4102 Acked Read 10 Done Sent Done false
12406 4103 Acked Read 2 Done Sent Done false
12407 4104 Acked Write 10 Done Sent Done false
12408 4105 Acked Write 2 Done Sent Done false
12409 4106 Acked Read 4 Done Sent Done false
12410 4107 Acked Read 5 Done Sent Done false
12411 4108 Acked Write 1 Done Sent Done false
12412 4109 Acked Read 8 Done Sent Done false
12413 4110 Acked Write 3 Done Sent Done false
12414 4111 Acked Read 10 Done Sent Done false
12415 4112 Acked Read 7 Done Sent Done false
12416 4113 Acked Write 8 Done Sent Done false
12417 4114 Acked Read 6 Done Sent Done false
12418 4115 Acked Write 8 Done Sent Done false
12419 4116 Acked Read 4 Done Sent Done false
12420 4117 Acked Write 10 Done Sent Done false
12421 4118 Acked Read 9 Done Sent Done false
12422 4119 Acked Read 7 Done Sent Done false
12423 4120 Acked Read 2 Done Sent Done false
12424 4121 Acked Read 10 Done Sent Done false
12425 4122 Acked Write 5 Done Sent Done false
12426 4123 Acked Read 5 Done Sent Done false
12427 4124 Acked Write 4 Done Sent Done false
12428 4125 Acked Write 6 Done Sent Done false
12429 4126 Acked Flush 0 Done Sent Done false
12430 4127 Acked Write 5 Done Sent Done false
12431 4128 Acked Read 10 Done Sent Done false
12432 4129 Acked Write 6 Done Sent Done false
12433 4130 Acked Read 9 Done Sent Done false
12434 4131 Acked Flush 0 Done Sent Done false
12435 4132 Acked Write 8 Done Sent Done false
12436 4133 Acked Write 8 Done Sent Done false
12437 4134 Acked Read 8 Done Sent Done false
12438 4135 Acked Read 5 Done Sent Done false
12439 4136 Acked Read 2 Done Sent Done false
12440 4137 Acked Flush 0 Done Sent Done false
12441 4138 Acked Read 9 Done Sent Done false
12442 4139 Acked Write 7 Done Sent Done false
12443 4140 Acked Read 1 Done Sent Done false
12444 4141 Acked Write 1 Done Sent Done false
12445 4142 Acked Read 5 Done Sent Done false
12446 4143 Acked Read 1 Done Sent Done false
12447 4144 Acked Read 9 Done Sent Done false
12448 4145 Acked Write 10 Done Sent Done false
12449 4146 Acked Flush 0 Done Sent Done false
12450 4147 Acked Write 10 Done Sent Done false
12451 4148 Acked Read 4 Done Sent Done false
12452 4149 Acked Flush 0 Done Sent Done false
12453 4150 Acked Write 6 Done Sent Done false
12454 4151 Acked Read 3 Done Sent Done false
12455 4152 Acked Read 10 Done Sent Done false
12456 4153 Acked Read 3 Done Sent Done false
12457 4154 Acked Read 6 Done Sent Done false
12458 4155 Acked Write 2 Done Sent Done false
12459 4156 Acked Write 7 Done Sent Done false
12460 4157 Acked Write 5 Done Sent Done false
12461 4158 Acked Read 3 Done Sent Done false
12462 4159 Acked Write 6 Done Sent Done false
12463 4160 Acked Read 3 Done Sent Done false
12464 4161 Acked Read 10 Done Sent Done false
12465 4162 Acked Write 3 Done Sent Done false
12466 4163 Acked Write 4 Done Sent Done false
12467 4164 Acked Write 1 Done Sent Done false
12468 4165 Acked Read 1 Done Sent Done false
12469 4166 Acked Write 5 Done Sent Done false
12470 4167 Acked Read 2 Done Sent Done false
12471 4168 Acked Read 8 Done Sent Done false
12472 4169 Acked Read 7 Done Sent Done false
12473 4170 Acked Write 6 Done Sent Done false
12474 4171 Acked Write 4 Done Sent Done false
12475 4172 Acked Write 9 Done Sent Done false
12476 4173 Acked Read 7 Done Sent Done false
12477 4174 Acked Read 8 Done Sent Done false
12478 4175 Acked Flush 0 Done Sent Done false
12479 4176 Acked Read 1 Done Sent Done false
12480 4177 Acked Write 5 Done Sent Done false
12481 4178 Acked Read 3 Done Sent Done false
12482 4179 Acked Write 2 Done Sent Done false
12483 4180 Acked Write 7 Done Sent Done false
12484 4181 Acked Read 1 Done Sent Done false
12485 4182 Acked Read 5 Done Sent Done false
12486 4183 Acked Read 7 Done Sent Done false
12487 4184 Acked Read 2 Done Sent Done false
12488 4185 Acked Read 9 Done Sent Done false
12489 4186 Acked Read 5 Done Sent Done false
12490 4187 Acked Write 6 Done Sent Done false
12491 4188 Acked Read 3 Done Sent Done false
12492 4189 Acked Read 7 Done Sent Done false
12493 4190 Acked Read 9 Done Sent Done false
12494 4191 Acked Write 6 Done Sent Done false
12495 4192 Acked Read 2 Done Sent Done false
12496 4193 Acked Write 9 Done Sent Done false
12497 4194 Acked Write 4 Done Sent Done false
12498 4195 Acked Write 1 Done Sent Done false
12499 4196 Acked Read 1 Done Sent Done false
12500 4197 Acked Read 3 Done Sent Done false
12501 4198 Acked Read 1 Done Sent Done false
12502 4199 Acked Read 4 Done Sent Done false
12503 4200 Acked Read 1 Done Sent Done false
12504 4201 Acked Flush 0 Done Sent Done false
12505 4202 Acked Write 8 Done Sent Done false
12506 4203 Acked Read 4 Done Sent Done false
12507 4204 Acked Write 8 Done Sent Done false
12508 4205 Acked Read 5 Done Sent Done false
12509 4206 Acked Flush 0 Done Sent Done false
12510 4207 Acked Read 8 Done Sent Done false
12511 4208 Acked Read 9 Done Sent Done false
12512 4209 Acked Read 7 Done Sent Done false
12513 4210 Acked Flush 0 Done Sent Done false
12514 4211 Acked Read 9 Done Sent Done false
12515 4212 Acked Read 9 Done Sent Done false
12516 4213 Acked Read 9 Done Sent Done false
12517 4214 Acked Write 2 Done Sent Done false
12518 4215 Acked Flush 0 Done Sent Done false
12519 4216 Acked Flush 0 Done Sent Done false
12520 4217 Acked Flush 0 Done Sent Done false
12521 4218 Acked Write 8 Done Sent Done false
12522 4219 Acked Read 3 Done Sent Done false
12523 4220 Acked Write 7 Done Sent Done false
12524 4221 Acked Read 1 Done Sent Done false
12525 4222 Acked Flush 0 Done Sent Done false
12526 4223 Acked Read 4 Done Sent Done false
12527 4224 Acked Write 4 Done Sent Done false
12528 4225 Acked Read 8 Done Sent Done false
12529 4226 Acked Write 4 Done Sent Done false
12530 4227 Acked Read 6 Done Sent Done false
12531 4228 Acked Write 1 Done Sent Done false
12532 4229 Acked Flush 0 Done Sent Done false
12533 4230 Acked Read 3 Done Sent Done false
12534 4231 Acked Read 4 Done Sent Done false
12535 4232 Acked Write 2 Done Sent Done false
12536 4233 Acked Read 7 Done Sent Done false
12537 4234 Acked Flush 0 Done Sent Done false
12538 4235 Acked Read 7 Done Sent Done false
12539 4236 Acked Write 4 Done Sent Done false
12540 4237 Acked Write 7 Done Sent Done false
12541 4238 Acked Write 3 Done Sent Done false
12542 4239 Acked Write 2 Done Sent Done false
12543 4240 Acked Read 8 Done Sent Done false
12544 4241 Acked Write 5 Done Sent Done false
12545 4242 Acked Write 10 Done Sent Done false
12546 4243 Acked Write 4 Done Sent Done false
12547 4244 Acked Read 10 Done Sent Done false
12548 4245 Acked Read 6 Done Sent Done false
12549 4246 Acked Read 9 Done Sent Done false
12550 4247 Acked Write 8 Done Sent Done false
12551 4248 Acked Read 10 Done Sent Done false
12552 4249 Acked Read 3 Done Sent Done false
12553 4250 Acked Write 9 Done Sent Done false
12554 4251 Acked Read 3 Done Sent Done false
12555 4252 Acked Write 6 Done Sent Done false
12556 4253 Acked Write 1 Done Sent Done false
12557 4254 Acked Read 8 Done Sent Done false
12558 4255 Acked Write 10 Done Sent Done false
12559 4256 Acked Read 6 Done Sent Done false
12560 4257 Acked Write 7 Done Sent Done false
12561 4258 Acked Read 2 Done Sent Done false
12562 4259 Acked Write 1 Done Sent Done false
12563 4260 Acked Write 9 Done Sent Done false
12564 4261 Acked Write 10 Done Sent Done false
12565 4262 Acked Read 10 Done Sent Done false
12566 4263 Acked Read 7 Done Sent Done false
12567 4264 Acked Write 8 Done Sent Done false
12568 4265 Acked Read 9 Done Sent Done false
12569 4266 Acked Read 1 Done Sent Done false
12570 4267 Acked Write 8 Done Sent Done false
12571 4268 Acked Write 6 Done Sent Done false
12572 4269 Acked Read 1 Done Sent Done false
12573 4270 Acked Read 1 Done Sent Done false
12574 4271 Acked Write 7 Done Sent Done false
12575 4272 Acked Read 6 Done Sent Done false
12576 4273 Acked Write 6 Done Sent Done false
12577 4274 Acked Write 3 Done Sent Done false
12578 4275 Acked Read 4 Done Sent Done false
12579 4276 Acked Write 1 Done Sent Done false
12580 4277 Acked Write 10 Done Sent Done false
12581 4278 Acked Write 3 Done Sent Done false
12582 4279 Acked Write 5 Done Sent Done false
12583 4280 Acked Write 4 Done Sent Done false
12584 4281 Acked Read 4 Done Sent Done false
12585 4282 Acked Read 3 Done Sent Done false
12586 4283 Acked Write 1 Done Sent Done false
12587 4284 Acked Read 8 Done Sent Done false
12588 4285 Acked Write 4 Done Sent Done false
12589 4286 Acked Flush 0 Done Sent Done false
12590 4287 Acked Write 3 Done Sent Done false
12591 4288 Acked Write 1 Done Sent Done false
12592 4289 Acked Read 2 Done Sent Done false
12593 4290 Acked Read 3 Done Sent Done false
12594 4291 Acked Read 2 Done Sent Done false
12595 4292 Acked Write 7 Done Sent Done false
12596 4293 Acked Read 5 Done Sent Done false
12597 4294 Acked Read 5 Done Sent Done false
12598 4295 Acked Write 4 Done Sent Done false
12599 4296 Acked Write 1 Done Sent Done false
12600 4297 Acked Read 6 Done Sent Done false
12601 4298 Acked Read 6 Done Sent Done false
12602 4299 Acked Read 9 Done Sent Done false
12603 4300 Acked Read 10 Done Sent Done false
12604 4301 Acked Read 1 Done Sent Done false
12605 4302 Acked Write 2 Done Sent Done false
12606 4303 Acked Read 9 Done Sent Done false
12607 4304 Acked Write 9 Done Sent Done false
12608 4305 Acked Read 4 Done Sent Done false
12609 4306 Acked Write 10 Done Sent Done false
12610 4307 Acked Read 3 Done Sent Done false
12611 4308 Acked Write 3 Done Sent Done false
12612 4309 Acked Read 2 Done Sent Done false
12613 4310 Acked Read 6 Done Sent Done false
12614 4311 Acked Read 1 Done Sent Done false
12615 4312 Acked Read 3 Done Sent Done false
12616 4313 Acked Read 5 Done Sent Done false
12617 4314 Acked Read 7 Done Sent Done false
12618 4315 Acked Read 2 Done Sent Done false
12619 4316 Acked Write 8 Done Sent Done false
12620 4317 Acked Read 6 Done Sent Done false
12621 4318 Acked Read 1 Done Sent Done false
12622 4319 Acked Read 8 Done Sent Done false
12623 4320 Acked Read 6 Done Sent Done false
12624 STATES DS:0 DS:1 DS:2 TOTAL
12625 Sent 0 301 0 301
12626 Done 301 0 301 602
12627 Skipped 0 0 0 0
12628 Error 0 0 0 0
12629 Last Flush: Some(JobId(4286)) Some(JobId(4019)) Some(JobId(4286))
12630 Downstairs last five completed: 4019 4018 4017 4016 4015
12631 Upstairs last five completed: 4320 4319 4318 4317 4316
12632 CLIENT: Up:0 ds:301 act:2
126332025-02-25T17:21:01.543ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
126342025-02-25T17:21:02.545ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
126352025-02-25T17:21:03.547ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
12636 [8820] got start action
12637 [8820] I am going to restart
12638 Starting downstairs at port 8820
12639 Make output file at "/tmp/dsc/downstairs-8820.txt"
12640 [8820][1] reports Starting
12641 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1219)
12642 [8820][1] reports Running
126432025-02-25T17:21:04.549ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
12644 ----------------------------------------------------------------
12645 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12646 States: Active Offline Active
12647 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12648 4020 Acked Write 10 Done Sent Done false
12649 4021 Acked Read 9 Done Sent Done false
12650 4022 Acked Read 2 Done Sent Done false
12651 4023 Acked Flush 0 Done Sent Done false
12652 4024 Acked Read 3 Done Sent Done false
12653 4025 Acked Read 8 Done Sent Done false
12654 4026 Acked Write 1 Done Sent Done false
12655 4027 Acked Write 7 Done Sent Done false
12656 4028 Acked Read 4 Done Sent Done false
12657 4029 Acked Flush 0 Done Sent Done false
12658 4030 Acked Read 9 Done Sent Done false
12659 4031 Acked Read 5 Done Sent Done false
12660 4032 Acked Read 6 Done Sent Done false
12661 4033 Acked Write 6 Done Sent Done false
12662 4034 Acked Read 10 Done Sent Done false
12663 4035 Acked Read 8 Done Sent Done false
12664 4036 Acked Flush 0 Done Sent Done false
12665 4037 Acked Read 8 Done Sent Done false
12666 4038 Acked Write 5 Done Sent Done false
12667 4039 Acked Write 10 Done Sent Done false
12668 4040 Acked Write 3 Done Sent Done false
12669 4041 Acked Read 1 Done Sent Done false
12670 4042 Acked Write 9 Done Sent Done false
12671 4043 Acked Read 7 Done Sent Done false
12672 4044 Acked Write 7 Done Sent Done false
12673 4045 Acked Read 2 Done Sent Done false
12674 4046 Acked Read 1 Done Sent Done false
12675 4047 Acked Read 5 Done Sent Done false
12676 4048 Acked Write 2 Done Sent Done false
12677 4049 Acked Write 4 Done Sent Done false
12678 4050 Acked Read 1 Done Sent Done false
12679 4051 Acked Write 7 Done Sent Done false
12680 4052 Acked Write 4 Done Sent Done false
12681 4053 Acked Write 3 Done Sent Done false
12682 4054 Acked Read 9 Done Sent Done false
12683 4055 Acked Read 4 Done Sent Done false
12684 4056 Acked Read 3 Done Sent Done false
12685 4057 Acked Read 10 Done Sent Done false
12686 4058 Acked Write 1 Done Sent Done false
12687 4059 Acked Read 2 Done Sent Done false
12688 4060 Acked Read 7 Done Sent Done false
12689 4061 Acked Write 6 Done Sent Done false
12690 4062 Acked Flush 0 Done Sent Done false
12691 4063 Acked Read 8 Done Sent Done false
12692 4064 Acked Write 9 Done Sent Done false
12693 4065 Acked Read 6 Done Sent Done false
12694 4066 Acked Read 4 Done Sent Done false
12695 4067 Acked Write 5 Done Sent Done false
12696 4068 Acked Write 10 Done Sent Done false
12697 4069 Acked Read 5 Done Sent Done false
12698 4070 Acked Write 5 Done Sent Done false
12699 4071 Acked Read 8 Done Sent Done false
12700 4072 Acked Read 10 Done Sent Done false
12701 4073 Acked Write 2 Done Sent Done false
12702 4074 Acked Write 6 Done Sent Done false
12703 4075 Acked Write 9 Done Sent Done false
12704 4076 Acked Write 5 Done Sent Done false
12705 4077 Acked Read 7 Done Sent Done false
12706 4078 Acked Write 4 Done Sent Done false
12707 4079 Acked Read 7 Done Sent Done false
12708 4080 Acked Read 8 Done Sent Done false
12709 4081 Acked Write 8 Done Sent Done false
12710 4082 Acked Read 8 Done Sent Done false
12711 4083 Acked Read 3 Done Sent Done false
12712 4084 Acked Write 10 Done Sent Done false
12713 4085 Acked Write 10 Done Sent Done false
12714 4086 Acked Read 1 Done Sent Done false
12715 4087 Acked Flush 0 Done Sent Done false
12716 4088 Acked Read 1 Done Sent Done false
12717 4089 Acked Flush 0 Done Sent Done false
12718 4090 Acked Write 6 Done Sent Done false
12719 4091 Acked Flush 0 Done Sent Done false
12720 4092 Acked Write 9 Done Sent Done false
12721 4093 Acked Write 3 Done Sent Done false
12722 4094 Acked Read 8 Done Sent Done false
12723 4095 Acked Write 9 Done Sent Done false
12724 4096 Acked Read 5 Done Sent Done false
12725 4097 Acked Write 1 Done Sent Done false
12726 4098 Acked Read 9 Done Sent Done false
12727 4099 Acked Write 10 Done Sent Done false
12728 4100 Acked Write 9 Done Sent Done false
12729 4101 Acked Write 5 Done Sent Done false
12730 4102 Acked Read 10 Done Sent Done false
12731 4103 Acked Read 2 Done Sent Done false
12732 4104 Acked Write 10 Done Sent Done false
12733 4105 Acked Write 2 Done Sent Done false
12734 4106 Acked Read 4 Done Sent Done false
12735 4107 Acked Read 5 Done Sent Done false
12736 4108 Acked Write 1 Done Sent Done false
12737 4109 Acked Read 8 Done Sent Done false
12738 4110 Acked Write 3 Done Sent Done false
12739 4111 Acked Read 10 Done Sent Done false
12740 4112 Acked Read 7 Done Sent Done false
12741 4113 Acked Write 8 Done Sent Done false
12742 4114 Acked Read 6 Done Sent Done false
12743 4115 Acked Write 8 Done Sent Done false
12744 4116 Acked Read 4 Done Sent Done false
12745 4117 Acked Write 10 Done Sent Done false
12746 4118 Acked Read 9 Done Sent Done false
12747 4119 Acked Read 7 Done Sent Done false
12748 4120 Acked Read 2 Done Sent Done false
12749 4121 Acked Read 10 Done Sent Done false
12750 4122 Acked Write 5 Done Sent Done false
12751 4123 Acked Read 5 Done Sent Done false
12752 4124 Acked Write 4 Done Sent Done false
12753 4125 Acked Write 6 Done Sent Done false
12754 4126 Acked Flush 0 Done Sent Done false
12755 4127 Acked Write 5 Done Sent Done false
12756 4128 Acked Read 10 Done Sent Done false
12757 4129 Acked Write 6 Done Sent Done false
12758 4130 Acked Read 9 Done Sent Done false
12759 4131 Acked Flush 0 Done Sent Done false
12760 4132 Acked Write 8 Done Sent Done false
12761 4133 Acked Write 8 Done Sent Done false
12762 4134 Acked Read 8 Done Sent Done false
12763 4135 Acked Read 5 Done Sent Done false
12764 4136 Acked Read 2 Done Sent Done false
12765 4137 Acked Flush 0 Done Sent Done false
12766 4138 Acked Read 9 Done Sent Done false
12767 4139 Acked Write 7 Done Sent Done false
12768 4140 Acked Read 1 Done Sent Done false
12769 4141 Acked Write 1 Done Sent Done false
12770 4142 Acked Read 5 Done Sent Done false
12771 4143 Acked Read 1 Done Sent Done false
12772 4144 Acked Read 9 Done Sent Done false
12773 4145 Acked Write 10 Done Sent Done false
12774 4146 Acked Flush 0 Done Sent Done false
12775 4147 Acked Write 10 Done Sent Done false
12776 4148 Acked Read 4 Done Sent Done false
12777 4149 Acked Flush 0 Done Sent Done false
12778 4150 Acked Write 6 Done Sent Done false
12779 4151 Acked Read 3 Done Sent Done false
12780 4152 Acked Read 10 Done Sent Done false
12781 4153 Acked Read 3 Done Sent Done false
12782 4154 Acked Read 6 Done Sent Done false
12783 4155 Acked Write 2 Done Sent Done false
12784 4156 Acked Write 7 Done Sent Done false
12785 4157 Acked Write 5 Done Sent Done false
12786 4158 Acked Read 3 Done Sent Done false
12787 4159 Acked Write 6 Done Sent Done false
12788 4160 Acked Read 3 Done Sent Done false
12789 4161 Acked Read 10 Done Sent Done false
12790 4162 Acked Write 3 Done Sent Done false
12791 4163 Acked Write 4 Done Sent Done false
12792 4164 Acked Write 1 Done Sent Done false
12793 4165 Acked Read 1 Done Sent Done false
12794 4166 Acked Write 5 Done Sent Done false
12795 4167 Acked Read 2 Done Sent Done false
12796 4168 Acked Read 8 Done Sent Done false
12797 4169 Acked Read 7 Done Sent Done false
12798 4170 Acked Write 6 Done Sent Done false
12799 4171 Acked Write 4 Done Sent Done false
12800 4172 Acked Write 9 Done Sent Done false
12801 4173 Acked Read 7 Done Sent Done false
12802 4174 Acked Read 8 Done Sent Done false
12803 4175 Acked Flush 0 Done Sent Done false
12804 4176 Acked Read 1 Done Sent Done false
12805 4177 Acked Write 5 Done Sent Done false
12806 4178 Acked Read 3 Done Sent Done false
12807 4179 Acked Write 2 Done Sent Done false
12808 4180 Acked Write 7 Done Sent Done false
12809 4181 Acked Read 1 Done Sent Done false
12810 4182 Acked Read 5 Done Sent Done false
12811 4183 Acked Read 7 Done Sent Done false
12812 4184 Acked Read 2 Done Sent Done false
12813 4185 Acked Read 9 Done Sent Done false
12814 4186 Acked Read 5 Done Sent Done false
12815 4187 Acked Write 6 Done Sent Done false
12816 4188 Acked Read 3 Done Sent Done false
12817 4189 Acked Read 7 Done Sent Done false
12818 4190 Acked Read 9 Done Sent Done false
12819 4191 Acked Write 6 Done Sent Done false
12820 4192 Acked Read 2 Done Sent Done false
12821 4193 Acked Write 9 Done Sent Done false
12822 4194 Acked Write 4 Done Sent Done false
12823 4195 Acked Write 1 Done Sent Done false
12824 4196 Acked Read 1 Done Sent Done false
12825 4197 Acked Read 3 Done Sent Done false
12826 4198 Acked Read 1 Done Sent Done false
12827 4199 Acked Read 4 Done Sent Done false
12828 4200 Acked Read 1 Done Sent Done false
12829 4201 Acked Flush 0 Done Sent Done false
12830 4202 Acked Write 8 Done Sent Done false
12831 4203 Acked Read 4 Done Sent Done false
12832 4204 Acked Write 8 Done Sent Done false
12833 4205 Acked Read 5 Done Sent Done false
12834 4206 Acked Flush 0 Done Sent Done false
12835 4207 Acked Read 8 Done Sent Done false
12836 4208 Acked Read 9 Done Sent Done false
12837 4209 Acked Read 7 Done Sent Done false
12838 4210 Acked Flush 0 Done Sent Done false
12839 4211 Acked Read 9 Done Sent Done false
12840 4212 Acked Read 9 Done Sent Done false
12841 4213 Acked Read 9 Done Sent Done false
12842 4214 Acked Write 2 Done Sent Done false
12843 4215 Acked Flush 0 Done Sent Done false
12844 4216 Acked Flush 0 Done Sent Done false
12845 4217 Acked Flush 0 Done Sent Done false
12846 4218 Acked Write 8 Done Sent Done false
12847 4219 Acked Read 3 Done Sent Done false
12848 4220 Acked Write 7 Done Sent Done false
12849 4221 Acked Read 1 Done Sent Done false
12850 4222 Acked Flush 0 Done Sent Done false
12851 4223 Acked Read 4 Done Sent Done false
12852 4224 Acked Write 4 Done Sent Done false
12853 4225 Acked Read 8 Done Sent Done false
12854 4226 Acked Write 4 Done Sent Done false
12855 4227 Acked Read 6 Done Sent Done false
12856 4228 Acked Write 1 Done Sent Done false
12857 4229 Acked Flush 0 Done Sent Done false
12858 4230 Acked Read 3 Done Sent Done false
12859 4231 Acked Read 4 Done Sent Done false
12860 4232 Acked Write 2 Done Sent Done false
12861 4233 Acked Read 7 Done Sent Done false
12862 4234 Acked Flush 0 Done Sent Done false
12863 4235 Acked Read 7 Done Sent Done false
12864 4236 Acked Write 4 Done Sent Done false
12865 4237 Acked Write 7 Done Sent Done false
12866 4238 Acked Write 3 Done Sent Done false
12867 4239 Acked Write 2 Done Sent Done false
12868 4240 Acked Read 8 Done Sent Done false
12869 4241 Acked Write 5 Done Sent Done false
12870 4242 Acked Write 10 Done Sent Done false
12871 4243 Acked Write 4 Done Sent Done false
12872 4244 Acked Read 10 Done Sent Done false
12873 4245 Acked Read 6 Done Sent Done false
12874 4246 Acked Read 9 Done Sent Done false
12875 4247 Acked Write 8 Done Sent Done false
12876 4248 Acked Read 10 Done Sent Done false
12877 4249 Acked Read 3 Done Sent Done false
12878 4250 Acked Write 9 Done Sent Done false
12879 4251 Acked Read 3 Done Sent Done false
12880 4252 Acked Write 6 Done Sent Done false
12881 4253 Acked Write 1 Done Sent Done false
12882 4254 Acked Read 8 Done Sent Done false
12883 4255 Acked Write 10 Done Sent Done false
12884 4256 Acked Read 6 Done Sent Done false
12885 4257 Acked Write 7 Done Sent Done false
12886 4258 Acked Read 2 Done Sent Done false
12887 4259 Acked Write 1 Done Sent Done false
12888 4260 Acked Write 9 Done Sent Done false
12889 4261 Acked Write 10 Done Sent Done false
12890 4262 Acked Read 10 Done Sent Done false
12891 4263 Acked Read 7 Done Sent Done false
12892 4264 Acked Write 8 Done Sent Done false
12893 4265 Acked Read 9 Done Sent Done false
12894 4266 Acked Read 1 Done Sent Done false
12895 4267 Acked Write 8 Done Sent Done false
12896 4268 Acked Write 6 Done Sent Done false
12897 4269 Acked Read 1 Done Sent Done false
12898 4270 Acked Read 1 Done Sent Done false
12899 4271 Acked Write 7 Done Sent Done false
12900 4272 Acked Read 6 Done Sent Done false
12901 4273 Acked Write 6 Done Sent Done false
12902 4274 Acked Write 3 Done Sent Done false
12903 4275 Acked Read 4 Done Sent Done false
12904 4276 Acked Write 1 Done Sent Done false
12905 4277 Acked Write 10 Done Sent Done false
12906 4278 Acked Write 3 Done Sent Done false
12907 4279 Acked Write 5 Done Sent Done false
12908 4280 Acked Write 4 Done Sent Done false
12909 4281 Acked Read 4 Done Sent Done false
12910 4282 Acked Read 3 Done Sent Done false
12911 4283 Acked Write 1 Done Sent Done false
12912 4284 Acked Read 8 Done Sent Done false
12913 4285 Acked Write 4 Done Sent Done false
12914 4286 Acked Flush 0 Done Sent Done false
12915 4287 Acked Write 3 Done Sent Done false
12916 4288 Acked Write 1 Done Sent Done false
12917 4289 Acked Read 2 Done Sent Done false
12918 4290 Acked Read 3 Done Sent Done false
12919 4291 Acked Read 2 Done Sent Done false
12920 4292 Acked Write 7 Done Sent Done false
12921 4293 Acked Read 5 Done Sent Done false
12922 4294 Acked Read 5 Done Sent Done false
12923 4295 Acked Write 4 Done Sent Done false
12924 4296 Acked Write 1 Done Sent Done false
12925 4297 Acked Read 6 Done Sent Done false
12926 4298 Acked Read 6 Done Sent Done false
12927 4299 Acked Read 9 Done Sent Done false
12928 4300 Acked Read 10 Done Sent Done false
12929 4301 Acked Read 1 Done Sent Done false
12930 4302 Acked Write 2 Done Sent Done false
12931 4303 Acked Read 9 Done Sent Done false
12932 4304 Acked Write 9 Done Sent Done false
12933 4305 Acked Read 4 Done Sent Done false
12934 4306 Acked Write 10 Done Sent Done false
12935 4307 Acked Read 3 Done Sent Done false
12936 4308 Acked Write 3 Done Sent Done false
12937 4309 Acked Read 2 Done Sent Done false
12938 4310 Acked Read 6 Done Sent Done false
12939 4311 Acked Read 1 Done Sent Done false
12940 4312 Acked Read 3 Done Sent Done false
12941 4313 Acked Read 5 Done Sent Done false
12942 4314 Acked Read 7 Done Sent Done false
12943 4315 Acked Read 2 Done Sent Done false
12944 4316 Acked Write 8 Done Sent Done false
12945 4317 Acked Read 6 Done Sent Done false
12946 4318 Acked Read 1 Done Sent Done false
12947 4319 Acked Read 8 Done Sent Done false
12948 4320 Acked Read 6 Done Sent Done false
12949 STATES DS:0 DS:1 DS:2 TOTAL
12950 Sent 0 301 0 301
12951 Done 301 0 301 602
12952 Skipped 0 0 0 0
12953 Error 0 0 0 0
12954 Last Flush: Some(JobId(4286)) Some(JobId(4019)) Some(JobId(4286))
12955 Downstairs last five completed: 4019 4018 4017 4016 4015
12956 Upstairs last five completed: 4320 4319 4318 4317 4316
12957 CLIENT: Up:0 ds:301 act:2
129582025-02-25T17:21:05.550ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
129592025-02-25T17:21:06.481ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
129602025-02-25T17:21:06.481ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:40677) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
129612025-02-25T17:21:06.483ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
129622025-02-25T17:21:06.483ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
129632025-02-25T17:21:06.483ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
129642025-02-25T17:21:06.483ZINFOcrucible: send last flush ID to this DS: Some(JobId(4019)) = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
129652025-02-25T17:21:06.483ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
129662025-02-25T17:21:06.483ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4019)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
129672025-02-25T17:21:06.483ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4019)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
12968 ----------------------------------------------------------------
12969 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12970 Downstairs last five completed: 4321 4320 4319 4318 4317
12971 Upstairs last five completed: 4321 4320 4319 4318 4317
12972 CLIENT: Up:0 ds:0 act:3
12973 Replay: All jobs finished
12974 Main task has work to do, go find it
12975 got dsc Stop(2)
12976 stop 2
12977 Feb 25 17:21:08.742 INFO [8830] Got stop action so:false kr:false
12978 request completed, latency_us: 403, response_code: 204, uri: /stop/cid/2, method: POST, req_id: ee719626-06f1-427d-98a8-d1adc4e0c804, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
129792025-02-25T17:21:08.743ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
129802025-02-25T17:21:08.743ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
129812025-02-25T17:21:08.743ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
12982 Feb 25 17:21:08.743 INFO request completed, latency_us: 154, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6c620aa1-384f-4835-b758-d278b6382928, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
129832025-02-25T17:21:08.743ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
12984 [8830] Exited with: ExitStatus(unix_wait_status(9))
12985 [8830][2] reports Exit
12986 Feb 25 17:21:12.745 INFO request completed, latency_us: 212, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7d5ded4f-89d5-4bb0-964f-de235eb51e30, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
12987 001/300 Write block 510 len 16384 data: 3 2 2 3
12988 002/300 Flush
12989 003/300 Flush
12990 004/300 Write block 4550 len 16384 data: 3 3 3 3
12991 005/300 Write block 1490 len 32768 data: 5 4 4 4 3 3 3 3
12992 006/300 Read block 3386 len 8192
12993 007/300 Write block 92 len 40960 data: 2 2 3 3 3 4 4 5 5 4
12994 008/300 Read block 2273 len 4096
12995 009/300 Write block 268 len 20480 data: 4 4 5 5 5
12996 010/300 Read block 542 len 28672
12997 011/300 Read block 2697 len 32768
12998 012/300 Write block 2706 len 4096 data: 4
12999 013/300 Read block 554 len 32768
13000 014/300 Read block 77 len 8192
13001 015/300 Write block 2486 len 4096 data: 4
13002 016/300 Flush
13003 017/300 Read block 3249 len 24576
13004 018/300 Read block 2936 len 16384
13005 019/300 Write block 2260 len 8192 data: 3 4
13006 020/300 Write block 2029 len 36864 data: 3 4 5 5 5 5 5 5 6
13007 021/300 Read block 173 len 32768
13008 022/300 Read block 3301 len 32768
13009 023/300 Read block 266 len 32768
13010 024/300 Read block 2676 len 20480
13011 025/300 Read block 1974 len 24576
13012 026/300 Flush
13013 027/300 Read block 4152 len 12288
13014 028/300 Flush
13015 029/300 Read block 1900 len 32768
13016 030/300 Read block 705 len 8192
13017 031/300 Read block 4357 len 32768
13018 032/300 Write block 3722 len 24576 data: 4 3 4 5 5 4
13019 033/300 Read block 405 len 8192
13020 034/300 Write block 1260 len 36864 data: 4 5 5 5 4 4 3 2 2
13021 035/300 Read block 1114 len 36864
13022 036/300 Write block 588 len 8192 data: 3 3
13023 037/300 Write block 2395 len 36864 data: 5 5 4 4 4 3 2 2 2
13024 038/300 Write block 1361 len 28672 data: 4 5 5 6 6 6 5
13025 039/300 Write block 2763 len 12288 data: 3 3 4
13026 040/300 Read block 1991 len 40960
13027 041/300 Write block 3500 len 28672 data: 3 4 4 3 3 3 3
13028 042/300 Read block 3233 len 24576
13029 043/300 Read block 3403 len 32768
13030 044/300 Read block 4508 len 4096
13031 045/300 Read block 177 len 4096
13032 046/300 Read block 1325 len 36864
13033 047/300 Read block 4705 len 4096
13034 048/300 Write block 1106 len 24576 data: 5 3 3 4 5 6
13035 049/300 Read block 3863 len 16384
13036 050/300 Write block 3272 len 40960 data: 4 4 4 3 2 2 2 2 2 2
13037 051/300 Write block 2766 len 12288 data: 4 5 5
13038 052/300 Read block 2346 len 4096
13039 053/300 Read block 3790 len 24576
13040 054/300 Read block 4725 len 32768
13041 055/300 Write block 1407 len 8192 data: 3 3
13042 056/300 Flush
13043 057/300 Read block 3836 len 12288
13044 058/300 Read block 2845 len 16384
13045 059/300 Read block 427 len 4096
13046 060/300 Flush
13047 061/300 Read block 1132 len 28672
13048 062/300 Write block 4588 len 4096 data: 3
13049 063/300 Read block 2888 len 28672
13050 064/300 Read block 1558 len 12288
13051 065/300 Write block 1051 len 4096 data: 2
13052 066/300 Write block 3330 len 28672 data: 2 2 3 3 3 3 3
13053 067/300 Flush
13054 068/300 Read block 4722 len 12288
13055 069/300 Read block 3451 len 20480
13056 070/300 Write block 2366 len 12288 data: 3 3 2
13057 071/300 Write block 202 len 24576 data: 4 4 4 4 4 4
13058 072/300 Write block 343 len 4096 data: 2
13059 073/300 Read block 3199 len 20480
13060 074/300 Write block 1616 len 16384 data: 4 4 4 4
13061 075/300 Read block 2652 len 40960
13062 076/300 Write block 2341 len 8192 data: 2 2
13063 077/300 Write block 488 len 8192 data: 3 5
13064 078/300 Write block 2791 len 36864 data: 5 5 5 4 4 4 5 6 5
13065 079/300 Read block 2316 len 20480
13066 080/300 Write block 3751 len 4096 data: 3
13067 081/300 Read block 397 len 24576
13068 082/300 Write block 3133 len 4096 data: 3
13069 083/300 Read block 1871 len 28672
13070 084/300 Write block 2802 len 24576 data: 2 2 2 3 3 4
13071 085/300 Read block 2894 len 28672
13072 086/300 Write block 723 len 40960 data: 2 2 2 2 3 4 4 4 3 4
13073 087/300 Write block 1281 len 20480 data: 2 2 2 3 3
13074 088/300 Read block 4948 len 28672
13075 089/300 Read block 1159 len 36864
13076 090/300 Read block 3336 len 40960
13077 091/300 Write block 1131 len 8192 data: 2 2
13078 092/300 Flush
13079 093/300 Read block 2135 len 8192
13080 094/300 Read block 4814 len 20480
13081 095/300 Read block 4552 len 12288
13082 096/300 Read block 3825 len 20480
13083 097/300 Read block 3358 len 28672
13084 098/300 Write block 208 len 36864 data: 3 4 4 4 4 3 4 4 4
13085 099/300 Write block 3335 len 40960 data: 4 4 3 3 3 3 4 2 2 2
13086 100/300 Read block 4470 len 20480
13087 101/300 Read block 174 len 12288
13088 102/300 Read block 4143 len 32768
13089 103/300 Read block 3115 len 4096
13090 104/300 Read block 2200 len 4096
13091 105/300 Write block 1683 len 4096 data: 2
13092 106/300 Write block 3932 len 28672 data: 4 3 3 2 3 5 4
13093 107/300 Read block 4494 len 4096
13094 108/300 Write block 3171 len 36864 data: 2 2 2 2 2 3 3 3 4
13095 109/300 Read block 4949 len 16384
13096 110/300 Read block 4589 len 16384
13097 111/300 Write block 2914 len 20480 data: 2 2 2 3 4
13098 112/300 Read block 2175 len 12288
13099 113/300 Write block 4569 len 40960 data: 2 2 2 2 2 2 3 3 3 4
13100 114/300 Read block 3348 len 12288
13101 115/300 Read block 81 len 12288
13102 116/300 Read block 1160 len 20480
13103 117/300 Read block 1716 len 32768
13104 118/300 Flush
13105 119/300 Read block 2281 len 16384
13106 120/300 Write block 108 len 24576 data: 4 5 5 4 4 4
13107 121/300 Read block 1952 len 16384
13108 122/300 Write block 88 len 32768 data: 3 3 2 2 3 3 4 4
13109 123/300 Read block 1766 len 28672
13110 124/300 Read block 1695 len 40960
13111 125/300 Read block 2794 len 24576
13112 126/300 Flush
13113 127/300 Write block 1953 len 40960 data: 3 4 3 3 3 3 3 4 4 3
13114 128/300 Write block 10 len 20480 data: 2 2 3 4 4
13115 129/300 Read block 3933 len 20480
13116 130/300 Write block 4257 len 24576 data: 2 2 2 3 3 3
13117 131/300 Write block 1411 len 24576 data: 2 2 3 3 3 2
13118 132/300 Write block 3505 len 12288 data: 4 4 3
13119 133/300 Write block 79 len 28672 data: 2 3 3 3 3 3 3
13120 134/300 Read block 4692 len 16384
13121 135/300 Read block 157 len 16384
13122 136/300 Write block 4694 len 20480 data: 2 2 3 4 4
13123 137/300 Write block 29 len 4096 data: 4
13124 138/300 Write block 2325 len 24576 data: 5 4 4 4 5 6
13125 139/300 Read block 2022 len 12288
13126 140/300 Read block 112 len 16384
13127 141/300 Read block 4169 len 8192
13128 142/300 Flush
13129 143/300 Read block 1584 len 24576
13130 144/300 Flush
13131 145/300 Read block 3501 len 32768
13132 146/300 Write block 635 len 16384 data: 2 3 3 3
13133 147/300 Write block 2318 len 4096 data: 3
13134 148/300 Read block 2249 len 4096
13135 149/300 Read block 3181 len 12288
13136 150/300 Flush
13137 151/300 Read block 3758 len 12288
13138 152/300 Read block 2918 len 16384
13139 153/300 Write block 1312 len 32768 data: 3 3 4 3 3 2 2 2
13140 154/300 Write block 756 len 20480 data: 2 2 2 2 2
13141 155/300 Read block 3743 len 12288
13142 156/300 Read block 47 len 8192
13143 157/300 Read block 3376 len 8192
13144 158/300 Write block 629 len 32768 data: 7 7 7 6 5 4 3 4
13145 159/300 Flush
13146 160/300 Write block 4991 len 32768 data: 4 3 3 3 3 2 2 2
13147 161/300 Write block 445 len 24576 data: 2 2 2 2 2 2
13148 162/300 Read block 2325 len 16384
13149 163/300 Read block 487 len 24576
13150 164/300 Flush
13151 165/300 Flush
13152 166/300 Read block 4534 len 4096
13153 167/300 Read block 3662 len 16384
13154 168/300 Write block 2833 len 4096 data: 5
13155 169/300 Read block 1543 len 40960
13156 170/300 Read block 3868 len 4096
13157 171/300 Read block 3460 len 8192
13158 172/300 Write block 1768 len 8192 data: 4 4
13159 173/300 Write block 3296 len 20480 data: 4 4 3 4 4
13160 174/300 Read block 2055 len 36864
13161 175/300 Read block 3775 len 40960
13162 176/300 Read block 3369 len 36864
13163 177/300 Write block 3870 len 40960 data: 3 3 4 4 4 3 4 4 4 4
13164 178/300 Read block 543 len 40960
13165 179/300 Read block 4654 len 24576
13166 180/300 Read block 4170 len 28672
13167 181/300 Write block 1939 len 28672 data: 5 5 4 3 3 4 5
13168 182/300 Flush
13169 183/300 Write block 870 len 4096 data: 5
13170 184/300 Read block 4082 len 40960
13171 185/300 Flush
13172 186/300 Read block 1318 len 4096
13173 187/300 Flush
13174 188/300 Read block 4639 len 16384
13175 189/300 Write block 2069 len 36864 data: 3 3 3 3 2 2 2 2 2
13176 190/300 Read block 2601 len 8192
13177 191/300 Read block 4023 len 4096
13178 192/300 Read block 3431 len 36864
13179 193/300 Read block 3613 len 4096
13180 194/300 Write block 3648 len 28672 data: 4 4 4 4 4 5 5
13181 195/300 Read block 2163 len 20480
13182 196/300 Read block 3862 len 4096
13183 197/300 Read block 846 len 40960
13184 198/300 Write block 1831 len 4096 data: 3
13185 199/300 Write block 4348 len 20480 data: 5 4 3 4 3
13186 200/300 Read block 3480 len 40960
13187 201/300 Write block 3034 len 20480 data: 3 3 3 3 3
13188 202/300 Write block 3192 len 8192 data: 2 3
13189 203/300 Read block 2852 len 16384
13190 204/300 Read block 678 len 28672
13191 205/300 Write block 4519 len 24576 data: 5 5 4 4 2 2
13192 206/300 Read block 1413 len 20480
13193 207/300 Write block 1682 len 8192 data: 3 3
13194 208/300 Write block 3213 len 40960 data: 4 4 4 4 5 6 7 5 5 5
13195 209/300 Read block 1027 len 32768
13196 210/300 Read block 3658 len 4096
13197 211/300 Read block 4570 len 32768
13198 212/300 Read block 3445 len 40960
13199 213/300 Read block 1262 len 20480
13200 214/300 Flush
13201 215/300 Read block 4656 len 36864
13202 216/300 Write block 1704 len 12288 data: 5 5 5
13203 217/300 Read block 977 len 40960
13204 218/300 Read block 1472 len 4096
13205 219/300 Read block 2637 len 8192
13206 220/300 Write block 1774 len 28672 data: 5 3 3 3 2 4 3
13207 221/300 Read block 495 len 36864
13208 222/300 Read block 3918 len 36864
13209 223/300 Write block 438 len 16384 data: 3 3 3 4
13210 224/300 Read block 4730 len 24576
13211 225/300 Read block 3072 len 40960
13212 226/300 Read block 1093 len 8192
13213 227/300 Flush
13214 228/300 Read block 4289 len 28672
13215 229/300 Read block 421 len 12288
13216 230/300 Flush
13217 231/300 Read block 2251 len 12288
13218 232/300 Read block 3689 len 12288
13219 233/300 Read block 3331 len 20480
13220 234/300 Write block 2917 len 36864 data: 4 5 4 4 4 3 4 5 4
13221 235/300 Read block 3474 len 32768
13222 236/300 Read block 4953 len 36864
13223 237/300 Read block 3129 len 20480
13224 238/300 Read block 3266 len 4096
13225 239/300 Read block 2074 len 20480
13226 240/300 Read block 3868 len 8192
13227 241/300 Write block 1792 len 4096 data: 2
13228 242/300 Read block 275 len 20480
13229 243/300 Flush
13230 244/300 Read block 4328 len 16384
13231 245/300 Read block 1053 len 36864
13232 246/300 Read block 294 len 28672
13233 247/300 Write block 3919 len 24576 data: 3 4 4 4 4 3
13234 248/300 Write block 4556 len 8192 data: 3 4
13235 249/300 Read block 206 len 4096
13236 250/300 Read block 1396 len 36864
13237 251/300 Read block 1947 len 20480
13238 252/300 Read block 3580 len 40960
13239 253/300 Read block 680 len 40960
13240 254/300 Write block 2973 len 8192 data: 3 4
13241 255/300 Write block 599 len 12288 data: 3 3 3
13242 256/300 Read block 712 len 28672
13243 257/300 Write block 1781 len 12288 data: 2 2 2
13244 258/300 Write block 1639 len 20480 data: 4 3 3 3 3
13245 259/300 Write block 4055 len 40960 data: 2 2 2 2 2 2 4 4 4 4
13246 260/300 Write block 786 len 16384 data: 4 4 4 4
13247 261/300 Write block 713 len 4096 data: 3
13248 262/300 Read block 1493 len 4096
13249 263/300 Write block 4860 len 8192 data: 3 3
13250 264/300 Flush
13251 265/300 Read block 3347 len 8192
13252 266/300 Read block 236 len 4096
13253 267/300 Write block 2061 len 20480 data: 3 2 2 2 4
13254 268/300 Write block 4408 len 16384 data: 2 2 2 2
13255 269/300 Read block 89 len 32768
13256 270/300 Read block 1695 len 12288
13257 271/300 Write block 1501 len 40960 data: 2 2 3 3 3 3 3 3 3 3
13258 272/300 Read block 591 len 20480
13259 273/300 Write block 1646 len 24576 data: 6 6 6 6 5 4
13260 274/300 Read block 2896 len 16384
13261 275/300 Write block 3499 len 24576 data: 3 4 5 5 4 4
13262 276/300 Flush
13263 277/300 Flush
13264 278/300 Flush
13265 279/300 Read block 1237 len 40960
13266 280/300 Write block 3126 len 16384 data: 4 3 2 2
13267 281/300 Read block 126 len 4096
13268 282/300 Write block 3089 len 32768 data: 3 3 4 3 3 3 3 3
13269 283/300 Write block 776 len 40960 data: 3 3 3 4 4 4 4 3 3 3
13270 284/300 Read block 446 len 16384
13271 285/300 Write block 4514 len 40960 data: 3 6 5 5 5 6 6 5 5 3
13272 286/300 Flush
13273 287/300 Write block 3898 len 20480 data: 4 5 4 3 3
13274 288/300 Write block 491 len 40960 data: 4 4 4 4 4 4 4 3 2 2
13275 289/300 Read block 89 len 32768
13276 290/300 Write block 1101 len 16384 data: 2 2 2 3
13277 291/300 Write block 2321 len 24576 data: 4 4 6 5 6 5
13278 292/300 Write block 4418 len 20480 data: 2 2 2 4 5
13279 293/300 Read block 1859 len 32768
13280 294/300 Read block 3662 len 8192
13281 295/300 Read block 2008 len 20480
13282 296/300 Read block 3934 len 16384
13283 297/300 Write block 2502 len 32768 data: 5 5 4 6 4 4 4 4
13284 298/300 Flush
13285 299/300 Write block 23 len 8192 data: 3 3
13286 300/300 Read block 4191 len 36864
13287 301/300 Flush
13288 Main task has work to do, go find it
13289 got dsc Start(2)
13290 start 2
13291 Feb 25 17:21:12.999 INFO request completed, latency_us: 216, response_code: 204, uri: /start/cid/2, method: POST, req_id: a5e75afd-45ee-42b1-9b45-c456c22547c6, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
13292 Replay: started 2, returned:Ok(())
13293 ----------------------------------------------------------------
13294 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13295 States: Active Active Offline
13296 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13297 4322 Acked Write 4 Done Done Sent false
13298 4323 Acked Flush 0 Done Done Sent false
13299 4324 Acked Flush 0 Done Done Sent false
13300 4325 Acked Write 4 Done Done Sent false
13301 4326 Acked Write 8 Done Done Sent false
13302 4327 Acked Read 2 Done Done Sent false
13303 4328 Acked Write 10 Done Done Sent false
13304 4329 Acked Read 1 Done Done Sent false
13305 4330 Acked Write 5 Done Done Sent false
13306 4331 Acked Read 7 Done Done Sent false
13307 4332 Acked Read 8 Done Done Sent false
13308 4333 Acked Write 1 Done Done Sent false
13309 4334 Acked Read 8 Done Done Sent false
13310 4335 Acked Read 2 Done Done Sent false
13311 4336 Acked Write 1 Done Done Sent false
13312 4337 Acked Flush 0 Done Done Sent false
13313 4338 Acked Read 6 Done Done Sent false
13314 4339 Acked Read 4 Done Done Sent false
13315 4340 Acked Write 2 Done Done Sent false
13316 4341 Acked Write 9 Done Done Sent false
13317 4342 Acked Read 8 Done Done Sent false
13318 4343 Acked Read 8 Done Done Sent false
13319 4344 Acked Read 8 Done Done Sent false
13320 4345 Acked Read 5 Done Done Sent false
13321 4346 Acked Read 6 Done Done Sent false
13322 4347 Acked Flush 0 Done Done Sent false
13323 4348 Acked Read 3 Done Done Sent false
13324 4349 Acked Flush 0 Done Done Sent false
13325 4350 Acked Read 8 Done Done Sent false
13326 4351 Acked Read 2 Done Done Sent false
13327 4352 Acked Read 8 Done Done Sent false
13328 4353 Acked Write 6 Done Done Sent false
13329 4354 Acked Read 2 Done Done Sent false
13330 4355 Acked Write 9 Done Done Sent false
13331 4356 Acked Read 9 Done Done Sent false
13332 4357 Acked Write 2 Done Done Sent false
13333 4358 Acked Write 9 Done Done Sent false
13334 4359 Acked Write 7 Done Done Sent false
13335 4360 Acked Write 3 Done Done Sent false
13336 4361 Acked Read 10 Done Done Sent false
13337 4362 Acked Write 7 Done Done Sent false
13338 4363 Acked Read 6 Done Done Sent false
13339 4364 Acked Read 8 Done Done Sent false
13340 4365 Acked Read 1 Done Done Sent false
13341 4366 Acked Read 1 Done Done Sent false
13342 4367 Acked Read 9 Done Done Sent false
13343 4368 Acked Read 1 Done Done Sent false
13344 4369 Acked Write 6 Done Done Sent false
13345 4370 Acked Read 4 Done Done Sent false
13346 4371 Acked Write 10 Done Done Sent false
13347 4372 Acked Write 3 Done Done Sent false
13348 4373 Acked Read 1 Done Done Sent false
13349 4374 Acked Read 6 Done Done Sent false
13350 4375 Acked Read 8 Done Done Sent false
13351 4376 Acked Write 2 Done Done Sent false
13352 4377 Acked Flush 0 Done Done Sent false
13353 4378 Acked Read 3 Done Done Sent false
13354 4379 Acked Read 4 Done Done Sent false
13355 4380 Acked Read 1 Done Done Sent false
13356 4381 Acked Flush 0 Done Done Sent false
13357 4382 Acked Read 7 Done Done Sent false
13358 4383 Acked Write 1 Done Done Sent false
13359 4384 Acked Read 7 Done Done Sent false
13360 4385 Acked Read 3 Done Done Sent false
13361 4386 Acked Write 1 Done Done Sent false
13362 4387 Acked Write 7 Done Done Sent false
13363 4388 Acked Flush 0 Done Done Sent false
13364 4389 Acked Read 3 Done Done Sent false
13365 4390 Acked Read 5 Done Done Sent false
13366 4391 Acked Write 3 Done Done Sent false
13367 4392 Acked Write 6 Done Done Sent false
13368 4393 Acked Write 1 Done Done Sent false
13369 4394 Acked Read 5 Done Done Sent false
13370 4395 Acked Write 4 Done Done Sent false
13371 4396 Acked Read 10 Done Done Sent false
13372 4397 Acked Write 2 Done Done Sent false
13373 4398 Acked Write 2 Done Done Sent false
13374 4399 Acked Write 9 Done Done Sent false
13375 4400 Acked Read 5 Done Done Sent false
13376 4401 Acked Write 1 Done Done Sent false
13377 4402 Acked Read 6 Done Done Sent false
13378 4403 Acked Write 1 Done Done Sent false
13379 4404 Acked Read 7 Done Done Sent false
13380 4405 Acked Write 6 Done Done Sent false
13381 4406 Acked Read 7 Done Done Sent false
13382 4407 Acked Write 10 Done Done Sent false
13383 4408 Acked Write 5 Done Done Sent false
13384 4409 Acked Read 7 Done Done Sent false
13385 4410 Acked Read 9 Done Done Sent false
13386 4411 Acked Read 10 Done Done Sent false
13387 4412 Acked Write 2 Done Done Sent false
13388 4413 Acked Flush 0 Done Done Sent false
13389 4414 Acked Read 2 Done Done Sent false
13390 4415 Acked Read 5 Done Done Sent false
13391 4416 Acked Read 3 Done Done Sent false
13392 4417 Acked Read 5 Done Done Sent false
13393 4418 Acked Read 7 Done Done Sent false
13394 4419 Acked Write 9 Done Done Sent false
13395 4420 Acked Write 10 Done Done Sent false
13396 4421 Acked Read 5 Done Done Sent false
13397 4422 Acked Read 3 Done Done Sent false
13398 4423 Acked Read 8 Done Done Sent false
13399 4424 Acked Read 1 Done Done Sent false
13400 4425 Acked Read 1 Done Done Sent false
13401 4426 Acked Write 1 Done Done Sent false
13402 4427 Acked Write 7 Done Done Sent false
13403 4428 Acked Read 1 Done Done Sent false
13404 4429 Acked Write 9 Done Done Sent false
13405 4430 Acked Read 4 Done Done Sent false
13406 4431 Acked Read 4 Done Done Sent false
13407 4432 Acked Write 5 Done Done Sent false
13408 4433 Acked Read 3 Done Done Sent false
13409 4434 Acked Write 10 Done Done Sent false
13410 4435 Acked Read 3 Done Done Sent false
13411 4436 Acked Read 3 Done Done Sent false
13412 4437 Acked Read 5 Done Done Sent false
13413 4438 Acked Read 8 Done Done Sent false
13414 4439 Acked Flush 0 Done Done Sent false
13415 4440 Acked Read 4 Done Done Sent false
13416 4441 Acked Write 6 Done Done Sent false
13417 4442 Acked Read 4 Done Done Sent false
13418 4443 Acked Write 8 Done Done Sent false
13419 4444 Acked Read 7 Done Done Sent false
13420 4445 Acked Read 10 Done Done Sent false
13421 4446 Acked Read 6 Done Done Sent false
13422 4447 Acked Flush 0 Done Done Sent false
13423 4448 Acked Write 10 Done Done Sent false
13424 4449 Acked Write 5 Done Done Sent false
13425 4450 Acked Read 5 Done Done Sent false
13426 4451 Acked Write 6 Done Done Sent false
13427 4452 Acked Write 6 Done Done Sent false
13428 4453 Acked Write 3 Done Done Sent false
13429 4454 Acked Write 7 Done Done Sent false
13430 4455 Acked Read 4 Done Done Sent false
13431 4456 Acked Read 4 Done Done Sent false
13432 4457 Acked Write 5 Done Done Sent false
13433 4458 Acked Write 1 Done Done Sent false
13434 4459 Acked Write 6 Done Done Sent false
13435 4460 Acked Read 3 Done Done Sent false
13436 4461 Acked Read 4 Done Done Sent false
13437 4462 Acked Read 2 Done Done Sent false
13438 4463 Acked Flush 0 Done Done Sent false
13439 4464 Acked Read 6 Done Done Sent false
13440 4465 Acked Flush 0 Done Done Sent false
13441 4466 Acked Read 8 Done Done Sent false
13442 4467 Acked Write 4 Done Done Sent false
13443 4468 Acked Write 1 Done Done Sent false
13444 4469 Acked Read 1 Done Done Sent false
13445 4470 Acked Read 3 Done Done Sent false
13446 4471 Acked Flush 0 Done Done Sent false
13447 4472 Acked Read 3 Done Done Sent false
13448 4473 Acked Read 4 Done Done Sent false
13449 4474 Acked Write 8 Done Done Sent false
13450 4475 Acked Write 5 Done Done Sent false
13451 4476 Acked Read 3 Done Done Sent false
13452 4477 Acked Read 2 Done Done Sent false
13453 4478 Acked Read 2 Done Done Sent false
13454 4479 Acked Write 8 Done Done Sent false
13455 4480 Acked Flush 0 Done Done Sent false
13456 4481 Acked Write 8 Done Done Sent false
13457 4482 Acked Write 6 Done Done Sent false
13458 4483 Acked Read 4 Done Done Sent false
13459 4484 Acked Read 6 Done Done Sent false
13460 4485 Acked Flush 0 Done Done Sent false
13461 4486 Acked Flush 0 Done Done Sent false
13462 4487 Acked Read 1 Done Done Sent false
13463 4488 Acked Read 4 Done Done Sent false
13464 4489 Acked Write 1 Done Done Sent false
13465 4490 Acked Read 10 Done Done Sent false
13466 4491 Acked Read 1 Done Done Sent false
13467 4492 Acked Read 2 Done Done Sent false
13468 4493 Acked Write 2 Done Done Sent false
13469 4494 Acked Write 5 Done Done Sent false
13470 4495 Acked Read 9 Done Done Sent false
13471 4496 Acked Read 10 Done Done Sent false
13472 4497 Acked Read 9 Done Done Sent false
13473 4498 Acked Write 10 Done Done Sent false
13474 4499 Acked Read 10 Done Done Sent false
13475 4500 Acked Read 6 Done Done Sent false
13476 4501 Acked Read 7 Done Done Sent false
13477 4502 Acked Write 7 Done Done Sent false
13478 4503 Acked Flush 0 Done Done Sent false
13479 4504 Acked Write 1 Done Done Sent false
13480 4505 Acked Read 10 Done Done Sent false
13481 4506 Acked Flush 0 Done Done Sent false
13482 4507 Acked Read 1 Done Done Sent false
13483 4508 Acked Flush 0 Done Done Sent false
13484 4509 Acked Read 4 Done Done Sent false
13485 4510 Acked Write 9 Done Done Sent false
13486 4511 Acked Read 2 Done Done Sent false
13487 4512 Acked Read 1 Done Done Sent false
13488 4513 Acked Read 9 Done Done Sent false
13489 4514 Acked Read 1 Done Done Sent false
13490 4515 Acked Write 7 Done Done Sent false
13491 4516 Acked Read 5 Done Done Sent false
13492 4517 Acked Read 1 Done Done Sent false
13493 4518 Acked Read 10 Done Done Sent false
13494 4519 Acked Write 1 Done Done Sent false
13495 4520 Acked Write 5 Done Done Sent false
13496 4521 Acked Read 10 Done Done Sent false
13497 4522 Acked Write 5 Done Done Sent false
13498 4523 Acked Write 2 Done Done Sent false
13499 4524 Acked Read 4 Done Done Sent false
13500 4525 Acked Read 7 Done Done Sent false
13501 4526 Acked Write 6 Done Done Sent false
13502 4527 Acked Read 5 Done Done Sent false
13503 4528 Acked Write 2 Done Done Sent false
13504 4529 Acked Write 10 Done Done Sent false
13505 4530 Acked Read 8 Done Done Sent false
13506 4531 Acked Read 1 Done Done Sent false
13507 4532 Acked Read 8 Done Done Sent false
13508 4533 Acked Read 10 Done Done Sent false
13509 4534 Acked Read 5 Done Done Sent false
13510 4535 Acked Flush 0 Done Done Sent false
13511 4536 Acked Read 9 Done Done Sent false
13512 4537 Acked Write 3 Done Done Sent false
13513 4538 Acked Read 10 Done Done Sent false
13514 4539 Acked Read 1 Done Done Sent false
13515 4540 Acked Read 2 Done Done Sent false
13516 4541 Acked Write 7 Done Done Sent false
13517 4542 Acked Read 9 Done Done Sent false
13518 4543 Acked Read 9 Done Done Sent false
13519 4544 Acked Write 4 Done Done Sent false
13520 4545 Acked Read 6 Done Done Sent false
13521 4546 Acked Read 10 Done Done Sent false
13522 4547 Acked Read 2 Done Done Sent false
13523 4548 Acked Flush 0 Done Done Sent false
13524 4549 Acked Read 7 Done Done Sent false
13525 4550 Acked Read 3 Done Done Sent false
13526 4551 Acked Flush 0 Done Done Sent false
13527 4552 Acked Read 3 Done Done Sent false
13528 4553 Acked Read 3 Done Done Sent false
13529 4554 Acked Read 5 Done Done Sent false
13530 4555 Acked Write 9 Done Done Sent false
13531 4556 Acked Read 8 Done Done Sent false
13532 4557 Acked Read 9 Done Done Sent false
13533 4558 Acked Read 5 Done Done Sent false
13534 4559 Acked Read 1 Done Done Sent false
13535 4560 Acked Read 5 Done Done Sent false
13536 4561 Acked Read 2 Done Done Sent false
13537 4562 Acked Write 1 Done Done Sent false
13538 4563 Acked Read 5 Done Done Sent false
13539 4564 Acked Flush 0 Done Done Sent false
13540 4565 Acked Read 4 Done Done Sent false
13541 4566 Acked Read 9 Done Done Sent false
13542 4567 Acked Read 7 Done Done Sent false
13543 4568 Acked Write 6 Done Done Sent false
13544 4569 Acked Write 2 Done Done Sent false
13545 4570 Acked Read 1 Done Done Sent false
13546 4571 Acked Read 9 Done Done Sent false
13547 4572 Acked Read 5 Done Done Sent false
13548 4573 Acked Read 10 Done Done Sent false
13549 4574 Acked Read 10 Done Done Sent false
13550 4575 Acked Write 2 Done Done Sent false
13551 4576 Acked Write 3 Done Done Sent false
13552 4577 Acked Read 7 Done Done Sent false
13553 4578 Acked Write 3 Done Done Sent false
13554 4579 Acked Write 5 Done Done Sent false
13555 4580 Acked Write 10 Done Done Sent false
13556 4581 Acked Write 4 Done Done Sent false
13557 4582 Acked Write 1 Done Done Sent false
13558 4583 Acked Read 1 Done Done Sent false
13559 4584 Acked Write 2 Done Done Sent false
13560 4585 Acked Flush 0 Done Done Sent false
13561 4586 Acked Read 2 Done Done Sent false
13562 4587 Acked Read 1 Done Done Sent false
13563 4588 Acked Write 5 Done Done Sent false
13564 4589 Acked Write 4 Done Done Sent false
13565 4590 Acked Read 8 Done Done Sent false
13566 4591 Acked Read 3 Done Done Sent false
13567 4592 Acked Write 10 Done Done Sent false
13568 4593 Acked Read 5 Done Done Sent false
13569 4594 Acked Write 6 Done Done Sent false
13570 4595 Acked Read 4 Done Done Sent false
13571 4596 Acked Write 6 Done Done Sent false
13572 4597 Acked Flush 0 Done Done Sent false
13573 4598 Acked Flush 0 Done Done Sent false
13574 4599 Acked Flush 0 Done Done Sent false
13575 4600 Acked Read 10 Done Done Sent false
13576 4601 Acked Write 4 Done Done Sent false
13577 4602 Acked Read 1 Done Done Sent false
13578 4603 Acked Write 8 Done Done Sent false
13579 4604 Acked Write 10 Done Done Sent false
13580 4605 Acked Read 4 Done Done Sent false
13581 4606 Acked Write 10 Done Done Sent false
13582 4607 Acked Flush 0 Done Done Sent false
13583 4608 Acked Write 5 Done Done Sent false
13584 4609 Acked Write 10 Done Done Sent false
13585 4610 Acked Read 8 Done Done Sent false
13586 4611 Acked Write 4 Done Done Sent false
13587 4612 Acked Write 6 Done Done Sent false
13588 4613 Acked Write 5 Done Done Sent false
13589 4614 Acked Read 8 Done Done Sent false
13590 4615 Acked Read 2 Done Done Sent false
13591 4616 Acked Read 5 Done Done Sent false
13592 4617 Acked Read 4 Done Done Sent false
13593 4618 Acked Write 8 Done Done Sent false
13594 4619 Acked Flush 0 Done Done Sent false
13595 4620 Acked Write 2 Done Done Sent false
13596 4621 Acked Read 9 Done Done Sent false
13597 4622 Acked Flush 0 Done Done Sent false
13598 STATES DS:0 DS:1 DS:2 TOTAL
13599 Sent 0 0 301 301
13600 Done 301 301 0 602
13601 Skipped 0 0 0 0
13602 Error 0 0 0 0
13603 Last Flush: Some(JobId(4622)) Some(JobId(4622)) Some(JobId(4321))
13604 Downstairs last five completed: 4321 4320 4319 4318 4317
13605 Upstairs last five completed: 4622 4621 4620 4619 4618
13606 CLIENT: Up:0 ds:301 act:2
136072025-02-25T17:21:13.560ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
136082025-02-25T17:21:14.562ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
13609 [8830] got start action
13610 [8830] I am going to restart
13611 Starting downstairs at port 8830
13612 Make output file at "/tmp/dsc/downstairs-8830.txt"
13613 [8830][2] reports Starting
13614 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1220)
13615 [8830][2] reports Running
136162025-02-25T17:21:15.563ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
136172025-02-25T17:21:16.986ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
13618 ----------------------------------------------------------------
13619 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13620 States: Active Active Offline
13621 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13622 4322 Acked Write 4 Done Done Sent false
13623 4323 Acked Flush 0 Done Done Sent false
13624 4324 Acked Flush 0 Done Done Sent false
13625 4325 Acked Write 4 Done Done Sent false
13626 4326 Acked Write 8 Done Done Sent false
13627 4327 Acked Read 2 Done Done Sent false
13628 4328 Acked Write 10 Done Done Sent false
13629 4329 Acked Read 1 Done Done Sent false
13630 4330 Acked Write 5 Done Done Sent false
13631 4331 Acked Read 7 Done Done Sent false
13632 4332 Acked Read 8 Done Done Sent false
13633 4333 Acked Write 1 Done Done Sent false
13634 4334 Acked Read 8 Done Done Sent false
13635 4335 Acked Read 2 Done Done Sent false
13636 4336 Acked Write 1 Done Done Sent false
13637 4337 Acked Flush 0 Done Done Sent false
13638 4338 Acked Read 6 Done Done Sent false
13639 4339 Acked Read 4 Done Done Sent false
13640 4340 Acked Write 2 Done Done Sent false
13641 4341 Acked Write 9 Done Done Sent false
13642 4342 Acked Read 8 Done Done Sent false
13643 4343 Acked Read 8 Done Done Sent false
13644 4344 Acked Read 8 Done Done Sent false
13645 4345 Acked Read 5 Done Done Sent false
13646 4346 Acked Read 6 Done Done Sent false
13647 4347 Acked Flush 0 Done Done Sent false
13648 4348 Acked Read 3 Done Done Sent false
13649 4349 Acked Flush 0 Done Done Sent false
13650 4350 Acked Read 8 Done Done Sent false
13651 4351 Acked Read 2 Done Done Sent false
13652 4352 Acked Read 8 Done Done Sent false
13653 4353 Acked Write 6 Done Done Sent false
13654 4354 Acked Read 2 Done Done Sent false
13655 4355 Acked Write 9 Done Done Sent false
13656 4356 Acked Read 9 Done Done Sent false
13657 4357 Acked Write 2 Done Done Sent false
13658 4358 Acked Write 9 Done Done Sent false
13659 4359 Acked Write 7 Done Done Sent false
13660 4360 Acked Write 3 Done Done Sent false
13661 4361 Acked Read 10 Done Done Sent false
13662 4362 Acked Write 7 Done Done Sent false
13663 4363 Acked Read 6 Done Done Sent false
13664 4364 Acked Read 8 Done Done Sent false
13665 4365 Acked Read 1 Done Done Sent false
13666 4366 Acked Read 1 Done Done Sent false
13667 4367 Acked Read 9 Done Done Sent false
13668 4368 Acked Read 1 Done Done Sent false
13669 4369 Acked Write 6 Done Done Sent false
13670 4370 Acked Read 4 Done Done Sent false
13671 4371 Acked Write 10 Done Done Sent false
13672 4372 Acked Write 3 Done Done Sent false
13673 4373 Acked Read 1 Done Done Sent false
13674 4374 Acked Read 6 Done Done Sent false
13675 4375 Acked Read 8 Done Done Sent false
13676 4376 Acked Write 2 Done Done Sent false
13677 4377 Acked Flush 0 Done Done Sent false
13678 4378 Acked Read 3 Done Done Sent false
13679 4379 Acked Read 4 Done Done Sent false
13680 4380 Acked Read 1 Done Done Sent false
13681 4381 Acked Flush 0 Done Done Sent false
13682 4382 Acked Read 7 Done Done Sent false
13683 4383 Acked Write 1 Done Done Sent false
13684 4384 Acked Read 7 Done Done Sent false
13685 4385 Acked Read 3 Done Done Sent false
13686 4386 Acked Write 1 Done Done Sent false
13687 4387 Acked Write 7 Done Done Sent false
13688 4388 Acked Flush 0 Done Done Sent false
13689 4389 Acked Read 3 Done Done Sent false
13690 4390 Acked Read 5 Done Done Sent false
13691 4391 Acked Write 3 Done Done Sent false
13692 4392 Acked Write 6 Done Done Sent false
13693 4393 Acked Write 1 Done Done Sent false
13694 4394 Acked Read 5 Done Done Sent false
13695 4395 Acked Write 4 Done Done Sent false
13696 4396 Acked Read 10 Done Done Sent false
13697 4397 Acked Write 2 Done Done Sent false
13698 4398 Acked Write 2 Done Done Sent false
13699 4399 Acked Write 9 Done Done Sent false
13700 4400 Acked Read 5 Done Done Sent false
13701 4401 Acked Write 1 Done Done Sent false
13702 4402 Acked Read 6 Done Done Sent false
13703 4403 Acked Write 1 Done Done Sent false
13704 4404 Acked Read 7 Done Done Sent false
13705 4405 Acked Write 6 Done Done Sent false
13706 4406 Acked Read 7 Done Done Sent false
13707 4407 Acked Write 10 Done Done Sent false
13708 4408 Acked Write 5 Done Done Sent false
13709 4409 Acked Read 7 Done Done Sent false
13710 4410 Acked Read 9 Done Done Sent false
13711 4411 Acked Read 10 Done Done Sent false
13712 4412 Acked Write 2 Done Done Sent false
13713 4413 Acked Flush 0 Done Done Sent false
13714 4414 Acked Read 2 Done Done Sent false
13715 4415 Acked Read 5 Done Done Sent false
13716 4416 Acked Read 3 Done Done Sent false
13717 4417 Acked Read 5 Done Done Sent false
13718 4418 Acked Read 7 Done Done Sent false
13719 4419 Acked Write 9 Done Done Sent false
13720 4420 Acked Write 10 Done Done Sent false
13721 4421 Acked Read 5 Done Done Sent false
13722 4422 Acked Read 3 Done Done Sent false
13723 4423 Acked Read 8 Done Done Sent false
13724 4424 Acked Read 1 Done Done Sent false
13725 4425 Acked Read 1 Done Done Sent false
13726 4426 Acked Write 1 Done Done Sent false
13727 4427 Acked Write 7 Done Done Sent false
13728 4428 Acked Read 1 Done Done Sent false
13729 4429 Acked Write 9 Done Done Sent false
13730 4430 Acked Read 4 Done Done Sent false
13731 4431 Acked Read 4 Done Done Sent false
13732 4432 Acked Write 5 Done Done Sent false
13733 4433 Acked Read 3 Done Done Sent false
13734 4434 Acked Write 10 Done Done Sent false
13735 4435 Acked Read 3 Done Done Sent false
13736 4436 Acked Read 3 Done Done Sent false
13737 4437 Acked Read 5 Done Done Sent false
13738 4438 Acked Read 8 Done Done Sent false
13739 4439 Acked Flush 0 Done Done Sent false
13740 4440 Acked Read 4 Done Done Sent false
13741 4441 Acked Write 6 Done Done Sent false
13742 4442 Acked Read 4 Done Done Sent false
13743 4443 Acked Write 8 Done Done Sent false
13744 4444 Acked Read 7 Done Done Sent false
13745 4445 Acked Read 10 Done Done Sent false
13746 4446 Acked Read 6 Done Done Sent false
13747 4447 Acked Flush 0 Done Done Sent false
13748 4448 Acked Write 10 Done Done Sent false
13749 4449 Acked Write 5 Done Done Sent false
13750 4450 Acked Read 5 Done Done Sent false
13751 4451 Acked Write 6 Done Done Sent false
13752 4452 Acked Write 6 Done Done Sent false
13753 4453 Acked Write 3 Done Done Sent false
13754 4454 Acked Write 7 Done Done Sent false
13755 4455 Acked Read 4 Done Done Sent false
13756 4456 Acked Read 4 Done Done Sent false
13757 4457 Acked Write 5 Done Done Sent false
13758 4458 Acked Write 1 Done Done Sent false
13759 4459 Acked Write 6 Done Done Sent false
13760 4460 Acked Read 3 Done Done Sent false
13761 4461 Acked Read 4 Done Done Sent false
13762 4462 Acked Read 2 Done Done Sent false
13763 4463 Acked Flush 0 Done Done Sent false
13764 4464 Acked Read 6 Done Done Sent false
13765 4465 Acked Flush 0 Done Done Sent false
13766 4466 Acked Read 8 Done Done Sent false
13767 4467 Acked Write 4 Done Done Sent false
13768 4468 Acked Write 1 Done Done Sent false
13769 4469 Acked Read 1 Done Done Sent false
13770 4470 Acked Read 3 Done Done Sent false
13771 4471 Acked Flush 0 Done Done Sent false
13772 4472 Acked Read 3 Done Done Sent false
13773 4473 Acked Read 4 Done Done Sent false
13774 4474 Acked Write 8 Done Done Sent false
13775 4475 Acked Write 5 Done Done Sent false
13776 4476 Acked Read 3 Done Done Sent false
13777 4477 Acked Read 2 Done Done Sent false
13778 4478 Acked Read 2 Done Done Sent false
13779 4479 Acked Write 8 Done Done Sent false
13780 4480 Acked Flush 0 Done Done Sent false
13781 4481 Acked Write 8 Done Done Sent false
13782 4482 Acked Write 6 Done Done Sent false
13783 4483 Acked Read 4 Done Done Sent false
13784 4484 Acked Read 6 Done Done Sent false
13785 4485 Acked Flush 0 Done Done Sent false
13786 4486 Acked Flush 0 Done Done Sent false
13787 4487 Acked Read 1 Done Done Sent false
13788 4488 Acked Read 4 Done Done Sent false
13789 4489 Acked Write 1 Done Done Sent false
13790 4490 Acked Read 10 Done Done Sent false
13791 4491 Acked Read 1 Done Done Sent false
13792 4492 Acked Read 2 Done Done Sent false
13793 4493 Acked Write 2 Done Done Sent false
13794 4494 Acked Write 5 Done Done Sent false
13795 4495 Acked Read 9 Done Done Sent false
13796 4496 Acked Read 10 Done Done Sent false
13797 4497 Acked Read 9 Done Done Sent false
13798 4498 Acked Write 10 Done Done Sent false
13799 4499 Acked Read 10 Done Done Sent false
13800 4500 Acked Read 6 Done Done Sent false
13801 4501 Acked Read 7 Done Done Sent false
13802 4502 Acked Write 7 Done Done Sent false
13803 4503 Acked Flush 0 Done Done Sent false
13804 4504 Acked Write 1 Done Done Sent false
13805 4505 Acked Read 10 Done Done Sent false
13806 4506 Acked Flush 0 Done Done Sent false
13807 4507 Acked Read 1 Done Done Sent false
13808 4508 Acked Flush 0 Done Done Sent false
13809 4509 Acked Read 4 Done Done Sent false
13810 4510 Acked Write 9 Done Done Sent false
13811 4511 Acked Read 2 Done Done Sent false
13812 4512 Acked Read 1 Done Done Sent false
13813 4513 Acked Read 9 Done Done Sent false
13814 4514 Acked Read 1 Done Done Sent false
13815 4515 Acked Write 7 Done Done Sent false
13816 4516 Acked Read 5 Done Done Sent false
13817 4517 Acked Read 1 Done Done Sent false
13818 4518 Acked Read 10 Done Done Sent false
13819 4519 Acked Write 1 Done Done Sent false
13820 4520 Acked Write 5 Done Done Sent false
13821 4521 Acked Read 10 Done Done Sent false
13822 4522 Acked Write 5 Done Done Sent false
13823 4523 Acked Write 2 Done Done Sent false
13824 4524 Acked Read 4 Done Done Sent false
13825 4525 Acked Read 7 Done Done Sent false
13826 4526 Acked Write 6 Done Done Sent false
13827 4527 Acked Read 5 Done Done Sent false
13828 4528 Acked Write 2 Done Done Sent false
13829 4529 Acked Write 10 Done Done Sent false
13830 4530 Acked Read 8 Done Done Sent false
13831 4531 Acked Read 1 Done Done Sent false
13832 4532 Acked Read 8 Done Done Sent false
13833 4533 Acked Read 10 Done Done Sent false
13834 4534 Acked Read 5 Done Done Sent false
13835 4535 Acked Flush 0 Done Done Sent false
13836 4536 Acked Read 9 Done Done Sent false
13837 4537 Acked Write 3 Done Done Sent false
13838 4538 Acked Read 10 Done Done Sent false
13839 4539 Acked Read 1 Done Done Sent false
13840 4540 Acked Read 2 Done Done Sent false
13841 4541 Acked Write 7 Done Done Sent false
13842 4542 Acked Read 9 Done Done Sent false
13843 4543 Acked Read 9 Done Done Sent false
13844 4544 Acked Write 4 Done Done Sent false
13845 4545 Acked Read 6 Done Done Sent false
13846 4546 Acked Read 10 Done Done Sent false
13847 4547 Acked Read 2 Done Done Sent false
13848 4548 Acked Flush 0 Done Done Sent false
13849 4549 Acked Read 7 Done Done Sent false
13850 4550 Acked Read 3 Done Done Sent false
13851 4551 Acked Flush 0 Done Done Sent false
13852 4552 Acked Read 3 Done Done Sent false
13853 4553 Acked Read 3 Done Done Sent false
13854 4554 Acked Read 5 Done Done Sent false
13855 4555 Acked Write 9 Done Done Sent false
13856 4556 Acked Read 8 Done Done Sent false
13857 4557 Acked Read 9 Done Done Sent false
13858 4558 Acked Read 5 Done Done Sent false
13859 4559 Acked Read 1 Done Done Sent false
13860 4560 Acked Read 5 Done Done Sent false
13861 4561 Acked Read 2 Done Done Sent false
13862 4562 Acked Write 1 Done Done Sent false
13863 4563 Acked Read 5 Done Done Sent false
13864 4564 Acked Flush 0 Done Done Sent false
13865 4565 Acked Read 4 Done Done Sent false
13866 4566 Acked Read 9 Done Done Sent false
13867 4567 Acked Read 7 Done Done Sent false
13868 4568 Acked Write 6 Done Done Sent false
13869 4569 Acked Write 2 Done Done Sent false
13870 4570 Acked Read 1 Done Done Sent false
13871 4571 Acked Read 9 Done Done Sent false
13872 4572 Acked Read 5 Done Done Sent false
13873 4573 Acked Read 10 Done Done Sent false
13874 4574 Acked Read 10 Done Done Sent false
13875 4575 Acked Write 2 Done Done Sent false
13876 4576 Acked Write 3 Done Done Sent false
13877 4577 Acked Read 7 Done Done Sent false
13878 4578 Acked Write 3 Done Done Sent false
13879 4579 Acked Write 5 Done Done Sent false
13880 4580 Acked Write 10 Done Done Sent false
13881 4581 Acked Write 4 Done Done Sent false
13882 4582 Acked Write 1 Done Done Sent false
13883 4583 Acked Read 1 Done Done Sent false
13884 4584 Acked Write 2 Done Done Sent false
13885 4585 Acked Flush 0 Done Done Sent false
13886 4586 Acked Read 2 Done Done Sent false
13887 4587 Acked Read 1 Done Done Sent false
13888 4588 Acked Write 5 Done Done Sent false
13889 4589 Acked Write 4 Done Done Sent false
13890 4590 Acked Read 8 Done Done Sent false
13891 4591 Acked Read 3 Done Done Sent false
13892 4592 Acked Write 10 Done Done Sent false
13893 4593 Acked Read 5 Done Done Sent false
13894 4594 Acked Write 6 Done Done Sent false
13895 4595 Acked Read 4 Done Done Sent false
13896 4596 Acked Write 6 Done Done Sent false
13897 4597 Acked Flush 0 Done Done Sent false
13898 4598 Acked Flush 0 Done Done Sent false
13899 4599 Acked Flush 0 Done Done Sent false
13900 4600 Acked Read 10 Done Done Sent false
13901 4601 Acked Write 4 Done Done Sent false
13902 4602 Acked Read 1 Done Done Sent false
13903 4603 Acked Write 8 Done Done Sent false
13904 4604 Acked Write 10 Done Done Sent false
13905 4605 Acked Read 4 Done Done Sent false
13906 4606 Acked Write 10 Done Done Sent false
13907 4607 Acked Flush 0 Done Done Sent false
13908 4608 Acked Write 5 Done Done Sent false
13909 4609 Acked Write 10 Done Done Sent false
13910 4610 Acked Read 8 Done Done Sent false
13911 4611 Acked Write 4 Done Done Sent false
13912 4612 Acked Write 6 Done Done Sent false
13913 4613 Acked Write 5 Done Done Sent false
13914 4614 Acked Read 8 Done Done Sent false
13915 4615 Acked Read 2 Done Done Sent false
13916 4616 Acked Read 5 Done Done Sent false
13917 4617 Acked Read 4 Done Done Sent false
13918 4618 Acked Write 8 Done Done Sent false
13919 4619 Acked Flush 0 Done Done Sent false
13920 4620 Acked Write 2 Done Done Sent false
13921 4621 Acked Read 9 Done Done Sent false
13922 4622 Acked Flush 0 Done Done Sent false
13923 STATES DS:0 DS:1 DS:2 TOTAL
13924 Sent 0 0 301 301
13925 Done 301 301 0 602
13926 Skipped 0 0 0 0
13927 Error 0 0 0 0
13928 Last Flush: Some(JobId(4622)) Some(JobId(4622)) Some(JobId(4321))
13929 Downstairs last five completed: 4321 4320 4319 4318 4317
13930 Upstairs last five completed: 4622 4621 4620 4619 4618
13931 CLIENT: Up:0 ds:301 act:2
139322025-02-25T17:21:17.566ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139332025-02-25T17:21:18.568ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139342025-02-25T17:21:18.744ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139352025-02-25T17:21:18.744ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39963) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139362025-02-25T17:21:18.746ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139372025-02-25T17:21:18.746ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139382025-02-25T17:21:18.746ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139392025-02-25T17:21:18.746ZINFOcrucible: send last flush ID to this DS: Some(JobId(4321)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139402025-02-25T17:21:18.746ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139412025-02-25T17:21:18.746ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4321)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139422025-02-25T17:21:18.746ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4321)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
13943 ----------------------------------------------------------------
13944 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13945 Downstairs last five completed: 4622 4621 4620 4619 4618
13946 Upstairs last five completed: 4622 4621 4620 4619 4618
13947 CLIENT: Up:0 ds:0 act:3
13948 Replay: All jobs finished
13949 Main task has work to do, go find it
13950 got dsc Stop(2)
13951 stop 2
13952 Feb 25 17:21:21.004 INFO request completed, latency_us: 351, response_code: 204, [8830] Got stop action so:false kr:false
13953 uri: /stop/cid/2, method: POST, req_id: d0bab064-626c-4b33-93ba-ffe1dad192dc, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
13954 Feb 25 17:21:21.005 INFO request completed, latency_us: 116, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2efa8429-58ab-489f-9014-05e9fbf4eee1, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
139552025-02-25T17:21:21.005ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139562025-02-25T17:21:21.005ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139572025-02-25T17:21:21.005ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
139582025-02-25T17:21:21.005ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
13959 [8830] Exited with: ExitStatus(unix_wait_status(9))
13960 [8830][2] reports Exit
13961 Feb 25 17:21:25.008 INFO request completed, latency_us: 188, response_code: 200, uri: /state/cid/2, method: GET, req_id: af7aa9c4-dd6f-49e5-990f-44316ad58e8c, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
13962 001/300 Write block 1446 len 24576 data: 5 4 4 3 4 4
13963 002/300 Write block 4981 len 12288 data: 2 2 2
13964 003/300 Read block 2088 len 40960
13965 004/300 Flush
13966 005/300 Write block 3054 len 28672 data: 3 3 3 4 3 3 4
13967 006/300 Read block 4947 len 12288
13968 007/300 Write block 1102 len 12288 data: 3 3 4
13969 008/300 Read block 3799 len 8192
13970 009/300 Read block 3503 len 4096
13971 010/300 Write block 4202 len 32768 data: 4 4 4 4 4 4 4 3
13972 011/300 Read block 3227 len 28672
13973 012/300 Read block 4750 len 24576
13974 013/300 Flush
13975 014/300 Read block 758 len 12288
13976 015/300 Write block 3366 len 12288 data: 3 3 4
13977 016/300 Read block 1864 len 16384
13978 017/300 Read block 3974 len 24576
13979 018/300 Read block 4658 len 8192
13980 019/300 Write block 1231 len 16384 data: 2 3 3 5
13981 020/300 Write block 1969 len 16384 data: 2 2 2 2
13982 021/300 Read block 2415 len 16384
13983 022/300 Read block 2758 len 32768
13984 023/300 Write block 4411 len 20480 data: 3 2 3 3 3
13985 024/300 Read block 4129 len 40960
13986 025/300 Flush
13987 026/300 Write block 830 len 20480 data: 3 3 5 8 9
13988 027/300 Read block 514 len 16384
13989 028/300 Read block 4708 len 8192
13990 029/300 Read block 4421 len 8192
13991 030/300 Read block 1795 len 20480
13992 031/300 Flush
13993 032/300 Read block 4528 len 16384
13994 033/300 Write block 1005 len 36864 data: 4 2 2 2 2 2 2 3 3
13995 034/300 Write block 240 len 8192 data: 3 3
13996 035/300 Read block 875 len 12288
13997 036/300 Flush
13998 037/300 Read block 3102 len 36864
13999 038/300 Write block 4612 len 24576 data: 3 4 4 4 4 4
14000 039/300 Read block 1454 len 20480
14001 040/300 Write block 2449 len 4096 data: 3
14002 041/300 Read block 121 len 40960
14003 042/300 Read block 2917 len 24576
14004 043/300 Read block 4573 len 40960
14005 044/300 Read block 3101 len 28672
14006 045/300 Flush
14007 046/300 Flush
14008 047/300 Read block 4219 len 36864
14009 048/300 Read block 4668 len 12288
14010 049/300 Write block 2421 len 36864 data: 2 3 3 3 3 3 3 3 3
14011 050/300 Read block 4466 len 8192
14012 051/300 Read block 716 len 8192
14013 052/300 Read block 547 len 32768
14014 053/300 Read block 2228 len 20480
14015 054/300 Read block 162 len 20480
14016 055/300 Write block 2418 len 16384 data: 2 2 2 3
14017 056/300 Read block 33 len 32768
14018 057/300 Read block 3401 len 32768
14019 058/300 Write block 3058 len 28672 data: 4 4 5 4 4 3 3
14020 059/300 Read block 265 len 28672
14021 060/300 Read block 219 len 16384
14022 061/300 Read block 896 len 28672
14023 062/300 Flush
14024 063/300 Read block 4154 len 20480
14025 064/300 Read block 3639 len 40960
14026 065/300 Read block 4165 len 20480
14027 066/300 Write block 4020 len 32768 data: 3 4 6 5 6 7 6 6
14028 067/300 Write block 2219 len 24576 data: 6 5 4 4 4 3
14029 068/300 Write block 1621 len 32768 data: 3 3 3 3 5 6 5 6
14030 069/300 Read block 2428 len 40960
14031 070/300 Read block 1176 len 8192
14032 071/300 Read block 2226 len 16384
14033 072/300 Flush
14034 073/300 Read block 4024 len 24576
14035 074/300 Read block 4279 len 28672
14036 075/300 Read block 670 len 8192
14037 076/300 Flush
14038 077/300 Write block 3798 len 24576 data: 4 3 2 2 2 2
14039 078/300 Write block 1928 len 24576 data: 3 4 5 6 6 6
14040 079/300 Write block 3373 len 20480 data: 2 2 2 2 2
14041 080/300 Write block 2346 len 32768 data: 3 4 4 4 4 5 6 5
14042 081/300 Read block 3730 len 36864
14043 082/300 Write block 1914 len 32768 data: 4 3 3 3 3 2 2 3
14044 083/300 Read block 2802 len 8192
14045 084/300 Read block 3436 len 28672
14046 085/300 Read block 2414 len 4096
14047 086/300 Write block 2780 len 28672 data: 2 3 3 3 3 2 4
14048 087/300 Read block 1522 len 24576
14049 088/300 Write block 2297 len 40960 data: 3 4 4 4 3 4 5 5 4 3
14050 089/300 Write block 3485 len 20480 data: 4 4 5 5 5
14051 090/300 Write block 1721 len 16384 data: 5 5 4 4
14052 091/300 Flush
14053 092/300 Write block 35 len 32768 data: 3 4 4 4 3 4 4 3
14054 093/300 Flush
14055 094/300 Read block 786 len 32768
14056 095/300 Flush
14057 096/300 Write block 1703 len 28672 data: 5 6 6 6 3 2 2
14058 097/300 Read block 3256 len 8192
14059 098/300 Write block 4084 len 16384 data: 3 3 3 2
14060 099/300 Write block 4720 len 4096 data: 4
14061 100/300 Write block 2796 len 4096 data: 5
14062 101/300 Read block 4646 len 32768
14063 102/300 Write block 1100 len 36864 data: 4 3 4 4 5 3 6 4 4
14064 103/300 Write block 1706 len 28672 data: 7 4 3 3 4 4 5
14065 104/300 Write block 3110 len 12288 data: 5 5 4
14066 105/300 Write block 1333 len 16384 data: 2 2 2 3
14067 106/300 Read block 4499 len 32768
14068 107/300 Read block 588 len 32768
14069 108/300 Read block 1085 len 40960
14070 109/300 Read block 1005 len 4096
14071 110/300 Write block 2085 len 24576 data: 5 5 4 4 2 3
14072 111/300 Read block 830 len 24576
14073 112/300 Flush
14074 113/300 Read block 4044 len 4096
14075 114/300 Read block 486 len 40960
14076 115/300 Flush
14077 116/300 Read block 1483 len 32768
14078 117/300 Write block 3852 len 36864 data: 3 4 5 5 5 3 2 2 3
14079 118/300 Flush
14080 119/300 Write block 3730 len 28672 data: 6 6 6 5 6 4 5
14081 120/300 Flush
14082 121/300 Write block 3391 len 24576 data: 3 4 4 4 3 3
14083 122/300 Read block 1425 len 40960
14084 123/300 Write block 1053 len 28672 data: 4 4 4 4 4 3 3
14085 124/300 Write block 2589 len 16384 data: 3 4 4 4
14086 125/300 Flush
14087 126/300 Read block 4218 len 36864
14088 127/300 Read block 3687 len 8192
14089 128/300 Read block 2495 len 20480
14090 129/300 Read block 1021 len 20480
14091 130/300 Read block 3965 len 20480
14092 131/300 Read block 2208 len 4096
14093 132/300 Read block 3598 len 40960
14094 133/300 Write block 509 len 28672 data: 3 4 3 3 4 3 3
14095 134/300 Write block 2472 len 36864 data: 3 3 2 2 2 3 3 3 3
14096 135/300 Read block 2509 len 40960
14097 136/300 Read block 1111 len 4096
14098 137/300 Read block 4524 len 32768
14099 138/300 Read block 1606 len 36864
14100 139/300 Write block 2382 len 8192 data: 2 2
14101 140/300 Read block 4619 len 28672
14102 141/300 Read block 4083 len 32768
14103 142/300 Read block 2734 len 32768
14104 143/300 Read block 2017 len 16384
14105 144/300 Read block 1965 len 24576
14106 145/300 Flush
14107 146/300 Write block 3837 len 36864 data: 4 3 4 4 4 3 2 2 2
14108 147/300 Write block 2459 len 24576 data: 8 8 9 7 5 4
14109 148/300 Read block 319 len 20480
14110 149/300 Read block 2168 len 36864
14111 150/300 Read block 3166 len 8192
14112 151/300 Read block 982 len 32768
14113 152/300 Read block 994 len 36864
14114 153/300 Flush
14115 154/300 Write block 1145 len 28672 data: 3 3 3 3 3 3 3
14116 155/300 Write block 3449 len 20480 data: 4 4 3 3 3
14117 156/300 Write block 866 len 36864 data: 3 4 3 4 6 5 4 3 3
14118 157/300 Read block 3348 len 36864
14119 158/300 Flush
14120 159/300 Read block 3805 len 32768
14121 160/300 Write block 4943 len 36864 data: 4 4 3 4 3 3 3 3 3
14122 161/300 Read block 4879 len 28672
14123 162/300 Read block 2094 len 40960
14124 163/300 Write block 4991 len 12288 data: 5 4 4
14125 164/300 Read block 1127 len 4096
14126 165/300 Flush
14127 166/300 Read block 4220 len 4096
14128 167/300 Write block 601 len 16384 data: 4 3 3 3
14129 168/300 Write block 2352 len 8192 data: 7 6
14130 169/300 Write block 3076 len 32768 data: 4 3 3 3 4 4 5 5
14131 170/300 Write block 3419 len 8192 data: 3 3
14132 171/300 Read block 3598 len 8192
14133 172/300 Read block 1157 len 12288
14134 173/300 Write block 1260 len 8192 data: 5 6
14135 174/300 Write block 1718 len 12288 data: 5 7 6
14136 175/300 Write block 4535 len 40960 data: 5 4 5 5 3 3 3 3 2 2
14137 176/300 Flush
14138 177/300 Write block 1544 len 40960 data: 2 2 2 2 2 3 3 3 3 4
14139 178/300 Write block 456 len 20480 data: 3 3 2 2 2
14140 179/300 Write block 2446 len 28672 data: 3 3 3 4 4 3 3
14141 180/300 Flush
14142 181/300 Read block 2503 len 16384
14143 182/300 Write block 1168 len 16384 data: 4 5 5 3
14144 183/300 Read block 2753 len 8192
14145 184/300 Write block 4940 len 40960 data: 6 7 6 5 5 4 5 4 4 4
14146 185/300 Read block 225 len 36864
14147 186/300 Read block 3318 len 24576
14148 187/300 Read block 1764 len 8192
14149 188/300 Read block 751 len 8192
14150 189/300 Read block 4378 len 40960
14151 190/300 Write block 3023 len 12288 data: 3 2 2
14152 191/300 Write block 247 len 24576 data: 3 3 4 4 3 3
14153 192/300 Read block 3816 len 16384
14154 193/300 Write block 3806 len 8192 data: 3 3
14155 194/300 Read block 320 len 40960
14156 195/300 Read block 2893 len 20480
14157 196/300 Write block 2116 len 12288 data: 4 4 4
14158 197/300 Flush
14159 198/300 Read block 3364 len 12288
14160 199/300 Write block 2228 len 28672 data: 2 2 2 2 2 2 2
14161 200/300 Read block 1507 len 32768
14162 201/300 Read block 3800 len 16384
14163 202/300 Write block 3622 len 4096 data: 2
14164 203/300 Write block 2647 len 20480 data: 2 2 2 2 2
14165 204/300 Write block 3345 len 20480 data: 2 3 3 3 4
14166 205/300 Write block 2948 len 28672 data: 4 3 3 3 3 3 2
14167 206/300 Read block 2514 len 28672
14168 207/300 Read block 3659 len 36864
14169 208/300 Read block 3108 len 4096
14170 209/300 Read block 2247 len 36864
14171 210/300 Read block 1949 len 20480
14172 211/300 Read block 4893 len 40960
14173 212/300 Write block 231 len 12288 data: 4 4 4
14174 213/300 Write block 2637 len 8192 data: 3 3
14175 214/300 Flush
14176 215/300 Read block 622 len 36864
14177 216/300 Flush
14178 217/300 Write block 2601 len 16384 data: 5 4 4 3
14179 218/300 Write block 578 len 36864 data: 3 3 5 5 4 3 4 4 4
14180 219/300 Write block 4147 len 16384 data: 2 3 3 3
14181 220/300 Flush
14182 221/300 Read block 2003 len 16384
14183 222/300 Write block 4468 len 32768 data: 5 3 2 2 2 2 2 2
14184 223/300 Write block 3917 len 20480 data: 3 4 4 5 5
14185 224/300 Read block 1571 len 32768
14186 225/300 Write block 3328 len 36864 data: 2 2 3 3 4 4 4 5 5
14187 226/300 Read block 2167 len 12288
14188 227/300 Write block 3168 len 20480 data: 2 2 2 3 3
14189 228/300 Write block 2789 len 4096 data: 4
14190 229/300 Write block 25 len 16384 data: 3 3 4 4
14191 230/300 Write block 3440 len 32768 data: 2 2 2 2 2 2 3 3
14192 231/300 Flush
14193 232/300 Read block 4345 len 16384
14194 233/300 Read block 4772 len 8192
14195 234/300 Write block 3147 len 20480 data: 2 2 2 2 2
14196 235/300 Write block 1119 len 40960 data: 2 2 2 2 2 2 2 2 2 2
14197 236/300 Write block 1359 len 28672 data: 3 4 5 6 6 7 7
14198 237/300 Write block 3434 len 4096 data: 3
14199 238/300 Read block 4736 len 16384
14200 239/300 Write block 1419 len 36864 data: 2 2 2 3 2 2 2 4 5
14201 240/300 Write block 4970 len 16384 data: 3 2 2 2
14202 241/300 Write block 1163 len 12288 data: 3 3 3
14203 242/300 Flush
14204 243/300 Flush
14205 244/300 Write block 3700 len 28672 data: 5 5 5 4 5 5 4
14206 245/300 Write block 628 len 24576 data: 7 8 8 8 7 6
14207 246/300 Write block 4818 len 28672 data: 5 5 5 5 5 5 5
14208 247/300 Write block 1480 len 20480 data: 4 4 4 5 4
14209 248/300 Read block 89 len 16384
14210 249/300 Write block 1180 len 4096 data: 6
14211 250/300 Read block 1414 len 24576
14212 251/300 Write block 1349 len 12288 data: 3 3 3
14213 252/300 Write block 3553 len 16384 data: 6 6 4 4
14214 253/300 Write block 2752 len 24576 data: 4 3 3 3 2 2
14215 254/300 Write block 4796 len 40960 data: 2 2 4 4 4 4 4 4 4 3
14216 255/300 Read block 1416 len 28672
14217 256/300 Write block 4077 len 16384 data: 3 4 4 3
14218 257/300 Write block 3425 len 12288 data: 3 3 4
14219 258/300 Read block 727 len 24576
14220 259/300 Write block 1663 len 16384 data: 2 2 2 3
14221 260/300 Read block 3278 len 40960
14222 261/300 Read block 4800 len 24576
14223 262/300 Read block 6 len 28672
14224 263/300 Write block 2404 len 24576 data: 3 5 5 5 4 2
14225 264/300 Read block 2527 len 8192
14226 265/300 Read block 2590 len 24576
14227 266/300 Read block 4642 len 12288
14228 267/300 Write block 4255 len 24576 data: 2 2 3 3 3 4
14229 268/300 Write block 4583 len 4096 data: 3
14230 269/300 Read block 3904 len 12288
14231 270/300 Write block 329 len 4096 data: 3
14232 271/300 Write block 3515 len 16384 data: 3 3 3 3
14233 272/300 Flush
14234 273/300 Read block 4581 len 12288
14235 274/300 Read block 1583 len 16384
14236 275/300 Write block 3543 len 4096 data: 4
14237 276/300 Read block 3667 len 40960
14238 277/300 Read block 3882 len 24576
14239 278/300 Read block 1893 len 4096
14240 279/300 Read block 3070 len 36864
14241 280/300 Read block 3286 len 40960
14242 281/300 Read block 3628 len 32768
14243 282/300 Read block 830 len 28672
14244 283/300 Write block 3053 len 8192 data: 3 4
14245 284/300 Read block 1390 len 20480
14246 285/300 Write block 743 len 20480 data: 5 3 3 3 4
14247 286/300 Read block 1624 len 28672
14248 287/300 Flush
14249 288/300 Flush
14250 289/300 Read block 4554 len 28672
14251 290/300 Read block 172 len 28672
14252 291/300 Write block 1105 len 16384 data: 4 7 5 5
14253 292/300 Read block 82 len 32768
14254 293/300 Write block 1831 len 4096 data: 4
14255 294/300 Write block 4349 len 32768 data: 5 4 5 4 4 4 6 6
14256 295/300 Read block 3839 len 20480
14257 296/300 Write block 3461 len 36864 data: 3 3 3 2 2 2 2 2 3
14258 297/300 Flush
14259 298/300 Flush
14260 299/300 Write block 1447 len 16384 data: 5 5 4 5
14261 300/300 Read block 2280 len 40960
14262 301/300 Read block 4634 len 36864
14263 Main task has work to do, go find it
14264 got dsc Start(2)
14265 start 2
14266 Feb 25 17:21:25.261 INFO request completed, latency_us: 216, response_code: 204, uri: /start/cid/2, method: POST, req_id: 39f011cc-8afb-47c8-8475-888438de35bc, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
14267 Replay: started 2, returned:Ok(())
14268 ----------------------------------------------------------------
14269 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14270 States: Active Active Offline
14271 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14272 4623 Acked Write 6 Done Done Sent false
14273 4624 Acked Write 3 Done Done Sent false
14274 4625 Acked Read 10 Done Done Sent false
14275 4626 Acked Flush 0 Done Done Sent false
14276 4627 Acked Write 7 Done Done Sent false
14277 4628 Acked Read 3 Done Done Sent false
14278 4629 Acked Write 3 Done Done Sent false
14279 4630 Acked Read 2 Done Done Sent false
14280 4631 Acked Read 1 Done Done Sent false
14281 4632 Acked Write 8 Done Done Sent false
14282 4633 Acked Read 7 Done Done Sent false
14283 4634 Acked Read 6 Done Done Sent false
14284 4635 Acked Flush 0 Done Done Sent false
14285 4636 Acked Read 3 Done Done Sent false
14286 4637 Acked Write 3 Done Done Sent false
14287 4638 Acked Read 4 Done Done Sent false
14288 4639 Acked Read 6 Done Done Sent false
14289 4640 Acked Read 2 Done Done Sent false
14290 4641 Acked Write 4 Done Done Sent false
14291 4642 Acked Write 4 Done Done Sent false
14292 4643 Acked Read 4 Done Done Sent false
14293 4644 Acked Read 8 Done Done Sent false
14294 4645 Acked Write 5 Done Done Sent false
14295 4646 Acked Read 10 Done Done Sent false
14296 4647 Acked Flush 0 Done Done Sent false
14297 4648 Acked Write 5 Done Done Sent false
14298 4649 Acked Read 4 Done Done Sent false
14299 4650 Acked Read 2 Done Done Sent false
14300 4651 Acked Read 2 Done Done Sent false
14301 4652 Acked Read 5 Done Done Sent false
14302 4653 Acked Flush 0 Done Done Sent false
14303 4654 Acked Read 4 Done Done Sent false
14304 4655 Acked Write 9 Done Done Sent false
14305 4656 Acked Write 2 Done Done Sent false
14306 4657 Acked Read 3 Done Done Sent false
14307 4658 Acked Flush 0 Done Done Sent false
14308 4659 Acked Read 9 Done Done Sent false
14309 4660 Acked Write 6 Done Done Sent false
14310 4661 Acked Read 5 Done Done Sent false
14311 4662 Acked Write 1 Done Done Sent false
14312 4663 Acked Read 10 Done Done Sent false
14313 4664 Acked Read 6 Done Done Sent false
14314 4665 Acked Read 10 Done Done Sent false
14315 4666 Acked Read 7 Done Done Sent false
14316 4667 Acked Flush 0 Done Done Sent false
14317 4668 Acked Flush 0 Done Done Sent false
14318 4669 Acked Read 9 Done Done Sent false
14319 4670 Acked Read 3 Done Done Sent false
14320 4671 Acked Write 9 Done Done Sent false
14321 4672 Acked Read 2 Done Done Sent false
14322 4673 Acked Read 2 Done Done Sent false
14323 4674 Acked Read 8 Done Done Sent false
14324 4675 Acked Read 5 Done Done Sent false
14325 4676 Acked Read 5 Done Done Sent false
14326 4677 Acked Write 4 Done Done Sent false
14327 4678 Acked Read 8 Done Done Sent false
14328 4679 Acked Read 8 Done Done Sent false
14329 4680 Acked Write 7 Done Done Sent false
14330 4681 Acked Read 7 Done Done Sent false
14331 4682 Acked Read 4 Done Done Sent false
14332 4683 Acked Read 7 Done Done Sent false
14333 4684 Acked Flush 0 Done Done Sent false
14334 4685 Acked Read 5 Done Done Sent false
14335 4686 Acked Read 10 Done Done Sent false
14336 4687 Acked Read 5 Done Done Sent false
14337 4688 Acked Write 8 Done Done Sent false
14338 4689 Acked Write 6 Done Done Sent false
14339 4690 Acked Write 8 Done Done Sent false
14340 4691 Acked Read 10 Done Done Sent false
14341 4692 Acked Read 2 Done Done Sent false
14342 4693 Acked Read 4 Done Done Sent false
14343 4694 Acked Flush 0 Done Done Sent false
14344 4695 Acked Read 6 Done Done Sent false
14345 4696 Acked Read 7 Done Done Sent false
14346 4697 Acked Read 2 Done Done Sent false
14347 4698 Acked Flush 0 Done Done Sent false
14348 4699 Acked Write 6 Done Done Sent false
14349 4700 Acked Write 6 Done Done Sent false
14350 4701 Acked Write 5 Done Done Sent false
14351 4702 Acked Write 8 Done Done Sent false
14352 4703 Acked Read 9 Done Done Sent false
14353 4704 Acked Write 8 Done Done Sent false
14354 4705 Acked Read 2 Done Done Sent false
14355 4706 Acked Read 7 Done Done Sent false
14356 4707 Acked Read 1 Done Done Sent false
14357 4708 Acked Write 7 Done Done Sent false
14358 4709 Acked Read 6 Done Done Sent false
14359 4710 Acked Write 10 Done Done Sent false
14360 4711 Acked Write 5 Done Done Sent false
14361 4712 Acked Write 4 Done Done Sent false
14362 4713 Acked Flush 0 Done Done Sent false
14363 4714 Acked Write 8 Done Done Sent false
14364 4715 Acked Flush 0 Done Done Sent false
14365 4716 Acked Read 8 Done Done Sent false
14366 4717 Acked Flush 0 Done Done Sent false
14367 4718 Acked Write 7 Done Done Sent false
14368 4719 Acked Read 2 Done Done Sent false
14369 4720 Acked Write 4 Done Done Sent false
14370 4721 Acked Write 1 Done Done Sent false
14371 4722 Acked Write 1 Done Done Sent false
14372 4723 Acked Read 8 Done Done Sent false
14373 4724 Acked Write 9 Done Done Sent false
14374 4725 Acked Write 7 Done Done Sent false
14375 4726 Acked Write 3 Done Done Sent false
14376 4727 Acked Write 4 Done Done Sent false
14377 4728 Acked Read 8 Done Done Sent false
14378 4729 Acked Read 8 Done Done Sent false
14379 4730 Acked Read 10 Done Done Sent false
14380 4731 Acked Read 1 Done Done Sent false
14381 4732 Acked Write 6 Done Done Sent false
14382 4733 Acked Read 6 Done Done Sent false
14383 4734 Acked Flush 0 Done Done Sent false
14384 4735 Acked Read 1 Done Done Sent false
14385 4736 Acked Read 10 Done Done Sent false
14386 4737 Acked Flush 0 Done Done Sent false
14387 4738 Acked Read 8 Done Done Sent false
14388 4739 Acked Write 9 Done Done Sent false
14389 4740 Acked Flush 0 Done Done Sent false
14390 4741 Acked Write 7 Done Done Sent false
14391 4742 Acked Flush 0 Done Done Sent false
14392 4743 Acked Write 6 Done Done Sent false
14393 4744 Acked Read 10 Done Done Sent false
14394 4745 Acked Write 7 Done Done Sent false
14395 4746 Acked Write 4 Done Done Sent false
14396 4747 Acked Flush 0 Done Done Sent false
14397 4748 Acked Read 9 Done Done Sent false
14398 4749 Acked Read 2 Done Done Sent false
14399 4750 Acked Read 5 Done Done Sent false
14400 4751 Acked Read 5 Done Done Sent false
14401 4752 Acked Read 5 Done Done Sent false
14402 4753 Acked Read 1 Done Done Sent false
14403 4754 Acked Read 10 Done Done Sent false
14404 4755 Acked Write 7 Done Done Sent false
14405 4756 Acked Write 9 Done Done Sent false
14406 4757 Acked Read 10 Done Done Sent false
14407 4758 Acked Read 1 Done Done Sent false
14408 4759 Acked Read 8 Done Done Sent false
14409 4760 Acked Read 9 Done Done Sent false
14410 4761 Acked Write 2 Done Done Sent false
14411 4762 Acked Read 7 Done Done Sent false
14412 4763 Acked Read 8 Done Done Sent false
14413 4764 Acked Read 8 Done Done Sent false
14414 4765 Acked Read 4 Done Done Sent false
14415 4766 Acked Read 6 Done Done Sent false
14416 4767 Acked Flush 0 Done Done Sent false
14417 4768 Acked Write 9 Done Done Sent false
14418 4769 Acked Write 6 Done Done Sent false
14419 4770 Acked Read 5 Done Done Sent false
14420 4771 Acked Read 9 Done Done Sent false
14421 4772 Acked Read 2 Done Done Sent false
14422 4773 Acked Read 8 Done Done Sent false
14423 4774 Acked Read 9 Done Done Sent false
14424 4775 Acked Flush 0 Done Done Sent false
14425 4776 Acked Write 7 Done Done Sent false
14426 4777 Acked Write 5 Done Done Sent false
14427 4778 Acked Write 9 Done Done Sent false
14428 4779 Acked Read 9 Done Done Sent false
14429 4780 Acked Flush 0 Done Done Sent false
14430 4781 Acked Read 8 Done Done Sent false
14431 4782 Acked Write 9 Done Done Sent false
14432 4783 Acked Read 7 Done Done Sent false
14433 4784 Acked Read 10 Done Done Sent false
14434 4785 Acked Write 3 Done Done Sent false
14435 4786 Acked Read 1 Done Done Sent false
14436 4787 Acked Flush 0 Done Done Sent false
14437 4788 Acked Read 1 Done Done Sent false
14438 4789 Acked Write 4 Done Done Sent false
14439 4790 Acked Write 2 Done Done Sent false
14440 4791 Acked Write 8 Done Done Sent false
14441 4792 Acked Write 2 Done Done Sent false
14442 4793 Acked Read 2 Done Done Sent false
14443 4794 Acked Read 3 Done Done Sent false
14444 4795 Acked Write 2 Done Done Sent false
14445 4796 Acked Write 3 Done Done Sent false
14446 4797 Acked Write 10 Done Done Sent false
14447 4798 Acked Flush 0 Done Done Sent false
14448 4799 Acked Write 10 Done Done Sent false
14449 4800 Acked Write 5 Done Done Sent false
14450 4801 Acked Write 7 Done Done Sent false
14451 4802 Acked Flush 0 Done Done Sent false
14452 4803 Acked Read 4 Done Done Sent false
14453 4804 Acked Write 4 Done Done Sent false
14454 4805 Acked Read 2 Done Done Sent false
14455 4806 Acked Write 10 Done Done Sent false
14456 4807 Acked Read 9 Done Done Sent false
14457 4808 Acked Read 6 Done Done Sent false
14458 4809 Acked Read 2 Done Done Sent false
14459 4810 Acked Read 2 Done Done Sent false
14460 4811 Acked Read 10 Done Done Sent false
14461 4812 Acked Write 3 Done Done Sent false
14462 4813 Acked Write 6 Done Done Sent false
14463 4814 Acked Read 4 Done Done Sent false
14464 4815 Acked Write 2 Done Done Sent false
14465 4816 Acked Read 10 Done Done Sent false
14466 4817 Acked Read 5 Done Done Sent false
14467 4818 Acked Write 3 Done Done Sent false
14468 4819 Acked Flush 0 Done Done Sent false
14469 4820 Acked Read 3 Done Done Sent false
14470 4821 Acked Write 7 Done Done Sent false
14471 4822 Acked Read 8 Done Done Sent false
14472 4823 Acked Read 4 Done Done Sent false
14473 4824 Acked Write 1 Done Done Sent false
14474 4825 Acked Write 5 Done Done Sent false
14475 4826 Acked Write 5 Done Done Sent false
14476 4827 Acked Write 7 Done Done Sent false
14477 4828 Acked Read 7 Done Done Sent false
14478 4829 Acked Read 9 Done Done Sent false
14479 4830 Acked Read 1 Done Done Sent false
14480 4831 Acked Read 9 Done Done Sent false
14481 4832 Acked Read 5 Done Done Sent false
14482 4833 Acked Read 10 Done Done Sent false
14483 4834 Acked Write 3 Done Done Sent false
14484 4835 Acked Write 2 Done Done Sent false
14485 4836 Acked Flush 0 Done Done Sent false
14486 4837 Acked Read 9 Done Done Sent false
14487 4838 Acked Flush 0 Done Done Sent false
14488 4839 Acked Write 4 Done Done Sent false
14489 4840 Acked Write 9 Done Done Sent false
14490 4841 Acked Write 4 Done Done Sent false
14491 4842 Acked Flush 0 Done Done Sent false
14492 4843 Acked Read 4 Done Done Sent false
14493 4844 Acked Write 8 Done Done Sent false
14494 4845 Acked Write 5 Done Done Sent false
14495 4846 Acked Read 8 Done Done Sent false
14496 4847 Acked Write 9 Done Done Sent false
14497 4848 Acked Read 3 Done Done Sent false
14498 4849 Acked Write 5 Done Done Sent false
14499 4850 Acked Write 1 Done Done Sent false
14500 4851 Acked Write 4 Done Done Sent false
14501 4852 Acked Write 8 Done Done Sent false
14502 4853 Acked Flush 0 Done Done Sent false
14503 4854 Acked Read 4 Done Done Sent false
14504 4855 Acked Read 2 Done Done Sent false
14505 4856 Acked Write 5 Done Done Sent false
14506 4857 Acked Write 10 Done Done Sent false
14507 4858 Acked Write 7 Done Done Sent false
14508 4859 Acked Write 1 Done Done Sent false
14509 4860 Acked Read 4 Done Done Sent false
14510 4861 Acked Write 9 Done Done Sent false
14511 4862 Acked Write 4 Done Done Sent false
14512 4863 Acked Write 3 Done Done Sent false
14513 4864 Acked Flush 0 Done Done Sent false
14514 4865 Acked Flush 0 Done Done Sent false
14515 4866 Acked Write 7 Done Done Sent false
14516 4867 Acked Write 6 Done Done Sent false
14517 4868 Acked Write 7 Done Done Sent false
14518 4869 Acked Write 5 Done Done Sent false
14519 4870 Acked Read 4 Done Done Sent false
14520 4871 Acked Write 1 Done Done Sent false
14521 4872 Acked Read 6 Done Done Sent false
14522 4873 Acked Write 3 Done Done Sent false
14523 4874 Acked Write 4 Done Done Sent false
14524 4875 Acked Write 6 Done Done Sent false
14525 4876 Acked Write 10 Done Done Sent false
14526 4877 Acked Read 7 Done Done Sent false
14527 4878 Acked Write 4 Done Done Sent false
14528 4879 Acked Write 3 Done Done Sent false
14529 4880 Acked Read 6 Done Done Sent false
14530 4881 Acked Write 4 Done Done Sent false
14531 4882 Acked Read 10 Done Done Sent false
14532 4883 Acked Read 6 Done Done Sent false
14533 4884 Acked Read 7 Done Done Sent false
14534 4885 Acked Write 6 Done Done Sent false
14535 4886 Acked Read 2 Done Done Sent false
14536 4887 Acked Read 6 Done Done Sent false
14537 4888 Acked Read 3 Done Done Sent false
14538 4889 Acked Write 6 Done Done Sent false
14539 4890 Acked Write 1 Done Done Sent false
14540 4891 Acked Read 3 Done Done Sent false
14541 4892 Acked Write 1 Done Done Sent false
14542 4893 Acked Write 4 Done Done Sent false
14543 4894 Acked Flush 0 Done Done Sent false
14544 4895 Acked Read 3 Done Done Sent false
14545 4896 Acked Read 4 Done Done Sent false
14546 4897 Acked Write 1 Done Done Sent false
14547 4898 Acked Read 10 Done Done Sent false
14548 4899 Acked Read 6 Done Done Sent false
14549 4900 Acked Read 1 Done Done Sent false
14550 4901 Acked Read 9 Done Done Sent false
14551 4902 Acked Read 10 Done Done Sent false
14552 4903 Acked Read 8 Done Done Sent false
14553 4904 Acked Read 7 Done Done Sent false
14554 4905 Acked Write 2 Done Done Sent false
14555 4906 Acked Read 5 Done Done Sent false
14556 4907 Acked Write 5 Done Done Sent false
14557 4908 Acked Read 7 Done Done Sent false
14558 4909 Acked Flush 0 Done Done Sent false
14559 4910 Acked Flush 0 Done Done Sent false
14560 4911 Acked Read 7 Done Done Sent false
14561 4912 Acked Read 7 Done Done Sent false
14562 4913 Acked Write 4 Done Done Sent false
14563 4914 Acked Read 8 Done Done Sent false
14564 4915 Acked Write 1 Done Done Sent false
14565 4916 Acked Write 8 Done Done Sent false
14566 4917 Acked Read 5 Done Done Sent false
14567 4918 Acked Write 9 Done Done Sent false
14568 4919 Acked Flush 0 Done Done Sent false
14569 4920 Acked Flush 0 Done Done Sent false
14570 4921 Acked Write 4 Done Done Sent false
14571 4922 Acked Read 10 Done Done Sent false
14572 4923 Acked Read 9 Done Done Sent false
14573 STATES DS:0 DS:1 DS:2 TOTAL
14574 Sent 0 0 301 301
14575 Done 301 301 0 602
14576 Skipped 0 0 0 0
14577 Error 0 0 0 0
14578 Last Flush: Some(JobId(4920)) Some(JobId(4920)) Some(JobId(4622))
14579 Downstairs last five completed: 4622 4621 4620 4619 4618
14580 Upstairs last five completed: 4923 4922 4921 4920 4919
14581 CLIENT: Up:0 ds:301 act:2
145822025-02-25T17:21:25.763ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
145832025-02-25T17:21:26.987ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
145842025-02-25T17:21:27.582ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
145852025-02-25T17:21:28.583ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
14586 [8830] got start action
14587 [8830] I am going to restart
14588 Starting downstairs at port 8830
14589 Make output file at "/tmp/dsc/downstairs-8830.txt"
14590 [8830][2] reports Starting
14591 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1221)
14592 [8830][2] reports Running
14593 ----------------------------------------------------------------
14594 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14595 States: Active Active Offline
14596 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14597 4623 Acked Write 6 Done Done Sent false
14598 4624 Acked Write 3 Done Done Sent false
14599 4625 Acked Read 10 Done Done Sent false
14600 4626 Acked Flush 0 Done Done Sent false
14601 4627 Acked Write 7 Done Done Sent false
14602 4628 Acked Read 3 Done Done Sent false
14603 4629 Acked Write 3 Done Done Sent false
14604 4630 Acked Read 2 Done Done Sent false
14605 4631 Acked Read 1 Done Done Sent false
14606 4632 Acked Write 8 Done Done Sent false
14607 4633 Acked Read 7 Done Done Sent false
14608 4634 Acked Read 6 Done Done Sent false
14609 4635 Acked Flush 0 Done Done Sent false
14610 4636 Acked Read 3 Done Done Sent false
14611 4637 Acked Write 3 Done Done Sent false
14612 4638 Acked Read 4 Done Done Sent false
14613 4639 Acked Read 6 Done Done Sent false
14614 4640 Acked Read 2 Done Done Sent false
14615 4641 Acked Write 4 Done Done Sent false
14616 4642 Acked Write 4 Done Done Sent false
14617 4643 Acked Read 4 Done Done Sent false
14618 4644 Acked Read 8 Done Done Sent false
14619 4645 Acked Write 5 Done Done Sent false
14620 4646 Acked Read 10 Done Done Sent false
14621 4647 Acked Flush 0 Done Done Sent false
14622 4648 Acked Write 5 Done Done Sent false
14623 4649 Acked Read 4 Done Done Sent false
14624 4650 Acked Read 2 Done Done Sent false
14625 4651 Acked Read 2 Done Done Sent false
14626 4652 Acked Read 5 Done Done Sent false
14627 4653 Acked Flush 0 Done Done Sent false
14628 4654 Acked Read 4 Done Done Sent false
14629 4655 Acked Write 9 Done Done Sent false
14630 4656 Acked Write 2 Done Done Sent false
14631 4657 Acked Read 3 Done Done Sent false
14632 4658 Acked Flush 0 Done Done Sent false
14633 4659 Acked Read 9 Done Done Sent false
14634 4660 Acked Write 6 Done Done Sent false
14635 4661 Acked Read 5 Done Done Sent false
14636 4662 Acked Write 1 Done Done Sent false
14637 4663 Acked Read 10 Done Done Sent false
14638 4664 Acked Read 6 Done Done Sent false
14639 4665 Acked Read 10 Done Done Sent false
14640 4666 Acked Read 7 Done Done Sent false
14641 4667 Acked Flush 0 Done Done Sent false
14642 4668 Acked Flush 0 Done Done Sent false
14643 4669 Acked Read 9 Done Done Sent false
14644 4670 Acked Read 3 Done Done Sent false
14645 4671 Acked Write 9 Done Done Sent false
14646 4672 Acked Read 2 Done Done Sent false
14647 4673 Acked Read 2 Done Done Sent false
14648 4674 Acked Read 8 Done Done Sent false
14649 4675 Acked Read 5 Done Done Sent false
14650 4676 Acked Read 5 Done Done Sent false
14651 4677 Acked Write 4 Done Done Sent false
14652 4678 Acked Read 8 Done Done Sent false
14653 4679 Acked Read 8 Done Done Sent false
14654 4680 Acked Write 7 Done Done Sent false
14655 4681 Acked Read 7 Done Done Sent false
14656 4682 Acked Read 4 Done Done Sent false
14657 4683 Acked Read 7 Done Done Sent false
14658 4684 Acked Flush 0 Done Done Sent false
14659 4685 Acked Read 5 Done Done Sent false
14660 4686 Acked Read 10 Done Done Sent false
14661 4687 Acked Read 5 Done Done Sent false
14662 4688 Acked Write 8 Done Done Sent false
14663 4689 Acked Write 6 Done Done Sent false
14664 4690 Acked Write 8 Done Done Sent false
14665 4691 Acked Read 10 Done Done Sent false
14666 4692 Acked Read 2 Done Done Sent false
14667 4693 Acked Read 4 Done Done Sent false
14668 4694 Acked Flush 0 Done Done Sent false
14669 4695 Acked Read 6 Done Done Sent false
14670 4696 Acked Read 7 Done Done Sent false
14671 4697 Acked Read 2 Done Done Sent false
14672 4698 Acked Flush 0 Done Done Sent false
14673 4699 Acked Write 6 Done Done Sent false
14674 4700 Acked Write 6 Done Done Sent false
14675 4701 Acked Write 5 Done Done Sent false
14676 4702 Acked Write 8 Done Done Sent false
14677 4703 Acked Read 9 Done Done Sent false
14678 4704 Acked Write 8 Done Done Sent false
14679 4705 Acked Read 2 Done Done Sent false
14680 4706 Acked Read 7 Done Done Sent false
14681 4707 Acked Read 1 Done Done Sent false
14682 4708 Acked Write 7 Done Done Sent false
14683 4709 Acked Read 6 Done Done Sent false
14684 4710 Acked Write 10 Done Done Sent false
14685 4711 Acked Write 5 Done Done Sent false
14686 4712 Acked Write 4 Done Done Sent false
14687 4713 Acked Flush 0 Done Done Sent false
14688 4714 Acked Write 8 Done Done Sent false
14689 4715 Acked Flush 0 Done Done Sent false
14690 4716 Acked Read 8 Done Done Sent false
14691 4717 Acked Flush 0 Done Done Sent false
14692 4718 Acked Write 7 Done Done Sent false
14693 4719 Acked Read 2 Done Done Sent false
14694 4720 Acked Write 4 Done Done Sent false
14695 4721 Acked Write 1 Done Done Sent false
14696 4722 Acked Write 1 Done Done Sent false
14697 4723 Acked Read 8 Done Done Sent false
14698 4724 Acked Write 9 Done Done Sent false
14699 4725 Acked Write 7 Done Done Sent false
14700 4726 Acked Write 3 Done Done Sent false
14701 4727 Acked Write 4 Done Done Sent false
14702 4728 Acked Read 8 Done Done Sent false
14703 4729 Acked Read 8 Done Done Sent false
14704 4730 Acked Read 10 Done Done Sent false
14705 4731 Acked Read 1 Done Done Sent false
14706 4732 Acked Write 6 Done Done Sent false
14707 4733 Acked Read 6 Done Done Sent false
14708 4734 Acked Flush 0 Done Done Sent false
14709 4735 Acked Read 1 Done Done Sent false
14710 4736 Acked Read 10 Done Done Sent false
14711 4737 Acked Flush 0 Done Done Sent false
14712 4738 Acked Read 8 Done Done Sent false
14713 4739 Acked Write 9 Done Done Sent false
14714 4740 Acked Flush 0 Done Done Sent false
14715 4741 Acked Write 7 Done Done Sent false
14716 4742 Acked Flush 0 Done Done Sent false
14717 4743 Acked Write 6 Done Done Sent false
14718 4744 Acked Read 10 Done Done Sent false
14719 4745 Acked Write 7 Done Done Sent false
14720 4746 Acked Write 4 Done Done Sent false
14721 4747 Acked Flush 0 Done Done Sent false
14722 4748 Acked Read 9 Done Done Sent false
14723 4749 Acked Read 2 Done Done Sent false
14724 4750 Acked Read 5 Done Done Sent false
14725 4751 Acked Read 5 Done Done Sent false
14726 4752 Acked Read 5 Done Done Sent false
14727 4753 Acked Read 1 Done Done Sent false
14728 4754 Acked Read 10 Done Done Sent false
14729 4755 Acked Write 7 Done Done Sent false
14730 4756 Acked Write 9 Done Done Sent false
14731 4757 Acked Read 10 Done Done Sent false
14732 4758 Acked Read 1 Done Done Sent false
14733 4759 Acked Read 8 Done Done Sent false
14734 4760 Acked Read 9 Done Done Sent false
14735 4761 Acked Write 2 Done Done Sent false
14736 4762 Acked Read 7 Done Done Sent false
14737 4763 Acked Read 8 Done Done Sent false
14738 4764 Acked Read 8 Done Done Sent false
14739 4765 Acked Read 4 Done Done Sent false
14740 4766 Acked Read 6 Done Done Sent false
14741 4767 Acked Flush 0 Done Done Sent false
14742 4768 Acked Write 9 Done Done Sent false
14743 4769 Acked Write 6 Done Done Sent false
14744 4770 Acked Read 5 Done Done Sent false
14745 4771 Acked Read 9 Done Done Sent false
14746 4772 Acked Read 2 Done Done Sent false
14747 4773 Acked Read 8 Done Done Sent false
14748 4774 Acked Read 9 Done Done Sent false
14749 4775 Acked Flush 0 Done Done Sent false
14750 4776 Acked Write 7 Done Done Sent false
14751 4777 Acked Write 5 Done Done Sent false
14752 4778 Acked Write 9 Done Done Sent false
14753 4779 Acked Read 9 Done Done Sent false
14754 4780 Acked Flush 0 Done Done Sent false
14755 4781 Acked Read 8 Done Done Sent false
14756 4782 Acked Write 9 Done Done Sent false
14757 4783 Acked Read 7 Done Done Sent false
14758 4784 Acked Read 10 Done Done Sent false
14759 4785 Acked Write 3 Done Done Sent false
14760 4786 Acked Read 1 Done Done Sent false
14761 4787 Acked Flush 0 Done Done Sent false
14762 4788 Acked Read 1 Done Done Sent false
14763 4789 Acked Write 4 Done Done Sent false
14764 4790 Acked Write 2 Done Done Sent false
14765 4791 Acked Write 8 Done Done Sent false
14766 4792 Acked Write 2 Done Done Sent false
14767 4793 Acked Read 2 Done Done Sent false
14768 4794 Acked Read 3 Done Done Sent false
14769 4795 Acked Write 2 Done Done Sent false
14770 4796 Acked Write 3 Done Done Sent false
14771 4797 Acked Write 10 Done Done Sent false
14772 4798 Acked Flush 0 Done Done Sent false
14773 4799 Acked Write 10 Done Done Sent false
14774 4800 Acked Write 5 Done Done Sent false
14775 4801 Acked Write 7 Done Done Sent false
14776 4802 Acked Flush 0 Done Done Sent false
14777 4803 Acked Read 4 Done Done Sent false
14778 4804 Acked Write 4 Done Done Sent false
14779 4805 Acked Read 2 Done Done Sent false
14780 4806 Acked Write 10 Done Done Sent false
14781 4807 Acked Read 9 Done Done Sent false
14782 4808 Acked Read 6 Done Done Sent false
14783 4809 Acked Read 2 Done Done Sent false
14784 4810 Acked Read 2 Done Done Sent false
14785 4811 Acked Read 10 Done Done Sent false
14786 4812 Acked Write 3 Done Done Sent false
14787 4813 Acked Write 6 Done Done Sent false
14788 4814 Acked Read 4 Done Done Sent false
14789 4815 Acked Write 2 Done Done Sent false
14790 4816 Acked Read 10 Done Done Sent false
14791 4817 Acked Read 5 Done Done Sent false
14792 4818 Acked Write 3 Done Done Sent false
14793 4819 Acked Flush 0 Done Done Sent false
14794 4820 Acked Read 3 Done Done Sent false
14795 4821 Acked Write 7 Done Done Sent false
14796 4822 Acked Read 8 Done Done Sent false
14797 4823 Acked Read 4 Done Done Sent false
14798 4824 Acked Write 1 Done Done Sent false
14799 4825 Acked Write 5 Done Done Sent false
14800 4826 Acked Write 5 Done Done Sent false
14801 4827 Acked Write 7 Done Done Sent false
14802 4828 Acked Read 7 Done Done Sent false
14803 4829 Acked Read 9 Done Done Sent false
14804 4830 Acked Read 1 Done Done Sent false
14805 4831 Acked Read 9 Done Done Sent false
14806 4832 Acked Read 5 Done Done Sent false
14807 4833 Acked Read 10 Done Done Sent false
14808 4834 Acked Write 3 Done Done Sent false
14809 4835 Acked Write 2 Done Done Sent false
14810 4836 Acked Flush 0 Done Done Sent false
14811 4837 Acked Read 9 Done Done Sent false
14812 4838 Acked Flush 0 Done Done Sent false
14813 4839 Acked Write 4 Done Done Sent false
14814 4840 Acked Write 9 Done Done Sent false
14815 4841 Acked Write 4 Done Done Sent false
14816 4842 Acked Flush 0 Done Done Sent false
14817 4843 Acked Read 4 Done Done Sent false
14818 4844 Acked Write 8 Done Done Sent false
14819 4845 Acked Write 5 Done Done Sent false
14820 4846 Acked Read 8 Done Done Sent false
14821 4847 Acked Write 9 Done Done Sent false
14822 4848 Acked Read 3 Done Done Sent false
14823 4849 Acked Write 5 Done Done Sent false
14824 4850 Acked Write 1 Done Done Sent false
14825 4851 Acked Write 4 Done Done Sent false
14826 4852 Acked Write 8 Done Done Sent false
14827 4853 Acked Flush 0 Done Done Sent false
14828 4854 Acked Read 4 Done Done Sent false
14829 4855 Acked Read 2 Done Done Sent false
14830 4856 Acked Write 5 Done Done Sent false
14831 4857 Acked Write 10 Done Done Sent false
14832 4858 Acked Write 7 Done Done Sent false
14833 4859 Acked Write 1 Done Done Sent false
14834 4860 Acked Read 4 Done Done Sent false
14835 4861 Acked Write 9 Done Done Sent false
14836 4862 Acked Write 4 Done Done Sent false
14837 4863 Acked Write 3 Done Done Sent false
14838 4864 Acked Flush 0 Done Done Sent false
14839 4865 Acked Flush 0 Done Done Sent false
14840 4866 Acked Write 7 Done Done Sent false
14841 4867 Acked Write 6 Done Done Sent false
14842 4868 Acked Write 7 Done Done Sent false
14843 4869 Acked Write 5 Done Done Sent false
14844 4870 Acked Read 4 Done Done Sent false
14845 4871 Acked Write 1 Done Done Sent false
14846 4872 Acked Read 6 Done Done Sent false
14847 4873 Acked Write 3 Done Done Sent false
14848 4874 Acked Write 4 Done Done Sent false
14849 4875 Acked Write 6 Done Done Sent false
14850 4876 Acked Write 10 Done Done Sent false
14851 4877 Acked Read 7 Done Done Sent false
14852 4878 Acked Write 4 Done Done Sent false
14853 4879 Acked Write 3 Done Done Sent false
14854 4880 Acked Read 6 Done Done Sent false
14855 4881 Acked Write 4 Done Done Sent false
14856 4882 Acked Read 10 Done Done Sent false
14857 4883 Acked Read 6 Done Done Sent false
14858 4884 Acked Read 7 Done Done Sent false
14859 4885 Acked Write 6 Done Done Sent false
14860 4886 Acked Read 2 Done Done Sent false
14861 4887 Acked Read 6 Done Done Sent false
14862 4888 Acked Read 3 Done Done Sent false
14863 4889 Acked Write 6 Done Done Sent false
14864 4890 Acked Write 1 Done Done Sent false
14865 4891 Acked Read 3 Done Done Sent false
14866 4892 Acked Write 1 Done Done Sent false
14867 4893 Acked Write 4 Done Done Sent false
14868 4894 Acked Flush 0 Done Done Sent false
14869 4895 Acked Read 3 Done Done Sent false
14870 4896 Acked Read 4 Done Done Sent false
14871 4897 Acked Write 1 Done Done Sent false
14872 4898 Acked Read 10 Done Done Sent false
14873 4899 Acked Read 6 Done Done Sent false
14874 4900 Acked Read 1 Done Done Sent false
14875 4901 Acked Read 9 Done Done Sent false
14876 4902 Acked Read 10 Done Done Sent false
14877 4903 Acked Read 8 Done Done Sent false
14878 4904 Acked Read 7 Done Done Sent false
14879 4905 Acked Write 2 Done Done Sent false
14880 4906 Acked Read 5 Done Done Sent false
14881 4907 Acked Write 5 Done Done Sent false
14882 4908 Acked Read 7 Done Done Sent false
14883 4909 Acked Flush 0 Done Done Sent false
14884 4910 Acked Flush 0 Done Done Sent false
14885 4911 Acked Read 7 Done Done Sent false
14886 4912 Acked Read 7 Done Done Sent false
14887 4913 Acked Write 4 Done Done Sent false
14888 4914 Acked Read 8 Done Done Sent false
14889 4915 Acked Write 1 Done Done Sent false
14890 4916 Acked Write 8 Done Done Sent false
14891 4917 Acked Read 5 Done Done Sent false
14892 4918 Acked Write 9 Done Done Sent false
14893 4919 Acked Flush 0 Done Done Sent false
14894 4920 Acked Flush 0 Done Done Sent false
14895 4921 Acked Write 4 Done Done Sent false
14896 4922 Acked Read 10 Done Done Sent false
14897 4923 Acked Read 9 Done Done Sent false
14898 STATES DS:0 DS:1 DS:2 TOTAL
14899 Sent 0 0 301 301
14900 Done 301 301 0 602
14901 Skipped 0 0 0 0
14902 Error 0 0 0 0
14903 Last Flush: Some(JobId(4920)) Some(JobId(4920)) Some(JobId(4622))
14904 Downstairs last five completed: 4622 4621 4620 4619 4618
14905 Upstairs last five completed: 4923 4922 4921 4920 4919
14906 CLIENT: Up:0 ds:301 act:2
149072025-02-25T17:21:30.587ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
149082025-02-25T17:21:31.006ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
149092025-02-25T17:21:31.006ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44897) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
149102025-02-25T17:21:31.007ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
149112025-02-25T17:21:31.008ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
149122025-02-25T17:21:31.008ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
149132025-02-25T17:21:31.008ZINFOcrucible: send last flush ID to this DS: Some(JobId(4622)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
149142025-02-25T17:21:31.008ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
149152025-02-25T17:21:31.008ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4622)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
149162025-02-25T17:21:31.008ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4622)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
14917 ----------------------------------------------------------------
14918 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14919 Downstairs last five completed: 4924 4923 4922 4921 4920
14920 Upstairs last five completed: 4924 4923 4922 4921 4920
14921 CLIENT: Up:0 ds:0 act:3
14922 Replay: All jobs finished
14923 Main task has work to do, go find it
14924 got dsc Stop(0)
14925 stop 0
14926 Feb 25 17:21:33.266 INFO [8810] Got stop action so:false kr:false
14927 request completed, latency_us: 363, response_code: 204, uri: /stop/cid/0, method: POST, req_id: f4733348-58fe-48fd-a66d-6a6849272c28, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
149282025-02-25T17:21:33.267ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
149292025-02-25T17:21:33.267ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
149302025-02-25T17:21:33.267ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
14931 Feb 25 17:21:33.267 {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T17:21:33.26778255Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
14932 INFO request completed, latency_us: 154, response_code: 200, uri: /state/cid/0, method: GET, req_id: 67405207-6095-4e21-a46d-d7c211a4ffff, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
14933 [8810] Exited with: ExitStatus(unix_wait_status(9))
14934 [8810][0] reports Exit
14935 Feb 25 17:21:37.270 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/0, method: GET, req_id: 859bd4d3-b59f-438c-98fe-734b2544faa8, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
14936 001/300 Write block 516 len 24576 data: 3 2 3 4 4 4
14937 002/300 Read block 2352 len 32768
14938 003/300 Read block 518 len 16384
14939 004/300 Write block 1049 len 12288 data: 3 3 3
14940 005/300 Write block 1505 len 24576 data: 4 4 4 4 4 4
14941 006/300 Write block 5 len 12288 data: 2 2 2
14942 007/300 Write block 4032 len 8192 data: 4 3
14943 008/300 Read block 2267 len 20480
14944 009/300 Read block 306 len 32768
14945 010/300 Read block 2512 len 8192
14946 011/300 Read block 4599 len 40960
14947 012/300 Read block 1647 len 40960
14948 013/300 Read block 4499 len 36864
14949 014/300 Read block 2789 len 20480
14950 015/300 Write block 4588 len 24576 data: 4 3 3 4 4 3
14951 016/300 Read block 1088 len 4096
14952 017/300 Write block 4517 len 16384 data: 6 6 7 7
14953 018/300 Write block 1387 len 36864 data: 5 6 5 5 4 3 3 4 4
14954 019/300 Write block 1496 len 40960 data: 4 4 2 2 2 3 3 4 4 5
14955 020/300 Read block 1385 len 20480
14956 021/300 Read block 3075 len 20480
14957 022/300 Read block 3640 len 12288
14958 023/300 Read block 1020 len 24576
14959 024/300 Flush
14960 025/300 Read block 3101 len 24576
14961 026/300 Flush
14962 027/300 Read block 3925 len 28672
14963 028/300 Write block 4089 len 12288 data: 3 3 3
14964 029/300 Write block 532 len 24576 data: 3 4 4 4 4 3
14965 030/300 Write block 4721 len 8192 data: 4 4
14966 031/300 Read block 2488 len 36864
14967 032/300 Read block 3233 len 4096
14968 033/300 Write block 1821 len 16384 data: 2 2 3 3
14969 034/300 Write block 628 len 12288 data: 8 9 9
14970 035/300 Read block 2644 len 16384
14971 036/300 Read block 503 len 36864
14972 037/300 Read block 4932 len 4096
14973 038/300 Read block 4467 len 20480
14974 039/300 Write block 3303 len 28672 data: 6 6 5 4 4 4 4
14975 040/300 Write block 1319 len 28672 data: 3 4 4 4 4 4 3
14976 041/300 Read block 1517 len 16384
14977 042/300 Read block 2612 len 40960
14978 043/300 Write block 175 len 16384 data: 2 4 5 4
14979 044/300 Flush
14980 045/300 Write block 853 len 4096 data: 3
14981 046/300 Write block 3311 len 40960 data: 3 3 4 4 4 3 2 2 2 2
14982 047/300 Read block 1570 len 12288
14983 048/300 Write block 324 len 20480 data: 3 3 3 3 3
14984 049/300 Read block 3796 len 32768
14985 050/300 Read block 3999 len 8192
14986 051/300 Read block 4819 len 40960
14987 052/300 Read block 898 len 4096
14988 053/300 Read block 2532 len 28672
14989 054/300 Read block 1726 len 28672
14990 055/300 Read block 4166 len 36864
14991 056/300 Read block 941 len 16384
14992 057/300 Read block 2201 len 24576
14993 058/300 Read block 3329 len 4096
14994 059/300 Read block 4922 len 16384
14995 060/300 Read block 1741 len 36864
14996 061/300 Read block 4714 len 32768
14997 062/300 Read block 4443 len 12288
14998 063/300 Read block 1762 len 20480
14999 064/300 Write block 4379 len 32768 data: 3 3 3 2 2 3 3 4
15000 065/300 Read block 1649 len 12288
15001 066/300 Read block 1225 len 8192
15002 067/300 Read block 2456 len 32768
15003 068/300 Write block 813 len 16384 data: 2 2 2 2
15004 069/300 Read block 522 len 36864
15005 070/300 Write block 3686 len 16384 data: 5 5 5 5
15006 071/300 Write block 2715 len 36864 data: 4 5 5 4 4 4 2 2 2
15007 072/300 Write block 982 len 20480 data: 3 5 4 4 4
15008 073/300 Write block 1858 len 4096 data: 2
15009 074/300 Write block 3124 len 32768 data: 4 3 5 4 3 3 2 2
15010 075/300 Read block 3058 len 8192
15011 076/300 Read block 4717 len 40960
15012 077/300 Read block 56 len 28672
15013 078/300 Read block 4826 len 8192
15014 079/300 Write block 901 len 32768 data: 3 3 3 3 2 4 4 3
15015 080/300 Read block 2313 len 4096
15016 081/300 Write block 2246 len 4096 data: 2
15017 082/300 Write block 4498 len 24576 data: 3 3 3 2 3 2
15018 083/300 Read block 271 len 28672
15019 084/300 Read block 2174 len 28672
15020 085/300 Read block 4984 len 28672
15021 086/300 Read block 1949 len 12288
15022 087/300 Write block 1106 len 28672 data: 8 6 6 5 6 7 4
15023 088/300 Read block 4185 len 36864
15024 089/300 Write block 3355 len 40960 data: 6 6 5 5 4 5 3 3 3 2
15025 090/300 Read block 719 len 32768
15026 091/300 Write block 653 len 12288 data: 3 3 3
15027 092/300 Read block 3161 len 36864
15028 093/300 Read block 2452 len 24576
15029 094/300 Write block 429 len 20480 data: 2 2 2 3 3
15030 095/300 Write block 438 len 16384 data: 4 4 4 5
15031 096/300 Read block 2248 len 4096
15032 097/300 Read block 3962 len 24576
15033 098/300 Read block 2049 len 20480
15034 099/300 Flush
15035 100/300 Read block 3124 len 16384
15036 101/300 Write block 3670 len 40960 data: 6 4 3 3 4 4 4 3 3 3
15037 102/300 Read block 3822 len 32768
15038 103/300 Read block 2110 len 4096
15039 104/300 Write block 298 len 24576 data: 2 3 2 2 2 2
15040 105/300 Write block 2174 len 28672 data: 3 2 3 4 4 3 2
15041 106/300 Write block 942 len 16384 data: 3 3 4 3
15042 107/300 Flush
15043 108/300 Read block 1283 len 28672
15044 109/300 Write block 2581 len 16384 data: 5 5 4 4
15045 110/300 Write block 3537 len 20480 data: 3 5 5 6 5
15046 111/300 Write block 3163 len 28672 data: 5 3 3 3 3 3 3
15047 112/300 Write block 1197 len 16384 data: 5 5 3 3
15048 113/300 Write block 2186 len 4096 data: 2
15049 114/300 Read block 726 len 32768
15050 115/300 Write block 125 len 12288 data: 4 4 4
15051 116/300 Write block 412 len 24576 data: 4 4 5 6 5 4
15052 117/300 Write block 4610 len 4096 data: 2
15053 118/300 Read block 932 len 24576
15054 119/300 Write block 858 len 36864 data: 3 3 2 3 3 3 4 3 4
15055 120/300 Write block 10 len 28672 data: 3 3 4 5 5 4 5
15056 121/300 Flush
15057 122/300 Read block 163 len 36864
15058 123/300 Read block 6 len 20480
15059 124/300 Read block 2933 len 4096
15060 125/300 Write block 4714 len 36864 data: 3 3 3 3 4 3 5 5 5
15061 126/300 Write block 930 len 32768 data: 4 4 4 4 4 2 2 2
15062 127/300 Write block 1118 len 8192 data: 2 3
15063 128/300 Read block 4675 len 36864
15064 129/300 Read block 1387 len 32768
15065 130/300 Write block 4203 len 8192 data: 5 5
15066 131/300 Flush
15067 132/300 Read block 2727 len 40960
15068 133/300 Read block 4821 len 36864
15069 134/300 Write block 2077 len 28672 data: 3 2 3 4 4 5 5
15070 135/300 Read block 2666 len 20480
15071 136/300 Write block 2293 len 16384 data: 3 3 3 3
15072 137/300 Read block 1204 len 24576
15073 138/300 Flush
15074 139/300 Write block 2379 len 40960 data: 2 2 2 3 3 2 2 3 3 4
15075 140/300 Read block 3258 len 32768
15076 141/300 Write block 1085 len 32768 data: 3 3 4 5 3 5 5 6
15077 142/300 Write block 4651 len 20480 data: 2 2 2 2 2
15078 143/300 Write block 3715 len 20480 data: 5 5 6 5 5
15079 144/300 Read block 80 len 4096
15080 145/300 Write block 348 len 8192 data: 2 3
15081 146/300 Read block 264 len 36864
15082 147/300 Write block 141 len 12288 data: 2 2 3
15083 148/300 Read block 50 len 32768
15084 149/300 Write block 720 len 16384 data: 3 3 2 3
15085 150/300 Write block 512 len 32768 data: 4 5 4 4 4 3 4 5
15086 151/300 Flush
15087 152/300 Read block 3961 len 32768
15088 153/300 Read block 503 len 36864
15089 154/300 Read block 669 len 36864
15090 155/300 Read block 1776 len 36864
15091 156/300 Write block 3235 len 28672 data: 2 2 2 2 2 3 3
15092 157/300 Write block 3090 len 16384 data: 4 5 4 4
15093 158/300 Write block 2207 len 4096 data: 3
15094 159/300 Read block 2839 len 8192
15095 160/300 Read block 711 len 32768
15096 161/300 Read block 2949 len 32768
15097 162/300 Write block 252 len 4096 data: 4
15098 163/300 Write block 4700 len 24576 data: 5 6 5 5 5 4
15099 164/300 Read block 3921 len 20480
15100 165/300 Write block 1469 len 20480 data: 5 5 4 4 4
15101 166/300 Write block 4423 len 12288 data: 4 3 3
15102 167/300 Read block 1445 len 16384
15103 168/300 Write block 3244 len 16384 data: 3 3 3 3
15104 169/300 Flush
15105 170/300 Read block 2470 len 40960
15106 171/300 Read block 4251 len 32768
15107 172/300 Write block 197 len 28672 data: 4 5 4 4 4 5 5
15108 173/300 Read block 2529 len 36864
15109 174/300 Write block 2966 len 16384 data: 7 7 7 7
15110 175/300 Write block 4429 len 20480 data: 3 3 3 3 3
15111 176/300 Read block 4787 len 4096
15112 177/300 Write block 919 len 4096 data: 2
15113 178/300 Read block 3014 len 8192
15114 179/300 Read block 3875 len 24576
15115 180/300 Read block 58 len 20480
15116 181/300 Read block 2713 len 36864
15117 182/300 Read block 589 len 20480
15118 183/300 Write block 4870 len 32768 data: 4 4 3 3 3 3 3 3
15119 184/300 Read block 1057 len 12288
15120 185/300 Write block 1644 len 12288 data: 3 5 7
15121 186/300 Write block 4806 len 36864 data: 4 3 4 5 5 6 8 8 8
15122 187/300 Write block 1041 len 4096 data: 5
15123 188/300 Flush
15124 189/300 Flush
15125 190/300 Read block 2398 len 16384
15126 191/300 Read block 1992 len 32768
15127 192/300 Read block 4059 len 28672
15128 193/300 Write block 2274 len 32768 data: 2 2 2 3 3 4 4 4
15129 194/300 Read block 415 len 28672
15130 195/300 Write block 2121 len 8192 data: 4 4
15131 196/300 Read block 4131 len 16384
15132 197/300 Read block 319 len 12288
15133 198/300 Flush
15134 199/300 Flush
15135 200/300 Flush
15136 201/300 Write block 2554 len 40960 data: 4 3 3 3 4 4 4 3 3 2
15137 202/300 Write block 3221 len 24576 data: 6 6 5 5 5 5
15138 203/300 Flush
15139 204/300 Read block 2474 len 36864
15140 205/300 Write block 1437 len 36864 data: 4 4 4 5 4 5 5 5 5
15141 206/300 Write block 4207 len 32768 data: 5 5 4 3 3 4 3 3
15142 207/300 Read block 2220 len 24576
15143 208/300 Write block 12 len 4096 data: 5
15144 209/300 Read block 4718 len 32768
15145 210/300 Write block 365 len 28672 data: 2 4 4 4 3 3 3
15146 211/300 Read block 3583 len 4096
15147 212/300 Read block 2595 len 24576
15148 213/300 Read block 1775 len 28672
15149 214/300 Flush
15150 215/300 Flush
15151 216/300 Write block 2758 len 40960 data: 2 2 2 2 2 4 4 5 5 6
15152 217/300 Write block 4978 len 32768 data: 2 2 3 3 3 3 2 2
15153 218/300 Read block 448 len 20480
15154 219/300 Read block 592 len 28672
15155 220/300 Read block 4932 len 16384
15156 221/300 Read block 3658 len 36864
15157 222/300 Write block 1874 len 24576 data: 3 3 3 2 2 2
15158 223/300 Read block 3661 len 4096
15159 224/300 Write block 3197 len 16384 data: 3 3 4 3
15160 225/300 Flush
15161 226/300 Read block 4629 len 20480
15162 227/300 Read block 3780 len 20480
15163 228/300 Read block 2137 len 40960
15164 229/300 Read block 2621 len 20480
15165 230/300 Write block 2008 len 16384 data: 4 4 4 4
15166 231/300 Read block 1551 len 32768
15167 232/300 Read block 2850 len 20480
15168 233/300 Read block 3636 len 8192
15169 234/300 Read block 153 len 20480
15170 235/300 Read block 2172 len 8192
15171 236/300 Read block 1114 len 24576
15172 237/300 Write block 912 len 32768 data: 3 3 3 2 3 3 2 3
15173 238/300 Read block 70 len 24576
15174 239/300 Read block 2430 len 28672
15175 240/300 Write block 3651 len 24576 data: 5 5 6 6 6 6
15176 241/300 Read block 2920 len 20480
15177 242/300 Read block 1805 len 4096
15178 243/300 Flush
15179 244/300 Read block 816 len 12288
15180 245/300 Read block 42 len 36864
15181 246/300 Read block 1440 len 4096
15182 247/300 Read block 4612 len 40960
15183 248/300 Read block 4085 len 4096
15184 249/300 Read block 215 len 32768
15185 250/300 Write block 977 len 20480 data: 3 3 3 3 3
15186 251/300 Write block 981 len 12288 data: 4 4 6
15187 252/300 Read block 4081 len 16384
15188 253/300 Write block 3235 len 36864 data: 3 3 3 3 3 4 4 3 3
15189 254/300 Read block 1338 len 20480
15190 255/300 Write block 752 len 16384 data: 5 4 4 4
15191 256/300 Read block 1526 len 4096
15192 257/300 Write block 1409 len 8192 data: 3 2
15193 258/300 Write block 4743 len 4096 data: 2
15194 259/300 Read block 4251 len 32768
15195 260/300 Write block 3050 len 40960 data: 5 4 2 4 5 4 4 5 5 5
15196 261/300 Flush
15197 262/300 Read block 4865 len 28672
15198 263/300 Read block 4745 len 36864
15199 264/300 Read block 4107 len 12288
15200 265/300 Write block 4658 len 16384 data: 2 2 2 5
15201 266/300 Read block 892 len 32768
15202 267/300 Write block 3850 len 16384 data: 3 3 4 5
15203 268/300 Read block 2520 len 8192
15204 269/300 Read block 3504 len 36864
15205 270/300 Write block 2797 len 24576 data: 6 7 6 3 2 3
15206 271/300 Read block 1615 len 4096
15207 272/300 Flush
15208 273/300 Write block 3509 len 8192 data: 2 3
15209 274/300 Read block 1955 len 8192
15210 275/300 Flush
15211 276/300 Flush
15212 277/300 Read block 714 len 32768
15213 278/300 Flush
15214 279/300 Write block 156 len 8192 data: 4 5
15215 280/300 Write block 4126 len 16384 data: 3 4 4 4
15216 281/300 Write block 1237 len 28672 data: 6 5 3 3 4 3 3
15217 282/300 Write block 946 len 40960 data: 3 2 2 2 3 4 3 3 3 3
15218 283/300 Read block 874 len 36864
15219 284/300 Read block 2940 len 40960
15220 285/300 Read block 1518 len 40960
15221 286/300 Write block 2552 len 4096 data: 4
15222 287/300 Flush
15223 288/300 Read block 4619 len 36864
15224 289/300 Read block 2992 len 40960
15225 290/300 Write block 1278 len 4096 data: 2
15226 291/300 Read block 613 len 8192
15227 292/300 Read block 4435 len 36864
15228 293/300 Write block 3553 len 36864 data: 7 7 5 5 2 2 3 3 4
15229 294/300 Read block 1711 len 8192
15230 295/300 Flush
15231 296/300 Write block 2781 len 12288 data: 4 4 4
15232 297/300 Write block 324 len 28672 data: 4 4 4 4 4 4 3
15233 298/300 Write block 1264 len 28672 data: 5 5 4 3 3 2 2
15234 299/300 Flush
15235 300/300 Write block 4690 len 8192 data: 2 3
15236 301/300 Write block 3724 len 28672 data: 5 6 6 5 6 6 7
15237 Main task has work to do, go find it
15238 got dsc Start(0)
15239 start 0
15240 Feb 25 17:21:37.527 INFO request completed, latency_us: 221, response_code: 204, uri: /start/cid/0, method: POST, req_id: 4a3388d1-fd1b-4779-a30e-51c811b3d9eb, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
15241 Replay: started 0, returned:Ok(())
15242 ----------------------------------------------------------------
15243 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15244 States: Offline Active Active
15245 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15246 4925 Acked Write 6 Sent Done Done false
15247 4926 Acked Read 8 Sent Done Done false
15248 4927 Acked Read 4 Sent Done Done false
15249 4928 Acked Write 3 Sent Done Done false
15250 4929 Acked Write 6 Sent Done Done false
15251 4930 Acked Write 3 Sent Done Done false
15252 4931 Acked Write 2 Sent Done Done false
15253 4932 Acked Read 5 Sent Done Done false
15254 4933 Acked Read 8 Sent Done Done false
15255 4934 Acked Read 2 Sent Done Done false
15256 4935 Acked Read 10 Sent Done Done false
15257 4936 Acked Read 10 Sent Done Done false
15258 4937 Acked Read 9 Sent Done Done false
15259 4938 Acked Read 5 Sent Done Done false
15260 4939 Acked Write 6 Sent Done Done false
15261 4940 Acked Read 1 Sent Done Done false
15262 4941 Acked Write 4 Sent Done Done false
15263 4942 Acked Write 9 Sent Done Done false
15264 4943 Acked Write 10 Sent Done Done false
15265 4944 Acked Read 5 Sent Done Done false
15266 4945 Acked Read 5 Sent Done Done false
15267 4946 Acked Read 3 Sent Done Done false
15268 4947 Acked Read 6 Sent Done Done false
15269 4948 Acked Flush 0 Sent Done Done false
15270 4949 Acked Read 6 Sent Done Done false
15271 4950 Acked Flush 0 Sent Done Done false
15272 4951 Acked Read 7 Sent Done Done false
15273 4952 Acked Write 3 Sent Done Done false
15274 4953 Acked Write 6 Sent Done Done false
15275 4954 Acked Write 2 Sent Done Done false
15276 4955 Acked Read 9 Sent Done Done false
15277 4956 Acked Read 1 Sent Done Done false
15278 4957 Acked Write 4 Sent Done Done false
15279 4958 Acked Write 3 Sent Done Done false
15280 4959 Acked Read 4 Sent Done Done false
15281 4960 Acked Read 9 Sent Done Done false
15282 4961 Acked Read 1 Sent Done Done false
15283 4962 Acked Read 5 Sent Done Done false
15284 4963 Acked Write 7 Sent Done Done false
15285 4964 Acked Write 7 Sent Done Done false
15286 4965 Acked Read 4 Sent Done Done false
15287 4966 Acked Read 10 Sent Done Done false
15288 4967 Acked Write 4 Sent Done Done false
15289 4968 Acked Flush 0 Sent Done Done false
15290 4969 Acked Write 1 Sent Done Done false
15291 4970 Acked Write 10 Sent Done Done false
15292 4971 Acked Read 3 Sent Done Done false
15293 4972 Acked Write 5 Sent Done Done false
15294 4973 Acked Read 8 Sent Done Done false
15295 4974 Acked Read 2 Sent Done Done false
15296 4975 Acked Read 10 Sent Done Done false
15297 4976 Acked Read 1 Sent Done Done false
15298 4977 Acked Read 7 Sent Done Done false
15299 4978 Acked Read 7 Sent Done Done false
15300 4979 Acked Read 9 Sent Done Done false
15301 4980 Acked Read 4 Sent Done Done false
15302 4981 Acked Read 6 Sent Done Done false
15303 4982 Acked Read 1 Sent Done Done false
15304 4983 Acked Read 4 Sent Done Done false
15305 4984 Acked Read 9 Sent Done Done false
15306 4985 Acked Read 8 Sent Done Done false
15307 4986 Acked Read 3 Sent Done Done false
15308 4987 Acked Read 5 Sent Done Done false
15309 4988 Acked Write 8 Sent Done Done false
15310 4989 Acked Read 3 Sent Done Done false
15311 4990 Acked Read 2 Sent Done Done false
15312 4991 Acked Read 8 Sent Done Done false
15313 4992 Acked Write 4 Sent Done Done false
15314 4993 Acked Read 9 Sent Done Done false
15315 4994 Acked Write 4 Sent Done Done false
15316 4995 Acked Write 9 Sent Done Done false
15317 4996 Acked Write 5 Sent Done Done false
15318 4997 Acked Write 1 Sent Done Done false
15319 4998 Acked Write 8 Sent Done Done false
15320 4999 Acked Read 2 Sent Done Done false
15321 5000 Acked Read 10 Sent Done Done false
15322 5001 Acked Read 7 Sent Done Done false
15323 5002 Acked Read 2 Sent Done Done false
15324 5003 Acked Write 8 Sent Done Done false
15325 5004 Acked Read 1 Sent Done Done false
15326 5005 Acked Write 1 Sent Done Done false
15327 5006 Acked Write 6 Sent Done Done false
15328 5007 Acked Read 7 Sent Done Done false
15329 5008 Acked Read 7 Sent Done Done false
15330 5009 Acked Read 7 Sent Done Done false
15331 5010 Acked Read 3 Sent Done Done false
15332 5011 Acked Write 7 Sent Done Done false
15333 5012 Acked Read 9 Sent Done Done false
15334 5013 Acked Write 10 Sent Done Done false
15335 5014 Acked Read 8 Sent Done Done false
15336 5015 Acked Write 3 Sent Done Done false
15337 5016 Acked Read 9 Sent Done Done false
15338 5017 Acked Read 6 Sent Done Done false
15339 5018 Acked Write 5 Sent Done Done false
15340 5019 Acked Write 4 Sent Done Done false
15341 5020 Acked Read 1 Sent Done Done false
15342 5021 Acked Read 6 Sent Done Done false
15343 5022 Acked Read 5 Sent Done Done false
15344 5023 Acked Flush 0 Sent Done Done false
15345 5024 Acked Read 4 Sent Done Done false
15346 5025 Acked Write 10 Sent Done Done false
15347 5026 Acked Read 8 Sent Done Done false
15348 5027 Acked Read 1 Sent Done Done false
15349 5028 Acked Write 6 Sent Done Done false
15350 5029 Acked Write 7 Sent Done Done false
15351 5030 Acked Write 4 Sent Done Done false
15352 5031 Acked Flush 0 Sent Done Done false
15353 5032 Acked Read 7 Sent Done Done false
15354 5033 Acked Write 4 Sent Done Done false
15355 5034 Acked Write 5 Sent Done Done false
15356 5035 Acked Write 7 Sent Done Done false
15357 5036 Acked Write 4 Sent Done Done false
15358 5037 Acked Write 1 Sent Done Done false
15359 5038 Acked Read 8 Sent Done Done false
15360 5039 Acked Write 3 Sent Done Done false
15361 5040 Acked Write 6 Sent Done Done false
15362 5041 Acked Write 1 Sent Done Done false
15363 5042 Acked Read 6 Sent Done Done false
15364 5043 Acked Write 9 Sent Done Done false
15365 5044 Acked Write 7 Sent Done Done false
15366 5045 Acked Flush 0 Sent Done Done false
15367 5046 Acked Read 9 Sent Done Done false
15368 5047 Acked Read 5 Sent Done Done false
15369 5048 Acked Read 1 Sent Done Done false
15370 5049 Acked Write 9 Sent Done Done false
15371 5050 Acked Write 8 Sent Done Done false
15372 5051 Acked Write 2 Sent Done Done false
15373 5052 Acked Read 9 Sent Done Done false
15374 5053 Acked Read 8 Sent Done Done false
15375 5054 Acked Write 2 Sent Done Done false
15376 5055 Acked Flush 0 Sent Done Done false
15377 5056 Acked Read 10 Sent Done Done false
15378 5057 Acked Read 9 Sent Done Done false
15379 5058 Acked Write 7 Sent Done Done false
15380 5059 Acked Read 5 Sent Done Done false
15381 5060 Acked Write 4 Sent Done Done false
15382 5061 Acked Read 6 Sent Done Done false
15383 5062 Acked Flush 0 Sent Done Done false
15384 5063 Acked Write 10 Sent Done Done false
15385 5064 Acked Read 8 Sent Done Done false
15386 5065 Acked Write 8 Sent Done Done false
15387 5066 Acked Write 5 Sent Done Done false
15388 5067 Acked Write 5 Sent Done Done false
15389 5068 Acked Read 1 Sent Done Done false
15390 5069 Acked Write 2 Sent Done Done false
15391 5070 Acked Read 9 Sent Done Done false
15392 5071 Acked Write 3 Sent Done Done false
15393 5072 Acked Read 8 Sent Done Done false
15394 5073 Acked Write 4 Sent Done Done false
15395 5074 Acked Write 8 Sent Done Done false
15396 5075 Acked Flush 0 Sent Done Done false
15397 5076 Acked Read 8 Sent Done Done false
15398 5077 Acked Read 9 Sent Done Done false
15399 5078 Acked Read 9 Sent Done Done false
15400 5079 Acked Read 9 Sent Done Done false
15401 5080 Acked Write 7 Sent Done Done false
15402 5081 Acked Write 4 Sent Done Done false
15403 5082 Acked Write 1 Sent Done Done false
15404 5083 Acked Read 2 Sent Done Done false
15405 5084 Acked Read 8 Sent Done Done false
15406 5085 Acked Read 8 Sent Done Done false
15407 5086 Acked Write 1 Sent Done Done false
15408 5087 Acked Write 6 Sent Done Done false
15409 5088 Acked Read 5 Sent Done Done false
15410 5089 Acked Write 5 Sent Done Done false
15411 5090 Acked Write 3 Sent Done Done false
15412 5091 Acked Read 4 Sent Done Done false
15413 5092 Acked Write 4 Sent Done Done false
15414 5093 Acked Flush 0 Sent Done Done false
15415 5094 Acked Read 10 Sent Done Done false
15416 5095 Acked Read 8 Sent Done Done false
15417 5096 Acked Write 7 Sent Done Done false
15418 5097 Acked Read 9 Sent Done Done false
15419 5098 Acked Write 4 Sent Done Done false
15420 5099 Acked Write 5 Sent Done Done false
15421 5100 Acked Read 1 Sent Done Done false
15422 5101 Acked Write 1 Sent Done Done false
15423 5102 Acked Read 2 Sent Done Done false
15424 5103 Acked Read 6 Sent Done Done false
15425 5104 Acked Read 5 Sent Done Done false
15426 5105 Acked Read 9 Sent Done Done false
15427 5106 Acked Read 5 Sent Done Done false
15428 5107 Acked Write 8 Sent Done Done false
15429 5108 Acked Read 3 Sent Done Done false
15430 5109 Acked Write 3 Sent Done Done false
15431 5110 Acked Write 9 Sent Done Done false
15432 5111 Acked Write 1 Sent Done Done false
15433 5112 Acked Flush 0 Sent Done Done false
15434 5113 Acked Flush 0 Sent Done Done false
15435 5114 Acked Read 4 Sent Done Done false
15436 5115 Acked Read 8 Sent Done Done false
15437 5116 Acked Read 7 Sent Done Done false
15438 5117 Acked Write 8 Sent Done Done false
15439 5118 Acked Read 7 Sent Done Done false
15440 5119 Acked Write 2 Sent Done Done false
15441 5120 Acked Read 4 Sent Done Done false
15442 5121 Acked Read 3 Sent Done Done false
15443 5122 Acked Flush 0 Sent Done Done false
15444 5123 Acked Flush 0 Sent Done Done false
15445 5124 Acked Flush 0 Sent Done Done false
15446 5125 Acked Write 10 Sent Done Done false
15447 5126 Acked Write 6 Sent Done Done false
15448 5127 Acked Flush 0 Sent Done Done false
15449 5128 Acked Read 9 Sent Done Done false
15450 5129 Acked Write 9 Sent Done Done false
15451 5130 Acked Write 8 Sent Done Done false
15452 5131 Acked Read 6 Sent Done Done false
15453 5132 Acked Write 1 Sent Done Done false
15454 5133 Acked Read 8 Sent Done Done false
15455 5134 Acked Write 7 Sent Done Done false
15456 5135 Acked Read 1 Sent Done Done false
15457 5136 Acked Read 6 Sent Done Done false
15458 5137 Acked Read 7 Sent Done Done false
15459 5138 Acked Flush 0 Sent Done Done false
15460 5139 Acked Flush 0 Sent Done Done false
15461 5140 Acked Write 10 Sent Done Done false
15462 5141 Acked Write 8 Sent Done Done false
15463 5142 Acked Read 5 Sent Done Done false
15464 5143 Acked Read 7 Sent Done Done false
15465 5144 Acked Read 4 Sent Done Done false
15466 5145 Acked Read 9 Sent Done Done false
15467 5146 Acked Write 6 Sent Done Done false
15468 5147 Acked Read 1 Sent Done Done false
15469 5148 Acked Write 4 Sent Done Done false
15470 5149 Acked Flush 0 Sent Done Done false
15471 5150 Acked Read 5 Sent Done Done false
15472 5151 Acked Read 5 Sent Done Done false
15473 5152 Acked Read 10 Sent Done Done false
15474 5153 Acked Read 5 Sent Done Done false
15475 5154 Acked Write 4 Sent Done Done false
15476 5155 Acked Read 8 Sent Done Done false
15477 5156 Acked Read 5 Sent Done Done false
15478 5157 Acked Read 2 Sent Done Done false
15479 5158 Acked Read 5 Sent Done Done false
15480 5159 Acked Read 2 Sent Done Done false
15481 5160 Acked Read 6 Sent Done Done false
15482 5161 Acked Write 8 Sent Done Done false
15483 5162 Acked Read 6 Sent Done Done false
15484 5163 Acked Read 7 Sent Done Done false
15485 5164 Acked Write 6 Sent Done Done false
15486 5165 Acked Read 5 Sent Done Done false
15487 5166 Acked Read 1 Sent Done Done false
15488 5167 Acked Flush 0 Sent Done Done false
15489 5168 Acked Read 3 Sent Done Done false
15490 5169 Acked Read 9 Sent Done Done false
15491 5170 Acked Read 1 Sent Done Done false
15492 5171 Acked Read 10 Sent Done Done false
15493 5172 Acked Read 1 Sent Done Done false
15494 5173 Acked Read 8 Sent Done Done false
15495 5174 Acked Write 5 Sent Done Done false
15496 5175 Acked Write 3 Sent Done Done false
15497 5176 Acked Read 4 Sent Done Done false
15498 5177 Acked Write 9 Sent Done Done false
15499 5178 Acked Read 5 Sent Done Done false
15500 5179 Acked Write 4 Sent Done Done false
15501 5180 Acked Read 1 Sent Done Done false
15502 5181 Acked Write 2 Sent Done Done false
15503 5182 Acked Write 1 Sent Done Done false
15504 5183 Acked Read 8 Sent Done Done false
15505 5184 Acked Write 10 Sent Done Done false
15506 5185 Acked Flush 0 Sent Done Done false
15507 5186 Acked Read 7 Sent Done Done false
15508 5187 Acked Read 9 Sent Done Done false
15509 5188 Acked Read 3 Sent Done Done false
15510 5189 Acked Write 4 Sent Done Done false
15511 5190 Acked Read 8 Sent Done Done false
15512 5191 Acked Write 4 Sent Done Done false
15513 5192 Acked Read 2 Sent Done Done false
15514 5193 Acked Read 9 Sent Done Done false
15515 5194 Acked Write 6 Sent Done Done false
15516 5195 Acked Read 1 Sent Done Done false
15517 5196 Acked Flush 0 Sent Done Done false
15518 5197 Acked Write 2 Sent Done Done false
15519 5198 Acked Read 2 Sent Done Done false
15520 5199 Acked Flush 0 Sent Done Done false
15521 5200 Acked Flush 0 Sent Done Done false
15522 5201 Acked Read 8 Sent Done Done false
15523 5202 Acked Flush 0 Sent Done Done false
15524 5203 Acked Write 2 Sent Done Done false
15525 5204 Acked Write 4 Sent Done Done false
15526 5205 Acked Write 7 Sent Done Done false
15527 5206 Acked Write 10 Sent Done Done false
15528 5207 Acked Read 9 Sent Done Done false
15529 5208 Acked Read 10 Sent Done Done false
15530 5209 Acked Read 10 Sent Done Done false
15531 5210 Acked Write 1 Sent Done Done false
15532 5211 Acked Flush 0 Sent Done Done false
15533 5212 Acked Read 9 Sent Done Done false
15534 5213 Acked Read 10 Sent Done Done false
15535 5214 Acked Write 1 Sent Done Done false
15536 5215 Acked Read 2 Sent Done Done false
15537 5216 Acked Read 9 Sent Done Done false
15538 5217 Acked Write 9 Sent Done Done false
15539 5218 Acked Read 2 Sent Done Done false
15540 5219 Acked Flush 0 Sent Done Done false
15541 5220 Acked Write 3 Sent Done Done false
15542 5221 Acked Write 7 Sent Done Done false
15543 5222 Acked Write 7 Sent Done Done false
15544 5223 Acked Flush 0 Sent Done Done false
15545 5224 Acked Write 2 Sent Done Done false
15546 5225 Acked Write 7 Sent Done Done false
15547 STATES DS:0 DS:1 DS:2 TOTAL
15548 Sent 301 0 0 301
15549 Done 0 301 301 602
15550 Skipped 0 0 0 0
15551 Error 0 0 0 0
15552 Last Flush: Some(JobId(4924)) Some(JobId(5223)) Some(JobId(5223))
15553 Downstairs last five completed: 4924 4923 4922 4921 4920
15554 Upstairs last five completed: 5225 5224 5223 5222 5221
15555 CLIENT: Up:0 ds:301 act:2
155562025-02-25T17:21:38.029ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
155572025-02-25T17:21:38.600ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
15558 [8810] got start action
15559 [8810] I am going to restart
15560 Starting downstairs at port 8810
15561 Make output file at "/tmp/dsc/downstairs-8810.txt"
15562 [8810][0] reports Starting
15563 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1222)
15564 [8810][0] reports Running
155652025-02-25T17:21:39.601ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
155662025-02-25T17:21:40.603ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
15567 ----------------------------------------------------------------
15568 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15569 States: Offline Active Active
15570 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15571 4925 Acked Write 6 Sent Done Done false
15572 4926 Acked Read 8 Sent Done Done false
15573 4927 Acked Read 4 Sent Done Done false
15574 4928 Acked Write 3 Sent Done Done false
15575 4929 Acked Write 6 Sent Done Done false
15576 4930 Acked Write 3 Sent Done Done false
15577 4931 Acked Write 2 Sent Done Done false
15578 4932 Acked Read 5 Sent Done Done false
15579 4933 Acked Read 8 Sent Done Done false
15580 4934 Acked Read 2 Sent Done Done false
15581 4935 Acked Read 10 Sent Done Done false
15582 4936 Acked Read 10 Sent Done Done false
15583 4937 Acked Read 9 Sent Done Done false
15584 4938 Acked Read 5 Sent Done Done false
15585 4939 Acked Write 6 Sent Done Done false
15586 4940 Acked Read 1 Sent Done Done false
15587 4941 Acked Write 4 Sent Done Done false
15588 4942 Acked Write 9 Sent Done Done false
15589 4943 Acked Write 10 Sent Done Done false
15590 4944 Acked Read 5 Sent Done Done false
15591 4945 Acked Read 5 Sent Done Done false
15592 4946 Acked Read 3 Sent Done Done false
15593 4947 Acked Read 6 Sent Done Done false
15594 4948 Acked Flush 0 Sent Done Done false
15595 4949 Acked Read 6 Sent Done Done false
15596 4950 Acked Flush 0 Sent Done Done false
15597 4951 Acked Read 7 Sent Done Done false
15598 4952 Acked Write 3 Sent Done Done false
15599 4953 Acked Write 6 Sent Done Done false
15600 4954 Acked Write 2 Sent Done Done false
15601 4955 Acked Read 9 Sent Done Done false
15602 4956 Acked Read 1 Sent Done Done false
15603 4957 Acked Write 4 Sent Done Done false
15604 4958 Acked Write 3 Sent Done Done false
15605 4959 Acked Read 4 Sent Done Done false
15606 4960 Acked Read 9 Sent Done Done false
15607 4961 Acked Read 1 Sent Done Done false
15608 4962 Acked Read 5 Sent Done Done false
15609 4963 Acked Write 7 Sent Done Done false
15610 4964 Acked Write 7 Sent Done Done false
15611 4965 Acked Read 4 Sent Done Done false
15612 4966 Acked Read 10 Sent Done Done false
15613 4967 Acked Write 4 Sent Done Done false
15614 4968 Acked Flush 0 Sent Done Done false
15615 4969 Acked Write 1 Sent Done Done false
15616 4970 Acked Write 10 Sent Done Done false
15617 4971 Acked Read 3 Sent Done Done false
15618 4972 Acked Write 5 Sent Done Done false
15619 4973 Acked Read 8 Sent Done Done false
15620 4974 Acked Read 2 Sent Done Done false
15621 4975 Acked Read 10 Sent Done Done false
15622 4976 Acked Read 1 Sent Done Done false
15623 4977 Acked Read 7 Sent Done Done false
15624 4978 Acked Read 7 Sent Done Done false
15625 4979 Acked Read 9 Sent Done Done false
15626 4980 Acked Read 4 Sent Done Done false
15627 4981 Acked Read 6 Sent Done Done false
15628 4982 Acked Read 1 Sent Done Done false
15629 4983 Acked Read 4 Sent Done Done false
15630 4984 Acked Read 9 Sent Done Done false
15631 4985 Acked Read 8 Sent Done Done false
15632 4986 Acked Read 3 Sent Done Done false
15633 4987 Acked Read 5 Sent Done Done false
15634 4988 Acked Write 8 Sent Done Done false
15635 4989 Acked Read 3 Sent Done Done false
15636 4990 Acked Read 2 Sent Done Done false
15637 4991 Acked Read 8 Sent Done Done false
15638 4992 Acked Write 4 Sent Done Done false
15639 4993 Acked Read 9 Sent Done Done false
15640 4994 Acked Write 4 Sent Done Done false
15641 4995 Acked Write 9 Sent Done Done false
15642 4996 Acked Write 5 Sent Done Done false
15643 4997 Acked Write 1 Sent Done Done false
15644 4998 Acked Write 8 Sent Done Done false
15645 4999 Acked Read 2 Sent Done Done false
15646 5000 Acked Read 10 Sent Done Done false
15647 5001 Acked Read 7 Sent Done Done false
15648 5002 Acked Read 2 Sent Done Done false
15649 5003 Acked Write 8 Sent Done Done false
15650 5004 Acked Read 1 Sent Done Done false
15651 5005 Acked Write 1 Sent Done Done false
15652 5006 Acked Write 6 Sent Done Done false
15653 5007 Acked Read 7 Sent Done Done false
15654 5008 Acked Read 7 Sent Done Done false
15655 5009 Acked Read 7 Sent Done Done false
15656 5010 Acked Read 3 Sent Done Done false
15657 5011 Acked Write 7 Sent Done Done false
15658 5012 Acked Read 9 Sent Done Done false
15659 5013 Acked Write 10 Sent Done Done false
15660 5014 Acked Read 8 Sent Done Done false
15661 5015 Acked Write 3 Sent Done Done false
15662 5016 Acked Read 9 Sent Done Done false
15663 5017 Acked Read 6 Sent Done Done false
15664 5018 Acked Write 5 Sent Done Done false
15665 5019 Acked Write 4 Sent Done Done false
15666 5020 Acked Read 1 Sent Done Done false
15667 5021 Acked Read 6 Sent Done Done false
15668 5022 Acked Read 5 Sent Done Done false
15669 5023 Acked Flush 0 Sent Done Done false
15670 5024 Acked Read 4 Sent Done Done false
15671 5025 Acked Write 10 Sent Done Done false
15672 5026 Acked Read 8 Sent Done Done false
15673 5027 Acked Read 1 Sent Done Done false
15674 5028 Acked Write 6 Sent Done Done false
15675 5029 Acked Write 7 Sent Done Done false
15676 5030 Acked Write 4 Sent Done Done false
15677 5031 Acked Flush 0 Sent Done Done false
15678 5032 Acked Read 7 Sent Done Done false
15679 5033 Acked Write 4 Sent Done Done false
15680 5034 Acked Write 5 Sent Done Done false
15681 5035 Acked Write 7 Sent Done Done false
15682 5036 Acked Write 4 Sent Done Done false
15683 5037 Acked Write 1 Sent Done Done false
15684 5038 Acked Read 8 Sent Done Done false
15685 5039 Acked Write 3 Sent Done Done false
15686 5040 Acked Write 6 Sent Done Done false
15687 5041 Acked Write 1 Sent Done Done false
15688 5042 Acked Read 6 Sent Done Done false
15689 5043 Acked Write 9 Sent Done Done false
15690 5044 Acked Write 7 Sent Done Done false
15691 5045 Acked Flush 0 Sent Done Done false
15692 5046 Acked Read 9 Sent Done Done false
15693 5047 Acked Read 5 Sent Done Done false
15694 5048 Acked Read 1 Sent Done Done false
15695 5049 Acked Write 9 Sent Done Done false
15696 5050 Acked Write 8 Sent Done Done false
15697 5051 Acked Write 2 Sent Done Done false
15698 5052 Acked Read 9 Sent Done Done false
15699 5053 Acked Read 8 Sent Done Done false
15700 5054 Acked Write 2 Sent Done Done false
15701 5055 Acked Flush 0 Sent Done Done false
15702 5056 Acked Read 10 Sent Done Done false
15703 5057 Acked Read 9 Sent Done Done false
15704 5058 Acked Write 7 Sent Done Done false
15705 5059 Acked Read 5 Sent Done Done false
15706 5060 Acked Write 4 Sent Done Done false
15707 5061 Acked Read 6 Sent Done Done false
15708 5062 Acked Flush 0 Sent Done Done false
15709 5063 Acked Write 10 Sent Done Done false
15710 5064 Acked Read 8 Sent Done Done false
15711 5065 Acked Write 8 Sent Done Done false
15712 5066 Acked Write 5 Sent Done Done false
15713 5067 Acked Write 5 Sent Done Done false
15714 5068 Acked Read 1 Sent Done Done false
15715 5069 Acked Write 2 Sent Done Done false
15716 5070 Acked Read 9 Sent Done Done false
15717 5071 Acked Write 3 Sent Done Done false
15718 5072 Acked Read 8 Sent Done Done false
15719 5073 Acked Write 4 Sent Done Done false
15720 5074 Acked Write 8 Sent Done Done false
15721 5075 Acked Flush 0 Sent Done Done false
15722 5076 Acked Read 8 Sent Done Done false
15723 5077 Acked Read 9 Sent Done Done false
15724 5078 Acked Read 9 Sent Done Done false
15725 5079 Acked Read 9 Sent Done Done false
15726 5080 Acked Write 7 Sent Done Done false
15727 5081 Acked Write 4 Sent Done Done false
15728 5082 Acked Write 1 Sent Done Done false
15729 5083 Acked Read 2 Sent Done Done false
15730 5084 Acked Read 8 Sent Done Done false
15731 5085 Acked Read 8 Sent Done Done false
15732 5086 Acked Write 1 Sent Done Done false
15733 5087 Acked Write 6 Sent Done Done false
15734 5088 Acked Read 5 Sent Done Done false
15735 5089 Acked Write 5 Sent Done Done false
15736 5090 Acked Write 3 Sent Done Done false
15737 5091 Acked Read 4 Sent Done Done false
15738 5092 Acked Write 4 Sent Done Done false
15739 5093 Acked Flush 0 Sent Done Done false
15740 5094 Acked Read 10 Sent Done Done false
15741 5095 Acked Read 8 Sent Done Done false
15742 5096 Acked Write 7 Sent Done Done false
15743 5097 Acked Read 9 Sent Done Done false
15744 5098 Acked Write 4 Sent Done Done false
15745 5099 Acked Write 5 Sent Done Done false
15746 5100 Acked Read 1 Sent Done Done false
15747 5101 Acked Write 1 Sent Done Done false
15748 5102 Acked Read 2 Sent Done Done false
15749 5103 Acked Read 6 Sent Done Done false
15750 5104 Acked Read 5 Sent Done Done false
15751 5105 Acked Read 9 Sent Done Done false
15752 5106 Acked Read 5 Sent Done Done false
15753 5107 Acked Write 8 Sent Done Done false
15754 5108 Acked Read 3 Sent Done Done false
15755 5109 Acked Write 3 Sent Done Done false
15756 5110 Acked Write 9 Sent Done Done false
15757 5111 Acked Write 1 Sent Done Done false
15758 5112 Acked Flush 0 Sent Done Done false
15759 5113 Acked Flush 0 Sent Done Done false
15760 5114 Acked Read 4 Sent Done Done false
15761 5115 Acked Read 8 Sent Done Done false
15762 5116 Acked Read 7 Sent Done Done false
15763 5117 Acked Write 8 Sent Done Done false
15764 5118 Acked Read 7 Sent Done Done false
15765 5119 Acked Write 2 Sent Done Done false
15766 5120 Acked Read 4 Sent Done Done false
15767 5121 Acked Read 3 Sent Done Done false
15768 5122 Acked Flush 0 Sent Done Done false
15769 5123 Acked Flush 0 Sent Done Done false
15770 5124 Acked Flush 0 Sent Done Done false
15771 5125 Acked Write 10 Sent Done Done false
15772 5126 Acked Write 6 Sent Done Done false
15773 5127 Acked Flush 0 Sent Done Done false
15774 5128 Acked Read 9 Sent Done Done false
15775 5129 Acked Write 9 Sent Done Done false
15776 5130 Acked Write 8 Sent Done Done false
15777 5131 Acked Read 6 Sent Done Done false
15778 5132 Acked Write 1 Sent Done Done false
15779 5133 Acked Read 8 Sent Done Done false
15780 5134 Acked Write 7 Sent Done Done false
15781 5135 Acked Read 1 Sent Done Done false
15782 5136 Acked Read 6 Sent Done Done false
15783 5137 Acked Read 7 Sent Done Done false
15784 5138 Acked Flush 0 Sent Done Done false
15785 5139 Acked Flush 0 Sent Done Done false
15786 5140 Acked Write 10 Sent Done Done false
15787 5141 Acked Write 8 Sent Done Done false
15788 5142 Acked Read 5 Sent Done Done false
15789 5143 Acked Read 7 Sent Done Done false
15790 5144 Acked Read 4 Sent Done Done false
15791 5145 Acked Read 9 Sent Done Done false
15792 5146 Acked Write 6 Sent Done Done false
15793 5147 Acked Read 1 Sent Done Done false
15794 5148 Acked Write 4 Sent Done Done false
15795 5149 Acked Flush 0 Sent Done Done false
15796 5150 Acked Read 5 Sent Done Done false
15797 5151 Acked Read 5 Sent Done Done false
15798 5152 Acked Read 10 Sent Done Done false
15799 5153 Acked Read 5 Sent Done Done false
15800 5154 Acked Write 4 Sent Done Done false
15801 5155 Acked Read 8 Sent Done Done false
15802 5156 Acked Read 5 Sent Done Done false
15803 5157 Acked Read 2 Sent Done Done false
15804 5158 Acked Read 5 Sent Done Done false
15805 5159 Acked Read 2 Sent Done Done false
15806 5160 Acked Read 6 Sent Done Done false
15807 5161 Acked Write 8 Sent Done Done false
15808 5162 Acked Read 6 Sent Done Done false
15809 5163 Acked Read 7 Sent Done Done false
15810 5164 Acked Write 6 Sent Done Done false
15811 5165 Acked Read 5 Sent Done Done false
15812 5166 Acked Read 1 Sent Done Done false
15813 5167 Acked Flush 0 Sent Done Done false
15814 5168 Acked Read 3 Sent Done Done false
15815 5169 Acked Read 9 Sent Done Done false
15816 5170 Acked Read 1 Sent Done Done false
15817 5171 Acked Read 10 Sent Done Done false
15818 5172 Acked Read 1 Sent Done Done false
15819 5173 Acked Read 8 Sent Done Done false
15820 5174 Acked Write 5 Sent Done Done false
15821 5175 Acked Write 3 Sent Done Done false
15822 5176 Acked Read 4 Sent Done Done false
15823 5177 Acked Write 9 Sent Done Done false
15824 5178 Acked Read 5 Sent Done Done false
15825 5179 Acked Write 4 Sent Done Done false
15826 5180 Acked Read 1 Sent Done Done false
15827 5181 Acked Write 2 Sent Done Done false
15828 5182 Acked Write 1 Sent Done Done false
15829 5183 Acked Read 8 Sent Done Done false
15830 5184 Acked Write 10 Sent Done Done false
15831 5185 Acked Flush 0 Sent Done Done false
15832 5186 Acked Read 7 Sent Done Done false
15833 5187 Acked Read 9 Sent Done Done false
15834 5188 Acked Read 3 Sent Done Done false
15835 5189 Acked Write 4 Sent Done Done false
15836 5190 Acked Read 8 Sent Done Done false
15837 5191 Acked Write 4 Sent Done Done false
15838 5192 Acked Read 2 Sent Done Done false
15839 5193 Acked Read 9 Sent Done Done false
15840 5194 Acked Write 6 Sent Done Done false
15841 5195 Acked Read 1 Sent Done Done false
15842 5196 Acked Flush 0 Sent Done Done false
15843 5197 Acked Write 2 Sent Done Done false
15844 5198 Acked Read 2 Sent Done Done false
15845 5199 Acked Flush 0 Sent Done Done false
15846 5200 Acked Flush 0 Sent Done Done false
15847 5201 Acked Read 8 Sent Done Done false
15848 5202 Acked Flush 0 Sent Done Done false
15849 5203 Acked Write 2 Sent Done Done false
15850 5204 Acked Write 4 Sent Done Done false
15851 5205 Acked Write 7 Sent Done Done false
15852 5206 Acked Write 10 Sent Done Done false
15853 5207 Acked Read 9 Sent Done Done false
15854 5208 Acked Read 10 Sent Done Done false
15855 5209 Acked Read 10 Sent Done Done false
15856 5210 Acked Write 1 Sent Done Done false
15857 5211 Acked Flush 0 Sent Done Done false
15858 5212 Acked Read 9 Sent Done Done false
15859 5213 Acked Read 10 Sent Done Done false
15860 5214 Acked Write 1 Sent Done Done false
15861 5215 Acked Read 2 Sent Done Done false
15862 5216 Acked Read 9 Sent Done Done false
15863 5217 Acked Write 9 Sent Done Done false
15864 5218 Acked Read 2 Sent Done Done false
15865 5219 Acked Flush 0 Sent Done Done false
15866 5220 Acked Write 3 Sent Done Done false
15867 5221 Acked Write 7 Sent Done Done false
15868 5222 Acked Write 7 Sent Done Done false
15869 5223 Acked Flush 0 Sent Done Done false
15870 5224 Acked Write 2 Sent Done Done false
15871 5225 Acked Write 7 Sent Done Done false
15872 STATES DS:0 DS:1 DS:2 TOTAL
15873 Sent 301 0 0 301
15874 Done 0 301 301 602
15875 Skipped 0 0 0 0
15876 Error 0 0 0 0
15877 Last Flush: Some(JobId(4924)) Some(JobId(5223)) Some(JobId(5223))
15878 Downstairs last five completed: 4924 4923 4922 4921 4920
15879 Upstairs last five completed: 5225 5224 5223 5222 5221
15880 CLIENT: Up:0 ds:301 act:2
158812025-02-25T17:21:42.031ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
158822025-02-25T17:21:42.607ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
158832025-02-25T17:21:43.268ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
158842025-02-25T17:21:43.268ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61788) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
158852025-02-25T17:21:43.269ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
158862025-02-25T17:21:43.270ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
158872025-02-25T17:21:43.270ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
158882025-02-25T17:21:43.270ZINFOcrucible: send last flush ID to this DS: Some(JobId(4924)) = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
158892025-02-25T17:21:43.270ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
158902025-02-25T17:21:43.270ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4924)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
158912025-02-25T17:21:43.270ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4924)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
15892 ----------------------------------------------------------------
15893 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15894 Downstairs last five completed: 5226 5225 5224 5223 5222
15895 Upstairs last five completed: 5226 5225 5224 5223 5222
15896 CLIENT: Up:0 ds:0 act:3
15897 Replay: All jobs finished
15898 Main task has work to do, go find it
15899 got dsc Stop(1)
15900 stop 1
15901 Feb 25 17:21:45.532 INFO request completed[8820] Got stop action so:false kr:false
15902 , latency_us: 386, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 58e15f3b-3cc3-4c72-b187-ab3cbdf764c8, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
159032025-02-25T17:21:45.533ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
15904 Feb 25 17:21:45.533 INFO request completed, latency_us: 161, response_code: 200, uri: /state/cid/1, method: GET, req_id: 485136b8-6fd2-42c5-aa1f-f9eeeab131b7, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
159052025-02-25T17:21:45.533ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
159062025-02-25T17:21:45.533ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
159072025-02-25T17:21:45.533ZINFOcrucible: client task is exiting = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
15908 [8820] Exited with: ExitStatus(unix_wait_status(9))
15909 [8820][1] reports Exit
15910 Feb 25 17:21:49.536 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/1, method: GET, req_id: 47f2bff5-7c9a-4935-9dc8-b5801289669b, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
15911 001/300 Write block 2876 len 36864 data: 4 5 4 4 4 6 4 4 4
15912 002/300 Write block 4750 len 16384 data: 4 4 5 5
15913 003/300 Write block 3909 len 28672 data: 4 4 5 5 5 4 4
15914 004/300 Read block 1344 len 32768
15915 005/300 Write block 902 len 12288 data: 4 4 4
15916 006/300 Read block 1058 len 32768
15917 007/300 Write block 644 len 8192 data: 3 2
15918 008/300 Read block 684 len 16384
15919 009/300 Write block 3268 len 28672 data: 3 3 3 4 5 5 5
15920 010/300 Read block 1904 len 24576
15921 011/300 Flush
15922 012/300 Read block 4842 len 12288
15923 013/300 Write block 1768 len 8192 data: 5 5
15924 014/300 Write block 3993 len 8192 data: 4 4
15925 015/300 Write block 2256 len 40960 data: 3 4 3 3 4 5 4 4 3 3
15926 016/300 Write block 2515 len 40960 data: 3 3 5 6 4 4 3 3 3 3
15927 017/300 Read block 1136 len 8192
15928 018/300 Write block 4113 len 20480 data: 3 3 2 2 2
15929 019/300 Write block 2328 len 32768 data: 5 6 7 5 6 6 6 5
15930 020/300 Write block 2240 len 40960 data: 3 3 3 3 2 2 3 3 3 3
15931 021/300 Write block 1079 len 24576 data: 4 3 3 3 3 3
15932 022/300 Write block 1020 len 28672 data: 4 4 5 5 5 6 4
15933 023/300 Write block 1538 len 16384 data: 3 3 3 3
15934 024/300 Flush
15935 025/300 Write block 4791 len 40960 data: 3 2 2 2 2 3 3 5 5 5
15936 026/300 Write block 1893 len 32768 data: 3 3 4 4 4 4 5 5
15937 027/300 Flush
15938 028/300 Read block 636 len 8192
15939 029/300 Read block 311 len 8192
15940 030/300 Read block 1019 len 24576
15941 031/300 Write block 1313 len 4096 data: 4
15942 032/300 Write block 4598 len 36864 data: 4 5 5 5 6 6 5 4 3
15943 033/300 Flush
15944 034/300 Read block 1838 len 24576
15945 035/300 Write block 832 len 12288 data: 6 9 10
15946 036/300 Flush
15947 037/300 Write block 2633 len 32768 data: 4 3 2 2 4 4 3 2
15948 038/300 Write block 1194 len 4096 data: 5
15949 039/300 Read block 1491 len 40960
15950 040/300 Write block 4743 len 36864 data: 3 2 2 3 3 4 4 5 5
15951 041/300 Read block 835 len 32768
15952 042/300 Write block 1712 len 16384 data: 6 5 5 5
15953 043/300 Write block 134 len 16384 data: 2 2 2 2
15954 044/300 Write block 276 len 20480 data: 4 3 3 4 4
15955 045/300 Write block 1579 len 12288 data: 3 4 5
15956 046/300 Write block 3390 len 40960 data: 4 4 5 5 5 4 4 3 3 4
15957 047/300 Write block 1228 len 32768 data: 2 2 2 3 4 4 6 6
15958 048/300 Write block 2912 len 16384 data: 2 2 3 3
15959 049/300 Write block 71 len 20480 data: 6 6 6 6 5
15960 050/300 Flush
15961 051/300 Read block 3485 len 24576
15962 052/300 Read block 4077 len 28672
15963 053/300 Write block 907 len 24576 data: 5 4 3 3 3 4
15964 054/300 Flush
15965 055/300 Write block 2633 len 32768 data: 5 4 3 3 5 5 4 3
15966 056/300 Read block 1778 len 28672
15967 057/300 Write block 3090 len 28672 data: 5 6 5 5 4 4 4
15968 058/300 Read block 2695 len 4096
15969 059/300 Read block 835 len 12288
15970 060/300 Read block 4177 len 20480
15971 061/300 Write block 4284 len 4096 data: 3
15972 062/300 Write block 2047 len 16384 data: 2 2 2 3
15973 063/300 Write block 3283 len 16384 data: 4 4 4 4
15974 064/300 Write block 2762 len 36864 data: 3 5 5 6 6 7 6 5 5
15975 065/300 Read block 1446 len 8192
15976 066/300 Read block 2622 len 4096
15977 067/300 Read block 3982 len 20480
15978 068/300 Read block 155 len 20480
15979 069/300 Write block 3802 len 24576 data: 3 3 2 3 4 4
15980 070/300 Write block 2053 len 24576 data: 2 2 3 3 4 4
15981 071/300 Read block 795 len 20480
15982 072/300 Read block 3460 len 28672
15983 073/300 Read block 4254 len 12288
15984 074/300 Write block 2608 len 16384 data: 3 4 4 3
15985 075/300 Read block 391 len 8192
15986 076/300 Read block 3658 len 8192
15987 077/300 Flush
15988 078/300 Read block 4894 len 32768
15989 079/300 Read block 428 len 40960
15990 080/300 Write block 940 len 8192 data: 3 3
15991 081/300 Write block 2859 len 36864 data: 7 5 4 4 4 4 4 4 3
15992 082/300 Write block 3218 len 16384 data: 7 8 6 7
15993 083/300 Read block 3557 len 16384
15994 084/300 Read block 1172 len 12288
15995 085/300 Write block 3573 len 4096 data: 2
15996 086/300 Write block 2933 len 4096 data: 3
15997 087/300 Read block 3482 len 24576
15998 088/300 Read block 65 len 40960
15999 089/300 Write block 65 len 20480 data: 3 3 3 5 6
16000 090/300 Read block 4834 len 40960
16001 091/300 Read block 2407 len 36864
16002 092/300 Read block 1755 len 32768
16003 093/300 Write block 2092 len 40960 data: 5 6 6 6 6 5 5 5 4 4
16004 094/300 Write block 4740 len 8192 data: 2 2
16005 095/300 Read block 3630 len 4096
16006 096/300 Read block 3768 len 20480
16007 097/300 Read block 225 len 4096
16008 098/300 Write block 1755 len 8192 data: 4 4
16009 099/300 Read block 4996 len 16384
16010 100/300 Write block 3515 len 28672 data: 4 4 4 4 4 4 4
16011 101/300 Write block 3735 len 4096 data: 5
16012 102/300 Write block 1479 len 12288 data: 4 5 5
16013 103/300 Read block 4127 len 28672
16014 104/300 Read block 1254 len 24576
16015 105/300 Read block 1784 len 40960
16016 106/300 Read block 2769 len 4096
16017 107/300 Read block 90 len 40960
16018 108/300 Flush
16019 109/300 Write block 3892 len 16384 data: 2 3 3 3
16020 110/300 Write block 1361 len 40960 data: 6 7 7 8 8 7 6 4 4 2
16021 111/300 Write block 4908 len 20480 data: 2 2 2 2 2
16022 112/300 Write block 3389 len 12288 data: 4 5 5
16023 113/300 Flush
16024 114/300 Write block 2003 len 28672 data: 3 4 4 4 4 5 5
16025 115/300 Read block 2598 len 8192
16026 116/300 Read block 1342 len 24576
16027 117/300 Write block 596 len 36864 data: 4 5 4 4 4 5 4 4 4
16028 118/300 Write block 1983 len 16384 data: 4 3 3 3
16029 119/300 Read block 4985 len 16384
16030 120/300 Write block 2533 len 28672 data: 2 2 2 3 3 4 4
16031 121/300 Read block 2631 len 4096
16032 122/300 Write block 4624 len 28672 data: 4 4 4 3 5 5 5
16033 123/300 Read block 2235 len 12288
16034 124/300 Write block 3847 len 16384 data: 3 3 3 4
16035 125/300 Write block 4118 len 4096 data: 2
16036 126/300 Read block 1896 len 28672
16037 127/300 Read block 198 len 12288
16038 128/300 Read block 374 len 20480
16039 129/300 Write block 3958 len 16384 data: 2 2 2 2
16040 130/300 Read block 2992 len 16384
16041 131/300 Write block 1673 len 24576 data: 4 4 4 3 2 2
16042 132/300 Flush
16043 133/300 Read block 674 len 8192
16044 134/300 Read block 735 len 24576
16045 135/300 Read block 1972 len 8192
16046 136/300 Read block 1049 len 28672
16047 137/300 Write block 2039 len 24576 data: 5 4 4 5 6 5
16048 138/300 Write block 2764 len 20480 data: 6 7 7 8 7
16049 139/300 Flush
16050 140/300 Read block 3710 len 12288
16051 141/300 Flush
16052 142/300 Flush
16053 143/300 Write block 1915 len 8192 data: 4 4
16054 144/300 Read block 145 len 28672
16055 145/300 Read block 3241 len 8192
16056 146/300 Flush
16057 147/300 Read block 474 len 4096
16058 148/300 Write block 2845 len 24576 data: 2 2 2 2 2 3
16059 149/300 Flush
16060 150/300 Read block 80 len 20480
16061 151/300 Write block 1089 len 24576 data: 4 6 6 7 5 4
16062 152/300 Write block 1728 len 12288 data: 3 3 3
16063 153/300 Read block 1149 len 20480
16064 154/300 Read block 1725 len 20480
16065 155/300 Write block 2009 len 24576 data: 6 5 5 4 3 3
16066 156/300 Read block 2415 len 36864
16067 157/300 Flush
16068 158/300 Write block 4591 len 12288 data: 5 5 4
16069 159/300 Flush
16070 160/300 Write block 927 len 20480 data: 4 4 4 5 5
16071 161/300 Read block 2725 len 28672
16072 162/300 Write block 1835 len 32768 data: 2 3 4 4 4 4 3 3
16073 163/300 Read block 4385 len 32768
16074 164/300 Read block 1295 len 24576
16075 165/300 Write block 166 len 20480 data: 3 2 2 2 2
16076 166/300 Read block 799 len 28672
16077 167/300 Write block 1078 len 36864 data: 3 5 4 4 4 4 4 4 4
16078 168/300 Read block 875 len 24576
16079 169/300 Read block 4144 len 24576
16080 170/300 Write block 4580 len 24576 data: 4 4 4 4 3 3
16081 171/300 Read block 1788 len 40960
16082 172/300 Write block 515 len 8192 data: 5 5
16083 173/300 Read block 2472 len 12288
16084 174/300 Flush
16085 175/300 Read block 2455 len 8192
16086 176/300 Read block 4869 len 28672
16087 177/300 Write block 2399 len 24576 data: 5 4 3 3 3 4
16088 178/300 Read block 4709 len 36864
16089 179/300 Read block 2367 len 12288
16090 180/300 Write block 4139 len 40960 data: 3 3 3 3 3 3 3 3 3 4
16091 181/300 Write block 3896 len 4096 data: 4
16092 182/300 Read block 1417 len 12288
16093 183/300 Write block 1369 len 28672 data: 5 3 3 3 3 3 5
16094 184/300 Write block 524 len 16384 data: 3 3 3 2
16095 185/300 Read block 384 len 20480
16096 186/300 Read block 1484 len 4096
16097 187/300 Flush
16098 188/300 Write block 2289 len 28672 data: 2 2 3 3 4 4 4
16099 189/300 Write block 4064 len 16384 data: 5 4 3 3
16100 190/300 Read block 3327 len 12288
16101 191/300 Write block 1316 len 12288 data: 4 3 3
16102 192/300 Read block 1645 len 12288
16103 193/300 Write block 636 len 40960 data: 5 4 4 3 3 4 5 4 4 3
16104 194/300 Read block 2443 len 8192
16105 195/300 Read block 1523 len 20480
16106 196/300 Flush
16107 197/300 Read block 1102 len 12288
16108 198/300 Write block 3721 len 24576 data: 4 5 4 6 7 7
16109 199/300 Write block 2786 len 32768 data: 5 5 4 5 4 6 6 6
16110 200/300 Write block 2301 len 16384 data: 4 5 6 6
16111 201/300 Write block 3527 len 8192 data: 2 2
16112 202/300 Read block 4285 len 40960
16113 203/300 Flush
16114 204/300 Read block 4214 len 8192
16115 205/300 Write block 636 len 8192 data: 6 5
16116 206/300 Write block 2013 len 40960 data: 4 4 3 3 3 4 3 3 3 2
16117 207/300 Read block 3508 len 16384
16118 208/300 Read block 3090 len 36864
16119 209/300 Read block 3671 len 28672
16120 210/300 Write block 1073 len 4096 data: 4
16121 211/300 Read block 1747 len 28672
16122 212/300 Read block 2344 len 12288
16123 213/300 Read block 3050 len 32768
16124 214/300 Read block 3998 len 32768
16125 215/300 Write block 4641 len 28672 data: 2 3 4 3 2 2 2
16126 216/300 Read block 1191 len 8192
16127 217/300 Read block 2427 len 12288
16128 218/300 Read block 3943 len 36864
16129 219/300 Write block 1307 len 20480 data: 3 3 3 3 3
16130 220/300 Write block 4209 len 36864 data: 5 4 4 5 4 4 3 2 2
16131 221/300 Write block 138 len 40960 data: 2 2 2 3 3 4 3 3 3 3
16132 222/300 Read block 1102 len 12288
16133 223/300 Read block 2727 len 8192
16134 224/300 Write block 2222 len 16384 data: 5 5 4 3
16135 225/300 Read block 2989 len 12288
16136 226/300 Write block 4984 len 8192 data: 3 3
16137 227/300 Write block 2123 len 40960 data: 4 3 3 3 3 5 5 4 4 4
16138 228/300 Read block 4929 len 28672
16139 229/300 Read block 4911 len 8192
16140 230/300 Read block 1964 len 8192
16141 231/300 Write block 1816 len 40960 data: 3 3 3 3 3 3 3 4 4 3
16142 232/300 Read block 3600 len 8192
16143 233/300 Write block 134 len 20480 data: 3 3 3 3 3
16144 234/300 Write block 482 len 32768 data: 4 4 4 4 3 3 4 6
16145 235/300 Read block 4270 len 32768
16146 236/300 Write block 1743 len 28672 data: 2 2 2 2 2 3 3
16147 237/300 Read block 1372 len 36864
16148 238/300 Flush
16149 239/300 Write block 4309 len 4096 data: 4
16150 240/300 Read block 3326 len 12288
16151 241/300 Read block 3344 len 12288
16152 242/300 Write block 2339 len 28672 data: 2 2 3 3 2 3 3
16153 243/300 Write block 298 len 28672 data: 3 4 3 3 3 3 4
16154 244/300 Read block 4080 len 12288
16155 245/300 Read block 4653 len 12288
16156 246/300 Read block 2513 len 32768
16157 247/300 Read block 4751 len 40960
16158 248/300 Read block 484 len 40960
16159 249/300 Read block 412 len 32768
16160 250/300 Write block 3183 len 20480 data: 3 3 3 2 2
16161 251/300 Read block 4469 len 16384
16162 252/300 Read block 2854 len 40960
16163 253/300 Read block 4813 len 4096
16164 254/300 Write block 3833 len 12288 data: 3 3 3
16165 255/300 Read block 1901 len 36864
16166 256/300 Write block 42 len 36864 data: 4 5 5 6 6 4 3 3 3
16167 257/300 Read block 1870 len 40960
16168 258/300 Read block 2172 len 36864
16169 259/300 Write block 3840 len 36864 data: 5 5 4 3 3 3 2 4 4
16170 260/300 Write block 2733 len 8192 data: 4 5
16171 261/300 Write block 1364 len 4096 data: 9
16172 262/300 Read block 4907 len 32768
16173 263/300 Read block 1033 len 24576
16174 264/300 Flush
16175 265/300 Read block 714 len 32768
16176 266/300 Read block 3072 len 12288
16177 267/300 Write block 4349 len 40960 data: 6 5 6 5 5 5 7 7 4 5
16178 268/300 Read block 810 len 36864
16179 269/300 Read block 4397 len 20480
16180 270/300 Write block 1316 len 4096 data: 5
16181 271/300 Read block 2072 len 24576
16182 272/300 Read block 1780 len 28672
16183 273/300 Read block 3619 len 28672
16184 274/300 Read block 2629 len 40960
16185 275/300 Read block 4874 len 20480
16186 276/300 Read block 1943 len 16384
16187 277/300 Read block 3693 len 20480
16188 278/300 Write block 2939 len 32768 data: 3 3 4 4 5 4 4 5
16189 279/300 Read block 4918 len 4096
16190 280/300 Flush
16191 281/300 Write block 3446 len 40960 data: 4 4 4 5 5 4 4 4 4 4
16192 282/300 Write block 1171 len 16384 data: 4 4 3 4
16193 283/300 Read block 3777 len 16384
16194 284/300 Flush
16195 285/300 Read block 2654 len 28672
16196 286/300 Write block 1161 len 32768 data: 3 3 4 4 4 4 4 5
16197 287/300 Read block 4829 len 24576
16198 288/300 Read block 1506 len 24576
16199 289/300 Flush
16200 290/300 Read block 2643 len 20480
16201 291/300 Read block 3608 len 36864
16202 292/300 Read block 4074 len 24576
16203 293/300 Write block 3343 len 28672 data: 3 3 3 4 4 4 5
16204 294/300 Read block 3367 len 12288
16205 295/300 Read block 256 len 28672
16206 296/300 Read block 2064 len 36864
16207 297/300 Write block 3795 len 8192 data: 6 6
16208 298/300 Flush
16209 299/300 Read block 2921 len 40960
16210 300/300 Read block 3330 len 20480
16211 301/300 Read block 4830 len 36864
16212 Main task has work to do, go find it
16213 got dsc Start(1)
16214 start 1
16215 Feb 25 17:21:49.784 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/1, method: POST, req_id: c6434346-7ff2-4caf-a450-b708067783a2, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
16216 Replay: started 1, returned:Ok(())
16217 ----------------------------------------------------------------
16218 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16219 States: Active Offline Active
16220 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16221 5227 Acked Write 9 Done Sent Done false
16222 5228 Acked Write 4 Done Sent Done false
16223 5229 Acked Write 7 Done Sent Done false
16224 5230 Acked Read 8 Done Sent Done false
16225 5231 Acked Write 3 Done Sent Done false
16226 5232 Acked Read 8 Done Sent Done false
16227 5233 Acked Write 2 Done Sent Done false
16228 5234 Acked Read 4 Done Sent Done false
16229 5235 Acked Write 7 Done Sent Done false
16230 5236 Acked Read 6 Done Sent Done false
16231 5237 Acked Flush 0 Done Sent Done false
16232 5238 Acked Read 3 Done Sent Done false
16233 5239 Acked Write 2 Done Sent Done false
16234 5240 Acked Write 2 Done Sent Done false
16235 5241 Acked Write 10 Done Sent Done false
16236 5242 Acked Write 10 Done Sent Done false
16237 5243 Acked Read 2 Done Sent Done false
16238 5244 Acked Write 5 Done Sent Done false
16239 5245 Acked Write 8 Done Sent Done false
16240 5246 Acked Write 10 Done Sent Done false
16241 5247 Acked Write 6 Done Sent Done false
16242 5248 Acked Write 7 Done Sent Done false
16243 5249 Acked Write 4 Done Sent Done false
16244 5250 Acked Flush 0 Done Sent Done false
16245 5251 Acked Write 10 Done Sent Done false
16246 5252 Acked Write 8 Done Sent Done false
16247 5253 Acked Flush 0 Done Sent Done false
16248 5254 Acked Read 2 Done Sent Done false
16249 5255 Acked Read 2 Done Sent Done false
16250 5256 Acked Read 6 Done Sent Done false
16251 5257 Acked Write 1 Done Sent Done false
16252 5258 Acked Write 9 Done Sent Done false
16253 5259 Acked Flush 0 Done Sent Done false
16254 5260 Acked Read 6 Done Sent Done false
16255 5261 Acked Write 3 Done Sent Done false
16256 5262 Acked Flush 0 Done Sent Done false
16257 5263 Acked Write 8 Done Sent Done false
16258 5264 Acked Write 1 Done Sent Done false
16259 5265 Acked Read 10 Done Sent Done false
16260 5266 Acked Write 9 Done Sent Done false
16261 5267 Acked Read 8 Done Sent Done false
16262 5268 Acked Write 4 Done Sent Done false
16263 5269 Acked Write 4 Done Sent Done false
16264 5270 Acked Write 5 Done Sent Done false
16265 5271 Acked Write 3 Done Sent Done false
16266 5272 Acked Write 10 Done Sent Done false
16267 5273 Acked Write 8 Done Sent Done false
16268 5274 Acked Write 4 Done Sent Done false
16269 5275 Acked Write 5 Done Sent Done false
16270 5276 Acked Flush 0 Done Sent Done false
16271 5277 Acked Read 6 Done Sent Done false
16272 5278 Acked Read 7 Done Sent Done false
16273 5279 Acked Write 6 Done Sent Done false
16274 5280 Acked Flush 0 Done Sent Done false
16275 5281 Acked Write 8 Done Sent Done false
16276 5282 Acked Read 7 Done Sent Done false
16277 5283 Acked Write 7 Done Sent Done false
16278 5284 Acked Read 1 Done Sent Done false
16279 5285 Acked Read 3 Done Sent Done false
16280 5286 Acked Read 5 Done Sent Done false
16281 5287 Acked Write 1 Done Sent Done false
16282 5288 Acked Write 4 Done Sent Done false
16283 5289 Acked Write 4 Done Sent Done false
16284 5290 Acked Write 9 Done Sent Done false
16285 5291 Acked Read 2 Done Sent Done false
16286 5292 Acked Read 1 Done Sent Done false
16287 5293 Acked Read 5 Done Sent Done false
16288 5294 Acked Read 5 Done Sent Done false
16289 5295 Acked Write 6 Done Sent Done false
16290 5296 Acked Write 6 Done Sent Done false
16291 5297 Acked Read 5 Done Sent Done false
16292 5298 Acked Read 7 Done Sent Done false
16293 5299 Acked Read 3 Done Sent Done false
16294 5300 Acked Write 4 Done Sent Done false
16295 5301 Acked Read 2 Done Sent Done false
16296 5302 Acked Read 2 Done Sent Done false
16297 5303 Acked Flush 0 Done Sent Done false
16298 5304 Acked Read 8 Done Sent Done false
16299 5305 Acked Read 10 Done Sent Done false
16300 5306 Acked Write 2 Done Sent Done false
16301 5307 Acked Write 9 Done Sent Done false
16302 5308 Acked Write 4 Done Sent Done false
16303 5309 Acked Read 4 Done Sent Done false
16304 5310 Acked Read 3 Done Sent Done false
16305 5311 Acked Write 1 Done Sent Done false
16306 5312 Acked Write 1 Done Sent Done false
16307 5313 Acked Read 6 Done Sent Done false
16308 5314 Acked Read 10 Done Sent Done false
16309 5315 Acked Write 5 Done Sent Done false
16310 5316 Acked Read 10 Done Sent Done false
16311 5317 Acked Read 9 Done Sent Done false
16312 5318 Acked Read 8 Done Sent Done false
16313 5319 Acked Write 10 Done Sent Done false
16314 5320 Acked Write 2 Done Sent Done false
16315 5321 Acked Read 1 Done Sent Done false
16316 5322 Acked Read 5 Done Sent Done false
16317 5323 Acked Read 1 Done Sent Done false
16318 5324 Acked Write 2 Done Sent Done false
16319 5325 Acked Read 4 Done Sent Done false
16320 5326 Acked Write 7 Done Sent Done false
16321 5327 Acked Write 1 Done Sent Done false
16322 5328 Acked Write 3 Done Sent Done false
16323 5329 Acked Read 7 Done Sent Done false
16324 5330 Acked Read 6 Done Sent Done false
16325 5331 Acked Read 10 Done Sent Done false
16326 5332 Acked Read 1 Done Sent Done false
16327 5333 Acked Read 10 Done Sent Done false
16328 5334 Acked Flush 0 Done Sent Done false
16329 5335 Acked Write 4 Done Sent Done false
16330 5336 Acked Write 10 Done Sent Done false
16331 5337 Acked Write 5 Done Sent Done false
16332 5338 Acked Write 3 Done Sent Done false
16333 5339 Acked Flush 0 Done Sent Done false
16334 5340 Acked Write 7 Done Sent Done false
16335 5341 Acked Read 2 Done Sent Done false
16336 5342 Acked Read 6 Done Sent Done false
16337 5343 Acked Write 9 Done Sent Done false
16338 5344 Acked Write 4 Done Sent Done false
16339 5345 Acked Read 4 Done Sent Done false
16340 5346 Acked Write 7 Done Sent Done false
16341 5347 Acked Read 1 Done Sent Done false
16342 5348 Acked Write 7 Done Sent Done false
16343 5349 Acked Read 3 Done Sent Done false
16344 5350 Acked Write 4 Done Sent Done false
16345 5351 Acked Write 1 Done Sent Done false
16346 5352 Acked Read 7 Done Sent Done false
16347 5353 Acked Read 3 Done Sent Done false
16348 5354 Acked Read 5 Done Sent Done false
16349 5355 Acked Write 4 Done Sent Done false
16350 5356 Acked Read 4 Done Sent Done false
16351 5357 Acked Write 6 Done Sent Done false
16352 5358 Acked Flush 0 Done Sent Done false
16353 5359 Acked Read 2 Done Sent Done false
16354 5360 Acked Read 6 Done Sent Done false
16355 5361 Acked Read 2 Done Sent Done false
16356 5362 Acked Read 7 Done Sent Done false
16357 5363 Acked Write 6 Done Sent Done false
16358 5364 Acked Write 5 Done Sent Done false
16359 5365 Acked Flush 0 Done Sent Done false
16360 5366 Acked Read 3 Done Sent Done false
16361 5367 Acked Flush 0 Done Sent Done false
16362 5368 Acked Flush 0 Done Sent Done false
16363 5369 Acked Write 2 Done Sent Done false
16364 5370 Acked Read 7 Done Sent Done false
16365 5371 Acked Read 2 Done Sent Done false
16366 5372 Acked Flush 0 Done Sent Done false
16367 5373 Acked Read 1 Done Sent Done false
16368 5374 Acked Write 6 Done Sent Done false
16369 5375 Acked Flush 0 Done Sent Done false
16370 5376 Acked Read 5 Done Sent Done false
16371 5377 Acked Write 6 Done Sent Done false
16372 5378 Acked Write 3 Done Sent Done false
16373 5379 Acked Read 5 Done Sent Done false
16374 5380 Acked Read 5 Done Sent Done false
16375 5381 Acked Write 6 Done Sent Done false
16376 5382 Acked Read 9 Done Sent Done false
16377 5383 Acked Flush 0 Done Sent Done false
16378 5384 Acked Write 3 Done Sent Done false
16379 5385 Acked Flush 0 Done Sent Done false
16380 5386 Acked Write 5 Done Sent Done false
16381 5387 Acked Read 7 Done Sent Done false
16382 5388 Acked Write 8 Done Sent Done false
16383 5389 Acked Read 8 Done Sent Done false
16384 5390 Acked Read 6 Done Sent Done false
16385 5391 Acked Write 5 Done Sent Done false
16386 5392 Acked Read 7 Done Sent Done false
16387 5393 Acked Write 9 Done Sent Done false
16388 5394 Acked Read 6 Done Sent Done false
16389 5395 Acked Read 6 Done Sent Done false
16390 5396 Acked Write 6 Done Sent Done false
16391 5397 Acked Read 10 Done Sent Done false
16392 5398 Acked Write 2 Done Sent Done false
16393 5399 Acked Read 3 Done Sent Done false
16394 5400 Acked Flush 0 Done Sent Done false
16395 5401 Acked Read 2 Done Sent Done false
16396 5402 Acked Read 7 Done Sent Done false
16397 5403 Acked Write 6 Done Sent Done false
16398 5404 Acked Read 9 Done Sent Done false
16399 5405 Acked Read 3 Done Sent Done false
16400 5406 Acked Write 10 Done Sent Done false
16401 5407 Acked Write 1 Done Sent Done false
16402 5408 Acked Read 3 Done Sent Done false
16403 5409 Acked Write 7 Done Sent Done false
16404 5410 Acked Write 4 Done Sent Done false
16405 5411 Acked Read 5 Done Sent Done false
16406 5412 Acked Read 1 Done Sent Done false
16407 5413 Acked Flush 0 Done Sent Done false
16408 5414 Acked Write 7 Done Sent Done false
16409 5415 Acked Write 4 Done Sent Done false
16410 5416 Acked Read 3 Done Sent Done false
16411 5417 Acked Write 3 Done Sent Done false
16412 5418 Acked Read 3 Done Sent Done false
16413 5419 Acked Write 10 Done Sent Done false
16414 5420 Acked Read 2 Done Sent Done false
16415 5421 Acked Read 5 Done Sent Done false
16416 5422 Acked Flush 0 Done Sent Done false
16417 5423 Acked Read 3 Done Sent Done false
16418 5424 Acked Write 6 Done Sent Done false
16419 5425 Acked Write 8 Done Sent Done false
16420 5426 Acked Write 4 Done Sent Done false
16421 5427 Acked Write 2 Done Sent Done false
16422 5428 Acked Read 10 Done Sent Done false
16423 5429 Acked Flush 0 Done Sent Done false
16424 5430 Acked Read 2 Done Sent Done false
16425 5431 Acked Write 2 Done Sent Done false
16426 5432 Acked Write 10 Done Sent Done false
16427 5433 Acked Read 4 Done Sent Done false
16428 5434 Acked Read 9 Done Sent Done false
16429 5435 Acked Read 7 Done Sent Done false
16430 5436 Acked Write 1 Done Sent Done false
16431 5437 Acked Read 7 Done Sent Done false
16432 5438 Acked Read 3 Done Sent Done false
16433 5439 Acked Read 8 Done Sent Done false
16434 5440 Acked Read 8 Done Sent Done false
16435 5441 Acked Write 7 Done Sent Done false
16436 5442 Acked Read 2 Done Sent Done false
16437 5443 Acked Read 3 Done Sent Done false
16438 5444 Acked Read 9 Done Sent Done false
16439 5445 Acked Write 5 Done Sent Done false
16440 5446 Acked Write 9 Done Sent Done false
16441 5447 Acked Write 10 Done Sent Done false
16442 5448 Acked Read 3 Done Sent Done false
16443 5449 Acked Read 2 Done Sent Done false
16444 5450 Acked Write 4 Done Sent Done false
16445 5451 Acked Read 3 Done Sent Done false
16446 5452 Acked Write 2 Done Sent Done false
16447 5453 Acked Write 10 Done Sent Done false
16448 5454 Acked Read 7 Done Sent Done false
16449 5455 Acked Read 2 Done Sent Done false
16450 5456 Acked Read 2 Done Sent Done false
16451 5457 Acked Write 10 Done Sent Done false
16452 5458 Acked Read 2 Done Sent Done false
16453 5459 Acked Write 5 Done Sent Done false
16454 5460 Acked Write 8 Done Sent Done false
16455 5461 Acked Read 8 Done Sent Done false
16456 5462 Acked Write 7 Done Sent Done false
16457 5463 Acked Read 9 Done Sent Done false
16458 5464 Acked Flush 0 Done Sent Done false
16459 5465 Acked Write 1 Done Sent Done false
16460 5466 Acked Read 3 Done Sent Done false
16461 5467 Acked Read 3 Done Sent Done false
16462 5468 Acked Write 7 Done Sent Done false
16463 5469 Acked Write 7 Done Sent Done false
16464 5470 Acked Read 3 Done Sent Done false
16465 5471 Acked Read 3 Done Sent Done false
16466 5472 Acked Read 8 Done Sent Done false
16467 5473 Acked Read 10 Done Sent Done false
16468 5474 Acked Read 10 Done Sent Done false
16469 5475 Acked Read 8 Done Sent Done false
16470 5476 Acked Write 5 Done Sent Done false
16471 5477 Acked Read 4 Done Sent Done false
16472 5478 Acked Read 10 Done Sent Done false
16473 5479 Acked Read 1 Done Sent Done false
16474 5480 Acked Write 3 Done Sent Done false
16475 5481 Acked Read 9 Done Sent Done false
16476 5482 Acked Write 9 Done Sent Done false
16477 5483 Acked Read 10 Done Sent Done false
16478 5484 Acked Read 9 Done Sent Done false
16479 5485 Acked Write 9 Done Sent Done false
16480 5486 Acked Write 2 Done Sent Done false
16481 5487 Acked Write 1 Done Sent Done false
16482 5488 Acked Read 8 Done Sent Done false
16483 5489 Acked Read 6 Done Sent Done false
16484 5490 Acked Flush 0 Done Sent Done false
16485 5491 Acked Read 8 Done Sent Done false
16486 5492 Acked Read 3 Done Sent Done false
16487 5493 Acked Write 10 Done Sent Done false
16488 5494 Acked Read 9 Done Sent Done false
16489 5495 Acked Read 5 Done Sent Done false
16490 5496 Acked Write 1 Done Sent Done false
16491 5497 Acked Read 6 Done Sent Done false
16492 5498 Acked Read 7 Done Sent Done false
16493 5499 Acked Read 7 Done Sent Done false
16494 5500 Acked Read 10 Done Sent Done false
16495 5501 Acked Read 5 Done Sent Done false
16496 5502 Acked Read 4 Done Sent Done false
16497 5503 Acked Read 5 Done Sent Done false
16498 5504 Acked Write 8 Done Sent Done false
16499 5505 Acked Read 1 Done Sent Done false
16500 5506 Acked Flush 0 Done Sent Done false
16501 5507 Acked Write 10 Done Sent Done false
16502 5508 Acked Write 4 Done Sent Done false
16503 5509 Acked Read 4 Done Sent Done false
16504 5510 Acked Flush 0 Done Sent Done false
16505 5511 Acked Read 7 Done Sent Done false
16506 5512 Acked Write 8 Done Sent Done false
16507 5513 Acked Read 6 Done Sent Done false
16508 5514 Acked Read 6 Done Sent Done false
16509 5515 Acked Flush 0 Done Sent Done false
16510 5516 Acked Read 5 Done Sent Done false
16511 5517 Acked Read 9 Done Sent Done false
16512 5518 Acked Read 6 Done Sent Done false
16513 5519 Acked Write 7 Done Sent Done false
16514 5520 Acked Read 3 Done Sent Done false
16515 5521 Acked Read 7 Done Sent Done false
16516 5522 Acked Read 9 Done Sent Done false
16517 5523 Acked Write 2 Done Sent Done false
16518 5524 Acked Flush 0 Done Sent Done false
16519 5525 Acked Read 10 Done Sent Done false
16520 5526 Acked Read 5 Done Sent Done false
16521 5527 Acked Read 9 Done Sent Done false
16522 STATES DS:0 DS:1 DS:2 TOTAL
16523 Sent 0 301 0 301
16524 Done 301 0 301 602
16525 Skipped 0 0 0 0
16526 Error 0 0 0 0
16527 Last Flush: Some(JobId(5524)) Some(JobId(5226)) Some(JobId(5524))
16528 Downstairs last five completed: 5226 5225 5224 5223 5222
16529 Upstairs last five completed: 5527 5526 5525 5524 5523
16530 CLIENT: Up:0 ds:301 act:2
165312025-02-25T17:21:50.621ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
165322025-02-25T17:21:51.623ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
165332025-02-25T17:21:52.625ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
165342025-02-25T17:21:53.771ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
16535 ----------------------------------------------------------------
16536 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16537 States: Active Offline Active
16538 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16539 5227 Acked Write 9 Done Sent Done false
16540 5228 Acked Write 4 Done Sent Done false
16541 5229 Acked Write 7 Done Sent Done false
16542 5230 Acked Read 8 Done Sent Done false
16543 5231 Acked Write 3 Done Sent Done false
16544 5232 Acked Read 8 Done Sent Done false
16545 5233 Acked Write 2 Done Sent Done false
16546 5234 Acked Read 4 Done Sent Done false
16547 5235 Acked Write 7 Done Sent Done false
16548 5236 Acked Read 6 Done Sent Done false
16549 5237 Acked Flush 0 Done Sent Done false
16550 5238 Acked Read 3 Done Sent Done false
16551 5239 Acked Write 2 Done Sent Done false
16552 5240 Acked Write 2 Done Sent Done false
16553 5241 Acked Write 10 Done Sent Done false
16554 5242 Acked Write 10 Done Sent Done false
16555 5243 Acked Read 2 Done Sent Done false
16556 5244 Acked Write 5 Done Sent Done false
16557 5245 Acked Write 8 Done Sent Done false
16558 5246 Acked Write 10 Done Sent Done false
16559 5247 Acked Write 6 Done Sent Done false
16560 5248 Acked Write 7 Done Sent Done false
16561 5249 Acked Write 4 Done Sent Done false
16562 5250 Acked Flush 0 Done Sent Done false
16563 5251 Acked Write 10 Done Sent Done false
16564 5252 Acked Write 8 Done Sent Done false
16565 5253 Acked Flush 0 Done Sent Done false
16566 5254 Acked Read 2 Done Sent Done false
16567 5255 Acked Read 2 Done Sent Done false
16568 5256 Acked Read 6 Done Sent Done false
16569 5257 Acked Write 1 Done Sent Done false
16570 5258 Acked Write 9 Done Sent Done false
16571 5259 Acked Flush 0 Done Sent Done false
16572 5260 Acked Read 6 Done Sent Done false
16573 5261 Acked Write 3 Done Sent Done false
16574 5262 Acked Flush 0 Done Sent Done false
16575 5263 Acked Write 8 Done Sent Done false
16576 5264 Acked Write 1 Done Sent Done false
16577 5265 Acked Read 10 Done Sent Done false
16578 5266 Acked Write 9 Done Sent Done false
16579 5267 Acked Read 8 Done Sent Done false
16580 5268 Acked Write 4 Done Sent Done false
16581 5269 Acked Write 4 Done Sent Done false
16582 5270 Acked Write 5 Done Sent Done false
16583 5271 Acked Write 3 Done Sent Done false
16584 5272 Acked Write 10 Done Sent Done false
16585 5273 Acked Write 8 Done Sent Done false
16586 5274 Acked Write 4 Done Sent Done false
16587 5275 Acked Write 5 Done Sent Done false
16588 5276 Acked Flush 0 Done Sent Done false
16589 5277 Acked Read 6 Done Sent Done false
16590 5278 Acked Read 7 Done Sent Done false
16591 5279 Acked Write 6 Done Sent Done false
16592 5280 Acked Flush 0 Done Sent Done false
16593 5281 Acked Write 8 Done Sent Done false
16594 5282 Acked Read 7 Done Sent Done false
16595 5283 Acked Write 7 Done Sent Done false
16596 5284 Acked Read 1 Done Sent Done false
16597 5285 Acked Read 3 Done Sent Done false
16598 5286 Acked Read 5 Done Sent Done false
16599 5287 Acked Write 1 Done Sent Done false
16600 5288 Acked Write 4 Done Sent Done false
16601 5289 Acked Write 4 Done Sent Done false
16602 5290 Acked Write 9 Done Sent Done false
16603 5291 Acked Read 2 Done Sent Done false
16604 5292 Acked Read 1 Done Sent Done false
16605 5293 Acked Read 5 Done Sent Done false
16606 5294 Acked Read 5 Done Sent Done false
16607 5295 Acked Write 6 Done Sent Done false
16608 5296 Acked Write 6 Done Sent Done false
16609 5297 Acked Read 5 Done Sent Done false
16610 5298 Acked Read 7 Done Sent Done false
16611 5299 Acked Read 3 Done Sent Done false
16612 5300 Acked Write 4 Done Sent Done false
16613 5301 Acked Read 2 Done Sent Done false
16614 5302 Acked Read 2 Done Sent Done false
16615 5303 Acked Flush 0 Done Sent Done false
16616 5304 Acked Read 8 Done Sent Done false
16617 5305 Acked Read 10 Done Sent Done false
16618 5306 Acked Write 2 Done Sent Done false
16619 5307 Acked Write 9 Done Sent Done false
16620 5308 Acked Write 4 Done Sent Done false
16621 5309 Acked Read 4 Done Sent Done false
16622 5310 Acked Read 3 Done Sent Done false
16623 5311 Acked Write 1 Done Sent Done false
16624 5312 Acked Write 1 Done Sent Done false
16625 5313 Acked Read 6 Done Sent Done false
16626 5314 Acked Read 10 Done Sent Done false
16627 5315 Acked Write 5 Done Sent Done false
16628 5316 Acked Read 10 Done Sent Done false
16629 5317 Acked Read 9 Done Sent Done false
16630 5318 Acked Read 8 Done Sent Done false
16631 5319 Acked Write 10 Done Sent Done false
16632 5320 Acked Write 2 Done Sent Done false
16633 5321 Acked Read 1 Done Sent Done false
16634 5322 Acked Read 5 Done Sent Done false
16635 5323 Acked Read 1 Done Sent Done false
16636 5324 Acked Write 2 Done Sent Done false
16637 5325 Acked Read 4 Done Sent Done false
16638 5326 Acked Write 7 Done Sent Done false
16639 5327 Acked Write 1 Done Sent Done false
16640 5328 Acked Write 3 Done Sent Done false
16641 5329 Acked Read 7 Done Sent Done false
16642 5330 Acked Read 6 Done Sent Done false
16643 5331 Acked Read 10 Done Sent Done false
16644 5332 Acked Read 1 Done Sent Done false
16645 5333 Acked Read 10 Done Sent Done false
16646 5334 Acked Flush 0 Done Sent Done false
16647 5335 Acked Write 4 Done Sent Done false
16648 5336 Acked Write 10 Done Sent Done false
16649 5337 Acked Write 5 Done Sent Done false
16650 5338 Acked Write 3 Done Sent Done false
16651 5339 Acked Flush 0 Done Sent Done false
16652 5340 Acked Write 7 Done Sent Done false
16653 5341 Acked Read 2 Done Sent Done false
16654 5342 Acked Read 6 Done Sent Done false
16655 5343 Acked Write 9 Done Sent Done false
16656 5344 Acked Write 4 Done Sent Done false
16657 5345 Acked Read 4 Done Sent Done false
16658 5346 Acked Write 7 Done Sent Done false
16659 5347 Acked Read 1 Done Sent Done false
16660 5348 Acked Write 7 Done Sent Done false
16661 5349 Acked Read 3 Done Sent Done false
16662 5350 Acked Write 4 Done Sent Done false
16663 5351 Acked Write 1 Done Sent Done false
16664 5352 Acked Read 7 Done Sent Done false
16665 5353 Acked Read 3 Done Sent Done false
16666 5354 Acked Read 5 Done Sent Done false
16667 5355 Acked Write 4 Done Sent Done false
16668 5356 Acked Read 4 Done Sent Done false
16669 5357 Acked Write 6 Done Sent Done false
16670 5358 Acked Flush 0 Done Sent Done false
16671 5359 Acked Read 2 Done Sent Done false
16672 5360 Acked Read 6 Done Sent Done false
16673 5361 Acked Read 2 Done Sent Done false
16674 5362 Acked Read 7 Done Sent Done false
16675 5363 Acked Write 6 Done Sent Done false
16676 5364 Acked Write 5 Done Sent Done false
16677 5365 Acked Flush 0 Done Sent Done false
16678 5366 Acked Read 3 Done Sent Done false
16679 5367 Acked Flush 0 Done Sent Done false
16680 5368 Acked Flush 0 Done Sent Done false
16681 5369 Acked Write 2 Done Sent Done false
16682 5370 Acked Read 7 Done Sent Done false
16683 5371 Acked Read 2 Done Sent Done false
16684 5372 Acked Flush 0 Done Sent Done false
16685 5373 Acked Read 1 Done Sent Done false
16686 5374 Acked Write 6 Done Sent Done false
16687 5375 Acked Flush 0 Done Sent Done false
16688 5376 Acked Read 5 Done Sent Done false
16689 5377 Acked Write 6 Done Sent Done false
16690 5378 Acked Write 3 Done Sent Done false
16691 5379 Acked Read 5 Done Sent Done false
16692 5380 Acked Read 5 Done Sent Done false
16693 5381 Acked Write 6 Done Sent Done false
16694 5382 Acked Read 9 Done Sent Done false
16695 5383 Acked Flush 0 Done Sent Done false
16696 5384 Acked Write 3 Done Sent Done false
16697 5385 Acked Flush 0 Done Sent Done false
16698 5386 Acked Write 5 Done Sent Done false
16699 5387 Acked Read 7 Done Sent Done false
16700 5388 Acked Write 8 Done Sent Done false
16701 5389 Acked Read 8 Done Sent Done false
16702 5390 Acked Read 6 Done Sent Done false
16703 5391 Acked Write 5 Done Sent Done false
16704 5392 Acked Read 7 Done Sent Done false
16705 5393 Acked Write 9 Done Sent Done false
16706 5394 Acked Read 6 Done Sent Done false
16707 5395 Acked Read 6 Done Sent Done false
16708 5396 Acked Write 6 Done Sent Done false
16709 5397 Acked Read 10 Done Sent Done false
16710 5398 Acked Write 2 Done Sent Done false
16711 5399 Acked Read 3 Done Sent Done false
16712 5400 Acked Flush 0 Done Sent Done false
16713 5401 Acked Read 2 Done Sent Done false
16714 5402 Acked Read 7 Done Sent Done false
16715 5403 Acked Write 6 Done Sent Done false
16716 5404 Acked Read 9 Done Sent Done false
16717 5405 Acked Read 3 Done Sent Done false
16718 5406 Acked Write 10 Done Sent Done false
16719 5407 Acked Write 1 Done Sent Done false
16720 5408 Acked Read 3 Done Sent Done false
16721 5409 Acked Write 7 Done Sent Done false
16722 5410 Acked Write 4 Done Sent Done false
16723 5411 Acked Read 5 Done Sent Done false
16724 5412 Acked Read 1 Done Sent Done false
16725 5413 Acked Flush 0 Done Sent Done false
16726 5414 Acked Write 7 Done Sent Done false
16727 5415 Acked Write 4 Done Sent Done false
16728 5416 Acked Read 3 Done Sent Done false
16729 5417 Acked Write 3 Done Sent Done false
16730 5418 Acked Read 3 Done Sent Done false
16731 5419 Acked Write 10 Done Sent Done false
16732 5420 Acked Read 2 Done Sent Done false
16733 5421 Acked Read 5 Done Sent Done false
16734 5422 Acked Flush 0 Done Sent Done false
16735 5423 Acked Read 3 Done Sent Done false
16736 5424 Acked Write 6 Done Sent Done false
16737 5425 Acked Write 8 Done Sent Done false
16738 5426 Acked Write 4 Done Sent Done false
16739 5427 Acked Write 2 Done Sent Done false
16740 5428 Acked Read 10 Done Sent Done false
16741 5429 Acked Flush 0 Done Sent Done false
16742 5430 Acked Read 2 Done Sent Done false
16743 5431 Acked Write 2 Done Sent Done false
16744 5432 Acked Write 10 Done Sent Done false
16745 5433 Acked Read 4 Done Sent Done false
16746 5434 Acked Read 9 Done Sent Done false
16747 5435 Acked Read 7 Done Sent Done false
16748 5436 Acked Write 1 Done Sent Done false
16749 5437 Acked Read 7 Done Sent Done false
16750 5438 Acked Read 3 Done Sent Done false
16751 5439 Acked Read 8 Done Sent Done false
16752 5440 Acked Read 8 Done Sent Done false
16753 5441 Acked Write 7 Done Sent Done false
16754 5442 Acked Read 2 Done Sent Done false
16755 5443 Acked Read 3 Done Sent Done false
16756 5444 Acked Read 9 Done Sent Done false
16757 5445 Acked Write 5 Done Sent Done false
16758 5446 Acked Write 9 Done Sent Done false
16759 5447 Acked Write 10 Done Sent Done false
16760 5448 Acked Read 3 Done Sent Done false
16761 5449 Acked Read 2 Done Sent Done false
16762 5450 Acked Write 4 Done Sent Done false
16763 5451 Acked Read 3 Done Sent Done false
16764 5452 Acked Write 2 Done Sent Done false
16765 5453 Acked Write 10 Done Sent Done false
16766 5454 Acked Read 7 Done Sent Done false
16767 5455 Acked Read 2 Done Sent Done false
16768 5456 Acked Read 2 Done Sent Done false
16769 5457 Acked Write 10 Done Sent Done false
16770 5458 Acked Read 2 Done Sent Done false
16771 5459 Acked Write 5 Done Sent Done false
16772 5460 Acked Write 8 Done Sent Done false
16773 5461 Acked Read 8 Done Sent Done false
16774 5462 Acked Write 7 Done Sent Done false
16775 5463 Acked Read 9 Done Sent Done false
16776 5464 Acked Flush 0 Done Sent Done false
16777 5465 Acked Write 1 Done Sent Done false
16778 5466 Acked Read 3 Done Sent Done false
16779 5467 Acked Read 3 Done Sent Done false
16780 5468 Acked Write 7 Done Sent Done false
16781 5469 Acked Write 7 Done Sent Done false
16782 5470 Acked Read 3 Done Sent Done false
16783 5471 Acked Read 3 Done Sent Done false
16784 5472 Acked Read 8 Done Sent Done false
16785 5473 Acked Read 10 Done Sent Done false
16786 5474 Acked Read 10 Done Sent Done false
16787 5475 Acked Read 8 Done Sent Done false
16788 5476 Acked Write 5 Done Sent Done false
16789 5477 Acked Read 4 Done Sent Done false
16790 5478 Acked Read 10 Done Sent Done false
16791 5479 Acked Read 1 Done Sent Done false
16792 5480 Acked Write 3 Done Sent Done false
16793 5481 Acked Read 9 Done Sent Done false
16794 5482 Acked Write 9 Done Sent Done false
16795 5483 Acked Read 10 Done Sent Done false
16796 5484 Acked Read 9 Done Sent Done false
16797 5485 Acked Write 9 Done Sent Done false
16798 5486 Acked Write 2 Done Sent Done false
16799 5487 Acked Write 1 Done Sent Done false
16800 5488 Acked Read 8 Done Sent Done false
16801 5489 Acked Read 6 Done Sent Done false
16802 5490 Acked Flush 0 Done Sent Done false
16803 5491 Acked Read 8 Done Sent Done false
16804 5492 Acked Read 3 Done Sent Done false
16805 5493 Acked Write 10 Done Sent Done false
16806 5494 Acked Read 9 Done Sent Done false
16807 5495 Acked Read 5 Done Sent Done false
16808 5496 Acked Write 1 Done Sent Done false
16809 5497 Acked Read 6 Done Sent Done false
16810 5498 Acked Read 7 Done Sent Done false
16811 5499 Acked Read 7 Done Sent Done false
16812 5500 Acked Read 10 Done Sent Done false
16813 5501 Acked Read 5 Done Sent Done false
16814 5502 Acked Read 4 Done Sent Done false
16815 5503 Acked Read 5 Done Sent Done false
16816 5504 Acked Write 8 Done Sent Done false
16817 5505 Acked Read 1 Done Sent Done false
16818 5506 Acked Flush 0 Done Sent Done false
16819 5507 Acked Write 10 Done Sent Done false
16820 5508 Acked Write 4 Done Sent Done false
16821 5509 Acked Read 4 Done Sent Done false
16822 5510 Acked Flush 0 Done Sent Done false
16823 5511 Acked Read 7 Done Sent Done false
16824 5512 Acked Write 8 Done Sent Done false
16825 5513 Acked Read 6 Done Sent Done false
16826 5514 Acked Read 6 Done Sent Done false
16827 5515 Acked Flush 0 Done Sent Done false
16828 5516 Acked Read 5 Done Sent Done false
16829 5517 Acked Read 9 Done Sent Done false
16830 5518 Acked Read 6 Done Sent Done false
16831 5519 Acked Write 7 Done Sent Done false
16832 5520 Acked Read 3 Done Sent Done false
16833 5521 Acked Read 7 Done Sent Done false
16834 5522 Acked Read 9 Done Sent Done false
16835 5523 Acked Write 2 Done Sent Done false
16836 5524 Acked Flush 0 Done Sent Done false
16837 5525 Acked Read 10 Done Sent Done false
16838 5526 Acked Read 5 Done Sent Done false
16839 5527 Acked Read 9 Done Sent Done false
16840 STATES DS:0 DS:1 DS:2 TOTAL
16841 Sent 0 301 0 301
16842 Done 301 0 301 602
16843 Skipped 0 0 0 0
16844 Error 0 0 0 0
16845 Last Flush: Some(JobId(5524)) Some(JobId(5226)) Some(JobId(5524))
16846 Downstairs last five completed: 5226 5225 5224 5223 5222
16847 Upstairs last five completed: 5527 5526 5525 5524 5523
16848 CLIENT: Up:0 ds:301 act:2
168492025-02-25T17:21:54.628ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
168502025-02-25T17:21:55.534ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
168512025-02-25T17:21:55.534ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
168522025-02-25T17:21:55.534ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
168532025-02-25T17:21:55.534ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
168542025-02-25T17:21:55.534ZINFOcrucible: client task is exiting = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
16855 [8820] got start action
16856 [8820] I am going to restart
16857 Starting downstairs at port 8820
16858 Make output file at "/tmp/dsc/downstairs-8820.txt"
16859 [8820][1] reports Starting
16860 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1223)
16861 [8820][1] reports Running
168622025-02-25T17:21:56.632ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
168632025-02-25T17:21:57.633ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
16864 ----------------------------------------------------------------
16865 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16866 States: Active Offline Active
16867 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16868 5227 Acked Write 9 Done Sent Done false
16869 5228 Acked Write 4 Done Sent Done false
16870 5229 Acked Write 7 Done Sent Done false
16871 5230 Acked Read 8 Done Sent Done false
16872 5231 Acked Write 3 Done Sent Done false
16873 5232 Acked Read 8 Done Sent Done false
16874 5233 Acked Write 2 Done Sent Done false
16875 5234 Acked Read 4 Done Sent Done false
16876 5235 Acked Write 7 Done Sent Done false
16877 5236 Acked Read 6 Done Sent Done false
16878 5237 Acked Flush 0 Done Sent Done false
16879 5238 Acked Read 3 Done Sent Done false
16880 5239 Acked Write 2 Done Sent Done false
16881 5240 Acked Write 2 Done Sent Done false
16882 5241 Acked Write 10 Done Sent Done false
16883 5242 Acked Write 10 Done Sent Done false
16884 5243 Acked Read 2 Done Sent Done false
16885 5244 Acked Write 5 Done Sent Done false
16886 5245 Acked Write 8 Done Sent Done false
16887 5246 Acked Write 10 Done Sent Done false
16888 5247 Acked Write 6 Done Sent Done false
16889 5248 Acked Write 7 Done Sent Done false
16890 5249 Acked Write 4 Done Sent Done false
16891 5250 Acked Flush 0 Done Sent Done false
16892 5251 Acked Write 10 Done Sent Done false
16893 5252 Acked Write 8 Done Sent Done false
16894 5253 Acked Flush 0 Done Sent Done false
16895 5254 Acked Read 2 Done Sent Done false
16896 5255 Acked Read 2 Done Sent Done false
16897 5256 Acked Read 6 Done Sent Done false
16898 5257 Acked Write 1 Done Sent Done false
16899 5258 Acked Write 9 Done Sent Done false
16900 5259 Acked Flush 0 Done Sent Done false
16901 5260 Acked Read 6 Done Sent Done false
16902 5261 Acked Write 3 Done Sent Done false
16903 5262 Acked Flush 0 Done Sent Done false
16904 5263 Acked Write 8 Done Sent Done false
16905 5264 Acked Write 1 Done Sent Done false
16906 5265 Acked Read 10 Done Sent Done false
16907 5266 Acked Write 9 Done Sent Done false
16908 5267 Acked Read 8 Done Sent Done false
16909 5268 Acked Write 4 Done Sent Done false
16910 5269 Acked Write 4 Done Sent Done false
16911 5270 Acked Write 5 Done Sent Done false
16912 5271 Acked Write 3 Done Sent Done false
16913 5272 Acked Write 10 Done Sent Done false
16914 5273 Acked Write 8 Done Sent Done false
16915 5274 Acked Write 4 Done Sent Done false
16916 5275 Acked Write 5 Done Sent Done false
16917 5276 Acked Flush 0 Done Sent Done false
16918 5277 Acked Read 6 Done Sent Done false
16919 5278 Acked Read 7 Done Sent Done false
16920 5279 Acked Write 6 Done Sent Done false
16921 5280 Acked Flush 0 Done Sent Done false
16922 5281 Acked Write 8 Done Sent Done false
16923 5282 Acked Read 7 Done Sent Done false
16924 5283 Acked Write 7 Done Sent Done false
16925 5284 Acked Read 1 Done Sent Done false
16926 5285 Acked Read 3 Done Sent Done false
16927 5286 Acked Read 5 Done Sent Done false
16928 5287 Acked Write 1 Done Sent Done false
16929 5288 Acked Write 4 Done Sent Done false
16930 5289 Acked Write 4 Done Sent Done false
16931 5290 Acked Write 9 Done Sent Done false
16932 5291 Acked Read 2 Done Sent Done false
16933 5292 Acked Read 1 Done Sent Done false
16934 5293 Acked Read 5 Done Sent Done false
16935 5294 Acked Read 5 Done Sent Done false
16936 5295 Acked Write 6 Done Sent Done false
16937 5296 Acked Write 6 Done Sent Done false
16938 5297 Acked Read 5 Done Sent Done false
16939 5298 Acked Read 7 Done Sent Done false
16940 5299 Acked Read 3 Done Sent Done false
16941 5300 Acked Write 4 Done Sent Done false
16942 5301 Acked Read 2 Done Sent Done false
16943 5302 Acked Read 2 Done Sent Done false
16944 5303 Acked Flush 0 Done Sent Done false
16945 5304 Acked Read 8 Done Sent Done false
16946 5305 Acked Read 10 Done Sent Done false
16947 5306 Acked Write 2 Done Sent Done false
16948 5307 Acked Write 9 Done Sent Done false
16949 5308 Acked Write 4 Done Sent Done false
16950 5309 Acked Read 4 Done Sent Done false
16951 5310 Acked Read 3 Done Sent Done false
16952 5311 Acked Write 1 Done Sent Done false
16953 5312 Acked Write 1 Done Sent Done false
16954 5313 Acked Read 6 Done Sent Done false
16955 5314 Acked Read 10 Done Sent Done false
16956 5315 Acked Write 5 Done Sent Done false
16957 5316 Acked Read 10 Done Sent Done false
16958 5317 Acked Read 9 Done Sent Done false
16959 5318 Acked Read 8 Done Sent Done false
16960 5319 Acked Write 10 Done Sent Done false
16961 5320 Acked Write 2 Done Sent Done false
16962 5321 Acked Read 1 Done Sent Done false
16963 5322 Acked Read 5 Done Sent Done false
16964 5323 Acked Read 1 Done Sent Done false
16965 5324 Acked Write 2 Done Sent Done false
16966 5325 Acked Read 4 Done Sent Done false
16967 5326 Acked Write 7 Done Sent Done false
16968 5327 Acked Write 1 Done Sent Done false
16969 5328 Acked Write 3 Done Sent Done false
16970 5329 Acked Read 7 Done Sent Done false
16971 5330 Acked Read 6 Done Sent Done false
16972 5331 Acked Read 10 Done Sent Done false
16973 5332 Acked Read 1 Done Sent Done false
16974 5333 Acked Read 10 Done Sent Done false
16975 5334 Acked Flush 0 Done Sent Done false
16976 5335 Acked Write 4 Done Sent Done false
16977 5336 Acked Write 10 Done Sent Done false
16978 5337 Acked Write 5 Done Sent Done false
16979 5338 Acked Write 3 Done Sent Done false
16980 5339 Acked Flush 0 Done Sent Done false
16981 5340 Acked Write 7 Done Sent Done false
16982 5341 Acked Read 2 Done Sent Done false
16983 5342 Acked Read 6 Done Sent Done false
16984 5343 Acked Write 9 Done Sent Done false
16985 5344 Acked Write 4 Done Sent Done false
16986 5345 Acked Read 4 Done Sent Done false
16987 5346 Acked Write 7 Done Sent Done false
16988 5347 Acked Read 1 Done Sent Done false
16989 5348 Acked Write 7 Done Sent Done false
16990 5349 Acked Read 3 Done Sent Done false
16991 5350 Acked Write 4 Done Sent Done false
16992 5351 Acked Write 1 Done Sent Done false
16993 5352 Acked Read 7 Done Sent Done false
16994 5353 Acked Read 3 Done Sent Done false
16995 5354 Acked Read 5 Done Sent Done false
16996 5355 Acked Write 4 Done Sent Done false
16997 5356 Acked Read 4 Done Sent Done false
16998 5357 Acked Write 6 Done Sent Done false
16999 5358 Acked Flush 0 Done Sent Done false
17000 5359 Acked Read 2 Done Sent Done false
17001 5360 Acked Read 6 Done Sent Done false
17002 5361 Acked Read 2 Done Sent Done false
17003 5362 Acked Read 7 Done Sent Done false
17004 5363 Acked Write 6 Done Sent Done false
17005 5364 Acked Write 5 Done Sent Done false
17006 5365 Acked Flush 0 Done Sent Done false
17007 5366 Acked Read 3 Done Sent Done false
17008 5367 Acked Flush 0 Done Sent Done false
17009 5368 Acked Flush 0 Done Sent Done false
17010 5369 Acked Write 2 Done Sent Done false
17011 5370 Acked Read 7 Done Sent Done false
17012 5371 Acked Read 2 Done Sent Done false
17013 5372 Acked Flush 0 Done Sent Done false
17014 5373 Acked Read 1 Done Sent Done false
17015 5374 Acked Write 6 Done Sent Done false
17016 5375 Acked Flush 0 Done Sent Done false
17017 5376 Acked Read 5 Done Sent Done false
17018 5377 Acked Write 6 Done Sent Done false
17019 5378 Acked Write 3 Done Sent Done false
17020 5379 Acked Read 5 Done Sent Done false
17021 5380 Acked Read 5 Done Sent Done false
17022 5381 Acked Write 6 Done Sent Done false
17023 5382 Acked Read 9 Done Sent Done false
17024 5383 Acked Flush 0 Done Sent Done false
17025 5384 Acked Write 3 Done Sent Done false
17026 5385 Acked Flush 0 Done Sent Done false
17027 5386 Acked Write 5 Done Sent Done false
17028 5387 Acked Read 7 Done Sent Done false
17029 5388 Acked Write 8 Done Sent Done false
17030 5389 Acked Read 8 Done Sent Done false
17031 5390 Acked Read 6 Done Sent Done false
17032 5391 Acked Write 5 Done Sent Done false
17033 5392 Acked Read 7 Done Sent Done false
17034 5393 Acked Write 9 Done Sent Done false
17035 5394 Acked Read 6 Done Sent Done false
17036 5395 Acked Read 6 Done Sent Done false
17037 5396 Acked Write 6 Done Sent Done false
17038 5397 Acked Read 10 Done Sent Done false
17039 5398 Acked Write 2 Done Sent Done false
17040 5399 Acked Read 3 Done Sent Done false
17041 5400 Acked Flush 0 Done Sent Done false
17042 5401 Acked Read 2 Done Sent Done false
17043 5402 Acked Read 7 Done Sent Done false
17044 5403 Acked Write 6 Done Sent Done false
17045 5404 Acked Read 9 Done Sent Done false
17046 5405 Acked Read 3 Done Sent Done false
17047 5406 Acked Write 10 Done Sent Done false
17048 5407 Acked Write 1 Done Sent Done false
17049 5408 Acked Read 3 Done Sent Done false
17050 5409 Acked Write 7 Done Sent Done false
17051 5410 Acked Write 4 Done Sent Done false
17052 5411 Acked Read 5 Done Sent Done false
17053 5412 Acked Read 1 Done Sent Done false
17054 5413 Acked Flush 0 Done Sent Done false
17055 5414 Acked Write 7 Done Sent Done false
17056 5415 Acked Write 4 Done Sent Done false
17057 5416 Acked Read 3 Done Sent Done false
17058 5417 Acked Write 3 Done Sent Done false
17059 5418 Acked Read 3 Done Sent Done false
17060 5419 Acked Write 10 Done Sent Done false
17061 5420 Acked Read 2 Done Sent Done false
17062 5421 Acked Read 5 Done Sent Done false
17063 5422 Acked Flush 0 Done Sent Done false
17064 5423 Acked Read 3 Done Sent Done false
17065 5424 Acked Write 6 Done Sent Done false
17066 5425 Acked Write 8 Done Sent Done false
17067 5426 Acked Write 4 Done Sent Done false
17068 5427 Acked Write 2 Done Sent Done false
17069 5428 Acked Read 10 Done Sent Done false
17070 5429 Acked Flush 0 Done Sent Done false
17071 5430 Acked Read 2 Done Sent Done false
17072 5431 Acked Write 2 Done Sent Done false
17073 5432 Acked Write 10 Done Sent Done false
17074 5433 Acked Read 4 Done Sent Done false
17075 5434 Acked Read 9 Done Sent Done false
17076 5435 Acked Read 7 Done Sent Done false
17077 5436 Acked Write 1 Done Sent Done false
17078 5437 Acked Read 7 Done Sent Done false
17079 5438 Acked Read 3 Done Sent Done false
17080 5439 Acked Read 8 Done Sent Done false
17081 5440 Acked Read 8 Done Sent Done false
17082 5441 Acked Write 7 Done Sent Done false
17083 5442 Acked Read 2 Done Sent Done false
17084 5443 Acked Read 3 Done Sent Done false
17085 5444 Acked Read 9 Done Sent Done false
17086 5445 Acked Write 5 Done Sent Done false
17087 5446 Acked Write 9 Done Sent Done false
17088 5447 Acked Write 10 Done Sent Done false
17089 5448 Acked Read 3 Done Sent Done false
17090 5449 Acked Read 2 Done Sent Done false
17091 5450 Acked Write 4 Done Sent Done false
17092 5451 Acked Read 3 Done Sent Done false
17093 5452 Acked Write 2 Done Sent Done false
17094 5453 Acked Write 10 Done Sent Done false
17095 5454 Acked Read 7 Done Sent Done false
17096 5455 Acked Read 2 Done Sent Done false
17097 5456 Acked Read 2 Done Sent Done false
17098 5457 Acked Write 10 Done Sent Done false
17099 5458 Acked Read 2 Done Sent Done false
17100 5459 Acked Write 5 Done Sent Done false
17101 5460 Acked Write 8 Done Sent Done false
17102 5461 Acked Read 8 Done Sent Done false
17103 5462 Acked Write 7 Done Sent Done false
17104 5463 Acked Read 9 Done Sent Done false
17105 5464 Acked Flush 0 Done Sent Done false
17106 5465 Acked Write 1 Done Sent Done false
17107 5466 Acked Read 3 Done Sent Done false
17108 5467 Acked Read 3 Done Sent Done false
17109 5468 Acked Write 7 Done Sent Done false
17110 5469 Acked Write 7 Done Sent Done false
17111 5470 Acked Read 3 Done Sent Done false
17112 5471 Acked Read 3 Done Sent Done false
17113 5472 Acked Read 8 Done Sent Done false
17114 5473 Acked Read 10 Done Sent Done false
17115 5474 Acked Read 10 Done Sent Done false
17116 5475 Acked Read 8 Done Sent Done false
17117 5476 Acked Write 5 Done Sent Done false
17118 5477 Acked Read 4 Done Sent Done false
17119 5478 Acked Read 10 Done Sent Done false
17120 5479 Acked Read 1 Done Sent Done false
17121 5480 Acked Write 3 Done Sent Done false
17122 5481 Acked Read 9 Done Sent Done false
17123 5482 Acked Write 9 Done Sent Done false
17124 5483 Acked Read 10 Done Sent Done false
17125 5484 Acked Read 9 Done Sent Done false
17126 5485 Acked Write 9 Done Sent Done false
17127 5486 Acked Write 2 Done Sent Done false
17128 5487 Acked Write 1 Done Sent Done false
17129 5488 Acked Read 8 Done Sent Done false
17130 5489 Acked Read 6 Done Sent Done false
17131 5490 Acked Flush 0 Done Sent Done false
17132 5491 Acked Read 8 Done Sent Done false
17133 5492 Acked Read 3 Done Sent Done false
17134 5493 Acked Write 10 Done Sent Done false
17135 5494 Acked Read 9 Done Sent Done false
17136 5495 Acked Read 5 Done Sent Done false
17137 5496 Acked Write 1 Done Sent Done false
17138 5497 Acked Read 6 Done Sent Done false
17139 5498 Acked Read 7 Done Sent Done false
17140 5499 Acked Read 7 Done Sent Done false
17141 5500 Acked Read 10 Done Sent Done false
17142 5501 Acked Read 5 Done Sent Done false
17143 5502 Acked Read 4 Done Sent Done false
17144 5503 Acked Read 5 Done Sent Done false
17145 5504 Acked Write 8 Done Sent Done false
17146 5505 Acked Read 1 Done Sent Done false
17147 5506 Acked Flush 0 Done Sent Done false
17148 5507 Acked Write 10 Done Sent Done false
17149 5508 Acked Write 4 Done Sent Done false
17150 5509 Acked Read 4 Done Sent Done false
17151 5510 Acked Flush 0 Done Sent Done false
17152 5511 Acked Read 7 Done Sent Done false
17153 5512 Acked Write 8 Done Sent Done false
17154 5513 Acked Read 6 Done Sent Done false
17155 5514 Acked Read 6 Done Sent Done false
17156 5515 Acked Flush 0 Done Sent Done false
17157 5516 Acked Read 5 Done Sent Done false
17158 5517 Acked Read 9 Done Sent Done false
17159 5518 Acked Read 6 Done Sent Done false
17160 5519 Acked Write 7 Done Sent Done false
17161 5520 Acked Read 3 Done Sent Done false
17162 5521 Acked Read 7 Done Sent Done false
17163 5522 Acked Read 9 Done Sent Done false
17164 5523 Acked Write 2 Done Sent Done false
17165 5524 Acked Flush 0 Done Sent Done false
17166 5525 Acked Read 10 Done Sent Done false
17167 5526 Acked Read 5 Done Sent Done false
17168 5527 Acked Read 9 Done Sent Done false
17169 STATES DS:0 DS:1 DS:2 TOTAL
17170 Sent 0 301 0 301
17171 Done 301 0 301 602
17172 Skipped 0 0 0 0
17173 Error 0 0 0 0
17174 Last Flush: Some(JobId(5524)) Some(JobId(5226)) Some(JobId(5524))
17175 Downstairs last five completed: 5226 5225 5224 5223 5222
17176 Upstairs last five completed: 5527 5526 5525 5524 5523
17177 CLIENT: Up:0 ds:301 act:2
171782025-02-25T17:21:58.772ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
171792025-02-25T17:21:59.635ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
171802025-02-25T17:22:00.637ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
171812025-02-25T17:22:01.639ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
17182 ----------------------------------------------------------------
17183 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17184 States: Active Offline Active
17185 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17186 5227 Acked Write 9 Done Sent Done false
17187 5228 Acked Write 4 Done Sent Done false
17188 5229 Acked Write 7 Done Sent Done false
17189 5230 Acked Read 8 Done Sent Done false
17190 5231 Acked Write 3 Done Sent Done false
17191 5232 Acked Read 8 Done Sent Done false
17192 5233 Acked Write 2 Done Sent Done false
17193 5234 Acked Read 4 Done Sent Done false
17194 5235 Acked Write 7 Done Sent Done false
17195 5236 Acked Read 6 Done Sent Done false
17196 5237 Acked Flush 0 Done Sent Done false
17197 5238 Acked Read 3 Done Sent Done false
17198 5239 Acked Write 2 Done Sent Done false
17199 5240 Acked Write 2 Done Sent Done false
17200 5241 Acked Write 10 Done Sent Done false
17201 5242 Acked Write 10 Done Sent Done false
17202 5243 Acked Read 2 Done Sent Done false
17203 5244 Acked Write 5 Done Sent Done false
17204 5245 Acked Write 8 Done Sent Done false
17205 5246 Acked Write 10 Done Sent Done false
17206 5247 Acked Write 6 Done Sent Done false
17207 5248 Acked Write 7 Done Sent Done false
17208 5249 Acked Write 4 Done Sent Done false
17209 5250 Acked Flush 0 Done Sent Done false
17210 5251 Acked Write 10 Done Sent Done false
17211 5252 Acked Write 8 Done Sent Done false
17212 5253 Acked Flush 0 Done Sent Done false
17213 5254 Acked Read 2 Done Sent Done false
17214 5255 Acked Read 2 Done Sent Done false
17215 5256 Acked Read 6 Done Sent Done false
17216 5257 Acked Write 1 Done Sent Done false
17217 5258 Acked Write 9 Done Sent Done false
17218 5259 Acked Flush 0 Done Sent Done false
17219 5260 Acked Read 6 Done Sent Done false
17220 5261 Acked Write 3 Done Sent Done false
17221 5262 Acked Flush 0 Done Sent Done false
17222 5263 Acked Write 8 Done Sent Done false
17223 5264 Acked Write 1 Done Sent Done false
17224 5265 Acked Read 10 Done Sent Done false
17225 5266 Acked Write 9 Done Sent Done false
17226 5267 Acked Read 8 Done Sent Done false
17227 5268 Acked Write 4 Done Sent Done false
17228 5269 Acked Write 4 Done Sent Done false
17229 5270 Acked Write 5 Done Sent Done false
17230 5271 Acked Write 3 Done Sent Done false
17231 5272 Acked Write 10 Done Sent Done false
17232 5273 Acked Write 8 Done Sent Done false
17233 5274 Acked Write 4 Done Sent Done false
17234 5275 Acked Write 5 Done Sent Done false
17235 5276 Acked Flush 0 Done Sent Done false
17236 5277 Acked Read 6 Done Sent Done false
17237 5278 Acked Read 7 Done Sent Done false
17238 5279 Acked Write 6 Done Sent Done false
17239 5280 Acked Flush 0 Done Sent Done false
17240 5281 Acked Write 8 Done Sent Done false
17241 5282 Acked Read 7 Done Sent Done false
17242 5283 Acked Write 7 Done Sent Done false
17243 5284 Acked Read 1 Done Sent Done false
17244 5285 Acked Read 3 Done Sent Done false
17245 5286 Acked Read 5 Done Sent Done false
17246 5287 Acked Write 1 Done Sent Done false
17247 5288 Acked Write 4 Done Sent Done false
17248 5289 Acked Write 4 Done Sent Done false
17249 5290 Acked Write 9 Done Sent Done false
17250 5291 Acked Read 2 Done Sent Done false
17251 5292 Acked Read 1 Done Sent Done false
17252 5293 Acked Read 5 Done Sent Done false
17253 5294 Acked Read 5 Done Sent Done false
17254 5295 Acked Write 6 Done Sent Done false
17255 5296 Acked Write 6 Done Sent Done false
17256 5297 Acked Read 5 Done Sent Done false
17257 5298 Acked Read 7 Done Sent Done false
17258 5299 Acked Read 3 Done Sent Done false
17259 5300 Acked Write 4 Done Sent Done false
17260 5301 Acked Read 2 Done Sent Done false
17261 5302 Acked Read 2 Done Sent Done false
17262 5303 Acked Flush 0 Done Sent Done false
17263 5304 Acked Read 8 Done Sent Done false
17264 5305 Acked Read 10 Done Sent Done false
17265 5306 Acked Write 2 Done Sent Done false
17266 5307 Acked Write 9 Done Sent Done false
17267 5308 Acked Write 4 Done Sent Done false
17268 5309 Acked Read 4 Done Sent Done false
17269 5310 Acked Read 3 Done Sent Done false
17270 5311 Acked Write 1 Done Sent Done false
17271 5312 Acked Write 1 Done Sent Done false
17272 5313 Acked Read 6 Done Sent Done false
17273 5314 Acked Read 10 Done Sent Done false
17274 5315 Acked Write 5 Done Sent Done false
17275 5316 Acked Read 10 Done Sent Done false
17276 5317 Acked Read 9 Done Sent Done false
17277 5318 Acked Read 8 Done Sent Done false
17278 5319 Acked Write 10 Done Sent Done false
17279 5320 Acked Write 2 Done Sent Done false
17280 5321 Acked Read 1 Done Sent Done false
17281 5322 Acked Read 5 Done Sent Done false
17282 5323 Acked Read 1 Done Sent Done false
17283 5324 Acked Write 2 Done Sent Done false
17284 5325 Acked Read 4 Done Sent Done false
17285 5326 Acked Write 7 Done Sent Done false
17286 5327 Acked Write 1 Done Sent Done false
17287 5328 Acked Write 3 Done Sent Done false
17288 5329 Acked Read 7 Done Sent Done false
17289 5330 Acked Read 6 Done Sent Done false
17290 5331 Acked Read 10 Done Sent Done false
17291 5332 Acked Read 1 Done Sent Done false
17292 5333 Acked Read 10 Done Sent Done false
17293 5334 Acked Flush 0 Done Sent Done false
17294 5335 Acked Write 4 Done Sent Done false
17295 5336 Acked Write 10 Done Sent Done false
17296 5337 Acked Write 5 Done Sent Done false
17297 5338 Acked Write 3 Done Sent Done false
17298 5339 Acked Flush 0 Done Sent Done false
17299 5340 Acked Write 7 Done Sent Done false
17300 5341 Acked Read 2 Done Sent Done false
17301 5342 Acked Read 6 Done Sent Done false
17302 5343 Acked Write 9 Done Sent Done false
17303 5344 Acked Write 4 Done Sent Done false
17304 5345 Acked Read 4 Done Sent Done false
17305 5346 Acked Write 7 Done Sent Done false
17306 5347 Acked Read 1 Done Sent Done false
17307 5348 Acked Write 7 Done Sent Done false
17308 5349 Acked Read 3 Done Sent Done false
17309 5350 Acked Write 4 Done Sent Done false
17310 5351 Acked Write 1 Done Sent Done false
17311 5352 Acked Read 7 Done Sent Done false
17312 5353 Acked Read 3 Done Sent Done false
17313 5354 Acked Read 5 Done Sent Done false
17314 5355 Acked Write 4 Done Sent Done false
17315 5356 Acked Read 4 Done Sent Done false
17316 5357 Acked Write 6 Done Sent Done false
17317 5358 Acked Flush 0 Done Sent Done false
17318 5359 Acked Read 2 Done Sent Done false
17319 5360 Acked Read 6 Done Sent Done false
17320 5361 Acked Read 2 Done Sent Done false
17321 5362 Acked Read 7 Done Sent Done false
17322 5363 Acked Write 6 Done Sent Done false
17323 5364 Acked Write 5 Done Sent Done false
17324 5365 Acked Flush 0 Done Sent Done false
17325 5366 Acked Read 3 Done Sent Done false
17326 5367 Acked Flush 0 Done Sent Done false
17327 5368 Acked Flush 0 Done Sent Done false
17328 5369 Acked Write 2 Done Sent Done false
17329 5370 Acked Read 7 Done Sent Done false
17330 5371 Acked Read 2 Done Sent Done false
17331 5372 Acked Flush 0 Done Sent Done false
17332 5373 Acked Read 1 Done Sent Done false
17333 5374 Acked Write 6 Done Sent Done false
17334 5375 Acked Flush 0 Done Sent Done false
17335 5376 Acked Read 5 Done Sent Done false
17336 5377 Acked Write 6 Done Sent Done false
17337 5378 Acked Write 3 Done Sent Done false
17338 5379 Acked Read 5 Done Sent Done false
17339 5380 Acked Read 5 Done Sent Done false
17340 5381 Acked Write 6 Done Sent Done false
17341 5382 Acked Read 9 Done Sent Done false
17342 5383 Acked Flush 0 Done Sent Done false
17343 5384 Acked Write 3 Done Sent Done false
17344 5385 Acked Flush 0 Done Sent Done false
17345 5386 Acked Write 5 Done Sent Done false
17346 5387 Acked Read 7 Done Sent Done false
17347 5388 Acked Write 8 Done Sent Done false
17348 5389 Acked Read 8 Done Sent Done false
17349 5390 Acked Read 6 Done Sent Done false
17350 5391 Acked Write 5 Done Sent Done false
17351 5392 Acked Read 7 Done Sent Done false
17352 5393 Acked Write 9 Done Sent Done false
17353 5394 Acked Read 6 Done Sent Done false
17354 5395 Acked Read 6 Done Sent Done false
17355 5396 Acked Write 6 Done Sent Done false
17356 5397 Acked Read 10 Done Sent Done false
17357 5398 Acked Write 2 Done Sent Done false
17358 5399 Acked Read 3 Done Sent Done false
17359 5400 Acked Flush 0 Done Sent Done false
17360 5401 Acked Read 2 Done Sent Done false
17361 5402 Acked Read 7 Done Sent Done false
17362 5403 Acked Write 6 Done Sent Done false
17363 5404 Acked Read 9 Done Sent Done false
17364 5405 Acked Read 3 Done Sent Done false
17365 5406 Acked Write 10 Done Sent Done false
17366 5407 Acked Write 1 Done Sent Done false
17367 5408 Acked Read 3 Done Sent Done false
17368 5409 Acked Write 7 Done Sent Done false
17369 5410 Acked Write 4 Done Sent Done false
17370 5411 Acked Read 5 Done Sent Done false
17371 5412 Acked Read 1 Done Sent Done false
17372 5413 Acked Flush 0 Done Sent Done false
17373 5414 Acked Write 7 Done Sent Done false
17374 5415 Acked Write 4 Done Sent Done false
17375 5416 Acked Read 3 Done Sent Done false
17376 5417 Acked Write 3 Done Sent Done false
17377 5418 Acked Read 3 Done Sent Done false
17378 5419 Acked Write 10 Done Sent Done false
17379 5420 Acked Read 2 Done Sent Done false
17380 5421 Acked Read 5 Done Sent Done false
17381 5422 Acked Flush 0 Done Sent Done false
17382 5423 Acked Read 3 Done Sent Done false
17383 5424 Acked Write 6 Done Sent Done false
17384 5425 Acked Write 8 Done Sent Done false
17385 5426 Acked Write 4 Done Sent Done false
17386 5427 Acked Write 2 Done Sent Done false
17387 5428 Acked Read 10 Done Sent Done false
17388 5429 Acked Flush 0 Done Sent Done false
17389 5430 Acked Read 2 Done Sent Done false
17390 5431 Acked Write 2 Done Sent Done false
17391 5432 Acked Write 10 Done Sent Done false
17392 5433 Acked Read 4 Done Sent Done false
17393 5434 Acked Read 9 Done Sent Done false
17394 5435 Acked Read 7 Done Sent Done false
17395 5436 Acked Write 1 Done Sent Done false
17396 5437 Acked Read 7 Done Sent Done false
17397 5438 Acked Read 3 Done Sent Done false
17398 5439 Acked Read 8 Done Sent Done false
17399 5440 Acked Read 8 Done Sent Done false
17400 5441 Acked Write 7 Done Sent Done false
17401 5442 Acked Read 2 Done Sent Done false
17402 5443 Acked Read 3 Done Sent Done false
17403 5444 Acked Read 9 Done Sent Done false
17404 5445 Acked Write 5 Done Sent Done false
17405 5446 Acked Write 9 Done Sent Done false
17406 5447 Acked Write 10 Done Sent Done false
17407 5448 Acked Read 3 Done Sent Done false
17408 5449 Acked Read 2 Done Sent Done false
17409 5450 Acked Write 4 Done Sent Done false
17410 5451 Acked Read 3 Done Sent Done false
17411 5452 Acked Write 2 Done Sent Done false
17412 5453 Acked Write 10 Done Sent Done false
17413 5454 Acked Read 7 Done Sent Done false
17414 5455 Acked Read 2 Done Sent Done false
17415 5456 Acked Read 2 Done Sent Done false
17416 5457 Acked Write 10 Done Sent Done false
17417 5458 Acked Read 2 Done Sent Done false
17418 5459 Acked Write 5 Done Sent Done false
17419 5460 Acked Write 8 Done Sent Done false
17420 5461 Acked Read 8 Done Sent Done false
17421 5462 Acked Write 7 Done Sent Done false
17422 5463 Acked Read 9 Done Sent Done false
17423 5464 Acked Flush 0 Done Sent Done false
17424 5465 Acked Write 1 Done Sent Done false
17425 5466 Acked Read 3 Done Sent Done false
17426 5467 Acked Read 3 Done Sent Done false
17427 5468 Acked Write 7 Done Sent Done false
17428 5469 Acked Write 7 Done Sent Done false
17429 5470 Acked Read 3 Done Sent Done false
17430 5471 Acked Read 3 Done Sent Done false
17431 5472 Acked Read 8 Done Sent Done false
17432 5473 Acked Read 10 Done Sent Done false
17433 5474 Acked Read 10 Done Sent Done false
17434 5475 Acked Read 8 Done Sent Done false
17435 5476 Acked Write 5 Done Sent Done false
17436 5477 Acked Read 4 Done Sent Done false
17437 5478 Acked Read 10 Done Sent Done false
17438 5479 Acked Read 1 Done Sent Done false
17439 5480 Acked Write 3 Done Sent Done false
17440 5481 Acked Read 9 Done Sent Done false
17441 5482 Acked Write 9 Done Sent Done false
17442 5483 Acked Read 10 Done Sent Done false
17443 5484 Acked Read 9 Done Sent Done false
17444 5485 Acked Write 9 Done Sent Done false
17445 5486 Acked Write 2 Done Sent Done false
17446 5487 Acked Write 1 Done Sent Done false
17447 5488 Acked Read 8 Done Sent Done false
17448 5489 Acked Read 6 Done Sent Done false
17449 5490 Acked Flush 0 Done Sent Done false
17450 5491 Acked Read 8 Done Sent Done false
17451 5492 Acked Read 3 Done Sent Done false
17452 5493 Acked Write 10 Done Sent Done false
17453 5494 Acked Read 9 Done Sent Done false
17454 5495 Acked Read 5 Done Sent Done false
17455 5496 Acked Write 1 Done Sent Done false
17456 5497 Acked Read 6 Done Sent Done false
17457 5498 Acked Read 7 Done Sent Done false
17458 5499 Acked Read 7 Done Sent Done false
17459 5500 Acked Read 10 Done Sent Done false
17460 5501 Acked Read 5 Done Sent Done false
17461 5502 Acked Read 4 Done Sent Done false
17462 5503 Acked Read 5 Done Sent Done false
17463 5504 Acked Write 8 Done Sent Done false
17464 5505 Acked Read 1 Done Sent Done false
17465 5506 Acked Flush 0 Done Sent Done false
17466 5507 Acked Write 10 Done Sent Done false
17467 5508 Acked Write 4 Done Sent Done false
17468 5509 Acked Read 4 Done Sent Done false
17469 5510 Acked Flush 0 Done Sent Done false
17470 5511 Acked Read 7 Done Sent Done false
17471 5512 Acked Write 8 Done Sent Done false
17472 5513 Acked Read 6 Done Sent Done false
17473 5514 Acked Read 6 Done Sent Done false
17474 5515 Acked Flush 0 Done Sent Done false
17475 5516 Acked Read 5 Done Sent Done false
17476 5517 Acked Read 9 Done Sent Done false
17477 5518 Acked Read 6 Done Sent Done false
17478 5519 Acked Write 7 Done Sent Done false
17479 5520 Acked Read 3 Done Sent Done false
17480 5521 Acked Read 7 Done Sent Done false
17481 5522 Acked Read 9 Done Sent Done false
17482 5523 Acked Write 2 Done Sent Done false
17483 5524 Acked Flush 0 Done Sent Done false
17484 5525 Acked Read 10 Done Sent Done false
17485 5526 Acked Read 5 Done Sent Done false
17486 5527 Acked Read 9 Done Sent Done false
17487 STATES DS:0 DS:1 DS:2 TOTAL
17488 Sent 0 301 0 301
17489 Done 301 0 301 602
17490 Skipped 0 0 0 0
17491 Error 0 0 0 0
17492 Last Flush: Some(JobId(5524)) Some(JobId(5226)) Some(JobId(5524))
17493 Downstairs last five completed: 5226 5225 5224 5223 5222
17494 Upstairs last five completed: 5527 5526 5525 5524 5523
17495 CLIENT: Up:0 ds:301 act:2
174962025-02-25T17:22:02.640ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
174972025-02-25T17:22:03.773ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
174982025-02-25T17:22:04.642ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
174992025-02-25T17:22:05.535ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
175002025-02-25T17:22:05.535ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61642) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
175012025-02-25T17:22:05.536ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
175022025-02-25T17:22:05.537ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
175032025-02-25T17:22:05.537ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
175042025-02-25T17:22:05.537ZINFOcrucible: send last flush ID to this DS: Some(JobId(5226)) = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
175052025-02-25T17:22:05.537ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
175062025-02-25T17:22:05.537ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5226)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
175072025-02-25T17:22:05.537ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5226)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
17508 ----------------------------------------------------------------
17509 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:3
17510 States: Active Active Active
17511 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17512 5525 Acked Read 10 Done Done Done true
17513 5526 Acked Read 5 Done Done Done true
17514 5527 Acked Read 9 Done Done Done true
17515 STATES DS:0 DS:1 DS:2 TOTAL
17516 Sent 0 0 0 0
17517 Done 3 3 3 9
17518 Skipped 0 0 0 0
17519 Error 0 0 0 0
17520 Last Flush: Some(JobId(5524)) Some(JobId(5524)) Some(JobId(5524))
17521 Downstairs last five completed: 5524 5523 5522 5521 5520
17522 Upstairs last five completed: 5527 5526 5525 5524 5523
17523 CLIENT: Up:0 ds:3 act:3
17524 ----------------------------------------------------------------
17525 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17526 Downstairs last five completed: 5528 5527 5526 5525 5524
17527 Upstairs last five completed: 5528 5527 5526 5525 5524
17528 CLIENT: Up:0 ds:0 act:3
17529 Replay: All jobs finished
17530 Main task has work to do, go find it
17531 got dsc Stop(1)
17532 stop 1
17533 [8820] Got stop action so:false kr:false
17534 Feb 25 17:22:09.795 INFO request completed, latency_us: 386, response_code: 204, uri: /stop/cid/1, method: POST, req_id: f9ca8e20-1fbf-4ad4-b428-8938aefe5ed8, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
175352025-02-25T17:22:09.796ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
175362025-02-25T17:22:09.796ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
175372025-02-25T17:22:09.796ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
17538 Feb 25 17:22:09.796 INFO request completed, latency_us: 138, response_code: 200, uri: /state/cid/1, method: GET, req_id: 9b3cf0ed-5e07-4581-bc54-9d9f94b465fe, remote_addr: 127.0.0{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T17:22:09.796243221Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
17539 .1:52576, local_addr: 127.0.0.1:9998
17540 [8820] Exited with: ExitStatus(unix_wait_status(9))
17541 [8820][1] reports Exit
17542 Feb 25 17:22:13.797 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/1, method: GET, req_id: 0127ca41-2653-49e1-903a-f26db136d49b, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
17543 001/300 Write block 2167 len 16384 data: 2 2 2 2
17544 002/300 Read block 4132 len 4096
17545 003/300 Write block 3452 len 16384 data: 5 5 5 5
17546 004/300 Read block 3525 len 20480
17547 005/300 Write block 2941 len 8192 data: 5 5
17548 006/300 Read block 4075 len 28672
17549 007/300 Read block 2844 len 36864
17550 008/300 Flush
17551 009/300 Read block 1842 len 28672
17552 010/300 Write block 4850 len 24576 data: 5 4 4 3 4 4
17553 011/300 Flush
17554 012/300 Write block 4963 len 24576 data: 5 5 6 4 4 5
17555 013/300 Flush
17556 014/300 Read block 3000 len 16384
17557 015/300 Write block 2894 len 8192 data: 4 4
17558 016/300 Write block 4349 len 12288 data: 7 6 7
17559 017/300 Write block 490 len 36864 data: 4 5 5 5 5 5 5 5 4
17560 018/300 Read block 1739 len 40960
17561 019/300 Write block 3816 len 32768 data: 6 5 5 5 4 6 5 5
17562 020/300 Read block 1982 len 12288
17563 021/300 Write block 3517 len 16384 data: 5 5 5 5
17564 022/300 Write block 4356 len 24576 data: 8 5 6 4 4 4
17565 023/300 Write block 3143 len 16384 data: 4 4 2 2
17566 024/300 Read block 937 len 28672
17567 025/300 Write block 4130 len 20480 data: 4 4 3 4 5
17568 026/300 Read block 4432 len 20480
17569 027/300 Read block 185 len 36864
17570 028/300 Read block 1624 len 20480
17571 029/300 Write block 1481 len 24576 data: 6 5 6 5 5 5
17572 030/300 Read block 2233 len 24576
17573 031/300 Write block 2136 len 40960 data: 4 4 5 4 4 4 4 5 5 4
17574 032/300 Read block 4779 len 24576
17575 033/300 Read block 4411 len 28672
17576 034/300 Write block 4719 len 40960 data: 4 6 6 6 3 3 2 2 2 2
17577 035/300 Read block 959 len 4096
17578 036/300 Write block 1360 len 8192 data: 5 7
17579 037/300 Write block 1050 len 16384 data: 4 4 3 5
17580 038/300 Read block 2234 len 4096
17581 039/300 Read block 2367 len 40960
17582 040/300 Write block 1793 len 4096 data: 3
17583 041/300 Read block 521 len 36864
17584 042/300 Flush
17585 043/300 Read block 4163 len 28672
17586 044/300 Write block 451 len 40960 data: 2 2 3 3 3 4 4 3 3 3
17587 045/300 Write block 320 len 12288 data: 4 4 4
17588 046/300 Read block 3637 len 28672
17589 047/300 Write block 1440 len 36864 data: 6 5 6 6 6 6 6 6 6
17590 048/300 Read block 4174 len 36864
17591 049/300 Read block 4153 len 32768
17592 050/300 Write block 34 len 28672 data: 3 4 5 5 5 4 5
17593 051/300 Read block 2069 len 8192
17594 052/300 Read block 458 len 24576
17595 053/300 Read block 578 len 12288
17596 054/300 Read block 3170 len 20480
17597 055/300 Write block 2848 len 20480 data: 3 3 4 3 4
17598 056/300 Read block 4105 len 36864
17599 057/300 Write block 3232 len 32768 data: 2 2 2 4 4 4 4 4
17600 058/300 Read block 161 len 40960
17601 059/300 Write block 1847 len 28672 data: 2 2 2 2 2 2 3
17602 060/300 Write block 2601 len 4096 data: 6
17603 061/300 Write block 4921 len 32768 data: 2 3 3 3 3 3 2 2
17604 062/300 Write block 3296 len 24576 data: 5 5 4 5 5 6
17605 063/300 Read block 3730 len 12288
17606 064/300 Write block 4010 len 36864 data: 3 2 2 2 2 3 3 3 3
17607 065/300 Write block 1044 len 28672 data: 6 6 6 6 4 4 5
17608 066/300 Read block 3154 len 4096
17609 067/300 Read block 3111 len 24576
17610 068/300 Read block 3552 len 36864
17611 069/300 Write block 809 len 16384 data: 2 2 2 2
17612 070/300 Write block 1584 len 8192 data: 3 3
17613 071/300 Write block 455 len 36864 data: 4 5 5 4 4 4 2 2 2
17614 072/300 Read block 1203 len 20480
17615 073/300 Write block 1421 len 40960 data: 3 4 3 3 3 5 6 5 5 6
17616 074/300 Write block 3416 len 12288 data: 3 4 3
17617 075/300 Flush
17618 076/300 Write block 1327 len 32768 data: 4 4 4 3 3 3 3 3
17619 077/300 Write block 3395 len 12288 data: 5 5 4
17620 078/300 Read block 1720 len 40960
17621 079/300 Read block 3564 len 8192
17622 080/300 Read block 3277 len 24576
17623 081/300 Read block 4122 len 32768
17624 082/300 Read block 3025 len 8192
17625 083/300 Write block 410 len 12288 data: 4 5 5
17626 084/300 Read block 47 len 16384
17627 085/300 Write block 1108 len 40960 data: 7 6 7 8 5 4 3 2 2 2
17628 086/300 Flush
17629 087/300 Read block 4358 len 16384
17630 088/300 Read block 3685 len 4096
17631 089/300 Read block 2953 len 32768
17632 090/300 Read block 1918 len 12288
17633 091/300 Read block 1823 len 36864
17634 092/300 Read block 79 len 32768
17635 093/300 Write block 4568 len 24576 data: 3 3 3 3 3 3
17636 094/300 Read block 255 len 16384
17637 095/300 Write block 3544 len 12288 data: 3 3 3
17638 096/300 Read block 2908 len 8192
17639 097/300 Write block 3214 len 36864 data: 5 5 5 6 8 9 7 8 7
17640 098/300 Write block 1215 len 24576 data: 3 3 3 2 3 3
17641 099/300 Write block 3452 len 20480 data: 6 6 6 6 5
17642 100/300 Write block 697 len 16384 data: 3 3 3 4
17643 101/300 Read block 1753 len 12288
17644 102/300 Write block 2109 len 32768 data: 2 2 3 3 4 4 3 5
17645 103/300 Read block 3882 len 32768
17646 104/300 Read block 718 len 28672
17647 105/300 Read block 4 len 36864
17648 106/300 Write block 131 len 24576 data: 4 4 3 4 4 4
17649 107/300 Read block 491 len 32768
17650 108/300 Read block 53 len 40960
17651 109/300 Read block 4277 len 40960
17652 110/300 Flush
17653 111/300 Read block 3041 len 40960
17654 112/300 Read block 2334 len 40960
17655 113/300 Write block 4958 len 16384 data: 2 2 3 5
17656 114/300 Read block 1618 len 24576
17657 115/300 Read block 3082 len 40960
17658 116/300 Read block 4647 len 24576
17659 117/300 Write block 21 len 32768 data: 3 3 4 4 4 4 5 5
17660 118/300 Write block 1391 len 40960 data: 5 4 4 5 5 4 4 4 5 5
17661 119/300 Read block 897 len 12288
17662 120/300 Write block 1059 len 24576 data: 4 3 3 6 4 5
17663 121/300 Read block 3963 len 4096
17664 122/300 Read block 3267 len 12288
17665 123/300 Read block 2390 len 36864
17666 124/300 Write block 3604 len 40960 data: 4 4 5 5 5 5 3 3 3 3
17667 125/300 Write block 3005 len 24576 data: 6 6 5 5 4 5
17668 126/300 Read block 1018 len 20480
17669 127/300 Write block 214 len 12288 data: 5 5 5
17670 128/300 Write block 4954 len 32768 data: 3 2 3 2 3 3 4 6
17671 129/300 Flush
17672 130/300 Read block 4741 len 24576
17673 131/300 Read block 4543 len 16384
17674 132/300 Read block 807 len 28672
17675 133/300 Read block 2418 len 36864
17676 134/300 Read block 536 len 8192
17677 135/300 Read block 1566 len 28672
17678 136/300 Read block 1034 len 28672
17679 137/300 Flush
17680 138/300 Flush
17681 139/300 Write block 1812 len 12288 data: 3 4 4
17682 140/300 Write block 2176 len 40960 data: 4 5 5 4 3 2 2 3 3 3
17683 141/300 Write block 633 len 24576 data: 7 5 4 7 6 5
17684 142/300 Write block 2993 len 36864 data: 3 5 5 4 2 5 5 5 5
17685 143/300 Read block 1171 len 20480
17686 144/300 Read block 4309 len 8192
17687 145/300 Read block 2027 len 20480
17688 146/300 Flush
17689 147/300 Write block 963 len 40960 data: 5 5 5 6 6 6 4 4 3 2
17690 148/300 Write block 3073 len 16384 data: 4 3 3 5
17691 149/300 Read block 439 len 36864
17692 150/300 Flush
17693 151/300 Write block 2204 len 28672 data: 3 2 3 4 3 3 5
17694 152/300 Read block 3053 len 32768
17695 153/300 Write block 2117 len 16384 data: 5 5 4 3
17696 154/300 Write block 4766 len 8192 data: 7 6
17697 155/300 Read block 2524 len 8192
17698 156/300 Write block 2063 len 32768 data: 3 3 5 4 3 3 4 4
17699 157/300 Read block 410 len 12288
17700 158/300 Read block 453 len 20480
17701 159/300 Read block 3508 len 8192
17702 160/300 Write block 3095 len 40960 data: 5 5 3 3 4 4 3 4 4 4
17703 161/300 Write block 1704 len 40960 data: 7 7 8 5 4 4 5 5 7 6
17704 162/300 Read block 639 len 36864
17705 163/300 Write block 2965 len 4096 data: 6
17706 164/300 Write block 3721 len 16384 data: 5 6 5 7
17707 165/300 Read block 4524 len 20480
17708 166/300 Write block 1278 len 28672 data: 3 2 2 3 3 3 4
17709 167/300 Write block 2699 len 12288 data: 3 3 3
17710 168/300 Flush
17711 169/300 Write block 1215 len 4096 data: 4
17712 170/300 Read block 1494 len 4096
17713 171/300 Read block 835 len 16384
17714 172/300 Read block 893 len 40960
17715 173/300 Write block 2894 len 4096 data: 5
17716 174/300 Write block 3287 len 12288 data: 4 3 3
17717 175/300 Read block 1004 len 36864
17718 176/300 Read block 4732 len 20480
17719 177/300 Write block 1335 len 8192 data: 3 4
17720 178/300 Flush
17721 179/300 Read block 1565 len 36864
17722 180/300 Write block 3097 len 8192 data: 4 4
17723 181/300 Read block 4157 len 24576
17724 182/300 Read block 3165 len 4096
17725 183/300 Read block 300 len 4096
17726 184/300 Read block 2753 len 12288
17727 185/300 Write block 2302 len 24576 data: 6 7 7 5 4 3
17728 186/300 Flush
17729 187/300 Write block 536 len 28672 data: 5 4 3 4 4 3 3
17730 188/300 Read block 1822 len 40960
17731 189/300 Read block 1789 len 32768
17732 190/300 Read block 4264 len 24576
17733 191/300 Read block 2095 len 16384
17734 192/300 Read block 4018 len 28672
17735 193/300 Write block 4439 len 24576 data: 3 3 2 2 3 3
17736 194/300 Write block 2951 len 28672 data: 4 4 4 3 2 2 3
17737 195/300 Write block 2136 len 28672 data: 5 5 6 5 5 5 5
17738 196/300 Read block 3165 len 36864
17739 197/300 Write block 1787 len 4096 data: 3
17740 198/300 Read block 4947 len 24576
17741 199/300 Read block 1867 len 8192
17742 200/300 Flush
17743 201/300 Write block 2213 len 24576 data: 7 7 6 6 7 7
17744 202/300 Write block 4718 len 4096 data: 5
17745 203/300 Flush
17746 204/300 Read block 2374 len 32768
17747 205/300 Read block 4732 len 8192
17748 206/300 Read block 1182 len 20480
17749 207/300 Read block 830 len 8192
17750 208/300 Read block 4812 len 24576
17751 209/300 Read block 2006 len 40960
17752 210/300 Write block 3426 len 20480 data: 4 5 4 4 4
17753 211/300 Read block 76 len 28672
17754 212/300 Write block 1744 len 24576 data: 3 3 3 3 4 4
17755 213/300 Flush
17756 214/300 Read block 1821 len 16384
17757 215/300 Write block 2720 len 28672 data: 5 3 3 3 2 2 2
17758 216/300 Read block 4583 len 16384
17759 217/300 Write block 2677 len 12288 data: 4 4 3
17760 218/300 Read block 4277 len 40960
17761 219/300 Write block 3328 len 40960 data: 3 3 4 4 5 5 5 6 6 4
17762 220/300 Read block 142 len 24576
17763 221/300 Write block 553 len 20480 data: 3 3 3 3 3
17764 222/300 Write block 3249 len 12288 data: 3 3 3
17765 223/300 Read block 465 len 36864
17766 224/300 Read block 4737 len 12288
17767 225/300 Read block 34 len 36864
17768 226/300 Read block 934 len 4096
17769 227/300 Flush
17770 228/300 Write block 1608 len 16384 data: 5 5 4 4
17771 229/300 Write block 4549 len 24576 data: 3 4 4 4 4 3
17772 230/300 Write block 1324 len 4096 data: 5
17773 231/300 Write block 3762 len 8192 data: 4 4
17774 232/300 Write block 4974 len 24576 data: 4 4 4 3 3 3
17775 233/300 Write block 2129 len 20480 data: 6 5 5 5 5
17776 234/300 Write block 2703 len 32768 data: 2 3 3 5 4 4 4 3
17777 235/300 Flush
17778 236/300 Read block 2276 len 24576
17779 237/300 Write block 4577 len 16384 data: 4 5 4 5
17780 238/300 Write block 2178 len 32768 data: 6 5 4 3 3 4 4 4
17781 239/300 Read block 2565 len 4096
17782 240/300 Read block 933 len 20480
17783 241/300 Read block 362 len 32768
17784 242/300 Read block 2488 len 36864
17785 243/300 Read block 4122 len 8192
17786 244/300 Read block 1784 len 20480
17787 245/300 Read block 370 len 16384
17788 246/300 Read block 3404 len 24576
17789 247/300 Read block 4799 len 24576
17790 248/300 Write block 870 len 12288 data: 7 6 5
17791 249/300 Write block 3623 len 24576 data: 2 3 3 3 3 2
17792 250/300 Read block 819 len 32768
17793 251/300 Write block 837 len 20480 data: 7 6 6 5 4
17794 252/300 Write block 119 len 36864 data: 2 2 2 2 2 4 5 5 5
17795 253/300 Write block 3203 len 12288 data: 5 5 4
17796 254/300 Write block 2462 len 32768 data: 8 6 5 4 4 3 3 4
17797 255/300 Read block 897 len 12288
17798 256/300 Read block 108 len 24576
17799 257/300 Read block 2873 len 24576
17800 258/300 Read block 3308 len 16384
17801 259/300 Read block 3611 len 24576
17802 260/300 Write block 3649 len 40960 data: 5 5 6 6 7 7 7 7 5 6
17803 261/300 Write block 3688 len 4096 data: 6
17804 262/300 Read block 4411 len 20480
17805 263/300 Read block 3754 len 24576
17806 264/300 Read block 1479 len 40960
17807 265/300 Write block 2693 len 20480 data: 2 3 3 3 3
17808 266/300 Read block 1735 len 28672
17809 267/300 Read block 373 len 32768
17810 268/300 Read block 847 len 4096
17811 269/300 Read block 255 len 12288
17812 270/300 Read block 2213 len 28672
17813 271/300 Write block 105 len 24576 data: 3 2 3 5 6 6
17814 272/300 Read block 3458 len 4096
17815 273/300 Write block 2390 len 40960 data: 4 4 6 6 6 6 6 5 5 6
17816 274/300 Read block 2948 len 4096
17817 275/300 Write block 3774 len 20480 data: 5 3 5 5 4
17818 276/300 Write block 2744 len 36864 data: 5 5 5 4 4 4 4 5 5
17819 277/300 Read block 4248 len 36864
17820 278/300 Read block 613 len 16384
17821 279/300 Read block 391 len 16384
17822 280/300 Read block 3203 len 24576
17823 281/300 Read block 1869 len 36864
17824 282/300 Read block 1603 len 4096
17825 283/300 Write block 2292 len 36864 data: 4 5 5 5 4 4 5 5 5
17826 284/300 Write block 4149 len 20480 data: 4 4 2 2 2
17827 285/300 Write block 1126 len 12288 data: 3 3 3
17828 286/300 Read block 216 len 8192
17829 287/300 Write block 1788 len 20480 data: 2 2 2 2 3
17830 288/300 Write block 2291 len 28672 data: 4 5 6 6 6 5 5
17831 289/300 Write block 3941 len 4096 data: 6
17832 290/300 Flush
17833 291/300 Read block 4662 len 8192
17834 292/300 Read block 3982 len 32768
17835 293/300 Read block 4937 len 32768
17836 294/300 Read block 2443 len 36864
17837 295/300 Read block 1988 len 4096
17838 296/300 Write block 4565 len 4096 data: 4
17839 297/300 Read block 4901 len 40960
17840 298/300 Read block 4926 len 16384
17841 299/300 Read block 3789 len 32768
17842 300/300 Read block 4643 len 28672
17843 301/300 Read block 4641 len 32768
17844 Main task has work to do, go find it
17845 got dsc Start(1)
17846 start 1
17847 Feb 25 17:22:14.058 INFO request completed, latency_us: 261, response_code: 204, uri: /start/cid/1, method: POST, req_id: 99d453ab-66ca-4b70-9a0e-3db32bee5e3c, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
17848 Replay: started 1, returned:Ok(())
17849 ----------------------------------------------------------------
17850 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17851 States: Active Offline Active
17852 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17853 5529 Acked Write 4 Done Sent Done false
17854 5530 Acked Read 1 Done Sent Done false
17855 5531 Acked Write 4 Done Sent Done false
17856 5532 Acked Read 5 Done Sent Done false
17857 5533 Acked Write 2 Done Sent Done false
17858 5534 Acked Read 7 Done Sent Done false
17859 5535 Acked Read 9 Done Sent Done false
17860 5536 Acked Flush 0 Done Sent Done false
17861 5537 Acked Read 7 Done Sent Done false
17862 5538 Acked Write 6 Done Sent Done false
17863 5539 Acked Flush 0 Done Sent Done false
17864 5540 Acked Write 6 Done Sent Done false
17865 5541 Acked Flush 0 Done Sent Done false
17866 5542 Acked Read 4 Done Sent Done false
17867 5543 Acked Write 2 Done Sent Done false
17868 5544 Acked Write 3 Done Sent Done false
17869 5545 Acked Write 9 Done Sent Done false
17870 5546 Acked Read 10 Done Sent Done false
17871 5547 Acked Write 8 Done Sent Done false
17872 5548 Acked Read 3 Done Sent Done false
17873 5549 Acked Write 4 Done Sent Done false
17874 5550 Acked Write 6 Done Sent Done false
17875 5551 Acked Write 4 Done Sent Done false
17876 5552 Acked Read 7 Done Sent Done false
17877 5553 Acked Write 5 Done Sent Done false
17878 5554 Acked Read 5 Done Sent Done false
17879 5555 Acked Read 9 Done Sent Done false
17880 5556 Acked Read 5 Done Sent Done false
17881 5557 Acked Write 6 Done Sent Done false
17882 5558 Acked Read 6 Done Sent Done false
17883 5559 Acked Write 10 Done Sent Done false
17884 5560 Acked Read 6 Done Sent Done false
17885 5561 Acked Read 7 Done Sent Done false
17886 5562 Acked Write 10 Done Sent Done false
17887 5563 Acked Read 1 Done Sent Done false
17888 5564 Acked Write 2 Done Sent Done false
17889 5565 Acked Write 4 Done Sent Done false
17890 5566 Acked Read 1 Done Sent Done false
17891 5567 Acked Read 10 Done Sent Done false
17892 5568 Acked Write 1 Done Sent Done false
17893 5569 Acked Read 9 Done Sent Done false
17894 5570 Acked Flush 0 Done Sent Done false
17895 5571 Acked Read 7 Done Sent Done false
17896 5572 Acked Write 10 Done Sent Done false
17897 5573 Acked Write 3 Done Sent Done false
17898 5574 Acked Read 7 Done Sent Done false
17899 5575 Acked Write 9 Done Sent Done false
17900 5576 Acked Read 9 Done Sent Done false
17901 5577 Acked Read 8 Done Sent Done false
17902 5578 Acked Write 7 Done Sent Done false
17903 5579 Acked Read 2 Done Sent Done false
17904 5580 Acked Read 6 Done Sent Done false
17905 5581 Acked Read 3 Done Sent Done false
17906 5582 Acked Read 5 Done Sent Done false
17907 5583 Acked Write 5 Done Sent Done false
17908 5584 Acked Read 9 Done Sent Done false
17909 5585 Acked Write 8 Done Sent Done false
17910 5586 Acked Read 10 Done Sent Done false
17911 5587 Acked Write 7 Done Sent Done false
17912 5588 Acked Write 1 Done Sent Done false
17913 5589 Acked Write 8 Done Sent Done false
17914 5590 Acked Write 6 Done Sent Done false
17915 5591 Acked Read 3 Done Sent Done false
17916 5592 Acked Write 9 Done Sent Done false
17917 5593 Acked Write 7 Done Sent Done false
17918 5594 Acked Read 1 Done Sent Done false
17919 5595 Acked Read 6 Done Sent Done false
17920 5596 Acked Read 9 Done Sent Done false
17921 5597 Acked Write 4 Done Sent Done false
17922 5598 Acked Write 2 Done Sent Done false
17923 5599 Acked Write 9 Done Sent Done false
17924 5600 Acked Read 5 Done Sent Done false
17925 5601 Acked Write 10 Done Sent Done false
17926 5602 Acked Write 3 Done Sent Done false
17927 5603 Acked Flush 0 Done Sent Done false
17928 5604 Acked Write 8 Done Sent Done false
17929 5605 Acked Write 3 Done Sent Done false
17930 5606 Acked Read 10 Done Sent Done false
17931 5607 Acked Read 2 Done Sent Done false
17932 5608 Acked Read 6 Done Sent Done false
17933 5609 Acked Read 8 Done Sent Done false
17934 5610 Acked Read 2 Done Sent Done false
17935 5611 Acked Write 3 Done Sent Done false
17936 5612 Acked Read 4 Done Sent Done false
17937 5613 Acked Write 10 Done Sent Done false
17938 5614 Acked Flush 0 Done Sent Done false
17939 5615 Acked Read 4 Done Sent Done false
17940 5616 Acked Read 1 Done Sent Done false
17941 5617 Acked Read 8 Done Sent Done false
17942 5618 Acked Read 3 Done Sent Done false
17943 5619 Acked Read 9 Done Sent Done false
17944 5620 Acked Read 8 Done Sent Done false
17945 5621 Acked Write 6 Done Sent Done false
17946 5622 Acked Read 4 Done Sent Done false
17947 5623 Acked Write 3 Done Sent Done false
17948 5624 Acked Read 2 Done Sent Done false
17949 5625 Acked Write 9 Done Sent Done false
17950 5626 Acked Write 6 Done Sent Done false
17951 5627 Acked Write 5 Done Sent Done false
17952 5628 Acked Write 4 Done Sent Done false
17953 5629 Acked Read 3 Done Sent Done false
17954 5630 Acked Write 8 Done Sent Done false
17955 5631 Acked Read 8 Done Sent Done false
17956 5632 Acked Read 7 Done Sent Done false
17957 5633 Acked Read 9 Done Sent Done false
17958 5634 Acked Write 6 Done Sent Done false
17959 5635 Acked Read 8 Done Sent Done false
17960 5636 Acked Read 10 Done Sent Done false
17961 5637 Acked Read 10 Done Sent Done false
17962 5638 Acked Flush 0 Done Sent Done false
17963 5639 Acked Read 10 Done Sent Done false
17964 5640 Acked Read 10 Done Sent Done false
17965 5641 Acked Write 4 Done Sent Done false
17966 5642 Acked Read 6 Done Sent Done false
17967 5643 Acked Read 10 Done Sent Done false
17968 5644 Acked Read 6 Done Sent Done false
17969 5645 Acked Write 8 Done Sent Done false
17970 5646 Acked Write 10 Done Sent Done false
17971 5647 Acked Read 3 Done Sent Done false
17972 5648 Acked Write 6 Done Sent Done false
17973 5649 Acked Read 1 Done Sent Done false
17974 5650 Acked Read 3 Done Sent Done false
17975 5651 Acked Read 9 Done Sent Done false
17976 5652 Acked Write 10 Done Sent Done false
17977 5653 Acked Write 6 Done Sent Done false
17978 5654 Acked Read 5 Done Sent Done false
17979 5655 Acked Write 3 Done Sent Done false
17980 5656 Acked Write 8 Done Sent Done false
17981 5657 Acked Flush 0 Done Sent Done false
17982 5658 Acked Read 6 Done Sent Done false
17983 5659 Acked Read 4 Done Sent Done false
17984 5660 Acked Read 7 Done Sent Done false
17985 5661 Acked Read 9 Done Sent Done false
17986 5662 Acked Read 2 Done Sent Done false
17987 5663 Acked Read 7 Done Sent Done false
17988 5664 Acked Read 7 Done Sent Done false
17989 5665 Acked Flush 0 Done Sent Done false
17990 5666 Acked Flush 0 Done Sent Done false
17991 5667 Acked Write 3 Done Sent Done false
17992 5668 Acked Write 10 Done Sent Done false
17993 5669 Acked Write 6 Done Sent Done false
17994 5670 Acked Write 9 Done Sent Done false
17995 5671 Acked Read 5 Done Sent Done false
17996 5672 Acked Read 2 Done Sent Done false
17997 5673 Acked Read 5 Done Sent Done false
17998 5674 Acked Flush 0 Done Sent Done false
17999 5675 Acked Write 10 Done Sent Done false
18000 5676 Acked Write 4 Done Sent Done false
18001 5677 Acked Read 9 Done Sent Done false
18002 5678 Acked Flush 0 Done Sent Done false
18003 5679 Acked Write 7 Done Sent Done false
18004 5680 Acked Read 8 Done Sent Done false
18005 5681 Acked Write 4 Done Sent Done false
18006 5682 Acked Write 2 Done Sent Done false
18007 5683 Acked Read 2 Done Sent Done false
18008 5684 Acked Write 8 Done Sent Done false
18009 5685 Acked Read 3 Done Sent Done false
18010 5686 Acked Read 5 Done Sent Done false
18011 5687 Acked Read 2 Done Sent Done false
18012 5688 Acked Write 10 Done Sent Done false
18013 5689 Acked Write 10 Done Sent Done false
18014 5690 Acked Read 9 Done Sent Done false
18015 5691 Acked Write 1 Done Sent Done false
18016 5692 Acked Write 4 Done Sent Done false
18017 5693 Acked Read 5 Done Sent Done false
18018 5694 Acked Write 7 Done Sent Done false
18019 5695 Acked Write 3 Done Sent Done false
18020 5696 Acked Flush 0 Done Sent Done false
18021 5697 Acked Write 1 Done Sent Done false
18022 5698 Acked Read 1 Done Sent Done false
18023 5699 Acked Read 4 Done Sent Done false
18024 5700 Acked Read 10 Done Sent Done false
18025 5701 Acked Write 1 Done Sent Done false
18026 5702 Acked Write 3 Done Sent Done false
18027 5703 Acked Read 9 Done Sent Done false
18028 5704 Acked Read 5 Done Sent Done false
18029 5705 Acked Write 2 Done Sent Done false
18030 5706 Acked Flush 0 Done Sent Done false
18031 5707 Acked Read 9 Done Sent Done false
18032 5708 Acked Write 2 Done Sent Done false
18033 5709 Acked Read 6 Done Sent Done false
18034 5710 Acked Read 1 Done Sent Done false
18035 5711 Acked Read 1 Done Sent Done false
18036 5712 Acked Read 3 Done Sent Done false
18037 5713 Acked Write 6 Done Sent Done false
18038 5714 Acked Flush 0 Done Sent Done false
18039 5715 Acked Write 7 Done Sent Done false
18040 5716 Acked Read 10 Done Sent Done false
18041 5717 Acked Read 8 Done Sent Done false
18042 5718 Acked Read 6 Done Sent Done false
18043 5719 Acked Read 4 Done Sent Done false
18044 5720 Acked Read 7 Done Sent Done false
18045 5721 Acked Write 6 Done Sent Done false
18046 5722 Acked Write 7 Done Sent Done false
18047 5723 Acked Write 7 Done Sent Done false
18048 5724 Acked Read 9 Done Sent Done false
18049 5725 Acked Write 1 Done Sent Done false
18050 5726 Acked Read 6 Done Sent Done false
18051 5727 Acked Read 2 Done Sent Done false
18052 5728 Acked Flush 0 Done Sent Done false
18053 5729 Acked Write 6 Done Sent Done false
18054 5730 Acked Write 1 Done Sent Done false
18055 5731 Acked Flush 0 Done Sent Done false
18056 5732 Acked Read 8 Done Sent Done false
18057 5733 Acked Read 2 Done Sent Done false
18058 5734 Acked Read 5 Done Sent Done false
18059 5735 Acked Read 2 Done Sent Done false
18060 5736 Acked Read 6 Done Sent Done false
18061 5737 Acked Read 10 Done Sent Done false
18062 5738 Acked Write 5 Done Sent Done false
18063 5739 Acked Read 7 Done Sent Done false
18064 5740 Acked Write 6 Done Sent Done false
18065 5741 Acked Flush 0 Done Sent Done false
18066 5742 Acked Read 4 Done Sent Done false
18067 5743 Acked Write 7 Done Sent Done false
18068 5744 Acked Read 4 Done Sent Done false
18069 5745 Acked Write 3 Done Sent Done false
18070 5746 Acked Read 10 Done Sent Done false
18071 5747 Acked Write 10 Done Sent Done false
18072 5748 Acked Read 6 Done Sent Done false
18073 5749 Acked Write 5 Done Sent Done false
18074 5750 Acked Write 3 Done Sent Done false
18075 5751 Acked Read 9 Done Sent Done false
18076 5752 Acked Read 3 Done Sent Done false
18077 5753 Acked Read 9 Done Sent Done false
18078 5754 Acked Read 1 Done Sent Done false
18079 5755 Acked Flush 0 Done Sent Done false
18080 5756 Acked Write 4 Done Sent Done false
18081 5757 Acked Write 6 Done Sent Done false
18082 5758 Acked Write 1 Done Sent Done false
18083 5759 Acked Write 2 Done Sent Done false
18084 5760 Acked Write 6 Done Sent Done false
18085 5761 Acked Write 5 Done Sent Done false
18086 5762 Acked Write 8 Done Sent Done false
18087 5763 Acked Flush 0 Done Sent Done false
18088 5764 Acked Read 6 Done Sent Done false
18089 5765 Acked Write 4 Done Sent Done false
18090 5766 Acked Write 8 Done Sent Done false
18091 5767 Acked Read 1 Done Sent Done false
18092 5768 Acked Read 5 Done Sent Done false
18093 5769 Acked Read 8 Done Sent Done false
18094 5770 Acked Read 9 Done Sent Done false
18095 5771 Acked Read 2 Done Sent Done false
18096 5772 Acked Read 5 Done Sent Done false
18097 5773 Acked Read 4 Done Sent Done false
18098 5774 Acked Read 6 Done Sent Done false
18099 5775 Acked Read 6 Done Sent Done false
18100 5776 Acked Write 3 Done Sent Done false
18101 5777 Acked Write 6 Done Sent Done false
18102 5778 Acked Read 8 Done Sent Done false
18103 5779 Acked Write 5 Done Sent Done false
18104 5780 Acked Write 9 Done Sent Done false
18105 5781 Acked Write 3 Done Sent Done false
18106 5782 Acked Write 8 Done Sent Done false
18107 5783 Acked Read 3 Done Sent Done false
18108 5784 Acked Read 6 Done Sent Done false
18109 5785 Acked Read 6 Done Sent Done false
18110 5786 Acked Read 4 Done Sent Done false
18111 5787 Acked Read 6 Done Sent Done false
18112 5788 Acked Write 10 Done Sent Done false
18113 5789 Acked Write 1 Done Sent Done false
18114 5790 Acked Read 5 Done Sent Done false
18115 5791 Acked Read 6 Done Sent Done false
18116 5792 Acked Read 10 Done Sent Done false
18117 5793 Acked Write 5 Done Sent Done false
18118 5794 Acked Read 7 Done Sent Done false
18119 5795 Acked Read 8 Done Sent Done false
18120 5796 Acked Read 1 Done Sent Done false
18121 5797 Acked Read 3 Done Sent Done false
18122 5798 Acked Read 7 Done Sent Done false
18123 5799 Acked Write 6 Done Sent Done false
18124 5800 Acked Read 1 Done Sent Done false
18125 5801 Acked Write 10 Done Sent Done false
18126 5802 Acked Read 1 Done Sent Done false
18127 5803 Acked Write 5 Done Sent Done false
18128 5804 Acked Write 9 Done Sent Done false
18129 5805 Acked Read 9 Done Sent Done false
18130 5806 Acked Read 4 Done Sent Done false
18131 5807 Acked Read 4 Done Sent Done false
18132 5808 Acked Read 6 Done Sent Done false
18133 5809 Acked Read 9 Done Sent Done false
18134 5810 Acked Read 1 Done Sent Done false
18135 5811 Acked Write 9 Done Sent Done false
18136 5812 Acked Write 5 Done Sent Done false
18137 5813 Acked Write 3 Done Sent Done false
18138 5814 Acked Read 2 Done Sent Done false
18139 5815 Acked Write 5 Done Sent Done false
18140 5816 Acked Write 7 Done Sent Done false
18141 5817 Acked Write 1 Done Sent Done false
18142 5818 Acked Flush 0 Done Sent Done false
18143 5819 Acked Read 2 Done Sent Done false
18144 5820 Acked Read 8 Done Sent Done false
18145 5821 Acked Read 8 Done Sent Done false
18146 5822 Acked Read 9 Done Sent Done false
18147 5823 Acked Read 1 Done Sent Done false
18148 5824 Acked Write 1 Done Sent Done false
18149 5825 Acked Read 10 Done Sent Done false
18150 5826 Acked Read 4 Done Sent Done false
18151 5827 Acked Read 8 Done Sent Done false
18152 5828 Acked Read 7 Done Sent Done false
18153 5829 Acked Read 8 Done Sent Done false
18154 STATES DS:0 DS:1 DS:2 TOTAL
18155 Sent 0 301 0 301
18156 Done 301 0 301 602
18157 Skipped 0 0 0 0
18158 Error 0 0 0 0
18159 Last Flush: Some(JobId(5818)) Some(JobId(5528)) Some(JobId(5818))
18160 Downstairs last five completed: 5528 5527 5526 5525 5524
18161 Upstairs last five completed: 5829 5828 5827 5826 5825
18162 CLIENT: Up:0 ds:301 act:2
181632025-02-25T17:22:14.657ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
181642025-02-25T17:22:15.659ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
18165 [8820] got start action
18166 [8820] I am going to restart
18167 Starting downstairs at port 8820
18168 Make output file at "/tmp/dsc/downstairs-8820.txt"
18169 [8820][1] reports Starting
18170 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1224)
18171 [8820][1] reports Running
181722025-02-25T17:22:16.661ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
181732025-02-25T17:22:17.663ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
18174 ----------------------------------------------------------------
18175 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18176 States: Active Offline Active
18177 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18178 5529 Acked Write 4 Done Sent Done false
18179 5530 Acked Read 1 Done Sent Done false
18180 5531 Acked Write 4 Done Sent Done false
18181 5532 Acked Read 5 Done Sent Done false
18182 5533 Acked Write 2 Done Sent Done false
18183 5534 Acked Read 7 Done Sent Done false
18184 5535 Acked Read 9 Done Sent Done false
18185 5536 Acked Flush 0 Done Sent Done false
18186 5537 Acked Read 7 Done Sent Done false
18187 5538 Acked Write 6 Done Sent Done false
18188 5539 Acked Flush 0 Done Sent Done false
18189 5540 Acked Write 6 Done Sent Done false
18190 5541 Acked Flush 0 Done Sent Done false
18191 5542 Acked Read 4 Done Sent Done false
18192 5543 Acked Write 2 Done Sent Done false
18193 5544 Acked Write 3 Done Sent Done false
18194 5545 Acked Write 9 Done Sent Done false
18195 5546 Acked Read 10 Done Sent Done false
18196 5547 Acked Write 8 Done Sent Done false
18197 5548 Acked Read 3 Done Sent Done false
18198 5549 Acked Write 4 Done Sent Done false
18199 5550 Acked Write 6 Done Sent Done false
18200 5551 Acked Write 4 Done Sent Done false
18201 5552 Acked Read 7 Done Sent Done false
18202 5553 Acked Write 5 Done Sent Done false
18203 5554 Acked Read 5 Done Sent Done false
18204 5555 Acked Read 9 Done Sent Done false
18205 5556 Acked Read 5 Done Sent Done false
18206 5557 Acked Write 6 Done Sent Done false
18207 5558 Acked Read 6 Done Sent Done false
18208 5559 Acked Write 10 Done Sent Done false
18209 5560 Acked Read 6 Done Sent Done false
18210 5561 Acked Read 7 Done Sent Done false
18211 5562 Acked Write 10 Done Sent Done false
18212 5563 Acked Read 1 Done Sent Done false
18213 5564 Acked Write 2 Done Sent Done false
18214 5565 Acked Write 4 Done Sent Done false
18215 5566 Acked Read 1 Done Sent Done false
18216 5567 Acked Read 10 Done Sent Done false
18217 5568 Acked Write 1 Done Sent Done false
18218 5569 Acked Read 9 Done Sent Done false
18219 5570 Acked Flush 0 Done Sent Done false
18220 5571 Acked Read 7 Done Sent Done false
18221 5572 Acked Write 10 Done Sent Done false
18222 5573 Acked Write 3 Done Sent Done false
18223 5574 Acked Read 7 Done Sent Done false
18224 5575 Acked Write 9 Done Sent Done false
18225 5576 Acked Read 9 Done Sent Done false
18226 5577 Acked Read 8 Done Sent Done false
18227 5578 Acked Write 7 Done Sent Done false
18228 5579 Acked Read 2 Done Sent Done false
18229 5580 Acked Read 6 Done Sent Done false
18230 5581 Acked Read 3 Done Sent Done false
18231 5582 Acked Read 5 Done Sent Done false
18232 5583 Acked Write 5 Done Sent Done false
18233 5584 Acked Read 9 Done Sent Done false
18234 5585 Acked Write 8 Done Sent Done false
18235 5586 Acked Read 10 Done Sent Done false
18236 5587 Acked Write 7 Done Sent Done false
18237 5588 Acked Write 1 Done Sent Done false
18238 5589 Acked Write 8 Done Sent Done false
18239 5590 Acked Write 6 Done Sent Done false
18240 5591 Acked Read 3 Done Sent Done false
18241 5592 Acked Write 9 Done Sent Done false
18242 5593 Acked Write 7 Done Sent Done false
18243 5594 Acked Read 1 Done Sent Done false
18244 5595 Acked Read 6 Done Sent Done false
18245 5596 Acked Read 9 Done Sent Done false
18246 5597 Acked Write 4 Done Sent Done false
18247 5598 Acked Write 2 Done Sent Done false
18248 5599 Acked Write 9 Done Sent Done false
18249 5600 Acked Read 5 Done Sent Done false
18250 5601 Acked Write 10 Done Sent Done false
18251 5602 Acked Write 3 Done Sent Done false
18252 5603 Acked Flush 0 Done Sent Done false
18253 5604 Acked Write 8 Done Sent Done false
18254 5605 Acked Write 3 Done Sent Done false
18255 5606 Acked Read 10 Done Sent Done false
18256 5607 Acked Read 2 Done Sent Done false
18257 5608 Acked Read 6 Done Sent Done false
18258 5609 Acked Read 8 Done Sent Done false
18259 5610 Acked Read 2 Done Sent Done false
18260 5611 Acked Write 3 Done Sent Done false
18261 5612 Acked Read 4 Done Sent Done false
18262 5613 Acked Write 10 Done Sent Done false
18263 5614 Acked Flush 0 Done Sent Done false
18264 5615 Acked Read 4 Done Sent Done false
18265 5616 Acked Read 1 Done Sent Done false
18266 5617 Acked Read 8 Done Sent Done false
18267 5618 Acked Read 3 Done Sent Done false
18268 5619 Acked Read 9 Done Sent Done false
18269 5620 Acked Read 8 Done Sent Done false
18270 5621 Acked Write 6 Done Sent Done false
18271 5622 Acked Read 4 Done Sent Done false
18272 5623 Acked Write 3 Done Sent Done false
18273 5624 Acked Read 2 Done Sent Done false
18274 5625 Acked Write 9 Done Sent Done false
18275 5626 Acked Write 6 Done Sent Done false
18276 5627 Acked Write 5 Done Sent Done false
18277 5628 Acked Write 4 Done Sent Done false
18278 5629 Acked Read 3 Done Sent Done false
18279 5630 Acked Write 8 Done Sent Done false
18280 5631 Acked Read 8 Done Sent Done false
18281 5632 Acked Read 7 Done Sent Done false
18282 5633 Acked Read 9 Done Sent Done false
18283 5634 Acked Write 6 Done Sent Done false
18284 5635 Acked Read 8 Done Sent Done false
18285 5636 Acked Read 10 Done Sent Done false
18286 5637 Acked Read 10 Done Sent Done false
18287 5638 Acked Flush 0 Done Sent Done false
18288 5639 Acked Read 10 Done Sent Done false
18289 5640 Acked Read 10 Done Sent Done false
18290 5641 Acked Write 4 Done Sent Done false
18291 5642 Acked Read 6 Done Sent Done false
18292 5643 Acked Read 10 Done Sent Done false
18293 5644 Acked Read 6 Done Sent Done false
18294 5645 Acked Write 8 Done Sent Done false
18295 5646 Acked Write 10 Done Sent Done false
18296 5647 Acked Read 3 Done Sent Done false
18297 5648 Acked Write 6 Done Sent Done false
18298 5649 Acked Read 1 Done Sent Done false
18299 5650 Acked Read 3 Done Sent Done false
18300 5651 Acked Read 9 Done Sent Done false
18301 5652 Acked Write 10 Done Sent Done false
18302 5653 Acked Write 6 Done Sent Done false
18303 5654 Acked Read 5 Done Sent Done false
18304 5655 Acked Write 3 Done Sent Done false
18305 5656 Acked Write 8 Done Sent Done false
18306 5657 Acked Flush 0 Done Sent Done false
18307 5658 Acked Read 6 Done Sent Done false
18308 5659 Acked Read 4 Done Sent Done false
18309 5660 Acked Read 7 Done Sent Done false
18310 5661 Acked Read 9 Done Sent Done false
18311 5662 Acked Read 2 Done Sent Done false
18312 5663 Acked Read 7 Done Sent Done false
18313 5664 Acked Read 7 Done Sent Done false
18314 5665 Acked Flush 0 Done Sent Done false
18315 5666 Acked Flush 0 Done Sent Done false
18316 5667 Acked Write 3 Done Sent Done false
18317 5668 Acked Write 10 Done Sent Done false
18318 5669 Acked Write 6 Done Sent Done false
18319 5670 Acked Write 9 Done Sent Done false
18320 5671 Acked Read 5 Done Sent Done false
18321 5672 Acked Read 2 Done Sent Done false
18322 5673 Acked Read 5 Done Sent Done false
18323 5674 Acked Flush 0 Done Sent Done false
18324 5675 Acked Write 10 Done Sent Done false
18325 5676 Acked Write 4 Done Sent Done false
18326 5677 Acked Read 9 Done Sent Done false
18327 5678 Acked Flush 0 Done Sent Done false
18328 5679 Acked Write 7 Done Sent Done false
18329 5680 Acked Read 8 Done Sent Done false
18330 5681 Acked Write 4 Done Sent Done false
18331 5682 Acked Write 2 Done Sent Done false
18332 5683 Acked Read 2 Done Sent Done false
18333 5684 Acked Write 8 Done Sent Done false
18334 5685 Acked Read 3 Done Sent Done false
18335 5686 Acked Read 5 Done Sent Done false
18336 5687 Acked Read 2 Done Sent Done false
18337 5688 Acked Write 10 Done Sent Done false
18338 5689 Acked Write 10 Done Sent Done false
18339 5690 Acked Read 9 Done Sent Done false
18340 5691 Acked Write 1 Done Sent Done false
18341 5692 Acked Write 4 Done Sent Done false
18342 5693 Acked Read 5 Done Sent Done false
18343 5694 Acked Write 7 Done Sent Done false
18344 5695 Acked Write 3 Done Sent Done false
18345 5696 Acked Flush 0 Done Sent Done false
18346 5697 Acked Write 1 Done Sent Done false
18347 5698 Acked Read 1 Done Sent Done false
18348 5699 Acked Read 4 Done Sent Done false
18349 5700 Acked Read 10 Done Sent Done false
18350 5701 Acked Write 1 Done Sent Done false
18351 5702 Acked Write 3 Done Sent Done false
18352 5703 Acked Read 9 Done Sent Done false
18353 5704 Acked Read 5 Done Sent Done false
18354 5705 Acked Write 2 Done Sent Done false
18355 5706 Acked Flush 0 Done Sent Done false
18356 5707 Acked Read 9 Done Sent Done false
18357 5708 Acked Write 2 Done Sent Done false
18358 5709 Acked Read 6 Done Sent Done false
18359 5710 Acked Read 1 Done Sent Done false
18360 5711 Acked Read 1 Done Sent Done false
18361 5712 Acked Read 3 Done Sent Done false
18362 5713 Acked Write 6 Done Sent Done false
18363 5714 Acked Flush 0 Done Sent Done false
18364 5715 Acked Write 7 Done Sent Done false
18365 5716 Acked Read 10 Done Sent Done false
18366 5717 Acked Read 8 Done Sent Done false
18367 5718 Acked Read 6 Done Sent Done false
18368 5719 Acked Read 4 Done Sent Done false
18369 5720 Acked Read 7 Done Sent Done false
18370 5721 Acked Write 6 Done Sent Done false
18371 5722 Acked Write 7 Done Sent Done false
18372 5723 Acked Write 7 Done Sent Done false
18373 5724 Acked Read 9 Done Sent Done false
18374 5725 Acked Write 1 Done Sent Done false
18375 5726 Acked Read 6 Done Sent Done false
18376 5727 Acked Read 2 Done Sent Done false
18377 5728 Acked Flush 0 Done Sent Done false
18378 5729 Acked Write 6 Done Sent Done false
18379 5730 Acked Write 1 Done Sent Done false
18380 5731 Acked Flush 0 Done Sent Done false
18381 5732 Acked Read 8 Done Sent Done false
18382 5733 Acked Read 2 Done Sent Done false
18383 5734 Acked Read 5 Done Sent Done false
18384 5735 Acked Read 2 Done Sent Done false
18385 5736 Acked Read 6 Done Sent Done false
18386 5737 Acked Read 10 Done Sent Done false
18387 5738 Acked Write 5 Done Sent Done false
18388 5739 Acked Read 7 Done Sent Done false
18389 5740 Acked Write 6 Done Sent Done false
18390 5741 Acked Flush 0 Done Sent Done false
18391 5742 Acked Read 4 Done Sent Done false
18392 5743 Acked Write 7 Done Sent Done false
18393 5744 Acked Read 4 Done Sent Done false
18394 5745 Acked Write 3 Done Sent Done false
18395 5746 Acked Read 10 Done Sent Done false
18396 5747 Acked Write 10 Done Sent Done false
18397 5748 Acked Read 6 Done Sent Done false
18398 5749 Acked Write 5 Done Sent Done false
18399 5750 Acked Write 3 Done Sent Done false
18400 5751 Acked Read 9 Done Sent Done false
18401 5752 Acked Read 3 Done Sent Done false
18402 5753 Acked Read 9 Done Sent Done false
18403 5754 Acked Read 1 Done Sent Done false
18404 5755 Acked Flush 0 Done Sent Done false
18405 5756 Acked Write 4 Done Sent Done false
18406 5757 Acked Write 6 Done Sent Done false
18407 5758 Acked Write 1 Done Sent Done false
18408 5759 Acked Write 2 Done Sent Done false
18409 5760 Acked Write 6 Done Sent Done false
18410 5761 Acked Write 5 Done Sent Done false
18411 5762 Acked Write 8 Done Sent Done false
18412 5763 Acked Flush 0 Done Sent Done false
18413 5764 Acked Read 6 Done Sent Done false
18414 5765 Acked Write 4 Done Sent Done false
18415 5766 Acked Write 8 Done Sent Done false
18416 5767 Acked Read 1 Done Sent Done false
18417 5768 Acked Read 5 Done Sent Done false
18418 5769 Acked Read 8 Done Sent Done false
18419 5770 Acked Read 9 Done Sent Done false
18420 5771 Acked Read 2 Done Sent Done false
18421 5772 Acked Read 5 Done Sent Done false
18422 5773 Acked Read 4 Done Sent Done false
18423 5774 Acked Read 6 Done Sent Done false
18424 5775 Acked Read 6 Done Sent Done false
18425 5776 Acked Write 3 Done Sent Done false
18426 5777 Acked Write 6 Done Sent Done false
18427 5778 Acked Read 8 Done Sent Done false
18428 5779 Acked Write 5 Done Sent Done false
18429 5780 Acked Write 9 Done Sent Done false
18430 5781 Acked Write 3 Done Sent Done false
18431 5782 Acked Write 8 Done Sent Done false
18432 5783 Acked Read 3 Done Sent Done false
18433 5784 Acked Read 6 Done Sent Done false
18434 5785 Acked Read 6 Done Sent Done false
18435 5786 Acked Read 4 Done Sent Done false
18436 5787 Acked Read 6 Done Sent Done false
18437 5788 Acked Write 10 Done Sent Done false
18438 5789 Acked Write 1 Done Sent Done false
18439 5790 Acked Read 5 Done Sent Done false
18440 5791 Acked Read 6 Done Sent Done false
18441 5792 Acked Read 10 Done Sent Done false
18442 5793 Acked Write 5 Done Sent Done false
18443 5794 Acked Read 7 Done Sent Done false
18444 5795 Acked Read 8 Done Sent Done false
18445 5796 Acked Read 1 Done Sent Done false
18446 5797 Acked Read 3 Done Sent Done false
18447 5798 Acked Read 7 Done Sent Done false
18448 5799 Acked Write 6 Done Sent Done false
18449 5800 Acked Read 1 Done Sent Done false
18450 5801 Acked Write 10 Done Sent Done false
18451 5802 Acked Read 1 Done Sent Done false
18452 5803 Acked Write 5 Done Sent Done false
18453 5804 Acked Write 9 Done Sent Done false
18454 5805 Acked Read 9 Done Sent Done false
18455 5806 Acked Read 4 Done Sent Done false
18456 5807 Acked Read 4 Done Sent Done false
18457 5808 Acked Read 6 Done Sent Done false
18458 5809 Acked Read 9 Done Sent Done false
18459 5810 Acked Read 1 Done Sent Done false
18460 5811 Acked Write 9 Done Sent Done false
18461 5812 Acked Write 5 Done Sent Done false
18462 5813 Acked Write 3 Done Sent Done false
18463 5814 Acked Read 2 Done Sent Done false
18464 5815 Acked Write 5 Done Sent Done false
18465 5816 Acked Write 7 Done Sent Done false
18466 5817 Acked Write 1 Done Sent Done false
18467 5818 Acked Flush 0 Done Sent Done false
18468 5819 Acked Read 2 Done Sent Done false
18469 5820 Acked Read 8 Done Sent Done false
18470 5821 Acked Read 8 Done Sent Done false
18471 5822 Acked Read 9 Done Sent Done false
18472 5823 Acked Read 1 Done Sent Done false
18473 5824 Acked Write 1 Done Sent Done false
18474 5825 Acked Read 10 Done Sent Done false
18475 5826 Acked Read 4 Done Sent Done false
18476 5827 Acked Read 8 Done Sent Done false
18477 5828 Acked Read 7 Done Sent Done false
18478 5829 Acked Read 8 Done Sent Done false
18479 STATES DS:0 DS:1 DS:2 TOTAL
18480 Sent 0 301 0 301
18481 Done 301 0 301 602
18482 Skipped 0 0 0 0
18483 Error 0 0 0 0
18484 Last Flush: Some(JobId(5818)) Some(JobId(5528)) Some(JobId(5818))
18485 Downstairs last five completed: 5528 5527 5526 5525 5524
18486 Upstairs last five completed: 5829 5828 5827 5826 5825
18487 CLIENT: Up:0 ds:301 act:2
184882025-02-25T17:22:18.776ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
184892025-02-25T17:22:19.665ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
184902025-02-25T17:22:19.797ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
184912025-02-25T17:22:19.797ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34615) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
184922025-02-25T17:22:19.798ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
184932025-02-25T17:22:19.798ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
184942025-02-25T17:22:19.799ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
184952025-02-25T17:22:19.799ZINFOcrucible: send last flush ID to this DS: Some(JobId(5528)) = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
184962025-02-25T17:22:19.799ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
184972025-02-25T17:22:19.799ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5528)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
184982025-02-25T17:22:19.799ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5528)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
18499 ----------------------------------------------------------------
18500 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18501 Downstairs last five completed: 5830 5829 5828 5827 5826
18502 Upstairs last five completed: 5830 5829 5828 5827 5826
18503 CLIENT: Up:0 ds:0 act:3
18504 Replay: All jobs finished
18505 Main task has work to do, go find it
18506 got dsc Stop(1)
18507 stop 1
18508 [8820] Got stop action so:false kr:false
18509 Feb 25 17:22:22.063 INFO request completed, latency_us: 380, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 1a0cd410-e769-4393-b40e-065e7a3b0ff4, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
18510 Feb 25 17:22:22.063 INFO request completed, latency_us: 141, response_code: 200{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:22:22.063965364Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
18511 , uri: /state/cid/1, method: GET, req_id: b007bee7-4336-49ca-88dd-e869b6bba0c4, remote_addr: 127.0{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:22:22.064050322Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
18512 .0.1:52576, local_addr: 127.0.{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:22:22.064100725Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
18513 0.1:9998
185142025-02-25T17:22:22.064ZINFOcrucible: client task is exiting = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
18515 [8820] Exited with: ExitStatus(unix_wait_status(9))
18516 [8820][1] reports Exit
18517 Feb 25 17:22:26.066 INFO request completed, latency_us: 249, response_code: 200, uri: /state/cid/1, method: GET, req_id: b3224935-40bd-4952-aaab-0d1fdd136e73, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
18518 001/300 Read block 1857 len 40960
18519 002/300 Flush
18520 003/300 Write block 896 len 36864 data: 4 4 4 4 3 4 5 5 5
18521 004/300 Write block 1613 len 12288 data: 5 4 4
18522 005/300 Read block 1722 len 40960
18523 006/300 Read block 4691 len 8192
18524 007/300 Read block 2557 len 4096
18525 008/300 Read block 2131 len 4096
18526 009/300 Read block 2183 len 8192
18527 010/300 Read block 621 len 20480
18528 011/300 Read block 1254 len 24576
18529 012/300 Write block 478 len 36864 data: 4 4 4 4 5 5 5 5 4
18530 013/300 Write block 1191 len 8192 data: 4 4
18531 014/300 Write block 4158 len 16384 data: 5 5 4 3
18532 015/300 Read block 4683 len 40960
18533 016/300 Write block 4630 len 28672 data: 6 5 5 5 6 7 6
18534 017/300 Flush
18535 018/300 Flush
18536 019/300 Read block 2011 len 32768
18537 020/300 Write block 2056 len 24576 data: 4 5 5 4 4 4
18538 021/300 Read block 2718 len 20480
18539 022/300 Write block 3581 len 32768 data: 3 3 3 5 4 4 4 4
18540 023/300 Write block 451 len 28672 data: 3 3 4 4 5 6 6
18541 024/300 Write block 3825 len 36864 data: 3 3 2 3 3 3 3 3 4
18542 025/300 Read block 2014 len 40960
18543 026/300 Read block 1230 len 28672
18544 027/300 Read block 3888 len 12288
18545 028/300 Read block 667 len 40960
18546 029/300 Write block 2829 len 28672 data: 4 5 5 5 6 5 5
18547 030/300 Read block 1945 len 12288
18548 031/300 Write block 2255 len 32768 data: 3 4 5 4 4 5 6 5
18549 032/300 Write block 509 len 16384 data: 4 5 4 5
18550 033/300 Write block 1616 len 24576 data: 5 5 5 5 4 4
18551 034/300 Write block 551 len 32768 data: 4 4 4 4 4 4 4 3
18552 035/300 Read block 4523 len 32768
18553 036/300 Read block 1229 len 40960
18554 037/300 Write block 2825 len 4096 data: 5
18555 038/300 Read block 4130 len 12288
18556 039/300 Write block 3229 len 16384 data: 3 3 3 3
18557 040/300 Flush
18558 041/300 Write block 2671 len 4096 data: 3
18559 042/300 Read block 3450 len 8192
18560 043/300 Write block 4790 len 4096 data: 4
18561 044/300 Write block 2249 len 36864 data: 4 4 4 2 2 3 4 5 6
18562 045/300 Read block 1424 len 28672
18563 046/300 Write block 3935 len 4096 data: 3
18564 047/300 Read block 4361 len 28672
18565 048/300 Read block 2000 len 20480
18566 049/300 Read block 1896 len 28672
18567 050/300 Read block 735 len 24576
18568 051/300 Flush
18569 052/300 Read block 123 len 4096
18570 053/300 Write block 85 len 24576 data: 4 4 4 4 4 3
18571 054/300 Write block 1829 len 16384 data: 3 3 5 3
18572 055/300 Write block 1989 len 28672 data: 4 3 4 3 2 3 3
18573 056/300 Read block 1439 len 32768
18574 057/300 Read block 1245 len 40960
18575 058/300 Write block 2982 len 36864 data: 2 2 2 2 2 2 3 3 3
18576 059/300 Read block 4230 len 40960
18577 060/300 Write block 3521 len 24576 data: 5 4 4 3 3 3
18578 061/300 Read block 884 len 8192
18579 062/300 Read block 4780 len 4096
18580 063/300 Write block 2470 len 8192 data: 3 3
18581 064/300 Write block 4299 len 24576 data: 3 4 5 4 5 5
18582 065/300 Read block 2215 len 8192
18583 066/300 Read block 2868 len 4096
18584 067/300 Read block 489 len 24576
18585 068/300 Write block 4543 len 20480 data: 3 3 3 3 3
18586 069/300 Read block 973 len 4096
18587 070/300 Read block 4053 len 24576
18588 071/300 Write block 4536 len 16384 data: 5 6 6 4
18589 072/300 Read block 650 len 8192
18590 073/300 Write block 2136 len 28672 data: 6 6 7 6 6 6 6
18591 074/300 Flush
18592 075/300 Write block 4359 len 16384 data: 5 5 5 3
18593 076/300 Read block 2315 len 8192
18594 077/300 Write block 2143 len 8192 data: 6 6
18595 078/300 Write block 3617 len 32768 data: 3 3 3 4 3 3 3 4
18596 079/300 Write block 1471 len 40960 data: 5 5 5 4 3 3 4 4 5 6
18597 080/300 Write block 3706 len 36864 data: 5 4 4 5 6 5 4 5 5
18598 081/300 Write block 4390 len 20480 data: 5 6 6 5 5
18599 082/300 Read block 4549 len 12288
18600 083/300 Read block 3106 len 28672
18601 084/300 Read block 3156 len 16384
18602 085/300 Write block 3145 len 20480 data: 3 3 3 3 3
18603 086/300 Write block 3746 len 8192 data: 3 3
18604 087/300 Read block 1364 len 36864
18605 088/300 Write block 3919 len 8192 data: 5 6
18606 089/300 Flush
18607 090/300 Read block 3691 len 36864
18608 091/300 Write block 3670 len 36864 data: 7 5 4 4 5 5 5 4 4
18609 092/300 Write block 4929 len 36864 data: 2 2 3 3 4 4 4 4 4
18610 093/300 Read block 3885 len 36864
18611 094/300 Read block 4751 len 4096
18612 095/300 Write block 3715 len 24576 data: 6 6 7 6 6 4
18613 096/300 Write block 2094 len 24576 data: 7 7 7 6 6 6
18614 097/300 Flush
18615 098/300 Read block 3061 len 32768
18616 099/300 Read block 1839 len 24576
18617 100/300 Read block 797 len 40960
18618 101/300 Read block 2821 len 24576
18619 102/300 Read block 3347 len 4096
18620 103/300 Read block 4758 len 12288
18621 104/300 Read block 2039 len 20480
18622 105/300 Read block 392 len 16384
18623 106/300 Read block 4033 len 28672
18624 107/300 Read block 2505 len 40960
18625 108/300 Read block 2719 len 40960
18626 109/300 Flush
18627 110/300 Write block 4280 len 8192 data: 3 3
18628 111/300 Write block 599 len 24576 data: 5 5 6 5 5 5
18629 112/300 Read block 3003 len 36864
18630 113/300 Write block 2039 len 20480 data: 6 5 5 6 7
18631 114/300 Read block 4668 len 8192
18632 115/300 Write block 1752 len 12288 data: 3 3 4
18633 116/300 Read block 2738 len 20480
18634 117/300 Read block 757 len 36864
18635 118/300 Read block 3241 len 32768
18636 119/300 Read block 302 len 8192
18637 120/300 Write block 4451 len 32768 data: 2 3 3 3 2 3 3 3
18638 121/300 Read block 4047 len 20480
18639 122/300 Read block 2422 len 4096
18640 123/300 Write block 2696 len 24576 data: 4 4 3 4 4 4
18641 124/300 Write block 3554 len 12288 data: 8 6 6
18642 125/300 Flush
18643 126/300 Write block 1562 len 16384 data: 2 2 2 2
18644 127/300 Write block 4715 len 16384 data: 4 4 4 6
18645 128/300 Flush
18646 129/300 Write block 969 len 20480 data: 5 5 4 3 2
18647 130/300 Write block 2963 len 8192 data: 4 5
18648 131/300 Write block 1501 len 32768 data: 4 4 5 5 6 5 5 5
18649 132/300 Read block 4672 len 4096
18650 133/300 Flush
18651 134/300 Flush
18652 135/300 Write block 137 len 16384 data: 4 4 3 3
18653 136/300 Read block 3162 len 16384
18654 137/300 Write block 620 len 12288 data: 4 6 6
18655 138/300 Read block 4315 len 12288
18656 139/300 Write block 1485 len 28672 data: 6 6 5 5 6 6 5
18657 140/300 Read block 2657 len 4096
18658 141/300 Read block 2903 len 12288
18659 142/300 Read block 1834 len 20480
18660 143/300 Read block 4634 len 8192
18661 144/300 Flush
18662 145/300 Flush
18663 146/300 Write block 4641 len 28672 data: 3 4 5 4 3 3 3
18664 147/300 Read block 1796 len 4096
18665 148/300 Write block 225 len 12288 data: 2 2 2
18666 149/300 Write block 4658 len 28672 data: 3 3 3 6 6 7 7
18667 150/300 Read block 250 len 4096
18668 151/300 Write block 650 len 40960 data: 2 3 3 4 4 4 3 2 2 2
18669 152/300 Write block 3692 len 20480 data: 2 2 2 2 2
18670 153/300 Write block 2262 len 20480 data: 6 5 4 4 2
18671 154/300 Read block 2364 len 20480
18672 155/300 Read block 1111 len 4096
18673 156/300 Read block 639 len 40960
18674 157/300 Write block 102 len 16384 data: 3 3 3 4
18675 158/300 Write block 402 len 36864 data: 4 4 4 5 4 4 3 3 5
18676 159/300 Read block 4796 len 4096
18677 160/300 Read block 626 len 36864
18678 161/300 Read block 2860 len 16384
18679 162/300 Write block 1784 len 8192 data: 2 2
18680 163/300 Read block 2322 len 8192
18681 164/300 Read block 1415 len 8192
18682 165/300 Read block 1034 len 40960
18683 166/300 Read block 255 len 8192
18684 167/300 Read block 3615 len 12288
18685 168/300 Read block 1982 len 16384
18686 169/300 Write block 2871 len 8192 data: 4 4
18687 170/300 Read block 494 len 32768
18688 171/300 Read block 3352 len 20480
18689 172/300 Read block 2660 len 20480
18690 173/300 Write block 333 len 4096 data: 4
18691 174/300 Write block 4872 len 20480 data: 4 4 4 4 4
18692 175/300 Read block 1297 len 20480
18693 176/300 Read block 3767 len 24576
18694 177/300 Read block 864 len 40960
18695 178/300 Write block 2603 len 12288 data: 5 4 3
18696 179/300 Write block 4720 len 8192 data: 7 7
18697 180/300 Read block 3026 len 12288
18698 181/300 Write block 2439 len 32768 data: 5 3 2 2 2 2 3 4
18699 182/300 Write block 2812 len 4096 data: 3
18700 183/300 Read block 2519 len 40960
18701 184/300 Write block 2994 len 28672 data: 6 6 5 3 6 6 6
18702 185/300 Read block 230 len 32768
18703 186/300 Write block 4565 len 28672 data: 5 3 3 4 4 4 4
18704 187/300 Read block 4252 len 20480
18705 188/300 Flush
18706 189/300 Write block 3592 len 4096 data: 3
18707 190/300 Read block 3492 len 20480
18708 191/300 Write block 3418 len 20480 data: 4 4 4 2 2
18709 192/300 Read block 3362 len 40960
18710 193/300 Write block 559 len 12288 data: 3 3 3
18711 194/300 Read block 1144 len 36864
18712 195/300 Read block 1149 len 4096
18713 196/300 Flush
18714 197/300 Write block 4695 len 12288 data: 3 4 5
18715 198/300 Flush
18716 199/300 Write block 853 len 12288 data: 4 3 4
18717 200/300 Write block 2891 len 28672 data: 2 2 3 6 5 4 4
18718 201/300 Write block 2012 len 28672 data: 5 5 5 4 4 4 5
18719 202/300 Write block 1014 len 16384 data: 3 3 2 2
18720 203/300 Write block 2487 len 24576 data: 3 2 3 3 3 3
18721 204/300 Write block 51 len 8192 data: 3 4
18722 205/300 Read block 2167 len 16384
18723 206/300 Read block 2938 len 20480
18724 207/300 Flush
18725 208/300 Write block 1462 len 32768 data: 4 4 4 4 4 5 5 6
18726 209/300 Write block 1842 len 28672 data: 4 3 4 3 3 3 3
18727 210/300 Read block 2773 len 32768
18728 211/300 Write block 1979 len 24576 data: 5 4 4 4 5 4
18729 212/300 Read block 2887 len 40960
18730 213/300 Read block 4818 len 12288
18731 214/300 Write block 3570 len 28672 data: 2 2 2 3 2 2 2
18732 215/300 Write block 4148 len 12288 data: 5 5 5
18733 216/300 Write block 1039 len 20480 data: 3 4 6 5 5
18734 217/300 Write block 3281 len 40960 data: 3 2 5 5 5 5 5 4 4 3
18735 218/300 Write block 2030 len 36864 data: 5 6 6 6 6 6 6 7 6
18736 219/300 Read block 1359 len 40960
18737 220/300 Write block 3878 len 20480 data: 5 5 4 3 2
18738 221/300 Write block 1152 len 16384 data: 3 3 3 3
18739 222/300 Write block 1076 len 8192 data: 3 3
18740 223/300 Flush
18741 224/300 Read block 4432 len 12288
18742 225/300 Write block 2828 len 4096 data: 4
18743 226/300 Read block 1458 len 24576
18744 227/300 Write block 2583 len 36864 data: 5 5 3 2 2 2 4 5 5
18745 228/300 Write block 225 len 20480 data: 3 3 3 3 4
18746 229/300 Read block 2322 len 20480
18747 230/300 Write block 4826 len 12288 data: 3 3 3
18748 231/300 Read block 3541 len 40960
18749 232/300 Read block 2047 len 32768
18750 233/300 Read block 2953 len 40960
18751 234/300 Write block 4829 len 16384 data: 3 2 2 2
18752 235/300 Read block 2663 len 24576
18753 236/300 Write block 3572 len 8192 data: 3 4
18754 237/300 Read block 2289 len 28672
18755 238/300 Write block 2134 len 8192 data: 5 5
18756 239/300 Write block 808 len 16384 data: 2 3 3 3
18757 240/300 Read block 1143 len 40960
18758 241/300 Read block 257 len 16384
18759 242/300 Read block 3799 len 4096
18760 243/300 Write block 2819 len 16384 data: 2 3 3 4
18761 244/300 Write block 794 len 16384 data: 2 2 3 3
18762 245/300 Read block 4520 len 8192
18763 246/300 Read block 1010 len 40960
18764 247/300 Read block 1596 len 4096
18765 248/300 Read block 23 len 40960
18766 249/300 Write block 4059 len 16384 data: 3 3 5 5
18767 250/300 Write block 2763 len 12288 data: 6 7 8
18768 251/300 Read block 3499 len 24576
18769 252/300 Write block 1973 len 16384 data: 2 4 5 5
18770 253/300 Write block 4133 len 32768 data: 5 6 5 4 3 3 4 4
18771 254/300 Write block 1907 len 20480 data: 4 5 5 7 5
18772 255/300 Write block 1003 len 12288 data: 3 4 5
18773 256/300 Read block 113 len 16384
18774 257/300 Write block 1565 len 8192 data: 3 2
18775 258/300 Flush
18776 259/300 Write block 1767 len 16384 data: 4 6 6 4
18777 260/300 Write block 167 len 32768 data: 3 3 3 3 2 2 2 2
18778 261/300 Write block 4340 len 32768 data: 5 4 4 4 4 5 5 5
18779 262/300 Flush
18780 263/300 Read block 352 len 32768
18781 264/300 Read block 3970 len 28672
18782 265/300 Read block 2778 len 40960
18783 266/300 Read block 4475 len 36864
18784 267/300 Write block 2506 len 12288 data: 5 5 5
18785 268/300 Read block 1396 len 4096
18786 269/300 Write block 1602 len 8192 data: 4 4
18787 270/300 Read block 2728 len 36864
18788 271/300 Read block 598 len 32768
18789 272/300 Flush
18790 273/300 Flush
18791 274/300 Write block 13 len 36864 data: 6 6 5 6 4 5 4 3 4
18792 275/300 Write block 4520 len 20480 data: 8 6 6 4 3
18793 276/300 Read block 3845 len 12288
18794 277/300 Flush
18795 278/300 Read block 3388 len 32768
18796 279/300 Write block 2272 len 40960 data: 2 2 3 3 3 4 4 5 5 5
18797 280/300 Read block 2877 len 40960
18798 281/300 Read block 4944 len 40960
18799 282/300 Write block 4201 len 12288 data: 4 5 6
18800 283/300 Write block 1480 len 28672 data: 7 7 6 7 6 7 7
18801 284/300 Read block 1414 len 40960
18802 285/300 Read block 1620 len 24576
18803 286/300 Write block 4327 len 28672 data: 3 3 3 3 3 2 3
18804 287/300 Read block 2221 len 16384
18805 288/300 Read block 225 len 36864
18806 289/300 Read block 1033 len 16384
18807 290/300 Read block 3129 len 8192
18808 291/300 Read block 2096 len 12288
18809 292/300 Write block 2183 len 36864 data: 5 5 5 3 2 2 3 4 5
18810 293/300 Read block 2869 len 16384
18811 294/300 Read block 2008 len 8192
18812 295/300 Write block 2179 len 32768 data: 6 5 4 4 6 6 6 4
18813 296/300 Write block 2136 len 32768 data: 7 7 8 7 7 7 7 7
18814 297/300 Read block 3851 len 28672
18815 298/300 Write block 788 len 16384 data: 5 5 4 4
18816 299/300 Read block 1440 len 20480
18817 300/300 Read block 4799 len 28672
18818 301/300 Read block 868 len 8192
18819 Main task has work to do, go find it
18820 got dsc Start(1)
18821 start 1
18822 Feb 25 17:22:26.311 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/1, method: POST, req_id: 523241a9-4917-44f7-9cfd-27e92243d8cd, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
18823 Replay: started 1, returned:Ok(())
18824 ----------------------------------------------------------------
18825 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18826 States: Active Offline Active
18827 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18828 5831 Acked Read 10 Done Sent Done false
18829 5832 Acked Flush 0 Done Sent Done false
18830 5833 Acked Write 9 Done Sent Done false
18831 5834 Acked Write 3 Done Sent Done false
18832 5835 Acked Read 10 Done Sent Done false
18833 5836 Acked Read 2 Done Sent Done false
18834 5837 Acked Read 1 Done Sent Done false
18835 5838 Acked Read 1 Done Sent Done false
18836 5839 Acked Read 2 Done Sent Done false
18837 5840 Acked Read 5 Done Sent Done false
18838 5841 Acked Read 6 Done Sent Done false
18839 5842 Acked Write 9 Done Sent Done false
18840 5843 Acked Write 2 Done Sent Done false
18841 5844 Acked Write 4 Done Sent Done false
18842 5845 Acked Read 10 Done Sent Done false
18843 5846 Acked Write 7 Done Sent Done false
18844 5847 Acked Flush 0 Done Sent Done false
18845 5848 Acked Flush 0 Done Sent Done false
18846 5849 Acked Read 8 Done Sent Done false
18847 5850 Acked Write 6 Done Sent Done false
18848 5851 Acked Read 5 Done Sent Done false
18849 5852 Acked Write 8 Done Sent Done false
18850 5853 Acked Write 7 Done Sent Done false
18851 5854 Acked Write 9 Done Sent Done false
18852 5855 Acked Read 10 Done Sent Done false
18853 5856 Acked Read 7 Done Sent Done false
18854 5857 Acked Read 3 Done Sent Done false
18855 5858 Acked Read 10 Done Sent Done false
18856 5859 Acked Write 7 Done Sent Done false
18857 5860 Acked Read 3 Done Sent Done false
18858 5861 Acked Write 8 Done Sent Done false
18859 5862 Acked Write 4 Done Sent Done false
18860 5863 Acked Write 6 Done Sent Done false
18861 5864 Acked Write 8 Done Sent Done false
18862 5865 Acked Read 8 Done Sent Done false
18863 5866 Acked Read 10 Done Sent Done false
18864 5867 Acked Write 1 Done Sent Done false
18865 5868 Acked Read 3 Done Sent Done false
18866 5869 Acked Write 4 Done Sent Done false
18867 5870 Acked Flush 0 Done Sent Done false
18868 5871 Acked Write 1 Done Sent Done false
18869 5872 Acked Read 2 Done Sent Done false
18870 5873 Acked Write 1 Done Sent Done false
18871 5874 Acked Write 9 Done Sent Done false
18872 5875 Acked Read 7 Done Sent Done false
18873 5876 Acked Write 1 Done Sent Done false
18874 5877 Acked Read 7 Done Sent Done false
18875 5878 Acked Read 5 Done Sent Done false
18876 5879 Acked Read 7 Done Sent Done false
18877 5880 Acked Read 6 Done Sent Done false
18878 5881 Acked Flush 0 Done Sent Done false
18879 5882 Acked Read 1 Done Sent Done false
18880 5883 Acked Write 6 Done Sent Done false
18881 5884 Acked Write 4 Done Sent Done false
18882 5885 Acked Write 7 Done Sent Done false
18883 5886 Acked Read 8 Done Sent Done false
18884 5887 Acked Read 10 Done Sent Done false
18885 5888 Acked Write 9 Done Sent Done false
18886 5889 Acked Read 10 Done Sent Done false
18887 5890 Acked Write 6 Done Sent Done false
18888 5891 Acked Read 2 Done Sent Done false
18889 5892 Acked Read 1 Done Sent Done false
18890 5893 Acked Write 2 Done Sent Done false
18891 5894 Acked Write 6 Done Sent Done false
18892 5895 Acked Read 2 Done Sent Done false
18893 5896 Acked Read 1 Done Sent Done false
18894 5897 Acked Read 6 Done Sent Done false
18895 5898 Acked Write 5 Done Sent Done false
18896 5899 Acked Read 1 Done Sent Done false
18897 5900 Acked Read 6 Done Sent Done false
18898 5901 Acked Write 4 Done Sent Done false
18899 5902 Acked Read 2 Done Sent Done false
18900 5903 Acked Write 7 Done Sent Done false
18901 5904 Acked Flush 0 Done Sent Done false
18902 5905 Acked Write 4 Done Sent Done false
18903 5906 Acked Read 2 Done Sent Done false
18904 5907 Acked Write 2 Done Sent Done false
18905 5908 Acked Write 8 Done Sent Done false
18906 5909 Acked Write 10 Done Sent Done false
18907 5910 Acked Write 9 Done Sent Done false
18908 5911 Acked Write 5 Done Sent Done false
18909 5912 Acked Read 3 Done Sent Done false
18910 5913 Acked Read 7 Done Sent Done false
18911 5914 Acked Read 4 Done Sent Done false
18912 5915 Acked Write 5 Done Sent Done false
18913 5916 Acked Write 2 Done Sent Done false
18914 5917 Acked Read 9 Done Sent Done false
18915 5918 Acked Write 2 Done Sent Done false
18916 5919 Acked Flush 0 Done Sent Done false
18917 5920 Acked Read 9 Done Sent Done false
18918 5921 Acked Write 9 Done Sent Done false
18919 5922 Acked Write 9 Done Sent Done false
18920 5923 Acked Read 9 Done Sent Done false
18921 5924 Acked Read 1 Done Sent Done false
18922 5925 Acked Write 6 Done Sent Done false
18923 5926 Acked Write 6 Done Sent Done false
18924 5927 Acked Flush 0 Done Sent Done false
18925 5928 Acked Read 8 Done Sent Done false
18926 5929 Acked Read 6 Done Sent Done false
18927 5930 Acked Read 10 Done Sent Done false
18928 5931 Acked Read 6 Done Sent Done false
18929 5932 Acked Read 1 Done Sent Done false
18930 5933 Acked Read 3 Done Sent Done false
18931 5934 Acked Read 5 Done Sent Done false
18932 5935 Acked Read 4 Done Sent Done false
18933 5936 Acked Read 7 Done Sent Done false
18934 5937 Acked Read 10 Done Sent Done false
18935 5938 Acked Read 10 Done Sent Done false
18936 5939 Acked Flush 0 Done Sent Done false
18937 5940 Acked Write 2 Done Sent Done false
18938 5941 Acked Write 6 Done Sent Done false
18939 5942 Acked Read 9 Done Sent Done false
18940 5943 Acked Write 5 Done Sent Done false
18941 5944 Acked Read 2 Done Sent Done false
18942 5945 Acked Write 3 Done Sent Done false
18943 5946 Acked Read 5 Done Sent Done false
18944 5947 Acked Read 9 Done Sent Done false
18945 5948 Acked Read 8 Done Sent Done false
18946 5949 Acked Read 2 Done Sent Done false
18947 5950 Acked Write 8 Done Sent Done false
18948 5951 Acked Read 5 Done Sent Done false
18949 5952 Acked Read 1 Done Sent Done false
18950 5953 Acked Write 6 Done Sent Done false
18951 5954 Acked Write 3 Done Sent Done false
18952 5955 Acked Flush 0 Done Sent Done false
18953 5956 Acked Write 4 Done Sent Done false
18954 5957 Acked Write 4 Done Sent Done false
18955 5958 Acked Flush 0 Done Sent Done false
18956 5959 Acked Write 5 Done Sent Done false
18957 5960 Acked Write 2 Done Sent Done false
18958 5961 Acked Write 8 Done Sent Done false
18959 5962 Acked Read 1 Done Sent Done false
18960 5963 Acked Flush 0 Done Sent Done false
18961 5964 Acked Flush 0 Done Sent Done false
18962 5965 Acked Write 4 Done Sent Done false
18963 5966 Acked Read 4 Done Sent Done false
18964 5967 Acked Write 3 Done Sent Done false
18965 5968 Acked Read 3 Done Sent Done false
18966 5969 Acked Write 7 Done Sent Done false
18967 5970 Acked Read 1 Done Sent Done false
18968 5971 Acked Read 3 Done Sent Done false
18969 5972 Acked Read 5 Done Sent Done false
18970 5973 Acked Read 2 Done Sent Done false
18971 5974 Acked Flush 0 Done Sent Done false
18972 5975 Acked Flush 0 Done Sent Done false
18973 5976 Acked Write 7 Done Sent Done false
18974 5977 Acked Read 1 Done Sent Done false
18975 5978 Acked Write 3 Done Sent Done false
18976 5979 Acked Write 7 Done Sent Done false
18977 5980 Acked Read 1 Done Sent Done false
18978 5981 Acked Write 10 Done Sent Done false
18979 5982 Acked Write 5 Done Sent Done false
18980 5983 Acked Write 5 Done Sent Done false
18981 5984 Acked Read 5 Done Sent Done false
18982 5985 Acked Read 1 Done Sent Done false
18983 5986 Acked Read 10 Done Sent Done false
18984 5987 Acked Write 4 Done Sent Done false
18985 5988 Acked Write 9 Done Sent Done false
18986 5989 Acked Read 1 Done Sent Done false
18987 5990 Acked Read 9 Done Sent Done false
18988 5991 Acked Read 4 Done Sent Done false
18989 5992 Acked Write 2 Done Sent Done false
18990 5993 Acked Read 2 Done Sent Done false
18991 5994 Acked Read 2 Done Sent Done false
18992 5995 Acked Read 10 Done Sent Done false
18993 5996 Acked Read 2 Done Sent Done false
18994 5997 Acked Read 3 Done Sent Done false
18995 5998 Acked Read 4 Done Sent Done false
18996 5999 Acked Write 2 Done Sent Done false
18997 6000 Acked Read 8 Done Sent Done false
18998 6001 Acked Read 5 Done Sent Done false
18999 6002 Acked Read 5 Done Sent Done false
19000 6003 Acked Write 1 Done Sent Done false
19001 6004 Acked Write 5 Done Sent Done false
19002 6005 Acked Read 5 Done Sent Done false
19003 6006 Acked Read 6 Done Sent Done false
19004 6007 Acked Read 10 Done Sent Done false
19005 6008 Acked Write 3 Done Sent Done false
19006 6009 Acked Write 2 Done Sent Done false
19007 6010 Acked Read 3 Done Sent Done false
19008 6011 Acked Write 8 Done Sent Done false
19009 6012 Acked Write 1 Done Sent Done false
19010 6013 Acked Read 10 Done Sent Done false
19011 6014 Acked Write 7 Done Sent Done false
19012 6015 Acked Read 8 Done Sent Done false
19013 6016 Acked Write 7 Done Sent Done false
19014 6017 Acked Read 5 Done Sent Done false
19015 6018 Acked Flush 0 Done Sent Done false
19016 6019 Acked Write 1 Done Sent Done false
19017 6020 Acked Read 5 Done Sent Done false
19018 6021 Acked Write 5 Done Sent Done false
19019 6022 Acked Read 10 Done Sent Done false
19020 6023 Acked Write 3 Done Sent Done false
19021 6024 Acked Read 9 Done Sent Done false
19022 6025 Acked Read 1 Done Sent Done false
19023 6026 Acked Flush 0 Done Sent Done false
19024 6027 Acked Write 3 Done Sent Done false
19025 6028 Acked Flush 0 Done Sent Done false
19026 6029 Acked Write 3 Done Sent Done false
19027 6030 Acked Write 7 Done Sent Done false
19028 6031 Acked Write 7 Done Sent Done false
19029 6032 Acked Write 4 Done Sent Done false
19030 6033 Acked Write 6 Done Sent Done false
19031 6034 Acked Write 2 Done Sent Done false
19032 6035 Acked Read 4 Done Sent Done false
19033 6036 Acked Read 5 Done Sent Done false
19034 6037 Acked Flush 0 Done Sent Done false
19035 6038 Acked Write 8 Done Sent Done false
19036 6039 Acked Write 7 Done Sent Done false
19037 6040 Acked Read 8 Done Sent Done false
19038 6041 Acked Write 6 Done Sent Done false
19039 6042 Acked Read 10 Done Sent Done false
19040 6043 Acked Read 3 Done Sent Done false
19041 6044 Acked Write 7 Done Sent Done false
19042 6045 Acked Write 3 Done Sent Done false
19043 6046 Acked Write 5 Done Sent Done false
19044 6047 Acked Write 10 Done Sent Done false
19045 6048 Acked Write 9 Done Sent Done false
19046 6049 Acked Read 10 Done Sent Done false
19047 6050 Acked Write 5 Done Sent Done false
19048 6051 Acked Write 4 Done Sent Done false
19049 6052 Acked Write 2 Done Sent Done false
19050 6053 Acked Flush 0 Done Sent Done false
19051 6054 Acked Read 3 Done Sent Done false
19052 6055 Acked Write 1 Done Sent Done false
19053 6056 Acked Read 6 Done Sent Done false
19054 6057 Acked Write 9 Done Sent Done false
19055 6058 Acked Write 5 Done Sent Done false
19056 6059 Acked Read 5 Done Sent Done false
19057 6060 Acked Write 3 Done Sent Done false
19058 6061 Acked Read 10 Done Sent Done false
19059 6062 Acked Read 8 Done Sent Done false
19060 6063 Acked Read 10 Done Sent Done false
19061 6064 Acked Write 4 Done Sent Done false
19062 6065 Acked Read 6 Done Sent Done false
19063 6066 Acked Write 2 Done Sent Done false
19064 6067 Acked Read 7 Done Sent Done false
19065 6068 Acked Write 2 Done Sent Done false
19066 6069 Acked Write 4 Done Sent Done false
19067 6070 Acked Read 10 Done Sent Done false
19068 6071 Acked Read 4 Done Sent Done false
19069 6072 Acked Read 1 Done Sent Done false
19070 6073 Acked Write 4 Done Sent Done false
19071 6074 Acked Write 4 Done Sent Done false
19072 6075 Acked Read 2 Done Sent Done false
19073 6076 Acked Read 10 Done Sent Done false
19074 6077 Acked Read 1 Done Sent Done false
19075 6078 Acked Read 10 Done Sent Done false
19076 6079 Acked Write 4 Done Sent Done false
19077 6080 Acked Write 3 Done Sent Done false
19078 6081 Acked Read 6 Done Sent Done false
19079 6082 Acked Write 4 Done Sent Done false
19080 6083 Acked Write 8 Done Sent Done false
19081 6084 Acked Write 5 Done Sent Done false
19082 6085 Acked Write 3 Done Sent Done false
19083 6086 Acked Read 4 Done Sent Done false
19084 6087 Acked Write 2 Done Sent Done false
19085 6088 Acked Flush 0 Done Sent Done false
19086 6089 Acked Write 4 Done Sent Done false
19087 6090 Acked Write 8 Done Sent Done false
19088 6091 Acked Write 8 Done Sent Done false
19089 6092 Acked Flush 0 Done Sent Done false
19090 6093 Acked Read 8 Done Sent Done false
19091 6094 Acked Read 7 Done Sent Done false
19092 6095 Acked Read 10 Done Sent Done false
19093 6096 Acked Read 9 Done Sent Done false
19094 6097 Acked Write 3 Done Sent Done false
19095 6098 Acked Read 1 Done Sent Done false
19096 6099 Acked Write 2 Done Sent Done false
19097 6100 Acked Read 9 Done Sent Done false
19098 6101 Acked Read 8 Done Sent Done false
19099 6102 Acked Flush 0 Done Sent Done false
19100 6103 Acked Flush 0 Done Sent Done false
19101 6104 Acked Write 9 Done Sent Done false
19102 6105 Acked Write 5 Done Sent Done false
19103 6106 Acked Read 3 Done Sent Done false
19104 6107 Acked Flush 0 Done Sent Done false
19105 6108 Acked Read 8 Done Sent Done false
19106 6109 Acked Write 10 Done Sent Done false
19107 6110 Acked Read 10 Done Sent Done false
19108 6111 Acked Read 10 Done Sent Done false
19109 6112 Acked Write 3 Done Sent Done false
19110 6113 Acked Write 7 Done Sent Done false
19111 6114 Acked Read 10 Done Sent Done false
19112 6115 Acked Read 6 Done Sent Done false
19113 6116 Acked Write 7 Done Sent Done false
19114 6117 Acked Read 4 Done Sent Done false
19115 6118 Acked Read 9 Done Sent Done false
19116 6119 Acked Read 4 Done Sent Done false
19117 6120 Acked Read 2 Done Sent Done false
19118 6121 Acked Read 3 Done Sent Done false
19119 6122 Acked Write 9 Done Sent Done false
19120 6123 Acked Read 4 Done Sent Done false
19121 6124 Acked Read 2 Done Sent Done false
19122 6125 Acked Write 8 Done Sent Done false
19123 6126 Acked Write 8 Done Sent Done false
19124 6127 Acked Read 7 Done Sent Done false
19125 6128 Acked Write 4 Done Sent Done false
19126 6129 Acked Read 5 Done Sent Done false
19127 6130 Acked Read 7 Done Sent Done false
19128 6131 Acked Read 2 Done Sent Done false
19129 STATES DS:0 DS:1 DS:2 TOTAL
19130 Sent 0 301 0 301
19131 Done 301 0 301 602
19132 Skipped 0 0 0 0
19133 Error 0 0 0 0
19134 Last Flush: Some(JobId(6107)) Some(JobId(5830)) Some(JobId(6107))
19135 Downstairs last five completed: 5830 5829 5828 5827 5826
19136 Upstairs last five completed: 6131 6130 6129 6128 6127
19137 CLIENT: Up:0 ds:301 act:2
191382025-02-25T17:22:26.813ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
191392025-02-25T17:22:27.679ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
191402025-02-25T17:22:28.779ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
191412025-02-25T17:22:29.682ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
19142 [8820] got start action
19143 [8820] I am going to restart
19144 Starting downstairs at port 8820
19145 Make output file at "/tmp/dsc/downstairs-8820.txt"
19146 [8820][1] reports Starting
19147 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1225)
19148 [8820][1] reports Running
19149 ----------------------------------------------------------------
19150 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19151 States: Active Offline Active
19152 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19153 5831 Acked Read 10 Done Sent Done false
19154 5832 Acked Flush 0 Done Sent Done false
19155 5833 Acked Write 9 Done Sent Done false
19156 5834 Acked Write 3 Done Sent Done false
19157 5835 Acked Read 10 Done Sent Done false
19158 5836 Acked Read 2 Done Sent Done false
19159 5837 Acked Read 1 Done Sent Done false
19160 5838 Acked Read 1 Done Sent Done false
19161 5839 Acked Read 2 Done Sent Done false
19162 5840 Acked Read 5 Done Sent Done false
19163 5841 Acked Read 6 Done Sent Done false
19164 5842 Acked Write 9 Done Sent Done false
19165 5843 Acked Write 2 Done Sent Done false
19166 5844 Acked Write 4 Done Sent Done false
19167 5845 Acked Read 10 Done Sent Done false
19168 5846 Acked Write 7 Done Sent Done false
19169 5847 Acked Flush 0 Done Sent Done false
19170 5848 Acked Flush 0 Done Sent Done false
19171 5849 Acked Read 8 Done Sent Done false
19172 5850 Acked Write 6 Done Sent Done false
19173 5851 Acked Read 5 Done Sent Done false
19174 5852 Acked Write 8 Done Sent Done false
19175 5853 Acked Write 7 Done Sent Done false
19176 5854 Acked Write 9 Done Sent Done false
19177 5855 Acked Read 10 Done Sent Done false
19178 5856 Acked Read 7 Done Sent Done false
19179 5857 Acked Read 3 Done Sent Done false
19180 5858 Acked Read 10 Done Sent Done false
19181 5859 Acked Write 7 Done Sent Done false
19182 5860 Acked Read 3 Done Sent Done false
19183 5861 Acked Write 8 Done Sent Done false
19184 5862 Acked Write 4 Done Sent Done false
19185 5863 Acked Write 6 Done Sent Done false
19186 5864 Acked Write 8 Done Sent Done false
19187 5865 Acked Read 8 Done Sent Done false
19188 5866 Acked Read 10 Done Sent Done false
19189 5867 Acked Write 1 Done Sent Done false
19190 5868 Acked Read 3 Done Sent Done false
19191 5869 Acked Write 4 Done Sent Done false
19192 5870 Acked Flush 0 Done Sent Done false
19193 5871 Acked Write 1 Done Sent Done false
19194 5872 Acked Read 2 Done Sent Done false
19195 5873 Acked Write 1 Done Sent Done false
19196 5874 Acked Write 9 Done Sent Done false
19197 5875 Acked Read 7 Done Sent Done false
19198 5876 Acked Write 1 Done Sent Done false
19199 5877 Acked Read 7 Done Sent Done false
19200 5878 Acked Read 5 Done Sent Done false
19201 5879 Acked Read 7 Done Sent Done false
19202 5880 Acked Read 6 Done Sent Done false
19203 5881 Acked Flush 0 Done Sent Done false
19204 5882 Acked Read 1 Done Sent Done false
19205 5883 Acked Write 6 Done Sent Done false
19206 5884 Acked Write 4 Done Sent Done false
19207 5885 Acked Write 7 Done Sent Done false
19208 5886 Acked Read 8 Done Sent Done false
19209 5887 Acked Read 10 Done Sent Done false
19210 5888 Acked Write 9 Done Sent Done false
19211 5889 Acked Read 10 Done Sent Done false
19212 5890 Acked Write 6 Done Sent Done false
19213 5891 Acked Read 2 Done Sent Done false
19214 5892 Acked Read 1 Done Sent Done false
19215 5893 Acked Write 2 Done Sent Done false
19216 5894 Acked Write 6 Done Sent Done false
19217 5895 Acked Read 2 Done Sent Done false
19218 5896 Acked Read 1 Done Sent Done false
19219 5897 Acked Read 6 Done Sent Done false
19220 5898 Acked Write 5 Done Sent Done false
19221 5899 Acked Read 1 Done Sent Done false
19222 5900 Acked Read 6 Done Sent Done false
19223 5901 Acked Write 4 Done Sent Done false
19224 5902 Acked Read 2 Done Sent Done false
19225 5903 Acked Write 7 Done Sent Done false
19226 5904 Acked Flush 0 Done Sent Done false
19227 5905 Acked Write 4 Done Sent Done false
19228 5906 Acked Read 2 Done Sent Done false
19229 5907 Acked Write 2 Done Sent Done false
19230 5908 Acked Write 8 Done Sent Done false
19231 5909 Acked Write 10 Done Sent Done false
19232 5910 Acked Write 9 Done Sent Done false
19233 5911 Acked Write 5 Done Sent Done false
19234 5912 Acked Read 3 Done Sent Done false
19235 5913 Acked Read 7 Done Sent Done false
19236 5914 Acked Read 4 Done Sent Done false
19237 5915 Acked Write 5 Done Sent Done false
19238 5916 Acked Write 2 Done Sent Done false
19239 5917 Acked Read 9 Done Sent Done false
19240 5918 Acked Write 2 Done Sent Done false
19241 5919 Acked Flush 0 Done Sent Done false
19242 5920 Acked Read 9 Done Sent Done false
19243 5921 Acked Write 9 Done Sent Done false
19244 5922 Acked Write 9 Done Sent Done false
19245 5923 Acked Read 9 Done Sent Done false
19246 5924 Acked Read 1 Done Sent Done false
19247 5925 Acked Write 6 Done Sent Done false
19248 5926 Acked Write 6 Done Sent Done false
19249 5927 Acked Flush 0 Done Sent Done false
19250 5928 Acked Read 8 Done Sent Done false
19251 5929 Acked Read 6 Done Sent Done false
19252 5930 Acked Read 10 Done Sent Done false
19253 5931 Acked Read 6 Done Sent Done false
19254 5932 Acked Read 1 Done Sent Done false
19255 5933 Acked Read 3 Done Sent Done false
19256 5934 Acked Read 5 Done Sent Done false
19257 5935 Acked Read 4 Done Sent Done false
19258 5936 Acked Read 7 Done Sent Done false
19259 5937 Acked Read 10 Done Sent Done false
19260 5938 Acked Read 10 Done Sent Done false
19261 5939 Acked Flush 0 Done Sent Done false
19262 5940 Acked Write 2 Done Sent Done false
19263 5941 Acked Write 6 Done Sent Done false
19264 5942 Acked Read 9 Done Sent Done false
19265 5943 Acked Write 5 Done Sent Done false
19266 5944 Acked Read 2 Done Sent Done false
19267 5945 Acked Write 3 Done Sent Done false
19268 5946 Acked Read 5 Done Sent Done false
19269 5947 Acked Read 9 Done Sent Done false
19270 5948 Acked Read 8 Done Sent Done false
19271 5949 Acked Read 2 Done Sent Done false
19272 5950 Acked Write 8 Done Sent Done false
19273 5951 Acked Read 5 Done Sent Done false
19274 5952 Acked Read 1 Done Sent Done false
19275 5953 Acked Write 6 Done Sent Done false
19276 5954 Acked Write 3 Done Sent Done false
19277 5955 Acked Flush 0 Done Sent Done false
19278 5956 Acked Write 4 Done Sent Done false
19279 5957 Acked Write 4 Done Sent Done false
19280 5958 Acked Flush 0 Done Sent Done false
19281 5959 Acked Write 5 Done Sent Done false
19282 5960 Acked Write 2 Done Sent Done false
19283 5961 Acked Write 8 Done Sent Done false
19284 5962 Acked Read 1 Done Sent Done false
19285 5963 Acked Flush 0 Done Sent Done false
19286 5964 Acked Flush 0 Done Sent Done false
19287 5965 Acked Write 4 Done Sent Done false
19288 5966 Acked Read 4 Done Sent Done false
19289 5967 Acked Write 3 Done Sent Done false
19290 5968 Acked Read 3 Done Sent Done false
19291 5969 Acked Write 7 Done Sent Done false
19292 5970 Acked Read 1 Done Sent Done false
19293 5971 Acked Read 3 Done Sent Done false
19294 5972 Acked Read 5 Done Sent Done false
19295 5973 Acked Read 2 Done Sent Done false
19296 5974 Acked Flush 0 Done Sent Done false
19297 5975 Acked Flush 0 Done Sent Done false
19298 5976 Acked Write 7 Done Sent Done false
19299 5977 Acked Read 1 Done Sent Done false
19300 5978 Acked Write 3 Done Sent Done false
19301 5979 Acked Write 7 Done Sent Done false
19302 5980 Acked Read 1 Done Sent Done false
19303 5981 Acked Write 10 Done Sent Done false
19304 5982 Acked Write 5 Done Sent Done false
19305 5983 Acked Write 5 Done Sent Done false
19306 5984 Acked Read 5 Done Sent Done false
19307 5985 Acked Read 1 Done Sent Done false
19308 5986 Acked Read 10 Done Sent Done false
19309 5987 Acked Write 4 Done Sent Done false
19310 5988 Acked Write 9 Done Sent Done false
19311 5989 Acked Read 1 Done Sent Done false
19312 5990 Acked Read 9 Done Sent Done false
19313 5991 Acked Read 4 Done Sent Done false
19314 5992 Acked Write 2 Done Sent Done false
19315 5993 Acked Read 2 Done Sent Done false
19316 5994 Acked Read 2 Done Sent Done false
19317 5995 Acked Read 10 Done Sent Done false
19318 5996 Acked Read 2 Done Sent Done false
19319 5997 Acked Read 3 Done Sent Done false
19320 5998 Acked Read 4 Done Sent Done false
19321 5999 Acked Write 2 Done Sent Done false
19322 6000 Acked Read 8 Done Sent Done false
19323 6001 Acked Read 5 Done Sent Done false
19324 6002 Acked Read 5 Done Sent Done false
19325 6003 Acked Write 1 Done Sent Done false
19326 6004 Acked Write 5 Done Sent Done false
19327 6005 Acked Read 5 Done Sent Done false
19328 6006 Acked Read 6 Done Sent Done false
19329 6007 Acked Read 10 Done Sent Done false
19330 6008 Acked Write 3 Done Sent Done false
19331 6009 Acked Write 2 Done Sent Done false
19332 6010 Acked Read 3 Done Sent Done false
19333 6011 Acked Write 8 Done Sent Done false
19334 6012 Acked Write 1 Done Sent Done false
19335 6013 Acked Read 10 Done Sent Done false
19336 6014 Acked Write 7 Done Sent Done false
19337 6015 Acked Read 8 Done Sent Done false
19338 6016 Acked Write 7 Done Sent Done false
19339 6017 Acked Read 5 Done Sent Done false
19340 6018 Acked Flush 0 Done Sent Done false
19341 6019 Acked Write 1 Done Sent Done false
19342 6020 Acked Read 5 Done Sent Done false
19343 6021 Acked Write 5 Done Sent Done false
19344 6022 Acked Read 10 Done Sent Done false
19345 6023 Acked Write 3 Done Sent Done false
19346 6024 Acked Read 9 Done Sent Done false
19347 6025 Acked Read 1 Done Sent Done false
19348 6026 Acked Flush 0 Done Sent Done false
19349 6027 Acked Write 3 Done Sent Done false
19350 6028 Acked Flush 0 Done Sent Done false
19351 6029 Acked Write 3 Done Sent Done false
19352 6030 Acked Write 7 Done Sent Done false
19353 6031 Acked Write 7 Done Sent Done false
19354 6032 Acked Write 4 Done Sent Done false
19355 6033 Acked Write 6 Done Sent Done false
19356 6034 Acked Write 2 Done Sent Done false
19357 6035 Acked Read 4 Done Sent Done false
19358 6036 Acked Read 5 Done Sent Done false
19359 6037 Acked Flush 0 Done Sent Done false
19360 6038 Acked Write 8 Done Sent Done false
19361 6039 Acked Write 7 Done Sent Done false
19362 6040 Acked Read 8 Done Sent Done false
19363 6041 Acked Write 6 Done Sent Done false
19364 6042 Acked Read 10 Done Sent Done false
19365 6043 Acked Read 3 Done Sent Done false
19366 6044 Acked Write 7 Done Sent Done false
19367 6045 Acked Write 3 Done Sent Done false
19368 6046 Acked Write 5 Done Sent Done false
19369 6047 Acked Write 10 Done Sent Done false
19370 6048 Acked Write 9 Done Sent Done false
19371 6049 Acked Read 10 Done Sent Done false
19372 6050 Acked Write 5 Done Sent Done false
19373 6051 Acked Write 4 Done Sent Done false
19374 6052 Acked Write 2 Done Sent Done false
19375 6053 Acked Flush 0 Done Sent Done false
19376 6054 Acked Read 3 Done Sent Done false
19377 6055 Acked Write 1 Done Sent Done false
19378 6056 Acked Read 6 Done Sent Done false
19379 6057 Acked Write 9 Done Sent Done false
19380 6058 Acked Write 5 Done Sent Done false
19381 6059 Acked Read 5 Done Sent Done false
19382 6060 Acked Write 3 Done Sent Done false
19383 6061 Acked Read 10 Done Sent Done false
19384 6062 Acked Read 8 Done Sent Done false
19385 6063 Acked Read 10 Done Sent Done false
19386 6064 Acked Write 4 Done Sent Done false
19387 6065 Acked Read 6 Done Sent Done false
19388 6066 Acked Write 2 Done Sent Done false
19389 6067 Acked Read 7 Done Sent Done false
19390 6068 Acked Write 2 Done Sent Done false
19391 6069 Acked Write 4 Done Sent Done false
19392 6070 Acked Read 10 Done Sent Done false
19393 6071 Acked Read 4 Done Sent Done false
19394 6072 Acked Read 1 Done Sent Done false
19395 6073 Acked Write 4 Done Sent Done false
19396 6074 Acked Write 4 Done Sent Done false
19397 6075 Acked Read 2 Done Sent Done false
19398 6076 Acked Read 10 Done Sent Done false
19399 6077 Acked Read 1 Done Sent Done false
19400 6078 Acked Read 10 Done Sent Done false
19401 6079 Acked Write 4 Done Sent Done false
19402 6080 Acked Write 3 Done Sent Done false
19403 6081 Acked Read 6 Done Sent Done false
19404 6082 Acked Write 4 Done Sent Done false
19405 6083 Acked Write 8 Done Sent Done false
19406 6084 Acked Write 5 Done Sent Done false
19407 6085 Acked Write 3 Done Sent Done false
19408 6086 Acked Read 4 Done Sent Done false
19409 6087 Acked Write 2 Done Sent Done false
19410 6088 Acked Flush 0 Done Sent Done false
19411 6089 Acked Write 4 Done Sent Done false
19412 6090 Acked Write 8 Done Sent Done false
19413 6091 Acked Write 8 Done Sent Done false
19414 6092 Acked Flush 0 Done Sent Done false
19415 6093 Acked Read 8 Done Sent Done false
19416 6094 Acked Read 7 Done Sent Done false
19417 6095 Acked Read 10 Done Sent Done false
19418 6096 Acked Read 9 Done Sent Done false
19419 6097 Acked Write 3 Done Sent Done false
19420 6098 Acked Read 1 Done Sent Done false
19421 6099 Acked Write 2 Done Sent Done false
19422 6100 Acked Read 9 Done Sent Done false
19423 6101 Acked Read 8 Done Sent Done false
19424 6102 Acked Flush 0 Done Sent Done false
19425 6103 Acked Flush 0 Done Sent Done false
19426 6104 Acked Write 9 Done Sent Done false
19427 6105 Acked Write 5 Done Sent Done false
19428 6106 Acked Read 3 Done Sent Done false
19429 6107 Acked Flush 0 Done Sent Done false
19430 6108 Acked Read 8 Done Sent Done false
19431 6109 Acked Write 10 Done Sent Done false
19432 6110 Acked Read 10 Done Sent Done false
19433 6111 Acked Read 10 Done Sent Done false
19434 6112 Acked Write 3 Done Sent Done false
19435 6113 Acked Write 7 Done Sent Done false
19436 6114 Acked Read 10 Done Sent Done false
19437 6115 Acked Read 6 Done Sent Done false
19438 6116 Acked Write 7 Done Sent Done false
19439 6117 Acked Read 4 Done Sent Done false
19440 6118 Acked Read 9 Done Sent Done false
19441 6119 Acked Read 4 Done Sent Done false
19442 6120 Acked Read 2 Done Sent Done false
19443 6121 Acked Read 3 Done Sent Done false
19444 6122 Acked Write 9 Done Sent Done false
19445 6123 Acked Read 4 Done Sent Done false
19446 6124 Acked Read 2 Done Sent Done false
19447 6125 Acked Write 8 Done Sent Done false
19448 6126 Acked Write 8 Done Sent Done false
19449 6127 Acked Read 7 Done Sent Done false
19450 6128 Acked Write 4 Done Sent Done false
19451 6129 Acked Read 5 Done Sent Done false
19452 6130 Acked Read 7 Done Sent Done false
19453 6131 Acked Read 2 Done Sent Done false
19454 STATES DS:0 DS:1 DS:2 TOTAL
19455 Sent 0 301 0 301
19456 Done 301 0 301 602
19457 Skipped 0 0 0 0
19458 Error 0 0 0 0
19459 Last Flush: Some(JobId(6107)) Some(JobId(5830)) Some(JobId(6107))
19460 Downstairs last five completed: 5830 5829 5828 5827 5826
19461 Upstairs last five completed: 6131 6130 6129 6128 6127
19462 CLIENT: Up:0 ds:301 act:2
194632025-02-25T17:22:30.815ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194642025-02-25T17:22:31.686ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194652025-02-25T17:22:32.065ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194662025-02-25T17:22:32.065ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39783) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194672025-02-25T17:22:32.067ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194682025-02-25T17:22:32.067ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194692025-02-25T17:22:32.067ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194702025-02-25T17:22:32.067ZINFOcrucible: send last flush ID to this DS: Some(JobId(5830)) = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194712025-02-25T17:22:32.067ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194722025-02-25T17:22:32.067ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5830)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194732025-02-25T17:22:32.068ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5830)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
19474 ----------------------------------------------------------------
19475 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19476 Downstairs last five completed: 6132 6131 6130 6129 6128
19477 Upstairs last five completed: 6132 6131 6130 6129 6128
19478 CLIENT: Up:0 ds:0 act:3
19479 Replay: All jobs finished
19480 Main task has work to do, go find it
19481 got dsc Stop(2)
19482 stop 2
19483 Feb 25 17:22:34.317 INFO request completed, latency_us: 376, response_code: 204, uri: /stop/cid/2[8830] Got stop action so:false kr:false
19484 , method: POST, req_id: d2360917-9e9e-4f50-bac4-7287563b0b3c, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
194852025-02-25T17:22:34.318ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194862025-02-25T17:22:34.318ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194872025-02-25T17:22:34.318ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
194882025-02-25T17:22:34.318ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
19489 Feb 25 17:22:34.318 INFO request completed, latency_us: 228, response_code: 200, uri: /state/cid/2, method: GET, req_id: 249d83c8-d2fb-431b-ae91-382b3e352b51, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
19490 [8830] Exited with: ExitStatus(unix_wait_status(9))
19491 [8830][2] reports Exit
19492 Feb 25 17:22:38.319 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/2, method: GET, req_id: 426350b1-3eae-460d-8bb0-8b5cf9924278, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
19493 001/300 Write block 2009 len 40960 data: 7 6 6 6 6 6 5 5 5 6
19494 002/300 Read block 814 len 16384
19495 003/300 Flush
19496 004/300 Write block 4128 len 32768 data: 5 5 5 5 4 6 7 6
19497 005/300 Read block 1278 len 20480
19498 006/300 Read block 4603 len 12288
19499 007/300 Read block 3932 len 12288
19500 008/300 Write block 4967 len 20480 data: 5 6 3 4 3
19501 009/300 Write block 739 len 36864 data: 4 3 5 5 6 4 4 4 5
19502 010/300 Flush
19503 011/300 Write block 2952 len 32768 data: 5 5 4 3 3 4 3 3
19504 012/300 Flush
19505 013/300 Read block 28 len 32768
19506 014/300 Write block 1210 len 36864 data: 3 2 3 3 3 5 4 4 3
19507 015/300 Read block 1056 len 24576
19508 016/300 Flush
19509 017/300 Flush
19510 018/300 Write block 1314 len 24576 data: 5 4 6 4 4 4
19511 019/300 Read block 1398 len 28672
19512 020/300 Flush
19513 021/300 Read block 4371 len 32768
19514 022/300 Read block 278 len 4096
19515 023/300 Read block 1191 len 40960
19516 024/300 Read block 3870 len 20480
19517 025/300 Read block 160 len 32768
19518 026/300 Read block 1810 len 12288
19519 027/300 Read block 4588 len 20480
19520 028/300 Read block 1071 len 4096
19521 029/300 Flush
19522 030/300 Read block 4395 len 16384
19523 031/300 Read block 546 len 36864
19524 032/300 Read block 1802 len 4096
19525 033/300 Read block 4744 len 28672
19526 034/300 Write block 1571 len 4096 data: 2
19527 035/300 Write block 1416 len 20480 data: 3 2 2 3 3
19528 036/300 Read block 4795 len 20480
19529 037/300 Write block 4914 len 24576 data: 2 2 3 3 3 2
19530 038/300 Write block 3819 len 28672 data: 6 5 7 6 6 3 4
19531 039/300 Read block 1485 len 40960
19532 040/300 Read block 1751 len 40960
19533 041/300 Read block 4558 len 32768
19534 042/300 Read block 1967 len 40960
19535 043/300 Write block 993 len 20480 data: 5 5 5 5 3
19536 044/300 Write block 376 len 12288 data: 2 4 5
19537 045/300 Write block 1448 len 40960 data: 7 5 6 5 5 3 3 3 4 4
19538 046/300 Read block 2323 len 8192
19539 047/300 Write block 3990 len 16384 data: 3 5 4 5
19540 048/300 Write block 1170 len 40960 data: 6 5 5 4 5 5 5 5 7 6
19541 049/300 Read block 511 len 4096
19542 050/300 Read block 1099 len 24576
19543 051/300 Write block 1352 len 8192 data: 3 3
19544 052/300 Read block 3904 len 20480
19545 053/300 Read block 2694 len 24576
19546 054/300 Read block 3659 len 20480
19547 055/300 Write block 554 len 12288 data: 5 5 5
19548 056/300 Read block 4001 len 32768
19549 057/300 Read block 2440 len 8192
19550 058/300 Read block 2340 len 32768
19551 059/300 Write block 1747 len 16384 data: 4 5 5 3
19552 060/300 Flush
19553 061/300 Read block 648 len 4096
19554 062/300 Write block 1760 len 24576 data: 2 2 2 3 3 3
19555 063/300 Read block 4605 len 24576
19556 064/300 Write block 1853 len 12288 data: 4 3 3
19557 065/300 Write block 546 len 20480 data: 3 3 3 3 3
19558 066/300 Read block 2811 len 20480
19559 067/300 Read block 193 len 20480
19560 068/300 Write block 4573 len 28672 data: 4 3 4 4 5 6 5
19561 069/300 Read block 4926 len 32768
19562 070/300 Flush
19563 071/300 Write block 13 len 36864 data: 7 7 6 7 5 6 5 4 5
19564 072/300 Read block 1508 len 36864
19565 073/300 Write block 3897 len 4096 data: 4
19566 074/300 Write block 1573 len 4096 data: 3
19567 075/300 Write block 3053 len 36864 data: 5 6 5 5 6 6 6 6 5
19568 076/300 Flush
19569 077/300 Write block 1669 len 40960 data: 4 4 4 4 5 5 5 4 3 3
19570 078/300 Write block 2899 len 32768 data: 4 5 7 8 8 7 7 6
19571 079/300 Flush
19572 080/300 Read block 2908 len 36864
19573 081/300 Write block 4889 len 40960 data: 3 3 3 2 2 2 2 3 3 3
19574 082/300 Read block 3805 len 8192
19575 083/300 Read block 1034 len 4096
19576 084/300 Read block 1131 len 32768
19577 085/300 Write block 2165 len 12288 data: 2 2 3
19578 086/300 Read block 1002 len 16384
19579 087/300 Write block 3976 len 8192 data: 2 2
19580 088/300 Write block 4448 len 40960 data: 3 2 2 3 4 4 4 3 4 4
19581 089/300 Read block 2212 len 40960
19582 090/300 Write block 2685 len 4096 data: 3
19583 091/300 Read block 3083 len 16384
19584 092/300 Read block 4133 len 24576
19585 093/300 Write block 3958 len 16384 data: 3 3 3 3
19586 094/300 Read block 3580 len 8192
19587 095/300 Read block 2996 len 4096
19588 096/300 Read block 4116 len 32768
19589 097/300 Flush
19590 098/300 Write block 3656 len 24576 data: 8 6 7 6 5 5
19591 099/300 Read block 853 len 28672
19592 100/300 Read block 1911 len 8192
19593 101/300 Read block 1774 len 20480
19594 102/300 Read block 1473 len 8192
19595 103/300 Write block 1699 len 36864 data: 3 5 4 5 6 8 8 9 6
19596 104/300 Write block 1538 len 24576 data: 4 4 4 4 3 2
19597 105/300 Read block 2571 len 4096
19598 106/300 Read block 902 len 32768
19599 107/300 Read block 771 len 32768
19600 108/300 Read block 4559 len 4096
19601 109/300 Read block 1124 len 32768
19602 110/300 Read block 4049 len 36864
19603 111/300 Write block 2407 len 20480 data: 6 5 3 2 3
19604 112/300 Write block 3428 len 8192 data: 5 5
19605 113/300 Write block 4065 len 12288 data: 5 4 4
19606 114/300 Write block 4261 len 36864 data: 4 4 3 5 5 5 5 5 5
19607 115/300 Write block 1034 len 24576 data: 3 2 2 2 2 4
19608 116/300 Read block 1033 len 40960
19609 117/300 Write block 467 len 12288 data: 3 2 2
19610 118/300 Read block 242 len 4096
19611 119/300 Read block 2937 len 20480
19612 120/300 Read block 477 len 36864
19613 121/300 Write block 372 len 36864 data: 4 3 3 3 3 5 6 5 5
19614 122/300 Write block 568 len 12288 data: 5 5 4
19615 123/300 Read block 111 len 28672
19616 124/300 Write block 1745 len 40960 data: 4 4 5 6 6 4 3 4 4 5
19617 125/300 Read block 1377 len 20480
19618 126/300 Write block 2651 len 4096 data: 3
19619 127/300 Read block 2397 len 28672
19620 128/300 Write block 584 len 12288 data: 5 5 5
19621 129/300 Write block 2900 len 8192 data: 6 8
19622 130/300 Write block 3399 len 16384 data: 5 5 6 7
19623 131/300 Read block 3529 len 28672
19624 132/300 Read block 4275 len 4096
19625 133/300 Flush
19626 134/300 Read block 4154 len 36864
19627 135/300 Read block 2398 len 16384
19628 136/300 Write block 174 len 12288 data: 3 3 5
19629 137/300 Write block 3964 len 32768 data: 3 3 4 4 4 4 2 2
19630 138/300 Write block 4318 len 28672 data: 3 2 2 2 3 2 2
19631 139/300 Write block 2515 len 8192 data: 4 4
19632 140/300 Write block 3227 len 28672 data: 5 3 4 4 4 4 3
19633 141/300 Write block 2384 len 8192 data: 3 3
19634 142/300 Flush
19635 143/300 Read block 2292 len 12288
19636 144/300 Flush
19637 145/300 Write block 3868 len 36864 data: 3 3 4 4 5 5 5 4 5
19638 146/300 Read block 2444 len 8192
19639 147/300 Read block 2666 len 4096
19640 148/300 Read block 3992 len 24576
19641 149/300 Read block 1502 len 32768
19642 150/300 Read block 3554 len 8192
19643 151/300 Read block 1408 len 20480
19644 152/300 Flush
19645 153/300 Write block 1047 len 40960 data: 7 5 5 6 5 4 6 5 5 5
19646 154/300 Write block 573 len 24576 data: 4 4 4 3 3 4
19647 155/300 Read block 4432 len 16384
19648 156/300 Write block 3392 len 24576 data: 6 6 6 6 6 5
19649 157/300 Read block 743 len 32768
19650 158/300 Read block 4310 len 40960
19651 159/300 Write block 2494 len 28672 data: 3 3 3 3 3 4 4
19652 160/300 Flush
19653 161/300 Write block 4924 len 28672 data: 4 4 4 3 3 3 3
19654 162/300 Read block 3520 len 36864
19655 163/300 Read block 741 len 24576
19656 164/300 Write block 1557 len 20480 data: 4 4 3 2 2
19657 165/300 Read block 2603 len 36864
19658 166/300 Write block 4958 len 24576 data: 4 4 5 7 5 6
19659 167/300 Write block 300 len 16384 data: 4 4 4 4
19660 168/300 Read block 1288 len 40960
19661 169/300 Write block 413 len 36864 data: 5 6 7 6 5 5 6 6 5
19662 170/300 Write block 395 len 8192 data: 2 2
19663 171/300 Flush
19664 172/300 Read block 79 len 28672
19665 173/300 Flush
19666 174/300 Write block 3814 len 24576 data: 6 6 7 6 6 7
19667 175/300 Read block 1577 len 40960
19668 176/300 Write block 1963 len 8192 data: 3 2
19669 177/300 Write block 4744 len 40960 data: 3 3 4 4 5 5 6 6 6 6
19670 178/300 Write block 2585 len 24576 data: 4 3 3 3 5 6
19671 179/300 Write block 1300 len 16384 data: 3 3 2 2
19672 180/300 Flush
19673 181/300 Read block 3790 len 20480
19674 182/300 Write block 4614 len 4096 data: 5
19675 183/300 Write block 3822 len 16384 data: 7 7 4 5
19676 184/300 Read block 639 len 24576
19677 185/300 Read block 4342 len 28672
19678 186/300 Write block 923 len 32768 data: 5 5 5 4 5 5 5 6
19679 187/300 Write block 4136 len 16384 data: 5 4 4 5
19680 188/300 Write block 1601 len 8192 data: 3 5
19681 189/300 Read block 2435 len 12288
19682 190/300 Read block 2045 len 28672
19683 191/300 Read block 2343 len 20480
19684 192/300 Read block 1999 len 36864
19685 193/300 Write block 2190 len 36864 data: 5 6 5 4 3 4 4 5 5
19686 194/300 Flush
19687 195/300 Read block 2434 len 40960
19688 196/300 Write block 1934 len 16384 data: 5 6 6 5
19689 197/300 Write block 648 len 4096 data: 2
19690 198/300 Read block 2694 len 32768
19691 199/300 Write block 2528 len 20480 data: 4 3 3 3 2
19692 200/300 Read block 4360 len 28672
19693 201/300 Write block 390 len 4096 data: 3
19694 202/300 Write block 3392 len 40960 data: 7 7 7 7 7 6 4 6 6 7
19695 203/300 Read block 2279 len 8192
19696 204/300 Flush
19697 205/300 Read block 3487 len 4096
19698 206/300 Write block 3616 len 32768 data: 2 4 4 4 5 4 4 4
19699 207/300 Read block 2739 len 16384
19700 208/300 Write block 1376 len 40960 data: 4 4 3 3 4 4 6 6 5 5
19701 209/300 Read block 3297 len 12288
19702 210/300 Read block 3360 len 28672
19703 211/300 Write block 3616 len 24576 data: 3 5 5 5 6 5
19704 212/300 Read block 838 len 8192
19705 213/300 Read block 1989 len 20480
19706 214/300 Read block 1341 len 28672
19707 215/300 Read block 2313 len 8192
19708 216/300 Read block 1256 len 8192
19709 217/300 Write block 585 len 28672 data: 6 6 3 4 4 4 4
19710 218/300 Write block 93 len 24576 data: 4 5 5 4 5 5
19711 219/300 Flush
19712 220/300 Write block 3574 len 32768 data: 3 3 3 2 2 2 2 4
19713 221/300 Write block 2319 len 16384 data: 4 4 5 5
19714 222/300 Write block 1929 len 36864 data: 5 6 7 7 7 6 7 7 6
19715 223/300 Read block 1349 len 16384
19716 224/300 Read block 2307 len 32768
19717 225/300 Flush
19718 226/300 Write block 1562 len 16384 data: 3 3 3 4
19719 227/300 Read block 2895 len 4096
19720 228/300 Read block 2059 len 4096
19721 229/300 Flush
19722 230/300 Write block 1348 len 40960 data: 3 4 4 4 4 4 3 3 3 3
19723 231/300 Read block 2651 len 20480
19724 232/300 Write block 1412 len 16384 data: 3 4 4 4
19725 233/300 Read block 4094 len 4096
19726 234/300 Flush
19727 235/300 Write block 919 len 20480 data: 4 3 3 5 6
19728 236/300 Write block 1948 len 32768 data: 5 4 4 3 3 4 5 4
19729 237/300 Read block 3547 len 40960
19730 238/300 Write block 3147 len 36864 data: 4 4 4 3 3 3 3 3 3
19731 239/300 Write block 4140 len 12288 data: 5 4 4
19732 240/300 Read block 1282 len 4096
19733 241/300 Flush
19734 242/300 Write block 3753 len 16384 data: 3 3 3 3
19735 243/300 Read block 3890 len 28672
19736 244/300 Read block 1279 len 20480
19737 245/300 Write block 3410 len 4096 data: 4
19738 246/300 Read block 4492 len 36864
19739 247/300 Read block 2947 len 20480
19740 248/300 Write block 2240 len 24576 data: 4 4 4 4 3 3
19741 249/300 Read block 1123 len 4096
19742 250/300 Write block 2127 len 16384 data: 4 6 7 6
19743 251/300 Read block 564 len 32768
19744 252/300 Read block 4709 len 8192
19745 253/300 Write block 3724 len 16384 data: 8 8 8 6
19746 254/300 Read block 3752 len 8192
19747 255/300 Read block 4204 len 20480
19748 256/300 Read block 754 len 4096
19749 257/300 Write block 4761 len 28672 data: 4 5 5 5 5 8 7
19750 258/300 Read block 3767 len 20480
19751 259/300 Write block 3119 len 20480 data: 3 4 5 5 5
19752 260/300 Write block 3313 len 12288 data: 5 5 5
19753 261/300 Read block 1046 len 12288
19754 262/300 Flush
19755 263/300 Write block 2055 len 16384 data: 4 5 6 6
19756 264/300 Write block 3397 len 24576 data: 7 5 7 7 8 8
19757 265/300 Read block 159 len 32768
19758 266/300 Read block 1375 len 20480
19759 267/300 Write block 4689 len 32768 data: 2 3 4 3 3 3 4 5
19760 268/300 Read block 4448 len 40960
19761 269/300 Read block 4599 len 28672
19762 270/300 Flush
19763 271/300 Write block 809 len 24576 data: 4 4 4 3 3 3
19764 272/300 Flush
19765 273/300 Read block 3318 len 12288
19766 274/300 Write block 4934 len 8192 data: 5 5
19767 275/300 Read block 1776 len 12288
19768 276/300 Read block 448 len 16384
19769 277/300 Flush
19770 278/300 Read block 811 len 32768
19771 279/300 Read block 1493 len 20480
19772 280/300 Write block 3731 len 28672 data: 7 7 6 7 6 6 5
19773 281/300 Write block 4353 len 36864 data: 6 6 8 9 6 7 6 6 6
19774 282/300 Write block 1124 len 12288 data: 3 3 4
19775 283/300 Flush
19776 284/300 Write block 1962 len 16384 data: 4 4 3 2
19777 285/300 Read block 1776 len 16384
19778 286/300 Read block 624 len 28672
19779 287/300 Read block 3509 len 8192
19780 288/300 Flush
19781 289/300 Read block 1997 len 36864
19782 290/300 Write block 1657 len 12288 data: 2 2 2
19783 291/300 Read block 1992 len 40960
19784 292/300 Write block 1453 len 20480 data: 4 4 4 5 5
19785 293/300 Read block 326 len 12288
19786 294/300 Read block 2004 len 16384
19787 295/300 Read block 3530 len 24576
19788 296/300 Write block 3909 len 8192 data: 5 5
19789 297/300 Read block 2037 len 16384
19790 298/300 Read block 1081 len 28672
19791 299/300 Read block 3218 len 32768
19792 300/300 Write block 2635 len 32768 data: 4 4 6 6 5 4 2 2
19793 301/300 Read block 4465 len 24576
19794 Main task has work to do, go find it
19795 got dsc Start(2)
19796 start 2
19797 Feb 25 17:22:38.575 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/2, method: POST, req_id: 91b6f9b6-b5fd-49d3-a76e-a968c9e02376, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
19798 Replay: started 2, returned:Ok(())
19799 ----------------------------------------------------------------
19800 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19801 States: Active Active Offline
19802 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19803 6133 Acked Write 10 Done Done Sent false
19804 6134 Acked Read 4 Done Done Sent false
19805 6135 Acked Flush 0 Done Done Sent false
19806 6136 Acked Write 8 Done Done Sent false
19807 6137 Acked Read 5 Done Done Sent false
19808 6138 Acked Read 3 Done Done Sent false
19809 6139 Acked Read 3 Done Done Sent false
19810 6140 Acked Write 5 Done Done Sent false
19811 6141 Acked Write 9 Done Done Sent false
19812 6142 Acked Flush 0 Done Done Sent false
19813 6143 Acked Write 8 Done Done Sent false
19814 6144 Acked Flush 0 Done Done Sent false
19815 6145 Acked Read 8 Done Done Sent false
19816 6146 Acked Write 9 Done Done Sent false
19817 6147 Acked Read 6 Done Done Sent false
19818 6148 Acked Flush 0 Done Done Sent false
19819 6149 Acked Flush 0 Done Done Sent false
19820 6150 Acked Write 6 Done Done Sent false
19821 6151 Acked Read 7 Done Done Sent false
19822 6152 Acked Flush 0 Done Done Sent false
19823 6153 Acked Read 8 Done Done Sent false
19824 6154 Acked Read 1 Done Done Sent false
19825 6155 Acked Read 10 Done Done Sent false
19826 6156 Acked Read 5 Done Done Sent false
19827 6157 Acked Read 8 Done Done Sent false
19828 6158 Acked Read 3 Done Done Sent false
19829 6159 Acked Read 5 Done Done Sent false
19830 6160 Acked Read 1 Done Done Sent false
19831 6161 Acked Flush 0 Done Done Sent false
19832 6162 Acked Read 4 Done Done Sent false
19833 6163 Acked Read 9 Done Done Sent false
19834 6164 Acked Read 1 Done Done Sent false
19835 6165 Acked Read 7 Done Done Sent false
19836 6166 Acked Write 1 Done Done Sent false
19837 6167 Acked Write 5 Done Done Sent false
19838 6168 Acked Read 5 Done Done Sent false
19839 6169 Acked Write 6 Done Done Sent false
19840 6170 Acked Write 7 Done Done Sent false
19841 6171 Acked Read 10 Done Done Sent false
19842 6172 Acked Read 10 Done Done Sent false
19843 6173 Acked Read 8 Done Done Sent false
19844 6174 Acked Read 10 Done Done Sent false
19845 6175 Acked Write 5 Done Done Sent false
19846 6176 Acked Write 3 Done Done Sent false
19847 6177 Acked Write 10 Done Done Sent false
19848 6178 Acked Read 2 Done Done Sent false
19849 6179 Acked Write 4 Done Done Sent false
19850 6180 Acked Write 10 Done Done Sent false
19851 6181 Acked Read 1 Done Done Sent false
19852 6182 Acked Read 6 Done Done Sent false
19853 6183 Acked Write 2 Done Done Sent false
19854 6184 Acked Read 5 Done Done Sent false
19855 6185 Acked Read 6 Done Done Sent false
19856 6186 Acked Read 5 Done Done Sent false
19857 6187 Acked Write 3 Done Done Sent false
19858 6188 Acked Read 8 Done Done Sent false
19859 6189 Acked Read 2 Done Done Sent false
19860 6190 Acked Read 8 Done Done Sent false
19861 6191 Acked Write 4 Done Done Sent false
19862 6192 Acked Flush 0 Done Done Sent false
19863 6193 Acked Read 1 Done Done Sent false
19864 6194 Acked Write 6 Done Done Sent false
19865 6195 Acked Read 6 Done Done Sent false
19866 6196 Acked Write 3 Done Done Sent false
19867 6197 Acked Write 5 Done Done Sent false
19868 6198 Acked Read 5 Done Done Sent false
19869 6199 Acked Read 5 Done Done Sent false
19870 6200 Acked Write 7 Done Done Sent false
19871 6201 Acked Read 8 Done Done Sent false
19872 6202 Acked Flush 0 Done Done Sent false
19873 6203 Acked Write 9 Done Done Sent false
19874 6204 Acked Read 9 Done Done Sent false
19875 6205 Acked Write 1 Done Done Sent false
19876 6206 Acked Write 1 Done Done Sent false
19877 6207 Acked Write 9 Done Done Sent false
19878 6208 Acked Flush 0 Done Done Sent false
19879 6209 Acked Write 10 Done Done Sent false
19880 6210 Acked Write 8 Done Done Sent false
19881 6211 Acked Flush 0 Done Done Sent false
19882 6212 Acked Read 9 Done Done Sent false
19883 6213 Acked Write 10 Done Done Sent false
19884 6214 Acked Read 2 Done Done Sent false
19885 6215 Acked Read 1 Done Done Sent false
19886 6216 Acked Read 8 Done Done Sent false
19887 6217 Acked Write 3 Done Done Sent false
19888 6218 Acked Read 4 Done Done Sent false
19889 6219 Acked Write 2 Done Done Sent false
19890 6220 Acked Write 10 Done Done Sent false
19891 6221 Acked Read 10 Done Done Sent false
19892 6222 Acked Write 1 Done Done Sent false
19893 6223 Acked Read 4 Done Done Sent false
19894 6224 Acked Read 6 Done Done Sent false
19895 6225 Acked Write 4 Done Done Sent false
19896 6226 Acked Read 2 Done Done Sent false
19897 6227 Acked Read 1 Done Done Sent false
19898 6228 Acked Read 8 Done Done Sent false
19899 6229 Acked Flush 0 Done Done Sent false
19900 6230 Acked Write 6 Done Done Sent false
19901 6231 Acked Read 7 Done Done Sent false
19902 6232 Acked Read 2 Done Done Sent false
19903 6233 Acked Read 5 Done Done Sent false
19904 6234 Acked Read 2 Done Done Sent false
19905 6235 Acked Write 9 Done Done Sent false
19906 6236 Acked Write 6 Done Done Sent false
19907 6237 Acked Read 1 Done Done Sent false
19908 6238 Acked Read 8 Done Done Sent false
19909 6239 Acked Read 8 Done Done Sent false
19910 6240 Acked Read 1 Done Done Sent false
19911 6241 Acked Read 8 Done Done Sent false
19912 6242 Acked Read 9 Done Done Sent false
19913 6243 Acked Write 5 Done Done Sent false
19914 6244 Acked Write 2 Done Done Sent false
19915 6245 Acked Write 3 Done Done Sent false
19916 6246 Acked Write 9 Done Done Sent false
19917 6247 Acked Write 6 Done Done Sent false
19918 6248 Acked Read 10 Done Done Sent false
19919 6249 Acked Write 3 Done Done Sent false
19920 6250 Acked Read 1 Done Done Sent false
19921 6251 Acked Read 5 Done Done Sent false
19922 6252 Acked Read 9 Done Done Sent false
19923 6253 Acked Write 9 Done Done Sent false
19924 6254 Acked Write 3 Done Done Sent false
19925 6255 Acked Read 7 Done Done Sent false
19926 6256 Acked Write 10 Done Done Sent false
19927 6257 Acked Read 5 Done Done Sent false
19928 6258 Acked Write 1 Done Done Sent false
19929 6259 Acked Read 7 Done Done Sent false
19930 6260 Acked Write 3 Done Done Sent false
19931 6261 Acked Write 2 Done Done Sent false
19932 6262 Acked Write 4 Done Done Sent false
19933 6263 Acked Read 7 Done Done Sent false
19934 6264 Acked Read 1 Done Done Sent false
19935 6265 Acked Flush 0 Done Done Sent false
19936 6266 Acked Read 9 Done Done Sent false
19937 6267 Acked Read 4 Done Done Sent false
19938 6268 Acked Write 3 Done Done Sent false
19939 6269 Acked Write 8 Done Done Sent false
19940 6270 Acked Write 7 Done Done Sent false
19941 6271 Acked Write 2 Done Done Sent false
19942 6272 Acked Write 7 Done Done Sent false
19943 6273 Acked Write 2 Done Done Sent false
19944 6274 Acked Flush 0 Done Done Sent false
19945 6275 Acked Read 3 Done Done Sent false
19946 6276 Acked Flush 0 Done Done Sent false
19947 6277 Acked Write 9 Done Done Sent false
19948 6278 Acked Read 2 Done Done Sent false
19949 6279 Acked Read 1 Done Done Sent false
19950 6280 Acked Read 6 Done Done Sent false
19951 6281 Acked Read 8 Done Done Sent false
19952 6282 Acked Read 2 Done Done Sent false
19953 6283 Acked Read 5 Done Done Sent false
19954 6284 Acked Flush 0 Done Done Sent false
19955 6285 Acked Write 10 Done Done Sent false
19956 6286 Acked Write 6 Done Done Sent false
19957 6287 Acked Read 4 Done Done Sent false
19958 6288 Acked Write 6 Done Done Sent false
19959 6289 Acked Read 8 Done Done Sent false
19960 6290 Acked Read 10 Done Done Sent false
19961 6291 Acked Write 7 Done Done Sent false
19962 6292 Acked Flush 0 Done Done Sent false
19963 6293 Acked Write 7 Done Done Sent false
19964 6294 Acked Read 9 Done Done Sent false
19965 6295 Acked Read 6 Done Done Sent false
19966 6296 Acked Write 5 Done Done Sent false
19967 6297 Acked Read 9 Done Done Sent false
19968 6298 Acked Write 6 Done Done Sent false
19969 6299 Acked Write 4 Done Done Sent false
19970 6300 Acked Read 10 Done Done Sent false
19971 6301 Acked Write 9 Done Done Sent false
19972 6302 Acked Write 2 Done Done Sent false
19973 6303 Acked Flush 0 Done Done Sent false
19974 6304 Acked Read 7 Done Done Sent false
19975 6305 Acked Flush 0 Done Done Sent false
19976 6306 Acked Write 6 Done Done Sent false
19977 6307 Acked Read 10 Done Done Sent false
19978 6308 Acked Write 2 Done Done Sent false
19979 6309 Acked Write 10 Done Done Sent false
19980 6310 Acked Write 6 Done Done Sent false
19981 6311 Acked Write 4 Done Done Sent false
19982 6312 Acked Flush 0 Done Done Sent false
19983 6313 Acked Read 5 Done Done Sent false
19984 6314 Acked Write 1 Done Done Sent false
19985 6315 Acked Write 4 Done Done Sent false
19986 6316 Acked Read 6 Done Done Sent false
19987 6317 Acked Read 7 Done Done Sent false
19988 6318 Acked Write 8 Done Done Sent false
19989 6319 Acked Write 4 Done Done Sent false
19990 6320 Acked Write 2 Done Done Sent false
19991 6321 Acked Read 3 Done Done Sent false
19992 6322 Acked Read 7 Done Done Sent false
19993 6323 Acked Read 5 Done Done Sent false
19994 6324 Acked Read 9 Done Done Sent false
19995 6325 Acked Write 9 Done Done Sent false
19996 6326 Acked Flush 0 Done Done Sent false
19997 6327 Acked Read 10 Done Done Sent false
19998 6328 Acked Write 4 Done Done Sent false
19999 6329 Acked Write 1 Done Done Sent false
20000 6330 Acked Read 8 Done Done Sent false
20001 6331 Acked Write 5 Done Done Sent false
20002 6332 Acked Read 7 Done Done Sent false
20003 6333 Acked Write 1 Done Done Sent false
20004 6334 Acked Write 10 Done Done Sent false
20005 6335 Acked Read 2 Done Done Sent false
20006 6336 Acked Flush 0 Done Done Sent false
20007 6337 Acked Read 1 Done Done Sent false
20008 6338 Acked Write 8 Done Done Sent false
20009 6339 Acked Read 4 Done Done Sent false
20010 6340 Acked Write 10 Done Done Sent false
20011 6341 Acked Read 3 Done Done Sent false
20012 6342 Acked Read 7 Done Done Sent false
20013 6343 Acked Write 6 Done Done Sent false
20014 6344 Acked Read 2 Done Done Sent false
20015 6345 Acked Read 5 Done Done Sent false
20016 6346 Acked Read 7 Done Done Sent false
20017 6347 Acked Read 2 Done Done Sent false
20018 6348 Acked Read 2 Done Done Sent false
20019 6349 Acked Write 7 Done Done Sent false
20020 6350 Acked Write 6 Done Done Sent false
20021 6351 Acked Flush 0 Done Done Sent false
20022 6352 Acked Write 8 Done Done Sent false
20023 6353 Acked Write 4 Done Done Sent false
20024 6354 Acked Write 9 Done Done Sent false
20025 6355 Acked Read 4 Done Done Sent false
20026 6356 Acked Read 8 Done Done Sent false
20027 6357 Acked Flush 0 Done Done Sent false
20028 6358 Acked Write 4 Done Done Sent false
20029 6359 Acked Read 1 Done Done Sent false
20030 6360 Acked Read 1 Done Done Sent false
20031 6361 Acked Flush 0 Done Done Sent false
20032 6362 Acked Write 10 Done Done Sent false
20033 6363 Acked Read 5 Done Done Sent false
20034 6364 Acked Write 4 Done Done Sent false
20035 6365 Acked Read 1 Done Done Sent false
20036 6366 Acked Flush 0 Done Done Sent false
20037 6367 Acked Write 5 Done Done Sent false
20038 6368 Acked Write 8 Done Done Sent false
20039 6369 Acked Read 10 Done Done Sent false
20040 6370 Acked Write 9 Done Done Sent false
20041 6371 Acked Write 3 Done Done Sent false
20042 6372 Acked Read 1 Done Done Sent false
20043 6373 Acked Flush 0 Done Done Sent false
20044 6374 Acked Write 4 Done Done Sent false
20045 6375 Acked Read 7 Done Done Sent false
20046 6376 Acked Read 5 Done Done Sent false
20047 6377 Acked Write 1 Done Done Sent false
20048 6378 Acked Read 9 Done Done Sent false
20049 6379 Acked Read 5 Done Done Sent false
20050 6380 Acked Write 6 Done Done Sent false
20051 6381 Acked Read 1 Done Done Sent false
20052 6382 Acked Write 4 Done Done Sent false
20053 6383 Acked Read 8 Done Done Sent false
20054 6384 Acked Read 2 Done Done Sent false
20055 6385 Acked Write 4 Done Done Sent false
20056 6386 Acked Read 2 Done Done Sent false
20057 6387 Acked Read 5 Done Done Sent false
20058 6388 Acked Read 1 Done Done Sent false
20059 6389 Acked Write 7 Done Done Sent false
20060 6390 Acked Read 5 Done Done Sent false
20061 6391 Acked Write 5 Done Done Sent false
20062 6392 Acked Write 3 Done Done Sent false
20063 6393 Acked Read 3 Done Done Sent false
20064 6394 Acked Flush 0 Done Done Sent false
20065 6395 Acked Write 4 Done Done Sent false
20066 6396 Acked Write 6 Done Done Sent false
20067 6397 Acked Read 8 Done Done Sent false
20068 6398 Acked Read 5 Done Done Sent false
20069 6399 Acked Write 8 Done Done Sent false
20070 6400 Acked Read 10 Done Done Sent false
20071 6401 Acked Read 7 Done Done Sent false
20072 6402 Acked Flush 0 Done Done Sent false
20073 6403 Acked Write 6 Done Done Sent false
20074 6404 Acked Flush 0 Done Done Sent false
20075 6405 Acked Read 3 Done Done Sent false
20076 6406 Acked Write 2 Done Done Sent false
20077 6407 Acked Read 3 Done Done Sent false
20078 6408 Acked Read 4 Done Done Sent false
20079 6409 Acked Flush 0 Done Done Sent false
20080 6410 Acked Read 8 Done Done Sent false
20081 6411 Acked Read 5 Done Done Sent false
20082 6412 Acked Write 7 Done Done Sent false
20083 6413 Acked Write 9 Done Done Sent false
20084 6414 Acked Write 3 Done Done Sent false
20085 6415 Acked Flush 0 Done Done Sent false
20086 6416 Acked Write 4 Done Done Sent false
20087 6417 Acked Read 4 Done Done Sent false
20088 6418 Acked Read 7 Done Done Sent false
20089 6419 Acked Read 2 Done Done Sent false
20090 6420 Acked Flush 0 Done Done Sent false
20091 6421 Acked Read 9 Done Done Sent false
20092 6422 Acked Write 3 Done Done Sent false
20093 6423 Acked Read 10 Done Done Sent false
20094 6424 Acked Write 5 Done Done Sent false
20095 6425 Acked Read 3 Done Done Sent false
20096 6426 Acked Read 4 Done Done Sent false
20097 6427 Acked Read 6 Done Done Sent false
20098 6428 Acked Write 2 Done Done Sent false
20099 6429 Acked Read 4 Done Done Sent false
20100 6430 Acked Read 7 Done Done Sent false
20101 6431 Acked Read 8 Done Done Sent false
20102 6432 Acked Write 8 Done Done Sent false
20103 6433 Acked Read 6 Done Done Sent false
20104 STATES DS:0 DS:1 DS:2 TOTAL
20105 Sent 0 0 301 301
20106 Done 301 301 0 602
20107 Skipped 0 0 0 0
20108 Error 0 0 0 0
20109 Last Flush: Some(JobId(6420)) Some(JobId(6420)) Some(JobId(6132))
20110 Downstairs last five completed: 6132 6131 6130 6129 6128
20111 Upstairs last five completed: 6433 6432 6431 6430 6429
20112 CLIENT: Up:0 ds:301 act:2
201132025-02-25T17:22:39.077ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
201142025-02-25T17:22:39.698ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
20115 [8830] got start action
20116 [8830] I am going to restart
20117 Starting downstairs at port 8830
20118 Make output file at "/tmp/dsc/downstairs-8830.txt"
20119 [8830][2] reports Starting
20120 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1226)
20121 [8830][2] reports Running
201222025-02-25T17:22:40.699ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
201232025-02-25T17:22:41.700ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
20124 ----------------------------------------------------------------
20125 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20126 States: Active Active Offline
20127 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20128 6133 Acked Write 10 Done Done Sent false
20129 6134 Acked Read 4 Done Done Sent false
20130 6135 Acked Flush 0 Done Done Sent false
20131 6136 Acked Write 8 Done Done Sent false
20132 6137 Acked Read 5 Done Done Sent false
20133 6138 Acked Read 3 Done Done Sent false
20134 6139 Acked Read 3 Done Done Sent false
20135 6140 Acked Write 5 Done Done Sent false
20136 6141 Acked Write 9 Done Done Sent false
20137 6142 Acked Flush 0 Done Done Sent false
20138 6143 Acked Write 8 Done Done Sent false
20139 6144 Acked Flush 0 Done Done Sent false
20140 6145 Acked Read 8 Done Done Sent false
20141 6146 Acked Write 9 Done Done Sent false
20142 6147 Acked Read 6 Done Done Sent false
20143 6148 Acked Flush 0 Done Done Sent false
20144 6149 Acked Flush 0 Done Done Sent false
20145 6150 Acked Write 6 Done Done Sent false
20146 6151 Acked Read 7 Done Done Sent false
20147 6152 Acked Flush 0 Done Done Sent false
20148 6153 Acked Read 8 Done Done Sent false
20149 6154 Acked Read 1 Done Done Sent false
20150 6155 Acked Read 10 Done Done Sent false
20151 6156 Acked Read 5 Done Done Sent false
20152 6157 Acked Read 8 Done Done Sent false
20153 6158 Acked Read 3 Done Done Sent false
20154 6159 Acked Read 5 Done Done Sent false
20155 6160 Acked Read 1 Done Done Sent false
20156 6161 Acked Flush 0 Done Done Sent false
20157 6162 Acked Read 4 Done Done Sent false
20158 6163 Acked Read 9 Done Done Sent false
20159 6164 Acked Read 1 Done Done Sent false
20160 6165 Acked Read 7 Done Done Sent false
20161 6166 Acked Write 1 Done Done Sent false
20162 6167 Acked Write 5 Done Done Sent false
20163 6168 Acked Read 5 Done Done Sent false
20164 6169 Acked Write 6 Done Done Sent false
20165 6170 Acked Write 7 Done Done Sent false
20166 6171 Acked Read 10 Done Done Sent false
20167 6172 Acked Read 10 Done Done Sent false
20168 6173 Acked Read 8 Done Done Sent false
20169 6174 Acked Read 10 Done Done Sent false
20170 6175 Acked Write 5 Done Done Sent false
20171 6176 Acked Write 3 Done Done Sent false
20172 6177 Acked Write 10 Done Done Sent false
20173 6178 Acked Read 2 Done Done Sent false
20174 6179 Acked Write 4 Done Done Sent false
20175 6180 Acked Write 10 Done Done Sent false
20176 6181 Acked Read 1 Done Done Sent false
20177 6182 Acked Read 6 Done Done Sent false
20178 6183 Acked Write 2 Done Done Sent false
20179 6184 Acked Read 5 Done Done Sent false
20180 6185 Acked Read 6 Done Done Sent false
20181 6186 Acked Read 5 Done Done Sent false
20182 6187 Acked Write 3 Done Done Sent false
20183 6188 Acked Read 8 Done Done Sent false
20184 6189 Acked Read 2 Done Done Sent false
20185 6190 Acked Read 8 Done Done Sent false
20186 6191 Acked Write 4 Done Done Sent false
20187 6192 Acked Flush 0 Done Done Sent false
20188 6193 Acked Read 1 Done Done Sent false
20189 6194 Acked Write 6 Done Done Sent false
20190 6195 Acked Read 6 Done Done Sent false
20191 6196 Acked Write 3 Done Done Sent false
20192 6197 Acked Write 5 Done Done Sent false
20193 6198 Acked Read 5 Done Done Sent false
20194 6199 Acked Read 5 Done Done Sent false
20195 6200 Acked Write 7 Done Done Sent false
20196 6201 Acked Read 8 Done Done Sent false
20197 6202 Acked Flush 0 Done Done Sent false
20198 6203 Acked Write 9 Done Done Sent false
20199 6204 Acked Read 9 Done Done Sent false
20200 6205 Acked Write 1 Done Done Sent false
20201 6206 Acked Write 1 Done Done Sent false
20202 6207 Acked Write 9 Done Done Sent false
20203 6208 Acked Flush 0 Done Done Sent false
20204 6209 Acked Write 10 Done Done Sent false
20205 6210 Acked Write 8 Done Done Sent false
20206 6211 Acked Flush 0 Done Done Sent false
20207 6212 Acked Read 9 Done Done Sent false
20208 6213 Acked Write 10 Done Done Sent false
20209 6214 Acked Read 2 Done Done Sent false
20210 6215 Acked Read 1 Done Done Sent false
20211 6216 Acked Read 8 Done Done Sent false
20212 6217 Acked Write 3 Done Done Sent false
20213 6218 Acked Read 4 Done Done Sent false
20214 6219 Acked Write 2 Done Done Sent false
20215 6220 Acked Write 10 Done Done Sent false
20216 6221 Acked Read 10 Done Done Sent false
20217 6222 Acked Write 1 Done Done Sent false
20218 6223 Acked Read 4 Done Done Sent false
20219 6224 Acked Read 6 Done Done Sent false
20220 6225 Acked Write 4 Done Done Sent false
20221 6226 Acked Read 2 Done Done Sent false
20222 6227 Acked Read 1 Done Done Sent false
20223 6228 Acked Read 8 Done Done Sent false
20224 6229 Acked Flush 0 Done Done Sent false
20225 6230 Acked Write 6 Done Done Sent false
20226 6231 Acked Read 7 Done Done Sent false
20227 6232 Acked Read 2 Done Done Sent false
20228 6233 Acked Read 5 Done Done Sent false
20229 6234 Acked Read 2 Done Done Sent false
20230 6235 Acked Write 9 Done Done Sent false
20231 6236 Acked Write 6 Done Done Sent false
20232 6237 Acked Read 1 Done Done Sent false
20233 6238 Acked Read 8 Done Done Sent false
20234 6239 Acked Read 8 Done Done Sent false
20235 6240 Acked Read 1 Done Done Sent false
20236 6241 Acked Read 8 Done Done Sent false
20237 6242 Acked Read 9 Done Done Sent false
20238 6243 Acked Write 5 Done Done Sent false
20239 6244 Acked Write 2 Done Done Sent false
20240 6245 Acked Write 3 Done Done Sent false
20241 6246 Acked Write 9 Done Done Sent false
20242 6247 Acked Write 6 Done Done Sent false
20243 6248 Acked Read 10 Done Done Sent false
20244 6249 Acked Write 3 Done Done Sent false
20245 6250 Acked Read 1 Done Done Sent false
20246 6251 Acked Read 5 Done Done Sent false
20247 6252 Acked Read 9 Done Done Sent false
20248 6253 Acked Write 9 Done Done Sent false
20249 6254 Acked Write 3 Done Done Sent false
20250 6255 Acked Read 7 Done Done Sent false
20251 6256 Acked Write 10 Done Done Sent false
20252 6257 Acked Read 5 Done Done Sent false
20253 6258 Acked Write 1 Done Done Sent false
20254 6259 Acked Read 7 Done Done Sent false
20255 6260 Acked Write 3 Done Done Sent false
20256 6261 Acked Write 2 Done Done Sent false
20257 6262 Acked Write 4 Done Done Sent false
20258 6263 Acked Read 7 Done Done Sent false
20259 6264 Acked Read 1 Done Done Sent false
20260 6265 Acked Flush 0 Done Done Sent false
20261 6266 Acked Read 9 Done Done Sent false
20262 6267 Acked Read 4 Done Done Sent false
20263 6268 Acked Write 3 Done Done Sent false
20264 6269 Acked Write 8 Done Done Sent false
20265 6270 Acked Write 7 Done Done Sent false
20266 6271 Acked Write 2 Done Done Sent false
20267 6272 Acked Write 7 Done Done Sent false
20268 6273 Acked Write 2 Done Done Sent false
20269 6274 Acked Flush 0 Done Done Sent false
20270 6275 Acked Read 3 Done Done Sent false
20271 6276 Acked Flush 0 Done Done Sent false
20272 6277 Acked Write 9 Done Done Sent false
20273 6278 Acked Read 2 Done Done Sent false
20274 6279 Acked Read 1 Done Done Sent false
20275 6280 Acked Read 6 Done Done Sent false
20276 6281 Acked Read 8 Done Done Sent false
20277 6282 Acked Read 2 Done Done Sent false
20278 6283 Acked Read 5 Done Done Sent false
20279 6284 Acked Flush 0 Done Done Sent false
20280 6285 Acked Write 10 Done Done Sent false
20281 6286 Acked Write 6 Done Done Sent false
20282 6287 Acked Read 4 Done Done Sent false
20283 6288 Acked Write 6 Done Done Sent false
20284 6289 Acked Read 8 Done Done Sent false
20285 6290 Acked Read 10 Done Done Sent false
20286 6291 Acked Write 7 Done Done Sent false
20287 6292 Acked Flush 0 Done Done Sent false
20288 6293 Acked Write 7 Done Done Sent false
20289 6294 Acked Read 9 Done Done Sent false
20290 6295 Acked Read 6 Done Done Sent false
20291 6296 Acked Write 5 Done Done Sent false
20292 6297 Acked Read 9 Done Done Sent false
20293 6298 Acked Write 6 Done Done Sent false
20294 6299 Acked Write 4 Done Done Sent false
20295 6300 Acked Read 10 Done Done Sent false
20296 6301 Acked Write 9 Done Done Sent false
20297 6302 Acked Write 2 Done Done Sent false
20298 6303 Acked Flush 0 Done Done Sent false
20299 6304 Acked Read 7 Done Done Sent false
20300 6305 Acked Flush 0 Done Done Sent false
20301 6306 Acked Write 6 Done Done Sent false
20302 6307 Acked Read 10 Done Done Sent false
20303 6308 Acked Write 2 Done Done Sent false
20304 6309 Acked Write 10 Done Done Sent false
20305 6310 Acked Write 6 Done Done Sent false
20306 6311 Acked Write 4 Done Done Sent false
20307 6312 Acked Flush 0 Done Done Sent false
20308 6313 Acked Read 5 Done Done Sent false
20309 6314 Acked Write 1 Done Done Sent false
20310 6315 Acked Write 4 Done Done Sent false
20311 6316 Acked Read 6 Done Done Sent false
20312 6317 Acked Read 7 Done Done Sent false
20313 6318 Acked Write 8 Done Done Sent false
20314 6319 Acked Write 4 Done Done Sent false
20315 6320 Acked Write 2 Done Done Sent false
20316 6321 Acked Read 3 Done Done Sent false
20317 6322 Acked Read 7 Done Done Sent false
20318 6323 Acked Read 5 Done Done Sent false
20319 6324 Acked Read 9 Done Done Sent false
20320 6325 Acked Write 9 Done Done Sent false
20321 6326 Acked Flush 0 Done Done Sent false
20322 6327 Acked Read 10 Done Done Sent false
20323 6328 Acked Write 4 Done Done Sent false
20324 6329 Acked Write 1 Done Done Sent false
20325 6330 Acked Read 8 Done Done Sent false
20326 6331 Acked Write 5 Done Done Sent false
20327 6332 Acked Read 7 Done Done Sent false
20328 6333 Acked Write 1 Done Done Sent false
20329 6334 Acked Write 10 Done Done Sent false
20330 6335 Acked Read 2 Done Done Sent false
20331 6336 Acked Flush 0 Done Done Sent false
20332 6337 Acked Read 1 Done Done Sent false
20333 6338 Acked Write 8 Done Done Sent false
20334 6339 Acked Read 4 Done Done Sent false
20335 6340 Acked Write 10 Done Done Sent false
20336 6341 Acked Read 3 Done Done Sent false
20337 6342 Acked Read 7 Done Done Sent false
20338 6343 Acked Write 6 Done Done Sent false
20339 6344 Acked Read 2 Done Done Sent false
20340 6345 Acked Read 5 Done Done Sent false
20341 6346 Acked Read 7 Done Done Sent false
20342 6347 Acked Read 2 Done Done Sent false
20343 6348 Acked Read 2 Done Done Sent false
20344 6349 Acked Write 7 Done Done Sent false
20345 6350 Acked Write 6 Done Done Sent false
20346 6351 Acked Flush 0 Done Done Sent false
20347 6352 Acked Write 8 Done Done Sent false
20348 6353 Acked Write 4 Done Done Sent false
20349 6354 Acked Write 9 Done Done Sent false
20350 6355 Acked Read 4 Done Done Sent false
20351 6356 Acked Read 8 Done Done Sent false
20352 6357 Acked Flush 0 Done Done Sent false
20353 6358 Acked Write 4 Done Done Sent false
20354 6359 Acked Read 1 Done Done Sent false
20355 6360 Acked Read 1 Done Done Sent false
20356 6361 Acked Flush 0 Done Done Sent false
20357 6362 Acked Write 10 Done Done Sent false
20358 6363 Acked Read 5 Done Done Sent false
20359 6364 Acked Write 4 Done Done Sent false
20360 6365 Acked Read 1 Done Done Sent false
20361 6366 Acked Flush 0 Done Done Sent false
20362 6367 Acked Write 5 Done Done Sent false
20363 6368 Acked Write 8 Done Done Sent false
20364 6369 Acked Read 10 Done Done Sent false
20365 6370 Acked Write 9 Done Done Sent false
20366 6371 Acked Write 3 Done Done Sent false
20367 6372 Acked Read 1 Done Done Sent false
20368 6373 Acked Flush 0 Done Done Sent false
20369 6374 Acked Write 4 Done Done Sent false
20370 6375 Acked Read 7 Done Done Sent false
20371 6376 Acked Read 5 Done Done Sent false
20372 6377 Acked Write 1 Done Done Sent false
20373 6378 Acked Read 9 Done Done Sent false
20374 6379 Acked Read 5 Done Done Sent false
20375 6380 Acked Write 6 Done Done Sent false
20376 6381 Acked Read 1 Done Done Sent false
20377 6382 Acked Write 4 Done Done Sent false
20378 6383 Acked Read 8 Done Done Sent false
20379 6384 Acked Read 2 Done Done Sent false
20380 6385 Acked Write 4 Done Done Sent false
20381 6386 Acked Read 2 Done Done Sent false
20382 6387 Acked Read 5 Done Done Sent false
20383 6388 Acked Read 1 Done Done Sent false
20384 6389 Acked Write 7 Done Done Sent false
20385 6390 Acked Read 5 Done Done Sent false
20386 6391 Acked Write 5 Done Done Sent false
20387 6392 Acked Write 3 Done Done Sent false
20388 6393 Acked Read 3 Done Done Sent false
20389 6394 Acked Flush 0 Done Done Sent false
20390 6395 Acked Write 4 Done Done Sent false
20391 6396 Acked Write 6 Done Done Sent false
20392 6397 Acked Read 8 Done Done Sent false
20393 6398 Acked Read 5 Done Done Sent false
20394 6399 Acked Write 8 Done Done Sent false
20395 6400 Acked Read 10 Done Done Sent false
20396 6401 Acked Read 7 Done Done Sent false
20397 6402 Acked Flush 0 Done Done Sent false
20398 6403 Acked Write 6 Done Done Sent false
20399 6404 Acked Flush 0 Done Done Sent false
20400 6405 Acked Read 3 Done Done Sent false
20401 6406 Acked Write 2 Done Done Sent false
20402 6407 Acked Read 3 Done Done Sent false
20403 6408 Acked Read 4 Done Done Sent false
20404 6409 Acked Flush 0 Done Done Sent false
20405 6410 Acked Read 8 Done Done Sent false
20406 6411 Acked Read 5 Done Done Sent false
20407 6412 Acked Write 7 Done Done Sent false
20408 6413 Acked Write 9 Done Done Sent false
20409 6414 Acked Write 3 Done Done Sent false
20410 6415 Acked Flush 0 Done Done Sent false
20411 6416 Acked Write 4 Done Done Sent false
20412 6417 Acked Read 4 Done Done Sent false
20413 6418 Acked Read 7 Done Done Sent false
20414 6419 Acked Read 2 Done Done Sent false
20415 6420 Acked Flush 0 Done Done Sent false
20416 6421 Acked Read 9 Done Done Sent false
20417 6422 Acked Write 3 Done Done Sent false
20418 6423 Acked Read 10 Done Done Sent false
20419 6424 Acked Write 5 Done Done Sent false
20420 6425 Acked Read 3 Done Done Sent false
20421 6426 Acked Read 4 Done Done Sent false
20422 6427 Acked Read 6 Done Done Sent false
20423 6428 Acked Write 2 Done Done Sent false
20424 6429 Acked Read 4 Done Done Sent false
20425 6430 Acked Read 7 Done Done Sent false
20426 6431 Acked Read 8 Done Done Sent false
20427 6432 Acked Write 8 Done Done Sent false
20428 6433 Acked Read 6 Done Done Sent false
20429 STATES DS:0 DS:1 DS:2 TOTAL
20430 Sent 0 0 301 301
20431 Done 301 301 0 602
20432 Skipped 0 0 0 0
20433 Error 0 0 0 0
20434 Last Flush: Some(JobId(6420)) Some(JobId(6420)) Some(JobId(6132))
20435 Downstairs last five completed: 6132 6131 6130 6129 6128
20436 Upstairs last five completed: 6433 6432 6431 6430 6429
20437 CLIENT: Up:0 ds:301 act:2
204382025-02-25T17:22:43.079ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204392025-02-25T17:22:43.783ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204402025-02-25T17:22:44.319ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204412025-02-25T17:22:44.319ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54154) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204422025-02-25T17:22:44.320ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204432025-02-25T17:22:44.321ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204442025-02-25T17:22:44.321ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204452025-02-25T17:22:44.321ZINFOcrucible: send last flush ID to this DS: Some(JobId(6132)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204462025-02-25T17:22:44.321ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204472025-02-25T17:22:44.321ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6132)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204482025-02-25T17:22:44.321ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6132)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
20449 ----------------------------------------------------------------
20450 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20451 Downstairs last five completed: 6434 6433 6432 6431 6430
20452 Upstairs last five completed: 6434 6433 6432 6431 6430
20453 CLIENT: Up:0 ds:0 act:3
20454 Replay: All jobs finished
20455 Main task has work to do, go find it
20456 got dsc Stop(0)
20457 stop 0
20458 Feb 25 17:22:46.580 [8810] Got stop action so:false kr:false
20459 INFO request completed, latency_us: 377, response_code: 204, uri: /stop/cid/0, method: POST, req_id: a168d912-4792-4a5d-9426-37bd47b92c13, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
20460 Feb 25 17:22:46.581 INFO request completed, latency_us: 157, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1e6a4b50-42eb-4087-b4bc-59d33b5deaa0, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
204612025-02-25T17:22:46.581ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204622025-02-25T17:22:46.581ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204632025-02-25T17:22:46.581ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
204642025-02-25T17:22:46.581ZINFOcrucible: client task is exiting = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
20465 [8810] Exited with: ExitStatus(unix_wait_status(9))
20466 [8810][0] reports Exit
20467 Feb 25 17:22:50.584 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/0, method: GET, req_id: af204b03-a211-4b43-8578-b5b3e8ace5a4, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
20468 001/300 Read block 4245 len 40960
20469 002/300 Read block 4668 len 8192
20470 003/300 Flush
20471 004/300 Write block 3383 len 36864 data: 3 4 4 4 3 3 5 6 6
20472 005/300 Write block 4403 len 16384 data: 3 3 3 3
20473 006/300 Flush
20474 007/300 Read block 2083 len 28672
20475 008/300 Write block 2912 len 20480 data: 3 3 4 4 3
20476 009/300 Read block 1605 len 4096
20477 010/300 Read block 582 len 24576
20478 011/300 Write block 4352 len 4096 data: 6
20479 012/300 Read block 1238 len 32768
20480 013/300 Read block 1576 len 8192
20481 014/300 Read block 3716 len 12288
20482 015/300 Read block 1112 len 20480
20483 016/300 Read block 4607 len 36864
20484 017/300 Write block 1511 len 28672 data: 3 3 3 3 3 3 3
20485 018/300 Write block 823 len 40960 data: 3 3 3 2 2 2 3 4 4 7
20486 019/300 Read block 3256 len 40960
20487 020/300 Read block 1385 len 32768
20488 021/300 Write block 2797 len 12288 data: 7 8 7
20489 022/300 Read block 4810 len 24576
20490 023/300 Read block 4331 len 12288
20491 024/300 Flush
20492 025/300 Read block 2013 len 16384
20493 026/300 Flush
20494 027/300 Read block 1226 len 8192
20495 028/300 Read block 4390 len 8192
20496 029/300 Read block 2373 len 20480
20497 030/300 Write block 4367 len 40960 data: 3 3 3 3 3 3 2 2 2 2
20498 031/300 Read block 186 len 4096
20499 032/300 Read block 4187 len 4096
20500 033/300 Write block 451 len 12288 data: 4 4 5
20501 034/300 Flush
20502 035/300 Write block 3360 len 36864 data: 6 4 4 4 3 3 4 4 5
20503 036/300 Read block 3138 len 28672
20504 037/300 Read block 4280 len 32768
20505 038/300 Flush
20506 039/300 Read block 1516 len 16384
20507 040/300 Read block 2498 len 12288
20508 041/300 Write block 1877 len 12288 data: 3 3 3
20509 042/300 Write block 2279 len 28672 data: 6 6 6 4 4 4 3
20510 043/300 Read block 1980 len 32768
20511 044/300 Flush
20512 045/300 Write block 4404 len 12288 data: 4 4 4
20513 046/300 Read block 3322 len 4096
20514 047/300 Write block 301 len 24576 data: 5 5 5 5 3 3
20515 048/300 Read block 1003 len 40960
20516 049/300 Flush
20517 050/300 Write block 4368 len 8192 data: 4 4
20518 051/300 Read block 3842 len 8192
20519 052/300 Write block 1247 len 36864 data: 4 5 3 4 4 3 3 4 4
20520 053/300 Read block 4297 len 4096
20521 054/300 Write block 21 len 32768 data: 6 4 5 5 5 5 6 6
20522 055/300 Read block 4871 len 32768
20523 056/300 Write block 3305 len 4096 data: 6
20524 057/300 Write block 2121 len 28672 data: 5 5 5 4 4 4 5
20525 058/300 Flush
20526 059/300 Write block 3425 len 12288 data: 4 5 6
20527 060/300 Read block 2818 len 32768
20528 061/300 Read block 3141 len 36864
20529 062/300 Read block 4922 len 16384
20530 063/300 Write block 1661 len 12288 data: 2 2 3
20531 064/300 Write block 2946 len 28672 data: 6 5 5 4 4 5 6
20532 065/300 Flush
20533 066/300 Read block 2596 len 12288
20534 067/300 Read block 4748 len 4096
20535 068/300 Read block 3053 len 40960
20536 069/300 Write block 4064 len 12288 data: 6 6 5
20537 070/300 Read block 895 len 16384
20538 071/300 Read block 3 len 12288
20539 072/300 Write block 1266 len 8192 data: 5 4
20540 073/300 Write block 4329 len 32768 data: 4 4 4 3 4 3 3 5
20541 074/300 Read block 334 len 24576
20542 075/300 Write block 1460 len 8192 data: 3 4
20543 076/300 Read block 4678 len 36864
20544 077/300 Read block 3287 len 20480
20545 078/300 Read block 45 len 28672
20546 079/300 Write block 3069 len 16384 data: 4 4 4 3
20547 080/300 Read block 4699 len 16384
20548 081/300 Read block 4610 len 20480
20549 082/300 Flush
20550 083/300 Read block 765 len 24576
20551 084/300 Read block 4764 len 24576
20552 085/300 Write block 2348 len 16384 data: 5 5 5 6
20553 086/300 Read block 2062 len 8192
20554 087/300 Flush
20555 088/300 Read block 4672 len 12288
20556 089/300 Read block 1752 len 4096
20557 090/300 Flush
20558 091/300 Write block 1646 len 24576 data: 8 7 7 7 6 5
20559 092/300 Write block 2314 len 4096 data: 3
20560 093/300 Read block 1352 len 20480
20561 094/300 Write block 3102 len 24576 data: 5 5 5 3 3 4
20562 095/300 Write block 3017 len 36864 data: 3 2 2 2 2 2 4 3 3
20563 096/300 Read block 3832 len 32768
20564 097/300 Flush
20565 098/300 Write block 4587 len 20480 data: 3 5 4 4 6
20566 099/300 Read block 1799 len 12288
20567 100/300 Read block 3186 len 40960
20568 101/300 Write block 3322 len 16384 data: 3 3 2 2
20569 102/300 Read block 1290 len 4096
20570 103/300 Read block 3305 len 32768
20571 104/300 Read block 3228 len 32768
20572 105/300 Read block 4782 len 4096
20573 106/300 Read block 2702 len 8192
20574 107/300 Read block 256 len 36864
20575 108/300 Flush
20576 109/300 Write block 4050 len 24576 data: 3 3 2 2 2 3
20577 110/300 Read block 4330 len 32768
20578 111/300 Read block 4616 len 40960
20579 112/300 Read block 3345 len 24576
20580 113/300 Write block 367 len 8192 data: 5 5
20581 114/300 Read block 1758 len 16384
20582 115/300 Write block 2424 len 40960 data: 4 4 4 4 4 4 4 4 3 4
20583 116/300 Write block 4891 len 28672 data: 4 3 3 3 3 4 4
20584 117/300 Read block 2486 len 4096
20585 118/300 Read block 438 len 8192
20586 119/300 Write block 2573 len 24576 data: 4 4 4 4 3 5
20587 120/300 Flush
20588 121/300 Write block 4410 len 40960 data: 3 4 3 4 4 4 3 2 3 3
20589 122/300 Read block 4700 len 36864
20590 123/300 Read block 3062 len 32768
20591 124/300 Write block 4162 len 24576 data: 2 2 3 3 3 4
20592 125/300 Write block 4802 len 28672 data: 5 5 5 4 5 4 5
20593 126/300 Flush
20594 127/300 Write block 456 len 4096 data: 7
20595 128/300 Flush
20596 129/300 Write block 1455 len 24576 data: 5 6 6 4 5 4
20597 130/300 Write block 2255 len 28672 data: 5 6 7 5 5 6 7
20598 131/300 Write block 4369 len 8192 data: 5 4
20599 132/300 Flush
20600 133/300 Flush
20601 134/300 Read block 1383 len 8192
20602 135/300 Write block 1864 len 20480 data: 2 3 2 2 3
20603 136/300 Write block 2535 len 32768 data: 3 4 4 5 5 4 5 5
20604 137/300 Read block 1171 len 24576
20605 138/300 Write block 2817 len 32768 data: 2 2 3 4 4 5 4 4
20606 139/300 Read block 4263 len 32768
20607 140/300 Read block 2661 len 32768
20608 141/300 Write block 3479 len 40960 data: 4 4 5 4 3 4 5 5 6 6
20609 142/300 Read block 1531 len 16384
20610 143/300 Read block 514 len 40960
20611 144/300 Read block 1373 len 12288
20612 145/300 Flush
20613 146/300 Read block 3202 len 12288
20614 147/300 Read block 3286 len 36864
20615 148/300 Read block 2470 len 40960
20616 149/300 Write block 3075 len 36864 data: 4 6 4 4 4 5 5 6 6
20617 150/300 Read block 3143 len 36864
20618 151/300 Read block 3818 len 40960
20619 152/300 Read block 3296 len 4096
20620 153/300 Read block 827 len 24576
20621 154/300 Write block 2680 len 12288 data: 3 3 3
20622 155/300 Read block 1681 len 16384
20623 156/300 Write block 988 len 36864 data: 3 3 2 3 4 6 6 6 6
20624 157/300 Flush
20625 158/300 Write block 26 len 28672 data: 6 7 7 5 4 5 3
20626 159/300 Flush
20627 160/300 Read block 1345 len 20480
20628 161/300 Write block 912 len 36864 data: 5 4 4 3 4 4 3 5 4
20629 162/300 Read block 2054 len 4096
20630 163/300 Read block 1125 len 16384
20631 164/300 Write block 2021 len 12288 data: 4 3 2
20632 165/300 Write block 2710 len 28672 data: 4 3 3 4 3 5 6
20633 166/300 Write block 1627 len 40960 data: 6 7 5 5 6 5 5 5 6 4
20634 167/300 Write block 4177 len 20480 data: 3 3 3 4 3
20635 168/300 Write block 4302 len 4096 data: 5
20636 169/300 Read block 607 len 24576
20637 170/300 Read block 4408 len 40960
20638 171/300 Read block 3395 len 32768
20639 172/300 Read block 4954 len 16384
20640 173/300 Write block 3496 len 16384 data: 4 5 4 4
20641 174/300 Read block 4330 len 24576
20642 175/300 Write block 2421 len 36864 data: 4 4 4 5 5 5 5 5 5
20643 176/300 Read block 2594 len 40960
20644 177/300 Write block 1693 len 40960 data: 3 3 2 2 3 3 4 6 5 6
20645 178/300 Flush
20646 179/300 Write block 2430 len 8192 data: 5 5
20647 180/300 Read block 3551 len 4096
20648 181/300 Write block 1289 len 24576 data: 4 5 7 5 5 4
20649 182/300 Read block 324 len 36864
20650 183/300 Flush
20651 184/300 Read block 4368 len 4096
20652 185/300 Read block 3113 len 40960
20653 186/300 Read block 2336 len 24576
20654 187/300 Flush
20655 188/300 Read block 776 len 28672
20656 189/300 Write block 128 len 8192 data: 4 5
20657 190/300 Write block 734 len 32768 data: 4 3 3 4 4 5 4 6
20658 191/300 Read block 4147 len 28672
20659 192/300 Write block 3139 len 16384 data: 4 5 4 4
20660 193/300 Read block 2947 len 24576
20661 194/300 Write block 2896 len 24576 data: 5 5 4 5 7 9
20662 195/300 Read block 3130 len 32768
20663 196/300 Read block 2776 len 16384
20664 197/300 Read block 4605 len 16384
20665 198/300 Read block 3515 len 40960
20666 199/300 Read block 52 len 32768
20667 200/300 Read block 1268 len 40960
20668 201/300 Write block 886 len 20480 data: 6 5 5 4 4
20669 202/300 Read block 2033 len 12288
20670 203/300 Flush
20671 204/300 Write block 3668 len 28672 data: 6 6 8 6 5 5 6
20672 205/300 Write block 4212 len 12288 data: 6 5 5
20673 206/300 Write block 2572 len 4096 data: 5
20674 207/300 Read block 201 len 40960
20675 208/300 Write block 1761 len 16384 data: 3 3 4 4
20676 209/300 Read block 628 len 16384
20677 210/300 Flush
20678 211/300 Read block 2319 len 28672
20679 212/300 Write block 176 len 28672 data: 6 6 5 5 4 4 4
20680 213/300 Read block 2669 len 12288
20681 214/300 Read block 868 len 16384
20682 215/300 Read block 394 len 32768
20683 216/300 Read block 1252 len 32768
20684 217/300 Read block 3922 len 20480
20685 218/300 Write block 4392 len 12288 data: 7 6 6
20686 219/300 Read block 4954 len 12288
20687 220/300 Write block 3381 len 8192 data: 4 3
20688 221/300 Write block 2028 len 32768 data: 3 4 6 7 7 7 7 7
20689 222/300 Write block 3696 len 40960 data: 3 2 4 5 6 6 6 5 6 6
20690 223/300 Read block 3998 len 32768
20691 224/300 Write block 3487 len 20480 data: 7 7 6 5 5
20692 225/300 Read block 4558 len 28672
20693 226/300 Write block 641 len 8192 data: 5 6
20694 227/300 Write block 3140 len 8192 data: 6 5
20695 228/300 Write block 116 len 28672 data: 4 4 3 3 3 3 3
20696 229/300 Read block 907 len 32768
20697 230/300 Read block 750 len 12288
20698 231/300 Read block 3169 len 12288
20699 232/300 Write block 4705 len 16384 data: 5 4 4 5
20700 233/300 Read block 1744 len 12288
20701 234/300 Read block 2742 len 16384
20702 235/300 Write block 4167 len 28672 data: 5 4 4 5 4 4 4
20703 236/300 Read block 2700 len 32768
20704 237/300 Read block 2780 len 20480
20705 238/300 Read block 2608 len 8192
20706 239/300 Flush
20707 240/300 Write block 4799 len 40960 data: 6 6 5 6 6 6 5 6 5 6
20708 241/300 Read block 3381 len 36864
20709 242/300 Write block 2762 len 24576 data: 4 7 8 9 8 9
20710 243/300 Read block 1025 len 36864
20711 244/300 Read block 2745 len 40960
20712 245/300 Flush
20713 246/300 Read block 1463 len 40960
20714 247/300 Read block 2169 len 24576
20715 248/300 Read block 3252 len 12288
20716 249/300 Read block 2136 len 32768
20717 250/300 Read block 1730 len 28672
20718 251/300 Write block 2350 len 36864 data: 6 7 8 7 4 5 4 3 3
20719 252/300 Read block 4547 len 28672
20720 253/300 Write block 4189 len 12288 data: 3 3 3
20721 254/300 Write block 954 len 28672 data: 4 4 3 3 2 2 2
20722 255/300 Write block 4091 len 4096 data: 4
20723 256/300 Read block 4559 len 28672
20724 257/300 Read block 1142 len 24576
20725 258/300 Write block 4176 len 16384 data: 4 4 4 4
20726 259/300 Read block 4896 len 12288
20727 260/300 Flush
20728 261/300 Write block 2455 len 32768 data: 6 7 9 8 9 9 10 9
20729 262/300 Read block 880 len 16384
20730 263/300 Read block 4344 len 20480
20731 264/300 Read block 896 len 4096
20732 265/300 Write block 927 len 8192 data: 6 6
20733 266/300 Write block 4223 len 4096 data: 3
20734 267/300 Read block 1507 len 36864
20735 268/300 Read block 1212 len 4096
20736 269/300 Read block 4285 len 36864
20737 270/300 Read block 1544 len 16384
20738 271/300 Read block 660 len 12288
20739 272/300 Read block 1714 len 8192
20740 273/300 Read block 2450 len 20480
20741 274/300 Read block 2816 len 40960
20742 275/300 Flush
20743 276/300 Read block 4442 len 8192
20744 277/300 Read block 89 len 40960
20745 278/300 Read block 2094 len 36864
20746 279/300 Read block 1270 len 32768
20747 280/300 Flush
20748 281/300 Read block 890 len 4096
20749 282/300 Read block 3011 len 12288
20750 283/300 Read block 2477 len 32768
20751 284/300 Read block 718 len 20480
20752 285/300 Flush
20753 286/300 Write block 74 len 8192 data: 7 6
20754 287/300 Read block 3944 len 40960
20755 288/300 Write block 2772 len 16384 data: 5 5 5 5
20756 289/300 Flush
20757 290/300 Write block 2760 len 16384 data: 3 3 5 8
20758 291/300 Read block 2091 len 28672
20759 292/300 Flush
20760 293/300 Write block 338 len 8192 data: 4 3
20761 294/300 Read block 3427 len 8192
20762 295/300 Write block 1543 len 12288 data: 3 3 3
20763 296/300 Write block 2885 len 12288 data: 4 4 3
20764 297/300 Read block 4234 len 24576
20765 298/300 Write block 294 len 16384 data: 3 2 2 2
20766 299/300 Write block 133 len 32768 data: 4 5 5 5 5 5 4 4
20767 300/300 Write block 1387 len 16384 data: 6 7 6 6
20768 301/300 Read block 4233 len 28672
20769 Main task has work to do, go find it
20770 got dsc Start(0)
20771 start 0
20772 Feb 25 17:22:50.846 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/0, method: POST, req_id: 921aef52-0f39-436e-a54f-54e9172c129e, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
20773 Replay: started 0, returned:Ok(())
20774 ----------------------------------------------------------------
20775 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20776 States: Offline Active Active
20777 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20778 6435 Acked Read 10 Sent Done Done false
20779 6436 Acked Read 2 Sent Done Done false
20780 6437 Acked Flush 0 Sent Done Done false
20781 6438 Acked Write 9 Sent Done Done false
20782 6439 Acked Write 4 Sent Done Done false
20783 6440 Acked Flush 0 Sent Done Done false
20784 6441 Acked Read 7 Sent Done Done false
20785 6442 Acked Write 5 Sent Done Done false
20786 6443 Acked Read 1 Sent Done Done false
20787 6444 Acked Read 6 Sent Done Done false
20788 6445 Acked Write 1 Sent Done Done false
20789 6446 Acked Read 8 Sent Done Done false
20790 6447 Acked Read 2 Sent Done Done false
20791 6448 Acked Read 3 Sent Done Done false
20792 6449 Acked Read 5 Sent Done Done false
20793 6450 Acked Read 9 Sent Done Done false
20794 6451 Acked Write 7 Sent Done Done false
20795 6452 Acked Write 10 Sent Done Done false
20796 6453 Acked Read 10 Sent Done Done false
20797 6454 Acked Read 8 Sent Done Done false
20798 6455 Acked Write 3 Sent Done Done false
20799 6456 Acked Read 6 Sent Done Done false
20800 6457 Acked Read 3 Sent Done Done false
20801 6458 Acked Flush 0 Sent Done Done false
20802 6459 Acked Read 4 Sent Done Done false
20803 6460 Acked Flush 0 Sent Done Done false
20804 6461 Acked Read 2 Sent Done Done false
20805 6462 Acked Read 2 Sent Done Done false
20806 6463 Acked Read 5 Sent Done Done false
20807 6464 Acked Write 10 Sent Done Done false
20808 6465 Acked Read 1 Sent Done Done false
20809 6466 Acked Read 1 Sent Done Done false
20810 6467 Acked Write 3 Sent Done Done false
20811 6468 Acked Flush 0 Sent Done Done false
20812 6469 Acked Write 9 Sent Done Done false
20813 6470 Acked Read 7 Sent Done Done false
20814 6471 Acked Read 8 Sent Done Done false
20815 6472 Acked Flush 0 Sent Done Done false
20816 6473 Acked Read 4 Sent Done Done false
20817 6474 Acked Read 3 Sent Done Done false
20818 6475 Acked Write 3 Sent Done Done false
20819 6476 Acked Write 7 Sent Done Done false
20820 6477 Acked Read 8 Sent Done Done false
20821 6478 Acked Flush 0 Sent Done Done false
20822 6479 Acked Write 3 Sent Done Done false
20823 6480 Acked Read 1 Sent Done Done false
20824 6481 Acked Write 6 Sent Done Done false
20825 6482 Acked Read 10 Sent Done Done false
20826 6483 Acked Flush 0 Sent Done Done false
20827 6484 Acked Write 2 Sent Done Done false
20828 6485 Acked Read 2 Sent Done Done false
20829 6486 Acked Write 9 Sent Done Done false
20830 6487 Acked Read 1 Sent Done Done false
20831 6488 Acked Write 8 Sent Done Done false
20832 6489 Acked Read 8 Sent Done Done false
20833 6490 Acked Write 1 Sent Done Done false
20834 6491 Acked Write 7 Sent Done Done false
20835 6492 Acked Flush 0 Sent Done Done false
20836 6493 Acked Write 3 Sent Done Done false
20837 6494 Acked Read 8 Sent Done Done false
20838 6495 Acked Read 9 Sent Done Done false
20839 6496 Acked Read 4 Sent Done Done false
20840 6497 Acked Write 3 Sent Done Done false
20841 6498 Acked Write 7 Sent Done Done false
20842 6499 Acked Flush 0 Sent Done Done false
20843 6500 Acked Read 3 Sent Done Done false
20844 6501 Acked Read 1 Sent Done Done false
20845 6502 Acked Read 10 Sent Done Done false
20846 6503 Acked Write 3 Sent Done Done false
20847 6504 Acked Read 4 Sent Done Done false
20848 6505 Acked Read 3 Sent Done Done false
20849 6506 Acked Write 2 Sent Done Done false
20850 6507 Acked Write 8 Sent Done Done false
20851 6508 Acked Read 6 Sent Done Done false
20852 6509 Acked Write 2 Sent Done Done false
20853 6510 Acked Read 9 Sent Done Done false
20854 6511 Acked Read 5 Sent Done Done false
20855 6512 Acked Read 7 Sent Done Done false
20856 6513 Acked Write 4 Sent Done Done false
20857 6514 Acked Read 4 Sent Done Done false
20858 6515 Acked Read 5 Sent Done Done false
20859 6516 Acked Flush 0 Sent Done Done false
20860 6517 Acked Read 6 Sent Done Done false
20861 6518 Acked Read 6 Sent Done Done false
20862 6519 Acked Write 4 Sent Done Done false
20863 6520 Acked Read 2 Sent Done Done false
20864 6521 Acked Flush 0 Sent Done Done false
20865 6522 Acked Read 3 Sent Done Done false
20866 6523 Acked Read 1 Sent Done Done false
20867 6524 Acked Flush 0 Sent Done Done false
20868 6525 Acked Write 6 Sent Done Done false
20869 6526 Acked Write 1 Sent Done Done false
20870 6527 Acked Read 5 Sent Done Done false
20871 6528 Acked Write 6 Sent Done Done false
20872 6529 Acked Write 9 Sent Done Done false
20873 6530 Acked Read 8 Sent Done Done false
20874 6531 Acked Flush 0 Sent Done Done false
20875 6532 Acked Write 5 Sent Done Done false
20876 6533 Acked Read 3 Sent Done Done false
20877 6534 Acked Read 10 Sent Done Done false
20878 6535 Acked Write 4 Sent Done Done false
20879 6536 Acked Read 1 Sent Done Done false
20880 6537 Acked Read 8 Sent Done Done false
20881 6538 Acked Read 8 Sent Done Done false
20882 6539 Acked Read 1 Sent Done Done false
20883 6540 Acked Read 2 Sent Done Done false
20884 6541 Acked Read 9 Sent Done Done false
20885 6542 Acked Flush 0 Sent Done Done false
20886 6543 Acked Write 6 Sent Done Done false
20887 6544 Acked Read 8 Sent Done Done false
20888 6545 Acked Read 10 Sent Done Done false
20889 6546 Acked Read 6 Sent Done Done false
20890 6547 Acked Write 2 Sent Done Done false
20891 6548 Acked Read 4 Sent Done Done false
20892 6549 Acked Write 10 Sent Done Done false
20893 6550 Acked Write 7 Sent Done Done false
20894 6551 Acked Read 1 Sent Done Done false
20895 6552 Acked Read 2 Sent Done Done false
20896 6553 Acked Write 6 Sent Done Done false
20897 6554 Acked Flush 0 Sent Done Done false
20898 6555 Acked Write 10 Sent Done Done false
20899 6556 Acked Read 9 Sent Done Done false
20900 6557 Acked Read 8 Sent Done Done false
20901 6558 Acked Write 6 Sent Done Done false
20902 6559 Acked Write 7 Sent Done Done false
20903 6560 Acked Flush 0 Sent Done Done false
20904 6561 Acked Write 1 Sent Done Done false
20905 6562 Acked Flush 0 Sent Done Done false
20906 6563 Acked Write 6 Sent Done Done false
20907 6564 Acked Write 7 Sent Done Done false
20908 6565 Acked Write 2 Sent Done Done false
20909 6566 Acked Flush 0 Sent Done Done false
20910 6567 Acked Flush 0 Sent Done Done false
20911 6568 Acked Read 2 Sent Done Done false
20912 6569 Acked Write 5 Sent Done Done false
20913 6570 Acked Write 8 Sent Done Done false
20914 6571 Acked Read 6 Sent Done Done false
20915 6572 Acked Write 8 Sent Done Done false
20916 6573 Acked Read 8 Sent Done Done false
20917 6574 Acked Read 8 Sent Done Done false
20918 6575 Acked Write 10 Sent Done Done false
20919 6576 Acked Read 4 Sent Done Done false
20920 6577 Acked Read 10 Sent Done Done false
20921 6578 Acked Read 3 Sent Done Done false
20922 6579 Acked Flush 0 Sent Done Done false
20923 6580 Acked Read 3 Sent Done Done false
20924 6581 Acked Read 9 Sent Done Done false
20925 6582 Acked Read 10 Sent Done Done false
20926 6583 Acked Write 9 Sent Done Done false
20927 6584 Acked Read 9 Sent Done Done false
20928 6585 Acked Read 10 Sent Done Done false
20929 6586 Acked Read 1 Sent Done Done false
20930 6587 Acked Read 6 Sent Done Done false
20931 6588 Acked Write 3 Sent Done Done false
20932 6589 Acked Read 4 Sent Done Done false
20933 6590 Acked Write 9 Sent Done Done false
20934 6591 Acked Flush 0 Sent Done Done false
20935 6592 Acked Write 7 Sent Done Done false
20936 6593 Acked Flush 0 Sent Done Done false
20937 6594 Acked Read 5 Sent Done Done false
20938 6595 Acked Write 9 Sent Done Done false
20939 6596 Acked Read 1 Sent Done Done false
20940 6597 Acked Read 4 Sent Done Done false
20941 6598 Acked Write 3 Sent Done Done false
20942 6599 Acked Write 7 Sent Done Done false
20943 6600 Acked Write 10 Sent Done Done false
20944 6601 Acked Write 5 Sent Done Done false
20945 6602 Acked Write 1 Sent Done Done false
20946 6603 Acked Read 6 Sent Done Done false
20947 6604 Acked Read 10 Sent Done Done false
20948 6605 Acked Read 8 Sent Done Done false
20949 6606 Acked Read 4 Sent Done Done false
20950 6607 Acked Write 4 Sent Done Done false
20951 6608 Acked Read 6 Sent Done Done false
20952 6609 Acked Write 9 Sent Done Done false
20953 6610 Acked Read 10 Sent Done Done false
20954 6611 Acked Write 10 Sent Done Done false
20955 6612 Acked Flush 0 Sent Done Done false
20956 6613 Acked Write 2 Sent Done Done false
20957 6614 Acked Read 1 Sent Done Done false
20958 6615 Acked Write 6 Sent Done Done false
20959 6616 Acked Read 9 Sent Done Done false
20960 6617 Acked Flush 0 Sent Done Done false
20961 6618 Acked Read 1 Sent Done Done false
20962 6619 Acked Read 10 Sent Done Done false
20963 6620 Acked Read 6 Sent Done Done false
20964 6621 Acked Flush 0 Sent Done Done false
20965 6622 Acked Read 7 Sent Done Done false
20966 6623 Acked Write 2 Sent Done Done false
20967 6624 Acked Write 8 Sent Done Done false
20968 6625 Acked Read 7 Sent Done Done false
20969 6626 Acked Write 4 Sent Done Done false
20970 6627 Acked Read 6 Sent Done Done false
20971 6628 Acked Write 6 Sent Done Done false
20972 6629 Acked Read 8 Sent Done Done false
20973 6630 Acked Read 4 Sent Done Done false
20974 6631 Acked Read 4 Sent Done Done false
20975 6632 Acked Read 10 Sent Done Done false
20976 6633 Acked Read 8 Sent Done Done false
20977 6634 Acked Read 10 Sent Done Done false
20978 6635 Acked Write 5 Sent Done Done false
20979 6636 Acked Read 3 Sent Done Done false
20980 6637 Acked Flush 0 Sent Done Done false
20981 6638 Acked Write 7 Sent Done Done false
20982 6639 Acked Write 3 Sent Done Done false
20983 6640 Acked Write 1 Sent Done Done false
20984 6641 Acked Read 10 Sent Done Done false
20985 6642 Acked Write 4 Sent Done Done false
20986 6643 Acked Read 4 Sent Done Done false
20987 6644 Acked Flush 0 Sent Done Done false
20988 6645 Acked Read 7 Sent Done Done false
20989 6646 Acked Write 7 Sent Done Done false
20990 6647 Acked Read 3 Sent Done Done false
20991 6648 Acked Read 4 Sent Done Done false
20992 6649 Acked Read 8 Sent Done Done false
20993 6650 Acked Read 8 Sent Done Done false
20994 6651 Acked Read 5 Sent Done Done false
20995 6652 Acked Write 3 Sent Done Done false
20996 6653 Acked Read 3 Sent Done Done false
20997 6654 Acked Write 2 Sent Done Done false
20998 6655 Acked Write 8 Sent Done Done false
20999 6656 Acked Write 10 Sent Done Done false
21000 6657 Acked Read 8 Sent Done Done false
21001 6658 Acked Write 5 Sent Done Done false
21002 6659 Acked Read 7 Sent Done Done false
21003 6660 Acked Write 2 Sent Done Done false
21004 6661 Acked Write 2 Sent Done Done false
21005 6662 Acked Write 7 Sent Done Done false
21006 6663 Acked Read 8 Sent Done Done false
21007 6664 Acked Read 3 Sent Done Done false
21008 6665 Acked Read 3 Sent Done Done false
21009 6666 Acked Write 4 Sent Done Done false
21010 6667 Acked Read 3 Sent Done Done false
21011 6668 Acked Read 4 Sent Done Done false
21012 6669 Acked Write 7 Sent Done Done false
21013 6670 Acked Read 8 Sent Done Done false
21014 6671 Acked Read 5 Sent Done Done false
21015 6672 Acked Read 2 Sent Done Done false
21016 6673 Acked Flush 0 Sent Done Done false
21017 6674 Acked Write 10 Sent Done Done false
21018 6675 Acked Read 9 Sent Done Done false
21019 6676 Acked Write 6 Sent Done Done false
21020 6677 Acked Read 9 Sent Done Done false
21021 6678 Acked Read 10 Sent Done Done false
21022 6679 Acked Flush 0 Sent Done Done false
21023 6680 Acked Read 10 Sent Done Done false
21024 6681 Acked Read 6 Sent Done Done false
21025 6682 Acked Read 3 Sent Done Done false
21026 6683 Acked Read 8 Sent Done Done false
21027 6684 Acked Read 7 Sent Done Done false
21028 6685 Acked Write 9 Sent Done Done false
21029 6686 Acked Read 7 Sent Done Done false
21030 6687 Acked Write 3 Sent Done Done false
21031 6688 Acked Write 7 Sent Done Done false
21032 6689 Acked Write 1 Sent Done Done false
21033 6690 Acked Read 7 Sent Done Done false
21034 6691 Acked Read 6 Sent Done Done false
21035 6692 Acked Write 4 Sent Done Done false
21036 6693 Acked Read 3 Sent Done Done false
21037 6694 Acked Flush 0 Sent Done Done false
21038 6695 Acked Write 8 Sent Done Done false
21039 6696 Acked Read 4 Sent Done Done false
21040 6697 Acked Read 5 Sent Done Done false
21041 6698 Acked Read 1 Sent Done Done false
21042 6699 Acked Write 2 Sent Done Done false
21043 6700 Acked Write 1 Sent Done Done false
21044 6701 Acked Read 9 Sent Done Done false
21045 6702 Acked Read 1 Sent Done Done false
21046 6703 Acked Read 9 Sent Done Done false
21047 6704 Acked Read 4 Sent Done Done false
21048 6705 Acked Read 3 Sent Done Done false
21049 6706 Acked Read 2 Sent Done Done false
21050 6707 Acked Read 5 Sent Done Done false
21051 6708 Acked Read 10 Sent Done Done false
21052 6709 Acked Flush 0 Sent Done Done false
21053 6710 Acked Read 2 Sent Done Done false
21054 6711 Acked Read 10 Sent Done Done false
21055 6712 Acked Read 9 Sent Done Done false
21056 6713 Acked Read 8 Sent Done Done false
21057 6714 Acked Flush 0 Sent Done Done false
21058 6715 Acked Read 1 Sent Done Done false
21059 6716 Acked Read 3 Sent Done Done false
21060 6717 Acked Read 8 Sent Done Done false
21061 6718 Acked Read 5 Sent Done Done false
21062 6719 Acked Flush 0 Sent Done Done false
21063 6720 Acked Write 2 Sent Done Done false
21064 6721 Acked Read 10 Sent Done Done false
21065 6722 Acked Write 4 Sent Done Done false
21066 6723 Acked Flush 0 Sent Done Done false
21067 6724 Acked Write 4 Sent Done Done false
21068 6725 Acked Read 7 Sent Done Done false
21069 6726 Acked Flush 0 Sent Done Done false
21070 6727 Acked Write 2 Sent Done Done false
21071 6728 Acked Read 2 Sent Done Done false
21072 6729 Acked Write 3 Sent Done Done false
21073 6730 Acked Write 3 Sent Done Done false
21074 6731 Acked Read 6 Sent Done Done false
21075 6732 Acked Write 4 Sent Done Done false
21076 6733 Acked Write 8 Sent Done Done false
21077 6734 Acked Write 4 Sent Done Done false
21078 6735 Acked Read 7 Sent Done Done false
21079 STATES DS:0 DS:1 DS:2 TOTAL
21080 Sent 301 0 0 301
21081 Done 0 301 301 602
21082 Skipped 0 0 0 0
21083 Error 0 0 0 0
21084 Last Flush: Some(JobId(6434)) Some(JobId(6726)) Some(JobId(6726))
21085 Downstairs last five completed: 6434 6433 6432 6431 6430
21086 Upstairs last five completed: 6735 6734 6733 6732 6731
21087 CLIENT: Up:0 ds:301 act:2
210882025-02-25T17:22:51.718ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21089 [8810] got start action
21090 [8810] I am going to restart
21091 Starting downstairs at port 8810
21092 Make output file at "/tmp/dsc/downstairs-8810.txt"
21093 [8810][0] reports Starting
21094 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1227)
21095 [8810][0] reports Running
210962025-02-25T17:22:52.720ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
210972025-02-25T17:22:53.722ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
210982025-02-25T17:22:54.822ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21099 ----------------------------------------------------------------
21100 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21101 States: Offline Active Active
21102 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21103 6435 Acked Read 10 Sent Done Done false
21104 6436 Acked Read 2 Sent Done Done false
21105 6437 Acked Flush 0 Sent Done Done false
21106 6438 Acked Write 9 Sent Done Done false
21107 6439 Acked Write 4 Sent Done Done false
21108 6440 Acked Flush 0 Sent Done Done false
21109 6441 Acked Read 7 Sent Done Done false
21110 6442 Acked Write 5 Sent Done Done false
21111 6443 Acked Read 1 Sent Done Done false
21112 6444 Acked Read 6 Sent Done Done false
21113 6445 Acked Write 1 Sent Done Done false
21114 6446 Acked Read 8 Sent Done Done false
21115 6447 Acked Read 2 Sent Done Done false
21116 6448 Acked Read 3 Sent Done Done false
21117 6449 Acked Read 5 Sent Done Done false
21118 6450 Acked Read 9 Sent Done Done false
21119 6451 Acked Write 7 Sent Done Done false
21120 6452 Acked Write 10 Sent Done Done false
21121 6453 Acked Read 10 Sent Done Done false
21122 6454 Acked Read 8 Sent Done Done false
21123 6455 Acked Write 3 Sent Done Done false
21124 6456 Acked Read 6 Sent Done Done false
21125 6457 Acked Read 3 Sent Done Done false
21126 6458 Acked Flush 0 Sent Done Done false
21127 6459 Acked Read 4 Sent Done Done false
21128 6460 Acked Flush 0 Sent Done Done false
21129 6461 Acked Read 2 Sent Done Done false
21130 6462 Acked Read 2 Sent Done Done false
21131 6463 Acked Read 5 Sent Done Done false
21132 6464 Acked Write 10 Sent Done Done false
21133 6465 Acked Read 1 Sent Done Done false
21134 6466 Acked Read 1 Sent Done Done false
21135 6467 Acked Write 3 Sent Done Done false
21136 6468 Acked Flush 0 Sent Done Done false
21137 6469 Acked Write 9 Sent Done Done false
21138 6470 Acked Read 7 Sent Done Done false
21139 6471 Acked Read 8 Sent Done Done false
21140 6472 Acked Flush 0 Sent Done Done false
21141 6473 Acked Read 4 Sent Done Done false
21142 6474 Acked Read 3 Sent Done Done false
21143 6475 Acked Write 3 Sent Done Done false
21144 6476 Acked Write 7 Sent Done Done false
21145 6477 Acked Read 8 Sent Done Done false
21146 6478 Acked Flush 0 Sent Done Done false
21147 6479 Acked Write 3 Sent Done Done false
21148 6480 Acked Read 1 Sent Done Done false
21149 6481 Acked Write 6 Sent Done Done false
21150 6482 Acked Read 10 Sent Done Done false
21151 6483 Acked Flush 0 Sent Done Done false
21152 6484 Acked Write 2 Sent Done Done false
21153 6485 Acked Read 2 Sent Done Done false
21154 6486 Acked Write 9 Sent Done Done false
21155 6487 Acked Read 1 Sent Done Done false
21156 6488 Acked Write 8 Sent Done Done false
21157 6489 Acked Read 8 Sent Done Done false
21158 6490 Acked Write 1 Sent Done Done false
21159 6491 Acked Write 7 Sent Done Done false
21160 6492 Acked Flush 0 Sent Done Done false
21161 6493 Acked Write 3 Sent Done Done false
21162 6494 Acked Read 8 Sent Done Done false
21163 6495 Acked Read 9 Sent Done Done false
21164 6496 Acked Read 4 Sent Done Done false
21165 6497 Acked Write 3 Sent Done Done false
21166 6498 Acked Write 7 Sent Done Done false
21167 6499 Acked Flush 0 Sent Done Done false
21168 6500 Acked Read 3 Sent Done Done false
21169 6501 Acked Read 1 Sent Done Done false
21170 6502 Acked Read 10 Sent Done Done false
21171 6503 Acked Write 3 Sent Done Done false
21172 6504 Acked Read 4 Sent Done Done false
21173 6505 Acked Read 3 Sent Done Done false
21174 6506 Acked Write 2 Sent Done Done false
21175 6507 Acked Write 8 Sent Done Done false
21176 6508 Acked Read 6 Sent Done Done false
21177 6509 Acked Write 2 Sent Done Done false
21178 6510 Acked Read 9 Sent Done Done false
21179 6511 Acked Read 5 Sent Done Done false
21180 6512 Acked Read 7 Sent Done Done false
21181 6513 Acked Write 4 Sent Done Done false
21182 6514 Acked Read 4 Sent Done Done false
21183 6515 Acked Read 5 Sent Done Done false
21184 6516 Acked Flush 0 Sent Done Done false
21185 6517 Acked Read 6 Sent Done Done false
21186 6518 Acked Read 6 Sent Done Done false
21187 6519 Acked Write 4 Sent Done Done false
21188 6520 Acked Read 2 Sent Done Done false
21189 6521 Acked Flush 0 Sent Done Done false
21190 6522 Acked Read 3 Sent Done Done false
21191 6523 Acked Read 1 Sent Done Done false
21192 6524 Acked Flush 0 Sent Done Done false
21193 6525 Acked Write 6 Sent Done Done false
21194 6526 Acked Write 1 Sent Done Done false
21195 6527 Acked Read 5 Sent Done Done false
21196 6528 Acked Write 6 Sent Done Done false
21197 6529 Acked Write 9 Sent Done Done false
21198 6530 Acked Read 8 Sent Done Done false
21199 6531 Acked Flush 0 Sent Done Done false
21200 6532 Acked Write 5 Sent Done Done false
21201 6533 Acked Read 3 Sent Done Done false
21202 6534 Acked Read 10 Sent Done Done false
21203 6535 Acked Write 4 Sent Done Done false
21204 6536 Acked Read 1 Sent Done Done false
21205 6537 Acked Read 8 Sent Done Done false
21206 6538 Acked Read 8 Sent Done Done false
21207 6539 Acked Read 1 Sent Done Done false
21208 6540 Acked Read 2 Sent Done Done false
21209 6541 Acked Read 9 Sent Done Done false
21210 6542 Acked Flush 0 Sent Done Done false
21211 6543 Acked Write 6 Sent Done Done false
21212 6544 Acked Read 8 Sent Done Done false
21213 6545 Acked Read 10 Sent Done Done false
21214 6546 Acked Read 6 Sent Done Done false
21215 6547 Acked Write 2 Sent Done Done false
21216 6548 Acked Read 4 Sent Done Done false
21217 6549 Acked Write 10 Sent Done Done false
21218 6550 Acked Write 7 Sent Done Done false
21219 6551 Acked Read 1 Sent Done Done false
21220 6552 Acked Read 2 Sent Done Done false
21221 6553 Acked Write 6 Sent Done Done false
21222 6554 Acked Flush 0 Sent Done Done false
21223 6555 Acked Write 10 Sent Done Done false
21224 6556 Acked Read 9 Sent Done Done false
21225 6557 Acked Read 8 Sent Done Done false
21226 6558 Acked Write 6 Sent Done Done false
21227 6559 Acked Write 7 Sent Done Done false
21228 6560 Acked Flush 0 Sent Done Done false
21229 6561 Acked Write 1 Sent Done Done false
21230 6562 Acked Flush 0 Sent Done Done false
21231 6563 Acked Write 6 Sent Done Done false
21232 6564 Acked Write 7 Sent Done Done false
21233 6565 Acked Write 2 Sent Done Done false
21234 6566 Acked Flush 0 Sent Done Done false
21235 6567 Acked Flush 0 Sent Done Done false
21236 6568 Acked Read 2 Sent Done Done false
21237 6569 Acked Write 5 Sent Done Done false
21238 6570 Acked Write 8 Sent Done Done false
21239 6571 Acked Read 6 Sent Done Done false
21240 6572 Acked Write 8 Sent Done Done false
21241 6573 Acked Read 8 Sent Done Done false
21242 6574 Acked Read 8 Sent Done Done false
21243 6575 Acked Write 10 Sent Done Done false
21244 6576 Acked Read 4 Sent Done Done false
21245 6577 Acked Read 10 Sent Done Done false
21246 6578 Acked Read 3 Sent Done Done false
21247 6579 Acked Flush 0 Sent Done Done false
21248 6580 Acked Read 3 Sent Done Done false
21249 6581 Acked Read 9 Sent Done Done false
21250 6582 Acked Read 10 Sent Done Done false
21251 6583 Acked Write 9 Sent Done Done false
21252 6584 Acked Read 9 Sent Done Done false
21253 6585 Acked Read 10 Sent Done Done false
21254 6586 Acked Read 1 Sent Done Done false
21255 6587 Acked Read 6 Sent Done Done false
21256 6588 Acked Write 3 Sent Done Done false
21257 6589 Acked Read 4 Sent Done Done false
21258 6590 Acked Write 9 Sent Done Done false
21259 6591 Acked Flush 0 Sent Done Done false
21260 6592 Acked Write 7 Sent Done Done false
21261 6593 Acked Flush 0 Sent Done Done false
21262 6594 Acked Read 5 Sent Done Done false
21263 6595 Acked Write 9 Sent Done Done false
21264 6596 Acked Read 1 Sent Done Done false
21265 6597 Acked Read 4 Sent Done Done false
21266 6598 Acked Write 3 Sent Done Done false
21267 6599 Acked Write 7 Sent Done Done false
21268 6600 Acked Write 10 Sent Done Done false
21269 6601 Acked Write 5 Sent Done Done false
21270 6602 Acked Write 1 Sent Done Done false
21271 6603 Acked Read 6 Sent Done Done false
21272 6604 Acked Read 10 Sent Done Done false
21273 6605 Acked Read 8 Sent Done Done false
21274 6606 Acked Read 4 Sent Done Done false
21275 6607 Acked Write 4 Sent Done Done false
21276 6608 Acked Read 6 Sent Done Done false
21277 6609 Acked Write 9 Sent Done Done false
21278 6610 Acked Read 10 Sent Done Done false
21279 6611 Acked Write 10 Sent Done Done false
21280 6612 Acked Flush 0 Sent Done Done false
21281 6613 Acked Write 2 Sent Done Done false
21282 6614 Acked Read 1 Sent Done Done false
21283 6615 Acked Write 6 Sent Done Done false
21284 6616 Acked Read 9 Sent Done Done false
21285 6617 Acked Flush 0 Sent Done Done false
21286 6618 Acked Read 1 Sent Done Done false
21287 6619 Acked Read 10 Sent Done Done false
21288 6620 Acked Read 6 Sent Done Done false
21289 6621 Acked Flush 0 Sent Done Done false
21290 6622 Acked Read 7 Sent Done Done false
21291 6623 Acked Write 2 Sent Done Done false
21292 6624 Acked Write 8 Sent Done Done false
21293 6625 Acked Read 7 Sent Done Done false
21294 6626 Acked Write 4 Sent Done Done false
21295 6627 Acked Read 6 Sent Done Done false
21296 6628 Acked Write 6 Sent Done Done false
21297 6629 Acked Read 8 Sent Done Done false
21298 6630 Acked Read 4 Sent Done Done false
21299 6631 Acked Read 4 Sent Done Done false
21300 6632 Acked Read 10 Sent Done Done false
21301 6633 Acked Read 8 Sent Done Done false
21302 6634 Acked Read 10 Sent Done Done false
21303 6635 Acked Write 5 Sent Done Done false
21304 6636 Acked Read 3 Sent Done Done false
21305 6637 Acked Flush 0 Sent Done Done false
21306 6638 Acked Write 7 Sent Done Done false
21307 6639 Acked Write 3 Sent Done Done false
21308 6640 Acked Write 1 Sent Done Done false
21309 6641 Acked Read 10 Sent Done Done false
21310 6642 Acked Write 4 Sent Done Done false
21311 6643 Acked Read 4 Sent Done Done false
21312 6644 Acked Flush 0 Sent Done Done false
21313 6645 Acked Read 7 Sent Done Done false
21314 6646 Acked Write 7 Sent Done Done false
21315 6647 Acked Read 3 Sent Done Done false
21316 6648 Acked Read 4 Sent Done Done false
21317 6649 Acked Read 8 Sent Done Done false
21318 6650 Acked Read 8 Sent Done Done false
21319 6651 Acked Read 5 Sent Done Done false
21320 6652 Acked Write 3 Sent Done Done false
21321 6653 Acked Read 3 Sent Done Done false
21322 6654 Acked Write 2 Sent Done Done false
21323 6655 Acked Write 8 Sent Done Done false
21324 6656 Acked Write 10 Sent Done Done false
21325 6657 Acked Read 8 Sent Done Done false
21326 6658 Acked Write 5 Sent Done Done false
21327 6659 Acked Read 7 Sent Done Done false
21328 6660 Acked Write 2 Sent Done Done false
21329 6661 Acked Write 2 Sent Done Done false
21330 6662 Acked Write 7 Sent Done Done false
21331 6663 Acked Read 8 Sent Done Done false
21332 6664 Acked Read 3 Sent Done Done false
21333 6665 Acked Read 3 Sent Done Done false
21334 6666 Acked Write 4 Sent Done Done false
21335 6667 Acked Read 3 Sent Done Done false
21336 6668 Acked Read 4 Sent Done Done false
21337 6669 Acked Write 7 Sent Done Done false
21338 6670 Acked Read 8 Sent Done Done false
21339 6671 Acked Read 5 Sent Done Done false
21340 6672 Acked Read 2 Sent Done Done false
21341 6673 Acked Flush 0 Sent Done Done false
21342 6674 Acked Write 10 Sent Done Done false
21343 6675 Acked Read 9 Sent Done Done false
21344 6676 Acked Write 6 Sent Done Done false
21345 6677 Acked Read 9 Sent Done Done false
21346 6678 Acked Read 10 Sent Done Done false
21347 6679 Acked Flush 0 Sent Done Done false
21348 6680 Acked Read 10 Sent Done Done false
21349 6681 Acked Read 6 Sent Done Done false
21350 6682 Acked Read 3 Sent Done Done false
21351 6683 Acked Read 8 Sent Done Done false
21352 6684 Acked Read 7 Sent Done Done false
21353 6685 Acked Write 9 Sent Done Done false
21354 6686 Acked Read 7 Sent Done Done false
21355 6687 Acked Write 3 Sent Done Done false
21356 6688 Acked Write 7 Sent Done Done false
21357 6689 Acked Write 1 Sent Done Done false
21358 6690 Acked Read 7 Sent Done Done false
21359 6691 Acked Read 6 Sent Done Done false
21360 6692 Acked Write 4 Sent Done Done false
21361 6693 Acked Read 3 Sent Done Done false
21362 6694 Acked Flush 0 Sent Done Done false
21363 6695 Acked Write 8 Sent Done Done false
21364 6696 Acked Read 4 Sent Done Done false
21365 6697 Acked Read 5 Sent Done Done false
21366 6698 Acked Read 1 Sent Done Done false
21367 6699 Acked Write 2 Sent Done Done false
21368 6700 Acked Write 1 Sent Done Done false
21369 6701 Acked Read 9 Sent Done Done false
21370 6702 Acked Read 1 Sent Done Done false
21371 6703 Acked Read 9 Sent Done Done false
21372 6704 Acked Read 4 Sent Done Done false
21373 6705 Acked Read 3 Sent Done Done false
21374 6706 Acked Read 2 Sent Done Done false
21375 6707 Acked Read 5 Sent Done Done false
21376 6708 Acked Read 10 Sent Done Done false
21377 6709 Acked Flush 0 Sent Done Done false
21378 6710 Acked Read 2 Sent Done Done false
21379 6711 Acked Read 10 Sent Done Done false
21380 6712 Acked Read 9 Sent Done Done false
21381 6713 Acked Read 8 Sent Done Done false
21382 6714 Acked Flush 0 Sent Done Done false
21383 6715 Acked Read 1 Sent Done Done false
21384 6716 Acked Read 3 Sent Done Done false
21385 6717 Acked Read 8 Sent Done Done false
21386 6718 Acked Read 5 Sent Done Done false
21387 6719 Acked Flush 0 Sent Done Done false
21388 6720 Acked Write 2 Sent Done Done false
21389 6721 Acked Read 10 Sent Done Done false
21390 6722 Acked Write 4 Sent Done Done false
21391 6723 Acked Flush 0 Sent Done Done false
21392 6724 Acked Write 4 Sent Done Done false
21393 6725 Acked Read 7 Sent Done Done false
21394 6726 Acked Flush 0 Sent Done Done false
21395 6727 Acked Write 2 Sent Done Done false
21396 6728 Acked Read 2 Sent Done Done false
21397 6729 Acked Write 3 Sent Done Done false
21398 6730 Acked Write 3 Sent Done Done false
21399 6731 Acked Read 6 Sent Done Done false
21400 6732 Acked Write 4 Sent Done Done false
21401 6733 Acked Write 8 Sent Done Done false
21402 6734 Acked Write 4 Sent Done Done false
21403 6735 Acked Read 7 Sent Done Done false
21404 STATES DS:0 DS:1 DS:2 TOTAL
21405 Sent 301 0 0 301
21406 Done 0 301 301 602
21407 Skipped 0 0 0 0
21408 Error 0 0 0 0
21409 Last Flush: Some(JobId(6434)) Some(JobId(6726)) Some(JobId(6726))
21410 Downstairs last five completed: 6434 6433 6432 6431 6430
21411 Upstairs last five completed: 6735 6734 6733 6732 6731
21412 CLIENT: Up:0 ds:301 act:2
214132025-02-25T17:22:55.724ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
214142025-02-25T17:22:56.583ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
214152025-02-25T17:22:56.583ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33554) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
214162025-02-25T17:22:56.585ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
214172025-02-25T17:22:56.585ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
214182025-02-25T17:22:56.585ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
214192025-02-25T17:22:56.585ZINFOcrucible: send last flush ID to this DS: Some(JobId(6434)) = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
214202025-02-25T17:22:56.585ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
214212025-02-25T17:22:56.585ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(6434)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
214222025-02-25T17:22:56.585ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(6434)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21423 ----------------------------------------------------------------
21424 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21425 Downstairs last five completed: 6736 6735 6734 6733 6732
21426 Upstairs last five completed: 6736 6735 6734 6733 6732
21427 CLIENT: Up:0 ds:0 act:3
21428 Replay: All jobs finished
21429 Main task has work to do, go find it
21430 got dsc Stop(2)
21431 stop 2
21432 [8830] Got stop action so:false kr:false
21433 Feb 25 17:22:58.850 INFO request completed, latency_us: 364, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 53a2e289-924d-4a6f-8099-dc48c8c73a08, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
214342025-02-25T17:22:58.851ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
214352025-02-25T17:22:58.851ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
214362025-02-25T17:22:58.851ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
21437 Feb 25 17:22:58.851 INFO request completed, latency_us: 159, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3ac8bff9-55df-466e-a697-91a1fe18bad6, remote_addr: 127.0.0.1:52576, local_addr{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T17:22:58.851946984Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
21438 : 127.0.0.1:9998
21439 [8830] Exited with: ExitStatus(unix_wait_status(9))
21440 [8830][2] reports Exit
21441 Feb 25 17:23:02.854 INFO request completed, latency_us: 351, response_code: 200, uri: /state/cid/2, method: GET, req_id: a75f38c8-9684-4595-8515-9dd57621594f, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
21442 001/300 Write block 237 len 32768 data: 3 2 3 4 4 3 3 3
21443 002/300 Read block 2185 len 20480
21444 003/300 Write block 294 len 32768 data: 4 3 3 3 4 5 5 6
21445 004/300 Read block 3826 len 16384
21446 005/300 Read block 546 len 16384
21447 006/300 Write block 923 len 36864 data: 7 6 6 5 7 7 6 7 6
21448 007/300 Write block 3216 len 28672 data: 6 7 9 10 8 9 8
21449 008/300 Read block 3066 len 20480
21450 009/300 Write block 2632 len 28672 data: 4 6 5 5 5 7 7
21451 010/300 Read block 816 len 4096
21452 011/300 Write block 4554 len 12288 data: 4 3 4
21453 012/300 Write block 4988 len 8192 data: 3 5
21454 013/300 Read block 4231 len 36864
21455 014/300 Flush
21456 015/300 Read block 432 len 24576
21457 016/300 Read block 3281 len 16384
21458 017/300 Read block 4616 len 20480
21459 018/300 Read block 2270 len 40960
21460 019/300 Read block 4332 len 8192
21461 020/300 Read block 1618 len 24576
21462 021/300 Write block 4759 len 40960 data: 5 3 5 6 6 6 6 9 8 7
21463 022/300 Write block 4566 len 4096 data: 4
21464 023/300 Read block 2014 len 12288
21465 024/300 Read block 1659 len 16384
21466 025/300 Read block 1843 len 12288
21467 026/300 Write block 1407 len 36864 data: 4 4 4 3 3 4 5 5 5
21468 027/300 Write block 2156 len 16384 data: 4 4 3 3
21469 028/300 Write block 3842 len 36864 data: 5 4 4 4 3 5 5 4 5
21470 029/300 Flush
21471 030/300 Flush
21472 031/300 Read block 2108 len 40960
21473 032/300 Read block 636 len 32768
21474 033/300 Read block 3358 len 4096
21475 034/300 Read block 1439 len 12288
21476 035/300 Flush
21477 036/300 Write block 4493 len 40960 data: 6 5 4 3 3 4 4 4 3 4
21478 037/300 Write block 4612 len 16384 data: 4 5 6 5
21479 038/300 Write block 2721 len 24576 data: 4 4 4 3 3 3
21480 039/300 Read block 1622 len 16384
21481 040/300 Read block 499 len 24576
21482 041/300 Write block 630 len 32768 data: 10 9 8 8 6 5 8 7
21483 042/300 Read block 2608 len 20480
21484 043/300 Write block 1331 len 16384 data: 4 4 4 4
21485 044/300 Write block 2775 len 20480 data: 6 5 2 2 2
21486 045/300 Write block 560 len 8192 data: 4 4
21487 046/300 Read block 2448 len 4096
21488 047/300 Write block 2237 len 16384 data: 2 2 2 5
21489 048/300 Read block 4626 len 8192
21490 049/300 Write block 4011 len 40960 data: 3 3 3 3 4 4 4 4 3 4
21491 050/300 Read block 1441 len 8192
21492 051/300 Write block 26 len 28672 data: 7 8 8 6 5 6 4
21493 052/300 Write block 3115 len 20480 data: 4 4 4 5 4
21494 053/300 Write block 3308 len 20480 data: 5 5 2 4 4
21495 054/300 Write block 4669 len 12288 data: 4 4 4
21496 055/300 Read block 551 len 8192
21497 056/300 Read block 1433 len 4096
21498 057/300 Flush
21499 058/300 Read block 4532 len 16384
21500 059/300 Write block 2369 len 12288 data: 3 3 4
21501 060/300 Write block 1051 len 40960 data: 6 5 7 6 6 6 5 4 5 4
21502 061/300 Read block 2304 len 20480
21503 062/300 Write block 4465 len 28672 data: 5 5 5 6 4 3 3
21504 063/300 Write block 5 len 24576 data: 3 3 3 2 2 4
21505 064/300 Read block 3452 len 20480
21506 065/300 Write block 3785 len 32768 data: 3 3 3 4 4 4 4 4
21507 066/300 Read block 4363 len 28672
21508 067/300 Write block 1614 len 4096 data: 5
21509 068/300 Read block 4764 len 16384
21510 069/300 Read block 705 len 20480
21511 070/300 Write block 3671 len 24576 data: 7 6 6 7 6 6
21512 071/300 Write block 586 len 16384 data: 7 4 5 5
21513 072/300 Read block 1605 len 28672
21514 073/300 Write block 3290 len 4096 data: 4
21515 074/300 Read block 1375 len 12288
21516 075/300 Read block 1375 len 40960
21517 076/300 Read block 3271 len 12288
21518 077/300 Flush
21519 078/300 Read block 4119 len 40960
21520 079/300 Write block 3100 len 8192 data: 5 4
21521 080/300 Write block 188 len 36864 data: 4 3 3 2 2 4 4 4 4
21522 081/300 Read block 1473 len 4096
21523 082/300 Write block 2893 len 12288 data: 4 7 6
21524 083/300 Write block 4495 len 36864 data: 5 4 4 5 5 5 4 5 3
21525 084/300 Write block 1452 len 36864 data: 6 5 5 6 7 7 5 6 5
21526 085/300 Write block 3654 len 16384 data: 8 8 9 7
21527 086/300 Read block 4444 len 4096
21528 087/300 Read block 2307 len 12288
21529 088/300 Read block 1167 len 4096
21530 089/300 Read block 643 len 28672
21531 090/300 Write block 2608 len 16384 data: 4 5 5 4
21532 091/300 Write block 3051 len 20480 data: 5 3 6 7 6
21533 092/300 Write block 2901 len 8192 data: 10 9
21534 093/300 Write block 1974 len 32768 data: 5 6 6 4 5 6 5 5
21535 094/300 Write block 1031 len 32768 data: 4 3 3 4 3 3 3 3
21536 095/300 Write block 4530 len 40960 data: 4 6 5 6 5 6 6 7 7 5
21537 096/300 Read block 2563 len 40960
21538 097/300 Write block 4710 len 8192 data: 3 3
21539 098/300 Read block 199 len 40960
21540 099/300 Read block 1760 len 4096
21541 100/300 Read block 911 len 28672
21542 101/300 Flush
21543 102/300 Write block 1746 len 32768 data: 5 6 7 7 5 4 5 5
21544 103/300 Read block 3205 len 40960
21545 104/300 Read block 2704 len 8192
21546 105/300 Write block 3127 len 24576 data: 5 4 4 3 3 3
21547 106/300 Write block 4142 len 24576 data: 5 4 4 4 4 4
21548 107/300 Read block 4227 len 40960
21549 108/300 Write block 4754 len 24576 data: 5 4 4 4 5 6
21550 109/300 Write block 1384 len 4096 data: 6
21551 110/300 Read block 2705 len 16384
21552 111/300 Write block 1767 len 8192 data: 5 7
21553 112/300 Flush
21554 113/300 Read block 3392 len 36864
21555 114/300 Read block 4001 len 32768
21556 115/300 Read block 3345 len 28672
21557 116/300 Flush
21558 117/300 Read block 139 len 12288
21559 118/300 Write block 1962 len 36864 data: 5 5 4 3 2 2 2 3 3
21560 119/300 Flush
21561 120/300 Read block 2926 len 32768
21562 121/300 Read block 4603 len 40960
21563 122/300 Write block 3008 len 8192 data: 6 5
21564 123/300 Write block 3883 len 16384 data: 2 2 2 2
21565 124/300 Write block 1951 len 20480 data: 4 4 5 6 5
21566 125/300 Write block 1521 len 16384 data: 2 3 4 4
21567 126/300 Read block 1940 len 20480
21568 127/300 Write block 1139 len 32768 data: 2 2 2 2 2 2 4 4
21569 128/300 Write block 1060 len 32768 data: 5 4 7 5 6 4 4 4
21570 129/300 Flush
21571 130/300 Read block 909 len 20480
21572 131/300 Flush
21573 132/300 Read block 3988 len 4096
21574 133/300 Read block 2586 len 12288
21575 134/300 Write block 2450 len 8192 data: 5 4
21576 135/300 Write block 1979 len 36864 data: 7 6 6 5 6 5 4 4 4
21577 136/300 Read block 398 len 4096
21578 137/300 Write block 232 len 8192 data: 5 5
21579 138/300 Write block 2060 len 28672 data: 5 5 3 4 4 6 5
21580 139/300 Read block 1716 len 24576
21581 140/300 Read block 363 len 20480
21582 141/300 Read block 1400 len 24576
21583 142/300 Flush
21584 143/300 Read block 3076 len 4096
21585 144/300 Read block 4221 len 4096
21586 145/300 Read block 530 len 40960
21587 146/300 Write block 1525 len 8192 data: 4 4
21588 147/300 Read block 2604 len 20480
21589 148/300 Write block 3369 len 36864 data: 4 4 4 3 3 3 3 3 3
21590 149/300 Flush
21591 150/300 Read block 281 len 16384
21592 151/300 Write block 4381 len 8192 data: 4 3
21593 152/300 Read block 387 len 16384
21594 153/300 Flush
21595 154/300 Read block 881 len 4096
21596 155/300 Read block 712 len 4096
21597 156/300 Read block 4075 len 8192
21598 157/300 Read block 4748 len 36864
21599 158/300 Write block 3637 len 12288 data: 4 4 3
21600 159/300 Read block 4734 len 32768
21601 160/300 Write block 4911 len 36864 data: 3 3 2 3 3 4 4 4 3
21602 161/300 Read block 4665 len 16384
21603 162/300 Read block 990 len 20480
21604 163/300 Write block 2213 len 32768 data: 8 8 7 7 8 8 7 6
21605 164/300 Write block 2363 len 20480 data: 3 3 3 4 4
21606 165/300 Write block 4254 len 36864 data: 2 3 3 4 4 4 5 5 5
21607 166/300 Write block 2700 len 40960 data: 5 5 3 3 4 4 6 5 5 5
21608 167/300 Write block 2257 len 4096 data: 8
21609 168/300 Read block 4145 len 40960
21610 169/300 Write block 3282 len 12288 data: 3 6 6
21611 170/300 Read block 1146 len 36864
21612 171/300 Read block 4161 len 4096
21613 172/300 Flush
21614 173/300 Flush
21615 174/300 Write block 2780 len 32768 data: 3 5 5 5 4 3 6 6
21616 175/300 Write block 4508 len 32768 data: 2 3 3 3 3 3 4 7
21617 176/300 Flush
21618 177/300 Write block 1124 len 24576 data: 4 4 5 4 4 2
21619 178/300 Write block 4726 len 24576 data: 3 3 3 3 3 3
21620 179/300 Write block 381 len 4096 data: 4
21621 180/300 Read block 2964 len 24576
21622 181/300 Write block 3554 len 40960 data: 9 7 7 3 3 4 4 5 4 4
21623 182/300 Write block 499 len 4096 data: 3
21624 183/300 Read block 4687 len 12288
21625 184/300 Read block 650 len 8192
21626 185/300 Read block 4589 len 16384
21627 186/300 Read block 4341 len 40960
21628 187/300 Read block 2457 len 12288
21629 188/300 Read block 3101 len 32768
21630 189/300 Read block 4050 len 4096
21631 190/300 Read block 1697 len 24576
21632 191/300 Flush
21633 192/300 Write block 1209 len 4096 data: 2
21634 193/300 Write block 3430 len 40960 data: 5 4 3 3 4 3 3 3 3 3
21635 194/300 Write block 4152 len 36864 data: 3 3 3 2 3 4 6 6 5
21636 195/300 Write block 1408 len 32768 data: 5 5 4 4 5 6 6 6
21637 196/300 Write block 4952 len 8192 data: 3 2
21638 197/300 Write block 1405 len 36864 data: 4 3 5 6 6 5 5 6 7
21639 198/300 Read block 2461 len 16384
21640 199/300 Read block 2062 len 32768
21641 200/300 Write block 371 len 28672 data: 4 5 4 4 4 4 6
21642 201/300 Write block 3459 len 28672 data: 3 4 4 4 4 3 3
21643 202/300 Read block 4146 len 28672
21644 203/300 Flush
21645 204/300 Flush
21646 205/300 Write block 1569 len 20480 data: 2 2 3 2 4
21647 206/300 Read block 874 len 4096
21648 207/300 Read block 4587 len 4096
21649 208/300 Read block 3518 len 12288
21650 209/300 Read block 3166 len 12288
21651 210/300 Read block 3689 len 32768
21652 211/300 Flush
21653 212/300 Write block 546 len 40960 data: 4 4 4 4 4 5 5 5 6 6
21654 213/300 Write block 673 len 16384 data: 2 3 3 3
21655 214/300 Read block 3315 len 16384
21656 215/300 Read block 3712 len 20480
21657 216/300 Write block 4451 len 40960 data: 4 5 5 5 4 5 5 4 4 4
21658 217/300 Read block 3852 len 12288
21659 218/300 Read block 1035 len 12288
21660 219/300 Write block 3791 len 32768 data: 5 5 5 5 7 7 5 5
21661 220/300 Write block 2749 len 24576 data: 5 5 6 6 4 4
21662 221/300 Write block 493 len 16384 data: 6 6 6 6
21663 222/300 Write block 597 len 16384 data: 6 5 6 6
21664 223/300 Read block 2470 len 24576
21665 224/300 Write block 1427 len 28672 data: 7 6 6 7 6 4 3
21666 225/300 Write block 2230 len 20480 data: 3 3 3 3 3
21667 226/300 Flush
21668 227/300 Read block 2582 len 24576
21669 228/300 Read block 4771 len 12288
21670 229/300 Write block 45 len 4096 data: 7
21671 230/300 Read block 4279 len 24576
21672 231/300 Write block 1294 len 32768 data: 5 2 3 3 3 3 4 4
21673 232/300 Write block 464 len 16384 data: 2 2 3 4
21674 233/300 Write block 2855 len 4096 data: 4
21675 234/300 Write block 1540 len 24576 data: 5 5 4 4 4 4
21676 235/300 Read block 4710 len 12288
21677 236/300 Read block 4025 len 28672
21678 237/300 Flush
21679 238/300 Write block 904 len 28672 data: 6 3 5 6 5 4 4
21680 239/300 Write block 1713 len 12288 data: 7 6 6
21681 240/300 Read block 1408 len 12288
21682 241/300 Read block 2644 len 4096
21683 242/300 Write block 2307 len 40960 data: 4 3 3 3 3 2 2 4 3 3
21684 243/300 Read block 3644 len 16384
21685 244/300 Write block 2238 len 8192 data: 3 3
21686 245/300 Read block 1021 len 40960
21687 246/300 Read block 4207 len 20480
21688 247/300 Write block 4888 len 12288 data: 2 4 4
21689 248/300 Read block 572 len 16384
21690 249/300 Read block 855 len 36864
21691 250/300 Read block 4159 len 12288
21692 251/300 Flush
21693 252/300 Read block 849 len 20480
21694 253/300 Write block 991 len 4096 data: 4
21695 254/300 Read block 180 len 12288
21696 255/300 Read block 1394 len 12288
21697 256/300 Write block 2 len 12288 data: 2 2 2
21698 257/300 Write block 1934 len 20480 data: 7 8 8 7 4
21699 258/300 Read block 34 len 36864
21700 259/300 Write block 2903 len 24576 data: 9 8 8 7 6 5
21701 260/300 Write block 1666 len 12288 data: 4 4 5
21702 261/300 Read block 4715 len 36864
21703 262/300 Read block 495 len 16384
21704 263/300 Read block 4195 len 4096
21705 264/300 Write block 2518 len 16384 data: 7 5 5 4
21706 265/300 Write block 4455 len 20480 data: 5 6 6 5 5
21707 266/300 Write block 4527 len 16384 data: 2 2 2 5
21708 267/300 Read block 3716 len 24576
21709 268/300 Read block 628 len 16384
21710 269/300 Read block 3427 len 36864
21711 270/300 Write block 3163 len 32768 data: 6 4 4 4 4 4 4 3
21712 271/300 Write block 1936 len 20480 data: 9 8 5 6 6
21713 272/300 Write block 3817 len 32768 data: 7 7 8 6 8 8 8 5
21714 273/300 Read block 2656 len 32768
21715 274/300 Write block 4623 len 40960 data: 4 5 5 5 4 6 6 7 6 6
21716 275/300 Flush
21717 276/300 Read block 39 len 4096
21718 277/300 Flush
21719 278/300 Read block 4161 len 12288
21720 279/300 Read block 2233 len 16384
21721 280/300 Write block 578 len 8192 data: 5 4
21722 281/300 Read block 367 len 32768
21723 282/300 Read block 3437 len 20480
21724 283/300 Read block 579 len 20480
21725 284/300 Flush
21726 285/300 Write block 4377 len 40960 data: 2 3 4 4 5 4 3 4 4 5
21727 286/300 Read block 3214 len 36864
21728 287/300 Write block 2307 len 20480 data: 5 4 4 4 4
21729 288/300 Write block 4464 len 40960 data: 6 6 6 6 7 5 4 4 3 3
21730 289/300 Read block 3275 len 20480
21731 290/300 Read block 547 len 28672
21732 291/300 Write block 1097 len 32768 data: 5 4 4 5 4 5 5 6
21733 292/300 Read block 2715 len 8192
21734 293/300 Read block 17 len 32768
21735 294/300 Write block 2991 len 28672 data: 3 4 4 7 7 6 4
21736 295/300 Read block 1544 len 28672
21737 296/300 Read block 2947 len 12288
21738 297/300 Write block 2730 len 24576 data: 4 4 3 5 6 4
21739 298/300 Read block 1757 len 28672
21740 299/300 Write block 3514 len 12288 data: 3 5 5
21741 300/300 Write block 4774 len 8192 data: 2 2
21742 301/300 Write block 2797 len 4096 data: 8
21743 Main task has work to do, go find it
21744 got dsc Start(2)
21745 start 2
21746 Feb 25 17:23:03.092 INFO request completed, latency_us: 285, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5b94a3a7-fec1-4fa5-920f-76625b0cc54b, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
21747 Replay: started 2, returned:Ok(())
21748 ----------------------------------------------------------------
21749 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21750 States: Active Active Offline
21751 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21752 6737 Acked Write 8 Done Done Sent false
21753 6738 Acked Read 5 Done Done Sent false
21754 6739 Acked Write 8 Done Done Sent false
21755 6740 Acked Read 4 Done Done Sent false
21756 6741 Acked Read 4 Done Done Sent false
21757 6742 Acked Write 9 Done Done Sent false
21758 6743 Acked Write 7 Done Done Sent false
21759 6744 Acked Read 5 Done Done Sent false
21760 6745 Acked Write 7 Done Done Sent false
21761 6746 Acked Read 1 Done Done Sent false
21762 6747 Acked Write 3 Done Done Sent false
21763 6748 Acked Write 2 Done Done Sent false
21764 6749 Acked Read 9 Done Done Sent false
21765 6750 Acked Flush 0 Done Done Sent false
21766 6751 Acked Read 6 Done Done Sent false
21767 6752 Acked Read 4 Done Done Sent false
21768 6753 Acked Read 5 Done Done Sent false
21769 6754 Acked Read 10 Done Done Sent false
21770 6755 Acked Read 2 Done Done Sent false
21771 6756 Acked Read 6 Done Done Sent false
21772 6757 Acked Write 10 Done Done Sent false
21773 6758 Acked Write 1 Done Done Sent false
21774 6759 Acked Read 3 Done Done Sent false
21775 6760 Acked Read 4 Done Done Sent false
21776 6761 Acked Read 3 Done Done Sent false
21777 6762 Acked Write 9 Done Done Sent false
21778 6763 Acked Write 4 Done Done Sent false
21779 6764 Acked Write 9 Done Done Sent false
21780 6765 Acked Flush 0 Done Done Sent false
21781 6766 Acked Flush 0 Done Done Sent false
21782 6767 Acked Read 10 Done Done Sent false
21783 6768 Acked Read 8 Done Done Sent false
21784 6769 Acked Read 1 Done Done Sent false
21785 6770 Acked Read 3 Done Done Sent false
21786 6771 Acked Flush 0 Done Done Sent false
21787 6772 Acked Write 10 Done Done Sent false
21788 6773 Acked Write 4 Done Done Sent false
21789 6774 Acked Write 6 Done Done Sent false
21790 6775 Acked Read 4 Done Done Sent false
21791 6776 Acked Read 6 Done Done Sent false
21792 6777 Acked Write 8 Done Done Sent false
21793 6778 Acked Read 5 Done Done Sent false
21794 6779 Acked Write 4 Done Done Sent false
21795 6780 Acked Write 5 Done Done Sent false
21796 6781 Acked Write 2 Done Done Sent false
21797 6782 Acked Read 1 Done Done Sent false
21798 6783 Acked Write 4 Done Done Sent false
21799 6784 Acked Read 2 Done Done Sent false
21800 6785 Acked Write 10 Done Done Sent false
21801 6786 Acked Read 2 Done Done Sent false
21802 6787 Acked Write 7 Done Done Sent false
21803 6788 Acked Write 5 Done Done Sent false
21804 6789 Acked Write 5 Done Done Sent false
21805 6790 Acked Write 3 Done Done Sent false
21806 6791 Acked Read 2 Done Done Sent false
21807 6792 Acked Read 1 Done Done Sent false
21808 6793 Acked Flush 0 Done Done Sent false
21809 6794 Acked Read 4 Done Done Sent false
21810 6795 Acked Write 3 Done Done Sent false
21811 6796 Acked Write 10 Done Done Sent false
21812 6797 Acked Read 5 Done Done Sent false
21813 6798 Acked Write 7 Done Done Sent false
21814 6799 Acked Write 6 Done Done Sent false
21815 6800 Acked Read 5 Done Done Sent false
21816 6801 Acked Write 8 Done Done Sent false
21817 6802 Acked Read 7 Done Done Sent false
21818 6803 Acked Write 1 Done Done Sent false
21819 6804 Acked Read 4 Done Done Sent false
21820 6805 Acked Read 5 Done Done Sent false
21821 6806 Acked Write 6 Done Done Sent false
21822 6807 Acked Write 4 Done Done Sent false
21823 6808 Acked Read 7 Done Done Sent false
21824 6809 Acked Write 1 Done Done Sent false
21825 6810 Acked Read 3 Done Done Sent false
21826 6811 Acked Read 10 Done Done Sent false
21827 6812 Acked Read 3 Done Done Sent false
21828 6813 Acked Flush 0 Done Done Sent false
21829 6814 Acked Read 10 Done Done Sent false
21830 6815 Acked Write 2 Done Done Sent false
21831 6816 Acked Write 9 Done Done Sent false
21832 6817 Acked Read 1 Done Done Sent false
21833 6818 Acked Write 3 Done Done Sent false
21834 6819 Acked Write 9 Done Done Sent false
21835 6820 Acked Write 9 Done Done Sent false
21836 6821 Acked Write 4 Done Done Sent false
21837 6822 Acked Read 1 Done Done Sent false
21838 6823 Acked Read 3 Done Done Sent false
21839 6824 Acked Read 1 Done Done Sent false
21840 6825 Acked Read 7 Done Done Sent false
21841 6826 Acked Write 4 Done Done Sent false
21842 6827 Acked Write 5 Done Done Sent false
21843 6828 Acked Write 2 Done Done Sent false
21844 6829 Acked Write 8 Done Done Sent false
21845 6830 Acked Write 8 Done Done Sent false
21846 6831 Acked Write 10 Done Done Sent false
21847 6832 Acked Read 10 Done Done Sent false
21848 6833 Acked Write 2 Done Done Sent false
21849 6834 Acked Read 10 Done Done Sent false
21850 6835 Acked Read 1 Done Done Sent false
21851 6836 Acked Read 7 Done Done Sent false
21852 6837 Acked Flush 0 Done Done Sent false
21853 6838 Acked Write 8 Done Done Sent false
21854 6839 Acked Read 10 Done Done Sent false
21855 6840 Acked Read 2 Done Done Sent false
21856 6841 Acked Write 6 Done Done Sent false
21857 6842 Acked Write 6 Done Done Sent false
21858 6843 Acked Read 10 Done Done Sent false
21859 6844 Acked Write 6 Done Done Sent false
21860 6845 Acked Write 1 Done Done Sent false
21861 6846 Acked Read 4 Done Done Sent false
21862 6847 Acked Write 2 Done Done Sent false
21863 6848 Acked Flush 0 Done Done Sent false
21864 6849 Acked Read 9 Done Done Sent false
21865 6850 Acked Read 8 Done Done Sent false
21866 6851 Acked Read 7 Done Done Sent false
21867 6852 Acked Flush 0 Done Done Sent false
21868 6853 Acked Read 3 Done Done Sent false
21869 6854 Acked Write 9 Done Done Sent false
21870 6855 Acked Flush 0 Done Done Sent false
21871 6856 Acked Read 8 Done Done Sent false
21872 6857 Acked Read 10 Done Done Sent false
21873 6858 Acked Write 2 Done Done Sent false
21874 6859 Acked Write 4 Done Done Sent false
21875 6860 Acked Write 5 Done Done Sent false
21876 6861 Acked Write 4 Done Done Sent false
21877 6862 Acked Read 5 Done Done Sent false
21878 6863 Acked Write 8 Done Done Sent false
21879 6864 Acked Write 8 Done Done Sent false
21880 6865 Acked Flush 0 Done Done Sent false
21881 6866 Acked Read 5 Done Done Sent false
21882 6867 Acked Flush 0 Done Done Sent false
21883 6868 Acked Read 1 Done Done Sent false
21884 6869 Acked Read 3 Done Done Sent false
21885 6870 Acked Write 2 Done Done Sent false
21886 6871 Acked Write 9 Done Done Sent false
21887 6872 Acked Read 1 Done Done Sent false
21888 6873 Acked Write 2 Done Done Sent false
21889 6874 Acked Write 7 Done Done Sent false
21890 6875 Acked Read 6 Done Done Sent false
21891 6876 Acked Read 5 Done Done Sent false
21892 6877 Acked Read 6 Done Done Sent false
21893 6878 Acked Flush 0 Done Done Sent false
21894 6879 Acked Read 1 Done Done Sent false
21895 6880 Acked Read 1 Done Done Sent false
21896 6881 Acked Read 10 Done Done Sent false
21897 6882 Acked Write 2 Done Done Sent false
21898 6883 Acked Read 5 Done Done Sent false
21899 6884 Acked Write 9 Done Done Sent false
21900 6885 Acked Flush 0 Done Done Sent false
21901 6886 Acked Read 4 Done Done Sent false
21902 6887 Acked Write 2 Done Done Sent false
21903 6888 Acked Read 4 Done Done Sent false
21904 6889 Acked Flush 0 Done Done Sent false
21905 6890 Acked Read 1 Done Done Sent false
21906 6891 Acked Read 1 Done Done Sent false
21907 6892 Acked Read 2 Done Done Sent false
21908 6893 Acked Read 9 Done Done Sent false
21909 6894 Acked Write 3 Done Done Sent false
21910 6895 Acked Read 8 Done Done Sent false
21911 6896 Acked Write 9 Done Done Sent false
21912 6897 Acked Read 4 Done Done Sent false
21913 6898 Acked Read 5 Done Done Sent false
21914 6899 Acked Write 8 Done Done Sent false
21915 6900 Acked Write 5 Done Done Sent false
21916 6901 Acked Write 9 Done Done Sent false
21917 6902 Acked Write 10 Done Done Sent false
21918 6903 Acked Write 1 Done Done Sent false
21919 6904 Acked Read 10 Done Done Sent false
21920 6905 Acked Write 3 Done Done Sent false
21921 6906 Acked Read 9 Done Done Sent false
21922 6907 Acked Read 1 Done Done Sent false
21923 6908 Acked Flush 0 Done Done Sent false
21924 6909 Acked Flush 0 Done Done Sent false
21925 6910 Acked Write 8 Done Done Sent false
21926 6911 Acked Write 8 Done Done Sent false
21927 6912 Acked Flush 0 Done Done Sent false
21928 6913 Acked Write 6 Done Done Sent false
21929 6914 Acked Write 6 Done Done Sent false
21930 6915 Acked Write 1 Done Done Sent false
21931 6916 Acked Read 6 Done Done Sent false
21932 6917 Acked Write 10 Done Done Sent false
21933 6918 Acked Write 1 Done Done Sent false
21934 6919 Acked Read 3 Done Done Sent false
21935 6920 Acked Read 2 Done Done Sent false
21936 6921 Acked Read 4 Done Done Sent false
21937 6922 Acked Read 10 Done Done Sent false
21938 6923 Acked Read 3 Done Done Sent false
21939 6924 Acked Read 8 Done Done Sent false
21940 6925 Acked Read 1 Done Done Sent false
21941 6926 Acked Read 6 Done Done Sent false
21942 6927 Acked Flush 0 Done Done Sent false
21943 6928 Acked Write 1 Done Done Sent false
21944 6929 Acked Write 10 Done Done Sent false
21945 6930 Acked Write 9 Done Done Sent false
21946 6931 Acked Write 8 Done Done Sent false
21947 6932 Acked Write 2 Done Done Sent false
21948 6933 Acked Write 9 Done Done Sent false
21949 6934 Acked Read 4 Done Done Sent false
21950 6935 Acked Read 8 Done Done Sent false
21951 6936 Acked Write 7 Done Done Sent false
21952 6937 Acked Write 7 Done Done Sent false
21953 6938 Acked Read 7 Done Done Sent false
21954 6939 Acked Flush 0 Done Done Sent false
21955 6940 Acked Flush 0 Done Done Sent false
21956 6941 Acked Write 5 Done Done Sent false
21957 6942 Acked Read 1 Done Done Sent false
21958 6943 Acked Read 1 Done Done Sent false
21959 6944 Acked Read 3 Done Done Sent false
21960 6945 Acked Read 3 Done Done Sent false
21961 6946 Acked Read 8 Done Done Sent false
21962 6947 Acked Flush 0 Done Done Sent false
21963 6948 Acked Write 10 Done Done Sent false
21964 6949 Acked Write 4 Done Done Sent false
21965 6950 Acked Read 4 Done Done Sent false
21966 6951 Acked Read 5 Done Done Sent false
21967 6952 Acked Write 10 Done Done Sent false
21968 6953 Acked Read 3 Done Done Sent false
21969 6954 Acked Read 3 Done Done Sent false
21970 6955 Acked Write 8 Done Done Sent false
21971 6956 Acked Write 6 Done Done Sent false
21972 6957 Acked Write 4 Done Done Sent false
21973 6958 Acked Write 4 Done Done Sent false
21974 6959 Acked Read 6 Done Done Sent false
21975 6960 Acked Write 7 Done Done Sent false
21976 6961 Acked Write 5 Done Done Sent false
21977 6962 Acked Flush 0 Done Done Sent false
21978 6963 Acked Read 6 Done Done Sent false
21979 6964 Acked Read 3 Done Done Sent false
21980 6965 Acked Write 1 Done Done Sent false
21981 6966 Acked Read 6 Done Done Sent false
21982 6967 Acked Write 8 Done Done Sent false
21983 6968 Acked Write 4 Done Done Sent false
21984 6969 Acked Write 1 Done Done Sent false
21985 6970 Acked Write 6 Done Done Sent false
21986 6971 Acked Read 3 Done Done Sent false
21987 6972 Acked Read 7 Done Done Sent false
21988 6973 Acked Flush 0 Done Done Sent false
21989 6974 Acked Write 7 Done Done Sent false
21990 6975 Acked Write 3 Done Done Sent false
21991 6976 Acked Read 3 Done Done Sent false
21992 6977 Acked Read 1 Done Done Sent false
21993 6978 Acked Write 10 Done Done Sent false
21994 6979 Acked Read 4 Done Done Sent false
21995 6980 Acked Write 2 Done Done Sent false
21996 6981 Acked Read 10 Done Done Sent false
21997 6982 Acked Read 5 Done Done Sent false
21998 6983 Acked Write 3 Done Done Sent false
21999 6984 Acked Read 4 Done Done Sent false
22000 6985 Acked Read 9 Done Done Sent false
22001 6986 Acked Read 3 Done Done Sent false
22002 6987 Acked Flush 0 Done Done Sent false
22003 6988 Acked Read 5 Done Done Sent false
22004 6989 Acked Write 1 Done Done Sent false
22005 6990 Acked Read 3 Done Done Sent false
22006 6991 Acked Read 3 Done Done Sent false
22007 6992 Acked Write 3 Done Done Sent false
22008 6993 Acked Write 5 Done Done Sent false
22009 6994 Acked Read 9 Done Done Sent false
22010 6995 Acked Write 6 Done Done Sent false
22011 6996 Acked Write 3 Done Done Sent false
22012 6997 Acked Read 9 Done Done Sent false
22013 6998 Acked Read 4 Done Done Sent false
22014 6999 Acked Read 1 Done Done Sent false
22015 7000 Acked Write 4 Done Done Sent false
22016 7001 Acked Write 5 Done Done Sent false
22017 7002 Acked Write 4 Done Done Sent false
22018 7003 Acked Read 6 Done Done Sent false
22019 7004 Acked Read 4 Done Done Sent false
22020 7005 Acked Read 9 Done Done Sent false
22021 7006 Acked Write 8 Done Done Sent false
22022 7007 Acked Write 5 Done Done Sent false
22023 7008 Acked Write 8 Done Done Sent false
22024 7009 Acked Read 8 Done Done Sent false
22025 7010 Acked Write 10 Done Done Sent false
22026 7011 Acked Flush 0 Done Done Sent false
22027 7012 Acked Read 1 Done Done Sent false
22028 7013 Acked Flush 0 Done Done Sent false
22029 7014 Acked Read 3 Done Done Sent false
22030 7015 Acked Read 4 Done Done Sent false
22031 7016 Acked Write 2 Done Done Sent false
22032 7017 Acked Read 8 Done Done Sent false
22033 7018 Acked Read 5 Done Done Sent false
22034 7019 Acked Read 5 Done Done Sent false
22035 7020 Acked Flush 0 Done Done Sent false
22036 7021 Acked Write 10 Done Done Sent false
22037 7022 Acked Read 9 Done Done Sent false
22038 7023 Acked Write 5 Done Done Sent false
22039 7024 Acked Write 10 Done Done Sent false
22040 7025 Acked Read 5 Done Done Sent false
22041 7026 Acked Read 7 Done Done Sent false
22042 7027 Acked Write 8 Done Done Sent false
22043 7028 Acked Read 2 Done Done Sent false
22044 7029 Acked Read 8 Done Done Sent false
22045 7030 Acked Write 7 Done Done Sent false
22046 7031 Acked Read 7 Done Done Sent false
22047 7032 Acked Read 3 Done Done Sent false
22048 7033 Acked Write 6 Done Done Sent false
22049 7034 Acked Read 7 Done Done Sent false
22050 7035 Acked Write 3 Done Done Sent false
22051 7036 Acked Write 2 Done Done Sent false
22052 7037 Acked Write 1 Done Done Sent false
22053 STATES DS:0 DS:1 DS:2 TOTAL
22054 Sent 0 0 301 301
22055 Done 301 301 0 602
22056 Skipped 0 0 0 0
22057 Error 0 0 0 0
22058 Last Flush: Some(JobId(7020)) Some(JobId(7020)) Some(JobId(6736))
22059 Downstairs last five completed: 6736 6735 6734 6733 6732
22060 Upstairs last five completed: 7037 7036 7035 7034 7033
22061 CLIENT: Up:0 ds:301 act:2
220622025-02-25T17:23:03.736ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
220632025-02-25T17:23:04.737ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
220642025-02-25T17:23:05.739ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
22065 ----------------------------------------------------------------
22066 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22067 States: Active Active Offline
22068 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22069 6737 Acked Write 8 Done Done Sent false
22070 6738 Acked Read 5 Done Done Sent false
22071 6739 Acked Write 8 Done Done Sent false
22072 6740 Acked Read 4 Done Done Sent false
22073 6741 Acked Read 4 Done Done Sent false
22074 6742 Acked Write 9 Done Done Sent false
22075 6743 Acked Write 7 Done Done Sent false
22076 6744 Acked Read 5 Done Done Sent false
22077 6745 Acked Write 7 Done Done Sent false
22078 6746 Acked Read 1 Done Done Sent false
22079 6747 Acked Write 3 Done Done Sent false
22080 6748 Acked Write 2 Done Done Sent false
22081 6749 Acked Read 9 Done Done Sent false
22082 6750 Acked Flush 0 Done Done Sent false
22083 6751 Acked Read 6 Done Done Sent false
22084 6752 Acked Read 4 Done Done Sent false
22085 6753 Acked Read 5 Done Done Sent false
22086 6754 Acked Read 10 Done Done Sent false
22087 6755 Acked Read 2 Done Done Sent false
22088 6756 Acked Read 6 Done Done Sent false
22089 6757 Acked Write 10 Done Done Sent false
22090 6758 Acked Write 1 Done Done Sent false
22091 6759 Acked Read 3 Done Done Sent false
22092 6760 Acked Read 4 Done Done Sent false
22093 6761 Acked Read 3 Done Done Sent false
22094 6762 Acked Write 9 Done Done Sent false
22095 6763 Acked Write 4 Done Done Sent false
22096 6764 Acked Write 9 Done Done Sent false
22097 6765 Acked Flush 0 Done Done Sent false
22098 6766 Acked Flush 0 Done Done Sent false
22099 6767 Acked Read 10 Done Done Sent false
22100 6768 Acked Read 8 Done Done Sent false
22101 6769 Acked Read 1 Done Done Sent false
22102 6770 Acked Read 3 Done Done Sent false
22103 6771 Acked Flush 0 Done Done Sent false
22104 6772 Acked Write 10 Done Done Sent false
22105 6773 Acked Write 4 Done Done Sent false
22106 6774 Acked Write 6 Done Done Sent false
22107 6775 Acked Read 4 Done Done Sent false
22108 6776 Acked Read 6 Done Done Sent false
22109 6777 Acked Write 8 Done Done Sent false
22110 6778 Acked Read 5 Done Done Sent false
22111 6779 Acked Write 4 Done Done Sent false
22112 6780 Acked Write 5 Done Done Sent false
22113 6781 Acked Write 2 Done Done Sent false
22114 6782 Acked Read 1 Done Done Sent false
22115 6783 Acked Write 4 Done Done Sent false
22116 6784 Acked Read 2 Done Done Sent false
22117 6785 Acked Write 10 Done Done Sent false
22118 6786 Acked Read 2 Done Done Sent false
22119 6787 Acked Write 7 Done Done Sent false
22120 6788 Acked Write 5 Done Done Sent false
22121 6789 Acked Write 5 Done Done Sent false
22122 6790 Acked Write 3 Done Done Sent false
22123 6791 Acked Read 2 Done Done Sent false
22124 6792 Acked Read 1 Done Done Sent false
22125 6793 Acked Flush 0 Done Done Sent false
22126 6794 Acked Read 4 Done Done Sent false
22127 6795 Acked Write 3 Done Done Sent false
22128 6796 Acked Write 10 Done Done Sent false
22129 6797 Acked Read 5 Done Done Sent false
22130 6798 Acked Write 7 Done Done Sent false
22131 6799 Acked Write 6 Done Done Sent false
22132 6800 Acked Read 5 Done Done Sent false
22133 6801 Acked Write 8 Done Done Sent false
22134 6802 Acked Read 7 Done Done Sent false
22135 6803 Acked Write 1 Done Done Sent false
22136 6804 Acked Read 4 Done Done Sent false
22137 6805 Acked Read 5 Done Done Sent false
22138 6806 Acked Write 6 Done Done Sent false
22139 6807 Acked Write 4 Done Done Sent false
22140 6808 Acked Read 7 Done Done Sent false
22141 6809 Acked Write 1 Done Done Sent false
22142 6810 Acked Read 3 Done Done Sent false
22143 6811 Acked Read 10 Done Done Sent false
22144 6812 Acked Read 3 Done Done Sent false
22145 6813 Acked Flush 0 Done Done Sent false
22146 6814 Acked Read 10 Done Done Sent false
22147 6815 Acked Write 2 Done Done Sent false
22148 6816 Acked Write 9 Done Done Sent false
22149 6817 Acked Read 1 Done Done Sent false
22150 6818 Acked Write 3 Done Done Sent false
22151 6819 Acked Write 9 Done Done Sent false
22152 6820 Acked Write 9 Done Done Sent false
22153 6821 Acked Write 4 Done Done Sent false
22154 6822 Acked Read 1 Done Done Sent false
22155 6823 Acked Read 3 Done Done Sent false
22156 6824 Acked Read 1 Done Done Sent false
22157 6825 Acked Read 7 Done Done Sent false
22158 6826 Acked Write 4 Done Done Sent false
22159 6827 Acked Write 5 Done Done Sent false
22160 6828 Acked Write 2 Done Done Sent false
22161 6829 Acked Write 8 Done Done Sent false
22162 6830 Acked Write 8 Done Done Sent false
22163 6831 Acked Write 10 Done Done Sent false
22164 6832 Acked Read 10 Done Done Sent false
22165 6833 Acked Write 2 Done Done Sent false
22166 6834 Acked Read 10 Done Done Sent false
22167 6835 Acked Read 1 Done Done Sent false
22168 6836 Acked Read 7 Done Done Sent false
22169 6837 Acked Flush 0 Done Done Sent false
22170 6838 Acked Write 8 Done Done Sent false
22171 6839 Acked Read 10 Done Done Sent false
22172 6840 Acked Read 2 Done Done Sent false
22173 6841 Acked Write 6 Done Done Sent false
22174 6842 Acked Write 6 Done Done Sent false
22175 6843 Acked Read 10 Done Done Sent false
22176 6844 Acked Write 6 Done Done Sent false
22177 6845 Acked Write 1 Done Done Sent false
22178 6846 Acked Read 4 Done Done Sent false
22179 6847 Acked Write 2 Done Done Sent false
22180 6848 Acked Flush 0 Done Done Sent false
22181 6849 Acked Read 9 Done Done Sent false
22182 6850 Acked Read 8 Done Done Sent false
22183 6851 Acked Read 7 Done Done Sent false
22184 6852 Acked Flush 0 Done Done Sent false
22185 6853 Acked Read 3 Done Done Sent false
22186 6854 Acked Write 9 Done Done Sent false
22187 6855 Acked Flush 0 Done Done Sent false
22188 6856 Acked Read 8 Done Done Sent false
22189 6857 Acked Read 10 Done Done Sent false
22190 6858 Acked Write 2 Done Done Sent false
22191 6859 Acked Write 4 Done Done Sent false
22192 6860 Acked Write 5 Done Done Sent false
22193 6861 Acked Write 4 Done Done Sent false
22194 6862 Acked Read 5 Done Done Sent false
22195 6863 Acked Write 8 Done Done Sent false
22196 6864 Acked Write 8 Done Done Sent false
22197 6865 Acked Flush 0 Done Done Sent false
22198 6866 Acked Read 5 Done Done Sent false
22199 6867 Acked Flush 0 Done Done Sent false
22200 6868 Acked Read 1 Done Done Sent false
22201 6869 Acked Read 3 Done Done Sent false
22202 6870 Acked Write 2 Done Done Sent false
22203 6871 Acked Write 9 Done Done Sent false
22204 6872 Acked Read 1 Done Done Sent false
22205 6873 Acked Write 2 Done Done Sent false
22206 6874 Acked Write 7 Done Done Sent false
22207 6875 Acked Read 6 Done Done Sent false
22208 6876 Acked Read 5 Done Done Sent false
22209 6877 Acked Read 6 Done Done Sent false
22210 6878 Acked Flush 0 Done Done Sent false
22211 6879 Acked Read 1 Done Done Sent false
22212 6880 Acked Read 1 Done Done Sent false
22213 6881 Acked Read 10 Done Done Sent false
22214 6882 Acked Write 2 Done Done Sent false
22215 6883 Acked Read 5 Done Done Sent false
22216 6884 Acked Write 9 Done Done Sent false
22217 6885 Acked Flush 0 Done Done Sent false
22218 6886 Acked Read 4 Done Done Sent false
22219 6887 Acked Write 2 Done Done Sent false
22220 6888 Acked Read 4 Done Done Sent false
22221 6889 Acked Flush 0 Done Done Sent false
22222 6890 Acked Read 1 Done Done Sent false
22223 6891 Acked Read 1 Done Done Sent false
22224 6892 Acked Read 2 Done Done Sent false
22225 6893 Acked Read 9 Done Done Sent false
22226 6894 Acked Write 3 Done Done Sent false
22227 6895 Acked Read 8 Done Done Sent false
22228 6896 Acked Write 9 Done Done Sent false
22229 6897 Acked Read 4 Done Done Sent false
22230 6898 Acked Read 5 Done Done Sent false
22231 6899 Acked Write 8 Done Done Sent false
22232 6900 Acked Write 5 Done Done Sent false
22233 6901 Acked Write 9 Done Done Sent false
22234 6902 Acked Write 10 Done Done Sent false
22235 6903 Acked Write 1 Done Done Sent false
22236 6904 Acked Read 10 Done Done Sent false
22237 6905 Acked Write 3 Done Done Sent false
22238 6906 Acked Read 9 Done Done Sent false
22239 6907 Acked Read 1 Done Done Sent false
22240 6908 Acked Flush 0 Done Done Sent false
22241 6909 Acked Flush 0 Done Done Sent false
22242 6910 Acked Write 8 Done Done Sent false
22243 6911 Acked Write 8 Done Done Sent false
22244 6912 Acked Flush 0 Done Done Sent false
22245 6913 Acked Write 6 Done Done Sent false
22246 6914 Acked Write 6 Done Done Sent false
22247 6915 Acked Write 1 Done Done Sent false
22248 6916 Acked Read 6 Done Done Sent false
22249 6917 Acked Write 10 Done Done Sent false
22250 6918 Acked Write 1 Done Done Sent false
22251 6919 Acked Read 3 Done Done Sent false
22252 6920 Acked Read 2 Done Done Sent false
22253 6921 Acked Read 4 Done Done Sent false
22254 6922 Acked Read 10 Done Done Sent false
22255 6923 Acked Read 3 Done Done Sent false
22256 6924 Acked Read 8 Done Done Sent false
22257 6925 Acked Read 1 Done Done Sent false
22258 6926 Acked Read 6 Done Done Sent false
22259 6927 Acked Flush 0 Done Done Sent false
22260 6928 Acked Write 1 Done Done Sent false
22261 6929 Acked Write 10 Done Done Sent false
22262 6930 Acked Write 9 Done Done Sent false
22263 6931 Acked Write 8 Done Done Sent false
22264 6932 Acked Write 2 Done Done Sent false
22265 6933 Acked Write 9 Done Done Sent false
22266 6934 Acked Read 4 Done Done Sent false
22267 6935 Acked Read 8 Done Done Sent false
22268 6936 Acked Write 7 Done Done Sent false
22269 6937 Acked Write 7 Done Done Sent false
22270 6938 Acked Read 7 Done Done Sent false
22271 6939 Acked Flush 0 Done Done Sent false
22272 6940 Acked Flush 0 Done Done Sent false
22273 6941 Acked Write 5 Done Done Sent false
22274 6942 Acked Read 1 Done Done Sent false
22275 6943 Acked Read 1 Done Done Sent false
22276 6944 Acked Read 3 Done Done Sent false
22277 6945 Acked Read 3 Done Done Sent false
22278 6946 Acked Read 8 Done Done Sent false
22279 6947 Acked Flush 0 Done Done Sent false
22280 6948 Acked Write 10 Done Done Sent false
22281 6949 Acked Write 4 Done Done Sent false
22282 6950 Acked Read 4 Done Done Sent false
22283 6951 Acked Read 5 Done Done Sent false
22284 6952 Acked Write 10 Done Done Sent false
22285 6953 Acked Read 3 Done Done Sent false
22286 6954 Acked Read 3 Done Done Sent false
22287 6955 Acked Write 8 Done Done Sent false
22288 6956 Acked Write 6 Done Done Sent false
22289 6957 Acked Write 4 Done Done Sent false
22290 6958 Acked Write 4 Done Done Sent false
22291 6959 Acked Read 6 Done Done Sent false
22292 6960 Acked Write 7 Done Done Sent false
22293 6961 Acked Write 5 Done Done Sent false
22294 6962 Acked Flush 0 Done Done Sent false
22295 6963 Acked Read 6 Done Done Sent false
22296 6964 Acked Read 3 Done Done Sent false
22297 6965 Acked Write 1 Done Done Sent false
22298 6966 Acked Read 6 Done Done Sent false
22299 6967 Acked Write 8 Done Done Sent false
22300 6968 Acked Write 4 Done Done Sent false
22301 6969 Acked Write 1 Done Done Sent false
22302 6970 Acked Write 6 Done Done Sent false
22303 6971 Acked Read 3 Done Done Sent false
22304 6972 Acked Read 7 Done Done Sent false
22305 6973 Acked Flush 0 Done Done Sent false
22306 6974 Acked Write 7 Done Done Sent false
22307 6975 Acked Write 3 Done Done Sent false
22308 6976 Acked Read 3 Done Done Sent false
22309 6977 Acked Read 1 Done Done Sent false
22310 6978 Acked Write 10 Done Done Sent false
22311 6979 Acked Read 4 Done Done Sent false
22312 6980 Acked Write 2 Done Done Sent false
22313 6981 Acked Read 10 Done Done Sent false
22314 6982 Acked Read 5 Done Done Sent false
22315 6983 Acked Write 3 Done Done Sent false
22316 6984 Acked Read 4 Done Done Sent false
22317 6985 Acked Read 9 Done Done Sent false
22318 6986 Acked Read 3 Done Done Sent false
22319 6987 Acked Flush 0 Done Done Sent false
22320 6988 Acked Read 5 Done Done Sent false
22321 6989 Acked Write 1 Done Done Sent false
22322 6990 Acked Read 3 Done Done Sent false
22323 6991 Acked Read 3 Done Done Sent false
22324 6992 Acked Write 3 Done Done Sent false
22325 6993 Acked Write 5 Done Done Sent false
22326 6994 Acked Read 9 Done Done Sent false
22327 6995 Acked Write 6 Done Done Sent false
22328 6996 Acked Write 3 Done Done Sent false
22329 6997 Acked Read 9 Done Done Sent false
22330 6998 Acked Read 4 Done Done Sent false
22331 6999 Acked Read 1 Done Done Sent false
22332 7000 Acked Write 4 Done Done Sent false
22333 7001 Acked Write 5 Done Done Sent false
22334 7002 Acked Write 4 Done Done Sent false
22335 7003 Acked Read 6 Done Done Sent false
22336 7004 Acked Read 4 Done Done Sent false
22337 7005 Acked Read 9 Done Done Sent false
22338 7006 Acked Write 8 Done Done Sent false
22339 7007 Acked Write 5 Done Done Sent false
22340 7008 Acked Write 8 Done Done Sent false
22341 7009 Acked Read 8 Done Done Sent false
22342 7010 Acked Write 10 Done Done Sent false
22343 7011 Acked Flush 0 Done Done Sent false
22344 7012 Acked Read 1 Done Done Sent false
22345 7013 Acked Flush 0 Done Done Sent false
22346 7014 Acked Read 3 Done Done Sent false
22347 7015 Acked Read 4 Done Done Sent false
22348 7016 Acked Write 2 Done Done Sent false
22349 7017 Acked Read 8 Done Done Sent false
22350 7018 Acked Read 5 Done Done Sent false
22351 7019 Acked Read 5 Done Done Sent false
22352 7020 Acked Flush 0 Done Done Sent false
22353 7021 Acked Write 10 Done Done Sent false
22354 7022 Acked Read 9 Done Done Sent false
22355 7023 Acked Write 5 Done Done Sent false
22356 7024 Acked Write 10 Done Done Sent false
22357 7025 Acked Read 5 Done Done Sent false
22358 7026 Acked Read 7 Done Done Sent false
22359 7027 Acked Write 8 Done Done Sent false
22360 7028 Acked Read 2 Done Done Sent false
22361 7029 Acked Read 8 Done Done Sent false
22362 7030 Acked Write 7 Done Done Sent false
22363 7031 Acked Read 7 Done Done Sent false
22364 7032 Acked Read 3 Done Done Sent false
22365 7033 Acked Write 6 Done Done Sent false
22366 7034 Acked Read 7 Done Done Sent false
22367 7035 Acked Write 3 Done Done Sent false
22368 7036 Acked Write 2 Done Done Sent false
22369 7037 Acked Write 1 Done Done Sent false
22370 STATES DS:0 DS:1 DS:2 TOTAL
22371 Sent 0 0 301 301
22372 Done 301 301 0 602
22373 Skipped 0 0 0 0
22374 Error 0 0 0 0
22375 Last Flush: Some(JobId(7020)) Some(JobId(7020)) Some(JobId(6736))
22376 Downstairs last five completed: 6736 6735 6734 6733 6732
22377 Upstairs last five completed: 7037 7036 7035 7034 7033
22378 CLIENT: Up:0 ds:301 act:2
223792025-02-25T17:23:07.740ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
223802025-02-25T17:23:08.741ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
223812025-02-25T17:23:08.853ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
223822025-02-25T17:23:08.853ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
223832025-02-25T17:23:08.853ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
223842025-02-25T17:23:08.853ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
223852025-02-25T17:23:08.853ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
223862025-02-25T17:23:09.743ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
223872025-02-25T17:23:10.745ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
22388 [8830] got start action
22389 [8830] I am going to restart
22390 Starting downstairs at port 8830
22391 Make output file at "/tmp/dsc/downstairs-8830.txt"
22392 [8830][2] reports Starting
22393 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1228)
22394 [8830][2] reports Running
22395 ----------------------------------------------------------------
22396 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22397 States: Active Active Offline
22398 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22399 6737 Acked Write 8 Done Done Sent false
22400 6738 Acked Read 5 Done Done Sent false
22401 6739 Acked Write 8 Done Done Sent false
22402 6740 Acked Read 4 Done Done Sent false
22403 6741 Acked Read 4 Done Done Sent false
22404 6742 Acked Write 9 Done Done Sent false
22405 6743 Acked Write 7 Done Done Sent false
22406 6744 Acked Read 5 Done Done Sent false
22407 6745 Acked Write 7 Done Done Sent false
22408 6746 Acked Read 1 Done Done Sent false
22409 6747 Acked Write 3 Done Done Sent false
22410 6748 Acked Write 2 Done Done Sent false
22411 6749 Acked Read 9 Done Done Sent false
22412 6750 Acked Flush 0 Done Done Sent false
22413 6751 Acked Read 6 Done Done Sent false
22414 6752 Acked Read 4 Done Done Sent false
22415 6753 Acked Read 5 Done Done Sent false
22416 6754 Acked Read 10 Done Done Sent false
22417 6755 Acked Read 2 Done Done Sent false
22418 6756 Acked Read 6 Done Done Sent false
22419 6757 Acked Write 10 Done Done Sent false
22420 6758 Acked Write 1 Done Done Sent false
22421 6759 Acked Read 3 Done Done Sent false
22422 6760 Acked Read 4 Done Done Sent false
22423 6761 Acked Read 3 Done Done Sent false
22424 6762 Acked Write 9 Done Done Sent false
22425 6763 Acked Write 4 Done Done Sent false
22426 6764 Acked Write 9 Done Done Sent false
22427 6765 Acked Flush 0 Done Done Sent false
22428 6766 Acked Flush 0 Done Done Sent false
22429 6767 Acked Read 10 Done Done Sent false
22430 6768 Acked Read 8 Done Done Sent false
22431 6769 Acked Read 1 Done Done Sent false
22432 6770 Acked Read 3 Done Done Sent false
22433 6771 Acked Flush 0 Done Done Sent false
22434 6772 Acked Write 10 Done Done Sent false
22435 6773 Acked Write 4 Done Done Sent false
22436 6774 Acked Write 6 Done Done Sent false
22437 6775 Acked Read 4 Done Done Sent false
22438 6776 Acked Read 6 Done Done Sent false
22439 6777 Acked Write 8 Done Done Sent false
22440 6778 Acked Read 5 Done Done Sent false
22441 6779 Acked Write 4 Done Done Sent false
22442 6780 Acked Write 5 Done Done Sent false
22443 6781 Acked Write 2 Done Done Sent false
22444 6782 Acked Read 1 Done Done Sent false
22445 6783 Acked Write 4 Done Done Sent false
22446 6784 Acked Read 2 Done Done Sent false
22447 6785 Acked Write 10 Done Done Sent false
22448 6786 Acked Read 2 Done Done Sent false
22449 6787 Acked Write 7 Done Done Sent false
22450 6788 Acked Write 5 Done Done Sent false
22451 6789 Acked Write 5 Done Done Sent false
22452 6790 Acked Write 3 Done Done Sent false
22453 6791 Acked Read 2 Done Done Sent false
22454 6792 Acked Read 1 Done Done Sent false
22455 6793 Acked Flush 0 Done Done Sent false
22456 6794 Acked Read 4 Done Done Sent false
22457 6795 Acked Write 3 Done Done Sent false
22458 6796 Acked Write 10 Done Done Sent false
22459 6797 Acked Read 5 Done Done Sent false
22460 6798 Acked Write 7 Done Done Sent false
22461 6799 Acked Write 6 Done Done Sent false
22462 6800 Acked Read 5 Done Done Sent false
22463 6801 Acked Write 8 Done Done Sent false
22464 6802 Acked Read 7 Done Done Sent false
22465 6803 Acked Write 1 Done Done Sent false
22466 6804 Acked Read 4 Done Done Sent false
22467 6805 Acked Read 5 Done Done Sent false
22468 6806 Acked Write 6 Done Done Sent false
22469 6807 Acked Write 4 Done Done Sent false
22470 6808 Acked Read 7 Done Done Sent false
22471 6809 Acked Write 1 Done Done Sent false
22472 6810 Acked Read 3 Done Done Sent false
22473 6811 Acked Read 10 Done Done Sent false
22474 6812 Acked Read 3 Done Done Sent false
22475 6813 Acked Flush 0 Done Done Sent false
22476 6814 Acked Read 10 Done Done Sent false
22477 6815 Acked Write 2 Done Done Sent false
22478 6816 Acked Write 9 Done Done Sent false
22479 6817 Acked Read 1 Done Done Sent false
22480 6818 Acked Write 3 Done Done Sent false
22481 6819 Acked Write 9 Done Done Sent false
22482 6820 Acked Write 9 Done Done Sent false
22483 6821 Acked Write 4 Done Done Sent false
22484 6822 Acked Read 1 Done Done Sent false
22485 6823 Acked Read 3 Done Done Sent false
22486 6824 Acked Read 1 Done Done Sent false
22487 6825 Acked Read 7 Done Done Sent false
22488 6826 Acked Write 4 Done Done Sent false
22489 6827 Acked Write 5 Done Done Sent false
22490 6828 Acked Write 2 Done Done Sent false
22491 6829 Acked Write 8 Done Done Sent false
22492 6830 Acked Write 8 Done Done Sent false
22493 6831 Acked Write 10 Done Done Sent false
22494 6832 Acked Read 10 Done Done Sent false
22495 6833 Acked Write 2 Done Done Sent false
22496 6834 Acked Read 10 Done Done Sent false
22497 6835 Acked Read 1 Done Done Sent false
22498 6836 Acked Read 7 Done Done Sent false
22499 6837 Acked Flush 0 Done Done Sent false
22500 6838 Acked Write 8 Done Done Sent false
22501 6839 Acked Read 10 Done Done Sent false
22502 6840 Acked Read 2 Done Done Sent false
22503 6841 Acked Write 6 Done Done Sent false
22504 6842 Acked Write 6 Done Done Sent false
22505 6843 Acked Read 10 Done Done Sent false
22506 6844 Acked Write 6 Done Done Sent false
22507 6845 Acked Write 1 Done Done Sent false
22508 6846 Acked Read 4 Done Done Sent false
22509 6847 Acked Write 2 Done Done Sent false
22510 6848 Acked Flush 0 Done Done Sent false
22511 6849 Acked Read 9 Done Done Sent false
22512 6850 Acked Read 8 Done Done Sent false
22513 6851 Acked Read 7 Done Done Sent false
22514 6852 Acked Flush 0 Done Done Sent false
22515 6853 Acked Read 3 Done Done Sent false
22516 6854 Acked Write 9 Done Done Sent false
22517 6855 Acked Flush 0 Done Done Sent false
22518 6856 Acked Read 8 Done Done Sent false
22519 6857 Acked Read 10 Done Done Sent false
22520 6858 Acked Write 2 Done Done Sent false
22521 6859 Acked Write 4 Done Done Sent false
22522 6860 Acked Write 5 Done Done Sent false
22523 6861 Acked Write 4 Done Done Sent false
22524 6862 Acked Read 5 Done Done Sent false
22525 6863 Acked Write 8 Done Done Sent false
22526 6864 Acked Write 8 Done Done Sent false
22527 6865 Acked Flush 0 Done Done Sent false
22528 6866 Acked Read 5 Done Done Sent false
22529 6867 Acked Flush 0 Done Done Sent false
22530 6868 Acked Read 1 Done Done Sent false
22531 6869 Acked Read 3 Done Done Sent false
22532 6870 Acked Write 2 Done Done Sent false
22533 6871 Acked Write 9 Done Done Sent false
22534 6872 Acked Read 1 Done Done Sent false
22535 6873 Acked Write 2 Done Done Sent false
22536 6874 Acked Write 7 Done Done Sent false
22537 6875 Acked Read 6 Done Done Sent false
22538 6876 Acked Read 5 Done Done Sent false
22539 6877 Acked Read 6 Done Done Sent false
22540 6878 Acked Flush 0 Done Done Sent false
22541 6879 Acked Read 1 Done Done Sent false
22542 6880 Acked Read 1 Done Done Sent false
22543 6881 Acked Read 10 Done Done Sent false
22544 6882 Acked Write 2 Done Done Sent false
22545 6883 Acked Read 5 Done Done Sent false
22546 6884 Acked Write 9 Done Done Sent false
22547 6885 Acked Flush 0 Done Done Sent false
22548 6886 Acked Read 4 Done Done Sent false
22549 6887 Acked Write 2 Done Done Sent false
22550 6888 Acked Read 4 Done Done Sent false
22551 6889 Acked Flush 0 Done Done Sent false
22552 6890 Acked Read 1 Done Done Sent false
22553 6891 Acked Read 1 Done Done Sent false
22554 6892 Acked Read 2 Done Done Sent false
22555 6893 Acked Read 9 Done Done Sent false
22556 6894 Acked Write 3 Done Done Sent false
22557 6895 Acked Read 8 Done Done Sent false
22558 6896 Acked Write 9 Done Done Sent false
22559 6897 Acked Read 4 Done Done Sent false
22560 6898 Acked Read 5 Done Done Sent false
22561 6899 Acked Write 8 Done Done Sent false
22562 6900 Acked Write 5 Done Done Sent false
22563 6901 Acked Write 9 Done Done Sent false
22564 6902 Acked Write 10 Done Done Sent false
22565 6903 Acked Write 1 Done Done Sent false
22566 6904 Acked Read 10 Done Done Sent false
22567 6905 Acked Write 3 Done Done Sent false
22568 6906 Acked Read 9 Done Done Sent false
22569 6907 Acked Read 1 Done Done Sent false
22570 6908 Acked Flush 0 Done Done Sent false
22571 6909 Acked Flush 0 Done Done Sent false
22572 6910 Acked Write 8 Done Done Sent false
22573 6911 Acked Write 8 Done Done Sent false
22574 6912 Acked Flush 0 Done Done Sent false
22575 6913 Acked Write 6 Done Done Sent false
22576 6914 Acked Write 6 Done Done Sent false
22577 6915 Acked Write 1 Done Done Sent false
22578 6916 Acked Read 6 Done Done Sent false
22579 6917 Acked Write 10 Done Done Sent false
22580 6918 Acked Write 1 Done Done Sent false
22581 6919 Acked Read 3 Done Done Sent false
22582 6920 Acked Read 2 Done Done Sent false
22583 6921 Acked Read 4 Done Done Sent false
22584 6922 Acked Read 10 Done Done Sent false
22585 6923 Acked Read 3 Done Done Sent false
22586 6924 Acked Read 8 Done Done Sent false
22587 6925 Acked Read 1 Done Done Sent false
22588 6926 Acked Read 6 Done Done Sent false
22589 6927 Acked Flush 0 Done Done Sent false
22590 6928 Acked Write 1 Done Done Sent false
22591 6929 Acked Write 10 Done Done Sent false
22592 6930 Acked Write 9 Done Done Sent false
22593 6931 Acked Write 8 Done Done Sent false
22594 6932 Acked Write 2 Done Done Sent false
22595 6933 Acked Write 9 Done Done Sent false
22596 6934 Acked Read 4 Done Done Sent false
22597 6935 Acked Read 8 Done Done Sent false
22598 6936 Acked Write 7 Done Done Sent false
22599 6937 Acked Write 7 Done Done Sent false
22600 6938 Acked Read 7 Done Done Sent false
22601 6939 Acked Flush 0 Done Done Sent false
22602 6940 Acked Flush 0 Done Done Sent false
22603 6941 Acked Write 5 Done Done Sent false
22604 6942 Acked Read 1 Done Done Sent false
22605 6943 Acked Read 1 Done Done Sent false
22606 6944 Acked Read 3 Done Done Sent false
22607 6945 Acked Read 3 Done Done Sent false
22608 6946 Acked Read 8 Done Done Sent false
22609 6947 Acked Flush 0 Done Done Sent false
22610 6948 Acked Write 10 Done Done Sent false
22611 6949 Acked Write 4 Done Done Sent false
22612 6950 Acked Read 4 Done Done Sent false
22613 6951 Acked Read 5 Done Done Sent false
22614 6952 Acked Write 10 Done Done Sent false
22615 6953 Acked Read 3 Done Done Sent false
22616 6954 Acked Read 3 Done Done Sent false
22617 6955 Acked Write 8 Done Done Sent false
22618 6956 Acked Write 6 Done Done Sent false
22619 6957 Acked Write 4 Done Done Sent false
22620 6958 Acked Write 4 Done Done Sent false
22621 6959 Acked Read 6 Done Done Sent false
22622 6960 Acked Write 7 Done Done Sent false
22623 6961 Acked Write 5 Done Done Sent false
22624 6962 Acked Flush 0 Done Done Sent false
22625 6963 Acked Read 6 Done Done Sent false
22626 6964 Acked Read 3 Done Done Sent false
22627 6965 Acked Write 1 Done Done Sent false
22628 6966 Acked Read 6 Done Done Sent false
22629 6967 Acked Write 8 Done Done Sent false
22630 6968 Acked Write 4 Done Done Sent false
22631 6969 Acked Write 1 Done Done Sent false
22632 6970 Acked Write 6 Done Done Sent false
22633 6971 Acked Read 3 Done Done Sent false
22634 6972 Acked Read 7 Done Done Sent false
22635 6973 Acked Flush 0 Done Done Sent false
22636 6974 Acked Write 7 Done Done Sent false
22637 6975 Acked Write 3 Done Done Sent false
22638 6976 Acked Read 3 Done Done Sent false
22639 6977 Acked Read 1 Done Done Sent false
22640 6978 Acked Write 10 Done Done Sent false
22641 6979 Acked Read 4 Done Done Sent false
22642 6980 Acked Write 2 Done Done Sent false
22643 6981 Acked Read 10 Done Done Sent false
22644 6982 Acked Read 5 Done Done Sent false
22645 6983 Acked Write 3 Done Done Sent false
22646 6984 Acked Read 4 Done Done Sent false
22647 6985 Acked Read 9 Done Done Sent false
22648 6986 Acked Read 3 Done Done Sent false
22649 6987 Acked Flush 0 Done Done Sent false
22650 6988 Acked Read 5 Done Done Sent false
22651 6989 Acked Write 1 Done Done Sent false
22652 6990 Acked Read 3 Done Done Sent false
22653 6991 Acked Read 3 Done Done Sent false
22654 6992 Acked Write 3 Done Done Sent false
22655 6993 Acked Write 5 Done Done Sent false
22656 6994 Acked Read 9 Done Done Sent false
22657 6995 Acked Write 6 Done Done Sent false
22658 6996 Acked Write 3 Done Done Sent false
22659 6997 Acked Read 9 Done Done Sent false
22660 6998 Acked Read 4 Done Done Sent false
22661 6999 Acked Read 1 Done Done Sent false
22662 7000 Acked Write 4 Done Done Sent false
22663 7001 Acked Write 5 Done Done Sent false
22664 7002 Acked Write 4 Done Done Sent false
22665 7003 Acked Read 6 Done Done Sent false
22666 7004 Acked Read 4 Done Done Sent false
22667 7005 Acked Read 9 Done Done Sent false
22668 7006 Acked Write 8 Done Done Sent false
22669 7007 Acked Write 5 Done Done Sent false
22670 7008 Acked Write 8 Done Done Sent false
22671 7009 Acked Read 8 Done Done Sent false
22672 7010 Acked Write 10 Done Done Sent false
22673 7011 Acked Flush 0 Done Done Sent false
22674 7012 Acked Read 1 Done Done Sent false
22675 7013 Acked Flush 0 Done Done Sent false
22676 7014 Acked Read 3 Done Done Sent false
22677 7015 Acked Read 4 Done Done Sent false
22678 7016 Acked Write 2 Done Done Sent false
22679 7017 Acked Read 8 Done Done Sent false
22680 7018 Acked Read 5 Done Done Sent false
22681 7019 Acked Read 5 Done Done Sent false
22682 7020 Acked Flush 0 Done Done Sent false
22683 7021 Acked Write 10 Done Done Sent false
22684 7022 Acked Read 9 Done Done Sent false
22685 7023 Acked Write 5 Done Done Sent false
22686 7024 Acked Write 10 Done Done Sent false
22687 7025 Acked Read 5 Done Done Sent false
22688 7026 Acked Read 7 Done Done Sent false
22689 7027 Acked Write 8 Done Done Sent false
22690 7028 Acked Read 2 Done Done Sent false
22691 7029 Acked Read 8 Done Done Sent false
22692 7030 Acked Write 7 Done Done Sent false
22693 7031 Acked Read 7 Done Done Sent false
22694 7032 Acked Read 3 Done Done Sent false
22695 7033 Acked Write 6 Done Done Sent false
22696 7034 Acked Read 7 Done Done Sent false
22697 7035 Acked Write 3 Done Done Sent false
22698 7036 Acked Write 2 Done Done Sent false
22699 7037 Acked Write 1 Done Done Sent false
22700 STATES DS:0 DS:1 DS:2 TOTAL
22701 Sent 0 0 301 301
22702 Done 301 301 0 602
22703 Skipped 0 0 0 0
22704 Error 0 0 0 0
22705 Last Flush: Some(JobId(7020)) Some(JobId(7020)) Some(JobId(6736))
22706 Downstairs last five completed: 6736 6735 6734 6733 6732
22707 Upstairs last five completed: 7037 7036 7035 7034 7033
22708 CLIENT: Up:0 ds:301 act:2
227092025-02-25T17:23:12.747ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
227102025-02-25T17:23:13.748ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
227112025-02-25T17:23:14.749ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
22712 ----------------------------------------------------------------
22713 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22714 States: Active Active Offline
22715 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22716 6737 Acked Write 8 Done Done Sent false
22717 6738 Acked Read 5 Done Done Sent false
22718 6739 Acked Write 8 Done Done Sent false
22719 6740 Acked Read 4 Done Done Sent false
22720 6741 Acked Read 4 Done Done Sent false
22721 6742 Acked Write 9 Done Done Sent false
22722 6743 Acked Write 7 Done Done Sent false
22723 6744 Acked Read 5 Done Done Sent false
22724 6745 Acked Write 7 Done Done Sent false
22725 6746 Acked Read 1 Done Done Sent false
22726 6747 Acked Write 3 Done Done Sent false
22727 6748 Acked Write 2 Done Done Sent false
22728 6749 Acked Read 9 Done Done Sent false
22729 6750 Acked Flush 0 Done Done Sent false
22730 6751 Acked Read 6 Done Done Sent false
22731 6752 Acked Read 4 Done Done Sent false
22732 6753 Acked Read 5 Done Done Sent false
22733 6754 Acked Read 10 Done Done Sent false
22734 6755 Acked Read 2 Done Done Sent false
22735 6756 Acked Read 6 Done Done Sent false
22736 6757 Acked Write 10 Done Done Sent false
22737 6758 Acked Write 1 Done Done Sent false
22738 6759 Acked Read 3 Done Done Sent false
22739 6760 Acked Read 4 Done Done Sent false
22740 6761 Acked Read 3 Done Done Sent false
22741 6762 Acked Write 9 Done Done Sent false
22742 6763 Acked Write 4 Done Done Sent false
22743 6764 Acked Write 9 Done Done Sent false
22744 6765 Acked Flush 0 Done Done Sent false
22745 6766 Acked Flush 0 Done Done Sent false
22746 6767 Acked Read 10 Done Done Sent false
22747 6768 Acked Read 8 Done Done Sent false
22748 6769 Acked Read 1 Done Done Sent false
22749 6770 Acked Read 3 Done Done Sent false
22750 6771 Acked Flush 0 Done Done Sent false
22751 6772 Acked Write 10 Done Done Sent false
22752 6773 Acked Write 4 Done Done Sent false
22753 6774 Acked Write 6 Done Done Sent false
22754 6775 Acked Read 4 Done Done Sent false
22755 6776 Acked Read 6 Done Done Sent false
22756 6777 Acked Write 8 Done Done Sent false
22757 6778 Acked Read 5 Done Done Sent false
22758 6779 Acked Write 4 Done Done Sent false
22759 6780 Acked Write 5 Done Done Sent false
22760 6781 Acked Write 2 Done Done Sent false
22761 6782 Acked Read 1 Done Done Sent false
22762 6783 Acked Write 4 Done Done Sent false
22763 6784 Acked Read 2 Done Done Sent false
22764 6785 Acked Write 10 Done Done Sent false
22765 6786 Acked Read 2 Done Done Sent false
22766 6787 Acked Write 7 Done Done Sent false
22767 6788 Acked Write 5 Done Done Sent false
22768 6789 Acked Write 5 Done Done Sent false
22769 6790 Acked Write 3 Done Done Sent false
22770 6791 Acked Read 2 Done Done Sent false
22771 6792 Acked Read 1 Done Done Sent false
22772 6793 Acked Flush 0 Done Done Sent false
22773 6794 Acked Read 4 Done Done Sent false
22774 6795 Acked Write 3 Done Done Sent false
22775 6796 Acked Write 10 Done Done Sent false
22776 6797 Acked Read 5 Done Done Sent false
22777 6798 Acked Write 7 Done Done Sent false
22778 6799 Acked Write 6 Done Done Sent false
22779 6800 Acked Read 5 Done Done Sent false
22780 6801 Acked Write 8 Done Done Sent false
22781 6802 Acked Read 7 Done Done Sent false
22782 6803 Acked Write 1 Done Done Sent false
22783 6804 Acked Read 4 Done Done Sent false
22784 6805 Acked Read 5 Done Done Sent false
22785 6806 Acked Write 6 Done Done Sent false
22786 6807 Acked Write 4 Done Done Sent false
22787 6808 Acked Read 7 Done Done Sent false
22788 6809 Acked Write 1 Done Done Sent false
22789 6810 Acked Read 3 Done Done Sent false
22790 6811 Acked Read 10 Done Done Sent false
22791 6812 Acked Read 3 Done Done Sent false
22792 6813 Acked Flush 0 Done Done Sent false
22793 6814 Acked Read 10 Done Done Sent false
22794 6815 Acked Write 2 Done Done Sent false
22795 6816 Acked Write 9 Done Done Sent false
22796 6817 Acked Read 1 Done Done Sent false
22797 6818 Acked Write 3 Done Done Sent false
22798 6819 Acked Write 9 Done Done Sent false
22799 6820 Acked Write 9 Done Done Sent false
22800 6821 Acked Write 4 Done Done Sent false
22801 6822 Acked Read 1 Done Done Sent false
22802 6823 Acked Read 3 Done Done Sent false
22803 6824 Acked Read 1 Done Done Sent false
22804 6825 Acked Read 7 Done Done Sent false
22805 6826 Acked Write 4 Done Done Sent false
22806 6827 Acked Write 5 Done Done Sent false
22807 6828 Acked Write 2 Done Done Sent false
22808 6829 Acked Write 8 Done Done Sent false
22809 6830 Acked Write 8 Done Done Sent false
22810 6831 Acked Write 10 Done Done Sent false
22811 6832 Acked Read 10 Done Done Sent false
22812 6833 Acked Write 2 Done Done Sent false
22813 6834 Acked Read 10 Done Done Sent false
22814 6835 Acked Read 1 Done Done Sent false
22815 6836 Acked Read 7 Done Done Sent false
22816 6837 Acked Flush 0 Done Done Sent false
22817 6838 Acked Write 8 Done Done Sent false
22818 6839 Acked Read 10 Done Done Sent false
22819 6840 Acked Read 2 Done Done Sent false
22820 6841 Acked Write 6 Done Done Sent false
22821 6842 Acked Write 6 Done Done Sent false
22822 6843 Acked Read 10 Done Done Sent false
22823 6844 Acked Write 6 Done Done Sent false
22824 6845 Acked Write 1 Done Done Sent false
22825 6846 Acked Read 4 Done Done Sent false
22826 6847 Acked Write 2 Done Done Sent false
22827 6848 Acked Flush 0 Done Done Sent false
22828 6849 Acked Read 9 Done Done Sent false
22829 6850 Acked Read 8 Done Done Sent false
22830 6851 Acked Read 7 Done Done Sent false
22831 6852 Acked Flush 0 Done Done Sent false
22832 6853 Acked Read 3 Done Done Sent false
22833 6854 Acked Write 9 Done Done Sent false
22834 6855 Acked Flush 0 Done Done Sent false
22835 6856 Acked Read 8 Done Done Sent false
22836 6857 Acked Read 10 Done Done Sent false
22837 6858 Acked Write 2 Done Done Sent false
22838 6859 Acked Write 4 Done Done Sent false
22839 6860 Acked Write 5 Done Done Sent false
22840 6861 Acked Write 4 Done Done Sent false
22841 6862 Acked Read 5 Done Done Sent false
22842 6863 Acked Write 8 Done Done Sent false
22843 6864 Acked Write 8 Done Done Sent false
22844 6865 Acked Flush 0 Done Done Sent false
22845 6866 Acked Read 5 Done Done Sent false
22846 6867 Acked Flush 0 Done Done Sent false
22847 6868 Acked Read 1 Done Done Sent false
22848 6869 Acked Read 3 Done Done Sent false
22849 6870 Acked Write 2 Done Done Sent false
22850 6871 Acked Write 9 Done Done Sent false
22851 6872 Acked Read 1 Done Done Sent false
22852 6873 Acked Write 2 Done Done Sent false
22853 6874 Acked Write 7 Done Done Sent false
22854 6875 Acked Read 6 Done Done Sent false
22855 6876 Acked Read 5 Done Done Sent false
22856 6877 Acked Read 6 Done Done Sent false
22857 6878 Acked Flush 0 Done Done Sent false
22858 6879 Acked Read 1 Done Done Sent false
22859 6880 Acked Read 1 Done Done Sent false
22860 6881 Acked Read 10 Done Done Sent false
22861 6882 Acked Write 2 Done Done Sent false
22862 6883 Acked Read 5 Done Done Sent false
22863 6884 Acked Write 9 Done Done Sent false
22864 6885 Acked Flush 0 Done Done Sent false
22865 6886 Acked Read 4 Done Done Sent false
22866 6887 Acked Write 2 Done Done Sent false
22867 6888 Acked Read 4 Done Done Sent false
22868 6889 Acked Flush 0 Done Done Sent false
22869 6890 Acked Read 1 Done Done Sent false
22870 6891 Acked Read 1 Done Done Sent false
22871 6892 Acked Read 2 Done Done Sent false
22872 6893 Acked Read 9 Done Done Sent false
22873 6894 Acked Write 3 Done Done Sent false
22874 6895 Acked Read 8 Done Done Sent false
22875 6896 Acked Write 9 Done Done Sent false
22876 6897 Acked Read 4 Done Done Sent false
22877 6898 Acked Read 5 Done Done Sent false
22878 6899 Acked Write 8 Done Done Sent false
22879 6900 Acked Write 5 Done Done Sent false
22880 6901 Acked Write 9 Done Done Sent false
22881 6902 Acked Write 10 Done Done Sent false
22882 6903 Acked Write 1 Done Done Sent false
22883 6904 Acked Read 10 Done Done Sent false
22884 6905 Acked Write 3 Done Done Sent false
22885 6906 Acked Read 9 Done Done Sent false
22886 6907 Acked Read 1 Done Done Sent false
22887 6908 Acked Flush 0 Done Done Sent false
22888 6909 Acked Flush 0 Done Done Sent false
22889 6910 Acked Write 8 Done Done Sent false
22890 6911 Acked Write 8 Done Done Sent false
22891 6912 Acked Flush 0 Done Done Sent false
22892 6913 Acked Write 6 Done Done Sent false
22893 6914 Acked Write 6 Done Done Sent false
22894 6915 Acked Write 1 Done Done Sent false
22895 6916 Acked Read 6 Done Done Sent false
22896 6917 Acked Write 10 Done Done Sent false
22897 6918 Acked Write 1 Done Done Sent false
22898 6919 Acked Read 3 Done Done Sent false
22899 6920 Acked Read 2 Done Done Sent false
22900 6921 Acked Read 4 Done Done Sent false
22901 6922 Acked Read 10 Done Done Sent false
22902 6923 Acked Read 3 Done Done Sent false
22903 6924 Acked Read 8 Done Done Sent false
22904 6925 Acked Read 1 Done Done Sent false
22905 6926 Acked Read 6 Done Done Sent false
22906 6927 Acked Flush 0 Done Done Sent false
22907 6928 Acked Write 1 Done Done Sent false
22908 6929 Acked Write 10 Done Done Sent false
22909 6930 Acked Write 9 Done Done Sent false
22910 6931 Acked Write 8 Done Done Sent false
22911 6932 Acked Write 2 Done Done Sent false
22912 6933 Acked Write 9 Done Done Sent false
22913 6934 Acked Read 4 Done Done Sent false
22914 6935 Acked Read 8 Done Done Sent false
22915 6936 Acked Write 7 Done Done Sent false
22916 6937 Acked Write 7 Done Done Sent false
22917 6938 Acked Read 7 Done Done Sent false
22918 6939 Acked Flush 0 Done Done Sent false
22919 6940 Acked Flush 0 Done Done Sent false
22920 6941 Acked Write 5 Done Done Sent false
22921 6942 Acked Read 1 Done Done Sent false
22922 6943 Acked Read 1 Done Done Sent false
22923 6944 Acked Read 3 Done Done Sent false
22924 6945 Acked Read 3 Done Done Sent false
22925 6946 Acked Read 8 Done Done Sent false
22926 6947 Acked Flush 0 Done Done Sent false
22927 6948 Acked Write 10 Done Done Sent false
22928 6949 Acked Write 4 Done Done Sent false
22929 6950 Acked Read 4 Done Done Sent false
22930 6951 Acked Read 5 Done Done Sent false
22931 6952 Acked Write 10 Done Done Sent false
22932 6953 Acked Read 3 Done Done Sent false
22933 6954 Acked Read 3 Done Done Sent false
22934 6955 Acked Write 8 Done Done Sent false
22935 6956 Acked Write 6 Done Done Sent false
22936 6957 Acked Write 4 Done Done Sent false
22937 6958 Acked Write 4 Done Done Sent false
22938 6959 Acked Read 6 Done Done Sent false
22939 6960 Acked Write 7 Done Done Sent false
22940 6961 Acked Write 5 Done Done Sent false
22941 6962 Acked Flush 0 Done Done Sent false
22942 6963 Acked Read 6 Done Done Sent false
22943 6964 Acked Read 3 Done Done Sent false
22944 6965 Acked Write 1 Done Done Sent false
22945 6966 Acked Read 6 Done Done Sent false
22946 6967 Acked Write 8 Done Done Sent false
22947 6968 Acked Write 4 Done Done Sent false
22948 6969 Acked Write 1 Done Done Sent false
22949 6970 Acked Write 6 Done Done Sent false
22950 6971 Acked Read 3 Done Done Sent false
22951 6972 Acked Read 7 Done Done Sent false
22952 6973 Acked Flush 0 Done Done Sent false
22953 6974 Acked Write 7 Done Done Sent false
22954 6975 Acked Write 3 Done Done Sent false
22955 6976 Acked Read 3 Done Done Sent false
22956 6977 Acked Read 1 Done Done Sent false
22957 6978 Acked Write 10 Done Done Sent false
22958 6979 Acked Read 4 Done Done Sent false
22959 6980 Acked Write 2 Done Done Sent false
22960 6981 Acked Read 10 Done Done Sent false
22961 6982 Acked Read 5 Done Done Sent false
22962 6983 Acked Write 3 Done Done Sent false
22963 6984 Acked Read 4 Done Done Sent false
22964 6985 Acked Read 9 Done Done Sent false
22965 6986 Acked Read 3 Done Done Sent false
22966 6987 Acked Flush 0 Done Done Sent false
22967 6988 Acked Read 5 Done Done Sent false
22968 6989 Acked Write 1 Done Done Sent false
22969 6990 Acked Read 3 Done Done Sent false
22970 6991 Acked Read 3 Done Done Sent false
22971 6992 Acked Write 3 Done Done Sent false
22972 6993 Acked Write 5 Done Done Sent false
22973 6994 Acked Read 9 Done Done Sent false
22974 6995 Acked Write 6 Done Done Sent false
22975 6996 Acked Write 3 Done Done Sent false
22976 6997 Acked Read 9 Done Done Sent false
22977 6998 Acked Read 4 Done Done Sent false
22978 6999 Acked Read 1 Done Done Sent false
22979 7000 Acked Write 4 Done Done Sent false
22980 7001 Acked Write 5 Done Done Sent false
22981 7002 Acked Write 4 Done Done Sent false
22982 7003 Acked Read 6 Done Done Sent false
22983 7004 Acked Read 4 Done Done Sent false
22984 7005 Acked Read 9 Done Done Sent false
22985 7006 Acked Write 8 Done Done Sent false
22986 7007 Acked Write 5 Done Done Sent false
22987 7008 Acked Write 8 Done Done Sent false
22988 7009 Acked Read 8 Done Done Sent false
22989 7010 Acked Write 10 Done Done Sent false
22990 7011 Acked Flush 0 Done Done Sent false
22991 7012 Acked Read 1 Done Done Sent false
22992 7013 Acked Flush 0 Done Done Sent false
22993 7014 Acked Read 3 Done Done Sent false
22994 7015 Acked Read 4 Done Done Sent false
22995 7016 Acked Write 2 Done Done Sent false
22996 7017 Acked Read 8 Done Done Sent false
22997 7018 Acked Read 5 Done Done Sent false
22998 7019 Acked Read 5 Done Done Sent false
22999 7020 Acked Flush 0 Done Done Sent false
23000 7021 Acked Write 10 Done Done Sent false
23001 7022 Acked Read 9 Done Done Sent false
23002 7023 Acked Write 5 Done Done Sent false
23003 7024 Acked Write 10 Done Done Sent false
23004 7025 Acked Read 5 Done Done Sent false
23005 7026 Acked Read 7 Done Done Sent false
23006 7027 Acked Write 8 Done Done Sent false
23007 7028 Acked Read 2 Done Done Sent false
23008 7029 Acked Read 8 Done Done Sent false
23009 7030 Acked Write 7 Done Done Sent false
23010 7031 Acked Read 7 Done Done Sent false
23011 7032 Acked Read 3 Done Done Sent false
23012 7033 Acked Write 6 Done Done Sent false
23013 7034 Acked Read 7 Done Done Sent false
23014 7035 Acked Write 3 Done Done Sent false
23015 7036 Acked Write 2 Done Done Sent false
23016 7037 Acked Write 1 Done Done Sent false
23017 STATES DS:0 DS:1 DS:2 TOTAL
23018 Sent 0 0 301 301
23019 Done 301 301 0 602
23020 Skipped 0 0 0 0
23021 Error 0 0 0 0
23022 Last Flush: Some(JobId(7020)) Some(JobId(7020)) Some(JobId(6736))
23023 Downstairs last five completed: 6736 6735 6734 6733 6732
23024 Upstairs last five completed: 7037 7036 7035 7034 7033
23025 CLIENT: Up:0 ds:301 act:2
230262025-02-25T17:23:15.751ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
230272025-02-25T17:23:17.754ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
230282025-02-25T17:23:18.755ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
230292025-02-25T17:23:18.854ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
230302025-02-25T17:23:18.854ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37067) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
230312025-02-25T17:23:18.855ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
230322025-02-25T17:23:18.856ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
230332025-02-25T17:23:18.856ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
230342025-02-25T17:23:18.856ZINFOcrucible: send last flush ID to this DS: Some(JobId(6736)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
230352025-02-25T17:23:18.856ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
230362025-02-25T17:23:18.856ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6736)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
230372025-02-25T17:23:18.856ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6736)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
23038 ----------------------------------------------------------------
23039 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:17
23040 States: Active Active Active
23041 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23042 7021 Acked Write 10 Done Done Done true
23043 7022 Acked Read 9 Done Done Done true
23044 7023 Acked Write 5 Done Done Done true
23045 7024 Acked Write 10 Done Done Done true
23046 7025 Acked Read 5 Done Done Done true
23047 7026 Acked Read 7 Done Done Done true
23048 7027 Acked Write 8 Done Done Done true
23049 7028 Acked Read 2 Done Done Done true
23050 7029 Acked Read 8 Done Done Done true
23051 7030 Acked Write 7 Done Done Done true
23052 7031 Acked Read 7 Done Done Done true
23053 7032 Acked Read 3 Done Done Done true
23054 7033 Acked Write 6 Done Done Done true
23055 7034 Acked Read 7 Done Done Done true
23056 7035 Acked Write 3 Done Done Done true
23057 7036 Acked Write 2 Done Done Done true
23058 7037 Acked Write 1 Done Done Done true
23059 STATES DS:0 DS:1 DS:2 TOTAL
23060 Sent 0 0 0 0
23061 Done 17 17 17 51
23062 Skipped 0 0 0 0
23063 Error 0 0 0 0
23064 Last Flush: Some(JobId(7020)) Some(JobId(7020)) Some(JobId(7020))
23065 Downstairs last five completed: 7020 7019 7018 7017 7016
23066 Upstairs last five completed: 7037 7036 7035 7034 7033
23067 CLIENT: Up:0 ds:17 act:3
23068 ----------------------------------------------------------------
23069 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23070 Downstairs last five completed: 7038 7037 7036 7035 7034
23071 Upstairs last five completed: 7038 7037 7036 7035 7034
23072 CLIENT: Up:0 ds:0 act:3
23073 Replay: All jobs finished
23074 Main task has work to do, go find it
23075 got dsc Stop(1)
23076 stop 1
23077 Feb 25 17:23:23.101 [8820] Got stop action so:false kr:false
23078 INFO request completed, latency_us: 361, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 42a2f387-37d6-41a3-9517-763eaea26032, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
23079 Feb 25 17:23:23.102 INFO request completed, latency_us: 143, response_code: 200, uri{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:23:23.102674608Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
23080 : /state/cid/1, method: GET, req_id: e6d28ee0-ccb5-4b29-8417-84a27fb6d310, remote_addr: 127.0{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:23:23.102742808Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
23081 .0.1:52576, local_addr{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:23:23.102779163Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
23082 : 127.0.0.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T17:23:23.102808999Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
23083 1:9998
23084 [8820] Exited with: ExitStatus(unix_wait_status(9))
23085 [8820][1] reports Exit
23086 Feb 25 17:23:27.105 INFO request completed, latency_us: 220, response_code: 200, uri: /state/cid/1, method: GET, req_id: bae1c0bd-4db2-4727-ad56-70b62be15c9d, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
23087 001/300 Write block 4982 len 28672 data: 4 4 4 4 2 3 4
23088 002/300 Write block 3223 len 12288 data: 6 6 6
23089 003/300 Write block 269 len 28672 data: 5 6 6 6 5 5 5
23090 004/300 Read block 4487 len 20480
23091 005/300 Read block 4077 len 24576
23092 006/300 Write block 1741 len 4096 data: 3
23093 007/300 Read block 3817 len 36864
23094 008/300 Read block 4127 len 28672
23095 009/300 Read block 1190 len 20480
23096 010/300 Write block 4334 len 20480 data: 4 4 6 6 6
23097 011/300 Read block 997 len 20480
23098 012/300 Write block 950 len 4096 data: 4
23099 013/300 Read block 4225 len 28672
23100 014/300 Read block 146 len 24576
23101 015/300 Read block 3564 len 16384
23102 016/300 Read block 3304 len 4096
23103 017/300 Flush
23104 018/300 Write block 1304 len 28672 data: 2 2 2 4 4 4 4
23105 019/300 Write block 2052 len 20480 data: 2 3 3 5 6
23106 020/300 Write block 1784 len 36864 data: 3 3 2 4 3 3 3 3 4
23107 021/300 Write block 4042 len 28672 data: 3 3 3 3 3 3 3
23108 022/300 Read block 827 len 20480
23109 023/300 Write block 1946 len 36864 data: 5 5 6 5 5 5 5 6 7
23110 024/300 Write block 3325 len 8192 data: 3 2
23111 025/300 Read block 148 len 36864
23112 026/300 Read block 4034 len 20480
23113 027/300 Write block 3684 len 28672 data: 5 5 6 6 7 6 3
23114 028/300 Write block 3117 len 4096 data: 5
23115 029/300 Read block 897 len 36864
23116 030/300 Read block 2818 len 28672
23117 031/300 Read block 4300 len 4096
23118 032/300 Read block 2947 len 28672
23119 033/300 Write block 3387 len 16384 data: 4 4 6 7
23120 034/300 Write block 4325 len 32768 data: 3 3 4 4 5 5 5 4
23121 035/300 Flush
23122 036/300 Flush
23123 037/300 Read block 3829 len 8192
23124 038/300 Read block 4193 len 12288
23125 039/300 Flush
23126 040/300 Write block 1708 len 36864 data: 5 5 6 6 8 8 7 7 5
23127 041/300 Read block 2317 len 24576
23128 042/300 Read block 3999 len 4096
23129 043/300 Read block 4891 len 28672
23130 044/300 Read block 874 len 4096
23131 045/300 Read block 3808 len 16384
23132 046/300 Read block 3279 len 20480
23133 047/300 Read block 2857 len 40960
23134 048/300 Read block 3179 len 16384
23135 049/300 Read block 3757 len 40960
23136 050/300 Read block 4808 len 20480
23137 051/300 Read block 801 len 32768
23138 052/300 Read block 2254 len 12288
23139 053/300 Flush
23140 054/300 Read block 3554 len 16384
23141 055/300 Read block 4936 len 20480
23142 056/300 Read block 1027 len 4096
23143 057/300 Write block 2801 len 20480 data: 3 4 3 3 4
23144 058/300 Write block 2781 len 28672 data: 6 6 6 5 4 7 7
23145 059/300 Write block 2710 len 20480 data: 5 4 4 5 4
23146 060/300 Read block 3772 len 24576
23147 061/300 Write block 3498 len 4096 data: 5
23148 062/300 Flush
23149 063/300 Write block 4740 len 12288 data: 3 3 2
23150 064/300 Flush
23151 065/300 Read block 2709 len 28672
23152 066/300 Write block 1149 len 20480 data: 4 4 4 4 4
23153 067/300 Write block 2442 len 32768 data: 3 3 3 4 5 4 4 5
23154 068/300 Write block 4471 len 40960 data: 5 4 4 3 3 2 2 2 2 2
23155 069/300 Read block 192 len 12288
23156 070/300 Write block 626 len 16384 data: 5 6 9 10
23157 071/300 Write block 1731 len 20480 data: 2 3 3 4 4
23158 072/300 Flush
23159 073/300 Read block 2009 len 16384
23160 074/300 Write block 3896 len 24576 data: 5 5 5 6 5 4
23161 075/300 Write block 1737 len 12288 data: 4 3 3
23162 076/300 Read block 68 len 40960
23163 077/300 Read block 1620 len 36864
23164 078/300 Write block 839 len 32768 data: 7 6 5 3 2 2 2 2
23165 079/300 Read block 2234 len 12288
23166 080/300 Flush
23167 081/300 Read block 3080 len 20480
23168 082/300 Write block 2940 len 8192 data: 4 6
23169 083/300 Read block 4555 len 20480
23170 084/300 Write block 2754 len 4096 data: 5
23171 085/300 Read block 2631 len 4096
23172 086/300 Flush
23173 087/300 Read block 3820 len 24576
23174 088/300 Read block 3306 len 36864
23175 089/300 Write block 4493 len 12288 data: 7 6 6
23176 090/300 Write block 1470 len 20480 data: 6 6 6 6 5
23177 091/300 Write block 4556 len 24576 data: 5 5 3 4 4 4
23178 092/300 Write block 2264 len 12288 data: 5 5 3
23179 093/300 Write block 711 len 20480 data: 4 4 4 3 3
23180 094/300 Read block 4258 len 24576
23181 095/300 Write block 4196 len 24576 data: 6 6 6 6 3 5
23182 096/300 Flush
23183 097/300 Read block 704 len 20480
23184 098/300 Read block 2166 len 12288
23185 099/300 Read block 1991 len 20480
23186 100/300 Write block 3653 len 20480 data: 8 9 9 10 8
23187 101/300 Read block 4426 len 20480
23188 102/300 Write block 2701 len 8192 data: 6 4
23189 103/300 Read block 734 len 36864
23190 104/300 Write block 4038 len 40960 data: 2 2 2 2 4 4 4 4 4 4
23191 105/300 Read block 3561 len 36864
23192 106/300 Read block 649 len 28672
23193 107/300 Flush
23194 108/300 Read block 604 len 36864
23195 109/300 Read block 3192 len 20480
23196 110/300 Read block 2326 len 24576
23197 111/300 Write block 3876 len 32768 data: 6 5 6 6 5 4 3 3
23198 112/300 Write block 3607 len 28672 data: 6 6 6 4 4 4 4
23199 113/300 Flush
23200 114/300 Read block 1477 len 4096
23201 115/300 Read block 1662 len 16384
23202 116/300 Write block 4188 len 40960 data: 3 4 4 4 3 4 4 6 7 7
23203 117/300 Read block 432 len 24576
23204 118/300 Read block 596 len 28672
23205 119/300 Read block 4233 len 16384
23206 120/300 Read block 4149 len 40960
23207 121/300 Read block 2734 len 8192
23208 122/300 Read block 3334 len 20480
23209 123/300 Write block 635 len 8192 data: 6 9
23210 124/300 Read block 1098 len 20480
23211 125/300 Read block 2512 len 32768
23212 126/300 Read block 1386 len 36864
23213 127/300 Read block 2797 len 8192
23214 128/300 Read block 3145 len 40960
23215 129/300 Write block 3920 len 8192 data: 7 6
23216 130/300 Write block 3921 len 24576 data: 7 5 5 4 3 2
23217 131/300 Read block 282 len 16384
23218 132/300 Flush
23219 133/300 Write block 2164 len 28672 data: 2 3 3 4 3 3 3
23220 134/300 Read block 1626 len 36864
23221 135/300 Write block 3571 len 12288 data: 3 4 5
23222 136/300 Read block 1246 len 40960
23223 137/300 Write block 4586 len 28672 data: 3 4 6 5 5 7 6
23224 138/300 Write block 3015 len 32768 data: 3 3 4 3 3 3 3 3
23225 139/300 Read block 2893 len 8192
23226 140/300 Read block 2591 len 8192
23227 141/300 Write block 1296 len 20480 data: 4 4 4 4 5
23228 142/300 Write block 1122 len 8192 data: 3 3
23229 143/300 Read block 838 len 40960
23230 144/300 Read block 1398 len 40960
23231 145/300 Write block 1970 len 12288 data: 4 3 3
23232 146/300 Flush
23233 147/300 Read block 2956 len 8192
23234 148/300 Write block 3720 len 8192 data: 5 6
23235 149/300 Write block 1424 len 28672 data: 4 4 6 8 7 7 8
23236 150/300 Write block 2990 len 40960 data: 4 4 5 5 8 8 7 5 7 7
23237 151/300 Write block 4867 len 36864 data: 3 3 3 5 5 5 5 5 5
23238 152/300 Write block 4719 len 36864 data: 5 8 8 7 4 4 3 4 4
23239 153/300 Write block 1621 len 20480 data: 5 4 4 4 6
23240 154/300 Read block 1807 len 36864
23241 155/300 Write block 1088 len 28672 data: 6 5 7 7 8 6 5
23242 156/300 Write block 4840 len 32768 data: 3 4 4 4 5 5 6 5
23243 157/300 Read block 1529 len 8192
23244 158/300 Write block 2263 len 40960 data: 6 6 6 4 2 2 3 2 2 3
23245 159/300 Read block 858 len 28672
23246 160/300 Write block 3035 len 40960 data: 4 4 4 4 3 3 4 4 4 4
23247 161/300 Read block 1122 len 32768
23248 162/300 Write block 589 len 8192 data: 6 5
23249 163/300 Write block 2586 len 4096 data: 4
23250 164/300 Read block 1987 len 20480
23251 165/300 Write block 455 len 28672 data: 6 8 7 5 5 5 3
23252 166/300 Write block 927 len 20480 data: 8 8 7 8 7
23253 167/300 Read block 2704 len 40960
23254 168/300 Write block 1947 len 20480 data: 6 7 6 6 6
23255 169/300 Read block 1913 len 4096
23256 170/300 Write block 1471 len 20480 data: 7 7 7 6 4
23257 171/300 Write block 3608 len 32768 data: 7 7 5 5 5 5 3 2
23258 172/300 Flush
23259 173/300 Write block 1665 len 4096 data: 3
23260 174/300 Read block 4924 len 24576
23261 175/300 Flush
23262 176/300 Read block 1960 len 4096
23263 177/300 Read block 2035 len 24576
23264 178/300 Write block 1504 len 24576 data: 6 7 6 6 6 5
23265 179/300 Write block 2283 len 12288 data: 5 5 4
23266 180/300 Write block 2286 len 40960 data: 3 2 2 3 3 5 6 7 7 7
23267 181/300 Write block 2593 len 20480 data: 4 4 4 3 2
23268 182/300 Flush
23269 183/300 Flush
23270 184/300 Write block 3991 len 36864 data: 6 5 6 5 4 4 4 4 4
23271 185/300 Write block 1194 len 36864 data: 6 5 4 6 6 4 4 3 3
23272 186/300 Read block 2426 len 32768
23273 187/300 Read block 4791 len 4096
23274 188/300 Read block 4696 len 8192
23275 189/300 Write block 3862 len 20480 data: 3 3 3 3 4
23276 190/300 Write block 296 len 8192 data: 4 4
23277 191/300 Read block 2299 len 8192
23278 192/300 Read block 140 len 24576
23279 193/300 Write block 1472 len 4096 data: 8
23280 194/300 Read block 1930 len 32768
23281 195/300 Write block 2503 len 28672 data: 6 5 7 6 6 6 5
23282 196/300 Read block 4181 len 4096
23283 197/300 Read block 2297 len 40960
23284 198/300 Write block 565 len 8192 data: 5 4
23285 199/300 Write block 4708 len 28672 data: 6 3 4 4 3 3 4
23286 200/300 Read block 3515 len 12288
23287 201/300 Write block 1089 len 40960 data: 6 8 8 9 7 6 5 5 6 5
23288 202/300 Write block 2054 len 40960 data: 4 6 7 7 7 5 6 6 4 5
23289 203/300 Read block 862 len 20480
23290 204/300 Read block 4742 len 4096
23291 205/300 Read block 3534 len 32768
23292 206/300 Write block 4987 len 36864 data: 4 5 6 5 6 5 5 4 4
23293 207/300 Read block 2303 len 8192
23294 208/300 Write block 2218 len 16384 data: 9 8 7 5
23295 209/300 Write block 601 len 12288 data: 7 6 6
23296 210/300 Read block 2131 len 12288
23297 211/300 Write block 1686 len 8192 data: 3 3
23298 212/300 Read block 4818 len 28672
23299 213/300 Write block 3954 len 12288 data: 4 3 3
23300 214/300 Read block 1412 len 32768
23301 215/300 Read block 1738 len 32768
23302 216/300 Read block 1521 len 20480
23303 217/300 Read block 2260 len 40960
23304 218/300 Write block 2314 len 28672 data: 5 4 4 3 4 5 5
23305 219/300 Write block 4104 len 20480 data: 3 3 3 3 3
23306 220/300 Read block 3136 len 36864
23307 221/300 Write block 1452 len 28672 data: 7 6 6 7 8 8 6
23308 222/300 Read block 4870 len 8192
23309 223/300 Flush
23310 224/300 Read block 2215 len 4096
23311 225/300 Write block 470 len 8192 data: 2 2
23312 226/300 Write block 3280 len 36864 data: 3 4 4 7 7 6 6 6 5
23313 227/300 Read block 3838 len 36864
23314 228/300 Read block 2581 len 12288
23315 229/300 Write block 470 len 36864 data: 3 3 2 2 2 2 3 3 5
23316 230/300 Write block 3119 len 8192 data: 5 5
23317 231/300 Read block 3738 len 24576
23318 232/300 Read block 2592 len 24576
23319 233/300 Read block 3014 len 40960
23320 234/300 Read block 2170 len 12288
23321 235/300 Flush
23322 236/300 Read block 3673 len 24576
23323 237/300 Read block 3455 len 32768
23324 238/300 Read block 3288 len 32768
23325 239/300 Read block 1930 len 28672
23326 240/300 Write block 3990 len 24576 data: 4 7 6 7 6 5
23327 241/300 Write block 2075 len 28672 data: 3 3 4 3 4 5 5
23328 242/300 Read block 4484 len 20480
23329 243/300 Flush
23330 244/300 Read block 1583 len 36864
23331 245/300 Read block 1447 len 36864
23332 246/300 Write block 2222 len 40960 data: 6 6 5 4 2 2 3 3 4 4
23333 247/300 Read block 2581 len 16384
23334 248/300 Read block 4109 len 8192
23335 249/300 Read block 2193 len 24576
23336 250/300 Write block 4784 len 36864 data: 3 3 3 4 4 4 5 4 3
23337 251/300 Read block 4082 len 20480
23338 252/300 Write block 4622 len 28672 data: 4 5 6 6 6 5 7
23339 253/300 Write block 1045 len 16384 data: 7 7 8 6
23340 254/300 Write block 4798 len 16384 data: 6 7 7 6
23341 255/300 Write block 2881 len 20480 data: 7 5 5 5 5
23342 256/300 Flush
23343 257/300 Read block 3378 len 12288
23344 258/300 Read block 1698 len 40960
23345 259/300 Read block 185 len 32768
23346 260/300 Read block 2059 len 40960
23347 261/300 Flush
23348 262/300 Read block 3012 len 36864
23349 263/300 Write block 3223 len 12288 data: 7 7 7
23350 264/300 Write block 166 len 16384 data: 4 4 4 4
23351 265/300 Read block 4521 len 20480
23352 266/300 Write block 3169 len 20480 data: 5 4 4 4 3
23353 267/300 Read block 2429 len 4096
23354 268/300 Read block 232 len 28672
23355 269/300 Read block 4522 len 28672
23356 270/300 Read block 1305 len 8192
23357 271/300 Read block 1846 len 8192
23358 272/300 Read block 2015 len 16384
23359 273/300 Write block 2964 len 4096 data: 6
23360 274/300 Read block 3416 len 12288
23361 275/300 Write block 592 len 16384 data: 4 4 4 4
23362 276/300 Write block 1570 len 20480 data: 3 4 3 5 3
23363 277/300 Read block 64 len 24576
23364 278/300 Write block 2417 len 16384 data: 2 3 3 3
23365 279/300 Write block 689 len 32768 data: 4 4 4 4 3 3 2 3
23366 280/300 Read block 2585 len 20480
23367 281/300 Read block 2280 len 4096
23368 282/300 Flush
23369 283/300 Read block 551 len 32768
23370 284/300 Read block 4787 len 12288
23371 285/300 Write block 242 len 24576 data: 4 4 4 3 3 4
23372 286/300 Write block 2578 len 4096 data: 6
23373 287/300 Write block 261 len 8192 data: 2 2
23374 288/300 Read block 1688 len 36864
23375 289/300 Read block 4320 len 20480
23376 290/300 Read block 3245 len 16384
23377 291/300 Write block 2949 len 40960 data: 5 5 6 7 6 5 4 4 5 4
23378 292/300 Read block 4913 len 20480
23379 293/300 Write block 2364 len 40960 data: 4 4 5 5 3 4 4 5 3 3
23380 294/300 Write block 2978 len 20480 data: 3 3 2 2 3
23381 295/300 Read block 2218 len 8192
23382 296/300 Read block 1280 len 24576
23383 297/300 Read block 4889 len 36864
23384 298/300 Read block 2326 len 4096
23385 299/300 Write block 498 len 40960 data: 5 4 3 2 2 2 2 2 2 2
23386 300/300 Read block 827 len 40960
23387 301/300 Read block 3993 len 36864
23388 Main task has work to do, go find it
23389 got dsc Start(1)
23390 start 1
23391 Feb 25 17:23:27.359 INFO request completed, latency_us: 229, response_code: 204, uri: /start/cid/1, method: POST, req_id: dc49d568-d8e1-4fa9-a102-4b06d1f536f5, remote_addr: 127.0.0.1:52576, local_addr: 127.Replay: started 1, returned:Ok(())
23392 0.0.1:9998
23393 ----------------------------------------------------------------
23394 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23395 States: Active Offline Active
23396 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23397 7039 Acked Write 7 Done Sent Done false
23398 7040 Acked Write 3 Done Sent Done false
23399 7041 Acked Write 7 Done Sent Done false
23400 7042 Acked Read 5 Done Sent Done false
23401 7043 Acked Read 6 Done Sent Done false
23402 7044 Acked Write 1 Done Sent Done false
23403 7045 Acked Read 9 Done Sent Done false
23404 7046 Acked Read 7 Done Sent Done false
23405 7047 Acked Read 5 Done Sent Done false
23406 7048 Acked Write 5 Done Sent Done false
23407 7049 Acked Read 5 Done Sent Done false
23408 7050 Acked Write 1 Done Sent Done false
23409 7051 Acked Read 7 Done Sent Done false
23410 7052 Acked Read 6 Done Sent Done false
23411 7053 Acked Read 4 Done Sent Done false
23412 7054 Acked Read 1 Done Sent Done false
23413 7055 Acked Flush 0 Done Sent Done false
23414 7056 Acked Write 7 Done Sent Done false
23415 7057 Acked Write 5 Done Sent Done false
23416 7058 Acked Write 9 Done Sent Done false
23417 7059 Acked Write 7 Done Sent Done false
23418 7060 Acked Read 5 Done Sent Done false
23419 7061 Acked Write 9 Done Sent Done false
23420 7062 Acked Write 2 Done Sent Done false
23421 7063 Acked Read 9 Done Sent Done false
23422 7064 Acked Read 5 Done Sent Done false
23423 7065 Acked Write 7 Done Sent Done false
23424 7066 Acked Write 1 Done Sent Done false
23425 7067 Acked Read 9 Done Sent Done false
23426 7068 Acked Read 7 Done Sent Done false
23427 7069 Acked Read 1 Done Sent Done false
23428 7070 Acked Read 7 Done Sent Done false
23429 7071 Acked Write 4 Done Sent Done false
23430 7072 Acked Write 8 Done Sent Done false
23431 7073 Acked Flush 0 Done Sent Done false
23432 7074 Acked Flush 0 Done Sent Done false
23433 7075 Acked Read 2 Done Sent Done false
23434 7076 Acked Read 3 Done Sent Done false
23435 7077 Acked Flush 0 Done Sent Done false
23436 7078 Acked Write 9 Done Sent Done false
23437 7079 Acked Read 6 Done Sent Done false
23438 7080 Acked Read 1 Done Sent Done false
23439 7081 Acked Read 7 Done Sent Done false
23440 7082 Acked Read 1 Done Sent Done false
23441 7083 Acked Read 4 Done Sent Done false
23442 7084 Acked Read 5 Done Sent Done false
23443 7085 Acked Read 10 Done Sent Done false
23444 7086 Acked Read 4 Done Sent Done false
23445 7087 Acked Read 10 Done Sent Done false
23446 7088 Acked Read 5 Done Sent Done false
23447 7089 Acked Read 8 Done Sent Done false
23448 7090 Acked Read 3 Done Sent Done false
23449 7091 Acked Flush 0 Done Sent Done false
23450 7092 Acked Read 4 Done Sent Done false
23451 7093 Acked Read 5 Done Sent Done false
23452 7094 Acked Read 1 Done Sent Done false
23453 7095 Acked Write 5 Done Sent Done false
23454 7096 Acked Write 7 Done Sent Done false
23455 7097 Acked Write 5 Done Sent Done false
23456 7098 Acked Read 6 Done Sent Done false
23457 7099 Acked Write 1 Done Sent Done false
23458 7100 Acked Flush 0 Done Sent Done false
23459 7101 Acked Write 3 Done Sent Done false
23460 7102 Acked Flush 0 Done Sent Done false
23461 7103 Acked Read 7 Done Sent Done false
23462 7104 Acked Write 5 Done Sent Done false
23463 7105 Acked Write 8 Done Sent Done false
23464 7106 Acked Write 10 Done Sent Done false
23465 7107 Acked Read 3 Done Sent Done false
23466 7108 Acked Write 4 Done Sent Done false
23467 7109 Acked Write 5 Done Sent Done false
23468 7110 Acked Flush 0 Done Sent Done false
23469 7111 Acked Read 4 Done Sent Done false
23470 7112 Acked Write 6 Done Sent Done false
23471 7113 Acked Write 3 Done Sent Done false
23472 7114 Acked Read 10 Done Sent Done false
23473 7115 Acked Read 9 Done Sent Done false
23474 7116 Acked Write 8 Done Sent Done false
23475 7117 Acked Read 3 Done Sent Done false
23476 7118 Acked Flush 0 Done Sent Done false
23477 7119 Acked Read 5 Done Sent Done false
23478 7120 Acked Write 2 Done Sent Done false
23479 7121 Acked Read 5 Done Sent Done false
23480 7122 Acked Write 1 Done Sent Done false
23481 7123 Acked Read 1 Done Sent Done false
23482 7124 Acked Flush 0 Done Sent Done false
23483 7125 Acked Read 6 Done Sent Done false
23484 7126 Acked Read 9 Done Sent Done false
23485 7127 Acked Write 3 Done Sent Done false
23486 7128 Acked Write 5 Done Sent Done false
23487 7129 Acked Write 6 Done Sent Done false
23488 7130 Acked Write 3 Done Sent Done false
23489 7131 Acked Write 5 Done Sent Done false
23490 7132 Acked Read 6 Done Sent Done false
23491 7133 Acked Write 6 Done Sent Done false
23492 7134 Acked Flush 0 Done Sent Done false
23493 7135 Acked Read 5 Done Sent Done false
23494 7136 Acked Read 3 Done Sent Done false
23495 7137 Acked Read 5 Done Sent Done false
23496 7138 Acked Write 5 Done Sent Done false
23497 7139 Acked Read 5 Done Sent Done false
23498 7140 Acked Write 2 Done Sent Done false
23499 7141 Acked Read 9 Done Sent Done false
23500 7142 Acked Write 10 Done Sent Done false
23501 7143 Acked Read 9 Done Sent Done false
23502 7144 Acked Read 7 Done Sent Done false
23503 7145 Acked Flush 0 Done Sent Done false
23504 7146 Acked Read 9 Done Sent Done false
23505 7147 Acked Read 5 Done Sent Done false
23506 7148 Acked Read 6 Done Sent Done false
23507 7149 Acked Write 8 Done Sent Done false
23508 7150 Acked Write 7 Done Sent Done false
23509 7151 Acked Flush 0 Done Sent Done false
23510 7152 Acked Read 1 Done Sent Done false
23511 7153 Acked Read 4 Done Sent Done false
23512 7154 Acked Write 10 Done Sent Done false
23513 7155 Acked Read 6 Done Sent Done false
23514 7156 Acked Read 7 Done Sent Done false
23515 7157 Acked Read 4 Done Sent Done false
23516 7158 Acked Read 10 Done Sent Done false
23517 7159 Acked Read 2 Done Sent Done false
23518 7160 Acked Read 5 Done Sent Done false
23519 7161 Acked Write 2 Done Sent Done false
23520 7162 Acked Read 5 Done Sent Done false
23521 7163 Acked Read 8 Done Sent Done false
23522 7164 Acked Read 9 Done Sent Done false
23523 7165 Acked Read 2 Done Sent Done false
23524 7166 Acked Read 10 Done Sent Done false
23525 7167 Acked Write 2 Done Sent Done false
23526 7168 Acked Write 6 Done Sent Done false
23527 7169 Acked Read 4 Done Sent Done false
23528 7170 Acked Flush 0 Done Sent Done false
23529 7171 Acked Write 7 Done Sent Done false
23530 7172 Acked Read 9 Done Sent Done false
23531 7173 Acked Write 3 Done Sent Done false
23532 7174 Acked Read 10 Done Sent Done false
23533 7175 Acked Write 7 Done Sent Done false
23534 7176 Acked Write 8 Done Sent Done false
23535 7177 Acked Read 2 Done Sent Done false
23536 7178 Acked Read 2 Done Sent Done false
23537 7179 Acked Write 5 Done Sent Done false
23538 7180 Acked Write 2 Done Sent Done false
23539 7181 Acked Read 10 Done Sent Done false
23540 7182 Acked Read 10 Done Sent Done false
23541 7183 Acked Write 3 Done Sent Done false
23542 7184 Acked Flush 0 Done Sent Done false
23543 7185 Acked Read 2 Done Sent Done false
23544 7186 Acked Write 2 Done Sent Done false
23545 7187 Acked Write 7 Done Sent Done false
23546 7188 Acked Write 10 Done Sent Done false
23547 7189 Acked Write 9 Done Sent Done false
23548 7190 Acked Write 9 Done Sent Done false
23549 7191 Acked Write 5 Done Sent Done false
23550 7192 Acked Read 9 Done Sent Done false
23551 7193 Acked Write 7 Done Sent Done false
23552 7194 Acked Write 8 Done Sent Done false
23553 7195 Acked Read 2 Done Sent Done false
23554 7196 Acked Write 10 Done Sent Done false
23555 7197 Acked Read 7 Done Sent Done false
23556 7198 Acked Write 10 Done Sent Done false
23557 7199 Acked Read 8 Done Sent Done false
23558 7200 Acked Write 2 Done Sent Done false
23559 7201 Acked Write 1 Done Sent Done false
23560 7202 Acked Read 5 Done Sent Done false
23561 7203 Acked Write 7 Done Sent Done false
23562 7204 Acked Write 5 Done Sent Done false
23563 7205 Acked Read 10 Done Sent Done false
23564 7206 Acked Write 5 Done Sent Done false
23565 7207 Acked Read 1 Done Sent Done false
23566 7208 Acked Write 5 Done Sent Done false
23567 7209 Acked Write 8 Done Sent Done false
23568 7210 Acked Flush 0 Done Sent Done false
23569 7211 Acked Write 1 Done Sent Done false
23570 7212 Acked Read 6 Done Sent Done false
23571 7213 Acked Flush 0 Done Sent Done false
23572 7214 Acked Read 1 Done Sent Done false
23573 7215 Acked Read 6 Done Sent Done false
23574 7216 Acked Write 6 Done Sent Done false
23575 7217 Acked Write 3 Done Sent Done false
23576 7218 Acked Write 10 Done Sent Done false
23577 7219 Acked Write 5 Done Sent Done false
23578 7220 Acked Flush 0 Done Sent Done false
23579 7221 Acked Flush 0 Done Sent Done false
23580 7222 Acked Write 9 Done Sent Done false
23581 7223 Acked Write 9 Done Sent Done false
23582 7224 Acked Read 8 Done Sent Done false
23583 7225 Acked Read 1 Done Sent Done false
23584 7226 Acked Read 2 Done Sent Done false
23585 7227 Acked Write 5 Done Sent Done false
23586 7228 Acked Write 2 Done Sent Done false
23587 7229 Acked Read 2 Done Sent Done false
23588 7230 Acked Read 6 Done Sent Done false
23589 7231 Acked Write 1 Done Sent Done false
23590 7232 Acked Read 8 Done Sent Done false
23591 7233 Acked Write 7 Done Sent Done false
23592 7234 Acked Read 1 Done Sent Done false
23593 7235 Acked Read 10 Done Sent Done false
23594 7236 Acked Write 2 Done Sent Done false
23595 7237 Acked Write 7 Done Sent Done false
23596 7238 Acked Read 3 Done Sent Done false
23597 7239 Acked Write 10 Done Sent Done false
23598 7240 Acked Write 10 Done Sent Done false
23599 7241 Acked Read 5 Done Sent Done false
23600 7242 Acked Read 1 Done Sent Done false
23601 7243 Acked Read 8 Done Sent Done false
23602 7244 Acked Write 9 Done Sent Done false
23603 7245 Acked Read 2 Done Sent Done false
23604 7246 Acked Write 4 Done Sent Done false
23605 7247 Acked Write 3 Done Sent Done false
23606 7248 Acked Read 3 Done Sent Done false
23607 7249 Acked Write 2 Done Sent Done false
23608 7250 Acked Read 7 Done Sent Done false
23609 7251 Acked Write 3 Done Sent Done false
23610 7252 Acked Read 8 Done Sent Done false
23611 7253 Acked Read 8 Done Sent Done false
23612 7254 Acked Read 5 Done Sent Done false
23613 7255 Acked Read 10 Done Sent Done false
23614 7256 Acked Write 7 Done Sent Done false
23615 7257 Acked Write 5 Done Sent Done false
23616 7258 Acked Read 9 Done Sent Done false
23617 7259 Acked Write 7 Done Sent Done false
23618 7260 Acked Read 2 Done Sent Done false
23619 7261 Acked Flush 0 Done Sent Done false
23620 7262 Acked Read 1 Done Sent Done false
23621 7263 Acked Write 2 Done Sent Done false
23622 7264 Acked Write 9 Done Sent Done false
23623 7265 Acked Read 9 Done Sent Done false
23624 7266 Acked Read 3 Done Sent Done false
23625 7267 Acked Write 9 Done Sent Done false
23626 7268 Acked Write 2 Done Sent Done false
23627 7269 Acked Read 6 Done Sent Done false
23628 7270 Acked Read 6 Done Sent Done false
23629 7271 Acked Read 10 Done Sent Done false
23630 7272 Acked Read 3 Done Sent Done false
23631 7273 Acked Flush 0 Done Sent Done false
23632 7274 Acked Read 6 Done Sent Done false
23633 7275 Acked Read 8 Done Sent Done false
23634 7276 Acked Read 8 Done Sent Done false
23635 7277 Acked Read 7 Done Sent Done false
23636 7278 Acked Write 6 Done Sent Done false
23637 7279 Acked Write 7 Done Sent Done false
23638 7280 Acked Read 5 Done Sent Done false
23639 7281 Acked Flush 0 Done Sent Done false
23640 7282 Acked Read 9 Done Sent Done false
23641 7283 Acked Read 9 Done Sent Done false
23642 7284 Acked Write 10 Done Sent Done false
23643 7285 Acked Read 4 Done Sent Done false
23644 7286 Acked Read 2 Done Sent Done false
23645 7287 Acked Read 6 Done Sent Done false
23646 7288 Acked Write 9 Done Sent Done false
23647 7289 Acked Read 5 Done Sent Done false
23648 7290 Acked Write 7 Done Sent Done false
23649 7291 Acked Write 4 Done Sent Done false
23650 7292 Acked Write 4 Done Sent Done false
23651 7293 Acked Write 5 Done Sent Done false
23652 7294 Acked Flush 0 Done Sent Done false
23653 7295 Acked Read 3 Done Sent Done false
23654 7296 Acked Read 10 Done Sent Done false
23655 7297 Acked Read 8 Done Sent Done false
23656 7298 Acked Read 10 Done Sent Done false
23657 7299 Acked Flush 0 Done Sent Done false
23658 7300 Acked Read 9 Done Sent Done false
23659 7301 Acked Write 3 Done Sent Done false
23660 7302 Acked Write 4 Done Sent Done false
23661 7303 Acked Read 5 Done Sent Done false
23662 7304 Acked Write 5 Done Sent Done false
23663 7305 Acked Read 1 Done Sent Done false
23664 7306 Acked Read 7 Done Sent Done false
23665 7307 Acked Read 7 Done Sent Done false
23666 7308 Acked Read 2 Done Sent Done false
23667 7309 Acked Read 2 Done Sent Done false
23668 7310 Acked Read 4 Done Sent Done false
23669 7311 Acked Write 1 Done Sent Done false
23670 7312 Acked Read 3 Done Sent Done false
23671 7313 Acked Write 4 Done Sent Done false
23672 7314 Acked Write 5 Done Sent Done false
23673 7315 Acked Read 6 Done Sent Done false
23674 7316 Acked Write 4 Done Sent Done false
23675 7317 Acked Write 8 Done Sent Done false
23676 7318 Acked Read 5 Done Sent Done false
23677 7319 Acked Read 1 Done Sent Done false
23678 7320 Acked Flush 0 Done Sent Done false
23679 7321 Acked Read 8 Done Sent Done false
23680 7322 Acked Read 3 Done Sent Done false
23681 7323 Acked Write 6 Done Sent Done false
23682 7324 Acked Write 1 Done Sent Done false
23683 7325 Acked Write 2 Done Sent Done false
23684 7326 Acked Read 9 Done Sent Done false
23685 7327 Acked Read 5 Done Sent Done false
23686 7328 Acked Read 4 Done Sent Done false
23687 7329 Acked Write 10 Done Sent Done false
23688 7330 Acked Read 5 Done Sent Done false
23689 7331 Acked Write 10 Done Sent Done false
23690 7332 Acked Write 5 Done Sent Done false
23691 7333 Acked Read 2 Done Sent Done false
23692 7334 Acked Read 6 Done Sent Done false
23693 7335 Acked Read 9 Done Sent Done false
23694 7336 Acked Read 1 Done Sent Done false
23695 7337 Acked Write 10 Done Sent Done false
23696 7338 Acked Read 10 Done Sent Done false
23697 7339 Acked Read 9 Done Sent Done false
23698 STATES DS:0 DS:1 DS:2 TOTAL
23699 Sent 0 301 0 301
23700 Done 301 0 301 602
23701 Skipped 0 0 0 0
23702 Error 0 0 0 0
23703 Last Flush: Some(JobId(7320)) Some(JobId(7038)) Some(JobId(7320))
23704 Downstairs last five completed: 7038 7037 7036 7035 7034
23705 Upstairs last five completed: 7339 7338 7337 7336 7335
23706 CLIENT: Up:0 ds:301 act:2
237072025-02-25T17:23:27.862ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
237082025-02-25T17:23:28.766ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
23709 [8820] got start action
23710 [8820] I am going to restart
23711 Starting downstairs at port 8820
23712 Make output file at "/tmp/dsc/downstairs-8820.txt"
23713 [8820][1] reports Starting
23714 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1229)
23715 [8820][1] reports Running
237162025-02-25T17:23:29.767ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
237172025-02-25T17:23:30.268ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
237182025-02-25T17:23:30.769ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
237192025-02-25T17:23:31.270ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
23720 ----------------------------------------------------------------
23721 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23722 States: Active Offline Active
23723 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23724 7039 Acked Write 7 Done Sent Done false
23725 7040 Acked Write 3 Done Sent Done false
23726 7041 Acked Write 7 Done Sent Done false
23727 7042 Acked Read 5 Done Sent Done false
23728 7043 Acked Read 6 Done Sent Done false
23729 7044 Acked Write 1 Done Sent Done false
23730 7045 Acked Read 9 Done Sent Done false
23731 7046 Acked Read 7 Done Sent Done false
23732 7047 Acked Read 5 Done Sent Done false
23733 7048 Acked Write 5 Done Sent Done false
23734 7049 Acked Read 5 Done Sent Done false
23735 7050 Acked Write 1 Done Sent Done false
23736 7051 Acked Read 7 Done Sent Done false
23737 7052 Acked Read 6 Done Sent Done false
23738 7053 Acked Read 4 Done Sent Done false
23739 7054 Acked Read 1 Done Sent Done false
23740 7055 Acked Flush 0 Done Sent Done false
23741 7056 Acked Write 7 Done Sent Done false
23742 7057 Acked Write 5 Done Sent Done false
23743 7058 Acked Write 9 Done Sent Done false
23744 7059 Acked Write 7 Done Sent Done false
23745 7060 Acked Read 5 Done Sent Done false
23746 7061 Acked Write 9 Done Sent Done false
23747 7062 Acked Write 2 Done Sent Done false
23748 7063 Acked Read 9 Done Sent Done false
23749 7064 Acked Read 5 Done Sent Done false
23750 7065 Acked Write 7 Done Sent Done false
23751 7066 Acked Write 1 Done Sent Done false
23752 7067 Acked Read 9 Done Sent Done false
23753 7068 Acked Read 7 Done Sent Done false
23754 7069 Acked Read 1 Done Sent Done false
23755 7070 Acked Read 7 Done Sent Done false
23756 7071 Acked Write 4 Done Sent Done false
23757 7072 Acked Write 8 Done Sent Done false
23758 7073 Acked Flush 0 Done Sent Done false
23759 7074 Acked Flush 0 Done Sent Done false
23760 7075 Acked Read 2 Done Sent Done false
23761 7076 Acked Read 3 Done Sent Done false
23762 7077 Acked Flush 0 Done Sent Done false
23763 7078 Acked Write 9 Done Sent Done false
23764 7079 Acked Read 6 Done Sent Done false
23765 7080 Acked Read 1 Done Sent Done false
23766 7081 Acked Read 7 Done Sent Done false
23767 7082 Acked Read 1 Done Sent Done false
23768 7083 Acked Read 4 Done Sent Done false
23769 7084 Acked Read 5 Done Sent Done false
23770 7085 Acked Read 10 Done Sent Done false
23771 7086 Acked Read 4 Done Sent Done false
23772 7087 Acked Read 10 Done Sent Done false
23773 7088 Acked Read 5 Done Sent Done false
23774 7089 Acked Read 8 Done Sent Done false
23775 7090 Acked Read 3 Done Sent Done false
23776 7091 Acked Flush 0 Done Sent Done false
23777 7092 Acked Read 4 Done Sent Done false
23778 7093 Acked Read 5 Done Sent Done false
23779 7094 Acked Read 1 Done Sent Done false
23780 7095 Acked Write 5 Done Sent Done false
23781 7096 Acked Write 7 Done Sent Done false
23782 7097 Acked Write 5 Done Sent Done false
23783 7098 Acked Read 6 Done Sent Done false
23784 7099 Acked Write 1 Done Sent Done false
23785 7100 Acked Flush 0 Done Sent Done false
23786 7101 Acked Write 3 Done Sent Done false
23787 7102 Acked Flush 0 Done Sent Done false
23788 7103 Acked Read 7 Done Sent Done false
23789 7104 Acked Write 5 Done Sent Done false
23790 7105 Acked Write 8 Done Sent Done false
23791 7106 Acked Write 10 Done Sent Done false
23792 7107 Acked Read 3 Done Sent Done false
23793 7108 Acked Write 4 Done Sent Done false
23794 7109 Acked Write 5 Done Sent Done false
23795 7110 Acked Flush 0 Done Sent Done false
23796 7111 Acked Read 4 Done Sent Done false
23797 7112 Acked Write 6 Done Sent Done false
23798 7113 Acked Write 3 Done Sent Done false
23799 7114 Acked Read 10 Done Sent Done false
23800 7115 Acked Read 9 Done Sent Done false
23801 7116 Acked Write 8 Done Sent Done false
23802 7117 Acked Read 3 Done Sent Done false
23803 7118 Acked Flush 0 Done Sent Done false
23804 7119 Acked Read 5 Done Sent Done false
23805 7120 Acked Write 2 Done Sent Done false
23806 7121 Acked Read 5 Done Sent Done false
23807 7122 Acked Write 1 Done Sent Done false
23808 7123 Acked Read 1 Done Sent Done false
23809 7124 Acked Flush 0 Done Sent Done false
23810 7125 Acked Read 6 Done Sent Done false
23811 7126 Acked Read 9 Done Sent Done false
23812 7127 Acked Write 3 Done Sent Done false
23813 7128 Acked Write 5 Done Sent Done false
23814 7129 Acked Write 6 Done Sent Done false
23815 7130 Acked Write 3 Done Sent Done false
23816 7131 Acked Write 5 Done Sent Done false
23817 7132 Acked Read 6 Done Sent Done false
23818 7133 Acked Write 6 Done Sent Done false
23819 7134 Acked Flush 0 Done Sent Done false
23820 7135 Acked Read 5 Done Sent Done false
23821 7136 Acked Read 3 Done Sent Done false
23822 7137 Acked Read 5 Done Sent Done false
23823 7138 Acked Write 5 Done Sent Done false
23824 7139 Acked Read 5 Done Sent Done false
23825 7140 Acked Write 2 Done Sent Done false
23826 7141 Acked Read 9 Done Sent Done false
23827 7142 Acked Write 10 Done Sent Done false
23828 7143 Acked Read 9 Done Sent Done false
23829 7144 Acked Read 7 Done Sent Done false
23830 7145 Acked Flush 0 Done Sent Done false
23831 7146 Acked Read 9 Done Sent Done false
23832 7147 Acked Read 5 Done Sent Done false
23833 7148 Acked Read 6 Done Sent Done false
23834 7149 Acked Write 8 Done Sent Done false
23835 7150 Acked Write 7 Done Sent Done false
23836 7151 Acked Flush 0 Done Sent Done false
23837 7152 Acked Read 1 Done Sent Done false
23838 7153 Acked Read 4 Done Sent Done false
23839 7154 Acked Write 10 Done Sent Done false
23840 7155 Acked Read 6 Done Sent Done false
23841 7156 Acked Read 7 Done Sent Done false
23842 7157 Acked Read 4 Done Sent Done false
23843 7158 Acked Read 10 Done Sent Done false
23844 7159 Acked Read 2 Done Sent Done false
23845 7160 Acked Read 5 Done Sent Done false
23846 7161 Acked Write 2 Done Sent Done false
23847 7162 Acked Read 5 Done Sent Done false
23848 7163 Acked Read 8 Done Sent Done false
23849 7164 Acked Read 9 Done Sent Done false
23850 7165 Acked Read 2 Done Sent Done false
23851 7166 Acked Read 10 Done Sent Done false
23852 7167 Acked Write 2 Done Sent Done false
23853 7168 Acked Write 6 Done Sent Done false
23854 7169 Acked Read 4 Done Sent Done false
23855 7170 Acked Flush 0 Done Sent Done false
23856 7171 Acked Write 7 Done Sent Done false
23857 7172 Acked Read 9 Done Sent Done false
23858 7173 Acked Write 3 Done Sent Done false
23859 7174 Acked Read 10 Done Sent Done false
23860 7175 Acked Write 7 Done Sent Done false
23861 7176 Acked Write 8 Done Sent Done false
23862 7177 Acked Read 2 Done Sent Done false
23863 7178 Acked Read 2 Done Sent Done false
23864 7179 Acked Write 5 Done Sent Done false
23865 7180 Acked Write 2 Done Sent Done false
23866 7181 Acked Read 10 Done Sent Done false
23867 7182 Acked Read 10 Done Sent Done false
23868 7183 Acked Write 3 Done Sent Done false
23869 7184 Acked Flush 0 Done Sent Done false
23870 7185 Acked Read 2 Done Sent Done false
23871 7186 Acked Write 2 Done Sent Done false
23872 7187 Acked Write 7 Done Sent Done false
23873 7188 Acked Write 10 Done Sent Done false
23874 7189 Acked Write 9 Done Sent Done false
23875 7190 Acked Write 9 Done Sent Done false
23876 7191 Acked Write 5 Done Sent Done false
23877 7192 Acked Read 9 Done Sent Done false
23878 7193 Acked Write 7 Done Sent Done false
23879 7194 Acked Write 8 Done Sent Done false
23880 7195 Acked Read 2 Done Sent Done false
23881 7196 Acked Write 10 Done Sent Done false
23882 7197 Acked Read 7 Done Sent Done false
23883 7198 Acked Write 10 Done Sent Done false
23884 7199 Acked Read 8 Done Sent Done false
23885 7200 Acked Write 2 Done Sent Done false
23886 7201 Acked Write 1 Done Sent Done false
23887 7202 Acked Read 5 Done Sent Done false
23888 7203 Acked Write 7 Done Sent Done false
23889 7204 Acked Write 5 Done Sent Done false
23890 7205 Acked Read 10 Done Sent Done false
23891 7206 Acked Write 5 Done Sent Done false
23892 7207 Acked Read 1 Done Sent Done false
23893 7208 Acked Write 5 Done Sent Done false
23894 7209 Acked Write 8 Done Sent Done false
23895 7210 Acked Flush 0 Done Sent Done false
23896 7211 Acked Write 1 Done Sent Done false
23897 7212 Acked Read 6 Done Sent Done false
23898 7213 Acked Flush 0 Done Sent Done false
23899 7214 Acked Read 1 Done Sent Done false
23900 7215 Acked Read 6 Done Sent Done false
23901 7216 Acked Write 6 Done Sent Done false
23902 7217 Acked Write 3 Done Sent Done false
23903 7218 Acked Write 10 Done Sent Done false
23904 7219 Acked Write 5 Done Sent Done false
23905 7220 Acked Flush 0 Done Sent Done false
23906 7221 Acked Flush 0 Done Sent Done false
23907 7222 Acked Write 9 Done Sent Done false
23908 7223 Acked Write 9 Done Sent Done false
23909 7224 Acked Read 8 Done Sent Done false
23910 7225 Acked Read 1 Done Sent Done false
23911 7226 Acked Read 2 Done Sent Done false
23912 7227 Acked Write 5 Done Sent Done false
23913 7228 Acked Write 2 Done Sent Done false
23914 7229 Acked Read 2 Done Sent Done false
23915 7230 Acked Read 6 Done Sent Done false
23916 7231 Acked Write 1 Done Sent Done false
23917 7232 Acked Read 8 Done Sent Done false
23918 7233 Acked Write 7 Done Sent Done false
23919 7234 Acked Read 1 Done Sent Done false
23920 7235 Acked Read 10 Done Sent Done false
23921 7236 Acked Write 2 Done Sent Done false
23922 7237 Acked Write 7 Done Sent Done false
23923 7238 Acked Read 3 Done Sent Done false
23924 7239 Acked Write 10 Done Sent Done false
23925 7240 Acked Write 10 Done Sent Done false
23926 7241 Acked Read 5 Done Sent Done false
23927 7242 Acked Read 1 Done Sent Done false
23928 7243 Acked Read 8 Done Sent Done false
23929 7244 Acked Write 9 Done Sent Done false
23930 7245 Acked Read 2 Done Sent Done false
23931 7246 Acked Write 4 Done Sent Done false
23932 7247 Acked Write 3 Done Sent Done false
23933 7248 Acked Read 3 Done Sent Done false
23934 7249 Acked Write 2 Done Sent Done false
23935 7250 Acked Read 7 Done Sent Done false
23936 7251 Acked Write 3 Done Sent Done false
23937 7252 Acked Read 8 Done Sent Done false
23938 7253 Acked Read 8 Done Sent Done false
23939 7254 Acked Read 5 Done Sent Done false
23940 7255 Acked Read 10 Done Sent Done false
23941 7256 Acked Write 7 Done Sent Done false
23942 7257 Acked Write 5 Done Sent Done false
23943 7258 Acked Read 9 Done Sent Done false
23944 7259 Acked Write 7 Done Sent Done false
23945 7260 Acked Read 2 Done Sent Done false
23946 7261 Acked Flush 0 Done Sent Done false
23947 7262 Acked Read 1 Done Sent Done false
23948 7263 Acked Write 2 Done Sent Done false
23949 7264 Acked Write 9 Done Sent Done false
23950 7265 Acked Read 9 Done Sent Done false
23951 7266 Acked Read 3 Done Sent Done false
23952 7267 Acked Write 9 Done Sent Done false
23953 7268 Acked Write 2 Done Sent Done false
23954 7269 Acked Read 6 Done Sent Done false
23955 7270 Acked Read 6 Done Sent Done false
23956 7271 Acked Read 10 Done Sent Done false
23957 7272 Acked Read 3 Done Sent Done false
23958 7273 Acked Flush 0 Done Sent Done false
23959 7274 Acked Read 6 Done Sent Done false
23960 7275 Acked Read 8 Done Sent Done false
23961 7276 Acked Read 8 Done Sent Done false
23962 7277 Acked Read 7 Done Sent Done false
23963 7278 Acked Write 6 Done Sent Done false
23964 7279 Acked Write 7 Done Sent Done false
23965 7280 Acked Read 5 Done Sent Done false
23966 7281 Acked Flush 0 Done Sent Done false
23967 7282 Acked Read 9 Done Sent Done false
23968 7283 Acked Read 9 Done Sent Done false
23969 7284 Acked Write 10 Done Sent Done false
23970 7285 Acked Read 4 Done Sent Done false
23971 7286 Acked Read 2 Done Sent Done false
23972 7287 Acked Read 6 Done Sent Done false
23973 7288 Acked Write 9 Done Sent Done false
23974 7289 Acked Read 5 Done Sent Done false
23975 7290 Acked Write 7 Done Sent Done false
23976 7291 Acked Write 4 Done Sent Done false
23977 7292 Acked Write 4 Done Sent Done false
23978 7293 Acked Write 5 Done Sent Done false
23979 7294 Acked Flush 0 Done Sent Done false
23980 7295 Acked Read 3 Done Sent Done false
23981 7296 Acked Read 10 Done Sent Done false
23982 7297 Acked Read 8 Done Sent Done false
23983 7298 Acked Read 10 Done Sent Done false
23984 7299 Acked Flush 0 Done Sent Done false
23985 7300 Acked Read 9 Done Sent Done false
23986 7301 Acked Write 3 Done Sent Done false
23987 7302 Acked Write 4 Done Sent Done false
23988 7303 Acked Read 5 Done Sent Done false
23989 7304 Acked Write 5 Done Sent Done false
23990 7305 Acked Read 1 Done Sent Done false
23991 7306 Acked Read 7 Done Sent Done false
23992 7307 Acked Read 7 Done Sent Done false
23993 7308 Acked Read 2 Done Sent Done false
23994 7309 Acked Read 2 Done Sent Done false
23995 7310 Acked Read 4 Done Sent Done false
23996 7311 Acked Write 1 Done Sent Done false
23997 7312 Acked Read 3 Done Sent Done false
23998 7313 Acked Write 4 Done Sent Done false
23999 7314 Acked Write 5 Done Sent Done false
24000 7315 Acked Read 6 Done Sent Done false
24001 7316 Acked Write 4 Done Sent Done false
24002 7317 Acked Write 8 Done Sent Done false
24003 7318 Acked Read 5 Done Sent Done false
24004 7319 Acked Read 1 Done Sent Done false
24005 7320 Acked Flush 0 Done Sent Done false
24006 7321 Acked Read 8 Done Sent Done false
24007 7322 Acked Read 3 Done Sent Done false
24008 7323 Acked Write 6 Done Sent Done false
24009 7324 Acked Write 1 Done Sent Done false
24010 7325 Acked Write 2 Done Sent Done false
24011 7326 Acked Read 9 Done Sent Done false
24012 7327 Acked Read 5 Done Sent Done false
24013 7328 Acked Read 4 Done Sent Done false
24014 7329 Acked Write 10 Done Sent Done false
24015 7330 Acked Read 5 Done Sent Done false
24016 7331 Acked Write 10 Done Sent Done false
24017 7332 Acked Write 5 Done Sent Done false
24018 7333 Acked Read 2 Done Sent Done false
24019 7334 Acked Read 6 Done Sent Done false
24020 7335 Acked Read 9 Done Sent Done false
24021 7336 Acked Read 1 Done Sent Done false
24022 7337 Acked Write 10 Done Sent Done false
24023 7338 Acked Read 10 Done Sent Done false
24024 7339 Acked Read 9 Done Sent Done false
24025 STATES DS:0 DS:1 DS:2 TOTAL
24026 Sent 0 301 0 301
24027 Done 301 0 301 602
24028 Skipped 0 0 0 0
24029 Error 0 0 0 0
24030 Last Flush: Some(JobId(7320)) Some(JobId(7038)) Some(JobId(7320))
24031 Downstairs last five completed: 7038 7037 7036 7035 7034
24032 Upstairs last five completed: 7339 7338 7337 7336 7335
24033 CLIENT: Up:0 ds:301 act:2
240342025-02-25T17:23:32.093ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240352025-02-25T17:23:32.773ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240362025-02-25T17:23:33.104ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240372025-02-25T17:23:33.104ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33482) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240382025-02-25T17:23:33.105ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240392025-02-25T17:23:33.105ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240402025-02-25T17:23:33.105ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240412025-02-25T17:23:33.105ZINFOcrucible: send last flush ID to this DS: Some(JobId(7038)) = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240422025-02-25T17:23:33.105ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240432025-02-25T17:23:33.105ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7038)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240442025-02-25T17:23:33.106ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7038)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
24045 ----------------------------------------------------------------
24046 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24047 Downstairs last five completed: 7340 7339 7338 7337 7336
24048 Upstairs last five completed: 7340 7339 7338 7337 7336
24049 CLIENT: Up:0 ds:0 act:3
24050 Replay: All jobs finished
24051 Main task has work to do, go find it
24052 got dsc Stop(2)
24053 stop 2
24054 Feb 25 17:23:35.364 INFO request completed, latency_us: [8830] Got stop action so:false kr:false
24055 395, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 389d9798-e8c6-48f9-b9d9-a81f26fcfd95, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
24056 Feb 25 17:23:35.365 INFO request completed, latency_us: 154, response_code: 200, uri: /state/cid/2, method: GET, req_id: 98f3bae4-134a-4266-b418-37f9454d5458, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
240572025-02-25T17:23:35.365ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240582025-02-25T17:23:35.365ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240592025-02-25T17:23:35.365ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
240602025-02-25T17:23:35.365ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
24061 [8830] Exited with: ExitStatus(unix_wait_status(9))
24062 [8830][2] reports Exit
24063 Feb 25 17:23:39.367 INFO request completed, latency_us: 222, response_code: 200, uri: /state/cid/2, method: GET, req_id: e3057da0-085c-447d-a529-3e0358112f58, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
24064 001/300 Read block 1229 len 8192
24065 002/300 Write block 2085 len 28672 data: 6 6 5 5 3 4 3
24066 003/300 Read block 4338 len 24576
24067 004/300 Read block 2749 len 36864
24068 005/300 Write block 210 len 40960 data: 5 5 5 4 6 6 6 4 4 3
24069 006/300 Read block 4828 len 36864
24070 007/300 Read block 3348 len 16384
24071 008/300 Read block 1787 len 40960
24072 009/300 Write block 2939 len 28672 data: 4 5 7 6 6 5 5
24073 010/300 Read block 182 len 32768
24074 011/300 Flush
24075 012/300 Read block 1516 len 36864
24076 013/300 Write block 3575 len 36864 data: 4 4 3 3 3 3 5 4 4
24077 014/300 Write block 728 len 16384 data: 5 5 5 4
24078 015/300 Flush
24079 016/300 Read block 1489 len 28672
24080 017/300 Read block 1531 len 28672
24081 018/300 Read block 130 len 40960
24082 019/300 Write block 1377 len 16384 data: 5 4 4 5
24083 020/300 Write block 2219 len 24576 data: 9 8 6 7 7 6
24084 021/300 Write block 3340 len 16384 data: 4 5 3 4
24085 022/300 Flush
24086 023/300 Write block 4122 len 20480 data: 3 4 4 3 4
24087 024/300 Write block 315 len 32768 data: 5 5 4 5 4 5 5 5
24088 025/300 Write block 3931 len 8192 data: 5 5
24089 026/300 Read block 1926 len 12288
24090 027/300 Read block 4691 len 36864
24091 028/300 Read block 915 len 16384
24092 029/300 Read block 562 len 40960
24093 030/300 Read block 1222 len 32768
24094 031/300 Write block 2531 len 36864 data: 4 3 3 3 4 5 5 6 6
24095 032/300 Read block 1848 len 28672
24096 033/300 Read block 186 len 24576
24097 034/300 Read block 4064 len 16384
24098 035/300 Read block 1202 len 12288
24099 036/300 Write block 652 len 28672 data: 4 5 5 5 4 3 3
24100 037/300 Read block 3793 len 20480
24101 038/300 Write block 4091 len 8192 data: 5 3
24102 039/300 Flush
24103 040/300 Write block 1823 len 8192 data: 5 5
24104 041/300 Write block 4728 len 28672 data: 4 4 4 4 3 3 3
24105 042/300 Read block 3476 len 4096
24106 043/300 Read block 525 len 36864
24107 044/300 Read block 3792 len 24576
24108 045/300 Read block 2068 len 16384
24109 046/300 Read block 997 len 28672
24110 047/300 Write block 2617 len 32768 data: 2 3 3 2 3 3 3 3
24111 048/300 Flush
24112 049/300 Write block 2453 len 36864 data: 5 6 7 8 10 9 10 10 11
24113 050/300 Read block 2387 len 16384
24114 051/300 Write block 729 len 24576 data: 6 6 5 5 4 5
24115 052/300 Read block 4781 len 24576
24116 053/300 Write block 713 len 12288 data: 5 4 4
24117 054/300 Flush
24118 055/300 Read block 3032 len 40960
24119 056/300 Read block 4377 len 36864
24120 057/300 Write block 4395 len 36864 data: 4 4 3 3 3 3 3 3 4
24121 058/300 Read block 4393 len 40960
24122 059/300 Write block 1659 len 36864 data: 3 2 3 3 4 3 4 5 5
24123 060/300 Read block 1055 len 4096
24124 061/300 Write block 4083 len 8192 data: 3 4
24125 062/300 Write block 3430 len 28672 data: 6 5 4 4 5 4 4
24126 063/300 Read block 4306 len 24576
24127 064/300 Read block 3284 len 16384
24128 065/300 Write block 1789 len 28672 data: 4 4 4 5 4 3 3
24129 066/300 Flush
24130 067/300 Read block 1074 len 4096
24131 068/300 Read block 3630 len 32768
24132 069/300 Write block 2827 len 28672 data: 5 5 5 6 6 6 7
24133 070/300 Read block 1466 len 16384
24134 071/300 Write block 1098 len 28672 data: 6 5 6 5 6 6 7
24135 072/300 Write block 3657 len 40960 data: 9 8 7 6 6 5 5 5 6 5
24136 073/300 Read block 65 len 8192
24137 074/300 Read block 2168 len 36864
24138 075/300 Write block 3004 len 40960 data: 6 7 7 6 7 6 6 3 3 3
24139 076/300 Read block 2730 len 28672
24140 077/300 Flush
24141 078/300 Write block 3652 len 20480 data: 7 9 10 10 11
24142 079/300 Write block 1677 len 28672 data: 4 4 3 3 3 4 4
24143 080/300 Write block 329 len 36864 data: 5 4 3 3 5 5 5 5 5
24144 081/300 Write block 3650 len 4096 data: 6
24145 082/300 Write block 476 len 40960 data: 4 4 6 5 5 5 6 6 6 6
24146 083/300 Read block 199 len 40960
24147 084/300 Read block 184 len 24576
24148 085/300 Read block 3542 len 4096
24149 086/300 Read block 924 len 20480
24150 087/300 Write block 2492 len 36864 data: 4 4 4 4 4 4 4 5 5
24151 088/300 Read block 1968 len 40960
24152 089/300 Read block 4203 len 28672
24153 090/300 Read block 555 len 20480
24154 091/300 Read block 2466 len 4096
24155 092/300 Write block 1825 len 36864 data: 4 3 3 3 4 4 6 4 2
24156 093/300 Read block 4620 len 36864
24157 094/300 Write block 297 len 20480 data: 5 5 6 6 7
24158 095/300 Write block 1395 len 28672 data: 6 5 5 5 6 6 5
24159 096/300 Flush
24160 097/300 Write block 3997 len 4096 data: 5
24161 098/300 Read block 4493 len 4096
24162 099/300 Flush
24163 100/300 Write block 3328 len 4096 data: 4
24164 101/300 Write block 1393 len 12288 data: 5 6 7
24165 102/300 Read block 180 len 36864
24166 103/300 Read block 2339 len 8192
24167 104/300 Read block 1266 len 28672
24168 105/300 Write block 529 len 20480 data: 2 2 3 4 5
24169 106/300 Write block 3921 len 20480 data: 8 6 6 5 4
24170 107/300 Read block 1946 len 8192
24171 108/300 Read block 760 len 28672
24172 109/300 Write block 4394 len 16384 data: 7 5 5 4
24173 110/300 Read block 491 len 40960
24174 111/300 Write block 2093 len 24576 data: 7 8 8 8 7 7
24175 112/300 Write block 2454 len 36864 data: 7 8 9 11 10 11 11 12 10
24176 113/300 Read block 35 len 40960
24177 114/300 Read block 2542 len 4096
24178 115/300 Read block 1923 len 28672
24179 116/300 Write block 4249 len 40960 data: 5 4 3 3 3 3 4 4 5 5
24180 117/300 Read block 490 len 16384
24181 118/300 Read block 3899 len 40960
24182 119/300 Flush
24183 120/300 Read block 1431 len 28672
24184 121/300 Write block 859 len 12288 data: 4 3 4
24185 122/300 Read block 3077 len 40960
24186 123/300 Write block 3731 len 40960 data: 8 8 7 8 7 7 6 5 5 3
24187 124/300 Read block 2356 len 40960
24188 125/300 Write block 1262 len 4096 data: 6
24189 126/300 Write block 4888 len 4096 data: 3
24190 127/300 Read block 564 len 32768
24191 128/300 Read block 619 len 40960
24192 129/300 Write block 3333 len 8192 data: 6 6
24193 130/300 Read block 3945 len 4096
24194 131/300 Write block 199 len 16384 data: 5 5 5 6
24195 132/300 Read block 3346 len 40960
24196 133/300 Read block 10 len 20480
24197 134/300 Read block 4984 len 28672
24198 135/300 Write block 70 len 8192 data: 6 7
24199 136/300 Read block 3542 len 20480
24200 137/300 Read block 904 len 16384
24201 138/300 Write block 1116 len 36864 data: 3 3 3 4 3 3 4 4 5
24202 139/300 Write block 1784 len 40960 data: 4 4 3 5 4 5 5 5 6 5
24203 140/300 Write block 3093 len 40960 data: 6 5 6 6 5 5 5 6 5 6
24204 141/300 Read block 855 len 36864
24205 142/300 Write block 1063 len 4096 data: 6
24206 143/300 Write block 183 len 32768 data: 5 5 5 4 4 5 4 4
24207 144/300 Write block 892 len 4096 data: 3
24208 145/300 Flush
24209 146/300 Read block 3077 len 4096
24210 147/300 Read block 365 len 40960
24211 148/300 Write block 290 len 4096 data: 4
24212 149/300 Write block 2320 len 4096 data: 6
24213 150/300 Read block 2462 len 32768
24214 151/300 Write block 4143 len 12288 data: 5 5 5
24215 152/300 Read block 3886 len 12288
24216 153/300 Write block 1530 len 32768 data: 5 5 5 5 5 4 3 3
24217 154/300 Write block 4208 len 32768 data: 6 6 5 5 7 6 6 4
24218 155/300 Read block 4709 len 8192
24219 156/300 Read block 4523 len 36864
24220 157/300 Flush
24221 158/300 Write block 4556 len 8192 data: 6 6
24222 159/300 Read block 4220 len 36864
24223 160/300 Read block 752 len 16384
24224 161/300 Read block 771 len 16384
24225 162/300 Read block 402 len 12288
24226 163/300 Read block 3829 len 20480
24227 164/300 Write block 3271 len 28672 data: 5 6 6 6 4 3 3
24228 165/300 Write block 1569 len 12288 data: 3 4 5
24229 166/300 Flush
24230 167/300 Read block 3727 len 4096
24231 168/300 Write block 2089 len 16384 data: 4 5 4 6
24232 169/300 Write block 4784 len 12288 data: 4 4 4
24233 170/300 Read block 4537 len 28672
24234 171/300 Read block 78 len 20480
24235 172/300 Read block 3113 len 32768
24236 173/300 Write block 3132 len 32768 data: 4 4 3 4 4 4 4 5
24237 174/300 Read block 4114 len 16384
24238 175/300 Write block 799 len 32768 data: 3 2 2 2 2 2 2 2
24239 176/300 Read block 1045 len 8192
24240 177/300 Write block 2249 len 28672 data: 5 5 5 3 3 4 6
24241 178/300 Read block 2439 len 24576
24242 179/300 Write block 838 len 20480 data: 7 8 7 6 4
24243 180/300 Write block 3697 len 32768 data: 3 5 6 7 7 7 6 7
24244 181/300 Read block 1118 len 20480
24245 182/300 Read block 1946 len 16384
24246 183/300 Write block 1787 len 28672 data: 6 5 6 6 6 7 6
24247 184/300 Write block 1320 len 12288 data: 5 5 5
24248 185/300 Read block 1188 len 28672
24249 186/300 Read block 2650 len 28672
24250 187/300 Read block 3373 len 20480
24251 188/300 Flush
24252 189/300 Read block 400 len 16384
24253 190/300 Read block 4812 len 12288
24254 191/300 Flush
24255 192/300 Read block 2239 len 32768
24256 193/300 Write block 630 len 8192 data: 11 10
24257 194/300 Write block 3636 len 12288 data: 3 5 5
24258 195/300 Read block 2110 len 8192
24259 196/300 Read block 2917 len 28672
24260 197/300 Flush
24261 198/300 Write block 1887 len 28672 data: 2 2 3 3 3 3 4
24262 199/300 Flush
24263 200/300 Read block 2294 len 28672
24264 201/300 Write block 1314 len 8192 data: 6 5
24265 202/300 Read block 3073 len 16384
24266 203/300 Read block 4472 len 8192
24267 204/300 Read block 2465 len 8192
24268 205/300 Write block 1767 len 12288 data: 6 8 7
24269 206/300 Write block 1434 len 16384 data: 3 3 4 5
24270 207/300 Write block 580 len 4096 data: 6
24271 208/300 Write block 3046 len 12288 data: 5 5 5
24272 209/300 Read block 463 len 28672
24273 210/300 Write block 1750 len 40960 data: 6 5 6 6 6 5 5 3 2 2
24274 211/300 Write block 1316 len 28672 data: 7 5 5 5 6 6 6
24275 212/300 Read block 1986 len 24576
24276 213/300 Write block 4952 len 24576 data: 4 3 4 3 4 3
24277 214/300 Read block 3217 len 32768
24278 215/300 Write block 140 len 36864 data: 5 4 4 5 4 4 4 4 3
24279 216/300 Write block 160 len 40960 data: 3 4 5 5 5 3 5 5 5 5
24280 217/300 Write block 3286 len 20480 data: 7 7 6 5 5
24281 218/300 Write block 1016 len 4096 data: 3
24282 219/300 Read block 2308 len 8192
24283 220/300 Read block 4900 len 12288
24284 221/300 Read block 2513 len 20480
24285 222/300 Read block 4256 len 24576
24286 223/300 Write block 1906 len 12288 data: 4 5 6
24287 224/300 Write block 3788 len 28672 data: 5 5 5 6 6 6 6
24288 225/300 Write block 1677 len 24576 data: 5 5 4 4 4 5
24289 226/300 Write block 4092 len 20480 data: 4 3 4 4 2
24290 227/300 Write block 3636 len 24576 data: 4 6 6 4 3 3
24291 228/300 Write block 4284 len 28672 data: 4 3 2 3 4 4 4
24292 229/300 Flush
24293 230/300 Flush
24294 231/300 Write block 3428 len 12288 data: 6 6 7
24295 232/300 Write block 4833 len 12288 data: 2 2 2
24296 233/300 Read block 205 len 40960
24297 234/300 Read block 4287 len 36864
24298 235/300 Read block 4854 len 20480
24299 236/300 Read block 3196 len 12288
24300 237/300 Flush
24301 238/300 Read block 3237 len 40960
24302 239/300 Write block 1998 len 36864 data: 5 6 6 5 5 4 5 5 5
24303 240/300 Read block 2837 len 28672
24304 241/300 Write block 2815 len 4096 data: 2
24305 242/300 Write block 1980 len 16384 data: 7 7 6 7
24306 243/300 Read block 3740 len 12288
24307 244/300 Read block 1054 len 8192
24308 245/300 Read block 4260 len 40960
24309 246/300 Write block 3286 len 16384 data: 8 8 7 6
24310 247/300 Read block 2797 len 12288
24311 248/300 Write block 221 len 20480 data: 3 3 2 3 4
24312 249/300 Flush
24313 250/300 Read block 2805 len 8192
24314 251/300 Write block 4145 len 32768 data: 6 5 5 6 6 6 3 4
24315 252/300 Read block 1803 len 4096
24316 253/300 Read block 4084 len 16384
24317 254/300 Write block 1942 len 4096 data: 4
24318 255/300 Flush
24319 256/300 Read block 1264 len 20480
24320 257/300 Write block 801 len 32768 data: 3 3 3 3 3 3 2 3
24321 258/300 Write block 203 len 36864 data: 6 5 5 5 5 4 5 6 6
24322 259/300 Read block 3114 len 36864
24323 260/300 Read block 3063 len 36864
24324 261/300 Write block 1463 len 8192 data: 5 5
24325 262/300 Read block 692 len 16384
24326 263/300 Read block 829 len 28672
24327 264/300 Write block 2006 len 36864 data: 6 5 6 8 7 7 7 7 7
24328 265/300 Flush
24329 266/300 Write block 2666 len 32768 data: 6 5 5 5 3 4 4 3
24330 267/300 Write block 482 len 36864 data: 7 7 7 7 5 4 5 7 5
24331 268/300 Read block 1196 len 4096
24332 269/300 Read block 1111 len 36864
24333 270/300 Flush
24334 271/300 Write block 2788 len 28672 data: 5 6 5 7 7 7 5
24335 272/300 Write block 2834 len 40960 data: 6 6 5 5 5 4 3 2 2 2
24336 273/300 Read block 2029 len 20480
24337 274/300 Read block 4762 len 12288
24338 275/300 Write block 4721 len 8192 data: 9 8
24339 276/300 Read block 949 len 40960
24340 277/300 Write block 1838 len 28672 data: 5 5 5 4 5 4 5
24341 278/300 Read block 2644 len 20480
24342 279/300 Write block 4173 len 28672 data: 5 3 3 5 5 5 5
24343 280/300 Read block 1120 len 12288
24344 281/300 Read block 154 len 28672
24345 282/300 Write block 4195 len 20480 data: 7 8 8 7 7
24346 283/300 Write block 1888 len 24576 data: 3 4 4 4 4 5
24347 284/300 Read block 4190 len 36864
24348 285/300 Write block 3701 len 36864 data: 8 8 7 8 7 6 5 5 6
24349 286/300 Write block 730 len 12288 data: 7 6 6
24350 287/300 Read block 2238 len 28672
24351 288/300 Read block 3343 len 20480
24352 289/300 Write block 3623 len 12288 data: 5 5 4
24353 290/300 Read block 119 len 16384
24354 291/300 Write block 208 len 12288 data: 5 6 7
24355 292/300 Flush
24356 293/300 Flush
24357 294/300 Read block 743 len 24576
24358 295/300 Read block 1055 len 24576
24359 296/300 Read block 626 len 40960
24360 297/300 Write block 4664 len 12288 data: 8 6 7
24361 298/300 Read block 1764 len 28672
24362 299/300 Write block 4657 len 20480 data: 2 4 4 4 7
24363 300/300 Write block 989 len 20480 data: 4 3 5 5 7
24364 301/300 Write block 1436 len 24576 data: 5 6 5 5 7 6
24365 Main task has work to do, go find it
24366 got dsc Start(2)
24367 start 2
24368 Feb 25 17:23:39.628 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2a1891b8-a99b-4b1a-84a1-aabce433b732, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
24369 Replay: started 2, returned:Ok(())
24370 ----------------------------------------------------------------
24371 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24372 States: Active Active Offline
24373 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24374 7341 Acked Read 2 Done Done Sent false
24375 7342 Acked Write 7 Done Done Sent false
24376 7343 Acked Read 6 Done Done Sent false
24377 7344 Acked Read 9 Done Done Sent false
24378 7345 Acked Write 10 Done Done Sent false
24379 7346 Acked Read 9 Done Done Sent false
24380 7347 Acked Read 4 Done Done Sent false
24381 7348 Acked Read 10 Done Done Sent false
24382 7349 Acked Write 7 Done Done Sent false
24383 7350 Acked Read 8 Done Done Sent false
24384 7351 Acked Flush 0 Done Done Sent false
24385 7352 Acked Read 9 Done Done Sent false
24386 7353 Acked Write 9 Done Done Sent false
24387 7354 Acked Write 4 Done Done Sent false
24388 7355 Acked Flush 0 Done Done Sent false
24389 7356 Acked Read 7 Done Done Sent false
24390 7357 Acked Read 7 Done Done Sent false
24391 7358 Acked Read 10 Done Done Sent false
24392 7359 Acked Write 4 Done Done Sent false
24393 7360 Acked Write 6 Done Done Sent false
24394 7361 Acked Write 4 Done Done Sent false
24395 7362 Acked Flush 0 Done Done Sent false
24396 7363 Acked Write 5 Done Done Sent false
24397 7364 Acked Write 8 Done Done Sent false
24398 7365 Acked Write 2 Done Done Sent false
24399 7366 Acked Read 3 Done Done Sent false
24400 7367 Acked Read 9 Done Done Sent false
24401 7368 Acked Read 4 Done Done Sent false
24402 7369 Acked Read 10 Done Done Sent false
24403 7370 Acked Read 8 Done Done Sent false
24404 7371 Acked Write 9 Done Done Sent false
24405 7372 Acked Read 7 Done Done Sent false
24406 7373 Acked Read 6 Done Done Sent false
24407 7374 Acked Read 4 Done Done Sent false
24408 7375 Acked Read 3 Done Done Sent false
24409 7376 Acked Write 7 Done Done Sent false
24410 7377 Acked Read 5 Done Done Sent false
24411 7378 Acked Write 2 Done Done Sent false
24412 7379 Acked Flush 0 Done Done Sent false
24413 7380 Acked Write 2 Done Done Sent false
24414 7381 Acked Write 7 Done Done Sent false
24415 7382 Acked Read 1 Done Done Sent false
24416 7383 Acked Read 9 Done Done Sent false
24417 7384 Acked Read 6 Done Done Sent false
24418 7385 Acked Read 4 Done Done Sent false
24419 7386 Acked Read 7 Done Done Sent false
24420 7387 Acked Write 8 Done Done Sent false
24421 7388 Acked Flush 0 Done Done Sent false
24422 7389 Acked Write 9 Done Done Sent false
24423 7390 Acked Read 4 Done Done Sent false
24424 7391 Acked Write 6 Done Done Sent false
24425 7392 Acked Read 6 Done Done Sent false
24426 7393 Acked Write 3 Done Done Sent false
24427 7394 Acked Flush 0 Done Done Sent false
24428 7395 Acked Read 10 Done Done Sent false
24429 7396 Acked Read 9 Done Done Sent false
24430 7397 Acked Write 9 Done Done Sent false
24431 7398 Acked Read 10 Done Done Sent false
24432 7399 Acked Write 9 Done Done Sent false
24433 7400 Acked Read 1 Done Done Sent false
24434 7401 Acked Write 2 Done Done Sent false
24435 7402 Acked Write 7 Done Done Sent false
24436 7403 Acked Read 6 Done Done Sent false
24437 7404 Acked Read 4 Done Done Sent false
24438 7405 Acked Write 7 Done Done Sent false
24439 7406 Acked Flush 0 Done Done Sent false
24440 7407 Acked Read 1 Done Done Sent false
24441 7408 Acked Read 8 Done Done Sent false
24442 7409 Acked Write 7 Done Done Sent false
24443 7410 Acked Read 4 Done Done Sent false
24444 7411 Acked Write 7 Done Done Sent false
24445 7412 Acked Write 10 Done Done Sent false
24446 7413 Acked Read 2 Done Done Sent false
24447 7414 Acked Read 9 Done Done Sent false
24448 7415 Acked Write 10 Done Done Sent false
24449 7416 Acked Read 7 Done Done Sent false
24450 7417 Acked Flush 0 Done Done Sent false
24451 7418 Acked Write 5 Done Done Sent false
24452 7419 Acked Write 7 Done Done Sent false
24453 7420 Acked Write 9 Done Done Sent false
24454 7421 Acked Write 1 Done Done Sent false
24455 7422 Acked Write 10 Done Done Sent false
24456 7423 Acked Read 10 Done Done Sent false
24457 7424 Acked Read 6 Done Done Sent false
24458 7425 Acked Read 1 Done Done Sent false
24459 7426 Acked Read 5 Done Done Sent false
24460 7427 Acked Write 9 Done Done Sent false
24461 7428 Acked Read 10 Done Done Sent false
24462 7429 Acked Read 7 Done Done Sent false
24463 7430 Acked Read 5 Done Done Sent false
24464 7431 Acked Read 1 Done Done Sent false
24465 7432 Acked Write 9 Done Done Sent false
24466 7433 Acked Read 9 Done Done Sent false
24467 7434 Acked Write 5 Done Done Sent false
24468 7435 Acked Write 7 Done Done Sent false
24469 7436 Acked Flush 0 Done Done Sent false
24470 7437 Acked Write 1 Done Done Sent false
24471 7438 Acked Read 1 Done Done Sent false
24472 7439 Acked Flush 0 Done Done Sent false
24473 7440 Acked Write 1 Done Done Sent false
24474 7441 Acked Write 3 Done Done Sent false
24475 7442 Acked Read 9 Done Done Sent false
24476 7443 Acked Read 2 Done Done Sent false
24477 7444 Acked Read 7 Done Done Sent false
24478 7445 Acked Write 5 Done Done Sent false
24479 7446 Acked Write 5 Done Done Sent false
24480 7447 Acked Read 2 Done Done Sent false
24481 7448 Acked Read 7 Done Done Sent false
24482 7449 Acked Write 4 Done Done Sent false
24483 7450 Acked Read 10 Done Done Sent false
24484 7451 Acked Write 6 Done Done Sent false
24485 7452 Acked Write 9 Done Done Sent false
24486 7453 Acked Read 10 Done Done Sent false
24487 7454 Acked Read 1 Done Done Sent false
24488 7455 Acked Read 7 Done Done Sent false
24489 7456 Acked Write 10 Done Done Sent false
24490 7457 Acked Read 4 Done Done Sent false
24491 7458 Acked Read 10 Done Done Sent false
24492 7459 Acked Flush 0 Done Done Sent false
24493 7460 Acked Read 7 Done Done Sent false
24494 7461 Acked Write 3 Done Done Sent false
24495 7462 Acked Read 10 Done Done Sent false
24496 7463 Acked Write 10 Done Done Sent false
24497 7464 Acked Read 10 Done Done Sent false
24498 7465 Acked Write 1 Done Done Sent false
24499 7466 Acked Write 1 Done Done Sent false
24500 7467 Acked Read 8 Done Done Sent false
24501 7468 Acked Read 10 Done Done Sent false
24502 7469 Acked Write 2 Done Done Sent false
24503 7470 Acked Read 1 Done Done Sent false
24504 7471 Acked Write 4 Done Done Sent false
24505 7472 Acked Read 10 Done Done Sent false
24506 7473 Acked Read 5 Done Done Sent false
24507 7474 Acked Read 7 Done Done Sent false
24508 7475 Acked Write 2 Done Done Sent false
24509 7476 Acked Read 5 Done Done Sent false
24510 7477 Acked Read 4 Done Done Sent false
24511 7478 Acked Write 9 Done Done Sent false
24512 7479 Acked Write 10 Done Done Sent false
24513 7480 Acked Write 10 Done Done Sent false
24514 7481 Acked Read 9 Done Done Sent false
24515 7482 Acked Write 1 Done Done Sent false
24516 7483 Acked Write 8 Done Done Sent false
24517 7484 Acked Write 1 Done Done Sent false
24518 7485 Acked Flush 0 Done Done Sent false
24519 7486 Acked Read 1 Done Done Sent false
24520 7487 Acked Read 10 Done Done Sent false
24521 7488 Acked Write 1 Done Done Sent false
24522 7489 Acked Write 1 Done Done Sent false
24523 7490 Acked Read 8 Done Done Sent false
24524 7491 Acked Write 3 Done Done Sent false
24525 7492 Acked Read 3 Done Done Sent false
24526 7493 Acked Write 8 Done Done Sent false
24527 7494 Acked Write 8 Done Done Sent false
24528 7495 Acked Read 2 Done Done Sent false
24529 7496 Acked Read 9 Done Done Sent false
24530 7497 Acked Flush 0 Done Done Sent false
24531 7498 Acked Write 2 Done Done Sent false
24532 7499 Acked Read 9 Done Done Sent false
24533 7500 Acked Read 4 Done Done Sent false
24534 7501 Acked Read 4 Done Done Sent false
24535 7502 Acked Read 3 Done Done Sent false
24536 7503 Acked Read 5 Done Done Sent false
24537 7504 Acked Write 7 Done Done Sent false
24538 7505 Acked Write 3 Done Done Sent false
24539 7506 Acked Flush 0 Done Done Sent false
24540 7507 Acked Read 1 Done Done Sent false
24541 7508 Acked Write 4 Done Done Sent false
24542 7509 Acked Write 3 Done Done Sent false
24543 7510 Acked Read 7 Done Done Sent false
24544 7511 Acked Read 5 Done Done Sent false
24545 7512 Acked Read 8 Done Done Sent false
24546 7513 Acked Write 8 Done Done Sent false
24547 7514 Acked Read 4 Done Done Sent false
24548 7515 Acked Write 8 Done Done Sent false
24549 7516 Acked Read 2 Done Done Sent false
24550 7517 Acked Write 7 Done Done Sent false
24551 7518 Acked Read 6 Done Done Sent false
24552 7519 Acked Write 5 Done Done Sent false
24553 7520 Acked Write 8 Done Done Sent false
24554 7521 Acked Read 5 Done Done Sent false
24555 7522 Acked Read 4 Done Done Sent false
24556 7523 Acked Write 7 Done Done Sent false
24557 7524 Acked Write 3 Done Done Sent false
24558 7525 Acked Read 7 Done Done Sent false
24559 7526 Acked Read 7 Done Done Sent false
24560 7527 Acked Read 5 Done Done Sent false
24561 7528 Acked Flush 0 Done Done Sent false
24562 7529 Acked Read 4 Done Done Sent false
24563 7530 Acked Read 3 Done Done Sent false
24564 7531 Acked Flush 0 Done Done Sent false
24565 7532 Acked Read 8 Done Done Sent false
24566 7533 Acked Write 2 Done Done Sent false
24567 7534 Acked Write 3 Done Done Sent false
24568 7535 Acked Read 2 Done Done Sent false
24569 7536 Acked Read 7 Done Done Sent false
24570 7537 Acked Flush 0 Done Done Sent false
24571 7538 Acked Write 7 Done Done Sent false
24572 7539 Acked Flush 0 Done Done Sent false
24573 7540 Acked Read 7 Done Done Sent false
24574 7541 Acked Write 2 Done Done Sent false
24575 7542 Acked Read 4 Done Done Sent false
24576 7543 Acked Read 2 Done Done Sent false
24577 7544 Acked Read 2 Done Done Sent false
24578 7545 Acked Write 3 Done Done Sent false
24579 7546 Acked Write 4 Done Done Sent false
24580 7547 Acked Write 1 Done Done Sent false
24581 7548 Acked Write 3 Done Done Sent false
24582 7549 Acked Read 7 Done Done Sent false
24583 7550 Acked Write 10 Done Done Sent false
24584 7551 Acked Write 7 Done Done Sent false
24585 7552 Acked Read 6 Done Done Sent false
24586 7553 Acked Write 6 Done Done Sent false
24587 7554 Acked Read 8 Done Done Sent false
24588 7555 Acked Write 9 Done Done Sent false
24589 7556 Acked Write 10 Done Done Sent false
24590 7557 Acked Write 5 Done Done Sent false
24591 7558 Acked Write 1 Done Done Sent false
24592 7559 Acked Read 2 Done Done Sent false
24593 7560 Acked Read 3 Done Done Sent false
24594 7561 Acked Read 5 Done Done Sent false
24595 7562 Acked Read 6 Done Done Sent false
24596 7563 Acked Write 3 Done Done Sent false
24597 7564 Acked Write 7 Done Done Sent false
24598 7565 Acked Write 6 Done Done Sent false
24599 7566 Acked Write 5 Done Done Sent false
24600 7567 Acked Write 6 Done Done Sent false
24601 7568 Acked Write 7 Done Done Sent false
24602 7569 Acked Flush 0 Done Done Sent false
24603 7570 Acked Flush 0 Done Done Sent false
24604 7571 Acked Write 3 Done Done Sent false
24605 7572 Acked Write 3 Done Done Sent false
24606 7573 Acked Read 10 Done Done Sent false
24607 7574 Acked Read 9 Done Done Sent false
24608 7575 Acked Read 5 Done Done Sent false
24609 7576 Acked Read 3 Done Done Sent false
24610 7577 Acked Flush 0 Done Done Sent false
24611 7578 Acked Read 10 Done Done Sent false
24612 7579 Acked Write 9 Done Done Sent false
24613 7580 Acked Read 7 Done Done Sent false
24614 7581 Acked Write 1 Done Done Sent false
24615 7582 Acked Write 4 Done Done Sent false
24616 7583 Acked Read 3 Done Done Sent false
24617 7584 Acked Read 2 Done Done Sent false
24618 7585 Acked Read 10 Done Done Sent false
24619 7586 Acked Write 4 Done Done Sent false
24620 7587 Acked Read 3 Done Done Sent false
24621 7588 Acked Write 5 Done Done Sent false
24622 7589 Acked Flush 0 Done Done Sent false
24623 7590 Acked Read 2 Done Done Sent false
24624 7591 Acked Write 8 Done Done Sent false
24625 7592 Acked Read 1 Done Done Sent false
24626 7593 Acked Read 4 Done Done Sent false
24627 7594 Acked Write 1 Done Done Sent false
24628 7595 Acked Flush 0 Done Done Sent false
24629 7596 Acked Read 5 Done Done Sent false
24630 7597 Acked Write 8 Done Done Sent false
24631 7598 Acked Write 9 Done Done Sent false
24632 7599 Acked Read 9 Done Done Sent false
24633 7600 Acked Read 9 Done Done Sent false
24634 7601 Acked Write 2 Done Done Sent false
24635 7602 Acked Read 4 Done Done Sent false
24636 7603 Acked Read 7 Done Done Sent false
24637 7604 Acked Write 9 Done Done Sent false
24638 7605 Acked Flush 0 Done Done Sent false
24639 7606 Acked Write 8 Done Done Sent false
24640 7607 Acked Write 9 Done Done Sent false
24641 7608 Acked Read 1 Done Done Sent false
24642 7609 Acked Read 9 Done Done Sent false
24643 7610 Acked Flush 0 Done Done Sent false
24644 7611 Acked Write 7 Done Done Sent false
24645 7612 Acked Write 10 Done Done Sent false
24646 7613 Acked Read 5 Done Done Sent false
24647 7614 Acked Read 3 Done Done Sent false
24648 7615 Acked Write 2 Done Done Sent false
24649 7616 Acked Read 10 Done Done Sent false
24650 7617 Acked Write 7 Done Done Sent false
24651 7618 Acked Read 5 Done Done Sent false
24652 7619 Acked Write 7 Done Done Sent false
24653 7620 Acked Read 3 Done Done Sent false
24654 7621 Acked Read 7 Done Done Sent false
24655 7622 Acked Write 5 Done Done Sent false
24656 7623 Acked Write 6 Done Done Sent false
24657 7624 Acked Read 9 Done Done Sent false
24658 7625 Acked Write 9 Done Done Sent false
24659 7626 Acked Write 3 Done Done Sent false
24660 7627 Acked Read 7 Done Done Sent false
24661 7628 Acked Read 5 Done Done Sent false
24662 7629 Acked Write 3 Done Done Sent false
24663 7630 Acked Read 4 Done Done Sent false
24664 7631 Acked Write 3 Done Done Sent false
24665 7632 Acked Flush 0 Done Done Sent false
24666 7633 Acked Flush 0 Done Done Sent false
24667 7634 Acked Read 6 Done Done Sent false
24668 7635 Acked Read 6 Done Done Sent false
24669 7636 Acked Read 10 Done Done Sent false
24670 7637 Acked Write 3 Done Done Sent false
24671 7638 Acked Read 7 Done Done Sent false
24672 7639 Acked Write 5 Done Done Sent false
24673 7640 Acked Write 5 Done Done Sent false
24674 7641 Acked Write 6 Done Sent Sent false
24675 STATES DS:0 DS:1 DS:2 TOTAL
24676 Sent 0 1 301 302
24677 Done 301 300 0 601
24678 Skipped 0 0 0 0
24679 Error 0 0 0 0
24680 Last Flush: Some(JobId(7633)) Some(JobId(7633)) Some(JobId(7340))
24681 Downstairs last five completed: 7340 7339 7338 7337 7336
24682 Upstairs last five completed: 7641 7640 7639 7638 7637
24683 CLIENT: Up:0 ds:301 act:2
246842025-02-25T17:23:40.131ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
246852025-02-25T17:23:40.785ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
24686 [8830] got start action
24687 [8830] I am going to restart
24688 Starting downstairs at port 8830
24689 Make output file at "/tmp/dsc/downstairs-8830.txt"
24690 [8830][2] reports Starting
24691 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1230)
24692 [8830][2] reports Running
246932025-02-25T17:23:42.095ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
246942025-02-25T17:23:42.788ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
24695 ----------------------------------------------------------------
24696 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24697 States: Active Active Offline
24698 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24699 7341 Acked Read 2 Done Done Sent false
24700 7342 Acked Write 7 Done Done Sent false
24701 7343 Acked Read 6 Done Done Sent false
24702 7344 Acked Read 9 Done Done Sent false
24703 7345 Acked Write 10 Done Done Sent false
24704 7346 Acked Read 9 Done Done Sent false
24705 7347 Acked Read 4 Done Done Sent false
24706 7348 Acked Read 10 Done Done Sent false
24707 7349 Acked Write 7 Done Done Sent false
24708 7350 Acked Read 8 Done Done Sent false
24709 7351 Acked Flush 0 Done Done Sent false
24710 7352 Acked Read 9 Done Done Sent false
24711 7353 Acked Write 9 Done Done Sent false
24712 7354 Acked Write 4 Done Done Sent false
24713 7355 Acked Flush 0 Done Done Sent false
24714 7356 Acked Read 7 Done Done Sent false
24715 7357 Acked Read 7 Done Done Sent false
24716 7358 Acked Read 10 Done Done Sent false
24717 7359 Acked Write 4 Done Done Sent false
24718 7360 Acked Write 6 Done Done Sent false
24719 7361 Acked Write 4 Done Done Sent false
24720 7362 Acked Flush 0 Done Done Sent false
24721 7363 Acked Write 5 Done Done Sent false
24722 7364 Acked Write 8 Done Done Sent false
24723 7365 Acked Write 2 Done Done Sent false
24724 7366 Acked Read 3 Done Done Sent false
24725 7367 Acked Read 9 Done Done Sent false
24726 7368 Acked Read 4 Done Done Sent false
24727 7369 Acked Read 10 Done Done Sent false
24728 7370 Acked Read 8 Done Done Sent false
24729 7371 Acked Write 9 Done Done Sent false
24730 7372 Acked Read 7 Done Done Sent false
24731 7373 Acked Read 6 Done Done Sent false
24732 7374 Acked Read 4 Done Done Sent false
24733 7375 Acked Read 3 Done Done Sent false
24734 7376 Acked Write 7 Done Done Sent false
24735 7377 Acked Read 5 Done Done Sent false
24736 7378 Acked Write 2 Done Done Sent false
24737 7379 Acked Flush 0 Done Done Sent false
24738 7380 Acked Write 2 Done Done Sent false
24739 7381 Acked Write 7 Done Done Sent false
24740 7382 Acked Read 1 Done Done Sent false
24741 7383 Acked Read 9 Done Done Sent false
24742 7384 Acked Read 6 Done Done Sent false
24743 7385 Acked Read 4 Done Done Sent false
24744 7386 Acked Read 7 Done Done Sent false
24745 7387 Acked Write 8 Done Done Sent false
24746 7388 Acked Flush 0 Done Done Sent false
24747 7389 Acked Write 9 Done Done Sent false
24748 7390 Acked Read 4 Done Done Sent false
24749 7391 Acked Write 6 Done Done Sent false
24750 7392 Acked Read 6 Done Done Sent false
24751 7393 Acked Write 3 Done Done Sent false
24752 7394 Acked Flush 0 Done Done Sent false
24753 7395 Acked Read 10 Done Done Sent false
24754 7396 Acked Read 9 Done Done Sent false
24755 7397 Acked Write 9 Done Done Sent false
24756 7398 Acked Read 10 Done Done Sent false
24757 7399 Acked Write 9 Done Done Sent false
24758 7400 Acked Read 1 Done Done Sent false
24759 7401 Acked Write 2 Done Done Sent false
24760 7402 Acked Write 7 Done Done Sent false
24761 7403 Acked Read 6 Done Done Sent false
24762 7404 Acked Read 4 Done Done Sent false
24763 7405 Acked Write 7 Done Done Sent false
24764 7406 Acked Flush 0 Done Done Sent false
24765 7407 Acked Read 1 Done Done Sent false
24766 7408 Acked Read 8 Done Done Sent false
24767 7409 Acked Write 7 Done Done Sent false
24768 7410 Acked Read 4 Done Done Sent false
24769 7411 Acked Write 7 Done Done Sent false
24770 7412 Acked Write 10 Done Done Sent false
24771 7413 Acked Read 2 Done Done Sent false
24772 7414 Acked Read 9 Done Done Sent false
24773 7415 Acked Write 10 Done Done Sent false
24774 7416 Acked Read 7 Done Done Sent false
24775 7417 Acked Flush 0 Done Done Sent false
24776 7418 Acked Write 5 Done Done Sent false
24777 7419 Acked Write 7 Done Done Sent false
24778 7420 Acked Write 9 Done Done Sent false
24779 7421 Acked Write 1 Done Done Sent false
24780 7422 Acked Write 10 Done Done Sent false
24781 7423 Acked Read 10 Done Done Sent false
24782 7424 Acked Read 6 Done Done Sent false
24783 7425 Acked Read 1 Done Done Sent false
24784 7426 Acked Read 5 Done Done Sent false
24785 7427 Acked Write 9 Done Done Sent false
24786 7428 Acked Read 10 Done Done Sent false
24787 7429 Acked Read 7 Done Done Sent false
24788 7430 Acked Read 5 Done Done Sent false
24789 7431 Acked Read 1 Done Done Sent false
24790 7432 Acked Write 9 Done Done Sent false
24791 7433 Acked Read 9 Done Done Sent false
24792 7434 Acked Write 5 Done Done Sent false
24793 7435 Acked Write 7 Done Done Sent false
24794 7436 Acked Flush 0 Done Done Sent false
24795 7437 Acked Write 1 Done Done Sent false
24796 7438 Acked Read 1 Done Done Sent false
24797 7439 Acked Flush 0 Done Done Sent false
24798 7440 Acked Write 1 Done Done Sent false
24799 7441 Acked Write 3 Done Done Sent false
24800 7442 Acked Read 9 Done Done Sent false
24801 7443 Acked Read 2 Done Done Sent false
24802 7444 Acked Read 7 Done Done Sent false
24803 7445 Acked Write 5 Done Done Sent false
24804 7446 Acked Write 5 Done Done Sent false
24805 7447 Acked Read 2 Done Done Sent false
24806 7448 Acked Read 7 Done Done Sent false
24807 7449 Acked Write 4 Done Done Sent false
24808 7450 Acked Read 10 Done Done Sent false
24809 7451 Acked Write 6 Done Done Sent false
24810 7452 Acked Write 9 Done Done Sent false
24811 7453 Acked Read 10 Done Done Sent false
24812 7454 Acked Read 1 Done Done Sent false
24813 7455 Acked Read 7 Done Done Sent false
24814 7456 Acked Write 10 Done Done Sent false
24815 7457 Acked Read 4 Done Done Sent false
24816 7458 Acked Read 10 Done Done Sent false
24817 7459 Acked Flush 0 Done Done Sent false
24818 7460 Acked Read 7 Done Done Sent false
24819 7461 Acked Write 3 Done Done Sent false
24820 7462 Acked Read 10 Done Done Sent false
24821 7463 Acked Write 10 Done Done Sent false
24822 7464 Acked Read 10 Done Done Sent false
24823 7465 Acked Write 1 Done Done Sent false
24824 7466 Acked Write 1 Done Done Sent false
24825 7467 Acked Read 8 Done Done Sent false
24826 7468 Acked Read 10 Done Done Sent false
24827 7469 Acked Write 2 Done Done Sent false
24828 7470 Acked Read 1 Done Done Sent false
24829 7471 Acked Write 4 Done Done Sent false
24830 7472 Acked Read 10 Done Done Sent false
24831 7473 Acked Read 5 Done Done Sent false
24832 7474 Acked Read 7 Done Done Sent false
24833 7475 Acked Write 2 Done Done Sent false
24834 7476 Acked Read 5 Done Done Sent false
24835 7477 Acked Read 4 Done Done Sent false
24836 7478 Acked Write 9 Done Done Sent false
24837 7479 Acked Write 10 Done Done Sent false
24838 7480 Acked Write 10 Done Done Sent false
24839 7481 Acked Read 9 Done Done Sent false
24840 7482 Acked Write 1 Done Done Sent false
24841 7483 Acked Write 8 Done Done Sent false
24842 7484 Acked Write 1 Done Done Sent false
24843 7485 Acked Flush 0 Done Done Sent false
24844 7486 Acked Read 1 Done Done Sent false
24845 7487 Acked Read 10 Done Done Sent false
24846 7488 Acked Write 1 Done Done Sent false
24847 7489 Acked Write 1 Done Done Sent false
24848 7490 Acked Read 8 Done Done Sent false
24849 7491 Acked Write 3 Done Done Sent false
24850 7492 Acked Read 3 Done Done Sent false
24851 7493 Acked Write 8 Done Done Sent false
24852 7494 Acked Write 8 Done Done Sent false
24853 7495 Acked Read 2 Done Done Sent false
24854 7496 Acked Read 9 Done Done Sent false
24855 7497 Acked Flush 0 Done Done Sent false
24856 7498 Acked Write 2 Done Done Sent false
24857 7499 Acked Read 9 Done Done Sent false
24858 7500 Acked Read 4 Done Done Sent false
24859 7501 Acked Read 4 Done Done Sent false
24860 7502 Acked Read 3 Done Done Sent false
24861 7503 Acked Read 5 Done Done Sent false
24862 7504 Acked Write 7 Done Done Sent false
24863 7505 Acked Write 3 Done Done Sent false
24864 7506 Acked Flush 0 Done Done Sent false
24865 7507 Acked Read 1 Done Done Sent false
24866 7508 Acked Write 4 Done Done Sent false
24867 7509 Acked Write 3 Done Done Sent false
24868 7510 Acked Read 7 Done Done Sent false
24869 7511 Acked Read 5 Done Done Sent false
24870 7512 Acked Read 8 Done Done Sent false
24871 7513 Acked Write 8 Done Done Sent false
24872 7514 Acked Read 4 Done Done Sent false
24873 7515 Acked Write 8 Done Done Sent false
24874 7516 Acked Read 2 Done Done Sent false
24875 7517 Acked Write 7 Done Done Sent false
24876 7518 Acked Read 6 Done Done Sent false
24877 7519 Acked Write 5 Done Done Sent false
24878 7520 Acked Write 8 Done Done Sent false
24879 7521 Acked Read 5 Done Done Sent false
24880 7522 Acked Read 4 Done Done Sent false
24881 7523 Acked Write 7 Done Done Sent false
24882 7524 Acked Write 3 Done Done Sent false
24883 7525 Acked Read 7 Done Done Sent false
24884 7526 Acked Read 7 Done Done Sent false
24885 7527 Acked Read 5 Done Done Sent false
24886 7528 Acked Flush 0 Done Done Sent false
24887 7529 Acked Read 4 Done Done Sent false
24888 7530 Acked Read 3 Done Done Sent false
24889 7531 Acked Flush 0 Done Done Sent false
24890 7532 Acked Read 8 Done Done Sent false
24891 7533 Acked Write 2 Done Done Sent false
24892 7534 Acked Write 3 Done Done Sent false
24893 7535 Acked Read 2 Done Done Sent false
24894 7536 Acked Read 7 Done Done Sent false
24895 7537 Acked Flush 0 Done Done Sent false
24896 7538 Acked Write 7 Done Done Sent false
24897 7539 Acked Flush 0 Done Done Sent false
24898 7540 Acked Read 7 Done Done Sent false
24899 7541 Acked Write 2 Done Done Sent false
24900 7542 Acked Read 4 Done Done Sent false
24901 7543 Acked Read 2 Done Done Sent false
24902 7544 Acked Read 2 Done Done Sent false
24903 7545 Acked Write 3 Done Done Sent false
24904 7546 Acked Write 4 Done Done Sent false
24905 7547 Acked Write 1 Done Done Sent false
24906 7548 Acked Write 3 Done Done Sent false
24907 7549 Acked Read 7 Done Done Sent false
24908 7550 Acked Write 10 Done Done Sent false
24909 7551 Acked Write 7 Done Done Sent false
24910 7552 Acked Read 6 Done Done Sent false
24911 7553 Acked Write 6 Done Done Sent false
24912 7554 Acked Read 8 Done Done Sent false
24913 7555 Acked Write 9 Done Done Sent false
24914 7556 Acked Write 10 Done Done Sent false
24915 7557 Acked Write 5 Done Done Sent false
24916 7558 Acked Write 1 Done Done Sent false
24917 7559 Acked Read 2 Done Done Sent false
24918 7560 Acked Read 3 Done Done Sent false
24919 7561 Acked Read 5 Done Done Sent false
24920 7562 Acked Read 6 Done Done Sent false
24921 7563 Acked Write 3 Done Done Sent false
24922 7564 Acked Write 7 Done Done Sent false
24923 7565 Acked Write 6 Done Done Sent false
24924 7566 Acked Write 5 Done Done Sent false
24925 7567 Acked Write 6 Done Done Sent false
24926 7568 Acked Write 7 Done Done Sent false
24927 7569 Acked Flush 0 Done Done Sent false
24928 7570 Acked Flush 0 Done Done Sent false
24929 7571 Acked Write 3 Done Done Sent false
24930 7572 Acked Write 3 Done Done Sent false
24931 7573 Acked Read 10 Done Done Sent false
24932 7574 Acked Read 9 Done Done Sent false
24933 7575 Acked Read 5 Done Done Sent false
24934 7576 Acked Read 3 Done Done Sent false
24935 7577 Acked Flush 0 Done Done Sent false
24936 7578 Acked Read 10 Done Done Sent false
24937 7579 Acked Write 9 Done Done Sent false
24938 7580 Acked Read 7 Done Done Sent false
24939 7581 Acked Write 1 Done Done Sent false
24940 7582 Acked Write 4 Done Done Sent false
24941 7583 Acked Read 3 Done Done Sent false
24942 7584 Acked Read 2 Done Done Sent false
24943 7585 Acked Read 10 Done Done Sent false
24944 7586 Acked Write 4 Done Done Sent false
24945 7587 Acked Read 3 Done Done Sent false
24946 7588 Acked Write 5 Done Done Sent false
24947 7589 Acked Flush 0 Done Done Sent false
24948 7590 Acked Read 2 Done Done Sent false
24949 7591 Acked Write 8 Done Done Sent false
24950 7592 Acked Read 1 Done Done Sent false
24951 7593 Acked Read 4 Done Done Sent false
24952 7594 Acked Write 1 Done Done Sent false
24953 7595 Acked Flush 0 Done Done Sent false
24954 7596 Acked Read 5 Done Done Sent false
24955 7597 Acked Write 8 Done Done Sent false
24956 7598 Acked Write 9 Done Done Sent false
24957 7599 Acked Read 9 Done Done Sent false
24958 7600 Acked Read 9 Done Done Sent false
24959 7601 Acked Write 2 Done Done Sent false
24960 7602 Acked Read 4 Done Done Sent false
24961 7603 Acked Read 7 Done Done Sent false
24962 7604 Acked Write 9 Done Done Sent false
24963 7605 Acked Flush 0 Done Done Sent false
24964 7606 Acked Write 8 Done Done Sent false
24965 7607 Acked Write 9 Done Done Sent false
24966 7608 Acked Read 1 Done Done Sent false
24967 7609 Acked Read 9 Done Done Sent false
24968 7610 Acked Flush 0 Done Done Sent false
24969 7611 Acked Write 7 Done Done Sent false
24970 7612 Acked Write 10 Done Done Sent false
24971 7613 Acked Read 5 Done Done Sent false
24972 7614 Acked Read 3 Done Done Sent false
24973 7615 Acked Write 2 Done Done Sent false
24974 7616 Acked Read 10 Done Done Sent false
24975 7617 Acked Write 7 Done Done Sent false
24976 7618 Acked Read 5 Done Done Sent false
24977 7619 Acked Write 7 Done Done Sent false
24978 7620 Acked Read 3 Done Done Sent false
24979 7621 Acked Read 7 Done Done Sent false
24980 7622 Acked Write 5 Done Done Sent false
24981 7623 Acked Write 6 Done Done Sent false
24982 7624 Acked Read 9 Done Done Sent false
24983 7625 Acked Write 9 Done Done Sent false
24984 7626 Acked Write 3 Done Done Sent false
24985 7627 Acked Read 7 Done Done Sent false
24986 7628 Acked Read 5 Done Done Sent false
24987 7629 Acked Write 3 Done Done Sent false
24988 7630 Acked Read 4 Done Done Sent false
24989 7631 Acked Write 3 Done Done Sent false
24990 7632 Acked Flush 0 Done Done Sent false
24991 7633 Acked Flush 0 Done Done Sent false
24992 7634 Acked Read 6 Done Done Sent false
24993 7635 Acked Read 6 Done Done Sent false
24994 7636 Acked Read 10 Done Done Sent false
24995 7637 Acked Write 3 Done Done Sent false
24996 7638 Acked Read 7 Done Done Sent false
24997 7639 Acked Write 5 Done Done Sent false
24998 7640 Acked Write 5 Done Done Sent false
24999 7641 Acked Write 6 Done Done Sent false
25000 STATES DS:0 DS:1 DS:2 TOTAL
25001 Sent 0 0 301 301
25002 Done 301 301 0 602
25003 Skipped 0 0 0 0
25004 Error 0 0 0 0
25005 Last Flush: Some(JobId(7633)) Some(JobId(7633)) Some(JobId(7340))
25006 Downstairs last five completed: 7340 7339 7338 7337 7336
25007 Upstairs last five completed: 7641 7640 7639 7638 7637
25008 CLIENT: Up:0 ds:301 act:2
250092025-02-25T17:23:44.132ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
250102025-02-25T17:23:44.792ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
250112025-02-25T17:23:45.366ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
250122025-02-25T17:23:45.366ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39603) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
250132025-02-25T17:23:45.367ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
250142025-02-25T17:23:45.368ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
250152025-02-25T17:23:45.368ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
250162025-02-25T17:23:45.368ZINFOcrucible: send last flush ID to this DS: Some(JobId(7340)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
250172025-02-25T17:23:45.368ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
250182025-02-25T17:23:45.368ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7340)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
250192025-02-25T17:23:45.368ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7340)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
25020 ----------------------------------------------------------------
25021 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25022 Downstairs last five completed: 7642 7641 7640 7639 7638
25023 Upstairs last five completed: 7642 7641 7640 7639 7638
25024 CLIENT: Up:0 ds:0 act:3
25025 Replay: All jobs finished
25026 Main task has work to do, go find it
25027 got dsc Stop(0)
25028 stop 0
25029 [8810] Got stop action so:false kr:false
25030 Feb 25 17:23:47.633 INFO request completed, latency_us: 377, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 9cf8487e-8d92-4c4c-921d-1be9904aebea, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
25031 Feb 25 17:23:47.634 INFO request completed{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:23:47.634656393Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
25032 , latency_us: 144, response_code: 200, uri: /state/cid/0, method: GET, req_id: 766c83e2-4590-42bc-a5c9-d367af37722a, remote_addr: 127.0.0.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:23:47.634722284Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
25033 1:52576, local_addr: 127.0.0.1:9998
250342025-02-25T17:23:47.634ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
250352025-02-25T17:23:47.634ZINFOcrucible: client task is exiting = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
25036 [8810] Exited with: ExitStatus(unix_wait_status(9))
25037 [8810][0] reports Exit
25038 Feb 25 17:23:51.636 INFO request completed, latency_us: 227, response_code: 200, uri: /state/cid/0, method: GET, req_id: c7b228d6-0457-4f68-95ea-eb80c7fb9461, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
25039 001/300 Flush
25040 002/300 Flush
25041 003/300 Flush
25042 004/300 Write block 2933 len 12288 data: 4 3 3
25043 005/300 Flush
25044 006/300 Flush
25045 007/300 Write block 2943 len 24576 data: 7 6 6 7 6 6
25046 008/300 Read block 2497 len 32768
25047 009/300 Read block 205 len 4096
25048 010/300 Flush
25049 011/300 Read block 4283 len 12288
25050 012/300 Read block 208 len 24576
25051 013/300 Read block 3427 len 24576
25052 014/300 Read block 176 len 20480
25053 015/300 Flush
25054 016/300 Read block 1920 len 24576
25055 017/300 Write block 72 len 28672 data: 7 7 8 7 4 2 2
25056 018/300 Write block 1019 len 32768 data: 4 5 5 6 6 6 7 5
25057 019/300 Write block 2303 len 12288 data: 8 8 6
25058 020/300 Read block 3847 len 4096
25059 021/300 Write block 847 len 12288 data: 2 2 3
25060 022/300 Read block 2547 len 24576
25061 023/300 Read block 523 len 12288
25062 024/300 Write block 2943 len 40960 data: 8 7 7 8 7 7 6 6 7 8
25063 025/300 Write block 2990 len 16384 data: 5 5 6 6
25064 026/300 Read block 2499 len 4096
25065 027/300 Write block 3010 len 32768 data: 7 4 4 4 3 4 4 5
25066 028/300 Read block 3237 len 28672
25067 029/300 Write block 1568 len 28672 data: 2 4 5 6 4 6 4
25068 030/300 Write block 4087 len 40960 data: 3 2 4 4 6 5 4 5 5 3
25069 031/300 Flush
25070 032/300 Read block 4875 len 20480
25071 033/300 Write block 2807 len 24576 data: 5 4 4 4 4 4
25072 034/300 Write block 2852 len 8192 data: 5 4
25073 035/300 Read block 3618 len 4096
25074 036/300 Flush
25075 037/300 Flush
25076 038/300 Write block 1859 len 32768 data: 2 2 2 2 2 3 4 3
25077 039/300 Read block 4269 len 36864
25078 040/300 Read block 733 len 40960
25079 041/300 Write block 618 len 8192 data: 4 4
25080 042/300 Read block 3754 len 16384
25081 043/300 Write block 650 len 12288 data: 3 4 5
25082 044/300 Write block 2018 len 4096 data: 7
25083 045/300 Read block 55 len 32768
25084 046/300 Read block 2896 len 20480
25085 047/300 Flush
25086 048/300 Write block 2688 len 16384 data: 2 2 3 2
25087 049/300 Read block 3453 len 8192
25088 050/300 Write block 3756 len 28672 data: 4 3 4 4 4 3 5
25089 051/300 Read block 1991 len 32768
25090 052/300 Write block 3559 len 20480 data: 5 5 6 5 5
25091 053/300 Write block 3565 len 4096 data: 3
25092 054/300 Write block 4469 len 28672 data: 6 5 6 5 5 4 4
25093 055/300 Write block 536 len 40960 data: 6 5 4 5 5 4 4 3 3 4
25094 056/300 Read block 3286 len 16384
25095 057/300 Read block 700 len 8192
25096 058/300 Read block 4521 len 24576
25097 059/300 Write block 597 len 36864 data: 7 6 7 7 8 7 7 6 3
25098 060/300 Read block 4384 len 16384
25099 061/300 Read block 3078 len 16384
25100 062/300 Read block 541 len 20480
25101 063/300 Write block 1568 len 12288 data: 3 5 6
25102 064/300 Write block 3014 len 28672 data: 4 5 5 6 4 4 4
25103 065/300 Write block 328 len 12288 data: 5 6 5
25104 066/300 Read block 1405 len 20480
25105 067/300 Read block 2115 len 20480
25106 068/300 Read block 4498 len 4096
25107 069/300 Read block 1805 len 24576
25108 070/300 Flush
25109 071/300 Write block 714 len 24576 data: 5 5 4 3 3 3
25110 072/300 Write block 4663 len 4096 data: 8
25111 073/300 Read block 4816 len 4096
25112 074/300 Write block 3883 len 36864 data: 4 3 3 3 2 2 2 2 2
25113 075/300 Read block 4960 len 40960
25114 076/300 Read block 2978 len 12288
25115 077/300 Flush
25116 078/300 Read block 2099 len 16384
25117 079/300 Write block 2666 len 40960 data: 7 6 6 6 4 5 5 4 3 3
25118 080/300 Write block 1721 len 4096 data: 6
25119 081/300 Read block 2923 len 28672
25120 082/300 Read block 2914 len 20480
25121 083/300 Flush
25122 084/300 Read block 558 len 16384
25123 085/300 Read block 614 len 4096
25124 086/300 Read block 3780 len 36864
25125 087/300 Write block 3707 len 36864 data: 6 6 7 7 6 5 6 6 7
25126 088/300 Read block 2233 len 40960
25127 089/300 Write block 4302 len 36864 data: 6 6 6 4 3 3 4 5 4
25128 090/300 Read block 1277 len 24576
25129 091/300 Flush
25130 092/300 Read block 2112 len 36864
25131 093/300 Write block 4240 len 40960 data: 6 5 5 5 5 5 5 4 4 6
25132 094/300 Read block 2378 len 16384
25133 095/300 Read block 1107 len 12288
25134 096/300 Write block 299 len 40960 data: 7 7 8 6 6 6 4 4 4 4
25135 097/300 Flush
25136 098/300 Read block 4679 len 16384
25137 099/300 Read block 2900 len 32768
25138 100/300 Write block 920 len 4096 data: 5
25139 101/300 Write block 1888 len 4096 data: 4
25140 102/300 Flush
25141 103/300 Read block 4233 len 40960
25142 104/300 Flush
25143 105/300 Write block 1365 len 28672 data: 9 8 7 5 6 4 4
25144 106/300 Write block 4210 len 40960 data: 6 6 8 7 7 5 3 3 2 3
25145 107/300 Read block 3397 len 16384
25146 108/300 Write block 3465 len 8192 data: 4 3
25147 109/300 Read block 4773 len 36864
25148 110/300 Flush
25149 111/300 Write block 2914 len 28672 data: 5 5 4 5 6 5 5
25150 112/300 Write block 1372 len 12288 data: 4 4 4
25151 113/300 Read block 4904 len 4096
25152 114/300 Read block 384 len 40960
25153 115/300 Read block 586 len 12288
25154 116/300 Read block 1279 len 24576
25155 117/300 Read block 2639 len 16384
25156 118/300 Write block 1623 len 12288 data: 5 5 7
25157 119/300 Write block 2141 len 36864 data: 8 8 8 7 5 3 3 3 3
25158 120/300 Write block 1908 len 12288 data: 7 6 8
25159 121/300 Read block 2931 len 36864
25160 122/300 Write block 944 len 4096 data: 5
25161 123/300 Write block 1395 len 4096 data: 8
25162 124/300 Write block 2811 len 24576 data: 5 5 3 3 3 2
25163 125/300 Read block 2002 len 8192
25164 126/300 Write block 2340 len 40960 data: 3 4 4 3 4 4 4 5 6 6
25165 127/300 Read block 40 len 24576
25166 128/300 Write block 24 len 4096 data: 6
25167 129/300 Write block 1800 len 36864 data: 3 3 3 3 2 2 2 3 4
25168 130/300 Read block 898 len 24576
25169 131/300 Read block 1503 len 24576
25170 132/300 Read block 905 len 16384
25171 133/300 Write block 4922 len 36864 data: 4 4 5 5 5 4 4 4 4
25172 134/300 Write block 622 len 4096 data: 7
25173 135/300 Flush
25174 136/300 Read block 1404 len 24576
25175 137/300 Read block 1612 len 12288
25176 138/300 Read block 3682 len 12288
25177 139/300 Read block 3173 len 32768
25178 140/300 Read block 4199 len 12288
25179 141/300 Write block 2048 len 12288 data: 3 3 4
25180 142/300 Read block 4050 len 32768
25181 143/300 Read block 2088 len 16384
25182 144/300 Flush
25183 145/300 Read block 4741 len 28672
25184 146/300 Write block 2687 len 40960 data: 2 3 3 4 3 2 3 4 4 5
25185 147/300 Read block 1928 len 28672
25186 148/300 Read block 3323 len 36864
25187 149/300 Read block 1260 len 40960
25188 150/300 Read block 1331 len 16384
25189 151/300 Write block 2892 len 20480 data: 3 5 8 7 6
25190 152/300 Write block 3237 len 36864 data: 5 5 5 5 5 4 4 4 4
25191 153/300 Write block 1581 len 24576 data: 6 4 4 4 4 3
25192 154/300 Read block 1897 len 4096
25193 155/300 Write block 535 len 36864 data: 5 7 6 5 6 6 5 5 4
25194 156/300 Read block 2200 len 36864
25195 157/300 Write block 1008 len 32768 data: 3 3 3 3 4 4 4 4
25196 158/300 Write block 797 len 28672 data: 4 3 4 3 4 4 4
25197 159/300 Read block 2934 len 8192
25198 160/300 Write block 746 len 16384 data: 5 6 6 6
25199 161/300 Read block 2771 len 12288
25200 162/300 Read block 4462 len 16384
25201 163/300 Write block 1117 len 8192 data: 4 4
25202 164/300 Write block 3735 len 24576 data: 8 8 7 6 6 4
25203 165/300 Read block 4766 len 16384
25204 166/300 Read block 876 len 8192
25205 167/300 Write block 2249 len 8192 data: 6 6
25206 168/300 Flush
25207 169/300 Read block 1797 len 32768
25208 170/300 Write block 4763 len 36864 data: 7 7 7 10 9 8 6 6 6
25209 171/300 Read block 3960 len 36864
25210 172/300 Write block 280 len 24576 data: 5 4 2 2 2 2
25211 173/300 Read block 1979 len 8192
25212 174/300 Read block 3112 len 4096
25213 175/300 Write block 2501 len 8192 data: 4 6
25214 176/300 Flush
25215 177/300 Read block 4829 len 28672
25216 178/300 Write block 3928 len 8192 data: 2 2
25217 179/300 Read block 330 len 4096
25218 180/300 Flush
25219 181/300 Flush
25220 182/300 Read block 2107 len 40960
25221 183/300 Read block 3272 len 20480
25222 184/300 Read block 2806 len 12288
25223 185/300 Write block 3339 len 28672 data: 4 5 6 4 5 4 4
25224 186/300 Read block 616 len 32768
25225 187/300 Read block 2169 len 32768
25226 188/300 Write block 1236 len 32768 data: 6 7 6 4 4 5 4 4
25227 189/300 Write block 1202 len 4096 data: 4
25228 190/300 Read block 62 len 36864
25229 191/300 Write block 525 len 20480 data: 4 4 3 2 3
25230 192/300 Write block 4795 len 4096 data: 3
25231 193/300 Read block 2318 len 40960
25232 194/300 Flush
25233 195/300 Write block 2779 len 16384 data: 3 4 7 7
25234 196/300 Flush
25235 197/300 Flush
25236 198/300 Write block 617 len 16384 data: 3 5 5 5
25237 199/300 Read block 1864 len 24576
25238 200/300 Flush
25239 201/300 Write block 4114 len 32768 data: 4 3 3 3 3 2 3 3
25240 202/300 Read block 695 len 36864
25241 203/300 Read block 1350 len 40960
25242 204/300 Write block 890 len 20480 data: 5 4 4 5 5
25243 205/300 Read block 1717 len 28672
25244 206/300 Read block 995 len 12288
25245 207/300 Read block 835 len 12288
25246 208/300 Write block 1345 len 36864 data: 3 3 2 4 5 5 5 5 5
25247 209/300 Read block 1218 len 20480
25248 210/300 Read block 2819 len 24576
25249 211/300 Read block 296 len 16384
25250 212/300 Read block 1825 len 16384
25251 213/300 Write block 504 len 12288 data: 3 3 3
25252 214/300 Read block 1912 len 36864
25253 215/300 Read block 1702 len 40960
25254 216/300 Write block 4337 len 20480 data: 7 7 6 6 5
25255 217/300 Write block 4525 len 8192 data: 2 2
25256 218/300 Read block 564 len 36864
25257 219/300 Read block 2499 len 16384
25258 220/300 Write block 1171 len 28672 data: 6 6 5 6 6 6 6
25259 221/300 Write block 1107 len 16384 data: 7 8 7 8
25260 222/300 Read block 3753 len 32768
25261 223/300 Flush
25262 224/300 Read block 1777 len 12288
25263 225/300 Read block 4468 len 20480
25264 226/300 Read block 4011 len 12288
25265 227/300 Write block 2639 len 16384 data: 6 5 3 3
25266 228/300 Read block 1291 len 20480
25267 229/300 Flush
25268 230/300 Write block 4673 len 40960 data: 5 4 3 3 3 3 3 3 2 2
25269 231/300 Read block 3175 len 32768
25270 232/300 Write block 1037 len 40960 data: 4 4 5 5 7 6 6 7 8 8
25271 233/300 Write block 2810 len 20480 data: 5 6 6 4 4
25272 234/300 Write block 2507 len 40960 data: 7 7 6 4 4 3 3 3 5 5
25273 235/300 Write block 4277 len 20480 data: 3 2 3 4 4
25274 236/300 Write block 4400 len 32768 data: 4 4 4 5 5 5 5 2
25275 237/300 Write block 66 len 28672 data: 4 4 6 7 7 8 8
25276 238/300 Flush
25277 239/300 Write block 2874 len 36864 data: 3 5 5 6 5 5 5 8 6
25278 240/300 Read block 3335 len 8192
25279 241/300 Write block 2318 len 36864 data: 5 6 7 6 6 7 6 7 6
25280 242/300 Read block 1800 len 24576
25281 243/300 Write block 2925 len 20480 data: 5 4 4 3 2
25282 244/300 Read block 3810 len 32768
25283 245/300 Write block 3525 len 12288 data: 4 4 3
25284 246/300 Read block 1241 len 4096
25285 247/300 Read block 2909 len 32768
25286 248/300 Read block 3839 len 40960
25287 249/300 Read block 1747 len 4096
25288 250/300 Read block 4174 len 20480
25289 251/300 Read block 3235 len 4096
25290 252/300 Read block 3268 len 32768
25291 253/300 Read block 113 len 28672
25292 254/300 Write block 1994 len 36864 data: 4 4 4 3 6 7 7 6 6
25293 255/300 Read block 3542 len 32768
25294 256/300 Write block 3673 len 24576 data: 7 8 7 7 5 5
25295 257/300 Read block 2830 len 4096
25296 258/300 Write block 488 len 4096 data: 6
25297 259/300 Write block 4362 len 28672 data: 4 3 3 3 3 4 5
25298 260/300 Write block 1539 len 36864 data: 5 6 6 5 5 5 5 3 3
25299 261/300 Write block 4114 len 40960 data: 5 4 4 4 4 3 4 4 4 5
25300 262/300 Read block 3306 len 20480
25301 263/300 Flush
25302 264/300 Write block 1199 len 36864 data: 5 5 4 5 3 3 2 2 2
25303 265/300 Read block 1891 len 40960
25304 266/300 Write block 615 len 12288 data: 3 3 4
25305 267/300 Write block 4252 len 36864 data: 4 4 4 5 5 6 6 5 6
25306 268/300 Read block 2300 len 4096
25307 269/300 Read block 2914 len 36864
25308 270/300 Read block 2005 len 20480
25309 271/300 Read block 3852 len 32768
25310 272/300 Flush
25311 273/300 Read block 1966 len 4096
25312 274/300 Read block 324 len 28672
25313 275/300 Write block 345 len 8192 data: 3 2
25314 276/300 Write block 1701 len 28672 data: 6 7 7 9 9 10 7
25315 277/300 Read block 729 len 40960
25316 278/300 Flush
25317 279/300 Write block 4220 len 28672 data: 2 2 3 4 4 4 3
25318 280/300 Read block 1781 len 8192
25319 281/300 Write block 2453 len 12288 data: 6 8 9
25320 282/300 Write block 1178 len 16384 data: 8 7 7 5
25321 283/300 Read block 2425 len 32768
25322 284/300 Flush
25323 285/300 Read block 3778 len 28672
25324 286/300 Flush
25325 287/300 Read block 562 len 28672
25326 288/300 Read block 2333 len 12288
25327 289/300 Write block 3054 len 20480 data: 8 7 6 7 7
25328 290/300 Read block 1057 len 8192
25329 291/300 Read block 3700 len 12288
25330 292/300 Read block 683 len 8192
25331 293/300 Flush
25332 294/300 Flush
25333 295/300 Flush
25334 296/300 Write block 3304 len 32768 data: 7 7 5 5 6 6 3 5
25335 297/300 Read block 660 len 12288
25336 298/300 Write block 2395 len 32768 data: 7 7 6 6 7 5 4 4
25337 299/300 Read block 3510 len 24576
25338 300/300 Write block 3467 len 4096 data: 3
25339 301/300 Flush
25340 Main task has work to do, go find it
25341 got dsc Start(0)
25342 start 0
25343 Feb 25 17:23:51.891 INFO request completed, latency_us: 248, response_code: 204, uri: /start/cid/0, method: POST, req_id: b0a09223-2adb-4112-b135-0455c09adb78, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
25344 Replay: started 0, returned:Ok(())
25345 ----------------------------------------------------------------
25346 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25347 States: Offline Active Active
25348 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25349 7643 Acked Flush 0 Sent Done Done false
25350 7644 Acked Flush 0 Sent Done Done false
25351 7645 Acked Flush 0 Sent Done Done false
25352 7646 Acked Write 3 Sent Done Done false
25353 7647 Acked Flush 0 Sent Done Done false
25354 7648 Acked Flush 0 Sent Done Done false
25355 7649 Acked Write 6 Sent Done Done false
25356 7650 Acked Read 8 Sent Done Done false
25357 7651 Acked Read 1 Sent Done Done false
25358 7652 Acked Flush 0 Sent Done Done false
25359 7653 Acked Read 3 Sent Done Done false
25360 7654 Acked Read 6 Sent Done Done false
25361 7655 Acked Read 6 Sent Done Done false
25362 7656 Acked Read 5 Sent Done Done false
25363 7657 Acked Flush 0 Sent Done Done false
25364 7658 Acked Read 6 Sent Done Done false
25365 7659 Acked Write 7 Sent Done Done false
25366 7660 Acked Write 8 Sent Done Done false
25367 7661 Acked Write 3 Sent Done Done false
25368 7662 Acked Read 1 Sent Done Done false
25369 7663 Acked Write 3 Sent Done Done false
25370 7664 Acked Read 6 Sent Done Done false
25371 7665 Acked Read 3 Sent Done Done false
25372 7666 Acked Write 10 Sent Done Done false
25373 7667 Acked Write 4 Sent Done Done false
25374 7668 Acked Read 1 Sent Done Done false
25375 7669 Acked Write 8 Sent Done Done false
25376 7670 Acked Read 7 Sent Done Done false
25377 7671 Acked Write 7 Sent Done Done false
25378 7672 Acked Write 10 Sent Done Done false
25379 7673 Acked Flush 0 Sent Done Done false
25380 7674 Acked Read 5 Sent Done Done false
25381 7675 Acked Write 6 Sent Done Done false
25382 7676 Acked Write 2 Sent Done Done false
25383 7677 Acked Read 1 Sent Done Done false
25384 7678 Acked Flush 0 Sent Done Done false
25385 7679 Acked Flush 0 Sent Done Done false
25386 7680 Acked Write 8 Sent Done Done false
25387 7681 Acked Read 9 Sent Done Done false
25388 7682 Acked Read 10 Sent Done Done false
25389 7683 Acked Write 2 Sent Done Done false
25390 7684 Acked Read 4 Sent Done Done false
25391 7685 Acked Write 3 Sent Done Done false
25392 7686 Acked Write 1 Sent Done Done false
25393 7687 Acked Read 8 Sent Done Done false
25394 7688 Acked Read 5 Sent Done Done false
25395 7689 Acked Flush 0 Sent Done Done false
25396 7690 Acked Write 4 Sent Done Done false
25397 7691 Acked Read 2 Sent Done Done false
25398 7692 Acked Write 7 Sent Done Done false
25399 7693 Acked Read 8 Sent Done Done false
25400 7694 Acked Write 5 Sent Done Done false
25401 7695 Acked Write 1 Sent Done Done false
25402 7696 Acked Write 7 Sent Done Done false
25403 7697 Acked Write 10 Sent Done Done false
25404 7698 Acked Read 4 Sent Done Done false
25405 7699 Acked Read 2 Sent Done Done false
25406 7700 Acked Read 6 Sent Done Done false
25407 7701 Acked Write 9 Sent Done Done false
25408 7702 Acked Read 4 Sent Done Done false
25409 7703 Acked Read 4 Sent Done Done false
25410 7704 Acked Read 5 Sent Done Done false
25411 7705 Acked Write 3 Sent Done Done false
25412 7706 Acked Write 7 Sent Done Done false
25413 7707 Acked Write 3 Sent Done Done false
25414 7708 Acked Read 5 Sent Done Done false
25415 7709 Acked Read 5 Sent Done Done false
25416 7710 Acked Read 1 Sent Done Done false
25417 7711 Acked Read 6 Sent Done Done false
25418 7712 Acked Flush 0 Sent Done Done false
25419 7713 Acked Write 6 Sent Done Done false
25420 7714 Acked Write 1 Sent Done Done false
25421 7715 Acked Read 1 Sent Done Done false
25422 7716 Acked Write 9 Sent Done Done false
25423 7717 Acked Read 10 Sent Done Done false
25424 7718 Acked Read 3 Sent Done Done false
25425 7719 Acked Flush 0 Sent Done Done false
25426 7720 Acked Read 4 Sent Done Done false
25427 7721 Acked Write 10 Sent Done Done false
25428 7722 Acked Write 1 Sent Done Done false
25429 7723 Acked Read 7 Sent Done Done false
25430 7724 Acked Read 5 Sent Done Done false
25431 7725 Acked Flush 0 Sent Done Done false
25432 7726 Acked Read 4 Sent Done Done false
25433 7727 Acked Read 1 Sent Done Done false
25434 7728 Acked Read 9 Sent Done Done false
25435 7729 Acked Write 9 Sent Done Done false
25436 7730 Acked Read 10 Sent Done Done false
25437 7731 Acked Write 9 Sent Done Done false
25438 7732 Acked Read 6 Sent Done Done false
25439 7733 Acked Flush 0 Sent Done Done false
25440 7734 Acked Read 9 Sent Done Done false
25441 7735 Acked Write 10 Sent Done Done false
25442 7736 Acked Read 4 Sent Done Done false
25443 7737 Acked Read 3 Sent Done Done false
25444 7738 Acked Write 10 Sent Done Done false
25445 7739 Acked Flush 0 Sent Done Done false
25446 7740 Acked Read 4 Sent Done Done false
25447 7741 Acked Read 8 Sent Done Done false
25448 7742 Acked Write 1 Sent Done Done false
25449 7743 Acked Write 1 Sent Done Done false
25450 7744 Acked Flush 0 Sent Done Done false
25451 7745 Acked Read 10 Sent Done Done false
25452 7746 Acked Flush 0 Sent Done Done false
25453 7747 Acked Write 7 Sent Done Done false
25454 7748 Acked Write 10 Sent Done Done false
25455 7749 Acked Read 4 Sent Done Done false
25456 7750 Acked Write 2 Sent Done Done false
25457 7751 Acked Read 9 Sent Done Done false
25458 7752 Acked Flush 0 Sent Done Done false
25459 7753 Acked Write 7 Sent Done Done false
25460 7754 Acked Write 3 Sent Done Done false
25461 7755 Acked Read 1 Sent Done Done false
25462 7756 Acked Read 10 Sent Done Done false
25463 7757 Acked Read 3 Sent Done Done false
25464 7758 Acked Read 6 Sent Done Done false
25465 7759 Acked Read 4 Sent Done Done false
25466 7760 Acked Write 3 Sent Done Done false
25467 7761 Acked Write 9 Sent Done Done false
25468 7762 Acked Write 3 Sent Done Done false
25469 7763 Acked Read 9 Sent Done Done false
25470 7764 Acked Write 1 Sent Done Done false
25471 7765 Acked Write 1 Sent Done Done false
25472 7766 Acked Write 6 Sent Done Done false
25473 7767 Acked Read 2 Sent Done Done false
25474 7768 Acked Write 10 Sent Done Done false
25475 7769 Acked Read 6 Sent Done Done false
25476 7770 Acked Write 1 Sent Done Done false
25477 7771 Acked Write 9 Sent Done Done false
25478 7772 Acked Read 6 Sent Done Done false
25479 7773 Acked Read 6 Sent Done Done false
25480 7774 Acked Read 4 Sent Done Done false
25481 7775 Acked Write 9 Sent Done Done false
25482 7776 Acked Write 1 Sent Done Done false
25483 7777 Acked Flush 0 Sent Done Done false
25484 7778 Acked Read 6 Sent Done Done false
25485 7779 Acked Read 3 Sent Done Done false
25486 7780 Acked Read 3 Sent Done Done false
25487 7781 Acked Read 8 Sent Done Done false
25488 7782 Acked Read 3 Sent Done Done false
25489 7783 Acked Write 3 Sent Done Done false
25490 7784 Acked Read 8 Sent Done Done false
25491 7785 Acked Read 4 Sent Done Done false
25492 7786 Acked Flush 0 Sent Done Done false
25493 7787 Acked Read 7 Sent Done Done false
25494 7788 Acked Write 10 Sent Done Done false
25495 7789 Acked Read 7 Sent Done Done false
25496 7790 Acked Read 9 Sent Done Done false
25497 7791 Acked Read 10 Sent Done Done false
25498 7792 Acked Read 4 Sent Done Done false
25499 7793 Acked Write 5 Sent Done Done false
25500 7794 Acked Write 9 Sent Done Done false
25501 7795 Acked Write 6 Sent Done Done false
25502 7796 Acked Read 1 Sent Done Done false
25503 7797 Acked Write 9 Sent Done Done false
25504 7798 Acked Read 9 Sent Done Done false
25505 7799 Acked Write 8 Sent Done Done false
25506 7800 Acked Write 7 Sent Done Done false
25507 7801 Acked Read 2 Sent Done Done false
25508 7802 Acked Write 4 Sent Done Done false
25509 7803 Acked Read 3 Sent Done Done false
25510 7804 Acked Read 4 Sent Done Done false
25511 7805 Acked Write 2 Sent Done Done false
25512 7806 Acked Write 6 Sent Done Done false
25513 7807 Acked Read 4 Sent Done Done false
25514 7808 Acked Read 2 Sent Done Done false
25515 7809 Acked Write 2 Sent Done Done false
25516 7810 Acked Flush 0 Sent Done Done false
25517 7811 Acked Read 8 Sent Done Done false
25518 7812 Acked Write 9 Sent Done Done false
25519 7813 Acked Read 9 Sent Done Done false
25520 7814 Acked Write 6 Sent Done Done false
25521 7815 Acked Read 2 Sent Done Done false
25522 7816 Acked Read 1 Sent Done Done false
25523 7817 Acked Write 2 Sent Done Done false
25524 7818 Acked Flush 0 Sent Done Done false
25525 7819 Acked Read 7 Sent Done Done false
25526 7820 Acked Write 2 Sent Done Done false
25527 7821 Acked Read 1 Sent Done Done false
25528 7822 Acked Flush 0 Sent Done Done false
25529 7823 Acked Flush 0 Sent Done Done false
25530 7824 Acked Read 10 Sent Done Done false
25531 7825 Acked Read 5 Sent Done Done false
25532 7826 Acked Read 3 Sent Done Done false
25533 7827 Acked Write 7 Sent Done Done false
25534 7828 Acked Read 8 Sent Done Done false
25535 7829 Acked Read 8 Sent Done Done false
25536 7830 Acked Write 8 Sent Done Done false
25537 7831 Acked Write 1 Sent Done Done false
25538 7832 Acked Read 9 Sent Done Done false
25539 7833 Acked Write 5 Sent Done Done false
25540 7834 Acked Write 1 Sent Done Done false
25541 7835 Acked Read 10 Sent Done Done false
25542 7836 Acked Flush 0 Sent Done Done false
25543 7837 Acked Write 4 Sent Done Done false
25544 7838 Acked Flush 0 Sent Done Done false
25545 7839 Acked Flush 0 Sent Done Done false
25546 7840 Acked Write 4 Sent Done Done false
25547 7841 Acked Read 6 Sent Done Done false
25548 7842 Acked Flush 0 Sent Done Done false
25549 7843 Acked Write 8 Sent Done Done false
25550 7844 Acked Read 9 Sent Done Done false
25551 7845 Acked Read 10 Sent Done Done false
25552 7846 Acked Write 5 Sent Done Done false
25553 7847 Acked Read 7 Sent Done Done false
25554 7848 Acked Read 3 Sent Done Done false
25555 7849 Acked Read 3 Sent Done Done false
25556 7850 Acked Write 9 Sent Done Done false
25557 7851 Acked Read 5 Sent Done Done false
25558 7852 Acked Read 6 Sent Done Done false
25559 7853 Acked Read 4 Sent Done Done false
25560 7854 Acked Read 4 Sent Done Done false
25561 7855 Acked Write 3 Sent Done Done false
25562 7856 Acked Read 9 Sent Done Done false
25563 7857 Acked Read 10 Sent Done Done false
25564 7858 Acked Write 5 Sent Done Done false
25565 7859 Acked Write 2 Sent Done Done false
25566 7860 Acked Read 9 Sent Done Done false
25567 7861 Acked Read 4 Sent Done Done false
25568 7862 Acked Write 7 Sent Done Done false
25569 7863 Acked Write 4 Sent Done Done false
25570 7864 Acked Read 8 Sent Done Done false
25571 7865 Acked Flush 0 Sent Done Done false
25572 7866 Acked Read 3 Sent Done Done false
25573 7867 Acked Read 5 Sent Done Done false
25574 7868 Acked Read 3 Sent Done Done false
25575 7869 Acked Write 4 Sent Done Done false
25576 7870 Acked Read 5 Sent Done Done false
25577 7871 Acked Flush 0 Sent Done Done false
25578 7872 Acked Write 10 Sent Done Done false
25579 7873 Acked Read 8 Sent Done Done false
25580 7874 Acked Write 10 Sent Done Done false
25581 7875 Acked Write 5 Sent Done Done false
25582 7876 Acked Write 10 Sent Done Done false
25583 7877 Acked Write 5 Sent Done Done false
25584 7878 Acked Write 8 Sent Done Done false
25585 7879 Acked Write 7 Sent Done Done false
25586 7880 Acked Flush 0 Sent Done Done false
25587 7881 Acked Write 9 Sent Done Done false
25588 7882 Acked Read 2 Sent Done Done false
25589 7883 Acked Write 9 Sent Done Done false
25590 7884 Acked Read 6 Sent Done Done false
25591 7885 Acked Write 5 Sent Done Done false
25592 7886 Acked Read 8 Sent Done Done false
25593 7887 Acked Write 3 Sent Done Done false
25594 7888 Acked Read 1 Sent Done Done false
25595 7889 Acked Read 8 Sent Done Done false
25596 7890 Acked Read 10 Sent Done Done false
25597 7891 Acked Read 1 Sent Done Done false
25598 7892 Acked Read 5 Sent Done Done false
25599 7893 Acked Read 1 Sent Done Done false
25600 7894 Acked Read 8 Sent Done Done false
25601 7895 Acked Read 7 Sent Done Done false
25602 7896 Acked Write 9 Sent Done Done false
25603 7897 Acked Read 8 Sent Done Done false
25604 7898 Acked Write 6 Sent Done Done false
25605 7899 Acked Read 1 Sent Done Done false
25606 7900 Acked Write 1 Sent Done Done false
25607 7901 Acked Write 7 Sent Done Done false
25608 7902 Acked Write 9 Sent Done Done false
25609 7903 Acked Write 10 Sent Done Done false
25610 7904 Acked Read 5 Sent Done Done false
25611 7905 Acked Flush 0 Sent Done Done false
25612 7906 Acked Write 9 Sent Done Done false
25613 7907 Acked Read 10 Sent Done Done false
25614 7908 Acked Write 3 Sent Done Done false
25615 7909 Acked Write 9 Sent Done Done false
25616 7910 Acked Read 1 Sent Done Done false
25617 7911 Acked Read 9 Sent Done Done false
25618 7912 Acked Read 5 Sent Done Done false
25619 7913 Acked Read 8 Sent Done Done false
25620 7914 Acked Flush 0 Sent Done Done false
25621 7915 Acked Read 1 Sent Done Done false
25622 7916 Acked Read 7 Sent Done Done false
25623 7917 Acked Write 2 Sent Done Done false
25624 7918 Acked Write 7 Sent Done Done false
25625 7919 Acked Read 10 Sent Done Done false
25626 7920 Acked Flush 0 Sent Done Done false
25627 7921 Acked Write 7 Sent Done Done false
25628 7922 Acked Read 2 Sent Done Done false
25629 7923 Acked Write 3 Sent Done Done false
25630 7924 Acked Write 4 Sent Done Done false
25631 7925 Acked Read 8 Sent Done Done false
25632 7926 Acked Flush 0 Sent Done Done false
25633 7927 Acked Read 7 Sent Done Done false
25634 7928 Acked Flush 0 Sent Done Done false
25635 7929 Acked Read 7 Sent Done Done false
25636 7930 Acked Read 3 Sent Done Done false
25637 7931 Acked Write 5 Sent Done Done false
25638 7932 Acked Read 2 Sent Done Done false
25639 7933 Acked Read 3 Sent Done Done false
25640 7934 Acked Read 2 Sent Done Done false
25641 7935 Acked Flush 0 Sent Done Done false
25642 7936 Acked Flush 0 Sent Done Done false
25643 7937 Acked Flush 0 Sent Done Done false
25644 7938 Acked Write 8 Sent Done Done false
25645 7939 Acked Read 3 Sent Done Done false
25646 7940 Acked Write 8 Sent Done Done false
25647 7941 Acked Read 6 Sent Done Done false
25648 7942 Acked Write 1 Sent Done Done false
25649 7943 Acked Flush 0 Sent Done Done false
25650 STATES DS:0 DS:1 DS:2 TOTAL
25651 Sent 301 0 0 301
25652 Done 0 301 301 602
25653 Skipped 0 0 0 0
25654 Error 0 0 0 0
25655 Last Flush: Some(JobId(7642)) Some(JobId(7943)) Some(JobId(7943))
25656 Downstairs last five completed: 7642 7641 7640 7639 7638
25657 Upstairs last five completed: 7943 7942 7941 7940 7939
25658 CLIENT: Up:0 ds:301 act:2
256592025-02-25T17:23:52.803ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
256602025-02-25T17:23:53.804ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
256612025-02-25T17:23:54.806ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
256622025-02-25T17:23:55.307ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
256632025-02-25T17:23:55.870ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
25664 ----------------------------------------------------------------
25665 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25666 States: Offline Active Active
25667 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25668 7643 Acked Flush 0 Sent Done Done false
25669 7644 Acked Flush 0 Sent Done Done false
25670 7645 Acked Flush 0 Sent Done Done false
25671 7646 Acked Write 3 Sent Done Done false
25672 7647 Acked Flush 0 Sent Done Done false
25673 7648 Acked Flush 0 Sent Done Done false
25674 7649 Acked Write 6 Sent Done Done false
25675 7650 Acked Read 8 Sent Done Done false
25676 7651 Acked Read 1 Sent Done Done false
25677 7652 Acked Flush 0 Sent Done Done false
25678 7653 Acked Read 3 Sent Done Done false
25679 7654 Acked Read 6 Sent Done Done false
25680 7655 Acked Read 6 Sent Done Done false
25681 7656 Acked Read 5 Sent Done Done false
25682 7657 Acked Flush 0 Sent Done Done false
25683 7658 Acked Read 6 Sent Done Done false
25684 7659 Acked Write 7 Sent Done Done false
25685 7660 Acked Write 8 Sent Done Done false
25686 7661 Acked Write 3 Sent Done Done false
25687 7662 Acked Read 1 Sent Done Done false
25688 7663 Acked Write 3 Sent Done Done false
25689 7664 Acked Read 6 Sent Done Done false
25690 7665 Acked Read 3 Sent Done Done false
25691 7666 Acked Write 10 Sent Done Done false
25692 7667 Acked Write 4 Sent Done Done false
25693 7668 Acked Read 1 Sent Done Done false
25694 7669 Acked Write 8 Sent Done Done false
25695 7670 Acked Read 7 Sent Done Done false
25696 7671 Acked Write 7 Sent Done Done false
25697 7672 Acked Write 10 Sent Done Done false
25698 7673 Acked Flush 0 Sent Done Done false
25699 7674 Acked Read 5 Sent Done Done false
25700 7675 Acked Write 6 Sent Done Done false
25701 7676 Acked Write 2 Sent Done Done false
25702 7677 Acked Read 1 Sent Done Done false
25703 7678 Acked Flush 0 Sent Done Done false
25704 7679 Acked Flush 0 Sent Done Done false
25705 7680 Acked Write 8 Sent Done Done false
25706 7681 Acked Read 9 Sent Done Done false
25707 7682 Acked Read 10 Sent Done Done false
25708 7683 Acked Write 2 Sent Done Done false
25709 7684 Acked Read 4 Sent Done Done false
25710 7685 Acked Write 3 Sent Done Done false
25711 7686 Acked Write 1 Sent Done Done false
25712 7687 Acked Read 8 Sent Done Done false
25713 7688 Acked Read 5 Sent Done Done false
25714 7689 Acked Flush 0 Sent Done Done false
25715 7690 Acked Write 4 Sent Done Done false
25716 7691 Acked Read 2 Sent Done Done false
25717 7692 Acked Write 7 Sent Done Done false
25718 7693 Acked Read 8 Sent Done Done false
25719 7694 Acked Write 5 Sent Done Done false
25720 7695 Acked Write 1 Sent Done Done false
25721 7696 Acked Write 7 Sent Done Done false
25722 7697 Acked Write 10 Sent Done Done false
25723 7698 Acked Read 4 Sent Done Done false
25724 7699 Acked Read 2 Sent Done Done false
25725 7700 Acked Read 6 Sent Done Done false
25726 7701 Acked Write 9 Sent Done Done false
25727 7702 Acked Read 4 Sent Done Done false
25728 7703 Acked Read 4 Sent Done Done false
25729 7704 Acked Read 5 Sent Done Done false
25730 7705 Acked Write 3 Sent Done Done false
25731 7706 Acked Write 7 Sent Done Done false
25732 7707 Acked Write 3 Sent Done Done false
25733 7708 Acked Read 5 Sent Done Done false
25734 7709 Acked Read 5 Sent Done Done false
25735 7710 Acked Read 1 Sent Done Done false
25736 7711 Acked Read 6 Sent Done Done false
25737 7712 Acked Flush 0 Sent Done Done false
25738 7713 Acked Write 6 Sent Done Done false
25739 7714 Acked Write 1 Sent Done Done false
25740 7715 Acked Read 1 Sent Done Done false
25741 7716 Acked Write 9 Sent Done Done false
25742 7717 Acked Read 10 Sent Done Done false
25743 7718 Acked Read 3 Sent Done Done false
25744 7719 Acked Flush 0 Sent Done Done false
25745 7720 Acked Read 4 Sent Done Done false
25746 7721 Acked Write 10 Sent Done Done false
25747 7722 Acked Write 1 Sent Done Done false
25748 7723 Acked Read 7 Sent Done Done false
25749 7724 Acked Read 5 Sent Done Done false
25750 7725 Acked Flush 0 Sent Done Done false
25751 7726 Acked Read 4 Sent Done Done false
25752 7727 Acked Read 1 Sent Done Done false
25753 7728 Acked Read 9 Sent Done Done false
25754 7729 Acked Write 9 Sent Done Done false
25755 7730 Acked Read 10 Sent Done Done false
25756 7731 Acked Write 9 Sent Done Done false
25757 7732 Acked Read 6 Sent Done Done false
25758 7733 Acked Flush 0 Sent Done Done false
25759 7734 Acked Read 9 Sent Done Done false
25760 7735 Acked Write 10 Sent Done Done false
25761 7736 Acked Read 4 Sent Done Done false
25762 7737 Acked Read 3 Sent Done Done false
25763 7738 Acked Write 10 Sent Done Done false
25764 7739 Acked Flush 0 Sent Done Done false
25765 7740 Acked Read 4 Sent Done Done false
25766 7741 Acked Read 8 Sent Done Done false
25767 7742 Acked Write 1 Sent Done Done false
25768 7743 Acked Write 1 Sent Done Done false
25769 7744 Acked Flush 0 Sent Done Done false
25770 7745 Acked Read 10 Sent Done Done false
25771 7746 Acked Flush 0 Sent Done Done false
25772 7747 Acked Write 7 Sent Done Done false
25773 7748 Acked Write 10 Sent Done Done false
25774 7749 Acked Read 4 Sent Done Done false
25775 7750 Acked Write 2 Sent Done Done false
25776 7751 Acked Read 9 Sent Done Done false
25777 7752 Acked Flush 0 Sent Done Done false
25778 7753 Acked Write 7 Sent Done Done false
25779 7754 Acked Write 3 Sent Done Done false
25780 7755 Acked Read 1 Sent Done Done false
25781 7756 Acked Read 10 Sent Done Done false
25782 7757 Acked Read 3 Sent Done Done false
25783 7758 Acked Read 6 Sent Done Done false
25784 7759 Acked Read 4 Sent Done Done false
25785 7760 Acked Write 3 Sent Done Done false
25786 7761 Acked Write 9 Sent Done Done false
25787 7762 Acked Write 3 Sent Done Done false
25788 7763 Acked Read 9 Sent Done Done false
25789 7764 Acked Write 1 Sent Done Done false
25790 7765 Acked Write 1 Sent Done Done false
25791 7766 Acked Write 6 Sent Done Done false
25792 7767 Acked Read 2 Sent Done Done false
25793 7768 Acked Write 10 Sent Done Done false
25794 7769 Acked Read 6 Sent Done Done false
25795 7770 Acked Write 1 Sent Done Done false
25796 7771 Acked Write 9 Sent Done Done false
25797 7772 Acked Read 6 Sent Done Done false
25798 7773 Acked Read 6 Sent Done Done false
25799 7774 Acked Read 4 Sent Done Done false
25800 7775 Acked Write 9 Sent Done Done false
25801 7776 Acked Write 1 Sent Done Done false
25802 7777 Acked Flush 0 Sent Done Done false
25803 7778 Acked Read 6 Sent Done Done false
25804 7779 Acked Read 3 Sent Done Done false
25805 7780 Acked Read 3 Sent Done Done false
25806 7781 Acked Read 8 Sent Done Done false
25807 7782 Acked Read 3 Sent Done Done false
25808 7783 Acked Write 3 Sent Done Done false
25809 7784 Acked Read 8 Sent Done Done false
25810 7785 Acked Read 4 Sent Done Done false
25811 7786 Acked Flush 0 Sent Done Done false
25812 7787 Acked Read 7 Sent Done Done false
25813 7788 Acked Write 10 Sent Done Done false
25814 7789 Acked Read 7 Sent Done Done false
25815 7790 Acked Read 9 Sent Done Done false
25816 7791 Acked Read 10 Sent Done Done false
25817 7792 Acked Read 4 Sent Done Done false
25818 7793 Acked Write 5 Sent Done Done false
25819 7794 Acked Write 9 Sent Done Done false
25820 7795 Acked Write 6 Sent Done Done false
25821 7796 Acked Read 1 Sent Done Done false
25822 7797 Acked Write 9 Sent Done Done false
25823 7798 Acked Read 9 Sent Done Done false
25824 7799 Acked Write 8 Sent Done Done false
25825 7800 Acked Write 7 Sent Done Done false
25826 7801 Acked Read 2 Sent Done Done false
25827 7802 Acked Write 4 Sent Done Done false
25828 7803 Acked Read 3 Sent Done Done false
25829 7804 Acked Read 4 Sent Done Done false
25830 7805 Acked Write 2 Sent Done Done false
25831 7806 Acked Write 6 Sent Done Done false
25832 7807 Acked Read 4 Sent Done Done false
25833 7808 Acked Read 2 Sent Done Done false
25834 7809 Acked Write 2 Sent Done Done false
25835 7810 Acked Flush 0 Sent Done Done false
25836 7811 Acked Read 8 Sent Done Done false
25837 7812 Acked Write 9 Sent Done Done false
25838 7813 Acked Read 9 Sent Done Done false
25839 7814 Acked Write 6 Sent Done Done false
25840 7815 Acked Read 2 Sent Done Done false
25841 7816 Acked Read 1 Sent Done Done false
25842 7817 Acked Write 2 Sent Done Done false
25843 7818 Acked Flush 0 Sent Done Done false
25844 7819 Acked Read 7 Sent Done Done false
25845 7820 Acked Write 2 Sent Done Done false
25846 7821 Acked Read 1 Sent Done Done false
25847 7822 Acked Flush 0 Sent Done Done false
25848 7823 Acked Flush 0 Sent Done Done false
25849 7824 Acked Read 10 Sent Done Done false
25850 7825 Acked Read 5 Sent Done Done false
25851 7826 Acked Read 3 Sent Done Done false
25852 7827 Acked Write 7 Sent Done Done false
25853 7828 Acked Read 8 Sent Done Done false
25854 7829 Acked Read 8 Sent Done Done false
25855 7830 Acked Write 8 Sent Done Done false
25856 7831 Acked Write 1 Sent Done Done false
25857 7832 Acked Read 9 Sent Done Done false
25858 7833 Acked Write 5 Sent Done Done false
25859 7834 Acked Write 1 Sent Done Done false
25860 7835 Acked Read 10 Sent Done Done false
25861 7836 Acked Flush 0 Sent Done Done false
25862 7837 Acked Write 4 Sent Done Done false
25863 7838 Acked Flush 0 Sent Done Done false
25864 7839 Acked Flush 0 Sent Done Done false
25865 7840 Acked Write 4 Sent Done Done false
25866 7841 Acked Read 6 Sent Done Done false
25867 7842 Acked Flush 0 Sent Done Done false
25868 7843 Acked Write 8 Sent Done Done false
25869 7844 Acked Read 9 Sent Done Done false
25870 7845 Acked Read 10 Sent Done Done false
25871 7846 Acked Write 5 Sent Done Done false
25872 7847 Acked Read 7 Sent Done Done false
25873 7848 Acked Read 3 Sent Done Done false
25874 7849 Acked Read 3 Sent Done Done false
25875 7850 Acked Write 9 Sent Done Done false
25876 7851 Acked Read 5 Sent Done Done false
25877 7852 Acked Read 6 Sent Done Done false
25878 7853 Acked Read 4 Sent Done Done false
25879 7854 Acked Read 4 Sent Done Done false
25880 7855 Acked Write 3 Sent Done Done false
25881 7856 Acked Read 9 Sent Done Done false
25882 7857 Acked Read 10 Sent Done Done false
25883 7858 Acked Write 5 Sent Done Done false
25884 7859 Acked Write 2 Sent Done Done false
25885 7860 Acked Read 9 Sent Done Done false
25886 7861 Acked Read 4 Sent Done Done false
25887 7862 Acked Write 7 Sent Done Done false
25888 7863 Acked Write 4 Sent Done Done false
25889 7864 Acked Read 8 Sent Done Done false
25890 7865 Acked Flush 0 Sent Done Done false
25891 7866 Acked Read 3 Sent Done Done false
25892 7867 Acked Read 5 Sent Done Done false
25893 7868 Acked Read 3 Sent Done Done false
25894 7869 Acked Write 4 Sent Done Done false
25895 7870 Acked Read 5 Sent Done Done false
25896 7871 Acked Flush 0 Sent Done Done false
25897 7872 Acked Write 10 Sent Done Done false
25898 7873 Acked Read 8 Sent Done Done false
25899 7874 Acked Write 10 Sent Done Done false
25900 7875 Acked Write 5 Sent Done Done false
25901 7876 Acked Write 10 Sent Done Done false
25902 7877 Acked Write 5 Sent Done Done false
25903 7878 Acked Write 8 Sent Done Done false
25904 7879 Acked Write 7 Sent Done Done false
25905 7880 Acked Flush 0 Sent Done Done false
25906 7881 Acked Write 9 Sent Done Done false
25907 7882 Acked Read 2 Sent Done Done false
25908 7883 Acked Write 9 Sent Done Done false
25909 7884 Acked Read 6 Sent Done Done false
25910 7885 Acked Write 5 Sent Done Done false
25911 7886 Acked Read 8 Sent Done Done false
25912 7887 Acked Write 3 Sent Done Done false
25913 7888 Acked Read 1 Sent Done Done false
25914 7889 Acked Read 8 Sent Done Done false
25915 7890 Acked Read 10 Sent Done Done false
25916 7891 Acked Read 1 Sent Done Done false
25917 7892 Acked Read 5 Sent Done Done false
25918 7893 Acked Read 1 Sent Done Done false
25919 7894 Acked Read 8 Sent Done Done false
25920 7895 Acked Read 7 Sent Done Done false
25921 7896 Acked Write 9 Sent Done Done false
25922 7897 Acked Read 8 Sent Done Done false
25923 7898 Acked Write 6 Sent Done Done false
25924 7899 Acked Read 1 Sent Done Done false
25925 7900 Acked Write 1 Sent Done Done false
25926 7901 Acked Write 7 Sent Done Done false
25927 7902 Acked Write 9 Sent Done Done false
25928 7903 Acked Write 10 Sent Done Done false
25929 7904 Acked Read 5 Sent Done Done false
25930 7905 Acked Flush 0 Sent Done Done false
25931 7906 Acked Write 9 Sent Done Done false
25932 7907 Acked Read 10 Sent Done Done false
25933 7908 Acked Write 3 Sent Done Done false
25934 7909 Acked Write 9 Sent Done Done false
25935 7910 Acked Read 1 Sent Done Done false
25936 7911 Acked Read 9 Sent Done Done false
25937 7912 Acked Read 5 Sent Done Done false
25938 7913 Acked Read 8 Sent Done Done false
25939 7914 Acked Flush 0 Sent Done Done false
25940 7915 Acked Read 1 Sent Done Done false
25941 7916 Acked Read 7 Sent Done Done false
25942 7917 Acked Write 2 Sent Done Done false
25943 7918 Acked Write 7 Sent Done Done false
25944 7919 Acked Read 10 Sent Done Done false
25945 7920 Acked Flush 0 Sent Done Done false
25946 7921 Acked Write 7 Sent Done Done false
25947 7922 Acked Read 2 Sent Done Done false
25948 7923 Acked Write 3 Sent Done Done false
25949 7924 Acked Write 4 Sent Done Done false
25950 7925 Acked Read 8 Sent Done Done false
25951 7926 Acked Flush 0 Sent Done Done false
25952 7927 Acked Read 7 Sent Done Done false
25953 7928 Acked Flush 0 Sent Done Done false
25954 7929 Acked Read 7 Sent Done Done false
25955 7930 Acked Read 3 Sent Done Done false
25956 7931 Acked Write 5 Sent Done Done false
25957 7932 Acked Read 2 Sent Done Done false
25958 7933 Acked Read 3 Sent Done Done false
25959 7934 Acked Read 2 Sent Done Done false
25960 7935 Acked Flush 0 Sent Done Done false
25961 7936 Acked Flush 0 Sent Done Done false
25962 7937 Acked Flush 0 Sent Done Done false
25963 7938 Acked Write 8 Sent Done Done false
25964 7939 Acked Read 3 Sent Done Done false
25965 7940 Acked Write 8 Sent Done Done false
25966 7941 Acked Read 6 Sent Done Done false
25967 7942 Acked Write 1 Sent Done Done false
25968 7943 Acked Flush 0 Sent Done Done false
25969 STATES DS:0 DS:1 DS:2 TOTAL
25970 Sent 301 0 0 301
25971 Done 0 301 301 602
25972 Skipped 0 0 0 0
25973 Error 0 0 0 0
25974 Last Flush: Some(JobId(7642)) Some(JobId(7943)) Some(JobId(7943))
25975 Downstairs last five completed: 7642 7641 7640 7639 7638
25976 Upstairs last five completed: 7943 7942 7941 7940 7939
25977 CLIENT: Up:0 ds:301 act:2
259782025-02-25T17:23:56.809ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
259792025-02-25T17:23:57.635ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
259802025-02-25T17:23:57.635ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
259812025-02-25T17:23:57.635ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
259822025-02-25T17:23:57.635ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
259832025-02-25T17:23:57.635ZINFOcrucible: client task is exiting = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
259842025-02-25T17:23:58.811ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
259852025-02-25T17:23:59.812ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
25986 ----------------------------------------------------------------
25987 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25988 States: Offline Active Active
25989 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25990 7643 Acked Flush 0 Sent Done Done false
25991 7644 Acked Flush 0 Sent Done Done false
25992 7645 Acked Flush 0 Sent Done Done false
25993 7646 Acked Write 3 Sent Done Done false
25994 7647 Acked Flush 0 Sent Done Done false
25995 7648 Acked Flush 0 Sent Done Done false
25996 7649 Acked Write 6 Sent Done Done false
25997 7650 Acked Read 8 Sent Done Done false
25998 7651 Acked Read 1 Sent Done Done false
25999 7652 Acked Flush 0 Sent Done Done false
26000 7653 Acked Read 3 Sent Done Done false
26001 7654 Acked Read 6 Sent Done Done false
26002 7655 Acked Read 6 Sent Done Done false
26003 7656 Acked Read 5 Sent Done Done false
26004 7657 Acked Flush 0 Sent Done Done false
26005 7658 Acked Read 6 Sent Done Done false
26006 7659 Acked Write 7 Sent Done Done false
26007 7660 Acked Write 8 Sent Done Done false
26008 7661 Acked Write 3 Sent Done Done false
26009 7662 Acked Read 1 Sent Done Done false
26010 7663 Acked Write 3 Sent Done Done false
26011 7664 Acked Read 6 Sent Done Done false
26012 7665 Acked Read 3 Sent Done Done false
26013 7666 Acked Write 10 Sent Done Done false
26014 7667 Acked Write 4 Sent Done Done false
26015 7668 Acked Read 1 Sent Done Done false
26016 7669 Acked Write 8 Sent Done Done false
26017 7670 Acked Read 7 Sent Done Done false
26018 7671 Acked Write 7 Sent Done Done false
26019 7672 Acked Write 10 Sent Done Done false
26020 7673 Acked Flush 0 Sent Done Done false
26021 7674 Acked Read 5 Sent Done Done false
26022 7675 Acked Write 6 Sent Done Done false
26023 7676 Acked Write 2 Sent Done Done false
26024 7677 Acked Read 1 Sent Done Done false
26025 7678 Acked Flush 0 Sent Done Done false
26026 7679 Acked Flush 0 Sent Done Done false
26027 7680 Acked Write 8 Sent Done Done false
26028 7681 Acked Read 9 Sent Done Done false
26029 7682 Acked Read 10 Sent Done Done false
26030 7683 Acked Write 2 Sent Done Done false
26031 7684 Acked Read 4 Sent Done Done false
26032 7685 Acked Write 3 Sent Done Done false
26033 7686 Acked Write 1 Sent Done Done false
26034 7687 Acked Read 8 Sent Done Done false
26035 7688 Acked Read 5 Sent Done Done false
26036 7689 Acked Flush 0 Sent Done Done false
26037 7690 Acked Write 4 Sent Done Done false
26038 7691 Acked Read 2 Sent Done Done false
26039 7692 Acked Write 7 Sent Done Done false
26040 7693 Acked Read 8 Sent Done Done false
26041 7694 Acked Write 5 Sent Done Done false
26042 7695 Acked Write 1 Sent Done Done false
26043 7696 Acked Write 7 Sent Done Done false
26044 7697 Acked Write 10 Sent Done Done false
26045 7698 Acked Read 4 Sent Done Done false
26046 7699 Acked Read 2 Sent Done Done false
26047 7700 Acked Read 6 Sent Done Done false
26048 7701 Acked Write 9 Sent Done Done false
26049 7702 Acked Read 4 Sent Done Done false
26050 7703 Acked Read 4 Sent Done Done false
26051 7704 Acked Read 5 Sent Done Done false
26052 7705 Acked Write 3 Sent Done Done false
26053 7706 Acked Write 7 Sent Done Done false
26054 7707 Acked Write 3 Sent Done Done false
26055 7708 Acked Read 5 Sent Done Done false
26056 7709 Acked Read 5 Sent Done Done false
26057 7710 Acked Read 1 Sent Done Done false
26058 7711 Acked Read 6 Sent Done Done false
26059 7712 Acked Flush 0 Sent Done Done false
26060 7713 Acked Write 6 Sent Done Done false
26061 7714 Acked Write 1 Sent Done Done false
26062 7715 Acked Read 1 Sent Done Done false
26063 7716 Acked Write 9 Sent Done Done false
26064 7717 Acked Read 10 Sent Done Done false
26065 7718 Acked Read 3 Sent Done Done false
26066 7719 Acked Flush 0 Sent Done Done false
26067 7720 Acked Read 4 Sent Done Done false
26068 7721 Acked Write 10 Sent Done Done false
26069 7722 Acked Write 1 Sent Done Done false
26070 7723 Acked Read 7 Sent Done Done false
26071 7724 Acked Read 5 Sent Done Done false
26072 7725 Acked Flush 0 Sent Done Done false
26073 7726 Acked Read 4 Sent Done Done false
26074 7727 Acked Read 1 Sent Done Done false
26075 7728 Acked Read 9 Sent Done Done false
26076 7729 Acked Write 9 Sent Done Done false
26077 7730 Acked Read 10 Sent Done Done false
26078 7731 Acked Write 9 Sent Done Done false
26079 7732 Acked Read 6 Sent Done Done false
26080 7733 Acked Flush 0 Sent Done Done false
26081 7734 Acked Read 9 Sent Done Done false
26082 7735 Acked Write 10 Sent Done Done false
26083 7736 Acked Read 4 Sent Done Done false
26084 7737 Acked Read 3 Sent Done Done false
26085 7738 Acked Write 10 Sent Done Done false
26086 7739 Acked Flush 0 Sent Done Done false
26087 7740 Acked Read 4 Sent Done Done false
26088 7741 Acked Read 8 Sent Done Done false
26089 7742 Acked Write 1 Sent Done Done false
26090 7743 Acked Write 1 Sent Done Done false
26091 7744 Acked Flush 0 Sent Done Done false
26092 7745 Acked Read 10 Sent Done Done false
26093 7746 Acked Flush 0 Sent Done Done false
26094 7747 Acked Write 7 Sent Done Done false
26095 7748 Acked Write 10 Sent Done Done false
26096 7749 Acked Read 4 Sent Done Done false
26097 7750 Acked Write 2 Sent Done Done false
26098 7751 Acked Read 9 Sent Done Done false
26099 7752 Acked Flush 0 Sent Done Done false
26100 7753 Acked Write 7 Sent Done Done false
26101 7754 Acked Write 3 Sent Done Done false
26102 7755 Acked Read 1 Sent Done Done false
26103 7756 Acked Read 10 Sent Done Done false
26104 7757 Acked Read 3 Sent Done Done false
26105 7758 Acked Read 6 Sent Done Done false
26106 7759 Acked Read 4 Sent Done Done false
26107 7760 Acked Write 3 Sent Done Done false
26108 7761 Acked Write 9 Sent Done Done false
26109 7762 Acked Write 3 Sent Done Done false
26110 7763 Acked Read 9 Sent Done Done false
26111 7764 Acked Write 1 Sent Done Done false
26112 7765 Acked Write 1 Sent Done Done false
26113 7766 Acked Write 6 Sent Done Done false
26114 7767 Acked Read 2 Sent Done Done false
26115 7768 Acked Write 10 Sent Done Done false
26116 7769 Acked Read 6 Sent Done Done false
26117 7770 Acked Write 1 Sent Done Done false
26118 7771 Acked Write 9 Sent Done Done false
26119 7772 Acked Read 6 Sent Done Done false
26120 7773 Acked Read 6 Sent Done Done false
26121 7774 Acked Read 4 Sent Done Done false
26122 7775 Acked Write 9 Sent Done Done false
26123 7776 Acked Write 1 Sent Done Done false
26124 7777 Acked Flush 0 Sent Done Done false
26125 7778 Acked Read 6 Sent Done Done false
26126 7779 Acked Read 3 Sent Done Done false
26127 7780 Acked Read 3 Sent Done Done false
26128 7781 Acked Read 8 Sent Done Done false
26129 7782 Acked Read 3 Sent Done Done false
26130 7783 Acked Write 3 Sent Done Done false
26131 7784 Acked Read 8 Sent Done Done false
26132 7785 Acked Read 4 Sent Done Done false
26133 7786 Acked Flush 0 Sent Done Done false
26134 7787 Acked Read 7 Sent Done Done false
26135 7788 Acked Write 10 Sent Done Done false
26136 7789 Acked Read 7 Sent Done Done false
26137 7790 Acked Read 9 Sent Done Done false
26138 7791 Acked Read 10 Sent Done Done false
26139 7792 Acked Read 4 Sent Done Done false
26140 7793 Acked Write 5 Sent Done Done false
26141 7794 Acked Write 9 Sent Done Done false
26142 7795 Acked Write 6 Sent Done Done false
26143 7796 Acked Read 1 Sent Done Done false
26144 7797 Acked Write 9 Sent Done Done false
26145 7798 Acked Read 9 Sent Done Done false
26146 7799 Acked Write 8 Sent Done Done false
26147 7800 Acked Write 7 Sent Done Done false
26148 7801 Acked Read 2 Sent Done Done false
26149 7802 Acked Write 4 Sent Done Done false
26150 7803 Acked Read 3 Sent Done Done false
26151 7804 Acked Read 4 Sent Done Done false
26152 7805 Acked Write 2 Sent Done Done false
26153 7806 Acked Write 6 Sent Done Done false
26154 7807 Acked Read 4 Sent Done Done false
26155 7808 Acked Read 2 Sent Done Done false
26156 7809 Acked Write 2 Sent Done Done false
26157 7810 Acked Flush 0 Sent Done Done false
26158 7811 Acked Read 8 Sent Done Done false
26159 7812 Acked Write 9 Sent Done Done false
26160 7813 Acked Read 9 Sent Done Done false
26161 7814 Acked Write 6 Sent Done Done false
26162 7815 Acked Read 2 Sent Done Done false
26163 7816 Acked Read 1 Sent Done Done false
26164 7817 Acked Write 2 Sent Done Done false
26165 7818 Acked Flush 0 Sent Done Done false
26166 7819 Acked Read 7 Sent Done Done false
26167 7820 Acked Write 2 Sent Done Done false
26168 7821 Acked Read 1 Sent Done Done false
26169 7822 Acked Flush 0 Sent Done Done false
26170 7823 Acked Flush 0 Sent Done Done false
26171 7824 Acked Read 10 Sent Done Done false
26172 7825 Acked Read 5 Sent Done Done false
26173 7826 Acked Read 3 Sent Done Done false
26174 7827 Acked Write 7 Sent Done Done false
26175 7828 Acked Read 8 Sent Done Done false
26176 7829 Acked Read 8 Sent Done Done false
26177 7830 Acked Write 8 Sent Done Done false
26178 7831 Acked Write 1 Sent Done Done false
26179 7832 Acked Read 9 Sent Done Done false
26180 7833 Acked Write 5 Sent Done Done false
26181 7834 Acked Write 1 Sent Done Done false
26182 7835 Acked Read 10 Sent Done Done false
26183 7836 Acked Flush 0 Sent Done Done false
26184 7837 Acked Write 4 Sent Done Done false
26185 7838 Acked Flush 0 Sent Done Done false
26186 7839 Acked Flush 0 Sent Done Done false
26187 7840 Acked Write 4 Sent Done Done false
26188 7841 Acked Read 6 Sent Done Done false
26189 7842 Acked Flush 0 Sent Done Done false
26190 7843 Acked Write 8 Sent Done Done false
26191 7844 Acked Read 9 Sent Done Done false
26192 7845 Acked Read 10 Sent Done Done false
26193 7846 Acked Write 5 Sent Done Done false
26194 7847 Acked Read 7 Sent Done Done false
26195 7848 Acked Read 3 Sent Done Done false
26196 7849 Acked Read 3 Sent Done Done false
26197 7850 Acked Write 9 Sent Done Done false
26198 7851 Acked Read 5 Sent Done Done false
26199 7852 Acked Read 6 Sent Done Done false
26200 7853 Acked Read 4 Sent Done Done false
26201 7854 Acked Read 4 Sent Done Done false
26202 7855 Acked Write 3 Sent Done Done false
26203 7856 Acked Read 9 Sent Done Done false
26204 7857 Acked Read 10 Sent Done Done false
26205 7858 Acked Write 5 Sent Done Done false
26206 7859 Acked Write 2 Sent Done Done false
26207 7860 Acked Read 9 Sent Done Done false
26208 7861 Acked Read 4 Sent Done Done false
26209 7862 Acked Write 7 Sent Done Done false
26210 7863 Acked Write 4 Sent Done Done false
26211 7864 Acked Read 8 Sent Done Done false
26212 7865 Acked Flush 0 Sent Done Done false
26213 7866 Acked Read 3 Sent Done Done false
26214 7867 Acked Read 5 Sent Done Done false
26215 7868 Acked Read 3 Sent Done Done false
26216 7869 Acked Write 4 Sent Done Done false
26217 7870 Acked Read 5 Sent Done Done false
26218 7871 Acked Flush 0 Sent Done Done false
26219 7872 Acked Write 10 Sent Done Done false
26220 7873 Acked Read 8 Sent Done Done false
26221 7874 Acked Write 10 Sent Done Done false
26222 7875 Acked Write 5 Sent Done Done false
26223 7876 Acked Write 10 Sent Done Done false
26224 7877 Acked Write 5 Sent Done Done false
26225 7878 Acked Write 8 Sent Done Done false
26226 7879 Acked Write 7 Sent Done Done false
26227 7880 Acked Flush 0 Sent Done Done false
26228 7881 Acked Write 9 Sent Done Done false
26229 7882 Acked Read 2 Sent Done Done false
26230 7883 Acked Write 9 Sent Done Done false
26231 7884 Acked Read 6 Sent Done Done false
26232 7885 Acked Write 5 Sent Done Done false
26233 7886 Acked Read 8 Sent Done Done false
26234 7887 Acked Write 3 Sent Done Done false
26235 7888 Acked Read 1 Sent Done Done false
26236 7889 Acked Read 8 Sent Done Done false
26237 7890 Acked Read 10 Sent Done Done false
26238 7891 Acked Read 1 Sent Done Done false
26239 7892 Acked Read 5 Sent Done Done false
26240 7893 Acked Read 1 Sent Done Done false
26241 7894 Acked Read 8 Sent Done Done false
26242 7895 Acked Read 7 Sent Done Done false
26243 7896 Acked Write 9 Sent Done Done false
26244 7897 Acked Read 8 Sent Done Done false
26245 7898 Acked Write 6 Sent Done Done false
26246 7899 Acked Read 1 Sent Done Done false
26247 7900 Acked Write 1 Sent Done Done false
26248 7901 Acked Write 7 Sent Done Done false
26249 7902 Acked Write 9 Sent Done Done false
26250 7903 Acked Write 10 Sent Done Done false
26251 7904 Acked Read 5 Sent Done Done false
26252 7905 Acked Flush 0 Sent Done Done false
26253 7906 Acked Write 9 Sent Done Done false
26254 7907 Acked Read 10 Sent Done Done false
26255 7908 Acked Write 3 Sent Done Done false
26256 7909 Acked Write 9 Sent Done Done false
26257 7910 Acked Read 1 Sent Done Done false
26258 7911 Acked Read 9 Sent Done Done false
26259 7912 Acked Read 5 Sent Done Done false
26260 7913 Acked Read 8 Sent Done Done false
26261 7914 Acked Flush 0 Sent Done Done false
26262 7915 Acked Read 1 Sent Done Done false
26263 7916 Acked Read 7 Sent Done Done false
26264 7917 Acked Write 2 Sent Done Done false
26265 7918 Acked Write 7 Sent Done Done false
26266 7919 Acked Read 10 Sent Done Done false
26267 7920 Acked Flush 0 Sent Done Done false
26268 7921 Acked Write 7 Sent Done Done false
26269 7922 Acked Read 2 Sent Done Done false
26270 7923 Acked Write 3 Sent Done Done false
26271 7924 Acked Write 4 Sent Done Done false
26272 7925 Acked Read 8 Sent Done Done false
26273 7926 Acked Flush 0 Sent Done Done false
26274 7927 Acked Read 7 Sent Done Done false
26275 7928 Acked Flush 0 Sent Done Done false
26276 7929 Acked Read 7 Sent Done Done false
26277 7930 Acked Read 3 Sent Done Done false
26278 7931 Acked Write 5 Sent Done Done false
26279 7932 Acked Read 2 Sent Done Done false
26280 7933 Acked Read 3 Sent Done Done false
26281 7934 Acked Read 2 Sent Done Done false
26282 7935 Acked Flush 0 Sent Done Done false
26283 7936 Acked Flush 0 Sent Done Done false
26284 7937 Acked Flush 0 Sent Done Done false
26285 7938 Acked Write 8 Sent Done Done false
26286 7939 Acked Read 3 Sent Done Done false
26287 7940 Acked Write 8 Sent Done Done false
26288 7941 Acked Read 6 Sent Done Done false
26289 7942 Acked Write 1 Sent Done Done false
26290 7943 Acked Flush 0 Sent Done Done false
26291 STATES DS:0 DS:1 DS:2 TOTAL
26292 Sent 301 0 0 301
26293 Done 0 301 301 602
26294 Skipped 0 0 0 0
26295 Error 0 0 0 0
26296 Last Flush: Some(JobId(7642)) Some(JobId(7943)) Some(JobId(7943))
26297 Downstairs last five completed: 7642 7641 7640 7639 7638
26298 Upstairs last five completed: 7943 7942 7941 7940 7939
26299 CLIENT: Up:0 ds:301 act:2
263002025-02-25T17:24:00.871ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
26301 [8810] got start action
26302 [8810] I am going to restart
26303 Starting downstairs at port 8810
26304 Make output file at "/tmp/dsc/downstairs-8810.txt"
26305 [8810][0] reports Starting
26306 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1231)
26307 [8810][0] reports Running
263082025-02-25T17:24:01.815ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
263092025-02-25T17:24:02.817ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
263102025-02-25T17:24:03.818ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
26311 ----------------------------------------------------------------
26312 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26313 States: Offline Active Active
26314 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26315 7643 Acked Flush 0 Sent Done Done false
26316 7644 Acked Flush 0 Sent Done Done false
26317 7645 Acked Flush 0 Sent Done Done false
26318 7646 Acked Write 3 Sent Done Done false
26319 7647 Acked Flush 0 Sent Done Done false
26320 7648 Acked Flush 0 Sent Done Done false
26321 7649 Acked Write 6 Sent Done Done false
26322 7650 Acked Read 8 Sent Done Done false
26323 7651 Acked Read 1 Sent Done Done false
26324 7652 Acked Flush 0 Sent Done Done false
26325 7653 Acked Read 3 Sent Done Done false
26326 7654 Acked Read 6 Sent Done Done false
26327 7655 Acked Read 6 Sent Done Done false
26328 7656 Acked Read 5 Sent Done Done false
26329 7657 Acked Flush 0 Sent Done Done false
26330 7658 Acked Read 6 Sent Done Done false
26331 7659 Acked Write 7 Sent Done Done false
26332 7660 Acked Write 8 Sent Done Done false
26333 7661 Acked Write 3 Sent Done Done false
26334 7662 Acked Read 1 Sent Done Done false
26335 7663 Acked Write 3 Sent Done Done false
26336 7664 Acked Read 6 Sent Done Done false
26337 7665 Acked Read 3 Sent Done Done false
26338 7666 Acked Write 10 Sent Done Done false
26339 7667 Acked Write 4 Sent Done Done false
26340 7668 Acked Read 1 Sent Done Done false
26341 7669 Acked Write 8 Sent Done Done false
26342 7670 Acked Read 7 Sent Done Done false
26343 7671 Acked Write 7 Sent Done Done false
26344 7672 Acked Write 10 Sent Done Done false
26345 7673 Acked Flush 0 Sent Done Done false
26346 7674 Acked Read 5 Sent Done Done false
26347 7675 Acked Write 6 Sent Done Done false
26348 7676 Acked Write 2 Sent Done Done false
26349 7677 Acked Read 1 Sent Done Done false
26350 7678 Acked Flush 0 Sent Done Done false
26351 7679 Acked Flush 0 Sent Done Done false
26352 7680 Acked Write 8 Sent Done Done false
26353 7681 Acked Read 9 Sent Done Done false
26354 7682 Acked Read 10 Sent Done Done false
26355 7683 Acked Write 2 Sent Done Done false
26356 7684 Acked Read 4 Sent Done Done false
26357 7685 Acked Write 3 Sent Done Done false
26358 7686 Acked Write 1 Sent Done Done false
26359 7687 Acked Read 8 Sent Done Done false
26360 7688 Acked Read 5 Sent Done Done false
26361 7689 Acked Flush 0 Sent Done Done false
26362 7690 Acked Write 4 Sent Done Done false
26363 7691 Acked Read 2 Sent Done Done false
26364 7692 Acked Write 7 Sent Done Done false
26365 7693 Acked Read 8 Sent Done Done false
26366 7694 Acked Write 5 Sent Done Done false
26367 7695 Acked Write 1 Sent Done Done false
26368 7696 Acked Write 7 Sent Done Done false
26369 7697 Acked Write 10 Sent Done Done false
26370 7698 Acked Read 4 Sent Done Done false
26371 7699 Acked Read 2 Sent Done Done false
26372 7700 Acked Read 6 Sent Done Done false
26373 7701 Acked Write 9 Sent Done Done false
26374 7702 Acked Read 4 Sent Done Done false
26375 7703 Acked Read 4 Sent Done Done false
26376 7704 Acked Read 5 Sent Done Done false
26377 7705 Acked Write 3 Sent Done Done false
26378 7706 Acked Write 7 Sent Done Done false
26379 7707 Acked Write 3 Sent Done Done false
26380 7708 Acked Read 5 Sent Done Done false
26381 7709 Acked Read 5 Sent Done Done false
26382 7710 Acked Read 1 Sent Done Done false
26383 7711 Acked Read 6 Sent Done Done false
26384 7712 Acked Flush 0 Sent Done Done false
26385 7713 Acked Write 6 Sent Done Done false
26386 7714 Acked Write 1 Sent Done Done false
26387 7715 Acked Read 1 Sent Done Done false
26388 7716 Acked Write 9 Sent Done Done false
26389 7717 Acked Read 10 Sent Done Done false
26390 7718 Acked Read 3 Sent Done Done false
26391 7719 Acked Flush 0 Sent Done Done false
26392 7720 Acked Read 4 Sent Done Done false
26393 7721 Acked Write 10 Sent Done Done false
26394 7722 Acked Write 1 Sent Done Done false
26395 7723 Acked Read 7 Sent Done Done false
26396 7724 Acked Read 5 Sent Done Done false
26397 7725 Acked Flush 0 Sent Done Done false
26398 7726 Acked Read 4 Sent Done Done false
26399 7727 Acked Read 1 Sent Done Done false
26400 7728 Acked Read 9 Sent Done Done false
26401 7729 Acked Write 9 Sent Done Done false
26402 7730 Acked Read 10 Sent Done Done false
26403 7731 Acked Write 9 Sent Done Done false
26404 7732 Acked Read 6 Sent Done Done false
26405 7733 Acked Flush 0 Sent Done Done false
26406 7734 Acked Read 9 Sent Done Done false
26407 7735 Acked Write 10 Sent Done Done false
26408 7736 Acked Read 4 Sent Done Done false
26409 7737 Acked Read 3 Sent Done Done false
26410 7738 Acked Write 10 Sent Done Done false
26411 7739 Acked Flush 0 Sent Done Done false
26412 7740 Acked Read 4 Sent Done Done false
26413 7741 Acked Read 8 Sent Done Done false
26414 7742 Acked Write 1 Sent Done Done false
26415 7743 Acked Write 1 Sent Done Done false
26416 7744 Acked Flush 0 Sent Done Done false
26417 7745 Acked Read 10 Sent Done Done false
26418 7746 Acked Flush 0 Sent Done Done false
26419 7747 Acked Write 7 Sent Done Done false
26420 7748 Acked Write 10 Sent Done Done false
26421 7749 Acked Read 4 Sent Done Done false
26422 7750 Acked Write 2 Sent Done Done false
26423 7751 Acked Read 9 Sent Done Done false
26424 7752 Acked Flush 0 Sent Done Done false
26425 7753 Acked Write 7 Sent Done Done false
26426 7754 Acked Write 3 Sent Done Done false
26427 7755 Acked Read 1 Sent Done Done false
26428 7756 Acked Read 10 Sent Done Done false
26429 7757 Acked Read 3 Sent Done Done false
26430 7758 Acked Read 6 Sent Done Done false
26431 7759 Acked Read 4 Sent Done Done false
26432 7760 Acked Write 3 Sent Done Done false
26433 7761 Acked Write 9 Sent Done Done false
26434 7762 Acked Write 3 Sent Done Done false
26435 7763 Acked Read 9 Sent Done Done false
26436 7764 Acked Write 1 Sent Done Done false
26437 7765 Acked Write 1 Sent Done Done false
26438 7766 Acked Write 6 Sent Done Done false
26439 7767 Acked Read 2 Sent Done Done false
26440 7768 Acked Write 10 Sent Done Done false
26441 7769 Acked Read 6 Sent Done Done false
26442 7770 Acked Write 1 Sent Done Done false
26443 7771 Acked Write 9 Sent Done Done false
26444 7772 Acked Read 6 Sent Done Done false
26445 7773 Acked Read 6 Sent Done Done false
26446 7774 Acked Read 4 Sent Done Done false
26447 7775 Acked Write 9 Sent Done Done false
26448 7776 Acked Write 1 Sent Done Done false
26449 7777 Acked Flush 0 Sent Done Done false
26450 7778 Acked Read 6 Sent Done Done false
26451 7779 Acked Read 3 Sent Done Done false
26452 7780 Acked Read 3 Sent Done Done false
26453 7781 Acked Read 8 Sent Done Done false
26454 7782 Acked Read 3 Sent Done Done false
26455 7783 Acked Write 3 Sent Done Done false
26456 7784 Acked Read 8 Sent Done Done false
26457 7785 Acked Read 4 Sent Done Done false
26458 7786 Acked Flush 0 Sent Done Done false
26459 7787 Acked Read 7 Sent Done Done false
26460 7788 Acked Write 10 Sent Done Done false
26461 7789 Acked Read 7 Sent Done Done false
26462 7790 Acked Read 9 Sent Done Done false
26463 7791 Acked Read 10 Sent Done Done false
26464 7792 Acked Read 4 Sent Done Done false
26465 7793 Acked Write 5 Sent Done Done false
26466 7794 Acked Write 9 Sent Done Done false
26467 7795 Acked Write 6 Sent Done Done false
26468 7796 Acked Read 1 Sent Done Done false
26469 7797 Acked Write 9 Sent Done Done false
26470 7798 Acked Read 9 Sent Done Done false
26471 7799 Acked Write 8 Sent Done Done false
26472 7800 Acked Write 7 Sent Done Done false
26473 7801 Acked Read 2 Sent Done Done false
26474 7802 Acked Write 4 Sent Done Done false
26475 7803 Acked Read 3 Sent Done Done false
26476 7804 Acked Read 4 Sent Done Done false
26477 7805 Acked Write 2 Sent Done Done false
26478 7806 Acked Write 6 Sent Done Done false
26479 7807 Acked Read 4 Sent Done Done false
26480 7808 Acked Read 2 Sent Done Done false
26481 7809 Acked Write 2 Sent Done Done false
26482 7810 Acked Flush 0 Sent Done Done false
26483 7811 Acked Read 8 Sent Done Done false
26484 7812 Acked Write 9 Sent Done Done false
26485 7813 Acked Read 9 Sent Done Done false
26486 7814 Acked Write 6 Sent Done Done false
26487 7815 Acked Read 2 Sent Done Done false
26488 7816 Acked Read 1 Sent Done Done false
26489 7817 Acked Write 2 Sent Done Done false
26490 7818 Acked Flush 0 Sent Done Done false
26491 7819 Acked Read 7 Sent Done Done false
26492 7820 Acked Write 2 Sent Done Done false
26493 7821 Acked Read 1 Sent Done Done false
26494 7822 Acked Flush 0 Sent Done Done false
26495 7823 Acked Flush 0 Sent Done Done false
26496 7824 Acked Read 10 Sent Done Done false
26497 7825 Acked Read 5 Sent Done Done false
26498 7826 Acked Read 3 Sent Done Done false
26499 7827 Acked Write 7 Sent Done Done false
26500 7828 Acked Read 8 Sent Done Done false
26501 7829 Acked Read 8 Sent Done Done false
26502 7830 Acked Write 8 Sent Done Done false
26503 7831 Acked Write 1 Sent Done Done false
26504 7832 Acked Read 9 Sent Done Done false
26505 7833 Acked Write 5 Sent Done Done false
26506 7834 Acked Write 1 Sent Done Done false
26507 7835 Acked Read 10 Sent Done Done false
26508 7836 Acked Flush 0 Sent Done Done false
26509 7837 Acked Write 4 Sent Done Done false
26510 7838 Acked Flush 0 Sent Done Done false
26511 7839 Acked Flush 0 Sent Done Done false
26512 7840 Acked Write 4 Sent Done Done false
26513 7841 Acked Read 6 Sent Done Done false
26514 7842 Acked Flush 0 Sent Done Done false
26515 7843 Acked Write 8 Sent Done Done false
26516 7844 Acked Read 9 Sent Done Done false
26517 7845 Acked Read 10 Sent Done Done false
26518 7846 Acked Write 5 Sent Done Done false
26519 7847 Acked Read 7 Sent Done Done false
26520 7848 Acked Read 3 Sent Done Done false
26521 7849 Acked Read 3 Sent Done Done false
26522 7850 Acked Write 9 Sent Done Done false
26523 7851 Acked Read 5 Sent Done Done false
26524 7852 Acked Read 6 Sent Done Done false
26525 7853 Acked Read 4 Sent Done Done false
26526 7854 Acked Read 4 Sent Done Done false
26527 7855 Acked Write 3 Sent Done Done false
26528 7856 Acked Read 9 Sent Done Done false
26529 7857 Acked Read 10 Sent Done Done false
26530 7858 Acked Write 5 Sent Done Done false
26531 7859 Acked Write 2 Sent Done Done false
26532 7860 Acked Read 9 Sent Done Done false
26533 7861 Acked Read 4 Sent Done Done false
26534 7862 Acked Write 7 Sent Done Done false
26535 7863 Acked Write 4 Sent Done Done false
26536 7864 Acked Read 8 Sent Done Done false
26537 7865 Acked Flush 0 Sent Done Done false
26538 7866 Acked Read 3 Sent Done Done false
26539 7867 Acked Read 5 Sent Done Done false
26540 7868 Acked Read 3 Sent Done Done false
26541 7869 Acked Write 4 Sent Done Done false
26542 7870 Acked Read 5 Sent Done Done false
26543 7871 Acked Flush 0 Sent Done Done false
26544 7872 Acked Write 10 Sent Done Done false
26545 7873 Acked Read 8 Sent Done Done false
26546 7874 Acked Write 10 Sent Done Done false
26547 7875 Acked Write 5 Sent Done Done false
26548 7876 Acked Write 10 Sent Done Done false
26549 7877 Acked Write 5 Sent Done Done false
26550 7878 Acked Write 8 Sent Done Done false
26551 7879 Acked Write 7 Sent Done Done false
26552 7880 Acked Flush 0 Sent Done Done false
26553 7881 Acked Write 9 Sent Done Done false
26554 7882 Acked Read 2 Sent Done Done false
26555 7883 Acked Write 9 Sent Done Done false
26556 7884 Acked Read 6 Sent Done Done false
26557 7885 Acked Write 5 Sent Done Done false
26558 7886 Acked Read 8 Sent Done Done false
26559 7887 Acked Write 3 Sent Done Done false
26560 7888 Acked Read 1 Sent Done Done false
26561 7889 Acked Read 8 Sent Done Done false
26562 7890 Acked Read 10 Sent Done Done false
26563 7891 Acked Read 1 Sent Done Done false
26564 7892 Acked Read 5 Sent Done Done false
26565 7893 Acked Read 1 Sent Done Done false
26566 7894 Acked Read 8 Sent Done Done false
26567 7895 Acked Read 7 Sent Done Done false
26568 7896 Acked Write 9 Sent Done Done false
26569 7897 Acked Read 8 Sent Done Done false
26570 7898 Acked Write 6 Sent Done Done false
26571 7899 Acked Read 1 Sent Done Done false
26572 7900 Acked Write 1 Sent Done Done false
26573 7901 Acked Write 7 Sent Done Done false
26574 7902 Acked Write 9 Sent Done Done false
26575 7903 Acked Write 10 Sent Done Done false
26576 7904 Acked Read 5 Sent Done Done false
26577 7905 Acked Flush 0 Sent Done Done false
26578 7906 Acked Write 9 Sent Done Done false
26579 7907 Acked Read 10 Sent Done Done false
26580 7908 Acked Write 3 Sent Done Done false
26581 7909 Acked Write 9 Sent Done Done false
26582 7910 Acked Read 1 Sent Done Done false
26583 7911 Acked Read 9 Sent Done Done false
26584 7912 Acked Read 5 Sent Done Done false
26585 7913 Acked Read 8 Sent Done Done false
26586 7914 Acked Flush 0 Sent Done Done false
26587 7915 Acked Read 1 Sent Done Done false
26588 7916 Acked Read 7 Sent Done Done false
26589 7917 Acked Write 2 Sent Done Done false
26590 7918 Acked Write 7 Sent Done Done false
26591 7919 Acked Read 10 Sent Done Done false
26592 7920 Acked Flush 0 Sent Done Done false
26593 7921 Acked Write 7 Sent Done Done false
26594 7922 Acked Read 2 Sent Done Done false
26595 7923 Acked Write 3 Sent Done Done false
26596 7924 Acked Write 4 Sent Done Done false
26597 7925 Acked Read 8 Sent Done Done false
26598 7926 Acked Flush 0 Sent Done Done false
26599 7927 Acked Read 7 Sent Done Done false
26600 7928 Acked Flush 0 Sent Done Done false
26601 7929 Acked Read 7 Sent Done Done false
26602 7930 Acked Read 3 Sent Done Done false
26603 7931 Acked Write 5 Sent Done Done false
26604 7932 Acked Read 2 Sent Done Done false
26605 7933 Acked Read 3 Sent Done Done false
26606 7934 Acked Read 2 Sent Done Done false
26607 7935 Acked Flush 0 Sent Done Done false
26608 7936 Acked Flush 0 Sent Done Done false
26609 7937 Acked Flush 0 Sent Done Done false
26610 7938 Acked Write 8 Sent Done Done false
26611 7939 Acked Read 3 Sent Done Done false
26612 7940 Acked Write 8 Sent Done Done false
26613 7941 Acked Read 6 Sent Done Done false
26614 7942 Acked Write 1 Sent Done Done false
26615 7943 Acked Flush 0 Sent Done Done false
26616 STATES DS:0 DS:1 DS:2 TOTAL
26617 Sent 301 0 0 301
26618 Done 0 301 301 602
26619 Skipped 0 0 0 0
26620 Error 0 0 0 0
26621 Last Flush: Some(JobId(7642)) Some(JobId(7943)) Some(JobId(7943))
26622 Downstairs last five completed: 7642 7641 7640 7639 7638
26623 Upstairs last five completed: 7943 7942 7941 7940 7939
26624 CLIENT: Up:0 ds:301 act:2
266252025-02-25T17:24:04.819ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
266262025-02-25T17:24:05.872ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
266272025-02-25T17:24:06.822ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
266282025-02-25T17:24:07.636ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
266292025-02-25T17:24:07.636ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49127) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
266302025-02-25T17:24:07.637ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
266312025-02-25T17:24:07.638ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
266322025-02-25T17:24:07.638ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
266332025-02-25T17:24:07.638ZINFOcrucible: send last flush ID to this DS: Some(JobId(7642)) = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
266342025-02-25T17:24:07.638ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
266352025-02-25T17:24:07.638ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(7642)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
266362025-02-25T17:24:07.638ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(7642)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
26637 ----------------------------------------------------------------
26638 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26639 Downstairs last five completed: 7943 7942 7941 7940 7939
26640 Upstairs last five completed: 7943 7942 7941 7940 7939
26641 CLIENT: Up:0 ds:0 act:3
26642 Replay: All jobs finished
26643 Main task has work to do, go find it
26644 got dsc Stop(0)
26645 stop 0
26646 Feb 25 17:24:07.900 INFO request completed, latency_us: 361, response_code: 204, uri: /stop/cid/0, method: POST, req_id:[8810] Got stop action so:false kr:false
26647 5786d60b-1608-4a71-abf9-8f4ee75016fa, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
266482025-02-25T17:24:07.901ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
266492025-02-25T17:24:07.901ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
26650 Feb 25 17:24:07.901 INFO request completed, latency_us: {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:24:07.901198002Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
26651 220, response_code: 200, uri: /state/cid/0, method: GET, req_id: c02b1086-ae0c-4949-9707-87e2fd9cfc97, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
266522025-02-25T17:24:07.901ZINFOcrucible: client task is exiting = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
26653 [8810] Exited with: ExitStatus(unix_wait_status(9))
26654 [8810][0] reports Exit
26655 Feb 25 17:24:11.903 INFO request completed, latency_us: 250, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1fadaa4e-9033-40d1-b9b3-86059dc0c46c, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
26656 001/300 Read block 1221 len 8192
26657 002/300 Write block 1094 len 36864 data: 7 6 6 7 7 6 7 6 7
26658 003/300 Flush
26659 004/300 Read block 1074 len 4096
26660 005/300 Write block 2550 len 20480 data: 3 4 5 4 5
26661 006/300 Read block 1403 len 12288
26662 007/300 Write block 696 len 36864 data: 4 4 4 4 5 4 3 2 2
26663 008/300 Read block 1571 len 24576
26664 009/300 Write block 4649 len 28672 data: 2 2 3 3 3 3 3
26665 010/300 Read block 2425 len 40960
26666 011/300 Write block 525 len 12288 data: 5 5 4
26667 012/300 Write block 2165 len 16384 data: 4 4 5 4
26668 013/300 Read block 4540 len 20480
26669 014/300 Write block 2221 len 28672 data: 7 8 8 7 5 3 3
26670 015/300 Read block 1439 len 40960
26671 016/300 Read block 4440 len 28672
26672 017/300 Write block 1590 len 4096 data: 3
26673 018/300 Write block 1788 len 8192 data: 6 7
26674 019/300 Flush
26675 020/300 Read block 350 len 4096
26676 021/300 Read block 1118 len 12288
26677 022/300 Write block 454 len 20480 data: 5 7 9 8 6
26678 023/300 Read block 3849 len 40960
26679 024/300 Write block 1415 len 20480 data: 7 4 3 3 4
26680 025/300 Write block 3074 len 16384 data: 4 5 7 5
26681 026/300 Write block 648 len 12288 data: 3 2 4
26682 027/300 Flush
26683 028/300 Read block 1919 len 28672
26684 029/300 Write block 4590 len 12288 data: 6 8 7
26685 030/300 Read block 2879 len 20480
26686 031/300 Read block 3609 len 12288
26687 032/300 Write block 3945 len 24576 data: 8 6 7 7 6 6
26688 033/300 Read block 4307 len 40960
26689 034/300 Flush
26690 035/300 Flush
26691 036/300 Read block 3054 len 8192
26692 037/300 Read block 2029 len 40960
26693 038/300 Write block 1698 len 32768 data: 4 5 7 7 8 8 10 10
26694 039/300 Read block 152 len 16384
26695 040/300 Write block 775 len 36864 data: 2 4 4 4 5 5 5 5 4
26696 041/300 Flush
26697 042/300 Flush
26698 043/300 Write block 3136 len 28672 data: 5 5 5 6 7 6 5
26699 044/300 Read block 2492 len 36864
26700 045/300 Read block 2082 len 12288
26701 046/300 Write block 1697 len 28672 data: 4 5 6 8 8 9 9
26702 047/300 Read block 2312 len 40960
26703 048/300 Write block 2269 len 24576 data: 4 3 3 4 3 4
26704 049/300 Write block 2992 len 32768 data: 7 7 9 9 8 6 8 8
26705 050/300 Write block 188 len 16384 data: 6 5 5 3
26706 051/300 Write block 4731 len 36864 data: 5 4 4 4 3 3 2 2 2
26707 052/300 Read block 4059 len 8192
26708 053/300 Write block 1214 len 24576 data: 4 6 5 5 4 4
26709 054/300 Write block 1479 len 12288 data: 6 8 8
26710 055/300 Flush
26711 056/300 Read block 1703 len 32768
26712 057/300 Read block 2736 len 20480
26713 058/300 Write block 1699 len 40960 data: 7 9 9 10 10 11 11 11 8 6
26714 059/300 Read block 3478 len 8192
26715 060/300 Write block 431 len 32768 data: 3 4 4 3 2 2 2 5
26716 061/300 Flush
26717 062/300 Flush
26718 063/300 Write block 3711 len 4096 data: 7
26719 064/300 Read block 1556 len 12288
26720 065/300 Write block 355 len 28672 data: 3 5 5 4 4 4 4
26721 066/300 Write block 3166 len 24576 data: 5 5 5 6 5 5
26722 067/300 Read block 4271 len 36864
26723 068/300 Read block 2209 len 4096
26724 069/300 Read block 1066 len 12288
26725 070/300 Read block 4193 len 28672
26726 071/300 Write block 1153 len 20480 data: 5 4 4 3 6
26727 072/300 Flush
26728 073/300 Write block 1544 len 12288 data: 6 6 4
26729 074/300 Write block 1342 len 20480 data: 4 4 4 4 4
26730 075/300 Write block 2005 len 24576 data: 6 7 6 7 9 8
26731 076/300 Write block 3810 len 40960 data: 3 4 5 6 7 7 8 8 8 9
26732 077/300 Flush
26733 078/300 Write block 7 len 28672 data: 4 3 3 5 4 6 8
26734 079/300 Write block 2037 len 24576 data: 8 7 7 6 6 7
26735 080/300 Write block 1329 len 4096 data: 5
26736 081/300 Read block 4047 len 36864
26737 082/300 Read block 3993 len 12288
26738 083/300 Write block 3426 len 4096 data: 6
26739 084/300 Read block 2191 len 32768
26740 085/300 Read block 2483 len 4096
26741 086/300 Read block 3591 len 24576
26742 087/300 Write block 3119 len 32768 data: 6 6 6 6 6 5 4 6
26743 088/300 Read block 479 len 8192
26744 089/300 Write block 3054 len 20480 data: 9 8 7 8 8
26745 090/300 Write block 3595 len 4096 data: 3
26746 091/300 Write block 4899 len 28672 data: 3 3 3 3 3 3 2
26747 092/300 Write block 214 len 20480 data: 7 7 7 5 5
26748 093/300 Read block 2055 len 28672
26749 094/300 Read block 1627 len 4096
26750 095/300 Write block 4022 len 16384 data: 7 6 7 8
26751 096/300 Write block 1413 len 8192 data: 8 7
26752 097/300 Write block 3161 len 16384 data: 3 5 7 5
26753 098/300 Write block 4477 len 20480 data: 3 3 3 3 2
26754 099/300 Read block 3399 len 28672
26755 100/300 Read block 1346 len 40960
26756 101/300 Write block 812 len 40960 data: 4 4 4 3 3 2 2 2 2 3
26757 102/300 Flush
26758 103/300 Write block 912 len 12288 data: 6 5 5
26759 104/300 Read block 4478 len 8192
26760 105/300 Read block 1612 len 4096
26761 106/300 Read block 1205 len 16384
26762 107/300 Flush
26763 108/300 Read block 3101 len 24576
26764 109/300 Read block 107 len 4096
26765 110/300 Read block 1059 len 20480
26766 111/300 Read block 3238 len 40960
26767 112/300 Write block 3204 len 20480 data: 6 5 5 4 3
26768 113/300 Read block 2259 len 8192
26769 114/300 Write block 3703 len 20480 data: 8 9 8 7 7
26770 115/300 Read block 3920 len 12288
26771 116/300 Read block 3926 len 4096
26772 117/300 Read block 265 len 16384
26773 118/300 Write block 4061 len 4096 data: 6
26774 119/300 Write block 3750 len 12288 data: 2 4 4
26775 120/300 Flush
26776 121/300 Write block 3488 len 12288 data: 8 7 6
26777 122/300 Read block 330 len 32768
26778 123/300 Read block 1421 len 4096
26779 124/300 Read block 2368 len 4096
26780 125/300 Write block 2206 len 16384 data: 4 5 4 4
26781 126/300 Read block 4432 len 36864
26782 127/300 Read block 2967 len 36864
26783 128/300 Write block 4283 len 36864 data: 3 5 4 3 4 5 5 5 4
26784 129/300 Read block 941 len 36864
26785 130/300 Read block 58 len 8192
26786 131/300 Read block 3444 len 20480
26787 132/300 Read block 1585 len 12288
26788 133/300 Read block 1162 len 36864
26789 134/300 Write block 203 len 36864 data: 7 6 6 6 6 6 7 8 7
26790 135/300 Read block 521 len 24576
26791 136/300 Flush
26792 137/300 Write block 2463 len 24576 data: 7 6 5 5 4 4
26793 138/300 Read block 4001 len 12288
26794 139/300 Read block 3057 len 12288
26795 140/300 Write block 1435 len 40960 data: 4 6 7 6 6 8 7 7 7 7
26796 141/300 Write block 2874 len 12288 data: 4 6 6
26797 142/300 Write block 204 len 16384 data: 7 7 7 7
26798 143/300 Write block 4067 len 40960 data: 5 3 3 3 3 2 2 2 2 2
26799 144/300 Read block 832 len 36864
26800 145/300 Read block 1493 len 12288
26801 146/300 Write block 532 len 4096 data: 5
26802 147/300 Write block 2071 len 36864 data: 4 4 3 3 4 4 5 4 5
26803 148/300 Read block 2839 len 20480
26804 149/300 Read block 1617 len 28672
26805 150/300 Write block 2615 len 40960 data: 2 2 3 4 4 3 4 4 4 4
26806 151/300 Write block 1891 len 24576 data: 5 5 6 4 5 5
26807 152/300 Write block 2269 len 24576 data: 5 4 4 5 4 5
26808 153/300 Flush
26809 154/300 Read block 4625 len 8192
26810 155/300 Read block 2280 len 8192
26811 156/300 Read block 3069 len 12288
26812 157/300 Flush
26813 158/300 Read block 1462 len 20480
26814 159/300 Write block 2776 len 12288 data: 6 3 3
26815 160/300 Write block 3307 len 12288 data: 6 7 7
26816 161/300 Flush
26817 162/300 Write block 1004 len 40960 data: 5 6 3 3 4 4 4 4 5 5
26818 163/300 Write block 1556 len 16384 data: 4 5 5 4
26819 164/300 Read block 647 len 4096
26820 165/300 Write block 3642 len 24576 data: 4 4 4 4 5 4
26821 166/300 Write block 1398 len 40960 data: 6 7 7 6 5 3 3 5 4 6
26822 167/300 Write block 1153 len 24576 data: 6 5 5 4 7 5
26823 168/300 Write block 2451 len 36864 data: 5 4 7 9 10 10 12 11 12
26824 169/300 Flush
26825 170/300 Read block 4797 len 24576
26826 171/300 Flush
26827 172/300 Write block 2716 len 12288 data: 7 6 5
26828 173/300 Read block 2835 len 28672
26829 174/300 Write block 4390 len 32768 data: 6 7 8 7 8 6 6 5
26830 175/300 Read block 3484 len 12288
26831 176/300 Write block 1782 len 24576 data: 3 3 5 5 4 7
26832 177/300 Write block 4146 len 16384 data: 6 6 7 7
26833 178/300 Read block 764 len 4096
26834 179/300 Write block 210 len 28672 data: 9 8 6 5 8 8 8
26835 180/300 Read block 3604 len 12288
26836 181/300 Read block 3056 len 16384
26837 182/300 Write block 1155 len 28672 data: 6 5 8 6 4 3 4
26838 183/300 Write block 1167 len 16384 data: 5 6 6 7
26839 184/300 Read block 4702 len 12288
26840 185/300 Write block 2962 len 24576 data: 4 5 7 7 8 8
26841 186/300 Read block 943 len 28672
26842 187/300 Read block 485 len 28672
26843 188/300 Write block 3631 len 32768 data: 2 2 3 3 3 5 7 7
26844 189/300 Read block 110 len 28672
26845 190/300 Read block 4749 len 40960
26846 191/300 Read block 1199 len 40960
26847 192/300 Read block 4039 len 24576
26848 193/300 Read block 442 len 4096
26849 194/300 Read block 2499 len 40960
26850 195/300 Read block 61 len 36864
26851 196/300 Read block 3993 len 4096
26852 197/300 Write block 4529 len 36864 data: 3 6 7 6 7 6 7 7 8
26853 198/300 Write block 1413 len 12288 data: 9 8 8
26854 199/300 Read block 4780 len 40960
26855 200/300 Read block 2434 len 36864
26856 201/300 Read block 3161 len 40960
26857 202/300 Flush
26858 203/300 Read block 2250 len 16384
26859 204/300 Flush
26860 205/300 Flush
26861 206/300 Read block 671 len 32768
26862 207/300 Read block 4876 len 36864
26863 208/300 Write block 2543 len 8192 data: 4 3
26864 209/300 Read block 4855 len 32768
26865 210/300 Write block 527 len 32768 data: 5 3 4 3 4 6 6 5
26866 211/300 Write block 1637 len 28672 data: 4 4 5 4 4 4 4
26867 212/300 Write block 3538 len 20480 data: 6 6 7 6 5
26868 213/300 Write block 709 len 16384 data: 3 4 5 5
26869 214/300 Read block 707 len 20480
26870 215/300 Write block 3709 len 40960 data: 8 8 8 6 7 7 8 7 8 7
26871 216/300 Write block 4167 len 8192 data: 6 5
26872 217/300 Write block 511 len 28672 data: 5 6 6 5 6 6 4
26873 218/300 Read block 3740 len 36864
26874 219/300 Write block 85 len 16384 data: 5 5 5 5
26875 220/300 Write block 1612 len 28672 data: 4 6 6 5 6 6 6
26876 221/300 Read block 101 len 28672
26877 222/300 Read block 4920 len 12288
26878 223/300 Read block 3934 len 32768
26879 224/300 Write block 3211 len 40960 data: 4 4 5 6 6 7 8 10 11 9
26880 225/300 Read block 659 len 12288
26881 226/300 Flush
26882 227/300 Read block 3467 len 32768
26883 228/300 Write block 1581 len 32768 data: 7 5 5 5 5 4 3 3
26884 229/300 Read block 4733 len 16384
26885 230/300 Read block 1096 len 20480
26886 231/300 Read block 1560 len 36864
26887 232/300 Read block 318 len 36864
26888 233/300 Read block 4370 len 16384
26889 234/300 Read block 3145 len 20480
26890 235/300 Flush
26891 236/300 Write block 1671 len 20480 data: 5 5 6 6 6
26892 237/300 Write block 822 len 16384 data: 3 4 4 4
26893 238/300 Write block 4581 len 32768 data: 5 5 5 4 4 4 5 7
26894 239/300 Read block 2323 len 40960
26895 240/300 Write block 4025 len 12288 data: 9 7 7
26896 241/300 Read block 1193 len 8192
26897 242/300 Read block 1391 len 40960
26898 243/300 Read block 752 len 28672
26899 244/300 Flush
26900 245/300 Write block 1566 len 36864 data: 3 2 4 6 7 7 5 7 5
26901 246/300 Read block 3064 len 36864
26902 247/300 Write block 4403 len 12288 data: 6 6 6
26903 248/300 Write block 2454 len 36864 data: 10 11 11 13 12 13 12 13 11
26904 249/300 Read block 2620 len 28672
26905 250/300 Write block 896 len 32768 data: 5 5 5 5 4 5 6 6
26906 251/300 Flush
26907 252/300 Write block 3391 len 28672 data: 7 8 8 8 8 8 8
26908 253/300 Read block 68 len 36864
26909 254/300 Write block 4169 len 32768 data: 5 6 5 5 6 4 4 6
26910 255/300 Write block 2380 len 8192 data: 3 3
26911 256/300 Read block 3565 len 20480
26912 257/300 Read block 4153 len 32768
26913 258/300 Flush
26914 259/300 Read block 2007 len 8192
26915 260/300 Flush
26916 261/300 Read block 2698 len 24576
26917 262/300 Write block 993 len 24576 data: 8 7 7 7 4 3
26918 263/300 Read block 1228 len 8192
26919 264/300 Read block 2657 len 28672
26920 265/300 Write block 1646 len 4096 data: 9
26921 266/300 Read block 4367 len 32768
26922 267/300 Read block 4603 len 24576
26923 268/300 Write block 411 len 16384 data: 6 6 6 7
26924 269/300 Flush
26925 270/300 Write block 838 len 16384 data: 8 9 8 7
26926 271/300 Read block 4153 len 24576
26927 272/300 Write block 4623 len 24576 data: 6 7 7 7 6 8
26928 273/300 Write block 4704 len 4096 data: 6
26929 274/300 Read block 3215 len 40960
26930 275/300 Write block 3192 len 32768 data: 3 4 3 3 3 4 4 5
26931 276/300 Read block 3387 len 40960
26932 277/300 Flush
26933 278/300 Write block 3307 len 36864 data: 7 8 8 4 6 5 6 6 6
26934 279/300 Read block 3976 len 28672
26935 280/300 Flush
26936 281/300 Read block 1564 len 40960
26937 282/300 Write block 3080 len 4096 data: 6
26938 283/300 Write block 3277 len 24576 data: 4 3 3 4 5 5
26939 284/300 Read block 4089 len 40960
26940 285/300 Read block 3953 len 32768
26941 286/300 Write block 3600 len 12288 data: 3 4 4
26942 287/300 Write block 4958 len 20480 data: 5 5 6 8 6
26943 288/300 Read block 3730 len 16384
26944 289/300 Read block 3025 len 12288
26945 290/300 Read block 4866 len 8192
26946 291/300 Read block 2840 len 40960
26947 292/300 Read block 458 len 24576
26948 293/300 Write block 3990 len 20480 data: 5 8 7 8 7
26949 294/300 Write block 926 len 12288 data: 6 9 9
26950 295/300 Write block 2616 len 20480 data: 3 4 5 5 4
26951 296/300 Write block 1155 len 16384 data: 7 6 9 7
26952 297/300 Flush
26953 298/300 Read block 3552 len 24576
26954 299/300 Write block 4606 len 12288 data: 4 3 3
26955 300/300 Read block 2738 len 36864
26956 301/300 Write block 2044 len 12288 data: 6 4 2
26957 Main task has work to do, go find it
26958 got dsc Start(0)
26959 start 0
26960 Feb 25 17:24:12.154 INFO request completed, latency_us: 260, response_code: 204, uri: /start/cid/0, method: POST, req_id: d0ebe120-9b10-4e35-a994-2512dd769ee9, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
26961 Replay: started 0, returned:Ok(())
26962 ----------------------------------------------------------------
26963 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26964 States: Offline Active Active
26965 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26966 7944 Acked Read 2 Sent Done Done false
26967 7945 Acked Write 9 Sent Done Done false
26968 7946 Acked Flush 0 Sent Done Done false
26969 7947 Acked Read 1 Sent Done Done false
26970 7948 Acked Write 5 Sent Done Done false
26971 7949 Acked Read 3 Sent Done Done false
26972 7950 Acked Write 9 Sent Done Done false
26973 7951 Acked Read 6 Sent Done Done false
26974 7952 Acked Write 7 Sent Done Done false
26975 7953 Acked Read 10 Sent Done Done false
26976 7954 Acked Write 3 Sent Done Done false
26977 7955 Acked Write 4 Sent Done Done false
26978 7956 Acked Read 5 Sent Done Done false
26979 7957 Acked Write 7 Sent Done Done false
26980 7958 Acked Read 10 Sent Done Done false
26981 7959 Acked Read 7 Sent Done Done false
26982 7960 Acked Write 1 Sent Done Done false
26983 7961 Acked Write 2 Sent Done Done false
26984 7962 Acked Flush 0 Sent Done Done false
26985 7963 Acked Read 1 Sent Done Done false
26986 7964 Acked Read 3 Sent Done Done false
26987 7965 Acked Write 5 Sent Done Done false
26988 7966 Acked Read 10 Sent Done Done false
26989 7967 Acked Write 5 Sent Done Done false
26990 7968 Acked Write 4 Sent Done Done false
26991 7969 Acked Write 3 Sent Done Done false
26992 7970 Acked Flush 0 Sent Done Done false
26993 7971 Acked Read 7 Sent Done Done false
26994 7972 Acked Write 3 Sent Done Done false
26995 7973 Acked Read 5 Sent Done Done false
26996 7974 Acked Read 3 Sent Done Done false
26997 7975 Acked Write 6 Sent Done Done false
26998 7976 Acked Read 10 Sent Done Done false
26999 7977 Acked Flush 0 Sent Done Done false
27000 7978 Acked Flush 0 Sent Done Done false
27001 7979 Acked Read 2 Sent Done Done false
27002 7980 Acked Read 10 Sent Done Done false
27003 7981 Acked Write 8 Sent Done Done false
27004 7982 Acked Read 4 Sent Done Done false
27005 7983 Acked Write 9 Sent Done Done false
27006 7984 Acked Flush 0 Sent Done Done false
27007 7985 Acked Flush 0 Sent Done Done false
27008 7986 Acked Write 7 Sent Done Done false
27009 7987 Acked Read 9 Sent Done Done false
27010 7988 Acked Read 3 Sent Done Done false
27011 7989 Acked Write 7 Sent Done Done false
27012 7990 Acked Read 10 Sent Done Done false
27013 7991 Acked Write 6 Sent Done Done false
27014 7992 Acked Write 8 Sent Done Done false
27015 7993 Acked Write 4 Sent Done Done false
27016 7994 Acked Write 9 Sent Done Done false
27017 7995 Acked Read 2 Sent Done Done false
27018 7996 Acked Write 6 Sent Done Done false
27019 7997 Acked Write 3 Sent Done Done false
27020 7998 Acked Flush 0 Sent Done Done false
27021 7999 Acked Read 8 Sent Done Done false
27022 8000 Acked Read 5 Sent Done Done false
27023 8001 Acked Write 10 Sent Done Done false
27024 8002 Acked Read 2 Sent Done Done false
27025 8003 Acked Write 8 Sent Done Done false
27026 8004 Acked Flush 0 Sent Done Done false
27027 8005 Acked Flush 0 Sent Done Done false
27028 8006 Acked Write 1 Sent Done Done false
27029 8007 Acked Read 3 Sent Done Done false
27030 8008 Acked Write 7 Sent Done Done false
27031 8009 Acked Write 6 Sent Done Done false
27032 8010 Acked Read 9 Sent Done Done false
27033 8011 Acked Read 1 Sent Done Done false
27034 8012 Acked Read 3 Sent Done Done false
27035 8013 Acked Read 7 Sent Done Done false
27036 8014 Acked Write 5 Sent Done Done false
27037 8015 Acked Flush 0 Sent Done Done false
27038 8016 Acked Write 3 Sent Done Done false
27039 8017 Acked Write 5 Sent Done Done false
27040 8018 Acked Write 6 Sent Done Done false
27041 8019 Acked Write 10 Sent Done Done false
27042 8020 Acked Flush 0 Sent Done Done false
27043 8021 Acked Write 7 Sent Done Done false
27044 8022 Acked Write 6 Sent Done Done false
27045 8023 Acked Write 1 Sent Done Done false
27046 8024 Acked Read 9 Sent Done Done false
27047 8025 Acked Read 3 Sent Done Done false
27048 8026 Acked Write 1 Sent Done Done false
27049 8027 Acked Read 8 Sent Done Done false
27050 8028 Acked Read 1 Sent Done Done false
27051 8029 Acked Read 6 Sent Done Done false
27052 8030 Acked Write 8 Sent Done Done false
27053 8031 Acked Read 2 Sent Done Done false
27054 8032 Acked Write 5 Sent Done Done false
27055 8033 Acked Write 1 Sent Done Done false
27056 8034 Acked Write 7 Sent Done Done false
27057 8035 Acked Write 5 Sent Done Done false
27058 8036 Acked Read 7 Sent Done Done false
27059 8037 Acked Read 1 Sent Done Done false
27060 8038 Acked Write 4 Sent Done Done false
27061 8039 Acked Write 2 Sent Done Done false
27062 8040 Acked Write 4 Sent Done Done false
27063 8041 Acked Write 5 Sent Done Done false
27064 8042 Acked Read 7 Sent Done Done false
27065 8043 Acked Read 10 Sent Done Done false
27066 8044 Acked Write 10 Sent Done Done false
27067 8045 Acked Flush 0 Sent Done Done false
27068 8046 Acked Write 3 Sent Done Done false
27069 8047 Acked Read 2 Sent Done Done false
27070 8048 Acked Read 1 Sent Done Done false
27071 8049 Acked Read 4 Sent Done Done false
27072 8050 Acked Flush 0 Sent Done Done false
27073 8051 Acked Read 6 Sent Done Done false
27074 8052 Acked Read 1 Sent Done Done false
27075 8053 Acked Read 5 Sent Done Done false
27076 8054 Acked Read 10 Sent Done Done false
27077 8055 Acked Write 5 Sent Done Done false
27078 8056 Acked Read 2 Sent Done Done false
27079 8057 Acked Write 5 Sent Done Done false
27080 8058 Acked Read 3 Sent Done Done false
27081 8059 Acked Read 1 Sent Done Done false
27082 8060 Acked Read 4 Sent Done Done false
27083 8061 Acked Write 1 Sent Done Done false
27084 8062 Acked Write 3 Sent Done Done false
27085 8063 Acked Flush 0 Sent Done Done false
27086 8064 Acked Write 3 Sent Done Done false
27087 8065 Acked Read 8 Sent Done Done false
27088 8066 Acked Read 1 Sent Done Done false
27089 8067 Acked Read 1 Sent Done Done false
27090 8068 Acked Write 4 Sent Done Done false
27091 8069 Acked Read 9 Sent Done Done false
27092 8070 Acked Read 9 Sent Done Done false
27093 8071 Acked Write 9 Sent Done Done false
27094 8072 Acked Read 9 Sent Done Done false
27095 8073 Acked Read 2 Sent Done Done false
27096 8074 Acked Read 5 Sent Done Done false
27097 8075 Acked Read 3 Sent Done Done false
27098 8076 Acked Read 9 Sent Done Done false
27099 8077 Acked Write 9 Sent Done Done false
27100 8078 Acked Read 6 Sent Done Done false
27101 8079 Acked Flush 0 Sent Done Done false
27102 8080 Acked Write 6 Sent Done Done false
27103 8081 Acked Read 3 Sent Done Done false
27104 8082 Acked Read 3 Sent Done Done false
27105 8083 Acked Write 10 Sent Done Done false
27106 8084 Acked Write 3 Sent Done Done false
27107 8085 Acked Write 4 Sent Done Done false
27108 8086 Acked Write 10 Sent Done Done false
27109 8087 Acked Read 9 Sent Done Done false
27110 8088 Acked Read 3 Sent Done Done false
27111 8089 Acked Write 1 Sent Done Done false
27112 8090 Acked Write 9 Sent Done Done false
27113 8091 Acked Read 5 Sent Done Done false
27114 8092 Acked Read 7 Sent Done Done false
27115 8093 Acked Write 10 Sent Done Done false
27116 8094 Acked Write 6 Sent Done Done false
27117 8095 Acked Write 6 Sent Done Done false
27118 8096 Acked Flush 0 Sent Done Done false
27119 8097 Acked Read 2 Sent Done Done false
27120 8098 Acked Read 2 Sent Done Done false
27121 8099 Acked Read 3 Sent Done Done false
27122 8100 Acked Flush 0 Sent Done Done false
27123 8101 Acked Read 5 Sent Done Done false
27124 8102 Acked Write 3 Sent Done Done false
27125 8103 Acked Write 3 Sent Done Done false
27126 8104 Acked Flush 0 Sent Done Done false
27127 8105 Acked Write 10 Sent Done Done false
27128 8106 Acked Write 4 Sent Done Done false
27129 8107 Acked Read 1 Sent Done Done false
27130 8108 Acked Write 6 Sent Done Done false
27131 8109 Acked Write 10 Sent Done Done false
27132 8110 Acked Write 6 Sent Done Done false
27133 8111 Acked Write 9 Sent Done Done false
27134 8112 Acked Flush 0 Sent Done Done false
27135 8113 Acked Read 6 Sent Done Done false
27136 8114 Acked Flush 0 Sent Done Done false
27137 8115 Acked Write 3 Sent Done Done false
27138 8116 Acked Read 7 Sent Done Done false
27139 8117 Acked Write 8 Sent Done Done false
27140 8118 Acked Read 3 Sent Done Done false
27141 8119 Acked Write 6 Sent Done Done false
27142 8120 Acked Write 4 Sent Done Done false
27143 8121 Acked Read 1 Sent Done Done false
27144 8122 Acked Write 7 Sent Done Done false
27145 8123 Acked Read 3 Sent Done Done false
27146 8124 Acked Read 4 Sent Done Done false
27147 8125 Acked Write 7 Sent Done Done false
27148 8126 Acked Write 4 Sent Done Done false
27149 8127 Acked Read 3 Sent Done Done false
27150 8128 Acked Write 6 Sent Done Done false
27151 8129 Acked Read 7 Sent Done Done false
27152 8130 Acked Read 7 Sent Done Done false
27153 8131 Acked Write 8 Sent Done Done false
27154 8132 Acked Read 7 Sent Done Done false
27155 8133 Acked Read 10 Sent Done Done false
27156 8134 Acked Read 10 Sent Done Done false
27157 8135 Acked Read 6 Sent Done Done false
27158 8136 Acked Read 1 Sent Done Done false
27159 8137 Acked Read 10 Sent Done Done false
27160 8138 Acked Read 9 Sent Done Done false
27161 8139 Acked Read 1 Sent Done Done false
27162 8140 Acked Write 9 Sent Done Done false
27163 8141 Acked Write 3 Sent Done Done false
27164 8142 Acked Read 10 Sent Done Done false
27165 8143 Acked Read 9 Sent Done Done false
27166 8144 Acked Read 10 Sent Done Done false
27167 8145 Acked Flush 0 Sent Done Done false
27168 8146 Acked Read 4 Sent Done Done false
27169 8147 Acked Flush 0 Sent Done Done false
27170 8148 Acked Flush 0 Sent Done Done false
27171 8149 Acked Read 8 Sent Done Done false
27172 8150 Acked Read 9 Sent Done Done false
27173 8151 Acked Write 2 Sent Done Done false
27174 8152 Acked Read 8 Sent Done Done false
27175 8153 Acked Write 8 Sent Done Done false
27176 8154 Acked Write 7 Sent Done Done false
27177 8155 Acked Write 5 Sent Done Done false
27178 8156 Acked Write 4 Sent Done Done false
27179 8157 Acked Read 5 Sent Done Done false
27180 8158 Acked Write 10 Sent Done Done false
27181 8159 Acked Write 2 Sent Done Done false
27182 8160 Acked Write 7 Sent Done Done false
27183 8161 Acked Read 9 Sent Done Done false
27184 8162 Acked Write 4 Sent Done Done false
27185 8163 Acked Write 7 Sent Done Done false
27186 8164 Acked Read 7 Sent Done Done false
27187 8165 Acked Read 3 Sent Done Done false
27188 8166 Acked Read 8 Sent Done Done false
27189 8167 Acked Write 10 Sent Done Done false
27190 8168 Acked Read 3 Sent Done Done false
27191 8169 Acked Flush 0 Sent Done Done false
27192 8170 Acked Read 8 Sent Done Done false
27193 8171 Acked Write 8 Sent Done Done false
27194 8172 Acked Read 4 Sent Done Done false
27195 8173 Acked Read 5 Sent Done Done false
27196 8174 Acked Read 9 Sent Done Done false
27197 8175 Acked Read 9 Sent Done Done false
27198 8176 Acked Read 4 Sent Done Done false
27199 8177 Acked Read 5 Sent Done Done false
27200 8178 Acked Flush 0 Sent Done Done false
27201 8179 Acked Write 5 Sent Done Done false
27202 8180 Acked Write 4 Sent Done Done false
27203 8181 Acked Write 8 Sent Done Done false
27204 8182 Acked Read 10 Sent Done Done false
27205 8183 Acked Write 3 Sent Done Done false
27206 8184 Acked Read 2 Sent Done Done false
27207 8185 Acked Read 10 Sent Done Done false
27208 8186 Acked Read 7 Sent Done Done false
27209 8187 Acked Flush 0 Sent Done Done false
27210 8188 Acked Write 9 Sent Done Done false
27211 8189 Acked Read 9 Sent Done Done false
27212 8190 Acked Write 3 Sent Done Done false
27213 8191 Acked Write 9 Sent Done Done false
27214 8192 Acked Read 7 Sent Done Done false
27215 8193 Acked Write 8 Sent Done Done false
27216 8194 Acked Flush 0 Sent Done Done false
27217 8195 Acked Write 7 Sent Done Done false
27218 8196 Acked Read 9 Sent Done Done false
27219 8197 Acked Write 8 Sent Done Done false
27220 8198 Acked Write 2 Sent Done Done false
27221 8199 Acked Read 5 Sent Done Done false
27222 8200 Acked Read 8 Sent Done Done false
27223 8201 Acked Flush 0 Sent Done Done false
27224 8202 Acked Read 2 Sent Done Done false
27225 8203 Acked Flush 0 Sent Done Done false
27226 8204 Acked Read 6 Sent Done Done false
27227 8205 Acked Write 6 Sent Done Done false
27228 8206 Acked Read 2 Sent Done Done false
27229 8207 Acked Read 7 Sent Done Done false
27230 8208 Acked Write 1 Sent Done Done false
27231 8209 Acked Read 8 Sent Done Done false
27232 8210 Acked Read 6 Sent Done Done false
27233 8211 Acked Write 4 Sent Done Done false
27234 8212 Acked Flush 0 Sent Done Done false
27235 8213 Acked Write 4 Sent Done Done false
27236 8214 Acked Read 6 Sent Done Done false
27237 8215 Acked Write 6 Sent Done Done false
27238 8216 Acked Write 1 Sent Done Done false
27239 8217 Acked Read 10 Sent Done Done false
27240 8218 Acked Write 8 Sent Done Done false
27241 8219 Acked Read 10 Sent Done Done false
27242 8220 Acked Flush 0 Sent Done Done false
27243 8221 Acked Write 9 Sent Done Done false
27244 8222 Acked Read 7 Sent Done Done false
27245 8223 Acked Flush 0 Sent Done Done false
27246 8224 Acked Read 10 Sent Done Done false
27247 8225 Acked Write 1 Sent Done Done false
27248 8226 Acked Write 6 Sent Done Done false
27249 8227 Acked Read 10 Sent Done Done false
27250 8228 Acked Read 8 Sent Done Done false
27251 8229 Acked Write 3 Sent Done Done false
27252 8230 Acked Write 5 Sent Done Done false
27253 8231 Acked Read 4 Sent Done Done false
27254 8232 Acked Read 3 Sent Done Done false
27255 8233 Acked Read 2 Sent Done Done false
27256 8234 Acked Read 10 Sent Done Done false
27257 8235 Acked Read 6 Sent Done Done false
27258 8236 Acked Write 5 Sent Done Done false
27259 8237 Acked Write 3 Sent Done Done false
27260 8238 Acked Write 5 Sent Done Done false
27261 8239 Acked Write 4 Sent Done Done false
27262 8240 Acked Flush 0 Sent Done Done false
27263 8241 Acked Read 6 Sent Done Done false
27264 8242 Acked Write 3 Sent Done Done false
27265 8243 Acked Read 9 Sent Done Done false
27266 8244 Acked Write 3 Sent Done Done false
27267 STATES DS:0 DS:1 DS:2 TOTAL
27268 Sent 301 0 0 301
27269 Done 0 301 301 602
27270 Skipped 0 0 0 0
27271 Error 0 0 0 0
27272 Last Flush: Some(JobId(7943)) Some(JobId(8240)) Some(JobId(8240))
27273 Downstairs last five completed: 7943 7942 7941 7940 7939
27274 Upstairs last five completed: 8244 8243 8242 8241 8240
27275 CLIENT: Up:0 ds:301 act:2
272762025-02-25T17:24:12.830ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
272772025-02-25T17:24:13.831ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
27278 [8810] got start action
27279 [8810] I am going to restart
27280 Starting downstairs at port 8810
27281 Make output file at "/tmp/dsc/downstairs-8810.txt"
27282 [8810][0] reports Starting
27283 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1232)
27284 [8810][0] reports Running
272852025-02-25T17:24:14.832ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
272862025-02-25T17:24:15.874ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
27287 ----------------------------------------------------------------
27288 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27289 States: Offline Active Active
27290 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27291 7944 Acked Read 2 Sent Done Done false
27292 7945 Acked Write 9 Sent Done Done false
27293 7946 Acked Flush 0 Sent Done Done false
27294 7947 Acked Read 1 Sent Done Done false
27295 7948 Acked Write 5 Sent Done Done false
27296 7949 Acked Read 3 Sent Done Done false
27297 7950 Acked Write 9 Sent Done Done false
27298 7951 Acked Read 6 Sent Done Done false
27299 7952 Acked Write 7 Sent Done Done false
27300 7953 Acked Read 10 Sent Done Done false
27301 7954 Acked Write 3 Sent Done Done false
27302 7955 Acked Write 4 Sent Done Done false
27303 7956 Acked Read 5 Sent Done Done false
27304 7957 Acked Write 7 Sent Done Done false
27305 7958 Acked Read 10 Sent Done Done false
27306 7959 Acked Read 7 Sent Done Done false
27307 7960 Acked Write 1 Sent Done Done false
27308 7961 Acked Write 2 Sent Done Done false
27309 7962 Acked Flush 0 Sent Done Done false
27310 7963 Acked Read 1 Sent Done Done false
27311 7964 Acked Read 3 Sent Done Done false
27312 7965 Acked Write 5 Sent Done Done false
27313 7966 Acked Read 10 Sent Done Done false
27314 7967 Acked Write 5 Sent Done Done false
27315 7968 Acked Write 4 Sent Done Done false
27316 7969 Acked Write 3 Sent Done Done false
27317 7970 Acked Flush 0 Sent Done Done false
27318 7971 Acked Read 7 Sent Done Done false
27319 7972 Acked Write 3 Sent Done Done false
27320 7973 Acked Read 5 Sent Done Done false
27321 7974 Acked Read 3 Sent Done Done false
27322 7975 Acked Write 6 Sent Done Done false
27323 7976 Acked Read 10 Sent Done Done false
27324 7977 Acked Flush 0 Sent Done Done false
27325 7978 Acked Flush 0 Sent Done Done false
27326 7979 Acked Read 2 Sent Done Done false
27327 7980 Acked Read 10 Sent Done Done false
27328 7981 Acked Write 8 Sent Done Done false
27329 7982 Acked Read 4 Sent Done Done false
27330 7983 Acked Write 9 Sent Done Done false
27331 7984 Acked Flush 0 Sent Done Done false
27332 7985 Acked Flush 0 Sent Done Done false
27333 7986 Acked Write 7 Sent Done Done false
27334 7987 Acked Read 9 Sent Done Done false
27335 7988 Acked Read 3 Sent Done Done false
27336 7989 Acked Write 7 Sent Done Done false
27337 7990 Acked Read 10 Sent Done Done false
27338 7991 Acked Write 6 Sent Done Done false
27339 7992 Acked Write 8 Sent Done Done false
27340 7993 Acked Write 4 Sent Done Done false
27341 7994 Acked Write 9 Sent Done Done false
27342 7995 Acked Read 2 Sent Done Done false
27343 7996 Acked Write 6 Sent Done Done false
27344 7997 Acked Write 3 Sent Done Done false
27345 7998 Acked Flush 0 Sent Done Done false
27346 7999 Acked Read 8 Sent Done Done false
27347 8000 Acked Read 5 Sent Done Done false
27348 8001 Acked Write 10 Sent Done Done false
27349 8002 Acked Read 2 Sent Done Done false
27350 8003 Acked Write 8 Sent Done Done false
27351 8004 Acked Flush 0 Sent Done Done false
27352 8005 Acked Flush 0 Sent Done Done false
27353 8006 Acked Write 1 Sent Done Done false
27354 8007 Acked Read 3 Sent Done Done false
27355 8008 Acked Write 7 Sent Done Done false
27356 8009 Acked Write 6 Sent Done Done false
27357 8010 Acked Read 9 Sent Done Done false
27358 8011 Acked Read 1 Sent Done Done false
27359 8012 Acked Read 3 Sent Done Done false
27360 8013 Acked Read 7 Sent Done Done false
27361 8014 Acked Write 5 Sent Done Done false
27362 8015 Acked Flush 0 Sent Done Done false
27363 8016 Acked Write 3 Sent Done Done false
27364 8017 Acked Write 5 Sent Done Done false
27365 8018 Acked Write 6 Sent Done Done false
27366 8019 Acked Write 10 Sent Done Done false
27367 8020 Acked Flush 0 Sent Done Done false
27368 8021 Acked Write 7 Sent Done Done false
27369 8022 Acked Write 6 Sent Done Done false
27370 8023 Acked Write 1 Sent Done Done false
27371 8024 Acked Read 9 Sent Done Done false
27372 8025 Acked Read 3 Sent Done Done false
27373 8026 Acked Write 1 Sent Done Done false
27374 8027 Acked Read 8 Sent Done Done false
27375 8028 Acked Read 1 Sent Done Done false
27376 8029 Acked Read 6 Sent Done Done false
27377 8030 Acked Write 8 Sent Done Done false
27378 8031 Acked Read 2 Sent Done Done false
27379 8032 Acked Write 5 Sent Done Done false
27380 8033 Acked Write 1 Sent Done Done false
27381 8034 Acked Write 7 Sent Done Done false
27382 8035 Acked Write 5 Sent Done Done false
27383 8036 Acked Read 7 Sent Done Done false
27384 8037 Acked Read 1 Sent Done Done false
27385 8038 Acked Write 4 Sent Done Done false
27386 8039 Acked Write 2 Sent Done Done false
27387 8040 Acked Write 4 Sent Done Done false
27388 8041 Acked Write 5 Sent Done Done false
27389 8042 Acked Read 7 Sent Done Done false
27390 8043 Acked Read 10 Sent Done Done false
27391 8044 Acked Write 10 Sent Done Done false
27392 8045 Acked Flush 0 Sent Done Done false
27393 8046 Acked Write 3 Sent Done Done false
27394 8047 Acked Read 2 Sent Done Done false
27395 8048 Acked Read 1 Sent Done Done false
27396 8049 Acked Read 4 Sent Done Done false
27397 8050 Acked Flush 0 Sent Done Done false
27398 8051 Acked Read 6 Sent Done Done false
27399 8052 Acked Read 1 Sent Done Done false
27400 8053 Acked Read 5 Sent Done Done false
27401 8054 Acked Read 10 Sent Done Done false
27402 8055 Acked Write 5 Sent Done Done false
27403 8056 Acked Read 2 Sent Done Done false
27404 8057 Acked Write 5 Sent Done Done false
27405 8058 Acked Read 3 Sent Done Done false
27406 8059 Acked Read 1 Sent Done Done false
27407 8060 Acked Read 4 Sent Done Done false
27408 8061 Acked Write 1 Sent Done Done false
27409 8062 Acked Write 3 Sent Done Done false
27410 8063 Acked Flush 0 Sent Done Done false
27411 8064 Acked Write 3 Sent Done Done false
27412 8065 Acked Read 8 Sent Done Done false
27413 8066 Acked Read 1 Sent Done Done false
27414 8067 Acked Read 1 Sent Done Done false
27415 8068 Acked Write 4 Sent Done Done false
27416 8069 Acked Read 9 Sent Done Done false
27417 8070 Acked Read 9 Sent Done Done false
27418 8071 Acked Write 9 Sent Done Done false
27419 8072 Acked Read 9 Sent Done Done false
27420 8073 Acked Read 2 Sent Done Done false
27421 8074 Acked Read 5 Sent Done Done false
27422 8075 Acked Read 3 Sent Done Done false
27423 8076 Acked Read 9 Sent Done Done false
27424 8077 Acked Write 9 Sent Done Done false
27425 8078 Acked Read 6 Sent Done Done false
27426 8079 Acked Flush 0 Sent Done Done false
27427 8080 Acked Write 6 Sent Done Done false
27428 8081 Acked Read 3 Sent Done Done false
27429 8082 Acked Read 3 Sent Done Done false
27430 8083 Acked Write 10 Sent Done Done false
27431 8084 Acked Write 3 Sent Done Done false
27432 8085 Acked Write 4 Sent Done Done false
27433 8086 Acked Write 10 Sent Done Done false
27434 8087 Acked Read 9 Sent Done Done false
27435 8088 Acked Read 3 Sent Done Done false
27436 8089 Acked Write 1 Sent Done Done false
27437 8090 Acked Write 9 Sent Done Done false
27438 8091 Acked Read 5 Sent Done Done false
27439 8092 Acked Read 7 Sent Done Done false
27440 8093 Acked Write 10 Sent Done Done false
27441 8094 Acked Write 6 Sent Done Done false
27442 8095 Acked Write 6 Sent Done Done false
27443 8096 Acked Flush 0 Sent Done Done false
27444 8097 Acked Read 2 Sent Done Done false
27445 8098 Acked Read 2 Sent Done Done false
27446 8099 Acked Read 3 Sent Done Done false
27447 8100 Acked Flush 0 Sent Done Done false
27448 8101 Acked Read 5 Sent Done Done false
27449 8102 Acked Write 3 Sent Done Done false
27450 8103 Acked Write 3 Sent Done Done false
27451 8104 Acked Flush 0 Sent Done Done false
27452 8105 Acked Write 10 Sent Done Done false
27453 8106 Acked Write 4 Sent Done Done false
27454 8107 Acked Read 1 Sent Done Done false
27455 8108 Acked Write 6 Sent Done Done false
27456 8109 Acked Write 10 Sent Done Done false
27457 8110 Acked Write 6 Sent Done Done false
27458 8111 Acked Write 9 Sent Done Done false
27459 8112 Acked Flush 0 Sent Done Done false
27460 8113 Acked Read 6 Sent Done Done false
27461 8114 Acked Flush 0 Sent Done Done false
27462 8115 Acked Write 3 Sent Done Done false
27463 8116 Acked Read 7 Sent Done Done false
27464 8117 Acked Write 8 Sent Done Done false
27465 8118 Acked Read 3 Sent Done Done false
27466 8119 Acked Write 6 Sent Done Done false
27467 8120 Acked Write 4 Sent Done Done false
27468 8121 Acked Read 1 Sent Done Done false
27469 8122 Acked Write 7 Sent Done Done false
27470 8123 Acked Read 3 Sent Done Done false
27471 8124 Acked Read 4 Sent Done Done false
27472 8125 Acked Write 7 Sent Done Done false
27473 8126 Acked Write 4 Sent Done Done false
27474 8127 Acked Read 3 Sent Done Done false
27475 8128 Acked Write 6 Sent Done Done false
27476 8129 Acked Read 7 Sent Done Done false
27477 8130 Acked Read 7 Sent Done Done false
27478 8131 Acked Write 8 Sent Done Done false
27479 8132 Acked Read 7 Sent Done Done false
27480 8133 Acked Read 10 Sent Done Done false
27481 8134 Acked Read 10 Sent Done Done false
27482 8135 Acked Read 6 Sent Done Done false
27483 8136 Acked Read 1 Sent Done Done false
27484 8137 Acked Read 10 Sent Done Done false
27485 8138 Acked Read 9 Sent Done Done false
27486 8139 Acked Read 1 Sent Done Done false
27487 8140 Acked Write 9 Sent Done Done false
27488 8141 Acked Write 3 Sent Done Done false
27489 8142 Acked Read 10 Sent Done Done false
27490 8143 Acked Read 9 Sent Done Done false
27491 8144 Acked Read 10 Sent Done Done false
27492 8145 Acked Flush 0 Sent Done Done false
27493 8146 Acked Read 4 Sent Done Done false
27494 8147 Acked Flush 0 Sent Done Done false
27495 8148 Acked Flush 0 Sent Done Done false
27496 8149 Acked Read 8 Sent Done Done false
27497 8150 Acked Read 9 Sent Done Done false
27498 8151 Acked Write 2 Sent Done Done false
27499 8152 Acked Read 8 Sent Done Done false
27500 8153 Acked Write 8 Sent Done Done false
27501 8154 Acked Write 7 Sent Done Done false
27502 8155 Acked Write 5 Sent Done Done false
27503 8156 Acked Write 4 Sent Done Done false
27504 8157 Acked Read 5 Sent Done Done false
27505 8158 Acked Write 10 Sent Done Done false
27506 8159 Acked Write 2 Sent Done Done false
27507 8160 Acked Write 7 Sent Done Done false
27508 8161 Acked Read 9 Sent Done Done false
27509 8162 Acked Write 4 Sent Done Done false
27510 8163 Acked Write 7 Sent Done Done false
27511 8164 Acked Read 7 Sent Done Done false
27512 8165 Acked Read 3 Sent Done Done false
27513 8166 Acked Read 8 Sent Done Done false
27514 8167 Acked Write 10 Sent Done Done false
27515 8168 Acked Read 3 Sent Done Done false
27516 8169 Acked Flush 0 Sent Done Done false
27517 8170 Acked Read 8 Sent Done Done false
27518 8171 Acked Write 8 Sent Done Done false
27519 8172 Acked Read 4 Sent Done Done false
27520 8173 Acked Read 5 Sent Done Done false
27521 8174 Acked Read 9 Sent Done Done false
27522 8175 Acked Read 9 Sent Done Done false
27523 8176 Acked Read 4 Sent Done Done false
27524 8177 Acked Read 5 Sent Done Done false
27525 8178 Acked Flush 0 Sent Done Done false
27526 8179 Acked Write 5 Sent Done Done false
27527 8180 Acked Write 4 Sent Done Done false
27528 8181 Acked Write 8 Sent Done Done false
27529 8182 Acked Read 10 Sent Done Done false
27530 8183 Acked Write 3 Sent Done Done false
27531 8184 Acked Read 2 Sent Done Done false
27532 8185 Acked Read 10 Sent Done Done false
27533 8186 Acked Read 7 Sent Done Done false
27534 8187 Acked Flush 0 Sent Done Done false
27535 8188 Acked Write 9 Sent Done Done false
27536 8189 Acked Read 9 Sent Done Done false
27537 8190 Acked Write 3 Sent Done Done false
27538 8191 Acked Write 9 Sent Done Done false
27539 8192 Acked Read 7 Sent Done Done false
27540 8193 Acked Write 8 Sent Done Done false
27541 8194 Acked Flush 0 Sent Done Done false
27542 8195 Acked Write 7 Sent Done Done false
27543 8196 Acked Read 9 Sent Done Done false
27544 8197 Acked Write 8 Sent Done Done false
27545 8198 Acked Write 2 Sent Done Done false
27546 8199 Acked Read 5 Sent Done Done false
27547 8200 Acked Read 8 Sent Done Done false
27548 8201 Acked Flush 0 Sent Done Done false
27549 8202 Acked Read 2 Sent Done Done false
27550 8203 Acked Flush 0 Sent Done Done false
27551 8204 Acked Read 6 Sent Done Done false
27552 8205 Acked Write 6 Sent Done Done false
27553 8206 Acked Read 2 Sent Done Done false
27554 8207 Acked Read 7 Sent Done Done false
27555 8208 Acked Write 1 Sent Done Done false
27556 8209 Acked Read 8 Sent Done Done false
27557 8210 Acked Read 6 Sent Done Done false
27558 8211 Acked Write 4 Sent Done Done false
27559 8212 Acked Flush 0 Sent Done Done false
27560 8213 Acked Write 4 Sent Done Done false
27561 8214 Acked Read 6 Sent Done Done false
27562 8215 Acked Write 6 Sent Done Done false
27563 8216 Acked Write 1 Sent Done Done false
27564 8217 Acked Read 10 Sent Done Done false
27565 8218 Acked Write 8 Sent Done Done false
27566 8219 Acked Read 10 Sent Done Done false
27567 8220 Acked Flush 0 Sent Done Done false
27568 8221 Acked Write 9 Sent Done Done false
27569 8222 Acked Read 7 Sent Done Done false
27570 8223 Acked Flush 0 Sent Done Done false
27571 8224 Acked Read 10 Sent Done Done false
27572 8225 Acked Write 1 Sent Done Done false
27573 8226 Acked Write 6 Sent Done Done false
27574 8227 Acked Read 10 Sent Done Done false
27575 8228 Acked Read 8 Sent Done Done false
27576 8229 Acked Write 3 Sent Done Done false
27577 8230 Acked Write 5 Sent Done Done false
27578 8231 Acked Read 4 Sent Done Done false
27579 8232 Acked Read 3 Sent Done Done false
27580 8233 Acked Read 2 Sent Done Done false
27581 8234 Acked Read 10 Sent Done Done false
27582 8235 Acked Read 6 Sent Done Done false
27583 8236 Acked Write 5 Sent Done Done false
27584 8237 Acked Write 3 Sent Done Done false
27585 8238 Acked Write 5 Sent Done Done false
27586 8239 Acked Write 4 Sent Done Done false
27587 8240 Acked Flush 0 Sent Done Done false
27588 8241 Acked Read 6 Sent Done Done false
27589 8242 Acked Write 3 Sent Done Done false
27590 8243 Acked Read 9 Sent Done Done false
27591 8244 Acked Write 3 Sent Done Done false
27592 STATES DS:0 DS:1 DS:2 TOTAL
27593 Sent 301 0 0 301
27594 Done 0 301 301 602
27595 Skipped 0 0 0 0
27596 Error 0 0 0 0
27597 Last Flush: Some(JobId(7943)) Some(JobId(8240)) Some(JobId(8240))
27598 Downstairs last five completed: 7943 7942 7941 7940 7939
27599 Upstairs last five completed: 8244 8243 8242 8241 8240
27600 CLIENT: Up:0 ds:301 act:2
276012025-02-25T17:24:16.835ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276022025-02-25T17:24:17.837ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276032025-02-25T17:24:17.902ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276042025-02-25T17:24:17.902ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48209) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276052025-02-25T17:24:17.904ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276062025-02-25T17:24:17.904ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276072025-02-25T17:24:17.904ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276082025-02-25T17:24:17.904ZINFOcrucible: send last flush ID to this DS: Some(JobId(7943)) = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276092025-02-25T17:24:17.904ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276102025-02-25T17:24:17.904ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(7943)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276112025-02-25T17:24:17.905ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(7943)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
27612 ----------------------------------------------------------------
27613 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27614 Downstairs last five completed: 8245 8244 8243 8242 8241
27615 Upstairs last five completed: 8245 8244 8243 8242 8241
27616 CLIENT: Up:0 ds:0 act:3
27617 Replay: All jobs finished
27618 Main task has work to do, go find it
27619 got dsc Stop(1)
27620 stop 1
27621 Feb 25 17:24:20.159[8820] Got stop action so:false kr:false
27622 INFO request completed, latency_us: 362, response_code: 204, uri: /stop/cid/1, method: POST, req_id: f847ee1c-44c1-496e-a56d-dfde62042adb, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
27623 Feb 25 17:24:20.160 INFO request completed, latency_us: 125, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2d23db48-4873-44a1-b3ca-e97e985cdefd, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
276242025-02-25T17:24:20.160ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276252025-02-25T17:24:20.160ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276262025-02-25T17:24:20.160ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
276272025-02-25T17:24:20.160ZINFOcrucible: client task is exiting = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
27628 [8820] Exited with: ExitStatus(unix_wait_status(9))
27629 [8820][1] reports Exit
27630 Feb 25 17:24:24.162 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/1, method: GET, req_id: f51831d9-65ef-436e-a72c-dda5cc3f7673, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
27631 001/300 Flush
27632 002/300 Read block 3907 len 20480
27633 003/300 Read block 1513 len 20480
27634 004/300 Read block 4936 len 20480
27635 005/300 Write block 1877 len 12288 data: 4 4 4
27636 006/300 Write block 1421 len 36864 data: 4 5 4 5 5 7 9 8 8
27637 007/300 Flush
27638 008/300 Read block 2036 len 32768
27639 009/300 Flush
27640 010/300 Write block 210 len 40960 data: 10 9 7 6 9 9 9 6 6 4
27641 011/300 Flush
27642 012/300 Flush
27643 013/300 Read block 1711 len 36864
27644 014/300 Read block 2452 len 40960
27645 015/300 Write block 2342 len 12288 data: 5 4 5
27646 016/300 Write block 322 len 28672 data: 6 4 5 5 5 5 6
27647 017/300 Write block 1338 len 28672 data: 5 5 5 4 5 5 5
27648 018/300 Write block 862 len 20480 data: 4 4 5 4 5
27649 019/300 Flush
27650 020/300 Flush
27651 021/300 Write block 3316 len 12288 data: 4 3 3
27652 022/300 Flush
27653 023/300 Write block 3827 len 24576 data: 3 4 4 4 4 4
27654 024/300 Write block 3634 len 12288 data: 4 4 6
27655 025/300 Read block 3683 len 20480
27656 026/300 Write block 373 len 32768 data: 5 5 5 5 7 7 6 6
27657 027/300 Write block 1026 len 4096 data: 6
27658 028/300 Flush
27659 029/300 Read block 4382 len 4096
27660 030/300 Write block 1517 len 40960 data: 4 3 4 3 3 4 5 5 5 5
27661 031/300 Write block 1781 len 40960 data: 3 4 4 6 6 5 8 7 8 7
27662 032/300 Write block 3848 len 40960 data: 6 5 6 4 5 6 6 6 6 4
27663 033/300 Write block 1175 len 16384 data: 7 7 7 9
27664 034/300 Read block 295 len 24576
27665 035/300 Read block 709 len 36864
27666 036/300 Write block 2620 len 16384 data: 5 5 5 5
27667 037/300 Read block 2738 len 32768
27668 038/300 Write block 4532 len 32768 data: 7 8 7 8 8 9 8 6
27669 039/300 Write block 1536 len 40960 data: 4 4 5 6 7 7 6 6 7 7
27670 040/300 Read block 4652 len 8192
27671 041/300 Flush
27672 042/300 Read block 4569 len 32768
27673 043/300 Read block 2700 len 24576
27674 044/300 Write block 788 len 28672 data: 6 6 5 5 4 4 3
27675 045/300 Read block 1921 len 16384
27676 046/300 Write block 2555 len 40960 data: 4 4 4 5 5 5 4 4 3 2
27677 047/300 Read block 2578 len 12288
27678 048/300 Write block 3887 len 32768 data: 3 3 3 3 3 3 4 4
27679 049/300 Read block 4101 len 20480
27680 050/300 Write block 4493 len 8192 data: 8 7
27681 051/300 Write block 1910 len 20480 data: 9 6 4 4 5
27682 052/300 Write block 1864 len 28672 data: 4 5 4 3 4 3 3
27683 053/300 Flush
27684 054/300 Read block 3181 len 8192
27685 055/300 Flush
27686 056/300 Read block 2735 len 12288
27687 057/300 Write block 3977 len 28672 data: 3 2 2 2 2 2 2
27688 058/300 Write block 3966 len 36864 data: 5 5 5 5 3 3 2 2 2
27689 059/300 Write block 3203 len 28672 data: 6 7 6 6 5 4 3
27690 060/300 Write block 4733 len 24576 data: 5 5 4 4 3 3
27691 061/300 Write block 1685 len 24576 data: 3 4 4 3 3 3
27692 062/300 Read block 3258 len 4096
27693 063/300 Read block 4857 len 4096
27694 064/300 Read block 2431 len 4096
27695 065/300 Write block 4281 len 40960 data: 5 3 4 6 5 4 5 6 6 6
27696 066/300 Read block 4447 len 20480
27697 067/300 Read block 4225 len 24576
27698 068/300 Read block 1602 len 20480
27699 069/300 Read block 3061 len 32768
27700 070/300 Read block 3890 len 4096
27701 071/300 Write block 2611 len 24576 data: 5 2 2 2 3 4
27702 072/300 Write block 2556 len 36864 data: 5 5 6 6 6 5 5 4 3
27703 073/300 Read block 1074 len 36864
27704 074/300 Write block 429 len 36864 data: 3 3 4 5 5 4 3 3 3
27705 075/300 Write block 1990 len 40960 data: 4 5 4 3 5 5 5 4 7 8
27706 076/300 Write block 2656 len 28672 data: 2 2 2 2 3 3 3
27707 077/300 Write block 2467 len 16384 data: 5 5 5 4
27708 078/300 Read block 785 len 36864
27709 079/300 Write block 3367 len 36864 data: 5 6 5 5 5 4 4 4 4
27710 080/300 Read block 4041 len 32768
27711 081/300 Read block 990 len 12288
27712 082/300 Flush
27713 083/300 Write block 4060 len 20480 data: 4 7 6 5 7
27714 084/300 Read block 3823 len 20480
27715 085/300 Write block 2132 len 28672 data: 6 6 6 6 8 8 9
27716 086/300 Read block 2897 len 36864
27717 087/300 Write block 2589 len 4096 data: 6
27718 088/300 Write block 1909 len 4096 data: 7
27719 089/300 Read block 1069 len 20480
27720 090/300 Read block 293 len 20480
27721 091/300 Flush
27722 092/300 Read block 3623 len 28672
27723 093/300 Write block 1143 len 4096 data: 3
27724 094/300 Read block 2562 len 12288
27725 095/300 Write block 3494 len 4096 data: 3
27726 096/300 Write block 1771 len 40960 data: 4 5 5 6 4 4 4 3 5 4
27727 097/300 Write block 2209 len 12288 data: 5 6 8
27728 098/300 Read block 1819 len 40960
27729 099/300 Read block 1827 len 4096
27730 100/300 Read block 3868 len 20480
27731 101/300 Read block 2370 len 20480
27732 102/300 Read block 3891 len 8192
27733 103/300 Flush
27734 104/300 Write block 838 len 12288 data: 9 10 9
27735 105/300 Read block 4376 len 24576
27736 106/300 Read block 4806 len 24576
27737 107/300 Flush
27738 108/300 Write block 515 len 28672 data: 7 7 5 5 6 5 5
27739 109/300 Read block 1271 len 32768
27740 110/300 Read block 2679 len 32768
27741 111/300 Read block 4567 len 8192
27742 112/300 Read block 2757 len 20480
27743 113/300 Write block 4385 len 28672 data: 5 6 4 4 4 7 8
27744 114/300 Read block 3263 len 36864
27745 115/300 Read block 3767 len 36864
27746 116/300 Read block 4445 len 36864
27747 117/300 Write block 2156 len 40960 data: 5 5 4 4 3 3 2 2 3 5
27748 118/300 Write block 2987 len 8192 data: 3 4
27749 119/300 Write block 2532 len 32768 data: 4 4 4 5 6 6 7 7
27750 120/300 Read block 1849 len 24576
27751 121/300 Write block 2488 len 40960 data: 3 4 4 4 5 5 5 5 5 5
27752 122/300 Read block 3209 len 4096
27753 123/300 Write block 2541 len 32768 data: 6 6 5 4 4 4 3 3
27754 124/300 Write block 3082 len 28672 data: 7 7 5 3 4 3 3
27755 125/300 Read block 4047 len 4096
27756 126/300 Write block 4826 len 32768 data: 4 4 4 4 3 3 3 3
27757 127/300 Flush
27758 128/300 Read block 471 len 4096
27759 129/300 Write block 4658 len 24576 data: 5 5 5 8 7 9
27760 130/300 Read block 1513 len 20480
27761 131/300 Read block 469 len 40960
27762 132/300 Write block 131 len 4096 data: 5
27763 133/300 Write block 1232 len 40960 data: 5 5 7 7 7 8 7 5 5 6
27764 134/300 Write block 4074 len 40960 data: 3 3 3 4 5 5 4 3 3 4
27765 135/300 Read block 2591 len 12288
27766 136/300 Read block 2087 len 16384
27767 137/300 Write block 2431 len 12288 data: 6 4 5
27768 138/300 Read block 1189 len 20480
27769 139/300 Read block 1740 len 20480
27770 140/300 Read block 374 len 32768
27771 141/300 Read block 1581 len 16384
27772 142/300 Flush
27773 143/300 Write block 2199 len 36864 data: 6 6 5 4 4 4 3 5 6
27774 144/300 Read block 2408 len 4096
27775 145/300 Read block 1287 len 36864
27776 146/300 Read block 3492 len 12288
27777 147/300 Write block 4498 len 24576 data: 6 6 6 5 6 4
27778 148/300 Read block 2400 len 32768
27779 149/300 Write block 4976 len 32768 data: 5 4 4 4 4 4 5 5
27780 150/300 Write block 668 len 36864 data: 2 2 2 2 2 3 4 4 4
27781 151/300 Read block 4150 len 32768
27782 152/300 Read block 1797 len 12288
27783 153/300 Flush
27784 154/300 Read block 2903 len 40960
27785 155/300 Write block 4951 len 28672 data: 4 5 4 5 4 5 4
27786 156/300 Read block 1889 len 40960
27787 157/300 Flush
27788 158/300 Read block 585 len 28672
27789 159/300 Write block 2112 len 28672 data: 4 5 5 4 6 6 6
27790 160/300 Read block 740 len 36864
27791 161/300 Read block 4764 len 32768
27792 162/300 Read block 4094 len 4096
27793 163/300 Read block 198 len 20480
27794 164/300 Write block 4106 len 36864 data: 4 4 4 3 3 3 3 4 6
27795 165/300 Flush
27796 166/300 Write block 1316 len 40960 data: 8 6 6 6 7 7 7 5 6 4
27797 167/300 Write block 326 len 36864 data: 6 6 7 7 6 4 4 6 6
27798 168/300 Write block 3490 len 8192 data: 7 6
27799 169/300 Write block 3920 len 8192 data: 8 9
27800 170/300 Flush
27801 171/300 Read block 4708 len 4096
27802 172/300 Read block 3064 len 12288
27803 173/300 Read block 3281 len 40960
27804 174/300 Write block 3041 len 28672 data: 5 5 5 5 4 6 6
27805 175/300 Read block 2545 len 36864
27806 176/300 Read block 4476 len 12288
27807 177/300 Write block 271 len 16384 data: 7 7 6 6
27808 178/300 Read block 749 len 36864
27809 179/300 Flush
27810 180/300 Read block 4399 len 36864
27811 181/300 Read block 481 len 4096
27812 182/300 Read block 4905 len 16384
27813 183/300 Write block 3049 len 8192 data: 5 6
27814 184/300 Read block 225 len 16384
27815 185/300 Read block 1719 len 36864
27816 186/300 Read block 1601 len 20480
27817 187/300 Write block 1282 len 28672 data: 4 4 5 4 3 3 4
27818 188/300 Read block 450 len 8192
27819 189/300 Read block 1905 len 4096
27820 190/300 Write block 3656 len 32768 data: 12 10 9 8 7 7 6 6
27821 191/300 Write block 1854 len 8192 data: 4 4
27822 192/300 Read block 1508 len 40960
27823 193/300 Flush
27824 194/300 Read block 1968 len 12288
27825 195/300 Write block 4064 len 24576 data: 8 7 6 6 4 4
27826 196/300 Read block 1400 len 8192
27827 197/300 Write block 101 len 24576 data: 5 4 4 4 5 3
27828 198/300 Read block 1699 len 8192
27829 199/300 Read block 2484 len 8192
27830 200/300 Write block 2979 len 12288 data: 4 3 3
27831 201/300 Read block 1520 len 24576
27832 202/300 Write block 3110 len 4096 data: 6
27833 203/300 Write block 1762 len 8192 data: 4 5
27834 204/300 Write block 3425 len 20480 data: 5 7 7 7 7
27835 205/300 Write block 1118 len 20480 data: 5 5 4 4 5
27836 206/300 Flush
27837 207/300 Write block 1802 len 16384 data: 4 4 3 3
27838 208/300 Write block 120 len 32768 data: 4 4 4 3 5 6 6 6
27839 209/300 Read block 1542 len 20480
27840 210/300 Read block 4814 len 32768
27841 211/300 Write block 491 len 8192 data: 6 6
27842 212/300 Read block 2373 len 16384
27843 213/300 Write block 4665 len 24576 data: 7 8 4 4 5 5
27844 214/300 Read block 359 len 28672
27845 215/300 Read block 1695 len 28672
27846 216/300 Read block 2705 len 12288
27847 217/300 Read block 4873 len 4096
27848 218/300 Read block 1876 len 24576
27849 219/300 Write block 4013 len 28672 data: 4 4 5 5 5 5 4
27850 220/300 Read block 3802 len 24576
27851 221/300 Write block 4153 len 40960 data: 4 4 3 4 5 7 7 6 4 3
27852 222/300 Read block 4508 len 16384
27853 223/300 Read block 25 len 24576
27854 224/300 Read block 1189 len 4096
27855 225/300 Write block 1930 len 40960 data: 7 8 8 8 8 9 10 9 6 7
27856 226/300 Write block 849 len 32768 data: 4 4 4 4 5 4 5 4
27857 227/300 Read block 4440 len 40960
27858 228/300 Write block 855 len 28672 data: 6 5 4 4 5 4 5
27859 229/300 Write block 1498 len 8192 data: 3 3
27860 230/300 Read block 3117 len 8192
27861 231/300 Read block 1476 len 12288
27862 232/300 Flush
27863 233/300 Write block 3241 len 32768 data: 6 5 5 5 5 4 4 3
27864 234/300 Read block 2392 len 8192
27865 235/300 Write block 3474 len 36864 data: 2 3 4 4 4 5 5 6 5
27866 236/300 Read block 1315 len 8192
27867 237/300 Flush
27868 238/300 Read block 2675 len 28672
27869 239/300 Read block 2736 len 28672
27870 240/300 Write block 3547 len 12288 data: 4 4 5
27871 241/300 Read block 4653 len 28672
27872 242/300 Read block 3886 len 24576
27873 243/300 Write block 1693 len 40960 data: 4 4 3 3 5 6 8 10 10 11
27874 244/300 Read block 2466 len 4096
27875 245/300 Flush
27876 246/300 Read block 1694 len 12288
27877 247/300 Flush
27878 248/300 Read block 1665 len 16384
27879 249/300 Read block 1591 len 32768
27880 250/300 Write block 4807 len 16384 data: 6 7 6 6
27881 251/300 Write block 4144 len 8192 data: 6 7
27882 252/300 Write block 3102 len 12288 data: 7 6 6
27883 253/300 Read block 656 len 40960
27884 254/300 Write block 380 len 28672 data: 7 5 3 2 2 2 2
27885 255/300 Flush
27886 256/300 Write block 2221 len 12288 data: 8 9 9
27887 257/300 Read block 4365 len 8192
27888 258/300 Read block 1146 len 36864
27889 259/300 Write block 2080 len 16384 data: 6 6 6 6
27890 260/300 Read block 4914 len 36864
27891 261/300 Read block 563 len 8192
27892 262/300 Read block 1768 len 20480
27893 263/300 Write block 2724 len 36864 data: 4 4 4 3 4 4 5 5 4
27894 264/300 Write block 3551 len 16384 data: 6 6 8 10
27895 265/300 Read block 2234 len 8192
27896 266/300 Write block 758 len 24576 data: 3 3 3 3 4 4
27897 267/300 Read block 1753 len 36864
27898 268/300 Read block 2724 len 40960
27899 269/300 Read block 2282 len 36864
27900 270/300 Write block 1981 len 12288 data: 8 7 8
27901 271/300 Read block 4753 len 32768
27902 272/300 Write block 2487 len 32768 data: 4 4 5 5 5 6 6 6
27903 273/300 Read block 831 len 12288
27904 274/300 Write block 4906 len 12288 data: 2 2 3
27905 275/300 Read block 3334 len 28672
27906 276/300 Read block 4095 len 40960
27907 277/300 Write block 2328 len 32768 data: 6 7 8 6 7 7 7 6
27908 278/300 Read block 3326 len 32768
27909 279/300 Read block 939 len 4096
27910 280/300 Read block 2625 len 8192
27911 281/300 Write block 320 len 16384 data: 6 6 7 5
27912 282/300 Write block 1218 len 32768 data: 5 5 4 3 3 3 3 2
27913 283/300 Read block 1389 len 16384
27914 284/300 Write block 726 len 24576 data: 3 4 6 7 8 7
27915 285/300 Write block 1500 len 20480 data: 3 5 5 6 7
27916 286/300 Flush
27917 287/300 Read block 2011 len 32768
27918 288/300 Read block 1069 len 40960
27919 289/300 Read block 2223 len 8192
27920 290/300 Read block 1482 len 16384
27921 291/300 Flush
27922 292/300 Read block 16 len 8192
27923 293/300 Write block 2486 len 8192 data: 5 5
27924 294/300 Read block 2092 len 24576
27925 295/300 Read block 2012 len 32768
27926 296/300 Write block 4806 len 12288 data: 7 7 8
27927 297/300 Write block 4769 len 8192 data: 7 7
27928 298/300 Write block 4659 len 16384 data: 6 6 9 8
27929 299/300 Write block 3075 len 40960 data: 6 8 6 5 5 7 6 8 8 6
27930 300/300 Read block 4610 len 4096
27931 301/300 Read block 4080 len 24576
27932 Main task has work to do, go find it
27933 got dsc Start(1)
27934 start 1
27935 Feb 25 17:24:24.414 INFO request completed, latency_us: 231, response_code: 204, uri: /start/cid/1, method: POST, req_id: 797cafee-f091-4b49-86b3-1b8fb8ff1347, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
27936 Replay: started 1, returned:Ok(())
27937 ----------------------------------------------------------------
27938 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27939 States: Active Offline Active
27940 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27941 8246 Acked Flush 0 Done Sent Done false
27942 8247 Acked Read 5 Done Sent Done false
27943 8248 Acked Read 5 Done Sent Done false
27944 8249 Acked Read 5 Done Sent Done false
27945 8250 Acked Write 3 Done Sent Done false
27946 8251 Acked Write 9 Done Sent Done false
27947 8252 Acked Flush 0 Done Sent Done false
27948 8253 Acked Read 8 Done Sent Done false
27949 8254 Acked Flush 0 Done Sent Done false
27950 8255 Acked Write 10 Done Sent Done false
27951 8256 Acked Flush 0 Done Sent Done false
27952 8257 Acked Flush 0 Done Sent Done false
27953 8258 Acked Read 9 Done Sent Done false
27954 8259 Acked Read 10 Done Sent Done false
27955 8260 Acked Write 3 Done Sent Done false
27956 8261 Acked Write 7 Done Sent Done false
27957 8262 Acked Write 7 Done Sent Done false
27958 8263 Acked Write 5 Done Sent Done false
27959 8264 Acked Flush 0 Done Sent Done false
27960 8265 Acked Flush 0 Done Sent Done false
27961 8266 Acked Write 3 Done Sent Done false
27962 8267 Acked Flush 0 Done Sent Done false
27963 8268 Acked Write 6 Done Sent Done false
27964 8269 Acked Write 3 Done Sent Done false
27965 8270 Acked Read 5 Done Sent Done false
27966 8271 Acked Write 8 Done Sent Done false
27967 8272 Acked Write 1 Done Sent Done false
27968 8273 Acked Flush 0 Done Sent Done false
27969 8274 Acked Read 1 Done Sent Done false
27970 8275 Acked Write 10 Done Sent Done false
27971 8276 Acked Write 10 Done Sent Done false
27972 8277 Acked Write 10 Done Sent Done false
27973 8278 Acked Write 4 Done Sent Done false
27974 8279 Acked Read 6 Done Sent Done false
27975 8280 Acked Read 9 Done Sent Done false
27976 8281 Acked Write 4 Done Sent Done false
27977 8282 Acked Read 8 Done Sent Done false
27978 8283 Acked Write 8 Done Sent Done false
27979 8284 Acked Write 10 Done Sent Done false
27980 8285 Acked Read 2 Done Sent Done false
27981 8286 Acked Flush 0 Done Sent Done false
27982 8287 Acked Read 8 Done Sent Done false
27983 8288 Acked Read 6 Done Sent Done false
27984 8289 Acked Write 7 Done Sent Done false
27985 8290 Acked Read 4 Done Sent Done false
27986 8291 Acked Write 10 Done Sent Done false
27987 8292 Acked Read 3 Done Sent Done false
27988 8293 Acked Write 8 Done Sent Done false
27989 8294 Acked Read 5 Done Sent Done false
27990 8295 Acked Write 2 Done Sent Done false
27991 8296 Acked Write 5 Done Sent Done false
27992 8297 Acked Write 7 Done Sent Done false
27993 8298 Acked Flush 0 Done Sent Done false
27994 8299 Acked Read 2 Done Sent Done false
27995 8300 Acked Flush 0 Done Sent Done false
27996 8301 Acked Read 3 Done Sent Done false
27997 8302 Acked Write 7 Done Sent Done false
27998 8303 Acked Write 9 Done Sent Done false
27999 8304 Acked Write 7 Done Sent Done false
28000 8305 Acked Write 6 Done Sent Done false
28001 8306 Acked Write 6 Done Sent Done false
28002 8307 Acked Read 1 Done Sent Done false
28003 8308 Acked Read 1 Done Sent Done false
28004 8309 Acked Read 1 Done Sent Done false
28005 8310 Acked Write 10 Done Sent Done false
28006 8311 Acked Read 5 Done Sent Done false
28007 8312 Acked Read 6 Done Sent Done false
28008 8313 Acked Read 5 Done Sent Done false
28009 8314 Acked Read 8 Done Sent Done false
28010 8315 Acked Read 1 Done Sent Done false
28011 8316 Acked Write 6 Done Sent Done false
28012 8317 Acked Write 9 Done Sent Done false
28013 8318 Acked Read 9 Done Sent Done false
28014 8319 Acked Write 9 Done Sent Done false
28015 8320 Acked Write 10 Done Sent Done false
28016 8321 Acked Write 7 Done Sent Done false
28017 8322 Acked Write 4 Done Sent Done false
28018 8323 Acked Read 9 Done Sent Done false
28019 8324 Acked Write 9 Done Sent Done false
28020 8325 Acked Read 8 Done Sent Done false
28021 8326 Acked Read 3 Done Sent Done false
28022 8327 Acked Flush 0 Done Sent Done false
28023 8328 Acked Write 5 Done Sent Done false
28024 8329 Acked Read 5 Done Sent Done false
28025 8330 Acked Write 7 Done Sent Done false
28026 8331 Acked Read 9 Done Sent Done false
28027 8332 Acked Write 1 Done Sent Done false
28028 8333 Acked Write 1 Done Sent Done false
28029 8334 Acked Read 5 Done Sent Done false
28030 8335 Acked Read 5 Done Sent Done false
28031 8336 Acked Flush 0 Done Sent Done false
28032 8337 Acked Read 7 Done Sent Done false
28033 8338 Acked Write 1 Done Sent Done false
28034 8339 Acked Read 3 Done Sent Done false
28035 8340 Acked Write 1 Done Sent Done false
28036 8341 Acked Write 10 Done Sent Done false
28037 8342 Acked Write 3 Done Sent Done false
28038 8343 Acked Read 10 Done Sent Done false
28039 8344 Acked Read 1 Done Sent Done false
28040 8345 Acked Read 5 Done Sent Done false
28041 8346 Acked Read 5 Done Sent Done false
28042 8347 Acked Read 2 Done Sent Done false
28043 8348 Acked Flush 0 Done Sent Done false
28044 8349 Acked Write 3 Done Sent Done false
28045 8350 Acked Read 6 Done Sent Done false
28046 8351 Acked Read 6 Done Sent Done false
28047 8352 Acked Flush 0 Done Sent Done false
28048 8353 Acked Write 7 Done Sent Done false
28049 8354 Acked Read 8 Done Sent Done false
28050 8355 Acked Read 8 Done Sent Done false
28051 8356 Acked Read 2 Done Sent Done false
28052 8357 Acked Read 5 Done Sent Done false
28053 8358 Acked Write 7 Done Sent Done false
28054 8359 Acked Read 9 Done Sent Done false
28055 8360 Acked Read 9 Done Sent Done false
28056 8361 Acked Read 9 Done Sent Done false
28057 8362 Acked Write 10 Done Sent Done false
28058 8363 Acked Write 2 Done Sent Done false
28059 8364 Acked Write 8 Done Sent Done false
28060 8365 Acked Read 6 Done Sent Done false
28061 8366 Acked Write 10 Done Sent Done false
28062 8367 Acked Read 1 Done Sent Done false
28063 8368 Acked Write 8 Done Sent Done false
28064 8369 Acked Write 7 Done Sent Done false
28065 8370 Acked Read 1 Done Sent Done false
28066 8371 Acked Write 8 Done Sent Done false
28067 8372 Acked Flush 0 Done Sent Done false
28068 8373 Acked Read 1 Done Sent Done false
28069 8374 Acked Write 6 Done Sent Done false
28070 8375 Acked Read 5 Done Sent Done false
28071 8376 Acked Read 10 Done Sent Done false
28072 8377 Acked Write 1 Done Sent Done false
28073 8378 Acked Write 10 Done Sent Done false
28074 8379 Acked Write 10 Done Sent Done false
28075 8380 Acked Read 3 Done Sent Done false
28076 8381 Acked Read 4 Done Sent Done false
28077 8382 Acked Write 3 Done Sent Done false
28078 8383 Acked Read 5 Done Sent Done false
28079 8384 Acked Read 5 Done Sent Done false
28080 8385 Acked Read 8 Done Sent Done false
28081 8386 Acked Read 4 Done Sent Done false
28082 8387 Acked Flush 0 Done Sent Done false
28083 8388 Acked Write 9 Done Sent Done false
28084 8389 Acked Read 1 Done Sent Done false
28085 8390 Acked Read 9 Done Sent Done false
28086 8391 Acked Read 3 Done Sent Done false
28087 8392 Acked Write 6 Done Sent Done false
28088 8393 Acked Read 8 Done Sent Done false
28089 8394 Acked Write 8 Done Sent Done false
28090 8395 Acked Write 9 Done Sent Done false
28091 8396 Acked Read 8 Done Sent Done false
28092 8397 Acked Read 3 Done Sent Done false
28093 8398 Acked Flush 0 Done Sent Done false
28094 8399 Acked Read 10 Done Sent Done false
28095 8400 Acked Write 7 Done Sent Done false
28096 8401 Acked Read 10 Done Sent Done false
28097 8402 Acked Flush 0 Done Sent Done false
28098 8403 Acked Read 7 Done Sent Done false
28099 8404 Acked Write 7 Done Sent Done false
28100 8405 Acked Read 9 Done Sent Done false
28101 8406 Acked Read 8 Done Sent Done false
28102 8407 Acked Read 1 Done Sent Done false
28103 8408 Acked Read 5 Done Sent Done false
28104 8409 Acked Write 9 Done Sent Done false
28105 8410 Acked Flush 0 Done Sent Done false
28106 8411 Acked Write 10 Done Sent Done false
28107 8412 Acked Write 9 Done Sent Done false
28108 8413 Acked Write 2 Done Sent Done false
28109 8414 Acked Write 2 Done Sent Done false
28110 8415 Acked Flush 0 Done Sent Done false
28111 8416 Acked Read 1 Done Sent Done false
28112 8417 Acked Read 3 Done Sent Done false
28113 8418 Acked Read 10 Done Sent Done false
28114 8419 Acked Write 7 Done Sent Done false
28115 8420 Acked Read 9 Done Sent Done false
28116 8421 Acked Read 3 Done Sent Done false
28117 8422 Acked Write 4 Done Sent Done false
28118 8423 Acked Read 9 Done Sent Done false
28119 8424 Acked Flush 0 Done Sent Done false
28120 8425 Acked Read 9 Done Sent Done false
28121 8426 Acked Read 1 Done Sent Done false
28122 8427 Acked Read 4 Done Sent Done false
28123 8428 Acked Write 2 Done Sent Done false
28124 8429 Acked Read 4 Done Sent Done false
28125 8430 Acked Read 9 Done Sent Done false
28126 8431 Acked Read 5 Done Sent Done false
28127 8432 Acked Write 7 Done Sent Done false
28128 8433 Acked Read 2 Done Sent Done false
28129 8434 Acked Read 1 Done Sent Done false
28130 8435 Acked Write 8 Done Sent Done false
28131 8436 Acked Write 2 Done Sent Done false
28132 8437 Acked Read 10 Done Sent Done false
28133 8438 Acked Flush 0 Done Sent Done false
28134 8439 Acked Read 3 Done Sent Done false
28135 8440 Acked Write 6 Done Sent Done false
28136 8441 Acked Read 2 Done Sent Done false
28137 8442 Acked Write 6 Done Sent Done false
28138 8443 Acked Read 2 Done Sent Done false
28139 8444 Acked Read 2 Done Sent Done false
28140 8445 Acked Write 3 Done Sent Done false
28141 8446 Acked Read 6 Done Sent Done false
28142 8447 Acked Write 1 Done Sent Done false
28143 8448 Acked Write 2 Done Sent Done false
28144 8449 Acked Write 5 Done Sent Done false
28145 8450 Acked Write 5 Done Sent Done false
28146 8451 Acked Flush 0 Done Sent Done false
28147 8452 Acked Write 4 Done Sent Done false
28148 8453 Acked Write 8 Done Sent Done false
28149 8454 Acked Read 5 Done Sent Done false
28150 8455 Acked Read 8 Done Sent Done false
28151 8456 Acked Write 2 Done Sent Done false
28152 8457 Acked Read 4 Done Sent Done false
28153 8458 Acked Write 6 Done Sent Done false
28154 8459 Acked Read 7 Done Sent Done false
28155 8460 Acked Read 7 Done Sent Done false
28156 8461 Acked Read 3 Done Sent Done false
28157 8462 Acked Read 1 Done Sent Done false
28158 8463 Acked Read 6 Done Sent Done false
28159 8464 Acked Write 7 Done Sent Done false
28160 8465 Acked Read 6 Done Sent Done false
28161 8466 Acked Write 10 Done Sent Done false
28162 8467 Acked Read 4 Done Sent Done false
28163 8468 Acked Read 6 Done Sent Done false
28164 8469 Acked Read 1 Done Sent Done false
28165 8470 Acked Write 10 Done Sent Done false
28166 8471 Acked Write 8 Done Sent Done false
28167 8472 Acked Read 10 Done Sent Done false
28168 8473 Acked Write 7 Done Sent Done false
28169 8474 Acked Write 2 Done Sent Done false
28170 8475 Acked Read 2 Done Sent Done false
28171 8476 Acked Read 3 Done Sent Done false
28172 8477 Acked Flush 0 Done Sent Done false
28173 8478 Acked Write 8 Done Sent Done false
28174 8479 Acked Read 2 Done Sent Done false
28175 8480 Acked Write 9 Done Sent Done false
28176 8481 Acked Read 2 Done Sent Done false
28177 8482 Acked Flush 0 Done Sent Done false
28178 8483 Acked Read 7 Done Sent Done false
28179 8484 Acked Read 7 Done Sent Done false
28180 8485 Acked Write 3 Done Sent Done false
28181 8486 Acked Read 7 Done Sent Done false
28182 8487 Acked Read 6 Done Sent Done false
28183 8488 Acked Write 10 Done Sent Done false
28184 8489 Acked Read 1 Done Sent Done false
28185 8490 Acked Flush 0 Done Sent Done false
28186 8491 Acked Read 3 Done Sent Done false
28187 8492 Acked Flush 0 Done Sent Done false
28188 8493 Acked Read 4 Done Sent Done false
28189 8494 Acked Read 8 Done Sent Done false
28190 8495 Acked Write 4 Done Sent Done false
28191 8496 Acked Write 2 Done Sent Done false
28192 8497 Acked Write 3 Done Sent Done false
28193 8498 Acked Read 10 Done Sent Done false
28194 8499 Acked Write 7 Done Sent Done false
28195 8500 Acked Flush 0 Done Sent Done false
28196 8501 Acked Write 3 Done Sent Done false
28197 8502 Acked Read 2 Done Sent Done false
28198 8503 Acked Read 9 Done Sent Done false
28199 8504 Acked Write 4 Done Sent Done false
28200 8505 Acked Read 9 Done Sent Done false
28201 8506 Acked Read 2 Done Sent Done false
28202 8507 Acked Read 5 Done Sent Done false
28203 8508 Acked Write 9 Done Sent Done false
28204 8509 Acked Write 4 Done Sent Done false
28205 8510 Acked Read 2 Done Sent Done false
28206 8511 Acked Write 6 Done Sent Done false
28207 8512 Acked Read 9 Done Sent Done false
28208 8513 Acked Read 10 Done Sent Done false
28209 8514 Acked Read 9 Done Sent Done false
28210 8515 Acked Write 3 Done Sent Done false
28211 8516 Acked Read 8 Done Sent Done false
28212 8517 Acked Write 8 Done Sent Done false
28213 8518 Acked Read 3 Done Sent Done false
28214 8519 Acked Write 3 Done Sent Done false
28215 8520 Acked Read 7 Done Sent Done false
28216 8521 Acked Read 10 Done Sent Done false
28217 8522 Acked Write 8 Done Sent Done false
28218 8523 Acked Read 8 Done Sent Done false
28219 8524 Acked Read 1 Done Sent Done false
28220 8525 Acked Read 2 Done Sent Done false
28221 8526 Acked Write 4 Done Sent Done false
28222 8527 Acked Write 8 Done Sent Done false
28223 8528 Acked Read 4 Done Sent Done false
28224 8529 Acked Write 6 Done Sent Done false
28225 8530 Acked Write 5 Done Sent Done false
28226 8531 Acked Flush 0 Done Sent Done false
28227 8532 Acked Read 8 Done Sent Done false
28228 8533 Acked Read 10 Done Sent Done false
28229 8534 Acked Read 2 Done Sent Done false
28230 8535 Acked Read 4 Done Sent Done false
28231 8536 Acked Flush 0 Done Sent Done false
28232 8537 Acked Read 2 Done Sent Done false
28233 8538 Acked Write 2 Done Sent Done false
28234 8539 Acked Read 6 Done Sent Done false
28235 8540 Acked Read 8 Done Sent Done false
28236 8541 Acked Write 3 Done Sent Done false
28237 8542 Acked Write 2 Done Sent Done false
28238 8543 Acked Write 4 Done Sent Done false
28239 8544 Acked Write 10 Done Sent Done false
28240 8545 Acked Read 1 Done Sent Done false
28241 8546 Acked Read 6 Done Sent Done false
28242 STATES DS:0 DS:1 DS:2 TOTAL
28243 Sent 0 301 0 301
28244 Done 301 0 301 602
28245 Skipped 0 0 0 0
28246 Error 0 0 0 0
28247 Last Flush: Some(JobId(8536)) Some(JobId(8245)) Some(JobId(8536))
28248 Downstairs last five completed: 8245 8244 8243 8242 8241
28249 Upstairs last five completed: 8546 8545 8544 8543 8542
28250 CLIENT: Up:0 ds:301 act:2
282512025-02-25T17:24:24.916ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
282522025-02-25T17:24:25.876ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
282532025-02-25T17:24:26.851ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
282542025-02-25T17:24:27.852ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
28255 [8820] got start action
28256 [8820] I am going to restart
28257 Starting downstairs at port 8820
28258 Make output file at "/tmp/dsc/downstairs-8820.txt"
28259 [8820][1] reports Starting
28260 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1233)
28261 [8820][1] reports Running
28262 ----------------------------------------------------------------
28263 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28264 States: Active Offline Active
28265 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28266 8246 Acked Flush 0 Done Sent Done false
28267 8247 Acked Read 5 Done Sent Done false
28268 8248 Acked Read 5 Done Sent Done false
28269 8249 Acked Read 5 Done Sent Done false
28270 8250 Acked Write 3 Done Sent Done false
28271 8251 Acked Write 9 Done Sent Done false
28272 8252 Acked Flush 0 Done Sent Done false
28273 8253 Acked Read 8 Done Sent Done false
28274 8254 Acked Flush 0 Done Sent Done false
28275 8255 Acked Write 10 Done Sent Done false
28276 8256 Acked Flush 0 Done Sent Done false
28277 8257 Acked Flush 0 Done Sent Done false
28278 8258 Acked Read 9 Done Sent Done false
28279 8259 Acked Read 10 Done Sent Done false
28280 8260 Acked Write 3 Done Sent Done false
28281 8261 Acked Write 7 Done Sent Done false
28282 8262 Acked Write 7 Done Sent Done false
28283 8263 Acked Write 5 Done Sent Done false
28284 8264 Acked Flush 0 Done Sent Done false
28285 8265 Acked Flush 0 Done Sent Done false
28286 8266 Acked Write 3 Done Sent Done false
28287 8267 Acked Flush 0 Done Sent Done false
28288 8268 Acked Write 6 Done Sent Done false
28289 8269 Acked Write 3 Done Sent Done false
28290 8270 Acked Read 5 Done Sent Done false
28291 8271 Acked Write 8 Done Sent Done false
28292 8272 Acked Write 1 Done Sent Done false
28293 8273 Acked Flush 0 Done Sent Done false
28294 8274 Acked Read 1 Done Sent Done false
28295 8275 Acked Write 10 Done Sent Done false
28296 8276 Acked Write 10 Done Sent Done false
28297 8277 Acked Write 10 Done Sent Done false
28298 8278 Acked Write 4 Done Sent Done false
28299 8279 Acked Read 6 Done Sent Done false
28300 8280 Acked Read 9 Done Sent Done false
28301 8281 Acked Write 4 Done Sent Done false
28302 8282 Acked Read 8 Done Sent Done false
28303 8283 Acked Write 8 Done Sent Done false
28304 8284 Acked Write 10 Done Sent Done false
28305 8285 Acked Read 2 Done Sent Done false
28306 8286 Acked Flush 0 Done Sent Done false
28307 8287 Acked Read 8 Done Sent Done false
28308 8288 Acked Read 6 Done Sent Done false
28309 8289 Acked Write 7 Done Sent Done false
28310 8290 Acked Read 4 Done Sent Done false
28311 8291 Acked Write 10 Done Sent Done false
28312 8292 Acked Read 3 Done Sent Done false
28313 8293 Acked Write 8 Done Sent Done false
28314 8294 Acked Read 5 Done Sent Done false
28315 8295 Acked Write 2 Done Sent Done false
28316 8296 Acked Write 5 Done Sent Done false
28317 8297 Acked Write 7 Done Sent Done false
28318 8298 Acked Flush 0 Done Sent Done false
28319 8299 Acked Read 2 Done Sent Done false
28320 8300 Acked Flush 0 Done Sent Done false
28321 8301 Acked Read 3 Done Sent Done false
28322 8302 Acked Write 7 Done Sent Done false
28323 8303 Acked Write 9 Done Sent Done false
28324 8304 Acked Write 7 Done Sent Done false
28325 8305 Acked Write 6 Done Sent Done false
28326 8306 Acked Write 6 Done Sent Done false
28327 8307 Acked Read 1 Done Sent Done false
28328 8308 Acked Read 1 Done Sent Done false
28329 8309 Acked Read 1 Done Sent Done false
28330 8310 Acked Write 10 Done Sent Done false
28331 8311 Acked Read 5 Done Sent Done false
28332 8312 Acked Read 6 Done Sent Done false
28333 8313 Acked Read 5 Done Sent Done false
28334 8314 Acked Read 8 Done Sent Done false
28335 8315 Acked Read 1 Done Sent Done false
28336 8316 Acked Write 6 Done Sent Done false
28337 8317 Acked Write 9 Done Sent Done false
28338 8318 Acked Read 9 Done Sent Done false
28339 8319 Acked Write 9 Done Sent Done false
28340 8320 Acked Write 10 Done Sent Done false
28341 8321 Acked Write 7 Done Sent Done false
28342 8322 Acked Write 4 Done Sent Done false
28343 8323 Acked Read 9 Done Sent Done false
28344 8324 Acked Write 9 Done Sent Done false
28345 8325 Acked Read 8 Done Sent Done false
28346 8326 Acked Read 3 Done Sent Done false
28347 8327 Acked Flush 0 Done Sent Done false
28348 8328 Acked Write 5 Done Sent Done false
28349 8329 Acked Read 5 Done Sent Done false
28350 8330 Acked Write 7 Done Sent Done false
28351 8331 Acked Read 9 Done Sent Done false
28352 8332 Acked Write 1 Done Sent Done false
28353 8333 Acked Write 1 Done Sent Done false
28354 8334 Acked Read 5 Done Sent Done false
28355 8335 Acked Read 5 Done Sent Done false
28356 8336 Acked Flush 0 Done Sent Done false
28357 8337 Acked Read 7 Done Sent Done false
28358 8338 Acked Write 1 Done Sent Done false
28359 8339 Acked Read 3 Done Sent Done false
28360 8340 Acked Write 1 Done Sent Done false
28361 8341 Acked Write 10 Done Sent Done false
28362 8342 Acked Write 3 Done Sent Done false
28363 8343 Acked Read 10 Done Sent Done false
28364 8344 Acked Read 1 Done Sent Done false
28365 8345 Acked Read 5 Done Sent Done false
28366 8346 Acked Read 5 Done Sent Done false
28367 8347 Acked Read 2 Done Sent Done false
28368 8348 Acked Flush 0 Done Sent Done false
28369 8349 Acked Write 3 Done Sent Done false
28370 8350 Acked Read 6 Done Sent Done false
28371 8351 Acked Read 6 Done Sent Done false
28372 8352 Acked Flush 0 Done Sent Done false
28373 8353 Acked Write 7 Done Sent Done false
28374 8354 Acked Read 8 Done Sent Done false
28375 8355 Acked Read 8 Done Sent Done false
28376 8356 Acked Read 2 Done Sent Done false
28377 8357 Acked Read 5 Done Sent Done false
28378 8358 Acked Write 7 Done Sent Done false
28379 8359 Acked Read 9 Done Sent Done false
28380 8360 Acked Read 9 Done Sent Done false
28381 8361 Acked Read 9 Done Sent Done false
28382 8362 Acked Write 10 Done Sent Done false
28383 8363 Acked Write 2 Done Sent Done false
28384 8364 Acked Write 8 Done Sent Done false
28385 8365 Acked Read 6 Done Sent Done false
28386 8366 Acked Write 10 Done Sent Done false
28387 8367 Acked Read 1 Done Sent Done false
28388 8368 Acked Write 8 Done Sent Done false
28389 8369 Acked Write 7 Done Sent Done false
28390 8370 Acked Read 1 Done Sent Done false
28391 8371 Acked Write 8 Done Sent Done false
28392 8372 Acked Flush 0 Done Sent Done false
28393 8373 Acked Read 1 Done Sent Done false
28394 8374 Acked Write 6 Done Sent Done false
28395 8375 Acked Read 5 Done Sent Done false
28396 8376 Acked Read 10 Done Sent Done false
28397 8377 Acked Write 1 Done Sent Done false
28398 8378 Acked Write 10 Done Sent Done false
28399 8379 Acked Write 10 Done Sent Done false
28400 8380 Acked Read 3 Done Sent Done false
28401 8381 Acked Read 4 Done Sent Done false
28402 8382 Acked Write 3 Done Sent Done false
28403 8383 Acked Read 5 Done Sent Done false
28404 8384 Acked Read 5 Done Sent Done false
28405 8385 Acked Read 8 Done Sent Done false
28406 8386 Acked Read 4 Done Sent Done false
28407 8387 Acked Flush 0 Done Sent Done false
28408 8388 Acked Write 9 Done Sent Done false
28409 8389 Acked Read 1 Done Sent Done false
28410 8390 Acked Read 9 Done Sent Done false
28411 8391 Acked Read 3 Done Sent Done false
28412 8392 Acked Write 6 Done Sent Done false
28413 8393 Acked Read 8 Done Sent Done false
28414 8394 Acked Write 8 Done Sent Done false
28415 8395 Acked Write 9 Done Sent Done false
28416 8396 Acked Read 8 Done Sent Done false
28417 8397 Acked Read 3 Done Sent Done false
28418 8398 Acked Flush 0 Done Sent Done false
28419 8399 Acked Read 10 Done Sent Done false
28420 8400 Acked Write 7 Done Sent Done false
28421 8401 Acked Read 10 Done Sent Done false
28422 8402 Acked Flush 0 Done Sent Done false
28423 8403 Acked Read 7 Done Sent Done false
28424 8404 Acked Write 7 Done Sent Done false
28425 8405 Acked Read 9 Done Sent Done false
28426 8406 Acked Read 8 Done Sent Done false
28427 8407 Acked Read 1 Done Sent Done false
28428 8408 Acked Read 5 Done Sent Done false
28429 8409 Acked Write 9 Done Sent Done false
28430 8410 Acked Flush 0 Done Sent Done false
28431 8411 Acked Write 10 Done Sent Done false
28432 8412 Acked Write 9 Done Sent Done false
28433 8413 Acked Write 2 Done Sent Done false
28434 8414 Acked Write 2 Done Sent Done false
28435 8415 Acked Flush 0 Done Sent Done false
28436 8416 Acked Read 1 Done Sent Done false
28437 8417 Acked Read 3 Done Sent Done false
28438 8418 Acked Read 10 Done Sent Done false
28439 8419 Acked Write 7 Done Sent Done false
28440 8420 Acked Read 9 Done Sent Done false
28441 8421 Acked Read 3 Done Sent Done false
28442 8422 Acked Write 4 Done Sent Done false
28443 8423 Acked Read 9 Done Sent Done false
28444 8424 Acked Flush 0 Done Sent Done false
28445 8425 Acked Read 9 Done Sent Done false
28446 8426 Acked Read 1 Done Sent Done false
28447 8427 Acked Read 4 Done Sent Done false
28448 8428 Acked Write 2 Done Sent Done false
28449 8429 Acked Read 4 Done Sent Done false
28450 8430 Acked Read 9 Done Sent Done false
28451 8431 Acked Read 5 Done Sent Done false
28452 8432 Acked Write 7 Done Sent Done false
28453 8433 Acked Read 2 Done Sent Done false
28454 8434 Acked Read 1 Done Sent Done false
28455 8435 Acked Write 8 Done Sent Done false
28456 8436 Acked Write 2 Done Sent Done false
28457 8437 Acked Read 10 Done Sent Done false
28458 8438 Acked Flush 0 Done Sent Done false
28459 8439 Acked Read 3 Done Sent Done false
28460 8440 Acked Write 6 Done Sent Done false
28461 8441 Acked Read 2 Done Sent Done false
28462 8442 Acked Write 6 Done Sent Done false
28463 8443 Acked Read 2 Done Sent Done false
28464 8444 Acked Read 2 Done Sent Done false
28465 8445 Acked Write 3 Done Sent Done false
28466 8446 Acked Read 6 Done Sent Done false
28467 8447 Acked Write 1 Done Sent Done false
28468 8448 Acked Write 2 Done Sent Done false
28469 8449 Acked Write 5 Done Sent Done false
28470 8450 Acked Write 5 Done Sent Done false
28471 8451 Acked Flush 0 Done Sent Done false
28472 8452 Acked Write 4 Done Sent Done false
28473 8453 Acked Write 8 Done Sent Done false
28474 8454 Acked Read 5 Done Sent Done false
28475 8455 Acked Read 8 Done Sent Done false
28476 8456 Acked Write 2 Done Sent Done false
28477 8457 Acked Read 4 Done Sent Done false
28478 8458 Acked Write 6 Done Sent Done false
28479 8459 Acked Read 7 Done Sent Done false
28480 8460 Acked Read 7 Done Sent Done false
28481 8461 Acked Read 3 Done Sent Done false
28482 8462 Acked Read 1 Done Sent Done false
28483 8463 Acked Read 6 Done Sent Done false
28484 8464 Acked Write 7 Done Sent Done false
28485 8465 Acked Read 6 Done Sent Done false
28486 8466 Acked Write 10 Done Sent Done false
28487 8467 Acked Read 4 Done Sent Done false
28488 8468 Acked Read 6 Done Sent Done false
28489 8469 Acked Read 1 Done Sent Done false
28490 8470 Acked Write 10 Done Sent Done false
28491 8471 Acked Write 8 Done Sent Done false
28492 8472 Acked Read 10 Done Sent Done false
28493 8473 Acked Write 7 Done Sent Done false
28494 8474 Acked Write 2 Done Sent Done false
28495 8475 Acked Read 2 Done Sent Done false
28496 8476 Acked Read 3 Done Sent Done false
28497 8477 Acked Flush 0 Done Sent Done false
28498 8478 Acked Write 8 Done Sent Done false
28499 8479 Acked Read 2 Done Sent Done false
28500 8480 Acked Write 9 Done Sent Done false
28501 8481 Acked Read 2 Done Sent Done false
28502 8482 Acked Flush 0 Done Sent Done false
28503 8483 Acked Read 7 Done Sent Done false
28504 8484 Acked Read 7 Done Sent Done false
28505 8485 Acked Write 3 Done Sent Done false
28506 8486 Acked Read 7 Done Sent Done false
28507 8487 Acked Read 6 Done Sent Done false
28508 8488 Acked Write 10 Done Sent Done false
28509 8489 Acked Read 1 Done Sent Done false
28510 8490 Acked Flush 0 Done Sent Done false
28511 8491 Acked Read 3 Done Sent Done false
28512 8492 Acked Flush 0 Done Sent Done false
28513 8493 Acked Read 4 Done Sent Done false
28514 8494 Acked Read 8 Done Sent Done false
28515 8495 Acked Write 4 Done Sent Done false
28516 8496 Acked Write 2 Done Sent Done false
28517 8497 Acked Write 3 Done Sent Done false
28518 8498 Acked Read 10 Done Sent Done false
28519 8499 Acked Write 7 Done Sent Done false
28520 8500 Acked Flush 0 Done Sent Done false
28521 8501 Acked Write 3 Done Sent Done false
28522 8502 Acked Read 2 Done Sent Done false
28523 8503 Acked Read 9 Done Sent Done false
28524 8504 Acked Write 4 Done Sent Done false
28525 8505 Acked Read 9 Done Sent Done false
28526 8506 Acked Read 2 Done Sent Done false
28527 8507 Acked Read 5 Done Sent Done false
28528 8508 Acked Write 9 Done Sent Done false
28529 8509 Acked Write 4 Done Sent Done false
28530 8510 Acked Read 2 Done Sent Done false
28531 8511 Acked Write 6 Done Sent Done false
28532 8512 Acked Read 9 Done Sent Done false
28533 8513 Acked Read 10 Done Sent Done false
28534 8514 Acked Read 9 Done Sent Done false
28535 8515 Acked Write 3 Done Sent Done false
28536 8516 Acked Read 8 Done Sent Done false
28537 8517 Acked Write 8 Done Sent Done false
28538 8518 Acked Read 3 Done Sent Done false
28539 8519 Acked Write 3 Done Sent Done false
28540 8520 Acked Read 7 Done Sent Done false
28541 8521 Acked Read 10 Done Sent Done false
28542 8522 Acked Write 8 Done Sent Done false
28543 8523 Acked Read 8 Done Sent Done false
28544 8524 Acked Read 1 Done Sent Done false
28545 8525 Acked Read 2 Done Sent Done false
28546 8526 Acked Write 4 Done Sent Done false
28547 8527 Acked Write 8 Done Sent Done false
28548 8528 Acked Read 4 Done Sent Done false
28549 8529 Acked Write 6 Done Sent Done false
28550 8530 Acked Write 5 Done Sent Done false
28551 8531 Acked Flush 0 Done Sent Done false
28552 8532 Acked Read 8 Done Sent Done false
28553 8533 Acked Read 10 Done Sent Done false
28554 8534 Acked Read 2 Done Sent Done false
28555 8535 Acked Read 4 Done Sent Done false
28556 8536 Acked Flush 0 Done Sent Done false
28557 8537 Acked Read 2 Done Sent Done false
28558 8538 Acked Write 2 Done Sent Done false
28559 8539 Acked Read 6 Done Sent Done false
28560 8540 Acked Read 8 Done Sent Done false
28561 8541 Acked Write 3 Done Sent Done false
28562 8542 Acked Write 2 Done Sent Done false
28563 8543 Acked Write 4 Done Sent Done false
28564 8544 Acked Write 10 Done Sent Done false
28565 8545 Acked Read 1 Done Sent Done false
28566 8546 Acked Read 6 Done Sent Done false
28567 STATES DS:0 DS:1 DS:2 TOTAL
28568 Sent 0 301 0 301
28569 Done 301 0 301 602
28570 Skipped 0 0 0 0
28571 Error 0 0 0 0
28572 Last Flush: Some(JobId(8536)) Some(JobId(8245)) Some(JobId(8536))
28573 Downstairs last five completed: 8245 8244 8243 8242 8241
28574 Upstairs last five completed: 8546 8545 8544 8543 8542
28575 CLIENT: Up:0 ds:301 act:2
285762025-02-25T17:24:28.918ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
285772025-02-25T17:24:29.855ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
285782025-02-25T17:24:30.161ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
285792025-02-25T17:24:30.162ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:40691) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
285802025-02-25T17:24:30.163ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
285812025-02-25T17:24:30.163ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
285822025-02-25T17:24:30.163ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
285832025-02-25T17:24:30.163ZINFOcrucible: send last flush ID to this DS: Some(JobId(8245)) = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
285842025-02-25T17:24:30.163ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
285852025-02-25T17:24:30.163ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(8245)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
285862025-02-25T17:24:30.163ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(8245)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
28587 ----------------------------------------------------------------
28588 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28589 Downstairs last five completed: 8547 8546 8545 8544 8543
28590 Upstairs last five completed: 8547 8546 8545 8544 8543
28591 CLIENT: Up:0 ds:0 act:3
28592 Replay: All jobs finished
28593 Main task has work to do, go find it
28594 got dsc Stop(2)
28595 stop 2
28596 [8830] Got stop action so:false kr:false
28597 Feb 25 17:24:32.419 INFO request completed, latency_us: 384, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 016e675e-b927-49b5-a17a-bbe90b6f54a9, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
285982025-02-25T17:24:32.420ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
285992025-02-25T17:24:32.420ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
286002025-02-25T17:24:32.420ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
28601 Feb 25 17:24:32.420 INFO request completed, latency_us: 118, response_code: 200, uri: /state/cid/2, method: GET, req_id: 55544a2e-a2d6-471d-9ed4-d5ed3a88d9df, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
286022025-02-25T17:24:32.420ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
28603 [8830] Exited with: ExitStatus(unix_wait_status(9))
28604 [8830][2] reports Exit
28605 Feb 25 17:24:36.422 INFO request completed, latency_us: 279, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3df6c3ac-f071-4bf8-a9aa-8a4a92d97a38, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
28606 001/300 Write block 4060 len 28672 data: 5 8 7 6 9 8 7
28607 002/300 Write block 1717 len 40960 data: 5 6 8 7 7 6 5 5 4 3
28608 003/300 Write block 1588 len 24576 data: 4 3 4 3 5 5
28609 004/300 Read block 4852 len 28672
28610 005/300 Write block 2542 len 36864 data: 7 6 5 5 5 4 4 2 4
28611 006/300 Read block 4510 len 32768
28612 007/300 Write block 1146 len 4096 data: 5
28613 008/300 Read block 3192 len 32768
28614 009/300 Write block 3941 len 28672 data: 7 7 7 7 9 7 8
28615 010/300 Flush
28616 011/300 Write block 4140 len 12288 data: 6 5 6
28617 012/300 Read block 1976 len 28672
28618 013/300 Flush
28619 014/300 Write block 4079 len 4096 data: 6
28620 015/300 Write block 4986 len 40960 data: 3 5 6 7 6 7 6 6 5 5
28621 016/300 Read block 1 len 36864
28622 017/300 Read block 2625 len 20480
28623 018/300 Write block 822 len 12288 data: 4 5 5
28624 019/300 Flush
28625 020/300 Read block 1544 len 28672
28626 021/300 Read block 3146 len 36864
28627 022/300 Read block 2335 len 4096
28628 023/300 Write block 2438 len 40960 data: 4 6 4 3 4 4 4 5 6 5
28629 024/300 Read block 1333 len 4096
28630 025/300 Flush
28631 026/300 Read block 494 len 36864
28632 027/300 Write block 4639 len 28672 data: 4 3 4 5 6 5 4
28633 028/300 Read block 1978 len 20480
28634 029/300 Write block 1631 len 8192 data: 7 6
28635 030/300 Write block 3101 len 16384 data: 6 8 7 7
28636 031/300 Write block 4285 len 8192 data: 6 5
28637 032/300 Read block 3606 len 12288
28638 033/300 Write block 3016 len 36864 data: 6 7 5 5 5 4 4 5 4
28639 034/300 Write block 4863 len 36864 data: 3 3 4 4 4 4 4 6 6
28640 035/300 Flush
28641 036/300 Read block 4255 len 16384
28642 037/300 Flush
28643 038/300 Read block 1681 len 40960
28644 039/300 Read block 981 len 20480
28645 040/300 Write block 2717 len 16384 data: 7 6 5 6
28646 041/300 Read block 1783 len 16384
28647 042/300 Read block 1362 len 36864
28648 043/300 Write block 2152 len 40960 data: 2 3 3 3 6 6 5 5 4 4
28649 044/300 Read block 4536 len 8192
28650 045/300 Flush
28651 046/300 Read block 3799 len 20480
28652 047/300 Write block 4014 len 40960 data: 5 6 6 6 6 5 5 5 8 7
28653 048/300 Read block 4420 len 40960
28654 049/300 Write block 3617 len 4096 data: 6
28655 050/300 Read block 885 len 16384
28656 051/300 Read block 4797 len 4096
28657 052/300 Read block 4328 len 8192
28658 053/300 Flush
28659 054/300 Read block 3380 len 36864
28660 055/300 Write block 3358 len 20480 data: 6 5 7 5 5
28661 056/300 Flush
28662 057/300 Write block 4902 len 4096 data: 4
28663 058/300 Write block 4764 len 40960 data: 8 8 11 10 9 8 8 7 3 2
28664 059/300 Flush
28665 060/300 Read block 981 len 4096
28666 061/300 Write block 4583 len 40960 data: 6 5 5 5 6 8 6 7 9 8
28667 062/300 Flush
28668 063/300 Read block 2950 len 40960
28669 064/300 Flush
28670 065/300 Read block 966 len 28672
28671 066/300 Read block 4369 len 12288
28672 067/300 Read block 855 len 16384
28673 068/300 Write block 304 len 12288 data: 7 5 5
28674 069/300 Read block 4305 len 36864
28675 070/300 Write block 1834 len 28672 data: 2 3 4 5 6 6 6
28676 071/300 Read block 865 len 16384
28677 072/300 Write block 3389 len 28672 data: 7 8 8 9 9 9 9
28678 073/300 Read block 287 len 4096
28679 074/300 Read block 1094 len 28672
28680 075/300 Read block 728 len 20480
28681 076/300 Write block 1539 len 32768 data: 7 8 8 7 7 8 8 5
28682 077/300 Read block 542 len 4096
28683 078/300 Read block 4737 len 28672
28684 079/300 Read block 4362 len 40960
28685 080/300 Read block 374 len 40960
28686 081/300 Flush
28687 082/300 Read block 555 len 8192
28688 083/300 Write block 2798 len 12288 data: 9 8 4
28689 084/300 Write block 3093 len 8192 data: 7 6
28690 085/300 Read block 1276 len 24576
28691 086/300 Write block 900 len 20480 data: 5 6 7 7 7
28692 087/300 Read block 1649 len 36864
28693 088/300 Write block 3646 len 32768 data: 6 5 5 6 7 7 8 10
28694 089/300 Flush
28695 090/300 Read block 4100 len 16384
28696 091/300 Read block 3176 len 16384
28697 092/300 Write block 2689 len 4096 data: 4
28698 093/300 Read block 4549 len 16384
28699 094/300 Write block 2961 len 32768 data: 4 5 6 8 8 9 9 8
28700 095/300 Write block 1839 len 16384 data: 7 7 5 6
28701 096/300 Read block 2688 len 36864
28702 097/300 Write block 3228 len 20480 data: 4 5 5 5 5
28703 098/300 Read block 1785 len 4096
28704 099/300 Write block 4114 len 16384 data: 7 5 5 5
28705 100/300 Flush
28706 101/300 Read block 3837 len 36864
28707 102/300 Read block 324 len 36864
28708 103/300 Write block 2838 len 20480 data: 6 5 4 3 3
28709 104/300 Write block 2658 len 16384 data: 3 3 4 4
28710 105/300 Read block 1617 len 20480
28711 106/300 Read block 240 len 28672
28712 107/300 Write block 2591 len 36864 data: 6 5 5 5 5 4 3 4 5
28713 108/300 Write block 4560 len 20480 data: 5 5 5 5 4
28714 109/300 Write block 4076 len 16384 data: 4 5 6 7
28715 110/300 Read block 1730 len 36864
28716 111/300 Write block 2094 len 36864 data: 9 9 9 8 8 7 5 5 3
28717 112/300 Read block 3544 len 32768
28718 113/300 Read block 3117 len 24576
28719 114/300 Read block 2991 len 16384
28720 115/300 Read block 1712 len 4096
28721 116/300 Read block 4532 len 4096
28722 117/300 Flush
28723 118/300 Write block 234 len 20480 data: 3 3 3 4 3
28724 119/300 Read block 1872 len 16384
28725 120/300 Flush
28726 121/300 Write block 487 len 24576 data: 5 7 8 6 7 7
28727 122/300 Write block 4300 len 8192 data: 5 6
28728 123/300 Read block 116 len 8192
28729 124/300 Write block 4268 len 20480 data: 6 6 5 4 4
28730 125/300 Read block 2378 len 16384
28731 126/300 Read block 1746 len 8192
28732 127/300 Read block 28 len 4096
28733 128/300 Write block 3513 len 12288 data: 3 4 6
28734 129/300 Read block 2709 len 32768
28735 130/300 Write block 3958 len 8192 data: 4 4
28736 131/300 Write block 1203 len 24576 data: 4 4 3 3 3 2
28737 132/300 Read block 1922 len 40960
28738 133/300 Read block 4544 len 40960
28739 134/300 Write block 1149 len 40960 data: 5 5 5 5 7 6 8 7 10 8
28740 135/300 Write block 4167 len 4096 data: 7
28741 136/300 Read block 2530 len 28672
28742 137/300 Read block 3980 len 8192
28743 138/300 Read block 578 len 12288
28744 139/300 Flush
28745 140/300 Write block 3913 len 12288 data: 6 5 5
28746 141/300 Read block 4503 len 32768
28747 142/300 Write block 4392 len 8192 data: 9 8
28748 143/300 Write block 4398 len 20480 data: 4 4 5 5 5
28749 144/300 Read block 2253 len 12288
28750 145/300 Write block 3143 len 32768 data: 5 5 4 4 5 5 5 4
28751 146/300 Flush
28752 147/300 Read block 4070 len 24576
28753 148/300 Write block 1284 len 12288 data: 6 5 4
28754 149/300 Write block 2442 len 40960 data: 5 5 5 6 7 6 5 6 6 6
28755 150/300 Write block 1949 len 12288 data: 7 7 7
28756 151/300 Read block 4549 len 36864
28757 152/300 Write block 2596 len 40960 data: 5 4 5 6 4 7 5 6 5 4
28758 153/300 Flush
28759 154/300 Write block 3226 len 20480 data: 6 6 5 6 6
28760 155/300 Read block 840 len 24576
28761 156/300 Read block 3698 len 32768
28762 157/300 Write block 556 len 28672 data: 6 5 4 4 5 5 3
28763 158/300 Write block 1579 len 28672 data: 4 5 8 6 6 6 6
28764 159/300 Write block 775 len 36864 data: 3 5 5 5 6 6 6 6 5
28765 160/300 Write block 4249 len 20480 data: 7 5 4 5 5
28766 161/300 Read block 1760 len 24576
28767 162/300 Read block 583 len 20480
28768 163/300 Read block 2380 len 8192
28769 164/300 Flush
28770 165/300 Write block 753 len 8192 data: 5 5
28771 166/300 Read block 1165 len 12288
28772 167/300 Read block 10 len 12288
28773 168/300 Read block 4605 len 24576
28774 169/300 Read block 2098 len 32768
28775 170/300 Write block 3503 len 12288 data: 5 5 5
28776 171/300 Write block 1332 len 40960 data: 5 5 5 4 5 3 6 6 6 5
28777 172/300 Write block 1820 len 36864 data: 4 4 4 6 6 5 4 4 4
28778 173/300 Read block 2506 len 24576
28779 174/300 Write block 796 len 32768 data: 4 5 4 5 4 5 5 5
28780 175/300 Read block 2839 len 8192
28781 176/300 Write block 1727 len 40960 data: 3 4 4 4 3 4 4 5 5 4
28782 177/300 Read block 1518 len 4096
28783 178/300 Flush
28784 179/300 Write block 1843 len 20480 data: 5 6 4 4 4
28785 180/300 Read block 871 len 24576
28786 181/300 Read block 2572 len 32768
28787 182/300 Read block 446 len 4096
28788 183/300 Read block 517 len 24576
28789 184/300 Flush
28790 185/300 Write block 3169 len 28672 data: 7 6 6 5 4 3 3
28791 186/300 Write block 2439 len 16384 data: 7 5 4 6
28792 187/300 Read block 1609 len 12288
28793 188/300 Write block 1156 len 16384 data: 8 11 9 5
28794 189/300 Read block 4115 len 8192
28795 190/300 Read block 832 len 36864
28796 191/300 Flush
28797 192/300 Read block 3073 len 12288
28798 193/300 Write block 2859 len 24576 data: 8 6 5 5 5 5
28799 194/300 Write block 1717 len 36864 data: 6 7 9 8 8 7 6 6 5
28800 195/300 Read block 3232 len 12288
28801 196/300 Read block 2317 len 28672
28802 197/300 Write block 1552 len 12288 data: 4 5 4
28803 198/300 Read block 2825 len 12288
28804 199/300 Write block 2683 len 8192 data: 3 3
28805 200/300 Flush
28806 201/300 Write block 1136 len 32768 data: 2 2 2 3 3 3 3 4
28807 202/300 Write block 265 len 36864 data: 4 5 5 5 6 7 8 8 7
28808 203/300 Write block 1345 len 28672 data: 5 5 3 5 6 6 6
28809 204/300 Read block 4832 len 32768
28810 205/300 Write block 3418 len 20480 data: 5 5 5 3 3
28811 206/300 Write block 2947 len 16384 data: 8 8 7 7
28812 207/300 Write block 3221 len 16384 data: 10 9 8 8
28813 208/300 Flush
28814 209/300 Write block 2701 len 36864 data: 7 5 4 5 5 7 6 6 6
28815 210/300 Read block 1239 len 20480
28816 211/300 Write block 4333 len 20480 data: 5 5 5 7 8
28817 212/300 Read block 661 len 16384
28818 213/300 Read block 2288 len 32768
28819 214/300 Read block 463 len 36864
28820 215/300 Read block 1585 len 12288
28821 216/300 Write block 636 len 4096 data: 10
28822 217/300 Write block 3671 len 12288 data: 8 7 8
28823 218/300 Write block 1936 len 32768 data: 11 10 7 8 7 5 5 4
28824 219/300 Read block 2651 len 8192
28825 220/300 Read block 4168 len 20480
28826 221/300 Read block 4854 len 20480
28827 222/300 Read block 341 len 20480
28828 223/300 Flush
28829 224/300 Write block 2616 len 40960 data: 5 5 6 6 6 6 6 6 5 5
28830 225/300 Read block 4153 len 16384
28831 226/300 Read block 4258 len 32768
28832 227/300 Write block 285 len 16384 data: 3 2 3 3
28833 228/300 Write block 1227 len 24576 data: 2 3 3 3 4 6
28834 229/300 Read block 1475 len 24576
28835 230/300 Write block 117 len 12288 data: 5 4 4
28836 231/300 Read block 1478 len 28672
28837 232/300 Read block 3002 len 24576
28838 233/300 Write block 3462 len 36864 data: 5 5 4 5 4 4 3 4 3
28839 234/300 Write block 1015 len 12288 data: 5 4 3
28840 235/300 Write block 1726 len 16384 data: 4 4 5 5
28841 236/300 Read block 760 len 28672
28842 237/300 Read block 2138 len 36864
28843 238/300 Write block 4633 len 20480 data: 6 7 8 7 7
28844 239/300 Write block 4097 len 20480 data: 2 2 2 2 2
28845 240/300 Read block 2736 len 8192
28846 241/300 Read block 1823 len 28672
28847 242/300 Read block 3621 len 28672
28848 243/300 Read block 2017 len 8192
28849 244/300 Write block 20 len 40960 data: 5 7 5 6 7 6 8 9 9 7
28850 245/300 Write block 4317 len 32768 data: 2 4 3 3 3 4 3 3
28851 246/300 Read block 4687 len 20480
28852 247/300 Read block 1454 len 12288
28853 248/300 Write block 2619 len 20480 data: 7 7 7 7 7
28854 249/300 Write block 878 len 12288 data: 3 3 3
28855 250/300 Write block 4320 len 8192 data: 4 4
28856 251/300 Write block 4239 len 12288 data: 6 7 6
28857 252/300 Read block 2559 len 24576
28858 253/300 Read block 2440 len 36864
28859 254/300 Read block 4277 len 16384
28860 255/300 Read block 4982 len 20480
28861 256/300 Read block 250 len 24576
28862 257/300 Write block 1313 len 32768 data: 5 7 6 9 7 7 7 8
28863 258/300 Read block 2725 len 28672
28864 259/300 Write block 1235 len 20480 data: 8 8 9 8 6
28865 260/300 Write block 1207 len 40960 data: 4 3 3 4 3 4 4 5 7 6
28866 261/300 Write block 1718 len 36864 data: 8 10 9 9 8 7 7 6 5
28867 262/300 Write block 1220 len 36864 data: 5 4 4 4 4 3 2 3 4
28868 263/300 Read block 975 len 40960
28869 264/300 Write block 4537 len 4096 data: 10
28870 265/300 Write block 3221 len 24576 data: 11 10 9 9 8 7
28871 266/300 Read block 863 len 12288
28872 267/300 Write block 1018 len 12288 data: 3 5 6
28873 268/300 Read block 4736 len 20480
28874 269/300 Write block 1458 len 40960 data: 7 7 6 5 5 6 6 5 5 6
28875 270/300 Flush
28876 271/300 Read block 247 len 24576
28877 272/300 Read block 4914 len 4096
28878 273/300 Write block 2027 len 20480 data: 3 4 5 7 8
28879 274/300 Flush
28880 275/300 Read block 112 len 12288
28881 276/300 Read block 4058 len 36864
28882 277/300 Write block 3610 len 24576 data: 6 6 6 6 4 3
28883 278/300 Write block 2776 len 16384 data: 7 4 4 4
28884 279/300 Read block 3708 len 16384
28885 280/300 Read block 1939 len 16384
28886 281/300 Read block 3560 len 32768
28887 282/300 Flush
28888 283/300 Read block 4612 len 32768
28889 284/300 Read block 2911 len 12288
28890 285/300 Flush
28891 286/300 Write block 512 len 20480 data: 7 7 6 8 8
28892 287/300 Read block 1670 len 36864
28893 288/300 Write block 2534 len 4096 data: 5
28894 289/300 Write block 4294 len 28672 data: 3 3 3 2 2 4 6
28895 290/300 Write block 653 len 12288 data: 6 6 6
28896 291/300 Read block 271 len 20480
28897 292/300 Write block 1656 len 32768 data: 2 3 3 4 3 4 4 5
28898 293/300 Read block 3002 len 16384
28899 294/300 Read block 2384 len 28672
28900 295/300 Write block 1754 len 8192 data: 7 6
28901 296/300 Read block 1725 len 40960
28902 297/300 Write block 854 len 8192 data: 5 7
28903 298/300 Read block 1951 len 16384
28904 299/300 Write block 1940 len 24576 data: 8 6 6 5 5 6
28905 300/300 Flush
28906 301/300 Write block 628 len 36864 data: 10 11 12 11 9 9 7 7 11
28907 Main task has work to do, go find it
28908 got dsc Start(2)
28909 start 2
28910 Feb 25 17:24:36.673 INFO request completed, latency_us: 219, response_code: 204, uri: /start/cid/2, method: POST, req_id: fecc1c5f-8546-47fe-bde3-8b9d6ffaf68c, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
28911 Replay: started 2, returned:Ok(())
28912 ----------------------------------------------------------------
28913 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28914 States: Active Active Offline
28915 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28916 8548 Acked Write 7 Done Done Sent false
28917 8549 Acked Write 10 Done Done Sent false
28918 8550 Acked Write 6 Done Done Sent false
28919 8551 Acked Read 7 Done Done Sent false
28920 8552 Acked Write 9 Done Done Sent false
28921 8553 Acked Read 8 Done Done Sent false
28922 8554 Acked Write 1 Done Done Sent false
28923 8555 Acked Read 8 Done Done Sent false
28924 8556 Acked Write 7 Done Done Sent false
28925 8557 Acked Flush 0 Done Done Sent false
28926 8558 Acked Write 3 Done Done Sent false
28927 8559 Acked Read 7 Done Done Sent false
28928 8560 Acked Flush 0 Done Done Sent false
28929 8561 Acked Write 1 Done Done Sent false
28930 8562 Acked Write 10 Done Done Sent false
28931 8563 Acked Read 9 Done Done Sent false
28932 8564 Acked Read 5 Done Done Sent false
28933 8565 Acked Write 3 Done Done Sent false
28934 8566 Acked Flush 0 Done Done Sent false
28935 8567 Acked Read 7 Done Done Sent false
28936 8568 Acked Read 9 Done Done Sent false
28937 8569 Acked Read 1 Done Done Sent false
28938 8570 Acked Write 10 Done Done Sent false
28939 8571 Acked Read 1 Done Done Sent false
28940 8572 Acked Flush 0 Done Done Sent false
28941 8573 Acked Read 9 Done Done Sent false
28942 8574 Acked Write 7 Done Done Sent false
28943 8575 Acked Read 5 Done Done Sent false
28944 8576 Acked Write 2 Done Done Sent false
28945 8577 Acked Write 4 Done Done Sent false
28946 8578 Acked Write 2 Done Done Sent false
28947 8579 Acked Read 3 Done Done Sent false
28948 8580 Acked Write 9 Done Done Sent false
28949 8581 Acked Write 9 Done Done Sent false
28950 8582 Acked Flush 0 Done Done Sent false
28951 8583 Acked Read 4 Done Done Sent false
28952 8584 Acked Flush 0 Done Done Sent false
28953 8585 Acked Read 10 Done Done Sent false
28954 8586 Acked Read 5 Done Done Sent false
28955 8587 Acked Write 4 Done Done Sent false
28956 8588 Acked Read 4 Done Done Sent false
28957 8589 Acked Read 9 Done Done Sent false
28958 8590 Acked Write 10 Done Done Sent false
28959 8591 Acked Read 2 Done Done Sent false
28960 8592 Acked Flush 0 Done Done Sent false
28961 8593 Acked Read 5 Done Done Sent false
28962 8594 Acked Write 10 Done Done Sent false
28963 8595 Acked Read 10 Done Done Sent false
28964 8596 Acked Write 1 Done Done Sent false
28965 8597 Acked Read 4 Done Done Sent false
28966 8598 Acked Read 1 Done Done Sent false
28967 8599 Acked Read 2 Done Done Sent false
28968 8600 Acked Flush 0 Done Done Sent false
28969 8601 Acked Read 9 Done Done Sent false
28970 8602 Acked Write 5 Done Done Sent false
28971 8603 Acked Flush 0 Done Done Sent false
28972 8604 Acked Write 1 Done Done Sent false
28973 8605 Acked Write 10 Done Done Sent false
28974 8606 Acked Flush 0 Done Done Sent false
28975 8607 Acked Read 1 Done Done Sent false
28976 8608 Acked Write 10 Done Done Sent false
28977 8609 Acked Flush 0 Done Done Sent false
28978 8610 Acked Read 10 Done Done Sent false
28979 8611 Acked Flush 0 Done Done Sent false
28980 8612 Acked Read 7 Done Done Sent false
28981 8613 Acked Read 3 Done Done Sent false
28982 8614 Acked Read 4 Done Done Sent false
28983 8615 Acked Write 3 Done Done Sent false
28984 8616 Acked Read 9 Done Done Sent false
28985 8617 Acked Write 7 Done Done Sent false
28986 8618 Acked Read 4 Done Done Sent false
28987 8619 Acked Write 7 Done Done Sent false
28988 8620 Acked Read 1 Done Done Sent false
28989 8621 Acked Read 7 Done Done Sent false
28990 8622 Acked Read 5 Done Done Sent false
28991 8623 Acked Write 8 Done Done Sent false
28992 8624 Acked Read 1 Done Done Sent false
28993 8625 Acked Read 7 Done Done Sent false
28994 8626 Acked Read 10 Done Done Sent false
28995 8627 Acked Read 10 Done Done Sent false
28996 8628 Acked Flush 0 Done Done Sent false
28997 8629 Acked Read 2 Done Done Sent false
28998 8630 Acked Write 3 Done Done Sent false
28999 8631 Acked Write 2 Done Done Sent false
29000 8632 Acked Read 6 Done Done Sent false
29001 8633 Acked Write 5 Done Done Sent false
29002 8634 Acked Read 9 Done Done Sent false
29003 8635 Acked Write 8 Done Done Sent false
29004 8636 Acked Flush 0 Done Done Sent false
29005 8637 Acked Read 4 Done Done Sent false
29006 8638 Acked Read 4 Done Done Sent false
29007 8639 Acked Write 1 Done Done Sent false
29008 8640 Acked Read 4 Done Done Sent false
29009 8641 Acked Write 8 Done Done Sent false
29010 8642 Acked Write 4 Done Done Sent false
29011 8643 Acked Read 9 Done Done Sent false
29012 8644 Acked Write 5 Done Done Sent false
29013 8645 Acked Read 1 Done Done Sent false
29014 8646 Acked Write 4 Done Done Sent false
29015 8647 Acked Flush 0 Done Done Sent false
29016 8648 Acked Read 9 Done Done Sent false
29017 8649 Acked Read 9 Done Done Sent false
29018 8650 Acked Write 5 Done Done Sent false
29019 8651 Acked Write 4 Done Done Sent false
29020 8652 Acked Read 5 Done Done Sent false
29021 8653 Acked Read 7 Done Done Sent false
29022 8654 Acked Write 9 Done Done Sent false
29023 8655 Acked Write 5 Done Done Sent false
29024 8656 Acked Write 4 Done Done Sent false
29025 8657 Acked Read 9 Done Done Sent false
29026 8658 Acked Write 9 Done Done Sent false
29027 8659 Acked Read 8 Done Done Sent false
29028 8660 Acked Read 6 Done Done Sent false
29029 8661 Acked Read 4 Done Done Sent false
29030 8662 Acked Read 1 Done Done Sent false
29031 8663 Acked Read 1 Done Done Sent false
29032 8664 Acked Flush 0 Done Done Sent false
29033 8665 Acked Write 5 Done Done Sent false
29034 8666 Acked Read 4 Done Done Sent false
29035 8667 Acked Flush 0 Done Done Sent false
29036 8668 Acked Write 6 Done Done Sent false
29037 8669 Acked Write 2 Done Done Sent false
29038 8670 Acked Read 2 Done Done Sent false
29039 8671 Acked Write 5 Done Done Sent false
29040 8672 Acked Read 4 Done Done Sent false
29041 8673 Acked Read 2 Done Done Sent false
29042 8674 Acked Read 1 Done Done Sent false
29043 8675 Acked Write 3 Done Done Sent false
29044 8676 Acked Read 8 Done Done Sent false
29045 8677 Acked Write 2 Done Done Sent false
29046 8678 Acked Write 6 Done Done Sent false
29047 8679 Acked Read 10 Done Done Sent false
29048 8680 Acked Read 10 Done Done Sent false
29049 8681 Acked Write 10 Done Done Sent false
29050 8682 Acked Write 1 Done Done Sent false
29051 8683 Acked Read 7 Done Done Sent false
29052 8684 Acked Read 2 Done Done Sent false
29053 8685 Acked Read 3 Done Done Sent false
29054 8686 Acked Flush 0 Done Done Sent false
29055 8687 Acked Write 3 Done Done Sent false
29056 8688 Acked Read 8 Done Done Sent false
29057 8689 Acked Write 2 Done Done Sent false
29058 8690 Acked Write 5 Done Done Sent false
29059 8691 Acked Read 3 Done Done Sent false
29060 8692 Acked Write 8 Done Done Sent false
29061 8693 Acked Flush 0 Done Done Sent false
29062 8694 Acked Read 6 Done Done Sent false
29063 8695 Acked Write 3 Done Done Sent false
29064 8696 Acked Write 10 Done Done Sent false
29065 8697 Acked Write 3 Done Done Sent false
29066 8698 Acked Read 9 Done Done Sent false
29067 8699 Acked Write 10 Done Done Sent false
29068 8700 Acked Flush 0 Done Done Sent false
29069 8701 Acked Write 5 Done Done Sent false
29070 8702 Acked Read 6 Done Done Sent false
29071 8703 Acked Read 8 Done Done Sent false
29072 8704 Acked Write 7 Done Done Sent false
29073 8705 Acked Write 7 Done Done Sent false
29074 8706 Acked Write 9 Done Done Sent false
29075 8707 Acked Write 5 Done Done Sent false
29076 8708 Acked Read 6 Done Done Sent false
29077 8709 Acked Read 5 Done Done Sent false
29078 8710 Acked Read 2 Done Done Sent false
29079 8711 Acked Flush 0 Done Done Sent false
29080 8712 Acked Write 2 Done Done Sent false
29081 8713 Acked Read 3 Done Done Sent false
29082 8714 Acked Read 3 Done Done Sent false
29083 8715 Acked Read 6 Done Done Sent false
29084 8716 Acked Read 8 Done Done Sent false
29085 8717 Acked Write 3 Done Done Sent false
29086 8718 Acked Write 10 Done Done Sent false
29087 8719 Acked Write 9 Done Done Sent false
29088 8720 Acked Read 6 Done Done Sent false
29089 8721 Acked Write 8 Done Done Sent false
29090 8722 Acked Read 2 Done Done Sent false
29091 8723 Acked Write 10 Done Done Sent false
29092 8724 Acked Read 1 Done Done Sent false
29093 8725 Acked Flush 0 Done Done Sent false
29094 8726 Acked Write 5 Done Done Sent false
29095 8727 Acked Read 6 Done Done Sent false
29096 8728 Acked Read 8 Done Done Sent false
29097 8729 Acked Read 1 Done Done Sent false
29098 8730 Acked Read 6 Done Done Sent false
29099 8731 Acked Flush 0 Done Done Sent false
29100 8732 Acked Write 7 Done Done Sent false
29101 8733 Acked Write 4 Done Done Sent false
29102 8734 Acked Read 3 Done Done Sent false
29103 8735 Acked Write 4 Done Done Sent false
29104 8736 Acked Read 2 Done Done Sent false
29105 8737 Acked Read 9 Done Done Sent false
29106 8738 Acked Flush 0 Done Done Sent false
29107 8739 Acked Read 3 Done Done Sent false
29108 8740 Acked Write 6 Done Done Sent false
29109 8741 Acked Write 9 Done Done Sent false
29110 8742 Acked Read 3 Done Done Sent false
29111 8743 Acked Read 7 Done Done Sent false
29112 8744 Acked Write 3 Done Done Sent false
29113 8745 Acked Read 3 Done Done Sent false
29114 8746 Acked Write 2 Done Done Sent false
29115 8747 Acked Flush 0 Done Done Sent false
29116 8748 Acked Write 8 Done Done Sent false
29117 8749 Acked Write 9 Done Done Sent false
29118 8750 Acked Write 7 Done Done Sent false
29119 8751 Acked Read 8 Done Done Sent false
29120 8752 Acked Write 5 Done Done Sent false
29121 8753 Acked Write 4 Done Done Sent false
29122 8754 Acked Write 4 Done Done Sent false
29123 8755 Acked Flush 0 Done Done Sent false
29124 8756 Acked Write 9 Done Done Sent false
29125 8757 Acked Read 5 Done Done Sent false
29126 8758 Acked Write 5 Done Done Sent false
29127 8759 Acked Read 4 Done Done Sent false
29128 8760 Acked Read 8 Done Done Sent false
29129 8761 Acked Read 9 Done Done Sent false
29130 8762 Acked Read 3 Done Done Sent false
29131 8763 Acked Write 1 Done Done Sent false
29132 8764 Acked Write 3 Done Done Sent false
29133 8765 Acked Write 8 Done Done Sent false
29134 8766 Acked Read 2 Done Done Sent false
29135 8767 Acked Read 5 Done Done Sent false
29136 8768 Acked Read 5 Done Done Sent false
29137 8769 Acked Read 5 Done Done Sent false
29138 8770 Acked Flush 0 Done Done Sent false
29139 8771 Acked Write 10 Done Done Sent false
29140 8772 Acked Read 4 Done Done Sent false
29141 8773 Acked Read 8 Done Done Sent false
29142 8774 Acked Write 4 Done Done Sent false
29143 8775 Acked Write 6 Done Done Sent false
29144 8776 Acked Read 6 Done Done Sent false
29145 8777 Acked Write 3 Done Done Sent false
29146 8778 Acked Read 7 Done Done Sent false
29147 8779 Acked Read 6 Done Done Sent false
29148 8780 Acked Write 9 Done Done Sent false
29149 8781 Acked Write 3 Done Done Sent false
29150 8782 Acked Write 4 Done Done Sent false
29151 8783 Acked Read 7 Done Done Sent false
29152 8784 Acked Read 9 Done Done Sent false
29153 8785 Acked Write 5 Done Done Sent false
29154 8786 Acked Write 5 Done Done Sent false
29155 8787 Acked Read 2 Done Done Sent false
29156 8788 Acked Read 7 Done Done Sent false
29157 8789 Acked Read 7 Done Done Sent false
29158 8790 Acked Read 2 Done Done Sent false
29159 8791 Acked Write 10 Done Done Sent false
29160 8792 Acked Write 8 Done Done Sent false
29161 8793 Acked Read 5 Done Done Sent false
29162 8794 Acked Read 3 Done Done Sent false
29163 8795 Acked Write 5 Done Done Sent false
29164 8796 Acked Write 3 Done Done Sent false
29165 8797 Acked Write 2 Done Done Sent false
29166 8798 Acked Write 3 Done Done Sent false
29167 8799 Acked Read 6 Done Done Sent false
29168 8800 Acked Read 9 Done Done Sent false
29169 8801 Acked Read 4 Done Done Sent false
29170 8802 Acked Read 5 Done Done Sent false
29171 8803 Acked Read 6 Done Done Sent false
29172 8804 Acked Write 8 Done Done Sent false
29173 8805 Acked Read 7 Done Done Sent false
29174 8806 Acked Write 5 Done Done Sent false
29175 8807 Acked Write 10 Done Done Sent false
29176 8808 Acked Write 9 Done Done Sent false
29177 8809 Acked Write 9 Done Done Sent false
29178 8810 Acked Read 10 Done Done Sent false
29179 8811 Acked Write 1 Done Done Sent false
29180 8812 Acked Write 6 Done Done Sent false
29181 8813 Acked Read 3 Done Done Sent false
29182 8814 Acked Write 3 Done Done Sent false
29183 8815 Acked Read 5 Done Done Sent false
29184 8816 Acked Write 10 Done Done Sent false
29185 8817 Acked Flush 0 Done Done Sent false
29186 8818 Acked Read 6 Done Done Sent false
29187 8819 Acked Read 1 Done Done Sent false
29188 8820 Acked Write 5 Done Done Sent false
29189 8821 Acked Flush 0 Done Done Sent false
29190 8822 Acked Read 3 Done Done Sent false
29191 8823 Acked Read 9 Done Done Sent false
29192 8824 Acked Write 6 Done Done Sent false
29193 8825 Acked Write 4 Done Done Sent false
29194 8826 Acked Read 4 Done Done Sent false
29195 8827 Acked Read 4 Done Done Sent false
29196 8828 Acked Read 8 Done Done Sent false
29197 8829 Acked Flush 0 Done Done Sent false
29198 8830 Acked Read 8 Done Done Sent false
29199 8831 Acked Read 3 Done Done Sent false
29200 8832 Acked Flush 0 Done Done Sent false
29201 8833 Acked Write 5 Done Done Sent false
29202 8834 Acked Read 9 Done Done Sent false
29203 8835 Acked Write 1 Done Done Sent false
29204 8836 Acked Write 7 Done Done Sent false
29205 8837 Acked Write 3 Done Done Sent false
29206 8838 Acked Read 5 Done Done Sent false
29207 8839 Acked Write 8 Done Done Sent false
29208 8840 Acked Read 4 Done Done Sent false
29209 8841 Acked Read 7 Done Done Sent false
29210 8842 Acked Write 2 Done Done Sent false
29211 8843 Acked Read 10 Done Done Sent false
29212 8844 Acked Write 2 Done Done Sent false
29213 8845 Acked Read 4 Done Done Sent false
29214 8846 Acked Write 6 Done Done Sent false
29215 8847 Acked Flush 0 Done Done Sent false
29216 8848 Acked Write 9 Done Done Sent false
29217 STATES DS:0 DS:1 DS:2 TOTAL
29218 Sent 0 0 301 301
29219 Done 301 301 0 602
29220 Skipped 0 0 0 0
29221 Error 0 0 0 0
29222 Last Flush: Some(JobId(8847)) Some(JobId(8847)) Some(JobId(8547))
29223 Downstairs last five completed: 8547 8546 8545 8544 8543
29224 Upstairs last five completed: 8848 8847 8846 8845 8844
29225 CLIENT: Up:0 ds:301 act:2
292262025-02-25T17:24:37.175ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
292272025-02-25T17:24:37.866ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
292282025-02-25T17:24:38.868ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
292292025-02-25T17:24:39.868ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
29230 [8830] got start action
29231 [8830] I am going to restart
29232 Starting downstairs at port 8830
29233 Make output file at "/tmp/dsc/downstairs-8830.txt"
29234 [8830][2] reports Starting
29235 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1234)
29236 [8830][2] reports Running
29237 ----------------------------------------------------------------
29238 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29239 States: Active Active Offline
29240 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29241 8548 Acked Write 7 Done Done Sent false
29242 8549 Acked Write 10 Done Done Sent false
29243 8550 Acked Write 6 Done Done Sent false
29244 8551 Acked Read 7 Done Done Sent false
29245 8552 Acked Write 9 Done Done Sent false
29246 8553 Acked Read 8 Done Done Sent false
29247 8554 Acked Write 1 Done Done Sent false
29248 8555 Acked Read 8 Done Done Sent false
29249 8556 Acked Write 7 Done Done Sent false
29250 8557 Acked Flush 0 Done Done Sent false
29251 8558 Acked Write 3 Done Done Sent false
29252 8559 Acked Read 7 Done Done Sent false
29253 8560 Acked Flush 0 Done Done Sent false
29254 8561 Acked Write 1 Done Done Sent false
29255 8562 Acked Write 10 Done Done Sent false
29256 8563 Acked Read 9 Done Done Sent false
29257 8564 Acked Read 5 Done Done Sent false
29258 8565 Acked Write 3 Done Done Sent false
29259 8566 Acked Flush 0 Done Done Sent false
29260 8567 Acked Read 7 Done Done Sent false
29261 8568 Acked Read 9 Done Done Sent false
29262 8569 Acked Read 1 Done Done Sent false
29263 8570 Acked Write 10 Done Done Sent false
29264 8571 Acked Read 1 Done Done Sent false
29265 8572 Acked Flush 0 Done Done Sent false
29266 8573 Acked Read 9 Done Done Sent false
29267 8574 Acked Write 7 Done Done Sent false
29268 8575 Acked Read 5 Done Done Sent false
29269 8576 Acked Write 2 Done Done Sent false
29270 8577 Acked Write 4 Done Done Sent false
29271 8578 Acked Write 2 Done Done Sent false
29272 8579 Acked Read 3 Done Done Sent false
29273 8580 Acked Write 9 Done Done Sent false
29274 8581 Acked Write 9 Done Done Sent false
29275 8582 Acked Flush 0 Done Done Sent false
29276 8583 Acked Read 4 Done Done Sent false
29277 8584 Acked Flush 0 Done Done Sent false
29278 8585 Acked Read 10 Done Done Sent false
29279 8586 Acked Read 5 Done Done Sent false
29280 8587 Acked Write 4 Done Done Sent false
29281 8588 Acked Read 4 Done Done Sent false
29282 8589 Acked Read 9 Done Done Sent false
29283 8590 Acked Write 10 Done Done Sent false
29284 8591 Acked Read 2 Done Done Sent false
29285 8592 Acked Flush 0 Done Done Sent false
29286 8593 Acked Read 5 Done Done Sent false
29287 8594 Acked Write 10 Done Done Sent false
29288 8595 Acked Read 10 Done Done Sent false
29289 8596 Acked Write 1 Done Done Sent false
29290 8597 Acked Read 4 Done Done Sent false
29291 8598 Acked Read 1 Done Done Sent false
29292 8599 Acked Read 2 Done Done Sent false
29293 8600 Acked Flush 0 Done Done Sent false
29294 8601 Acked Read 9 Done Done Sent false
29295 8602 Acked Write 5 Done Done Sent false
29296 8603 Acked Flush 0 Done Done Sent false
29297 8604 Acked Write 1 Done Done Sent false
29298 8605 Acked Write 10 Done Done Sent false
29299 8606 Acked Flush 0 Done Done Sent false
29300 8607 Acked Read 1 Done Done Sent false
29301 8608 Acked Write 10 Done Done Sent false
29302 8609 Acked Flush 0 Done Done Sent false
29303 8610 Acked Read 10 Done Done Sent false
29304 8611 Acked Flush 0 Done Done Sent false
29305 8612 Acked Read 7 Done Done Sent false
29306 8613 Acked Read 3 Done Done Sent false
29307 8614 Acked Read 4 Done Done Sent false
29308 8615 Acked Write 3 Done Done Sent false
29309 8616 Acked Read 9 Done Done Sent false
29310 8617 Acked Write 7 Done Done Sent false
29311 8618 Acked Read 4 Done Done Sent false
29312 8619 Acked Write 7 Done Done Sent false
29313 8620 Acked Read 1 Done Done Sent false
29314 8621 Acked Read 7 Done Done Sent false
29315 8622 Acked Read 5 Done Done Sent false
29316 8623 Acked Write 8 Done Done Sent false
29317 8624 Acked Read 1 Done Done Sent false
29318 8625 Acked Read 7 Done Done Sent false
29319 8626 Acked Read 10 Done Done Sent false
29320 8627 Acked Read 10 Done Done Sent false
29321 8628 Acked Flush 0 Done Done Sent false
29322 8629 Acked Read 2 Done Done Sent false
29323 8630 Acked Write 3 Done Done Sent false
29324 8631 Acked Write 2 Done Done Sent false
29325 8632 Acked Read 6 Done Done Sent false
29326 8633 Acked Write 5 Done Done Sent false
29327 8634 Acked Read 9 Done Done Sent false
29328 8635 Acked Write 8 Done Done Sent false
29329 8636 Acked Flush 0 Done Done Sent false
29330 8637 Acked Read 4 Done Done Sent false
29331 8638 Acked Read 4 Done Done Sent false
29332 8639 Acked Write 1 Done Done Sent false
29333 8640 Acked Read 4 Done Done Sent false
29334 8641 Acked Write 8 Done Done Sent false
29335 8642 Acked Write 4 Done Done Sent false
29336 8643 Acked Read 9 Done Done Sent false
29337 8644 Acked Write 5 Done Done Sent false
29338 8645 Acked Read 1 Done Done Sent false
29339 8646 Acked Write 4 Done Done Sent false
29340 8647 Acked Flush 0 Done Done Sent false
29341 8648 Acked Read 9 Done Done Sent false
29342 8649 Acked Read 9 Done Done Sent false
29343 8650 Acked Write 5 Done Done Sent false
29344 8651 Acked Write 4 Done Done Sent false
29345 8652 Acked Read 5 Done Done Sent false
29346 8653 Acked Read 7 Done Done Sent false
29347 8654 Acked Write 9 Done Done Sent false
29348 8655 Acked Write 5 Done Done Sent false
29349 8656 Acked Write 4 Done Done Sent false
29350 8657 Acked Read 9 Done Done Sent false
29351 8658 Acked Write 9 Done Done Sent false
29352 8659 Acked Read 8 Done Done Sent false
29353 8660 Acked Read 6 Done Done Sent false
29354 8661 Acked Read 4 Done Done Sent false
29355 8662 Acked Read 1 Done Done Sent false
29356 8663 Acked Read 1 Done Done Sent false
29357 8664 Acked Flush 0 Done Done Sent false
29358 8665 Acked Write 5 Done Done Sent false
29359 8666 Acked Read 4 Done Done Sent false
29360 8667 Acked Flush 0 Done Done Sent false
29361 8668 Acked Write 6 Done Done Sent false
29362 8669 Acked Write 2 Done Done Sent false
29363 8670 Acked Read 2 Done Done Sent false
29364 8671 Acked Write 5 Done Done Sent false
29365 8672 Acked Read 4 Done Done Sent false
29366 8673 Acked Read 2 Done Done Sent false
29367 8674 Acked Read 1 Done Done Sent false
29368 8675 Acked Write 3 Done Done Sent false
29369 8676 Acked Read 8 Done Done Sent false
29370 8677 Acked Write 2 Done Done Sent false
29371 8678 Acked Write 6 Done Done Sent false
29372 8679 Acked Read 10 Done Done Sent false
29373 8680 Acked Read 10 Done Done Sent false
29374 8681 Acked Write 10 Done Done Sent false
29375 8682 Acked Write 1 Done Done Sent false
29376 8683 Acked Read 7 Done Done Sent false
29377 8684 Acked Read 2 Done Done Sent false
29378 8685 Acked Read 3 Done Done Sent false
29379 8686 Acked Flush 0 Done Done Sent false
29380 8687 Acked Write 3 Done Done Sent false
29381 8688 Acked Read 8 Done Done Sent false
29382 8689 Acked Write 2 Done Done Sent false
29383 8690 Acked Write 5 Done Done Sent false
29384 8691 Acked Read 3 Done Done Sent false
29385 8692 Acked Write 8 Done Done Sent false
29386 8693 Acked Flush 0 Done Done Sent false
29387 8694 Acked Read 6 Done Done Sent false
29388 8695 Acked Write 3 Done Done Sent false
29389 8696 Acked Write 10 Done Done Sent false
29390 8697 Acked Write 3 Done Done Sent false
29391 8698 Acked Read 9 Done Done Sent false
29392 8699 Acked Write 10 Done Done Sent false
29393 8700 Acked Flush 0 Done Done Sent false
29394 8701 Acked Write 5 Done Done Sent false
29395 8702 Acked Read 6 Done Done Sent false
29396 8703 Acked Read 8 Done Done Sent false
29397 8704 Acked Write 7 Done Done Sent false
29398 8705 Acked Write 7 Done Done Sent false
29399 8706 Acked Write 9 Done Done Sent false
29400 8707 Acked Write 5 Done Done Sent false
29401 8708 Acked Read 6 Done Done Sent false
29402 8709 Acked Read 5 Done Done Sent false
29403 8710 Acked Read 2 Done Done Sent false
29404 8711 Acked Flush 0 Done Done Sent false
29405 8712 Acked Write 2 Done Done Sent false
29406 8713 Acked Read 3 Done Done Sent false
29407 8714 Acked Read 3 Done Done Sent false
29408 8715 Acked Read 6 Done Done Sent false
29409 8716 Acked Read 8 Done Done Sent false
29410 8717 Acked Write 3 Done Done Sent false
29411 8718 Acked Write 10 Done Done Sent false
29412 8719 Acked Write 9 Done Done Sent false
29413 8720 Acked Read 6 Done Done Sent false
29414 8721 Acked Write 8 Done Done Sent false
29415 8722 Acked Read 2 Done Done Sent false
29416 8723 Acked Write 10 Done Done Sent false
29417 8724 Acked Read 1 Done Done Sent false
29418 8725 Acked Flush 0 Done Done Sent false
29419 8726 Acked Write 5 Done Done Sent false
29420 8727 Acked Read 6 Done Done Sent false
29421 8728 Acked Read 8 Done Done Sent false
29422 8729 Acked Read 1 Done Done Sent false
29423 8730 Acked Read 6 Done Done Sent false
29424 8731 Acked Flush 0 Done Done Sent false
29425 8732 Acked Write 7 Done Done Sent false
29426 8733 Acked Write 4 Done Done Sent false
29427 8734 Acked Read 3 Done Done Sent false
29428 8735 Acked Write 4 Done Done Sent false
29429 8736 Acked Read 2 Done Done Sent false
29430 8737 Acked Read 9 Done Done Sent false
29431 8738 Acked Flush 0 Done Done Sent false
29432 8739 Acked Read 3 Done Done Sent false
29433 8740 Acked Write 6 Done Done Sent false
29434 8741 Acked Write 9 Done Done Sent false
29435 8742 Acked Read 3 Done Done Sent false
29436 8743 Acked Read 7 Done Done Sent false
29437 8744 Acked Write 3 Done Done Sent false
29438 8745 Acked Read 3 Done Done Sent false
29439 8746 Acked Write 2 Done Done Sent false
29440 8747 Acked Flush 0 Done Done Sent false
29441 8748 Acked Write 8 Done Done Sent false
29442 8749 Acked Write 9 Done Done Sent false
29443 8750 Acked Write 7 Done Done Sent false
29444 8751 Acked Read 8 Done Done Sent false
29445 8752 Acked Write 5 Done Done Sent false
29446 8753 Acked Write 4 Done Done Sent false
29447 8754 Acked Write 4 Done Done Sent false
29448 8755 Acked Flush 0 Done Done Sent false
29449 8756 Acked Write 9 Done Done Sent false
29450 8757 Acked Read 5 Done Done Sent false
29451 8758 Acked Write 5 Done Done Sent false
29452 8759 Acked Read 4 Done Done Sent false
29453 8760 Acked Read 8 Done Done Sent false
29454 8761 Acked Read 9 Done Done Sent false
29455 8762 Acked Read 3 Done Done Sent false
29456 8763 Acked Write 1 Done Done Sent false
29457 8764 Acked Write 3 Done Done Sent false
29458 8765 Acked Write 8 Done Done Sent false
29459 8766 Acked Read 2 Done Done Sent false
29460 8767 Acked Read 5 Done Done Sent false
29461 8768 Acked Read 5 Done Done Sent false
29462 8769 Acked Read 5 Done Done Sent false
29463 8770 Acked Flush 0 Done Done Sent false
29464 8771 Acked Write 10 Done Done Sent false
29465 8772 Acked Read 4 Done Done Sent false
29466 8773 Acked Read 8 Done Done Sent false
29467 8774 Acked Write 4 Done Done Sent false
29468 8775 Acked Write 6 Done Done Sent false
29469 8776 Acked Read 6 Done Done Sent false
29470 8777 Acked Write 3 Done Done Sent false
29471 8778 Acked Read 7 Done Done Sent false
29472 8779 Acked Read 6 Done Done Sent false
29473 8780 Acked Write 9 Done Done Sent false
29474 8781 Acked Write 3 Done Done Sent false
29475 8782 Acked Write 4 Done Done Sent false
29476 8783 Acked Read 7 Done Done Sent false
29477 8784 Acked Read 9 Done Done Sent false
29478 8785 Acked Write 5 Done Done Sent false
29479 8786 Acked Write 5 Done Done Sent false
29480 8787 Acked Read 2 Done Done Sent false
29481 8788 Acked Read 7 Done Done Sent false
29482 8789 Acked Read 7 Done Done Sent false
29483 8790 Acked Read 2 Done Done Sent false
29484 8791 Acked Write 10 Done Done Sent false
29485 8792 Acked Write 8 Done Done Sent false
29486 8793 Acked Read 5 Done Done Sent false
29487 8794 Acked Read 3 Done Done Sent false
29488 8795 Acked Write 5 Done Done Sent false
29489 8796 Acked Write 3 Done Done Sent false
29490 8797 Acked Write 2 Done Done Sent false
29491 8798 Acked Write 3 Done Done Sent false
29492 8799 Acked Read 6 Done Done Sent false
29493 8800 Acked Read 9 Done Done Sent false
29494 8801 Acked Read 4 Done Done Sent false
29495 8802 Acked Read 5 Done Done Sent false
29496 8803 Acked Read 6 Done Done Sent false
29497 8804 Acked Write 8 Done Done Sent false
29498 8805 Acked Read 7 Done Done Sent false
29499 8806 Acked Write 5 Done Done Sent false
29500 8807 Acked Write 10 Done Done Sent false
29501 8808 Acked Write 9 Done Done Sent false
29502 8809 Acked Write 9 Done Done Sent false
29503 8810 Acked Read 10 Done Done Sent false
29504 8811 Acked Write 1 Done Done Sent false
29505 8812 Acked Write 6 Done Done Sent false
29506 8813 Acked Read 3 Done Done Sent false
29507 8814 Acked Write 3 Done Done Sent false
29508 8815 Acked Read 5 Done Done Sent false
29509 8816 Acked Write 10 Done Done Sent false
29510 8817 Acked Flush 0 Done Done Sent false
29511 8818 Acked Read 6 Done Done Sent false
29512 8819 Acked Read 1 Done Done Sent false
29513 8820 Acked Write 5 Done Done Sent false
29514 8821 Acked Flush 0 Done Done Sent false
29515 8822 Acked Read 3 Done Done Sent false
29516 8823 Acked Read 9 Done Done Sent false
29517 8824 Acked Write 6 Done Done Sent false
29518 8825 Acked Write 4 Done Done Sent false
29519 8826 Acked Read 4 Done Done Sent false
29520 8827 Acked Read 4 Done Done Sent false
29521 8828 Acked Read 8 Done Done Sent false
29522 8829 Acked Flush 0 Done Done Sent false
29523 8830 Acked Read 8 Done Done Sent false
29524 8831 Acked Read 3 Done Done Sent false
29525 8832 Acked Flush 0 Done Done Sent false
29526 8833 Acked Write 5 Done Done Sent false
29527 8834 Acked Read 9 Done Done Sent false
29528 8835 Acked Write 1 Done Done Sent false
29529 8836 Acked Write 7 Done Done Sent false
29530 8837 Acked Write 3 Done Done Sent false
29531 8838 Acked Read 5 Done Done Sent false
29532 8839 Acked Write 8 Done Done Sent false
29533 8840 Acked Read 4 Done Done Sent false
29534 8841 Acked Read 7 Done Done Sent false
29535 8842 Acked Write 2 Done Done Sent false
29536 8843 Acked Read 10 Done Done Sent false
29537 8844 Acked Write 2 Done Done Sent false
29538 8845 Acked Read 4 Done Done Sent false
29539 8846 Acked Write 6 Done Done Sent false
29540 8847 Acked Flush 0 Done Done Sent false
29541 8848 Acked Write 9 Done Done Sent false
29542 STATES DS:0 DS:1 DS:2 TOTAL
29543 Sent 0 0 301 301
29544 Done 301 301 0 602
29545 Skipped 0 0 0 0
29546 Error 0 0 0 0
29547 Last Flush: Some(JobId(8847)) Some(JobId(8847)) Some(JobId(8547))
29548 Downstairs last five completed: 8547 8546 8545 8544 8543
29549 Upstairs last five completed: 8848 8847 8846 8845 8844
29550 CLIENT: Up:0 ds:301 act:2
295512025-02-25T17:24:41.177ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295522025-02-25T17:24:41.871ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295532025-02-25T17:24:42.372ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295542025-02-25T17:24:42.421ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295552025-02-25T17:24:42.421ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54552) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295562025-02-25T17:24:42.423ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295572025-02-25T17:24:42.423ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295582025-02-25T17:24:42.423ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295592025-02-25T17:24:42.423ZINFOcrucible: send last flush ID to this DS: Some(JobId(8547)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295602025-02-25T17:24:42.423ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295612025-02-25T17:24:42.423ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(8547)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295622025-02-25T17:24:42.424ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(8547)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
29563 ----------------------------------------------------------------
29564 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29565 Downstairs last five completed: 8849 8848 8847 8846 8845
29566 Upstairs last five completed: 8849 8848 8847 8846 8845
29567 CLIENT: Up:0 ds:0 act:3
29568 Replay: All jobs finished
29569 Main task has work to do, go find it
29570 got dsc Stop(0)
29571 stop 0
29572 Feb 25 17:24:44.678[8810] Got stop action so:false kr:false
29573 INFO request completed, latency_us: 375, response_code: 204, uri: /stop/cid/0, method: POST, req_id: c11f1980-062f-403a-9f63-e4f9ea8407ef, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
29574 Feb 25 17:24:44.679 INFO request completed, latency_us: 139, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1ffbfc01-4740-4938-9619-398e30e0b0b3, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
295752025-02-25T17:24:44.679ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295762025-02-25T17:24:44.679ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295772025-02-25T17:24:44.679ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
295782025-02-25T17:24:44.679ZINFOcrucible: client task is exiting = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
29579 [8810] Exited with: ExitStatus(unix_wait_status(9))
29580 [8810][0] reports Exit
29581 Feb 25 17:24:48.681 INFO request completed, latency_us: 228, response_code: 200, uri: /state/cid/0, method: GET, req_id: df010e39-abb2-443b-8a12-663cc33be9f0, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
29582 001/300 Read block 1902 len 8192
29583 002/300 Read block 365 len 24576
29584 003/300 Read block 4976 len 40960
29585 004/300 Read block 813 len 12288
29586 005/300 Read block 456 len 8192
29587 006/300 Write block 681 len 16384 data: 5 5 6 6
29588 007/300 Write block 4452 len 8192 data: 6 6
29589 008/300 Read block 251 len 36864
29590 009/300 Flush
29591 010/300 Read block 2748 len 28672
29592 011/300 Write block 2631 len 16384 data: 4 5 7 6
29593 012/300 Flush
29594 013/300 Read block 246 len 32768
29595 014/300 Read block 843 len 12288
29596 015/300 Write block 1378 len 24576 data: 5 5 6 5 7 7
29597 016/300 Write block 2856 len 40960 data: 4 6 7 9 7 6 6 6 6 5
29598 017/300 Write block 1049 len 32768 data: 6 7 7 6 8 7 7 7
29599 018/300 Read block 2482 len 12288
29600 019/300 Write block 2770 len 12288 data: 6 5 6
29601 020/300 Flush
29602 021/300 Flush
29603 022/300 Read block 3664 len 32768
29604 023/300 Write block 2748 len 40960 data: 5 6 6 7 7 5 6 4 3 3
29605 024/300 Write block 1078 len 32768 data: 4 6 5 5 5 5 5 5
29606 025/300 Read block 4657 len 24576
29607 026/300 Read block 1901 len 32768
29608 027/300 Flush
29609 028/300 Read block 272 len 28672
29610 029/300 Read block 4749 len 24576
29611 030/300 Read block 1649 len 16384
29612 031/300 Read block 4541 len 12288
29613 032/300 Flush
29614 033/300 Read block 4502 len 28672
29615 034/300 Read block 183 len 32768
29616 035/300 Read block 4210 len 36864
29617 036/300 Flush
29618 037/300 Write block 1493 len 32768 data: 5 4 4 5 5 4 4 4
29619 038/300 Read block 3498 len 32768
29620 039/300 Flush
29621 040/300 Write block 2804 len 32768 data: 4 5 4 6 5 5 6 7
29622 041/300 Read block 1597 len 32768
29623 042/300 Write block 3627 len 20480 data: 4 3 2 2 3
29624 043/300 Write block 4740 len 12288 data: 4 4 3
29625 044/300 Write block 4700 len 28672 data: 6 7 6 6 7 6 5
29626 045/300 Read block 4827 len 20480
29627 046/300 Read block 783 len 16384
29628 047/300 Write block 649 len 4096 data: 3
29629 048/300 Read block 971 len 24576
29630 049/300 Read block 373 len 12288
29631 050/300 Flush
29632 051/300 Read block 3794 len 4096
29633 052/300 Read block 3549 len 8192
29634 053/300 Write block 4577 len 12288 data: 6 7 6
29635 054/300 Read block 1573 len 16384
29636 055/300 Read block 4442 len 32768
29637 056/300 Read block 3125 len 8192
29638 057/300 Flush
29639 058/300 Flush
29640 059/300 Read block 1108 len 32768
29641 060/300 Write block 1666 len 12288 data: 6 6 6
29642 061/300 Write block 3040 len 20480 data: 4 6 6 6 6
29643 062/300 Read block 3238 len 8192
29644 063/300 Write block 242 len 24576 data: 5 5 5 4 4 5
29645 064/300 Read block 523 len 32768
29646 065/300 Write block 1323 len 12288 data: 6 7 5
29647 066/300 Read block 4229 len 16384
29648 067/300 Write block 1604 len 32768 data: 4 4 5 5 6 6 5 5
29649 068/300 Read block 2024 len 8192
29650 069/300 Read block 704 len 16384
29651 070/300 Read block 2381 len 12288
29652 071/300 Write block 750 len 24576 data: 5 5 6 6 6 5
29653 072/300 Write block 4255 len 28672 data: 6 6 7 7 6 7 6
29654 073/300 Write block 704 len 24576 data: 3 3 3 3 3 4
29655 074/300 Flush
29656 075/300 Read block 2115 len 28672
29657 076/300 Flush
29658 077/300 Write block 809 len 40960 data: 5 5 5 5 5 5 4 4 3 3
29659 078/300 Write block 2282 len 8192 data: 5 6
29660 079/300 Write block 3679 len 32768 data: 4 3 4 5 5 6 6 7
29661 080/300 Write block 3305 len 20480 data: 8 6 8 9 9
29662 081/300 Write block 2908 len 40960 data: 6 3 2 2 4 4 6 6 5 6
29663 082/300 Read block 3573 len 20480
29664 083/300 Read block 2334 len 32768
29665 084/300 Read block 2278 len 16384
29666 085/300 Read block 4360 len 36864
29667 086/300 Write block 1116 len 12288 data: 4 5 6
29668 087/300 Read block 2406 len 28672
29669 088/300 Read block 3149 len 8192
29670 089/300 Write block 2753 len 16384 data: 6 7 5 4
29671 090/300 Write block 3497 len 24576 data: 6 6 5 5 6 6
29672 091/300 Write block 3722 len 36864 data: 7 6 9 9 9 7 7 7 8
29673 092/300 Write block 4063 len 4096 data: 7
29674 093/300 Write block 1791 len 40960 data: 7 8 7 4 4 2 2 2 2 4
29675 094/300 Write block 4590 len 20480 data: 8 10 9 5 3
29676 095/300 Read block 2111 len 4096
29677 096/300 Write block 1945 len 32768 data: 7 6 7 8 8 8 8 6
29678 097/300 Write block 724 len 40960 data: 3 3 4 5 7 8 9 8 7 5
29679 098/300 Flush
29680 099/300 Write block 4029 len 4096 data: 5
29681 100/300 Read block 1480 len 4096
29682 101/300 Read block 2152 len 24576
29683 102/300 Flush
29684 103/300 Write block 4294 len 36864 data: 4 4 4 3 3 5 7 7 7
29685 104/300 Write block 4591 len 16384 data: 11 10 6 4
29686 105/300 Read block 2497 len 12288
29687 106/300 Read block 610 len 12288
29688 107/300 Write block 1283 len 8192 data: 5 7
29689 108/300 Read block 3123 len 20480
29690 109/300 Read block 2595 len 20480
29691 110/300 Read block 3562 len 12288
29692 111/300 Read block 535 len 20480
29693 112/300 Read block 4170 len 20480
29694 113/300 Write block 4651 len 36864 data: 4 4 4 4 4 2 3 6 7
29695 114/300 Write block 3507 len 32768 data: 4 4 3 4 3 3 4 5
29696 115/300 Read block 2588 len 32768
29697 116/300 Read block 535 len 36864
29698 117/300 Read block 3547 len 8192
29699 118/300 Read block 2581 len 36864
29700 119/300 Read block 4298 len 16384
29701 120/300 Read block 4316 len 16384
29702 121/300 Write block 4744 len 16384 data: 4 4 5 5
29703 122/300 Read block 196 len 40960
29704 123/300 Read block 121 len 32768
29705 124/300 Read block 3400 len 16384
29706 125/300 Read block 1198 len 16384
29707 126/300 Flush
29708 127/300 Write block 3752 len 12288 data: 5 4 4
29709 128/300 Read block 618 len 40960
29710 129/300 Read block 1187 len 40960
29711 130/300 Write block 1732 len 20480 data: 5 5 6 6 5
29712 131/300 Write block 2144 len 32768 data: 8 6 4 4 4 4 2 2
29713 132/300 Read block 3496 len 12288
29714 133/300 Read block 3580 len 4096
29715 134/300 Read block 4376 len 20480
29716 135/300 Read block 2090 len 8192
29717 136/300 Write block 1484 len 36864 data: 7 8 8 6 6 7 7 6 5
29718 137/300 Read block 314 len 12288
29719 138/300 Write block 751 len 24576 data: 6 7 7 7 6 3
29720 139/300 Read block 1496 len 4096
29721 140/300 Read block 128 len 12288
29722 141/300 Read block 4254 len 40960
29723 142/300 Write block 1946 len 12288 data: 7 8 9
29724 143/300 Read block 319 len 8192
29725 144/300 Write block 2294 len 20480 data: 8 8 6 6 6
29726 145/300 Read block 2132 len 36864
29727 146/300 Write block 4932 len 40960 data: 4 5 6 6 5 5 4 6 7 8
29728 147/300 Read block 239 len 4096
29729 148/300 Write block 4302 len 8192 data: 8 7
29730 149/300 Read block 2261 len 24576
29731 150/300 Write block 4538 len 28672 data: 9 7 4 4 4 4 4
29732 151/300 Write block 4990 len 28672 data: 7 8 7 7 6 6 3
29733 152/300 Flush
29734 153/300 Write block 4356 len 4096 data: 10
29735 154/300 Read block 940 len 36864
29736 155/300 Write block 3997 len 12288 data: 6 5 5
29737 156/300 Read block 4476 len 28672
29738 157/300 Read block 2140 len 40960
29739 158/300 Flush
29740 159/300 Read block 3728 len 12288
29741 160/300 Write block 3895 len 4096 data: 4
29742 161/300 Read block 3210 len 16384
29743 162/300 Write block 4994 len 8192 data: 7 7
29744 163/300 Write block 3994 len 20480 data: 8 6 5 7 6
29745 164/300 Read block 1736 len 24576
29746 165/300 Write block 288 len 20480 data: 4 3 5 4 3
29747 166/300 Read block 4687 len 8192
29748 167/300 Write block 3941 len 24576 data: 8 8 8 8 10 8
29749 168/300 Write block 1598 len 28672 data: 3 3 3 4 6 5 5
29750 169/300 Read block 1298 len 8192
29751 170/300 Write block 478 len 4096 data: 7
29752 171/300 Read block 1676 len 36864
29753 172/300 Write block 72 len 32768 data: 9 8 9 8 5 3 3 3
29754 173/300 Read block 1900 len 24576
29755 174/300 Flush
29756 175/300 Write block 1795 len 12288 data: 5 3 3
29757 176/300 Write block 2062 len 12288 data: 5 6 5
29758 177/300 Write block 1551 len 16384 data: 4 5 6 5
29759 178/300 Read block 4078 len 4096
29760 179/300 Read block 4486 len 36864
29761 180/300 Write block 2331 len 36864 data: 7 8 8 8 7 4 4 4 3
29762 181/300 Write block 2477 len 24576 data: 4 4 4 4 3 4
29763 182/300 Write block 3568 len 16384 data: 3 2 3 4
29764 183/300 Write block 4098 len 24576 data: 3 3 3 3 4 3
29765 184/300 Read block 3960 len 8192
29766 185/300 Write block 304 len 8192 data: 8 6
29767 186/300 Write block 3109 len 16384 data: 4 7 6 5
29768 187/300 Read block 242 len 16384
29769 188/300 Read block 1797 len 28672
29770 189/300 Read block 551 len 20480
29771 190/300 Flush
29772 191/300 Write block 3194 len 32768 data: 4 4 4 5 5 6 4 4
29773 192/300 Write block 3265 len 16384 data: 4 3 3 4
29774 193/300 Read block 4967 len 24576
29775 194/300 Read block 2018 len 16384
29776 195/300 Read block 3253 len 32768
29777 196/300 Read block 969 len 36864
29778 197/300 Read block 4151 len 24576
29779 198/300 Read block 1693 len 16384
29780 199/300 Write block 1585 len 20480 data: 7 5 4 5 4
29781 200/300 Read block 4482 len 20480
29782 201/300 Read block 3195 len 12288
29783 202/300 Flush
29784 203/300 Write block 1571 len 40960 data: 8 6 8 6 3 3 3 3 5 6
29785 204/300 Flush
29786 205/300 Write block 3321 len 36864 data: 3 4 4 3 4 3 2 5 4
29787 206/300 Read block 2134 len 8192
29788 207/300 Write block 1573 len 24576 data: 9 7 4 4 4 4
29789 208/300 Write block 3152 len 8192 data: 4 4
29790 209/300 Read block 1318 len 40960
29791 210/300 Read block 1111 len 16384
29792 211/300 Read block 2026 len 36864
29793 212/300 Read block 69 len 36864
29794 213/300 Read block 2463 len 36864
29795 214/300 Write block 4189 len 12288 data: 5 5 5
29796 215/300 Read block 3999 len 36864
29797 216/300 Read block 1485 len 24576
29798 217/300 Write block 3321 len 16384 data: 4 5 5 4
29799 218/300 Read block 1490 len 20480
29800 219/300 Read block 700 len 20480
29801 220/300 Write block 2987 len 8192 data: 4 5
29802 221/300 Read block 2034 len 36864
29803 222/300 Read block 4758 len 12288
29804 223/300 Read block 2189 len 36864
29805 224/300 Read block 960 len 40960
29806 225/300 Flush
29807 226/300 Read block 1003 len 40960
29808 227/300 Read block 2703 len 12288
29809 228/300 Read block 3868 len 40960
29810 229/300 Write block 4948 len 28672 data: 5 5 4 5 6 5 6
29811 230/300 Read block 4126 len 28672
29812 231/300 Read block 2928 len 8192
29813 232/300 Write block 1808 len 4096 data: 5
29814 233/300 Write block 2730 len 28672 data: 6 6 5 6 7 5 5
29815 234/300 Write block 2635 len 36864 data: 6 6 8 8 7 6 4 4 2
29816 235/300 Flush
29817 236/300 Read block 1321 len 20480
29818 237/300 Read block 1003 len 4096
29819 238/300 Write block 3550 len 16384 data: 6 7 7 9
29820 239/300 Read block 1217 len 36864
29821 240/300 Write block 3051 len 12288 data: 6 4 7
29822 241/300 Flush
29823 242/300 Read block 4032 len 4096
29824 243/300 Read block 4035 len 36864
29825 244/300 Write block 1937 len 28672 data: 11 8 9 9 7 7 6
29826 245/300 Read block 4389 len 28672
29827 246/300 Read block 3524 len 40960
29828 247/300 Write block 3451 len 28672 data: 5 7 7 7 7 6 5
29829 248/300 Read block 4949 len 28672
29830 249/300 Read block 290 len 36864
29831 250/300 Write block 3387 len 8192 data: 5 5
29832 251/300 Write block 4438 len 8192 data: 3 4
29833 252/300 Read block 3708 len 32768
29834 253/300 Write block 2949 len 20480 data: 8 8 8 9 7
29835 254/300 Write block 2537 len 28672 data: 7 8 8 5 7 8 7
29836 255/300 Read block 2400 len 8192
29837 256/300 Write block 3369 len 20480 data: 6 6 6 5 5
29838 257/300 Write block 1512 len 40960 data: 4 4 4 4 4 5 4 5 4 4
29839 258/300 Write block 1819 len 24576 data: 4 5 5 5 7 7
29840 259/300 Read block 3674 len 36864
29841 260/300 Read block 3466 len 28672
29842 261/300 Read block 545 len 20480
29843 262/300 Write block 1342 len 28672 data: 6 6 6 6 6 4 6
29844 263/300 Read block 2464 len 36864
29845 264/300 Read block 4977 len 32768
29846 265/300 Read block 3615 len 12288
29847 266/300 Write block 4703 len 36864 data: 7 8 7 6 5 7 4 5 5
29848 267/300 Read block 4540 len 28672
29849 268/300 Read block 79 len 20480
29850 269/300 Write block 3575 len 24576 data: 5 5 4 4 4 4
29851 270/300 Flush
29852 271/300 Read block 2214 len 20480
29853 272/300 Read block 4498 len 8192
29854 273/300 Read block 579 len 32768
29855 274/300 Read block 413 len 32768
29856 275/300 Read block 4854 len 16384
29857 276/300 Read block 290 len 16384
29858 277/300 Write block 1924 len 12288 data: 3 3 3
29859 278/300 Write block 2217 len 24576 data: 9 10 10 9 9 10
29860 279/300 Write block 1720 len 16384 data: 10 10 9 8
29861 280/300 Read block 945 len 40960
29862 281/300 Read block 3542 len 16384
29863 282/300 Write block 3321 len 16384 data: 5 6 6 5
29864 283/300 Write block 4263 len 28672 data: 4 6 6 6 6 7 7
29865 284/300 Read block 752 len 12288
29866 285/300 Read block 2045 len 32768
29867 286/300 Write block 1018 len 28672 data: 4 6 7 6 7 7 7
29868 287/300 Read block 1461 len 8192
29869 288/300 Write block 3541 len 32768 data: 7 6 5 4 4 4 5 5
29870 289/300 Read block 2182 len 28672
29871 290/300 Write block 3259 len 32768 data: 3 2 2 2 3 3 5 4
29872 291/300 Write block 1685 len 12288 data: 4 5 5
29873 292/300 Read block 743 len 32768
29874 293/300 Write block 437 len 28672 data: 4 6 5 5 6 4 3
29875 294/300 Read block 2767 len 16384
29876 295/300 Write block 2979 len 8192 data: 5 4
29877 296/300 Write block 2763 len 8192 data: 9 9
29878 297/300 Read block 2253 len 32768
29879 298/300 Write block 779 len 4096 data: 7
29880 299/300 Read block 98 len 16384
29881 300/300 Read block 3027 len 8192
29882 301/300 Read block 4929 len 40960
29883 Main task has work to do, go find it
29884 got dsc Start(0)
29885 start 0
29886 Feb 25 17:24:48.944 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/0, method: POST, req_id: e7132790-950a-4fe9-8a8c-312cd49d1e18, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
29887 Replay: started 0, returned:Ok(())
29888 ----------------------------------------------------------------
29889 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29890 States: Offline Active Active
29891 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29892 8850 Acked Read 2 Sent Done Done false
29893 8851 Acked Read 6 Sent Done Done false
29894 8852 Acked Read 10 Sent Done Done false
29895 8853 Acked Read 3 Sent Done Done false
29896 8854 Acked Read 2 Sent Done Done false
29897 8855 Acked Write 4 Sent Done Done false
29898 8856 Acked Write 2 Sent Done Done false
29899 8857 Acked Read 9 Sent Done Done false
29900 8858 Acked Flush 0 Sent Done Done false
29901 8859 Acked Read 7 Sent Done Done false
29902 8860 Acked Write 4 Sent Done Done false
29903 8861 Acked Flush 0 Sent Done Done false
29904 8862 Acked Read 8 Sent Done Done false
29905 8863 Acked Read 3 Sent Done Done false
29906 8864 Acked Write 6 Sent Done Done false
29907 8865 Acked Write 10 Sent Done Done false
29908 8866 Acked Write 8 Sent Done Done false
29909 8867 Acked Read 3 Sent Done Done false
29910 8868 Acked Write 3 Sent Done Done false
29911 8869 Acked Flush 0 Sent Done Done false
29912 8870 Acked Flush 0 Sent Done Done false
29913 8871 Acked Read 8 Sent Done Done false
29914 8872 Acked Write 10 Sent Done Done false
29915 8873 Acked Write 8 Sent Done Done false
29916 8874 Acked Read 6 Sent Done Done false
29917 8875 Acked Read 8 Sent Done Done false
29918 8876 Acked Flush 0 Sent Done Done false
29919 8877 Acked Read 7 Sent Done Done false
29920 8878 Acked Read 6 Sent Done Done false
29921 8879 Acked Read 4 Sent Done Done false
29922 8880 Acked Read 3 Sent Done Done false
29923 8881 Acked Flush 0 Sent Done Done false
29924 8882 Acked Read 7 Sent Done Done false
29925 8883 Acked Read 8 Sent Done Done false
29926 8884 Acked Read 9 Sent Done Done false
29927 8885 Acked Flush 0 Sent Done Done false
29928 8886 Acked Write 8 Sent Done Done false
29929 8887 Acked Read 8 Sent Done Done false
29930 8888 Acked Flush 0 Sent Done Done false
29931 8889 Acked Write 8 Sent Done Done false
29932 8890 Acked Read 8 Sent Done Done false
29933 8891 Acked Write 5 Sent Done Done false
29934 8892 Acked Write 3 Sent Done Done false
29935 8893 Acked Write 7 Sent Done Done false
29936 8894 Acked Read 5 Sent Done Done false
29937 8895 Acked Read 4 Sent Done Done false
29938 8896 Acked Write 1 Sent Done Done false
29939 8897 Acked Read 6 Sent Done Done false
29940 8898 Acked Read 3 Sent Done Done false
29941 8899 Acked Flush 0 Sent Done Done false
29942 8900 Acked Read 1 Sent Done Done false
29943 8901 Acked Read 2 Sent Done Done false
29944 8902 Acked Write 3 Sent Done Done false
29945 8903 Acked Read 4 Sent Done Done false
29946 8904 Acked Read 8 Sent Done Done false
29947 8905 Acked Read 2 Sent Done Done false
29948 8906 Acked Flush 0 Sent Done Done false
29949 8907 Acked Flush 0 Sent Done Done false
29950 8908 Acked Read 8 Sent Done Done false
29951 8909 Acked Write 3 Sent Done Done false
29952 8910 Acked Write 5 Sent Done Done false
29953 8911 Acked Read 2 Sent Done Done false
29954 8912 Acked Write 6 Sent Done Done false
29955 8913 Acked Read 8 Sent Done Done false
29956 8914 Acked Write 3 Sent Done Done false
29957 8915 Acked Read 4 Sent Done Done false
29958 8916 Acked Write 8 Sent Done Done false
29959 8917 Acked Read 2 Sent Done Done false
29960 8918 Acked Read 4 Sent Done Done false
29961 8919 Acked Read 3 Sent Done Done false
29962 8920 Acked Write 6 Sent Done Done false
29963 8921 Acked Write 7 Sent Done Done false
29964 8922 Acked Write 6 Sent Done Done false
29965 8923 Acked Flush 0 Sent Done Done false
29966 8924 Acked Read 7 Sent Done Done false
29967 8925 Acked Flush 0 Sent Done Done false
29968 8926 Acked Write 10 Sent Done Done false
29969 8927 Acked Write 2 Sent Done Done false
29970 8928 Acked Write 8 Sent Done Done false
29971 8929 Acked Write 5 Sent Done Done false
29972 8930 Acked Write 10 Sent Done Done false
29973 8931 Acked Read 5 Sent Done Done false
29974 8932 Acked Read 8 Sent Done Done false
29975 8933 Acked Read 4 Sent Done Done false
29976 8934 Acked Read 9 Sent Done Done false
29977 8935 Acked Write 3 Sent Done Done false
29978 8936 Acked Read 7 Sent Done Done false
29979 8937 Acked Read 2 Sent Done Done false
29980 8938 Acked Write 4 Sent Done Done false
29981 8939 Acked Write 6 Sent Done Done false
29982 8940 Acked Write 9 Sent Done Done false
29983 8941 Acked Write 1 Sent Done Done false
29984 8942 Acked Write 10 Sent Done Done false
29985 8943 Acked Write 5 Sent Done Done false
29986 8944 Acked Read 1 Sent Done Done false
29987 8945 Acked Write 8 Sent Done Done false
29988 8946 Acked Write 10 Sent Done Done false
29989 8947 Acked Flush 0 Sent Done Done false
29990 8948 Acked Write 1 Sent Done Done false
29991 8949 Acked Read 1 Sent Done Done false
29992 8950 Acked Read 6 Sent Done Done false
29993 8951 Acked Flush 0 Sent Done Done false
29994 8952 Acked Write 9 Sent Done Done false
29995 8953 Acked Write 4 Sent Done Done false
29996 8954 Acked Read 3 Sent Done Done false
29997 8955 Acked Read 3 Sent Done Done false
29998 8956 Acked Write 2 Sent Done Done false
29999 8957 Acked Read 5 Sent Done Done false
30000 8958 Acked Read 5 Sent Done Done false
30001 8959 Acked Read 3 Sent Done Done false
30002 8960 Acked Read 5 Sent Done Done false
30003 8961 Acked Read 5 Sent Done Done false
30004 8962 Acked Write 9 Sent Done Done false
30005 8963 Acked Write 8 Sent Done Done false
30006 8964 Acked Read 8 Sent Done Done false
30007 8965 Acked Read 9 Sent Done Done false
30008 8966 Acked Read 2 Sent Done Done false
30009 8967 Acked Read 9 Sent Done Done false
30010 8968 Acked Read 4 Sent Done Done false
30011 8969 Acked Read 4 Sent Done Done false
30012 8970 Acked Write 4 Sent Done Done false
30013 8971 Acked Read 10 Sent Done Done false
30014 8972 Acked Read 8 Sent Done Done false
30015 8973 Acked Read 4 Sent Done Done false
30016 8974 Acked Read 4 Sent Done Done false
30017 8975 Acked Flush 0 Sent Done Done false
30018 8976 Acked Write 3 Sent Done Done false
30019 8977 Acked Read 10 Sent Done Done false
30020 8978 Acked Read 10 Sent Done Done false
30021 8979 Acked Write 5 Sent Done Done false
30022 8980 Acked Write 8 Sent Done Done false
30023 8981 Acked Read 3 Sent Done Done false
30024 8982 Acked Read 1 Sent Done Done false
30025 8983 Acked Read 5 Sent Done Done false
30026 8984 Acked Read 2 Sent Done Done false
30027 8985 Acked Write 9 Sent Done Done false
30028 8986 Acked Read 3 Sent Done Done false
30029 8987 Acked Write 6 Sent Done Done false
30030 8988 Acked Read 1 Sent Done Done false
30031 8989 Acked Read 3 Sent Done Done false
30032 8990 Acked Read 10 Sent Done Done false
30033 8991 Acked Write 3 Sent Done Done false
30034 8992 Acked Read 2 Sent Done Done false
30035 8993 Acked Write 5 Sent Done Done false
30036 8994 Acked Read 9 Sent Done Done false
30037 8995 Acked Write 10 Sent Done Done false
30038 8996 Acked Read 1 Sent Done Done false
30039 8997 Acked Write 2 Sent Done Done false
30040 8998 Acked Read 6 Sent Done Done false
30041 8999 Acked Write 7 Sent Done Done false
30042 9000 Acked Write 7 Sent Done Done false
30043 9001 Acked Flush 0 Sent Done Done false
30044 9002 Acked Write 1 Sent Done Done false
30045 9003 Acked Read 9 Sent Done Done false
30046 9004 Acked Write 3 Sent Done Done false
30047 9005 Acked Read 7 Sent Done Done false
30048 9006 Acked Read 10 Sent Done Done false
30049 9007 Acked Flush 0 Sent Done Done false
30050 9008 Acked Read 3 Sent Done Done false
30051 9009 Acked Write 1 Sent Done Done false
30052 9010 Acked Read 4 Sent Done Done false
30053 9011 Acked Write 2 Sent Done Done false
30054 9012 Acked Write 5 Sent Done Done false
30055 9013 Acked Read 6 Sent Done Done false
30056 9014 Acked Write 5 Sent Done Done false
30057 9015 Acked Read 2 Sent Done Done false
30058 9016 Acked Write 6 Sent Done Done false
30059 9017 Acked Write 7 Sent Done Done false
30060 9018 Acked Read 2 Sent Done Done false
30061 9019 Acked Write 1 Sent Done Done false
30062 9020 Acked Read 9 Sent Done Done false
30063 9021 Acked Write 8 Sent Done Done false
30064 9022 Acked Read 6 Sent Done Done false
30065 9023 Acked Flush 0 Sent Done Done false
30066 9024 Acked Write 3 Sent Done Done false
30067 9025 Acked Write 3 Sent Done Done false
30068 9026 Acked Write 4 Sent Done Done false
30069 9027 Acked Read 1 Sent Done Done false
30070 9028 Acked Read 9 Sent Done Done false
30071 9029 Acked Write 9 Sent Done Done false
30072 9030 Acked Write 6 Sent Done Done false
30073 9031 Acked Write 4 Sent Done Done false
30074 9032 Acked Write 6 Sent Done Done false
30075 9033 Acked Read 2 Sent Done Done false
30076 9034 Acked Write 2 Sent Done Done false
30077 9035 Acked Write 4 Sent Done Done false
30078 9036 Acked Read 4 Sent Done Done false
30079 9037 Acked Read 7 Sent Done Done false
30080 9038 Acked Read 5 Sent Done Done false
30081 9039 Acked Flush 0 Sent Done Done false
30082 9040 Acked Write 8 Sent Done Done false
30083 9041 Acked Write 4 Sent Done Done false
30084 9042 Acked Read 6 Sent Done Done false
30085 9043 Acked Read 4 Sent Done Done false
30086 9044 Acked Read 8 Sent Done Done false
30087 9045 Acked Read 9 Sent Done Done false
30088 9046 Acked Read 6 Sent Done Done false
30089 9047 Acked Read 4 Sent Done Done false
30090 9048 Acked Write 5 Sent Done Done false
30091 9049 Acked Read 5 Sent Done Done false
30092 9050 Acked Read 3 Sent Done Done false
30093 9051 Acked Flush 0 Sent Done Done false
30094 9052 Acked Write 10 Sent Done Done false
30095 9053 Acked Flush 0 Sent Done Done false
30096 9054 Acked Write 9 Sent Done Done false
30097 9055 Acked Read 2 Sent Done Done false
30098 9056 Acked Write 6 Sent Done Done false
30099 9057 Acked Write 2 Sent Done Done false
30100 9058 Acked Read 10 Sent Done Done false
30101 9059 Acked Read 4 Sent Done Done false
30102 9060 Acked Read 9 Sent Done Done false
30103 9061 Acked Read 9 Sent Done Done false
30104 9062 Acked Read 9 Sent Done Done false
30105 9063 Acked Write 3 Sent Done Done false
30106 9064 Acked Read 9 Sent Done Done false
30107 9065 Acked Read 6 Sent Done Done false
30108 9066 Acked Write 4 Sent Done Done false
30109 9067 Acked Read 5 Sent Done Done false
30110 9068 Acked Read 5 Sent Done Done false
30111 9069 Acked Write 2 Sent Done Done false
30112 9070 Acked Read 9 Sent Done Done false
30113 9071 Acked Read 3 Sent Done Done false
30114 9072 Acked Read 9 Sent Done Done false
30115 9073 Acked Read 10 Sent Done Done false
30116 9074 Acked Flush 0 Sent Done Done false
30117 9075 Acked Read 10 Sent Done Done false
30118 9076 Acked Read 3 Sent Done Done false
30119 9077 Acked Read 10 Sent Done Done false
30120 9078 Acked Write 7 Sent Done Done false
30121 9079 Acked Read 7 Sent Done Done false
30122 9080 Acked Read 2 Sent Done Done false
30123 9081 Acked Write 1 Sent Done Done false
30124 9082 Acked Write 7 Sent Done Done false
30125 9083 Acked Write 9 Sent Done Done false
30126 9084 Acked Flush 0 Sent Done Done false
30127 9085 Acked Read 5 Sent Done Done false
30128 9086 Acked Read 1 Sent Done Done false
30129 9087 Acked Write 4 Sent Done Done false
30130 9088 Acked Read 9 Sent Done Done false
30131 9089 Acked Write 3 Sent Done Done false
30132 9090 Acked Flush 0 Sent Done Done false
30133 9091 Acked Read 1 Sent Done Done false
30134 9092 Acked Read 9 Sent Done Done false
30135 9093 Acked Write 7 Sent Done Done false
30136 9094 Acked Read 7 Sent Done Done false
30137 9095 Acked Read 10 Sent Done Done false
30138 9096 Acked Write 7 Sent Done Done false
30139 9097 Acked Read 7 Sent Done Done false
30140 9098 Acked Read 9 Sent Done Done false
30141 9099 Acked Write 2 Sent Done Done false
30142 9100 Acked Write 2 Sent Done Done false
30143 9101 Acked Read 8 Sent Done Done false
30144 9102 Acked Write 5 Sent Done Done false
30145 9103 Acked Write 7 Sent Done Done false
30146 9104 Acked Read 2 Sent Done Done false
30147 9105 Acked Write 5 Sent Done Done false
30148 9106 Acked Write 10 Sent Done Done false
30149 9107 Acked Write 6 Sent Done Done false
30150 9108 Acked Read 9 Sent Done Done false
30151 9109 Acked Read 7 Sent Done Done false
30152 9110 Acked Read 5 Sent Done Done false
30153 9111 Acked Write 7 Sent Done Done false
30154 9112 Acked Read 9 Sent Done Done false
30155 9113 Acked Read 8 Sent Done Done false
30156 9114 Acked Read 3 Sent Done Done false
30157 9115 Acked Write 9 Sent Done Done false
30158 9116 Acked Read 7 Sent Done Done false
30159 9117 Acked Read 5 Sent Done Done false
30160 9118 Acked Write 6 Sent Done Done false
30161 9119 Acked Flush 0 Sent Done Done false
30162 9120 Acked Read 5 Sent Done Done false
30163 9121 Acked Read 2 Sent Done Done false
30164 9122 Acked Read 8 Sent Done Done false
30165 9123 Acked Read 8 Sent Done Done false
30166 9124 Acked Read 4 Sent Done Done false
30167 9125 Acked Read 4 Sent Done Done false
30168 9126 Acked Write 3 Sent Done Done false
30169 9127 Acked Write 6 Sent Done Done false
30170 9128 Acked Write 4 Sent Done Done false
30171 9129 Acked Read 10 Sent Done Done false
30172 9130 Acked Read 4 Sent Done Done false
30173 9131 Acked Write 4 Sent Done Done false
30174 9132 Acked Write 7 Sent Done Done false
30175 9133 Acked Read 3 Sent Done Done false
30176 9134 Acked Read 8 Sent Done Done false
30177 9135 Acked Write 7 Sent Done Done false
30178 9136 Acked Read 2 Sent Done Done false
30179 9137 Acked Write 8 Sent Done Done false
30180 9138 Acked Read 7 Sent Done Done false
30181 9139 Acked Write 8 Sent Done Done false
30182 9140 Acked Write 3 Sent Done Done false
30183 9141 Acked Read 8 Sent Done Done false
30184 9142 Acked Write 7 Sent Done Done false
30185 9143 Acked Read 4 Sent Done Done false
30186 9144 Acked Write 2 Sent Done Done false
30187 9145 Acked Write 2 Sent Done Done false
30188 9146 Acked Read 8 Sent Done Done false
30189 9147 Acked Write 1 Sent Done Done false
30190 9148 Acked Read 4 Sent Done Done false
30191 9149 Acked Read 2 Sent Done Done false
30192 9150 Acked Read 10 Sent Done Done false
30193 STATES DS:0 DS:1 DS:2 TOTAL
30194 Sent 301 0 0 301
30195 Done 0 301 301 602
30196 Skipped 0 0 0 0
30197 Error 0 0 0 0
30198 Last Flush: Some(JobId(8849)) Some(JobId(9119)) Some(JobId(9119))
30199 Downstairs last five completed: 8849 8848 8847 8846 8845
30200 Upstairs last five completed: 9150 9149 9148 9147 9146
30201 CLIENT: Up:0 ds:301 act:2
302022025-02-25T17:24:49.884ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
30203 [8810] got start action
30204 [8810] I am going to restart
30205 Starting downstairs at port 8810
30206 Make output file at "/tmp/dsc/downstairs-8810.txt"
30207 [8810][0] reports Starting
30208 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1235)
30209 [8810][0] reports Running
302102025-02-25T17:24:50.886ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
302112025-02-25T17:24:51.888ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
302122025-02-25T17:24:52.926ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
30213 ----------------------------------------------------------------
30214 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30215 States: Offline Active Active
30216 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30217 8850 Acked Read 2 Sent Done Done false
30218 8851 Acked Read 6 Sent Done Done false
30219 8852 Acked Read 10 Sent Done Done false
30220 8853 Acked Read 3 Sent Done Done false
30221 8854 Acked Read 2 Sent Done Done false
30222 8855 Acked Write 4 Sent Done Done false
30223 8856 Acked Write 2 Sent Done Done false
30224 8857 Acked Read 9 Sent Done Done false
30225 8858 Acked Flush 0 Sent Done Done false
30226 8859 Acked Read 7 Sent Done Done false
30227 8860 Acked Write 4 Sent Done Done false
30228 8861 Acked Flush 0 Sent Done Done false
30229 8862 Acked Read 8 Sent Done Done false
30230 8863 Acked Read 3 Sent Done Done false
30231 8864 Acked Write 6 Sent Done Done false
30232 8865 Acked Write 10 Sent Done Done false
30233 8866 Acked Write 8 Sent Done Done false
30234 8867 Acked Read 3 Sent Done Done false
30235 8868 Acked Write 3 Sent Done Done false
30236 8869 Acked Flush 0 Sent Done Done false
30237 8870 Acked Flush 0 Sent Done Done false
30238 8871 Acked Read 8 Sent Done Done false
30239 8872 Acked Write 10 Sent Done Done false
30240 8873 Acked Write 8 Sent Done Done false
30241 8874 Acked Read 6 Sent Done Done false
30242 8875 Acked Read 8 Sent Done Done false
30243 8876 Acked Flush 0 Sent Done Done false
30244 8877 Acked Read 7 Sent Done Done false
30245 8878 Acked Read 6 Sent Done Done false
30246 8879 Acked Read 4 Sent Done Done false
30247 8880 Acked Read 3 Sent Done Done false
30248 8881 Acked Flush 0 Sent Done Done false
30249 8882 Acked Read 7 Sent Done Done false
30250 8883 Acked Read 8 Sent Done Done false
30251 8884 Acked Read 9 Sent Done Done false
30252 8885 Acked Flush 0 Sent Done Done false
30253 8886 Acked Write 8 Sent Done Done false
30254 8887 Acked Read 8 Sent Done Done false
30255 8888 Acked Flush 0 Sent Done Done false
30256 8889 Acked Write 8 Sent Done Done false
30257 8890 Acked Read 8 Sent Done Done false
30258 8891 Acked Write 5 Sent Done Done false
30259 8892 Acked Write 3 Sent Done Done false
30260 8893 Acked Write 7 Sent Done Done false
30261 8894 Acked Read 5 Sent Done Done false
30262 8895 Acked Read 4 Sent Done Done false
30263 8896 Acked Write 1 Sent Done Done false
30264 8897 Acked Read 6 Sent Done Done false
30265 8898 Acked Read 3 Sent Done Done false
30266 8899 Acked Flush 0 Sent Done Done false
30267 8900 Acked Read 1 Sent Done Done false
30268 8901 Acked Read 2 Sent Done Done false
30269 8902 Acked Write 3 Sent Done Done false
30270 8903 Acked Read 4 Sent Done Done false
30271 8904 Acked Read 8 Sent Done Done false
30272 8905 Acked Read 2 Sent Done Done false
30273 8906 Acked Flush 0 Sent Done Done false
30274 8907 Acked Flush 0 Sent Done Done false
30275 8908 Acked Read 8 Sent Done Done false
30276 8909 Acked Write 3 Sent Done Done false
30277 8910 Acked Write 5 Sent Done Done false
30278 8911 Acked Read 2 Sent Done Done false
30279 8912 Acked Write 6 Sent Done Done false
30280 8913 Acked Read 8 Sent Done Done false
30281 8914 Acked Write 3 Sent Done Done false
30282 8915 Acked Read 4 Sent Done Done false
30283 8916 Acked Write 8 Sent Done Done false
30284 8917 Acked Read 2 Sent Done Done false
30285 8918 Acked Read 4 Sent Done Done false
30286 8919 Acked Read 3 Sent Done Done false
30287 8920 Acked Write 6 Sent Done Done false
30288 8921 Acked Write 7 Sent Done Done false
30289 8922 Acked Write 6 Sent Done Done false
30290 8923 Acked Flush 0 Sent Done Done false
30291 8924 Acked Read 7 Sent Done Done false
30292 8925 Acked Flush 0 Sent Done Done false
30293 8926 Acked Write 10 Sent Done Done false
30294 8927 Acked Write 2 Sent Done Done false
30295 8928 Acked Write 8 Sent Done Done false
30296 8929 Acked Write 5 Sent Done Done false
30297 8930 Acked Write 10 Sent Done Done false
30298 8931 Acked Read 5 Sent Done Done false
30299 8932 Acked Read 8 Sent Done Done false
30300 8933 Acked Read 4 Sent Done Done false
30301 8934 Acked Read 9 Sent Done Done false
30302 8935 Acked Write 3 Sent Done Done false
30303 8936 Acked Read 7 Sent Done Done false
30304 8937 Acked Read 2 Sent Done Done false
30305 8938 Acked Write 4 Sent Done Done false
30306 8939 Acked Write 6 Sent Done Done false
30307 8940 Acked Write 9 Sent Done Done false
30308 8941 Acked Write 1 Sent Done Done false
30309 8942 Acked Write 10 Sent Done Done false
30310 8943 Acked Write 5 Sent Done Done false
30311 8944 Acked Read 1 Sent Done Done false
30312 8945 Acked Write 8 Sent Done Done false
30313 8946 Acked Write 10 Sent Done Done false
30314 8947 Acked Flush 0 Sent Done Done false
30315 8948 Acked Write 1 Sent Done Done false
30316 8949 Acked Read 1 Sent Done Done false
30317 8950 Acked Read 6 Sent Done Done false
30318 8951 Acked Flush 0 Sent Done Done false
30319 8952 Acked Write 9 Sent Done Done false
30320 8953 Acked Write 4 Sent Done Done false
30321 8954 Acked Read 3 Sent Done Done false
30322 8955 Acked Read 3 Sent Done Done false
30323 8956 Acked Write 2 Sent Done Done false
30324 8957 Acked Read 5 Sent Done Done false
30325 8958 Acked Read 5 Sent Done Done false
30326 8959 Acked Read 3 Sent Done Done false
30327 8960 Acked Read 5 Sent Done Done false
30328 8961 Acked Read 5 Sent Done Done false
30329 8962 Acked Write 9 Sent Done Done false
30330 8963 Acked Write 8 Sent Done Done false
30331 8964 Acked Read 8 Sent Done Done false
30332 8965 Acked Read 9 Sent Done Done false
30333 8966 Acked Read 2 Sent Done Done false
30334 8967 Acked Read 9 Sent Done Done false
30335 8968 Acked Read 4 Sent Done Done false
30336 8969 Acked Read 4 Sent Done Done false
30337 8970 Acked Write 4 Sent Done Done false
30338 8971 Acked Read 10 Sent Done Done false
30339 8972 Acked Read 8 Sent Done Done false
30340 8973 Acked Read 4 Sent Done Done false
30341 8974 Acked Read 4 Sent Done Done false
30342 8975 Acked Flush 0 Sent Done Done false
30343 8976 Acked Write 3 Sent Done Done false
30344 8977 Acked Read 10 Sent Done Done false
30345 8978 Acked Read 10 Sent Done Done false
30346 8979 Acked Write 5 Sent Done Done false
30347 8980 Acked Write 8 Sent Done Done false
30348 8981 Acked Read 3 Sent Done Done false
30349 8982 Acked Read 1 Sent Done Done false
30350 8983 Acked Read 5 Sent Done Done false
30351 8984 Acked Read 2 Sent Done Done false
30352 8985 Acked Write 9 Sent Done Done false
30353 8986 Acked Read 3 Sent Done Done false
30354 8987 Acked Write 6 Sent Done Done false
30355 8988 Acked Read 1 Sent Done Done false
30356 8989 Acked Read 3 Sent Done Done false
30357 8990 Acked Read 10 Sent Done Done false
30358 8991 Acked Write 3 Sent Done Done false
30359 8992 Acked Read 2 Sent Done Done false
30360 8993 Acked Write 5 Sent Done Done false
30361 8994 Acked Read 9 Sent Done Done false
30362 8995 Acked Write 10 Sent Done Done false
30363 8996 Acked Read 1 Sent Done Done false
30364 8997 Acked Write 2 Sent Done Done false
30365 8998 Acked Read 6 Sent Done Done false
30366 8999 Acked Write 7 Sent Done Done false
30367 9000 Acked Write 7 Sent Done Done false
30368 9001 Acked Flush 0 Sent Done Done false
30369 9002 Acked Write 1 Sent Done Done false
30370 9003 Acked Read 9 Sent Done Done false
30371 9004 Acked Write 3 Sent Done Done false
30372 9005 Acked Read 7 Sent Done Done false
30373 9006 Acked Read 10 Sent Done Done false
30374 9007 Acked Flush 0 Sent Done Done false
30375 9008 Acked Read 3 Sent Done Done false
30376 9009 Acked Write 1 Sent Done Done false
30377 9010 Acked Read 4 Sent Done Done false
30378 9011 Acked Write 2 Sent Done Done false
30379 9012 Acked Write 5 Sent Done Done false
30380 9013 Acked Read 6 Sent Done Done false
30381 9014 Acked Write 5 Sent Done Done false
30382 9015 Acked Read 2 Sent Done Done false
30383 9016 Acked Write 6 Sent Done Done false
30384 9017 Acked Write 7 Sent Done Done false
30385 9018 Acked Read 2 Sent Done Done false
30386 9019 Acked Write 1 Sent Done Done false
30387 9020 Acked Read 9 Sent Done Done false
30388 9021 Acked Write 8 Sent Done Done false
30389 9022 Acked Read 6 Sent Done Done false
30390 9023 Acked Flush 0 Sent Done Done false
30391 9024 Acked Write 3 Sent Done Done false
30392 9025 Acked Write 3 Sent Done Done false
30393 9026 Acked Write 4 Sent Done Done false
30394 9027 Acked Read 1 Sent Done Done false
30395 9028 Acked Read 9 Sent Done Done false
30396 9029 Acked Write 9 Sent Done Done false
30397 9030 Acked Write 6 Sent Done Done false
30398 9031 Acked Write 4 Sent Done Done false
30399 9032 Acked Write 6 Sent Done Done false
30400 9033 Acked Read 2 Sent Done Done false
30401 9034 Acked Write 2 Sent Done Done false
30402 9035 Acked Write 4 Sent Done Done false
30403 9036 Acked Read 4 Sent Done Done false
30404 9037 Acked Read 7 Sent Done Done false
30405 9038 Acked Read 5 Sent Done Done false
30406 9039 Acked Flush 0 Sent Done Done false
30407 9040 Acked Write 8 Sent Done Done false
30408 9041 Acked Write 4 Sent Done Done false
30409 9042 Acked Read 6 Sent Done Done false
30410 9043 Acked Read 4 Sent Done Done false
30411 9044 Acked Read 8 Sent Done Done false
30412 9045 Acked Read 9 Sent Done Done false
30413 9046 Acked Read 6 Sent Done Done false
30414 9047 Acked Read 4 Sent Done Done false
30415 9048 Acked Write 5 Sent Done Done false
30416 9049 Acked Read 5 Sent Done Done false
30417 9050 Acked Read 3 Sent Done Done false
30418 9051 Acked Flush 0 Sent Done Done false
30419 9052 Acked Write 10 Sent Done Done false
30420 9053 Acked Flush 0 Sent Done Done false
30421 9054 Acked Write 9 Sent Done Done false
30422 9055 Acked Read 2 Sent Done Done false
30423 9056 Acked Write 6 Sent Done Done false
30424 9057 Acked Write 2 Sent Done Done false
30425 9058 Acked Read 10 Sent Done Done false
30426 9059 Acked Read 4 Sent Done Done false
30427 9060 Acked Read 9 Sent Done Done false
30428 9061 Acked Read 9 Sent Done Done false
30429 9062 Acked Read 9 Sent Done Done false
30430 9063 Acked Write 3 Sent Done Done false
30431 9064 Acked Read 9 Sent Done Done false
30432 9065 Acked Read 6 Sent Done Done false
30433 9066 Acked Write 4 Sent Done Done false
30434 9067 Acked Read 5 Sent Done Done false
30435 9068 Acked Read 5 Sent Done Done false
30436 9069 Acked Write 2 Sent Done Done false
30437 9070 Acked Read 9 Sent Done Done false
30438 9071 Acked Read 3 Sent Done Done false
30439 9072 Acked Read 9 Sent Done Done false
30440 9073 Acked Read 10 Sent Done Done false
30441 9074 Acked Flush 0 Sent Done Done false
30442 9075 Acked Read 10 Sent Done Done false
30443 9076 Acked Read 3 Sent Done Done false
30444 9077 Acked Read 10 Sent Done Done false
30445 9078 Acked Write 7 Sent Done Done false
30446 9079 Acked Read 7 Sent Done Done false
30447 9080 Acked Read 2 Sent Done Done false
30448 9081 Acked Write 1 Sent Done Done false
30449 9082 Acked Write 7 Sent Done Done false
30450 9083 Acked Write 9 Sent Done Done false
30451 9084 Acked Flush 0 Sent Done Done false
30452 9085 Acked Read 5 Sent Done Done false
30453 9086 Acked Read 1 Sent Done Done false
30454 9087 Acked Write 4 Sent Done Done false
30455 9088 Acked Read 9 Sent Done Done false
30456 9089 Acked Write 3 Sent Done Done false
30457 9090 Acked Flush 0 Sent Done Done false
30458 9091 Acked Read 1 Sent Done Done false
30459 9092 Acked Read 9 Sent Done Done false
30460 9093 Acked Write 7 Sent Done Done false
30461 9094 Acked Read 7 Sent Done Done false
30462 9095 Acked Read 10 Sent Done Done false
30463 9096 Acked Write 7 Sent Done Done false
30464 9097 Acked Read 7 Sent Done Done false
30465 9098 Acked Read 9 Sent Done Done false
30466 9099 Acked Write 2 Sent Done Done false
30467 9100 Acked Write 2 Sent Done Done false
30468 9101 Acked Read 8 Sent Done Done false
30469 9102 Acked Write 5 Sent Done Done false
30470 9103 Acked Write 7 Sent Done Done false
30471 9104 Acked Read 2 Sent Done Done false
30472 9105 Acked Write 5 Sent Done Done false
30473 9106 Acked Write 10 Sent Done Done false
30474 9107 Acked Write 6 Sent Done Done false
30475 9108 Acked Read 9 Sent Done Done false
30476 9109 Acked Read 7 Sent Done Done false
30477 9110 Acked Read 5 Sent Done Done false
30478 9111 Acked Write 7 Sent Done Done false
30479 9112 Acked Read 9 Sent Done Done false
30480 9113 Acked Read 8 Sent Done Done false
30481 9114 Acked Read 3 Sent Done Done false
30482 9115 Acked Write 9 Sent Done Done false
30483 9116 Acked Read 7 Sent Done Done false
30484 9117 Acked Read 5 Sent Done Done false
30485 9118 Acked Write 6 Sent Done Done false
30486 9119 Acked Flush 0 Sent Done Done false
30487 9120 Acked Read 5 Sent Done Done false
30488 9121 Acked Read 2 Sent Done Done false
30489 9122 Acked Read 8 Sent Done Done false
30490 9123 Acked Read 8 Sent Done Done false
30491 9124 Acked Read 4 Sent Done Done false
30492 9125 Acked Read 4 Sent Done Done false
30493 9126 Acked Write 3 Sent Done Done false
30494 9127 Acked Write 6 Sent Done Done false
30495 9128 Acked Write 4 Sent Done Done false
30496 9129 Acked Read 10 Sent Done Done false
30497 9130 Acked Read 4 Sent Done Done false
30498 9131 Acked Write 4 Sent Done Done false
30499 9132 Acked Write 7 Sent Done Done false
30500 9133 Acked Read 3 Sent Done Done false
30501 9134 Acked Read 8 Sent Done Done false
30502 9135 Acked Write 7 Sent Done Done false
30503 9136 Acked Read 2 Sent Done Done false
30504 9137 Acked Write 8 Sent Done Done false
30505 9138 Acked Read 7 Sent Done Done false
30506 9139 Acked Write 8 Sent Done Done false
30507 9140 Acked Write 3 Sent Done Done false
30508 9141 Acked Read 8 Sent Done Done false
30509 9142 Acked Write 7 Sent Done Done false
30510 9143 Acked Read 4 Sent Done Done false
30511 9144 Acked Write 2 Sent Done Done false
30512 9145 Acked Write 2 Sent Done Done false
30513 9146 Acked Read 8 Sent Done Done false
30514 9147 Acked Write 1 Sent Done Done false
30515 9148 Acked Read 4 Sent Done Done false
30516 9149 Acked Read 2 Sent Done Done false
30517 9150 Acked Read 10 Sent Done Done false
30518 STATES DS:0 DS:1 DS:2 TOTAL
30519 Sent 301 0 0 301
30520 Done 0 301 301 602
30521 Skipped 0 0 0 0
30522 Error 0 0 0 0
30523 Last Flush: Some(JobId(8849)) Some(JobId(9119)) Some(JobId(9119))
30524 Downstairs last five completed: 8849 8848 8847 8846 8845
30525 Upstairs last five completed: 9150 9149 9148 9147 9146
30526 CLIENT: Up:0 ds:301 act:2
305272025-02-25T17:24:53.891ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
305282025-02-25T17:24:54.680ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
305292025-02-25T17:24:54.680ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56528) = io task client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
305302025-02-25T17:24:54.681ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
305312025-02-25T17:24:54.682ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
305322025-02-25T17:24:54.682ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
305332025-02-25T17:24:54.682ZINFOcrucible: send last flush ID to this DS: Some(JobId(8849)) = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
305342025-02-25T17:24:54.682ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
305352025-02-25T17:24:54.682ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(8849)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
305362025-02-25T17:24:54.682ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(8849)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
30537 ----------------------------------------------------------------
30538 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30539 Downstairs last five completed: 9151 9150 9149 9148 9147
30540 Upstairs last five completed: 9151 9150 9149 9148 9147
30541 CLIENT: Up:0 ds:0 act:3
30542 Replay: All jobs finished
30543 Main task has work to do, go find it
30544 got dsc Stop(1)
30545 stop 1
30546 Feb 25 17:24:56.950 INFO request completed, latency_us: 425, response_code: 204, uri: /stop/cid/1, method:[8820] Got stop action so:false kr:false
30547 POST, req_id: 78a50feb-d049-49a1-8425-b6c7d40c3e5b, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
305482025-02-25T17:24:56.950ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
305492025-02-25T17:24:56.950ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
305502025-02-25T17:24:56.950ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
305512025-02-25T17:24:56.951ZINFOcrucible: client task is exiting = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
30552 Feb 25 17:24:56.951 INFO request completed, latency_us: 197, response_code: 200, uri: /state/cid/1, method: GET, req_id: dc72ebb9-b979-4c5d-ad90-249acd28a6f7, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
30553 [8820] Exited with: ExitStatus(unix_wait_status(9))
30554 [8820][1] reports Exit
30555 Feb 25 17:25:00.952 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/1, method: GET, req_id: 11c217f0-83fe-4447-a514-bd2aa4889e45, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
30556 001/300 Read block 2344 len 8192
30557 002/300 Read block 1144 len 16384
30558 003/300 Read block 2878 len 12288
30559 004/300 Read block 2662 len 40960
30560 005/300 Read block 43 len 24576
30561 006/300 Flush
30562 007/300 Write block 2068 len 16384 data: 4 5 5 5
30563 008/300 Read block 868 len 16384
30564 009/300 Read block 4385 len 20480
30565 010/300 Write block 1215 len 24576 data: 8 7 6 6 6 6
30566 011/300 Flush
30567 012/300 Write block 4310 len 4096 data: 5
30568 013/300 Read block 775 len 8192
30569 014/300 Write block 760 len 12288 data: 4 4 5
30570 015/300 Read block 2622 len 40960
30571 016/300 Read block 3655 len 28672
30572 017/300 Read block 3690 len 24576
30573 018/300 Write block 1223 len 8192 data: 5 5
30574 019/300 Write block 4939 len 40960 data: 7 8 9 7 6 6 5 6 5 6
30575 020/300 Flush
30576 021/300 Flush
30577 022/300 Write block 4066 len 40960 data: 8 7 5 5 4 4 3 3 4 4
30578 023/300 Read block 458 len 36864
30579 024/300 Read block 2608 len 24576
30580 025/300 Write block 3947 len 24576 data: 9 8 7 7 4 4
30581 026/300 Write block 4878 len 12288 data: 2 2 2
30582 027/300 Write block 3341 len 8192 data: 7 5
30583 028/300 Read block 2648 len 8192
30584 029/300 Read block 3668 len 40960
30585 030/300 Read block 1113 len 36864
30586 031/300 Read block 879 len 20480
30587 032/300 Read block 3583 len 28672
30588 033/300 Write block 866 len 36864 data: 6 5 4 5 8 7 6 4 4
30589 034/300 Read block 3100 len 36864
30590 035/300 Write block 1980 len 20480 data: 8 9 8 9 6
30591 036/300 Read block 1028 len 12288
30592 037/300 Write block 4912 len 12288 data: 4 3 4
30593 038/300 Write block 1253 len 36864 data: 4 5 5 4 5 5 4 6 7
30594 039/300 Write block 6 len 36864 data: 4 5 4 4 6 5 7 9 8
30595 040/300 Write block 4753 len 36864 data: 7 6 5 5 5 6 7 4 6
30596 041/300 Read block 175 len 20480
30597 042/300 Read block 2088 len 36864
30598 043/300 Read block 2074 len 12288
30599 044/300 Write block 1476 len 16384 data: 4 5 5 7
30600 045/300 Write block 2835 len 8192 data: 7 6
30601 046/300 Write block 872 len 36864 data: 7 5 5 3 4 5 4 4 4
30602 047/300 Read block 3354 len 36864
30603 048/300 Read block 1943 len 4096
30604 049/300 Write block 3510 len 24576 data: 5 4 4 5 6 7
30605 050/300 Read block 1984 len 28672
30606 051/300 Write block 4144 len 32768 data: 7 8 7 7 8 8 7 4
30607 052/300 Read block 4601 len 24576
30608 053/300 Write block 4494 len 12288 data: 8 7 5
30609 054/300 Write block 3907 len 36864 data: 3 3 6 6 6 6 7 6 6
30610 055/300 Read block 263 len 4096
30611 056/300 Write block 3446 len 4096 data: 5
30612 057/300 Read block 2006 len 28672
30613 058/300 Read block 3742 len 40960
30614 059/300 Read block 2349 len 24576
30615 060/300 Flush
30616 061/300 Write block 2840 len 32768 data: 5 4 4 3 2 3 3 3
30617 062/300 Read block 3857 len 4096
30618 063/300 Flush
30619 064/300 Read block 344 len 32768
30620 065/300 Read block 2494 len 36864
30621 066/300 Read block 3011 len 8192
30622 067/300 Write block 1136 len 40960 data: 3 3 3 4 4 4 4 5 3 5
30623 068/300 Read block 3937 len 36864
30624 069/300 Read block 4848 len 8192
30625 070/300 Write block 1098 len 8192 data: 8 7
30626 071/300 Read block 195 len 20480
30627 072/300 Read block 2764 len 8192
30628 073/300 Write block 1053 len 32768 data: 9 8 8 8 6 5 6 6
30629 074/300 Write block 1948 len 36864 data: 10 9 9 9 7 7 8 6 4
30630 075/300 Flush
30631 076/300 Write block 3290 len 32768 data: 6 3 2 2 3 3 6 6
30632 077/300 Read block 475 len 12288
30633 078/300 Write block 2425 len 36864 data: 6 6 6 6 6 6 7 5 6
30634 079/300 Write block 2484 len 12288 data: 4 4 6
30635 080/300 Read block 357 len 20480
30636 081/300 Write block 3644 len 4096 data: 5
30637 082/300 Read block 4987 len 32768
30638 083/300 Read block 3836 len 20480
30639 084/300 Write block 3376 len 8192 data: 4 4
30640 085/300 Write block 2723 len 16384 data: 5 5 5 5
30641 086/300 Write block 832 len 16384 data: 8 10 11 9
30642 087/300 Read block 4869 len 8192
30643 088/300 Write block 2699 len 40960 data: 5 6 8 6 5 6 6 8 7 7
30644 089/300 Read block 3117 len 16384
30645 090/300 Flush
30646 091/300 Write block 4092 len 24576 data: 6 5 6 6 4 3
30647 092/300 Read block 1776 len 24576
30648 093/300 Flush
30649 094/300 Flush
30650 095/300 Read block 1939 len 20480
30651 096/300 Read block 2001 len 40960
30652 097/300 Write block 3185 len 8192 data: 4 3
30653 098/300 Read block 2419 len 20480
30654 099/300 Write block 4911 len 24576 data: 4 5 4 5 4 5
30655 100/300 Write block 1660 len 28672 data: 4 5 5 6 4 5 7
30656 101/300 Write block 4127 len 16384 data: 5 6 6 6
30657 102/300 Read block 361 len 32768
30658 103/300 Read block 717 len 8192
30659 104/300 Read block 4828 len 24576
30660 105/300 Flush
30661 106/300 Write block 3964 len 24576 data: 4 4 6 6 6 6
30662 107/300 Read block 955 len 20480
30663 108/300 Read block 759 len 20480
30664 109/300 Read block 3634 len 36864
30665 110/300 Read block 4330 len 32768
30666 111/300 Write block 4343 len 32768 data: 5 5 6 6 6 6 8 7
30667 112/300 Write block 4838 len 12288 data: 2 2 4
30668 113/300 Write block 3762 len 32768 data: 6 5 4 3 5 5 4 5
30669 114/300 Read block 1016 len 36864
30670 115/300 Write block 1956 len 40960 data: 5 4 4 4 5 5 6 6 5 4
30671 116/300 Read block 1453 len 36864
30672 117/300 Read block 2881 len 24576
30673 118/300 Read block 1131 len 8192
30674 119/300 Read block 4132 len 12288
30675 120/300 Read block 4929 len 8192
30676 121/300 Read block 2200 len 40960
30677 122/300 Write block 23 len 20480 data: 7 8 7 9 10
30678 123/300 Write block 1295 len 40960 data: 3 5 5 5 5 6 5 3 3 3
30679 124/300 Read block 2945 len 32768
30680 125/300 Read block 1819 len 4096
30681 126/300 Write block 4289 len 28672 data: 7 7 5 3 3 5 5
30682 127/300 Read block 2574 len 4096
30683 128/300 Read block 1952 len 12288
30684 129/300 Write block 4121 len 28672 data: 5 5 6 5 4 5 6
30685 130/300 Read block 3901 len 24576
30686 131/300 Read block 3298 len 12288
30687 132/300 Read block 4688 len 8192
30688 133/300 Read block 1054 len 28672
30689 134/300 Read block 4453 len 28672
30690 135/300 Read block 1781 len 20480
30691 136/300 Write block 773 len 36864 data: 2 2 4 6 6 6 8 7 7
30692 137/300 Write block 3387 len 40960 data: 6 6 8 9 9 10 10 10 10 9
30693 138/300 Read block 4208 len 28672
30694 139/300 Read block 2743 len 12288
30695 140/300 Write block 3716 len 40960 data: 8 9 8 7 6 7 8 7 10 10
30696 141/300 Flush
30697 142/300 Read block 2042 len 40960
30698 143/300 Flush
30699 144/300 Read block 2787 len 32768
30700 145/300 Read block 4148 len 32768
30701 146/300 Read block 3963 len 16384
30702 147/300 Write block 3009 len 16384 data: 7 8 5 5
30703 148/300 Write block 404 len 40960 data: 5 6 5 5 4 4 6 7 7 7
30704 149/300 Write block 3208 len 4096 data: 5
30705 150/300 Write block 3260 len 28672 data: 3 3 3 4 4 6 5
30706 151/300 Write block 433 len 40960 data: 6 5 4 4 5 7 6 6 7 5
30707 152/300 Read block 2389 len 28672
30708 153/300 Read block 375 len 20480
30709 154/300 Read block 64 len 8192
30710 155/300 Read block 2258 len 12288
30711 156/300 Write block 218 len 12288 data: 7 5 3
30712 157/300 Read block 2776 len 32768
30713 158/300 Write block 675 len 32768 data: 5 5 3 5 5 4 6 6
30714 159/300 Flush
30715 160/300 Read block 3786 len 40960
30716 161/300 Read block 1599 len 24576
30717 162/300 Write block 2525 len 32768 data: 3 2 3 5 4 4 5 5
30718 163/300 Read block 2472 len 12288
30719 164/300 Flush
30720 165/300 Write block 432 len 12288 data: 6 7 6
30721 166/300 Read block 2847 len 36864
30722 167/300 Write block 1311 len 20480 data: 4 4 6 8 7
30723 168/300 Read block 3510 len 4096
30724 169/300 Write block 4817 len 8192 data: 5 6
30725 170/300 Write block 1200 len 12288 data: 6 5 6
30726 171/300 Write block 4300 len 8192 data: 8 8
30727 172/300 Write block 733 len 36864 data: 6 6 4 4 5 5 6 5 7
30728 173/300 Write block 508 len 36864 data: 3 5 6 6 8 8 7 9 9
30729 174/300 Read block 748 len 36864
30730 175/300 Read block 2428 len 4096
30731 176/300 Read block 54 len 8192
30732 177/300 Read block 1362 len 8192
30733 178/300 Read block 4304 len 32768
30734 179/300 Read block 2588 len 4096
30735 180/300 Read block 1099 len 16384
30736 181/300 Write block 1606 len 32768 data: 6 6 7 7 6 6 5 7
30737 182/300 Write block 456 len 20480 data: 10 9 7 6 6
30738 183/300 Read block 4005 len 28672
30739 184/300 Write block 4614 len 20480 data: 7 6 5 5 4
30740 185/300 Write block 3382 len 12288 data: 4 4 5
30741 186/300 Flush
30742 187/300 Write block 482 len 4096 data: 8
30743 188/300 Write block 667 len 8192 data: 4 3
30744 189/300 Write block 2164 len 36864 data: 4 6 5 6 5 4 4 2 3
30745 190/300 Write block 4106 len 20480 data: 5 5 5 4 4
30746 191/300 Read block 2892 len 16384
30747 192/300 Flush
30748 193/300 Read block 4000 len 40960
30749 194/300 Write block 4322 len 20480 data: 5 4 4 4 4
30750 195/300 Read block 1868 len 20480
30751 196/300 Write block 3609 len 12288 data: 8 7 7
30752 197/300 Read block 884 len 20480
30753 198/300 Read block 4804 len 32768
30754 199/300 Read block 2073 len 24576
30755 200/300 Read block 1488 len 4096
30756 201/300 Read block 2045 len 12288
30757 202/300 Write block 1277 len 20480 data: 2 4 3 3 4
30758 203/300 Write block 3993 len 24576 data: 9 9 7 6 8 7
30759 204/300 Write block 2204 len 8192 data: 5 4
30760 205/300 Write block 2031 len 36864 data: 9 8 8 8 8 7 9 8 8
30761 206/300 Read block 3197 len 12288
30762 207/300 Read block 1925 len 4096
30763 208/300 Write block 174 len 28672 data: 4 4 7 7 6 6 5
30764 209/300 Read block 2323 len 28672
30765 210/300 Read block 2080 len 28672
30766 211/300 Write block 511 len 16384 data: 7 9 9 8
30767 212/300 Write block 776 len 36864 data: 7 7 7 9 8 8 7 6 4
30768 213/300 Read block 2974 len 32768
30769 214/300 Read block 2943 len 8192
30770 215/300 Write block 2205 len 12288 data: 5 6 7
30771 216/300 Read block 4288 len 28672
30772 217/300 Read block 1159 len 8192
30773 218/300 Read block 2836 len 40960
30774 219/300 Read block 3676 len 12288
30775 220/300 Write block 4141 len 20480 data: 6 7 6 8 9
30776 221/300 Write block 4609 len 36864 data: 2 3 2 5 6 8 7 6 6
30777 222/300 Write block 1790 len 32768 data: 8 8 9 8 5 6 4 4
30778 223/300 Write block 2974 len 20480 data: 5 4 3 3 4
30779 224/300 Read block 3904 len 16384
30780 225/300 Read block 3022 len 24576
30781 226/300 Write block 4380 len 8192 data: 5 6
30782 227/300 Read block 3630 len 12288
30783 228/300 Write block 669 len 28672 data: 3 3 3 3 4 5 6
30784 229/300 Write block 2639 len 28672 data: 8 7 5 5 3 2 2
30785 230/300 Read block 985 len 40960
30786 231/300 Read block 3399 len 36864
30787 232/300 Read block 4588 len 16384
30788 233/300 Read block 743 len 20480
30789 234/300 Write block 3172 len 28672 data: 6 5 4 4 4 4 4
30790 235/300 Read block 2850 len 36864
30791 236/300 Flush
30792 237/300 Read block 953 len 16384
30793 238/300 Write block 561 len 8192 data: 6 4
30794 239/300 Read block 1611 len 40960
30795 240/300 Write block 4455 len 4096 data: 6
30796 241/300 Flush
30797 242/300 Read block 386 len 16384
30798 243/300 Read block 1359 len 4096
30799 244/300 Read block 851 len 24576
30800 245/300 Write block 3197 len 40960 data: 6 6 7 5 5 4 7 8 7 7
30801 246/300 Flush
30802 247/300 Read block 2592 len 28672
30803 248/300 Read block 1121 len 8192
30804 249/300 Write block 1714 len 12288 data: 8 8 6
30805 250/300 Flush
30806 251/300 Read block 2315 len 24576
30807 252/300 Read block 189 len 24576
30808 253/300 Write block 4362 len 40960 data: 5 4 4 4 4 5 6 6 5 4
30809 254/300 Write block 2764 len 4096 data: 10
30810 255/300 Write block 3535 len 24576 data: 3 3 4 7 7 8
30811 256/300 Write block 1675 len 20480 data: 7 5 6 6 5
30812 257/300 Read block 4930 len 28672
30813 258/300 Read block 4717 len 32768
30814 259/300 Write block 4612 len 36864 data: 6 7 9 8 7 7 5 5 4
30815 260/300 Flush
30816 261/300 Write block 801 len 4096 data: 6
30817 262/300 Write block 2841 len 32768 data: 5 5 4 3 4 4 4 4
30818 263/300 Read block 4742 len 36864
30819 264/300 Read block 1076 len 8192
30820 265/300 Read block 2157 len 12288
30821 266/300 Write block 3020 len 16384 data: 6 5 5 6
30822 267/300 Read block 3000 len 40960
30823 268/300 Read block 2324 len 28672
30824 269/300 Read block 4273 len 12288
30825 270/300 Write block 1851 len 4096 data: 3
30826 271/300 Read block 3999 len 28672
30827 272/300 Read block 264 len 24576
30828 273/300 Read block 4863 len 36864
30829 274/300 Read block 4940 len 4096
30830 275/300 Read block 1449 len 12288
30831 276/300 Write block 1457 len 28672 data: 9 8 8 7 6 6 7
30832 277/300 Read block 3752 len 28672
30833 278/300 Read block 1244 len 36864
30834 279/300 Read block 4376 len 16384
30835 280/300 Write block 4964 len 32768 data: 6 7 5 6 7 4 5 4
30836 281/300 Read block 3288 len 8192
30837 282/300 Read block 2457 len 36864
30838 283/300 Read block 981 len 36864
30839 284/300 Write block 1183 len 28672 data: 4 4 4 3 2 2 4
30840 285/300 Flush
30841 286/300 Write block 1835 len 16384 data: 4 5 6 7
30842 287/300 Write block 1738 len 36864 data: 4 4 3 4 2 3 4 5 6
30843 288/300 Read block 3323 len 40960
30844 289/300 Flush
30845 290/300 Write block 4277 len 8192 data: 4 3
30846 291/300 Write block 882 len 12288 data: 4 5 5
30847 292/300 Write block 3073 len 40960 data: 5 5 7 9 7 6 6 8 7 9
30848 293/300 Read block 3141 len 40960
30849 294/300 Read block 4038 len 36864
30850 295/300 Read block 493 len 16384
30851 296/300 Write block 725 len 20480 data: 4 5 6 8 9
30852 297/300 Read block 2759 len 12288
30853 298/300 Read block 1820 len 12288
30854 299/300 Read block 3033 len 20480
30855 300/300 Write block 4099 len 24576 data: 4 4 4 5 4 4
30856 301/300 Read block 1852 len 12288
30857 Main task has work to do, go find it
30858 got dsc Start(1)
30859 start 1
30860 Feb 25 17:25:01.211 INFO request completed, latency_us: 247, response_code: 204, uri: /start/cid/1, method: POST, req_id: dcf36bf8-ee3c-4a6c-85ae-fd7aad8890a9, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
30861 Replay: started 1, returned:Ok(())
30862 ----------------------------------------------------------------
30863 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30864 States: Active Offline Active
30865 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30866 9152 Acked Read 2 Done Sent Done false
30867 9153 Acked Read 4 Done Sent Done false
30868 9154 Acked Read 3 Done Sent Done false
30869 9155 Acked Read 10 Done Sent Done false
30870 9156 Acked Read 6 Done Sent Done false
30871 9157 Acked Flush 0 Done Sent Done false
30872 9158 Acked Write 4 Done Sent Done false
30873 9159 Acked Read 4 Done Sent Done false
30874 9160 Acked Read 5 Done Sent Done false
30875 9161 Acked Write 6 Done Sent Done false
30876 9162 Acked Flush 0 Done Sent Done false
30877 9163 Acked Write 1 Done Sent Done false
30878 9164 Acked Read 2 Done Sent Done false
30879 9165 Acked Write 3 Done Sent Done false
30880 9166 Acked Read 10 Done Sent Done false
30881 9167 Acked Read 7 Done Sent Done false
30882 9168 Acked Read 6 Done Sent Done false
30883 9169 Acked Write 2 Done Sent Done false
30884 9170 Acked Write 10 Done Sent Done false
30885 9171 Acked Flush 0 Done Sent Done false
30886 9172 Acked Flush 0 Done Sent Done false
30887 9173 Acked Write 10 Done Sent Done false
30888 9174 Acked Read 9 Done Sent Done false
30889 9175 Acked Read 6 Done Sent Done false
30890 9176 Acked Write 6 Done Sent Done false
30891 9177 Acked Write 3 Done Sent Done false
30892 9178 Acked Write 2 Done Sent Done false
30893 9179 Acked Read 2 Done Sent Done false
30894 9180 Acked Read 10 Done Sent Done false
30895 9181 Acked Read 9 Done Sent Done false
30896 9182 Acked Read 5 Done Sent Done false
30897 9183 Acked Read 7 Done Sent Done false
30898 9184 Acked Write 9 Done Sent Done false
30899 9185 Acked Read 9 Done Sent Done false
30900 9186 Acked Write 5 Done Sent Done false
30901 9187 Acked Read 3 Done Sent Done false
30902 9188 Acked Write 3 Done Sent Done false
30903 9189 Acked Write 9 Done Sent Done false
30904 9190 Acked Write 9 Done Sent Done false
30905 9191 Acked Write 9 Done Sent Done false
30906 9192 Acked Read 5 Done Sent Done false
30907 9193 Acked Read 9 Done Sent Done false
30908 9194 Acked Read 3 Done Sent Done false
30909 9195 Acked Write 4 Done Sent Done false
30910 9196 Acked Write 2 Done Sent Done false
30911 9197 Acked Write 9 Done Sent Done false
30912 9198 Acked Read 9 Done Sent Done false
30913 9199 Acked Read 1 Done Sent Done false
30914 9200 Acked Write 6 Done Sent Done false
30915 9201 Acked Read 7 Done Sent Done false
30916 9202 Acked Write 8 Done Sent Done false
30917 9203 Acked Read 6 Done Sent Done false
30918 9204 Acked Write 3 Done Sent Done false
30919 9205 Acked Write 9 Done Sent Done false
30920 9206 Acked Read 1 Done Sent Done false
30921 9207 Acked Write 1 Done Sent Done false
30922 9208 Acked Read 7 Done Sent Done false
30923 9209 Acked Read 10 Done Sent Done false
30924 9210 Acked Read 6 Done Sent Done false
30925 9211 Acked Flush 0 Done Sent Done false
30926 9212 Acked Write 8 Done Sent Done false
30927 9213 Acked Read 1 Done Sent Done false
30928 9214 Acked Flush 0 Done Sent Done false
30929 9215 Acked Read 8 Done Sent Done false
30930 9216 Acked Read 9 Done Sent Done false
30931 9217 Acked Read 2 Done Sent Done false
30932 9218 Acked Write 10 Done Sent Done false
30933 9219 Acked Read 9 Done Sent Done false
30934 9220 Acked Read 2 Done Sent Done false
30935 9221 Acked Write 2 Done Sent Done false
30936 9222 Acked Read 5 Done Sent Done false
30937 9223 Acked Read 2 Done Sent Done false
30938 9224 Acked Write 8 Done Sent Done false
30939 9225 Acked Write 9 Done Sent Done false
30940 9226 Acked Flush 0 Done Sent Done false
30941 9227 Acked Write 8 Done Sent Done false
30942 9228 Acked Read 3 Done Sent Done false
30943 9229 Acked Write 9 Done Sent Done false
30944 9230 Acked Write 3 Done Sent Done false
30945 9231 Acked Read 5 Done Sent Done false
30946 9232 Acked Write 1 Done Sent Done false
30947 9233 Acked Read 8 Done Sent Done false
30948 9234 Acked Read 5 Done Sent Done false
30949 9235 Acked Write 2 Done Sent Done false
30950 9236 Acked Write 4 Done Sent Done false
30951 9237 Acked Write 4 Done Sent Done false
30952 9238 Acked Read 2 Done Sent Done false
30953 9239 Acked Write 10 Done Sent Done false
30954 9240 Acked Read 4 Done Sent Done false
30955 9241 Acked Flush 0 Done Sent Done false
30956 9242 Acked Write 6 Done Sent Done false
30957 9243 Acked Read 6 Done Sent Done false
30958 9244 Acked Flush 0 Done Sent Done false
30959 9245 Acked Flush 0 Done Sent Done false
30960 9246 Acked Read 5 Done Sent Done false
30961 9247 Acked Read 10 Done Sent Done false
30962 9248 Acked Write 2 Done Sent Done false
30963 9249 Acked Read 5 Done Sent Done false
30964 9250 Acked Write 6 Done Sent Done false
30965 9251 Acked Write 7 Done Sent Done false
30966 9252 Acked Write 4 Done Sent Done false
30967 9253 Acked Read 8 Done Sent Done false
30968 9254 Acked Read 2 Done Sent Done false
30969 9255 Acked Read 6 Done Sent Done false
30970 9256 Acked Flush 0 Done Sent Done false
30971 9257 Acked Write 6 Done Sent Done false
30972 9258 Acked Read 5 Done Sent Done false
30973 9259 Acked Read 5 Done Sent Done false
30974 9260 Acked Read 9 Done Sent Done false
30975 9261 Acked Read 8 Done Sent Done false
30976 9262 Acked Write 8 Done Sent Done false
30977 9263 Acked Write 3 Done Sent Done false
30978 9264 Acked Write 8 Done Sent Done false
30979 9265 Acked Read 9 Done Sent Done false
30980 9266 Acked Write 10 Done Sent Done false
30981 9267 Acked Read 9 Done Sent Done false
30982 9268 Acked Read 6 Done Sent Done false
30983 9269 Acked Read 2 Done Sent Done false
30984 9270 Acked Read 3 Done Sent Done false
30985 9271 Acked Read 2 Done Sent Done false
30986 9272 Acked Read 10 Done Sent Done false
30987 9273 Acked Write 5 Done Sent Done false
30988 9274 Acked Write 10 Done Sent Done false
30989 9275 Acked Read 8 Done Sent Done false
30990 9276 Acked Read 1 Done Sent Done false
30991 9277 Acked Write 7 Done Sent Done false
30992 9278 Acked Read 1 Done Sent Done false
30993 9279 Acked Read 3 Done Sent Done false
30994 9280 Acked Write 7 Done Sent Done false
30995 9281 Acked Read 6 Done Sent Done false
30996 9282 Acked Read 3 Done Sent Done false
30997 9283 Acked Read 2 Done Sent Done false
30998 9284 Acked Read 7 Done Sent Done false
30999 9285 Acked Read 7 Done Sent Done false
31000 9286 Acked Read 5 Done Sent Done false
31001 9287 Acked Write 9 Done Sent Done false
31002 9288 Acked Write 10 Done Sent Done false
31003 9289 Acked Read 7 Done Sent Done false
31004 9290 Acked Read 3 Done Sent Done false
31005 9291 Acked Write 10 Done Sent Done false
31006 9292 Acked Flush 0 Done Sent Done false
31007 9293 Acked Read 10 Done Sent Done false
31008 9294 Acked Flush 0 Done Sent Done false
31009 9295 Acked Read 8 Done Sent Done false
31010 9296 Acked Read 8 Done Sent Done false
31011 9297 Acked Read 4 Done Sent Done false
31012 9298 Acked Write 4 Done Sent Done false
31013 9299 Acked Write 10 Done Sent Done false
31014 9300 Acked Write 1 Done Sent Done false
31015 9301 Acked Write 7 Done Sent Done false
31016 9302 Acked Write 10 Done Sent Done false
31017 9303 Acked Read 7 Done Sent Done false
31018 9304 Acked Read 5 Done Sent Done false
31019 9305 Acked Read 2 Done Sent Done false
31020 9306 Acked Read 3 Done Sent Done false
31021 9307 Acked Write 3 Done Sent Done false
31022 9308 Acked Read 8 Done Sent Done false
31023 9309 Acked Write 8 Done Sent Done false
31024 9310 Acked Flush 0 Done Sent Done false
31025 9311 Acked Read 10 Done Sent Done false
31026 9312 Acked Read 6 Done Sent Done false
31027 9313 Acked Write 8 Done Sent Done false
31028 9314 Acked Read 3 Done Sent Done false
31029 9315 Acked Flush 0 Done Sent Done false
31030 9316 Acked Write 3 Done Sent Done false
31031 9317 Acked Read 9 Done Sent Done false
31032 9318 Acked Write 5 Done Sent Done false
31033 9319 Acked Read 1 Done Sent Done false
31034 9320 Acked Write 2 Done Sent Done false
31035 9321 Acked Write 3 Done Sent Done false
31036 9322 Acked Write 2 Done Sent Done false
31037 9323 Acked Write 9 Done Sent Done false
31038 9324 Acked Write 9 Done Sent Done false
31039 9325 Acked Read 9 Done Sent Done false
31040 9326 Acked Read 1 Done Sent Done false
31041 9327 Acked Read 2 Done Sent Done false
31042 9328 Acked Read 2 Done Sent Done false
31043 9329 Acked Read 8 Done Sent Done false
31044 9330 Acked Read 1 Done Sent Done false
31045 9331 Acked Read 4 Done Sent Done false
31046 9332 Acked Write 8 Done Sent Done false
31047 9333 Acked Write 5 Done Sent Done false
31048 9334 Acked Read 7 Done Sent Done false
31049 9335 Acked Write 5 Done Sent Done false
31050 9336 Acked Write 3 Done Sent Done false
31051 9337 Acked Flush 0 Done Sent Done false
31052 9338 Acked Write 1 Done Sent Done false
31053 9339 Acked Write 2 Done Sent Done false
31054 9340 Acked Write 9 Done Sent Done false
31055 9341 Acked Write 5 Done Sent Done false
31056 9342 Acked Read 4 Done Sent Done false
31057 9343 Acked Flush 0 Done Sent Done false
31058 9344 Acked Read 10 Done Sent Done false
31059 9345 Acked Write 5 Done Sent Done false
31060 9346 Acked Read 5 Done Sent Done false
31061 9347 Acked Write 3 Done Sent Done false
31062 9348 Acked Read 5 Done Sent Done false
31063 9349 Acked Read 8 Done Sent Done false
31064 9350 Acked Read 6 Done Sent Done false
31065 9351 Acked Read 1 Done Sent Done false
31066 9352 Acked Read 3 Done Sent Done false
31067 9353 Acked Write 5 Done Sent Done false
31068 9354 Acked Write 6 Done Sent Done false
31069 9355 Acked Write 2 Done Sent Done false
31070 9356 Acked Write 9 Done Sent Done false
31071 9357 Acked Read 3 Done Sent Done false
31072 9358 Acked Read 1 Done Sent Done false
31073 9359 Acked Write 7 Done Sent Done false
31074 9360 Acked Read 7 Done Sent Done false
31075 9361 Acked Read 7 Done Sent Done false
31076 9362 Acked Write 4 Done Sent Done false
31077 9363 Acked Write 9 Done Sent Done false
31078 9364 Acked Read 8 Done Sent Done false
31079 9365 Acked Read 2 Done Sent Done false
31080 9366 Acked Write 3 Done Sent Done false
31081 9367 Acked Read 7 Done Sent Done false
31082 9368 Acked Read 2 Done Sent Done false
31083 9369 Acked Read 10 Done Sent Done false
31084 9370 Acked Read 3 Done Sent Done false
31085 9371 Acked Write 5 Done Sent Done false
31086 9372 Acked Write 9 Done Sent Done false
31087 9373 Acked Write 8 Done Sent Done false
31088 9374 Acked Write 5 Done Sent Done false
31089 9375 Acked Read 4 Done Sent Done false
31090 9376 Acked Read 6 Done Sent Done false
31091 9377 Acked Write 2 Done Sent Done false
31092 9378 Acked Read 3 Done Sent Done false
31093 9379 Acked Write 7 Done Sent Done false
31094 9380 Acked Write 7 Done Sent Done false
31095 9381 Acked Read 10 Done Sent Done false
31096 9382 Acked Read 9 Done Sent Done false
31097 9383 Acked Read 4 Done Sent Done false
31098 9384 Acked Read 5 Done Sent Done false
31099 9385 Acked Write 7 Done Sent Done false
31100 9386 Acked Read 9 Done Sent Done false
31101 9387 Acked Flush 0 Done Sent Done false
31102 9388 Acked Read 4 Done Sent Done false
31103 9389 Acked Write 2 Done Sent Done false
31104 9390 Acked Read 10 Done Sent Done false
31105 9391 Acked Write 1 Done Sent Done false
31106 9392 Acked Flush 0 Done Sent Done false
31107 9393 Acked Read 4 Done Sent Done false
31108 9394 Acked Read 1 Done Sent Done false
31109 9395 Acked Read 6 Done Sent Done false
31110 9396 Acked Write 10 Done Sent Done false
31111 9397 Acked Flush 0 Done Sent Done false
31112 9398 Acked Read 7 Done Sent Done false
31113 9399 Acked Read 2 Done Sent Done false
31114 9400 Acked Write 3 Done Sent Done false
31115 9401 Acked Flush 0 Done Sent Done false
31116 9402 Acked Read 6 Done Sent Done false
31117 9403 Acked Read 6 Done Sent Done false
31118 9404 Acked Write 10 Done Sent Done false
31119 9405 Acked Write 1 Done Sent Done false
31120 9406 Acked Write 6 Done Sent Done false
31121 9407 Acked Write 5 Done Sent Done false
31122 9408 Acked Read 7 Done Sent Done false
31123 9409 Acked Read 8 Done Sent Done false
31124 9410 Acked Write 9 Done Sent Done false
31125 9411 Acked Flush 0 Done Sent Done false
31126 9412 Acked Write 1 Done Sent Done false
31127 9413 Acked Write 8 Done Sent Done false
31128 9414 Acked Read 9 Done Sent Done false
31129 9415 Acked Read 2 Done Sent Done false
31130 9416 Acked Read 3 Done Sent Done false
31131 9417 Acked Write 4 Done Sent Done false
31132 9418 Acked Read 10 Done Sent Done false
31133 9419 Acked Read 7 Done Sent Done false
31134 9420 Acked Read 3 Done Sent Done false
31135 9421 Acked Write 1 Done Sent Done false
31136 9422 Acked Read 7 Done Sent Done false
31137 9423 Acked Read 6 Done Sent Done false
31138 9424 Acked Read 9 Done Sent Done false
31139 9425 Acked Read 1 Done Sent Done false
31140 9426 Acked Read 3 Done Sent Done false
31141 9427 Acked Write 7 Done Sent Done false
31142 9428 Acked Read 7 Done Sent Done false
31143 9429 Acked Read 9 Done Sent Done false
31144 9430 Acked Read 4 Done Sent Done false
31145 9431 Acked Write 8 Done Sent Done false
31146 9432 Acked Read 2 Done Sent Done false
31147 9433 Acked Read 9 Done Sent Done false
31148 9434 Acked Read 9 Done Sent Done false
31149 9435 Acked Write 7 Done Sent Done false
31150 9436 Acked Flush 0 Done Sent Done false
31151 9437 Acked Write 4 Done Sent Done false
31152 9438 Acked Write 9 Done Sent Done false
31153 9439 Acked Read 10 Done Sent Done false
31154 9440 Acked Flush 0 Done Sent Done false
31155 9441 Acked Write 2 Done Sent Done false
31156 9442 Acked Write 3 Done Sent Done false
31157 9443 Acked Write 10 Done Sent Done false
31158 9444 Acked Read 10 Done Sent Done false
31159 9445 Acked Read 9 Done Sent Done false
31160 9446 Acked Read 4 Done Sent Done false
31161 9447 Acked Write 5 Done Sent Done false
31162 9448 Acked Read 3 Done Sent Done false
31163 9449 Acked Read 3 Done Sent Done false
31164 9450 Acked Read 5 Done Sent Done false
31165 9451 Acked Write 6 Done Sent Done false
31166 9452 Acked Read 3 Done Sent Done false
31167 STATES DS:0 DS:1 DS:2 TOTAL
31168 Sent 0 301 0 301
31169 Done 301 0 301 602
31170 Skipped 0 0 0 0
31171 Error 0 0 0 0
31172 Last Flush: Some(JobId(9440)) Some(JobId(9151)) Some(JobId(9440))
31173 Downstairs last five completed: 9151 9150 9149 9148 9147
31174 Upstairs last five completed: 9452 9451 9450 9449 9448
31175 CLIENT: Up:0 ds:301 act:2
311762025-02-25T17:25:01.904ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
311772025-02-25T17:25:02.928ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31178 [8820] got start action
31179 [8820] I am going to restart
31180 Starting downstairs at port 8820
31181 Make output file at "/tmp/dsc/downstairs-8820.txt"
31182 [8820][1] reports Starting
31183 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1236)
31184 [8820][1] reports Running
311852025-02-25T17:25:03.907ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
311862025-02-25T17:25:05.184ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31187 ----------------------------------------------------------------
31188 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31189 States: Active Offline Active
31190 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31191 9152 Acked Read 2 Done Sent Done false
31192 9153 Acked Read 4 Done Sent Done false
31193 9154 Acked Read 3 Done Sent Done false
31194 9155 Acked Read 10 Done Sent Done false
31195 9156 Acked Read 6 Done Sent Done false
31196 9157 Acked Flush 0 Done Sent Done false
31197 9158 Acked Write 4 Done Sent Done false
31198 9159 Acked Read 4 Done Sent Done false
31199 9160 Acked Read 5 Done Sent Done false
31200 9161 Acked Write 6 Done Sent Done false
31201 9162 Acked Flush 0 Done Sent Done false
31202 9163 Acked Write 1 Done Sent Done false
31203 9164 Acked Read 2 Done Sent Done false
31204 9165 Acked Write 3 Done Sent Done false
31205 9166 Acked Read 10 Done Sent Done false
31206 9167 Acked Read 7 Done Sent Done false
31207 9168 Acked Read 6 Done Sent Done false
31208 9169 Acked Write 2 Done Sent Done false
31209 9170 Acked Write 10 Done Sent Done false
31210 9171 Acked Flush 0 Done Sent Done false
31211 9172 Acked Flush 0 Done Sent Done false
31212 9173 Acked Write 10 Done Sent Done false
31213 9174 Acked Read 9 Done Sent Done false
31214 9175 Acked Read 6 Done Sent Done false
31215 9176 Acked Write 6 Done Sent Done false
31216 9177 Acked Write 3 Done Sent Done false
31217 9178 Acked Write 2 Done Sent Done false
31218 9179 Acked Read 2 Done Sent Done false
31219 9180 Acked Read 10 Done Sent Done false
31220 9181 Acked Read 9 Done Sent Done false
31221 9182 Acked Read 5 Done Sent Done false
31222 9183 Acked Read 7 Done Sent Done false
31223 9184 Acked Write 9 Done Sent Done false
31224 9185 Acked Read 9 Done Sent Done false
31225 9186 Acked Write 5 Done Sent Done false
31226 9187 Acked Read 3 Done Sent Done false
31227 9188 Acked Write 3 Done Sent Done false
31228 9189 Acked Write 9 Done Sent Done false
31229 9190 Acked Write 9 Done Sent Done false
31230 9191 Acked Write 9 Done Sent Done false
31231 9192 Acked Read 5 Done Sent Done false
31232 9193 Acked Read 9 Done Sent Done false
31233 9194 Acked Read 3 Done Sent Done false
31234 9195 Acked Write 4 Done Sent Done false
31235 9196 Acked Write 2 Done Sent Done false
31236 9197 Acked Write 9 Done Sent Done false
31237 9198 Acked Read 9 Done Sent Done false
31238 9199 Acked Read 1 Done Sent Done false
31239 9200 Acked Write 6 Done Sent Done false
31240 9201 Acked Read 7 Done Sent Done false
31241 9202 Acked Write 8 Done Sent Done false
31242 9203 Acked Read 6 Done Sent Done false
31243 9204 Acked Write 3 Done Sent Done false
31244 9205 Acked Write 9 Done Sent Done false
31245 9206 Acked Read 1 Done Sent Done false
31246 9207 Acked Write 1 Done Sent Done false
31247 9208 Acked Read 7 Done Sent Done false
31248 9209 Acked Read 10 Done Sent Done false
31249 9210 Acked Read 6 Done Sent Done false
31250 9211 Acked Flush 0 Done Sent Done false
31251 9212 Acked Write 8 Done Sent Done false
31252 9213 Acked Read 1 Done Sent Done false
31253 9214 Acked Flush 0 Done Sent Done false
31254 9215 Acked Read 8 Done Sent Done false
31255 9216 Acked Read 9 Done Sent Done false
31256 9217 Acked Read 2 Done Sent Done false
31257 9218 Acked Write 10 Done Sent Done false
31258 9219 Acked Read 9 Done Sent Done false
31259 9220 Acked Read 2 Done Sent Done false
31260 9221 Acked Write 2 Done Sent Done false
31261 9222 Acked Read 5 Done Sent Done false
31262 9223 Acked Read 2 Done Sent Done false
31263 9224 Acked Write 8 Done Sent Done false
31264 9225 Acked Write 9 Done Sent Done false
31265 9226 Acked Flush 0 Done Sent Done false
31266 9227 Acked Write 8 Done Sent Done false
31267 9228 Acked Read 3 Done Sent Done false
31268 9229 Acked Write 9 Done Sent Done false
31269 9230 Acked Write 3 Done Sent Done false
31270 9231 Acked Read 5 Done Sent Done false
31271 9232 Acked Write 1 Done Sent Done false
31272 9233 Acked Read 8 Done Sent Done false
31273 9234 Acked Read 5 Done Sent Done false
31274 9235 Acked Write 2 Done Sent Done false
31275 9236 Acked Write 4 Done Sent Done false
31276 9237 Acked Write 4 Done Sent Done false
31277 9238 Acked Read 2 Done Sent Done false
31278 9239 Acked Write 10 Done Sent Done false
31279 9240 Acked Read 4 Done Sent Done false
31280 9241 Acked Flush 0 Done Sent Done false
31281 9242 Acked Write 6 Done Sent Done false
31282 9243 Acked Read 6 Done Sent Done false
31283 9244 Acked Flush 0 Done Sent Done false
31284 9245 Acked Flush 0 Done Sent Done false
31285 9246 Acked Read 5 Done Sent Done false
31286 9247 Acked Read 10 Done Sent Done false
31287 9248 Acked Write 2 Done Sent Done false
31288 9249 Acked Read 5 Done Sent Done false
31289 9250 Acked Write 6 Done Sent Done false
31290 9251 Acked Write 7 Done Sent Done false
31291 9252 Acked Write 4 Done Sent Done false
31292 9253 Acked Read 8 Done Sent Done false
31293 9254 Acked Read 2 Done Sent Done false
31294 9255 Acked Read 6 Done Sent Done false
31295 9256 Acked Flush 0 Done Sent Done false
31296 9257 Acked Write 6 Done Sent Done false
31297 9258 Acked Read 5 Done Sent Done false
31298 9259 Acked Read 5 Done Sent Done false
31299 9260 Acked Read 9 Done Sent Done false
31300 9261 Acked Read 8 Done Sent Done false
31301 9262 Acked Write 8 Done Sent Done false
31302 9263 Acked Write 3 Done Sent Done false
31303 9264 Acked Write 8 Done Sent Done false
31304 9265 Acked Read 9 Done Sent Done false
31305 9266 Acked Write 10 Done Sent Done false
31306 9267 Acked Read 9 Done Sent Done false
31307 9268 Acked Read 6 Done Sent Done false
31308 9269 Acked Read 2 Done Sent Done false
31309 9270 Acked Read 3 Done Sent Done false
31310 9271 Acked Read 2 Done Sent Done false
31311 9272 Acked Read 10 Done Sent Done false
31312 9273 Acked Write 5 Done Sent Done false
31313 9274 Acked Write 10 Done Sent Done false
31314 9275 Acked Read 8 Done Sent Done false
31315 9276 Acked Read 1 Done Sent Done false
31316 9277 Acked Write 7 Done Sent Done false
31317 9278 Acked Read 1 Done Sent Done false
31318 9279 Acked Read 3 Done Sent Done false
31319 9280 Acked Write 7 Done Sent Done false
31320 9281 Acked Read 6 Done Sent Done false
31321 9282 Acked Read 3 Done Sent Done false
31322 9283 Acked Read 2 Done Sent Done false
31323 9284 Acked Read 7 Done Sent Done false
31324 9285 Acked Read 7 Done Sent Done false
31325 9286 Acked Read 5 Done Sent Done false
31326 9287 Acked Write 9 Done Sent Done false
31327 9288 Acked Write 10 Done Sent Done false
31328 9289 Acked Read 7 Done Sent Done false
31329 9290 Acked Read 3 Done Sent Done false
31330 9291 Acked Write 10 Done Sent Done false
31331 9292 Acked Flush 0 Done Sent Done false
31332 9293 Acked Read 10 Done Sent Done false
31333 9294 Acked Flush 0 Done Sent Done false
31334 9295 Acked Read 8 Done Sent Done false
31335 9296 Acked Read 8 Done Sent Done false
31336 9297 Acked Read 4 Done Sent Done false
31337 9298 Acked Write 4 Done Sent Done false
31338 9299 Acked Write 10 Done Sent Done false
31339 9300 Acked Write 1 Done Sent Done false
31340 9301 Acked Write 7 Done Sent Done false
31341 9302 Acked Write 10 Done Sent Done false
31342 9303 Acked Read 7 Done Sent Done false
31343 9304 Acked Read 5 Done Sent Done false
31344 9305 Acked Read 2 Done Sent Done false
31345 9306 Acked Read 3 Done Sent Done false
31346 9307 Acked Write 3 Done Sent Done false
31347 9308 Acked Read 8 Done Sent Done false
31348 9309 Acked Write 8 Done Sent Done false
31349 9310 Acked Flush 0 Done Sent Done false
31350 9311 Acked Read 10 Done Sent Done false
31351 9312 Acked Read 6 Done Sent Done false
31352 9313 Acked Write 8 Done Sent Done false
31353 9314 Acked Read 3 Done Sent Done false
31354 9315 Acked Flush 0 Done Sent Done false
31355 9316 Acked Write 3 Done Sent Done false
31356 9317 Acked Read 9 Done Sent Done false
31357 9318 Acked Write 5 Done Sent Done false
31358 9319 Acked Read 1 Done Sent Done false
31359 9320 Acked Write 2 Done Sent Done false
31360 9321 Acked Write 3 Done Sent Done false
31361 9322 Acked Write 2 Done Sent Done false
31362 9323 Acked Write 9 Done Sent Done false
31363 9324 Acked Write 9 Done Sent Done false
31364 9325 Acked Read 9 Done Sent Done false
31365 9326 Acked Read 1 Done Sent Done false
31366 9327 Acked Read 2 Done Sent Done false
31367 9328 Acked Read 2 Done Sent Done false
31368 9329 Acked Read 8 Done Sent Done false
31369 9330 Acked Read 1 Done Sent Done false
31370 9331 Acked Read 4 Done Sent Done false
31371 9332 Acked Write 8 Done Sent Done false
31372 9333 Acked Write 5 Done Sent Done false
31373 9334 Acked Read 7 Done Sent Done false
31374 9335 Acked Write 5 Done Sent Done false
31375 9336 Acked Write 3 Done Sent Done false
31376 9337 Acked Flush 0 Done Sent Done false
31377 9338 Acked Write 1 Done Sent Done false
31378 9339 Acked Write 2 Done Sent Done false
31379 9340 Acked Write 9 Done Sent Done false
31380 9341 Acked Write 5 Done Sent Done false
31381 9342 Acked Read 4 Done Sent Done false
31382 9343 Acked Flush 0 Done Sent Done false
31383 9344 Acked Read 10 Done Sent Done false
31384 9345 Acked Write 5 Done Sent Done false
31385 9346 Acked Read 5 Done Sent Done false
31386 9347 Acked Write 3 Done Sent Done false
31387 9348 Acked Read 5 Done Sent Done false
31388 9349 Acked Read 8 Done Sent Done false
31389 9350 Acked Read 6 Done Sent Done false
31390 9351 Acked Read 1 Done Sent Done false
31391 9352 Acked Read 3 Done Sent Done false
31392 9353 Acked Write 5 Done Sent Done false
31393 9354 Acked Write 6 Done Sent Done false
31394 9355 Acked Write 2 Done Sent Done false
31395 9356 Acked Write 9 Done Sent Done false
31396 9357 Acked Read 3 Done Sent Done false
31397 9358 Acked Read 1 Done Sent Done false
31398 9359 Acked Write 7 Done Sent Done false
31399 9360 Acked Read 7 Done Sent Done false
31400 9361 Acked Read 7 Done Sent Done false
31401 9362 Acked Write 4 Done Sent Done false
31402 9363 Acked Write 9 Done Sent Done false
31403 9364 Acked Read 8 Done Sent Done false
31404 9365 Acked Read 2 Done Sent Done false
31405 9366 Acked Write 3 Done Sent Done false
31406 9367 Acked Read 7 Done Sent Done false
31407 9368 Acked Read 2 Done Sent Done false
31408 9369 Acked Read 10 Done Sent Done false
31409 9370 Acked Read 3 Done Sent Done false
31410 9371 Acked Write 5 Done Sent Done false
31411 9372 Acked Write 9 Done Sent Done false
31412 9373 Acked Write 8 Done Sent Done false
31413 9374 Acked Write 5 Done Sent Done false
31414 9375 Acked Read 4 Done Sent Done false
31415 9376 Acked Read 6 Done Sent Done false
31416 9377 Acked Write 2 Done Sent Done false
31417 9378 Acked Read 3 Done Sent Done false
31418 9379 Acked Write 7 Done Sent Done false
31419 9380 Acked Write 7 Done Sent Done false
31420 9381 Acked Read 10 Done Sent Done false
31421 9382 Acked Read 9 Done Sent Done false
31422 9383 Acked Read 4 Done Sent Done false
31423 9384 Acked Read 5 Done Sent Done false
31424 9385 Acked Write 7 Done Sent Done false
31425 9386 Acked Read 9 Done Sent Done false
31426 9387 Acked Flush 0 Done Sent Done false
31427 9388 Acked Read 4 Done Sent Done false
31428 9389 Acked Write 2 Done Sent Done false
31429 9390 Acked Read 10 Done Sent Done false
31430 9391 Acked Write 1 Done Sent Done false
31431 9392 Acked Flush 0 Done Sent Done false
31432 9393 Acked Read 4 Done Sent Done false
31433 9394 Acked Read 1 Done Sent Done false
31434 9395 Acked Read 6 Done Sent Done false
31435 9396 Acked Write 10 Done Sent Done false
31436 9397 Acked Flush 0 Done Sent Done false
31437 9398 Acked Read 7 Done Sent Done false
31438 9399 Acked Read 2 Done Sent Done false
31439 9400 Acked Write 3 Done Sent Done false
31440 9401 Acked Flush 0 Done Sent Done false
31441 9402 Acked Read 6 Done Sent Done false
31442 9403 Acked Read 6 Done Sent Done false
31443 9404 Acked Write 10 Done Sent Done false
31444 9405 Acked Write 1 Done Sent Done false
31445 9406 Acked Write 6 Done Sent Done false
31446 9407 Acked Write 5 Done Sent Done false
31447 9408 Acked Read 7 Done Sent Done false
31448 9409 Acked Read 8 Done Sent Done false
31449 9410 Acked Write 9 Done Sent Done false
31450 9411 Acked Flush 0 Done Sent Done false
31451 9412 Acked Write 1 Done Sent Done false
31452 9413 Acked Write 8 Done Sent Done false
31453 9414 Acked Read 9 Done Sent Done false
31454 9415 Acked Read 2 Done Sent Done false
31455 9416 Acked Read 3 Done Sent Done false
31456 9417 Acked Write 4 Done Sent Done false
31457 9418 Acked Read 10 Done Sent Done false
31458 9419 Acked Read 7 Done Sent Done false
31459 9420 Acked Read 3 Done Sent Done false
31460 9421 Acked Write 1 Done Sent Done false
31461 9422 Acked Read 7 Done Sent Done false
31462 9423 Acked Read 6 Done Sent Done false
31463 9424 Acked Read 9 Done Sent Done false
31464 9425 Acked Read 1 Done Sent Done false
31465 9426 Acked Read 3 Done Sent Done false
31466 9427 Acked Write 7 Done Sent Done false
31467 9428 Acked Read 7 Done Sent Done false
31468 9429 Acked Read 9 Done Sent Done false
31469 9430 Acked Read 4 Done Sent Done false
31470 9431 Acked Write 8 Done Sent Done false
31471 9432 Acked Read 2 Done Sent Done false
31472 9433 Acked Read 9 Done Sent Done false
31473 9434 Acked Read 9 Done Sent Done false
31474 9435 Acked Write 7 Done Sent Done false
31475 9436 Acked Flush 0 Done Sent Done false
31476 9437 Acked Write 4 Done Sent Done false
31477 9438 Acked Write 9 Done Sent Done false
31478 9439 Acked Read 10 Done Sent Done false
31479 9440 Acked Flush 0 Done Sent Done false
31480 9441 Acked Write 2 Done Sent Done false
31481 9442 Acked Write 3 Done Sent Done false
31482 9443 Acked Write 10 Done Sent Done false
31483 9444 Acked Read 10 Done Sent Done false
31484 9445 Acked Read 9 Done Sent Done false
31485 9446 Acked Read 4 Done Sent Done false
31486 9447 Acked Write 5 Done Sent Done false
31487 9448 Acked Read 3 Done Sent Done false
31488 9449 Acked Read 3 Done Sent Done false
31489 9450 Acked Read 5 Done Sent Done false
31490 9451 Acked Write 6 Done Sent Done false
31491 9452 Acked Read 3 Done Sent Done false
31492 STATES DS:0 DS:1 DS:2 TOTAL
31493 Sent 0 301 0 301
31494 Done 301 0 301 602
31495 Skipped 0 0 0 0
31496 Error 0 0 0 0
31497 Last Flush: Some(JobId(9440)) Some(JobId(9151)) Some(JobId(9440))
31498 Downstairs last five completed: 9151 9150 9149 9148 9147
31499 Upstairs last five completed: 9452 9451 9450 9449 9448
31500 CLIENT: Up:0 ds:301 act:2
315012025-02-25T17:25:05.910ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315022025-02-25T17:25:06.910ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315032025-02-25T17:25:06.952ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315042025-02-25T17:25:06.952ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:57552) = io task client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315052025-02-25T17:25:06.953ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315062025-02-25T17:25:06.953ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315072025-02-25T17:25:06.953ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315082025-02-25T17:25:06.954ZINFOcrucible: send last flush ID to this DS: Some(JobId(9151)) = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315092025-02-25T17:25:06.954ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315102025-02-25T17:25:06.954ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(9151)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315112025-02-25T17:25:06.954ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(9151)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31512 ----------------------------------------------------------------
31513 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31514 Downstairs last five completed: 9453 9452 9451 9450 9449
31515 Upstairs last five completed: 9453 9452 9451 9450 9449
31516 CLIENT: Up:0 ds:0 act:3
31517 Replay: All jobs finished
31518 Main task has work to do, go find it
31519 got dsc Stop(2)
31520 stop 2
31521 Feb 25 17:25:09.216 INFO request completed, latency_us[8830] Got stop action so:false kr:false
31522 : 392, response_code: 204, uri: /stop/cid/2, method: POST, req_id: ea329ccb-41eb-4d14-a988-1fe6a1a23679, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
31523 Feb 25 17:25:09.217 INFO request completed, latency_us: 110, response_code: 200, uri: /state/cid/2, method: GET, req_id: 617f5697-ce0f-41ed-97b5-a18eba14ff6a, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
315242025-02-25T17:25:09.217ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315252025-02-25T17:25:09.217ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315262025-02-25T17:25:09.217ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
315272025-02-25T17:25:09.217ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
31528 [8830] Exited with: ExitStatus(unix_wait_status(9))
31529 [8830][2] reports Exit
31530 Feb 25 17:25:13.219 INFO request completed, latency_us: 220, response_code: 200, uri: /state/cid/2, method: GET, req_id: d37c9c8c-b610-4542-add7-b69aa4ffd7b3, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
31531 001/300 Read block 3858 len 12288
31532 002/300 Read block 3918 len 16384
31533 003/300 Read block 1993 len 4096
31534 004/300 Read block 4484 len 12288
31535 005/300 Read block 803 len 36864
31536 006/300 Flush
31537 007/300 Write block 60 len 16384 data: 3 3 2 3
31538 008/300 Read block 374 len 32768
31539 009/300 Read block 2879 len 20480
31540 010/300 Write block 4287 len 32768 data: 6 7 8 8 6 4 4 6
31541 011/300 Read block 4279 len 4096
31542 012/300 Write block 3591 len 40960 data: 3 4 4 4 4 3 3 3 3 4
31543 013/300 Write block 4912 len 36864 data: 6 5 6 5 6 5 5 4 2
31544 014/300 Read block 431 len 20480
31545 015/300 Read block 2146 len 4096
31546 016/300 Read block 2607 len 20480
31547 017/300 Write block 4031 len 28672 data: 3 5 4 3 2 2 2
31548 018/300 Write block 3 len 4096 data: 3
31549 019/300 Write block 3238 len 8192 data: 6 6
31550 020/300 Read block 1498 len 12288
31551 021/300 Read block 3053 len 16384
31552 022/300 Read block 1111 len 40960
31553 023/300 Read block 784 len 20480
31554 024/300 Write block 3493 len 8192 data: 4 4
31555 025/300 Flush
31556 026/300 Read block 509 len 40960
31557 027/300 Write block 3190 len 24576 data: 2 2 4 5 5 5
31558 028/300 Flush
31559 029/300 Write block 2645 len 32768 data: 3 2 3 3 3 3 4 2
31560 030/300 Write block 3442 len 32768 data: 3 3 3 3 6 5 5 6
31561 031/300 Write block 3050 len 36864 data: 7 7 5 8 10 9 8 9 9
31562 032/300 Write block 4558 len 24576 data: 4 5 6 6 6 6
31563 033/300 Read block 2721 len 20480
31564 034/300 Read block 2648 len 28672
31565 035/300 Read block 4543 len 36864
31566 036/300 Read block 2910 len 36864
31567 037/300 Read block 739 len 24576
31568 038/300 Read block 3769 len 8192
31569 039/300 Read block 2500 len 32768
31570 040/300 Write block 3344 len 8192 data: 5 5
31571 041/300 Write block 2755 len 8192 data: 6 5
31572 042/300 Read block 3466 len 4096
31573 043/300 Write block 713 len 12288 data: 6 6 6
31574 044/300 Read block 4662 len 20480
31575 045/300 Read block 4355 len 32768
31576 046/300 Read block 2540 len 32768
31577 047/300 Write block 1991 len 28672 data: 6 5 4 6 6 6 5
31578 048/300 Write block 2349 len 32768 data: 7 7 8 9 8 5 6 5
31579 049/300 Read block 1303 len 28672
31580 050/300 Write block 4663 len 24576 data: 10 9 8 9 5 5
31581 051/300 Write block 2209 len 24576 data: 6 7 9 7 9 9
31582 052/300 Write block 3694 len 36864 data: 3 3 4 4 6 7 8 9 9
31583 053/300 Read block 4921 len 4096
31584 054/300 Write block 4873 len 16384 data: 6 6 6 5
31585 055/300 Read block 235 len 12288
31586 056/300 Write block 2943 len 40960 data: 9 8 8 9 9 9 9 9 9 10
31587 057/300 Read block 4893 len 4096
31588 058/300 Read block 2398 len 16384
31589 059/300 Read block 2024 len 32768
31590 060/300 Read block 3653 len 8192
31591 061/300 Write block 4689 len 8192 data: 3 4
31592 062/300 Flush
31593 063/300 Write block 4307 len 24576 data: 4 5 6 6 4 4
31594 064/300 Flush
31595 065/300 Read block 3181 len 20480
31596 066/300 Read block 8 len 4096
31597 067/300 Read block 3062 len 40960
31598 068/300 Read block 2372 len 16384
31599 069/300 Read block 4508 len 40960
31600 070/300 Write block 1897 len 16384 data: 5 5 6 6
31601 071/300 Write block 3876 len 4096 data: 7
31602 072/300 Read block 3906 len 32768
31603 073/300 Write block 3514 len 40960 data: 7 8 6 6 6 6 6 6 5 5
31604 074/300 Flush
31605 075/300 Write block 4992 len 28672 data: 8 8 8 8 4 3 3
31606 076/300 Read block 2582 len 36864
31607 077/300 Read block 1902 len 28672
31608 078/300 Write block 3558 len 24576 data: 4 6 6 7 6 6
31609 079/300 Write block 1496 len 32768 data: 6 6 5 5 5 6 6 7
31610 080/300 Read block 444 len 24576
31611 081/300 Write block 2598 len 8192 data: 6 7
31612 082/300 Read block 4832 len 16384
31613 083/300 Write block 1939 len 16384 data: 10 10 8 8
31614 084/300 Write block 2352 len 20480 data: 10 9 6 7 6
31615 085/300 Write block 4888 len 12288 data: 4 5 5
31616 086/300 Read block 2309 len 28672
31617 087/300 Read block 2352 len 20480
31618 088/300 Read block 2273 len 8192
31619 089/300 Write block 1401 len 28672 data: 7 6 4 4 6 5 7
31620 090/300 Read block 4075 len 32768
31621 091/300 Read block 103 len 16384
31622 092/300 Read block 3485 len 36864
31623 093/300 Read block 2633 len 8192
31624 094/300 Read block 4703 len 24576
31625 095/300 Write block 36 len 24576 data: 6 6 6 5 6 5
31626 096/300 Write block 914 len 28672 data: 6 4 5 5 4 6 6
31627 097/300 Read block 120 len 20480
31628 098/300 Read block 722 len 24576
31629 099/300 Read block 3712 len 36864
31630 100/300 Read block 2527 len 4096
31631 101/300 Read block 491 len 36864
31632 102/300 Write block 47 len 16384 data: 5 4 4 4
31633 103/300 Flush
31634 104/300 Write block 4072 len 32768 data: 4 4 5 5 5 6 7 8
31635 105/300 Read block 2982 len 8192
31636 106/300 Read block 4945 len 20480
31637 107/300 Flush
31638 108/300 Write block 1280 len 16384 data: 4 5 5 6
31639 109/300 Read block 4078 len 20480
31640 110/300 Read block 375 len 24576
31641 111/300 Read block 4738 len 20480
31642 112/300 Read block 1742 len 24576
31643 113/300 Read block 2865 len 24576
31644 114/300 Read block 1837 len 32768
31645 115/300 Read block 434 len 24576
31646 116/300 Read block 4756 len 20480
31647 117/300 Write block 1605 len 28672 data: 5 7 7 8 8 7 7
31648 118/300 Read block 3651 len 20480
31649 119/300 Write block 1288 len 36864 data: 5 5 6 8 6 6 6 4 6
31650 120/300 Write block 1876 len 20480 data: 4 5 5 5 2
31651 121/300 Write block 14 len 16384 data: 9 7 8 6
31652 122/300 Read block 467 len 40960
31653 123/300 Read block 3426 len 36864
31654 124/300 Write block 1751 len 32768 data: 6 7 7 8 7 6 4 3
31655 125/300 Write block 4320 len 36864 data: 5 5 6 5 5 5 5 5 5
31656 126/300 Write block 4487 len 16384 data: 3 4 5 4
31657 127/300 Read block 4391 len 20480
31658 128/300 Read block 4507 len 16384
31659 129/300 Read block 3823 len 12288
31660 130/300 Read block 3535 len 28672
31661 131/300 Read block 1418 len 4096
31662 132/300 Write block 2080 len 20480 data: 7 7 7 7 5
31663 133/300 Flush
31664 134/300 Write block 2003 len 8192 data: 5 6
31665 135/300 Write block 2021 len 4096 data: 5
31666 136/300 Flush
31667 137/300 Read block 3299 len 28672
31668 138/300 Read block 1551 len 4096
31669 139/300 Flush
31670 140/300 Read block 4557 len 28672
31671 141/300 Write block 3936 len 4096 data: 4
31672 142/300 Write block 3764 len 12288 data: 5 4 6
31673 143/300 Write block 1627 len 36864 data: 7 8 6 6 8 7 6 6 7
31674 144/300 Read block 457 len 40960
31675 145/300 Write block 4238 len 16384 data: 4 7 8 7
31676 146/300 Read block 1482 len 28672
31677 147/300 Write block 526 len 12288 data: 6 6 4
31678 148/300 Read block 243 len 24576
31679 149/300 Read block 4540 len 20480
31680 150/300 Read block 4074 len 24576
31681 151/300 Read block 2010 len 28672
31682 152/300 Write block 3738 len 24576 data: 7 7 5 3 3 3
31683 153/300 Write block 3647 len 4096 data: 6
31684 154/300 Read block 1506 len 32768
31685 155/300 Read block 2213 len 40960
31686 156/300 Write block 2534 len 36864 data: 6 6 7 8 9 9 6 8 9
31687 157/300 Write block 110 len 24576 data: 7 5 5 5 4 4
31688 158/300 Write block 791 len 28672 data: 6 5 5 4 3 5 6
31689 159/300 Read block 4014 len 12288
31690 160/300 Write block 4646 len 32768 data: 4 4 2 3 3 5 5 5
31691 161/300 Read block 139 len 12288
31692 162/300 Read block 3455 len 20480
31693 163/300 Read block 4257 len 24576
31694 164/300 Read block 223 len 28672
31695 165/300 Read block 882 len 36864
31696 166/300 Write block 3964 len 12288 data: 5 5 7
31697 167/300 Write block 505 len 40960 data: 4 4 3 4 6 7 8 10 10 9
31698 168/300 Write block 2317 len 12288 data: 4 6 7
31699 169/300 Write block 3929 len 32768 data: 3 4 6 6 4 4 4 5
31700 170/300 Read block 4645 len 32768
31701 171/300 Read block 267 len 16384
31702 172/300 Read block 4612 len 28672
31703 173/300 Read block 3007 len 40960
31704 174/300 Read block 821 len 28672
31705 175/300 Read block 4072 len 24576
31706 176/300 Write block 4748 len 4096 data: 6
31707 177/300 Flush
31708 178/300 Flush
31709 179/300 Write block 1055 len 36864 data: 9 9 7 6 7 7 5 8 7
31710 180/300 Write block 183 len 32768 data: 6 6 6 5 5 7 6 6
31711 181/300 Write block 2682 len 4096 data: 4
31712 182/300 Write block 3162 len 32768 data: 6 8 6 5 6 6 6 8
31713 183/300 Write block 4305 len 8192 data: 5 4
31714 184/300 Read block 731 len 4096
31715 185/300 Write block 2419 len 28672 data: 4 4 5 5 5 6 7
31716 186/300 Flush
31717 187/300 Write block 3761 len 32768 data: 4 7 6 6 5 7 6 5
31718 188/300 Write block 56 len 28672 data: 2 3 3 4 4 4 3
31719 189/300 Flush
31720 190/300 Write block 1939 len 36864 data: 11 11 9 9 7 6 8 8 9
31721 191/300 Flush
31722 192/300 Read block 370 len 4096
31723 193/300 Write block 799 len 32768 data: 6 5 7 6 6 4 4 4
31724 194/300 Write block 2187 len 20480 data: 3 3 4 6 7
31725 195/300 Read block 2174 len 28672
31726 196/300 Read block 4803 len 36864
31727 197/300 Flush
31728 198/300 Read block 4754 len 20480
31729 199/300 Write block 1384 len 28672 data: 7 6 5 7 8 7 7
31730 200/300 Write block 3371 len 36864 data: 7 6 6 5 5 5 5 3 3
31731 201/300 Read block 1665 len 24576
31732 202/300 Read block 3411 len 4096
31733 203/300 Read block 4642 len 20480
31734 204/300 Write block 4016 len 8192 data: 7 7
31735 205/300 Write block 3683 len 4096 data: 6
31736 206/300 Flush
31737 207/300 Read block 4222 len 32768
31738 208/300 Read block 819 len 28672
31739 209/300 Write block 4360 len 24576 data: 7 7 6 5 5 5
31740 210/300 Read block 1009 len 28672
31741 211/300 Read block 4251 len 12288
31742 212/300 Read block 3198 len 8192
31743 213/300 Flush
31744 214/300 Write block 2868 len 12288 data: 3 3 3
31745 215/300 Read block 1280 len 24576
31746 216/300 Write block 3213 len 12288 data: 6 7 7
31747 217/300 Read block 732 len 24576
31748 218/300 Write block 669 len 20480 data: 4 4 4 4 5
31749 219/300 Flush
31750 220/300 Read block 2166 len 40960
31751 221/300 Write block 3795 len 40960 data: 8 8 6 6 4 3 3 4 4 3
31752 222/300 Write block 3073 len 32768 data: 6 6 8 10 8 7 7 9
31753 223/300 Write block 4830 len 28672 data: 4 4 4 4 3 3 2
31754 224/300 Read block 2321 len 32768
31755 225/300 Write block 326 len 32768 data: 7 7 8 8 7 5 5 7
31756 226/300 Write block 1130 len 24576 data: 2 3 3 3 2 2
31757 227/300 Read block 1961 len 28672
31758 228/300 Read block 3152 len 16384
31759 229/300 Read block 18 len 24576
31760 230/300 Write block 1423 len 32768 data: 5 6 6 8 10 9 9 9
31761 231/300 Read block 4936 len 28672
31762 232/300 Read block 2417 len 16384
31763 233/300 Read block 83 len 40960
31764 234/300 Write block 3104 len 20480 data: 8 4 4 5 4
31765 235/300 Flush
31766 236/300 Write block 901 len 40960 data: 7 8 8 8 4 6 7 6 5 5
31767 237/300 Write block 4413 len 8192 data: 5 5
31768 238/300 Write block 1761 len 28672 data: 4 5 6 5 4 3 7
31769 239/300 Write block 2653 len 16384 data: 3 3 3 3
31770 240/300 Write block 4652 len 40960 data: 6 6 5 5 3 4 7 8 7 10
31771 241/300 Read block 1407 len 16384
31772 242/300 Write block 1431 len 32768 data: 7 5 4 4 5 7 8 7
31773 243/300 Write block 4666 len 40960 data: 10 6 6 6 6 5 5 6 5 4
31774 244/300 Read block 3901 len 12288
31775 245/300 Write block 4151 len 36864 data: 5 5 5 5 4 5 6 8 8
31776 246/300 Write block 3693 len 28672 data: 3 4 4 5 5 7 8
31777 247/300 Read block 1354 len 4096
31778 248/300 Read block 151 len 40960
31779 249/300 Write block 272 len 36864 data: 9 8 7 6 5 4 4 5 6
31780 250/300 Write block 3905 len 4096 data: 4
31781 251/300 Write block 1344 len 8192 data: 7 7
31782 252/300 Flush
31783 253/300 Write block 96 len 24576 data: 5 6 6 6 6 6
31784 254/300 Write block 4266 len 36864 data: 7 7 8 8 6 5 5 3 3
31785 255/300 Read block 2349 len 16384
31786 256/300 Read block 2202 len 12288
31787 257/300 Read block 1865 len 8192
31788 258/300 Read block 2935 len 36864
31789 259/300 Write block 4085 len 32768 data: 4 4 4 3 5 5 7 7
31790 260/300 Read block 4005 len 20480
31791 261/300 Flush
31792 262/300 Write block 667 len 40960 data: 5 4 5 5 5 5 6 6 7 6
31793 263/300 Write block 1055 len 24576 data: 10 10 8 7 8 8
31794 264/300 Read block 2803 len 24576
31795 265/300 Read block 3338 len 36864
31796 266/300 Read block 4887 len 8192
31797 267/300 Write block 1912 len 28672 data: 5 5 6 5 5 4 4
31798 268/300 Write block 1942 len 24576 data: 10 8 7 9 9 10
31799 269/300 Write block 625 len 36864 data: 5 6 7 11 12 13 12 10 10
31800 270/300 Read block 3946 len 4096
31801 271/300 Write block 4358 len 20480 data: 8 7 8 8 7
31802 272/300 Read block 1829 len 20480
31803 273/300 Read block 1009 len 28672
31804 274/300 Flush
31805 275/300 Read block 791 len 16384
31806 276/300 Write block 998 len 16384 data: 4 4 3 3
31807 277/300 Read block 3326 len 4096
31808 278/300 Read block 4076 len 8192
31809 279/300 Read block 2218 len 4096
31810 280/300 Read block 4166 len 8192
31811 281/300 Write block 3796 len 40960 data: 9 7 7 5 4 4 5 5 4 4
31812 282/300 Read block 1353 len 16384
31813 283/300 Flush
31814 284/300 Write block 2948 len 4096 data: 10
31815 285/300 Read block 4846 len 20480
31816 286/300 Read block 444 len 36864
31817 287/300 Write block 1046 len 12288 data: 9 9 7
31818 288/300 Read block 1873 len 4096
31819 289/300 Read block 4160 len 40960
31820 290/300 Write block 2536 len 24576 data: 8 9 10 10 7 9
31821 291/300 Write block 4064 len 32768 data: 10 9 9 8 6 6 5 5
31822 292/300 Write block 4210 len 32768 data: 7 7 9 8 8 6 4 4
31823 293/300 Flush
31824 294/300 Write block 3164 len 24576 data: 7 6 7 7 7 9
31825 295/300 Write block 4097 len 32768 data: 4 4 5 5 5 6 5 5
31826 296/300 Write block 531 len 12288 data: 5 7 7
31827 297/300 Flush
31828 298/300 Flush
31829 299/300 Read block 4595 len 24576
31830 300/300 Read block 749 len 28672
31831 301/300 Read block 2237 len 40960
31832 Main task has work to do, go find it
31833 got dsc Start(2)
31834 start 2
31835 Feb 25 17:25:13.472 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2a9606d8-bd1d-4462-93f1-dd07abd680ca, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
31836 Replay: started 2, returned:Ok(())
31837 ----------------------------------------------------------------
31838 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31839 States: Active Active Offline
31840 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31841 9454 Acked Read 3 Done Done Sent false
31842 9455 Acked Read 4 Done Done Sent false
31843 9456 Acked Read 1 Done Done Sent false
31844 9457 Acked Read 3 Done Done Sent false
31845 9458 Acked Read 9 Done Done Sent false
31846 9459 Acked Flush 0 Done Done Sent false
31847 9460 Acked Write 4 Done Done Sent false
31848 9461 Acked Read 8 Done Done Sent false
31849 9462 Acked Read 5 Done Done Sent false
31850 9463 Acked Write 8 Done Done Sent false
31851 9464 Acked Read 1 Done Done Sent false
31852 9465 Acked Write 10 Done Done Sent false
31853 9466 Acked Write 9 Done Done Sent false
31854 9467 Acked Read 5 Done Done Sent false
31855 9468 Acked Read 1 Done Done Sent false
31856 9469 Acked Read 5 Done Done Sent false
31857 9470 Acked Write 7 Done Done Sent false
31858 9471 Acked Write 1 Done Done Sent false
31859 9472 Acked Write 2 Done Done Sent false
31860 9473 Acked Read 3 Done Done Sent false
31861 9474 Acked Read 4 Done Done Sent false
31862 9475 Acked Read 10 Done Done Sent false
31863 9476 Acked Read 5 Done Done Sent false
31864 9477 Acked Write 2 Done Done Sent false
31865 9478 Acked Flush 0 Done Done Sent false
31866 9479 Acked Read 10 Done Done Sent false
31867 9480 Acked Write 6 Done Done Sent false
31868 9481 Acked Flush 0 Done Done Sent false
31869 9482 Acked Write 8 Done Done Sent false
31870 9483 Acked Write 8 Done Done Sent false
31871 9484 Acked Write 9 Done Done Sent false
31872 9485 Acked Write 6 Done Done Sent false
31873 9486 Acked Read 5 Done Done Sent false
31874 9487 Acked Read 7 Done Done Sent false
31875 9488 Acked Read 9 Done Done Sent false
31876 9489 Acked Read 9 Done Done Sent false
31877 9490 Acked Read 6 Done Done Sent false
31878 9491 Acked Read 2 Done Done Sent false
31879 9492 Acked Read 8 Done Done Sent false
31880 9493 Acked Write 2 Done Done Sent false
31881 9494 Acked Write 2 Done Done Sent false
31882 9495 Acked Read 1 Done Done Sent false
31883 9496 Acked Write 3 Done Done Sent false
31884 9497 Acked Read 5 Done Done Sent false
31885 9498 Acked Read 8 Done Done Sent false
31886 9499 Acked Read 8 Done Done Sent false
31887 9500 Acked Write 7 Done Done Sent false
31888 9501 Acked Write 8 Done Done Sent false
31889 9502 Acked Read 7 Done Done Sent false
31890 9503 Acked Write 6 Done Done Sent false
31891 9504 Acked Write 6 Done Done Sent false
31892 9505 Acked Write 9 Done Done Sent false
31893 9506 Acked Read 1 Done Done Sent false
31894 9507 Acked Write 4 Done Done Sent false
31895 9508 Acked Read 3 Done Done Sent false
31896 9509 Acked Write 10 Done Done Sent false
31897 9510 Acked Read 1 Done Done Sent false
31898 9511 Acked Read 4 Done Done Sent false
31899 9512 Acked Read 8 Done Done Sent false
31900 9513 Acked Read 2 Done Done Sent false
31901 9514 Acked Write 2 Done Done Sent false
31902 9515 Acked Flush 0 Done Done Sent false
31903 9516 Acked Write 6 Done Done Sent false
31904 9517 Acked Flush 0 Done Done Sent false
31905 9518 Acked Read 5 Done Done Sent false
31906 9519 Acked Read 1 Done Done Sent false
31907 9520 Acked Read 10 Done Done Sent false
31908 9521 Acked Read 4 Done Done Sent false
31909 9522 Acked Read 10 Done Done Sent false
31910 9523 Acked Write 4 Done Done Sent false
31911 9524 Acked Write 1 Done Done Sent false
31912 9525 Acked Read 8 Done Done Sent false
31913 9526 Acked Write 10 Done Done Sent false
31914 9527 Acked Flush 0 Done Done Sent false
31915 9528 Acked Write 7 Done Done Sent false
31916 9529 Acked Read 9 Done Done Sent false
31917 9530 Acked Read 7 Done Done Sent false
31918 9531 Acked Write 6 Done Done Sent false
31919 9532 Acked Write 8 Done Done Sent false
31920 9533 Acked Read 6 Done Done Sent false
31921 9534 Acked Write 2 Done Done Sent false
31922 9535 Acked Read 4 Done Done Sent false
31923 9536 Acked Write 4 Done Done Sent false
31924 9537 Acked Write 5 Done Done Sent false
31925 9538 Acked Write 3 Done Done Sent false
31926 9539 Acked Read 7 Done Done Sent false
31927 9540 Acked Read 5 Done Done Sent false
31928 9541 Acked Read 2 Done Done Sent false
31929 9542 Acked Write 7 Done Done Sent false
31930 9543 Acked Read 8 Done Done Sent false
31931 9544 Acked Read 4 Done Done Sent false
31932 9545 Acked Read 9 Done Done Sent false
31933 9546 Acked Read 2 Done Done Sent false
31934 9547 Acked Read 6 Done Done Sent false
31935 9548 Acked Write 6 Done Done Sent false
31936 9549 Acked Write 7 Done Done Sent false
31937 9550 Acked Read 5 Done Done Sent false
31938 9551 Acked Read 6 Done Done Sent false
31939 9552 Acked Read 9 Done Done Sent false
31940 9553 Acked Read 1 Done Done Sent false
31941 9554 Acked Read 9 Done Done Sent false
31942 9555 Acked Write 4 Done Done Sent false
31943 9556 Acked Flush 0 Done Done Sent false
31944 9557 Acked Write 8 Done Done Sent false
31945 9558 Acked Read 2 Done Done Sent false
31946 9559 Acked Read 5 Done Done Sent false
31947 9560 Acked Flush 0 Done Done Sent false
31948 9561 Acked Write 4 Done Done Sent false
31949 9562 Acked Read 5 Done Done Sent false
31950 9563 Acked Read 6 Done Done Sent false
31951 9564 Acked Read 5 Done Done Sent false
31952 9565 Acked Read 6 Done Done Sent false
31953 9566 Acked Read 6 Done Done Sent false
31954 9567 Acked Read 8 Done Done Sent false
31955 9568 Acked Read 6 Done Done Sent false
31956 9569 Acked Read 5 Done Done Sent false
31957 9570 Acked Write 7 Done Done Sent false
31958 9571 Acked Read 5 Done Done Sent false
31959 9572 Acked Write 9 Done Done Sent false
31960 9573 Acked Write 5 Done Done Sent false
31961 9574 Acked Write 4 Done Done Sent false
31962 9575 Acked Read 10 Done Done Sent false
31963 9576 Acked Read 9 Done Done Sent false
31964 9577 Acked Write 8 Done Done Sent false
31965 9578 Acked Write 9 Done Done Sent false
31966 9579 Acked Write 4 Done Done Sent false
31967 9580 Acked Read 5 Done Done Sent false
31968 9581 Acked Read 4 Done Done Sent false
31969 9582 Acked Read 3 Done Done Sent false
31970 9583 Acked Read 7 Done Done Sent false
31971 9584 Acked Read 1 Done Done Sent false
31972 9585 Acked Write 5 Done Done Sent false
31973 9586 Acked Flush 0 Done Done Sent false
31974 9587 Acked Write 2 Done Done Sent false
31975 9588 Acked Write 1 Done Done Sent false
31976 9589 Acked Flush 0 Done Done Sent false
31977 9590 Acked Read 7 Done Done Sent false
31978 9591 Acked Read 1 Done Done Sent false
31979 9592 Acked Flush 0 Done Done Sent false
31980 9593 Acked Read 7 Done Done Sent false
31981 9594 Acked Write 1 Done Done Sent false
31982 9595 Acked Write 3 Done Done Sent false
31983 9596 Acked Write 9 Done Done Sent false
31984 9597 Acked Read 10 Done Done Sent false
31985 9598 Acked Write 4 Done Done Sent false
31986 9599 Acked Read 7 Done Done Sent false
31987 9600 Acked Write 3 Done Done Sent false
31988 9601 Acked Read 6 Done Done Sent false
31989 9602 Acked Read 5 Done Done Sent false
31990 9603 Acked Read 6 Done Done Sent false
31991 9604 Acked Read 7 Done Done Sent false
31992 9605 Acked Write 6 Done Done Sent false
31993 9606 Acked Write 1 Done Done Sent false
31994 9607 Acked Read 8 Done Done Sent false
31995 9608 Acked Read 10 Done Done Sent false
31996 9609 Acked Write 9 Done Done Sent false
31997 9610 Acked Write 6 Done Done Sent false
31998 9611 Acked Write 7 Done Done Sent false
31999 9612 Acked Read 3 Done Done Sent false
32000 9613 Acked Write 8 Done Done Sent false
32001 9614 Acked Read 3 Done Done Sent false
32002 9615 Acked Read 5 Done Done Sent false
32003 9616 Acked Read 6 Done Done Sent false
32004 9617 Acked Read 7 Done Done Sent false
32005 9618 Acked Read 9 Done Done Sent false
32006 9619 Acked Write 3 Done Done Sent false
32007 9620 Acked Write 10 Done Done Sent false
32008 9621 Acked Write 3 Done Done Sent false
32009 9622 Acked Write 8 Done Done Sent false
32010 9623 Acked Read 8 Done Done Sent false
32011 9624 Acked Read 4 Done Done Sent false
32012 9625 Acked Read 7 Done Done Sent false
32013 9626 Acked Read 10 Done Done Sent false
32014 9627 Acked Read 7 Done Done Sent false
32015 9628 Acked Read 6 Done Done Sent false
32016 9629 Acked Write 1 Done Done Sent false
32017 9630 Acked Flush 0 Done Done Sent false
32018 9631 Acked Flush 0 Done Done Sent false
32019 9632 Acked Write 9 Done Done Sent false
32020 9633 Acked Write 8 Done Done Sent false
32021 9634 Acked Write 1 Done Done Sent false
32022 9635 Acked Write 8 Done Done Sent false
32023 9636 Acked Write 2 Done Done Sent false
32024 9637 Acked Read 1 Done Done Sent false
32025 9638 Acked Write 7 Done Done Sent false
32026 9639 Acked Flush 0 Done Done Sent false
32027 9640 Acked Write 8 Done Done Sent false
32028 9641 Acked Write 7 Done Done Sent false
32029 9642 Acked Flush 0 Done Done Sent false
32030 9643 Acked Write 9 Done Done Sent false
32031 9644 Acked Flush 0 Done Done Sent false
32032 9645 Acked Read 1 Done Done Sent false
32033 9646 Acked Write 8 Done Done Sent false
32034 9647 Acked Write 5 Done Done Sent false
32035 9648 Acked Read 7 Done Done Sent false
32036 9649 Acked Read 9 Done Done Sent false
32037 9650 Acked Flush 0 Done Done Sent false
32038 9651 Acked Read 5 Done Done Sent false
32039 9652 Acked Write 7 Done Done Sent false
32040 9653 Acked Write 9 Done Done Sent false
32041 9654 Acked Read 6 Done Done Sent false
32042 9655 Acked Read 1 Done Done Sent false
32043 9656 Acked Read 5 Done Done Sent false
32044 9657 Acked Write 2 Done Done Sent false
32045 9658 Acked Write 1 Done Done Sent false
32046 9659 Acked Flush 0 Done Done Sent false
32047 9660 Acked Read 8 Done Done Sent false
32048 9661 Acked Read 7 Done Done Sent false
32049 9662 Acked Write 6 Done Done Sent false
32050 9663 Acked Read 7 Done Done Sent false
32051 9664 Acked Read 3 Done Done Sent false
32052 9665 Acked Read 2 Done Done Sent false
32053 9666 Acked Flush 0 Done Done Sent false
32054 9667 Acked Write 3 Done Done Sent false
32055 9668 Acked Read 6 Done Done Sent false
32056 9669 Acked Write 3 Done Done Sent false
32057 9670 Acked Read 6 Done Done Sent false
32058 9671 Acked Write 5 Done Done Sent false
32059 9672 Acked Flush 0 Done Done Sent false
32060 9673 Acked Read 10 Done Done Sent false
32061 9674 Acked Write 10 Done Done Sent false
32062 9675 Acked Write 8 Done Done Sent false
32063 9676 Acked Write 7 Done Done Sent false
32064 9677 Acked Read 8 Done Done Sent false
32065 9678 Acked Write 8 Done Done Sent false
32066 9679 Acked Write 6 Done Done Sent false
32067 9680 Acked Read 7 Done Done Sent false
32068 9681 Acked Read 4 Done Done Sent false
32069 9682 Acked Read 6 Done Done Sent false
32070 9683 Acked Write 8 Done Done Sent false
32071 9684 Acked Read 7 Done Done Sent false
32072 9685 Acked Read 4 Done Done Sent false
32073 9686 Acked Read 10 Done Done Sent false
32074 9687 Acked Write 5 Done Done Sent false
32075 9688 Acked Flush 0 Done Done Sent false
32076 9689 Acked Write 10 Done Done Sent false
32077 9690 Acked Write 2 Done Done Sent false
32078 9691 Acked Write 7 Done Done Sent false
32079 9692 Acked Write 4 Done Done Sent false
32080 9693 Acked Write 10 Done Done Sent false
32081 9694 Acked Read 4 Done Done Sent false
32082 9695 Acked Write 8 Done Done Sent false
32083 9696 Acked Write 10 Done Done Sent false
32084 9697 Acked Read 3 Done Done Sent false
32085 9698 Acked Write 9 Done Done Sent false
32086 9699 Acked Write 7 Done Done Sent false
32087 9700 Acked Read 1 Done Done Sent false
32088 9701 Acked Read 10 Done Done Sent false
32089 9702 Acked Write 9 Done Done Sent false
32090 9703 Acked Write 1 Done Done Sent false
32091 9704 Acked Write 2 Done Done Sent false
32092 9705 Acked Flush 0 Done Done Sent false
32093 9706 Acked Write 6 Done Done Sent false
32094 9707 Acked Write 9 Done Done Sent false
32095 9708 Acked Read 4 Done Done Sent false
32096 9709 Acked Read 3 Done Done Sent false
32097 9710 Acked Read 2 Done Done Sent false
32098 9711 Acked Read 9 Done Done Sent false
32099 9712 Acked Write 8 Done Done Sent false
32100 9713 Acked Read 5 Done Done Sent false
32101 9714 Acked Flush 0 Done Done Sent false
32102 9715 Acked Write 10 Done Done Sent false
32103 9716 Acked Write 6 Done Done Sent false
32104 9717 Acked Read 6 Done Done Sent false
32105 9718 Acked Read 9 Done Done Sent false
32106 9719 Acked Read 2 Done Done Sent false
32107 9720 Acked Write 7 Done Done Sent false
32108 9721 Acked Write 6 Done Done Sent false
32109 9722 Acked Write 9 Done Done Sent false
32110 9723 Acked Read 1 Done Done Sent false
32111 9724 Acked Write 5 Done Done Sent false
32112 9725 Acked Read 5 Done Done Sent false
32113 9726 Acked Read 7 Done Done Sent false
32114 9727 Acked Flush 0 Done Done Sent false
32115 9728 Acked Read 4 Done Done Sent false
32116 9729 Acked Write 4 Done Done Sent false
32117 9730 Acked Read 1 Done Done Sent false
32118 9731 Acked Read 2 Done Done Sent false
32119 9732 Acked Read 1 Done Done Sent false
32120 9733 Acked Read 2 Done Done Sent false
32121 9734 Acked Write 10 Done Done Sent false
32122 9735 Acked Read 4 Done Done Sent false
32123 9736 Acked Flush 0 Done Done Sent false
32124 9737 Acked Write 1 Done Done Sent false
32125 9738 Acked Read 5 Done Done Sent false
32126 9739 Acked Read 9 Done Done Sent false
32127 9740 Acked Write 3 Done Done Sent false
32128 9741 Acked Read 1 Done Done Sent false
32129 9742 Acked Read 10 Done Done Sent false
32130 9743 Acked Write 6 Done Done Sent false
32131 9744 Acked Write 8 Done Done Sent false
32132 9745 Acked Write 8 Done Done Sent false
32133 9746 Acked Flush 0 Done Done Sent false
32134 9747 Acked Write 6 Done Done Sent false
32135 9748 Acked Write 8 Done Done Sent false
32136 9749 Acked Write 3 Done Done Sent false
32137 9750 Acked Flush 0 Done Done Sent false
32138 9751 Acked Flush 0 Done Done Sent false
32139 9752 Acked Read 6 Done Done Sent false
32140 9753 Acked Read 7 Done Done Sent false
32141 9754 Acked Read 10 Done Done Sent false
32142 STATES DS:0 DS:1 DS:2 TOTAL
32143 Sent 0 0 301 301
32144 Done 301 301 0 602
32145 Skipped 0 0 0 0
32146 Error 0 0 0 0
32147 Last Flush: Some(JobId(9751)) Some(JobId(9751)) Some(JobId(9453))
32148 Downstairs last five completed: 9453 9452 9451 9450 9449
32149 Upstairs last five completed: 9754 9753 9752 9751 9750
32150 CLIENT: Up:0 ds:301 act:2
321512025-02-25T17:25:13.975ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
321522025-02-25T17:25:15.186ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
32153 [8830] got start action
32154 [8830] I am going to restart
32155 Starting downstairs at port 8830
32156 Make output file at "/tmp/dsc/downstairs-8830.txt"
32157 [8830][2] reports Starting
32158 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1237)
32159 [8830][2] reports Running
321602025-02-25T17:25:15.922ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
321612025-02-25T17:25:16.924ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
32162 ----------------------------------------------------------------
32163 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32164 States: Active Active Offline
32165 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32166 9454 Acked Read 3 Done Done Sent false
32167 9455 Acked Read 4 Done Done Sent false
32168 9456 Acked Read 1 Done Done Sent false
32169 9457 Acked Read 3 Done Done Sent false
32170 9458 Acked Read 9 Done Done Sent false
32171 9459 Acked Flush 0 Done Done Sent false
32172 9460 Acked Write 4 Done Done Sent false
32173 9461 Acked Read 8 Done Done Sent false
32174 9462 Acked Read 5 Done Done Sent false
32175 9463 Acked Write 8 Done Done Sent false
32176 9464 Acked Read 1 Done Done Sent false
32177 9465 Acked Write 10 Done Done Sent false
32178 9466 Acked Write 9 Done Done Sent false
32179 9467 Acked Read 5 Done Done Sent false
32180 9468 Acked Read 1 Done Done Sent false
32181 9469 Acked Read 5 Done Done Sent false
32182 9470 Acked Write 7 Done Done Sent false
32183 9471 Acked Write 1 Done Done Sent false
32184 9472 Acked Write 2 Done Done Sent false
32185 9473 Acked Read 3 Done Done Sent false
32186 9474 Acked Read 4 Done Done Sent false
32187 9475 Acked Read 10 Done Done Sent false
32188 9476 Acked Read 5 Done Done Sent false
32189 9477 Acked Write 2 Done Done Sent false
32190 9478 Acked Flush 0 Done Done Sent false
32191 9479 Acked Read 10 Done Done Sent false
32192 9480 Acked Write 6 Done Done Sent false
32193 9481 Acked Flush 0 Done Done Sent false
32194 9482 Acked Write 8 Done Done Sent false
32195 9483 Acked Write 8 Done Done Sent false
32196 9484 Acked Write 9 Done Done Sent false
32197 9485 Acked Write 6 Done Done Sent false
32198 9486 Acked Read 5 Done Done Sent false
32199 9487 Acked Read 7 Done Done Sent false
32200 9488 Acked Read 9 Done Done Sent false
32201 9489 Acked Read 9 Done Done Sent false
32202 9490 Acked Read 6 Done Done Sent false
32203 9491 Acked Read 2 Done Done Sent false
32204 9492 Acked Read 8 Done Done Sent false
32205 9493 Acked Write 2 Done Done Sent false
32206 9494 Acked Write 2 Done Done Sent false
32207 9495 Acked Read 1 Done Done Sent false
32208 9496 Acked Write 3 Done Done Sent false
32209 9497 Acked Read 5 Done Done Sent false
32210 9498 Acked Read 8 Done Done Sent false
32211 9499 Acked Read 8 Done Done Sent false
32212 9500 Acked Write 7 Done Done Sent false
32213 9501 Acked Write 8 Done Done Sent false
32214 9502 Acked Read 7 Done Done Sent false
32215 9503 Acked Write 6 Done Done Sent false
32216 9504 Acked Write 6 Done Done Sent false
32217 9505 Acked Write 9 Done Done Sent false
32218 9506 Acked Read 1 Done Done Sent false
32219 9507 Acked Write 4 Done Done Sent false
32220 9508 Acked Read 3 Done Done Sent false
32221 9509 Acked Write 10 Done Done Sent false
32222 9510 Acked Read 1 Done Done Sent false
32223 9511 Acked Read 4 Done Done Sent false
32224 9512 Acked Read 8 Done Done Sent false
32225 9513 Acked Read 2 Done Done Sent false
32226 9514 Acked Write 2 Done Done Sent false
32227 9515 Acked Flush 0 Done Done Sent false
32228 9516 Acked Write 6 Done Done Sent false
32229 9517 Acked Flush 0 Done Done Sent false
32230 9518 Acked Read 5 Done Done Sent false
32231 9519 Acked Read 1 Done Done Sent false
32232 9520 Acked Read 10 Done Done Sent false
32233 9521 Acked Read 4 Done Done Sent false
32234 9522 Acked Read 10 Done Done Sent false
32235 9523 Acked Write 4 Done Done Sent false
32236 9524 Acked Write 1 Done Done Sent false
32237 9525 Acked Read 8 Done Done Sent false
32238 9526 Acked Write 10 Done Done Sent false
32239 9527 Acked Flush 0 Done Done Sent false
32240 9528 Acked Write 7 Done Done Sent false
32241 9529 Acked Read 9 Done Done Sent false
32242 9530 Acked Read 7 Done Done Sent false
32243 9531 Acked Write 6 Done Done Sent false
32244 9532 Acked Write 8 Done Done Sent false
32245 9533 Acked Read 6 Done Done Sent false
32246 9534 Acked Write 2 Done Done Sent false
32247 9535 Acked Read 4 Done Done Sent false
32248 9536 Acked Write 4 Done Done Sent false
32249 9537 Acked Write 5 Done Done Sent false
32250 9538 Acked Write 3 Done Done Sent false
32251 9539 Acked Read 7 Done Done Sent false
32252 9540 Acked Read 5 Done Done Sent false
32253 9541 Acked Read 2 Done Done Sent false
32254 9542 Acked Write 7 Done Done Sent false
32255 9543 Acked Read 8 Done Done Sent false
32256 9544 Acked Read 4 Done Done Sent false
32257 9545 Acked Read 9 Done Done Sent false
32258 9546 Acked Read 2 Done Done Sent false
32259 9547 Acked Read 6 Done Done Sent false
32260 9548 Acked Write 6 Done Done Sent false
32261 9549 Acked Write 7 Done Done Sent false
32262 9550 Acked Read 5 Done Done Sent false
32263 9551 Acked Read 6 Done Done Sent false
32264 9552 Acked Read 9 Done Done Sent false
32265 9553 Acked Read 1 Done Done Sent false
32266 9554 Acked Read 9 Done Done Sent false
32267 9555 Acked Write 4 Done Done Sent false
32268 9556 Acked Flush 0 Done Done Sent false
32269 9557 Acked Write 8 Done Done Sent false
32270 9558 Acked Read 2 Done Done Sent false
32271 9559 Acked Read 5 Done Done Sent false
32272 9560 Acked Flush 0 Done Done Sent false
32273 9561 Acked Write 4 Done Done Sent false
32274 9562 Acked Read 5 Done Done Sent false
32275 9563 Acked Read 6 Done Done Sent false
32276 9564 Acked Read 5 Done Done Sent false
32277 9565 Acked Read 6 Done Done Sent false
32278 9566 Acked Read 6 Done Done Sent false
32279 9567 Acked Read 8 Done Done Sent false
32280 9568 Acked Read 6 Done Done Sent false
32281 9569 Acked Read 5 Done Done Sent false
32282 9570 Acked Write 7 Done Done Sent false
32283 9571 Acked Read 5 Done Done Sent false
32284 9572 Acked Write 9 Done Done Sent false
32285 9573 Acked Write 5 Done Done Sent false
32286 9574 Acked Write 4 Done Done Sent false
32287 9575 Acked Read 10 Done Done Sent false
32288 9576 Acked Read 9 Done Done Sent false
32289 9577 Acked Write 8 Done Done Sent false
32290 9578 Acked Write 9 Done Done Sent false
32291 9579 Acked Write 4 Done Done Sent false
32292 9580 Acked Read 5 Done Done Sent false
32293 9581 Acked Read 4 Done Done Sent false
32294 9582 Acked Read 3 Done Done Sent false
32295 9583 Acked Read 7 Done Done Sent false
32296 9584 Acked Read 1 Done Done Sent false
32297 9585 Acked Write 5 Done Done Sent false
32298 9586 Acked Flush 0 Done Done Sent false
32299 9587 Acked Write 2 Done Done Sent false
32300 9588 Acked Write 1 Done Done Sent false
32301 9589 Acked Flush 0 Done Done Sent false
32302 9590 Acked Read 7 Done Done Sent false
32303 9591 Acked Read 1 Done Done Sent false
32304 9592 Acked Flush 0 Done Done Sent false
32305 9593 Acked Read 7 Done Done Sent false
32306 9594 Acked Write 1 Done Done Sent false
32307 9595 Acked Write 3 Done Done Sent false
32308 9596 Acked Write 9 Done Done Sent false
32309 9597 Acked Read 10 Done Done Sent false
32310 9598 Acked Write 4 Done Done Sent false
32311 9599 Acked Read 7 Done Done Sent false
32312 9600 Acked Write 3 Done Done Sent false
32313 9601 Acked Read 6 Done Done Sent false
32314 9602 Acked Read 5 Done Done Sent false
32315 9603 Acked Read 6 Done Done Sent false
32316 9604 Acked Read 7 Done Done Sent false
32317 9605 Acked Write 6 Done Done Sent false
32318 9606 Acked Write 1 Done Done Sent false
32319 9607 Acked Read 8 Done Done Sent false
32320 9608 Acked Read 10 Done Done Sent false
32321 9609 Acked Write 9 Done Done Sent false
32322 9610 Acked Write 6 Done Done Sent false
32323 9611 Acked Write 7 Done Done Sent false
32324 9612 Acked Read 3 Done Done Sent false
32325 9613 Acked Write 8 Done Done Sent false
32326 9614 Acked Read 3 Done Done Sent false
32327 9615 Acked Read 5 Done Done Sent false
32328 9616 Acked Read 6 Done Done Sent false
32329 9617 Acked Read 7 Done Done Sent false
32330 9618 Acked Read 9 Done Done Sent false
32331 9619 Acked Write 3 Done Done Sent false
32332 9620 Acked Write 10 Done Done Sent false
32333 9621 Acked Write 3 Done Done Sent false
32334 9622 Acked Write 8 Done Done Sent false
32335 9623 Acked Read 8 Done Done Sent false
32336 9624 Acked Read 4 Done Done Sent false
32337 9625 Acked Read 7 Done Done Sent false
32338 9626 Acked Read 10 Done Done Sent false
32339 9627 Acked Read 7 Done Done Sent false
32340 9628 Acked Read 6 Done Done Sent false
32341 9629 Acked Write 1 Done Done Sent false
32342 9630 Acked Flush 0 Done Done Sent false
32343 9631 Acked Flush 0 Done Done Sent false
32344 9632 Acked Write 9 Done Done Sent false
32345 9633 Acked Write 8 Done Done Sent false
32346 9634 Acked Write 1 Done Done Sent false
32347 9635 Acked Write 8 Done Done Sent false
32348 9636 Acked Write 2 Done Done Sent false
32349 9637 Acked Read 1 Done Done Sent false
32350 9638 Acked Write 7 Done Done Sent false
32351 9639 Acked Flush 0 Done Done Sent false
32352 9640 Acked Write 8 Done Done Sent false
32353 9641 Acked Write 7 Done Done Sent false
32354 9642 Acked Flush 0 Done Done Sent false
32355 9643 Acked Write 9 Done Done Sent false
32356 9644 Acked Flush 0 Done Done Sent false
32357 9645 Acked Read 1 Done Done Sent false
32358 9646 Acked Write 8 Done Done Sent false
32359 9647 Acked Write 5 Done Done Sent false
32360 9648 Acked Read 7 Done Done Sent false
32361 9649 Acked Read 9 Done Done Sent false
32362 9650 Acked Flush 0 Done Done Sent false
32363 9651 Acked Read 5 Done Done Sent false
32364 9652 Acked Write 7 Done Done Sent false
32365 9653 Acked Write 9 Done Done Sent false
32366 9654 Acked Read 6 Done Done Sent false
32367 9655 Acked Read 1 Done Done Sent false
32368 9656 Acked Read 5 Done Done Sent false
32369 9657 Acked Write 2 Done Done Sent false
32370 9658 Acked Write 1 Done Done Sent false
32371 9659 Acked Flush 0 Done Done Sent false
32372 9660 Acked Read 8 Done Done Sent false
32373 9661 Acked Read 7 Done Done Sent false
32374 9662 Acked Write 6 Done Done Sent false
32375 9663 Acked Read 7 Done Done Sent false
32376 9664 Acked Read 3 Done Done Sent false
32377 9665 Acked Read 2 Done Done Sent false
32378 9666 Acked Flush 0 Done Done Sent false
32379 9667 Acked Write 3 Done Done Sent false
32380 9668 Acked Read 6 Done Done Sent false
32381 9669 Acked Write 3 Done Done Sent false
32382 9670 Acked Read 6 Done Done Sent false
32383 9671 Acked Write 5 Done Done Sent false
32384 9672 Acked Flush 0 Done Done Sent false
32385 9673 Acked Read 10 Done Done Sent false
32386 9674 Acked Write 10 Done Done Sent false
32387 9675 Acked Write 8 Done Done Sent false
32388 9676 Acked Write 7 Done Done Sent false
32389 9677 Acked Read 8 Done Done Sent false
32390 9678 Acked Write 8 Done Done Sent false
32391 9679 Acked Write 6 Done Done Sent false
32392 9680 Acked Read 7 Done Done Sent false
32393 9681 Acked Read 4 Done Done Sent false
32394 9682 Acked Read 6 Done Done Sent false
32395 9683 Acked Write 8 Done Done Sent false
32396 9684 Acked Read 7 Done Done Sent false
32397 9685 Acked Read 4 Done Done Sent false
32398 9686 Acked Read 10 Done Done Sent false
32399 9687 Acked Write 5 Done Done Sent false
32400 9688 Acked Flush 0 Done Done Sent false
32401 9689 Acked Write 10 Done Done Sent false
32402 9690 Acked Write 2 Done Done Sent false
32403 9691 Acked Write 7 Done Done Sent false
32404 9692 Acked Write 4 Done Done Sent false
32405 9693 Acked Write 10 Done Done Sent false
32406 9694 Acked Read 4 Done Done Sent false
32407 9695 Acked Write 8 Done Done Sent false
32408 9696 Acked Write 10 Done Done Sent false
32409 9697 Acked Read 3 Done Done Sent false
32410 9698 Acked Write 9 Done Done Sent false
32411 9699 Acked Write 7 Done Done Sent false
32412 9700 Acked Read 1 Done Done Sent false
32413 9701 Acked Read 10 Done Done Sent false
32414 9702 Acked Write 9 Done Done Sent false
32415 9703 Acked Write 1 Done Done Sent false
32416 9704 Acked Write 2 Done Done Sent false
32417 9705 Acked Flush 0 Done Done Sent false
32418 9706 Acked Write 6 Done Done Sent false
32419 9707 Acked Write 9 Done Done Sent false
32420 9708 Acked Read 4 Done Done Sent false
32421 9709 Acked Read 3 Done Done Sent false
32422 9710 Acked Read 2 Done Done Sent false
32423 9711 Acked Read 9 Done Done Sent false
32424 9712 Acked Write 8 Done Done Sent false
32425 9713 Acked Read 5 Done Done Sent false
32426 9714 Acked Flush 0 Done Done Sent false
32427 9715 Acked Write 10 Done Done Sent false
32428 9716 Acked Write 6 Done Done Sent false
32429 9717 Acked Read 6 Done Done Sent false
32430 9718 Acked Read 9 Done Done Sent false
32431 9719 Acked Read 2 Done Done Sent false
32432 9720 Acked Write 7 Done Done Sent false
32433 9721 Acked Write 6 Done Done Sent false
32434 9722 Acked Write 9 Done Done Sent false
32435 9723 Acked Read 1 Done Done Sent false
32436 9724 Acked Write 5 Done Done Sent false
32437 9725 Acked Read 5 Done Done Sent false
32438 9726 Acked Read 7 Done Done Sent false
32439 9727 Acked Flush 0 Done Done Sent false
32440 9728 Acked Read 4 Done Done Sent false
32441 9729 Acked Write 4 Done Done Sent false
32442 9730 Acked Read 1 Done Done Sent false
32443 9731 Acked Read 2 Done Done Sent false
32444 9732 Acked Read 1 Done Done Sent false
32445 9733 Acked Read 2 Done Done Sent false
32446 9734 Acked Write 10 Done Done Sent false
32447 9735 Acked Read 4 Done Done Sent false
32448 9736 Acked Flush 0 Done Done Sent false
32449 9737 Acked Write 1 Done Done Sent false
32450 9738 Acked Read 5 Done Done Sent false
32451 9739 Acked Read 9 Done Done Sent false
32452 9740 Acked Write 3 Done Done Sent false
32453 9741 Acked Read 1 Done Done Sent false
32454 9742 Acked Read 10 Done Done Sent false
32455 9743 Acked Write 6 Done Done Sent false
32456 9744 Acked Write 8 Done Done Sent false
32457 9745 Acked Write 8 Done Done Sent false
32458 9746 Acked Flush 0 Done Done Sent false
32459 9747 Acked Write 6 Done Done Sent false
32460 9748 Acked Write 8 Done Done Sent false
32461 9749 Acked Write 3 Done Done Sent false
32462 9750 Acked Flush 0 Done Done Sent false
32463 9751 Acked Flush 0 Done Done Sent false
32464 9752 Acked Read 6 Done Done Sent false
32465 9753 Acked Read 7 Done Done Sent false
32466 9754 Acked Read 10 Done Done Sent false
32467 STATES DS:0 DS:1 DS:2 TOTAL
32468 Sent 0 0 301 301
32469 Done 301 301 0 602
32470 Skipped 0 0 0 0
32471 Error 0 0 0 0
32472 Last Flush: Some(JobId(9751)) Some(JobId(9751)) Some(JobId(9453))
32473 Downstairs last five completed: 9453 9452 9451 9450 9449
32474 Upstairs last five completed: 9754 9753 9752 9751 9750
32475 CLIENT: Up:0 ds:301 act:2
324762025-02-25T17:25:17.977ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
324772025-02-25T17:25:18.927ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
324782025-02-25T17:25:19.218ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
324792025-02-25T17:25:19.219ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54315) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
324802025-02-25T17:25:19.220ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
324812025-02-25T17:25:19.220ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
324822025-02-25T17:25:19.220ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
324832025-02-25T17:25:19.220ZINFOcrucible: send last flush ID to this DS: Some(JobId(9453)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
324842025-02-25T17:25:19.220ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
324852025-02-25T17:25:19.220ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9453)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
324862025-02-25T17:25:19.221ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9453)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
32487 ----------------------------------------------------------------
32488 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
32489 Downstairs last five completed: 9755 9754 9753 9752 9751
32490 Upstairs last five completed: 9755 9754 9753 9752 9751
32491 CLIENT: Up:0 ds:0 act:3
32492 Replay: All jobs finished
32493 Main task has work to do, go find it
32494 got dsc Stop(2)
32495 stop 2
32496 Feb 25 17:25:21.478 INFO [8830] Got stop action so:false kr:false
32497 request completed, latency_us: 389, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 4e8ae25b-c587-4d33-bf77-7660eacbdeb6, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
324982025-02-25T17:25:21.479ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
32499 Feb 25 17:25:21.479 INFO request completed, latency_us: 133, response_code: 200, uri{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:25:21.479754545Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
32500 : /state/cid/2, method: GET, req_id: 67b82000-9468-4d94-9b20-fcc920e2c466, remote_addr:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-02-25T17:25:21.4797945Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
32501 127.0.0.1{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-02-25T17:25:21.479827225Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"df7322ab-c489-4662-a11c-3ea4e180b6a3"}
32502 :52576, local_addr: 127.0.0.1:9998
32503 [8830] Exited with: ExitStatus(unix_wait_status(9))
32504 [8830][2] reports Exit
32505 Feb 25 17:25:25.482 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/2, method: GET, req_id: 697f030a-384b-4453-9f77-c1ac5d49ee73, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
32506 001/300 Read block 4124 len 16384
32507 002/300 Read block 3360 len 8192
32508 003/300 Flush
32509 004/300 Write block 501 len 16384 data: 3 3 3 4
32510 005/300 Read block 4249 len 8192
32511 006/300 Write block 3977 len 40960 data: 4 3 3 3 3 3 3 2 2 3
32512 007/300 Read block 2280 len 40960
32513 008/300 Write block 992 len 40960 data: 6 9 8 8 8 5 5 5 4 4
32514 009/300 Read block 4590 len 28672
32515 010/300 Flush
32516 011/300 Read block 345 len 28672
32517 012/300 Read block 4973 len 36864
32518 013/300 Write block 1025 len 8192 data: 8 7
32519 014/300 Write block 4941 len 4096 data: 10
32520 015/300 Write block 2896 len 36864 data: 7 6 5 6 8 11 10 10 9
32521 016/300 Write block 2835 len 40960 data: 8 7 6 7 6 6 6 6 5 4
32522 017/300 Write block 1455 len 8192 data: 8 9
32523 018/300 Write block 2785 len 40960 data: 5 8 8 6 7 6 8 8 8 6
32524 019/300 Read block 3120 len 20480
32525 020/300 Read block 3509 len 20480
32526 021/300 Read block 4859 len 36864
32527 022/300 Write block 910 len 24576 data: 6 4 7 6 7 5
32528 023/300 Read block 3877 len 12288
32529 024/300 Read block 880 len 20480
32530 025/300 Read block 913 len 32768
32531 026/300 Read block 2965 len 28672
32532 027/300 Write block 2767 len 20480 data: 10 8 6 7 6
32533 028/300 Read block 1165 len 16384
32534 029/300 Write block 1837 len 8192 data: 7 8
32535 030/300 Read block 2761 len 24576
32536 031/300 Write block 4946 len 4096 data: 7
32537 032/300 Read block 1262 len 12288
32538 033/300 Write block 2624 len 40960 data: 6 6 6 5 6 6 6 5 6 8
32539 034/300 Write block 3762 len 12288 data: 8 7 7
32540 035/300 Write block 4748 len 40960 data: 7 6 7 7 7 8 7 6 6 6
32541 036/300 Read block 3735 len 36864
32542 037/300 Read block 180 len 20480
32543 038/300 Read block 3699 len 4096
32544 039/300 Read block 1641 len 4096
32545 040/300 Read block 2568 len 20480
32546 041/300 Flush
32547 042/300 Write block 4566 len 16384 data: 5 4 5 5
32548 043/300 Read block 3254 len 12288
32549 044/300 Read block 4208 len 8192
32550 045/300 Write block 2825 len 36864 data: 6 5 6 6 6 7 7 7 8
32551 046/300 Write block 1730 len 28672 data: 5 4 6 6 7 7 6
32552 047/300 Flush
32553 048/300 Write block 2677 len 16384 data: 5 5 4 4
32554 049/300 Read block 1047 len 16384
32555 050/300 Write block 2886 len 28672 data: 5 4 3 3 3 3 4
32556 051/300 Flush
32557 052/300 Read block 2679 len 20480
32558 053/300 Write block 1209 len 4096 data: 4
32559 054/300 Write block 2420 len 16384 data: 5 6 6 6
32560 055/300 Write block 2175 len 16384 data: 3 5 6 7
32561 056/300 Read block 1953 len 16384
32562 057/300 Read block 4246 len 20480
32563 058/300 Read block 695 len 20480
32564 059/300 Read block 3392 len 32768
32565 060/300 Write block 2388 len 28672 data: 5 4 5 5 7 7 7
32566 061/300 Read block 2840 len 12288
32567 062/300 Write block 4178 len 16384 data: 6 6 5 4
32568 063/300 Write block 2210 len 4096 data: 8
32569 064/300 Write block 2742 len 16384 data: 3 5 6 6
32570 065/300 Read block 2556 len 16384
32571 066/300 Read block 1435 len 8192
32572 067/300 Write block 4504 len 36864 data: 2 2 2 2 3 4 4 4 4
32573 068/300 Read block 3633 len 8192
32574 069/300 Read block 2138 len 12288
32575 070/300 Write block 37 len 16384 data: 7 7 6 7
32576 071/300 Flush
32577 072/300 Read block 891 len 12288
32578 073/300 Write block 3286 len 32768 data: 9 9 8 7 7 4 3 3
32579 074/300 Read block 1105 len 28672
32580 075/300 Write block 1640 len 28672 data: 5 5 5 5 4 6 10
32581 076/300 Flush
32582 077/300 Write block 2887 len 32768 data: 5 4 4 4 4 5 6 9
32583 078/300 Write block 759 len 12288 data: 4 5 5
32584 079/300 Read block 2815 len 4096
32585 080/300 Flush
32586 081/300 Write block 4519 len 8192 data: 8 9
32587 082/300 Read block 681 len 8192
32588 083/300 Write block 4362 len 4096 data: 8
32589 084/300 Flush
32590 085/300 Write block 2918 len 16384 data: 7 6 6 5
32591 086/300 Read block 386 len 24576
32592 087/300 Read block 3669 len 20480
32593 088/300 Read block 3473 len 12288
32594 089/300 Write block 3902 len 24576 data: 4 3 4 5 4 4
32595 090/300 Write block 3494 len 36864 data: 5 3 5 7 7 6 6 7 7
32596 091/300 Write block 1991 len 20480 data: 7 6 5 7 7
32597 092/300 Flush
32598 093/300 Flush
32599 094/300 Flush
32600 095/300 Read block 452 len 36864
32601 096/300 Read block 3337 len 32768
32602 097/300 Flush
32603 098/300 Write block 829 len 28672 data: 4 5 5 9 11 12 10
32604 099/300 Read block 1845 len 16384
32605 100/300 Read block 1379 len 16384
32606 101/300 Read block 1779 len 4096
32607 102/300 Read block 4460 len 12288
32608 103/300 Write block 4850 len 12288 data: 6 5 5
32609 104/300 Read block 1867 len 16384
32610 105/300 Read block 1047 len 4096
32611 106/300 Write block 3421 len 32768 data: 4 4 3 3 6 8 8 8
32612 107/300 Read block 1268 len 16384
32613 108/300 Read block 371 len 8192
32614 109/300 Write block 1168 len 40960 data: 7 7 8 7 7 6 7 8 8 8
32615 110/300 Read block 1952 len 28672
32616 111/300 Flush
32617 112/300 Write block 825 len 12288 data: 5 3 3
32618 113/300 Read block 940 len 20480
32619 114/300 Write block 3460 len 32768 data: 5 5 6 6 5 6 5 5
32620 115/300 Write block 2118 len 40960 data: 7 5 4 6 6 6 5 5 5 6
32621 116/300 Write block 826 len 40960 data: 4 4 3 5 6 6 10 12 13 11
32622 117/300 Read block 1850 len 24576
32623 118/300 Read block 2604 len 24576
32624 119/300 Write block 2501 len 4096 data: 5
32625 120/300 Read block 1451 len 24576
32626 121/300 Read block 4857 len 36864
32627 122/300 Read block 4029 len 20480
32628 123/300 Write block 3779 len 4096 data: 4
32629 124/300 Read block 793 len 28672
32630 125/300 Read block 1600 len 24576
32631 126/300 Write block 2500 len 8192 data: 6 6
32632 127/300 Write block 3639 len 12288 data: 5 4 4
32633 128/300 Write block 3774 len 8192 data: 6 4
32634 129/300 Write block 1939 len 32768 data: 12 12 10 11 9 8 10 10
32635 130/300 Write block 575 len 40960 data: 5 4 4 6 5 7 6 5 4 6
32636 131/300 Read block 1047 len 8192
32637 132/300 Flush
32638 133/300 Read block 2476 len 12288
32639 134/300 Write block 3692 len 32768 data: 3 4 5 5 6 6 8 9
32640 135/300 Read block 3538 len 16384
32641 136/300 Read block 2734 len 12288
32642 137/300 Flush
32643 138/300 Read block 4960 len 24576
32644 139/300 Read block 3244 len 4096
32645 140/300 Write block 1242 len 36864 data: 5 5 3 4 4 5 6 4 5
32646 141/300 Read block 4113 len 8192
32647 142/300 Write block 4670 len 24576 data: 7 6 6 7 6 5
32648 143/300 Write block 1484 len 20480 data: 8 9 9 7 7
32649 144/300 Read block 1149 len 32768
32650 145/300 Write block 3860 len 8192 data: 4 3
32651 146/300 Read block 1316 len 28672
32652 147/300 Write block 3765 len 12288 data: 6 8 7
32653 148/300 Read block 1874 len 12288
32654 149/300 Read block 3164 len 4096
32655 150/300 Write block 271 len 12288 data: 9 10 9
32656 151/300 Write block 4224 len 24576 data: 5 5 4 4 4 3
32657 152/300 Write block 2874 len 28672 data: 5 7 7 7 6 6 6
32658 153/300 Read block 4563 len 4096
32659 154/300 Read block 2720 len 36864
32660 155/300 Write block 2682 len 4096 data: 5
32661 156/300 Write block 3710 len 12288 data: 9 9 7
32662 157/300 Read block 1488 len 36864
32663 158/300 Write block 776 len 36864 data: 8 8 8 10 9 9 8 7 5
32664 159/300 Read block 2975 len 36864
32665 160/300 Read block 2570 len 8192
32666 161/300 Write block 4053 len 20480 data: 3 3 4 3 3
32667 162/300 Read block 4615 len 24576
32668 163/300 Write block 4057 len 32768 data: 4 3 4 6 9 8 8 11
32669 164/300 Flush
32670 165/300 Write block 708 len 28672 data: 4 5 5 6 6 7 7
32671 166/300 Write block 90 len 40960 data: 4 3 4 5 6 6 6 7 7 7
32672 167/300 Read block 2035 len 40960
32673 168/300 Read block 466 len 16384
32674 169/300 Read block 2986 len 28672
32675 170/300 Write block 3304 len 12288 data: 8 9 7
32676 171/300 Read block 1406 len 36864
32677 172/300 Write block 4477 len 40960 data: 4 4 4 4 3 2 2 2 2 2
32678 173/300 Read block 4615 len 28672
32679 174/300 Read block 2183 len 24576
32680 175/300 Read block 899 len 40960
32681 176/300 Read block 3017 len 12288
32682 177/300 Read block 4823 len 24576
32683 178/300 Write block 2838 len 8192 data: 8 7
32684 179/300 Write block 1773 len 36864 data: 6 7 5 5 5 4 6 5 4
32685 180/300 Write block 2297 len 8192 data: 7 7
32686 181/300 Flush
32687 182/300 Read block 260 len 8192
32688 183/300 Read block 4151 len 20480
32689 184/300 Read block 1809 len 16384
32690 185/300 Read block 4763 len 12288
32691 186/300 Read block 2172 len 28672
32692 187/300 Read block 1711 len 4096
32693 188/300 Flush
32694 189/300 Read block 1247 len 36864
32695 190/300 Read block 3940 len 32768
32696 191/300 Read block 1472 len 24576
32697 192/300 Read block 2596 len 12288
32698 193/300 Write block 338 len 8192 data: 5 4
32699 194/300 Write block 552 len 24576 data: 6 6 7 7 7 6
32700 195/300 Read block 2709 len 8192
32701 196/300 Read block 371 len 24576
32702 197/300 Flush
32703 198/300 Write block 915 len 8192 data: 6 6
32704 199/300 Read block 3557 len 20480
32705 200/300 Read block 1061 len 12288
32706 201/300 Read block 2295 len 36864
32707 202/300 Read block 3850 len 28672
32708 203/300 Write block 501 len 8192 data: 4 4
32709 204/300 Read block 3071 len 28672
32710 205/300 Write block 2769 len 36864 data: 7 8 7 7 6 6 7 8 5
32711 206/300 Flush
32712 207/300 Write block 1486 len 16384 data: 10 8 8 8
32713 208/300 Write block 3191 len 32768 data: 3 5 6 6 6 5 7 7
32714 209/300 Write block 3794 len 36864 data: 7 9 10 8 8 6 5 5 6
32715 210/300 Read block 582 len 40960
32716 211/300 Flush
32717 212/300 Write block 1239 len 40960 data: 7 6 7 6 6 4 5 5 6 7
32718 213/300 Read block 2153 len 20480
32719 214/300 Write block 557 len 24576 data: 7 5 5 6 7 5
32720 215/300 Write block 4831 len 32768 data: 5 5 5 4 4 3 2 3
32721 216/300 Read block 4252 len 4096
32722 217/300 Read block 1528 len 20480
32723 218/300 Write block 752 len 40960 data: 8 8 8 7 4 3 4 5 6 6
32724 219/300 Write block 585 len 28672 data: 7 8 5 6 7 6 5
32725 220/300 Write block 3605 len 12288 data: 5 6 7
32726 221/300 Write block 406 len 40960 data: 6 6 5 5 7 8 8 8 8 8
32727 222/300 Write block 676 len 8192 data: 7 4
32728 223/300 Write block 2174 len 24576 data: 4 4 6 7 8 7
32729 224/300 Write block 3542 len 4096 data: 7
32730 225/300 Read block 36 len 24576
32731 226/300 Read block 4931 len 8192
32732 227/300 Write block 3222 len 32768 data: 11 10 10 9 8 7 6 7
32733 228/300 Flush
32734 229/300 Write block 4848 len 20480 data: 5 6 7 6 6
32735 230/300 Read block 49 len 12288
32736 231/300 Read block 681 len 20480
32737 232/300 Read block 3443 len 8192
32738 233/300 Write block 1687 len 20480 data: 6 4 4 4 3
32739 234/300 Write block 4913 len 12288 data: 6 7 6
32740 235/300 Write block 3788 len 24576 data: 6 6 6 7 7 7
32741 236/300 Read block 1957 len 32768
32742 237/300 Write block 2390 len 16384 data: 6 6 8 8
32743 238/300 Write block 552 len 4096 data: 7
32744 239/300 Read block 4991 len 36864
32745 240/300 Write block 109 len 24576 data: 7 8 6 6 6 5
32746 241/300 Write block 4982 len 4096 data: 6
32747 242/300 Write block 2301 len 40960 data: 5 7 9 9 7 5 6 5 5 5
32748 243/300 Read block 1400 len 28672
32749 244/300 Read block 3927 len 32768
32750 245/300 Read block 2827 len 32768
32751 246/300 Write block 3503 len 36864 data: 6 6 6 5 5 5 4 6 5
32752 247/300 Write block 4861 len 36864 data: 4 4 4 4 5 5 5 5 5
32753 248/300 Write block 818 len 8192 data: 4 3
32754 249/300 Read block 1721 len 8192
32755 250/300 Read block 1639 len 4096
32756 251/300 Read block 4064 len 32768
32757 252/300 Write block 1764 len 8192 data: 6 5
32758 253/300 Flush
32759 254/300 Read block 730 len 24576
32760 255/300 Read block 3642 len 12288
32761 256/300 Write block 4681 len 20480 data: 3 3 3 3 2
32762 257/300 Write block 4645 len 40960 data: 5 5 5 3 4 4 6 7 7 6
32763 258/300 Read block 3126 len 4096
32764 259/300 Read block 3297 len 16384
32765 260/300 Read block 4243 len 8192
32766 261/300 Write block 2505 len 28672 data: 8 7 8 8 7 5 5
32767 262/300 Write block 3483 len 16384 data: 4 5 6 6
32768 263/300 Write block 2207 len 24576 data: 8 5 7 9 10 8
32769 264/300 Read block 485 len 40960
32770 265/300 Write block 1366 len 16384 data: 9 8 6 7
32771 266/300 Write block 357 len 32768 data: 6 5 5 5 5 2 2 2
32772 267/300 Flush
32773 268/300 Read block 3219 len 32768
32774 269/300 Read block 139 len 8192
32775 270/300 Write block 1614 len 20480 data: 7 6 7 7 7
32776 271/300 Flush
32777 272/300 Flush
32778 273/300 Write block 3692 len 20480 data: 4 5 6 6 7
32779 274/300 Read block 2431 len 40960
32780 275/300 Write block 776 len 16384 data: 9 9 9 11
32781 276/300 Write block 4575 len 28672 data: 5 5 7 8 7 6 6
32782 277/300 Write block 819 len 32768 data: 4 3 4 5 6 6 6 5
32783 278/300 Read block 1222 len 24576
32784 279/300 Read block 2906 len 4096
32785 280/300 Write block 1519 len 12288 data: 6 5 5
32786 281/300 Read block 4533 len 24576
32787 282/300 Write block 3291 len 40960 data: 5 4 4 4 4 7 7 5 6 6
32788 283/300 Read block 2066 len 8192
32789 284/300 Write block 550 len 20480 data: 5 6 8 7 8
32790 285/300 Read block 1546 len 8192
32791 286/300 Read block 980 len 40960
32792 287/300 Read block 729 len 12288
32793 288/300 Write block 2364 len 4096 data: 5
32794 289/300 Read block 4432 len 16384
32795 290/300 Write block 347 len 40960 data: 2 3 4 3 3 3 2 2 4 6
32796 291/300 Read block 876 len 40960
32797 292/300 Read block 717 len 32768
32798 293/300 Read block 206 len 16384
32799 294/300 Write block 4872 len 4096 data: 6
32800 295/300 Write block 3634 len 8192 data: 5 5
32801 296/300 Read block 3652 len 20480
32802 297/300 Read block 4803 len 28672
32803 298/300 Write block 2601 len 12288 data: 8 6 7
32804 299/300 Write block 3347 len 8192 data: 5 5
32805 300/300 Write block 575 len 20480 data: 6 5 5 7 6
32806 301/300 Write block 3104 len 36864 data: 9 5 5 6 5 5 8 7 6
32807 Main task has work to do, go find it
32808 got dsc Start(2)
32809 start 2
32810 Feb 25 17:25:25.723 INFO request completed, latency_us: 223, response_code: 204, uri: /start/cid/2, method: POST, req_id: 84e56637-bece-445d-bda2-9b11397ec476, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
32811 Replay: started 2, returned:Ok(())
32812 ----------------------------------------------------------------
32813 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32814 States: Active Active Offline
32815 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32816 9756 Acked Read 4 Done Done Sent false
32817 9757 Acked Read 2 Done Done Sent false
32818 9758 Acked Flush 0 Done Done Sent false
32819 9759 Acked Write 4 Done Done Sent false
32820 9760 Acked Read 2 Done Done Sent false
32821 9761 Acked Write 10 Done Done Sent false
32822 9762 Acked Read 10 Done Done Sent false
32823 9763 Acked Write 10 Done Done Sent false
32824 9764 Acked Read 7 Done Done Sent false
32825 9765 Acked Flush 0 Done Done Sent false
32826 9766 Acked Read 7 Done Done Sent false
32827 9767 Acked Read 9 Done Done Sent false
32828 9768 Acked Write 2 Done Done Sent false
32829 9769 Acked Write 1 Done Done Sent false
32830 9770 Acked Write 9 Done Done Sent false
32831 9771 Acked Write 10 Done Done Sent false
32832 9772 Acked Write 2 Done Done Sent false
32833 9773 Acked Write 10 Done Done Sent false
32834 9774 Acked Read 5 Done Done Sent false
32835 9775 Acked Read 5 Done Done Sent false
32836 9776 Acked Read 9 Done Done Sent false
32837 9777 Acked Write 6 Done Done Sent false
32838 9778 Acked Read 3 Done Done Sent false
32839 9779 Acked Read 5 Done Done Sent false
32840 9780 Acked Read 8 Done Done Sent false
32841 9781 Acked Read 7 Done Done Sent false
32842 9782 Acked Write 5 Done Done Sent false
32843 9783 Acked Read 4 Done Done Sent false
32844 9784 Acked Write 2 Done Done Sent false
32845 9785 Acked Read 6 Done Done Sent false
32846 9786 Acked Write 1 Done Done Sent false
32847 9787 Acked Read 3 Done Done Sent false
32848 9788 Acked Write 10 Done Done Sent false
32849 9789 Acked Write 3 Done Done Sent false
32850 9790 Acked Write 10 Done Done Sent false
32851 9791 Acked Read 9 Done Done Sent false
32852 9792 Acked Read 5 Done Done Sent false
32853 9793 Acked Read 1 Done Done Sent false
32854 9794 Acked Read 1 Done Done Sent false
32855 9795 Acked Read 5 Done Done Sent false
32856 9796 Acked Flush 0 Done Done Sent false
32857 9797 Acked Write 4 Done Done Sent false
32858 9798 Acked Read 3 Done Done Sent false
32859 9799 Acked Read 2 Done Done Sent false
32860 9800 Acked Write 9 Done Done Sent false
32861 9801 Acked Write 7 Done Done Sent false
32862 9802 Acked Flush 0 Done Done Sent false
32863 9803 Acked Write 4 Done Done Sent false
32864 9804 Acked Read 4 Done Done Sent false
32865 9805 Acked Write 7 Done Done Sent false
32866 9806 Acked Flush 0 Done Done Sent false
32867 9807 Acked Read 5 Done Done Sent false
32868 9808 Acked Write 1 Done Done Sent false
32869 9809 Acked Write 4 Done Done Sent false
32870 9810 Acked Write 4 Done Done Sent false
32871 9811 Acked Read 4 Done Done Sent false
32872 9812 Acked Read 5 Done Done Sent false
32873 9813 Acked Read 5 Done Done Sent false
32874 9814 Acked Read 8 Done Done Sent false
32875 9815 Acked Write 7 Done Done Sent false
32876 9816 Acked Read 3 Done Done Sent false
32877 9817 Acked Write 4 Done Done Sent false
32878 9818 Acked Write 1 Done Done Sent false
32879 9819 Acked Write 4 Done Done Sent false
32880 9820 Acked Read 4 Done Done Sent false
32881 9821 Acked Read 2 Done Done Sent false
32882 9822 Acked Write 9 Done Done Sent false
32883 9823 Acked Read 2 Done Done Sent false
32884 9824 Acked Read 3 Done Done Sent false
32885 9825 Acked Write 4 Done Done Sent false
32886 9826 Acked Flush 0 Done Done Sent false
32887 9827 Acked Read 3 Done Done Sent false
32888 9828 Acked Write 8 Done Done Sent false
32889 9829 Acked Read 7 Done Done Sent false
32890 9830 Acked Write 7 Done Done Sent false
32891 9831 Acked Flush 0 Done Done Sent false
32892 9832 Acked Write 8 Done Done Sent false
32893 9833 Acked Write 3 Done Done Sent false
32894 9834 Acked Read 1 Done Done Sent false
32895 9835 Acked Flush 0 Done Done Sent false
32896 9836 Acked Write 2 Done Done Sent false
32897 9837 Acked Read 2 Done Done Sent false
32898 9838 Acked Write 1 Done Done Sent false
32899 9839 Acked Flush 0 Done Done Sent false
32900 9840 Acked Write 4 Done Done Sent false
32901 9841 Acked Read 6 Done Done Sent false
32902 9842 Acked Read 5 Done Done Sent false
32903 9843 Acked Read 3 Done Done Sent false
32904 9844 Acked Write 6 Done Done Sent false
32905 9845 Acked Write 9 Done Done Sent false
32906 9846 Acked Write 5 Done Done Sent false
32907 9847 Acked Flush 0 Done Done Sent false
32908 9848 Acked Flush 0 Done Done Sent false
32909 9849 Acked Flush 0 Done Done Sent false
32910 9850 Acked Read 9 Done Done Sent false
32911 9851 Acked Read 8 Done Done Sent false
32912 9852 Acked Flush 0 Done Done Sent false
32913 9853 Acked Write 7 Done Done Sent false
32914 9854 Acked Read 4 Done Done Sent false
32915 9855 Acked Read 4 Done Done Sent false
32916 9856 Acked Read 1 Done Done Sent false
32917 9857 Acked Read 3 Done Done Sent false
32918 9858 Acked Write 3 Done Done Sent false
32919 9859 Acked Read 4 Done Done Sent false
32920 9860 Acked Read 1 Done Done Sent false
32921 9861 Acked Write 8 Done Done Sent false
32922 9862 Acked Read 4 Done Done Sent false
32923 9863 Acked Read 2 Done Done Sent false
32924 9864 Acked Write 10 Done Done Sent false
32925 9865 Acked Read 7 Done Done Sent false
32926 9866 Acked Flush 0 Done Done Sent false
32927 9867 Acked Write 3 Done Done Sent false
32928 9868 Acked Read 5 Done Done Sent false
32929 9869 Acked Write 8 Done Done Sent false
32930 9870 Acked Write 10 Done Done Sent false
32931 9871 Acked Write 10 Done Done Sent false
32932 9872 Acked Read 6 Done Done Sent false
32933 9873 Acked Read 6 Done Done Sent false
32934 9874 Acked Write 1 Done Done Sent false
32935 9875 Acked Read 6 Done Done Sent false
32936 9876 Acked Read 9 Done Done Sent false
32937 9877 Acked Read 5 Done Done Sent false
32938 9878 Acked Write 1 Done Done Sent false
32939 9879 Acked Read 7 Done Done Sent false
32940 9880 Acked Read 6 Done Done Sent false
32941 9881 Acked Write 2 Done Done Sent false
32942 9882 Acked Write 3 Done Done Sent false
32943 9883 Acked Write 2 Done Done Sent false
32944 9884 Acked Write 8 Done Done Sent false
32945 9885 Acked Write 10 Done Done Sent false
32946 9886 Acked Read 2 Done Done Sent false
32947 9887 Acked Flush 0 Done Done Sent false
32948 9888 Acked Read 3 Done Done Sent false
32949 9889 Acked Write 8 Done Done Sent false
32950 9890 Acked Read 4 Done Done Sent false
32951 9891 Acked Read 3 Done Done Sent false
32952 9892 Acked Flush 0 Done Done Sent false
32953 9893 Acked Read 6 Done Done Sent false
32954 9894 Acked Read 1 Done Done Sent false
32955 9895 Acked Write 9 Done Done Sent false
32956 9896 Acked Read 2 Done Done Sent false
32957 9897 Acked Write 6 Done Done Sent false
32958 9898 Acked Write 5 Done Done Sent false
32959 9899 Acked Read 8 Done Done Sent false
32960 9900 Acked Write 2 Done Done Sent false
32961 9901 Acked Read 7 Done Done Sent false
32962 9902 Acked Write 3 Done Done Sent false
32963 9903 Acked Read 3 Done Done Sent false
32964 9904 Acked Read 1 Done Done Sent false
32965 9905 Acked Write 3 Done Done Sent false
32966 9906 Acked Write 6 Done Done Sent false
32967 9907 Acked Write 7 Done Done Sent false
32968 9908 Acked Read 1 Done Done Sent false
32969 9909 Acked Read 9 Done Done Sent false
32970 9910 Acked Write 1 Done Done Sent false
32971 9911 Acked Write 3 Done Done Sent false
32972 9912 Acked Read 9 Done Done Sent false
32973 9913 Acked Write 9 Done Done Sent false
32974 9914 Acked Read 9 Done Done Sent false
32975 9915 Acked Read 2 Done Done Sent false
32976 9916 Acked Write 5 Done Done Sent false
32977 9917 Acked Read 6 Done Done Sent false
32978 9918 Acked Write 8 Done Done Sent false
32979 9919 Acked Flush 0 Done Done Sent false
32980 9920 Acked Write 7 Done Done Sent false
32981 9921 Acked Write 10 Done Done Sent false
32982 9922 Acked Read 10 Done Done Sent false
32983 9923 Acked Read 4 Done Done Sent false
32984 9924 Acked Read 7 Done Done Sent false
32985 9925 Acked Write 3 Done Done Sent false
32986 9926 Acked Read 9 Done Done Sent false
32987 9927 Acked Write 10 Done Done Sent false
32988 9928 Acked Read 7 Done Done Sent false
32989 9929 Acked Read 6 Done Done Sent false
32990 9930 Acked Read 10 Done Done Sent false
32991 9931 Acked Read 3 Done Done Sent false
32992 9932 Acked Read 6 Done Done Sent false
32993 9933 Acked Write 2 Done Done Sent false
32994 9934 Acked Write 9 Done Done Sent false
32995 9935 Acked Write 2 Done Done Sent false
32996 9936 Acked Flush 0 Done Done Sent false
32997 9937 Acked Read 2 Done Done Sent false
32998 9938 Acked Read 5 Done Done Sent false
32999 9939 Acked Read 4 Done Done Sent false
33000 9940 Acked Read 3 Done Done Sent false
33001 9941 Acked Read 7 Done Done Sent false
33002 9942 Acked Read 1 Done Done Sent false
33003 9943 Acked Flush 0 Done Done Sent false
33004 9944 Acked Read 9 Done Done Sent false
33005 9945 Acked Read 8 Done Done Sent false
33006 9946 Acked Read 6 Done Done Sent false
33007 9947 Acked Read 3 Done Done Sent false
33008 9948 Acked Write 2 Done Done Sent false
33009 9949 Acked Write 6 Done Done Sent false
33010 9950 Acked Read 2 Done Done Sent false
33011 9951 Acked Read 6 Done Done Sent false
33012 9952 Acked Flush 0 Done Done Sent false
33013 9953 Acked Write 2 Done Done Sent false
33014 9954 Acked Read 5 Done Done Sent false
33015 9955 Acked Read 3 Done Done Sent false
33016 9956 Acked Read 9 Done Done Sent false
33017 9957 Acked Read 7 Done Done Sent false
33018 9958 Acked Write 2 Done Done Sent false
33019 9959 Acked Read 7 Done Done Sent false
33020 9960 Acked Write 9 Done Done Sent false
33021 9961 Acked Flush 0 Done Done Sent false
33022 9962 Acked Write 4 Done Done Sent false
33023 9963 Acked Write 8 Done Done Sent false
33024 9964 Acked Write 9 Done Done Sent false
33025 9965 Acked Read 10 Done Done Sent false
33026 9966 Acked Flush 0 Done Done Sent false
33027 9967 Acked Write 10 Done Done Sent false
33028 9968 Acked Read 5 Done Done Sent false
33029 9969 Acked Write 6 Done Done Sent false
33030 9970 Acked Write 8 Done Done Sent false
33031 9971 Acked Read 1 Done Done Sent false
33032 9972 Acked Read 5 Done Done Sent false
33033 9973 Acked Write 10 Done Done Sent false
33034 9974 Acked Write 7 Done Done Sent false
33035 9975 Acked Write 3 Done Done Sent false
33036 9976 Acked Write 10 Done Done Sent false
33037 9977 Acked Write 2 Done Done Sent false
33038 9978 Acked Write 6 Done Done Sent false
33039 9979 Acked Write 1 Done Done Sent false
33040 9980 Acked Read 6 Done Done Sent false
33041 9981 Acked Read 2 Done Done Sent false
33042 9982 Acked Write 8 Done Done Sent false
33043 9983 Acked Flush 0 Done Done Sent false
33044 9984 Acked Write 5 Done Done Sent false
33045 9985 Acked Read 3 Done Done Sent false
33046 9986 Acked Read 5 Done Done Sent false
33047 9987 Acked Read 2 Done Done Sent false
33048 9988 Acked Write 5 Done Done Sent false
33049 9989 Acked Write 3 Done Done Sent false
33050 9990 Acked Write 6 Done Done Sent false
33051 9991 Acked Read 8 Done Done Sent false
33052 9992 Acked Write 4 Done Done Sent false
33053 9993 Acked Write 1 Done Done Sent false
33054 9994 Acked Read 9 Done Done Sent false
33055 9995 Acked Write 6 Done Done Sent false
33056 9996 Acked Write 1 Done Done Sent false
33057 9997 Acked Write 10 Done Done Sent false
33058 9998 Acked Read 7 Done Done Sent false
33059 9999 Acked Read 8 Done Done Sent false
33060 10000 Acked Read 8 Done Done Sent false
33061 10001 Acked Write 9 Done Done Sent false
33062 10002 Acked Write 9 Done Done Sent false
33063 10003 Acked Write 2 Done Done Sent false
33064 10004 Acked Read 2 Done Done Sent false
33065 10005 Acked Read 1 Done Done Sent false
33066 10006 Acked Read 8 Done Done Sent false
33067 10007 Acked Write 2 Done Done Sent false
33068 10008 Acked Flush 0 Done Done Sent false
33069 10009 Acked Read 6 Done Done Sent false
33070 10010 Acked Read 3 Done Done Sent false
33071 10011 Acked Write 5 Done Done Sent false
33072 10012 Acked Write 10 Done Done Sent false
33073 10013 Acked Read 1 Done Done Sent false
33074 10014 Acked Read 4 Done Done Sent false
33075 10015 Acked Read 2 Done Done Sent false
33076 10016 Acked Write 7 Done Done Sent false
33077 10017 Acked Write 4 Done Done Sent false
33078 10018 Acked Write 6 Done Done Sent false
33079 10019 Acked Read 10 Done Done Sent false
33080 10020 Acked Write 4 Done Done Sent false
33081 10021 Acked Write 8 Done Done Sent false
33082 10022 Acked Flush 0 Done Done Sent false
33083 10023 Acked Read 8 Done Done Sent false
33084 10024 Acked Read 2 Done Done Sent false
33085 10025 Acked Write 5 Done Done Sent false
33086 10026 Acked Flush 0 Done Done Sent false
33087 10027 Acked Flush 0 Done Done Sent false
33088 10028 Acked Write 5 Done Done Sent false
33089 10029 Acked Read 10 Done Done Sent false
33090 10030 Acked Write 4 Done Done Sent false
33091 10031 Acked Write 7 Done Done Sent false
33092 10032 Acked Write 8 Done Done Sent false
33093 10033 Acked Read 6 Done Done Sent false
33094 10034 Acked Read 1 Done Done Sent false
33095 10035 Acked Write 3 Done Done Sent false
33096 10036 Acked Read 6 Done Done Sent false
33097 10037 Acked Write 10 Done Done Sent false
33098 10038 Acked Read 2 Done Done Sent false
33099 10039 Acked Write 5 Done Done Sent false
33100 10040 Acked Read 2 Done Done Sent false
33101 10041 Acked Read 10 Done Done Sent false
33102 10042 Acked Read 3 Done Done Sent false
33103 10043 Acked Write 1 Done Done Sent false
33104 10044 Acked Read 4 Done Done Sent false
33105 10045 Acked Write 10 Done Done Sent false
33106 10046 Acked Read 10 Done Done Sent false
33107 10047 Acked Read 8 Done Done Sent false
33108 10048 Acked Read 4 Done Done Sent false
33109 10049 Acked Write 1 Done Done Sent false
33110 10050 Acked Write 2 Done Done Sent false
33111 10051 Acked Read 5 Done Done Sent false
33112 10052 Acked Read 7 Done Done Sent false
33113 10053 Acked Write 3 Done Done Sent false
33114 10054 Acked Write 2 Done Done Sent false
33115 10055 Acked Write 5 Done Done Sent false
33116 10056 Acked Write 9 Done Done Sent false
33117 STATES DS:0 DS:1 DS:2 TOTAL
33118 Sent 0 0 301 301
33119 Done 301 301 0 602
33120 Skipped 0 0 0 0
33121 Error 0 0 0 0
33122 Last Flush: Some(JobId(10027)) Some(JobId(10027)) Some(JobId(9755))
33123 Downstairs last five completed: 9755 9754 9753 9752 9751
33124 Upstairs last five completed: 10056 10055 10054 10053 10052
33125 CLIENT: Up:0 ds:301 act:2
331262025-02-25T17:25:26.225ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
331272025-02-25T17:25:26.941ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
33128 [8830] got start action
33129 [8830] I am going to restart
33130 Starting downstairs at port 8830
33131 Make output file at "/tmp/dsc/downstairs-8830.txt"
33132 [8830][2] reports Starting
33133 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1238)
33134 [8830][2] reports Running
331352025-02-25T17:25:27.942ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
331362025-02-25T17:25:28.943ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
33137 ----------------------------------------------------------------
33138 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33139 States: Active Active Offline
33140 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33141 9756 Acked Read 4 Done Done Sent false
33142 9757 Acked Read 2 Done Done Sent false
33143 9758 Acked Flush 0 Done Done Sent false
33144 9759 Acked Write 4 Done Done Sent false
33145 9760 Acked Read 2 Done Done Sent false
33146 9761 Acked Write 10 Done Done Sent false
33147 9762 Acked Read 10 Done Done Sent false
33148 9763 Acked Write 10 Done Done Sent false
33149 9764 Acked Read 7 Done Done Sent false
33150 9765 Acked Flush 0 Done Done Sent false
33151 9766 Acked Read 7 Done Done Sent false
33152 9767 Acked Read 9 Done Done Sent false
33153 9768 Acked Write 2 Done Done Sent false
33154 9769 Acked Write 1 Done Done Sent false
33155 9770 Acked Write 9 Done Done Sent false
33156 9771 Acked Write 10 Done Done Sent false
33157 9772 Acked Write 2 Done Done Sent false
33158 9773 Acked Write 10 Done Done Sent false
33159 9774 Acked Read 5 Done Done Sent false
33160 9775 Acked Read 5 Done Done Sent false
33161 9776 Acked Read 9 Done Done Sent false
33162 9777 Acked Write 6 Done Done Sent false
33163 9778 Acked Read 3 Done Done Sent false
33164 9779 Acked Read 5 Done Done Sent false
33165 9780 Acked Read 8 Done Done Sent false
33166 9781 Acked Read 7 Done Done Sent false
33167 9782 Acked Write 5 Done Done Sent false
33168 9783 Acked Read 4 Done Done Sent false
33169 9784 Acked Write 2 Done Done Sent false
33170 9785 Acked Read 6 Done Done Sent false
33171 9786 Acked Write 1 Done Done Sent false
33172 9787 Acked Read 3 Done Done Sent false
33173 9788 Acked Write 10 Done Done Sent false
33174 9789 Acked Write 3 Done Done Sent false
33175 9790 Acked Write 10 Done Done Sent false
33176 9791 Acked Read 9 Done Done Sent false
33177 9792 Acked Read 5 Done Done Sent false
33178 9793 Acked Read 1 Done Done Sent false
33179 9794 Acked Read 1 Done Done Sent false
33180 9795 Acked Read 5 Done Done Sent false
33181 9796 Acked Flush 0 Done Done Sent false
33182 9797 Acked Write 4 Done Done Sent false
33183 9798 Acked Read 3 Done Done Sent false
33184 9799 Acked Read 2 Done Done Sent false
33185 9800 Acked Write 9 Done Done Sent false
33186 9801 Acked Write 7 Done Done Sent false
33187 9802 Acked Flush 0 Done Done Sent false
33188 9803 Acked Write 4 Done Done Sent false
33189 9804 Acked Read 4 Done Done Sent false
33190 9805 Acked Write 7 Done Done Sent false
33191 9806 Acked Flush 0 Done Done Sent false
33192 9807 Acked Read 5 Done Done Sent false
33193 9808 Acked Write 1 Done Done Sent false
33194 9809 Acked Write 4 Done Done Sent false
33195 9810 Acked Write 4 Done Done Sent false
33196 9811 Acked Read 4 Done Done Sent false
33197 9812 Acked Read 5 Done Done Sent false
33198 9813 Acked Read 5 Done Done Sent false
33199 9814 Acked Read 8 Done Done Sent false
33200 9815 Acked Write 7 Done Done Sent false
33201 9816 Acked Read 3 Done Done Sent false
33202 9817 Acked Write 4 Done Done Sent false
33203 9818 Acked Write 1 Done Done Sent false
33204 9819 Acked Write 4 Done Done Sent false
33205 9820 Acked Read 4 Done Done Sent false
33206 9821 Acked Read 2 Done Done Sent false
33207 9822 Acked Write 9 Done Done Sent false
33208 9823 Acked Read 2 Done Done Sent false
33209 9824 Acked Read 3 Done Done Sent false
33210 9825 Acked Write 4 Done Done Sent false
33211 9826 Acked Flush 0 Done Done Sent false
33212 9827 Acked Read 3 Done Done Sent false
33213 9828 Acked Write 8 Done Done Sent false
33214 9829 Acked Read 7 Done Done Sent false
33215 9830 Acked Write 7 Done Done Sent false
33216 9831 Acked Flush 0 Done Done Sent false
33217 9832 Acked Write 8 Done Done Sent false
33218 9833 Acked Write 3 Done Done Sent false
33219 9834 Acked Read 1 Done Done Sent false
33220 9835 Acked Flush 0 Done Done Sent false
33221 9836 Acked Write 2 Done Done Sent false
33222 9837 Acked Read 2 Done Done Sent false
33223 9838 Acked Write 1 Done Done Sent false
33224 9839 Acked Flush 0 Done Done Sent false
33225 9840 Acked Write 4 Done Done Sent false
33226 9841 Acked Read 6 Done Done Sent false
33227 9842 Acked Read 5 Done Done Sent false
33228 9843 Acked Read 3 Done Done Sent false
33229 9844 Acked Write 6 Done Done Sent false
33230 9845 Acked Write 9 Done Done Sent false
33231 9846 Acked Write 5 Done Done Sent false
33232 9847 Acked Flush 0 Done Done Sent false
33233 9848 Acked Flush 0 Done Done Sent false
33234 9849 Acked Flush 0 Done Done Sent false
33235 9850 Acked Read 9 Done Done Sent false
33236 9851 Acked Read 8 Done Done Sent false
33237 9852 Acked Flush 0 Done Done Sent false
33238 9853 Acked Write 7 Done Done Sent false
33239 9854 Acked Read 4 Done Done Sent false
33240 9855 Acked Read 4 Done Done Sent false
33241 9856 Acked Read 1 Done Done Sent false
33242 9857 Acked Read 3 Done Done Sent false
33243 9858 Acked Write 3 Done Done Sent false
33244 9859 Acked Read 4 Done Done Sent false
33245 9860 Acked Read 1 Done Done Sent false
33246 9861 Acked Write 8 Done Done Sent false
33247 9862 Acked Read 4 Done Done Sent false
33248 9863 Acked Read 2 Done Done Sent false
33249 9864 Acked Write 10 Done Done Sent false
33250 9865 Acked Read 7 Done Done Sent false
33251 9866 Acked Flush 0 Done Done Sent false
33252 9867 Acked Write 3 Done Done Sent false
33253 9868 Acked Read 5 Done Done Sent false
33254 9869 Acked Write 8 Done Done Sent false
33255 9870 Acked Write 10 Done Done Sent false
33256 9871 Acked Write 10 Done Done Sent false
33257 9872 Acked Read 6 Done Done Sent false
33258 9873 Acked Read 6 Done Done Sent false
33259 9874 Acked Write 1 Done Done Sent false
33260 9875 Acked Read 6 Done Done Sent false
33261 9876 Acked Read 9 Done Done Sent false
33262 9877 Acked Read 5 Done Done Sent false
33263 9878 Acked Write 1 Done Done Sent false
33264 9879 Acked Read 7 Done Done Sent false
33265 9880 Acked Read 6 Done Done Sent false
33266 9881 Acked Write 2 Done Done Sent false
33267 9882 Acked Write 3 Done Done Sent false
33268 9883 Acked Write 2 Done Done Sent false
33269 9884 Acked Write 8 Done Done Sent false
33270 9885 Acked Write 10 Done Done Sent false
33271 9886 Acked Read 2 Done Done Sent false
33272 9887 Acked Flush 0 Done Done Sent false
33273 9888 Acked Read 3 Done Done Sent false
33274 9889 Acked Write 8 Done Done Sent false
33275 9890 Acked Read 4 Done Done Sent false
33276 9891 Acked Read 3 Done Done Sent false
33277 9892 Acked Flush 0 Done Done Sent false
33278 9893 Acked Read 6 Done Done Sent false
33279 9894 Acked Read 1 Done Done Sent false
33280 9895 Acked Write 9 Done Done Sent false
33281 9896 Acked Read 2 Done Done Sent false
33282 9897 Acked Write 6 Done Done Sent false
33283 9898 Acked Write 5 Done Done Sent false
33284 9899 Acked Read 8 Done Done Sent false
33285 9900 Acked Write 2 Done Done Sent false
33286 9901 Acked Read 7 Done Done Sent false
33287 9902 Acked Write 3 Done Done Sent false
33288 9903 Acked Read 3 Done Done Sent false
33289 9904 Acked Read 1 Done Done Sent false
33290 9905 Acked Write 3 Done Done Sent false
33291 9906 Acked Write 6 Done Done Sent false
33292 9907 Acked Write 7 Done Done Sent false
33293 9908 Acked Read 1 Done Done Sent false
33294 9909 Acked Read 9 Done Done Sent false
33295 9910 Acked Write 1 Done Done Sent false
33296 9911 Acked Write 3 Done Done Sent false
33297 9912 Acked Read 9 Done Done Sent false
33298 9913 Acked Write 9 Done Done Sent false
33299 9914 Acked Read 9 Done Done Sent false
33300 9915 Acked Read 2 Done Done Sent false
33301 9916 Acked Write 5 Done Done Sent false
33302 9917 Acked Read 6 Done Done Sent false
33303 9918 Acked Write 8 Done Done Sent false
33304 9919 Acked Flush 0 Done Done Sent false
33305 9920 Acked Write 7 Done Done Sent false
33306 9921 Acked Write 10 Done Done Sent false
33307 9922 Acked Read 10 Done Done Sent false
33308 9923 Acked Read 4 Done Done Sent false
33309 9924 Acked Read 7 Done Done Sent false
33310 9925 Acked Write 3 Done Done Sent false
33311 9926 Acked Read 9 Done Done Sent false
33312 9927 Acked Write 10 Done Done Sent false
33313 9928 Acked Read 7 Done Done Sent false
33314 9929 Acked Read 6 Done Done Sent false
33315 9930 Acked Read 10 Done Done Sent false
33316 9931 Acked Read 3 Done Done Sent false
33317 9932 Acked Read 6 Done Done Sent false
33318 9933 Acked Write 2 Done Done Sent false
33319 9934 Acked Write 9 Done Done Sent false
33320 9935 Acked Write 2 Done Done Sent false
33321 9936 Acked Flush 0 Done Done Sent false
33322 9937 Acked Read 2 Done Done Sent false
33323 9938 Acked Read 5 Done Done Sent false
33324 9939 Acked Read 4 Done Done Sent false
33325 9940 Acked Read 3 Done Done Sent false
33326 9941 Acked Read 7 Done Done Sent false
33327 9942 Acked Read 1 Done Done Sent false
33328 9943 Acked Flush 0 Done Done Sent false
33329 9944 Acked Read 9 Done Done Sent false
33330 9945 Acked Read 8 Done Done Sent false
33331 9946 Acked Read 6 Done Done Sent false
33332 9947 Acked Read 3 Done Done Sent false
33333 9948 Acked Write 2 Done Done Sent false
33334 9949 Acked Write 6 Done Done Sent false
33335 9950 Acked Read 2 Done Done Sent false
33336 9951 Acked Read 6 Done Done Sent false
33337 9952 Acked Flush 0 Done Done Sent false
33338 9953 Acked Write 2 Done Done Sent false
33339 9954 Acked Read 5 Done Done Sent false
33340 9955 Acked Read 3 Done Done Sent false
33341 9956 Acked Read 9 Done Done Sent false
33342 9957 Acked Read 7 Done Done Sent false
33343 9958 Acked Write 2 Done Done Sent false
33344 9959 Acked Read 7 Done Done Sent false
33345 9960 Acked Write 9 Done Done Sent false
33346 9961 Acked Flush 0 Done Done Sent false
33347 9962 Acked Write 4 Done Done Sent false
33348 9963 Acked Write 8 Done Done Sent false
33349 9964 Acked Write 9 Done Done Sent false
33350 9965 Acked Read 10 Done Done Sent false
33351 9966 Acked Flush 0 Done Done Sent false
33352 9967 Acked Write 10 Done Done Sent false
33353 9968 Acked Read 5 Done Done Sent false
33354 9969 Acked Write 6 Done Done Sent false
33355 9970 Acked Write 8 Done Done Sent false
33356 9971 Acked Read 1 Done Done Sent false
33357 9972 Acked Read 5 Done Done Sent false
33358 9973 Acked Write 10 Done Done Sent false
33359 9974 Acked Write 7 Done Done Sent false
33360 9975 Acked Write 3 Done Done Sent false
33361 9976 Acked Write 10 Done Done Sent false
33362 9977 Acked Write 2 Done Done Sent false
33363 9978 Acked Write 6 Done Done Sent false
33364 9979 Acked Write 1 Done Done Sent false
33365 9980 Acked Read 6 Done Done Sent false
33366 9981 Acked Read 2 Done Done Sent false
33367 9982 Acked Write 8 Done Done Sent false
33368 9983 Acked Flush 0 Done Done Sent false
33369 9984 Acked Write 5 Done Done Sent false
33370 9985 Acked Read 3 Done Done Sent false
33371 9986 Acked Read 5 Done Done Sent false
33372 9987 Acked Read 2 Done Done Sent false
33373 9988 Acked Write 5 Done Done Sent false
33374 9989 Acked Write 3 Done Done Sent false
33375 9990 Acked Write 6 Done Done Sent false
33376 9991 Acked Read 8 Done Done Sent false
33377 9992 Acked Write 4 Done Done Sent false
33378 9993 Acked Write 1 Done Done Sent false
33379 9994 Acked Read 9 Done Done Sent false
33380 9995 Acked Write 6 Done Done Sent false
33381 9996 Acked Write 1 Done Done Sent false
33382 9997 Acked Write 10 Done Done Sent false
33383 9998 Acked Read 7 Done Done Sent false
33384 9999 Acked Read 8 Done Done Sent false
33385 10000 Acked Read 8 Done Done Sent false
33386 10001 Acked Write 9 Done Done Sent false
33387 10002 Acked Write 9 Done Done Sent false
33388 10003 Acked Write 2 Done Done Sent false
33389 10004 Acked Read 2 Done Done Sent false
33390 10005 Acked Read 1 Done Done Sent false
33391 10006 Acked Read 8 Done Done Sent false
33392 10007 Acked Write 2 Done Done Sent false
33393 10008 Acked Flush 0 Done Done Sent false
33394 10009 Acked Read 6 Done Done Sent false
33395 10010 Acked Read 3 Done Done Sent false
33396 10011 Acked Write 5 Done Done Sent false
33397 10012 Acked Write 10 Done Done Sent false
33398 10013 Acked Read 1 Done Done Sent false
33399 10014 Acked Read 4 Done Done Sent false
33400 10015 Acked Read 2 Done Done Sent false
33401 10016 Acked Write 7 Done Done Sent false
33402 10017 Acked Write 4 Done Done Sent false
33403 10018 Acked Write 6 Done Done Sent false
33404 10019 Acked Read 10 Done Done Sent false
33405 10020 Acked Write 4 Done Done Sent false
33406 10021 Acked Write 8 Done Done Sent false
33407 10022 Acked Flush 0 Done Done Sent false
33408 10023 Acked Read 8 Done Done Sent false
33409 10024 Acked Read 2 Done Done Sent false
33410 10025 Acked Write 5 Done Done Sent false
33411 10026 Acked Flush 0 Done Done Sent false
33412 10027 Acked Flush 0 Done Done Sent false
33413 10028 Acked Write 5 Done Done Sent false
33414 10029 Acked Read 10 Done Done Sent false
33415 10030 Acked Write 4 Done Done Sent false
33416 10031 Acked Write 7 Done Done Sent false
33417 10032 Acked Write 8 Done Done Sent false
33418 10033 Acked Read 6 Done Done Sent false
33419 10034 Acked Read 1 Done Done Sent false
33420 10035 Acked Write 3 Done Done Sent false
33421 10036 Acked Read 6 Done Done Sent false
33422 10037 Acked Write 10 Done Done Sent false
33423 10038 Acked Read 2 Done Done Sent false
33424 10039 Acked Write 5 Done Done Sent false
33425 10040 Acked Read 2 Done Done Sent false
33426 10041 Acked Read 10 Done Done Sent false
33427 10042 Acked Read 3 Done Done Sent false
33428 10043 Acked Write 1 Done Done Sent false
33429 10044 Acked Read 4 Done Done Sent false
33430 10045 Acked Write 10 Done Done Sent false
33431 10046 Acked Read 10 Done Done Sent false
33432 10047 Acked Read 8 Done Done Sent false
33433 10048 Acked Read 4 Done Done Sent false
33434 10049 Acked Write 1 Done Done Sent false
33435 10050 Acked Write 2 Done Done Sent false
33436 10051 Acked Read 5 Done Done Sent false
33437 10052 Acked Read 7 Done Done Sent false
33438 10053 Acked Write 3 Done Done Sent false
33439 10054 Acked Write 2 Done Done Sent false
33440 10055 Acked Write 5 Done Done Sent false
33441 10056 Acked Write 9 Done Done Sent false
33442 STATES DS:0 DS:1 DS:2 TOTAL
33443 Sent 0 0 301 301
33444 Done 301 301 0 602
33445 Skipped 0 0 0 0
33446 Error 0 0 0 0
33447 Last Flush: Some(JobId(10027)) Some(JobId(10027)) Some(JobId(9755))
33448 Downstairs last five completed: 9755 9754 9753 9752 9751
33449 Upstairs last five completed: 10056 10055 10054 10053 10052
33450 CLIENT: Up:0 ds:301 act:2
334512025-02-25T17:25:30.227ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
334522025-02-25T17:25:30.946ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
334532025-02-25T17:25:31.480ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
334542025-02-25T17:25:31.480ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47506) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
334552025-02-25T17:25:31.481ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
334562025-02-25T17:25:31.482ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
334572025-02-25T17:25:31.482ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
334582025-02-25T17:25:31.482ZINFOcrucible: send last flush ID to this DS: Some(JobId(9755)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
334592025-02-25T17:25:31.482ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
334602025-02-25T17:25:31.482ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9755)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
334612025-02-25T17:25:31.482ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9755)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
33462 ----------------------------------------------------------------
33463 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33464 Downstairs last five completed: 10057 10056 10055 10054 10053
33465 Upstairs last five completed: 10057 10056 10055 10054 10053
33466 CLIENT: Up:0 ds:0 act:3
33467 Replay: All jobs finished
33468 Main task has work to do, go find it
33469 got dsc Stop(2)
33470 stop 2
33471 Feb 25 17:25:33.729[8830] Got stop action so:false kr:false
33472 INFO request completed, latency_us: 406, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c9f4da22-dee2-4bc7-8a7c-3ce423aae8d7, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
334732025-02-25T17:25:33.729ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
334742025-02-25T17:25:33.729ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
334752025-02-25T17:25:33.730ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
33476 Feb 25 17:25:33.730 INFO request completed, latency_us: 139, response_code: 200, uri: /state/cid/2, method: GET, req_id: a28d396b-f402-48ef-be8b-65c1837a71b0, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
334772025-02-25T17:25:33.730ZINFOcrucible: client task is exiting = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
33478 [8830] Exited with: ExitStatus(unix_wait_status(9))
33479 [8830][2] reports Exit
33480 Feb 25 17:25:37.732 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9cae63f5-8881-47e2-94d8-3113581efd84, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
33481 001/300 Read block 3485 len 20480
33482 002/300 Read block 613 len 32768
33483 003/300 Write block 4795 len 20480 data: 4 4 4 7 8
33484 004/300 Read block 2160 len 24576
33485 005/300 Write block 1299 len 12288 data: 6 7 6
33486 006/300 Read block 3381 len 8192
33487 007/300 Write block 3132 len 36864 data: 5 5 4 5 6 6 6 7 8
33488 008/300 Flush
33489 009/300 Read block 3875 len 24576
33490 010/300 Flush
33491 011/300 Flush
33492 012/300 Write block 4694 len 4096 data: 4
33493 013/300 Read block 4268 len 28672
33494 014/300 Write block 4819 len 16384 data: 6 6 6 6
33495 015/300 Read block 3942 len 24576
33496 016/300 Read block 1466 len 8192
33497 017/300 Write block 2342 len 12288 data: 6 5 6
33498 018/300 Write block 1211 len 8192 data: 4 5
33499 019/300 Read block 1991 len 24576
33500 020/300 Write block 3130 len 20480 data: 4 4 6 6 5
33501 021/300 Read block 2330 len 4096
33502 022/300 Read block 1555 len 40960
33503 023/300 Write block 4154 len 40960 data: 6 5 6 7 9 9 7 5 4 3
33504 024/300 Write block 1063 len 12288 data: 8 7 5
33505 025/300 Write block 4669 len 32768 data: 7 8 7 7 8 7 6 4
33506 026/300 Read block 46 len 24576
33507 027/300 Read block 1313 len 36864
33508 028/300 Write block 1263 len 20480 data: 6 6 6 6 5
33509 029/300 Write block 3830 len 20480 data: 5 5 5 5 4
33510 030/300 Write block 1363 len 32768 data: 8 10 10 10 9 7 8 5
33511 031/300 Write block 2193 len 8192 data: 5 4
33512 032/300 Flush
33513 033/300 Write block 262 len 16384 data: 3 2 3 5
33514 034/300 Write block 1035 len 12288 data: 4 4 5
33515 035/300 Write block 529 len 36864 data: 5 4 6 8 8 6 6 8 7
33516 036/300 Read block 3779 len 28672
33517 037/300 Write block 4777 len 12288 data: 4 4 4
33518 038/300 Read block 1924 len 8192
33519 039/300 Read block 3568 len 20480
33520 040/300 Write block 1932 len 12288 data: 9 9 9
33521 041/300 Read block 2590 len 24576
33522 042/300 Write block 988 len 12288 data: 4 5 4
33523 043/300 Read block 2028 len 4096
33524 044/300 Write block 2065 len 28672 data: 7 6 4 5 6 6 6
33525 045/300 Read block 872 len 12288
33526 046/300 Read block 1893 len 24576
33527 047/300 Read block 2868 len 24576
33528 048/300 Read block 4296 len 8192
33529 049/300 Read block 3800 len 12288
33530 050/300 Read block 4712 len 16384
33531 051/300 Read block 2877 len 28672
33532 052/300 Read block 2558 len 8192
33533 053/300 Write block 1266 len 20480 data: 7 6 4 3 3
33534 054/300 Write block 4884 len 40960 data: 3 3 2 2 5 6 6 5 4 4
33535 055/300 Write block 22 len 36864 data: 6 8 9 8 10 11 10 8 6
33536 056/300 Read block 488 len 32768
33537 057/300 Read block 3952 len 12288
33538 058/300 Write block 4195 len 28672 data: 8 9 9 8 8 4 6
33539 059/300 Read block 4904 len 8192
33540 060/300 Read block 2328 len 16384
33541 061/300 Flush
33542 062/300 Read block 535 len 20480
33543 063/300 Read block 1903 len 32768
33544 064/300 Write block 2404 len 12288 data: 5 6 6
33545 065/300 Flush
33546 066/300 Read block 1426 len 16384
33547 067/300 Write block 1722 len 8192 data: 10 9
33548 068/300 Write block 3470 len 12288 data: 4 2 2
33549 069/300 Write block 2421 len 40960 data: 7 7 7 7 8 7 7 7 7 7
33550 070/300 Write block 2397 len 8192 data: 7 7
33551 071/300 Read block 3253 len 28672
33552 072/300 Write block 1969 len 20480 data: 4 5 4 4 3
33553 073/300 Flush
33554 074/300 Write block 3232 len 28672 data: 6 4 3 5 5 6 7
33555 075/300 Write block 1168 len 4096 data: 8
33556 076/300 Read block 3043 len 40960
33557 077/300 Write block 1992 len 36864 data: 7 6 8 8 7 6 8 9 8
33558 078/300 Write block 1550 len 8192 data: 4 5
33559 079/300 Write block 2144 len 12288 data: 9 7 5
33560 080/300 Read block 4185 len 20480
33561 081/300 Write block 807 len 16384 data: 3 4 6 6
33562 082/300 Write block 1094 len 20480 data: 8 7 7 8 9
33563 083/300 Write block 1359 len 40960 data: 4 6 8 8 9 11 11 11 10 8
33564 084/300 Write block 2144 len 20480 data: 10 8 6 5 5
33565 085/300 Flush
33566 086/300 Read block 871 len 8192
33567 087/300 Write block 2518 len 4096 data: 8
33568 088/300 Read block 4749 len 28672
33569 089/300 Read block 1136 len 12288
33570 090/300 Read block 3964 len 16384
33571 091/300 Write block 2879 len 36864 data: 7 7 9 7 6 6 6 6 6
33572 092/300 Write block 967 len 24576 data: 7 7 6 6 5 4
33573 093/300 Flush
33574 094/300 Write block 1929 len 24576 data: 6 8 9 10 10 10
33575 095/300 Read block 2341 len 20480
33576 096/300 Read block 4027 len 20480
33577 097/300 Write block 428 len 12288 data: 2 4 4
33578 098/300 Read block 2255 len 40960
33579 099/300 Read block 111 len 28672
33580 100/300 Write block 2005 len 8192 data: 7 8
33581 101/300 Read block 1280 len 32768
33582 102/300 Read block 709 len 40960
33583 103/300 Write block 2871 len 28672 data: 5 5 4 6 8 8 8
33584 104/300 Flush
33585 105/300 Read block 2135 len 16384
33586 106/300 Write block 93 len 20480 data: 6 7 7 7 8
33587 107/300 Read block 3702 len 12288
33588 108/300 Read block 983 len 36864
33589 109/300 Write block 707 len 32768 data: 4 5 6 6 7 7 8 8
33590 110/300 Read block 4341 len 16384
33591 111/300 Write block 2180 len 12288 data: 6 5 5
33592 112/300 Write block 2296 len 12288 data: 7 8 8
33593 113/300 Read block 3647 len 16384
33594 114/300 Read block 3482 len 20480
33595 115/300 Read block 881 len 12288
33596 116/300 Read block 760 len 24576
33597 117/300 Read block 4487 len 20480
33598 118/300 Read block 734 len 16384
33599 119/300 Read block 371 len 20480
33600 120/300 Write block 770 len 4096 data: 2
33601 121/300 Write block 2467 len 8192 data: 6 6
33602 122/300 Flush
33603 123/300 Write block 1417 len 20480 data: 4 4 5 4 5
33604 124/300 Write block 3024 len 4096 data: 5
33605 125/300 Read block 3136 len 16384
33606 126/300 Write block 1005 len 20480 data: 7 4 4 5 5
33607 127/300 Read block 4723 len 8192
33608 128/300 Write block 592 len 40960 data: 5 5 5 5 5 8 7 8 8 9
33609 129/300 Write block 84 len 16384 data: 4 6 6 6
33610 130/300 Write block 4729 len 4096 data: 5
33611 131/300 Read block 3015 len 40960
33612 132/300 Write block 3181 len 20480 data: 3 3 4 4 5
33613 133/300 Read block 4745 len 20480
33614 134/300 Read block 911 len 8192
33615 135/300 Write block 4735 len 12288 data: 5 5 4
33616 136/300 Write block 3797 len 32768 data: 9 9 7 6 6 7 6 5
33617 137/300 Write block 673 len 32768 data: 7 7 8 8 5 6 6 5
33618 138/300 Write block 221 len 16384 data: 4 4 3 4
33619 139/300 Write block 3198 len 12288 data: 8 8 6
33620 140/300 Read block 2944 len 40960
33621 141/300 Write block 2534 len 4096 data: 7
33622 142/300 Read block 2975 len 20480
33623 143/300 Read block 4712 len 32768
33624 144/300 Read block 3456 len 40960
33625 145/300 Read block 748 len 40960
33626 146/300 Write block 1018 len 28672 data: 5 7 8 7 8 8 8
33627 147/300 Write block 1743 len 20480 data: 4 5 6 7 7
33628 148/300 Read block 941 len 36864
33629 149/300 Flush
33630 150/300 Read block 1029 len 20480
33631 151/300 Write block 514 len 24576 data: 10 10 10 6 6 7
33632 152/300 Write block 4009 len 8192 data: 3 4
33633 153/300 Write block 2383 len 20480 data: 4 4 4 4 4
33634 154/300 Flush
33635 155/300 Read block 734 len 24576
33636 156/300 Read block 3642 len 16384
33637 157/300 Read block 3772 len 4096
33638 158/300 Read block 282 len 16384
33639 159/300 Write block 2630 len 12288 data: 7 6 7
33640 160/300 Write block 4270 len 8192 data: 7 6
33641 161/300 Write block 4803 len 24576 data: 7 7 6 8 8 9
33642 162/300 Write block 1591 len 32768 data: 4 6 6 4 4 3 3 4
33643 163/300 Read block 384 len 4096
33644 164/300 Write block 3404 len 8192 data: 4 4
33645 165/300 Write block 1515 len 24576 data: 5 5 6 5 7 6
33646 166/300 Write block 2416 len 24576 data: 2 3 4 5 6 8
33647 167/300 Read block 506 len 36864
33648 168/300 Flush
33649 169/300 Read block 4183 len 36864
33650 170/300 Read block 474 len 32768
33651 171/300 Read block 2482 len 28672
33652 172/300 Read block 686 len 24576
33653 173/300 Flush
33654 174/300 Read block 10 len 36864
33655 175/300 Read block 1165 len 20480
33656 176/300 Write block 1766 len 20480 data: 4 8 9 8 5
33657 177/300 Write block 1341 len 4096 data: 6
33658 178/300 Read block 1062 len 32768
33659 179/300 Write block 3922 len 36864 data: 7 7 6 5 3 2 3 4 5
33660 180/300 Write block 4513 len 28672 data: 4 5 8 6 7 7 9
33661 181/300 Read block 1009 len 40960
33662 182/300 Write block 1666 len 28672 data: 8 7 7 5 5 6 6
33663 183/300 Read block 1405 len 8192
33664 184/300 Flush
33665 185/300 Read block 4260 len 36864
33666 186/300 Read block 3468 len 20480
33667 187/300 Flush
33668 188/300 Read block 940 len 12288
33669 189/300 Write block 2431 len 32768 data: 8 6 7 4 4 4 4 5
33670 190/300 Flush
33671 191/300 Write block 4811 len 12288 data: 7 9 9
33672 192/300 Read block 2639 len 20480
33673 193/300 Write block 337 len 8192 data: 6 6
33674 194/300 Write block 2260 len 4096 data: 7
33675 195/300 Write block 4357 len 12288 data: 7 9 8
33676 196/300 Write block 2198 len 16384 data: 6 7 7 6
33677 197/300 Read block 233 len 12288
33678 198/300 Write block 3980 len 40960 data: 4 4 4 4 3 3 4 2 2 2
33679 199/300 Write block 4186 len 12288 data: 3 3 4
33680 200/300 Read block 4205 len 28672
33681 201/300 Read block 1918 len 32768
33682 202/300 Write block 3343 len 8192 data: 6 6
33683 203/300 Flush
33684 204/300 Read block 4895 len 16384
33685 205/300 Write block 557 len 40960 data: 8 6 6 7 8 6 3 4 6 5
33686 206/300 Write block 3724 len 12288 data: 11 11 10
33687 207/300 Read block 4212 len 28672
33688 208/300 Write block 2378 len 24576 data: 3 3 4 4 4 5
33689 209/300 Write block 3910 len 16384 data: 7 7 7 8
33690 210/300 Read block 1934 len 36864
33691 211/300 Read block 1588 len 16384
33692 212/300 Read block 3642 len 36864
33693 213/300 Read block 2595 len 24576
33694 214/300 Write block 226 len 16384 data: 4 4 4 5
33695 215/300 Read block 562 len 24576
33696 216/300 Write block 1923 len 36864 data: 3 4 4 4 3 4 7 9 10
33697 217/300 Write block 2854 len 40960 data: 4 5 5 7 8 10 8 7 7 7
33698 218/300 Write block 4864 len 8192 data: 5 6
33699 219/300 Read block 1541 len 24576
33700 220/300 Write block 2988 len 16384 data: 6 4 6 6
33701 221/300 Write block 1761 len 16384 data: 5 6 7 7
33702 222/300 Read block 2779 len 4096
33703 223/300 Flush
33704 224/300 Write block 1011 len 24576 data: 5 6 6 5 6 5
33705 225/300 Read block 4213 len 12288
33706 226/300 Flush
33707 227/300 Write block 2522 len 36864 data: 4 4 4 4 3 4 6 5 5
33708 228/300 Read block 1414 len 20480
33709 229/300 Read block 3074 len 36864
33710 230/300 Flush
33711 231/300 Write block 89 len 12288 data: 5 5 4
33712 232/300 Read block 3758 len 8192
33713 233/300 Read block 4558 len 36864
33714 234/300 Write block 3643 len 32768 data: 5 6 5 7 7 6 7 8
33715 235/300 Flush
33716 236/300 Read block 3119 len 16384
33717 237/300 Read block 4601 len 4096
33718 238/300 Read block 2880 len 36864
33719 239/300 Write block 3784 len 16384 data: 4 4 4 4
33720 240/300 Read block 4490 len 24576
33721 241/300 Read block 65 len 8192
33722 242/300 Read block 1840 len 36864
33723 243/300 Read block 3264 len 36864
33724 244/300 Read block 1846 len 12288
33725 245/300 Read block 3253 len 24576
33726 246/300 Write block 4954 len 32768 data: 7 5 6 5 6 6 7 9
33727 247/300 Read block 1388 len 32768
33728 248/300 Flush
33729 249/300 Flush
33730 250/300 Flush
33731 251/300 Write block 2033 len 4096 data: 9
33732 252/300 Write block 3881 len 20480 data: 5 4 5 4 4
33733 253/300 Write block 4765 len 16384 data: 9 12 11 10
33734 254/300 Read block 925 len 16384
33735 255/300 Read block 1479 len 4096
33736 256/300 Read block 2537 len 4096
33737 257/300 Write block 1162 len 12288 data: 4 5 5
33738 258/300 Read block 3453 len 40960
33739 259/300 Write block 1301 len 32768 data: 7 4 4 4 3 3 5 5
33740 260/300 Read block 1509 len 36864
33741 261/300 Write block 3493 len 8192 data: 5 6
33742 262/300 Read block 2051 len 40960
33743 263/300 Read block 1519 len 8192
33744 264/300 Write block 3770 len 4096 data: 5
33745 265/300 Flush
33746 266/300 Read block 2959 len 20480
33747 267/300 Write block 4883 len 36864 data: 3 4 4 3 3 6 7 7 6
33748 268/300 Flush
33749 269/300 Read block 1573 len 16384
33750 270/300 Read block 2322 len 40960
33751 271/300 Read block 3995 len 40960
33752 272/300 Read block 2686 len 40960
33753 273/300 Read block 2428 len 12288
33754 274/300 Read block 1371 len 16384
33755 275/300 Flush
33756 276/300 Write block 4382 len 8192 data: 5 4
33757 277/300 Read block 2446 len 40960
33758 278/300 Write block 594 len 4096 data: 6
33759 279/300 Write block 3924 len 28672 data: 7 6 4 3 4 5 6
33760 280/300 Write block 1493 len 20480 data: 6 5 5 7 7
33761 281/300 Flush
33762 282/300 Flush
33763 283/300 Read block 767 len 4096
33764 284/300 Flush
33765 285/300 Write block 2329 len 24576 data: 8 9 8 9 9 9
33766 286/300 Write block 842 len 4096 data: 5
33767 287/300 Write block 4627 len 28672 data: 7 9 7 8 7 7 7
33768 288/300 Read block 67 len 8192
33769 289/300 Read block 1733 len 36864
33770 290/300 Write block 3161 len 40960 data: 4 7 9 8 7 8 8 8 10 7
33771 291/300 Write block 854 len 8192 data: 6 8
33772 292/300 Read block 1971 len 20480
33773 293/300 Read block 4063 len 20480
33774 294/300 Read block 1231 len 36864
33775 295/300 Read block 444 len 40960
33776 296/300 Flush
33777 297/300 Write block 700 len 28672 data: 6 5 4 3 4 4 4
33778 298/300 Write block 564 len 8192 data: 5 7
33779 299/300 Write block 2595 len 40960 data: 6 6 5 7 8 5 9 7 8 6
33780 300/300 Read block 2552 len 16384
33781 301/300 Write block 4309 len 40960 data: 7 7 5 5 4 4 3 4 3 5
33782 Main task has work to do, go find it
33783 got dsc Start(2)
33784 start 2
33785 Feb 25 17:25:37.974 INFO request completed, latency_us: 243, response_code: 204, uri: /start/cid/2, method: POST, req_id: 0c5ff0a8-0a24-46c1-a3e4-3ed57353eb13, remote_addr: 127.0.0.1:52576, local_addr: 127.0.0.1:9998
33786 Replay: started 2, returned:Ok(())
33787 ----------------------------------------------------------------
33788 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33789 States: Active Active Offline
33790 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33791 10058 Acked Read 5 Done Done Sent false
33792 10059 Acked Read 8 Done Done Sent false
33793 10060 Acked Write 5 Done Done Sent false
33794 10061 Acked Read 6 Done Done Sent false
33795 10062 Acked Write 3 Done Done Sent false
33796 10063 Acked Read 2 Done Done Sent false
33797 10064 Acked Write 9 Done Done Sent false
33798 10065 Acked Flush 0 Done Done Sent false
33799 10066 Acked Read 6 Done Done Sent false
33800 10067 Acked Flush 0 Done Done Sent false
33801 10068 Acked Flush 0 Done Done Sent false
33802 10069 Acked Write 1 Done Done Sent false
33803 10070 Acked Read 7 Done Done Sent false
33804 10071 Acked Write 4 Done Done Sent false
33805 10072 Acked Read 6 Done Done Sent false
33806 10073 Acked Read 2 Done Done Sent false
33807 10074 Acked Write 3 Done Done Sent false
33808 10075 Acked Write 2 Done Done Sent false
33809 10076 Acked Read 6 Done Done Sent false
33810 10077 Acked Write 5 Done Done Sent false
33811 10078 Acked Read 1 Done Done Sent false
33812 10079 Acked Read 10 Done Done Sent false
33813 10080 Acked Write 10 Done Done Sent false
33814 10081 Acked Write 3 Done Done Sent false
33815 10082 Acked Write 8 Done Done Sent false
33816 10083 Acked Read 6 Done Done Sent false
33817 10084 Acked Read 9 Done Done Sent false
33818 10085 Acked Write 5 Done Done Sent false
33819 10086 Acked Write 5 Done Done Sent false
33820 10087 Acked Write 8 Done Done Sent false
33821 10088 Acked Write 2 Done Done Sent false
33822 10089 Acked Flush 0 Done Done Sent false
33823 10090 Acked Write 4 Done Done Sent false
33824 10091 Acked Write 3 Done Done Sent false
33825 10092 Acked Write 9 Done Done Sent false
33826 10093 Acked Read 7 Done Done Sent false
33827 10094 Acked Write 3 Done Done Sent false
33828 10095 Acked Read 2 Done Done Sent false
33829 10096 Acked Read 5 Done Done Sent false
33830 10097 Acked Write 3 Done Done Sent false
33831 10098 Acked Read 6 Done Done Sent false
33832 10099 Acked Write 3 Done Done Sent false
33833 10100 Acked Read 1 Done Done Sent false
33834 10101 Acked Write 7 Done Done Sent false
33835 10102 Acked Read 3 Done Done Sent false
33836 10103 Acked Read 6 Done Done Sent false
33837 10104 Acked Read 6 Done Done Sent false
33838 10105 Acked Read 2 Done Done Sent false
33839 10106 Acked Read 3 Done Done Sent false
33840 10107 Acked Read 4 Done Done Sent false
33841 10108 Acked Read 7 Done Done Sent false
33842 10109 Acked Read 2 Done Done Sent false
33843 10110 Acked Write 5 Done Done Sent false
33844 10111 Acked Write 10 Done Done Sent false
33845 10112 Acked Write 9 Done Done Sent false
33846 10113 Acked Read 8 Done Done Sent false
33847 10114 Acked Read 3 Done Done Sent false
33848 10115 Acked Write 7 Done Done Sent false
33849 10116 Acked Read 2 Done Done Sent false
33850 10117 Acked Read 4 Done Done Sent false
33851 10118 Acked Flush 0 Done Done Sent false
33852 10119 Acked Read 5 Done Done Sent false
33853 10120 Acked Read 8 Done Done Sent false
33854 10121 Acked Write 3 Done Done Sent false
33855 10122 Acked Flush 0 Done Done Sent false
33856 10123 Acked Read 4 Done Done Sent false
33857 10124 Acked Write 2 Done Done Sent false
33858 10125 Acked Write 3 Done Done Sent false
33859 10126 Acked Write 10 Done Done Sent false
33860 10127 Acked Write 2 Done Done Sent false
33861 10128 Acked Read 7 Done Done Sent false
33862 10129 Acked Write 5 Done Done Sent false
33863 10130 Acked Flush 0 Done Done Sent false
33864 10131 Acked Write 7 Done Done Sent false
33865 10132 Acked Write 1 Done Done Sent false
33866 10133 Acked Read 10 Done Done Sent false
33867 10134 Acked Write 9 Done Done Sent false
33868 10135 Acked Write 2 Done Done Sent false
33869 10136 Acked Write 3 Done Done Sent false
33870 10137 Acked Read 5 Done Done Sent false
33871 10138 Acked Write 4 Done Done Sent false
33872 10139 Acked Write 5 Done Done Sent false
33873 10140 Acked Write 10 Done Done Sent false
33874 10141 Acked Write 5 Done Done Sent false
33875 10142 Acked Flush 0 Done Done Sent false
33876 10143 Acked Read 2 Done Done Sent false
33877 10144 Acked Write 1 Done Done Sent false
33878 10145 Acked Read 7 Done Done Sent false
33879 10146 Acked Read 3 Done Done Sent false
33880 10147 Acked Read 4 Done Done Sent false
33881 10148 Acked Write 9 Done Done Sent false
33882 10149 Acked Write 6 Done Done Sent false
33883 10150 Acked Flush 0 Done Done Sent false
33884 10151 Acked Write 6 Done Done Sent false
33885 10152 Acked Read 5 Done Done Sent false
33886 10153 Acked Read 5 Done Done Sent false
33887 10154 Acked Write 3 Done Done Sent false
33888 10155 Acked Read 10 Done Done Sent false
33889 10156 Acked Read 7 Done Done Sent false
33890 10157 Acked Write 2 Done Done Sent false
33891 10158 Acked Read 8 Done Done Sent false
33892 10159 Acked Read 10 Done Done Sent false
33893 10160 Acked Write 7 Done Done Sent false
33894 10161 Acked Flush 0 Done Done Sent false
33895 10162 Acked Read 4 Done Done Sent false
33896 10163 Acked Write 5 Done Done Sent false
33897 10164 Acked Read 3 Done Done Sent false
33898 10165 Acked Read 9 Done Done Sent false
33899 10166 Acked Write 8 Done Done Sent false
33900 10167 Acked Read 4 Done Done Sent false
33901 10168 Acked Write 3 Done Done Sent false
33902 10169 Acked Write 3 Done Done Sent false
33903 10170 Acked Read 4 Done Done Sent false
33904 10171 Acked Read 5 Done Done Sent false
33905 10172 Acked Read 3 Done Done Sent false
33906 10173 Acked Read 6 Done Done Sent false
33907 10174 Acked Read 5 Done Done Sent false
33908 10175 Acked Read 4 Done Done Sent false
33909 10176 Acked Read 5 Done Done Sent false
33910 10177 Acked Write 1 Done Done Sent false
33911 10178 Acked Write 2 Done Done Sent false
33912 10179 Acked Flush 0 Done Done Sent false
33913 10180 Acked Write 5 Done Done Sent false
33914 10181 Acked Write 1 Done Done Sent false
33915 10182 Acked Read 4 Done Done Sent false
33916 10183 Acked Write 5 Done Done Sent false
33917 10184 Acked Read 2 Done Done Sent false
33918 10185 Acked Write 10 Done Done Sent false
33919 10186 Acked Write 4 Done Done Sent false
33920 10187 Acked Write 1 Done Done Sent false
33921 10188 Acked Read 10 Done Done Sent false
33922 10189 Acked Write 5 Done Done Sent false
33923 10190 Acked Read 5 Done Done Sent false
33924 10191 Acked Read 2 Done Done Sent false
33925 10192 Acked Write 3 Done Done Sent false
33926 10193 Acked Write 8 Done Done Sent false
33927 10194 Acked Write 8 Done Done Sent false
33928 10195 Acked Write 4 Done Done Sent false
33929 10196 Acked Write 3 Done Done Sent false
33930 10197 Acked Read 10 Done Done Sent false
33931 10198 Acked Write 1 Done Done Sent false
33932 10199 Acked Read 5 Done Done Sent false
33933 10200 Acked Read 8 Done Done Sent false
33934 10201 Acked Read 10 Done Done Sent false
33935 10202 Acked Read 10 Done Done Sent false
33936 10203 Acked Write 7 Done Done Sent false
33937 10204 Acked Write 5 Done Done Sent false
33938 10205 Acked Read 9 Done Done Sent false
33939 10206 Acked Flush 0 Done Done Sent false
33940 10207 Acked Read 5 Done Done Sent false
33941 10208 Acked Write 6 Done Done Sent false
33942 10209 Acked Write 2 Done Done Sent false
33943 10210 Acked Write 5 Done Done Sent false
33944 10211 Acked Flush 0 Done Done Sent false
33945 10212 Acked Read 6 Done Done Sent false
33946 10213 Acked Read 4 Done Done Sent false
33947 10214 Acked Read 1 Done Done Sent false
33948 10215 Acked Read 4 Done Done Sent false
33949 10216 Acked Write 3 Done Done Sent false
33950 10217 Acked Write 2 Done Done Sent false
33951 10218 Acked Write 6 Done Done Sent false
33952 10219 Acked Write 8 Done Done Sent false
33953 10220 Acked Read 1 Done Done Sent false
33954 10221 Acked Write 2 Done Done Sent false
33955 10222 Acked Write 6 Done Done Sent false
33956 10223 Acked Write 6 Done Done Sent false
33957 10224 Acked Read 9 Done Done Sent false
33958 10225 Acked Flush 0 Done Done Sent false
33959 10226 Acked Read 9 Done Done Sent false
33960 10227 Acked Read 8 Done Done Sent false
33961 10228 Acked Read 7 Done Done Sent false
33962 10229 Acked Read 6 Done Done Sent false
33963 10230 Acked Flush 0 Done Done Sent false
33964 10231 Acked Read 9 Done Done Sent false
33965 10232 Acked Read 5 Done Done Sent false
33966 10233 Acked Write 5 Done Done Sent false
33967 10234 Acked Write 1 Done Done Sent false
33968 10235 Acked Read 8 Done Done Sent false
33969 10236 Acked Write 9 Done Done Sent false
33970 10237 Acked Write 7 Done Done Sent false
33971 10238 Acked Read 10 Done Done Sent false
33972 10239 Acked Write 7 Done Done Sent false
33973 10240 Acked Read 2 Done Done Sent false
33974 10241 Acked Flush 0 Done Done Sent false
33975 10242 Acked Read 9 Done Done Sent false
33976 10243 Acked Read 5 Done Done Sent false
33977 10244 Acked Flush 0 Done Done Sent false
33978 10245 Acked Read 3 Done Done Sent false
33979 10246 Acked Write 8 Done Done Sent false
33980 10247 Acked Flush 0 Done Done Sent false
33981 10248 Acked Write 3 Done Done Sent false
33982 10249 Acked Read 5 Done Done Sent false
33983 10250 Acked Write 2 Done Done Sent false
33984 10251 Acked Write 1 Done Done Sent false
33985 10252 Acked Write 3 Done Done Sent false
33986 10253 Acked Write 4 Done Done Sent false
33987 10254 Acked Read 3 Done Done Sent false
33988 10255 Acked Write 10 Done Done Sent false
33989 10256 Acked Write 3 Done Done Sent false
33990 10257 Acked Read 7 Done Done Sent false
33991 10258 Acked Read 8 Done Done Sent false
33992 10259 Acked Write 2 Done Done Sent false
33993 10260 Acked Flush 0 Done Done Sent false
33994 10261 Acked Read 4 Done Done Sent false
33995 10262 Acked Write 10 Done Done Sent false
33996 10263 Acked Write 3 Done Done Sent false
33997 10264 Acked Read 7 Done Done Sent false
33998 10265 Acked Write 6 Done Done Sent false
33999 10266 Acked Write 4 Done Done Sent false
34000 10267 Acked Read 9 Done Done Sent false
34001 10268 Acked Read 4 Done Done Sent false
34002 10269 Acked Read 9 Done Done Sent false
34003 10270 Acked Read 6 Done Done Sent false
34004 10271 Acked Write 4 Done Done Sent false
34005 10272 Acked Read 6 Done Done Sent false
34006 10273 Acked Write 9 Done Done Sent false
34007 10274 Acked Write 10 Done Done Sent false
34008 10275 Acked Write 2 Done Done Sent false
34009 10276 Acked Read 6 Done Done Sent false
34010 10277 Acked Write 4 Done Done Sent false
34011 10278 Acked Write 4 Done Done Sent false
34012 10279 Acked Read 1 Done Done Sent false
34013 10280 Acked Flush 0 Done Done Sent false
34014 10281 Acked Write 6 Done Done Sent false
34015 10282 Acked Read 3 Done Done Sent false
34016 10283 Acked Flush 0 Done Done Sent false
34017 10284 Acked Write 9 Done Done Sent false
34018 10285 Acked Read 5 Done Done Sent false
34019 10286 Acked Read 9 Done Done Sent false
34020 10287 Acked Flush 0 Done Done Sent false
34021 10288 Acked Write 3 Done Done Sent false
34022 10289 Acked Read 2 Done Done Sent false
34023 10290 Acked Read 9 Done Done Sent false
34024 10291 Acked Write 8 Done Done Sent false
34025 10292 Acked Flush 0 Done Done Sent false
34026 10293 Acked Read 4 Done Done Sent false
34027 10294 Acked Read 1 Done Done Sent false
34028 10295 Acked Read 9 Done Done Sent false
34029 10296 Acked Write 4 Done Done Sent false
34030 10297 Acked Read 6 Done Done Sent false
34031 10298 Acked Read 2 Done Done Sent false
34032 10299 Acked Read 9 Done Done Sent false
34033 10300 Acked Read 9 Done Done Sent false
34034 10301 Acked Read 3 Done Done Sent false
34035 10302 Acked Read 6 Done Done Sent false
34036 10303 Acked Write 8 Done Done Sent false
34037 10304 Acked Read 8 Done Done Sent false
34038 10305 Acked Flush 0 Done Done Sent false
34039 10306 Acked Flush 0 Done Done Sent false
34040 10307 Acked Flush 0 Done Done Sent false
34041 10308 Acked Write 1 Done Done Sent false
34042 10309 Acked Write 5 Done Done Sent false
34043 10310 Acked Write 4 Done Done Sent false
34044 10311 Acked Read 4 Done Done Sent false
34045 10312 Acked Read 1 Done Done Sent false
34046 10313 Acked Read 1 Done Done Sent false
34047 10314 Acked Write 3 Done Done Sent false
34048 10315 Acked Read 10 Done Done Sent false
34049 10316 Acked Write 8 Done Done Sent false
34050 10317 Acked Read 9 Done Done Sent false
34051 10318 Acked Write 2 Done Done Sent false
34052 10319 Acked Read 10 Done Done Sent false
34053 10320 Acked Read 2 Done Done Sent false
34054 10321 Acked Write 1 Done Done Sent false
34055 10322 Acked Flush 0 Done Done Sent false
34056 10323 Acked Read 5 Done Done Sent false
34057 10324 Acked Write 9 Done Done Sent false
34058 10325 Acked Flush 0 Done Done Sent false
34059 10326 Acked Read 4 Done Done Sent false
34060 10327 Acked Read 10 Done Done Sent false
34061 10328 Acked Read 10 Done Done Sent false
34062 10329 Acked Read 10 Done Done Sent false
34063 10330 Acked Read 3 Done Done Sent false
34064 10331 Acked Read 4 Done Done Sent false
34065 10332 Acked Flush 0 Done Done Sent false
34066 10333 Acked Write 2 Done Done Sent false
34067 10334 Acked Read 10 Done Done Sent false
34068 10335 Acked Write 1 Done Done Sent false
34069 10336 Acked Write 7 Done Done Sent false
34070 10337 Acked Write 5 Done Done Sent false
34071 10338 Acked Flush 0 Done Done Sent false
34072 10339 Acked Flush 0 Done Done Sent false
34073 10340 Acked Read 1 Done Done Sent false
34074 10341 Acked Flush 0 Done Done Sent false
34075 10342 Acked Write 6 Done Done Sent false
34076 10343 Acked Write 1 Done Done Sent false
34077 10344 Acked Write 7 Done Done Sent false
34078 10345 Acked Read 2 Done Done Sent false
34079 10346 Acked Read 9 Done Done Sent false
34080 10347 Acked Write 10 Done Done Sent false
34081 10348 Acked Write 2 Done Done Sent false
34082 10349 Acked Read 5 Done Done Sent false
34083 10350 Acked Read 5 Done Done Sent false
34084 10351 Acked Read 9 Done Done Sent false
34085 10352 Acked Read 10 Done Done Sent false
34086 10353 Acked Flush 0 Done Done Sent false
34087 10354 Acked Write 7 Done Done Sent false
34088 10355 Acked Write 2 Done Done Sent false
34089 10356 Acked Write 10 Done Done Sent false
34090 10357 Acked Read 4 Done Done Sent false
34091 10358 Acked Write 10 Done Done Sent false
34092 STATES DS:0 DS:1 DS:2 TOTAL
34093 Sent 0 0 301 301
34094 Done 301 301 0 602
34095 Skipped 0 0 0 0
34096 Error 0 0 0 0
34097 Last Flush: Some(JobId(10353)) Some(JobId(10353)) Some(JobId(10057))
34098 Downstairs last five completed: 10057 10056 10055 10054 10053
34099 Upstairs last five completed: 10358 10357 10356 10355 10354
34100 CLIENT: Up:0 ds:301 act:2
341012025-02-25T17:25:38.961ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
341022025-02-25T17:25:40.191ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
341032025-02-25T17:25:40.964ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
34104 [8830] got start action
34105 [8830] I am going to restart
34106 Starting downstairs at port 8830
34107 Make output file at "/tmp/dsc/downstairs-8830.txt"
34108 [8830][2] reports Starting
34109 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1239)
34110 [8830][2] reports Running
34111 ----------------------------------------------------------------
34112 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34113 States: Active Active Offline
34114 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34115 10058 Acked Read 5 Done Done Sent false
34116 10059 Acked Read 8 Done Done Sent false
34117 10060 Acked Write 5 Done Done Sent false
34118 10061 Acked Read 6 Done Done Sent false
34119 10062 Acked Write 3 Done Done Sent false
34120 10063 Acked Read 2 Done Done Sent false
34121 10064 Acked Write 9 Done Done Sent false
34122 10065 Acked Flush 0 Done Done Sent false
34123 10066 Acked Read 6 Done Done Sent false
34124 10067 Acked Flush 0 Done Done Sent false
34125 10068 Acked Flush 0 Done Done Sent false
34126 10069 Acked Write 1 Done Done Sent false
34127 10070 Acked Read 7 Done Done Sent false
34128 10071 Acked Write 4 Done Done Sent false
34129 10072 Acked Read 6 Done Done Sent false
34130 10073 Acked Read 2 Done Done Sent false
34131 10074 Acked Write 3 Done Done Sent false
34132 10075 Acked Write 2 Done Done Sent false
34133 10076 Acked Read 6 Done Done Sent false
34134 10077 Acked Write 5 Done Done Sent false
34135 10078 Acked Read 1 Done Done Sent false
34136 10079 Acked Read 10 Done Done Sent false
34137 10080 Acked Write 10 Done Done Sent false
34138 10081 Acked Write 3 Done Done Sent false
34139 10082 Acked Write 8 Done Done Sent false
34140 10083 Acked Read 6 Done Done Sent false
34141 10084 Acked Read 9 Done Done Sent false
34142 10085 Acked Write 5 Done Done Sent false
34143 10086 Acked Write 5 Done Done Sent false
34144 10087 Acked Write 8 Done Done Sent false
34145 10088 Acked Write 2 Done Done Sent false
34146 10089 Acked Flush 0 Done Done Sent false
34147 10090 Acked Write 4 Done Done Sent false
34148 10091 Acked Write 3 Done Done Sent false
34149 10092 Acked Write 9 Done Done Sent false
34150 10093 Acked Read 7 Done Done Sent false
34151 10094 Acked Write 3 Done Done Sent false
34152 10095 Acked Read 2 Done Done Sent false
34153 10096 Acked Read 5 Done Done Sent false
34154 10097 Acked Write 3 Done Done Sent false
34155 10098 Acked Read 6 Done Done Sent false
34156 10099 Acked Write 3 Done Done Sent false
34157 10100 Acked Read 1 Done Done Sent false
34158 10101 Acked Write 7 Done Done Sent false
34159 10102 Acked Read 3 Done Done Sent false
34160 10103 Acked Read 6 Done Done Sent false
34161 10104 Acked Read 6 Done Done Sent false
34162 10105 Acked Read 2 Done Done Sent false
34163 10106 Acked Read 3 Done Done Sent false
34164 10107 Acked Read 4 Done Done Sent false
34165 10108 Acked Read 7 Done Done Sent false
34166 10109 Acked Read 2 Done Done Sent false
34167 10110 Acked Write 5 Done Done Sent false
34168 10111 Acked Write 10 Done Done Sent false
34169 10112 Acked Write 9 Done Done Sent false
34170 10113 Acked Read 8 Done Done Sent false
34171 10114 Acked Read 3 Done Done Sent false
34172 10115 Acked Write 7 Done Done Sent false
34173 10116 Acked Read 2 Done Done Sent false
34174 10117 Acked Read 4 Done Done Sent false
34175 10118 Acked Flush 0 Done Done Sent false
34176 10119 Acked Read 5 Done Done Sent false
34177 10120 Acked Read 8 Done Done Sent false
34178 10121 Acked Write 3 Done Done Sent false
34179 10122 Acked Flush 0 Done Done Sent false
34180 10123 Acked Read 4 Done Done Sent false
34181 10124 Acked Write 2 Done Done Sent false
34182 10125 Acked Write 3 Done Done Sent false
34183 10126 Acked Write 10 Done Done Sent false
34184 10127 Acked Write 2 Done Done Sent false
34185 10128 Acked Read 7 Done Done Sent false
34186 10129 Acked Write 5 Done Done Sent false
34187 10130 Acked Flush 0 Done Done Sent false
34188 10131 Acked Write 7 Done Done Sent false
34189 10132 Acked Write 1 Done Done Sent false
34190 10133 Acked Read 10 Done Done Sent false
34191 10134 Acked Write 9 Done Done Sent false
34192 10135 Acked Write 2 Done Done Sent false
34193 10136 Acked Write 3 Done Done Sent false
34194 10137 Acked Read 5 Done Done Sent false
34195 10138 Acked Write 4 Done Done Sent false
34196 10139 Acked Write 5 Done Done Sent false
34197 10140 Acked Write 10 Done Done Sent false
34198 10141 Acked Write 5 Done Done Sent false
34199 10142 Acked Flush 0 Done Done Sent false
34200 10143 Acked Read 2 Done Done Sent false
34201 10144 Acked Write 1 Done Done Sent false
34202 10145 Acked Read 7 Done Done Sent false
34203 10146 Acked Read 3 Done Done Sent false
34204 10147 Acked Read 4 Done Done Sent false
34205 10148 Acked Write 9 Done Done Sent false
34206 10149 Acked Write 6 Done Done Sent false
34207 10150 Acked Flush 0 Done Done Sent false
34208 10151 Acked Write 6 Done Done Sent false
34209 10152 Acked Read 5 Done Done Sent false
34210 10153 Acked Read 5 Done Done Sent false
34211 10154 Acked Write 3 Done Done Sent false
34212 10155 Acked Read 10 Done Done Sent false
34213 10156 Acked Read 7 Done Done Sent false
34214 10157 Acked Write 2 Done Done Sent false
34215 10158 Acked Read 8 Done Done Sent false
34216 10159 Acked Read 10 Done Done Sent false
34217 10160 Acked Write 7 Done Done Sent false
34218 10161 Acked Flush 0 Done Done Sent false
34219 10162 Acked Read 4 Done Done Sent false
34220 10163 Acked Write 5 Done Done Sent false
34221 10164 Acked Read 3 Done Done Sent false
34222 10165 Acked Read 9 Done Done Sent false
34223 10166 Acked Write 8 Done Done Sent false
34224 10167 Acked Read 4 Done Done Sent false
34225 10168 Acked Write 3 Done Done Sent false
34226 10169 Acked Write 3 Done Done Sent false
34227 10170 Acked Read 4 Done Done Sent false
34228 10171 Acked Read 5 Done Done Sent false
34229 10172 Acked Read 3 Done Done Sent false
34230 10173 Acked Read 6 Done Done Sent false
34231 10174 Acked Read 5 Done Done Sent false
34232 10175 Acked Read 4 Done Done Sent false
34233 10176 Acked Read 5 Done Done Sent false
34234 10177 Acked Write 1 Done Done Sent false
34235 10178 Acked Write 2 Done Done Sent false
34236 10179 Acked Flush 0 Done Done Sent false
34237 10180 Acked Write 5 Done Done Sent false
34238 10181 Acked Write 1 Done Done Sent false
34239 10182 Acked Read 4 Done Done Sent false
34240 10183 Acked Write 5 Done Done Sent false
34241 10184 Acked Read 2 Done Done Sent false
34242 10185 Acked Write 10 Done Done Sent false
34243 10186 Acked Write 4 Done Done Sent false
34244 10187 Acked Write 1 Done Done Sent false
34245 10188 Acked Read 10 Done Done Sent false
34246 10189 Acked Write 5 Done Done Sent false
34247 10190 Acked Read 5 Done Done Sent false
34248 10191 Acked Read 2 Done Done Sent false
34249 10192 Acked Write 3 Done Done Sent false
34250 10193 Acked Write 8 Done Done Sent false
34251 10194 Acked Write 8 Done Done Sent false
34252 10195 Acked Write 4 Done Done Sent false
34253 10196 Acked Write 3 Done Done Sent false
34254 10197 Acked Read 10 Done Done Sent false
34255 10198 Acked Write 1 Done Done Sent false
34256 10199 Acked Read 5 Done Done Sent false
34257 10200 Acked Read 8 Done Done Sent false
34258 10201 Acked Read 10 Done Done Sent false
34259 10202 Acked Read 10 Done Done Sent false
34260 10203 Acked Write 7 Done Done Sent false
34261 10204 Acked Write 5 Done Done Sent false
34262 10205 Acked Read 9 Done Done Sent false
34263 10206 Acked Flush 0 Done Done Sent false
34264 10207 Acked Read 5 Done Done Sent false
34265 10208 Acked Write 6 Done Done Sent false
34266 10209 Acked Write 2 Done Done Sent false
34267 10210 Acked Write 5 Done Done Sent false
34268 10211 Acked Flush 0 Done Done Sent false
34269 10212 Acked Read 6 Done Done Sent false
34270 10213 Acked Read 4 Done Done Sent false
34271 10214 Acked Read 1 Done Done Sent false
34272 10215 Acked Read 4 Done Done Sent false
34273 10216 Acked Write 3 Done Done Sent false
34274 10217 Acked Write 2 Done Done Sent false
34275 10218 Acked Write 6 Done Done Sent false
34276 10219 Acked Write 8 Done Done Sent false
34277 10220 Acked Read 1 Done Done Sent false
34278 10221 Acked Write 2 Done Done Sent false
34279 10222 Acked Write 6 Done Done Sent false
34280 10223 Acked Write 6 Done Done Sent false
34281 10224 Acked Read 9 Done Done Sent false
34282 10225 Acked Flush 0 Done Done Sent false
34283 10226 Acked Read 9 Done Done Sent false
34284 10227 Acked Read 8 Done Done Sent false
34285 10228 Acked Read 7 Done Done Sent false
34286 10229 Acked Read 6 Done Done Sent false
34287 10230 Acked Flush 0 Done Done Sent false
34288 10231 Acked Read 9 Done Done Sent false
34289 10232 Acked Read 5 Done Done Sent false
34290 10233 Acked Write 5 Done Done Sent false
34291 10234 Acked Write 1 Done Done Sent false
34292 10235 Acked Read 8 Done Done Sent false
34293 10236 Acked Write 9 Done Done Sent false
34294 10237 Acked Write 7 Done Done Sent false
34295 10238 Acked Read 10 Done Done Sent false
34296 10239 Acked Write 7 Done Done Sent false
34297 10240 Acked Read 2 Done Done Sent false
34298 10241 Acked Flush 0 Done Done Sent false
34299 10242 Acked Read 9 Done Done Sent false
34300 10243 Acked Read 5 Done Done Sent false
34301 10244 Acked Flush 0 Done Done Sent false
34302 10245 Acked Read 3 Done Done Sent false
34303 10246 Acked Write 8 Done Done Sent false
34304 10247 Acked Flush 0 Done Done Sent false
34305 10248 Acked Write 3 Done Done Sent false
34306 10249 Acked Read 5 Done Done Sent false
34307 10250 Acked Write 2 Done Done Sent false
34308 10251 Acked Write 1 Done Done Sent false
34309 10252 Acked Write 3 Done Done Sent false
34310 10253 Acked Write 4 Done Done Sent false
34311 10254 Acked Read 3 Done Done Sent false
34312 10255 Acked Write 10 Done Done Sent false
34313 10256 Acked Write 3 Done Done Sent false
34314 10257 Acked Read 7 Done Done Sent false
34315 10258 Acked Read 8 Done Done Sent false
34316 10259 Acked Write 2 Done Done Sent false
34317 10260 Acked Flush 0 Done Done Sent false
34318 10261 Acked Read 4 Done Done Sent false
34319 10262 Acked Write 10 Done Done Sent false
34320 10263 Acked Write 3 Done Done Sent false
34321 10264 Acked Read 7 Done Done Sent false
34322 10265 Acked Write 6 Done Done Sent false
34323 10266 Acked Write 4 Done Done Sent false
34324 10267 Acked Read 9 Done Done Sent false
34325 10268 Acked Read 4 Done Done Sent false
34326 10269 Acked Read 9 Done Done Sent false
34327 10270 Acked Read 6 Done Done Sent false
34328 10271 Acked Write 4 Done Done Sent false
34329 10272 Acked Read 6 Done Done Sent false
34330 10273 Acked Write 9 Done Done Sent false
34331 10274 Acked Write 10 Done Done Sent false
34332 10275 Acked Write 2 Done Done Sent false
34333 10276 Acked Read 6 Done Done Sent false
34334 10277 Acked Write 4 Done Done Sent false
34335 10278 Acked Write 4 Done Done Sent false
34336 10279 Acked Read 1 Done Done Sent false
34337 10280 Acked Flush 0 Done Done Sent false
34338 10281 Acked Write 6 Done Done Sent false
34339 10282 Acked Read 3 Done Done Sent false
34340 10283 Acked Flush 0 Done Done Sent false
34341 10284 Acked Write 9 Done Done Sent false
34342 10285 Acked Read 5 Done Done Sent false
34343 10286 Acked Read 9 Done Done Sent false
34344 10287 Acked Flush 0 Done Done Sent false
34345 10288 Acked Write 3 Done Done Sent false
34346 10289 Acked Read 2 Done Done Sent false
34347 10290 Acked Read 9 Done Done Sent false
34348 10291 Acked Write 8 Done Done Sent false
34349 10292 Acked Flush 0 Done Done Sent false
34350 10293 Acked Read 4 Done Done Sent false
34351 10294 Acked Read 1 Done Done Sent false
34352 10295 Acked Read 9 Done Done Sent false
34353 10296 Acked Write 4 Done Done Sent false
34354 10297 Acked Read 6 Done Done Sent false
34355 10298 Acked Read 2 Done Done Sent false
34356 10299 Acked Read 9 Done Done Sent false
34357 10300 Acked Read 9 Done Done Sent false
34358 10301 Acked Read 3 Done Done Sent false
34359 10302 Acked Read 6 Done Done Sent false
34360 10303 Acked Write 8 Done Done Sent false
34361 10304 Acked Read 8 Done Done Sent false
34362 10305 Acked Flush 0 Done Done Sent false
34363 10306 Acked Flush 0 Done Done Sent false
34364 10307 Acked Flush 0 Done Done Sent false
34365 10308 Acked Write 1 Done Done Sent false
34366 10309 Acked Write 5 Done Done Sent false
34367 10310 Acked Write 4 Done Done Sent false
34368 10311 Acked Read 4 Done Done Sent false
34369 10312 Acked Read 1 Done Done Sent false
34370 10313 Acked Read 1 Done Done Sent false
34371 10314 Acked Write 3 Done Done Sent false
34372 10315 Acked Read 10 Done Done Sent false
34373 10316 Acked Write 8 Done Done Sent false
34374 10317 Acked Read 9 Done Done Sent false
34375 10318 Acked Write 2 Done Done Sent false
34376 10319 Acked Read 10 Done Done Sent false
34377 10320 Acked Read 2 Done Done Sent false
34378 10321 Acked Write 1 Done Done Sent false
34379 10322 Acked Flush 0 Done Done Sent false
34380 10323 Acked Read 5 Done Done Sent false
34381 10324 Acked Write 9 Done Done Sent false
34382 10325 Acked Flush 0 Done Done Sent false
34383 10326 Acked Read 4 Done Done Sent false
34384 10327 Acked Read 10 Done Done Sent false
34385 10328 Acked Read 10 Done Done Sent false
34386 10329 Acked Read 10 Done Done Sent false
34387 10330 Acked Read 3 Done Done Sent false
34388 10331 Acked Read 4 Done Done Sent false
34389 10332 Acked Flush 0 Done Done Sent false
34390 10333 Acked Write 2 Done Done Sent false
34391 10334 Acked Read 10 Done Done Sent false
34392 10335 Acked Write 1 Done Done Sent false
34393 10336 Acked Write 7 Done Done Sent false
34394 10337 Acked Write 5 Done Done Sent false
34395 10338 Acked Flush 0 Done Done Sent false
34396 10339 Acked Flush 0 Done Done Sent false
34397 10340 Acked Read 1 Done Done Sent false
34398 10341 Acked Flush 0 Done Done Sent false
34399 10342 Acked Write 6 Done Done Sent false
34400 10343 Acked Write 1 Done Done Sent false
34401 10344 Acked Write 7 Done Done Sent false
34402 10345 Acked Read 2 Done Done Sent false
34403 10346 Acked Read 9 Done Done Sent false
34404 10347 Acked Write 10 Done Done Sent false
34405 10348 Acked Write 2 Done Done Sent false
34406 10349 Acked Read 5 Done Done Sent false
34407 10350 Acked Read 5 Done Done Sent false
34408 10351 Acked Read 9 Done Done Sent false
34409 10352 Acked Read 10 Done Done Sent false
34410 10353 Acked Flush 0 Done Done Sent false
34411 10354 Acked Write 7 Done Done Sent false
34412 10355 Acked Write 2 Done Done Sent false
34413 10356 Acked Write 10 Done Done Sent false
34414 10357 Acked Read 4 Done Done Sent false
34415 10358 Acked Write 10 Done Done Sent false
34416 STATES DS:0 DS:1 DS:2 TOTAL
34417 Sent 0 0 301 301
34418 Done 301 301 0 602
34419 Skipped 0 0 0 0
34420 Error 0 0 0 0
34421 Last Flush: Some(JobId(10353)) Some(JobId(10353)) Some(JobId(10057))
34422 Downstairs last five completed: 10057 10056 10055 10054 10053
34423 Upstairs last five completed: 10358 10357 10356 10355 10354
34424 CLIENT: Up:0 ds:301 act:2
344252025-02-25T17:25:42.967ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
344262025-02-25T17:25:43.731ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
344272025-02-25T17:25:43.731ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43652) = io task client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
344282025-02-25T17:25:43.732ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
344292025-02-25T17:25:43.733ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
344302025-02-25T17:25:43.733ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
344312025-02-25T17:25:43.733ZINFOcrucible: send last flush ID to this DS: Some(JobId(10057)) = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
344322025-02-25T17:25:43.733ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
344332025-02-25T17:25:43.733ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(10057)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
344342025-02-25T17:25:43.733ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(10057)) = downstairs session_id = df7322ab-c489-4662-a11c-3ea4e180b6a3
34435 ----------------------------------------------------------------
34436 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
34437 Downstairs last five completed: 10359 10358 10357 10356 10355
34438 Upstairs last five completed: 10359 10358 10357 10356 10355
34439 CLIENT: Up:0 ds:0 act:3
34440 Replay: All jobs finished
34441 Test replay has completed
34442 Wrote out file "/tmp/test_replay_verify.log"
34443 CLIENT: Tests done. All submitted work has been ACK'd
34444 ----------------------------------------------------------------
34445 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
34446 Downstairs last five completed: 10359 10358 10357 10356 10355
34447 Upstairs last five completed: 10359 10358 10357 10356 10355
34448 CLIENT: Up:0 ds:0 act:3
34449 CLIENT: All jobs finished, all DS active.
34450 Replays:30 time: 7:28
34451 Do final verify
34452 Feb 25 17:25:46.056 INFO accepted connection, remote_addr: 127.0.0.1:45723, local_addr: 127.0.0.1:9998
34453 Feb 25 17:25:46.056 INFO request completed, latency_us: 214, response_code: 200, uri: /regioninfo, method: GET, req_id: c3ecc414-6710-4cb6-95a7-1062e16cbee1, remote_addr: 127.0.0.1:45723, local_addr: 127.0.0.1:9998
34454 Feb 25 17:25:46.057 INFO request completed, latency_us: 102{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 }","v":0,"name":"crucible","level":30,"time":"2025-02-25T17:25:46.05756408Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1244,"task":"crutest"}
34455 , response_code: 200, uri: /regioncount, method: GET, req_id: fa74a6f9-b8e8-42fb-860e-e05770dee982, remote_addr: 127.0.0.1:45723, local_addr: 127.0.0.1:9998
344562025-02-25T17:25:46.058ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
34457 Feb 25 17:25:46.058 INFO request completed, latency_us: 127, response_code: 200, uri: /port/cid/0, method: GET, req_id: 67df5ac7-a50e-494b-948e-814dde4d680c, remote_addr: 127.0.0.1:45723, local_addr: 127.0.0.1:9998
34458 Feb 25 17:25:46.059 INFO request completed, latency_us: 109, response_code: 200, uri: /port/cid/1, method: GET, req_id: 4084202f-7bfd-4594-b2fe-7dfa0d3c79f5, remote_addr: 127.0.0.1:45723, local_addr: 127.0.0.1:9998
34459 Feb 25 17:25:46.059 INFO request completed, latency_us: 118, response_code: 200, uri: /port/cid/2, method: GET, req_id: 1e7530e0-4806-4c68-8081-bfbef32186cc, remote_addr: 127.0.0.1:45723, local_addr: 127.0.0.1:9998
344602025-02-25T17:25:46.059ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
344612025-02-25T17:25:46.060ZINFOcrucible: Upstairs starts
344622025-02-25T17:25:46.060ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "47aa4c863f974975b86a01cc6893b0f87d94aae1", git_commit_timestamp: "2025-02-25T16:27:18.000000000Z", git_branch: "renovate/thiserror-1.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
344632025-02-25T17:25:46.060ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
344642025-02-25T17:25:46.060ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
344652025-02-25T17:25:46.060ZINFOcrucible: Crucible 91a0f9f5-064c-4ccb-9828-0a4c7160e70d has session id: 37ebfe4e-5178-4804-b1d1-57b8f21df6e1 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344662025-02-25T17:25:46.060ZINFOcrucible: Upstairs opts: Upstairs UUID: 91a0f9f5-064c-4ccb-9828-0a4c7160e70d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344672025-02-25T17:25:46.060ZINFOcrucible: Crucible stats registered with UUID: 91a0f9f5-064c-4ccb-9828-0a4c7160e70d session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344682025-02-25T17:25:46.061ZWARNcrucible: could not find Downstairs address for Nexus session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344692025-02-25T17:25:46.062ZINFOcrucible: Downstairs in volume = 3 task = crutest
344702025-02-25T17:25:46.062ZINFOcrucible: The guest has requested activation with gen:3
344712025-02-25T17:25:46.062ZINFOcrucible: 91a0f9f5-064c-4ccb-9828-0a4c7160e70d active request set session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344722025-02-25T17:25:46.062ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344732025-02-25T17:25:46.062ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344742025-02-25T17:25:46.062ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344752025-02-25T17:25:46.062ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344762025-02-25T17:25:46.062ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344772025-02-25T17:25:46.062ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344782025-02-25T17:25:46.062ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344792025-02-25T17:25:46.062ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344802025-02-25T17:25:46.062ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344812025-02-25T17:25:46.062ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65209) = io task client = 0 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344822025-02-25T17:25:46.062ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58352) = io task client = 2 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344832025-02-25T17:25:46.062ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43644) = io task client = 1 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344842025-02-25T17:25:46.063ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344852025-02-25T17:25:46.063ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344862025-02-25T17:25:46.063ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344872025-02-25T17:25:46.063ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344882025-02-25T17:25:46.064ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344892025-02-25T17:25:46.064ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344902025-02-25T17:25:46.064ZINFOcrucible: Waiting for more clients to be ready session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344912025-02-25T17:25:46.064ZINFOcrucible: Waiting for more clients to be ready session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344922025-02-25T17:25:46.064ZINFOcrucible: [0]R flush_numbers[0..12]: [923, 896, 920, 919, 910, 934, 912, 934, 933, 909, 921, 927] = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344932025-02-25T17:25:46.064ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344942025-02-25T17:25:46.064ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344952025-02-25T17:25:46.064ZINFOcrucible: [1]R flush_numbers[0..12]: [923, 896, 920, 919, 910, 934, 912, 934, 933, 909, 921, 927] = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344962025-02-25T17:25:46.064ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344972025-02-25T17:25:46.064ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344982025-02-25T17:25:46.064ZINFOcrucible: [2]R flush_numbers[0..12]: [923, 896, 920, 919, 910, 934, 912, 934, 933, 909, 921, 927] = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
344992025-02-25T17:25:46.064ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
345002025-02-25T17:25:46.064ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
345012025-02-25T17:25:46.064ZINFOcrucible: Max found gen is 3 = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
345022025-02-25T17:25:46.064ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
345032025-02-25T17:25:46.064ZINFOcrucible: Next flush: 935 = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
345042025-02-25T17:25:46.064ZINFOcrucible: All extents match = downstairs session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
345052025-02-25T17:25:46.064ZINFOcrucible: No downstairs reconciliation required session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
345062025-02-25T17:25:46.065ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
34507 {Wait for a query_work_queue command to finish before sending IO
34508 "msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-02-25T17:25:46.065032666Z","hostname":"w-01JMZ1S36N5PABBMBP949XFW1G","pid":1244,"client":"1","":"downstairs","session_id":"37ebfe4e-5178-4804-b1d1-57b8f21df6e1"}
345092025-02-25T17:25:46.065ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
345102025-02-25T17:25:46.065ZINFOcrucible: All required reconciliation work is completed session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
345112025-02-25T17:25:46.065ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
345122025-02-25T17:25:46.065ZINFOcrucible: 91a0f9f5-064c-4ccb-9828-0a4c7160e70d is now active with session: 37ebfe4e-5178-4804-b1d1-57b8f21df6e1 session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
345132025-02-25T17:25:46.065ZINFOcrucible: Set Active after no reconciliation session_id = 37ebfe4e-5178-4804-b1d1-57b8f21df6e1
345142025-02-25T17:25:46.065ZINFOcrucible: The guest has finished waiting for activation with:3
34515 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
34516 Loading write count information from file "/tmp/test_replay_verify.log"
34517 Read and Verify all blocks (0..5000 range:false)
34518 Wrote out file "/tmp/test_replay_verify.log"
34519 Verify test completed
34520 Wrote out file "/tmp/test_replay_verify.log"
34521 CLIENT: Tests done. All submitted work has been ACK'd
34522 ----------------------------------------------------------------
34523 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
34524 States: Active Active Active
34525 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34526 1000 Acked Read 256 Done Done Done false
34527 1001 Acked Read 256 Done Done Done false
34528 1002 Acked Read 256 Done Done Done false
34529 1003 Acked Read 256 Done Done Done false
34530 1004 Acked Read 256 Done Done Done false
34531 1005 Acked Read 256 Done Done Done false
34532 1006 Acked Read 256 Done Done Done false
34533 1007 Acked Read 256 Done Done Done false
34534 1008 Acked Read 256 Done Done Done false
34535 1009 Acked Read 256 Done Done Done false
34536 1010 Acked Read 256 Done Done Done false
34537 1011 Acked Read 256 Done Done Done false
34538 1012 Acked Read 256 Done Done Done false
34539 1013 Acked Read 256 Done Done Done false
34540 1014 Acked Read 256 Done Done Done false
34541 1015 Acked Read 256 Done Done Done false
34542 1016 Acked Read 256 Done Done Done false
34543 1017 Acked Read 256 Done Done Done false
34544 1018 Acked Read 136 Done Done Done false
34545 1019 Acked Read 256 Done Done Done false
34546 STATES DS:0 DS:1 DS:2 TOTAL
34547 Sent 0 0 0 0
34548 Done 20 20 20 60
34549 Skipped 0 0 0 0
34550 Error 0 0 0 0
34551 Last Flush: None None None
34552 Downstairs last five completed:
34553 Upstairs last five completed: 1019 1018 1017 1016 1015
34554 CLIENT: Up:0 ds:20 act:3
34555 ----------------------------------------------------------------
34556 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
34557 Downstairs last five completed: 1020 1019 1018 1017 1016
34558 Upstairs last five completed: 1020 1019 1018 1017 1016
34559 CLIENT: Up:0 ds:0 act:3
34560 CLIENT: All crucible jobs finished, exiting program
34561 Feb 25 17:25:50.532 INFO accepted connection, remote_addr: 127.0.0.1:42785, local_addr: 127.0.0.1:9998
34562 Main task has work to do, go find it
34563 got dsc Shutdown
34564 Shutdown
34565 Feb 25 17:25:50.533 INFO request completed, latency_us: 163, response_code: 204, uri: /shutdown, method: POST, req_id: bd3c09d9-4913-42a6-9194-bcd2719fdcb2, remote_addr: 127.0.0.1:42785, local_addr: 127.0[8810] Disable keep_running
34566 .Shut it down
34567 0.1:9998
34568 [8810] Got stop action so:false kr:false
34569 [8820] Disable keep_running
34570 [8820] Got stop action so:false kr:false
34571 [8830] Disable keep_running
34572 [8830] Got stop action so:false kr:false
34573 [8830] Exited with: ExitStatus(unix_wait_status(9))
34574 [8820] Exited with: ExitStatus(unix_wait_status(9))
34575 [8830][2] reports Exit
34576 [8810] Exited with: ExitStatus(unix_wait_status(9))
34577 [8820][1] reports Exit
34578 [8810][0] reports Exit
34579 February 25, 2025 at 05:25:55 PM UTC Test ends with 0